2024-04-26T22:48:11.057 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T22:48:11.057 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_rishabh@teuthology --name rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/multimds_misc} -- /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279/orig.config.yaml 2024-04-26T22:48:11.104 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T22:48:11.194 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279 branch: wip-rishabh-testing-20240426.111959 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/multimds_misc} email: null first_in_suite: false job_id: '7675279' kernel: client: branch: testing flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240426.111959 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Scrub error on inode sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240426.111959 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 816 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 sleep_before_teardown: 0 subset: 166/512 suite: fs suite_branch: wip-rishabh-testing-20240426.111959 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCmwad1FdyL8tEL6I5NyZOm1iLJXHdqc4gYsXpz/ZVh+8X0V5pjDOmkQYIHl1kwFTeIuREUa686wdnNywdeHJM= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKq3pNTox0Ssyv/rsyIIVqcF0i9bOossn7/BwWuFayw7fcHef7wXimfAPIVgFS6INsRneFpNwvEHH4OFhGppse8= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_multimds_misc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_19:30:57 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T22:48:11.194 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa; will attempt to use it 2024-04-26T22:48:11.195 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks 2024-04-26T22:48:11.195 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T22:48:11.197 INFO:teuthology.task.internal:Checking packages... 2024-04-26T22:48:11.224 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bc612ffb9c1d5bc27536bc917a705262a36d4387' 2024-04-26T22:48:11.224 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T22:48:11.224 INFO:teuthology.packaging:ref: None 2024-04-26T22:48:11.224 INFO:teuthology.packaging:tag: None 2024-04-26T22:48:11.224 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240426.111959 2024-04-26T22:48:11.224 INFO:teuthology.packaging:sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:48:11.224 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240426.111959 2024-04-26T22:48:11.374 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3306.gbc612ffb 2024-04-26T22:48:11.376 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T22:48:11.382 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T22:48:11.382 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T22:48:11.400 INFO:teuthology.task.internal:Saving configuration 2024-04-26T22:48:11.417 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T22:48:11.422 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T22:48:11.447 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 22:42:44.985132', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCmwad1FdyL8tEL6I5NyZOm1iLJXHdqc4gYsXpz/ZVh+8X0V5pjDOmkQYIHl1kwFTeIuREUa686wdnNywdeHJM='} 2024-04-26T22:48:11.470 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 22:42:44.986741', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKq3pNTox0Ssyv/rsyIIVqcF0i9bOossn7/BwWuFayw7fcHef7wXimfAPIVgFS6INsRneFpNwvEHH4OFhGppse8='} 2024-04-26T22:48:11.470 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T22:48:11.477 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-26T22:48:11.477 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-26T22:48:11.477 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T22:48:11.549 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fee7ef32160>, signals=[15]) 2024-04-26T22:48:11.550 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T22:48:11.556 INFO:teuthology.task.internal:Opening connections... 2024-04-26T22:48:11.556 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T22:48:11.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:48:11.634 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T22:48:11.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:48:11.705 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T22:48:11.711 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-04-26T22:48:11.729 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-04-26T22:48:11.729 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-04-26T22:48:11.787 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-04-26T22:48:11.787 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="9" 2024-04-26T22:48:11.787 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-04-26T22:48:11.788 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-04-26T22:48:11.788 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="9" 2024-04-26T22:48:11.788 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el9" 2024-04-26T22:48:11.788 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T22:48:11.788 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-04-26T22:48:11.788 INFO:teuthology.orchestra.run.smithi040.stdout:LOGO="fedora-logo-icon" 2024-04-26T22:48:11.788 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T22:48:11.788 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-04-26T22:48:11.788 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T22:48:11.788 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T22:48:11.788 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T22:48:11.789 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-04-26T22:48:11.813 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-04-26T22:48:11.831 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-04-26T22:48:11.831 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-04-26T22:48:11.888 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-04-26T22:48:11.888 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-04-26T22:48:11.888 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-04-26T22:48:11.888 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-04-26T22:48:11.888 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-04-26T22:48:11.888 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-04-26T22:48:11.888 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T22:48:11.888 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-04-26T22:48:11.888 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-04-26T22:48:11.888 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T22:48:11.888 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-04-26T22:48:11.888 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T22:48:11.888 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T22:48:11.889 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T22:48:11.889 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-04-26T22:48:11.913 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T22:48:11.923 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T22:48:11.931 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T22:48:11.931 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-04-26T22:48:11.934 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-04-26T22:48:11.949 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T22:48:11.955 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T22:48:11.955 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-04-26T22:48:11.991 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-04-26T22:48:12.096 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T22:48:12.150 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T22:48:12.164 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-26T22:48:12.165 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-04-26T22:48:12.165 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-04-26T22:48:12.165 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T22:48:12.165 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T22:48:12.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:48:12.182 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-26T22:48:12.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:48:12.186 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-26T22:48:12.416 DEBUG:teuthology.task.kernel:sha1 for client.1 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T22:48:12.417 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-26T22:48:12.417 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-04-26T22:48:12.433 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T22:48:12.433 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-26T22:48:12.433 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-04-26T22:48:12.434 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-437.el9.x86_64 2024-04-26T22:48:12.434 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T22:48:12.435 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-26T22:48:12.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum upgrade -y linux-firmware 2024-04-26T22:48:12.453 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-437.el9.x86_64 2024-04-26T22:48:12.453 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T22:48:12.453 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-26T22:48:12.453 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum upgrade -y linux-firmware 2024-04-26T22:48:14.667 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-26T22:48:15.051 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.0 MB 00:00 2024-04-26T22:48:18.958 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 6.7 MB/s | 19 MB 00:02 2024-04-26T22:48:21.276 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 3.5 MB/s | 19 MB 00:05 2024-04-26T22:48:23.517 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 6.6 MB/s | 6.2 MB 00:00 2024-04-26T22:48:25.669 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.2 MB 00:00 2024-04-26T22:48:26.035 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-04-26T22:48:27.320 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-26T22:48:27.630 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-26T22:48:28.412 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-26T22:48:32.786 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 10 kB/s | 1.7 kB 00:00 2024-04-26T22:48:33.712 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-26T22:48:34.730 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-26T22:48:34.731 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-26T22:48:34.731 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-26T22:48:34.863 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.1... 2024-04-26T22:48:34.864 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T22:48:35.082 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-26T22:48:35.083 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-26T22:48:35.085 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi103.front.sepia.ceph.com to complete... 2024-04-26T22:48:35.783 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-26T22:48:35.785 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-26T22:48:35.785 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-26T22:48:35.893 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T22:48:35.893 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T22:48:36.040 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-26T22:48:36.041 DEBUG:teuthology.orchestra.run.smithi040:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-26T22:48:36.043 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi040.front.sepia.ceph.com to complete... 2024-04-26T22:49:03.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26 22:49:03 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-26T22:49:03.558 INFO:teuthology.orchestra.run.smithi103.stderr:FINISHED --2024-04-26 22:49:03-- 2024-04-26T22:49:03.558 INFO:teuthology.orchestra.run.smithi103.stderr:Total wall clock time: 28s 2024-04-26T22:49:03.558 INFO:teuthology.orchestra.run.smithi103.stderr:Downloaded: 1 files, 341M in 28s (12.2 MB/s) 2024-04-26T22:49:03.569 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.1... 2024-04-26T22:49:03.570 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-26T22:49:05.789 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying... ######################################## 2024-04-26T22:49:05.912 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing... ######################################## 2024-04-26T22:49:06.022 INFO:teuthology.orchestra.run.smithi103.stdout:Updating / installing... 2024-04-26T22:49:13.985 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-26T22:49:20.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26 22:49:20 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-26T22:49:20.312 INFO:teuthology.orchestra.run.smithi040.stderr:FINISHED --2024-04-26 22:49:20-- 2024-04-26T22:49:20.312 INFO:teuthology.orchestra.run.smithi040.stderr:Total wall clock time: 44s 2024-04-26T22:49:20.312 INFO:teuthology.orchestra.run.smithi040.stderr:Downloaded: 1 files, 341M in 43s (7.87 MB/s) 2024-04-26T22:49:20.315 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T22:49:20.315 DEBUG:teuthology.orchestra.run.smithi040:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-26T22:49:22.521 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying... ######################################## 2024-04-26T22:49:22.703 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing... ######################################## 2024-04-26T22:49:22.835 INFO:teuthology.orchestra.run.smithi040.stdout:Updating / installing... 2024-04-26T22:49:31.123 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-26T22:49:51.771 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-26T22:49:51.771 DEBUG:teuthology.orchestra.run.smithi103:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-26T22:49:51.857 INFO:teuthology.orchestra.run.smithi103.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-26T22:49:51.857 INFO:teuthology.orchestra.run.smithi103.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-26T22:49:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-26T22:49:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-26T22:49:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-26T22:49:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-26T22:49:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-26T22:49:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-26T22:49:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-26T22:49:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-26T22:49:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-26T22:49:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-26T22:49:51.858 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-26T22:49:51.859 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-26T22:49:51.859 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-26T22:49:51.859 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-26T22:49:51.859 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-26T22:49:51.859 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-26T22:49:51.859 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-26T22:49:51.859 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-26T22:49:51.859 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-26T22:49:51.859 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-26T22:49:51.859 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-26T22:49:51.859 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-26T22:49:51.859 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-26T22:49:51.860 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-26T22:49:51.860 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-26T22:49:51.860 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-26T22:49:51.860 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-26T22:49:51.860 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-26T22:49:51.860 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-26T22:49:51.860 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-26T22:49:51.860 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-26T22:49:51.860 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-26T22:49:51.860 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-26T22:49:51.861 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-26T22:49:51.861 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-26T22:49:51.861 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-26T22:49:51.861 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-26T22:49:51.861 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-26T22:49:51.861 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-26T22:49:51.861 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-26T22:49:51.861 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-26T22:49:51.861 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-26T22:49:51.861 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-26T22:49:51.861 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-26T22:49:51.861 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-26T22:49:51.861 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-26T22:49:51.862 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-26T22:49:51.862 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-26T22:49:51.862 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-26T22:49:51.862 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-26T22:49:51.862 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-26T22:49:51.862 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-26T22:49:51.862 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-26T22:49:51.862 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-26T22:49:51.862 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-26T22:49:51.862 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-26T22:49:51.862 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-26T22:49:51.862 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-26T22:49:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-26T22:49:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-26T22:49:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-26T22:49:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-26T22:49:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-26T22:49:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-26T22:49:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-26T22:49:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-26T22:49:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-26T22:49:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-26T22:49:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-26T22:49:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-26T22:49:51.863 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-26T22:49:51.864 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-26T22:49:51.865 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-26T22:49:51.866 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-26T22:49:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-26T22:49:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-26T22:49:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-26T22:49:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-26T22:49:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-26T22:49:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-26T22:49:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-26T22:49:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-26T22:49:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-26T22:49:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-26T22:49:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-26T22:49:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-26T22:49:51.867 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-26T22:49:51.868 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-26T22:49:51.869 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-26T22:49:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-26T22:49:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-26T22:49:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-26T22:49:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-26T22:49:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-26T22:49:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-26T22:49:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-26T22:49:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-26T22:49:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-26T22:49:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-26T22:49:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-26T22:49:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-26T22:49:51.870 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-26T22:49:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-26T22:49:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-26T22:49:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-26T22:49:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-26T22:49:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-26T22:49:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-26T22:49:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-26T22:49:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-26T22:49:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-26T22:49:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-26T22:49:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-26T22:49:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-26T22:49:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-26T22:49:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-26T22:49:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-26T22:49:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-26T22:49:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-26T22:49:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-26T22:49:51.872 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-26T22:49:51.873 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-26T22:49:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-26T22:49:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-26T22:49:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-26T22:49:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-26T22:49:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-26T22:49:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-26T22:49:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-26T22:49:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-26T22:49:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-26T22:49:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-26T22:49:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-26T22:49:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-26T22:49:51.874 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-26T22:49:51.875 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-26T22:49:51.876 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-26T22:49:51.876 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-26T22:49:51.876 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-26T22:49:51.876 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-26T22:49:51.876 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-26T22:49:51.876 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-26T22:49:51.876 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-26T22:49:51.876 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-26T22:49:51.876 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-26T22:49:51.876 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-26T22:49:51.876 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-26T22:49:51.876 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-26T22:49:51.876 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-26T22:49:51.877 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-26T22:49:51.877 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-26T22:49:51.877 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-26T22:49:51.877 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-26T22:49:51.877 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-26T22:49:51.877 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-26T22:49:51.877 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-26T22:49:51.877 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-26T22:49:51.877 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-26T22:49:51.877 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-26T22:49:51.877 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-26T22:49:51.877 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-26T22:49:51.877 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-26T22:49:51.878 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-26T22:49:51.878 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-26T22:49:51.878 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-26T22:49:51.878 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-26T22:49:51.878 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-26T22:49:51.878 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-26T22:49:51.878 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-26T22:49:51.878 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-26T22:49:51.878 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-26T22:49:51.878 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-26T22:49:51.878 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-26T22:49:51.878 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-26T22:49:51.878 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-26T22:49:51.879 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-26T22:49:51.879 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-26T22:49:51.879 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-26T22:49:51.879 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-26T22:49:51.879 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-26T22:49:51.879 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-26T22:49:51.879 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-26T22:49:51.879 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-26T22:49:51.879 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-26T22:49:51.879 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-26T22:49:51.879 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-26T22:49:51.879 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-26T22:49:51.879 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-26T22:49:51.880 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-26T22:49:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-26T22:49:51.882 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-26T22:49:51.882 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-26T22:49:51.882 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-26T22:49:51.882 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-26T22:49:51.882 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-26T22:49:51.882 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-26T22:49:51.882 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-26T22:49:51.882 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-26T22:49:51.882 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-26T22:49:51.882 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-26T22:49:51.882 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-26T22:49:51.882 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-26T22:49:51.882 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-26T22:49:51.883 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-26T22:49:51.884 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-26T22:49:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-26T22:49:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-26T22:49:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-26T22:49:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-26T22:49:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-26T22:49:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-26T22:49:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-26T22:49:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-26T22:49:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-26T22:49:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-26T22:49:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-26T22:49:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-26T22:49:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-26T22:49:51.886 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-26T22:49:51.887 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-26T22:49:51.888 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-26T22:49:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-26T22:49:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-26T22:49:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-26T22:49:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-26T22:49:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-26T22:49:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-26T22:49:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-26T22:49:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-26T22:49:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-26T22:49:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-26T22:49:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-26T22:49:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-26T22:49:51.889 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-26T22:49:51.890 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-26T22:49:51.891 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-26T22:49:51.892 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-26T22:49:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-26T22:49:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-26T22:49:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-26T22:49:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-26T22:49:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-26T22:49:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-26T22:49:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-26T22:49:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-26T22:49:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-26T22:49:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-26T22:49:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-26T22:49:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-26T22:49:51.893 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-26T22:49:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-26T22:49:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-26T22:49:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-26T22:49:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-26T22:49:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-26T22:49:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-26T22:49:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-26T22:49:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-26T22:49:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-26T22:49:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-26T22:49:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-26T22:49:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-26T22:49:51.894 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-26T22:49:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-26T22:49:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-26T22:49:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-26T22:49:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-26T22:49:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-26T22:49:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-26T22:49:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-26T22:49:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-26T22:49:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-26T22:49:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-26T22:49:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-26T22:49:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-26T22:49:51.895 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-26T22:49:51.896 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-26T22:49:51.897 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-26T22:49:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-26T22:49:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-26T22:49:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-26T22:49:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-26T22:49:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-26T22:49:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-26T22:49:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-26T22:49:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-26T22:49:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-26T22:49:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-26T22:49:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-26T22:49:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-26T22:49:51.898 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-26T22:49:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-26T22:49:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-26T22:49:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-26T22:49:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-26T22:49:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-26T22:49:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-26T22:49:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-26T22:49:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-26T22:49:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-26T22:49:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-26T22:49:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-26T22:49:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-26T22:49:51.899 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-26T22:49:51.900 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-26T22:49:51.901 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-26T22:49:51.902 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-26T22:49:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-26T22:49:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-26T22:49:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-26T22:49:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-26T22:49:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-26T22:49:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-26T22:49:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-26T22:49:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-26T22:49:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-26T22:49:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-26T22:49:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-26T22:49:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-26T22:49:51.903 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-26T22:49:51.904 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-26T22:49:51.905 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-26T22:49:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-26T22:49:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-26T22:49:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-26T22:49:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-26T22:49:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-26T22:49:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-26T22:49:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-26T22:49:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-26T22:49:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-26T22:49:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-26T22:49:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-26T22:49:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-26T22:49:51.906 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-26T22:49:51.907 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-26T22:49:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-26T22:49:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-26T22:49:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-26T22:49:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-26T22:49:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-26T22:49:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-26T22:49:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-26T22:49:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-26T22:49:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-26T22:49:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-26T22:49:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-26T22:49:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-26T22:49:51.908 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-26T22:49:51.909 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-26T22:49:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-26T22:49:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-26T22:49:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-26T22:49:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-26T22:49:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-26T22:49:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-26T22:49:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-26T22:49:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-26T22:49:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-26T22:49:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-26T22:49:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-26T22:49:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-26T22:49:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-26T22:49:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-26T22:49:51.912 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-26T22:49:51.913 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-26T22:49:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-26T22:49:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-26T22:49:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-26T22:49:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-26T22:49:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-26T22:49:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-26T22:49:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-26T22:49:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-26T22:49:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-26T22:49:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-26T22:49:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-26T22:49:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-26T22:49:51.914 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-26T22:49:51.915 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-26T22:49:51.916 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-26T22:49:51.917 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-26T22:49:51.918 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-26T22:49:51.918 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-26T22:49:51.918 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-26T22:49:51.918 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-26T22:49:51.918 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-26T22:49:51.918 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-26T22:49:51.918 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-26T22:49:51.918 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-26T22:49:51.918 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-26T22:49:51.918 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-26T22:49:51.918 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-26T22:49:51.918 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-26T22:49:51.918 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-26T22:49:51.919 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-26T22:49:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-26T22:49:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-26T22:49:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-26T22:49:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-26T22:49:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-26T22:49:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-26T22:49:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-26T22:49:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-26T22:49:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-26T22:49:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-26T22:49:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-26T22:49:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-26T22:49:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-26T22:49:51.922 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-26T22:49:51.923 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-26T22:49:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-26T22:49:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-26T22:49:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-26T22:49:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-26T22:49:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-26T22:49:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-26T22:49:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-26T22:49:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-26T22:49:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-26T22:49:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-26T22:49:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-26T22:49:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-26T22:49:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-26T22:49:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-26T22:49:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-26T22:49:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-26T22:49:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-26T22:49:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-26T22:49:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-26T22:49:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-26T22:49:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-26T22:49:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-26T22:49:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-26T22:49:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-26T22:49:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-26T22:49:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-26T22:49:51.926 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-26T22:49:51.927 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-26T22:49:51.928 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-26T22:49:51.929 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-26T22:49:51.930 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-26T22:49:51.931 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-26T22:49:51.932 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-26T22:49:51.933 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-26T22:49:51.934 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-26T22:49:51.934 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-26T22:49:51.934 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-26T22:49:51.934 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-26T22:49:51.934 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-26T22:49:51.934 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-26T22:49:51.934 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-26T22:49:51.934 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-26T22:49:51.934 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-26T22:49:51.934 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-26T22:49:51.934 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-26T22:49:51.934 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-26T22:49:51.934 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-26T22:49:51.935 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-26T22:49:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-26T22:49:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-26T22:49:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-26T22:49:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-26T22:49:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-26T22:49:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-26T22:49:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-26T22:49:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-26T22:49:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-26T22:49:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-26T22:49:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-26T22:49:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-26T22:49:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-26T22:49:51.937 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-26T22:49:51.938 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-26T22:49:51.939 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-26T22:49:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-26T22:49:51.941 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-26T22:49:51.942 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-26T22:49:51.943 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-26T22:49:51.944 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-26T22:49:51.945 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-26T22:49:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-26T22:49:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-26T22:49:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-26T22:49:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-26T22:49:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-26T22:49:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-26T22:49:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-26T22:49:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-26T22:49:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-26T22:49:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-26T22:49:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-26T22:49:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-26T22:49:51.946 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-26T22:49:51.947 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-26T22:49:51.948 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-26T22:49:51.949 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-26T22:49:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-26T22:49:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-26T22:49:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-26T22:49:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-26T22:49:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-26T22:49:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-26T22:49:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-26T22:49:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-26T22:49:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-26T22:49:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-26T22:49:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-26T22:49:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-26T22:49:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-26T22:49:51.951 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-26T22:49:51.952 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-26T22:49:51.953 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-26T22:49:51.954 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-26T22:49:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-26T22:49:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-26T22:49:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-26T22:49:51.958 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-26T22:49:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-26T22:49:51.960 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-26T22:49:51.961 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-26T22:49:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-26T22:49:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-26T22:49:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-26T22:49:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-26T22:49:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-26T22:49:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-26T22:49:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-26T22:49:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-26T22:49:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-26T22:49:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-26T22:49:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-26T22:49:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-26T22:49:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-26T22:49:51.963 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-26T22:49:51.964 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-26T22:49:51.964 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-26T22:49:51.964 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-26T22:49:51.964 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-26T22:49:51.964 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-26T22:49:51.964 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-26T22:49:51.964 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-26T22:49:51.964 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-26T22:49:51.964 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-26T22:49:51.964 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-26T22:49:51.964 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-26T22:49:51.964 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-26T22:49:51.964 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-26T22:49:51.965 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-26T22:49:51.966 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-26T22:49:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-26T22:49:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-26T22:49:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-26T22:49:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-26T22:49:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-26T22:49:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-26T22:49:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-26T22:49:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-26T22:49:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-26T22:49:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-26T22:49:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-26T22:49:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-26T22:49:51.967 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-26T22:49:51.968 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-26T22:49:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-26T22:49:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-26T22:49:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-26T22:49:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-26T22:49:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-26T22:49:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-26T22:49:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-26T22:49:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-26T22:49:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-26T22:49:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-26T22:49:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-26T22:49:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-26T22:49:51.969 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-26T22:49:51.970 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-26T22:49:51.971 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-26T22:49:51.972 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-26T22:49:51.973 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-26T22:49:51.974 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-26T22:49:51.975 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-26T22:49:51.976 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-26T22:49:51.977 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-26T22:49:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-26T22:49:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-26T22:49:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-26T22:49:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-26T22:49:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-26T22:49:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-26T22:49:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-26T22:49:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-26T22:49:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-26T22:49:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-26T22:49:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-26T22:49:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-26T22:49:51.978 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-26T22:49:51.979 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-26T22:49:51.980 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-26T22:49:51.981 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-26T22:49:51.982 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-26T22:49:51.983 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-26T22:49:51.984 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-26T22:49:51.985 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-26T22:49:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-26T22:49:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-26T22:49:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-26T22:49:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-26T22:49:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-26T22:49:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-26T22:49:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-26T22:49:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-26T22:49:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-26T22:49:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-26T22:49:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-26T22:49:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-26T22:49:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-26T22:49:51.987 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-26T22:49:51.988 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-26T22:49:51.989 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-26T22:49:51.990 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-26T22:49:51.991 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-26T22:49:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-26T22:49:51.993 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-26T22:49:51.994 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-26T22:49:51.995 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-26T22:49:51.996 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-26T22:49:51.997 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-26T22:49:51.998 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-26T22:49:51.999 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-26T22:49:51.999 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-26T22:49:51.999 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T22:49:52.003 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-26T22:49:52.003 DEBUG:teuthology.orchestra.run.smithi103:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-26T22:49:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:postinstall scriptlet (using /bin/sh): 2024-04-26T22:49:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T22:49:52.026 INFO:teuthology.orchestra.run.smithi103.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T22:49:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-04-26T22:49:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:for file in vmlinuz System.map config; do 2024-04-26T22:49:52.026 INFO:teuthology.orchestra.run.smithi103.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-26T22:49:52.026 INFO:teuthology.orchestra.run.smithi103.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-26T22:49:52.026 INFO:teuthology.orchestra.run.smithi103.stdout: fi 2024-04-26T22:49:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:done 2024-04-26T22:49:52.026 INFO:teuthology.orchestra.run.smithi103.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-26T22:49:52.026 INFO:teuthology.orchestra.run.smithi103.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-26T22:49:52.027 INFO:teuthology.orchestra.run.smithi103.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T22:49:52.027 INFO:teuthology.orchestra.run.smithi103.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T22:49:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-04-26T22:49:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-26T22:49:52.027 INFO:teuthology.orchestra.run.smithi103.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-26T22:49:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-04-26T22:49:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-26T22:49:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T22:49:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-26T22:49:52.027 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-04-26T22:49:52.028 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-04-26T22:49:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-04-26T22:49:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-04-26T22:49:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.06 2024-04-26T22:49:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 68.el9 2024-04-26T22:49:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-04-26T22:49:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-26T22:49:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Group : Unspecified 2024-04-26T22:49:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 8264254 2024-04-26T22:49:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-04-26T22:49:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-26T22:49:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-26T22:49:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-26T22:49:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-26T22:49:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : builder@centos.org 2024-04-26T22:49:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-04-26T22:49:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-26T22:49:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-04-26T22:49:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-04-26T22:49:52.110 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:49:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-26T22:49:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-26T22:49:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-26T22:49:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-04-26T22:49:52.110 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:49:52.110 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-04-26T22:49:52.111 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-26T22:49:52.111 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 6.9.0-rc3-g806ec36b2112 2024-04-26T22:49:52.112 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-26T22:49:52.654 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-04-26T22:49:54.026 INFO:teuthology.orchestra.run.smithi103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-26T22:49:54.060 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-04-26T22:49:54.063 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-04-26T22:49:54.081 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.33RExqhyvd 2024-04-26T22:49:54.081 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.33RExqhyvd 2024-04-26T22:49:54.172 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.33RExqhyvd 2024-04-26T22:49:54.274 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.33RExqhyvd is 6KB 2024-04-26T22:49:54.294 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.33RExqhyvd 2024-04-26T22:49:54.311 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-04-26T22:49:54.383 INFO:teuthology.orchestra.run.smithi103.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-26T22:49:54.383 INFO:teuthology.orchestra.run.smithi103.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-26T22:49:54.383 INFO:teuthology.orchestra.run.smithi103.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-26T22:49:54.383 INFO:teuthology.orchestra.run.smithi103.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-26T22:49:54.384 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-26T22:49:54.538 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-04-26T22:50:09.748 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-26T22:50:09.748 DEBUG:teuthology.orchestra.run.smithi040:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-26T22:50:09.820 INFO:teuthology.orchestra.run.smithi040.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-26T22:50:09.820 INFO:teuthology.orchestra.run.smithi040.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-26T22:50:09.820 INFO:teuthology.orchestra.run.smithi040.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-26T22:50:09.820 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-26T22:50:09.820 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-26T22:50:09.820 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-26T22:50:09.821 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-26T22:50:09.822 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-26T22:50:09.822 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-26T22:50:09.822 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-26T22:50:09.822 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-26T22:50:09.822 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-26T22:50:09.822 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-26T22:50:09.822 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-26T22:50:09.822 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-26T22:50:09.822 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-26T22:50:09.822 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-26T22:50:09.822 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-26T22:50:09.822 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-26T22:50:09.822 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-26T22:50:09.823 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-26T22:50:09.824 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-26T22:50:09.825 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-26T22:50:09.826 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-26T22:50:09.826 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-26T22:50:09.826 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-26T22:50:09.826 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-26T22:50:09.826 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-26T22:50:09.826 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-26T22:50:09.826 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-26T22:50:09.826 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-26T22:50:09.826 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-26T22:50:09.826 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-26T22:50:09.826 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-26T22:50:09.826 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-26T22:50:09.826 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-26T22:50:09.827 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-26T22:50:09.828 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-26T22:50:09.829 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-26T22:50:09.830 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-26T22:50:09.830 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-26T22:50:09.830 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-26T22:50:09.830 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-26T22:50:09.830 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-26T22:50:09.830 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-26T22:50:09.830 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-26T22:50:09.830 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-26T22:50:09.830 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-26T22:50:09.830 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-26T22:50:09.830 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-26T22:50:09.830 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-26T22:50:09.830 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-26T22:50:09.831 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-26T22:50:09.832 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-26T22:50:09.833 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-26T22:50:09.834 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-26T22:50:09.834 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-26T22:50:09.834 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-26T22:50:09.834 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-26T22:50:09.834 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-26T22:50:09.834 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-26T22:50:09.834 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-26T22:50:09.834 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-26T22:50:09.834 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-26T22:50:09.834 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-26T22:50:09.834 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-26T22:50:09.834 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-26T22:50:09.835 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-26T22:50:09.836 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-26T22:50:09.837 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-26T22:50:09.838 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-26T22:50:09.838 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-26T22:50:09.838 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-26T22:50:09.838 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-26T22:50:09.838 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-26T22:50:09.838 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-26T22:50:09.838 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-26T22:50:09.838 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-26T22:50:09.838 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-26T22:50:09.838 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-26T22:50:09.838 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-26T22:50:09.838 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-26T22:50:09.838 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-26T22:50:09.839 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-26T22:50:09.840 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-26T22:50:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-26T22:50:09.842 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-26T22:50:09.843 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-26T22:50:09.844 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-26T22:50:09.845 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-26T22:50:09.846 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-26T22:50:09.847 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-26T22:50:09.848 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-26T22:50:09.849 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-26T22:50:09.850 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-26T22:50:09.851 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-26T22:50:09.852 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-26T22:50:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-26T22:50:09.854 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-26T22:50:09.855 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-26T22:50:09.856 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-26T22:50:09.857 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-26T22:50:09.858 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-26T22:50:09.859 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-26T22:50:09.860 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-26T22:50:09.861 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-26T22:50:09.862 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-26T22:50:09.863 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-26T22:50:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-26T22:50:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-26T22:50:09.866 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-26T22:50:09.867 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-26T22:50:09.868 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-26T22:50:09.869 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-26T22:50:09.870 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-26T22:50:09.871 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-26T22:50:09.872 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-26T22:50:09.873 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-26T22:50:09.874 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-26T22:50:09.874 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-26T22:50:09.874 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-26T22:50:09.874 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-26T22:50:09.874 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-26T22:50:09.874 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-26T22:50:09.874 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-26T22:50:09.874 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-26T22:50:09.874 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-26T22:50:09.874 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-26T22:50:09.874 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-26T22:50:09.874 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-26T22:50:09.874 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-26T22:50:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-26T22:50:09.876 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-26T22:50:09.877 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-26T22:50:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-26T22:50:09.879 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-26T22:50:09.880 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-26T22:50:09.881 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-26T22:50:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-26T22:50:09.883 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-26T22:50:09.884 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-26T22:50:09.885 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-26T22:50:09.886 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-26T22:50:09.887 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-26T22:50:09.888 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-26T22:50:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-26T22:50:09.890 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-26T22:50:09.891 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-26T22:50:09.892 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-26T22:50:09.893 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-26T22:50:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-26T22:50:09.895 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-26T22:50:09.896 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-26T22:50:09.897 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-26T22:50:09.898 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-26T22:50:09.899 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-26T22:50:09.900 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-26T22:50:09.901 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-26T22:50:09.902 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-26T22:50:09.903 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-26T22:50:09.904 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-26T22:50:09.905 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-26T22:50:09.906 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-26T22:50:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-26T22:50:09.908 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-26T22:50:09.909 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-26T22:50:09.910 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-26T22:50:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-26T22:50:09.912 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-26T22:50:09.913 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-26T22:50:09.914 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-26T22:50:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-26T22:50:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-26T22:50:09.917 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-26T22:50:09.918 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-26T22:50:09.919 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-26T22:50:09.920 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-26T22:50:09.921 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-26T22:50:09.922 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-26T22:50:09.923 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-26T22:50:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-26T22:50:09.925 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-26T22:50:09.926 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-26T22:50:09.927 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-26T22:50:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-26T22:50:09.929 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-26T22:50:09.930 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-26T22:50:09.931 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-26T22:50:09.932 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-26T22:50:09.933 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-26T22:50:09.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-26T22:50:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-26T22:50:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-26T22:50:09.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-26T22:50:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-26T22:50:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-26T22:50:09.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-26T22:50:09.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-26T22:50:09.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-26T22:50:09.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-26T22:50:09.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-26T22:50:09.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-26T22:50:09.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-26T22:50:09.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-26T22:50:09.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-26T22:50:09.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-26T22:50:09.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-26T22:50:09.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-26T22:50:09.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-26T22:50:09.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-26T22:50:09.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T22:50:09.954 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-26T22:50:09.954 DEBUG:teuthology.orchestra.run.smithi040:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-26T22:50:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:postinstall scriptlet (using /bin/sh): 2024-04-26T22:50:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T22:50:09.977 INFO:teuthology.orchestra.run.smithi040.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T22:50:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:fi 2024-04-26T22:50:09.977 INFO:teuthology.orchestra.run.smithi040.stdout:for file in vmlinuz System.map config; do 2024-04-26T22:50:09.977 INFO:teuthology.orchestra.run.smithi040.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-26T22:50:09.978 INFO:teuthology.orchestra.run.smithi040.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-26T22:50:09.978 INFO:teuthology.orchestra.run.smithi040.stdout: fi 2024-04-26T22:50:09.978 INFO:teuthology.orchestra.run.smithi040.stdout:done 2024-04-26T22:50:09.978 INFO:teuthology.orchestra.run.smithi040.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-26T22:50:09.978 INFO:teuthology.orchestra.run.smithi040.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-26T22:50:09.978 INFO:teuthology.orchestra.run.smithi040.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T22:50:09.978 INFO:teuthology.orchestra.run.smithi040.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T22:50:09.978 INFO:teuthology.orchestra.run.smithi040.stdout:fi 2024-04-26T22:50:09.978 INFO:teuthology.orchestra.run.smithi040.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-26T22:50:09.978 INFO:teuthology.orchestra.run.smithi040.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-26T22:50:09.978 INFO:teuthology.orchestra.run.smithi040.stdout:fi 2024-04-26T22:50:09.978 INFO:teuthology.orchestra.run.smithi040.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-26T22:50:09.978 INFO:teuthology.orchestra.run.smithi040.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T22:50:09.979 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-26T22:50:09.979 INFO:teuthology.orchestra.run.smithi040.stdout:fi 2024-04-26T22:50:09.979 DEBUG:teuthology.orchestra.run.smithi040:> sudo rpm -qi grub2-tools 2024-04-26T22:50:10.056 INFO:teuthology.orchestra.run.smithi040.stdout:Name : grub2-tools 2024-04-26T22:50:10.056 INFO:teuthology.orchestra.run.smithi040.stdout:Epoch : 1 2024-04-26T22:50:10.056 INFO:teuthology.orchestra.run.smithi040.stdout:Version : 2.06 2024-04-26T22:50:10.056 INFO:teuthology.orchestra.run.smithi040.stdout:Release : 68.el9 2024-04-26T22:50:10.056 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: x86_64 2024-04-26T22:50:10.056 INFO:teuthology.orchestra.run.smithi040.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-26T22:50:10.056 INFO:teuthology.orchestra.run.smithi040.stdout:Group : Unspecified 2024-04-26T22:50:10.056 INFO:teuthology.orchestra.run.smithi040.stdout:Size : 8264254 2024-04-26T22:50:10.056 INFO:teuthology.orchestra.run.smithi040.stdout:License : GPLv3+ 2024-04-26T22:50:10.056 INFO:teuthology.orchestra.run.smithi040.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-26T22:50:10.056 INFO:teuthology.orchestra.run.smithi040.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout:Packager : builder@centos.org 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout:Vendor : CentOS 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout:Summary : Support tools for GRUB. 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout:Description : 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout:hardware devices. 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:50:10.057 INFO:teuthology.orchestra.run.smithi040.stdout:This subpackage provides tools for support of all platforms. 2024-04-26T22:50:10.058 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-26T22:50:10.058 INFO:teuthology.task.kernel:Updating grub on smithi040 to boot 6.9.0-rc3-g806ec36b2112 2024-04-26T22:50:10.058 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-26T22:50:10.646 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2024-04-26T22:50:12.166 INFO:teuthology.orchestra.run.smithi040.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-26T22:50:12.183 INFO:teuthology.orchestra.run.smithi040.stderr:done 2024-04-26T22:50:12.185 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-04-26T22:50:12.241 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.Ob8zzGOdur 2024-04-26T22:50:12.242 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Ob8zzGOdur 2024-04-26T22:50:12.320 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.Ob8zzGOdur 2024-04-26T22:50:12.475 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.Ob8zzGOdur is 6KB 2024-04-26T22:50:12.495 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.Ob8zzGOdur 2024-04-26T22:50:12.510 DEBUG:teuthology.orchestra.run.smithi040:> sudo /bin/ls /boot/loader/entries || true 2024-04-26T22:50:12.571 INFO:teuthology.orchestra.run.smithi040.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-26T22:50:12.571 INFO:teuthology.orchestra.run.smithi040.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-26T22:50:12.571 INFO:teuthology.orchestra.run.smithi040.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-26T22:50:12.571 INFO:teuthology.orchestra.run.smithi040.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-26T22:50:12.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-26T22:50:12.723 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2024-04-26T22:50:24.548 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-04-26T22:50:24.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-26T22:50:24.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:50:42.742 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-26T22:50:42.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-26T22:50:42.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:50:43.106 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-04-26T22:50:52.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-26T22:50:52.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:50:55.170 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-04-26T22:51:01.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-04-26T22:51:07.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-26T22:51:07.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:51:10.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-26T22:51:10.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:52:07.231 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T22:52:10.289 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T22:52:22.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-26T22:52:22.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:52:22.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-26T22:52:22.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:52:22.639 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-26T22:52:22.687 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-04-26T22:52:23.245 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-26T22:52:23.245 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.9.0-rc3-g806ec36b2112"... 2024-04-26T22:52:23.245 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-04-26T22:52:23.299 INFO:teuthology.orchestra.run.smithi103.stdout:6.9.0-rc3-g806ec36b2112 2024-04-26T22:52:23.299 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-26T22:52:23.299 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T22:52:23.299 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-26T22:52:23.427 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-26T22:52:23.427 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-26T22:52:23.427 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-04-26T22:52:23.484 INFO:teuthology.orchestra.run.smithi040.stdout:6.9.0-rc3-g806ec36b2112 2024-04-26T22:52:23.484 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-26T22:52:23.484 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T22:52:23.484 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-26T22:52:24.300 DEBUG:teuthology.parallel:result is None 2024-04-26T22:52:24.485 DEBUG:teuthology.parallel:result is None 2024-04-26T22:52:24.486 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T22:52:24.496 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T22:52:24.497 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T22:52:24.504 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T22:52:24.560 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T22:52:24.568 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T22:52:24.574 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T22:52:24.574 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T22:52:24.580 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T22:52:24.646 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T22:52:24.653 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T22:52:24.653 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T22:52:24.660 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T22:52:24.789 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T22:52:24.796 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T22:52:24.839 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T22:52:24.858 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T22:52:24.863 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T22:52:24.872 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T22:52:24.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T22:52:24.877 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T22:52:24.964 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T22:52:24.974 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T22:52:24.974 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T22:52:24.979 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T22:52:25.035 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T22:52:25.125 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T22:52:25.226 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T22:52:25.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T22:52:25.404 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T22:52:25.404 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T22:52:25.493 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T22:52:25.581 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T22:52:25.675 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T22:52:25.760 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T22:52:25.843 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T22:52:25.843 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T22:52:25.925 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-04-26T22:52:25.933 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-04-26T22:52:26.042 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T22:52:26.075 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T22:52:26.537 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T22:52:26.545 INFO:teuthology.task.internal:Starting timer... 2024-04-26T22:52:26.545 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T22:52:26.588 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T22:52:26.597 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-26T22:52:26.597 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2024-04-26T22:52:26.706 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2024-04-26T22:52:26.710 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-04-26T22:52:26.826 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-04-26T22:52:26.831 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T22:52:26.831 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2024-04-26T22:52:26.895 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2024-04-26T22:52:26.895 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-04-26T22:52:26.967 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-04-26T22:52:26.968 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive'} 2024-04-26T22:52:26.968 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T22:52:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:52:27.041 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T22:52:27.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:52:27.121 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T22:52:27.121 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2024-04-26T22:52:27.199 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-04-26T22:52:27.268 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T22:52:27.278 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T22:52:27.296 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T22:52:27.297 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com,smithi103.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T22:57:41.769 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com')] 2024-04-26T22:57:41.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-26T22:57:41.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:57:41.882 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-04-26T22:57:42.034 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-26T22:57:42.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-26T22:57:42.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T22:57:42.141 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-26T22:57:42.305 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-26T22:57:42.305 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T22:57:42.316 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T22:57:42.316 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T22:57:42.317 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T22:57:42.324 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T22:57:42.324 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T22:57:42.414 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T22:57:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T22:57:42.452 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T22:57:42.464 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T22:57:42.518 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-04-26T22:57:42.543 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2024-04-26T22:57:42.548 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-04-26T22:57:42.573 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2024-04-26T22:57:42.589 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T22:57:42.617 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T22:57:42.632 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T22:57:42.662 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T22:57:42.770 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-26T22:57:42.779 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T22:57:42.779 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-26T22:57:42.779 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:57:42.779 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:57:42.779 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:57:42.779 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:57:42.812 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-04-26T22:57:42.821 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T22:57:42.821 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-26T22:57:42.821 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:57:42.821 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:57:42.822 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:57:42.822 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:57:42.823 INFO:teuthology.run_tasks:Running task install... 2024-04-26T22:57:42.833 DEBUG:teuthology.task.install:project ceph 2024-04-26T22:57:42.833 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}} 2024-04-26T22:57:42.833 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'} 2024-04-26T22:57:42.833 INFO:teuthology.task.install:Using flavor: default 2024-04-26T22:57:42.840 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T22:57:42.840 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-26T22:57:42.840 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T22:57:42.841 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:57:42.843 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T22:57:42.843 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:57:42.986 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T22:57:42.986 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T22:57:43.056 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T22:57:43.057 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T22:57:43.158 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T22:57:43.158 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T22:57:43.158 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T22:57:43.164 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T22:57:43.164 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T22:57:43.164 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T22:57:43.252 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T22:57:43.253 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T22:57:43.255 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T22:57:43.255 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T22:57:43.384 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T22:57:43.385 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T22:57:43.495 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T22:57:43.499 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T22:57:43.576 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-04-26T22:57:43.580 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-04-26T22:57:43.585 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2024-04-26T22:57:43.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2024-04-26T22:57:43.896 INFO:teuthology.orchestra.run.smithi103.stdout:48 files removed 2024-04-26T22:57:43.923 INFO:teuthology.orchestra.run.smithi040.stdout:48 files removed 2024-04-26T22:57:43.934 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T22:57:43.962 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T22:57:44.894 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 159 kB/s | 84 kB 00:00 2024-04-26T22:57:45.064 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 127 kB/s | 84 kB 00:00 2024-04-26T22:57:45.611 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 19 kB/s | 12 kB 00:00 2024-04-26T22:57:45.773 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 19 kB/s | 12 kB 00:00 2024-04-26T22:57:46.128 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 4.4 kB/s | 1.9 kB 00:00 2024-04-26T22:57:46.348 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-04-26T22:57:47.247 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 9.9 MB/s | 8.0 MB 00:00 2024-04-26T22:57:47.586 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 5.9 MB/s | 8.0 MB 00:01 2024-04-26T22:57:51.727 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 6.5 MB/s | 19 MB 00:02 2024-04-26T22:57:51.739 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 19 MB 00:03 2024-04-26T22:57:56.037 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-04-26T22:57:56.507 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 6.2 MB 00:00 2024-04-26T22:57:57.646 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 16 kB 00:00 2024-04-26T22:57:58.182 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-04-26T22:57:58.895 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-26T22:57:59.840 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-26T22:58:04.865 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-26T22:58:06.093 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-26T22:58:07.251 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T22:58:07.252 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T22:58:07.262 INFO:teuthology.orchestra.run.smithi103.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T22:58:07.263 INFO:teuthology.orchestra.run.smithi103.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T22:58:07.264 INFO:teuthology.orchestra.run.smithi103.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T22:58:07.265 INFO:teuthology.orchestra.run.smithi103.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T22:58:07.267 INFO:teuthology.orchestra.run.smithi103.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T22:58:07.269 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T22:58:07.270 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T22:58:07.333 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-26T22:58:07.347 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-26T22:58:07.347 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-26T22:58:07.347 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-26T22:58:07.347 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-26T22:58:07.347 INFO:teuthology.orchestra.run.smithi103.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T22:58:07.348 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T22:58:07.349 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T22:58:07.349 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T22:58:07.349 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T22:58:07.349 INFO:teuthology.orchestra.run.smithi103.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T22:58:07.349 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T22:58:07.349 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T22:58:07.349 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T22:58:07.349 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T22:58:07.349 INFO:teuthology.orchestra.run.smithi103.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T22:58:07.349 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T22:58:07.349 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T22:58:07.349 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T22:58:07.349 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T22:58:07.350 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T22:58:07.350 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T22:58:07.350 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T22:58:07.350 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T22:58:07.350 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T22:58:07.350 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T22:58:07.350 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T22:58:07.350 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T22:58:07.350 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T22:58:07.350 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-04-26T22:58:07.350 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T22:58:07.350 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T22:58:07.350 INFO:teuthology.orchestra.run.smithi103.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T22:58:07.351 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T22:58:07.352 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T22:58:07.352 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T22:58:07.352 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T22:58:07.352 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T22:58:07.352 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T22:58:07.352 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T22:58:07.352 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T22:58:07.352 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T22:58:07.352 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T22:58:07.352 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T22:58:07.352 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T22:58:07.352 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T22:58:07.352 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T22:58:07.353 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T22:58:07.354 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T22:58:07.354 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T22:58:07.354 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T22:58:07.354 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T22:58:07.354 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T22:58:07.354 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T22:58:07.354 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T22:58:07.354 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T22:58:07.354 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T22:58:07.354 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T22:58:07.354 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T22:58:07.354 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T22:58:07.354 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T22:58:07.355 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T22:58:07.356 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T22:58:07.357 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T22:58:07.358 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T22:58:07.358 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T22:58:07.358 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T22:58:07.358 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T22:58:07.358 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T22:58:07.358 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T22:58:07.358 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T22:58:07.358 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T22:58:07.358 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T22:58:07.358 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T22:58:07.358 INFO:teuthology.orchestra.run.smithi103.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T22:58:07.358 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T22:58:07.358 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T22:58:07.359 INFO:teuthology.orchestra.run.smithi103.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T22:58:07.359 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T22:58:07.359 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T22:58:07.359 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-04-26T22:58:07.359 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T22:58:07.359 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:58:07.359 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-26T22:58:07.359 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-26T22:58:07.359 INFO:teuthology.orchestra.run.smithi103.stdout:Install 148 Packages 2024-04-26T22:58:07.359 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 4 Packages 2024-04-26T22:58:07.359 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:58:07.360 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 222 M 2024-04-26T22:58:07.360 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-26T22:58:08.604 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T22:58:08.606 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T22:58:08.628 INFO:teuthology.orchestra.run.smithi040.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T22:58:08.629 INFO:teuthology.orchestra.run.smithi040.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T22:58:08.631 INFO:teuthology.orchestra.run.smithi040.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T22:58:08.632 INFO:teuthology.orchestra.run.smithi040.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T22:58:08.635 INFO:teuthology.orchestra.run.smithi040.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T22:58:08.639 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T22:58:08.640 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T22:58:08.641 INFO:teuthology.orchestra.run.smithi103.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-04-26T22:58:08.717 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-26T22:58:08.737 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-26T22:58:08.737 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-26T22:58:08.737 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-26T22:58:08.738 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-26T22:58:08.738 INFO:teuthology.orchestra.run.smithi040.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T22:58:08.738 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T22:58:08.738 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T22:58:08.738 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T22:58:08.738 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T22:58:08.738 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T22:58:08.738 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T22:58:08.738 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T22:58:08.738 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T22:58:08.738 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T22:58:08.738 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T22:58:08.739 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2024-04-26T22:58:08.740 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T22:58:08.741 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T22:58:08.742 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T22:58:08.743 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T22:58:08.743 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T22:58:08.743 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T22:58:08.743 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T22:58:08.743 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T22:58:08.743 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T22:58:08.743 INFO:teuthology.orchestra.run.smithi040.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T22:58:08.743 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T22:58:08.743 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T22:58:08.743 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T22:58:08.743 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T22:58:08.743 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T22:58:08.743 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T22:58:08.744 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T22:58:08.744 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T22:58:08.744 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T22:58:08.744 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T22:58:08.744 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T22:58:08.744 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T22:58:08.744 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T22:58:08.744 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T22:58:08.744 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T22:58:08.744 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T22:58:08.744 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T22:58:08.744 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T22:58:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T22:58:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T22:58:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T22:58:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T22:58:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T22:58:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T22:58:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T22:58:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T22:58:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T22:58:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T22:58:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T22:58:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T22:58:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T22:58:08.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T22:58:08.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T22:58:08.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T22:58:08.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T22:58:08.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T22:58:08.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T22:58:08.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T22:58:08.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T22:58:08.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T22:58:08.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T22:58:08.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T22:58:08.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T22:58:08.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T22:58:08.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T22:58:08.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T22:58:08.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T22:58:08.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T22:58:08.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T22:58:08.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T22:58:08.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T22:58:08.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T22:58:08.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T22:58:08.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T22:58:08.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T22:58:08.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T22:58:08.748 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T22:58:08.749 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T22:58:08.749 INFO:teuthology.orchestra.run.smithi040.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T22:58:08.749 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T22:58:08.749 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T22:58:08.749 INFO:teuthology.orchestra.run.smithi040.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T22:58:08.749 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T22:58:08.749 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T22:58:08.749 INFO:teuthology.orchestra.run.smithi040.stdout:Installing weak dependencies: 2024-04-26T22:58:08.749 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T22:58:08.750 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:58:08.750 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-26T22:58:08.750 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-26T22:58:08.750 INFO:teuthology.orchestra.run.smithi040.stdout:Install 148 Packages 2024-04-26T22:58:08.750 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 4 Packages 2024-04-26T22:58:08.750 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:58:08.751 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 222 M 2024-04-26T22:58:08.751 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-26T22:58:08.983 INFO:teuthology.orchestra.run.smithi103.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 2.5 MB/s | 889 kB 00:00 2024-04-26T22:58:09.059 INFO:teuthology.orchestra.run.smithi103.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 1.9 MB/s | 145 kB 00:00 2024-04-26T22:58:09.476 INFO:teuthology.orchestra.run.smithi103.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 5.7 MB/s | 2.4 MB 00:00 2024-04-26T22:58:09.601 INFO:teuthology.orchestra.run.smithi103.stdout:(5/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 4.3 MB/s | 5.4 MB 00:01 2024-04-26T22:58:09.751 INFO:teuthology.orchestra.run.smithi103.stdout:(6/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 5.5 MB/s | 1.5 MB 00:00 2024-04-26T22:58:10.180 INFO:teuthology.orchestra.run.smithi040.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 14 kB/s | 6.6 kB 00:00 2024-04-26T22:58:10.218 INFO:teuthology.orchestra.run.smithi103.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 7.6 MB/s | 4.7 MB 00:00 2024-04-26T22:58:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 817 kB/s | 889 kB 00:01 2024-04-26T22:58:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:(8/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 7.2 MB/s | 21 MB 00:02 2024-04-26T22:58:11.360 INFO:teuthology.orchestra.run.smithi103.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 384 kB/s | 25 kB 00:00 2024-04-26T22:58:11.403 INFO:teuthology.orchestra.run.smithi040.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 1.1 MB/s | 145 kB 00:00 2024-04-26T22:58:11.544 INFO:teuthology.orchestra.run.smithi103.stdout:(10/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 8.0 MB/s | 11 MB 00:01 2024-04-26T22:58:11.644 INFO:teuthology.orchestra.run.smithi103.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 2.2 MB/s | 225 kB 00:00 2024-04-26T22:58:11.728 INFO:teuthology.orchestra.run.smithi103.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 327 kB/s | 27 kB 00:00 2024-04-26T22:58:11.961 INFO:teuthology.orchestra.run.smithi103.stdout:(13/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 7.6 MB/s | 17 MB 00:02 2024-04-26T22:58:11.986 INFO:teuthology.orchestra.run.smithi103.stdout:(14/152): libcephfs-devel-19.0.0-3306.gbc612ffb 123 kB/s | 32 kB 00:00 2024-04-26T22:58:12.095 INFO:teuthology.orchestra.run.smithi103.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 1.5 MB/s | 163 kB 00:00 2024-04-26T22:58:12.137 INFO:teuthology.orchestra.run.smithi103.stdout:(16/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 4.2 MB/s | 747 kB 00:00 2024-04-26T22:58:12.171 INFO:teuthology.orchestra.run.smithi103.stdout:(17/152): librados-devel-19.0.0-3306.gbc612ffb. 1.7 MB/s | 127 kB 00:00 2024-04-26T22:58:12.195 INFO:teuthology.orchestra.run.smithi040.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 3.0 MB/s | 2.4 MB 00:00 2024-04-26T22:58:12.238 INFO:teuthology.orchestra.run.smithi103.stdout:(18/152): libradosstriper1-19.0.0-3306.gbc612ff 4.9 MB/s | 501 kB 00:00 2024-04-26T22:58:12.338 INFO:teuthology.orchestra.run.smithi103.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 455 kB/s | 45 kB 00:00 2024-04-26T22:58:12.455 INFO:teuthology.orchestra.run.smithi103.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 1.1 MB/s | 135 kB 00:00 2024-04-26T22:58:12.564 INFO:teuthology.orchestra.run.smithi103.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 1.5 MB/s | 161 kB 00:00 2024-04-26T22:58:12.689 INFO:teuthology.orchestra.run.smithi103.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 2.5 MB/s | 321 kB 00:00 2024-04-26T22:58:12.737 INFO:teuthology.orchestra.run.smithi040.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-04-26T22:58:12.797 INFO:teuthology.orchestra.run.smithi103.stdout:(23/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 8.6 MB/s | 5.4 MB 00:00 2024-04-26T22:58:12.839 INFO:teuthology.orchestra.run.smithi103.stdout:(24/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 1.9 MB/s | 298 kB 00:00 2024-04-26T22:58:12.881 INFO:teuthology.orchestra.run.smithi103.stdout:(25/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.2 MB/s | 99 kB 00:00 2024-04-26T22:58:12.924 INFO:teuthology.orchestra.run.smithi103.stdout:(26/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.0 MB/s | 85 kB 00:00 2024-04-26T22:58:13.049 INFO:teuthology.orchestra.run.smithi103.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 1.3 MB/s | 171 kB 00:00 2024-04-26T22:58:13.166 INFO:teuthology.orchestra.run.smithi103.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 295 kB/s | 34 kB 00:00 2024-04-26T22:58:13.250 INFO:teuthology.orchestra.run.smithi103.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.7 MB/s | 142 kB 00:00 2024-04-26T22:58:13.296 INFO:teuthology.orchestra.run.smithi040.stdout:(6/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-04-26T22:58:13.324 INFO:teuthology.orchestra.run.smithi103.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 7.0 MB/s | 3.1 MB 00:00 2024-04-26T22:58:13.808 INFO:teuthology.orchestra.run.smithi103.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 6.4 MB/s | 3.5 MB 00:00 2024-04-26T22:58:13.813 INFO:teuthology.orchestra.run.smithi040.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 4.4 MB/s | 4.7 MB 00:01 2024-04-26T22:58:13.909 INFO:teuthology.orchestra.run.smithi103.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 2.4 MB/s | 249 kB 00:00 2024-04-26T22:58:14.009 INFO:teuthology.orchestra.run.smithi103.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 497 kB/s | 49 kB 00:00 2024-04-26T22:58:14.209 INFO:teuthology.orchestra.run.smithi103.stdout:(34/152): ceph-mgr-diskprediction-local-19.0.0- 8.4 MB/s | 7.4 MB 00:00 2024-04-26T22:58:14.234 INFO:teuthology.orchestra.run.smithi103.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3306.gb 73 kB/s | 16 kB 00:00 2024-04-26T22:58:14.318 INFO:teuthology.orchestra.run.smithi103.stdout:(36/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 2.5 MB/s | 273 kB 00:00 2024-04-26T22:58:14.418 INFO:teuthology.orchestra.run.smithi103.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 184 kB/s | 18 kB 00:00 2024-04-26T22:58:14.485 INFO:teuthology.orchestra.run.smithi103.stdout:(38/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 3.0 MB/s | 762 kB 00:00 2024-04-26T22:58:15.311 INFO:teuthology.orchestra.run.smithi103.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 90 kB/s | 80 kB 00:00 2024-04-26T22:58:15.604 INFO:teuthology.orchestra.run.smithi040.stdout:(8/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 5.9 MB/s | 11 MB 00:01 2024-04-26T22:58:15.671 INFO:teuthology.orchestra.run.smithi040.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 384 kB/s | 25 kB 00:00 2024-04-26T22:58:16.253 INFO:teuthology.orchestra.run.smithi103.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 208 kB/s | 195 kB 00:00 2024-04-26T22:58:16.319 INFO:teuthology.orchestra.run.smithi103.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 438 kB/s | 803 kB 00:01 2024-04-26T22:58:16.945 INFO:teuthology.orchestra.run.smithi103.stdout:(42/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 8.8 MB/s | 49 MB 00:05 2024-04-26T22:58:17.172 INFO:teuthology.orchestra.run.smithi040.stdout:(10/152): ceph-common-19.0.0-3306.gbc612ffb.el9 2.8 MB/s | 21 MB 00:07 2024-04-26T22:58:17.255 INFO:teuthology.orchestra.run.smithi040.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 2.7 MB/s | 225 kB 00:00 2024-04-26T22:58:17.498 INFO:teuthology.orchestra.run.smithi040.stdout:(12/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 4.0 MB/s | 17 MB 00:04 2024-04-26T22:58:17.539 INFO:teuthology.orchestra.run.smithi040.stdout:(13/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 96 kB/s | 27 kB 00:00 2024-04-26T22:58:17.564 INFO:teuthology.orchestra.run.smithi040.stdout:(14/152): libcephfs-devel-19.0.0-3306.gbc612ffb 488 kB/s | 32 kB 00:00 2024-04-26T22:58:17.658 INFO:teuthology.orchestra.run.smithi040.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 1.7 MB/s | 163 kB 00:00 2024-04-26T22:58:17.723 INFO:teuthology.orchestra.run.smithi040.stdout:(16/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 4.0 MB/s | 747 kB 00:00 2024-04-26T22:58:17.745 INFO:teuthology.orchestra.run.smithi103.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 170 kB/s | 253 kB 00:01 2024-04-26T22:58:17.757 INFO:teuthology.orchestra.run.smithi040.stdout:(17/152): librados-devel-19.0.0-3306.gbc612ffb. 1.3 MB/s | 127 kB 00:00 2024-04-26T22:58:17.832 INFO:teuthology.orchestra.run.smithi040.stdout:(18/152): libradosstriper1-19.0.0-3306.gbc612ff 4.5 MB/s | 501 kB 00:00 2024-04-26T22:58:17.862 INFO:teuthology.orchestra.run.smithi103.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 129 kB/s | 106 kB 00:00 2024-04-26T22:58:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 917 kB/s | 45 kB 00:00 2024-04-26T22:58:17.932 INFO:teuthology.orchestra.run.smithi040.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 2.7 MB/s | 135 kB 00:00 2024-04-26T22:58:17.983 INFO:teuthology.orchestra.run.smithi040.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 3.2 MB/s | 161 kB 00:00 2024-04-26T22:58:18.091 INFO:teuthology.orchestra.run.smithi040.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 2.9 MB/s | 321 kB 00:00 2024-04-26T22:58:18.167 INFO:teuthology.orchestra.run.smithi040.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 3.9 MB/s | 298 kB 00:00 2024-04-26T22:58:18.251 INFO:teuthology.orchestra.run.smithi040.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.2 MB/s | 99 kB 00:00 2024-04-26T22:58:18.310 INFO:teuthology.orchestra.run.smithi040.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.5 MB/s | 85 kB 00:00 2024-04-26T22:58:18.668 INFO:teuthology.orchestra.run.smithi040.stdout:(26/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 5.9 MB/s | 5.4 MB 00:00 2024-04-26T22:58:18.683 INFO:teuthology.orchestra.run.smithi103.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 144 kB/s | 135 kB 00:00 2024-04-26T22:58:18.726 INFO:teuthology.orchestra.run.smithi040.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 2.9 MB/s | 171 kB 00:00 2024-04-26T22:58:18.777 INFO:teuthology.orchestra.run.smithi040.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 696 kB/s | 34 kB 00:00 2024-04-26T22:58:18.804 INFO:teuthology.orchestra.run.smithi103.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 133 kB/s | 125 kB 00:00 2024-04-26T22:58:18.885 INFO:teuthology.orchestra.run.smithi040.stdout:(29/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 5.4 MB/s | 3.1 MB 00:00 2024-04-26T22:58:18.911 INFO:teuthology.orchestra.run.smithi040.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.0 MB/s | 142 kB 00:00 2024-04-26T22:58:19.096 INFO:teuthology.orchestra.run.smithi103.stdout:(47/152): python3-cryptography-36.0.1-4.el9.x86 452 kB/s | 1.2 MB 00:02 2024-04-26T22:58:19.263 INFO:teuthology.orchestra.run.smithi103.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 1.5 MB/s | 689 kB 00:00 2024-04-26T22:58:19.380 INFO:teuthology.orchestra.run.smithi103.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 934 kB/s | 107 kB 00:00 2024-04-26T22:58:19.394 INFO:teuthology.orchestra.run.smithi040.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 7.0 MB/s | 3.5 MB 00:00 2024-04-26T22:58:19.455 INFO:teuthology.orchestra.run.smithi103.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 289 kB/s | 21 kB 00:00 2024-04-26T22:58:19.478 INFO:teuthology.orchestra.run.smithi040.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 3.0 MB/s | 249 kB 00:00 2024-04-26T22:58:19.528 INFO:teuthology.orchestra.run.smithi040.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 1.0 MB/s | 49 kB 00:00 2024-04-26T22:58:19.570 INFO:teuthology.orchestra.run.smithi040.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 404 kB/s | 16 kB 00:00 2024-04-26T22:58:19.586 INFO:teuthology.orchestra.run.smithi103.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 2.4 MB/s | 313 kB 00:00 2024-04-26T22:58:19.633 INFO:teuthology.orchestra.run.smithi103.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 1.7 MB/s | 943 kB 00:00 2024-04-26T22:58:19.654 INFO:teuthology.orchestra.run.smithi040.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 3.2 MB/s | 273 kB 00:00 2024-04-26T22:58:19.692 INFO:teuthology.orchestra.run.smithi103.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 313 kB/s | 33 kB 00:00 2024-04-26T22:58:19.750 INFO:teuthology.orchestra.run.smithi103.stdout:(54/152): python3-urllib3-1.26.5-5.el9.noarch.r 201 kB/s | 215 kB 00:01 2024-04-26T22:58:19.796 INFO:teuthology.orchestra.run.smithi040.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 5.3 MB/s | 762 kB 00:00 2024-04-26T22:58:19.809 INFO:teuthology.orchestra.run.smithi103.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 157 kB/s | 18 kB 00:00 2024-04-26T22:58:19.838 INFO:teuthology.orchestra.run.smithi040.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 448 kB/s | 18 kB 00:00 2024-04-26T22:58:19.959 INFO:teuthology.orchestra.run.smithi103.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-04-26T22:58:20.001 INFO:teuthology.orchestra.run.smithi103.stdout:(57/152): libaio-devel-0.3.111-13.el9.x86_64.rp 45 kB/s | 11 kB 00:00 2024-04-26T22:58:20.005 INFO:teuthology.orchestra.run.smithi040.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 6.8 MB/s | 7.4 MB 00:01 2024-04-26T22:58:20.030 INFO:teuthology.orchestra.run.smithi040.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 420 kB/s | 80 kB 00:00 2024-04-26T22:58:20.093 INFO:teuthology.orchestra.run.smithi103.stdout:(58/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.5 MB/s | 3.0 MB 00:00 2024-04-26T22:58:20.118 INFO:teuthology.orchestra.run.smithi103.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 286 kB/s | 45 kB 00:00 2024-04-26T22:58:20.122 INFO:teuthology.orchestra.run.smithi040.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-26T22:58:20.194 INFO:teuthology.orchestra.run.smithi103.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-04-26T22:58:20.197 INFO:teuthology.orchestra.run.smithi040.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-04-26T22:58:20.243 INFO:teuthology.orchestra.run.smithi103.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 3.8 MB/s | 582 kB 00:00 2024-04-26T22:58:20.248 INFO:teuthology.orchestra.run.smithi040.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 803 kB 00:00 2024-04-26T22:58:20.269 INFO:teuthology.orchestra.run.smithi103.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 170 kB/s | 25 kB 00:00 2024-04-26T22:58:20.294 INFO:teuthology.orchestra.run.smithi103.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 476 kB/s | 47 kB 00:00 2024-04-26T22:58:20.298 INFO:teuthology.orchestra.run.smithi040.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-26T22:58:20.328 INFO:teuthology.orchestra.run.smithi103.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-26T22:58:20.357 INFO:teuthology.orchestra.run.smithi040.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 7.8 MB/s | 1.2 MB 00:00 2024-04-26T22:58:20.361 INFO:teuthology.orchestra.run.smithi103.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-04-26T22:58:20.387 INFO:teuthology.orchestra.run.smithi103.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 385 kB/s | 35 kB 00:00 2024-04-26T22:58:20.390 INFO:teuthology.orchestra.run.smithi040.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 135 kB 00:00 2024-04-26T22:58:20.432 INFO:teuthology.orchestra.run.smithi040.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-26T22:58:20.466 INFO:teuthology.orchestra.run.smithi040.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-04-26T22:58:20.662 INFO:teuthology.orchestra.run.smithi103.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-04-26T22:58:20.837 INFO:teuthology.orchestra.run.smithi103.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2024-04-26T22:58:20.879 INFO:teuthology.orchestra.run.smithi103.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 4.1 MB/s | 911 kB 00:00 2024-04-26T22:58:20.938 INFO:teuthology.orchestra.run.smithi103.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 133 kB 00:00 2024-04-26T22:58:20.941 INFO:teuthology.orchestra.run.smithi040.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 1.3 MB/s | 689 kB 00:00 2024-04-26T22:58:20.988 INFO:teuthology.orchestra.run.smithi103.stdout:(71/152): postgresql-13.14-1.el9.x86_64.rpm 2.6 MB/s | 1.6 MB 00:00 2024-04-26T22:58:21.000 INFO:teuthology.orchestra.run.smithi040.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 1.7 MB/s | 943 kB 00:00 2024-04-26T22:58:21.059 INFO:teuthology.orchestra.run.smithi040.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 919 kB/s | 107 kB 00:00 2024-04-26T22:58:21.088 INFO:teuthology.orchestra.run.smithi103.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 6.8 MB/s | 1.0 MB 00:00 2024-04-26T22:58:21.114 INFO:teuthology.orchestra.run.smithi103.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 703 kB/s | 87 kB 00:00 2024-04-26T22:58:21.121 INFO:teuthology.orchestra.run.smithi040.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 179 kB/s | 21 kB 00:00 2024-04-26T22:58:21.201 INFO:teuthology.orchestra.run.smithi040.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 2.2 MB/s | 313 kB 00:00 2024-04-26T22:58:21.251 INFO:teuthology.orchestra.run.smithi040.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 254 kB/s | 33 kB 00:00 2024-04-26T22:58:21.314 INFO:teuthology.orchestra.run.smithi103.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2024-04-26T22:58:21.368 INFO:teuthology.orchestra.run.smithi040.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 157 kB/s | 18 kB 00:00 2024-04-26T22:58:21.502 INFO:teuthology.orchestra.run.smithi040.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 84 kB/s | 11 kB 00:00 2024-04-26T22:58:21.531 INFO:teuthology.orchestra.run.smithi103.stdout:(75/152): postgresql-server-13.14-1.el9.x86_64. 8.9 MB/s | 5.8 MB 00:00 2024-04-26T22:58:21.566 INFO:teuthology.orchestra.run.smithi103.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 996 kB/s | 248 kB 00:00 2024-04-26T22:58:21.594 INFO:teuthology.orchestra.run.smithi040.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-04-26T22:58:21.698 INFO:teuthology.orchestra.run.smithi103.stdout:(77/152): python3-babel-2.9.1-2.el9.noarch.rpm 9.8 MB/s | 6.0 MB 00:00 2024-04-26T22:58:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:(57/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.9 MB/s | 3.0 MB 00:00 2024-04-26T22:58:21.723 INFO:teuthology.orchestra.run.smithi103.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 134 kB/s | 26 kB 00:00 2024-04-26T22:58:21.749 INFO:teuthology.orchestra.run.smithi103.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 104 kB/s | 19 kB 00:00 2024-04-26T22:58:21.782 INFO:teuthology.orchestra.run.smithi103.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-04-26T22:58:21.808 INFO:teuthology.orchestra.run.smithi103.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 415 kB/s | 35 kB 00:00 2024-04-26T22:58:21.900 INFO:teuthology.orchestra.run.smithi103.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-04-26T22:58:21.950 INFO:teuthology.orchestra.run.smithi103.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 222 kB 00:00 2024-04-26T22:58:22.000 INFO:teuthology.orchestra.run.smithi103.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 776 kB/s | 77 kB 00:00 2024-04-26T22:58:22.017 INFO:teuthology.orchestra.run.smithi103.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 637 kB/s | 42 kB 00:00 2024-04-26T22:58:22.093 INFO:teuthology.orchestra.run.smithi103.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 2.9 MB/s | 269 kB 00:00 2024-04-26T22:58:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-04-26T22:58:22.236 INFO:teuthology.orchestra.run.smithi040.stdout:(58/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 7.5 MB/s | 49 MB 00:06 2024-04-26T22:58:22.243 INFO:teuthology.orchestra.run.smithi103.stdout:(88/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-04-26T22:58:22.277 INFO:teuthology.orchestra.run.smithi103.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 1.5 MB/s | 279 kB 00:00 2024-04-26T22:58:22.278 INFO:teuthology.orchestra.run.smithi040.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 66 kB/s | 45 kB 00:00 2024-04-26T22:58:22.302 INFO:teuthology.orchestra.run.smithi103.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 292 kB/s | 51 kB 00:00 2024-04-26T22:58:22.327 INFO:teuthology.orchestra.run.smithi103.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 647 kB/s | 54 kB 00:00 2024-04-26T22:58:22.386 INFO:teuthology.orchestra.run.smithi103.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 503 kB/s | 42 kB 00:00 2024-04-26T22:58:22.420 INFO:teuthology.orchestra.run.smithi103.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 485 kB/s | 44 kB 00:00 2024-04-26T22:58:22.437 INFO:teuthology.orchestra.run.smithi040.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 910 kB/s | 662 kB 00:00 2024-04-26T22:58:22.478 INFO:teuthology.orchestra.run.smithi103.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.4 MB/s | 221 kB 00:00 2024-04-26T22:58:22.479 INFO:teuthology.orchestra.run.smithi040.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 2.4 MB/s | 582 kB 00:00 2024-04-26T22:58:22.503 INFO:teuthology.orchestra.run.smithi103.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 434 kB/s | 36 kB 00:00 2024-04-26T22:58:22.512 INFO:teuthology.orchestra.run.smithi040.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 109 kB/s | 25 kB 00:00 2024-04-26T22:58:22.538 INFO:teuthology.orchestra.run.smithi040.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 486 kB/s | 47 kB 00:00 2024-04-26T22:58:22.554 INFO:teuthology.orchestra.run.smithi103.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 637 kB/s | 47 kB 00:00 2024-04-26T22:58:22.579 INFO:teuthology.orchestra.run.smithi040.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 292 kB 00:00 2024-04-26T22:58:22.612 INFO:teuthology.orchestra.run.smithi103.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.8 MB/s | 305 kB 00:00 2024-04-26T22:58:22.613 INFO:teuthology.orchestra.run.smithi040.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-04-26T22:58:22.638 INFO:teuthology.orchestra.run.smithi103.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 699 kB/s | 58 kB 00:00 2024-04-26T22:58:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 355 kB/s | 35 kB 00:00 2024-04-26T22:58:22.696 INFO:teuthology.orchestra.run.smithi103.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 94 kB 00:00 2024-04-26T22:58:22.730 INFO:teuthology.orchestra.run.smithi103.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 699 kB/s | 64 kB 00:00 2024-04-26T22:58:22.788 INFO:teuthology.orchestra.run.smithi103.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 491 kB/s | 45 kB 00:00 2024-04-26T22:58:22.864 INFO:teuthology.orchestra.run.smithi103.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 168 kB/s | 22 kB 00:00 2024-04-26T22:58:23.047 INFO:teuthology.orchestra.run.smithi040.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-26T22:58:23.147 INFO:teuthology.orchestra.run.smithi040.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 8.4 MB/s | 4.7 MB 00:00 2024-04-26T22:58:23.239 INFO:teuthology.orchestra.run.smithi040.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-04-26T22:58:23.281 INFO:teuthology.orchestra.run.smithi103.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:01 2024-04-26T22:58:23.289 INFO:teuthology.orchestra.run.smithi040.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 3.7 MB/s | 911 kB 00:00 2024-04-26T22:58:23.331 INFO:teuthology.orchestra.run.smithi103.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.6 MB/s | 863 kB 00:00 2024-04-26T22:58:23.356 INFO:teuthology.orchestra.run.smithi040.stdout:(71/152): postgresql-13.14-1.el9.x86_64.rpm 2.2 MB/s | 1.6 MB 00:00 2024-04-26T22:58:23.373 INFO:teuthology.orchestra.run.smithi103.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-26T22:58:23.465 INFO:teuthology.orchestra.run.smithi040.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 815 kB/s | 87 kB 00:00 2024-04-26T22:58:23.473 INFO:teuthology.orchestra.run.smithi103.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.6 MB/s | 309 kB 00:00 2024-04-26T22:58:23.499 INFO:teuthology.orchestra.run.smithi103.stdout:(107/152): grpc-data-1.46.7-10.el9.noarch.rpm 118 kB/s | 19 kB 00:00 2024-04-26T22:58:23.532 INFO:teuthology.orchestra.run.smithi040.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2024-04-26T22:58:23.749 INFO:teuthology.orchestra.run.smithi103.stdout:(108/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 95 kB/s | 26 kB 00:00 2024-04-26T22:58:23.782 INFO:teuthology.orchestra.run.smithi040.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-04-26T22:58:23.818 INFO:teuthology.orchestra.run.smithi040.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 862 kB/s | 244 kB 00:00 2024-04-26T22:58:23.866 INFO:teuthology.orchestra.run.smithi103.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 132 kB/s | 49 kB 00:00 2024-04-26T22:58:23.925 INFO:teuthology.orchestra.run.smithi040.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-04-26T22:58:23.950 INFO:teuthology.orchestra.run.smithi040.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 192 kB/s | 26 kB 00:00 2024-04-26T22:58:24.059 INFO:teuthology.orchestra.run.smithi040.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 145 kB/s | 19 kB 00:00 2024-04-26T22:58:24.150 INFO:teuthology.orchestra.run.smithi103.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 169 kB/s | 67 kB 00:00 2024-04-26T22:58:24.268 INFO:teuthology.orchestra.run.smithi040.stdout:(79/152): python3-babel-2.9.1-2.el9.noarch.rpm 7.4 MB/s | 6.0 MB 00:00 2024-04-26T22:58:24.293 INFO:teuthology.orchestra.run.smithi040.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 149 kB/s | 35 kB 00:00 2024-04-26T22:58:24.317 INFO:teuthology.orchestra.run.smithi103.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 337 kB/s | 151 kB 00:00 2024-04-26T22:58:24.360 INFO:teuthology.orchestra.run.smithi040.stdout:(81/152): python3-mako-1.1.4-6.el9.noarch.rpm 421 kB/s | 172 kB 00:00 2024-04-26T22:58:24.430 INFO:teuthology.orchestra.run.smithi040.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.8 MB/s | 250 kB 00:00 2024-04-26T22:58:24.469 INFO:teuthology.orchestra.run.smithi040.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-04-26T22:58:24.528 INFO:teuthology.orchestra.run.smithi040.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 776 kB/s | 77 kB 00:00 2024-04-26T22:58:24.595 INFO:teuthology.orchestra.run.smithi040.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 339 kB/s | 42 kB 00:00 2024-04-26T22:58:24.645 INFO:teuthology.orchestra.run.smithi040.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 2.3 MB/s | 269 kB 00:00 2024-04-26T22:58:24.703 INFO:teuthology.orchestra.run.smithi040.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-04-26T22:58:24.710 INFO:teuthology.orchestra.run.smithi103.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 1.4 MB/s | 548 kB 00:00 2024-04-26T22:58:24.801 INFO:teuthology.orchestra.run.smithi103.stdout:(113/152): libarrow-9.0.0-11.el9.x86_64.rpm 3.1 MB/s | 4.4 MB 00:01 2024-04-26T22:58:24.838 INFO:teuthology.orchestra.run.smithi040.stdout:(88/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.0 MB/s | 5.1 MB 00:00 2024-04-26T22:58:24.851 INFO:teuthology.orchestra.run.smithi103.stdout:(114/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.2 MB/s | 839 kB 00:00 2024-04-26T22:58:24.876 INFO:teuthology.orchestra.run.smithi103.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 269 kB/s | 43 kB 00:00 2024-04-26T22:58:24.879 INFO:teuthology.orchestra.run.smithi040.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 1.2 MB/s | 279 kB 00:00 2024-04-26T22:58:24.902 INFO:teuthology.orchestra.run.smithi103.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 328 kB/s | 32 kB 00:00 2024-04-26T22:58:24.904 INFO:teuthology.orchestra.run.smithi040.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 255 kB/s | 51 kB 00:00 2024-04-26T22:58:24.938 INFO:teuthology.orchestra.run.smithi040.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 537 kB/s | 54 kB 00:00 2024-04-26T22:58:24.943 INFO:teuthology.orchestra.run.smithi103.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 229 kB/s | 15 kB 00:00 2024-04-26T22:58:24.977 INFO:teuthology.orchestra.run.smithi103.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-26T22:58:24.996 INFO:teuthology.orchestra.run.smithi040.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 456 kB/s | 42 kB 00:00 2024-04-26T22:58:25.019 INFO:teuthology.orchestra.run.smithi103.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-26T22:58:25.022 INFO:teuthology.orchestra.run.smithi040.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 532 kB/s | 44 kB 00:00 2024-04-26T22:58:25.061 INFO:teuthology.orchestra.run.smithi103.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.6 MB/s | 303 kB 00:00 2024-04-26T22:58:25.086 INFO:teuthology.orchestra.run.smithi103.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 332 kB/s | 35 kB 00:00 2024-04-26T22:58:25.089 INFO:teuthology.orchestra.run.smithi040.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.4 MB/s | 221 kB 00:00 2024-04-26T22:58:25.106 INFO:teuthology.orchestra.run.smithi040.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 435 kB/s | 36 kB 00:00 2024-04-26T22:58:25.120 INFO:teuthology.orchestra.run.smithi103.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.1 MB/s | 206 kB 00:00 2024-04-26T22:58:25.164 INFO:teuthology.orchestra.run.smithi040.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 638 kB/s | 47 kB 00:00 2024-04-26T22:58:25.248 INFO:teuthology.orchestra.run.smithi040.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.1 MB/s | 305 kB 00:00 2024-04-26T22:58:25.265 INFO:teuthology.orchestra.run.smithi040.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 581 kB/s | 58 kB 00:00 2024-04-26T22:58:25.304 INFO:teuthology.orchestra.run.smithi103.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 59 kB/s | 11 kB 00:00 2024-04-26T22:58:25.331 INFO:teuthology.orchestra.run.smithi040.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 94 kB 00:00 2024-04-26T22:58:25.337 INFO:teuthology.orchestra.run.smithi103.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 583 kB/s | 144 kB 00:00 2024-04-26T22:58:25.349 INFO:teuthology.orchestra.run.smithi040.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 768 kB/s | 64 kB 00:00 2024-04-26T22:58:25.407 INFO:teuthology.orchestra.run.smithi040.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 599 kB/s | 45 kB 00:00 2024-04-26T22:58:25.421 INFO:teuthology.orchestra.run.smithi103.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 5.7 MB/s | 2.0 MB 00:00 2024-04-26T22:58:25.424 INFO:teuthology.orchestra.run.smithi040.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 299 kB/s | 22 kB 00:00 2024-04-26T22:58:25.446 INFO:teuthology.orchestra.run.smithi103.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 127 kB/s | 18 kB 00:00 2024-04-26T22:58:25.471 INFO:teuthology.orchestra.run.smithi103.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 180 kB/s | 23 kB 00:00 2024-04-26T22:58:25.496 INFO:teuthology.orchestra.run.smithi103.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 262 kB/s | 19 kB 00:00 2024-04-26T22:58:25.522 INFO:teuthology.orchestra.run.smithi103.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 402 kB/s | 20 kB 00:00 2024-04-26T22:58:25.566 INFO:teuthology.orchestra.run.smithi040.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 5.3 MB/s | 863 kB 00:00 2024-04-26T22:58:25.647 INFO:teuthology.orchestra.run.smithi103.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 5.9 MB/s | 1.0 MB 00:00 2024-04-26T22:58:25.672 INFO:teuthology.orchestra.run.smithi103.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 266 kB/s | 46 kB 00:00 2024-04-26T22:58:25.698 INFO:teuthology.orchestra.run.smithi103.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 457 kB/s | 79 kB 00:00 2024-04-26T22:58:25.722 INFO:teuthology.orchestra.run.smithi103.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 786 kB/s | 58 kB 00:00 2024-04-26T22:58:25.781 INFO:teuthology.orchestra.run.smithi103.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-26T22:58:25.815 INFO:teuthology.orchestra.run.smithi103.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 142 kB/s | 16 kB 00:00 2024-04-26T22:58:25.840 INFO:teuthology.orchestra.run.smithi103.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 770 kB/s | 90 kB 00:00 2024-04-26T22:58:25.865 INFO:teuthology.orchestra.run.smithi103.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 345 kB/s | 28 kB 00:00 2024-04-26T22:58:25.891 INFO:teuthology.orchestra.run.smithi103.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 627 kB/s | 31 kB 00:00 2024-04-26T22:58:25.933 INFO:teuthology.orchestra.run.smithi103.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-04-26T22:58:25.950 INFO:teuthology.orchestra.run.smithi103.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 716 kB/s | 59 kB 00:00 2024-04-26T22:58:25.966 INFO:teuthology.orchestra.run.smithi103.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 490 kB/s | 36 kB 00:00 2024-04-26T22:58:25.992 INFO:teuthology.orchestra.run.smithi103.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 893 kB/s | 51 kB 00:00 2024-04-26T22:58:26.034 INFO:teuthology.orchestra.run.smithi103.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-04-26T22:58:26.068 INFO:teuthology.orchestra.run.smithi103.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 908 kB/s | 90 kB 00:00 2024-04-26T22:58:26.092 INFO:teuthology.orchestra.run.smithi040.stdout:(104/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-04-26T22:58:26.101 INFO:teuthology.orchestra.run.smithi103.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.9 MB/s | 427 kB 00:00 2024-04-26T22:58:26.118 INFO:teuthology.orchestra.run.smithi103.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 254 kB/s | 20 kB 00:00 2024-04-26T22:58:26.152 INFO:teuthology.orchestra.run.smithi103.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.3 MB/s | 192 kB 00:00 2024-04-26T22:58:26.209 INFO:teuthology.orchestra.run.smithi040.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 700 kB/s | 548 kB 00:00 2024-04-26T22:58:26.234 INFO:teuthology.orchestra.run.smithi040.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 140 kB/s | 19 kB 00:00 2024-04-26T22:58:26.302 INFO:teuthology.orchestra.run.smithi103.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 8.8 MB/s | 1.6 MB 00:00 2024-04-26T22:58:26.343 INFO:teuthology.orchestra.run.smithi040.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 398 kB/s | 309 kB 00:00 2024-04-26T22:58:26.418 INFO:teuthology.orchestra.run.smithi040.stdout:(108/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 142 kB/s | 26 kB 00:00 2024-04-26T22:58:26.468 INFO:teuthology.orchestra.run.smithi040.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 401 kB/s | 49 kB 00:00 2024-04-26T22:58:26.560 INFO:teuthology.orchestra.run.smithi040.stdout:(110/152): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-26T22:58:26.586 INFO:teuthology.orchestra.run.smithi040.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 441 kB/s | 67 kB 00:00 2024-04-26T22:58:26.627 INFO:teuthology.orchestra.run.smithi040.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-04-26T22:58:26.635 INFO:teuthology.orchestra.run.smithi103.stdout:(149/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 6.5 MB/s | 3.1 MB 00:00 2024-04-26T22:58:26.678 INFO:teuthology.orchestra.run.smithi040.stdout:(113/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.2 MB/s | 839 kB 00:00 2024-04-26T22:58:26.711 INFO:teuthology.orchestra.run.smithi103.stdout:(150/152): librados2-19.0.0-3306.gbc612ffb.el9. 5.8 MB/s | 3.4 MB 00:00 2024-04-26T22:58:26.720 INFO:teuthology.orchestra.run.smithi040.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 6.5 MB/s | 548 kB 00:00 2024-04-26T22:58:26.745 INFO:teuthology.orchestra.run.smithi040.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 404 kB/s | 43 kB 00:00 2024-04-26T22:58:26.770 INFO:teuthology.orchestra.run.smithi040.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 351 kB/s | 32 kB 00:00 2024-04-26T22:58:26.820 INFO:teuthology.orchestra.run.smithi040.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 203 kB/s | 15 kB 00:00 2024-04-26T22:58:26.886 INFO:teuthology.orchestra.run.smithi040.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.2 MB/s | 172 kB 00:00 2024-04-26T22:58:26.921 INFO:teuthology.orchestra.run.smithi040.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-04-26T22:58:26.963 INFO:teuthology.orchestra.run.smithi040.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-04-26T22:58:27.005 INFO:teuthology.orchestra.run.smithi040.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 444 kB/s | 35 kB 00:00 2024-04-26T22:58:27.030 INFO:teuthology.orchestra.run.smithi040.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.9 MB/s | 206 kB 00:00 2024-04-26T22:58:27.114 INFO:teuthology.orchestra.run.smithi040.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-26T22:58:27.139 INFO:teuthology.orchestra.run.smithi040.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-04-26T22:58:27.164 INFO:teuthology.orchestra.run.smithi040.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 89 kB/s | 11 kB 00:00 2024-04-26T22:58:27.189 INFO:teuthology.orchestra.run.smithi040.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 242 kB/s | 18 kB 00:00 2024-04-26T22:58:27.231 INFO:teuthology.orchestra.run.smithi040.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 353 kB/s | 23 kB 00:00 2024-04-26T22:58:27.256 INFO:teuthology.orchestra.run.smithi040.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 215 kB/s | 19 kB 00:00 2024-04-26T22:58:27.273 INFO:teuthology.orchestra.run.smithi040.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 236 kB/s | 20 kB 00:00 2024-04-26T22:58:27.373 INFO:teuthology.orchestra.run.smithi040.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 8.9 MB/s | 1.0 MB 00:00 2024-04-26T22:58:27.390 INFO:teuthology.orchestra.run.smithi040.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 350 kB/s | 46 kB 00:00 2024-04-26T22:58:27.416 INFO:teuthology.orchestra.run.smithi040.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 556 kB/s | 79 kB 00:00 2024-04-26T22:58:27.466 INFO:teuthology.orchestra.run.smithi040.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 786 kB/s | 58 kB 00:00 2024-04-26T22:58:27.502 INFO:teuthology.orchestra.run.smithi040.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-04-26T22:58:27.516 INFO:teuthology.orchestra.run.smithi040.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-26T22:58:27.542 INFO:teuthology.orchestra.run.smithi040.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-26T22:58:27.600 INFO:teuthology.orchestra.run.smithi040.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 340 kB/s | 28 kB 00:00 2024-04-26T22:58:27.626 INFO:teuthology.orchestra.run.smithi040.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 285 kB/s | 31 kB 00:00 2024-04-26T22:58:27.659 INFO:teuthology.orchestra.run.smithi040.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-04-26T22:58:27.743 INFO:teuthology.orchestra.run.smithi040.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 519 kB/s | 59 kB 00:00 2024-04-26T22:58:27.768 INFO:teuthology.orchestra.run.smithi040.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 257 kB/s | 36 kB 00:00 2024-04-26T22:58:27.793 INFO:teuthology.orchestra.run.smithi040.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 383 kB/s | 51 kB 00:00 2024-04-26T22:58:27.885 INFO:teuthology.orchestra.run.smithi040.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-04-26T22:58:27.919 INFO:teuthology.orchestra.run.smithi040.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 603 kB/s | 90 kB 00:00 2024-04-26T22:58:27.952 INFO:teuthology.orchestra.run.smithi040.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-04-26T22:58:27.978 INFO:teuthology.orchestra.run.smithi040.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 232 kB/s | 20 kB 00:00 2024-04-26T22:58:28.053 INFO:teuthology.orchestra.run.smithi040.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-26T22:58:28.120 INFO:teuthology.orchestra.run.smithi040.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-04-26T22:58:28.321 INFO:teuthology.orchestra.run.smithi040.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 6.8 MB/s | 1.3 MB 00:00 2024-04-26T22:58:28.437 INFO:teuthology.orchestra.run.smithi040.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 21 MB/s | 2.4 MB 00:00 2024-04-26T22:58:28.786 INFO:teuthology.orchestra.run.smithi103.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 557 kB/s | 1.3 MB 00:02 2024-04-26T22:58:29.179 INFO:teuthology.orchestra.run.smithi040.stdout:(151/152): librados2-19.0.0-3306.gbc612ffb.el9. 2.8 MB/s | 3.4 MB 00:01 2024-04-26T22:58:29.354 INFO:teuthology.orchestra.run.smithi040.stdout:(152/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 2.4 MB/s | 3.1 MB 00:01 2024-04-26T22:58:29.371 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-26T22:58:29.371 INFO:teuthology.orchestra.run.smithi040.stdout:Total 11 MB/s | 222 MB 00:20 2024-04-26T22:58:30.178 INFO:teuthology.orchestra.run.smithi103.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 692 kB/s | 2.4 MB 00:03 2024-04-26T22:58:30.192 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-26T22:58:30.193 INFO:teuthology.orchestra.run.smithi103.stdout:Total 9.7 MB/s | 222 MB 00:22 2024-04-26T22:58:32.387 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-26T22:58:32.644 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-26T22:58:32.645 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-26T22:58:33.065 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-26T22:58:33.308 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-26T22:58:33.309 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-26T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-26T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-26T22:58:35.994 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-26T22:58:35.994 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-26T22:58:41.979 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-26T22:58:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T22:58:42.272 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T22:58:42.342 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-26T22:58:42.429 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T22:58:42.469 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T22:58:42.509 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T22:58:42.550 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T22:58:42.560 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T22:58:42.697 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T22:58:42.811 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T22:58:42.886 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T22:58:42.921 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T22:58:43.057 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T22:58:43.150 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T22:58:43.255 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:58:43.422 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:58:43.456 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:58:43.477 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T22:58:43.585 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:58:43.610 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:58:43.639 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T22:58:43.812 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:58:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:58:43.954 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:58:43.993 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T22:58:44.009 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T22:58:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T22:58:44.093 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:58:44.261 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:58:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T22:58:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T22:58:44.373 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T22:58:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T22:58:44.404 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T22:58:44.415 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:58:44.480 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T22:58:44.564 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:58:44.600 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T22:58:44.694 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T22:58:44.726 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T22:58:44.738 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T22:58:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T22:58:44.797 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T22:58:45.001 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T22:58:45.088 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T22:58:45.133 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T22:58:45.143 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T22:58:45.172 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T22:58:45.222 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T22:58:45.330 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T22:58:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T22:58:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T22:58:45.451 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T22:58:45.483 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T22:58:45.498 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T22:58:45.527 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T22:58:45.537 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T22:58:45.575 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T22:58:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T22:58:45.679 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T22:58:45.730 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T22:58:45.734 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T22:58:45.772 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T22:58:45.794 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T22:58:45.841 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T22:58:45.843 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T22:58:45.883 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T22:58:45.981 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T22:58:45.987 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T22:58:46.017 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T22:58:46.063 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T22:58:46.073 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T22:58:46.102 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T22:58:46.116 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T22:58:46.161 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T22:58:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T22:58:46.305 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T22:58:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T22:58:46.348 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T22:58:46.403 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T22:58:46.426 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T22:58:46.446 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T22:58:46.568 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T22:58:46.625 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T22:58:46.633 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T22:58:46.698 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T22:58:46.741 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T22:58:46.916 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T22:58:46.985 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T22:58:47.385 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T22:58:47.498 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T22:58:47.539 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T22:58:47.584 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T22:58:47.659 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T22:58:47.686 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T22:58:47.732 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T22:58:47.799 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T22:58:47.836 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T22:58:47.882 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T22:58:47.947 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T22:58:48.006 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T22:58:48.017 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T22:58:48.068 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T22:58:48.148 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T22:58:48.278 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T22:58:48.295 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T22:58:48.317 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T22:58:48.357 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T22:58:48.358 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T22:58:48.438 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T22:58:48.442 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T22:58:48.498 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T22:58:48.536 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T22:58:48.564 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T22:58:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T22:58:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T22:58:48.629 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T22:58:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T22:58:48.720 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T22:58:48.781 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T22:58:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T22:58:48.864 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T22:58:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T22:58:48.933 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T22:58:48.993 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T22:58:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T22:58:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T22:58:49.133 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T22:58:49.164 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T22:58:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T22:58:49.222 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T22:58:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T22:58:49.281 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T22:58:49.346 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T22:58:49.370 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T22:58:49.422 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T22:58:49.472 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T22:58:49.504 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T22:58:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T22:58:49.545 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T22:58:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T22:58:49.737 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T22:58:49.770 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T22:58:49.808 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T22:58:49.859 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T22:58:49.912 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T22:58:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T22:58:49.969 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T22:58:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T22:58:50.013 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T22:58:50.056 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T22:58:50.068 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T22:58:50.121 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T22:58:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T22:58:50.181 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T22:58:50.181 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T22:58:50.224 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T22:58:50.258 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T22:58:50.279 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T22:58:50.304 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T22:58:50.321 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T22:58:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T22:58:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T22:58:50.453 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T22:58:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T22:58:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T22:58:50.608 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T22:58:50.646 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T22:58:50.680 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T22:58:50.724 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T22:58:50.794 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T22:58:50.806 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T22:58:50.873 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T22:58:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T22:58:51.005 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T22:58:51.863 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T22:58:51.922 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T22:58:51.948 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T22:58:51.955 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T22:58:52.004 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T22:58:52.037 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T22:58:52.086 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T22:58:52.158 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T22:58:52.159 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T22:58:52.248 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T22:58:52.270 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T22:58:52.315 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T22:58:52.355 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T22:58:52.397 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T22:58:52.560 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T22:58:52.587 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T22:58:52.632 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T22:58:52.669 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T22:58:52.685 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T22:58:52.786 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T22:58:53.564 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T22:58:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T22:58:53.715 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T22:58:53.784 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T22:58:55.358 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T22:58:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T22:58:55.460 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T22:58:55.486 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T22:58:55.501 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T22:58:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T22:58:55.540 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T22:58:55.590 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T22:58:55.922 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T22:58:55.972 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T22:58:56.018 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T22:58:56.095 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T22:58:56.129 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T22:58:56.139 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T22:58:56.196 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T22:58:56.301 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T22:58:56.347 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T22:58:56.411 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T22:58:56.763 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T22:58:56.792 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T22:58:56.884 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T22:58:56.911 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T22:58:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T22:58:57.051 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T22:58:57.152 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T22:58:57.178 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T22:58:59.618 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:58:59.658 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:58:59.995 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:59:00.041 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:59:00.307 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:59:00.353 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T22:59:00.538 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T22:59:00.539 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T22:59:00.539 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T22:59:00.539 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:00.587 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T22:59:00.729 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T22:59:00.778 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T22:59:00.963 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T22:59:00.963 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T22:59:00.963 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T22:59:00.963 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:01.012 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T22:59:13.829 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T22:59:13.829 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-04-26T22:59:13.829 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-04-26T22:59:13.829 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-04-26T22:59:13.829 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-04-26T22:59:13.829 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-04-26T22:59:13.829 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-04-26T22:59:13.829 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-04-26T22:59:13.829 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:14.080 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T22:59:14.126 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T22:59:14.126 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:14.126 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T22:59:14.126 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:14.411 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T22:59:14.411 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2024-04-26T22:59:14.412 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2024-04-26T22:59:14.412 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2024-04-26T22:59:14.412 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2024-04-26T22:59:14.412 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2024-04-26T22:59:14.412 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /root 2024-04-26T22:59:14.412 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2024-04-26T22:59:14.412 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:14.644 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T22:59:14.696 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T22:59:14.697 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:14.697 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T22:59:14.697 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:15.747 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T22:59:15.854 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T22:59:15.854 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:15.854 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T22:59:15.854 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T22:59:15.854 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T22:59:15.855 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:16.114 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T22:59:16.178 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T22:59:16.178 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:16.178 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T22:59:16.178 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T22:59:16.178 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T22:59:16.178 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:16.386 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T22:59:16.496 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T22:59:16.496 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:16.496 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T22:59:16.497 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T22:59:16.497 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T22:59:16.497 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:16.665 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T22:59:16.751 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T22:59:16.751 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:16.751 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T22:59:16.751 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T22:59:16.751 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T22:59:16.752 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:16.762 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T22:59:16.855 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T22:59:16.855 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:16.855 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T22:59:16.855 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T22:59:16.855 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T22:59:16.856 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:16.882 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T22:59:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T22:59:17.026 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T22:59:17.260 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T22:59:17.311 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T22:59:17.371 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T22:59:17.423 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T22:59:17.468 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T22:59:17.468 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:17.468 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T22:59:17.468 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T22:59:17.468 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T22:59:17.468 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:17.531 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T22:59:17.606 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T22:59:17.670 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T22:59:17.697 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T22:59:17.753 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T22:59:17.909 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T22:59:17.957 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T22:59:17.985 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T22:59:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T22:59:18.078 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T22:59:18.169 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T22:59:18.297 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T22:59:18.486 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T22:59:18.538 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T22:59:18.610 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T22:59:18.618 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T22:59:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T22:59:18.663 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T22:59:18.702 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T22:59:18.813 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T22:59:18.849 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T22:59:19.276 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T22:59:19.328 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T22:59:19.388 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T22:59:19.509 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T22:59:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T22:59:20.104 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T22:59:20.151 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T22:59:20.926 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T22:59:20.966 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T22:59:21.245 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T22:59:21.284 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T22:59:21.430 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T22:59:21.606 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T22:59:21.636 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T22:59:21.731 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T22:59:21.731 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:21.732 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T22:59:21.732 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T22:59:21.732 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T22:59:21.732 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:21.787 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T22:59:21.825 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T22:59:21.937 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T22:59:22.003 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T22:59:22.127 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T22:59:22.156 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T22:59:22.310 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T22:59:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T22:59:22.541 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T22:59:22.641 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T22:59:22.641 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:22.641 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T22:59:22.641 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T22:59:22.641 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T22:59:22.641 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:22.709 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T22:59:22.751 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T22:59:22.887 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T22:59:22.969 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T22:59:23.002 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T22:59:23.040 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T22:59:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T22:59:23.108 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:23.108 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T22:59:23.108 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T22:59:23.108 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T22:59:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T22:59:23.234 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T22:59:23.234 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:23.234 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T22:59:23.234 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:23.316 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T22:59:23.401 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T22:59:23.402 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:23.402 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T22:59:23.402 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T22:59:23.402 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T22:59:23.402 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:23.767 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T22:59:23.835 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T22:59:23.835 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:23.835 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T22:59:23.835 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T22:59:23.836 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T22:59:23.836 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T22:59:24.120 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T22:59:24.215 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T22:59:24.215 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:24.215 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T22:59:24.215 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T22:59:24.215 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T22:59:24.215 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:24.262 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T22:59:24.358 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T22:59:24.358 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:24.358 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T22:59:24.359 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:24.438 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T22:59:24.539 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T22:59:24.539 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:24.539 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T22:59:24.539 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T22:59:24.539 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T22:59:24.540 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T22:59:25.012 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T22:59:25.013 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T22:59:25.013 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T22:59:25.013 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T22:59:25.013 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T22:59:25.013 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:28.831 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T22:59:28.886 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T22:59:28.972 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T22:59:29.017 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T22:59:29.066 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T22:59:29.419 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T22:59:29.478 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T22:59:29.506 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T22:59:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T22:59:30.333 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:59:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T22:59:30.379 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:59:30.437 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:59:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T22:59:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T22:59:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T22:59:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T22:59:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T22:59:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T22:59:31.806 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T22:59:31.880 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:59:31.920 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T22:59:31.934 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:59:31.992 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T22:59:32.008 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T22:59:32.183 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T22:59:32.229 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T22:59:32.230 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T22:59:32.352 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T22:59:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T22:59:32.392 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T22:59:32.532 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T22:59:32.691 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:59:32.962 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:59:32.962 INFO:teuthology.orchestra.run.smithi103.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-26T22:59:32.962 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:33.403 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T22:59:33.516 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T22:59:33.593 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T22:59:33.761 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T22:59:33.805 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T22:59:33.805 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T22:59:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T22:59:34.013 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T22:59:34.013 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T22:59:34.162 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T22:59:34.293 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:59:34.601 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:59:34.601 INFO:teuthology.orchestra.run.smithi040.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-26T22:59:34.602 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:38.752 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T22:59:38.752 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T22:59:38.752 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T22:59:38.752 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T22:59:38.753 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T22:59:38.753 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T22:59:38.753 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T22:59:38.753 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:59:38.753 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:59:38.753 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T22:59:38.753 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T22:59:38.754 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T22:59:38.754 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:59:38.754 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T22:59:38.754 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T22:59:38.754 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T22:59:38.755 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T22:59:38.755 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T22:59:38.755 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T22:59:38.755 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T22:59:38.755 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T22:59:38.755 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T22:59:38.756 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T22:59:38.756 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T22:59:38.756 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T22:59:38.756 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T22:59:38.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T22:59:38.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T22:59:38.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T22:59:38.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T22:59:38.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T22:59:38.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T22:59:38.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T22:59:38.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T22:59:38.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T22:59:38.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T22:59:38.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T22:59:38.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T22:59:38.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T22:59:38.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T22:59:38.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T22:59:38.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T22:59:38.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T22:59:38.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T22:59:38.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T22:59:38.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T22:59:38.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T22:59:38.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T22:59:38.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T22:59:38.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T22:59:38.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T22:59:38.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T22:59:38.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T22:59:38.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T22:59:38.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T22:59:38.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T22:59:38.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T22:59:38.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T22:59:38.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T22:59:38.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T22:59:38.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T22:59:38.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T22:59:38.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T22:59:38.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T22:59:38.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T22:59:38.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T22:59:38.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T22:59:38.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T22:59:38.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T22:59:38.789 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T22:59:38.789 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T22:59:38.789 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T22:59:38.789 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T22:59:38.789 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T22:59:38.789 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T22:59:38.789 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T22:59:38.790 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T22:59:38.790 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T22:59:38.790 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T22:59:38.790 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T22:59:38.790 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T22:59:38.791 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T22:59:38.791 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T22:59:38.791 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T22:59:38.791 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T22:59:38.791 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T22:59:38.791 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T22:59:38.792 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T22:59:38.792 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T22:59:38.792 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T22:59:38.792 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T22:59:38.792 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T22:59:38.792 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T22:59:38.793 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T22:59:38.793 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T22:59:38.793 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T22:59:38.793 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T22:59:38.793 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T22:59:38.793 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T22:59:38.794 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T22:59:38.794 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T22:59:38.794 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T22:59:38.794 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T22:59:38.794 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T22:59:38.794 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T22:59:38.795 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T22:59:38.795 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T22:59:38.795 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T22:59:38.795 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T22:59:38.795 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T22:59:38.796 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T22:59:38.796 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T22:59:38.796 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T22:59:38.796 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T22:59:38.796 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T22:59:38.796 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T22:59:38.797 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T22:59:38.797 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T22:59:38.797 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T22:59:38.797 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T22:59:38.797 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T22:59:38.797 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T22:59:38.798 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T22:59:38.798 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T22:59:38.798 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T22:59:38.798 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T22:59:38.798 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T22:59:38.798 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T22:59:38.799 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T22:59:38.799 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T22:59:38.799 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T22:59:38.799 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T22:59:38.799 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T22:59:38.800 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T22:59:38.800 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T22:59:38.800 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T22:59:38.800 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T22:59:38.800 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T22:59:38.800 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T22:59:38.801 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T22:59:38.801 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T22:59:38.801 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T22:59:38.801 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T22:59:38.801 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T22:59:38.801 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T22:59:38.801 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T22:59:38.802 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T22:59:38.802 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T22:59:38.802 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T22:59:38.802 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T22:59:38.802 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T22:59:38.803 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T22:59:38.803 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T22:59:38.803 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T22:59:38.803 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T22:59:40.831 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T22:59:40.831 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T22:59:40.831 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T22:59:40.831 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T22:59:40.831 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T22:59:40.831 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T22:59:40.831 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T22:59:40.831 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T22:59:40.831 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T22:59:40.831 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T22:59:40.832 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T22:59:40.833 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T22:59:40.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T22:59:40.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T22:59:40.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T22:59:40.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T22:59:40.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T22:59:40.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T22:59:40.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T22:59:40.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T22:59:40.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T22:59:40.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T22:59:40.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T22:59:40.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T22:59:40.834 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T22:59:40.835 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T22:59:40.836 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T22:59:40.836 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T22:59:40.836 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T22:59:40.836 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T22:59:40.836 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T22:59:40.836 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T22:59:40.836 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T22:59:40.836 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T22:59:40.836 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T22:59:40.836 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T22:59:40.836 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T22:59:40.836 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T22:59:40.837 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T22:59:40.837 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T22:59:40.837 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T22:59:40.837 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T22:59:40.837 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T22:59:40.837 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T22:59:40.837 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T22:59:40.837 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T22:59:40.837 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T22:59:40.837 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T22:59:40.837 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T22:59:40.837 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T22:59:40.837 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T22:59:40.838 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T22:59:40.838 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T22:59:40.838 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T22:59:40.838 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T22:59:40.838 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T22:59:40.838 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T22:59:40.838 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T22:59:40.838 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T22:59:40.838 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T22:59:40.838 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T22:59:40.838 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T22:59:40.838 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T22:59:40.838 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T22:59:40.839 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T22:59:40.840 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T22:59:40.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T22:59:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T22:59:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T22:59:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T22:59:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T22:59:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T22:59:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T22:59:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T22:59:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T22:59:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T22:59:41.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:59:41.022 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:41.022 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-04-26T22:59:41.022 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.022 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.022 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T22:59:41.022 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T22:59:41.022 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-26T22:59:41.022 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T22:59:41.023 INFO:teuthology.orchestra.run.smithi103.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T22:59:41.023 INFO:teuthology.orchestra.run.smithi103.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T22:59:41.023 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T22:59:41.023 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.023 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.023 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.023 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.023 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:41.023 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.023 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.023 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.023 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:41.023 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.024 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T22:59:41.026 INFO:teuthology.orchestra.run.smithi103.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T22:59:41.027 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T22:59:41.027 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T22:59:41.027 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T22:59:41.027 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T22:59:41.027 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T22:59:41.027 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T22:59:41.027 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T22:59:41.027 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T22:59:41.027 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T22:59:41.027 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T22:59:41.027 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T22:59:41.027 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T22:59:41.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T22:59:41.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T22:59:41.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T22:59:41.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T22:59:41.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T22:59:41.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T22:59:41.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T22:59:41.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T22:59:41.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T22:59:41.028 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T22:59:41.029 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T22:59:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.031 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T22:59:41.032 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.033 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.033 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:41.033 INFO:teuthology.orchestra.run.smithi103.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T22:59:41.033 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T22:59:41.033 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T22:59:41.033 INFO:teuthology.orchestra.run.smithi103.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T22:59:41.033 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T22:59:41.033 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T22:59:41.034 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T22:59:41.034 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:41.034 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-26T22:59:41.608 DEBUG:teuthology.parallel:result is None 2024-04-26T22:59:43.262 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T22:59:43.262 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:43.263 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2024-04-26T22:59:43.263 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.263 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.263 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T22:59:43.263 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T22:59:43.263 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-26T22:59:43.263 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T22:59:43.263 INFO:teuthology.orchestra.run.smithi040.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T22:59:43.263 INFO:teuthology.orchestra.run.smithi040.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T22:59:43.263 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T22:59:43.263 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.263 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.263 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:43.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.265 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.265 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.265 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:43.265 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:43.265 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.265 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.265 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T22:59:43.265 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T22:59:43.265 INFO:teuthology.orchestra.run.smithi040.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T22:59:43.265 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T22:59:43.265 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T22:59:43.265 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T22:59:43.265 INFO:teuthology.orchestra.run.smithi040.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T22:59:43.266 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T22:59:43.266 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T22:59:43.266 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T22:59:43.266 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T22:59:43.266 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T22:59:43.266 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.266 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.266 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.266 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T22:59:43.266 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T22:59:43.266 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T22:59:43.266 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T22:59:43.267 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T22:59:43.268 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T22:59:43.268 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T22:59:43.268 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T22:59:43.268 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T22:59:43.268 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T22:59:43.268 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T22:59:43.268 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T22:59:43.268 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T22:59:43.268 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T22:59:43.269 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T22:59:43.270 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T22:59:43.270 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T22:59:43.270 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T22:59:43.270 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T22:59:43.270 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T22:59:43.270 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T22:59:43.270 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T22:59:43.270 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T22:59:43.270 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T22:59:43.271 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T22:59:43.271 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T22:59:43.271 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T22:59:43.271 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T22:59:43.271 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T22:59:43.271 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T22:59:43.271 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T22:59:43.271 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T22:59:43.271 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T22:59:43.271 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T22:59:43.271 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T22:59:43.271 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T22:59:43.272 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T22:59:43.272 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T22:59:43.272 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T22:59:43.272 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T22:59:43.272 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T22:59:43.272 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T22:59:43.272 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T22:59:43.272 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T22:59:43.272 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T22:59:43.272 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T22:59:43.273 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T22:59:43.273 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T22:59:43.273 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T22:59:43.273 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.273 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.273 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T22:59:43.273 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T22:59:43.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T22:59:43.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T22:59:43.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T22:59:43.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T22:59:43.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T22:59:43.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T22:59:43.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T22:59:43.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T22:59:43.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T22:59:43.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T22:59:43.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T22:59:43.275 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T22:59:43.275 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T22:59:43.275 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T22:59:43.275 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T22:59:43.275 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T22:59:43.275 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.275 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.275 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T22:59:43.275 INFO:teuthology.orchestra.run.smithi040.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T22:59:43.275 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T22:59:43.275 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T22:59:43.276 INFO:teuthology.orchestra.run.smithi040.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T22:59:43.276 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T22:59:43.276 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T22:59:43.276 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T22:59:43.276 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:43.276 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-26T22:59:43.999 DEBUG:teuthology.parallel:result is None 2024-04-26T22:59:44.000 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:59:44.260 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T22:59:44.316 INFO:teuthology.orchestra.run.smithi040.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T22:59:44.316 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T22:59:44.316 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T22:59:44.319 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T22:59:44.470 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T22:59:44.524 INFO:teuthology.orchestra.run.smithi103.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T22:59:44.524 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T22:59:44.524 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T22:59:44.526 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T22:59:44.527 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T22:59:44.527 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T22:59:44.596 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T22:59:44.596 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T22:59:44.671 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T22:59:44.672 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T22:59:44.672 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T22:59:44.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T22:59:44.866 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T22:59:44.866 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T22:59:44.941 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T22:59:45.043 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T22:59:45.043 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T22:59:45.043 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T22:59:45.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T22:59:45.211 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T22:59:45.211 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T22:59:45.274 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T22:59:45.374 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T22:59:45.374 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T22:59:45.374 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T22:59:45.441 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T22:59:45.542 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T22:59:45.542 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T22:59:45.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T22:59:45.706 INFO:teuthology.run_tasks:Running task ceph... 2024-04-26T22:59:45.793 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-26T22:59:45.793 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /var/log/ceph 2024-04-26T22:59:45.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /var/log/ceph 2024-04-26T22:59:45.868 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-26T22:59:45.868 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T22:59:45.872 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T22:59:45.930 INFO:tasks.ceph:Creating extra log directories... 2024-04-26T22:59:45.930 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T22:59:45.934 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T22:59:45.998 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-26T22:59:45.998 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-26T22:59:45.998 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-26T22:59:45.998 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T22:59:45.999 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-26T22:59:46.035 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-26T22:59:46.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-26T22:59:46.281 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T22:59:46.282 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-26T22:59:46.322 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-26T22:59:46.507 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-26T22:59:46.572 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-26T22:59:46.572 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Scrub error on inode'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-26T22:59:46.572 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279', 'branch': 'wip-rishabh-testing-20240426.111959', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/multimds_misc}', 'email': None, 'first_in_suite': False, 'job_id': '7675279', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-rishabh-testing-20240426.111959'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Scrub error on inode'], 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rishabh-testing-20240426.111959', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 816, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'sleep_before_teardown': 0, 'subset': '166/512', 'suite': 'fs', 'suite_branch': 'wip-rishabh-testing-20240426.111959', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi040.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCmwad1FdyL8tEL6I5NyZOm1iLJXHdqc4gYsXpz/ZVh+8X0V5pjDOmkQYIHl1kwFTeIuREUa686wdnNywdeHJM=', 'smithi103.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKq3pNTox0Ssyv/rsyIIVqcF0i9bOossn7/BwWuFayw7fcHef7wXimfAPIVgFS6INsRneFpNwvEHH4OFhGppse8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Scrub error on inode'], 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_multimds_misc']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-26_19:30:57', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-26T22:59:46.573 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T22:59:46.578 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T22:59:46.616 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T22:59:46.660 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T22:59:46.722 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T22:59:46.722 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T22:59:46.762 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T22:59:46.762 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-04-26T22:59:46.844 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T22:59:46.844 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:59:46.845 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-04-26T22:59:46.845 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:59:46.845 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:59:46.845 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-26 22:59:35.135556202 +0000 2024-04-26T22:59:46.845 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-26 22:57:13.559815094 +0000 2024-04-26T22:59:46.845 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-26 22:57:13.559815094 +0000 2024-04-26T22:59:46.845 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-26 22:57:13.559815094 +0000 2024-04-26T22:59:46.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T22:59:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-26T22:59:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-26T22:59:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000364335 s, 1.4 MB/s 2024-04-26T22:59:46.947 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T22:59:46.988 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-04-26T22:59:47.072 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T22:59:47.072 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:59:47.072 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-04-26T22:59:47.073 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:59:47.073 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:59:47.073 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-26 22:59:35.137556175 +0000 2024-04-26T22:59:47.073 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-26 22:57:14.185806638 +0000 2024-04-26T22:59:47.073 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-26 22:57:14.185806638 +0000 2024-04-26T22:59:47.073 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-26 22:57:14.185806638 +0000 2024-04-26T22:59:47.073 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T22:59:47.175 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-26T22:59:47.175 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-26T22:59:47.175 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000280171 s, 1.8 MB/s 2024-04-26T22:59:47.178 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T22:59:47.219 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-04-26T22:59:47.300 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T22:59:47.301 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:59:47.301 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 670 Links: 1 2024-04-26T22:59:47.301 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:59:47.301 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:59:47.301 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-26 22:59:35.137556175 +0000 2024-04-26T22:59:47.301 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-26 22:57:14.815798127 +0000 2024-04-26T22:59:47.301 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-26 22:57:14.815798127 +0000 2024-04-26T22:59:47.301 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-26 22:57:14.815798127 +0000 2024-04-26T22:59:47.301 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T22:59:47.397 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-26T22:59:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-26T22:59:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000279356 s, 1.8 MB/s 2024-04-26T22:59:47.400 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T22:59:47.441 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-04-26T22:59:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T22:59:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:59:47.522 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-04-26T22:59:47.522 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:59:47.522 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:59:47.522 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-26 22:59:35.138556161 +0000 2024-04-26T22:59:47.523 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-26 22:57:15.436789738 +0000 2024-04-26T22:59:47.523 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-26 22:57:15.436789738 +0000 2024-04-26T22:59:47.523 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-26 22:57:15.436789738 +0000 2024-04-26T22:59:47.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T22:59:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-26T22:59:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-26T22:59:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000270208 s, 1.9 MB/s 2024-04-26T22:59:47.631 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T22:59:47.671 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T22:59:47.672 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T22:59:47.672 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T22:59:47.717 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T22:59:47.717 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-04-26T22:59:47.795 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T22:59:47.796 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:59:47.796 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-04-26T22:59:47.796 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:59:47.796 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:59:47.796 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-26 22:59:33.459879063 +0000 2024-04-26T22:59:47.796 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-26 22:57:05.585971841 +0000 2024-04-26T22:59:47.796 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-26 22:57:05.585971841 +0000 2024-04-26T22:59:47.796 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-26 22:57:05.585971841 +0000 2024-04-26T22:59:47.796 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T22:59:47.892 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-26T22:59:47.900 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-26T22:59:47.900 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000339878 s, 1.5 MB/s 2024-04-26T22:59:47.901 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T22:59:47.939 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-04-26T22:59:48.019 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T22:59:48.020 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:59:48.020 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-04-26T22:59:48.020 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:59:48.020 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:59:48.020 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-26 22:59:33.460879045 +0000 2024-04-26T22:59:48.020 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-26 22:57:06.191960274 +0000 2024-04-26T22:59:48.020 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-26 22:57:06.191960274 +0000 2024-04-26T22:59:48.020 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-26 22:57:06.191960274 +0000 2024-04-26T22:59:48.020 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T22:59:48.129 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-26T22:59:48.129 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-26T22:59:48.129 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000333787 s, 1.5 MB/s 2024-04-26T22:59:48.133 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T22:59:48.176 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-04-26T22:59:48.257 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T22:59:48.257 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:59:48.257 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 670 Links: 1 2024-04-26T22:59:48.257 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:59:48.258 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:59:48.258 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-26 22:59:33.461879025 +0000 2024-04-26T22:59:48.258 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-26 22:57:06.790948840 +0000 2024-04-26T22:59:48.258 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-26 22:57:06.790948840 +0000 2024-04-26T22:59:48.258 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-26 22:57:06.790948840 +0000 2024-04-26T22:59:48.258 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T22:59:48.358 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-26T22:59:48.359 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-26T22:59:48.359 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000337057 s, 1.5 MB/s 2024-04-26T22:59:48.361 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T22:59:48.402 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-04-26T22:59:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T22:59:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T22:59:48.494 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-04-26T22:59:48.494 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T22:59:48.494 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T22:59:48.494 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-26 22:59:33.463878987 +0000 2024-04-26T22:59:48.494 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-26 22:57:07.384937503 +0000 2024-04-26T22:59:48.495 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-26 22:57:07.384937503 +0000 2024-04-26T22:59:48.495 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-26 22:57:07.384937503 +0000 2024-04-26T22:59:48.495 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T22:59:48.556 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-26T22:59:48.556 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-26T22:59:48.556 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000281239 s, 1.8 MB/s 2024-04-26T22:59:48.559 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T22:59:48.609 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-26T22:59:48.609 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-26T22:59:48.609 INFO:tasks.ceph:Generating config... 2024-04-26T22:59:48.610 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-26T22:59:48.610 INFO:tasks.ceph:[client] debug client = 20 2024-04-26T22:59:48.610 INFO:tasks.ceph:[client] debug ms = 1 2024-04-26T22:59:48.611 INFO:tasks.ceph:[client] pid file = 2024-04-26T22:59:48.611 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-26T22:59:48.611 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-26T22:59:48.611 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-26T22:59:48.611 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-26T22:59:48.611 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-26T22:59:48.611 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-26T22:59:48.611 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-26T22:59:48.611 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-26T22:59:48.611 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-26T22:59:48.611 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-26T22:59:48.611 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-26T22:59:48.611 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-26T22:59:48.611 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-26T22:59:48.611 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-26T22:59:48.612 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-26T22:59:48.612 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-26T22:59:48.612 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-26T22:59:48.612 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-26T22:59:48.612 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-26T22:59:48.612 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-26T22:59:48.612 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-26T22:59:48.612 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-26T22:59:48.612 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-26T22:59:48.612 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-26T22:59:48.612 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-26T22:59:48.612 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-26T22:59:48.612 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-26T22:59:48.612 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-26T22:59:48.612 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-26T22:59:48.613 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-26T22:59:48.613 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-26T22:59:48.613 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-26T22:59:48.613 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-26T22:59:48.613 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-26T22:59:48.613 INFO:tasks.ceph:Setting up mon.a... 2024-04-26T22:59:48.613 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-26T22:59:48.714 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.keyring 2024-04-26T22:59:48.720 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-26T22:59:48.818 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T22:59:48.882 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.40'), ('mon.b', '172.21.15.103'), ('mon.c', '[v2:172.21.15.103:3301,v1:172.21.15.103:6790]')] 2024-04-26T22:59:48.883 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.40,172.21.15.103,[v2:172.21.15.103:3301,v1:172.21.15.103:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-26T22:59:48.883 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T22:59:48.884 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T22:59:48.923 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.40 --add b 172.21.15.103 --addv c '[v2:172.21.15.103:3301,v1:172.21.15.103:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-26T22:59:49.056 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-26T22:59:49.056 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: generated fsid 815936ab-d4ac-471e-b069-1a0dc70078ee 2024-04-26T22:59:49.056 INFO:teuthology.orchestra.run.smithi040.stdout:setting min_mon_release = quincy 2024-04-26T22:59:49.056 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 0 2024-04-26T22:59:49.056 INFO:teuthology.orchestra.run.smithi040.stdout:fsid 815936ab-d4ac-471e-b069-1a0dc70078ee 2024-04-26T22:59:49.056 INFO:teuthology.orchestra.run.smithi040.stdout:last_changed 2024-04-26T22:59:49.056690+0000 2024-04-26T22:59:49.056 INFO:teuthology.orchestra.run.smithi040.stdout:created 2024-04-26T22:59:49.056690+0000 2024-04-26T22:59:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:min_mon_release 17 (quincy) 2024-04-26T22:59:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:election_strategy: 1 2024-04-26T22:59:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-26T22:59:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:1: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-04-26T22:59:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:2: [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] mon.c 2024-04-26T22:59:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-26T22:59:49.059 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T22:59:49.097 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 815936ab-d4ac-471e-b069-1a0dc70078ee... 2024-04-26T22:59:49.098 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T22:59:49.142 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T22:59:49.245 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2024-04-26T22:59:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: chdir = "" 2024-04-26T22:59:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T22:59:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: auth supported = cephx 2024-04-26T22:59:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: filestore xattr use omap = true 2024-04-26T22:59:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: mon clock drift allowed = 1.000 2024-04-26T22:59:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: osd crush chooseleaf type = 0 2024-04-26T22:59:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: auth debug = true 2024-04-26T22:59:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on old message = true 2024-04-26T22:59:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on bug = true 2024-04-26T22:59:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T22:59:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: mon pg warn max object skew = 0 2024-04-26T22:59:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T22:59:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T22:59:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default size = 2 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow primary affinity = true 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow pg remap = true 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on legacy crush tunables = false 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on crush straw calc version zero = false 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on no sortbitwise = false 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on osd down out interval zero = false 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on too few osds = false 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: mon_allow_pool_size_one = true 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.247 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T22:59:49.248 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.248 INFO:teuthology.orchestra.run.smithi040.stdout: osd default data pool replay window = 5 2024-04-26T22:59:49.248 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.248 INFO:teuthology.orchestra.run.smithi040.stdout: mon allow pool delete = true 2024-04-26T22:59:49.248 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.248 INFO:teuthology.orchestra.run.smithi040.stdout: mon cluster log file level = debug 2024-04-26T22:59:49.248 INFO:teuthology.orchestra.run.smithi040.stdout: debug asserts on shutdown = true 2024-04-26T22:59:49.248 INFO:teuthology.orchestra.run.smithi040.stdout: mon health detail to clog = false 2024-04-26T22:59:49.248 INFO:teuthology.orchestra.run.smithi040.stdout: mon host = "172.21.15.40,172.21.15.103,[v2:172.21.15.103:3301,v1:172.21.15.103:6790]" 2024-04-26T22:59:49.248 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on skipped message = False 2024-04-26T22:59:49.248 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = 815936ab-d4ac-471e-b069-1a0dc70078ee 2024-04-26T22:59:49.248 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout:[osd] 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: osd journal size = 100 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub load threshold = 5.0 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub max interval = 600 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock profile = high_recovery_ops 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: osd recover clone overlap = true 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: osd recovery max chunk = 1048576 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug shutdown = true 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug op order = true 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify stray on activate = true 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug trim objects = true 2024-04-26T22:59:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: osd open classes on start = true 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug pg log writeout = true 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: osd deep scrub update digest min age = 30 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: osd map max advance = 10 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: journal zero on create = true 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: filestore ondisk finisher threads = 3 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: filestore apply finisher threads = 3 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: bdev debug aio = true 2024-04-26T22:59:49.250 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug misdirected ops = true 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore block size = 96636764160 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore fsck on mount = True 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluefs = 20 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluestore = 20 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: debug osd = 20 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: debug rocksdb = 10 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd full ratio = 0.9 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd nearfull ratio = 0.8 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: osd failsafe full ratio = 0.95 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: osd objectstore = bluestore 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: osd op complaint time = 180 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.251 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr] 2024-04-26T22:59:49.252 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-04-26T22:59:49.252 INFO:teuthology.orchestra.run.smithi040.stdout: debug mgr = 20 2024-04-26T22:59:49.253 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2024-04-26T22:59:49.253 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2024-04-26T22:59:49.253 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2024-04-26T22:59:49.253 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2024-04-26T22:59:49.253 INFO:teuthology.orchestra.run.smithi040.stdout: mgr/telemetry/nag = false 2024-04-26T22:59:49.253 INFO:teuthology.orchestra.run.smithi040.stdout: debug client = 20 2024-04-26T22:59:49.253 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.253 INFO:teuthology.orchestra.run.smithi040.stdout:[mon] 2024-04-26T22:59:49.253 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-04-26T22:59:49.253 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2024-04-26T22:59:49.254 INFO:teuthology.orchestra.run.smithi040.stdout: debug paxos = 20 2024-04-26T22:59:49.254 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2024-04-26T22:59:49.254 INFO:teuthology.orchestra.run.smithi040.stdout: mon data avail warn = 5 2024-04-26T22:59:49.254 INFO:teuthology.orchestra.run.smithi040.stdout: mon mgr mkfs grace = 240 2024-04-26T22:59:49.254 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2024-04-26T22:59:49.254 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd reporter subtree level = osd 2024-04-26T22:59:49.254 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd prime pg temp = true 2024-04-26T22:59:49.254 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2024-04-26T22:59:49.254 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.254 INFO:teuthology.orchestra.run.smithi040.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T22:59:49.254 INFO:teuthology.orchestra.run.smithi040.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T22:59:49.254 INFO:teuthology.orchestra.run.smithi040.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T22:59:49.255 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.255 INFO:teuthology.orchestra.run.smithi040.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T22:59:49.255 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T22:59:49.255 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T22:59:49.255 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.255 INFO:teuthology.orchestra.run.smithi040.stdout: # 1m isn't quite enough 2024-04-26T22:59:49.255 INFO:teuthology.orchestra.run.smithi040.stdout: mon_down_mkfs_grace = 2m 2024-04-26T22:59:49.255 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.255 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_filestore_osds = false 2024-04-26T22:59:49.255 INFO:teuthology.orchestra.run.smithi040.stdout: mon down mkfs grace = 300 2024-04-26T22:59:49.255 INFO:teuthology.orchestra.run.smithi040.stdout: mon op complaint time = 120 2024-04-26T22:59:49.255 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout:[client] 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout: rgw cache enabled = true 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable ops log = true 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable usage log = true 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout: client mount timeout = 600 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout: debug client = 20 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout: pid file = "" 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout: rados mon op timeout = 900 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout: rados osd op timeout = 900 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.a] 2024-04-26T22:59:49.256 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.b] 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.c] 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.a] 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.c] 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.b] 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.d] 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout:[mds] 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout: debug mds = 20 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout: debug mds balancer = 20 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout: mds debug frag = True 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout: mds debug scatterstat = True 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout: mds op complaint time = 180 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout: mds verify scatter = True 2024-04-26T22:59:49.257 INFO:teuthology.orchestra.run.smithi040.stdout: osd op complaint time = 180 2024-04-26T22:59:49.258 INFO:teuthology.orchestra.run.smithi040.stdout: rados mon op timeout = 900 2024-04-26T22:59:49.258 INFO:teuthology.orchestra.run.smithi040.stdout: rados osd op timeout = 900 2024-04-26T22:59:49.283 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-04-26T22:59:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2024-04-26T22:59:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T22:59:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2024-04-26T22:59:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2024-04-26T22:59:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2024-04-26T22:59:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2024-04-26T22:59:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2024-04-26T22:59:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2024-04-26T22:59:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2024-04-26T22:59:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T22:59:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2024-04-26T22:59:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T22:59:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T22:59:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T22:59:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = "172.21.15.40,172.21.15.103,[v2:172.21.15.103:3301,v1:172.21.15.103:6790]" 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on skipped message = False 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = 815936ab-d4ac-471e-b069-1a0dc70078ee 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2024-04-26T22:59:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 600 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug trim objects = true 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2024-04-26T22:59:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore block size = 96636764160 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore fsck on mount = True 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluefs = 20 2024-04-26T22:59:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluestore = 20 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: debug rocksdb = 10 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd full ratio = 0.9 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd nearfull ratio = 0.8 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: osd failsafe full ratio = 0.95 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: osd objectstore = bluestore 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: osd op complaint time = 180 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-26T22:59:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: debug client = 20 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-26T22:59:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: mon down mkfs grace = 300 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: mon op complaint time = 120 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: client mount timeout = 600 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: debug client = 20 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = "" 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: rados mon op timeout = 900 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: rados osd op timeout = 900 2024-04-26T22:59:49.292 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.b] 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.c] 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.a] 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.c] 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.b] 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.d] 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout:[mds] 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: debug mds = 20 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: debug mds balancer = 20 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: mds debug frag = True 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: mds debug scatterstat = True 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: mds op complaint time = 180 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: mds verify scatter = True 2024-04-26T22:59:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: osd op complaint time = 180 2024-04-26T22:59:49.294 INFO:teuthology.orchestra.run.smithi103.stdout: rados mon op timeout = 900 2024-04-26T22:59:49.294 INFO:teuthology.orchestra.run.smithi103.stdout: rados osd op timeout = 900 2024-04-26T22:59:49.315 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-26T22:59:49.315 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-26T22:59:49.426 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-26T22:59:49.426 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T22:59:49.426 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-26T22:59:49.467 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T22:59:49.467 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-26T22:59:49.554 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T22:59:49.555 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T22:59:49.555 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T22:59:49.555 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T22:59:49.700 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T22:59:49.700 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T22:59:49.783 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T22:59:49.784 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T22:59:49.784 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T22:59:49.784 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T22:59:49.891 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T22:59:49.891 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T22:59:49.986 INFO:tasks.ceph:Setting up mon nodes... 2024-04-26T22:59:49.986 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-26T22:59:49.986 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-26T22:59:50.139 INFO:teuthology.orchestra.run.smithi040.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-26T22:59:50.146 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T22:59:50.313 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T22:59:50.323 INFO:tasks.ceph:Setting up mds nodes... 2024-04-26T22:59:50.323 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-26T22:59:50.469 INFO:teuthology.orchestra.run.smithi040.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-26T22:59:50.475 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-26T22:59:50.542 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-26T22:59:50.659 INFO:teuthology.orchestra.run.smithi040.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-26T22:59:50.665 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-26T22:59:50.731 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-26T22:59:50.866 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-26T22:59:50.876 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-26T22:59:50.939 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-26T22:59:51.064 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-26T22:59:51.071 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-26T22:59:51.150 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-26T22:59:51.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T22:59:51.273 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-26T22:59:51.314 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-26T22:59:51.424 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-26T22:59:51.462 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-26T22:59:51.585 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-26T22:59:51.624 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-26T22:59:51.737 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-26T22:59:51.779 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-26T22:59:51.779 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-26T22:59:51.779 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-26T22:59:51.873 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T22:59:51.874 INFO:tasks.ceph:role: osd.0 2024-04-26T22:59:51.874 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T22:59:51.874 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T22:59:51.957 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T22:59:51.957 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T22:59:51.957 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T22:59:51.957 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-26T22:59:51.957 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T22:59:51.957 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-04-26T22:59:51.957 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T22:59:51.957 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-26T22:59:51.957 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T22:59:51.957 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T22:59:52.371 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-04-26T22:59:52.379 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-04-26T22:59:52.379 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-26T22:59:52.540 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-26T22:59:52.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-26T22:59:52.725 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T22:59:52.725 INFO:tasks.ceph:role: osd.1 2024-04-26T22:59:52.725 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T22:59:52.725 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T22:59:52.812 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T22:59:52.813 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T22:59:52.813 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T22:59:52.813 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-26T22:59:52.813 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T22:59:52.813 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-04-26T22:59:52.813 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T22:59:52.813 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-26T22:59:52.813 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T22:59:52.813 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T22:59:53.166 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-04-26T22:59:53.173 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-04-26T22:59:53.173 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-26T22:59:53.325 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-26T22:59:53.413 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-26T22:59:53.490 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T22:59:53.491 INFO:tasks.ceph:role: osd.2 2024-04-26T22:59:53.491 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T22:59:53.491 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T22:59:53.584 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T22:59:53.584 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T22:59:53.584 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T22:59:53.584 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-26T22:59:53.584 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T22:59:53.584 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-04-26T22:59:53.584 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T22:59:53.584 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-26T22:59:53.584 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T22:59:53.585 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T22:59:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-04-26T22:59:53.997 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-04-26T22:59:53.997 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-26T22:59:54.162 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-26T22:59:54.262 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-26T22:59:54.348 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T22:59:54.349 INFO:tasks.ceph:role: osd.3 2024-04-26T22:59:54.349 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T22:59:54.349 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T22:59:54.446 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T22:59:54.447 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T22:59:54.447 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T22:59:54.447 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-26T22:59:54.447 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T22:59:54.447 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-04-26T22:59:54.447 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T22:59:54.447 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-26T22:59:54.447 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T22:59:54.447 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T22:59:54.850 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-04-26T22:59:54.855 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-04-26T22:59:54.855 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-26T22:59:54.969 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-26T22:59:55.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T22:59:55.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T22:59:55.192+0000 7fac32c0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-26T22:59:55.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T22:59:55.193+0000 7fac32c0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-26T22:59:55.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T22:59:55.193+0000 7fac32c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T22:59:55.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T22:59:55.193+0000 7fac32c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T22:59:55.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T22:59:55.193+0000 7fac32c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T22:59:55.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T22:59:55.197+0000 7fac32c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-26T22:59:58.429 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-26T22:59:58.519 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T22:59:58.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T22:59:58.650+0000 7fda31e33640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-26T22:59:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T22:59:58.650+0000 7fda31e33640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-26T22:59:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T22:59:58.650+0000 7fda31e33640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T22:59:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T22:59:58.650+0000 7fda31e33640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T22:59:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T22:59:58.650+0000 7fda31e33640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T22:59:58.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T22:59:58.653+0000 7fda31e33640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-26T23:00:01.879 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-26T23:00:01.971 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T23:00:02.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:02.106+0000 7f61b8b88640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-26T23:00:02.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:02.107+0000 7f61b8b88640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-26T23:00:02.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:02.107+0000 7f61b8b88640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T23:00:02.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:02.107+0000 7f61b8b88640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T23:00:02.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:02.107+0000 7f61b8b88640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T23:00:02.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:02.109+0000 7f61b8b88640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-26T23:00:05.075 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-26T23:00:05.169 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T23:00:05.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:05.305+0000 7fe7ba3a5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-26T23:00:05.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:05.306+0000 7fe7ba3a5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-26T23:00:05.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:05.306+0000 7fe7ba3a5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T23:00:05.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:05.306+0000 7fe7ba3a5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T23:00:05.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:05.306+0000 7fe7ba3a5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T23:00:05.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:05.308+0000 7fe7ba3a5640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-26T23:00:08.291 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-26T23:00:08.375 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-26T23:00:08.463 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-26T23:00:08.463 INFO:tasks.ceph:role: osd.4 2024-04-26T23:00:08.463 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T23:00:08.463 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T23:00:08.552 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T23:00:08.553 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T23:00:08.553 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T23:00:08.553 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-26T23:00:08.553 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T23:00:08.553 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-26T23:00:08.553 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T23:00:08.553 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-26T23:00:08.553 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T23:00:08.553 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T23:00:08.965 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-26T23:00:08.973 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-26T23:00:08.973 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-26T23:00:09.133 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-26T23:00:09.229 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-26T23:00:09.308 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-26T23:00:09.308 INFO:tasks.ceph:role: osd.5 2024-04-26T23:00:09.308 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T23:00:09.308 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T23:00:09.392 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T23:00:09.392 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T23:00:09.392 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T23:00:09.393 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-26T23:00:09.393 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T23:00:09.393 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-26T23:00:09.393 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T23:00:09.393 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-26T23:00:09.393 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T23:00:09.393 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T23:00:09.748 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-26T23:00:09.755 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-26T23:00:09.755 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-26T23:00:09.868 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-26T23:00:09.939 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-26T23:00:10.000 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-26T23:00:10.000 INFO:tasks.ceph:role: osd.6 2024-04-26T23:00:10.000 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T23:00:10.000 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T23:00:10.074 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T23:00:10.075 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T23:00:10.075 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T23:00:10.075 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-26T23:00:10.075 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T23:00:10.075 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-26T23:00:10.075 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T23:00:10.075 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-26T23:00:10.075 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T23:00:10.075 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T23:00:10.462 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-26T23:00:10.468 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-26T23:00:10.468 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-26T23:00:10.587 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-26T23:00:10.659 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-26T23:00:10.726 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-26T23:00:10.726 INFO:tasks.ceph:role: osd.7 2024-04-26T23:00:10.726 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T23:00:10.726 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T23:00:10.801 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T23:00:10.801 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T23:00:10.801 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T23:00:10.801 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-26T23:00:10.801 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T23:00:10.801 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-26T23:00:10.801 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T23:00:10.801 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-26T23:00:10.801 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T23:00:10.802 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T23:00:11.210 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-26T23:00:11.217 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-26T23:00:11.218 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-26T23:00:11.361 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-26T23:00:11.452 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T23:00:11.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:11.594+0000 7f7f3b01e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-26T23:00:11.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:11.595+0000 7f7f3b01e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-26T23:00:11.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:11.595+0000 7f7f3b01e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T23:00:11.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:11.595+0000 7f7f3b01e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T23:00:11.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:11.595+0000 7f7f3b01e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T23:00:11.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:11.599+0000 7f7f3b01e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-26T23:00:14.554 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-26T23:00:14.642 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T23:00:14.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:14.764+0000 7f900d811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-26T23:00:14.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:14.764+0000 7f900d811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-26T23:00:14.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:14.764+0000 7f900d811640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T23:00:14.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:14.764+0000 7f900d811640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T23:00:14.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:14.764+0000 7f900d811640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T23:00:14.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:14.767+0000 7f900d811640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-26T23:00:16.574 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:00:16.582 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:00:17.986 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-26T23:00:18.070 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T23:00:18.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:18.199+0000 7f5e6222f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-26T23:00:18.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:18.199+0000 7f5e6222f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-26T23:00:18.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:18.199+0000 7f5e6222f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-26T23:00:18.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:18.199+0000 7f5e6222f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-26T23:00:18.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:18.199+0000 7f5e6222f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-26T23:00:18.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:18.202+0000 7f5e6222f640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-26T23:00:21.412 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-26T23:00:21.495 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T23:00:21.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:21.621+0000 7f1674618640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-26T23:00:21.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:21.621+0000 7f1674618640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-26T23:00:21.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:21.621+0000 7f1674618640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-26T23:00:21.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:21.621+0000 7f1674618640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-26T23:00:21.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:21.621+0000 7f1674618640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-26T23:00:21.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:21.624+0000 7f1674618640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-26T23:00:24.565 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-26T23:00:24.650 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-26T23:00:24.650 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:00:24.650 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-26T23:00:24.747 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:00:24.747 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-26T23:00:24.834 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:00:24.834 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-26T23:00:24.925 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:00:24.925 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-26T23:00:25.016 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:00:25.016 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-26T23:00:25.104 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:00:25.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-26T23:00:25.189 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:00:25.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-26T23:00:25.276 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:00:25.276 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-26T23:00:25.367 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:00:25.367 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-26T23:00:25.450 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:00:25.450 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-26T23:00:25.531 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:00:25.531 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-26T23:00:25.611 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:00:25.611 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-26T23:00:25.694 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:00:25.694 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-26T23:00:25.776 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:00:25.776 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-26T23:00:25.862 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:00:25.862 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-26T23:00:25.920 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:00:25.920 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-26T23:00:25.974 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:00:25.975 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-26T23:00:26.031 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:00:26.031 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-26T23:00:26.084 INFO:tasks.ceph:Adding keys to all mons... 2024-04-26T23:00:26.084 DEBUG:teuthology.orchestra.run.smithi040:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T23:00:26.092 DEBUG:teuthology.orchestra.run.smithi103:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T23:00:26.171 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.y] 2024-04-26T23:00:26.171 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDmMSxmjjpcCBAAjDQRUBn/TK9hF3T2WBObyw== 2024-04-26T23:00:26.171 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.a] 2024-04-26T23:00:26.171 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDmMSxmF/UQHBAACdb8iskp2MuBMV6S5tYFJg== 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.c] 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDmMSxmLGZhJxAAg/fsrRrNoF/Jx1PnBVjjeQ== 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.0] 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDrMSxmbeONCxAATB67ujBDn2tVxpbMju7Mfg== 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.1] 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDuMSxmqE/PJhAAsNNu5ar8tHWQauxaXOvagA== 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.2] 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDyMSxmd2trBhAAbnl7EBldxgtOd1s0Ll0A9w== 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.3] 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQD1MSxmqdtKEhAA9oxBxOu/musCKyladrJgYA== 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.x] 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDmMSxmVf+9EhAAcpk8NgFqgCLXeHA2z7oHmA== 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.b] 2024-04-26T23:00:26.172 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDmMSxmniCyMxAAT/rVbIKj9EnvwDiblKMCeg== 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.d] 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDnMSxms3DnAxAAxJj/rybrv/0cLK8PmQ619A== 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.4] 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQD7MSxm+kl7IxAAlPAGIemdC8gcvp+koeULaQ== 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.5] 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQD+MSxmzKiaLRAA2UPS1wwJHgMxQf/zLMjMVQ== 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.6] 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQACMixmluHrCxAAmbmb33f5MVSTWNuy1D0Zsg== 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.7] 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAFMixm6GQjJRAArmdnX7ioekT8GSdsbeB1Ow== 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg== 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout:[client.2] 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDnMSxm3qhbGRAAK62blgPVShIYskDsMcaffQ== 2024-04-26T23:00:26.173 INFO:teuthology.orchestra.run.smithi103.stdout:[client.1] 2024-04-26T23:00:26.174 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDnMSxmVhn4IhAA94XsDk1x5Oax5smXPoFQKg== 2024-04-26T23:00:26.174 INFO:teuthology.orchestra.run.smithi103.stdout:[client.3] 2024-04-26T23:00:26.174 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDnMSxmxof/KxAAUmXVV4Vdp2DZnRjQzQ0Inw== 2024-04-26T23:00:26.175 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.y] 2024-04-26T23:00:26.175 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDmMSxmjjpcCBAAjDQRUBn/TK9hF3T2WBObyw== 2024-04-26T23:00:26.175 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.a] 2024-04-26T23:00:26.175 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDmMSxmF/UQHBAACdb8iskp2MuBMV6S5tYFJg== 2024-04-26T23:00:26.175 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.c] 2024-04-26T23:00:26.175 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDmMSxmLGZhJxAAg/fsrRrNoF/Jx1PnBVjjeQ== 2024-04-26T23:00:26.175 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.0] 2024-04-26T23:00:26.175 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDrMSxmbeONCxAATB67ujBDn2tVxpbMju7Mfg== 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.1] 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDuMSxmqE/PJhAAsNNu5ar8tHWQauxaXOvagA== 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.2] 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDyMSxmd2trBhAAbnl7EBldxgtOd1s0Ll0A9w== 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.3] 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQD1MSxmqdtKEhAA9oxBxOu/musCKyladrJgYA== 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.x] 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDmMSxmVf+9EhAAcpk8NgFqgCLXeHA2z7oHmA== 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.b] 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDmMSxmniCyMxAAT/rVbIKj9EnvwDiblKMCeg== 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.d] 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDnMSxms3DnAxAAxJj/rybrv/0cLK8PmQ619A== 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.4] 2024-04-26T23:00:26.176 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQD7MSxm+kl7IxAAlPAGIemdC8gcvp+koeULaQ== 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.5] 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQD+MSxmzKiaLRAA2UPS1wwJHgMxQf/zLMjMVQ== 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.6] 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQACMixmluHrCxAAmbmb33f5MVSTWNuy1D0Zsg== 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.7] 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAFMixm6GQjJRAArmdnX7ioekT8GSdsbeB1Ow== 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg== 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout:[client.2] 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDnMSxm3qhbGRAAK62blgPVShIYskDsMcaffQ== 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout:[client.1] 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDnMSxmVhn4IhAA94XsDk1x5Oax5smXPoFQKg== 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout:[client.3] 2024-04-26T23:00:26.177 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDnMSxmxof/KxAAUmXVV4Vdp2DZnRjQzQ0Inw== 2024-04-26T23:00:26.179 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T23:00:26.228 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T23:00:26.365 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-26T23:00:26.373 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-26T23:00:26.515 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-26T23:00:26.521 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-26T23:00:26.660 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:26.666 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:26.802 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:26.808 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:26.939 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:26.944 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:27.074 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:27.080 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:27.204 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T23:00:27.209 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T23:00:27.342 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-26T23:00:27.349 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-26T23:00:27.488 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-26T23:00:27.496 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-26T23:00:27.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:27.640 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:27.767 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:27.772 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:27.906 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:27.912 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:28.044 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:28.049 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T23:00:28.187 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T23:00:28.192 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T23:00:28.330 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T23:00:28.336 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T23:00:28.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T23:00:28.480 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T23:00:28.621 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T23:00:28.626 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T23:00:28.759 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-26T23:00:28.759 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-26T23:00:28.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T23:00:29.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-26T23:00:29.088 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-26T23:00:29.177 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T23:00:29.323 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-26T23:00:29.402 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-26T23:00:29.483 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T23:00:29.628 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-26T23:00:29.702 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T23:00:29.710 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T23:00:29.762 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-26T23:00:29.762 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T23:00:29.762 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T23:00:29.770 INFO:tasks.ceph.mon.a:Started 2024-04-26T23:00:29.770 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T23:00:29.770 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T23:00:29.776 INFO:tasks.ceph.mon.b:Started 2024-04-26T23:00:29.777 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T23:00:29.777 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T23:00:29.784 INFO:tasks.ceph.mon.c:Started 2024-04-26T23:00:29.784 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-26T23:00:29.784 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-26T23:00:29.784 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-26T23:00:29.792 INFO:tasks.ceph.mgr.y:Started 2024-04-26T23:00:29.792 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-26T23:00:29.792 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-26T23:00:29.796 INFO:tasks.ceph.mgr.x:Started 2024-04-26T23:00:29.796 INFO:tasks.ceph:Setting crush tunables to default 2024-04-26T23:00:29.796 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd crush tunables default 2024-04-26T23:00:29.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:29.972+0000 7fcd0ca00640 1 Processor -- start 2024-04-26T23:00:29.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:29.972+0000 7fcd0ca00640 1 -- start start 2024-04-26T23:00:29.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:29.973+0000 7fcd0ca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:29.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:29.973+0000 7fcd0ca00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd080d5d30 con 0x7fcd0815f680 2024-04-26T23:00:29.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:29.973+0000 7fcd0ca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd080d5a70 con 0x7fcd080d1770 2024-04-26T23:00:29.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:29.973+0000 7fcd0ca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcd080d3760 con 0x7fcd080b6470 2024-04-26T23:00:29.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:29.974+0000 7fcd06a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:29.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:29.974+0000 7fcd07e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcd0815f680 0x7fcd080cff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:40400/0 (socket says 172.21.15.40:40400) 2024-04-26T23:00:29.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:29.974+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 learned_addr learned my addr 172.21.15.40:0/2953148663 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:00:29.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:29.974+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcd080cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T23:00:29.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:29.974+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcd080cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:29.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:29.974+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:29.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:29.974+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:00:30.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:30.176+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:30.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:30.176+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcd080cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T23:00:30.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:30.176+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcd080cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:30.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:30.176+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:30.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:30.176+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:00:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:30.578+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:30.578+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcd080cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T23:00:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:30.578+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcd080cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:30.578+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:30.578+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T23:00:31.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:31.380+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:31.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:31.380+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcd080cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T23:00:31.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:31.380+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcd080cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:31.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:31.380+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:31.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:31.380+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T23:00:32.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.973+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcd080cf9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T23:00:32.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.973+0000 7fcd04c00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcd080cf9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:32.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.973+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 >> v1:172.21.15.103:6789/0 conn(0x7fcd080d1770 legacy=0x7fcd0815ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:32.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.974+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 >> v1:172.21.15.40:6789/0 conn(0x7fcd0815f680 legacy=0x7fcd080cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:32.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.974+0000 7fcd04c00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:32.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.974+0000 7fcd04c00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:32.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.974+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fcd080d3760 con 0x7fcce80061f0 2024-04-26T23:00:32.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.974+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcce8008d20 con 0x7fcce8003710 2024-04-26T23:00:32.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.974+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:32.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.975+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:32.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.975+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 msgr2=0x7fcce8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T23:00:32.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.975+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 msgr2=0x7fcce8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:32.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.975+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:32.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.975+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 msgr2=0x7fcce8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T23:00:32.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.975+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 msgr2=0x7fcce8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:32.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.975+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:32.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.975+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:00:32.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:32.975+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:00:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.176+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.176+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.177+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 msgr2=0x7fcce8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T23:00:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.177+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 msgr2=0x7fcce8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.177+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.177+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 msgr2=0x7fcce8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T23:00:33.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.177+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 msgr2=0x7fcce8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:33.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.177+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:33.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.177+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:00:33.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.177+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:00:33.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.578+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:33.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.578+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:33.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.578+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 msgr2=0x7fcce8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T23:00:33.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.579+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 msgr2=0x7fcce8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:33.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.579+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:33.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.579+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 msgr2=0x7fcce8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T23:00:33.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.579+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 msgr2=0x7fcce8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:33.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.579+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:33.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.579+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T23:00:33.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:33.579+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T23:00:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:34.380+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:34.381+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:34.381+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 msgr2=0x7fcce8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T23:00:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:34.381+0000 7fcd06a00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 msgr2=0x7fcce8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:34.381+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:34.381+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 msgr2=0x7fcce8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T23:00:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:34.381+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 msgr2=0x7fcce8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:34.381+0000 7fcd06a00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T23:00:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:34.381+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:34.381+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T23:00:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.974+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 msgr2=0x7fcce8005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T23:00:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.974+0000 7fcd04c00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003710 0x7fcce8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.974+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 msgr2=0x7fcce8008690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T23:00:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.974+0000 7fcd04c00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80061f0 0x7fcce8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.974+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 reap_dead start 2024-04-26T23:00:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.974+0000 7fcd04c00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.974+0000 7fcd04c00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.974+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd080b5b00 con 0x7fcd0815f680 2024-04-26T23:00:35.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.975+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcd080d3760 con 0x7fcd080b6470 2024-04-26T23:00:35.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.975+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcce8008d20 con 0x7fcce8003b50 2024-04-26T23:00:35.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.975+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:35.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.975+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:35.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.976+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcce800b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T23:00:35.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.976+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcce800b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:35.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.976+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:35.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.976+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:00:35.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.976+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 msgr2=0x7fcce8006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T23:00:35.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.976+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 msgr2=0x7fcce8006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:35.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.976+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:35.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:35.976+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:00:36.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.178+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:36.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.178+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:36.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.178+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcce800b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T23:00:36.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.178+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcce800b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:36.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.178+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 msgr2=0x7fcce8006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T23:00:36.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.178+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:36.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.178+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 msgr2=0x7fcce8006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:36.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.178+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:36.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.178+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:00:36.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.178+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:00:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.580+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.580+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.580+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcce800b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T23:00:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.580+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcce800b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.580+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.580+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 msgr2=0x7fcce8006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T23:00:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.580+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 msgr2=0x7fcce8006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.580+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.580+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T23:00:36.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:36.580+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T23:00:37.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:37.382+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:37.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:37.382+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:37.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:37.383+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcce800b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T23:00:37.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:37.383+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcce800b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:37.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:37.383+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:37.383+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T23:00:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:37.383+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 msgr2=0x7fcce8006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T23:00:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:37.383+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 msgr2=0x7fcce8006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:37.383+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:37.383+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T23:00:38.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:38.975+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 msgr2=0x7fcce8006680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T23:00:38.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:38.975+0000 7fcd04c00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce8003b50 0x7fcce8006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:38.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:38.975+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 msgr2=0x7fcce800b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T23:00:38.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:38.975+0000 7fcd04c00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.103:3300/0 conn(0x7fcd080b6470 0x7fcce800b190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:38.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:38.975+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 >> v1:172.21.15.40:6789/0 conn(0x7fcd0815f680 legacy=0x7fcce800d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:38.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:38.975+0000 7fcd04c00640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 0x7fcce80084b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:38.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:38.975+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcce8007d50 con 0x7fcce8006ca0 2024-04-26T23:00:38.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:38.975+0000 7fcd04c00640 1 -- 172.21.15.40:0/2953148663 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fcd080d3760 con 0x7fcce80080a0 2024-04-26T23:00:38.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:38.976+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 0x7fcce80084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:38.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:38.976+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 msgr2=0x7fcce80084b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T23:00:38.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:38.977+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 msgr2=0x7fcce80084b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:38.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:38.977+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 0x7fcce80084b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:38.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:38.977+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 0x7fcce80084b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:00:39.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:39.178+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 0x7fcce80084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:39.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:39.179+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 msgr2=0x7fcce80084b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T23:00:39.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:39.179+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 msgr2=0x7fcce80084b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:39.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:39.179+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 0x7fcce80084b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:39.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:39.179+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 0x7fcce80084b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:00:39.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:39.580+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 0x7fcce80084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:39.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:39.580+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 msgr2=0x7fcce80084b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T23:00:39.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:39.580+0000 7fcd07400640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 msgr2=0x7fcce80084b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T23:00:39.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:39.580+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 0x7fcce80084b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:00:39.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:39.580+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 0x7fcce80084b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T23:00:40.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.044+0000 7fcd06000640 1 -- 172.21.15.40:0/2953148663 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3133136523 0 0) 0x7fcce8007d50 con 0x7fcce8006ca0 2024-04-26T23:00:40.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.044+0000 7fcd06000640 1 -- 172.21.15.40:0/2953148663 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccec003520 con 0x7fcce8006ca0 2024-04-26T23:00:40.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.051+0000 7fcd06000640 1 -- 172.21.15.40:0/2953148663 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 683200925 0 0) 0x7fccec003520 con 0x7fcce8006ca0 2024-04-26T23:00:40.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.051+0000 7fcd06000640 1 -- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 msgr2=0x7fcce80084b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T23:00:40.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.051+0000 7fcd06000640 1 --2- 172.21.15.40:0/2953148663 >> v2:172.21.15.40:3300/0 conn(0x7fcce80080a0 0x7fcce80084b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:40.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.052+0000 7fcd06000640 1 -- 172.21.15.40:0/2953148663 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce8007920 con 0x7fcce8006ca0 2024-04-26T23:00:40.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.052+0000 7fcd06000640 1 -- 172.21.15.40:0/2953148663 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fccf4015580 con 0x7fcce8006ca0 2024-04-26T23:00:40.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.052+0000 7fcd06000640 1 -- 172.21.15.40:0/2953148663 >> v1:172.21.15.103:6789/0 conn(0x7fcce8006ca0 legacy=0x7fcce8007110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:40.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.052+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 reap_dead start 2024-04-26T23:00:40.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.052+0000 7fcd06000640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccec0039b0 0x7fccec003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:40.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.052+0000 7fcd06000640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccec004800 0x7fccec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:40.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.052+0000 7fcd06000640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccec005210 0x7fccec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:40.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.052+0000 7fcd06000640 1 -- 172.21.15.40:0/2953148663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcd080d3760 con 0x7fccec004800 2024-04-26T23:00:40.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.052+0000 7fcd06000640 1 -- 172.21.15.40:0/2953148663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fccec00c140 con 0x7fccec005210 2024-04-26T23:00:40.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.052+0000 7fcd06000640 1 -- 172.21.15.40:0/2953148663 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fccec00c340 con 0x7fccec0039b0 2024-04-26T23:00:40.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.053+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccec005210 0x7fccec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:40.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.053+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccec0039b0 0x7fccec003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:40.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.054+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccec0039b0 msgr2=0x7fccec003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:40.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.054+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccec0039b0 0x7fccec003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:40.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.055+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccec004800 msgr2=0x7fccec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:40.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.055+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccec004800 0x7fccec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:40.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.055+0000 7fcd07e00640 1 -- 172.21.15.40:0/2953148663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce8007920 con 0x7fccec005210 2024-04-26T23:00:40.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.055+0000 7fcd07400640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccec0039b0 0x7fccec003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:00:40.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.058+0000 7fcd07e00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccec005210 0x7fccec00b9c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcd0808d510 tx=0x7fccec005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8cf11a741e75036 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:40.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.060+0000 7fcd06000640 1 -- 172.21.15.40:0/2953148663 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd00039070 con 0x7fccec005210 2024-04-26T23:00:40.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.060+0000 7fcd06000640 1 -- 172.21.15.40:0/2953148663 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0002ba80 con 0x7fccec005210 2024-04-26T23:00:40.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.060+0000 7fcd06000640 1 -- 172.21.15.40:0/2953148663 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd00031070 con 0x7fccec005210 2024-04-26T23:00:40.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.061+0000 7fcd0ca00640 1 -- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccec005210 msgr2=0x7fccec00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:40.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.061+0000 7fcd0ca00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccec005210 0x7fccec00b9c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcd0808d510 tx=0x7fccec005830 comp rx=0 tx=0).stop 2024-04-26T23:00:40.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.061+0000 7fcd0ca00640 1 -- 172.21.15.40:0/2953148663 shutdown_connections 2024-04-26T23:00:40.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.061+0000 7fcd0ca00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccec005210 0x7fccec00b9c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:40.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.061+0000 7fcd0ca00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccec004800 0x7fccec004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:40.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.061+0000 7fcd0ca00640 1 --2- 172.21.15.40:0/2953148663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccec0039b0 0x7fccec003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:40.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.061+0000 7fcd0ca00640 1 -- 172.21.15.40:0/2953148663 >> 172.21.15.40:0/2953148663 conn(0x7fcd080d4dd0 msgr2=0x7fcd080d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:40.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.062+0000 7fcd0ca00640 1 -- 172.21.15.40:0/2953148663 shutdown_connections 2024-04-26T23:00:40.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.062+0000 7fcd0ca00640 1 -- 172.21.15.40:0/2953148663 wait complete. 2024-04-26T23:00:40.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.066+0000 7fcd0ca00640 1 Processor -- start 2024-04-26T23:00:40.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.067+0000 7fcd0ca00640 1 -- start start 2024-04-26T23:00:40.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.068+0000 7fcd0ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccec0039b0 0x7fcd081513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:40.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.068+0000 7fcd0ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccec004800 0x7fcd080cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:40.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.068+0000 7fcd0ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccec005210 0x7fcd080cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:40.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.068+0000 7fcd0ca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcd080d3d70 con 0x7fccec005210 2024-04-26T23:00:40.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.068+0000 7fcd0ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcd080d3bf0 con 0x7fccec004800 2024-04-26T23:00:40.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.068+0000 7fcd0ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcd080d1b50 con 0x7fccec0039b0 2024-04-26T23:00:40.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.069+0000 7fcd07e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccec005210 0x7fcd080cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:40.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.069+0000 7fcd07e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccec005210 0x7fcd080cbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39616/0 (socket says 172.21.15.40:39616) 2024-04-26T23:00:40.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.069+0000 7fcd07e00640 1 -- 172.21.15.40:0/4238113345 learned_addr learned my addr 172.21.15.40:0/4238113345 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:00:40.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.069+0000 7fcd06a00640 1 --2- 172.21.15.40:0/4238113345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccec004800 0x7fcd080cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:40.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.069+0000 7fcd07400640 1 --2- 172.21.15.40:0/4238113345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccec0039b0 0x7fcd081513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:40.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.069+0000 7fcd07e00640 1 -- 172.21.15.40:0/4238113345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccec0039b0 msgr2=0x7fcd081513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:40.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.069+0000 7fcd07e00640 1 --2- 172.21.15.40:0/4238113345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccec0039b0 0x7fcd081513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:40.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.070+0000 7fcd07e00640 1 -- 172.21.15.40:0/4238113345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccec004800 msgr2=0x7fcd080cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:40.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.070+0000 7fcd07e00640 1 --2- 172.21.15.40:0/4238113345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccec004800 0x7fcd080cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:40.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.070+0000 7fcd07e00640 1 -- 172.21.15.40:0/4238113345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd08151900 con 0x7fccec005210 2024-04-26T23:00:40.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.071+0000 7fcd07e00640 1 --2- 172.21.15.40:0/4238113345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccec005210 0x7fcd080cbea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcd00037040 tx=0x7fcd000014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:40.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.072+0000 7fccff400640 1 -- 172.21.15.40:0/4238113345 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd00046070 con 0x7fccec005210 2024-04-26T23:00:40.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.072+0000 7fccff400640 1 -- 172.21.15.40:0/4238113345 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd000053e0 con 0x7fccec005210 2024-04-26T23:00:40.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.072+0000 7fccff400640 1 -- 172.21.15.40:0/4238113345 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd00039040 con 0x7fccec005210 2024-04-26T23:00:40.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.072+0000 7fcd0ca00640 1 -- 172.21.15.40:0/4238113345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd080cc860 con 0x7fccec005210 2024-04-26T23:00:40.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.073+0000 7fcd0ca00640 1 -- 172.21.15.40:0/4238113345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd080cce20 con 0x7fccec005210 2024-04-26T23:00:40.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.073+0000 7fccff400640 1 -- 172.21.15.40:0/4238113345 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcd00049070 con 0x7fccec005210 2024-04-26T23:00:40.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.073+0000 7fccff400640 1 -- 172.21.15.40:0/4238113345 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcd0003fab0 con 0x7fccec005210 2024-04-26T23:00:40.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.074+0000 7fcd0ca00640 1 -- 172.21.15.40:0/4238113345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd08069370 con 0x7fccec005210 2024-04-26T23:00:40.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.077+0000 7fccff400640 1 -- 172.21.15.40:0/4238113345 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fcd0004f060 con 0x7fccec005210 2024-04-26T23:00:40.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.154+0000 7fcd0ca00640 1 -- 172.21.15.40:0/4238113345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fcd080cfa40 con 0x7fccec005210 2024-04-26T23:00:40.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.159+0000 7fccff400640 1 -- 172.21.15.40:0/4238113345 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fcd08151900 con 0x7fccec005210 2024-04-26T23:00:40.159 INFO:teuthology.orchestra.run.smithi040.stderr:adjusted tunables profile to default 2024-04-26T23:00:40.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.161+0000 7fcd0ca00640 1 -- 172.21.15.40:0/4238113345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccec005210 msgr2=0x7fcd080cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:40.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.161+0000 7fcd0ca00640 1 --2- 172.21.15.40:0/4238113345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccec005210 0x7fcd080cbea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcd00037040 tx=0x7fcd000014a0 comp rx=0 tx=0).stop 2024-04-26T23:00:40.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.161+0000 7fcd0ca00640 1 -- 172.21.15.40:0/4238113345 shutdown_connections 2024-04-26T23:00:40.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.161+0000 7fcd0ca00640 1 --2- 172.21.15.40:0/4238113345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccec005210 0x7fcd080cbea0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:40.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.161+0000 7fcd0ca00640 1 --2- 172.21.15.40:0/4238113345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccec004800 0x7fcd080cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:40.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.161+0000 7fcd0ca00640 1 --2- 172.21.15.40:0/4238113345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccec0039b0 0x7fcd081513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:40.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.161+0000 7fcd0ca00640 1 -- 172.21.15.40:0/4238113345 >> 172.21.15.40:0/4238113345 conn(0x7fcd080d4dd0 msgr2=0x7fcd0815d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:40.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.161+0000 7fcd0ca00640 1 -- 172.21.15.40:0/4238113345 shutdown_connections 2024-04-26T23:00:40.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:40.161+0000 7fcd0ca00640 1 -- 172.21.15.40:0/4238113345 wait complete. 2024-04-26T23:00:40.180 INFO:tasks.ceph:check_enable_crimson: False 2024-04-26T23:00:40.180 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-26T23:00:40.181 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:00:40.181 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-26T23:00:40.227 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:40.227+0000 7f12fb01f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T23:00:40.237 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:40.237+0000 7f251a9ce040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T23:00:40.258 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:00:40.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-26T23:00:40.386 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:00:40.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-26T23:00:40.449 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:40.449+0000 7f12fb01f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:00:40.451 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:40.450+0000 7f251a9ce040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:00:40.463 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:00:40.463 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-26T23:00:40.518 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:40.518+0000 7f251a9ce040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T23:00:40.519 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:40.519+0000 7f12fb01f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T23:00:40.543 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:00:40.543 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-26T23:00:40.620 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:00:40.620 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-26T23:00:40.688 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:00:40.688 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-26T23:00:40.804 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:00:40.804 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-26T23:00:40.824 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:40.824+0000 7f251a9ce040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T23:00:40.832 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:40.832+0000 7f12fb01f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T23:00:40.877 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 042d5bb0-3c0d-45e5-83d2-a51234697407 0 2024-04-26T23:00:40.885 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:40.885+0000 7f251a9ce040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T23:00:40.890 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:40.889+0000 7f12fb01f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T23:00:41.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.058+0000 7fe2d5c00640 1 Processor -- start 2024-04-26T23:00:41.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.058+0000 7fe2d5c00640 1 -- start start 2024-04-26T23:00:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.060+0000 7fe2d5c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2d009aaf0 0x7fe2d00d0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.060+0000 7fe2d5c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2d00cdf10 con 0x7fe2d0074090 2024-04-26T23:00:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.060+0000 7fe2d5c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2d00ce1f0 con 0x7fe2d01627f0 2024-04-26T23:00:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.060+0000 7fe2d5c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe2d00cdcc0 con 0x7fe2d009aaf0 2024-04-26T23:00:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.060+0000 7fe2cfe00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe2d01627f0 0x7fe2d0073980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:34828/0 (socket says 172.21.15.103:34828) 2024-04-26T23:00:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.060+0000 7fe2cfe00640 1 -- 172.21.15.103:0/545964433 learned_addr learned my addr 172.21.15.103:0/545964433 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-04-26T23:00:41.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.060+0000 7fe2d5200640 1 --2- 172.21.15.103:0/545964433 >> v2:172.21.15.40:3300/0 conn(0x7fe2d009aaf0 0x7fe2d00d0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.060+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2094808720 0 0) 0x7fe2d00ce1f0 con 0x7fe2d01627f0 2024-04-26T23:00:41.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.060+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2b4003520 con 0x7fe2d01627f0 2024-04-26T23:00:41.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2530429011 0 0) 0x7fe2b4003520 con 0x7fe2d01627f0 2024-04-26T23:00:41.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 >> v2:172.21.15.40:3300/0 conn(0x7fe2d009aaf0 msgr2=0x7fe2d00d0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:41.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cea00640 1 --2- 172.21.15.103:0/545964433 >> v2:172.21.15.40:3300/0 conn(0x7fe2d009aaf0 0x7fe2d00d0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 >> v1:172.21.15.40:6789/0 conn(0x7fe2d0074090 legacy=0x7fe2d00cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d00ce4d0 con 0x7fe2d01627f0 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fe2c0001670 con 0x7fe2d01627f0 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 >> v1:172.21.15.103:6789/0 conn(0x7fe2d01627f0 legacy=0x7fe2d0073980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cea00640 1 --2- 172.21.15.103:0/545964433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b40039b0 0x7fe2b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cea00640 1 --2- 172.21.15.103:0/545964433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b4004800 0x7fe2b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cea00640 1 --2- 172.21.15.103:0/545964433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2b40051c0 0x7fe2b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d00cdcc0 con 0x7fe2b40051c0 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b400c160 con 0x7fe2b4004800 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2b400c360 con 0x7fe2b40039b0 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cfe00640 1 --2- 172.21.15.103:0/545964433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b40039b0 0x7fe2b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.061+0000 7fe2cf400640 1 --2- 172.21.15.103:0/545964433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b4004800 0x7fe2b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.062+0000 7fe2cfe00640 1 -- 172.21.15.103:0/545964433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b4004800 msgr2=0x7fe2b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.062+0000 7fe2cfe00640 1 --2- 172.21.15.103:0/545964433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b4004800 0x7fe2b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.062+0000 7fe2cfe00640 1 -- 172.21.15.103:0/545964433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2b40051c0 msgr2=0x7fe2b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:41.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.062+0000 7fe2d5200640 1 --2- 172.21.15.103:0/545964433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2b40051c0 0x7fe2b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.062+0000 7fe2cfe00640 1 --2- 172.21.15.103:0/545964433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2b40051c0 0x7fe2b400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.062+0000 7fe2cfe00640 1 -- 172.21.15.103:0/545964433 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b4003520 con 0x7fe2b40039b0 2024-04-26T23:00:41.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.062+0000 7fe2d5200640 1 -- 172.21.15.103:0/545964433 reap_dead start 2024-04-26T23:00:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.063+0000 7fe2cfe00640 1 --2- 172.21.15.103:0/545964433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b40039b0 0x7fe2b4003dd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c210 tx=0x7fe2b4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25217ac35d7feec0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.063+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0007d30 con 0x7fe2b40039b0 2024-04-26T23:00:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.063+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2c0005ad0 con 0x7fe2b40039b0 2024-04-26T23:00:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.063+0000 7fe2cea00640 1 -- 172.21.15.103:0/545964433 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0007070 con 0x7fe2b40039b0 2024-04-26T23:00:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.064+0000 7fe2d5c00640 1 -- 172.21.15.103:0/545964433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b40039b0 msgr2=0x7fe2b4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.064+0000 7fe2d5c00640 1 --2- 172.21.15.103:0/545964433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b40039b0 0x7fe2b4003dd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c210 tx=0x7fe2b4005830 comp rx=0 tx=0).stop 2024-04-26T23:00:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.064+0000 7fe2d5c00640 1 -- 172.21.15.103:0/545964433 shutdown_connections 2024-04-26T23:00:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.064+0000 7fe2d5c00640 1 --2- 172.21.15.103:0/545964433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b40039b0 0x7fe2b4003dd0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.064+0000 7fe2d5c00640 1 -- 172.21.15.103:0/545964433 >> 172.21.15.103:0/545964433 conn(0x7fe2d00b2ee0 msgr2=0x7fe2d00b32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:41.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.064+0000 7fe2d5c00640 1 -- 172.21.15.103:0/545964433 shutdown_connections 2024-04-26T23:00:41.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.064+0000 7fe2d5c00640 1 -- 172.21.15.103:0/545964433 wait complete. 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.066+0000 7fe2d5c00640 1 Processor -- start 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.066+0000 7fe2d5c00640 1 -- start start 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2d5c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2b40039b0 0x7fe2d0150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2d5c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d0159790 0x7fe2d00b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2d5c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d0159ce0 0x7fe2d00b5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2d5c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d00d0cc0 con 0x7fe2b40039b0 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2d5c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d00cea90 con 0x7fe2d0159ce0 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2d5c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d00d0e40 con 0x7fe2d0159790 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2cf400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d0159790 0x7fe2d00b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2cf400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d0159790 0x7fe2d00b53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:60368/0 (socket says 172.21.15.103:60368) 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2cf400640 1 -- 172.21.15.103:0/3849972151 learned_addr learned my addr 172.21.15.103:0/3849972151 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2d5200640 1 --2- 172.21.15.103:0/3849972151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d0159ce0 0x7fe2d00b5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2cfe00640 1 --2- 172.21.15.103:0/3849972151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2b40039b0 0x7fe2d0150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2cf400640 1 -- 172.21.15.103:0/3849972151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d0159ce0 msgr2=0x7fe2d00b5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2cf400640 1 --2- 172.21.15.103:0/3849972151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d0159ce0 0x7fe2d00b5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2cf400640 1 -- 172.21.15.103:0/3849972151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2b40039b0 msgr2=0x7fe2d0150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2cf400640 1 --2- 172.21.15.103:0/3849972151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2b40039b0 0x7fe2d0150660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.067+0000 7fe2cf400640 1 -- 172.21.15.103:0/3849972151 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d00b4340 con 0x7fe2d0159790 2024-04-26T23:00:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.068+0000 7fe2cf400640 1 --2- 172.21.15.103:0/3849972151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d0159790 0x7fe2d00b53e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc007810 tx=0x7fe2bc00fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.068+0000 7fe2c7e00640 1 -- 172.21.15.103:0/3849972151 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc0050d0 con 0x7fe2d0159790 2024-04-26T23:00:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.068+0000 7fe2c7e00640 1 -- 172.21.15.103:0/3849972151 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2bc018070 con 0x7fe2d0159790 2024-04-26T23:00:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.068+0000 7fe2c7e00640 1 -- 172.21.15.103:0/3849972151 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc005490 con 0x7fe2d0159790 2024-04-26T23:00:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.068+0000 7fe2d5c00640 1 -- 172.21.15.103:0/3849972151 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d00b4620 con 0x7fe2d0159790 2024-04-26T23:00:41.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.068+0000 7fe2d5c00640 1 -- 172.21.15.103:0/3849972151 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d00b4be0 con 0x7fe2d0159790 2024-04-26T23:00:41.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.068+0000 7fe2c7e00640 1 -- 172.21.15.103:0/3849972151 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe2bc00d6d0 con 0x7fe2d0159790 2024-04-26T23:00:41.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.068+0000 7fe2c7e00640 1 -- 172.21.15.103:0/3849972151 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fe2bc003c10 con 0x7fe2d0159790 2024-04-26T23:00:41.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.069+0000 7fe2d5c00640 1 -- 172.21.15.103:0/3849972151 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2d0069a80 con 0x7fe2d0159790 2024-04-26T23:00:41.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.074+0000 7fe2c7e00640 1 -- 172.21.15.103:0/3849972151 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fe2bc003e10 con 0x7fe2d0159790 2024-04-26T23:00:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.154+0000 7fe2d5c00640 1 -- 172.21.15.103:0/3849972151 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd new", "uuid": "042d5bb0-3c0d-45e5-83d2-a51234697407", "id": 0} v 0) -- 0x7fe2d0075a40 con 0x7fe2d0159790 2024-04-26T23:00:41.161 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.162+0000 7fe2c7e00640 1 -- 172.21.15.103:0/3849972151 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "042d5bb0-3c0d-45e5-83d2-a51234697407", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fe2bc0035f0 con 0x7fe2d0159790 2024-04-26T23:00:41.162 INFO:teuthology.orchestra.run.smithi103.stdout:0 2024-04-26T23:00:41.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.163+0000 7fe2d5c00640 1 -- 172.21.15.103:0/3849972151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d0159790 msgr2=0x7fe2d00b53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.163+0000 7fe2d5c00640 1 --2- 172.21.15.103:0/3849972151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d0159790 0x7fe2d00b53e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc007810 tx=0x7fe2bc00fc00 comp rx=0 tx=0).stop 2024-04-26T23:00:41.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.163+0000 7fe2d5c00640 1 -- 172.21.15.103:0/3849972151 shutdown_connections 2024-04-26T23:00:41.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.163+0000 7fe2d5c00640 1 --2- 172.21.15.103:0/3849972151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d0159ce0 0x7fe2d00b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.163+0000 7fe2d5c00640 1 --2- 172.21.15.103:0/3849972151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2d0159790 0x7fe2d00b53e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.163+0000 7fe2d5c00640 1 --2- 172.21.15.103:0/3849972151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2b40039b0 0x7fe2d0150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.163+0000 7fe2d5c00640 1 -- 172.21.15.103:0/3849972151 >> 172.21.15.103:0/3849972151 conn(0x7fe2d00b2ee0 msgr2=0x7fe2d00c64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:41.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.163+0000 7fe2d5c00640 1 -- 172.21.15.103:0/3849972151 shutdown_connections 2024-04-26T23:00:41.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.163+0000 7fe2d5c00640 1 -- 172.21.15.103:0/3849972151 wait complete. 2024-04-26T23:00:41.180 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 6973225f-5f76-4c21-9204-a664b8eed95e 1 2024-04-26T23:00:41.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.360+0000 7f7f87800640 1 Processor -- start 2024-04-26T23:00:41.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.361+0000 7f7f87800640 1 -- start start 2024-04-26T23:00:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.362+0000 7f7f87800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800d06f0 0x7f7f800d1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.362+0000 7f7f87800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7f800b2850 0x7f7f800b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.362+0000 7f7f87800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f800d0b70 con 0x7f7f8015f3a0 2024-04-26T23:00:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.362+0000 7f7f87800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f7f800d0e20 con 0x7f7f800b2850 2024-04-26T23:00:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.362+0000 7f7f87800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7f800b1d70 con 0x7f7f800d06f0 2024-04-26T23:00:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.362+0000 7f7f7fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800d06f0 0x7f7f800d1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.362+0000 7f7f7fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800d06f0 0x7f7f800d1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:60370/0 (socket says 172.21.15.103:60370) 2024-04-26T23:00:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.362+0000 7f7f7fe00640 1 -- 172.21.15.103:0/4229163804 learned_addr learned my addr 172.21.15.103:0/4229163804 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:00:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.363+0000 7f7f85800640 1 --2- 172.21.15.103:0/4229163804 >> v2:172.21.15.40:3300/0 conn(0x7f7f800b2850 0x7f7f800b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.363+0000 7f7f7fe00640 1 -- 172.21.15.103:0/4229163804 >> v2:172.21.15.40:3300/0 conn(0x7f7f800b2850 msgr2=0x7f7f800b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:41.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.363+0000 7f7f7fe00640 1 --2- 172.21.15.103:0/4229163804 >> v2:172.21.15.40:3300/0 conn(0x7f7f800b2850 0x7f7f800b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.363+0000 7f7f7fe00640 1 -- 172.21.15.103:0/4229163804 >> v1:172.21.15.40:6789/0 conn(0x7f7f8015f3a0 legacy=0x7f7f800d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.363+0000 7f7f7fe00640 1 -- 172.21.15.103:0/4229163804 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f800b2050 con 0x7f7f800d06f0 2024-04-26T23:00:41.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.364+0000 7f7f7fe00640 1 --2- 172.21.15.103:0/4229163804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800d06f0 0x7f7f800d1e80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009870 tx=0x7f7f6c02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbfa718efe84d4f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:41.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.365+0000 7f7f7f400640 1 -- 172.21.15.103:0/4229163804 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c037070 con 0x7f7f800d06f0 2024-04-26T23:00:41.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.365+0000 7f7f7f400640 1 -- 172.21.15.103:0/4229163804 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f6c032440 con 0x7f7f800d06f0 2024-04-26T23:00:41.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.365+0000 7f7f7f400640 1 -- 172.21.15.103:0/4229163804 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c032810 con 0x7f7f800d06f0 2024-04-26T23:00:41.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.365+0000 7f7f87800640 1 -- 172.21.15.103:0/4229163804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800d06f0 msgr2=0x7f7f800d1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.365+0000 7f7f87800640 1 --2- 172.21.15.103:0/4229163804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800d06f0 0x7f7f800d1e80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009870 tx=0x7f7f6c02a250 comp rx=0 tx=0).stop 2024-04-26T23:00:41.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.365+0000 7f7f87800640 1 -- 172.21.15.103:0/4229163804 shutdown_connections 2024-04-26T23:00:41.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.365+0000 7f7f87800640 1 --2- 172.21.15.103:0/4229163804 >> v2:172.21.15.40:3300/0 conn(0x7f7f800b2850 0x7f7f800b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.365+0000 7f7f87800640 1 --2- 172.21.15.103:0/4229163804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800d06f0 0x7f7f800d1e80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.365+0000 7f7f87800640 1 -- 172.21.15.103:0/4229163804 >> 172.21.15.103:0/4229163804 conn(0x7f7f800d4f90 msgr2=0x7f7f800d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:41.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.366+0000 7f7f87800640 1 -- 172.21.15.103:0/4229163804 shutdown_connections 2024-04-26T23:00:41.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.366+0000 7f7f87800640 1 -- 172.21.15.103:0/4229163804 wait complete. 2024-04-26T23:00:41.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.367+0000 7f7f87800640 1 Processor -- start 2024-04-26T23:00:41.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.367+0000 7f7f87800640 1 -- start start 2024-04-26T23:00:41.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.368+0000 7f7f87800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800b2850 0x7f7f80152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.368+0000 7f7f87800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f800d06f0 0x7f7f80153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.368+0000 7f7f87800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8015f3a0 0x7f7f80151b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.368+0000 7f7f87800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8015f900 con 0x7f7f800d06f0 2024-04-26T23:00:41.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.368+0000 7f7f87800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8015f780 con 0x7f7f8015f3a0 2024-04-26T23:00:41.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.368+0000 7f7f87800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7f800d23c0 con 0x7f7f800b2850 2024-04-26T23:00:41.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.369+0000 7f7f84e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800b2850 0x7f7f80152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.369+0000 7f7f84e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800b2850 0x7f7f80152540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:60378/0 (socket says 172.21.15.103:60378) 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.369+0000 7f7f84e00640 1 -- 172.21.15.103:0/1009696115 learned_addr learned my addr 172.21.15.103:0/1009696115 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.369+0000 7f7f85800640 1 --2- 172.21.15.103:0/1009696115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8015f3a0 0x7f7f80151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.369+0000 7f7f7fe00640 1 --2- 172.21.15.103:0/1009696115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f800d06f0 0x7f7f80153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.369+0000 7f7f84e00640 1 -- 172.21.15.103:0/1009696115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8015f3a0 msgr2=0x7f7f80151b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.369+0000 7f7f84e00640 1 --2- 172.21.15.103:0/1009696115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8015f3a0 0x7f7f80151b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.369+0000 7f7f84e00640 1 -- 172.21.15.103:0/1009696115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f800d06f0 msgr2=0x7f7f80153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.369+0000 7f7f84e00640 1 --2- 172.21.15.103:0/1009696115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f800d06f0 0x7f7f80153c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.369+0000 7f7f84e00640 1 -- 172.21.15.103:0/1009696115 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f80152b90 con 0x7f7f800b2850 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.369+0000 7f7f84e00640 1 --2- 172.21.15.103:0/1009696115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800b2850 0x7f7f80152540 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400c400 tx=0x7f7f7400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.370+0000 7f7f7cc00640 1 -- 172.21.15.103:0/1009696115 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f74017070 con 0x7f7f800b2850 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.370+0000 7f7f7cc00640 1 -- 172.21.15.103:0/1009696115 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f740052d0 con 0x7f7f800b2850 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.370+0000 7f7f7cc00640 1 -- 172.21.15.103:0/1009696115 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f74003830 con 0x7f7f800b2850 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.370+0000 7f7f87800640 1 -- 172.21.15.103:0/1009696115 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f80152e70 con 0x7f7f800b2850 2024-04-26T23:00:41.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.370+0000 7f7f87800640 1 -- 172.21.15.103:0/1009696115 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f80153430 con 0x7f7f800b2850 2024-04-26T23:00:41.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.370+0000 7f7f7cc00640 1 -- 172.21.15.103:0/1009696115 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7f7401a070 con 0x7f7f800b2850 2024-04-26T23:00:41.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.370+0000 7f7f7cc00640 1 -- 172.21.15.103:0/1009696115 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f7f74011b80 con 0x7f7f800b2850 2024-04-26T23:00:41.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.371+0000 7f7f87800640 1 -- 172.21.15.103:0/1009696115 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f800694f0 con 0x7f7f800b2850 2024-04-26T23:00:41.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.374+0000 7f7f7cc00640 1 -- 172.21.15.103:0/1009696115 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f7f74021070 con 0x7f7f800b2850 2024-04-26T23:00:41.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.451+0000 7f7f87800640 1 -- 172.21.15.103:0/1009696115 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6973225f-5f76-4c21-9204-a664b8eed95e", "id": 1} v 0) -- 0x7f7f800b6860 con 0x7f7f800b2850 2024-04-26T23:00:41.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.459+0000 7f7f7cc00640 1 -- 172.21.15.103:0/1009696115 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6973225f-5f76-4c21-9204-a664b8eed95e", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f7f74011df0 con 0x7f7f800b2850 2024-04-26T23:00:41.459 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-26T23:00:41.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.460+0000 7f7f87800640 1 -- 172.21.15.103:0/1009696115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800b2850 msgr2=0x7f7f80152540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.460+0000 7f7f87800640 1 --2- 172.21.15.103:0/1009696115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800b2850 0x7f7f80152540 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400c400 tx=0x7f7f7400c8c0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.460+0000 7f7f87800640 1 -- 172.21.15.103:0/1009696115 shutdown_connections 2024-04-26T23:00:41.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.460+0000 7f7f87800640 1 --2- 172.21.15.103:0/1009696115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8015f3a0 0x7f7f80151b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.460+0000 7f7f87800640 1 --2- 172.21.15.103:0/1009696115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f800d06f0 0x7f7f80153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.460+0000 7f7f87800640 1 --2- 172.21.15.103:0/1009696115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f800b2850 0x7f7f80152540 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.460+0000 7f7f87800640 1 -- 172.21.15.103:0/1009696115 >> 172.21.15.103:0/1009696115 conn(0x7f7f800d4f90 msgr2=0x7f7f8006cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:41.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.460+0000 7f7f87800640 1 -- 172.21.15.103:0/1009696115 shutdown_connections 2024-04-26T23:00:41.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.460+0000 7f7f87800640 1 -- 172.21.15.103:0/1009696115 wait complete. 2024-04-26T23:00:41.476 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1 2 2024-04-26T23:00:41.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.642+0000 7f13de400640 1 Processor -- start 2024-04-26T23:00:41.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.643+0000 7f13de400640 1 -- start start 2024-04-26T23:00:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.644+0000 7f13de400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f13d815f3a0 0x7f13d80d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.644+0000 7f13de400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f13d80d1770 0x7f13d80d1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.644+0000 7f13de400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d80d2590 con 0x7f13d80d2160 2024-04-26T23:00:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.644+0000 7f13de400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f13d80d0e20 con 0x7f13d815f3a0 2024-04-26T23:00:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.644+0000 7f13de400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f13d80b1d70 con 0x7f13d80d1770 2024-04-26T23:00:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.644+0000 7f13d7e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f13d80d2160 0x7f13d80d2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:34832/0 (socket says 172.21.15.103:34832) 2024-04-26T23:00:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.644+0000 7f13d7e00640 1 -- 172.21.15.103:0/3126457911 learned_addr learned my addr 172.21.15.103:0/3126457911 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-04-26T23:00:41.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.644+0000 7f13d6a00640 1 --2- 172.21.15.103:0/3126457911 >> v2:172.21.15.103:3300/0 conn(0x7f13d80d1770 0x7f13d80d1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.644+0000 7f13d7400640 1 --2- 172.21.15.103:0/3126457911 >> v2:172.21.15.40:3300/0 conn(0x7f13d815f3a0 0x7f13d80d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.645+0000 7f13d6a00640 1 -- 172.21.15.103:0/3126457911 >> v2:172.21.15.40:3300/0 conn(0x7f13d815f3a0 msgr2=0x7f13d80d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.645+0000 7f13d6a00640 1 --2- 172.21.15.103:0/3126457911 >> v2:172.21.15.40:3300/0 conn(0x7f13d815f3a0 0x7f13d80d06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.644+0000 7f13d6000640 1 -- 172.21.15.103:0/3126457911 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2028327160 0 0) 0x7f13d80d2590 con 0x7f13d80d2160 2024-04-26T23:00:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.645+0000 7f13d6a00640 1 -- 172.21.15.103:0/3126457911 >> v1:172.21.15.103:6789/0 conn(0x7f13d80d2160 legacy=0x7f13d80d2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.645+0000 7f13d6a00640 1 -- 172.21.15.103:0/3126457911 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d80b2050 con 0x7f13d80d1770 2024-04-26T23:00:41.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.646+0000 7f13d6a00640 1 --2- 172.21.15.103:0/3126457911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d80d1770 0x7f13d80d1c20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f13d80b7b80 tx=0x7f13c402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4eec7966d27c2454 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:41.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.646+0000 7f13d6000640 1 -- 172.21.15.103:0/3126457911 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4037070 con 0x7f13d80d1770 2024-04-26T23:00:41.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.646+0000 7f13d6000640 1 -- 172.21.15.103:0/3126457911 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c4032440 con 0x7f13d80d1770 2024-04-26T23:00:41.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.646+0000 7f13d6000640 1 -- 172.21.15.103:0/3126457911 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4032810 con 0x7f13d80d1770 2024-04-26T23:00:41.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.646+0000 7f13de400640 1 -- 172.21.15.103:0/3126457911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d80d1770 msgr2=0x7f13d80d1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.646+0000 7f13de400640 1 --2- 172.21.15.103:0/3126457911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d80d1770 0x7f13d80d1c20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f13d80b7b80 tx=0x7f13c402a1a0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.647+0000 7f13de400640 1 -- 172.21.15.103:0/3126457911 shutdown_connections 2024-04-26T23:00:41.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.647+0000 7f13de400640 1 --2- 172.21.15.103:0/3126457911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d80d1770 0x7f13d80d1c20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.647+0000 7f13de400640 1 --2- 172.21.15.103:0/3126457911 >> v2:172.21.15.40:3300/0 conn(0x7f13d815f3a0 0x7f13d80d06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.647+0000 7f13de400640 1 -- 172.21.15.103:0/3126457911 >> 172.21.15.103:0/3126457911 conn(0x7f13d80d4f90 msgr2=0x7f13d80d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:41.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.647+0000 7f13de400640 1 -- 172.21.15.103:0/3126457911 shutdown_connections 2024-04-26T23:00:41.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.647+0000 7f13de400640 1 -- 172.21.15.103:0/3126457911 wait complete. 2024-04-26T23:00:41.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.648+0000 7f13de400640 1 Processor -- start 2024-04-26T23:00:41.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.648+0000 7f13de400640 1 -- start start 2024-04-26T23:00:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.649+0000 7f13de400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d80d1770 0x7f13d80cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.649+0000 7f13de400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13d80d2160 0x7f13d80cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.649+0000 7f13de400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d815f3a0 0x7f13d80cef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.649+0000 7f13de400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f13d815f900 con 0x7f13d80d1770 2024-04-26T23:00:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.649+0000 7f13de400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13d815f780 con 0x7f13d815f3a0 2024-04-26T23:00:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.649+0000 7f13de400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f13d80b3860 con 0x7f13d80d2160 2024-04-26T23:00:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.649+0000 7f13d7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d815f3a0 0x7f13d80cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.650+0000 7f13d6a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13d80d2160 0x7f13d80cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.650+0000 7f13d7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d815f3a0 0x7f13d80cef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42436/0 (socket says 172.21.15.103:42436) 2024-04-26T23:00:41.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.650+0000 7f13d6a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13d80d2160 0x7f13d80cd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:60380/0 (socket says 172.21.15.103:60380) 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.650+0000 7f13d7e00640 1 -- 172.21.15.103:0/1844059195 learned_addr learned my addr 172.21.15.103:0/1844059195 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.650+0000 7f13d7400640 1 --2- 172.21.15.103:0/1844059195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d80d1770 0x7f13d80cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.650+0000 7f13d6a00640 1 -- 172.21.15.103:0/1844059195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d815f3a0 msgr2=0x7f13d80cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.650+0000 7f13d6a00640 1 --2- 172.21.15.103:0/1844059195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d815f3a0 0x7f13d80cef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.650+0000 7f13d6a00640 1 -- 172.21.15.103:0/1844059195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d80d1770 msgr2=0x7f13d80cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.650+0000 7f13d6a00640 1 --2- 172.21.15.103:0/1844059195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d80d1770 0x7f13d80cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.650+0000 7f13d6a00640 1 -- 172.21.15.103:0/1844059195 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d80cf4d0 con 0x7f13d80d2160 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.650+0000 7f13d7400640 1 --2- 172.21.15.103:0/1844059195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d80d1770 0x7f13d80cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.650+0000 7f13d7e00640 1 --2- 172.21.15.103:0/1844059195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d815f3a0 0x7f13d80cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.650+0000 7f13d6a00640 1 --2- 172.21.15.103:0/1844059195 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13d80d2160 0x7f13d80cd770 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f13c4031000 tx=0x7f13c40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.651+0000 7f13cf400640 1 -- 172.21.15.103:0/1844059195 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4032e60 con 0x7f13d80d2160 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.651+0000 7f13cf400640 1 -- 172.21.15.103:0/1844059195 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c4030770 con 0x7f13d80d2160 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.651+0000 7f13cf400640 1 -- 172.21.15.103:0/1844059195 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4037040 con 0x7f13d80d2160 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.651+0000 7f13de400640 1 -- 172.21.15.103:0/1844059195 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d80ccc40 con 0x7f13d80d2160 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.651+0000 7f13de400640 1 -- 172.21.15.103:0/1844059195 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d80cc960 con 0x7f13d80d2160 2024-04-26T23:00:41.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.651+0000 7f13cf400640 1 -- 172.21.15.103:0/1844059195 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f13c4044020 con 0x7f13d80d2160 2024-04-26T23:00:41.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.651+0000 7f13cf400640 1 -- 172.21.15.103:0/1844059195 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f13c4046410 con 0x7f13d80d2160 2024-04-26T23:00:41.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.652+0000 7f13de400640 1 -- 172.21.15.103:0/1844059195 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d80d49d0 con 0x7f13d80d2160 2024-04-26T23:00:41.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.654+0000 7f13cf400640 1 -- 172.21.15.103:0/1844059195 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f13d80d49d0 con 0x7f13d80d2160 2024-04-26T23:00:41.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.733+0000 7f13de400640 1 -- 172.21.15.103:0/1844059195 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1", "id": 2} v 0) -- 0x7f13d80694f0 con 0x7f13d80d2160 2024-04-26T23:00:41.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.740+0000 7f13cf400640 1 -- 172.21.15.103:0/1844059195 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f13c403db70 con 0x7f13d80d2160 2024-04-26T23:00:41.740 INFO:teuthology.orchestra.run.smithi103.stdout:2 2024-04-26T23:00:41.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.741+0000 7f13de400640 1 -- 172.21.15.103:0/1844059195 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13d80d2160 msgr2=0x7f13d80cd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.741+0000 7f13de400640 1 --2- 172.21.15.103:0/1844059195 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13d80d2160 0x7f13d80cd770 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f13c4031000 tx=0x7f13c40046c0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.741+0000 7f13de400640 1 -- 172.21.15.103:0/1844059195 shutdown_connections 2024-04-26T23:00:41.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.741+0000 7f13de400640 1 --2- 172.21.15.103:0/1844059195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13d815f3a0 0x7f13d80cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.741+0000 7f13de400640 1 --2- 172.21.15.103:0/1844059195 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13d80d2160 0x7f13d80cd770 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.741+0000 7f13de400640 1 --2- 172.21.15.103:0/1844059195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d80d1770 0x7f13d80cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.741+0000 7f13de400640 1 -- 172.21.15.103:0/1844059195 >> 172.21.15.103:0/1844059195 conn(0x7f13d80d4f90 msgr2=0x7f13d80d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:41.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.741+0000 7f13de400640 1 -- 172.21.15.103:0/1844059195 shutdown_connections 2024-04-26T23:00:41.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.741+0000 7f13de400640 1 -- 172.21.15.103:0/1844059195 wait complete. 2024-04-26T23:00:41.757 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 13b77b2e-3161-450c-8aa0-557988e3ce81 3 2024-04-26T23:00:41.824 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:41.824+0000 7f251a9ce040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T23:00:41.866 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:41.866+0000 7f12fb01f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T23:00:41.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.931+0000 7f5280600640 1 Processor -- start 2024-04-26T23:00:41.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.931+0000 7f5280600640 1 -- start start 2024-04-26T23:00:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.932+0000 7f5280600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52780d06f0 0x7f52780d1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.932+0000 7f5280600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f52780b2850 0x7f52780b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.932+0000 7f5280600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52780d0b70 con 0x7f527815f3a0 2024-04-26T23:00:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.932+0000 7f5280600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f52780d0e20 con 0x7f52780b2850 2024-04-26T23:00:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.932+0000 7f5280600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f52780b1d70 con 0x7f52780d06f0 2024-04-26T23:00:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.932+0000 7f527e600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f52780b2850 0x7f52780b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.933+0000 7f527e600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f52780b2850 0x7f52780b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60102/0 (socket says 172.21.15.103:60102) 2024-04-26T23:00:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.933+0000 7f527e600640 1 -- 172.21.15.103:0/1649537670 learned_addr learned my addr 172.21.15.103:0/1649537670 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:00:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.933+0000 7f527d200640 1 --2- 172.21.15.103:0/1649537670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52780d06f0 0x7f52780d1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.933+0000 7f527e600640 1 -- 172.21.15.103:0/1649537670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52780d06f0 msgr2=0x7f52780d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:41.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.933+0000 7f527e600640 1 --2- 172.21.15.103:0/1649537670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52780d06f0 0x7f52780d1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.933+0000 7f527e600640 1 -- 172.21.15.103:0/1649537670 >> v1:172.21.15.40:6789/0 conn(0x7f527815f3a0 legacy=0x7f52780d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.933+0000 7f527e600640 1 -- 172.21.15.103:0/1649537670 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52780b2050 con 0x7f52780b2850 2024-04-26T23:00:41.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.935+0000 7f527e600640 1 --2- 172.21.15.103:0/1649537670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780b2850 0x7f52780b2cc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5270009e50 tx=0x7f527002a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d3e1fd03e3e4a03 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:41.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.935+0000 7f5277e00640 1 -- 172.21.15.103:0/1649537670 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5270036070 con 0x7f52780b2850 2024-04-26T23:00:41.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.935+0000 7f5277e00640 1 -- 172.21.15.103:0/1649537670 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5270031600 con 0x7f52780b2850 2024-04-26T23:00:41.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.935+0000 7f5277e00640 1 -- 172.21.15.103:0/1649537670 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52700319d0 con 0x7f52780b2850 2024-04-26T23:00:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.936+0000 7f5280600640 1 -- 172.21.15.103:0/1649537670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780b2850 msgr2=0x7f52780b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.936+0000 7f5280600640 1 --2- 172.21.15.103:0/1649537670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780b2850 0x7f52780b2cc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5270009e50 tx=0x7f527002a730 comp rx=0 tx=0).stop 2024-04-26T23:00:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.936+0000 7f5280600640 1 -- 172.21.15.103:0/1649537670 shutdown_connections 2024-04-26T23:00:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.936+0000 7f5280600640 1 --2- 172.21.15.103:0/1649537670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780b2850 0x7f52780b2cc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.936+0000 7f5280600640 1 --2- 172.21.15.103:0/1649537670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52780d06f0 0x7f52780d1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.936+0000 7f5280600640 1 -- 172.21.15.103:0/1649537670 >> 172.21.15.103:0/1649537670 conn(0x7f52780d4f90 msgr2=0x7f52780d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.936+0000 7f5280600640 1 -- 172.21.15.103:0/1649537670 shutdown_connections 2024-04-26T23:00:41.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.936+0000 7f5280600640 1 -- 172.21.15.103:0/1649537670 wait complete. 2024-04-26T23:00:41.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.938+0000 7f5280600640 1 Processor -- start 2024-04-26T23:00:41.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.938+0000 7f5280600640 1 -- start start 2024-04-26T23:00:41.938 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:41.938+0000 7f251a9ce040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T23:00:41.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f5280600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780b2850 0x7f5278159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f5280600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52780d06f0 0x7f52781f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f5280600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f527815f3a0 0x7f52780d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:41.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f5280600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f527815f900 con 0x7f52780d06f0 2024-04-26T23:00:41.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f5280600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f527815f780 con 0x7f52780b2850 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f5280600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f52780d23c0 con 0x7f527815f3a0 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f527dc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780b2850 0x7f5278159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f527dc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780b2850 0x7f5278159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60116/0 (socket says 172.21.15.103:60116) 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f527e600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f527815f3a0 0x7f52780d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f527dc00640 1 -- 172.21.15.103:0/219008439 learned_addr learned my addr 172.21.15.103:0/219008439 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f527d200640 1 --2- 172.21.15.103:0/219008439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52780d06f0 0x7f52781f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f527dc00640 1 -- 172.21.15.103:0/219008439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f527815f3a0 msgr2=0x7f52780d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f527dc00640 1 --2- 172.21.15.103:0/219008439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f527815f3a0 0x7f52780d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f527dc00640 1 -- 172.21.15.103:0/219008439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52780d06f0 msgr2=0x7f52781f59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f527dc00640 1 --2- 172.21.15.103:0/219008439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52780d06f0 0x7f52781f59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.939+0000 7f527dc00640 1 -- 172.21.15.103:0/219008439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52781f5ee0 con 0x7f52780b2850 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.940+0000 7f527dc00640 1 --2- 172.21.15.103:0/219008439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780b2850 0x7f5278159690 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f526c00c450 tx=0x7f526c00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.940+0000 7f5275600640 1 -- 172.21.15.103:0/219008439 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c0052d0 con 0x7f52780b2850 2024-04-26T23:00:41.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.940+0000 7f5275600640 1 -- 172.21.15.103:0/219008439 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f526c005470 con 0x7f52780b2850 2024-04-26T23:00:41.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.940+0000 7f5275600640 1 -- 172.21.15.103:0/219008439 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c012870 con 0x7f52780b2850 2024-04-26T23:00:41.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.940+0000 7f5280600640 1 -- 172.21.15.103:0/219008439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5278161c30 con 0x7f52780b2850 2024-04-26T23:00:41.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.940+0000 7f5280600640 1 -- 172.21.15.103:0/219008439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52780b7dd0 con 0x7f52780b2850 2024-04-26T23:00:41.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.940+0000 7f5275600640 1 -- 172.21.15.103:0/219008439 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f526c012a10 con 0x7f52780b2850 2024-04-26T23:00:41.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.941+0000 7f5275600640 1 -- 172.21.15.103:0/219008439 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f526c012d00 con 0x7f52780b2850 2024-04-26T23:00:41.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.942+0000 7f5280600640 1 -- 172.21.15.103:0/219008439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52780694f0 con 0x7f52780b2850 2024-04-26T23:00:41.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:41.947+0000 7f5275600640 1 -- 172.21.15.103:0/219008439 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f52781f5ee0 con 0x7f52780b2850 2024-04-26T23:00:41.973 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:41.973+0000 7f12fb01f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T23:00:41.996 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:41.997+0000 7f251a9ce040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T23:00:42.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.022+0000 7f5280600640 1 -- 172.21.15.103:0/219008439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "13b77b2e-3161-450c-8aa0-557988e3ce81", "id": 3} v 0) -- 0x7f52780b3870 con 0x7f52780b2850 2024-04-26T23:00:42.026 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:42.025+0000 7f12fb01f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T23:00:42.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.030+0000 7f5275600640 1 -- 172.21.15.103:0/219008439 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "13b77b2e-3161-450c-8aa0-557988e3ce81", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f526c0036e0 con 0x7f52780b2850 2024-04-26T23:00:42.030 INFO:teuthology.orchestra.run.smithi103.stdout:3 2024-04-26T23:00:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.032+0000 7f5280600640 1 -- 172.21.15.103:0/219008439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780b2850 msgr2=0x7f5278159690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.032+0000 7f5280600640 1 --2- 172.21.15.103:0/219008439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780b2850 0x7f5278159690 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f526c00c450 tx=0x7f526c00c910 comp rx=0 tx=0).stop 2024-04-26T23:00:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.032+0000 7f5280600640 1 -- 172.21.15.103:0/219008439 shutdown_connections 2024-04-26T23:00:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.032+0000 7f5280600640 1 --2- 172.21.15.103:0/219008439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f527815f3a0 0x7f52780d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.032+0000 7f5280600640 1 --2- 172.21.15.103:0/219008439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52780d06f0 0x7f52781f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.032+0000 7f5280600640 1 --2- 172.21.15.103:0/219008439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780b2850 0x7f5278159690 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.032+0000 7f5280600640 1 -- 172.21.15.103:0/219008439 >> 172.21.15.103:0/219008439 conn(0x7f52780d4f90 msgr2=0x7f52780b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:42.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.032+0000 7f5280600640 1 -- 172.21.15.103:0/219008439 shutdown_connections 2024-04-26T23:00:42.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.032+0000 7f5280600640 1 -- 172.21.15.103:0/219008439 wait complete. 2024-04-26T23:00:42.049 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new e1c95d9d-13fd-4263-ae34-96862877c738 4 2024-04-26T23:00:42.211 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:42.211+0000 7f251a9ce040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T23:00:42.223 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:42.223+0000 7f12fb01f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T23:00:42.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.228+0000 7f084a800640 1 Processor -- start 2024-04-26T23:00:42.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.228+0000 7f084a800640 1 -- start start 2024-04-26T23:00:42.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.229+0000 7f084a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08440b6470 0x7f08440cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.229+0000 7f084a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f084415f680 0x7f08440cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.229+0000 7f084a800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08440d5d30 con 0x7f08440d1770 2024-04-26T23:00:42.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.230+0000 7f084a800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f08440d3760 con 0x7f08440b6470 2024-04-26T23:00:42.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.230+0000 7f084a800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f08440d5a70 con 0x7f084415f680 2024-04-26T23:00:42.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.230+0000 7f0849e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f084415f680 0x7f08440cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.230+0000 7f0849e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f084415f680 0x7f08440cff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60128/0 (socket says 172.21.15.103:60128) 2024-04-26T23:00:42.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.230+0000 7f0849e00640 1 -- 172.21.15.103:0/1620204341 learned_addr learned my addr 172.21.15.103:0/1620204341 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:00:42.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.230+0000 7f0848a00640 1 --2- 172.21.15.103:0/1620204341 >> v2:172.21.15.40:3300/0 conn(0x7f08440b6470 0x7f08440cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.230+0000 7f0849e00640 1 -- 172.21.15.103:0/1620204341 >> v2:172.21.15.40:3300/0 conn(0x7f08440b6470 msgr2=0x7f08440cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:42.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.230+0000 7f0849e00640 1 --2- 172.21.15.103:0/1620204341 >> v2:172.21.15.40:3300/0 conn(0x7f08440b6470 0x7f08440cf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.230+0000 7f0849e00640 1 -- 172.21.15.103:0/1620204341 >> v1:172.21.15.40:6789/0 conn(0x7f08440d1770 legacy=0x7f084415ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:42.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.230+0000 7f0849e00640 1 -- 172.21.15.103:0/1620204341 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08440b5b00 con 0x7f084415f680 2024-04-26T23:00:42.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.231+0000 7f0849e00640 1 --2- 172.21.15.103:0/1620204341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f084415f680 0x7f08440cff20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f084408f5c0 tx=0x7f083c02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d032b393a08fc92f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:42.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.232+0000 7f0843e00640 1 -- 172.21.15.103:0/1620204341 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083c036070 con 0x7f084415f680 2024-04-26T23:00:42.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.232+0000 7f0843e00640 1 -- 172.21.15.103:0/1620204341 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f083c031600 con 0x7f084415f680 2024-04-26T23:00:42.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.232+0000 7f0843e00640 1 -- 172.21.15.103:0/1620204341 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083c0319d0 con 0x7f084415f680 2024-04-26T23:00:42.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.232+0000 7f084a800640 1 -- 172.21.15.103:0/1620204341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f084415f680 msgr2=0x7f08440cff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.232+0000 7f084a800640 1 --2- 172.21.15.103:0/1620204341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f084415f680 0x7f08440cff20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f084408f5c0 tx=0x7f083c02a6f0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.233+0000 7f084a800640 1 -- 172.21.15.103:0/1620204341 shutdown_connections 2024-04-26T23:00:42.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.233+0000 7f084a800640 1 --2- 172.21.15.103:0/1620204341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f084415f680 0x7f08440cff20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.233+0000 7f084a800640 1 --2- 172.21.15.103:0/1620204341 >> v2:172.21.15.40:3300/0 conn(0x7f08440b6470 0x7f08440cf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.233+0000 7f084a800640 1 -- 172.21.15.103:0/1620204341 >> 172.21.15.103:0/1620204341 conn(0x7f08440d4dd0 msgr2=0x7f08440d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:42.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.233+0000 7f084a800640 1 -- 172.21.15.103:0/1620204341 shutdown_connections 2024-04-26T23:00:42.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.233+0000 7f084a800640 1 -- 172.21.15.103:0/1620204341 wait complete. 2024-04-26T23:00:42.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.234+0000 7f084a800640 1 Processor -- start 2024-04-26T23:00:42.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.234+0000 7f084a800640 1 -- start start 2024-04-26T23:00:42.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.235+0000 7f084a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08440b6470 0x7f08440b8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.235+0000 7f084a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08440d1770 0x7f08440ba100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.235+0000 7f084a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f084415f680 0x7f08440ce570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.235+0000 7f084a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f08440d3d70 con 0x7f08440d1770 2024-04-26T23:00:42.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.235+0000 7f084a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f08440d3bf0 con 0x7f084415f680 2024-04-26T23:00:42.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.235+0000 7f084a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f08440d1b50 con 0x7f08440b6470 2024-04-26T23:00:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.236+0000 7f0849400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08440b6470 0x7f08440b8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.236+0000 7f0849400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08440b6470 0x7f08440b8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:33662/0 (socket says 172.21.15.103:33662) 2024-04-26T23:00:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.236+0000 7f0849e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f084415f680 0x7f08440ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.236+0000 7f0849400640 1 -- 172.21.15.103:0/1699600038 learned_addr learned my addr 172.21.15.103:0/1699600038 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:00:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.236+0000 7f0848a00640 1 --2- 172.21.15.103:0/1699600038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08440d1770 0x7f08440ba100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.236+0000 7f0849e00640 1 -- 172.21.15.103:0/1699600038 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08440b6470 msgr2=0x7f08440b8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.236+0000 7f0849e00640 1 --2- 172.21.15.103:0/1699600038 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08440b6470 0x7f08440b8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.236+0000 7f0849e00640 1 -- 172.21.15.103:0/1699600038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08440d1770 msgr2=0x7f08440ba100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.236+0000 7f0849e00640 1 --2- 172.21.15.103:0/1699600038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08440d1770 0x7f08440ba100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.236+0000 7f0849e00640 1 -- 172.21.15.103:0/1699600038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08440b7d00 con 0x7f084415f680 2024-04-26T23:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.236+0000 7f0849400640 1 --2- 172.21.15.103:0/1699600038 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08440b6470 0x7f08440b8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.237+0000 7f0849e00640 1 --2- 172.21.15.103:0/1699600038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f084415f680 0x7f08440ce570 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f083c030000 tx=0x7f083c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.237+0000 7f0841600640 1 -- 172.21.15.103:0/1699600038 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083c036050 con 0x7f084415f680 2024-04-26T23:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.237+0000 7f0841600640 1 -- 172.21.15.103:0/1699600038 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f083c042070 con 0x7f084415f680 2024-04-26T23:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.237+0000 7f084a800640 1 -- 172.21.15.103:0/1699600038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08440b7fe0 con 0x7f084415f680 2024-04-26T23:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.237+0000 7f0841600640 1 -- 172.21.15.103:0/1699600038 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083c009510 con 0x7f084415f680 2024-04-26T23:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.237+0000 7f084a800640 1 -- 172.21.15.103:0/1699600038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08440b85a0 con 0x7f084415f680 2024-04-26T23:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.237+0000 7f0841600640 1 -- 172.21.15.103:0/1699600038 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f083c007bd0 con 0x7f084415f680 2024-04-26T23:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.237+0000 7f0841600640 1 -- 172.21.15.103:0/1699600038 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f083c047b40 con 0x7f084415f680 2024-04-26T23:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.238+0000 7f084a800640 1 -- 172.21.15.103:0/1699600038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f084408ee40 con 0x7f084415f680 2024-04-26T23:00:42.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.240+0000 7f0841600640 1 -- 172.21.15.103:0/1699600038 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f084408ee40 con 0x7f084415f680 2024-04-26T23:00:42.277 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:42.277+0000 7f251a9ce040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T23:00:42.287 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:42.287+0000 7f12fb01f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T23:00:42.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.320+0000 7f084a800640 1 -- 172.21.15.103:0/1699600038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e1c95d9d-13fd-4263-ae34-96862877c738", "id": 4} v 0) -- 0x7f084415f430 con 0x7f084415f680 2024-04-26T23:00:42.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.329+0000 7f0841600640 1 -- 172.21.15.103:0/1699600038 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e1c95d9d-13fd-4263-ae34-96862877c738", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f083c047460 con 0x7f084415f680 2024-04-26T23:00:42.330 INFO:teuthology.orchestra.run.smithi103.stdout:4 2024-04-26T23:00:42.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.331+0000 7f084a800640 1 -- 172.21.15.103:0/1699600038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f084415f680 msgr2=0x7f08440ce570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.331+0000 7f084a800640 1 --2- 172.21.15.103:0/1699600038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f084415f680 0x7f08440ce570 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f083c030000 tx=0x7f083c004a40 comp rx=0 tx=0).stop 2024-04-26T23:00:42.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.331+0000 7f084a800640 1 -- 172.21.15.103:0/1699600038 shutdown_connections 2024-04-26T23:00:42.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.331+0000 7f084a800640 1 --2- 172.21.15.103:0/1699600038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f084415f680 0x7f08440ce570 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.331+0000 7f084a800640 1 --2- 172.21.15.103:0/1699600038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08440d1770 0x7f08440ba100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.331+0000 7f084a800640 1 --2- 172.21.15.103:0/1699600038 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08440b6470 0x7f08440b8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.331+0000 7f084a800640 1 -- 172.21.15.103:0/1699600038 >> 172.21.15.103:0/1699600038 conn(0x7f08440d4dd0 msgr2=0x7f084415e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:42.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.331+0000 7f084a800640 1 -- 172.21.15.103:0/1699600038 shutdown_connections 2024-04-26T23:00:42.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.331+0000 7f084a800640 1 -- 172.21.15.103:0/1699600038 wait complete. 2024-04-26T23:00:42.341 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:42.341+0000 7f251a9ce040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T23:00:42.347 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 222ecc63-1194-48ec-abc2-d1f1272c7ff8 5 2024-04-26T23:00:42.350 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:42.350+0000 7f12fb01f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T23:00:42.478 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:42.478+0000 7f251a9ce040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:00:42.483 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:42.483+0000 7f12fb01f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:00:42.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.528+0000 7f108b200640 1 Processor -- start 2024-04-26T23:00:42.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.529+0000 7f108b200640 1 -- start start 2024-04-26T23:00:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.530+0000 7f108b200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f10840d1770 0x7f108415ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.530+0000 7f108b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f10840d1b50 0x7f108415f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.530+0000 7f108b200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10840d5d30 con 0x7f10840b6470 2024-04-26T23:00:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.530+0000 7f108b200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f10840d3760 con 0x7f10840d1770 2024-04-26T23:00:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.530+0000 7f108b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f10840d09f0 con 0x7f10840d1b50 2024-04-26T23:00:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.531+0000 7f108a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f10840d1b50 0x7f108415f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.531+0000 7f108a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f10840d1b50 0x7f108415f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60150/0 (socket says 172.21.15.103:60150) 2024-04-26T23:00:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.531+0000 7f108a800640 1 -- 172.21.15.103:0/2879335221 learned_addr learned my addr 172.21.15.103:0/2879335221 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:00:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.531+0000 7f1089e00640 1 --2- 172.21.15.103:0/2879335221 >> v2:172.21.15.40:3300/0 conn(0x7f10840d1770 0x7f108415ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.531+0000 7f108a800640 1 -- 172.21.15.103:0/2879335221 >> v2:172.21.15.40:3300/0 conn(0x7f10840d1770 msgr2=0x7f108415ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.531+0000 7f108a800640 1 --2- 172.21.15.103:0/2879335221 >> v2:172.21.15.40:3300/0 conn(0x7f10840d1770 0x7f108415ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.531+0000 7f108a800640 1 -- 172.21.15.103:0/2879335221 >> v1:172.21.15.40:6789/0 conn(0x7f10840b6470 legacy=0x7f10840b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:42.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.531+0000 7f108a800640 1 -- 172.21.15.103:0/2879335221 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10840d5a70 con 0x7f10840d1b50 2024-04-26T23:00:42.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.532+0000 7f108a800640 1 --2- 172.21.15.103:0/2879335221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10840d1b50 0x7f108415f4b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f108408f5c0 tx=0x7f108002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=962a779b237e0e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:42.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.532+0000 7f1088a00640 1 -- 172.21.15.103:0/2879335221 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080036070 con 0x7f10840d1b50 2024-04-26T23:00:42.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.532+0000 7f1088a00640 1 -- 172.21.15.103:0/2879335221 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1080031600 con 0x7f10840d1b50 2024-04-26T23:00:42.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.532+0000 7f1088a00640 1 -- 172.21.15.103:0/2879335221 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10800319d0 con 0x7f10840d1b50 2024-04-26T23:00:42.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.533+0000 7f108b200640 1 -- 172.21.15.103:0/2879335221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10840d1b50 msgr2=0x7f108415f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.533+0000 7f108b200640 1 --2- 172.21.15.103:0/2879335221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10840d1b50 0x7f108415f4b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f108408f5c0 tx=0x7f108002a660 comp rx=0 tx=0).stop 2024-04-26T23:00:42.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.533+0000 7f108b200640 1 -- 172.21.15.103:0/2879335221 shutdown_connections 2024-04-26T23:00:42.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.533+0000 7f108b200640 1 --2- 172.21.15.103:0/2879335221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10840d1b50 0x7f108415f4b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.533+0000 7f108b200640 1 --2- 172.21.15.103:0/2879335221 >> v2:172.21.15.40:3300/0 conn(0x7f10840d1770 0x7f108415ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.533+0000 7f108b200640 1 -- 172.21.15.103:0/2879335221 >> 172.21.15.103:0/2879335221 conn(0x7f10840d4dd0 msgr2=0x7f10840d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:42.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.533+0000 7f108b200640 1 -- 172.21.15.103:0/2879335221 shutdown_connections 2024-04-26T23:00:42.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.533+0000 7f108b200640 1 -- 172.21.15.103:0/2879335221 wait complete. 2024-04-26T23:00:42.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.535+0000 7f108b200640 1 Processor -- start 2024-04-26T23:00:42.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.535+0000 7f108b200640 1 -- start start 2024-04-26T23:00:42.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.536+0000 7f108b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10840b6470 0x7f10840ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.536+0000 7f108b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10840d1770 0x7f1084158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.536+0000 7f108b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f10840d1b50 0x7f1084158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.536+0000 7f108b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f10840d3bf0 con 0x7f10840d1770 2024-04-26T23:00:42.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.536+0000 7f108b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f10840d2670 con 0x7f10840b6470 2024-04-26T23:00:42.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.536+0000 7f108b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f10840d3d70 con 0x7f10840d1b50 2024-04-26T23:00:42.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.536+0000 7f1089e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10840b6470 0x7f10840ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.536+0000 7f1089e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10840b6470 0x7f10840ce2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60154/0 (socket says 172.21.15.103:60154) 2024-04-26T23:00:42.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.536+0000 7f1089e00640 1 -- 172.21.15.103:0/169918163 learned_addr learned my addr 172.21.15.103:0/169918163 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.536+0000 7f108a800640 1 --2- 172.21.15.103:0/169918163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f10840d1b50 0x7f1084158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.537+0000 7f1089400640 1 --2- 172.21.15.103:0/169918163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10840d1770 0x7f1084158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.537+0000 7f1089e00640 1 -- 172.21.15.103:0/169918163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f10840d1b50 msgr2=0x7f1084158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.537+0000 7f1089e00640 1 --2- 172.21.15.103:0/169918163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f10840d1b50 0x7f1084158760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.537+0000 7f1089e00640 1 -- 172.21.15.103:0/169918163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10840d1770 msgr2=0x7f1084158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.537+0000 7f1089e00640 1 --2- 172.21.15.103:0/169918163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10840d1770 0x7f1084158220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.537+0000 7f1089e00640 1 -- 172.21.15.103:0/169918163 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10840ce820 con 0x7f10840b6470 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.537+0000 7f1089e00640 1 --2- 172.21.15.103:0/169918163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10840b6470 0x7f10840ce2e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f107800d5f0 tx=0x7f107800dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.537+0000 7f107e000640 1 -- 172.21.15.103:0/169918163 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10780125f0 con 0x7f10840b6470 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.537+0000 7f107e000640 1 -- 172.21.15.103:0/169918163 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1078019070 con 0x7f10840b6470 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.537+0000 7f108b200640 1 -- 172.21.15.103:0/169918163 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10840cd240 con 0x7f10840b6470 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.537+0000 7f107e000640 1 -- 172.21.15.103:0/169918163 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10780129d0 con 0x7f10840b6470 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.538+0000 7f108b200640 1 -- 172.21.15.103:0/169918163 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10840cd800 con 0x7f10840b6470 2024-04-26T23:00:42.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.538+0000 7f107e000640 1 -- 172.21.15.103:0/169918163 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1078012b70 con 0x7f10840b6470 2024-04-26T23:00:42.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.538+0000 7f107e000640 1 -- 172.21.15.103:0/169918163 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f107801e9e0 con 0x7f10840b6470 2024-04-26T23:00:42.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.539+0000 7f108b200640 1 -- 172.21.15.103:0/169918163 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1084069370 con 0x7f10840b6470 2024-04-26T23:00:42.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.541+0000 7f107e000640 1 -- 172.21.15.103:0/169918163 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f107801e460 con 0x7f10840b6470 2024-04-26T23:00:42.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.619+0000 7f108b200640 1 -- 172.21.15.103:0/169918163 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "222ecc63-1194-48ec-abc2-d1f1272c7ff8", "id": 5} v 0) -- 0x7f10840b6230 con 0x7f10840b6470 2024-04-26T23:00:42.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.630+0000 7f107e000640 1 -- 172.21.15.103:0/169918163 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "222ecc63-1194-48ec-abc2-d1f1272c7ff8", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f107801e640 con 0x7f10840b6470 2024-04-26T23:00:42.630 INFO:teuthology.orchestra.run.smithi103.stdout:5 2024-04-26T23:00:42.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.631+0000 7f108b200640 1 -- 172.21.15.103:0/169918163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10840b6470 msgr2=0x7f10840ce2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.631+0000 7f108b200640 1 --2- 172.21.15.103:0/169918163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10840b6470 0x7f10840ce2e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f107800d5f0 tx=0x7f107800dab0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.631+0000 7f108b200640 1 -- 172.21.15.103:0/169918163 shutdown_connections 2024-04-26T23:00:42.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.631+0000 7f108b200640 1 --2- 172.21.15.103:0/169918163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f10840d1b50 0x7f1084158760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.631+0000 7f108b200640 1 --2- 172.21.15.103:0/169918163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10840d1770 0x7f1084158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.631+0000 7f108b200640 1 --2- 172.21.15.103:0/169918163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10840b6470 0x7f10840ce2e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.631+0000 7f108b200640 1 -- 172.21.15.103:0/169918163 >> 172.21.15.103:0/169918163 conn(0x7f10840d4dd0 msgr2=0x7f10840d51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:42.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.631+0000 7f108b200640 1 -- 172.21.15.103:0/169918163 shutdown_connections 2024-04-26T23:00:42.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.632+0000 7f108b200640 1 -- 172.21.15.103:0/169918163 wait complete. 2024-04-26T23:00:42.647 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 4710c272-359f-4fb7-a4bc-0fc444c29a12 6 2024-04-26T23:00:42.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.814+0000 7f4336800640 1 Processor -- start 2024-04-26T23:00:42.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.814+0000 7f4336800640 1 -- start start 2024-04-26T23:00:42.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f4336800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43300d06f0 0x7f43300d1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f4336800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f43300b2850 0x7f43300b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f4336800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43300d0b70 con 0x7f433015f3a0 2024-04-26T23:00:42.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f4336800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f43300d0e20 con 0x7f43300b2850 2024-04-26T23:00:42.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f4336800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f43300b1d70 con 0x7f43300d06f0 2024-04-26T23:00:42.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f432f400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f433015f3a0 0x7f43300d1770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:51726/0 (socket says 172.21.15.103:51726) 2024-04-26T23:00:42.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f432ea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43300d06f0 0x7f43300d1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f432f400640 1 -- 172.21.15.103:0/1485191378 learned_addr learned my addr 172.21.15.103:0/1485191378 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-04-26T23:00:42.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f432fe00640 1 --2- 172.21.15.103:0/1485191378 >> v2:172.21.15.40:3300/0 conn(0x7f43300b2850 0x7f43300b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f432ea00640 1 -- 172.21.15.103:0/1485191378 >> v2:172.21.15.40:3300/0 conn(0x7f43300b2850 msgr2=0x7f43300b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:42.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f432ea00640 1 --2- 172.21.15.103:0/1485191378 >> v2:172.21.15.40:3300/0 conn(0x7f43300b2850 0x7f43300b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f432ea00640 1 -- 172.21.15.103:0/1485191378 >> v1:172.21.15.103:6789/0 conn(0x7f433015f3a0 legacy=0x7f43300d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f432ea00640 1 -- 172.21.15.103:0/1485191378 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43300b2050 con 0x7f43300d06f0 2024-04-26T23:00:42.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.816+0000 7f432e000640 1 -- 172.21.15.103:0/1485191378 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2299210655 0 0) 0x7f43300d0b70 con 0x7f433015f3a0 2024-04-26T23:00:42.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.817+0000 7f432ea00640 1 --2- 172.21.15.103:0/1485191378 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43300d06f0 0x7f43300d1e80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f431c0097a0 tx=0x7f431c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11b2e0c3a6bb641 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:42.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.818+0000 7f432e000640 1 -- 172.21.15.103:0/1485191378 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431c037070 con 0x7f43300d06f0 2024-04-26T23:00:42.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.818+0000 7f432e000640 1 -- 172.21.15.103:0/1485191378 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f431c032440 con 0x7f43300d06f0 2024-04-26T23:00:42.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.818+0000 7f432e000640 1 -- 172.21.15.103:0/1485191378 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431c032810 con 0x7f43300d06f0 2024-04-26T23:00:42.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.818+0000 7f4336800640 1 -- 172.21.15.103:0/1485191378 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43300d06f0 msgr2=0x7f43300d1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.818+0000 7f4336800640 1 --2- 172.21.15.103:0/1485191378 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43300d06f0 0x7f43300d1e80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f431c0097a0 tx=0x7f431c02a180 comp rx=0 tx=0).stop 2024-04-26T23:00:42.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.819+0000 7f4336800640 1 -- 172.21.15.103:0/1485191378 shutdown_connections 2024-04-26T23:00:42.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.819+0000 7f4336800640 1 --2- 172.21.15.103:0/1485191378 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43300d06f0 0x7f43300d1e80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.819+0000 7f4336800640 1 --2- 172.21.15.103:0/1485191378 >> v2:172.21.15.40:3300/0 conn(0x7f43300b2850 0x7f43300b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.819+0000 7f4336800640 1 -- 172.21.15.103:0/1485191378 >> 172.21.15.103:0/1485191378 conn(0x7f43300d4f90 msgr2=0x7f43300d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:42.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.819+0000 7f4336800640 1 -- 172.21.15.103:0/1485191378 shutdown_connections 2024-04-26T23:00:42.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.819+0000 7f4336800640 1 -- 172.21.15.103:0/1485191378 wait complete. 2024-04-26T23:00:42.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.820+0000 7f4336800640 1 Processor -- start 2024-04-26T23:00:42.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.820+0000 7f4336800640 1 -- start start 2024-04-26T23:00:42.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.821+0000 7f4336800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43300b2850 0x7f43300ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.821+0000 7f4336800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43300d06f0 0x7f4330159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.821+0000 7f4336800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f433015f3a0 0x7f43300c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:42.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.821+0000 7f4336800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f433015f900 con 0x7f43300d06f0 2024-04-26T23:00:42.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.821+0000 7f4336800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f433015f780 con 0x7f43300b2850 2024-04-26T23:00:42.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.821+0000 7f4336800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f43300d23c0 con 0x7f433015f3a0 2024-04-26T23:00:42.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.822+0000 7f432f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43300b2850 0x7f43300ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.822+0000 7f432f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43300b2850 0x7f43300ca400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60166/0 (socket says 172.21.15.103:60166) 2024-04-26T23:00:42.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.822+0000 7f432f400640 1 -- 172.21.15.103:0/2397667373 learned_addr learned my addr 172.21.15.103:0/2397667373 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:00:42.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.822+0000 7f432fe00640 1 --2- 172.21.15.103:0/2397667373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f433015f3a0 0x7f43300c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.822+0000 7f432ea00640 1 --2- 172.21.15.103:0/2397667373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43300d06f0 0x7f4330159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.822+0000 7f432f400640 1 -- 172.21.15.103:0/2397667373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f433015f3a0 msgr2=0x7f43300c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.822+0000 7f432f400640 1 --2- 172.21.15.103:0/2397667373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f433015f3a0 0x7f43300c94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.822+0000 7f432f400640 1 -- 172.21.15.103:0/2397667373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43300d06f0 msgr2=0x7f4330159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.822+0000 7f432f400640 1 --2- 172.21.15.103:0/2397667373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43300d06f0 0x7f4330159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.822+0000 7f432f400640 1 -- 172.21.15.103:0/2397667373 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43300cc100 con 0x7f43300b2850 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.822+0000 7f432f400640 1 --2- 172.21.15.103:0/2397667373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43300b2850 0x7f43300ca400 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f432000c500 tx=0x7f432000c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.823+0000 7f4327400640 1 -- 172.21.15.103:0/2397667373 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320018070 con 0x7f43300b2850 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.823+0000 7f4327400640 1 -- 172.21.15.103:0/2397667373 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4320007580 con 0x7f43300b2850 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.823+0000 7f4327400640 1 -- 172.21.15.103:0/2397667373 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320004a20 con 0x7f43300b2850 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.823+0000 7f4336800640 1 -- 172.21.15.103:0/2397667373 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43300cc3e0 con 0x7f43300b2850 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.823+0000 7f4336800640 1 -- 172.21.15.103:0/2397667373 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43300cc9a0 con 0x7f43300b2850 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.823+0000 7f4327400640 1 -- 172.21.15.103:0/2397667373 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4320004bc0 con 0x7f43300b2850 2024-04-26T23:00:42.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.823+0000 7f4327400640 1 -- 172.21.15.103:0/2397667373 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f43200129f0 con 0x7f43300b2850 2024-04-26T23:00:42.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.824+0000 7f4336800640 1 -- 172.21.15.103:0/2397667373 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43300694f0 con 0x7f43300b2850 2024-04-26T23:00:42.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.826+0000 7f4327400640 1 -- 172.21.15.103:0/2397667373 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f4320012400 con 0x7f43300b2850 2024-04-26T23:00:42.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.901+0000 7f4336800640 1 -- 172.21.15.103:0/2397667373 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4710c272-359f-4fb7-a4bc-0fc444c29a12", "id": 6} v 0) -- 0x7f43300b4280 con 0x7f43300b2850 2024-04-26T23:00:42.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.911+0000 7f4327400640 1 -- 172.21.15.103:0/2397667373 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4710c272-359f-4fb7-a4bc-0fc444c29a12", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f43200125e0 con 0x7f43300b2850 2024-04-26T23:00:42.911 INFO:teuthology.orchestra.run.smithi103.stdout:6 2024-04-26T23:00:42.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.912+0000 7f4336800640 1 -- 172.21.15.103:0/2397667373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43300b2850 msgr2=0x7f43300ca400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:42.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.912+0000 7f4336800640 1 --2- 172.21.15.103:0/2397667373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43300b2850 0x7f43300ca400 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f432000c500 tx=0x7f432000c9c0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.912+0000 7f4336800640 1 -- 172.21.15.103:0/2397667373 shutdown_connections 2024-04-26T23:00:42.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.912+0000 7f4336800640 1 --2- 172.21.15.103:0/2397667373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f433015f3a0 0x7f43300c94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.912+0000 7f4336800640 1 --2- 172.21.15.103:0/2397667373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43300d06f0 0x7f4330159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.912+0000 7f4336800640 1 --2- 172.21.15.103:0/2397667373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43300b2850 0x7f43300ca400 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:42.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.912+0000 7f4336800640 1 -- 172.21.15.103:0/2397667373 >> 172.21.15.103:0/2397667373 conn(0x7f43300d4f90 msgr2=0x7f43300d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:42.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.912+0000 7f4336800640 1 -- 172.21.15.103:0/2397667373 shutdown_connections 2024-04-26T23:00:42.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:42.912+0000 7f4336800640 1 -- 172.21.15.103:0/2397667373 wait complete. 2024-04-26T23:00:42.928 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 6e81e8c8-355c-4411-89ef-c9f95e5dc00d 7 2024-04-26T23:00:42.958 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:42.958+0000 7f12fb01f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T23:00:42.961 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:42.961+0000 7f251a9ce040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T23:00:43.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.109+0000 7f8b16000640 1 Processor -- start 2024-04-26T23:00:43.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.109+0000 7f8b16000640 1 -- start start 2024-04-26T23:00:43.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.110+0000 7f8b16000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b1015f060 0x7f8b1015f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:43.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.111+0000 7f8b16000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b100d5c20 con 0x7f8b100d1f70 2024-04-26T23:00:43.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.111+0000 7f8b16000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b100cfc40 con 0x7f8b100d3640 2024-04-26T23:00:43.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.111+0000 7f8b16000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8b100c99b0 con 0x7f8b1015f060 2024-04-26T23:00:43.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.111+0000 7f8b14c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b1015f060 0x7f8b1015f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:43.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.111+0000 7f8b14c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b1015f060 0x7f8b1015f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:33712/0 (socket says 172.21.15.103:33712) 2024-04-26T23:00:43.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.111+0000 7f8b14c00640 1 -- 172.21.15.103:0/1389486434 learned_addr learned my addr 172.21.15.103:0/1389486434 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:00:43.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.111+0000 7f8b14c00640 1 -- 172.21.15.103:0/1389486434 >> v1:172.21.15.103:6789/0 conn(0x7f8b100d3640 legacy=0x7f8b100d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:43.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.111+0000 7f8b14c00640 1 -- 172.21.15.103:0/1389486434 >> v1:172.21.15.40:6789/0 conn(0x7f8b100d1f70 legacy=0x7f8b100cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:43.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.111+0000 7f8b14c00640 1 -- 172.21.15.103:0/1389486434 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b100cff20 con 0x7f8b1015f060 2024-04-26T23:00:43.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.112+0000 7f8b14c00640 1 --2- 172.21.15.103:0/1389486434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b1015f060 0x7f8b1015f440 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8b040098a0 tx=0x7f8b0402a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=757e276584a2149a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:43.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.113+0000 7f8b0f400640 1 -- 172.21.15.103:0/1389486434 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b04037070 con 0x7f8b1015f060 2024-04-26T23:00:43.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.113+0000 7f8b0f400640 1 -- 172.21.15.103:0/1389486434 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b04032440 con 0x7f8b1015f060 2024-04-26T23:00:43.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.113+0000 7f8b0f400640 1 -- 172.21.15.103:0/1389486434 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b04032810 con 0x7f8b1015f060 2024-04-26T23:00:43.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.113+0000 7f8b16000640 1 -- 172.21.15.103:0/1389486434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b1015f060 msgr2=0x7f8b1015f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:43.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.113+0000 7f8b16000640 1 --2- 172.21.15.103:0/1389486434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b1015f060 0x7f8b1015f440 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8b040098a0 tx=0x7f8b0402a290 comp rx=0 tx=0).stop 2024-04-26T23:00:43.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.113+0000 7f8b16000640 1 -- 172.21.15.103:0/1389486434 shutdown_connections 2024-04-26T23:00:43.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.113+0000 7f8b16000640 1 --2- 172.21.15.103:0/1389486434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b1015f060 0x7f8b1015f440 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.113+0000 7f8b16000640 1 -- 172.21.15.103:0/1389486434 >> 172.21.15.103:0/1389486434 conn(0x7f8b100d4e50 msgr2=0x7f8b100d5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:43.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.114+0000 7f8b16000640 1 -- 172.21.15.103:0/1389486434 shutdown_connections 2024-04-26T23:00:43.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.114+0000 7f8b16000640 1 -- 172.21.15.103:0/1389486434 wait complete. 2024-04-26T23:00:43.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.116+0000 7f8b16000640 1 Processor -- start 2024-04-26T23:00:43.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.116+0000 7f8b16000640 1 -- start start 2024-04-26T23:00:43.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.118+0000 7f8b16000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b100d1f70 0x7f8b100ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:43.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.118+0000 7f8b16000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b100d3640 0x7f8b10159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:43.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.118+0000 7f8b16000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b1015f060 0x7f8b101f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:43.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.118+0000 7f8b16000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8b100d0ce0 con 0x7f8b100d1f70 2024-04-26T23:00:43.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.118+0000 7f8b16000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8b100d04e0 con 0x7f8b1015f060 2024-04-26T23:00:43.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.118+0000 7f8b16000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8b100ceed0 con 0x7f8b100d3640 2024-04-26T23:00:43.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.118+0000 7f8b0fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b100d3640 0x7f8b10159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:43.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.118+0000 7f8b15600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b1015f060 0x7f8b101f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:43.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.118+0000 7f8b0fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b100d3640 0x7f8b10159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:33724/0 (socket says 172.21.15.103:33724) 2024-04-26T23:00:43.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.118+0000 7f8b0fe00640 1 -- 172.21.15.103:0/2822046793 learned_addr learned my addr 172.21.15.103:0/2822046793 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:00:43.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.118+0000 7f8b14c00640 1 --2- 172.21.15.103:0/2822046793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b100d1f70 0x7f8b100ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:43.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.119+0000 7f8b0fe00640 1 -- 172.21.15.103:0/2822046793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b1015f060 msgr2=0x7f8b101f5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:43.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.119+0000 7f8b0fe00640 1 --2- 172.21.15.103:0/2822046793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b1015f060 0x7f8b101f5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.119+0000 7f8b0fe00640 1 -- 172.21.15.103:0/2822046793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b100d1f70 msgr2=0x7f8b100ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:43.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.119+0000 7f8b0fe00640 1 --2- 172.21.15.103:0/2822046793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b100d1f70 0x7f8b100ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.119+0000 7f8b0fe00640 1 -- 172.21.15.103:0/2822046793 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b101f4ce0 con 0x7f8b100d3640 2024-04-26T23:00:43.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.119+0000 7f8b0fe00640 1 --2- 172.21.15.103:0/2822046793 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b100d3640 0x7f8b10159a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00c400 tx=0x7f8afc00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:43.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.119+0000 7f8b0cc00640 1 -- 172.21.15.103:0/2822046793 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc0052d0 con 0x7f8b100d3640 2024-04-26T23:00:43.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.119+0000 7f8b0cc00640 1 -- 172.21.15.103:0/2822046793 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8afc005490 con 0x7f8b100d3640 2024-04-26T23:00:43.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.119+0000 7f8b16000640 1 -- 172.21.15.103:0/2822046793 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b10157f70 con 0x7f8b100d3640 2024-04-26T23:00:43.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.119+0000 7f8b0cc00640 1 -- 172.21.15.103:0/2822046793 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc007260 con 0x7f8b100d3640 2024-04-26T23:00:43.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.119+0000 7f8b16000640 1 -- 172.21.15.103:0/2822046793 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b10158810 con 0x7f8b100d3640 2024-04-26T23:00:43.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.120+0000 7f8b0cc00640 1 -- 172.21.15.103:0/2822046793 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8afc007400 con 0x7f8b100d3640 2024-04-26T23:00:43.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.120+0000 7f8b0cc00640 1 -- 172.21.15.103:0/2822046793 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f8afc0035f0 con 0x7f8b100d3640 2024-04-26T23:00:43.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.120+0000 7f8b16000640 1 -- 172.21.15.103:0/2822046793 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b101f4eb0 con 0x7f8b100d3640 2024-04-26T23:00:43.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.123+0000 7f8b0cc00640 1 -- 172.21.15.103:0/2822046793 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f8b101f4eb0 con 0x7f8b100d3640 2024-04-26T23:00:43.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.197+0000 7f8b16000640 1 -- 172.21.15.103:0/2822046793 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6e81e8c8-355c-4411-89ef-c9f95e5dc00d", "id": 7} v 0) -- 0x7f8b10069a80 con 0x7f8b100d3640 2024-04-26T23:00:43.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.204+0000 7f8b0cc00640 1 -- 172.21.15.103:0/2822046793 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6e81e8c8-355c-4411-89ef-c9f95e5dc00d", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f8afc003a50 con 0x7f8b100d3640 2024-04-26T23:00:43.204 INFO:teuthology.orchestra.run.smithi103.stdout:7 2024-04-26T23:00:43.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.206+0000 7f8b16000640 1 -- 172.21.15.103:0/2822046793 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b100d3640 msgr2=0x7f8b10159a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:43.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.206+0000 7f8b16000640 1 --2- 172.21.15.103:0/2822046793 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b100d3640 0x7f8b10159a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00c400 tx=0x7f8afc00c8c0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.206+0000 7f8b16000640 1 -- 172.21.15.103:0/2822046793 shutdown_connections 2024-04-26T23:00:43.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.206+0000 7f8b16000640 1 --2- 172.21.15.103:0/2822046793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b1015f060 0x7f8b101f5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.206+0000 7f8b16000640 1 --2- 172.21.15.103:0/2822046793 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8b100d3640 0x7f8b10159a30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.206+0000 7f8b16000640 1 --2- 172.21.15.103:0/2822046793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b100d1f70 0x7f8b100ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.206+0000 7f8b16000640 1 -- 172.21.15.103:0/2822046793 >> 172.21.15.103:0/2822046793 conn(0x7f8b100d4e50 msgr2=0x7f8b1015b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:43.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.206+0000 7f8b16000640 1 -- 172.21.15.103:0/2822046793 shutdown_connections 2024-04-26T23:00:43.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:00:43.206+0000 7f8b16000640 1 -- 172.21.15.103:0/2822046793 wait complete. 2024-04-26T23:00:43.222 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-26T23:00:43.223 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-26T23:00:43.230 INFO:tasks.ceph.osd.0:Started 2024-04-26T23:00:43.230 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-26T23:00:43.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-26T23:00:43.240 INFO:tasks.ceph.osd.1:Started 2024-04-26T23:00:43.240 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-26T23:00:43.240 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-26T23:00:43.245 INFO:tasks.ceph.osd.2:Started 2024-04-26T23:00:43.246 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-26T23:00:43.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-26T23:00:43.251 INFO:tasks.ceph.osd.3:Started 2024-04-26T23:00:43.251 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-26T23:00:43.251 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-26T23:00:43.270 INFO:tasks.ceph.osd.4:Started 2024-04-26T23:00:43.270 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-26T23:00:43.270 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-26T23:00:43.275 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:43.275+0000 7f12fb01f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T23:00:43.277 INFO:tasks.ceph.osd.5:Started 2024-04-26T23:00:43.277 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-26T23:00:43.277 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-26T23:00:43.278 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:43.274+0000 7f251a9ce040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T23:00:43.284 INFO:tasks.ceph.osd.6:Started 2024-04-26T23:00:43.284 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-26T23:00:43.285 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-26T23:00:43.290 INFO:tasks.ceph.osd.7:Started 2024-04-26T23:00:43.291 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:00:43.345 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:43.345+0000 7f251a9ce040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T23:00:43.354 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:43.355+0000 7f12fb01f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T23:00:43.496 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:43.496+0000 7f251a9ce040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T23:00:43.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.497+0000 7fdde9600640 1 Processor -- start 2024-04-26T23:00:43.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.497+0000 7fdde9600640 1 -- start start 2024-04-26T23:00:43.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.498+0000 7fdde9600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdde40b2340 0x7fdde40cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:43.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.498+0000 7fdde9600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdde40cd930 0x7fdde40d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:43.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.498+0000 7fdde9600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdde40d10d0 con 0x7fdde4162810 2024-04-26T23:00:43.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.498+0000 7fdde9600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fdde40d0750 con 0x7fdde40b2340 2024-04-26T23:00:43.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.498+0000 7fdde9600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdde40d16a0 con 0x7fdde40cd930 2024-04-26T23:00:43.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.499+0000 7fdde2000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdde40b2340 0x7fdde40cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:43.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.499+0000 7fdde3400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdde40cd930 0x7fdde40d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:43.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.499+0000 7fdde2a00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fdde4162810 0x7fdde4070350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:43258/0 (socket says 172.21.15.40:43258) 2024-04-26T23:00:43.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.499+0000 7fdde2a00640 1 -- 172.21.15.40:0/1138173054 learned_addr learned my addr 172.21.15.40:0/1138173054 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:00:43.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.499+0000 7fdde3400640 1 -- 172.21.15.40:0/1138173054 >> v2:172.21.15.40:3300/0 conn(0x7fdde40b2340 msgr2=0x7fdde40cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:43.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.499+0000 7fdde3400640 1 --2- 172.21.15.40:0/1138173054 >> v2:172.21.15.40:3300/0 conn(0x7fdde40b2340 0x7fdde40cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.499+0000 7fdde3400640 1 -- 172.21.15.40:0/1138173054 >> v1:172.21.15.103:6789/0 conn(0x7fdde4162810 legacy=0x7fdde4070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:43.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.499+0000 7fdde1600640 1 -- 172.21.15.40:0/1138173054 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3307035220 0 0) 0x7fdde40d10d0 con 0x7fdde4162810 2024-04-26T23:00:43.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.499+0000 7fdde3400640 1 -- 172.21.15.40:0/1138173054 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde40d1370 con 0x7fdde40cd930 2024-04-26T23:00:43.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.500+0000 7fdde3400640 1 --2- 172.21.15.40:0/1138173054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdde40cd930 0x7fdde40d1b30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdde40d1820 tx=0x7fdddc02a640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=592f9879a2c9e359 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:43.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.501+0000 7fdde1600640 1 -- 172.21.15.40:0/1138173054 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc02ae60 con 0x7fdde40cd930 2024-04-26T23:00:43.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.501+0000 7fdde1600640 1 -- 172.21.15.40:0/1138173054 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdddc007740 con 0x7fdde40cd930 2024-04-26T23:00:43.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.501+0000 7fdde1600640 1 -- 172.21.15.40:0/1138173054 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc007b10 con 0x7fdde40cd930 2024-04-26T23:00:43.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.501+0000 7fdde9600640 1 -- 172.21.15.40:0/1138173054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdde40cd930 msgr2=0x7fdde40d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:43.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.501+0000 7fdde9600640 1 --2- 172.21.15.40:0/1138173054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdde40cd930 0x7fdde40d1b30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdde40d1820 tx=0x7fdddc02a640 comp rx=0 tx=0).stop 2024-04-26T23:00:43.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.501+0000 7fdde9600640 1 -- 172.21.15.40:0/1138173054 shutdown_connections 2024-04-26T23:00:43.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.501+0000 7fdde9600640 1 --2- 172.21.15.40:0/1138173054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdde40cd930 0x7fdde40d1b30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.501+0000 7fdde9600640 1 --2- 172.21.15.40:0/1138173054 >> v2:172.21.15.40:3300/0 conn(0x7fdde40b2340 0x7fdde40cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.501+0000 7fdde9600640 1 -- 172.21.15.40:0/1138173054 >> 172.21.15.40:0/1138173054 conn(0x7fdde40d2da0 msgr2=0x7fdde40d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:43.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.501+0000 7fdde9600640 1 -- 172.21.15.40:0/1138173054 shutdown_connections 2024-04-26T23:00:43.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.501+0000 7fdde9600640 1 -- 172.21.15.40:0/1138173054 wait complete. 2024-04-26T23:00:43.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.503+0000 7fdde9600640 1 Processor -- start 2024-04-26T23:00:43.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.503+0000 7fdde9600640 1 -- start start 2024-04-26T23:00:43.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.504+0000 7fdde9600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdde40b2340 0x7fdde4160da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:43.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.504+0000 7fdde9600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdde40cd930 0x7fdde41f4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:43.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.504+0000 7fdde9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdde4162810 0x7fdde41f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:43.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.504+0000 7fdde9600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdde40d2070 con 0x7fdde4162810 2024-04-26T23:00:43.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.504+0000 7fdde9600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdde40d0470 con 0x7fdde40cd930 2024-04-26T23:00:43.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.504+0000 7fdde9600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdde40d21f0 con 0x7fdde40b2340 2024-04-26T23:00:43.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.504+0000 7fdde2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdde40b2340 0x7fdde4160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:43.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.504+0000 7fdde2000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdde40cd930 0x7fdde41f4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:43.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.504+0000 7fdde3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdde4162810 0x7fdde41f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:43.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.504+0000 7fdde2000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdde40cd930 0x7fdde41f4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:40338/0 (socket says 172.21.15.40:40338) 2024-04-26T23:00:43.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.504+0000 7fdde2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdde40b2340 0x7fdde4160da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:42370/0 (socket says 172.21.15.40:42370) 2024-04-26T23:00:43.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.504+0000 7fdde2000640 1 -- 172.21.15.40:0/3061816975 learned_addr learned my addr 172.21.15.40:0/3061816975 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:00:43.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.505+0000 7fdde2a00640 1 -- 172.21.15.40:0/3061816975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdde40cd930 msgr2=0x7fdde41f4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:43.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.505+0000 7fdde2a00640 1 --2- 172.21.15.40:0/3061816975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdde40cd930 0x7fdde41f4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.505+0000 7fdde2a00640 1 -- 172.21.15.40:0/3061816975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdde4162810 msgr2=0x7fdde41f5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:43.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.505+0000 7fdde2a00640 1 --2- 172.21.15.40:0/3061816975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdde4162810 0x7fdde41f5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.505+0000 7fdde2a00640 1 -- 172.21.15.40:0/3061816975 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde406d350 con 0x7fdde40b2340 2024-04-26T23:00:43.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.505+0000 7fdde2000640 1 --2- 172.21.15.40:0/3061816975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdde40cd930 0x7fdde41f4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:00:43.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.505+0000 7fdde2a00640 1 --2- 172.21.15.40:0/3061816975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdde40b2340 0x7fdde4160da0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fddd400c410 tx=0x7fddd400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:43.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.505+0000 7fdddaa00640 1 -- 172.21.15.40:0/3061816975 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd40074e0 con 0x7fdde40b2340 2024-04-26T23:00:43.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.505+0000 7fdddaa00640 1 -- 172.21.15.40:0/3061816975 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddd40047e0 con 0x7fdde40b2340 2024-04-26T23:00:43.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.505+0000 7fdddaa00640 1 -- 172.21.15.40:0/3061816975 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd4004bd0 con 0x7fdde40b2340 2024-04-26T23:00:43.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.506+0000 7fdde9600640 1 -- 172.21.15.40:0/3061816975 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde415f440 con 0x7fdde40b2340 2024-04-26T23:00:43.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.506+0000 7fdde9600640 1 -- 172.21.15.40:0/3061816975 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde415fa00 con 0x7fdde40b2340 2024-04-26T23:00:43.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.506+0000 7fdddaa00640 1 -- 172.21.15.40:0/3061816975 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fddd4011070 con 0x7fdde40b2340 2024-04-26T23:00:43.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.506+0000 7fdddaa00640 1 -- 172.21.15.40:0/3061816975 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fddd4004d70 con 0x7fdde40b2340 2024-04-26T23:00:43.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.507+0000 7fdde9600640 1 -- 172.21.15.40:0/3061816975 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdde40694f0 con 0x7fdde40b2340 2024-04-26T23:00:43.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.509+0000 7fdddaa00640 1 -- 172.21.15.40:0/3061816975 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fddd401d3f0 con 0x7fdde40b2340 2024-04-26T23:00:43.522 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:43.522+0000 7f12fb01f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T23:00:43.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.575+0000 7fdde9600640 1 -- 172.21.15.40:0/3061816975 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdde40cd380 con 0x7fdde40b2340 2024-04-26T23:00:43.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.576+0000 7fdddaa00640 1 -- 172.21.15.40:0/3061816975 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fddd401d5d0 con 0x7fdde40b2340 2024-04-26T23:00:43.576 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:00:43.576 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":10,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:00:43.199900+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:00:43.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.577+0000 7fdde9600640 1 -- 172.21.15.40:0/3061816975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdde40b2340 msgr2=0x7fdde4160da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:43.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.577+0000 7fdde9600640 1 --2- 172.21.15.40:0/3061816975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdde40b2340 0x7fdde4160da0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fddd400c410 tx=0x7fddd400c8d0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.577+0000 7fdde9600640 1 -- 172.21.15.40:0/3061816975 shutdown_connections 2024-04-26T23:00:43.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.577+0000 7fdde9600640 1 --2- 172.21.15.40:0/3061816975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdde4162810 0x7fdde41f5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.577+0000 7fdde9600640 1 --2- 172.21.15.40:0/3061816975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdde40cd930 0x7fdde41f4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.577+0000 7fdde9600640 1 --2- 172.21.15.40:0/3061816975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdde40b2340 0x7fdde4160da0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:43.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.577+0000 7fdde9600640 1 -- 172.21.15.40:0/3061816975 >> 172.21.15.40:0/3061816975 conn(0x7fdde40d2da0 msgr2=0x7fdde415cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:43.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.578+0000 7fdde9600640 1 -- 172.21.15.40:0/3061816975 shutdown_connections 2024-04-26T23:00:43.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:43.578+0000 7fdde9600640 1 -- 172.21.15.40:0/3061816975 wait complete. 2024-04-26T23:00:43.594 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-26T23:00:43.594 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-26T23:00:43.825 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:43.825+0000 7f251a9ce040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T23:00:43.870 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:43.870+0000 7f12fb01f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T23:00:43.880 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:43.880+0000 7f251a9ce040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T23:00:43.928 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:43.928+0000 7f12fb01f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T23:00:43.939 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-04-26T23:00:43.939+0000 7ff4c6c14640 -1 Falling back to public interface 2024-04-26T23:00:43.957 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-04-26T23:00:43.957+0000 7f60ce025640 -1 Falling back to public interface 2024-04-26T23:00:43.961 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-04-26T23:00:43.961+0000 7f559e024640 -1 Falling back to public interface 2024-04-26T23:00:43.962 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-26T23:00:43.962+0000 7fe8d4eee640 -1 Falling back to public interface 2024-04-26T23:00:43.963 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-26T23:00:43.963+0000 7f9f8e612640 -1 Falling back to public interface 2024-04-26T23:00:43.973 INFO:tasks.ceph.osd.3.smithi040.stderr:2024-04-26T23:00:43.973+0000 7f85731fc640 -1 Falling back to public interface 2024-04-26T23:00:43.974 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-04-26T23:00:43.974+0000 7f559c600640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6823/0: (98) Address already in use 2024-04-26T23:00:43.979 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-04-26T23:00:43.980+0000 7fe0a432c640 -1 Falling back to public interface 2024-04-26T23:00:43.983 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-04-26T23:00:43.984+0000 7fd533815640 -1 Falling back to public interface 2024-04-26T23:00:44.078 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:44.077+0000 7f251a9ce040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T23:00:44.125 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:44.125+0000 7f12fb01f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T23:00:44.243 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:44.243+0000 7f251a9ce040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T23:00:44.299 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:44.299+0000 7f12fb01f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T23:00:44.304 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:44.304+0000 7f251a9ce040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T23:00:44.359 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:44.359+0000 7f12fb01f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T23:00:44.385 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:44.385+0000 7f251a9ce040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T23:00:44.399 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T23:00:44.437 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:44.437+0000 7f251a9ce040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T23:00:44.443 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:44.444+0000 7f12fb01f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T23:00:44.500 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:44.500+0000 7f12fb01f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T23:00:44.566 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.566+0000 7fe7da000640 1 Processor -- start 2024-04-26T23:00:44.567 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.566+0000 7fe7da000640 1 -- start start 2024-04-26T23:00:44.568 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.568+0000 7fe7da000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe7d41626a0 0x7fe7d40d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:44.568 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.568+0000 7fe7da000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7d40cf990 0x7fe7d40d31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:44.569 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.568+0000 7fe7da000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7d40cfe70 con 0x7fe7d40d36f0 2024-04-26T23:00:44.569 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.568+0000 7fe7da000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe7d40d3ba0 con 0x7fe7d41626a0 2024-04-26T23:00:44.569 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.568+0000 7fe7da000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe7d40d0150 con 0x7fe7d40cf990 2024-04-26T23:00:44.569 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.568+0000 7fe7d3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7d40cf990 0x7fe7d40d31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:44.569 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.568+0000 7fe7d9600640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe7d40d36f0 0x7fe7d4162db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:43262/0 (socket says 172.21.15.40:43262) 2024-04-26T23:00:44.569 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.569+0000 7fe7d9600640 1 -- 172.21.15.40:0/3267634069 learned_addr learned my addr 172.21.15.40:0/3267634069 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:00:44.569 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.569+0000 7fe7d8c00640 1 --2- 172.21.15.40:0/3267634069 >> v2:172.21.15.103:3300/0 conn(0x7fe7d41626a0 0x7fe7d40d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:44.570 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.569+0000 7fe7d3400640 1 -- 172.21.15.40:0/3267634069 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1328043993 0 0) 0x7fe7d40cfe70 con 0x7fe7d40d36f0 2024-04-26T23:00:44.570 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.569+0000 7fe7d3400640 1 -- 172.21.15.40:0/3267634069 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7bc003520 con 0x7fe7d40d36f0 2024-04-26T23:00:44.570 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.569+0000 7fe7d8c00640 1 -- 172.21.15.40:0/3267634069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7d40cf990 msgr2=0x7fe7d40d31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:44.570 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.569+0000 7fe7d8c00640 1 --2- 172.21.15.40:0/3267634069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7d40cf990 0x7fe7d40d31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:44.570 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.569+0000 7fe7d8c00640 1 -- 172.21.15.40:0/3267634069 >> v1:172.21.15.103:6789/0 conn(0x7fe7d40d36f0 legacy=0x7fe7d4162db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:44.570 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.569+0000 7fe7d8c00640 1 -- 172.21.15.40:0/3267634069 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d40d0430 con 0x7fe7d41626a0 2024-04-26T23:00:44.570 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.569+0000 7fe7d3e00640 1 --2- 172.21.15.40:0/3267634069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7d40cf990 0x7fe7d40d31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:00:44.571 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.570+0000 7fe7d8c00640 1 --2- 172.21.15.40:0/3267634069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d41626a0 0x7fe7d40d2b60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe7c80097a0 tx=0x7fe7c802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8103476b7880b5ce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:44.571 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.571+0000 7fe7d3400640 1 -- 172.21.15.40:0/3267634069 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8037070 con 0x7fe7d41626a0 2024-04-26T23:00:44.572 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.571+0000 7fe7d3400640 1 -- 172.21.15.40:0/3267634069 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c8032440 con 0x7fe7d41626a0 2024-04-26T23:00:44.572 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.571+0000 7fe7d3400640 1 -- 172.21.15.40:0/3267634069 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8032810 con 0x7fe7d41626a0 2024-04-26T23:00:44.572 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.571+0000 7fe7da000640 1 -- 172.21.15.40:0/3267634069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d41626a0 msgr2=0x7fe7d40d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:44.572 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.571+0000 7fe7da000640 1 --2- 172.21.15.40:0/3267634069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d41626a0 0x7fe7d40d2b60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe7c80097a0 tx=0x7fe7c802a180 comp rx=0 tx=0).stop 2024-04-26T23:00:44.572 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.571+0000 7fe7da000640 1 -- 172.21.15.40:0/3267634069 shutdown_connections 2024-04-26T23:00:44.572 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.571+0000 7fe7da000640 1 --2- 172.21.15.40:0/3267634069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7d40cf990 0x7fe7d40d31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:44.572 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.571+0000 7fe7da000640 1 --2- 172.21.15.40:0/3267634069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d41626a0 0x7fe7d40d2b60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:44.572 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.571+0000 7fe7da000640 1 -- 172.21.15.40:0/3267634069 >> 172.21.15.40:0/3267634069 conn(0x7fe7d40d4d80 msgr2=0x7fe7d40d5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:44.572 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.571+0000 7fe7da000640 1 -- 172.21.15.40:0/3267634069 shutdown_connections 2024-04-26T23:00:44.572 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.571+0000 7fe7da000640 1 -- 172.21.15.40:0/3267634069 wait complete. 2024-04-26T23:00:44.573 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.573+0000 7fe7da000640 1 Processor -- start 2024-04-26T23:00:44.573 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.573+0000 7fe7da000640 1 -- start start 2024-04-26T23:00:44.575 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.574+0000 7fe7da000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d40cf990 0x7fe7d4152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:44.575 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.574+0000 7fe7da000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d40d36f0 0x7fe7d406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:44.575 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.574+0000 7fe7da000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7d41626a0 0x7fe7d406d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:44.575 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.574+0000 7fe7da000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d4162a80 con 0x7fe7d40cf990 2024-04-26T23:00:44.575 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.574+0000 7fe7da000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d40d09f0 con 0x7fe7d40d36f0 2024-04-26T23:00:44.575 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.574+0000 7fe7da000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe7d40d59f0 con 0x7fe7d41626a0 2024-04-26T23:00:44.575 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.574+0000 7fe7d8c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d40cf990 0x7fe7d4152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:44.575 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.574+0000 7fe7d8c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d40cf990 0x7fe7d4152910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39650/0 (socket says 172.21.15.40:39650) 2024-04-26T23:00:44.575 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.574+0000 7fe7d8c00640 1 -- 172.21.15.40:0/2551495337 learned_addr learned my addr 172.21.15.40:0/2551495337 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:00:44.576 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.574+0000 7fe7d3e00640 1 --2- 172.21.15.40:0/2551495337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d40d36f0 0x7fe7d406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:44.576 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.574+0000 7fe7d9600640 1 --2- 172.21.15.40:0/2551495337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7d41626a0 0x7fe7d406d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:44.576 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.575+0000 7fe7d8c00640 1 -- 172.21.15.40:0/2551495337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7d41626a0 msgr2=0x7fe7d406d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:44.576 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.575+0000 7fe7d8c00640 1 --2- 172.21.15.40:0/2551495337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7d41626a0 0x7fe7d406d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:44.576 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.575+0000 7fe7d8c00640 1 -- 172.21.15.40:0/2551495337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d40d36f0 msgr2=0x7fe7d406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:44.576 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.575+0000 7fe7d8c00640 1 --2- 172.21.15.40:0/2551495337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d40d36f0 0x7fe7d406d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:44.576 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.575+0000 7fe7d8c00640 1 -- 172.21.15.40:0/2551495337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d41f8190 con 0x7fe7d40cf990 2024-04-26T23:00:44.576 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.575+0000 7fe7d8c00640 1 --2- 172.21.15.40:0/2551495337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d40cf990 0x7fe7d4152910 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8035040 tx=0x7fe7c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:44.577 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.575+0000 7fe7d0c00640 1 -- 172.21.15.40:0/2551495337 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8041030 con 0x7fe7d40cf990 2024-04-26T23:00:44.577 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.576+0000 7fe7d0c00640 1 -- 172.21.15.40:0/2551495337 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c803d4e0 con 0x7fe7d40cf990 2024-04-26T23:00:44.577 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.576+0000 7fe7d0c00640 1 -- 172.21.15.40:0/2551495337 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8037040 con 0x7fe7d40cf990 2024-04-26T23:00:44.577 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.576+0000 7fe7da000640 1 -- 172.21.15.40:0/2551495337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d41f8470 con 0x7fe7d40cf990 2024-04-26T23:00:44.577 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.576+0000 7fe7da000640 1 -- 172.21.15.40:0/2551495337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d41f8a30 con 0x7fe7d40cf990 2024-04-26T23:00:44.577 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.576+0000 7fe7d0c00640 1 -- 172.21.15.40:0/2551495337 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe7c8044070 con 0x7fe7d40cf990 2024-04-26T23:00:44.577 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.576+0000 7fe7d0c00640 1 -- 172.21.15.40:0/2551495337 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fe7c8046690 con 0x7fe7d40cf990 2024-04-26T23:00:44.577 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.577+0000 7fe7da000640 1 -- 172.21.15.40:0/2551495337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d40506c0 con 0x7fe7d40cf990 2024-04-26T23:00:44.579 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.579+0000 7fe7d0c00640 1 -- 172.21.15.40:0/2551495337 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fe7c8046900 con 0x7fe7d40cf990 2024-04-26T23:00:44.647 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.647+0000 7fe7da000640 1 -- 172.21.15.40:0/2551495337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe7d40ceb60 con 0x7fe7d40cf990 2024-04-26T23:00:44.648 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.648+0000 7fe7d0c00640 1 -- 172.21.15.40:0/2551495337 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fe7c8046ae0 con 0x7fe7d40cf990 2024-04-26T23:00:44.648 INFO:teuthology.misc.health.smithi040.stdout: 2024-04-26T23:00:44.648 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":10,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:00:43.199900+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:00:44.649 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.649+0000 7fe7da000640 1 -- 172.21.15.40:0/2551495337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d40cf990 msgr2=0x7fe7d4152910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:44.650 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.649+0000 7fe7da000640 1 --2- 172.21.15.40:0/2551495337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d40cf990 0x7fe7d4152910 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8035040 tx=0x7fe7c80046c0 comp rx=0 tx=0).stop 2024-04-26T23:00:44.650 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.649+0000 7fe7da000640 1 -- 172.21.15.40:0/2551495337 shutdown_connections 2024-04-26T23:00:44.650 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.649+0000 7fe7da000640 1 --2- 172.21.15.40:0/2551495337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7d41626a0 0x7fe7d406d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:44.650 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.649+0000 7fe7da000640 1 --2- 172.21.15.40:0/2551495337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d40d36f0 0x7fe7d406d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:44.650 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.649+0000 7fe7da000640 1 --2- 172.21.15.40:0/2551495337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d40cf990 0x7fe7d4152910 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:44.650 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.649+0000 7fe7da000640 1 -- 172.21.15.40:0/2551495337 >> 172.21.15.40:0/2551495337 conn(0x7fe7d40d4d80 msgr2=0x7fe7d40ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:44.650 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.649+0000 7fe7da000640 1 -- 172.21.15.40:0/2551495337 shutdown_connections 2024-04-26T23:00:44.650 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:44.649+0000 7fe7da000640 1 -- 172.21.15.40:0/2551495337 wait complete. 2024-04-26T23:00:44.661 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-26T23:00:44.981 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:44.981+0000 7f251a9ce040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T23:00:45.066 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:45.066+0000 7f12fb01f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T23:00:45.116 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:45.116+0000 7f251a9ce040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T23:00:45.210 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-04-26T23:00:45.209+0000 7f12fb01f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T23:00:46.156 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:46.156+0000 7f24bb000640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-26T23:00:46.159 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-26T23:00:46.159+0000 7f24bb000640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-26T23:00:46.726 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:00:46.732 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:00:47.367 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-04-26T23:00:47.367+0000 7fd533815640 -1 osd.7 0 log_to_monitors true 2024-04-26T23:00:47.563 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-26T23:00:47.563+0000 7f9f8e612640 -1 osd.4 0 log_to_monitors true 2024-04-26T23:00:47.572 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-26T23:00:47.572+0000 7fe8d4eee640 -1 osd.5 0 log_to_monitors true 2024-04-26T23:00:47.597 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-04-26T23:00:47.597+0000 7fe0a432c640 -1 osd.6 0 log_to_monitors true 2024-04-26T23:00:47.823 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-04-26T23:00:47.823+0000 7ff4c6c14640 -1 osd.0 0 log_to_monitors true 2024-04-26T23:00:47.834 INFO:tasks.ceph.osd.3.smithi040.stderr:2024-04-26T23:00:47.834+0000 7f85731fc640 -1 osd.3 0 log_to_monitors true 2024-04-26T23:00:47.838 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-04-26T23:00:47.839+0000 7f559e024640 -1 osd.1 0 log_to_monitors true 2024-04-26T23:00:47.842 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-04-26T23:00:47.843+0000 7f60ce025640 -1 osd.2 0 log_to_monitors true 2024-04-26T23:00:51.415 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-26T23:00:51.415+0000 7fe8cfe00640 -1 osd.5 0 waiting for initial osdmap 2024-04-26T23:00:51.427 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-26T23:00:51.428+0000 7fe8c5e00640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:00:51.467 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T23:00:51.525 INFO:tasks.ceph.osd.3.smithi040.stderr:2024-04-26T23:00:51.525+0000 7f856e000640 -1 osd.3 0 waiting for initial osdmap 2024-04-26T23:00:51.526 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-04-26T23:00:51.526+0000 7f5599400640 -1 osd.1 0 waiting for initial osdmap 2024-04-26T23:00:51.532 INFO:tasks.ceph.osd.3.smithi040.stderr:2024-04-26T23:00:51.532+0000 7f8564000640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:00:51.532 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-04-26T23:00:51.532+0000 7f558ea00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:00:51.536 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-04-26T23:00:51.534+0000 7fe09f200640 -1 osd.6 0 waiting for initial osdmap 2024-04-26T23:00:51.551 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-04-26T23:00:51.551+0000 7fe095200640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:00:51.572 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-04-26T23:00:51.573+0000 7f60c8a00640 -1 osd.2 0 waiting for initial osdmap 2024-04-26T23:00:51.577 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-04-26T23:00:51.577+0000 7f60bea00640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:00:51.578 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-26T23:00:51.578+0000 7f9f89800640 -1 osd.4 0 waiting for initial osdmap 2024-04-26T23:00:51.581 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-04-26T23:00:51.581+0000 7ff4c1600640 -1 osd.0 0 waiting for initial osdmap 2024-04-26T23:00:51.589 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-04-26T23:00:51.589+0000 7ff4b7600640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:00:51.589 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-04-26T23:00:51.589+0000 7fd52e000640 -1 osd.7 0 waiting for initial osdmap 2024-04-26T23:00:51.592 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-26T23:00:51.593+0000 7f9f7ee00640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:00:51.602 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-04-26T23:00:51.602+0000 7fd524000640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:00:51.642 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.642+0000 7fcd2a400640 1 Processor -- start 2024-04-26T23:00:51.643 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.643+0000 7fcd2a400640 1 -- start start 2024-04-26T23:00:51.644 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.644+0000 7fcd2a400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd24146940 0x7fcd240b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:51.644 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.644+0000 7fcd2a400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd24161250 0x7fcd240722c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:51.644 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.644+0000 7fcd2a400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd2406df40 con 0x7fcd24073c40 2024-04-26T23:00:51.644 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.644+0000 7fcd2a400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcd240b4080 con 0x7fcd24146940 2024-04-26T23:00:51.644 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.644+0000 7fcd2a400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcd240b61e0 con 0x7fcd24161250 2024-04-26T23:00:51.645 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.645+0000 7fcd23e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd24161250 0x7fcd240722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:51.645 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.645+0000 7fcd23400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd24146940 0x7fcd240b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:51.645 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.645+0000 7fcd1be00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fcd24073c40 0x7fcd24160b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:38342/0 (socket says 172.21.15.40:38342) 2024-04-26T23:00:51.645 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.645+0000 7fcd1be00640 1 -- 172.21.15.40:0/3216375154 learned_addr learned my addr 172.21.15.40:0/3216375154 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:00:51.645 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.645+0000 7fcd23400640 1 -- 172.21.15.40:0/3216375154 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd24161250 msgr2=0x7fcd240722c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:51.645 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.645+0000 7fcd23400640 1 --2- 172.21.15.40:0/3216375154 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd24161250 0x7fcd240722c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:51.646 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.645+0000 7fcd23400640 1 -- 172.21.15.40:0/3216375154 >> v1:172.21.15.103:6789/0 conn(0x7fcd24073c40 legacy=0x7fcd24160b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:51.646 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.645+0000 7fcd23400640 1 -- 172.21.15.40:0/3216375154 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd240b64c0 con 0x7fcd24146940 2024-04-26T23:00:51.646 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.646+0000 7fcd23400640 1 --2- 172.21.15.40:0/3216375154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd24146940 0x7fcd240b3b40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcd140097a0 tx=0x7fcd1402a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=254e42f454e2e018 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:51.647 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.647+0000 7fcd22a00640 1 -- 172.21.15.40:0/3216375154 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd14037070 con 0x7fcd24146940 2024-04-26T23:00:51.647 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.647+0000 7fcd22a00640 1 -- 172.21.15.40:0/3216375154 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd1402adb0 con 0x7fcd24146940 2024-04-26T23:00:51.647 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.647+0000 7fcd22a00640 1 -- 172.21.15.40:0/3216375154 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd14032660 con 0x7fcd24146940 2024-04-26T23:00:51.647 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.647+0000 7fcd2a400640 1 -- 172.21.15.40:0/3216375154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd24146940 msgr2=0x7fcd240b3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:51.648 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.647+0000 7fcd2a400640 1 --2- 172.21.15.40:0/3216375154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd24146940 0x7fcd240b3b40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcd140097a0 tx=0x7fcd1402a060 comp rx=0 tx=0).stop 2024-04-26T23:00:51.648 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.647+0000 7fcd2a400640 1 -- 172.21.15.40:0/3216375154 shutdown_connections 2024-04-26T23:00:51.648 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.647+0000 7fcd2a400640 1 --2- 172.21.15.40:0/3216375154 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd24161250 0x7fcd240722c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:51.648 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.647+0000 7fcd2a400640 1 --2- 172.21.15.40:0/3216375154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd24146940 0x7fcd240b3b40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:51.648 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.647+0000 7fcd2a400640 1 -- 172.21.15.40:0/3216375154 >> 172.21.15.40:0/3216375154 conn(0x7fcd240d0b30 msgr2=0x7fcd240d0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:51.648 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.647+0000 7fcd2a400640 1 -- 172.21.15.40:0/3216375154 shutdown_connections 2024-04-26T23:00:51.648 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.647+0000 7fcd2a400640 1 -- 172.21.15.40:0/3216375154 wait complete. 2024-04-26T23:00:51.649 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.649+0000 7fcd2a400640 1 Processor -- start 2024-04-26T23:00:51.649 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.649+0000 7fcd2a400640 1 -- start start 2024-04-26T23:00:51.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.650+0000 7fcd2a400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd24073c40 0x7fcd24143c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:51.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.650+0000 7fcd2a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd24146940 0x7fcd24145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:51.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.650+0000 7fcd2a400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd24161250 0x7fcd2415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:51.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.650+0000 7fcd2a400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcd24072980 con 0x7fcd24146940 2024-04-26T23:00:51.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.650+0000 7fcd2a400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcd24072800 con 0x7fcd24073c40 2024-04-26T23:00:51.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.650+0000 7fcd2a400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcd24072b00 con 0x7fcd24161250 2024-04-26T23:00:51.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.650+0000 7fcd1be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd24146940 0x7fcd24145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:51.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.650+0000 7fcd1be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd24146940 0x7fcd24145cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39200/0 (socket says 172.21.15.40:39200) 2024-04-26T23:00:51.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.650+0000 7fcd1be00640 1 -- 172.21.15.40:0/438494218 learned_addr learned my addr 172.21.15.40:0/438494218 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:00:51.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd23400640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd24073c40 0x7fcd24143c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:51.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd23e00640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd24161250 0x7fcd2415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:51.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd1be00640 1 -- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd24161250 msgr2=0x7fcd2415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:51.652 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd1be00640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd24161250 0x7fcd2415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:51.652 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd1be00640 1 -- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd24073c40 msgr2=0x7fcd24143c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:51.652 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd1be00640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd24073c40 0x7fcd24143c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:51.652 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd1be00640 1 -- 172.21.15.40:0/438494218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd241442e0 con 0x7fcd24146940 2024-04-26T23:00:51.652 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd23400640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd24073c40 0x7fcd24143c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:00:51.652 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd23e00640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd24161250 0x7fcd2415ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:00:51.652 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd1be00640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd24146940 0x7fcd24145cd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00c400 tx=0x7fcd0c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:51.653 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd1b400640 1 -- 172.21.15.40:0/438494218 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c017070 con 0x7fcd24146940 2024-04-26T23:00:51.653 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd1b400640 1 -- 172.21.15.40:0/438494218 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0c005ce0 con 0x7fcd24146940 2024-04-26T23:00:51.653 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd1b400640 1 -- 172.21.15.40:0/438494218 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c0072f0 con 0x7fcd24146940 2024-04-26T23:00:51.653 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.651+0000 7fcd2a400640 1 -- 172.21.15.40:0/438494218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd241445c0 con 0x7fcd24146940 2024-04-26T23:00:51.653 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.652+0000 7fcd2a400640 1 -- 172.21.15.40:0/438494218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd24144b80 con 0x7fcd24146940 2024-04-26T23:00:51.653 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.653+0000 7fcd2a400640 1 -- 172.21.15.40:0/438494218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd24076610 con 0x7fcd24146940 2024-04-26T23:00:51.653 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.653+0000 7fcd1b400640 1 -- 172.21.15.40:0/438494218 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7fcd0c01a070 con 0x7fcd24146940 2024-04-26T23:00:51.654 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.654+0000 7fcd1b400640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd0006c860 0x7fcd0006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:51.654 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.654+0000 7fcd1b400640 1 -- 172.21.15.40:0/438494218 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7fcd0c08a0d0 con 0x7fcd24146940 2024-04-26T23:00:51.655 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.655+0000 7fcd23400640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd0006c860 0x7fcd0006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:51.656 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.656+0000 7fcd23400640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd0006c860 0x7fcd0006ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcd14034000 tx=0x7fcd1402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:51.660 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.660+0000 7fcd1b400640 1 -- 172.21.15.40:0/438494218 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fcd0c059370 con 0x7fcd24146940 2024-04-26T23:00:51.883 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.883+0000 7fcd2a400640 1 -- 172.21.15.40:0/438494218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcd2415cf50 con 0x7fcd24146940 2024-04-26T23:00:51.884 INFO:teuthology.misc.health.smithi040.stdout: 2024-04-26T23:00:51.884 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":12,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:00:49.171522+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:00:52.173 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.884+0000 7fcd1b400640 1 -- 172.21.15.40:0/438494218 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7fcd0c05f030 con 0x7fcd24146940 2024-04-26T23:00:52.173 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.886+0000 7fcd2a400640 1 -- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd0006c860 msgr2=0x7fcd0006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:52.173 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.886+0000 7fcd2a400640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd0006c860 0x7fcd0006ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcd14034000 tx=0x7fcd1402e000 comp rx=0 tx=0).stop 2024-04-26T23:00:52.173 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.886+0000 7fcd2a400640 1 -- 172.21.15.40:0/438494218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd24146940 msgr2=0x7fcd24145cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:52.174 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.886+0000 7fcd2a400640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd24146940 0x7fcd24145cd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00c400 tx=0x7fcd0c00c8c0 comp rx=0 tx=0).stop 2024-04-26T23:00:52.174 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.887+0000 7fcd2a400640 1 -- 172.21.15.40:0/438494218 shutdown_connections 2024-04-26T23:00:52.174 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.887+0000 7fcd2a400640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd0006c860 0x7fcd0006ed20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:52.174 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.887+0000 7fcd2a400640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd24161250 0x7fcd2415ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:52.174 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.887+0000 7fcd2a400640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd24146940 0x7fcd24145cd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:52.174 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.887+0000 7fcd2a400640 1 --2- 172.21.15.40:0/438494218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd24073c40 0x7fcd24143c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:52.174 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.887+0000 7fcd2a400640 1 -- 172.21.15.40:0/438494218 >> 172.21.15.40:0/438494218 conn(0x7fcd240d0b30 msgr2=0x7fcd240afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:52.174 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.887+0000 7fcd2a400640 1 -- 172.21.15.40:0/438494218 shutdown_connections 2024-04-26T23:00:52.174 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:51.887+0000 7fcd2a400640 1 -- 172.21.15.40:0/438494218 wait complete. 2024-04-26T23:00:52.196 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-26T23:00:59.005 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T23:00:59.207 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.206+0000 7fef96a00640 1 Processor -- start 2024-04-26T23:00:59.207 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.207+0000 7fef96a00640 1 -- start start 2024-04-26T23:00:59.209 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.208+0000 7fef96a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fef900d3b60 0x7fef900d5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:59.209 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.208+0000 7fef96a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef900d2c50 0x7fef900d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:59.209 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.208+0000 7fef96a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef900cfaf0 con 0x7fef9015f0f0 2024-04-26T23:00:59.209 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.208+0000 7fef96a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fef900d0e00 con 0x7fef900d2c50 2024-04-26T23:00:59.209 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.208+0000 7fef96a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fef900cfd60 con 0x7fef900d3b60 2024-04-26T23:00:59.209 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.209+0000 7fef95600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fef9015f0f0 0x7fef900d3450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52594/0 (socket says 172.21.15.40:52594) 2024-04-26T23:00:59.209 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.209+0000 7fef95600640 1 -- 172.21.15.40:0/2136175085 learned_addr learned my addr 172.21.15.40:0/2136175085 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:00:59.209 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.209+0000 7fef96000640 1 --2- 172.21.15.40:0/2136175085 >> v2:172.21.15.40:3300/0 conn(0x7fef900d2c50 0x7fef900d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:59.209 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.209+0000 7fef94c00640 1 --2- 172.21.15.40:0/2136175085 >> v2:172.21.15.103:3300/0 conn(0x7fef900d3b60 0x7fef900d5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:59.210 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.209+0000 7fef8fe00640 1 -- 172.21.15.40:0/2136175085 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1092321917 0 0) 0x7fef900cfaf0 con 0x7fef9015f0f0 2024-04-26T23:00:59.210 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.209+0000 7fef8fe00640 1 -- 172.21.15.40:0/2136175085 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef78003520 con 0x7fef9015f0f0 2024-04-26T23:00:59.210 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.209+0000 7fef96000640 1 -- 172.21.15.40:0/2136175085 >> v2:172.21.15.103:3300/0 conn(0x7fef900d3b60 msgr2=0x7fef900d5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:00:59.210 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.209+0000 7fef96000640 1 --2- 172.21.15.40:0/2136175085 >> v2:172.21.15.103:3300/0 conn(0x7fef900d3b60 0x7fef900d5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.210 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.209+0000 7fef96000640 1 -- 172.21.15.40:0/2136175085 >> v1:172.21.15.40:6789/0 conn(0x7fef9015f0f0 legacy=0x7fef900d3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:59.210 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.209+0000 7fef96000640 1 -- 172.21.15.40:0/2136175085 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef900d0040 con 0x7fef900d2c50 2024-04-26T23:00:59.212 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.211+0000 7fef96000640 1 --2- 172.21.15.40:0/2136175085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef900d2c50 0x7fef900d1630 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fef9008f5a0 tx=0x7fef8802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa52e979a985175 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:59.212 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.211+0000 7fef8fe00640 1 -- 172.21.15.40:0/2136175085 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef88007740 con 0x7fef900d2c50 2024-04-26T23:00:59.212 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.211+0000 7fef8fe00640 1 -- 172.21.15.40:0/2136175085 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef880078e0 con 0x7fef900d2c50 2024-04-26T23:00:59.212 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.211+0000 7fef8fe00640 1 -- 172.21.15.40:0/2136175085 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef88007cb0 con 0x7fef900d2c50 2024-04-26T23:00:59.212 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.211+0000 7fef96a00640 1 -- 172.21.15.40:0/2136175085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef900d2c50 msgr2=0x7fef900d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:59.212 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.211+0000 7fef96a00640 1 --2- 172.21.15.40:0/2136175085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef900d2c50 0x7fef900d1630 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fef9008f5a0 tx=0x7fef8802a710 comp rx=0 tx=0).stop 2024-04-26T23:00:59.212 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.212+0000 7fef96a00640 1 -- 172.21.15.40:0/2136175085 shutdown_connections 2024-04-26T23:00:59.212 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.212+0000 7fef96a00640 1 --2- 172.21.15.40:0/2136175085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef900d2c50 0x7fef900d1630 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.212 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.212+0000 7fef96a00640 1 --2- 172.21.15.40:0/2136175085 >> v2:172.21.15.103:3300/0 conn(0x7fef900d3b60 0x7fef900d5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.212 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.212+0000 7fef96a00640 1 -- 172.21.15.40:0/2136175085 >> 172.21.15.40:0/2136175085 conn(0x7fef900d4e70 msgr2=0x7fef900d5280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:59.212 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.212+0000 7fef96a00640 1 -- 172.21.15.40:0/2136175085 shutdown_connections 2024-04-26T23:00:59.213 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.212+0000 7fef96a00640 1 -- 172.21.15.40:0/2136175085 wait complete. 2024-04-26T23:00:59.213 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.213+0000 7fef96a00640 1 Processor -- start 2024-04-26T23:00:59.213 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.214+0000 7fef96a00640 1 -- start start 2024-04-26T23:00:59.216 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.214+0000 7fef96a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef900d2c50 0x7fef9015c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:59.216 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.214+0000 7fef96a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef900d3b60 0x7fef900ba020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:59.216 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef96a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef9015f0f0 0x7fef900b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:59.216 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef96a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fef900d08e0 con 0x7fef9015f0f0 2024-04-26T23:00:59.216 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef96a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef900d0600 con 0x7fef900d2c50 2024-04-26T23:00:59.216 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef96a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fef9015f4d0 con 0x7fef900d3b60 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef96000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef9015f0f0 0x7fef900b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef96000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef9015f0f0 0x7fef900b8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55400/0 (socket says 172.21.15.40:55400) 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef96000640 1 -- 172.21.15.40:0/3935898496 learned_addr learned my addr 172.21.15.40:0/3935898496 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef95600640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef900d2c50 0x7fef9015c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef96000640 1 -- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef900d3b60 msgr2=0x7fef900ba020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef94c00640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef900d3b60 0x7fef900ba020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef96000640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef900d3b60 0x7fef900ba020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef96000640 1 -- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef900d2c50 msgr2=0x7fef9015c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef96000640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef900d2c50 0x7fef9015c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef96000640 1 -- 172.21.15.40:0/3935898496 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9015c930 con 0x7fef9015f0f0 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.215+0000 7fef95600640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef900d2c50 0x7fef9015c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.216+0000 7fef96000640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef9015f0f0 0x7fef900b8c00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fef88009e50 tx=0x7fef88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.216+0000 7fef8d600640 1 -- 172.21.15.40:0/3935898496 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef8803f070 con 0x7fef9015f0f0 2024-04-26T23:00:59.217 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.216+0000 7fef8d600640 1 -- 172.21.15.40:0/3935898496 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef88031bf0 con 0x7fef9015f0f0 2024-04-26T23:00:59.218 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.216+0000 7fef8d600640 1 -- 172.21.15.40:0/3935898496 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef88009450 con 0x7fef9015f0f0 2024-04-26T23:00:59.218 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.216+0000 7fef96a00640 1 -- 172.21.15.40:0/3935898496 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef900ba970 con 0x7fef9015f0f0 2024-04-26T23:00:59.218 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.216+0000 7fef96a00640 1 -- 172.21.15.40:0/3935898496 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef900b8120 con 0x7fef9015f0f0 2024-04-26T23:00:59.218 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.217+0000 7fef96a00640 1 -- 172.21.15.40:0/3935898496 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef900506c0 con 0x7fef9015f0f0 2024-04-26T23:00:59.218 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.218+0000 7fef8d600640 1 -- 172.21.15.40:0/3935898496 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fef8803b410 con 0x7fef9015f0f0 2024-04-26T23:00:59.220 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.220+0000 7fef8d600640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fef7006c8f0 0x7fef7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:59.220 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.220+0000 7fef8d600640 1 -- 172.21.15.40:0/3935898496 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fef880acf00 con 0x7fef9015f0f0 2024-04-26T23:00:59.220 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.220+0000 7fef95600640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fef7006c8f0 0x7fef7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:59.221 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.221+0000 7fef95600640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fef7006c8f0 0x7fef7006edb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fef84005880 tx=0x7fef84009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:59.225 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.225+0000 7fef8d600640 1 -- 172.21.15.40:0/3935898496 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fef9015c930 con 0x7fef9015f0f0 2024-04-26T23:00:59.444 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.443+0000 7fef96a00640 1 -- 172.21.15.40:0/3935898496 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fef900d3230 con 0x7fef9015f0f0 2024-04-26T23:00:59.445 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.445+0000 7fef8d600640 1 -- 172.21.15.40:0/3935898496 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12848 (secure 0 0 0) 0x7fef8807b020 con 0x7fef9015f0f0 2024-04-26T23:00:59.445 INFO:teuthology.misc.health.smithi040.stdout: 2024-04-26T23:00:59.446 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":16,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:00:55.462707+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:00:59.448 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.448+0000 7fef96a00640 1 -- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fef7006c8f0 msgr2=0x7fef7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:59.448 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.448+0000 7fef96a00640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fef7006c8f0 0x7fef7006edb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fef84005880 tx=0x7fef84009040 comp rx=0 tx=0).stop 2024-04-26T23:00:59.448 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.448+0000 7fef96a00640 1 -- 172.21.15.40:0/3935898496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef9015f0f0 msgr2=0x7fef900b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:59.448 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.448+0000 7fef96a00640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef9015f0f0 0x7fef900b8c00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fef88009e50 tx=0x7fef88004a40 comp rx=0 tx=0).stop 2024-04-26T23:00:59.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.448+0000 7fef96a00640 1 -- 172.21.15.40:0/3935898496 shutdown_connections 2024-04-26T23:00:59.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.448+0000 7fef96a00640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fef7006c8f0 0x7fef7006edb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.448+0000 7fef96a00640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef9015f0f0 0x7fef900b8c00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.448+0000 7fef96a00640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef900d3b60 0x7fef900ba020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.448+0000 7fef96a00640 1 --2- 172.21.15.40:0/3935898496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef900d2c50 0x7fef9015c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.448+0000 7fef96a00640 1 -- 172.21.15.40:0/3935898496 >> 172.21.15.40:0/3935898496 conn(0x7fef900d4e70 msgr2=0x7fef90163410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:59.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.449+0000 7fef96a00640 1 -- 172.21.15.40:0/3935898496 shutdown_connections 2024-04-26T23:00:59.450 INFO:teuthology.misc.health.smithi040.stderr:2024-04-26T23:00:59.449+0000 7fef96a00640 1 -- 172.21.15.40:0/3935898496 wait complete. 2024-04-26T23:00:59.464 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-26T23:00:59.464 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-26T23:00:59.464 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-26T23:00:59.464 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-26T23:00:59.471 INFO:tasks.ceph.mds.a:Started 2024-04-26T23:00:59.471 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-26T23:00:59.472 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-26T23:00:59.476 INFO:tasks.ceph.mds.c:Started 2024-04-26T23:00:59.476 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-26T23:00:59.477 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-26T23:00:59.484 INFO:tasks.ceph.mds.b:Started 2024-04-26T23:00:59.484 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-26T23:00:59.485 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-26T23:00:59.492 INFO:tasks.ceph.mds.d:Started 2024-04-26T23:00:59.492 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-26T23:00:59.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:00:59.618 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-04-26T23:00:59.621 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:00:59.642 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-26T23:00:59.642 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-26T23:00:59.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.674+0000 7fe118000640 1 Processor -- start 2024-04-26T23:00:59.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.674+0000 7fe118000640 1 -- start start 2024-04-26T23:00:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.675+0000 7fe118000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe11015f540 0x7fe1100d3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.675+0000 7fe118000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1100d2d70 con 0x7fe1100b5ab0 2024-04-26T23:00:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.675+0000 7fe118000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1100d3050 con 0x7fe1100b3ff0 2024-04-26T23:00:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.675+0000 7fe118000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe11015f920 con 0x7fe11015f540 2024-04-26T23:00:59.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.675+0000 7fe115600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe11015f540 0x7fe1100d3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:59.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.675+0000 7fe116000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe1100b5ab0 0x7fe1100d1e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52608/0 (socket says 172.21.15.40:52608) 2024-04-26T23:00:59.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.675+0000 7fe116000640 1 -- 172.21.15.40:0/591638467 learned_addr learned my addr 172.21.15.40:0/591638467 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:00:59.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.676+0000 7fe115600640 1 -- 172.21.15.40:0/591638467 >> v1:172.21.15.103:6789/0 conn(0x7fe1100b3ff0 legacy=0x7fe1100d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:59.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.676+0000 7fe115600640 1 -- 172.21.15.40:0/591638467 >> v1:172.21.15.40:6789/0 conn(0x7fe1100b5ab0 legacy=0x7fe1100d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:59.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.676+0000 7fe115600640 1 -- 172.21.15.40:0/591638467 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1100d08e0 con 0x7fe11015f540 2024-04-26T23:00:59.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.676+0000 7fe10fe00640 1 -- 172.21.15.40:0/591638467 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4118145633 0 0) 0x7fe1100d2d70 con 0x7fe1100b5ab0 2024-04-26T23:00:59.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.677+0000 7fe115600640 1 --2- 172.21.15.40:0/591638467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe11015f540 0x7fe1100d3470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe1040097a0 tx=0x7fe10402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f2a6096b1eadc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:59.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.677+0000 7fe10fe00640 1 -- 172.21.15.40:0/591638467 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104037070 con 0x7fe11015f540 2024-04-26T23:00:59.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.677+0000 7fe10fe00640 1 -- 172.21.15.40:0/591638467 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe104032440 con 0x7fe11015f540 2024-04-26T23:00:59.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.677+0000 7fe10fe00640 1 -- 172.21.15.40:0/591638467 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104032810 con 0x7fe11015f540 2024-04-26T23:00:59.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.678+0000 7fe118000640 1 -- 172.21.15.40:0/591638467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe11015f540 msgr2=0x7fe1100d3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:59.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.678+0000 7fe118000640 1 --2- 172.21.15.40:0/591638467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe11015f540 0x7fe1100d3470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe1040097a0 tx=0x7fe10402a1a0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.678+0000 7fe118000640 1 -- 172.21.15.40:0/591638467 shutdown_connections 2024-04-26T23:00:59.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.678+0000 7fe118000640 1 --2- 172.21.15.40:0/591638467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe11015f540 0x7fe1100d3470 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.678+0000 7fe118000640 1 -- 172.21.15.40:0/591638467 >> 172.21.15.40:0/591638467 conn(0x7fe1100d4f90 msgr2=0x7fe1100d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:59.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.678+0000 7fe118000640 1 -- 172.21.15.40:0/591638467 shutdown_connections 2024-04-26T23:00:59.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.678+0000 7fe118000640 1 -- 172.21.15.40:0/591638467 wait complete. 2024-04-26T23:00:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.680+0000 7fe118000640 1 Processor -- start 2024-04-26T23:00:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.680+0000 7fe118000640 1 -- start start 2024-04-26T23:00:59.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe118000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1100b3ff0 0x7fe1100ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:59.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe118000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1100b5ab0 0x7fe110152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:59.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe118000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe11015f540 0x7fe1100b9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:59.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe118000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe110068e50 con 0x7fe1100b3ff0 2024-04-26T23:00:59.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe118000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe1100d0f30 con 0x7fe11015f540 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe118000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe1100d2590 con 0x7fe1100b5ab0 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe115600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1100b3ff0 0x7fe1100ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe115600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1100b3ff0 0x7fe1100ce8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55444/0 (socket says 172.21.15.40:55444) 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe115600640 1 -- 172.21.15.40:0/1469709386 learned_addr learned my addr 172.21.15.40:0/1469709386 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe116000640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe11015f540 0x7fe1100b9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe114c00640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1100b5ab0 0x7fe110152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe115600640 1 -- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1100b5ab0 msgr2=0x7fe110152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe115600640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1100b5ab0 0x7fe110152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe115600640 1 -- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe11015f540 msgr2=0x7fe1100b9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe115600640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe11015f540 0x7fe1100b9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe115600640 1 -- 172.21.15.40:0/1469709386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1100ceed0 con 0x7fe1100b3ff0 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe116000640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe11015f540 0x7fe1100b9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.681+0000 7fe114c00640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1100b5ab0 0x7fe110152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:00:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.682+0000 7fe115600640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1100b3ff0 0x7fe1100ce8d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe104000c00 tx=0x7fe1040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:59.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.682+0000 7fe10d600640 1 -- 172.21.15.40:0/1469709386 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104037040 con 0x7fe1100b3ff0 2024-04-26T23:00:59.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.682+0000 7fe10d600640 1 -- 172.21.15.40:0/1469709386 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe104042070 con 0x7fe1100b3ff0 2024-04-26T23:00:59.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.682+0000 7fe10d600640 1 -- 172.21.15.40:0/1469709386 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104030a70 con 0x7fe1100b3ff0 2024-04-26T23:00:59.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.682+0000 7fe118000640 1 -- 172.21.15.40:0/1469709386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11014fa20 con 0x7fe1100b3ff0 2024-04-26T23:00:59.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.682+0000 7fe118000640 1 -- 172.21.15.40:0/1469709386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe11014ffe0 con 0x7fe1100b3ff0 2024-04-26T23:00:59.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.683+0000 7fe118000640 1 -- 172.21.15.40:0/1469709386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1100694f0 con 0x7fe1100b3ff0 2024-04-26T23:00:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.684+0000 7fe10d600640 1 -- 172.21.15.40:0/1469709386 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe104030c10 con 0x7fe1100b3ff0 2024-04-26T23:00:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.686+0000 7fe10d600640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe0f006c8f0 0x7fe0f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:00:59.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.686+0000 7fe10d600640 1 -- 172.21.15.40:0/1469709386 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fe1040ac240 con 0x7fe1100b3ff0 2024-04-26T23:00:59.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.686+0000 7fe114c00640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe0f006c8f0 0x7fe0f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:00:59.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.687+0000 7fe114c00640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe0f006c8f0 0x7fe0f006edb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc007fd0 tx=0x7fe0fc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:00:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.691+0000 7fe10d600640 1 -- 172.21.15.40:0/1469709386 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe10407a9b0 con 0x7fe1100b3ff0 2024-04-26T23:00:59.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.892+0000 7fe118000640 1 -- 172.21.15.40:0/1469709386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe110162d00 con 0x7fe1100b3ff0 2024-04-26T23:00:59.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.895+0000 7fe10d600640 1 -- 172.21.15.40:0/1469709386 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12848 (secure 0 0 0) 0x7fe10407a2b0 con 0x7fe1100b3ff0 2024-04-26T23:00:59.895 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:00:59.895 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":16,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:00:55.462707+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:00:59.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.897+0000 7fe118000640 1 -- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe0f006c8f0 msgr2=0x7fe0f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:59.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.897+0000 7fe118000640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe0f006c8f0 0x7fe0f006edb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc007fd0 tx=0x7fe0fc009040 comp rx=0 tx=0).stop 2024-04-26T23:00:59.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.897+0000 7fe118000640 1 -- 172.21.15.40:0/1469709386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1100b3ff0 msgr2=0x7fe1100ce8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:00:59.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.897+0000 7fe118000640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1100b3ff0 0x7fe1100ce8d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe104000c00 tx=0x7fe1040046c0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.897+0000 7fe118000640 1 -- 172.21.15.40:0/1469709386 shutdown_connections 2024-04-26T23:00:59.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.897+0000 7fe118000640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe0f006c8f0 0x7fe0f006edb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.897+0000 7fe118000640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe11015f540 0x7fe1100b9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.897+0000 7fe118000640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1100b5ab0 0x7fe110152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.897+0000 7fe118000640 1 --2- 172.21.15.40:0/1469709386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1100b3ff0 0x7fe1100ce8d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:00:59.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.897+0000 7fe118000640 1 -- 172.21.15.40:0/1469709386 >> 172.21.15.40:0/1469709386 conn(0x7fe1100d4f90 msgr2=0x7fe1100cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:00:59.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.897+0000 7fe118000640 1 -- 172.21.15.40:0/1469709386 shutdown_connections 2024-04-26T23:00:59.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:00:59.897+0000 7fe118000640 1 -- 172.21.15.40:0/1469709386 wait complete. 2024-04-26T23:00:59.916 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:00:59.916 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:01:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.132+0000 7f1f6d400640 1 Processor -- start 2024-04-26T23:01:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.132+0000 7f1f6d400640 1 -- start start 2024-04-26T23:01:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.134+0000 7f1f6d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1f6815bc70 0x7f1f680cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.134+0000 7f1f6d400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f6806dde0 con 0x7f1f68071ef0 2024-04-26T23:01:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.134+0000 7f1f6d400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f6806e0c0 con 0x7f1f680730a0 2024-04-26T23:01:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.134+0000 7f1f6d400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1f6806e3a0 con 0x7f1f6815bc70 2024-04-26T23:01:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.134+0000 7f1f67e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1f6815bc70 0x7f1f680cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.134+0000 7f1f67e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1f6815bc70 0x7f1f680cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55458/0 (socket says 172.21.15.40:55458) 2024-04-26T23:01:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.134+0000 7f1f67e00640 1 -- 172.21.15.40:0/1533596797 learned_addr learned my addr 172.21.15.40:0/1533596797 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.134+0000 7f1f5fe00640 1 --1- 172.21.15.40:0/1533596797 >> v1:172.21.15.40:6789/0 conn(0x7f1f68071ef0 0x7f1f68072990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52616/0 (socket says 172.21.15.40:52616) 2024-04-26T23:01:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.135+0000 7f1f67e00640 1 -- 172.21.15.40:0/1533596797 >> v1:172.21.15.103:6789/0 conn(0x7f1f680730a0 legacy=0x7f1f680b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.135+0000 7f1f67e00640 1 -- 172.21.15.40:0/1533596797 >> v1:172.21.15.40:6789/0 conn(0x7f1f68071ef0 legacy=0x7f1f68072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.135+0000 7f1f67e00640 1 -- 172.21.15.40:0/1533596797 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f680b44d0 con 0x7f1f6815bc70 2024-04-26T23:01:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.136+0000 7f1f67e00640 1 --2- 172.21.15.40:0/1533596797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f6815bc70 0x7f1f680cdf30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009870 tx=0x7f1f5802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e52484246c309d3b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.136+0000 7f1f67400640 1 -- 172.21.15.40:0/1533596797 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58037070 con 0x7f1f6815bc70 2024-04-26T23:01:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.136+0000 7f1f67400640 1 -- 172.21.15.40:0/1533596797 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f58032440 con 0x7f1f6815bc70 2024-04-26T23:01:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.136+0000 7f1f67400640 1 -- 172.21.15.40:0/1533596797 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58032810 con 0x7f1f6815bc70 2024-04-26T23:01:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.137+0000 7f1f6d400640 1 -- 172.21.15.40:0/1533596797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f6815bc70 msgr2=0x7f1f680cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.137+0000 7f1f6d400640 1 --2- 172.21.15.40:0/1533596797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f6815bc70 0x7f1f680cdf30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009870 tx=0x7f1f5802a2a0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.137+0000 7f1f6d400640 1 -- 172.21.15.40:0/1533596797 shutdown_connections 2024-04-26T23:01:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.137+0000 7f1f6d400640 1 --2- 172.21.15.40:0/1533596797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f6815bc70 0x7f1f680cdf30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.137+0000 7f1f6d400640 1 -- 172.21.15.40:0/1533596797 >> 172.21.15.40:0/1533596797 conn(0x7f1f680b2720 msgr2=0x7f1f680b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:00.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.137+0000 7f1f6d400640 1 -- 172.21.15.40:0/1533596797 shutdown_connections 2024-04-26T23:01:00.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.137+0000 7f1f6d400640 1 -- 172.21.15.40:0/1533596797 wait complete. 2024-04-26T23:01:00.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.139+0000 7f1f6d400640 1 Processor -- start 2024-04-26T23:01:00.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.139+0000 7f1f6d400640 1 -- start start 2024-04-26T23:01:00.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.140+0000 7f1f6d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f68071ef0 0x7f1f6814c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:00.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.140+0000 7f1f6d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f680730a0 0x7f1f68157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:00.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.140+0000 7f1f6d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f6815bc70 0x7f1f68158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:00.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.140+0000 7f1f6d400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6815c050 con 0x7f1f6815bc70 2024-04-26T23:01:00.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.140+0000 7f1f6d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1f680b4840 con 0x7f1f68071ef0 2024-04-26T23:01:00.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.140+0000 7f1f6d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1f6815c1d0 con 0x7f1f680730a0 2024-04-26T23:01:00.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.140+0000 7f1f6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f6815bc70 0x7f1f68158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:00.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.140+0000 7f1f6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f6815bc70 0x7f1f68158310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55470/0 (socket says 172.21.15.40:55470) 2024-04-26T23:01:00.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.140+0000 7f1f6ca00640 1 -- 172.21.15.40:0/2917147695 learned_addr learned my addr 172.21.15.40:0/2917147695 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:00.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.141+0000 7f1f67e00640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f68071ef0 0x7f1f6814c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:00.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.141+0000 7f1f5fe00640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f680730a0 0x7f1f68157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:00.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.141+0000 7f1f6ca00640 1 -- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f680730a0 msgr2=0x7f1f68157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:00.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.141+0000 7f1f6ca00640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f680730a0 0x7f1f68157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.141+0000 7f1f6ca00640 1 -- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f68071ef0 msgr2=0x7f1f6814c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:00.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.141+0000 7f1f6ca00640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f68071ef0 0x7f1f6814c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.141+0000 7f1f6ca00640 1 -- 172.21.15.40:0/2917147695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f681f7540 con 0x7f1f6815bc70 2024-04-26T23:01:00.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.141+0000 7f1f6ca00640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f6815bc70 0x7f1f68158310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000c700 tx=0x7f1f6000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:00.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.141+0000 7f1f64c00640 1 -- 172.21.15.40:0/2917147695 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60016020 con 0x7f1f6815bc70 2024-04-26T23:01:00.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.141+0000 7f1f64c00640 1 -- 172.21.15.40:0/2917147695 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f6000aa50 con 0x7f1f6815bc70 2024-04-26T23:01:00.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.141+0000 7f1f64c00640 1 -- 172.21.15.40:0/2917147695 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6000ae40 con 0x7f1f6815bc70 2024-04-26T23:01:00.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.142+0000 7f1f6d400640 1 -- 172.21.15.40:0/2917147695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f6815ac10 con 0x7f1f6815bc70 2024-04-26T23:01:00.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.142+0000 7f1f6d400640 1 -- 172.21.15.40:0/2917147695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f681f6a60 con 0x7f1f6815bc70 2024-04-26T23:01:00.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.143+0000 7f1f6d400640 1 -- 172.21.15.40:0/2917147695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f68069370 con 0x7f1f6815bc70 2024-04-26T23:01:00.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.144+0000 7f1f64c00640 1 -- 172.21.15.40:0/2917147695 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1f60019020 con 0x7f1f6815bc70 2024-04-26T23:01:00.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.145+0000 7f1f64c00640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1f4406c8f0 0x7f1f4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:00.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.145+0000 7f1f64c00640 1 -- 172.21.15.40:0/2917147695 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f1f6008b820 con 0x7f1f6815bc70 2024-04-26T23:01:00.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.146+0000 7f1f67e00640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1f4406c8f0 0x7f1f4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:00.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.147+0000 7f1f67e00640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1f4406c8f0 0x7f1f4406edb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1f58034000 tx=0x7f1f5802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:00.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.150+0000 7f1f64c00640 1 -- 172.21.15.40:0/2917147695 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1f60020030 con 0x7f1f6815bc70 2024-04-26T23:01:00.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.355+0000 7f1f6d400640 1 -- 172.21.15.40:0/2917147695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1f680d0e80 con 0x7f1f6815bc70 2024-04-26T23:01:00.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.356+0000 7f1f64c00640 1 -- 172.21.15.40:0/2917147695 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f1f60059f00 con 0x7f1f6815bc70 2024-04-26T23:01:00.356 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:01:00.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.359+0000 7f1f6d400640 1 -- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1f4406c8f0 msgr2=0x7f1f4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:00.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.359+0000 7f1f6d400640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1f4406c8f0 0x7f1f4406edb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1f58034000 tx=0x7f1f5802e000 comp rx=0 tx=0).stop 2024-04-26T23:01:00.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.359+0000 7f1f6d400640 1 -- 172.21.15.40:0/2917147695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f6815bc70 msgr2=0x7f1f68158310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:00.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.359+0000 7f1f6d400640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f6815bc70 0x7f1f68158310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000c700 tx=0x7f1f6000cbc0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.359+0000 7f1f6d400640 1 -- 172.21.15.40:0/2917147695 shutdown_connections 2024-04-26T23:01:00.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.359+0000 7f1f6d400640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1f4406c8f0 0x7f1f4406edb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.359+0000 7f1f6d400640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f6815bc70 0x7f1f68158310 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.359+0000 7f1f6d400640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f680730a0 0x7f1f68157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.359+0000 7f1f6d400640 1 --2- 172.21.15.40:0/2917147695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f68071ef0 0x7f1f6814c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.359+0000 7f1f6d400640 1 -- 172.21.15.40:0/2917147695 >> 172.21.15.40:0/2917147695 conn(0x7f1f680b2720 msgr2=0x7f1f68156560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:00.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.360+0000 7f1f6d400640 1 -- 172.21.15.40:0/2917147695 shutdown_connections 2024-04-26T23:01:00.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.360+0000 7f1f6d400640 1 -- 172.21.15.40:0/2917147695 wait complete. 2024-04-26T23:01:00.379 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:01:00.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.588+0000 7fd3e2a00640 1 Processor -- start 2024-04-26T23:01:00.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.589+0000 7fd3e2a00640 1 -- start start 2024-04-26T23:01:00.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.590+0000 7fd3e2a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd3dc15ce60 0x7fd3dc0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:00.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.590+0000 7fd3e2a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd3dc0af7a0 0x7fd3dc0ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:00.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.590+0000 7fd3e2a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3dc0d0820 con 0x7fd3dc071fe0 2024-04-26T23:01:00.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.590+0000 7fd3e2a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fd3dc0cc450 con 0x7fd3dc15ce60 2024-04-26T23:01:00.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.590+0000 7fd3e2a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd3dc1614c0 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.591+0000 7fd3e1600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd3dc15ce60 0x7fd3dc0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:00.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.591+0000 7fd3e2000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd3dc071fe0 0x7fd3dc073760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52626/0 (socket says 172.21.15.40:52626) 2024-04-26T23:01:00.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.591+0000 7fd3e2000640 1 -- 172.21.15.40:0/655398080 learned_addr learned my addr 172.21.15.40:0/655398080 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:00.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.591+0000 7fd3e0c00640 1 --2- 172.21.15.40:0/655398080 >> v2:172.21.15.103:3300/0 conn(0x7fd3dc0af7a0 0x7fd3dc0ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.591+0000 7fd3dbe00640 1 -- 172.21.15.40:0/655398080 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 126381089 0 0) 0x7fd3dc0d0820 con 0x7fd3dc071fe0 2024-04-26T23:01:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.591+0000 7fd3dbe00640 1 -- 172.21.15.40:0/655398080 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3c4003520 con 0x7fd3dc071fe0 2024-04-26T23:01:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.591+0000 7fd3e1600640 1 -- 172.21.15.40:0/655398080 >> v2:172.21.15.103:3300/0 conn(0x7fd3dc0af7a0 msgr2=0x7fd3dc0ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.591+0000 7fd3e1600640 1 --2- 172.21.15.40:0/655398080 >> v2:172.21.15.103:3300/0 conn(0x7fd3dc0af7a0 0x7fd3dc0ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.591+0000 7fd3e1600640 1 -- 172.21.15.40:0/655398080 >> v1:172.21.15.40:6789/0 conn(0x7fd3dc071fe0 legacy=0x7fd3dc073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.591+0000 7fd3e1600640 1 -- 172.21.15.40:0/655398080 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3dc161760 con 0x7fd3dc15ce60 2024-04-26T23:01:00.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.593+0000 7fd3e1600640 1 --2- 172.21.15.40:0/655398080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc15ce60 0x7fd3dc0afc60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0d1940 tx=0x7fd3d002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7820a3d36d789c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:00.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.593+0000 7fd3dbe00640 1 -- 172.21.15.40:0/655398080 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0037070 con 0x7fd3dc15ce60 2024-04-26T23:01:00.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.593+0000 7fd3dbe00640 1 -- 172.21.15.40:0/655398080 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3d0032440 con 0x7fd3dc15ce60 2024-04-26T23:01:00.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.593+0000 7fd3dbe00640 1 -- 172.21.15.40:0/655398080 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d002ae60 con 0x7fd3dc15ce60 2024-04-26T23:01:00.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.594+0000 7fd3e2a00640 1 -- 172.21.15.40:0/655398080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc15ce60 msgr2=0x7fd3dc0afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:00.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.594+0000 7fd3e2a00640 1 --2- 172.21.15.40:0/655398080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc15ce60 0x7fd3dc0afc60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0d1940 tx=0x7fd3d002a110 comp rx=0 tx=0).stop 2024-04-26T23:01:00.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.594+0000 7fd3e2a00640 1 -- 172.21.15.40:0/655398080 shutdown_connections 2024-04-26T23:01:00.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.594+0000 7fd3e2a00640 1 --2- 172.21.15.40:0/655398080 >> v2:172.21.15.103:3300/0 conn(0x7fd3dc0af7a0 0x7fd3dc0ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.594+0000 7fd3e2a00640 1 --2- 172.21.15.40:0/655398080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc15ce60 0x7fd3dc0afc60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.594+0000 7fd3e2a00640 1 -- 172.21.15.40:0/655398080 >> 172.21.15.40:0/655398080 conn(0x7fd3dc0d0a10 msgr2=0x7fd3dc0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:00.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.594+0000 7fd3e2a00640 1 -- 172.21.15.40:0/655398080 shutdown_connections 2024-04-26T23:01:00.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.594+0000 7fd3e2a00640 1 -- 172.21.15.40:0/655398080 wait complete. 2024-04-26T23:01:00.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.596+0000 7fd3e2a00640 1 Processor -- start 2024-04-26T23:01:00.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.596+0000 7fd3e2a00640 1 -- start start 2024-04-26T23:01:00.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.597+0000 7fd3e2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3dc071fe0 0x7fd3dc153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:00.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.597+0000 7fd3e2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0af7a0 0x7fd3dc15dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:00.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.597+0000 7fd3e2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3dc15ce60 0x7fd3dc1f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:00.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.597+0000 7fd3e2a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc073e70 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.597+0000 7fd3e2a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc0cf530 con 0x7fd3dc071fe0 2024-04-26T23:01:00.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.597+0000 7fd3e2a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd3dc073ff0 con 0x7fd3dc15ce60 2024-04-26T23:01:00.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.597+0000 7fd3e0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0af7a0 0x7fd3dc15dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:00.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.597+0000 7fd3e0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0af7a0 0x7fd3dc15dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55492/0 (socket says 172.21.15.40:55492) 2024-04-26T23:01:00.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.597+0000 7fd3e0c00640 1 -- 172.21.15.40:0/1869973432 learned_addr learned my addr 172.21.15.40:0/1869973432 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:00.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.598+0000 7fd3e0c00640 1 -- 172.21.15.40:0/1869973432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3dc15ce60 msgr2=0x7fd3dc1f5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:00.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.598+0000 7fd3e0c00640 1 --2- 172.21.15.40:0/1869973432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3dc15ce60 0x7fd3dc1f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.598+0000 7fd3e0c00640 1 -- 172.21.15.40:0/1869973432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3dc071fe0 msgr2=0x7fd3dc153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:00.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.598+0000 7fd3e1600640 1 --2- 172.21.15.40:0/1869973432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3dc071fe0 0x7fd3dc153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:00.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.598+0000 7fd3e0c00640 1 --2- 172.21.15.40:0/1869973432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3dc071fe0 0x7fd3dc153880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.598+0000 7fd3e0c00640 1 -- 172.21.15.40:0/1869973432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3dc0c7e50 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.598+0000 7fd3e0c00640 1 --2- 172.21.15.40:0/1869973432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0af7a0 0x7fd3dc15dc60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800d800 tx=0x7fd3c800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:00.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.598+0000 7fd3d9600640 1 -- 172.21.15.40:0/1869973432 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c800b6c0 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.598+0000 7fd3d9600640 1 -- 172.21.15.40:0/1869973432 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3c8019070 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.598+0000 7fd3d9600640 1 -- 172.21.15.40:0/1869973432 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c800ba80 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.598+0000 7fd3e2a00640 1 -- 172.21.15.40:0/1869973432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3dc0c8130 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.598+0000 7fd3e2a00640 1 -- 172.21.15.40:0/1869973432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3dc0c86f0 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.600+0000 7fd3e2a00640 1 -- 172.21.15.40:0/1869973432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3dc069370 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.601+0000 7fd3d9600640 1 -- 172.21.15.40:0/1869973432 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd3c800bc20 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.602+0000 7fd3d9600640 1 --2- 172.21.15.40:0/1869973432 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd3bc06c8f0 0x7fd3bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.602+0000 7fd3d9600640 1 -- 172.21.15.40:0/1869973432 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fd3c808bcf0 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.602+0000 7fd3e1600640 1 --2- 172.21.15.40:0/1869973432 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd3bc06c8f0 0x7fd3bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:00.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.603+0000 7fd3e1600640 1 --2- 172.21.15.40:0/1869973432 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd3bc06c8f0 0x7fd3bc06edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0034000 tx=0x7fd3d002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:00.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.606+0000 7fd3d9600640 1 -- 172.21.15.40:0/1869973432 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd3c8090070 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.858+0000 7fd3e2a00640 1 -- 172.21.15.40:0/1869973432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3dc0af000 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.859+0000 7fd3d9600640 1 -- 172.21.15.40:0/1869973432 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3870 (secure 0 0 0) 0x7fd3c801c360 con 0x7fd3dc0af7a0 2024-04-26T23:01:00.860 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:00.860 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1253648754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1253648754},{"type":"v1","addr":"172.21.15.103:6836","nonce":1253648754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3239652611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":3239652611},{"type":"v1","addr":"172.21.15.103:6837","nonce":3239652611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3670144900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3670144900},{"type":"v1","addr":"172.21.15.40:6833","nonce":3670144900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/3989190340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3989190340},{"type":"v1","addr":"172.21.15.40:6835","nonce":3989190340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T23:01:00.860 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 3 2024-04-26T23:01:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.862+0000 7fd3e2a00640 1 -- 172.21.15.40:0/1869973432 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd3bc06c8f0 msgr2=0x7fd3bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.862+0000 7fd3e2a00640 1 --2- 172.21.15.40:0/1869973432 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd3bc06c8f0 0x7fd3bc06edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0034000 tx=0x7fd3d002e000 comp rx=0 tx=0).stop 2024-04-26T23:01:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.862+0000 7fd3e2a00640 1 -- 172.21.15.40:0/1869973432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0af7a0 msgr2=0x7fd3dc15dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.862+0000 7fd3e2a00640 1 --2- 172.21.15.40:0/1869973432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0af7a0 0x7fd3dc15dc60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800d800 tx=0x7fd3c800dcc0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.862+0000 7fd3e2a00640 1 -- 172.21.15.40:0/1869973432 shutdown_connections 2024-04-26T23:01:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.862+0000 7fd3e2a00640 1 --2- 172.21.15.40:0/1869973432 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd3bc06c8f0 0x7fd3bc06edb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.862+0000 7fd3e2a00640 1 --2- 172.21.15.40:0/1869973432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3dc15ce60 0x7fd3dc1f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.862+0000 7fd3e2a00640 1 --2- 172.21.15.40:0/1869973432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0af7a0 0x7fd3dc15dc60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.862+0000 7fd3e2a00640 1 --2- 172.21.15.40:0/1869973432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3dc071fe0 0x7fd3dc153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.862+0000 7fd3e2a00640 1 -- 172.21.15.40:0/1869973432 >> 172.21.15.40:0/1869973432 conn(0x7fd3dc0d0a10 msgr2=0x7fd3dc1574b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.863+0000 7fd3e2a00640 1 -- 172.21.15.40:0/1869973432 shutdown_connections 2024-04-26T23:01:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:00.863+0000 7fd3e2a00640 1 -- 172.21.15.40:0/1869973432 wait complete. 2024-04-26T23:01:00.882 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:01.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.096+0000 7f9c83800640 1 Processor -- start 2024-04-26T23:01:01.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.096+0000 7f9c83800640 1 -- start start 2024-04-26T23:01:01.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.097+0000 7f9c83800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c7c0cc480 0x7f9c7c072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.097+0000 7f9c83800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9c7c072780 0x7f9c7c166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.098+0000 7f9c83800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c7c15d7c0 con 0x7f9c7c071e60 2024-04-26T23:01:01.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.098+0000 7f9c83800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f9c7c15daa0 con 0x7f9c7c072780 2024-04-26T23:01:01.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.098+0000 7f9c83800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9c7c15dd80 con 0x7f9c7c0cc480 2024-04-26T23:01:01.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.098+0000 7f9c80e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9c7c071e60 0x7f9c7c06f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52636/0 (socket says 172.21.15.40:52636) 2024-04-26T23:01:01.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.098+0000 7f9c80e00640 1 -- 172.21.15.40:0/328897292 learned_addr learned my addr 172.21.15.40:0/328897292 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:01.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.098+0000 7f9c81800640 1 --2- 172.21.15.40:0/328897292 >> v2:172.21.15.40:3300/0 conn(0x7f9c7c072780 0x7f9c7c166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.098+0000 7f9c7be00640 1 --2- 172.21.15.40:0/328897292 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c7c0cc480 0x7f9c7c072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.098+0000 7f9c7b400640 1 -- 172.21.15.40:0/328897292 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2055022538 0 0) 0x7f9c7c15d7c0 con 0x7f9c7c071e60 2024-04-26T23:01:01.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.098+0000 7f9c7b400640 1 -- 172.21.15.40:0/328897292 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c64003520 con 0x7f9c7c071e60 2024-04-26T23:01:01.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.098+0000 7f9c81800640 1 -- 172.21.15.40:0/328897292 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c7c0cc480 msgr2=0x7f9c7c072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:01.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.098+0000 7f9c81800640 1 --2- 172.21.15.40:0/328897292 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c7c0cc480 0x7f9c7c072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.098+0000 7f9c81800640 1 -- 172.21.15.40:0/328897292 >> v1:172.21.15.40:6789/0 conn(0x7f9c7c071e60 legacy=0x7f9c7c06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.098+0000 7f9c81800640 1 -- 172.21.15.40:0/328897292 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c7c15e060 con 0x7f9c7c072780 2024-04-26T23:01:01.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.099+0000 7f9c81800640 1 --2- 172.21.15.40:0/328897292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7c072780 0x7f9c7c166ad0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9c74009d90 tx=0x7f9c7402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d71f7485485f4a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:01.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.100+0000 7f9c7b400640 1 -- 172.21.15.40:0/328897292 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c74007740 con 0x7f9c7c072780 2024-04-26T23:01:01.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.100+0000 7f9c7b400640 1 -- 172.21.15.40:0/328897292 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c740078e0 con 0x7f9c7c072780 2024-04-26T23:01:01.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.100+0000 7f9c7b400640 1 -- 172.21.15.40:0/328897292 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c74007cb0 con 0x7f9c7c072780 2024-04-26T23:01:01.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.100+0000 7f9c83800640 1 -- 172.21.15.40:0/328897292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7c072780 msgr2=0x7f9c7c166ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.100+0000 7f9c83800640 1 --2- 172.21.15.40:0/328897292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7c072780 0x7f9c7c166ad0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9c74009d90 tx=0x7f9c7402a660 comp rx=0 tx=0).stop 2024-04-26T23:01:01.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.101+0000 7f9c83800640 1 -- 172.21.15.40:0/328897292 shutdown_connections 2024-04-26T23:01:01.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.101+0000 7f9c83800640 1 --2- 172.21.15.40:0/328897292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7c072780 0x7f9c7c166ad0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.101+0000 7f9c83800640 1 --2- 172.21.15.40:0/328897292 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c7c0cc480 0x7f9c7c072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.101+0000 7f9c83800640 1 -- 172.21.15.40:0/328897292 >> 172.21.15.40:0/328897292 conn(0x7f9c7c0d0be0 msgr2=0x7f9c7c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:01.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.101+0000 7f9c83800640 1 -- 172.21.15.40:0/328897292 shutdown_connections 2024-04-26T23:01:01.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.101+0000 7f9c83800640 1 -- 172.21.15.40:0/328897292 wait complete. 2024-04-26T23:01:01.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.102+0000 7f9c83800640 1 Processor -- start 2024-04-26T23:01:01.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.103+0000 7f9c83800640 1 -- start start 2024-04-26T23:01:01.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.103+0000 7f9c83800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c7c071e60 0x7f9c7c152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.103+0000 7f9c83800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7c072780 0x7f9c7c0c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.103+0000 7f9c83800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c7c0cc480 0x7f9c7c1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.103+0000 7f9c83800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7c15d3f0 con 0x7f9c7c072780 2024-04-26T23:01:01.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.103+0000 7f9c83800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7c072c10 con 0x7f9c7c071e60 2024-04-26T23:01:01.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.103+0000 7f9c83800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9c7c15d570 con 0x7f9c7c0cc480 2024-04-26T23:01:01.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.104+0000 7f9c7be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7c072780 0x7f9c7c0c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.104+0000 7f9c7be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7c072780 0x7f9c7c0c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55518/0 (socket says 172.21.15.40:55518) 2024-04-26T23:01:01.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.104+0000 7f9c7be00640 1 -- 172.21.15.40:0/1625864151 learned_addr learned my addr 172.21.15.40:0/1625864151 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:01.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.104+0000 7f9c81800640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c7c0cc480 0x7f9c7c1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.104+0000 7f9c80e00640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c7c071e60 0x7f9c7c152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.104+0000 7f9c7be00640 1 -- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c7c0cc480 msgr2=0x7f9c7c1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.104+0000 7f9c7be00640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c7c0cc480 0x7f9c7c1539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.104+0000 7f9c7be00640 1 -- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c7c071e60 msgr2=0x7f9c7c152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.104+0000 7f9c7be00640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c7c071e60 0x7f9c7c152fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.104+0000 7f9c7be00640 1 -- 172.21.15.40:0/1625864151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c7c0c8f10 con 0x7f9c7c072780 2024-04-26T23:01:01.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.104+0000 7f9c80e00640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c7c071e60 0x7f9c7c152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:01.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.105+0000 7f9c7be00640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7c072780 0x7f9c7c0c9fb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9c6800c7f0 tx=0x7f9c6800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:01.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.105+0000 7f9c78c00640 1 -- 172.21.15.40:0/1625864151 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c68015070 con 0x7f9c7c072780 2024-04-26T23:01:01.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.105+0000 7f9c78c00640 1 -- 172.21.15.40:0/1625864151 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c68004220 con 0x7f9c7c072780 2024-04-26T23:01:01.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.105+0000 7f9c78c00640 1 -- 172.21.15.40:0/1625864151 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c68007620 con 0x7f9c7c072780 2024-04-26T23:01:01.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.105+0000 7f9c83800640 1 -- 172.21.15.40:0/1625864151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c7c0c91f0 con 0x7f9c7c072780 2024-04-26T23:01:01.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.105+0000 7f9c83800640 1 -- 172.21.15.40:0/1625864151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c7c0c97b0 con 0x7f9c7c072780 2024-04-26T23:01:01.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.106+0000 7f9c83800640 1 -- 172.21.15.40:0/1625864151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c7c0694f0 con 0x7f9c7c072780 2024-04-26T23:01:01.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.107+0000 7f9c78c00640 1 -- 172.21.15.40:0/1625864151 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9c680079d0 con 0x7f9c7c072780 2024-04-26T23:01:01.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.108+0000 7f9c78c00640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9c5c06c8f0 0x7f9c5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.108+0000 7f9c78c00640 1 -- 172.21.15.40:0/1625864151 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f9c6808bb30 con 0x7f9c7c072780 2024-04-26T23:01:01.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.109+0000 7f9c80e00640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9c5c06c8f0 0x7f9c5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.110+0000 7f9c80e00640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9c5c06c8f0 0x7f9c5c06edb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9c70005880 tx=0x7f9c70009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:01.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.113+0000 7f9c78c00640 1 -- 172.21.15.40:0/1625864151 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9c6805a340 con 0x7f9c7c072780 2024-04-26T23:01:01.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.324+0000 7f9c83800640 1 -- 172.21.15.40:0/1625864151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c7c069e30 con 0x7f9c7c072780 2024-04-26T23:01:01.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.326+0000 7f9c78c00640 1 -- 172.21.15.40:0/1625864151 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12848 (secure 0 0 0) 0x7f9c68059c40 con 0x7f9c7c072780 2024-04-26T23:01:01.327 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:01.327 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":16,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:00:55.462707+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.329+0000 7f9c83800640 1 -- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9c5c06c8f0 msgr2=0x7f9c5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.329+0000 7f9c83800640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9c5c06c8f0 0x7f9c5c06edb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9c70005880 tx=0x7f9c70009040 comp rx=0 tx=0).stop 2024-04-26T23:01:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.329+0000 7f9c83800640 1 -- 172.21.15.40:0/1625864151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7c072780 msgr2=0x7f9c7c0c9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.329+0000 7f9c83800640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7c072780 0x7f9c7c0c9fb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9c6800c7f0 tx=0x7f9c6800ccb0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.329+0000 7f9c83800640 1 -- 172.21.15.40:0/1625864151 shutdown_connections 2024-04-26T23:01:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.329+0000 7f9c83800640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9c5c06c8f0 0x7f9c5c06edb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.329+0000 7f9c83800640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c7c0cc480 0x7f9c7c1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.329+0000 7f9c83800640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7c072780 0x7f9c7c0c9fb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.329+0000 7f9c83800640 1 --2- 172.21.15.40:0/1625864151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c7c071e60 0x7f9c7c152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.329+0000 7f9c83800640 1 -- 172.21.15.40:0/1625864151 >> 172.21.15.40:0/1625864151 conn(0x7f9c7c0d0be0 msgr2=0x7f9c7c072e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.329+0000 7f9c83800640 1 -- 172.21.15.40:0/1625864151 shutdown_connections 2024-04-26T23:01:01.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.330+0000 7f9c83800640 1 -- 172.21.15.40:0/1625864151 wait complete. 2024-04-26T23:01:01.349 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:01:01.350 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:01:01.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.546+0000 7ff20a400640 1 Processor -- start 2024-04-26T23:01:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.546+0000 7ff20a400640 1 -- start start 2024-04-26T23:01:01.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.548+0000 7ff20a400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff204073600 0x7ff204072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.548+0000 7ff20a400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2041604e0 con 0x7ff20415fe80 2024-04-26T23:01:01.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.548+0000 7ff20a400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff204072200 con 0x7ff2040739e0 2024-04-26T23:01:01.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.548+0000 7ff20a400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff204073220 con 0x7ff204073600 2024-04-26T23:01:01.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.548+0000 7ff209000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff204073600 0x7ff204072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.548+0000 7ff209000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff204073600 0x7ff204072ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55526/0 (socket says 172.21.15.40:55526) 2024-04-26T23:01:01.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.548+0000 7ff209a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff20415fe80 0x7ff2040ae8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52644/0 (socket says 172.21.15.40:52644) 2024-04-26T23:01:01.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.548+0000 7ff209000640 1 -- 172.21.15.40:0/2980935759 learned_addr learned my addr 172.21.15.40:0/2980935759 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:01.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.549+0000 7ff209000640 1 -- 172.21.15.40:0/2980935759 >> v1:172.21.15.103:6789/0 conn(0x7ff2040739e0 legacy=0x7ff20415f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.549+0000 7ff209000640 1 -- 172.21.15.40:0/2980935759 >> v1:172.21.15.40:6789/0 conn(0x7ff20415fe80 legacy=0x7ff2040ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.549+0000 7ff209000640 1 -- 172.21.15.40:0/2980935759 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2040724e0 con 0x7ff204073600 2024-04-26T23:01:01.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.549+0000 7ff203400640 1 -- 172.21.15.40:0/2980935759 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 357764454 0 0) 0x7ff2041604e0 con 0x7ff20415fe80 2024-04-26T23:01:01.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.550+0000 7ff209000640 1 --2- 172.21.15.40:0/2980935759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204073600 0x7ff204072ce0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8009870 tx=0x7ff1f802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157cae2483275bbe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:01.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.550+0000 7ff203400640 1 -- 172.21.15.40:0/2980935759 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8037070 con 0x7ff204073600 2024-04-26T23:01:01.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.550+0000 7ff203400640 1 -- 172.21.15.40:0/2980935759 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1f802ae50 con 0x7ff204073600 2024-04-26T23:01:01.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.550+0000 7ff203400640 1 -- 172.21.15.40:0/2980935759 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8032860 con 0x7ff204073600 2024-04-26T23:01:01.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.551+0000 7ff20a400640 1 -- 172.21.15.40:0/2980935759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204073600 msgr2=0x7ff204072ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.551+0000 7ff20a400640 1 --2- 172.21.15.40:0/2980935759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204073600 0x7ff204072ce0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8009870 tx=0x7ff1f802a300 comp rx=0 tx=0).stop 2024-04-26T23:01:01.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.551+0000 7ff20a400640 1 -- 172.21.15.40:0/2980935759 shutdown_connections 2024-04-26T23:01:01.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.551+0000 7ff20a400640 1 --2- 172.21.15.40:0/2980935759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204073600 0x7ff204072ce0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.551+0000 7ff20a400640 1 -- 172.21.15.40:0/2980935759 >> 172.21.15.40:0/2980935759 conn(0x7ff2040b2720 msgr2=0x7ff2040b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:01.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.551+0000 7ff20a400640 1 -- 172.21.15.40:0/2980935759 shutdown_connections 2024-04-26T23:01:01.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.551+0000 7ff20a400640 1 -- 172.21.15.40:0/2980935759 wait complete. 2024-04-26T23:01:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.553+0000 7ff20a400640 1 Processor -- start 2024-04-26T23:01:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.553+0000 7ff20a400640 1 -- start start 2024-04-26T23:01:01.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.554+0000 7ff20a400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff204073600 0x7ff204152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.554+0000 7ff20a400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2040739e0 0x7ff204155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.554+0000 7ff20a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff20415fe80 0x7ff2041510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.554+0000 7ff20a400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff204160290 con 0x7ff20415fe80 2024-04-26T23:01:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.554+0000 7ff20a400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff204072850 con 0x7ff2040739e0 2024-04-26T23:01:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.554+0000 7ff20a400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff204071e20 con 0x7ff204073600 2024-04-26T23:01:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.554+0000 7ff209a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff20415fe80 0x7ff2041510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.554+0000 7ff209a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff20415fe80 0x7ff2041510c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55542/0 (socket says 172.21.15.40:55542) 2024-04-26T23:01:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.554+0000 7ff209a00640 1 -- 172.21.15.40:0/2813702606 learned_addr learned my addr 172.21.15.40:0/2813702606 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:01.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.555+0000 7ff209000640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff204073600 0x7ff204152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.555+0000 7ff203e00640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2040739e0 0x7ff204155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.555+0000 7ff209a00640 1 -- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff204073600 msgr2=0x7ff204152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.555+0000 7ff209a00640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff204073600 0x7ff204152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.555+0000 7ff209a00640 1 -- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2040739e0 msgr2=0x7ff204155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.555+0000 7ff209a00640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2040739e0 0x7ff204155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.555+0000 7ff209a00640 1 -- 172.21.15.40:0/2813702606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff204151600 con 0x7ff20415fe80 2024-04-26T23:01:01.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.555+0000 7ff209000640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff204073600 0x7ff204152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:01.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.555+0000 7ff203e00640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2040739e0 0x7ff204155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:01.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.555+0000 7ff209a00640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff20415fe80 0x7ff2041510c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00c730 tx=0x7ff1fc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:01.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.555+0000 7ff200c00640 1 -- 172.21.15.40:0/2813702606 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc018070 con 0x7ff20415fe80 2024-04-26T23:01:01.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.555+0000 7ff200c00640 1 -- 172.21.15.40:0/2813702606 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1fc00ce40 con 0x7ff20415fe80 2024-04-26T23:01:01.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.556+0000 7ff200c00640 1 -- 172.21.15.40:0/2813702606 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc00a860 con 0x7ff20415fe80 2024-04-26T23:01:01.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.556+0000 7ff20a400640 1 -- 172.21.15.40:0/2813702606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff204160660 con 0x7ff20415fe80 2024-04-26T23:01:01.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.556+0000 7ff20a400640 1 -- 172.21.15.40:0/2813702606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff204160c00 con 0x7ff20415fe80 2024-04-26T23:01:01.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.557+0000 7ff20a400640 1 -- 172.21.15.40:0/2813702606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff204069370 con 0x7ff20415fe80 2024-04-26T23:01:01.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.558+0000 7ff200c00640 1 -- 172.21.15.40:0/2813702606 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff1fc00aa00 con 0x7ff20415fe80 2024-04-26T23:01:01.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.559+0000 7ff200c00640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff1e406c920 0x7ff1e406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.559+0000 7ff200c00640 1 -- 172.21.15.40:0/2813702606 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7ff1fc08acd0 con 0x7ff20415fe80 2024-04-26T23:01:01.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.559+0000 7ff209000640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff1e406c920 0x7ff1e406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.561+0000 7ff209000640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff1e406c920 0x7ff1e406ede0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8034000 tx=0x7ff1f802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:01.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.563+0000 7ff200c00640 1 -- 172.21.15.40:0/2813702606 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff204151600 con 0x7ff20415fe80 2024-04-26T23:01:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.762+0000 7ff20a400640 1 -- 172.21.15.40:0/2813702606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff2040b3480 con 0x7ff20415fe80 2024-04-26T23:01:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.762+0000 7ff200c00640 1 -- 172.21.15.40:0/2813702606 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7ff1fc059440 con 0x7ff20415fe80 2024-04-26T23:01:01.763 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:01:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.765+0000 7ff20a400640 1 -- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff1e406c920 msgr2=0x7ff1e406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.765+0000 7ff20a400640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff1e406c920 0x7ff1e406ede0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8034000 tx=0x7ff1f802e000 comp rx=0 tx=0).stop 2024-04-26T23:01:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.765+0000 7ff20a400640 1 -- 172.21.15.40:0/2813702606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff20415fe80 msgr2=0x7ff2041510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.765+0000 7ff20a400640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff20415fe80 0x7ff2041510c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00c730 tx=0x7ff1fc00cbf0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.765+0000 7ff20a400640 1 -- 172.21.15.40:0/2813702606 shutdown_connections 2024-04-26T23:01:01.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.765+0000 7ff20a400640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff1e406c920 0x7ff1e406ede0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.765+0000 7ff20a400640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff20415fe80 0x7ff2041510c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.765+0000 7ff20a400640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2040739e0 0x7ff204155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.765+0000 7ff20a400640 1 --2- 172.21.15.40:0/2813702606 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff204073600 0x7ff204152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.765+0000 7ff20a400640 1 -- 172.21.15.40:0/2813702606 >> 172.21.15.40:0/2813702606 conn(0x7ff2040b2720 msgr2=0x7ff2040b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:01.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.765+0000 7ff20a400640 1 -- 172.21.15.40:0/2813702606 shutdown_connections 2024-04-26T23:01:01.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.765+0000 7ff20a400640 1 -- 172.21.15.40:0/2813702606 wait complete. 2024-04-26T23:01:01.783 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T23:01:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.989+0000 7f5622c00640 1 Processor -- start 2024-04-26T23:01:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.989+0000 7f5622c00640 1 -- start start 2024-04-26T23:01:01.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.990+0000 7f5622c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f561c073e10 0x7f561c1600d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.990+0000 7f5622c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f561c15c160 0x7f561c0b4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.990+0000 7f5622c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f561c15c5f0 0x7f561c0b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.990+0000 7f5622c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f561c153040 con 0x7f561c073e10 2024-04-26T23:01:01.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.990+0000 7f5622c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f561c15fc40 con 0x7f561c15c160 2024-04-26T23:01:01.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.990+0000 7f5622c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f561c160610 con 0x7f561c15c5f0 2024-04-26T23:01:01.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.991+0000 7f5621800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f561c073e10 0x7f561c1600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.991+0000 7f5621800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f561c073e10 0x7f561c1600d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55556/0 (socket says 172.21.15.40:55556) 2024-04-26T23:01:01.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.991+0000 7f5621800640 1 -- 172.21.15.40:0/59758895 learned_addr learned my addr 172.21.15.40:0/59758895 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:01.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.991+0000 7f5622200640 1 --2- 172.21.15.40:0/59758895 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f561c15c5f0 0x7f561c0b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.991+0000 7f5620e00640 1 --2- 172.21.15.40:0/59758895 >> v2:172.21.15.103:3300/0 conn(0x7f561c15c160 0x7f561c0b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.991+0000 7f5621800640 1 -- 172.21.15.40:0/59758895 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f561c15c5f0 msgr2=0x7f561c0b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:01.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.991+0000 7f5621800640 1 --2- 172.21.15.40:0/59758895 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f561c15c5f0 0x7f561c0b4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.991+0000 7f5621800640 1 -- 172.21.15.40:0/59758895 >> v2:172.21.15.103:3300/0 conn(0x7f561c15c160 msgr2=0x7f561c0b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:01.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.991+0000 7f5621800640 1 --2- 172.21.15.40:0/59758895 >> v2:172.21.15.103:3300/0 conn(0x7f561c15c160 0x7f561c0b4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.991+0000 7f5621800640 1 -- 172.21.15.40:0/59758895 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561c15cb90 con 0x7f561c073e10 2024-04-26T23:01:01.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.992+0000 7f5621800640 1 --2- 172.21.15.40:0/59758895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c073e10 0x7f561c1600d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f561c0b6140 tx=0x7f561002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6452fd29f78b5b4b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:01.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.993+0000 7f561be00640 1 -- 172.21.15.40:0/59758895 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5610037070 con 0x7f561c073e10 2024-04-26T23:01:01.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.993+0000 7f561be00640 1 -- 172.21.15.40:0/59758895 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5610032440 con 0x7f561c073e10 2024-04-26T23:01:01.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.993+0000 7f561be00640 1 -- 172.21.15.40:0/59758895 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5610032810 con 0x7f561c073e10 2024-04-26T23:01:01.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.993+0000 7f5622c00640 1 -- 172.21.15.40:0/59758895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c073e10 msgr2=0x7f561c1600d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.993+0000 7f5622c00640 1 --2- 172.21.15.40:0/59758895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c073e10 0x7f561c1600d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f561c0b6140 tx=0x7f561002a1e0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.994+0000 7f5622c00640 1 -- 172.21.15.40:0/59758895 shutdown_connections 2024-04-26T23:01:01.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.994+0000 7f5622c00640 1 --2- 172.21.15.40:0/59758895 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f561c15c5f0 0x7f561c0b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.994+0000 7f5622c00640 1 --2- 172.21.15.40:0/59758895 >> v2:172.21.15.103:3300/0 conn(0x7f561c15c160 0x7f561c0b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.994+0000 7f5622c00640 1 --2- 172.21.15.40:0/59758895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c073e10 0x7f561c1600d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.994+0000 7f5622c00640 1 -- 172.21.15.40:0/59758895 >> 172.21.15.40:0/59758895 conn(0x7f561c075eb0 msgr2=0x7f561c068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:01.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.994+0000 7f5622c00640 1 -- 172.21.15.40:0/59758895 shutdown_connections 2024-04-26T23:01:01.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.994+0000 7f5622c00640 1 -- 172.21.15.40:0/59758895 wait complete. 2024-04-26T23:01:01.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.995+0000 7f5622c00640 1 Processor -- start 2024-04-26T23:01:01.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.996+0000 7f5622c00640 1 -- start start 2024-04-26T23:01:01.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.996+0000 7f5622c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f561c073e10 0x7f561c142680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.996+0000 7f5622c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f561c15c160 0x7f561c146320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.996+0000 7f5622c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c15c5f0 0x7f561c146860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:01.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.996+0000 7f5622c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f561c0733c0 con 0x7f561c15c5f0 2024-04-26T23:01:01.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.997+0000 7f5622c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f561c15f8d0 con 0x7f561c15c160 2024-04-26T23:01:01.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.997+0000 7f5622c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f561c073540 con 0x7f561c073e10 2024-04-26T23:01:01.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.997+0000 7f5622200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c15c5f0 0x7f561c146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:01.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.997+0000 7f5622200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c15c5f0 0x7f561c146860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55558/0 (socket says 172.21.15.40:55558) 2024-04-26T23:01:01.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.997+0000 7f5622200640 1 -- 172.21.15.40:0/1142503499 learned_addr learned my addr 172.21.15.40:0/1142503499 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:01.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.997+0000 7f5622200640 1 -- 172.21.15.40:0/1142503499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f561c073e10 msgr2=0x7f561c142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.997+0000 7f5622200640 1 --2- 172.21.15.40:0/1142503499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f561c073e10 0x7f561c142680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.997+0000 7f5622200640 1 -- 172.21.15.40:0/1142503499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f561c15c160 msgr2=0x7f561c146320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:01.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.997+0000 7f5622200640 1 --2- 172.21.15.40:0/1142503499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f561c15c160 0x7f561c146320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:01.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.997+0000 7f5622200640 1 -- 172.21.15.40:0/1142503499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561c155220 con 0x7f561c15c5f0 2024-04-26T23:01:01.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.997+0000 7f5622200640 1 --2- 172.21.15.40:0/1142503499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c15c5f0 0x7f561c146860 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5614007e60 tx=0x7f561400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:01.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.998+0000 7f5619600640 1 -- 172.21.15.40:0/1142503499 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614014420 con 0x7f561c15c5f0 2024-04-26T23:01:01.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.998+0000 7f5619600640 1 -- 172.21.15.40:0/1142503499 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5614019070 con 0x7f561c15c5f0 2024-04-26T23:01:01.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.998+0000 7f5619600640 1 -- 172.21.15.40:0/1142503499 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614014800 con 0x7f561c15c5f0 2024-04-26T23:01:01.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.998+0000 7f5622c00640 1 -- 172.21.15.40:0/1142503499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f561c142bc0 con 0x7f561c15c5f0 2024-04-26T23:01:01.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.998+0000 7f5622c00640 1 -- 172.21.15.40:0/1142503499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f561c154740 con 0x7f561c15c5f0 2024-04-26T23:01:01.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:01.999+0000 7f5622c00640 1 -- 172.21.15.40:0/1142503499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f561c069370 con 0x7f561c15c5f0 2024-04-26T23:01:02.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.000+0000 7f5619600640 1 -- 172.21.15.40:0/1142503499 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f56140149a0 con 0x7f561c15c5f0 2024-04-26T23:01:02.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.001+0000 7f5619600640 1 --2- 172.21.15.40:0/1142503499 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f55fc06c920 0x7f55fc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:02.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.001+0000 7f5619600640 1 -- 172.21.15.40:0/1142503499 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f561408c0e0 con 0x7f561c15c5f0 2024-04-26T23:01:02.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.002+0000 7f5621800640 1 --2- 172.21.15.40:0/1142503499 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f55fc06c920 0x7f55fc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:02.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.003+0000 7f5621800640 1 --2- 172.21.15.40:0/1142503499 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f55fc06c920 0x7f55fc06ede0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5610034000 tx=0x7f561002c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:02.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.005+0000 7f5619600640 1 -- 172.21.15.40:0/1142503499 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f561405a7c0 con 0x7f561c15c5f0 2024-04-26T23:01:02.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.211+0000 7f5622c00640 1 -- 172.21.15.40:0/1142503499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f561c06f4e0 con 0x7f561c15c5f0 2024-04-26T23:01:02.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.212+0000 7f5619600640 1 -- 172.21.15.40:0/1142503499 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f561405a0c0 con 0x7f561c15c5f0 2024-04-26T23:01:02.212 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:02.212 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-04-26T23:01:02.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:01:02.212 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-04-26T23:01:02.212 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-04-26T23:01:02.213 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:01:02.213 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-04-26T23:01:02.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.214+0000 7f5622c00640 1 -- 172.21.15.40:0/1142503499 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f55fc06c920 msgr2=0x7f55fc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:02.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.215+0000 7f5622c00640 1 --2- 172.21.15.40:0/1142503499 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f55fc06c920 0x7f55fc06ede0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5610034000 tx=0x7f561002c000 comp rx=0 tx=0).stop 2024-04-26T23:01:02.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.215+0000 7f5622c00640 1 -- 172.21.15.40:0/1142503499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c15c5f0 msgr2=0x7f561c146860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:02.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.215+0000 7f5622c00640 1 --2- 172.21.15.40:0/1142503499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c15c5f0 0x7f561c146860 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5614007e60 tx=0x7f561400e4b0 comp rx=0 tx=0).stop 2024-04-26T23:01:02.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.215+0000 7f5622c00640 1 -- 172.21.15.40:0/1142503499 shutdown_connections 2024-04-26T23:01:02.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.215+0000 7f5622c00640 1 --2- 172.21.15.40:0/1142503499 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f55fc06c920 0x7f55fc06ede0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:02.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.215+0000 7f5622c00640 1 --2- 172.21.15.40:0/1142503499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c15c5f0 0x7f561c146860 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:02.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.215+0000 7f5622c00640 1 --2- 172.21.15.40:0/1142503499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f561c15c160 0x7f561c146320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:02.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.215+0000 7f5622c00640 1 --2- 172.21.15.40:0/1142503499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f561c073e10 0x7f561c142680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:02.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.215+0000 7f5622c00640 1 -- 172.21.15.40:0/1142503499 >> 172.21.15.40:0/1142503499 conn(0x7f561c075eb0 msgr2=0x7f561c0b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:02.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.215+0000 7f5622c00640 1 -- 172.21.15.40:0/1142503499 shutdown_connections 2024-04-26T23:01:02.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.215+0000 7f5622c00640 1 -- 172.21.15.40:0/1142503499 wait complete. 2024-04-26T23:01:02.233 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T23:01:02.233 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T23:01:02.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.450+0000 7fc9b7e00640 1 Processor -- start 2024-04-26T23:01:02.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.450+0000 7fc9b7e00640 1 -- start start 2024-04-26T23:01:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc9b00b3df0 0x7fc9b0164d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9b01652b0 0x7fc9b00b5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b7e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b00b4280 con 0x7fc9b014d090 2024-04-26T23:01:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b7e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc9b00b4560 con 0x7fc9b01652b0 2024-04-26T23:01:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc9b00b4840 con 0x7fc9b00b3df0 2024-04-26T23:01:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b5e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9b01652b0 0x7fc9b00b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b5e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9b01652b0 0x7fc9b00b5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55566/0 (socket says 172.21.15.40:55566) 2024-04-26T23:01:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b5e00640 1 -- 172.21.15.40:0/4156492078 learned_addr learned my addr 172.21.15.40:0/4156492078 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b5400640 1 --1- 172.21.15.40:0/4156492078 >> v1:172.21.15.40:6789/0 conn(0x7fc9b014d090 0x7fc9b0164620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52658/0 (socket says 172.21.15.40:52658) 2024-04-26T23:01:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b4a00640 1 --2- 172.21.15.40:0/4156492078 >> v2:172.21.15.103:3300/0 conn(0x7fc9b00b3df0 0x7fc9b0164d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b5e00640 1 -- 172.21.15.40:0/4156492078 >> v2:172.21.15.103:3300/0 conn(0x7fc9b00b3df0 msgr2=0x7fc9b0164d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b5e00640 1 --2- 172.21.15.40:0/4156492078 >> v2:172.21.15.103:3300/0 conn(0x7fc9b00b3df0 0x7fc9b0164d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b5e00640 1 -- 172.21.15.40:0/4156492078 >> v1:172.21.15.40:6789/0 conn(0x7fc9b014d090 legacy=0x7fc9b0164620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.452+0000 7fc9b5e00640 1 -- 172.21.15.40:0/4156492078 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b00b4b20 con 0x7fc9b01652b0 2024-04-26T23:01:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.454+0000 7fc9b5e00640 1 --2- 172.21.15.40:0/4156492078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b01652b0 0x7fc9b00b5320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0167100 tx=0x7fc9a802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00380762bc34c70 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.454+0000 7fc9afe00640 1 -- 172.21.15.40:0/4156492078 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8036070 con 0x7fc9b01652b0 2024-04-26T23:01:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.454+0000 7fc9afe00640 1 -- 172.21.15.40:0/4156492078 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9a8031600 con 0x7fc9b01652b0 2024-04-26T23:01:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.454+0000 7fc9afe00640 1 -- 172.21.15.40:0/4156492078 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a80319d0 con 0x7fc9b01652b0 2024-04-26T23:01:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.454+0000 7fc9b7e00640 1 -- 172.21.15.40:0/4156492078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b01652b0 msgr2=0x7fc9b00b5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.454+0000 7fc9b7e00640 1 --2- 172.21.15.40:0/4156492078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b01652b0 0x7fc9b00b5320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0167100 tx=0x7fc9a802a730 comp rx=0 tx=0).stop 2024-04-26T23:01:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.455+0000 7fc9b7e00640 1 -- 172.21.15.40:0/4156492078 shutdown_connections 2024-04-26T23:01:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.455+0000 7fc9b7e00640 1 --2- 172.21.15.40:0/4156492078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b01652b0 0x7fc9b00b5320 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.455+0000 7fc9b7e00640 1 --2- 172.21.15.40:0/4156492078 >> v2:172.21.15.103:3300/0 conn(0x7fc9b00b3df0 0x7fc9b0164d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:02.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.455+0000 7fc9b7e00640 1 -- 172.21.15.40:0/4156492078 >> 172.21.15.40:0/4156492078 conn(0x7fc9b006f310 msgr2=0x7fc9b006f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.455+0000 7fc9b7e00640 1 -- 172.21.15.40:0/4156492078 shutdown_connections 2024-04-26T23:01:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.455+0000 7fc9b7e00640 1 -- 172.21.15.40:0/4156492078 wait complete. 2024-04-26T23:01:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.456+0000 7fc9b7e00640 1 Processor -- start 2024-04-26T23:01:02.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.457+0000 7fc9b7e00640 1 -- start start 2024-04-26T23:01:02.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9b014d090 0x7fc9b00cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:02.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9b01652b0 0x7fc9b00c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b00c9120 0x7fc9b00af0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b00b59e0 con 0x7fc9b00c9120 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b00b5860 con 0x7fc9b01652b0 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc9b00b5b60 con 0x7fc9b014d090 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b00c9120 0x7fc9b00af0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b00c9120 0x7fc9b00af0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55574/0 (socket says 172.21.15.40:55574) 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b5e00640 1 -- 172.21.15.40:0/3813655857 learned_addr learned my addr 172.21.15.40:0/3813655857 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b5400640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9b014d090 0x7fc9b00cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b4a00640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9b01652b0 0x7fc9b00c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b5e00640 1 -- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9b014d090 msgr2=0x7fc9b00cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b5e00640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9b014d090 0x7fc9b00cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b5e00640 1 -- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9b01652b0 msgr2=0x7fc9b00c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.458+0000 7fc9b5e00640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9b01652b0 0x7fc9b00c8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.459+0000 7fc9b5e00640 1 -- 172.21.15.40:0/3813655857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b00afd90 con 0x7fc9b00c9120 2024-04-26T23:01:02.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.459+0000 7fc9b5e00640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b00c9120 0x7fc9b00af0a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009f40 tx=0x7fc9a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:02.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.459+0000 7fc9ad600640 1 -- 172.21.15.40:0/3813655857 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8036050 con 0x7fc9b00c9120 2024-04-26T23:01:02.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.459+0000 7fc9ad600640 1 -- 172.21.15.40:0/3813655857 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9a8042070 con 0x7fc9b00c9120 2024-04-26T23:01:02.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.459+0000 7fc9ad600640 1 -- 172.21.15.40:0/3813655857 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8009510 con 0x7fc9b00c9120 2024-04-26T23:01:02.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.459+0000 7fc9b7e00640 1 -- 172.21.15.40:0/3813655857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b00c6540 con 0x7fc9b00c9120 2024-04-26T23:01:02.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.459+0000 7fc9b7e00640 1 -- 172.21.15.40:0/3813655857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b00cd840 con 0x7fc9b00c9120 2024-04-26T23:01:02.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.461+0000 7fc9b7e00640 1 -- 172.21.15.40:0/3813655857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b0069600 con 0x7fc9b00c9120 2024-04-26T23:01:02.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.461+0000 7fc9ad600640 1 -- 172.21.15.40:0/3813655857 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc9a8007bf0 con 0x7fc9b00c9120 2024-04-26T23:01:02.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.462+0000 7fc9ad600640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc99006c920 0x7fc99006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:02.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.462+0000 7fc9ad600640 1 -- 172.21.15.40:0/3813655857 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fc9a80ac790 con 0x7fc9b00c9120 2024-04-26T23:01:02.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.463+0000 7fc9b5400640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc99006c920 0x7fc99006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:02.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.464+0000 7fc9b5400640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc99006c920 0x7fc99006ede0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4000f60 tx=0x7fc9a4007cc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:02.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.467+0000 7fc9ad600640 1 -- 172.21.15.40:0/3813655857 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc9a807ae70 con 0x7fc9b00c9120 2024-04-26T23:01:02.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:02.677+0000 7fc9b7e00640 1 -- 172.21.15.40:0/3813655857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc9b00cf2c0 con 0x7fc9b00c9120 2024-04-26T23:01:03.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.670+0000 7fc9ad600640 1 -- 172.21.15.40:0/3813655857 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7fc9a807a770 con 0x7fc9b00c9120 2024-04-26T23:01:03.671 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-04-26T23:01:03.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.674+0000 7fc9b7e00640 1 -- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc99006c920 msgr2=0x7fc99006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:03.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.674+0000 7fc9b7e00640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc99006c920 0x7fc99006ede0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4000f60 tx=0x7fc9a4007cc0 comp rx=0 tx=0).stop 2024-04-26T23:01:03.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.674+0000 7fc9b7e00640 1 -- 172.21.15.40:0/3813655857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b00c9120 msgr2=0x7fc9b00af0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:03.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.674+0000 7fc9b7e00640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b00c9120 0x7fc9b00af0a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009f40 tx=0x7fc9a8004a40 comp rx=0 tx=0).stop 2024-04-26T23:01:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.675+0000 7fc9b7e00640 1 -- 172.21.15.40:0/3813655857 shutdown_connections 2024-04-26T23:01:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.675+0000 7fc9b7e00640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc99006c920 0x7fc99006ede0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.675+0000 7fc9b7e00640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b00c9120 0x7fc9b00af0a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.675+0000 7fc9b7e00640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9b01652b0 0x7fc9b00c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.675+0000 7fc9b7e00640 1 --2- 172.21.15.40:0/3813655857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9b014d090 0x7fc9b00cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.675+0000 7fc9b7e00640 1 -- 172.21.15.40:0/3813655857 >> 172.21.15.40:0/3813655857 conn(0x7fc9b006f310 msgr2=0x7fc9b006f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.675+0000 7fc9b7e00640 1 -- 172.21.15.40:0/3813655857 shutdown_connections 2024-04-26T23:01:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.675+0000 7fc9b7e00640 1 -- 172.21.15.40:0/3813655857 wait complete. 2024-04-26T23:01:03.706 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-26T23:01:03.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.904+0000 7ff29e400640 1 Processor -- start 2024-04-26T23:01:03.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.905+0000 7ff29e400640 1 -- start start 2024-04-26T23:01:03.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.906+0000 7ff29e400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff29815f4f0 0x7ff2980d34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:03.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.906+0000 7ff29e400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2980d3e10 con 0x7ff29815f110 2024-04-26T23:01:03.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.906+0000 7ff29e400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2980b7df0 con 0x7ff2980d39e0 2024-04-26T23:01:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.906+0000 7ff29e400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff2980b80d0 con 0x7ff29815f4f0 2024-04-26T23:01:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.907+0000 7ff297e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff29815f4f0 0x7ff2980d34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.907+0000 7ff297e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff29815f4f0 0x7ff2980d34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55588/0 (socket says 172.21.15.40:55588) 2024-04-26T23:01:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.907+0000 7ff297e00640 1 -- 172.21.15.40:0/3572144344 learned_addr learned my addr 172.21.15.40:0/3572144344 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.907+0000 7ff297e00640 1 -- 172.21.15.40:0/3572144344 >> v1:172.21.15.103:6789/0 conn(0x7ff2980d39e0 legacy=0x7ff2980b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.907+0000 7ff297e00640 1 -- 172.21.15.40:0/3572144344 >> v1:172.21.15.40:6789/0 conn(0x7ff29815f110 legacy=0x7ff2980d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.907+0000 7ff297e00640 1 -- 172.21.15.40:0/3572144344 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2980b83b0 con 0x7ff29815f4f0 2024-04-26T23:01:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.908+0000 7ff297e00640 1 --2- 172.21.15.40:0/3572144344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff29815f4f0 0x7ff2980d34a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff280009870 tx=0x7ff28002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6735a00946668fb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.909+0000 7ff297400640 1 -- 172.21.15.40:0/3572144344 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff280037070 con 0x7ff29815f4f0 2024-04-26T23:01:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.909+0000 7ff297400640 1 -- 172.21.15.40:0/3572144344 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff280032440 con 0x7ff29815f4f0 2024-04-26T23:01:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.909+0000 7ff297400640 1 -- 172.21.15.40:0/3572144344 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff280032810 con 0x7ff29815f4f0 2024-04-26T23:01:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.909+0000 7ff29e400640 1 -- 172.21.15.40:0/3572144344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff29815f4f0 msgr2=0x7ff2980d34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.909+0000 7ff29e400640 1 --2- 172.21.15.40:0/3572144344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff29815f4f0 0x7ff2980d34a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff280009870 tx=0x7ff28002a2a0 comp rx=0 tx=0).stop 2024-04-26T23:01:03.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.909+0000 7ff29e400640 1 -- 172.21.15.40:0/3572144344 shutdown_connections 2024-04-26T23:01:03.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.909+0000 7ff29e400640 1 --2- 172.21.15.40:0/3572144344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff29815f4f0 0x7ff2980d34a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:03.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.910+0000 7ff29e400640 1 -- 172.21.15.40:0/3572144344 >> 172.21.15.40:0/3572144344 conn(0x7ff29809e210 msgr2=0x7ff2980d4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:03.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.910+0000 7ff29e400640 1 -- 172.21.15.40:0/3572144344 shutdown_connections 2024-04-26T23:01:03.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.910+0000 7ff29e400640 1 -- 172.21.15.40:0/3572144344 wait complete. 2024-04-26T23:01:03.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.911+0000 7ff29e400640 1 Processor -- start 2024-04-26T23:01:03.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.912+0000 7ff29e400640 1 -- start start 2024-04-26T23:01:03.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.912+0000 7ff29e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2980d39e0 0x7ff2980cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:03.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.912+0000 7ff29e400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff29815f110 0x7ff29815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:03.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff29e400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff29815f4f0 0x7ff2980a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:03.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff29e400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff2980d2fa0 con 0x7ff2980d39e0 2024-04-26T23:01:03.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff29e400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff2980b8970 con 0x7ff29815f110 2024-04-26T23:01:03.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff29e400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff2980d0c30 con 0x7ff29815f4f0 2024-04-26T23:01:03.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff29d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2980d39e0 0x7ff2980cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:03.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff29d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2980d39e0 0x7ff2980cb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55594/0 (socket says 172.21.15.40:55594) 2024-04-26T23:01:03.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff29d000640 1 -- 172.21.15.40:0/4097445498 learned_addr learned my addr 172.21.15.40:0/4097445498 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:03.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff297e00640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff29815f110 0x7ff29815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:03.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff29da00640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff29815f4f0 0x7ff2980a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff29d000640 1 -- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff29815f4f0 msgr2=0x7ff2980a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff29d000640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff29815f4f0 0x7ff2980a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff29d000640 1 -- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff29815f110 msgr2=0x7ff29815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff29d000640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff29815f110 0x7ff29815bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.913+0000 7ff29d000640 1 -- 172.21.15.40:0/4097445498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2980c9650 con 0x7ff2980d39e0 2024-04-26T23:01:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.914+0000 7ff29d000640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2980d39e0 0x7ff2980cb4f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff28800c790 tx=0x7ff28800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.914+0000 7ff294c00640 1 -- 172.21.15.40:0/4097445498 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288016020 con 0x7ff2980d39e0 2024-04-26T23:01:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.914+0000 7ff294c00640 1 -- 172.21.15.40:0/4097445498 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff288006d50 con 0x7ff2980d39e0 2024-04-26T23:01:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.914+0000 7ff294c00640 1 -- 172.21.15.40:0/4097445498 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288003bb0 con 0x7ff2980d39e0 2024-04-26T23:01:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.914+0000 7ff29e400640 1 -- 172.21.15.40:0/4097445498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2980cba30 con 0x7ff2980d39e0 2024-04-26T23:01:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.914+0000 7ff29e400640 1 -- 172.21.15.40:0/4097445498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2980c9bd0 con 0x7ff2980d39e0 2024-04-26T23:01:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.916+0000 7ff29e400640 1 -- 172.21.15.40:0/4097445498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff298069370 con 0x7ff2980d39e0 2024-04-26T23:01:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.917+0000 7ff294c00640 1 -- 172.21.15.40:0/4097445498 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff28800a850 con 0x7ff2980d39e0 2024-04-26T23:01:03.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.917+0000 7ff294c00640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff27406c8f0 0x7ff27406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:03.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.918+0000 7ff294c00640 1 -- 172.21.15.40:0/4097445498 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7ff28808b630 con 0x7ff2980d39e0 2024-04-26T23:01:03.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.918+0000 7ff297e00640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff27406c8f0 0x7ff27406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:03.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.919+0000 7ff297e00640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff27406c8f0 0x7ff27406edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff280034000 tx=0x7ff28002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:03.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:03.922+0000 7ff294c00640 1 -- 172.21.15.40:0/4097445498 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff288020020 con 0x7ff2980d39e0 2024-04-26T23:01:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.128+0000 7ff29e400640 1 -- 172.21.15.40:0/4097445498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7ff2980d5470 con 0x7ff2980d39e0 2024-04-26T23:01:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.686+0000 7ff294c00640 1 -- 172.21.15.40:0/4097445498 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) ==== 171+0+0 (secure 0 0 0) 0x7ff2980d3e10 con 0x7ff2980d39e0 2024-04-26T23:01:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-04-26T23:01:04.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.690+0000 7ff29e400640 1 -- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff27406c8f0 msgr2=0x7ff27406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:04.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.690+0000 7ff29e400640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff27406c8f0 0x7ff27406edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff280034000 tx=0x7ff28002e000 comp rx=0 tx=0).stop 2024-04-26T23:01:04.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.690+0000 7ff29e400640 1 -- 172.21.15.40:0/4097445498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2980d39e0 msgr2=0x7ff2980cb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:04.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.690+0000 7ff29e400640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2980d39e0 0x7ff2980cb4f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff28800c790 tx=0x7ff28800cc50 comp rx=0 tx=0).stop 2024-04-26T23:01:04.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.690+0000 7ff29e400640 1 -- 172.21.15.40:0/4097445498 shutdown_connections 2024-04-26T23:01:04.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.690+0000 7ff29e400640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff27406c8f0 0x7ff27406edb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:04.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.690+0000 7ff29e400640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff29815f4f0 0x7ff2980a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:04.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.690+0000 7ff29e400640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff29815f110 0x7ff29815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:04.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.690+0000 7ff29e400640 1 --2- 172.21.15.40:0/4097445498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2980d39e0 0x7ff2980cb4f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:04.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.690+0000 7ff29e400640 1 -- 172.21.15.40:0/4097445498 >> 172.21.15.40:0/4097445498 conn(0x7ff29809e210 msgr2=0x7ff2980b72c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:04.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.691+0000 7ff29e400640 1 -- 172.21.15.40:0/4097445498 shutdown_connections 2024-04-26T23:01:04.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.691+0000 7ff29e400640 1 -- 172.21.15.40:0/4097445498 wait complete. 2024-04-26T23:01:04.725 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T23:01:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.905+0000 7fa527200640 1 Processor -- start 2024-04-26T23:01:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.906+0000 7fa527200640 1 -- start start 2024-04-26T23:01:04.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.907+0000 7fa527200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa52815f060 0x7fa52815f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:04.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.907+0000 7fa527200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5280d3640 0x7fa5280d3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:04.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.907+0000 7fa527200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5280d1860 0x7fa5280d1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:04.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.907+0000 7fa527200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa5280d5c20 con 0x7fa5280d3640 2024-04-26T23:01:04.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.907+0000 7fa527200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa5280cfc40 con 0x7fa52815f060 2024-04-26T23:01:04.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.907+0000 7fa527200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa5280cff20 con 0x7fa5280d1860 2024-04-26T23:01:04.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.908+0000 7fa525400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5280d3640 0x7fa5280d3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.908+0000 7fa525400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5280d3640 0x7fa5280d3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45236/0 (socket says 172.21.15.40:45236) 2024-04-26T23:01:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.908+0000 7fa525400640 1 -- 172.21.15.40:0/529133086 learned_addr learned my addr 172.21.15.40:0/529133086 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.908+0000 7fa526800640 1 --2- 172.21.15.40:0/529133086 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5280d1860 0x7fa5280d1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.908+0000 7fa525e00640 1 --2- 172.21.15.40:0/529133086 >> v2:172.21.15.103:3300/0 conn(0x7fa52815f060 0x7fa52815f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.908+0000 7fa525400640 1 -- 172.21.15.40:0/529133086 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5280d1860 msgr2=0x7fa5280d1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.908+0000 7fa525400640 1 --2- 172.21.15.40:0/529133086 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5280d1860 0x7fa5280d1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.908+0000 7fa525400640 1 -- 172.21.15.40:0/529133086 >> v2:172.21.15.103:3300/0 conn(0x7fa52815f060 msgr2=0x7fa52815f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.908+0000 7fa525400640 1 --2- 172.21.15.40:0/529133086 >> v2:172.21.15.103:3300/0 conn(0x7fa52815f060 0x7fa52815f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.908+0000 7fa525400640 1 -- 172.21.15.40:0/529133086 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5280d0200 con 0x7fa5280d3640 2024-04-26T23:01:04.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.909+0000 7fa525400640 1 --2- 172.21.15.40:0/529133086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280d3640 0x7fa5280d3a80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa52808d7b0 tx=0x7fa51002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac502e82d6fd38cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:04.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.910+0000 7fa524a00640 1 -- 172.21.15.40:0/529133086 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa510036070 con 0x7fa5280d3640 2024-04-26T23:01:04.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.910+0000 7fa524a00640 1 -- 172.21.15.40:0/529133086 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa510031600 con 0x7fa5280d3640 2024-04-26T23:01:04.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.910+0000 7fa524a00640 1 -- 172.21.15.40:0/529133086 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5100319d0 con 0x7fa5280d3640 2024-04-26T23:01:04.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.911+0000 7fa527200640 1 -- 172.21.15.40:0/529133086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280d3640 msgr2=0x7fa5280d3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:04.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.911+0000 7fa527200640 1 --2- 172.21.15.40:0/529133086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280d3640 0x7fa5280d3a80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa52808d7b0 tx=0x7fa51002a6a0 comp rx=0 tx=0).stop 2024-04-26T23:01:04.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.911+0000 7fa527200640 1 -- 172.21.15.40:0/529133086 shutdown_connections 2024-04-26T23:01:04.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.911+0000 7fa527200640 1 --2- 172.21.15.40:0/529133086 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5280d1860 0x7fa5280d1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:04.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.911+0000 7fa527200640 1 --2- 172.21.15.40:0/529133086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280d3640 0x7fa5280d3a80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:04.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.911+0000 7fa527200640 1 --2- 172.21.15.40:0/529133086 >> v2:172.21.15.103:3300/0 conn(0x7fa52815f060 0x7fa52815f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:04.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.911+0000 7fa527200640 1 -- 172.21.15.40:0/529133086 >> 172.21.15.40:0/529133086 conn(0x7fa5280d4e50 msgr2=0x7fa5280d5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:04.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.911+0000 7fa527200640 1 -- 172.21.15.40:0/529133086 shutdown_connections 2024-04-26T23:01:04.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.911+0000 7fa527200640 1 -- 172.21.15.40:0/529133086 wait complete. 2024-04-26T23:01:04.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.913+0000 7fa527200640 1 Processor -- start 2024-04-26T23:01:04.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.913+0000 7fa527200640 1 -- start start 2024-04-26T23:01:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa527200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5280d1860 0x7fa528161390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa527200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280d3640 0x7fa5280ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa527200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa52815f060 0x7fa5281540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa527200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa5280d0ce0 con 0x7fa5280d3640 2024-04-26T23:01:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa527200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa5280cab70 con 0x7fa52815f060 2024-04-26T23:01:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa527200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa5280ce030 con 0x7fa5280d1860 2024-04-26T23:01:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa525400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280d3640 0x7fa5280ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa525400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280d3640 0x7fa5280ccc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45250/0 (socket says 172.21.15.40:45250) 2024-04-26T23:01:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa525400640 1 -- 172.21.15.40:0/1378351583 learned_addr learned my addr 172.21.15.40:0/1378351583 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa525e00640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5280d1860 0x7fa528161390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa526800640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa52815f060 0x7fa5281540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:04.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa525400640 1 -- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5280d1860 msgr2=0x7fa528161390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:04.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa525400640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5280d1860 0x7fa528161390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:04.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa525400640 1 -- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa52815f060 msgr2=0x7fa5281540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:04.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa525400640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa52815f060 0x7fa5281540b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:04.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.914+0000 7fa525400640 1 -- 172.21.15.40:0/1378351583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5280cd180 con 0x7fa5280d3640 2024-04-26T23:01:04.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.915+0000 7fa525400640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280d3640 0x7fa5280ccc40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa510009f40 tx=0x7fa510004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:04.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.915+0000 7fa51e000640 1 -- 172.21.15.40:0/1378351583 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51003f020 con 0x7fa5280d3640 2024-04-26T23:01:04.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.915+0000 7fa51e000640 1 -- 172.21.15.40:0/1378351583 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa510037040 con 0x7fa5280d3640 2024-04-26T23:01:04.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.915+0000 7fa51e000640 1 -- 172.21.15.40:0/1378351583 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa510036070 con 0x7fa5280d3640 2024-04-26T23:01:04.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.915+0000 7fa527200640 1 -- 172.21.15.40:0/1378351583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5280cd420 con 0x7fa5280d3640 2024-04-26T23:01:04.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.915+0000 7fa527200640 1 -- 172.21.15.40:0/1378351583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5280cd9e0 con 0x7fa5280d3640 2024-04-26T23:01:04.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.916+0000 7fa527200640 1 -- 172.21.15.40:0/1378351583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa528069a80 con 0x7fa5280d3640 2024-04-26T23:01:04.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.918+0000 7fa51e000640 1 -- 172.21.15.40:0/1378351583 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa510042020 con 0x7fa5280d3640 2024-04-26T23:01:04.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.919+0000 7fa51e000640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa50406c940 0x7fa50406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:04.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.919+0000 7fa51e000640 1 -- 172.21.15.40:0/1378351583 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7fa5100abfe0 con 0x7fa5280d3640 2024-04-26T23:01:04.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.919+0000 7fa525e00640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa50406c940 0x7fa50406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:04.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.920+0000 7fa525e00640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa50406c940 0x7fa50406ee00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa518006fd0 tx=0x7fa518008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:04.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:04.924+0000 7fa51e000640 1 -- 172.21.15.40:0/1378351583 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa510049030 con 0x7fa5280d3640 2024-04-26T23:01:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.169+0000 7fa527200640 1 -- 172.21.15.40:0/1378351583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa528068e00 con 0x7fa5280d3640 2024-04-26T23:01:05.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.713+0000 7fa51e000640 1 -- 172.21.15.40:0/1378351583 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fa51007a4c0 con 0x7fa5280d3640 2024-04-26T23:01:05.714 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 2 and data pool 3 2024-04-26T23:01:05.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.719+0000 7fa527200640 1 -- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa50406c940 msgr2=0x7fa50406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:05.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.719+0000 7fa527200640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa50406c940 0x7fa50406ee00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa518006fd0 tx=0x7fa518008040 comp rx=0 tx=0).stop 2024-04-26T23:01:05.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.719+0000 7fa527200640 1 -- 172.21.15.40:0/1378351583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280d3640 msgr2=0x7fa5280ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:05.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.719+0000 7fa527200640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280d3640 0x7fa5280ccc40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa510009f40 tx=0x7fa510004a40 comp rx=0 tx=0).stop 2024-04-26T23:01:05.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.720+0000 7fa527200640 1 -- 172.21.15.40:0/1378351583 shutdown_connections 2024-04-26T23:01:05.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.720+0000 7fa527200640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa50406c940 0x7fa50406ee00 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa518006fd0 tx=0x7fa518008040 comp rx=0 tx=0).stop 2024-04-26T23:01:05.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.720+0000 7fa527200640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa52815f060 0x7fa5281540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:05.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.720+0000 7fa527200640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280d3640 0x7fa5280ccc40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:05.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.720+0000 7fa527200640 1 --2- 172.21.15.40:0/1378351583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5280d1860 0x7fa528161390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:05.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.720+0000 7fa527200640 1 -- 172.21.15.40:0/1378351583 >> 172.21.15.40:0/1378351583 conn(0x7fa5280d4e50 msgr2=0x7fa5280d52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:05.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.728+0000 7fa527200640 1 -- 172.21.15.40:0/1378351583 shutdown_connections 2024-04-26T23:01:05.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.729+0000 7fa527200640 1 -- 172.21.15.40:0/1378351583 wait complete. 2024-04-26T23:01:05.760 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-26T23:01:05.760 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-26T23:01:05.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.943+0000 7ffb68e00640 1 Processor -- start 2024-04-26T23:01:05.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.943+0000 7ffb68e00640 1 -- start start 2024-04-26T23:01:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.944+0000 7ffb68e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffb6415cf90 0x7ffb640d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.944+0000 7ffb68e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffb640ce370 0x7ffb640ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.944+0000 7ffb68e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb640d0b30 con 0x7ffb64160b80 2024-04-26T23:01:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.944+0000 7ffb68e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ffb640d08e0 con 0x7ffb6415cf90 2024-04-26T23:01:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.944+0000 7ffb68e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffb640d0e10 con 0x7ffb640ce370 2024-04-26T23:01:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.945+0000 7ffb63400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffb6415cf90 0x7ffb640d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.945+0000 7ffb63400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffb6415cf90 0x7ffb640d18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45260/0 (socket says 172.21.15.40:45260) 2024-04-26T23:01:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.945+0000 7ffb63400640 1 -- 172.21.15.40:0/4191607358 learned_addr learned my addr 172.21.15.40:0/4191607358 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.945+0000 7ffb63e00640 1 --1- 172.21.15.40:0/4191607358 >> v1:172.21.15.40:6789/0 conn(0x7ffb64160b80 0x7ffb64070050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51434/0 (socket says 172.21.15.40:51434) 2024-04-26T23:01:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.945+0000 7ffb62a00640 1 --2- 172.21.15.40:0/4191607358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffb640ce370 0x7ffb640ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.945+0000 7ffb63400640 1 -- 172.21.15.40:0/4191607358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffb640ce370 msgr2=0x7ffb640ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.945+0000 7ffb63400640 1 --2- 172.21.15.40:0/4191607358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffb640ce370 0x7ffb640ce7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.945+0000 7ffb63400640 1 -- 172.21.15.40:0/4191607358 >> v1:172.21.15.40:6789/0 conn(0x7ffb64160b80 legacy=0x7ffb64070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.945+0000 7ffb63400640 1 -- 172.21.15.40:0/4191607358 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb640d10f0 con 0x7ffb6415cf90 2024-04-26T23:01:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.946+0000 7ffb63400640 1 --2- 172.21.15.40:0/4191607358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6415cf90 0x7ffb640d18f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009e50 tx=0x7ffb5402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a89b15941b9d44 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.946+0000 7ffb62000640 1 -- 172.21.15.40:0/4191607358 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb54036070 con 0x7ffb6415cf90 2024-04-26T23:01:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.946+0000 7ffb62000640 1 -- 172.21.15.40:0/4191607358 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb54031600 con 0x7ffb6415cf90 2024-04-26T23:01:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.947+0000 7ffb62000640 1 -- 172.21.15.40:0/4191607358 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb540319d0 con 0x7ffb6415cf90 2024-04-26T23:01:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.947+0000 7ffb68e00640 1 -- 172.21.15.40:0/4191607358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6415cf90 msgr2=0x7ffb640d18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.947+0000 7ffb68e00640 1 --2- 172.21.15.40:0/4191607358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6415cf90 0x7ffb640d18f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009e50 tx=0x7ffb5402a680 comp rx=0 tx=0).stop 2024-04-26T23:01:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.947+0000 7ffb68e00640 1 -- 172.21.15.40:0/4191607358 shutdown_connections 2024-04-26T23:01:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.947+0000 7ffb68e00640 1 --2- 172.21.15.40:0/4191607358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffb640ce370 0x7ffb640ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.947+0000 7ffb68e00640 1 --2- 172.21.15.40:0/4191607358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6415cf90 0x7ffb640d18f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.947+0000 7ffb68e00640 1 -- 172.21.15.40:0/4191607358 >> 172.21.15.40:0/4191607358 conn(0x7ffb640b4860 msgr2=0x7ffb640b4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.947+0000 7ffb68e00640 1 -- 172.21.15.40:0/4191607358 shutdown_connections 2024-04-26T23:01:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.947+0000 7ffb68e00640 1 -- 172.21.15.40:0/4191607358 wait complete. 2024-04-26T23:01:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.949+0000 7ffb68e00640 1 Processor -- start 2024-04-26T23:01:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.949+0000 7ffb68e00640 1 -- start start 2024-04-26T23:01:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.950+0000 7ffb68e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb640ce370 0x7ffb640cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.950+0000 7ffb68e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6415cf90 0x7ffb64152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.950+0000 7ffb68e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffb64160b80 0x7ffb6406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.950+0000 7ffb68e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6415d370 con 0x7ffb6415cf90 2024-04-26T23:01:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.950+0000 7ffb68e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffb640b54f0 con 0x7ffb640ce370 2024-04-26T23:01:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.950+0000 7ffb68e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffb640cc2a0 con 0x7ffb64160b80 2024-04-26T23:01:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.950+0000 7ffb62a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6415cf90 0x7ffb64152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.950+0000 7ffb62a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6415cf90 0x7ffb64152110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45268/0 (socket says 172.21.15.40:45268) 2024-04-26T23:01:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.950+0000 7ffb62a00640 1 -- 172.21.15.40:0/2807002579 learned_addr learned my addr 172.21.15.40:0/2807002579 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.951+0000 7ffb63400640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb640ce370 0x7ffb640cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.951+0000 7ffb62a00640 1 -- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffb64160b80 msgr2=0x7ffb6406e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.951+0000 7ffb62a00640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffb64160b80 0x7ffb6406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.951+0000 7ffb62a00640 1 -- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb640ce370 msgr2=0x7ffb640cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.951+0000 7ffb62a00640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb640ce370 0x7ffb640cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.951+0000 7ffb62a00640 1 -- 172.21.15.40:0/2807002579 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb640b5950 con 0x7ffb6415cf90 2024-04-26T23:01:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.951+0000 7ffb63400640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb640ce370 0x7ffb640cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.951+0000 7ffb62a00640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6415cf90 0x7ffb64152110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00d830 tx=0x7ffb4c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.951+0000 7ffb5b400640 1 -- 172.21.15.40:0/2807002579 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c00b6c0 con 0x7ffb6415cf90 2024-04-26T23:01:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.951+0000 7ffb5b400640 1 -- 172.21.15.40:0/2807002579 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4c019070 con 0x7ffb6415cf90 2024-04-26T23:01:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.951+0000 7ffb68e00640 1 -- 172.21.15.40:0/2807002579 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb640b5c30 con 0x7ffb6415cf90 2024-04-26T23:01:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.951+0000 7ffb5b400640 1 -- 172.21.15.40:0/2807002579 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c00ba80 con 0x7ffb6415cf90 2024-04-26T23:01:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.952+0000 7ffb68e00640 1 -- 172.21.15.40:0/2807002579 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb640b61f0 con 0x7ffb6415cf90 2024-04-26T23:01:05.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.953+0000 7ffb68e00640 1 -- 172.21.15.40:0/2807002579 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb64069a80 con 0x7ffb6415cf90 2024-04-26T23:01:05.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.954+0000 7ffb5b400640 1 -- 172.21.15.40:0/2807002579 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffb4c01e460 con 0x7ffb6415cf90 2024-04-26T23:01:05.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.955+0000 7ffb5b400640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ffb4006c940 0x7ffb4006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:05.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.955+0000 7ffb5b400640 1 -- 172.21.15.40:0/2807002579 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7ffb4c08c1a0 con 0x7ffb6415cf90 2024-04-26T23:01:05.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.955+0000 7ffb63400640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ffb4006c940 0x7ffb4006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:05.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.956+0000 7ffb63400640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ffb4006c940 0x7ffb4006ee00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffb54033000 tx=0x7ffb54009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:05.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:05.959+0000 7ffb5b400640 1 -- 172.21.15.40:0/2807002579 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ffb4c05a640 con 0x7ffb6415cf90 2024-04-26T23:01:06.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.169+0000 7ffb68e00640 1 -- 172.21.15.40:0/2807002579 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7ffb640b8c90 con 0x7ffb6415cf90 2024-04-26T23:01:06.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.726+0000 7ffb5b400640 1 -- 172.21.15.40:0/2807002579 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) ==== 157+0+0 (secure 0 0 0) 0x7ffb4c059f40 con 0x7ffb6415cf90 2024-04-26T23:01:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.730+0000 7ffb68e00640 1 -- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ffb4006c940 msgr2=0x7ffb4006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.731+0000 7ffb68e00640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ffb4006c940 0x7ffb4006ee00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffb54033000 tx=0x7ffb54009000 comp rx=0 tx=0).stop 2024-04-26T23:01:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.731+0000 7ffb68e00640 1 -- 172.21.15.40:0/2807002579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6415cf90 msgr2=0x7ffb64152110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.731+0000 7ffb68e00640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6415cf90 0x7ffb64152110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00d830 tx=0x7ffb4c00dcf0 comp rx=0 tx=0).stop 2024-04-26T23:01:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.731+0000 7ffb68e00640 1 -- 172.21.15.40:0/2807002579 shutdown_connections 2024-04-26T23:01:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.731+0000 7ffb68e00640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ffb4006c940 0x7ffb4006ee00 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffb54033000 tx=0x7ffb54009000 comp rx=0 tx=0).stop 2024-04-26T23:01:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.731+0000 7ffb68e00640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffb64160b80 0x7ffb6406e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.731+0000 7ffb68e00640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6415cf90 0x7ffb64152110 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.731+0000 7ffb68e00640 1 --2- 172.21.15.40:0/2807002579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb640ce370 0x7ffb640cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.731+0000 7ffb68e00640 1 -- 172.21.15.40:0/2807002579 >> 172.21.15.40:0/2807002579 conn(0x7ffb640b4860 msgr2=0x7ffb64070840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.731+0000 7ffb68e00640 1 -- 172.21.15.40:0/2807002579 shutdown_connections 2024-04-26T23:01:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.731+0000 7ffb68e00640 1 -- 172.21.15.40:0/2807002579 wait complete. 2024-04-26T23:01:06.753 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T23:01:06.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.955+0000 7fd277200640 1 Processor -- start 2024-04-26T23:01:06.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.956+0000 7fd277200640 1 -- start start 2024-04-26T23:01:06.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.957+0000 7fd277200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd27806f9a0 0x7fd2780ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:06.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.957+0000 7fd277200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd278146a90 con 0x7fd2780d1370 2024-04-26T23:01:06.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.958+0000 7fd277200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd27806ec20 con 0x7fd278143550 2024-04-26T23:01:06.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.958+0000 7fd277200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd2780cef90 con 0x7fd27806f9a0 2024-04-26T23:01:06.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.958+0000 7fd26fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd2780d1370 0x7fd2780b48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51440/0 (socket says 172.21.15.40:51440) 2024-04-26T23:01:06.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.958+0000 7fd26fe00640 1 -- 172.21.15.40:0/3070205927 learned_addr learned my addr 172.21.15.40:0/3070205927 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:06.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.958+0000 7fd276800640 1 --2- 172.21.15.40:0/3070205927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd27806f9a0 0x7fd2780ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:06.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.958+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4173211642 0 0) 0x7fd260001110 con 0x7fd2780d1370 2024-04-26T23:01:06.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.958+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd258003520 con 0x7fd2780d1370 2024-04-26T23:01:06.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2896810448 0 0) 0x7fd27806ec20 con 0x7fd278143550 2024-04-26T23:01:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd260001110 con 0x7fd278143550 2024-04-26T23:01:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 509434218 0 0) 0x7fd258003520 con 0x7fd2780d1370 2024-04-26T23:01:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd27806f9a0 msgr2=0x7fd2780ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 --2- 172.21.15.40:0/3070205927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd27806f9a0 0x7fd2780ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 >> v1:172.21.15.103:6789/0 conn(0x7fd278143550 legacy=0x7fd2780d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd278143930 con 0x7fd2780d1370 2024-04-26T23:01:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fd260001840 con 0x7fd2780d1370 2024-04-26T23:01:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 >> v1:172.21.15.40:6789/0 conn(0x7fd2780d1370 legacy=0x7fd2780b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 --2- 172.21.15.40:0/3070205927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd27806f9a0 0x7fd258003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 --2- 172.21.15.40:0/3070205927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258004120 0x7fd258004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 --2- 172.21.15.40:0/3070205927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd258004d40 0x7fd2580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd2780cef90 con 0x7fd258004120 2024-04-26T23:01:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd25800bb00 con 0x7fd258004d40 2024-04-26T23:01:06.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd25800bd00 con 0x7fd27806f9a0 2024-04-26T23:01:06.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.959+0000 7fd26fe00640 1 --2- 172.21.15.40:0/3070205927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258004120 0x7fd258004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:06.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.960+0000 7fd275e00640 1 --2- 172.21.15.40:0/3070205927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd27806f9a0 0x7fd258003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:06.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.960+0000 7fd276800640 1 --2- 172.21.15.40:0/3070205927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd258004d40 0x7fd2580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:06.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.960+0000 7fd26fe00640 1 -- 172.21.15.40:0/3070205927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd27806f9a0 msgr2=0x7fd258003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:06.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.960+0000 7fd26fe00640 1 --2- 172.21.15.40:0/3070205927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd27806f9a0 0x7fd258003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:06.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.960+0000 7fd26fe00640 1 -- 172.21.15.40:0/3070205927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd258004d40 msgr2=0x7fd2580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:06.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.960+0000 7fd26fe00640 1 --2- 172.21.15.40:0/3070205927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd258004d40 0x7fd2580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:06.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.960+0000 7fd26fe00640 1 -- 172.21.15.40:0/3070205927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd258003520 con 0x7fd258004120 2024-04-26T23:01:06.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.961+0000 7fd26fe00640 1 --2- 172.21.15.40:0/3070205927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258004120 0x7fd258004800 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd26000c320 tx=0x7fd258005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8782460cd582fba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:06.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.961+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd260007dd0 con 0x7fd258004120 2024-04-26T23:01:06.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.961+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2600058d0 con 0x7fd258004120 2024-04-26T23:01:06.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.961+0000 7fd275400640 1 -- 172.21.15.40:0/3070205927 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd260005d20 con 0x7fd258004120 2024-04-26T23:01:06.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.962+0000 7fd277200640 1 -- 172.21.15.40:0/3070205927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258004120 msgr2=0x7fd258004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:06.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.962+0000 7fd277200640 1 --2- 172.21.15.40:0/3070205927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258004120 0x7fd258004800 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd26000c320 tx=0x7fd258005830 comp rx=0 tx=0).stop 2024-04-26T23:01:06.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.962+0000 7fd276800640 1 -- 172.21.15.40:0/3070205927 reap_dead start 2024-04-26T23:01:06.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.962+0000 7fd277200640 1 -- 172.21.15.40:0/3070205927 shutdown_connections 2024-04-26T23:01:06.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.962+0000 7fd277200640 1 -- 172.21.15.40:0/3070205927 >> 172.21.15.40:0/3070205927 conn(0x7fd278068270 msgr2=0x7fd278068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:06.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.962+0000 7fd277200640 1 -- 172.21.15.40:0/3070205927 shutdown_connections 2024-04-26T23:01:06.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.962+0000 7fd277200640 1 -- 172.21.15.40:0/3070205927 wait complete. 2024-04-26T23:01:06.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.964+0000 7fd277200640 1 Processor -- start 2024-04-26T23:01:06.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.964+0000 7fd277200640 1 -- start start 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.965+0000 7fd277200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd258004120 0x7fd278073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.965+0000 7fd277200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2781520d0 0x7fd27815efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.965+0000 7fd277200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd278154ae0 0x7fd27815fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.965+0000 7fd277200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd27806f480 con 0x7fd278154ae0 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.965+0000 7fd277200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd27806f1a0 con 0x7fd258004120 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.965+0000 7fd277200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd278146290 con 0x7fd2781520d0 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.965+0000 7fd276800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd278154ae0 0x7fd27815fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.965+0000 7fd276800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd278154ae0 0x7fd27815fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45284/0 (socket says 172.21.15.40:45284) 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.965+0000 7fd276800640 1 -- 172.21.15.40:0/1192963170 learned_addr learned my addr 172.21.15.40:0/1192963170 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.965+0000 7fd275e00640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd258004120 0x7fd278073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.966+0000 7fd26fe00640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2781520d0 0x7fd27815efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.966+0000 7fd276800640 1 -- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2781520d0 msgr2=0x7fd27815efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.966+0000 7fd276800640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2781520d0 0x7fd27815efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.966+0000 7fd276800640 1 -- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd258004120 msgr2=0x7fd278073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:06.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.966+0000 7fd276800640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd258004120 0x7fd278073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:06.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.966+0000 7fd276800640 1 -- 172.21.15.40:0/1192963170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27815f750 con 0x7fd278154ae0 2024-04-26T23:01:06.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.966+0000 7fd276800640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd278154ae0 0x7fd27815fd80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd27806e320 tx=0x7fd2700115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:06.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.966+0000 7fd26e000640 1 -- 172.21.15.40:0/1192963170 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270011e10 con 0x7fd278154ae0 2024-04-26T23:01:06.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.966+0000 7fd26e000640 1 -- 172.21.15.40:0/1192963170 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27001b070 con 0x7fd278154ae0 2024-04-26T23:01:06.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.966+0000 7fd26e000640 1 -- 172.21.15.40:0/1192963170 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270003990 con 0x7fd278154ae0 2024-04-26T23:01:06.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.966+0000 7fd277200640 1 -- 172.21.15.40:0/1192963170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd27815fa30 con 0x7fd278154ae0 2024-04-26T23:01:06.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.967+0000 7fd277200640 1 -- 172.21.15.40:0/1192963170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2780aeba0 con 0x7fd278154ae0 2024-04-26T23:01:06.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.968+0000 7fd277200640 1 -- 172.21.15.40:0/1192963170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd278069370 con 0x7fd278154ae0 2024-04-26T23:01:06.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.969+0000 7fd26e000640 1 -- 172.21.15.40:0/1192963170 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd270003b30 con 0x7fd278154ae0 2024-04-26T23:01:06.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.970+0000 7fd26e000640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd25406c8f0 0x7fd25406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:06.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.970+0000 7fd26e000640 1 -- 172.21.15.40:0/1192963170 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 6007+0+0 (secure 0 0 0) 0x7fd27008c4f0 con 0x7fd278154ae0 2024-04-26T23:01:06.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.971+0000 7fd275e00640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd25406c8f0 0x7fd25406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:06.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.972+0000 7fd275e00640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd25406c8f0 0x7fd25406edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd268007ae0 tx=0x7fd268009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:06.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:06.974+0000 7fd26e000640 1 -- 172.21.15.40:0/1192963170 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd270003e20 con 0x7fd278154ae0 2024-04-26T23:01:07.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:07.178+0000 7fd277200640 1 -- 172.21.15.40:0/1192963170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd27806e350 con 0x7fd278154ae0 2024-04-26T23:01:08.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.748+0000 7fd26e000640 1 -- 172.21.15.40:0/1192963170 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) ==== 228+0+0 (secure 0 0 0) 0x7fd27005a750 con 0x7fd278154ae0 2024-04-26T23:01:08.749 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data_ec' created 2024-04-26T23:01:08.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.752+0000 7fd277200640 1 -- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd25406c8f0 msgr2=0x7fd25406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:08.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.752+0000 7fd277200640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd25406c8f0 0x7fd25406edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd268007ae0 tx=0x7fd268009040 comp rx=0 tx=0).stop 2024-04-26T23:01:08.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.752+0000 7fd277200640 1 -- 172.21.15.40:0/1192963170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd278154ae0 msgr2=0x7fd27815fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:08.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.752+0000 7fd277200640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd278154ae0 0x7fd27815fd80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd27806e320 tx=0x7fd2700115a0 comp rx=0 tx=0).stop 2024-04-26T23:01:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.752+0000 7fd277200640 1 -- 172.21.15.40:0/1192963170 shutdown_connections 2024-04-26T23:01:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.752+0000 7fd277200640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd25406c8f0 0x7fd25406edb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.752+0000 7fd277200640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd278154ae0 0x7fd27815fd80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.752+0000 7fd277200640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd2781520d0 0x7fd27815efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.752+0000 7fd277200640 1 --2- 172.21.15.40:0/1192963170 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd258004120 0x7fd278073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.752+0000 7fd277200640 1 -- 172.21.15.40:0/1192963170 >> 172.21.15.40:0/1192963170 conn(0x7fd278068270 msgr2=0x7fd2780cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.752+0000 7fd277200640 1 -- 172.21.15.40:0/1192963170 shutdown_connections 2024-04-26T23:01:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.752+0000 7fd277200640 1 -- 172.21.15.40:0/1192963170 wait complete. 2024-04-26T23:01:08.780 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-26T23:01:08.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.968+0000 7f85c7200640 1 Processor -- start 2024-04-26T23:01:08.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.968+0000 7f85c7200640 1 -- start start 2024-04-26T23:01:08.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.969+0000 7f85c7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85c80b6b40 0x7f85c80b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:08.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.969+0000 7f85c7200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f85c81633a0 0x7f85c8163800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:08.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.970+0000 7f85c7200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85c80b7880 con 0x7f85c8163d40 2024-04-26T23:01:08.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.970+0000 7f85c7200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f85c80d0bb0 con 0x7f85c81633a0 2024-04-26T23:01:08.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.970+0000 7f85c7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f85c80d05a0 con 0x7f85c80b6b40 2024-04-26T23:01:08.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.970+0000 7f85c6800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f85c8163d40 0x7f85c80d2a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51442/0 (socket says 172.21.15.40:51442) 2024-04-26T23:01:08.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.970+0000 7f85c6800640 1 -- 172.21.15.40:0/3653738657 learned_addr learned my addr 172.21.15.40:0/3653738657 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:08.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.970+0000 7f85c5e00640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85c80b6b40 0x7f85c80b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:08.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.970+0000 7f85c5400640 1 --2- 172.21.15.40:0/3653738657 >> v2:172.21.15.103:3300/0 conn(0x7f85c81633a0 0x7f85c8163800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:08.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.970+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3024660793 0 0) 0x7f85c80b7880 con 0x7f85c8163d40 2024-04-26T23:01:08.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85a8003520 con 0x7f85c8163d40 2024-04-26T23:01:08.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1705715489 0 0) 0x7f85a8003520 con 0x7f85c8163d40 2024-04-26T23:01:08.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85c80b6b40 msgr2=0x7f85c80b6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:08.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85c80b6b40 0x7f85c80b6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:08.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 >> v2:172.21.15.103:3300/0 conn(0x7f85c81633a0 msgr2=0x7f85c8163800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:08.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 --2- 172.21.15.40:0/3653738657 >> v2:172.21.15.103:3300/0 conn(0x7f85c81633a0 0x7f85c8163800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c81641b0 con 0x7f85c8163d40 2024-04-26T23:01:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f85c0001670 con 0x7f85c8163d40 2024-04-26T23:01:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 >> v1:172.21.15.40:6789/0 conn(0x7f85c8163d40 legacy=0x7f85c80d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a80039b0 0x7f85a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85a8004800 0x7f85a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a80051c0 0x7f85a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f85c80d0bb0 con 0x7f85a80039b0 2024-04-26T23:01:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f85c80d05a0 con 0x7f85a80051c0 2024-04-26T23:01:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f85a800bf80 con 0x7f85a8004800 2024-04-26T23:01:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c5e00640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85c80b6b40 0x7f85c80b6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:01:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.971+0000 7f85c5400640 1 --2- 172.21.15.40:0/3653738657 >> v2:172.21.15.103:3300/0 conn(0x7f85c81633a0 0x7f85c8163800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:01:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.972+0000 7f85c5e00640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a80039b0 0x7f85a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.972+0000 7f85c5400640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85a8004800 0x7f85a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.972+0000 7f85c5e00640 1 -- 172.21.15.40:0/3653738657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85a8004800 msgr2=0x7f85a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.972+0000 7f85c5e00640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85a8004800 0x7f85a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.972+0000 7f85c6800640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a80051c0 0x7f85a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.972+0000 7f85c5e00640 1 -- 172.21.15.40:0/3653738657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a80051c0 msgr2=0x7f85a800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.972+0000 7f85c5e00640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a80051c0 0x7f85a800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.972+0000 7f85c5e00640 1 -- 172.21.15.40:0/3653738657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a8003520 con 0x7f85a80039b0 2024-04-26T23:01:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.972+0000 7f85c5400640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85a8004800 0x7f85a8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.973+0000 7f85c5e00640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a80039b0 0x7f85a8003dd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f85c808daf0 tx=0x7f85a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d7e8bbd48456a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.973+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b8036070 con 0x7f85a80039b0 2024-04-26T23:01:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.973+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b8004030 con 0x7f85a80039b0 2024-04-26T23:01:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.974+0000 7f85c4a00640 1 -- 172.21.15.40:0/3653738657 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b80097d0 con 0x7f85a80039b0 2024-04-26T23:01:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.974+0000 7f85c7200640 1 -- 172.21.15.40:0/3653738657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a80039b0 msgr2=0x7f85a8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.974+0000 7f85c7200640 1 --2- 172.21.15.40:0/3653738657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a80039b0 0x7f85a8003dd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f85c808daf0 tx=0x7f85a8005830 comp rx=0 tx=0).stop 2024-04-26T23:01:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.974+0000 7f85c6800640 1 -- 172.21.15.40:0/3653738657 reap_dead start 2024-04-26T23:01:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.974+0000 7f85c7200640 1 -- 172.21.15.40:0/3653738657 shutdown_connections 2024-04-26T23:01:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.974+0000 7f85c7200640 1 -- 172.21.15.40:0/3653738657 >> 172.21.15.40:0/3653738657 conn(0x7f85c80d4e50 msgr2=0x7f85c80d5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.974+0000 7f85c7200640 1 -- 172.21.15.40:0/3653738657 shutdown_connections 2024-04-26T23:01:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.974+0000 7f85c7200640 1 -- 172.21.15.40:0/3653738657 wait complete. 2024-04-26T23:01:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.976+0000 7f85c7200640 1 Processor -- start 2024-04-26T23:01:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.976+0000 7f85c7200640 1 -- start start 2024-04-26T23:01:08.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.977+0000 7f85c7200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85c80ba910 0x7f85c81f5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:08.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.977+0000 7f85c7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85c80c9180 0x7f85c80b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:08.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.977+0000 7f85c7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85c80b7fb0 0x7f85c80d2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.977+0000 7f85c7200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f85c80b6180 con 0x7f85c80ba910 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.977+0000 7f85c7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f85c80d0840 con 0x7f85c80b7fb0 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.977+0000 7f85c7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f85c80b6300 con 0x7f85c80c9180 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.977+0000 7f85c5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85c80ba910 0x7f85c81f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.977+0000 7f85c5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85c80ba910 0x7f85c81f5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45306/0 (socket says 172.21.15.40:45306) 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.977+0000 7f85c5e00640 1 -- 172.21.15.40:0/2301391941 learned_addr learned my addr 172.21.15.40:0/2301391941 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.978+0000 7f85c6800640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85c80b7fb0 0x7f85c80d2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.978+0000 7f85c5400640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85c80c9180 0x7f85c80b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.978+0000 7f85c5e00640 1 -- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85c80c9180 msgr2=0x7f85c80b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.978+0000 7f85c5e00640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85c80c9180 0x7f85c80b7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.978+0000 7f85c5e00640 1 -- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85c80b7fb0 msgr2=0x7f85c80d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.978+0000 7f85c5e00640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85c80b7fb0 0x7f85c80d2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.978+0000 7f85c5e00640 1 -- 172.21.15.40:0/2301391941 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c81f5c40 con 0x7f85c80ba910 2024-04-26T23:01:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.978+0000 7f85c6800640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85c80b7fb0 0x7f85c80d2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:08.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.978+0000 7f85c5e00640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85c80ba910 0x7f85c81f5700 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009bf0 tx=0x7f85b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:08.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.978+0000 7f85be000640 1 -- 172.21.15.40:0/2301391941 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b8036070 con 0x7f85c80ba910 2024-04-26T23:01:08.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.978+0000 7f85be000640 1 -- 172.21.15.40:0/2301391941 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b8041070 con 0x7f85c80ba910 2024-04-26T23:01:08.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.978+0000 7f85be000640 1 -- 172.21.15.40:0/2301391941 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b803c420 con 0x7f85c80ba910 2024-04-26T23:01:08.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.979+0000 7f85c7200640 1 -- 172.21.15.40:0/2301391941 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85c80bacf0 con 0x7f85c80ba910 2024-04-26T23:01:08.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.979+0000 7f85c7200640 1 -- 172.21.15.40:0/2301391941 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85c80c8af0 con 0x7f85c80ba910 2024-04-26T23:01:08.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.980+0000 7f85c7200640 1 -- 172.21.15.40:0/2301391941 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85c8069a80 con 0x7f85c80ba910 2024-04-26T23:01:08.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.981+0000 7f85be000640 1 -- 172.21.15.40:0/2301391941 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f85b803c5c0 con 0x7f85c80ba910 2024-04-26T23:01:08.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.982+0000 7f85be000640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85a406c920 0x7f85a406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:08.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.982+0000 7f85be000640 1 -- 172.21.15.40:0/2301391941 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6500+0+0 (secure 0 0 0) 0x7f85b80ace10 con 0x7f85c80ba910 2024-04-26T23:01:08.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.983+0000 7f85c5400640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85a406c920 0x7f85a406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.984+0000 7f85c5400640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85a406c920 0x7f85a406ede0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f85b0009fd0 tx=0x7f85b000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:08.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:08.987+0000 7f85be000640 1 -- 172.21.15.40:0/2301391941 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f85c81f5c40 con 0x7f85c80ba910 2024-04-26T23:01:09.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.191+0000 7f85c7200640 1 -- 172.21.15.40:0/2301391941 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f85c80b4590 con 0x7f85c80ba910 2024-04-26T23:01:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.759+0000 7f85be000640 1 -- 172.21.15.40:0/2301391941 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) ==== 169+0+0 (secure 0 0 0) 0x7f85b807af00 con 0x7f85c80ba910 2024-04-26T23:01:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:set pool 4 allow_ec_overwrites to true 2024-04-26T23:01:09.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.762+0000 7f85c7200640 1 -- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85a406c920 msgr2=0x7f85a406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:09.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.762+0000 7f85c7200640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85a406c920 0x7f85a406ede0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f85b0009fd0 tx=0x7f85b000b040 comp rx=0 tx=0).stop 2024-04-26T23:01:09.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.762+0000 7f85c7200640 1 -- 172.21.15.40:0/2301391941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85c80ba910 msgr2=0x7f85c81f5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:09.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.762+0000 7f85c7200640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85c80ba910 0x7f85c81f5700 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009bf0 tx=0x7f85b8004730 comp rx=0 tx=0).stop 2024-04-26T23:01:09.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.763+0000 7f85c7200640 1 -- 172.21.15.40:0/2301391941 shutdown_connections 2024-04-26T23:01:09.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.763+0000 7f85c7200640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85a406c920 0x7f85a406ede0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:09.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.763+0000 7f85c7200640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85c80b7fb0 0x7f85c80d2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:09.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.763+0000 7f85c7200640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85c80c9180 0x7f85c80b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:09.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.763+0000 7f85c7200640 1 --2- 172.21.15.40:0/2301391941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85c80ba910 0x7f85c81f5700 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.763+0000 7f85c7200640 1 -- 172.21.15.40:0/2301391941 >> 172.21.15.40:0/2301391941 conn(0x7f85c80d4e50 msgr2=0x7f85c80b3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.766+0000 7f85c7200640 1 -- 172.21.15.40:0/2301391941 shutdown_connections 2024-04-26T23:01:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.766+0000 7f85c7200640 1 -- 172.21.15.40:0/2301391941 wait complete. 2024-04-26T23:01:09.810 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-26T23:01:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.987+0000 7f3561a00640 1 Processor -- start 2024-04-26T23:01:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.988+0000 7f3561a00640 1 -- start start 2024-04-26T23:01:09.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.989+0000 7f3561a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f355c0b2340 0x7f355c0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:09.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.989+0000 7f3561a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f355c0d10d0 con 0x7f355c162810 2024-04-26T23:01:09.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.989+0000 7f3561a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f355c0d1370 con 0x7f355c0cd930 2024-04-26T23:01:09.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.989+0000 7f3561a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f355c0d0750 con 0x7f355c0b2340 2024-04-26T23:01:09.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.989+0000 7f355a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f355c0b2340 0x7f355c0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:09.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.989+0000 7f355aa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f355c162810 0x7f355c070350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51444/0 (socket says 172.21.15.40:51444) 2024-04-26T23:01:09.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.989+0000 7f355aa00640 1 -- 172.21.15.40:0/3062700271 learned_addr learned my addr 172.21.15.40:0/3062700271 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:09.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.990+0000 7f355a000640 1 -- 172.21.15.40:0/3062700271 >> v1:172.21.15.103:6789/0 conn(0x7f355c0cd930 legacy=0x7f355c0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:09.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.990+0000 7f3559600640 1 -- 172.21.15.40:0/3062700271 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 392944355 0 0) 0x7f355c0d10d0 con 0x7f355c162810 2024-04-26T23:01:09.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.990+0000 7f355a000640 1 -- 172.21.15.40:0/3062700271 >> v1:172.21.15.40:6789/0 conn(0x7f355c162810 legacy=0x7f355c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:09.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.990+0000 7f355a000640 1 -- 172.21.15.40:0/3062700271 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355c0d01b0 con 0x7f355c0b2340 2024-04-26T23:01:09.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.991+0000 7f355a000640 1 --2- 172.21.15.40:0/3062700271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c0b2340 0x7f355c0cd3f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f355c0d1540 tx=0x7f354802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abeb0cd7f986fbf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:09.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.991+0000 7f3559600640 1 -- 172.21.15.40:0/3062700271 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3548037070 con 0x7f355c0b2340 2024-04-26T23:01:09.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.991+0000 7f3559600640 1 -- 172.21.15.40:0/3062700271 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3548032440 con 0x7f355c0b2340 2024-04-26T23:01:09.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.992+0000 7f3559600640 1 -- 172.21.15.40:0/3062700271 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3548032810 con 0x7f355c0b2340 2024-04-26T23:01:09.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.992+0000 7f3561a00640 1 -- 172.21.15.40:0/3062700271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c0b2340 msgr2=0x7f355c0cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:09.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.992+0000 7f3561a00640 1 --2- 172.21.15.40:0/3062700271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c0b2340 0x7f355c0cd3f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f355c0d1540 tx=0x7f354802a210 comp rx=0 tx=0).stop 2024-04-26T23:01:09.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.992+0000 7f3561a00640 1 -- 172.21.15.40:0/3062700271 shutdown_connections 2024-04-26T23:01:09.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.992+0000 7f3561a00640 1 --2- 172.21.15.40:0/3062700271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c0b2340 0x7f355c0cd3f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:09.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.992+0000 7f3561a00640 1 -- 172.21.15.40:0/3062700271 >> 172.21.15.40:0/3062700271 conn(0x7f355c0d2da0 msgr2=0x7f355c0d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:09.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.992+0000 7f3561a00640 1 -- 172.21.15.40:0/3062700271 shutdown_connections 2024-04-26T23:01:09.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.992+0000 7f3561a00640 1 -- 172.21.15.40:0/3062700271 wait complete. 2024-04-26T23:01:09.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.994+0000 7f3561a00640 1 Processor -- start 2024-04-26T23:01:09.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.994+0000 7f3561a00640 1 -- start start 2024-04-26T23:01:09.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.995+0000 7f3561a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c0b2340 0x7f355c150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:09.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.995+0000 7f3561a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f355c0cd930 0x7f355c0b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:09.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.995+0000 7f3561a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f355c162810 0x7f355c0c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:09.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.995+0000 7f3561a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f355c0d2240 con 0x7f355c0b2340 2024-04-26T23:01:09.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.995+0000 7f3561a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f355c0d16a0 con 0x7f355c162810 2024-04-26T23:01:09.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.995+0000 7f3561a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f355c0d23c0 con 0x7f355c0cd930 2024-04-26T23:01:09.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.995+0000 7f355aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c0b2340 0x7f355c150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:09.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.996+0000 7f355aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c0b2340 0x7f355c150180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45326/0 (socket says 172.21.15.40:45326) 2024-04-26T23:01:09.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.996+0000 7f355aa00640 1 -- 172.21.15.40:0/722923608 learned_addr learned my addr 172.21.15.40:0/722923608 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:09.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.996+0000 7f355aa00640 1 -- 172.21.15.40:0/722923608 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f355c0cd930 msgr2=0x7f355c0b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:09.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.996+0000 7f355aa00640 1 --2- 172.21.15.40:0/722923608 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f355c0cd930 0x7f355c0b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:09.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.996+0000 7f355aa00640 1 -- 172.21.15.40:0/722923608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f355c162810 msgr2=0x7f355c0c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:09.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.996+0000 7f355aa00640 1 --2- 172.21.15.40:0/722923608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f355c162810 0x7f355c0c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:09.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.996+0000 7f355aa00640 1 -- 172.21.15.40:0/722923608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355c1617a0 con 0x7f355c0b2340 2024-04-26T23:01:09.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.996+0000 7f355aa00640 1 --2- 172.21.15.40:0/722923608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c0b2340 0x7f355c150180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f354c00c420 tx=0x7f354c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:09.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.996+0000 7f3552a00640 1 -- 172.21.15.40:0/722923608 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c0074e0 con 0x7f355c0b2340 2024-04-26T23:01:09.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.996+0000 7f3552a00640 1 -- 172.21.15.40:0/722923608 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f354c0047e0 con 0x7f355c0b2340 2024-04-26T23:01:09.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.996+0000 7f3552a00640 1 -- 172.21.15.40:0/722923608 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c004c10 con 0x7f355c0b2340 2024-04-26T23:01:09.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.996+0000 7f3561a00640 1 -- 172.21.15.40:0/722923608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f355c160b90 con 0x7f355c0b2340 2024-04-26T23:01:09.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.997+0000 7f3561a00640 1 -- 172.21.15.40:0/722923608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f355c161150 con 0x7f355c0b2340 2024-04-26T23:01:09.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.998+0000 7f3561a00640 1 -- 172.21.15.40:0/722923608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f355c0694f0 con 0x7f355c0b2340 2024-04-26T23:01:09.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:09.999+0000 7f3552a00640 1 -- 172.21.15.40:0/722923608 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f354c011070 con 0x7f355c0b2340 2024-04-26T23:01:10.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.000+0000 7f3552a00640 1 --2- 172.21.15.40:0/722923608 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f353c06c8f0 0x7f353c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:10.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.000+0000 7f3552a00640 1 -- 172.21.15.40:0/722923608 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f354c08bae0 con 0x7f355c0b2340 2024-04-26T23:01:10.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.000+0000 7f355a000640 1 --2- 172.21.15.40:0/722923608 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f353c06c8f0 0x7f353c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:10.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.001+0000 7f355a000640 1 --2- 172.21.15.40:0/722923608 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f353c06c8f0 0x7f353c06edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3548034000 tx=0x7f354802c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:10.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.004+0000 7f3552a00640 1 -- 172.21.15.40:0/722923608 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f355c1617a0 con 0x7f355c0b2340 2024-04-26T23:01:10.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.252+0000 7f3561a00640 1 -- 172.21.15.40:0/722923608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f355c163b70 con 0x7f355c0b2340 2024-04-26T23:01:10.654 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:01:10.653+0000 7fecfe400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T23:01:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.763+0000 7f3552a00640 1 -- 172.21.15.40:0/722923608 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7f354c00e280 con 0x7f355c0b2340 2024-04-26T23:01:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:added data pool 4 to fsmap 2024-04-26T23:01:10.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.767+0000 7f3561a00640 1 -- 172.21.15.40:0/722923608 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f353c06c8f0 msgr2=0x7f353c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:10.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.767+0000 7f3561a00640 1 --2- 172.21.15.40:0/722923608 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f353c06c8f0 0x7f353c06edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3548034000 tx=0x7f354802c000 comp rx=0 tx=0).stop 2024-04-26T23:01:10.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.767+0000 7f3561a00640 1 -- 172.21.15.40:0/722923608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c0b2340 msgr2=0x7f355c150180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:10.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.767+0000 7f3561a00640 1 --2- 172.21.15.40:0/722923608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c0b2340 0x7f355c150180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f354c00c420 tx=0x7f354c00c8e0 comp rx=0 tx=0).stop 2024-04-26T23:01:10.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.768+0000 7f3561a00640 1 -- 172.21.15.40:0/722923608 shutdown_connections 2024-04-26T23:01:10.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.768+0000 7f3561a00640 1 --2- 172.21.15.40:0/722923608 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f353c06c8f0 0x7f353c06edb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:10.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.768+0000 7f3561a00640 1 --2- 172.21.15.40:0/722923608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f355c162810 0x7f355c0c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:10.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.768+0000 7f3561a00640 1 --2- 172.21.15.40:0/722923608 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f355c0cd930 0x7f355c0b36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:10.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.768+0000 7f3561a00640 1 --2- 172.21.15.40:0/722923608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c0b2340 0x7f355c150180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:10.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.768+0000 7f3561a00640 1 -- 172.21.15.40:0/722923608 >> 172.21.15.40:0/722923608 conn(0x7f355c0d2da0 msgr2=0x7f355c0b0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:10.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.768+0000 7f3561a00640 1 -- 172.21.15.40:0/722923608 shutdown_connections 2024-04-26T23:01:10.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.768+0000 7f3561a00640 1 -- 172.21.15.40:0/722923608 wait complete. 2024-04-26T23:01:10.790 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:01:10.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.994+0000 7f42c2e00640 1 Processor -- start 2024-04-26T23:01:10.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.994+0000 7f42c2e00640 1 -- start start 2024-04-26T23:01:10.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.995+0000 7f42c2e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f42bc0c6f10 0x7f42bc0c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:10.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.996+0000 7f42c2e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42bc0c6010 0x7f42bc158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:10.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.996+0000 7f42c2e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42bc0cce10 con 0x7f42bc0cc980 2024-04-26T23:01:10.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.996+0000 7f42c2e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f42bc0c72f0 con 0x7f42bc0c6f10 2024-04-26T23:01:10.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.996+0000 7f42c2e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f42bc0cb860 con 0x7f42bc0c6010 2024-04-26T23:01:10.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.996+0000 7f42c1000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f42bc0cc980 0x7f42bc0cbcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51456/0 (socket says 172.21.15.40:51456) 2024-04-26T23:01:10.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.996+0000 7f42c1000640 1 -- 172.21.15.40:0/2371944087 learned_addr learned my addr 172.21.15.40:0/2371944087 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:10.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.996+0000 7f42c2400640 1 --2- 172.21.15.40:0/2371944087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42bc0c6010 0x7f42bc158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:10.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.996+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1943570150 0 0) 0x7f42bc0cce10 con 0x7f42bc0cc980 2024-04-26T23:01:10.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.996+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42a4003520 con 0x7f42bc0cc980 2024-04-26T23:01:10.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.996+0000 7f42c1a00640 1 --2- 172.21.15.40:0/2371944087 >> v2:172.21.15.103:3300/0 conn(0x7f42bc0c6f10 0x7f42bc0c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1282758495 0 0) 0x7f42a4003520 con 0x7f42bc0cc980 2024-04-26T23:01:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42bc0c6010 msgr2=0x7f42bc158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 --2- 172.21.15.40:0/2371944087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42bc0c6010 0x7f42bc158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 >> v2:172.21.15.103:3300/0 conn(0x7f42bc0c6f10 msgr2=0x7f42bc0c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 --2- 172.21.15.40:0/2371944087 >> v2:172.21.15.103:3300/0 conn(0x7f42bc0c6f10 0x7f42bc0c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42bc0cd0f0 con 0x7f42bc0cc980 2024-04-26T23:01:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f42ac001670 con 0x7f42bc0cc980 2024-04-26T23:01:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 >> v1:172.21.15.40:6789/0 conn(0x7f42bc0cc980 legacy=0x7f42bc0cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 --2- 172.21.15.40:0/2371944087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42a40039b0 0x7f42a4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 --2- 172.21.15.40:0/2371944087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42bc0c6010 0x7f42a4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 --2- 172.21.15.40:0/2371944087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42a4004d40 0x7f42a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc0c72f0 con 0x7f42a40039b0 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc0cb860 con 0x7f42a4004d40 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f42a400baa0 con 0x7f42bc0c6010 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.997+0000 7f42c1a00640 1 --2- 172.21.15.40:0/2371944087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42a40039b0 0x7f42a4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.998+0000 7f42c1000640 1 --2- 172.21.15.40:0/2371944087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42bc0c6010 0x7f42a4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.998+0000 7f42c1a00640 1 -- 172.21.15.40:0/2371944087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42bc0c6010 msgr2=0x7f42a4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.998+0000 7f42c2400640 1 --2- 172.21.15.40:0/2371944087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42a4004d40 0x7f42a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.998+0000 7f42c1a00640 1 --2- 172.21.15.40:0/2371944087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42bc0c6010 0x7f42a4004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.998+0000 7f42c1a00640 1 -- 172.21.15.40:0/2371944087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42a4004d40 msgr2=0x7f42a40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.998+0000 7f42c1a00640 1 --2- 172.21.15.40:0/2371944087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42a4004d40 0x7f42a40051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.998+0000 7f42c1a00640 1 -- 172.21.15.40:0/2371944087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a4003520 con 0x7f42a40039b0 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.999+0000 7f42c1a00640 1 --2- 172.21.15.40:0/2371944087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42a40039b0 0x7f42a4003dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f42b000ba30 tx=0x7f42a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f120a16b6631e5c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.999+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0030070 con 0x7f42a40039b0 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.999+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b002ae10 con 0x7f42a40039b0 2024-04-26T23:01:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.999+0000 7f42bbe00640 1 -- 172.21.15.40:0/2371944087 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b000b600 con 0x7f42a40039b0 2024-04-26T23:01:11.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.999+0000 7f42c2e00640 1 -- 172.21.15.40:0/2371944087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42a40039b0 msgr2=0x7f42a4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.999+0000 7f42c2e00640 1 --2- 172.21.15.40:0/2371944087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42a40039b0 0x7f42a4003dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f42b000ba30 tx=0x7f42a4005830 comp rx=0 tx=0).stop 2024-04-26T23:01:11.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:10.999+0000 7f42c2400640 1 -- 172.21.15.40:0/2371944087 reap_dead start 2024-04-26T23:01:11.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.000+0000 7f42c2e00640 1 -- 172.21.15.40:0/2371944087 shutdown_connections 2024-04-26T23:01:11.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.000+0000 7f42c2e00640 1 -- 172.21.15.40:0/2371944087 >> 172.21.15.40:0/2371944087 conn(0x7f42bc0d0a10 msgr2=0x7f42bc0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:11.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.000+0000 7f42c2e00640 1 -- 172.21.15.40:0/2371944087 shutdown_connections 2024-04-26T23:01:11.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.000+0000 7f42c2e00640 1 -- 172.21.15.40:0/2371944087 wait complete. 2024-04-26T23:01:11.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.002+0000 7f42c2e00640 1 Processor -- start 2024-04-26T23:01:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.002+0000 7f42c2e00640 1 -- start start 2024-04-26T23:01:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.003+0000 7f42c2e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42a40039b0 0x7f42bc146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.003+0000 7f42c2e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42bc0b3730 0x7f42bc0b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.003+0000 7f42c2e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42bc161aa0 0x7f42bc161f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.003+0000 7f42c2e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc0cc400 con 0x7f42bc161aa0 2024-04-26T23:01:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.003+0000 7f42c2e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc0cb1a0 con 0x7f42a40039b0 2024-04-26T23:01:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.003+0000 7f42c2e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f42bc0cc580 con 0x7f42bc0b3730 2024-04-26T23:01:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.003+0000 7f42c2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42bc161aa0 0x7f42bc161f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.003+0000 7f42c2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42bc161aa0 0x7f42bc161f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45346/0 (socket says 172.21.15.40:45346) 2024-04-26T23:01:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.003+0000 7f42c2400640 1 -- 172.21.15.40:0/1663398706 learned_addr learned my addr 172.21.15.40:0/1663398706 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.004+0000 7f42c1000640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42bc0b3730 0x7f42bc0b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.004+0000 7f42c1a00640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42a40039b0 0x7f42bc146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.004+0000 7f42c2400640 1 -- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42bc0b3730 msgr2=0x7f42bc0b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.004+0000 7f42c2400640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42bc0b3730 0x7f42bc0b3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.004+0000 7f42c2400640 1 -- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42a40039b0 msgr2=0x7f42bc146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.004+0000 7f42c2400640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42a40039b0 0x7f42bc146bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.004+0000 7f42c2400640 1 -- 172.21.15.40:0/1663398706 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42bc14a4d0 con 0x7f42bc161aa0 2024-04-26T23:01:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.004+0000 7f42c2400640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42bc161aa0 0x7f42bc161f50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42bc069090 tx=0x7f42b400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.004+0000 7f42b9600640 1 -- 172.21.15.40:0/1663398706 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b401c070 con 0x7f42bc161aa0 2024-04-26T23:01:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.004+0000 7f42b9600640 1 -- 172.21.15.40:0/1663398706 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b4017400 con 0x7f42bc161aa0 2024-04-26T23:01:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.004+0000 7f42b9600640 1 -- 172.21.15.40:0/1663398706 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b40177f0 con 0x7f42bc161aa0 2024-04-26T23:01:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.004+0000 7f42c2e00640 1 -- 172.21.15.40:0/1663398706 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42bc0b56b0 con 0x7f42bc161aa0 2024-04-26T23:01:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.005+0000 7f42c2e00640 1 -- 172.21.15.40:0/1663398706 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42bc0b5c70 con 0x7f42bc161aa0 2024-04-26T23:01:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.006+0000 7f42c2e00640 1 -- 172.21.15.40:0/1663398706 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42bc0717e0 con 0x7f42bc161aa0 2024-04-26T23:01:11.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.007+0000 7f42b9600640 1 -- 172.21.15.40:0/1663398706 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f42b401f070 con 0x7f42bc161aa0 2024-04-26T23:01:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.008+0000 7f42b9600640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f42a006c920 0x7f42a006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.009+0000 7f42b9600640 1 -- 172.21.15.40:0/1663398706 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f42b408bfe0 con 0x7f42bc161aa0 2024-04-26T23:01:11.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.009+0000 7f42c1a00640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f42a006c920 0x7f42a006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.010+0000 7f42c1a00640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f42a006c920 0x7f42a006ede0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f42b000ba00 tx=0x7f42b0030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:11.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.013+0000 7f42b9600640 1 -- 172.21.15.40:0/1663398706 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f42bc0717e0 con 0x7f42bc161aa0 2024-04-26T23:01:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.254+0000 7f42c2e00640 1 -- 172.21.15.40:0/1663398706 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42bc069370 con 0x7f42bc161aa0 2024-04-26T23:01:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.256+0000 7f42b9600640 1 -- 172.21.15.40:0/1663398706 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5183 (secure 0 0 0) 0x7f42b4012070 con 0x7f42bc161aa0 2024-04-26T23:01:11.256 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:11.256 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1253648754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1253648754},{"type":"v1","addr":"172.21.15.103:6836","nonce":1253648754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3239652611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":3239652611},{"type":"v1","addr":"172.21.15.103:6837","nonce":3239652611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3670144900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3670144900},{"type":"v1","addr":"172.21.15.40:6833","nonce":3670144900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:01:05.171276+0000","modified":"2024-04-26T23:01:10.254243+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3989190340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3989190340},{"type":"v1","addr":"172.21.15.40:6835","nonce":3989190340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-26T23:01:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 7 2024-04-26T23:01:11.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.258+0000 7f42c2e00640 1 -- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f42a006c920 msgr2=0x7f42a006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.258+0000 7f42c2e00640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f42a006c920 0x7f42a006ede0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f42b000ba00 tx=0x7f42b0030000 comp rx=0 tx=0).stop 2024-04-26T23:01:11.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.258+0000 7f42c2e00640 1 -- 172.21.15.40:0/1663398706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42bc161aa0 msgr2=0x7f42bc161f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.258+0000 7f42c2e00640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42bc161aa0 0x7f42bc161f50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42bc069090 tx=0x7f42b400f4a0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.259+0000 7f42c2e00640 1 -- 172.21.15.40:0/1663398706 shutdown_connections 2024-04-26T23:01:11.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.259+0000 7f42c2e00640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f42a006c920 0x7f42a006ede0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.259+0000 7f42c2e00640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42bc161aa0 0x7f42bc161f50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.259+0000 7f42c2e00640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42bc0b3730 0x7f42bc0b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.259+0000 7f42c2e00640 1 --2- 172.21.15.40:0/1663398706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42a40039b0 0x7f42bc146bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.259+0000 7f42c2e00640 1 -- 172.21.15.40:0/1663398706 >> 172.21.15.40:0/1663398706 conn(0x7f42bc0d0a10 msgr2=0x7f42bc162ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:11.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.259+0000 7f42c2e00640 1 -- 172.21.15.40:0/1663398706 shutdown_connections 2024-04-26T23:01:11.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.259+0000 7f42c2e00640 1 -- 172.21.15.40:0/1663398706 wait complete. 2024-04-26T23:01:11.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:11.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.483+0000 7f933f200640 1 Processor -- start 2024-04-26T23:01:11.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.483+0000 7f933f200640 1 -- start start 2024-04-26T23:01:11.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.485+0000 7f933f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93400c72d0 0x7f9340167670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.485+0000 7f933f200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9340163810 con 0x7f9340167bb0 2024-04-26T23:01:11.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.485+0000 7f933f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93400cb100 con 0x7f9340163430 2024-04-26T23:01:11.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.485+0000 7f933f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f93400cb400 con 0x7f93400c72d0 2024-04-26T23:01:11.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.486+0000 7f933e800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9340167bb0 0x7f93400ce800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51468/0 (socket says 172.21.15.40:51468) 2024-04-26T23:01:11.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.486+0000 7f933e800640 1 -- 172.21.15.40:0/3106711398 learned_addr learned my addr 172.21.15.40:0/3106711398 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:11.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.486+0000 7f9335400640 1 --2- 172.21.15.40:0/3106711398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93400c72d0 0x7f9340167670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.486+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4152673927 0 0) 0x7f9340163810 con 0x7f9340167bb0 2024-04-26T23:01:11.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.486+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9324003520 con 0x7f9340167bb0 2024-04-26T23:01:11.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.486+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4108096689 0 0) 0x7f9324003520 con 0x7f9340167bb0 2024-04-26T23:01:11.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.486+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93400c72d0 msgr2=0x7f9340167670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:11.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.486+0000 7f933d400640 1 --2- 172.21.15.40:0/3106711398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93400c72d0 0x7f9340167670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.486+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 >> v1:172.21.15.103:6789/0 conn(0x7f9340163430 legacy=0x7f9340166f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.486+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93400ccc60 con 0x7f9340167bb0 2024-04-26T23:01:11.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.486+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f93380019f0 con 0x7f9340167bb0 2024-04-26T23:01:11.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.486+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 >> v1:172.21.15.40:6789/0 conn(0x7f9340167bb0 legacy=0x7f93400ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f933d400640 1 --2- 172.21.15.40:0/3106711398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93240039e0 0x7f9324003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f933d400640 1 --2- 172.21.15.40:0/3106711398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93400c72d0 0x7f9324004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f933d400640 1 --2- 172.21.15.40:0/3106711398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9324004d40 0x7f93240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f93400cb400 con 0x7f9324004d40 2024-04-26T23:01:11.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f932400baa0 con 0x7f93240039e0 2024-04-26T23:01:11.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f932400bca0 con 0x7f93400c72d0 2024-04-26T23:01:11.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f933e800640 1 --2- 172.21.15.40:0/3106711398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9324004d40 0x7f93240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f9335400640 1 --2- 172.21.15.40:0/3106711398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93400c72d0 0x7f9324004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f933de00640 1 --2- 172.21.15.40:0/3106711398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93240039e0 0x7f9324003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f933e800640 1 -- 172.21.15.40:0/3106711398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93400c72d0 msgr2=0x7f9324004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:11.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f933e800640 1 --2- 172.21.15.40:0/3106711398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93400c72d0 0x7f9324004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f933e800640 1 -- 172.21.15.40:0/3106711398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93240039e0 msgr2=0x7f9324003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:11.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f933e800640 1 --2- 172.21.15.40:0/3106711398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93240039e0 0x7f9324003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.487+0000 7f933e800640 1 -- 172.21.15.40:0/3106711398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9324003520 con 0x7f9324004d40 2024-04-26T23:01:11.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.488+0000 7f933e800640 1 --2- 172.21.15.40:0/3106711398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9324004d40 0x7f93240051c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f933800a740 tx=0x7f9324005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64ece6e266d51b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:11.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.489+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933800a390 con 0x7f9324004d40 2024-04-26T23:01:11.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.489+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f933802f070 con 0x7f9324004d40 2024-04-26T23:01:11.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.489+0000 7f933d400640 1 -- 172.21.15.40:0/3106711398 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933802bc40 con 0x7f9324004d40 2024-04-26T23:01:11.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.489+0000 7f933f200640 1 -- 172.21.15.40:0/3106711398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9324004d40 msgr2=0x7f93240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.489+0000 7f933f200640 1 --2- 172.21.15.40:0/3106711398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9324004d40 0x7f93240051c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f933800a740 tx=0x7f9324005830 comp rx=0 tx=0).stop 2024-04-26T23:01:11.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.489+0000 7f933e800640 1 -- 172.21.15.40:0/3106711398 reap_dead start 2024-04-26T23:01:11.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.489+0000 7f933f200640 1 -- 172.21.15.40:0/3106711398 shutdown_connections 2024-04-26T23:01:11.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.489+0000 7f933f200640 1 -- 172.21.15.40:0/3106711398 >> 172.21.15.40:0/3106711398 conn(0x7f93400d0a10 msgr2=0x7f93400d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:11.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.490+0000 7f933f200640 1 -- 172.21.15.40:0/3106711398 shutdown_connections 2024-04-26T23:01:11.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.490+0000 7f933f200640 1 -- 172.21.15.40:0/3106711398 wait complete. 2024-04-26T23:01:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.491+0000 7f933f200640 1 Processor -- start 2024-04-26T23:01:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.491+0000 7f933f200640 1 -- start start 2024-04-26T23:01:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.492+0000 7f933f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9324004d40 0x7f93400c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.492+0000 7f933f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9340158e40 0x7f934015e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.492+0000 7f933f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93401592c0 0x7f934015e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.492+0000 7f933f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f93400cd1e0 con 0x7f9340158e40 2024-04-26T23:01:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.492+0000 7f933f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f93400afe80 con 0x7f9324004d40 2024-04-26T23:01:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.492+0000 7f933f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9340167670 con 0x7f93401592c0 2024-04-26T23:01:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.493+0000 7f9335400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9340158e40 0x7f934015e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.493+0000 7f9335400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9340158e40 0x7f934015e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45366/0 (socket says 172.21.15.40:45366) 2024-04-26T23:01:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.493+0000 7f9335400640 1 -- 172.21.15.40:0/3405814882 learned_addr learned my addr 172.21.15.40:0/3405814882 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.493+0000 7f933e800640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93401592c0 0x7f934015e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.493+0000 7f933de00640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9324004d40 0x7f93400c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.493+0000 7f9335400640 1 -- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93401592c0 msgr2=0x7f934015e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.493+0000 7f9335400640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93401592c0 0x7f934015e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.493+0000 7f9335400640 1 -- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9324004d40 msgr2=0x7f93400c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.493+0000 7f9335400640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9324004d40 0x7f93400c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.493+0000 7f9335400640 1 -- 172.21.15.40:0/3405814882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9340164ac0 con 0x7f9340158e40 2024-04-26T23:01:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.493+0000 7f933de00640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9324004d40 0x7f93400c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.493+0000 7f9335400640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9340158e40 0x7f934015e3b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9328011560 tx=0x7f9328011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.494+0000 7f9336a00640 1 -- 172.21.15.40:0/3405814882 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328015870 con 0x7f9340158e40 2024-04-26T23:01:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.494+0000 7f9336a00640 1 -- 172.21.15.40:0/3405814882 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9328015a10 con 0x7f9340158e40 2024-04-26T23:01:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.494+0000 7f9336a00640 1 -- 172.21.15.40:0/3405814882 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328015e00 con 0x7f9340158e40 2024-04-26T23:01:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.494+0000 7f933f200640 1 -- 172.21.15.40:0/3405814882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9340164da0 con 0x7f9340158e40 2024-04-26T23:01:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.494+0000 7f933f200640 1 -- 172.21.15.40:0/3405814882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9340165360 con 0x7f9340158e40 2024-04-26T23:01:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.495+0000 7f933f200640 1 -- 172.21.15.40:0/3405814882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9340069370 con 0x7f9340158e40 2024-04-26T23:01:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.496+0000 7f9336a00640 1 -- 172.21.15.40:0/3405814882 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f932800f460 con 0x7f9340158e40 2024-04-26T23:01:11.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.497+0000 7f9336a00640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f931c06c8f0 0x7f931c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.497+0000 7f9336a00640 1 -- 172.21.15.40:0/3405814882 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f932808c8a0 con 0x7f9340158e40 2024-04-26T23:01:11.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.497+0000 7f933de00640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f931c06c8f0 0x7f931c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.498+0000 7f933de00640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f931c06c8f0 0x7f931c06edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9330001630 tx=0x7f9330009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:11.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.501+0000 7f9336a00640 1 -- 172.21.15.40:0/3405814882 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9328019030 con 0x7f9340158e40 2024-04-26T23:01:11.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.702+0000 7f933f200640 1 -- 172.21.15.40:0/3405814882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93400ce5e0 con 0x7f9340158e40 2024-04-26T23:01:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.706+0000 7f9336a00640 1 -- 172.21.15.40:0/3405814882 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18491 (secure 0 0 0) 0x7f932805a8f0 con 0x7f9340158e40 2024-04-26T23:01:11.706 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:11.707 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":24,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:10.253857+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:11.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.708+0000 7f933f200640 1 -- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f931c06c8f0 msgr2=0x7f931c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.708+0000 7f933f200640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f931c06c8f0 0x7f931c06edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9330001630 tx=0x7f9330009040 comp rx=0 tx=0).stop 2024-04-26T23:01:11.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.708+0000 7f933f200640 1 -- 172.21.15.40:0/3405814882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9340158e40 msgr2=0x7f934015e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.708+0000 7f933f200640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9340158e40 0x7f934015e3b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9328011560 tx=0x7f9328011a20 comp rx=0 tx=0).stop 2024-04-26T23:01:11.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.709+0000 7f933f200640 1 -- 172.21.15.40:0/3405814882 shutdown_connections 2024-04-26T23:01:11.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.709+0000 7f933f200640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f931c06c8f0 0x7f931c06edb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.709+0000 7f933f200640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93401592c0 0x7f934015e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.709+0000 7f933f200640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9340158e40 0x7f934015e3b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.709+0000 7f933f200640 1 --2- 172.21.15.40:0/3405814882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9324004d40 0x7f93400c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.709+0000 7f933f200640 1 -- 172.21.15.40:0/3405814882 >> 172.21.15.40:0/3405814882 conn(0x7f93400d0a10 msgr2=0x7f93400b19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:11.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.709+0000 7f933f200640 1 -- 172.21.15.40:0/3405814882 shutdown_connections 2024-04-26T23:01:11.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.709+0000 7f933f200640 1 -- 172.21.15.40:0/3405814882 wait complete. 2024-04-26T23:01:11.727 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.933+0000 7f47f8a00640 1 Processor -- start 2024-04-26T23:01:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.933+0000 7f47f8a00640 1 -- start start 2024-04-26T23:01:11.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.934+0000 7f47f8a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f415f380 0x7f47f40cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.935+0000 7f47f8a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47f40cfc00 con 0x7f47f4073b70 2024-04-26T23:01:11.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.935+0000 7f47f8a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47f4072c00 con 0x7f47f40d0620 2024-04-26T23:01:11.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.935+0000 7f47f8a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f47f40729b0 con 0x7f47f415f380 2024-04-26T23:01:11.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.935+0000 7f47ebe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f47f4073b70 0x7f47f40cede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51474/0 (socket says 172.21.15.40:51474) 2024-04-26T23:01:11.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.935+0000 7f47ebe00640 1 -- 172.21.15.40:0/3657710997 learned_addr learned my addr 172.21.15.40:0/3657710997 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.935+0000 7f47f1e00640 1 --2- 172.21.15.40:0/3657710997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f415f380 0x7f47f40cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.935+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 917772702 0 0) 0x7f47f40cfc00 con 0x7f47f4073b70 2024-04-26T23:01:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.935+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47d8003520 con 0x7f47f4073b70 2024-04-26T23:01:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3034724440 0 0) 0x7f47d8003520 con 0x7f47f4073b70 2024-04-26T23:01:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f415f380 msgr2=0x7f47f40cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47f1400640 1 --2- 172.21.15.40:0/3657710997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f415f380 0x7f47f40cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 >> v1:172.21.15.103:6789/0 conn(0x7f47f40d0620 legacy=0x7f47f40cf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47f4072ee0 con 0x7f47f4073b70 2024-04-26T23:01:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f47dc001670 con 0x7f47f4073b70 2024-04-26T23:01:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 >> v1:172.21.15.40:6789/0 conn(0x7f47f4073b70 legacy=0x7f47f40cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47f1400640 1 --2- 172.21.15.40:0/3657710997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80039e0 0x7f47d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47f1400640 1 --2- 172.21.15.40:0/3657710997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d8004800 0x7f47d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47f1400640 1 --2- 172.21.15.40:0/3657710997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f415f380 0x7f47d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f47f40729b0 con 0x7f47d8004800 2024-04-26T23:01:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f47d800baa0 con 0x7f47d80039e0 2024-04-26T23:01:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f47d800bca0 con 0x7f47f415f380 2024-04-26T23:01:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.936+0000 7f47ebe00640 1 --2- 172.21.15.40:0/3657710997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d8004800 0x7f47d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.937+0000 7f47f1e00640 1 --2- 172.21.15.40:0/3657710997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80039e0 0x7f47d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.937+0000 7f47f2800640 1 --2- 172.21.15.40:0/3657710997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f415f380 0x7f47d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.937+0000 7f47ebe00640 1 -- 172.21.15.40:0/3657710997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f415f380 msgr2=0x7f47d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.937+0000 7f47ebe00640 1 --2- 172.21.15.40:0/3657710997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f415f380 0x7f47d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.937+0000 7f47ebe00640 1 -- 172.21.15.40:0/3657710997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80039e0 msgr2=0x7f47d8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.937+0000 7f47ebe00640 1 --2- 172.21.15.40:0/3657710997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80039e0 0x7f47d8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.937+0000 7f47ebe00640 1 -- 172.21.15.40:0/3657710997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d8003520 con 0x7f47d8004800 2024-04-26T23:01:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.938+0000 7f47ebe00640 1 --2- 172.21.15.40:0/3657710997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d8004800 0x7f47d8004c80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f47dc007fd0 tx=0x7f47d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e89a4afe325652 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.938+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc007c20 con 0x7f47d8004800 2024-04-26T23:01:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.938+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47dc005af0 con 0x7f47d8004800 2024-04-26T23:01:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.938+0000 7f47f1400640 1 -- 172.21.15.40:0/3657710997 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc007070 con 0x7f47d8004800 2024-04-26T23:01:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.939+0000 7f47f8a00640 1 -- 172.21.15.40:0/3657710997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d8004800 msgr2=0x7f47d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.939+0000 7f47f8a00640 1 --2- 172.21.15.40:0/3657710997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d8004800 0x7f47d8004c80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f47dc007fd0 tx=0x7f47d8005830 comp rx=0 tx=0).stop 2024-04-26T23:01:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.939+0000 7f47f2800640 1 -- 172.21.15.40:0/3657710997 reap_dead start 2024-04-26T23:01:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.939+0000 7f47f8a00640 1 -- 172.21.15.40:0/3657710997 shutdown_connections 2024-04-26T23:01:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.939+0000 7f47f8a00640 1 -- 172.21.15.40:0/3657710997 >> 172.21.15.40:0/3657710997 conn(0x7f47f40d0be0 msgr2=0x7f47f40d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.939+0000 7f47f8a00640 1 -- 172.21.15.40:0/3657710997 shutdown_connections 2024-04-26T23:01:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.939+0000 7f47f8a00640 1 -- 172.21.15.40:0/3657710997 wait complete. 2024-04-26T23:01:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.941+0000 7f47f8a00640 1 Processor -- start 2024-04-26T23:01:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.941+0000 7f47f8a00640 1 -- start start 2024-04-26T23:01:11.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47f8a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d8004800 0x7f47f414f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47f8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f41523f0 0x7f47f41f7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47f8a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f41f8530 0x7f47f406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47f8a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f47f40afe40 con 0x7f47f41523f0 2024-04-26T23:01:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47f8a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f47f40afcc0 con 0x7f47d8004800 2024-04-26T23:01:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47f8a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f47f40affc0 con 0x7f47f41f8530 2024-04-26T23:01:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47ebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f41523f0 0x7f47f41f7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47ebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f41523f0 0x7f47f41f7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45376/0 (socket says 172.21.15.40:45376) 2024-04-26T23:01:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47ebe00640 1 -- 172.21.15.40:0/2303912103 learned_addr learned my addr 172.21.15.40:0/2303912103 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47f1e00640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d8004800 0x7f47f414f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47f2800640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f41f8530 0x7f47f406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47ebe00640 1 -- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f41f8530 msgr2=0x7f47f406e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47ebe00640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f41f8530 0x7f47f406e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47ebe00640 1 -- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d8004800 msgr2=0x7f47f414f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47ebe00640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d8004800 0x7f47f414f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.942+0000 7f47ebe00640 1 -- 172.21.15.40:0/2303912103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47f4151950 con 0x7f47f41523f0 2024-04-26T23:01:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.943+0000 7f47ebe00640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f41523f0 0x7f47f41f7ff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00c1a0 tx=0x7f47f4072dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.943+0000 7f47ea000640 1 -- 172.21.15.40:0/2303912103 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc037040 con 0x7f47f41523f0 2024-04-26T23:01:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.943+0000 7f47ea000640 1 -- 172.21.15.40:0/2303912103 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47dc041070 con 0x7f47f41523f0 2024-04-26T23:01:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.943+0000 7f47f8a00640 1 -- 172.21.15.40:0/2303912103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47f41f8a40 con 0x7f47f41523f0 2024-04-26T23:01:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.943+0000 7f47ea000640 1 -- 172.21.15.40:0/2303912103 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc004a20 con 0x7f47f41523f0 2024-04-26T23:01:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.943+0000 7f47f8a00640 1 -- 172.21.15.40:0/2303912103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47f4151bf0 con 0x7f47f41523f0 2024-04-26T23:01:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.945+0000 7f47f8a00640 1 -- 172.21.15.40:0/2303912103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47f40694f0 con 0x7f47f41523f0 2024-04-26T23:01:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.945+0000 7f47ea000640 1 -- 172.21.15.40:0/2303912103 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f47dc004bc0 con 0x7f47f41523f0 2024-04-26T23:01:11.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.946+0000 7f47ea000640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f47d006c920 0x7f47d006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:11.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.946+0000 7f47ea000640 1 -- 172.21.15.40:0/2303912103 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f47dc0abc40 con 0x7f47f41523f0 2024-04-26T23:01:11.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.947+0000 7f47f1e00640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f47d006c920 0x7f47d006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.948+0000 7f47f1e00640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f47d006c920 0x7f47d006ede0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f47f406f650 tx=0x7f47e4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:11.950+0000 7f47ea000640 1 -- 172.21.15.40:0/2303912103 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f47dc079ca0 con 0x7f47f41523f0 2024-04-26T23:01:12.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.155+0000 7f47f8a00640 1 -- 172.21.15.40:0/2303912103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47f4152870 con 0x7f47f41523f0 2024-04-26T23:01:12.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.157+0000 7f47ea000640 1 -- 172.21.15.40:0/2303912103 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f47dc0ab070 con 0x7f47f41523f0 2024-04-26T23:01:12.158 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:12.158 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:11.763965+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:12.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.162+0000 7f47f8a00640 1 -- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f47d006c920 msgr2=0x7f47d006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:12.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.162+0000 7f47f8a00640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f47d006c920 0x7f47d006ede0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f47f406f650 tx=0x7f47e4008040 comp rx=0 tx=0).stop 2024-04-26T23:01:12.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.162+0000 7f47f8a00640 1 -- 172.21.15.40:0/2303912103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f41523f0 msgr2=0x7f47f41f7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:12.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.162+0000 7f47f8a00640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f41523f0 0x7f47f41f7ff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00c1a0 tx=0x7f47f4072dd0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.162+0000 7f47f8a00640 1 -- 172.21.15.40:0/2303912103 shutdown_connections 2024-04-26T23:01:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.162+0000 7f47f8a00640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f47d006c920 0x7f47d006ede0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.162+0000 7f47f8a00640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f41f8530 0x7f47f406e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.162+0000 7f47f8a00640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f41523f0 0x7f47f41f7ff0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.162+0000 7f47f8a00640 1 --2- 172.21.15.40:0/2303912103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d8004800 0x7f47f414f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.162+0000 7f47f8a00640 1 -- 172.21.15.40:0/2303912103 >> 172.21.15.40:0/2303912103 conn(0x7f47f40d0be0 msgr2=0x7f47f40b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.162+0000 7f47f8a00640 1 -- 172.21.15.40:0/2303912103 shutdown_connections 2024-04-26T23:01:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.162+0000 7f47f8a00640 1 -- 172.21.15.40:0/2303912103 wait complete. 2024-04-26T23:01:12.197 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:01:12.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.409+0000 7f016ce00640 1 Processor -- start 2024-04-26T23:01:12.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.409+0000 7f016ce00640 1 -- start start 2024-04-26T23:01:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.410+0000 7f016ce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01680d3b80 0x7f01680b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.410+0000 7f016ce00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f01680b3ff0 0x7f01680b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.411+0000 7f016ce00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01680d2d70 con 0x7f016815f540 2024-04-26T23:01:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.411+0000 7f016ce00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f016815f920 con 0x7f01680b3ff0 2024-04-26T23:01:12.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.411+0000 7f016ce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f01680d0f30 con 0x7f01680d3b80 2024-04-26T23:01:12.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.411+0000 7f0165e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f016815f540 0x7f01680d3470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51490/0 (socket says 172.21.15.40:51490) 2024-04-26T23:01:12.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.411+0000 7f0165e00640 1 -- 172.21.15.40:0/2399478289 learned_addr learned my addr 172.21.15.40:0/2399478289 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:12.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.411+0000 7f0165400640 1 --2- 172.21.15.40:0/2399478289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01680d3b80 0x7f01680b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:12.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.411+0000 7f0166800640 1 --2- 172.21.15.40:0/2399478289 >> v2:172.21.15.103:3300/0 conn(0x7f01680b3ff0 0x7f01680b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:12.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.411+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1345075303 0 0) 0x7f01680d2d70 con 0x7f016815f540 2024-04-26T23:01:12.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.411+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0148003520 con 0x7f016815f540 2024-04-26T23:01:12.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3855851827 0 0) 0x7f0148003520 con 0x7f016815f540 2024-04-26T23:01:12.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01680d3b80 msgr2=0x7f01680b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:12.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 --2- 172.21.15.40:0/2399478289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01680d3b80 0x7f01680b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 >> v2:172.21.15.103:3300/0 conn(0x7f01680b3ff0 msgr2=0x7f01680b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:12.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 --2- 172.21.15.40:0/2399478289 >> v2:172.21.15.103:3300/0 conn(0x7f01680b3ff0 0x7f01680b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01680d3050 con 0x7f016815f540 2024-04-26T23:01:12.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f0158001670 con 0x7f016815f540 2024-04-26T23:01:12.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 >> v1:172.21.15.40:6789/0 conn(0x7f016815f540 legacy=0x7f01680d3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:12.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 --2- 172.21.15.40:0/2399478289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01680d3b80 0x7f01480039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 --2- 172.21.15.40:0/2399478289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0148003f80 0x7f0148004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 --2- 172.21.15.40:0/2399478289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0148004d40 0x7f01480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f016815f920 con 0x7f0148003f80 2024-04-26T23:01:12.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01680d0f30 con 0x7f0148004d40 2024-04-26T23:01:12.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f014800b9c0 con 0x7f01680d3b80 2024-04-26T23:01:12.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0165400640 1 --2- 172.21.15.40:0/2399478289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0148003f80 0x7f0148004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:12.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.412+0000 7f0165e00640 1 --2- 172.21.15.40:0/2399478289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01680d3b80 0x7f01480039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:12.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.413+0000 7f0165400640 1 -- 172.21.15.40:0/2399478289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01680d3b80 msgr2=0x7f01480039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:12.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.413+0000 7f0165400640 1 --2- 172.21.15.40:0/2399478289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01680d3b80 0x7f01480039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.413+0000 7f0165400640 1 -- 172.21.15.40:0/2399478289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0148004d40 msgr2=0x7f01480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.413+0000 7f0165400640 1 --2- 172.21.15.40:0/2399478289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0148004d40 0x7f01480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.413+0000 7f0165400640 1 -- 172.21.15.40:0/2399478289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0148003520 con 0x7f0148003f80 2024-04-26T23:01:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.413+0000 7f0165e00640 1 --2- 172.21.15.40:0/2399478289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01680d3b80 0x7f01480039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.414+0000 7f0165400640 1 --2- 172.21.15.40:0/2399478289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0148003f80 0x7f0148004800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f015000e170 tx=0x7f0148005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa050c8a3ae2f97a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.414+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150034020 con 0x7f0148003f80 2024-04-26T23:01:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.414+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01500305f0 con 0x7f0148003f80 2024-04-26T23:01:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.414+0000 7f0164a00640 1 -- 172.21.15.40:0/2399478289 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150009c10 con 0x7f0148003f80 2024-04-26T23:01:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.414+0000 7f016ce00640 1 -- 172.21.15.40:0/2399478289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0148003f80 msgr2=0x7f0148004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.415+0000 7f016ce00640 1 --2- 172.21.15.40:0/2399478289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0148003f80 0x7f0148004800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f015000e170 tx=0x7f0148005830 comp rx=0 tx=0).stop 2024-04-26T23:01:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.415+0000 7f0166800640 1 -- 172.21.15.40:0/2399478289 reap_dead start 2024-04-26T23:01:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.415+0000 7f016ce00640 1 -- 172.21.15.40:0/2399478289 shutdown_connections 2024-04-26T23:01:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.415+0000 7f016ce00640 1 -- 172.21.15.40:0/2399478289 >> 172.21.15.40:0/2399478289 conn(0x7f01680d4f90 msgr2=0x7f01680d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.415+0000 7f016ce00640 1 -- 172.21.15.40:0/2399478289 shutdown_connections 2024-04-26T23:01:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.415+0000 7f016ce00640 1 -- 172.21.15.40:0/2399478289 wait complete. 2024-04-26T23:01:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.417+0000 7f016ce00640 1 Processor -- start 2024-04-26T23:01:12.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.417+0000 7f016ce00640 1 -- start start 2024-04-26T23:01:12.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f016ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0148003f80 0x7f0168162580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f016ce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f016815fd40 0x7f01680ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f016ce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01680ba120 0x7f01680b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f016ce00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0168068e50 con 0x7f0148003f80 2024-04-26T23:01:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f016ce00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01680d0bc0 con 0x7f016815fd40 2024-04-26T23:01:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f016ce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f01680d2780 con 0x7f01680ba120 2024-04-26T23:01:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f0165e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0148003f80 0x7f0168162580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f0165e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0148003f80 0x7f0168162580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45402/0 (socket says 172.21.15.40:45402) 2024-04-26T23:01:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f0165e00640 1 -- 172.21.15.40:0/578746904 learned_addr learned my addr 172.21.15.40:0/578746904 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f0165400640 1 --2- 172.21.15.40:0/578746904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f016815fd40 0x7f01680ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f0165e00640 1 -- 172.21.15.40:0/578746904 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01680ba120 msgr2=0x7f01680b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f0165e00640 1 --2- 172.21.15.40:0/578746904 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01680ba120 0x7f01680b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f0165e00640 1 -- 172.21.15.40:0/578746904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f016815fd40 msgr2=0x7f01680ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f0165e00640 1 --2- 172.21.15.40:0/578746904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f016815fd40 0x7f01680ce9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.418+0000 7f0165e00640 1 -- 172.21.15.40:0/578746904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01680b9080 con 0x7f0148003f80 2024-04-26T23:01:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.419+0000 7f0165e00640 1 --2- 172.21.15.40:0/578746904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0148003f80 0x7f0168162580 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f015800d940 tx=0x7f015800de00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:12.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.419+0000 7f015e000640 1 -- 172.21.15.40:0/578746904 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158019070 con 0x7f0148003f80 2024-04-26T23:01:12.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.419+0000 7f015e000640 1 -- 172.21.15.40:0/578746904 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01580074e0 con 0x7f0148003f80 2024-04-26T23:01:12.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.419+0000 7f015e000640 1 -- 172.21.15.40:0/578746904 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158004a20 con 0x7f0148003f80 2024-04-26T23:01:12.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.419+0000 7f016ce00640 1 -- 172.21.15.40:0/578746904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01680b9360 con 0x7f0148003f80 2024-04-26T23:01:12.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.419+0000 7f016ce00640 1 -- 172.21.15.40:0/578746904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01680b9920 con 0x7f0148003f80 2024-04-26T23:01:12.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.420+0000 7f016ce00640 1 -- 172.21.15.40:0/578746904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01680694f0 con 0x7f0148003f80 2024-04-26T23:01:12.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.421+0000 7f015e000640 1 -- 172.21.15.40:0/578746904 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f015801c070 con 0x7f0148003f80 2024-04-26T23:01:12.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.423+0000 7f015e000640 1 --2- 172.21.15.40:0/578746904 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f014406c8f0 0x7f014406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.423+0000 7f015e000640 1 -- 172.21.15.40:0/578746904 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f015808baa0 con 0x7f0148003f80 2024-04-26T23:01:12.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.423+0000 7f0165400640 1 --2- 172.21.15.40:0/578746904 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f014406c8f0 0x7f014406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:12.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.425+0000 7f0165400640 1 --2- 172.21.15.40:0/578746904 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f014406c8f0 0x7f014406edb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0150035000 tx=0x7f015002f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.428+0000 7f015e000640 1 -- 172.21.15.40:0/578746904 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0158059af0 con 0x7f0148003f80 2024-04-26T23:01:12.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.679+0000 7f016ce00640 1 -- 172.21.15.40:0/578746904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f016806e040 con 0x7f0148003f80 2024-04-26T23:01:12.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.680+0000 7f015e000640 1 -- 172.21.15.40:0/578746904 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5183 (secure 0 0 0) 0x7f01580593f0 con 0x7f0148003f80 2024-04-26T23:01:12.681 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:12.681 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1253648754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1253648754},{"type":"v1","addr":"172.21.15.103:6836","nonce":1253648754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3239652611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":3239652611},{"type":"v1","addr":"172.21.15.103:6837","nonce":3239652611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3670144900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3670144900},{"type":"v1","addr":"172.21.15.40:6833","nonce":3670144900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:01:05.171276+0000","modified":"2024-04-26T23:01:10.254243+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3989190340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3989190340},{"type":"v1","addr":"172.21.15.40:6835","nonce":3989190340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-26T23:01:12.681 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 7 2024-04-26T23:01:12.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.683+0000 7f016ce00640 1 -- 172.21.15.40:0/578746904 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f014406c8f0 msgr2=0x7f014406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:12.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.683+0000 7f016ce00640 1 --2- 172.21.15.40:0/578746904 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f014406c8f0 0x7f014406edb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0150035000 tx=0x7f015002f000 comp rx=0 tx=0).stop 2024-04-26T23:01:12.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.684+0000 7f016ce00640 1 -- 172.21.15.40:0/578746904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0148003f80 msgr2=0x7f0168162580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:12.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.684+0000 7f016ce00640 1 --2- 172.21.15.40:0/578746904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0148003f80 0x7f0168162580 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f015800d940 tx=0x7f015800de00 comp rx=0 tx=0).stop 2024-04-26T23:01:12.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.684+0000 7f016ce00640 1 -- 172.21.15.40:0/578746904 shutdown_connections 2024-04-26T23:01:12.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.684+0000 7f016ce00640 1 --2- 172.21.15.40:0/578746904 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f014406c8f0 0x7f014406edb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.684+0000 7f016ce00640 1 --2- 172.21.15.40:0/578746904 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01680ba120 0x7f01680b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.684+0000 7f016ce00640 1 --2- 172.21.15.40:0/578746904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f016815fd40 0x7f01680ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.684+0000 7f016ce00640 1 --2- 172.21.15.40:0/578746904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0148003f80 0x7f0168162580 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.684+0000 7f016ce00640 1 -- 172.21.15.40:0/578746904 >> 172.21.15.40:0/578746904 conn(0x7f01680d4f90 msgr2=0x7f01680d55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:12.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.684+0000 7f016ce00640 1 -- 172.21.15.40:0/578746904 shutdown_connections 2024-04-26T23:01:12.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.684+0000 7f016ce00640 1 -- 172.21.15.40:0/578746904 wait complete. 2024-04-26T23:01:12.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:12.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.918+0000 7f817f200640 1 Processor -- start 2024-04-26T23:01:12.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.918+0000 7f817f200640 1 -- start start 2024-04-26T23:01:12.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.919+0000 7f817f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81800cfc70 0x7f8180162dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.920+0000 7f817f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8180163770 0x7f818015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.920+0000 7f817f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8180163c00 con 0x7f8180163310 2024-04-26T23:01:12.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.920+0000 7f817f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f818006dd60 con 0x7f8180163770 2024-04-26T23:01:12.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.920+0000 7f817f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f818006e040 con 0x7f81800cfc70 2024-04-26T23:01:12.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.920+0000 7f817de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81800cfc70 0x7f8180162dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:12.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.920+0000 7f817de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81800cfc70 0x7f8180162dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:53096/0 (socket says 172.21.15.40:53096) 2024-04-26T23:01:12.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.920+0000 7f817de00640 1 -- 172.21.15.40:0/2566704121 learned_addr learned my addr 172.21.15.40:0/2566704121 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:12.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.921+0000 7f817e800640 1 --2- 172.21.15.40:0/2566704121 >> v2:172.21.15.103:3300/0 conn(0x7f8180163770 0x7f818015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:12.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.921+0000 7f817ca00640 1 -- 172.21.15.40:0/2566704121 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2684294428 0 0) 0x7f816c001490 con 0x7f8180163310 2024-04-26T23:01:12.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.921+0000 7f817ca00640 1 -- 172.21.15.40:0/2566704121 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8164003520 con 0x7f8180163310 2024-04-26T23:01:12.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.921+0000 7f817de00640 1 -- 172.21.15.40:0/2566704121 >> v2:172.21.15.103:3300/0 conn(0x7f8180163770 msgr2=0x7f818015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:12.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.921+0000 7f817de00640 1 --2- 172.21.15.40:0/2566704121 >> v2:172.21.15.103:3300/0 conn(0x7f8180163770 0x7f818015ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.921+0000 7f817de00640 1 -- 172.21.15.40:0/2566704121 >> v1:172.21.15.103:6789/0 conn(0x7f8180163310 legacy=0x7f818015e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:12.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.921+0000 7f817de00640 1 -- 172.21.15.40:0/2566704121 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818006d690 con 0x7f81800cfc70 2024-04-26T23:01:12.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.923+0000 7f817de00640 1 --2- 172.21.15.40:0/2566704121 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81800cfc70 0x7f8180162dd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8170009870 tx=0x7f817002a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93319568ebcc4de1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:12.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.923+0000 7f817ca00640 1 -- 172.21.15.40:0/2566704121 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8170037070 con 0x7f81800cfc70 2024-04-26T23:01:12.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.923+0000 7f817ca00640 1 -- 172.21.15.40:0/2566704121 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8170032440 con 0x7f81800cfc70 2024-04-26T23:01:12.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.923+0000 7f817ca00640 1 -- 172.21.15.40:0/2566704121 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8170032810 con 0x7f81800cfc70 2024-04-26T23:01:12.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.924+0000 7f817f200640 1 -- 172.21.15.40:0/2566704121 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81800cfc70 msgr2=0x7f8180162dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:12.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.924+0000 7f817f200640 1 --2- 172.21.15.40:0/2566704121 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81800cfc70 0x7f8180162dd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8170009870 tx=0x7f817002a1c0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.924+0000 7f817f200640 1 -- 172.21.15.40:0/2566704121 shutdown_connections 2024-04-26T23:01:12.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.924+0000 7f817f200640 1 --2- 172.21.15.40:0/2566704121 >> v2:172.21.15.103:3300/0 conn(0x7f8180163770 0x7f818015ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.924+0000 7f817f200640 1 --2- 172.21.15.40:0/2566704121 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f81800cfc70 0x7f8180162dd0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.924+0000 7f817f200640 1 -- 172.21.15.40:0/2566704121 >> 172.21.15.40:0/2566704121 conn(0x7f81800d2c10 msgr2=0x7f81800d3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:12.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.924+0000 7f817f200640 1 -- 172.21.15.40:0/2566704121 shutdown_connections 2024-04-26T23:01:12.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.924+0000 7f817f200640 1 -- 172.21.15.40:0/2566704121 wait complete. 2024-04-26T23:01:12.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.926+0000 7f817f200640 1 Processor -- start 2024-04-26T23:01:12.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.926+0000 7f817f200640 1 -- start start 2024-04-26T23:01:12.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.927+0000 7f817f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81800cfc70 0x7f81800cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.927+0000 7f817f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8180163310 0x7f81800b8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.927+0000 7f817f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8180163770 0x7f81801e9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.927+0000 7f817f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f81800d2530 con 0x7f81800cfc70 2024-04-26T23:01:12.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.927+0000 7f817f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f818006e4d0 con 0x7f8180163310 2024-04-26T23:01:12.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.927+0000 7f817f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f81800d26b0 con 0x7f8180163770 2024-04-26T23:01:12.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.927+0000 7f817de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81800cfc70 0x7f81800cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:12.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.927+0000 7f817de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81800cfc70 0x7f81800cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45408/0 (socket says 172.21.15.40:45408) 2024-04-26T23:01:12.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.927+0000 7f817de00640 1 -- 172.21.15.40:0/3442409184 learned_addr learned my addr 172.21.15.40:0/3442409184 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:12.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.928+0000 7f817d400640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8180163310 0x7f81800b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:12.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.928+0000 7f817e800640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8180163770 0x7f81801e9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:12.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.928+0000 7f817de00640 1 -- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8180163770 msgr2=0x7f81801e9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:12.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.928+0000 7f817de00640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8180163770 0x7f81801e9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.928+0000 7f817de00640 1 -- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8180163310 msgr2=0x7f81800b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:12.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.928+0000 7f817de00640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8180163310 0x7f81800b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:12.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.928+0000 7f817de00640 1 -- 172.21.15.40:0/3442409184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81800b6cd0 con 0x7f81800cfc70 2024-04-26T23:01:12.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.928+0000 7f817de00640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81800cfc70 0x7f81800cd4b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8170000c00 tx=0x7f81700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:12.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.929+0000 7f8176000640 1 -- 172.21.15.40:0/3442409184 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8170041020 con 0x7f81800cfc70 2024-04-26T23:01:12.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.929+0000 7f8176000640 1 -- 172.21.15.40:0/3442409184 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f817003d6e0 con 0x7f81800cfc70 2024-04-26T23:01:12.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.929+0000 7f8176000640 1 -- 172.21.15.40:0/3442409184 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8170037040 con 0x7f81800cfc70 2024-04-26T23:01:12.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.929+0000 7f817f200640 1 -- 172.21.15.40:0/3442409184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81800b6fb0 con 0x7f81800cfc70 2024-04-26T23:01:12.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.929+0000 7f817f200640 1 -- 172.21.15.40:0/3442409184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81800b7570 con 0x7f81800cfc70 2024-04-26T23:01:12.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.930+0000 7f817f200640 1 -- 172.21.15.40:0/3442409184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8180069a80 con 0x7f81800cfc70 2024-04-26T23:01:12.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.931+0000 7f8176000640 1 -- 172.21.15.40:0/3442409184 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8170044020 con 0x7f81800cfc70 2024-04-26T23:01:12.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.932+0000 7f8176000640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f816006c920 0x7f816006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:12.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.932+0000 7f8176000640 1 -- 172.21.15.40:0/3442409184 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f81700ac4e0 con 0x7f81800cfc70 2024-04-26T23:01:12.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.933+0000 7f817d400640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f816006c920 0x7f816006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:12.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.934+0000 7f817d400640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f816006c920 0x7f816006ede0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f81800c9c20 tx=0x7f816c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:12.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:12.937+0000 7f8176000640 1 -- 172.21.15.40:0/3442409184 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f817004b030 con 0x7f81800cfc70 2024-04-26T23:01:13.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.143+0000 7f817f200640 1 -- 172.21.15.40:0/3442409184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81800ce130 con 0x7f81800cfc70 2024-04-26T23:01:13.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.146+0000 7f8176000640 1 -- 172.21.15.40:0/3442409184 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f817007a530 con 0x7f81800cfc70 2024-04-26T23:01:13.147 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:13.147 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:11.763965+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:13.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.149+0000 7f817f200640 1 -- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f816006c920 msgr2=0x7f816006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:13.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.149+0000 7f817f200640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f816006c920 0x7f816006ede0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f81800c9c20 tx=0x7f816c009040 comp rx=0 tx=0).stop 2024-04-26T23:01:13.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.149+0000 7f817f200640 1 -- 172.21.15.40:0/3442409184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81800cfc70 msgr2=0x7f81800cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.149+0000 7f817f200640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81800cfc70 0x7f81800cd4b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8170000c00 tx=0x7f81700046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.149+0000 7f817f200640 1 -- 172.21.15.40:0/3442409184 shutdown_connections 2024-04-26T23:01:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.149+0000 7f817f200640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f816006c920 0x7f816006ede0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.149+0000 7f817f200640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8180163770 0x7f81801e9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.149+0000 7f817f200640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8180163310 0x7f81800b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.149+0000 7f817f200640 1 --2- 172.21.15.40:0/3442409184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81800cfc70 0x7f81800cd4b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.149+0000 7f817f200640 1 -- 172.21.15.40:0/3442409184 >> 172.21.15.40:0/3442409184 conn(0x7f81800d2c10 msgr2=0x7f8180071430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.149+0000 7f817f200640 1 -- 172.21.15.40:0/3442409184 shutdown_connections 2024-04-26T23:01:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.149+0000 7f817f200640 1 -- 172.21.15.40:0/3442409184 wait complete. 2024-04-26T23:01:13.571 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:01:05.171276+0000', 'modified': '2024-04-26T23:01:10.254243+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/3989190340', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 3989190340}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 3989190340}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4257, 'qdb_cluster': [4257]} 2024-04-26T23:01:13.571 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-26T23:01:13.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T23:01:13.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.797+0000 7f592c600640 1 Processor -- start 2024-04-26T23:01:13.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.797+0000 7f592c600640 1 -- start start 2024-04-26T23:01:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.799+0000 7f592c600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f592415f540 0x7f59240d3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.799+0000 7f592c600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59240d2d70 con 0x7f59240b3ff0 2024-04-26T23:01:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.799+0000 7f592c600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59240d3050 con 0x7f59240b5ab0 2024-04-26T23:01:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.799+0000 7f592c600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f592415f920 con 0x7f592415f540 2024-04-26T23:01:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.799+0000 7f5929200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f59240b3ff0 0x7f59240d1770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51494/0 (socket says 172.21.15.40:51494) 2024-04-26T23:01:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.799+0000 7f5929200640 1 -- 172.21.15.40:0/1475111565 learned_addr learned my addr 172.21.15.40:0/1475111565 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:13.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.799+0000 7f5929c00640 1 --2- 172.21.15.40:0/1475111565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f592415f540 0x7f59240d3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.799+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4000765154 0 0) 0x7f59240d2d70 con 0x7f59240b3ff0 2024-04-26T23:01:13.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f590c003520 con 0x7f59240b3ff0 2024-04-26T23:01:13.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 933327248 0 0) 0x7f590c003520 con 0x7f59240b3ff0 2024-04-26T23:01:13.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f592415f540 msgr2=0x7f59240d3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:13.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 --2- 172.21.15.40:0/1475111565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f592415f540 0x7f59240d3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 >> v1:172.21.15.103:6789/0 conn(0x7f59240b5ab0 legacy=0x7f59240d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:13.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59240d08e0 con 0x7f59240b3ff0 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f5910001670 con 0x7f59240b3ff0 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 >> v1:172.21.15.40:6789/0 conn(0x7f59240b3ff0 legacy=0x7f59240d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 --2- 172.21.15.40:0/1475111565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f590c0039e0 0x7f590c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 --2- 172.21.15.40:0/1475111565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f590c004800 0x7f590c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 --2- 172.21.15.40:0/1475111565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f590c0051c0 0x7f590c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f592415f920 con 0x7f590c0051c0 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f590c00bfe0 con 0x7f590c0039e0 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f590c00c1e0 con 0x7f590c004800 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.800+0000 7f592a600640 1 --2- 172.21.15.40:0/1475111565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f590c0051c0 0x7f590c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.801+0000 7f5929200640 1 --2- 172.21.15.40:0/1475111565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f590c004800 0x7f590c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.801+0000 7f5929c00640 1 --2- 172.21.15.40:0/1475111565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f590c0039e0 0x7f590c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.801+0000 7f592a600640 1 -- 172.21.15.40:0/1475111565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f590c004800 msgr2=0x7f590c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.801+0000 7f592a600640 1 --2- 172.21.15.40:0/1475111565 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f590c004800 0x7f590c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.801+0000 7f592a600640 1 -- 172.21.15.40:0/1475111565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f590c0039e0 msgr2=0x7f590c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:13.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.801+0000 7f592a600640 1 --2- 172.21.15.40:0/1475111565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f590c0039e0 0x7f590c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.801+0000 7f592a600640 1 -- 172.21.15.40:0/1475111565 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590c003520 con 0x7f590c0051c0 2024-04-26T23:01:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.802+0000 7f592a600640 1 --2- 172.21.15.40:0/1475111565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f590c0051c0 0x7f590c00b9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f59240b5f40 tx=0x7f590c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166c6d537749ad4a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.803+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c001a70 con 0x7f590c0051c0 2024-04-26T23:01:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.803+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f591c02ae10 con 0x7f590c0051c0 2024-04-26T23:01:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.803+0000 7f5923e00640 1 -- 172.21.15.40:0/1475111565 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c003bb0 con 0x7f590c0051c0 2024-04-26T23:01:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.803+0000 7f592c600640 1 -- 172.21.15.40:0/1475111565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f590c0051c0 msgr2=0x7f590c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.803+0000 7f592c600640 1 --2- 172.21.15.40:0/1475111565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f590c0051c0 0x7f590c00b9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f59240b5f40 tx=0x7f590c005830 comp rx=0 tx=0).stop 2024-04-26T23:01:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.803+0000 7f592a600640 1 -- 172.21.15.40:0/1475111565 reap_dead start 2024-04-26T23:01:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.803+0000 7f592c600640 1 -- 172.21.15.40:0/1475111565 shutdown_connections 2024-04-26T23:01:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.803+0000 7f592c600640 1 -- 172.21.15.40:0/1475111565 >> 172.21.15.40:0/1475111565 conn(0x7f59240d4f90 msgr2=0x7f59240d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.804+0000 7f592c600640 1 -- 172.21.15.40:0/1475111565 shutdown_connections 2024-04-26T23:01:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.804+0000 7f592c600640 1 -- 172.21.15.40:0/1475111565 wait complete. 2024-04-26T23:01:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.805+0000 7f592c600640 1 Processor -- start 2024-04-26T23:01:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.806+0000 7f592c600640 1 -- start start 2024-04-26T23:01:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.806+0000 7f592c600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f590c0051c0 0x7f59241f92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.806+0000 7f592c600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5924151b10 0x7f59240c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.807+0000 7f592c600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59240ce070 0x7f59240ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.807+0000 7f592c600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5924068e50 con 0x7f5924151b10 2024-04-26T23:01:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.807+0000 7f592c600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f59240d0f30 con 0x7f590c0051c0 2024-04-26T23:01:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.807+0000 7f592c600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f59240d2590 con 0x7f59240ce070 2024-04-26T23:01:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.807+0000 7f5929200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5924151b10 0x7f59240c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.807+0000 7f5929200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5924151b10 0x7f59240c9b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45434/0 (socket says 172.21.15.40:45434) 2024-04-26T23:01:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.807+0000 7f5929200640 1 -- 172.21.15.40:0/3842688946 learned_addr learned my addr 172.21.15.40:0/3842688946 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.807+0000 7f5929c00640 1 --2- 172.21.15.40:0/3842688946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f590c0051c0 0x7f59241f92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.807+0000 7f5929200640 1 -- 172.21.15.40:0/3842688946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59240ce070 msgr2=0x7f59240ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.807+0000 7f5929200640 1 --2- 172.21.15.40:0/3842688946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59240ce070 0x7f59240ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.807+0000 7f5929200640 1 -- 172.21.15.40:0/3842688946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f590c0051c0 msgr2=0x7f59241f92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.807+0000 7f5929200640 1 --2- 172.21.15.40:0/3842688946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f590c0051c0 0x7f59241f92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.807+0000 7f5929200640 1 -- 172.21.15.40:0/3842688946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59240ca050 con 0x7f5924151b10 2024-04-26T23:01:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.808+0000 7f5929200640 1 --2- 172.21.15.40:0/3842688946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5924151b10 0x7f59240c9b10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5910007320 tx=0x7f59100077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.808+0000 7f5921600640 1 -- 172.21.15.40:0/3842688946 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591001a070 con 0x7f5924151b10 2024-04-26T23:01:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.808+0000 7f5921600640 1 -- 172.21.15.40:0/3842688946 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f591000f920 con 0x7f5924151b10 2024-04-26T23:01:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.808+0000 7f592c600640 1 -- 172.21.15.40:0/3842688946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59240ca320 con 0x7f5924151b10 2024-04-26T23:01:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.808+0000 7f5921600640 1 -- 172.21.15.40:0/3842688946 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591000fcf0 con 0x7f5924151b10 2024-04-26T23:01:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.808+0000 7f592c600640 1 -- 172.21.15.40:0/3842688946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59240ca8e0 con 0x7f5924151b10 2024-04-26T23:01:13.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.811+0000 7f5921600640 1 -- 172.21.15.40:0/3842688946 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f591001d070 con 0x7f5924151b10 2024-04-26T23:01:13.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.812+0000 7f5921600640 1 --2- 172.21.15.40:0/3842688946 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f590406c8f0 0x7f590406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.812+0000 7f5921600640 1 -- 172.21.15.40:0/3842688946 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f591008b7e0 con 0x7f5924151b10 2024-04-26T23:01:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.813+0000 7f592f583740 1 Processor -- start 2024-04-26T23:01:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.813+0000 7f5929c00640 1 --2- 172.21.15.40:0/3842688946 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f590406c8f0 0x7f590406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.813+0000 7f592f583740 1 -- start start 2024-04-26T23:01:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.814+0000 7f592f583740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8055e0 0x55b74d807740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.814+0000 7f592f583740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d841cb0 0x55b74d8059c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.814+0000 7f592f583740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8300a0 0x55b74d830520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.814+0000 7f592f583740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b74d769600 con 0x55b74d8055e0 2024-04-26T23:01:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.814+0000 7f592f583740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b74d706100 con 0x55b74d8300a0 2024-04-26T23:01:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.814+0000 7f592f583740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b74d7655c0 con 0x55b74d841cb0 2024-04-26T23:01:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.814+0000 7f5929c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8055e0 0x55b74d807740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.814+0000 7f5929c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8055e0 0x55b74d807740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45448/0 (socket says 172.21.15.40:45448) 2024-04-26T23:01:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.814+0000 7f5929c00640 1 -- 172.21.15.40:0/2454528727 learned_addr learned my addr 172.21.15.40:0/2454528727 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.814+0000 7f5929c00640 1 --2- 172.21.15.40:0/3842688946 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f590406c8f0 0x7f590406edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f592408d6d0 tx=0x7f5918008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.815+0000 7f5929200640 1 --2- 172.21.15.40:0/2454528727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d841cb0 0x55b74d8059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.815+0000 7f5929c00640 1 -- 172.21.15.40:0/2454528727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d841cb0 msgr2=0x55b74d8059c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.815+0000 7f5929c00640 1 --2- 172.21.15.40:0/2454528727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d841cb0 0x55b74d8059c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.815+0000 7f5929c00640 1 -- 172.21.15.40:0/2454528727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8300a0 msgr2=0x55b74d830520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.815+0000 7f5929c00640 1 --2- 172.21.15.40:0/2454528727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8300a0 0x55b74d830520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.815+0000 7f5929c00640 1 -- 172.21.15.40:0/2454528727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b74d47b2d0 con 0x55b74d8055e0 2024-04-26T23:01:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.815+0000 7f5929c00640 1 --2- 172.21.15.40:0/2454528727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8055e0 0x55b74d807740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5918012c10 tx=0x7f5918015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf6ad2ff9edeed3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:13.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.815+0000 7f5916000640 1 -- 172.21.15.40:0/2454528727 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5918015ca0 con 0x55b74d8055e0 2024-04-26T23:01:13.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.815+0000 7f5916000640 1 -- 172.21.15.40:0/2454528727 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5918027070 con 0x55b74d8055e0 2024-04-26T23:01:13.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.815+0000 7f5916000640 1 -- 172.21.15.40:0/2454528727 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5918022640 con 0x55b74d8055e0 2024-04-26T23:01:13.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.816+0000 7f592f583740 1 -- 172.21.15.40:0/2454528727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8055e0 msgr2=0x55b74d807740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:13.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.816+0000 7f592f583740 1 --2- 172.21.15.40:0/2454528727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8055e0 0x55b74d807740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5918012c10 tx=0x7f5918015000 comp rx=0 tx=0).stop 2024-04-26T23:01:13.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.816+0000 7f592f583740 1 -- 172.21.15.40:0/2454528727 shutdown_connections 2024-04-26T23:01:13.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.816+0000 7f592f583740 1 --2- 172.21.15.40:0/2454528727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8300a0 0x55b74d830520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.816+0000 7f592f583740 1 --2- 172.21.15.40:0/2454528727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d841cb0 0x55b74d8059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.816+0000 7f592f583740 1 --2- 172.21.15.40:0/2454528727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8055e0 0x55b74d807740 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.816+0000 7f592f583740 1 -- 172.21.15.40:0/2454528727 >> 172.21.15.40:0/2454528727 conn(0x55b74d799d10 msgr2=0x55b74d84b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:13.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.816+0000 7f592f583740 1 -- 172.21.15.40:0/2454528727 shutdown_connections 2024-04-26T23:01:13.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.816+0000 7f592f583740 1 -- 172.21.15.40:0/2454528727 wait complete. 2024-04-26T23:01:13.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.817+0000 7f592f583740 1 -- start start 2024-04-26T23:01:13.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.817+0000 7f592f583740 1 Processor -- start 2024-04-26T23:01:13.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.818+0000 7f592f583740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8055e0 0x55b74d8455b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.818+0000 7f592f583740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d8300a0 0x55b74d846b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.818+0000 7f592f583740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 0x55b74d847080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.818+0000 7f592f583740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b74d6f0f70 con 0x55b74d841cb0 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.818+0000 7f592f583740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b74d82dbf0 con 0x55b74d8055e0 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.818+0000 7f592f583740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b74d831de0 con 0x55b74d8300a0 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.818+0000 7f592f583740 10 client.0 _ll_register_callbacks cb 0x55b74d75f480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.819+0000 7f592a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 0x55b74d847080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.819+0000 7f592a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 0x55b74d847080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45462/0 (socket says 172.21.15.40:45462) 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.819+0000 7f592a600640 1 -- 172.21.15.40:0/623073507 learned_addr learned my addr 172.21.15.40:0/623073507 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.819+0000 7f5929200640 1 --2- 172.21.15.40:0/623073507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d8300a0 0x55b74d846b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.819+0000 7f5929c00640 1 --2- 172.21.15.40:0/623073507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8055e0 0x55b74d8455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.819+0000 7f592a600640 1 -- 172.21.15.40:0/623073507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d8300a0 msgr2=0x55b74d846b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.819+0000 7f592a600640 1 --2- 172.21.15.40:0/623073507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d8300a0 0x55b74d846b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.819+0000 7f592a600640 1 -- 172.21.15.40:0/623073507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8055e0 msgr2=0x55b74d8455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.819+0000 7f592a600640 1 --2- 172.21.15.40:0/623073507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8055e0 0x55b74d8455b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.819+0000 7f592a600640 1 -- 172.21.15.40:0/623073507 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b74d5456b0 con 0x55b74d841cb0 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.819+0000 7f592a600640 1 --2- 172.21.15.40:0/623073507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 0x55b74d847080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f591c001f10 tx=0x7f591c001390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deafdcec5ce656c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:13.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.819+0000 7f590aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.820+0000 7f590aa00640 1 -- 172.21.15.40:0/623073507 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c03e030 con 0x55b74d841cb0 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.820+0000 7f590aa00640 1 -- 172.21.15.40:0/623073507 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f591c037040 con 0x55b74d841cb0 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.820+0000 7f592f583740 1 -- 172.21.15.40:0/623073507 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5918027070 con 0x55b74d841cb0 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.820+0000 7f590aa00640 1 -- 172.21.15.40:0/623073507 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f591c047050 con 0x55b74d841cb0 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.820+0000 7f592f583740 10 client.4416 fetch_fsmap learned FSMap version 7 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.820+0000 7f592f583740 1 -- 172.21.15.40:0/623073507 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55b74d831b20 con 0x55b74d841cb0 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.820+0000 7f590aa00640 1 -- 172.21.15.40:0/623073507 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5918027070 con 0x55b74d841cb0 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.820+0000 7f590aa00640 1 -- 172.21.15.40:0/623073507 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7f591c04a020 con 0x55b74d841cb0 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.820+0000 7f592f583740 10 client.4416 fetch_fsmap finished waiting for FSMap version 7 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.820+0000 7f592f583740 10 client.4416 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.820+0000 7f592f583740 20 client.4416 populate_metadata read hostname 'smithi040' 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.821+0000 7f592f583740 1 --2- 172.21.15.40:0/623073507 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x55b74d85e720 0x55b74d831120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.821+0000 7f592f583740 4 client.4416 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:01:13.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.821+0000 7f592f583740 1 -- 172.21.15.40:0/623073507 --> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b74d777210 con 0x55b74d85e720 2024-04-26T23:01:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.821+0000 7f5929c00640 1 --2- 172.21.15.40:0/623073507 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x55b74d85e720 0x55b74d831120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.821+0000 7f5929c00640 1 --2- 172.21.15.40:0/623073507 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x55b74d85e720 0x55b74d831120 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0ec8027c9f4984c server_cookie=9c4361b3267b8438 in_seq=0 out_seq=0 2024-04-26T23:01:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.821+0000 7f590aa00640 10 client.4416 ms_handle_connect on v2:172.21.15.40:6834/3989190340 2024-04-26T23:01:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.822+0000 7f590aa00640 1 -- 172.21.15.40:0/623073507 <== mds.0 v2:172.21.15.40:6834/3989190340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b74d777210 con 0x55b74d85e720 2024-04-26T23:01:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.822+0000 7f590aa00640 10 client.4416 handle_command_reply: tid=0 2024-04-26T23:01:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.822+0000 7f592f583740 1 client.4416 shutdown 2024-04-26T23:01:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.823+0000 7f592f583740 1 -- 172.21.15.40:0/623073507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 msgr2=0x55b74d847080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.823+0000 7f592f583740 1 --2- 172.21.15.40:0/623073507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 0x55b74d847080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f591c001f10 tx=0x7f591c001390 comp rx=0 tx=0).stop 2024-04-26T23:01:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.823+0000 7f592f583740 1 -- 172.21.15.40:0/623073507 shutdown_connections 2024-04-26T23:01:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.823+0000 7f592f583740 1 --2- 172.21.15.40:0/623073507 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x55b74d85e720 0x55b74d831120 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.823+0000 7f592f583740 1 --2- 172.21.15.40:0/623073507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 0x55b74d847080 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.823+0000 7f592f583740 1 --2- 172.21.15.40:0/623073507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d8300a0 0x55b74d846b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.823+0000 7f592f583740 1 --2- 172.21.15.40:0/623073507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8055e0 0x55b74d8455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.823+0000 7f590aa00640 0 client.4416 ms_handle_reset on v2:172.21.15.40:6834/3989190340 2024-04-26T23:01:13.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.823+0000 7f592f583740 1 -- 172.21.15.40:0/623073507 >> 172.21.15.40:0/623073507 conn(0x55b74d799d10 msgr2=0x55b74d82d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:13.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.823+0000 7f592f583740 1 -- 172.21.15.40:0/623073507 shutdown_connections 2024-04-26T23:01:13.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.823+0000 7f592f583740 1 -- 172.21.15.40:0/623073507 wait complete. 2024-04-26T23:01:13.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.823+0000 7f592f583740 20 client.4416 trim_cache size 0 max 16384 2024-04-26T23:01:13.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.843+0000 7f592f583740 1 Processor -- start 2024-04-26T23:01:13.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.843+0000 7f592f583740 1 -- start start 2024-04-26T23:01:13.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.844+0000 7f592f583740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d8055e0 0x55b74d847a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.844+0000 7f592f583740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8300a0 0x55b74d8450c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.844+0000 7f592f583740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 0x55b74d86d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.844+0000 7f592f583740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f59242024e0 con 0x55b74d841cb0 2024-04-26T23:01:13.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.844+0000 7f592f583740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5924153ac0 con 0x55b74d8300a0 2024-04-26T23:01:13.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.844+0000 7f592f583740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5924201e90 con 0x55b74d8055e0 2024-04-26T23:01:13.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.845+0000 7f5929200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8300a0 0x55b74d8450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.845+0000 7f592a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 0x55b74d86d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.845+0000 7f5929c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d8055e0 0x55b74d847a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.845+0000 7f592a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 0x55b74d86d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45466/0 (socket says 172.21.15.40:45466) 2024-04-26T23:01:13.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.845+0000 7f592a600640 1 -- 172.21.15.40:0/1018535096 learned_addr learned my addr 172.21.15.40:0/1018535096 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:13.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.845+0000 7f592a600640 1 -- 172.21.15.40:0/1018535096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d8055e0 msgr2=0x55b74d847a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:13.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.845+0000 7f592a600640 1 --2- 172.21.15.40:0/1018535096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d8055e0 0x55b74d847a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.845+0000 7f592a600640 1 -- 172.21.15.40:0/1018535096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8300a0 msgr2=0x55b74d8450c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:13.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.845+0000 7f592a600640 1 --2- 172.21.15.40:0/1018535096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8300a0 0x55b74d8450c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.845+0000 7f592a600640 1 -- 172.21.15.40:0/1018535096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b74d845b90 con 0x55b74d841cb0 2024-04-26T23:01:13.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.845+0000 7f592a600640 1 --2- 172.21.15.40:0/1018535096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 0x55b74d86d6e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f591c007f20 tx=0x7f591c034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f5a8d8da6cf193 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:13.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.846+0000 7f5915600640 1 -- 172.21.15.40:0/1018535096 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c03e040 con 0x55b74d841cb0 2024-04-26T23:01:13.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.846+0000 7f5915600640 1 -- 172.21.15.40:0/1018535096 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f591c005280 con 0x55b74d841cb0 2024-04-26T23:01:13.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.846+0000 7f5915600640 1 -- 172.21.15.40:0/1018535096 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c005670 con 0x55b74d841cb0 2024-04-26T23:01:13.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.846+0000 7f592f583740 1 -- 172.21.15.40:0/1018535096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 msgr2=0x55b74d86d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:13.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.846+0000 7f592f583740 1 --2- 172.21.15.40:0/1018535096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 0x55b74d86d6e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f591c007f20 tx=0x7f591c034000 comp rx=0 tx=0).stop 2024-04-26T23:01:13.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.846+0000 7f592f583740 1 -- 172.21.15.40:0/1018535096 shutdown_connections 2024-04-26T23:01:13.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.846+0000 7f592f583740 1 --2- 172.21.15.40:0/1018535096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d841cb0 0x55b74d86d6e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.846+0000 7f592f583740 1 --2- 172.21.15.40:0/1018535096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8300a0 0x55b74d8450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.846+0000 7f592f583740 1 --2- 172.21.15.40:0/1018535096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d8055e0 0x55b74d847a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.846+0000 7f592f583740 1 -- 172.21.15.40:0/1018535096 >> 172.21.15.40:0/1018535096 conn(0x55b74d799d10 msgr2=0x55b74d85b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:13.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.846+0000 7f592f583740 1 -- 172.21.15.40:0/1018535096 shutdown_connections 2024-04-26T23:01:13.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.846+0000 7f592f583740 1 -- 172.21.15.40:0/1018535096 wait complete. 2024-04-26T23:01:13.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.847+0000 7f592f583740 1 -- start start 2024-04-26T23:01:13.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.847+0000 7f592f583740 1 Processor -- start 2024-04-26T23:01:13.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.848+0000 7f592f583740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8055e0 0x55b74d846b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.848+0000 7f592f583740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8300a0 0x55b74d847080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.848+0000 7f592f583740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d841cb0 0x55b74d820ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.848+0000 7f592f583740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b74d85e5a0 con 0x55b74d8300a0 2024-04-26T23:01:13.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.848+0000 7f592f583740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b74d821de0 con 0x55b74d8055e0 2024-04-26T23:01:13.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.848+0000 7f592f583740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b74d85ebf0 con 0x55b74d841cb0 2024-04-26T23:01:13.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.848+0000 7f592f583740 10 client.0 _ll_register_callbacks cb 0x7f590c0036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:01:13.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.848+0000 7f5929200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8300a0 0x55b74d847080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f5929200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8300a0 0x55b74d847080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45476/0 (socket says 172.21.15.40:45476) 2024-04-26T23:01:13.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f5929200640 1 -- 172.21.15.40:0/3868809029 learned_addr learned my addr 172.21.15.40:0/3868809029 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:13.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f592a600640 1 --2- 172.21.15.40:0/3868809029 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d841cb0 0x55b74d820ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f5929200640 1 -- 172.21.15.40:0/3868809029 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d841cb0 msgr2=0x55b74d820ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:13.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f5929200640 1 --2- 172.21.15.40:0/3868809029 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d841cb0 0x55b74d820ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f5929200640 1 -- 172.21.15.40:0/3868809029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8055e0 msgr2=0x55b74d846b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:13.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f5929200640 1 --2- 172.21.15.40:0/3868809029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8055e0 0x55b74d846b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f5929200640 1 -- 172.21.15.40:0/3868809029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b74d8c5440 con 0x55b74d8300a0 2024-04-26T23:01:13.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f592a600640 1 --2- 172.21.15.40:0/3868809029 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d841cb0 0x55b74d820ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:13.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f5929200640 1 --2- 172.21.15.40:0/3868809029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8300a0 0x55b74d847080 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5910059150 tx=0x7f5910055580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e8c705cb25018a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:13.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f590aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:01:13.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f590aa00640 1 -- 172.21.15.40:0/3868809029 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591006ddd0 con 0x55b74d8300a0 2024-04-26T23:01:13.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f590aa00640 1 -- 172.21.15.40:0/3868809029 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5910086e00 con 0x55b74d8300a0 2024-04-26T23:01:13.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.849+0000 7f592f583740 1 -- 172.21.15.40:0/3868809029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f591c005280 con 0x55b74d8300a0 2024-04-26T23:01:13.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.850+0000 7f590aa00640 1 -- 172.21.15.40:0/3868809029 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f5910076050 con 0x55b74d8300a0 2024-04-26T23:01:13.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.850+0000 7f592f583740 10 client.4422 fetch_fsmap learned FSMap version 7 2024-04-26T23:01:13.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.850+0000 7f592f583740 1 -- 172.21.15.40:0/3868809029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55b74d8c5e80 con 0x55b74d8300a0 2024-04-26T23:01:13.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.850+0000 7f590aa00640 1 -- 172.21.15.40:0/3868809029 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f591c005280 con 0x55b74d8300a0 2024-04-26T23:01:13.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.850+0000 7f590aa00640 1 -- 172.21.15.40:0/3868809029 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7f5910061910 con 0x55b74d8300a0 2024-04-26T23:01:13.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.850+0000 7f592f583740 10 client.4422 fetch_fsmap finished waiting for FSMap version 7 2024-04-26T23:01:13.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.850+0000 7f592f583740 10 client.4422 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.850+0000 7f592f583740 20 client.4422 populate_metadata read hostname 'smithi040' 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.850+0000 7f592f583740 1 --2- 172.21.15.40:0/3868809029 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x55b74d840a30 0x55b74d840e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.850+0000 7f592f583740 4 client.4422 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.850+0000 7f592f583740 1 -- 172.21.15.40:0/3868809029 --> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] -- command(tid 0: {"prefix": "status"}) -- 0x55b74d771080 con 0x55b74d840a30 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.850+0000 7f5929c00640 1 --2- 172.21.15.40:0/3868809029 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x55b74d840a30 0x55b74d840e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f5929c00640 1 --2- 172.21.15.40:0/3868809029 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x55b74d840a30 0x55b74d840e10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18ef58f1afb3e72f server_cookie=6aa88cbf14c021dc in_seq=0 out_seq=0 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f590aa00640 10 client.4422 ms_handle_connect on v2:172.21.15.40:6834/3989190340 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f590aa00640 1 -- 172.21.15.40:0/3868809029 <== mds.0 v2:172.21.15.40:6834/3989190340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55b74d771080 con 0x55b74d840a30 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f590aa00640 10 client.4422 handle_command_reply: tid=0 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f592f583740 1 client.4422 shutdown 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f592f583740 1 -- 172.21.15.40:0/3868809029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8300a0 msgr2=0x55b74d847080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f592f583740 1 --2- 172.21.15.40:0/3868809029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8300a0 0x55b74d847080 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5910059150 tx=0x7f5910055580 comp rx=0 tx=0).stop 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f592f583740 1 -- 172.21.15.40:0/3868809029 shutdown_connections 2024-04-26T23:01:13.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f592f583740 1 --2- 172.21.15.40:0/3868809029 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x55b74d840a30 0x55b74d840e10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f592f583740 1 --2- 172.21.15.40:0/3868809029 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b74d841cb0 0x55b74d820ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f592f583740 1 --2- 172.21.15.40:0/3868809029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b74d8300a0 0x55b74d847080 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f592f583740 1 --2- 172.21.15.40:0/3868809029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b74d8055e0 0x55b74d846b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f590aa00640 0 client.4422 ms_handle_reset on v2:172.21.15.40:6834/3989190340 2024-04-26T23:01:13.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.851+0000 7f592f583740 1 -- 172.21.15.40:0/3868809029 >> 172.21.15.40:0/3868809029 conn(0x55b74d799d10 msgr2=0x55b74d7ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:13.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.852+0000 7f592f583740 1 -- 172.21.15.40:0/3868809029 shutdown_connections 2024-04-26T23:01:13.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.852+0000 7f592f583740 1 -- 172.21.15.40:0/3868809029 wait complete. 2024-04-26T23:01:13.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.852+0000 7f592f583740 20 client.4422 trim_cache size 0 max 16384 2024-04-26T23:01:13.855 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:01:13.855 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:01:13.855 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:01:13.855 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4257, 2024-04-26T23:01:13.856 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:01:13.856 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:01:13.856 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:01:13.856 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 8.1354750960000004, 2024-04-26T23:01:13.856 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 7, 2024-04-26T23:01:13.856 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 25, 2024-04-26T23:01:13.856 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 23, 2024-04-26T23:01:13.856 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 14.225155856000001 2024-04-26T23:01:13.856 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:01:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.852+0000 7f592c600640 1 -- 172.21.15.40:0/3842688946 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f590406c8f0 msgr2=0x7f590406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.852+0000 7f592c600640 1 --2- 172.21.15.40:0/3842688946 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f590406c8f0 0x7f590406edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f592408d6d0 tx=0x7f5918008040 comp rx=0 tx=0).stop 2024-04-26T23:01:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.852+0000 7f592c600640 1 -- 172.21.15.40:0/3842688946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5924151b10 msgr2=0x7f59240c9b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.852+0000 7f592c600640 1 --2- 172.21.15.40:0/3842688946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5924151b10 0x7f59240c9b10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5910007320 tx=0x7f59100077e0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.852+0000 7f592c600640 1 -- 172.21.15.40:0/3842688946 shutdown_connections 2024-04-26T23:01:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.852+0000 7f592c600640 1 --2- 172.21.15.40:0/3842688946 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f590406c8f0 0x7f590406edb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.852+0000 7f592c600640 1 --2- 172.21.15.40:0/3842688946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59240ce070 0x7f59240ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.852+0000 7f592c600640 1 --2- 172.21.15.40:0/3842688946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5924151b10 0x7f59240c9b10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.852+0000 7f592c600640 1 --2- 172.21.15.40:0/3842688946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f590c0051c0 0x7f59241f92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.852+0000 7f592c600640 1 -- 172.21.15.40:0/3842688946 >> 172.21.15.40:0/3842688946 conn(0x7f59240d4f90 msgr2=0x7f59240b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.853+0000 7f592c600640 1 -- 172.21.15.40:0/3842688946 shutdown_connections 2024-04-26T23:01:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:13.853+0000 7f592c600640 1 -- 172.21.15.40:0/3842688946 wait complete. 2024-04-26T23:01:13.870 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T23:01:13.870 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:01:13.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-26T23:01:14.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.072+0000 7fc687200640 1 Processor -- start 2024-04-26T23:01:14.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.072+0000 7fc687200640 1 -- start start 2024-04-26T23:01:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.074+0000 7fc687200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc6880d11a0 0x7fc68806e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.074+0000 7fc687200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc68806eaa0 0x7fc6880b5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.074+0000 7fc687200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc68815fca0 con 0x7fc688155f20 2024-04-26T23:01:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.074+0000 7fc687200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc6880d1610 con 0x7fc6880d11a0 2024-04-26T23:01:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.074+0000 7fc687200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc68815ff80 con 0x7fc68806eaa0 2024-04-26T23:01:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.074+0000 7fc685400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc6880d11a0 0x7fc68806e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.074+0000 7fc685400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc6880d11a0 0x7fc68806e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45480/0 (socket says 172.21.15.40:45480) 2024-04-26T23:01:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.074+0000 7fc685400640 1 -- 172.21.15.40:0/952699195 learned_addr learned my addr 172.21.15.40:0/952699195 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.074+0000 7fc686800640 1 --2- 172.21.15.40:0/952699195 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc68806eaa0 0x7fc6880b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.075+0000 7fc685400640 1 -- 172.21.15.40:0/952699195 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc68806eaa0 msgr2=0x7fc6880b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.075+0000 7fc685400640 1 --2- 172.21.15.40:0/952699195 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc68806eaa0 0x7fc6880b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.075+0000 7fc685400640 1 -- 172.21.15.40:0/952699195 >> v1:172.21.15.103:6789/0 conn(0x7fc688155f20 legacy=0x7fc68815f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.075+0000 7fc685400640 1 -- 172.21.15.40:0/952699195 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc688160260 con 0x7fc6880d11a0 2024-04-26T23:01:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.076+0000 7fc685400640 1 --2- 172.21.15.40:0/952699195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6880d11a0 0x7fc68806e560 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc674009870 tx=0x7fc67402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c03011802941c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.076+0000 7fc684a00640 1 -- 172.21.15.40:0/952699195 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674037070 con 0x7fc6880d11a0 2024-04-26T23:01:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.076+0000 7fc684a00640 1 -- 172.21.15.40:0/952699195 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc674032440 con 0x7fc6880d11a0 2024-04-26T23:01:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.076+0000 7fc684a00640 1 -- 172.21.15.40:0/952699195 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674032810 con 0x7fc6880d11a0 2024-04-26T23:01:14.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.077+0000 7fc687200640 1 -- 172.21.15.40:0/952699195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6880d11a0 msgr2=0x7fc68806e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:14.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.077+0000 7fc687200640 1 --2- 172.21.15.40:0/952699195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6880d11a0 0x7fc68806e560 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc674009870 tx=0x7fc67402a250 comp rx=0 tx=0).stop 2024-04-26T23:01:14.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.077+0000 7fc687200640 1 -- 172.21.15.40:0/952699195 shutdown_connections 2024-04-26T23:01:14.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.077+0000 7fc687200640 1 --2- 172.21.15.40:0/952699195 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc68806eaa0 0x7fc6880b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:14.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.077+0000 7fc687200640 1 --2- 172.21.15.40:0/952699195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6880d11a0 0x7fc68806e560 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:14.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.077+0000 7fc687200640 1 -- 172.21.15.40:0/952699195 >> 172.21.15.40:0/952699195 conn(0x7fc688075eb0 msgr2=0x7fc688068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:14.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.077+0000 7fc687200640 1 -- 172.21.15.40:0/952699195 shutdown_connections 2024-04-26T23:01:14.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.077+0000 7fc687200640 1 -- 172.21.15.40:0/952699195 wait complete. 2024-04-26T23:01:14.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.079+0000 7fc687200640 1 Processor -- start 2024-04-26T23:01:14.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.079+0000 7fc687200640 1 -- start start 2024-04-26T23:01:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.080+0000 7fc687200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc68806eaa0 0x7fc6881f7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.080+0000 7fc687200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6880d11a0 0x7fc6881584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.080+0000 7fc687200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc688155f20 0x7fc688158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.080+0000 7fc687200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc6880b5a40 con 0x7fc688155f20 2024-04-26T23:01:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.080+0000 7fc687200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc688156300 con 0x7fc68806eaa0 2024-04-26T23:01:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.080+0000 7fc687200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc6880b5bc0 con 0x7fc6880d11a0 2024-04-26T23:01:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.080+0000 7fc686800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc688155f20 0x7fc688158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.080+0000 7fc686800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc688155f20 0x7fc688158a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45492/0 (socket says 172.21.15.40:45492) 2024-04-26T23:01:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.081+0000 7fc686800640 1 -- 172.21.15.40:0/1883934000 learned_addr learned my addr 172.21.15.40:0/1883934000 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.081+0000 7fc685400640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6880d11a0 0x7fc6881584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.081+0000 7fc685e00640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc68806eaa0 0x7fc6881f7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.081+0000 7fc686800640 1 -- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6880d11a0 msgr2=0x7fc6881584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.081+0000 7fc686800640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6880d11a0 0x7fc6881584f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.081+0000 7fc686800640 1 -- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc68806eaa0 msgr2=0x7fc6881f7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.081+0000 7fc686800640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc68806eaa0 0x7fc6881f7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.081+0000 7fc686800640 1 -- 172.21.15.40:0/1883934000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc688150ad0 con 0x7fc688155f20 2024-04-26T23:01:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.081+0000 7fc686800640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc688155f20 0x7fc688158a30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc680007e40 tx=0x7fc68000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.082+0000 7fc67e000640 1 -- 172.21.15.40:0/1883934000 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc680014620 con 0x7fc688155f20 2024-04-26T23:01:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.082+0000 7fc67e000640 1 -- 172.21.15.40:0/1883934000 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc680019070 con 0x7fc688155f20 2024-04-26T23:01:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.082+0000 7fc687200640 1 -- 172.21.15.40:0/1883934000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc688150db0 con 0x7fc688155f20 2024-04-26T23:01:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.082+0000 7fc67e000640 1 -- 172.21.15.40:0/1883934000 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc680014a20 con 0x7fc688155f20 2024-04-26T23:01:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.082+0000 7fc687200640 1 -- 172.21.15.40:0/1883934000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc688151370 con 0x7fc688155f20 2024-04-26T23:01:14.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.083+0000 7fc687200640 1 -- 172.21.15.40:0/1883934000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc688069370 con 0x7fc688155f20 2024-04-26T23:01:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.084+0000 7fc67e000640 1 -- 172.21.15.40:0/1883934000 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc680014bc0 con 0x7fc688155f20 2024-04-26T23:01:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.085+0000 7fc67e000640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc66806c8f0 0x7fc66806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.085+0000 7fc67e000640 1 -- 172.21.15.40:0/1883934000 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc68008c6f0 con 0x7fc688155f20 2024-04-26T23:01:14.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.086+0000 7fc685e00640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc66806c8f0 0x7fc66806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:14.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.087+0000 7fc685e00640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc66806c8f0 0x7fc66806edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc678007fd0 tx=0x7fc678009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:14.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.089+0000 7fc67e000640 1 -- 172.21.15.40:0/1883934000 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc68005a870 con 0x7fc688155f20 2024-04-26T23:01:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.335+0000 7fc687200640 1 -- 172.21.15.40:0/1883934000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc68806f3e0 con 0x7fc688155f20 2024-04-26T23:01:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.336+0000 7fc67e000640 1 -- 172.21.15.40:0/1883934000 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7fc68005a170 con 0x7fc688155f20 2024-04-26T23:01:14.336 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T23:01:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.338+0000 7fc687200640 1 -- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc66806c8f0 msgr2=0x7fc66806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.338+0000 7fc687200640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc66806c8f0 0x7fc66806edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc678007fd0 tx=0x7fc678009040 comp rx=0 tx=0).stop 2024-04-26T23:01:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.338+0000 7fc687200640 1 -- 172.21.15.40:0/1883934000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc688155f20 msgr2=0x7fc688158a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.338+0000 7fc687200640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc688155f20 0x7fc688158a30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc680007e40 tx=0x7fc68000e4b0 comp rx=0 tx=0).stop 2024-04-26T23:01:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.339+0000 7fc687200640 1 -- 172.21.15.40:0/1883934000 shutdown_connections 2024-04-26T23:01:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.339+0000 7fc687200640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc66806c8f0 0x7fc66806edb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.339+0000 7fc687200640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc688155f20 0x7fc688158a30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.339+0000 7fc687200640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc6880d11a0 0x7fc6881584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.339+0000 7fc687200640 1 --2- 172.21.15.40:0/1883934000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc68806eaa0 0x7fc6881f7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:14.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.339+0000 7fc687200640 1 -- 172.21.15.40:0/1883934000 >> 172.21.15.40:0/1883934000 conn(0x7fc688075eb0 msgr2=0x7fc68815f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:14.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.339+0000 7fc687200640 1 -- 172.21.15.40:0/1883934000 shutdown_connections 2024-04-26T23:01:14.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14.339+0000 7fc687200640 1 -- 172.21.15.40:0/1883934000 wait complete. 2024-04-26T23:01:14.358 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-26T23:01:14.358 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:01:14.358 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-04-26T23:01:14.359 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-26T23:01:14.359 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:01:14.359 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-26T23:01:14.359 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T23:01:14.359 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:01:14.359 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-26T23:01:14.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:14.359 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-04-26T23:01:14.412 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:01:14.412 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:01:14.412 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42640sec preferred_lft 42640sec 2024-04-26T23:01:14.413 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-26T23:01:14.414 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:14.414 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:01:14.414 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:14.414 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-04-26T23:01:14.414 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T23:01:14.415 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:14.415 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:14.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add name ceph-brx type bridge 2024-04-26T23:01:14.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr flush dev ceph-brx 2024-04-26T23:01:14.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set ceph-brx up 2024-04-26T23:01:14.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T23:01:14.415 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:14.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:14.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:14.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:14.916 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:01:14.990 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-04-26T23:01:14.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:14.993 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-04-26T23:01:15.051 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-04-26T23:01:15.051 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:01:15.051 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T23:01:15.051 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T23:01:15.051 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:01:15.052 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:15.052 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:15.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T23:01:15.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T23:01:15.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T23:01:15.052 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:15.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:15.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:15.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:15.895 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:01:15.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:15.940 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list-id 2024-04-26T23:01:16.030 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:16.030 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:16.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:01:16.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-26T23:01:16.031 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:16.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:16.207 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-26T23:01:16.207 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:16.208 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:16.208 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-26T23:01:16.208 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T23:01:16.208 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-26T23:01:16.208 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-26T23:01:16.208 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-26T23:01:16.208 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:16.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:16.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:16.500 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:16.500 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:16.500 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-04-26T23:01:16.500 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T23:01:16.500 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:16.700 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-26T23:01:16.700 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:01:16.700 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:01:16.739 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-26T23:01:16.739 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:01:16.739 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:01:16.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-04-26T23:01:16.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:01:16.913 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:01:16.933 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T23:01:17.028 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-04-26T23:01:17.028 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-04-26T23:01:17.028 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-04-26T23:01:17.028 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-04-26T23:01:17.028 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-04-26T23:01:17.028 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-04-26T23:01:17.028 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-04-26T23:01:17.028 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-04-26T23:01:17.028 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-04-26T23:01:17.028 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-04-26T23:01:17.028 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/selinux 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-04-26T23:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-04-26T23:01:17.030 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-04-26T23:01:17.030 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-04-26T23:01:17.030 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-04-26T23:01:17.030 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-04-26T23:01:17.030 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:01:17.030 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:01:17.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:17.031 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-04-26T23:01:17.128 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-26T23:01:17.128 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-26T23:01:17.134 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-04-26T23:01:17.206 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T23:01:17.253 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-04-26T23:01:17.253 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-04-26T23:01:17.253 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-04-26T23:01:17.253 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-04-26T23:01:17.253 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-04-26T23:01:17.253 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-04-26T23:01:17.253 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-04-26T23:01:17.253 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-04-26T23:01:17.253 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-04-26T23:01:17.253 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-04-26T23:01:17.253 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-04-26T23:01:17.253 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/selinux 2024-04-26T23:01:17.253 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-04-26T23:01:17.254 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-04-26T23:01:17.255 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-04-26T23:01:17.255 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-04-26T23:01:17.255 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-04-26T23:01:17.255 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:01:17.255 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:01:17.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:17.256 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-04-26T23:01:17.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:2024-04-26T23:01:17.324+0000 7f131551c480 -1 init, newargv = 0x55fa677e74b0 newargc=15 2024-04-26T23:01:17.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:ceph-fuse[39680]: starting ceph client 2024-04-26T23:01:17.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:ceph-fuse[39680]: starting fuse 2024-04-26T23:01:18.396 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-04-26T23:01:18.490 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T23:01:18.547 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/selinux 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-04-26T23:01:18.548 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-04-26T23:01:18.549 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-04-26T23:01:18.550 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:01:18.550 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:01:18.550 INFO:teuthology.orchestra.run.smithi040.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-26T23:01:18.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:18.551 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-04-26T23:01:18.604 INFO:teuthology.orchestra.run.smithi040.stdout:42 2024-04-26T23:01:18.604 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-04-26T23:01:18.605 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-04-26T23:01:18.605 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-04-26T23:01:18.605 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-04-26T23:01:18.605 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:01:18.605 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-04-26T23:01:18.605 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:01:18.605 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-04-26T23:01:18.605 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-26T23:01:18.605 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-04-26T23:01:18.605 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-26T23:01:18.605 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:01:18.605 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-04-26T23:01:18.605 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.admin")) 2024-04-26T23:01:18.606 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:01:18.781 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.admin.39680.asok 2024-04-26T23:01:18.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:18.799 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.39680.asok 2024-04-26T23:01:18.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:18.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.39680.asok status 2024-04-26T23:01:19.040 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:01:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "bc612ffb9c1d5bc27536bc917a705262a36d4387", 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "admin", 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "39680", 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4434, 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-04-26T23:01:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4434 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 217721616 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 217721616 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.4434 192.168.0.1:0/217721616", 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "192.168.0.1:0/217721616", 2024-04-26T23:01:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-04-26T23:01:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 7, 2024-04-26T23:01:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 25, 2024-04-26T23:01:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-04-26T23:01:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-04-26T23:01:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-04-26T23:01:19.043 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:01:19.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:19.056 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-04-26T23:01:19.056 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:01:19.109 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-04-26T23:01:19.109 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:01:19.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:19.110 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:01:19.198 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-26T23:01:19.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:01:19.348 INFO:teuthology.orchestra.run.smithi040.stdout:# file: . 2024-04-26T23:01:19.348 INFO:teuthology.orchestra.run.smithi040.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-26T23:01:19.348 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:19.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:19.358 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:01:19.359 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:01:19.408 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:19.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.667+0000 7fd611200640 1 Processor -- start 2024-04-26T23:01:19.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.668+0000 7fd611200640 1 -- start start 2024-04-26T23:01:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.669+0000 7fd611200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd60c0d3740 0x7fd60c0b9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.669+0000 7fd611200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd60c162c40 con 0x7fd60c0b6220 2024-04-26T23:01:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.669+0000 7fd611200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd60c0d2d80 con 0x7fd60c0d32e0 2024-04-26T23:01:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.669+0000 7fd611200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fd60c0b7c80 con 0x7fd60c0d3740 2024-04-26T23:01:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.670+0000 7fd60b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd60c0b6220 0x7fd60c0d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53940/0 (socket says 172.21.15.40:53940) 2024-04-26T23:01:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.670+0000 7fd60b400640 1 -- 172.21.15.40:0/218854636 learned_addr learned my addr 172.21.15.40:0/218854636 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.670+0000 7fd60be00640 1 --2- 172.21.15.40:0/218854636 >> v2:172.21.15.40:3300/0 conn(0x7fd60c0d3740 0x7fd60c0b9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:19.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.670+0000 7fd60a000640 1 -- 172.21.15.40:0/218854636 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1108832048 0 0) 0x7fd60c162c40 con 0x7fd60c0b6220 2024-04-26T23:01:19.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.670+0000 7fd60a000640 1 -- 172.21.15.40:0/218854636 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5f0003520 con 0x7fd60c0b6220 2024-04-26T23:01:19.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.670+0000 7fd60be00640 1 -- 172.21.15.40:0/218854636 >> v1:172.21.15.103:6789/0 conn(0x7fd60c0d32e0 legacy=0x7fd60c0b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:19.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.670+0000 7fd60be00640 1 -- 172.21.15.40:0/218854636 >> v1:172.21.15.40:6789/0 conn(0x7fd60c0b6220 legacy=0x7fd60c0d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:19.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.670+0000 7fd60be00640 1 -- 172.21.15.40:0/218854636 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60c0b7f60 con 0x7fd60c0d3740 2024-04-26T23:01:19.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.671+0000 7fd60be00640 1 --2- 172.21.15.40:0/218854636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60c0d3740 0x7fd60c0b9430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd6040097a0 tx=0x7fd60402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=676eb6a90d7afa31 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:19.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.672+0000 7fd60a000640 1 -- 172.21.15.40:0/218854636 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd604037070 con 0x7fd60c0d3740 2024-04-26T23:01:19.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.672+0000 7fd60a000640 1 -- 172.21.15.40:0/218854636 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd604032440 con 0x7fd60c0d3740 2024-04-26T23:01:19.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.672+0000 7fd60a000640 1 -- 172.21.15.40:0/218854636 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6040327f0 con 0x7fd60c0d3740 2024-04-26T23:01:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.672+0000 7fd611200640 1 -- 172.21.15.40:0/218854636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60c0d3740 msgr2=0x7fd60c0b9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.672+0000 7fd611200640 1 --2- 172.21.15.40:0/218854636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60c0d3740 0x7fd60c0b9430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd6040097a0 tx=0x7fd60402a1e0 comp rx=0 tx=0).stop 2024-04-26T23:01:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.672+0000 7fd611200640 1 -- 172.21.15.40:0/218854636 shutdown_connections 2024-04-26T23:01:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.672+0000 7fd611200640 1 --2- 172.21.15.40:0/218854636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60c0d3740 0x7fd60c0b9430 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.672+0000 7fd611200640 1 -- 172.21.15.40:0/218854636 >> 172.21.15.40:0/218854636 conn(0x7fd60c0d4d90 msgr2=0x7fd60c0d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.673+0000 7fd611200640 1 -- 172.21.15.40:0/218854636 shutdown_connections 2024-04-26T23:01:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.673+0000 7fd611200640 1 -- 172.21.15.40:0/218854636 wait complete. 2024-04-26T23:01:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.674+0000 7fd611200640 1 Processor -- start 2024-04-26T23:01:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.674+0000 7fd611200640 1 -- start start 2024-04-26T23:01:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.675+0000 7fd611200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60c0b6220 0x7fd60c1f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.675+0000 7fd611200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd60c0d32e0 0x7fd60c0cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd611200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd60c0d3740 0x7fd60c15d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd611200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c0d3b80 con 0x7fd60c0b6220 2024-04-26T23:01:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd611200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c0b8520 con 0x7fd60c0d32e0 2024-04-26T23:01:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd611200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd60c0d0aa0 con 0x7fd60c0d3740 2024-04-26T23:01:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd60b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60c0b6220 0x7fd60c1f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd60b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60c0b6220 0x7fd60c1f49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35066/0 (socket says 172.21.15.40:35066) 2024-04-26T23:01:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd60b400640 1 -- 172.21.15.40:0/2657395222 learned_addr learned my addr 172.21.15.40:0/2657395222 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd60aa00640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd60c0d32e0 0x7fd60c0cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd60be00640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd60c0d3740 0x7fd60c15d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:19.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd60b400640 1 -- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd60c0d3740 msgr2=0x7fd60c15d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:19.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd60b400640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd60c0d3740 0x7fd60c15d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:19.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd60b400640 1 -- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd60c0d32e0 msgr2=0x7fd60c0cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:19.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd60b400640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd60c0d32e0 0x7fd60c0cbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:19.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd60b400640 1 -- 172.21.15.40:0/2657395222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60c15ad20 con 0x7fd60c0b6220 2024-04-26T23:01:19.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.676+0000 7fd60be00640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd60c0d3740 0x7fd60c15d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:19.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.677+0000 7fd60b400640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60c0b6220 0x7fd60c1f49b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00cb40 tx=0x7fd5fc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:19.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.677+0000 7fd603400640 1 -- 172.21.15.40:0/2657395222 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc017070 con 0x7fd60c0b6220 2024-04-26T23:01:19.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.677+0000 7fd603400640 1 -- 172.21.15.40:0/2657395222 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5fc00aca0 con 0x7fd60c0b6220 2024-04-26T23:01:19.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.677+0000 7fd603400640 1 -- 172.21.15.40:0/2657395222 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc012620 con 0x7fd60c0b6220 2024-04-26T23:01:19.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.677+0000 7fd611200640 1 -- 172.21.15.40:0/2657395222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd60c0cc460 con 0x7fd60c0b6220 2024-04-26T23:01:19.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.677+0000 7fd611200640 1 -- 172.21.15.40:0/2657395222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd60c0cca20 con 0x7fd60c0b6220 2024-04-26T23:01:19.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.679+0000 7fd611200640 1 -- 172.21.15.40:0/2657395222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd60c069370 con 0x7fd60c0b6220 2024-04-26T23:01:19.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.680+0000 7fd603400640 1 -- 172.21.15.40:0/2657395222 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd5fc01a070 con 0x7fd60c0b6220 2024-04-26T23:01:19.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.681+0000 7fd603400640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd5e806c8f0 0x7fd5e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:19.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.681+0000 7fd603400640 1 -- 172.21.15.40:0/2657395222 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd5fc08bdc0 con 0x7fd60c0b6220 2024-04-26T23:01:19.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.681+0000 7fd60aa00640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd5e806c8f0 0x7fd5e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:19.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.682+0000 7fd60aa00640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd5e806c8f0 0x7fd5e806edb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0b5bc0 tx=0x7fd5f4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:19.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.686+0000 7fd603400640 1 -- 172.21.15.40:0/2657395222 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd5fc059e10 con 0x7fd60c0b6220 2024-04-26T23:01:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.892+0000 7fd611200640 1 -- 172.21.15.40:0/2657395222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd60c0d30c0 con 0x7fd60c0b6220 2024-04-26T23:01:19.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.895+0000 7fd603400640 1 -- 172.21.15.40:0/2657395222 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7fd5fc059710 con 0x7fd60c0b6220 2024-04-26T23:01:19.896 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:19.896 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:11.763965+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:19.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.897+0000 7fd611200640 1 -- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd5e806c8f0 msgr2=0x7fd5e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:19.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.897+0000 7fd611200640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd5e806c8f0 0x7fd5e806edb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0b5bc0 tx=0x7fd5f4009040 comp rx=0 tx=0).stop 2024-04-26T23:01:19.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.897+0000 7fd611200640 1 -- 172.21.15.40:0/2657395222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60c0b6220 msgr2=0x7fd60c1f49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:19.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.897+0000 7fd611200640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60c0b6220 0x7fd60c1f49b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00cb40 tx=0x7fd5fc00a060 comp rx=0 tx=0).stop 2024-04-26T23:01:19.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.898+0000 7fd611200640 1 -- 172.21.15.40:0/2657395222 shutdown_connections 2024-04-26T23:01:19.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.898+0000 7fd611200640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd5e806c8f0 0x7fd5e806edb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:19.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.898+0000 7fd611200640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd60c0d3740 0x7fd60c15d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:19.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.898+0000 7fd611200640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd60c0d32e0 0x7fd60c0cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:19.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.898+0000 7fd611200640 1 --2- 172.21.15.40:0/2657395222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60c0b6220 0x7fd60c1f49b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:19.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.898+0000 7fd611200640 1 -- 172.21.15.40:0/2657395222 >> 172.21.15.40:0/2657395222 conn(0x7fd60c0d4d90 msgr2=0x7fd60c0d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:19.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.898+0000 7fd611200640 1 -- 172.21.15.40:0/2657395222 shutdown_connections 2024-04-26T23:01:19.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:19.898+0000 7fd611200640 1 -- 172.21.15.40:0/2657395222 wait complete. 2024-04-26T23:01:19.919 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:01:19.920 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:01:20.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.133+0000 7f2a1d600640 1 Processor -- start 2024-04-26T23:01:20.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.134+0000 7f2a1d600640 1 -- start start 2024-04-26T23:01:20.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.136+0000 7f2a1d600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a180cfaf0 0x7f2a18073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:20.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.136+0000 7f2a1d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a1806dd40 0x7f2a18072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:20.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.136+0000 7f2a1d600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a18161880 con 0x7f2a18160a50 2024-04-26T23:01:20.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.136+0000 7f2a1d600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f2a18071ce0 con 0x7f2a180cfaf0 2024-04-26T23:01:20.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.136+0000 7f2a1d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2a180722a0 con 0x7f2a1806dd40 2024-04-26T23:01:20.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.136+0000 7f2a17e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2a18160a50 0x7f2a18160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53954/0 (socket says 172.21.15.40:53954) 2024-04-26T23:01:20.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.136+0000 7f2a17e00640 1 -- 172.21.15.40:0/3797947098 learned_addr learned my addr 172.21.15.40:0/3797947098 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:20.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.136+0000 7f2a0fe00640 1 --2- 172.21.15.40:0/3797947098 >> v2:172.21.15.40:3300/0 conn(0x7f2a180cfaf0 0x7f2a18073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:20.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.137+0000 7f2a1cc00640 1 --2- 172.21.15.40:0/3797947098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a1806dd40 0x7f2a18072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:20.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.137+0000 7f2a0fe00640 1 -- 172.21.15.40:0/3797947098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a1806dd40 msgr2=0x7f2a18072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:20.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.137+0000 7f2a17400640 1 -- 172.21.15.40:0/3797947098 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4281524664 0 0) 0x7f2a18161880 con 0x7f2a18160a50 2024-04-26T23:01:20.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.137+0000 7f2a0fe00640 1 --2- 172.21.15.40:0/3797947098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a1806dd40 0x7f2a18072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.137+0000 7f2a0fe00640 1 -- 172.21.15.40:0/3797947098 >> v1:172.21.15.40:6789/0 conn(0x7f2a18160a50 legacy=0x7f2a18160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:20.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.137+0000 7f2a0fe00640 1 -- 172.21.15.40:0/3797947098 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a180cff80 con 0x7f2a180cfaf0 2024-04-26T23:01:20.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.137+0000 7f2a1cc00640 1 --2- 172.21.15.40:0/3797947098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a1806dd40 0x7f2a18072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.138+0000 7f2a0fe00640 1 --2- 172.21.15.40:0/3797947098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a180cfaf0 0x7f2a18073a50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2a000097a0 tx=0x7f2a0002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6d13d104abd8eed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.138+0000 7f2a17400640 1 -- 172.21.15.40:0/3797947098 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a00037070 con 0x7f2a180cfaf0 2024-04-26T23:01:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.138+0000 7f2a17400640 1 -- 172.21.15.40:0/3797947098 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a00032440 con 0x7f2a180cfaf0 2024-04-26T23:01:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.138+0000 7f2a17400640 1 -- 172.21.15.40:0/3797947098 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a00032810 con 0x7f2a180cfaf0 2024-04-26T23:01:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.139+0000 7f2a1d600640 1 -- 172.21.15.40:0/3797947098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a180cfaf0 msgr2=0x7f2a18073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.139+0000 7f2a1d600640 1 --2- 172.21.15.40:0/3797947098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a180cfaf0 0x7f2a18073a50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2a000097a0 tx=0x7f2a0002a180 comp rx=0 tx=0).stop 2024-04-26T23:01:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.139+0000 7f2a1d600640 1 -- 172.21.15.40:0/3797947098 shutdown_connections 2024-04-26T23:01:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.139+0000 7f2a1d600640 1 --2- 172.21.15.40:0/3797947098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a1806dd40 0x7f2a18072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.139+0000 7f2a1d600640 1 --2- 172.21.15.40:0/3797947098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a180cfaf0 0x7f2a18073a50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.139+0000 7f2a1d600640 1 -- 172.21.15.40:0/3797947098 >> 172.21.15.40:0/3797947098 conn(0x7f2a180b2720 msgr2=0x7f2a180b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.139+0000 7f2a1d600640 1 -- 172.21.15.40:0/3797947098 shutdown_connections 2024-04-26T23:01:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.139+0000 7f2a1d600640 1 -- 172.21.15.40:0/3797947098 wait complete. 2024-04-26T23:01:20.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.141+0000 7f2a1d600640 1 Processor -- start 2024-04-26T23:01:20.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.141+0000 7f2a1d600640 1 -- start start 2024-04-26T23:01:20.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.142+0000 7f2a1d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a1806dd40 0x7f2a18143720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:20.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.142+0000 7f2a1d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a18160a50 0x7f2a1815a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:20.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.142+0000 7f2a1d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a18144430 0x7f2a181448c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:20.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.142+0000 7f2a1d600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2a18071f30 con 0x7f2a1806dd40 2024-04-26T23:01:20.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.142+0000 7f2a1d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2a18073f90 con 0x7f2a18160a50 2024-04-26T23:01:20.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.142+0000 7f2a1d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2a1806e1f0 con 0x7f2a18144430 2024-04-26T23:01:20.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.143+0000 7f2a17e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a1806dd40 0x7f2a18143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:20.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.143+0000 7f2a17e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a1806dd40 0x7f2a18143720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35090/0 (socket says 172.21.15.40:35090) 2024-04-26T23:01:20.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.143+0000 7f2a17e00640 1 -- 172.21.15.40:0/3428924871 learned_addr learned my addr 172.21.15.40:0/3428924871 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:20.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.143+0000 7f2a1cc00640 1 --2- 172.21.15.40:0/3428924871 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a18144430 0x7f2a181448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:20.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.143+0000 7f2a17e00640 1 -- 172.21.15.40:0/3428924871 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a18144430 msgr2=0x7f2a181448c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:20.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.143+0000 7f2a17e00640 1 --2- 172.21.15.40:0/3428924871 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a18144430 0x7f2a181448c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.143+0000 7f2a17e00640 1 -- 172.21.15.40:0/3428924871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a18160a50 msgr2=0x7f2a1815a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:20.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.143+0000 7f2a17e00640 1 --2- 172.21.15.40:0/3428924871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a18160a50 0x7f2a1815a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.143+0000 7f2a17e00640 1 -- 172.21.15.40:0/3428924871 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a18142680 con 0x7f2a1806dd40 2024-04-26T23:01:20.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.143+0000 7f2a17e00640 1 --2- 172.21.15.40:0/3428924871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a1806dd40 0x7f2a18143720 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800c430 tx=0x7f2a0800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:20.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.144+0000 7f2a14c00640 1 -- 172.21.15.40:0/3428924871 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a080074e0 con 0x7f2a1806dd40 2024-04-26T23:01:20.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.144+0000 7f2a14c00640 1 -- 172.21.15.40:0/3428924871 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a080047e0 con 0x7f2a1806dd40 2024-04-26T23:01:20.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.144+0000 7f2a1d600640 1 -- 172.21.15.40:0/3428924871 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a18142960 con 0x7f2a1806dd40 2024-04-26T23:01:20.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.144+0000 7f2a14c00640 1 -- 172.21.15.40:0/3428924871 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a08004bd0 con 0x7f2a1806dd40 2024-04-26T23:01:20.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.144+0000 7f2a1d600640 1 -- 172.21.15.40:0/3428924871 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a18142f20 con 0x7f2a1806dd40 2024-04-26T23:01:20.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.145+0000 7f2a1d600640 1 -- 172.21.15.40:0/3428924871 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a18069370 con 0x7f2a1806dd40 2024-04-26T23:01:20.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.146+0000 7f2a14c00640 1 -- 172.21.15.40:0/3428924871 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a08011070 con 0x7f2a1806dd40 2024-04-26T23:01:20.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.147+0000 7f2a14c00640 1 --2- 172.21.15.40:0/3428924871 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f29f406c8f0 0x7f29f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:20.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.147+0000 7f2a14c00640 1 -- 172.21.15.40:0/3428924871 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2a0808bb80 con 0x7f2a1806dd40 2024-04-26T23:01:20.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.148+0000 7f2a0fe00640 1 --2- 172.21.15.40:0/3428924871 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f29f406c8f0 0x7f29f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:20.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.149+0000 7f2a0fe00640 1 --2- 172.21.15.40:0/3428924871 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f29f406c8f0 0x7f29f406edb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2a00034000 tx=0x7f2a0002c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:20.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.152+0000 7f2a14c00640 1 -- 172.21.15.40:0/3428924871 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2a08059c60 con 0x7f2a1806dd40 2024-04-26T23:01:20.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.353+0000 7f2a1d600640 1 -- 172.21.15.40:0/3428924871 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2a1806fa60 con 0x7f2a1806dd40 2024-04-26T23:01:20.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.354+0000 7f2a14c00640 1 -- 172.21.15.40:0/3428924871 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f2a08059560 con 0x7f2a1806dd40 2024-04-26T23:01:20.354 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:01:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.357+0000 7f2a1d600640 1 -- 172.21.15.40:0/3428924871 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f29f406c8f0 msgr2=0x7f29f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.357+0000 7f2a1d600640 1 --2- 172.21.15.40:0/3428924871 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f29f406c8f0 0x7f29f406edb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2a00034000 tx=0x7f2a0002c000 comp rx=0 tx=0).stop 2024-04-26T23:01:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.357+0000 7f2a1d600640 1 -- 172.21.15.40:0/3428924871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a1806dd40 msgr2=0x7f2a18143720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.357+0000 7f2a1d600640 1 --2- 172.21.15.40:0/3428924871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a1806dd40 0x7f2a18143720 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800c430 tx=0x7f2a0800c8f0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.357+0000 7f2a1d600640 1 -- 172.21.15.40:0/3428924871 shutdown_connections 2024-04-26T23:01:20.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.357+0000 7f2a1d600640 1 --2- 172.21.15.40:0/3428924871 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f29f406c8f0 0x7f29f406edb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.357+0000 7f2a1d600640 1 --2- 172.21.15.40:0/3428924871 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a18144430 0x7f2a181448c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.357+0000 7f2a1d600640 1 --2- 172.21.15.40:0/3428924871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a18160a50 0x7f2a1815a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.357+0000 7f2a1d600640 1 --2- 172.21.15.40:0/3428924871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a1806dd40 0x7f2a18143720 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.357+0000 7f2a1d600640 1 -- 172.21.15.40:0/3428924871 >> 172.21.15.40:0/3428924871 conn(0x7f2a180b2720 msgr2=0x7f2a180d0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:20.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.357+0000 7f2a1d600640 1 -- 172.21.15.40:0/3428924871 shutdown_connections 2024-04-26T23:01:20.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.357+0000 7f2a1d600640 1 -- 172.21.15.40:0/3428924871 wait complete. 2024-04-26T23:01:20.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:01:20.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.593+0000 7fa468e00640 1 Processor -- start 2024-04-26T23:01:20.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.593+0000 7fa468e00640 1 -- start start 2024-04-26T23:01:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.595+0000 7fa468e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4640cfaf0 0x7fa464073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.595+0000 7fa468e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa46406dd40 0x7fa464072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.595+0000 7fa468e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa464161880 con 0x7fa464160a50 2024-04-26T23:01:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.595+0000 7fa468e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa464071ce0 con 0x7fa46406dd40 2024-04-26T23:01:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.595+0000 7fa468e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa4640722a0 con 0x7fa4640cfaf0 2024-04-26T23:01:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.595+0000 7fa463e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa46406dd40 0x7fa464072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.595+0000 7fa463e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa46406dd40 0x7fa464072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35092/0 (socket says 172.21.15.40:35092) 2024-04-26T23:01:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.595+0000 7fa463e00640 1 -- 172.21.15.40:0/3866268321 learned_addr learned my addr 172.21.15.40:0/3866268321 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.596+0000 7fa462a00640 1 --2- 172.21.15.40:0/3866268321 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4640cfaf0 0x7fa464073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.596+0000 7fa463e00640 1 -- 172.21.15.40:0/3866268321 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4640cfaf0 msgr2=0x7fa464073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.596+0000 7fa463e00640 1 --2- 172.21.15.40:0/3866268321 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4640cfaf0 0x7fa464073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.596+0000 7fa463e00640 1 -- 172.21.15.40:0/3866268321 >> v1:172.21.15.103:6789/0 conn(0x7fa464160a50 legacy=0x7fa464160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.596+0000 7fa463e00640 1 -- 172.21.15.40:0/3866268321 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4640cff80 con 0x7fa46406dd40 2024-04-26T23:01:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.597+0000 7fa463e00640 1 --2- 172.21.15.40:0/3866268321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa46406dd40 0x7fa464072730 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009870 tx=0x7fa45c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471fe508616c9bdd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.597+0000 7fa462000640 1 -- 172.21.15.40:0/3866268321 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c037070 con 0x7fa46406dd40 2024-04-26T23:01:20.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.597+0000 7fa462000640 1 -- 172.21.15.40:0/3866268321 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa45c032440 con 0x7fa46406dd40 2024-04-26T23:01:20.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.597+0000 7fa462000640 1 -- 172.21.15.40:0/3866268321 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c032810 con 0x7fa46406dd40 2024-04-26T23:01:20.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.598+0000 7fa468e00640 1 -- 172.21.15.40:0/3866268321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa46406dd40 msgr2=0x7fa464072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:20.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.598+0000 7fa468e00640 1 --2- 172.21.15.40:0/3866268321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa46406dd40 0x7fa464072730 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009870 tx=0x7fa45c02a250 comp rx=0 tx=0).stop 2024-04-26T23:01:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.598+0000 7fa468e00640 1 -- 172.21.15.40:0/3866268321 shutdown_connections 2024-04-26T23:01:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.598+0000 7fa468e00640 1 --2- 172.21.15.40:0/3866268321 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4640cfaf0 0x7fa464073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.598+0000 7fa468e00640 1 --2- 172.21.15.40:0/3866268321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa46406dd40 0x7fa464072730 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.598+0000 7fa468e00640 1 -- 172.21.15.40:0/3866268321 >> 172.21.15.40:0/3866268321 conn(0x7fa4640b2720 msgr2=0x7fa4640b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.598+0000 7fa468e00640 1 -- 172.21.15.40:0/3866268321 shutdown_connections 2024-04-26T23:01:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.598+0000 7fa468e00640 1 -- 172.21.15.40:0/3866268321 wait complete. 2024-04-26T23:01:20.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.600+0000 7fa468e00640 1 Processor -- start 2024-04-26T23:01:20.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.600+0000 7fa468e00640 1 -- start start 2024-04-26T23:01:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.601+0000 7fa468e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa46406dd40 0x7fa46415cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.601+0000 7fa468e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa464160a50 0x7fa46413c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.601+0000 7fa468e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4640b20a0 0x7fa46415f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.601+0000 7fa468e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa464071f30 con 0x7fa464160a50 2024-04-26T23:01:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.601+0000 7fa468e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa464073f90 con 0x7fa46406dd40 2024-04-26T23:01:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.601+0000 7fa468e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa46406e1f0 con 0x7fa4640b20a0 2024-04-26T23:01:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.601+0000 7fa462a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa464160a50 0x7fa46413c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.601+0000 7fa462a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa464160a50 0x7fa46413c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35100/0 (socket says 172.21.15.40:35100) 2024-04-26T23:01:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.601+0000 7fa462a00640 1 -- 172.21.15.40:0/2038133951 learned_addr learned my addr 172.21.15.40:0/2038133951 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.602+0000 7fa463400640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa46406dd40 0x7fa46415cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.602+0000 7fa463e00640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4640b20a0 0x7fa46415f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.602+0000 7fa462a00640 1 -- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4640b20a0 msgr2=0x7fa46415f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.602+0000 7fa462a00640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4640b20a0 0x7fa46415f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.602+0000 7fa462a00640 1 -- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa46406dd40 msgr2=0x7fa46415cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.602+0000 7fa462a00640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa46406dd40 0x7fa46415cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.602+0000 7fa462a00640 1 -- 172.21.15.40:0/2038133951 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46413da70 con 0x7fa464160a50 2024-04-26T23:01:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.602+0000 7fa462a00640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa464160a50 0x7fa46413c930 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa450007ef0 tx=0x7fa45000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.602+0000 7fa45b400640 1 -- 172.21.15.40:0/2038133951 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa450014620 con 0x7fa464160a50 2024-04-26T23:01:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.602+0000 7fa45b400640 1 -- 172.21.15.40:0/2038133951 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa450019070 con 0x7fa464160a50 2024-04-26T23:01:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.602+0000 7fa45b400640 1 -- 172.21.15.40:0/2038133951 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa450014a00 con 0x7fa464160a50 2024-04-26T23:01:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.602+0000 7fa468e00640 1 -- 172.21.15.40:0/2038133951 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46413dd50 con 0x7fa464160a50 2024-04-26T23:01:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.603+0000 7fa468e00640 1 -- 172.21.15.40:0/2038133951 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa46415aa70 con 0x7fa464160a50 2024-04-26T23:01:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.604+0000 7fa468e00640 1 -- 172.21.15.40:0/2038133951 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa464069370 con 0x7fa464160a50 2024-04-26T23:01:20.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.605+0000 7fa45b400640 1 -- 172.21.15.40:0/2038133951 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa450014ba0 con 0x7fa464160a50 2024-04-26T23:01:20.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.606+0000 7fa45b400640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa44406c8f0 0x7fa44406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:20.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.606+0000 7fa45b400640 1 -- 172.21.15.40:0/2038133951 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa45008c820 con 0x7fa464160a50 2024-04-26T23:01:20.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.606+0000 7fa463400640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa44406c8f0 0x7fa44406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:20.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.607+0000 7fa463400640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa44406c8f0 0x7fa44406edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa46413d530 tx=0x7fa454009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:20.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.610+0000 7fa45b400640 1 -- 172.21.15.40:0/2038133951 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa45005a870 con 0x7fa464160a50 2024-04-26T23:01:20.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.821+0000 7fa468e00640 1 -- 172.21.15.40:0/2038133951 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa4640cf550 con 0x7fa464160a50 2024-04-26T23:01:20.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.823+0000 7fa45b400640 1 -- 172.21.15.40:0/2038133951 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fa45005a170 con 0x7fa464160a50 2024-04-26T23:01:20.823 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:01:20.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.826+0000 7fa468e00640 1 -- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa44406c8f0 msgr2=0x7fa44406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:20.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.826+0000 7fa468e00640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa44406c8f0 0x7fa44406edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa46413d530 tx=0x7fa454009040 comp rx=0 tx=0).stop 2024-04-26T23:01:20.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.826+0000 7fa468e00640 1 -- 172.21.15.40:0/2038133951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa464160a50 msgr2=0x7fa46413c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:20.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.826+0000 7fa468e00640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa464160a50 0x7fa46413c930 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa450007ef0 tx=0x7fa45000e4b0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.826+0000 7fa468e00640 1 -- 172.21.15.40:0/2038133951 shutdown_connections 2024-04-26T23:01:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.826+0000 7fa468e00640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa44406c8f0 0x7fa44406edb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.826+0000 7fa468e00640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4640b20a0 0x7fa46415f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.826+0000 7fa468e00640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa464160a50 0x7fa46413c930 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.826+0000 7fa468e00640 1 --2- 172.21.15.40:0/2038133951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa46406dd40 0x7fa46415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.826+0000 7fa468e00640 1 -- 172.21.15.40:0/2038133951 >> 172.21.15.40:0/2038133951 conn(0x7fa4640b2720 msgr2=0x7fa4640ae950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.826+0000 7fa468e00640 1 -- 172.21.15.40:0/2038133951 shutdown_connections 2024-04-26T23:01:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:20.826+0000 7fa468e00640 1 -- 172.21.15.40:0/2038133951 wait complete. 2024-04-26T23:01:20.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:01:21.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.058+0000 7efc30e00640 1 Processor -- start 2024-04-26T23:01:21.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.059+0000 7efc30e00640 1 -- start start 2024-04-26T23:01:21.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.060+0000 7efc30e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc2c162a20 0x7efc2c0d1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.060+0000 7efc30e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efc2c0d1c40 0x7efc2c0d20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.061+0000 7efc30e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc2c0c8bc0 con 0x7efc2c0d2ed0 2024-04-26T23:01:21.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.061+0000 7efc30e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7efc2c0d07a0 con 0x7efc2c162a20 2024-04-26T23:01:21.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.061+0000 7efc30e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7efc2c0d09f0 con 0x7efc2c0d1c40 2024-04-26T23:01:21.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.061+0000 7efc2a800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efc2c0d2ed0 0x7efc2c0d3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53962/0 (socket says 172.21.15.40:53962) 2024-04-26T23:01:21.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.061+0000 7efc2a800640 1 -- 172.21.15.40:0/3158434579 learned_addr learned my addr 172.21.15.40:0/3158434579 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:21.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.061+0000 7efc29e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc2c162a20 0x7efc2c0d1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.061+0000 7efc23e00640 1 --2- 172.21.15.40:0/3158434579 >> v2:172.21.15.103:3300/0 conn(0x7efc2c0d1c40 0x7efc2c0d20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.062+0000 7efc29e00640 1 -- 172.21.15.40:0/3158434579 >> v2:172.21.15.103:3300/0 conn(0x7efc2c0d1c40 msgr2=0x7efc2c0d20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:21.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.062+0000 7efc29e00640 1 --2- 172.21.15.40:0/3158434579 >> v2:172.21.15.103:3300/0 conn(0x7efc2c0d1c40 0x7efc2c0d20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.062+0000 7efc29e00640 1 -- 172.21.15.40:0/3158434579 >> v1:172.21.15.40:6789/0 conn(0x7efc2c0d2ed0 legacy=0x7efc2c0d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.062+0000 7efc29400640 1 -- 172.21.15.40:0/3158434579 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4125309576 0 0) 0x7efc2c0c8bc0 con 0x7efc2c0d2ed0 2024-04-26T23:01:21.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.062+0000 7efc29e00640 1 -- 172.21.15.40:0/3158434579 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc2c0d0cd0 con 0x7efc2c162a20 2024-04-26T23:01:21.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.063+0000 7efc29e00640 1 --2- 172.21.15.40:0/3158434579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc2c162a20 0x7efc2c0d1700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0762e0 tx=0x7efc1c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bad8749b4640144c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:21.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.063+0000 7efc29400640 1 -- 172.21.15.40:0/3158434579 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc1c037070 con 0x7efc2c162a20 2024-04-26T23:01:21.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.063+0000 7efc29400640 1 -- 172.21.15.40:0/3158434579 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc1c032440 con 0x7efc2c162a20 2024-04-26T23:01:21.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.063+0000 7efc29400640 1 -- 172.21.15.40:0/3158434579 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc1c02ae60 con 0x7efc2c162a20 2024-04-26T23:01:21.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.064+0000 7efc30e00640 1 -- 172.21.15.40:0/3158434579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc2c162a20 msgr2=0x7efc2c0d1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.064+0000 7efc30e00640 1 --2- 172.21.15.40:0/3158434579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc2c162a20 0x7efc2c0d1700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0762e0 tx=0x7efc1c02a110 comp rx=0 tx=0).stop 2024-04-26T23:01:21.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.064+0000 7efc30e00640 1 -- 172.21.15.40:0/3158434579 shutdown_connections 2024-04-26T23:01:21.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.064+0000 7efc30e00640 1 --2- 172.21.15.40:0/3158434579 >> v2:172.21.15.103:3300/0 conn(0x7efc2c0d1c40 0x7efc2c0d20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.064+0000 7efc30e00640 1 --2- 172.21.15.40:0/3158434579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc2c162a20 0x7efc2c0d1700 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.064+0000 7efc30e00640 1 -- 172.21.15.40:0/3158434579 >> 172.21.15.40:0/3158434579 conn(0x7efc2c0d4fd0 msgr2=0x7efc2c0d53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:21.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.064+0000 7efc30e00640 1 -- 172.21.15.40:0/3158434579 shutdown_connections 2024-04-26T23:01:21.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.064+0000 7efc30e00640 1 -- 172.21.15.40:0/3158434579 wait complete. 2024-04-26T23:01:21.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.066+0000 7efc30e00640 1 Processor -- start 2024-04-26T23:01:21.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.066+0000 7efc30e00640 1 -- start start 2024-04-26T23:01:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.067+0000 7efc30e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc2c0d1c40 0x7efc2c15a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.067+0000 7efc30e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc2c0d2ed0 0x7efc2c15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.067+0000 7efc30e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc2c162a20 0x7efc2c15cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.067+0000 7efc30e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc2c0cbd90 con 0x7efc2c162a20 2024-04-26T23:01:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.067+0000 7efc30e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc2c0d0fb0 con 0x7efc2c0d2ed0 2024-04-26T23:01:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.067+0000 7efc30e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efc2c162e00 con 0x7efc2c0d1c40 2024-04-26T23:01:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.067+0000 7efc2a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc2c162a20 0x7efc2c15cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.067+0000 7efc2a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc2c162a20 0x7efc2c15cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35122/0 (socket says 172.21.15.40:35122) 2024-04-26T23:01:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.067+0000 7efc2a800640 1 -- 172.21.15.40:0/3160080350 learned_addr learned my addr 172.21.15.40:0/3160080350 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.068+0000 7efc29e00640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc2c0d1c40 0x7efc2c15a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.068+0000 7efc23e00640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc2c0d2ed0 0x7efc2c15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.068+0000 7efc2a800640 1 -- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc2c0d1c40 msgr2=0x7efc2c15a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.068+0000 7efc2a800640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc2c0d1c40 0x7efc2c15a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.068+0000 7efc2a800640 1 -- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc2c0d2ed0 msgr2=0x7efc2c15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.068+0000 7efc2a800640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc2c0d2ed0 0x7efc2c15c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.068+0000 7efc2a800640 1 -- 172.21.15.40:0/3160080350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc2c15b010 con 0x7efc2c162a20 2024-04-26T23:01:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.068+0000 7efc2a800640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc2c162a20 0x7efc2c15cd00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efc2400c7a0 tx=0x7efc2400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:21.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.068+0000 7efc22000640 1 -- 172.21.15.40:0/3160080350 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc24018070 con 0x7efc2c162a20 2024-04-26T23:01:21.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.068+0000 7efc22000640 1 -- 172.21.15.40:0/3160080350 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc2400a640 con 0x7efc2c162a20 2024-04-26T23:01:21.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.068+0000 7efc22000640 1 -- 172.21.15.40:0/3160080350 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2400aab0 con 0x7efc2c162a20 2024-04-26T23:01:21.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.068+0000 7efc30e00640 1 -- 172.21.15.40:0/3160080350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc2c15b2f0 con 0x7efc2c162a20 2024-04-26T23:01:21.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.069+0000 7efc30e00640 1 -- 172.21.15.40:0/3160080350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc2c15b8b0 con 0x7efc2c162a20 2024-04-26T23:01:21.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.070+0000 7efc30e00640 1 -- 172.21.15.40:0/3160080350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc2c0694f0 con 0x7efc2c162a20 2024-04-26T23:01:21.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.071+0000 7efc22000640 1 -- 172.21.15.40:0/3160080350 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efc2400ac50 con 0x7efc2c162a20 2024-04-26T23:01:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.072+0000 7efc22000640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc0806c8f0 0x7efc0806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.072+0000 7efc22000640 1 -- 172.21.15.40:0/3160080350 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7efc2408b600 con 0x7efc2c162a20 2024-04-26T23:01:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.073+0000 7efc29e00640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc0806c8f0 0x7efc0806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.074+0000 7efc29e00640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc0806c8f0 0x7efc0806edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efc1c034000 tx=0x7efc1c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.076+0000 7efc22000640 1 -- 172.21.15.40:0/3160080350 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7efc24059650 con 0x7efc2c162a20 2024-04-26T23:01:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.286+0000 7efc30e00640 1 -- 172.21.15.40:0/3160080350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efc2c159c20 con 0x7efc2c162a20 2024-04-26T23:01:21.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.287+0000 7efc22000640 1 -- 172.21.15.40:0/3160080350 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7efc24058f50 con 0x7efc2c162a20 2024-04-26T23:01:21.287 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:01:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.290+0000 7efc30e00640 1 -- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc0806c8f0 msgr2=0x7efc0806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.290+0000 7efc30e00640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc0806c8f0 0x7efc0806edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efc1c034000 tx=0x7efc1c02e000 comp rx=0 tx=0).stop 2024-04-26T23:01:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.290+0000 7efc30e00640 1 -- 172.21.15.40:0/3160080350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc2c162a20 msgr2=0x7efc2c15cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.290+0000 7efc30e00640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc2c162a20 0x7efc2c15cd00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efc2400c7a0 tx=0x7efc2400cc60 comp rx=0 tx=0).stop 2024-04-26T23:01:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.290+0000 7efc30e00640 1 -- 172.21.15.40:0/3160080350 shutdown_connections 2024-04-26T23:01:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.290+0000 7efc30e00640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc0806c8f0 0x7efc0806edb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.290+0000 7efc30e00640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc2c162a20 0x7efc2c15cd00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.290+0000 7efc30e00640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc2c0d2ed0 0x7efc2c15c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.290+0000 7efc30e00640 1 --2- 172.21.15.40:0/3160080350 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc2c0d1c40 0x7efc2c15a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.290+0000 7efc30e00640 1 -- 172.21.15.40:0/3160080350 >> 172.21.15.40:0/3160080350 conn(0x7efc2c0d4fd0 msgr2=0x7efc2c160fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:21.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.290+0000 7efc30e00640 1 -- 172.21.15.40:0/3160080350 shutdown_connections 2024-04-26T23:01:21.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.290+0000 7efc30e00640 1 -- 172.21.15.40:0/3160080350 wait complete. 2024-04-26T23:01:21.309 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:01:21.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.528+0000 7f0d82c00640 1 Processor -- start 2024-04-26T23:01:21.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.529+0000 7f0d82c00640 1 -- start start 2024-04-26T23:01:21.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.530+0000 7f0d82c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d7c160a50 0x7f0d7c073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.530+0000 7f0d82c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0d7c06dd40 0x7f0d7c072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.530+0000 7f0d82c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d7c161880 con 0x7f0d7c0cfaf0 2024-04-26T23:01:21.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.530+0000 7f0d82c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0d7c071ce0 con 0x7f0d7c06dd40 2024-04-26T23:01:21.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.530+0000 7f0d82c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0d7c0722a0 con 0x7f0d7c160a50 2024-04-26T23:01:21.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.531+0000 7f0d81800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d7c160a50 0x7f0d7c073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.531+0000 7f0d81800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d7c160a50 0x7f0d7c073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:40686/0 (socket says 172.21.15.40:40686) 2024-04-26T23:01:21.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.531+0000 7f0d81800640 1 -- 172.21.15.40:0/1210482337 learned_addr learned my addr 172.21.15.40:0/1210482337 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:21.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.531+0000 7f0d82200640 1 --2- 172.21.15.40:0/1210482337 >> v2:172.21.15.103:3300/0 conn(0x7f0d7c06dd40 0x7f0d7c072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.532+0000 7f0d81800640 1 -- 172.21.15.40:0/1210482337 >> v2:172.21.15.103:3300/0 conn(0x7f0d7c06dd40 msgr2=0x7f0d7c072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:21.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.532+0000 7f0d81800640 1 --2- 172.21.15.40:0/1210482337 >> v2:172.21.15.103:3300/0 conn(0x7f0d7c06dd40 0x7f0d7c072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.532+0000 7f0d81800640 1 -- 172.21.15.40:0/1210482337 >> v1:172.21.15.103:6789/0 conn(0x7f0d7c0cfaf0 legacy=0x7f0d7c160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.532+0000 7f0d81800640 1 -- 172.21.15.40:0/1210482337 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7c0cff80 con 0x7f0d7c160a50 2024-04-26T23:01:21.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.533+0000 7f0d81800640 1 --2- 172.21.15.40:0/1210482337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d7c160a50 0x7f0d7c073a50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c009870 tx=0x7f0d6c02a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b28cbca8213b44e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:21.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.534+0000 7f0d7be00640 1 -- 172.21.15.40:0/1210482337 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c037070 con 0x7f0d7c160a50 2024-04-26T23:01:21.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.534+0000 7f0d7be00640 1 -- 172.21.15.40:0/1210482337 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d6c02adf0 con 0x7f0d7c160a50 2024-04-26T23:01:21.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.534+0000 7f0d7be00640 1 -- 172.21.15.40:0/1210482337 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c032660 con 0x7f0d7c160a50 2024-04-26T23:01:21.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.534+0000 7f0d82c00640 1 -- 172.21.15.40:0/1210482337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d7c160a50 msgr2=0x7f0d7c073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.534+0000 7f0d82c00640 1 --2- 172.21.15.40:0/1210482337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d7c160a50 0x7f0d7c073a50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c009870 tx=0x7f0d6c02a160 comp rx=0 tx=0).stop 2024-04-26T23:01:21.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.534+0000 7f0d82c00640 1 -- 172.21.15.40:0/1210482337 shutdown_connections 2024-04-26T23:01:21.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.535+0000 7f0d82c00640 1 --2- 172.21.15.40:0/1210482337 >> v2:172.21.15.103:3300/0 conn(0x7f0d7c06dd40 0x7f0d7c072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.535+0000 7f0d82c00640 1 --2- 172.21.15.40:0/1210482337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d7c160a50 0x7f0d7c073a50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.535+0000 7f0d82c00640 1 -- 172.21.15.40:0/1210482337 >> 172.21.15.40:0/1210482337 conn(0x7f0d7c0b2720 msgr2=0x7f0d7c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:21.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.535+0000 7f0d82c00640 1 -- 172.21.15.40:0/1210482337 shutdown_connections 2024-04-26T23:01:21.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.535+0000 7f0d82c00640 1 -- 172.21.15.40:0/1210482337 wait complete. 2024-04-26T23:01:21.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.536+0000 7f0d82c00640 1 Processor -- start 2024-04-26T23:01:21.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.537+0000 7f0d82c00640 1 -- start start 2024-04-26T23:01:21.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.537+0000 7f0d82c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d7c06dd40 0x7f0d7c15a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.537+0000 7f0d82c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d7c160a50 0x7f0d7c145a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.538+0000 7f0d82c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d7c0b4100 0x7f0d7c139220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.538+0000 7f0d82c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7c071f30 con 0x7f0d7c0b4100 2024-04-26T23:01:21.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.538+0000 7f0d82c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7c073f90 con 0x7f0d7c06dd40 2024-04-26T23:01:21.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.538+0000 7f0d82c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0d7c1468f0 con 0x7f0d7c160a50 2024-04-26T23:01:21.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.538+0000 7f0d82200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d7c0b4100 0x7f0d7c139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.538+0000 7f0d82200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d7c0b4100 0x7f0d7c139220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35128/0 (socket says 172.21.15.40:35128) 2024-04-26T23:01:21.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.538+0000 7f0d82200640 1 -- 172.21.15.40:0/1989003561 learned_addr learned my addr 172.21.15.40:0/1989003561 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:21.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.538+0000 7f0d81800640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d7c06dd40 0x7f0d7c15a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.538+0000 7f0d80e00640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d7c160a50 0x7f0d7c145a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.538+0000 7f0d82200640 1 -- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d7c160a50 msgr2=0x7f0d7c145a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.538+0000 7f0d82200640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d7c160a50 0x7f0d7c145a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.538+0000 7f0d82200640 1 -- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d7c06dd40 msgr2=0x7f0d7c15a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.538+0000 7f0d82200640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d7c06dd40 0x7f0d7c15a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.539+0000 7f0d82200640 1 -- 172.21.15.40:0/1989003561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7c1f5620 con 0x7f0d7c0b4100 2024-04-26T23:01:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.539+0000 7f0d82200640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d7c0b4100 0x7f0d7c139220 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d7400efc0 tx=0x7f0d7400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.539+0000 7f0d79600640 1 -- 172.21.15.40:0/1989003561 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d74018020 con 0x7f0d7c0b4100 2024-04-26T23:01:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.539+0000 7f0d79600640 1 -- 172.21.15.40:0/1989003561 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d74014480 con 0x7f0d7c0b4100 2024-04-26T23:01:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.539+0000 7f0d79600640 1 -- 172.21.15.40:0/1989003561 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d74014870 con 0x7f0d7c0b4100 2024-04-26T23:01:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.539+0000 7f0d82c00640 1 -- 172.21.15.40:0/1989003561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d7c15ad80 con 0x7f0d7c0b4100 2024-04-26T23:01:21.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.539+0000 7f0d82c00640 1 -- 172.21.15.40:0/1989003561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d7c136c60 con 0x7f0d7c0b4100 2024-04-26T23:01:21.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.541+0000 7f0d82c00640 1 -- 172.21.15.40:0/1989003561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d7c069370 con 0x7f0d7c0b4100 2024-04-26T23:01:21.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.542+0000 7f0d79600640 1 -- 172.21.15.40:0/1989003561 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0d74005d10 con 0x7f0d7c0b4100 2024-04-26T23:01:21.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.543+0000 7f0d79600640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0d6006c8f0 0x7f0d6006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.543+0000 7f0d79600640 1 -- 172.21.15.40:0/1989003561 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f0d7408ccd0 con 0x7f0d7c0b4100 2024-04-26T23:01:21.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.544+0000 7f0d81800640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0d6006c8f0 0x7f0d6006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.545+0000 7f0d81800640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0d6006c8f0 0x7f0d6006edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c034000 tx=0x7f0d6c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:21.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.547+0000 7f0d79600640 1 -- 172.21.15.40:0/1989003561 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0d74022030 con 0x7f0d7c0b4100 2024-04-26T23:01:21.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.755+0000 7f0d82c00640 1 -- 172.21.15.40:0/1989003561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0d7c13d0b0 con 0x7f0d7c0b4100 2024-04-26T23:01:21.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.756+0000 7f0d79600640 1 -- 172.21.15.40:0/1989003561 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f0d74010040 con 0x7f0d7c0b4100 2024-04-26T23:01:21.756 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:01:21.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.759+0000 7f0d82c00640 1 -- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0d6006c8f0 msgr2=0x7f0d6006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.759+0000 7f0d82c00640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0d6006c8f0 0x7f0d6006edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c034000 tx=0x7f0d6c02e000 comp rx=0 tx=0).stop 2024-04-26T23:01:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.759+0000 7f0d82c00640 1 -- 172.21.15.40:0/1989003561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d7c0b4100 msgr2=0x7f0d7c139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.759+0000 7f0d82c00640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d7c0b4100 0x7f0d7c139220 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d7400efc0 tx=0x7f0d7400c490 comp rx=0 tx=0).stop 2024-04-26T23:01:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.759+0000 7f0d82c00640 1 -- 172.21.15.40:0/1989003561 shutdown_connections 2024-04-26T23:01:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.759+0000 7f0d82c00640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0d6006c8f0 0x7f0d6006edb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.759+0000 7f0d82c00640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d7c0b4100 0x7f0d7c139220 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.759+0000 7f0d82c00640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d7c160a50 0x7f0d7c145a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.759+0000 7f0d82c00640 1 --2- 172.21.15.40:0/1989003561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d7c06dd40 0x7f0d7c15a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.759+0000 7f0d82c00640 1 -- 172.21.15.40:0/1989003561 >> 172.21.15.40:0/1989003561 conn(0x7f0d7c0b2720 msgr2=0x7f0d7c15fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.759+0000 7f0d82c00640 1 -- 172.21.15.40:0/1989003561 shutdown_connections 2024-04-26T23:01:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.759+0000 7f0d82c00640 1 -- 172.21.15.40:0/1989003561 wait complete. 2024-04-26T23:01:21.779 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:01:21.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.986+0000 7f721dc00640 1 Processor -- start 2024-04-26T23:01:21.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.986+0000 7f721dc00640 1 -- start start 2024-04-26T23:01:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.987+0000 7f721dc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7218071e60 0x7f721806f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.988+0000 7f721dc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f72180cc480 0x7f7218166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.988+0000 7f721dc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f721815d7c0 con 0x7f721806fe20 2024-04-26T23:01:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.988+0000 7f721dc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f721815daa0 con 0x7f72180cc480 2024-04-26T23:01:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.988+0000 7f721dc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f721815dd80 con 0x7f7218071e60 2024-04-26T23:01:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.988+0000 7f720e000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f721806fe20 0x7f7218072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53972/0 (socket says 172.21.15.40:53972) 2024-04-26T23:01:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.988+0000 7f720e000640 1 -- 172.21.15.40:0/1278482162 learned_addr learned my addr 172.21.15.40:0/1278482162 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.988+0000 7f7217400640 1 --2- 172.21.15.40:0/1278482162 >> v2:172.21.15.103:3300/0 conn(0x7f72180cc480 0x7f7218166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.988+0000 7f7216a00640 1 --2- 172.21.15.40:0/1278482162 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7218071e60 0x7f721806f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.989+0000 7f7216000640 1 -- 172.21.15.40:0/1278482162 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 185502227 0 0) 0x7f721815d7c0 con 0x7f721806fe20 2024-04-26T23:01:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.989+0000 7f7216000640 1 -- 172.21.15.40:0/1278482162 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71fc003520 con 0x7f721806fe20 2024-04-26T23:01:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.989+0000 7f7217400640 1 -- 172.21.15.40:0/1278482162 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7218071e60 msgr2=0x7f721806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.989+0000 7f7217400640 1 --2- 172.21.15.40:0/1278482162 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7218071e60 0x7f721806f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.989+0000 7f7217400640 1 -- 172.21.15.40:0/1278482162 >> v1:172.21.15.40:6789/0 conn(0x7f721806fe20 legacy=0x7f7218072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.989+0000 7f7216000640 1 -- 172.21.15.40:0/1278482162 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1653060795 0 0) 0x7f71fc003520 con 0x7f721806fe20 2024-04-26T23:01:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.989+0000 7f7217400640 1 -- 172.21.15.40:0/1278482162 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f721815e060 con 0x7f72180cc480 2024-04-26T23:01:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.989+0000 7f7216a00640 1 --2- 172.21.15.40:0/1278482162 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7218071e60 0x7f721806f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:01:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.990+0000 7f7217400640 1 --2- 172.21.15.40:0/1278482162 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72180cc480 0x7f7218166ad0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7210009d90 tx=0x7f721002a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db16014d170a7a1f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.991+0000 7f7216000640 1 -- 172.21.15.40:0/1278482162 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7200001670 con 0x7f72180cc480 2024-04-26T23:01:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.991+0000 7f7216000640 1 -- 172.21.15.40:0/1278482162 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7210007540 con 0x7f72180cc480 2024-04-26T23:01:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.991+0000 7f7216000640 1 -- 172.21.15.40:0/1278482162 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7210007910 con 0x7f72180cc480 2024-04-26T23:01:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.992+0000 7f721dc00640 1 -- 172.21.15.40:0/1278482162 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72180cc480 msgr2=0x7f7218166ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.992+0000 7f721dc00640 1 --2- 172.21.15.40:0/1278482162 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72180cc480 0x7f7218166ad0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7210009d90 tx=0x7f721002a5f0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.992+0000 7f721dc00640 1 -- 172.21.15.40:0/1278482162 shutdown_connections 2024-04-26T23:01:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.992+0000 7f721dc00640 1 --2- 172.21.15.40:0/1278482162 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72180cc480 0x7f7218166ad0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.992+0000 7f721dc00640 1 --2- 172.21.15.40:0/1278482162 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7218071e60 0x7f721806f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.992+0000 7f721dc00640 1 -- 172.21.15.40:0/1278482162 >> 172.21.15.40:0/1278482162 conn(0x7f72180d0be0 msgr2=0x7f72180d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.992+0000 7f721dc00640 1 -- 172.21.15.40:0/1278482162 shutdown_connections 2024-04-26T23:01:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.992+0000 7f721dc00640 1 -- 172.21.15.40:0/1278482162 wait complete. 2024-04-26T23:01:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.994+0000 7f721dc00640 1 Processor -- start 2024-04-26T23:01:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.994+0000 7f721dc00640 1 -- start start 2024-04-26T23:01:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.995+0000 7f721dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f721806fe20 0x7f72181f7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.995+0000 7f721dc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7218071e60 0x7f721815e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.995+0000 7f721dc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72180cc480 0x7f721815ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.995+0000 7f721dc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f721815d570 con 0x7f721806fe20 2024-04-26T23:01:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.995+0000 7f721dc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f721815d3f0 con 0x7f7218071e60 2024-04-26T23:01:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.995+0000 7f721dc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7218072950 con 0x7f72180cc480 2024-04-26T23:01:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.995+0000 7f7216a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f721806fe20 0x7f72181f7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.995+0000 7f7216a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f721806fe20 0x7f72181f7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35136/0 (socket says 172.21.15.40:35136) 2024-04-26T23:01:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.995+0000 7f7216a00640 1 -- 172.21.15.40:0/2969023812 learned_addr learned my addr 172.21.15.40:0/2969023812 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.996+0000 7f7216a00640 1 -- 172.21.15.40:0/2969023812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72180cc480 msgr2=0x7f721815ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.996+0000 7f7216a00640 1 --2- 172.21.15.40:0/2969023812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72180cc480 0x7f721815ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.996+0000 7f7216a00640 1 -- 172.21.15.40:0/2969023812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7218071e60 msgr2=0x7f721815e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.996+0000 7f7216a00640 1 --2- 172.21.15.40:0/2969023812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7218071e60 0x7f721815e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.996+0000 7f7216a00640 1 -- 172.21.15.40:0/2969023812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f721815fe00 con 0x7f721806fe20 2024-04-26T23:01:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.996+0000 7f7216a00640 1 --2- 172.21.15.40:0/2969023812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f721806fe20 0x7f72181f7bc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f720800ec10 tx=0x7f720800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.996+0000 7f720f400640 1 -- 172.21.15.40:0/2969023812 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7208017070 con 0x7f721806fe20 2024-04-26T23:01:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.996+0000 7f720f400640 1 -- 172.21.15.40:0/2969023812 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f720800cbb0 con 0x7f721806fe20 2024-04-26T23:01:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.996+0000 7f721dc00640 1 -- 172.21.15.40:0/2969023812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f721815f560 con 0x7f721806fe20 2024-04-26T23:01:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.996+0000 7f720f400640 1 -- 172.21.15.40:0/2969023812 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7208013470 con 0x7f721806fe20 2024-04-26T23:01:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.997+0000 7f721dc00640 1 -- 172.21.15.40:0/2969023812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f721815fb20 con 0x7f721806fe20 2024-04-26T23:01:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.998+0000 7f721dc00640 1 -- 172.21.15.40:0/2969023812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72180694f0 con 0x7f721806fe20 2024-04-26T23:01:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:21.999+0000 7f720f400640 1 -- 172.21.15.40:0/2969023812 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f720801a070 con 0x7f721806fe20 2024-04-26T23:01:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.000+0000 7f720f400640 1 --2- 172.21.15.40:0/2969023812 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f71f406c8f0 0x7f71f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.000+0000 7f720f400640 1 -- 172.21.15.40:0/2969023812 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f720808c0e0 con 0x7f721806fe20 2024-04-26T23:01:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.000+0000 7f720e000640 1 --2- 172.21.15.40:0/2969023812 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f71f406c8f0 0x7f71f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.001+0000 7f720e000640 1 --2- 172.21.15.40:0/2969023812 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f71f406c8f0 0x7f71f406edb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7200000f60 tx=0x7f7200001060 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:22.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.004+0000 7f720f400640 1 -- 172.21.15.40:0/2969023812 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f721815fe00 con 0x7f721806fe20 2024-04-26T23:01:22.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.223+0000 7f721dc00640 1 -- 172.21.15.40:0/2969023812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7218072450 con 0x7f721806fe20 2024-04-26T23:01:22.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.224+0000 7f720f400640 1 -- 172.21.15.40:0/2969023812 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7f720805a260 con 0x7f721806fe20 2024-04-26T23:01:22.224 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-04-26T23:01:22.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.226+0000 7f721dc00640 1 -- 172.21.15.40:0/2969023812 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f71f406c8f0 msgr2=0x7f71f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:22.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.226+0000 7f721dc00640 1 --2- 172.21.15.40:0/2969023812 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f71f406c8f0 0x7f71f406edb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7200000f60 tx=0x7f7200001060 comp rx=0 tx=0).stop 2024-04-26T23:01:22.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.226+0000 7f721dc00640 1 -- 172.21.15.40:0/2969023812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f721806fe20 msgr2=0x7f72181f7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:22.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.226+0000 7f721dc00640 1 --2- 172.21.15.40:0/2969023812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f721806fe20 0x7f72181f7bc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f720800ec10 tx=0x7f720800c520 comp rx=0 tx=0).stop 2024-04-26T23:01:22.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.226+0000 7f721dc00640 1 -- 172.21.15.40:0/2969023812 shutdown_connections 2024-04-26T23:01:22.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.226+0000 7f721dc00640 1 --2- 172.21.15.40:0/2969023812 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f71f406c8f0 0x7f71f406edb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.226+0000 7f721dc00640 1 --2- 172.21.15.40:0/2969023812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72180cc480 0x7f721815ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.226+0000 7f721dc00640 1 --2- 172.21.15.40:0/2969023812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7218071e60 0x7f721815e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.226+0000 7f721dc00640 1 --2- 172.21.15.40:0/2969023812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f721806fe20 0x7f72181f7bc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.226+0000 7f721dc00640 1 -- 172.21.15.40:0/2969023812 >> 172.21.15.40:0/2969023812 conn(0x7f72180d0be0 msgr2=0x7f7218159340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:22.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.227+0000 7f721dc00640 1 -- 172.21.15.40:0/2969023812 shutdown_connections 2024-04-26T23:01:22.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.227+0000 7f721dc00640 1 -- 172.21.15.40:0/2969023812 wait complete. 2024-04-26T23:01:22.246 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:01:22.246 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:01:22.301 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:22.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.512+0000 7f28e7200640 1 Processor -- start 2024-04-26T23:01:22.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.512+0000 7f28e7200640 1 -- start start 2024-04-26T23:01:22.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.513+0000 7f28e7200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f28e80b6220 0x7f28e80d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.513+0000 7f28e7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28e80d08d0 0x7f28e80d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.513+0000 7f28e7200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f28e80d3820 0x7f28e80b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.513+0000 7f28e7200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f28e8162c40 con 0x7f28e80b6220 2024-04-26T23:01:22.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.513+0000 7f28e7200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f28e80d2d80 con 0x7f28e80d3820 2024-04-26T23:01:22.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.513+0000 7f28e7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f28e80b7c80 con 0x7f28e80d08d0 2024-04-26T23:01:22.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.514+0000 7f28e5e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f28e80b6220 0x7f28e80d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.514+0000 7f28e5e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f28e80b6220 0x7f28e80d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35138/0 (socket says 172.21.15.40:35138) 2024-04-26T23:01:22.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.514+0000 7f28e5e00640 1 -- 172.21.15.40:0/1221724366 learned_addr learned my addr 172.21.15.40:0/1221724366 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:22.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.514+0000 7f28e6800640 1 --2- 172.21.15.40:0/1221724366 >> v2:172.21.15.103:3300/0 conn(0x7f28e80d3820 0x7f28e80b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.514+0000 7f28e5400640 1 --2- 172.21.15.40:0/1221724366 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28e80d08d0 0x7f28e80d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.514+0000 7f28e5e00640 1 -- 172.21.15.40:0/1221724366 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28e80d08d0 msgr2=0x7f28e80d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:22.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.514+0000 7f28e5e00640 1 --2- 172.21.15.40:0/1221724366 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28e80d08d0 0x7f28e80d32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.514+0000 7f28e5e00640 1 -- 172.21.15.40:0/1221724366 >> v2:172.21.15.103:3300/0 conn(0x7f28e80d3820 msgr2=0x7f28e80b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:22.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.514+0000 7f28e5e00640 1 --2- 172.21.15.40:0/1221724366 >> v2:172.21.15.103:3300/0 conn(0x7f28e80d3820 0x7f28e80b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.514+0000 7f28e5e00640 1 -- 172.21.15.40:0/1221724366 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e80b7f60 con 0x7f28e80b6220 2024-04-26T23:01:22.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.515+0000 7f28e5e00640 1 --2- 172.21.15.40:0/1221724366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28e80b6220 0x7f28e80d0390 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f28e80b7920 tx=0x7f28d802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e776a1a69f4414 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:22.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.516+0000 7f28e4a00640 1 -- 172.21.15.40:0/1221724366 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d8037070 con 0x7f28e80b6220 2024-04-26T23:01:22.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.516+0000 7f28e4a00640 1 -- 172.21.15.40:0/1221724366 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28d8032440 con 0x7f28e80b6220 2024-04-26T23:01:22.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.516+0000 7f28e4a00640 1 -- 172.21.15.40:0/1221724366 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d8032810 con 0x7f28e80b6220 2024-04-26T23:01:22.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.516+0000 7f28e7200640 1 -- 172.21.15.40:0/1221724366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28e80b6220 msgr2=0x7f28e80d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:22.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.516+0000 7f28e7200640 1 --2- 172.21.15.40:0/1221724366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28e80b6220 0x7f28e80d0390 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f28e80b7920 tx=0x7f28d802a1e0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.516+0000 7f28e7200640 1 -- 172.21.15.40:0/1221724366 shutdown_connections 2024-04-26T23:01:22.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.516+0000 7f28e7200640 1 --2- 172.21.15.40:0/1221724366 >> v2:172.21.15.103:3300/0 conn(0x7f28e80d3820 0x7f28e80b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.516+0000 7f28e7200640 1 --2- 172.21.15.40:0/1221724366 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28e80d08d0 0x7f28e80d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.516+0000 7f28e7200640 1 --2- 172.21.15.40:0/1221724366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28e80b6220 0x7f28e80d0390 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.516+0000 7f28e7200640 1 -- 172.21.15.40:0/1221724366 >> 172.21.15.40:0/1221724366 conn(0x7f28e80d4d90 msgr2=0x7f28e80d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:22.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.516+0000 7f28e7200640 1 -- 172.21.15.40:0/1221724366 shutdown_connections 2024-04-26T23:01:22.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.517+0000 7f28e7200640 1 -- 172.21.15.40:0/1221724366 wait complete. 2024-04-26T23:01:22.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.518+0000 7f28e7200640 1 Processor -- start 2024-04-26T23:01:22.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.518+0000 7f28e7200640 1 -- start start 2024-04-26T23:01:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.519+0000 7f28e7200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28e80b6220 0x7f28e80caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.519+0000 7f28e7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28e80d08d0 0x7f28e815cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.519+0000 7f28e7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28e80d3820 0x7f28e81e91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.519+0000 7f28e7200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f28e80b6d50 con 0x7f28e80b6220 2024-04-26T23:01:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.519+0000 7f28e7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f28e80b8520 con 0x7f28e80d08d0 2024-04-26T23:01:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.519+0000 7f28e7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f28e80b6ed0 con 0x7f28e80d3820 2024-04-26T23:01:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.520+0000 7f28e5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28e80b6220 0x7f28e80caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.520+0000 7f28e5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28e80b6220 0x7f28e80caaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35146/0 (socket says 172.21.15.40:35146) 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.520+0000 7f28e5e00640 1 -- 172.21.15.40:0/1441650373 learned_addr learned my addr 172.21.15.40:0/1441650373 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.520+0000 7f28e6800640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28e80d3820 0x7f28e81e91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.520+0000 7f28e5400640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28e80d08d0 0x7f28e815cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.520+0000 7f28e5e00640 1 -- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28e80d3820 msgr2=0x7f28e81e91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.520+0000 7f28e5e00640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28e80d3820 0x7f28e81e91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.520+0000 7f28e5e00640 1 -- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28e80d08d0 msgr2=0x7f28e815cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.520+0000 7f28e5e00640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28e80d08d0 0x7f28e815cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.520+0000 7f28e5e00640 1 -- 172.21.15.40:0/1441650373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e815d5d0 con 0x7f28e80b6220 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.520+0000 7f28e6800640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28e80d3820 0x7f28e81e91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.521+0000 7f28e5e00640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28e80b6220 0x7f28e80caaa0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f28d802a220 tx=0x7f28d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.521+0000 7f28de000640 1 -- 172.21.15.40:0/1441650373 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d8041020 con 0x7f28e80b6220 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.521+0000 7f28de000640 1 -- 172.21.15.40:0/1441650373 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28d80306c0 con 0x7f28e80b6220 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.521+0000 7f28de000640 1 -- 172.21.15.40:0/1441650373 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d8037040 con 0x7f28e80b6220 2024-04-26T23:01:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.521+0000 7f28e7200640 1 -- 172.21.15.40:0/1441650373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e815d8b0 con 0x7f28e80b6220 2024-04-26T23:01:22.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.521+0000 7f28e7200640 1 -- 172.21.15.40:0/1441650373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e815de70 con 0x7f28e80b6220 2024-04-26T23:01:22.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.522+0000 7f28e7200640 1 -- 172.21.15.40:0/1441650373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28e80b9f40 con 0x7f28e80b6220 2024-04-26T23:01:22.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.524+0000 7f28de000640 1 -- 172.21.15.40:0/1441650373 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f28d8044020 con 0x7f28e80b6220 2024-04-26T23:01:22.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.524+0000 7f28de000640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f28c806c920 0x7f28c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.525+0000 7f28de000640 1 -- 172.21.15.40:0/1441650373 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f28d80ac760 con 0x7f28e80b6220 2024-04-26T23:01:22.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.525+0000 7f28e5400640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f28c806c920 0x7f28c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.526+0000 7f28e5400640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f28c806c920 0x7f28c806ede0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f28e815c580 tx=0x7f28d4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:22.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.529+0000 7f28de000640 1 -- 172.21.15.40:0/1441650373 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f28e80b9f40 con 0x7f28e80b6220 2024-04-26T23:01:22.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.730+0000 7f28e7200640 1 -- 172.21.15.40:0/1441650373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28e8069370 con 0x7f28e80b6220 2024-04-26T23:01:22.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.733+0000 7f28de000640 1 -- 172.21.15.40:0/1441650373 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f28d804b030 con 0x7f28e80b6220 2024-04-26T23:01:22.734 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:22.734 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:11.763965+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:22.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.736+0000 7f28e7200640 1 -- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f28c806c920 msgr2=0x7f28c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:22.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.736+0000 7f28e7200640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f28c806c920 0x7f28c806ede0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f28e815c580 tx=0x7f28d4008040 comp rx=0 tx=0).stop 2024-04-26T23:01:22.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.736+0000 7f28e7200640 1 -- 172.21.15.40:0/1441650373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28e80b6220 msgr2=0x7f28e80caaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:22.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.736+0000 7f28e7200640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28e80b6220 0x7f28e80caaa0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f28d802a220 tx=0x7f28d80046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.736+0000 7f28e7200640 1 -- 172.21.15.40:0/1441650373 shutdown_connections 2024-04-26T23:01:22.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.736+0000 7f28e7200640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f28c806c920 0x7f28c806ede0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.736+0000 7f28e7200640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28e80d3820 0x7f28e81e91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.736+0000 7f28e7200640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28e80d08d0 0x7f28e815cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.736+0000 7f28e7200640 1 --2- 172.21.15.40:0/1441650373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28e80b6220 0x7f28e80caaa0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.736+0000 7f28e7200640 1 -- 172.21.15.40:0/1441650373 >> 172.21.15.40:0/1441650373 conn(0x7f28e80d4d90 msgr2=0x7f28e8162320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:22.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.736+0000 7f28e7200640 1 -- 172.21.15.40:0/1441650373 shutdown_connections 2024-04-26T23:01:22.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.737+0000 7f28e7200640 1 -- 172.21.15.40:0/1441650373 wait complete. 2024-04-26T23:01:22.756 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:01:22.757 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:01:22.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.966+0000 7f2974a00640 1 Processor -- start 2024-04-26T23:01:22.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.966+0000 7f2974a00640 1 -- start start 2024-04-26T23:01:22.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.968+0000 7f2974a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f29700cfaf0 0x7f2970073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.968+0000 7f2974a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f297006dd40 0x7f2970072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.968+0000 7f2974a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2970161880 con 0x7f2970160a50 2024-04-26T23:01:22.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.968+0000 7f2974a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f2970071ce0 con 0x7f29700cfaf0 2024-04-26T23:01:22.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.968+0000 7f2974a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f29700722a0 con 0x7f297006dd40 2024-04-26T23:01:22.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.968+0000 7f2967e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f29700cfaf0 0x7f2970073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.968+0000 7f2967e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f29700cfaf0 0x7f2970073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35156/0 (socket says 172.21.15.40:35156) 2024-04-26T23:01:22.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.968+0000 7f2967e00640 1 -- 172.21.15.40:0/115080975 learned_addr learned my addr 172.21.15.40:0/115080975 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:22.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.969+0000 7f296f200640 1 --2- 172.21.15.40:0/115080975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f297006dd40 0x7f2970072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.969+0000 7f2967e00640 1 -- 172.21.15.40:0/115080975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f297006dd40 msgr2=0x7f2970072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:22.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.969+0000 7f2967e00640 1 --2- 172.21.15.40:0/115080975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f297006dd40 0x7f2970072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.969+0000 7f2967e00640 1 -- 172.21.15.40:0/115080975 >> v1:172.21.15.103:6789/0 conn(0x7f2970160a50 legacy=0x7f2970160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:22.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.969+0000 7f2967e00640 1 -- 172.21.15.40:0/115080975 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29700cff80 con 0x7f29700cfaf0 2024-04-26T23:01:22.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.969+0000 7f296f200640 1 --2- 172.21.15.40:0/115080975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f297006dd40 0x7f2970072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:22.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.970+0000 7f2967e00640 1 --2- 172.21.15.40:0/115080975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29700cfaf0 0x7f2970073a50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2958009870 tx=0x7f295802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9391c86141b03399 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:22.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.970+0000 7f296de00640 1 -- 172.21.15.40:0/115080975 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958037070 con 0x7f29700cfaf0 2024-04-26T23:01:22.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.970+0000 7f296de00640 1 -- 172.21.15.40:0/115080975 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2958032440 con 0x7f29700cfaf0 2024-04-26T23:01:22.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.970+0000 7f296de00640 1 -- 172.21.15.40:0/115080975 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958032810 con 0x7f29700cfaf0 2024-04-26T23:01:22.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.971+0000 7f2974a00640 1 -- 172.21.15.40:0/115080975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29700cfaf0 msgr2=0x7f2970073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:22.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.971+0000 7f2974a00640 1 --2- 172.21.15.40:0/115080975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29700cfaf0 0x7f2970073a50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2958009870 tx=0x7f295802a210 comp rx=0 tx=0).stop 2024-04-26T23:01:22.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.971+0000 7f2974a00640 1 -- 172.21.15.40:0/115080975 shutdown_connections 2024-04-26T23:01:22.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.971+0000 7f2974a00640 1 --2- 172.21.15.40:0/115080975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f297006dd40 0x7f2970072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.971+0000 7f2974a00640 1 --2- 172.21.15.40:0/115080975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29700cfaf0 0x7f2970073a50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.971+0000 7f2974a00640 1 -- 172.21.15.40:0/115080975 >> 172.21.15.40:0/115080975 conn(0x7f29700b2720 msgr2=0x7f29700b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:22.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.971+0000 7f2974a00640 1 -- 172.21.15.40:0/115080975 shutdown_connections 2024-04-26T23:01:22.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.971+0000 7f2974a00640 1 -- 172.21.15.40:0/115080975 wait complete. 2024-04-26T23:01:22.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.973+0000 7f2974a00640 1 Processor -- start 2024-04-26T23:01:22.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.973+0000 7f2974a00640 1 -- start start 2024-04-26T23:01:22.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.974+0000 7f2974a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f297006dd40 0x7f2970144d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.974+0000 7f2974a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2970160a50 0x7f29700ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.974+0000 7f2974a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29700aee30 0x7f29700b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.974+0000 7f2974a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2970071f30 con 0x7f2970160a50 2024-04-26T23:01:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.974+0000 7f2974a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2970073f90 con 0x7f297006dd40 2024-04-26T23:01:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.974+0000 7f2974a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f297006e1f0 con 0x7f29700aee30 2024-04-26T23:01:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.974+0000 7f2967e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2970160a50 0x7f29700ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.974+0000 7f2967e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2970160a50 0x7f29700ae8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35164/0 (socket says 172.21.15.40:35164) 2024-04-26T23:01:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.974+0000 7f2967e00640 1 -- 172.21.15.40:0/891993745 learned_addr learned my addr 172.21.15.40:0/891993745 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.975+0000 7f296f200640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29700aee30 0x7f29700b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.975+0000 7f296e800640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f297006dd40 0x7f2970144d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.975+0000 7f2967e00640 1 -- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29700aee30 msgr2=0x7f29700b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:22.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.975+0000 7f2967e00640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29700aee30 0x7f29700b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.975+0000 7f2967e00640 1 -- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f297006dd40 msgr2=0x7f2970144d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:22.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.975+0000 7f2967e00640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f297006dd40 0x7f2970144d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:22.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.975+0000 7f2967e00640 1 -- 172.21.15.40:0/891993745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29700b59e0 con 0x7f2970160a50 2024-04-26T23:01:22.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.975+0000 7f2967e00640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2970160a50 0x7f29700ae8f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f29580097c0 tx=0x7f29580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:22.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.975+0000 7f2966a00640 1 -- 172.21.15.40:0/891993745 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958041020 con 0x7f2970160a50 2024-04-26T23:01:22.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.975+0000 7f2966a00640 1 -- 172.21.15.40:0/891993745 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2958030970 con 0x7f2970160a50 2024-04-26T23:01:22.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.975+0000 7f2966a00640 1 -- 172.21.15.40:0/891993745 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958037040 con 0x7f2970160a50 2024-04-26T23:01:22.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.975+0000 7f2974a00640 1 -- 172.21.15.40:0/891993745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29701433c0 con 0x7f2970160a50 2024-04-26T23:01:22.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.976+0000 7f2974a00640 1 -- 172.21.15.40:0/891993745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2970143980 con 0x7f2970160a50 2024-04-26T23:01:22.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.977+0000 7f2974a00640 1 -- 172.21.15.40:0/891993745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2970069370 con 0x7f2970160a50 2024-04-26T23:01:22.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.978+0000 7f2966a00640 1 -- 172.21.15.40:0/891993745 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2958044020 con 0x7f2970160a50 2024-04-26T23:01:22.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.978+0000 7f2966a00640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f294c06c8f0 0x7f294c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:22.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.979+0000 7f2966a00640 1 -- 172.21.15.40:0/891993745 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f29580ac870 con 0x7f2970160a50 2024-04-26T23:01:22.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.979+0000 7f296e800640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f294c06c8f0 0x7f294c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:22.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.980+0000 7f296e800640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f294c06c8f0 0x7f294c06edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2960004ea0 tx=0x7f2960009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:22.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:22.983+0000 7f2966a00640 1 -- 172.21.15.40:0/891993745 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f295804b030 con 0x7f2970160a50 2024-04-26T23:01:23.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.188+0000 7f2974a00640 1 -- 172.21.15.40:0/891993745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f29700b2fd0 con 0x7f2970160a50 2024-04-26T23:01:23.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.189+0000 7f2966a00640 1 -- 172.21.15.40:0/891993745 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f295807a8c0 con 0x7f2970160a50 2024-04-26T23:01:23.190 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:01:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.192+0000 7f2974a00640 1 -- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f294c06c8f0 msgr2=0x7f294c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.192+0000 7f2974a00640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f294c06c8f0 0x7f294c06edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2960004ea0 tx=0x7f2960009040 comp rx=0 tx=0).stop 2024-04-26T23:01:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.192+0000 7f2974a00640 1 -- 172.21.15.40:0/891993745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2970160a50 msgr2=0x7f29700ae8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.192+0000 7f2974a00640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2970160a50 0x7f29700ae8f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f29580097c0 tx=0x7f29580046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.192+0000 7f2974a00640 1 -- 172.21.15.40:0/891993745 shutdown_connections 2024-04-26T23:01:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.192+0000 7f2974a00640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f294c06c8f0 0x7f294c06edb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.192+0000 7f2974a00640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f29700aee30 0x7f29700b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.192+0000 7f2974a00640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2970160a50 0x7f29700ae8f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.192+0000 7f2974a00640 1 --2- 172.21.15.40:0/891993745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f297006dd40 0x7f2970144d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.192+0000 7f2974a00640 1 -- 172.21.15.40:0/891993745 >> 172.21.15.40:0/891993745 conn(0x7f29700b2720 msgr2=0x7f29700b0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.193+0000 7f2974a00640 1 -- 172.21.15.40:0/891993745 shutdown_connections 2024-04-26T23:01:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.193+0000 7f2974a00640 1 -- 172.21.15.40:0/891993745 wait complete. 2024-04-26T23:01:23.212 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:01:23.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.422+0000 7fdc0be00640 1 Processor -- start 2024-04-26T23:01:23.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.422+0000 7fdc0be00640 1 -- start start 2024-04-26T23:01:23.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.423+0000 7fdc0be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc0c0b4e60 0x7fdc0c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:23.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.423+0000 7fdc0be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdc0c072220 0x7fdc0c06dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:23.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.423+0000 7fdc0be00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc0c06f800 con 0x7fdc0c06f420 2024-04-26T23:01:23.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.423+0000 7fdc0be00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fdc0c072670 con 0x7fdc0c072220 2024-04-26T23:01:23.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.423+0000 7fdc0be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdc0c0b3dc0 con 0x7fdc0c0b4e60 2024-04-26T23:01:23.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.424+0000 7fdc0aa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdc0c06f420 0x7fdc0c1644c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53984/0 (socket says 172.21.15.40:53984) 2024-04-26T23:01:23.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.424+0000 7fdc0aa00640 1 -- 172.21.15.40:0/726382839 learned_addr learned my addr 172.21.15.40:0/726382839 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:23.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.424+0000 7fdc0b400640 1 --2- 172.21.15.40:0/726382839 >> v2:172.21.15.40:3300/0 conn(0x7fdc0c072220 0x7fdc0c06dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:23.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.424+0000 7fdc03e00640 1 --2- 172.21.15.40:0/726382839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc0c0b4e60 0x7fdc0c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:23.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.424+0000 7fdc0a000640 1 -- 172.21.15.40:0/726382839 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3547928376 0 0) 0x7fdc0c06f800 con 0x7fdc0c06f420 2024-04-26T23:01:23.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.424+0000 7fdc0a000640 1 -- 172.21.15.40:0/726382839 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbf0003520 con 0x7fdc0c06f420 2024-04-26T23:01:23.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.424+0000 7fdc0b400640 1 -- 172.21.15.40:0/726382839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc0c0b4e60 msgr2=0x7fdc0c071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:23.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.424+0000 7fdc0b400640 1 --2- 172.21.15.40:0/726382839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc0c0b4e60 0x7fdc0c071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.424+0000 7fdc0b400640 1 -- 172.21.15.40:0/726382839 >> v1:172.21.15.40:6789/0 conn(0x7fdc0c06f420 legacy=0x7fdc0c1644c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:23.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.424+0000 7fdc0b400640 1 -- 172.21.15.40:0/726382839 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0c0b40a0 con 0x7fdc0c072220 2024-04-26T23:01:23.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.424+0000 7fdc03e00640 1 --2- 172.21.15.40:0/726382839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc0c0b4e60 0x7fdc0c071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:01:23.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.425+0000 7fdc0b400640 1 --2- 172.21.15.40:0/726382839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc0c072220 0x7fdc0c06dd40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009d90 tx=0x7fdc0402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dbee201e6d61365 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:23.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.426+0000 7fdc0a000640 1 -- 172.21.15.40:0/726382839 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04007740 con 0x7fdc0c072220 2024-04-26T23:01:23.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.426+0000 7fdc0a000640 1 -- 172.21.15.40:0/726382839 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc040078e0 con 0x7fdc0c072220 2024-04-26T23:01:23.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.426+0000 7fdc0a000640 1 -- 172.21.15.40:0/726382839 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04007cb0 con 0x7fdc0c072220 2024-04-26T23:01:23.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.426+0000 7fdc0be00640 1 -- 172.21.15.40:0/726382839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc0c072220 msgr2=0x7fdc0c06dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:23.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.426+0000 7fdc0be00640 1 --2- 172.21.15.40:0/726382839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc0c072220 0x7fdc0c06dd40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009d90 tx=0x7fdc0402a660 comp rx=0 tx=0).stop 2024-04-26T23:01:23.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.426+0000 7fdc0be00640 1 -- 172.21.15.40:0/726382839 shutdown_connections 2024-04-26T23:01:23.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.426+0000 7fdc0be00640 1 --2- 172.21.15.40:0/726382839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc0c072220 0x7fdc0c06dd40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.426+0000 7fdc0be00640 1 --2- 172.21.15.40:0/726382839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc0c0b4e60 0x7fdc0c071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.426+0000 7fdc0be00640 1 -- 172.21.15.40:0/726382839 >> 172.21.15.40:0/726382839 conn(0x7fdc0c0b2720 msgr2=0x7fdc0c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:23.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.427+0000 7fdc0be00640 1 -- 172.21.15.40:0/726382839 shutdown_connections 2024-04-26T23:01:23.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.427+0000 7fdc0be00640 1 -- 172.21.15.40:0/726382839 wait complete. 2024-04-26T23:01:23.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.428+0000 7fdc0be00640 1 Processor -- start 2024-04-26T23:01:23.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.428+0000 7fdc0be00640 1 -- start start 2024-04-26T23:01:23.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.429+0000 7fdc0be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc0c06f420 0x7fdc0c1494c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:23.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.429+0000 7fdc0be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc0c072220 0x7fdc0c14b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:23.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.429+0000 7fdc0be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc0c0b4e60 0x7fdc0c14ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:23.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.429+0000 7fdc0be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0c0b4940 con 0x7fdc0c072220 2024-04-26T23:01:23.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.429+0000 7fdc0be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0c0b4660 con 0x7fdc0c0b4e60 2024-04-26T23:01:23.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.429+0000 7fdc0be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdc0c073f50 con 0x7fdc0c06f420 2024-04-26T23:01:23.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.430+0000 7fdc03e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc0c072220 0x7fdc0c14b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:23.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.430+0000 7fdc03e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc0c072220 0x7fdc0c14b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35188/0 (socket says 172.21.15.40:35188) 2024-04-26T23:01:23.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.430+0000 7fdc03e00640 1 -- 172.21.15.40:0/1167124566 learned_addr learned my addr 172.21.15.40:0/1167124566 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:23.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.430+0000 7fdc0aa00640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc0c06f420 0x7fdc0c1494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:23.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.430+0000 7fdc0b400640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc0c0b4e60 0x7fdc0c14ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:23.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.430+0000 7fdc03e00640 1 -- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc0c06f420 msgr2=0x7fdc0c1494c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:23.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.430+0000 7fdc03e00640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc0c06f420 0x7fdc0c1494c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.430+0000 7fdc03e00640 1 -- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc0c0b4e60 msgr2=0x7fdc0c14ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:23.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.430+0000 7fdc03e00640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc0c0b4e60 0x7fdc0c14ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.430+0000 7fdc03e00640 1 -- 172.21.15.40:0/1167124566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0c149f60 con 0x7fdc0c072220 2024-04-26T23:01:23.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.431+0000 7fdc03e00640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc0c072220 0x7fdc0c14b550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400e7e0 tx=0x7fdbf400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:23.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.431+0000 7fdc02a00640 1 -- 172.21.15.40:0/1167124566 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf400f040 con 0x7fdc0c072220 2024-04-26T23:01:23.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.431+0000 7fdc02a00640 1 -- 172.21.15.40:0/1167124566 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbf4004030 con 0x7fdc0c072220 2024-04-26T23:01:23.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.431+0000 7fdc02a00640 1 -- 172.21.15.40:0/1167124566 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4004400 con 0x7fdc0c072220 2024-04-26T23:01:23.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.431+0000 7fdc0be00640 1 -- 172.21.15.40:0/1167124566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc0c14a240 con 0x7fdc0c072220 2024-04-26T23:01:23.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.431+0000 7fdc0be00640 1 -- 172.21.15.40:0/1167124566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc0c14a800 con 0x7fdc0c072220 2024-04-26T23:01:23.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.432+0000 7fdc0be00640 1 -- 172.21.15.40:0/1167124566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc0c069370 con 0x7fdc0c072220 2024-04-26T23:01:23.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.433+0000 7fdc02a00640 1 -- 172.21.15.40:0/1167124566 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdbf400c6d0 con 0x7fdc0c072220 2024-04-26T23:01:23.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.434+0000 7fdc02a00640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdbe806c8f0 0x7fdbe806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:23.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.434+0000 7fdc02a00640 1 -- 172.21.15.40:0/1167124566 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdbf408bd10 con 0x7fdc0c072220 2024-04-26T23:01:23.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.435+0000 7fdc0aa00640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdbe806c8f0 0x7fdbe806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:23.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.436+0000 7fdc0aa00640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdbe806c8f0 0x7fdbe806edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc005880 tx=0x7fdbfc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:23.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.438+0000 7fdc02a00640 1 -- 172.21.15.40:0/1167124566 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fdbf4091020 con 0x7fdc0c072220 2024-04-26T23:01:23.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.642+0000 7fdc0be00640 1 -- 172.21.15.40:0/1167124566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdc0c162230 con 0x7fdc0c072220 2024-04-26T23:01:23.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.643+0000 7fdc02a00640 1 -- 172.21.15.40:0/1167124566 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fdbf4017020 con 0x7fdc0c072220 2024-04-26T23:01:23.644 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:01:23.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.646+0000 7fdc0be00640 1 -- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdbe806c8f0 msgr2=0x7fdbe806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:23.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.646+0000 7fdc0be00640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdbe806c8f0 0x7fdbe806edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc005880 tx=0x7fdbfc009040 comp rx=0 tx=0).stop 2024-04-26T23:01:23.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.646+0000 7fdc0be00640 1 -- 172.21.15.40:0/1167124566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc0c072220 msgr2=0x7fdc0c14b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:23.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.646+0000 7fdc0be00640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc0c072220 0x7fdc0c14b550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400e7e0 tx=0x7fdbf400eca0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.646+0000 7fdc0be00640 1 -- 172.21.15.40:0/1167124566 shutdown_connections 2024-04-26T23:01:23.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.646+0000 7fdc0be00640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdbe806c8f0 0x7fdbe806edb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.646+0000 7fdc0be00640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc0c0b4e60 0x7fdc0c14ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.646+0000 7fdc0be00640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc0c072220 0x7fdc0c14b550 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.646+0000 7fdc0be00640 1 --2- 172.21.15.40:0/1167124566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc0c06f420 0x7fdc0c1494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.646+0000 7fdc0be00640 1 -- 172.21.15.40:0/1167124566 >> 172.21.15.40:0/1167124566 conn(0x7fdc0c0b2720 msgr2=0x7fdc0c06df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:23.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.646+0000 7fdc0be00640 1 -- 172.21.15.40:0/1167124566 shutdown_connections 2024-04-26T23:01:23.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.646+0000 7fdc0be00640 1 -- 172.21.15.40:0/1167124566 wait complete. 2024-04-26T23:01:23.666 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:01:23.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.883+0000 7f52cca00640 1 Processor -- start 2024-04-26T23:01:23.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.884+0000 7f52cca00640 1 -- start start 2024-04-26T23:01:23.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.885+0000 7f52cca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52c80c6f10 0x7f52c80c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:23.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.885+0000 7f52cca00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52c80c6010 0x7f52c8158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:23.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.885+0000 7f52cca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52c80cce10 con 0x7f52c80cc980 2024-04-26T23:01:23.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.885+0000 7f52cca00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f52c80c72f0 con 0x7f52c80c6010 2024-04-26T23:01:23.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.885+0000 7f52cca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f52c80cb860 con 0x7f52c80c6f10 2024-04-26T23:01:23.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.886+0000 7f52c7200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52c80c6010 0x7f52c8158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:23.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.886+0000 7f52c7200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52c80c6010 0x7f52c8158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35200/0 (socket says 172.21.15.40:35200) 2024-04-26T23:01:23.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.886+0000 7f52c7200640 1 -- 172.21.15.40:0/2422883698 learned_addr learned my addr 172.21.15.40:0/2422883698 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:23.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.886+0000 7f52c6800640 1 --2- 172.21.15.40:0/2422883698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52c80c6f10 0x7f52c80c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:23.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.886+0000 7f52c7200640 1 -- 172.21.15.40:0/2422883698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52c80c6f10 msgr2=0x7f52c80c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:23.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.886+0000 7f52c7200640 1 --2- 172.21.15.40:0/2422883698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52c80c6f10 0x7f52c80c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.886+0000 7f52c7200640 1 -- 172.21.15.40:0/2422883698 >> v1:172.21.15.103:6789/0 conn(0x7f52c80cc980 legacy=0x7f52c80cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:23.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.886+0000 7f52c7200640 1 -- 172.21.15.40:0/2422883698 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c80cd0f0 con 0x7f52c80c6010 2024-04-26T23:01:23.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.887+0000 7f52c7200640 1 --2- 172.21.15.40:0/2422883698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52c80c6010 0x7f52c8158cd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f52c0009870 tx=0x7f52c002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7939e9322d1404b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:23.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.888+0000 7f52c5e00640 1 -- 172.21.15.40:0/2422883698 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c0037070 con 0x7f52c80c6010 2024-04-26T23:01:23.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.888+0000 7f52c5e00640 1 -- 172.21.15.40:0/2422883698 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c0032440 con 0x7f52c80c6010 2024-04-26T23:01:23.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.888+0000 7f52c5e00640 1 -- 172.21.15.40:0/2422883698 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c0032810 con 0x7f52c80c6010 2024-04-26T23:01:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.888+0000 7f52cca00640 1 -- 172.21.15.40:0/2422883698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52c80c6010 msgr2=0x7f52c8158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.888+0000 7f52cca00640 1 --2- 172.21.15.40:0/2422883698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52c80c6010 0x7f52c8158cd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f52c0009870 tx=0x7f52c002a1c0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.888+0000 7f52cca00640 1 -- 172.21.15.40:0/2422883698 shutdown_connections 2024-04-26T23:01:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.888+0000 7f52cca00640 1 --2- 172.21.15.40:0/2422883698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52c80c6010 0x7f52c8158cd0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.888+0000 7f52cca00640 1 --2- 172.21.15.40:0/2422883698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52c80c6f10 0x7f52c80c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.888+0000 7f52cca00640 1 -- 172.21.15.40:0/2422883698 >> 172.21.15.40:0/2422883698 conn(0x7f52c80d0a10 msgr2=0x7f52c80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.889+0000 7f52cca00640 1 -- 172.21.15.40:0/2422883698 shutdown_connections 2024-04-26T23:01:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.889+0000 7f52cca00640 1 -- 172.21.15.40:0/2422883698 wait complete. 2024-04-26T23:01:23.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.890+0000 7f52cca00640 1 Processor -- start 2024-04-26T23:01:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.890+0000 7f52cca00640 1 -- start start 2024-04-26T23:01:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.891+0000 7f52cca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52c80c6010 0x7f52c815c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.891+0000 7f52cca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52c80c6f10 0x7f52c80b6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.891+0000 7f52cca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52c80b40f0 0x7f52c8148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.891+0000 7f52cca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f52c80cc400 con 0x7f52c80c6f10 2024-04-26T23:01:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.891+0000 7f52cca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f52c80cb1a0 con 0x7f52c80b40f0 2024-04-26T23:01:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.891+0000 7f52cca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f52c80cc580 con 0x7f52c80c6010 2024-04-26T23:01:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.892+0000 7f52bde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52c80c6f10 0x7f52c80b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.892+0000 7f52bde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52c80c6f10 0x7f52c80b6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35214/0 (socket says 172.21.15.40:35214) 2024-04-26T23:01:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.892+0000 7f52bde00640 1 -- 172.21.15.40:0/4202035033 learned_addr learned my addr 172.21.15.40:0/4202035033 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.892+0000 7f52c7200640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52c80b40f0 0x7f52c8148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.892+0000 7f52c6800640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52c80c6010 0x7f52c815c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.892+0000 7f52bde00640 1 -- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52c80c6010 msgr2=0x7f52c815c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.892+0000 7f52bde00640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52c80c6010 0x7f52c815c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.892+0000 7f52bde00640 1 -- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52c80b40f0 msgr2=0x7f52c8148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.892+0000 7f52bde00640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52c80b40f0 0x7f52c8148f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.892+0000 7f52bde00640 1 -- 172.21.15.40:0/4202035033 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c80b3980 con 0x7f52c80c6f10 2024-04-26T23:01:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.892+0000 7f52c6800640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52c80c6010 0x7f52c815c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.892+0000 7f52bde00640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52c80c6f10 0x7f52c80b6010 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f52b000c400 tx=0x7f52b000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.893+0000 7f52bf400640 1 -- 172.21.15.40:0/4202035033 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b00052d0 con 0x7f52c80c6f10 2024-04-26T23:01:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.893+0000 7f52bf400640 1 -- 172.21.15.40:0/4202035033 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b0005490 con 0x7f52c80c6f10 2024-04-26T23:01:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.893+0000 7f52bf400640 1 -- 172.21.15.40:0/4202035033 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0007260 con 0x7f52c80c6f10 2024-04-26T23:01:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.893+0000 7f52cca00640 1 -- 172.21.15.40:0/4202035033 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c80b3c60 con 0x7f52c80c6f10 2024-04-26T23:01:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.893+0000 7f52cca00640 1 -- 172.21.15.40:0/4202035033 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c815c330 con 0x7f52c80c6f10 2024-04-26T23:01:23.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.894+0000 7f52cca00640 1 -- 172.21.15.40:0/4202035033 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52c815c530 con 0x7f52c80c6f10 2024-04-26T23:01:23.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.895+0000 7f52bf400640 1 -- 172.21.15.40:0/4202035033 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f52b0007400 con 0x7f52c80c6f10 2024-04-26T23:01:23.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.896+0000 7f52bf400640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f52a406c8f0 0x7f52a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:23.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.896+0000 7f52bf400640 1 -- 172.21.15.40:0/4202035033 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f52b008b420 con 0x7f52c80c6f10 2024-04-26T23:01:23.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.897+0000 7f52c6800640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f52a406c8f0 0x7f52a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:23.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.898+0000 7f52c6800640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f52a406c8f0 0x7f52a406edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f52c815d230 tx=0x7f52b800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:23.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:23.901+0000 7f52bf400640 1 -- 172.21.15.40:0/4202035033 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f52c815c530 con 0x7f52c80c6f10 2024-04-26T23:01:24.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.105+0000 7f52cca00640 1 -- 172.21.15.40:0/4202035033 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f52c80b3e30 con 0x7f52c80c6f10 2024-04-26T23:01:24.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.106+0000 7f52bf400640 1 -- 172.21.15.40:0/4202035033 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f52b0059510 con 0x7f52c80c6f10 2024-04-26T23:01:24.106 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:01:24.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.109+0000 7f52cca00640 1 -- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f52a406c8f0 msgr2=0x7f52a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:24.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.109+0000 7f52cca00640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f52a406c8f0 0x7f52a406edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f52c815d230 tx=0x7f52b800b040 comp rx=0 tx=0).stop 2024-04-26T23:01:24.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.109+0000 7f52cca00640 1 -- 172.21.15.40:0/4202035033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52c80c6f10 msgr2=0x7f52c80b6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:24.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.109+0000 7f52cca00640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52c80c6f10 0x7f52c80b6010 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f52b000c400 tx=0x7f52b000c8c0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.109+0000 7f52cca00640 1 -- 172.21.15.40:0/4202035033 shutdown_connections 2024-04-26T23:01:24.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.109+0000 7f52cca00640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f52a406c8f0 0x7f52a406edb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.109+0000 7f52cca00640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52c80b40f0 0x7f52c8148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.109+0000 7f52cca00640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52c80c6f10 0x7f52c80b6010 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.109+0000 7f52cca00640 1 --2- 172.21.15.40:0/4202035033 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52c80c6010 0x7f52c815c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.109+0000 7f52cca00640 1 -- 172.21.15.40:0/4202035033 >> 172.21.15.40:0/4202035033 conn(0x7f52c80d0a10 msgr2=0x7f52c80aeda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:24.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.109+0000 7f52cca00640 1 -- 172.21.15.40:0/4202035033 shutdown_connections 2024-04-26T23:01:24.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.109+0000 7f52cca00640 1 -- 172.21.15.40:0/4202035033 wait complete. 2024-04-26T23:01:24.129 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:01:24.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.329+0000 7ff8e2e00640 1 Processor -- start 2024-04-26T23:01:24.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.329+0000 7ff8e2e00640 1 -- start start 2024-04-26T23:01:24.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.330+0000 7ff8e2e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff8dc15bc70 0x7ff8dc0cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.330+0000 7ff8e2e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8dc0730a0 0x7ff8dc0b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.330+0000 7ff8e2e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8dc06dde0 con 0x7ff8dc071ef0 2024-04-26T23:01:24.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.331+0000 7ff8e2e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff8dc06e3a0 con 0x7ff8dc0730a0 2024-04-26T23:01:24.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.331+0000 7ff8e2e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff8dc0b4840 con 0x7ff8dc15bc70 2024-04-26T23:01:24.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.331+0000 7ff8e2400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8dc0730a0 0x7ff8dc0b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.331+0000 7ff8e1000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff8dc071ef0 0x7ff8dc072990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53994/0 (socket says 172.21.15.40:53994) 2024-04-26T23:01:24.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.331+0000 7ff8e1000640 1 -- 172.21.15.40:0/1558844392 learned_addr learned my addr 172.21.15.40:0/1558844392 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:24.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.331+0000 7ff8e1a00640 1 --2- 172.21.15.40:0/1558844392 >> v2:172.21.15.103:3300/0 conn(0x7ff8dc15bc70 0x7ff8dc0cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.331+0000 7ff8e2400640 1 -- 172.21.15.40:0/1558844392 >> v2:172.21.15.103:3300/0 conn(0x7ff8dc15bc70 msgr2=0x7ff8dc0cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:24.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.331+0000 7ff8e2400640 1 --2- 172.21.15.40:0/1558844392 >> v2:172.21.15.103:3300/0 conn(0x7ff8dc15bc70 0x7ff8dc0cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.331+0000 7ff8e2400640 1 -- 172.21.15.40:0/1558844392 >> v1:172.21.15.40:6789/0 conn(0x7ff8dc071ef0 legacy=0x7ff8dc072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:24.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.332+0000 7ff8e2400640 1 -- 172.21.15.40:0/1558844392 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc06e0c0 con 0x7ff8dc0730a0 2024-04-26T23:01:24.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.333+0000 7ff8e2400640 1 --2- 172.21.15.40:0/1558844392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8dc0730a0 0x7ff8dc0b4cd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0b0640 tx=0x7ff8d402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8da1c5fab9d6a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:24.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.333+0000 7ff8dbe00640 1 -- 172.21.15.40:0/1558844392 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4007740 con 0x7ff8dc0730a0 2024-04-26T23:01:24.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.333+0000 7ff8dbe00640 1 -- 172.21.15.40:0/1558844392 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d40078e0 con 0x7ff8dc0730a0 2024-04-26T23:01:24.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.333+0000 7ff8dbe00640 1 -- 172.21.15.40:0/1558844392 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4007cb0 con 0x7ff8dc0730a0 2024-04-26T23:01:24.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.334+0000 7ff8e2e00640 1 -- 172.21.15.40:0/1558844392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8dc0730a0 msgr2=0x7ff8dc0b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:24.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.334+0000 7ff8e2e00640 1 --2- 172.21.15.40:0/1558844392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8dc0730a0 0x7ff8dc0b4cd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0b0640 tx=0x7ff8d402a680 comp rx=0 tx=0).stop 2024-04-26T23:01:24.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.334+0000 7ff8e2e00640 1 -- 172.21.15.40:0/1558844392 shutdown_connections 2024-04-26T23:01:24.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.334+0000 7ff8e2e00640 1 --2- 172.21.15.40:0/1558844392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8dc0730a0 0x7ff8dc0b4cd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.334+0000 7ff8e2e00640 1 --2- 172.21.15.40:0/1558844392 >> v2:172.21.15.103:3300/0 conn(0x7ff8dc15bc70 0x7ff8dc0cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.334+0000 7ff8e2e00640 1 -- 172.21.15.40:0/1558844392 >> 172.21.15.40:0/1558844392 conn(0x7ff8dc0b2720 msgr2=0x7ff8dc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:24.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.334+0000 7ff8e2e00640 1 -- 172.21.15.40:0/1558844392 shutdown_connections 2024-04-26T23:01:24.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.334+0000 7ff8e2e00640 1 -- 172.21.15.40:0/1558844392 wait complete. 2024-04-26T23:01:24.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.336+0000 7ff8e2e00640 1 Processor -- start 2024-04-26T23:01:24.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.336+0000 7ff8e2e00640 1 -- start start 2024-04-26T23:01:24.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.337+0000 7ff8e2e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8dc071ef0 0x7ff8dc14e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.337+0000 7ff8e2e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8dc0730a0 0x7ff8dc0b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.337+0000 7ff8e2e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8dc15bc70 0x7ff8dc0b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.337+0000 7ff8e2e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc15c050 con 0x7ff8dc15bc70 2024-04-26T23:01:24.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.337+0000 7ff8e2e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc06fad0 con 0x7ff8dc071ef0 2024-04-26T23:01:24.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.337+0000 7ff8e2e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff8dc15c1d0 con 0x7ff8dc0730a0 2024-04-26T23:01:24.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.337+0000 7ff8e2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8dc15bc70 0x7ff8dc0b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.337+0000 7ff8e2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8dc15bc70 0x7ff8dc0b5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35234/0 (socket says 172.21.15.40:35234) 2024-04-26T23:01:24.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.337+0000 7ff8e2400640 1 -- 172.21.15.40:0/3297351081 learned_addr learned my addr 172.21.15.40:0/3297351081 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:24.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.337+0000 7ff8e1a00640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8dc071ef0 0x7ff8dc14e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.337+0000 7ff8e1000640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8dc0730a0 0x7ff8dc0b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.338+0000 7ff8e2400640 1 -- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8dc0730a0 msgr2=0x7ff8dc0b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.338+0000 7ff8e2400640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8dc0730a0 0x7ff8dc0b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.338+0000 7ff8e2400640 1 -- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8dc071ef0 msgr2=0x7ff8dc14e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.338+0000 7ff8e2400640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8dc071ef0 0x7ff8dc14e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.338+0000 7ff8e2400640 1 -- 172.21.15.40:0/3297351081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc0aa560 con 0x7ff8dc15bc70 2024-04-26T23:01:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.338+0000 7ff8e1000640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8dc0730a0 0x7ff8dc0b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.338+0000 7ff8e2400640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8dc15bc70 0x7ff8dc0b5110 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009e50 tx=0x7ff8d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.338+0000 7ff8d9600640 1 -- 172.21.15.40:0/3297351081 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4040070 con 0x7ff8dc15bc70 2024-04-26T23:01:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.338+0000 7ff8d9600640 1 -- 172.21.15.40:0/3297351081 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d4031bf0 con 0x7ff8dc15bc70 2024-04-26T23:01:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.338+0000 7ff8d9600640 1 -- 172.21.15.40:0/3297351081 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4045460 con 0x7ff8dc15bc70 2024-04-26T23:01:24.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.338+0000 7ff8e2e00640 1 -- 172.21.15.40:0/3297351081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8dc14d250 con 0x7ff8dc15bc70 2024-04-26T23:01:24.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.339+0000 7ff8e2e00640 1 -- 172.21.15.40:0/3297351081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8dc14d810 con 0x7ff8dc15bc70 2024-04-26T23:01:24.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.340+0000 7ff8e2e00640 1 -- 172.21.15.40:0/3297351081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8dc069370 con 0x7ff8dc15bc70 2024-04-26T23:01:24.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.341+0000 7ff8d9600640 1 -- 172.21.15.40:0/3297351081 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff8d4039040 con 0x7ff8dc15bc70 2024-04-26T23:01:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.342+0000 7ff8d9600640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8bc06c8f0 0x7ff8bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.342+0000 7ff8d9600640 1 -- 172.21.15.40:0/3297351081 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff8d40ad570 con 0x7ff8dc15bc70 2024-04-26T23:01:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.343+0000 7ff8e1a00640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8bc06c8f0 0x7ff8bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.344+0000 7ff8e1a00640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8bc06c8f0 0x7ff8bc06edb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0006fd0 tx=0x7ff8d0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:24.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.347+0000 7ff8d9600640 1 -- 172.21.15.40:0/3297351081 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff8d404c030 con 0x7ff8dc15bc70 2024-04-26T23:01:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.557+0000 7ff8e2e00640 1 -- 172.21.15.40:0/3297351081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff8dc151e10 con 0x7ff8dc15bc70 2024-04-26T23:01:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.558+0000 7ff8d9600640 1 -- 172.21.15.40:0/3297351081 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7ff8dc0aa560 con 0x7ff8dc15bc70 2024-04-26T23:01:24.558 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:01:24.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.560+0000 7ff8e2e00640 1 -- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8bc06c8f0 msgr2=0x7ff8bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:24.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.560+0000 7ff8e2e00640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8bc06c8f0 0x7ff8bc06edb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0006fd0 tx=0x7ff8d0008040 comp rx=0 tx=0).stop 2024-04-26T23:01:24.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.560+0000 7ff8e2e00640 1 -- 172.21.15.40:0/3297351081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8dc15bc70 msgr2=0x7ff8dc0b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:24.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.560+0000 7ff8e2e00640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8dc15bc70 0x7ff8dc0b5110 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009e50 tx=0x7ff8d4004a40 comp rx=0 tx=0).stop 2024-04-26T23:01:24.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.561+0000 7ff8e2e00640 1 -- 172.21.15.40:0/3297351081 shutdown_connections 2024-04-26T23:01:24.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.561+0000 7ff8e2e00640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8bc06c8f0 0x7ff8bc06edb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.561+0000 7ff8e2e00640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8dc15bc70 0x7ff8dc0b5110 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.561+0000 7ff8e2e00640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8dc0730a0 0x7ff8dc0b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.561+0000 7ff8e2e00640 1 --2- 172.21.15.40:0/3297351081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8dc071ef0 0x7ff8dc14e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.561+0000 7ff8e2e00640 1 -- 172.21.15.40:0/3297351081 >> 172.21.15.40:0/3297351081 conn(0x7ff8dc0b2720 msgr2=0x7ff8dc0d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:24.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.561+0000 7ff8e2e00640 1 -- 172.21.15.40:0/3297351081 shutdown_connections 2024-04-26T23:01:24.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.561+0000 7ff8e2e00640 1 -- 172.21.15.40:0/3297351081 wait complete. 2024-04-26T23:01:24.581 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:01:24.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.780+0000 7f3c83e00640 1 Processor -- start 2024-04-26T23:01:24.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.780+0000 7f3c83e00640 1 -- start start 2024-04-26T23:01:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.782+0000 7f3c83e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3c840ccca0 0x7f3c840afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.782+0000 7f3c83e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c840af7b0 con 0x7f3c840ce5e0 2024-04-26T23:01:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.782+0000 7f3c83e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c840afa90 con 0x7f3c8415f2d0 2024-04-26T23:01:24.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.782+0000 7f3c83e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3c840cbd90 con 0x7f3c840ccca0 2024-04-26T23:01:24.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.782+0000 7f3c82000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3c840ce5e0 0x7f3c840cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53996/0 (socket says 172.21.15.40:53996) 2024-04-26T23:01:24.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.782+0000 7f3c82000640 1 -- 172.21.15.40:0/778321323 learned_addr learned my addr 172.21.15.40:0/778321323 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:24.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c83400640 1 --2- 172.21.15.40:0/778321323 >> v2:172.21.15.103:3300/0 conn(0x7f3c840ccca0 0x7f3c840afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3289863528 0 0) 0x7f3c840af7b0 con 0x7f3c840ce5e0 2024-04-26T23:01:24.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c68005730 con 0x7f3c840ce5e0 2024-04-26T23:01:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1392117698 0 0) 0x7f3c68005730 con 0x7f3c840ce5e0 2024-04-26T23:01:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 >> v2:172.21.15.103:3300/0 conn(0x7f3c840ccca0 msgr2=0x7f3c840afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 --2- 172.21.15.40:0/778321323 >> v2:172.21.15.103:3300/0 conn(0x7f3c840ccca0 0x7f3c840afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 >> v1:172.21.15.103:6789/0 conn(0x7f3c8415f2d0 legacy=0x7f3c840c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c840cc070 con 0x7f3c840ce5e0 2024-04-26T23:01:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f3c70001670 con 0x7f3c840ce5e0 2024-04-26T23:01:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 >> v1:172.21.15.40:6789/0 conn(0x7f3c840ce5e0 legacy=0x7f3c840cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 --2- 172.21.15.40:0/778321323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c68005bf0 0x7f3c68006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 --2- 172.21.15.40:0/778321323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c68006a10 0x7f3c68006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 --2- 172.21.15.40:0/778321323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c680073d0 0x7f3c6800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c840cbd90 con 0x7f3c68005bf0 2024-04-26T23:01:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6800e2a0 con 0x7f3c680073d0 2024-04-26T23:01:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.783+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c68003520 con 0x7f3c68006a10 2024-04-26T23:01:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.784+0000 7f3c82a00640 1 --2- 172.21.15.40:0/778321323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c68005bf0 0x7f3c68006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.784+0000 7f3c82000640 1 --2- 172.21.15.40:0/778321323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c68006a10 0x7f3c68006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.784+0000 7f3c83400640 1 --2- 172.21.15.40:0/778321323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c680073d0 0x7f3c6800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.784+0000 7f3c82a00640 1 -- 172.21.15.40:0/778321323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c68006a10 msgr2=0x7f3c68006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.784+0000 7f3c82a00640 1 --2- 172.21.15.40:0/778321323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c68006a10 0x7f3c68006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.784+0000 7f3c82a00640 1 -- 172.21.15.40:0/778321323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c680073d0 msgr2=0x7f3c6800dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.784+0000 7f3c82a00640 1 --2- 172.21.15.40:0/778321323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c680073d0 0x7f3c6800dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.784+0000 7f3c83400640 1 -- 172.21.15.40:0/778321323 reap_dead start 2024-04-26T23:01:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.784+0000 7f3c82a00640 1 -- 172.21.15.40:0/778321323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c68005730 con 0x7f3c68005bf0 2024-04-26T23:01:24.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.785+0000 7f3c82a00640 1 --2- 172.21.15.40:0/778321323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c68005bf0 0x7f3c68006010 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3c74001750 tx=0x7f3c68007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01b4de440c2e153 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:24.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.786+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74001370 con 0x7f3c68005bf0 2024-04-26T23:01:24.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.786+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c740052d0 con 0x7f3c68005bf0 2024-04-26T23:01:24.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.786+0000 7f3c81600640 1 -- 172.21.15.40:0/778321323 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7402abc0 con 0x7f3c68005bf0 2024-04-26T23:01:24.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.786+0000 7f3c83e00640 1 -- 172.21.15.40:0/778321323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c68005bf0 msgr2=0x7f3c68006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:24.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.786+0000 7f3c83e00640 1 --2- 172.21.15.40:0/778321323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c68005bf0 0x7f3c68006010 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3c74001750 tx=0x7f3c68007a40 comp rx=0 tx=0).stop 2024-04-26T23:01:24.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.786+0000 7f3c83e00640 1 -- 172.21.15.40:0/778321323 shutdown_connections 2024-04-26T23:01:24.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.786+0000 7f3c83e00640 1 --2- 172.21.15.40:0/778321323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c68005bf0 0x7f3c68006010 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.786+0000 7f3c83e00640 1 -- 172.21.15.40:0/778321323 >> 172.21.15.40:0/778321323 conn(0x7f3c840d0a10 msgr2=0x7f3c840d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:24.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.786+0000 7f3c83e00640 1 -- 172.21.15.40:0/778321323 shutdown_connections 2024-04-26T23:01:24.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.786+0000 7f3c83e00640 1 -- 172.21.15.40:0/778321323 wait complete. 2024-04-26T23:01:24.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.788+0000 7f3c83e00640 1 Processor -- start 2024-04-26T23:01:24.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.788+0000 7f3c83e00640 1 -- start start 2024-04-26T23:01:24.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.789+0000 7f3c83e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c68005bf0 0x7f3c8415b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.789+0000 7f3c83e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c84147300 0x7f3c840b4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.789+0000 7f3c83e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c84071c00 0x7f3c84072080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.789+0000 7f3c83e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c840afdf0 con 0x7f3c84147300 2024-04-26T23:01:24.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.789+0000 7f3c83e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8406e270 con 0x7f3c84071c00 2024-04-26T23:01:24.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.789+0000 7f3c83e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c840aff70 con 0x7f3c68005bf0 2024-04-26T23:01:24.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.790+0000 7f3c82000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c84147300 0x7f3c840b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.790+0000 7f3c82000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c84147300 0x7f3c840b4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35254/0 (socket says 172.21.15.40:35254) 2024-04-26T23:01:24.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.790+0000 7f3c82000640 1 -- 172.21.15.40:0/1080286488 learned_addr learned my addr 172.21.15.40:0/1080286488 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:24.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.790+0000 7f3c82a00640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c68005bf0 0x7f3c8415b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.790+0000 7f3c83400640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c84071c00 0x7f3c84072080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.790+0000 7f3c82000640 1 -- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c68005bf0 msgr2=0x7f3c8415b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:24.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.790+0000 7f3c82000640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c68005bf0 0x7f3c8415b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.790+0000 7f3c82000640 1 -- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c84071c00 msgr2=0x7f3c84072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:24.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.790+0000 7f3c82000640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c84071c00 0x7f3c84072080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:24.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.790+0000 7f3c82000640 1 -- 172.21.15.40:0/1080286488 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c840c8c90 con 0x7f3c84147300 2024-04-26T23:01:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.791+0000 7f3c82000640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c84147300 0x7f3c840b4120 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000e700 tx=0x7f3c7000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.791+0000 7f3c7aa00640 1 -- 172.21.15.40:0/1080286488 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7000ee10 con 0x7f3c84147300 2024-04-26T23:01:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.791+0000 7f3c7aa00640 1 -- 172.21.15.40:0/1080286488 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c70018070 con 0x7f3c84147300 2024-04-26T23:01:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.791+0000 7f3c7aa00640 1 -- 172.21.15.40:0/1080286488 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7000ca90 con 0x7f3c84147300 2024-04-26T23:01:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.791+0000 7f3c83e00640 1 -- 172.21.15.40:0/1080286488 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c840c8f70 con 0x7f3c84147300 2024-04-26T23:01:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.791+0000 7f3c83e00640 1 -- 172.21.15.40:0/1080286488 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c840c9530 con 0x7f3c84147300 2024-04-26T23:01:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.792+0000 7f3c83e00640 1 -- 172.21.15.40:0/1080286488 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c84069370 con 0x7f3c84147300 2024-04-26T23:01:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.793+0000 7f3c7aa00640 1 -- 172.21.15.40:0/1080286488 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3c7000cc30 con 0x7f3c84147300 2024-04-26T23:01:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.794+0000 7f3c7aa00640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c6406c940 0x7f3c6406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.794+0000 7f3c7aa00640 1 -- 172.21.15.40:0/1080286488 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3c7008bc00 con 0x7f3c84147300 2024-04-26T23:01:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.795+0000 7f3c82a00640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c6406c940 0x7f3c6406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.796+0000 7f3c82a00640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c6406c940 0x7f3c6406ee00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3c74032000 tx=0x7f3c7402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:24.799+0000 7f3c7aa00640 1 -- 172.21.15.40:0/1080286488 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3c70059d60 con 0x7f3c84147300 2024-04-26T23:01:25.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.021+0000 7f3c83e00640 1 -- 172.21.15.40:0/1080286488 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3c8406de10 con 0x7f3c84147300 2024-04-26T23:01:25.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.022+0000 7f3c7aa00640 1 -- 172.21.15.40:0/1080286488 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7f3c70059660 con 0x7f3c84147300 2024-04-26T23:01:25.022 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-04-26T23:01:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.025+0000 7f3c83e00640 1 -- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c6406c940 msgr2=0x7f3c6406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.025+0000 7f3c83e00640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c6406c940 0x7f3c6406ee00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3c74032000 tx=0x7f3c7402e000 comp rx=0 tx=0).stop 2024-04-26T23:01:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.025+0000 7f3c83e00640 1 -- 172.21.15.40:0/1080286488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c84147300 msgr2=0x7f3c840b4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.025+0000 7f3c83e00640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c84147300 0x7f3c840b4120 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000e700 tx=0x7f3c7000ebc0 comp rx=0 tx=0).stop 2024-04-26T23:01:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.025+0000 7f3c83e00640 1 -- 172.21.15.40:0/1080286488 shutdown_connections 2024-04-26T23:01:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.025+0000 7f3c83e00640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c6406c940 0x7f3c6406ee00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.025+0000 7f3c83e00640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c84071c00 0x7f3c84072080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:25.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.025+0000 7f3c83e00640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c84147300 0x7f3c840b4120 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:25.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.025+0000 7f3c83e00640 1 --2- 172.21.15.40:0/1080286488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c68005bf0 0x7f3c8415b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:25.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.025+0000 7f3c83e00640 1 -- 172.21.15.40:0/1080286488 >> 172.21.15.40:0/1080286488 conn(0x7f3c840d0a10 msgr2=0x7f3c840cd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:25.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.025+0000 7f3c83e00640 1 -- 172.21.15.40:0/1080286488 shutdown_connections 2024-04-26T23:01:25.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:25.025+0000 7f3c83e00640 1 -- 172.21.15.40:0/1080286488 wait complete. 2024-04-26T23:01:25.045 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-04-26T23:01:25.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:25.045 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:01:25.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:ceph-fuse[39680]: fuse finished with error 0 and tester_r 0 2024-04-26T23:01:25.138 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:01:31.239 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:01:31.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:31.240 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-26T23:01:31.324 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:31.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.542+0000 7fcdeb200640 1 Processor -- start 2024-04-26T23:01:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.542+0000 7fcdeb200640 1 -- start start 2024-04-26T23:01:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.543+0000 7fcdeb200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcdec0b6220 0x7fcdec0d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.543+0000 7fcdeb200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcdec0d08d0 0x7fcdec0d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.543+0000 7fcdeb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdec0d3820 0x7fcdec0b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.543+0000 7fcdeb200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fcdec162c40 con 0x7fcdec0d08d0 2024-04-26T23:01:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.544+0000 7fcdeb200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcdec0d2d80 con 0x7fcdec0b6220 2024-04-26T23:01:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.544+0000 7fcdeb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcdec0b7c80 con 0x7fcdec0d3820 2024-04-26T23:01:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.544+0000 7fcde9400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcdec0d08d0 0x7fcdec0d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.544+0000 7fcde9400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcdec0d08d0 0x7fcdec0d32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42964/0 (socket says 172.21.15.40:42964) 2024-04-26T23:01:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.544+0000 7fcde9400640 1 -- 172.21.15.40:0/1750102801 learned_addr learned my addr 172.21.15.40:0/1750102801 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.544+0000 7fcdea800640 1 --2- 172.21.15.40:0/1750102801 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdec0d3820 0x7fcdec0b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.544+0000 7fcde9e00640 1 --2- 172.21.15.40:0/1750102801 >> v2:172.21.15.103:3300/0 conn(0x7fcdec0b6220 0x7fcdec0d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.544+0000 7fcde9400640 1 -- 172.21.15.40:0/1750102801 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdec0d3820 msgr2=0x7fcdec0b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.544+0000 7fcde9400640 1 --2- 172.21.15.40:0/1750102801 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdec0d3820 0x7fcdec0b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.544+0000 7fcde9400640 1 -- 172.21.15.40:0/1750102801 >> v2:172.21.15.103:3300/0 conn(0x7fcdec0b6220 msgr2=0x7fcdec0d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.544+0000 7fcde9400640 1 --2- 172.21.15.40:0/1750102801 >> v2:172.21.15.103:3300/0 conn(0x7fcdec0b6220 0x7fcdec0d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.544+0000 7fcde9400640 1 -- 172.21.15.40:0/1750102801 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdec0b7f60 con 0x7fcdec0d08d0 2024-04-26T23:01:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.546+0000 7fcde9400640 1 --2- 172.21.15.40:0/1750102801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdec0d08d0 0x7fcdec0d32e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0d4cf0 tx=0x7fcdd802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2079148b59d00f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.546+0000 7fcde8a00640 1 -- 172.21.15.40:0/1750102801 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8037070 con 0x7fcdec0d08d0 2024-04-26T23:01:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.546+0000 7fcde8a00640 1 -- 172.21.15.40:0/1750102801 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdd8032440 con 0x7fcdec0d08d0 2024-04-26T23:01:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.546+0000 7fcde8a00640 1 -- 172.21.15.40:0/1750102801 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8032810 con 0x7fcdec0d08d0 2024-04-26T23:01:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.547+0000 7fcdeb200640 1 -- 172.21.15.40:0/1750102801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdec0d08d0 msgr2=0x7fcdec0d32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.547+0000 7fcdeb200640 1 --2- 172.21.15.40:0/1750102801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdec0d08d0 0x7fcdec0d32e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0d4cf0 tx=0x7fcdd802a1e0 comp rx=0 tx=0).stop 2024-04-26T23:01:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.547+0000 7fcdeb200640 1 -- 172.21.15.40:0/1750102801 shutdown_connections 2024-04-26T23:01:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.547+0000 7fcdeb200640 1 --2- 172.21.15.40:0/1750102801 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdec0d3820 0x7fcdec0b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.547+0000 7fcdeb200640 1 --2- 172.21.15.40:0/1750102801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdec0d08d0 0x7fcdec0d32e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.547+0000 7fcdeb200640 1 --2- 172.21.15.40:0/1750102801 >> v2:172.21.15.103:3300/0 conn(0x7fcdec0b6220 0x7fcdec0d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.547+0000 7fcdeb200640 1 -- 172.21.15.40:0/1750102801 >> 172.21.15.40:0/1750102801 conn(0x7fcdec0d4d90 msgr2=0x7fcdec0d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.547+0000 7fcdeb200640 1 -- 172.21.15.40:0/1750102801 shutdown_connections 2024-04-26T23:01:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.547+0000 7fcdeb200640 1 -- 172.21.15.40:0/1750102801 wait complete. 2024-04-26T23:01:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.549+0000 7fcdeb200640 1 Processor -- start 2024-04-26T23:01:31.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.550+0000 7fcdeb200640 1 -- start start 2024-04-26T23:01:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.551+0000 7fcdeb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdec0b6220 0x7fcdec0b5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.551+0000 7fcdeb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdec0d08d0 0x7fcdec0b4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.551+0000 7fcdeb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdec0d3820 0x7fcdec0a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.551+0000 7fcdeb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec0b6d50 con 0x7fcdec0d3820 2024-04-26T23:01:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.551+0000 7fcdeb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec0b8520 con 0x7fcdec0d08d0 2024-04-26T23:01:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.551+0000 7fcdeb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcdec0b6ed0 con 0x7fcdec0b6220 2024-04-26T23:01:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.552+0000 7fcdea800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdec0d3820 0x7fcdec0a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.552+0000 7fcdea800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdec0d3820 0x7fcdec0a1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42978/0 (socket says 172.21.15.40:42978) 2024-04-26T23:01:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.552+0000 7fcdea800640 1 -- 172.21.15.40:0/623636103 learned_addr learned my addr 172.21.15.40:0/623636103 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.552+0000 7fcde9e00640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdec0b6220 0x7fcdec0b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.552+0000 7fcde9400640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdec0d08d0 0x7fcdec0b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.552+0000 7fcdea800640 1 -- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdec0b6220 msgr2=0x7fcdec0b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.552+0000 7fcdea800640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdec0b6220 0x7fcdec0b5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.552+0000 7fcdea800640 1 -- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdec0d08d0 msgr2=0x7fcdec0b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.552+0000 7fcdea800640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdec0d08d0 0x7fcdec0b4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.552+0000 7fcdea800640 1 -- 172.21.15.40:0/623636103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdec1f8d20 con 0x7fcdec0d3820 2024-04-26T23:01:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.552+0000 7fcdea800640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdec0d3820 0x7fcdec0a1b30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcde400d5d0 tx=0x7fcde400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.553+0000 7fcde2000640 1 -- 172.21.15.40:0/623636103 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde40125f0 con 0x7fcdec0d3820 2024-04-26T23:01:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.553+0000 7fcde2000640 1 -- 172.21.15.40:0/623636103 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcde4019070 con 0x7fcdec0d3820 2024-04-26T23:01:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.553+0000 7fcde2000640 1 -- 172.21.15.40:0/623636103 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde40129f0 con 0x7fcdec0d3820 2024-04-26T23:01:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.553+0000 7fcdeb200640 1 -- 172.21.15.40:0/623636103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdec0a2360 con 0x7fcdec0d3820 2024-04-26T23:01:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.553+0000 7fcdeb200640 1 -- 172.21.15.40:0/623636103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdec0a2920 con 0x7fcdec0d3820 2024-04-26T23:01:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.554+0000 7fcdeb200640 1 -- 172.21.15.40:0/623636103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdec069370 con 0x7fcdec0d3820 2024-04-26T23:01:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.555+0000 7fcde2000640 1 -- 172.21.15.40:0/623636103 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcde4012b90 con 0x7fcdec0d3820 2024-04-26T23:01:31.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.556+0000 7fcde2000640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdcc06c8f0 0x7fcdcc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:31.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.556+0000 7fcde2000640 1 -- 172.21.15.40:0/623636103 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fcde408c2f0 con 0x7fcdec0d3820 2024-04-26T23:01:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.557+0000 7fcde9e00640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdcc06c8f0 0x7fcdcc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.558+0000 7fcde9e00640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdcc06c8f0 0x7fcdcc06edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0b38b0 tx=0x7fcddc003a10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.563+0000 7fcde2000640 1 -- 172.21.15.40:0/623636103 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fcdec1f8d20 con 0x7fcdec0d3820 2024-04-26T23:01:31.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.768+0000 7fcdeb200640 1 -- 172.21.15.40:0/623636103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcdec0d30c0 con 0x7fcdec0d3820 2024-04-26T23:01:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.771+0000 7fcde2000640 1 -- 172.21.15.40:0/623636103 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7fcde405a350 con 0x7fcdec0d3820 2024-04-26T23:01:31.772 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:31.772 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:11.763965+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:31.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.773+0000 7fcdeb200640 1 -- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdcc06c8f0 msgr2=0x7fcdcc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:31.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.774+0000 7fcdeb200640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdcc06c8f0 0x7fcdcc06edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0b38b0 tx=0x7fcddc003a10 comp rx=0 tx=0).stop 2024-04-26T23:01:31.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.774+0000 7fcdeb200640 1 -- 172.21.15.40:0/623636103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdec0d3820 msgr2=0x7fcdec0a1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:31.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.774+0000 7fcdeb200640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdec0d3820 0x7fcdec0a1b30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcde400d5d0 tx=0x7fcde400da90 comp rx=0 tx=0).stop 2024-04-26T23:01:31.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.774+0000 7fcdeb200640 1 -- 172.21.15.40:0/623636103 shutdown_connections 2024-04-26T23:01:31.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.774+0000 7fcdeb200640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdcc06c8f0 0x7fcdcc06edb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:31.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.774+0000 7fcdeb200640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdec0d3820 0x7fcdec0a1b30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:31.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.774+0000 7fcdeb200640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdec0d08d0 0x7fcdec0b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:31.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.774+0000 7fcdeb200640 1 --2- 172.21.15.40:0/623636103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdec0b6220 0x7fcdec0b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:31.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.774+0000 7fcdeb200640 1 -- 172.21.15.40:0/623636103 >> 172.21.15.40:0/623636103 conn(0x7fcdec0d4d90 msgr2=0x7fcdec06e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:31.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.774+0000 7fcdeb200640 1 -- 172.21.15.40:0/623636103 shutdown_connections 2024-04-26T23:01:31.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:31.774+0000 7fcdeb200640 1 -- 172.21.15.40:0/623636103 wait complete. 2024-04-26T23:01:31.795 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.008+0000 7fd319000640 1 Processor -- start 2024-04-26T23:01:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.008+0000 7fd319000640 1 -- start start 2024-04-26T23:01:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.010+0000 7fd319000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd3140ce5e0 0x7fd3140afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.010+0000 7fd319000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3140cc590 0x7fd3140cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.010+0000 7fd319000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3140af7b0 con 0x7fd31415f2d0 2024-04-26T23:01:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.010+0000 7fd319000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd3140afa90 con 0x7fd3140ce5e0 2024-04-26T23:01:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.010+0000 7fd319000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd3140cbd90 con 0x7fd3140cc590 2024-04-26T23:01:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.010+0000 7fd313400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd31415f2d0 0x7fd3140c58b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:47492/0 (socket says 172.21.15.40:47492) 2024-04-26T23:01:32.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.010+0000 7fd313400640 1 -- 172.21.15.40:0/1144160929 learned_addr learned my addr 172.21.15.40:0/1144160929 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:32.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.010+0000 7fd313e00640 1 --2- 172.21.15.40:0/1144160929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3140cc590 0x7fd3140cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.011+0000 7fd30be00640 1 --2- 172.21.15.40:0/1144160929 >> v2:172.21.15.103:3300/0 conn(0x7fd3140ce5e0 0x7fd3140afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.011+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 80586530 0 0) 0x7fd3140af7b0 con 0x7fd31415f2d0 2024-04-26T23:01:32.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.011+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2f8003520 con 0x7fd31415f2d0 2024-04-26T23:01:32.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.011+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1960561476 0 0) 0x7fd2f8003520 con 0x7fd31415f2d0 2024-04-26T23:01:32.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.011+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3140cc590 msgr2=0x7fd3140cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:32.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.011+0000 7fd312a00640 1 --2- 172.21.15.40:0/1144160929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3140cc590 0x7fd3140cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.011+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 >> v2:172.21.15.103:3300/0 conn(0x7fd3140ce5e0 msgr2=0x7fd3140afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:32.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.011+0000 7fd312a00640 1 --2- 172.21.15.40:0/1144160929 >> v2:172.21.15.103:3300/0 conn(0x7fd3140ce5e0 0x7fd3140afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.011+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3140cc070 con 0x7fd31415f2d0 2024-04-26T23:01:32.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.011+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fd304001670 con 0x7fd31415f2d0 2024-04-26T23:01:32.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.011+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 >> v1:172.21.15.40:6789/0 conn(0x7fd31415f2d0 legacy=0x7fd3140c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:32.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.011+0000 7fd312a00640 1 --2- 172.21.15.40:0/1144160929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3140cc590 0x7fd2f80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd312a00640 1 --2- 172.21.15.40:0/1144160929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2f8003f30 0x7fd2f8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd312a00640 1 --2- 172.21.15.40:0/1144160929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f8004d40 0x7fd2f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd3140afa90 con 0x7fd2f8003f30 2024-04-26T23:01:32.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd3140cbd90 con 0x7fd2f8004d40 2024-04-26T23:01:32.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd2f800dbd0 con 0x7fd3140cc590 2024-04-26T23:01:32.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd30be00640 1 --2- 172.21.15.40:0/1144160929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2f8003f30 0x7fd2f8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd313400640 1 --2- 172.21.15.40:0/1144160929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3140cc590 0x7fd2f80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd313e00640 1 --2- 172.21.15.40:0/1144160929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f8004d40 0x7fd2f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd30be00640 1 -- 172.21.15.40:0/1144160929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3140cc590 msgr2=0x7fd2f80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:32.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd30be00640 1 --2- 172.21.15.40:0/1144160929 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3140cc590 0x7fd2f80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd30be00640 1 -- 172.21.15.40:0/1144160929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f8004d40 msgr2=0x7fd2f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:32.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd30be00640 1 --2- 172.21.15.40:0/1144160929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f8004d40 0x7fd2f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd30be00640 1 -- 172.21.15.40:0/1144160929 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f8003520 con 0x7fd2f8003f30 2024-04-26T23:01:32.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.012+0000 7fd313e00640 1 --2- 172.21.15.40:0/1144160929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f8004d40 0x7fd2f80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:32.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.013+0000 7fd30be00640 1 --2- 172.21.15.40:0/1144160929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2f8003f30 0x7fd2f8004800 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009ac0 tx=0x7fd2f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c7fb45d765d864 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:32.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.014+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc036070 con 0x7fd2f8003f30 2024-04-26T23:01:32.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.014+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2fc031400 con 0x7fd2f8003f30 2024-04-26T23:01:32.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.014+0000 7fd312a00640 1 -- 172.21.15.40:0/1144160929 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc009690 con 0x7fd2f8003f30 2024-04-26T23:01:32.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.014+0000 7fd319000640 1 -- 172.21.15.40:0/1144160929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2f8003f30 msgr2=0x7fd2f8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:32.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.014+0000 7fd319000640 1 --2- 172.21.15.40:0/1144160929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2f8003f30 0x7fd2f8004800 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009ac0 tx=0x7fd2f8005830 comp rx=0 tx=0).stop 2024-04-26T23:01:32.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.014+0000 7fd313e00640 1 -- 172.21.15.40:0/1144160929 reap_dead start 2024-04-26T23:01:32.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.014+0000 7fd319000640 1 -- 172.21.15.40:0/1144160929 shutdown_connections 2024-04-26T23:01:32.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.014+0000 7fd319000640 1 -- 172.21.15.40:0/1144160929 >> 172.21.15.40:0/1144160929 conn(0x7fd3140d0a10 msgr2=0x7fd3140d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:32.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.014+0000 7fd319000640 1 -- 172.21.15.40:0/1144160929 shutdown_connections 2024-04-26T23:01:32.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.014+0000 7fd319000640 1 -- 172.21.15.40:0/1144160929 wait complete. 2024-04-26T23:01:32.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.016+0000 7fd319000640 1 Processor -- start 2024-04-26T23:01:32.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.016+0000 7fd319000640 1 -- start start 2024-04-26T23:01:32.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.017+0000 7fd319000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f8003f30 0x7fd3140ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.017+0000 7fd319000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3141555f0 0x7fd3140ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd319000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3140aee00 0x7fd3141f7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd319000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd3140c5fc0 con 0x7fd3141555f0 2024-04-26T23:01:32.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd319000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd31406e270 con 0x7fd2f8003f30 2024-04-26T23:01:32.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd319000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd3141f4d10 con 0x7fd3140aee00 2024-04-26T23:01:32.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd30be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3141555f0 0x7fd3140ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd30be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3141555f0 0x7fd3140ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42988/0 (socket says 172.21.15.40:42988) 2024-04-26T23:01:32.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd30be00640 1 -- 172.21.15.40:0/1335706761 learned_addr learned my addr 172.21.15.40:0/1335706761 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:32.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd313400640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f8003f30 0x7fd3140ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd313e00640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3140aee00 0x7fd3141f7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd30be00640 1 -- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3140aee00 msgr2=0x7fd3141f7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:32.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd30be00640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3140aee00 0x7fd3141f7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd30be00640 1 -- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f8003f30 msgr2=0x7fd3140ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:32.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd30be00640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f8003f30 0x7fd3140ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.018+0000 7fd30be00640 1 -- 172.21.15.40:0/1335706761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3141f4ff0 con 0x7fd3141555f0 2024-04-26T23:01:32.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.019+0000 7fd30be00640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3141555f0 0x7fd3140ae8c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc000c00 tx=0x7fd2fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:32.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.019+0000 7fd30b400640 1 -- 172.21.15.40:0/1335706761 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc040020 con 0x7fd3141555f0 2024-04-26T23:01:32.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.019+0000 7fd30b400640 1 -- 172.21.15.40:0/1335706761 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2fc003c60 con 0x7fd3141555f0 2024-04-26T23:01:32.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.019+0000 7fd319000640 1 -- 172.21.15.40:0/1335706761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3141f52d0 con 0x7fd3141555f0 2024-04-26T23:01:32.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.019+0000 7fd30b400640 1 -- 172.21.15.40:0/1335706761 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc036040 con 0x7fd3141555f0 2024-04-26T23:01:32.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.019+0000 7fd319000640 1 -- 172.21.15.40:0/1335706761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd314155a70 con 0x7fd3141555f0 2024-04-26T23:01:32.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.021+0000 7fd319000640 1 -- 172.21.15.40:0/1335706761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd314069370 con 0x7fd3141555f0 2024-04-26T23:01:32.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.021+0000 7fd30b400640 1 -- 172.21.15.40:0/1335706761 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd2fc043020 con 0x7fd3141555f0 2024-04-26T23:01:32.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.022+0000 7fd30b400640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd2f006c8f0 0x7fd2f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.022+0000 7fd30b400640 1 -- 172.21.15.40:0/1335706761 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd2fc0ac3a0 con 0x7fd3141555f0 2024-04-26T23:01:32.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.023+0000 7fd313400640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd2f006c8f0 0x7fd2f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.024+0000 7fd313400640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd2f006c8f0 0x7fd2f006edb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd3141f9000 tx=0x7fd304009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:32.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.027+0000 7fd30b400640 1 -- 172.21.15.40:0/1335706761 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd2fc04a030 con 0x7fd3141555f0 2024-04-26T23:01:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.239+0000 7fd319000640 1 -- 172.21.15.40:0/1335706761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd3140af1e0 con 0x7fd3141555f0 2024-04-26T23:01:32.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.242+0000 7fd30b400640 1 -- 172.21.15.40:0/1335706761 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7fd2fc0044e0 con 0x7fd3141555f0 2024-04-26T23:01:32.243 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:32.243 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:11.763965+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:32.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.245+0000 7fd319000640 1 -- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd2f006c8f0 msgr2=0x7fd2f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:32.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.245+0000 7fd319000640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd2f006c8f0 0x7fd2f006edb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd3141f9000 tx=0x7fd304009040 comp rx=0 tx=0).stop 2024-04-26T23:01:32.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.245+0000 7fd319000640 1 -- 172.21.15.40:0/1335706761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3141555f0 msgr2=0x7fd3140ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:32.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.245+0000 7fd319000640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3141555f0 0x7fd3140ae8c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc000c00 tx=0x7fd2fc004730 comp rx=0 tx=0).stop 2024-04-26T23:01:32.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.245+0000 7fd319000640 1 -- 172.21.15.40:0/1335706761 shutdown_connections 2024-04-26T23:01:32.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.245+0000 7fd319000640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd2f006c8f0 0x7fd2f006edb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.245+0000 7fd319000640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3140aee00 0x7fd3141f7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.245+0000 7fd319000640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3141555f0 0x7fd3140ae8c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.245+0000 7fd319000640 1 --2- 172.21.15.40:0/1335706761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f8003f30 0x7fd3140ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.245+0000 7fd319000640 1 -- 172.21.15.40:0/1335706761 >> 172.21.15.40:0/1335706761 conn(0x7fd3140d0a10 msgr2=0x7fd3140c6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:32.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.245+0000 7fd319000640 1 -- 172.21.15.40:0/1335706761 shutdown_connections 2024-04-26T23:01:32.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.245+0000 7fd319000640 1 -- 172.21.15.40:0/1335706761 wait complete. 2024-04-26T23:01:32.264 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T23:01:32.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.480+0000 7f5a1a200640 1 Processor -- start 2024-04-26T23:01:32.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.480+0000 7f5a1a200640 1 -- start start 2024-04-26T23:01:32.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.482+0000 7f5a1a200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5a140b5120 0x7f5a140b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.482+0000 7f5a1a200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a140b5500 0x7f5a140b6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.482+0000 7f5a1a200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a140d3dc0 con 0x7f5a140b59b0 2024-04-26T23:01:32.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.482+0000 7f5a1a200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f5a140d03b0 con 0x7f5a140b5500 2024-04-26T23:01:32.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.482+0000 7f5a1a200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5a140b75b0 con 0x7f5a140b5120 2024-04-26T23:01:32.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.482+0000 7f5a13e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a140b5500 0x7f5a140b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.482+0000 7f5a13e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a140b5500 0x7f5a140b6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42994/0 (socket says 172.21.15.40:42994) 2024-04-26T23:01:32.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.482+0000 7f5a13e00640 1 -- 172.21.15.40:0/617439142 learned_addr learned my addr 172.21.15.40:0/617439142 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:32.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.483+0000 7f5a18e00640 1 --2- 172.21.15.40:0/617439142 >> v2:172.21.15.103:3300/0 conn(0x7f5a140b5120 0x7f5a140b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.483+0000 7f5a13e00640 1 -- 172.21.15.40:0/617439142 >> v2:172.21.15.103:3300/0 conn(0x7f5a140b5120 msgr2=0x7f5a140b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:32.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.483+0000 7f5a13e00640 1 --2- 172.21.15.40:0/617439142 >> v2:172.21.15.103:3300/0 conn(0x7f5a140b5120 0x7f5a140b5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.483+0000 7f5a13e00640 1 -- 172.21.15.40:0/617439142 >> v1:172.21.15.103:6789/0 conn(0x7f5a140b59b0 legacy=0x7f5a140b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:32.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.483+0000 7f5a13e00640 1 -- 172.21.15.40:0/617439142 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a140d0600 con 0x7f5a140b5500 2024-04-26T23:01:32.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.484+0000 7f5a13e00640 1 --2- 172.21.15.40:0/617439142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a140b5500 0x7f5a140b6830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009870 tx=0x7f5a0002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fef1d81b7388066 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:32.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.484+0000 7f5a13400640 1 -- 172.21.15.40:0/617439142 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00037070 con 0x7f5a140b5500 2024-04-26T23:01:32.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.484+0000 7f5a13400640 1 -- 172.21.15.40:0/617439142 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a0002ae40 con 0x7f5a140b5500 2024-04-26T23:01:32.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.484+0000 7f5a13400640 1 -- 172.21.15.40:0/617439142 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00032660 con 0x7f5a140b5500 2024-04-26T23:01:32.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.485+0000 7f5a1a200640 1 -- 172.21.15.40:0/617439142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a140b5500 msgr2=0x7f5a140b6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:32.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.485+0000 7f5a1a200640 1 --2- 172.21.15.40:0/617439142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a140b5500 0x7f5a140b6830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009870 tx=0x7f5a0002a130 comp rx=0 tx=0).stop 2024-04-26T23:01:32.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.485+0000 7f5a1a200640 1 -- 172.21.15.40:0/617439142 shutdown_connections 2024-04-26T23:01:32.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.485+0000 7f5a1a200640 1 --2- 172.21.15.40:0/617439142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a140b5500 0x7f5a140b6830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.485+0000 7f5a1a200640 1 --2- 172.21.15.40:0/617439142 >> v2:172.21.15.103:3300/0 conn(0x7f5a140b5120 0x7f5a140b5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.485+0000 7f5a1a200640 1 -- 172.21.15.40:0/617439142 >> 172.21.15.40:0/617439142 conn(0x7f5a140d4db0 msgr2=0x7f5a140d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:32.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.485+0000 7f5a1a200640 1 -- 172.21.15.40:0/617439142 shutdown_connections 2024-04-26T23:01:32.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.485+0000 7f5a1a200640 1 -- 172.21.15.40:0/617439142 wait complete. 2024-04-26T23:01:32.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.487+0000 7f5a1a200640 1 Processor -- start 2024-04-26T23:01:32.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.487+0000 7f5a1a200640 1 -- start start 2024-04-26T23:01:32.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.488+0000 7f5a1a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a140b5120 0x7f5a140b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.488+0000 7f5a1a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a140b5500 0x7f5a140b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.488+0000 7f5a1a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a140b59b0 0x7f5a140b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.488+0000 7f5a1a200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5a140b7c00 con 0x7f5a140b5500 2024-04-26T23:01:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.488+0000 7f5a1a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5a140d0bc0 con 0x7f5a140b59b0 2024-04-26T23:01:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.488+0000 7f5a1a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5a140b7d80 con 0x7f5a140b5120 2024-04-26T23:01:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.488+0000 7f5a13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a140b5500 0x7f5a140b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.489+0000 7f5a13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a140b5500 0x7f5a140b3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43010/0 (socket says 172.21.15.40:43010) 2024-04-26T23:01:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.489+0000 7f5a13e00640 1 -- 172.21.15.40:0/2252083845 learned_addr learned my addr 172.21.15.40:0/2252083845 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.489+0000 7f5a19800640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a140b59b0 0x7f5a140b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.489+0000 7f5a18e00640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a140b5120 0x7f5a140b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.489+0000 7f5a13e00640 1 -- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a140b5120 msgr2=0x7f5a140b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.489+0000 7f5a13e00640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a140b5120 0x7f5a140b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.489+0000 7f5a13e00640 1 -- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a140b59b0 msgr2=0x7f5a140b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.489+0000 7f5a13e00640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a140b59b0 0x7f5a140b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.489+0000 7f5a13e00640 1 -- 172.21.15.40:0/2252083845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a140b2a40 con 0x7f5a140b5500 2024-04-26T23:01:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.489+0000 7f5a19800640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a140b59b0 0x7f5a140b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:32.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.489+0000 7f5a18e00640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a140b5120 0x7f5a140b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:32.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.490+0000 7f5a13e00640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a140b5500 0x7f5a140b3ea0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009870 tx=0x7f5a000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:32.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.490+0000 7f5a10c00640 1 -- 172.21.15.40:0/2252083845 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00037040 con 0x7f5a140b5500 2024-04-26T23:01:32.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.490+0000 7f5a10c00640 1 -- 172.21.15.40:0/2252083845 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a00004030 con 0x7f5a140b5500 2024-04-26T23:01:32.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.490+0000 7f5a10c00640 1 -- 172.21.15.40:0/2252083845 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00004420 con 0x7f5a140b5500 2024-04-26T23:01:32.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.490+0000 7f5a1a200640 1 -- 172.21.15.40:0/2252083845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a141f4f60 con 0x7f5a140b5500 2024-04-26T23:01:32.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.490+0000 7f5a1a200640 1 -- 172.21.15.40:0/2252083845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a141f5520 con 0x7f5a140b5500 2024-04-26T23:01:32.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.491+0000 7f5a1a200640 1 -- 172.21.15.40:0/2252083845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a14162940 con 0x7f5a140b5500 2024-04-26T23:01:32.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.492+0000 7f5a10c00640 1 -- 172.21.15.40:0/2252083845 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5a0003a070 con 0x7f5a140b5500 2024-04-26T23:01:32.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.493+0000 7f5a10c00640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f59f406c8f0 0x7f59f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:32.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.493+0000 7f5a10c00640 1 -- 172.21.15.40:0/2252083845 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5a000ac420 con 0x7f5a140b5500 2024-04-26T23:01:32.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.494+0000 7f5a18e00640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f59f406c8f0 0x7f59f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:32.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.495+0000 7f5a18e00640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f59f406c8f0 0x7f59f406edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5a14069b30 tx=0x7f5a08008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:32.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.498+0000 7f5a10c00640 1 -- 172.21.15.40:0/2252083845 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5a14162940 con 0x7f5a140b5500 2024-04-26T23:01:32.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.746+0000 7f5a1a200640 1 -- 172.21.15.40:0/2252083845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5a14069370 con 0x7f5a140b5500 2024-04-26T23:01:32.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.884+0000 7f5a10c00640 1 -- 172.21.15.40:0/2252083845 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7f5a140b2a40 con 0x7f5a140b5500 2024-04-26T23:01:32.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.889+0000 7f5a1a200640 1 -- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f59f406c8f0 msgr2=0x7f59f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:32.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.889+0000 7f5a1a200640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f59f406c8f0 0x7f59f406edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5a14069b30 tx=0x7f5a08008040 comp rx=0 tx=0).stop 2024-04-26T23:01:32.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.889+0000 7f5a1a200640 1 -- 172.21.15.40:0/2252083845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a140b5500 msgr2=0x7f5a140b3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:32.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.889+0000 7f5a1a200640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a140b5500 0x7f5a140b3ea0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009870 tx=0x7f5a000046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.889+0000 7f5a1a200640 1 -- 172.21.15.40:0/2252083845 shutdown_connections 2024-04-26T23:01:32.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.889+0000 7f5a1a200640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f59f406c8f0 0x7f59f406edb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.889+0000 7f5a1a200640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a140b59b0 0x7f5a140b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.889+0000 7f5a1a200640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a140b5500 0x7f5a140b3ea0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.889+0000 7f5a1a200640 1 --2- 172.21.15.40:0/2252083845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5a140b5120 0x7f5a140b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:32.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.889+0000 7f5a1a200640 1 -- 172.21.15.40:0/2252083845 >> 172.21.15.40:0/2252083845 conn(0x7f5a140d4db0 msgr2=0x7f5a14162230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:32.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.889+0000 7f5a1a200640 1 -- 172.21.15.40:0/2252083845 shutdown_connections 2024-04-26T23:01:32.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:32.889+0000 7f5a1a200640 1 -- 172.21.15.40:0/2252083845 wait complete. 2024-04-26T23:01:32.915 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-26T23:01:32.915 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-26T23:01:33.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.132+0000 7fabe4c00640 1 Processor -- start 2024-04-26T23:01:33.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.132+0000 7fabe4c00640 1 -- start start 2024-04-26T23:01:33.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.134+0000 7fabe4c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabe00b5120 0x7fabe00b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:33.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.134+0000 7fabe4c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fabe00b59b0 0x7fabe00b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:33.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.134+0000 7fabe4c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabe00d3dc0 con 0x7fabe00b5500 2024-04-26T23:01:33.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.134+0000 7fabe4c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fabe00d03b0 con 0x7fabe00b59b0 2024-04-26T23:01:33.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.134+0000 7fabe4c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fabe00b75b0 con 0x7fabe00b5120 2024-04-26T23:01:33.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.134+0000 7fabdea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fabe00b5500 0x7fabe00b6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:47500/0 (socket says 172.21.15.40:47500) 2024-04-26T23:01:33.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.134+0000 7fabdea00640 1 -- 172.21.15.40:0/3408380255 learned_addr learned my addr 172.21.15.40:0/3408380255 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:33.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.135+0000 7fabdf400640 1 --2- 172.21.15.40:0/3408380255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabe00b5120 0x7fabe00b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:33.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.135+0000 7fabdfe00640 1 --2- 172.21.15.40:0/3408380255 >> v2:172.21.15.103:3300/0 conn(0x7fabe00b59b0 0x7fabe00b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:33.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.135+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1608525807 0 0) 0x7fabe00d3dc0 con 0x7fabe00b5500 2024-04-26T23:01:33.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.135+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabc4003520 con 0x7fabe00b5500 2024-04-26T23:01:33.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.135+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 791795514 0 0) 0x7fabc4003520 con 0x7fabe00b5500 2024-04-26T23:01:33.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.135+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabe00b5120 msgr2=0x7fabe00b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:33.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.135+0000 7fabde000640 1 --2- 172.21.15.40:0/3408380255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabe00b5120 0x7fabe00b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:33.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.135+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 >> v2:172.21.15.103:3300/0 conn(0x7fabe00b59b0 msgr2=0x7fabe00b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:33.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.135+0000 7fabde000640 1 --2- 172.21.15.40:0/3408380255 >> v2:172.21.15.103:3300/0 conn(0x7fabe00b59b0 0x7fabe00b6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:33.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe00d0600 con 0x7fabe00b5500 2024-04-26T23:01:33.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fabcc001670 con 0x7fabe00b5500 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 >> v1:172.21.15.40:6789/0 conn(0x7fabe00b5500 legacy=0x7fabe00b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabde000640 1 --2- 172.21.15.40:0/3408380255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabc40039b0 0x7fabc4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabde000640 1 --2- 172.21.15.40:0/3408380255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc4004800 0x7fabc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabde000640 1 --2- 172.21.15.40:0/3408380255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabe00b5120 0x7fabc40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fabe00d03b0 con 0x7fabc4004800 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fabe00b75b0 con 0x7fabc40039b0 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fabc400ba40 con 0x7fabe00b5120 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabdea00640 1 --2- 172.21.15.40:0/3408380255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc4004800 0x7fabc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabdfe00640 1 --2- 172.21.15.40:0/3408380255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabe00b5120 0x7fabc40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabdf400640 1 --2- 172.21.15.40:0/3408380255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabc40039b0 0x7fabc4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabdea00640 1 -- 172.21.15.40:0/3408380255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabe00b5120 msgr2=0x7fabc40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabdea00640 1 --2- 172.21.15.40:0/3408380255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabe00b5120 0x7fabc40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.136+0000 7fabdea00640 1 -- 172.21.15.40:0/3408380255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabc40039b0 msgr2=0x7fabc4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.137+0000 7fabdea00640 1 --2- 172.21.15.40:0/3408380255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabc40039b0 0x7fabc4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:33.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.137+0000 7fabdea00640 1 -- 172.21.15.40:0/3408380255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc4003520 con 0x7fabc4004800 2024-04-26T23:01:33.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.138+0000 7fabdea00640 1 --2- 172.21.15.40:0/3408380255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc4004800 0x7fabc4004c80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fabcc007fd0 tx=0x7fabc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60c3a9801881393 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:33.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.138+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc007d50 con 0x7fabc4004800 2024-04-26T23:01:33.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.138+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabcc005ad0 con 0x7fabc4004800 2024-04-26T23:01:33.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.138+0000 7fabde000640 1 -- 172.21.15.40:0/3408380255 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc007070 con 0x7fabc4004800 2024-04-26T23:01:33.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.138+0000 7fabe4c00640 1 -- 172.21.15.40:0/3408380255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc4004800 msgr2=0x7fabc4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:33.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.138+0000 7fabe4c00640 1 --2- 172.21.15.40:0/3408380255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc4004800 0x7fabc4004c80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fabcc007fd0 tx=0x7fabc4005830 comp rx=0 tx=0).stop 2024-04-26T23:01:33.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.138+0000 7fabdfe00640 1 -- 172.21.15.40:0/3408380255 reap_dead start 2024-04-26T23:01:33.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.138+0000 7fabe4c00640 1 -- 172.21.15.40:0/3408380255 shutdown_connections 2024-04-26T23:01:33.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.138+0000 7fabe4c00640 1 -- 172.21.15.40:0/3408380255 >> 172.21.15.40:0/3408380255 conn(0x7fabe00d4db0 msgr2=0x7fabe00d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:33.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.139+0000 7fabe4c00640 1 -- 172.21.15.40:0/3408380255 shutdown_connections 2024-04-26T23:01:33.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.139+0000 7fabe4c00640 1 -- 172.21.15.40:0/3408380255 wait complete. 2024-04-26T23:01:33.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.140+0000 7fabe4c00640 1 Processor -- start 2024-04-26T23:01:33.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.141+0000 7fabe4c00640 1 -- start start 2024-04-26T23:01:33.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.141+0000 7fabe4c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabc4004800 0x7fabe00cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:33.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.141+0000 7fabe4c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabe015d5e0 0x7fabe015da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.141+0000 7fabe4c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabe0154d70 0x7fabe01551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.141+0000 7fabe4c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fabe00b5ee0 con 0x7fabe0154d70 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.141+0000 7fabe4c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fabe00d0bc0 con 0x7fabc4004800 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.141+0000 7fabe4c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fabe00b6060 con 0x7fabe015d5e0 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.142+0000 7fabdfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabe0154d70 0x7fabe01551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.142+0000 7fabdfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabe0154d70 0x7fabe01551f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43034/0 (socket says 172.21.15.40:43034) 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.142+0000 7fabdfe00640 1 -- 172.21.15.40:0/3645615355 learned_addr learned my addr 172.21.15.40:0/3645615355 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.142+0000 7fabdf400640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabc4004800 0x7fabe00cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.142+0000 7fabdea00640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabe015d5e0 0x7fabe015da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.142+0000 7fabdfe00640 1 -- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabe015d5e0 msgr2=0x7fabe015da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.142+0000 7fabdfe00640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabe015d5e0 0x7fabe015da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.142+0000 7fabdfe00640 1 -- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabc4004800 msgr2=0x7fabe00cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.142+0000 7fabdfe00640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabc4004800 0x7fabe00cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.142+0000 7fabdfe00640 1 -- 172.21.15.40:0/3645615355 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe00cdcb0 con 0x7fabe0154d70 2024-04-26T23:01:33.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.143+0000 7fabdfe00640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabe0154d70 0x7fabe01551f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fabe008e270 tx=0x7fabd80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:33.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.143+0000 7fabd7400640 1 -- 172.21.15.40:0/3645615355 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd8011e40 con 0x7fabe0154d70 2024-04-26T23:01:33.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.143+0000 7fabd7400640 1 -- 172.21.15.40:0/3645615355 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabd801b070 con 0x7fabe0154d70 2024-04-26T23:01:33.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.143+0000 7fabd7400640 1 -- 172.21.15.40:0/3645615355 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd8003990 con 0x7fabe0154d70 2024-04-26T23:01:33.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.143+0000 7fabe4c00640 1 -- 172.21.15.40:0/3645615355 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe00cdf90 con 0x7fabe0154d70 2024-04-26T23:01:33.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.143+0000 7fabe4c00640 1 -- 172.21.15.40:0/3645615355 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe00ce550 con 0x7fabe0154d70 2024-04-26T23:01:33.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.144+0000 7fabe4c00640 1 -- 172.21.15.40:0/3645615355 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabe0069370 con 0x7fabe0154d70 2024-04-26T23:01:33.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.145+0000 7fabd7400640 1 -- 172.21.15.40:0/3645615355 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fabd8005d10 con 0x7fabe0154d70 2024-04-26T23:01:33.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.146+0000 7fabd7400640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fabc006c920 0x7fabc006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:33.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.146+0000 7fabd7400640 1 -- 172.21.15.40:0/3645615355 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fabd808c4f0 con 0x7fabe0154d70 2024-04-26T23:01:33.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.147+0000 7fabdf400640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fabc006c920 0x7fabc006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:33.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.148+0000 7fabdf400640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fabc006c920 0x7fabc006ede0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fabe00b2030 tx=0x7fabd0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:33.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.151+0000 7fabd7400640 1 -- 172.21.15.40:0/3645615355 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fabd8025020 con 0x7fabe0154d70 2024-04-26T23:01:33.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.398+0000 7fabe4c00640 1 -- 172.21.15.40:0/3645615355 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fabe00b6ae0 con 0x7fabe0154d70 2024-04-26T23:01:33.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.904+0000 7fabd7400640 1 -- 172.21.15.40:0/3645615355 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) ==== 122+0+0 (secure 0 0 0) 0x7fabd8014350 con 0x7fabe0154d70 2024-04-26T23:01:33.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.907+0000 7fabe4c00640 1 -- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fabc006c920 msgr2=0x7fabc006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:33.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.907+0000 7fabe4c00640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fabc006c920 0x7fabc006ede0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fabe00b2030 tx=0x7fabd0008040 comp rx=0 tx=0).stop 2024-04-26T23:01:33.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.907+0000 7fabe4c00640 1 -- 172.21.15.40:0/3645615355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabe0154d70 msgr2=0x7fabe01551f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:33.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.907+0000 7fabe4c00640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabe0154d70 0x7fabe01551f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fabe008e270 tx=0x7fabd80115a0 comp rx=0 tx=0).stop 2024-04-26T23:01:33.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.907+0000 7fabe4c00640 1 -- 172.21.15.40:0/3645615355 shutdown_connections 2024-04-26T23:01:33.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.907+0000 7fabe4c00640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fabc006c920 0x7fabc006ede0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:33.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.907+0000 7fabe4c00640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabe0154d70 0x7fabe01551f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:33.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.907+0000 7fabe4c00640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fabe015d5e0 0x7fabe015da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:33.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.907+0000 7fabe4c00640 1 --2- 172.21.15.40:0/3645615355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabc4004800 0x7fabe00cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:33.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.908+0000 7fabe4c00640 1 -- 172.21.15.40:0/3645615355 >> 172.21.15.40:0/3645615355 conn(0x7fabe00d4db0 msgr2=0x7fabe015f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:33.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.908+0000 7fabe4c00640 1 -- 172.21.15.40:0/3645615355 shutdown_connections 2024-04-26T23:01:33.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:33.908+0000 7fabe4c00640 1 -- 172.21.15.40:0/3645615355 wait complete. 2024-04-26T23:01:33.930 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-26T23:01:33.930 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:01:34.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.146+0000 7f82c2200640 1 Processor -- start 2024-04-26T23:01:34.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.146+0000 7f82c2200640 1 -- start start 2024-04-26T23:01:34.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.147+0000 7f82c2200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82bc0b4770 0x7f82bc0d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.147+0000 7f82c2200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f82bc0b63e0 0x7f82bc161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.147+0000 7f82c2200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82bc0b4110 con 0x7f82bc161850 2024-04-26T23:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.147+0000 7f82c2200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f82bc0d3850 con 0x7f82bc0b63e0 2024-04-26T23:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.147+0000 7f82c2200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f82bc0d3b30 con 0x7f82bc0b4770 2024-04-26T23:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.148+0000 7f82c1800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f82bc0b63e0 0x7f82bc161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.148+0000 7f82c0e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f82bc161850 0x7f82bc0d1440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:47504/0 (socket says 172.21.15.40:47504) 2024-04-26T23:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.148+0000 7f82c0e00640 1 -- 172.21.15.40:0/3303415352 learned_addr learned my addr 172.21.15.40:0/3303415352 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.148+0000 7f82c1800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f82bc0b63e0 0x7f82bc161c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43040/0 (socket says 172.21.15.40:43040) 2024-04-26T23:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.148+0000 7f82bbe00640 1 --2- 172.21.15.40:0/3303415352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82bc0b4770 0x7f82bc0d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.148+0000 7f82c1800640 1 -- 172.21.15.40:0/3303415352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82bc0b4770 msgr2=0x7f82bc0d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.148+0000 7f82c1800640 1 --2- 172.21.15.40:0/3303415352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82bc0b4770 0x7f82bc0d1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.148+0000 7f82c1800640 1 -- 172.21.15.40:0/3303415352 >> v1:172.21.15.40:6789/0 conn(0x7f82bc161850 legacy=0x7f82bc0d1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.148+0000 7f82bb400640 1 -- 172.21.15.40:0/3303415352 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2236692440 0 0) 0x7f82bc0b4110 con 0x7f82bc161850 2024-04-26T23:01:34.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.148+0000 7f82c1800640 1 -- 172.21.15.40:0/3303415352 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82bc0b3d70 con 0x7f82bc0b63e0 2024-04-26T23:01:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.149+0000 7f82c1800640 1 --2- 172.21.15.40:0/3303415352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82bc0b63e0 0x7f82bc161c30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f82b4009d90 tx=0x7f82b402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ce18bd4c3e3687 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.150+0000 7f82bb400640 1 -- 172.21.15.40:0/3303415352 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b4031600 con 0x7f82bc0b63e0 2024-04-26T23:01:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.150+0000 7f82bb400640 1 -- 172.21.15.40:0/3303415352 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82b40317a0 con 0x7f82bc0b63e0 2024-04-26T23:01:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.150+0000 7f82bb400640 1 -- 172.21.15.40:0/3303415352 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b4031b70 con 0x7f82bc0b63e0 2024-04-26T23:01:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.150+0000 7f82c2200640 1 -- 172.21.15.40:0/3303415352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82bc0b63e0 msgr2=0x7f82bc161c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.150+0000 7f82c2200640 1 --2- 172.21.15.40:0/3303415352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82bc0b63e0 0x7f82bc161c30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f82b4009d90 tx=0x7f82b402a5d0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.150+0000 7f82c2200640 1 -- 172.21.15.40:0/3303415352 shutdown_connections 2024-04-26T23:01:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.150+0000 7f82c2200640 1 --2- 172.21.15.40:0/3303415352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82bc0b63e0 0x7f82bc161c30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.150+0000 7f82c2200640 1 --2- 172.21.15.40:0/3303415352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82bc0b4770 0x7f82bc0d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.150+0000 7f82c2200640 1 -- 172.21.15.40:0/3303415352 >> 172.21.15.40:0/3303415352 conn(0x7f82bc0d4e10 msgr2=0x7f82bc0d5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.150+0000 7f82c2200640 1 -- 172.21.15.40:0/3303415352 shutdown_connections 2024-04-26T23:01:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.150+0000 7f82c2200640 1 -- 172.21.15.40:0/3303415352 wait complete. 2024-04-26T23:01:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.152+0000 7f82c2200640 1 Processor -- start 2024-04-26T23:01:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.152+0000 7f82c2200640 1 -- start start 2024-04-26T23:01:34.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.153+0000 7f82c2200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82bc0b4770 0x7f82bc0ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:34.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.153+0000 7f82c2200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82bc0b63e0 0x7f82bc153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.153+0000 7f82c2200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82bc161850 0x7f82bc154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.153+0000 7f82c2200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc0d2210 con 0x7f82bc0b63e0 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.153+0000 7f82c2200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc0d2090 con 0x7f82bc0b4770 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.153+0000 7f82c2200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f82bc162170 con 0x7f82bc161850 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.154+0000 7f82bbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82bc0b63e0 0x7f82bc153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.154+0000 7f82bbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82bc0b63e0 0x7f82bc153b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43052/0 (socket says 172.21.15.40:43052) 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.154+0000 7f82bbe00640 1 -- 172.21.15.40:0/172487091 learned_addr learned my addr 172.21.15.40:0/172487091 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.154+0000 7f82c1800640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82bc161850 0x7f82bc154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.154+0000 7f82c0e00640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82bc0b4770 0x7f82bc0ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.154+0000 7f82bbe00640 1 -- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82bc161850 msgr2=0x7f82bc154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.154+0000 7f82bbe00640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82bc161850 0x7f82bc154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.154+0000 7f82bbe00640 1 -- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82bc0b4770 msgr2=0x7f82bc0ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.154+0000 7f82bbe00640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82bc0b4770 0x7f82bc0ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.154+0000 7f82bbe00640 1 -- 172.21.15.40:0/172487091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82bc0cd6f0 con 0x7f82bc0b63e0 2024-04-26T23:01:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.154+0000 7f82c0e00640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82bc0b4770 0x7f82bc0ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.155+0000 7f82bbe00640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82bc0b63e0 0x7f82bc153b20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f82a400d7d0 tx=0x7f82a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.155+0000 7f82b8c00640 1 -- 172.21.15.40:0/172487091 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a400b6c0 con 0x7f82bc0b63e0 2024-04-26T23:01:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.155+0000 7f82b8c00640 1 -- 172.21.15.40:0/172487091 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82a4019070 con 0x7f82bc0b63e0 2024-04-26T23:01:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.155+0000 7f82b8c00640 1 -- 172.21.15.40:0/172487091 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a400ba80 con 0x7f82bc0b63e0 2024-04-26T23:01:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.155+0000 7f82c2200640 1 -- 172.21.15.40:0/172487091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82bc0cd9d0 con 0x7f82bc0b63e0 2024-04-26T23:01:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.155+0000 7f82c2200640 1 -- 172.21.15.40:0/172487091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82bc151ce0 con 0x7f82bc0b63e0 2024-04-26T23:01:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.156+0000 7f82c2200640 1 -- 172.21.15.40:0/172487091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82bc069a80 con 0x7f82bc0b63e0 2024-04-26T23:01:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.157+0000 7f82b8c00640 1 -- 172.21.15.40:0/172487091 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f82a400bc20 con 0x7f82bc0b63e0 2024-04-26T23:01:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.158+0000 7f82b8c00640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f829806c8f0 0x7f829806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.158+0000 7f82b8c00640 1 -- 172.21.15.40:0/172487091 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f82a408c320 con 0x7f82bc0b63e0 2024-04-26T23:01:34.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.159+0000 7f82c0e00640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f829806c8f0 0x7f829806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:34.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.160+0000 7f82c0e00640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f829806c8f0 0x7f829806edb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f82ac000f60 tx=0x7f82ac009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.162+0000 7f82b8c00640 1 -- 172.21.15.40:0/172487091 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f82a401c360 con 0x7f82bc0b63e0 2024-04-26T23:01:34.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.418+0000 7f82c2200640 1 -- 172.21.15.40:0/172487091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82bc0bac70 con 0x7f82bc0b63e0 2024-04-26T23:01:34.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.419+0000 7f82b8c00640 1 -- 172.21.15.40:0/172487091 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f82a405a400 con 0x7f82bc0b63e0 2024-04-26T23:01:34.419 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:34.419 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1253648754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1253648754},{"type":"v1","addr":"172.21.15.103:6836","nonce":1253648754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3239652611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":3239652611},{"type":"v1","addr":"172.21.15.103:6837","nonce":3239652611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3670144900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3670144900},{"type":"v1","addr":"172.21.15.40:6833","nonce":3670144900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:01:05.171276+0000","modified":"2024-04-26T23:01:33.399801+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3989190340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3989190340},{"type":"v1","addr":"172.21.15.40:6835","nonce":3989190340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-26T23:01:34.419 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 9 2024-04-26T23:01:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.422+0000 7f82c2200640 1 -- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f829806c8f0 msgr2=0x7f829806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.422+0000 7f82c2200640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f829806c8f0 0x7f829806edb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f82ac000f60 tx=0x7f82ac009040 comp rx=0 tx=0).stop 2024-04-26T23:01:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.422+0000 7f82c2200640 1 -- 172.21.15.40:0/172487091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82bc0b63e0 msgr2=0x7f82bc153b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.422+0000 7f82c2200640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82bc0b63e0 0x7f82bc153b20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f82a400d7d0 tx=0x7f82a400dc90 comp rx=0 tx=0).stop 2024-04-26T23:01:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.422+0000 7f82c2200640 1 -- 172.21.15.40:0/172487091 shutdown_connections 2024-04-26T23:01:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.422+0000 7f82c2200640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f829806c8f0 0x7f829806edb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.422+0000 7f82c2200640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82bc161850 0x7f82bc154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.422+0000 7f82c2200640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82bc0b63e0 0x7f82bc153b20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.422+0000 7f82c2200640 1 --2- 172.21.15.40:0/172487091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82bc0b4770 0x7f82bc0ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.422+0000 7f82c2200640 1 -- 172.21.15.40:0/172487091 >> 172.21.15.40:0/172487091 conn(0x7f82bc0d4e10 msgr2=0x7f82bc0d2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.422+0000 7f82c2200640 1 -- 172.21.15.40:0/172487091 shutdown_connections 2024-04-26T23:01:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.422+0000 7f82c2200640 1 -- 172.21.15.40:0/172487091 wait complete. 2024-04-26T23:01:34.441 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:34.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.650+0000 7fef67e00640 1 Processor -- start 2024-04-26T23:01:34.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.651+0000 7fef67e00640 1 -- start start 2024-04-26T23:01:34.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.652+0000 7fef67e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef680b6220 0x7fef680d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:34.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.652+0000 7fef67e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef680d39f0 0x7fef680b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:34.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.652+0000 7fef67e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef68162c40 con 0x7fef680d08d0 2024-04-26T23:01:34.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.652+0000 7fef67e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fef680d2d80 con 0x7fef680d39f0 2024-04-26T23:01:34.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.652+0000 7fef67e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fef680b7c80 con 0x7fef680b6220 2024-04-26T23:01:34.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.653+0000 7fef67400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef680d39f0 0x7fef680b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:34.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.653+0000 7fef67400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef680d39f0 0x7fef680b8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43068/0 (socket says 172.21.15.40:43068) 2024-04-26T23:01:34.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.653+0000 7fef67400640 1 -- 172.21.15.40:0/2975204796 learned_addr learned my addr 172.21.15.40:0/2975204796 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:34.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.653+0000 7fef66a00640 1 --2- 172.21.15.40:0/2975204796 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef680b6220 0x7fef680d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:34.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.653+0000 7fef67400640 1 -- 172.21.15.40:0/2975204796 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef680b6220 msgr2=0x7fef680d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:34.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.653+0000 7fef67400640 1 --2- 172.21.15.40:0/2975204796 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef680b6220 0x7fef680d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.653+0000 7fef67400640 1 -- 172.21.15.40:0/2975204796 >> v1:172.21.15.103:6789/0 conn(0x7fef680d08d0 legacy=0x7fef680d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:34.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.653+0000 7fef67400640 1 -- 172.21.15.40:0/2975204796 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef680b7f60 con 0x7fef680d39f0 2024-04-26T23:01:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.655+0000 7fef67400640 1 --2- 172.21.15.40:0/2975204796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef680d39f0 0x7fef680b8d20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fef60009870 tx=0x7fef6002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1441257731284c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.655+0000 7fef65600640 1 -- 172.21.15.40:0/2975204796 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60037070 con 0x7fef680d39f0 2024-04-26T23:01:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.655+0000 7fef65600640 1 -- 172.21.15.40:0/2975204796 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef60032440 con 0x7fef680d39f0 2024-04-26T23:01:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.655+0000 7fef65600640 1 -- 172.21.15.40:0/2975204796 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60032810 con 0x7fef680d39f0 2024-04-26T23:01:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.655+0000 7fef67e00640 1 -- 172.21.15.40:0/2975204796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef680d39f0 msgr2=0x7fef680b8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.655+0000 7fef67e00640 1 --2- 172.21.15.40:0/2975204796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef680d39f0 0x7fef680b8d20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fef60009870 tx=0x7fef6002a270 comp rx=0 tx=0).stop 2024-04-26T23:01:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.656+0000 7fef67e00640 1 -- 172.21.15.40:0/2975204796 shutdown_connections 2024-04-26T23:01:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.656+0000 7fef67e00640 1 --2- 172.21.15.40:0/2975204796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef680d39f0 0x7fef680b8d20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.656+0000 7fef67e00640 1 --2- 172.21.15.40:0/2975204796 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef680b6220 0x7fef680d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.656+0000 7fef67e00640 1 -- 172.21.15.40:0/2975204796 >> 172.21.15.40:0/2975204796 conn(0x7fef680d4d90 msgr2=0x7fef680d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.656+0000 7fef67e00640 1 -- 172.21.15.40:0/2975204796 shutdown_connections 2024-04-26T23:01:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.656+0000 7fef67e00640 1 -- 172.21.15.40:0/2975204796 wait complete. 2024-04-26T23:01:34.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.657+0000 7fef67e00640 1 Processor -- start 2024-04-26T23:01:34.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.658+0000 7fef67e00640 1 -- start start 2024-04-26T23:01:34.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.658+0000 7fef67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef680b6220 0x7fef681f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:34.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.658+0000 7fef67e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef680d08d0 0x7fef680b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:34.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.658+0000 7fef67e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef680d39f0 0x7fef680a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:34.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef67e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fef680b9260 con 0x7fef680b6220 2024-04-26T23:01:34.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef67e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef680b8520 con 0x7fef680d39f0 2024-04-26T23:01:34.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef67e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fef680b93e0 con 0x7fef680d08d0 2024-04-26T23:01:34.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef66a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef680b6220 0x7fef681f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef66a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef680b6220 0x7fef681f57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43070/0 (socket says 172.21.15.40:43070) 2024-04-26T23:01:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef66a00640 1 -- 172.21.15.40:0/2552393528 learned_addr learned my addr 172.21.15.40:0/2552393528 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef66000640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef680d08d0 0x7fef680b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef67400640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef680d39f0 0x7fef680a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef66a00640 1 -- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef680d08d0 msgr2=0x7fef680b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef66a00640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef680d08d0 0x7fef680b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef66a00640 1 -- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef680d39f0 msgr2=0x7fef680a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef66a00640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef680d39f0 0x7fef680a2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef66a00640 1 -- 172.21.15.40:0/2552393528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef680b2a70 con 0x7fef680b6220 2024-04-26T23:01:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.659+0000 7fef67400640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef680d39f0 0x7fef680a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.660+0000 7fef66a00640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef680b6220 0x7fef681f57b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fef58009db0 tx=0x7fef5800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.660+0000 7fef5ea00640 1 -- 172.21.15.40:0/2552393528 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef58018020 con 0x7fef680b6220 2024-04-26T23:01:34.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.660+0000 7fef5ea00640 1 -- 172.21.15.40:0/2552393528 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef58014480 con 0x7fef680b6220 2024-04-26T23:01:34.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.660+0000 7fef67e00640 1 -- 172.21.15.40:0/2552393528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef680b2d50 con 0x7fef680b6220 2024-04-26T23:01:34.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.660+0000 7fef5ea00640 1 -- 172.21.15.40:0/2552393528 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef58014850 con 0x7fef680b6220 2024-04-26T23:01:34.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.660+0000 7fef67e00640 1 -- 172.21.15.40:0/2552393528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef680b3310 con 0x7fef680b6220 2024-04-26T23:01:34.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.661+0000 7fef67e00640 1 -- 172.21.15.40:0/2552393528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef68069370 con 0x7fef680b6220 2024-04-26T23:01:34.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.662+0000 7fef5ea00640 1 -- 172.21.15.40:0/2552393528 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fef58005d10 con 0x7fef680b6220 2024-04-26T23:01:34.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.663+0000 7fef5ea00640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fef4406c8f0 0x7fef4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:34.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.663+0000 7fef5ea00640 1 -- 172.21.15.40:0/2552393528 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fef5808cd70 con 0x7fef680b6220 2024-04-26T23:01:34.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.664+0000 7fef66000640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fef4406c8f0 0x7fef4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:34.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.665+0000 7fef66000640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fef4406c8f0 0x7fef4406edb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fef680cb3e0 tx=0x7fef50009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:34.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.669+0000 7fef5ea00640 1 -- 172.21.15.40:0/2552393528 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fef58022030 con 0x7fef680b6220 2024-04-26T23:01:34.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.875+0000 7fef67e00640 1 -- 172.21.15.40:0/2552393528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fef680d30c0 con 0x7fef680b6220 2024-04-26T23:01:34.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.877+0000 7fef5ea00640 1 -- 172.21.15.40:0/2552393528 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7fef58010040 con 0x7fef680b6220 2024-04-26T23:01:34.877 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:34.878 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:11.763965+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.880+0000 7fef67e00640 1 -- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fef4406c8f0 msgr2=0x7fef4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.880+0000 7fef67e00640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fef4406c8f0 0x7fef4406edb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fef680cb3e0 tx=0x7fef50009040 comp rx=0 tx=0).stop 2024-04-26T23:01:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.880+0000 7fef67e00640 1 -- 172.21.15.40:0/2552393528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef680b6220 msgr2=0x7fef681f57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.880+0000 7fef67e00640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef680b6220 0x7fef681f57b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fef58009db0 tx=0x7fef5800c490 comp rx=0 tx=0).stop 2024-04-26T23:01:34.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.880+0000 7fef67e00640 1 -- 172.21.15.40:0/2552393528 shutdown_connections 2024-04-26T23:01:34.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.880+0000 7fef67e00640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fef4406c8f0 0x7fef4406edb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.880+0000 7fef67e00640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef680d39f0 0x7fef680a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.880+0000 7fef67e00640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef680d08d0 0x7fef680b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.880+0000 7fef67e00640 1 --2- 172.21.15.40:0/2552393528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef680b6220 0x7fef681f57b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:34.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.880+0000 7fef67e00640 1 -- 172.21.15.40:0/2552393528 >> 172.21.15.40:0/2552393528 conn(0x7fef680d4d90 msgr2=0x7fef680cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:34.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.880+0000 7fef67e00640 1 -- 172.21.15.40:0/2552393528 shutdown_connections 2024-04-26T23:01:34.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:34.880+0000 7fef67e00640 1 -- 172.21.15.40:0/2552393528 wait complete. 2024-04-26T23:01:34.900 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T23:01:34.900 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:01:35.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.117+0000 7fd7edc00640 1 Processor -- start 2024-04-26T23:01:35.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.118+0000 7fd7edc00640 1 -- start start 2024-04-26T23:01:35.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.119+0000 7fd7edc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd7e80b6220 0x7fd7e80d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.119+0000 7fd7edc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd7e80d39f0 0x7fd7e80b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.119+0000 7fd7edc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7e8162c40 con 0x7fd7e80d08d0 2024-04-26T23:01:35.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.119+0000 7fd7edc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fd7e80d2d80 con 0x7fd7e80d39f0 2024-04-26T23:01:35.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.119+0000 7fd7edc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd7e80b7c80 con 0x7fd7e80b6220 2024-04-26T23:01:35.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.120+0000 7fd7ed200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd7e80d39f0 0x7fd7e80b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.120+0000 7fd7ed200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd7e80d39f0 0x7fd7e80b8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38622/0 (socket says 172.21.15.40:38622) 2024-04-26T23:01:35.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.120+0000 7fd7ed200640 1 -- 172.21.15.40:0/2035876624 learned_addr learned my addr 172.21.15.40:0/2035876624 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:35.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.120+0000 7fd7e7e00640 1 --2- 172.21.15.40:0/2035876624 >> v2:172.21.15.103:3300/0 conn(0x7fd7e80b6220 0x7fd7e80d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.120+0000 7fd7ed200640 1 -- 172.21.15.40:0/2035876624 >> v2:172.21.15.103:3300/0 conn(0x7fd7e80b6220 msgr2=0x7fd7e80d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:35.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.120+0000 7fd7ed200640 1 --2- 172.21.15.40:0/2035876624 >> v2:172.21.15.103:3300/0 conn(0x7fd7e80b6220 0x7fd7e80d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.120+0000 7fd7ed200640 1 -- 172.21.15.40:0/2035876624 >> v1:172.21.15.103:6789/0 conn(0x7fd7e80d08d0 legacy=0x7fd7e80d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:35.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.120+0000 7fd7ed200640 1 -- 172.21.15.40:0/2035876624 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e80b7f60 con 0x7fd7e80d39f0 2024-04-26T23:01:35.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.121+0000 7fd7ed200640 1 --2- 172.21.15.40:0/2035876624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80d39f0 0x7fd7e80b8d20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd7e80d4cf0 tx=0x7fd7e002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59bdac640049ef3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:35.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.122+0000 7fd7e6a00640 1 -- 172.21.15.40:0/2035876624 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0037070 con 0x7fd7e80d39f0 2024-04-26T23:01:35.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.122+0000 7fd7e6a00640 1 -- 172.21.15.40:0/2035876624 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e0032440 con 0x7fd7e80d39f0 2024-04-26T23:01:35.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.122+0000 7fd7e6a00640 1 -- 172.21.15.40:0/2035876624 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0032810 con 0x7fd7e80d39f0 2024-04-26T23:01:35.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.122+0000 7fd7edc00640 1 -- 172.21.15.40:0/2035876624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80d39f0 msgr2=0x7fd7e80b8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:35.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.122+0000 7fd7edc00640 1 --2- 172.21.15.40:0/2035876624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80d39f0 0x7fd7e80b8d20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd7e80d4cf0 tx=0x7fd7e002a230 comp rx=0 tx=0).stop 2024-04-26T23:01:35.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.122+0000 7fd7edc00640 1 -- 172.21.15.40:0/2035876624 shutdown_connections 2024-04-26T23:01:35.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.122+0000 7fd7edc00640 1 --2- 172.21.15.40:0/2035876624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80d39f0 0x7fd7e80b8d20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.122+0000 7fd7edc00640 1 --2- 172.21.15.40:0/2035876624 >> v2:172.21.15.103:3300/0 conn(0x7fd7e80b6220 0x7fd7e80d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.122+0000 7fd7edc00640 1 -- 172.21.15.40:0/2035876624 >> 172.21.15.40:0/2035876624 conn(0x7fd7e80d4d90 msgr2=0x7fd7e80d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:35.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.122+0000 7fd7edc00640 1 -- 172.21.15.40:0/2035876624 shutdown_connections 2024-04-26T23:01:35.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.123+0000 7fd7edc00640 1 -- 172.21.15.40:0/2035876624 wait complete. 2024-04-26T23:01:35.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.124+0000 7fd7edc00640 1 Processor -- start 2024-04-26T23:01:35.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.124+0000 7fd7edc00640 1 -- start start 2024-04-26T23:01:35.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.125+0000 7fd7edc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e80b6220 0x7fd7e81f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.125+0000 7fd7edc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80d08d0 0x7fd7e80b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.125+0000 7fd7edc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7e80d39f0 0x7fd7e80b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.125+0000 7fd7edc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e80b9260 con 0x7fd7e80d08d0 2024-04-26T23:01:35.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.125+0000 7fd7edc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e80b8520 con 0x7fd7e80d39f0 2024-04-26T23:01:35.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.125+0000 7fd7edc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd7e80b93e0 con 0x7fd7e80b6220 2024-04-26T23:01:35.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.126+0000 7fd7e7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80d08d0 0x7fd7e80b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.126+0000 7fd7e7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80d08d0 0x7fd7e80b5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38632/0 (socket says 172.21.15.40:38632) 2024-04-26T23:01:35.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.126+0000 7fd7e7400640 1 -- 172.21.15.40:0/2394869411 learned_addr learned my addr 172.21.15.40:0/2394869411 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:35.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.126+0000 7fd7e7e00640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e80b6220 0x7fd7e81f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.126+0000 7fd7ed200640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7e80d39f0 0x7fd7e80b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.126+0000 7fd7e7400640 1 -- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e80b6220 msgr2=0x7fd7e81f4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:35.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.126+0000 7fd7e7400640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e80b6220 0x7fd7e81f4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.126+0000 7fd7e7400640 1 -- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7e80d39f0 msgr2=0x7fd7e80b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:35.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.126+0000 7fd7e7400640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7e80d39f0 0x7fd7e80b6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.126+0000 7fd7e7400640 1 -- 172.21.15.40:0/2394869411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e80b3d60 con 0x7fd7e80d08d0 2024-04-26T23:01:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.126+0000 7fd7e7e00640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e80b6220 0x7fd7e81f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.126+0000 7fd7e7400640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80d08d0 0x7fd7e80b5110 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd7d000c430 tx=0x7fd7d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.127+0000 7fd7dfe00640 1 -- 172.21.15.40:0/2394869411 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0011790 con 0x7fd7e80d08d0 2024-04-26T23:01:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.127+0000 7fd7dfe00640 1 -- 172.21.15.40:0/2394869411 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d0017070 con 0x7fd7e80d08d0 2024-04-26T23:01:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.127+0000 7fd7edc00640 1 -- 172.21.15.40:0/2394869411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e80b4040 con 0x7fd7e80d08d0 2024-04-26T23:01:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.127+0000 7fd7dfe00640 1 -- 172.21.15.40:0/2394869411 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0011bb0 con 0x7fd7e80d08d0 2024-04-26T23:01:35.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.127+0000 7fd7edc00640 1 -- 172.21.15.40:0/2394869411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e80b4600 con 0x7fd7e80d08d0 2024-04-26T23:01:35.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.128+0000 7fd7edc00640 1 -- 172.21.15.40:0/2394869411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e8069370 con 0x7fd7e80d08d0 2024-04-26T23:01:35.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.129+0000 7fd7dfe00640 1 -- 172.21.15.40:0/2394869411 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd7d001c460 con 0x7fd7e80d08d0 2024-04-26T23:01:35.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.130+0000 7fd7dfe00640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd7c406c8f0 0x7fd7c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.130+0000 7fd7dfe00640 1 -- 172.21.15.40:0/2394869411 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd7d008bdd0 con 0x7fd7e80d08d0 2024-04-26T23:01:35.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.131+0000 7fd7e7e00640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd7c406c8f0 0x7fd7c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.132+0000 7fd7e7e00640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd7c406c8f0 0x7fd7c406edb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd7e80d4930 tx=0x7fd7d800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:35.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.135+0000 7fd7dfe00640 1 -- 172.21.15.40:0/2394869411 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd7d0059e20 con 0x7fd7e80d08d0 2024-04-26T23:01:35.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.337+0000 7fd7edc00640 1 -- 172.21.15.40:0/2394869411 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd7e80d30c0 con 0x7fd7c406c8f0 2024-04-26T23:01:35.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.369+0000 7fd7dfe00640 1 -- 172.21.15.40:0/2394869411 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332538 (secure 0 0 0) 0x7fd7e80d30c0 con 0x7fd7c406c8f0 2024-04-26T23:01:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:35.376 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:01:35.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.374+0000 7fd7edc00640 1 -- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd7c406c8f0 msgr2=0x7fd7c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:35.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.374+0000 7fd7edc00640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd7c406c8f0 0x7fd7c406edb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd7e80d4930 tx=0x7fd7d800b040 comp rx=0 tx=0).stop 2024-04-26T23:01:35.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.374+0000 7fd7edc00640 1 -- 172.21.15.40:0/2394869411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80d08d0 msgr2=0x7fd7e80b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:35.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.374+0000 7fd7edc00640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80d08d0 0x7fd7e80b5110 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd7d000c430 tx=0x7fd7d000c8f0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.374+0000 7fd7edc00640 1 -- 172.21.15.40:0/2394869411 shutdown_connections 2024-04-26T23:01:35.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.374+0000 7fd7edc00640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd7c406c8f0 0x7fd7c406edb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.374+0000 7fd7edc00640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7e80d39f0 0x7fd7e80b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.374+0000 7fd7edc00640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80d08d0 0x7fd7e80b5110 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.374+0000 7fd7edc00640 1 --2- 172.21.15.40:0/2394869411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7e80b6220 0x7fd7e81f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.374+0000 7fd7edc00640 1 -- 172.21.15.40:0/2394869411 >> 172.21.15.40:0/2394869411 conn(0x7fd7e80d4d90 msgr2=0x7fd7e80ce1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:35.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.375+0000 7fd7edc00640 1 -- 172.21.15.40:0/2394869411 shutdown_connections 2024-04-26T23:01:35.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.375+0000 7fd7edc00640 1 -- 172.21.15.40:0/2394869411 wait complete. 2024-04-26T23:01:35.394 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-26T23:01:35.151963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1781,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004933"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487297+0000","last_change":"2024-04-26T23:01:04.726404+0000","last_active":"2024-04-26T23:01:12.487297+0000","last_peered":"2024-04-26T23:01:12.487297+0000","last_clean":"2024-04-26T23:01:12.487297+0000","last_became_active":"2024-04-26T23:01:04.726160+0000","last_became_peered":"2024-04-26T23:01:04.726160+0000","last_unstale":"2024-04-26T23:01:12.487297+0000","last_undegraded":"2024-04-26T23:01:12.487297+0000","last_fullsized":"2024-04-26T23:01:12.487297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:21:22.581263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771048+0000","last_change":"2024-04-26T23:01:05.746741+0000","last_active":"2024-04-26T23:01:11.771048+0000","last_peered":"2024-04-26T23:01:11.771048+0000","last_clean":"2024-04-26T23:01:11.771048+0000","last_became_active":"2024-04-26T23:01:05.743495+0000","last_became_peered":"2024-04-26T23:01:05.743495+0000","last_unstale":"2024-04-26T23:01:11.771048+0000","last_undegraded":"2024-04-26T23:01:11.771048+0000","last_fullsized":"2024-04-26T23:01:11.771048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769795+0000","last_change":"2024-04-26T23:01:10.403533+0000","last_active":"2024-04-26T23:01:11.769795+0000","last_peered":"2024-04-26T23:01:11.769795+0000","last_clean":"2024-04-26T23:01:11.769795+0000","last_became_active":"2024-04-26T23:01:10.402972+0000","last_became_peered":"2024-04-26T23:01:10.402972+0000","last_unstale":"2024-04-26T23:01:11.769795+0000","last_undegraded":"2024-04-26T23:01:11.769795+0000","last_fullsized":"2024-04-26T23:01:11.769795+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:17.916674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771280+0000","last_change":"2024-04-26T23:01:04.715636+0000","last_active":"2024-04-26T23:01:11.771280+0000","last_peered":"2024-04-26T23:01:11.771280+0000","last_clean":"2024-04-26T23:01:11.771280+0000","last_became_active":"2024-04-26T23:01:04.715322+0000","last_became_peered":"2024-04-26T23:01:04.715322+0000","last_unstale":"2024-04-26T23:01:11.771280+0000","last_undegraded":"2024-04-26T23:01:11.771280+0000","last_fullsized":"2024-04-26T23:01:11.771280+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:55.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486825+0000","last_change":"2024-04-26T23:01:05.723149+0000","last_active":"2024-04-26T23:01:12.486825+0000","last_peered":"2024-04-26T23:01:12.486825+0000","last_clean":"2024-04-26T23:01:12.486825+0000","last_became_active":"2024-04-26T23:01:05.722809+0000","last_became_peered":"2024-04-26T23:01:05.722809+0000","last_unstale":"2024-04-26T23:01:12.486825+0000","last_undegraded":"2024-04-26T23:01:12.486825+0000","last_fullsized":"2024-04-26T23:01:12.486825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:26.157667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.881757+0000","last_change":"2024-04-26T23:01:09.805189+0000","last_active":"2024-04-26T23:01:11.881757+0000","last_peered":"2024-04-26T23:01:11.881757+0000","last_clean":"2024-04-26T23:01:11.881757+0000","last_became_active":"2024-04-26T23:01:09.804560+0000","last_became_peered":"2024-04-26T23:01:09.804560+0000","last_unstale":"2024-04-26T23:01:11.881757+0000","last_undegraded":"2024-04-26T23:01:11.881757+0000","last_fullsized":"2024-04-26T23:01:11.881757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:47.817790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775805+0000","last_change":"2024-04-26T23:01:04.734037+0000","last_active":"2024-04-26T23:01:11.775805+0000","last_peered":"2024-04-26T23:01:11.775805+0000","last_clean":"2024-04-26T23:01:11.775805+0000","last_became_active":"2024-04-26T23:01:04.733352+0000","last_became_peered":"2024-04-26T23:01:04.733352+0000","last_unstale":"2024-04-26T23:01:11.775805+0000","last_undegraded":"2024-04-26T23:01:11.775805+0000","last_fullsized":"2024-04-26T23:01:11.775805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:16.098138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484327+0000","last_change":"2024-04-26T23:01:05.738238+0000","last_active":"2024-04-26T23:01:12.484327+0000","last_peered":"2024-04-26T23:01:12.484327+0000","last_clean":"2024-04-26T23:01:12.484327+0000","last_became_active":"2024-04-26T23:01:05.737945+0000","last_became_peered":"2024-04-26T23:01:05.737945+0000","last_unstale":"2024-04-26T23:01:12.484327+0000","last_undegraded":"2024-04-26T23:01:12.484327+0000","last_fullsized":"2024-04-26T23:01:12.484327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:22.679803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.881869+0000","last_change":"2024-04-26T23:01:09.800139+0000","last_active":"2024-04-26T23:01:11.881869+0000","last_peered":"2024-04-26T23:01:11.881869+0000","last_clean":"2024-04-26T23:01:11.881869+0000","last_became_active":"2024-04-26T23:01:09.799214+0000","last_became_peered":"2024-04-26T23:01:09.799214+0000","last_unstale":"2024-04-26T23:01:11.881869+0000","last_undegraded":"2024-04-26T23:01:11.881869+0000","last_fullsized":"2024-04-26T23:01:11.881869+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:46:57.684555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776133+0000","last_change":"2024-04-26T23:01:04.735697+0000","last_active":"2024-04-26T23:01:11.776133+0000","last_peered":"2024-04-26T23:01:11.776133+0000","last_clean":"2024-04-26T23:01:11.776133+0000","last_became_active":"2024-04-26T23:01:04.735327+0000","last_became_peered":"2024-04-26T23:01:04.735327+0000","last_unstale":"2024-04-26T23:01:11.776133+0000","last_undegraded":"2024-04-26T23:01:11.776133+0000","last_fullsized":"2024-04-26T23:01:11.776133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:58.527150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484663+0000","last_change":"2024-04-26T23:01:05.729770+0000","last_active":"2024-04-26T23:01:12.484663+0000","last_peered":"2024-04-26T23:01:12.484663+0000","last_clean":"2024-04-26T23:01:12.484663+0000","last_became_active":"2024-04-26T23:01:05.727922+0000","last_became_peered":"2024-04-26T23:01:05.727922+0000","last_unstale":"2024-04-26T23:01:12.484663+0000","last_undegraded":"2024-04-26T23:01:12.484663+0000","last_fullsized":"2024-04-26T23:01:12.484663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:44:40.404454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882761+0000","last_change":"2024-04-26T23:01:09.804570+0000","last_active":"2024-04-26T23:01:11.882761+0000","last_peered":"2024-04-26T23:01:11.882761+0000","last_clean":"2024-04-26T23:01:11.882761+0000","last_became_active":"2024-04-26T23:01:09.803489+0000","last_became_peered":"2024-04-26T23:01:09.803489+0000","last_unstale":"2024-04-26T23:01:11.882761+0000","last_undegraded":"2024-04-26T23:01:11.882761+0000","last_fullsized":"2024-04-26T23:01:11.882761+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774588+0000","last_change":"2024-04-26T23:01:05.729351+0000","last_active":"2024-04-26T23:01:11.774588+0000","last_peered":"2024-04-26T23:01:11.774588+0000","last_clean":"2024-04-26T23:01:11.774588+0000","last_became_active":"2024-04-26T23:01:05.729024+0000","last_became_peered":"2024-04-26T23:01:05.729024+0000","last_unstale":"2024-04-26T23:01:11.774588+0000","last_undegraded":"2024-04-26T23:01:11.774588+0000","last_fullsized":"2024-04-26T23:01:11.774588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776686+0000","last_change":"2024-04-26T23:01:04.730260+0000","last_active":"2024-04-26T23:01:11.776686+0000","last_peered":"2024-04-26T23:01:11.776686+0000","last_clean":"2024-04-26T23:01:11.776686+0000","last_became_active":"2024-04-26T23:01:04.729209+0000","last_became_peered":"2024-04-26T23:01:04.729209+0000","last_unstale":"2024-04-26T23:01:11.776686+0000","last_undegraded":"2024-04-26T23:01:11.776686+0000","last_fullsized":"2024-04-26T23:01:11.776686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:45.523274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.883045+0000","last_change":"2024-04-26T23:01:09.796341+0000","last_active":"2024-04-26T23:01:11.883045+0000","last_peered":"2024-04-26T23:01:11.883045+0000","last_clean":"2024-04-26T23:01:11.883045+0000","last_became_active":"2024-04-26T23:01:09.795939+0000","last_became_peered":"2024-04-26T23:01:09.795939+0000","last_unstale":"2024-04-26T23:01:11.883045+0000","last_undegraded":"2024-04-26T23:01:11.883045+0000","last_fullsized":"2024-04-26T23:01:11.883045+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:09:55.216454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770614+0000","last_change":"2024-04-26T23:01:05.735100+0000","last_active":"2024-04-26T23:01:11.770614+0000","last_peered":"2024-04-26T23:01:11.770614+0000","last_clean":"2024-04-26T23:01:11.770614+0000","last_became_active":"2024-04-26T23:01:05.734138+0000","last_became_peered":"2024-04-26T23:01:05.734138+0000","last_unstale":"2024-04-26T23:01:11.770614+0000","last_undegraded":"2024-04-26T23:01:11.770614+0000","last_fullsized":"2024-04-26T23:01:11.770614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486466+0000","last_change":"2024-04-26T23:01:04.723885+0000","last_active":"2024-04-26T23:01:12.486466+0000","last_peered":"2024-04-26T23:01:12.486466+0000","last_clean":"2024-04-26T23:01:12.486466+0000","last_became_active":"2024-04-26T23:01:04.723597+0000","last_became_peered":"2024-04-26T23:01:04.723597+0000","last_unstale":"2024-04-26T23:01:12.486466+0000","last_undegraded":"2024-04-26T23:01:12.486466+0000","last_fullsized":"2024-04-26T23:01:12.486466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:45.523274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882064+0000","last_change":"2024-04-26T23:01:09.803721+0000","last_active":"2024-04-26T23:01:11.882064+0000","last_peered":"2024-04-26T23:01:11.882064+0000","last_clean":"2024-04-26T23:01:11.882064+0000","last_became_active":"2024-04-26T23:01:09.803061+0000","last_became_peered":"2024-04-26T23:01:09.803061+0000","last_unstale":"2024-04-26T23:01:11.882064+0000","last_undegraded":"2024-04-26T23:01:11.882064+0000","last_fullsized":"2024-04-26T23:01:11.882064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:41.540020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769949+0000","last_change":"2024-04-26T23:01:05.738344+0000","last_active":"2024-04-26T23:01:11.769949+0000","last_peered":"2024-04-26T23:01:11.769949+0000","last_clean":"2024-04-26T23:01:11.769949+0000","last_became_active":"2024-04-26T23:01:05.738082+0000","last_became_peered":"2024-04-26T23:01:05.738082+0000","last_unstale":"2024-04-26T23:01:11.769949+0000","last_undegraded":"2024-04-26T23:01:11.769949+0000","last_fullsized":"2024-04-26T23:01:11.769949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:26.157667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484208+0000","last_change":"2024-04-26T23:01:04.727252+0000","last_active":"2024-04-26T23:01:12.484208+0000","last_peered":"2024-04-26T23:01:12.484208+0000","last_clean":"2024-04-26T23:01:12.484208+0000","last_became_active":"2024-04-26T23:01:04.726980+0000","last_became_peered":"2024-04-26T23:01:04.726980+0000","last_unstale":"2024-04-26T23:01:12.484208+0000","last_undegraded":"2024-04-26T23:01:12.484208+0000","last_fullsized":"2024-04-26T23:01:12.484208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486354+0000","last_change":"2024-04-26T23:01:10.399273+0000","last_active":"2024-04-26T23:01:12.486354+0000","last_peered":"2024-04-26T23:01:12.486354+0000","last_clean":"2024-04-26T23:01:12.486354+0000","last_became_active":"2024-04-26T23:01:10.397193+0000","last_became_peered":"2024-04-26T23:01:10.397193+0000","last_unstale":"2024-04-26T23:01:12.486354+0000","last_undegraded":"2024-04-26T23:01:12.486354+0000","last_fullsized":"2024-04-26T23:01:12.486354+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882811+0000","last_change":"2024-04-26T23:01:05.734542+0000","last_active":"2024-04-26T23:01:11.882811+0000","last_peered":"2024-04-26T23:01:11.882811+0000","last_clean":"2024-04-26T23:01:11.882811+0000","last_became_active":"2024-04-26T23:01:05.734191+0000","last_became_peered":"2024-04-26T23:01:05.734191+0000","last_unstale":"2024-04-26T23:01:11.882811+0000","last_undegraded":"2024-04-26T23:01:11.882811+0000","last_fullsized":"2024-04-26T23:01:11.882811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:49.765162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774127+0000","last_change":"2024-04-26T23:01:04.732728+0000","last_active":"2024-04-26T23:01:11.774127+0000","last_peered":"2024-04-26T23:01:11.774127+0000","last_clean":"2024-04-26T23:01:11.774127+0000","last_became_active":"2024-04-26T23:01:04.732413+0000","last_became_peered":"2024-04-26T23:01:04.732413+0000","last_unstale":"2024-04-26T23:01:11.774127+0000","last_undegraded":"2024-04-26T23:01:11.774127+0000","last_fullsized":"2024-04-26T23:01:11.774127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882964+0000","last_change":"2024-04-26T23:01:09.800234+0000","last_active":"2024-04-26T23:01:11.882964+0000","last_peered":"2024-04-26T23:01:11.882964+0000","last_clean":"2024-04-26T23:01:11.882964+0000","last_became_active":"2024-04-26T23:01:09.799644+0000","last_became_peered":"2024-04-26T23:01:09.799644+0000","last_unstale":"2024-04-26T23:01:11.882964+0000","last_undegraded":"2024-04-26T23:01:11.882964+0000","last_fullsized":"2024-04-26T23:01:11.882964+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:37:23.404492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486201+0000","last_change":"2024-04-26T23:01:05.735371+0000","last_active":"2024-04-26T23:01:12.486201+0000","last_peered":"2024-04-26T23:01:12.486201+0000","last_clean":"2024-04-26T23:01:12.486201+0000","last_became_active":"2024-04-26T23:01:05.729667+0000","last_became_peered":"2024-04-26T23:01:05.729667+0000","last_unstale":"2024-04-26T23:01:12.486201+0000","last_undegraded":"2024-04-26T23:01:12.486201+0000","last_fullsized":"2024-04-26T23:01:12.486201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778387+0000","last_change":"2024-04-26T23:01:04.726450+0000","last_active":"2024-04-26T23:01:11.778387+0000","last_peered":"2024-04-26T23:01:11.778387+0000","last_clean":"2024-04-26T23:01:11.778387+0000","last_became_active":"2024-04-26T23:01:04.726023+0000","last_became_peered":"2024-04-26T23:01:04.726023+0000","last_unstale":"2024-04-26T23:01:11.778387+0000","last_undegraded":"2024-04-26T23:01:11.778387+0000","last_fullsized":"2024-04-26T23:01:11.778387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:45.523274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.881567+0000","last_change":"2024-04-26T23:01:10.401428+0000","last_active":"2024-04-26T23:01:11.881567+0000","last_peered":"2024-04-26T23:01:11.881567+0000","last_clean":"2024-04-26T23:01:11.881567+0000","last_became_active":"2024-04-26T23:01:10.400621+0000","last_became_peered":"2024-04-26T23:01:10.400621+0000","last_unstale":"2024-04-26T23:01:11.881567+0000","last_undegraded":"2024-04-26T23:01:11.881567+0000","last_fullsized":"2024-04-26T23:01:11.881567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:59.067859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771304+0000","last_change":"2024-04-26T23:01:05.748687+0000","last_active":"2024-04-26T23:01:11.771304+0000","last_peered":"2024-04-26T23:01:11.771304+0000","last_clean":"2024-04-26T23:01:11.771304+0000","last_became_active":"2024-04-26T23:01:05.744038+0000","last_became_peered":"2024-04-26T23:01:05.744038+0000","last_unstale":"2024-04-26T23:01:11.771304+0000","last_undegraded":"2024-04-26T23:01:11.771304+0000","last_fullsized":"2024-04-26T23:01:11.771304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:13.482334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486378+0000","last_change":"2024-04-26T23:01:04.718414+0000","last_active":"2024-04-26T23:01:12.486378+0000","last_peered":"2024-04-26T23:01:12.486378+0000","last_clean":"2024-04-26T23:01:12.486378+0000","last_became_active":"2024-04-26T23:01:04.717764+0000","last_became_peered":"2024-04-26T23:01:04.717764+0000","last_unstale":"2024-04-26T23:01:12.486378+0000","last_undegraded":"2024-04-26T23:01:12.486378+0000","last_fullsized":"2024-04-26T23:01:12.486378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769756+0000","last_change":"2024-04-26T23:01:10.386716+0000","last_active":"2024-04-26T23:01:11.769756+0000","last_peered":"2024-04-26T23:01:11.769756+0000","last_clean":"2024-04-26T23:01:11.769756+0000","last_became_active":"2024-04-26T23:01:10.385392+0000","last_became_peered":"2024-04-26T23:01:10.385392+0000","last_unstale":"2024-04-26T23:01:11.769756+0000","last_undegraded":"2024-04-26T23:01:11.769756+0000","last_fullsized":"2024-04-26T23:01:11.769756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:09:55.216454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485210+0000","last_change":"2024-04-26T23:01:05.728786+0000","last_active":"2024-04-26T23:01:12.485210+0000","last_peered":"2024-04-26T23:01:12.485210+0000","last_clean":"2024-04-26T23:01:12.485210+0000","last_became_active":"2024-04-26T23:01:05.728430+0000","last_became_peered":"2024-04-26T23:01:05.728430+0000","last_unstale":"2024-04-26T23:01:12.485210+0000","last_undegraded":"2024-04-26T23:01:12.485210+0000","last_fullsized":"2024-04-26T23:01:12.485210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776238+0000","last_change":"2024-04-26T23:01:04.734438+0000","last_active":"2024-04-26T23:01:11.776238+0000","last_peered":"2024-04-26T23:01:11.776238+0000","last_clean":"2024-04-26T23:01:11.776238+0000","last_became_active":"2024-04-26T23:01:04.733372+0000","last_became_peered":"2024-04-26T23:01:04.733372+0000","last_unstale":"2024-04-26T23:01:11.776238+0000","last_undegraded":"2024-04-26T23:01:11.776238+0000","last_fullsized":"2024-04-26T23:01:11.776238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:55.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483768+0000","last_change":"2024-04-26T23:01:10.391451+0000","last_active":"2024-04-26T23:01:12.483768+0000","last_peered":"2024-04-26T23:01:12.483768+0000","last_clean":"2024-04-26T23:01:12.483768+0000","last_became_active":"2024-04-26T23:01:10.389477+0000","last_became_peered":"2024-04-26T23:01:10.389477+0000","last_unstale":"2024-04-26T23:01:12.483768+0000","last_undegraded":"2024-04-26T23:01:12.483768+0000","last_fullsized":"2024-04-26T23:01:12.483768+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:46:57.684555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485051+0000","last_change":"2024-04-26T23:01:05.746004+0000","last_active":"2024-04-26T23:01:12.485051+0000","last_peered":"2024-04-26T23:01:12.485051+0000","last_clean":"2024-04-26T23:01:12.485051+0000","last_became_active":"2024-04-26T23:01:05.745576+0000","last_became_peered":"2024-04-26T23:01:05.745576+0000","last_unstale":"2024-04-26T23:01:12.485051+0000","last_undegraded":"2024-04-26T23:01:12.485051+0000","last_fullsized":"2024-04-26T23:01:12.485051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:06.384302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485393+0000","last_change":"2024-04-26T23:01:04.717206+0000","last_active":"2024-04-26T23:01:12.485393+0000","last_peered":"2024-04-26T23:01:12.485393+0000","last_clean":"2024-04-26T23:01:12.485393+0000","last_became_active":"2024-04-26T23:01:04.715952+0000","last_became_peered":"2024-04-26T23:01:04.715952+0000","last_unstale":"2024-04-26T23:01:12.485393+0000","last_undegraded":"2024-04-26T23:01:12.485393+0000","last_fullsized":"2024-04-26T23:01:12.485393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485772+0000","last_change":"2024-04-26T23:01:09.799096+0000","last_active":"2024-04-26T23:01:12.485772+0000","last_peered":"2024-04-26T23:01:12.485772+0000","last_clean":"2024-04-26T23:01:12.485772+0000","last_became_active":"2024-04-26T23:01:09.798586+0000","last_became_peered":"2024-04-26T23:01:09.798586+0000","last_unstale":"2024-04-26T23:01:12.485772+0000","last_undegraded":"2024-04-26T23:01:12.485772+0000","last_fullsized":"2024-04-26T23:01:12.485772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:30.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.883148+0000","last_change":"2024-04-26T23:01:05.732977+0000","last_active":"2024-04-26T23:01:11.883148+0000","last_peered":"2024-04-26T23:01:11.883148+0000","last_clean":"2024-04-26T23:01:11.883148+0000","last_became_active":"2024-04-26T23:01:05.732617+0000","last_became_peered":"2024-04-26T23:01:05.732617+0000","last_unstale":"2024-04-26T23:01:11.883148+0000","last_undegraded":"2024-04-26T23:01:11.883148+0000","last_fullsized":"2024-04-26T23:01:11.883148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:59.540986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484829+0000","last_change":"2024-04-26T23:01:04.730386+0000","last_active":"2024-04-26T23:01:12.484829+0000","last_peered":"2024-04-26T23:01:12.484829+0000","last_clean":"2024-04-26T23:01:12.484829+0000","last_became_active":"2024-04-26T23:01:04.727828+0000","last_became_peered":"2024-04-26T23:01:04.727828+0000","last_unstale":"2024-04-26T23:01:12.484829+0000","last_undegraded":"2024-04-26T23:01:12.484829+0000","last_fullsized":"2024-04-26T23:01:12.484829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485312+0000","last_change":"2024-04-26T23:01:09.805894+0000","last_active":"2024-04-26T23:01:12.485312+0000","last_peered":"2024-04-26T23:01:12.485312+0000","last_clean":"2024-04-26T23:01:12.485312+0000","last_became_active":"2024-04-26T23:01:09.803400+0000","last_became_peered":"2024-04-26T23:01:09.803400+0000","last_unstale":"2024-04-26T23:01:12.485312+0000","last_undegraded":"2024-04-26T23:01:12.485312+0000","last_fullsized":"2024-04-26T23:01:12.485312+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:46:57.684555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484288+0000","last_change":"2024-04-26T23:01:05.722407+0000","last_active":"2024-04-26T23:01:12.484288+0000","last_peered":"2024-04-26T23:01:12.484288+0000","last_clean":"2024-04-26T23:01:12.484288+0000","last_became_active":"2024-04-26T23:01:05.722106+0000","last_became_peered":"2024-04-26T23:01:05.722106+0000","last_unstale":"2024-04-26T23:01:12.484288+0000","last_undegraded":"2024-04-26T23:01:12.484288+0000","last_fullsized":"2024-04-26T23:01:12.484288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487617+0000","last_change":"2024-04-26T23:01:04.712172+0000","last_active":"2024-04-26T23:01:12.487617+0000","last_peered":"2024-04-26T23:01:12.487617+0000","last_clean":"2024-04-26T23:01:12.487617+0000","last_became_active":"2024-04-26T23:01:04.711777+0000","last_became_peered":"2024-04-26T23:01:04.711777+0000","last_unstale":"2024-04-26T23:01:12.487617+0000","last_undegraded":"2024-04-26T23:01:12.487617+0000","last_fullsized":"2024-04-26T23:01:12.487617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486748+0000","last_change":"2024-04-26T23:01:09.808963+0000","last_active":"2024-04-26T23:01:12.486748+0000","last_peered":"2024-04-26T23:01:12.486748+0000","last_clean":"2024-04-26T23:01:12.486748+0000","last_became_active":"2024-04-26T23:01:09.808470+0000","last_became_peered":"2024-04-26T23:01:09.808470+0000","last_unstale":"2024-04-26T23:01:12.486748+0000","last_undegraded":"2024-04-26T23:01:12.486748+0000","last_fullsized":"2024-04-26T23:01:12.486748+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.482634+0000","last_change":"2024-04-26T23:01:05.742128+0000","last_active":"2024-04-26T23:01:12.482634+0000","last_peered":"2024-04-26T23:01:12.482634+0000","last_clean":"2024-04-26T23:01:12.482634+0000","last_became_active":"2024-04-26T23:01:05.735959+0000","last_became_peered":"2024-04-26T23:01:05.735959+0000","last_unstale":"2024-04-26T23:01:12.482634+0000","last_undegraded":"2024-04-26T23:01:12.482634+0000","last_fullsized":"2024-04-26T23:01:12.482634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774374+0000","last_change":"2024-04-26T23:01:04.732093+0000","last_active":"2024-04-26T23:01:11.774374+0000","last_peered":"2024-04-26T23:01:11.774374+0000","last_clean":"2024-04-26T23:01:11.774374+0000","last_became_active":"2024-04-26T23:01:04.731804+0000","last_became_peered":"2024-04-26T23:01:04.731804+0000","last_unstale":"2024-04-26T23:01:11.774374+0000","last_undegraded":"2024-04-26T23:01:11.774374+0000","last_fullsized":"2024-04-26T23:01:11.774374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485713+0000","last_change":"2024-04-26T23:01:10.389206+0000","last_active":"2024-04-26T23:01:12.485713+0000","last_peered":"2024-04-26T23:01:12.485713+0000","last_clean":"2024-04-26T23:01:12.485713+0000","last_became_active":"2024-04-26T23:01:10.384213+0000","last_became_peered":"2024-04-26T23:01:10.384213+0000","last_unstale":"2024-04-26T23:01:12.485713+0000","last_undegraded":"2024-04-26T23:01:12.485713+0000","last_fullsized":"2024-04-26T23:01:12.485713+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770463+0000","last_change":"2024-04-26T23:01:05.744836+0000","last_active":"2024-04-26T23:01:11.770463+0000","last_peered":"2024-04-26T23:01:11.770463+0000","last_clean":"2024-04-26T23:01:11.770463+0000","last_became_active":"2024-04-26T23:01:05.744600+0000","last_became_peered":"2024-04-26T23:01:05.744600+0000","last_unstale":"2024-04-26T23:01:11.770463+0000","last_undegraded":"2024-04-26T23:01:11.770463+0000","last_fullsized":"2024-04-26T23:01:11.770463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484137+0000","last_change":"2024-04-26T23:01:04.729176+0000","last_active":"2024-04-26T23:01:12.484137+0000","last_peered":"2024-04-26T23:01:12.484137+0000","last_clean":"2024-04-26T23:01:12.484137+0000","last_became_active":"2024-04-26T23:01:04.722896+0000","last_became_peered":"2024-04-26T23:01:04.722896+0000","last_unstale":"2024-04-26T23:01:12.484137+0000","last_undegraded":"2024-04-26T23:01:12.484137+0000","last_fullsized":"2024-04-26T23:01:12.484137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:16.585834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":180,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774617+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:01:11.774617+0000","last_peered":"2024-04-26T23:01:11.774617+0000","last_clean":"2024-04-26T23:01:11.774617+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:01:11.774617+0000","last_undegraded":"2024-04-26T23:01:11.774617+0000","last_fullsized":"2024-04-26T23:01:11.774617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777714+0000","last_change":"2024-04-26T23:01:10.402293+0000","last_active":"2024-04-26T23:01:11.777714+0000","last_peered":"2024-04-26T23:01:11.777714+0000","last_clean":"2024-04-26T23:01:11.777714+0000","last_became_active":"2024-04-26T23:01:10.396954+0000","last_became_peered":"2024-04-26T23:01:10.396954+0000","last_unstale":"2024-04-26T23:01:11.777714+0000","last_undegraded":"2024-04-26T23:01:11.777714+0000","last_fullsized":"2024-04-26T23:01:11.777714+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:44.757523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775315+0000","last_change":"2024-04-26T23:01:05.731266+0000","last_active":"2024-04-26T23:01:11.775315+0000","last_peered":"2024-04-26T23:01:11.775315+0000","last_clean":"2024-04-26T23:01:11.775315+0000","last_became_active":"2024-04-26T23:01:05.728530+0000","last_became_peered":"2024-04-26T23:01:05.728530+0000","last_unstale":"2024-04-26T23:01:11.775315+0000","last_undegraded":"2024-04-26T23:01:11.775315+0000","last_fullsized":"2024-04-26T23:01:11.775315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:44:40.404454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.488463+0000","last_change":"2024-04-26T23:01:04.728315+0000","last_active":"2024-04-26T23:01:12.488463+0000","last_peered":"2024-04-26T23:01:12.488463+0000","last_clean":"2024-04-26T23:01:12.488463+0000","last_became_active":"2024-04-26T23:01:04.727609+0000","last_became_peered":"2024-04-26T23:01:04.727609+0000","last_unstale":"2024-04-26T23:01:12.488463+0000","last_undegraded":"2024-04-26T23:01:12.488463+0000","last_fullsized":"2024-04-26T23:01:12.488463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775771+0000","last_change":"2024-04-26T23:01:10.391227+0000","last_active":"2024-04-26T23:01:11.775771+0000","last_peered":"2024-04-26T23:01:11.775771+0000","last_clean":"2024-04-26T23:01:11.775771+0000","last_became_active":"2024-04-26T23:01:10.387715+0000","last_became_peered":"2024-04-26T23:01:10.387715+0000","last_unstale":"2024-04-26T23:01:11.775771+0000","last_undegraded":"2024-04-26T23:01:11.775771+0000","last_fullsized":"2024-04-26T23:01:11.775771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:28.895329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484465+0000","last_change":"2024-04-26T23:01:05.753852+0000","last_active":"2024-04-26T23:01:12.484465+0000","last_peered":"2024-04-26T23:01:12.484465+0000","last_clean":"2024-04-26T23:01:12.484465+0000","last_became_active":"2024-04-26T23:01:05.753607+0000","last_became_peered":"2024-04-26T23:01:05.753607+0000","last_unstale":"2024-04-26T23:01:12.484465+0000","last_undegraded":"2024-04-26T23:01:12.484465+0000","last_fullsized":"2024-04-26T23:01:12.484465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:59:57.750603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485129+0000","last_change":"2024-04-26T23:01:04.713352+0000","last_active":"2024-04-26T23:01:12.485129+0000","last_peered":"2024-04-26T23:01:12.485129+0000","last_clean":"2024-04-26T23:01:12.485129+0000","last_became_active":"2024-04-26T23:01:04.712352+0000","last_became_peered":"2024-04-26T23:01:04.712352+0000","last_unstale":"2024-04-26T23:01:12.485129+0000","last_undegraded":"2024-04-26T23:01:12.485129+0000","last_fullsized":"2024-04-26T23:01:12.485129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:55.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484995+0000","last_change":"2024-04-26T23:01:09.803857+0000","last_active":"2024-04-26T23:01:12.484995+0000","last_peered":"2024-04-26T23:01:12.484995+0000","last_clean":"2024-04-26T23:01:12.484995+0000","last_became_active":"2024-04-26T23:01:09.799482+0000","last_became_peered":"2024-04-26T23:01:09.799482+0000","last_unstale":"2024-04-26T23:01:12.484995+0000","last_undegraded":"2024-04-26T23:01:12.484995+0000","last_fullsized":"2024-04-26T23:01:12.484995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:41.540020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776723+0000","last_change":"2024-04-26T23:01:05.741597+0000","last_active":"2024-04-26T23:01:11.776723+0000","last_peered":"2024-04-26T23:01:11.776723+0000","last_clean":"2024-04-26T23:01:11.776723+0000","last_became_active":"2024-04-26T23:01:05.741308+0000","last_became_peered":"2024-04-26T23:01:05.741308+0000","last_unstale":"2024-04-26T23:01:11.776723+0000","last_undegraded":"2024-04-26T23:01:11.776723+0000","last_fullsized":"2024-04-26T23:01:11.776723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:54.996731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487896+0000","last_change":"2024-04-26T23:01:04.720208+0000","last_active":"2024-04-26T23:01:12.487896+0000","last_peered":"2024-04-26T23:01:12.487896+0000","last_clean":"2024-04-26T23:01:12.487896+0000","last_became_active":"2024-04-26T23:01:04.719867+0000","last_became_peered":"2024-04-26T23:01:04.719867+0000","last_unstale":"2024-04-26T23:01:12.487896+0000","last_undegraded":"2024-04-26T23:01:12.487896+0000","last_fullsized":"2024-04-26T23:01:12.487896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:16.585834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483634+0000","last_change":"2024-04-26T23:01:10.396595+0000","last_active":"2024-04-26T23:01:12.483634+0000","last_peered":"2024-04-26T23:01:12.483634+0000","last_clean":"2024-04-26T23:01:12.483634+0000","last_became_active":"2024-04-26T23:01:10.395189+0000","last_became_peered":"2024-04-26T23:01:10.395189+0000","last_unstale":"2024-04-26T23:01:12.483634+0000","last_undegraded":"2024-04-26T23:01:12.483634+0000","last_fullsized":"2024-04-26T23:01:12.483634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776069+0000","last_change":"2024-04-26T23:01:05.733520+0000","last_active":"2024-04-26T23:01:11.776069+0000","last_peered":"2024-04-26T23:01:11.776069+0000","last_clean":"2024-04-26T23:01:11.776069+0000","last_became_active":"2024-04-26T23:01:05.733170+0000","last_became_peered":"2024-04-26T23:01:05.733170+0000","last_unstale":"2024-04-26T23:01:11.776069+0000","last_undegraded":"2024-04-26T23:01:11.776069+0000","last_fullsized":"2024-04-26T23:01:11.776069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:06.384302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.883050+0000","last_change":"2024-04-26T23:01:04.717930+0000","last_active":"2024-04-26T23:01:11.883050+0000","last_peered":"2024-04-26T23:01:11.883050+0000","last_clean":"2024-04-26T23:01:11.883050+0000","last_became_active":"2024-04-26T23:01:04.717576+0000","last_became_peered":"2024-04-26T23:01:04.717576+0000","last_unstale":"2024-04-26T23:01:11.883050+0000","last_undegraded":"2024-04-26T23:01:11.883050+0000","last_fullsized":"2024-04-26T23:01:11.883050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:11:48.078530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882003+0000","last_change":"2024-04-26T23:01:09.797064+0000","last_active":"2024-04-26T23:01:11.882003+0000","last_peered":"2024-04-26T23:01:11.882003+0000","last_clean":"2024-04-26T23:01:11.882003+0000","last_became_active":"2024-04-26T23:01:09.796589+0000","last_became_peered":"2024-04-26T23:01:09.796589+0000","last_unstale":"2024-04-26T23:01:11.882003+0000","last_undegraded":"2024-04-26T23:01:11.882003+0000","last_fullsized":"2024-04-26T23:01:11.882003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:30.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777169+0000","last_change":"2024-04-26T23:01:05.738519+0000","last_active":"2024-04-26T23:01:11.777169+0000","last_peered":"2024-04-26T23:01:11.777169+0000","last_clean":"2024-04-26T23:01:11.777169+0000","last_became_active":"2024-04-26T23:01:05.738201+0000","last_became_peered":"2024-04-26T23:01:05.738201+0000","last_unstale":"2024-04-26T23:01:11.777169+0000","last_undegraded":"2024-04-26T23:01:11.777169+0000","last_fullsized":"2024-04-26T23:01:11.777169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487284+0000","last_change":"2024-04-26T23:01:04.711562+0000","last_active":"2024-04-26T23:01:12.487284+0000","last_peered":"2024-04-26T23:01:12.487284+0000","last_clean":"2024-04-26T23:01:12.487284+0000","last_became_active":"2024-04-26T23:01:04.708852+0000","last_became_peered":"2024-04-26T23:01:04.708852+0000","last_unstale":"2024-04-26T23:01:12.487284+0000","last_undegraded":"2024-04-26T23:01:12.487284+0000","last_fullsized":"2024-04-26T23:01:12.487284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:21:22.581263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485270+0000","last_change":"2024-04-26T23:01:09.805302+0000","last_active":"2024-04-26T23:01:12.485270+0000","last_peered":"2024-04-26T23:01:12.485270+0000","last_clean":"2024-04-26T23:01:12.485270+0000","last_became_active":"2024-04-26T23:01:09.804878+0000","last_became_peered":"2024-04-26T23:01:09.804878+0000","last_unstale":"2024-04-26T23:01:12.485270+0000","last_undegraded":"2024-04-26T23:01:12.485270+0000","last_fullsized":"2024-04-26T23:01:12.485270+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774404+0000","last_change":"2024-04-26T23:01:05.717338+0000","last_active":"2024-04-26T23:01:11.774404+0000","last_peered":"2024-04-26T23:01:11.774404+0000","last_clean":"2024-04-26T23:01:11.774404+0000","last_became_active":"2024-04-26T23:01:05.716983+0000","last_became_peered":"2024-04-26T23:01:05.716983+0000","last_unstale":"2024-04-26T23:01:11.774404+0000","last_undegraded":"2024-04-26T23:01:11.774404+0000","last_fullsized":"2024-04-26T23:01:11.774404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774791+0000","last_change":"2024-04-26T23:01:04.715584+0000","last_active":"2024-04-26T23:01:11.774791+0000","last_peered":"2024-04-26T23:01:11.774791+0000","last_clean":"2024-04-26T23:01:11.774791+0000","last_became_active":"2024-04-26T23:01:04.715240+0000","last_became_peered":"2024-04-26T23:01:04.715240+0000","last_unstale":"2024-04-26T23:01:11.774791+0000","last_undegraded":"2024-04-26T23:01:11.774791+0000","last_fullsized":"2024-04-26T23:01:11.774791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771265+0000","last_change":"2024-04-26T23:01:10.401101+0000","last_active":"2024-04-26T23:01:11.771265+0000","last_peered":"2024-04-26T23:01:11.771265+0000","last_clean":"2024-04-26T23:01:11.771265+0000","last_became_active":"2024-04-26T23:01:10.396052+0000","last_became_peered":"2024-04-26T23:01:10.396052+0000","last_unstale":"2024-04-26T23:01:11.771265+0000","last_undegraded":"2024-04-26T23:01:11.771265+0000","last_fullsized":"2024-04-26T23:01:11.771265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:15.222701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776466+0000","last_change":"2024-04-26T23:01:05.732489+0000","last_active":"2024-04-26T23:01:11.776466+0000","last_peered":"2024-04-26T23:01:11.776466+0000","last_clean":"2024-04-26T23:01:11.776466+0000","last_became_active":"2024-04-26T23:01:05.732228+0000","last_became_peered":"2024-04-26T23:01:05.732228+0000","last_unstale":"2024-04-26T23:01:11.776466+0000","last_undegraded":"2024-04-26T23:01:11.776466+0000","last_fullsized":"2024-04-26T23:01:11.776466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:22:15.301548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484809+0000","last_change":"2024-04-26T23:01:04.731558+0000","last_active":"2024-04-26T23:01:12.484809+0000","last_peered":"2024-04-26T23:01:12.484809+0000","last_clean":"2024-04-26T23:01:12.484809+0000","last_became_active":"2024-04-26T23:01:04.731032+0000","last_became_peered":"2024-04-26T23:01:04.731032+0000","last_unstale":"2024-04-26T23:01:12.484809+0000","last_undegraded":"2024-04-26T23:01:12.484809+0000","last_fullsized":"2024-04-26T23:01:12.484809+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770866+0000","last_change":"2024-04-26T23:01:10.386434+0000","last_active":"2024-04-26T23:01:11.770866+0000","last_peered":"2024-04-26T23:01:11.770866+0000","last_clean":"2024-04-26T23:01:11.770866+0000","last_became_active":"2024-04-26T23:01:10.385162+0000","last_became_peered":"2024-04-26T23:01:10.385162+0000","last_unstale":"2024-04-26T23:01:11.770866+0000","last_undegraded":"2024-04-26T23:01:11.770866+0000","last_fullsized":"2024-04-26T23:01:11.770866+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:19.562651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483154+0000","last_change":"2024-04-26T23:01:05.740528+0000","last_active":"2024-04-26T23:01:12.483154+0000","last_peered":"2024-04-26T23:01:12.483154+0000","last_clean":"2024-04-26T23:01:12.483154+0000","last_became_active":"2024-04-26T23:01:05.740211+0000","last_became_peered":"2024-04-26T23:01:05.740211+0000","last_unstale":"2024-04-26T23:01:12.483154+0000","last_undegraded":"2024-04-26T23:01:12.483154+0000","last_fullsized":"2024-04-26T23:01:12.483154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484416+0000","last_change":"2024-04-26T23:01:04.712658+0000","last_active":"2024-04-26T23:01:12.484416+0000","last_peered":"2024-04-26T23:01:12.484416+0000","last_clean":"2024-04-26T23:01:12.484416+0000","last_became_active":"2024-04-26T23:01:04.712325+0000","last_became_peered":"2024-04-26T23:01:04.712325+0000","last_unstale":"2024-04-26T23:01:12.484416+0000","last_undegraded":"2024-04-26T23:01:12.484416+0000","last_fullsized":"2024-04-26T23:01:12.484416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:52.271211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485257+0000","last_change":"2024-04-26T23:01:10.385938+0000","last_active":"2024-04-26T23:01:12.485257+0000","last_peered":"2024-04-26T23:01:12.485257+0000","last_clean":"2024-04-26T23:01:12.485257+0000","last_became_active":"2024-04-26T23:01:10.384600+0000","last_became_peered":"2024-04-26T23:01:10.384600+0000","last_unstale":"2024-04-26T23:01:12.485257+0000","last_undegraded":"2024-04-26T23:01:12.485257+0000","last_fullsized":"2024-04-26T23:01:12.485257+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775962+0000","last_change":"2024-04-26T23:01:05.744208+0000","last_active":"2024-04-26T23:01:11.775962+0000","last_peered":"2024-04-26T23:01:11.775962+0000","last_clean":"2024-04-26T23:01:11.775962+0000","last_became_active":"2024-04-26T23:01:05.743897+0000","last_became_peered":"2024-04-26T23:01:05.743897+0000","last_unstale":"2024-04-26T23:01:11.775962+0000","last_undegraded":"2024-04-26T23:01:11.775962+0000","last_fullsized":"2024-04-26T23:01:11.775962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:22.679803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:25.136494+0000","last_change":"2024-04-26T23:01:04.723670+0000","last_active":"2024-04-26T23:01:25.136494+0000","last_peered":"2024-04-26T23:01:25.136494+0000","last_clean":"2024-04-26T23:01:25.136494+0000","last_became_active":"2024-04-26T23:01:04.723361+0000","last_became_peered":"2024-04-26T23:01:04.723361+0000","last_unstale":"2024-04-26T23:01:25.136494+0000","last_undegraded":"2024-04-26T23:01:25.136494+0000","last_fullsized":"2024-04-26T23:01:25.136494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:58.527150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485782+0000","last_change":"2024-04-26T23:01:09.799732+0000","last_active":"2024-04-26T23:01:12.485782+0000","last_peered":"2024-04-26T23:01:12.485782+0000","last_clean":"2024-04-26T23:01:12.485782+0000","last_became_active":"2024-04-26T23:01:09.797973+0000","last_became_peered":"2024-04-26T23:01:09.797973+0000","last_unstale":"2024-04-26T23:01:12.485782+0000","last_undegraded":"2024-04-26T23:01:12.485782+0000","last_fullsized":"2024-04-26T23:01:12.485782+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:41.540020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882483+0000","last_change":"2024-04-26T23:01:05.732675+0000","last_active":"2024-04-26T23:01:11.882483+0000","last_peered":"2024-04-26T23:01:11.882483+0000","last_clean":"2024-04-26T23:01:11.882483+0000","last_became_active":"2024-04-26T23:01:05.732334+0000","last_became_peered":"2024-04-26T23:01:05.732334+0000","last_unstale":"2024-04-26T23:01:11.882483+0000","last_undegraded":"2024-04-26T23:01:11.882483+0000","last_fullsized":"2024-04-26T23:01:11.882483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:46.537111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484886+0000","last_change":"2024-04-26T23:01:04.731957+0000","last_active":"2024-04-26T23:01:12.484886+0000","last_peered":"2024-04-26T23:01:12.484886+0000","last_clean":"2024-04-26T23:01:12.484886+0000","last_became_active":"2024-04-26T23:01:04.731688+0000","last_became_peered":"2024-04-26T23:01:04.731688+0000","last_unstale":"2024-04-26T23:01:12.484886+0000","last_undegraded":"2024-04-26T23:01:12.484886+0000","last_fullsized":"2024-04-26T23:01:12.484886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:16.098138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882537+0000","last_change":"2024-04-26T23:01:09.793585+0000","last_active":"2024-04-26T23:01:11.882537+0000","last_peered":"2024-04-26T23:01:11.882537+0000","last_clean":"2024-04-26T23:01:11.882537+0000","last_became_active":"2024-04-26T23:01:09.793177+0000","last_became_peered":"2024-04-26T23:01:09.793177+0000","last_unstale":"2024-04-26T23:01:11.882537+0000","last_undegraded":"2024-04-26T23:01:11.882537+0000","last_fullsized":"2024-04-26T23:01:11.882537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487118+0000","last_change":"2024-04-26T23:01:05.723191+0000","last_active":"2024-04-26T23:01:12.487118+0000","last_peered":"2024-04-26T23:01:12.487118+0000","last_clean":"2024-04-26T23:01:12.487118+0000","last_became_active":"2024-04-26T23:01:05.722903+0000","last_became_peered":"2024-04-26T23:01:05.722903+0000","last_unstale":"2024-04-26T23:01:12.487118+0000","last_undegraded":"2024-04-26T23:01:12.487118+0000","last_fullsized":"2024-04-26T23:01:12.487118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484699+0000","last_change":"2024-04-26T23:01:04.729093+0000","last_active":"2024-04-26T23:01:12.484699+0000","last_peered":"2024-04-26T23:01:12.484699+0000","last_clean":"2024-04-26T23:01:12.484699+0000","last_became_active":"2024-04-26T23:01:04.728125+0000","last_became_peered":"2024-04-26T23:01:04.728125+0000","last_unstale":"2024-04-26T23:01:12.484699+0000","last_undegraded":"2024-04-26T23:01:12.484699+0000","last_fullsized":"2024-04-26T23:01:12.484699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769989+0000","last_change":"2024-04-26T23:01:09.802623+0000","last_active":"2024-04-26T23:01:11.769989+0000","last_peered":"2024-04-26T23:01:11.769989+0000","last_clean":"2024-04-26T23:01:11.769989+0000","last_became_active":"2024-04-26T23:01:09.798323+0000","last_became_peered":"2024-04-26T23:01:09.798323+0000","last_unstale":"2024-04-26T23:01:11.769989+0000","last_undegraded":"2024-04-26T23:01:11.769989+0000","last_fullsized":"2024-04-26T23:01:11.769989+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:56.662302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484537+0000","last_change":"2024-04-26T23:01:05.727245+0000","last_active":"2024-04-26T23:01:12.484537+0000","last_peered":"2024-04-26T23:01:12.484537+0000","last_clean":"2024-04-26T23:01:12.484537+0000","last_became_active":"2024-04-26T23:01:05.726786+0000","last_became_peered":"2024-04-26T23:01:05.726786+0000","last_unstale":"2024-04-26T23:01:12.484537+0000","last_undegraded":"2024-04-26T23:01:12.484537+0000","last_fullsized":"2024-04-26T23:01:12.484537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:24.824202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778010+0000","last_change":"2024-04-26T23:01:04.732475+0000","last_active":"2024-04-26T23:01:11.778010+0000","last_peered":"2024-04-26T23:01:11.778010+0000","last_clean":"2024-04-26T23:01:11.778010+0000","last_became_active":"2024-04-26T23:01:04.732185+0000","last_became_peered":"2024-04-26T23:01:04.732185+0000","last_unstale":"2024-04-26T23:01:11.778010+0000","last_undegraded":"2024-04-26T23:01:11.778010+0000","last_fullsized":"2024-04-26T23:01:11.778010+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770229+0000","last_change":"2024-04-26T23:01:10.396108+0000","last_active":"2024-04-26T23:01:11.770229+0000","last_peered":"2024-04-26T23:01:11.770229+0000","last_clean":"2024-04-26T23:01:11.770229+0000","last_became_active":"2024-04-26T23:01:10.394701+0000","last_became_peered":"2024-04-26T23:01:10.394701+0000","last_unstale":"2024-04-26T23:01:11.770229+0000","last_undegraded":"2024-04-26T23:01:11.770229+0000","last_fullsized":"2024-04-26T23:01:11.770229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:00:11.874893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484827+0000","last_change":"2024-04-26T23:01:10.396863+0000","last_active":"2024-04-26T23:01:12.484827+0000","last_peered":"2024-04-26T23:01:12.484827+0000","last_clean":"2024-04-26T23:01:12.484827+0000","last_became_active":"2024-04-26T23:01:10.395914+0000","last_became_peered":"2024-04-26T23:01:10.395914+0000","last_unstale":"2024-04-26T23:01:12.484827+0000","last_undegraded":"2024-04-26T23:01:12.484827+0000","last_fullsized":"2024-04-26T23:01:12.484827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:30.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483687+0000","last_change":"2024-04-26T23:01:05.749537+0000","last_active":"2024-04-26T23:01:12.483687+0000","last_peered":"2024-04-26T23:01:12.483687+0000","last_clean":"2024-04-26T23:01:12.483687+0000","last_became_active":"2024-04-26T23:01:05.749254+0000","last_became_peered":"2024-04-26T23:01:05.749254+0000","last_unstale":"2024-04-26T23:01:12.483687+0000","last_undegraded":"2024-04-26T23:01:12.483687+0000","last_fullsized":"2024-04-26T23:01:12.483687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483804+0000","last_change":"2024-04-26T23:01:04.730472+0000","last_active":"2024-04-26T23:01:12.483804+0000","last_peered":"2024-04-26T23:01:12.483804+0000","last_clean":"2024-04-26T23:01:12.483804+0000","last_became_active":"2024-04-26T23:01:04.729294+0000","last_became_peered":"2024-04-26T23:01:04.729294+0000","last_unstale":"2024-04-26T23:01:12.483804+0000","last_undegraded":"2024-04-26T23:01:12.483804+0000","last_fullsized":"2024-04-26T23:01:12.483804+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:58.527150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778171+0000","last_change":"2024-04-26T23:01:09.800700+0000","last_active":"2024-04-26T23:01:11.778171+0000","last_peered":"2024-04-26T23:01:11.778171+0000","last_clean":"2024-04-26T23:01:11.778171+0000","last_became_active":"2024-04-26T23:01:09.799085+0000","last_became_peered":"2024-04-26T23:01:09.799085+0000","last_unstale":"2024-04-26T23:01:11.778171+0000","last_undegraded":"2024-04-26T23:01:11.778171+0000","last_fullsized":"2024-04-26T23:01:11.778171+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:59.067859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769853+0000","last_change":"2024-04-26T23:01:05.737423+0000","last_active":"2024-04-26T23:01:11.769853+0000","last_peered":"2024-04-26T23:01:11.769853+0000","last_clean":"2024-04-26T23:01:11.769853+0000","last_became_active":"2024-04-26T23:01:05.736881+0000","last_became_peered":"2024-04-26T23:01:05.736881+0000","last_unstale":"2024-04-26T23:01:11.769853+0000","last_undegraded":"2024-04-26T23:01:11.769853+0000","last_fullsized":"2024-04-26T23:01:11.769853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:35:08.020426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770314+0000","last_change":"2024-04-26T23:01:04.723415+0000","last_active":"2024-04-26T23:01:11.770314+0000","last_peered":"2024-04-26T23:01:11.770314+0000","last_clean":"2024-04-26T23:01:11.770314+0000","last_became_active":"2024-04-26T23:01:04.723109+0000","last_became_peered":"2024-04-26T23:01:04.723109+0000","last_unstale":"2024-04-26T23:01:11.770314+0000","last_undegraded":"2024-04-26T23:01:11.770314+0000","last_fullsized":"2024-04-26T23:01:11.770314+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:52.271211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775686+0000","last_change":"2024-04-26T23:01:04.712783+0000","last_active":"2024-04-26T23:01:11.775686+0000","last_peered":"2024-04-26T23:01:11.775686+0000","last_clean":"2024-04-26T23:01:11.775686+0000","last_became_active":"2024-04-26T23:01:04.712436+0000","last_became_peered":"2024-04-26T23:01:04.712436+0000","last_unstale":"2024-04-26T23:01:11.775686+0000","last_undegraded":"2024-04-26T23:01:11.775686+0000","last_fullsized":"2024-04-26T23:01:11.775686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484177+0000","last_change":"2024-04-26T23:01:05.742581+0000","last_active":"2024-04-26T23:01:12.484177+0000","last_peered":"2024-04-26T23:01:12.484177+0000","last_clean":"2024-04-26T23:01:12.484177+0000","last_became_active":"2024-04-26T23:01:05.740918+0000","last_became_peered":"2024-04-26T23:01:05.740918+0000","last_unstale":"2024-04-26T23:01:12.484177+0000","last_undegraded":"2024-04-26T23:01:12.484177+0000","last_fullsized":"2024-04-26T23:01:12.484177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:35:08.020426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.488392+0000","last_change":"2024-04-26T23:01:09.798160+0000","last_active":"2024-04-26T23:01:12.488392+0000","last_peered":"2024-04-26T23:01:12.488392+0000","last_clean":"2024-04-26T23:01:12.488392+0000","last_became_active":"2024-04-26T23:01:09.797746+0000","last_became_peered":"2024-04-26T23:01:09.797746+0000","last_unstale":"2024-04-26T23:01:12.488392+0000","last_undegraded":"2024-04-26T23:01:12.488392+0000","last_fullsized":"2024-04-26T23:01:12.488392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:11.186360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770724+0000","last_change":"2024-04-26T23:01:04.723482+0000","last_active":"2024-04-26T23:01:11.770724+0000","last_peered":"2024-04-26T23:01:11.770724+0000","last_clean":"2024-04-26T23:01:11.770724+0000","last_became_active":"2024-04-26T23:01:04.722816+0000","last_became_peered":"2024-04-26T23:01:04.722816+0000","last_unstale":"2024-04-26T23:01:11.770724+0000","last_undegraded":"2024-04-26T23:01:11.770724+0000","last_fullsized":"2024-04-26T23:01:11.770724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882061+0000","last_change":"2024-04-26T23:01:05.745436+0000","last_active":"2024-04-26T23:01:11.882061+0000","last_peered":"2024-04-26T23:01:11.882061+0000","last_clean":"2024-04-26T23:01:11.882061+0000","last_became_active":"2024-04-26T23:01:05.745148+0000","last_became_peered":"2024-04-26T23:01:05.745148+0000","last_unstale":"2024-04-26T23:01:11.882061+0000","last_undegraded":"2024-04-26T23:01:11.882061+0000","last_fullsized":"2024-04-26T23:01:11.882061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:17.111974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771111+0000","last_change":"2024-04-26T23:01:09.795297+0000","last_active":"2024-04-26T23:01:11.771111+0000","last_peered":"2024-04-26T23:01:11.771111+0000","last_clean":"2024-04-26T23:01:11.771111+0000","last_became_active":"2024-04-26T23:01:09.794873+0000","last_became_peered":"2024-04-26T23:01:09.794873+0000","last_unstale":"2024-04-26T23:01:11.771111+0000","last_undegraded":"2024-04-26T23:01:11.771111+0000","last_fullsized":"2024-04-26T23:01:11.771111+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:16:59.104896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.773782+0000","last_change":"2024-04-26T23:01:05.713935+0000","last_active":"2024-04-26T23:01:11.773782+0000","last_peered":"2024-04-26T23:01:11.773782+0000","last_clean":"2024-04-26T23:01:11.773782+0000","last_became_active":"2024-04-26T23:01:05.713606+0000","last_became_peered":"2024-04-26T23:01:05.713606+0000","last_unstale":"2024-04-26T23:01:11.773782+0000","last_undegraded":"2024-04-26T23:01:11.773782+0000","last_fullsized":"2024-04-26T23:01:11.773782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:26.157667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487634+0000","last_change":"2024-04-26T23:01:05.722889+0000","last_active":"2024-04-26T23:01:12.487634+0000","last_peered":"2024-04-26T23:01:12.487634+0000","last_clean":"2024-04-26T23:01:12.487634+0000","last_became_active":"2024-04-26T23:01:05.721634+0000","last_became_peered":"2024-04-26T23:01:05.721634+0000","last_unstale":"2024-04-26T23:01:12.487634+0000","last_undegraded":"2024-04-26T23:01:12.487634+0000","last_fullsized":"2024-04-26T23:01:12.487634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:46.537111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484476+0000","last_change":"2024-04-26T23:01:05.749096+0000","last_active":"2024-04-26T23:01:12.484476+0000","last_peered":"2024-04-26T23:01:12.484476+0000","last_clean":"2024-04-26T23:01:12.484476+0000","last_became_active":"2024-04-26T23:01:05.747770+0000","last_became_peered":"2024-04-26T23:01:05.747770+0000","last_unstale":"2024-04-26T23:01:12.484476+0000","last_undegraded":"2024-04-26T23:01:12.484476+0000","last_fullsized":"2024-04-26T23:01:12.484476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:57.106236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483913+0000","last_change":"2024-04-26T23:01:05.747157+0000","last_active":"2024-04-26T23:01:12.483913+0000","last_peered":"2024-04-26T23:01:12.483913+0000","last_clean":"2024-04-26T23:01:12.483913+0000","last_became_active":"2024-04-26T23:01:05.746805+0000","last_became_peered":"2024-04-26T23:01:05.746805+0000","last_unstale":"2024-04-26T23:01:12.483913+0000","last_undegraded":"2024-04-26T23:01:12.483913+0000","last_fullsized":"2024-04-26T23:01:12.483913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:07.115233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770959+0000","last_change":"2024-04-26T23:01:05.747433+0000","last_active":"2024-04-26T23:01:11.770959+0000","last_peered":"2024-04-26T23:01:11.770959+0000","last_clean":"2024-04-26T23:01:11.770959+0000","last_became_active":"2024-04-26T23:01:05.747108+0000","last_became_peered":"2024-04-26T23:01:05.747108+0000","last_unstale":"2024-04-26T23:01:11.770959+0000","last_undegraded":"2024-04-26T23:01:11.770959+0000","last_fullsized":"2024-04-26T23:01:11.770959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:57.106236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484633+0000","last_change":"2024-04-26T23:01:05.753692+0000","last_active":"2024-04-26T23:01:12.484633+0000","last_peered":"2024-04-26T23:01:12.484633+0000","last_clean":"2024-04-26T23:01:12.484633+0000","last_became_active":"2024-04-26T23:01:05.752474+0000","last_became_peered":"2024-04-26T23:01:05.752474+0000","last_unstale":"2024-04-26T23:01:12.484633+0000","last_undegraded":"2024-04-26T23:01:12.484633+0000","last_fullsized":"2024-04-26T23:01:12.484633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778393+0000","last_change":"2024-04-26T23:01:05.731062+0000","last_active":"2024-04-26T23:01:11.778393+0000","last_peered":"2024-04-26T23:01:11.778393+0000","last_clean":"2024-04-26T23:01:11.778393+0000","last_became_active":"2024-04-26T23:01:05.727960+0000","last_became_peered":"2024-04-26T23:01:05.727960+0000","last_unstale":"2024-04-26T23:01:11.778393+0000","last_undegraded":"2024-04-26T23:01:11.778393+0000","last_fullsized":"2024-04-26T23:01:11.778393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771192+0000","last_change":"2024-04-26T23:01:05.750232+0000","last_active":"2024-04-26T23:01:11.771192+0000","last_peered":"2024-04-26T23:01:11.771192+0000","last_clean":"2024-04-26T23:01:11.771192+0000","last_became_active":"2024-04-26T23:01:05.749885+0000","last_became_peered":"2024-04-26T23:01:05.749885+0000","last_unstale":"2024-04-26T23:01:11.771192+0000","last_undegraded":"2024-04-26T23:01:11.771192+0000","last_fullsized":"2024-04-26T23:01:11.771192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483726+0000","last_change":"2024-04-26T23:01:05.749914+0000","last_active":"2024-04-26T23:01:12.483726+0000","last_peered":"2024-04-26T23:01:12.483726+0000","last_clean":"2024-04-26T23:01:12.483726+0000","last_became_active":"2024-04-26T23:01:05.745797+0000","last_became_peered":"2024-04-26T23:01:05.745797+0000","last_unstale":"2024-04-26T23:01:12.483726+0000","last_undegraded":"2024-04-26T23:01:12.483726+0000","last_fullsized":"2024-04-26T23:01:12.483726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770217+0000","last_change":"2024-04-26T23:01:05.750181+0000","last_active":"2024-04-26T23:01:11.770217+0000","last_peered":"2024-04-26T23:01:11.770217+0000","last_clean":"2024-04-26T23:01:11.770217+0000","last_became_active":"2024-04-26T23:01:05.749810+0000","last_became_peered":"2024-04-26T23:01:05.749810+0000","last_unstale":"2024-04-26T23:01:11.770217+0000","last_undegraded":"2024-04-26T23:01:11.770217+0000","last_fullsized":"2024-04-26T23:01:11.770217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484216+0000","last_change":"2024-04-26T23:01:05.730857+0000","last_active":"2024-04-26T23:01:12.484216+0000","last_peered":"2024-04-26T23:01:12.484216+0000","last_clean":"2024-04-26T23:01:12.484216+0000","last_became_active":"2024-04-26T23:01:05.730543+0000","last_became_peered":"2024-04-26T23:01:05.730543+0000","last_unstale":"2024-04-26T23:01:12.484216+0000","last_undegraded":"2024-04-26T23:01:12.484216+0000","last_fullsized":"2024-04-26T23:01:12.484216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.482980+0000","last_change":"2024-04-26T23:01:05.741407+0000","last_active":"2024-04-26T23:01:12.482980+0000","last_peered":"2024-04-26T23:01:12.482980+0000","last_clean":"2024-04-26T23:01:12.482980+0000","last_became_active":"2024-04-26T23:01:05.740354+0000","last_became_peered":"2024-04-26T23:01:05.740354+0000","last_unstale":"2024-04-26T23:01:12.482980+0000","last_undegraded":"2024-04-26T23:01:12.482980+0000","last_fullsized":"2024-04-26T23:01:12.482980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:24:06.318595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777299+0000","last_change":"2024-04-26T23:01:05.730302+0000","last_active":"2024-04-26T23:01:11.777299+0000","last_peered":"2024-04-26T23:01:11.777299+0000","last_clean":"2024-04-26T23:01:11.777299+0000","last_became_active":"2024-04-26T23:01:05.729149+0000","last_became_peered":"2024-04-26T23:01:05.729149+0000","last_unstale":"2024-04-26T23:01:11.777299+0000","last_undegraded":"2024-04-26T23:01:11.777299+0000","last_fullsized":"2024-04-26T23:01:11.777299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776184+0000","last_change":"2024-04-26T23:01:05.731828+0000","last_active":"2024-04-26T23:01:11.776184+0000","last_peered":"2024-04-26T23:01:11.776184+0000","last_clean":"2024-04-26T23:01:11.776184+0000","last_became_active":"2024-04-26T23:01:05.731537+0000","last_became_peered":"2024-04-26T23:01:05.731537+0000","last_unstale":"2024-04-26T23:01:11.776184+0000","last_undegraded":"2024-04-26T23:01:11.776184+0000","last_fullsized":"2024-04-26T23:01:11.776184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:59:57.750603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487750+0000","last_change":"2024-04-26T23:01:05.726729+0000","last_active":"2024-04-26T23:01:12.487750+0000","last_peered":"2024-04-26T23:01:12.487750+0000","last_clean":"2024-04-26T23:01:12.487750+0000","last_became_active":"2024-04-26T23:01:05.726364+0000","last_became_peered":"2024-04-26T23:01:05.726364+0000","last_unstale":"2024-04-26T23:01:12.487750+0000","last_undegraded":"2024-04-26T23:01:12.487750+0000","last_fullsized":"2024-04-26T23:01:12.487750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:17.111974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485129+0000","last_change":"2024-04-26T23:01:05.728806+0000","last_active":"2024-04-26T23:01:12.485129+0000","last_peered":"2024-04-26T23:01:12.485129+0000","last_clean":"2024-04-26T23:01:12.485129+0000","last_became_active":"2024-04-26T23:01:05.728418+0000","last_became_peered":"2024-04-26T23:01:05.728418+0000","last_unstale":"2024-04-26T23:01:12.485129+0000","last_undegraded":"2024-04-26T23:01:12.485129+0000","last_fullsized":"2024-04-26T23:01:12.485129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:40.686396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483463+0000","last_change":"2024-04-26T23:01:05.721739+0000","last_active":"2024-04-26T23:01:12.483463+0000","last_peered":"2024-04-26T23:01:12.483463+0000","last_clean":"2024-04-26T23:01:12.483463+0000","last_became_active":"2024-04-26T23:01:05.721409+0000","last_became_peered":"2024-04-26T23:01:05.721409+0000","last_unstale":"2024-04-26T23:01:12.483463+0000","last_undegraded":"2024-04-26T23:01:12.483463+0000","last_fullsized":"2024-04-26T23:01:12.483463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483935+0000","last_change":"2024-04-26T23:01:05.739575+0000","last_active":"2024-04-26T23:01:12.483935+0000","last_peered":"2024-04-26T23:01:12.483935+0000","last_clean":"2024-04-26T23:01:12.483935+0000","last_became_active":"2024-04-26T23:01:05.734760+0000","last_became_peered":"2024-04-26T23:01:05.734760+0000","last_unstale":"2024-04-26T23:01:12.483935+0000","last_undegraded":"2024-04-26T23:01:12.483935+0000","last_fullsized":"2024-04-26T23:01:12.483935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483111+0000","last_change":"2024-04-26T23:01:05.732699+0000","last_active":"2024-04-26T23:01:12.483111+0000","last_peered":"2024-04-26T23:01:12.483111+0000","last_clean":"2024-04-26T23:01:12.483111+0000","last_became_active":"2024-04-26T23:01:05.732371+0000","last_became_peered":"2024-04-26T23:01:05.732371+0000","last_unstale":"2024-04-26T23:01:12.483111+0000","last_undegraded":"2024-04-26T23:01:12.483111+0000","last_fullsized":"2024-04-26T23:01:12.483111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770589+0000","last_change":"2024-04-26T23:01:05.745445+0000","last_active":"2024-04-26T23:01:11.770589+0000","last_peered":"2024-04-26T23:01:11.770589+0000","last_clean":"2024-04-26T23:01:11.770589+0000","last_became_active":"2024-04-26T23:01:05.742977+0000","last_became_peered":"2024-04-26T23:01:05.742977+0000","last_unstale":"2024-04-26T23:01:11.770589+0000","last_undegraded":"2024-04-26T23:01:11.770589+0000","last_fullsized":"2024-04-26T23:01:11.770589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882409+0000","last_change":"2024-04-26T23:01:05.739107+0000","last_active":"2024-04-26T23:01:11.882409+0000","last_peered":"2024-04-26T23:01:11.882409+0000","last_clean":"2024-04-26T23:01:11.882409+0000","last_became_active":"2024-04-26T23:01:05.738786+0000","last_became_peered":"2024-04-26T23:01:05.738786+0000","last_unstale":"2024-04-26T23:01:11.882409+0000","last_undegraded":"2024-04-26T23:01:11.882409+0000","last_fullsized":"2024-04-26T23:01:11.882409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:58.394848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778055+0000","last_change":"2024-04-26T23:01:05.710270+0000","last_active":"2024-04-26T23:01:11.778055+0000","last_peered":"2024-04-26T23:01:11.778055+0000","last_clean":"2024-04-26T23:01:11.778055+0000","last_became_active":"2024-04-26T23:01:05.709943+0000","last_became_peered":"2024-04-26T23:01:05.709943+0000","last_unstale":"2024-04-26T23:01:11.778055+0000","last_undegraded":"2024-04-26T23:01:11.778055+0000","last_fullsized":"2024-04-26T23:01:11.778055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778339+0000","last_change":"2024-04-26T23:01:05.717127+0000","last_active":"2024-04-26T23:01:11.778339+0000","last_peered":"2024-04-26T23:01:11.778339+0000","last_clean":"2024-04-26T23:01:11.778339+0000","last_became_active":"2024-04-26T23:01:05.716647+0000","last_became_peered":"2024-04-26T23:01:05.716647+0000","last_unstale":"2024-04-26T23:01:11.778339+0000","last_undegraded":"2024-04-26T23:01:11.778339+0000","last_fullsized":"2024-04-26T23:01:11.778339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776723+0000","last_change":"2024-04-26T23:01:05.730733+0000","last_active":"2024-04-26T23:01:11.776723+0000","last_peered":"2024-04-26T23:01:11.776723+0000","last_clean":"2024-04-26T23:01:11.776723+0000","last_became_active":"2024-04-26T23:01:05.727115+0000","last_became_peered":"2024-04-26T23:01:05.727115+0000","last_unstale":"2024-04-26T23:01:11.776723+0000","last_undegraded":"2024-04-26T23:01:11.776723+0000","last_fullsized":"2024-04-26T23:01:11.776723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:40.686396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769732+0000","last_change":"2024-04-26T23:01:05.737620+0000","last_active":"2024-04-26T23:01:11.769732+0000","last_peered":"2024-04-26T23:01:11.769732+0000","last_clean":"2024-04-26T23:01:11.769732+0000","last_became_active":"2024-04-26T23:01:05.737220+0000","last_became_peered":"2024-04-26T23:01:05.737220+0000","last_unstale":"2024-04-26T23:01:11.769732+0000","last_undegraded":"2024-04-26T23:01:11.769732+0000","last_fullsized":"2024-04-26T23:01:11.769732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:07.115233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777054+0000","last_change":"2024-04-26T23:01:05.731417+0000","last_active":"2024-04-26T23:01:11.777054+0000","last_peered":"2024-04-26T23:01:11.777054+0000","last_clean":"2024-04-26T23:01:11.777054+0000","last_became_active":"2024-04-26T23:01:05.730556+0000","last_became_peered":"2024-04-26T23:01:05.730556+0000","last_unstale":"2024-04-26T23:01:11.777054+0000","last_undegraded":"2024-04-26T23:01:11.777054+0000","last_fullsized":"2024-04-26T23:01:11.777054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:13.482334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483844+0000","last_change":"2024-04-26T23:01:05.721675+0000","last_active":"2024-04-26T23:01:12.483844+0000","last_peered":"2024-04-26T23:01:12.483844+0000","last_clean":"2024-04-26T23:01:12.483844+0000","last_became_active":"2024-04-26T23:01:05.721363+0000","last_became_peered":"2024-04-26T23:01:05.721363+0000","last_unstale":"2024-04-26T23:01:12.483844+0000","last_undegraded":"2024-04-26T23:01:12.483844+0000","last_fullsized":"2024-04-26T23:01:12.483844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:13.482334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776212+0000","last_change":"2024-04-26T23:01:05.729610+0000","last_active":"2024-04-26T23:01:11.776212+0000","last_peered":"2024-04-26T23:01:11.776212+0000","last_clean":"2024-04-26T23:01:11.776212+0000","last_became_active":"2024-04-26T23:01:05.729307+0000","last_became_peered":"2024-04-26T23:01:05.729307+0000","last_unstale":"2024-04-26T23:01:11.776212+0000","last_undegraded":"2024-04-26T23:01:11.776212+0000","last_fullsized":"2024-04-26T23:01:11.776212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:57.106236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484169+0000","last_change":"2024-04-26T23:01:05.753873+0000","last_active":"2024-04-26T23:01:12.484169+0000","last_peered":"2024-04-26T23:01:12.484169+0000","last_clean":"2024-04-26T23:01:12.484169+0000","last_became_active":"2024-04-26T23:01:05.753603+0000","last_became_peered":"2024-04-26T23:01:05.753603+0000","last_unstale":"2024-04-26T23:01:12.484169+0000","last_undegraded":"2024-04-26T23:01:12.484169+0000","last_fullsized":"2024-04-26T23:01:12.484169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:22:15.301548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484214+0000","last_change":"2024-04-26T23:01:05.720420+0000","last_active":"2024-04-26T23:01:12.484214+0000","last_peered":"2024-04-26T23:01:12.484214+0000","last_clean":"2024-04-26T23:01:12.484214+0000","last_became_active":"2024-04-26T23:01:05.720114+0000","last_became_peered":"2024-04-26T23:01:05.720114+0000","last_unstale":"2024-04-26T23:01:12.484214+0000","last_undegraded":"2024-04-26T23:01:12.484214+0000","last_fullsized":"2024-04-26T23:01:12.484214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486621+0000","last_change":"2024-04-26T23:01:05.730995+0000","last_active":"2024-04-26T23:01:12.486621+0000","last_peered":"2024-04-26T23:01:12.486621+0000","last_clean":"2024-04-26T23:01:12.486621+0000","last_became_active":"2024-04-26T23:01:05.730644+0000","last_became_peered":"2024-04-26T23:01:05.730644+0000","last_unstale":"2024-04-26T23:01:12.486621+0000","last_undegraded":"2024-04-26T23:01:12.486621+0000","last_fullsized":"2024-04-26T23:01:12.486621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776188+0000","last_change":"2024-04-26T23:01:05.727391+0000","last_active":"2024-04-26T23:01:11.776188+0000","last_peered":"2024-04-26T23:01:11.776188+0000","last_clean":"2024-04-26T23:01:11.776188+0000","last_became_active":"2024-04-26T23:01:05.726729+0000","last_became_peered":"2024-04-26T23:01:05.726729+0000","last_unstale":"2024-04-26T23:01:11.776188+0000","last_undegraded":"2024-04-26T23:01:11.776188+0000","last_fullsized":"2024-04-26T23:01:11.776188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:24.824202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:01:35.401 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:01:35.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.652+0000 7ff725400640 1 Processor -- start 2024-04-26T23:01:35.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.652+0000 7ff725400640 1 -- start start 2024-04-26T23:01:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.654+0000 7ff725400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff720163430 0x7ff720166f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.654+0000 7ff725400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff720163810 con 0x7ff7201674a0 2024-04-26T23:01:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.654+0000 7ff725400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7200cb100 con 0x7ff7200cf810 2024-04-26T23:01:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.654+0000 7ff725400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff7200cb400 con 0x7ff720163430 2024-04-26T23:01:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.654+0000 7ff71f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff7201674a0 0x7ff7200c62c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41448/0 (socket says 172.21.15.40:41448) 2024-04-26T23:01:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.654+0000 7ff71f400640 1 -- 172.21.15.40:0/1681999104 learned_addr learned my addr 172.21.15.40:0/1681999104 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2124084324 0 0) 0x7ff720163810 con 0x7ff7201674a0 2024-04-26T23:01:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff704003520 con 0x7ff7201674a0 2024-04-26T23:01:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71fe00640 1 --2- 172.21.15.40:0/1681999104 >> v2:172.21.15.103:3300/0 conn(0x7ff720163430 0x7ff720166f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2128290347 0 0) 0x7ff704003520 con 0x7ff7201674a0 2024-04-26T23:01:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 >> v2:172.21.15.103:3300/0 conn(0x7ff720163430 msgr2=0x7ff720166f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 --2- 172.21.15.40:0/1681999104 >> v2:172.21.15.103:3300/0 conn(0x7ff720163430 0x7ff720166f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 >> v1:172.21.15.103:6789/0 conn(0x7ff7200cf810 legacy=0x7ff7200c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7200ccc60 con 0x7ff7201674a0 2024-04-26T23:01:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7ff70c001670 con 0x7ff7201674a0 2024-04-26T23:01:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 >> v1:172.21.15.40:6789/0 conn(0x7ff7201674a0 legacy=0x7ff7200c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 --2- 172.21.15.40:0/1681999104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7040039b0 0x7ff704003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 --2- 172.21.15.40:0/1681999104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff704004800 0x7ff704004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 --2- 172.21.15.40:0/1681999104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff704005210 0x7ff70400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff7200cb400 con 0x7ff7040039b0 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff70400c140 con 0x7ff704005210 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.655+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff70400c340 con 0x7ff704004800 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.656+0000 7ff71fe00640 1 --2- 172.21.15.40:0/1681999104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7040039b0 0x7ff704003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.656+0000 7ff71f400640 1 --2- 172.21.15.40:0/1681999104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff704004800 0x7ff704004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.656+0000 7ff71fe00640 1 -- 172.21.15.40:0/1681999104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff704004800 msgr2=0x7ff704004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.656+0000 7ff71fe00640 1 --2- 172.21.15.40:0/1681999104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff704004800 0x7ff704004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.656+0000 7ff71fe00640 1 -- 172.21.15.40:0/1681999104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff704005210 msgr2=0x7ff70400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.656+0000 7ff724a00640 1 --2- 172.21.15.40:0/1681999104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff704005210 0x7ff70400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.656+0000 7ff71fe00640 1 --2- 172.21.15.40:0/1681999104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff704005210 0x7ff70400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.656+0000 7ff71fe00640 1 -- 172.21.15.40:0/1681999104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff704003520 con 0x7ff7040039b0 2024-04-26T23:01:35.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.656+0000 7ff724a00640 1 --2- 172.21.15.40:0/1681999104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff704005210 0x7ff70400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:35.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.656+0000 7ff724a00640 1 -- 172.21.15.40:0/1681999104 reap_dead start 2024-04-26T23:01:35.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.656+0000 7ff71f400640 1 --2- 172.21.15.40:0/1681999104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff704004800 0x7ff704004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:35.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.657+0000 7ff71fe00640 1 --2- 172.21.15.40:0/1681999104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7040039b0 0x7ff704003df0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff71000ba30 tx=0x7ff704005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7145e2c5817cdf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:35.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.658+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710030070 con 0x7ff7040039b0 2024-04-26T23:01:35.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.658+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff71002ade0 con 0x7ff7040039b0 2024-04-26T23:01:35.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.658+0000 7ff71ea00640 1 -- 172.21.15.40:0/1681999104 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71000b600 con 0x7ff7040039b0 2024-04-26T23:01:35.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.658+0000 7ff725400640 1 -- 172.21.15.40:0/1681999104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7040039b0 msgr2=0x7ff704003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:35.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.658+0000 7ff725400640 1 --2- 172.21.15.40:0/1681999104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7040039b0 0x7ff704003df0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff71000ba30 tx=0x7ff704005830 comp rx=0 tx=0).stop 2024-04-26T23:01:35.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.658+0000 7ff725400640 1 -- 172.21.15.40:0/1681999104 shutdown_connections 2024-04-26T23:01:35.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.658+0000 7ff725400640 1 --2- 172.21.15.40:0/1681999104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7040039b0 0x7ff704003df0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.658+0000 7ff725400640 1 -- 172.21.15.40:0/1681999104 >> 172.21.15.40:0/1681999104 conn(0x7ff7200d0a10 msgr2=0x7ff7200d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:35.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.658+0000 7ff725400640 1 -- 172.21.15.40:0/1681999104 shutdown_connections 2024-04-26T23:01:35.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.658+0000 7ff725400640 1 -- 172.21.15.40:0/1681999104 wait complete. 2024-04-26T23:01:35.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.660+0000 7ff725400640 1 Processor -- start 2024-04-26T23:01:35.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.660+0000 7ff725400640 1 -- start start 2024-04-26T23:01:35.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.661+0000 7ff725400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7040039b0 0x7ff7200b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.661+0000 7ff725400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7200c8c20 0x7ff7200b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.661+0000 7ff725400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff72015a910 0x7ff72006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.661+0000 7ff725400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff7200cd1e0 con 0x7ff72015a910 2024-04-26T23:01:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.661+0000 7ff725400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff7200afe80 con 0x7ff7200c8c20 2024-04-26T23:01:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.661+0000 7ff725400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff720166f60 con 0x7ff7040039b0 2024-04-26T23:01:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.662+0000 7ff724a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff72015a910 0x7ff72006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.662+0000 7ff724a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff72015a910 0x7ff72006ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38652/0 (socket says 172.21.15.40:38652) 2024-04-26T23:01:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.662+0000 7ff724a00640 1 -- 172.21.15.40:0/3528017644 learned_addr learned my addr 172.21.15.40:0/3528017644 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.662+0000 7ff71fe00640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7040039b0 0x7ff7200b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.662+0000 7ff71f400640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7200c8c20 0x7ff7200b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.662+0000 7ff724a00640 1 -- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7040039b0 msgr2=0x7ff7200b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.662+0000 7ff724a00640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7040039b0 0x7ff7200b4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.662+0000 7ff724a00640 1 -- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7200c8c20 msgr2=0x7ff7200b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.662+0000 7ff724a00640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7200c8c20 0x7ff7200b45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.662+0000 7ff724a00640 1 -- 172.21.15.40:0/3528017644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7200c7b80 con 0x7ff72015a910 2024-04-26T23:01:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.663+0000 7ff724a00640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff72015a910 0x7ff72006ed50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff7200b6b80 tx=0x7ff71800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.663+0000 7ff717e00640 1 -- 172.21.15.40:0/3528017644 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff718018070 con 0x7ff72015a910 2024-04-26T23:01:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.663+0000 7ff717e00640 1 -- 172.21.15.40:0/3528017644 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff718003b70 con 0x7ff72015a910 2024-04-26T23:01:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.663+0000 7ff717e00640 1 -- 172.21.15.40:0/3528017644 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff718005070 con 0x7ff72015a910 2024-04-26T23:01:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.663+0000 7ff725400640 1 -- 172.21.15.40:0/3528017644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7200c7e60 con 0x7ff72015a910 2024-04-26T23:01:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.663+0000 7ff725400640 1 -- 172.21.15.40:0/3528017644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7200c8420 con 0x7ff72015a910 2024-04-26T23:01:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.664+0000 7ff725400640 1 -- 172.21.15.40:0/3528017644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff720069370 con 0x7ff72015a910 2024-04-26T23:01:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.665+0000 7ff717e00640 1 -- 172.21.15.40:0/3528017644 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff71801b070 con 0x7ff72015a910 2024-04-26T23:01:35.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.666+0000 7ff717e00640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff70006c920 0x7ff70006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:35.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.666+0000 7ff717e00640 1 -- 172.21.15.40:0/3528017644 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff71808b690 con 0x7ff72015a910 2024-04-26T23:01:35.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.667+0000 7ff71fe00640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff70006c920 0x7ff70006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.668+0000 7ff71fe00640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff70006c920 0x7ff70006ede0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff71000ba00 tx=0x7ff710030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:35.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.671+0000 7ff717e00640 1 -- 172.21.15.40:0/3528017644 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff7180596e0 con 0x7ff72015a910 2024-04-26T23:01:35.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.870+0000 7ff725400640 1 -- 172.21.15.40:0/3528017644 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7200c63b0 con 0x7ff70006c920 2024-04-26T23:01:35.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.895+0000 7ff717e00640 1 -- 172.21.15.40:0/3528017644 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332538 (secure 0 0 0) 0x7ff7200c63b0 con 0x7ff70006c920 2024-04-26T23:01:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:35.902 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:01:35.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.903+0000 7ff725400640 1 -- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff70006c920 msgr2=0x7ff70006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:35.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.903+0000 7ff725400640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff70006c920 0x7ff70006ede0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff71000ba00 tx=0x7ff710030000 comp rx=0 tx=0).stop 2024-04-26T23:01:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.903+0000 7ff725400640 1 -- 172.21.15.40:0/3528017644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff72015a910 msgr2=0x7ff72006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.903+0000 7ff725400640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff72015a910 0x7ff72006ed50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff7200b6b80 tx=0x7ff71800dc00 comp rx=0 tx=0).stop 2024-04-26T23:01:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.903+0000 7ff725400640 1 -- 172.21.15.40:0/3528017644 shutdown_connections 2024-04-26T23:01:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.903+0000 7ff725400640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff70006c920 0x7ff70006ede0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.903+0000 7ff725400640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff72015a910 0x7ff72006ed50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.903+0000 7ff725400640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7200c8c20 0x7ff7200b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.903+0000 7ff725400640 1 --2- 172.21.15.40:0/3528017644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7040039b0 0x7ff7200b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.903+0000 7ff725400640 1 -- 172.21.15.40:0/3528017644 >> 172.21.15.40:0/3528017644 conn(0x7ff7200d0a10 msgr2=0x7ff72015ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.903+0000 7ff725400640 1 -- 172.21.15.40:0/3528017644 shutdown_connections 2024-04-26T23:01:35.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:35.903+0000 7ff725400640 1 -- 172.21.15.40:0/3528017644 wait complete. 2024-04-26T23:01:35.926 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-26T23:01:35.151963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1781,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004933"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487297+0000","last_change":"2024-04-26T23:01:04.726404+0000","last_active":"2024-04-26T23:01:12.487297+0000","last_peered":"2024-04-26T23:01:12.487297+0000","last_clean":"2024-04-26T23:01:12.487297+0000","last_became_active":"2024-04-26T23:01:04.726160+0000","last_became_peered":"2024-04-26T23:01:04.726160+0000","last_unstale":"2024-04-26T23:01:12.487297+0000","last_undegraded":"2024-04-26T23:01:12.487297+0000","last_fullsized":"2024-04-26T23:01:12.487297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:21:22.581263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771048+0000","last_change":"2024-04-26T23:01:05.746741+0000","last_active":"2024-04-26T23:01:11.771048+0000","last_peered":"2024-04-26T23:01:11.771048+0000","last_clean":"2024-04-26T23:01:11.771048+0000","last_became_active":"2024-04-26T23:01:05.743495+0000","last_became_peered":"2024-04-26T23:01:05.743495+0000","last_unstale":"2024-04-26T23:01:11.771048+0000","last_undegraded":"2024-04-26T23:01:11.771048+0000","last_fullsized":"2024-04-26T23:01:11.771048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769795+0000","last_change":"2024-04-26T23:01:10.403533+0000","last_active":"2024-04-26T23:01:11.769795+0000","last_peered":"2024-04-26T23:01:11.769795+0000","last_clean":"2024-04-26T23:01:11.769795+0000","last_became_active":"2024-04-26T23:01:10.402972+0000","last_became_peered":"2024-04-26T23:01:10.402972+0000","last_unstale":"2024-04-26T23:01:11.769795+0000","last_undegraded":"2024-04-26T23:01:11.769795+0000","last_fullsized":"2024-04-26T23:01:11.769795+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:17.916674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771280+0000","last_change":"2024-04-26T23:01:04.715636+0000","last_active":"2024-04-26T23:01:11.771280+0000","last_peered":"2024-04-26T23:01:11.771280+0000","last_clean":"2024-04-26T23:01:11.771280+0000","last_became_active":"2024-04-26T23:01:04.715322+0000","last_became_peered":"2024-04-26T23:01:04.715322+0000","last_unstale":"2024-04-26T23:01:11.771280+0000","last_undegraded":"2024-04-26T23:01:11.771280+0000","last_fullsized":"2024-04-26T23:01:11.771280+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:55.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486825+0000","last_change":"2024-04-26T23:01:05.723149+0000","last_active":"2024-04-26T23:01:12.486825+0000","last_peered":"2024-04-26T23:01:12.486825+0000","last_clean":"2024-04-26T23:01:12.486825+0000","last_became_active":"2024-04-26T23:01:05.722809+0000","last_became_peered":"2024-04-26T23:01:05.722809+0000","last_unstale":"2024-04-26T23:01:12.486825+0000","last_undegraded":"2024-04-26T23:01:12.486825+0000","last_fullsized":"2024-04-26T23:01:12.486825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:26.157667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.881757+0000","last_change":"2024-04-26T23:01:09.805189+0000","last_active":"2024-04-26T23:01:11.881757+0000","last_peered":"2024-04-26T23:01:11.881757+0000","last_clean":"2024-04-26T23:01:11.881757+0000","last_became_active":"2024-04-26T23:01:09.804560+0000","last_became_peered":"2024-04-26T23:01:09.804560+0000","last_unstale":"2024-04-26T23:01:11.881757+0000","last_undegraded":"2024-04-26T23:01:11.881757+0000","last_fullsized":"2024-04-26T23:01:11.881757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:47.817790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775805+0000","last_change":"2024-04-26T23:01:04.734037+0000","last_active":"2024-04-26T23:01:11.775805+0000","last_peered":"2024-04-26T23:01:11.775805+0000","last_clean":"2024-04-26T23:01:11.775805+0000","last_became_active":"2024-04-26T23:01:04.733352+0000","last_became_peered":"2024-04-26T23:01:04.733352+0000","last_unstale":"2024-04-26T23:01:11.775805+0000","last_undegraded":"2024-04-26T23:01:11.775805+0000","last_fullsized":"2024-04-26T23:01:11.775805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:16.098138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484327+0000","last_change":"2024-04-26T23:01:05.738238+0000","last_active":"2024-04-26T23:01:12.484327+0000","last_peered":"2024-04-26T23:01:12.484327+0000","last_clean":"2024-04-26T23:01:12.484327+0000","last_became_active":"2024-04-26T23:01:05.737945+0000","last_became_peered":"2024-04-26T23:01:05.737945+0000","last_unstale":"2024-04-26T23:01:12.484327+0000","last_undegraded":"2024-04-26T23:01:12.484327+0000","last_fullsized":"2024-04-26T23:01:12.484327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:22.679803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.881869+0000","last_change":"2024-04-26T23:01:09.800139+0000","last_active":"2024-04-26T23:01:11.881869+0000","last_peered":"2024-04-26T23:01:11.881869+0000","last_clean":"2024-04-26T23:01:11.881869+0000","last_became_active":"2024-04-26T23:01:09.799214+0000","last_became_peered":"2024-04-26T23:01:09.799214+0000","last_unstale":"2024-04-26T23:01:11.881869+0000","last_undegraded":"2024-04-26T23:01:11.881869+0000","last_fullsized":"2024-04-26T23:01:11.881869+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:46:57.684555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776133+0000","last_change":"2024-04-26T23:01:04.735697+0000","last_active":"2024-04-26T23:01:11.776133+0000","last_peered":"2024-04-26T23:01:11.776133+0000","last_clean":"2024-04-26T23:01:11.776133+0000","last_became_active":"2024-04-26T23:01:04.735327+0000","last_became_peered":"2024-04-26T23:01:04.735327+0000","last_unstale":"2024-04-26T23:01:11.776133+0000","last_undegraded":"2024-04-26T23:01:11.776133+0000","last_fullsized":"2024-04-26T23:01:11.776133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:58.527150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484663+0000","last_change":"2024-04-26T23:01:05.729770+0000","last_active":"2024-04-26T23:01:12.484663+0000","last_peered":"2024-04-26T23:01:12.484663+0000","last_clean":"2024-04-26T23:01:12.484663+0000","last_became_active":"2024-04-26T23:01:05.727922+0000","last_became_peered":"2024-04-26T23:01:05.727922+0000","last_unstale":"2024-04-26T23:01:12.484663+0000","last_undegraded":"2024-04-26T23:01:12.484663+0000","last_fullsized":"2024-04-26T23:01:12.484663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:44:40.404454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882761+0000","last_change":"2024-04-26T23:01:09.804570+0000","last_active":"2024-04-26T23:01:11.882761+0000","last_peered":"2024-04-26T23:01:11.882761+0000","last_clean":"2024-04-26T23:01:11.882761+0000","last_became_active":"2024-04-26T23:01:09.803489+0000","last_became_peered":"2024-04-26T23:01:09.803489+0000","last_unstale":"2024-04-26T23:01:11.882761+0000","last_undegraded":"2024-04-26T23:01:11.882761+0000","last_fullsized":"2024-04-26T23:01:11.882761+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774588+0000","last_change":"2024-04-26T23:01:05.729351+0000","last_active":"2024-04-26T23:01:11.774588+0000","last_peered":"2024-04-26T23:01:11.774588+0000","last_clean":"2024-04-26T23:01:11.774588+0000","last_became_active":"2024-04-26T23:01:05.729024+0000","last_became_peered":"2024-04-26T23:01:05.729024+0000","last_unstale":"2024-04-26T23:01:11.774588+0000","last_undegraded":"2024-04-26T23:01:11.774588+0000","last_fullsized":"2024-04-26T23:01:11.774588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776686+0000","last_change":"2024-04-26T23:01:04.730260+0000","last_active":"2024-04-26T23:01:11.776686+0000","last_peered":"2024-04-26T23:01:11.776686+0000","last_clean":"2024-04-26T23:01:11.776686+0000","last_became_active":"2024-04-26T23:01:04.729209+0000","last_became_peered":"2024-04-26T23:01:04.729209+0000","last_unstale":"2024-04-26T23:01:11.776686+0000","last_undegraded":"2024-04-26T23:01:11.776686+0000","last_fullsized":"2024-04-26T23:01:11.776686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:45.523274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.883045+0000","last_change":"2024-04-26T23:01:09.796341+0000","last_active":"2024-04-26T23:01:11.883045+0000","last_peered":"2024-04-26T23:01:11.883045+0000","last_clean":"2024-04-26T23:01:11.883045+0000","last_became_active":"2024-04-26T23:01:09.795939+0000","last_became_peered":"2024-04-26T23:01:09.795939+0000","last_unstale":"2024-04-26T23:01:11.883045+0000","last_undegraded":"2024-04-26T23:01:11.883045+0000","last_fullsized":"2024-04-26T23:01:11.883045+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:09:55.216454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770614+0000","last_change":"2024-04-26T23:01:05.735100+0000","last_active":"2024-04-26T23:01:11.770614+0000","last_peered":"2024-04-26T23:01:11.770614+0000","last_clean":"2024-04-26T23:01:11.770614+0000","last_became_active":"2024-04-26T23:01:05.734138+0000","last_became_peered":"2024-04-26T23:01:05.734138+0000","last_unstale":"2024-04-26T23:01:11.770614+0000","last_undegraded":"2024-04-26T23:01:11.770614+0000","last_fullsized":"2024-04-26T23:01:11.770614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486466+0000","last_change":"2024-04-26T23:01:04.723885+0000","last_active":"2024-04-26T23:01:12.486466+0000","last_peered":"2024-04-26T23:01:12.486466+0000","last_clean":"2024-04-26T23:01:12.486466+0000","last_became_active":"2024-04-26T23:01:04.723597+0000","last_became_peered":"2024-04-26T23:01:04.723597+0000","last_unstale":"2024-04-26T23:01:12.486466+0000","last_undegraded":"2024-04-26T23:01:12.486466+0000","last_fullsized":"2024-04-26T23:01:12.486466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:45.523274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882064+0000","last_change":"2024-04-26T23:01:09.803721+0000","last_active":"2024-04-26T23:01:11.882064+0000","last_peered":"2024-04-26T23:01:11.882064+0000","last_clean":"2024-04-26T23:01:11.882064+0000","last_became_active":"2024-04-26T23:01:09.803061+0000","last_became_peered":"2024-04-26T23:01:09.803061+0000","last_unstale":"2024-04-26T23:01:11.882064+0000","last_undegraded":"2024-04-26T23:01:11.882064+0000","last_fullsized":"2024-04-26T23:01:11.882064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:41.540020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769949+0000","last_change":"2024-04-26T23:01:05.738344+0000","last_active":"2024-04-26T23:01:11.769949+0000","last_peered":"2024-04-26T23:01:11.769949+0000","last_clean":"2024-04-26T23:01:11.769949+0000","last_became_active":"2024-04-26T23:01:05.738082+0000","last_became_peered":"2024-04-26T23:01:05.738082+0000","last_unstale":"2024-04-26T23:01:11.769949+0000","last_undegraded":"2024-04-26T23:01:11.769949+0000","last_fullsized":"2024-04-26T23:01:11.769949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:26.157667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484208+0000","last_change":"2024-04-26T23:01:04.727252+0000","last_active":"2024-04-26T23:01:12.484208+0000","last_peered":"2024-04-26T23:01:12.484208+0000","last_clean":"2024-04-26T23:01:12.484208+0000","last_became_active":"2024-04-26T23:01:04.726980+0000","last_became_peered":"2024-04-26T23:01:04.726980+0000","last_unstale":"2024-04-26T23:01:12.484208+0000","last_undegraded":"2024-04-26T23:01:12.484208+0000","last_fullsized":"2024-04-26T23:01:12.484208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486354+0000","last_change":"2024-04-26T23:01:10.399273+0000","last_active":"2024-04-26T23:01:12.486354+0000","last_peered":"2024-04-26T23:01:12.486354+0000","last_clean":"2024-04-26T23:01:12.486354+0000","last_became_active":"2024-04-26T23:01:10.397193+0000","last_became_peered":"2024-04-26T23:01:10.397193+0000","last_unstale":"2024-04-26T23:01:12.486354+0000","last_undegraded":"2024-04-26T23:01:12.486354+0000","last_fullsized":"2024-04-26T23:01:12.486354+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882811+0000","last_change":"2024-04-26T23:01:05.734542+0000","last_active":"2024-04-26T23:01:11.882811+0000","last_peered":"2024-04-26T23:01:11.882811+0000","last_clean":"2024-04-26T23:01:11.882811+0000","last_became_active":"2024-04-26T23:01:05.734191+0000","last_became_peered":"2024-04-26T23:01:05.734191+0000","last_unstale":"2024-04-26T23:01:11.882811+0000","last_undegraded":"2024-04-26T23:01:11.882811+0000","last_fullsized":"2024-04-26T23:01:11.882811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:49.765162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774127+0000","last_change":"2024-04-26T23:01:04.732728+0000","last_active":"2024-04-26T23:01:11.774127+0000","last_peered":"2024-04-26T23:01:11.774127+0000","last_clean":"2024-04-26T23:01:11.774127+0000","last_became_active":"2024-04-26T23:01:04.732413+0000","last_became_peered":"2024-04-26T23:01:04.732413+0000","last_unstale":"2024-04-26T23:01:11.774127+0000","last_undegraded":"2024-04-26T23:01:11.774127+0000","last_fullsized":"2024-04-26T23:01:11.774127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882964+0000","last_change":"2024-04-26T23:01:09.800234+0000","last_active":"2024-04-26T23:01:11.882964+0000","last_peered":"2024-04-26T23:01:11.882964+0000","last_clean":"2024-04-26T23:01:11.882964+0000","last_became_active":"2024-04-26T23:01:09.799644+0000","last_became_peered":"2024-04-26T23:01:09.799644+0000","last_unstale":"2024-04-26T23:01:11.882964+0000","last_undegraded":"2024-04-26T23:01:11.882964+0000","last_fullsized":"2024-04-26T23:01:11.882964+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:37:23.404492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486201+0000","last_change":"2024-04-26T23:01:05.735371+0000","last_active":"2024-04-26T23:01:12.486201+0000","last_peered":"2024-04-26T23:01:12.486201+0000","last_clean":"2024-04-26T23:01:12.486201+0000","last_became_active":"2024-04-26T23:01:05.729667+0000","last_became_peered":"2024-04-26T23:01:05.729667+0000","last_unstale":"2024-04-26T23:01:12.486201+0000","last_undegraded":"2024-04-26T23:01:12.486201+0000","last_fullsized":"2024-04-26T23:01:12.486201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778387+0000","last_change":"2024-04-26T23:01:04.726450+0000","last_active":"2024-04-26T23:01:11.778387+0000","last_peered":"2024-04-26T23:01:11.778387+0000","last_clean":"2024-04-26T23:01:11.778387+0000","last_became_active":"2024-04-26T23:01:04.726023+0000","last_became_peered":"2024-04-26T23:01:04.726023+0000","last_unstale":"2024-04-26T23:01:11.778387+0000","last_undegraded":"2024-04-26T23:01:11.778387+0000","last_fullsized":"2024-04-26T23:01:11.778387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:45.523274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.881567+0000","last_change":"2024-04-26T23:01:10.401428+0000","last_active":"2024-04-26T23:01:11.881567+0000","last_peered":"2024-04-26T23:01:11.881567+0000","last_clean":"2024-04-26T23:01:11.881567+0000","last_became_active":"2024-04-26T23:01:10.400621+0000","last_became_peered":"2024-04-26T23:01:10.400621+0000","last_unstale":"2024-04-26T23:01:11.881567+0000","last_undegraded":"2024-04-26T23:01:11.881567+0000","last_fullsized":"2024-04-26T23:01:11.881567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:59.067859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771304+0000","last_change":"2024-04-26T23:01:05.748687+0000","last_active":"2024-04-26T23:01:11.771304+0000","last_peered":"2024-04-26T23:01:11.771304+0000","last_clean":"2024-04-26T23:01:11.771304+0000","last_became_active":"2024-04-26T23:01:05.744038+0000","last_became_peered":"2024-04-26T23:01:05.744038+0000","last_unstale":"2024-04-26T23:01:11.771304+0000","last_undegraded":"2024-04-26T23:01:11.771304+0000","last_fullsized":"2024-04-26T23:01:11.771304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:13.482334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486378+0000","last_change":"2024-04-26T23:01:04.718414+0000","last_active":"2024-04-26T23:01:12.486378+0000","last_peered":"2024-04-26T23:01:12.486378+0000","last_clean":"2024-04-26T23:01:12.486378+0000","last_became_active":"2024-04-26T23:01:04.717764+0000","last_became_peered":"2024-04-26T23:01:04.717764+0000","last_unstale":"2024-04-26T23:01:12.486378+0000","last_undegraded":"2024-04-26T23:01:12.486378+0000","last_fullsized":"2024-04-26T23:01:12.486378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769756+0000","last_change":"2024-04-26T23:01:10.386716+0000","last_active":"2024-04-26T23:01:11.769756+0000","last_peered":"2024-04-26T23:01:11.769756+0000","last_clean":"2024-04-26T23:01:11.769756+0000","last_became_active":"2024-04-26T23:01:10.385392+0000","last_became_peered":"2024-04-26T23:01:10.385392+0000","last_unstale":"2024-04-26T23:01:11.769756+0000","last_undegraded":"2024-04-26T23:01:11.769756+0000","last_fullsized":"2024-04-26T23:01:11.769756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:09:55.216454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485210+0000","last_change":"2024-04-26T23:01:05.728786+0000","last_active":"2024-04-26T23:01:12.485210+0000","last_peered":"2024-04-26T23:01:12.485210+0000","last_clean":"2024-04-26T23:01:12.485210+0000","last_became_active":"2024-04-26T23:01:05.728430+0000","last_became_peered":"2024-04-26T23:01:05.728430+0000","last_unstale":"2024-04-26T23:01:12.485210+0000","last_undegraded":"2024-04-26T23:01:12.485210+0000","last_fullsized":"2024-04-26T23:01:12.485210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776238+0000","last_change":"2024-04-26T23:01:04.734438+0000","last_active":"2024-04-26T23:01:11.776238+0000","last_peered":"2024-04-26T23:01:11.776238+0000","last_clean":"2024-04-26T23:01:11.776238+0000","last_became_active":"2024-04-26T23:01:04.733372+0000","last_became_peered":"2024-04-26T23:01:04.733372+0000","last_unstale":"2024-04-26T23:01:11.776238+0000","last_undegraded":"2024-04-26T23:01:11.776238+0000","last_fullsized":"2024-04-26T23:01:11.776238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:55.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483768+0000","last_change":"2024-04-26T23:01:10.391451+0000","last_active":"2024-04-26T23:01:12.483768+0000","last_peered":"2024-04-26T23:01:12.483768+0000","last_clean":"2024-04-26T23:01:12.483768+0000","last_became_active":"2024-04-26T23:01:10.389477+0000","last_became_peered":"2024-04-26T23:01:10.389477+0000","last_unstale":"2024-04-26T23:01:12.483768+0000","last_undegraded":"2024-04-26T23:01:12.483768+0000","last_fullsized":"2024-04-26T23:01:12.483768+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:46:57.684555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485051+0000","last_change":"2024-04-26T23:01:05.746004+0000","last_active":"2024-04-26T23:01:12.485051+0000","last_peered":"2024-04-26T23:01:12.485051+0000","last_clean":"2024-04-26T23:01:12.485051+0000","last_became_active":"2024-04-26T23:01:05.745576+0000","last_became_peered":"2024-04-26T23:01:05.745576+0000","last_unstale":"2024-04-26T23:01:12.485051+0000","last_undegraded":"2024-04-26T23:01:12.485051+0000","last_fullsized":"2024-04-26T23:01:12.485051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:06.384302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485393+0000","last_change":"2024-04-26T23:01:04.717206+0000","last_active":"2024-04-26T23:01:12.485393+0000","last_peered":"2024-04-26T23:01:12.485393+0000","last_clean":"2024-04-26T23:01:12.485393+0000","last_became_active":"2024-04-26T23:01:04.715952+0000","last_became_peered":"2024-04-26T23:01:04.715952+0000","last_unstale":"2024-04-26T23:01:12.485393+0000","last_undegraded":"2024-04-26T23:01:12.485393+0000","last_fullsized":"2024-04-26T23:01:12.485393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485772+0000","last_change":"2024-04-26T23:01:09.799096+0000","last_active":"2024-04-26T23:01:12.485772+0000","last_peered":"2024-04-26T23:01:12.485772+0000","last_clean":"2024-04-26T23:01:12.485772+0000","last_became_active":"2024-04-26T23:01:09.798586+0000","last_became_peered":"2024-04-26T23:01:09.798586+0000","last_unstale":"2024-04-26T23:01:12.485772+0000","last_undegraded":"2024-04-26T23:01:12.485772+0000","last_fullsized":"2024-04-26T23:01:12.485772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:30.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.883148+0000","last_change":"2024-04-26T23:01:05.732977+0000","last_active":"2024-04-26T23:01:11.883148+0000","last_peered":"2024-04-26T23:01:11.883148+0000","last_clean":"2024-04-26T23:01:11.883148+0000","last_became_active":"2024-04-26T23:01:05.732617+0000","last_became_peered":"2024-04-26T23:01:05.732617+0000","last_unstale":"2024-04-26T23:01:11.883148+0000","last_undegraded":"2024-04-26T23:01:11.883148+0000","last_fullsized":"2024-04-26T23:01:11.883148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:59.540986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484829+0000","last_change":"2024-04-26T23:01:04.730386+0000","last_active":"2024-04-26T23:01:12.484829+0000","last_peered":"2024-04-26T23:01:12.484829+0000","last_clean":"2024-04-26T23:01:12.484829+0000","last_became_active":"2024-04-26T23:01:04.727828+0000","last_became_peered":"2024-04-26T23:01:04.727828+0000","last_unstale":"2024-04-26T23:01:12.484829+0000","last_undegraded":"2024-04-26T23:01:12.484829+0000","last_fullsized":"2024-04-26T23:01:12.484829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485312+0000","last_change":"2024-04-26T23:01:09.805894+0000","last_active":"2024-04-26T23:01:12.485312+0000","last_peered":"2024-04-26T23:01:12.485312+0000","last_clean":"2024-04-26T23:01:12.485312+0000","last_became_active":"2024-04-26T23:01:09.803400+0000","last_became_peered":"2024-04-26T23:01:09.803400+0000","last_unstale":"2024-04-26T23:01:12.485312+0000","last_undegraded":"2024-04-26T23:01:12.485312+0000","last_fullsized":"2024-04-26T23:01:12.485312+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:46:57.684555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484288+0000","last_change":"2024-04-26T23:01:05.722407+0000","last_active":"2024-04-26T23:01:12.484288+0000","last_peered":"2024-04-26T23:01:12.484288+0000","last_clean":"2024-04-26T23:01:12.484288+0000","last_became_active":"2024-04-26T23:01:05.722106+0000","last_became_peered":"2024-04-26T23:01:05.722106+0000","last_unstale":"2024-04-26T23:01:12.484288+0000","last_undegraded":"2024-04-26T23:01:12.484288+0000","last_fullsized":"2024-04-26T23:01:12.484288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487617+0000","last_change":"2024-04-26T23:01:04.712172+0000","last_active":"2024-04-26T23:01:12.487617+0000","last_peered":"2024-04-26T23:01:12.487617+0000","last_clean":"2024-04-26T23:01:12.487617+0000","last_became_active":"2024-04-26T23:01:04.711777+0000","last_became_peered":"2024-04-26T23:01:04.711777+0000","last_unstale":"2024-04-26T23:01:12.487617+0000","last_undegraded":"2024-04-26T23:01:12.487617+0000","last_fullsized":"2024-04-26T23:01:12.487617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486748+0000","last_change":"2024-04-26T23:01:09.808963+0000","last_active":"2024-04-26T23:01:12.486748+0000","last_peered":"2024-04-26T23:01:12.486748+0000","last_clean":"2024-04-26T23:01:12.486748+0000","last_became_active":"2024-04-26T23:01:09.808470+0000","last_became_peered":"2024-04-26T23:01:09.808470+0000","last_unstale":"2024-04-26T23:01:12.486748+0000","last_undegraded":"2024-04-26T23:01:12.486748+0000","last_fullsized":"2024-04-26T23:01:12.486748+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.482634+0000","last_change":"2024-04-26T23:01:05.742128+0000","last_active":"2024-04-26T23:01:12.482634+0000","last_peered":"2024-04-26T23:01:12.482634+0000","last_clean":"2024-04-26T23:01:12.482634+0000","last_became_active":"2024-04-26T23:01:05.735959+0000","last_became_peered":"2024-04-26T23:01:05.735959+0000","last_unstale":"2024-04-26T23:01:12.482634+0000","last_undegraded":"2024-04-26T23:01:12.482634+0000","last_fullsized":"2024-04-26T23:01:12.482634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774374+0000","last_change":"2024-04-26T23:01:04.732093+0000","last_active":"2024-04-26T23:01:11.774374+0000","last_peered":"2024-04-26T23:01:11.774374+0000","last_clean":"2024-04-26T23:01:11.774374+0000","last_became_active":"2024-04-26T23:01:04.731804+0000","last_became_peered":"2024-04-26T23:01:04.731804+0000","last_unstale":"2024-04-26T23:01:11.774374+0000","last_undegraded":"2024-04-26T23:01:11.774374+0000","last_fullsized":"2024-04-26T23:01:11.774374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485713+0000","last_change":"2024-04-26T23:01:10.389206+0000","last_active":"2024-04-26T23:01:12.485713+0000","last_peered":"2024-04-26T23:01:12.485713+0000","last_clean":"2024-04-26T23:01:12.485713+0000","last_became_active":"2024-04-26T23:01:10.384213+0000","last_became_peered":"2024-04-26T23:01:10.384213+0000","last_unstale":"2024-04-26T23:01:12.485713+0000","last_undegraded":"2024-04-26T23:01:12.485713+0000","last_fullsized":"2024-04-26T23:01:12.485713+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770463+0000","last_change":"2024-04-26T23:01:05.744836+0000","last_active":"2024-04-26T23:01:11.770463+0000","last_peered":"2024-04-26T23:01:11.770463+0000","last_clean":"2024-04-26T23:01:11.770463+0000","last_became_active":"2024-04-26T23:01:05.744600+0000","last_became_peered":"2024-04-26T23:01:05.744600+0000","last_unstale":"2024-04-26T23:01:11.770463+0000","last_undegraded":"2024-04-26T23:01:11.770463+0000","last_fullsized":"2024-04-26T23:01:11.770463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484137+0000","last_change":"2024-04-26T23:01:04.729176+0000","last_active":"2024-04-26T23:01:12.484137+0000","last_peered":"2024-04-26T23:01:12.484137+0000","last_clean":"2024-04-26T23:01:12.484137+0000","last_became_active":"2024-04-26T23:01:04.722896+0000","last_became_peered":"2024-04-26T23:01:04.722896+0000","last_unstale":"2024-04-26T23:01:12.484137+0000","last_undegraded":"2024-04-26T23:01:12.484137+0000","last_fullsized":"2024-04-26T23:01:12.484137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:16.585834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":180,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774617+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:01:11.774617+0000","last_peered":"2024-04-26T23:01:11.774617+0000","last_clean":"2024-04-26T23:01:11.774617+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:01:11.774617+0000","last_undegraded":"2024-04-26T23:01:11.774617+0000","last_fullsized":"2024-04-26T23:01:11.774617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777714+0000","last_change":"2024-04-26T23:01:10.402293+0000","last_active":"2024-04-26T23:01:11.777714+0000","last_peered":"2024-04-26T23:01:11.777714+0000","last_clean":"2024-04-26T23:01:11.777714+0000","last_became_active":"2024-04-26T23:01:10.396954+0000","last_became_peered":"2024-04-26T23:01:10.396954+0000","last_unstale":"2024-04-26T23:01:11.777714+0000","last_undegraded":"2024-04-26T23:01:11.777714+0000","last_fullsized":"2024-04-26T23:01:11.777714+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:44.757523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775315+0000","last_change":"2024-04-26T23:01:05.731266+0000","last_active":"2024-04-26T23:01:11.775315+0000","last_peered":"2024-04-26T23:01:11.775315+0000","last_clean":"2024-04-26T23:01:11.775315+0000","last_became_active":"2024-04-26T23:01:05.728530+0000","last_became_peered":"2024-04-26T23:01:05.728530+0000","last_unstale":"2024-04-26T23:01:11.775315+0000","last_undegraded":"2024-04-26T23:01:11.775315+0000","last_fullsized":"2024-04-26T23:01:11.775315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:44:40.404454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.488463+0000","last_change":"2024-04-26T23:01:04.728315+0000","last_active":"2024-04-26T23:01:12.488463+0000","last_peered":"2024-04-26T23:01:12.488463+0000","last_clean":"2024-04-26T23:01:12.488463+0000","last_became_active":"2024-04-26T23:01:04.727609+0000","last_became_peered":"2024-04-26T23:01:04.727609+0000","last_unstale":"2024-04-26T23:01:12.488463+0000","last_undegraded":"2024-04-26T23:01:12.488463+0000","last_fullsized":"2024-04-26T23:01:12.488463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775771+0000","last_change":"2024-04-26T23:01:10.391227+0000","last_active":"2024-04-26T23:01:11.775771+0000","last_peered":"2024-04-26T23:01:11.775771+0000","last_clean":"2024-04-26T23:01:11.775771+0000","last_became_active":"2024-04-26T23:01:10.387715+0000","last_became_peered":"2024-04-26T23:01:10.387715+0000","last_unstale":"2024-04-26T23:01:11.775771+0000","last_undegraded":"2024-04-26T23:01:11.775771+0000","last_fullsized":"2024-04-26T23:01:11.775771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:28.895329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484465+0000","last_change":"2024-04-26T23:01:05.753852+0000","last_active":"2024-04-26T23:01:12.484465+0000","last_peered":"2024-04-26T23:01:12.484465+0000","last_clean":"2024-04-26T23:01:12.484465+0000","last_became_active":"2024-04-26T23:01:05.753607+0000","last_became_peered":"2024-04-26T23:01:05.753607+0000","last_unstale":"2024-04-26T23:01:12.484465+0000","last_undegraded":"2024-04-26T23:01:12.484465+0000","last_fullsized":"2024-04-26T23:01:12.484465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:59:57.750603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485129+0000","last_change":"2024-04-26T23:01:04.713352+0000","last_active":"2024-04-26T23:01:12.485129+0000","last_peered":"2024-04-26T23:01:12.485129+0000","last_clean":"2024-04-26T23:01:12.485129+0000","last_became_active":"2024-04-26T23:01:04.712352+0000","last_became_peered":"2024-04-26T23:01:04.712352+0000","last_unstale":"2024-04-26T23:01:12.485129+0000","last_undegraded":"2024-04-26T23:01:12.485129+0000","last_fullsized":"2024-04-26T23:01:12.485129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:55.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484995+0000","last_change":"2024-04-26T23:01:09.803857+0000","last_active":"2024-04-26T23:01:12.484995+0000","last_peered":"2024-04-26T23:01:12.484995+0000","last_clean":"2024-04-26T23:01:12.484995+0000","last_became_active":"2024-04-26T23:01:09.799482+0000","last_became_peered":"2024-04-26T23:01:09.799482+0000","last_unstale":"2024-04-26T23:01:12.484995+0000","last_undegraded":"2024-04-26T23:01:12.484995+0000","last_fullsized":"2024-04-26T23:01:12.484995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:41.540020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776723+0000","last_change":"2024-04-26T23:01:05.741597+0000","last_active":"2024-04-26T23:01:11.776723+0000","last_peered":"2024-04-26T23:01:11.776723+0000","last_clean":"2024-04-26T23:01:11.776723+0000","last_became_active":"2024-04-26T23:01:05.741308+0000","last_became_peered":"2024-04-26T23:01:05.741308+0000","last_unstale":"2024-04-26T23:01:11.776723+0000","last_undegraded":"2024-04-26T23:01:11.776723+0000","last_fullsized":"2024-04-26T23:01:11.776723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:54.996731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487896+0000","last_change":"2024-04-26T23:01:04.720208+0000","last_active":"2024-04-26T23:01:12.487896+0000","last_peered":"2024-04-26T23:01:12.487896+0000","last_clean":"2024-04-26T23:01:12.487896+0000","last_became_active":"2024-04-26T23:01:04.719867+0000","last_became_peered":"2024-04-26T23:01:04.719867+0000","last_unstale":"2024-04-26T23:01:12.487896+0000","last_undegraded":"2024-04-26T23:01:12.487896+0000","last_fullsized":"2024-04-26T23:01:12.487896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:16.585834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483634+0000","last_change":"2024-04-26T23:01:10.396595+0000","last_active":"2024-04-26T23:01:12.483634+0000","last_peered":"2024-04-26T23:01:12.483634+0000","last_clean":"2024-04-26T23:01:12.483634+0000","last_became_active":"2024-04-26T23:01:10.395189+0000","last_became_peered":"2024-04-26T23:01:10.395189+0000","last_unstale":"2024-04-26T23:01:12.483634+0000","last_undegraded":"2024-04-26T23:01:12.483634+0000","last_fullsized":"2024-04-26T23:01:12.483634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776069+0000","last_change":"2024-04-26T23:01:05.733520+0000","last_active":"2024-04-26T23:01:11.776069+0000","last_peered":"2024-04-26T23:01:11.776069+0000","last_clean":"2024-04-26T23:01:11.776069+0000","last_became_active":"2024-04-26T23:01:05.733170+0000","last_became_peered":"2024-04-26T23:01:05.733170+0000","last_unstale":"2024-04-26T23:01:11.776069+0000","last_undegraded":"2024-04-26T23:01:11.776069+0000","last_fullsized":"2024-04-26T23:01:11.776069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:06.384302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.883050+0000","last_change":"2024-04-26T23:01:04.717930+0000","last_active":"2024-04-26T23:01:11.883050+0000","last_peered":"2024-04-26T23:01:11.883050+0000","last_clean":"2024-04-26T23:01:11.883050+0000","last_became_active":"2024-04-26T23:01:04.717576+0000","last_became_peered":"2024-04-26T23:01:04.717576+0000","last_unstale":"2024-04-26T23:01:11.883050+0000","last_undegraded":"2024-04-26T23:01:11.883050+0000","last_fullsized":"2024-04-26T23:01:11.883050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:11:48.078530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882003+0000","last_change":"2024-04-26T23:01:09.797064+0000","last_active":"2024-04-26T23:01:11.882003+0000","last_peered":"2024-04-26T23:01:11.882003+0000","last_clean":"2024-04-26T23:01:11.882003+0000","last_became_active":"2024-04-26T23:01:09.796589+0000","last_became_peered":"2024-04-26T23:01:09.796589+0000","last_unstale":"2024-04-26T23:01:11.882003+0000","last_undegraded":"2024-04-26T23:01:11.882003+0000","last_fullsized":"2024-04-26T23:01:11.882003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:30.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777169+0000","last_change":"2024-04-26T23:01:05.738519+0000","last_active":"2024-04-26T23:01:11.777169+0000","last_peered":"2024-04-26T23:01:11.777169+0000","last_clean":"2024-04-26T23:01:11.777169+0000","last_became_active":"2024-04-26T23:01:05.738201+0000","last_became_peered":"2024-04-26T23:01:05.738201+0000","last_unstale":"2024-04-26T23:01:11.777169+0000","last_undegraded":"2024-04-26T23:01:11.777169+0000","last_fullsized":"2024-04-26T23:01:11.777169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487284+0000","last_change":"2024-04-26T23:01:04.711562+0000","last_active":"2024-04-26T23:01:12.487284+0000","last_peered":"2024-04-26T23:01:12.487284+0000","last_clean":"2024-04-26T23:01:12.487284+0000","last_became_active":"2024-04-26T23:01:04.708852+0000","last_became_peered":"2024-04-26T23:01:04.708852+0000","last_unstale":"2024-04-26T23:01:12.487284+0000","last_undegraded":"2024-04-26T23:01:12.487284+0000","last_fullsized":"2024-04-26T23:01:12.487284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:21:22.581263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485270+0000","last_change":"2024-04-26T23:01:09.805302+0000","last_active":"2024-04-26T23:01:12.485270+0000","last_peered":"2024-04-26T23:01:12.485270+0000","last_clean":"2024-04-26T23:01:12.485270+0000","last_became_active":"2024-04-26T23:01:09.804878+0000","last_became_peered":"2024-04-26T23:01:09.804878+0000","last_unstale":"2024-04-26T23:01:12.485270+0000","last_undegraded":"2024-04-26T23:01:12.485270+0000","last_fullsized":"2024-04-26T23:01:12.485270+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774404+0000","last_change":"2024-04-26T23:01:05.717338+0000","last_active":"2024-04-26T23:01:11.774404+0000","last_peered":"2024-04-26T23:01:11.774404+0000","last_clean":"2024-04-26T23:01:11.774404+0000","last_became_active":"2024-04-26T23:01:05.716983+0000","last_became_peered":"2024-04-26T23:01:05.716983+0000","last_unstale":"2024-04-26T23:01:11.774404+0000","last_undegraded":"2024-04-26T23:01:11.774404+0000","last_fullsized":"2024-04-26T23:01:11.774404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774791+0000","last_change":"2024-04-26T23:01:04.715584+0000","last_active":"2024-04-26T23:01:11.774791+0000","last_peered":"2024-04-26T23:01:11.774791+0000","last_clean":"2024-04-26T23:01:11.774791+0000","last_became_active":"2024-04-26T23:01:04.715240+0000","last_became_peered":"2024-04-26T23:01:04.715240+0000","last_unstale":"2024-04-26T23:01:11.774791+0000","last_undegraded":"2024-04-26T23:01:11.774791+0000","last_fullsized":"2024-04-26T23:01:11.774791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771265+0000","last_change":"2024-04-26T23:01:10.401101+0000","last_active":"2024-04-26T23:01:11.771265+0000","last_peered":"2024-04-26T23:01:11.771265+0000","last_clean":"2024-04-26T23:01:11.771265+0000","last_became_active":"2024-04-26T23:01:10.396052+0000","last_became_peered":"2024-04-26T23:01:10.396052+0000","last_unstale":"2024-04-26T23:01:11.771265+0000","last_undegraded":"2024-04-26T23:01:11.771265+0000","last_fullsized":"2024-04-26T23:01:11.771265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:15.222701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776466+0000","last_change":"2024-04-26T23:01:05.732489+0000","last_active":"2024-04-26T23:01:11.776466+0000","last_peered":"2024-04-26T23:01:11.776466+0000","last_clean":"2024-04-26T23:01:11.776466+0000","last_became_active":"2024-04-26T23:01:05.732228+0000","last_became_peered":"2024-04-26T23:01:05.732228+0000","last_unstale":"2024-04-26T23:01:11.776466+0000","last_undegraded":"2024-04-26T23:01:11.776466+0000","last_fullsized":"2024-04-26T23:01:11.776466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:22:15.301548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484809+0000","last_change":"2024-04-26T23:01:04.731558+0000","last_active":"2024-04-26T23:01:12.484809+0000","last_peered":"2024-04-26T23:01:12.484809+0000","last_clean":"2024-04-26T23:01:12.484809+0000","last_became_active":"2024-04-26T23:01:04.731032+0000","last_became_peered":"2024-04-26T23:01:04.731032+0000","last_unstale":"2024-04-26T23:01:12.484809+0000","last_undegraded":"2024-04-26T23:01:12.484809+0000","last_fullsized":"2024-04-26T23:01:12.484809+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770866+0000","last_change":"2024-04-26T23:01:10.386434+0000","last_active":"2024-04-26T23:01:11.770866+0000","last_peered":"2024-04-26T23:01:11.770866+0000","last_clean":"2024-04-26T23:01:11.770866+0000","last_became_active":"2024-04-26T23:01:10.385162+0000","last_became_peered":"2024-04-26T23:01:10.385162+0000","last_unstale":"2024-04-26T23:01:11.770866+0000","last_undegraded":"2024-04-26T23:01:11.770866+0000","last_fullsized":"2024-04-26T23:01:11.770866+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:19.562651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483154+0000","last_change":"2024-04-26T23:01:05.740528+0000","last_active":"2024-04-26T23:01:12.483154+0000","last_peered":"2024-04-26T23:01:12.483154+0000","last_clean":"2024-04-26T23:01:12.483154+0000","last_became_active":"2024-04-26T23:01:05.740211+0000","last_became_peered":"2024-04-26T23:01:05.740211+0000","last_unstale":"2024-04-26T23:01:12.483154+0000","last_undegraded":"2024-04-26T23:01:12.483154+0000","last_fullsized":"2024-04-26T23:01:12.483154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484416+0000","last_change":"2024-04-26T23:01:04.712658+0000","last_active":"2024-04-26T23:01:12.484416+0000","last_peered":"2024-04-26T23:01:12.484416+0000","last_clean":"2024-04-26T23:01:12.484416+0000","last_became_active":"2024-04-26T23:01:04.712325+0000","last_became_peered":"2024-04-26T23:01:04.712325+0000","last_unstale":"2024-04-26T23:01:12.484416+0000","last_undegraded":"2024-04-26T23:01:12.484416+0000","last_fullsized":"2024-04-26T23:01:12.484416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:52.271211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485257+0000","last_change":"2024-04-26T23:01:10.385938+0000","last_active":"2024-04-26T23:01:12.485257+0000","last_peered":"2024-04-26T23:01:12.485257+0000","last_clean":"2024-04-26T23:01:12.485257+0000","last_became_active":"2024-04-26T23:01:10.384600+0000","last_became_peered":"2024-04-26T23:01:10.384600+0000","last_unstale":"2024-04-26T23:01:12.485257+0000","last_undegraded":"2024-04-26T23:01:12.485257+0000","last_fullsized":"2024-04-26T23:01:12.485257+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775962+0000","last_change":"2024-04-26T23:01:05.744208+0000","last_active":"2024-04-26T23:01:11.775962+0000","last_peered":"2024-04-26T23:01:11.775962+0000","last_clean":"2024-04-26T23:01:11.775962+0000","last_became_active":"2024-04-26T23:01:05.743897+0000","last_became_peered":"2024-04-26T23:01:05.743897+0000","last_unstale":"2024-04-26T23:01:11.775962+0000","last_undegraded":"2024-04-26T23:01:11.775962+0000","last_fullsized":"2024-04-26T23:01:11.775962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:22.679803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:25.136494+0000","last_change":"2024-04-26T23:01:04.723670+0000","last_active":"2024-04-26T23:01:25.136494+0000","last_peered":"2024-04-26T23:01:25.136494+0000","last_clean":"2024-04-26T23:01:25.136494+0000","last_became_active":"2024-04-26T23:01:04.723361+0000","last_became_peered":"2024-04-26T23:01:04.723361+0000","last_unstale":"2024-04-26T23:01:25.136494+0000","last_undegraded":"2024-04-26T23:01:25.136494+0000","last_fullsized":"2024-04-26T23:01:25.136494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:58.527150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485782+0000","last_change":"2024-04-26T23:01:09.799732+0000","last_active":"2024-04-26T23:01:12.485782+0000","last_peered":"2024-04-26T23:01:12.485782+0000","last_clean":"2024-04-26T23:01:12.485782+0000","last_became_active":"2024-04-26T23:01:09.797973+0000","last_became_peered":"2024-04-26T23:01:09.797973+0000","last_unstale":"2024-04-26T23:01:12.485782+0000","last_undegraded":"2024-04-26T23:01:12.485782+0000","last_fullsized":"2024-04-26T23:01:12.485782+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:41.540020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882483+0000","last_change":"2024-04-26T23:01:05.732675+0000","last_active":"2024-04-26T23:01:11.882483+0000","last_peered":"2024-04-26T23:01:11.882483+0000","last_clean":"2024-04-26T23:01:11.882483+0000","last_became_active":"2024-04-26T23:01:05.732334+0000","last_became_peered":"2024-04-26T23:01:05.732334+0000","last_unstale":"2024-04-26T23:01:11.882483+0000","last_undegraded":"2024-04-26T23:01:11.882483+0000","last_fullsized":"2024-04-26T23:01:11.882483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:46.537111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484886+0000","last_change":"2024-04-26T23:01:04.731957+0000","last_active":"2024-04-26T23:01:12.484886+0000","last_peered":"2024-04-26T23:01:12.484886+0000","last_clean":"2024-04-26T23:01:12.484886+0000","last_became_active":"2024-04-26T23:01:04.731688+0000","last_became_peered":"2024-04-26T23:01:04.731688+0000","last_unstale":"2024-04-26T23:01:12.484886+0000","last_undegraded":"2024-04-26T23:01:12.484886+0000","last_fullsized":"2024-04-26T23:01:12.484886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:16.098138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882537+0000","last_change":"2024-04-26T23:01:09.793585+0000","last_active":"2024-04-26T23:01:11.882537+0000","last_peered":"2024-04-26T23:01:11.882537+0000","last_clean":"2024-04-26T23:01:11.882537+0000","last_became_active":"2024-04-26T23:01:09.793177+0000","last_became_peered":"2024-04-26T23:01:09.793177+0000","last_unstale":"2024-04-26T23:01:11.882537+0000","last_undegraded":"2024-04-26T23:01:11.882537+0000","last_fullsized":"2024-04-26T23:01:11.882537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487118+0000","last_change":"2024-04-26T23:01:05.723191+0000","last_active":"2024-04-26T23:01:12.487118+0000","last_peered":"2024-04-26T23:01:12.487118+0000","last_clean":"2024-04-26T23:01:12.487118+0000","last_became_active":"2024-04-26T23:01:05.722903+0000","last_became_peered":"2024-04-26T23:01:05.722903+0000","last_unstale":"2024-04-26T23:01:12.487118+0000","last_undegraded":"2024-04-26T23:01:12.487118+0000","last_fullsized":"2024-04-26T23:01:12.487118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484699+0000","last_change":"2024-04-26T23:01:04.729093+0000","last_active":"2024-04-26T23:01:12.484699+0000","last_peered":"2024-04-26T23:01:12.484699+0000","last_clean":"2024-04-26T23:01:12.484699+0000","last_became_active":"2024-04-26T23:01:04.728125+0000","last_became_peered":"2024-04-26T23:01:04.728125+0000","last_unstale":"2024-04-26T23:01:12.484699+0000","last_undegraded":"2024-04-26T23:01:12.484699+0000","last_fullsized":"2024-04-26T23:01:12.484699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769989+0000","last_change":"2024-04-26T23:01:09.802623+0000","last_active":"2024-04-26T23:01:11.769989+0000","last_peered":"2024-04-26T23:01:11.769989+0000","last_clean":"2024-04-26T23:01:11.769989+0000","last_became_active":"2024-04-26T23:01:09.798323+0000","last_became_peered":"2024-04-26T23:01:09.798323+0000","last_unstale":"2024-04-26T23:01:11.769989+0000","last_undegraded":"2024-04-26T23:01:11.769989+0000","last_fullsized":"2024-04-26T23:01:11.769989+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:56.662302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484537+0000","last_change":"2024-04-26T23:01:05.727245+0000","last_active":"2024-04-26T23:01:12.484537+0000","last_peered":"2024-04-26T23:01:12.484537+0000","last_clean":"2024-04-26T23:01:12.484537+0000","last_became_active":"2024-04-26T23:01:05.726786+0000","last_became_peered":"2024-04-26T23:01:05.726786+0000","last_unstale":"2024-04-26T23:01:12.484537+0000","last_undegraded":"2024-04-26T23:01:12.484537+0000","last_fullsized":"2024-04-26T23:01:12.484537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:24.824202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778010+0000","last_change":"2024-04-26T23:01:04.732475+0000","last_active":"2024-04-26T23:01:11.778010+0000","last_peered":"2024-04-26T23:01:11.778010+0000","last_clean":"2024-04-26T23:01:11.778010+0000","last_became_active":"2024-04-26T23:01:04.732185+0000","last_became_peered":"2024-04-26T23:01:04.732185+0000","last_unstale":"2024-04-26T23:01:11.778010+0000","last_undegraded":"2024-04-26T23:01:11.778010+0000","last_fullsized":"2024-04-26T23:01:11.778010+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770229+0000","last_change":"2024-04-26T23:01:10.396108+0000","last_active":"2024-04-26T23:01:11.770229+0000","last_peered":"2024-04-26T23:01:11.770229+0000","last_clean":"2024-04-26T23:01:11.770229+0000","last_became_active":"2024-04-26T23:01:10.394701+0000","last_became_peered":"2024-04-26T23:01:10.394701+0000","last_unstale":"2024-04-26T23:01:11.770229+0000","last_undegraded":"2024-04-26T23:01:11.770229+0000","last_fullsized":"2024-04-26T23:01:11.770229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:00:11.874893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484827+0000","last_change":"2024-04-26T23:01:10.396863+0000","last_active":"2024-04-26T23:01:12.484827+0000","last_peered":"2024-04-26T23:01:12.484827+0000","last_clean":"2024-04-26T23:01:12.484827+0000","last_became_active":"2024-04-26T23:01:10.395914+0000","last_became_peered":"2024-04-26T23:01:10.395914+0000","last_unstale":"2024-04-26T23:01:12.484827+0000","last_undegraded":"2024-04-26T23:01:12.484827+0000","last_fullsized":"2024-04-26T23:01:12.484827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:30.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483687+0000","last_change":"2024-04-26T23:01:05.749537+0000","last_active":"2024-04-26T23:01:12.483687+0000","last_peered":"2024-04-26T23:01:12.483687+0000","last_clean":"2024-04-26T23:01:12.483687+0000","last_became_active":"2024-04-26T23:01:05.749254+0000","last_became_peered":"2024-04-26T23:01:05.749254+0000","last_unstale":"2024-04-26T23:01:12.483687+0000","last_undegraded":"2024-04-26T23:01:12.483687+0000","last_fullsized":"2024-04-26T23:01:12.483687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483804+0000","last_change":"2024-04-26T23:01:04.730472+0000","last_active":"2024-04-26T23:01:12.483804+0000","last_peered":"2024-04-26T23:01:12.483804+0000","last_clean":"2024-04-26T23:01:12.483804+0000","last_became_active":"2024-04-26T23:01:04.729294+0000","last_became_peered":"2024-04-26T23:01:04.729294+0000","last_unstale":"2024-04-26T23:01:12.483804+0000","last_undegraded":"2024-04-26T23:01:12.483804+0000","last_fullsized":"2024-04-26T23:01:12.483804+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:58.527150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778171+0000","last_change":"2024-04-26T23:01:09.800700+0000","last_active":"2024-04-26T23:01:11.778171+0000","last_peered":"2024-04-26T23:01:11.778171+0000","last_clean":"2024-04-26T23:01:11.778171+0000","last_became_active":"2024-04-26T23:01:09.799085+0000","last_became_peered":"2024-04-26T23:01:09.799085+0000","last_unstale":"2024-04-26T23:01:11.778171+0000","last_undegraded":"2024-04-26T23:01:11.778171+0000","last_fullsized":"2024-04-26T23:01:11.778171+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:59.067859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769853+0000","last_change":"2024-04-26T23:01:05.737423+0000","last_active":"2024-04-26T23:01:11.769853+0000","last_peered":"2024-04-26T23:01:11.769853+0000","last_clean":"2024-04-26T23:01:11.769853+0000","last_became_active":"2024-04-26T23:01:05.736881+0000","last_became_peered":"2024-04-26T23:01:05.736881+0000","last_unstale":"2024-04-26T23:01:11.769853+0000","last_undegraded":"2024-04-26T23:01:11.769853+0000","last_fullsized":"2024-04-26T23:01:11.769853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:35:08.020426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770314+0000","last_change":"2024-04-26T23:01:04.723415+0000","last_active":"2024-04-26T23:01:11.770314+0000","last_peered":"2024-04-26T23:01:11.770314+0000","last_clean":"2024-04-26T23:01:11.770314+0000","last_became_active":"2024-04-26T23:01:04.723109+0000","last_became_peered":"2024-04-26T23:01:04.723109+0000","last_unstale":"2024-04-26T23:01:11.770314+0000","last_undegraded":"2024-04-26T23:01:11.770314+0000","last_fullsized":"2024-04-26T23:01:11.770314+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:52.271211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775686+0000","last_change":"2024-04-26T23:01:04.712783+0000","last_active":"2024-04-26T23:01:11.775686+0000","last_peered":"2024-04-26T23:01:11.775686+0000","last_clean":"2024-04-26T23:01:11.775686+0000","last_became_active":"2024-04-26T23:01:04.712436+0000","last_became_peered":"2024-04-26T23:01:04.712436+0000","last_unstale":"2024-04-26T23:01:11.775686+0000","last_undegraded":"2024-04-26T23:01:11.775686+0000","last_fullsized":"2024-04-26T23:01:11.775686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484177+0000","last_change":"2024-04-26T23:01:05.742581+0000","last_active":"2024-04-26T23:01:12.484177+0000","last_peered":"2024-04-26T23:01:12.484177+0000","last_clean":"2024-04-26T23:01:12.484177+0000","last_became_active":"2024-04-26T23:01:05.740918+0000","last_became_peered":"2024-04-26T23:01:05.740918+0000","last_unstale":"2024-04-26T23:01:12.484177+0000","last_undegraded":"2024-04-26T23:01:12.484177+0000","last_fullsized":"2024-04-26T23:01:12.484177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:35:08.020426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.488392+0000","last_change":"2024-04-26T23:01:09.798160+0000","last_active":"2024-04-26T23:01:12.488392+0000","last_peered":"2024-04-26T23:01:12.488392+0000","last_clean":"2024-04-26T23:01:12.488392+0000","last_became_active":"2024-04-26T23:01:09.797746+0000","last_became_peered":"2024-04-26T23:01:09.797746+0000","last_unstale":"2024-04-26T23:01:12.488392+0000","last_undegraded":"2024-04-26T23:01:12.488392+0000","last_fullsized":"2024-04-26T23:01:12.488392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:11.186360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770724+0000","last_change":"2024-04-26T23:01:04.723482+0000","last_active":"2024-04-26T23:01:11.770724+0000","last_peered":"2024-04-26T23:01:11.770724+0000","last_clean":"2024-04-26T23:01:11.770724+0000","last_became_active":"2024-04-26T23:01:04.722816+0000","last_became_peered":"2024-04-26T23:01:04.722816+0000","last_unstale":"2024-04-26T23:01:11.770724+0000","last_undegraded":"2024-04-26T23:01:11.770724+0000","last_fullsized":"2024-04-26T23:01:11.770724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882061+0000","last_change":"2024-04-26T23:01:05.745436+0000","last_active":"2024-04-26T23:01:11.882061+0000","last_peered":"2024-04-26T23:01:11.882061+0000","last_clean":"2024-04-26T23:01:11.882061+0000","last_became_active":"2024-04-26T23:01:05.745148+0000","last_became_peered":"2024-04-26T23:01:05.745148+0000","last_unstale":"2024-04-26T23:01:11.882061+0000","last_undegraded":"2024-04-26T23:01:11.882061+0000","last_fullsized":"2024-04-26T23:01:11.882061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:17.111974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771111+0000","last_change":"2024-04-26T23:01:09.795297+0000","last_active":"2024-04-26T23:01:11.771111+0000","last_peered":"2024-04-26T23:01:11.771111+0000","last_clean":"2024-04-26T23:01:11.771111+0000","last_became_active":"2024-04-26T23:01:09.794873+0000","last_became_peered":"2024-04-26T23:01:09.794873+0000","last_unstale":"2024-04-26T23:01:11.771111+0000","last_undegraded":"2024-04-26T23:01:11.771111+0000","last_fullsized":"2024-04-26T23:01:11.771111+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:16:59.104896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.773782+0000","last_change":"2024-04-26T23:01:05.713935+0000","last_active":"2024-04-26T23:01:11.773782+0000","last_peered":"2024-04-26T23:01:11.773782+0000","last_clean":"2024-04-26T23:01:11.773782+0000","last_became_active":"2024-04-26T23:01:05.713606+0000","last_became_peered":"2024-04-26T23:01:05.713606+0000","last_unstale":"2024-04-26T23:01:11.773782+0000","last_undegraded":"2024-04-26T23:01:11.773782+0000","last_fullsized":"2024-04-26T23:01:11.773782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:26.157667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487634+0000","last_change":"2024-04-26T23:01:05.722889+0000","last_active":"2024-04-26T23:01:12.487634+0000","last_peered":"2024-04-26T23:01:12.487634+0000","last_clean":"2024-04-26T23:01:12.487634+0000","last_became_active":"2024-04-26T23:01:05.721634+0000","last_became_peered":"2024-04-26T23:01:05.721634+0000","last_unstale":"2024-04-26T23:01:12.487634+0000","last_undegraded":"2024-04-26T23:01:12.487634+0000","last_fullsized":"2024-04-26T23:01:12.487634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:46.537111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484476+0000","last_change":"2024-04-26T23:01:05.749096+0000","last_active":"2024-04-26T23:01:12.484476+0000","last_peered":"2024-04-26T23:01:12.484476+0000","last_clean":"2024-04-26T23:01:12.484476+0000","last_became_active":"2024-04-26T23:01:05.747770+0000","last_became_peered":"2024-04-26T23:01:05.747770+0000","last_unstale":"2024-04-26T23:01:12.484476+0000","last_undegraded":"2024-04-26T23:01:12.484476+0000","last_fullsized":"2024-04-26T23:01:12.484476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:57.106236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483913+0000","last_change":"2024-04-26T23:01:05.747157+0000","last_active":"2024-04-26T23:01:12.483913+0000","last_peered":"2024-04-26T23:01:12.483913+0000","last_clean":"2024-04-26T23:01:12.483913+0000","last_became_active":"2024-04-26T23:01:05.746805+0000","last_became_peered":"2024-04-26T23:01:05.746805+0000","last_unstale":"2024-04-26T23:01:12.483913+0000","last_undegraded":"2024-04-26T23:01:12.483913+0000","last_fullsized":"2024-04-26T23:01:12.483913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:07.115233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770959+0000","last_change":"2024-04-26T23:01:05.747433+0000","last_active":"2024-04-26T23:01:11.770959+0000","last_peered":"2024-04-26T23:01:11.770959+0000","last_clean":"2024-04-26T23:01:11.770959+0000","last_became_active":"2024-04-26T23:01:05.747108+0000","last_became_peered":"2024-04-26T23:01:05.747108+0000","last_unstale":"2024-04-26T23:01:11.770959+0000","last_undegraded":"2024-04-26T23:01:11.770959+0000","last_fullsized":"2024-04-26T23:01:11.770959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:57.106236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484633+0000","last_change":"2024-04-26T23:01:05.753692+0000","last_active":"2024-04-26T23:01:12.484633+0000","last_peered":"2024-04-26T23:01:12.484633+0000","last_clean":"2024-04-26T23:01:12.484633+0000","last_became_active":"2024-04-26T23:01:05.752474+0000","last_became_peered":"2024-04-26T23:01:05.752474+0000","last_unstale":"2024-04-26T23:01:12.484633+0000","last_undegraded":"2024-04-26T23:01:12.484633+0000","last_fullsized":"2024-04-26T23:01:12.484633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778393+0000","last_change":"2024-04-26T23:01:05.731062+0000","last_active":"2024-04-26T23:01:11.778393+0000","last_peered":"2024-04-26T23:01:11.778393+0000","last_clean":"2024-04-26T23:01:11.778393+0000","last_became_active":"2024-04-26T23:01:05.727960+0000","last_became_peered":"2024-04-26T23:01:05.727960+0000","last_unstale":"2024-04-26T23:01:11.778393+0000","last_undegraded":"2024-04-26T23:01:11.778393+0000","last_fullsized":"2024-04-26T23:01:11.778393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771192+0000","last_change":"2024-04-26T23:01:05.750232+0000","last_active":"2024-04-26T23:01:11.771192+0000","last_peered":"2024-04-26T23:01:11.771192+0000","last_clean":"2024-04-26T23:01:11.771192+0000","last_became_active":"2024-04-26T23:01:05.749885+0000","last_became_peered":"2024-04-26T23:01:05.749885+0000","last_unstale":"2024-04-26T23:01:11.771192+0000","last_undegraded":"2024-04-26T23:01:11.771192+0000","last_fullsized":"2024-04-26T23:01:11.771192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483726+0000","last_change":"2024-04-26T23:01:05.749914+0000","last_active":"2024-04-26T23:01:12.483726+0000","last_peered":"2024-04-26T23:01:12.483726+0000","last_clean":"2024-04-26T23:01:12.483726+0000","last_became_active":"2024-04-26T23:01:05.745797+0000","last_became_peered":"2024-04-26T23:01:05.745797+0000","last_unstale":"2024-04-26T23:01:12.483726+0000","last_undegraded":"2024-04-26T23:01:12.483726+0000","last_fullsized":"2024-04-26T23:01:12.483726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770217+0000","last_change":"2024-04-26T23:01:05.750181+0000","last_active":"2024-04-26T23:01:11.770217+0000","last_peered":"2024-04-26T23:01:11.770217+0000","last_clean":"2024-04-26T23:01:11.770217+0000","last_became_active":"2024-04-26T23:01:05.749810+0000","last_became_peered":"2024-04-26T23:01:05.749810+0000","last_unstale":"2024-04-26T23:01:11.770217+0000","last_undegraded":"2024-04-26T23:01:11.770217+0000","last_fullsized":"2024-04-26T23:01:11.770217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484216+0000","last_change":"2024-04-26T23:01:05.730857+0000","last_active":"2024-04-26T23:01:12.484216+0000","last_peered":"2024-04-26T23:01:12.484216+0000","last_clean":"2024-04-26T23:01:12.484216+0000","last_became_active":"2024-04-26T23:01:05.730543+0000","last_became_peered":"2024-04-26T23:01:05.730543+0000","last_unstale":"2024-04-26T23:01:12.484216+0000","last_undegraded":"2024-04-26T23:01:12.484216+0000","last_fullsized":"2024-04-26T23:01:12.484216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.482980+0000","last_change":"2024-04-26T23:01:05.741407+0000","last_active":"2024-04-26T23:01:12.482980+0000","last_peered":"2024-04-26T23:01:12.482980+0000","last_clean":"2024-04-26T23:01:12.482980+0000","last_became_active":"2024-04-26T23:01:05.740354+0000","last_became_peered":"2024-04-26T23:01:05.740354+0000","last_unstale":"2024-04-26T23:01:12.482980+0000","last_undegraded":"2024-04-26T23:01:12.482980+0000","last_fullsized":"2024-04-26T23:01:12.482980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:24:06.318595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777299+0000","last_change":"2024-04-26T23:01:05.730302+0000","last_active":"2024-04-26T23:01:11.777299+0000","last_peered":"2024-04-26T23:01:11.777299+0000","last_clean":"2024-04-26T23:01:11.777299+0000","last_became_active":"2024-04-26T23:01:05.729149+0000","last_became_peered":"2024-04-26T23:01:05.729149+0000","last_unstale":"2024-04-26T23:01:11.777299+0000","last_undegraded":"2024-04-26T23:01:11.777299+0000","last_fullsized":"2024-04-26T23:01:11.777299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776184+0000","last_change":"2024-04-26T23:01:05.731828+0000","last_active":"2024-04-26T23:01:11.776184+0000","last_peered":"2024-04-26T23:01:11.776184+0000","last_clean":"2024-04-26T23:01:11.776184+0000","last_became_active":"2024-04-26T23:01:05.731537+0000","last_became_peered":"2024-04-26T23:01:05.731537+0000","last_unstale":"2024-04-26T23:01:11.776184+0000","last_undegraded":"2024-04-26T23:01:11.776184+0000","last_fullsized":"2024-04-26T23:01:11.776184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:59:57.750603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487750+0000","last_change":"2024-04-26T23:01:05.726729+0000","last_active":"2024-04-26T23:01:12.487750+0000","last_peered":"2024-04-26T23:01:12.487750+0000","last_clean":"2024-04-26T23:01:12.487750+0000","last_became_active":"2024-04-26T23:01:05.726364+0000","last_became_peered":"2024-04-26T23:01:05.726364+0000","last_unstale":"2024-04-26T23:01:12.487750+0000","last_undegraded":"2024-04-26T23:01:12.487750+0000","last_fullsized":"2024-04-26T23:01:12.487750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:17.111974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485129+0000","last_change":"2024-04-26T23:01:05.728806+0000","last_active":"2024-04-26T23:01:12.485129+0000","last_peered":"2024-04-26T23:01:12.485129+0000","last_clean":"2024-04-26T23:01:12.485129+0000","last_became_active":"2024-04-26T23:01:05.728418+0000","last_became_peered":"2024-04-26T23:01:05.728418+0000","last_unstale":"2024-04-26T23:01:12.485129+0000","last_undegraded":"2024-04-26T23:01:12.485129+0000","last_fullsized":"2024-04-26T23:01:12.485129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:40.686396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483463+0000","last_change":"2024-04-26T23:01:05.721739+0000","last_active":"2024-04-26T23:01:12.483463+0000","last_peered":"2024-04-26T23:01:12.483463+0000","last_clean":"2024-04-26T23:01:12.483463+0000","last_became_active":"2024-04-26T23:01:05.721409+0000","last_became_peered":"2024-04-26T23:01:05.721409+0000","last_unstale":"2024-04-26T23:01:12.483463+0000","last_undegraded":"2024-04-26T23:01:12.483463+0000","last_fullsized":"2024-04-26T23:01:12.483463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483935+0000","last_change":"2024-04-26T23:01:05.739575+0000","last_active":"2024-04-26T23:01:12.483935+0000","last_peered":"2024-04-26T23:01:12.483935+0000","last_clean":"2024-04-26T23:01:12.483935+0000","last_became_active":"2024-04-26T23:01:05.734760+0000","last_became_peered":"2024-04-26T23:01:05.734760+0000","last_unstale":"2024-04-26T23:01:12.483935+0000","last_undegraded":"2024-04-26T23:01:12.483935+0000","last_fullsized":"2024-04-26T23:01:12.483935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483111+0000","last_change":"2024-04-26T23:01:05.732699+0000","last_active":"2024-04-26T23:01:12.483111+0000","last_peered":"2024-04-26T23:01:12.483111+0000","last_clean":"2024-04-26T23:01:12.483111+0000","last_became_active":"2024-04-26T23:01:05.732371+0000","last_became_peered":"2024-04-26T23:01:05.732371+0000","last_unstale":"2024-04-26T23:01:12.483111+0000","last_undegraded":"2024-04-26T23:01:12.483111+0000","last_fullsized":"2024-04-26T23:01:12.483111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770589+0000","last_change":"2024-04-26T23:01:05.745445+0000","last_active":"2024-04-26T23:01:11.770589+0000","last_peered":"2024-04-26T23:01:11.770589+0000","last_clean":"2024-04-26T23:01:11.770589+0000","last_became_active":"2024-04-26T23:01:05.742977+0000","last_became_peered":"2024-04-26T23:01:05.742977+0000","last_unstale":"2024-04-26T23:01:11.770589+0000","last_undegraded":"2024-04-26T23:01:11.770589+0000","last_fullsized":"2024-04-26T23:01:11.770589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882409+0000","last_change":"2024-04-26T23:01:05.739107+0000","last_active":"2024-04-26T23:01:11.882409+0000","last_peered":"2024-04-26T23:01:11.882409+0000","last_clean":"2024-04-26T23:01:11.882409+0000","last_became_active":"2024-04-26T23:01:05.738786+0000","last_became_peered":"2024-04-26T23:01:05.738786+0000","last_unstale":"2024-04-26T23:01:11.882409+0000","last_undegraded":"2024-04-26T23:01:11.882409+0000","last_fullsized":"2024-04-26T23:01:11.882409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:58.394848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778055+0000","last_change":"2024-04-26T23:01:05.710270+0000","last_active":"2024-04-26T23:01:11.778055+0000","last_peered":"2024-04-26T23:01:11.778055+0000","last_clean":"2024-04-26T23:01:11.778055+0000","last_became_active":"2024-04-26T23:01:05.709943+0000","last_became_peered":"2024-04-26T23:01:05.709943+0000","last_unstale":"2024-04-26T23:01:11.778055+0000","last_undegraded":"2024-04-26T23:01:11.778055+0000","last_fullsized":"2024-04-26T23:01:11.778055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778339+0000","last_change":"2024-04-26T23:01:05.717127+0000","last_active":"2024-04-26T23:01:11.778339+0000","last_peered":"2024-04-26T23:01:11.778339+0000","last_clean":"2024-04-26T23:01:11.778339+0000","last_became_active":"2024-04-26T23:01:05.716647+0000","last_became_peered":"2024-04-26T23:01:05.716647+0000","last_unstale":"2024-04-26T23:01:11.778339+0000","last_undegraded":"2024-04-26T23:01:11.778339+0000","last_fullsized":"2024-04-26T23:01:11.778339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776723+0000","last_change":"2024-04-26T23:01:05.730733+0000","last_active":"2024-04-26T23:01:11.776723+0000","last_peered":"2024-04-26T23:01:11.776723+0000","last_clean":"2024-04-26T23:01:11.776723+0000","last_became_active":"2024-04-26T23:01:05.727115+0000","last_became_peered":"2024-04-26T23:01:05.727115+0000","last_unstale":"2024-04-26T23:01:11.776723+0000","last_undegraded":"2024-04-26T23:01:11.776723+0000","last_fullsized":"2024-04-26T23:01:11.776723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:40.686396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769732+0000","last_change":"2024-04-26T23:01:05.737620+0000","last_active":"2024-04-26T23:01:11.769732+0000","last_peered":"2024-04-26T23:01:11.769732+0000","last_clean":"2024-04-26T23:01:11.769732+0000","last_became_active":"2024-04-26T23:01:05.737220+0000","last_became_peered":"2024-04-26T23:01:05.737220+0000","last_unstale":"2024-04-26T23:01:11.769732+0000","last_undegraded":"2024-04-26T23:01:11.769732+0000","last_fullsized":"2024-04-26T23:01:11.769732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:07.115233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777054+0000","last_change":"2024-04-26T23:01:05.731417+0000","last_active":"2024-04-26T23:01:11.777054+0000","last_peered":"2024-04-26T23:01:11.777054+0000","last_clean":"2024-04-26T23:01:11.777054+0000","last_became_active":"2024-04-26T23:01:05.730556+0000","last_became_peered":"2024-04-26T23:01:05.730556+0000","last_unstale":"2024-04-26T23:01:11.777054+0000","last_undegraded":"2024-04-26T23:01:11.777054+0000","last_fullsized":"2024-04-26T23:01:11.777054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:13.482334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483844+0000","last_change":"2024-04-26T23:01:05.721675+0000","last_active":"2024-04-26T23:01:12.483844+0000","last_peered":"2024-04-26T23:01:12.483844+0000","last_clean":"2024-04-26T23:01:12.483844+0000","last_became_active":"2024-04-26T23:01:05.721363+0000","last_became_peered":"2024-04-26T23:01:05.721363+0000","last_unstale":"2024-04-26T23:01:12.483844+0000","last_undegraded":"2024-04-26T23:01:12.483844+0000","last_fullsized":"2024-04-26T23:01:12.483844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:13.482334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776212+0000","last_change":"2024-04-26T23:01:05.729610+0000","last_active":"2024-04-26T23:01:11.776212+0000","last_peered":"2024-04-26T23:01:11.776212+0000","last_clean":"2024-04-26T23:01:11.776212+0000","last_became_active":"2024-04-26T23:01:05.729307+0000","last_became_peered":"2024-04-26T23:01:05.729307+0000","last_unstale":"2024-04-26T23:01:11.776212+0000","last_undegraded":"2024-04-26T23:01:11.776212+0000","last_fullsized":"2024-04-26T23:01:11.776212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:57.106236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484169+0000","last_change":"2024-04-26T23:01:05.753873+0000","last_active":"2024-04-26T23:01:12.484169+0000","last_peered":"2024-04-26T23:01:12.484169+0000","last_clean":"2024-04-26T23:01:12.484169+0000","last_became_active":"2024-04-26T23:01:05.753603+0000","last_became_peered":"2024-04-26T23:01:05.753603+0000","last_unstale":"2024-04-26T23:01:12.484169+0000","last_undegraded":"2024-04-26T23:01:12.484169+0000","last_fullsized":"2024-04-26T23:01:12.484169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:22:15.301548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484214+0000","last_change":"2024-04-26T23:01:05.720420+0000","last_active":"2024-04-26T23:01:12.484214+0000","last_peered":"2024-04-26T23:01:12.484214+0000","last_clean":"2024-04-26T23:01:12.484214+0000","last_became_active":"2024-04-26T23:01:05.720114+0000","last_became_peered":"2024-04-26T23:01:05.720114+0000","last_unstale":"2024-04-26T23:01:12.484214+0000","last_undegraded":"2024-04-26T23:01:12.484214+0000","last_fullsized":"2024-04-26T23:01:12.484214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486621+0000","last_change":"2024-04-26T23:01:05.730995+0000","last_active":"2024-04-26T23:01:12.486621+0000","last_peered":"2024-04-26T23:01:12.486621+0000","last_clean":"2024-04-26T23:01:12.486621+0000","last_became_active":"2024-04-26T23:01:05.730644+0000","last_became_peered":"2024-04-26T23:01:05.730644+0000","last_unstale":"2024-04-26T23:01:12.486621+0000","last_undegraded":"2024-04-26T23:01:12.486621+0000","last_fullsized":"2024-04-26T23:01:12.486621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776188+0000","last_change":"2024-04-26T23:01:05.727391+0000","last_active":"2024-04-26T23:01:11.776188+0000","last_peered":"2024-04-26T23:01:11.776188+0000","last_clean":"2024-04-26T23:01:11.776188+0000","last_became_active":"2024-04-26T23:01:05.726729+0000","last_became_peered":"2024-04-26T23:01:05.726729+0000","last_unstale":"2024-04-26T23:01:11.776188+0000","last_undegraded":"2024-04-26T23:01:11.776188+0000","last_fullsized":"2024-04-26T23:01:11.776188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:24.824202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:01:35.931 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T23:01:35.931 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T23:01:35.931 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-26T23:01:35.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-26T23:01:35.932 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-26T23:01:36.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.180+0000 7f676ea00640 1 Processor -- start 2024-04-26T23:01:36.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.180+0000 7f676ea00640 1 -- start start 2024-04-26T23:01:36.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.181+0000 7f676ea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67680ce5e0 0x7f67680afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.182+0000 7f676ea00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67680af7b0 con 0x7f67680cc590 2024-04-26T23:01:36.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.182+0000 7f676ea00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67680afa90 con 0x7f676815f2d0 2024-04-26T23:01:36.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.182+0000 7f676ea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f67680cbd90 con 0x7f67680ce5e0 2024-04-26T23:01:36.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.182+0000 7f676e000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f67680cc590 0x7f676806f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41452/0 (socket says 172.21.15.40:41452) 2024-04-26T23:01:36.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.182+0000 7f676e000640 1 -- 172.21.15.40:0/405539711 learned_addr learned my addr 172.21.15.40:0/405539711 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:36.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.182+0000 7f676cc00640 1 --2- 172.21.15.40:0/405539711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67680ce5e0 0x7f67680afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3285992405 0 0) 0x7f67680af7b0 con 0x7f67680cc590 2024-04-26T23:01:36.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6750005730 con 0x7f67680cc590 2024-04-26T23:01:36.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1615023603 0 0) 0x7f6750005730 con 0x7f67680cc590 2024-04-26T23:01:36.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67680ce5e0 msgr2=0x7f67680afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:36.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 --2- 172.21.15.40:0/405539711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67680ce5e0 0x7f67680afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 >> v1:172.21.15.103:6789/0 conn(0x7f676815f2d0 legacy=0x7f67680c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67680cc070 con 0x7f67680cc590 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f67600019f0 con 0x7f67680cc590 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 >> v1:172.21.15.40:6789/0 conn(0x7f67680cc590 legacy=0x7f676806f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 --2- 172.21.15.40:0/405539711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67680ce5e0 0x7f6750005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 --2- 172.21.15.40:0/405539711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67500061c0 0x7f6750006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 --2- 172.21.15.40:0/405539711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6750006f50 0x7f67500073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f67680cbd90 con 0x7f67500061c0 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f675000dd10 con 0x7f6750006f50 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.183+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f675000df10 con 0x7f67680ce5e0 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.184+0000 7f676cc00640 1 --2- 172.21.15.40:0/405539711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67500061c0 0x7f6750006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.184+0000 7f676d600640 1 --2- 172.21.15.40:0/405539711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67680ce5e0 0x7f6750005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.184+0000 7f676e000640 1 --2- 172.21.15.40:0/405539711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6750006f50 0x7f67500073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.184+0000 7f676cc00640 1 -- 172.21.15.40:0/405539711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67680ce5e0 msgr2=0x7f6750005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.184+0000 7f676cc00640 1 --2- 172.21.15.40:0/405539711 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67680ce5e0 0x7f6750005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.184+0000 7f676cc00640 1 -- 172.21.15.40:0/405539711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6750006f50 msgr2=0x7f67500073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.184+0000 7f676cc00640 1 --2- 172.21.15.40:0/405539711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6750006f50 0x7f67500073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.184+0000 7f676cc00640 1 -- 172.21.15.40:0/405539711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6750005730 con 0x7f67500061c0 2024-04-26T23:01:36.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.185+0000 7f676cc00640 1 --2- 172.21.15.40:0/405539711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67500061c0 0x7f6750006a10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6754009ac0 tx=0x7f6750007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e78f934d67fe1e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:36.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.186+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754036070 con 0x7f67500061c0 2024-04-26T23:01:36.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.186+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6754031400 con 0x7f67500061c0 2024-04-26T23:01:36.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.186+0000 7f6767e00640 1 -- 172.21.15.40:0/405539711 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754009690 con 0x7f67500061c0 2024-04-26T23:01:36.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.186+0000 7f676ea00640 1 -- 172.21.15.40:0/405539711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67500061c0 msgr2=0x7f6750006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:36.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.186+0000 7f676ea00640 1 --2- 172.21.15.40:0/405539711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67500061c0 0x7f6750006a10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6754009ac0 tx=0x7f6750007a40 comp rx=0 tx=0).stop 2024-04-26T23:01:36.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.186+0000 7f676e000640 1 -- 172.21.15.40:0/405539711 reap_dead start 2024-04-26T23:01:36.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.186+0000 7f676ea00640 1 -- 172.21.15.40:0/405539711 shutdown_connections 2024-04-26T23:01:36.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.186+0000 7f676ea00640 1 -- 172.21.15.40:0/405539711 >> 172.21.15.40:0/405539711 conn(0x7f67680d0a10 msgr2=0x7f67680d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:36.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.186+0000 7f676ea00640 1 -- 172.21.15.40:0/405539711 shutdown_connections 2024-04-26T23:01:36.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.187+0000 7f676ea00640 1 -- 172.21.15.40:0/405539711 wait complete. 2024-04-26T23:01:36.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.188+0000 7f676ea00640 1 Processor -- start 2024-04-26T23:01:36.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.189+0000 7f676ea00640 1 -- start start 2024-04-26T23:01:36.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.190+0000 7f676ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67500061c0 0x7f6768151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.190+0000 7f676ea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6768151d80 0x7f67680c9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.190+0000 7f676ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67680c8480 0x7f6768073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.190+0000 7f676ea00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f67680c5fc0 con 0x7f67680c8480 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.190+0000 7f676ea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f676806e270 con 0x7f67500061c0 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.190+0000 7f676ea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f67680c7ff0 con 0x7f6768151d80 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.190+0000 7f676e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67680c8480 0x7f6768073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.190+0000 7f676e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67680c8480 0x7f6768073270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38680/0 (socket says 172.21.15.40:38680) 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.190+0000 7f676e000640 1 -- 172.21.15.40:0/3134045653 learned_addr learned my addr 172.21.15.40:0/3134045653 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.190+0000 7f676cc00640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6768151d80 0x7f67680c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.190+0000 7f676d600640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67500061c0 0x7f6768151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.191+0000 7f676e000640 1 -- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6768151d80 msgr2=0x7f67680c9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.191+0000 7f676e000640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6768151d80 0x7f67680c9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.191+0000 7f676e000640 1 -- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67500061c0 msgr2=0x7f6768151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.191+0000 7f676e000640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67500061c0 0x7f6768151640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.191+0000 7f676e000640 1 -- 172.21.15.40:0/3134045653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6768151b80 con 0x7f67680c8480 2024-04-26T23:01:36.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.191+0000 7f676e000640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67680c8480 0x7f6768073270 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f67680d1b10 tx=0x7f676000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:36.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.191+0000 7f6765600640 1 -- 172.21.15.40:0/3134045653 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676000ec10 con 0x7f67680c8480 2024-04-26T23:01:36.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.191+0000 7f6765600640 1 -- 172.21.15.40:0/3134045653 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f676000edb0 con 0x7f67680c8480 2024-04-26T23:01:36.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.191+0000 7f6765600640 1 -- 172.21.15.40:0/3134045653 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6760014630 con 0x7f67680c8480 2024-04-26T23:01:36.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.191+0000 7f676ea00640 1 -- 172.21.15.40:0/3134045653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67680c9c70 con 0x7f67680c8480 2024-04-26T23:01:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.191+0000 7f676ea00640 1 -- 172.21.15.40:0/3134045653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67680ca1f0 con 0x7f67680c8480 2024-04-26T23:01:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.193+0000 7f676ea00640 1 -- 172.21.15.40:0/3134045653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6768069370 con 0x7f67680c8480 2024-04-26T23:01:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.194+0000 7f6765600640 1 -- 172.21.15.40:0/3134045653 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f67600147d0 con 0x7f67680c8480 2024-04-26T23:01:36.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.195+0000 7f6765600640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f674806c920 0x7f674806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.195+0000 7f6765600640 1 -- 172.21.15.40:0/3134045653 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f676008bd40 con 0x7f67680c8480 2024-04-26T23:01:36.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.195+0000 7f676d600640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f674806c920 0x7f674806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.196+0000 7f676d600640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f674806c920 0x7f674806ede0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f675c000c00 tx=0x7f675c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:36.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.199+0000 7f6765600640 1 -- 172.21.15.40:0/3134045653 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f6768151b80 con 0x7f67680c8480 2024-04-26T23:01:36.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.453+0000 7f676ea00640 1 -- 172.21.15.40:0/3134045653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f67680c78a0 con 0x7f67680c8480 2024-04-26T23:01:36.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.458+0000 7f6765600640 1 -- 172.21.15.40:0/3134045653 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+174144 (secure 0 0 0) 0x7f6760059da0 con 0x7f67680c8480 2024-04-26T23:01:36.460 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:36.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.462+0000 7f676ea00640 1 -- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f674806c920 msgr2=0x7f674806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:36.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.462+0000 7f676ea00640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f674806c920 0x7f674806ede0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f675c000c00 tx=0x7f675c009040 comp rx=0 tx=0).stop 2024-04-26T23:01:36.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.462+0000 7f676ea00640 1 -- 172.21.15.40:0/3134045653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67680c8480 msgr2=0x7f6768073270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:36.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.462+0000 7f676ea00640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67680c8480 0x7f6768073270 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f67680d1b10 tx=0x7f676000e530 comp rx=0 tx=0).stop 2024-04-26T23:01:36.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.462+0000 7f676ea00640 1 -- 172.21.15.40:0/3134045653 shutdown_connections 2024-04-26T23:01:36.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.463+0000 7f676ea00640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f674806c920 0x7f674806ede0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.463+0000 7f676ea00640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67680c8480 0x7f6768073270 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.463+0000 7f676ea00640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6768151d80 0x7f67680c9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.463+0000 7f676ea00640 1 --2- 172.21.15.40:0/3134045653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67500061c0 0x7f6768151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.463+0000 7f676ea00640 1 -- 172.21.15.40:0/3134045653 >> 172.21.15.40:0/3134045653 conn(0x7f67680d0a10 msgr2=0x7f676806e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:36.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.463+0000 7f676ea00640 1 -- 172.21.15.40:0/3134045653 shutdown_connections 2024-04-26T23:01:36.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.463+0000 7f676ea00640 1 -- 172.21.15.40:0/3134045653 wait complete. 2024-04-26T23:01:36.483 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":5,"flags":0,"active_gid":4111,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1151751020},{"type":"v1","addr":"172.21.15.103:6833","nonce":1151751020}]},"active_addr":"172.21.15.103:6833/1151751020","active_change":"2024-04-26T23:00:45.120539+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4108,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":2637926314}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":1583486003}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":1729674856}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":1110698999}]}]} 2024-04-26T23:01:36.489 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-26T23:01:36.490 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-26T23:01:36.490 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:36.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.695+0000 7fd827200640 1 Processor -- start 2024-04-26T23:01:36.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.695+0000 7fd827200640 1 -- start start 2024-04-26T23:01:36.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.696+0000 7fd827200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd828145870 0x7fd828070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.696+0000 7fd827200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd8280b4b10 0x7fd8280b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.697+0000 7fd827200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8280b54d0 con 0x7fd8280b6b20 2024-04-26T23:01:36.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.697+0000 7fd827200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd82806f540 con 0x7fd8280b4b10 2024-04-26T23:01:36.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.697+0000 7fd827200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd82806f7a0 con 0x7fd828145870 2024-04-26T23:01:36.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.697+0000 7fd825e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd828145870 0x7fd828070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.697+0000 7fd825e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd828145870 0x7fd828070840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:46694/0 (socket says 172.21.15.40:46694) 2024-04-26T23:01:36.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.697+0000 7fd825e00640 1 -- 172.21.15.40:0/3228128053 learned_addr learned my addr 172.21.15.40:0/3228128053 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:36.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.697+0000 7fd826800640 1 --2- 172.21.15.40:0/3228128053 >> v2:172.21.15.103:3300/0 conn(0x7fd8280b4b10 0x7fd8280b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.698+0000 7fd825e00640 1 -- 172.21.15.40:0/3228128053 >> v2:172.21.15.103:3300/0 conn(0x7fd8280b4b10 msgr2=0x7fd8280b4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:36.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.698+0000 7fd825e00640 1 --2- 172.21.15.40:0/3228128053 >> v2:172.21.15.103:3300/0 conn(0x7fd8280b4b10 0x7fd8280b4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.698+0000 7fd824a00640 1 -- 172.21.15.40:0/3228128053 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 309027509 0 0) 0x7fd8280b54d0 con 0x7fd8280b6b20 2024-04-26T23:01:36.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.698+0000 7fd825e00640 1 -- 172.21.15.40:0/3228128053 >> v1:172.21.15.103:6789/0 conn(0x7fd8280b6b20 legacy=0x7fd82806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:36.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.698+0000 7fd825e00640 1 -- 172.21.15.40:0/3228128053 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd82806fa80 con 0x7fd828145870 2024-04-26T23:01:36.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.700+0000 7fd825e00640 1 --2- 172.21.15.40:0/3228128053 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd828145870 0x7fd828070840 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd818009870 tx=0x7fd81802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89c09d54623210c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:36.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.700+0000 7fd824a00640 1 -- 172.21.15.40:0/3228128053 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818037070 con 0x7fd828145870 2024-04-26T23:01:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.700+0000 7fd824a00640 1 -- 172.21.15.40:0/3228128053 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd818032440 con 0x7fd828145870 2024-04-26T23:01:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.700+0000 7fd824a00640 1 -- 172.21.15.40:0/3228128053 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818032810 con 0x7fd828145870 2024-04-26T23:01:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.701+0000 7fd827200640 1 -- 172.21.15.40:0/3228128053 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd828145870 msgr2=0x7fd828070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.701+0000 7fd827200640 1 --2- 172.21.15.40:0/3228128053 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd828145870 0x7fd828070840 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd818009870 tx=0x7fd81802a270 comp rx=0 tx=0).stop 2024-04-26T23:01:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.701+0000 7fd827200640 1 -- 172.21.15.40:0/3228128053 shutdown_connections 2024-04-26T23:01:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.701+0000 7fd827200640 1 --2- 172.21.15.40:0/3228128053 >> v2:172.21.15.103:3300/0 conn(0x7fd8280b4b10 0x7fd8280b4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.701+0000 7fd827200640 1 --2- 172.21.15.40:0/3228128053 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd828145870 0x7fd828070840 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.701+0000 7fd827200640 1 -- 172.21.15.40:0/3228128053 >> 172.21.15.40:0/3228128053 conn(0x7fd8280b3810 msgr2=0x7fd8280b3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.701+0000 7fd827200640 1 -- 172.21.15.40:0/3228128053 shutdown_connections 2024-04-26T23:01:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.701+0000 7fd827200640 1 -- 172.21.15.40:0/3228128053 wait complete. 2024-04-26T23:01:36.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.703+0000 7fd827200640 1 Processor -- start 2024-04-26T23:01:36.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.703+0000 7fd827200640 1 -- start start 2024-04-26T23:01:36.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.704+0000 7fd827200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280b4b10 0x7fd8280c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.704+0000 7fd827200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8280b6b20 0x7fd828142520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.704+0000 7fd827200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd828145870 0x7fd8281f46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.704+0000 7fd827200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd828070320 con 0x7fd8280b4b10 2024-04-26T23:01:36.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.704+0000 7fd827200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd828070040 con 0x7fd828145870 2024-04-26T23:01:36.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.704+0000 7fd827200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd8280ae8c0 con 0x7fd8280b6b20 2024-04-26T23:01:36.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.705+0000 7fd825e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280b4b10 0x7fd8280c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.705+0000 7fd825e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280b4b10 0x7fd8280c6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38684/0 (socket says 172.21.15.40:38684) 2024-04-26T23:01:36.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.705+0000 7fd825e00640 1 -- 172.21.15.40:0/3283604990 learned_addr learned my addr 172.21.15.40:0/3283604990 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:36.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.705+0000 7fd826800640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd828145870 0x7fd8281f46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.705+0000 7fd825400640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8280b6b20 0x7fd828142520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.705+0000 7fd825e00640 1 -- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8280b6b20 msgr2=0x7fd828142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.705+0000 7fd825e00640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8280b6b20 0x7fd828142520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.705+0000 7fd825e00640 1 -- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd828145870 msgr2=0x7fd8281f46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.705+0000 7fd825e00640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd828145870 0x7fd8281f46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.705+0000 7fd825e00640 1 -- 172.21.15.40:0/3283604990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8280c70f0 con 0x7fd8280b4b10 2024-04-26T23:01:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.705+0000 7fd826800640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd828145870 0x7fd8281f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:01:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.706+0000 7fd825e00640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280b4b10 0x7fd8280c6a80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd818009870 tx=0x7fd8180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.706+0000 7fd81e000640 1 -- 172.21.15.40:0/3283604990 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818041020 con 0x7fd8280b4b10 2024-04-26T23:01:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.706+0000 7fd81e000640 1 -- 172.21.15.40:0/3283604990 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd81803d6e0 con 0x7fd8280b4b10 2024-04-26T23:01:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.706+0000 7fd81e000640 1 -- 172.21.15.40:0/3283604990 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818037040 con 0x7fd8280b4b10 2024-04-26T23:01:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.706+0000 7fd827200640 1 -- 172.21.15.40:0/3283604990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8280c73d0 con 0x7fd8280b4b10 2024-04-26T23:01:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.706+0000 7fd827200640 1 -- 172.21.15.40:0/3283604990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8280c7990 con 0x7fd8280b4b10 2024-04-26T23:01:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.707+0000 7fd827200640 1 -- 172.21.15.40:0/3283604990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd828069a80 con 0x7fd8280b4b10 2024-04-26T23:01:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.708+0000 7fd81e000640 1 -- 172.21.15.40:0/3283604990 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd818044020 con 0x7fd8280b4b10 2024-04-26T23:01:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.709+0000 7fd81e000640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd80406c8f0 0x7fd80406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.710+0000 7fd81e000640 1 -- 172.21.15.40:0/3283604990 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd8180ac620 con 0x7fd8280b4b10 2024-04-26T23:01:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.710+0000 7fd825400640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd80406c8f0 0x7fd80406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.711+0000 7fd825400640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd80406c8f0 0x7fd80406edb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd8280d15e0 tx=0x7fd810009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.715+0000 7fd81e000640 1 -- 172.21.15.40:0/3283604990 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd81804b030 con 0x7fd8280b4b10 2024-04-26T23:01:36.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.923+0000 7fd827200640 1 -- 172.21.15.40:0/3283604990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd8280b2500 con 0x7fd8280b4b10 2024-04-26T23:01:36.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.925+0000 7fd81e000640 1 -- 172.21.15.40:0/3283604990 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7fd81807a720 con 0x7fd8280b4b10 2024-04-26T23:01:36.926 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:36.926 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:11.763965+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:36.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.928+0000 7fd827200640 1 -- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd80406c8f0 msgr2=0x7fd80406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:36.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.928+0000 7fd827200640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd80406c8f0 0x7fd80406edb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd8280d15e0 tx=0x7fd810009040 comp rx=0 tx=0).stop 2024-04-26T23:01:36.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.928+0000 7fd827200640 1 -- 172.21.15.40:0/3283604990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280b4b10 msgr2=0x7fd8280c6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:36.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.928+0000 7fd827200640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280b4b10 0x7fd8280c6a80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd818009870 tx=0x7fd8180046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.928+0000 7fd827200640 1 -- 172.21.15.40:0/3283604990 shutdown_connections 2024-04-26T23:01:36.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.928+0000 7fd827200640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd80406c8f0 0x7fd80406edb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.928+0000 7fd827200640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd828145870 0x7fd8281f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.928+0000 7fd827200640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8280b6b20 0x7fd828142520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.929+0000 7fd827200640 1 --2- 172.21.15.40:0/3283604990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280b4b10 0x7fd8280c6a80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:36.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.929+0000 7fd827200640 1 -- 172.21.15.40:0/3283604990 >> 172.21.15.40:0/3283604990 conn(0x7fd8280b3810 msgr2=0x7fd8280b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:36.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.929+0000 7fd827200640 1 -- 172.21.15.40:0/3283604990 shutdown_connections 2024-04-26T23:01:36.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:36.929+0000 7fd827200640 1 -- 172.21.15.40:0/3283604990 wait complete. 2024-04-26T23:01:36.950 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-26T23:01:36.950 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:37.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.151+0000 7f2449400640 1 Processor -- start 2024-04-26T23:01:37.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.152+0000 7f2449400640 1 -- start start 2024-04-26T23:01:37.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.153+0000 7f2449400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f24440b6220 0x7f24440d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.153+0000 7f2449400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f24440d39f0 0x7f24440b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.153+0000 7f2449400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2444162c40 con 0x7f24440d08d0 2024-04-26T23:01:37.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.153+0000 7f2449400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f24440d2d80 con 0x7f24440d39f0 2024-04-26T23:01:37.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.153+0000 7f2449400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f24440b7c80 con 0x7f24440b6220 2024-04-26T23:01:37.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.154+0000 7f243b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f24440d08d0 0x7f24440d32e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41466/0 (socket says 172.21.15.40:41466) 2024-04-26T23:01:37.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.154+0000 7f243b400640 1 -- 172.21.15.40:0/3585685224 learned_addr learned my addr 172.21.15.40:0/3585685224 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:37.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.154+0000 7f2448a00640 1 --2- 172.21.15.40:0/3585685224 >> v2:172.21.15.40:3300/0 conn(0x7f24440d39f0 0x7f24440b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.154+0000 7f2443e00640 1 --2- 172.21.15.40:0/3585685224 >> v2:172.21.15.103:3300/0 conn(0x7f24440b6220 0x7f24440d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.154+0000 7f2448a00640 1 -- 172.21.15.40:0/3585685224 >> v2:172.21.15.103:3300/0 conn(0x7f24440b6220 msgr2=0x7f24440d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.154+0000 7f2448a00640 1 --2- 172.21.15.40:0/3585685224 >> v2:172.21.15.103:3300/0 conn(0x7f24440b6220 0x7f24440d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.154+0000 7f2448a00640 1 -- 172.21.15.40:0/3585685224 >> v1:172.21.15.40:6789/0 conn(0x7f24440d08d0 legacy=0x7f24440d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.154+0000 7f2448a00640 1 -- 172.21.15.40:0/3585685224 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24440b7f60 con 0x7f24440d39f0 2024-04-26T23:01:37.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.155+0000 7f2448a00640 1 --2- 172.21.15.40:0/3585685224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24440d39f0 0x7f24440b8d20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f24440d4cf0 tx=0x7f243c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1083fa1a4f57016 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.156+0000 7f2443400640 1 -- 172.21.15.40:0/3585685224 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c007740 con 0x7f24440d39f0 2024-04-26T23:01:37.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.156+0000 7f2443400640 1 -- 172.21.15.40:0/3585685224 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f243c0078e0 con 0x7f24440d39f0 2024-04-26T23:01:37.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.156+0000 7f2443400640 1 -- 172.21.15.40:0/3585685224 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c007cb0 con 0x7f24440d39f0 2024-04-26T23:01:37.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.156+0000 7f2449400640 1 -- 172.21.15.40:0/3585685224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24440d39f0 msgr2=0x7f24440b8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.156+0000 7f2449400640 1 --2- 172.21.15.40:0/3585685224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24440d39f0 0x7f24440b8d20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f24440d4cf0 tx=0x7f243c02a680 comp rx=0 tx=0).stop 2024-04-26T23:01:37.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.156+0000 7f2449400640 1 -- 172.21.15.40:0/3585685224 shutdown_connections 2024-04-26T23:01:37.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.156+0000 7f2449400640 1 --2- 172.21.15.40:0/3585685224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24440d39f0 0x7f24440b8d20 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.156+0000 7f2449400640 1 --2- 172.21.15.40:0/3585685224 >> v2:172.21.15.103:3300/0 conn(0x7f24440b6220 0x7f24440d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.156+0000 7f2449400640 1 -- 172.21.15.40:0/3585685224 >> 172.21.15.40:0/3585685224 conn(0x7f24440d4d90 msgr2=0x7f24440d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.157+0000 7f2449400640 1 -- 172.21.15.40:0/3585685224 shutdown_connections 2024-04-26T23:01:37.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.157+0000 7f2449400640 1 -- 172.21.15.40:0/3585685224 wait complete. 2024-04-26T23:01:37.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.158+0000 7f2449400640 1 Processor -- start 2024-04-26T23:01:37.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.159+0000 7f2449400640 1 -- start start 2024-04-26T23:01:37.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f2449400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24440b6220 0x7f24440a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f2449400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24440d08d0 0x7f244415f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f2449400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440d39f0 0x7f244415be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f2449400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f24440b9260 con 0x7f24440b6220 2024-04-26T23:01:37.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f2449400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24440b8520 con 0x7f24440d08d0 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f2449400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f24440b93e0 con 0x7f24440d39f0 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f2443e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24440b6220 0x7f24440a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f2443e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24440b6220 0x7f24440a2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38692/0 (socket says 172.21.15.40:38692) 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f2443e00640 1 -- 172.21.15.40:0/3037479627 learned_addr learned my addr 172.21.15.40:0/3037479627 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f2443e00640 1 -- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440d39f0 msgr2=0x7f244415be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f2448a00640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440d39f0 0x7f244415be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f2443e00640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440d39f0 0x7f244415be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f243b400640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24440d08d0 0x7f244415f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.160+0000 7f2443e00640 1 -- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24440d08d0 msgr2=0x7f244415f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.161+0000 7f2443e00640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24440d08d0 0x7f244415f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.161+0000 7f2443e00640 1 -- 172.21.15.40:0/3037479627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244415c370 con 0x7f24440b6220 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.161+0000 7f2443e00640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24440b6220 0x7f24440a2230 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f243400d830 tx=0x7f243400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.161+0000 7f2440c00640 1 -- 172.21.15.40:0/3037479627 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243400b660 con 0x7f24440b6220 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.161+0000 7f2440c00640 1 -- 172.21.15.40:0/3037479627 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2434019070 con 0x7f24440b6220 2024-04-26T23:01:37.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.161+0000 7f2440c00640 1 -- 172.21.15.40:0/3037479627 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243400ba20 con 0x7f24440b6220 2024-04-26T23:01:37.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.161+0000 7f2449400640 1 -- 172.21.15.40:0/3037479627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24441f4f90 con 0x7f24440b6220 2024-04-26T23:01:37.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.161+0000 7f2449400640 1 -- 172.21.15.40:0/3037479627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f244409ed70 con 0x7f24440b6220 2024-04-26T23:01:37.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.162+0000 7f2449400640 1 -- 172.21.15.40:0/3037479627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24440d5b50 con 0x7f24440b6220 2024-04-26T23:01:37.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.163+0000 7f2440c00640 1 -- 172.21.15.40:0/3037479627 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f243401e460 con 0x7f24440b6220 2024-04-26T23:01:37.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.164+0000 7f2440c00640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f242006c8f0 0x7f242006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.164+0000 7f2440c00640 1 -- 172.21.15.40:0/3037479627 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f243408c640 con 0x7f24440b6220 2024-04-26T23:01:37.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.165+0000 7f243b400640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f242006c8f0 0x7f242006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.166+0000 7f243b400640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f242006c8f0 0x7f242006edb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f24440d4d60 tx=0x7f242c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.169+0000 7f2440c00640 1 -- 172.21.15.40:0/3037479627 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f24440d5b50 con 0x7f24440b6220 2024-04-26T23:01:37.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.372+0000 7f2449400640 1 -- 172.21.15.40:0/3037479627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2444069370 con 0x7f24440b6220 2024-04-26T23:01:37.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.374+0000 7f2440c00640 1 -- 172.21.15.40:0/3037479627 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f24441f4f90 con 0x7f24440b6220 2024-04-26T23:01:37.375 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:37.375 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:11.763965+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:37.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.377+0000 7f2449400640 1 -- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f242006c8f0 msgr2=0x7f242006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.377+0000 7f2449400640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f242006c8f0 0x7f242006edb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f24440d4d60 tx=0x7f242c009040 comp rx=0 tx=0).stop 2024-04-26T23:01:37.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.377+0000 7f2449400640 1 -- 172.21.15.40:0/3037479627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24440b6220 msgr2=0x7f24440a2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.377+0000 7f2449400640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24440b6220 0x7f24440a2230 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f243400d830 tx=0x7f243400dcf0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.377+0000 7f2449400640 1 -- 172.21.15.40:0/3037479627 shutdown_connections 2024-04-26T23:01:37.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.377+0000 7f2449400640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f242006c8f0 0x7f242006edb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.377+0000 7f2449400640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440d39f0 0x7f244415be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.377+0000 7f2449400640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24440d08d0 0x7f244415f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.377+0000 7f2449400640 1 --2- 172.21.15.40:0/3037479627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24440b6220 0x7f24440a2230 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.377+0000 7f2449400640 1 -- 172.21.15.40:0/3037479627 >> 172.21.15.40:0/3037479627 conn(0x7f24440d4d90 msgr2=0x7f24441623b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.377+0000 7f2449400640 1 -- 172.21.15.40:0/3037479627 shutdown_connections 2024-04-26T23:01:37.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.377+0000 7f2449400640 1 -- 172.21.15.40:0/3037479627 wait complete. 2024-04-26T23:01:37.395 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T23:01:37.396 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-26T23:01:37.396 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-26T23:01:37.397 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-26T23:01:37.397 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-26T23:01:37.397 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-26T23:01:37.398 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-26T23:01:37.398 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-26T23:01:37.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.686+0000 7efef5400640 1 Processor -- start 2024-04-26T23:01:37.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.688+0000 7ff9d9200640 1 Processor -- start 2024-04-26T23:01:37.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.688+0000 7ff9d9200640 1 -- start start 2024-04-26T23:01:37.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.687+0000 7efef5400640 1 -- start start 2024-04-26T23:01:37.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.689+0000 7efef5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef00b5f20 0x7efef00d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.689+0000 7efef5400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef00b7e50 con 0x7efef00d3bc0 2024-04-26T23:01:37.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.689+0000 7efef5400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef00d1800 con 0x7efef0162880 2024-04-26T23:01:37.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.689+0000 7efef5400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efef00d0580 con 0x7efef00b5f20 2024-04-26T23:01:37.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.690+0000 7ff9d9200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff9d415f3a0 0x7ff9d40d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.690+0000 7ff9d9200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff9d40d1770 0x7ff9d40d1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.690+0000 7ff9d9200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d40d2160 0x7ff9d40d2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.690+0000 7efeede00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efef00d3bc0 0x7efef00b8300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41468/0 (socket says 172.21.15.40:41468) 2024-04-26T23:01:37.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.690+0000 7efeede00640 1 -- 172.21.15.40:0/3634734946 learned_addr learned my addr 172.21.15.40:0/3634734946 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:37.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.691+0000 7efeef200640 1 --2- 172.21.15.40:0/3634734946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef00b5f20 0x7efef00d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.691+0000 7ff9d9200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff9d40d25d0 con 0x7ff9d415f3a0 2024-04-26T23:01:37.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.691+0000 7ff9d9200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff9d40d0e20 con 0x7ff9d40d1770 2024-04-26T23:01:37.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.691+0000 7ff9d9200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d40b1d70 con 0x7ff9d40d2160 2024-04-26T23:01:37.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.691+0000 7ff9d1400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff9d40d1770 0x7ff9d40d1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.691+0000 7ff9d1400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff9d40d1770 0x7ff9d40d1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49666/0 (socket says 172.21.15.40:49666) 2024-04-26T23:01:37.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.691+0000 7ff9d1400640 1 -- 172.21.15.40:0/2514895938 learned_addr learned my addr 172.21.15.40:0/2514895938 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:37.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.691+0000 7ff9d1e00640 1 --2- 172.21.15.40:0/2514895938 >> v2:172.21.15.40:3300/0 conn(0x7ff9d415f3a0 0x7ff9d40d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.691+0000 7ff9d2800640 1 --2- 172.21.15.40:0/2514895938 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d40d2160 0x7ff9d40d2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.692+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3566300793 0 0) 0x7efedc001110 con 0x7efef0162880 2024-04-26T23:01:37.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.692+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efed4003520 con 0x7efef0162880 2024-04-26T23:01:37.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.692+0000 7ff9d2800640 1 -- 172.21.15.40:0/2514895938 >> v2:172.21.15.103:3300/0 conn(0x7ff9d40d1770 msgr2=0x7ff9d40d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.692+0000 7ff9d2800640 1 --2- 172.21.15.40:0/2514895938 >> v2:172.21.15.103:3300/0 conn(0x7ff9d40d1770 0x7ff9d40d1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.693+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3216865219 0 0) 0x7efef00b7e50 con 0x7efef00d3bc0 2024-04-26T23:01:37.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.693+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efed4003880 con 0x7efef00d3bc0 2024-04-26T23:01:37.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.692+0000 7ff9d2800640 1 -- 172.21.15.40:0/2514895938 >> v2:172.21.15.40:3300/0 conn(0x7ff9d415f3a0 msgr2=0x7ff9d40d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.693+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 207807805 0 0) 0x7efed4003520 con 0x7efef0162880 2024-04-26T23:01:37.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.693+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef00b5f20 msgr2=0x7efef00d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.693+0000 7efeed400640 1 --2- 172.21.15.40:0/3634734946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef00b5f20 0x7efef00d2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.693+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 >> v1:172.21.15.40:6789/0 conn(0x7efef00d3bc0 legacy=0x7efef00b8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.692+0000 7ff9d2800640 1 --2- 172.21.15.40:0/2514895938 >> v2:172.21.15.40:3300/0 conn(0x7ff9d415f3a0 0x7ff9d40d06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.692+0000 7ff9d2800640 1 -- 172.21.15.40:0/2514895938 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d40b2050 con 0x7ff9d40d2160 2024-04-26T23:01:37.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.693+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef00d07d0 con 0x7efef0162880 2024-04-26T23:01:37.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.694+0000 7ff9d2800640 1 --2- 172.21.15.40:0/2514895938 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d40d2160 0x7ff9d40d2e50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40b7b80 tx=0x7ff9c402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af5f643e01e17500 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.694+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7efedc001820 con 0x7efef0162880 2024-04-26T23:01:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.694+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 >> v1:172.21.15.103:6789/0 conn(0x7efef0162880 legacy=0x7efef00d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.694+0000 7efeed400640 1 --2- 172.21.15.40:0/3634734946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef00b5f20 0x7efed4003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.695+0000 7efeed400640 1 --2- 172.21.15.40:0/3634734946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed40042b0 0x7efed4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.695+0000 7efeed400640 1 --2- 172.21.15.40:0/3634734946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efed4004eb0 0x7efed400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.695+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efef00d0580 con 0x7efed40042b0 2024-04-26T23:01:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.695+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efed400bf00 con 0x7efed4004eb0 2024-04-26T23:01:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.695+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efed400c080 con 0x7efef00b5f20 2024-04-26T23:01:37.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.695+0000 7ff9d0a00640 1 -- 172.21.15.40:0/2514895938 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4037070 con 0x7ff9d40d2160 2024-04-26T23:01:37.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.695+0000 7efeee800640 1 --2- 172.21.15.40:0/3634734946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef00b5f20 0x7efed4003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.695+0000 7efeef200640 1 --2- 172.21.15.40:0/3634734946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efed4004eb0 0x7efed400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.696+0000 7f0c3d000640 1 Processor -- start 2024-04-26T23:01:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.695+0000 7ff9d0a00640 1 -- 172.21.15.40:0/2514895938 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c402ae50 con 0x7ff9d40d2160 2024-04-26T23:01:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.695+0000 7ff9d0a00640 1 -- 172.21.15.40:0/2514895938 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4032660 con 0x7ff9d40d2160 2024-04-26T23:01:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.696+0000 7efeede00640 1 --2- 172.21.15.40:0/3634734946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed40042b0 0x7efed4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.696+0000 7efeef200640 1 -- 172.21.15.40:0/3634734946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef00b5f20 msgr2=0x7efed4003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.696+0000 7efeef200640 1 --2- 172.21.15.40:0/3634734946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef00b5f20 0x7efed4003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.696+0000 7efeef200640 1 -- 172.21.15.40:0/3634734946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed40042b0 msgr2=0x7efed4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.696+0000 7ff9d9200640 1 -- 172.21.15.40:0/2514895938 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d40d2160 msgr2=0x7ff9d40d2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.696+0000 7ff9d9200640 1 --2- 172.21.15.40:0/2514895938 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d40d2160 0x7ff9d40d2e50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40b7b80 tx=0x7ff9c402a100 comp rx=0 tx=0).stop 2024-04-26T23:01:37.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.696+0000 7efeef200640 1 --2- 172.21.15.40:0/3634734946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed40042b0 0x7efed4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.696+0000 7efeef200640 1 -- 172.21.15.40:0/3634734946 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed4003520 con 0x7efed4004eb0 2024-04-26T23:01:37.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.697+0000 7efeede00640 1 --2- 172.21.15.40:0/3634734946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed40042b0 0x7efed4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:01:37.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.696+0000 7f0c3d000640 1 -- start start 2024-04-26T23:01:37.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.697+0000 7ff9d9200640 1 -- 172.21.15.40:0/2514895938 shutdown_connections 2024-04-26T23:01:37.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.697+0000 7ff9d9200640 1 --2- 172.21.15.40:0/2514895938 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d40d2160 0x7ff9d40d2e50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.697+0000 7ff9d9200640 1 --2- 172.21.15.40:0/2514895938 >> v2:172.21.15.103:3300/0 conn(0x7ff9d40d1770 0x7ff9d40d1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.697+0000 7ff9d9200640 1 --2- 172.21.15.40:0/2514895938 >> v2:172.21.15.40:3300/0 conn(0x7ff9d415f3a0 0x7ff9d40d06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.697+0000 7ff9d9200640 1 -- 172.21.15.40:0/2514895938 >> 172.21.15.40:0/2514895938 conn(0x7ff9d40d4f90 msgr2=0x7ff9d40d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.698+0000 7f0c3d000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0c3806f600 0x7f0c380722f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.698+0000 7f0c3d000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0c3806e500 0x7f0c3815ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.698+0000 7f0c3d000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c38073ce0 con 0x7f0c380aff60 2024-04-26T23:01:37.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.698+0000 7f0c3d000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f0c38071e60 con 0x7f0c3806e500 2024-04-26T23:01:37.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.698+0000 7f0c3d000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0c380d06a0 con 0x7f0c3806f600 2024-04-26T23:01:37.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.698+0000 7f0c2d400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0c380aff60 0x7f0c3806ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41476/0 (socket says 172.21.15.40:41476) 2024-04-26T23:01:37.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.698+0000 7f0c2d400640 1 -- 172.21.15.40:0/3961722674 learned_addr learned my addr 172.21.15.40:0/3961722674 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:37.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.698+0000 7f0c36800640 1 --2- 172.21.15.40:0/3961722674 >> v2:172.21.15.40:3300/0 conn(0x7f0c3806e500 0x7f0c3815ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.699+0000 7efeef200640 1 --2- 172.21.15.40:0/3634734946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efed4004eb0 0x7efed400b9c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efee8007360 tx=0x7efed4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5b4fcb343a40007 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.699+0000 7f0c36800640 1 -- 172.21.15.40:0/3961722674 >> v2:172.21.15.103:3300/0 conn(0x7f0c3806f600 msgr2=0x7f0c380722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.699+0000 7f0c36800640 1 --2- 172.21.15.40:0/3961722674 >> v2:172.21.15.103:3300/0 conn(0x7f0c3806f600 0x7f0c380722f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.699+0000 7f0c36800640 1 -- 172.21.15.40:0/3961722674 >> v1:172.21.15.40:6789/0 conn(0x7f0c380aff60 legacy=0x7f0c3806ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.699+0000 7f0c36800640 1 -- 172.21.15.40:0/3961722674 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c38073f80 con 0x7f0c3806e500 2024-04-26T23:01:37.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.699+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8033070 con 0x7efed4004eb0 2024-04-26T23:01:37.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.700+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee802f680 con 0x7efed4004eb0 2024-04-26T23:01:37.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.700+0000 7efeed400640 1 -- 172.21.15.40:0/3634734946 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8009be0 con 0x7efed4004eb0 2024-04-26T23:01:37.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.701+0000 7f0c36800640 1 --2- 172.21.15.40:0/3961722674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3806e500 0x7f0c3815ece0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0c3806c4e0 tx=0x7f0c3002b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0ea35a57d5ec8a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.698+0000 7ff9d9200640 1 -- 172.21.15.40:0/2514895938 shutdown_connections 2024-04-26T23:01:37.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.698+0000 7ff9d9200640 1 -- 172.21.15.40:0/2514895938 wait complete. 2024-04-26T23:01:37.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.701+0000 7ff9d9200640 1 Processor -- start 2024-04-26T23:01:37.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.701+0000 7efef5400640 1 -- 172.21.15.40:0/3634734946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efed4004eb0 msgr2=0x7efed400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.701+0000 7efef5400640 1 --2- 172.21.15.40:0/3634734946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efed4004eb0 0x7efed400b9c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efee8007360 tx=0x7efed4005830 comp rx=0 tx=0).stop 2024-04-26T23:01:37.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.701+0000 7efeef200640 1 -- 172.21.15.40:0/3634734946 reap_dead start 2024-04-26T23:01:37.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.701+0000 7efef5400640 1 -- 172.21.15.40:0/3634734946 shutdown_connections 2024-04-26T23:01:37.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.701+0000 7efef5400640 1 -- 172.21.15.40:0/3634734946 >> 172.21.15.40:0/3634734946 conn(0x7efef00d4f60 msgr2=0x7efef00d5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.702+0000 7f0c35400640 1 -- 172.21.15.40:0/3961722674 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c30007740 con 0x7f0c3806e500 2024-04-26T23:01:37.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.702+0000 7f0c35400640 1 -- 172.21.15.40:0/3961722674 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c300078e0 con 0x7f0c3806e500 2024-04-26T23:01:37.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.702+0000 7efef5400640 1 -- 172.21.15.40:0/3634734946 shutdown_connections 2024-04-26T23:01:37.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.702+0000 7f0c35400640 1 -- 172.21.15.40:0/3961722674 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c30007c90 con 0x7f0c3806e500 2024-04-26T23:01:37.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.702+0000 7efef5400640 1 -- 172.21.15.40:0/3634734946 wait complete. 2024-04-26T23:01:37.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.703+0000 7f0c3d000640 1 -- 172.21.15.40:0/3961722674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3806e500 msgr2=0x7f0c3815ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.703+0000 7f0c3d000640 1 --2- 172.21.15.40:0/3961722674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3806e500 0x7f0c3815ece0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0c3806c4e0 tx=0x7f0c3002b720 comp rx=0 tx=0).stop 2024-04-26T23:01:37.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.703+0000 7ff9d9200640 1 -- start start 2024-04-26T23:01:37.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.703+0000 7f0c3d000640 1 -- 172.21.15.40:0/3961722674 shutdown_connections 2024-04-26T23:01:37.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.703+0000 7f0c3d000640 1 --2- 172.21.15.40:0/3961722674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3806e500 0x7f0c3815ece0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.703+0000 7f0c3d000640 1 --2- 172.21.15.40:0/3961722674 >> v2:172.21.15.103:3300/0 conn(0x7f0c3806f600 0x7f0c380722f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.703+0000 7f0c3d000640 1 -- 172.21.15.40:0/3961722674 >> 172.21.15.40:0/3961722674 conn(0x7f0c380d0be0 msgr2=0x7f0c380d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.704+0000 7f0c3d000640 1 -- 172.21.15.40:0/3961722674 shutdown_connections 2024-04-26T23:01:37.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.705+0000 7f0c3d000640 1 -- 172.21.15.40:0/3961722674 wait complete. 2024-04-26T23:01:37.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.705+0000 7efef5400640 1 Processor -- start 2024-04-26T23:01:37.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.705+0000 7efef5400640 1 -- start start 2024-04-26T23:01:37.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.705+0000 7ff9d9200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d40d1770 0x7ff9d4204ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.705+0000 7ff9d9200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d40d2160 0x7ff9d40cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.706+0000 7ff9d9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9d415f3a0 0x7ff9d4205310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.706+0000 7ff9d9200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d415f900 con 0x7ff9d415f3a0 2024-04-26T23:01:37.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.706+0000 7ff9d9200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d415f780 con 0x7ff9d40d2160 2024-04-26T23:01:37.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.706+0000 7ff9d9200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d40d3390 con 0x7ff9d40d1770 2024-04-26T23:01:37.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.707+0000 7efef5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efed4004eb0 0x7efef01f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.707+0000 7f0c3d000640 1 Processor -- start 2024-04-26T23:01:37.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.707+0000 7efef5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef015c580 0x7efef00b5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.707+0000 7efef5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef015e290 0x7efef00cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.707+0000 7ff9d2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9d415f3a0 0x7ff9d4205310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.707+0000 7ff9d2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9d415f3a0 0x7ff9d4205310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38720/0 (socket says 172.21.15.40:38720) 2024-04-26T23:01:37.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.707+0000 7ff9d2800640 1 -- 172.21.15.40:0/1595731131 learned_addr learned my addr 172.21.15.40:0/1595731131 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:37.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.708+0000 7f0c3d000640 1 -- start start 2024-04-26T23:01:37.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.708+0000 7efef5400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efef0162c60 con 0x7efef015e290 2024-04-26T23:01:37.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.708+0000 7efef5400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efef00d0d90 con 0x7efed4004eb0 2024-04-26T23:01:37.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.708+0000 7efef5400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efef0162de0 con 0x7efef015c580 2024-04-26T23:01:37.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.708+0000 7ff9d1e00640 1 --2- 172.21.15.40:0/1595731131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d40d1770 0x7ff9d4204ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.708+0000 7efeede00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef015c580 0x7efef00b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.708+0000 7efeee800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efed4004eb0 0x7efef01f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.708+0000 7efeede00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef015c580 0x7efef00b5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:46744/0 (socket says 172.21.15.40:46744) 2024-04-26T23:01:37.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.708+0000 7efeede00640 1 -- 172.21.15.40:0/819510412 learned_addr learned my addr 172.21.15.40:0/819510412 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:37.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.708+0000 7fbadde00640 1 Processor -- start 2024-04-26T23:01:37.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.708+0000 7fbadde00640 1 -- start start 2024-04-26T23:01:37.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.709+0000 7ff9d2800640 1 -- 172.21.15.40:0/1595731131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d40d1770 msgr2=0x7ff9d4204ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.709+0000 7ff9d2800640 1 --2- 172.21.15.40:0/1595731131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9d40d1770 0x7ff9d4204ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.709+0000 7ff9d2800640 1 -- 172.21.15.40:0/1595731131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d40d2160 msgr2=0x7ff9d40cc060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:01:37.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.709+0000 7ff9d2800640 1 --2- 172.21.15.40:0/1595731131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d40d2160 0x7ff9d40cc060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.709+0000 7ff9d2800640 1 -- 172.21.15.40:0/1595731131 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d40b81f0 con 0x7ff9d415f3a0 2024-04-26T23:01:37.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.708+0000 7efeef200640 1 --2- 172.21.15.40:0/819510412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef015e290 0x7efef00cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.709+0000 7f44fb200640 1 Processor -- start 2024-04-26T23:01:37.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.709+0000 7efeee800640 1 -- 172.21.15.40:0/819510412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef015c580 msgr2=0x7efef00b5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.709+0000 7efeee800640 1 --2- 172.21.15.40:0/819510412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef015c580 0x7efef00b5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.709+0000 7efeee800640 1 -- 172.21.15.40:0/819510412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef015e290 msgr2=0x7efef00cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.710+0000 7fbadde00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbad815f5a0 0x7fbad80d5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.710+0000 7fbadde00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbad80d2830 0x7fbad8162f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.710+0000 7fbadde00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbad80d1a70 con 0x7fbad80d3c20 2024-04-26T23:01:37.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.710+0000 7ff9d2800640 1 --2- 172.21.15.40:0/1595731131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9d415f3a0 0x7ff9d4205310 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4035040 tx=0x7ff9c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.710+0000 7fbadde00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fbad8163d30 con 0x7fbad80d2830 2024-04-26T23:01:37.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.710+0000 7fbadde00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbad80d1d50 con 0x7fbad815f5a0 2024-04-26T23:01:37.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.711+0000 7fbad6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbad80d2830 0x7fbad8162f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.711+0000 7fbad6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbad80d2830 0x7fbad8162f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38744/0 (socket says 172.21.15.40:38744) 2024-04-26T23:01:37.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.711+0000 7fbad6000640 1 -- 172.21.15.40:0/561184982 learned_addr learned my addr 172.21.15.40:0/561184982 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:37.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.709+0000 7efeee800640 1 --2- 172.21.15.40:0/819510412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef015e290 0x7efef00cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.709+0000 7efeee800640 1 -- 172.21.15.40:0/819510412 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef00b2d40 con 0x7efed4004eb0 2024-04-26T23:01:37.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.709+0000 7f44fb200640 1 -- start start 2024-04-26T23:01:37.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.711+0000 7f44fb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44f415f420 0x7f44f40d0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.711+0000 7f44fb200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f44f40d3b60 0x7f44f40b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.711+0000 7f44fb200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f40d0d50 con 0x7f44f40b3d90 2024-04-26T23:01:37.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.711+0000 7efeee800640 1 --2- 172.21.15.40:0/819510412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efed4004eb0 0x7efef01f8ea0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efedc00f8e0 tx=0x7efedc00fda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.711+0000 7f44fb200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f44f40d0a70 con 0x7f44f40d3b60 2024-04-26T23:01:37.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.711+0000 7f44fb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f44f40b1bf0 con 0x7f44f415f420 2024-04-26T23:01:37.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.711+0000 7ff9ca000640 1 -- 172.21.15.40:0/1595731131 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4041070 con 0x7ff9d415f3a0 2024-04-26T23:01:37.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.711+0000 7ff9ca000640 1 -- 172.21.15.40:0/1595731131 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c403c460 con 0x7ff9d415f3a0 2024-04-26T23:01:37.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.711+0000 7ff9ca000640 1 -- 172.21.15.40:0/1595731131 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4037040 con 0x7ff9d415f3a0 2024-04-26T23:01:37.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7fbad6000640 1 -- 172.21.15.40:0/561184982 >> v2:172.21.15.103:3300/0 conn(0x7fbad815f5a0 msgr2=0x7fbad80d5cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T23:01:37.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7f0c3d000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c3806e500 0x7f0c381f9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7fbad6000640 1 --2- 172.21.15.40:0/561184982 >> v2:172.21.15.103:3300/0 conn(0x7fbad815f5a0 0x7fbad80d5cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7f0c3d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3806f600 0x7f0c3815c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7f0c3d000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c380aff60 0x7f0c3815cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7f0c3d000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0c380d1700 con 0x7f0c3806f600 2024-04-26T23:01:37.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7f0c3d000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3806f9e0 con 0x7f0c3806e500 2024-04-26T23:01:37.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7f0c3d000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0c380d1880 con 0x7f0c380aff60 2024-04-26T23:01:37.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7fbad6000640 1 -- 172.21.15.40:0/561184982 >> v1:172.21.15.103:6789/0 conn(0x7fbad80d3c20 legacy=0x7fbad815f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7efee6a00640 1 -- 172.21.15.40:0/819510412 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc0052e0 con 0x7efed4004eb0 2024-04-26T23:01:37.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7efee6a00640 1 -- 172.21.15.40:0/819510412 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efedc018070 con 0x7efed4004eb0 2024-04-26T23:01:37.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7efee6a00640 1 -- 172.21.15.40:0/819510412 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc00d950 con 0x7efed4004eb0 2024-04-26T23:01:37.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.713+0000 7efef5400640 1 -- 172.21.15.40:0/819510412 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef00b3020 con 0x7efed4004eb0 2024-04-26T23:01:37.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.713+0000 7f0c36800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c380aff60 0x7f0c3815cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.713+0000 7f0c2d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3806f600 0x7f0c3815c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7f44fa800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f44f40d3b60 0x7f44f40b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.713+0000 7f44fa800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f44f40d3b60 0x7f44f40b26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49704/0 (socket says 172.21.15.40:49704) 2024-04-26T23:01:37.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.713+0000 7f44fa800640 1 -- 172.21.15.40:0/212644834 learned_addr learned my addr 172.21.15.40:0/212644834 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:37.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7f44f9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44f415f420 0x7f44f40d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.713+0000 7ff9d9200640 1 -- 172.21.15.40:0/1595731131 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d40b84d0 con 0x7ff9d415f3a0 2024-04-26T23:01:37.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.713+0000 7ff9d9200640 1 -- 172.21.15.40:0/1595731131 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d415bf90 con 0x7ff9d415f3a0 2024-04-26T23:01:37.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.713+0000 7f0c36800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c380aff60 0x7f0c3815cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:46772/0 (socket says 172.21.15.40:46772) 2024-04-26T23:01:37.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.713+0000 7f0c36800640 1 -- 172.21.15.40:0/528069097 learned_addr learned my addr 172.21.15.40:0/528069097 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:37.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.713+0000 7f0c35e00640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c3806e500 0x7f0c381f9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7fbad5600640 1 -- 172.21.15.40:0/561184982 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 864895558 0 0) 0x7fbad80d1a70 con 0x7fbad80d3c20 2024-04-26T23:01:37.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.712+0000 7fbad6000640 1 -- 172.21.15.40:0/561184982 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad80d2030 con 0x7fbad80d2830 2024-04-26T23:01:37.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.714+0000 7efef5400640 1 -- 172.21.15.40:0/819510412 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef00b35e0 con 0x7efed4004eb0 2024-04-26T23:01:37.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.714+0000 7f0c2d400640 1 -- 172.21.15.40:0/528069097 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c380aff60 msgr2=0x7f0c3815cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.714+0000 7f0c2d400640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c380aff60 0x7f0c3815cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.714+0000 7ff9d9200640 1 -- 172.21.15.40:0/1595731131 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff9d42050e0 con 0x7ff9d415f3a0 2024-04-26T23:01:37.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.715+0000 7f0c2d400640 1 -- 172.21.15.40:0/528069097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c3806e500 msgr2=0x7f0c381f9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.714+0000 7f44f8a00640 1 -- 172.21.15.40:0/212644834 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3089912683 0 0) 0x7f44f40d0d50 con 0x7f44f40b3d90 2024-04-26T23:01:37.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.715+0000 7f0c2d400640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c3806e500 0x7f0c381f9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.715+0000 7f0c2d400640 1 -- 172.21.15.40:0/528069097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c3815d870 con 0x7f0c3806f600 2024-04-26T23:01:37.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.715+0000 7f0c36800640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c380aff60 0x7f0c3815cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:01:37.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.715+0000 7f0c35e00640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c3806e500 0x7f0c381f9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:01:37.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.715+0000 7fbad6000640 1 --2- 172.21.15.40:0/561184982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80d2830 0x7fbad8162f00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbad808d2f0 tx=0x7fbac002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbae62521cbbc31e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.715+0000 7f44f8a00640 1 -- 172.21.15.40:0/212644834 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44e0003520 con 0x7f44f40b3d90 2024-04-26T23:01:37.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.715+0000 7f44f9e00640 1 -- 172.21.15.40:0/212644834 >> v2:172.21.15.103:3300/0 conn(0x7f44f40d3b60 msgr2=0x7f44f40b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.715+0000 7f44f9e00640 1 --2- 172.21.15.40:0/212644834 >> v2:172.21.15.103:3300/0 conn(0x7f44f40d3b60 0x7f44f40b26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.715+0000 7f44f9e00640 1 -- 172.21.15.40:0/212644834 >> v1:172.21.15.103:6789/0 conn(0x7f44f40b3d90 legacy=0x7f44f40d3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.715+0000 7f44f9e00640 1 -- 172.21.15.40:0/212644834 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f40b1ed0 con 0x7f44f415f420 2024-04-26T23:01:37.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.716+0000 7fbad5600640 1 -- 172.21.15.40:0/561184982 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0037070 con 0x7fbad80d2830 2024-04-26T23:01:37.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.716+0000 7fbad5600640 1 -- 172.21.15.40:0/561184982 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac0032440 con 0x7fbad80d2830 2024-04-26T23:01:37.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.716+0000 7f0c2d400640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3806f600 0x7f0c3815c960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0c2800b8e0 tx=0x7f0c2800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.715+0000 7efef5400640 1 -- 172.21.15.40:0/819510412 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efef00b5ad0 con 0x7efed4004eb0 2024-04-26T23:01:37.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.717+0000 7f0c2ea00640 1 -- 172.21.15.40:0/528069097 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c28001660 con 0x7f0c3806f600 2024-04-26T23:01:37.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.717+0000 7f44f9e00640 1 --2- 172.21.15.40:0/212644834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44f415f420 0x7f44f40d0440 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009d90 tx=0x7f44ec032e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a441685e73533c7e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.717+0000 7f0c2ea00640 1 -- 172.21.15.40:0/528069097 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c28015020 con 0x7f0c3806f600 2024-04-26T23:01:37.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.717+0000 7fbadde00640 1 -- 172.21.15.40:0/561184982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80d2830 msgr2=0x7fbad8162f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.717+0000 7fbad5600640 1 -- 172.21.15.40:0/561184982 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0032810 con 0x7fbad80d2830 2024-04-26T23:01:37.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.717+0000 7fbadde00640 1 --2- 172.21.15.40:0/561184982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80d2830 0x7fbad8162f00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbad808d2f0 tx=0x7fbac002a210 comp rx=0 tx=0).stop 2024-04-26T23:01:37.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.717+0000 7f0c3d000640 1 -- 172.21.15.40:0/528069097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c3815db50 con 0x7f0c3806f600 2024-04-26T23:01:37.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.717+0000 7f44f8a00640 1 -- 172.21.15.40:0/212644834 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ec040070 con 0x7f44f415f420 2024-04-26T23:01:37.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.718+0000 7f44f8a00640 1 -- 172.21.15.40:0/212644834 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44ec033b70 con 0x7f44f415f420 2024-04-26T23:01:37.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.718+0000 7f44f8a00640 1 -- 172.21.15.40:0/212644834 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ec03b440 con 0x7f44f415f420 2024-04-26T23:01:37.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.717+0000 7fcaa8a00640 1 Processor -- start 2024-04-26T23:01:37.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.718+0000 7fbadde00640 1 -- 172.21.15.40:0/561184982 shutdown_connections 2024-04-26T23:01:37.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.718+0000 7fbadde00640 1 --2- 172.21.15.40:0/561184982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80d2830 0x7fbad8162f00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.718+0000 7fbadde00640 1 --2- 172.21.15.40:0/561184982 >> v2:172.21.15.103:3300/0 conn(0x7fbad815f5a0 0x7fbad80d5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.718+0000 7fbadde00640 1 -- 172.21.15.40:0/561184982 >> 172.21.15.40:0/561184982 conn(0x7fbad80d5190 msgr2=0x7fbad80d55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.718+0000 7fcaa8a00640 1 -- start start 2024-04-26T23:01:37.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.718+0000 7f0c2ea00640 1 -- 172.21.15.40:0/528069097 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c28001a00 con 0x7f0c3806f600 2024-04-26T23:01:37.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.719+0000 7f44fb200640 1 -- 172.21.15.40:0/212644834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44f415f420 msgr2=0x7f44f40d0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.719+0000 7f44fb200640 1 --2- 172.21.15.40:0/212644834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44f415f420 0x7f44f40d0440 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009d90 tx=0x7f44ec032e20 comp rx=0 tx=0).stop 2024-04-26T23:01:37.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.718+0000 7f0c3d000640 1 -- 172.21.15.40:0/528069097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c381f9540 con 0x7f0c3806f600 2024-04-26T23:01:37.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.719+0000 7f44fb200640 1 -- 172.21.15.40:0/212644834 shutdown_connections 2024-04-26T23:01:37.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.719+0000 7fbadde00640 1 -- 172.21.15.40:0/561184982 shutdown_connections 2024-04-26T23:01:37.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.719+0000 7fbadde00640 1 -- 172.21.15.40:0/561184982 wait complete. 2024-04-26T23:01:37.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.719+0000 7ff9ca000640 1 -- 172.21.15.40:0/1595731131 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff9c4044070 con 0x7ff9d415f3a0 2024-04-26T23:01:37.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.718+0000 7efee6a00640 1 -- 172.21.15.40:0/819510412 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efedc00daf0 con 0x7efed4004eb0 2024-04-26T23:01:37.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.719+0000 7efee6a00640 1 --2- 172.21.15.40:0/819510412 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efed006c9c0 0x7efed006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.719+0000 7f44fb200640 1 --2- 172.21.15.40:0/212644834 >> v2:172.21.15.103:3300/0 conn(0x7f44f40d3b60 0x7f44f40b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.719+0000 7f44fb200640 1 --2- 172.21.15.40:0/212644834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44f415f420 0x7f44f40d0440 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.719+0000 7f44fb200640 1 -- 172.21.15.40:0/212644834 >> 172.21.15.40:0/212644834 conn(0x7f44f409e1d0 msgr2=0x7f44f40d4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.720+0000 7f44fb200640 1 -- 172.21.15.40:0/212644834 shutdown_connections 2024-04-26T23:01:37.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.720+0000 7ff616000640 1 Processor -- start 2024-04-26T23:01:37.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.720+0000 7efee6a00640 1 -- 172.21.15.40:0/819510412 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7efedc08be70 con 0x7efed4004eb0 2024-04-26T23:01:37.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.720+0000 7efee6a00640 1 --2- 172.21.15.40:0/819510412 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x7efed00760c0 0x7efed0078500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.720+0000 7efee6a00640 1 -- 172.21.15.40:0/819510412 --> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efed0078bd0 con 0x7efed00760c0 2024-04-26T23:01:37.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.720+0000 7efeede00640 1 --2- 172.21.15.40:0/819510412 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efed006c9c0 0x7efed006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.720+0000 7ff616000640 1 -- start start 2024-04-26T23:01:37.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.720+0000 7efeef200640 1 --2- 172.21.15.40:0/819510412 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x7efed00760c0 0x7efed0078500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.720+0000 7f44fb200640 1 -- 172.21.15.40:0/212644834 wait complete. 2024-04-26T23:01:37.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.721+0000 7efee6a00640 1 -- 172.21.15.40:0/819510412 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efedc0559a0 con 0x7efed4004eb0 2024-04-26T23:01:37.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.721+0000 7f0c3d000640 1 -- 172.21.15.40:0/528069097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0c10001030 con 0x7f0c3806f600 2024-04-26T23:01:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.722+0000 7fbadde00640 1 Processor -- start 2024-04-26T23:01:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.722+0000 7ff616000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff61015f3a0 0x7ff6100d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.722+0000 7ff616000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff6100d2300 0x7ff6100b2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.722+0000 7ff616000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6100d0e20 con 0x7ff6100d1770 2024-04-26T23:01:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.722+0000 7ff616000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff6100b1d70 con 0x7ff6100d2300 2024-04-26T23:01:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.722+0000 7ff616000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff6100b2050 con 0x7ff61015f3a0 2024-04-26T23:01:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.722+0000 7f0c2ea00640 1 -- 172.21.15.40:0/528069097 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0c28001ba0 con 0x7f0c3806f600 2024-04-26T23:01:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7fbadde00640 1 -- start start 2024-04-26T23:01:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7ff60ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff6100d1770 0x7ff6100d1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41488/0 (socket says 172.21.15.40:41488) 2024-04-26T23:01:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7ff60ea00640 1 -- 172.21.15.40:0/914730248 learned_addr learned my addr 172.21.15.40:0/914730248 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7ff60fe00640 1 --2- 172.21.15.40:0/914730248 >> v2:172.21.15.103:3300/0 conn(0x7ff6100d2300 0x7ff6100b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7fcaa8a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcaa415e830 0x7fcaa406f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7f44fb200640 1 Processor -- start 2024-04-26T23:01:37.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7fcaa8a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcaa40cb940 0x7fcaa4071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7fcaa8a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaa40cfa50 con 0x7fcaa40ceaf0 2024-04-26T23:01:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7fcaa8a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcaa406e160 con 0x7fcaa40cb940 2024-04-26T23:01:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7fcaa8a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcaa406e440 con 0x7fcaa415e830 2024-04-26T23:01:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2904987434 0 0) 0x7ff6100d0e20 con 0x7ff6100d1770 2024-04-26T23:01:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5f4003520 con 0x7ff6100d1770 2024-04-26T23:01:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.721+0000 7ff9ca000640 1 --2- 172.21.15.40:0/1595731131 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff9b406c940 0x7ff9b406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.721+0000 7ff9ca000640 1 -- 172.21.15.40:0/1595731131 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff9c40aca50 con 0x7ff9d415f3a0 2024-04-26T23:01:37.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.722+0000 7ff9d1e00640 1 --2- 172.21.15.40:0/1595731131 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff9b406c940 0x7ff9b406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7ff9ca000640 1 --2- 172.21.15.40:0/1595731131 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x7ff9b40760c0 0x7ff9b4078500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7ff9ca000640 1 -- 172.21.15.40:0/1595731131 --> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff9b4078bd0 con 0x7ff9b40760c0 2024-04-26T23:01:37.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7ff9ca000640 1 -- 172.21.15.40:0/1595731131 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff9c40acf30 con 0x7ff9d415f3a0 2024-04-26T23:01:37.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.722+0000 7efeef200640 1 --2- 172.21.15.40:0/819510412 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x7efed00760c0 0x7efed0078500 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.723+0000 7efeede00640 1 --2- 172.21.15.40:0/819510412 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efed006c9c0 0x7efed006ee80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efef00cee00 tx=0x7efee0007f90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7f44fb200640 1 -- start start 2024-04-26T23:01:37.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7fcaa1400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcaa40ceaf0 0x7fcaa40cb230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41490/0 (socket says 172.21.15.40:41490) 2024-04-26T23:01:37.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 991403086 0 0) 0x7ff5f4003520 con 0x7ff6100d1770 2024-04-26T23:01:37.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff61015f3a0 msgr2=0x7ff6100d06f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T23:01:37.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7ff9d1e00640 1 --2- 172.21.15.40:0/1595731131 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff9b406c940 0x7ff9b406ee00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc011340 tx=0x7ff9cc00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7f0c2ea00640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0c1406c8f0 0x7f0c1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7f0c2ea00640 1 -- 172.21.15.40:0/528069097 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f0c2808c560 con 0x7f0c3806f600 2024-04-26T23:01:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7f0c2ea00640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] conn(0x7f0c14075f60 0x7f0c140783a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7f0c2ea00640 1 -- 172.21.15.40:0/528069097 --> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c2801fec0 con 0x7f0c14075f60 2024-04-26T23:01:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7f0c2ea00640 1 -- 172.21.15.40:0/528069097 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0c28093050 con 0x7f0c3806f600 2024-04-26T23:01:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7f0c36800640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] conn(0x7f0c14075f60 0x7f0c140783a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7ff60f400640 1 --2- 172.21.15.40:0/914730248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff61015f3a0 0x7ff6100d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7ff60e000640 1 --2- 172.21.15.40:0/914730248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff61015f3a0 0x7ff6100d06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 >> v2:172.21.15.103:3300/0 conn(0x7ff6100d2300 msgr2=0x7ff6100b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7ff60e000640 1 --2- 172.21.15.40:0/914730248 >> v2:172.21.15.103:3300/0 conn(0x7ff6100d2300 0x7ff6100b2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6100b2330 con 0x7ff6100d1770 2024-04-26T23:01:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7ff600001670 con 0x7ff6100d1770 2024-04-26T23:01:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 >> v1:172.21.15.40:6789/0 conn(0x7ff6100d1770 legacy=0x7ff6100d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7ff60e000640 1 --2- 172.21.15.40:0/914730248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5f40039b0 0x7ff5f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7ff60e000640 1 --2- 172.21.15.40:0/914730248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5f4004800 0x7ff5f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7ff60e000640 1 --2- 172.21.15.40:0/914730248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5f40051c0 0x7ff5f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff6100b1d70 con 0x7ff5f40039b0 2024-04-26T23:01:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff6100b2050 con 0x7ff5f4004800 2024-04-26T23:01:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff5f400c060 con 0x7ff5f40051c0 2024-04-26T23:01:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7fcaa1400640 1 -- 172.21.15.40:0/1660593657 learned_addr learned my addr 172.21.15.40:0/1660593657 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7fcaa2800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcaa40cb940 0x7fcaa4071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.724+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/1660593657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcaa415e830 0x7fcaa406f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.726+0000 7ff60f400640 1 --2- 172.21.15.40:0/914730248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5f40039b0 0x7ff5f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.726+0000 7f44fb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44f40b3d90 0x7f44f41472c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.726+0000 7ff60ea00640 1 --2- 172.21.15.40:0/914730248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5f4004800 0x7ff5f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.726+0000 7ff9d1400640 1 --2- 172.21.15.40:0/1595731131 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x7ff9b40760c0 0x7ff9b4078500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.726+0000 7f0c35e00640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0c1406c8f0 0x7f0c1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.726+0000 7f44fb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44f40d3b60 0x7f44f4147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.726+0000 7f44fb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f415f420 0x7f44f4158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.726+0000 7f0c36800640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] conn(0x7f0c14075f60 0x7f0c140783a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.726+0000 7efee6a00640 1 -- 172.21.15.40:0/819510412 <== osd.1 v2:172.21.15.40:6815/2323014490 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efed0078bd0 con 0x7efed00760c0 2024-04-26T23:01:37.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.726+0000 7f44fb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f44f40b2c10 con 0x7f44f415f420 2024-04-26T23:01:37.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7f44f9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44f40b3d90 0x7f44f41472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7f44fb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f44f40d2e50 con 0x7f44f40d3b60 2024-04-26T23:01:37.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.726+0000 7ff60fe00640 1 --2- 172.21.15.40:0/914730248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5f40051c0 0x7ff5f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7ff60ea00640 1 -- 172.21.15.40:0/914730248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5f40051c0 msgr2=0x7ff5f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7ff60ea00640 1 --2- 172.21.15.40:0/914730248 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5f40051c0 0x7ff5f400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7ff60ea00640 1 -- 172.21.15.40:0/914730248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5f40039b0 msgr2=0x7ff5f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7ff60ea00640 1 --2- 172.21.15.40:0/914730248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5f40039b0 0x7ff5f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7ff60ea00640 1 -- 172.21.15.40:0/914730248 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f4003520 con 0x7ff5f4004800 2024-04-26T23:01:37.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7f44fb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f44f40b2d90 con 0x7f44f40b3d90 2024-04-26T23:01:37.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7ff9d1400640 1 --2- 172.21.15.40:0/1595731131 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x7ff9b40760c0 0x7ff9b4078500 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7fbadde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80d2830 0x7fbad81fb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7fbadde00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad80d3c20 0x7fbad8160330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7f44f9400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44f40d3b60 0x7f44f4147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7f44fa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f415f420 0x7f44f4158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7fbadde00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad8201a90 0x7fbad8161120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7fcaa1e00640 1 -- 172.21.15.40:0/1660593657 >> v2:172.21.15.103:3300/0 conn(0x7fcaa40cb940 msgr2=0x7fcaa4071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/1660593657 >> v2:172.21.15.103:3300/0 conn(0x7fcaa40cb940 0x7fcaa4071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7fcaa1e00640 1 -- 172.21.15.40:0/1660593657 >> v1:172.21.15.40:6789/0 conn(0x7fcaa40ceaf0 legacy=0x7fcaa40cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.725+0000 7fcaa1e00640 1 -- 172.21.15.40:0/1660593657 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa406e720 con 0x7fcaa415e830 2024-04-26T23:01:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7fbadde00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbad8163440 con 0x7fbad80d2830 2024-04-26T23:01:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7fbadde00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbad80d4190 con 0x7fbad80d3c20 2024-04-26T23:01:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7fbadde00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbad81635c0 con 0x7fbad8201a90 2024-04-26T23:01:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7fbad6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80d2830 0x7fbad81fb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7f44fa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f415f420 0x7f44f4158d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38760/0 (socket says 172.21.15.40:38760) 2024-04-26T23:01:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.727+0000 7f44fa800640 1 -- 172.21.15.40:0/4281960216 learned_addr learned my addr 172.21.15.40:0/4281960216 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7fbad6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80d2830 0x7fbad81fb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38768/0 (socket says 172.21.15.40:38768) 2024-04-26T23:01:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7fbad6a00640 1 -- 172.21.15.40:0/1339843568 learned_addr learned my addr 172.21.15.40:0/1339843568 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7fbad6a00640 1 -- 172.21.15.40:0/1339843568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad8201a90 msgr2=0x7fbad8161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7fbad6a00640 1 --2- 172.21.15.40:0/1339843568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad8201a90 0x7fbad8161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7fbad6a00640 1 -- 172.21.15.40:0/1339843568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad80d3c20 msgr2=0x7fbad8160330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7fbad6a00640 1 --2- 172.21.15.40:0/1339843568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad80d3c20 0x7fbad8160330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7fbad6a00640 1 -- 172.21.15.40:0/1339843568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad81fbf00 con 0x7fbad80d2830 2024-04-26T23:01:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7ff9ca000640 1 -- 172.21.15.40:0/1595731131 <== osd.0 v2:172.21.15.40:6800/2897145808 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff9b4078bd0 con 0x7ff9b40760c0 2024-04-26T23:01:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7f44fa800640 1 -- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44f40b3d90 msgr2=0x7f44f41472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7f44fa800640 1 --2- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44f40b3d90 0x7f44f41472c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7f44fa800640 1 -- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44f40d3b60 msgr2=0x7f44f4147800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7f0c35e00640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0c1406c8f0 0x7f0c1406edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009890 tx=0x7f0c20008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7f44fa800640 1 --2- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44f40d3b60 0x7f44f4147800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.728+0000 7f44fa800640 1 -- 172.21.15.40:0/4281960216 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f41598f0 con 0x7f44f415f420 2024-04-26T23:01:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.729+0000 7f0c2ea00640 1 -- 172.21.15.40:0/528069097 <== osd.2 v2:172.21.15.40:6808/2108759315 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0c2801fec0 con 0x7f0c14075f60 2024-04-26T23:01:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.729+0000 7f44fa800640 1 --2- 172.21.15.40:0/4281960216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f415f420 0x7f44f4158d10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f44e400e870 tx=0x7f44e400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.729+0000 7fbad6a00640 1 --2- 172.21.15.40:0/1339843568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80d2830 0x7fbad81fb9c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbac800ba80 tx=0x7fbac800bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.729+0000 7fbacf400640 1 -- 172.21.15.40:0/1339843568 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac8019070 con 0x7fbad80d2830 2024-04-26T23:01:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.729+0000 7f44ea000640 1 -- 172.21.15.40:0/4281960216 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4017070 con 0x7f44f415f420 2024-04-26T23:01:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.730+0000 7f44ea000640 1 -- 172.21.15.40:0/4281960216 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e400c8e0 con 0x7f44f415f420 2024-04-26T23:01:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.730+0000 7f44ea000640 1 -- 172.21.15.40:0/4281960216 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e400ccd0 con 0x7f44f415f420 2024-04-26T23:01:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.729+0000 7ff60ea00640 1 --2- 172.21.15.40:0/914730248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5f4004800 0x7ff5f4004c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff60000c190 tx=0x7ff5f4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ac46e10411ba5f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.730+0000 7fbacf400640 1 -- 172.21.15.40:0/1339843568 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac800cc90 con 0x7fbad80d2830 2024-04-26T23:01:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.730+0000 7fbacf400640 1 -- 172.21.15.40:0/1339843568 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac8014680 con 0x7fbad80d2830 2024-04-26T23:01:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.730+0000 7fbadde00640 1 -- 172.21.15.40:0/1339843568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad82024a0 con 0x7fbad80d2830 2024-04-26T23:01:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.730+0000 7fbadde00640 1 -- 172.21.15.40:0/1339843568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad815fb30 con 0x7fbad80d2830 2024-04-26T23:01:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.730+0000 7f44fb200640 1 -- 172.21.15.40:0/4281960216 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f4159bd0 con 0x7f44f415f420 2024-04-26T23:01:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.731+0000 7f44fb200640 1 -- 172.21.15.40:0/4281960216 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f415a190 con 0x7f44f415f420 2024-04-26T23:01:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.731+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600007c80 con 0x7ff5f4004800 2024-04-26T23:01:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.731+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff600005ad0 con 0x7ff5f4004800 2024-04-26T23:01:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.731+0000 7ff60e000640 1 -- 172.21.15.40:0/914730248 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600007070 con 0x7ff5f4004800 2024-04-26T23:01:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.730+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/1660593657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcaa415e830 0x7fcaa406f200 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fca900097a0 tx=0x7fca9002a010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0ed5b8f9a79a8a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.731+0000 7fcaa0a00640 1 -- 172.21.15.40:0/1660593657 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca90037070 con 0x7fcaa415e830 2024-04-26T23:01:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.732+0000 7fbaccc00640 1 -- 172.21.15.40:0/1339843568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbaa8001030 con 0x7fbad80d2830 2024-04-26T23:01:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.732+0000 7fcaa0a00640 1 -- 172.21.15.40:0/1660593657 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca9002ad20 con 0x7fcaa415e830 2024-04-26T23:01:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.732+0000 7fcaa0a00640 1 -- 172.21.15.40:0/1660593657 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca90037070 con 0x7fcaa415e830 2024-04-26T23:01:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.732+0000 7ff616000640 1 -- 172.21.15.40:0/914730248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5f4004800 msgr2=0x7ff5f4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.732+0000 7ff616000640 1 --2- 172.21.15.40:0/914730248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5f4004800 0x7ff5f4004c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff60000c190 tx=0x7ff5f4005830 comp rx=0 tx=0).stop 2024-04-26T23:01:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.732+0000 7ff60fe00640 1 -- 172.21.15.40:0/914730248 reap_dead start 2024-04-26T23:01:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.733+0000 7ff616000640 1 -- 172.21.15.40:0/914730248 shutdown_connections 2024-04-26T23:01:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.733+0000 7ff616000640 1 -- 172.21.15.40:0/914730248 >> 172.21.15.40:0/914730248 conn(0x7ff6100d4f90 msgr2=0x7ff6100d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.733+0000 7fcaa8a00640 1 -- 172.21.15.40:0/1660593657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcaa415e830 msgr2=0x7fcaa406f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.733+0000 7fcaa8a00640 1 --2- 172.21.15.40:0/1660593657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcaa415e830 0x7fcaa406f200 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fca900097a0 tx=0x7fca9002a010 comp rx=0 tx=0).stop 2024-04-26T23:01:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.733+0000 7ff616000640 1 -- 172.21.15.40:0/914730248 shutdown_connections 2024-04-26T23:01:37.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.734+0000 7f44ea000640 1 -- 172.21.15.40:0/4281960216 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f44e401a070 con 0x7f44f415f420 2024-04-26T23:01:37.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.733+0000 7ff616000640 1 -- 172.21.15.40:0/914730248 wait complete. 2024-04-26T23:01:37.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.734+0000 7feaaf200640 1 Processor -- start 2024-04-26T23:01:37.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.734+0000 7feaaf200640 1 -- start start 2024-04-26T23:01:37.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.734+0000 7fcaa8a00640 1 -- 172.21.15.40:0/1660593657 shutdown_connections 2024-04-26T23:01:37.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.734+0000 7fcaa8a00640 1 --2- 172.21.15.40:0/1660593657 >> v2:172.21.15.103:3300/0 conn(0x7fcaa40cb940 0x7fcaa4071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.734+0000 7fcaa8a00640 1 --2- 172.21.15.40:0/1660593657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcaa415e830 0x7fcaa406f200 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.734+0000 7fcaa8a00640 1 -- 172.21.15.40:0/1660593657 >> 172.21.15.40:0/1660593657 conn(0x7fcaa40d0be0 msgr2=0x7fcaa40d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.734+0000 7f44fb200640 1 -- 172.21.15.40:0/4281960216 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f44f40b26d0 con 0x7f44f415f420 2024-04-26T23:01:37.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.736+0000 7f44ea000640 1 --2- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f44d806c940 0x7f44d806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.736+0000 7f44ea000640 1 -- 172.21.15.40:0/4281960216 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f44e408c070 con 0x7f44f415f420 2024-04-26T23:01:37.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.736+0000 7feaaf200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feaa815f3a0 0x7feaa80d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.736+0000 7feaaf200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feaa80d1770 0x7feaa80d1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.737+0000 7feaaf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feaa80d2160 0x7feaa80d2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.737+0000 7feaaf200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7feaa80d25d0 con 0x7feaa815f3a0 2024-04-26T23:01:37.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.737+0000 7feaaf200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7feaa80d0e20 con 0x7feaa80d1770 2024-04-26T23:01:37.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.737+0000 7feaaf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feaa80b1d70 con 0x7feaa80d2160 2024-04-26T23:01:37.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.736+0000 7f44ea000640 1 --2- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x7f44d80760c0 0x7f44d8078500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.736+0000 7f44ea000640 1 -- 172.21.15.40:0/4281960216 --> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44e4016ec0 con 0x7f44d80760c0 2024-04-26T23:01:37.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.736+0000 7f44ea000640 1 -- 172.21.15.40:0/4281960216 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f44e4093050 con 0x7f44f415f420 2024-04-26T23:01:37.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.734+0000 7fcaa8a00640 1 -- 172.21.15.40:0/1660593657 shutdown_connections 2024-04-26T23:01:37.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.734+0000 7fcaa8a00640 1 -- 172.21.15.40:0/1660593657 wait complete. 2024-04-26T23:01:37.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.737+0000 7feaa7400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feaa80d1770 0x7feaa80d1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.737+0000 7fcaa8a00640 1 Processor -- start 2024-04-26T23:01:37.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.737+0000 7fcaa8a00640 1 -- start start 2024-04-26T23:01:37.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.737+0000 7feaa7400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feaa80d1770 0x7feaa80d1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49776/0 (socket says 172.21.15.40:49776) 2024-04-26T23:01:37.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.737+0000 7feaa7400640 1 -- 172.21.15.40:0/796942942 learned_addr learned my addr 172.21.15.40:0/796942942 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:37.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7feaa7e00640 1 --2- 172.21.15.40:0/796942942 >> v2:172.21.15.40:3300/0 conn(0x7feaa815f3a0 0x7feaa80d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7fbacf400640 1 -- 172.21.15.40:0/1339843568 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbac801c070 con 0x7fbad80d2830 2024-04-26T23:01:37.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7f44f9e00640 1 --2- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f44d806c940 0x7f44d806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7f44f9400640 1 --2- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x7f44d80760c0 0x7f44d8078500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7feaad200640 1 --2- 172.21.15.40:0/796942942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feaa80d2160 0x7feaa80d2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7feaa7400640 1 -- 172.21.15.40:0/796942942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feaa80d2160 msgr2=0x7feaa80d2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7feaa7400640 1 --2- 172.21.15.40:0/796942942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feaa80d2160 0x7feaa80d2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7feaa7400640 1 -- 172.21.15.40:0/796942942 >> v2:172.21.15.40:3300/0 conn(0x7feaa815f3a0 msgr2=0x7feaa80d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7feaa7400640 1 --2- 172.21.15.40:0/796942942 >> v2:172.21.15.40:3300/0 conn(0x7feaa815f3a0 0x7feaa80d06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7feaa7400640 1 -- 172.21.15.40:0/796942942 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa80b2050 con 0x7feaa80d1770 2024-04-26T23:01:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7fcaa8a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcaa40cb940 0x7fcaa415d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7fcaa8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa40ceaf0 0x7fcaa4155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7fcaa8a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcaa415e830 0x7fcaa408c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7fcaa8a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa406ddf0 con 0x7fcaa40ceaf0 2024-04-26T23:01:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7fcaa8a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa406ece0 con 0x7fcaa40cb940 2024-04-26T23:01:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.738+0000 7fcaa8a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcaa406f740 con 0x7fcaa415e830 2024-04-26T23:01:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.739+0000 7fcaa1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa40ceaf0 0x7fcaa4155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.739+0000 7fcaa1e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcaa40cb940 0x7fcaa415d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.739+0000 7fcaa1e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcaa40cb940 0x7fcaa415d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49792/0 (socket says 172.21.15.40:49792) 2024-04-26T23:01:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.739+0000 7fcaa1e00640 1 -- 172.21.15.40:0/4076855249 learned_addr learned my addr 172.21.15.40:0/4076855249 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.739+0000 7fcaa1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa40ceaf0 0x7fcaa4155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38780/0 (socket says 172.21.15.40:38780) 2024-04-26T23:01:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.739+0000 7fcaa2800640 1 --2- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcaa415e830 0x7fcaa408c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.740+0000 7fcaa1400640 1 -- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcaa415e830 msgr2=0x7fcaa408c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.740+0000 7fcaa1400640 1 --2- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcaa415e830 0x7fcaa408c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.740+0000 7feaa7e00640 1 --2- 172.21.15.40:0/796942942 >> v2:172.21.15.40:3300/0 conn(0x7feaa815f3a0 0x7feaa80d06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:01:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.740+0000 7fcaa1400640 1 -- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcaa40cb940 msgr2=0x7fcaa415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.740+0000 7fbacf400640 1 --2- 172.21.15.40:0/1339843568 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbab406c9c0 0x7fbab406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.740+0000 7fbacf400640 1 -- 172.21.15.40:0/1339843568 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbac808c370 con 0x7fbad80d2830 2024-04-26T23:01:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.740+0000 7fcaa1400640 1 --2- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcaa40cb940 0x7fcaa415d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.740+0000 7fcaa1400640 1 -- 172.21.15.40:0/4076855249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa40c6bc0 con 0x7fcaa40ceaf0 2024-04-26T23:01:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.741+0000 7fcaa1400640 1 --2- 172.21.15.40:0/4076855249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa40ceaf0 0x7fcaa4155570 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fca9400b410 tx=0x7fca9400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.741+0000 7f44f9e00640 1 --2- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f44d806c940 0x7f44d806ee00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f44ec03d000 tx=0x7f44ec037000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.741+0000 7fbacf400640 1 --2- 172.21.15.40:0/1339843568 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x7fbab4076140 0x7fbab4078580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.741+0000 7fbacf400640 1 -- 172.21.15.40:0/1339843568 --> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbac8018ec0 con 0x7fbab4076140 2024-04-26T23:01:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.741+0000 7fca9a000640 1 -- 172.21.15.40:0/4076855249 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94015020 con 0x7fcaa40ceaf0 2024-04-26T23:01:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.741+0000 7fbacf400640 1 -- 172.21.15.40:0/1339843568 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbac808c760 con 0x7fbad80d2830 2024-04-26T23:01:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.741+0000 7fcaa8a00640 1 -- 172.21.15.40:0/4076855249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaa40c6ea0 con 0x7fcaa40ceaf0 2024-04-26T23:01:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.741+0000 7fbad6000640 1 --2- 172.21.15.40:0/1339843568 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbab406c9c0 0x7fbab406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.742+0000 7f44f9400640 1 --2- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x7f44d80760c0 0x7f44d8078500 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.741+0000 7fca9a000640 1 -- 172.21.15.40:0/4076855249 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca94011850 con 0x7fcaa40ceaf0 2024-04-26T23:01:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.741+0000 7fbad7400640 1 --2- 172.21.15.40:0/1339843568 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x7fbab4076140 0x7fbab4078580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.742+0000 7ff616000640 1 Processor -- start 2024-04-26T23:01:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.743+0000 7fbad7400640 1 --2- 172.21.15.40:0/1339843568 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x7fbab4076140 0x7fbab4078580 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.742+0000 7fca9a000640 1 -- 172.21.15.40:0/4076855249 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94011c80 con 0x7fcaa40ceaf0 2024-04-26T23:01:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.742+0000 7fcaa8a00640 1 -- 172.21.15.40:0/4076855249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaa40c7460 con 0x7fcaa40ceaf0 2024-04-26T23:01:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.743+0000 7fcaa8a00640 1 -- 172.21.15.40:0/4076855249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fca7c001030 con 0x7fcaa40ceaf0 2024-04-26T23:01:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.741+0000 7feaa7400640 1 --2- 172.21.15.40:0/796942942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa80d1770 0x7feaa80d1c20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feaa80b7b80 tx=0x7fea9402a200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c152006c57eeba01 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.742+0000 7feaa6a00640 1 -- 172.21.15.40:0/796942942 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea94037070 con 0x7feaa80d1770 2024-04-26T23:01:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.742+0000 7feaa6a00640 1 -- 172.21.15.40:0/796942942 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea94032440 con 0x7feaa80d1770 2024-04-26T23:01:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.742+0000 7feaa6a00640 1 -- 172.21.15.40:0/796942942 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea94032850 con 0x7feaa80d1770 2024-04-26T23:01:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.743+0000 7feaaf200640 1 -- 172.21.15.40:0/796942942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa80d1770 msgr2=0x7feaa80d1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.743+0000 7feaaf200640 1 --2- 172.21.15.40:0/796942942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa80d1770 0x7feaa80d1c20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feaa80b7b80 tx=0x7fea9402a200 comp rx=0 tx=0).stop 2024-04-26T23:01:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.743+0000 7feaaf200640 1 -- 172.21.15.40:0/796942942 shutdown_connections 2024-04-26T23:01:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.743+0000 7feaaf200640 1 --2- 172.21.15.40:0/796942942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feaa80d2160 0x7feaa80d2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.743+0000 7feaaf200640 1 --2- 172.21.15.40:0/796942942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa80d1770 0x7feaa80d1c20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.743+0000 7feaaf200640 1 --2- 172.21.15.40:0/796942942 >> v2:172.21.15.40:3300/0 conn(0x7feaa815f3a0 0x7feaa80d06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.744+0000 7feaaf200640 1 -- 172.21.15.40:0/796942942 >> 172.21.15.40:0/796942942 conn(0x7feaa80d4f90 msgr2=0x7feaa80d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.742+0000 7ff616000640 1 -- start start 2024-04-26T23:01:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.743+0000 7ff616000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5f4004800 0x7ff6101603a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.743+0000 7ff616000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6101f4da0 0x7ff6100b9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.744+0000 7feaaf200640 1 -- 172.21.15.40:0/796942942 shutdown_connections 2024-04-26T23:01:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.744+0000 7ff616000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff610162a60 0x7ff6100ba290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.744+0000 7ff616000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff6100d0870 con 0x7ff5f4004800 2024-04-26T23:01:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.744+0000 7ff616000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff6100d06f0 con 0x7ff610162a60 2024-04-26T23:01:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.744+0000 7ff616000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff6100d09f0 con 0x7ff6101f4da0 2024-04-26T23:01:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.744+0000 7ff60ea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6101f4da0 0x7ff6100b9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.744+0000 7ff60ea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6101f4da0 0x7ff6100b9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:46868/0 (socket says 172.21.15.40:46868) 2024-04-26T23:01:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.744+0000 7ff60ea00640 1 -- 172.21.15.40:0/2160319872 learned_addr learned my addr 172.21.15.40:0/2160319872 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.745+0000 7fbacf400640 1 -- 172.21.15.40:0/1339843568 <== osd.3 v2:172.21.15.40:6823/378872239 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbac8018ec0 con 0x7fbab4076140 2024-04-26T23:01:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.745+0000 7ff60ea00640 1 -- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff610162a60 msgr2=0x7ff6100ba290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.745+0000 7ff60ea00640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff610162a60 0x7ff6100ba290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.745+0000 7ff60ea00640 1 -- 172.21.15.40:0/2160319872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5f4004800 msgr2=0x7ff6101603a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.745+0000 7fbad6000640 1 --2- 172.21.15.40:0/1339843568 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbab406c9c0 0x7fbab406ee80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbac0034000 tx=0x7fbac002c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.746+0000 7ff60f400640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5f4004800 0x7ff6101603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.746+0000 7ff60ea00640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5f4004800 0x7ff6101603a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.746+0000 7ff60ea00640 1 -- 172.21.15.40:0/2160319872 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6100cde00 con 0x7ff6101f4da0 2024-04-26T23:01:37.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.747+0000 7f44ea000640 1 -- 172.21.15.40:0/4281960216 <== osd.4 v2:172.21.15.103:6803/801343304 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44e4016ec0 con 0x7f44d80760c0 2024-04-26T23:01:37.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.744+0000 7feaaf200640 1 -- 172.21.15.40:0/796942942 wait complete. 2024-04-26T23:01:37.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.747+0000 7ff60f400640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5f4004800 0x7ff6101603a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:37.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.747+0000 7ff60ea00640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6101f4da0 0x7ff6100b9d50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff60000c2b0 tx=0x7ff600001970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.747+0000 7feaaf200640 1 Processor -- start 2024-04-26T23:01:37.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.748+0000 7feaaf200640 1 -- start start 2024-04-26T23:01:37.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.748+0000 7ff607400640 1 -- 172.21.15.40:0/2160319872 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600041020 con 0x7ff6101f4da0 2024-04-26T23:01:37.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.746+0000 7fca9a000640 1 -- 172.21.15.40:0/4076855249 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fca94018020 con 0x7fcaa40ceaf0 2024-04-26T23:01:37.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.747+0000 7fca9a000640 1 --2- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fca8406c920 0x7fca8406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.748+0000 7fca9a000640 1 -- 172.21.15.40:0/4076855249 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fca9408c370 con 0x7fcaa40ceaf0 2024-04-26T23:01:37.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.748+0000 7fca9a000640 1 --2- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x7fca840760a0 0x7fca840784e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.748+0000 7fca9a000640 1 -- 172.21.15.40:0/4076855249 --> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca9400de50 con 0x7fca840760a0 2024-04-26T23:01:37.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.748+0000 7fca9a000640 1 -- 172.21.15.40:0/4076855249 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fca9408c760 con 0x7fcaa40ceaf0 2024-04-26T23:01:37.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.748+0000 7ff607400640 1 -- 172.21.15.40:0/2160319872 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff600001430 con 0x7ff6101f4da0 2024-04-26T23:01:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.748+0000 7ff607400640 1 -- 172.21.15.40:0/2160319872 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600007070 con 0x7ff6101f4da0 2024-04-26T23:01:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.748+0000 7ff616000640 1 -- 172.21.15.40:0/2160319872 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6100cc100 con 0x7ff6101f4da0 2024-04-26T23:01:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.748+0000 7ff616000640 1 -- 172.21.15.40:0/2160319872 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6100ce0a0 con 0x7ff6101f4da0 2024-04-26T23:01:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.749+0000 7ff616000640 1 -- 172.21.15.40:0/2160319872 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff61015e8c0 con 0x7ff6101f4da0 2024-04-26T23:01:37.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.750+0000 7feaaf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feaa80d2160 0x7feaa81f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.750+0000 7feaaf200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa815f3a0 0x7feaa80cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.750+0000 7feaa7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa815f3a0 0x7feaa80cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.750+0000 7feaa7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feaa80d2160 0x7feaa81f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.751+0000 7feaa7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feaa80d2160 0x7feaa81f4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:46880/0 (socket says 172.21.15.40:46880) 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.751+0000 7feaa7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa815f3a0 0x7feaa80cad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38808/0 (socket says 172.21.15.40:38808) 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.751+0000 7feaaf200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa80cb270 0x7feaa80b8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.751+0000 7feaa7e00640 1 -- 172.21.15.40:0/416937634 learned_addr learned my addr 172.21.15.40:0/416937634 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.751+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feaa815f900 con 0x7feaa815f3a0 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.751+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feaa815f780 con 0x7feaa80cb270 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.751+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feaa80d3390 con 0x7feaa80d2160 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.751+0000 7feaad200640 1 --2- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa80cb270 0x7feaa80b8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.752+0000 7feaa7400640 1 -- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feaa80d2160 msgr2=0x7feaa81f4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.752+0000 7feaa7400640 1 --2- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feaa80d2160 0x7feaa81f4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.752+0000 7feaa7400640 1 -- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa80cb270 msgr2=0x7feaa80b8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.752+0000 7feaa7400640 1 --2- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa80cb270 0x7feaa80b8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.752+0000 7feaa7400640 1 -- 172.21.15.40:0/416937634 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa80b93a0 con 0x7feaa815f3a0 2024-04-26T23:01:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.752+0000 7feaa7400640 1 --2- 172.21.15.40:0/416937634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa815f3a0 0x7feaa80cad30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fea94031000 tx=0x7fea940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.753+0000 7fea9fe00640 1 -- 172.21.15.40:0/416937634 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea94041030 con 0x7feaa815f3a0 2024-04-26T23:01:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.753+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fca8406c920 0x7fca8406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.753+0000 7fea9fe00640 1 -- 172.21.15.40:0/416937634 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea94004020 con 0x7feaa815f3a0 2024-04-26T23:01:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.753+0000 7fcaa2800640 1 --2- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x7fca840760a0 0x7fca840784e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.753+0000 7fea9fe00640 1 -- 172.21.15.40:0/416937634 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea94037040 con 0x7feaa815f3a0 2024-04-26T23:01:37.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.753+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaa80b9680 con 0x7feaa815f3a0 2024-04-26T23:01:37.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.754+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaa80b9c40 con 0x7feaa815f3a0 2024-04-26T23:01:37.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.754+0000 7ff607400640 1 -- 172.21.15.40:0/2160319872 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff600044020 con 0x7ff6101f4da0 2024-04-26T23:01:37.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.755+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feaa80694f0 con 0x7feaa815f3a0 2024-04-26T23:01:37.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.756+0000 7ff607400640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5ec06c8f0 0x7ff5ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.756+0000 7fcaa2800640 1 --2- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x7fca840760a0 0x7fca840784e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.756+0000 7ff607400640 1 -- 172.21.15.40:0/2160319872 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff6000aca40 con 0x7ff6101f4da0 2024-04-26T23:01:37.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.756+0000 7ff607400640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x7ff5ec075ff0 0x7ff5ec078430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.756+0000 7ff607400640 1 -- 172.21.15.40:0/2160319872 --> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff600009eb0 con 0x7ff5ec075ff0 2024-04-26T23:01:37.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.756+0000 7ff607400640 1 -- 172.21.15.40:0/2160319872 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff6000acdf0 con 0x7ff6101f4da0 2024-04-26T23:01:37.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.756+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fca8406c920 0x7fca8406ede0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fca90009560 tx=0x7fca9002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.757+0000 7ff60f400640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5ec06c8f0 0x7ff5ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.757+0000 7ff60fe00640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x7ff5ec075ff0 0x7ff5ec078430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.758+0000 7ff60f400640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5ec06c8f0 0x7ff5ec06edb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff610073b50 tx=0x7ff6080075a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.758+0000 7ff60fe00640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x7ff5ec075ff0 0x7ff5ec078430 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.759+0000 7fca9a000640 1 -- 172.21.15.40:0/4076855249 <== osd.6 v2:172.21.15.103:6816/293480649 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fca9400de50 con 0x7fca840760a0 2024-04-26T23:01:37.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.757+0000 7fea9fe00640 1 -- 172.21.15.40:0/416937634 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fea94044070 con 0x7feaa815f3a0 2024-04-26T23:01:37.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.759+0000 7fea9fe00640 1 --2- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea8806c8f0 0x7fea8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.759+0000 7fea9fe00640 1 -- 172.21.15.40:0/416937634 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fea940ac390 con 0x7feaa815f3a0 2024-04-26T23:01:37.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.759+0000 7fea9fe00640 1 --2- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x7fea88076060 0x7fea880784a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:37.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.759+0000 7fea9fe00640 1 -- 172.21.15.40:0/416937634 --> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fea94032440 con 0x7fea88076060 2024-04-26T23:01:37.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.759+0000 7fea9fe00640 1 -- 172.21.15.40:0/416937634 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fea940b3050 con 0x7feaa815f3a0 2024-04-26T23:01:37.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.760+0000 7feaa7e00640 1 --2- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea8806c8f0 0x7fea8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.760+0000 7feaad200640 1 --2- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x7fea88076060 0x7fea880784a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:37.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.761+0000 7feaa7e00640 1 --2- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea8806c8f0 0x7fea8806edb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feaa80cf060 tx=0x7fea98009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.761+0000 7ff607400640 1 -- 172.21.15.40:0/2160319872 <== osd.5 v2:172.21.15.103:6800/157083760 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff600009eb0 con 0x7ff5ec075ff0 2024-04-26T23:01:37.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.762+0000 7feaad200640 1 --2- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x7fea88076060 0x7fea880784a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:37.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.764+0000 7fea9fe00640 1 -- 172.21.15.40:0/416937634 <== osd.7 v2:172.21.15.103:6819/3769540391 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fea94032440 con 0x7fea88076060 2024-04-26T23:01:37.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.769+0000 7ff9d9200640 1 -- 172.21.15.40:0/1595731131 --> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff9d40694f0 con 0x7ff9b40760c0 2024-04-26T23:01:37.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.769+0000 7ff9ca000640 1 -- 172.21.15.40:0/1595731131 <== osd.0 v2:172.21.15.40:6800/2897145808 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff9d40694f0 con 0x7ff9b40760c0 2024-04-26T23:01:37.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.770+0000 7ff9d9200640 1 -- 172.21.15.40:0/1595731131 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x7ff9b40760c0 msgr2=0x7ff9b4078500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.770+0000 7ff9d9200640 1 --2- 172.21.15.40:0/1595731131 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x7ff9b40760c0 0x7ff9b4078500 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.771+0000 7ff9d9200640 1 -- 172.21.15.40:0/1595731131 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff9b406c940 msgr2=0x7ff9b406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.771+0000 7efef5400640 1 -- 172.21.15.40:0/819510412 --> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efef00b5870 con 0x7efed00760c0 2024-04-26T23:01:37.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.771+0000 7ff9d9200640 1 --2- 172.21.15.40:0/1595731131 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff9b406c940 0x7ff9b406ee00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc011340 tx=0x7ff9cc00b040 comp rx=0 tx=0).stop 2024-04-26T23:01:37.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.771+0000 7ff9d9200640 1 -- 172.21.15.40:0/1595731131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9d415f3a0 msgr2=0x7ff9d4205310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.771+0000 7ff9d9200640 1 --2- 172.21.15.40:0/1595731131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9d415f3a0 0x7ff9d4205310 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4035040 tx=0x7ff9c40046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.771+0000 7ff9d2800640 1 -- 172.21.15.40:0/1595731131 reap_dead start 2024-04-26T23:01:37.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.771+0000 7ff9d9200640 1 -- 172.21.15.40:0/1595731131 shutdown_connections 2024-04-26T23:01:37.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.771+0000 7ff9d9200640 1 -- 172.21.15.40:0/1595731131 >> 172.21.15.40:0/1595731131 conn(0x7ff9d40d4f90 msgr2=0x7ff9d406e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.771+0000 7ff9d9200640 1 -- 172.21.15.40:0/1595731131 shutdown_connections 2024-04-26T23:01:37.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.771+0000 7ff9d9200640 1 -- 172.21.15.40:0/1595731131 wait complete. 2024-04-26T23:01:37.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.772+0000 7efee6a00640 1 -- 172.21.15.40:0/819510412 <== osd.1 v2:172.21.15.40:6815/2323014490 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7efee8003990 con 0x7efed00760c0 2024-04-26T23:01:37.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.772+0000 7efedbe00640 1 -- 172.21.15.40:0/819510412 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x7efed00760c0 msgr2=0x7efed0078500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.772+0000 7efedbe00640 1 --2- 172.21.15.40:0/819510412 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x7efed00760c0 0x7efed0078500 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.772+0000 7f0c3d000640 1 -- 172.21.15.40:0/528069097 --> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0c10002dd0 con 0x7f0c14075f60 2024-04-26T23:01:37.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.773+0000 7efedbe00640 1 -- 172.21.15.40:0/819510412 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efed006c9c0 msgr2=0x7efed006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.773+0000 7efedbe00640 1 --2- 172.21.15.40:0/819510412 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efed006c9c0 0x7efed006ee80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efef00cee00 tx=0x7efee0007f90 comp rx=0 tx=0).stop 2024-04-26T23:01:37.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.773+0000 7efedbe00640 1 -- 172.21.15.40:0/819510412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efed4004eb0 msgr2=0x7efef01f8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.773+0000 7efedbe00640 1 --2- 172.21.15.40:0/819510412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efed4004eb0 0x7efef01f8ea0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efedc00f8e0 tx=0x7efedc00fda0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.773+0000 7f0c2ea00640 1 -- 172.21.15.40:0/528069097 <== osd.2 v2:172.21.15.40:6808/2108759315 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0c10002dd0 con 0x7f0c14075f60 2024-04-26T23:01:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.773+0000 7efeef200640 1 -- 172.21.15.40:0/819510412 reap_dead start 2024-04-26T23:01:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.773+0000 7efedbe00640 1 -- 172.21.15.40:0/819510412 shutdown_connections 2024-04-26T23:01:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.773+0000 7efedbe00640 1 -- 172.21.15.40:0/819510412 >> 172.21.15.40:0/819510412 conn(0x7efef00d4f60 msgr2=0x7efef0160160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.773+0000 7efedbe00640 1 -- 172.21.15.40:0/819510412 shutdown_connections 2024-04-26T23:01:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.773+0000 7efedbe00640 1 -- 172.21.15.40:0/819510412 wait complete. 2024-04-26T23:01:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.775+0000 7f0c3d000640 1 -- 172.21.15.40:0/528069097 >> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] conn(0x7f0c14075f60 msgr2=0x7f0c140783a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.775+0000 7f0c3d000640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] conn(0x7f0c14075f60 0x7f0c140783a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.775+0000 7f0c3d000640 1 -- 172.21.15.40:0/528069097 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0c1406c8f0 msgr2=0x7f0c1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.775+0000 7f0c3d000640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0c1406c8f0 0x7f0c1406edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009890 tx=0x7f0c20008040 comp rx=0 tx=0).stop 2024-04-26T23:01:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.775+0000 7f0c3d000640 1 -- 172.21.15.40:0/528069097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3806f600 msgr2=0x7f0c3815c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.775+0000 7f0c3d000640 1 --2- 172.21.15.40:0/528069097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3806f600 0x7f0c3815c960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0c2800b8e0 tx=0x7f0c2800bda0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.775+0000 7f0c36800640 1 -- 172.21.15.40:0/528069097 reap_dead start 2024-04-26T23:01:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.775+0000 7f0c3d000640 1 -- 172.21.15.40:0/528069097 shutdown_connections 2024-04-26T23:01:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.775+0000 7f0c3d000640 1 -- 172.21.15.40:0/528069097 >> 172.21.15.40:0/528069097 conn(0x7f0c380d0be0 msgr2=0x7f0c380c4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.776+0000 7f0c3d000640 1 -- 172.21.15.40:0/528069097 shutdown_connections 2024-04-26T23:01:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.776+0000 7f0c3d000640 1 -- 172.21.15.40:0/528069097 wait complete. 2024-04-26T23:01:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.782+0000 7fbaccc00640 1 -- 172.21.15.40:0/1339843568 --> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbaa8002dd0 con 0x7fbab4076140 2024-04-26T23:01:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.783+0000 7fbacf400640 1 -- 172.21.15.40:0/1339843568 <== osd.3 v2:172.21.15.40:6823/378872239 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbaa8002dd0 con 0x7fbab4076140 2024-04-26T23:01:37.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.784+0000 7f44fb200640 1 -- 172.21.15.40:0/4281960216 --> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f44f415a5b0 con 0x7f44d80760c0 2024-04-26T23:01:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.784+0000 7fbaccc00640 1 -- 172.21.15.40:0/1339843568 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x7fbab4076140 msgr2=0x7fbab4078580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.784+0000 7fbaccc00640 1 --2- 172.21.15.40:0/1339843568 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x7fbab4076140 0x7fbab4078580 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.784+0000 7fbaccc00640 1 -- 172.21.15.40:0/1339843568 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbab406c9c0 msgr2=0x7fbab406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.784+0000 7fbaccc00640 1 --2- 172.21.15.40:0/1339843568 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbab406c9c0 0x7fbab406ee80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbac0034000 tx=0x7fbac002c000 comp rx=0 tx=0).stop 2024-04-26T23:01:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.784+0000 7fbaccc00640 1 -- 172.21.15.40:0/1339843568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80d2830 msgr2=0x7fbad81fb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.784+0000 7fbaccc00640 1 --2- 172.21.15.40:0/1339843568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80d2830 0x7fbad81fb9c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbac800ba80 tx=0x7fbac800bf40 comp rx=0 tx=0).stop 2024-04-26T23:01:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.784+0000 7fbad7400640 1 -- 172.21.15.40:0/1339843568 reap_dead start 2024-04-26T23:01:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.784+0000 7fbaccc00640 1 -- 172.21.15.40:0/1339843568 shutdown_connections 2024-04-26T23:01:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.784+0000 7fbaccc00640 1 -- 172.21.15.40:0/1339843568 >> 172.21.15.40:0/1339843568 conn(0x7fbad80d5190 msgr2=0x7fbad815d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.785+0000 7fbaccc00640 1 -- 172.21.15.40:0/1339843568 shutdown_connections 2024-04-26T23:01:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.785+0000 7fbaccc00640 1 -- 172.21.15.40:0/1339843568 wait complete. 2024-04-26T23:01:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.786+0000 7f44ea000640 1 -- 172.21.15.40:0/4281960216 <== osd.4 v2:172.21.15.103:6803/801343304 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f44f415a5b0 con 0x7f44d80760c0 2024-04-26T23:01:37.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.787+0000 7f44fb200640 1 -- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x7f44d80760c0 msgr2=0x7f44d8078500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.787+0000 7f44fb200640 1 --2- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x7f44d80760c0 0x7f44d8078500 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.787+0000 7f44fb200640 1 -- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f44d806c940 msgr2=0x7f44d806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.787+0000 7f44fb200640 1 --2- 172.21.15.40:0/4281960216 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f44d806c940 0x7f44d806ee00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f44ec03d000 tx=0x7f44ec037000 comp rx=0 tx=0).stop 2024-04-26T23:01:37.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.787+0000 7f44fb200640 1 -- 172.21.15.40:0/4281960216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f415f420 msgr2=0x7f44f4158d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.787+0000 7f44fb200640 1 --2- 172.21.15.40:0/4281960216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f415f420 0x7f44f4158d10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f44e400e870 tx=0x7f44e400ed30 comp rx=0 tx=0).stop 2024-04-26T23:01:37.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.788+0000 7f44fa800640 1 -- 172.21.15.40:0/4281960216 reap_dead start 2024-04-26T23:01:37.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.788+0000 7f44fb200640 1 -- 172.21.15.40:0/4281960216 shutdown_connections 2024-04-26T23:01:37.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.788+0000 7f44fb200640 1 -- 172.21.15.40:0/4281960216 >> 172.21.15.40:0/4281960216 conn(0x7f44f409e1d0 msgr2=0x7f44f406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.788+0000 7f44fb200640 1 -- 172.21.15.40:0/4281960216 shutdown_connections 2024-04-26T23:01:37.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.788+0000 7f44fb200640 1 -- 172.21.15.40:0/4281960216 wait complete. 2024-04-26T23:01:37.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.794+0000 7ff616000640 1 -- 172.21.15.40:0/2160319872 --> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff6100694f0 con 0x7ff5ec075ff0 2024-04-26T23:01:37.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.796+0000 7fcaa8a00640 1 -- 172.21.15.40:0/4076855249 --> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fca7c002dd0 con 0x7fca840760a0 2024-04-26T23:01:37.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.796+0000 7ff607400640 1 -- 172.21.15.40:0/2160319872 <== osd.5 v2:172.21.15.103:6800/157083760 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff6100694f0 con 0x7ff5ec075ff0 2024-04-26T23:01:37.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.796+0000 7fca9a000640 1 -- 172.21.15.40:0/4076855249 <== osd.6 v2:172.21.15.103:6816/293480649 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fca7c002dd0 con 0x7fca840760a0 2024-04-26T23:01:37.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.797+0000 7ff616000640 1 -- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x7ff5ec075ff0 msgr2=0x7ff5ec078430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.797+0000 7ff616000640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x7ff5ec075ff0 0x7ff5ec078430 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.797+0000 7ff616000640 1 -- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5ec06c8f0 msgr2=0x7ff5ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.797+0000 7fcaa8a00640 1 -- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x7fca840760a0 msgr2=0x7fca840784e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.797+0000 7fcaa8a00640 1 --2- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x7fca840760a0 0x7fca840784e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.797+0000 7ff616000640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5ec06c8f0 0x7ff5ec06edb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff610073b50 tx=0x7ff6080075a0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.797+0000 7ff616000640 1 -- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6101f4da0 msgr2=0x7ff6100b9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.797+0000 7ff616000640 1 --2- 172.21.15.40:0/2160319872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6101f4da0 0x7ff6100b9d50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff60000c2b0 tx=0x7ff600001970 comp rx=0 tx=0).stop 2024-04-26T23:01:37.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.797+0000 7ff60fe00640 1 -- 172.21.15.40:0/2160319872 reap_dead start 2024-04-26T23:01:37.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.798+0000 7fcaa8a00640 1 -- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fca8406c920 msgr2=0x7fca8406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.798+0000 7fcaa8a00640 1 --2- 172.21.15.40:0/4076855249 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fca8406c920 0x7fca8406ede0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fca90009560 tx=0x7fca9002e000 comp rx=0 tx=0).stop 2024-04-26T23:01:37.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.798+0000 7fcaa8a00640 1 -- 172.21.15.40:0/4076855249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa40ceaf0 msgr2=0x7fcaa4155570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.798+0000 7fcaa8a00640 1 --2- 172.21.15.40:0/4076855249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa40ceaf0 0x7fcaa4155570 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fca9400b410 tx=0x7fca9400b8d0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.798+0000 7fcaa2800640 1 -- 172.21.15.40:0/4076855249 reap_dead start 2024-04-26T23:01:37.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.798+0000 7ff616000640 1 -- 172.21.15.40:0/2160319872 shutdown_connections 2024-04-26T23:01:37.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.798+0000 7ff616000640 1 -- 172.21.15.40:0/2160319872 >> 172.21.15.40:0/2160319872 conn(0x7ff6100d4f90 msgr2=0x7ff61006d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.798+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 --> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feaa80b6860 con 0x7fea88076060 2024-04-26T23:01:37.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.798+0000 7ff616000640 1 -- 172.21.15.40:0/2160319872 shutdown_connections 2024-04-26T23:01:37.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.798+0000 7ff616000640 1 -- 172.21.15.40:0/2160319872 wait complete. 2024-04-26T23:01:37.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.799+0000 7fcaa8a00640 1 -- 172.21.15.40:0/4076855249 shutdown_connections 2024-04-26T23:01:37.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.799+0000 7fcaa8a00640 1 -- 172.21.15.40:0/4076855249 >> 172.21.15.40:0/4076855249 conn(0x7fcaa40d0be0 msgr2=0x7fcaa40b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.800+0000 7fea9fe00640 1 -- 172.21.15.40:0/416937634 <== osd.7 v2:172.21.15.103:6819/3769540391 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7feaa80b6860 con 0x7fea88076060 2024-04-26T23:01:37.811 INFO:teuthology.orchestra.run.smithi040.stdout:55834574859 2024-04-26T23:01:37.812 INFO:teuthology.orchestra.run.smithi040.stdout:55834574858 2024-04-26T23:01:37.812 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T23:01:37.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.800+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x7fea88076060 msgr2=0x7fea880784a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.800+0000 7feaaf200640 1 --2- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x7fea88076060 0x7fea880784a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.813 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T23:01:37.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.803+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea8806c8f0 msgr2=0x7fea8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.803+0000 7feaaf200640 1 --2- 172.21.15.40:0/416937634 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea8806c8f0 0x7fea8806edb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feaa80cf060 tx=0x7fea98009190 comp rx=0 tx=0).stop 2024-04-26T23:01:37.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.803+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa815f3a0 msgr2=0x7feaa80cad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:37.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.803+0000 7feaaf200640 1 --2- 172.21.15.40:0/416937634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa815f3a0 0x7feaa80cad30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fea94031000 tx=0x7fea940046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:37.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.803+0000 7feaad200640 1 -- 172.21.15.40:0/416937634 reap_dead start 2024-04-26T23:01:37.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.803+0000 7fcaa8a00640 1 -- 172.21.15.40:0/4076855249 shutdown_connections 2024-04-26T23:01:37.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.803+0000 7fcaa8a00640 1 -- 172.21.15.40:0/4076855249 wait complete. 2024-04-26T23:01:37.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.803+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 shutdown_connections 2024-04-26T23:01:37.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.803+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 >> 172.21.15.40:0/416937634 conn(0x7feaa80d4f90 msgr2=0x7feaa80d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:37.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.803+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 shutdown_connections 2024-04-26T23:01:37.816 INFO:teuthology.orchestra.run.smithi040.stdout:55834574860 2024-04-26T23:01:37.816 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T23:01:37.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:37.804+0000 7feaaf200640 1 -- 172.21.15.40:0/416937634 wait complete. 2024-04-26T23:01:37.817 INFO:teuthology.orchestra.run.smithi040.stdout:55834574858 2024-04-26T23:01:37.817 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T23:01:37.818 INFO:teuthology.orchestra.run.smithi040.stdout:55834574858 2024-04-26T23:01:37.819 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T23:01:37.820 INFO:teuthology.orchestra.run.smithi040.stdout:55834574858 2024-04-26T23:01:37.820 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T23:01:37.833 INFO:teuthology.orchestra.run.smithi040.stdout:55834574858 2024-04-26T23:01:37.833 INFO:teuthology.orchestra.run.smithi040.stdout:55834574858 2024-04-26T23:01:37.834 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-26T23:01:37.834 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-26T23:01:38.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.056+0000 7f9893200640 1 Processor -- start 2024-04-26T23:01:38.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.057+0000 7f9893200640 1 -- start start 2024-04-26T23:01:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.059+0000 7f9893200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f988c0e3160 0x7f988c0e7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.060+0000 7f9893200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f988c03c0a0 0x7f988c058050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.060+0000 7f9893200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c053cb0 0x7f988c054140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.060+0000 7f9893200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f988c058590 con 0x7f988c0e3160 2024-04-26T23:01:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.060+0000 7f9893200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f988c054680 con 0x7f988c03c0a0 2024-04-26T23:01:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.060+0000 7fd3a2200640 1 Processor -- start 2024-04-26T23:01:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.061+0000 7fd3a2200640 1 -- start start 2024-04-26T23:01:38.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.061+0000 7f9893200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f988c058810 con 0x7f988c053cb0 2024-04-26T23:01:38.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.061+0000 7f9891400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f988c03c0a0 0x7f988c058050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.061+0000 7f5e16c00640 1 Processor -- start 2024-04-26T23:01:38.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.061+0000 7f9892800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c053cb0 0x7f988c054140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.062+0000 7f9891400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f988c03c0a0 0x7f988c058050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49820/0 (socket says 172.21.15.40:49820) 2024-04-26T23:01:38.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.062+0000 7f9892800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c053cb0 0x7f988c054140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:46890/0 (socket says 172.21.15.40:46890) 2024-04-26T23:01:38.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.062+0000 7f9892800640 1 -- 172.21.15.40:0/2159016017 learned_addr learned my addr 172.21.15.40:0/2159016017 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:38.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.062+0000 7f9891e00640 1 --2- 172.21.15.40:0/2159016017 >> v2:172.21.15.40:3300/0 conn(0x7f988c0e3160 0x7f988c0e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.063+0000 7f9891e00640 1 -- 172.21.15.40:0/2159016017 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c053cb0 msgr2=0x7f988c054140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.063+0000 7f9891e00640 1 --2- 172.21.15.40:0/2159016017 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c053cb0 0x7f988c054140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.063+0000 7f9891e00640 1 -- 172.21.15.40:0/2159016017 >> v2:172.21.15.103:3300/0 conn(0x7f988c03c0a0 msgr2=0x7f988c058050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.063+0000 7f9891e00640 1 --2- 172.21.15.40:0/2159016017 >> v2:172.21.15.103:3300/0 conn(0x7f988c03c0a0 0x7f988c058050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.063+0000 7f9891e00640 1 -- 172.21.15.40:0/2159016017 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c058af0 con 0x7f988c0e3160 2024-04-26T23:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.062+0000 7f5e16c00640 1 -- start start 2024-04-26T23:01:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.063+0000 7fd3a2200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd39c0d0d10 0x7fd39c163580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.063+0000 7fd3a2200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd39c0d2b60 0x7fd39c0d3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.063+0000 7fd3a2200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c0cfcb0 con 0x7fd39c15f210 2024-04-26T23:01:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.063+0000 7fd3a2200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd39c0cff50 con 0x7fd39c0d0d10 2024-04-26T23:01:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.063+0000 7fd3a2200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd39c0d0230 con 0x7fd39c0d2b60 2024-04-26T23:01:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.064+0000 7f1469400640 1 Processor -- start 2024-04-26T23:01:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.064+0000 7f5e16c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e080548a0 0x7f5e08054ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.064+0000 7f5e16c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5e0803c360 0x7f5e0803c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.064+0000 7f5e16c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e080e7e30 con 0x7f5e080e7a50 2024-04-26T23:01:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.064+0000 7f5e16c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5e0803b5a0 con 0x7f5e0803c360 2024-04-26T23:01:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.064+0000 7f5e16c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5e0803b880 con 0x7f5e080548a0 2024-04-26T23:01:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.064+0000 7f9891e00640 1 --2- 172.21.15.40:0/2159016017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c0e3160 0x7f988c0e7780 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9894051fc0 tx=0x7f98940a6c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac215e90d50c46ab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.064+0000 7fd39be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd39c0d0d10 0x7fd39c163580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.064+0000 7fd3a0e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd39c15f210 0x7fd39c162e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:55412/0 (socket says 172.21.15.40:55412) 2024-04-26T23:01:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.064+0000 7fd3a0e00640 1 -- 172.21.15.40:0/854093153 learned_addr learned my addr 172.21.15.40:0/854093153 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.064+0000 7f1469400640 1 -- start start 2024-04-26T23:01:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.065+0000 7fd3a1800640 1 --2- 172.21.15.40:0/854093153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd39c0d2b60 0x7fd39c0d3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.065+0000 7f5e14e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e080548a0 0x7f5e08054ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.065+0000 7f5e14e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e080548a0 0x7f5e08054ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:46906/0 (socket says 172.21.15.40:46906) 2024-04-26T23:01:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.065+0000 7f5e14e00640 1 -- 172.21.15.40:0/2568752510 learned_addr learned my addr 172.21.15.40:0/2568752510 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.065+0000 7f5e16200640 1 --2- 172.21.15.40:0/2568752510 >> v2:172.21.15.103:3300/0 conn(0x7f5e0803c360 0x7f5e0803c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7fd39b400640 1 -- 172.21.15.40:0/854093153 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2066988286 0 0) 0x7fd39c0cfcb0 con 0x7fd39c15f210 2024-04-26T23:01:38.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7fd39b400640 1 -- 172.21.15.40:0/854093153 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd384003520 con 0x7fd39c15f210 2024-04-26T23:01:38.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7fd39be00640 1 -- 172.21.15.40:0/854093153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd39c0d2b60 msgr2=0x7fd39c0d3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:38.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7fd39be00640 1 --2- 172.21.15.40:0/854093153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd39c0d2b60 0x7fd39c0d3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7fd39be00640 1 -- 172.21.15.40:0/854093153 >> v1:172.21.15.103:6789/0 conn(0x7fd39c15f210 legacy=0x7fd39c162e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7fd39be00640 1 -- 172.21.15.40:0/854093153 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39c0d0510 con 0x7fd39c0d0d10 2024-04-26T23:01:38.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7f9890a00640 1 -- 172.21.15.40:0/2159016017 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98940b4070 con 0x7f988c0e3160 2024-04-26T23:01:38.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7f9890a00640 1 -- 172.21.15.40:0/2159016017 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98940a7a70 con 0x7f988c0e3160 2024-04-26T23:01:38.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7f9890a00640 1 -- 172.21.15.40:0/2159016017 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98940af440 con 0x7f988c0e3160 2024-04-26T23:01:38.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7f5e0fe00640 1 -- 172.21.15.40:0/2568752510 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3279512192 0 0) 0x7f5e080e7e30 con 0x7f5e080e7a50 2024-04-26T23:01:38.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7f5e0fe00640 1 -- 172.21.15.40:0/2568752510 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5df8003520 con 0x7f5e080e7a50 2024-04-26T23:01:38.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.067+0000 7f1469400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14640b6470 0x7f14640cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.067+0000 7f1469400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14640d5d30 con 0x7f14640d1770 2024-04-26T23:01:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.067+0000 7f1469400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14640d5a70 con 0x7f146415f680 2024-04-26T23:01:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.067+0000 7f1469400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f14640d3760 con 0x7f14640b6470 2024-04-26T23:01:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7f5e16200640 1 -- 172.21.15.40:0/2568752510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e080548a0 msgr2=0x7f5e08054ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7f5e16200640 1 --2- 172.21.15.40:0/2568752510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e080548a0 0x7f5e08054ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7f5e16200640 1 -- 172.21.15.40:0/2568752510 >> v1:172.21.15.40:6789/0 conn(0x7f5e080e7a50 legacy=0x7f5e08049b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.066+0000 7f5e16200640 1 -- 172.21.15.40:0/2568752510 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0803bb60 con 0x7f5e0803c360 2024-04-26T23:01:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.068+0000 7f9893200640 1 -- 172.21.15.40:0/2159016017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c0e3160 msgr2=0x7f988c0e7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.068+0000 7f9893200640 1 --2- 172.21.15.40:0/2159016017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c0e3160 0x7f988c0e7780 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9894051fc0 tx=0x7f98940a6c60 comp rx=0 tx=0).stop 2024-04-26T23:01:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.068+0000 7f5e16200640 1 --2- 172.21.15.40:0/2568752510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e0803c360 0x7f5e0803c7f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5e10051fc0 tx=0x7f5e100a6c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edfb08953ff88050 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.068+0000 7fd39be00640 1 --2- 172.21.15.40:0/854093153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd39c0d0d10 0x7fd39c163580 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd3880097a0 tx=0x7fd38802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3e13c6ba10a3575 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.068+0000 7f9893200640 1 -- 172.21.15.40:0/2159016017 shutdown_connections 2024-04-26T23:01:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.068+0000 7f1463e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f14640d1770 0x7f146415ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41506/0 (socket says 172.21.15.40:41506) 2024-04-26T23:01:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.068+0000 7f1463400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14640b6470 0x7f14640cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.068+0000 7f1463e00640 1 -- 172.21.15.40:0/3773469488 learned_addr learned my addr 172.21.15.40:0/3773469488 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.068+0000 7f1468a00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f146415f680 0x7f14640cff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:55416/0 (socket says 172.21.15.40:55416) 2024-04-26T23:01:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.068+0000 7f9893200640 1 --2- 172.21.15.40:0/2159016017 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c053cb0 0x7f988c054140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.068+0000 7f9893200640 1 --2- 172.21.15.40:0/2159016017 >> v2:172.21.15.103:3300/0 conn(0x7f988c03c0a0 0x7f988c058050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.068+0000 7f9893200640 1 --2- 172.21.15.40:0/2159016017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c0e3160 0x7f988c0e7780 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.068+0000 7f9893200640 1 -- 172.21.15.40:0/2159016017 >> 172.21.15.40:0/2159016017 conn(0x7f988c058ee0 msgr2=0x7f988c0592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.069+0000 7fd39b400640 1 -- 172.21.15.40:0/854093153 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388037070 con 0x7fd39c0d0d10 2024-04-26T23:01:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.069+0000 7fd39b400640 1 -- 172.21.15.40:0/854093153 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd388032440 con 0x7fd39c0d0d10 2024-04-26T23:01:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.069+0000 7fd39b400640 1 -- 172.21.15.40:0/854093153 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388032850 con 0x7fd39c0d0d10 2024-04-26T23:01:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.069+0000 7f9893200640 1 -- 172.21.15.40:0/2159016017 shutdown_connections 2024-04-26T23:01:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.069+0000 7f9893200640 1 -- 172.21.15.40:0/2159016017 wait complete. 2024-04-26T23:01:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.069+0000 7f1463400640 1 -- 172.21.15.40:0/3773469488 >> v1:172.21.15.103:6789/0 conn(0x7f146415f680 legacy=0x7f14640cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.069+0000 7f1463400640 1 -- 172.21.15.40:0/3773469488 >> v1:172.21.15.40:6789/0 conn(0x7f14640d1770 legacy=0x7f146415ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.070+0000 7f5e0fe00640 1 -- 172.21.15.40:0/2568752510 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e100b4070 con 0x7f5e0803c360 2024-04-26T23:01:38.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.070+0000 7fd3a2200640 1 -- 172.21.15.40:0/854093153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd39c0d0d10 msgr2=0x7fd39c163580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.070+0000 7f5e0fe00640 1 -- 172.21.15.40:0/2568752510 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e100a7a90 con 0x7f5e0803c360 2024-04-26T23:01:38.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.070+0000 7f5e0fe00640 1 -- 172.21.15.40:0/2568752510 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e100af440 con 0x7f5e0803c360 2024-04-26T23:01:38.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.070+0000 7fd3a2200640 1 --2- 172.21.15.40:0/854093153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd39c0d0d10 0x7fd39c163580 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd3880097a0 tx=0x7fd38802a1c0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.070+0000 7f1463400640 1 -- 172.21.15.40:0/3773469488 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14640b5b00 con 0x7f14640b6470 2024-04-26T23:01:38.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.071+0000 7fd3a2200640 1 -- 172.21.15.40:0/854093153 shutdown_connections 2024-04-26T23:01:38.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.071+0000 7fd3a2200640 1 --2- 172.21.15.40:0/854093153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd39c0d2b60 0x7fd39c0d3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.071+0000 7f1463400640 1 --2- 172.21.15.40:0/3773469488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14640b6470 0x7f14640cf9e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f14540097a0 tx=0x7f145402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ba142da820cf1f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.071+0000 7fd3a2200640 1 --2- 172.21.15.40:0/854093153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd39c0d0d10 0x7fd39c163580 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.072+0000 7f1462a00640 1 -- 172.21.15.40:0/3773469488 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454037070 con 0x7f14640b6470 2024-04-26T23:01:38.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.072+0000 7f1462a00640 1 -- 172.21.15.40:0/3773469488 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1454032440 con 0x7f14640b6470 2024-04-26T23:01:38.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.072+0000 7f1462a00640 1 -- 172.21.15.40:0/3773469488 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454032810 con 0x7f14640b6470 2024-04-26T23:01:38.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.071+0000 7fd3a2200640 1 -- 172.21.15.40:0/854093153 >> 172.21.15.40:0/854093153 conn(0x7fd39c0d4f90 msgr2=0x7fd39c0d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.072+0000 7f5e16c00640 1 -- 172.21.15.40:0/2568752510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e0803c360 msgr2=0x7f5e0803c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.072+0000 7f5e16c00640 1 --2- 172.21.15.40:0/2568752510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e0803c360 0x7f5e0803c7f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5e10051fc0 tx=0x7f5e100a6c80 comp rx=0 tx=0).stop 2024-04-26T23:01:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.072+0000 7fd3a2200640 1 -- 172.21.15.40:0/854093153 shutdown_connections 2024-04-26T23:01:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.073+0000 7f1469400640 1 -- 172.21.15.40:0/3773469488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14640b6470 msgr2=0x7f14640cf9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.073+0000 7f1469400640 1 --2- 172.21.15.40:0/3773469488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14640b6470 0x7f14640cf9e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f14540097a0 tx=0x7f145402a230 comp rx=0 tx=0).stop 2024-04-26T23:01:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.073+0000 7f1469400640 1 -- 172.21.15.40:0/3773469488 shutdown_connections 2024-04-26T23:01:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.073+0000 7f1469400640 1 --2- 172.21.15.40:0/3773469488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14640b6470 0x7f14640cf9e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.073+0000 7f1469400640 1 -- 172.21.15.40:0/3773469488 >> 172.21.15.40:0/3773469488 conn(0x7f14640d4dd0 msgr2=0x7f14640d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.073+0000 7f9893200640 1 Processor -- start 2024-04-26T23:01:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.073+0000 7f5e16c00640 1 -- 172.21.15.40:0/2568752510 shutdown_connections 2024-04-26T23:01:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.073+0000 7f5e16c00640 1 --2- 172.21.15.40:0/2568752510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e0803c360 0x7f5e0803c7f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.073+0000 7f5e16c00640 1 --2- 172.21.15.40:0/2568752510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e080548a0 0x7f5e08054ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.073+0000 7f5e16c00640 1 -- 172.21.15.40:0/2568752510 >> 172.21.15.40:0/2568752510 conn(0x7f5e08056cd0 msgr2=0x7f5e080570e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.073+0000 7fd3a2200640 1 -- 172.21.15.40:0/854093153 wait complete. 2024-04-26T23:01:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.074+0000 7f9893200640 1 -- start start 2024-04-26T23:01:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.074+0000 7f5e16c00640 1 -- 172.21.15.40:0/2568752510 shutdown_connections 2024-04-26T23:01:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.074+0000 7f5e16c00640 1 -- 172.21.15.40:0/2568752510 wait complete. 2024-04-26T23:01:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.075+0000 7f1469400640 1 -- 172.21.15.40:0/3773469488 shutdown_connections 2024-04-26T23:01:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.075+0000 7f1469400640 1 -- 172.21.15.40:0/3773469488 wait complete. 2024-04-26T23:01:38.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.075+0000 7fd3a2200640 1 Processor -- start 2024-04-26T23:01:38.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.075+0000 7f9893200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c03c0a0 0x7f988c186e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.075+0000 7f9893200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f988c053cb0 0x7f988c188950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.075+0000 7f9893200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c03ca40 0x7f988c03cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.076+0000 7f9892800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c03ca40 0x7f988c03cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.075+0000 7fd3a2200640 1 -- start start 2024-04-26T23:01:38.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.076+0000 7f9891400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f988c053cb0 0x7f988c188950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.076+0000 7f9892800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c03ca40 0x7f988c03cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38826/0 (socket says 172.21.15.40:38826) 2024-04-26T23:01:38.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.076+0000 7f9891400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f988c053cb0 0x7f988c188950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49832/0 (socket says 172.21.15.40:49832) 2024-04-26T23:01:38.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.077+0000 7f9892800640 1 -- 172.21.15.40:0/1751149442 learned_addr learned my addr 172.21.15.40:0/1751149442 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:38.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.077+0000 7f9891e00640 1 --2- 172.21.15.40:0/1751149442 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c03c0a0 0x7f988c186e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.077+0000 7fd3a2200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd39c0d0d10 0x7fd39c160e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.077+0000 7f9893200640 1 -- 172.21.15.40:0/1751149442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f988c0e3540 con 0x7f988c03ca40 2024-04-26T23:01:38.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.077+0000 7fd3a2200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd39c0d2b60 0x7fd39c0cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.078+0000 7fd3a2200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c15f210 0x7fd39c0b8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.078+0000 7f9893200640 1 -- 172.21.15.40:0/1751149442 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f988c006640 con 0x7f988c053cb0 2024-04-26T23:01:38.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.078+0000 7f5e16c00640 1 Processor -- start 2024-04-26T23:01:38.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.078+0000 7fd3a0e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd39c0d0d10 0x7fd39c160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.078+0000 7fd3a0e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd39c0d0d10 0x7fd39c160e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49834/0 (socket says 172.21.15.40:49834) 2024-04-26T23:01:38.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.078+0000 7fd3a0e00640 1 -- 172.21.15.40:0/4257870309 learned_addr learned my addr 172.21.15.40:0/4257870309 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:38.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.078+0000 7fd3a2200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd39c15f770 con 0x7fd39c15f210 2024-04-26T23:01:38.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.078+0000 7fd3a2200640 1 -- 172.21.15.40:0/4257870309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd39c15f5f0 con 0x7fd39c0d0d10 2024-04-26T23:01:38.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.078+0000 7fd3a2200640 1 -- 172.21.15.40:0/4257870309 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd39c0d27c0 con 0x7fd39c0d2b60 2024-04-26T23:01:38.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.078+0000 7fd39be00640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd39c0d2b60 0x7fd39c0cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.078+0000 7f9893200640 1 -- 172.21.15.40:0/1751149442 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f988c056e10 con 0x7f988c03c0a0 2024-04-26T23:01:38.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.078+0000 7f5e16c00640 1 -- start start 2024-04-26T23:01:38.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.079+0000 7fd3a1800640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c15f210 0x7fd39c0b8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.079+0000 7f1469400640 1 Processor -- start 2024-04-26T23:01:38.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.079+0000 7f1469400640 1 -- start start 2024-04-26T23:01:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.080+0000 7fd39be00640 1 -- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd39c0d0d10 msgr2=0x7fd39c160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.080+0000 7fd39be00640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd39c0d0d10 0x7fd39c160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.080+0000 7fd39be00640 1 -- 172.21.15.40:0/4257870309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c15f210 msgr2=0x7fd39c0b8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.080+0000 7fd39be00640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c15f210 0x7fd39c0b8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.080+0000 7fd39be00640 1 -- 172.21.15.40:0/4257870309 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39c161510 con 0x7fd39c0d2b60 2024-04-26T23:01:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.080+0000 7fd3a1800640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c15f210 0x7fd39c0b8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:01:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.079+0000 7f9892800640 1 -- 172.21.15.40:0/1751149442 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c03c0a0 msgr2=0x7f988c186e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.079+0000 7f9892800640 1 --2- 172.21.15.40:0/1751149442 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c03c0a0 0x7f988c186e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.079+0000 7f9892800640 1 -- 172.21.15.40:0/1751149442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f988c053cb0 msgr2=0x7f988c188950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.079+0000 7f9892800640 1 --2- 172.21.15.40:0/1751149442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f988c053cb0 0x7f988c188950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.079+0000 7f9892800640 1 -- 172.21.15.40:0/1751149442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c04d9e0 con 0x7f988c03ca40 2024-04-26T23:01:38.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.080+0000 7f9892800640 1 --2- 172.21.15.40:0/1751149442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c03ca40 0x7f988c03cef0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f988000e8a0 tx=0x7f988000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.081+0000 7f5e16c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e0803c360 0x7f5e080d8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.081+0000 7f5e16c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e080548a0 0x7f5e08055830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.081+0000 7fd39be00640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd39c0d2b60 0x7fd39c0cde70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd388031000 tx=0x7fd3880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.081+0000 7f1469400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14640d1770 0x7f14640cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.081+0000 7f988a000640 1 -- 172.21.15.40:0/1751149442 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9880017070 con 0x7f988c03ca40 2024-04-26T23:01:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.081+0000 7f5e15800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e0803c360 0x7f5e080d8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.082+0000 7f5e15800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e0803c360 0x7f5e080d8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38836/0 (socket says 172.21.15.40:38836) 2024-04-26T23:01:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.082+0000 7f5e15800640 1 -- 172.21.15.40:0/4011854130 learned_addr learned my addr 172.21.15.40:0/4011854130 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.082+0000 7f5e16c00640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e080e7a50 0x7f5e08055d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.082+0000 7f1469400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f146415f680 0x7f14640a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.081+0000 7f988a000640 1 -- 172.21.15.40:0/1751149442 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f988000c940 con 0x7f988c03ca40 2024-04-26T23:01:38.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.081+0000 7f988a000640 1 -- 172.21.15.40:0/1751149442 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f988000cd50 con 0x7f988c03ca40 2024-04-26T23:01:38.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.082+0000 7f9893200640 1 -- 172.21.15.40:0/1751149442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988c04df40 con 0x7f988c03ca40 2024-04-26T23:01:38.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.082+0000 7f9893200640 1 -- 172.21.15.40:0/1751149442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f988c04e500 con 0x7f988c03ca40 2024-04-26T23:01:38.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.082+0000 7f5e16c00640 1 -- 172.21.15.40:0/4011854130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0804a270 con 0x7f5e0803c360 2024-04-26T23:01:38.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.082+0000 7f5e16c00640 1 -- 172.21.15.40:0/4011854130 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0804a5b0 con 0x7f5e080e7a50 2024-04-26T23:01:38.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.082+0000 7f5e16c00640 1 -- 172.21.15.40:0/4011854130 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5e08039cb0 con 0x7f5e080548a0 2024-04-26T23:01:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.082+0000 7f1469400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14640cdc60 0x7f14640ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.082+0000 7f1469400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f14640d3d70 con 0x7f14640d1770 2024-04-26T23:01:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7f5e16200640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e080e7a50 0x7f5e08055d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.082+0000 7fd398c00640 1 -- 172.21.15.40:0/4257870309 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388042020 con 0x7fd39c0d2b60 2024-04-26T23:01:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.082+0000 7f1469400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f14640d3bf0 con 0x7f146415f680 2024-04-26T23:01:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7f1463400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f146415f680 0x7f14640a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7f1468a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14640cdc60 0x7f14640ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7f1468a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14640cdc60 0x7f14640ce0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:46934/0 (socket says 172.21.15.40:46934) 2024-04-26T23:01:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7f1468a00640 1 -- 172.21.15.40:0/2638236267 learned_addr learned my addr 172.21.15.40:0/2638236267 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7f1469400640 1 -- 172.21.15.40:0/2638236267 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f14640d1b50 con 0x7f14640cdc60 2024-04-26T23:01:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7f1463e00640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14640d1770 0x7f14640cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7fd3a2200640 1 -- 172.21.15.40:0/4257870309 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd39c1617f0 con 0x7fd39c0d2b60 2024-04-26T23:01:38.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7fd3a2200640 1 -- 172.21.15.40:0/4257870309 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd39c162090 con 0x7fd39c0d2b60 2024-04-26T23:01:38.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7f5e15800640 1 -- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e080548a0 msgr2=0x7f5e08055830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7f5e15800640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e080548a0 0x7f5e08055830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7f5e15800640 1 -- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e080e7a50 msgr2=0x7f5e08055d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7f5e15800640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e080e7a50 0x7f5e08055d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.083+0000 7f5e15800640 1 -- 172.21.15.40:0/4011854130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e080e8430 con 0x7f5e0803c360 2024-04-26T23:01:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.084+0000 7f5e16200640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e080e7a50 0x7f5e08055d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:01:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.084+0000 7f1463400640 1 -- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14640cdc60 msgr2=0x7f14640ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.084+0000 7f1463400640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14640cdc60 0x7f14640ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.084+0000 7f1463400640 1 -- 172.21.15.40:0/2638236267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14640d1770 msgr2=0x7f14640cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.084+0000 7f5e15800640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e0803c360 0x7f5e080d8280 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400c430 tx=0x7f5e0400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.084+0000 7fd398c00640 1 -- 172.21.15.40:0/4257870309 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd388038040 con 0x7fd39c0d2b60 2024-04-26T23:01:38.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.084+0000 7fd398c00640 1 -- 172.21.15.40:0/4257870309 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388042020 con 0x7fd39c0d2b60 2024-04-26T23:01:38.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.085+0000 7f1463400640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14640d1770 0x7f14640cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.085+0000 7f1463400640 1 -- 172.21.15.40:0/2638236267 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14640cd880 con 0x7f146415f680 2024-04-26T23:01:38.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.085+0000 7f1463e00640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14640d1770 0x7f14640cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:01:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.085+0000 7f5e0d600640 1 -- 172.21.15.40:0/4011854130 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04017070 con 0x7f5e0803c360 2024-04-26T23:01:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.085+0000 7f1468a00640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14640cdc60 0x7f14640ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:01:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.085+0000 7f5e0d600640 1 -- 172.21.15.40:0/4011854130 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e040052f0 con 0x7f5e0803c360 2024-04-26T23:01:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.085+0000 7f5e16c00640 1 -- 172.21.15.40:0/4011854130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e080e8710 con 0x7f5e0803c360 2024-04-26T23:01:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.086+0000 7f1463400640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f146415f680 0x7f14640a20f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1454031000 tx=0x7f14540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.086+0000 7f5e16c00640 1 -- 172.21.15.40:0/4011854130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e080e8cd0 con 0x7f5e0803c360 2024-04-26T23:01:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.086+0000 7f5e0d600640 1 -- 172.21.15.40:0/4011854130 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04011890 con 0x7f5e0803c360 2024-04-26T23:01:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.086+0000 7f145be00640 1 -- 172.21.15.40:0/2638236267 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454040030 con 0x7f146415f680 2024-04-26T23:01:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.087+0000 7f1469400640 1 -- 172.21.15.40:0/2638236267 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14640ccfe0 con 0x7f146415f680 2024-04-26T23:01:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.087+0000 7f1469400640 1 -- 172.21.15.40:0/2638236267 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14640cd5a0 con 0x7f146415f680 2024-04-26T23:01:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.087+0000 7f988a000640 1 -- 172.21.15.40:0/1751149442 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f988001a070 con 0x7f988c03ca40 2024-04-26T23:01:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.087+0000 7f145be00640 1 -- 172.21.15.40:0/2638236267 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1454038040 con 0x7f146415f680 2024-04-26T23:01:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.088+0000 7f145be00640 1 -- 172.21.15.40:0/2638236267 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454040030 con 0x7f146415f680 2024-04-26T23:01:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.089+0000 7f988a000640 1 --2- 172.21.15.40:0/1751149442 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f987406c9f0 0x7f987406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.090+0000 7f988a000640 1 -- 172.21.15.40:0/1751149442 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f988008bb60 con 0x7f988c03ca40 2024-04-26T23:01:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.089+0000 7fd398c00640 1 -- 172.21.15.40:0/4257870309 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd388045070 con 0x7fd39c0d2b60 2024-04-26T23:01:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.090+0000 7fd3a2200640 1 -- 172.21.15.40:0/4257870309 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd378005290 con 0x7fd39c0d2b60 2024-04-26T23:01:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.090+0000 7f1469400640 1 -- 172.21.15.40:0/2638236267 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f143c005290 con 0x7f146415f680 2024-04-26T23:01:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.090+0000 7f5e16c00640 1 -- 172.21.15.40:0/4011854130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e080057e0 con 0x7f5e0803c360 2024-04-26T23:01:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.091+0000 7f9893200640 1 -- 172.21.15.40:0/1751149442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f986c005290 con 0x7f988c03ca40 2024-04-26T23:01:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.091+0000 7f9891e00640 1 --2- 172.21.15.40:0/1751149442 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f987406c9f0 0x7f987406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.092+0000 7fd398c00640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd37c06caf0 0x7fd37c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.092+0000 7fd398c00640 1 -- 172.21.15.40:0/4257870309 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd3880ab890 con 0x7fd39c0d2b60 2024-04-26T23:01:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.093+0000 7f9891e00640 1 --2- 172.21.15.40:0/1751149442 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f987406c9f0 0x7f987406eeb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f988c0568f0 tx=0x7f98940ab000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.093+0000 7fd3a0e00640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd37c06caf0 0x7fd37c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.094+0000 7f5e0d600640 1 -- 172.21.15.40:0/4011854130 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5e0401a070 con 0x7f5e0803c360 2024-04-26T23:01:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.094+0000 7f145be00640 1 -- 172.21.15.40:0/2638236267 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1454043070 con 0x7f146415f680 2024-04-26T23:01:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.095+0000 7fd3a0e00640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd37c06caf0 0x7fd37c06efb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd3900011e0 tx=0x7fd390009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.096+0000 7f5e0d600640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5df006c8f0 0x7f5df006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.096+0000 7f5e0d600640 1 -- 172.21.15.40:0/4011854130 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5e0408b200 con 0x7f5e0803c360 2024-04-26T23:01:38.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.096+0000 7f145be00640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f144006cab0 0x7f144006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.096+0000 7f145be00640 1 -- 172.21.15.40:0/2638236267 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f14540ac5a0 con 0x7f146415f680 2024-04-26T23:01:38.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.096+0000 7f5e14e00640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5df006c8f0 0x7f5df006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.097+0000 7f1463e00640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f144006cab0 0x7f144006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.098+0000 7f5e14e00640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5df006c8f0 0x7f5df006edb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc006f40 tx=0x7f5dfc006e90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.098+0000 7f1463e00640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f144006cab0 0x7f144006ef70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f144c005880 tx=0x7f144c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.101+0000 7f988a000640 1 -- 172.21.15.40:0/1751149442 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9880059670 con 0x7f988c03ca40 2024-04-26T23:01:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.110+0000 7fd398c00640 1 -- 172.21.15.40:0/4257870309 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd388079210 con 0x7fd39c0d2b60 2024-04-26T23:01:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.110+0000 7f145be00640 1 -- 172.21.15.40:0/2638236267 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f145407a670 con 0x7f146415f680 2024-04-26T23:01:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.111+0000 7f5e0d600640 1 -- 172.21.15.40:0/4011854130 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5e04058c90 con 0x7f5e0803c360 2024-04-26T23:01:38.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.167+0000 7f2107c00640 1 Processor -- start 2024-04-26T23:01:38.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.167+0000 7f2107c00640 1 -- start start 2024-04-26T23:01:38.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.169+0000 7f2107c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f21000b3ff0 0x7f21000b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.169+0000 7f2107c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21000d2d70 con 0x7f21000d3b80 2024-04-26T23:01:38.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.169+0000 7f2107c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21000d3050 con 0x7f210015f540 2024-04-26T23:01:38.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.169+0000 7f2107c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f210015f920 con 0x7f21000b3ff0 2024-04-26T23:01:38.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.169+0000 7f6de9400640 1 Processor -- start 2024-04-26T23:01:38.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.169+0000 7f6de9400640 1 -- start start 2024-04-26T23:01:38.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.170+0000 7f2105200640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f210015f540 0x7f21000d3470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:55418/0 (socket says 172.21.15.40:55418) 2024-04-26T23:01:38.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.170+0000 7f2105200640 1 -- 172.21.15.40:0/1968926192 learned_addr learned my addr 172.21.15.40:0/1968926192 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:38.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.171+0000 7f6de9400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6de40d2090 0x7f6de40d0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.171+0000 7f6de9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6de40b65d0 0x7f6de40d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.171+0000 7f6de9400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de40b7f80 con 0x7f6de40b61f0 2024-04-26T23:01:38.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.171+0000 7f6de9400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6de40b88b0 con 0x7f6de40d2090 2024-04-26T23:01:38.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.171+0000 7f6de9400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6de40d2800 con 0x7f6de40b65d0 2024-04-26T23:01:38.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.171+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3162081452 0 0) 0x7f21000d2d70 con 0x7f21000d3b80 2024-04-26T23:01:38.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.172+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20e8003520 con 0x7f21000d3b80 2024-04-26T23:01:38.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.172+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4052327006 0 0) 0x7f21000d3050 con 0x7f210015f540 2024-04-26T23:01:38.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.172+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21000d2d70 con 0x7f210015f540 2024-04-26T23:01:38.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.172+0000 7f2105c00640 1 --2- 172.21.15.40:0/1968926192 >> v2:172.21.15.40:3300/0 conn(0x7f21000b3ff0 0x7f21000b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.172+0000 7f6de3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6de40b65d0 0x7f6de40d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.172+0000 7f6de3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6de40b65d0 0x7f6de40d0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:46936/0 (socket says 172.21.15.40:46936) 2024-04-26T23:01:38.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.172+0000 7f6de3200640 1 -- 172.21.15.40:0/848699080 learned_addr learned my addr 172.21.15.40:0/848699080 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:38.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.172+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1289386999 0 0) 0x7f20e8003520 con 0x7f21000d3b80 2024-04-26T23:01:38.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.172+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 >> v2:172.21.15.40:3300/0 conn(0x7f21000b3ff0 msgr2=0x7f21000b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:38.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f20ff400640 1 --2- 172.21.15.40:0/1968926192 >> v2:172.21.15.40:3300/0 conn(0x7f21000b3ff0 0x7f21000b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 >> v1:172.21.15.103:6789/0 conn(0x7f210015f540 legacy=0x7f21000d3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21000d08e0 con 0x7f21000d3b80 2024-04-26T23:01:38.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f20ec001670 con 0x7f21000d3b80 2024-04-26T23:01:38.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 >> v1:172.21.15.40:6789/0 conn(0x7f21000d3b80 legacy=0x7f21000b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f6de3200640 1 -- 172.21.15.40:0/848699080 >> v2:172.21.15.103:3300/0 conn(0x7f6de40d2090 msgr2=0x7f6de40d0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:38.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f20ff400640 1 --2- 172.21.15.40:0/1968926192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e8004bd0 0x7f20e8004fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f20ff400640 1 --2- 172.21.15.40:0/1968926192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20e80037a0 0x7f20e8003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f20ff400640 1 --2- 172.21.15.40:0/1968926192 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20e8004160 0x7f20e800ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f210015f920 con 0x7f20e8004bd0 2024-04-26T23:01:38.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f20e800c170 con 0x7f20e80037a0 2024-04-26T23:01:38.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f20e800c370 con 0x7f20e8004160 2024-04-26T23:01:38.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f6de3200640 1 --2- 172.21.15.40:0/848699080 >> v2:172.21.15.103:3300/0 conn(0x7f6de40d2090 0x7f6de40d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f6de3200640 1 -- 172.21.15.40:0/848699080 >> v1:172.21.15.103:6789/0 conn(0x7f6de40b61f0 legacy=0x7f6de40d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f6de3200640 1 -- 172.21.15.40:0/848699080 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de40b8260 con 0x7f6de40b65d0 2024-04-26T23:01:38.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.173+0000 7f6de1400640 1 -- 172.21.15.40:0/848699080 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2872178061 0 0) 0x7f6de40b7f80 con 0x7f6de40b61f0 2024-04-26T23:01:38.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.174+0000 7f2105c00640 1 --2- 172.21.15.40:0/1968926192 >> v2:172.21.15.40:3300/0 conn(0x7f21000b3ff0 0x7f21000b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:01:38.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.174+0000 7f2105200640 1 --2- 172.21.15.40:0/1968926192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e8004bd0 0x7f20e8004fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.174+0000 7f20ffe00640 1 --2- 172.21.15.40:0/1968926192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20e80037a0 0x7f20e8003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.175+0000 7f2105c00640 1 --2- 172.21.15.40:0/1968926192 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20e8004160 0x7f20e800ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.175+0000 7f2105200640 1 -- 172.21.15.40:0/1968926192 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20e8004160 msgr2=0x7f20e800ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:38.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.175+0000 7f2105200640 1 --2- 172.21.15.40:0/1968926192 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20e8004160 0x7f20e800ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.175+0000 7f6de3200640 1 --2- 172.21.15.40:0/848699080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6de40b65d0 0x7f6de40d0c70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009e50 tx=0x7f6ddc02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aba176233c22cc5b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.175+0000 7f2105200640 1 -- 172.21.15.40:0/1968926192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20e80037a0 msgr2=0x7f20e8003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:38.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.175+0000 7f2105200640 1 --2- 172.21.15.40:0/1968926192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20e80037a0 0x7f20e8003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.175+0000 7f2105200640 1 -- 172.21.15.40:0/1968926192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21000d08e0 con 0x7f20e8004bd0 2024-04-26T23:01:38.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.175+0000 7f20ffe00640 1 --2- 172.21.15.40:0/1968926192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20e80037a0 0x7f20e8003c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:01:38.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.177+0000 7f2105c00640 1 -- 172.21.15.40:0/1968926192 reap_dead start 2024-04-26T23:01:38.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.177+0000 7f2105200640 1 --2- 172.21.15.40:0/1968926192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e8004bd0 0x7f20e8004fb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f20f4007eb0 tx=0x7f20e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a00d558517fcb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.176+0000 7f6de1400640 1 -- 172.21.15.40:0/848699080 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc036070 con 0x7f6de40b65d0 2024-04-26T23:01:38.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.176+0000 7f6de1400640 1 -- 172.21.15.40:0/848699080 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ddc031600 con 0x7f6de40b65d0 2024-04-26T23:01:38.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.176+0000 7f6de1400640 1 -- 172.21.15.40:0/848699080 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc0319d0 con 0x7f6de40b65d0 2024-04-26T23:01:38.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.177+0000 7f6de9400640 1 -- 172.21.15.40:0/848699080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6de40b65d0 msgr2=0x7f6de40d0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.177+0000 7f6de9400640 1 --2- 172.21.15.40:0/848699080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6de40b65d0 0x7f6de40d0c70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009e50 tx=0x7f6ddc02a710 comp rx=0 tx=0).stop 2024-04-26T23:01:38.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.177+0000 7f6de9400640 1 -- 172.21.15.40:0/848699080 shutdown_connections 2024-04-26T23:01:38.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.177+0000 7f6de9400640 1 --2- 172.21.15.40:0/848699080 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6de40b65d0 0x7f6de40d0c70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.177+0000 7f6de9400640 1 --2- 172.21.15.40:0/848699080 >> v2:172.21.15.103:3300/0 conn(0x7f6de40d2090 0x7f6de40d0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.177+0000 7f6de9400640 1 -- 172.21.15.40:0/848699080 >> 172.21.15.40:0/848699080 conn(0x7f6de40d4f90 msgr2=0x7f6de40d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.178+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4007ab0 con 0x7f20e8004bd0 2024-04-26T23:01:38.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.178+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20f4004aa0 con 0x7f20e8004bd0 2024-04-26T23:01:38.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.178+0000 7f20ff400640 1 -- 172.21.15.40:0/1968926192 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f40070d0 con 0x7f20e8004bd0 2024-04-26T23:01:38.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.179+0000 7f2107c00640 1 -- 172.21.15.40:0/1968926192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e8004bd0 msgr2=0x7f20e8004fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.178+0000 7f6de9400640 1 -- 172.21.15.40:0/848699080 shutdown_connections 2024-04-26T23:01:38.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.179+0000 7f6de9400640 1 -- 172.21.15.40:0/848699080 wait complete. 2024-04-26T23:01:38.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.179+0000 7f2107c00640 1 --2- 172.21.15.40:0/1968926192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e8004bd0 0x7f20e8004fb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f20f4007eb0 tx=0x7f20e8005830 comp rx=0 tx=0).stop 2024-04-26T23:01:38.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.180+0000 7f2107c00640 1 -- 172.21.15.40:0/1968926192 shutdown_connections 2024-04-26T23:01:38.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.180+0000 7f2107c00640 1 --2- 172.21.15.40:0/1968926192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e8004bd0 0x7f20e8004fb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.180+0000 7f2107c00640 1 -- 172.21.15.40:0/1968926192 >> 172.21.15.40:0/1968926192 conn(0x7f21000d4f90 msgr2=0x7f21000d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.181+0000 7f2107c00640 1 -- 172.21.15.40:0/1968926192 shutdown_connections 2024-04-26T23:01:38.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.181+0000 7f6de9400640 1 Processor -- start 2024-04-26T23:01:38.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.182+0000 7f6de9400640 1 -- start start 2024-04-26T23:01:38.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.181+0000 7f2107c00640 1 -- 172.21.15.40:0/1968926192 wait complete. 2024-04-26T23:01:38.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.183+0000 7f2107c00640 1 Processor -- start 2024-04-26T23:01:38.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.183+0000 7f6de9400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6de40b61f0 0x7f6de40cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.183+0000 7f6de9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6de40b65d0 0x7f6de4164050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.183+0000 7f6de9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6de40d2090 0x7f6de4164590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.184+0000 7f6de9400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6de40c7d10 con 0x7f6de40b65d0 2024-04-26T23:01:38.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.184+0000 7f6de9400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6de40c7b90 con 0x7f6de40b61f0 2024-04-26T23:01:38.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.184+0000 7f6de9400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6de40c7e90 con 0x7f6de40d2090 2024-04-26T23:01:38.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.184+0000 7f2107c00640 1 -- start start 2024-04-26T23:01:38.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.184+0000 7f6de3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6de40d2090 0x7f6de4164590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.184+0000 7f6de3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6de40d2090 0x7f6de4164590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:46952/0 (socket says 172.21.15.40:46952) 2024-04-26T23:01:38.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.185+0000 7f6de1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6de40b65d0 0x7f6de4164050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.185+0000 7f6de1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6de40b65d0 0x7f6de4164050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38872/0 (socket says 172.21.15.40:38872) 2024-04-26T23:01:38.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.185+0000 7f6de1e00640 1 -- 172.21.15.40:0/2221406398 learned_addr learned my addr 172.21.15.40:0/2221406398 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:38.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.185+0000 7f6de2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6de40b61f0 0x7f6de40cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.186+0000 7f6de1e00640 1 -- 172.21.15.40:0/2221406398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6de40d2090 msgr2=0x7f6de4164590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.186+0000 7f6de1e00640 1 --2- 172.21.15.40:0/2221406398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6de40d2090 0x7f6de4164590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.186+0000 7f6de1e00640 1 -- 172.21.15.40:0/2221406398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6de40b61f0 msgr2=0x7f6de40cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.186+0000 7f6de1e00640 1 --2- 172.21.15.40:0/2221406398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6de40b61f0 0x7f6de40cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.186+0000 7f6de1e00640 1 -- 172.21.15.40:0/2221406398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de4164c30 con 0x7f6de40b65d0 2024-04-26T23:01:38.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.187+0000 7f6de1e00640 1 --2- 172.21.15.40:0/2221406398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6de40b65d0 0x7f6de4164050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400b440 tx=0x7f6dd400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.187+0000 7f6ddaa00640 1 -- 172.21.15.40:0/2221406398 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4015020 con 0x7f6de40b65d0 2024-04-26T23:01:38.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.188+0000 7f6ddaa00640 1 -- 172.21.15.40:0/2221406398 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd4011890 con 0x7f6de40b65d0 2024-04-26T23:01:38.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.188+0000 7f6ddaa00640 1 -- 172.21.15.40:0/2221406398 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4011c80 con 0x7f6de40b65d0 2024-04-26T23:01:38.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.188+0000 7f6de9400640 1 -- 172.21.15.40:0/2221406398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6de4164f10 con 0x7f6de40b65d0 2024-04-26T23:01:38.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.188+0000 7f6de9400640 1 -- 172.21.15.40:0/2221406398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6de41654d0 con 0x7f6de40b65d0 2024-04-26T23:01:38.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.189+0000 7f2107c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20e8004bd0 0x7f210015b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.189+0000 7f2107c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2100161520 0x7f2100161980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.189+0000 7f2107c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21000b2860 0x7f21000b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.189+0000 7f2107c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2100068e50 con 0x7f21000b2860 2024-04-26T23:01:38.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.189+0000 7f2107c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f21000d0f30 con 0x7f20e8004bd0 2024-04-26T23:01:38.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.189+0000 7f2107c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f21000d3470 con 0x7f2100161520 2024-04-26T23:01:38.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.190+0000 7f2105200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20e8004bd0 0x7f210015b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.190+0000 7f20ffe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2100161520 0x7f2100161980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.190+0000 7f2105200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20e8004bd0 0x7f210015b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49868/0 (socket says 172.21.15.40:49868) 2024-04-26T23:01:38.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.190+0000 7f2105200640 1 -- 172.21.15.40:0/4270901991 learned_addr learned my addr 172.21.15.40:0/4270901991 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:38.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.191+0000 7f2105c00640 1 --2- 172.21.15.40:0/4270901991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21000b2860 0x7f21000b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.191+0000 7f2105200640 1 -- 172.21.15.40:0/4270901991 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2100161520 msgr2=0x7f2100161980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.191+0000 7f2105200640 1 --2- 172.21.15.40:0/4270901991 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2100161520 0x7f2100161980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.191+0000 7f2105200640 1 -- 172.21.15.40:0/4270901991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21000b2860 msgr2=0x7f21000b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.191+0000 7f2105200640 1 --2- 172.21.15.40:0/4270901991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21000b2860 0x7f21000b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.191+0000 7f2105200640 1 -- 172.21.15.40:0/4270901991 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21000c8ac0 con 0x7f20e8004bd0 2024-04-26T23:01:38.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.191+0000 7f2105200640 1 --2- 172.21.15.40:0/4270901991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20e8004bd0 0x7f210015b590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20f4007fd0 tx=0x7f20f4001640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.193+0000 7f20fcc00640 1 -- 172.21.15.40:0/4270901991 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4044070 con 0x7f20e8004bd0 2024-04-26T23:01:38.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.193+0000 7f20fcc00640 1 -- 172.21.15.40:0/4270901991 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20f40070f0 con 0x7f20e8004bd0 2024-04-26T23:01:38.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.193+0000 7f20fcc00640 1 -- 172.21.15.40:0/4270901991 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f40074a0 con 0x7f20e8004bd0 2024-04-26T23:01:38.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.193+0000 7f6de9400640 1 -- 172.21.15.40:0/2221406398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6de40694f0 con 0x7f6de40b65d0 2024-04-26T23:01:38.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.193+0000 7f6ddaa00640 1 -- 172.21.15.40:0/2221406398 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6dd4018020 con 0x7f6de40b65d0 2024-04-26T23:01:38.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.194+0000 7f6ddaa00640 1 --2- 172.21.15.40:0/2221406398 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6dc006c8f0 0x7f6dc006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.195+0000 7f6ddaa00640 1 -- 172.21.15.40:0/2221406398 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6dd40579c0 con 0x7f6de40b65d0 2024-04-26T23:01:38.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.195+0000 7f2107c00640 1 -- 172.21.15.40:0/4270901991 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2100162070 con 0x7f20e8004bd0 2024-04-26T23:01:38.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.195+0000 7f2107c00640 1 -- 172.21.15.40:0/4270901991 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2100162630 con 0x7f20e8004bd0 2024-04-26T23:01:38.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.197+0000 7f2107c00640 1 -- 172.21.15.40:0/4270901991 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21000694f0 con 0x7f20e8004bd0 2024-04-26T23:01:38.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.200+0000 7f20fcc00640 1 -- 172.21.15.40:0/4270901991 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f20f4047070 con 0x7f20e8004bd0 2024-04-26T23:01:38.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.195+0000 7f6de2800640 1 --2- 172.21.15.40:0/2221406398 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6dc006c8f0 0x7f6dc006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.201+0000 7f20fcc00640 1 --2- 172.21.15.40:0/4270901991 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f20e006c940 0x7f20e006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.201+0000 7f20fcc00640 1 -- 172.21.15.40:0/4270901991 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f20f40ac090 con 0x7f20e8004bd0 2024-04-26T23:01:38.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.201+0000 7f6de2800640 1 --2- 172.21.15.40:0/2221406398 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6dc006c8f0 0x7f6dc006edb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6de41fc5a0 tx=0x7f6dcc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.202+0000 7f20ffe00640 1 --2- 172.21.15.40:0/4270901991 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f20e006c940 0x7f20e006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.204+0000 7f20ffe00640 1 --2- 172.21.15.40:0/4270901991 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f20e006c940 0x7f20e006ee00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f210015e810 tx=0x7f20ec009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.208+0000 7f6ddaa00640 1 -- 172.21.15.40:0/2221406398 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f6dd401f020 con 0x7f6de40b65d0 2024-04-26T23:01:38.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.212+0000 7fc32a800640 1 Processor -- start 2024-04-26T23:01:38.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.213+0000 7fc32a800640 1 -- start start 2024-04-26T23:01:38.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.213+0000 7f9cebe00640 1 Processor -- start 2024-04-26T23:01:38.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.213+0000 7f9cebe00640 1 -- start start 2024-04-26T23:01:38.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.214+0000 7f20fcc00640 1 -- 172.21.15.40:0/4270901991 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f20f4079cc0 con 0x7f20e8004bd0 2024-04-26T23:01:38.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.215+0000 7fc32a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc324161850 0x7fc3240d1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.215+0000 7fc32a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3240b4770 0x7fc3240d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.215+0000 7fc32a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc3240d1ec0 0x7fc3240d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.215+0000 7fc32a800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc3240b4110 con 0x7fc3240d1ec0 2024-04-26T23:01:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.215+0000 7fc32a800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc3240d3850 con 0x7fc324161850 2024-04-26T23:01:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.215+0000 7fc32a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc3240d3b30 con 0x7fc3240b4770 2024-04-26T23:01:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.215+0000 7f9cebe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cec161850 0x7f9cec0d1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.215+0000 7f9cebe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9cec0b63e0 0x7f9cec161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.215+0000 7f9cebe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cec0b4110 con 0x7f9cec0b4770 2024-04-26T23:01:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.215+0000 7f9cebe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9cec0d3850 con 0x7f9cec0b63e0 2024-04-26T23:01:38.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.215+0000 7f9cebe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9cec0d3b30 con 0x7f9cec161850 2024-04-26T23:01:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.217+0000 7fc328a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3240b4770 0x7fc3240d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.217+0000 7fc328a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3240b4770 0x7fc3240d1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:46970/0 (socket says 172.21.15.40:46970) 2024-04-26T23:01:38.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.217+0000 7fc328a00640 1 -- 172.21.15.40:0/825175492 learned_addr learned my addr 172.21.15.40:0/825175492 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:38.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.217+0000 7f9cea000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9cec0b4770 0x7f9cec0d1980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41532/0 (socket says 172.21.15.40:41532) 2024-04-26T23:01:38.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.217+0000 7f9cea000640 1 -- 172.21.15.40:0/1957089749 learned_addr learned my addr 172.21.15.40:0/1957089749 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:38.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.217+0000 7f9ceb400640 1 --2- 172.21.15.40:0/1957089749 >> v2:172.21.15.103:3300/0 conn(0x7f9cec0b63e0 0x7f9cec161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.218+0000 7fc329400640 1 --2- 172.21.15.40:0/825175492 >> v2:172.21.15.103:3300/0 conn(0x7fc324161850 0x7fc3240d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.218+0000 7fc328a00640 1 -- 172.21.15.40:0/825175492 >> v2:172.21.15.103:3300/0 conn(0x7fc324161850 msgr2=0x7fc3240d1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:38.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.218+0000 7fc328a00640 1 --2- 172.21.15.40:0/825175492 >> v2:172.21.15.103:3300/0 conn(0x7fc324161850 0x7fc3240d1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.218+0000 7fc328a00640 1 -- 172.21.15.40:0/825175492 >> v2:172.21.15.40:3300/0 conn(0x7fc3240d1ec0 msgr2=0x7fc3240d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:38.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.218+0000 7fc328a00640 1 --2- 172.21.15.40:0/825175492 >> v2:172.21.15.40:3300/0 conn(0x7fc3240d1ec0 0x7fc3240d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.218+0000 7fc328a00640 1 -- 172.21.15.40:0/825175492 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3240b3d70 con 0x7fc3240b4770 2024-04-26T23:01:38.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.218+0000 7f9ceb400640 1 -- 172.21.15.40:0/1957089749 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cec161850 msgr2=0x7f9cec0d1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:38.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.218+0000 7f9ceb400640 1 --2- 172.21.15.40:0/1957089749 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cec161850 0x7f9cec0d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.218+0000 7f9ceb400640 1 -- 172.21.15.40:0/1957089749 >> v1:172.21.15.40:6789/0 conn(0x7f9cec0b4770 legacy=0x7f9cec0d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.218+0000 7f9ceb400640 1 -- 172.21.15.40:0/1957089749 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec0b3d70 con 0x7f9cec0b63e0 2024-04-26T23:01:38.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.220+0000 7f9ceb400640 1 --2- 172.21.15.40:0/1957089749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cec0b63e0 0x7f9cec161c30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9cd80097a0 tx=0x7f9cd802a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4621aabc001adce1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.220+0000 7fc328a00640 1 --2- 172.21.15.40:0/825175492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3240b4770 0x7fc3240d1980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc3240d4d70 tx=0x7fc31002a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16c7fbaeedd03 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.221+0000 7f9ce9600640 1 -- 172.21.15.40:0/1957089749 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8037070 con 0x7f9cec0b63e0 2024-04-26T23:01:38.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.221+0000 7f9ce9600640 1 -- 172.21.15.40:0/1957089749 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cd8032440 con 0x7f9cec0b63e0 2024-04-26T23:01:38.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.221+0000 7f9ce9600640 1 -- 172.21.15.40:0/1957089749 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd802ae60 con 0x7f9cec0b63e0 2024-04-26T23:01:38.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.221+0000 7fc323e00640 1 -- 172.21.15.40:0/825175492 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc310037070 con 0x7fc3240b4770 2024-04-26T23:01:38.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.221+0000 7fc323e00640 1 -- 172.21.15.40:0/825175492 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc310032440 con 0x7fc3240b4770 2024-04-26T23:01:38.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.222+0000 7fc323e00640 1 -- 172.21.15.40:0/825175492 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc310032810 con 0x7fc3240b4770 2024-04-26T23:01:38.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.222+0000 7fc32a800640 1 -- 172.21.15.40:0/825175492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3240b4770 msgr2=0x7fc3240d1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.223+0000 7fc32a800640 1 --2- 172.21.15.40:0/825175492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3240b4770 0x7fc3240d1980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc3240d4d70 tx=0x7fc31002a1e0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.222+0000 7f9cebe00640 1 -- 172.21.15.40:0/1957089749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cec0b63e0 msgr2=0x7f9cec161c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.222+0000 7f9cebe00640 1 --2- 172.21.15.40:0/1957089749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cec0b63e0 0x7f9cec161c30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9cd80097a0 tx=0x7f9cd802a110 comp rx=0 tx=0).stop 2024-04-26T23:01:38.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.222+0000 7f9cebe00640 1 -- 172.21.15.40:0/1957089749 shutdown_connections 2024-04-26T23:01:38.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.222+0000 7f9cebe00640 1 --2- 172.21.15.40:0/1957089749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cec0b63e0 0x7f9cec161c30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.222+0000 7f9cebe00640 1 --2- 172.21.15.40:0/1957089749 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cec161850 0x7f9cec0d1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.222+0000 7f9cebe00640 1 -- 172.21.15.40:0/1957089749 >> 172.21.15.40:0/1957089749 conn(0x7f9cec0d4e10 msgr2=0x7f9cec0d5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.222+0000 7f9cebe00640 1 -- 172.21.15.40:0/1957089749 shutdown_connections 2024-04-26T23:01:38.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.222+0000 7f9cebe00640 1 -- 172.21.15.40:0/1957089749 wait complete. 2024-04-26T23:01:38.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.223+0000 7fc32a800640 1 -- 172.21.15.40:0/825175492 shutdown_connections 2024-04-26T23:01:38.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.223+0000 7fc32a800640 1 --2- 172.21.15.40:0/825175492 >> v2:172.21.15.40:3300/0 conn(0x7fc3240d1ec0 0x7fc3240d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.223+0000 7fc32a800640 1 --2- 172.21.15.40:0/825175492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3240b4770 0x7fc3240d1980 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.223+0000 7fc32a800640 1 --2- 172.21.15.40:0/825175492 >> v2:172.21.15.103:3300/0 conn(0x7fc324161850 0x7fc3240d1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.223+0000 7fc32a800640 1 -- 172.21.15.40:0/825175492 >> 172.21.15.40:0/825175492 conn(0x7fc3240d4e10 msgr2=0x7fc3240d5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.225+0000 7f9cebe00640 1 Processor -- start 2024-04-26T23:01:38.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.225+0000 7f9cebe00640 1 -- start start 2024-04-26T23:01:38.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.223+0000 7fc32a800640 1 -- 172.21.15.40:0/825175492 shutdown_connections 2024-04-26T23:01:38.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.223+0000 7fc32a800640 1 -- 172.21.15.40:0/825175492 wait complete. 2024-04-26T23:01:38.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.226+0000 7fc32a800640 1 Processor -- start 2024-04-26T23:01:38.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.227+0000 7f9cebe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cec0b4770 0x7f9cec159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.227+0000 7fc32a800640 1 -- start start 2024-04-26T23:01:38.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.227+0000 7f9cebe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cec0b63e0 0x7f9cec1f8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.227+0000 7f9cebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cec161850 0x7f9cec155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.228+0000 7f9cebe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec0d2210 con 0x7f9cec161850 2024-04-26T23:01:38.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.228+0000 7f9cebe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec0d2090 con 0x7f9cec0b63e0 2024-04-26T23:01:38.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.228+0000 7f9cebe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9cec162170 con 0x7f9cec0b4770 2024-04-26T23:01:38.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.228+0000 7f9cea000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cec0b63e0 0x7f9cec1f8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.228+0000 7f9cea000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cec0b63e0 0x7f9cec1f8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49900/0 (socket says 172.21.15.40:49900) 2024-04-26T23:01:38.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.228+0000 7f9cea000640 1 -- 172.21.15.40:0/147112201 learned_addr learned my addr 172.21.15.40:0/147112201 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:38.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.228+0000 7f9ceaa00640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cec0b4770 0x7f9cec159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.229+0000 7f9ceb400640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cec161850 0x7f9cec155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.229+0000 7f9cea000640 1 -- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cec0b4770 msgr2=0x7f9cec159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.229+0000 7f9cea000640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cec0b4770 0x7f9cec159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.229+0000 7f9cea000640 1 -- 172.21.15.40:0/147112201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cec161850 msgr2=0x7f9cec155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.229+0000 7f9cea000640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cec161850 0x7f9cec155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.229+0000 7f9cea000640 1 -- 172.21.15.40:0/147112201 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec15ab70 con 0x7f9cec0b63e0 2024-04-26T23:01:38.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.229+0000 7f9ceb400640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cec161850 0x7f9cec155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.230+0000 7fc32a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3240b4770 0x7fc3241604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.230+0000 7fc32a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3240d1ec0 0x7fc3240cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.230+0000 7fc329400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3240b4770 0x7fc3241604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.230+0000 7fc32a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc324161850 0x7fc3240b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.230+0000 7fc329400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3240b4770 0x7fc3241604f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38908/0 (socket says 172.21.15.40:38908) 2024-04-26T23:01:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.230+0000 7fc329400640 1 -- 172.21.15.40:0/3746284678 learned_addr learned my addr 172.21.15.40:0/3746284678 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.231+0000 7fc32a800640 1 -- 172.21.15.40:0/3746284678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc3240b6560 con 0x7fc3240b4770 2024-04-26T23:01:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.231+0000 7fc32a800640 1 -- 172.21.15.40:0/3746284678 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc3240b63e0 con 0x7fc324161850 2024-04-26T23:01:38.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.231+0000 7fc32a800640 1 -- 172.21.15.40:0/3746284678 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc3240b66e0 con 0x7fc3240d1ec0 2024-04-26T23:01:38.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.231+0000 7fc328a00640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3240d1ec0 0x7fc3240cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.231+0000 7f9cea000640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cec0b63e0 0x7f9cec1f8cb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c530 tx=0x7f9cdc00c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.231+0000 7fc329e00640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc324161850 0x7fc3240b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.231+0000 7f9ce2a00640 1 -- 172.21.15.40:0/147112201 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc017070 con 0x7f9cec0b63e0 2024-04-26T23:01:38.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.231+0000 7f9ce2a00640 1 -- 172.21.15.40:0/147112201 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cdc005ce0 con 0x7f9cec0b63e0 2024-04-26T23:01:38.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.231+0000 7f9ce2a00640 1 -- 172.21.15.40:0/147112201 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc007270 con 0x7f9cec0b63e0 2024-04-26T23:01:38.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.231+0000 7f9cebe00640 1 -- 172.21.15.40:0/147112201 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cec159ff0 con 0x7f9cec0b63e0 2024-04-26T23:01:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.231+0000 7f9cebe00640 1 -- 172.21.15.40:0/147112201 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cec15a5b0 con 0x7f9cec0b63e0 2024-04-26T23:01:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.232+0000 7fc329400640 1 -- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3240d1ec0 msgr2=0x7fc3240cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.232+0000 7fc329400640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3240d1ec0 0x7fc3240cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.232+0000 7fc329400640 1 -- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc324161850 msgr2=0x7fc3240b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.232+0000 7fc328a00640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3240d1ec0 0x7fc3240cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.233+0000 7fc329400640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc324161850 0x7fc3240b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.233+0000 7fc329400640 1 -- 172.21.15.40:0/3746284678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32409ebb0 con 0x7fc3240b4770 2024-04-26T23:01:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.233+0000 7fc329400640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3240b4770 0x7fc3241604f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc31800e870 tx=0x7fc31800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.234+0000 7fc321600640 1 -- 172.21.15.40:0/3746284678 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc318017070 con 0x7fc3240b4770 2024-04-26T23:01:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.234+0000 7fc32a800640 1 -- 172.21.15.40:0/3746284678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc32409ee50 con 0x7fc3240b4770 2024-04-26T23:01:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.234+0000 7f9cebe00640 1 -- 172.21.15.40:0/147112201 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cec069a80 con 0x7f9cec0b63e0 2024-04-26T23:01:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.234+0000 7fc32a800640 1 -- 172.21.15.40:0/3746284678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc32409f410 con 0x7fc3240b4770 2024-04-26T23:01:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.235+0000 7fc321600640 1 -- 172.21.15.40:0/3746284678 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc31800c940 con 0x7fc3240b4770 2024-04-26T23:01:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.235+0000 7fc321600640 1 -- 172.21.15.40:0/3746284678 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31800cd30 con 0x7fc3240b4770 2024-04-26T23:01:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.237+0000 7f9ce2a00640 1 -- 172.21.15.40:0/147112201 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9cdc01a070 con 0x7f9cec0b63e0 2024-04-26T23:01:38.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.238+0000 7fc321600640 1 -- 172.21.15.40:0/3746284678 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc31801a070 con 0x7fc3240b4770 2024-04-26T23:01:38.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.239+0000 7fc32a800640 1 -- 172.21.15.40:0/3746284678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc300005290 con 0x7fc3240b4770 2024-04-26T23:01:38.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.239+0000 7f9ce2a00640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9ccc06c9c0 0x7f9ccc06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.239+0000 7f9ce2a00640 1 -- 172.21.15.40:0/147112201 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9cdc08b290 con 0x7f9cec0b63e0 2024-04-26T23:01:38.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.241+0000 7f9ceaa00640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9ccc06c9c0 0x7f9ccc06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.240+0000 7fc321600640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc30406c8f0 0x7fc30406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:38.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.240+0000 7fc321600640 1 -- 172.21.15.40:0/3746284678 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc31808bbd0 con 0x7fc3240b4770 2024-04-26T23:01:38.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.241+0000 7fc328a00640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc30406c8f0 0x7fc30406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:38.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.243+0000 7fc328a00640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc30406c8f0 0x7fc30406edb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc310034000 tx=0x7fc31002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.243+0000 7f9ceaa00640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9ccc06c9c0 0x7f9ccc06ee80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0cb530 tx=0x7f9ce400d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:38.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.250+0000 7f9ce2a00640 1 -- 172.21.15.40:0/147112201 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9cdc058d10 con 0x7f9cec0b63e0 2024-04-26T23:01:38.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.250+0000 7fc321600640 1 -- 172.21.15.40:0/3746284678 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc318059700 con 0x7fc3240b4770 2024-04-26T23:01:38.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.532+0000 7f9893200640 1 -- 172.21.15.40:0/1751149442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f986c005580 con 0x7f988c03ca40 2024-04-26T23:01:38.534 INFO:teuthology.orchestra.run.smithi040.stdout:55834574857 2024-04-26T23:01:38.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.533+0000 7f988a000640 1 -- 172.21.15.40:0/1751149442 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f988005d970 con 0x7f988c03ca40 2024-04-26T23:01:38.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.533+0000 7fd3a2200640 1 -- 172.21.15.40:0/4257870309 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd378005580 con 0x7fd39c0d2b60 2024-04-26T23:01:38.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.534+0000 7fd398c00640 1 -- 172.21.15.40:0/4257870309 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd38807d510 con 0x7fd39c0d2b60 2024-04-26T23:01:38.537 INFO:teuthology.orchestra.run.smithi040.stdout:55834574857 2024-04-26T23:01:38.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.536+0000 7f9893200640 1 -- 172.21.15.40:0/1751149442 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f987406c9f0 msgr2=0x7f987406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.536+0000 7f9893200640 1 --2- 172.21.15.40:0/1751149442 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f987406c9f0 0x7f987406eeb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f988c0568f0 tx=0x7f98940ab000 comp rx=0 tx=0).stop 2024-04-26T23:01:38.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.536+0000 7f9893200640 1 -- 172.21.15.40:0/1751149442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c03ca40 msgr2=0x7f988c03cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.536+0000 7f9893200640 1 --2- 172.21.15.40:0/1751149442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c03ca40 0x7f988c03cef0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f988000e8a0 tx=0x7f988000ed60 comp rx=0 tx=0).stop 2024-04-26T23:01:38.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.537+0000 7f9893200640 1 -- 172.21.15.40:0/1751149442 shutdown_connections 2024-04-26T23:01:38.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.537+0000 7f9893200640 1 --2- 172.21.15.40:0/1751149442 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f987406c9f0 0x7f987406eeb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.537+0000 7f9893200640 1 --2- 172.21.15.40:0/1751149442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c03ca40 0x7f988c03cef0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.537+0000 7f9893200640 1 --2- 172.21.15.40:0/1751149442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f988c053cb0 0x7f988c188950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.537+0000 7f9893200640 1 --2- 172.21.15.40:0/1751149442 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c03c0a0 0x7f988c186e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.537+0000 7f9893200640 1 -- 172.21.15.40:0/1751149442 >> 172.21.15.40:0/1751149442 conn(0x7f988c058ee0 msgr2=0x7f988c038940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.537+0000 7f9893200640 1 -- 172.21.15.40:0/1751149442 shutdown_connections 2024-04-26T23:01:38.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.538+0000 7f9893200640 1 -- 172.21.15.40:0/1751149442 wait complete. 2024-04-26T23:01:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.538+0000 7fd3a2200640 1 -- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd37c06caf0 msgr2=0x7fd37c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.538+0000 7fd3a2200640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd37c06caf0 0x7fd37c06efb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd3900011e0 tx=0x7fd390009040 comp rx=0 tx=0).stop 2024-04-26T23:01:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.538+0000 7fd3a2200640 1 -- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd39c0d2b60 msgr2=0x7fd39c0cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.538+0000 7fd3a2200640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd39c0d2b60 0x7fd39c0cde70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd388031000 tx=0x7fd3880046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.539+0000 7fd3a2200640 1 -- 172.21.15.40:0/4257870309 shutdown_connections 2024-04-26T23:01:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.539+0000 7fd3a2200640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd37c06caf0 0x7fd37c06efb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.539+0000 7fd3a2200640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c15f210 0x7fd39c0b8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.539+0000 7fd3a2200640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd39c0d2b60 0x7fd39c0cde70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.539+0000 7fd3a2200640 1 --2- 172.21.15.40:0/4257870309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd39c0d0d10 0x7fd39c160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.539+0000 7fd3a2200640 1 -- 172.21.15.40:0/4257870309 >> 172.21.15.40:0/4257870309 conn(0x7fd39c0d4f90 msgr2=0x7fd39c0b5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.540+0000 7fd3a2200640 1 -- 172.21.15.40:0/4257870309 shutdown_connections 2024-04-26T23:01:38.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.540+0000 7fd3a2200640 1 -- 172.21.15.40:0/4257870309 wait complete. 2024-04-26T23:01:38.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.559+0000 7f1469400640 1 -- 172.21.15.40:0/2638236267 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f143c005850 con 0x7f146415f680 2024-04-26T23:01:38.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.561+0000 7f145be00640 1 -- 172.21.15.40:0/2638236267 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1454079f70 con 0x7f146415f680 2024-04-26T23:01:38.561 INFO:teuthology.orchestra.run.smithi040.stdout:55834574859 2024-04-26T23:01:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.564+0000 7f1469400640 1 -- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f144006cab0 msgr2=0x7f144006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.564+0000 7f1469400640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f144006cab0 0x7f144006ef70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f144c005880 tx=0x7f144c009040 comp rx=0 tx=0).stop 2024-04-26T23:01:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.564+0000 7f1469400640 1 -- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f146415f680 msgr2=0x7f14640a20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.564+0000 7f1469400640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f146415f680 0x7f14640a20f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1454031000 tx=0x7f14540046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.565+0000 7f1469400640 1 -- 172.21.15.40:0/2638236267 shutdown_connections 2024-04-26T23:01:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.565+0000 7f1469400640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f144006cab0 0x7f144006ef70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.565+0000 7f1469400640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14640cdc60 0x7f14640ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.565+0000 7f1469400640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f146415f680 0x7f14640a20f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.565+0000 7f1469400640 1 --2- 172.21.15.40:0/2638236267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14640d1770 0x7f14640cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.565+0000 7f1469400640 1 -- 172.21.15.40:0/2638236267 >> 172.21.15.40:0/2638236267 conn(0x7f14640d4dd0 msgr2=0x7f14640d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.565+0000 7f1469400640 1 -- 172.21.15.40:0/2638236267 shutdown_connections 2024-04-26T23:01:38.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.565+0000 7f1469400640 1 -- 172.21.15.40:0/2638236267 wait complete. 2024-04-26T23:01:38.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.566+0000 7f5e16c00640 1 -- 172.21.15.40:0/4011854130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f5e080578f0 con 0x7f5e0803c360 2024-04-26T23:01:38.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.566+0000 7f5e0d600640 1 -- 172.21.15.40:0/4011854130 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5e0405cf90 con 0x7f5e0803c360 2024-04-26T23:01:38.567 INFO:teuthology.orchestra.run.smithi040.stdout:55834574858 2024-04-26T23:01:38.568 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.3 2024-04-26T23:01:38.568 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.1 2024-04-26T23:01:38.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.570+0000 7f5e16c00640 1 -- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5df006c8f0 msgr2=0x7f5df006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.570+0000 7f5e16c00640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5df006c8f0 0x7f5df006edb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc006f40 tx=0x7f5dfc006e90 comp rx=0 tx=0).stop 2024-04-26T23:01:38.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.570+0000 7f5e16c00640 1 -- 172.21.15.40:0/4011854130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e0803c360 msgr2=0x7f5e080d8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.570+0000 7f5e16c00640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e0803c360 0x7f5e080d8280 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400c430 tx=0x7f5e0400c8f0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.572+0000 7f5e16c00640 1 -- 172.21.15.40:0/4011854130 shutdown_connections 2024-04-26T23:01:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.572+0000 7f5e16c00640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5df006c8f0 0x7f5df006edb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.572+0000 7f5e16c00640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e080e7a50 0x7f5e08055d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.572+0000 7f5e16c00640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e080548a0 0x7f5e08055830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.572+0000 7f5e16c00640 1 --2- 172.21.15.40:0/4011854130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e0803c360 0x7f5e080d8280 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.572+0000 7f5e16c00640 1 -- 172.21.15.40:0/4011854130 >> 172.21.15.40:0/4011854130 conn(0x7f5e08056cd0 msgr2=0x7f5e08008460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.572+0000 7f5e16c00640 1 -- 172.21.15.40:0/4011854130 shutdown_connections 2024-04-26T23:01:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.572+0000 7f5e16c00640 1 -- 172.21.15.40:0/4011854130 wait complete. 2024-04-26T23:01:38.588 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.2 2024-04-26T23:01:38.593 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.0 2024-04-26T23:01:38.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.607+0000 7f6de9400640 1 -- 172.21.15.40:0/2221406398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f6de40d2d00 con 0x7f6de40b65d0 2024-04-26T23:01:38.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.608+0000 7f6ddaa00640 1 -- 172.21.15.40:0/2221406398 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6dd400d040 con 0x7f6de40b65d0 2024-04-26T23:01:38.608 INFO:teuthology.orchestra.run.smithi040.stdout:55834574857 2024-04-26T23:01:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.610+0000 7f6de9400640 1 -- 172.21.15.40:0/2221406398 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6dc006c8f0 msgr2=0x7f6dc006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.610+0000 7f6de9400640 1 --2- 172.21.15.40:0/2221406398 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6dc006c8f0 0x7f6dc006edb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6de41fc5a0 tx=0x7f6dcc009040 comp rx=0 tx=0).stop 2024-04-26T23:01:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.610+0000 7f6de9400640 1 -- 172.21.15.40:0/2221406398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6de40b65d0 msgr2=0x7f6de4164050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.610+0000 7f6de9400640 1 --2- 172.21.15.40:0/2221406398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6de40b65d0 0x7f6de4164050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400b440 tx=0x7f6dd400b900 comp rx=0 tx=0).stop 2024-04-26T23:01:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.610+0000 7f6de9400640 1 -- 172.21.15.40:0/2221406398 shutdown_connections 2024-04-26T23:01:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.610+0000 7f6de9400640 1 --2- 172.21.15.40:0/2221406398 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6dc006c8f0 0x7f6dc006edb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.610+0000 7f6de9400640 1 --2- 172.21.15.40:0/2221406398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6de40d2090 0x7f6de4164590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.610+0000 7f6de9400640 1 --2- 172.21.15.40:0/2221406398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6de40b65d0 0x7f6de4164050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.610+0000 7f6de9400640 1 --2- 172.21.15.40:0/2221406398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6de40b61f0 0x7f6de40cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.610+0000 7f6de9400640 1 -- 172.21.15.40:0/2221406398 >> 172.21.15.40:0/2221406398 conn(0x7f6de40d4f90 msgr2=0x7f6de40b9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.610+0000 7f6de9400640 1 -- 172.21.15.40:0/2221406398 shutdown_connections 2024-04-26T23:01:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.611+0000 7f6de9400640 1 -- 172.21.15.40:0/2221406398 wait complete. 2024-04-26T23:01:38.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.625+0000 7f9cebe00640 1 -- 172.21.15.40:0/147112201 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f9cec0b5270 con 0x7f9cec0b63e0 2024-04-26T23:01:38.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.626+0000 7f9ce2a00640 1 -- 172.21.15.40:0/147112201 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9cdc05d010 con 0x7f9cec0b63e0 2024-04-26T23:01:38.627 INFO:teuthology.orchestra.run.smithi040.stdout:55834574857 2024-04-26T23:01:38.628 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.5 2024-04-26T23:01:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.629+0000 7f9cebe00640 1 -- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9ccc06c9c0 msgr2=0x7f9ccc06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.629+0000 7f9cebe00640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9ccc06c9c0 0x7f9ccc06ee80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0cb530 tx=0x7f9ce400d040 comp rx=0 tx=0).stop 2024-04-26T23:01:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.629+0000 7f9cebe00640 1 -- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cec0b63e0 msgr2=0x7f9cec1f8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.629+0000 7f9cebe00640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cec0b63e0 0x7f9cec1f8cb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c530 tx=0x7f9cdc00c9f0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.629+0000 7f9cebe00640 1 -- 172.21.15.40:0/147112201 shutdown_connections 2024-04-26T23:01:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.629+0000 7f9cebe00640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9ccc06c9c0 0x7f9ccc06ee80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.629+0000 7f9cebe00640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cec161850 0x7f9cec155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.629+0000 7f9cebe00640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cec0b63e0 0x7f9cec1f8cb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.629+0000 7f9cebe00640 1 --2- 172.21.15.40:0/147112201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9cec0b4770 0x7f9cec159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.629+0000 7f9cebe00640 1 -- 172.21.15.40:0/147112201 >> 172.21.15.40:0/147112201 conn(0x7f9cec0d4e10 msgr2=0x7f9cec1602c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.630+0000 7f9cebe00640 1 -- 172.21.15.40:0/147112201 shutdown_connections 2024-04-26T23:01:38.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.630+0000 7f9cebe00640 1 -- 172.21.15.40:0/147112201 wait complete. 2024-04-26T23:01:38.645 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.7 2024-04-26T23:01:38.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.665+0000 7f2107c00640 1 -- 172.21.15.40:0/4270901991 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f2100162830 con 0x7f20e8004bd0 2024-04-26T23:01:38.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.666+0000 7f20fcc00640 1 -- 172.21.15.40:0/4270901991 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f20f407dfc0 con 0x7f20e8004bd0 2024-04-26T23:01:38.666 INFO:teuthology.orchestra.run.smithi040.stdout:55834574857 2024-04-26T23:01:38.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.668+0000 7f2107c00640 1 -- 172.21.15.40:0/4270901991 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f20e006c940 msgr2=0x7f20e006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.668+0000 7f2107c00640 1 --2- 172.21.15.40:0/4270901991 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f20e006c940 0x7f20e006ee00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f210015e810 tx=0x7f20ec009040 comp rx=0 tx=0).stop 2024-04-26T23:01:38.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.668+0000 7f2107c00640 1 -- 172.21.15.40:0/4270901991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20e8004bd0 msgr2=0x7f210015b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.668+0000 7f2107c00640 1 --2- 172.21.15.40:0/4270901991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20e8004bd0 0x7f210015b590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20f4007fd0 tx=0x7f20f4001640 comp rx=0 tx=0).stop 2024-04-26T23:01:38.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.668+0000 7f2107c00640 1 -- 172.21.15.40:0/4270901991 shutdown_connections 2024-04-26T23:01:38.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.668+0000 7f2107c00640 1 --2- 172.21.15.40:0/4270901991 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f20e006c940 0x7f20e006ee00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.668+0000 7f2107c00640 1 --2- 172.21.15.40:0/4270901991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21000b2860 0x7f21000b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.668+0000 7f2107c00640 1 --2- 172.21.15.40:0/4270901991 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2100161520 0x7f2100161980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.668+0000 7f2107c00640 1 --2- 172.21.15.40:0/4270901991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20e8004bd0 0x7f210015b590 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.668+0000 7f2107c00640 1 -- 172.21.15.40:0/4270901991 >> 172.21.15.40:0/4270901991 conn(0x7f21000d4f90 msgr2=0x7f210015d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.668+0000 7f2107c00640 1 -- 172.21.15.40:0/4270901991 shutdown_connections 2024-04-26T23:01:38.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.668+0000 7f2107c00640 1 -- 172.21.15.40:0/4270901991 wait complete. 2024-04-26T23:01:38.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.680+0000 7fc32a800640 1 -- 172.21.15.40:0/3746284678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fc300005850 con 0x7fc3240b4770 2024-04-26T23:01:38.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.681+0000 7fc321600640 1 -- 172.21.15.40:0/3746284678 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc31805da00 con 0x7fc3240b4770 2024-04-26T23:01:38.681 INFO:teuthology.orchestra.run.smithi040.stdout:55834574857 2024-04-26T23:01:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.683+0000 7fc32a800640 1 -- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc30406c8f0 msgr2=0x7fc30406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.683+0000 7fc32a800640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc30406c8f0 0x7fc30406edb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc310034000 tx=0x7fc31002e000 comp rx=0 tx=0).stop 2024-04-26T23:01:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.683+0000 7fc32a800640 1 -- 172.21.15.40:0/3746284678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3240b4770 msgr2=0x7fc3241604f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.683+0000 7fc32a800640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3240b4770 0x7fc3241604f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc31800e870 tx=0x7fc31800ed30 comp rx=0 tx=0).stop 2024-04-26T23:01:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.684+0000 7fc32a800640 1 -- 172.21.15.40:0/3746284678 shutdown_connections 2024-04-26T23:01:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.684+0000 7fc32a800640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc30406c8f0 0x7fc30406edb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.684+0000 7fc32a800640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc324161850 0x7fc3240b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.684+0000 7fc32a800640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3240d1ec0 0x7fc3240cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.684+0000 7fc32a800640 1 --2- 172.21.15.40:0/3746284678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3240b4770 0x7fc3241604f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.684+0000 7fc32a800640 1 -- 172.21.15.40:0/3746284678 >> 172.21.15.40:0/3746284678 conn(0x7fc3240d4e10 msgr2=0x7fc32406e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:38.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.684+0000 7fc32a800640 1 -- 172.21.15.40:0/3746284678 shutdown_connections 2024-04-26T23:01:38.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:38.684+0000 7fc32a800640 1 -- 172.21.15.40:0/3746284678 wait complete. 2024-04-26T23:01:38.685 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.4 2024-04-26T23:01:38.703 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.6 2024-04-26T23:01:39.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T23:01:39.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T23:01:39.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T23:01:39.594 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T23:01:39.628 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T23:01:39.646 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-26T23:01:39.687 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T23:01:39.703 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-26T23:01:39.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.801+0000 7f8813800640 1 Processor -- start 2024-04-26T23:01:39.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.801+0000 7f8813800640 1 -- start start 2024-04-26T23:01:39.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.803+0000 7f8813800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f880c0d2090 0x7f880c0d0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.803+0000 7f8813800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f880c0b65d0 0x7f880c0d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.803+0000 7f8813800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f880c0b7f80 con 0x7f880c0b61f0 2024-04-26T23:01:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.803+0000 7f8813800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f880c0b88b0 con 0x7f880c0b65d0 2024-04-26T23:01:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.803+0000 7f8813800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f880c0d2800 con 0x7f880c0d2090 2024-04-26T23:01:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.804+0000 7f880be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f880c0d2090 0x7f880c0d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.804+0000 7f880be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f880c0d2090 0x7f880c0d0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49918/0 (socket says 172.21.15.40:49918) 2024-04-26T23:01:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.804+0000 7f880be00640 1 -- 172.21.15.40:0/1847912935 learned_addr learned my addr 172.21.15.40:0/1847912935 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.805+0000 7fe1d7a00640 1 Processor -- start 2024-04-26T23:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.804+0000 7f8811800640 1 --2- 172.21.15.40:0/1847912935 >> v2:172.21.15.40:3300/0 conn(0x7f880c0b65d0 0x7f880c0d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.805+0000 7f880be00640 1 -- 172.21.15.40:0/1847912935 >> v2:172.21.15.40:3300/0 conn(0x7f880c0b65d0 msgr2=0x7f880c0d0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.805+0000 7fe1d7a00640 1 -- start start 2024-04-26T23:01:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.805+0000 7f880be00640 1 --2- 172.21.15.40:0/1847912935 >> v2:172.21.15.40:3300/0 conn(0x7f880c0b65d0 0x7f880c0d0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.805+0000 7f880be00640 1 -- 172.21.15.40:0/1847912935 >> v1:172.21.15.40:6789/0 conn(0x7f880c0b61f0 legacy=0x7f880c0d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.805+0000 7f880be00640 1 -- 172.21.15.40:0/1847912935 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880c0b8260 con 0x7f880c0d2090 2024-04-26T23:01:39.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.807+0000 7fe1d7a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1d0073f30 0x7fe1d00cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.807+0000 7fe1d7a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe1d01671e0 0x7fe1d0167690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.807+0000 7fe1d7a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1d0163a50 con 0x7fe1d00c41d0 2024-04-26T23:01:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.807+0000 7fe1d7a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe1d00cf7a0 con 0x7fe1d01671e0 2024-04-26T23:01:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.807+0000 7fe1d7a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d00ceb90 con 0x7fe1d0073f30 2024-04-26T23:01:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.806+0000 7f8811800640 1 --2- 172.21.15.40:0/1847912935 >> v2:172.21.15.40:3300/0 conn(0x7f880c0b65d0 0x7f880c0d0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:39.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.807+0000 7fe1d5000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe1d00c41d0 0x7fe1d00cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41560/0 (socket says 172.21.15.40:41560) 2024-04-26T23:01:39.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.807+0000 7fe1d5000640 1 -- 172.21.15.40:0/2548769852 learned_addr learned my addr 172.21.15.40:0/2548769852 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:39.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.808+0000 7fe1cfe00640 1 --2- 172.21.15.40:0/2548769852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1d0073f30 0x7fe1d00cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.808+0000 7f880be00640 1 --2- 172.21.15.40:0/1847912935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f880c0d2090 0x7f880c0d0730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f880c0b8a30 tx=0x7f87fc02a760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e53a1a7cfe5d5c66 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.808+0000 7fe1d5a00640 1 --2- 172.21.15.40:0/2548769852 >> v2:172.21.15.103:3300/0 conn(0x7fe1d01671e0 0x7fe1d0167690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.808+0000 7f880b400640 1 -- 172.21.15.40:0/1847912935 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc036070 con 0x7f880c0d2090 2024-04-26T23:01:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.809+0000 7f880b400640 1 -- 172.21.15.40:0/1847912935 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87fc031600 con 0x7f880c0d2090 2024-04-26T23:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.809+0000 7fe1d5a00640 1 -- 172.21.15.40:0/2548769852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1d0073f30 msgr2=0x7fe1d00cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.809+0000 7fe1d5a00640 1 --2- 172.21.15.40:0/2548769852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1d0073f30 0x7fe1d00cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.809+0000 7fe1d5a00640 1 -- 172.21.15.40:0/2548769852 >> v1:172.21.15.40:6789/0 conn(0x7fe1d00c41d0 legacy=0x7fe1d00cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.809+0000 7fe1d5a00640 1 -- 172.21.15.40:0/2548769852 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d00cee70 con 0x7fe1d01671e0 2024-04-26T23:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.809+0000 7f880b400640 1 -- 172.21.15.40:0/1847912935 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc031a10 con 0x7f880c0d2090 2024-04-26T23:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.810+0000 7f8813800640 1 -- 172.21.15.40:0/1847912935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f880c0d2090 msgr2=0x7f880c0d0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.810+0000 7f8813800640 1 --2- 172.21.15.40:0/1847912935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f880c0d2090 0x7f880c0d0730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f880c0b8a30 tx=0x7f87fc02a760 comp rx=0 tx=0).stop 2024-04-26T23:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.810+0000 7f8813800640 1 -- 172.21.15.40:0/1847912935 shutdown_connections 2024-04-26T23:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.810+0000 7f8813800640 1 --2- 172.21.15.40:0/1847912935 >> v2:172.21.15.40:3300/0 conn(0x7f880c0b65d0 0x7f880c0d0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.810+0000 7f8813800640 1 --2- 172.21.15.40:0/1847912935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f880c0d2090 0x7f880c0d0730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.810+0000 7f8813800640 1 -- 172.21.15.40:0/1847912935 >> 172.21.15.40:0/1847912935 conn(0x7f880c0d4f90 msgr2=0x7f880c0d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.811+0000 7f8813800640 1 -- 172.21.15.40:0/1847912935 shutdown_connections 2024-04-26T23:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.811+0000 7f8813800640 1 -- 172.21.15.40:0/1847912935 wait complete. 2024-04-26T23:01:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.811+0000 7fe1d5a00640 1 --2- 172.21.15.40:0/2548769852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d01671e0 0x7fe1d0167690 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00cf920 tx=0x7fe1c802a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59d011476efdf7e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.812+0000 7fe1cf400640 1 -- 172.21.15.40:0/2548769852 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8007740 con 0x7fe1d01671e0 2024-04-26T23:01:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.812+0000 7fe1cf400640 1 -- 172.21.15.40:0/2548769852 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c80078e0 con 0x7fe1d01671e0 2024-04-26T23:01:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.812+0000 7fe1cf400640 1 -- 172.21.15.40:0/2548769852 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8007cb0 con 0x7fe1d01671e0 2024-04-26T23:01:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.813+0000 7fe1d7a00640 1 -- 172.21.15.40:0/2548769852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d01671e0 msgr2=0x7fe1d0167690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.813+0000 7fe1d7a00640 1 --2- 172.21.15.40:0/2548769852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d01671e0 0x7fe1d0167690 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00cf920 tx=0x7fe1c802a560 comp rx=0 tx=0).stop 2024-04-26T23:01:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.814+0000 7fe1d7a00640 1 -- 172.21.15.40:0/2548769852 shutdown_connections 2024-04-26T23:01:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.814+0000 7fe1d7a00640 1 --2- 172.21.15.40:0/2548769852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d01671e0 0x7fe1d0167690 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.814+0000 7fe1d7a00640 1 --2- 172.21.15.40:0/2548769852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1d0073f30 0x7fe1d00cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.814+0000 7fe1d7a00640 1 -- 172.21.15.40:0/2548769852 >> 172.21.15.40:0/2548769852 conn(0x7fe1d00d0be0 msgr2=0x7fe1d00d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:39.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.814+0000 7fe1d7a00640 1 -- 172.21.15.40:0/2548769852 shutdown_connections 2024-04-26T23:01:39.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.815+0000 7f8813800640 1 Processor -- start 2024-04-26T23:01:39.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.815+0000 7f8813800640 1 -- start start 2024-04-26T23:01:39.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.814+0000 7fe1d7a00640 1 -- 172.21.15.40:0/2548769852 wait complete. 2024-04-26T23:01:39.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.817+0000 7fe1d7a00640 1 Processor -- start 2024-04-26T23:01:39.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.817+0000 7fe1d7a00640 1 -- start start 2024-04-26T23:01:39.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.818+0000 7f8813800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f880c0b61f0 0x7f880c0cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.818+0000 7f8813800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f880c0b65d0 0x7f880c0b97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.818+0000 7f8813800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f880c0d2090 0x7f880c0b9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.818+0000 7f8811800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f880c0d2090 0x7f880c0b9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.819+0000 7f8811800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f880c0d2090 0x7f880c0b9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38922/0 (socket says 172.21.15.40:38922) 2024-04-26T23:01:39.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.819+0000 7f8811800640 1 -- 172.21.15.40:0/2373855407 learned_addr learned my addr 172.21.15.40:0/2373855407 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:39.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.819+0000 7f880be00640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f880c0b65d0 0x7f880c0b97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.819+0000 7f8813800640 1 -- 172.21.15.40:0/2373855407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f880c0c7d10 con 0x7f880c0d2090 2024-04-26T23:01:39.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.819+0000 7f8813800640 1 -- 172.21.15.40:0/2373855407 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f880c0c7b90 con 0x7f880c0b65d0 2024-04-26T23:01:39.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.819+0000 7f8813800640 1 -- 172.21.15.40:0/2373855407 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f880c0c7e90 con 0x7f880c0b61f0 2024-04-26T23:01:39.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.819+0000 7f8810e00640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f880c0b61f0 0x7f880c0cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.820+0000 7f8811800640 1 -- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f880c0b61f0 msgr2=0x7f880c0cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.821+0000 7f8811800640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f880c0b61f0 0x7f880c0cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.821+0000 7f8811800640 1 -- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f880c0b65d0 msgr2=0x7f880c0b97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.821+0000 7fe1d7a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1d0073f30 0x7fe1d015c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.821+0000 7f8811800640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f880c0b65d0 0x7f880c0b97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.821+0000 7fe1d7a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d00c41d0 0x7fe1d0163c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.821+0000 7f8811800640 1 -- 172.21.15.40:0/2373855407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880c0ce100 con 0x7f880c0d2090 2024-04-26T23:01:39.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.821+0000 7f880be00640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f880c0b65d0 0x7f880c0b97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:01:39.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.822+0000 7fe1d7a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d01671e0 0x7fe1d00c6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.822+0000 7fe1d5000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1d0073f30 0x7fe1d015c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.822+0000 7fe1d5000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1d0073f30 0x7fe1d015c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:47030/0 (socket says 172.21.15.40:47030) 2024-04-26T23:01:39.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.822+0000 7fe1d7a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d00ce7b0 con 0x7fe1d01671e0 2024-04-26T23:01:39.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.822+0000 7fe1d5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d01671e0 0x7fe1d00c6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.822+0000 7f8811800640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f880c0d2090 0x7f880c0b9d30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f880400c7f0 tx=0x7f880400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.822+0000 7fe1d5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d01671e0 0x7fe1d00c6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38934/0 (socket says 172.21.15.40:38934) 2024-04-26T23:01:39.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.822+0000 7fe1d5a00640 1 -- 172.21.15.40:0/1380773247 learned_addr learned my addr 172.21.15.40:0/1380773247 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:39.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.822+0000 7fe1d7a00640 1 -- 172.21.15.40:0/1380773247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d00cf430 con 0x7fe1d00c41d0 2024-04-26T23:01:39.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.823+0000 7fe1d7a00640 1 -- 172.21.15.40:0/1380773247 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d00ce930 con 0x7fe1d0073f30 2024-04-26T23:01:39.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.823+0000 7fe1cfe00640 1 --2- 172.21.15.40:0/1380773247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d00c41d0 0x7fe1d0163c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.823+0000 7fe1d5a00640 1 -- 172.21.15.40:0/1380773247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1d0073f30 msgr2=0x7fe1d015c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.823+0000 7fe1d5a00640 1 --2- 172.21.15.40:0/1380773247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1d0073f30 0x7fe1d015c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.823+0000 7fe1d5a00640 1 -- 172.21.15.40:0/1380773247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d00c41d0 msgr2=0x7fe1d0163c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.824+0000 7fe1d5a00640 1 --2- 172.21.15.40:0/1380773247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d00c41d0 0x7fe1d0163c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.824+0000 7fe1d5a00640 1 -- 172.21.15.40:0/1380773247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d01646c0 con 0x7fe1d01671e0 2024-04-26T23:01:39.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.824+0000 7f8808c00640 1 -- 172.21.15.40:0/2373855407 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8804015070 con 0x7f880c0d2090 2024-04-26T23:01:39.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.824+0000 7f8808c00640 1 -- 172.21.15.40:0/2373855407 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88040077c0 con 0x7f880c0d2090 2024-04-26T23:01:39.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.824+0000 7f8808c00640 1 -- 172.21.15.40:0/2373855407 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8804007bf0 con 0x7f880c0d2090 2024-04-26T23:01:39.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.825+0000 7fe1d5a00640 1 --2- 172.21.15.40:0/1380773247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d01671e0 0x7fe1d00c6790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009e80 tx=0x7fe1c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.825+0000 7f8813800640 1 -- 172.21.15.40:0/2373855407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f880c1fcc20 con 0x7f880c0d2090 2024-04-26T23:01:39.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.825+0000 7f8813800640 1 -- 172.21.15.40:0/2373855407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f880c1fd1e0 con 0x7f880c0d2090 2024-04-26T23:01:39.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.826+0000 7fe1ccc00640 1 -- 172.21.15.40:0/1380773247 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c803f070 con 0x7fe1d01671e0 2024-04-26T23:01:39.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.827+0000 7fe1d7a00640 1 -- 172.21.15.40:0/1380773247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d01649a0 con 0x7fe1d01671e0 2024-04-26T23:01:39.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.827+0000 7fe1d7a00640 1 -- 172.21.15.40:0/1380773247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d00b3c30 con 0x7fe1d01671e0 2024-04-26T23:01:39.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.827+0000 7f8813800640 1 -- 172.21.15.40:0/2373855407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f880c0738b0 con 0x7f880c0d2090 2024-04-26T23:01:39.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.832+0000 7fe1d7a00640 1 -- 172.21.15.40:0/1380773247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1d00c7970 con 0x7fe1d01671e0 2024-04-26T23:01:39.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.833+0000 7fe1ccc00640 1 -- 172.21.15.40:0/1380773247 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c8007840 con 0x7fe1d01671e0 2024-04-26T23:01:39.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.833+0000 7fe1ccc00640 1 -- 172.21.15.40:0/1380773247 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c803f070 con 0x7fe1d01671e0 2024-04-26T23:01:39.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.833+0000 7fe1ccc00640 1 -- 172.21.15.40:0/1380773247 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe1c8003b90 con 0x7fe1d01671e0 2024-04-26T23:01:39.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.835+0000 7fe1ccc00640 1 --2- 172.21.15.40:0/1380773247 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1b006cb70 0x7fe1b006f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.835+0000 7fe1ccc00640 1 -- 172.21.15.40:0/1380773247 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe1c80aceb0 con 0x7fe1d01671e0 2024-04-26T23:01:39.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.836+0000 7f8808c00640 1 -- 172.21.15.40:0/2373855407 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8804018070 con 0x7f880c0d2090 2024-04-26T23:01:39.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.837+0000 7f8808c00640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f87e806c9c0 0x7f87e806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.837+0000 7f8808c00640 1 -- 172.21.15.40:0/2373855407 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f880408bce0 con 0x7f880c0d2090 2024-04-26T23:01:39.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.839+0000 7fe1d5000640 1 --2- 172.21.15.40:0/1380773247 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1b006cb70 0x7fe1b006f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.839+0000 7f8810e00640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f87e806c9c0 0x7f87e806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.841+0000 7f8810e00640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f87e806c9c0 0x7f87e806ee80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f880c162ee0 tx=0x7f87f4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.843+0000 7f8808c00640 1 -- 172.21.15.40:0/2373855407 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f88040597f0 con 0x7f880c0d2090 2024-04-26T23:01:39.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.846+0000 7fe1d5000640 1 --2- 172.21.15.40:0/1380773247 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1b006cb70 0x7fe1b006f030 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc000f60 tx=0x7fe1bc000bc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.853+0000 7fe1ccc00640 1 -- 172.21.15.40:0/1380773247 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe1d00c7970 con 0x7fe1d01671e0 2024-04-26T23:01:39.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.875+0000 7fea4fe00640 1 Processor -- start 2024-04-26T23:01:39.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.875+0000 7fea4fe00640 1 -- start start 2024-04-26T23:01:39.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.878+0000 7fea4fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fea5015b4d0 0x7fea5006ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.878+0000 7fea4fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fea500b3290 0x7fea50165310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.879+0000 7fea4fe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea500b44c0 con 0x7fea500727a0 2024-04-26T23:01:39.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.879+0000 7fea4fe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fea500b5c90 con 0x7fea500b3290 2024-04-26T23:01:39.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.879+0000 7fea4fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fea500b5f30 con 0x7fea5015b4d0 2024-04-26T23:01:39.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.879+0000 7fea4e000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fea500727a0 0x7fea5015b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41570/0 (socket says 172.21.15.40:41570) 2024-04-26T23:01:39.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.879+0000 7fea4e000640 1 -- 172.21.15.40:0/2503861722 learned_addr learned my addr 172.21.15.40:0/2503861722 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:39.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.879+0000 7fea4f400640 1 --2- 172.21.15.40:0/2503861722 >> v2:172.21.15.40:3300/0 conn(0x7fea500b3290 0x7fea50165310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.879+0000 7fea4ea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fea5015b4d0 0x7fea5006ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.880+0000 7fea4f400640 1 -- 172.21.15.40:0/2503861722 >> v2:172.21.15.103:3300/0 conn(0x7fea5015b4d0 msgr2=0x7fea5006ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.880+0000 7fea4f400640 1 --2- 172.21.15.40:0/2503861722 >> v2:172.21.15.103:3300/0 conn(0x7fea5015b4d0 0x7fea5006ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.880+0000 7fea4f400640 1 -- 172.21.15.40:0/2503861722 >> v1:172.21.15.40:6789/0 conn(0x7fea500727a0 legacy=0x7fea5015b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.880+0000 7fea4f400640 1 -- 172.21.15.40:0/2503861722 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea50072ff0 con 0x7fea500b3290 2024-04-26T23:01:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.883+0000 7fea4f400640 1 --2- 172.21.15.40:0/2503861722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea500b3290 0x7fea50165310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fea500b3770 tx=0x7fea4802ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0dcdf0645fbe00d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.883+0000 7fea4d600640 1 -- 172.21.15.40:0/2503861722 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea48036020 con 0x7fea500b3290 2024-04-26T23:01:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.883+0000 7fea4d600640 1 -- 172.21.15.40:0/2503861722 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea48032880 con 0x7fea500b3290 2024-04-26T23:01:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.883+0000 7fea4d600640 1 -- 172.21.15.40:0/2503861722 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea48036020 con 0x7fea500b3290 2024-04-26T23:01:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.884+0000 7fea4fe00640 1 -- 172.21.15.40:0/2503861722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea500b3290 msgr2=0x7fea50165310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.884+0000 7fea4fe00640 1 --2- 172.21.15.40:0/2503861722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea500b3290 0x7fea50165310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fea500b3770 tx=0x7fea4802ca40 comp rx=0 tx=0).stop 2024-04-26T23:01:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.885+0000 7fea4fe00640 1 -- 172.21.15.40:0/2503861722 shutdown_connections 2024-04-26T23:01:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.886+0000 7fea4fe00640 1 --2- 172.21.15.40:0/2503861722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea500b3290 0x7fea50165310 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.886+0000 7fea4fe00640 1 --2- 172.21.15.40:0/2503861722 >> v2:172.21.15.103:3300/0 conn(0x7fea5015b4d0 0x7fea5006ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.886+0000 7fea4fe00640 1 -- 172.21.15.40:0/2503861722 >> 172.21.15.40:0/2503861722 conn(0x7fea500b2770 msgr2=0x7fea500b2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.886+0000 7fea4fe00640 1 -- 172.21.15.40:0/2503861722 shutdown_connections 2024-04-26T23:01:39.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.886+0000 7fea4fe00640 1 -- 172.21.15.40:0/2503861722 wait complete. 2024-04-26T23:01:39.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.888+0000 7fe7f9200640 1 Processor -- start 2024-04-26T23:01:39.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.888+0000 7fe7f9200640 1 -- start start 2024-04-26T23:01:39.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.889+0000 7fea4fe00640 1 Processor -- start 2024-04-26T23:01:39.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.890+0000 7fea4fe00640 1 -- start start 2024-04-26T23:01:39.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.890+0000 7fe7f9200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7f40d32e0 0x7fe7f40d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.891+0000 7fe7f9200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7f4162c40 con 0x7fe7f40b8d20 2024-04-26T23:01:39.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.891+0000 7fe7f9200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7f40d2d80 con 0x7fe7f40b6220 2024-04-26T23:01:39.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.891+0000 7fe7f9200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe7f40d3c80 con 0x7fe7f40d32e0 2024-04-26T23:01:39.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.891+0000 7fe7f2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7f40d32e0 0x7fe7f40d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.891+0000 7fe7f3400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe7f40b6220 0x7fe7f40d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:55444/0 (socket says 172.21.15.40:55444) 2024-04-26T23:01:39.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.892+0000 7fe7f3400640 1 -- 172.21.15.40:0/3958023028 learned_addr learned my addr 172.21.15.40:0/3958023028 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:39.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.892+0000 7fea4fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea500727a0 0x7fea501434a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.892+0000 7fea4fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea500b3290 0x7fea5015a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fe7f2000640 1 -- 172.21.15.40:0/3958023028 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2995856749 0 0) 0x7fe7f40d2d80 con 0x7fe7f40b6220 2024-04-26T23:01:39.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fe7f2000640 1 -- 172.21.15.40:0/3958023028 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7d8003520 con 0x7fe7f40b6220 2024-04-26T23:01:39.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fe7f2a00640 1 -- 172.21.15.40:0/3958023028 >> v1:172.21.15.103:6789/0 conn(0x7fe7f40b6220 legacy=0x7fe7f40d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fea4fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5015b4d0 0x7fea501544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fea4fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fea500b4760 con 0x7fea500b3290 2024-04-26T23:01:39.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fea4fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea500735b0 con 0x7fea500727a0 2024-04-26T23:01:39.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fea4fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fea50072c20 con 0x7fea5015b4d0 2024-04-26T23:01:39.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fea4e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea500b3290 0x7fea5015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fea4e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea500b3290 0x7fea5015a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38946/0 (socket says 172.21.15.40:38946) 2024-04-26T23:01:39.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fea4e000640 1 -- 172.21.15.40:0/3447025234 learned_addr learned my addr 172.21.15.40:0/3447025234 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:39.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fea4ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea500727a0 0x7fea501434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fe7f2a00640 1 -- 172.21.15.40:0/3958023028 >> v1:172.21.15.40:6789/0 conn(0x7fe7f40b8d20 legacy=0x7fe7f40b91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fe7f2a00640 1 -- 172.21.15.40:0/3958023028 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f40b7c80 con 0x7fe7f40d32e0 2024-04-26T23:01:39.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.893+0000 7fea4f400640 1 --2- 172.21.15.40:0/3447025234 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5015b4d0 0x7fea501544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.894+0000 7fea4e000640 1 -- 172.21.15.40:0/3447025234 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5015b4d0 msgr2=0x7fea501544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.894+0000 7fea4e000640 1 --2- 172.21.15.40:0/3447025234 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5015b4d0 0x7fea501544b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.894+0000 7fea4e000640 1 -- 172.21.15.40:0/3447025234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea500727a0 msgr2=0x7fea501434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.894+0000 7fea4e000640 1 --2- 172.21.15.40:0/3447025234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea500727a0 0x7fea501434a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.894+0000 7fea4e000640 1 -- 172.21.15.40:0/3447025234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea50157df0 con 0x7fea500b3290 2024-04-26T23:01:39.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.895+0000 7fe7f2a00640 1 --2- 172.21.15.40:0/3958023028 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7f40d32e0 0x7fe7f40d3740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe7e40097d0 tx=0x7fe7e402a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5c637fd79f5af6c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.895+0000 7fea4e000640 1 --2- 172.21.15.40:0/3447025234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea500b3290 0x7fea5015a810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fea40001be0 tx=0x7fea40014420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.895+0000 7fea46a00640 1 -- 172.21.15.40:0/3447025234 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40014d40 con 0x7fea500b3290 2024-04-26T23:01:39.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.895+0000 7fe7f2000640 1 -- 172.21.15.40:0/3958023028 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4037070 con 0x7fe7f40d32e0 2024-04-26T23:01:39.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.896+0000 7fe7f2000640 1 -- 172.21.15.40:0/3958023028 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7e4032440 con 0x7fe7f40d32e0 2024-04-26T23:01:39.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.896+0000 7fe7f2000640 1 -- 172.21.15.40:0/3958023028 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4032850 con 0x7fe7f40d32e0 2024-04-26T23:01:39.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.896+0000 7fea4fe00640 1 -- 172.21.15.40:0/3447025234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea501580d0 con 0x7fea500b3290 2024-04-26T23:01:39.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.896+0000 7fea4fe00640 1 -- 172.21.15.40:0/3447025234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea50158690 con 0x7fea500b3290 2024-04-26T23:01:39.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.896+0000 7fea46a00640 1 -- 172.21.15.40:0/3447025234 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea400070d0 con 0x7fea500b3290 2024-04-26T23:01:39.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.896+0000 7fea46a00640 1 -- 172.21.15.40:0/3447025234 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40007500 con 0x7fea500b3290 2024-04-26T23:01:39.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.898+0000 7fea4fe00640 1 -- 172.21.15.40:0/3447025234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea28005290 con 0x7fea500b3290 2024-04-26T23:01:39.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.896+0000 7fe7f9200640 1 -- 172.21.15.40:0/3958023028 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7f40d32e0 msgr2=0x7fe7f40d3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.896+0000 7fe7f9200640 1 --2- 172.21.15.40:0/3958023028 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7f40d32e0 0x7fe7f40d3740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe7e40097d0 tx=0x7fe7e402a250 comp rx=0 tx=0).stop 2024-04-26T23:01:39.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.897+0000 7fe7f9200640 1 -- 172.21.15.40:0/3958023028 shutdown_connections 2024-04-26T23:01:39.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.897+0000 7fe7f9200640 1 --2- 172.21.15.40:0/3958023028 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7f40d32e0 0x7fe7f40d3740 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.897+0000 7fe7f9200640 1 -- 172.21.15.40:0/3958023028 >> 172.21.15.40:0/3958023028 conn(0x7fe7f40d4d90 msgr2=0x7fe7f40d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:39.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.897+0000 7fe7f9200640 1 -- 172.21.15.40:0/3958023028 shutdown_connections 2024-04-26T23:01:39.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.897+0000 7fe7f9200640 1 -- 172.21.15.40:0/3958023028 wait complete. 2024-04-26T23:01:39.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.900+0000 7fe7f9200640 1 Processor -- start 2024-04-26T23:01:39.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.900+0000 7fe7f9200640 1 -- start start 2024-04-26T23:01:39.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.902+0000 7fe7f9200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7f40b6220 0x7fe7f40cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.902+0000 7fe7f9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f40b8d20 0x7fe7f41f8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.902+0000 7fe7f9200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7f40d32e0 0x7fe7f41fcb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.903+0000 7fe7f3400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7f40b6220 0x7fe7f40cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.903+0000 7fe7f2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f40b8d20 0x7fe7f41f8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.903+0000 7fe7f3400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7f40b6220 0x7fe7f40cb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49978/0 (socket says 172.21.15.40:49978) 2024-04-26T23:01:39.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.903+0000 7fe7f3400640 1 -- 172.21.15.40:0/2229224975 learned_addr learned my addr 172.21.15.40:0/2229224975 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:39.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.903+0000 7fe7f9200640 1 -- 172.21.15.40:0/2229224975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f40b8520 con 0x7fe7f40b8d20 2024-04-26T23:01:39.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.903+0000 7fe7f9200640 1 -- 172.21.15.40:0/2229224975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f40b8240 con 0x7fe7f40b6220 2024-04-26T23:01:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.903+0000 7fe7f9200640 1 -- 172.21.15.40:0/2229224975 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe7f40d0aa0 con 0x7fe7f40d32e0 2024-04-26T23:01:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.903+0000 7fe7f3e00640 1 --2- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7f40d32e0 0x7fe7f41fcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.904+0000 7fe7f3400640 1 -- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7f40d32e0 msgr2=0x7fe7f41fcb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.904+0000 7fe7f3400640 1 --2- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7f40d32e0 0x7fe7f41fcb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.904+0000 7fe7f3400640 1 -- 172.21.15.40:0/2229224975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f40b8d20 msgr2=0x7fe7f41f8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.904+0000 7fe7f3400640 1 --2- 172.21.15.40:0/2229224975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f40b8d20 0x7fe7f41f8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.904+0000 7fe7f3400640 1 -- 172.21.15.40:0/2229224975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f41fc3e0 con 0x7fe7f40b6220 2024-04-26T23:01:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.907+0000 7fe7f3400640 1 --2- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7f40b6220 0x7fe7f40cb6a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe7e000c450 tx=0x7fe7e000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.905+0000 7fea46a00640 1 -- 172.21.15.40:0/3447025234 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fea40012110 con 0x7fea500b3290 2024-04-26T23:01:39.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.908+0000 7fe7eb400640 1 -- 172.21.15.40:0/2229224975 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0018070 con 0x7fe7f40b6220 2024-04-26T23:01:39.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.908+0000 7fe7f9200640 1 -- 172.21.15.40:0/2229224975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f41f98d0 con 0x7fe7f40b6220 2024-04-26T23:01:39.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.908+0000 7fe7f9200640 1 -- 172.21.15.40:0/2229224975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f41f9290 con 0x7fe7f40b6220 2024-04-26T23:01:39.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.907+0000 7fea46a00640 1 --2- 172.21.15.40:0/3447025234 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea3006c8f0 0x7fea3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.907+0000 7fea46a00640 1 -- 172.21.15.40:0/3447025234 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fea40060230 con 0x7fea500b3290 2024-04-26T23:01:39.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.908+0000 7fe7eb400640 1 -- 172.21.15.40:0/2229224975 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7e00074e0 con 0x7fe7f40b6220 2024-04-26T23:01:39.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.909+0000 7fe7eb400640 1 -- 172.21.15.40:0/2229224975 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0018070 con 0x7fe7f40b6220 2024-04-26T23:01:39.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.910+0000 7fe7f9200640 1 -- 172.21.15.40:0/2229224975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7cc005290 con 0x7fe7f40b6220 2024-04-26T23:01:39.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.910+0000 7fea4ea00640 1 --2- 172.21.15.40:0/3447025234 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea3006c8f0 0x7fea3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.911+0000 7fea4ea00640 1 --2- 172.21.15.40:0/3447025234 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea3006c8f0 0x7fea3006edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009ad0 tx=0x7fea3c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.911+0000 7fea46a00640 1 -- 172.21.15.40:0/3447025234 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fea4002b020 con 0x7fea500b3290 2024-04-26T23:01:39.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.915+0000 7fe7eb400640 1 -- 172.21.15.40:0/2229224975 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe7e00047e0 con 0x7fe7f40b6220 2024-04-26T23:01:39.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.917+0000 7fe7eb400640 1 --2- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe7d406ccf0 0x7fe7d406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.917+0000 7fe7eb400640 1 -- 172.21.15.40:0/2229224975 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe7e00072e0 con 0x7fe7f40b6220 2024-04-26T23:01:39.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.918+0000 7fe7f2a00640 1 --2- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe7d406ccf0 0x7fe7d406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.919+0000 7fe7f2a00640 1 --2- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe7d406ccf0 0x7fe7d406f1b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4034000 tx=0x7fe7e402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.927+0000 7fe7eb400640 1 -- 172.21.15.40:0/2229224975 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe7e0021080 con 0x7fe7f40b6220 2024-04-26T23:01:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.952+0000 7fce22800640 1 Processor -- start 2024-04-26T23:01:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.952+0000 7fce22800640 1 -- start start 2024-04-26T23:01:39.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.955+0000 7fce22800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce140e1300 0x7fce1403a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.956+0000 7fce22800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fce1400a450 0x7fce14055da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.956+0000 7fce1b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce140e1300 0x7fce1403a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.956+0000 7fce1b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce140e1300 0x7fce1403a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:47066/0 (socket says 172.21.15.40:47066) 2024-04-26T23:01:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.957+0000 7fce22800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce1403a6a0 con 0x7fce1400ab30 2024-04-26T23:01:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.957+0000 7fce22800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fce14055910 con 0x7fce1400a450 2024-04-26T23:01:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.957+0000 7fce22800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fce14054d00 con 0x7fce140e1300 2024-04-26T23:01:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.957+0000 7fce1be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fce1400a450 0x7fce14055da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.957+0000 7fce1b400640 1 -- 172.21.15.40:0/3344258104 learned_addr learned my addr 172.21.15.40:0/3344258104 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.958+0000 7fce1a000640 1 -- 172.21.15.40:0/3344258104 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 533157765 0 0) 0x7fce1403a6a0 con 0x7fce1400ab30 2024-04-26T23:01:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.958+0000 7fce1a000640 1 -- 172.21.15.40:0/3344258104 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce04003520 con 0x7fce1400ab30 2024-04-26T23:01:39.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.959+0000 7fce1b400640 1 -- 172.21.15.40:0/3344258104 >> v2:172.21.15.103:3300/0 conn(0x7fce1400a450 msgr2=0x7fce14055da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:39.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.959+0000 7fce1b400640 1 --2- 172.21.15.40:0/3344258104 >> v2:172.21.15.103:3300/0 conn(0x7fce1400a450 0x7fce14055da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.959+0000 7fce1b400640 1 -- 172.21.15.40:0/3344258104 >> v1:172.21.15.103:6789/0 conn(0x7fce1400ab30 legacy=0x7fce14009d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.959+0000 7fce1b400640 1 -- 172.21.15.40:0/3344258104 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce14054fe0 con 0x7fce140e1300 2024-04-26T23:01:39.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.961+0000 7fce1b400640 1 --2- 172.21.15.40:0/3344258104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce140e1300 0x7fce1403a160 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fce08009870 tx=0x7fce0802a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c11caf3f8004772 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.963+0000 7fce1a000640 1 -- 172.21.15.40:0/3344258104 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce08037070 con 0x7fce140e1300 2024-04-26T23:01:39.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.963+0000 7fce1a000640 1 -- 172.21.15.40:0/3344258104 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce0802ac70 con 0x7fce140e1300 2024-04-26T23:01:39.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.963+0000 7fce1a000640 1 -- 172.21.15.40:0/3344258104 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce10001670 con 0x7fce140e1300 2024-04-26T23:01:39.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.963+0000 7f551cc00640 1 Processor -- start 2024-04-26T23:01:39.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.964+0000 7fce22800640 1 -- 172.21.15.40:0/3344258104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce140e1300 msgr2=0x7fce1403a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.964+0000 7fce22800640 1 --2- 172.21.15.40:0/3344258104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce140e1300 0x7fce1403a160 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fce08009870 tx=0x7fce0802a1e0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.965+0000 7fce22800640 1 -- 172.21.15.40:0/3344258104 shutdown_connections 2024-04-26T23:01:39.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.965+0000 7fce22800640 1 --2- 172.21.15.40:0/3344258104 >> v2:172.21.15.103:3300/0 conn(0x7fce1400a450 0x7fce14055da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.965+0000 7fce22800640 1 --2- 172.21.15.40:0/3344258104 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce140e1300 0x7fce1403a160 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fce08009870 tx=0x7fce0802a1e0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.965+0000 7fce22800640 1 -- 172.21.15.40:0/3344258104 >> 172.21.15.40:0/3344258104 conn(0x7fce14056cc0 msgr2=0x7fce140570d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:39.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.964+0000 7f551cc00640 1 -- start start 2024-04-26T23:01:39.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.966+0000 7f551cc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f551815f540 0x7f55180d3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.966+0000 7f551cc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55180d2d70 con 0x7f55180b5ab0 2024-04-26T23:01:39.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.966+0000 7f551cc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55180d3050 con 0x7f55180b3ff0 2024-04-26T23:01:39.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.966+0000 7f551cc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f551815f920 con 0x7f551815f540 2024-04-26T23:01:39.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.967+0000 7fce22800640 1 -- 172.21.15.40:0/3344258104 shutdown_connections 2024-04-26T23:01:39.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.967+0000 7fce22800640 1 -- 172.21.15.40:0/3344258104 wait complete. 2024-04-26T23:01:39.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.967+0000 7f5515e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f551815f540 0x7f55180d3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.967+0000 7f550d400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f55180b3ff0 0x7f55180d1770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:55458/0 (socket says 172.21.15.40:55458) 2024-04-26T23:01:39.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.967+0000 7f550d400640 1 -- 172.21.15.40:0/2675668041 learned_addr learned my addr 172.21.15.40:0/2675668041 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:39.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.968+0000 7f5515400640 1 -- 172.21.15.40:0/2675668041 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3683141210 0 0) 0x7f55180d3050 con 0x7f55180b3ff0 2024-04-26T23:01:39.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.968+0000 7f5515400640 1 -- 172.21.15.40:0/2675668041 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54f8003520 con 0x7f55180b3ff0 2024-04-26T23:01:39.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.969+0000 7f5515e00640 1 -- 172.21.15.40:0/2675668041 >> v1:172.21.15.103:6789/0 conn(0x7f55180b3ff0 legacy=0x7f55180d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.969+0000 7f5515e00640 1 -- 172.21.15.40:0/2675668041 >> v1:172.21.15.40:6789/0 conn(0x7f55180b5ab0 legacy=0x7f55180d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.969+0000 7f5515400640 1 -- 172.21.15.40:0/2675668041 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3296092965 0 0) 0x7f55180d2d70 con 0x7f55180b5ab0 2024-04-26T23:01:39.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.969+0000 7f5515e00640 1 -- 172.21.15.40:0/2675668041 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55180d08e0 con 0x7f551815f540 2024-04-26T23:01:39.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.971+0000 7fce22800640 1 Processor -- start 2024-04-26T23:01:39.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.971+0000 7f5515e00640 1 --2- 172.21.15.40:0/2675668041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f551815f540 0x7f55180d3470 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f55000097a0 tx=0x7f550002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19f333e9f3de1be9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.971+0000 7fce22800640 1 -- start start 2024-04-26T23:01:39.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.972+0000 7f5515400640 1 -- 172.21.15.40:0/2675668041 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5500037070 con 0x7f551815f540 2024-04-26T23:01:39.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.972+0000 7f5515400640 1 -- 172.21.15.40:0/2675668041 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5500032440 con 0x7f551815f540 2024-04-26T23:01:39.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.972+0000 7f5515400640 1 -- 172.21.15.40:0/2675668041 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5500032810 con 0x7f551815f540 2024-04-26T23:01:39.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.973+0000 7fce22800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce1400a450 0x7fce14050670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.973+0000 7fce22800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1400ab30 0x7fce1402d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.973+0000 7f551cc00640 1 -- 172.21.15.40:0/2675668041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f551815f540 msgr2=0x7f55180d3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.973+0000 7f551cc00640 1 --2- 172.21.15.40:0/2675668041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f551815f540 0x7f55180d3470 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f55000097a0 tx=0x7f550002a1a0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.973+0000 7f551cc00640 1 -- 172.21.15.40:0/2675668041 shutdown_connections 2024-04-26T23:01:39.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.973+0000 7f551cc00640 1 --2- 172.21.15.40:0/2675668041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f551815f540 0x7f55180d3470 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.973+0000 7fce1b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce1400a450 0x7fce14050670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.973+0000 7f551cc00640 1 -- 172.21.15.40:0/2675668041 >> 172.21.15.40:0/2675668041 conn(0x7f55180d4f90 msgr2=0x7f55180d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:39.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.974+0000 7fce1b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce1400a450 0x7fce14050670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38960/0 (socket says 172.21.15.40:38960) 2024-04-26T23:01:39.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.974+0000 7f551cc00640 1 -- 172.21.15.40:0/2675668041 shutdown_connections 2024-04-26T23:01:39.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.974+0000 7fce1b400640 1 -- 172.21.15.40:0/2427236251 learned_addr learned my addr 172.21.15.40:0/2427236251 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:39.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.974+0000 7f551cc00640 1 -- 172.21.15.40:0/2675668041 wait complete. 2024-04-26T23:01:39.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.974+0000 7fce22800640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce140d6150 0x7fce140377a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.974+0000 7fce22800640 1 -- 172.21.15.40:0/2427236251 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fce14054890 con 0x7fce1400a450 2024-04-26T23:01:39.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.974+0000 7fce1aa00640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1400ab30 0x7fce1402d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.975+0000 7fce22800640 1 -- 172.21.15.40:0/2427236251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fce140555a0 con 0x7fce140d6150 2024-04-26T23:01:39.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.975+0000 7fce22800640 1 -- 172.21.15.40:0/2427236251 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fce14054a10 con 0x7fce1400ab30 2024-04-26T23:01:39.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.975+0000 7fce1be00640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce140d6150 0x7fce140377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.976+0000 7fce1aa00640 1 -- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce140d6150 msgr2=0x7fce140377a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.976+0000 7fce1aa00640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce140d6150 0x7fce140377a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.976+0000 7fce1aa00640 1 -- 172.21.15.40:0/2427236251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce1400a450 msgr2=0x7fce14050670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.976+0000 7fce1aa00640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce1400a450 0x7fce14050670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.977+0000 7fce1aa00640 1 -- 172.21.15.40:0/2427236251 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce14050d00 con 0x7fce1400ab30 2024-04-26T23:01:39.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.977+0000 7f551cc00640 1 Processor -- start 2024-04-26T23:01:39.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.977+0000 7fce1b400640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce1400a450 0x7fce14050670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:01:39.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.977+0000 7fce1be00640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce140d6150 0x7fce140377a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:01:39.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.977+0000 7f551cc00640 1 -- start start 2024-04-26T23:01:39.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.977+0000 7fce1aa00640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1400ab30 0x7fce1402d830 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fce1400af50 tx=0x7fce1000cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.978+0000 7fce0f400640 1 -- 172.21.15.40:0/2427236251 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce10016020 con 0x7fce1400ab30 2024-04-26T23:01:39.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.978+0000 7fce0f400640 1 -- 172.21.15.40:0/2427236251 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce1000a840 con 0x7fce1400ab30 2024-04-26T23:01:39.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.978+0000 7fce22800640 1 -- 172.21.15.40:0/2427236251 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce1404c100 con 0x7fce1400ab30 2024-04-26T23:01:39.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.978+0000 7fce22800640 1 -- 172.21.15.40:0/2427236251 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fce1404c6c0 con 0x7fce1400ab30 2024-04-26T23:01:39.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.978+0000 7fce0f400640 1 -- 172.21.15.40:0/2427236251 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1000ac30 con 0x7fce1400ab30 2024-04-26T23:01:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.980+0000 7f551cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55180b3ff0 0x7f551815b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.980+0000 7f551cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55180b5ab0 0x7f55180b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.980+0000 7f551cc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f551815f540 0x7f55180b9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.980+0000 7f551cc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5518068e50 con 0x7f55180b3ff0 2024-04-26T23:01:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.980+0000 7f551cc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f55180d0f30 con 0x7f55180b5ab0 2024-04-26T23:01:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.980+0000 7f551cc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f55180d2590 con 0x7f551815f540 2024-04-26T23:01:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.981+0000 7f5515e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55180b3ff0 0x7f551815b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.981+0000 7f5515e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55180b3ff0 0x7f551815b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38972/0 (socket says 172.21.15.40:38972) 2024-04-26T23:01:39.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.981+0000 7f5516800640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f551815f540 0x7f55180b9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.981+0000 7f550d400640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55180b5ab0 0x7f55180b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.981+0000 7f5515e00640 1 -- 172.21.15.40:0/1970361614 learned_addr learned my addr 172.21.15.40:0/1970361614 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:39.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.982+0000 7f5516800640 1 -- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55180b5ab0 msgr2=0x7f55180b8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.982+0000 7f5516800640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55180b5ab0 0x7f55180b8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.982+0000 7f5516800640 1 -- 172.21.15.40:0/1970361614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55180b3ff0 msgr2=0x7f551815b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:39.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.982+0000 7f5516800640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55180b3ff0 0x7f551815b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:39.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.982+0000 7f5516800640 1 -- 172.21.15.40:0/1970361614 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551815b580 con 0x7f551815f540 2024-04-26T23:01:39.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.983+0000 7f5516800640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f551815f540 0x7f55180b9140 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f551000c780 tx=0x7f551000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.983+0000 7f550ea00640 1 -- 172.21.15.40:0/1970361614 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510017070 con 0x7f551815f540 2024-04-26T23:01:39.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.983+0000 7f550ea00640 1 -- 172.21.15.40:0/1970361614 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5510003da0 con 0x7f551815f540 2024-04-26T23:01:39.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.983+0000 7f550ea00640 1 -- 172.21.15.40:0/1970361614 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510017070 con 0x7f551815f540 2024-04-26T23:01:39.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.983+0000 7f551cc00640 1 -- 172.21.15.40:0/1970361614 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55180b9770 con 0x7f551815f540 2024-04-26T23:01:39.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.984+0000 7f551cc00640 1 -- 172.21.15.40:0/1970361614 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55180b9d30 con 0x7f551815f540 2024-04-26T23:01:39.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.983+0000 7fce0f400640 1 -- 172.21.15.40:0/2427236251 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fce10019020 con 0x7fce1400ab30 2024-04-26T23:01:39.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.984+0000 7fce22800640 1 -- 172.21.15.40:0/2427236251 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce140057d0 con 0x7fce1400ab30 2024-04-26T23:01:39.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.986+0000 7fce0f400640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdfc06c8f0 0x7fcdfc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.985+0000 7f551cc00640 1 -- 172.21.15.40:0/1970361614 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55180694f0 con 0x7f551815f540 2024-04-26T23:01:39.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.986+0000 7fce0f400640 1 -- 172.21.15.40:0/2427236251 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fce1008c070 con 0x7fce1400ab30 2024-04-26T23:01:39.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.986+0000 7fce1b400640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdfc06c8f0 0x7fcdfc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.988+0000 7fce1b400640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdfc06c8f0 0x7fcdfc06edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce0802a220 tx=0x7fce0802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:39.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.988+0000 7f550ea00640 1 -- 172.21.15.40:0/1970361614 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f551001a070 con 0x7f551815f540 2024-04-26T23:01:39.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.990+0000 7f550ea00640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f54f406caf0 0x7f54f406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:39.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.990+0000 7f550ea00640 1 -- 172.21.15.40:0/1970361614 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5510056a00 con 0x7f551815f540 2024-04-26T23:01:39.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.991+0000 7f5515e00640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f54f406caf0 0x7f54f406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:39.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:39.992+0000 7f5515e00640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f54f406caf0 0x7f54f406efb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5500009560 tx=0x7f550002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:40.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.005+0000 7fce0f400640 1 -- 172.21.15.40:0/2427236251 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fce10020020 con 0x7fce1400ab30 2024-04-26T23:01:40.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.016+0000 7fdb26000640 1 Processor -- start 2024-04-26T23:01:40.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.017+0000 7fdb26000640 1 -- start start 2024-04-26T23:01:40.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.019+0000 7fdb26000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdb200d3b80 0x7fdb200b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.019+0000 7fdb26000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 0x7fdb200b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.019+0000 7fdb26000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb200d2d70 con 0x7fdb2015f540 2024-04-26T23:01:40.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.019+0000 7fdb26000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdb2015f920 con 0x7fdb200d3b80 2024-04-26T23:01:40.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.019+0000 7fdb26000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdb200d0f30 con 0x7fdb200b3ff0 2024-04-26T23:01:40.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.019+0000 7f550ea00640 1 -- 172.21.15.40:0/1970361614 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5510058960 con 0x7f551815f540 2024-04-26T23:01:40.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.020+0000 7fdb1ea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdb200d3b80 0x7fdb200b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.020+0000 7fdb1ea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdb200d3b80 0x7fdb200b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50024/0 (socket says 172.21.15.40:50024) 2024-04-26T23:01:40.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.020+0000 7fdb1fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 0x7fdb200b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.020+0000 7fdb1fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 0x7fdb200b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:47096/0 (socket says 172.21.15.40:47096) 2024-04-26T23:01:40.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.020+0000 7fdb1fe00640 1 -- 172.21.15.40:0/2773124074 learned_addr learned my addr 172.21.15.40:0/2773124074 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:40.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.021+0000 7fdb1fe00640 1 -- 172.21.15.40:0/2773124074 >> v2:172.21.15.103:3300/0 conn(0x7fdb200d3b80 msgr2=0x7fdb200b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:40.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.021+0000 7fdb1fe00640 1 --2- 172.21.15.40:0/2773124074 >> v2:172.21.15.103:3300/0 conn(0x7fdb200d3b80 0x7fdb200b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.021+0000 7fdb1fe00640 1 -- 172.21.15.40:0/2773124074 >> v1:172.21.15.103:6789/0 conn(0x7fdb2015f540 legacy=0x7fdb200d3470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T23:01:40.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.021+0000 7fdb1fe00640 1 -- 172.21.15.40:0/2773124074 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb200d3050 con 0x7fdb200b3ff0 2024-04-26T23:01:40.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.023+0000 7fdb1fe00640 1 --2- 172.21.15.40:0/2773124074 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 0x7fdb200b4fe0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009e50 tx=0x7fdb1802a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca51b726cc5b04a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:40.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.023+0000 7fdb1e000640 1 -- 172.21.15.40:0/2773124074 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18036070 con 0x7fdb200b3ff0 2024-04-26T23:01:40.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.024+0000 7fdb1e000640 1 -- 172.21.15.40:0/2773124074 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb18031600 con 0x7fdb200b3ff0 2024-04-26T23:01:40.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.024+0000 7fdb1e000640 1 -- 172.21.15.40:0/2773124074 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb180319d0 con 0x7fdb200b3ff0 2024-04-26T23:01:40.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.025+0000 7fdb26000640 1 -- 172.21.15.40:0/2773124074 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 msgr2=0x7fdb200b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.025+0000 7fdb26000640 1 --2- 172.21.15.40:0/2773124074 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 0x7fdb200b4fe0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009e50 tx=0x7fdb1802a6d0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.025+0000 7fdb26000640 1 -- 172.21.15.40:0/2773124074 shutdown_connections 2024-04-26T23:01:40.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.025+0000 7fdb26000640 1 --2- 172.21.15.40:0/2773124074 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 0x7fdb200b4fe0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.025+0000 7fdb26000640 1 --2- 172.21.15.40:0/2773124074 >> v2:172.21.15.103:3300/0 conn(0x7fdb200d3b80 0x7fdb200b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.026+0000 7fdb26000640 1 -- 172.21.15.40:0/2773124074 >> 172.21.15.40:0/2773124074 conn(0x7fdb200d4f90 msgr2=0x7fdb200d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:40.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.026+0000 7fdb26000640 1 -- 172.21.15.40:0/2773124074 shutdown_connections 2024-04-26T23:01:40.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.026+0000 7fdb26000640 1 -- 172.21.15.40:0/2773124074 wait complete. 2024-04-26T23:01:40.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.029+0000 7f3c48800640 1 Processor -- start 2024-04-26T23:01:40.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.029+0000 7fdb26000640 1 Processor -- start 2024-04-26T23:01:40.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.030+0000 7fdb26000640 1 -- start start 2024-04-26T23:01:40.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.029+0000 7f3c48800640 1 -- start start 2024-04-26T23:01:40.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.031+0000 7f3c48800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c4006f600 0x7f3c400722f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.031+0000 7f3c48800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3c400aff60 0x7f3c4006ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.031+0000 7f3c48800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3c4006e330 0x7f3c4006e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.031+0000 7f3c48800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f3c40071e60 con 0x7f3c4006e330 2024-04-26T23:01:40.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.031+0000 7f3c48800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3c40073ce0 con 0x7f3c400aff60 2024-04-26T23:01:40.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.031+0000 7f3c48800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c400d06a0 con 0x7f3c4006f600 2024-04-26T23:01:40.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.032+0000 7f3c45400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3c400aff60 0x7f3c4006ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.032+0000 7f3c45400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3c400aff60 0x7f3c4006ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50040/0 (socket says 172.21.15.40:50040) 2024-04-26T23:01:40.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.032+0000 7f3c45400640 1 -- 172.21.15.40:0/4037826250 learned_addr learned my addr 172.21.15.40:0/4037826250 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:40.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.032+0000 7fdb26000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 0x7fdb2014fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.033+0000 7f3c46800640 1 --2- 172.21.15.40:0/4037826250 >> v2:172.21.15.40:3300/0 conn(0x7f3c4006e330 0x7f3c4006e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.033+0000 7fdb26000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb200d3b80 0x7fdb200b8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.033+0000 7fdb26000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2015f540 0x7fdb200b94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.034+0000 7fdb1f400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 0x7fdb2014fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.034+0000 7fdb1f400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 0x7fdb2014fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:47106/0 (socket says 172.21.15.40:47106) 2024-04-26T23:01:40.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.034+0000 7fdb1ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb200d3b80 0x7fdb200b8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.034+0000 7fdb1ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb200d3b80 0x7fdb200b8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50052/0 (socket says 172.21.15.40:50052) 2024-04-26T23:01:40.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.035+0000 7fdb1f400640 1 -- 172.21.15.40:0/2431672857 learned_addr learned my addr 172.21.15.40:0/2431672857 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:40.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.035+0000 7fdb26000640 1 -- 172.21.15.40:0/2431672857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdb20068e50 con 0x7fdb2015f540 2024-04-26T23:01:40.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.035+0000 7fdb26000640 1 -- 172.21.15.40:0/2431672857 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb200d0bc0 con 0x7fdb200d3b80 2024-04-26T23:01:40.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.035+0000 7fdb26000640 1 -- 172.21.15.40:0/2431672857 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdb200d2780 con 0x7fdb200b3ff0 2024-04-26T23:01:40.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.035+0000 7fdb1fe00640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2015f540 0x7fdb200b94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.033+0000 7f3c45400640 1 -- 172.21.15.40:0/4037826250 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c4006f600 msgr2=0x7f3c400722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:40.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.033+0000 7f3c45400640 1 --2- 172.21.15.40:0/4037826250 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c4006f600 0x7f3c400722f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.033+0000 7f3c45400640 1 -- 172.21.15.40:0/4037826250 >> v2:172.21.15.40:3300/0 conn(0x7f3c4006e330 msgr2=0x7f3c4006e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:40.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.033+0000 7f3c45400640 1 --2- 172.21.15.40:0/4037826250 >> v2:172.21.15.40:3300/0 conn(0x7f3c4006e330 0x7f3c4006e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.033+0000 7f3c45400640 1 -- 172.21.15.40:0/4037826250 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c40073f80 con 0x7f3c400aff60 2024-04-26T23:01:40.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.036+0000 7f3c45400640 1 --2- 172.21.15.40:0/4037826250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c400aff60 0x7f3c4006ddf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c400ce720 tx=0x7f3c3002a160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce000db5930e0cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:40.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.037+0000 7fdb1fe00640 1 -- 172.21.15.40:0/2431672857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 msgr2=0x7fdb2014fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.037+0000 7fdb1fe00640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 0x7fdb2014fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.037+0000 7fdb1fe00640 1 -- 172.21.15.40:0/2431672857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb200d3b80 msgr2=0x7fdb200b8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.037+0000 7fdb1fe00640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb200d3b80 0x7fdb200b8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.037+0000 7fdb1f400640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 0x7fdb2014fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:01:40.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.037+0000 7fdb1fe00640 1 -- 172.21.15.40:0/2431672857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb201502a0 con 0x7fdb2015f540 2024-04-26T23:01:40.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.038+0000 7f3c44a00640 1 -- 172.21.15.40:0/4037826250 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c30037070 con 0x7f3c400aff60 2024-04-26T23:01:40.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.038+0000 7f3c44a00640 1 -- 172.21.15.40:0/4037826250 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c30032440 con 0x7f3c400aff60 2024-04-26T23:01:40.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.038+0000 7f3c44a00640 1 -- 172.21.15.40:0/4037826250 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c30032850 con 0x7f3c400aff60 2024-04-26T23:01:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.038+0000 7fdb1fe00640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2015f540 0x7fdb200b94d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdb18030000 tx=0x7fdb18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.039+0000 7f3c48800640 1 -- 172.21.15.40:0/4037826250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c400aff60 msgr2=0x7f3c4006ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.039+0000 7fdb17400640 1 -- 172.21.15.40:0/2431672857 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18036050 con 0x7fdb2015f540 2024-04-26T23:01:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.039+0000 7fdb17400640 1 -- 172.21.15.40:0/2431672857 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb18042070 con 0x7fdb2015f540 2024-04-26T23:01:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.039+0000 7fdb17400640 1 -- 172.21.15.40:0/2431672857 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18009510 con 0x7fdb2015f540 2024-04-26T23:01:40.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.039+0000 7f3c48800640 1 --2- 172.21.15.40:0/4037826250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c400aff60 0x7f3c4006ddf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c400ce720 tx=0x7f3c3002a160 comp rx=0 tx=0).stop 2024-04-26T23:01:40.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.039+0000 7f3c48800640 1 -- 172.21.15.40:0/4037826250 shutdown_connections 2024-04-26T23:01:40.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.039+0000 7f3c48800640 1 --2- 172.21.15.40:0/4037826250 >> v2:172.21.15.40:3300/0 conn(0x7f3c4006e330 0x7f3c4006e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.039+0000 7f3c48800640 1 --2- 172.21.15.40:0/4037826250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c400aff60 0x7f3c4006ddf0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.039+0000 7fdb26000640 1 -- 172.21.15.40:0/2431672857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb20150730 con 0x7fdb2015f540 2024-04-26T23:01:40.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.039+0000 7f3c48800640 1 --2- 172.21.15.40:0/4037826250 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c4006f600 0x7f3c400722f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.039+0000 7f3c48800640 1 -- 172.21.15.40:0/4037826250 >> 172.21.15.40:0/4037826250 conn(0x7f3c400d0be0 msgr2=0x7f3c400d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:40.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.039+0000 7fdb26000640 1 -- 172.21.15.40:0/2431672857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb20150cf0 con 0x7fdb2015f540 2024-04-26T23:01:40.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.041+0000 7fdb26000640 1 -- 172.21.15.40:0/2431672857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb200694f0 con 0x7fdb2015f540 2024-04-26T23:01:40.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.042+0000 7f3c48800640 1 -- 172.21.15.40:0/4037826250 shutdown_connections 2024-04-26T23:01:40.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.042+0000 7f3c48800640 1 -- 172.21.15.40:0/4037826250 wait complete. 2024-04-26T23:01:40.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.045+0000 7f3c48800640 1 Processor -- start 2024-04-26T23:01:40.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.047+0000 7f3c48800640 1 -- start start 2024-04-26T23:01:40.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.049+0000 7fdb17400640 1 -- 172.21.15.40:0/2431672857 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb18007b70 con 0x7fdb2015f540 2024-04-26T23:01:40.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.049+0000 7f3c48800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c4006e330 0x7f3c40158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.049+0000 7f3c48800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c4006f600 0x7f3c401f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.049+0000 7f3c48800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c400aff60 0x7f3c401f64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.049+0000 7f3c48800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c400d1700 con 0x7f3c4006e330 2024-04-26T23:01:40.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.049+0000 7f3c48800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4006f9e0 con 0x7f3c400aff60 2024-04-26T23:01:40.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.049+0000 7f3c48800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c400d1880 con 0x7f3c4006f600 2024-04-26T23:01:40.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.051+0000 7f3c45e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c4006e330 0x7f3c40158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.052+0000 7f3c45400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c4006f600 0x7f3c401f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.052+0000 7f3c46800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c400aff60 0x7f3c401f64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.052+0000 7f3c46800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c400aff60 0x7f3c401f64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50064/0 (socket says 172.21.15.40:50064) 2024-04-26T23:01:40.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.052+0000 7f3c46800640 1 -- 172.21.15.40:0/3240826907 learned_addr learned my addr 172.21.15.40:0/3240826907 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:40.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.053+0000 7f3c45e00640 1 -- 172.21.15.40:0/3240826907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c4006f600 msgr2=0x7f3c401f5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.053+0000 7fdb17400640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb0006c9c0 0x7fdb0006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.054+0000 7fdb17400640 1 -- 172.21.15.40:0/2431672857 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdb180ad660 con 0x7fdb2015f540 2024-04-26T23:01:40.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.053+0000 7f3c45e00640 1 --2- 172.21.15.40:0/3240826907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c4006f600 0x7f3c401f5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.053+0000 7f3c45e00640 1 -- 172.21.15.40:0/3240826907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c400aff60 msgr2=0x7f3c401f64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.055+0000 7fdb17400640 1 -- 172.21.15.40:0/2431672857 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fdb1807b050 con 0x7fdb2015f540 2024-04-26T23:01:40.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.053+0000 7f3c45e00640 1 --2- 172.21.15.40:0/3240826907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c400aff60 0x7f3c401f64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.053+0000 7f3c45e00640 1 -- 172.21.15.40:0/3240826907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c400cb240 con 0x7f3c4006e330 2024-04-26T23:01:40.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.053+0000 7f3c45e00640 1 --2- 172.21.15.40:0/3240826907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c4006e330 0x7f3c40158b10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400b410 tx=0x7f3c3400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:40.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.055+0000 7f3c3a000640 1 -- 172.21.15.40:0/3240826907 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34015020 con 0x7f3c4006e330 2024-04-26T23:01:40.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.055+0000 7f3c48800640 1 -- 172.21.15.40:0/3240826907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c400cb4b0 con 0x7f3c4006e330 2024-04-26T23:01:40.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.056+0000 7f3c48800640 1 -- 172.21.15.40:0/3240826907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c401f6d80 con 0x7f3c4006e330 2024-04-26T23:01:40.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.056+0000 7f3c3a000640 1 -- 172.21.15.40:0/3240826907 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c34011850 con 0x7f3c4006e330 2024-04-26T23:01:40.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.056+0000 7f3c3a000640 1 -- 172.21.15.40:0/3240826907 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34011c40 con 0x7f3c4006e330 2024-04-26T23:01:40.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.059+0000 7f3c48800640 1 -- 172.21.15.40:0/3240826907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c1c005290 con 0x7f3c4006e330 2024-04-26T23:01:40.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.062+0000 7fdb1f400640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb0006c9c0 0x7fdb0006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.063+0000 7fdb1f400640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb0006c9c0 0x7fdb0006ee80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009940 tx=0x7fdb10008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:40.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.061+0000 7f3c3a000640 1 -- 172.21.15.40:0/3240826907 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3c34018020 con 0x7f3c4006e330 2024-04-26T23:01:40.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.063+0000 7f3c3a000640 1 --2- 172.21.15.40:0/3240826907 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c2406c8f0 0x7f3c2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.063+0000 7f3c3a000640 1 -- 172.21.15.40:0/3240826907 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3c3408bc00 con 0x7f3c4006e330 2024-04-26T23:01:40.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.064+0000 7f3c45400640 1 --2- 172.21.15.40:0/3240826907 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c2406c8f0 0x7f3c2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.069+0000 7f3c45400640 1 --2- 172.21.15.40:0/3240826907 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c2406c8f0 0x7f3c2406edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3c30034000 tx=0x7f3c3002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:40.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.076+0000 7f3c3a000640 1 -- 172.21.15.40:0/3240826907 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3c3401f020 con 0x7f3c4006e330 2024-04-26T23:01:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.269+0000 7fe1d7a00640 1 -- 172.21.15.40:0/1380773247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fe1d00694f0 con 0x7fe1d01671e0 2024-04-26T23:01:40.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.270+0000 7fe1ccc00640 1 -- 172.21.15.40:0/1380773247 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe1c807af80 con 0x7fe1d01671e0 2024-04-26T23:01:40.271 INFO:teuthology.orchestra.run.smithi040.stdout:55834574859 2024-04-26T23:01:40.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.271+0000 7f8813800640 1 -- 172.21.15.40:0/2373855407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f880c0694f0 con 0x7f880c0d2090 2024-04-26T23:01:40.273 INFO:teuthology.orchestra.run.smithi040.stdout:55834574859 2024-04-26T23:01:40.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.273+0000 7f8808c00640 1 -- 172.21.15.40:0/2373855407 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f880405daf0 con 0x7f880c0d2090 2024-04-26T23:01:40.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.275+0000 7fe1d7a00640 1 -- 172.21.15.40:0/1380773247 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1b006cb70 msgr2=0x7fe1b006f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.275+0000 7fe1d7a00640 1 --2- 172.21.15.40:0/1380773247 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1b006cb70 0x7fe1b006f030 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc000f60 tx=0x7fe1bc000bc0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.275+0000 7fe1d7a00640 1 -- 172.21.15.40:0/1380773247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d01671e0 msgr2=0x7fe1d00c6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.275+0000 7fe1d7a00640 1 --2- 172.21.15.40:0/1380773247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d01671e0 0x7fe1d00c6790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009e80 tx=0x7fe1c8004a40 comp rx=0 tx=0).stop 2024-04-26T23:01:40.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.276+0000 7fe1d7a00640 1 -- 172.21.15.40:0/1380773247 shutdown_connections 2024-04-26T23:01:40.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.276+0000 7fe1d7a00640 1 --2- 172.21.15.40:0/1380773247 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1b006cb70 0x7fe1b006f030 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.276+0000 7fe1d7a00640 1 --2- 172.21.15.40:0/1380773247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1d01671e0 0x7fe1d00c6790 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.276+0000 7fe1d7a00640 1 --2- 172.21.15.40:0/1380773247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d00c41d0 0x7fe1d0163c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.276+0000 7fe1d7a00640 1 --2- 172.21.15.40:0/1380773247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1d0073f30 0x7fe1d015c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.276+0000 7fe1d7a00640 1 -- 172.21.15.40:0/1380773247 >> 172.21.15.40:0/1380773247 conn(0x7fe1d00d0be0 msgr2=0x7fe1d00b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:40.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.276+0000 7fe1d7a00640 1 -- 172.21.15.40:0/1380773247 shutdown_connections 2024-04-26T23:01:40.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.277+0000 7fe1d7a00640 1 -- 172.21.15.40:0/1380773247 wait complete. 2024-04-26T23:01:40.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.277+0000 7f8813800640 1 -- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f87e806c9c0 msgr2=0x7f87e806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.277+0000 7f8813800640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f87e806c9c0 0x7f87e806ee80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f880c162ee0 tx=0x7f87f4009040 comp rx=0 tx=0).stop 2024-04-26T23:01:40.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.277+0000 7f8813800640 1 -- 172.21.15.40:0/2373855407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f880c0d2090 msgr2=0x7f880c0b9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.277+0000 7f8813800640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f880c0d2090 0x7f880c0b9d30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f880400c7f0 tx=0x7f880400ccb0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.278+0000 7f8813800640 1 -- 172.21.15.40:0/2373855407 shutdown_connections 2024-04-26T23:01:40.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.278+0000 7f8813800640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f87e806c9c0 0x7f87e806ee80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.278+0000 7f8813800640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f880c0d2090 0x7f880c0b9d30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.278+0000 7f8813800640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f880c0b65d0 0x7f880c0b97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.278+0000 7f8813800640 1 --2- 172.21.15.40:0/2373855407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f880c0b61f0 0x7f880c0cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.278+0000 7f8813800640 1 -- 172.21.15.40:0/2373855407 >> 172.21.15.40:0/2373855407 conn(0x7f880c0d4f90 msgr2=0x7f880c06e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:40.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.278+0000 7f8813800640 1 -- 172.21.15.40:0/2373855407 shutdown_connections 2024-04-26T23:01:40.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.279+0000 7f8813800640 1 -- 172.21.15.40:0/2373855407 wait complete. 2024-04-26T23:01:40.304 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.3 2024-04-26T23:01:40.304 DEBUG:teuthology.parallel:result is None 2024-04-26T23:01:40.306 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.1 2024-04-26T23:01:40.306 DEBUG:teuthology.parallel:result is None 2024-04-26T23:01:40.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.328+0000 7fe7f9200640 1 -- 172.21.15.40:0/2229224975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fe7cc005850 con 0x7fe7f40b6220 2024-04-26T23:01:40.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.328+0000 7fe7eb400640 1 -- 172.21.15.40:0/2229224975 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe7e00587d0 con 0x7fe7f40b6220 2024-04-26T23:01:40.329 INFO:teuthology.orchestra.run.smithi040.stdout:55834574860 2024-04-26T23:01:40.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.331+0000 7fe7f9200640 1 -- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe7d406ccf0 msgr2=0x7fe7d406f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.331+0000 7fe7f9200640 1 --2- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe7d406ccf0 0x7fe7d406f1b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4034000 tx=0x7fe7e402e000 comp rx=0 tx=0).stop 2024-04-26T23:01:40.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.331+0000 7fe7f9200640 1 -- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7f40b6220 msgr2=0x7fe7f40cb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.331+0000 7fe7f9200640 1 --2- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7f40b6220 0x7fe7f40cb6a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe7e000c450 tx=0x7fe7e000c910 comp rx=0 tx=0).stop 2024-04-26T23:01:40.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.331+0000 7fe7f9200640 1 -- 172.21.15.40:0/2229224975 shutdown_connections 2024-04-26T23:01:40.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.331+0000 7fe7f9200640 1 --2- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe7d406ccf0 0x7fe7d406f1b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.331+0000 7fe7f9200640 1 --2- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7f40d32e0 0x7fe7f41fcb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.331+0000 7fe7f9200640 1 --2- 172.21.15.40:0/2229224975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f40b8d20 0x7fe7f41f8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.331+0000 7fe7f9200640 1 --2- 172.21.15.40:0/2229224975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7f40b6220 0x7fe7f40cb6a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.331+0000 7fe7f9200640 1 -- 172.21.15.40:0/2229224975 >> 172.21.15.40:0/2229224975 conn(0x7fe7f40d4d90 msgr2=0x7fe7f406e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:40.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.331+0000 7fe7f9200640 1 -- 172.21.15.40:0/2229224975 shutdown_connections 2024-04-26T23:01:40.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.331+0000 7fe7f9200640 1 -- 172.21.15.40:0/2229224975 wait complete. 2024-04-26T23:01:40.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.334+0000 7fea4fe00640 1 -- 172.21.15.40:0/3447025234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fea28005850 con 0x7fea500b3290 2024-04-26T23:01:40.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.334+0000 7fea46a00640 1 -- 172.21.15.40:0/3447025234 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fea40062890 con 0x7fea500b3290 2024-04-26T23:01:40.334 INFO:teuthology.orchestra.run.smithi040.stdout:55834574861 2024-04-26T23:01:40.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.337+0000 7fea4fe00640 1 -- 172.21.15.40:0/3447025234 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea3006c8f0 msgr2=0x7fea3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.337+0000 7fea4fe00640 1 --2- 172.21.15.40:0/3447025234 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea3006c8f0 0x7fea3006edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009ad0 tx=0x7fea3c008040 comp rx=0 tx=0).stop 2024-04-26T23:01:40.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.337+0000 7fea4fe00640 1 -- 172.21.15.40:0/3447025234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea500b3290 msgr2=0x7fea5015a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.337+0000 7fea4fe00640 1 --2- 172.21.15.40:0/3447025234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea500b3290 0x7fea5015a810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fea40001be0 tx=0x7fea40014420 comp rx=0 tx=0).stop 2024-04-26T23:01:40.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.338+0000 7fea4fe00640 1 -- 172.21.15.40:0/3447025234 shutdown_connections 2024-04-26T23:01:40.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.338+0000 7fea4fe00640 1 --2- 172.21.15.40:0/3447025234 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea3006c8f0 0x7fea3006edb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.338+0000 7fea4fe00640 1 --2- 172.21.15.40:0/3447025234 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea5015b4d0 0x7fea501544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.338+0000 7fea4fe00640 1 --2- 172.21.15.40:0/3447025234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea500b3290 0x7fea5015a810 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.338+0000 7fea4fe00640 1 --2- 172.21.15.40:0/3447025234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea500727a0 0x7fea501434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.338+0000 7fea4fe00640 1 -- 172.21.15.40:0/3447025234 >> 172.21.15.40:0/3447025234 conn(0x7fea500b2770 msgr2=0x7fea5006e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.338+0000 7fea4fe00640 1 -- 172.21.15.40:0/3447025234 shutdown_connections 2024-04-26T23:01:40.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.338+0000 7fea4fe00640 1 -- 172.21.15.40:0/3447025234 wait complete. 2024-04-26T23:01:40.359 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574860 for osd.0 2024-04-26T23:01:40.359 DEBUG:teuthology.parallel:result is None 2024-04-26T23:01:40.362 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574861 for osd.2 2024-04-26T23:01:40.362 DEBUG:teuthology.parallel:result is None 2024-04-26T23:01:40.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.387+0000 7f551cc00640 1 -- 172.21.15.40:0/1970361614 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f55180d0420 con 0x7f551815f540 2024-04-26T23:01:40.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.388+0000 7f550ea00640 1 -- 172.21.15.40:0/1970361614 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f551005cc60 con 0x7f551815f540 2024-04-26T23:01:40.388 INFO:teuthology.orchestra.run.smithi040.stdout:55834574859 2024-04-26T23:01:40.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.390+0000 7f551cc00640 1 -- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f54f406caf0 msgr2=0x7f54f406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.390+0000 7f551cc00640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f54f406caf0 0x7f54f406efb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5500009560 tx=0x7f550002e000 comp rx=0 tx=0).stop 2024-04-26T23:01:40.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.390+0000 7f551cc00640 1 -- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f551815f540 msgr2=0x7f55180b9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.390+0000 7f551cc00640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f551815f540 0x7f55180b9140 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f551000c780 tx=0x7f551000cc40 comp rx=0 tx=0).stop 2024-04-26T23:01:40.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.390+0000 7f551cc00640 1 -- 172.21.15.40:0/1970361614 shutdown_connections 2024-04-26T23:01:40.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.390+0000 7f551cc00640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f54f406caf0 0x7f54f406efb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.390+0000 7f551cc00640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f551815f540 0x7f55180b9140 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.390+0000 7f551cc00640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55180b5ab0 0x7f55180b8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.390+0000 7f551cc00640 1 --2- 172.21.15.40:0/1970361614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55180b3ff0 0x7f551815b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.390+0000 7f551cc00640 1 -- 172.21.15.40:0/1970361614 >> 172.21.15.40:0/1970361614 conn(0x7f55180d4f90 msgr2=0x7f55180b6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:40.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.390+0000 7f551cc00640 1 -- 172.21.15.40:0/1970361614 shutdown_connections 2024-04-26T23:01:40.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.390+0000 7f551cc00640 1 -- 172.21.15.40:0/1970361614 wait complete. 2024-04-26T23:01:40.407 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.7 2024-04-26T23:01:40.407 DEBUG:teuthology.parallel:result is None 2024-04-26T23:01:40.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.424+0000 7fce22800640 1 -- 172.21.15.40:0/2427236251 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fce14051160 con 0x7fce1400ab30 2024-04-26T23:01:40.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.425+0000 7fce0f400640 1 -- 172.21.15.40:0/2427236251 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fce10059960 con 0x7fce1400ab30 2024-04-26T23:01:40.425 INFO:teuthology.orchestra.run.smithi040.stdout:55834574859 2024-04-26T23:01:40.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.426+0000 7f3c48800640 1 -- 172.21.15.40:0/3240826907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f3c1c005580 con 0x7f3c4006e330 2024-04-26T23:01:40.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.426+0000 7f3c3a000640 1 -- 172.21.15.40:0/3240826907 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3c34059600 con 0x7f3c4006e330 2024-04-26T23:01:40.426 INFO:teuthology.orchestra.run.smithi040.stdout:55834574859 2024-04-26T23:01:40.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7fce22800640 1 -- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdfc06c8f0 msgr2=0x7fcdfc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7fce22800640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdfc06c8f0 0x7fcdfc06edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce0802a220 tx=0x7fce0802e000 comp rx=0 tx=0).stop 2024-04-26T23:01:40.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7fce22800640 1 -- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1400ab30 msgr2=0x7fce1402d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7fce22800640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1400ab30 0x7fce1402d830 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fce1400af50 tx=0x7fce1000cbe0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7fce22800640 1 -- 172.21.15.40:0/2427236251 shutdown_connections 2024-04-26T23:01:40.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7fce22800640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdfc06c8f0 0x7fcdfc06edb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7fce22800640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce140d6150 0x7fce140377a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7fce22800640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce1400ab30 0x7fce1402d830 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7fce22800640 1 --2- 172.21.15.40:0/2427236251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce1400a450 0x7fce14050670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7fce22800640 1 -- 172.21.15.40:0/2427236251 >> 172.21.15.40:0/2427236251 conn(0x7fce14056cc0 msgr2=0x7fce1403ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:40.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7fce22800640 1 -- 172.21.15.40:0/2427236251 shutdown_connections 2024-04-26T23:01:40.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7fce22800640 1 -- 172.21.15.40:0/2427236251 wait complete. 2024-04-26T23:01:40.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7f3c48800640 1 -- 172.21.15.40:0/3240826907 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c2406c8f0 msgr2=0x7f3c2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7f3c48800640 1 --2- 172.21.15.40:0/3240826907 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c2406c8f0 0x7f3c2406edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3c30034000 tx=0x7f3c3002e000 comp rx=0 tx=0).stop 2024-04-26T23:01:40.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7f3c48800640 1 -- 172.21.15.40:0/3240826907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c4006e330 msgr2=0x7f3c40158b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7f3c48800640 1 --2- 172.21.15.40:0/3240826907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c4006e330 0x7f3c40158b10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400b410 tx=0x7f3c3400b8d0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7f3c48800640 1 -- 172.21.15.40:0/3240826907 shutdown_connections 2024-04-26T23:01:40.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7f3c48800640 1 --2- 172.21.15.40:0/3240826907 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c2406c8f0 0x7f3c2406edb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7f3c48800640 1 --2- 172.21.15.40:0/3240826907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c400aff60 0x7f3c401f64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7f3c48800640 1 --2- 172.21.15.40:0/3240826907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c4006f600 0x7f3c401f5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7f3c48800640 1 --2- 172.21.15.40:0/3240826907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c4006e330 0x7f3c40158b10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7f3c48800640 1 -- 172.21.15.40:0/3240826907 >> 172.21.15.40:0/3240826907 conn(0x7f3c400d0be0 msgr2=0x7f3c4015d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:40.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7f3c48800640 1 -- 172.21.15.40:0/3240826907 shutdown_connections 2024-04-26T23:01:40.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.428+0000 7f3c48800640 1 -- 172.21.15.40:0/3240826907 wait complete. 2024-04-26T23:01:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.442+0000 7fdb26000640 1 -- 172.21.15.40:0/2431672857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fdb200b8320 con 0x7fdb2015f540 2024-04-26T23:01:40.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.443+0000 7fdb17400640 1 -- 172.21.15.40:0/2431672857 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdb1807f350 con 0x7fdb2015f540 2024-04-26T23:01:40.443 INFO:teuthology.orchestra.run.smithi040.stdout:55834574859 2024-04-26T23:01:40.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.5 2024-04-26T23:01:40.445 DEBUG:teuthology.parallel:result is None 2024-04-26T23:01:40.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.446+0000 7fdb26000640 1 -- 172.21.15.40:0/2431672857 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb0006c9c0 msgr2=0x7fdb0006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.446+0000 7fdb26000640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb0006c9c0 0x7fdb0006ee80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009940 tx=0x7fdb10008040 comp rx=0 tx=0).stop 2024-04-26T23:01:40.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.446+0000 7fdb26000640 1 -- 172.21.15.40:0/2431672857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2015f540 msgr2=0x7fdb200b94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.446+0000 7fdb26000640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2015f540 0x7fdb200b94d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdb18030000 tx=0x7fdb18004a40 comp rx=0 tx=0).stop 2024-04-26T23:01:40.447 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.6 2024-04-26T23:01:40.447 DEBUG:teuthology.parallel:result is None 2024-04-26T23:01:40.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.446+0000 7fdb26000640 1 -- 172.21.15.40:0/2431672857 shutdown_connections 2024-04-26T23:01:40.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.446+0000 7fdb26000640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb0006c9c0 0x7fdb0006ee80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.446+0000 7fdb26000640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2015f540 0x7fdb200b94d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.446+0000 7fdb26000640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb200d3b80 0x7fdb200b8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.446+0000 7fdb26000640 1 --2- 172.21.15.40:0/2431672857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb200b3ff0 0x7fdb2014fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.446+0000 7fdb26000640 1 -- 172.21.15.40:0/2431672857 >> 172.21.15.40:0/2431672857 conn(0x7fdb200d4f90 msgr2=0x7fdb200d1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:40.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.446+0000 7fdb26000640 1 -- 172.21.15.40:0/2431672857 shutdown_connections 2024-04-26T23:01:40.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.446+0000 7fdb26000640 1 -- 172.21.15.40:0/2431672857 wait complete. 2024-04-26T23:01:40.462 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.4 2024-04-26T23:01:40.462 DEBUG:teuthology.parallel:result is None 2024-04-26T23:01:40.462 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T23:01:40.463 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:01:40.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.703+0000 7f42d3e00640 1 Processor -- start 2024-04-26T23:01:40.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.704+0000 7f42d3e00640 1 -- start start 2024-04-26T23:01:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.705+0000 7f42d3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f42cc071e60 0x7f42cc06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.705+0000 7f42d3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42cc06fe20 0x7f42cc072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.705+0000 7f42d3e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42cc15d7c0 con 0x7f42cc072780 2024-04-26T23:01:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.705+0000 7f42d3e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f42cc15daa0 con 0x7f42cc071e60 2024-04-26T23:01:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.705+0000 7f42d3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f42cc15dd80 con 0x7f42cc06fe20 2024-04-26T23:01:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.706+0000 7f42d1400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f42cc071e60 0x7f42cc06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.706+0000 7f42d1e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f42cc072780 0x7f42cc0cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41596/0 (socket says 172.21.15.40:41596) 2024-04-26T23:01:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.706+0000 7f42d1e00640 1 -- 172.21.15.40:0/591835284 learned_addr learned my addr 172.21.15.40:0/591835284 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.706+0000 7f42d0a00640 1 --2- 172.21.15.40:0/591835284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42cc06fe20 0x7f42cc072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.706+0000 7f42d1400640 1 -- 172.21.15.40:0/591835284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42cc06fe20 msgr2=0x7f42cc072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.706+0000 7f42d1400640 1 --2- 172.21.15.40:0/591835284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42cc06fe20 0x7f42cc072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.706+0000 7f42d1400640 1 -- 172.21.15.40:0/591835284 >> v1:172.21.15.40:6789/0 conn(0x7f42cc072780 legacy=0x7f42cc0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.706+0000 7f42d1400640 1 -- 172.21.15.40:0/591835284 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42cc15e060 con 0x7f42cc071e60 2024-04-26T23:01:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.706+0000 7f42cbe00640 1 -- 172.21.15.40:0/591835284 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1337524703 0 0) 0x7f42cc15d7c0 con 0x7f42cc072780 2024-04-26T23:01:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.707+0000 7f42d1400640 1 --2- 172.21.15.40:0/591835284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc071e60 0x7f42cc06f8e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f42c00097a0 tx=0x7f42c002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe6a4422eb997238 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.708+0000 7f42cbe00640 1 -- 172.21.15.40:0/591835284 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c0037070 con 0x7f42cc071e60 2024-04-26T23:01:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.708+0000 7f42cbe00640 1 -- 172.21.15.40:0/591835284 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42c0032440 con 0x7f42cc071e60 2024-04-26T23:01:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.708+0000 7f42cbe00640 1 -- 172.21.15.40:0/591835284 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c0032810 con 0x7f42cc071e60 2024-04-26T23:01:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.708+0000 7f42d3e00640 1 -- 172.21.15.40:0/591835284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc071e60 msgr2=0x7f42cc06f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.708+0000 7f42d3e00640 1 --2- 172.21.15.40:0/591835284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc071e60 0x7f42cc06f8e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f42c00097a0 tx=0x7f42c002a180 comp rx=0 tx=0).stop 2024-04-26T23:01:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.708+0000 7f42d3e00640 1 -- 172.21.15.40:0/591835284 shutdown_connections 2024-04-26T23:01:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.708+0000 7f42d3e00640 1 --2- 172.21.15.40:0/591835284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42cc06fe20 0x7f42cc072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.708+0000 7f42d3e00640 1 --2- 172.21.15.40:0/591835284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc071e60 0x7f42cc06f8e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.708+0000 7f42d3e00640 1 -- 172.21.15.40:0/591835284 >> 172.21.15.40:0/591835284 conn(0x7f42cc0d0be0 msgr2=0x7f42cc0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.708+0000 7f42d3e00640 1 -- 172.21.15.40:0/591835284 shutdown_connections 2024-04-26T23:01:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.708+0000 7f42d3e00640 1 -- 172.21.15.40:0/591835284 wait complete. 2024-04-26T23:01:40.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.710+0000 7f42d3e00640 1 Processor -- start 2024-04-26T23:01:40.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.710+0000 7f42d3e00640 1 -- start start 2024-04-26T23:01:40.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.711+0000 7f42d3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42cc06fe20 0x7f42cc14b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.711+0000 7f42d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc071e60 0x7f42cc0b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.711+0000 7f42d3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42cc072780 0x7f42cc0c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.711+0000 7f42d3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f42cc15d3f0 con 0x7f42cc071e60 2024-04-26T23:01:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.711+0000 7f42d3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f42cc072c10 con 0x7f42cc06fe20 2024-04-26T23:01:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.711+0000 7f42d3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f42cc15d570 con 0x7f42cc072780 2024-04-26T23:01:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.711+0000 7f42d0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc071e60 0x7f42cc0b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.711+0000 7f42d0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc071e60 0x7f42cc0b0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39028/0 (socket says 172.21.15.40:39028) 2024-04-26T23:01:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.711+0000 7f42d0a00640 1 -- 172.21.15.40:0/3567882780 learned_addr learned my addr 172.21.15.40:0/3567882780 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.712+0000 7f42d1e00640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42cc072780 0x7f42cc0c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.712+0000 7f42d1400640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42cc06fe20 0x7f42cc14b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.712+0000 7f42d0a00640 1 -- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42cc072780 msgr2=0x7f42cc0c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.712+0000 7f42d0a00640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42cc072780 0x7f42cc0c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.712+0000 7f42d0a00640 1 -- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42cc06fe20 msgr2=0x7f42cc14b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.712+0000 7f42d0a00640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42cc06fe20 0x7f42cc14b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.712+0000 7f42d0a00640 1 -- 172.21.15.40:0/3567882780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42cc0c9240 con 0x7f42cc071e60 2024-04-26T23:01:40.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.712+0000 7f42d1400640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42cc06fe20 0x7f42cc14b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:40.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.712+0000 7f42d0a00640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc071e60 0x7f42cc0b0070 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00d830 tx=0x7f42bc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:40.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.712+0000 7f42c9600640 1 -- 172.21.15.40:0/3567882780 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc00b6c0 con 0x7f42cc071e60 2024-04-26T23:01:40.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.712+0000 7f42c9600640 1 -- 172.21.15.40:0/3567882780 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42bc019070 con 0x7f42cc071e60 2024-04-26T23:01:40.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.712+0000 7f42c9600640 1 -- 172.21.15.40:0/3567882780 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc00ba80 con 0x7f42cc071e60 2024-04-26T23:01:40.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.712+0000 7f42d3e00640 1 -- 172.21.15.40:0/3567882780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42cc0c94e0 con 0x7f42cc071e60 2024-04-26T23:01:40.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.713+0000 7f42d3e00640 1 -- 172.21.15.40:0/3567882780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42cc0c9aa0 con 0x7f42cc071e60 2024-04-26T23:01:40.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.714+0000 7f42d3e00640 1 -- 172.21.15.40:0/3567882780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42cc0694f0 con 0x7f42cc071e60 2024-04-26T23:01:40.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.714+0000 7f42c9600640 1 -- 172.21.15.40:0/3567882780 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f42bc01e460 con 0x7f42cc071e60 2024-04-26T23:01:40.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.716+0000 7f42c9600640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f42b006c8f0 0x7f42b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:40.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.716+0000 7f42c9600640 1 -- 172.21.15.40:0/3567882780 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f42bc08c4d0 con 0x7f42cc071e60 2024-04-26T23:01:40.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.716+0000 7f42d1400640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f42b006c8f0 0x7f42b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:40.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.717+0000 7f42d1400640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f42b006c8f0 0x7f42b006edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f42c0034000 tx=0x7f42c002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:40.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.719+0000 7f42c9600640 1 -- 172.21.15.40:0/3567882780 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f42bc05a650 con 0x7f42cc071e60 2024-04-26T23:01:40.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.912+0000 7f42d3e00640 1 -- 172.21.15.40:0/3567882780 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f42cc06ce60 con 0x7f42b006c8f0 2024-04-26T23:01:40.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.938+0000 7f42c9600640 1 -- 172.21.15.40:0/3567882780 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332537 (secure 0 0 0) 0x7f42cc06ce60 con 0x7f42b006c8f0 2024-04-26T23:01:40.945 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:40.946 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:01:40.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.946+0000 7f42d3e00640 1 -- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f42b006c8f0 msgr2=0x7f42b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.946+0000 7f42d3e00640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f42b006c8f0 0x7f42b006edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f42c0034000 tx=0x7f42c002e000 comp rx=0 tx=0).stop 2024-04-26T23:01:40.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.946+0000 7f42d3e00640 1 -- 172.21.15.40:0/3567882780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc071e60 msgr2=0x7f42cc0b0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:40.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.946+0000 7f42d3e00640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc071e60 0x7f42cc0b0070 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00d830 tx=0x7f42bc00dcf0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.947+0000 7f42d3e00640 1 -- 172.21.15.40:0/3567882780 shutdown_connections 2024-04-26T23:01:40.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.947+0000 7f42d3e00640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f42b006c8f0 0x7f42b006edb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.947+0000 7f42d3e00640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f42cc072780 0x7f42cc0c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.947+0000 7f42d3e00640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc071e60 0x7f42cc0b0070 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.947+0000 7f42d3e00640 1 --2- 172.21.15.40:0/3567882780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42cc06fe20 0x7f42cc14b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:40.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.947+0000 7f42d3e00640 1 -- 172.21.15.40:0/3567882780 >> 172.21.15.40:0/3567882780 conn(0x7f42cc0d0be0 msgr2=0x7f42cc0cb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:40.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.947+0000 7f42d3e00640 1 -- 172.21.15.40:0/3567882780 shutdown_connections 2024-04-26T23:01:40.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:40.947+0000 7f42d3e00640 1 -- 172.21.15.40:0/3567882780 wait complete. 2024-04-26T23:01:40.964 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-04-26T23:01:39.153714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004868"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487297+0000","last_change":"2024-04-26T23:01:04.726404+0000","last_active":"2024-04-26T23:01:12.487297+0000","last_peered":"2024-04-26T23:01:12.487297+0000","last_clean":"2024-04-26T23:01:12.487297+0000","last_became_active":"2024-04-26T23:01:04.726160+0000","last_became_peered":"2024-04-26T23:01:04.726160+0000","last_unstale":"2024-04-26T23:01:12.487297+0000","last_undegraded":"2024-04-26T23:01:12.487297+0000","last_fullsized":"2024-04-26T23:01:12.487297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:21:22.581263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771048+0000","last_change":"2024-04-26T23:01:05.746741+0000","last_active":"2024-04-26T23:01:11.771048+0000","last_peered":"2024-04-26T23:01:11.771048+0000","last_clean":"2024-04-26T23:01:11.771048+0000","last_became_active":"2024-04-26T23:01:05.743495+0000","last_became_peered":"2024-04-26T23:01:05.743495+0000","last_unstale":"2024-04-26T23:01:11.771048+0000","last_undegraded":"2024-04-26T23:01:11.771048+0000","last_fullsized":"2024-04-26T23:01:11.771048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769795+0000","last_change":"2024-04-26T23:01:10.403533+0000","last_active":"2024-04-26T23:01:11.769795+0000","last_peered":"2024-04-26T23:01:11.769795+0000","last_clean":"2024-04-26T23:01:11.769795+0000","last_became_active":"2024-04-26T23:01:10.402972+0000","last_became_peered":"2024-04-26T23:01:10.402972+0000","last_unstale":"2024-04-26T23:01:11.769795+0000","last_undegraded":"2024-04-26T23:01:11.769795+0000","last_fullsized":"2024-04-26T23:01:11.769795+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:17.916674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771280+0000","last_change":"2024-04-26T23:01:04.715636+0000","last_active":"2024-04-26T23:01:11.771280+0000","last_peered":"2024-04-26T23:01:11.771280+0000","last_clean":"2024-04-26T23:01:11.771280+0000","last_became_active":"2024-04-26T23:01:04.715322+0000","last_became_peered":"2024-04-26T23:01:04.715322+0000","last_unstale":"2024-04-26T23:01:11.771280+0000","last_undegraded":"2024-04-26T23:01:11.771280+0000","last_fullsized":"2024-04-26T23:01:11.771280+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:55.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486825+0000","last_change":"2024-04-26T23:01:05.723149+0000","last_active":"2024-04-26T23:01:12.486825+0000","last_peered":"2024-04-26T23:01:12.486825+0000","last_clean":"2024-04-26T23:01:12.486825+0000","last_became_active":"2024-04-26T23:01:05.722809+0000","last_became_peered":"2024-04-26T23:01:05.722809+0000","last_unstale":"2024-04-26T23:01:12.486825+0000","last_undegraded":"2024-04-26T23:01:12.486825+0000","last_fullsized":"2024-04-26T23:01:12.486825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:26.157667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.881757+0000","last_change":"2024-04-26T23:01:09.805189+0000","last_active":"2024-04-26T23:01:11.881757+0000","last_peered":"2024-04-26T23:01:11.881757+0000","last_clean":"2024-04-26T23:01:11.881757+0000","last_became_active":"2024-04-26T23:01:09.804560+0000","last_became_peered":"2024-04-26T23:01:09.804560+0000","last_unstale":"2024-04-26T23:01:11.881757+0000","last_undegraded":"2024-04-26T23:01:11.881757+0000","last_fullsized":"2024-04-26T23:01:11.881757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:47.817790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775805+0000","last_change":"2024-04-26T23:01:04.734037+0000","last_active":"2024-04-26T23:01:11.775805+0000","last_peered":"2024-04-26T23:01:11.775805+0000","last_clean":"2024-04-26T23:01:11.775805+0000","last_became_active":"2024-04-26T23:01:04.733352+0000","last_became_peered":"2024-04-26T23:01:04.733352+0000","last_unstale":"2024-04-26T23:01:11.775805+0000","last_undegraded":"2024-04-26T23:01:11.775805+0000","last_fullsized":"2024-04-26T23:01:11.775805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:16.098138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484327+0000","last_change":"2024-04-26T23:01:05.738238+0000","last_active":"2024-04-26T23:01:12.484327+0000","last_peered":"2024-04-26T23:01:12.484327+0000","last_clean":"2024-04-26T23:01:12.484327+0000","last_became_active":"2024-04-26T23:01:05.737945+0000","last_became_peered":"2024-04-26T23:01:05.737945+0000","last_unstale":"2024-04-26T23:01:12.484327+0000","last_undegraded":"2024-04-26T23:01:12.484327+0000","last_fullsized":"2024-04-26T23:01:12.484327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:22.679803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.881869+0000","last_change":"2024-04-26T23:01:09.800139+0000","last_active":"2024-04-26T23:01:11.881869+0000","last_peered":"2024-04-26T23:01:11.881869+0000","last_clean":"2024-04-26T23:01:11.881869+0000","last_became_active":"2024-04-26T23:01:09.799214+0000","last_became_peered":"2024-04-26T23:01:09.799214+0000","last_unstale":"2024-04-26T23:01:11.881869+0000","last_undegraded":"2024-04-26T23:01:11.881869+0000","last_fullsized":"2024-04-26T23:01:11.881869+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:46:57.684555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776133+0000","last_change":"2024-04-26T23:01:04.735697+0000","last_active":"2024-04-26T23:01:11.776133+0000","last_peered":"2024-04-26T23:01:11.776133+0000","last_clean":"2024-04-26T23:01:11.776133+0000","last_became_active":"2024-04-26T23:01:04.735327+0000","last_became_peered":"2024-04-26T23:01:04.735327+0000","last_unstale":"2024-04-26T23:01:11.776133+0000","last_undegraded":"2024-04-26T23:01:11.776133+0000","last_fullsized":"2024-04-26T23:01:11.776133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:58.527150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484663+0000","last_change":"2024-04-26T23:01:05.729770+0000","last_active":"2024-04-26T23:01:12.484663+0000","last_peered":"2024-04-26T23:01:12.484663+0000","last_clean":"2024-04-26T23:01:12.484663+0000","last_became_active":"2024-04-26T23:01:05.727922+0000","last_became_peered":"2024-04-26T23:01:05.727922+0000","last_unstale":"2024-04-26T23:01:12.484663+0000","last_undegraded":"2024-04-26T23:01:12.484663+0000","last_fullsized":"2024-04-26T23:01:12.484663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:44:40.404454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882761+0000","last_change":"2024-04-26T23:01:09.804570+0000","last_active":"2024-04-26T23:01:11.882761+0000","last_peered":"2024-04-26T23:01:11.882761+0000","last_clean":"2024-04-26T23:01:11.882761+0000","last_became_active":"2024-04-26T23:01:09.803489+0000","last_became_peered":"2024-04-26T23:01:09.803489+0000","last_unstale":"2024-04-26T23:01:11.882761+0000","last_undegraded":"2024-04-26T23:01:11.882761+0000","last_fullsized":"2024-04-26T23:01:11.882761+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774588+0000","last_change":"2024-04-26T23:01:05.729351+0000","last_active":"2024-04-26T23:01:11.774588+0000","last_peered":"2024-04-26T23:01:11.774588+0000","last_clean":"2024-04-26T23:01:11.774588+0000","last_became_active":"2024-04-26T23:01:05.729024+0000","last_became_peered":"2024-04-26T23:01:05.729024+0000","last_unstale":"2024-04-26T23:01:11.774588+0000","last_undegraded":"2024-04-26T23:01:11.774588+0000","last_fullsized":"2024-04-26T23:01:11.774588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776686+0000","last_change":"2024-04-26T23:01:04.730260+0000","last_active":"2024-04-26T23:01:11.776686+0000","last_peered":"2024-04-26T23:01:11.776686+0000","last_clean":"2024-04-26T23:01:11.776686+0000","last_became_active":"2024-04-26T23:01:04.729209+0000","last_became_peered":"2024-04-26T23:01:04.729209+0000","last_unstale":"2024-04-26T23:01:11.776686+0000","last_undegraded":"2024-04-26T23:01:11.776686+0000","last_fullsized":"2024-04-26T23:01:11.776686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:45.523274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.883045+0000","last_change":"2024-04-26T23:01:09.796341+0000","last_active":"2024-04-26T23:01:11.883045+0000","last_peered":"2024-04-26T23:01:11.883045+0000","last_clean":"2024-04-26T23:01:11.883045+0000","last_became_active":"2024-04-26T23:01:09.795939+0000","last_became_peered":"2024-04-26T23:01:09.795939+0000","last_unstale":"2024-04-26T23:01:11.883045+0000","last_undegraded":"2024-04-26T23:01:11.883045+0000","last_fullsized":"2024-04-26T23:01:11.883045+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:09:55.216454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770614+0000","last_change":"2024-04-26T23:01:05.735100+0000","last_active":"2024-04-26T23:01:11.770614+0000","last_peered":"2024-04-26T23:01:11.770614+0000","last_clean":"2024-04-26T23:01:11.770614+0000","last_became_active":"2024-04-26T23:01:05.734138+0000","last_became_peered":"2024-04-26T23:01:05.734138+0000","last_unstale":"2024-04-26T23:01:11.770614+0000","last_undegraded":"2024-04-26T23:01:11.770614+0000","last_fullsized":"2024-04-26T23:01:11.770614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486466+0000","last_change":"2024-04-26T23:01:04.723885+0000","last_active":"2024-04-26T23:01:12.486466+0000","last_peered":"2024-04-26T23:01:12.486466+0000","last_clean":"2024-04-26T23:01:12.486466+0000","last_became_active":"2024-04-26T23:01:04.723597+0000","last_became_peered":"2024-04-26T23:01:04.723597+0000","last_unstale":"2024-04-26T23:01:12.486466+0000","last_undegraded":"2024-04-26T23:01:12.486466+0000","last_fullsized":"2024-04-26T23:01:12.486466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:45.523274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882064+0000","last_change":"2024-04-26T23:01:09.803721+0000","last_active":"2024-04-26T23:01:11.882064+0000","last_peered":"2024-04-26T23:01:11.882064+0000","last_clean":"2024-04-26T23:01:11.882064+0000","last_became_active":"2024-04-26T23:01:09.803061+0000","last_became_peered":"2024-04-26T23:01:09.803061+0000","last_unstale":"2024-04-26T23:01:11.882064+0000","last_undegraded":"2024-04-26T23:01:11.882064+0000","last_fullsized":"2024-04-26T23:01:11.882064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:41.540020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769949+0000","last_change":"2024-04-26T23:01:05.738344+0000","last_active":"2024-04-26T23:01:11.769949+0000","last_peered":"2024-04-26T23:01:11.769949+0000","last_clean":"2024-04-26T23:01:11.769949+0000","last_became_active":"2024-04-26T23:01:05.738082+0000","last_became_peered":"2024-04-26T23:01:05.738082+0000","last_unstale":"2024-04-26T23:01:11.769949+0000","last_undegraded":"2024-04-26T23:01:11.769949+0000","last_fullsized":"2024-04-26T23:01:11.769949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:26.157667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484208+0000","last_change":"2024-04-26T23:01:04.727252+0000","last_active":"2024-04-26T23:01:12.484208+0000","last_peered":"2024-04-26T23:01:12.484208+0000","last_clean":"2024-04-26T23:01:12.484208+0000","last_became_active":"2024-04-26T23:01:04.726980+0000","last_became_peered":"2024-04-26T23:01:04.726980+0000","last_unstale":"2024-04-26T23:01:12.484208+0000","last_undegraded":"2024-04-26T23:01:12.484208+0000","last_fullsized":"2024-04-26T23:01:12.484208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486354+0000","last_change":"2024-04-26T23:01:10.399273+0000","last_active":"2024-04-26T23:01:12.486354+0000","last_peered":"2024-04-26T23:01:12.486354+0000","last_clean":"2024-04-26T23:01:12.486354+0000","last_became_active":"2024-04-26T23:01:10.397193+0000","last_became_peered":"2024-04-26T23:01:10.397193+0000","last_unstale":"2024-04-26T23:01:12.486354+0000","last_undegraded":"2024-04-26T23:01:12.486354+0000","last_fullsized":"2024-04-26T23:01:12.486354+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882811+0000","last_change":"2024-04-26T23:01:05.734542+0000","last_active":"2024-04-26T23:01:11.882811+0000","last_peered":"2024-04-26T23:01:11.882811+0000","last_clean":"2024-04-26T23:01:11.882811+0000","last_became_active":"2024-04-26T23:01:05.734191+0000","last_became_peered":"2024-04-26T23:01:05.734191+0000","last_unstale":"2024-04-26T23:01:11.882811+0000","last_undegraded":"2024-04-26T23:01:11.882811+0000","last_fullsized":"2024-04-26T23:01:11.882811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:49.765162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774127+0000","last_change":"2024-04-26T23:01:04.732728+0000","last_active":"2024-04-26T23:01:11.774127+0000","last_peered":"2024-04-26T23:01:11.774127+0000","last_clean":"2024-04-26T23:01:11.774127+0000","last_became_active":"2024-04-26T23:01:04.732413+0000","last_became_peered":"2024-04-26T23:01:04.732413+0000","last_unstale":"2024-04-26T23:01:11.774127+0000","last_undegraded":"2024-04-26T23:01:11.774127+0000","last_fullsized":"2024-04-26T23:01:11.774127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882964+0000","last_change":"2024-04-26T23:01:09.800234+0000","last_active":"2024-04-26T23:01:11.882964+0000","last_peered":"2024-04-26T23:01:11.882964+0000","last_clean":"2024-04-26T23:01:11.882964+0000","last_became_active":"2024-04-26T23:01:09.799644+0000","last_became_peered":"2024-04-26T23:01:09.799644+0000","last_unstale":"2024-04-26T23:01:11.882964+0000","last_undegraded":"2024-04-26T23:01:11.882964+0000","last_fullsized":"2024-04-26T23:01:11.882964+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:37:23.404492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486201+0000","last_change":"2024-04-26T23:01:05.735371+0000","last_active":"2024-04-26T23:01:12.486201+0000","last_peered":"2024-04-26T23:01:12.486201+0000","last_clean":"2024-04-26T23:01:12.486201+0000","last_became_active":"2024-04-26T23:01:05.729667+0000","last_became_peered":"2024-04-26T23:01:05.729667+0000","last_unstale":"2024-04-26T23:01:12.486201+0000","last_undegraded":"2024-04-26T23:01:12.486201+0000","last_fullsized":"2024-04-26T23:01:12.486201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778387+0000","last_change":"2024-04-26T23:01:04.726450+0000","last_active":"2024-04-26T23:01:11.778387+0000","last_peered":"2024-04-26T23:01:11.778387+0000","last_clean":"2024-04-26T23:01:11.778387+0000","last_became_active":"2024-04-26T23:01:04.726023+0000","last_became_peered":"2024-04-26T23:01:04.726023+0000","last_unstale":"2024-04-26T23:01:11.778387+0000","last_undegraded":"2024-04-26T23:01:11.778387+0000","last_fullsized":"2024-04-26T23:01:11.778387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:45.523274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.881567+0000","last_change":"2024-04-26T23:01:10.401428+0000","last_active":"2024-04-26T23:01:11.881567+0000","last_peered":"2024-04-26T23:01:11.881567+0000","last_clean":"2024-04-26T23:01:11.881567+0000","last_became_active":"2024-04-26T23:01:10.400621+0000","last_became_peered":"2024-04-26T23:01:10.400621+0000","last_unstale":"2024-04-26T23:01:11.881567+0000","last_undegraded":"2024-04-26T23:01:11.881567+0000","last_fullsized":"2024-04-26T23:01:11.881567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:59.067859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771304+0000","last_change":"2024-04-26T23:01:05.748687+0000","last_active":"2024-04-26T23:01:11.771304+0000","last_peered":"2024-04-26T23:01:11.771304+0000","last_clean":"2024-04-26T23:01:11.771304+0000","last_became_active":"2024-04-26T23:01:05.744038+0000","last_became_peered":"2024-04-26T23:01:05.744038+0000","last_unstale":"2024-04-26T23:01:11.771304+0000","last_undegraded":"2024-04-26T23:01:11.771304+0000","last_fullsized":"2024-04-26T23:01:11.771304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:13.482334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486378+0000","last_change":"2024-04-26T23:01:04.718414+0000","last_active":"2024-04-26T23:01:12.486378+0000","last_peered":"2024-04-26T23:01:12.486378+0000","last_clean":"2024-04-26T23:01:12.486378+0000","last_became_active":"2024-04-26T23:01:04.717764+0000","last_became_peered":"2024-04-26T23:01:04.717764+0000","last_unstale":"2024-04-26T23:01:12.486378+0000","last_undegraded":"2024-04-26T23:01:12.486378+0000","last_fullsized":"2024-04-26T23:01:12.486378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769756+0000","last_change":"2024-04-26T23:01:10.386716+0000","last_active":"2024-04-26T23:01:11.769756+0000","last_peered":"2024-04-26T23:01:11.769756+0000","last_clean":"2024-04-26T23:01:11.769756+0000","last_became_active":"2024-04-26T23:01:10.385392+0000","last_became_peered":"2024-04-26T23:01:10.385392+0000","last_unstale":"2024-04-26T23:01:11.769756+0000","last_undegraded":"2024-04-26T23:01:11.769756+0000","last_fullsized":"2024-04-26T23:01:11.769756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:09:55.216454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485210+0000","last_change":"2024-04-26T23:01:05.728786+0000","last_active":"2024-04-26T23:01:12.485210+0000","last_peered":"2024-04-26T23:01:12.485210+0000","last_clean":"2024-04-26T23:01:12.485210+0000","last_became_active":"2024-04-26T23:01:05.728430+0000","last_became_peered":"2024-04-26T23:01:05.728430+0000","last_unstale":"2024-04-26T23:01:12.485210+0000","last_undegraded":"2024-04-26T23:01:12.485210+0000","last_fullsized":"2024-04-26T23:01:12.485210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776238+0000","last_change":"2024-04-26T23:01:04.734438+0000","last_active":"2024-04-26T23:01:11.776238+0000","last_peered":"2024-04-26T23:01:11.776238+0000","last_clean":"2024-04-26T23:01:11.776238+0000","last_became_active":"2024-04-26T23:01:04.733372+0000","last_became_peered":"2024-04-26T23:01:04.733372+0000","last_unstale":"2024-04-26T23:01:11.776238+0000","last_undegraded":"2024-04-26T23:01:11.776238+0000","last_fullsized":"2024-04-26T23:01:11.776238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:55.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483768+0000","last_change":"2024-04-26T23:01:10.391451+0000","last_active":"2024-04-26T23:01:12.483768+0000","last_peered":"2024-04-26T23:01:12.483768+0000","last_clean":"2024-04-26T23:01:12.483768+0000","last_became_active":"2024-04-26T23:01:10.389477+0000","last_became_peered":"2024-04-26T23:01:10.389477+0000","last_unstale":"2024-04-26T23:01:12.483768+0000","last_undegraded":"2024-04-26T23:01:12.483768+0000","last_fullsized":"2024-04-26T23:01:12.483768+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:46:57.684555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485051+0000","last_change":"2024-04-26T23:01:05.746004+0000","last_active":"2024-04-26T23:01:12.485051+0000","last_peered":"2024-04-26T23:01:12.485051+0000","last_clean":"2024-04-26T23:01:12.485051+0000","last_became_active":"2024-04-26T23:01:05.745576+0000","last_became_peered":"2024-04-26T23:01:05.745576+0000","last_unstale":"2024-04-26T23:01:12.485051+0000","last_undegraded":"2024-04-26T23:01:12.485051+0000","last_fullsized":"2024-04-26T23:01:12.485051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:06.384302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485393+0000","last_change":"2024-04-26T23:01:04.717206+0000","last_active":"2024-04-26T23:01:12.485393+0000","last_peered":"2024-04-26T23:01:12.485393+0000","last_clean":"2024-04-26T23:01:12.485393+0000","last_became_active":"2024-04-26T23:01:04.715952+0000","last_became_peered":"2024-04-26T23:01:04.715952+0000","last_unstale":"2024-04-26T23:01:12.485393+0000","last_undegraded":"2024-04-26T23:01:12.485393+0000","last_fullsized":"2024-04-26T23:01:12.485393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485772+0000","last_change":"2024-04-26T23:01:09.799096+0000","last_active":"2024-04-26T23:01:12.485772+0000","last_peered":"2024-04-26T23:01:12.485772+0000","last_clean":"2024-04-26T23:01:12.485772+0000","last_became_active":"2024-04-26T23:01:09.798586+0000","last_became_peered":"2024-04-26T23:01:09.798586+0000","last_unstale":"2024-04-26T23:01:12.485772+0000","last_undegraded":"2024-04-26T23:01:12.485772+0000","last_fullsized":"2024-04-26T23:01:12.485772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:30.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.883148+0000","last_change":"2024-04-26T23:01:05.732977+0000","last_active":"2024-04-26T23:01:11.883148+0000","last_peered":"2024-04-26T23:01:11.883148+0000","last_clean":"2024-04-26T23:01:11.883148+0000","last_became_active":"2024-04-26T23:01:05.732617+0000","last_became_peered":"2024-04-26T23:01:05.732617+0000","last_unstale":"2024-04-26T23:01:11.883148+0000","last_undegraded":"2024-04-26T23:01:11.883148+0000","last_fullsized":"2024-04-26T23:01:11.883148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:59.540986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484829+0000","last_change":"2024-04-26T23:01:04.730386+0000","last_active":"2024-04-26T23:01:12.484829+0000","last_peered":"2024-04-26T23:01:12.484829+0000","last_clean":"2024-04-26T23:01:12.484829+0000","last_became_active":"2024-04-26T23:01:04.727828+0000","last_became_peered":"2024-04-26T23:01:04.727828+0000","last_unstale":"2024-04-26T23:01:12.484829+0000","last_undegraded":"2024-04-26T23:01:12.484829+0000","last_fullsized":"2024-04-26T23:01:12.484829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485312+0000","last_change":"2024-04-26T23:01:09.805894+0000","last_active":"2024-04-26T23:01:12.485312+0000","last_peered":"2024-04-26T23:01:12.485312+0000","last_clean":"2024-04-26T23:01:12.485312+0000","last_became_active":"2024-04-26T23:01:09.803400+0000","last_became_peered":"2024-04-26T23:01:09.803400+0000","last_unstale":"2024-04-26T23:01:12.485312+0000","last_undegraded":"2024-04-26T23:01:12.485312+0000","last_fullsized":"2024-04-26T23:01:12.485312+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:46:57.684555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484288+0000","last_change":"2024-04-26T23:01:05.722407+0000","last_active":"2024-04-26T23:01:12.484288+0000","last_peered":"2024-04-26T23:01:12.484288+0000","last_clean":"2024-04-26T23:01:12.484288+0000","last_became_active":"2024-04-26T23:01:05.722106+0000","last_became_peered":"2024-04-26T23:01:05.722106+0000","last_unstale":"2024-04-26T23:01:12.484288+0000","last_undegraded":"2024-04-26T23:01:12.484288+0000","last_fullsized":"2024-04-26T23:01:12.484288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487617+0000","last_change":"2024-04-26T23:01:04.712172+0000","last_active":"2024-04-26T23:01:12.487617+0000","last_peered":"2024-04-26T23:01:12.487617+0000","last_clean":"2024-04-26T23:01:12.487617+0000","last_became_active":"2024-04-26T23:01:04.711777+0000","last_became_peered":"2024-04-26T23:01:04.711777+0000","last_unstale":"2024-04-26T23:01:12.487617+0000","last_undegraded":"2024-04-26T23:01:12.487617+0000","last_fullsized":"2024-04-26T23:01:12.487617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486748+0000","last_change":"2024-04-26T23:01:09.808963+0000","last_active":"2024-04-26T23:01:12.486748+0000","last_peered":"2024-04-26T23:01:12.486748+0000","last_clean":"2024-04-26T23:01:12.486748+0000","last_became_active":"2024-04-26T23:01:09.808470+0000","last_became_peered":"2024-04-26T23:01:09.808470+0000","last_unstale":"2024-04-26T23:01:12.486748+0000","last_undegraded":"2024-04-26T23:01:12.486748+0000","last_fullsized":"2024-04-26T23:01:12.486748+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.482634+0000","last_change":"2024-04-26T23:01:05.742128+0000","last_active":"2024-04-26T23:01:12.482634+0000","last_peered":"2024-04-26T23:01:12.482634+0000","last_clean":"2024-04-26T23:01:12.482634+0000","last_became_active":"2024-04-26T23:01:05.735959+0000","last_became_peered":"2024-04-26T23:01:05.735959+0000","last_unstale":"2024-04-26T23:01:12.482634+0000","last_undegraded":"2024-04-26T23:01:12.482634+0000","last_fullsized":"2024-04-26T23:01:12.482634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774374+0000","last_change":"2024-04-26T23:01:04.732093+0000","last_active":"2024-04-26T23:01:11.774374+0000","last_peered":"2024-04-26T23:01:11.774374+0000","last_clean":"2024-04-26T23:01:11.774374+0000","last_became_active":"2024-04-26T23:01:04.731804+0000","last_became_peered":"2024-04-26T23:01:04.731804+0000","last_unstale":"2024-04-26T23:01:11.774374+0000","last_undegraded":"2024-04-26T23:01:11.774374+0000","last_fullsized":"2024-04-26T23:01:11.774374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485713+0000","last_change":"2024-04-26T23:01:10.389206+0000","last_active":"2024-04-26T23:01:12.485713+0000","last_peered":"2024-04-26T23:01:12.485713+0000","last_clean":"2024-04-26T23:01:12.485713+0000","last_became_active":"2024-04-26T23:01:10.384213+0000","last_became_peered":"2024-04-26T23:01:10.384213+0000","last_unstale":"2024-04-26T23:01:12.485713+0000","last_undegraded":"2024-04-26T23:01:12.485713+0000","last_fullsized":"2024-04-26T23:01:12.485713+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770463+0000","last_change":"2024-04-26T23:01:05.744836+0000","last_active":"2024-04-26T23:01:11.770463+0000","last_peered":"2024-04-26T23:01:11.770463+0000","last_clean":"2024-04-26T23:01:11.770463+0000","last_became_active":"2024-04-26T23:01:05.744600+0000","last_became_peered":"2024-04-26T23:01:05.744600+0000","last_unstale":"2024-04-26T23:01:11.770463+0000","last_undegraded":"2024-04-26T23:01:11.770463+0000","last_fullsized":"2024-04-26T23:01:11.770463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484137+0000","last_change":"2024-04-26T23:01:04.729176+0000","last_active":"2024-04-26T23:01:12.484137+0000","last_peered":"2024-04-26T23:01:12.484137+0000","last_clean":"2024-04-26T23:01:12.484137+0000","last_became_active":"2024-04-26T23:01:04.722896+0000","last_became_peered":"2024-04-26T23:01:04.722896+0000","last_unstale":"2024-04-26T23:01:12.484137+0000","last_undegraded":"2024-04-26T23:01:12.484137+0000","last_fullsized":"2024-04-26T23:01:12.484137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:16.585834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":180,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774617+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:01:11.774617+0000","last_peered":"2024-04-26T23:01:11.774617+0000","last_clean":"2024-04-26T23:01:11.774617+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:01:11.774617+0000","last_undegraded":"2024-04-26T23:01:11.774617+0000","last_fullsized":"2024-04-26T23:01:11.774617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777714+0000","last_change":"2024-04-26T23:01:10.402293+0000","last_active":"2024-04-26T23:01:11.777714+0000","last_peered":"2024-04-26T23:01:11.777714+0000","last_clean":"2024-04-26T23:01:11.777714+0000","last_became_active":"2024-04-26T23:01:10.396954+0000","last_became_peered":"2024-04-26T23:01:10.396954+0000","last_unstale":"2024-04-26T23:01:11.777714+0000","last_undegraded":"2024-04-26T23:01:11.777714+0000","last_fullsized":"2024-04-26T23:01:11.777714+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:44.757523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775315+0000","last_change":"2024-04-26T23:01:05.731266+0000","last_active":"2024-04-26T23:01:11.775315+0000","last_peered":"2024-04-26T23:01:11.775315+0000","last_clean":"2024-04-26T23:01:11.775315+0000","last_became_active":"2024-04-26T23:01:05.728530+0000","last_became_peered":"2024-04-26T23:01:05.728530+0000","last_unstale":"2024-04-26T23:01:11.775315+0000","last_undegraded":"2024-04-26T23:01:11.775315+0000","last_fullsized":"2024-04-26T23:01:11.775315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:44:40.404454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.488463+0000","last_change":"2024-04-26T23:01:04.728315+0000","last_active":"2024-04-26T23:01:12.488463+0000","last_peered":"2024-04-26T23:01:12.488463+0000","last_clean":"2024-04-26T23:01:12.488463+0000","last_became_active":"2024-04-26T23:01:04.727609+0000","last_became_peered":"2024-04-26T23:01:04.727609+0000","last_unstale":"2024-04-26T23:01:12.488463+0000","last_undegraded":"2024-04-26T23:01:12.488463+0000","last_fullsized":"2024-04-26T23:01:12.488463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775771+0000","last_change":"2024-04-26T23:01:10.391227+0000","last_active":"2024-04-26T23:01:11.775771+0000","last_peered":"2024-04-26T23:01:11.775771+0000","last_clean":"2024-04-26T23:01:11.775771+0000","last_became_active":"2024-04-26T23:01:10.387715+0000","last_became_peered":"2024-04-26T23:01:10.387715+0000","last_unstale":"2024-04-26T23:01:11.775771+0000","last_undegraded":"2024-04-26T23:01:11.775771+0000","last_fullsized":"2024-04-26T23:01:11.775771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:28.895329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484465+0000","last_change":"2024-04-26T23:01:05.753852+0000","last_active":"2024-04-26T23:01:12.484465+0000","last_peered":"2024-04-26T23:01:12.484465+0000","last_clean":"2024-04-26T23:01:12.484465+0000","last_became_active":"2024-04-26T23:01:05.753607+0000","last_became_peered":"2024-04-26T23:01:05.753607+0000","last_unstale":"2024-04-26T23:01:12.484465+0000","last_undegraded":"2024-04-26T23:01:12.484465+0000","last_fullsized":"2024-04-26T23:01:12.484465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:59:57.750603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485129+0000","last_change":"2024-04-26T23:01:04.713352+0000","last_active":"2024-04-26T23:01:12.485129+0000","last_peered":"2024-04-26T23:01:12.485129+0000","last_clean":"2024-04-26T23:01:12.485129+0000","last_became_active":"2024-04-26T23:01:04.712352+0000","last_became_peered":"2024-04-26T23:01:04.712352+0000","last_unstale":"2024-04-26T23:01:12.485129+0000","last_undegraded":"2024-04-26T23:01:12.485129+0000","last_fullsized":"2024-04-26T23:01:12.485129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:55.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484995+0000","last_change":"2024-04-26T23:01:09.803857+0000","last_active":"2024-04-26T23:01:12.484995+0000","last_peered":"2024-04-26T23:01:12.484995+0000","last_clean":"2024-04-26T23:01:12.484995+0000","last_became_active":"2024-04-26T23:01:09.799482+0000","last_became_peered":"2024-04-26T23:01:09.799482+0000","last_unstale":"2024-04-26T23:01:12.484995+0000","last_undegraded":"2024-04-26T23:01:12.484995+0000","last_fullsized":"2024-04-26T23:01:12.484995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:41.540020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776723+0000","last_change":"2024-04-26T23:01:05.741597+0000","last_active":"2024-04-26T23:01:11.776723+0000","last_peered":"2024-04-26T23:01:11.776723+0000","last_clean":"2024-04-26T23:01:11.776723+0000","last_became_active":"2024-04-26T23:01:05.741308+0000","last_became_peered":"2024-04-26T23:01:05.741308+0000","last_unstale":"2024-04-26T23:01:11.776723+0000","last_undegraded":"2024-04-26T23:01:11.776723+0000","last_fullsized":"2024-04-26T23:01:11.776723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:54.996731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487896+0000","last_change":"2024-04-26T23:01:04.720208+0000","last_active":"2024-04-26T23:01:12.487896+0000","last_peered":"2024-04-26T23:01:12.487896+0000","last_clean":"2024-04-26T23:01:12.487896+0000","last_became_active":"2024-04-26T23:01:04.719867+0000","last_became_peered":"2024-04-26T23:01:04.719867+0000","last_unstale":"2024-04-26T23:01:12.487896+0000","last_undegraded":"2024-04-26T23:01:12.487896+0000","last_fullsized":"2024-04-26T23:01:12.487896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:16.585834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483634+0000","last_change":"2024-04-26T23:01:10.396595+0000","last_active":"2024-04-26T23:01:12.483634+0000","last_peered":"2024-04-26T23:01:12.483634+0000","last_clean":"2024-04-26T23:01:12.483634+0000","last_became_active":"2024-04-26T23:01:10.395189+0000","last_became_peered":"2024-04-26T23:01:10.395189+0000","last_unstale":"2024-04-26T23:01:12.483634+0000","last_undegraded":"2024-04-26T23:01:12.483634+0000","last_fullsized":"2024-04-26T23:01:12.483634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776069+0000","last_change":"2024-04-26T23:01:05.733520+0000","last_active":"2024-04-26T23:01:11.776069+0000","last_peered":"2024-04-26T23:01:11.776069+0000","last_clean":"2024-04-26T23:01:11.776069+0000","last_became_active":"2024-04-26T23:01:05.733170+0000","last_became_peered":"2024-04-26T23:01:05.733170+0000","last_unstale":"2024-04-26T23:01:11.776069+0000","last_undegraded":"2024-04-26T23:01:11.776069+0000","last_fullsized":"2024-04-26T23:01:11.776069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:06.384302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.883050+0000","last_change":"2024-04-26T23:01:04.717930+0000","last_active":"2024-04-26T23:01:11.883050+0000","last_peered":"2024-04-26T23:01:11.883050+0000","last_clean":"2024-04-26T23:01:11.883050+0000","last_became_active":"2024-04-26T23:01:04.717576+0000","last_became_peered":"2024-04-26T23:01:04.717576+0000","last_unstale":"2024-04-26T23:01:11.883050+0000","last_undegraded":"2024-04-26T23:01:11.883050+0000","last_fullsized":"2024-04-26T23:01:11.883050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:11:48.078530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882003+0000","last_change":"2024-04-26T23:01:09.797064+0000","last_active":"2024-04-26T23:01:11.882003+0000","last_peered":"2024-04-26T23:01:11.882003+0000","last_clean":"2024-04-26T23:01:11.882003+0000","last_became_active":"2024-04-26T23:01:09.796589+0000","last_became_peered":"2024-04-26T23:01:09.796589+0000","last_unstale":"2024-04-26T23:01:11.882003+0000","last_undegraded":"2024-04-26T23:01:11.882003+0000","last_fullsized":"2024-04-26T23:01:11.882003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:30.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777169+0000","last_change":"2024-04-26T23:01:05.738519+0000","last_active":"2024-04-26T23:01:11.777169+0000","last_peered":"2024-04-26T23:01:11.777169+0000","last_clean":"2024-04-26T23:01:11.777169+0000","last_became_active":"2024-04-26T23:01:05.738201+0000","last_became_peered":"2024-04-26T23:01:05.738201+0000","last_unstale":"2024-04-26T23:01:11.777169+0000","last_undegraded":"2024-04-26T23:01:11.777169+0000","last_fullsized":"2024-04-26T23:01:11.777169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487284+0000","last_change":"2024-04-26T23:01:04.711562+0000","last_active":"2024-04-26T23:01:12.487284+0000","last_peered":"2024-04-26T23:01:12.487284+0000","last_clean":"2024-04-26T23:01:12.487284+0000","last_became_active":"2024-04-26T23:01:04.708852+0000","last_became_peered":"2024-04-26T23:01:04.708852+0000","last_unstale":"2024-04-26T23:01:12.487284+0000","last_undegraded":"2024-04-26T23:01:12.487284+0000","last_fullsized":"2024-04-26T23:01:12.487284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:21:22.581263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485270+0000","last_change":"2024-04-26T23:01:09.805302+0000","last_active":"2024-04-26T23:01:12.485270+0000","last_peered":"2024-04-26T23:01:12.485270+0000","last_clean":"2024-04-26T23:01:12.485270+0000","last_became_active":"2024-04-26T23:01:09.804878+0000","last_became_peered":"2024-04-26T23:01:09.804878+0000","last_unstale":"2024-04-26T23:01:12.485270+0000","last_undegraded":"2024-04-26T23:01:12.485270+0000","last_fullsized":"2024-04-26T23:01:12.485270+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774404+0000","last_change":"2024-04-26T23:01:05.717338+0000","last_active":"2024-04-26T23:01:11.774404+0000","last_peered":"2024-04-26T23:01:11.774404+0000","last_clean":"2024-04-26T23:01:11.774404+0000","last_became_active":"2024-04-26T23:01:05.716983+0000","last_became_peered":"2024-04-26T23:01:05.716983+0000","last_unstale":"2024-04-26T23:01:11.774404+0000","last_undegraded":"2024-04-26T23:01:11.774404+0000","last_fullsized":"2024-04-26T23:01:11.774404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774791+0000","last_change":"2024-04-26T23:01:04.715584+0000","last_active":"2024-04-26T23:01:11.774791+0000","last_peered":"2024-04-26T23:01:11.774791+0000","last_clean":"2024-04-26T23:01:11.774791+0000","last_became_active":"2024-04-26T23:01:04.715240+0000","last_became_peered":"2024-04-26T23:01:04.715240+0000","last_unstale":"2024-04-26T23:01:11.774791+0000","last_undegraded":"2024-04-26T23:01:11.774791+0000","last_fullsized":"2024-04-26T23:01:11.774791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771265+0000","last_change":"2024-04-26T23:01:10.401101+0000","last_active":"2024-04-26T23:01:11.771265+0000","last_peered":"2024-04-26T23:01:11.771265+0000","last_clean":"2024-04-26T23:01:11.771265+0000","last_became_active":"2024-04-26T23:01:10.396052+0000","last_became_peered":"2024-04-26T23:01:10.396052+0000","last_unstale":"2024-04-26T23:01:11.771265+0000","last_undegraded":"2024-04-26T23:01:11.771265+0000","last_fullsized":"2024-04-26T23:01:11.771265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:15.222701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776466+0000","last_change":"2024-04-26T23:01:05.732489+0000","last_active":"2024-04-26T23:01:11.776466+0000","last_peered":"2024-04-26T23:01:11.776466+0000","last_clean":"2024-04-26T23:01:11.776466+0000","last_became_active":"2024-04-26T23:01:05.732228+0000","last_became_peered":"2024-04-26T23:01:05.732228+0000","last_unstale":"2024-04-26T23:01:11.776466+0000","last_undegraded":"2024-04-26T23:01:11.776466+0000","last_fullsized":"2024-04-26T23:01:11.776466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:22:15.301548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484809+0000","last_change":"2024-04-26T23:01:04.731558+0000","last_active":"2024-04-26T23:01:12.484809+0000","last_peered":"2024-04-26T23:01:12.484809+0000","last_clean":"2024-04-26T23:01:12.484809+0000","last_became_active":"2024-04-26T23:01:04.731032+0000","last_became_peered":"2024-04-26T23:01:04.731032+0000","last_unstale":"2024-04-26T23:01:12.484809+0000","last_undegraded":"2024-04-26T23:01:12.484809+0000","last_fullsized":"2024-04-26T23:01:12.484809+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770866+0000","last_change":"2024-04-26T23:01:10.386434+0000","last_active":"2024-04-26T23:01:11.770866+0000","last_peered":"2024-04-26T23:01:11.770866+0000","last_clean":"2024-04-26T23:01:11.770866+0000","last_became_active":"2024-04-26T23:01:10.385162+0000","last_became_peered":"2024-04-26T23:01:10.385162+0000","last_unstale":"2024-04-26T23:01:11.770866+0000","last_undegraded":"2024-04-26T23:01:11.770866+0000","last_fullsized":"2024-04-26T23:01:11.770866+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:19.562651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483154+0000","last_change":"2024-04-26T23:01:05.740528+0000","last_active":"2024-04-26T23:01:12.483154+0000","last_peered":"2024-04-26T23:01:12.483154+0000","last_clean":"2024-04-26T23:01:12.483154+0000","last_became_active":"2024-04-26T23:01:05.740211+0000","last_became_peered":"2024-04-26T23:01:05.740211+0000","last_unstale":"2024-04-26T23:01:12.483154+0000","last_undegraded":"2024-04-26T23:01:12.483154+0000","last_fullsized":"2024-04-26T23:01:12.483154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484416+0000","last_change":"2024-04-26T23:01:04.712658+0000","last_active":"2024-04-26T23:01:12.484416+0000","last_peered":"2024-04-26T23:01:12.484416+0000","last_clean":"2024-04-26T23:01:12.484416+0000","last_became_active":"2024-04-26T23:01:04.712325+0000","last_became_peered":"2024-04-26T23:01:04.712325+0000","last_unstale":"2024-04-26T23:01:12.484416+0000","last_undegraded":"2024-04-26T23:01:12.484416+0000","last_fullsized":"2024-04-26T23:01:12.484416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:52.271211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485257+0000","last_change":"2024-04-26T23:01:10.385938+0000","last_active":"2024-04-26T23:01:12.485257+0000","last_peered":"2024-04-26T23:01:12.485257+0000","last_clean":"2024-04-26T23:01:12.485257+0000","last_became_active":"2024-04-26T23:01:10.384600+0000","last_became_peered":"2024-04-26T23:01:10.384600+0000","last_unstale":"2024-04-26T23:01:12.485257+0000","last_undegraded":"2024-04-26T23:01:12.485257+0000","last_fullsized":"2024-04-26T23:01:12.485257+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775962+0000","last_change":"2024-04-26T23:01:05.744208+0000","last_active":"2024-04-26T23:01:11.775962+0000","last_peered":"2024-04-26T23:01:11.775962+0000","last_clean":"2024-04-26T23:01:11.775962+0000","last_became_active":"2024-04-26T23:01:05.743897+0000","last_became_peered":"2024-04-26T23:01:05.743897+0000","last_unstale":"2024-04-26T23:01:11.775962+0000","last_undegraded":"2024-04-26T23:01:11.775962+0000","last_fullsized":"2024-04-26T23:01:11.775962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:22.679803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:25.136494+0000","last_change":"2024-04-26T23:01:04.723670+0000","last_active":"2024-04-26T23:01:25.136494+0000","last_peered":"2024-04-26T23:01:25.136494+0000","last_clean":"2024-04-26T23:01:25.136494+0000","last_became_active":"2024-04-26T23:01:04.723361+0000","last_became_peered":"2024-04-26T23:01:04.723361+0000","last_unstale":"2024-04-26T23:01:25.136494+0000","last_undegraded":"2024-04-26T23:01:25.136494+0000","last_fullsized":"2024-04-26T23:01:25.136494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:58.527150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485782+0000","last_change":"2024-04-26T23:01:09.799732+0000","last_active":"2024-04-26T23:01:12.485782+0000","last_peered":"2024-04-26T23:01:12.485782+0000","last_clean":"2024-04-26T23:01:12.485782+0000","last_became_active":"2024-04-26T23:01:09.797973+0000","last_became_peered":"2024-04-26T23:01:09.797973+0000","last_unstale":"2024-04-26T23:01:12.485782+0000","last_undegraded":"2024-04-26T23:01:12.485782+0000","last_fullsized":"2024-04-26T23:01:12.485782+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:41.540020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882483+0000","last_change":"2024-04-26T23:01:05.732675+0000","last_active":"2024-04-26T23:01:11.882483+0000","last_peered":"2024-04-26T23:01:11.882483+0000","last_clean":"2024-04-26T23:01:11.882483+0000","last_became_active":"2024-04-26T23:01:05.732334+0000","last_became_peered":"2024-04-26T23:01:05.732334+0000","last_unstale":"2024-04-26T23:01:11.882483+0000","last_undegraded":"2024-04-26T23:01:11.882483+0000","last_fullsized":"2024-04-26T23:01:11.882483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:46.537111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484886+0000","last_change":"2024-04-26T23:01:04.731957+0000","last_active":"2024-04-26T23:01:12.484886+0000","last_peered":"2024-04-26T23:01:12.484886+0000","last_clean":"2024-04-26T23:01:12.484886+0000","last_became_active":"2024-04-26T23:01:04.731688+0000","last_became_peered":"2024-04-26T23:01:04.731688+0000","last_unstale":"2024-04-26T23:01:12.484886+0000","last_undegraded":"2024-04-26T23:01:12.484886+0000","last_fullsized":"2024-04-26T23:01:12.484886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:16.098138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882537+0000","last_change":"2024-04-26T23:01:09.793585+0000","last_active":"2024-04-26T23:01:11.882537+0000","last_peered":"2024-04-26T23:01:11.882537+0000","last_clean":"2024-04-26T23:01:11.882537+0000","last_became_active":"2024-04-26T23:01:09.793177+0000","last_became_peered":"2024-04-26T23:01:09.793177+0000","last_unstale":"2024-04-26T23:01:11.882537+0000","last_undegraded":"2024-04-26T23:01:11.882537+0000","last_fullsized":"2024-04-26T23:01:11.882537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487118+0000","last_change":"2024-04-26T23:01:05.723191+0000","last_active":"2024-04-26T23:01:12.487118+0000","last_peered":"2024-04-26T23:01:12.487118+0000","last_clean":"2024-04-26T23:01:12.487118+0000","last_became_active":"2024-04-26T23:01:05.722903+0000","last_became_peered":"2024-04-26T23:01:05.722903+0000","last_unstale":"2024-04-26T23:01:12.487118+0000","last_undegraded":"2024-04-26T23:01:12.487118+0000","last_fullsized":"2024-04-26T23:01:12.487118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484699+0000","last_change":"2024-04-26T23:01:04.729093+0000","last_active":"2024-04-26T23:01:12.484699+0000","last_peered":"2024-04-26T23:01:12.484699+0000","last_clean":"2024-04-26T23:01:12.484699+0000","last_became_active":"2024-04-26T23:01:04.728125+0000","last_became_peered":"2024-04-26T23:01:04.728125+0000","last_unstale":"2024-04-26T23:01:12.484699+0000","last_undegraded":"2024-04-26T23:01:12.484699+0000","last_fullsized":"2024-04-26T23:01:12.484699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769989+0000","last_change":"2024-04-26T23:01:09.802623+0000","last_active":"2024-04-26T23:01:11.769989+0000","last_peered":"2024-04-26T23:01:11.769989+0000","last_clean":"2024-04-26T23:01:11.769989+0000","last_became_active":"2024-04-26T23:01:09.798323+0000","last_became_peered":"2024-04-26T23:01:09.798323+0000","last_unstale":"2024-04-26T23:01:11.769989+0000","last_undegraded":"2024-04-26T23:01:11.769989+0000","last_fullsized":"2024-04-26T23:01:11.769989+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:56.662302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484537+0000","last_change":"2024-04-26T23:01:05.727245+0000","last_active":"2024-04-26T23:01:12.484537+0000","last_peered":"2024-04-26T23:01:12.484537+0000","last_clean":"2024-04-26T23:01:12.484537+0000","last_became_active":"2024-04-26T23:01:05.726786+0000","last_became_peered":"2024-04-26T23:01:05.726786+0000","last_unstale":"2024-04-26T23:01:12.484537+0000","last_undegraded":"2024-04-26T23:01:12.484537+0000","last_fullsized":"2024-04-26T23:01:12.484537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:24.824202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778010+0000","last_change":"2024-04-26T23:01:04.732475+0000","last_active":"2024-04-26T23:01:11.778010+0000","last_peered":"2024-04-26T23:01:11.778010+0000","last_clean":"2024-04-26T23:01:11.778010+0000","last_became_active":"2024-04-26T23:01:04.732185+0000","last_became_peered":"2024-04-26T23:01:04.732185+0000","last_unstale":"2024-04-26T23:01:11.778010+0000","last_undegraded":"2024-04-26T23:01:11.778010+0000","last_fullsized":"2024-04-26T23:01:11.778010+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770229+0000","last_change":"2024-04-26T23:01:10.396108+0000","last_active":"2024-04-26T23:01:11.770229+0000","last_peered":"2024-04-26T23:01:11.770229+0000","last_clean":"2024-04-26T23:01:11.770229+0000","last_became_active":"2024-04-26T23:01:10.394701+0000","last_became_peered":"2024-04-26T23:01:10.394701+0000","last_unstale":"2024-04-26T23:01:11.770229+0000","last_undegraded":"2024-04-26T23:01:11.770229+0000","last_fullsized":"2024-04-26T23:01:11.770229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:00:11.874893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484827+0000","last_change":"2024-04-26T23:01:10.396863+0000","last_active":"2024-04-26T23:01:12.484827+0000","last_peered":"2024-04-26T23:01:12.484827+0000","last_clean":"2024-04-26T23:01:12.484827+0000","last_became_active":"2024-04-26T23:01:10.395914+0000","last_became_peered":"2024-04-26T23:01:10.395914+0000","last_unstale":"2024-04-26T23:01:12.484827+0000","last_undegraded":"2024-04-26T23:01:12.484827+0000","last_fullsized":"2024-04-26T23:01:12.484827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:30.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483687+0000","last_change":"2024-04-26T23:01:05.749537+0000","last_active":"2024-04-26T23:01:12.483687+0000","last_peered":"2024-04-26T23:01:12.483687+0000","last_clean":"2024-04-26T23:01:12.483687+0000","last_became_active":"2024-04-26T23:01:05.749254+0000","last_became_peered":"2024-04-26T23:01:05.749254+0000","last_unstale":"2024-04-26T23:01:12.483687+0000","last_undegraded":"2024-04-26T23:01:12.483687+0000","last_fullsized":"2024-04-26T23:01:12.483687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483804+0000","last_change":"2024-04-26T23:01:04.730472+0000","last_active":"2024-04-26T23:01:12.483804+0000","last_peered":"2024-04-26T23:01:12.483804+0000","last_clean":"2024-04-26T23:01:12.483804+0000","last_became_active":"2024-04-26T23:01:04.729294+0000","last_became_peered":"2024-04-26T23:01:04.729294+0000","last_unstale":"2024-04-26T23:01:12.483804+0000","last_undegraded":"2024-04-26T23:01:12.483804+0000","last_fullsized":"2024-04-26T23:01:12.483804+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:58.527150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778171+0000","last_change":"2024-04-26T23:01:09.800700+0000","last_active":"2024-04-26T23:01:11.778171+0000","last_peered":"2024-04-26T23:01:11.778171+0000","last_clean":"2024-04-26T23:01:11.778171+0000","last_became_active":"2024-04-26T23:01:09.799085+0000","last_became_peered":"2024-04-26T23:01:09.799085+0000","last_unstale":"2024-04-26T23:01:11.778171+0000","last_undegraded":"2024-04-26T23:01:11.778171+0000","last_fullsized":"2024-04-26T23:01:11.778171+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:59.067859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769853+0000","last_change":"2024-04-26T23:01:05.737423+0000","last_active":"2024-04-26T23:01:11.769853+0000","last_peered":"2024-04-26T23:01:11.769853+0000","last_clean":"2024-04-26T23:01:11.769853+0000","last_became_active":"2024-04-26T23:01:05.736881+0000","last_became_peered":"2024-04-26T23:01:05.736881+0000","last_unstale":"2024-04-26T23:01:11.769853+0000","last_undegraded":"2024-04-26T23:01:11.769853+0000","last_fullsized":"2024-04-26T23:01:11.769853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:35:08.020426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770314+0000","last_change":"2024-04-26T23:01:04.723415+0000","last_active":"2024-04-26T23:01:11.770314+0000","last_peered":"2024-04-26T23:01:11.770314+0000","last_clean":"2024-04-26T23:01:11.770314+0000","last_became_active":"2024-04-26T23:01:04.723109+0000","last_became_peered":"2024-04-26T23:01:04.723109+0000","last_unstale":"2024-04-26T23:01:11.770314+0000","last_undegraded":"2024-04-26T23:01:11.770314+0000","last_fullsized":"2024-04-26T23:01:11.770314+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:52.271211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775686+0000","last_change":"2024-04-26T23:01:04.712783+0000","last_active":"2024-04-26T23:01:11.775686+0000","last_peered":"2024-04-26T23:01:11.775686+0000","last_clean":"2024-04-26T23:01:11.775686+0000","last_became_active":"2024-04-26T23:01:04.712436+0000","last_became_peered":"2024-04-26T23:01:04.712436+0000","last_unstale":"2024-04-26T23:01:11.775686+0000","last_undegraded":"2024-04-26T23:01:11.775686+0000","last_fullsized":"2024-04-26T23:01:11.775686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484177+0000","last_change":"2024-04-26T23:01:05.742581+0000","last_active":"2024-04-26T23:01:12.484177+0000","last_peered":"2024-04-26T23:01:12.484177+0000","last_clean":"2024-04-26T23:01:12.484177+0000","last_became_active":"2024-04-26T23:01:05.740918+0000","last_became_peered":"2024-04-26T23:01:05.740918+0000","last_unstale":"2024-04-26T23:01:12.484177+0000","last_undegraded":"2024-04-26T23:01:12.484177+0000","last_fullsized":"2024-04-26T23:01:12.484177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:35:08.020426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.488392+0000","last_change":"2024-04-26T23:01:09.798160+0000","last_active":"2024-04-26T23:01:12.488392+0000","last_peered":"2024-04-26T23:01:12.488392+0000","last_clean":"2024-04-26T23:01:12.488392+0000","last_became_active":"2024-04-26T23:01:09.797746+0000","last_became_peered":"2024-04-26T23:01:09.797746+0000","last_unstale":"2024-04-26T23:01:12.488392+0000","last_undegraded":"2024-04-26T23:01:12.488392+0000","last_fullsized":"2024-04-26T23:01:12.488392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:11.186360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770724+0000","last_change":"2024-04-26T23:01:04.723482+0000","last_active":"2024-04-26T23:01:11.770724+0000","last_peered":"2024-04-26T23:01:11.770724+0000","last_clean":"2024-04-26T23:01:11.770724+0000","last_became_active":"2024-04-26T23:01:04.722816+0000","last_became_peered":"2024-04-26T23:01:04.722816+0000","last_unstale":"2024-04-26T23:01:11.770724+0000","last_undegraded":"2024-04-26T23:01:11.770724+0000","last_fullsized":"2024-04-26T23:01:11.770724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882061+0000","last_change":"2024-04-26T23:01:05.745436+0000","last_active":"2024-04-26T23:01:11.882061+0000","last_peered":"2024-04-26T23:01:11.882061+0000","last_clean":"2024-04-26T23:01:11.882061+0000","last_became_active":"2024-04-26T23:01:05.745148+0000","last_became_peered":"2024-04-26T23:01:05.745148+0000","last_unstale":"2024-04-26T23:01:11.882061+0000","last_undegraded":"2024-04-26T23:01:11.882061+0000","last_fullsized":"2024-04-26T23:01:11.882061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:17.111974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771111+0000","last_change":"2024-04-26T23:01:09.795297+0000","last_active":"2024-04-26T23:01:11.771111+0000","last_peered":"2024-04-26T23:01:11.771111+0000","last_clean":"2024-04-26T23:01:11.771111+0000","last_became_active":"2024-04-26T23:01:09.794873+0000","last_became_peered":"2024-04-26T23:01:09.794873+0000","last_unstale":"2024-04-26T23:01:11.771111+0000","last_undegraded":"2024-04-26T23:01:11.771111+0000","last_fullsized":"2024-04-26T23:01:11.771111+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:16:59.104896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.773782+0000","last_change":"2024-04-26T23:01:05.713935+0000","last_active":"2024-04-26T23:01:11.773782+0000","last_peered":"2024-04-26T23:01:11.773782+0000","last_clean":"2024-04-26T23:01:11.773782+0000","last_became_active":"2024-04-26T23:01:05.713606+0000","last_became_peered":"2024-04-26T23:01:05.713606+0000","last_unstale":"2024-04-26T23:01:11.773782+0000","last_undegraded":"2024-04-26T23:01:11.773782+0000","last_fullsized":"2024-04-26T23:01:11.773782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:26.157667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487634+0000","last_change":"2024-04-26T23:01:05.722889+0000","last_active":"2024-04-26T23:01:12.487634+0000","last_peered":"2024-04-26T23:01:12.487634+0000","last_clean":"2024-04-26T23:01:12.487634+0000","last_became_active":"2024-04-26T23:01:05.721634+0000","last_became_peered":"2024-04-26T23:01:05.721634+0000","last_unstale":"2024-04-26T23:01:12.487634+0000","last_undegraded":"2024-04-26T23:01:12.487634+0000","last_fullsized":"2024-04-26T23:01:12.487634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:46.537111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484476+0000","last_change":"2024-04-26T23:01:05.749096+0000","last_active":"2024-04-26T23:01:12.484476+0000","last_peered":"2024-04-26T23:01:12.484476+0000","last_clean":"2024-04-26T23:01:12.484476+0000","last_became_active":"2024-04-26T23:01:05.747770+0000","last_became_peered":"2024-04-26T23:01:05.747770+0000","last_unstale":"2024-04-26T23:01:12.484476+0000","last_undegraded":"2024-04-26T23:01:12.484476+0000","last_fullsized":"2024-04-26T23:01:12.484476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:57.106236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483913+0000","last_change":"2024-04-26T23:01:05.747157+0000","last_active":"2024-04-26T23:01:12.483913+0000","last_peered":"2024-04-26T23:01:12.483913+0000","last_clean":"2024-04-26T23:01:12.483913+0000","last_became_active":"2024-04-26T23:01:05.746805+0000","last_became_peered":"2024-04-26T23:01:05.746805+0000","last_unstale":"2024-04-26T23:01:12.483913+0000","last_undegraded":"2024-04-26T23:01:12.483913+0000","last_fullsized":"2024-04-26T23:01:12.483913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:07.115233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770959+0000","last_change":"2024-04-26T23:01:05.747433+0000","last_active":"2024-04-26T23:01:11.770959+0000","last_peered":"2024-04-26T23:01:11.770959+0000","last_clean":"2024-04-26T23:01:11.770959+0000","last_became_active":"2024-04-26T23:01:05.747108+0000","last_became_peered":"2024-04-26T23:01:05.747108+0000","last_unstale":"2024-04-26T23:01:11.770959+0000","last_undegraded":"2024-04-26T23:01:11.770959+0000","last_fullsized":"2024-04-26T23:01:11.770959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:57.106236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484633+0000","last_change":"2024-04-26T23:01:05.753692+0000","last_active":"2024-04-26T23:01:12.484633+0000","last_peered":"2024-04-26T23:01:12.484633+0000","last_clean":"2024-04-26T23:01:12.484633+0000","last_became_active":"2024-04-26T23:01:05.752474+0000","last_became_peered":"2024-04-26T23:01:05.752474+0000","last_unstale":"2024-04-26T23:01:12.484633+0000","last_undegraded":"2024-04-26T23:01:12.484633+0000","last_fullsized":"2024-04-26T23:01:12.484633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778393+0000","last_change":"2024-04-26T23:01:05.731062+0000","last_active":"2024-04-26T23:01:11.778393+0000","last_peered":"2024-04-26T23:01:11.778393+0000","last_clean":"2024-04-26T23:01:11.778393+0000","last_became_active":"2024-04-26T23:01:05.727960+0000","last_became_peered":"2024-04-26T23:01:05.727960+0000","last_unstale":"2024-04-26T23:01:11.778393+0000","last_undegraded":"2024-04-26T23:01:11.778393+0000","last_fullsized":"2024-04-26T23:01:11.778393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771192+0000","last_change":"2024-04-26T23:01:05.750232+0000","last_active":"2024-04-26T23:01:11.771192+0000","last_peered":"2024-04-26T23:01:11.771192+0000","last_clean":"2024-04-26T23:01:11.771192+0000","last_became_active":"2024-04-26T23:01:05.749885+0000","last_became_peered":"2024-04-26T23:01:05.749885+0000","last_unstale":"2024-04-26T23:01:11.771192+0000","last_undegraded":"2024-04-26T23:01:11.771192+0000","last_fullsized":"2024-04-26T23:01:11.771192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483726+0000","last_change":"2024-04-26T23:01:05.749914+0000","last_active":"2024-04-26T23:01:12.483726+0000","last_peered":"2024-04-26T23:01:12.483726+0000","last_clean":"2024-04-26T23:01:12.483726+0000","last_became_active":"2024-04-26T23:01:05.745797+0000","last_became_peered":"2024-04-26T23:01:05.745797+0000","last_unstale":"2024-04-26T23:01:12.483726+0000","last_undegraded":"2024-04-26T23:01:12.483726+0000","last_fullsized":"2024-04-26T23:01:12.483726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770217+0000","last_change":"2024-04-26T23:01:05.750181+0000","last_active":"2024-04-26T23:01:11.770217+0000","last_peered":"2024-04-26T23:01:11.770217+0000","last_clean":"2024-04-26T23:01:11.770217+0000","last_became_active":"2024-04-26T23:01:05.749810+0000","last_became_peered":"2024-04-26T23:01:05.749810+0000","last_unstale":"2024-04-26T23:01:11.770217+0000","last_undegraded":"2024-04-26T23:01:11.770217+0000","last_fullsized":"2024-04-26T23:01:11.770217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484216+0000","last_change":"2024-04-26T23:01:05.730857+0000","last_active":"2024-04-26T23:01:12.484216+0000","last_peered":"2024-04-26T23:01:12.484216+0000","last_clean":"2024-04-26T23:01:12.484216+0000","last_became_active":"2024-04-26T23:01:05.730543+0000","last_became_peered":"2024-04-26T23:01:05.730543+0000","last_unstale":"2024-04-26T23:01:12.484216+0000","last_undegraded":"2024-04-26T23:01:12.484216+0000","last_fullsized":"2024-04-26T23:01:12.484216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.482980+0000","last_change":"2024-04-26T23:01:05.741407+0000","last_active":"2024-04-26T23:01:12.482980+0000","last_peered":"2024-04-26T23:01:12.482980+0000","last_clean":"2024-04-26T23:01:12.482980+0000","last_became_active":"2024-04-26T23:01:05.740354+0000","last_became_peered":"2024-04-26T23:01:05.740354+0000","last_unstale":"2024-04-26T23:01:12.482980+0000","last_undegraded":"2024-04-26T23:01:12.482980+0000","last_fullsized":"2024-04-26T23:01:12.482980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:24:06.318595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777299+0000","last_change":"2024-04-26T23:01:05.730302+0000","last_active":"2024-04-26T23:01:11.777299+0000","last_peered":"2024-04-26T23:01:11.777299+0000","last_clean":"2024-04-26T23:01:11.777299+0000","last_became_active":"2024-04-26T23:01:05.729149+0000","last_became_peered":"2024-04-26T23:01:05.729149+0000","last_unstale":"2024-04-26T23:01:11.777299+0000","last_undegraded":"2024-04-26T23:01:11.777299+0000","last_fullsized":"2024-04-26T23:01:11.777299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776184+0000","last_change":"2024-04-26T23:01:05.731828+0000","last_active":"2024-04-26T23:01:11.776184+0000","last_peered":"2024-04-26T23:01:11.776184+0000","last_clean":"2024-04-26T23:01:11.776184+0000","last_became_active":"2024-04-26T23:01:05.731537+0000","last_became_peered":"2024-04-26T23:01:05.731537+0000","last_unstale":"2024-04-26T23:01:11.776184+0000","last_undegraded":"2024-04-26T23:01:11.776184+0000","last_fullsized":"2024-04-26T23:01:11.776184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:59:57.750603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487750+0000","last_change":"2024-04-26T23:01:05.726729+0000","last_active":"2024-04-26T23:01:12.487750+0000","last_peered":"2024-04-26T23:01:12.487750+0000","last_clean":"2024-04-26T23:01:12.487750+0000","last_became_active":"2024-04-26T23:01:05.726364+0000","last_became_peered":"2024-04-26T23:01:05.726364+0000","last_unstale":"2024-04-26T23:01:12.487750+0000","last_undegraded":"2024-04-26T23:01:12.487750+0000","last_fullsized":"2024-04-26T23:01:12.487750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:17.111974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485129+0000","last_change":"2024-04-26T23:01:05.728806+0000","last_active":"2024-04-26T23:01:12.485129+0000","last_peered":"2024-04-26T23:01:12.485129+0000","last_clean":"2024-04-26T23:01:12.485129+0000","last_became_active":"2024-04-26T23:01:05.728418+0000","last_became_peered":"2024-04-26T23:01:05.728418+0000","last_unstale":"2024-04-26T23:01:12.485129+0000","last_undegraded":"2024-04-26T23:01:12.485129+0000","last_fullsized":"2024-04-26T23:01:12.485129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:40.686396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483463+0000","last_change":"2024-04-26T23:01:05.721739+0000","last_active":"2024-04-26T23:01:12.483463+0000","last_peered":"2024-04-26T23:01:12.483463+0000","last_clean":"2024-04-26T23:01:12.483463+0000","last_became_active":"2024-04-26T23:01:05.721409+0000","last_became_peered":"2024-04-26T23:01:05.721409+0000","last_unstale":"2024-04-26T23:01:12.483463+0000","last_undegraded":"2024-04-26T23:01:12.483463+0000","last_fullsized":"2024-04-26T23:01:12.483463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483935+0000","last_change":"2024-04-26T23:01:05.739575+0000","last_active":"2024-04-26T23:01:12.483935+0000","last_peered":"2024-04-26T23:01:12.483935+0000","last_clean":"2024-04-26T23:01:12.483935+0000","last_became_active":"2024-04-26T23:01:05.734760+0000","last_became_peered":"2024-04-26T23:01:05.734760+0000","last_unstale":"2024-04-26T23:01:12.483935+0000","last_undegraded":"2024-04-26T23:01:12.483935+0000","last_fullsized":"2024-04-26T23:01:12.483935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483111+0000","last_change":"2024-04-26T23:01:05.732699+0000","last_active":"2024-04-26T23:01:12.483111+0000","last_peered":"2024-04-26T23:01:12.483111+0000","last_clean":"2024-04-26T23:01:12.483111+0000","last_became_active":"2024-04-26T23:01:05.732371+0000","last_became_peered":"2024-04-26T23:01:05.732371+0000","last_unstale":"2024-04-26T23:01:12.483111+0000","last_undegraded":"2024-04-26T23:01:12.483111+0000","last_fullsized":"2024-04-26T23:01:12.483111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770589+0000","last_change":"2024-04-26T23:01:05.745445+0000","last_active":"2024-04-26T23:01:11.770589+0000","last_peered":"2024-04-26T23:01:11.770589+0000","last_clean":"2024-04-26T23:01:11.770589+0000","last_became_active":"2024-04-26T23:01:05.742977+0000","last_became_peered":"2024-04-26T23:01:05.742977+0000","last_unstale":"2024-04-26T23:01:11.770589+0000","last_undegraded":"2024-04-26T23:01:11.770589+0000","last_fullsized":"2024-04-26T23:01:11.770589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882409+0000","last_change":"2024-04-26T23:01:05.739107+0000","last_active":"2024-04-26T23:01:11.882409+0000","last_peered":"2024-04-26T23:01:11.882409+0000","last_clean":"2024-04-26T23:01:11.882409+0000","last_became_active":"2024-04-26T23:01:05.738786+0000","last_became_peered":"2024-04-26T23:01:05.738786+0000","last_unstale":"2024-04-26T23:01:11.882409+0000","last_undegraded":"2024-04-26T23:01:11.882409+0000","last_fullsized":"2024-04-26T23:01:11.882409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:58.394848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778055+0000","last_change":"2024-04-26T23:01:05.710270+0000","last_active":"2024-04-26T23:01:11.778055+0000","last_peered":"2024-04-26T23:01:11.778055+0000","last_clean":"2024-04-26T23:01:11.778055+0000","last_became_active":"2024-04-26T23:01:05.709943+0000","last_became_peered":"2024-04-26T23:01:05.709943+0000","last_unstale":"2024-04-26T23:01:11.778055+0000","last_undegraded":"2024-04-26T23:01:11.778055+0000","last_fullsized":"2024-04-26T23:01:11.778055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778339+0000","last_change":"2024-04-26T23:01:05.717127+0000","last_active":"2024-04-26T23:01:11.778339+0000","last_peered":"2024-04-26T23:01:11.778339+0000","last_clean":"2024-04-26T23:01:11.778339+0000","last_became_active":"2024-04-26T23:01:05.716647+0000","last_became_peered":"2024-04-26T23:01:05.716647+0000","last_unstale":"2024-04-26T23:01:11.778339+0000","last_undegraded":"2024-04-26T23:01:11.778339+0000","last_fullsized":"2024-04-26T23:01:11.778339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776723+0000","last_change":"2024-04-26T23:01:05.730733+0000","last_active":"2024-04-26T23:01:11.776723+0000","last_peered":"2024-04-26T23:01:11.776723+0000","last_clean":"2024-04-26T23:01:11.776723+0000","last_became_active":"2024-04-26T23:01:05.727115+0000","last_became_peered":"2024-04-26T23:01:05.727115+0000","last_unstale":"2024-04-26T23:01:11.776723+0000","last_undegraded":"2024-04-26T23:01:11.776723+0000","last_fullsized":"2024-04-26T23:01:11.776723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:40.686396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769732+0000","last_change":"2024-04-26T23:01:05.737620+0000","last_active":"2024-04-26T23:01:11.769732+0000","last_peered":"2024-04-26T23:01:11.769732+0000","last_clean":"2024-04-26T23:01:11.769732+0000","last_became_active":"2024-04-26T23:01:05.737220+0000","last_became_peered":"2024-04-26T23:01:05.737220+0000","last_unstale":"2024-04-26T23:01:11.769732+0000","last_undegraded":"2024-04-26T23:01:11.769732+0000","last_fullsized":"2024-04-26T23:01:11.769732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:07.115233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777054+0000","last_change":"2024-04-26T23:01:05.731417+0000","last_active":"2024-04-26T23:01:11.777054+0000","last_peered":"2024-04-26T23:01:11.777054+0000","last_clean":"2024-04-26T23:01:11.777054+0000","last_became_active":"2024-04-26T23:01:05.730556+0000","last_became_peered":"2024-04-26T23:01:05.730556+0000","last_unstale":"2024-04-26T23:01:11.777054+0000","last_undegraded":"2024-04-26T23:01:11.777054+0000","last_fullsized":"2024-04-26T23:01:11.777054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:13.482334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483844+0000","last_change":"2024-04-26T23:01:05.721675+0000","last_active":"2024-04-26T23:01:12.483844+0000","last_peered":"2024-04-26T23:01:12.483844+0000","last_clean":"2024-04-26T23:01:12.483844+0000","last_became_active":"2024-04-26T23:01:05.721363+0000","last_became_peered":"2024-04-26T23:01:05.721363+0000","last_unstale":"2024-04-26T23:01:12.483844+0000","last_undegraded":"2024-04-26T23:01:12.483844+0000","last_fullsized":"2024-04-26T23:01:12.483844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:13.482334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776212+0000","last_change":"2024-04-26T23:01:05.729610+0000","last_active":"2024-04-26T23:01:11.776212+0000","last_peered":"2024-04-26T23:01:11.776212+0000","last_clean":"2024-04-26T23:01:11.776212+0000","last_became_active":"2024-04-26T23:01:05.729307+0000","last_became_peered":"2024-04-26T23:01:05.729307+0000","last_unstale":"2024-04-26T23:01:11.776212+0000","last_undegraded":"2024-04-26T23:01:11.776212+0000","last_fullsized":"2024-04-26T23:01:11.776212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:57.106236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484169+0000","last_change":"2024-04-26T23:01:05.753873+0000","last_active":"2024-04-26T23:01:12.484169+0000","last_peered":"2024-04-26T23:01:12.484169+0000","last_clean":"2024-04-26T23:01:12.484169+0000","last_became_active":"2024-04-26T23:01:05.753603+0000","last_became_peered":"2024-04-26T23:01:05.753603+0000","last_unstale":"2024-04-26T23:01:12.484169+0000","last_undegraded":"2024-04-26T23:01:12.484169+0000","last_fullsized":"2024-04-26T23:01:12.484169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:22:15.301548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484214+0000","last_change":"2024-04-26T23:01:05.720420+0000","last_active":"2024-04-26T23:01:12.484214+0000","last_peered":"2024-04-26T23:01:12.484214+0000","last_clean":"2024-04-26T23:01:12.484214+0000","last_became_active":"2024-04-26T23:01:05.720114+0000","last_became_peered":"2024-04-26T23:01:05.720114+0000","last_unstale":"2024-04-26T23:01:12.484214+0000","last_undegraded":"2024-04-26T23:01:12.484214+0000","last_fullsized":"2024-04-26T23:01:12.484214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486621+0000","last_change":"2024-04-26T23:01:05.730995+0000","last_active":"2024-04-26T23:01:12.486621+0000","last_peered":"2024-04-26T23:01:12.486621+0000","last_clean":"2024-04-26T23:01:12.486621+0000","last_became_active":"2024-04-26T23:01:05.730644+0000","last_became_peered":"2024-04-26T23:01:05.730644+0000","last_unstale":"2024-04-26T23:01:12.486621+0000","last_undegraded":"2024-04-26T23:01:12.486621+0000","last_fullsized":"2024-04-26T23:01:12.486621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776188+0000","last_change":"2024-04-26T23:01:05.727391+0000","last_active":"2024-04-26T23:01:11.776188+0000","last_peered":"2024-04-26T23:01:11.776188+0000","last_clean":"2024-04-26T23:01:11.776188+0000","last_became_active":"2024-04-26T23:01:05.726729+0000","last_became_peered":"2024-04-26T23:01:05.726729+0000","last_unstale":"2024-04-26T23:01:11.776188+0000","last_undegraded":"2024-04-26T23:01:11.776188+0000","last_fullsized":"2024-04-26T23:01:11.776188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:24.824202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:01:40.969 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:01:41.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.209+0000 7f503ea00640 1 Processor -- start 2024-04-26T23:01:41.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.210+0000 7f503ea00640 1 -- start start 2024-04-26T23:01:41.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.211+0000 7f503ea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50380b4240 0x7f503806d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.211+0000 7f503ea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f503806d890 0x7f50380c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.211+0000 7f503ea00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5038163a40 con 0x7f50380c60e0 2024-04-26T23:01:41.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.211+0000 7f503ea00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5038167130 con 0x7f503806d890 2024-04-26T23:01:41.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.211+0000 7f503ea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5038167410 con 0x7f50380b4240 2024-04-26T23:01:41.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.212+0000 7f503d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50380b4240 0x7f503806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.212+0000 7f503cc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f503806d890 0x7f50380c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.212+0000 7f503e000640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f50380c60e0 0x7f50380b2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:55464/0 (socket says 172.21.15.40:55464) 2024-04-26T23:01:41.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.212+0000 7f503d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50380b4240 0x7f503806d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:47140/0 (socket says 172.21.15.40:47140) 2024-04-26T23:01:41.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.212+0000 7f503e000640 1 -- 172.21.15.40:0/2952044964 learned_addr learned my addr 172.21.15.40:0/2952044964 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:41.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.213+0000 7f503d600640 1 -- 172.21.15.40:0/2952044964 >> v2:172.21.15.103:3300/0 conn(0x7f503806d890 msgr2=0x7f50380c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:41.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.213+0000 7f503d600640 1 --2- 172.21.15.40:0/2952044964 >> v2:172.21.15.103:3300/0 conn(0x7f503806d890 0x7f50380c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.213+0000 7f503d600640 1 -- 172.21.15.40:0/2952044964 >> v1:172.21.15.103:6789/0 conn(0x7f50380c60e0 legacy=0x7f50380b2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:41.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.213+0000 7f5037e00640 1 -- 172.21.15.40:0/2952044964 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2704795294 0 0) 0x7f5038163a40 con 0x7f50380c60e0 2024-04-26T23:01:41.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.213+0000 7f503d600640 1 -- 172.21.15.40:0/2952044964 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50381676f0 con 0x7f50380b4240 2024-04-26T23:01:41.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.215+0000 7f503d600640 1 --2- 172.21.15.40:0/2952044964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50380b4240 0x7f503806d350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5028009d90 tx=0x7f502802a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfe6e55e4020ccea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:41.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.215+0000 7f5037e00640 1 -- 172.21.15.40:0/2952044964 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028007740 con 0x7f50380b4240 2024-04-26T23:01:41.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.215+0000 7f5037e00640 1 -- 172.21.15.40:0/2952044964 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50280078e0 con 0x7f50380b4240 2024-04-26T23:01:41.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.215+0000 7f5037e00640 1 -- 172.21.15.40:0/2952044964 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028007cb0 con 0x7f50380b4240 2024-04-26T23:01:41.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.216+0000 7f503ea00640 1 -- 172.21.15.40:0/2952044964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50380b4240 msgr2=0x7f503806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:41.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.216+0000 7f503ea00640 1 --2- 172.21.15.40:0/2952044964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50380b4240 0x7f503806d350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5028009d90 tx=0x7f502802a6f0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.216+0000 7f503ea00640 1 -- 172.21.15.40:0/2952044964 shutdown_connections 2024-04-26T23:01:41.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.216+0000 7f503ea00640 1 --2- 172.21.15.40:0/2952044964 >> v2:172.21.15.103:3300/0 conn(0x7f503806d890 0x7f50380c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.216+0000 7f503ea00640 1 --2- 172.21.15.40:0/2952044964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50380b4240 0x7f503806d350 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.216+0000 7f503ea00640 1 -- 172.21.15.40:0/2952044964 >> 172.21.15.40:0/2952044964 conn(0x7f50380d2bd0 msgr2=0x7f50380d2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:41.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.216+0000 7f503ea00640 1 -- 172.21.15.40:0/2952044964 shutdown_connections 2024-04-26T23:01:41.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.216+0000 7f503ea00640 1 -- 172.21.15.40:0/2952044964 wait complete. 2024-04-26T23:01:41.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.218+0000 7f503ea00640 1 Processor -- start 2024-04-26T23:01:41.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.218+0000 7f503ea00640 1 -- start start 2024-04-26T23:01:41.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.219+0000 7f503ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f503806d890 0x7f5038162640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.219+0000 7f503ea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50380b4240 0x7f5038166620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.219+0000 7f503ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50380c60e0 0x7f50380b9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.219+0000 7f503ea00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5038167f90 con 0x7f50380c60e0 2024-04-26T23:01:41.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.219+0000 7f503ea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5038167cb0 con 0x7f503806d890 2024-04-26T23:01:41.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.219+0000 7f503ea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f50380d1a90 con 0x7f50380b4240 2024-04-26T23:01:41.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.220+0000 7f503e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50380c60e0 0x7f50380b9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.220+0000 7f503e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50380c60e0 0x7f50380b9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39032/0 (socket says 172.21.15.40:39032) 2024-04-26T23:01:41.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.220+0000 7f503e000640 1 -- 172.21.15.40:0/1816313161 learned_addr learned my addr 172.21.15.40:0/1816313161 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:41.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.220+0000 7f503cc00640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50380b4240 0x7f5038166620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.220+0000 7f503d600640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f503806d890 0x7f5038162640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.220+0000 7f503e000640 1 -- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50380b4240 msgr2=0x7f5038166620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:41.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.220+0000 7f503e000640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50380b4240 0x7f5038166620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.220+0000 7f503e000640 1 -- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f503806d890 msgr2=0x7f5038162640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:41.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.220+0000 7f503e000640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f503806d890 0x7f5038162640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.220+0000 7f503e000640 1 -- 172.21.15.40:0/1816313161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50380c6ac0 con 0x7f50380c60e0 2024-04-26T23:01:41.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.221+0000 7f503e000640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50380c60e0 0x7f50380b9930 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f503000c410 tx=0x7f503000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:41.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.221+0000 7f5035600640 1 -- 172.21.15.40:0/1816313161 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50300074e0 con 0x7f50380c60e0 2024-04-26T23:01:41.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.221+0000 7f5035600640 1 -- 172.21.15.40:0/1816313161 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50300047e0 con 0x7f50380c60e0 2024-04-26T23:01:41.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.221+0000 7f5035600640 1 -- 172.21.15.40:0/1816313161 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5030004c10 con 0x7f50380c60e0 2024-04-26T23:01:41.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.221+0000 7f503ea00640 1 -- 172.21.15.40:0/1816313161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50380c6da0 con 0x7f50380c60e0 2024-04-26T23:01:41.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.221+0000 7f503ea00640 1 -- 172.21.15.40:0/1816313161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50380c7640 con 0x7f50380c60e0 2024-04-26T23:01:41.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.222+0000 7f503ea00640 1 -- 172.21.15.40:0/1816313161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50380c7840 con 0x7f50380c60e0 2024-04-26T23:01:41.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.223+0000 7f5035600640 1 -- 172.21.15.40:0/1816313161 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5030011070 con 0x7f50380c60e0 2024-04-26T23:01:41.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.225+0000 7f5035600640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f501806c920 0x7f501806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.225+0000 7f5035600640 1 -- 172.21.15.40:0/1816313161 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f503008b910 con 0x7f50380c60e0 2024-04-26T23:01:41.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.225+0000 7f503d600640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f501806c920 0x7f501806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.226+0000 7f503d600640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f501806c920 0x7f501806ede0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5028034000 tx=0x7f502802f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:41.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.228+0000 7f5035600640 1 -- 172.21.15.40:0/1816313161 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f50380c7840 con 0x7f50380c60e0 2024-04-26T23:01:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.421+0000 7f503ea00640 1 -- 172.21.15.40:0/1816313161 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5038069a80 con 0x7f501806c920 2024-04-26T23:01:41.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.446+0000 7f5035600640 1 -- 172.21.15.40:0/1816313161 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332535 (secure 0 0 0) 0x7f5038069a80 con 0x7f501806c920 2024-04-26T23:01:41.450 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:41.452 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:01:41.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.451+0000 7f503ea00640 1 -- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f501806c920 msgr2=0x7f501806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:41.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.451+0000 7f503ea00640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f501806c920 0x7f501806ede0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5028034000 tx=0x7f502802f000 comp rx=0 tx=0).stop 2024-04-26T23:01:41.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.451+0000 7f503ea00640 1 -- 172.21.15.40:0/1816313161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50380c60e0 msgr2=0x7f50380b9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:41.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.451+0000 7f503ea00640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50380c60e0 0x7f50380b9930 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f503000c410 tx=0x7f503000c8d0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.451+0000 7f503ea00640 1 -- 172.21.15.40:0/1816313161 shutdown_connections 2024-04-26T23:01:41.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.451+0000 7f503ea00640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f501806c920 0x7f501806ede0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.451+0000 7f503ea00640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50380c60e0 0x7f50380b9930 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.451+0000 7f503ea00640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50380b4240 0x7f5038166620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.451+0000 7f503ea00640 1 --2- 172.21.15.40:0/1816313161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f503806d890 0x7f5038162640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.451+0000 7f503ea00640 1 -- 172.21.15.40:0/1816313161 >> 172.21.15.40:0/1816313161 conn(0x7f50380d2bd0 msgr2=0x7f50380b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:41.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.451+0000 7f503ea00640 1 -- 172.21.15.40:0/1816313161 shutdown_connections 2024-04-26T23:01:41.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.451+0000 7f503ea00640 1 -- 172.21.15.40:0/1816313161 wait complete. 2024-04-26T23:01:41.470 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-04-26T23:01:41.154948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005459"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487297+0000","last_change":"2024-04-26T23:01:04.726404+0000","last_active":"2024-04-26T23:01:12.487297+0000","last_peered":"2024-04-26T23:01:12.487297+0000","last_clean":"2024-04-26T23:01:12.487297+0000","last_became_active":"2024-04-26T23:01:04.726160+0000","last_became_peered":"2024-04-26T23:01:04.726160+0000","last_unstale":"2024-04-26T23:01:12.487297+0000","last_undegraded":"2024-04-26T23:01:12.487297+0000","last_fullsized":"2024-04-26T23:01:12.487297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:21:22.581263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771048+0000","last_change":"2024-04-26T23:01:05.746741+0000","last_active":"2024-04-26T23:01:11.771048+0000","last_peered":"2024-04-26T23:01:11.771048+0000","last_clean":"2024-04-26T23:01:11.771048+0000","last_became_active":"2024-04-26T23:01:05.743495+0000","last_became_peered":"2024-04-26T23:01:05.743495+0000","last_unstale":"2024-04-26T23:01:11.771048+0000","last_undegraded":"2024-04-26T23:01:11.771048+0000","last_fullsized":"2024-04-26T23:01:11.771048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769795+0000","last_change":"2024-04-26T23:01:10.403533+0000","last_active":"2024-04-26T23:01:11.769795+0000","last_peered":"2024-04-26T23:01:11.769795+0000","last_clean":"2024-04-26T23:01:11.769795+0000","last_became_active":"2024-04-26T23:01:10.402972+0000","last_became_peered":"2024-04-26T23:01:10.402972+0000","last_unstale":"2024-04-26T23:01:11.769795+0000","last_undegraded":"2024-04-26T23:01:11.769795+0000","last_fullsized":"2024-04-26T23:01:11.769795+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:17.916674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771280+0000","last_change":"2024-04-26T23:01:04.715636+0000","last_active":"2024-04-26T23:01:11.771280+0000","last_peered":"2024-04-26T23:01:11.771280+0000","last_clean":"2024-04-26T23:01:11.771280+0000","last_became_active":"2024-04-26T23:01:04.715322+0000","last_became_peered":"2024-04-26T23:01:04.715322+0000","last_unstale":"2024-04-26T23:01:11.771280+0000","last_undegraded":"2024-04-26T23:01:11.771280+0000","last_fullsized":"2024-04-26T23:01:11.771280+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:55.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486825+0000","last_change":"2024-04-26T23:01:05.723149+0000","last_active":"2024-04-26T23:01:12.486825+0000","last_peered":"2024-04-26T23:01:12.486825+0000","last_clean":"2024-04-26T23:01:12.486825+0000","last_became_active":"2024-04-26T23:01:05.722809+0000","last_became_peered":"2024-04-26T23:01:05.722809+0000","last_unstale":"2024-04-26T23:01:12.486825+0000","last_undegraded":"2024-04-26T23:01:12.486825+0000","last_fullsized":"2024-04-26T23:01:12.486825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:26.157667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.881757+0000","last_change":"2024-04-26T23:01:09.805189+0000","last_active":"2024-04-26T23:01:11.881757+0000","last_peered":"2024-04-26T23:01:11.881757+0000","last_clean":"2024-04-26T23:01:11.881757+0000","last_became_active":"2024-04-26T23:01:09.804560+0000","last_became_peered":"2024-04-26T23:01:09.804560+0000","last_unstale":"2024-04-26T23:01:11.881757+0000","last_undegraded":"2024-04-26T23:01:11.881757+0000","last_fullsized":"2024-04-26T23:01:11.881757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:47.817790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775805+0000","last_change":"2024-04-26T23:01:04.734037+0000","last_active":"2024-04-26T23:01:11.775805+0000","last_peered":"2024-04-26T23:01:11.775805+0000","last_clean":"2024-04-26T23:01:11.775805+0000","last_became_active":"2024-04-26T23:01:04.733352+0000","last_became_peered":"2024-04-26T23:01:04.733352+0000","last_unstale":"2024-04-26T23:01:11.775805+0000","last_undegraded":"2024-04-26T23:01:11.775805+0000","last_fullsized":"2024-04-26T23:01:11.775805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:16.098138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484327+0000","last_change":"2024-04-26T23:01:05.738238+0000","last_active":"2024-04-26T23:01:12.484327+0000","last_peered":"2024-04-26T23:01:12.484327+0000","last_clean":"2024-04-26T23:01:12.484327+0000","last_became_active":"2024-04-26T23:01:05.737945+0000","last_became_peered":"2024-04-26T23:01:05.737945+0000","last_unstale":"2024-04-26T23:01:12.484327+0000","last_undegraded":"2024-04-26T23:01:12.484327+0000","last_fullsized":"2024-04-26T23:01:12.484327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:22.679803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.881869+0000","last_change":"2024-04-26T23:01:09.800139+0000","last_active":"2024-04-26T23:01:11.881869+0000","last_peered":"2024-04-26T23:01:11.881869+0000","last_clean":"2024-04-26T23:01:11.881869+0000","last_became_active":"2024-04-26T23:01:09.799214+0000","last_became_peered":"2024-04-26T23:01:09.799214+0000","last_unstale":"2024-04-26T23:01:11.881869+0000","last_undegraded":"2024-04-26T23:01:11.881869+0000","last_fullsized":"2024-04-26T23:01:11.881869+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:46:57.684555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776133+0000","last_change":"2024-04-26T23:01:04.735697+0000","last_active":"2024-04-26T23:01:11.776133+0000","last_peered":"2024-04-26T23:01:11.776133+0000","last_clean":"2024-04-26T23:01:11.776133+0000","last_became_active":"2024-04-26T23:01:04.735327+0000","last_became_peered":"2024-04-26T23:01:04.735327+0000","last_unstale":"2024-04-26T23:01:11.776133+0000","last_undegraded":"2024-04-26T23:01:11.776133+0000","last_fullsized":"2024-04-26T23:01:11.776133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:58.527150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484663+0000","last_change":"2024-04-26T23:01:05.729770+0000","last_active":"2024-04-26T23:01:12.484663+0000","last_peered":"2024-04-26T23:01:12.484663+0000","last_clean":"2024-04-26T23:01:12.484663+0000","last_became_active":"2024-04-26T23:01:05.727922+0000","last_became_peered":"2024-04-26T23:01:05.727922+0000","last_unstale":"2024-04-26T23:01:12.484663+0000","last_undegraded":"2024-04-26T23:01:12.484663+0000","last_fullsized":"2024-04-26T23:01:12.484663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:44:40.404454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882761+0000","last_change":"2024-04-26T23:01:09.804570+0000","last_active":"2024-04-26T23:01:11.882761+0000","last_peered":"2024-04-26T23:01:11.882761+0000","last_clean":"2024-04-26T23:01:11.882761+0000","last_became_active":"2024-04-26T23:01:09.803489+0000","last_became_peered":"2024-04-26T23:01:09.803489+0000","last_unstale":"2024-04-26T23:01:11.882761+0000","last_undegraded":"2024-04-26T23:01:11.882761+0000","last_fullsized":"2024-04-26T23:01:11.882761+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774588+0000","last_change":"2024-04-26T23:01:05.729351+0000","last_active":"2024-04-26T23:01:11.774588+0000","last_peered":"2024-04-26T23:01:11.774588+0000","last_clean":"2024-04-26T23:01:11.774588+0000","last_became_active":"2024-04-26T23:01:05.729024+0000","last_became_peered":"2024-04-26T23:01:05.729024+0000","last_unstale":"2024-04-26T23:01:11.774588+0000","last_undegraded":"2024-04-26T23:01:11.774588+0000","last_fullsized":"2024-04-26T23:01:11.774588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776686+0000","last_change":"2024-04-26T23:01:04.730260+0000","last_active":"2024-04-26T23:01:11.776686+0000","last_peered":"2024-04-26T23:01:11.776686+0000","last_clean":"2024-04-26T23:01:11.776686+0000","last_became_active":"2024-04-26T23:01:04.729209+0000","last_became_peered":"2024-04-26T23:01:04.729209+0000","last_unstale":"2024-04-26T23:01:11.776686+0000","last_undegraded":"2024-04-26T23:01:11.776686+0000","last_fullsized":"2024-04-26T23:01:11.776686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:45.523274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.883045+0000","last_change":"2024-04-26T23:01:09.796341+0000","last_active":"2024-04-26T23:01:11.883045+0000","last_peered":"2024-04-26T23:01:11.883045+0000","last_clean":"2024-04-26T23:01:11.883045+0000","last_became_active":"2024-04-26T23:01:09.795939+0000","last_became_peered":"2024-04-26T23:01:09.795939+0000","last_unstale":"2024-04-26T23:01:11.883045+0000","last_undegraded":"2024-04-26T23:01:11.883045+0000","last_fullsized":"2024-04-26T23:01:11.883045+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:09:55.216454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770614+0000","last_change":"2024-04-26T23:01:05.735100+0000","last_active":"2024-04-26T23:01:11.770614+0000","last_peered":"2024-04-26T23:01:11.770614+0000","last_clean":"2024-04-26T23:01:11.770614+0000","last_became_active":"2024-04-26T23:01:05.734138+0000","last_became_peered":"2024-04-26T23:01:05.734138+0000","last_unstale":"2024-04-26T23:01:11.770614+0000","last_undegraded":"2024-04-26T23:01:11.770614+0000","last_fullsized":"2024-04-26T23:01:11.770614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486466+0000","last_change":"2024-04-26T23:01:04.723885+0000","last_active":"2024-04-26T23:01:12.486466+0000","last_peered":"2024-04-26T23:01:12.486466+0000","last_clean":"2024-04-26T23:01:12.486466+0000","last_became_active":"2024-04-26T23:01:04.723597+0000","last_became_peered":"2024-04-26T23:01:04.723597+0000","last_unstale":"2024-04-26T23:01:12.486466+0000","last_undegraded":"2024-04-26T23:01:12.486466+0000","last_fullsized":"2024-04-26T23:01:12.486466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:45.523274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882064+0000","last_change":"2024-04-26T23:01:09.803721+0000","last_active":"2024-04-26T23:01:11.882064+0000","last_peered":"2024-04-26T23:01:11.882064+0000","last_clean":"2024-04-26T23:01:11.882064+0000","last_became_active":"2024-04-26T23:01:09.803061+0000","last_became_peered":"2024-04-26T23:01:09.803061+0000","last_unstale":"2024-04-26T23:01:11.882064+0000","last_undegraded":"2024-04-26T23:01:11.882064+0000","last_fullsized":"2024-04-26T23:01:11.882064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:41.540020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769949+0000","last_change":"2024-04-26T23:01:05.738344+0000","last_active":"2024-04-26T23:01:11.769949+0000","last_peered":"2024-04-26T23:01:11.769949+0000","last_clean":"2024-04-26T23:01:11.769949+0000","last_became_active":"2024-04-26T23:01:05.738082+0000","last_became_peered":"2024-04-26T23:01:05.738082+0000","last_unstale":"2024-04-26T23:01:11.769949+0000","last_undegraded":"2024-04-26T23:01:11.769949+0000","last_fullsized":"2024-04-26T23:01:11.769949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:26.157667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484208+0000","last_change":"2024-04-26T23:01:04.727252+0000","last_active":"2024-04-26T23:01:12.484208+0000","last_peered":"2024-04-26T23:01:12.484208+0000","last_clean":"2024-04-26T23:01:12.484208+0000","last_became_active":"2024-04-26T23:01:04.726980+0000","last_became_peered":"2024-04-26T23:01:04.726980+0000","last_unstale":"2024-04-26T23:01:12.484208+0000","last_undegraded":"2024-04-26T23:01:12.484208+0000","last_fullsized":"2024-04-26T23:01:12.484208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486354+0000","last_change":"2024-04-26T23:01:10.399273+0000","last_active":"2024-04-26T23:01:12.486354+0000","last_peered":"2024-04-26T23:01:12.486354+0000","last_clean":"2024-04-26T23:01:12.486354+0000","last_became_active":"2024-04-26T23:01:10.397193+0000","last_became_peered":"2024-04-26T23:01:10.397193+0000","last_unstale":"2024-04-26T23:01:12.486354+0000","last_undegraded":"2024-04-26T23:01:12.486354+0000","last_fullsized":"2024-04-26T23:01:12.486354+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882811+0000","last_change":"2024-04-26T23:01:05.734542+0000","last_active":"2024-04-26T23:01:11.882811+0000","last_peered":"2024-04-26T23:01:11.882811+0000","last_clean":"2024-04-26T23:01:11.882811+0000","last_became_active":"2024-04-26T23:01:05.734191+0000","last_became_peered":"2024-04-26T23:01:05.734191+0000","last_unstale":"2024-04-26T23:01:11.882811+0000","last_undegraded":"2024-04-26T23:01:11.882811+0000","last_fullsized":"2024-04-26T23:01:11.882811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:49.765162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774127+0000","last_change":"2024-04-26T23:01:04.732728+0000","last_active":"2024-04-26T23:01:11.774127+0000","last_peered":"2024-04-26T23:01:11.774127+0000","last_clean":"2024-04-26T23:01:11.774127+0000","last_became_active":"2024-04-26T23:01:04.732413+0000","last_became_peered":"2024-04-26T23:01:04.732413+0000","last_unstale":"2024-04-26T23:01:11.774127+0000","last_undegraded":"2024-04-26T23:01:11.774127+0000","last_fullsized":"2024-04-26T23:01:11.774127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882964+0000","last_change":"2024-04-26T23:01:09.800234+0000","last_active":"2024-04-26T23:01:11.882964+0000","last_peered":"2024-04-26T23:01:11.882964+0000","last_clean":"2024-04-26T23:01:11.882964+0000","last_became_active":"2024-04-26T23:01:09.799644+0000","last_became_peered":"2024-04-26T23:01:09.799644+0000","last_unstale":"2024-04-26T23:01:11.882964+0000","last_undegraded":"2024-04-26T23:01:11.882964+0000","last_fullsized":"2024-04-26T23:01:11.882964+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:37:23.404492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486201+0000","last_change":"2024-04-26T23:01:05.735371+0000","last_active":"2024-04-26T23:01:12.486201+0000","last_peered":"2024-04-26T23:01:12.486201+0000","last_clean":"2024-04-26T23:01:12.486201+0000","last_became_active":"2024-04-26T23:01:05.729667+0000","last_became_peered":"2024-04-26T23:01:05.729667+0000","last_unstale":"2024-04-26T23:01:12.486201+0000","last_undegraded":"2024-04-26T23:01:12.486201+0000","last_fullsized":"2024-04-26T23:01:12.486201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778387+0000","last_change":"2024-04-26T23:01:04.726450+0000","last_active":"2024-04-26T23:01:11.778387+0000","last_peered":"2024-04-26T23:01:11.778387+0000","last_clean":"2024-04-26T23:01:11.778387+0000","last_became_active":"2024-04-26T23:01:04.726023+0000","last_became_peered":"2024-04-26T23:01:04.726023+0000","last_unstale":"2024-04-26T23:01:11.778387+0000","last_undegraded":"2024-04-26T23:01:11.778387+0000","last_fullsized":"2024-04-26T23:01:11.778387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:45.523274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.881567+0000","last_change":"2024-04-26T23:01:10.401428+0000","last_active":"2024-04-26T23:01:11.881567+0000","last_peered":"2024-04-26T23:01:11.881567+0000","last_clean":"2024-04-26T23:01:11.881567+0000","last_became_active":"2024-04-26T23:01:10.400621+0000","last_became_peered":"2024-04-26T23:01:10.400621+0000","last_unstale":"2024-04-26T23:01:11.881567+0000","last_undegraded":"2024-04-26T23:01:11.881567+0000","last_fullsized":"2024-04-26T23:01:11.881567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:59.067859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771304+0000","last_change":"2024-04-26T23:01:05.748687+0000","last_active":"2024-04-26T23:01:11.771304+0000","last_peered":"2024-04-26T23:01:11.771304+0000","last_clean":"2024-04-26T23:01:11.771304+0000","last_became_active":"2024-04-26T23:01:05.744038+0000","last_became_peered":"2024-04-26T23:01:05.744038+0000","last_unstale":"2024-04-26T23:01:11.771304+0000","last_undegraded":"2024-04-26T23:01:11.771304+0000","last_fullsized":"2024-04-26T23:01:11.771304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:13.482334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486378+0000","last_change":"2024-04-26T23:01:04.718414+0000","last_active":"2024-04-26T23:01:12.486378+0000","last_peered":"2024-04-26T23:01:12.486378+0000","last_clean":"2024-04-26T23:01:12.486378+0000","last_became_active":"2024-04-26T23:01:04.717764+0000","last_became_peered":"2024-04-26T23:01:04.717764+0000","last_unstale":"2024-04-26T23:01:12.486378+0000","last_undegraded":"2024-04-26T23:01:12.486378+0000","last_fullsized":"2024-04-26T23:01:12.486378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769756+0000","last_change":"2024-04-26T23:01:10.386716+0000","last_active":"2024-04-26T23:01:11.769756+0000","last_peered":"2024-04-26T23:01:11.769756+0000","last_clean":"2024-04-26T23:01:11.769756+0000","last_became_active":"2024-04-26T23:01:10.385392+0000","last_became_peered":"2024-04-26T23:01:10.385392+0000","last_unstale":"2024-04-26T23:01:11.769756+0000","last_undegraded":"2024-04-26T23:01:11.769756+0000","last_fullsized":"2024-04-26T23:01:11.769756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:09:55.216454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485210+0000","last_change":"2024-04-26T23:01:05.728786+0000","last_active":"2024-04-26T23:01:12.485210+0000","last_peered":"2024-04-26T23:01:12.485210+0000","last_clean":"2024-04-26T23:01:12.485210+0000","last_became_active":"2024-04-26T23:01:05.728430+0000","last_became_peered":"2024-04-26T23:01:05.728430+0000","last_unstale":"2024-04-26T23:01:12.485210+0000","last_undegraded":"2024-04-26T23:01:12.485210+0000","last_fullsized":"2024-04-26T23:01:12.485210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776238+0000","last_change":"2024-04-26T23:01:04.734438+0000","last_active":"2024-04-26T23:01:11.776238+0000","last_peered":"2024-04-26T23:01:11.776238+0000","last_clean":"2024-04-26T23:01:11.776238+0000","last_became_active":"2024-04-26T23:01:04.733372+0000","last_became_peered":"2024-04-26T23:01:04.733372+0000","last_unstale":"2024-04-26T23:01:11.776238+0000","last_undegraded":"2024-04-26T23:01:11.776238+0000","last_fullsized":"2024-04-26T23:01:11.776238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:55.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483768+0000","last_change":"2024-04-26T23:01:10.391451+0000","last_active":"2024-04-26T23:01:12.483768+0000","last_peered":"2024-04-26T23:01:12.483768+0000","last_clean":"2024-04-26T23:01:12.483768+0000","last_became_active":"2024-04-26T23:01:10.389477+0000","last_became_peered":"2024-04-26T23:01:10.389477+0000","last_unstale":"2024-04-26T23:01:12.483768+0000","last_undegraded":"2024-04-26T23:01:12.483768+0000","last_fullsized":"2024-04-26T23:01:12.483768+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:46:57.684555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485051+0000","last_change":"2024-04-26T23:01:05.746004+0000","last_active":"2024-04-26T23:01:12.485051+0000","last_peered":"2024-04-26T23:01:12.485051+0000","last_clean":"2024-04-26T23:01:12.485051+0000","last_became_active":"2024-04-26T23:01:05.745576+0000","last_became_peered":"2024-04-26T23:01:05.745576+0000","last_unstale":"2024-04-26T23:01:12.485051+0000","last_undegraded":"2024-04-26T23:01:12.485051+0000","last_fullsized":"2024-04-26T23:01:12.485051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:06.384302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485393+0000","last_change":"2024-04-26T23:01:04.717206+0000","last_active":"2024-04-26T23:01:12.485393+0000","last_peered":"2024-04-26T23:01:12.485393+0000","last_clean":"2024-04-26T23:01:12.485393+0000","last_became_active":"2024-04-26T23:01:04.715952+0000","last_became_peered":"2024-04-26T23:01:04.715952+0000","last_unstale":"2024-04-26T23:01:12.485393+0000","last_undegraded":"2024-04-26T23:01:12.485393+0000","last_fullsized":"2024-04-26T23:01:12.485393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485772+0000","last_change":"2024-04-26T23:01:09.799096+0000","last_active":"2024-04-26T23:01:12.485772+0000","last_peered":"2024-04-26T23:01:12.485772+0000","last_clean":"2024-04-26T23:01:12.485772+0000","last_became_active":"2024-04-26T23:01:09.798586+0000","last_became_peered":"2024-04-26T23:01:09.798586+0000","last_unstale":"2024-04-26T23:01:12.485772+0000","last_undegraded":"2024-04-26T23:01:12.485772+0000","last_fullsized":"2024-04-26T23:01:12.485772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:30.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.883148+0000","last_change":"2024-04-26T23:01:05.732977+0000","last_active":"2024-04-26T23:01:11.883148+0000","last_peered":"2024-04-26T23:01:11.883148+0000","last_clean":"2024-04-26T23:01:11.883148+0000","last_became_active":"2024-04-26T23:01:05.732617+0000","last_became_peered":"2024-04-26T23:01:05.732617+0000","last_unstale":"2024-04-26T23:01:11.883148+0000","last_undegraded":"2024-04-26T23:01:11.883148+0000","last_fullsized":"2024-04-26T23:01:11.883148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:59.540986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484829+0000","last_change":"2024-04-26T23:01:04.730386+0000","last_active":"2024-04-26T23:01:12.484829+0000","last_peered":"2024-04-26T23:01:12.484829+0000","last_clean":"2024-04-26T23:01:12.484829+0000","last_became_active":"2024-04-26T23:01:04.727828+0000","last_became_peered":"2024-04-26T23:01:04.727828+0000","last_unstale":"2024-04-26T23:01:12.484829+0000","last_undegraded":"2024-04-26T23:01:12.484829+0000","last_fullsized":"2024-04-26T23:01:12.484829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485312+0000","last_change":"2024-04-26T23:01:09.805894+0000","last_active":"2024-04-26T23:01:12.485312+0000","last_peered":"2024-04-26T23:01:12.485312+0000","last_clean":"2024-04-26T23:01:12.485312+0000","last_became_active":"2024-04-26T23:01:09.803400+0000","last_became_peered":"2024-04-26T23:01:09.803400+0000","last_unstale":"2024-04-26T23:01:12.485312+0000","last_undegraded":"2024-04-26T23:01:12.485312+0000","last_fullsized":"2024-04-26T23:01:12.485312+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:46:57.684555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484288+0000","last_change":"2024-04-26T23:01:05.722407+0000","last_active":"2024-04-26T23:01:12.484288+0000","last_peered":"2024-04-26T23:01:12.484288+0000","last_clean":"2024-04-26T23:01:12.484288+0000","last_became_active":"2024-04-26T23:01:05.722106+0000","last_became_peered":"2024-04-26T23:01:05.722106+0000","last_unstale":"2024-04-26T23:01:12.484288+0000","last_undegraded":"2024-04-26T23:01:12.484288+0000","last_fullsized":"2024-04-26T23:01:12.484288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487617+0000","last_change":"2024-04-26T23:01:04.712172+0000","last_active":"2024-04-26T23:01:12.487617+0000","last_peered":"2024-04-26T23:01:12.487617+0000","last_clean":"2024-04-26T23:01:12.487617+0000","last_became_active":"2024-04-26T23:01:04.711777+0000","last_became_peered":"2024-04-26T23:01:04.711777+0000","last_unstale":"2024-04-26T23:01:12.487617+0000","last_undegraded":"2024-04-26T23:01:12.487617+0000","last_fullsized":"2024-04-26T23:01:12.487617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486748+0000","last_change":"2024-04-26T23:01:09.808963+0000","last_active":"2024-04-26T23:01:12.486748+0000","last_peered":"2024-04-26T23:01:12.486748+0000","last_clean":"2024-04-26T23:01:12.486748+0000","last_became_active":"2024-04-26T23:01:09.808470+0000","last_became_peered":"2024-04-26T23:01:09.808470+0000","last_unstale":"2024-04-26T23:01:12.486748+0000","last_undegraded":"2024-04-26T23:01:12.486748+0000","last_fullsized":"2024-04-26T23:01:12.486748+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.482634+0000","last_change":"2024-04-26T23:01:05.742128+0000","last_active":"2024-04-26T23:01:12.482634+0000","last_peered":"2024-04-26T23:01:12.482634+0000","last_clean":"2024-04-26T23:01:12.482634+0000","last_became_active":"2024-04-26T23:01:05.735959+0000","last_became_peered":"2024-04-26T23:01:05.735959+0000","last_unstale":"2024-04-26T23:01:12.482634+0000","last_undegraded":"2024-04-26T23:01:12.482634+0000","last_fullsized":"2024-04-26T23:01:12.482634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774374+0000","last_change":"2024-04-26T23:01:04.732093+0000","last_active":"2024-04-26T23:01:11.774374+0000","last_peered":"2024-04-26T23:01:11.774374+0000","last_clean":"2024-04-26T23:01:11.774374+0000","last_became_active":"2024-04-26T23:01:04.731804+0000","last_became_peered":"2024-04-26T23:01:04.731804+0000","last_unstale":"2024-04-26T23:01:11.774374+0000","last_undegraded":"2024-04-26T23:01:11.774374+0000","last_fullsized":"2024-04-26T23:01:11.774374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485713+0000","last_change":"2024-04-26T23:01:10.389206+0000","last_active":"2024-04-26T23:01:12.485713+0000","last_peered":"2024-04-26T23:01:12.485713+0000","last_clean":"2024-04-26T23:01:12.485713+0000","last_became_active":"2024-04-26T23:01:10.384213+0000","last_became_peered":"2024-04-26T23:01:10.384213+0000","last_unstale":"2024-04-26T23:01:12.485713+0000","last_undegraded":"2024-04-26T23:01:12.485713+0000","last_fullsized":"2024-04-26T23:01:12.485713+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770463+0000","last_change":"2024-04-26T23:01:05.744836+0000","last_active":"2024-04-26T23:01:11.770463+0000","last_peered":"2024-04-26T23:01:11.770463+0000","last_clean":"2024-04-26T23:01:11.770463+0000","last_became_active":"2024-04-26T23:01:05.744600+0000","last_became_peered":"2024-04-26T23:01:05.744600+0000","last_unstale":"2024-04-26T23:01:11.770463+0000","last_undegraded":"2024-04-26T23:01:11.770463+0000","last_fullsized":"2024-04-26T23:01:11.770463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484137+0000","last_change":"2024-04-26T23:01:04.729176+0000","last_active":"2024-04-26T23:01:12.484137+0000","last_peered":"2024-04-26T23:01:12.484137+0000","last_clean":"2024-04-26T23:01:12.484137+0000","last_became_active":"2024-04-26T23:01:04.722896+0000","last_became_peered":"2024-04-26T23:01:04.722896+0000","last_unstale":"2024-04-26T23:01:12.484137+0000","last_undegraded":"2024-04-26T23:01:12.484137+0000","last_fullsized":"2024-04-26T23:01:12.484137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:16.585834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":180,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774617+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:01:11.774617+0000","last_peered":"2024-04-26T23:01:11.774617+0000","last_clean":"2024-04-26T23:01:11.774617+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:01:11.774617+0000","last_undegraded":"2024-04-26T23:01:11.774617+0000","last_fullsized":"2024-04-26T23:01:11.774617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777714+0000","last_change":"2024-04-26T23:01:10.402293+0000","last_active":"2024-04-26T23:01:11.777714+0000","last_peered":"2024-04-26T23:01:11.777714+0000","last_clean":"2024-04-26T23:01:11.777714+0000","last_became_active":"2024-04-26T23:01:10.396954+0000","last_became_peered":"2024-04-26T23:01:10.396954+0000","last_unstale":"2024-04-26T23:01:11.777714+0000","last_undegraded":"2024-04-26T23:01:11.777714+0000","last_fullsized":"2024-04-26T23:01:11.777714+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:44.757523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775315+0000","last_change":"2024-04-26T23:01:05.731266+0000","last_active":"2024-04-26T23:01:11.775315+0000","last_peered":"2024-04-26T23:01:11.775315+0000","last_clean":"2024-04-26T23:01:11.775315+0000","last_became_active":"2024-04-26T23:01:05.728530+0000","last_became_peered":"2024-04-26T23:01:05.728530+0000","last_unstale":"2024-04-26T23:01:11.775315+0000","last_undegraded":"2024-04-26T23:01:11.775315+0000","last_fullsized":"2024-04-26T23:01:11.775315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:44:40.404454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.488463+0000","last_change":"2024-04-26T23:01:04.728315+0000","last_active":"2024-04-26T23:01:12.488463+0000","last_peered":"2024-04-26T23:01:12.488463+0000","last_clean":"2024-04-26T23:01:12.488463+0000","last_became_active":"2024-04-26T23:01:04.727609+0000","last_became_peered":"2024-04-26T23:01:04.727609+0000","last_unstale":"2024-04-26T23:01:12.488463+0000","last_undegraded":"2024-04-26T23:01:12.488463+0000","last_fullsized":"2024-04-26T23:01:12.488463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775771+0000","last_change":"2024-04-26T23:01:10.391227+0000","last_active":"2024-04-26T23:01:11.775771+0000","last_peered":"2024-04-26T23:01:11.775771+0000","last_clean":"2024-04-26T23:01:11.775771+0000","last_became_active":"2024-04-26T23:01:10.387715+0000","last_became_peered":"2024-04-26T23:01:10.387715+0000","last_unstale":"2024-04-26T23:01:11.775771+0000","last_undegraded":"2024-04-26T23:01:11.775771+0000","last_fullsized":"2024-04-26T23:01:11.775771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:28.895329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484465+0000","last_change":"2024-04-26T23:01:05.753852+0000","last_active":"2024-04-26T23:01:12.484465+0000","last_peered":"2024-04-26T23:01:12.484465+0000","last_clean":"2024-04-26T23:01:12.484465+0000","last_became_active":"2024-04-26T23:01:05.753607+0000","last_became_peered":"2024-04-26T23:01:05.753607+0000","last_unstale":"2024-04-26T23:01:12.484465+0000","last_undegraded":"2024-04-26T23:01:12.484465+0000","last_fullsized":"2024-04-26T23:01:12.484465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:59:57.750603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485129+0000","last_change":"2024-04-26T23:01:04.713352+0000","last_active":"2024-04-26T23:01:12.485129+0000","last_peered":"2024-04-26T23:01:12.485129+0000","last_clean":"2024-04-26T23:01:12.485129+0000","last_became_active":"2024-04-26T23:01:04.712352+0000","last_became_peered":"2024-04-26T23:01:04.712352+0000","last_unstale":"2024-04-26T23:01:12.485129+0000","last_undegraded":"2024-04-26T23:01:12.485129+0000","last_fullsized":"2024-04-26T23:01:12.485129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:55.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484995+0000","last_change":"2024-04-26T23:01:09.803857+0000","last_active":"2024-04-26T23:01:12.484995+0000","last_peered":"2024-04-26T23:01:12.484995+0000","last_clean":"2024-04-26T23:01:12.484995+0000","last_became_active":"2024-04-26T23:01:09.799482+0000","last_became_peered":"2024-04-26T23:01:09.799482+0000","last_unstale":"2024-04-26T23:01:12.484995+0000","last_undegraded":"2024-04-26T23:01:12.484995+0000","last_fullsized":"2024-04-26T23:01:12.484995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:41.540020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776723+0000","last_change":"2024-04-26T23:01:05.741597+0000","last_active":"2024-04-26T23:01:11.776723+0000","last_peered":"2024-04-26T23:01:11.776723+0000","last_clean":"2024-04-26T23:01:11.776723+0000","last_became_active":"2024-04-26T23:01:05.741308+0000","last_became_peered":"2024-04-26T23:01:05.741308+0000","last_unstale":"2024-04-26T23:01:11.776723+0000","last_undegraded":"2024-04-26T23:01:11.776723+0000","last_fullsized":"2024-04-26T23:01:11.776723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:54.996731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487896+0000","last_change":"2024-04-26T23:01:04.720208+0000","last_active":"2024-04-26T23:01:12.487896+0000","last_peered":"2024-04-26T23:01:12.487896+0000","last_clean":"2024-04-26T23:01:12.487896+0000","last_became_active":"2024-04-26T23:01:04.719867+0000","last_became_peered":"2024-04-26T23:01:04.719867+0000","last_unstale":"2024-04-26T23:01:12.487896+0000","last_undegraded":"2024-04-26T23:01:12.487896+0000","last_fullsized":"2024-04-26T23:01:12.487896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:16.585834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483634+0000","last_change":"2024-04-26T23:01:10.396595+0000","last_active":"2024-04-26T23:01:12.483634+0000","last_peered":"2024-04-26T23:01:12.483634+0000","last_clean":"2024-04-26T23:01:12.483634+0000","last_became_active":"2024-04-26T23:01:10.395189+0000","last_became_peered":"2024-04-26T23:01:10.395189+0000","last_unstale":"2024-04-26T23:01:12.483634+0000","last_undegraded":"2024-04-26T23:01:12.483634+0000","last_fullsized":"2024-04-26T23:01:12.483634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776069+0000","last_change":"2024-04-26T23:01:05.733520+0000","last_active":"2024-04-26T23:01:11.776069+0000","last_peered":"2024-04-26T23:01:11.776069+0000","last_clean":"2024-04-26T23:01:11.776069+0000","last_became_active":"2024-04-26T23:01:05.733170+0000","last_became_peered":"2024-04-26T23:01:05.733170+0000","last_unstale":"2024-04-26T23:01:11.776069+0000","last_undegraded":"2024-04-26T23:01:11.776069+0000","last_fullsized":"2024-04-26T23:01:11.776069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:06.384302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.883050+0000","last_change":"2024-04-26T23:01:04.717930+0000","last_active":"2024-04-26T23:01:11.883050+0000","last_peered":"2024-04-26T23:01:11.883050+0000","last_clean":"2024-04-26T23:01:11.883050+0000","last_became_active":"2024-04-26T23:01:04.717576+0000","last_became_peered":"2024-04-26T23:01:04.717576+0000","last_unstale":"2024-04-26T23:01:11.883050+0000","last_undegraded":"2024-04-26T23:01:11.883050+0000","last_fullsized":"2024-04-26T23:01:11.883050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:11:48.078530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882003+0000","last_change":"2024-04-26T23:01:09.797064+0000","last_active":"2024-04-26T23:01:11.882003+0000","last_peered":"2024-04-26T23:01:11.882003+0000","last_clean":"2024-04-26T23:01:11.882003+0000","last_became_active":"2024-04-26T23:01:09.796589+0000","last_became_peered":"2024-04-26T23:01:09.796589+0000","last_unstale":"2024-04-26T23:01:11.882003+0000","last_undegraded":"2024-04-26T23:01:11.882003+0000","last_fullsized":"2024-04-26T23:01:11.882003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:30.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777169+0000","last_change":"2024-04-26T23:01:05.738519+0000","last_active":"2024-04-26T23:01:11.777169+0000","last_peered":"2024-04-26T23:01:11.777169+0000","last_clean":"2024-04-26T23:01:11.777169+0000","last_became_active":"2024-04-26T23:01:05.738201+0000","last_became_peered":"2024-04-26T23:01:05.738201+0000","last_unstale":"2024-04-26T23:01:11.777169+0000","last_undegraded":"2024-04-26T23:01:11.777169+0000","last_fullsized":"2024-04-26T23:01:11.777169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487284+0000","last_change":"2024-04-26T23:01:04.711562+0000","last_active":"2024-04-26T23:01:12.487284+0000","last_peered":"2024-04-26T23:01:12.487284+0000","last_clean":"2024-04-26T23:01:12.487284+0000","last_became_active":"2024-04-26T23:01:04.708852+0000","last_became_peered":"2024-04-26T23:01:04.708852+0000","last_unstale":"2024-04-26T23:01:12.487284+0000","last_undegraded":"2024-04-26T23:01:12.487284+0000","last_fullsized":"2024-04-26T23:01:12.487284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:21:22.581263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485270+0000","last_change":"2024-04-26T23:01:09.805302+0000","last_active":"2024-04-26T23:01:12.485270+0000","last_peered":"2024-04-26T23:01:12.485270+0000","last_clean":"2024-04-26T23:01:12.485270+0000","last_became_active":"2024-04-26T23:01:09.804878+0000","last_became_peered":"2024-04-26T23:01:09.804878+0000","last_unstale":"2024-04-26T23:01:12.485270+0000","last_undegraded":"2024-04-26T23:01:12.485270+0000","last_fullsized":"2024-04-26T23:01:12.485270+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774404+0000","last_change":"2024-04-26T23:01:05.717338+0000","last_active":"2024-04-26T23:01:11.774404+0000","last_peered":"2024-04-26T23:01:11.774404+0000","last_clean":"2024-04-26T23:01:11.774404+0000","last_became_active":"2024-04-26T23:01:05.716983+0000","last_became_peered":"2024-04-26T23:01:05.716983+0000","last_unstale":"2024-04-26T23:01:11.774404+0000","last_undegraded":"2024-04-26T23:01:11.774404+0000","last_fullsized":"2024-04-26T23:01:11.774404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.774791+0000","last_change":"2024-04-26T23:01:04.715584+0000","last_active":"2024-04-26T23:01:11.774791+0000","last_peered":"2024-04-26T23:01:11.774791+0000","last_clean":"2024-04-26T23:01:11.774791+0000","last_became_active":"2024-04-26T23:01:04.715240+0000","last_became_peered":"2024-04-26T23:01:04.715240+0000","last_unstale":"2024-04-26T23:01:11.774791+0000","last_undegraded":"2024-04-26T23:01:11.774791+0000","last_fullsized":"2024-04-26T23:01:11.774791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771265+0000","last_change":"2024-04-26T23:01:10.401101+0000","last_active":"2024-04-26T23:01:11.771265+0000","last_peered":"2024-04-26T23:01:11.771265+0000","last_clean":"2024-04-26T23:01:11.771265+0000","last_became_active":"2024-04-26T23:01:10.396052+0000","last_became_peered":"2024-04-26T23:01:10.396052+0000","last_unstale":"2024-04-26T23:01:11.771265+0000","last_undegraded":"2024-04-26T23:01:11.771265+0000","last_fullsized":"2024-04-26T23:01:11.771265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:15.222701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776466+0000","last_change":"2024-04-26T23:01:05.732489+0000","last_active":"2024-04-26T23:01:11.776466+0000","last_peered":"2024-04-26T23:01:11.776466+0000","last_clean":"2024-04-26T23:01:11.776466+0000","last_became_active":"2024-04-26T23:01:05.732228+0000","last_became_peered":"2024-04-26T23:01:05.732228+0000","last_unstale":"2024-04-26T23:01:11.776466+0000","last_undegraded":"2024-04-26T23:01:11.776466+0000","last_fullsized":"2024-04-26T23:01:11.776466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:22:15.301548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484809+0000","last_change":"2024-04-26T23:01:04.731558+0000","last_active":"2024-04-26T23:01:12.484809+0000","last_peered":"2024-04-26T23:01:12.484809+0000","last_clean":"2024-04-26T23:01:12.484809+0000","last_became_active":"2024-04-26T23:01:04.731032+0000","last_became_peered":"2024-04-26T23:01:04.731032+0000","last_unstale":"2024-04-26T23:01:12.484809+0000","last_undegraded":"2024-04-26T23:01:12.484809+0000","last_fullsized":"2024-04-26T23:01:12.484809+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770866+0000","last_change":"2024-04-26T23:01:10.386434+0000","last_active":"2024-04-26T23:01:11.770866+0000","last_peered":"2024-04-26T23:01:11.770866+0000","last_clean":"2024-04-26T23:01:11.770866+0000","last_became_active":"2024-04-26T23:01:10.385162+0000","last_became_peered":"2024-04-26T23:01:10.385162+0000","last_unstale":"2024-04-26T23:01:11.770866+0000","last_undegraded":"2024-04-26T23:01:11.770866+0000","last_fullsized":"2024-04-26T23:01:11.770866+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:19.562651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483154+0000","last_change":"2024-04-26T23:01:05.740528+0000","last_active":"2024-04-26T23:01:12.483154+0000","last_peered":"2024-04-26T23:01:12.483154+0000","last_clean":"2024-04-26T23:01:12.483154+0000","last_became_active":"2024-04-26T23:01:05.740211+0000","last_became_peered":"2024-04-26T23:01:05.740211+0000","last_unstale":"2024-04-26T23:01:12.483154+0000","last_undegraded":"2024-04-26T23:01:12.483154+0000","last_fullsized":"2024-04-26T23:01:12.483154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484416+0000","last_change":"2024-04-26T23:01:04.712658+0000","last_active":"2024-04-26T23:01:12.484416+0000","last_peered":"2024-04-26T23:01:12.484416+0000","last_clean":"2024-04-26T23:01:12.484416+0000","last_became_active":"2024-04-26T23:01:04.712325+0000","last_became_peered":"2024-04-26T23:01:04.712325+0000","last_unstale":"2024-04-26T23:01:12.484416+0000","last_undegraded":"2024-04-26T23:01:12.484416+0000","last_fullsized":"2024-04-26T23:01:12.484416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:52.271211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485257+0000","last_change":"2024-04-26T23:01:10.385938+0000","last_active":"2024-04-26T23:01:12.485257+0000","last_peered":"2024-04-26T23:01:12.485257+0000","last_clean":"2024-04-26T23:01:12.485257+0000","last_became_active":"2024-04-26T23:01:10.384600+0000","last_became_peered":"2024-04-26T23:01:10.384600+0000","last_unstale":"2024-04-26T23:01:12.485257+0000","last_undegraded":"2024-04-26T23:01:12.485257+0000","last_fullsized":"2024-04-26T23:01:12.485257+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:25.672514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775962+0000","last_change":"2024-04-26T23:01:05.744208+0000","last_active":"2024-04-26T23:01:11.775962+0000","last_peered":"2024-04-26T23:01:11.775962+0000","last_clean":"2024-04-26T23:01:11.775962+0000","last_became_active":"2024-04-26T23:01:05.743897+0000","last_became_peered":"2024-04-26T23:01:05.743897+0000","last_unstale":"2024-04-26T23:01:11.775962+0000","last_undegraded":"2024-04-26T23:01:11.775962+0000","last_fullsized":"2024-04-26T23:01:11.775962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:22.679803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:25.136494+0000","last_change":"2024-04-26T23:01:04.723670+0000","last_active":"2024-04-26T23:01:25.136494+0000","last_peered":"2024-04-26T23:01:25.136494+0000","last_clean":"2024-04-26T23:01:25.136494+0000","last_became_active":"2024-04-26T23:01:04.723361+0000","last_became_peered":"2024-04-26T23:01:04.723361+0000","last_unstale":"2024-04-26T23:01:25.136494+0000","last_undegraded":"2024-04-26T23:01:25.136494+0000","last_fullsized":"2024-04-26T23:01:25.136494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:58.527150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485782+0000","last_change":"2024-04-26T23:01:09.799732+0000","last_active":"2024-04-26T23:01:12.485782+0000","last_peered":"2024-04-26T23:01:12.485782+0000","last_clean":"2024-04-26T23:01:12.485782+0000","last_became_active":"2024-04-26T23:01:09.797973+0000","last_became_peered":"2024-04-26T23:01:09.797973+0000","last_unstale":"2024-04-26T23:01:12.485782+0000","last_undegraded":"2024-04-26T23:01:12.485782+0000","last_fullsized":"2024-04-26T23:01:12.485782+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:41.540020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882483+0000","last_change":"2024-04-26T23:01:05.732675+0000","last_active":"2024-04-26T23:01:11.882483+0000","last_peered":"2024-04-26T23:01:11.882483+0000","last_clean":"2024-04-26T23:01:11.882483+0000","last_became_active":"2024-04-26T23:01:05.732334+0000","last_became_peered":"2024-04-26T23:01:05.732334+0000","last_unstale":"2024-04-26T23:01:11.882483+0000","last_undegraded":"2024-04-26T23:01:11.882483+0000","last_fullsized":"2024-04-26T23:01:11.882483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:46.537111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484886+0000","last_change":"2024-04-26T23:01:04.731957+0000","last_active":"2024-04-26T23:01:12.484886+0000","last_peered":"2024-04-26T23:01:12.484886+0000","last_clean":"2024-04-26T23:01:12.484886+0000","last_became_active":"2024-04-26T23:01:04.731688+0000","last_became_peered":"2024-04-26T23:01:04.731688+0000","last_unstale":"2024-04-26T23:01:12.484886+0000","last_undegraded":"2024-04-26T23:01:12.484886+0000","last_fullsized":"2024-04-26T23:01:12.484886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:16.098138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882537+0000","last_change":"2024-04-26T23:01:09.793585+0000","last_active":"2024-04-26T23:01:11.882537+0000","last_peered":"2024-04-26T23:01:11.882537+0000","last_clean":"2024-04-26T23:01:11.882537+0000","last_became_active":"2024-04-26T23:01:09.793177+0000","last_became_peered":"2024-04-26T23:01:09.793177+0000","last_unstale":"2024-04-26T23:01:11.882537+0000","last_undegraded":"2024-04-26T23:01:11.882537+0000","last_fullsized":"2024-04-26T23:01:11.882537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:42.777098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487118+0000","last_change":"2024-04-26T23:01:05.723191+0000","last_active":"2024-04-26T23:01:12.487118+0000","last_peered":"2024-04-26T23:01:12.487118+0000","last_clean":"2024-04-26T23:01:12.487118+0000","last_became_active":"2024-04-26T23:01:05.722903+0000","last_became_peered":"2024-04-26T23:01:05.722903+0000","last_unstale":"2024-04-26T23:01:12.487118+0000","last_undegraded":"2024-04-26T23:01:12.487118+0000","last_fullsized":"2024-04-26T23:01:12.487118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484699+0000","last_change":"2024-04-26T23:01:04.729093+0000","last_active":"2024-04-26T23:01:12.484699+0000","last_peered":"2024-04-26T23:01:12.484699+0000","last_clean":"2024-04-26T23:01:12.484699+0000","last_became_active":"2024-04-26T23:01:04.728125+0000","last_became_peered":"2024-04-26T23:01:04.728125+0000","last_unstale":"2024-04-26T23:01:12.484699+0000","last_undegraded":"2024-04-26T23:01:12.484699+0000","last_fullsized":"2024-04-26T23:01:12.484699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769989+0000","last_change":"2024-04-26T23:01:09.802623+0000","last_active":"2024-04-26T23:01:11.769989+0000","last_peered":"2024-04-26T23:01:11.769989+0000","last_clean":"2024-04-26T23:01:11.769989+0000","last_became_active":"2024-04-26T23:01:09.798323+0000","last_became_peered":"2024-04-26T23:01:09.798323+0000","last_unstale":"2024-04-26T23:01:11.769989+0000","last_undegraded":"2024-04-26T23:01:11.769989+0000","last_fullsized":"2024-04-26T23:01:11.769989+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:56.662302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484537+0000","last_change":"2024-04-26T23:01:05.727245+0000","last_active":"2024-04-26T23:01:12.484537+0000","last_peered":"2024-04-26T23:01:12.484537+0000","last_clean":"2024-04-26T23:01:12.484537+0000","last_became_active":"2024-04-26T23:01:05.726786+0000","last_became_peered":"2024-04-26T23:01:05.726786+0000","last_unstale":"2024-04-26T23:01:12.484537+0000","last_undegraded":"2024-04-26T23:01:12.484537+0000","last_fullsized":"2024-04-26T23:01:12.484537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:24.824202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778010+0000","last_change":"2024-04-26T23:01:04.732475+0000","last_active":"2024-04-26T23:01:11.778010+0000","last_peered":"2024-04-26T23:01:11.778010+0000","last_clean":"2024-04-26T23:01:11.778010+0000","last_became_active":"2024-04-26T23:01:04.732185+0000","last_became_peered":"2024-04-26T23:01:04.732185+0000","last_unstale":"2024-04-26T23:01:11.778010+0000","last_undegraded":"2024-04-26T23:01:11.778010+0000","last_fullsized":"2024-04-26T23:01:11.778010+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:48.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770229+0000","last_change":"2024-04-26T23:01:10.396108+0000","last_active":"2024-04-26T23:01:11.770229+0000","last_peered":"2024-04-26T23:01:11.770229+0000","last_clean":"2024-04-26T23:01:11.770229+0000","last_became_active":"2024-04-26T23:01:10.394701+0000","last_became_peered":"2024-04-26T23:01:10.394701+0000","last_unstale":"2024-04-26T23:01:11.770229+0000","last_undegraded":"2024-04-26T23:01:11.770229+0000","last_fullsized":"2024-04-26T23:01:11.770229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:00:11.874893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484827+0000","last_change":"2024-04-26T23:01:10.396863+0000","last_active":"2024-04-26T23:01:12.484827+0000","last_peered":"2024-04-26T23:01:12.484827+0000","last_clean":"2024-04-26T23:01:12.484827+0000","last_became_active":"2024-04-26T23:01:10.395914+0000","last_became_peered":"2024-04-26T23:01:10.395914+0000","last_unstale":"2024-04-26T23:01:12.484827+0000","last_undegraded":"2024-04-26T23:01:12.484827+0000","last_fullsized":"2024-04-26T23:01:12.484827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:30.804166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483687+0000","last_change":"2024-04-26T23:01:05.749537+0000","last_active":"2024-04-26T23:01:12.483687+0000","last_peered":"2024-04-26T23:01:12.483687+0000","last_clean":"2024-04-26T23:01:12.483687+0000","last_became_active":"2024-04-26T23:01:05.749254+0000","last_became_peered":"2024-04-26T23:01:05.749254+0000","last_unstale":"2024-04-26T23:01:12.483687+0000","last_undegraded":"2024-04-26T23:01:12.483687+0000","last_fullsized":"2024-04-26T23:01:12.483687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:00.924105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483804+0000","last_change":"2024-04-26T23:01:04.730472+0000","last_active":"2024-04-26T23:01:12.483804+0000","last_peered":"2024-04-26T23:01:12.483804+0000","last_clean":"2024-04-26T23:01:12.483804+0000","last_became_active":"2024-04-26T23:01:04.729294+0000","last_became_peered":"2024-04-26T23:01:04.729294+0000","last_unstale":"2024-04-26T23:01:12.483804+0000","last_undegraded":"2024-04-26T23:01:12.483804+0000","last_fullsized":"2024-04-26T23:01:12.483804+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:58.527150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778171+0000","last_change":"2024-04-26T23:01:09.800700+0000","last_active":"2024-04-26T23:01:11.778171+0000","last_peered":"2024-04-26T23:01:11.778171+0000","last_clean":"2024-04-26T23:01:11.778171+0000","last_became_active":"2024-04-26T23:01:09.799085+0000","last_became_peered":"2024-04-26T23:01:09.799085+0000","last_unstale":"2024-04-26T23:01:11.778171+0000","last_undegraded":"2024-04-26T23:01:11.778171+0000","last_fullsized":"2024-04-26T23:01:11.778171+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:59.067859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769853+0000","last_change":"2024-04-26T23:01:05.737423+0000","last_active":"2024-04-26T23:01:11.769853+0000","last_peered":"2024-04-26T23:01:11.769853+0000","last_clean":"2024-04-26T23:01:11.769853+0000","last_became_active":"2024-04-26T23:01:05.736881+0000","last_became_peered":"2024-04-26T23:01:05.736881+0000","last_unstale":"2024-04-26T23:01:11.769853+0000","last_undegraded":"2024-04-26T23:01:11.769853+0000","last_fullsized":"2024-04-26T23:01:11.769853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:35:08.020426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770314+0000","last_change":"2024-04-26T23:01:04.723415+0000","last_active":"2024-04-26T23:01:11.770314+0000","last_peered":"2024-04-26T23:01:11.770314+0000","last_clean":"2024-04-26T23:01:11.770314+0000","last_became_active":"2024-04-26T23:01:04.723109+0000","last_became_peered":"2024-04-26T23:01:04.723109+0000","last_unstale":"2024-04-26T23:01:11.770314+0000","last_undegraded":"2024-04-26T23:01:11.770314+0000","last_fullsized":"2024-04-26T23:01:11.770314+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:52.271211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.775686+0000","last_change":"2024-04-26T23:01:04.712783+0000","last_active":"2024-04-26T23:01:11.775686+0000","last_peered":"2024-04-26T23:01:11.775686+0000","last_clean":"2024-04-26T23:01:11.775686+0000","last_became_active":"2024-04-26T23:01:04.712436+0000","last_became_peered":"2024-04-26T23:01:04.712436+0000","last_unstale":"2024-04-26T23:01:11.775686+0000","last_undegraded":"2024-04-26T23:01:11.775686+0000","last_fullsized":"2024-04-26T23:01:11.775686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:35.395090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484177+0000","last_change":"2024-04-26T23:01:05.742581+0000","last_active":"2024-04-26T23:01:12.484177+0000","last_peered":"2024-04-26T23:01:12.484177+0000","last_clean":"2024-04-26T23:01:12.484177+0000","last_became_active":"2024-04-26T23:01:05.740918+0000","last_became_peered":"2024-04-26T23:01:05.740918+0000","last_unstale":"2024-04-26T23:01:12.484177+0000","last_undegraded":"2024-04-26T23:01:12.484177+0000","last_fullsized":"2024-04-26T23:01:12.484177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:35:08.020426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.488392+0000","last_change":"2024-04-26T23:01:09.798160+0000","last_active":"2024-04-26T23:01:12.488392+0000","last_peered":"2024-04-26T23:01:12.488392+0000","last_clean":"2024-04-26T23:01:12.488392+0000","last_became_active":"2024-04-26T23:01:09.797746+0000","last_became_peered":"2024-04-26T23:01:09.797746+0000","last_unstale":"2024-04-26T23:01:12.488392+0000","last_undegraded":"2024-04-26T23:01:12.488392+0000","last_fullsized":"2024-04-26T23:01:12.488392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:11.186360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770724+0000","last_change":"2024-04-26T23:01:04.723482+0000","last_active":"2024-04-26T23:01:11.770724+0000","last_peered":"2024-04-26T23:01:11.770724+0000","last_clean":"2024-04-26T23:01:11.770724+0000","last_became_active":"2024-04-26T23:01:04.722816+0000","last_became_peered":"2024-04-26T23:01:04.722816+0000","last_unstale":"2024-04-26T23:01:11.770724+0000","last_undegraded":"2024-04-26T23:01:11.770724+0000","last_fullsized":"2024-04-26T23:01:11.770724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:03.658782+0000","last_clean_scrub_stamp":"2024-04-26T23:01:03.658782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:57.381012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882061+0000","last_change":"2024-04-26T23:01:05.745436+0000","last_active":"2024-04-26T23:01:11.882061+0000","last_peered":"2024-04-26T23:01:11.882061+0000","last_clean":"2024-04-26T23:01:11.882061+0000","last_became_active":"2024-04-26T23:01:05.745148+0000","last_became_peered":"2024-04-26T23:01:05.745148+0000","last_unstale":"2024-04-26T23:01:11.882061+0000","last_undegraded":"2024-04-26T23:01:11.882061+0000","last_fullsized":"2024-04-26T23:01:11.882061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:17.111974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771111+0000","last_change":"2024-04-26T23:01:09.795297+0000","last_active":"2024-04-26T23:01:11.771111+0000","last_peered":"2024-04-26T23:01:11.771111+0000","last_clean":"2024-04-26T23:01:11.771111+0000","last_became_active":"2024-04-26T23:01:09.794873+0000","last_became_peered":"2024-04-26T23:01:09.794873+0000","last_unstale":"2024-04-26T23:01:11.771111+0000","last_undegraded":"2024-04-26T23:01:11.771111+0000","last_fullsized":"2024-04-26T23:01:11.771111+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:08.743746+0000","last_clean_scrub_stamp":"2024-04-26T23:01:08.743746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:16:59.104896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.773782+0000","last_change":"2024-04-26T23:01:05.713935+0000","last_active":"2024-04-26T23:01:11.773782+0000","last_peered":"2024-04-26T23:01:11.773782+0000","last_clean":"2024-04-26T23:01:11.773782+0000","last_became_active":"2024-04-26T23:01:05.713606+0000","last_became_peered":"2024-04-26T23:01:05.713606+0000","last_unstale":"2024-04-26T23:01:11.773782+0000","last_undegraded":"2024-04-26T23:01:11.773782+0000","last_fullsized":"2024-04-26T23:01:11.773782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:26.157667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487634+0000","last_change":"2024-04-26T23:01:05.722889+0000","last_active":"2024-04-26T23:01:12.487634+0000","last_peered":"2024-04-26T23:01:12.487634+0000","last_clean":"2024-04-26T23:01:12.487634+0000","last_became_active":"2024-04-26T23:01:05.721634+0000","last_became_peered":"2024-04-26T23:01:05.721634+0000","last_unstale":"2024-04-26T23:01:12.487634+0000","last_undegraded":"2024-04-26T23:01:12.487634+0000","last_fullsized":"2024-04-26T23:01:12.487634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:46.537111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484476+0000","last_change":"2024-04-26T23:01:05.749096+0000","last_active":"2024-04-26T23:01:12.484476+0000","last_peered":"2024-04-26T23:01:12.484476+0000","last_clean":"2024-04-26T23:01:12.484476+0000","last_became_active":"2024-04-26T23:01:05.747770+0000","last_became_peered":"2024-04-26T23:01:05.747770+0000","last_unstale":"2024-04-26T23:01:12.484476+0000","last_undegraded":"2024-04-26T23:01:12.484476+0000","last_fullsized":"2024-04-26T23:01:12.484476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:57.106236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483913+0000","last_change":"2024-04-26T23:01:05.747157+0000","last_active":"2024-04-26T23:01:12.483913+0000","last_peered":"2024-04-26T23:01:12.483913+0000","last_clean":"2024-04-26T23:01:12.483913+0000","last_became_active":"2024-04-26T23:01:05.746805+0000","last_became_peered":"2024-04-26T23:01:05.746805+0000","last_unstale":"2024-04-26T23:01:12.483913+0000","last_undegraded":"2024-04-26T23:01:12.483913+0000","last_fullsized":"2024-04-26T23:01:12.483913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:07.115233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770959+0000","last_change":"2024-04-26T23:01:05.747433+0000","last_active":"2024-04-26T23:01:11.770959+0000","last_peered":"2024-04-26T23:01:11.770959+0000","last_clean":"2024-04-26T23:01:11.770959+0000","last_became_active":"2024-04-26T23:01:05.747108+0000","last_became_peered":"2024-04-26T23:01:05.747108+0000","last_unstale":"2024-04-26T23:01:11.770959+0000","last_undegraded":"2024-04-26T23:01:11.770959+0000","last_fullsized":"2024-04-26T23:01:11.770959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:57.106236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484633+0000","last_change":"2024-04-26T23:01:05.753692+0000","last_active":"2024-04-26T23:01:12.484633+0000","last_peered":"2024-04-26T23:01:12.484633+0000","last_clean":"2024-04-26T23:01:12.484633+0000","last_became_active":"2024-04-26T23:01:05.752474+0000","last_became_peered":"2024-04-26T23:01:05.752474+0000","last_unstale":"2024-04-26T23:01:12.484633+0000","last_undegraded":"2024-04-26T23:01:12.484633+0000","last_fullsized":"2024-04-26T23:01:12.484633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778393+0000","last_change":"2024-04-26T23:01:05.731062+0000","last_active":"2024-04-26T23:01:11.778393+0000","last_peered":"2024-04-26T23:01:11.778393+0000","last_clean":"2024-04-26T23:01:11.778393+0000","last_became_active":"2024-04-26T23:01:05.727960+0000","last_became_peered":"2024-04-26T23:01:05.727960+0000","last_unstale":"2024-04-26T23:01:11.778393+0000","last_undegraded":"2024-04-26T23:01:11.778393+0000","last_fullsized":"2024-04-26T23:01:11.778393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.771192+0000","last_change":"2024-04-26T23:01:05.750232+0000","last_active":"2024-04-26T23:01:11.771192+0000","last_peered":"2024-04-26T23:01:11.771192+0000","last_clean":"2024-04-26T23:01:11.771192+0000","last_became_active":"2024-04-26T23:01:05.749885+0000","last_became_peered":"2024-04-26T23:01:05.749885+0000","last_unstale":"2024-04-26T23:01:11.771192+0000","last_undegraded":"2024-04-26T23:01:11.771192+0000","last_fullsized":"2024-04-26T23:01:11.771192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483726+0000","last_change":"2024-04-26T23:01:05.749914+0000","last_active":"2024-04-26T23:01:12.483726+0000","last_peered":"2024-04-26T23:01:12.483726+0000","last_clean":"2024-04-26T23:01:12.483726+0000","last_became_active":"2024-04-26T23:01:05.745797+0000","last_became_peered":"2024-04-26T23:01:05.745797+0000","last_unstale":"2024-04-26T23:01:12.483726+0000","last_undegraded":"2024-04-26T23:01:12.483726+0000","last_fullsized":"2024-04-26T23:01:12.483726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770217+0000","last_change":"2024-04-26T23:01:05.750181+0000","last_active":"2024-04-26T23:01:11.770217+0000","last_peered":"2024-04-26T23:01:11.770217+0000","last_clean":"2024-04-26T23:01:11.770217+0000","last_became_active":"2024-04-26T23:01:05.749810+0000","last_became_peered":"2024-04-26T23:01:05.749810+0000","last_unstale":"2024-04-26T23:01:11.770217+0000","last_undegraded":"2024-04-26T23:01:11.770217+0000","last_fullsized":"2024-04-26T23:01:11.770217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:01.346004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484216+0000","last_change":"2024-04-26T23:01:05.730857+0000","last_active":"2024-04-26T23:01:12.484216+0000","last_peered":"2024-04-26T23:01:12.484216+0000","last_clean":"2024-04-26T23:01:12.484216+0000","last_became_active":"2024-04-26T23:01:05.730543+0000","last_became_peered":"2024-04-26T23:01:05.730543+0000","last_unstale":"2024-04-26T23:01:12.484216+0000","last_undegraded":"2024-04-26T23:01:12.484216+0000","last_fullsized":"2024-04-26T23:01:12.484216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.482980+0000","last_change":"2024-04-26T23:01:05.741407+0000","last_active":"2024-04-26T23:01:12.482980+0000","last_peered":"2024-04-26T23:01:12.482980+0000","last_clean":"2024-04-26T23:01:12.482980+0000","last_became_active":"2024-04-26T23:01:05.740354+0000","last_became_peered":"2024-04-26T23:01:05.740354+0000","last_unstale":"2024-04-26T23:01:12.482980+0000","last_undegraded":"2024-04-26T23:01:12.482980+0000","last_fullsized":"2024-04-26T23:01:12.482980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:24:06.318595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777299+0000","last_change":"2024-04-26T23:01:05.730302+0000","last_active":"2024-04-26T23:01:11.777299+0000","last_peered":"2024-04-26T23:01:11.777299+0000","last_clean":"2024-04-26T23:01:11.777299+0000","last_became_active":"2024-04-26T23:01:05.729149+0000","last_became_peered":"2024-04-26T23:01:05.729149+0000","last_unstale":"2024-04-26T23:01:11.777299+0000","last_undegraded":"2024-04-26T23:01:11.777299+0000","last_fullsized":"2024-04-26T23:01:11.777299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776184+0000","last_change":"2024-04-26T23:01:05.731828+0000","last_active":"2024-04-26T23:01:11.776184+0000","last_peered":"2024-04-26T23:01:11.776184+0000","last_clean":"2024-04-26T23:01:11.776184+0000","last_became_active":"2024-04-26T23:01:05.731537+0000","last_became_peered":"2024-04-26T23:01:05.731537+0000","last_unstale":"2024-04-26T23:01:11.776184+0000","last_undegraded":"2024-04-26T23:01:11.776184+0000","last_fullsized":"2024-04-26T23:01:11.776184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:59:57.750603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.487750+0000","last_change":"2024-04-26T23:01:05.726729+0000","last_active":"2024-04-26T23:01:12.487750+0000","last_peered":"2024-04-26T23:01:12.487750+0000","last_clean":"2024-04-26T23:01:12.487750+0000","last_became_active":"2024-04-26T23:01:05.726364+0000","last_became_peered":"2024-04-26T23:01:05.726364+0000","last_unstale":"2024-04-26T23:01:12.487750+0000","last_undegraded":"2024-04-26T23:01:12.487750+0000","last_fullsized":"2024-04-26T23:01:12.487750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:17.111974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.485129+0000","last_change":"2024-04-26T23:01:05.728806+0000","last_active":"2024-04-26T23:01:12.485129+0000","last_peered":"2024-04-26T23:01:12.485129+0000","last_clean":"2024-04-26T23:01:12.485129+0000","last_became_active":"2024-04-26T23:01:05.728418+0000","last_became_peered":"2024-04-26T23:01:05.728418+0000","last_unstale":"2024-04-26T23:01:12.485129+0000","last_undegraded":"2024-04-26T23:01:12.485129+0000","last_fullsized":"2024-04-26T23:01:12.485129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:40.686396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483463+0000","last_change":"2024-04-26T23:01:05.721739+0000","last_active":"2024-04-26T23:01:12.483463+0000","last_peered":"2024-04-26T23:01:12.483463+0000","last_clean":"2024-04-26T23:01:12.483463+0000","last_became_active":"2024-04-26T23:01:05.721409+0000","last_became_peered":"2024-04-26T23:01:05.721409+0000","last_unstale":"2024-04-26T23:01:12.483463+0000","last_undegraded":"2024-04-26T23:01:12.483463+0000","last_fullsized":"2024-04-26T23:01:12.483463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483935+0000","last_change":"2024-04-26T23:01:05.739575+0000","last_active":"2024-04-26T23:01:12.483935+0000","last_peered":"2024-04-26T23:01:12.483935+0000","last_clean":"2024-04-26T23:01:12.483935+0000","last_became_active":"2024-04-26T23:01:05.734760+0000","last_became_peered":"2024-04-26T23:01:05.734760+0000","last_unstale":"2024-04-26T23:01:12.483935+0000","last_undegraded":"2024-04-26T23:01:12.483935+0000","last_fullsized":"2024-04-26T23:01:12.483935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483111+0000","last_change":"2024-04-26T23:01:05.732699+0000","last_active":"2024-04-26T23:01:12.483111+0000","last_peered":"2024-04-26T23:01:12.483111+0000","last_clean":"2024-04-26T23:01:12.483111+0000","last_became_active":"2024-04-26T23:01:05.732371+0000","last_became_peered":"2024-04-26T23:01:05.732371+0000","last_unstale":"2024-04-26T23:01:12.483111+0000","last_undegraded":"2024-04-26T23:01:12.483111+0000","last_fullsized":"2024-04-26T23:01:12.483111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:56.515639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.770589+0000","last_change":"2024-04-26T23:01:05.745445+0000","last_active":"2024-04-26T23:01:11.770589+0000","last_peered":"2024-04-26T23:01:11.770589+0000","last_clean":"2024-04-26T23:01:11.770589+0000","last_became_active":"2024-04-26T23:01:05.742977+0000","last_became_peered":"2024-04-26T23:01:05.742977+0000","last_unstale":"2024-04-26T23:01:11.770589+0000","last_undegraded":"2024-04-26T23:01:11.770589+0000","last_fullsized":"2024-04-26T23:01:11.770589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:18:18.696196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.882409+0000","last_change":"2024-04-26T23:01:05.739107+0000","last_active":"2024-04-26T23:01:11.882409+0000","last_peered":"2024-04-26T23:01:11.882409+0000","last_clean":"2024-04-26T23:01:11.882409+0000","last_became_active":"2024-04-26T23:01:05.738786+0000","last_became_peered":"2024-04-26T23:01:05.738786+0000","last_unstale":"2024-04-26T23:01:11.882409+0000","last_undegraded":"2024-04-26T23:01:11.882409+0000","last_fullsized":"2024-04-26T23:01:11.882409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:15:58.394848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778055+0000","last_change":"2024-04-26T23:01:05.710270+0000","last_active":"2024-04-26T23:01:11.778055+0000","last_peered":"2024-04-26T23:01:11.778055+0000","last_clean":"2024-04-26T23:01:11.778055+0000","last_became_active":"2024-04-26T23:01:05.709943+0000","last_became_peered":"2024-04-26T23:01:05.709943+0000","last_unstale":"2024-04-26T23:01:11.778055+0000","last_undegraded":"2024-04-26T23:01:11.778055+0000","last_fullsized":"2024-04-26T23:01:11.778055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.778339+0000","last_change":"2024-04-26T23:01:05.717127+0000","last_active":"2024-04-26T23:01:11.778339+0000","last_peered":"2024-04-26T23:01:11.778339+0000","last_clean":"2024-04-26T23:01:11.778339+0000","last_became_active":"2024-04-26T23:01:05.716647+0000","last_became_peered":"2024-04-26T23:01:05.716647+0000","last_unstale":"2024-04-26T23:01:11.778339+0000","last_undegraded":"2024-04-26T23:01:11.778339+0000","last_fullsized":"2024-04-26T23:01:11.778339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776723+0000","last_change":"2024-04-26T23:01:05.730733+0000","last_active":"2024-04-26T23:01:11.776723+0000","last_peered":"2024-04-26T23:01:11.776723+0000","last_clean":"2024-04-26T23:01:11.776723+0000","last_became_active":"2024-04-26T23:01:05.727115+0000","last_became_peered":"2024-04-26T23:01:05.727115+0000","last_unstale":"2024-04-26T23:01:11.776723+0000","last_undegraded":"2024-04-26T23:01:11.776723+0000","last_fullsized":"2024-04-26T23:01:11.776723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:40.686396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.769732+0000","last_change":"2024-04-26T23:01:05.737620+0000","last_active":"2024-04-26T23:01:11.769732+0000","last_peered":"2024-04-26T23:01:11.769732+0000","last_clean":"2024-04-26T23:01:11.769732+0000","last_became_active":"2024-04-26T23:01:05.737220+0000","last_became_peered":"2024-04-26T23:01:05.737220+0000","last_unstale":"2024-04-26T23:01:11.769732+0000","last_undegraded":"2024-04-26T23:01:11.769732+0000","last_fullsized":"2024-04-26T23:01:11.769732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:07.115233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.777054+0000","last_change":"2024-04-26T23:01:05.731417+0000","last_active":"2024-04-26T23:01:11.777054+0000","last_peered":"2024-04-26T23:01:11.777054+0000","last_clean":"2024-04-26T23:01:11.777054+0000","last_became_active":"2024-04-26T23:01:05.730556+0000","last_became_peered":"2024-04-26T23:01:05.730556+0000","last_unstale":"2024-04-26T23:01:11.777054+0000","last_undegraded":"2024-04-26T23:01:11.777054+0000","last_fullsized":"2024-04-26T23:01:11.777054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:13.482334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.483844+0000","last_change":"2024-04-26T23:01:05.721675+0000","last_active":"2024-04-26T23:01:12.483844+0000","last_peered":"2024-04-26T23:01:12.483844+0000","last_clean":"2024-04-26T23:01:12.483844+0000","last_became_active":"2024-04-26T23:01:05.721363+0000","last_became_peered":"2024-04-26T23:01:05.721363+0000","last_unstale":"2024-04-26T23:01:12.483844+0000","last_undegraded":"2024-04-26T23:01:12.483844+0000","last_fullsized":"2024-04-26T23:01:12.483844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:13.482334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776212+0000","last_change":"2024-04-26T23:01:05.729610+0000","last_active":"2024-04-26T23:01:11.776212+0000","last_peered":"2024-04-26T23:01:11.776212+0000","last_clean":"2024-04-26T23:01:11.776212+0000","last_became_active":"2024-04-26T23:01:05.729307+0000","last_became_peered":"2024-04-26T23:01:05.729307+0000","last_unstale":"2024-04-26T23:01:11.776212+0000","last_undegraded":"2024-04-26T23:01:11.776212+0000","last_fullsized":"2024-04-26T23:01:11.776212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:15:57.106236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484169+0000","last_change":"2024-04-26T23:01:05.753873+0000","last_active":"2024-04-26T23:01:12.484169+0000","last_peered":"2024-04-26T23:01:12.484169+0000","last_clean":"2024-04-26T23:01:12.484169+0000","last_became_active":"2024-04-26T23:01:05.753603+0000","last_became_peered":"2024-04-26T23:01:05.753603+0000","last_unstale":"2024-04-26T23:01:12.484169+0000","last_undegraded":"2024-04-26T23:01:12.484169+0000","last_fullsized":"2024-04-26T23:01:12.484169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:22:15.301548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.484214+0000","last_change":"2024-04-26T23:01:05.720420+0000","last_active":"2024-04-26T23:01:12.484214+0000","last_peered":"2024-04-26T23:01:12.484214+0000","last_clean":"2024-04-26T23:01:12.484214+0000","last_became_active":"2024-04-26T23:01:05.720114+0000","last_became_peered":"2024-04-26T23:01:05.720114+0000","last_unstale":"2024-04-26T23:01:12.484214+0000","last_undegraded":"2024-04-26T23:01:12.484214+0000","last_fullsized":"2024-04-26T23:01:12.484214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:41.636867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:12.486621+0000","last_change":"2024-04-26T23:01:05.730995+0000","last_active":"2024-04-26T23:01:12.486621+0000","last_peered":"2024-04-26T23:01:12.486621+0000","last_clean":"2024-04-26T23:01:12.486621+0000","last_became_active":"2024-04-26T23:01:05.730644+0000","last_became_peered":"2024-04-26T23:01:05.730644+0000","last_unstale":"2024-04-26T23:01:12.486621+0000","last_undegraded":"2024-04-26T23:01:12.486621+0000","last_fullsized":"2024-04-26T23:01:12.486621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:56.414070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-26T23:01:11.776188+0000","last_change":"2024-04-26T23:01:05.727391+0000","last_active":"2024-04-26T23:01:11.776188+0000","last_peered":"2024-04-26T23:01:11.776188+0000","last_clean":"2024-04-26T23:01:11.776188+0000","last_became_active":"2024-04-26T23:01:05.726729+0000","last_became_peered":"2024-04-26T23:01:05.726729+0000","last_unstale":"2024-04-26T23:01:11.776188+0000","last_undegraded":"2024-04-26T23:01:11.776188+0000","last_fullsized":"2024-04-26T23:01:11.776188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:01:04.672619+0000","last_clean_scrub_stamp":"2024-04-26T23:01:04.672619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:24.824202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:01:41.474 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T23:01:41.475 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T23:01:41.475 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-26T23:01:41.475 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-26T23:01:41.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.721+0000 7f974b200640 1 Processor -- start 2024-04-26T23:01:41.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.722+0000 7f974b200640 1 -- start start 2024-04-26T23:01:41.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.723+0000 7f974b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f974c145970 0x7f974c1612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.723+0000 7f974b200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f974c0b59f0 con 0x7f974c0d06b0 2024-04-26T23:01:41.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.723+0000 7f974b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f974c1617f0 con 0x7f974c146060 2024-04-26T23:01:41.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.723+0000 7f974b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f974c0ceed0 con 0x7f974c145970 2024-04-26T23:01:41.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.724+0000 7f974a800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f974c0d06b0 0x7f974c0cf900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41604/0 (socket says 172.21.15.40:41604) 2024-04-26T23:01:41.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.724+0000 7f974a800640 1 -- 172.21.15.40:0/1254538968 learned_addr learned my addr 172.21.15.40:0/1254538968 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:41.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.724+0000 7f9749e00640 1 --2- 172.21.15.40:0/1254538968 >> v2:172.21.15.103:3300/0 conn(0x7f974c145970 0x7f974c1612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.724+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3753791345 0 0) 0x7f974c0b59f0 con 0x7f974c0d06b0 2024-04-26T23:01:41.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.724+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9730003520 con 0x7f974c0d06b0 2024-04-26T23:01:41.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2486856676 0 0) 0x7f9730003520 con 0x7f974c0d06b0 2024-04-26T23:01:41.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 >> v2:172.21.15.103:3300/0 conn(0x7f974c145970 msgr2=0x7f974c1612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749400640 1 --2- 172.21.15.40:0/1254538968 >> v2:172.21.15.103:3300/0 conn(0x7f974c145970 0x7f974c1612b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 >> v1:172.21.15.103:6789/0 conn(0x7f974c146060 legacy=0x7f974c0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f974c0b4180 con 0x7f974c0d06b0 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f97440019f0 con 0x7f974c0d06b0 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 >> v1:172.21.15.40:6789/0 conn(0x7f974c0d06b0 legacy=0x7f974c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749400640 1 --2- 172.21.15.40:0/1254538968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97300039b0 0x7f9730003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749400640 1 --2- 172.21.15.40:0/1254538968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9730004800 0x7f9730004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749400640 1 --2- 172.21.15.40:0/1254538968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97300051c0 0x7f973000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f973000c030 con 0x7f97300039b0 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f973000c1e0 con 0x7f97300051c0 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f973000c3e0 con 0x7f9730004800 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9749e00640 1 --2- 172.21.15.40:0/1254538968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97300039b0 0x7f9730003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f9741400640 1 --2- 172.21.15.40:0/1254538968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9730004800 0x7f9730004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.725+0000 7f974a800640 1 --2- 172.21.15.40:0/1254538968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97300051c0 0x7f973000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.726+0000 7f9749e00640 1 -- 172.21.15.40:0/1254538968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9730004800 msgr2=0x7f9730004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:41.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.726+0000 7f9749e00640 1 --2- 172.21.15.40:0/1254538968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9730004800 0x7f9730004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.726+0000 7f9749e00640 1 -- 172.21.15.40:0/1254538968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97300051c0 msgr2=0x7f973000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:41.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.726+0000 7f9749e00640 1 --2- 172.21.15.40:0/1254538968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97300051c0 0x7f973000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.726+0000 7f9749e00640 1 -- 172.21.15.40:0/1254538968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9730003520 con 0x7f97300039b0 2024-04-26T23:01:41.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.726+0000 7f974a800640 1 --2- 172.21.15.40:0/1254538968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97300051c0 0x7f973000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:41.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.726+0000 7f974a800640 1 -- 172.21.15.40:0/1254538968 reap_dead start 2024-04-26T23:01:41.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.727+0000 7f9749e00640 1 --2- 172.21.15.40:0/1254538968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97300039b0 0x7f9730003dd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f973c009ac0 tx=0x7f9730005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa6a5270cb27a3b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:41.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.727+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973c036070 con 0x7f97300039b0 2024-04-26T23:01:41.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.727+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f973c031400 con 0x7f97300039b0 2024-04-26T23:01:41.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.727+0000 7f9749400640 1 -- 172.21.15.40:0/1254538968 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973c009690 con 0x7f97300039b0 2024-04-26T23:01:41.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.727+0000 7f974b200640 1 -- 172.21.15.40:0/1254538968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97300039b0 msgr2=0x7f9730003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:41.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.727+0000 7f974b200640 1 --2- 172.21.15.40:0/1254538968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97300039b0 0x7f9730003dd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f973c009ac0 tx=0x7f9730005830 comp rx=0 tx=0).stop 2024-04-26T23:01:41.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.728+0000 7f974b200640 1 -- 172.21.15.40:0/1254538968 shutdown_connections 2024-04-26T23:01:41.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.728+0000 7f974b200640 1 --2- 172.21.15.40:0/1254538968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97300039b0 0x7f9730003dd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.728+0000 7f974b200640 1 -- 172.21.15.40:0/1254538968 >> 172.21.15.40:0/1254538968 conn(0x7f974c075eb0 msgr2=0x7f974c068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:41.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.728+0000 7f974b200640 1 -- 172.21.15.40:0/1254538968 shutdown_connections 2024-04-26T23:01:41.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.728+0000 7f974b200640 1 -- 172.21.15.40:0/1254538968 wait complete. 2024-04-26T23:01:41.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.729+0000 7f974b200640 1 Processor -- start 2024-04-26T23:01:41.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.730+0000 7f974b200640 1 -- start start 2024-04-26T23:01:41.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.730+0000 7f974b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97300039b0 0x7f974c1f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f974b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f974c072670 0x7f974c1e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f974b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f974c1e7e30 0x7f974c1f8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f974b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f974c0cf100 con 0x7f97300039b0 2024-04-26T23:01:41.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f974b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f974c0b5c60 con 0x7f974c1e7e30 2024-04-26T23:01:41.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f974b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f974c0cf3e0 con 0x7f974c072670 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f9749e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97300039b0 0x7f974c1f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f9749e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97300039b0 0x7f974c1f2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39060/0 (socket says 172.21.15.40:39060) 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f9749e00640 1 -- 172.21.15.40:0/2837720500 learned_addr learned my addr 172.21.15.40:0/2837720500 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f9741400640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f974c072670 0x7f974c1e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f974a800640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f974c1e7e30 0x7f974c1f8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f9749e00640 1 -- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f974c072670 msgr2=0x7f974c1e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f9749e00640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f974c072670 0x7f974c1e78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f9749e00640 1 -- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f974c1e7e30 msgr2=0x7f974c1f8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f9749e00640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f974c1e7e30 0x7f974c1f8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f9749e00640 1 -- 172.21.15.40:0/2837720500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f974c072b90 con 0x7f97300039b0 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.731+0000 7f974a800640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f974c1e7e30 0x7f974c1f8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.732+0000 7f9749e00640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97300039b0 0x7f974c1f2be0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f973c034040 tx=0x7f973c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.732+0000 7f9742a00640 1 -- 172.21.15.40:0/2837720500 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973c03f020 con 0x7f97300039b0 2024-04-26T23:01:41.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.732+0000 7f9742a00640 1 -- 172.21.15.40:0/2837720500 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f973c037040 con 0x7f97300039b0 2024-04-26T23:01:41.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.732+0000 7f974b200640 1 -- 172.21.15.40:0/2837720500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f974c072e70 con 0x7f97300039b0 2024-04-26T23:01:41.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.732+0000 7f9742a00640 1 -- 172.21.15.40:0/2837720500 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973c036070 con 0x7f97300039b0 2024-04-26T23:01:41.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.732+0000 7f974b200640 1 -- 172.21.15.40:0/2837720500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f974c073430 con 0x7f97300039b0 2024-04-26T23:01:41.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.733+0000 7f974b200640 1 -- 172.21.15.40:0/2837720500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f974c0ce210 con 0x7f97300039b0 2024-04-26T23:01:41.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.734+0000 7f9742a00640 1 -- 172.21.15.40:0/2837720500 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f973c042020 con 0x7f97300039b0 2024-04-26T23:01:41.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.735+0000 7f9742a00640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f972806c920 0x7f972806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:41.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.735+0000 7f9742a00640 1 -- 172.21.15.40:0/2837720500 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f973c0abec0 con 0x7f97300039b0 2024-04-26T23:01:41.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.736+0000 7f9741400640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f972806c920 0x7f972806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:41.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.737+0000 7f9741400640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f972806c920 0x7f972806ede0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9734000c00 tx=0x7f9734009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:41.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.740+0000 7f9742a00640 1 -- 172.21.15.40:0/2837720500 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f974c0ce210 con 0x7f97300039b0 2024-04-26T23:01:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.991+0000 7f974b200640 1 -- 172.21.15.40:0/2837720500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f974c069370 con 0x7f97300039b0 2024-04-26T23:01:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.991+0000 7f9742a00640 1 -- 172.21.15.40:0/2837720500 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f973c049030 con 0x7f97300039b0 2024-04-26T23:01:41.991 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:41.992 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T23:01:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.994+0000 7f974b200640 1 -- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f972806c920 msgr2=0x7f972806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.994+0000 7f974b200640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f972806c920 0x7f972806ede0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9734000c00 tx=0x7f9734009040 comp rx=0 tx=0).stop 2024-04-26T23:01:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.994+0000 7f974b200640 1 -- 172.21.15.40:0/2837720500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97300039b0 msgr2=0x7f974c1f2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.994+0000 7f974b200640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97300039b0 0x7f974c1f2be0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f973c034040 tx=0x7f973c004730 comp rx=0 tx=0).stop 2024-04-26T23:01:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.994+0000 7f974b200640 1 -- 172.21.15.40:0/2837720500 shutdown_connections 2024-04-26T23:01:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.994+0000 7f974b200640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f972806c920 0x7f972806ede0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.994+0000 7f974b200640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f974c1e7e30 0x7f974c1f8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.994+0000 7f974b200640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f974c072670 0x7f974c1e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.994+0000 7f974b200640 1 --2- 172.21.15.40:0/2837720500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97300039b0 0x7f974c1f2be0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.994+0000 7f974b200640 1 -- 172.21.15.40:0/2837720500 >> 172.21.15.40:0/2837720500 conn(0x7f974c075eb0 msgr2=0x7f974c0b4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.994+0000 7f974b200640 1 -- 172.21.15.40:0/2837720500 shutdown_connections 2024-04-26T23:01:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:41.994+0000 7f974b200640 1 -- 172.21.15.40:0/2837720500 wait complete. 2024-04-26T23:01:42.013 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-26T23:01:42.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:42.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.218+0000 7f46c2200640 1 Processor -- start 2024-04-26T23:01:42.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.218+0000 7f46c2200640 1 -- start start 2024-04-26T23:01:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.220+0000 7f46c2200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46bc071e60 0x7f46bc06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.220+0000 7f46c2200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f46bc06fe20 0x7f46bc072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.220+0000 7f46c2200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46bc15d7c0 con 0x7f46bc072780 2024-04-26T23:01:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.220+0000 7f46c2200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f46bc15daa0 con 0x7f46bc06fe20 2024-04-26T23:01:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.220+0000 7f46c2200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f46bc15dd80 con 0x7f46bc071e60 2024-04-26T23:01:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.221+0000 7f46bbe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f46bc072780 0x7f46bc0cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41618/0 (socket says 172.21.15.40:41618) 2024-04-26T23:01:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.221+0000 7f46bbe00640 1 -- 172.21.15.40:0/720214907 learned_addr learned my addr 172.21.15.40:0/720214907 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.221+0000 7f46b3e00640 1 --2- 172.21.15.40:0/720214907 >> v2:172.21.15.103:3300/0 conn(0x7f46bc06fe20 0x7f46bc072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.221+0000 7f46bb400640 1 --2- 172.21.15.40:0/720214907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46bc071e60 0x7f46bc06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.221+0000 7f46baa00640 1 -- 172.21.15.40:0/720214907 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3354480481 0 0) 0x7f46bc15d7c0 con 0x7f46bc072780 2024-04-26T23:01:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.221+0000 7f46baa00640 1 -- 172.21.15.40:0/720214907 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46a0003520 con 0x7f46bc072780 2024-04-26T23:01:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.221+0000 7f46baa00640 1 -- 172.21.15.40:0/720214907 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3317336448 0 0) 0x7f46a0003520 con 0x7f46bc072780 2024-04-26T23:01:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.222+0000 7f46b3e00640 1 -- 172.21.15.40:0/720214907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46bc071e60 msgr2=0x7f46bc06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.222+0000 7f46b3e00640 1 --2- 172.21.15.40:0/720214907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46bc071e60 0x7f46bc06f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.222+0000 7f46b3e00640 1 -- 172.21.15.40:0/720214907 >> v1:172.21.15.40:6789/0 conn(0x7f46bc072780 legacy=0x7f46bc0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.222+0000 7f46b3e00640 1 -- 172.21.15.40:0/720214907 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46bc15e060 con 0x7f46bc06fe20 2024-04-26T23:01:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.223+0000 7f46b3e00640 1 --2- 172.21.15.40:0/720214907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46bc06fe20 0x7f46bc072240 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f46a40097a0 tx=0x7f46a402a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d30fb6860cf4bdb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.223+0000 7f46baa00640 1 -- 172.21.15.40:0/720214907 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a4037070 con 0x7f46bc06fe20 2024-04-26T23:01:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.223+0000 7f46baa00640 1 -- 172.21.15.40:0/720214907 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46a402aba0 con 0x7f46bc06fe20 2024-04-26T23:01:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.223+0000 7f46baa00640 1 -- 172.21.15.40:0/720214907 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b40019f0 con 0x7f46bc06fe20 2024-04-26T23:01:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.224+0000 7f46c2200640 1 -- 172.21.15.40:0/720214907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46bc06fe20 msgr2=0x7f46bc072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.224+0000 7f46c2200640 1 --2- 172.21.15.40:0/720214907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46bc06fe20 0x7f46bc072240 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f46a40097a0 tx=0x7f46a402a110 comp rx=0 tx=0).stop 2024-04-26T23:01:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.224+0000 7f46c2200640 1 -- 172.21.15.40:0/720214907 shutdown_connections 2024-04-26T23:01:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.224+0000 7f46c2200640 1 --2- 172.21.15.40:0/720214907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46bc06fe20 0x7f46bc072240 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.224+0000 7f46c2200640 1 --2- 172.21.15.40:0/720214907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46bc071e60 0x7f46bc06f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.224+0000 7f46c2200640 1 -- 172.21.15.40:0/720214907 >> 172.21.15.40:0/720214907 conn(0x7f46bc0d0be0 msgr2=0x7f46bc0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:42.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.224+0000 7f46c2200640 1 -- 172.21.15.40:0/720214907 shutdown_connections 2024-04-26T23:01:42.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.224+0000 7f46c2200640 1 -- 172.21.15.40:0/720214907 wait complete. 2024-04-26T23:01:42.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.226+0000 7f46c2200640 1 Processor -- start 2024-04-26T23:01:42.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.226+0000 7f46c2200640 1 -- start start 2024-04-26T23:01:42.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.227+0000 7f46c2200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46bc06fe20 0x7f46bc15ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.227+0000 7f46c2200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc071e60 0x7f46bc157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.227+0000 7f46c2200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46bc072780 0x7f46bc157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.227+0000 7f46c2200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc15d3f0 con 0x7f46bc071e60 2024-04-26T23:01:42.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.227+0000 7f46c2200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc072c10 con 0x7f46bc06fe20 2024-04-26T23:01:42.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.227+0000 7f46c2200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f46bc15d570 con 0x7f46bc072780 2024-04-26T23:01:42.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.227+0000 7f46b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc071e60 0x7f46bc157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:42.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.227+0000 7f46b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc071e60 0x7f46bc157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39070/0 (socket says 172.21.15.40:39070) 2024-04-26T23:01:42.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.227+0000 7f46b3e00640 1 -- 172.21.15.40:0/38182277 learned_addr learned my addr 172.21.15.40:0/38182277 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:42.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46bb400640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46bc06fe20 0x7f46bc15ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:42.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46bbe00640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46bc072780 0x7f46bc157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:42.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46b3e00640 1 -- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46bc072780 msgr2=0x7f46bc157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:42.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46b3e00640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46bc072780 0x7f46bc157860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46b3e00640 1 -- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46bc06fe20 msgr2=0x7f46bc15ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:42.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46b3e00640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46bc06fe20 0x7f46bc15ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46b3e00640 1 -- 172.21.15.40:0/38182277 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46bc15c5e0 con 0x7f46bc071e60 2024-04-26T23:01:42.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46bbe00640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46bc072780 0x7f46bc157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:42.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46bb400640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46bc06fe20 0x7f46bc15ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:42.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46b3e00640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc071e60 0x7f46bc157320 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f46a40098c0 tx=0x7f46a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:42.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46b3400640 1 -- 172.21.15.40:0/38182277 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a4041030 con 0x7f46bc071e60 2024-04-26T23:01:42.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46b3400640 1 -- 172.21.15.40:0/38182277 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46a403d420 con 0x7f46bc071e60 2024-04-26T23:01:42.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46c2200640 1 -- 172.21.15.40:0/38182277 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46bc0b5490 con 0x7f46bc071e60 2024-04-26T23:01:42.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.228+0000 7f46b3400640 1 -- 172.21.15.40:0/38182277 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a4037040 con 0x7f46bc071e60 2024-04-26T23:01:42.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.229+0000 7f46c2200640 1 -- 172.21.15.40:0/38182277 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46bc0b5a50 con 0x7f46bc071e60 2024-04-26T23:01:42.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.230+0000 7f46c2200640 1 -- 172.21.15.40:0/38182277 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46bc0694f0 con 0x7f46bc071e60 2024-04-26T23:01:42.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.231+0000 7f46b3400640 1 -- 172.21.15.40:0/38182277 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f46a4044070 con 0x7f46bc071e60 2024-04-26T23:01:42.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.231+0000 7f46b3400640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f469806c8f0 0x7f469806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.232+0000 7f46b3400640 1 -- 172.21.15.40:0/38182277 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f46a40ac3b0 con 0x7f46bc071e60 2024-04-26T23:01:42.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.232+0000 7f46bb400640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f469806c8f0 0x7f469806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:42.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.233+0000 7f46bb400640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f469806c8f0 0x7f469806edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0cb0c0 tx=0x7f46ac008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:42.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.236+0000 7f46b3400640 1 -- 172.21.15.40:0/38182277 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f46a407a400 con 0x7f46bc071e60 2024-04-26T23:01:42.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.447+0000 7f46c2200640 1 -- 172.21.15.40:0/38182277 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46bc073b20 con 0x7f46bc071e60 2024-04-26T23:01:42.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.450+0000 7f46b3400640 1 -- 172.21.15.40:0/38182277 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f46a4079d00 con 0x7f46bc071e60 2024-04-26T23:01:42.451 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:42.451 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:11.763965+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:42.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.452+0000 7f46c2200640 1 -- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f469806c8f0 msgr2=0x7f469806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:42.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.452+0000 7f46c2200640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f469806c8f0 0x7f469806edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0cb0c0 tx=0x7f46ac008040 comp rx=0 tx=0).stop 2024-04-26T23:01:42.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.452+0000 7f46c2200640 1 -- 172.21.15.40:0/38182277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc071e60 msgr2=0x7f46bc157320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:42.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.452+0000 7f46c2200640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc071e60 0x7f46bc157320 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f46a40098c0 tx=0x7f46a40046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.453+0000 7f46c2200640 1 -- 172.21.15.40:0/38182277 shutdown_connections 2024-04-26T23:01:42.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.453+0000 7f46c2200640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f469806c8f0 0x7f469806edb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.453+0000 7f46c2200640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46bc072780 0x7f46bc157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.453+0000 7f46c2200640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc071e60 0x7f46bc157320 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.453+0000 7f46c2200640 1 --2- 172.21.15.40:0/38182277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46bc06fe20 0x7f46bc15ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.453+0000 7f46c2200640 1 -- 172.21.15.40:0/38182277 >> 172.21.15.40:0/38182277 conn(0x7f46bc0d0be0 msgr2=0x7f46bc0cdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:42.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.453+0000 7f46c2200640 1 -- 172.21.15.40:0/38182277 shutdown_connections 2024-04-26T23:01:42.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.453+0000 7f46c2200640 1 -- 172.21.15.40:0/38182277 wait complete. 2024-04-26T23:01:42.473 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:01:42.473 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:01:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.681+0000 7f7e7be00640 1 Processor -- start 2024-04-26T23:01:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.681+0000 7f7e7be00640 1 -- start start 2024-04-26T23:01:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.683+0000 7f7e7be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e7c0b5500 0x7f7e7c0b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.683+0000 7f7e7be00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e7c0d3dc0 con 0x7f7e7c0b5120 2024-04-26T23:01:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.683+0000 7f7e7be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e7c0b75b0 con 0x7f7e7c0b59b0 2024-04-26T23:01:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.683+0000 7f7e7be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7e7c0d03b0 con 0x7f7e7c0b5500 2024-04-26T23:01:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.683+0000 7f7e7aa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7e7c0b5120 0x7f7e7c0b6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41628/0 (socket says 172.21.15.40:41628) 2024-04-26T23:01:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.683+0000 7f7e7aa00640 1 -- 172.21.15.40:0/2380110443 learned_addr learned my addr 172.21.15.40:0/2380110443 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e7a000640 1 --2- 172.21.15.40:0/2380110443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e7c0b5500 0x7f7e7c0b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1241519641 0 0) 0x7f7e7c0d3dc0 con 0x7f7e7c0b5120 2024-04-26T23:01:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e60003520 con 0x7f7e7c0b5120 2024-04-26T23:01:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1999528948 0 0) 0x7f7e60003520 con 0x7f7e7c0b5120 2024-04-26T23:01:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e7c0b5500 msgr2=0x7f7e7c0b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 --2- 172.21.15.40:0/2380110443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e7c0b5500 0x7f7e7c0b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 >> v1:172.21.15.103:6789/0 conn(0x7f7e7c0b59b0 legacy=0x7f7e7c0b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c0d0600 con 0x7f7e7c0b5120 2024-04-26T23:01:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f7e6c001670 con 0x7f7e7c0b5120 2024-04-26T23:01:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 >> v1:172.21.15.40:6789/0 conn(0x7f7e7c0b5120 legacy=0x7f7e7c0b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 --2- 172.21.15.40:0/2380110443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e600039e0 0x7f7e60003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 --2- 172.21.15.40:0/2380110443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e7c0b5500 0x7f7e60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 --2- 172.21.15.40:0/2380110443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e60004d40 0x7f7e600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c0d03b0 con 0x7f7e600039e0 2024-04-26T23:01:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.684+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6000bb80 con 0x7f7e60004d40 2024-04-26T23:01:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.685+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6000bd80 con 0x7f7e7c0b5500 2024-04-26T23:01:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.685+0000 7f7e7aa00640 1 --2- 172.21.15.40:0/2380110443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e600039e0 0x7f7e60003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.685+0000 7f7e7a000640 1 --2- 172.21.15.40:0/2380110443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e7c0b5500 0x7f7e60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.685+0000 7f7e7aa00640 1 -- 172.21.15.40:0/2380110443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e7c0b5500 msgr2=0x7f7e60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.685+0000 7f7e7aa00640 1 --2- 172.21.15.40:0/2380110443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e7c0b5500 0x7f7e60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.685+0000 7f7e7aa00640 1 -- 172.21.15.40:0/2380110443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e60004d40 msgr2=0x7f7e600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.685+0000 7f7e7aa00640 1 --2- 172.21.15.40:0/2380110443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e60004d40 0x7f7e600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.685+0000 7f7e7aa00640 1 -- 172.21.15.40:0/2380110443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e60003520 con 0x7f7e600039e0 2024-04-26T23:01:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.686+0000 7f7e7aa00640 1 --2- 172.21.15.40:0/2380110443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e600039e0 0x7f7e60003e00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00c110 tx=0x7f7e60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04b6cd28e7fee5f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.686+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c007c40 con 0x7f7e600039e0 2024-04-26T23:01:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.686+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e6c005af0 con 0x7f7e600039e0 2024-04-26T23:01:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.686+0000 7f7e79600640 1 -- 172.21.15.40:0/2380110443 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c007070 con 0x7f7e600039e0 2024-04-26T23:01:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.687+0000 7f7e7be00640 1 -- 172.21.15.40:0/2380110443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e600039e0 msgr2=0x7f7e60003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.687+0000 7f7e7be00640 1 --2- 172.21.15.40:0/2380110443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e600039e0 0x7f7e60003e00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00c110 tx=0x7f7e60005830 comp rx=0 tx=0).stop 2024-04-26T23:01:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.687+0000 7f7e7b400640 1 -- 172.21.15.40:0/2380110443 reap_dead start 2024-04-26T23:01:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.687+0000 7f7e7be00640 1 -- 172.21.15.40:0/2380110443 shutdown_connections 2024-04-26T23:01:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.687+0000 7f7e7be00640 1 -- 172.21.15.40:0/2380110443 >> 172.21.15.40:0/2380110443 conn(0x7f7e7c0d4db0 msgr2=0x7f7e7c0d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.687+0000 7f7e7be00640 1 -- 172.21.15.40:0/2380110443 shutdown_connections 2024-04-26T23:01:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.687+0000 7f7e7be00640 1 -- 172.21.15.40:0/2380110443 wait complete. 2024-04-26T23:01:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.689+0000 7f7e7be00640 1 Processor -- start 2024-04-26T23:01:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.689+0000 7f7e7be00640 1 -- start start 2024-04-26T23:01:42.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.690+0000 7f7e7be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e600039e0 0x7f7e7c0b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.690+0000 7f7e7be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e7c154e00 0x7f7e7c0b8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.690+0000 7f7e7be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e7c0b9db0 0x7f7e7c0ba230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.690+0000 7f7e7be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c0b5ee0 con 0x7f7e600039e0 2024-04-26T23:01:42.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.690+0000 7f7e7be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c0d0bc0 con 0x7f7e7c154e00 2024-04-26T23:01:42.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.690+0000 7f7e7be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7e7c0b6060 con 0x7f7e7c0b9db0 2024-04-26T23:01:42.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.690+0000 7f7e7aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e600039e0 0x7f7e7c0b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:42.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.690+0000 7f7e7aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e600039e0 0x7f7e7c0b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39092/0 (socket says 172.21.15.40:39092) 2024-04-26T23:01:42.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.690+0000 7f7e7aa00640 1 -- 172.21.15.40:0/1784816016 learned_addr learned my addr 172.21.15.40:0/1784816016 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:42.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.691+0000 7f7e7aa00640 1 -- 172.21.15.40:0/1784816016 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e7c0b9db0 msgr2=0x7f7e7c0ba230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:42.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.691+0000 7f7e7aa00640 1 --2- 172.21.15.40:0/1784816016 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e7c0b9db0 0x7f7e7c0ba230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.691+0000 7f7e7aa00640 1 -- 172.21.15.40:0/1784816016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e7c154e00 msgr2=0x7f7e7c0b8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:42.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.691+0000 7f7e7aa00640 1 --2- 172.21.15.40:0/1784816016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e7c154e00 0x7f7e7c0b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.691+0000 7f7e7aa00640 1 -- 172.21.15.40:0/1784816016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c0ca2b0 con 0x7f7e600039e0 2024-04-26T23:01:42.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.691+0000 7f7e7aa00640 1 --2- 172.21.15.40:0/1784816016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e600039e0 0x7f7e7c0b2470 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c02ad60 tx=0x7f7e7c15fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:42.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.691+0000 7f7e72a00640 1 -- 172.21.15.40:0/1784816016 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c041020 con 0x7f7e600039e0 2024-04-26T23:01:42.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.692+0000 7f7e72a00640 1 -- 172.21.15.40:0/1784816016 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e6c005ca0 con 0x7f7e600039e0 2024-04-26T23:01:42.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.692+0000 7f7e72a00640 1 -- 172.21.15.40:0/1784816016 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c007070 con 0x7f7e600039e0 2024-04-26T23:01:42.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.692+0000 7f7e7be00640 1 -- 172.21.15.40:0/1784816016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7c0ca570 con 0x7f7e600039e0 2024-04-26T23:01:42.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.692+0000 7f7e7be00640 1 -- 172.21.15.40:0/1784816016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7c0cab30 con 0x7f7e600039e0 2024-04-26T23:01:42.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.693+0000 7f7e7be00640 1 -- 172.21.15.40:0/1784816016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e7c0763b0 con 0x7f7e600039e0 2024-04-26T23:01:42.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.694+0000 7f7e72a00640 1 -- 172.21.15.40:0/1784816016 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7e6c044020 con 0x7f7e600039e0 2024-04-26T23:01:42.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.695+0000 7f7e72a00640 1 --2- 172.21.15.40:0/1784816016 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e5806c8f0 0x7f7e5806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:42.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.695+0000 7f7e72a00640 1 -- 172.21.15.40:0/1784816016 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7e6c0ac4b0 con 0x7f7e600039e0 2024-04-26T23:01:42.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.696+0000 7f7e7a000640 1 --2- 172.21.15.40:0/1784816016 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e5806c8f0 0x7f7e5806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:42.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.697+0000 7f7e7a000640 1 --2- 172.21.15.40:0/1784816016 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e5806c8f0 0x7f7e5806edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c15c6f0 tx=0x7f7e64008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:42.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.700+0000 7f7e72a00640 1 -- 172.21.15.40:0/1784816016 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7e7c0763b0 con 0x7f7e600039e0 2024-04-26T23:01:42.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.904+0000 7f7e7be00640 1 -- 172.21.15.40:0/1784816016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7e7c08eed0 con 0x7f7e600039e0 2024-04-26T23:01:42.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.905+0000 7f7e72a00640 1 -- 172.21.15.40:0/1784816016 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f7e7c08eed0 con 0x7f7e600039e0 2024-04-26T23:01:42.905 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:01:42.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.907+0000 7f7e7be00640 1 -- 172.21.15.40:0/1784816016 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e5806c8f0 msgr2=0x7f7e5806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:42.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.908+0000 7f7e7be00640 1 --2- 172.21.15.40:0/1784816016 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e5806c8f0 0x7f7e5806edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c15c6f0 tx=0x7f7e64008040 comp rx=0 tx=0).stop 2024-04-26T23:01:42.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.908+0000 7f7e7be00640 1 -- 172.21.15.40:0/1784816016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e600039e0 msgr2=0x7f7e7c0b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:42.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.908+0000 7f7e7be00640 1 --2- 172.21.15.40:0/1784816016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e600039e0 0x7f7e7c0b2470 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c02ad60 tx=0x7f7e7c15fa80 comp rx=0 tx=0).stop 2024-04-26T23:01:42.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.908+0000 7f7e7be00640 1 -- 172.21.15.40:0/1784816016 shutdown_connections 2024-04-26T23:01:42.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.908+0000 7f7e7be00640 1 --2- 172.21.15.40:0/1784816016 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e5806c8f0 0x7f7e5806edb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.908+0000 7f7e7be00640 1 --2- 172.21.15.40:0/1784816016 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e7c0b9db0 0x7f7e7c0ba230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.908+0000 7f7e7be00640 1 --2- 172.21.15.40:0/1784816016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e7c154e00 0x7f7e7c0b8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.908+0000 7f7e7be00640 1 --2- 172.21.15.40:0/1784816016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e600039e0 0x7f7e7c0b2470 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:42.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.908+0000 7f7e7be00640 1 -- 172.21.15.40:0/1784816016 >> 172.21.15.40:0/1784816016 conn(0x7f7e7c0d4db0 msgr2=0x7f7e7c0b6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:42.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.908+0000 7f7e7be00640 1 -- 172.21.15.40:0/1784816016 shutdown_connections 2024-04-26T23:01:42.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:42.908+0000 7f7e7be00640 1 -- 172.21.15.40:0/1784816016 wait complete. 2024-04-26T23:01:42.927 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:01:43.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.139+0000 7fa8abe00640 1 Processor -- start 2024-04-26T23:01:43.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.140+0000 7fa8abe00640 1 -- start start 2024-04-26T23:01:43.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.141+0000 7fa8abe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b5120 0x7fa8ac0b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.141+0000 7fa8abe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa8ac0b5500 0x7fa8ac0b6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.141+0000 7fa8abe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8ac0d3dc0 con 0x7fa8ac0b59b0 2024-04-26T23:01:43.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.141+0000 7fa8abe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa8ac0d03b0 con 0x7fa8ac0b5500 2024-04-26T23:01:43.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.141+0000 7fa8abe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa8ac0b75b0 con 0x7fa8ac0b5120 2024-04-26T23:01:43.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.142+0000 7fa8ab400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa8ac0b59b0 0x7fa8ac0b6d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41638/0 (socket says 172.21.15.40:41638) 2024-04-26T23:01:43.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.142+0000 7fa8ab400640 1 -- 172.21.15.40:0/3298542936 learned_addr learned my addr 172.21.15.40:0/3298542936 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:43.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.142+0000 7fa8aaa00640 1 --2- 172.21.15.40:0/3298542936 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b5120 0x7fa8ac0b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.142+0000 7fa8a3e00640 1 --2- 172.21.15.40:0/3298542936 >> v2:172.21.15.103:3300/0 conn(0x7fa8ac0b5500 0x7fa8ac0b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.142+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1298691142 0 0) 0x7fa8ac0d3dc0 con 0x7fa8ac0b59b0 2024-04-26T23:01:43.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.142+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa890003520 con 0x7fa8ac0b59b0 2024-04-26T23:01:43.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.142+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3563092186 0 0) 0x7fa890003520 con 0x7fa8ac0b59b0 2024-04-26T23:01:43.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b5120 msgr2=0x7fa8ac0b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:43.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aa000640 1 --2- 172.21.15.40:0/3298542936 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b5120 0x7fa8ac0b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 >> v2:172.21.15.103:3300/0 conn(0x7fa8ac0b5500 msgr2=0x7fa8ac0b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:43.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aa000640 1 --2- 172.21.15.40:0/3298542936 >> v2:172.21.15.103:3300/0 conn(0x7fa8ac0b5500 0x7fa8ac0b6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8ac0d0600 con 0x7fa8ac0b59b0 2024-04-26T23:01:43.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fa8a40019f0 con 0x7fa8ac0b59b0 2024-04-26T23:01:43.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 >> v1:172.21.15.40:6789/0 conn(0x7fa8ac0b59b0 legacy=0x7fa8ac0b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:43.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aa000640 1 --2- 172.21.15.40:0/3298542936 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b5120 0x7fa8900039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aa000640 1 --2- 172.21.15.40:0/3298542936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa890003f80 0x7fa890004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aa000640 1 --2- 172.21.15.40:0/3298542936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa890004d40 0x7fa8900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac0d03b0 con 0x7fa890004d40 2024-04-26T23:01:43.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac0b75b0 con 0x7fa890003f80 2024-04-26T23:01:43.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa89000baf0 con 0x7fa8ac0b5120 2024-04-26T23:01:43.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8ab400640 1 --2- 172.21.15.40:0/3298542936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa890004d40 0x7fa8900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8aaa00640 1 --2- 172.21.15.40:0/3298542936 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b5120 0x7fa8900039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8a3e00640 1 --2- 172.21.15.40:0/3298542936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa890003f80 0x7fa890004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8ab400640 1 -- 172.21.15.40:0/3298542936 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b5120 msgr2=0x7fa8900039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:43.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8ab400640 1 --2- 172.21.15.40:0/3298542936 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b5120 0x7fa8900039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8ab400640 1 -- 172.21.15.40:0/3298542936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa890003f80 msgr2=0x7fa890004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:43.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.143+0000 7fa8ab400640 1 --2- 172.21.15.40:0/3298542936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa890003f80 0x7fa890004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.144+0000 7fa8ab400640 1 -- 172.21.15.40:0/3298542936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa890003520 con 0x7fa890004d40 2024-04-26T23:01:43.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.145+0000 7fa8ab400640 1 --2- 172.21.15.40:0/3298542936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa890004d40 0x7fa8900051c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400a7a0 tx=0x7fa890005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31afae57c30023b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:43.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.145+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a400a3f0 con 0x7fa890004d40 2024-04-26T23:01:43.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.145+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8a402f070 con 0x7fa890004d40 2024-04-26T23:01:43.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.145+0000 7fa8aa000640 1 -- 172.21.15.40:0/3298542936 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a402bc60 con 0x7fa890004d40 2024-04-26T23:01:43.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.145+0000 7fa8abe00640 1 -- 172.21.15.40:0/3298542936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa890004d40 msgr2=0x7fa8900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:43.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.145+0000 7fa8abe00640 1 --2- 172.21.15.40:0/3298542936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa890004d40 0x7fa8900051c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400a7a0 tx=0x7fa890005830 comp rx=0 tx=0).stop 2024-04-26T23:01:43.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.145+0000 7fa8ab400640 1 -- 172.21.15.40:0/3298542936 reap_dead start 2024-04-26T23:01:43.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.146+0000 7fa8abe00640 1 -- 172.21.15.40:0/3298542936 shutdown_connections 2024-04-26T23:01:43.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.146+0000 7fa8abe00640 1 -- 172.21.15.40:0/3298542936 >> 172.21.15.40:0/3298542936 conn(0x7fa8ac0d4db0 msgr2=0x7fa8ac0d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:43.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.146+0000 7fa8abe00640 1 -- 172.21.15.40:0/3298542936 shutdown_connections 2024-04-26T23:01:43.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.146+0000 7fa8abe00640 1 -- 172.21.15.40:0/3298542936 wait complete. 2024-04-26T23:01:43.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.147+0000 7fa8abe00640 1 Processor -- start 2024-04-26T23:01:43.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.148+0000 7fa8abe00640 1 -- start start 2024-04-26T23:01:43.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.148+0000 7fa8abe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa890004d40 0x7fa8ac0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.148+0000 7fa8abe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b4250 0x7fa8ac0b97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.148+0000 7fa8abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8ac0b46d0 0x7fa8ac0b9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.148+0000 7fa8abe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac0b5ee0 con 0x7fa8ac0b46d0 2024-04-26T23:01:43.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.148+0000 7fa8abe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac0d0bc0 con 0x7fa890004d40 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.148+0000 7fa8abe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa8ac0b6060 con 0x7fa8ac0b4250 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.149+0000 7fa8ab400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8ac0b46d0 0x7fa8ac0b9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.149+0000 7fa8ab400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8ac0b46d0 0x7fa8ac0b9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39106/0 (socket says 172.21.15.40:39106) 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.149+0000 7fa8ab400640 1 -- 172.21.15.40:0/3524436452 learned_addr learned my addr 172.21.15.40:0/3524436452 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.149+0000 7fa8a3e00640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b4250 0x7fa8ac0b97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.149+0000 7fa8aaa00640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa890004d40 0x7fa8ac0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.149+0000 7fa8ab400640 1 -- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b4250 msgr2=0x7fa8ac0b97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.149+0000 7fa8ab400640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b4250 0x7fa8ac0b97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.149+0000 7fa8ab400640 1 -- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa890004d40 msgr2=0x7fa8ac0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.149+0000 7fa8ab400640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa890004d40 0x7fa8ac0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.149+0000 7fa8ab400640 1 -- 172.21.15.40:0/3524436452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8ac0cd550 con 0x7fa8ac0b46d0 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.149+0000 7fa8a3e00640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b4250 0x7fa8ac0b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.149+0000 7fa8aaa00640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa890004d40 0x7fa8ac0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:43.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.149+0000 7fa8ab400640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8ac0b46d0 0x7fa8ac0b9d10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400a8c0 tx=0x7fa8ac15fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:43.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.150+0000 7fa8a2a00640 1 -- 172.21.15.40:0/3524436452 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4044070 con 0x7fa8ac0b46d0 2024-04-26T23:01:43.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.150+0000 7fa8a2a00640 1 -- 172.21.15.40:0/3524436452 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8a400a3d0 con 0x7fa8ac0b46d0 2024-04-26T23:01:43.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.150+0000 7fa8abe00640 1 -- 172.21.15.40:0/3524436452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8ac0cd830 con 0x7fa8ac0b46d0 2024-04-26T23:01:43.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.150+0000 7fa8a2a00640 1 -- 172.21.15.40:0/3524436452 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4001f30 con 0x7fa8ac0b46d0 2024-04-26T23:01:43.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.150+0000 7fa8abe00640 1 -- 172.21.15.40:0/3524436452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8ac0cddf0 con 0x7fa8ac0b46d0 2024-04-26T23:01:43.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.151+0000 7fa8abe00640 1 -- 172.21.15.40:0/3524436452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8ac069370 con 0x7fa8ac0b46d0 2024-04-26T23:01:43.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.152+0000 7fa8a2a00640 1 -- 172.21.15.40:0/3524436452 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa8a4047070 con 0x7fa8ac0b46d0 2024-04-26T23:01:43.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.153+0000 7fa8a2a00640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa88806c8f0 0x7fa88806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.153+0000 7fa8a2a00640 1 -- 172.21.15.40:0/3524436452 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa8a40abf10 con 0x7fa8ac0b46d0 2024-04-26T23:01:43.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.154+0000 7fa8aaa00640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa88806c8f0 0x7fa88806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.155+0000 7fa8aaa00640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa88806c8f0 0x7fa88806edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa89c009a00 tx=0x7fa89c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:43.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.158+0000 7fa8a2a00640 1 -- 172.21.15.40:0/3524436452 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa8a4079ff0 con 0x7fa8ac0b46d0 2024-04-26T23:01:43.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.359+0000 7fa8abe00640 1 -- 172.21.15.40:0/3524436452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa8ac162020 con 0x7fa8ac0b46d0 2024-04-26T23:01:43.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.360+0000 7fa8a2a00640 1 -- 172.21.15.40:0/3524436452 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fa8a40798f0 con 0x7fa8ac0b46d0 2024-04-26T23:01:43.360 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:01:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.363+0000 7fa8abe00640 1 -- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa88806c8f0 msgr2=0x7fa88806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.363+0000 7fa8abe00640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa88806c8f0 0x7fa88806edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa89c009a00 tx=0x7fa89c008040 comp rx=0 tx=0).stop 2024-04-26T23:01:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.363+0000 7fa8abe00640 1 -- 172.21.15.40:0/3524436452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8ac0b46d0 msgr2=0x7fa8ac0b9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.363+0000 7fa8abe00640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8ac0b46d0 0x7fa8ac0b9d10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400a8c0 tx=0x7fa8ac15fa80 comp rx=0 tx=0).stop 2024-04-26T23:01:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.363+0000 7fa8abe00640 1 -- 172.21.15.40:0/3524436452 shutdown_connections 2024-04-26T23:01:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.363+0000 7fa8abe00640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa88806c8f0 0x7fa88806edb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.363+0000 7fa8abe00640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8ac0b46d0 0x7fa8ac0b9d10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.363+0000 7fa8abe00640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8ac0b4250 0x7fa8ac0b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.363+0000 7fa8abe00640 1 --2- 172.21.15.40:0/3524436452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa890004d40 0x7fa8ac0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.363+0000 7fa8abe00640 1 -- 172.21.15.40:0/3524436452 >> 172.21.15.40:0/3524436452 conn(0x7fa8ac0d4db0 msgr2=0x7fa8ac160420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.363+0000 7fa8abe00640 1 -- 172.21.15.40:0/3524436452 shutdown_connections 2024-04-26T23:01:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.363+0000 7fa8abe00640 1 -- 172.21.15.40:0/3524436452 wait complete. 2024-04-26T23:01:43.382 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:01:43.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.600+0000 7f4906600640 1 Processor -- start 2024-04-26T23:01:43.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.600+0000 7f4906600640 1 -- start start 2024-04-26T23:01:43.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.602+0000 7f4906600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4900156460 0x7f49000cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.602+0000 7f4906600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f490006fea0 0x7f4900072c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.602+0000 7f4906600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4900156840 con 0x7f49000cba10 2024-04-26T23:01:43.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.602+0000 7f4906600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f49000cbf20 con 0x7f4900156460 2024-04-26T23:01:43.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.602+0000 7f4906600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f49000cc200 con 0x7f490006fea0 2024-04-26T23:01:43.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.603+0000 7f48ffe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f490006fea0 0x7f4900072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.603+0000 7f48ffe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f490006fea0 0x7f4900072c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:47262/0 (socket says 172.21.15.40:47262) 2024-04-26T23:01:43.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.603+0000 7f48ff400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4900156460 0x7f49000cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.603+0000 7f48ffe00640 1 -- 172.21.15.40:0/1551795701 learned_addr learned my addr 172.21.15.40:0/1551795701 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:43.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.603+0000 7f48ffe00640 1 -- 172.21.15.40:0/1551795701 >> v2:172.21.15.103:3300/0 conn(0x7f4900156460 msgr2=0x7f49000cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:43.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.603+0000 7f48ffe00640 1 --2- 172.21.15.40:0/1551795701 >> v2:172.21.15.103:3300/0 conn(0x7f4900156460 0x7f49000cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.603+0000 7f48ffe00640 1 -- 172.21.15.40:0/1551795701 >> v1:172.21.15.103:6789/0 conn(0x7f49000cba10 legacy=0x7f490006f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:43.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.603+0000 7f48ffe00640 1 -- 172.21.15.40:0/1551795701 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49000cc4e0 con 0x7f490006fea0 2024-04-26T23:01:43.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.605+0000 7f48ffe00640 1 --2- 172.21.15.40:0/1551795701 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f490006fea0 0x7f4900072c50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009870 tx=0x7f48f802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=889ae52b400d9718 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.605+0000 7f48fea00640 1 -- 172.21.15.40:0/1551795701 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f8037070 con 0x7f490006fea0 2024-04-26T23:01:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.605+0000 7f48fea00640 1 -- 172.21.15.40:0/1551795701 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48f8032440 con 0x7f490006fea0 2024-04-26T23:01:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.605+0000 7f48fea00640 1 -- 172.21.15.40:0/1551795701 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f8032810 con 0x7f490006fea0 2024-04-26T23:01:43.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.606+0000 7f4906600640 1 -- 172.21.15.40:0/1551795701 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f490006fea0 msgr2=0x7f4900072c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:43.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.606+0000 7f4906600640 1 --2- 172.21.15.40:0/1551795701 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f490006fea0 0x7f4900072c50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009870 tx=0x7f48f802a270 comp rx=0 tx=0).stop 2024-04-26T23:01:43.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.606+0000 7f4906600640 1 -- 172.21.15.40:0/1551795701 shutdown_connections 2024-04-26T23:01:43.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.606+0000 7f4906600640 1 --2- 172.21.15.40:0/1551795701 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f490006fea0 0x7f4900072c50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.606+0000 7f4906600640 1 --2- 172.21.15.40:0/1551795701 >> v2:172.21.15.103:3300/0 conn(0x7f4900156460 0x7f49000cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.606+0000 7f4906600640 1 -- 172.21.15.40:0/1551795701 >> 172.21.15.40:0/1551795701 conn(0x7f49000d0be0 msgr2=0x7f49000d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:43.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.606+0000 7f4906600640 1 -- 172.21.15.40:0/1551795701 shutdown_connections 2024-04-26T23:01:43.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.606+0000 7f4906600640 1 -- 172.21.15.40:0/1551795701 wait complete. 2024-04-26T23:01:43.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.608+0000 7f4906600640 1 Processor -- start 2024-04-26T23:01:43.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.608+0000 7f4906600640 1 -- start start 2024-04-26T23:01:43.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.609+0000 7f4906600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490006fea0 0x7f4900149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.609+0000 7f4906600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49000cba10 0x7f49000cef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.609+0000 7f4906600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4900156460 0x7f49000cf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.609+0000 7f4906600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f490006e260 con 0x7f4900156460 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.609+0000 7f4906600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4900072700 con 0x7f490006fea0 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.609+0000 7f4906600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f490006e3e0 con 0x7f49000cba10 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.609+0000 7f48ffe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4900156460 0x7f49000cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.609+0000 7f48ffe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4900156460 0x7f49000cf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39120/0 (socket says 172.21.15.40:39120) 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.609+0000 7f48ffe00640 1 -- 172.21.15.40:0/742460681 learned_addr learned my addr 172.21.15.40:0/742460681 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.609+0000 7f48f6a00640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49000cba10 0x7f49000cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.609+0000 7f48ff400640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490006fea0 0x7f4900149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.610+0000 7f48ffe00640 1 -- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49000cba10 msgr2=0x7f49000cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.610+0000 7f48ffe00640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49000cba10 0x7f49000cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.610+0000 7f48ffe00640 1 -- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490006fea0 msgr2=0x7f4900149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.610+0000 7f48ffe00640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490006fea0 0x7f4900149990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.610+0000 7f48ffe00640 1 -- 172.21.15.40:0/742460681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f490014a5d0 con 0x7f4900156460 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.610+0000 7f48ff400640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490006fea0 0x7f4900149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.610+0000 7f48ffe00640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4900156460 0x7f49000cf460 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009870 tx=0x7f48f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:43.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.610+0000 7f48f7e00640 1 -- 172.21.15.40:0/742460681 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f8041020 con 0x7f4900156460 2024-04-26T23:01:43.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.610+0000 7f48f7e00640 1 -- 172.21.15.40:0/742460681 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48f803d640 con 0x7f4900156460 2024-04-26T23:01:43.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.610+0000 7f48f7e00640 1 -- 172.21.15.40:0/742460681 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f8037040 con 0x7f4900156460 2024-04-26T23:01:43.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.610+0000 7f4906600640 1 -- 172.21.15.40:0/742460681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f490014a8b0 con 0x7f4900156460 2024-04-26T23:01:43.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.610+0000 7f4906600640 1 -- 172.21.15.40:0/742460681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f490014ae70 con 0x7f4900156460 2024-04-26T23:01:43.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.611+0000 7f4906600640 1 -- 172.21.15.40:0/742460681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49000694f0 con 0x7f4900156460 2024-04-26T23:01:43.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.612+0000 7f48f7e00640 1 -- 172.21.15.40:0/742460681 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f48f8044020 con 0x7f4900156460 2024-04-26T23:01:43.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.613+0000 7f48f7e00640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48dc06c8f0 0x7f48dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:43.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.613+0000 7f48f7e00640 1 -- 172.21.15.40:0/742460681 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f48f80ac690 con 0x7f4900156460 2024-04-26T23:01:43.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.614+0000 7f48ff400640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48dc06c8f0 0x7f48dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:43.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.615+0000 7f48ff400640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48dc06c8f0 0x7f48dc06edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f48f0007440 tx=0x7f48f000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:43.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.618+0000 7f48f7e00640 1 -- 172.21.15.40:0/742460681 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f48f804b030 con 0x7f4900156460 2024-04-26T23:01:43.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.818+0000 7f4906600640 1 -- 172.21.15.40:0/742460681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f490006de80 con 0x7f4900156460 2024-04-26T23:01:43.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.820+0000 7f48f7e00640 1 -- 172.21.15.40:0/742460681 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f48f807a6f0 con 0x7f4900156460 2024-04-26T23:01:43.820 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:01:43.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.822+0000 7f4906600640 1 -- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48dc06c8f0 msgr2=0x7f48dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.822+0000 7f4906600640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48dc06c8f0 0x7f48dc06edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f48f0007440 tx=0x7f48f000b040 comp rx=0 tx=0).stop 2024-04-26T23:01:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.822+0000 7f4906600640 1 -- 172.21.15.40:0/742460681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4900156460 msgr2=0x7f49000cf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.822+0000 7f4906600640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4900156460 0x7f49000cf460 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009870 tx=0x7f48f80046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.822+0000 7f4906600640 1 -- 172.21.15.40:0/742460681 shutdown_connections 2024-04-26T23:01:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.822+0000 7f4906600640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48dc06c8f0 0x7f48dc06edb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.822+0000 7f4906600640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4900156460 0x7f49000cf460 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.822+0000 7f4906600640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49000cba10 0x7f49000cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.822+0000 7f4906600640 1 --2- 172.21.15.40:0/742460681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490006fea0 0x7f4900149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.822+0000 7f4906600640 1 -- 172.21.15.40:0/742460681 >> 172.21.15.40:0/742460681 conn(0x7f49000d0be0 msgr2=0x7f4900073460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.823+0000 7f4906600640 1 -- 172.21.15.40:0/742460681 shutdown_connections 2024-04-26T23:01:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:43.823+0000 7f4906600640 1 -- 172.21.15.40:0/742460681 wait complete. 2024-04-26T23:01:43.842 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:01:44.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.058+0000 7f279fe00640 1 Processor -- start 2024-04-26T23:01:44.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.059+0000 7f279fe00640 1 -- start start 2024-04-26T23:01:44.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.061+0000 7f279fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f27a015bc70 0x7f27a00cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:44.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.061+0000 7f279fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f27a0071ef0 0x7f27a0072990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:44.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.061+0000 7f279fe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27a006dde0 con 0x7f27a0072440 2024-04-26T23:01:44.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.061+0000 7f279fe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f27a006e3a0 con 0x7f27a0071ef0 2024-04-26T23:01:44.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.061+0000 7f279fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f27a00b4840 con 0x7f27a015bc70 2024-04-26T23:01:44.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.061+0000 7f2797e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f27a0071ef0 0x7f27a0072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:44.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.061+0000 7f279f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f27a0072440 0x7f27a015f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41640/0 (socket says 172.21.15.40:41640) 2024-04-26T23:01:44.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.061+0000 7f279f400640 1 -- 172.21.15.40:0/3936389673 learned_addr learned my addr 172.21.15.40:0/3936389673 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:44.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.062+0000 7f279ea00640 1 --2- 172.21.15.40:0/3936389673 >> v2:172.21.15.103:3300/0 conn(0x7f27a015bc70 0x7f27a00cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:44.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.062+0000 7f2797e00640 1 -- 172.21.15.40:0/3936389673 >> v2:172.21.15.103:3300/0 conn(0x7f27a015bc70 msgr2=0x7f27a00cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:44.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.062+0000 7f2797e00640 1 --2- 172.21.15.40:0/3936389673 >> v2:172.21.15.103:3300/0 conn(0x7f27a015bc70 0x7f27a00cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.062+0000 7f2797e00640 1 -- 172.21.15.40:0/3936389673 >> v1:172.21.15.40:6789/0 conn(0x7f27a0072440 legacy=0x7f27a015f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:44.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.062+0000 7f279e000640 1 -- 172.21.15.40:0/3936389673 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1969897 0 0) 0x7f27a006dde0 con 0x7f27a0072440 2024-04-26T23:01:44.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.062+0000 7f2797e00640 1 -- 172.21.15.40:0/3936389673 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a006e0c0 con 0x7f27a0071ef0 2024-04-26T23:01:44.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.063+0000 7f2797e00640 1 --2- 172.21.15.40:0/3936389673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a0071ef0 0x7f27a0072990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f27a00b0640 tx=0x7f278802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e20c22658d3295e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:44.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.064+0000 7f279e000640 1 -- 172.21.15.40:0/3936389673 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2788037070 con 0x7f27a0071ef0 2024-04-26T23:01:44.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.064+0000 7f279e000640 1 -- 172.21.15.40:0/3936389673 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2788032440 con 0x7f27a0071ef0 2024-04-26T23:01:44.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.064+0000 7f279e000640 1 -- 172.21.15.40:0/3936389673 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2788032810 con 0x7f27a0071ef0 2024-04-26T23:01:44.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.064+0000 7f279fe00640 1 -- 172.21.15.40:0/3936389673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a0071ef0 msgr2=0x7f27a0072990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:44.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.064+0000 7f279fe00640 1 --2- 172.21.15.40:0/3936389673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a0071ef0 0x7f27a0072990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f27a00b0640 tx=0x7f278802a1a0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.064+0000 7f279fe00640 1 -- 172.21.15.40:0/3936389673 shutdown_connections 2024-04-26T23:01:44.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.064+0000 7f279fe00640 1 --2- 172.21.15.40:0/3936389673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a0071ef0 0x7f27a0072990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.064+0000 7f279fe00640 1 --2- 172.21.15.40:0/3936389673 >> v2:172.21.15.103:3300/0 conn(0x7f27a015bc70 0x7f27a00cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.064+0000 7f279fe00640 1 -- 172.21.15.40:0/3936389673 >> 172.21.15.40:0/3936389673 conn(0x7f27a00b2720 msgr2=0x7f27a00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:44.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.064+0000 7f279fe00640 1 -- 172.21.15.40:0/3936389673 shutdown_connections 2024-04-26T23:01:44.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.064+0000 7f279fe00640 1 -- 172.21.15.40:0/3936389673 wait complete. 2024-04-26T23:01:44.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.066+0000 7f279fe00640 1 Processor -- start 2024-04-26T23:01:44.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.066+0000 7f279fe00640 1 -- start start 2024-04-26T23:01:44.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.067+0000 7f279fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a0071ef0 0x7f27a0159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:44.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.067+0000 7f279fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27a0072440 0x7f27a015a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:44.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.067+0000 7f279fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015bc70 0x7f27a00d0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:44.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.067+0000 7f279fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f27a015c050 con 0x7f27a015bc70 2024-04-26T23:01:44.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.067+0000 7f279fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f27a006fad0 con 0x7f27a0071ef0 2024-04-26T23:01:44.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.067+0000 7f279fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f27a015c1d0 con 0x7f27a0072440 2024-04-26T23:01:44.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.068+0000 7f279f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015bc70 0x7f27a00d0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:44.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.068+0000 7f279f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015bc70 0x7f27a00d0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39148/0 (socket says 172.21.15.40:39148) 2024-04-26T23:01:44.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.068+0000 7f279f400640 1 -- 172.21.15.40:0/134607149 learned_addr learned my addr 172.21.15.40:0/134607149 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:44.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.068+0000 7f2797e00640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27a0072440 0x7f27a015a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:44.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.068+0000 7f279ea00640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a0071ef0 0x7f27a0159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:44.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.068+0000 7f279f400640 1 -- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27a0072440 msgr2=0x7f27a015a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:44.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.068+0000 7f279f400640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27a0072440 0x7f27a015a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.068+0000 7f279f400640 1 -- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a0071ef0 msgr2=0x7f27a0159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:44.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.068+0000 7f279f400640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a0071ef0 0x7f27a0159d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.068+0000 7f279f400640 1 -- 172.21.15.40:0/134607149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a0149670 con 0x7f27a015bc70 2024-04-26T23:01:44.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.068+0000 7f279f400640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015bc70 0x7f27a00d0c80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f279800c7a0 tx=0x7f279800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:44.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.069+0000 7f2796a00640 1 -- 172.21.15.40:0/134607149 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2798018070 con 0x7f27a015bc70 2024-04-26T23:01:44.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.069+0000 7f2796a00640 1 -- 172.21.15.40:0/134607149 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f279800a640 con 0x7f27a015bc70 2024-04-26T23:01:44.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.069+0000 7f2796a00640 1 -- 172.21.15.40:0/134607149 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279800aab0 con 0x7f27a015bc70 2024-04-26T23:01:44.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.069+0000 7f279fe00640 1 -- 172.21.15.40:0/134607149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a015a8d0 con 0x7f27a015bc70 2024-04-26T23:01:44.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.069+0000 7f279fe00640 1 -- 172.21.15.40:0/134607149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a0149bf0 con 0x7f27a015bc70 2024-04-26T23:01:44.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.070+0000 7f279fe00640 1 -- 172.21.15.40:0/134607149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27a0069370 con 0x7f27a015bc70 2024-04-26T23:01:44.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.072+0000 7f2796a00640 1 -- 172.21.15.40:0/134607149 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f279801b070 con 0x7f27a015bc70 2024-04-26T23:01:44.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.073+0000 7f2796a00640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f277c06c8f0 0x7f277c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:44.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.073+0000 7f2796a00640 1 -- 172.21.15.40:0/134607149 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f279808b490 con 0x7f27a015bc70 2024-04-26T23:01:44.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.073+0000 7f279ea00640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f277c06c8f0 0x7f277c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:44.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.074+0000 7f279ea00640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f277c06c8f0 0x7f277c06edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2790006fd0 tx=0x7f2790008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:44.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.077+0000 7f2796a00640 1 -- 172.21.15.40:0/134607149 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2798059570 con 0x7f27a015bc70 2024-04-26T23:01:44.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.281+0000 7f279fe00640 1 -- 172.21.15.40:0/134607149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f27a0160470 con 0x7f27a015bc70 2024-04-26T23:01:44.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.282+0000 7f2796a00640 1 -- 172.21.15.40:0/134607149 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f2798058e70 con 0x7f27a015bc70 2024-04-26T23:01:44.282 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:01:44.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.285+0000 7f279fe00640 1 -- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f277c06c8f0 msgr2=0x7f277c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:44.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.285+0000 7f279fe00640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f277c06c8f0 0x7f277c06edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2790006fd0 tx=0x7f2790008040 comp rx=0 tx=0).stop 2024-04-26T23:01:44.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.285+0000 7f279fe00640 1 -- 172.21.15.40:0/134607149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015bc70 msgr2=0x7f27a00d0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:44.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.285+0000 7f279fe00640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015bc70 0x7f27a00d0c80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f279800c7a0 tx=0x7f279800cc60 comp rx=0 tx=0).stop 2024-04-26T23:01:44.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.285+0000 7f279fe00640 1 -- 172.21.15.40:0/134607149 shutdown_connections 2024-04-26T23:01:44.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.285+0000 7f279fe00640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f277c06c8f0 0x7f277c06edb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.285+0000 7f279fe00640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015bc70 0x7f27a00d0c80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.285+0000 7f279fe00640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27a0072440 0x7f27a015a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.285+0000 7f279fe00640 1 --2- 172.21.15.40:0/134607149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a0071ef0 0x7f27a0159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.285+0000 7f279fe00640 1 -- 172.21.15.40:0/134607149 >> 172.21.15.40:0/134607149 conn(0x7f27a00b2720 msgr2=0x7f27a00b0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:44.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.286+0000 7f279fe00640 1 -- 172.21.15.40:0/134607149 shutdown_connections 2024-04-26T23:01:44.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.286+0000 7f279fe00640 1 -- 172.21.15.40:0/134607149 wait complete. 2024-04-26T23:01:44.305 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:01:44.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.519+0000 7f4ecd400640 1 Processor -- start 2024-04-26T23:01:44.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.519+0000 7f4ecd400640 1 -- start start 2024-04-26T23:01:44.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.521+0000 7f4ecd400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ec815ce60 0x7f4ec80afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:44.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.521+0000 7f4ecd400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4ec80af7a0 0x7f4ec80ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:44.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.521+0000 7f4ecd400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec80d0820 con 0x7f4ec8071fe0 2024-04-26T23:01:44.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.521+0000 7f4ecd400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f4ec80cc450 con 0x7f4ec815ce60 2024-04-26T23:01:44.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.521+0000 7f4ecd400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4ec81614c0 con 0x7f4ec80af7a0 2024-04-26T23:01:44.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.521+0000 7f4ecca00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4ec8071fe0 0x7f4ec8073760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41650/0 (socket says 172.21.15.40:41650) 2024-04-26T23:01:44.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.521+0000 7f4ecca00640 1 -- 172.21.15.40:0/3581878552 learned_addr learned my addr 172.21.15.40:0/3581878552 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:44.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.521+0000 7f4ec7e00640 1 --2- 172.21.15.40:0/3581878552 >> v2:172.21.15.40:3300/0 conn(0x7f4ec815ce60 0x7f4ec80afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:44.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.521+0000 7f4ec7400640 1 --2- 172.21.15.40:0/3581878552 >> v2:172.21.15.103:3300/0 conn(0x7f4ec80af7a0 0x7f4ec80ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.522+0000 7f4ec6a00640 1 -- 172.21.15.40:0/3581878552 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 372829898 0 0) 0x7f4ec80d0820 con 0x7f4ec8071fe0 2024-04-26T23:01:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.522+0000 7f4ec7e00640 1 -- 172.21.15.40:0/3581878552 >> v2:172.21.15.103:3300/0 conn(0x7f4ec80af7a0 msgr2=0x7f4ec80ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.522+0000 7f4ec7e00640 1 --2- 172.21.15.40:0/3581878552 >> v2:172.21.15.103:3300/0 conn(0x7f4ec80af7a0 0x7f4ec80ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.522+0000 7f4ec7e00640 1 -- 172.21.15.40:0/3581878552 >> v1:172.21.15.40:6789/0 conn(0x7f4ec8071fe0 legacy=0x7f4ec8073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.522+0000 7f4ec7e00640 1 -- 172.21.15.40:0/3581878552 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec8161760 con 0x7f4ec815ce60 2024-04-26T23:01:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.523+0000 7f4ec7e00640 1 --2- 172.21.15.40:0/3581878552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec815ce60 0x7f4ec80afc60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4ec80d1940 tx=0x7f4eb802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a6a7c8d208c77a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.523+0000 7f4ec6a00640 1 -- 172.21.15.40:0/3581878552 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8037070 con 0x7f4ec815ce60 2024-04-26T23:01:44.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.523+0000 7f4ec6a00640 1 -- 172.21.15.40:0/3581878552 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eb8032440 con 0x7f4ec815ce60 2024-04-26T23:01:44.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.523+0000 7f4ec6a00640 1 -- 172.21.15.40:0/3581878552 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb802ae60 con 0x7f4ec815ce60 2024-04-26T23:01:44.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.524+0000 7f4ecd400640 1 -- 172.21.15.40:0/3581878552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec815ce60 msgr2=0x7f4ec80afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:44.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.524+0000 7f4ecd400640 1 --2- 172.21.15.40:0/3581878552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec815ce60 0x7f4ec80afc60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4ec80d1940 tx=0x7f4eb802a110 comp rx=0 tx=0).stop 2024-04-26T23:01:44.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.524+0000 7f4ecd400640 1 -- 172.21.15.40:0/3581878552 shutdown_connections 2024-04-26T23:01:44.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.524+0000 7f4ecd400640 1 --2- 172.21.15.40:0/3581878552 >> v2:172.21.15.103:3300/0 conn(0x7f4ec80af7a0 0x7f4ec80ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.524+0000 7f4ecd400640 1 --2- 172.21.15.40:0/3581878552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec815ce60 0x7f4ec80afc60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.524+0000 7f4ecd400640 1 -- 172.21.15.40:0/3581878552 >> 172.21.15.40:0/3581878552 conn(0x7f4ec80d0a10 msgr2=0x7f4ec80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:44.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.524+0000 7f4ecd400640 1 -- 172.21.15.40:0/3581878552 shutdown_connections 2024-04-26T23:01:44.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.524+0000 7f4ecd400640 1 -- 172.21.15.40:0/3581878552 wait complete. 2024-04-26T23:01:44.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.526+0000 7f4ecd400640 1 Processor -- start 2024-04-26T23:01:44.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.526+0000 7f4ecd400640 1 -- start start 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.527+0000 7f4ecd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec8071fe0 0x7f4ec81f1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.527+0000 7f4ecd400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ec80af7a0 0x7f4ec815fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.527+0000 7f4ecd400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ec815ce60 0x7f4ec81573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.527+0000 7f4ecd400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec8073e70 con 0x7f4ec8071fe0 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.527+0000 7f4ecd400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec80cf530 con 0x7f4ec80af7a0 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.527+0000 7f4ecd400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ec8073ff0 con 0x7f4ec815ce60 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.527+0000 7f4ec7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec8071fe0 0x7f4ec81f1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.527+0000 7f4ec7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec8071fe0 0x7f4ec81f1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39180/0 (socket says 172.21.15.40:39180) 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.527+0000 7f4ec7e00640 1 -- 172.21.15.40:0/2362382443 learned_addr learned my addr 172.21.15.40:0/2362382443 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.528+0000 7f4ec7e00640 1 -- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ec815ce60 msgr2=0x7f4ec81573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.528+0000 7f4ecca00640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ec815ce60 0x7f4ec81573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.528+0000 7f4ec7e00640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ec815ce60 0x7f4ec81573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.528+0000 7f4ec7400640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ec80af7a0 0x7f4ec815fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.528+0000 7f4ec7e00640 1 -- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ec80af7a0 msgr2=0x7f4ec815fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:44.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.528+0000 7f4ec7e00640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ec80af7a0 0x7f4ec815fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.528+0000 7f4ec7e00640 1 -- 172.21.15.40:0/2362382443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec8160250 con 0x7f4ec8071fe0 2024-04-26T23:01:44.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.528+0000 7f4ec7e00640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec8071fe0 0x7f4ec81f1e80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4eb802a700 tx=0x7f4eb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:44.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.528+0000 7f4ebfe00640 1 -- 172.21.15.40:0/2362382443 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8037050 con 0x7f4ec8071fe0 2024-04-26T23:01:44.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.528+0000 7f4ebfe00640 1 -- 172.21.15.40:0/2362382443 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eb8038040 con 0x7f4ec8071fe0 2024-04-26T23:01:44.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.528+0000 7f4ebfe00640 1 -- 172.21.15.40:0/2362382443 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8030bd0 con 0x7f4ec8071fe0 2024-04-26T23:01:44.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.529+0000 7f4ecd400640 1 -- 172.21.15.40:0/2362382443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec815ef50 con 0x7f4ec8071fe0 2024-04-26T23:01:44.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.529+0000 7f4ecd400640 1 -- 172.21.15.40:0/2362382443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec815f510 con 0x7f4ec8071fe0 2024-04-26T23:01:44.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.530+0000 7f4ecd400640 1 -- 172.21.15.40:0/2362382443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ec8069370 con 0x7f4ec8071fe0 2024-04-26T23:01:44.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.531+0000 7f4ebfe00640 1 -- 172.21.15.40:0/2362382443 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4eb803c440 con 0x7f4ec8071fe0 2024-04-26T23:01:44.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.532+0000 7f4ebfe00640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4ea806c8f0 0x7f4ea806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:44.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.532+0000 7f4ebfe00640 1 -- 172.21.15.40:0/2362382443 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4eb80acaa0 con 0x7f4ec8071fe0 2024-04-26T23:01:44.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.532+0000 7f4ec7400640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4ea806c8f0 0x7f4ea806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:44.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.533+0000 7f4ec7400640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4ea806c8f0 0x7f4ea806edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ec815eaf0 tx=0x7f4eb4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:44.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.537+0000 7f4ebfe00640 1 -- 172.21.15.40:0/2362382443 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4ec8160250 con 0x7f4ec8071fe0 2024-04-26T23:01:44.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.784+0000 7f4ecd400640 1 -- 172.21.15.40:0/2362382443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ec80d01f0 con 0x7f4ec8071fe0 2024-04-26T23:01:44.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.785+0000 7f4ebfe00640 1 -- 172.21.15.40:0/2362382443 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f4eb807ab80 con 0x7f4ec8071fe0 2024-04-26T23:01:44.786 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:44.786 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1253648754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1253648754},{"type":"v1","addr":"172.21.15.103:6836","nonce":1253648754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3239652611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":3239652611},{"type":"v1","addr":"172.21.15.103:6837","nonce":3239652611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3670144900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3670144900},{"type":"v1","addr":"172.21.15.40:6833","nonce":3670144900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:01:05.171276+0000","modified":"2024-04-26T23:01:33.399801+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3989190340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3989190340},{"type":"v1","addr":"172.21.15.40:6835","nonce":3989190340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-26T23:01:44.786 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 9 2024-04-26T23:01:44.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.788+0000 7f4ecd400640 1 -- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4ea806c8f0 msgr2=0x7f4ea806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:44.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.788+0000 7f4ecd400640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4ea806c8f0 0x7f4ea806edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ec815eaf0 tx=0x7f4eb4008040 comp rx=0 tx=0).stop 2024-04-26T23:01:44.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.788+0000 7f4ecd400640 1 -- 172.21.15.40:0/2362382443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec8071fe0 msgr2=0x7f4ec81f1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:44.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.788+0000 7f4ecd400640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec8071fe0 0x7f4ec81f1e80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4eb802a700 tx=0x7f4eb80046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.788+0000 7f4ecd400640 1 -- 172.21.15.40:0/2362382443 shutdown_connections 2024-04-26T23:01:44.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.788+0000 7f4ecd400640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4ea806c8f0 0x7f4ea806edb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.788+0000 7f4ecd400640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ec815ce60 0x7f4ec81573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.788+0000 7f4ecd400640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ec80af7a0 0x7f4ec815fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.788+0000 7f4ecd400640 1 --2- 172.21.15.40:0/2362382443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec8071fe0 0x7f4ec81f1e80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:44.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.788+0000 7f4ecd400640 1 -- 172.21.15.40:0/2362382443 >> 172.21.15.40:0/2362382443 conn(0x7f4ec80d0a10 msgr2=0x7f4ec80b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:44.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.789+0000 7f4ecd400640 1 -- 172.21.15.40:0/2362382443 shutdown_connections 2024-04-26T23:01:44.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:44.789+0000 7f4ecd400640 1 -- 172.21.15.40:0/2362382443 wait complete. 2024-04-26T23:01:44.808 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:45.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.020+0000 7f49ce600640 1 Processor -- start 2024-04-26T23:01:45.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.020+0000 7f49ce600640 1 -- start start 2024-04-26T23:01:45.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.021+0000 7f49ce600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f49c815f050 0x7f49c80d1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.021+0000 7f49ce600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49c80d2c20 0x7f49c80d0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.021+0000 7f49ce600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49c80cfa90 con 0x7f49c815f430 2024-04-26T23:01:45.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.021+0000 7f49ce600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f49c80d5ad0 con 0x7f49c815f050 2024-04-26T23:01:45.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.021+0000 7f49ce600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f49c80cfd30 con 0x7f49c80d2c20 2024-04-26T23:01:45.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.021+0000 7f49cd200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f49c815f050 0x7f49c80d1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.021+0000 7f49cdc00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f49c815f430 0x7f49c80d3350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:47898/0 (socket says 172.21.15.40:47898) 2024-04-26T23:01:45.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.021+0000 7f49cdc00640 1 -- 172.21.15.40:0/1913457398 learned_addr learned my addr 172.21.15.40:0/1913457398 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:45.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.022+0000 7f49c7e00640 1 --2- 172.21.15.40:0/1913457398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49c80d2c20 0x7f49c80d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.022+0000 7f49cd200640 1 -- 172.21.15.40:0/1913457398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49c80d2c20 msgr2=0x7f49c80d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:45.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.022+0000 7f49cd200640 1 --2- 172.21.15.40:0/1913457398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49c80d2c20 0x7f49c80d0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.022+0000 7f49cd200640 1 -- 172.21.15.40:0/1913457398 >> v1:172.21.15.40:6789/0 conn(0x7f49c815f430 legacy=0x7f49c80d3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.022+0000 7f49cd200640 1 -- 172.21.15.40:0/1913457398 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c80d0010 con 0x7f49c815f050 2024-04-26T23:01:45.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.022+0000 7f49c7400640 1 -- 172.21.15.40:0/1913457398 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2474600423 0 0) 0x7f49c80cfa90 con 0x7f49c815f430 2024-04-26T23:01:45.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.023+0000 7f49cd200640 1 --2- 172.21.15.40:0/1913457398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c815f050 0x7f49c80d1f00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f49b8009d90 tx=0x7f49b802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430f918c59fc01da server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:45.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.023+0000 7f49c7400640 1 -- 172.21.15.40:0/1913457398 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b8007740 con 0x7f49c815f050 2024-04-26T23:01:45.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.023+0000 7f49c7400640 1 -- 172.21.15.40:0/1913457398 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49b80078e0 con 0x7f49c815f050 2024-04-26T23:01:45.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.023+0000 7f49c7400640 1 -- 172.21.15.40:0/1913457398 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b8007cb0 con 0x7f49c815f050 2024-04-26T23:01:45.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.024+0000 7f49ce600640 1 -- 172.21.15.40:0/1913457398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c815f050 msgr2=0x7f49c80d1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.024+0000 7f49ce600640 1 --2- 172.21.15.40:0/1913457398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c815f050 0x7f49c80d1f00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f49b8009d90 tx=0x7f49b802a6f0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.024+0000 7f49ce600640 1 -- 172.21.15.40:0/1913457398 shutdown_connections 2024-04-26T23:01:45.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.024+0000 7f49ce600640 1 --2- 172.21.15.40:0/1913457398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49c80d2c20 0x7f49c80d0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.024+0000 7f49ce600640 1 --2- 172.21.15.40:0/1913457398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c815f050 0x7f49c80d1f00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.024+0000 7f49ce600640 1 -- 172.21.15.40:0/1913457398 >> 172.21.15.40:0/1913457398 conn(0x7f49c809e140 msgr2=0x7f49c80d4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:45.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.024+0000 7f49ce600640 1 -- 172.21.15.40:0/1913457398 shutdown_connections 2024-04-26T23:01:45.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.024+0000 7f49ce600640 1 -- 172.21.15.40:0/1913457398 wait complete. 2024-04-26T23:01:45.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.026+0000 7f49ce600640 1 Processor -- start 2024-04-26T23:01:45.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.026+0000 7f49ce600640 1 -- start start 2024-04-26T23:01:45.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.027+0000 7f49ce600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c80d2c20 0x7f49c80b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.027+0000 7f49ce600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c815f430 0x7f49c8152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.027+0000 7f49ce600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49c80b4730 0x7f49c80b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.027+0000 7f49ce600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f49c80d0d50 con 0x7f49c815f430 2024-04-26T23:01:45.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.027+0000 7f49ce600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f49c80d56f0 con 0x7f49c80d2c20 2024-04-26T23:01:45.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.027+0000 7f49ce600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f49c80cceb0 con 0x7f49c80b4730 2024-04-26T23:01:45.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.027+0000 7f49c7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c815f430 0x7f49c8152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.027+0000 7f49c7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c815f430 0x7f49c8152960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48244/0 (socket says 172.21.15.40:48244) 2024-04-26T23:01:45.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.027+0000 7f49c7e00640 1 -- 172.21.15.40:0/1394600662 learned_addr learned my addr 172.21.15.40:0/1394600662 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:45.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.028+0000 7f49cdc00640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49c80b4730 0x7f49c80b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.028+0000 7f49c7e00640 1 -- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49c80b4730 msgr2=0x7f49c80b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.028+0000 7f49c7e00640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49c80b4730 0x7f49c80b4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.028+0000 7f49c7e00640 1 -- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c80d2c20 msgr2=0x7f49c80b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.028+0000 7f49cd200640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c80d2c20 0x7f49c80b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.028+0000 7f49c7e00640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c80d2c20 0x7f49c80b71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.028+0000 7f49c7e00640 1 -- 172.21.15.40:0/1394600662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c8162170 con 0x7f49c815f430 2024-04-26T23:01:45.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.028+0000 7f49cd200640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c80d2c20 0x7f49c80b71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:45.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.028+0000 7f49c7e00640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c815f430 0x7f49c8152960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f49b400d7d0 tx=0x7f49b400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:45.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.029+0000 7f49c4c00640 1 -- 172.21.15.40:0/1394600662 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b4004020 con 0x7f49c815f430 2024-04-26T23:01:45.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.029+0000 7f49c4c00640 1 -- 172.21.15.40:0/1394600662 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49b40041e0 con 0x7f49c815f430 2024-04-26T23:01:45.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.029+0000 7f49ce600640 1 -- 172.21.15.40:0/1394600662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49c8162450 con 0x7f49c815f430 2024-04-26T23:01:45.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.029+0000 7f49c4c00640 1 -- 172.21.15.40:0/1394600662 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b40043d0 con 0x7f49c815f430 2024-04-26T23:01:45.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.029+0000 7f49ce600640 1 -- 172.21.15.40:0/1394600662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49c80cd3e0 con 0x7f49c815f430 2024-04-26T23:01:45.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.030+0000 7f49ce600640 1 -- 172.21.15.40:0/1394600662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49c8162620 con 0x7f49c815f430 2024-04-26T23:01:45.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.031+0000 7f49c4c00640 1 -- 172.21.15.40:0/1394600662 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f49b401b020 con 0x7f49c815f430 2024-04-26T23:01:45.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.032+0000 7f49c4c00640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f49a806c8f0 0x7f49a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.032+0000 7f49c4c00640 1 -- 172.21.15.40:0/1394600662 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f49b408bc00 con 0x7f49c815f430 2024-04-26T23:01:45.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.033+0000 7f49cd200640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f49a806c8f0 0x7f49a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.034+0000 7f49cd200640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f49a806c8f0 0x7f49a806edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f49b8034000 tx=0x7f49b802f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:45.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.037+0000 7f49c4c00640 1 -- 172.21.15.40:0/1394600662 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f49b4092080 con 0x7f49c815f430 2024-04-26T23:01:45.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.242+0000 7f49ce600640 1 -- 172.21.15.40:0/1394600662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49c8069370 con 0x7f49c815f430 2024-04-26T23:01:45.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.244+0000 7f49c4c00640 1 -- 172.21.15.40:0/1394600662 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f49b4059d80 con 0x7f49c815f430 2024-04-26T23:01:45.244 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:45.245 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:11.763965+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:45.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.247+0000 7f49ce600640 1 -- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f49a806c8f0 msgr2=0x7f49a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.247+0000 7f49ce600640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f49a806c8f0 0x7f49a806edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f49b8034000 tx=0x7f49b802f000 comp rx=0 tx=0).stop 2024-04-26T23:01:45.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.247+0000 7f49ce600640 1 -- 172.21.15.40:0/1394600662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c815f430 msgr2=0x7f49c8152960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.247+0000 7f49ce600640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c815f430 0x7f49c8152960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f49b400d7d0 tx=0x7f49b400dc90 comp rx=0 tx=0).stop 2024-04-26T23:01:45.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.247+0000 7f49ce600640 1 -- 172.21.15.40:0/1394600662 shutdown_connections 2024-04-26T23:01:45.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.247+0000 7f49ce600640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f49a806c8f0 0x7f49a806edb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.247+0000 7f49ce600640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49c80b4730 0x7f49c80b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.247+0000 7f49ce600640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c815f430 0x7f49c8152960 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.247+0000 7f49ce600640 1 --2- 172.21.15.40:0/1394600662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c80d2c20 0x7f49c80b71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.247+0000 7f49ce600640 1 -- 172.21.15.40:0/1394600662 >> 172.21.15.40:0/1394600662 conn(0x7f49c809e140 msgr2=0x7f49c80b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:45.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.247+0000 7f49ce600640 1 -- 172.21.15.40:0/1394600662 shutdown_connections 2024-04-26T23:01:45.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.247+0000 7f49ce600640 1 -- 172.21.15.40:0/1394600662 wait complete. 2024-04-26T23:01:45.267 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:01:45.268 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:01:45.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.482+0000 7f3733e00640 1 Processor -- start 2024-04-26T23:01:45.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.482+0000 7f3733e00640 1 -- start start 2024-04-26T23:01:45.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.483+0000 7f3733e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f373415bc70 0x7f37340cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.483+0000 7f3733e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f373406dde0 con 0x7f37340730a0 2024-04-26T23:01:45.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.483+0000 7f3733e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f373406e0c0 con 0x7f3734071ef0 2024-04-26T23:01:45.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.483+0000 7f3733e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f373406e3a0 con 0x7f373415bc70 2024-04-26T23:01:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.484+0000 7f3733400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f37340730a0 0x7f37340b4cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:47906/0 (socket says 172.21.15.40:47906) 2024-04-26T23:01:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.484+0000 7f3733400640 1 -- 172.21.15.40:0/2265275613 learned_addr learned my addr 172.21.15.40:0/2265275613 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.484+0000 7f3732a00640 1 --2- 172.21.15.40:0/2265275613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f373415bc70 0x7f37340cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.484+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2286176709 0 0) 0x7f373406dde0 con 0x7f37340730a0 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.484+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3718003520 con 0x7f37340730a0 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 376998976 0 0) 0x7f3718003520 con 0x7f37340730a0 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f373415bc70 msgr2=0x7f37340cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3731600640 1 --2- 172.21.15.40:0/2265275613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f373415bc70 0x7f37340cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 >> v1:172.21.15.103:6789/0 conn(0x7f3734071ef0 legacy=0x7f3734072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37340b44d0 con 0x7f37340730a0 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f372c0019f0 con 0x7f37340730a0 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 >> v1:172.21.15.40:6789/0 conn(0x7f37340730a0 legacy=0x7f37340b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3731600640 1 --2- 172.21.15.40:0/2265275613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f373415bc70 0x7f37180039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3731600640 1 --2- 172.21.15.40:0/2265275613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3718003fb0 0x7f3718004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3731600640 1 --2- 172.21.15.40:0/2265275613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3718004d40 0x7f37180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f373406e3a0 con 0x7f3718003fb0 2024-04-26T23:01:45.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f371800bb00 con 0x7f3718004d40 2024-04-26T23:01:45.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f371800df10 con 0x7f373415bc70 2024-04-26T23:01:45.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3732000640 1 --2- 172.21.15.40:0/2265275613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3718003fb0 0x7f3718004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.485+0000 7f3732a00640 1 --2- 172.21.15.40:0/2265275613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f373415bc70 0x7f37180039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.486+0000 7f3733400640 1 --2- 172.21.15.40:0/2265275613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3718004d40 0x7f37180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.486+0000 7f3732000640 1 -- 172.21.15.40:0/2265275613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f373415bc70 msgr2=0x7f37180039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:45.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.486+0000 7f3732000640 1 --2- 172.21.15.40:0/2265275613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f373415bc70 0x7f37180039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.486+0000 7f3732000640 1 -- 172.21.15.40:0/2265275613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3718004d40 msgr2=0x7f37180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.486+0000 7f3732000640 1 --2- 172.21.15.40:0/2265275613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3718004d40 0x7f37180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.486+0000 7f3732000640 1 -- 172.21.15.40:0/2265275613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3718003520 con 0x7f3718003fb0 2024-04-26T23:01:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.487+0000 7f3732000640 1 --2- 172.21.15.40:0/2265275613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3718003fb0 0x7f3718004800 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3720001750 tx=0x7f3718005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d47d681999d192 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.487+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3720001370 con 0x7f3718003fb0 2024-04-26T23:01:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.487+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37200052d0 con 0x7f3718003fb0 2024-04-26T23:01:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.487+0000 7f3731600640 1 -- 172.21.15.40:0/2265275613 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372002ab20 con 0x7f3718003fb0 2024-04-26T23:01:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.488+0000 7f3733e00640 1 -- 172.21.15.40:0/2265275613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3718003fb0 msgr2=0x7f3718004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.488+0000 7f3733e00640 1 --2- 172.21.15.40:0/2265275613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3718003fb0 0x7f3718004800 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3720001750 tx=0x7f3718005830 comp rx=0 tx=0).stop 2024-04-26T23:01:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.488+0000 7f3733400640 1 -- 172.21.15.40:0/2265275613 reap_dead start 2024-04-26T23:01:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.488+0000 7f3733e00640 1 -- 172.21.15.40:0/2265275613 shutdown_connections 2024-04-26T23:01:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.488+0000 7f3733e00640 1 -- 172.21.15.40:0/2265275613 >> 172.21.15.40:0/2265275613 conn(0x7f37340b2720 msgr2=0x7f37340b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.488+0000 7f3733e00640 1 -- 172.21.15.40:0/2265275613 shutdown_connections 2024-04-26T23:01:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.488+0000 7f3733e00640 1 -- 172.21.15.40:0/2265275613 wait complete. 2024-04-26T23:01:45.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.490+0000 7f3733e00640 1 Processor -- start 2024-04-26T23:01:45.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.490+0000 7f3733e00640 1 -- start start 2024-04-26T23:01:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.491+0000 7f3733e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3718003fb0 0x7f373414b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.491+0000 7f3733e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3734150fb0 0x7f37341505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.491+0000 7f3733e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3734150af0 0x7f3734155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.491+0000 7f3733e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f373415c050 con 0x7f3734150af0 2024-04-26T23:01:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.491+0000 7f3733e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f37340b4840 con 0x7f3734150fb0 2024-04-26T23:01:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.491+0000 7f3733e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f373415c1d0 con 0x7f3718003fb0 2024-04-26T23:01:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.491+0000 7f3733400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3734150af0 0x7f3734155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.491+0000 7f3733400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3734150af0 0x7f3734155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48256/0 (socket says 172.21.15.40:48256) 2024-04-26T23:01:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.491+0000 7f3733400640 1 -- 172.21.15.40:0/1810203935 learned_addr learned my addr 172.21.15.40:0/1810203935 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:45.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.492+0000 7f3732000640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3734150fb0 0x7f37341505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.492+0000 7f3732a00640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3718003fb0 0x7f373414b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.492+0000 7f3733400640 1 -- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3718003fb0 msgr2=0x7f373414b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.492+0000 7f3733400640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3718003fb0 0x7f373414b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.492+0000 7f3733400640 1 -- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3734150fb0 msgr2=0x7f37341505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.492+0000 7f3733400640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3734150fb0 0x7f37341505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.492+0000 7f3733400640 1 -- 172.21.15.40:0/1810203935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3734151970 con 0x7f3734150af0 2024-04-26T23:01:45.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.492+0000 7f3733400640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3734150af0 0x7f3734155830 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f372c002230 tx=0x7f372c007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.492+0000 7f372aa00640 1 -- 172.21.15.40:0/1810203935 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c00abf0 con 0x7f3734150af0 2024-04-26T23:01:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.492+0000 7f372aa00640 1 -- 172.21.15.40:0/1810203935 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f372c018070 con 0x7f3734150af0 2024-04-26T23:01:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.492+0000 7f372aa00640 1 -- 172.21.15.40:0/1810203935 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c0133f0 con 0x7f3734150af0 2024-04-26T23:01:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.493+0000 7f3733e00640 1 -- 172.21.15.40:0/1810203935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f373414fad0 con 0x7f3734150af0 2024-04-26T23:01:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.493+0000 7f3733e00640 1 -- 172.21.15.40:0/1810203935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f373414b9f0 con 0x7f3734150af0 2024-04-26T23:01:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.494+0000 7f3733e00640 1 -- 172.21.15.40:0/1810203935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3734070650 con 0x7f3734150af0 2024-04-26T23:01:45.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.495+0000 7f372aa00640 1 -- 172.21.15.40:0/1810203935 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f372c013590 con 0x7f3734150af0 2024-04-26T23:01:45.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.496+0000 7f372aa00640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f371406c8f0 0x7f371406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.496+0000 7f372aa00640 1 -- 172.21.15.40:0/1810203935 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f372c08c5b0 con 0x7f3734150af0 2024-04-26T23:01:45.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.496+0000 7f3732a00640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f371406c8f0 0x7f371406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.497+0000 7f3732a00640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f371406c8f0 0x7f371406edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f37340b3ea0 tx=0x7f3724008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:45.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.500+0000 7f372aa00640 1 -- 172.21.15.40:0/1810203935 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3734070650 con 0x7f3734150af0 2024-04-26T23:01:45.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.703+0000 7f3733e00640 1 -- 172.21.15.40:0/1810203935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f373414f8c0 con 0x7f3734150af0 2024-04-26T23:01:45.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.703+0000 7f372aa00640 1 -- 172.21.15.40:0/1810203935 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f372c05a600 con 0x7f3734150af0 2024-04-26T23:01:45.704 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:01:45.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.706+0000 7f3733e00640 1 -- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f371406c8f0 msgr2=0x7f371406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.706+0000 7f3733e00640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f371406c8f0 0x7f371406edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f37340b3ea0 tx=0x7f3724008040 comp rx=0 tx=0).stop 2024-04-26T23:01:45.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.706+0000 7f3733e00640 1 -- 172.21.15.40:0/1810203935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3734150af0 msgr2=0x7f3734155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.706+0000 7f3733e00640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3734150af0 0x7f3734155830 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f372c002230 tx=0x7f372c007bd0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.706+0000 7f3733e00640 1 -- 172.21.15.40:0/1810203935 shutdown_connections 2024-04-26T23:01:45.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.706+0000 7f3733e00640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f371406c8f0 0x7f371406edb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.706+0000 7f3733e00640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3734150af0 0x7f3734155830 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.706+0000 7f3733e00640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3734150fb0 0x7f37341505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.706+0000 7f3733e00640 1 --2- 172.21.15.40:0/1810203935 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3718003fb0 0x7f373414b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.706+0000 7f3733e00640 1 -- 172.21.15.40:0/1810203935 >> 172.21.15.40:0/1810203935 conn(0x7f37340b2720 msgr2=0x7f37340b1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:45.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.707+0000 7f3733e00640 1 -- 172.21.15.40:0/1810203935 shutdown_connections 2024-04-26T23:01:45.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.707+0000 7f3733e00640 1 -- 172.21.15.40:0/1810203935 wait complete. 2024-04-26T23:01:45.727 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:01:45.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.938+0000 7ff662400640 1 Processor -- start 2024-04-26T23:01:45.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.938+0000 7ff662400640 1 -- start start 2024-04-26T23:01:45.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.939+0000 7ff662400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff65c071e60 0x7ff65c06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.939+0000 7ff662400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff65c0cc480 0x7ff65c166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.939+0000 7ff662400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff65c15d7c0 con 0x7ff65c06fe20 2024-04-26T23:01:45.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.939+0000 7ff662400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff65c15daa0 con 0x7ff65c071e60 2024-04-26T23:01:45.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.939+0000 7ff662400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff65c15dd80 con 0x7ff65c0cc480 2024-04-26T23:01:45.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.940+0000 7ff65b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff65c071e60 0x7ff65c06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.940+0000 7ff65b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff65c071e60 0x7ff65c06f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48260/0 (socket says 172.21.15.40:48260) 2024-04-26T23:01:45.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.940+0000 7ff65b400640 1 -- 172.21.15.40:0/1735985614 learned_addr learned my addr 172.21.15.40:0/1735985614 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:45.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.940+0000 7ff65b400640 1 -- 172.21.15.40:0/1735985614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff65c0cc480 msgr2=0x7ff65c166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:45.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.940+0000 7ff65be00640 1 --2- 172.21.15.40:0/1735985614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff65c0cc480 0x7ff65c166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.940+0000 7ff65aa00640 1 -- 172.21.15.40:0/1735985614 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2735255706 0 0) 0x7ff65c15d7c0 con 0x7ff65c06fe20 2024-04-26T23:01:45.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.940+0000 7ff65b400640 1 --2- 172.21.15.40:0/1735985614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff65c0cc480 0x7ff65c166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.940+0000 7ff65b400640 1 -- 172.21.15.40:0/1735985614 >> v1:172.21.15.40:6789/0 conn(0x7ff65c06fe20 legacy=0x7ff65c072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.940+0000 7ff65b400640 1 -- 172.21.15.40:0/1735985614 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65c15e060 con 0x7ff65c071e60 2024-04-26T23:01:45.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.941+0000 7ff65be00640 1 --2- 172.21.15.40:0/1735985614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff65c0cc480 0x7ff65c166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:45.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.942+0000 7ff65b400640 1 --2- 172.21.15.40:0/1735985614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65c071e60 0x7ff65c06f8e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009870 tx=0x7ff64c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67333235077f962 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:45.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.942+0000 7ff65aa00640 1 -- 172.21.15.40:0/1735985614 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c037070 con 0x7ff65c071e60 2024-04-26T23:01:45.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.942+0000 7ff65aa00640 1 -- 172.21.15.40:0/1735985614 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff64c032440 con 0x7ff65c071e60 2024-04-26T23:01:45.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.942+0000 7ff65aa00640 1 -- 172.21.15.40:0/1735985614 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c032810 con 0x7ff65c071e60 2024-04-26T23:01:45.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.942+0000 7ff662400640 1 -- 172.21.15.40:0/1735985614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65c071e60 msgr2=0x7ff65c06f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.942+0000 7ff662400640 1 --2- 172.21.15.40:0/1735985614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65c071e60 0x7ff65c06f8e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009870 tx=0x7ff64c02a250 comp rx=0 tx=0).stop 2024-04-26T23:01:45.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.943+0000 7ff662400640 1 -- 172.21.15.40:0/1735985614 shutdown_connections 2024-04-26T23:01:45.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.943+0000 7ff662400640 1 --2- 172.21.15.40:0/1735985614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff65c0cc480 0x7ff65c166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.943+0000 7ff662400640 1 --2- 172.21.15.40:0/1735985614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65c071e60 0x7ff65c06f8e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.943+0000 7ff662400640 1 -- 172.21.15.40:0/1735985614 >> 172.21.15.40:0/1735985614 conn(0x7ff65c0d0be0 msgr2=0x7ff65c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:45.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.943+0000 7ff662400640 1 -- 172.21.15.40:0/1735985614 shutdown_connections 2024-04-26T23:01:45.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.943+0000 7ff662400640 1 -- 172.21.15.40:0/1735985614 wait complete. 2024-04-26T23:01:45.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.944+0000 7ff662400640 1 Processor -- start 2024-04-26T23:01:45.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.945+0000 7ff662400640 1 -- start start 2024-04-26T23:01:45.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff662400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff65c06fe20 0x7ff65c148860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff662400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff65c071e60 0x7ff65c14b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff662400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65c0cc480 0x7ff65c0b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff662400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c15d570 con 0x7ff65c0cc480 2024-04-26T23:01:45.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff662400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c15d3f0 con 0x7ff65c071e60 2024-04-26T23:01:45.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff662400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff65c072950 con 0x7ff65c06fe20 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff65be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65c0cc480 0x7ff65c0b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff65be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65c0cc480 0x7ff65c0b0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48268/0 (socket says 172.21.15.40:48268) 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff65be00640 1 -- 172.21.15.40:0/4038888549 learned_addr learned my addr 172.21.15.40:0/4038888549 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff65b400640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff65c06fe20 0x7ff65c148860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff653e00640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff65c071e60 0x7ff65c14b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff65be00640 1 -- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff65c06fe20 msgr2=0x7ff65c148860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff65be00640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff65c06fe20 0x7ff65c148860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff65be00640 1 -- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff65c071e60 msgr2=0x7ff65c14b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff65be00640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff65c071e60 0x7ff65c14b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.946+0000 7ff65be00640 1 -- 172.21.15.40:0/4038888549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65c0c7cb0 con 0x7ff65c0cc480 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.947+0000 7ff65be00640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65c0cc480 0x7ff65c0b0700 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff65400cd80 tx=0x7ff65400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.947+0000 7ff653400640 1 -- 172.21.15.40:0/4038888549 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff654003b90 con 0x7ff65c0cc480 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.947+0000 7ff653400640 1 -- 172.21.15.40:0/4038888549 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff654003d50 con 0x7ff65c0cc480 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.947+0000 7ff662400640 1 -- 172.21.15.40:0/4038888549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff65c14c9e0 con 0x7ff65c0cc480 2024-04-26T23:01:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.947+0000 7ff653400640 1 -- 172.21.15.40:0/4038888549 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff654004260 con 0x7ff65c0cc480 2024-04-26T23:01:45.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.947+0000 7ff662400640 1 -- 172.21.15.40:0/4038888549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff65c14c420 con 0x7ff65c0cc480 2024-04-26T23:01:45.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.948+0000 7ff662400640 1 -- 172.21.15.40:0/4038888549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff65c0694f0 con 0x7ff65c0cc480 2024-04-26T23:01:45.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.949+0000 7ff653400640 1 -- 172.21.15.40:0/4038888549 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff654004400 con 0x7ff65c0cc480 2024-04-26T23:01:45.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.950+0000 7ff653400640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff63806c920 0x7ff63806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:45.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.951+0000 7ff653400640 1 -- 172.21.15.40:0/4038888549 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff65408c8f0 con 0x7ff65c0cc480 2024-04-26T23:01:45.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.951+0000 7ff65b400640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff63806c920 0x7ff63806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:45.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.952+0000 7ff65b400640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff63806c920 0x7ff63806ede0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff64c034000 tx=0x7ff64c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:45.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:45.955+0000 7ff653400640 1 -- 172.21.15.40:0/4038888549 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff654014030 con 0x7ff65c0cc480 2024-04-26T23:01:46.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.170+0000 7ff662400640 1 -- 172.21.15.40:0/4038888549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff65c06e6f0 con 0x7ff65c0cc480 2024-04-26T23:01:46.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.170+0000 7ff653400640 1 -- 172.21.15.40:0/4038888549 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7ff65405a9e0 con 0x7ff65c0cc480 2024-04-26T23:01:46.171 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:01:46.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.173+0000 7ff662400640 1 -- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff63806c920 msgr2=0x7ff63806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:46.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.173+0000 7ff662400640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff63806c920 0x7ff63806ede0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff64c034000 tx=0x7ff64c02e000 comp rx=0 tx=0).stop 2024-04-26T23:01:46.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.173+0000 7ff662400640 1 -- 172.21.15.40:0/4038888549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65c0cc480 msgr2=0x7ff65c0b0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:46.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.173+0000 7ff662400640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65c0cc480 0x7ff65c0b0700 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff65400cd80 tx=0x7ff65400a650 comp rx=0 tx=0).stop 2024-04-26T23:01:46.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.173+0000 7ff662400640 1 -- 172.21.15.40:0/4038888549 shutdown_connections 2024-04-26T23:01:46.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.173+0000 7ff662400640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff63806c920 0x7ff63806ede0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.173+0000 7ff662400640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65c0cc480 0x7ff65c0b0700 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.173+0000 7ff662400640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff65c071e60 0x7ff65c14b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.173+0000 7ff662400640 1 --2- 172.21.15.40:0/4038888549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff65c06fe20 0x7ff65c148860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.173+0000 7ff662400640 1 -- 172.21.15.40:0/4038888549 >> 172.21.15.40:0/4038888549 conn(0x7ff65c0d0be0 msgr2=0x7ff65c073b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:46.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.174+0000 7ff662400640 1 -- 172.21.15.40:0/4038888549 shutdown_connections 2024-04-26T23:01:46.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.174+0000 7ff662400640 1 -- 172.21.15.40:0/4038888549 wait complete. 2024-04-26T23:01:46.193 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:01:46.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.405+0000 7f4bfa800640 1 Processor -- start 2024-04-26T23:01:46.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.405+0000 7f4bfa800640 1 -- start start 2024-04-26T23:01:46.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.407+0000 7f4bfa800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4bf415ce60 0x7f4bf40afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.407+0000 7f4bfa800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4bf40af7a0 0x7f4bf40ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.407+0000 7f4bfa800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bf40d0820 con 0x7f4bf4071fe0 2024-04-26T23:01:46.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.407+0000 7f4bfa800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f4bf40cc450 con 0x7f4bf40af7a0 2024-04-26T23:01:46.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.407+0000 7f4bfa800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4bf41614c0 con 0x7f4bf415ce60 2024-04-26T23:01:46.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.408+0000 7f4bf8a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4bf40af7a0 0x7f4bf40ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.408+0000 7f4bf8a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4bf40af7a0 0x7f4bf40ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48282/0 (socket says 172.21.15.40:48282) 2024-04-26T23:01:46.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.408+0000 7f4bf8a00640 1 -- 172.21.15.40:0/1178711645 learned_addr learned my addr 172.21.15.40:0/1178711645 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:46.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.408+0000 7f4bf9400640 1 --2- 172.21.15.40:0/1178711645 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4bf415ce60 0x7f4bf40afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.408+0000 7f4bf8a00640 1 -- 172.21.15.40:0/1178711645 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4bf415ce60 msgr2=0x7f4bf40afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:46.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.408+0000 7f4bf8a00640 1 --2- 172.21.15.40:0/1178711645 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4bf415ce60 0x7f4bf40afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.408+0000 7f4bf8a00640 1 -- 172.21.15.40:0/1178711645 >> v1:172.21.15.40:6789/0 conn(0x7f4bf4071fe0 legacy=0x7f4bf4073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:46.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.408+0000 7f4bf8a00640 1 -- 172.21.15.40:0/1178711645 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf4161760 con 0x7f4bf40af7a0 2024-04-26T23:01:46.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.409+0000 7f4bf8a00640 1 --2- 172.21.15.40:0/1178711645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bf40af7a0 0x7f4bf40ceff0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4be4009870 tx=0x7f4be402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139ab5c355bbeac6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:46.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.410+0000 7f4bf3e00640 1 -- 172.21.15.40:0/1178711645 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4037070 con 0x7f4bf40af7a0 2024-04-26T23:01:46.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.410+0000 7f4bf3e00640 1 -- 172.21.15.40:0/1178711645 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be4032440 con 0x7f4bf40af7a0 2024-04-26T23:01:46.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.410+0000 7f4bf3e00640 1 -- 172.21.15.40:0/1178711645 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4032810 con 0x7f4bf40af7a0 2024-04-26T23:01:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.410+0000 7f4bfa800640 1 -- 172.21.15.40:0/1178711645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bf40af7a0 msgr2=0x7f4bf40ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.410+0000 7f4bfa800640 1 --2- 172.21.15.40:0/1178711645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bf40af7a0 0x7f4bf40ceff0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4be4009870 tx=0x7f4be402a1c0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.410+0000 7f4bfa800640 1 -- 172.21.15.40:0/1178711645 shutdown_connections 2024-04-26T23:01:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.410+0000 7f4bfa800640 1 --2- 172.21.15.40:0/1178711645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bf40af7a0 0x7f4bf40ceff0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.410+0000 7f4bfa800640 1 --2- 172.21.15.40:0/1178711645 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4bf415ce60 0x7f4bf40afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.410+0000 7f4bfa800640 1 -- 172.21.15.40:0/1178711645 >> 172.21.15.40:0/1178711645 conn(0x7f4bf40d0a10 msgr2=0x7f4bf40d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.410+0000 7f4bfa800640 1 -- 172.21.15.40:0/1178711645 shutdown_connections 2024-04-26T23:01:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.410+0000 7f4bfa800640 1 -- 172.21.15.40:0/1178711645 wait complete. 2024-04-26T23:01:46.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.412+0000 7f4bfa800640 1 Processor -- start 2024-04-26T23:01:46.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.412+0000 7f4bfa800640 1 -- start start 2024-04-26T23:01:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.413+0000 7f4bfa800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bf4071fe0 0x7f4bf41526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.413+0000 7f4bfa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bf40af7a0 0x7f4bf40ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.413+0000 7f4bfa800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4bf415ce60 0x7f4bf40aee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.413+0000 7f4bfa800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf4073e70 con 0x7f4bf40af7a0 2024-04-26T23:01:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.413+0000 7f4bfa800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf40cf530 con 0x7f4bf4071fe0 2024-04-26T23:01:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.414+0000 7f4bfa800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4bf4073ff0 con 0x7f4bf415ce60 2024-04-26T23:01:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.414+0000 7f4bf8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bf40af7a0 0x7f4bf40ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.414+0000 7f4bf8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bf40af7a0 0x7f4bf40ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48292/0 (socket says 172.21.15.40:48292) 2024-04-26T23:01:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.414+0000 7f4bf8a00640 1 -- 172.21.15.40:0/3469474803 learned_addr learned my addr 172.21.15.40:0/3469474803 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.414+0000 7f4bf9400640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bf4071fe0 0x7f4bf41526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.414+0000 7f4bf9e00640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4bf415ce60 0x7f4bf40aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.414+0000 7f4bf8a00640 1 -- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4bf415ce60 msgr2=0x7f4bf40aee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.414+0000 7f4bf8a00640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4bf415ce60 0x7f4bf40aee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.414+0000 7f4bf8a00640 1 -- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bf4071fe0 msgr2=0x7f4bf41526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.414+0000 7f4bf8a00640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bf4071fe0 0x7f4bf41526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.414+0000 7f4bf8a00640 1 -- 172.21.15.40:0/3469474803 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf41f50d0 con 0x7f4bf40af7a0 2024-04-26T23:01:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.415+0000 7f4bf8a00640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bf40af7a0 0x7f4bf40ae8c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4be4000c00 tx=0x7f4be40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.415+0000 7f4bf1600640 1 -- 172.21.15.40:0/3469474803 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4037040 con 0x7f4bf40af7a0 2024-04-26T23:01:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.415+0000 7f4bf1600640 1 -- 172.21.15.40:0/3469474803 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be4042070 con 0x7f4bf40af7a0 2024-04-26T23:01:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.415+0000 7f4bf1600640 1 -- 172.21.15.40:0/3469474803 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4030900 con 0x7f4bf40af7a0 2024-04-26T23:01:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.415+0000 7f4bfa800640 1 -- 172.21.15.40:0/3469474803 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf41f53b0 con 0x7f4bf40af7a0 2024-04-26T23:01:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.415+0000 7f4bfa800640 1 -- 172.21.15.40:0/3469474803 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf41f5970 con 0x7f4bf40af7a0 2024-04-26T23:01:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.417+0000 7f4bfa800640 1 -- 172.21.15.40:0/3469474803 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf4069370 con 0x7f4bf40af7a0 2024-04-26T23:01:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.418+0000 7f4bf1600640 1 -- 172.21.15.40:0/3469474803 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4be402ad10 con 0x7f4bf40af7a0 2024-04-26T23:01:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.420+0000 7f4bf1600640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bd806c8f0 0x7f4bd806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.420+0000 7f4bf1600640 1 -- 172.21.15.40:0/3469474803 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4be40ac5b0 con 0x7f4bf40af7a0 2024-04-26T23:01:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.420+0000 7f4bf9400640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bd806c8f0 0x7f4bd806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.421+0000 7f4bf9400640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bd806c8f0 0x7f4bd806edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4bf40af560 tx=0x7f4be8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.424+0000 7f4bf1600640 1 -- 172.21.15.40:0/3469474803 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4be407a540 con 0x7f4bf40af7a0 2024-04-26T23:01:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.628+0000 7f4bfa800640 1 -- 172.21.15.40:0/3469474803 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4bf4069a80 con 0x7f4bf40af7a0 2024-04-26T23:01:46.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.630+0000 7f4bf1600640 1 -- 172.21.15.40:0/3469474803 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f4be4079e40 con 0x7f4bf40af7a0 2024-04-26T23:01:46.630 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:01:46.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.632+0000 7f4bfa800640 1 -- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bd806c8f0 msgr2=0x7f4bd806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:46.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.632+0000 7f4bfa800640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bd806c8f0 0x7f4bd806edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4bf40af560 tx=0x7f4be8008040 comp rx=0 tx=0).stop 2024-04-26T23:01:46.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.632+0000 7f4bfa800640 1 -- 172.21.15.40:0/3469474803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bf40af7a0 msgr2=0x7f4bf40ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:46.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.632+0000 7f4bfa800640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bf40af7a0 0x7f4bf40ae8c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4be4000c00 tx=0x7f4be40046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.632+0000 7f4bfa800640 1 -- 172.21.15.40:0/3469474803 shutdown_connections 2024-04-26T23:01:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.632+0000 7f4bfa800640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bd806c8f0 0x7f4bd806edb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.632+0000 7f4bfa800640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4bf415ce60 0x7f4bf40aee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.632+0000 7f4bfa800640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bf40af7a0 0x7f4bf40ae8c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.632+0000 7f4bfa800640 1 --2- 172.21.15.40:0/3469474803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bf4071fe0 0x7f4bf41526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.632+0000 7f4bfa800640 1 -- 172.21.15.40:0/3469474803 >> 172.21.15.40:0/3469474803 conn(0x7f4bf40d0a10 msgr2=0x7f4bf4160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.633+0000 7f4bfa800640 1 -- 172.21.15.40:0/3469474803 shutdown_connections 2024-04-26T23:01:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.633+0000 7f4bfa800640 1 -- 172.21.15.40:0/3469474803 wait complete. 2024-04-26T23:01:46.653 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:01:46.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.873+0000 7f95e9000640 1 Processor -- start 2024-04-26T23:01:46.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.873+0000 7f95e9000640 1 -- start start 2024-04-26T23:01:46.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.875+0000 7f95e9000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f95e406c4b0 0x7f95e406c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.875+0000 7f95e9000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e40cf6c0 con 0x7f95e406ff20 2024-04-26T23:01:46.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.875+0000 7f95e9000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e40cf9a0 con 0x7f95e40cda50 2024-04-26T23:01:46.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.875+0000 7f95e9000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f95e40d0060 con 0x7f95e406c4b0 2024-04-26T23:01:46.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.876+0000 7f95e3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f95e406ff20 0x7f95e40cc0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:47928/0 (socket says 172.21.15.40:47928) 2024-04-26T23:01:46.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.876+0000 7f95e3e00640 1 -- 172.21.15.40:0/3353324466 learned_addr learned my addr 172.21.15.40:0/3353324466 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:46.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.876+0000 7f95e3400640 1 --2- 172.21.15.40:0/3353324466 >> v2:172.21.15.103:3300/0 conn(0x7f95e406c4b0 0x7f95e406c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.876+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 533262714 0 0) 0x7f95e40cf6c0 con 0x7f95e406ff20 2024-04-26T23:01:46.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.876+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95c8003520 con 0x7f95e406ff20 2024-04-26T23:01:46.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4132352688 0 0) 0x7f95c8003520 con 0x7f95e406ff20 2024-04-26T23:01:46.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 >> v2:172.21.15.103:3300/0 conn(0x7f95e406c4b0 msgr2=0x7f95e406c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:46.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e2000640 1 --2- 172.21.15.40:0/3353324466 >> v2:172.21.15.103:3300/0 conn(0x7f95e406c4b0 0x7f95e406c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 >> v1:172.21.15.103:6789/0 conn(0x7f95e40cda50 legacy=0x7f95e406f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e40cd7c0 con 0x7f95e406ff20 2024-04-26T23:01:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f95dc0019f0 con 0x7f95e406ff20 2024-04-26T23:01:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 >> v1:172.21.15.40:6789/0 conn(0x7f95e406ff20 legacy=0x7f95e40cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e2000640 1 --2- 172.21.15.40:0/3353324466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95c8003a00 0x7f95c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e2000640 1 --2- 172.21.15.40:0/3353324466 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95c8004800 0x7f95c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e2000640 1 --2- 172.21.15.40:0/3353324466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95c8005210 0x7f95c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f95e40d0060 con 0x7f95c8005210 2024-04-26T23:01:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f95c800c160 con 0x7f95c8003a00 2024-04-26T23:01:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f95c800c360 con 0x7f95c8004800 2024-04-26T23:01:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.877+0000 7f95e3e00640 1 --2- 172.21.15.40:0/3353324466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95c8005210 0x7f95c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.878+0000 7f95e2a00640 1 --2- 172.21.15.40:0/3353324466 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95c8004800 0x7f95c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.878+0000 7f95e3400640 1 --2- 172.21.15.40:0/3353324466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95c8003a00 0x7f95c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.878+0000 7f95e3e00640 1 -- 172.21.15.40:0/3353324466 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95c8004800 msgr2=0x7f95c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:46.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.878+0000 7f95e3e00640 1 --2- 172.21.15.40:0/3353324466 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95c8004800 0x7f95c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.878+0000 7f95e3e00640 1 -- 172.21.15.40:0/3353324466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95c8003a00 msgr2=0x7f95c8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:46.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.878+0000 7f95e3e00640 1 --2- 172.21.15.40:0/3353324466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95c8003a00 0x7f95c8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.878+0000 7f95e3e00640 1 -- 172.21.15.40:0/3353324466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c8003520 con 0x7f95c8005210 2024-04-26T23:01:46.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.878+0000 7f95e3400640 1 --2- 172.21.15.40:0/3353324466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95c8003a00 0x7f95c8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:46.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.879+0000 7f95e3e00640 1 -- 172.21.15.40:0/3353324466 reap_dead start 2024-04-26T23:01:46.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.879+0000 7f95e3e00640 1 --2- 172.21.15.40:0/3353324466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95c8005210 0x7f95c800b9c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f95e40d0750 tx=0x7f95c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ca4d0571e5c4bf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:46.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.879+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc00a390 con 0x7f95c8005210 2024-04-26T23:01:46.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.879+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95dc02f070 con 0x7f95c8005210 2024-04-26T23:01:46.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.879+0000 7f95e2000640 1 -- 172.21.15.40:0/3353324466 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc02bb60 con 0x7f95c8005210 2024-04-26T23:01:46.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.880+0000 7f95e9000640 1 -- 172.21.15.40:0/3353324466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95c8005210 msgr2=0x7f95c800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:46.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.880+0000 7f95e9000640 1 --2- 172.21.15.40:0/3353324466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95c8005210 0x7f95c800b9c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f95e40d0750 tx=0x7f95c8005830 comp rx=0 tx=0).stop 2024-04-26T23:01:46.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.880+0000 7f95e9000640 1 -- 172.21.15.40:0/3353324466 shutdown_connections 2024-04-26T23:01:46.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.880+0000 7f95e9000640 1 --2- 172.21.15.40:0/3353324466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95c8005210 0x7f95c800b9c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.880+0000 7f95e9000640 1 -- 172.21.15.40:0/3353324466 >> 172.21.15.40:0/3353324466 conn(0x7f95e40b3730 msgr2=0x7f95e40b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:46.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.880+0000 7f95e9000640 1 -- 172.21.15.40:0/3353324466 shutdown_connections 2024-04-26T23:01:46.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.880+0000 7f95e9000640 1 -- 172.21.15.40:0/3353324466 wait complete. 2024-04-26T23:01:46.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.882+0000 7f95e9000640 1 Processor -- start 2024-04-26T23:01:46.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.882+0000 7f95e9000640 1 -- start start 2024-04-26T23:01:46.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.883+0000 7f95e9000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95c8005210 0x7f95e414c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.883+0000 7f95e9000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95e4158e20 0x7f95e4160bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.883+0000 7f95e9000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95e4159280 0x7f95e4161110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.883+0000 7f95e9000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f95e4068ea0 con 0x7f95c8005210 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.883+0000 7f95e9000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f95e4068d20 con 0x7f95e4158e20 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.883+0000 7f95e9000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f95e40cdfa0 con 0x7f95e4159280 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.883+0000 7f95e3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95c8005210 0x7f95e414c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.883+0000 7f95e3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95c8005210 0x7f95e414c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48310/0 (socket says 172.21.15.40:48310) 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.883+0000 7f95e3400640 1 -- 172.21.15.40:0/1002300613 learned_addr learned my addr 172.21.15.40:0/1002300613 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.884+0000 7f95e2a00640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95e4158e20 0x7f95e4160bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.884+0000 7f95e3e00640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95e4159280 0x7f95e4161110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.884+0000 7f95e3400640 1 -- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95e4159280 msgr2=0x7f95e4161110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.884+0000 7f95e3400640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95e4159280 0x7f95e4161110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.884+0000 7f95e3400640 1 -- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95e4158e20 msgr2=0x7f95e4160bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.884+0000 7f95e3400640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95e4158e20 0x7f95e4160bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:46.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.884+0000 7f95e3400640 1 -- 172.21.15.40:0/1002300613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e4159800 con 0x7f95c8005210 2024-04-26T23:01:46.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.884+0000 7f95e2a00640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95e4158e20 0x7f95e4160bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:46.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.884+0000 7f95e3400640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95c8005210 0x7f95e414c480 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f95d4010910 tx=0x7f95d4010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:46.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.884+0000 7f95db400640 1 -- 172.21.15.40:0/1002300613 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d400f040 con 0x7f95c8005210 2024-04-26T23:01:46.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.884+0000 7f95db400640 1 -- 172.21.15.40:0/1002300613 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d400ebc0 con 0x7f95c8005210 2024-04-26T23:01:46.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.884+0000 7f95db400640 1 -- 172.21.15.40:0/1002300613 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d401c400 con 0x7f95c8005210 2024-04-26T23:01:46.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.884+0000 7f95e9000640 1 -- 172.21.15.40:0/1002300613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e4159ae0 con 0x7f95c8005210 2024-04-26T23:01:46.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.885+0000 7f95e9000640 1 -- 172.21.15.40:0/1002300613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e415a0a0 con 0x7f95c8005210 2024-04-26T23:01:46.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.886+0000 7f95e9000640 1 -- 172.21.15.40:0/1002300613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e4069370 con 0x7f95c8005210 2024-04-26T23:01:46.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.887+0000 7f95db400640 1 -- 172.21.15.40:0/1002300613 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f95d401c5a0 con 0x7f95c8005210 2024-04-26T23:01:46.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.888+0000 7f95db400640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f95c406c8f0 0x7f95c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:46.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.888+0000 7f95db400640 1 -- 172.21.15.40:0/1002300613 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f95d408c560 con 0x7f95c8005210 2024-04-26T23:01:46.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.888+0000 7f95e2a00640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f95c406c8f0 0x7f95c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:46.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.889+0000 7f95e2a00640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f95c406c8f0 0x7f95c406edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f95e41f7d70 tx=0x7f95d0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:46.892+0000 7f95db400640 1 -- 172.21.15.40:0/1002300613 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f95d4018030 con 0x7f95c8005210 2024-04-26T23:01:47.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:01:47.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:01:47.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.103+0000 7f95e9000640 1 -- 172.21.15.40:0/1002300613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f95e4073ce0 con 0x7f95c8005210 2024-04-26T23:01:47.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.104+0000 7f95db400640 1 -- 172.21.15.40:0/1002300613 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f95d4059f40 con 0x7f95c8005210 2024-04-26T23:01:47.104 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:01:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.106+0000 7f95e9000640 1 -- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f95c406c8f0 msgr2=0x7f95c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.106+0000 7f95e9000640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f95c406c8f0 0x7f95c406edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f95e41f7d70 tx=0x7f95d0009040 comp rx=0 tx=0).stop 2024-04-26T23:01:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.106+0000 7f95e9000640 1 -- 172.21.15.40:0/1002300613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95c8005210 msgr2=0x7f95e414c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.106+0000 7f95e9000640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95c8005210 0x7f95e414c480 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f95d4010910 tx=0x7f95d4010dd0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.106+0000 7f95e9000640 1 -- 172.21.15.40:0/1002300613 shutdown_connections 2024-04-26T23:01:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.106+0000 7f95e9000640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f95c406c8f0 0x7f95c406edb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.106+0000 7f95e9000640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95e4159280 0x7f95e4161110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.106+0000 7f95e9000640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95e4158e20 0x7f95e4160bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.106+0000 7f95e9000640 1 --2- 172.21.15.40:0/1002300613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95c8005210 0x7f95e414c480 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.106+0000 7f95e9000640 1 -- 172.21.15.40:0/1002300613 >> 172.21.15.40:0/1002300613 conn(0x7f95e40b3730 msgr2=0x7f95e4158560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.107+0000 7f95e9000640 1 -- 172.21.15.40:0/1002300613 shutdown_connections 2024-04-26T23:01:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.107+0000 7f95e9000640 1 -- 172.21.15.40:0/1002300613 wait complete. 2024-04-26T23:01:47.127 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:01:47.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.333+0000 7f2258600640 1 Processor -- start 2024-04-26T23:01:47.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.334+0000 7f2258600640 1 -- start start 2024-04-26T23:01:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.335+0000 7f2258600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2250162880 0x7f22500d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.335+0000 7f2258600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f22500b5f20 0x7f22500d34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.335+0000 7f2258600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22500b7e50 con 0x7f22500d39f0 2024-04-26T23:01:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.335+0000 7f2258600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f22500d3e40 con 0x7f2250162880 2024-04-26T23:01:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.335+0000 7f2258600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f22500d0580 con 0x7f22500b5f20 2024-04-26T23:01:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.335+0000 7f2255c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2250162880 0x7f22500d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.336+0000 7f2255c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2250162880 0x7f22500d2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48312/0 (socket says 172.21.15.40:48312) 2024-04-26T23:01:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.336+0000 7f2255c00640 1 -- 172.21.15.40:0/503888789 learned_addr learned my addr 172.21.15.40:0/503888789 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.336+0000 7f2256600640 1 --1- 172.21.15.40:0/503888789 >> v1:172.21.15.40:6789/0 conn(0x7f22500d39f0 0x7f22500b69e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:47940/0 (socket says 172.21.15.40:47940) 2024-04-26T23:01:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.336+0000 7f2255200640 1 --2- 172.21.15.40:0/503888789 >> v2:172.21.15.103:3300/0 conn(0x7f22500b5f20 0x7f22500d34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.336+0000 7f2255c00640 1 -- 172.21.15.40:0/503888789 >> v2:172.21.15.103:3300/0 conn(0x7f22500b5f20 msgr2=0x7f22500d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.336+0000 7f2255c00640 1 --2- 172.21.15.40:0/503888789 >> v2:172.21.15.103:3300/0 conn(0x7f22500b5f20 0x7f22500d34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.336+0000 7f2255c00640 1 -- 172.21.15.40:0/503888789 >> v1:172.21.15.40:6789/0 conn(0x7f22500d39f0 legacy=0x7f22500b69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.336+0000 7f2255c00640 1 -- 172.21.15.40:0/503888789 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22500d1800 con 0x7f2250162880 2024-04-26T23:01:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.336+0000 7f224fe00640 1 -- 172.21.15.40:0/503888789 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 221326021 0 0) 0x7f22500b7e50 con 0x7f22500d39f0 2024-04-26T23:01:47.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.337+0000 7f2255c00640 1 --2- 172.21.15.40:0/503888789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2250162880 0x7f22500d2cb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f225008d3d0 tx=0x7f224402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0f892745af4684 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:47.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.337+0000 7f224fe00640 1 -- 172.21.15.40:0/503888789 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2244037070 con 0x7f2250162880 2024-04-26T23:01:47.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.337+0000 7f224fe00640 1 -- 172.21.15.40:0/503888789 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2244032440 con 0x7f2250162880 2024-04-26T23:01:47.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.337+0000 7f224fe00640 1 -- 172.21.15.40:0/503888789 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2244032810 con 0x7f2250162880 2024-04-26T23:01:47.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.338+0000 7f2258600640 1 -- 172.21.15.40:0/503888789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2250162880 msgr2=0x7f22500d2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:47.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.338+0000 7f2258600640 1 --2- 172.21.15.40:0/503888789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2250162880 0x7f22500d2cb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f225008d3d0 tx=0x7f224402a270 comp rx=0 tx=0).stop 2024-04-26T23:01:47.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.338+0000 7f2258600640 1 -- 172.21.15.40:0/503888789 shutdown_connections 2024-04-26T23:01:47.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.338+0000 7f2258600640 1 --2- 172.21.15.40:0/503888789 >> v2:172.21.15.103:3300/0 conn(0x7f22500b5f20 0x7f22500d34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.338+0000 7f2258600640 1 --2- 172.21.15.40:0/503888789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2250162880 0x7f22500d2cb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.338+0000 7f2258600640 1 -- 172.21.15.40:0/503888789 >> 172.21.15.40:0/503888789 conn(0x7f22500d4f60 msgr2=0x7f22500d5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:47.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.338+0000 7f2258600640 1 -- 172.21.15.40:0/503888789 shutdown_connections 2024-04-26T23:01:47.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.338+0000 7f2258600640 1 -- 172.21.15.40:0/503888789 wait complete. 2024-04-26T23:01:47.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.340+0000 7f2258600640 1 Processor -- start 2024-04-26T23:01:47.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.340+0000 7f2258600640 1 -- start start 2024-04-26T23:01:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.341+0000 7f2258600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22500b5f20 0x7f22500b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.341+0000 7f2258600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22500d39f0 0x7f22500b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.341+0000 7f2258600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2250162880 0x7f22501614d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.341+0000 7f2258600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f22500d0d90 con 0x7f22500d39f0 2024-04-26T23:01:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.341+0000 7f2258600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f22500d0ab0 con 0x7f2250162880 2024-04-26T23:01:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.341+0000 7f2258600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2250162c60 con 0x7f22500b5f20 2024-04-26T23:01:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.341+0000 7f2255200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22500d39f0 0x7f22500b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.341+0000 7f2255200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22500d39f0 0x7f22500b53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48328/0 (socket says 172.21.15.40:48328) 2024-04-26T23:01:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.341+0000 7f2255200640 1 -- 172.21.15.40:0/2114806472 learned_addr learned my addr 172.21.15.40:0/2114806472 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f2255c00640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22500b5f20 0x7f22500b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f2256600640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2250162880 0x7f22501614d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f2255200640 1 -- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22500b5f20 msgr2=0x7f22500b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f2255200640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22500b5f20 0x7f22500b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f2255200640 1 -- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2250162880 msgr2=0x7f22501614d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f2255200640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2250162880 0x7f22501614d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f2255200640 1 -- 172.21.15.40:0/2114806472 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22500b1b50 con 0x7f22500d39f0 2024-04-26T23:01:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f2255c00640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22500b5f20 0x7f22500b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f2256600640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2250162880 0x7f22501614d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f2255200640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22500d39f0 0x7f22500b53f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f224000d620 tx=0x7f224000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f224d600640 1 -- 172.21.15.40:0/2114806472 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224000b620 con 0x7f22500d39f0 2024-04-26T23:01:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f224d600640 1 -- 172.21.15.40:0/2114806472 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2240019070 con 0x7f22500d39f0 2024-04-26T23:01:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f2258600640 1 -- 172.21.15.40:0/2114806472 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22500b1e30 con 0x7f22500d39f0 2024-04-26T23:01:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.342+0000 7f224d600640 1 -- 172.21.15.40:0/2114806472 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224000ba00 con 0x7f22500d39f0 2024-04-26T23:01:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.343+0000 7f2258600640 1 -- 172.21.15.40:0/2114806472 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22500b23f0 con 0x7f22500d39f0 2024-04-26T23:01:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.344+0000 7f2258600640 1 -- 172.21.15.40:0/2114806472 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22500694f0 con 0x7f22500d39f0 2024-04-26T23:01:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.345+0000 7f224d600640 1 -- 172.21.15.40:0/2114806472 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f224000bba0 con 0x7f22500d39f0 2024-04-26T23:01:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.346+0000 7f224d600640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f223406c8f0 0x7f223406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.346+0000 7f224d600640 1 -- 172.21.15.40:0/2114806472 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f224008c3e0 con 0x7f22500d39f0 2024-04-26T23:01:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.347+0000 7f2255c00640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f223406c8f0 0x7f223406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.348+0000 7f2255c00640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f223406c8f0 0x7f223406edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2244034000 tx=0x7f224402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.351+0000 7f224d600640 1 -- 172.21.15.40:0/2114806472 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f224005a370 con 0x7f22500d39f0 2024-04-26T23:01:47.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.601+0000 7f2258600640 1 -- 172.21.15.40:0/2114806472 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2250074590 con 0x7f22500d39f0 2024-04-26T23:01:47.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.602+0000 7f224d600640 1 -- 172.21.15.40:0/2114806472 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f2240060030 con 0x7f22500d39f0 2024-04-26T23:01:47.602 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:47.603 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1253648754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1253648754},{"type":"v1","addr":"172.21.15.103:6836","nonce":1253648754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3239652611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":3239652611},{"type":"v1","addr":"172.21.15.103:6837","nonce":3239652611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3670144900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3670144900},{"type":"v1","addr":"172.21.15.40:6833","nonce":3670144900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:01:05.171276+0000","modified":"2024-04-26T23:01:33.399801+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3989190340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3989190340},{"type":"v1","addr":"172.21.15.40:6835","nonce":3989190340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-26T23:01:47.603 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 9 2024-04-26T23:01:47.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.604+0000 7f2258600640 1 -- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f223406c8f0 msgr2=0x7f223406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:47.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.604+0000 7f2258600640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f223406c8f0 0x7f223406edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2244034000 tx=0x7f224402e000 comp rx=0 tx=0).stop 2024-04-26T23:01:47.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.604+0000 7f2258600640 1 -- 172.21.15.40:0/2114806472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22500d39f0 msgr2=0x7f22500b53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:47.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.604+0000 7f2258600640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22500d39f0 0x7f22500b53f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f224000d620 tx=0x7f224000dae0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.605+0000 7f2258600640 1 -- 172.21.15.40:0/2114806472 shutdown_connections 2024-04-26T23:01:47.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.605+0000 7f2258600640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f223406c8f0 0x7f223406edb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.605+0000 7f2258600640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2250162880 0x7f22501614d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.605+0000 7f2258600640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22500d39f0 0x7f22500b53f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.605+0000 7f2258600640 1 --2- 172.21.15.40:0/2114806472 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22500b5f20 0x7f22500b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.605+0000 7f2258600640 1 -- 172.21.15.40:0/2114806472 >> 172.21.15.40:0/2114806472 conn(0x7f22500d4f60 msgr2=0x7f225006dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:47.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.605+0000 7f2258600640 1 -- 172.21.15.40:0/2114806472 shutdown_connections 2024-04-26T23:01:47.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.605+0000 7f2258600640 1 -- 172.21.15.40:0/2114806472 wait complete. 2024-04-26T23:01:47.624 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:01:47.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.828+0000 7fd673e00640 1 Processor -- start 2024-04-26T23:01:47.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.829+0000 7fd673e00640 1 -- start start 2024-04-26T23:01:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.830+0000 7fd673e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6740b6220 0x7fd6740d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.830+0000 7fd673e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd6740d08d0 0x7fd6740d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.830+0000 7fd673e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6740d3820 0x7fd6740b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.830+0000 7fd673e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fd674162c40 con 0x7fd6740b6220 2024-04-26T23:01:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.830+0000 7fd673e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd6740d2d80 con 0x7fd6740d08d0 2024-04-26T23:01:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.830+0000 7fd673e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd6740b7c80 con 0x7fd6740d3820 2024-04-26T23:01:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.831+0000 7fd672a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6740b6220 0x7fd6740d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.831+0000 7fd672a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6740b6220 0x7fd6740d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48330/0 (socket says 172.21.15.40:48330) 2024-04-26T23:01:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.831+0000 7fd672a00640 1 -- 172.21.15.40:0/2905263545 learned_addr learned my addr 172.21.15.40:0/2905263545 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.831+0000 7fd672000640 1 --2- 172.21.15.40:0/2905263545 >> v2:172.21.15.103:3300/0 conn(0x7fd6740d08d0 0x7fd6740d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.831+0000 7fd673400640 1 --2- 172.21.15.40:0/2905263545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6740d3820 0x7fd6740b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.831+0000 7fd672a00640 1 -- 172.21.15.40:0/2905263545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6740d3820 msgr2=0x7fd6740b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.831+0000 7fd672a00640 1 --2- 172.21.15.40:0/2905263545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6740d3820 0x7fd6740b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.831+0000 7fd672a00640 1 -- 172.21.15.40:0/2905263545 >> v2:172.21.15.103:3300/0 conn(0x7fd6740d08d0 msgr2=0x7fd6740d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.831+0000 7fd672a00640 1 --2- 172.21.15.40:0/2905263545 >> v2:172.21.15.103:3300/0 conn(0x7fd6740d08d0 0x7fd6740d32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.831+0000 7fd672a00640 1 -- 172.21.15.40:0/2905263545 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6740b7f60 con 0x7fd6740b6220 2024-04-26T23:01:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.831+0000 7fd672000640 1 --2- 172.21.15.40:0/2905263545 >> v2:172.21.15.103:3300/0 conn(0x7fd6740d08d0 0x7fd6740d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:01:47.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.832+0000 7fd672a00640 1 --2- 172.21.15.40:0/2905263545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6740b6220 0x7fd6740d0390 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd67408f300 tx=0x7fd65c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13505ff436562fa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:47.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.833+0000 7fd671600640 1 -- 172.21.15.40:0/2905263545 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c037070 con 0x7fd6740b6220 2024-04-26T23:01:47.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.833+0000 7fd671600640 1 -- 172.21.15.40:0/2905263545 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd65c032440 con 0x7fd6740b6220 2024-04-26T23:01:47.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.833+0000 7fd671600640 1 -- 172.21.15.40:0/2905263545 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c032810 con 0x7fd6740b6220 2024-04-26T23:01:47.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.833+0000 7fd673e00640 1 -- 172.21.15.40:0/2905263545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6740b6220 msgr2=0x7fd6740d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:47.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.833+0000 7fd673e00640 1 --2- 172.21.15.40:0/2905263545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6740b6220 0x7fd6740d0390 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd67408f300 tx=0x7fd65c02a1e0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.833+0000 7fd673e00640 1 -- 172.21.15.40:0/2905263545 shutdown_connections 2024-04-26T23:01:47.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.833+0000 7fd673e00640 1 --2- 172.21.15.40:0/2905263545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6740d3820 0x7fd6740b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.833+0000 7fd673e00640 1 --2- 172.21.15.40:0/2905263545 >> v2:172.21.15.103:3300/0 conn(0x7fd6740d08d0 0x7fd6740d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.833+0000 7fd673e00640 1 --2- 172.21.15.40:0/2905263545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6740b6220 0x7fd6740d0390 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.833+0000 7fd673e00640 1 -- 172.21.15.40:0/2905263545 >> 172.21.15.40:0/2905263545 conn(0x7fd6740d4d90 msgr2=0x7fd6740d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:47.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.834+0000 7fd673e00640 1 -- 172.21.15.40:0/2905263545 shutdown_connections 2024-04-26T23:01:47.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.834+0000 7fd673e00640 1 -- 172.21.15.40:0/2905263545 wait complete. 2024-04-26T23:01:47.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.835+0000 7fd673e00640 1 Processor -- start 2024-04-26T23:01:47.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.835+0000 7fd673e00640 1 -- start start 2024-04-26T23:01:47.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.836+0000 7fd673e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6740b6220 0x7fd674159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:47.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.836+0000 7fd673e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6740d08d0 0x7fd6741f7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:47.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.836+0000 7fd673e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6740d3820 0x7fd67415c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.836+0000 7fd673e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd6740b6d50 con 0x7fd6740b6220 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.836+0000 7fd673e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd6740b8520 con 0x7fd6740d3820 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.836+0000 7fd673e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd6740b6ed0 con 0x7fd6740d08d0 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.837+0000 7fd672a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6740b6220 0x7fd674159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.837+0000 7fd672a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6740b6220 0x7fd674159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48336/0 (socket says 172.21.15.40:48336) 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.837+0000 7fd672a00640 1 -- 172.21.15.40:0/913015974 learned_addr learned my addr 172.21.15.40:0/913015974 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.837+0000 7fd672000640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6740d08d0 0x7fd6741f7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.837+0000 7fd673400640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6740d3820 0x7fd67415c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.837+0000 7fd672a00640 1 -- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6740d08d0 msgr2=0x7fd6741f7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.837+0000 7fd672a00640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6740d08d0 0x7fd6741f7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.837+0000 7fd672a00640 1 -- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6740d3820 msgr2=0x7fd67415c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.837+0000 7fd672a00640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6740d3820 0x7fd67415c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.837+0000 7fd672a00640 1 -- 172.21.15.40:0/913015974 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67414f8a0 con 0x7fd6740b6220 2024-04-26T23:01:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.837+0000 7fd673400640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6740d3820 0x7fd67415c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:47.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.837+0000 7fd672a00640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6740b6220 0x7fd674159420 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd65c02a220 tx=0x7fd65c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:47.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.838+0000 7fd66b400640 1 -- 172.21.15.40:0/913015974 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c041020 con 0x7fd6740b6220 2024-04-26T23:01:47.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.838+0000 7fd66b400640 1 -- 172.21.15.40:0/913015974 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd65c0306c0 con 0x7fd6740b6220 2024-04-26T23:01:47.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.838+0000 7fd66b400640 1 -- 172.21.15.40:0/913015974 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c037040 con 0x7fd6740b6220 2024-04-26T23:01:47.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.838+0000 7fd673e00640 1 -- 172.21.15.40:0/913015974 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd674159a00 con 0x7fd6740b6220 2024-04-26T23:01:47.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.838+0000 7fd673e00640 1 -- 172.21.15.40:0/913015974 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd67415ced0 con 0x7fd6740b6220 2024-04-26T23:01:47.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.839+0000 7fd673e00640 1 -- 172.21.15.40:0/913015974 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd674069370 con 0x7fd6740b6220 2024-04-26T23:01:47.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.840+0000 7fd66b400640 1 -- 172.21.15.40:0/913015974 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd65c044020 con 0x7fd6740b6220 2024-04-26T23:01:47.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.841+0000 7fd66b400640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd65006c8f0 0x7fd65006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:47.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.841+0000 7fd66b400640 1 -- 172.21.15.40:0/913015974 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd65c0ac800 con 0x7fd6740b6220 2024-04-26T23:01:47.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.842+0000 7fd672000640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd65006c8f0 0x7fd65006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:47.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.843+0000 7fd672000640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd65006c8f0 0x7fd65006edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd67414fbe0 tx=0x7fd66400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:47.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:47.845+0000 7fd66b400640 1 -- 172.21.15.40:0/913015974 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd65c04b030 con 0x7fd6740b6220 2024-04-26T23:01:48.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.057+0000 7fd673e00640 1 -- 172.21.15.40:0/913015974 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd6740d30c0 con 0x7fd6740b6220 2024-04-26T23:01:48.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.059+0000 7fd66b400640 1 -- 172.21.15.40:0/913015974 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18812 (secure 0 0 0) 0x7fd67414f8a0 con 0x7fd6740b6220 2024-04-26T23:01:48.059 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:48.060 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:46.975683+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:01:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.062+0000 7fd673e00640 1 -- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd65006c8f0 msgr2=0x7fd65006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.062+0000 7fd673e00640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd65006c8f0 0x7fd65006edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd67414fbe0 tx=0x7fd66400b040 comp rx=0 tx=0).stop 2024-04-26T23:01:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.062+0000 7fd673e00640 1 -- 172.21.15.40:0/913015974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6740b6220 msgr2=0x7fd674159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.062+0000 7fd673e00640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6740b6220 0x7fd674159420 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd65c02a220 tx=0x7fd65c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.062+0000 7fd673e00640 1 -- 172.21.15.40:0/913015974 shutdown_connections 2024-04-26T23:01:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.062+0000 7fd673e00640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd65006c8f0 0x7fd65006edb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.062+0000 7fd673e00640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6740d3820 0x7fd67415c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.062+0000 7fd673e00640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6740d08d0 0x7fd6741f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.062+0000 7fd673e00640 1 --2- 172.21.15.40:0/913015974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6740b6220 0x7fd674159420 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.062+0000 7fd673e00640 1 -- 172.21.15.40:0/913015974 >> 172.21.15.40:0/913015974 conn(0x7fd6740d4d90 msgr2=0x7fd6740d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.062+0000 7fd673e00640 1 -- 172.21.15.40:0/913015974 shutdown_connections 2024-04-26T23:01:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.063+0000 7fd673e00640 1 -- 172.21.15.40:0/913015974 wait complete. 2024-04-26T23:01:48.087 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:01:48.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.307+0000 7f4289600640 1 Processor -- start 2024-04-26T23:01:48.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.307+0000 7f4289600640 1 -- start start 2024-04-26T23:01:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.309+0000 7f4289600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f42840af7a0 0x7f42840afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.309+0000 7f4289600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4284071fe0 0x7f4284073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.309+0000 7f4289600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42840d0820 con 0x7f428415ce60 2024-04-26T23:01:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.309+0000 7f4289600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f42840cc450 con 0x7f42840af7a0 2024-04-26T23:01:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.309+0000 7f4289600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f42841614c0 con 0x7f4284071fe0 2024-04-26T23:01:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.309+0000 7f4283e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f428415ce60 0x7f42840ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:47946/0 (socket says 172.21.15.40:47946) 2024-04-26T23:01:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.309+0000 7f4283e00640 1 -- 172.21.15.40:0/2509080394 learned_addr learned my addr 172.21.15.40:0/2509080394 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4288c00640 1 --2- 172.21.15.40:0/2509080394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4284071fe0 0x7f4284073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4283400640 1 --2- 172.21.15.40:0/2509080394 >> v2:172.21.15.103:3300/0 conn(0x7f42840af7a0 0x7f42840afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 447267596 0 0) 0x7f42840d0820 con 0x7f428415ce60 2024-04-26T23:01:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4268003520 con 0x7f428415ce60 2024-04-26T23:01:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2274826957 0 0) 0x7f4268003520 con 0x7f428415ce60 2024-04-26T23:01:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4284071fe0 msgr2=0x7f4284073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 --2- 172.21.15.40:0/2509080394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4284071fe0 0x7f4284073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 >> v2:172.21.15.103:3300/0 conn(0x7f42840af7a0 msgr2=0x7f42840afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 --2- 172.21.15.40:0/2509080394 >> v2:172.21.15.103:3300/0 conn(0x7f42840af7a0 0x7f42840afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4284161760 con 0x7f428415ce60 2024-04-26T23:01:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f42740019f0 con 0x7f428415ce60 2024-04-26T23:01:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 >> v1:172.21.15.40:6789/0 conn(0x7f428415ce60 legacy=0x7f42840ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 --2- 172.21.15.40:0/2509080394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4284071fe0 0x7f42680039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:48.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 --2- 172.21.15.40:0/2509080394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4268003f80 0x7f4268004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:48.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 --2- 172.21.15.40:0/2509080394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4268004d40 0x7f42680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:48.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f42840cc450 con 0x7f4268003f80 2024-04-26T23:01:48.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f42841614c0 con 0x7f4268004d40 2024-04-26T23:01:48.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.310+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f426800ba70 con 0x7f4284071fe0 2024-04-26T23:01:48.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.311+0000 7f4283400640 1 --2- 172.21.15.40:0/2509080394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4268003f80 0x7f4268004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:48.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.311+0000 7f4283e00640 1 --2- 172.21.15.40:0/2509080394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4284071fe0 0x7f42680039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:48.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.311+0000 7f4288c00640 1 --2- 172.21.15.40:0/2509080394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4268004d40 0x7f42680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:48.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.311+0000 7f4283400640 1 -- 172.21.15.40:0/2509080394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4284071fe0 msgr2=0x7f42680039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:48.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.311+0000 7f4283400640 1 --2- 172.21.15.40:0/2509080394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4284071fe0 0x7f42680039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.311+0000 7f4283400640 1 -- 172.21.15.40:0/2509080394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4268004d40 msgr2=0x7f42680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:48.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.311+0000 7f4283400640 1 --2- 172.21.15.40:0/2509080394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4268004d40 0x7f42680051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.311+0000 7f4283400640 1 -- 172.21.15.40:0/2509080394 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4268003520 con 0x7f4268003f80 2024-04-26T23:01:48.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.312+0000 7f4283400640 1 --2- 172.21.15.40:0/2509080394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4268003f80 0x7f4268004800 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f426c009ac0 tx=0x7f4268005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=781e94a0dee71cff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:48.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.312+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c036070 con 0x7f4268003f80 2024-04-26T23:01:48.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.312+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f426c031400 con 0x7f4268003f80 2024-04-26T23:01:48.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.312+0000 7f4282a00640 1 -- 172.21.15.40:0/2509080394 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c009690 con 0x7f4268003f80 2024-04-26T23:01:48.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.313+0000 7f4289600640 1 -- 172.21.15.40:0/2509080394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4268003f80 msgr2=0x7f4268004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:48.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.313+0000 7f4289600640 1 --2- 172.21.15.40:0/2509080394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4268003f80 0x7f4268004800 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f426c009ac0 tx=0x7f4268005830 comp rx=0 tx=0).stop 2024-04-26T23:01:48.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.313+0000 7f4288c00640 1 -- 172.21.15.40:0/2509080394 reap_dead start 2024-04-26T23:01:48.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.313+0000 7f4289600640 1 -- 172.21.15.40:0/2509080394 shutdown_connections 2024-04-26T23:01:48.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.313+0000 7f4289600640 1 -- 172.21.15.40:0/2509080394 >> 172.21.15.40:0/2509080394 conn(0x7f42840d0a10 msgr2=0x7f42840d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:48.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.313+0000 7f4289600640 1 -- 172.21.15.40:0/2509080394 shutdown_connections 2024-04-26T23:01:48.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.313+0000 7f4289600640 1 -- 172.21.15.40:0/2509080394 wait complete. 2024-04-26T23:01:48.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.315+0000 7f4289600640 1 Processor -- start 2024-04-26T23:01:48.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.315+0000 7f4289600640 1 -- start start 2024-04-26T23:01:48.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.316+0000 7f4289600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4268003f80 0x7f428415f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:48.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.316+0000 7f4289600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f428415b270 0x7f428415d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:48.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.316+0000 7f4289600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f428415dad0 0x7f42840b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:48.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.316+0000 7f4289600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f42840cc1f0 con 0x7f4268003f80 2024-04-26T23:01:48.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.316+0000 7f4289600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f42840cc070 con 0x7f428415b270 2024-04-26T23:01:48.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.316+0000 7f4289600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4284073760 con 0x7f428415dad0 2024-04-26T23:01:48.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.316+0000 7f4283e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4268003f80 0x7f428415f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:48.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.316+0000 7f4283e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4268003f80 0x7f428415f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48344/0 (socket says 172.21.15.40:48344) 2024-04-26T23:01:48.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.316+0000 7f4283e00640 1 -- 172.21.15.40:0/2169818068 learned_addr learned my addr 172.21.15.40:0/2169818068 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:48.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.317+0000 7f4283400640 1 --2- 172.21.15.40:0/2169818068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f428415b270 0x7f428415d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:48.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.317+0000 7f4283e00640 1 -- 172.21.15.40:0/2169818068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f428415dad0 msgr2=0x7f42840b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:48.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.317+0000 7f4283e00640 1 --2- 172.21.15.40:0/2169818068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f428415dad0 0x7f42840b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.317+0000 7f4283e00640 1 -- 172.21.15.40:0/2169818068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f428415b270 msgr2=0x7f428415d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:48.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.317+0000 7f4283e00640 1 --2- 172.21.15.40:0/2169818068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f428415b270 0x7f428415d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.317+0000 7f4283e00640 1 -- 172.21.15.40:0/2169818068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428415fc40 con 0x7f4268003f80 2024-04-26T23:01:48.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.317+0000 7f4283e00640 1 --2- 172.21.15.40:0/2169818068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4268003f80 0x7f428415f700 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f42740079e0 tx=0x7f427400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:48.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.317+0000 7f427be00640 1 -- 172.21.15.40:0/2169818068 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427400ed70 con 0x7f4268003f80 2024-04-26T23:01:48.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.317+0000 7f427be00640 1 -- 172.21.15.40:0/2169818068 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4274018070 con 0x7f4268003f80 2024-04-26T23:01:48.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.317+0000 7f4289600640 1 -- 172.21.15.40:0/2169818068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42840b3a70 con 0x7f4268003f80 2024-04-26T23:01:48.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.318+0000 7f427be00640 1 -- 172.21.15.40:0/2169818068 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274013660 con 0x7f4268003f80 2024-04-26T23:01:48.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.318+0000 7f4289600640 1 -- 172.21.15.40:0/2169818068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42840b4030 con 0x7f4268003f80 2024-04-26T23:01:48.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.319+0000 7f4289600640 1 -- 172.21.15.40:0/2169818068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42840706d0 con 0x7f4268003f80 2024-04-26T23:01:48.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.320+0000 7f427be00640 1 -- 172.21.15.40:0/2169818068 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4274013800 con 0x7f4268003f80 2024-04-26T23:01:48.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.321+0000 7f427be00640 1 --2- 172.21.15.40:0/2169818068 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f426006c920 0x7f426006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:48.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.321+0000 7f427be00640 1 -- 172.21.15.40:0/2169818068 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f427408c630 con 0x7f4268003f80 2024-04-26T23:01:48.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.321+0000 7f4283400640 1 --2- 172.21.15.40:0/2169818068 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f426006c920 0x7f426006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:48.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.322+0000 7f4283400640 1 --2- 172.21.15.40:0/2169818068 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f426006c920 0x7f426006ede0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f426c033000 tx=0x7f426c008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:48.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.325+0000 7f427be00640 1 -- 172.21.15.40:0/2169818068 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f42840706d0 con 0x7f4268003f80 2024-04-26T23:01:48.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.572+0000 7f4289600640 1 -- 172.21.15.40:0/2169818068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42840d01f0 con 0x7f4268003f80 2024-04-26T23:01:48.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.573+0000 7f427be00640 1 -- 172.21.15.40:0/2169818068 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f427405a5c0 con 0x7f4268003f80 2024-04-26T23:01:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:01:48.574 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1253648754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1253648754},{"type":"v1","addr":"172.21.15.103:6836","nonce":1253648754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3239652611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":3239652611},{"type":"v1","addr":"172.21.15.103:6837","nonce":3239652611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3670144900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3670144900},{"type":"v1","addr":"172.21.15.40:6833","nonce":3670144900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:01:05.171276+0000","modified":"2024-04-26T23:01:33.399801+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3989190340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3989190340},{"type":"v1","addr":"172.21.15.40:6835","nonce":3989190340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-26T23:01:48.574 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 9 2024-04-26T23:01:48.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.576+0000 7f4289600640 1 -- 172.21.15.40:0/2169818068 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f426006c920 msgr2=0x7f426006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:48.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.576+0000 7f4289600640 1 --2- 172.21.15.40:0/2169818068 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f426006c920 0x7f426006ede0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f426c033000 tx=0x7f426c008030 comp rx=0 tx=0).stop 2024-04-26T23:01:48.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.576+0000 7f4289600640 1 -- 172.21.15.40:0/2169818068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4268003f80 msgr2=0x7f428415f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:48.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.576+0000 7f4289600640 1 --2- 172.21.15.40:0/2169818068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4268003f80 0x7f428415f700 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f42740079e0 tx=0x7f427400e530 comp rx=0 tx=0).stop 2024-04-26T23:01:48.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.576+0000 7f4289600640 1 -- 172.21.15.40:0/2169818068 shutdown_connections 2024-04-26T23:01:48.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.576+0000 7f4289600640 1 --2- 172.21.15.40:0/2169818068 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f426006c920 0x7f426006ede0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.576+0000 7f4289600640 1 --2- 172.21.15.40:0/2169818068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f428415dad0 0x7f42840b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.576+0000 7f4289600640 1 --2- 172.21.15.40:0/2169818068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f428415b270 0x7f428415d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.576+0000 7f4289600640 1 --2- 172.21.15.40:0/2169818068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4268003f80 0x7f428415f700 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:48.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.576+0000 7f4289600640 1 -- 172.21.15.40:0/2169818068 >> 172.21.15.40:0/2169818068 conn(0x7f42840d0a10 msgr2=0x7f42840b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:48.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.577+0000 7f4289600640 1 -- 172.21.15.40:0/2169818068 shutdown_connections 2024-04-26T23:01:48.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:48.577+0000 7f4289600640 1 -- 172.21.15.40:0/2169818068 wait complete. 2024-04-26T23:01:48.997 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:01:05.171276+0000', 'modified': '2024-04-26T23:01:33.399801+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/3989190340', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 3989190340}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 3989190340}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4257, 'qdb_cluster': [4257]} 2024-04-26T23:01:48.997 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-26T23:01:48.998 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T23:01:49.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.218+0000 7f50f2800640 1 Processor -- start 2024-04-26T23:01:49.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.218+0000 7f50f2800640 1 -- start start 2024-04-26T23:01:49.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.220+0000 7f50f2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50ec15f2d0 0x7f50ec0afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.220+0000 7f50f2800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f50ec0cc590 0x7f50ec0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.220+0000 7f50f2800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50ec0af7b0 con 0x7f50ec0ce5e0 2024-04-26T23:01:49.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.220+0000 7f50f2800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f50ec0afa90 con 0x7f50ec0cc590 2024-04-26T23:01:49.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.220+0000 7f50f2800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f50ec0cbd90 con 0x7f50ec15f2d0 2024-04-26T23:01:49.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.220+0000 7f50f0a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f50ec0ce5e0 0x7f50ec0c58b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:47956/0 (socket says 172.21.15.40:47956) 2024-04-26T23:01:49.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.220+0000 7f50f0a00640 1 -- 172.21.15.40:0/1793801627 learned_addr learned my addr 172.21.15.40:0/1793801627 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:49.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50f1400640 1 --2- 172.21.15.40:0/1793801627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50ec15f2d0 0x7f50ec0afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50f1e00640 1 --2- 172.21.15.40:0/1793801627 >> v2:172.21.15.103:3300/0 conn(0x7f50ec0cc590 0x7f50ec0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1742444707 0 0) 0x7f50ec0af7b0 con 0x7f50ec0ce5e0 2024-04-26T23:01:49.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50d4003520 con 0x7f50ec0ce5e0 2024-04-26T23:01:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3744069463 0 0) 0x7f50d4003520 con 0x7f50ec0ce5e0 2024-04-26T23:01:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50ec15f2d0 msgr2=0x7f50ec0afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50ebe00640 1 --2- 172.21.15.40:0/1793801627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50ec15f2d0 0x7f50ec0afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 >> v2:172.21.15.103:3300/0 conn(0x7f50ec0cc590 msgr2=0x7f50ec0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50ebe00640 1 --2- 172.21.15.40:0/1793801627 >> v2:172.21.15.103:3300/0 conn(0x7f50ec0cc590 0x7f50ec0cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ec0cc070 con 0x7f50ec0ce5e0 2024-04-26T23:01:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f50d8001670 con 0x7f50ec0ce5e0 2024-04-26T23:01:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 >> v1:172.21.15.40:6789/0 conn(0x7f50ec0ce5e0 legacy=0x7f50ec0c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50ebe00640 1 --2- 172.21.15.40:0/1793801627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50d40039b0 0x7f50d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.221+0000 7f50ebe00640 1 --2- 172.21.15.40:0/1793801627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50ec15f2d0 0x7f50d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.222+0000 7f50ebe00640 1 --2- 172.21.15.40:0/1793801627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d4004d40 0x7f50d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.222+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec0afa90 con 0x7f50d4004d40 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.222+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec0cbd90 con 0x7f50d40039b0 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.222+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f50d400b9c0 con 0x7f50ec15f2d0 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.222+0000 7f50f1e00640 1 --2- 172.21.15.40:0/1793801627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d4004d40 0x7f50d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.222+0000 7f50f0a00640 1 --2- 172.21.15.40:0/1793801627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50ec15f2d0 0x7f50d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.222+0000 7f50f1400640 1 --2- 172.21.15.40:0/1793801627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50d40039b0 0x7f50d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.222+0000 7f50f1e00640 1 -- 172.21.15.40:0/1793801627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50ec15f2d0 msgr2=0x7f50d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.222+0000 7f50f1e00640 1 --2- 172.21.15.40:0/1793801627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50ec15f2d0 0x7f50d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.222+0000 7f50f1e00640 1 -- 172.21.15.40:0/1793801627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50d40039b0 msgr2=0x7f50d4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.222+0000 7f50f1e00640 1 --2- 172.21.15.40:0/1793801627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50d40039b0 0x7f50d4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.222+0000 7f50f1e00640 1 -- 172.21.15.40:0/1793801627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d4003520 con 0x7f50d4004d40 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.222+0000 7f50f1400640 1 --2- 172.21.15.40:0/1793801627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50d40039b0 0x7f50d4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.223+0000 7f50f1e00640 1 --2- 172.21.15.40:0/1793801627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d4004d40 0x7f50d40051c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f50e4009fc0 tx=0x7f50d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1fd427c6c3e65c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.224+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e4009c10 con 0x7f50d4004d40 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.224+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50e4036070 con 0x7f50d4004d40 2024-04-26T23:01:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.224+0000 7f50ebe00640 1 -- 172.21.15.40:0/1793801627 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e402d070 con 0x7f50d4004d40 2024-04-26T23:01:49.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.224+0000 7f50f2800640 1 -- 172.21.15.40:0/1793801627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d4004d40 msgr2=0x7f50d40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.224+0000 7f50f2800640 1 --2- 172.21.15.40:0/1793801627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50d4004d40 0x7f50d40051c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f50e4009fc0 tx=0x7f50d4005830 comp rx=0 tx=0).stop 2024-04-26T23:01:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.224+0000 7f50f1e00640 1 -- 172.21.15.40:0/1793801627 reap_dead start 2024-04-26T23:01:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.224+0000 7f50f2800640 1 -- 172.21.15.40:0/1793801627 shutdown_connections 2024-04-26T23:01:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.224+0000 7f50f2800640 1 -- 172.21.15.40:0/1793801627 >> 172.21.15.40:0/1793801627 conn(0x7f50ec0d0a10 msgr2=0x7f50ec0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.224+0000 7f50f2800640 1 -- 172.21.15.40:0/1793801627 shutdown_connections 2024-04-26T23:01:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.224+0000 7f50f2800640 1 -- 172.21.15.40:0/1793801627 wait complete. 2024-04-26T23:01:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.226+0000 7f50f2800640 1 Processor -- start 2024-04-26T23:01:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.226+0000 7f50f2800640 1 -- start start 2024-04-26T23:01:49.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.227+0000 7f50f2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50d4004d40 0x7f50ec1524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.227+0000 7f50f2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50ec0b1900 0x7f50ec0b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.227+0000 7f50f2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50ec0c6e90 0x7f50ec0c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.227+0000 7f50f2800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec0c5fc0 con 0x7f50ec0b1900 2024-04-26T23:01:49.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.228+0000 7f50f2800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec06e270 con 0x7f50d4004d40 2024-04-26T23:01:49.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.228+0000 7f50f2800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f50ec0c8720 con 0x7f50ec0c6e90 2024-04-26T23:01:49.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.228+0000 7f50f0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50ec0b1900 0x7f50ec0b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.228+0000 7f50f0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50ec0b1900 0x7f50ec0b1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48366/0 (socket says 172.21.15.40:48366) 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.228+0000 7f50f0a00640 1 -- 172.21.15.40:0/1187287362 learned_addr learned my addr 172.21.15.40:0/1187287362 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.228+0000 7f50f1400640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50d4004d40 0x7f50ec1524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.228+0000 7f50f1e00640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50ec0c6e90 0x7f50ec0c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.228+0000 7f50f0a00640 1 -- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50ec0c6e90 msgr2=0x7f50ec0c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.228+0000 7f50f0a00640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50ec0c6e90 0x7f50ec0c7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.228+0000 7f50f0a00640 1 -- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50d4004d40 msgr2=0x7f50ec1524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.228+0000 7f50f0a00640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50d4004d40 0x7f50ec1524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.228+0000 7f50f0a00640 1 -- 172.21.15.40:0/1187287362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ec0c8a00 con 0x7f50ec0b1900 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.228+0000 7f50f1400640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50d4004d40 0x7f50ec1524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.229+0000 7f50f0a00640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50ec0b1900 0x7f50ec0b1d80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f50d800e870 tx=0x7f50d800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.229+0000 7f50e9600640 1 -- 172.21.15.40:0/1187287362 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d800c8d0 con 0x7f50ec0b1900 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.229+0000 7f50e9600640 1 -- 172.21.15.40:0/1187287362 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d8018070 con 0x7f50ec0b1900 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.229+0000 7f50e9600640 1 -- 172.21.15.40:0/1187287362 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d800ccb0 con 0x7f50ec0b1900 2024-04-26T23:01:49.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.229+0000 7f50f2800640 1 -- 172.21.15.40:0/1187287362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50ec0c8ce0 con 0x7f50ec0b1900 2024-04-26T23:01:49.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.229+0000 7f50f2800640 1 -- 172.21.15.40:0/1187287362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50ec152a20 con 0x7f50ec0b1900 2024-04-26T23:01:49.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.231+0000 7f50e9600640 1 -- 172.21.15.40:0/1187287362 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f50d801d460 con 0x7f50ec0b1900 2024-04-26T23:01:49.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.232+0000 7f50e9600640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f50cc06c920 0x7f50cc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.232+0000 7f50e9600640 1 -- 172.21.15.40:0/1187287362 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f50d808be50 con 0x7f50ec0b1900 2024-04-26T23:01:49.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.233+0000 7f50f6c37740 1 Processor -- start 2024-04-26T23:01:49.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.233+0000 7f50f1400640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f50cc06c920 0x7f50cc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.233+0000 7f50f6c37740 1 -- start start 2024-04-26T23:01:49.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.234+0000 7f50f6c37740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c317b0 0x5608d0c33910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.234+0000 7f50f6c37740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 0x5608d0c31b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.234+0000 7f50f6c37740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c5c270 0x5608d0c5c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.234+0000 7f50f6c37740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5608d0b95600 con 0x5608d0c6de80 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.234+0000 7f50f6c37740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5608d0b32100 con 0x5608d0c5c270 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.234+0000 7f50f6c37740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5608d0b915c0 con 0x5608d0c317b0 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.234+0000 7f50f1400640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f50cc06c920 0x7f50cc06ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f50e0009aa0 tx=0x7f50e0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.234+0000 7f50f0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 0x5608d0c31b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.234+0000 7f50f0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 0x5608d0c31b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48372/0 (socket says 172.21.15.40:48372) 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.234+0000 7f50f0a00640 1 -- 172.21.15.40:0/3773867474 learned_addr learned my addr 172.21.15.40:0/3773867474 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.235+0000 7f50f1400640 1 --2- 172.21.15.40:0/3773867474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c317b0 0x5608d0c33910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.235+0000 7f50f0a00640 1 -- 172.21.15.40:0/3773867474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c317b0 msgr2=0x5608d0c33910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.235+0000 7f50f0a00640 1 --2- 172.21.15.40:0/3773867474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c317b0 0x5608d0c33910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.235+0000 7f50f0a00640 1 -- 172.21.15.40:0/3773867474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c5c270 msgr2=0x5608d0c5c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.235+0000 7f50f1e00640 1 --2- 172.21.15.40:0/3773867474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c5c270 0x5608d0c5c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.235+0000 7f50f0a00640 1 --2- 172.21.15.40:0/3773867474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c5c270 0x5608d0c5c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.235+0000 7f50f0a00640 1 -- 172.21.15.40:0/3773867474 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608d08a72d0 con 0x5608d0c6de80 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.235+0000 7f50f0a00640 1 --2- 172.21.15.40:0/3773867474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 0x5608d0c31b90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f50d8089ee0 tx=0x7f50d8059f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab59c08475af533 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.235+0000 7f50ddc00640 1 -- 172.21.15.40:0/3773867474 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d8062350 con 0x5608d0c6de80 2024-04-26T23:01:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.235+0000 7f50ddc00640 1 -- 172.21.15.40:0/3773867474 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d806f070 con 0x5608d0c6de80 2024-04-26T23:01:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.235+0000 7f50ddc00640 1 -- 172.21.15.40:0/3773867474 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d8057a70 con 0x5608d0c6de80 2024-04-26T23:01:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.236+0000 7f50f6c37740 1 -- 172.21.15.40:0/3773867474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 msgr2=0x5608d0c31b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.236+0000 7f50f6c37740 1 --2- 172.21.15.40:0/3773867474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 0x5608d0c31b90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f50d8089ee0 tx=0x7f50d8059f30 comp rx=0 tx=0).stop 2024-04-26T23:01:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.236+0000 7f50f6c37740 1 -- 172.21.15.40:0/3773867474 shutdown_connections 2024-04-26T23:01:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.236+0000 7f50f6c37740 1 --2- 172.21.15.40:0/3773867474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c5c270 0x5608d0c5c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.236+0000 7f50f6c37740 1 --2- 172.21.15.40:0/3773867474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 0x5608d0c31b90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.236+0000 7f50f6c37740 1 --2- 172.21.15.40:0/3773867474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c317b0 0x5608d0c33910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.236+0000 7f50f6c37740 1 -- 172.21.15.40:0/3773867474 >> 172.21.15.40:0/3773867474 conn(0x5608d0bc5d10 msgr2=0x5608d0c77270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.236+0000 7f50f6c37740 1 -- 172.21.15.40:0/3773867474 shutdown_connections 2024-04-26T23:01:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.236+0000 7f50f6c37740 1 -- 172.21.15.40:0/3773867474 wait complete. 2024-04-26T23:01:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.237+0000 7f50f6c37740 1 -- start start 2024-04-26T23:01:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.237+0000 7f50f6c37740 1 Processor -- start 2024-04-26T23:01:49.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f6c37740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 0x5608d0c71780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f6c37740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c5c270 0x5608d0c58d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f6c37740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c6de80 0x5608d0c701d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f6c37740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5608d0b1cf70 con 0x5608d0c317b0 2024-04-26T23:01:49.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f6c37740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5608d0c59dc0 con 0x5608d0c6de80 2024-04-26T23:01:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f6c37740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5608d0c5c6f0 con 0x5608d0c5c270 2024-04-26T23:01:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f6c37740 10 client.0 _ll_register_callbacks cb 0x5608d0b8b480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:01:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 0x5608d0c71780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 0x5608d0c71780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48386/0 (socket says 172.21.15.40:48386) 2024-04-26T23:01:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f1400640 1 -- 172.21.15.40:0/754953039 learned_addr learned my addr 172.21.15.40:0/754953039 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f0a00640 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c5c270 0x5608d0c58d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f1e00640 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c6de80 0x5608d0c701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f1400640 1 -- 172.21.15.40:0/754953039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c5c270 msgr2=0x5608d0c58d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f1400640 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c5c270 0x5608d0c58d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f1400640 1 -- 172.21.15.40:0/754953039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c6de80 msgr2=0x5608d0c701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f1400640 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c6de80 0x5608d0c701d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f1400640 1 -- 172.21.15.40:0/754953039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608d09716b0 con 0x5608d0c317b0 2024-04-26T23:01:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.239+0000 7f50f1e00640 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c6de80 0x5608d0c701d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:49.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.240+0000 7f50f1400640 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 0x5608d0c71780 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f50e0015790 tx=0x7f50e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c9dc440e10952fa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:49.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.240+0000 7f50d2000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:01:49.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.240+0000 7f50d2000640 1 -- 172.21.15.40:0/754953039 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e0026070 con 0x5608d0c317b0 2024-04-26T23:01:49.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.240+0000 7f50d2000640 1 -- 172.21.15.40:0/754953039 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f50e0014960 con 0x5608d0c317b0 2024-04-26T23:01:49.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.240+0000 7f50f6c37740 1 -- 172.21.15.40:0/754953039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50d806f070 con 0x5608d0c317b0 2024-04-26T23:01:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.241+0000 7f50d2000640 1 -- 172.21.15.40:0/754953039 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f50e0014b60 con 0x5608d0c317b0 2024-04-26T23:01:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.241+0000 7f50f6c37740 10 client.4839 fetch_fsmap learned FSMap version 9 2024-04-26T23:01:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.241+0000 7f50f6c37740 1 -- 172.21.15.40:0/754953039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x5608d0c59790 con 0x5608d0c317b0 2024-04-26T23:01:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.241+0000 7f50d2000640 1 -- 172.21.15.40:0/754953039 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d806f070 con 0x5608d0c317b0 2024-04-26T23:01:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.241+0000 7f50d2000640 1 -- 172.21.15.40:0/754953039 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f50e002b440 con 0x5608d0c317b0 2024-04-26T23:01:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.241+0000 7f50f6c37740 10 client.4839 fetch_fsmap finished waiting for FSMap version 9 2024-04-26T23:01:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.241+0000 7f50f6c37740 10 client.4839 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.241+0000 7f50f6c37740 20 client.4839 populate_metadata read hostname 'smithi040' 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.241+0000 7f50f6c37740 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x5608d0c908c0 0x5608d0c94b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.241+0000 7f50f6c37740 4 client.4839 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.241+0000 7f50f6c37740 1 -- 172.21.15.40:0/754953039 --> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608d0ba3210 con 0x5608d0c908c0 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.242+0000 7f50f0a00640 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x5608d0c908c0 0x5608d0c94b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.242+0000 7f50f0a00640 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x5608d0c908c0 0x5608d0c94b50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c9db138b7497c7 server_cookie=32ae0ded37714670 in_seq=0 out_seq=0 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.242+0000 7f50d2000640 10 client.4839 ms_handle_connect on v2:172.21.15.40:6834/3989190340 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50d2000640 1 -- 172.21.15.40:0/754953039 <== mds.0 v2:172.21.15.40:6834/3989190340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5608d0ba3210 con 0x5608d0c908c0 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50d2000640 10 client.4839 handle_command_reply: tid=0 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50f6c37740 1 client.4839 shutdown 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50f6c37740 1 -- 172.21.15.40:0/754953039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 msgr2=0x5608d0c71780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50f6c37740 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 0x5608d0c71780 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f50e0015790 tx=0x7f50e0004730 comp rx=0 tx=0).stop 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50f6c37740 1 -- 172.21.15.40:0/754953039 shutdown_connections 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50f6c37740 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x5608d0c908c0 0x5608d0c94b50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50f6c37740 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c6de80 0x5608d0c701d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50f6c37740 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c5c270 0x5608d0c58d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50f6c37740 1 --2- 172.21.15.40:0/754953039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 0x5608d0c71780 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50d2000640 0 client.4839 ms_handle_reset on v2:172.21.15.40:6834/3989190340 2024-04-26T23:01:49.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50f6c37740 1 -- 172.21.15.40:0/754953039 >> 172.21.15.40:0/754953039 conn(0x5608d0bc5d10 msgr2=0x5608d0c59fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:49.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50f6c37740 1 -- 172.21.15.40:0/754953039 shutdown_connections 2024-04-26T23:01:49.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.243+0000 7f50f6c37740 1 -- 172.21.15.40:0/754953039 wait complete. 2024-04-26T23:01:49.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.244+0000 7f50f6c37740 20 client.4839 trim_cache size 0 max 16384 2024-04-26T23:01:49.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.264+0000 7f50f6c37740 1 Processor -- start 2024-04-26T23:01:49.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.264+0000 7f50f6c37740 1 -- start start 2024-04-26T23:01:49.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.265+0000 7f50f6c37740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c317b0 0x5608d0c89ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.265+0000 7f50f6c37740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c5c270 0x5608d0ceb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.265+0000 7f50f6c37740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 0x5608d0c89020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.265+0000 7f50f6c37740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec2022d0 con 0x5608d0c6de80 2024-04-26T23:01:49.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.265+0000 7f50f6c37740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec15c120 con 0x5608d0c5c270 2024-04-26T23:01:49.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.265+0000 7f50f6c37740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f50ec201c00 con 0x5608d0c317b0 2024-04-26T23:01:49.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.265+0000 7f50f1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 0x5608d0c89020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.265+0000 7f50f1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 0x5608d0c89020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48398/0 (socket says 172.21.15.40:48398) 2024-04-26T23:01:49.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.265+0000 7f50f1e00640 1 -- 172.21.15.40:0/3264409584 learned_addr learned my addr 172.21.15.40:0/3264409584 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:49.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.266+0000 7f50f1400640 1 --2- 172.21.15.40:0/3264409584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c317b0 0x5608d0c89ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.266+0000 7f50f1e00640 1 -- 172.21.15.40:0/3264409584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c317b0 msgr2=0x5608d0c89ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:49.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.266+0000 7f50f1e00640 1 --2- 172.21.15.40:0/3264409584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c317b0 0x5608d0c89ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.266+0000 7f50f1e00640 1 -- 172.21.15.40:0/3264409584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c5c270 msgr2=0x5608d0ceb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:49.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.266+0000 7f50f1e00640 1 --2- 172.21.15.40:0/3264409584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c5c270 0x5608d0ceb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.266+0000 7f50f1e00640 1 -- 172.21.15.40:0/3264409584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608d0c28680 con 0x5608d0c6de80 2024-04-26T23:01:49.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.266+0000 7f50f1e00640 1 --2- 172.21.15.40:0/3264409584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 0x5608d0c89020 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f50e400e200 tx=0x7f50e402d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a302bcd7056f4f8c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:49.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.266+0000 7f50dd200640 1 -- 172.21.15.40:0/3264409584 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e4003990 con 0x5608d0c6de80 2024-04-26T23:01:49.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.266+0000 7f50dd200640 1 -- 172.21.15.40:0/3264409584 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50e4036040 con 0x5608d0c6de80 2024-04-26T23:01:49.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.266+0000 7f50dd200640 1 -- 172.21.15.40:0/3264409584 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e4003d70 con 0x5608d0c6de80 2024-04-26T23:01:49.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.267+0000 7f50f6c37740 1 -- 172.21.15.40:0/3264409584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 msgr2=0x5608d0c89020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:49.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.267+0000 7f50f6c37740 1 --2- 172.21.15.40:0/3264409584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 0x5608d0c89020 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f50e400e200 tx=0x7f50e402d000 comp rx=0 tx=0).stop 2024-04-26T23:01:49.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.267+0000 7f50f6c37740 1 -- 172.21.15.40:0/3264409584 shutdown_connections 2024-04-26T23:01:49.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.267+0000 7f50f6c37740 1 --2- 172.21.15.40:0/3264409584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c6de80 0x5608d0c89020 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.267+0000 7f50f6c37740 1 --2- 172.21.15.40:0/3264409584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c5c270 0x5608d0ceb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.267+0000 7f50f6c37740 1 --2- 172.21.15.40:0/3264409584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c317b0 0x5608d0c89ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.267+0000 7f50f6c37740 1 -- 172.21.15.40:0/3264409584 >> 172.21.15.40:0/3264409584 conn(0x5608d0bc5d10 msgr2=0x5608d0c27df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:49.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.267+0000 7f50f6c37740 1 -- 172.21.15.40:0/3264409584 shutdown_connections 2024-04-26T23:01:49.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.267+0000 7f50f6c37740 1 -- 172.21.15.40:0/3264409584 wait complete. 2024-04-26T23:01:49.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.268+0000 7f50f6c37740 1 -- start start 2024-04-26T23:01:49.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.268+0000 7f50f6c37740 1 Processor -- start 2024-04-26T23:01:49.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.269+0000 7f50f6c37740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 0x5608d0c98fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.269+0000 7f50f6c37740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c5c270 0x5608d0c994e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.269+0000 7f50f6c37740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c6de80 0x5608d0c6cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.269+0000 7f50f6c37740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5608d0c90740 con 0x5608d0c317b0 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.269+0000 7f50f6c37740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5608d0c5f0a0 con 0x5608d0c6de80 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.269+0000 7f50f6c37740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5608d0c90cd0 con 0x5608d0c5c270 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.269+0000 7f50f6c37740 10 client.0 _ll_register_callbacks cb 0x7f50e00097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.269+0000 7f50f1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 0x5608d0c98fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.269+0000 7f50f1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 0x5608d0c98fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48408/0 (socket says 172.21.15.40:48408) 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.269+0000 7f50f1400640 1 -- 172.21.15.40:0/4115437393 learned_addr learned my addr 172.21.15.40:0/4115437393 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.270+0000 7f50f0a00640 1 --2- 172.21.15.40:0/4115437393 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c5c270 0x5608d0c994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.270+0000 7f50f1e00640 1 --2- 172.21.15.40:0/4115437393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c6de80 0x5608d0c6cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.270+0000 7f50f1400640 1 -- 172.21.15.40:0/4115437393 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c5c270 msgr2=0x5608d0c994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.270+0000 7f50f1400640 1 --2- 172.21.15.40:0/4115437393 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c5c270 0x5608d0c994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.270+0000 7f50f1400640 1 -- 172.21.15.40:0/4115437393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c6de80 msgr2=0x5608d0c6cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:49.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.270+0000 7f50f1400640 1 --2- 172.21.15.40:0/4115437393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c6de80 0x5608d0c6cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.270+0000 7f50f1400640 1 -- 172.21.15.40:0/4115437393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608d0cf1450 con 0x5608d0c317b0 2024-04-26T23:01:49.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.270+0000 7f50f1400640 1 --2- 172.21.15.40:0/4115437393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 0x5608d0c98fa0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f50e0004700 tx=0x7f50e0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56612d90b269be1e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:49.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.270+0000 7f50d2000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:01:49.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.270+0000 7f50d2000640 1 -- 172.21.15.40:0/4115437393 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e003b060 con 0x5608d0c317b0 2024-04-26T23:01:49.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.270+0000 7f50f6c37740 1 -- 172.21.15.40:0/4115437393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50e4036040 con 0x5608d0c317b0 2024-04-26T23:01:49.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.270+0000 7f50d2000640 1 -- 172.21.15.40:0/4115437393 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f50e002b440 con 0x5608d0c317b0 2024-04-26T23:01:49.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.271+0000 7f50d2000640 1 -- 172.21.15.40:0/4115437393 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f50e0041050 con 0x5608d0c317b0 2024-04-26T23:01:49.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.271+0000 7f50f6c37740 10 client.4845 fetch_fsmap learned FSMap version 9 2024-04-26T23:01:49.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.271+0000 7f50f6c37740 1 -- 172.21.15.40:0/4115437393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x5608d0ce76a0 con 0x5608d0c317b0 2024-04-26T23:01:49.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.271+0000 7f50d2000640 1 -- 172.21.15.40:0/4115437393 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50e4036040 con 0x5608d0c317b0 2024-04-26T23:01:49.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.271+0000 7f50d2000640 1 -- 172.21.15.40:0/4115437393 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f50e0044020 con 0x5608d0c317b0 2024-04-26T23:01:49.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.271+0000 7f50f6c37740 10 client.4845 fetch_fsmap finished waiting for FSMap version 9 2024-04-26T23:01:49.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.271+0000 7f50f6c37740 10 client.4845 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.271+0000 7f50f6c37740 20 client.4845 populate_metadata read hostname 'smithi040' 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.271+0000 7f50f6c37740 1 --2- 172.21.15.40:0/4115437393 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x5608d0c6d710 0x5608d0ce7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.271+0000 7f50f6c37740 4 client.4845 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.271+0000 7f50f6c37740 1 -- 172.21.15.40:0/4115437393 --> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] -- command(tid 0: {"prefix": "status"}) -- 0x5608d0c32a00 con 0x5608d0c6d710 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.272+0000 7f50f0a00640 1 --2- 172.21.15.40:0/4115437393 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x5608d0c6d710 0x5608d0ce7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.272+0000 7f50f0a00640 1 --2- 172.21.15.40:0/4115437393 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x5608d0c6d710 0x5608d0ce7bc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c316cef4908248c7 server_cookie=c7ff3746e912ad29 in_seq=0 out_seq=0 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.272+0000 7f50d2000640 10 client.4845 ms_handle_connect on v2:172.21.15.40:6834/3989190340 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.272+0000 7f50d2000640 1 -- 172.21.15.40:0/4115437393 <== mds.0 v2:172.21.15.40:6834/3989190340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5608d0c32a00 con 0x5608d0c6d710 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.272+0000 7f50d2000640 10 client.4845 handle_command_reply: tid=0 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.272+0000 7f50f6c37740 1 client.4845 shutdown 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.273+0000 7f50f6c37740 1 -- 172.21.15.40:0/4115437393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 msgr2=0x5608d0c98fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.273+0000 7f50f6c37740 1 --2- 172.21.15.40:0/4115437393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 0x5608d0c98fa0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f50e0004700 tx=0x7f50e0016000 comp rx=0 tx=0).stop 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.273+0000 7f50f6c37740 1 -- 172.21.15.40:0/4115437393 shutdown_connections 2024-04-26T23:01:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.273+0000 7f50f6c37740 1 --2- 172.21.15.40:0/4115437393 >> [v2:172.21.15.40:6834/3989190340,v1:172.21.15.40:6835/3989190340] conn(0x5608d0c6d710 0x5608d0ce7bc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.273+0000 7f50f6c37740 1 --2- 172.21.15.40:0/4115437393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608d0c6de80 0x5608d0c6cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.273+0000 7f50f6c37740 1 --2- 172.21.15.40:0/4115437393 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5608d0c5c270 0x5608d0c994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.273+0000 7f50f6c37740 1 --2- 172.21.15.40:0/4115437393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608d0c317b0 0x5608d0c98fa0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.273+0000 7f50f6c37740 1 -- 172.21.15.40:0/4115437393 >> 172.21.15.40:0/4115437393 conn(0x5608d0bc5d10 msgr2=0x5608d0c71d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.273+0000 7f50d2000640 0 client.4845 ms_handle_reset on v2:172.21.15.40:6834/3989190340 2024-04-26T23:01:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.273+0000 7f50f6c37740 1 -- 172.21.15.40:0/4115437393 shutdown_connections 2024-04-26T23:01:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.273+0000 7f50f6c37740 1 -- 172.21.15.40:0/4115437393 wait complete. 2024-04-26T23:01:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.273+0000 7f50f6c37740 20 client.4845 trim_cache size 0 max 16384 2024-04-26T23:01:49.276 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:01:49.276 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:01:49.276 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:01:49.277 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4257, 2024-04-26T23:01:49.277 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:01:49.277 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:01:49.277 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:01:49.277 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 43.556725444000001, 2024-04-26T23:01:49.277 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 9, 2024-04-26T23:01:49.277 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 27, 2024-04-26T23:01:49.277 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 25, 2024-04-26T23:01:49.277 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 49.6464116 2024-04-26T23:01:49.277 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:01:49.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.274+0000 7f50f2800640 1 -- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f50cc06c920 msgr2=0x7f50cc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:49.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.274+0000 7f50f2800640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f50cc06c920 0x7f50cc06ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f50e0009aa0 tx=0x7f50e0008040 comp rx=0 tx=0).stop 2024-04-26T23:01:49.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.274+0000 7f50f2800640 1 -- 172.21.15.40:0/1187287362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50ec0b1900 msgr2=0x7f50ec0b1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:49.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.274+0000 7f50f2800640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50ec0b1900 0x7f50ec0b1d80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f50d800e870 tx=0x7f50d800ed30 comp rx=0 tx=0).stop 2024-04-26T23:01:49.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.274+0000 7f50f2800640 1 -- 172.21.15.40:0/1187287362 shutdown_connections 2024-04-26T23:01:49.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.274+0000 7f50f2800640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f50cc06c920 0x7f50cc06ede0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.274+0000 7f50f2800640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50ec0c6e90 0x7f50ec0c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.274+0000 7f50f2800640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50ec0b1900 0x7f50ec0b1d80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.274+0000 7f50f2800640 1 --2- 172.21.15.40:0/1187287362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50d4004d40 0x7f50ec1524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:49.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.274+0000 7f50f2800640 1 -- 172.21.15.40:0/1187287362 >> 172.21.15.40:0/1187287362 conn(0x7f50ec0d0a10 msgr2=0x7f50ec06e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:49.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.274+0000 7f50f2800640 1 -- 172.21.15.40:0/1187287362 shutdown_connections 2024-04-26T23:01:49.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:49.274+0000 7f50f2800640 1 -- 172.21.15.40:0/1187287362 wait complete. 2024-04-26T23:01:49.293 INFO:teuthology.run_tasks:Running task sequential... 2024-04-26T23:01:49.302 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-26T23:01:49.303 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-26T23:01:49.303 INFO:teuthology.task.print:Enabling mgr modules 2024-04-26T23:01:49.303 INFO:teuthology.run_tasks:Running task kclient... 2024-04-26T23:01:49.320 INFO:tasks.kclient:Mounting kernel clients... 2024-04-26T23:01:49.320 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-26T23:01:49.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:49.320 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:01:49.378 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-26T23:01:49.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:49.379 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:01:49.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:49.464 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2024-04-26T23:01:49.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:49.581 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:01:49.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:49.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2024-04-26T23:01:49.762 INFO:teuthology.orchestra.run.smithi040.stderr:Cannot find device "ceph-brx" 2024-04-26T23:01:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:01:49.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:49.767 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-26T23:01:49.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:49.824 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-04-26T23:01:49.903 INFO:teuthology.orchestra.run.smithi103.stderr:Cannot find device "ceph-brx" 2024-04-26T23:01:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:01:49.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:49.908 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-26T23:01:49.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:49.957 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-04-26T23:01:50.041 INFO:teuthology.orchestra.run.smithi103.stderr:Cannot find device "ceph-brx" 2024-04-26T23:01:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:01:50.046 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-04-26T23:01:50.046 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T23:01:50.046 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T23:01:50.046 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-26T23:01:50.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.265+0000 7f5361c00640 1 Processor -- start 2024-04-26T23:01:50.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.265+0000 7f5361c00640 1 -- start start 2024-04-26T23:01:50.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.268+0000 7f5361c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f535c146060 0x7f535c0b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:50.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.268+0000 7f5361c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f535c0b3fb0 0x7f535c0d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:50.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.268+0000 7f5361c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f535c0b59f0 con 0x7f535c145970 2024-04-26T23:01:50.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.268+0000 7f5361c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f535c0ceed0 con 0x7f535c146060 2024-04-26T23:01:50.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.268+0000 7f5361c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f535c0b5f40 con 0x7f535c0b3fb0 2024-04-26T23:01:50.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.268+0000 7f5353e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f535c146060 0x7f535c0b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:50.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.268+0000 7f5353e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f535c146060 0x7f535c0b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48420/0 (socket says 172.21.15.40:48420) 2024-04-26T23:01:50.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.268+0000 7f5353e00640 1 -- 172.21.15.40:0/1139973117 learned_addr learned my addr 172.21.15.40:0/1139973117 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:50.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.268+0000 7f5361200640 1 --2- 172.21.15.40:0/1139973117 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f535c0b3fb0 0x7f535c0d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:50.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.268+0000 7f5353e00640 1 -- 172.21.15.40:0/1139973117 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f535c0b3fb0 msgr2=0x7f535c0d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:50.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.268+0000 7f5353e00640 1 --2- 172.21.15.40:0/1139973117 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f535c0b3fb0 0x7f535c0d06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:50.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.268+0000 7f5353e00640 1 -- 172.21.15.40:0/1139973117 >> v1:172.21.15.103:6789/0 conn(0x7f535c145970 legacy=0x7f535c1612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:50.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.269+0000 7f5353e00640 1 -- 172.21.15.40:0/1139973117 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f535c0b5c60 con 0x7f535c146060 2024-04-26T23:01:50.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.270+0000 7f5353e00640 1 --2- 172.21.15.40:0/1139973117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f535c146060 0x7f535c0b3a70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f53440098a0 tx=0x7f534402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e6af96de8e51a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:50.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.270+0000 7f535b400640 1 -- 172.21.15.40:0/1139973117 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5344037070 con 0x7f535c146060 2024-04-26T23:01:50.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.270+0000 7f535b400640 1 -- 172.21.15.40:0/1139973117 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5344032440 con 0x7f535c146060 2024-04-26T23:01:50.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.270+0000 7f535b400640 1 -- 172.21.15.40:0/1139973117 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5344032810 con 0x7f535c146060 2024-04-26T23:01:50.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.270+0000 7f5361c00640 1 -- 172.21.15.40:0/1139973117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f535c146060 msgr2=0x7f535c0b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:50.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.270+0000 7f5361c00640 1 --2- 172.21.15.40:0/1139973117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f535c146060 0x7f535c0b3a70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f53440098a0 tx=0x7f534402a250 comp rx=0 tx=0).stop 2024-04-26T23:01:50.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.271+0000 7f5361c00640 1 -- 172.21.15.40:0/1139973117 shutdown_connections 2024-04-26T23:01:50.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.271+0000 7f5361c00640 1 --2- 172.21.15.40:0/1139973117 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f535c0b3fb0 0x7f535c0d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:50.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.271+0000 7f5361c00640 1 --2- 172.21.15.40:0/1139973117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f535c146060 0x7f535c0b3a70 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:50.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.271+0000 7f5361c00640 1 -- 172.21.15.40:0/1139973117 >> 172.21.15.40:0/1139973117 conn(0x7f535c075eb0 msgr2=0x7f535c068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:50.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.271+0000 7f5361c00640 1 -- 172.21.15.40:0/1139973117 shutdown_connections 2024-04-26T23:01:50.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.271+0000 7f5361c00640 1 -- 172.21.15.40:0/1139973117 wait complete. 2024-04-26T23:01:50.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.272+0000 7f5361c00640 1 Processor -- start 2024-04-26T23:01:50.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.273+0000 7f5361c00640 1 -- start start 2024-04-26T23:01:50.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.273+0000 7f5361c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f535c145970 0x7f535c15bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:50.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.273+0000 7f5361c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f535c146060 0x7f535c071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:50.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.273+0000 7f5361c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f535c15c2c0 0x7f535c072380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:50.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.273+0000 7f5361c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f535c145d50 con 0x7f535c145970 2024-04-26T23:01:50.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.273+0000 7f5361c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f535c0cf3e0 con 0x7f535c146060 2024-04-26T23:01:50.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.273+0000 7f5361c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f535c0d0bf0 con 0x7f535c15c2c0 2024-04-26T23:01:50.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.274+0000 7f535be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f535c145970 0x7f535c15bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.274+0000 7f535be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f535c145970 0x7f535c15bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48430/0 (socket says 172.21.15.40:48430) 2024-04-26T23:01:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.274+0000 7f535be00640 1 -- 172.21.15.40:0/3883526808 learned_addr learned my addr 172.21.15.40:0/3883526808 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.274+0000 7f5361200640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f535c15c2c0 0x7f535c072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.274+0000 7f5353e00640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f535c146060 0x7f535c071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.274+0000 7f535be00640 1 -- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f535c15c2c0 msgr2=0x7f535c072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.274+0000 7f535be00640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f535c15c2c0 0x7f535c072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.274+0000 7f535be00640 1 -- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f535c146060 msgr2=0x7f535c071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.274+0000 7f535be00640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f535c146060 0x7f535c071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.274+0000 7f535be00640 1 -- 172.21.15.40:0/3883526808 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f535c072a00 con 0x7f535c145970 2024-04-26T23:01:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.274+0000 7f5361200640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f535c15c2c0 0x7f535c072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:01:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.275+0000 7f535be00640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f535c145970 0x7f535c15bd80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f534c00c430 tx=0x7f534c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.275+0000 7f5358c00640 1 -- 172.21.15.40:0/3883526808 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c011790 con 0x7f535c145970 2024-04-26T23:01:50.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.275+0000 7f5358c00640 1 -- 172.21.15.40:0/3883526808 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f534c017070 con 0x7f535c145970 2024-04-26T23:01:50.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.275+0000 7f5358c00640 1 -- 172.21.15.40:0/3883526808 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c011bb0 con 0x7f535c145970 2024-04-26T23:01:50.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.275+0000 7f5361c00640 1 -- 172.21.15.40:0/3883526808 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f535c072ce0 con 0x7f535c145970 2024-04-26T23:01:50.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.275+0000 7f5361c00640 1 -- 172.21.15.40:0/3883526808 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f535c0732a0 con 0x7f535c145970 2024-04-26T23:01:50.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.277+0000 7f5361c00640 1 -- 172.21.15.40:0/3883526808 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f535c069370 con 0x7f535c145970 2024-04-26T23:01:50.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.278+0000 7f5358c00640 1 -- 172.21.15.40:0/3883526808 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f534c01c460 con 0x7f535c145970 2024-04-26T23:01:50.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.279+0000 7f5358c00640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f533806c8f0 0x7f533806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:50.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.279+0000 7f5358c00640 1 -- 172.21.15.40:0/3883526808 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f534c08be90 con 0x7f535c145970 2024-04-26T23:01:50.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.279+0000 7f5353e00640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f533806c8f0 0x7f533806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:50.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.281+0000 7f5353e00640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f533806c8f0 0x7f533806edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5344034000 tx=0x7f534402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:50.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.284+0000 7f5358c00640 1 -- 172.21.15.40:0/3883526808 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f534c059e20 con 0x7f535c145970 2024-04-26T23:01:50.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.537+0000 7f5361c00640 1 -- 172.21.15.40:0/3883526808 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f535c0d0fc0 con 0x7f535c145970 2024-04-26T23:01:50.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.537+0000 7f5358c00640 1 -- 172.21.15.40:0/3883526808 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f534c08d070 con 0x7f535c145970 2024-04-26T23:01:50.538 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T23:01:50.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.540+0000 7f5361c00640 1 -- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f533806c8f0 msgr2=0x7f533806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:50.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.540+0000 7f5361c00640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f533806c8f0 0x7f533806edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5344034000 tx=0x7f534402e000 comp rx=0 tx=0).stop 2024-04-26T23:01:50.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.540+0000 7f5361c00640 1 -- 172.21.15.40:0/3883526808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f535c145970 msgr2=0x7f535c15bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:50.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.540+0000 7f5361c00640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f535c145970 0x7f535c15bd80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f534c00c430 tx=0x7f534c00c8f0 comp rx=0 tx=0).stop 2024-04-26T23:01:50.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.540+0000 7f5361c00640 1 -- 172.21.15.40:0/3883526808 shutdown_connections 2024-04-26T23:01:50.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.541+0000 7f5361c00640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f533806c8f0 0x7f533806edb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:50.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.541+0000 7f5361c00640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f535c15c2c0 0x7f535c072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:50.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.541+0000 7f5361c00640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f535c146060 0x7f535c071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:50.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.541+0000 7f5361c00640 1 --2- 172.21.15.40:0/3883526808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f535c145970 0x7f535c15bd80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:50.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.541+0000 7f5361c00640 1 -- 172.21.15.40:0/3883526808 >> 172.21.15.40:0/3883526808 conn(0x7f535c075eb0 msgr2=0x7f535c0b5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:50.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.541+0000 7f5361c00640 1 -- 172.21.15.40:0/3883526808 shutdown_connections 2024-04-26T23:01:50.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50.541+0000 7f5361c00640 1 -- 172.21.15.40:0/3883526808 wait complete. 2024-04-26T23:01:50.561 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T23:01:50.561 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:01:50.561 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-04-26T23:01:50.562 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T23:01:50.562 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T23:01:50.562 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T23:01:50.562 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T23:01:50.562 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:01:50.562 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:01:50.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:50.562 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-04-26T23:01:50.614 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:50.615 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-04-26T23:01:50.616 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T23:01:50.616 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42604sec preferred_lft 42604sec 2024-04-26T23:01:50.616 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-26T23:01:50.616 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:50.616 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:01:50.616 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:50.616 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-04-26T23:01:50.617 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T23:01:50.617 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:50.617 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:50.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add name ceph-brx type bridge 2024-04-26T23:01:50.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr flush dev ceph-brx 2024-04-26T23:01:50.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set ceph-brx up 2024-04-26T23:01:50.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T23:01:50.617 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:50.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:50.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:50.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:50.870 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:01:50.942 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-04-26T23:01:50.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:50.946 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-04-26T23:01:51.003 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-04-26T23:01:51.003 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:01:51.003 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T23:01:51.003 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T23:01:51.003 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:01:51.004 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:51.004 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:51.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T23:01:51.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T23:01:51.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T23:01:51.004 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:51.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:51.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:51.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:51.226 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:01:51.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:51.277 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list-id 2024-04-26T23:01:51.331 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:51.331 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:51.331 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T23:01:51.331 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-26T23:01:51.331 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:51.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:51.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:51.522 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-26T23:01:51.522 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:51.523 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:51.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-26T23:01:51.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T23:01:51.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-26T23:01:51.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-26T23:01:51.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-26T23:01:51.523 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:51.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:51.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:51.812 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:51.812 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:51.813 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-04-26T23:01:51.813 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T23:01:51.813 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:51.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:52.025 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:01:52.026 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T23:01:52.075 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:01:52.076 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:01:52.076 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:01:52.175 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-26T23:01:52.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:52.175 DEBUG:teuthology.orchestra.run.smithi040:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:01:52.311 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:01:52.311 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:01:52.311 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:01:53.048 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:01:53.048 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:01:53.048 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:01:53.049 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:01:53.049 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:01:53.085 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:01:53.085 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:01:53.085 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:01:53.085 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:01:53.086 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:01:53.086 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-26T23:01:53.086 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-26T23:01:53.086 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=815936ab-d4ac-471e-b069-1a0dc70078ee" will pass to kernel 2024-04-26T23:01:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T23:01:53.091 INFO:teuthology.orchestra.run.smithi040.stdout: You just mounted a file system that supports labels which does not 2024-04-26T23:01:53.091 INFO:teuthology.orchestra.run.smithi040.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T23:01:53.091 INFO:teuthology.orchestra.run.smithi040.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T23:01:53.092 INFO:teuthology.orchestra.run.smithi040.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T23:01:53.097 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T23:01:53.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:53.098 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:01:53.191 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:01:53.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:01:53.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:53.324 INFO:teuthology.orchestra.run.smithi040.stdout:client4562 2024-04-26T23:01:53.324 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:01:53.324 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:01:53.378 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:01:53.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:01:53.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:53.494 INFO:teuthology.orchestra.run.smithi040.stdout:client4562 2024-04-26T23:01:53.495 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:01:53.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:01:53.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:53.614 INFO:teuthology.orchestra.run.smithi040.stdout:client4562 2024-04-26T23:01:53.614 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/815936ab-d4ac-471e-b069-1a0dc70078ee.client4562/status') 2024-04-26T23:01:53.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:53.752 INFO:teuthology.orchestra.run.smithi040.stdout:instance: client.4562 (3)172.21.15.40:0/3197704350 2024-04-26T23:01:53.752 INFO:teuthology.orchestra.run.smithi040.stdout:blocklisted: false 2024-04-26T23:01:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:0+1 records in 2024-04-26T23:01:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:0+1 records out 2024-04-26T23:01:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:70 bytes copied, 7.1037e-05 s, 985 kB/s 2024-04-26T23:01:53.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:53.766 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-04-26T23:01:53.766 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T23:01:53.766 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-26T23:01:53.766 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-26T23:01:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.967+0000 7f194d600640 1 Processor -- start 2024-04-26T23:01:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.967+0000 7f194d600640 1 -- start start 2024-04-26T23:01:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.969+0000 7f194d600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f194815c160 0x7f19480b4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.969+0000 7f194d600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f194815c5c0 0x7f19480b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.969+0000 7f194d600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f194815cb90 con 0x7f1948073e10 2024-04-26T23:01:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.969+0000 7f194d600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1948153040 con 0x7f194815c160 2024-04-26T23:01:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.969+0000 7f194d600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f194815fc40 con 0x7f194815c5c0 2024-04-26T23:01:53.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.969+0000 7f1947e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1948073e10 0x7f19481600d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:47962/0 (socket says 172.21.15.40:47962) 2024-04-26T23:01:53.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.969+0000 7f1947e00640 1 -- 172.21.15.40:0/938374932 learned_addr learned my addr 172.21.15.40:0/938374932 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:53.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.969+0000 7f1947400640 1 --2- 172.21.15.40:0/938374932 >> v2:172.21.15.40:3300/0 conn(0x7f194815c160 0x7f19480b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:53.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.970+0000 7f194cc00640 1 --2- 172.21.15.40:0/938374932 >> v2:172.21.15.103:3300/0 conn(0x7f194815c5c0 0x7f19480b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:53.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.970+0000 7f1947400640 1 -- 172.21.15.40:0/938374932 >> v2:172.21.15.103:3300/0 conn(0x7f194815c5c0 msgr2=0x7f19480b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.970+0000 7f1946a00640 1 -- 172.21.15.40:0/938374932 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592237306 0 0) 0x7f194815cb90 con 0x7f1948073e10 2024-04-26T23:01:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.970+0000 7f1947400640 1 --2- 172.21.15.40:0/938374932 >> v2:172.21.15.103:3300/0 conn(0x7f194815c5c0 0x7f19480b4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.970+0000 7f1947400640 1 -- 172.21.15.40:0/938374932 >> v1:172.21.15.40:6789/0 conn(0x7f1948073e10 legacy=0x7f19481600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.970+0000 7f1947400640 1 -- 172.21.15.40:0/938374932 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194815ce70 con 0x7f194815c160 2024-04-26T23:01:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.970+0000 7f194cc00640 1 --2- 172.21.15.40:0/938374932 >> v2:172.21.15.103:3300/0 conn(0x7f194815c5c0 0x7f19480b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:53.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.971+0000 7f1947400640 1 --2- 172.21.15.40:0/938374932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f194815c160 0x7f19480b4840 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1948159ca0 tx=0x7f193402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214f2cc6a4cd2a58 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:53.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.971+0000 7f1946a00640 1 -- 172.21.15.40:0/938374932 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934037070 con 0x7f194815c160 2024-04-26T23:01:53.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.971+0000 7f1946a00640 1 -- 172.21.15.40:0/938374932 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1934032440 con 0x7f194815c160 2024-04-26T23:01:53.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.971+0000 7f1946a00640 1 -- 172.21.15.40:0/938374932 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934032810 con 0x7f194815c160 2024-04-26T23:01:53.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.972+0000 7f194d600640 1 -- 172.21.15.40:0/938374932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f194815c160 msgr2=0x7f19480b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:53.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.972+0000 7f194d600640 1 --2- 172.21.15.40:0/938374932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f194815c160 0x7f19480b4840 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1948159ca0 tx=0x7f193402a1a0 comp rx=0 tx=0).stop 2024-04-26T23:01:53.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.972+0000 7f194d600640 1 -- 172.21.15.40:0/938374932 shutdown_connections 2024-04-26T23:01:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.972+0000 7f194d600640 1 --2- 172.21.15.40:0/938374932 >> v2:172.21.15.103:3300/0 conn(0x7f194815c5c0 0x7f19480b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.972+0000 7f194d600640 1 --2- 172.21.15.40:0/938374932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f194815c160 0x7f19480b4840 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.972+0000 7f194d600640 1 -- 172.21.15.40:0/938374932 >> 172.21.15.40:0/938374932 conn(0x7f1948075eb0 msgr2=0x7f1948068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.972+0000 7f194d600640 1 -- 172.21.15.40:0/938374932 shutdown_connections 2024-04-26T23:01:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.972+0000 7f194d600640 1 -- 172.21.15.40:0/938374932 wait complete. 2024-04-26T23:01:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.974+0000 7f194d600640 1 Processor -- start 2024-04-26T23:01:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.974+0000 7f194d600640 1 -- start start 2024-04-26T23:01:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.975+0000 7f194d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1948073e10 0x7f1948146200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.975+0000 7f194d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f194815c160 0x7f19481571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.975+0000 7f194d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194815c5c0 0x7f19481576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.975+0000 7f194d600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1948073540 con 0x7f194815c160 2024-04-26T23:01:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.975+0000 7f194d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f19480733c0 con 0x7f1948073e10 2024-04-26T23:01:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.975+0000 7f194d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f194815d150 con 0x7f194815c5c0 2024-04-26T23:01:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.975+0000 7f1947400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f194815c160 0x7f19481571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.975+0000 7f1947400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f194815c160 0x7f19481571a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48446/0 (socket says 172.21.15.40:48446) 2024-04-26T23:01:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.975+0000 7f1947400640 1 -- 172.21.15.40:0/2218682398 learned_addr learned my addr 172.21.15.40:0/2218682398 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.976+0000 7f1947e00640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1948073e10 0x7f1948146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.976+0000 7f194cc00640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194815c5c0 0x7f19481576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.976+0000 7f1947400640 1 -- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194815c5c0 msgr2=0x7f19481576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.976+0000 7f1947400640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194815c5c0 0x7f19481576e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.976+0000 7f1947400640 1 -- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1948073e10 msgr2=0x7f1948146200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.976+0000 7f1947400640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1948073e10 0x7f1948146200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.976+0000 7f1947400640 1 -- 172.21.15.40:0/2218682398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194814d670 con 0x7f194815c160 2024-04-26T23:01:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.976+0000 7f1947e00640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1948073e10 0x7f1948146200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.976+0000 7f1947400640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f194815c160 0x7f19481571a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1934035040 tx=0x7f19340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.977+0000 7f193fe00640 1 -- 172.21.15.40:0/2218682398 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934037050 con 0x7f194815c160 2024-04-26T23:01:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.977+0000 7f193fe00640 1 -- 172.21.15.40:0/2218682398 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1934038040 con 0x7f194815c160 2024-04-26T23:01:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.977+0000 7f193fe00640 1 -- 172.21.15.40:0/2218682398 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934004450 con 0x7f194815c160 2024-04-26T23:01:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.977+0000 7f194d600640 1 -- 172.21.15.40:0/2218682398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f194814d910 con 0x7f194815c160 2024-04-26T23:01:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.977+0000 7f194d600640 1 -- 172.21.15.40:0/2218682398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f194814ded0 con 0x7f194815c160 2024-04-26T23:01:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.978+0000 7f194d600640 1 -- 172.21.15.40:0/2218682398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1948069370 con 0x7f194815c160 2024-04-26T23:01:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.979+0000 7f193fe00640 1 -- 172.21.15.40:0/2218682398 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1934030980 con 0x7f194815c160 2024-04-26T23:01:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.980+0000 7f193fe00640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f192806c920 0x7f192806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.980+0000 7f193fe00640 1 -- 172.21.15.40:0/2218682398 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f19340acc90 con 0x7f194815c160 2024-04-26T23:01:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.981+0000 7f1947e00640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f192806c920 0x7f192806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.982+0000 7f1947e00640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f192806c920 0x7f192806ede0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1938000f60 tx=0x7f1938009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:53.985+0000 7f193fe00640 1 -- 172.21.15.40:0/2218682398 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f193407ad40 con 0x7f194815c160 2024-04-26T23:01:54.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.233+0000 7f194d600640 1 -- 172.21.15.40:0/2218682398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f194806f4e0 con 0x7f194815c160 2024-04-26T23:01:54.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.234+0000 7f193fe00640 1 -- 172.21.15.40:0/2218682398 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f193407a640 con 0x7f194815c160 2024-04-26T23:01:54.234 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T23:01:54.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.237+0000 7f194d600640 1 -- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f192806c920 msgr2=0x7f192806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:54.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.237+0000 7f194d600640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f192806c920 0x7f192806ede0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1938000f60 tx=0x7f1938009040 comp rx=0 tx=0).stop 2024-04-26T23:01:54.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.237+0000 7f194d600640 1 -- 172.21.15.40:0/2218682398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f194815c160 msgr2=0x7f19481571a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:54.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.237+0000 7f194d600640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f194815c160 0x7f19481571a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1934035040 tx=0x7f19340046c0 comp rx=0 tx=0).stop 2024-04-26T23:01:54.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.237+0000 7f194d600640 1 -- 172.21.15.40:0/2218682398 shutdown_connections 2024-04-26T23:01:54.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.237+0000 7f194d600640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f192806c920 0x7f192806ede0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:54.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.237+0000 7f194d600640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f194815c5c0 0x7f19481576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:54.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.237+0000 7f194d600640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f194815c160 0x7f19481571a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:54.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.237+0000 7f194d600640 1 --2- 172.21.15.40:0/2218682398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1948073e10 0x7f1948146200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:54.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.237+0000 7f194d600640 1 -- 172.21.15.40:0/2218682398 >> 172.21.15.40:0/2218682398 conn(0x7f1948075eb0 msgr2=0x7f1948159cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:54.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.237+0000 7f194d600640 1 -- 172.21.15.40:0/2218682398 shutdown_connections 2024-04-26T23:01:54.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54.237+0000 7f194d600640 1 -- 172.21.15.40:0/2218682398 wait complete. 2024-04-26T23:01:54.257 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T23:01:54.257 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:01:54.257 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-04-26T23:01:54.257 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-26T23:01:54.257 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-26T23:01:54.257 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T23:01:54.257 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T23:01:54.257 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:01:54.257 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-26T23:01:54.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:54.258 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-04-26T23:01:54.313 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:01:54.313 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:01:54.313 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:01:54.313 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:54.313 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-26T23:01:54.313 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:54.313 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:01:54.313 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:54.313 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-04-26T23:01:54.313 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:01:54.313 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:54.313 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-04-26T23:01:54.314 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:01:54.314 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:54.314 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-04-26T23:01:54.314 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T23:01:54.314 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42600sec preferred_lft 42600sec 2024-04-26T23:01:54.314 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-26T23:01:54.314 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:54.314 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:01:54.314 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:54.314 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-04-26T23:01:54.314 INFO:teuthology.orchestra.run.smithi040.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-26T23:01:54.314 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether be:f1:28:e1:99:21 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:54.315 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-26T23:01:54.315 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:54.315 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::4ce7:9bff:fe51:b1/64 scope link 2024-04-26T23:01:54.315 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:54.315 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:01:54.315 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether be:f1:28:e1:99:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T23:01:54.315 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::bcf1:28ff:fee1:9921/64 scope link 2024-04-26T23:01:54.315 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:54.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:54.316 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:01:54.396 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-04-26T23:01:54.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:54.401 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-04-26T23:01:54.453 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-04-26T23:01:54.453 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:01:54.453 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T23:01:54.454 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T23:01:54.454 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:01:54.454 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:54.454 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:54.454 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T23:01:54.454 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T23:01:54.454 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T23:01:54.454 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:54.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:54.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:54.674 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:01:54.721 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T23:01:54.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:54.723 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list-id 2024-04-26T23:01:54.822 INFO:teuthology.orchestra.run.smithi040.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-26T23:01:54.823 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:54.823 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:54.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-26T23:01:54.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-26T23:01:54.823 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:55.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:55.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:55.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-26T23:01:55.097 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:01:55.097 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:01:55.097 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:01:55.097 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:55.097 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-26T23:01:55.097 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:55.098 INFO:teuthology.orchestra.run.smithi040.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-26T23:01:55.098 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 4e:67:97:2a:43:ce brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-26T23:01:55.098 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-26T23:01:55.098 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:55.098 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::4c67:97ff:fe2a:43ce/64 scope link 2024-04-26T23:01:55.098 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:55.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:55.103 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-26T23:01:55.186 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:01:55.186 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:01:55.186 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:01:55.186 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:55.186 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-26T23:01:55.186 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:55.187 INFO:teuthology.orchestra.run.smithi040.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-26T23:01:55.187 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 4e:67:97:2a:43:ce brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-26T23:01:55.187 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-26T23:01:55.187 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:55.187 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::4c67:97ff:fe2a:43ce/64 scope link 2024-04-26T23:01:55.187 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:55.190 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-26T23:01:55.190 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:55.190 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:55.190 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-26T23:01:55.190 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-26T23:01:55.190 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-26T23:01:55.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-26T23:01:55.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-26T23:01:55.191 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:55.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:55.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:55.481 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:55.481 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:01:55.481 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.1 up 2024-04-26T23:01:55.481 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set dev brx.1 master ceph-brx 2024-04-26T23:01:55.481 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:01:55.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:55.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:55.694 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:01:55.694 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-26T23:01:55.736 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-26T23:01:55.737 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:01:55.737 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-26T23:01:55.833 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-26T23:01:55.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:55.833 DEBUG:teuthology.orchestra.run.smithi040:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-26T23:01:55.970 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-26T23:01:55.970 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-04-26T23:01:55.970 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:01:55.993 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-26T23:01:55.993 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-04-26T23:01:55.993 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:01:55.993 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:01:55.993 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:01:56.020 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-26T23:01:56.020 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-04-26T23:01:56.020 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:01:56.020 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:01:56.020 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:01:56.021 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-26T23:01:56.021 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-26T23:01:56.021 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=815936ab-d4ac-471e-b069-1a0dc70078ee" will pass to kernel 2024-04-26T23:01:56.024 INFO:teuthology.orchestra.run.smithi040.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-04-26T23:01:56.025 INFO:teuthology.orchestra.run.smithi040.stdout: You just mounted a file system that supports labels which does not 2024-04-26T23:01:56.025 INFO:teuthology.orchestra.run.smithi040.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T23:01:56.025 INFO:teuthology.orchestra.run.smithi040.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T23:01:56.025 INFO:teuthology.orchestra.run.smithi040.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T23:01:56.029 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T23:01:56.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:56.029 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-26T23:01:56.106 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:01:56.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:01:56.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:56.230 INFO:teuthology.orchestra.run.smithi040.stdout:client4684 2024-04-26T23:01:56.231 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:01:56.231 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:01:56.286 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:01:56.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:01:56.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:56.414 INFO:teuthology.orchestra.run.smithi040.stdout:client4684 2024-04-26T23:01:56.415 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:01:56.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:01:56.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:56.540 INFO:teuthology.orchestra.run.smithi040.stdout:client4684 2024-04-26T23:01:56.541 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/815936ab-d4ac-471e-b069-1a0dc70078ee.client4684/status') 2024-04-26T23:01:56.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:56.684 INFO:teuthology.orchestra.run.smithi040.stdout:instance: client.4684 (3)172.21.15.40:0/2892693232 2024-04-26T23:01:56.684 INFO:teuthology.orchestra.run.smithi040.stdout:blocklisted: false 2024-04-26T23:01:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:0+1 records in 2024-04-26T23:01:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:0+1 records out 2024-04-26T23:01:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:70 bytes copied, 8.3014e-05 s, 843 kB/s 2024-04-26T23:01:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:56.698 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-04-26T23:01:56.698 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T23:01:56.698 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-26T23:01:56.699 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-26T23:01:56.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.910+0000 7f1105400640 1 Processor -- start 2024-04-26T23:01:56.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.911+0000 7f1105400640 1 -- start start 2024-04-26T23:01:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.912+0000 7f1105400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f110015f380 0x7f11000cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.912+0000 7f1105400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1100073b70 0x7f11000cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.912+0000 7f1105400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1100072c00 con 0x7f11000d0620 2024-04-26T23:01:56.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.912+0000 7f1105400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f11000729b0 con 0x7f1100073b70 2024-04-26T23:01:56.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.912+0000 7f1105400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1100072ee0 con 0x7f110015f380 2024-04-26T23:01:56.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.912+0000 7f10ff200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f11000d0620 0x7f11000cede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51706/0 (socket says 172.21.15.40:51706) 2024-04-26T23:01:56.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.912+0000 7f10ff200640 1 -- 172.21.15.40:0/732986329 learned_addr learned my addr 172.21.15.40:0/732986329 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:01:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.913+0000 7f10fe800640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f110015f380 0x7f11000cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.913+0000 7f10f5e00640 1 --2- 172.21.15.40:0/732986329 >> v2:172.21.15.103:3300/0 conn(0x7f1100073b70 0x7f11000cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.913+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3193140247 0 0) 0x7f1100072c00 con 0x7f11000d0620 2024-04-26T23:01:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.913+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10e4003520 con 0x7f11000d0620 2024-04-26T23:01:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.913+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3076273850 0 0) 0x7f10e4003520 con 0x7f11000d0620 2024-04-26T23:01:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.913+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f110015f380 msgr2=0x7f11000cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.913+0000 7f10fde00640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f110015f380 0x7f11000cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.913+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 >> v2:172.21.15.103:3300/0 conn(0x7f1100073b70 msgr2=0x7f11000cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.913+0000 7f10fde00640 1 --2- 172.21.15.40:0/732986329 >> v2:172.21.15.103:3300/0 conn(0x7f1100073b70 0x7f11000cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.913+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11000731c0 con 0x7f11000d0620 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.913+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f10f80019f0 con 0x7f11000d0620 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.913+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 >> v1:172.21.15.40:6789/0 conn(0x7f11000d0620 legacy=0x7f11000cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10fde00640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e40039b0 0x7f10e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10fde00640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10e4004800 0x7f10e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10fde00640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f110015f380 0x7f10e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f11000729b0 con 0x7f10e40039b0 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1100072ee0 con 0x7f10e4004800 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f10e400b9c0 con 0x7f110015f380 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10fe800640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e40039b0 0x7f10e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10f5e00640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10e4004800 0x7f10e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10ff200640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f110015f380 0x7f10e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10fe800640 1 -- 172.21.15.40:0/732986329 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f110015f380 msgr2=0x7f10e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10fe800640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f110015f380 0x7f10e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10fe800640 1 -- 172.21.15.40:0/732986329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10e4004800 msgr2=0x7f10e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:01:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10fe800640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10e4004800 0x7f10e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10fe800640 1 -- 172.21.15.40:0/732986329 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e4003520 con 0x7f10e40039b0 2024-04-26T23:01:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10f5e00640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10e4004800 0x7f10e4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.914+0000 7f10ff200640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f110015f380 0x7f10e40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:01:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.915+0000 7f10fe800640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e40039b0 0x7f10e4003dd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009ac0 tx=0x7f10e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4d2748496b7ab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.916+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0036070 con 0x7f10e40039b0 2024-04-26T23:01:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.916+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f0031400 con 0x7f10e40039b0 2024-04-26T23:01:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.916+0000 7f10fde00640 1 -- 172.21.15.40:0/732986329 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0009690 con 0x7f10e40039b0 2024-04-26T23:01:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.916+0000 7f1105400640 1 -- 172.21.15.40:0/732986329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e40039b0 msgr2=0x7f10e4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.916+0000 7f1105400640 1 --2- 172.21.15.40:0/732986329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e40039b0 0x7f10e4003dd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009ac0 tx=0x7f10e4005830 comp rx=0 tx=0).stop 2024-04-26T23:01:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.916+0000 7f10ff200640 1 -- 172.21.15.40:0/732986329 reap_dead start 2024-04-26T23:01:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.916+0000 7f1105400640 1 -- 172.21.15.40:0/732986329 shutdown_connections 2024-04-26T23:01:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.916+0000 7f1105400640 1 -- 172.21.15.40:0/732986329 >> 172.21.15.40:0/732986329 conn(0x7f11000d0be0 msgr2=0x7f11000d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.916+0000 7f1105400640 1 -- 172.21.15.40:0/732986329 shutdown_connections 2024-04-26T23:01:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.917+0000 7f1105400640 1 -- 172.21.15.40:0/732986329 wait complete. 2024-04-26T23:01:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.918+0000 7f1105400640 1 Processor -- start 2024-04-26T23:01:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.919+0000 7f1105400640 1 -- start start 2024-04-26T23:01:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.919+0000 7f1105400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e40039b0 0x7f1100151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.919+0000 7f1105400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f110014ec10 0x7f11001f8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.920+0000 7f1105400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11000c6f80 0x7f11000c4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.920+0000 7f1105400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1100073620 con 0x7f10e40039b0 2024-04-26T23:01:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.920+0000 7f1105400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f11000aff10 con 0x7f110014ec10 2024-04-26T23:01:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.920+0000 7f1105400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f11000737a0 con 0x7f11000c6f80 2024-04-26T23:01:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.920+0000 7f10fe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e40039b0 0x7f1100151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.920+0000 7f10fe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e40039b0 0x7f1100151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34234/0 (socket says 172.21.15.40:34234) 2024-04-26T23:01:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.920+0000 7f10fe800640 1 -- 172.21.15.40:0/3720615088 learned_addr learned my addr 172.21.15.40:0/3720615088 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:01:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.920+0000 7f10f5e00640 1 --2- 172.21.15.40:0/3720615088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f110014ec10 0x7f11001f8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.920+0000 7f10fe800640 1 -- 172.21.15.40:0/3720615088 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11000c6f80 msgr2=0x7f11000c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.920+0000 7f10fe800640 1 --2- 172.21.15.40:0/3720615088 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11000c6f80 0x7f11000c4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.920+0000 7f10fe800640 1 -- 172.21.15.40:0/3720615088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f110014ec10 msgr2=0x7f11001f8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.920+0000 7f10fe800640 1 --2- 172.21.15.40:0/3720615088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f110014ec10 0x7f11001f8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.920+0000 7f10fe800640 1 -- 172.21.15.40:0/3720615088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110014f090 con 0x7f10e40039b0 2024-04-26T23:01:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.921+0000 7f10fe800640 1 --2- 172.21.15.40:0/3720615088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e40039b0 0x7f1100151fa0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009be0 tx=0x7f10f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.921+0000 7f10f7400640 1 -- 172.21.15.40:0/3720615088 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0040020 con 0x7f10e40039b0 2024-04-26T23:01:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.921+0000 7f10f7400640 1 -- 172.21.15.40:0/3720615088 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f0003bc0 con 0x7f10e40039b0 2024-04-26T23:01:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.921+0000 7f10f7400640 1 -- 172.21.15.40:0/3720615088 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0036040 con 0x7f10e40039b0 2024-04-26T23:01:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.921+0000 7f1105400640 1 -- 172.21.15.40:0/3720615088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11001f8e10 con 0x7f10e40039b0 2024-04-26T23:01:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.921+0000 7f1105400640 1 -- 172.21.15.40:0/3720615088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11001f93d0 con 0x7f10e40039b0 2024-04-26T23:01:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.922+0000 7f1105400640 1 -- 172.21.15.40:0/3720615088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11000694f0 con 0x7f10e40039b0 2024-04-26T23:01:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.923+0000 7f10f7400640 1 -- 172.21.15.40:0/3720615088 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f10f0043020 con 0x7f10e40039b0 2024-04-26T23:01:56.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.925+0000 7f10f7400640 1 --2- 172.21.15.40:0/3720615088 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f10dc06c920 0x7f10dc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:01:56.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.925+0000 7f10f7400640 1 -- 172.21.15.40:0/3720615088 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f10f00ac150 con 0x7f10e40039b0 2024-04-26T23:01:56.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.925+0000 7f10f5e00640 1 --2- 172.21.15.40:0/3720615088 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f10dc06c920 0x7f10dc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:01:56.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.926+0000 7f10f5e00640 1 --2- 172.21.15.40:0/3720615088 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f10dc06c920 0x7f10dc06ede0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f11000b23e0 tx=0x7f10e8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:01:56.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:56.930+0000 7f10f7400640 1 -- 172.21.15.40:0/3720615088 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f10f004a070 con 0x7f10e40039b0 2024-04-26T23:01:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.175+0000 7f1105400640 1 -- 172.21.15.40:0/3720615088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1100070180 con 0x7f10e40039b0 2024-04-26T23:01:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.176+0000 7f10f7400640 1 -- 172.21.15.40:0/3720615088 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f10f00044e0 con 0x7f10e40039b0 2024-04-26T23:01:57.177 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T23:01:57.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.179+0000 7f1105400640 1 -- 172.21.15.40:0/3720615088 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f10dc06c920 msgr2=0x7f10dc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:57.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.179+0000 7f1105400640 1 --2- 172.21.15.40:0/3720615088 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f10dc06c920 0x7f10dc06ede0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f11000b23e0 tx=0x7f10e8008040 comp rx=0 tx=0).stop 2024-04-26T23:01:57.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.179+0000 7f1105400640 1 -- 172.21.15.40:0/3720615088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e40039b0 msgr2=0x7f1100151fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:01:57.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.179+0000 7f1105400640 1 --2- 172.21.15.40:0/3720615088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e40039b0 0x7f1100151fa0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009be0 tx=0x7f10f0004730 comp rx=0 tx=0).stop 2024-04-26T23:01:57.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.179+0000 7f1105400640 1 -- 172.21.15.40:0/3720615088 shutdown_connections 2024-04-26T23:01:57.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.179+0000 7f1105400640 1 --2- 172.21.15.40:0/3720615088 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f10dc06c920 0x7f10dc06ede0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:57.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.179+0000 7f1105400640 1 --2- 172.21.15.40:0/3720615088 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11000c6f80 0x7f11000c4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:57.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.179+0000 7f1105400640 1 --2- 172.21.15.40:0/3720615088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f110014ec10 0x7f11001f8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:57.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.179+0000 7f1105400640 1 --2- 172.21.15.40:0/3720615088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e40039b0 0x7f1100151fa0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:01:57.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.179+0000 7f1105400640 1 -- 172.21.15.40:0/3720615088 >> 172.21.15.40:0/3720615088 conn(0x7f11000d0be0 msgr2=0x7f11000cd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:01:57.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.179+0000 7f1105400640 1 -- 172.21.15.40:0/3720615088 shutdown_connections 2024-04-26T23:01:57.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:01:57.179+0000 7f1105400640 1 -- 172.21.15.40:0/3720615088 wait complete. 2024-04-26T23:01:57.200 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T23:01:57.200 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:01:57.200 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-26T23:01:57.201 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-26T23:01:57.201 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-26T23:01:57.201 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T23:01:57.201 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T23:01:57.201 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:01:57.201 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-26T23:01:57.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:57.201 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-26T23:01:57.258 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:01:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:01:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:01:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:57.258 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42592sec preferred_lft 42592sec 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-26T23:01:57.259 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:01:57.260 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T23:01:57.260 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:57.261 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-26T23:01:57.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add name ceph-brx type bridge 2024-04-26T23:01:57.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr flush dev ceph-brx 2024-04-26T23:01:57.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set ceph-brx up 2024-04-26T23:01:57.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T23:01:57.261 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-26T23:01:57.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:01:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:57.754 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:01:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:57.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:57.764 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:01:57.831 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-26T23:01:57.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:57.835 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-26T23:01:57.893 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-26T23:01:57.893 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:01:57.893 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-26T23:01:57.893 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-26T23:01:57.894 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:01:57.895 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:57.895 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-26T23:01:57.895 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-26T23:01:57.895 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-26T23:01:57.895 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-26T23:01:57.895 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-26T23:01:57.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:01:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:58.682 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:01:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:58.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:58.693 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-26T23:01:58.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:58.743 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list-id 2024-04-26T23:01:58.789 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:58.789 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-26T23:01:58.789 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-26T23:01:58.789 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-26T23:01:58.790 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-26T23:01:58.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:01:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:58.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:01:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:58.976 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-26T23:01:58.977 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:58.977 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-26T23:01:58.977 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-26T23:01:58.977 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T23:01:58.977 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-26T23:01:58.977 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-26T23:01:58.977 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-26T23:01:58.977 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-26T23:01:59.079 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:01:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:59.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:01:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:59.265 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:01:59.265 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-26T23:01:59.265 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-04-26T23:01:59.265 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T23:01:59.265 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-26T23:01:59.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:01:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:01:59.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:01:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:01:59.462 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:01:59.462 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-26T23:01:59.503 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-26T23:01:59.503 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:01:59.503 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-26T23:01:59.597 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-26T23:01:59.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:01:59.598 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-26T23:01:59.727 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-26T23:01:59.728 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-26T23:01:59.728 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-04-26T23:02:00.426 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-26T23:02:00.426 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-26T23:02:00.427 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-04-26T23:02:00.427 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:02:00.427 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:02:00.462 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-26T23:02:00.463 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-26T23:02:00.463 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-04-26T23:02:00.463 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:02:00.463 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:02:00.463 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-26T23:02:00.463 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-26T23:02:00.463 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=815936ab-d4ac-471e-b069-1a0dc70078ee" will pass to kernel 2024-04-26T23:02:00.465 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-26T23:02:00.465 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-26T23:02:00.465 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T23:02:00.466 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T23:02:00.466 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T23:02:00.471 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T23:02:00.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:00.471 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-26T23:02:00.549 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:02:00.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:00 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:02:00.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:02:00.670 INFO:teuthology.orchestra.run.smithi103.stdout:client4690 2024-04-26T23:02:00.670 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:02:00.670 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:00.722 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:02:00.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:00 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:02:00.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:02:00.840 INFO:teuthology.orchestra.run.smithi103.stdout:client4690 2024-04-26T23:02:00.840 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:02:00.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:00 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:02:00.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:02:00.958 INFO:teuthology.orchestra.run.smithi103.stdout:client4690 2024-04-26T23:02:00.958 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/815936ab-d4ac-471e-b069-1a0dc70078ee.client4690/status') 2024-04-26T23:02:01.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:02:01.088 INFO:teuthology.orchestra.run.smithi103.stdout:instance: client.4690 (3)192.168.0.1:0/636514655 2024-04-26T23:02:01.088 INFO:teuthology.orchestra.run.smithi103.stdout:blocklisted: false 2024-04-26T23:02:01.088 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records in 2024-04-26T23:02:01.088 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records out 2024-04-26T23:02:01.088 INFO:teuthology.orchestra.run.smithi103.stderr:68 bytes copied, 7.9481e-05 s, 856 kB/s 2024-04-26T23:02:01.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:02:01.101 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-04-26T23:02:01.101 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T23:02:01.101 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-26T23:02:01.101 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-26T23:02:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.322+0000 7fa231000640 1 Processor -- start 2024-04-26T23:02:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.322+0000 7fa231000640 1 -- start start 2024-04-26T23:02:01.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.324+0000 7fa231000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa22c156460 0x7fa22c0cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:01.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.324+0000 7fa231000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa22c06fea0 0x7fa22c072c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:01.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.324+0000 7fa231000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa22c156840 con 0x7fa22c0cba10 2024-04-26T23:02:01.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.324+0000 7fa231000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa22c0cbf20 con 0x7fa22c06fea0 2024-04-26T23:02:01.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.324+0000 7fa231000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa22c0cc200 con 0x7fa22c156460 2024-04-26T23:02:01.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.324+0000 7fa22a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa22c06fea0 0x7fa22c072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:01.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.324+0000 7fa22a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa22c06fea0 0x7fa22c072c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34242/0 (socket says 172.21.15.40:34242) 2024-04-26T23:02:01.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.324+0000 7fa22a800640 1 -- 172.21.15.40:0/2975065278 learned_addr learned my addr 172.21.15.40:0/2975065278 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:01.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.325+0000 7fa229e00640 1 --2- 172.21.15.40:0/2975065278 >> v2:172.21.15.103:3300/0 conn(0x7fa22c156460 0x7fa22c0cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:01.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.325+0000 7fa22a800640 1 -- 172.21.15.40:0/2975065278 >> v2:172.21.15.103:3300/0 conn(0x7fa22c156460 msgr2=0x7fa22c0cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:01.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.325+0000 7fa22a800640 1 --2- 172.21.15.40:0/2975065278 >> v2:172.21.15.103:3300/0 conn(0x7fa22c156460 0x7fa22c0cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:01.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.325+0000 7fa22a800640 1 -- 172.21.15.40:0/2975065278 >> v1:172.21.15.103:6789/0 conn(0x7fa22c0cba10 legacy=0x7fa22c06f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:01.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.325+0000 7fa22a800640 1 -- 172.21.15.40:0/2975065278 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa22c0cc4e0 con 0x7fa22c06fea0 2024-04-26T23:02:01.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.325+0000 7fa229e00640 1 --2- 172.21.15.40:0/2975065278 >> v2:172.21.15.103:3300/0 conn(0x7fa22c156460 0x7fa22c0cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:01.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.326+0000 7fa22a800640 1 --2- 172.21.15.40:0/2975065278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa22c06fea0 0x7fa22c072c50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa22c0b05d0 tx=0x7fa22402c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0205704d5e3263d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:01.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.327+0000 7fa228a00640 1 -- 172.21.15.40:0/2975065278 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22400b2d0 con 0x7fa22c06fea0 2024-04-26T23:02:01.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.327+0000 7fa228a00640 1 -- 172.21.15.40:0/2975065278 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa22400b470 con 0x7fa22c06fea0 2024-04-26T23:02:01.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.327+0000 7fa228a00640 1 -- 172.21.15.40:0/2975065278 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa224009820 con 0x7fa22c06fea0 2024-04-26T23:02:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.327+0000 7fa231000640 1 -- 172.21.15.40:0/2975065278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa22c06fea0 msgr2=0x7fa22c072c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.327+0000 7fa231000640 1 --2- 172.21.15.40:0/2975065278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa22c06fea0 0x7fa22c072c50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa22c0b05d0 tx=0x7fa22402c8f0 comp rx=0 tx=0).stop 2024-04-26T23:02:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.327+0000 7fa231000640 1 -- 172.21.15.40:0/2975065278 shutdown_connections 2024-04-26T23:02:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.327+0000 7fa231000640 1 --2- 172.21.15.40:0/2975065278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa22c06fea0 0x7fa22c072c50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.327+0000 7fa231000640 1 --2- 172.21.15.40:0/2975065278 >> v2:172.21.15.103:3300/0 conn(0x7fa22c156460 0x7fa22c0cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.327+0000 7fa231000640 1 -- 172.21.15.40:0/2975065278 >> 172.21.15.40:0/2975065278 conn(0x7fa22c0d0be0 msgr2=0x7fa22c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.327+0000 7fa231000640 1 -- 172.21.15.40:0/2975065278 shutdown_connections 2024-04-26T23:02:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.328+0000 7fa231000640 1 -- 172.21.15.40:0/2975065278 wait complete. 2024-04-26T23:02:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.329+0000 7fa231000640 1 Processor -- start 2024-04-26T23:02:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.329+0000 7fa231000640 1 -- start start 2024-04-26T23:02:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.330+0000 7fa231000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa22c06fea0 0x7fa22c0c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.330+0000 7fa231000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa22c0cba10 0x7fa22c1f7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.330+0000 7fa231000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa22c156460 0x7fa22c15ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.330+0000 7fa231000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c06e260 con 0x7fa22c06fea0 2024-04-26T23:02:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.330+0000 7fa231000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c072700 con 0x7fa22c156460 2024-04-26T23:02:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.330+0000 7fa231000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa22c06e3e0 con 0x7fa22c0cba10 2024-04-26T23:02:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.331+0000 7fa229e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa22c06fea0 0x7fa22c0c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.331+0000 7fa229e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa22c06fea0 0x7fa22c0c59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34256/0 (socket says 172.21.15.40:34256) 2024-04-26T23:02:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.331+0000 7fa229e00640 1 -- 172.21.15.40:0/1958558367 learned_addr learned my addr 172.21.15.40:0/1958558367 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.331+0000 7fa22a800640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa22c156460 0x7fa22c15ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.331+0000 7fa229400640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa22c0cba10 0x7fa22c1f7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.331+0000 7fa229e00640 1 -- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa22c0cba10 msgr2=0x7fa22c1f7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.331+0000 7fa229e00640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa22c0cba10 0x7fa22c1f7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.331+0000 7fa229e00640 1 -- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa22c156460 msgr2=0x7fa22c15ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.331+0000 7fa229e00640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa22c156460 0x7fa22c15ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.331+0000 7fa229e00640 1 -- 172.21.15.40:0/1958558367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa22c1f82c0 con 0x7fa22c06fea0 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.331+0000 7fa229400640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa22c0cba10 0x7fa22c1f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.332+0000 7fa229e00640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa22c06fea0 0x7fa22c0c59c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00c820 tx=0x7fa21c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.332+0000 7fa222000640 1 -- 172.21.15.40:0/1958558367 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c015070 con 0x7fa22c06fea0 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.332+0000 7fa222000640 1 -- 172.21.15.40:0/1958558367 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa21c007a30 con 0x7fa22c06fea0 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.332+0000 7fa222000640 1 -- 172.21.15.40:0/1958558367 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c007de0 con 0x7fa22c06fea0 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.332+0000 7fa231000640 1 -- 172.21.15.40:0/1958558367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa22c0c4220 con 0x7fa22c06fea0 2024-04-26T23:02:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.332+0000 7fa231000640 1 -- 172.21.15.40:0/1958558367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa22c0c62b0 con 0x7fa22c06fea0 2024-04-26T23:02:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.333+0000 7fa231000640 1 -- 172.21.15.40:0/1958558367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa22c0694f0 con 0x7fa22c06fea0 2024-04-26T23:02:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.334+0000 7fa222000640 1 -- 172.21.15.40:0/1958558367 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa21c018070 con 0x7fa22c06fea0 2024-04-26T23:02:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.335+0000 7fa222000640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa20c06c8f0 0x7fa20c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.335+0000 7fa222000640 1 -- 172.21.15.40:0/1958558367 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa21c08be70 con 0x7fa22c06fea0 2024-04-26T23:02:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.336+0000 7fa229400640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa20c06c8f0 0x7fa20c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:01.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.337+0000 7fa229400640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa20c06c8f0 0x7fa20c06edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa22c0c5f00 tx=0x7fa218009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.340+0000 7fa222000640 1 -- 172.21.15.40:0/1958558367 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa22c1f82c0 con 0x7fa22c06fea0 2024-04-26T23:02:01.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.591+0000 7fa231000640 1 -- 172.21.15.40:0/1958558367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa22c06de80 con 0x7fa22c06fea0 2024-04-26T23:02:01.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.591+0000 7fa222000640 1 -- 172.21.15.40:0/1958558367 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7fa21c059e00 con 0x7fa22c06fea0 2024-04-26T23:02:01.592 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T23:02:01.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.594+0000 7fa231000640 1 -- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa20c06c8f0 msgr2=0x7fa20c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:01.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.594+0000 7fa231000640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa20c06c8f0 0x7fa20c06edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa22c0c5f00 tx=0x7fa218009040 comp rx=0 tx=0).stop 2024-04-26T23:02:01.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.594+0000 7fa231000640 1 -- 172.21.15.40:0/1958558367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa22c06fea0 msgr2=0x7fa22c0c59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:01.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.594+0000 7fa231000640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa22c06fea0 0x7fa22c0c59c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00c820 tx=0x7fa21c00cce0 comp rx=0 tx=0).stop 2024-04-26T23:02:01.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.595+0000 7fa231000640 1 -- 172.21.15.40:0/1958558367 shutdown_connections 2024-04-26T23:02:01.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.595+0000 7fa231000640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa20c06c8f0 0x7fa20c06edb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:01.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.595+0000 7fa231000640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa22c156460 0x7fa22c15ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:01.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.595+0000 7fa231000640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa22c0cba10 0x7fa22c1f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:01.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.595+0000 7fa231000640 1 --2- 172.21.15.40:0/1958558367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa22c06fea0 0x7fa22c0c59c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:01.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.595+0000 7fa231000640 1 -- 172.21.15.40:0/1958558367 >> 172.21.15.40:0/1958558367 conn(0x7fa22c0d0be0 msgr2=0x7fa22c0b1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:01.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.595+0000 7fa231000640 1 -- 172.21.15.40:0/1958558367 shutdown_connections 2024-04-26T23:02:01.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:01.595+0000 7fa231000640 1 -- 172.21.15.40:0/1958558367 wait complete. 2024-04-26T23:02:01.614 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T23:02:01.615 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:02:01.615 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-26T23:02:01.615 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-26T23:02:01.615 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-26T23:02:01.615 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T23:02:01.615 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T23:02:01.615 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:02:01.615 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-26T23:02:01.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:01.615 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-26T23:02:01.673 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:02:01.674 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:02:01.674 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:02:01.674 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:01.674 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-26T23:02:01.674 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:01.674 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:02:01.674 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:02:01.674 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-26T23:02:01.674 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:02:01.674 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:02:01.674 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42588sec preferred_lft 42588sec 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 12:34:77:b5:8c:9a brd ff:ff:ff:ff:ff:ff 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::60b7:faff:fe51:2658/64 scope link 2024-04-26T23:02:01.675 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:01.676 INFO:teuthology.orchestra.run.smithi103.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:02:01.676 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 12:34:77:b5:8c:9a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-26T23:02:01.676 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::1034:77ff:feb5:8c9a/64 scope link 2024-04-26T23:02:01.676 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:01.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:01.677 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:02:01.758 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-26T23:02:01.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:01.762 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-26T23:02:01.810 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-26T23:02:01.810 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:02:01.810 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-26T23:02:01.810 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-26T23:02:01.810 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:02:01.810 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:02:01.810 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-26T23:02:01.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-26T23:02:01.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-26T23:02:01.811 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-26T23:02:01.811 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-26T23:02:01.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:02:02.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:02:02.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:02.014 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-26T23:02:02.061 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-26T23:02:02.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:02.062 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list-id 2024-04-26T23:02:02.106 INFO:teuthology.orchestra.run.smithi103.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-26T23:02:02.107 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:02:02.107 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-26T23:02:02.107 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-26T23:02:02.107 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-26T23:02:02.107 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-26T23:02:02.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:02:02.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:02:02.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:02.340 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-26T23:02:02.417 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:02:02.418 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:02:02.418 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:02:02.418 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:02.418 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-26T23:02:02.418 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:02.418 INFO:teuthology.orchestra.run.smithi103.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-26T23:02:02.418 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 72:03:60:93:d8:60 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-26T23:02:02.418 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-26T23:02:02.418 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:02.418 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::7003:60ff:fe93:d860/64 scope link 2024-04-26T23:02:02.418 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:02.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:02.422 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-26T23:02:02.508 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:02:02.508 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:02:02.508 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:02:02.508 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:02.508 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-26T23:02:02.508 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:02.508 INFO:teuthology.orchestra.run.smithi103.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-26T23:02:02.509 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 72:03:60:93:d8:60 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-26T23:02:02.509 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-26T23:02:02.509 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:02.509 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::7003:60ff:fe93:d860/64 scope link 2024-04-26T23:02:02.509 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:02:02.513 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-26T23:02:02.513 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:02:02.513 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-26T23:02:02.513 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-26T23:02:02.513 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-26T23:02:02.514 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-26T23:02:02.514 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-26T23:02:02.514 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-26T23:02:02.514 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-26T23:02:02.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:02:02.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:02:02.785 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:02:02.786 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-26T23:02:02.786 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.1 up 2024-04-26T23:02:02.786 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set dev brx.1 master ceph-brx 2024-04-26T23:02:02.786 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-26T23:02:02.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:02:02.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:02:02.987 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:02:02.987 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-26T23:02:03.031 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-26T23:02:03.031 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:02:03.031 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-26T23:02:03.124 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-26T23:02:03.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:03.125 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-26T23:02:03.252 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-26T23:02:03.252 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-04-26T23:02:03.252 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-04-26T23:02:03.273 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-26T23:02:03.273 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-04-26T23:02:03.273 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-04-26T23:02:03.273 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:02:03.273 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:02:03.303 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-26T23:02:03.303 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-04-26T23:02:03.303 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-04-26T23:02:03.303 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:02:03.303 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:02:03.303 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-26T23:02:03.303 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-26T23:02:03.303 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=815936ab-d4ac-471e-b069-1a0dc70078ee" will pass to kernel 2024-04-26T23:02:03.304 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-04-26T23:02:03.305 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-26T23:02:03.305 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T23:02:03.305 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T23:02:03.305 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T23:02:03.310 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T23:02:03.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:03.310 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-26T23:02:03.384 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:02:03.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:02:03.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:02:03.500 INFO:teuthology.orchestra.run.smithi103.stdout:client4875 2024-04-26T23:02:03.501 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:02:03.501 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:03.551 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:02:03.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:02:03.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:02:03.711 INFO:teuthology.orchestra.run.smithi103.stdout:client4875 2024-04-26T23:02:03.711 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:02:03.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:02:03.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:02:03.828 INFO:teuthology.orchestra.run.smithi103.stdout:client4875 2024-04-26T23:02:03.828 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/815936ab-d4ac-471e-b069-1a0dc70078ee.client4875/status') 2024-04-26T23:02:03.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:02:03.965 INFO:teuthology.orchestra.run.smithi103.stdout:instance: client.4875 (3)172.21.15.103:0/3310872551 2024-04-26T23:02:03.966 INFO:teuthology.orchestra.run.smithi103.stdout:blocklisted: false 2024-04-26T23:02:03.966 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records in 2024-04-26T23:02:03.966 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records out 2024-04-26T23:02:03.966 INFO:teuthology.orchestra.run.smithi103.stderr:71 bytes copied, 9.2095e-05 s, 771 kB/s 2024-04-26T23:02:03.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:02:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:02:03.979 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-26T23:02:03.996 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:04.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.221+0000 7f85aea00640 1 Processor -- start 2024-04-26T23:02:04.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.221+0000 7f85aea00640 1 -- start start 2024-04-26T23:02:04.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.223+0000 7f85aea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f85a815ce60 0x7f85a80afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:04.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.223+0000 7f85aea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85a80af7a0 0x7f85a80ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:04.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.223+0000 7f85aea00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85a80d0820 con 0x7f85a8071fe0 2024-04-26T23:02:04.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.223+0000 7f85aea00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f85a80cc450 con 0x7f85a80af7a0 2024-04-26T23:02:04.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.223+0000 7f85aea00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f85a81614c0 con 0x7f85a815ce60 2024-04-26T23:02:04.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.224+0000 7f85acc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85a80af7a0 0x7f85a80ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:04.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.224+0000 7f85ae000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f85a8071fe0 0x7f85a8073760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51722/0 (socket says 172.21.15.40:51722) 2024-04-26T23:02:04.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.224+0000 7f85ae000640 1 -- 172.21.15.40:0/1207693196 learned_addr learned my addr 172.21.15.40:0/1207693196 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:04.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.224+0000 7f85ad600640 1 --2- 172.21.15.40:0/1207693196 >> v2:172.21.15.103:3300/0 conn(0x7f85a815ce60 0x7f85a80afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:04.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.224+0000 7f85acc00640 1 -- 172.21.15.40:0/1207693196 >> v2:172.21.15.103:3300/0 conn(0x7f85a815ce60 msgr2=0x7f85a80afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:04.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.224+0000 7f85acc00640 1 --2- 172.21.15.40:0/1207693196 >> v2:172.21.15.103:3300/0 conn(0x7f85a815ce60 0x7f85a80afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.224+0000 7f85acc00640 1 -- 172.21.15.40:0/1207693196 >> v1:172.21.15.40:6789/0 conn(0x7f85a8071fe0 legacy=0x7f85a8073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:04.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.224+0000 7f85acc00640 1 -- 172.21.15.40:0/1207693196 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a8161760 con 0x7f85a80af7a0 2024-04-26T23:02:04.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.224+0000 7f85a7e00640 1 -- 172.21.15.40:0/1207693196 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2114034499 0 0) 0x7f85a80d0820 con 0x7f85a8071fe0 2024-04-26T23:02:04.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.224+0000 7f85ad600640 1 --2- 172.21.15.40:0/1207693196 >> v2:172.21.15.103:3300/0 conn(0x7f85a815ce60 0x7f85a80afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:04.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.225+0000 7f85acc00640 1 --2- 172.21.15.40:0/1207693196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a80af7a0 0x7f85a80ceff0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f85a80741b0 tx=0x7f859402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6df3905b6ba7ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.226+0000 7f85a7e00640 1 -- 172.21.15.40:0/1207693196 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8594037070 con 0x7f85a80af7a0 2024-04-26T23:02:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.226+0000 7f85a7e00640 1 -- 172.21.15.40:0/1207693196 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8594032440 con 0x7f85a80af7a0 2024-04-26T23:02:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.226+0000 7f85a7e00640 1 -- 172.21.15.40:0/1207693196 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859402ae60 con 0x7f85a80af7a0 2024-04-26T23:02:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.226+0000 7f85aea00640 1 -- 172.21.15.40:0/1207693196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a80af7a0 msgr2=0x7f85a80ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.226+0000 7f85aea00640 1 --2- 172.21.15.40:0/1207693196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a80af7a0 0x7f85a80ceff0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f85a80741b0 tx=0x7f859402a110 comp rx=0 tx=0).stop 2024-04-26T23:02:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.226+0000 7f85aea00640 1 -- 172.21.15.40:0/1207693196 shutdown_connections 2024-04-26T23:02:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.226+0000 7f85aea00640 1 --2- 172.21.15.40:0/1207693196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a80af7a0 0x7f85a80ceff0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.226+0000 7f85aea00640 1 --2- 172.21.15.40:0/1207693196 >> v2:172.21.15.103:3300/0 conn(0x7f85a815ce60 0x7f85a80afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.226+0000 7f85aea00640 1 -- 172.21.15.40:0/1207693196 >> 172.21.15.40:0/1207693196 conn(0x7f85a80d0a10 msgr2=0x7f85a80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.226+0000 7f85aea00640 1 -- 172.21.15.40:0/1207693196 shutdown_connections 2024-04-26T23:02:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.226+0000 7f85aea00640 1 -- 172.21.15.40:0/1207693196 wait complete. 2024-04-26T23:02:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.228+0000 7f85aea00640 1 Processor -- start 2024-04-26T23:02:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.228+0000 7f85aea00640 1 -- start start 2024-04-26T23:02:04.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.229+0000 7f85aea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a8071fe0 0x7f85a815f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:04.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.229+0000 7f85aea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85a80af7a0 0x7f85a80c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:04.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.229+0000 7f85aea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a815ce60 0x7f85a814a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:04.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.229+0000 7f85aea00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f85a8073e70 con 0x7f85a8071fe0 2024-04-26T23:02:04.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.229+0000 7f85aea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f85a80cf530 con 0x7f85a815ce60 2024-04-26T23:02:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.229+0000 7f85aea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f85a8073ff0 con 0x7f85a80af7a0 2024-04-26T23:02:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.229+0000 7f85ad600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a8071fe0 0x7f85a815f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.230+0000 7f85ad600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a8071fe0 0x7f85a815f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34270/0 (socket says 172.21.15.40:34270) 2024-04-26T23:02:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.230+0000 7f85ad600640 1 -- 172.21.15.40:0/463109341 learned_addr learned my addr 172.21.15.40:0/463109341 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.230+0000 7f85acc00640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85a80af7a0 0x7f85a80c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.230+0000 7f85ae000640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a815ce60 0x7f85a814a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.230+0000 7f85ad600640 1 -- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85a80af7a0 msgr2=0x7f85a80c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.230+0000 7f85ad600640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85a80af7a0 0x7f85a80c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.230+0000 7f85ad600640 1 -- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a815ce60 msgr2=0x7f85a814a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.230+0000 7f85ad600640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a815ce60 0x7f85a814a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.230+0000 7f85ad600640 1 -- 172.21.15.40:0/463109341 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a80c84c0 con 0x7f85a8071fe0 2024-04-26T23:02:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.230+0000 7f85ad600640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a8071fe0 0x7f85a815f890 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f859c00c7f0 tx=0x7f859c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.231+0000 7f85a5600640 1 -- 172.21.15.40:0/463109341 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c015070 con 0x7f85a8071fe0 2024-04-26T23:02:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.231+0000 7f85a5600640 1 -- 172.21.15.40:0/463109341 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f859c007820 con 0x7f85a8071fe0 2024-04-26T23:02:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.231+0000 7f85a5600640 1 -- 172.21.15.40:0/463109341 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c007bd0 con 0x7f85a8071fe0 2024-04-26T23:02:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.231+0000 7f85aea00640 1 -- 172.21.15.40:0/463109341 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a80c87a0 con 0x7f85a8071fe0 2024-04-26T23:02:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.231+0000 7f85aea00640 1 -- 172.21.15.40:0/463109341 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a815d4f0 con 0x7f85a8071fe0 2024-04-26T23:02:04.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.232+0000 7f85aea00640 1 -- 172.21.15.40:0/463109341 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a80d0df0 con 0x7f85a8071fe0 2024-04-26T23:02:04.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.233+0000 7f85a5600640 1 -- 172.21.15.40:0/463109341 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f859c018070 con 0x7f85a8071fe0 2024-04-26T23:02:04.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.234+0000 7f85a5600640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f858806c8f0 0x7f858806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:04.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.234+0000 7f85a5600640 1 -- 172.21.15.40:0/463109341 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f859c08c0d0 con 0x7f85a8071fe0 2024-04-26T23:02:04.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.235+0000 7f85acc00640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f858806c8f0 0x7f858806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:04.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.236+0000 7f85acc00640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f858806c8f0 0x7f858806edb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8594034000 tx=0x7f859402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:04.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.238+0000 7f85a5600640 1 -- 172.21.15.40:0/463109341 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f85a80d0df0 con 0x7f85a8071fe0 2024-04-26T23:02:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.445+0000 7f85aea00640 1 -- 172.21.15.40:0/463109341 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85a8069370 con 0x7f85a8071fe0 2024-04-26T23:02:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.448+0000 7f85a5600640 1 -- 172.21.15.40:0/463109341 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18812 (secure 0 0 0) 0x7f859c05a0f0 con 0x7f85a8071fe0 2024-04-26T23:02:04.449 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:04.449 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:46.975683+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.451+0000 7f85aea00640 1 -- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f858806c8f0 msgr2=0x7f858806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.451+0000 7f85aea00640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f858806c8f0 0x7f858806edb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8594034000 tx=0x7f859402e000 comp rx=0 tx=0).stop 2024-04-26T23:02:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.451+0000 7f85aea00640 1 -- 172.21.15.40:0/463109341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a8071fe0 msgr2=0x7f85a815f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.451+0000 7f85aea00640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a8071fe0 0x7f85a815f890 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f859c00c7f0 tx=0x7f859c00ccb0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.451+0000 7f85aea00640 1 -- 172.21.15.40:0/463109341 shutdown_connections 2024-04-26T23:02:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.451+0000 7f85aea00640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f858806c8f0 0x7f858806edb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.451+0000 7f85aea00640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a815ce60 0x7f85a814a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.451+0000 7f85aea00640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85a80af7a0 0x7f85a80c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.451+0000 7f85aea00640 1 --2- 172.21.15.40:0/463109341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a8071fe0 0x7f85a815f890 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.451+0000 7f85aea00640 1 -- 172.21.15.40:0/463109341 >> 172.21.15.40:0/463109341 conn(0x7f85a80d0a10 msgr2=0x7f85a8160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.451+0000 7f85aea00640 1 -- 172.21.15.40:0/463109341 shutdown_connections 2024-04-26T23:02:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.452+0000 7f85aea00640 1 -- 172.21.15.40:0/463109341 wait complete. 2024-04-26T23:02:04.472 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:02:04.473 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:02:04.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.684+0000 7efe13e00640 1 Processor -- start 2024-04-26T23:02:04.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.684+0000 7efe13e00640 1 -- start start 2024-04-26T23:02:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.686+0000 7efe13e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe1415f060 0x7efe1415f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.686+0000 7efe13e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efe140d3640 0x7efe140d3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.686+0000 7efe13e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe140d1860 0x7efe140d1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.686+0000 7efe13e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7efe140d5c20 con 0x7efe1415f060 2024-04-26T23:02:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.686+0000 7efe13e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7efe140cfc40 con 0x7efe140d3640 2024-04-26T23:02:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.686+0000 7efe13e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efe140cff20 con 0x7efe140d1860 2024-04-26T23:02:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.686+0000 7efe12a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe1415f060 0x7efe1415f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.686+0000 7efe12a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe1415f060 0x7efe1415f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34276/0 (socket says 172.21.15.40:34276) 2024-04-26T23:02:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.686+0000 7efe12a00640 1 -- 172.21.15.40:0/2716537741 learned_addr learned my addr 172.21.15.40:0/2716537741 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.686+0000 7efe13400640 1 --2- 172.21.15.40:0/2716537741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe140d1860 0x7efe140d1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.686+0000 7efe12000640 1 --2- 172.21.15.40:0/2716537741 >> v2:172.21.15.103:3300/0 conn(0x7efe140d3640 0x7efe140d3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.686+0000 7efe12a00640 1 -- 172.21.15.40:0/2716537741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe140d1860 msgr2=0x7efe140d1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.686+0000 7efe12a00640 1 --2- 172.21.15.40:0/2716537741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe140d1860 0x7efe140d1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.687+0000 7efe12a00640 1 -- 172.21.15.40:0/2716537741 >> v2:172.21.15.103:3300/0 conn(0x7efe140d3640 msgr2=0x7efe140d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:04.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.687+0000 7efe12a00640 1 --2- 172.21.15.40:0/2716537741 >> v2:172.21.15.103:3300/0 conn(0x7efe140d3640 0x7efe140d3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.687+0000 7efe12a00640 1 -- 172.21.15.40:0/2716537741 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe140d0200 con 0x7efe1415f060 2024-04-26T23:02:04.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.688+0000 7efe12a00640 1 --2- 172.21.15.40:0/2716537741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe1415f060 0x7efe1415f440 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efe14162de0 tx=0x7efe0402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56288a622689854d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:04.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.688+0000 7efe11600640 1 -- 172.21.15.40:0/2716537741 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04037070 con 0x7efe1415f060 2024-04-26T23:02:04.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.688+0000 7efe11600640 1 -- 172.21.15.40:0/2716537741 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe04032440 con 0x7efe1415f060 2024-04-26T23:02:04.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.688+0000 7efe11600640 1 -- 172.21.15.40:0/2716537741 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04032810 con 0x7efe1415f060 2024-04-26T23:02:04.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.689+0000 7efe13e00640 1 -- 172.21.15.40:0/2716537741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe1415f060 msgr2=0x7efe1415f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:04.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.689+0000 7efe13e00640 1 --2- 172.21.15.40:0/2716537741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe1415f060 0x7efe1415f440 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efe14162de0 tx=0x7efe0402a1e0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.689+0000 7efe13e00640 1 -- 172.21.15.40:0/2716537741 shutdown_connections 2024-04-26T23:02:04.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.689+0000 7efe13e00640 1 --2- 172.21.15.40:0/2716537741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe140d1860 0x7efe140d1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.689+0000 7efe13e00640 1 --2- 172.21.15.40:0/2716537741 >> v2:172.21.15.103:3300/0 conn(0x7efe140d3640 0x7efe140d3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.689+0000 7efe13e00640 1 --2- 172.21.15.40:0/2716537741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe1415f060 0x7efe1415f440 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.689+0000 7efe13e00640 1 -- 172.21.15.40:0/2716537741 >> 172.21.15.40:0/2716537741 conn(0x7efe140d4e50 msgr2=0x7efe140d5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:04.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.689+0000 7efe13e00640 1 -- 172.21.15.40:0/2716537741 shutdown_connections 2024-04-26T23:02:04.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.689+0000 7efe13e00640 1 -- 172.21.15.40:0/2716537741 wait complete. 2024-04-26T23:02:04.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.691+0000 7efe13e00640 1 Processor -- start 2024-04-26T23:02:04.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.692+0000 7efe13e00640 1 -- start start 2024-04-26T23:02:04.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.693+0000 7efe13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe140d1860 0x7efe1409eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:04.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.693+0000 7efe13e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe140d3640 0x7efe14157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:04.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.693+0000 7efe13e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe1415f060 0x7efe140b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:04.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.693+0000 7efe13e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efe140d0ce0 con 0x7efe140d1860 2024-04-26T23:02:04.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.693+0000 7efe13e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efe140cab70 con 0x7efe1415f060 2024-04-26T23:02:04.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.693+0000 7efe13e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efe140b56b0 con 0x7efe140d3640 2024-04-26T23:02:04.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.693+0000 7efe12a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe140d1860 0x7efe1409eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:04.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.693+0000 7efe12a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe140d1860 0x7efe1409eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34282/0 (socket says 172.21.15.40:34282) 2024-04-26T23:02:04.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.693+0000 7efe12a00640 1 -- 172.21.15.40:0/3145630213 learned_addr learned my addr 172.21.15.40:0/3145630213 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:04.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.694+0000 7efe12a00640 1 -- 172.21.15.40:0/3145630213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe140d3640 msgr2=0x7efe14157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:04.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.694+0000 7efe12a00640 1 --2- 172.21.15.40:0/3145630213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe140d3640 0x7efe14157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.694+0000 7efe12a00640 1 -- 172.21.15.40:0/3145630213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe1415f060 msgr2=0x7efe140b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:04.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.694+0000 7efe12a00640 1 --2- 172.21.15.40:0/3145630213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe1415f060 0x7efe140b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.694+0000 7efe12a00640 1 -- 172.21.15.40:0/3145630213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe140b1bc0 con 0x7efe140d1860 2024-04-26T23:02:04.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.694+0000 7efe12a00640 1 --2- 172.21.15.40:0/3145630213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe140d1860 0x7efe1409eb30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efe0402a220 tx=0x7efe040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:04.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.694+0000 7efe0aa00640 1 -- 172.21.15.40:0/3145630213 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04037040 con 0x7efe140d1860 2024-04-26T23:02:04.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.694+0000 7efe0aa00640 1 -- 172.21.15.40:0/3145630213 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe04042070 con 0x7efe140d1860 2024-04-26T23:02:04.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.694+0000 7efe0aa00640 1 -- 172.21.15.40:0/3145630213 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04030bb0 con 0x7efe140d1860 2024-04-26T23:02:04.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.695+0000 7efe13e00640 1 -- 172.21.15.40:0/3145630213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe140b1ea0 con 0x7efe140d1860 2024-04-26T23:02:04.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.695+0000 7efe13e00640 1 -- 172.21.15.40:0/3145630213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe14157f90 con 0x7efe140d1860 2024-04-26T23:02:04.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.696+0000 7efe13e00640 1 -- 172.21.15.40:0/3145630213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe14069a80 con 0x7efe140d1860 2024-04-26T23:02:04.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.697+0000 7efe0aa00640 1 -- 172.21.15.40:0/3145630213 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efe04047460 con 0x7efe140d1860 2024-04-26T23:02:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.698+0000 7efe0aa00640 1 --2- 172.21.15.40:0/3145630213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efdf406c8f0 0x7efdf406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.698+0000 7efe0aa00640 1 -- 172.21.15.40:0/3145630213 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7efe040acb80 con 0x7efe140d1860 2024-04-26T23:02:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.699+0000 7efe12000640 1 --2- 172.21.15.40:0/3145630213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efdf406c8f0 0x7efdf406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:04.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.700+0000 7efe12000640 1 --2- 172.21.15.40:0/3145630213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efdf406c8f0 0x7efdf406edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efe00006fd0 tx=0x7efe00008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:04.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.702+0000 7efe0aa00640 1 -- 172.21.15.40:0/3145630213 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7efe04045360 con 0x7efe140d1860 2024-04-26T23:02:04.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.905+0000 7efe13e00640 1 -- 172.21.15.40:0/3145630213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efe14068e00 con 0x7efe140d1860 2024-04-26T23:02:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.906+0000 7efe0aa00640 1 -- 172.21.15.40:0/3145630213 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7efe0407ac40 con 0x7efe140d1860 2024-04-26T23:02:04.906 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:02:04.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.908+0000 7efe13e00640 1 -- 172.21.15.40:0/3145630213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efdf406c8f0 msgr2=0x7efdf406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:04.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.908+0000 7efe13e00640 1 --2- 172.21.15.40:0/3145630213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efdf406c8f0 0x7efdf406edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efe00006fd0 tx=0x7efe00008040 comp rx=0 tx=0).stop 2024-04-26T23:02:04.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.908+0000 7efe13e00640 1 -- 172.21.15.40:0/3145630213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe140d1860 msgr2=0x7efe1409eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:04.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.908+0000 7efe13e00640 1 --2- 172.21.15.40:0/3145630213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe140d1860 0x7efe1409eb30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efe0402a220 tx=0x7efe040046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.908+0000 7efe13e00640 1 -- 172.21.15.40:0/3145630213 shutdown_connections 2024-04-26T23:02:04.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.908+0000 7efe13e00640 1 --2- 172.21.15.40:0/3145630213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efdf406c8f0 0x7efdf406edb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.908+0000 7efe13e00640 1 --2- 172.21.15.40:0/3145630213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe1415f060 0x7efe140b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.908+0000 7efe13e00640 1 --2- 172.21.15.40:0/3145630213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe140d3640 0x7efe14157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.908+0000 7efe13e00640 1 --2- 172.21.15.40:0/3145630213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe140d1860 0x7efe1409eb30 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.908+0000 7efe13e00640 1 -- 172.21.15.40:0/3145630213 >> 172.21.15.40:0/3145630213 conn(0x7efe140d4e50 msgr2=0x7efe140d5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.908+0000 7efe13e00640 1 -- 172.21.15.40:0/3145630213 shutdown_connections 2024-04-26T23:02:04.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:04.909+0000 7efe13e00640 1 -- 172.21.15.40:0/3145630213 wait complete. 2024-04-26T23:02:04.927 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:02:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.122+0000 7fa5bcc00640 1 Processor -- start 2024-04-26T23:02:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.123+0000 7fa5bcc00640 1 -- start start 2024-04-26T23:02:05.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.124+0000 7fa5bcc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5b8073600 0x7fa5b8072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:05.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.124+0000 7fa5bcc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5b80739e0 0x7fa5b815f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:05.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.125+0000 7fa5bcc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5b81604e0 con 0x7fa5b815fcb0 2024-04-26T23:02:05.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.125+0000 7fa5bcc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa5b8073220 con 0x7fa5b80739e0 2024-04-26T23:02:05.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.125+0000 7fa5bcc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa5b8072850 con 0x7fa5b8073600 2024-04-26T23:02:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.125+0000 7fa5b6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5b80739e0 0x7fa5b815f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.125+0000 7fa5b7e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa5b815fcb0 0x7fa5b80ae8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51982/0 (socket says 172.21.15.40:51982) 2024-04-26T23:02:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.125+0000 7fa5b7e00640 1 -- 172.21.15.40:0/2208471739 learned_addr learned my addr 172.21.15.40:0/2208471739 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.125+0000 7fa5b7400640 1 --2- 172.21.15.40:0/2208471739 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5b8073600 0x7fa5b8072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.126+0000 7fa5b6a00640 1 -- 172.21.15.40:0/2208471739 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5b8073600 msgr2=0x7fa5b8072ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.126+0000 7fa5b6a00640 1 --2- 172.21.15.40:0/2208471739 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5b8073600 0x7fa5b8072ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.126+0000 7fa5b6a00640 1 -- 172.21.15.40:0/2208471739 >> v1:172.21.15.40:6789/0 conn(0x7fa5b815fcb0 legacy=0x7fa5b80ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.126+0000 7fa5b6a00640 1 -- 172.21.15.40:0/2208471739 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b8072200 con 0x7fa5b80739e0 2024-04-26T23:02:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.126+0000 7fa5b6000640 1 -- 172.21.15.40:0/2208471739 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3793312190 0 0) 0x7fa5b81604e0 con 0x7fa5b815fcb0 2024-04-26T23:02:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.127+0000 7fa5b6a00640 1 --2- 172.21.15.40:0/2208471739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5b80739e0 0x7fa5b815f770 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa5a40097a0 tx=0x7fa5a402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=370cf9080d2fd014 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.127+0000 7fa5b6000640 1 -- 172.21.15.40:0/2208471739 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4037070 con 0x7fa5b80739e0 2024-04-26T23:02:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.127+0000 7fa5b6000640 1 -- 172.21.15.40:0/2208471739 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5a4032440 con 0x7fa5b80739e0 2024-04-26T23:02:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.127+0000 7fa5b6000640 1 -- 172.21.15.40:0/2208471739 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4032810 con 0x7fa5b80739e0 2024-04-26T23:02:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.128+0000 7fa5bcc00640 1 -- 172.21.15.40:0/2208471739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5b80739e0 msgr2=0x7fa5b815f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.128+0000 7fa5bcc00640 1 --2- 172.21.15.40:0/2208471739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5b80739e0 0x7fa5b815f770 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa5a40097a0 tx=0x7fa5a402a180 comp rx=0 tx=0).stop 2024-04-26T23:02:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.128+0000 7fa5bcc00640 1 -- 172.21.15.40:0/2208471739 shutdown_connections 2024-04-26T23:02:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.128+0000 7fa5bcc00640 1 --2- 172.21.15.40:0/2208471739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5b80739e0 0x7fa5b815f770 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.128+0000 7fa5bcc00640 1 --2- 172.21.15.40:0/2208471739 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5b8073600 0x7fa5b8072ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.128+0000 7fa5bcc00640 1 -- 172.21.15.40:0/2208471739 >> 172.21.15.40:0/2208471739 conn(0x7fa5b80b2720 msgr2=0x7fa5b80b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.128+0000 7fa5bcc00640 1 -- 172.21.15.40:0/2208471739 shutdown_connections 2024-04-26T23:02:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.128+0000 7fa5bcc00640 1 -- 172.21.15.40:0/2208471739 wait complete. 2024-04-26T23:02:05.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.130+0000 7fa5bcc00640 1 Processor -- start 2024-04-26T23:02:05.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.130+0000 7fa5bcc00640 1 -- start start 2024-04-26T23:02:05.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.131+0000 7fa5bcc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5b8073600 0x7fa5b806f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:05.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.131+0000 7fa5bcc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5b80739e0 0x7fa5b815af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:05.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.131+0000 7fa5bcc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5b815fcb0 0x7fa5b8158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:05.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.131+0000 7fa5bcc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b8071e20 con 0x7fa5b815fcb0 2024-04-26T23:02:05.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.131+0000 7fa5bcc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b8073e60 con 0x7fa5b8073600 2024-04-26T23:02:05.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.131+0000 7fa5bcc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa5b8071fa0 con 0x7fa5b80739e0 2024-04-26T23:02:05.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.131+0000 7fa5b7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5b815fcb0 0x7fa5b8158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.131+0000 7fa5b7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5b815fcb0 0x7fa5b8158e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36164/0 (socket says 172.21.15.40:36164) 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.131+0000 7fa5b7e00640 1 -- 172.21.15.40:0/4213807118 learned_addr learned my addr 172.21.15.40:0/4213807118 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.132+0000 7fa5b6a00640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5b80739e0 0x7fa5b815af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.132+0000 7fa5b7400640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5b8073600 0x7fa5b806f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.132+0000 7fa5b7e00640 1 -- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5b80739e0 msgr2=0x7fa5b815af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.132+0000 7fa5b7e00640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5b80739e0 0x7fa5b815af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.132+0000 7fa5b7e00640 1 -- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5b8073600 msgr2=0x7fa5b806f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.132+0000 7fa5b7e00640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5b8073600 0x7fa5b806f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.132+0000 7fa5b7e00640 1 -- 172.21.15.40:0/4213807118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b8162390 con 0x7fa5b815fcb0 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.132+0000 7fa5b6a00640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5b80739e0 0x7fa5b815af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.132+0000 7fa5b7e00640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5b815fcb0 0x7fa5b8158e50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000c700 tx=0x7fa5b000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.132+0000 7fa5af400640 1 -- 172.21.15.40:0/4213807118 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0018070 con 0x7fa5b815fcb0 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.132+0000 7fa5af400640 1 -- 172.21.15.40:0/4213807118 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b000ce10 con 0x7fa5b815fcb0 2024-04-26T23:02:05.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.132+0000 7fa5af400640 1 -- 172.21.15.40:0/4213807118 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b000a860 con 0x7fa5b815fcb0 2024-04-26T23:02:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.132+0000 7fa5bcc00640 1 -- 172.21.15.40:0/4213807118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b8162670 con 0x7fa5b815fcb0 2024-04-26T23:02:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.133+0000 7fa5bcc00640 1 -- 172.21.15.40:0/4213807118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b815b470 con 0x7fa5b815fcb0 2024-04-26T23:02:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.134+0000 7fa5bcc00640 1 -- 172.21.15.40:0/4213807118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b8163130 con 0x7fa5b815fcb0 2024-04-26T23:02:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.135+0000 7fa5af400640 1 -- 172.21.15.40:0/4213807118 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa5b000aa00 con 0x7fa5b815fcb0 2024-04-26T23:02:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.136+0000 7fa5af400640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa59806c8f0 0x7fa59806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.136+0000 7fa5af400640 1 -- 172.21.15.40:0/4213807118 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa5b008b560 con 0x7fa5b815fcb0 2024-04-26T23:02:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.136+0000 7fa5b7400640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa59806c8f0 0x7fa59806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.138+0000 7fa5b7400640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa59806c8f0 0x7fa59806edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8006fd0 tx=0x7fa5a8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.141+0000 7fa5af400640 1 -- 172.21.15.40:0/4213807118 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa5b8163130 con 0x7fa5b815fcb0 2024-04-26T23:02:05.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.346+0000 7fa5bcc00640 1 -- 172.21.15.40:0/4213807118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa5b81f6360 con 0x7fa5b815fcb0 2024-04-26T23:02:05.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.347+0000 7fa5af400640 1 -- 172.21.15.40:0/4213807118 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fa5b0059580 con 0x7fa5b815fcb0 2024-04-26T23:02:05.348 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.350+0000 7fa5bcc00640 1 -- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa59806c8f0 msgr2=0x7fa59806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.350+0000 7fa5bcc00640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa59806c8f0 0x7fa59806edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8006fd0 tx=0x7fa5a8008040 comp rx=0 tx=0).stop 2024-04-26T23:02:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.350+0000 7fa5bcc00640 1 -- 172.21.15.40:0/4213807118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5b815fcb0 msgr2=0x7fa5b8158e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.350+0000 7fa5bcc00640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5b815fcb0 0x7fa5b8158e50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000c700 tx=0x7fa5b000cbc0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.350+0000 7fa5bcc00640 1 -- 172.21.15.40:0/4213807118 shutdown_connections 2024-04-26T23:02:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.350+0000 7fa5bcc00640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa59806c8f0 0x7fa59806edb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.350+0000 7fa5bcc00640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5b815fcb0 0x7fa5b8158e50 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.350+0000 7fa5bcc00640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5b80739e0 0x7fa5b815af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.350+0000 7fa5bcc00640 1 --2- 172.21.15.40:0/4213807118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5b8073600 0x7fa5b806f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.350+0000 7fa5bcc00640 1 -- 172.21.15.40:0/4213807118 >> 172.21.15.40:0/4213807118 conn(0x7fa5b80b2720 msgr2=0x7fa5b815d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.350+0000 7fa5bcc00640 1 -- 172.21.15.40:0/4213807118 shutdown_connections 2024-04-26T23:02:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.351+0000 7fa5bcc00640 1 -- 172.21.15.40:0/4213807118 wait complete. 2024-04-26T23:02:05.370 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:02:05.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.573+0000 7fac5b200640 1 Processor -- start 2024-04-26T23:02:05.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.574+0000 7fac5b200640 1 -- start start 2024-04-26T23:02:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.575+0000 7fac5b200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fac5c071fe0 0x7fac5c0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.575+0000 7fac5b200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac5c0d0820 con 0x7fac5c15ce60 2024-04-26T23:02:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.575+0000 7fac5b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac5c1614c0 con 0x7fac5c0af7a0 2024-04-26T23:02:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.575+0000 7fac5b200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fac5c0cc450 con 0x7fac5c071fe0 2024-04-26T23:02:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.576+0000 7fac5a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fac5c071fe0 0x7fac5c0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.576+0000 7fac59e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fac5c15ce60 0x7fac5c0ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51994/0 (socket says 172.21.15.40:51994) 2024-04-26T23:02:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.576+0000 7fac59e00640 1 -- 172.21.15.40:0/2251303831 learned_addr learned my addr 172.21.15.40:0/2251303831 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:05.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.576+0000 7fac5a800640 1 -- 172.21.15.40:0/2251303831 >> v1:172.21.15.103:6789/0 conn(0x7fac5c0af7a0 legacy=0x7fac5c073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:05.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.576+0000 7fac5a800640 1 -- 172.21.15.40:0/2251303831 >> v1:172.21.15.40:6789/0 conn(0x7fac5c15ce60 legacy=0x7fac5c0ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:05.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.576+0000 7fac5a800640 1 -- 172.21.15.40:0/2251303831 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5c161760 con 0x7fac5c071fe0 2024-04-26T23:02:05.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.578+0000 7fac5a800640 1 --2- 172.21.15.40:0/2251303831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac5c071fe0 0x7fac5c0afc60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fac54009d90 tx=0x7fac5402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f585d9bf7721bdd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:05.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.578+0000 7fac59400640 1 -- 172.21.15.40:0/2251303831 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac54007740 con 0x7fac5c071fe0 2024-04-26T23:02:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.578+0000 7fac59400640 1 -- 172.21.15.40:0/2251303831 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac540078e0 con 0x7fac5c071fe0 2024-04-26T23:02:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.578+0000 7fac59400640 1 -- 172.21.15.40:0/2251303831 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac54007cb0 con 0x7fac5c071fe0 2024-04-26T23:02:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.578+0000 7fac5b200640 1 -- 172.21.15.40:0/2251303831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac5c071fe0 msgr2=0x7fac5c0afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.578+0000 7fac5b200640 1 --2- 172.21.15.40:0/2251303831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac5c071fe0 0x7fac5c0afc60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fac54009d90 tx=0x7fac5402a660 comp rx=0 tx=0).stop 2024-04-26T23:02:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.578+0000 7fac5b200640 1 -- 172.21.15.40:0/2251303831 shutdown_connections 2024-04-26T23:02:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.578+0000 7fac5b200640 1 --2- 172.21.15.40:0/2251303831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac5c071fe0 0x7fac5c0afc60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.578+0000 7fac5b200640 1 -- 172.21.15.40:0/2251303831 >> 172.21.15.40:0/2251303831 conn(0x7fac5c0d0a10 msgr2=0x7fac5c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.579+0000 7fac5b200640 1 -- 172.21.15.40:0/2251303831 shutdown_connections 2024-04-26T23:02:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.579+0000 7fac5b200640 1 -- 172.21.15.40:0/2251303831 wait complete. 2024-04-26T23:02:05.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.580+0000 7fac5b200640 1 Processor -- start 2024-04-26T23:02:05.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.580+0000 7fac5b200640 1 -- start start 2024-04-26T23:02:05.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.581+0000 7fac5b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac5c071fe0 0x7fac5c15fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:05.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.581+0000 7fac5b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac5c0af7a0 0x7fac5c0c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:05.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.581+0000 7fac5b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac5c15ce60 0x7fac5c0ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:05.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.581+0000 7fac5b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c0cc1f0 con 0x7fac5c071fe0 2024-04-26T23:02:05.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.581+0000 7fac5b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c0cc070 con 0x7fac5c15ce60 2024-04-26T23:02:05.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.581+0000 7fac5b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fac5c073e70 con 0x7fac5c0af7a0 2024-04-26T23:02:05.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.582+0000 7fac59e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac5c071fe0 0x7fac5c15fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:05.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.582+0000 7fac59e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac5c071fe0 0x7fac5c15fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36192/0 (socket says 172.21.15.40:36192) 2024-04-26T23:02:05.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.582+0000 7fac59e00640 1 -- 172.21.15.40:0/151305243 learned_addr learned my addr 172.21.15.40:0/151305243 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.582+0000 7fac53e00640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac5c0af7a0 0x7fac5c0c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.582+0000 7fac59e00640 1 -- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac5c0af7a0 msgr2=0x7fac5c0c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.582+0000 7fac5a800640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac5c15ce60 0x7fac5c0ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.582+0000 7fac59e00640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac5c0af7a0 0x7fac5c0c8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.582+0000 7fac59e00640 1 -- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac5c15ce60 msgr2=0x7fac5c0ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.582+0000 7fac59e00640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac5c15ce60 0x7fac5c0ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.582+0000 7fac59e00640 1 -- 172.21.15.40:0/151305243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5c0c7cc0 con 0x7fac5c071fe0 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.582+0000 7fac53e00640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac5c0af7a0 0x7fac5c0c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.582+0000 7fac59e00640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac5c071fe0 0x7fac5c15fd50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00c480 tx=0x7fac4c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.583+0000 7fac52000640 1 -- 172.21.15.40:0/151305243 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c017070 con 0x7fac5c071fe0 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.583+0000 7fac52000640 1 -- 172.21.15.40:0/151305243 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac4c005ce0 con 0x7fac5c071fe0 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.583+0000 7fac5b200640 1 -- 172.21.15.40:0/151305243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5c0c7fa0 con 0x7fac5c071fe0 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.583+0000 7fac52000640 1 -- 172.21.15.40:0/151305243 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c0072f0 con 0x7fac5c071fe0 2024-04-26T23:02:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.583+0000 7fac5b200640 1 -- 172.21.15.40:0/151305243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5c0c8fc0 con 0x7fac5c071fe0 2024-04-26T23:02:05.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.584+0000 7fac5b200640 1 -- 172.21.15.40:0/151305243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac5c069370 con 0x7fac5c071fe0 2024-04-26T23:02:05.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.585+0000 7fac52000640 1 -- 172.21.15.40:0/151305243 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fac4c01a070 con 0x7fac5c071fe0 2024-04-26T23:02:05.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.586+0000 7fac52000640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fac3806c920 0x7fac3806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:05.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.586+0000 7fac52000640 1 -- 172.21.15.40:0/151305243 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fac4c08b210 con 0x7fac5c071fe0 2024-04-26T23:02:05.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.587+0000 7fac53e00640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fac3806c920 0x7fac3806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:05.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.588+0000 7fac53e00640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fac3806c920 0x7fac3806ede0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fac5c073230 tx=0x7fac44009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:05.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.591+0000 7fac52000640 1 -- 172.21.15.40:0/151305243 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fac4c058be0 con 0x7fac5c071fe0 2024-04-26T23:02:05.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.785+0000 7fac5b200640 1 -- 172.21.15.40:0/151305243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fac5c0b6980 con 0x7fac5c071fe0 2024-04-26T23:02:05.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.786+0000 7fac52000640 1 -- 172.21.15.40:0/151305243 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fac4c05cee0 con 0x7fac5c071fe0 2024-04-26T23:02:05.786 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:02:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.788+0000 7fac5b200640 1 -- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fac3806c920 msgr2=0x7fac3806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.788+0000 7fac5b200640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fac3806c920 0x7fac3806ede0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fac5c073230 tx=0x7fac44009040 comp rx=0 tx=0).stop 2024-04-26T23:02:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.788+0000 7fac5b200640 1 -- 172.21.15.40:0/151305243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac5c071fe0 msgr2=0x7fac5c15fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.788+0000 7fac5b200640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac5c071fe0 0x7fac5c15fd50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00c480 tx=0x7fac4c00c940 comp rx=0 tx=0).stop 2024-04-26T23:02:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.789+0000 7fac5b200640 1 -- 172.21.15.40:0/151305243 shutdown_connections 2024-04-26T23:02:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.789+0000 7fac5b200640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fac3806c920 0x7fac3806ede0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.789+0000 7fac5b200640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac5c15ce60 0x7fac5c0ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.789+0000 7fac5b200640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac5c0af7a0 0x7fac5c0c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.789+0000 7fac5b200640 1 --2- 172.21.15.40:0/151305243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac5c071fe0 0x7fac5c15fd50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.789+0000 7fac5b200640 1 -- 172.21.15.40:0/151305243 >> 172.21.15.40:0/151305243 conn(0x7fac5c0d0a10 msgr2=0x7fac5c160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.789+0000 7fac5b200640 1 -- 172.21.15.40:0/151305243 shutdown_connections 2024-04-26T23:02:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:05.789+0000 7fac5b200640 1 -- 172.21.15.40:0/151305243 wait complete. 2024-04-26T23:02:05.807 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:02:06.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.011+0000 7fbe95000640 1 Processor -- start 2024-04-26T23:02:06.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.011+0000 7fbe95000640 1 -- start start 2024-04-26T23:02:06.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.013+0000 7fbe95000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbe900cc480 0x7fbe90072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.013+0000 7fbe95000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe90072780 0x7fbe90166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.013+0000 7fbe95000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe9015d7c0 con 0x7fbe90071e60 2024-04-26T23:02:06.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.013+0000 7fbe95000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbe9015daa0 con 0x7fbe900cc480 2024-04-26T23:02:06.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.013+0000 7fbe95000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbe9015dd80 con 0x7fbe90072780 2024-04-26T23:02:06.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.014+0000 7fbe85400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbe900cc480 0x7fbe90072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.014+0000 7fbe8e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe90072780 0x7fbe90166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.014+0000 7fbe8e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe90072780 0x7fbe90166ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:44780/0 (socket says 172.21.15.40:44780) 2024-04-26T23:02:06.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.014+0000 7fbe8e800640 1 -- 172.21.15.40:0/1518957529 learned_addr learned my addr 172.21.15.40:0/1518957529 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:06.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.014+0000 7fbe8e800640 1 -- 172.21.15.40:0/1518957529 >> v2:172.21.15.103:3300/0 conn(0x7fbe900cc480 msgr2=0x7fbe90072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:06.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.014+0000 7fbe8e800640 1 --2- 172.21.15.40:0/1518957529 >> v2:172.21.15.103:3300/0 conn(0x7fbe900cc480 0x7fbe90072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.014+0000 7fbe8e800640 1 -- 172.21.15.40:0/1518957529 >> v1:172.21.15.103:6789/0 conn(0x7fbe90071e60 legacy=0x7fbe9006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.014+0000 7fbe8e800640 1 -- 172.21.15.40:0/1518957529 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9015e060 con 0x7fbe90072780 2024-04-26T23:02:06.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.016+0000 7fbe8e800640 1 --2- 172.21.15.40:0/1518957529 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe90072780 0x7fbe90166ad0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009e50 tx=0x7fbe8802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c9815bdb716f251 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:06.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.017+0000 7fbe8d400640 1 -- 172.21.15.40:0/1518957529 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88036070 con 0x7fbe90072780 2024-04-26T23:02:06.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.017+0000 7fbe8d400640 1 -- 172.21.15.40:0/1518957529 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe88031600 con 0x7fbe90072780 2024-04-26T23:02:06.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.017+0000 7fbe8d400640 1 -- 172.21.15.40:0/1518957529 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe880319d0 con 0x7fbe90072780 2024-04-26T23:02:06.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.017+0000 7fbe95000640 1 -- 172.21.15.40:0/1518957529 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe90072780 msgr2=0x7fbe90166ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.017+0000 7fbe95000640 1 --2- 172.21.15.40:0/1518957529 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe90072780 0x7fbe90166ad0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009e50 tx=0x7fbe8802a710 comp rx=0 tx=0).stop 2024-04-26T23:02:06.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.017+0000 7fbe95000640 1 -- 172.21.15.40:0/1518957529 shutdown_connections 2024-04-26T23:02:06.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.017+0000 7fbe95000640 1 --2- 172.21.15.40:0/1518957529 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe90072780 0x7fbe90166ad0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.017+0000 7fbe95000640 1 --2- 172.21.15.40:0/1518957529 >> v2:172.21.15.103:3300/0 conn(0x7fbe900cc480 0x7fbe90072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.017+0000 7fbe95000640 1 -- 172.21.15.40:0/1518957529 >> 172.21.15.40:0/1518957529 conn(0x7fbe900d0be0 msgr2=0x7fbe900d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:06.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.018+0000 7fbe95000640 1 -- 172.21.15.40:0/1518957529 shutdown_connections 2024-04-26T23:02:06.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.018+0000 7fbe95000640 1 -- 172.21.15.40:0/1518957529 wait complete. 2024-04-26T23:02:06.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.020+0000 7fbe95000640 1 Processor -- start 2024-04-26T23:02:06.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.020+0000 7fbe95000640 1 -- start start 2024-04-26T23:02:06.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.021+0000 7fbe95000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe90071e60 0x7fbe9015a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.021+0000 7fbe95000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe90072780 0x7fbe900c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.021+0000 7fbe95000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe900cc480 0x7fbe90150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe95000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9015d3f0 con 0x7fbe90072780 2024-04-26T23:02:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe95000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbe90072c10 con 0x7fbe90071e60 2024-04-26T23:02:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe95000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbe9015d570 con 0x7fbe900cc480 2024-04-26T23:02:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe85400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe90072780 0x7fbe900c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe85400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe90072780 0x7fbe900c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36194/0 (socket says 172.21.15.40:36194) 2024-04-26T23:02:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe85400640 1 -- 172.21.15.40:0/3019813937 learned_addr learned my addr 172.21.15.40:0/3019813937 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe8de00640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe90071e60 0x7fbe9015a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe8e800640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe900cc480 0x7fbe90150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe85400640 1 -- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe900cc480 msgr2=0x7fbe90150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe85400640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe900cc480 0x7fbe90150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe85400640 1 -- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe90071e60 msgr2=0x7fbe9015a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe85400640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe90071e60 0x7fbe9015a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe85400640 1 -- 172.21.15.40:0/3019813937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe900b6590 con 0x7fbe90072780 2024-04-26T23:02:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.022+0000 7fbe8e800640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe900cc480 0x7fbe90150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:06.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.023+0000 7fbe85400640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe90072780 0x7fbe900c8d00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800e870 tx=0x7fbe7800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:06.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.023+0000 7fbe86a00640 1 -- 172.21.15.40:0/3019813937 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78017070 con 0x7fbe90072780 2024-04-26T23:02:06.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.023+0000 7fbe86a00640 1 -- 172.21.15.40:0/3019813937 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe7800c8e0 con 0x7fbe90072780 2024-04-26T23:02:06.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.023+0000 7fbe86a00640 1 -- 172.21.15.40:0/3019813937 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7800ccd0 con 0x7fbe90072780 2024-04-26T23:02:06.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.023+0000 7fbe95000640 1 -- 172.21.15.40:0/3019813937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe900c9450 con 0x7fbe90072780 2024-04-26T23:02:06.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.023+0000 7fbe95000640 1 -- 172.21.15.40:0/3019813937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe900c9a10 con 0x7fbe90072780 2024-04-26T23:02:06.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.024+0000 7fbe95000640 1 -- 172.21.15.40:0/3019813937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe900694f0 con 0x7fbe90072780 2024-04-26T23:02:06.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.025+0000 7fbe86a00640 1 -- 172.21.15.40:0/3019813937 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbe7801a070 con 0x7fbe90072780 2024-04-26T23:02:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.026+0000 7fbe86a00640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbe6c06c8f0 0x7fbe6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.026+0000 7fbe86a00640 1 -- 172.21.15.40:0/3019813937 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbe7808bd70 con 0x7fbe90072780 2024-04-26T23:02:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.027+0000 7fbe8de00640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbe6c06c8f0 0x7fbe6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.028+0000 7fbe8de00640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbe6c06c8f0 0x7fbe6c06edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbe80001640 tx=0x7fbe80009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:06.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.031+0000 7fbe86a00640 1 -- 172.21.15.40:0/3019813937 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fbe78090070 con 0x7fbe90072780 2024-04-26T23:02:06.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.236+0000 7fbe95000640 1 -- 172.21.15.40:0/3019813937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbe900b3b30 con 0x7fbe90072780 2024-04-26T23:02:06.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.237+0000 7fbe86a00640 1 -- 172.21.15.40:0/3019813937 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fbe78059dd0 con 0x7fbe90072780 2024-04-26T23:02:06.238 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:06.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.240+0000 7fbe95000640 1 -- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbe6c06c8f0 msgr2=0x7fbe6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.240+0000 7fbe95000640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbe6c06c8f0 0x7fbe6c06edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbe80001640 tx=0x7fbe80009040 comp rx=0 tx=0).stop 2024-04-26T23:02:06.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.240+0000 7fbe95000640 1 -- 172.21.15.40:0/3019813937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe90072780 msgr2=0x7fbe900c8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.241+0000 7fbe95000640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe90072780 0x7fbe900c8d00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800e870 tx=0x7fbe7800ed30 comp rx=0 tx=0).stop 2024-04-26T23:02:06.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.241+0000 7fbe95000640 1 -- 172.21.15.40:0/3019813937 shutdown_connections 2024-04-26T23:02:06.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.241+0000 7fbe95000640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbe6c06c8f0 0x7fbe6c06edb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.241+0000 7fbe95000640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbe900cc480 0x7fbe90150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.241+0000 7fbe95000640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe90072780 0x7fbe900c8d00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.241+0000 7fbe95000640 1 --2- 172.21.15.40:0/3019813937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe90071e60 0x7fbe9015a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.241+0000 7fbe95000640 1 -- 172.21.15.40:0/3019813937 >> 172.21.15.40:0/3019813937 conn(0x7fbe900d0be0 msgr2=0x7fbe900d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:06.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.241+0000 7fbe95000640 1 -- 172.21.15.40:0/3019813937 shutdown_connections 2024-04-26T23:02:06.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.241+0000 7fbe95000640 1 -- 172.21.15.40:0/3019813937 wait complete. 2024-04-26T23:02:06.259 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:06.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.470+0000 7fa886400640 1 Processor -- start 2024-04-26T23:02:06.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.470+0000 7fa886400640 1 -- start start 2024-04-26T23:02:06.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.471+0000 7fa886400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa880071e60 0x7fa88006f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.471+0000 7fa886400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa88015d7c0 con 0x7fa88006fe20 2024-04-26T23:02:06.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.471+0000 7fa886400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa88015daa0 con 0x7fa8800cc480 2024-04-26T23:02:06.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.471+0000 7fa886400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa88015dd80 con 0x7fa880071e60 2024-04-26T23:02:06.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.472+0000 7fa87f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa880071e60 0x7fa88006f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.472+0000 7fa87ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa88006fe20 0x7fa880072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52010/0 (socket says 172.21.15.40:52010) 2024-04-26T23:02:06.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.472+0000 7fa87ea00640 1 -- 172.21.15.40:0/78086934 learned_addr learned my addr 172.21.15.40:0/78086934 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:06.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.472+0000 7fa87f400640 1 -- 172.21.15.40:0/78086934 >> v1:172.21.15.103:6789/0 conn(0x7fa8800cc480 legacy=0x7fa880166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:06.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.472+0000 7fa87f400640 1 -- 172.21.15.40:0/78086934 >> v1:172.21.15.40:6789/0 conn(0x7fa88006fe20 legacy=0x7fa880072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.472+0000 7fa87f400640 1 -- 172.21.15.40:0/78086934 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88015e060 con 0x7fa880071e60 2024-04-26T23:02:06.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.472+0000 7fa87e000640 1 -- 172.21.15.40:0/78086934 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 87192325 0 0) 0x7fa88015d7c0 con 0x7fa88006fe20 2024-04-26T23:02:06.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.473+0000 7fa87f400640 1 --2- 172.21.15.40:0/78086934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa880071e60 0x7fa88006f8e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa8700097a0 tx=0x7fa87002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=951b6d7822f4712d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:06.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.474+0000 7fa87e000640 1 -- 172.21.15.40:0/78086934 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870037070 con 0x7fa880071e60 2024-04-26T23:02:06.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.474+0000 7fa87e000640 1 -- 172.21.15.40:0/78086934 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa870032440 con 0x7fa880071e60 2024-04-26T23:02:06.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.474+0000 7fa87e000640 1 -- 172.21.15.40:0/78086934 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870032810 con 0x7fa880071e60 2024-04-26T23:02:06.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.474+0000 7fa886400640 1 -- 172.21.15.40:0/78086934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa880071e60 msgr2=0x7fa88006f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.474+0000 7fa886400640 1 --2- 172.21.15.40:0/78086934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa880071e60 0x7fa88006f8e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa8700097a0 tx=0x7fa87002a1e0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.474+0000 7fa886400640 1 -- 172.21.15.40:0/78086934 shutdown_connections 2024-04-26T23:02:06.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.474+0000 7fa886400640 1 --2- 172.21.15.40:0/78086934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa880071e60 0x7fa88006f8e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.474+0000 7fa886400640 1 -- 172.21.15.40:0/78086934 >> 172.21.15.40:0/78086934 conn(0x7fa8800d0be0 msgr2=0x7fa8800d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:06.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.474+0000 7fa886400640 1 -- 172.21.15.40:0/78086934 shutdown_connections 2024-04-26T23:02:06.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.474+0000 7fa886400640 1 -- 172.21.15.40:0/78086934 wait complete. 2024-04-26T23:02:06.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.476+0000 7fa886400640 1 Processor -- start 2024-04-26T23:02:06.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.476+0000 7fa886400640 1 -- start start 2024-04-26T23:02:06.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.477+0000 7fa886400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88006fe20 0x7fa880151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.477+0000 7fa886400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa880071e60 0x7fa8801f7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.477+0000 7fa886400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8800cc480 0x7fa8801f5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.477+0000 7fa886400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa88015d570 con 0x7fa88006fe20 2024-04-26T23:02:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.477+0000 7fa886400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa88015d3f0 con 0x7fa8800cc480 2024-04-26T23:02:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.477+0000 7fa886400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa880072950 con 0x7fa880071e60 2024-04-26T23:02:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.477+0000 7fa87f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88006fe20 0x7fa880151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.478+0000 7fa87f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88006fe20 0x7fa880151080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36214/0 (socket says 172.21.15.40:36214) 2024-04-26T23:02:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.478+0000 7fa87f400640 1 -- 172.21.15.40:0/1388187258 learned_addr learned my addr 172.21.15.40:0/1388187258 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.478+0000 7fa87ea00640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa880071e60 0x7fa8801f7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.478+0000 7fa87f400640 1 -- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa880071e60 msgr2=0x7fa8801f7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.478+0000 7fa87f400640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa880071e60 0x7fa8801f7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.478+0000 7fa87f400640 1 -- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8800cc480 msgr2=0x7fa8801f5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.478+0000 7fa87f400640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8800cc480 0x7fa8801f5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.478+0000 7fa87f400640 1 -- 172.21.15.40:0/1388187258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8801515c0 con 0x7fa88006fe20 2024-04-26T23:02:06.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.478+0000 7fa87ea00640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa880071e60 0x7fa8801f7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:06.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.478+0000 7fa87f400640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88006fe20 0x7fa880151080 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa8700098c0 tx=0x7fa8700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:06.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.479+0000 7fa877400640 1 -- 172.21.15.40:0/1388187258 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870041070 con 0x7fa88006fe20 2024-04-26T23:02:06.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.479+0000 7fa877400640 1 -- 172.21.15.40:0/1388187258 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa87003c870 con 0x7fa88006fe20 2024-04-26T23:02:06.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.479+0000 7fa877400640 1 -- 172.21.15.40:0/1388187258 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870037040 con 0x7fa88006fe20 2024-04-26T23:02:06.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.479+0000 7fa886400640 1 -- 172.21.15.40:0/1388187258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8801f6680 con 0x7fa88006fe20 2024-04-26T23:02:06.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.479+0000 7fa886400640 1 -- 172.21.15.40:0/1388187258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8801f6c40 con 0x7fa88006fe20 2024-04-26T23:02:06.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.480+0000 7fa886400640 1 -- 172.21.15.40:0/1388187258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8800694f0 con 0x7fa88006fe20 2024-04-26T23:02:06.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.481+0000 7fa877400640 1 -- 172.21.15.40:0/1388187258 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa870044070 con 0x7fa88006fe20 2024-04-26T23:02:06.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.482+0000 7fa877400640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa86006c8f0 0x7fa86006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.482+0000 7fa877400640 1 -- 172.21.15.40:0/1388187258 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa8700acad0 con 0x7fa88006fe20 2024-04-26T23:02:06.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.483+0000 7fa87ea00640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa86006c8f0 0x7fa86006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.484+0000 7fa87ea00640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa86006c8f0 0x7fa86006edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa86c000f60 tx=0x7fa86c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:06.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.486+0000 7fa877400640 1 -- 172.21.15.40:0/1388187258 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa8801515c0 con 0x7fa88006fe20 2024-04-26T23:02:06.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.693+0000 7fa886400640 1 -- 172.21.15.40:0/1388187258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa8801f72f0 con 0x7fa88006fe20 2024-04-26T23:02:06.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.695+0000 7fa877400640 1 -- 172.21.15.40:0/1388187258 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18812 (secure 0 0 0) 0x7fa87007ac20 con 0x7fa88006fe20 2024-04-26T23:02:06.695 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:06.695 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:46.975683+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:06.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.697+0000 7fa886400640 1 -- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa86006c8f0 msgr2=0x7fa86006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.697+0000 7fa886400640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa86006c8f0 0x7fa86006edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa86c000f60 tx=0x7fa86c009040 comp rx=0 tx=0).stop 2024-04-26T23:02:06.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.697+0000 7fa886400640 1 -- 172.21.15.40:0/1388187258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88006fe20 msgr2=0x7fa880151080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.697+0000 7fa886400640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88006fe20 0x7fa880151080 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa8700098c0 tx=0x7fa8700046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.697+0000 7fa886400640 1 -- 172.21.15.40:0/1388187258 shutdown_connections 2024-04-26T23:02:06.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.697+0000 7fa886400640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa86006c8f0 0x7fa86006edb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.697+0000 7fa886400640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8800cc480 0x7fa8801f5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.697+0000 7fa886400640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa880071e60 0x7fa8801f7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.697+0000 7fa886400640 1 --2- 172.21.15.40:0/1388187258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88006fe20 0x7fa880151080 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.697+0000 7fa886400640 1 -- 172.21.15.40:0/1388187258 >> 172.21.15.40:0/1388187258 conn(0x7fa8800d0be0 msgr2=0x7fa8800b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:06.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.698+0000 7fa886400640 1 -- 172.21.15.40:0/1388187258 shutdown_connections 2024-04-26T23:02:06.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.698+0000 7fa886400640 1 -- 172.21.15.40:0/1388187258 wait complete. 2024-04-26T23:02:06.719 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:02:06.719 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:02:06.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.936+0000 7f3a15600640 1 Processor -- start 2024-04-26T23:02:06.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.936+0000 7f3a15600640 1 -- start start 2024-04-26T23:02:06.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.937+0000 7f3a15600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3a10073b70 0x7f3a100aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.937+0000 7f3a15600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3a100cbb50 0x7f3a100af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.937+0000 7f3a15600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a10072c00 con 0x7f3a1015f380 2024-04-26T23:02:06.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.937+0000 7f3a15600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f3a100729b0 con 0x7f3a100cbb50 2024-04-26T23:02:06.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.937+0000 7f3a15600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3a10072ee0 con 0x7f3a10073b70 2024-04-26T23:02:06.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.938+0000 7f3a0f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3a100cbb50 0x7f3a100af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.938+0000 7f3a0f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3a100cbb50 0x7f3a100af1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36228/0 (socket says 172.21.15.40:36228) 2024-04-26T23:02:06.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.938+0000 7f3a0f400640 1 -- 172.21.15.40:0/1139008856 learned_addr learned my addr 172.21.15.40:0/1139008856 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:06.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.938+0000 7f3a07e00640 1 --2- 172.21.15.40:0/1139008856 >> v2:172.21.15.103:3300/0 conn(0x7f3a10073b70 0x7f3a100aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.938+0000 7f3a0f400640 1 -- 172.21.15.40:0/1139008856 >> v2:172.21.15.103:3300/0 conn(0x7f3a10073b70 msgr2=0x7f3a100aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:06.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.938+0000 7f3a0f400640 1 --2- 172.21.15.40:0/1139008856 >> v2:172.21.15.103:3300/0 conn(0x7f3a10073b70 0x7f3a100aecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.938+0000 7f3a0f400640 1 -- 172.21.15.40:0/1139008856 >> v1:172.21.15.103:6789/0 conn(0x7f3a1015f380 legacy=0x7f3a100cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:06.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.939+0000 7f3a0f400640 1 -- 172.21.15.40:0/1139008856 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a100731c0 con 0x7f3a100cbb50 2024-04-26T23:02:06.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.940+0000 7f3a0f400640 1 --2- 172.21.15.40:0/1139008856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a100cbb50 0x7f3a100af1f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3a100ce700 tx=0x7f3a0802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa67476d3ba054c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:06.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.940+0000 7f3a0e000640 1 -- 172.21.15.40:0/1139008856 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08036070 con 0x7f3a100cbb50 2024-04-26T23:02:06.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.940+0000 7f3a0e000640 1 -- 172.21.15.40:0/1139008856 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a08031600 con 0x7f3a100cbb50 2024-04-26T23:02:06.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.940+0000 7f3a0e000640 1 -- 172.21.15.40:0/1139008856 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a080319d0 con 0x7f3a100cbb50 2024-04-26T23:02:06.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.941+0000 7f3a15600640 1 -- 172.21.15.40:0/1139008856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a100cbb50 msgr2=0x7f3a100af1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.941+0000 7f3a15600640 1 --2- 172.21.15.40:0/1139008856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a100cbb50 0x7f3a100af1f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3a100ce700 tx=0x7f3a0802a6f0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.941+0000 7f3a15600640 1 -- 172.21.15.40:0/1139008856 shutdown_connections 2024-04-26T23:02:06.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.941+0000 7f3a15600640 1 --2- 172.21.15.40:0/1139008856 >> v2:172.21.15.103:3300/0 conn(0x7f3a10073b70 0x7f3a100aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.941+0000 7f3a15600640 1 --2- 172.21.15.40:0/1139008856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a100cbb50 0x7f3a100af1f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.941+0000 7f3a15600640 1 -- 172.21.15.40:0/1139008856 >> 172.21.15.40:0/1139008856 conn(0x7f3a100d0be0 msgr2=0x7f3a100d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:06.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.941+0000 7f3a15600640 1 -- 172.21.15.40:0/1139008856 shutdown_connections 2024-04-26T23:02:06.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.941+0000 7f3a15600640 1 -- 172.21.15.40:0/1139008856 wait complete. 2024-04-26T23:02:06.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.943+0000 7f3a15600640 1 Processor -- start 2024-04-26T23:02:06.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.943+0000 7f3a15600640 1 -- start start 2024-04-26T23:02:06.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.944+0000 7f3a15600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a100cbb50 0x7f3a1014ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.944+0000 7f3a15600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a1015f380 0x7f3a10151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.944+0000 7f3a15600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a100c9bb0 0x7f3a10151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.944+0000 7f3a15600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3a10073620 con 0x7f3a100c9bb0 2024-04-26T23:02:06.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.944+0000 7f3a15600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3a100d08b0 con 0x7f3a100cbb50 2024-04-26T23:02:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.944+0000 7f3a15600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3a100737a0 con 0x7f3a1015f380 2024-04-26T23:02:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.945+0000 7f3a0f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a100c9bb0 0x7f3a10151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.945+0000 7f3a0f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a100c9bb0 0x7f3a10151c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36244/0 (socket says 172.21.15.40:36244) 2024-04-26T23:02:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.945+0000 7f3a0f400640 1 -- 172.21.15.40:0/91056948 learned_addr learned my addr 172.21.15.40:0/91056948 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.945+0000 7f3a0ea00640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a100cbb50 0x7f3a1014ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.945+0000 7f3a07e00640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a1015f380 0x7f3a10151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.945+0000 7f3a0f400640 1 -- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a1015f380 msgr2=0x7f3a10151700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.945+0000 7f3a0f400640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a1015f380 0x7f3a10151700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.945+0000 7f3a0f400640 1 -- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a100cbb50 msgr2=0x7f3a1014ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.945+0000 7f3a0f400640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a100cbb50 0x7f3a1014ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.945+0000 7f3a0f400640 1 -- 172.21.15.40:0/91056948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a100c4cb0 con 0x7f3a100c9bb0 2024-04-26T23:02:06.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.946+0000 7f3a0f400640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a100c9bb0 0x7f3a10151c40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3a0802ace0 tx=0x7f3a08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:06.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.946+0000 7f3a06a00640 1 -- 172.21.15.40:0/91056948 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08036050 con 0x7f3a100c9bb0 2024-04-26T23:02:06.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.946+0000 7f3a06a00640 1 -- 172.21.15.40:0/91056948 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a08042070 con 0x7f3a100c9bb0 2024-04-26T23:02:06.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.946+0000 7f3a15600640 1 -- 172.21.15.40:0/91056948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a100c90d0 con 0x7f3a100c9bb0 2024-04-26T23:02:06.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.946+0000 7f3a06a00640 1 -- 172.21.15.40:0/91056948 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08009510 con 0x7f3a100c9bb0 2024-04-26T23:02:06.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.946+0000 7f3a15600640 1 -- 172.21.15.40:0/91056948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a100c9690 con 0x7f3a100c9bb0 2024-04-26T23:02:06.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.947+0000 7f3a15600640 1 -- 172.21.15.40:0/91056948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a100c9890 con 0x7f3a100c9bb0 2024-04-26T23:02:06.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.948+0000 7f3a06a00640 1 -- 172.21.15.40:0/91056948 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3a08007bd0 con 0x7f3a100c9bb0 2024-04-26T23:02:06.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.949+0000 7f3a06a00640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f39ec06c8f0 0x7f39ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:06.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.949+0000 7f3a06a00640 1 -- 172.21.15.40:0/91056948 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f3a080ace50 con 0x7f3a100c9bb0 2024-04-26T23:02:06.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.950+0000 7f3a0ea00640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f39ec06c8f0 0x7f39ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:06.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.951+0000 7f3a0ea00640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f39ec06c8f0 0x7f39ec06edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3a00006fd0 tx=0x7f3a00008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:06.954+0000 7f3a06a00640 1 -- 172.21.15.40:0/91056948 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3a100c9890 con 0x7f3a100c9bb0 2024-04-26T23:02:07.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.160+0000 7f3a15600640 1 -- 172.21.15.40:0/91056948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3a100cb590 con 0x7f3a100c9bb0 2024-04-26T23:02:07.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.161+0000 7f3a06a00640 1 -- 172.21.15.40:0/91056948 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f3a0807ae70 con 0x7f3a100c9bb0 2024-04-26T23:02:07.161 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:02:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.163+0000 7f3a15600640 1 -- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f39ec06c8f0 msgr2=0x7f39ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.163+0000 7f3a15600640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f39ec06c8f0 0x7f39ec06edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3a00006fd0 tx=0x7f3a00008040 comp rx=0 tx=0).stop 2024-04-26T23:02:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.163+0000 7f3a15600640 1 -- 172.21.15.40:0/91056948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a100c9bb0 msgr2=0x7f3a10151c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.163+0000 7f3a15600640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a100c9bb0 0x7f3a10151c40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3a0802ace0 tx=0x7f3a08004a40 comp rx=0 tx=0).stop 2024-04-26T23:02:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.164+0000 7f3a15600640 1 -- 172.21.15.40:0/91056948 shutdown_connections 2024-04-26T23:02:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.164+0000 7f3a15600640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f39ec06c8f0 0x7f39ec06edb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.164+0000 7f3a15600640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a100c9bb0 0x7f3a10151c40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.164+0000 7f3a15600640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3a1015f380 0x7f3a10151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.164+0000 7f3a15600640 1 --2- 172.21.15.40:0/91056948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a100cbb50 0x7f3a1014ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.164+0000 7f3a15600640 1 -- 172.21.15.40:0/91056948 >> 172.21.15.40:0/91056948 conn(0x7f3a100d0be0 msgr2=0x7f3a100b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.164+0000 7f3a15600640 1 -- 172.21.15.40:0/91056948 shutdown_connections 2024-04-26T23:02:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.164+0000 7f3a15600640 1 -- 172.21.15.40:0/91056948 wait complete. 2024-04-26T23:02:07.184 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:02:07.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.399+0000 7f6e82600640 1 Processor -- start 2024-04-26T23:02:07.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.400+0000 7f6e82600640 1 -- start start 2024-04-26T23:02:07.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.401+0000 7f6e82600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6e7c15f440 0x7f6e7c0d1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.401+0000 7f6e82600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e7c0d1da0 0x7f6e7c0cab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.401+0000 7f6e82600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e7c0d5c20 con 0x7f6e7c15f060 2024-04-26T23:02:07.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.401+0000 7f6e82600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6e7c0c99b0 con 0x7f6e7c15f440 2024-04-26T23:02:07.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.401+0000 7f6e82600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6e7c0cfc40 con 0x7f6e7c0d1da0 2024-04-26T23:02:07.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.401+0000 7f6e81200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6e7c15f060 0x7f6e7c0d3640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52020/0 (socket says 172.21.15.40:52020) 2024-04-26T23:02:07.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.401+0000 7f6e81200640 1 -- 172.21.15.40:0/1256179412 learned_addr learned my addr 172.21.15.40:0/1256179412 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:07.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e73e00640 1 --2- 172.21.15.40:0/1256179412 >> v2:172.21.15.103:3300/0 conn(0x7f6e7c15f440 0x7f6e7c0d1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:07.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3210070924 0 0) 0x7f6e7c0d5c20 con 0x7f6e7c15f060 2024-04-26T23:02:07.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e81c00640 1 --2- 172.21.15.40:0/1256179412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e7c0d1da0 0x7f6e7c0cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:07.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e60003520 con 0x7f6e7c15f060 2024-04-26T23:02:07.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2663835470 0 0) 0x7f6e60003520 con 0x7f6e7c15f060 2024-04-26T23:02:07.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e7c0d1da0 msgr2=0x7f6e7c0cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:07.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 --2- 172.21.15.40:0/1256179412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e7c0d1da0 0x7f6e7c0cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 >> v2:172.21.15.103:3300/0 conn(0x7f6e7c15f440 msgr2=0x7f6e7c0d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:07.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 --2- 172.21.15.40:0/1256179412 >> v2:172.21.15.103:3300/0 conn(0x7f6e7c15f440 0x7f6e7c0d1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e7c0cff20 con 0x7f6e7c15f060 2024-04-26T23:02:07.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f6e6c001670 con 0x7f6e7c15f060 2024-04-26T23:02:07.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 >> v1:172.21.15.40:6789/0 conn(0x7f6e7c15f060 legacy=0x7f6e7c0d3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:07.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 --2- 172.21.15.40:0/1256179412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e600039b0 0x7f6e60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 --2- 172.21.15.40:0/1256179412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e60004800 0x7f6e60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 --2- 172.21.15.40:0/1256179412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e600051c0 0x7f6e6000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c0c99b0 con 0x7f6e600039b0 2024-04-26T23:02:07.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c0cfc40 con 0x7f6e600051c0 2024-04-26T23:02:07.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6e6000c060 con 0x7f6e60004800 2024-04-26T23:02:07.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e73e00640 1 --2- 172.21.15.40:0/1256179412 >> v2:172.21.15.103:3300/0 conn(0x7f6e7c15f440 0x7f6e7c0d1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:02:07.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e81c00640 1 --2- 172.21.15.40:0/1256179412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e7c0d1da0 0x7f6e7c0cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:02:07.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.402+0000 7f6e81200640 1 --2- 172.21.15.40:0/1256179412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e600039b0 0x7f6e60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:07.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.403+0000 7f6e73e00640 1 --2- 172.21.15.40:0/1256179412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e60004800 0x7f6e60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:07.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.403+0000 7f6e81200640 1 -- 172.21.15.40:0/1256179412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e60004800 msgr2=0x7f6e60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:07.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.403+0000 7f6e81200640 1 --2- 172.21.15.40:0/1256179412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e60004800 0x7f6e60004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.403+0000 7f6e81200640 1 -- 172.21.15.40:0/1256179412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e600051c0 msgr2=0x7f6e6000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:07.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.403+0000 7f6e81200640 1 --2- 172.21.15.40:0/1256179412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e600051c0 0x7f6e6000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.403+0000 7f6e81200640 1 -- 172.21.15.40:0/1256179412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e60003520 con 0x7f6e600039b0 2024-04-26T23:02:07.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.404+0000 7f6e81200640 1 --2- 172.21.15.40:0/1256179412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e600039b0 0x7f6e60003dd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00c160 tx=0x7f6e60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5136a7ea71bae330 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:07.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.404+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c007c80 con 0x7f6e600039b0 2024-04-26T23:02:07.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.404+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e6c005ad0 con 0x7f6e600039b0 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.404+0000 7f6e7be00640 1 -- 172.21.15.40:0/1256179412 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c007070 con 0x7f6e600039b0 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.405+0000 7f6e82600640 1 -- 172.21.15.40:0/1256179412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e600039b0 msgr2=0x7f6e60003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.405+0000 7f6e82600640 1 --2- 172.21.15.40:0/1256179412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e600039b0 0x7f6e60003dd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00c160 tx=0x7f6e60005830 comp rx=0 tx=0).stop 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.405+0000 7f6e81c00640 1 -- 172.21.15.40:0/1256179412 reap_dead start 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.405+0000 7f6e82600640 1 -- 172.21.15.40:0/1256179412 shutdown_connections 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.405+0000 7f6e82600640 1 -- 172.21.15.40:0/1256179412 >> 172.21.15.40:0/1256179412 conn(0x7f6e7c0d4e50 msgr2=0x7f6e7c0d5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.405+0000 7f6e82600640 1 -- 172.21.15.40:0/1256179412 shutdown_connections 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.405+0000 7f6e82600640 1 -- 172.21.15.40:0/1256179412 wait complete. 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.407+0000 7f6e82600640 1 Processor -- start 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.407+0000 7f6e82600640 1 -- start start 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.408+0000 7f6e82600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e600039b0 0x7f6e7c15d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.408+0000 7f6e82600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e7c1503a0 0x7f6e7c1f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.408+0000 7f6e82600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e7c0ce0f0 0x7f6e7c0ce570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.408+0000 7f6e82600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c0d0ce0 con 0x7f6e7c0ce0f0 2024-04-26T23:02:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.408+0000 7f6e82600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c0d04e0 con 0x7f6e600039b0 2024-04-26T23:02:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.408+0000 7f6e82600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6e7c150820 con 0x7f6e7c1503a0 2024-04-26T23:02:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.409+0000 7f6e81c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e7c0ce0f0 0x7f6e7c0ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.409+0000 7f6e81c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e7c0ce0f0 0x7f6e7c0ce570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36256/0 (socket says 172.21.15.40:36256) 2024-04-26T23:02:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.409+0000 7f6e81c00640 1 -- 172.21.15.40:0/3828907881 learned_addr learned my addr 172.21.15.40:0/3828907881 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.409+0000 7f6e81200640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e600039b0 0x7f6e7c15d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.409+0000 7f6e73e00640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e7c1503a0 0x7f6e7c1f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.409+0000 7f6e81c00640 1 -- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e7c1503a0 msgr2=0x7f6e7c1f50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.409+0000 7f6e81c00640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e7c1503a0 0x7f6e7c1f50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.409+0000 7f6e81c00640 1 -- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e600039b0 msgr2=0x7f6e7c15d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.409+0000 7f6e81c00640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e600039b0 0x7f6e7c15d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.409+0000 7f6e81c00640 1 -- 172.21.15.40:0/3828907881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e7c1f5900 con 0x7f6e7c0ce0f0 2024-04-26T23:02:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.409+0000 7f6e81c00640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e7c0ce0f0 0x7f6e7c0ce570 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6e74007cb0 tx=0x7f6e7400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.410+0000 7f6e79600640 1 -- 172.21.15.40:0/3828907881 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7401a070 con 0x7f6e7c0ce0f0 2024-04-26T23:02:07.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.410+0000 7f6e79600640 1 -- 172.21.15.40:0/3828907881 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e7400de80 con 0x7f6e7c0ce0f0 2024-04-26T23:02:07.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.410+0000 7f6e79600640 1 -- 172.21.15.40:0/3828907881 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74003bd0 con 0x7f6e7c0ce0f0 2024-04-26T23:02:07.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.410+0000 7f6e82600640 1 -- 172.21.15.40:0/3828907881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e7c1f5be0 con 0x7f6e7c0ce0f0 2024-04-26T23:02:07.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.410+0000 7f6e82600640 1 -- 172.21.15.40:0/3828907881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e7c1f6210 con 0x7f6e7c0ce0f0 2024-04-26T23:02:07.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.411+0000 7f6e82600640 1 -- 172.21.15.40:0/3828907881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e7c0ceab0 con 0x7f6e7c0ce0f0 2024-04-26T23:02:07.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.412+0000 7f6e79600640 1 -- 172.21.15.40:0/3828907881 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6e7401d070 con 0x7f6e7c0ce0f0 2024-04-26T23:02:07.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.413+0000 7f6e79600640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6e5806c8f0 0x7f6e5806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.414+0000 7f6e79600640 1 -- 172.21.15.40:0/3828907881 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f6e7408be70 con 0x7f6e7c0ce0f0 2024-04-26T23:02:07.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.414+0000 7f6e81200640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6e5806c8f0 0x7f6e5806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:07.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.415+0000 7f6e81200640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6e5806c8f0 0x7f6e5806edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c032000 tx=0x7f6e6c007000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.418+0000 7f6e79600640 1 -- 172.21.15.40:0/3828907881 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f6e74010070 con 0x7f6e7c0ce0f0 2024-04-26T23:02:07.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.632+0000 7f6e82600640 1 -- 172.21.15.40:0/3828907881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6e7c069a80 con 0x7f6e7c0ce0f0 2024-04-26T23:02:07.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.632+0000 7f6e79600640 1 -- 172.21.15.40:0/3828907881 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f6e74007e60 con 0x7f6e7c0ce0f0 2024-04-26T23:02:07.632 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.635+0000 7f6e82600640 1 -- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6e5806c8f0 msgr2=0x7f6e5806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.635+0000 7f6e82600640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6e5806c8f0 0x7f6e5806edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c032000 tx=0x7f6e6c007000 comp rx=0 tx=0).stop 2024-04-26T23:02:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.635+0000 7f6e82600640 1 -- 172.21.15.40:0/3828907881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e7c0ce0f0 msgr2=0x7f6e7c0ce570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.635+0000 7f6e82600640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e7c0ce0f0 0x7f6e7c0ce570 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6e74007cb0 tx=0x7f6e7400d4a0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.635+0000 7f6e82600640 1 -- 172.21.15.40:0/3828907881 shutdown_connections 2024-04-26T23:02:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.635+0000 7f6e82600640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6e5806c8f0 0x7f6e5806edb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.635+0000 7f6e82600640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e7c0ce0f0 0x7f6e7c0ce570 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.635+0000 7f6e82600640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6e7c1503a0 0x7f6e7c1f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.635+0000 7f6e82600640 1 --2- 172.21.15.40:0/3828907881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e600039b0 0x7f6e7c15d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.635+0000 7f6e82600640 1 -- 172.21.15.40:0/3828907881 >> 172.21.15.40:0/3828907881 conn(0x7f6e7c0d4e50 msgr2=0x7f6e7c06e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:07.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.635+0000 7f6e82600640 1 -- 172.21.15.40:0/3828907881 shutdown_connections 2024-04-26T23:02:07.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.635+0000 7f6e82600640 1 -- 172.21.15.40:0/3828907881 wait complete. 2024-04-26T23:02:07.655 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:02:07.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.876+0000 7f5bcd200640 1 Processor -- start 2024-04-26T23:02:07.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.877+0000 7f5bcd200640 1 -- start start 2024-04-26T23:02:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.878+0000 7f5bcd200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5bc80c6f10 0x7f5bc80c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.878+0000 7f5bcd200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bc80cc980 0x7f5bc80cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.878+0000 7f5bcd200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc80c6010 0x7f5bc80cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.878+0000 7f5bcd200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f5bc80c72f0 con 0x7f5bc80cc980 2024-04-26T23:02:07.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.878+0000 7f5bcd200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5bc80cb860 con 0x7f5bc80c6f10 2024-04-26T23:02:07.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.878+0000 7f5bcd200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc80cce10 con 0x7f5bc80c6010 2024-04-26T23:02:07.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.879+0000 7f5bc6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bc80cc980 0x7f5bc80cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:07.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.879+0000 7f5bc6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bc80cc980 0x7f5bc80cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36258/0 (socket says 172.21.15.40:36258) 2024-04-26T23:02:07.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.879+0000 7f5bc6a00640 1 -- 172.21.15.40:0/422076972 learned_addr learned my addr 172.21.15.40:0/422076972 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:07.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.879+0000 7f5bc7400640 1 --2- 172.21.15.40:0/422076972 >> v2:172.21.15.103:3300/0 conn(0x7f5bc80c6f10 0x7f5bc80c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:07.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.879+0000 7f5bc6a00640 1 -- 172.21.15.40:0/422076972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc80c6010 msgr2=0x7f5bc80cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:07.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.879+0000 7f5bc6a00640 1 --2- 172.21.15.40:0/422076972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc80c6010 0x7f5bc80cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.879+0000 7f5bc6a00640 1 -- 172.21.15.40:0/422076972 >> v2:172.21.15.103:3300/0 conn(0x7f5bc80c6f10 msgr2=0x7f5bc80c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:07.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.879+0000 7f5bc6a00640 1 --2- 172.21.15.40:0/422076972 >> v2:172.21.15.103:3300/0 conn(0x7f5bc80c6f10 0x7f5bc80c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.879+0000 7f5bc6a00640 1 -- 172.21.15.40:0/422076972 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc80cd0f0 con 0x7f5bc80cc980 2024-04-26T23:02:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.880+0000 7f5bc6a00640 1 --2- 172.21.15.40:0/422076972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bc80cc980 0x7f5bc80cbcf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80ce590 tx=0x7f5bb402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce395ae9761d2805 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.881+0000 7f5bc6000640 1 -- 172.21.15.40:0/422076972 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4037070 con 0x7f5bc80cc980 2024-04-26T23:02:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.881+0000 7f5bc6000640 1 -- 172.21.15.40:0/422076972 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bb4032440 con 0x7f5bc80cc980 2024-04-26T23:02:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.881+0000 7f5bc6000640 1 -- 172.21.15.40:0/422076972 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4032810 con 0x7f5bc80cc980 2024-04-26T23:02:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.881+0000 7f5bcd200640 1 -- 172.21.15.40:0/422076972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bc80cc980 msgr2=0x7f5bc80cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.881+0000 7f5bcd200640 1 --2- 172.21.15.40:0/422076972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bc80cc980 0x7f5bc80cbcf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80ce590 tx=0x7f5bb402a140 comp rx=0 tx=0).stop 2024-04-26T23:02:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.882+0000 7f5bcd200640 1 -- 172.21.15.40:0/422076972 shutdown_connections 2024-04-26T23:02:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.882+0000 7f5bcd200640 1 --2- 172.21.15.40:0/422076972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc80c6010 0x7f5bc80cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.882+0000 7f5bcd200640 1 --2- 172.21.15.40:0/422076972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bc80cc980 0x7f5bc80cbcf0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.882+0000 7f5bcd200640 1 --2- 172.21.15.40:0/422076972 >> v2:172.21.15.103:3300/0 conn(0x7f5bc80c6f10 0x7f5bc80c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.882+0000 7f5bcd200640 1 -- 172.21.15.40:0/422076972 >> 172.21.15.40:0/422076972 conn(0x7f5bc80d0a10 msgr2=0x7f5bc80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.882+0000 7f5bcd200640 1 -- 172.21.15.40:0/422076972 shutdown_connections 2024-04-26T23:02:07.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.882+0000 7f5bcd200640 1 -- 172.21.15.40:0/422076972 wait complete. 2024-04-26T23:02:07.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.883+0000 7f5bcd200640 1 Processor -- start 2024-04-26T23:02:07.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.884+0000 7f5bcd200640 1 -- start start 2024-04-26T23:02:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.884+0000 7f5bcd200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc80c6010 0x7f5bc81453f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.884+0000 7f5bcd200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bc80c6f10 0x7f5bc80b6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.884+0000 7f5bcd200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc80b3950 0x7f5bc80b5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.884+0000 7f5bcd200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc80c76a0 con 0x7f5bc80c6f10 2024-04-26T23:02:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.884+0000 7f5bcd200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc80cb1a0 con 0x7f5bc80c6010 2024-04-26T23:02:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.884+0000 7f5bcd200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc80c7820 con 0x7f5bc80b3950 2024-04-26T23:02:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.885+0000 7f5bc6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bc80c6f10 0x7f5bc80b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.885+0000 7f5bc6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bc80c6f10 0x7f5bc80b6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36274/0 (socket says 172.21.15.40:36274) 2024-04-26T23:02:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.885+0000 7f5bc6a00640 1 -- 172.21.15.40:0/4049149325 learned_addr learned my addr 172.21.15.40:0/4049149325 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.885+0000 7f5bc7400640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc80c6010 0x7f5bc81453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.885+0000 7f5bc6a00640 1 -- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc80b3950 msgr2=0x7f5bc80b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.885+0000 7f5bc6a00640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc80b3950 0x7f5bc80b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.885+0000 7f5bc6a00640 1 -- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc80c6010 msgr2=0x7f5bc81453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.885+0000 7f5bc6a00640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc80c6010 0x7f5bc81453f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.885+0000 7f5bc6a00640 1 -- 172.21.15.40:0/4049149325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc80b3de0 con 0x7f5bc80c6f10 2024-04-26T23:02:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.885+0000 7f5bc7400640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc80c6010 0x7f5bc81453f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.886+0000 7f5bc6a00640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bc80c6f10 0x7f5bc80b6030 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4000c00 tx=0x7f5bb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.886+0000 7f5bbf400640 1 -- 172.21.15.40:0/4049149325 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4041020 con 0x7f5bc80c6f10 2024-04-26T23:02:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.886+0000 7f5bbf400640 1 -- 172.21.15.40:0/4049149325 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bb4030740 con 0x7f5bc80c6f10 2024-04-26T23:02:07.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.886+0000 7f5bbf400640 1 -- 172.21.15.40:0/4049149325 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4037040 con 0x7f5bc80c6f10 2024-04-26T23:02:07.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.886+0000 7f5bcd200640 1 -- 172.21.15.40:0/4049149325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc80b6730 con 0x7f5bc80c6f10 2024-04-26T23:02:07.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.886+0000 7f5bcd200640 1 -- 172.21.15.40:0/4049149325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc8156d60 con 0x7f5bc80c6f10 2024-04-26T23:02:07.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.887+0000 7f5bcd200640 1 -- 172.21.15.40:0/4049149325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc8069370 con 0x7f5bc80c6f10 2024-04-26T23:02:07.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.888+0000 7f5bbf400640 1 -- 172.21.15.40:0/4049149325 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5bb4044020 con 0x7f5bc80c6f10 2024-04-26T23:02:07.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.889+0000 7f5bbf400640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ba806c8f0 0x7f5ba806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:07.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.889+0000 7f5bbf400640 1 -- 172.21.15.40:0/4049149325 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f5bb40ac890 con 0x7f5bc80c6f10 2024-04-26T23:02:07.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.890+0000 7f5bc7400640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ba806c8f0 0x7f5ba806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:07.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.891+0000 7f5bc7400640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ba806c8f0 0x7f5ba806edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8006fd0 tx=0x7f5bb8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:07.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:07.894+0000 7f5bbf400640 1 -- 172.21.15.40:0/4049149325 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5bb404b030 con 0x7f5bc80c6f10 2024-04-26T23:02:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.096+0000 7f5bcd200640 1 -- 172.21.15.40:0/4049149325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5bc80b5e10 con 0x7f5bc80c6f10 2024-04-26T23:02:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.096+0000 7f5bbf400640 1 -- 172.21.15.40:0/4049149325 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f5bc80b3de0 con 0x7f5bc80c6f10 2024-04-26T23:02:08.097 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:02:08.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.099+0000 7f5bcd200640 1 -- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ba806c8f0 msgr2=0x7f5ba806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:08.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.099+0000 7f5bcd200640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ba806c8f0 0x7f5ba806edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8006fd0 tx=0x7f5bb8008040 comp rx=0 tx=0).stop 2024-04-26T23:02:08.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.099+0000 7f5bcd200640 1 -- 172.21.15.40:0/4049149325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bc80c6f10 msgr2=0x7f5bc80b6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:08.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.099+0000 7f5bcd200640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bc80c6f10 0x7f5bc80b6030 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4000c00 tx=0x7f5bb40046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.099+0000 7f5bcd200640 1 -- 172.21.15.40:0/4049149325 shutdown_connections 2024-04-26T23:02:08.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.099+0000 7f5bcd200640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ba806c8f0 0x7f5ba806edb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.099+0000 7f5bcd200640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5bc80b3950 0x7f5bc80b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.099+0000 7f5bcd200640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bc80c6f10 0x7f5bc80b6030 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.099+0000 7f5bcd200640 1 --2- 172.21.15.40:0/4049149325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bc80c6010 0x7f5bc81453f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.099+0000 7f5bcd200640 1 -- 172.21.15.40:0/4049149325 >> 172.21.15.40:0/4049149325 conn(0x7f5bc80d0a10 msgr2=0x7f5bc806e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:08.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.099+0000 7f5bcd200640 1 -- 172.21.15.40:0/4049149325 shutdown_connections 2024-04-26T23:02:08.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.099+0000 7f5bcd200640 1 -- 172.21.15.40:0/4049149325 wait complete. 2024-04-26T23:02:08.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:02:08.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.330+0000 7f1689200640 1 Processor -- start 2024-04-26T23:02:08.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.331+0000 7f1689200640 1 -- start start 2024-04-26T23:02:08.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.332+0000 7f1689200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f16840b5500 0x7f16840b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:08.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.332+0000 7f1689200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16840b59b0 0x7f16840b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:08.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.332+0000 7f1689200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16840d3dc0 con 0x7f16840b5120 2024-04-26T23:02:08.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.332+0000 7f1689200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f16840d03b0 con 0x7f16840b5500 2024-04-26T23:02:08.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.332+0000 7f1689200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f16840b75b0 con 0x7f16840b59b0 2024-04-26T23:02:08.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.332+0000 7f1682a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f16840b5500 0x7f16840b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:08.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.332+0000 7f1683400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f16840b5120 0x7f16840b6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52034/0 (socket says 172.21.15.40:52034) 2024-04-26T23:02:08.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.332+0000 7f1683400640 1 -- 172.21.15.40:0/4077759660 learned_addr learned my addr 172.21.15.40:0/4077759660 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:08.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.333+0000 7f1683e00640 1 --2- 172.21.15.40:0/4077759660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16840b59b0 0x7f16840b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:08.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.333+0000 7f1682a00640 1 -- 172.21.15.40:0/4077759660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16840b59b0 msgr2=0x7f16840b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:08.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.333+0000 7f1682a00640 1 --2- 172.21.15.40:0/4077759660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16840b59b0 0x7f16840b6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.333+0000 7f1682a00640 1 -- 172.21.15.40:0/4077759660 >> v1:172.21.15.40:6789/0 conn(0x7f16840b5120 legacy=0x7f16840b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:08.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.333+0000 7f1682a00640 1 -- 172.21.15.40:0/4077759660 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16840d0600 con 0x7f16840b5500 2024-04-26T23:02:08.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.333+0000 7f1682000640 1 -- 172.21.15.40:0/4077759660 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 573851123 0 0) 0x7f16840d3dc0 con 0x7f16840b5120 2024-04-26T23:02:08.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.334+0000 7f1682a00640 1 --2- 172.21.15.40:0/4077759660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16840b5500 0x7f16840b5ee0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f166c0097d0 tx=0x7f166c02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e84dfcef0efd2a82 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:08.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.334+0000 7f1682000640 1 -- 172.21.15.40:0/4077759660 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c037070 con 0x7f16840b5500 2024-04-26T23:02:08.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.334+0000 7f1682000640 1 -- 172.21.15.40:0/4077759660 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f166c02adb0 con 0x7f16840b5500 2024-04-26T23:02:08.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.334+0000 7f1682000640 1 -- 172.21.15.40:0/4077759660 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c032660 con 0x7f16840b5500 2024-04-26T23:02:08.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.335+0000 7f1689200640 1 -- 172.21.15.40:0/4077759660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16840b5500 msgr2=0x7f16840b5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:08.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.335+0000 7f1689200640 1 --2- 172.21.15.40:0/4077759660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16840b5500 0x7f16840b5ee0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f166c0097d0 tx=0x7f166c02a0a0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.335+0000 7f1689200640 1 -- 172.21.15.40:0/4077759660 shutdown_connections 2024-04-26T23:02:08.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.335+0000 7f1689200640 1 --2- 172.21.15.40:0/4077759660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16840b59b0 0x7f16840b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.335+0000 7f1689200640 1 --2- 172.21.15.40:0/4077759660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16840b5500 0x7f16840b5ee0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.335+0000 7f1689200640 1 -- 172.21.15.40:0/4077759660 >> 172.21.15.40:0/4077759660 conn(0x7f16840d4db0 msgr2=0x7f16840d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:08.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.335+0000 7f1689200640 1 -- 172.21.15.40:0/4077759660 shutdown_connections 2024-04-26T23:02:08.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.335+0000 7f1689200640 1 -- 172.21.15.40:0/4077759660 wait complete. 2024-04-26T23:02:08.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.337+0000 7f1689200640 1 Processor -- start 2024-04-26T23:02:08.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.337+0000 7f1689200640 1 -- start start 2024-04-26T23:02:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1689200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16840b5120 0x7f168414ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1689200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16840b5500 0x7f168406df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1689200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16840b59b0 0x7f168406e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1689200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f16840b7c00 con 0x7f16840b59b0 2024-04-26T23:02:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1689200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f16840d0bc0 con 0x7f16840b5500 2024-04-26T23:02:08.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1689200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f16840b7d80 con 0x7f16840b5120 2024-04-26T23:02:08.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1683e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16840b59b0 0x7f168406e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:08.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1683e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16840b59b0 0x7f168406e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36280/0 (socket says 172.21.15.40:36280) 2024-04-26T23:02:08.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1683e00640 1 -- 172.21.15.40:0/942684962 learned_addr learned my addr 172.21.15.40:0/942684962 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:08.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1682a00640 1 --2- 172.21.15.40:0/942684962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16840b5500 0x7f168406df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:08.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1683e00640 1 -- 172.21.15.40:0/942684962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16840b5120 msgr2=0x7f168414ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:08.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1683e00640 1 --2- 172.21.15.40:0/942684962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16840b5120 0x7f168414ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1683e00640 1 -- 172.21.15.40:0/942684962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16840b5500 msgr2=0x7f168406df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:08.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1683e00640 1 --2- 172.21.15.40:0/942684962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16840b5500 0x7f168406df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.338+0000 7f1683e00640 1 -- 172.21.15.40:0/942684962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f168414b3e0 con 0x7f16840b59b0 2024-04-26T23:02:08.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.339+0000 7f1683e00640 1 --2- 172.21.15.40:0/942684962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16840b59b0 0x7f168406e4d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f167c00cc60 tx=0x7f167c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:08.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.339+0000 7f167b400640 1 -- 172.21.15.40:0/942684962 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167c003990 con 0x7f16840b59b0 2024-04-26T23:02:08.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.339+0000 7f167b400640 1 -- 172.21.15.40:0/942684962 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f167c003b30 con 0x7f16840b59b0 2024-04-26T23:02:08.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.339+0000 7f1689200640 1 -- 172.21.15.40:0/942684962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f168414b6c0 con 0x7f16840b59b0 2024-04-26T23:02:08.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.339+0000 7f167b400640 1 -- 172.21.15.40:0/942684962 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167c004020 con 0x7f16840b59b0 2024-04-26T23:02:08.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.339+0000 7f1689200640 1 -- 172.21.15.40:0/942684962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f168414bc80 con 0x7f16840b59b0 2024-04-26T23:02:08.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.340+0000 7f1689200640 1 -- 172.21.15.40:0/942684962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1684069370 con 0x7f16840b59b0 2024-04-26T23:02:08.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.341+0000 7f167b400640 1 -- 172.21.15.40:0/942684962 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f167c0041c0 con 0x7f16840b59b0 2024-04-26T23:02:08.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.342+0000 7f167b400640 1 --2- 172.21.15.40:0/942684962 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f166006c8f0 0x7f166006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:08.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.342+0000 7f167b400640 1 -- 172.21.15.40:0/942684962 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f167c08c450 con 0x7f16840b59b0 2024-04-26T23:02:08.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.343+0000 7f1683400640 1 --2- 172.21.15.40:0/942684962 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f166006c8f0 0x7f166006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:08.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.344+0000 7f1683400640 1 --2- 172.21.15.40:0/942684962 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f166006c8f0 0x7f166006edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1674007fd0 tx=0x7f1674007920 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:08.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.347+0000 7f167b400640 1 -- 172.21.15.40:0/942684962 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f167c014030 con 0x7f16840b59b0 2024-04-26T23:02:08.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.547+0000 7f1689200640 1 -- 172.21.15.40:0/942684962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f16840d2660 con 0x7f16840b59b0 2024-04-26T23:02:08.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.548+0000 7f167b400640 1 -- 172.21.15.40:0/942684962 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f167c05a470 con 0x7f16840b59b0 2024-04-26T23:02:08.548 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:08.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.550+0000 7f1689200640 1 -- 172.21.15.40:0/942684962 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f166006c8f0 msgr2=0x7f166006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:08.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.550+0000 7f1689200640 1 --2- 172.21.15.40:0/942684962 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f166006c8f0 0x7f166006edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1674007fd0 tx=0x7f1674007920 comp rx=0 tx=0).stop 2024-04-26T23:02:08.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.550+0000 7f1689200640 1 -- 172.21.15.40:0/942684962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16840b59b0 msgr2=0x7f168406e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:08.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.550+0000 7f1689200640 1 --2- 172.21.15.40:0/942684962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16840b59b0 0x7f168406e4d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f167c00cc60 tx=0x7f167c00a5c0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.550+0000 7f1689200640 1 -- 172.21.15.40:0/942684962 shutdown_connections 2024-04-26T23:02:08.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.550+0000 7f1689200640 1 --2- 172.21.15.40:0/942684962 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f166006c8f0 0x7f166006edb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.550+0000 7f1689200640 1 --2- 172.21.15.40:0/942684962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16840b59b0 0x7f168406e4d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.550+0000 7f1689200640 1 --2- 172.21.15.40:0/942684962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16840b5500 0x7f168406df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.550+0000 7f1689200640 1 --2- 172.21.15.40:0/942684962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16840b5120 0x7f168414ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.550+0000 7f1689200640 1 -- 172.21.15.40:0/942684962 >> 172.21.15.40:0/942684962 conn(0x7f16840d4db0 msgr2=0x7f16840b9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:08.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.551+0000 7f1689200640 1 -- 172.21.15.40:0/942684962 shutdown_connections 2024-04-26T23:02:08.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.551+0000 7f1689200640 1 -- 172.21.15.40:0/942684962 wait complete. 2024-04-26T23:02:08.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:08.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.772+0000 7fd1f8a00640 1 Processor -- start 2024-04-26T23:02:08.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.772+0000 7fd1f8a00640 1 -- start start 2024-04-26T23:02:08.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.773+0000 7fd1f8a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd1f4160630 0x7fd1f40d0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:08.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.773+0000 7fd1f8a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1f40cfb00 0x7fd1f40c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:08.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.773+0000 7fd1f8a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1f40d10e0 con 0x7fd1f4160a10 2024-04-26T23:02:08.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.773+0000 7fd1f8a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd1f40d0570 con 0x7fd1f4160630 2024-04-26T23:02:08.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.773+0000 7fd1f8a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd1f40cff30 con 0x7fd1f40cfb00 2024-04-26T23:02:08.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.774+0000 7fd1f2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1f40cfb00 0x7fd1f40c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:08.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.774+0000 7fd1f1400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd1f4160a10 0x7fd1f40cf3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:47352/0 (socket says 172.21.15.40:47352) 2024-04-26T23:02:08.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.774+0000 7fd1f1400640 1 -- 172.21.15.40:0/2860172162 learned_addr learned my addr 172.21.15.40:0/2860172162 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:08.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.774+0000 7fd1f1e00640 1 --2- 172.21.15.40:0/2860172162 >> v2:172.21.15.103:3300/0 conn(0x7fd1f4160630 0x7fd1f40d0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:08.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.775+0000 7fd1f2800640 1 -- 172.21.15.40:0/2860172162 >> v2:172.21.15.103:3300/0 conn(0x7fd1f4160630 msgr2=0x7fd1f40d0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:08.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.775+0000 7fd1f2800640 1 --2- 172.21.15.40:0/2860172162 >> v2:172.21.15.103:3300/0 conn(0x7fd1f4160630 0x7fd1f40d0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.775+0000 7fd1f0a00640 1 -- 172.21.15.40:0/2860172162 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2124014969 0 0) 0x7fd1f40d10e0 con 0x7fd1f4160a10 2024-04-26T23:02:08.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.775+0000 7fd1f2800640 1 -- 172.21.15.40:0/2860172162 >> v1:172.21.15.103:6789/0 conn(0x7fd1f4160a10 legacy=0x7fd1f40cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:08.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.775+0000 7fd1f2800640 1 -- 172.21.15.40:0/2860172162 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f40b41d0 con 0x7fd1f40cfb00 2024-04-26T23:02:08.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.776+0000 7fd1f2800640 1 --2- 172.21.15.40:0/2860172162 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1f40cfb00 0x7fd1f40c6a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec0097a0 tx=0x7fd1ec02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3559e342eb47b464 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:08.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.777+0000 7fd1f0a00640 1 -- 172.21.15.40:0/2860172162 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec037070 con 0x7fd1f40cfb00 2024-04-26T23:02:08.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.777+0000 7fd1f0a00640 1 -- 172.21.15.40:0/2860172162 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1ec032440 con 0x7fd1f40cfb00 2024-04-26T23:02:08.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.777+0000 7fd1f0a00640 1 -- 172.21.15.40:0/2860172162 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec032810 con 0x7fd1f40cfb00 2024-04-26T23:02:08.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.777+0000 7fd1f8a00640 1 -- 172.21.15.40:0/2860172162 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1f40cfb00 msgr2=0x7fd1f40c6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:08.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.777+0000 7fd1f8a00640 1 --2- 172.21.15.40:0/2860172162 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1f40cfb00 0x7fd1f40c6a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec0097a0 tx=0x7fd1ec02a210 comp rx=0 tx=0).stop 2024-04-26T23:02:08.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.777+0000 7fd1f8a00640 1 -- 172.21.15.40:0/2860172162 shutdown_connections 2024-04-26T23:02:08.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.777+0000 7fd1f8a00640 1 --2- 172.21.15.40:0/2860172162 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1f40cfb00 0x7fd1f40c6a80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.777+0000 7fd1f8a00640 1 --2- 172.21.15.40:0/2860172162 >> v2:172.21.15.103:3300/0 conn(0x7fd1f4160630 0x7fd1f40d0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.777+0000 7fd1f8a00640 1 -- 172.21.15.40:0/2860172162 >> 172.21.15.40:0/2860172162 conn(0x7fd1f40d2dc0 msgr2=0x7fd1f40d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:08.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.777+0000 7fd1f8a00640 1 -- 172.21.15.40:0/2860172162 shutdown_connections 2024-04-26T23:02:08.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.778+0000 7fd1f8a00640 1 -- 172.21.15.40:0/2860172162 wait complete. 2024-04-26T23:02:08.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.779+0000 7fd1f8a00640 1 Processor -- start 2024-04-26T23:02:08.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.779+0000 7fd1f8a00640 1 -- start start 2024-04-26T23:02:08.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.780+0000 7fd1f8a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1f40cfb00 0x7fd1f415c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:08.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.780+0000 7fd1f8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1f4160630 0x7fd1f409dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:08.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.780+0000 7fd1f8a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1f4160a10 0x7fd1f409fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:08.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.780+0000 7fd1f8a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f40b3e80 con 0x7fd1f4160630 2024-04-26T23:02:08.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.780+0000 7fd1f8a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f40d0200 con 0x7fd1f40cfb00 2024-04-26T23:02:08.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.780+0000 7fd1f8a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd1f40d3b10 con 0x7fd1f4160a10 2024-04-26T23:02:08.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.781+0000 7fd1f1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1f4160630 0x7fd1f409dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:08.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.781+0000 7fd1f1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1f4160630 0x7fd1f409dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36296/0 (socket says 172.21.15.40:36296) 2024-04-26T23:02:08.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.781+0000 7fd1f1400640 1 -- 172.21.15.40:0/3594092226 learned_addr learned my addr 172.21.15.40:0/3594092226 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:08.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.781+0000 7fd1f2800640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1f4160a10 0x7fd1f409fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:08.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.781+0000 7fd1f1e00640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1f40cfb00 0x7fd1f415c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:08.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.781+0000 7fd1f1400640 1 -- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1f4160a10 msgr2=0x7fd1f409fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:08.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.781+0000 7fd1f1400640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1f4160a10 0x7fd1f409fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.781+0000 7fd1f1400640 1 -- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1f40cfb00 msgr2=0x7fd1f415c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:08.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.781+0000 7fd1f1400640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1f40cfb00 0x7fd1f415c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:08.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.781+0000 7fd1f1400640 1 -- 172.21.15.40:0/3594092226 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f409e610 con 0x7fd1f4160630 2024-04-26T23:02:08.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.781+0000 7fd1f1400640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1f4160630 0x7fd1f409dc30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000c460 tx=0x7fd1e000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:08.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.782+0000 7fd1ea000640 1 -- 172.21.15.40:0/3594092226 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0018070 con 0x7fd1f4160630 2024-04-26T23:02:08.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.782+0000 7fd1ea000640 1 -- 172.21.15.40:0/3594092226 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1e0007500 con 0x7fd1f4160630 2024-04-26T23:02:08.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.782+0000 7fd1ea000640 1 -- 172.21.15.40:0/3594092226 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0004a20 con 0x7fd1f4160630 2024-04-26T23:02:08.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.782+0000 7fd1f8a00640 1 -- 172.21.15.40:0/3594092226 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f409e8f0 con 0x7fd1f4160630 2024-04-26T23:02:08.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.782+0000 7fd1f8a00640 1 -- 172.21.15.40:0/3594092226 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f40cb360 con 0x7fd1f4160630 2024-04-26T23:02:08.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.783+0000 7fd1f8a00640 1 -- 172.21.15.40:0/3594092226 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1f40694f0 con 0x7fd1f4160630 2024-04-26T23:02:08.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.784+0000 7fd1ea000640 1 -- 172.21.15.40:0/3594092226 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd1e001b070 con 0x7fd1f4160630 2024-04-26T23:02:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.785+0000 7fd1ea000640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd1d406c8f0 0x7fd1d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.785+0000 7fd1ea000640 1 -- 172.21.15.40:0/3594092226 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd1e008b210 con 0x7fd1f4160630 2024-04-26T23:02:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.786+0000 7fd1f1e00640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd1d406c8f0 0x7fd1d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:08.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.787+0000 7fd1f1e00640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd1d406c8f0 0x7fd1d406edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40074a0 tx=0x7fd1e400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:08.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.789+0000 7fd1ea000640 1 -- 172.21.15.40:0/3594092226 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd1e0059230 con 0x7fd1f4160630 2024-04-26T23:02:08.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.993+0000 7fd1f8a00640 1 -- 172.21.15.40:0/3594092226 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd1f409eac0 con 0x7fd1f4160630 2024-04-26T23:02:08.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.997+0000 7fd1ea000640 1 -- 172.21.15.40:0/3594092226 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18812 (secure 0 0 0) 0x7fd1e0058b30 con 0x7fd1f4160630 2024-04-26T23:02:08.997 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:08.997 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:46.975683+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:09.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.999+0000 7fd1f8a00640 1 -- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd1d406c8f0 msgr2=0x7fd1d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:09.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.999+0000 7fd1f8a00640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd1d406c8f0 0x7fd1d406edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40074a0 tx=0x7fd1e400b040 comp rx=0 tx=0).stop 2024-04-26T23:02:09.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.999+0000 7fd1f8a00640 1 -- 172.21.15.40:0/3594092226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1f4160630 msgr2=0x7fd1f409dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:09.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:08.999+0000 7fd1f8a00640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1f4160630 0x7fd1f409dc30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000c460 tx=0x7fd1e000c920 comp rx=0 tx=0).stop 2024-04-26T23:02:09.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.000+0000 7fd1f8a00640 1 -- 172.21.15.40:0/3594092226 shutdown_connections 2024-04-26T23:02:09.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.000+0000 7fd1f8a00640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd1d406c8f0 0x7fd1d406edb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.000+0000 7fd1f8a00640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1f4160a10 0x7fd1f409fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.000+0000 7fd1f8a00640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1f4160630 0x7fd1f409dc30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.000+0000 7fd1f8a00640 1 --2- 172.21.15.40:0/3594092226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1f40cfb00 0x7fd1f415c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.000+0000 7fd1f8a00640 1 -- 172.21.15.40:0/3594092226 >> 172.21.15.40:0/3594092226 conn(0x7fd1f40d2dc0 msgr2=0x7fd1f415ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:09.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.000+0000 7fd1f8a00640 1 -- 172.21.15.40:0/3594092226 shutdown_connections 2024-04-26T23:02:09.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.000+0000 7fd1f8a00640 1 -- 172.21.15.40:0/3594092226 wait complete. 2024-04-26T23:02:09.019 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:02:09.019 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:02:09.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.244+0000 7ff198a00640 1 Processor -- start 2024-04-26T23:02:09.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.244+0000 7ff198a00640 1 -- start start 2024-04-26T23:02:09.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.246+0000 7ff198a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff194071e60 0x7ff19406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:09.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.246+0000 7ff198a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff19406fe20 0x7ff194072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:09.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.246+0000 7ff198a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff19415d7c0 con 0x7ff194072780 2024-04-26T23:02:09.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.246+0000 7ff198a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff19415daa0 con 0x7ff19406fe20 2024-04-26T23:02:09.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.246+0000 7ff198a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff19415dd80 con 0x7ff194071e60 2024-04-26T23:02:09.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.246+0000 7ff192800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff194072780 0x7ff1940cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52042/0 (socket says 172.21.15.40:52042) 2024-04-26T23:02:09.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.246+0000 7ff192800640 1 -- 172.21.15.40:0/251873986 learned_addr learned my addr 172.21.15.40:0/251873986 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:09.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.246+0000 7ff191400640 1 --2- 172.21.15.40:0/251873986 >> v2:172.21.15.40:3300/0 conn(0x7ff19406fe20 0x7ff194072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:09.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.246+0000 7ff191e00640 1 --2- 172.21.15.40:0/251873986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff194071e60 0x7ff19406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:09.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.247+0000 7ff190a00640 1 -- 172.21.15.40:0/251873986 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 765790950 0 0) 0x7ff19415d7c0 con 0x7ff194072780 2024-04-26T23:02:09.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.247+0000 7ff190a00640 1 -- 172.21.15.40:0/251873986 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff178003520 con 0x7ff194072780 2024-04-26T23:02:09.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.247+0000 7ff191400640 1 -- 172.21.15.40:0/251873986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff194071e60 msgr2=0x7ff19406f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:09.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.247+0000 7ff191400640 1 --2- 172.21.15.40:0/251873986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff194071e60 0x7ff19406f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.247+0000 7ff191400640 1 -- 172.21.15.40:0/251873986 >> v1:172.21.15.40:6789/0 conn(0x7ff194072780 legacy=0x7ff1940cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:09.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.247+0000 7ff191400640 1 -- 172.21.15.40:0/251873986 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19415e060 con 0x7ff19406fe20 2024-04-26T23:02:09.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.248+0000 7ff191400640 1 --2- 172.21.15.40:0/251873986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19406fe20 0x7ff194072240 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff1800097a0 tx=0x7ff18002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f798030a5b8310a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:09.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.248+0000 7ff190a00640 1 -- 172.21.15.40:0/251873986 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180037070 con 0x7ff19406fe20 2024-04-26T23:02:09.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.248+0000 7ff190a00640 1 -- 172.21.15.40:0/251873986 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff180032440 con 0x7ff19406fe20 2024-04-26T23:02:09.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.248+0000 7ff190a00640 1 -- 172.21.15.40:0/251873986 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180032810 con 0x7ff19406fe20 2024-04-26T23:02:09.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.249+0000 7ff198a00640 1 -- 172.21.15.40:0/251873986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19406fe20 msgr2=0x7ff194072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:09.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.249+0000 7ff198a00640 1 --2- 172.21.15.40:0/251873986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19406fe20 0x7ff194072240 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff1800097a0 tx=0x7ff18002a180 comp rx=0 tx=0).stop 2024-04-26T23:02:09.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.249+0000 7ff198a00640 1 -- 172.21.15.40:0/251873986 shutdown_connections 2024-04-26T23:02:09.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.249+0000 7ff198a00640 1 --2- 172.21.15.40:0/251873986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff19406fe20 0x7ff194072240 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.249+0000 7ff198a00640 1 --2- 172.21.15.40:0/251873986 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff194071e60 0x7ff19406f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.249+0000 7ff198a00640 1 -- 172.21.15.40:0/251873986 >> 172.21.15.40:0/251873986 conn(0x7ff1940d0be0 msgr2=0x7ff1940d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:09.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.249+0000 7ff198a00640 1 -- 172.21.15.40:0/251873986 shutdown_connections 2024-04-26T23:02:09.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.249+0000 7ff198a00640 1 -- 172.21.15.40:0/251873986 wait complete. 2024-04-26T23:02:09.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.251+0000 7ff198a00640 1 Processor -- start 2024-04-26T23:02:09.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.251+0000 7ff198a00640 1 -- start start 2024-04-26T23:02:09.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.252+0000 7ff198a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff19406fe20 0x7ff19415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:09.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.252+0000 7ff198a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff194071e60 0x7ff19414ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:09.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.252+0000 7ff198a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff194072780 0x7ff1941590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:09.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.252+0000 7ff198a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff19415d3f0 con 0x7ff194072780 2024-04-26T23:02:09.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.252+0000 7ff198a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff194072c10 con 0x7ff194071e60 2024-04-26T23:02:09.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.252+0000 7ff198a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff19415d570 con 0x7ff19406fe20 2024-04-26T23:02:09.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.252+0000 7ff192800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff194072780 0x7ff1941590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:09.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.252+0000 7ff192800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff194072780 0x7ff1941590c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36320/0 (socket says 172.21.15.40:36320) 2024-04-26T23:02:09.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.252+0000 7ff192800640 1 -- 172.21.15.40:0/1937687927 learned_addr learned my addr 172.21.15.40:0/1937687927 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:09.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.253+0000 7ff191e00640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff19406fe20 0x7ff19415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:09.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.253+0000 7ff191400640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff194071e60 0x7ff19414ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:09.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.253+0000 7ff192800640 1 -- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff19406fe20 msgr2=0x7ff19415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:09.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.253+0000 7ff192800640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff19406fe20 0x7ff19415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.253+0000 7ff192800640 1 -- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff194071e60 msgr2=0x7ff19414ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:09.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.253+0000 7ff192800640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff194071e60 0x7ff19414ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.253+0000 7ff192800640 1 -- 172.21.15.40:0/1937687927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff194159b40 con 0x7ff194072780 2024-04-26T23:02:09.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.253+0000 7ff192800640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff194072780 0x7ff1941590c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff18c00caa0 tx=0x7ff18c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:09.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.254+0000 7ff18a000640 1 -- 172.21.15.40:0/1937687927 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c017070 con 0x7ff194072780 2024-04-26T23:02:09.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.254+0000 7ff18a000640 1 -- 172.21.15.40:0/1937687927 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff18c00ac00 con 0x7ff194072780 2024-04-26T23:02:09.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.254+0000 7ff18a000640 1 -- 172.21.15.40:0/1937687927 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c012400 con 0x7ff194072780 2024-04-26T23:02:09.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.254+0000 7ff198a00640 1 -- 172.21.15.40:0/1937687927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff194159e20 con 0x7ff194072780 2024-04-26T23:02:09.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.254+0000 7ff198a00640 1 -- 172.21.15.40:0/1937687927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19415a3e0 con 0x7ff194072780 2024-04-26T23:02:09.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.255+0000 7ff198a00640 1 -- 172.21.15.40:0/1937687927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1940694f0 con 0x7ff194072780 2024-04-26T23:02:09.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.256+0000 7ff18a000640 1 -- 172.21.15.40:0/1937687927 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff18c01a070 con 0x7ff194072780 2024-04-26T23:02:09.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.257+0000 7ff18a000640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff17406c8f0 0x7ff17406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:09.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.258+0000 7ff18a000640 1 -- 172.21.15.40:0/1937687927 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ff18c08bd60 con 0x7ff194072780 2024-04-26T23:02:09.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.258+0000 7ff191e00640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff17406c8f0 0x7ff17406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:09.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.259+0000 7ff191e00640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff17406c8f0 0x7ff17406edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff184006fd0 tx=0x7ff184008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:09.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.262+0000 7ff18a000640 1 -- 172.21.15.40:0/1937687927 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff18c059cf0 con 0x7ff194072780 2024-04-26T23:02:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.463+0000 7ff198a00640 1 -- 172.21.15.40:0/1937687927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff194160570 con 0x7ff194072780 2024-04-26T23:02:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.464+0000 7ff18a000640 1 -- 172.21.15.40:0/1937687927 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7ff18c0595f0 con 0x7ff194072780 2024-04-26T23:02:09.464 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:02:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.466+0000 7ff198a00640 1 -- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff17406c8f0 msgr2=0x7ff17406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.466+0000 7ff198a00640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff17406c8f0 0x7ff17406edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff184006fd0 tx=0x7ff184008040 comp rx=0 tx=0).stop 2024-04-26T23:02:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.466+0000 7ff198a00640 1 -- 172.21.15.40:0/1937687927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff194072780 msgr2=0x7ff1941590c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.466+0000 7ff198a00640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff194072780 0x7ff1941590c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff18c00caa0 tx=0x7ff18c00cf60 comp rx=0 tx=0).stop 2024-04-26T23:02:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.466+0000 7ff198a00640 1 -- 172.21.15.40:0/1937687927 shutdown_connections 2024-04-26T23:02:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.466+0000 7ff198a00640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff17406c8f0 0x7ff17406edb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.466+0000 7ff198a00640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff194072780 0x7ff1941590c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.466+0000 7ff198a00640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff194071e60 0x7ff19414ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.466+0000 7ff198a00640 1 --2- 172.21.15.40:0/1937687927 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff19406fe20 0x7ff19415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.466+0000 7ff198a00640 1 -- 172.21.15.40:0/1937687927 >> 172.21.15.40:0/1937687927 conn(0x7ff1940d0be0 msgr2=0x7ff1940cb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.466+0000 7ff198a00640 1 -- 172.21.15.40:0/1937687927 shutdown_connections 2024-04-26T23:02:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.467+0000 7ff198a00640 1 -- 172.21.15.40:0/1937687927 wait complete. 2024-04-26T23:02:09.485 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:02:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.695+0000 7fcb18e00640 1 Processor -- start 2024-04-26T23:02:09.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.696+0000 7fcb18e00640 1 -- start start 2024-04-26T23:02:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.697+0000 7fcb18e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcb14073600 0x7fcb14072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.697+0000 7fcb18e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb140739e0 0x7fcb1415f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.698+0000 7fcb18e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcb1415fcb0 0x7fcb140ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.698+0000 7fcb18e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fcb14073220 con 0x7fcb1415fcb0 2024-04-26T23:02:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.698+0000 7fcb18e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcb14072850 con 0x7fcb14073600 2024-04-26T23:02:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.698+0000 7fcb18e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcb141604e0 con 0x7fcb140739e0 2024-04-26T23:02:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.698+0000 7fcb13e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcb1415fcb0 0x7fcb140ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.698+0000 7fcb13e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcb1415fcb0 0x7fcb140ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36336/0 (socket says 172.21.15.40:36336) 2024-04-26T23:02:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.698+0000 7fcb13e00640 1 -- 172.21.15.40:0/663762575 learned_addr learned my addr 172.21.15.40:0/663762575 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.698+0000 7fcb12a00640 1 --2- 172.21.15.40:0/663762575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb140739e0 0x7fcb1415f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.698+0000 7fcb13400640 1 --2- 172.21.15.40:0/663762575 >> v2:172.21.15.103:3300/0 conn(0x7fcb14073600 0x7fcb14072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.698+0000 7fcb13e00640 1 -- 172.21.15.40:0/663762575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb140739e0 msgr2=0x7fcb1415f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.698+0000 7fcb13e00640 1 --2- 172.21.15.40:0/663762575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb140739e0 0x7fcb1415f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.698+0000 7fcb13e00640 1 -- 172.21.15.40:0/663762575 >> v2:172.21.15.103:3300/0 conn(0x7fcb14073600 msgr2=0x7fcb14072ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.699+0000 7fcb13e00640 1 --2- 172.21.15.40:0/663762575 >> v2:172.21.15.103:3300/0 conn(0x7fcb14073600 0x7fcb14072ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.699+0000 7fcb13e00640 1 -- 172.21.15.40:0/663762575 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb14072200 con 0x7fcb1415fcb0 2024-04-26T23:02:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.699+0000 7fcb13400640 1 --2- 172.21.15.40:0/663762575 >> v2:172.21.15.103:3300/0 conn(0x7fcb14073600 0x7fcb14072ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:09.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.700+0000 7fcb13e00640 1 --2- 172.21.15.40:0/663762575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb1415fcb0 0x7fcb140ae8c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcb1406dac0 tx=0x7fcb0c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd00ff4b44f08f73 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:09.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.700+0000 7fcb12000640 1 -- 172.21.15.40:0/663762575 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c036070 con 0x7fcb1415fcb0 2024-04-26T23:02:09.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.700+0000 7fcb12000640 1 -- 172.21.15.40:0/663762575 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb0c031600 con 0x7fcb1415fcb0 2024-04-26T23:02:09.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.700+0000 7fcb12000640 1 -- 172.21.15.40:0/663762575 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c0319d0 con 0x7fcb1415fcb0 2024-04-26T23:02:09.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.701+0000 7fcb18e00640 1 -- 172.21.15.40:0/663762575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb1415fcb0 msgr2=0x7fcb140ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:09.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.701+0000 7fcb18e00640 1 --2- 172.21.15.40:0/663762575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb1415fcb0 0x7fcb140ae8c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcb1406dac0 tx=0x7fcb0c02a6a0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.701+0000 7fcb18e00640 1 -- 172.21.15.40:0/663762575 shutdown_connections 2024-04-26T23:02:09.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.701+0000 7fcb18e00640 1 --2- 172.21.15.40:0/663762575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb1415fcb0 0x7fcb140ae8c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.701+0000 7fcb18e00640 1 --2- 172.21.15.40:0/663762575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb140739e0 0x7fcb1415f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.701+0000 7fcb18e00640 1 --2- 172.21.15.40:0/663762575 >> v2:172.21.15.103:3300/0 conn(0x7fcb14073600 0x7fcb14072ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.701+0000 7fcb18e00640 1 -- 172.21.15.40:0/663762575 >> 172.21.15.40:0/663762575 conn(0x7fcb140b2720 msgr2=0x7fcb140b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:09.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.701+0000 7fcb18e00640 1 -- 172.21.15.40:0/663762575 shutdown_connections 2024-04-26T23:02:09.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.701+0000 7fcb18e00640 1 -- 172.21.15.40:0/663762575 wait complete. 2024-04-26T23:02:09.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.703+0000 7fcb18e00640 1 Processor -- start 2024-04-26T23:02:09.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.703+0000 7fcb18e00640 1 -- start start 2024-04-26T23:02:09.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.704+0000 7fcb18e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb14073600 0x7fcb14148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:09.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.704+0000 7fcb18e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb140739e0 0x7fcb14156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:09.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.704+0000 7fcb18e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1415fcb0 0x7fcb14160e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:09.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.704+0000 7fcb18e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcb14071e20 con 0x7fcb14073600 2024-04-26T23:02:09.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.704+0000 7fcb18e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcb14073e60 con 0x7fcb140739e0 2024-04-26T23:02:09.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.704+0000 7fcb18e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcb14071fa0 con 0x7fcb1415fcb0 2024-04-26T23:02:09.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.704+0000 7fcb13400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb14073600 0x7fcb14148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:09.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.704+0000 7fcb13400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb14073600 0x7fcb14148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36340/0 (socket says 172.21.15.40:36340) 2024-04-26T23:02:09.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.704+0000 7fcb13400640 1 -- 172.21.15.40:0/3839328577 learned_addr learned my addr 172.21.15.40:0/3839328577 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:09.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.705+0000 7fcb13400640 1 -- 172.21.15.40:0/3839328577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1415fcb0 msgr2=0x7fcb14160e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:09.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.705+0000 7fcb13400640 1 --2- 172.21.15.40:0/3839328577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1415fcb0 0x7fcb14160e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.705+0000 7fcb13400640 1 -- 172.21.15.40:0/3839328577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb140739e0 msgr2=0x7fcb14156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:09.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.705+0000 7fcb13400640 1 --2- 172.21.15.40:0/3839328577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb140739e0 0x7fcb14156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.705+0000 7fcb13400640 1 -- 172.21.15.40:0/3839328577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb14157990 con 0x7fcb14073600 2024-04-26T23:02:09.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.705+0000 7fcb13400640 1 --2- 172.21.15.40:0/3839328577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb14073600 0x7fcb14148610 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400c7c0 tx=0x7fcb0400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:09.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.705+0000 7fcb0b400640 1 -- 172.21.15.40:0/3839328577 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04015070 con 0x7fcb14073600 2024-04-26T23:02:09.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.705+0000 7fcb0b400640 1 -- 172.21.15.40:0/3839328577 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb04007a30 con 0x7fcb14073600 2024-04-26T23:02:09.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.705+0000 7fcb18e00640 1 -- 172.21.15.40:0/3839328577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb14157450 con 0x7fcb14073600 2024-04-26T23:02:09.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.705+0000 7fcb0b400640 1 -- 172.21.15.40:0/3839328577 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04007e00 con 0x7fcb14073600 2024-04-26T23:02:09.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.705+0000 7fcb18e00640 1 -- 172.21.15.40:0/3839328577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb140cd580 con 0x7fcb14073600 2024-04-26T23:02:09.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.707+0000 7fcb18e00640 1 -- 172.21.15.40:0/3839328577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb14069370 con 0x7fcb14073600 2024-04-26T23:02:09.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.708+0000 7fcb0b400640 1 -- 172.21.15.40:0/3839328577 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcb04018070 con 0x7fcb14073600 2024-04-26T23:02:09.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.709+0000 7fcb0b400640 1 --2- 172.21.15.40:0/3839328577 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcaf006c8f0 0x7fcaf006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:09.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.709+0000 7fcb0b400640 1 -- 172.21.15.40:0/3839328577 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fcb0408be60 con 0x7fcb14073600 2024-04-26T23:02:09.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.709+0000 7fcb12a00640 1 --2- 172.21.15.40:0/3839328577 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcaf006c8f0 0x7fcaf006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:09.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.711+0000 7fcb12a00640 1 --2- 172.21.15.40:0/3839328577 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcaf006c8f0 0x7fcaf006edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcafc006fd0 tx=0x7fcafc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:09.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.714+0000 7fcb0b400640 1 -- 172.21.15.40:0/3839328577 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fcb04059e80 con 0x7fcb14073600 2024-04-26T23:02:09.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.922+0000 7fcb18e00640 1 -- 172.21.15.40:0/3839328577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcb140b3370 con 0x7fcb14073600 2024-04-26T23:02:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.923+0000 7fcb0b400640 1 -- 172.21.15.40:0/3839328577 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fcb04059780 con 0x7fcb14073600 2024-04-26T23:02:09.924 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.926+0000 7fcb18e00640 1 -- 172.21.15.40:0/3839328577 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcaf006c8f0 msgr2=0x7fcaf006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.926+0000 7fcb18e00640 1 --2- 172.21.15.40:0/3839328577 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcaf006c8f0 0x7fcaf006edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcafc006fd0 tx=0x7fcafc008040 comp rx=0 tx=0).stop 2024-04-26T23:02:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.926+0000 7fcb18e00640 1 -- 172.21.15.40:0/3839328577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb14073600 msgr2=0x7fcb14148610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.926+0000 7fcb18e00640 1 --2- 172.21.15.40:0/3839328577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb14073600 0x7fcb14148610 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400c7c0 tx=0x7fcb0400cc80 comp rx=0 tx=0).stop 2024-04-26T23:02:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.926+0000 7fcb18e00640 1 -- 172.21.15.40:0/3839328577 shutdown_connections 2024-04-26T23:02:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.926+0000 7fcb18e00640 1 --2- 172.21.15.40:0/3839328577 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcaf006c8f0 0x7fcaf006edb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.926+0000 7fcb18e00640 1 --2- 172.21.15.40:0/3839328577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1415fcb0 0x7fcb14160e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.926+0000 7fcb18e00640 1 --2- 172.21.15.40:0/3839328577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb140739e0 0x7fcb14156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.926+0000 7fcb18e00640 1 --2- 172.21.15.40:0/3839328577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb14073600 0x7fcb14148610 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.926+0000 7fcb18e00640 1 -- 172.21.15.40:0/3839328577 >> 172.21.15.40:0/3839328577 conn(0x7fcb140b2720 msgr2=0x7fcb1415b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.926+0000 7fcb18e00640 1 -- 172.21.15.40:0/3839328577 shutdown_connections 2024-04-26T23:02:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:09.926+0000 7fcb18e00640 1 -- 172.21.15.40:0/3839328577 wait complete. 2024-04-26T23:02:09.946 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:02:10.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.162+0000 7fba6d400640 1 Processor -- start 2024-04-26T23:02:10.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.162+0000 7fba6d400640 1 -- start start 2024-04-26T23:02:10.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.163+0000 7fba6d400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fba68072950 0x7fba68167ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.164+0000 7fba6d400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba6815d7c0 con 0x7fba680cc480 2024-04-26T23:02:10.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.164+0000 7fba6d400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba6815daa0 con 0x7fba68071e60 2024-04-26T23:02:10.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.164+0000 7fba6d400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fba6815dd80 con 0x7fba68072950 2024-04-26T23:02:10.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.164+0000 7fba65e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fba680cc480 0x7fba68072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52048/0 (socket says 172.21.15.40:52048) 2024-04-26T23:02:10.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.164+0000 7fba65e00640 1 -- 172.21.15.40:0/3299463383 learned_addr learned my addr 172.21.15.40:0/3299463383 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:10.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.164+0000 7fba67200640 1 --2- 172.21.15.40:0/3299463383 >> v2:172.21.15.103:3300/0 conn(0x7fba68072950 0x7fba68167ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2023925352 0 0) 0x7fba6815d7c0 con 0x7fba680cc480 2024-04-26T23:02:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba4c003520 con 0x7fba680cc480 2024-04-26T23:02:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1513874453 0 0) 0x7fba4c003520 con 0x7fba680cc480 2024-04-26T23:02:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 >> v2:172.21.15.103:3300/0 conn(0x7fba68072950 msgr2=0x7fba68167ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 --2- 172.21.15.40:0/3299463383 >> v2:172.21.15.103:3300/0 conn(0x7fba68072950 0x7fba68167ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 >> v1:172.21.15.103:6789/0 conn(0x7fba68071e60 legacy=0x7fba6806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6815e060 con 0x7fba680cc480 2024-04-26T23:02:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fba54001670 con 0x7fba680cc480 2024-04-26T23:02:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 >> v1:172.21.15.40:6789/0 conn(0x7fba680cc480 legacy=0x7fba68072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 --2- 172.21.15.40:0/3299463383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba4c0039b0 0x7fba4c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 --2- 172.21.15.40:0/3299463383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba4c004800 0x7fba4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 --2- 172.21.15.40:0/3299463383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba4c0051c0 0x7fba4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fba6815dd80 con 0x7fba4c0039b0 2024-04-26T23:02:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c00c040 con 0x7fba4c0051c0 2024-04-26T23:02:10.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.165+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fba4c00c240 con 0x7fba4c004800 2024-04-26T23:02:10.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.166+0000 7fba66800640 1 --2- 172.21.15.40:0/3299463383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba4c0039b0 0x7fba4c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.166+0000 7fba65e00640 1 --2- 172.21.15.40:0/3299463383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba4c004800 0x7fba4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.166+0000 7fba67200640 1 --2- 172.21.15.40:0/3299463383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba4c0051c0 0x7fba4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.166+0000 7fba66800640 1 -- 172.21.15.40:0/3299463383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba4c004800 msgr2=0x7fba4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:10.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.166+0000 7fba66800640 1 --2- 172.21.15.40:0/3299463383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba4c004800 0x7fba4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.166+0000 7fba66800640 1 -- 172.21.15.40:0/3299463383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba4c0051c0 msgr2=0x7fba4c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:10.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.166+0000 7fba66800640 1 --2- 172.21.15.40:0/3299463383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba4c0051c0 0x7fba4c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.166+0000 7fba66800640 1 -- 172.21.15.40:0/3299463383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4c003520 con 0x7fba4c0039b0 2024-04-26T23:02:10.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.166+0000 7fba67200640 1 --2- 172.21.15.40:0/3299463383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba4c0051c0 0x7fba4c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:10.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.166+0000 7fba67200640 1 -- 172.21.15.40:0/3299463383 reap_dead start 2024-04-26T23:02:10.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.167+0000 7fba66800640 1 --2- 172.21.15.40:0/3299463383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba4c0039b0 0x7fba4c003db0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fba580044b0 tx=0x7fba4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=910e2a70add2365a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:10.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.168+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58037070 con 0x7fba4c0039b0 2024-04-26T23:02:10.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.168+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba5802ad60 con 0x7fba4c0039b0 2024-04-26T23:02:10.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.168+0000 7fba65400640 1 -- 172.21.15.40:0/3299463383 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58004080 con 0x7fba4c0039b0 2024-04-26T23:02:10.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.168+0000 7fba6d400640 1 -- 172.21.15.40:0/3299463383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba4c0039b0 msgr2=0x7fba4c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:10.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.168+0000 7fba6d400640 1 --2- 172.21.15.40:0/3299463383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba4c0039b0 0x7fba4c003db0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fba580044b0 tx=0x7fba4c005830 comp rx=0 tx=0).stop 2024-04-26T23:02:10.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.168+0000 7fba6d400640 1 -- 172.21.15.40:0/3299463383 shutdown_connections 2024-04-26T23:02:10.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.168+0000 7fba6d400640 1 --2- 172.21.15.40:0/3299463383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba4c0039b0 0x7fba4c003db0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.168+0000 7fba6d400640 1 -- 172.21.15.40:0/3299463383 >> 172.21.15.40:0/3299463383 conn(0x7fba680d0be0 msgr2=0x7fba680d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:10.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.169+0000 7fba6d400640 1 -- 172.21.15.40:0/3299463383 shutdown_connections 2024-04-26T23:02:10.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.169+0000 7fba6d400640 1 -- 172.21.15.40:0/3299463383 wait complete. 2024-04-26T23:02:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.171+0000 7fba6d400640 1 Processor -- start 2024-04-26T23:02:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.171+0000 7fba6d400640 1 -- start start 2024-04-26T23:02:10.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.172+0000 7fba6d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681f9110 0x7fba680b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.172+0000 7fba6d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba680b6650 0x7fba68157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.172+0000 7fba6d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba681f94f0 0x7fba68158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.172+0000 7fba6d400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fba6815d570 con 0x7fba681f9110 2024-04-26T23:02:10.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.172+0000 7fba6d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fba6815d3f0 con 0x7fba680b6650 2024-04-26T23:02:10.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.172+0000 7fba6d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fba68072240 con 0x7fba681f94f0 2024-04-26T23:02:10.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.172+0000 7fba66800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681f9110 0x7fba680b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.172+0000 7fba66800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681f9110 0x7fba680b6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36372/0 (socket says 172.21.15.40:36372) 2024-04-26T23:02:10.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.172+0000 7fba66800640 1 -- 172.21.15.40:0/152940945 learned_addr learned my addr 172.21.15.40:0/152940945 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:10.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.172+0000 7fba65e00640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba680b6650 0x7fba68157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.173+0000 7fba67200640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba681f94f0 0x7fba68158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.173+0000 7fba66800640 1 -- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba681f94f0 msgr2=0x7fba68158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:10.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.173+0000 7fba66800640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba681f94f0 0x7fba68158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.173+0000 7fba66800640 1 -- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba680b6650 msgr2=0x7fba68157d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:10.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.173+0000 7fba66800640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba680b6650 0x7fba68157d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.173+0000 7fba66800640 1 -- 172.21.15.40:0/152940945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba680cf360 con 0x7fba681f9110 2024-04-26T23:02:10.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.173+0000 7fba66800640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681f9110 0x7fba680b6110 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fba58000f80 tx=0x7fba58003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:10.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.173+0000 7fba5ea00640 1 -- 172.21.15.40:0/152940945 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58037050 con 0x7fba681f9110 2024-04-26T23:02:10.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.173+0000 7fba5ea00640 1 -- 172.21.15.40:0/152940945 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba58038040 con 0x7fba681f9110 2024-04-26T23:02:10.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.173+0000 7fba5ea00640 1 -- 172.21.15.40:0/152940945 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58007df0 con 0x7fba681f9110 2024-04-26T23:02:10.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.174+0000 7fba6d400640 1 -- 172.21.15.40:0/152940945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba680cf600 con 0x7fba681f9110 2024-04-26T23:02:10.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.174+0000 7fba6d400640 1 -- 172.21.15.40:0/152940945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba680cfbc0 con 0x7fba681f9110 2024-04-26T23:02:10.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.175+0000 7fba6d400640 1 -- 172.21.15.40:0/152940945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba680725d0 con 0x7fba681f9110 2024-04-26T23:02:10.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.176+0000 7fba5ea00640 1 -- 172.21.15.40:0/152940945 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fba58045020 con 0x7fba681f9110 2024-04-26T23:02:10.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.177+0000 7fba5ea00640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fba4806c8f0 0x7fba4806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.177+0000 7fba5ea00640 1 -- 172.21.15.40:0/152940945 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fba580abfd0 con 0x7fba681f9110 2024-04-26T23:02:10.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.177+0000 7fba65e00640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fba4806c8f0 0x7fba4806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.179+0000 7fba65e00640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fba4806c8f0 0x7fba4806edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fba680cb270 tx=0x7fba54009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:10.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.182+0000 7fba5ea00640 1 -- 172.21.15.40:0/152940945 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fba680725d0 con 0x7fba681f9110 2024-04-26T23:02:10.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.384+0000 7fba6d400640 1 -- 172.21.15.40:0/152940945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fba680b19d0 con 0x7fba681f9110 2024-04-26T23:02:10.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.385+0000 7fba5ea00640 1 -- 172.21.15.40:0/152940945 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fba680b19d0 con 0x7fba681f9110 2024-04-26T23:02:10.386 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:02:10.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.388+0000 7fba6d400640 1 -- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fba4806c8f0 msgr2=0x7fba4806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:10.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.388+0000 7fba6d400640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fba4806c8f0 0x7fba4806edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fba680cb270 tx=0x7fba54009040 comp rx=0 tx=0).stop 2024-04-26T23:02:10.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.388+0000 7fba6d400640 1 -- 172.21.15.40:0/152940945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681f9110 msgr2=0x7fba680b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:10.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.388+0000 7fba6d400640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681f9110 0x7fba680b6110 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fba58000f80 tx=0x7fba58003a70 comp rx=0 tx=0).stop 2024-04-26T23:02:10.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.388+0000 7fba6d400640 1 -- 172.21.15.40:0/152940945 shutdown_connections 2024-04-26T23:02:10.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.388+0000 7fba6d400640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fba4806c8f0 0x7fba4806edb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.388+0000 7fba6d400640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba681f94f0 0x7fba68158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.388+0000 7fba6d400640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba680b6650 0x7fba68157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.388+0000 7fba6d400640 1 --2- 172.21.15.40:0/152940945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba681f9110 0x7fba680b6110 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.388+0000 7fba6d400640 1 -- 172.21.15.40:0/152940945 >> 172.21.15.40:0/152940945 conn(0x7fba680d0be0 msgr2=0x7fba680afa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:10.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.388+0000 7fba6d400640 1 -- 172.21.15.40:0/152940945 shutdown_connections 2024-04-26T23:02:10.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.388+0000 7fba6d400640 1 -- 172.21.15.40:0/152940945 wait complete. 2024-04-26T23:02:10.406 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:02:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.623+0000 7fc44b000640 1 Processor -- start 2024-04-26T23:02:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.623+0000 7fc44b000640 1 -- start start 2024-04-26T23:02:10.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.625+0000 7fc44b000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc444160a50 0x7fc444073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.625+0000 7fc44b000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc4440cfaf0 0x7fc444160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.625+0000 7fc44b000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc44406dd40 0x7fc444072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.625+0000 7fc44b000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc444071ce0 con 0x7fc44406dd40 2024-04-26T23:02:10.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.625+0000 7fc44b000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc4440722a0 con 0x7fc4440cfaf0 2024-04-26T23:02:10.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.625+0000 7fc44b000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc444073f90 con 0x7fc444160a50 2024-04-26T23:02:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.625+0000 7fc44a600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc44406dd40 0x7fc444072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.626+0000 7fc44a600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc44406dd40 0x7fc444072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36388/0 (socket says 172.21.15.40:36388) 2024-04-26T23:02:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.626+0000 7fc44a600640 1 -- 172.21.15.40:0/1396434868 learned_addr learned my addr 172.21.15.40:0/1396434868 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.626+0000 7fc449c00640 1 --2- 172.21.15.40:0/1396434868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc444160a50 0x7fc444073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.626+0000 7fc449200640 1 --2- 172.21.15.40:0/1396434868 >> v2:172.21.15.103:3300/0 conn(0x7fc4440cfaf0 0x7fc444160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.626+0000 7fc44a600640 1 -- 172.21.15.40:0/1396434868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc444160a50 msgr2=0x7fc444073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.626+0000 7fc44a600640 1 --2- 172.21.15.40:0/1396434868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc444160a50 0x7fc444073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.626+0000 7fc44a600640 1 -- 172.21.15.40:0/1396434868 >> v2:172.21.15.103:3300/0 conn(0x7fc4440cfaf0 msgr2=0x7fc444160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.626+0000 7fc44a600640 1 --2- 172.21.15.40:0/1396434868 >> v2:172.21.15.103:3300/0 conn(0x7fc4440cfaf0 0x7fc444160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.626+0000 7fc44a600640 1 -- 172.21.15.40:0/1396434868 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc444161880 con 0x7fc44406dd40 2024-04-26T23:02:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.626+0000 7fc449c00640 1 --2- 172.21.15.40:0/1396434868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc444160a50 0x7fc444073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.626+0000 7fc449200640 1 --2- 172.21.15.40:0/1396434868 >> v2:172.21.15.103:3300/0 conn(0x7fc4440cfaf0 0x7fc444160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.627+0000 7fc44a600640 1 --2- 172.21.15.40:0/1396434868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44406dd40 0x7fc444072730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc444074700 tx=0x7fc43c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b963016fc05d85ba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.627+0000 7fc443e00640 1 -- 172.21.15.40:0/1396434868 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c036070 con 0x7fc44406dd40 2024-04-26T23:02:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.628+0000 7fc443e00640 1 -- 172.21.15.40:0/1396434868 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc43c031600 con 0x7fc44406dd40 2024-04-26T23:02:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.628+0000 7fc443e00640 1 -- 172.21.15.40:0/1396434868 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c0319d0 con 0x7fc44406dd40 2024-04-26T23:02:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.628+0000 7fc44b000640 1 -- 172.21.15.40:0/1396434868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44406dd40 msgr2=0x7fc444072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.628+0000 7fc44b000640 1 --2- 172.21.15.40:0/1396434868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44406dd40 0x7fc444072730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc444074700 tx=0x7fc43c02a6a0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.628+0000 7fc44b000640 1 -- 172.21.15.40:0/1396434868 shutdown_connections 2024-04-26T23:02:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.628+0000 7fc44b000640 1 --2- 172.21.15.40:0/1396434868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44406dd40 0x7fc444072730 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.628+0000 7fc44b000640 1 --2- 172.21.15.40:0/1396434868 >> v2:172.21.15.103:3300/0 conn(0x7fc4440cfaf0 0x7fc444160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.628+0000 7fc44b000640 1 --2- 172.21.15.40:0/1396434868 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc444160a50 0x7fc444073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.628+0000 7fc44b000640 1 -- 172.21.15.40:0/1396434868 >> 172.21.15.40:0/1396434868 conn(0x7fc4440b2720 msgr2=0x7fc4440b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.628+0000 7fc44b000640 1 -- 172.21.15.40:0/1396434868 shutdown_connections 2024-04-26T23:02:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.628+0000 7fc44b000640 1 -- 172.21.15.40:0/1396434868 wait complete. 2024-04-26T23:02:10.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.630+0000 7fc44b000640 1 Processor -- start 2024-04-26T23:02:10.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.630+0000 7fc44b000640 1 -- start start 2024-04-26T23:02:10.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.631+0000 7fc44b000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44406dd40 0x7fc44415a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.631+0000 7fc44b000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc444160a50 0x7fc44413caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.631+0000 7fc44b000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc44415cc90 0x7fc44415f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.631+0000 7fc44b000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc4440d0260 con 0x7fc44406dd40 2024-04-26T23:02:10.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.631+0000 7fc44b000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc4440cff80 con 0x7fc44415cc90 2024-04-26T23:02:10.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.631+0000 7fc44b000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc444071f30 con 0x7fc444160a50 2024-04-26T23:02:10.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.631+0000 7fc449c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44406dd40 0x7fc44415a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.631+0000 7fc449c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44406dd40 0x7fc44415a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36394/0 (socket says 172.21.15.40:36394) 2024-04-26T23:02:10.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.631+0000 7fc449c00640 1 -- 172.21.15.40:0/1296894103 learned_addr learned my addr 172.21.15.40:0/1296894103 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:10.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.632+0000 7fc449200640 1 --2- 172.21.15.40:0/1296894103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc444160a50 0x7fc44413caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.632+0000 7fc449c00640 1 -- 172.21.15.40:0/1296894103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc444160a50 msgr2=0x7fc44413caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:10.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.632+0000 7fc449c00640 1 --2- 172.21.15.40:0/1296894103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc444160a50 0x7fc44413caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.632+0000 7fc449c00640 1 -- 172.21.15.40:0/1296894103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc44415cc90 msgr2=0x7fc44415f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:10.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.632+0000 7fc449c00640 1 --2- 172.21.15.40:0/1296894103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc44415cc90 0x7fc44415f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.632+0000 7fc449c00640 1 -- 172.21.15.40:0/1296894103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44413dcf0 con 0x7fc44406dd40 2024-04-26T23:02:10.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.632+0000 7fc449c00640 1 --2- 172.21.15.40:0/1296894103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44406dd40 0x7fc44415a680 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c7f0 tx=0x7fc43800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:10.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.632+0000 7fc441600640 1 -- 172.21.15.40:0/1296894103 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438015070 con 0x7fc44406dd40 2024-04-26T23:02:10.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.632+0000 7fc441600640 1 -- 172.21.15.40:0/1296894103 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4380077c0 con 0x7fc44406dd40 2024-04-26T23:02:10.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.632+0000 7fc441600640 1 -- 172.21.15.40:0/1296894103 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438007b70 con 0x7fc44406dd40 2024-04-26T23:02:10.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.633+0000 7fc44b000640 1 -- 172.21.15.40:0/1296894103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc44413dfd0 con 0x7fc44406dd40 2024-04-26T23:02:10.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.633+0000 7fc44b000640 1 -- 172.21.15.40:0/1296894103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc44413d9b0 con 0x7fc44406dd40 2024-04-26T23:02:10.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.634+0000 7fc44b000640 1 -- 172.21.15.40:0/1296894103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc444069370 con 0x7fc44406dd40 2024-04-26T23:02:10.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.635+0000 7fc441600640 1 -- 172.21.15.40:0/1296894103 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc438018070 con 0x7fc44406dd40 2024-04-26T23:02:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.636+0000 7fc441600640 1 --2- 172.21.15.40:0/1296894103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc42406c920 0x7fc42406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.636+0000 7fc441600640 1 -- 172.21.15.40:0/1296894103 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc43808bf60 con 0x7fc44406dd40 2024-04-26T23:02:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.637+0000 7fc449200640 1 --2- 172.21.15.40:0/1296894103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc42406c920 0x7fc42406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:10.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.638+0000 7fc449200640 1 --2- 172.21.15.40:0/1296894103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc42406c920 0x7fc42406ede0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc430006fd0 tx=0x7fc430008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:10.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.640+0000 7fc441600640 1 -- 172.21.15.40:0/1296894103 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc438059f80 con 0x7fc44406dd40 2024-04-26T23:02:10.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.844+0000 7fc44b000640 1 -- 172.21.15.40:0/1296894103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc44406fa60 con 0x7fc44406dd40 2024-04-26T23:02:10.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.845+0000 7fc441600640 1 -- 172.21.15.40:0/1296894103 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fc438059880 con 0x7fc44406dd40 2024-04-26T23:02:10.846 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:10.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.848+0000 7fc44b000640 1 -- 172.21.15.40:0/1296894103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc42406c920 msgr2=0x7fc42406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:10.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.848+0000 7fc44b000640 1 --2- 172.21.15.40:0/1296894103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc42406c920 0x7fc42406ede0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc430006fd0 tx=0x7fc430008040 comp rx=0 tx=0).stop 2024-04-26T23:02:10.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.848+0000 7fc44b000640 1 -- 172.21.15.40:0/1296894103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44406dd40 msgr2=0x7fc44415a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:10.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.848+0000 7fc44b000640 1 --2- 172.21.15.40:0/1296894103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44406dd40 0x7fc44415a680 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c7f0 tx=0x7fc43800ccb0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.848+0000 7fc44b000640 1 -- 172.21.15.40:0/1296894103 shutdown_connections 2024-04-26T23:02:10.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.848+0000 7fc44b000640 1 --2- 172.21.15.40:0/1296894103 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc42406c920 0x7fc42406ede0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.848+0000 7fc44b000640 1 --2- 172.21.15.40:0/1296894103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc44415cc90 0x7fc44415f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.848+0000 7fc44b000640 1 --2- 172.21.15.40:0/1296894103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc444160a50 0x7fc44413caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.848+0000 7fc44b000640 1 --2- 172.21.15.40:0/1296894103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc44406dd40 0x7fc44415a680 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:10.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.848+0000 7fc44b000640 1 -- 172.21.15.40:0/1296894103 >> 172.21.15.40:0/1296894103 conn(0x7fc4440b2720 msgr2=0x7fc4440ae950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:10.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.849+0000 7fc44b000640 1 -- 172.21.15.40:0/1296894103 shutdown_connections 2024-04-26T23:02:10.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:10.849+0000 7fc44b000640 1 -- 172.21.15.40:0/1296894103 wait complete. 2024-04-26T23:02:10.867 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:11.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.081+0000 7fd595400640 1 Processor -- start 2024-04-26T23:02:11.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.081+0000 7fd595400640 1 -- start start 2024-04-26T23:02:11.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.083+0000 7fd595400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd59015f1d0 0x7fd590070a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:11.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.083+0000 7fd595400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5900701e0 con 0x7fd5900d2c30 2024-04-26T23:02:11.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.083+0000 7fd595400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5900704c0 con 0x7fd590163610 2024-04-26T23:02:11.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.083+0000 7fd595400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fd59015f5b0 con 0x7fd59015f1d0 2024-04-26T23:02:11.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.083+0000 7fd58fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd59015f1d0 0x7fd590070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:11.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.083+0000 7fd594a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd5900d2c30 0x7fd59006e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52060/0 (socket says 172.21.15.40:52060) 2024-04-26T23:02:11.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.083+0000 7fd594a00640 1 -- 172.21.15.40:0/2698171944 learned_addr learned my addr 172.21.15.40:0/2698171944 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:11.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.084+0000 7fd58fe00640 1 -- 172.21.15.40:0/2698171944 >> v1:172.21.15.103:6789/0 conn(0x7fd590163610 legacy=0x7fd5900d0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:11.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.084+0000 7fd58fe00640 1 -- 172.21.15.40:0/2698171944 >> v1:172.21.15.40:6789/0 conn(0x7fd5900d2c30 legacy=0x7fd59006e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:11.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.084+0000 7fd58fe00640 1 -- 172.21.15.40:0/2698171944 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5900707a0 con 0x7fd59015f1d0 2024-04-26T23:02:11.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.085+0000 7fd58fe00640 1 --2- 172.21.15.40:0/2698171944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd59015f1d0 0x7fd590070a80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd580009d90 tx=0x7fd58002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ace3679257cfd4e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:11.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.085+0000 7fd58ea00640 1 -- 172.21.15.40:0/2698171944 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580031600 con 0x7fd59015f1d0 2024-04-26T23:02:11.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.085+0000 7fd58ea00640 1 -- 172.21.15.40:0/2698171944 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5800317a0 con 0x7fd59015f1d0 2024-04-26T23:02:11.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.085+0000 7fd58ea00640 1 -- 172.21.15.40:0/2698171944 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580031b70 con 0x7fd59015f1d0 2024-04-26T23:02:11.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.086+0000 7fd595400640 1 -- 172.21.15.40:0/2698171944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd59015f1d0 msgr2=0x7fd590070a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:11.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.086+0000 7fd595400640 1 --2- 172.21.15.40:0/2698171944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd59015f1d0 0x7fd590070a80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd580009d90 tx=0x7fd58002a610 comp rx=0 tx=0).stop 2024-04-26T23:02:11.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.086+0000 7fd595400640 1 -- 172.21.15.40:0/2698171944 shutdown_connections 2024-04-26T23:02:11.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.086+0000 7fd595400640 1 --2- 172.21.15.40:0/2698171944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd59015f1d0 0x7fd590070a80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.086+0000 7fd595400640 1 -- 172.21.15.40:0/2698171944 >> 172.21.15.40:0/2698171944 conn(0x7fd59009ba50 msgr2=0x7fd59009be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:11.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.086+0000 7fd595400640 1 -- 172.21.15.40:0/2698171944 shutdown_connections 2024-04-26T23:02:11.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.086+0000 7fd595400640 1 -- 172.21.15.40:0/2698171944 wait complete. 2024-04-26T23:02:11.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.088+0000 7fd595400640 1 Processor -- start 2024-04-26T23:02:11.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.088+0000 7fd595400640 1 -- start start 2024-04-26T23:02:11.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.089+0000 7fd595400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5900d2c30 0x7fd59015dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:11.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.089+0000 7fd595400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59015f1d0 0x7fd5901e9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:11.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.089+0000 7fd595400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd590163610 0x7fd5901f6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:11.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.089+0000 7fd595400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd590163a90 con 0x7fd5900d2c30 2024-04-26T23:02:11.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.089+0000 7fd595400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd59006da80 con 0x7fd59015f1d0 2024-04-26T23:02:11.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.089+0000 7fd595400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd5900d1580 con 0x7fd590163610 2024-04-26T23:02:11.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.089+0000 7fd58fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5900d2c30 0x7fd59015dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:11.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.089+0000 7fd58fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5900d2c30 0x7fd59015dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36418/0 (socket says 172.21.15.40:36418) 2024-04-26T23:02:11.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.089+0000 7fd58fe00640 1 -- 172.21.15.40:0/131636155 learned_addr learned my addr 172.21.15.40:0/131636155 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.090+0000 7fd58f400640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59015f1d0 0x7fd5901e9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.090+0000 7fd58fe00640 1 -- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd590163610 msgr2=0x7fd5901f6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.090+0000 7fd594a00640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd590163610 0x7fd5901f6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.090+0000 7fd58fe00640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd590163610 0x7fd5901f6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.090+0000 7fd58fe00640 1 -- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59015f1d0 msgr2=0x7fd5901e9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.090+0000 7fd58fe00640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59015f1d0 0x7fd5901e9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.090+0000 7fd58fe00640 1 -- 172.21.15.40:0/131636155 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5901e8c90 con 0x7fd5900d2c30 2024-04-26T23:02:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.090+0000 7fd58f400640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59015f1d0 0x7fd5901e9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.090+0000 7fd58fe00640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5900d2c30 0x7fd59015dd70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd580009d90 tx=0x7fd580004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.090+0000 7fd587e00640 1 -- 172.21.15.40:0/131636155 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580040070 con 0x7fd5900d2c30 2024-04-26T23:02:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.090+0000 7fd587e00640 1 -- 172.21.15.40:0/131636155 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd580030b60 con 0x7fd5900d2c30 2024-04-26T23:02:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.090+0000 7fd587e00640 1 -- 172.21.15.40:0/131636155 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580045460 con 0x7fd5900d2c30 2024-04-26T23:02:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.090+0000 7fd595400640 1 -- 172.21.15.40:0/131636155 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5901e8f70 con 0x7fd5900d2c30 2024-04-26T23:02:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.091+0000 7fd595400640 1 -- 172.21.15.40:0/131636155 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5901e9810 con 0x7fd5900d2c30 2024-04-26T23:02:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.092+0000 7fd595400640 1 -- 172.21.15.40:0/131636155 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5901e9360 con 0x7fd5900d2c30 2024-04-26T23:02:11.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.093+0000 7fd587e00640 1 -- 172.21.15.40:0/131636155 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd580039040 con 0x7fd5900d2c30 2024-04-26T23:02:11.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.094+0000 7fd587e00640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd57006c8f0 0x7fd57006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:11.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.094+0000 7fd587e00640 1 -- 172.21.15.40:0/131636155 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd5800ad850 con 0x7fd5900d2c30 2024-04-26T23:02:11.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.094+0000 7fd58f400640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd57006c8f0 0x7fd57006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:11.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.096+0000 7fd58f400640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd57006c8f0 0x7fd57006edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00a830 tx=0x7fd57c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:11.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.098+0000 7fd587e00640 1 -- 172.21.15.40:0/131636155 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd58007b7e0 con 0x7fd5900d2c30 2024-04-26T23:02:11.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.303+0000 7fd595400640 1 -- 172.21.15.40:0/131636155 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd590069a80 con 0x7fd5900d2c30 2024-04-26T23:02:11.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.306+0000 7fd587e00640 1 -- 172.21.15.40:0/131636155 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18812 (secure 0 0 0) 0x7fd58007b0e0 con 0x7fd5900d2c30 2024-04-26T23:02:11.307 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:11.307 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:46.975683+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:11.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.309+0000 7fd595400640 1 -- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd57006c8f0 msgr2=0x7fd57006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:11.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.309+0000 7fd595400640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd57006c8f0 0x7fd57006edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00a830 tx=0x7fd57c008040 comp rx=0 tx=0).stop 2024-04-26T23:02:11.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.309+0000 7fd595400640 1 -- 172.21.15.40:0/131636155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5900d2c30 msgr2=0x7fd59015dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:11.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.309+0000 7fd595400640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5900d2c30 0x7fd59015dd70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd580009d90 tx=0x7fd580004a40 comp rx=0 tx=0).stop 2024-04-26T23:02:11.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.309+0000 7fd595400640 1 -- 172.21.15.40:0/131636155 shutdown_connections 2024-04-26T23:02:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.309+0000 7fd595400640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd57006c8f0 0x7fd57006edb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.309+0000 7fd595400640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd590163610 0x7fd5901f6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.309+0000 7fd595400640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59015f1d0 0x7fd5901e9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.309+0000 7fd595400640 1 --2- 172.21.15.40:0/131636155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5900d2c30 0x7fd59015dd70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.309+0000 7fd595400640 1 -- 172.21.15.40:0/131636155 >> 172.21.15.40:0/131636155 conn(0x7fd59009ba50 msgr2=0x7fd5900d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.309+0000 7fd595400640 1 -- 172.21.15.40:0/131636155 shutdown_connections 2024-04-26T23:02:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.309+0000 7fd595400640 1 -- 172.21.15.40:0/131636155 wait complete. 2024-04-26T23:02:11.330 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:02:11.330 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:02:11.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.544+0000 7f1fbf200640 1 Processor -- start 2024-04-26T23:02:11.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.544+0000 7f1fbf200640 1 -- start start 2024-04-26T23:02:11.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.546+0000 7f1fbf200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1fc00d0db0 0x7f1fc00d1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:11.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.546+0000 7f1fbf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fc006ed30 0x7f1fc006f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:11.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.546+0000 7f1fbf200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc00ce930 con 0x7f1fc0146850 2024-04-26T23:02:11.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.546+0000 7f1fbf200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1fc006f6a0 con 0x7f1fc00d0db0 2024-04-26T23:02:11.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.546+0000 7f1fbf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1fc00cec10 con 0x7f1fc006ed30 2024-04-26T23:02:11.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.546+0000 7f1fbde00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1fc0146850 0x7f1fc0073720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52066/0 (socket says 172.21.15.40:52066) 2024-04-26T23:02:11.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.546+0000 7f1fbde00640 1 -- 172.21.15.40:0/2582059282 learned_addr learned my addr 172.21.15.40:0/2582059282 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:11.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.547+0000 7f1fbd400640 1 --2- 172.21.15.40:0/2582059282 >> v2:172.21.15.40:3300/0 conn(0x7f1fc00d0db0 0x7f1fc00d1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:11.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.547+0000 7f1fbe800640 1 --2- 172.21.15.40:0/2582059282 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fc006ed30 0x7f1fc006f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:11.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.547+0000 7f1fbd400640 1 -- 172.21.15.40:0/2582059282 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fc006ed30 msgr2=0x7f1fc006f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:11.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.547+0000 7f1fbd400640 1 --2- 172.21.15.40:0/2582059282 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fc006ed30 0x7f1fc006f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.547+0000 7f1fbca00640 1 -- 172.21.15.40:0/2582059282 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 980737034 0 0) 0x7f1fc00ce930 con 0x7f1fc0146850 2024-04-26T23:02:11.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.547+0000 7f1fbd400640 1 -- 172.21.15.40:0/2582059282 >> v1:172.21.15.40:6789/0 conn(0x7f1fc0146850 legacy=0x7f1fc0073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:11.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.547+0000 7f1fbd400640 1 -- 172.21.15.40:0/2582059282 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc00d05d0 con 0x7f1fc00d0db0 2024-04-26T23:02:11.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.547+0000 7f1fbe800640 1 --2- 172.21.15.40:0/2582059282 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fc006ed30 0x7f1fc006f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:11.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.548+0000 7f1fbd400640 1 --2- 172.21.15.40:0/2582059282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc00d0db0 0x7f1fc00d1220 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0097a0 tx=0x7f1fac02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab3476a355ee3cc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.548+0000 7f1fbca00640 1 -- 172.21.15.40:0/2582059282 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac037070 con 0x7f1fc00d0db0 2024-04-26T23:02:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.549+0000 7f1fbca00640 1 -- 172.21.15.40:0/2582059282 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fac032440 con 0x7f1fc00d0db0 2024-04-26T23:02:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.549+0000 7f1fbca00640 1 -- 172.21.15.40:0/2582059282 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac032810 con 0x7f1fc00d0db0 2024-04-26T23:02:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.549+0000 7f1fbf200640 1 -- 172.21.15.40:0/2582059282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc00d0db0 msgr2=0x7f1fc00d1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.549+0000 7f1fbf200640 1 --2- 172.21.15.40:0/2582059282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc00d0db0 0x7f1fc00d1220 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0097a0 tx=0x7f1fac02a180 comp rx=0 tx=0).stop 2024-04-26T23:02:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.549+0000 7f1fbf200640 1 -- 172.21.15.40:0/2582059282 shutdown_connections 2024-04-26T23:02:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.549+0000 7f1fbf200640 1 --2- 172.21.15.40:0/2582059282 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fc006ed30 0x7f1fc006f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.549+0000 7f1fbf200640 1 --2- 172.21.15.40:0/2582059282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc00d0db0 0x7f1fc00d1220 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.549+0000 7f1fbf200640 1 -- 172.21.15.40:0/2582059282 >> 172.21.15.40:0/2582059282 conn(0x7f1fc00b38a0 msgr2=0x7f1fc00b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.549+0000 7f1fbf200640 1 -- 172.21.15.40:0/2582059282 shutdown_connections 2024-04-26T23:02:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.549+0000 7f1fbf200640 1 -- 172.21.15.40:0/2582059282 wait complete. 2024-04-26T23:02:11.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.551+0000 7f1fbf200640 1 Processor -- start 2024-04-26T23:02:11.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.551+0000 7f1fbf200640 1 -- start start 2024-04-26T23:02:11.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.552+0000 7f1fbf200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc006ed30 0x7f1fc0158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:11.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.552+0000 7f1fbf200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc0146850 0x7f1fc015cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:11.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.552+0000 7f1fbf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fc01382d0 0x7f1fc015e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:11.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.552+0000 7f1fbf200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc00ce5c0 con 0x7f1fc0146850 2024-04-26T23:02:11.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.552+0000 7f1fbf200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc00b4610 con 0x7f1fc006ed30 2024-04-26T23:02:11.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.552+0000 7f1fbf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1fc00d0230 con 0x7f1fc01382d0 2024-04-26T23:02:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.553+0000 7f1fbd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc0146850 0x7f1fc015cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.553+0000 7f1fbd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc0146850 0x7f1fc015cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36440/0 (socket says 172.21.15.40:36440) 2024-04-26T23:02:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.553+0000 7f1fbd400640 1 -- 172.21.15.40:0/1385448192 learned_addr learned my addr 172.21.15.40:0/1385448192 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.553+0000 7f1fbde00640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc006ed30 0x7f1fc0158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.553+0000 7f1fbe800640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fc01382d0 0x7f1fc015e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.553+0000 7f1fbd400640 1 -- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fc01382d0 msgr2=0x7f1fc015e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.553+0000 7f1fbd400640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fc01382d0 0x7f1fc015e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.553+0000 7f1fbd400640 1 -- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc006ed30 msgr2=0x7f1fc0158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.553+0000 7f1fbd400640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc006ed30 0x7f1fc0158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.553+0000 7f1fbd400640 1 -- 172.21.15.40:0/1385448192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc0138a60 con 0x7f1fc0146850 2024-04-26T23:02:11.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.554+0000 7f1fbd400640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc0146850 0x7f1fc015cc50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0097a0 tx=0x7f1fac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:11.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.554+0000 7f1fb6000640 1 -- 172.21.15.40:0/1385448192 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac037050 con 0x7f1fc0146850 2024-04-26T23:02:11.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.554+0000 7f1fb6000640 1 -- 172.21.15.40:0/1385448192 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fac038040 con 0x7f1fc0146850 2024-04-26T23:02:11.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.554+0000 7f1fbf200640 1 -- 172.21.15.40:0/1385448192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc0155f50 con 0x7f1fc0146850 2024-04-26T23:02:11.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.554+0000 7f1fb6000640 1 -- 172.21.15.40:0/1385448192 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac006e10 con 0x7f1fc0146850 2024-04-26T23:02:11.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.554+0000 7f1fbf200640 1 -- 172.21.15.40:0/1385448192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc0156510 con 0x7f1fc0146850 2024-04-26T23:02:11.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.555+0000 7f1fbf200640 1 -- 172.21.15.40:0/1385448192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc0069a80 con 0x7f1fc0146850 2024-04-26T23:02:11.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.556+0000 7f1fb6000640 1 -- 172.21.15.40:0/1385448192 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1fac030980 con 0x7f1fc0146850 2024-04-26T23:02:11.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.558+0000 7f1fb6000640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1fa006c920 0x7f1fa006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:11.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.558+0000 7f1fb6000640 1 -- 172.21.15.40:0/1385448192 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1fac0acb70 con 0x7f1fc0146850 2024-04-26T23:02:11.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.558+0000 7f1fbde00640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1fa006c920 0x7f1fa006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:11.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.560+0000 7f1fbde00640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1fa006c920 0x7f1fa006ede0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0000f60 tx=0x7f1fb0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:11.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.562+0000 7f1fb6000640 1 -- 172.21.15.40:0/1385448192 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1fac07ab00 con 0x7f1fc0146850 2024-04-26T23:02:11.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.767+0000 7f1fbf200640 1 -- 172.21.15.40:0/1385448192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1fc006ffa0 con 0x7f1fc0146850 2024-04-26T23:02:11.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.768+0000 7f1fb6000640 1 -- 172.21.15.40:0/1385448192 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f1fac07a400 con 0x7f1fc0146850 2024-04-26T23:02:11.768 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:02:11.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.771+0000 7f1fbf200640 1 -- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1fa006c920 msgr2=0x7f1fa006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:11.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.771+0000 7f1fbf200640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1fa006c920 0x7f1fa006ede0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0000f60 tx=0x7f1fb0009040 comp rx=0 tx=0).stop 2024-04-26T23:02:11.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.771+0000 7f1fbf200640 1 -- 172.21.15.40:0/1385448192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc0146850 msgr2=0x7f1fc015cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:11.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.771+0000 7f1fbf200640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc0146850 0x7f1fc015cc50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0097a0 tx=0x7f1fac0046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.771+0000 7f1fbf200640 1 -- 172.21.15.40:0/1385448192 shutdown_connections 2024-04-26T23:02:11.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.771+0000 7f1fbf200640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1fa006c920 0x7f1fa006ede0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.771+0000 7f1fbf200640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fc01382d0 0x7f1fc015e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.771+0000 7f1fbf200640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fc0146850 0x7f1fc015cc50 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.771+0000 7f1fbf200640 1 --2- 172.21.15.40:0/1385448192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fc006ed30 0x7f1fc0158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:11.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.771+0000 7f1fbf200640 1 -- 172.21.15.40:0/1385448192 >> 172.21.15.40:0/1385448192 conn(0x7f1fc00b38a0 msgr2=0x7f1fc00b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:11.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.771+0000 7f1fbf200640 1 -- 172.21.15.40:0/1385448192 shutdown_connections 2024-04-26T23:02:11.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:11.771+0000 7f1fbf200640 1 -- 172.21.15.40:0/1385448192 wait complete. 2024-04-26T23:02:11.791 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:02:12.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.011+0000 7fa185000640 1 Processor -- start 2024-04-26T23:02:12.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.011+0000 7fa185000640 1 -- start start 2024-04-26T23:02:12.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.013+0000 7fa185000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa1800b5500 0x7fa1800b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.013+0000 7fa185000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b59b0 0x7fa1800b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.013+0000 7fa185000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1800d3dc0 con 0x7fa1800b5120 2024-04-26T23:02:12.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.013+0000 7fa185000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa1800d03b0 con 0x7fa1800b5500 2024-04-26T23:02:12.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.013+0000 7fa185000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa1800b75b0 con 0x7fa1800b59b0 2024-04-26T23:02:12.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.013+0000 7fa17f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa1800b5120 0x7fa1800b6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52068/0 (socket says 172.21.15.40:52068) 2024-04-26T23:02:12.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.013+0000 7fa17f400640 1 -- 172.21.15.40:0/1775181954 learned_addr learned my addr 172.21.15.40:0/1775181954 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:12.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.013+0000 7fa17ea00640 1 --2- 172.21.15.40:0/1775181954 >> v2:172.21.15.103:3300/0 conn(0x7fa1800b5500 0x7fa1800b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.013+0000 7fa17fe00640 1 --2- 172.21.15.40:0/1775181954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b59b0 0x7fa1800b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1007203157 0 0) 0x7fa1800d3dc0 con 0x7fa1800b5120 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa164003520 con 0x7fa1800b5120 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 326871624 0 0) 0x7fa164003520 con 0x7fa1800b5120 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b59b0 msgr2=0x7fa1800b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 --2- 172.21.15.40:0/1775181954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b59b0 0x7fa1800b6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 >> v2:172.21.15.103:3300/0 conn(0x7fa1800b5500 msgr2=0x7fa1800b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 --2- 172.21.15.40:0/1775181954 >> v2:172.21.15.103:3300/0 conn(0x7fa1800b5500 0x7fa1800b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1800d0600 con 0x7fa1800b5120 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fa170001670 con 0x7fa1800b5120 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 >> v1:172.21.15.40:6789/0 conn(0x7fa1800b5120 legacy=0x7fa1800b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 --2- 172.21.15.40:0/1775181954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1640039b0 0x7fa164003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 --2- 172.21.15.40:0/1775181954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa164004800 0x7fa164004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 --2- 172.21.15.40:0/1775181954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b59b0 0x7fa1640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa1800d03b0 con 0x7fa1640039b0 2024-04-26T23:02:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1800b75b0 con 0x7fa164004800 2024-04-26T23:02:12.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa16400ba40 con 0x7fa1800b59b0 2024-04-26T23:02:12.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.014+0000 7fa17f400640 1 --2- 172.21.15.40:0/1775181954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1640039b0 0x7fa164003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.015+0000 7fa17ea00640 1 --2- 172.21.15.40:0/1775181954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa164004800 0x7fa164004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.015+0000 7fa17fe00640 1 --2- 172.21.15.40:0/1775181954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b59b0 0x7fa1640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.015+0000 7fa17f400640 1 -- 172.21.15.40:0/1775181954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b59b0 msgr2=0x7fa1640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:12.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.015+0000 7fa17f400640 1 --2- 172.21.15.40:0/1775181954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b59b0 0x7fa1640051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.015+0000 7fa17f400640 1 -- 172.21.15.40:0/1775181954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa164004800 msgr2=0x7fa164004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:12.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.015+0000 7fa17f400640 1 --2- 172.21.15.40:0/1775181954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa164004800 0x7fa164004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.015+0000 7fa17f400640 1 -- 172.21.15.40:0/1775181954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa164003520 con 0x7fa1640039b0 2024-04-26T23:02:12.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.016+0000 7fa17f400640 1 --2- 172.21.15.40:0/1775181954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1640039b0 0x7fa164003dd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa17000c240 tx=0x7fa164005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f37e5ffa1000cc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:12.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.016+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa170007d10 con 0x7fa1640039b0 2024-04-26T23:02:12.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.016+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa170005ad0 con 0x7fa1640039b0 2024-04-26T23:02:12.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.016+0000 7fa17e000640 1 -- 172.21.15.40:0/1775181954 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa170007070 con 0x7fa1640039b0 2024-04-26T23:02:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.017+0000 7fa185000640 1 -- 172.21.15.40:0/1775181954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1640039b0 msgr2=0x7fa164003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.017+0000 7fa185000640 1 --2- 172.21.15.40:0/1775181954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1640039b0 0x7fa164003dd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa17000c240 tx=0x7fa164005830 comp rx=0 tx=0).stop 2024-04-26T23:02:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.017+0000 7fa17fe00640 1 -- 172.21.15.40:0/1775181954 reap_dead start 2024-04-26T23:02:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.017+0000 7fa185000640 1 -- 172.21.15.40:0/1775181954 shutdown_connections 2024-04-26T23:02:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.017+0000 7fa185000640 1 -- 172.21.15.40:0/1775181954 >> 172.21.15.40:0/1775181954 conn(0x7fa1800d4db0 msgr2=0x7fa1800d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.017+0000 7fa185000640 1 -- 172.21.15.40:0/1775181954 shutdown_connections 2024-04-26T23:02:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.017+0000 7fa185000640 1 -- 172.21.15.40:0/1775181954 wait complete. 2024-04-26T23:02:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.019+0000 7fa185000640 1 Processor -- start 2024-04-26T23:02:12.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.019+0000 7fa185000640 1 -- start start 2024-04-26T23:02:12.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.020+0000 7fa185000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1640039b0 0x7fa1801f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.020+0000 7fa185000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa180154460 0x7fa1800b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.020+0000 7fa185000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b4180 0x7fa1800c8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.020+0000 7fa185000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa1800b6f40 con 0x7fa180154460 2024-04-26T23:02:12.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.020+0000 7fa185000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1800d0bc0 con 0x7fa1640039b0 2024-04-26T23:02:12.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.020+0000 7fa185000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa1800b70c0 con 0x7fa1800b4180 2024-04-26T23:02:12.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.020+0000 7fa17ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa180154460 0x7fa1800b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.020+0000 7fa17ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa180154460 0x7fa1800b3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36452/0 (socket says 172.21.15.40:36452) 2024-04-26T23:02:12.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.020+0000 7fa17ea00640 1 -- 172.21.15.40:0/86248719 learned_addr learned my addr 172.21.15.40:0/86248719 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:12.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.021+0000 7fa17f400640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1640039b0 0x7fa1801f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.021+0000 7fa17fe00640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b4180 0x7fa1800c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.021+0000 7fa17ea00640 1 -- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b4180 msgr2=0x7fa1800c8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.021+0000 7fa17ea00640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b4180 0x7fa1800c8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.021+0000 7fa17ea00640 1 -- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1640039b0 msgr2=0x7fa1801f5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.021+0000 7fa17ea00640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1640039b0 0x7fa1801f5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.021+0000 7fa17ea00640 1 -- 172.21.15.40:0/86248719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1800c94f0 con 0x7fa180154460 2024-04-26T23:02:12.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.021+0000 7fa17fe00640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b4180 0x7fa1800c8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:12.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.021+0000 7fa17ea00640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa180154460 0x7fa1800b3c40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa16c007f00 tx=0x7fa16c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:12.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.022+0000 7fa177400640 1 -- 172.21.15.40:0/86248719 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c01a020 con 0x7fa180154460 2024-04-26T23:02:12.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.022+0000 7fa177400640 1 -- 172.21.15.40:0/86248719 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa16c016470 con 0x7fa180154460 2024-04-26T23:02:12.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.022+0000 7fa177400640 1 -- 172.21.15.40:0/86248719 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c016860 con 0x7fa180154460 2024-04-26T23:02:12.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.022+0000 7fa185000640 1 -- 172.21.15.40:0/86248719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1800b1b40 con 0x7fa180154460 2024-04-26T23:02:12.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.022+0000 7fa185000640 1 -- 172.21.15.40:0/86248719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1800b2100 con 0x7fa180154460 2024-04-26T23:02:12.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.023+0000 7fa185000640 1 -- 172.21.15.40:0/86248719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa180069370 con 0x7fa180154460 2024-04-26T23:02:12.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.024+0000 7fa177400640 1 -- 172.21.15.40:0/86248719 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa16c005d10 con 0x7fa180154460 2024-04-26T23:02:12.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.025+0000 7fa177400640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa16006c8f0 0x7fa16006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.025+0000 7fa177400640 1 -- 172.21.15.40:0/86248719 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa16c08cd70 con 0x7fa180154460 2024-04-26T23:02:12.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.026+0000 7fa17f400640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa16006c8f0 0x7fa16006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.027+0000 7fa17f400640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa16006c8f0 0x7fa16006edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa170033000 tx=0x7fa17002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:12.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.029+0000 7fa177400640 1 -- 172.21.15.40:0/86248719 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa16c024030 con 0x7fa180154460 2024-04-26T23:02:12.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.237+0000 7fa185000640 1 -- 172.21.15.40:0/86248719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa1800b4560 con 0x7fa180154460 2024-04-26T23:02:12.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.239+0000 7fa177400640 1 -- 172.21.15.40:0/86248719 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fa16c05ada0 con 0x7fa180154460 2024-04-26T23:02:12.239 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:12.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.241+0000 7fa185000640 1 -- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa16006c8f0 msgr2=0x7fa16006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.241+0000 7fa185000640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa16006c8f0 0x7fa16006edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa170033000 tx=0x7fa17002e000 comp rx=0 tx=0).stop 2024-04-26T23:02:12.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.241+0000 7fa185000640 1 -- 172.21.15.40:0/86248719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa180154460 msgr2=0x7fa1800b3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.241+0000 7fa185000640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa180154460 0x7fa1800b3c40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa16c007f00 tx=0x7fa16c0115a0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.242+0000 7fa185000640 1 -- 172.21.15.40:0/86248719 shutdown_connections 2024-04-26T23:02:12.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.242+0000 7fa185000640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa16006c8f0 0x7fa16006edb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.242+0000 7fa185000640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1800b4180 0x7fa1800c8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.242+0000 7fa185000640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa180154460 0x7fa1800b3c40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.242+0000 7fa185000640 1 --2- 172.21.15.40:0/86248719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1640039b0 0x7fa1801f5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.242+0000 7fa185000640 1 -- 172.21.15.40:0/86248719 >> 172.21.15.40:0/86248719 conn(0x7fa1800d4db0 msgr2=0x7fa1800b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:12.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.242+0000 7fa185000640 1 -- 172.21.15.40:0/86248719 shutdown_connections 2024-04-26T23:02:12.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.242+0000 7fa185000640 1 -- 172.21.15.40:0/86248719 wait complete. 2024-04-26T23:02:12.263 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:02:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.483+0000 7f9b21000640 1 Processor -- start 2024-04-26T23:02:12.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.483+0000 7f9b21000640 1 -- start start 2024-04-26T23:02:12.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.485+0000 7f9b21000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9b1c15f2d0 0x7f9b1c0afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.485+0000 7f9b21000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1c0af7b0 con 0x7f9b1c0cc590 2024-04-26T23:02:12.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.485+0000 7f9b21000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1c0afa90 con 0x7f9b1c0ce5e0 2024-04-26T23:02:12.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.485+0000 7f9b21000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f9b1c0cbd90 con 0x7f9b1c15f2d0 2024-04-26T23:02:12.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.485+0000 7f9b1be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9b1c0cc590 0x7f9b1c06f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52076/0 (socket says 172.21.15.40:52076) 2024-04-26T23:02:12.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.485+0000 7f9b1be00640 1 -- 172.21.15.40:0/2875879866 learned_addr learned my addr 172.21.15.40:0/2875879866 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:12.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.485+0000 7f9b1b400640 1 --2- 172.21.15.40:0/2875879866 >> v2:172.21.15.40:3300/0 conn(0x7f9b1c15f2d0 0x7f9b1c0afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.486+0000 7f9b1a000640 1 -- 172.21.15.40:0/2875879866 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1353826181 0 0) 0x7f9b1c0af7b0 con 0x7f9b1c0cc590 2024-04-26T23:02:12.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.486+0000 7f9b1a000640 1 -- 172.21.15.40:0/2875879866 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b00003520 con 0x7f9b1c0cc590 2024-04-26T23:02:12.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.486+0000 7f9b1b400640 1 -- 172.21.15.40:0/2875879866 >> v1:172.21.15.103:6789/0 conn(0x7f9b1c0ce5e0 legacy=0x7f9b1c0c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:12.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.486+0000 7f9b1b400640 1 -- 172.21.15.40:0/2875879866 >> v1:172.21.15.40:6789/0 conn(0x7f9b1c0cc590 legacy=0x7f9b1c06f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.486+0000 7f9b1b400640 1 -- 172.21.15.40:0/2875879866 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1c0cc070 con 0x7f9b1c15f2d0 2024-04-26T23:02:12.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.487+0000 7f9b1b400640 1 --2- 172.21.15.40:0/2875879866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b1c15f2d0 0x7f9b1c0afdf0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0097a0 tx=0x7f9b0c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c41eb38e6a8cb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:12.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.487+0000 7f9b1a000640 1 -- 172.21.15.40:0/2875879866 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c037070 con 0x7f9b1c15f2d0 2024-04-26T23:02:12.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.487+0000 7f9b1a000640 1 -- 172.21.15.40:0/2875879866 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b0c032440 con 0x7f9b1c15f2d0 2024-04-26T23:02:12.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.487+0000 7f9b1a000640 1 -- 172.21.15.40:0/2875879866 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c032810 con 0x7f9b1c15f2d0 2024-04-26T23:02:12.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.488+0000 7f9b21000640 1 -- 172.21.15.40:0/2875879866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b1c15f2d0 msgr2=0x7f9b1c0afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.488+0000 7f9b21000640 1 --2- 172.21.15.40:0/2875879866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b1c15f2d0 0x7f9b1c0afdf0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0097a0 tx=0x7f9b0c02a230 comp rx=0 tx=0).stop 2024-04-26T23:02:12.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.488+0000 7f9b21000640 1 -- 172.21.15.40:0/2875879866 shutdown_connections 2024-04-26T23:02:12.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.488+0000 7f9b21000640 1 --2- 172.21.15.40:0/2875879866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b1c15f2d0 0x7f9b1c0afdf0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.488+0000 7f9b21000640 1 -- 172.21.15.40:0/2875879866 >> 172.21.15.40:0/2875879866 conn(0x7f9b1c0d0a10 msgr2=0x7f9b1c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:12.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.488+0000 7f9b21000640 1 -- 172.21.15.40:0/2875879866 shutdown_connections 2024-04-26T23:02:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.488+0000 7f9b21000640 1 -- 172.21.15.40:0/2875879866 wait complete. 2024-04-26T23:02:12.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.490+0000 7f9b21000640 1 Processor -- start 2024-04-26T23:02:12.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.490+0000 7f9b21000640 1 -- start start 2024-04-26T23:02:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.491+0000 7f9b21000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b1c0cc590 0x7f9b1c1f9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.491+0000 7f9b21000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b1c0ce5e0 0x7f9b1c152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.491+0000 7f9b21000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b1c15f2d0 0x7f9b1c0b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.491+0000 7f9b21000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c0c5fc0 con 0x7f9b1c0cc590 2024-04-26T23:02:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.491+0000 7f9b21000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c06e270 con 0x7f9b1c0ce5e0 2024-04-26T23:02:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.491+0000 7f9b21000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9b1c1fa280 con 0x7f9b1c15f2d0 2024-04-26T23:02:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.491+0000 7f9b1b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b1c0cc590 0x7f9b1c1f9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.491+0000 7f9b1b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b1c0cc590 0x7f9b1c1f9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36474/0 (socket says 172.21.15.40:36474) 2024-04-26T23:02:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.491+0000 7f9b1b400640 1 -- 172.21.15.40:0/1451430768 learned_addr learned my addr 172.21.15.40:0/1451430768 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.492+0000 7f9b1b400640 1 -- 172.21.15.40:0/1451430768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b1c15f2d0 msgr2=0x7f9b1c0b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.492+0000 7f9b1b400640 1 --2- 172.21.15.40:0/1451430768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b1c15f2d0 0x7f9b1c0b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.492+0000 7f9b1b400640 1 -- 172.21.15.40:0/1451430768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b1c0ce5e0 msgr2=0x7f9b1c152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.492+0000 7f9b1aa00640 1 --2- 172.21.15.40:0/1451430768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b1c0ce5e0 0x7f9b1c152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.492+0000 7f9b1b400640 1 --2- 172.21.15.40:0/1451430768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b1c0ce5e0 0x7f9b1c152440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.492+0000 7f9b1b400640 1 -- 172.21.15.40:0/1451430768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1c0b6250 con 0x7f9b1c0cc590 2024-04-26T23:02:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.492+0000 7f9b1b400640 1 --2- 172.21.15.40:0/1451430768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b1c0cc590 0x7f9b1c1f9d40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c035040 tx=0x7f9b0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.492+0000 7f9b13400640 1 -- 172.21.15.40:0/1451430768 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c037040 con 0x7f9b1c0cc590 2024-04-26T23:02:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.492+0000 7f9b13400640 1 -- 172.21.15.40:0/1451430768 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b0c030500 con 0x7f9b1c0cc590 2024-04-26T23:02:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.492+0000 7f9b13400640 1 -- 172.21.15.40:0/1451430768 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c030910 con 0x7f9b1c0cc590 2024-04-26T23:02:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.493+0000 7f9b21000640 1 -- 172.21.15.40:0/1451430768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b1c0b6530 con 0x7f9b1c0cc590 2024-04-26T23:02:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.493+0000 7f9b21000640 1 -- 172.21.15.40:0/1451430768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b1c1fa8e0 con 0x7f9b1c0cc590 2024-04-26T23:02:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.494+0000 7f9b21000640 1 -- 172.21.15.40:0/1451430768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b1c069370 con 0x7f9b1c0cc590 2024-04-26T23:02:12.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.495+0000 7f9b13400640 1 -- 172.21.15.40:0/1451430768 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9b0c044020 con 0x7f9b1c0cc590 2024-04-26T23:02:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.496+0000 7f9b13400640 1 --2- 172.21.15.40:0/1451430768 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9afc06c8f0 0x7f9afc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.496+0000 7f9b13400640 1 -- 172.21.15.40:0/1451430768 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f9b0c0ac1f0 con 0x7f9b1c0cc590 2024-04-26T23:02:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.496+0000 7f9b1aa00640 1 --2- 172.21.15.40:0/1451430768 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9afc06c8f0 0x7f9afc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.498+0000 7f9b1aa00640 1 --2- 172.21.15.40:0/1451430768 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9afc06c8f0 0x7f9afc06edb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c0c5cb0 tx=0x7f9b08009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:12.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.500+0000 7f9b13400640 1 -- 172.21.15.40:0/1451430768 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9b0c07a220 con 0x7f9b1c0cc590 2024-04-26T23:02:12.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.703+0000 7f9b21000640 1 -- 172.21.15.40:0/1451430768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9b1c0b6700 con 0x7f9b1c0cc590 2024-04-26T23:02:12.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.704+0000 7f9b13400640 1 -- 172.21.15.40:0/1451430768 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f9b0c079b20 con 0x7f9b1c0cc590 2024-04-26T23:02:12.705 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:02:12.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.707+0000 7f9b21000640 1 -- 172.21.15.40:0/1451430768 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9afc06c8f0 msgr2=0x7f9afc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.707+0000 7f9b21000640 1 --2- 172.21.15.40:0/1451430768 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9afc06c8f0 0x7f9afc06edb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c0c5cb0 tx=0x7f9b08009040 comp rx=0 tx=0).stop 2024-04-26T23:02:12.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.707+0000 7f9b21000640 1 -- 172.21.15.40:0/1451430768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b1c0cc590 msgr2=0x7f9b1c1f9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.707+0000 7f9b21000640 1 --2- 172.21.15.40:0/1451430768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b1c0cc590 0x7f9b1c1f9d40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c035040 tx=0x7f9b0c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.707+0000 7f9b21000640 1 -- 172.21.15.40:0/1451430768 shutdown_connections 2024-04-26T23:02:12.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.707+0000 7f9b21000640 1 --2- 172.21.15.40:0/1451430768 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9afc06c8f0 0x7f9afc06edb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.707+0000 7f9b21000640 1 --2- 172.21.15.40:0/1451430768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b1c15f2d0 0x7f9b1c0b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.707+0000 7f9b21000640 1 --2- 172.21.15.40:0/1451430768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b1c0ce5e0 0x7f9b1c152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.707+0000 7f9b21000640 1 --2- 172.21.15.40:0/1451430768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b1c0cc590 0x7f9b1c1f9d40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.707+0000 7f9b21000640 1 -- 172.21.15.40:0/1451430768 >> 172.21.15.40:0/1451430768 conn(0x7f9b1c0d0a10 msgr2=0x7f9b1c06f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:12.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.707+0000 7f9b21000640 1 -- 172.21.15.40:0/1451430768 shutdown_connections 2024-04-26T23:02:12.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.707+0000 7f9b21000640 1 -- 172.21.15.40:0/1451430768 wait complete. 2024-04-26T23:02:12.727 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:02:12.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.943+0000 7f7c46200640 1 Processor -- start 2024-04-26T23:02:12.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.943+0000 7f7c46200640 1 -- start start 2024-04-26T23:02:12.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.944+0000 7f7c46200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7c400b5500 0x7f7c400b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.944+0000 7f7c46200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7c400b59b0 0x7f7c400b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.944+0000 7f7c46200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c400d3dc0 con 0x7f7c400b5120 2024-04-26T23:02:12.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.944+0000 7f7c46200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f7c400d03b0 con 0x7f7c400b5500 2024-04-26T23:02:12.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.944+0000 7f7c46200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7c400b75b0 con 0x7f7c400b59b0 2024-04-26T23:02:12.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.945+0000 7f7c44e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7c400b5120 0x7f7c400b6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52080/0 (socket says 172.21.15.40:52080) 2024-04-26T23:02:12.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.945+0000 7f7c44e00640 1 -- 172.21.15.40:0/1595183449 learned_addr learned my addr 172.21.15.40:0/1595183449 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:12.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.945+0000 7f7c3fe00640 1 --2- 172.21.15.40:0/1595183449 >> v2:172.21.15.40:3300/0 conn(0x7f7c400b5500 0x7f7c400b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.945+0000 7f7c45800640 1 --2- 172.21.15.40:0/1595183449 >> v2:172.21.15.103:3300/0 conn(0x7f7c400b59b0 0x7f7c400b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.945+0000 7f7c3fe00640 1 -- 172.21.15.40:0/1595183449 >> v2:172.21.15.103:3300/0 conn(0x7f7c400b59b0 msgr2=0x7f7c400b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:12.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.945+0000 7f7c3f400640 1 -- 172.21.15.40:0/1595183449 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3579072282 0 0) 0x7f7c400d3dc0 con 0x7f7c400b5120 2024-04-26T23:02:12.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.945+0000 7f7c3fe00640 1 --2- 172.21.15.40:0/1595183449 >> v2:172.21.15.103:3300/0 conn(0x7f7c400b59b0 0x7f7c400b6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.945+0000 7f7c3fe00640 1 -- 172.21.15.40:0/1595183449 >> v1:172.21.15.40:6789/0 conn(0x7f7c400b5120 legacy=0x7f7c400b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.945+0000 7f7c3fe00640 1 -- 172.21.15.40:0/1595183449 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c400d0600 con 0x7f7c400b5500 2024-04-26T23:02:12.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.946+0000 7f7c3fe00640 1 --2- 172.21.15.40:0/1595183449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c400b5500 0x7f7c400b5ee0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7c28009d90 tx=0x7f7c2802a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d53526fed4552de6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:12.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.947+0000 7f7c3f400640 1 -- 172.21.15.40:0/1595183449 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c28007740 con 0x7f7c400b5500 2024-04-26T23:02:12.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.947+0000 7f7c3f400640 1 -- 172.21.15.40:0/1595183449 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c280078e0 con 0x7f7c400b5500 2024-04-26T23:02:12.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.947+0000 7f7c3f400640 1 -- 172.21.15.40:0/1595183449 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c28007cb0 con 0x7f7c400b5500 2024-04-26T23:02:12.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.947+0000 7f7c46200640 1 -- 172.21.15.40:0/1595183449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c400b5500 msgr2=0x7f7c400b5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.947+0000 7f7c46200640 1 --2- 172.21.15.40:0/1595183449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c400b5500 0x7f7c400b5ee0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7c28009d90 tx=0x7f7c2802a580 comp rx=0 tx=0).stop 2024-04-26T23:02:12.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.948+0000 7f7c46200640 1 -- 172.21.15.40:0/1595183449 shutdown_connections 2024-04-26T23:02:12.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.948+0000 7f7c46200640 1 --2- 172.21.15.40:0/1595183449 >> v2:172.21.15.103:3300/0 conn(0x7f7c400b59b0 0x7f7c400b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.948+0000 7f7c46200640 1 --2- 172.21.15.40:0/1595183449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c400b5500 0x7f7c400b5ee0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.948+0000 7f7c46200640 1 -- 172.21.15.40:0/1595183449 >> 172.21.15.40:0/1595183449 conn(0x7f7c400d4db0 msgr2=0x7f7c400d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:12.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.948+0000 7f7c46200640 1 -- 172.21.15.40:0/1595183449 shutdown_connections 2024-04-26T23:02:12.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.948+0000 7f7c46200640 1 -- 172.21.15.40:0/1595183449 wait complete. 2024-04-26T23:02:12.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.949+0000 7f7c46200640 1 Processor -- start 2024-04-26T23:02:12.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.950+0000 7f7c46200640 1 -- start start 2024-04-26T23:02:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.951+0000 7f7c46200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c400b5120 0x7f7c401f4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.951+0000 7f7c46200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c400b5500 0x7f7c400c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.951+0000 7f7c46200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c400b59b0 0x7f7c4006d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.951+0000 7f7c46200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7c400b7c00 con 0x7f7c400b5120 2024-04-26T23:02:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.951+0000 7f7c46200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7c400d0bc0 con 0x7f7c400b59b0 2024-04-26T23:02:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.951+0000 7f7c46200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7c400b7d80 con 0x7f7c400b5500 2024-04-26T23:02:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.951+0000 7f7c44e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c400b5120 0x7f7c401f4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.951+0000 7f7c44e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c400b5120 0x7f7c401f4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36502/0 (socket says 172.21.15.40:36502) 2024-04-26T23:02:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.951+0000 7f7c44e00640 1 -- 172.21.15.40:0/2402948664 learned_addr learned my addr 172.21.15.40:0/2402948664 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:12.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.951+0000 7f7c45800640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c400b59b0 0x7f7c4006d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.951+0000 7f7c3fe00640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c400b5500 0x7f7c400c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.952+0000 7f7c44e00640 1 -- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c400b5500 msgr2=0x7f7c400c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.952+0000 7f7c44e00640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c400b5500 0x7f7c400c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.952+0000 7f7c44e00640 1 -- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c400b59b0 msgr2=0x7f7c4006d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:12.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.952+0000 7f7c44e00640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c400b59b0 0x7f7c4006d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:12.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.952+0000 7f7c44e00640 1 -- 172.21.15.40:0/2402948664 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c400c9a30 con 0x7f7c400b5120 2024-04-26T23:02:12.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.952+0000 7f7c44e00640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c400b5120 0x7f7c401f4ad0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000c4d0 tx=0x7f7c3000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:12.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.952+0000 7f7c3cc00640 1 -- 172.21.15.40:0/2402948664 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c30018070 con 0x7f7c400b5120 2024-04-26T23:02:12.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.952+0000 7f7c3cc00640 1 -- 172.21.15.40:0/2402948664 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c30007500 con 0x7f7c400b5120 2024-04-26T23:02:12.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.952+0000 7f7c46200640 1 -- 172.21.15.40:0/2402948664 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c401f5240 con 0x7f7c400b5120 2024-04-26T23:02:12.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.952+0000 7f7c3cc00640 1 -- 172.21.15.40:0/2402948664 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c30004a20 con 0x7f7c400b5120 2024-04-26T23:02:12.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.953+0000 7f7c46200640 1 -- 172.21.15.40:0/2402948664 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c401f5800 con 0x7f7c400b5120 2024-04-26T23:02:12.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.954+0000 7f7c46200640 1 -- 172.21.15.40:0/2402948664 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c40069370 con 0x7f7c400b5120 2024-04-26T23:02:12.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.955+0000 7f7c3cc00640 1 -- 172.21.15.40:0/2402948664 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7c3001b070 con 0x7f7c400b5120 2024-04-26T23:02:12.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.956+0000 7f7c3cc00640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7c1c06c8f0 0x7f7c1c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:12.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.956+0000 7f7c3cc00640 1 -- 172.21.15.40:0/2402948664 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7c3008b140 con 0x7f7c400b5120 2024-04-26T23:02:12.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.957+0000 7f7c3fe00640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7c1c06c8f0 0x7f7c1c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:12.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.958+0000 7f7c3fe00640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7c1c06c8f0 0x7f7c1c06edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7c28034000 tx=0x7f7c2802f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:12.961+0000 7f7c3cc00640 1 -- 172.21.15.40:0/2402948664 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7c30058bb0 con 0x7f7c400b5120 2024-04-26T23:02:13.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.169+0000 7f7c46200640 1 -- 172.21.15.40:0/2402948664 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7c400b80b0 con 0x7f7c400b5120 2024-04-26T23:02:13.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.171+0000 7f7c3cc00640 1 -- 172.21.15.40:0/2402948664 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f7c3005ceb0 con 0x7f7c400b5120 2024-04-26T23:02:13.171 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:13.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.173+0000 7f7c46200640 1 -- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7c1c06c8f0 msgr2=0x7f7c1c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:13.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.173+0000 7f7c46200640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7c1c06c8f0 0x7f7c1c06edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7c28034000 tx=0x7f7c2802f000 comp rx=0 tx=0).stop 2024-04-26T23:02:13.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.173+0000 7f7c46200640 1 -- 172.21.15.40:0/2402948664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c400b5120 msgr2=0x7f7c401f4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:13.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.173+0000 7f7c46200640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c400b5120 0x7f7c401f4ad0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000c4d0 tx=0x7f7c3000c990 comp rx=0 tx=0).stop 2024-04-26T23:02:13.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.173+0000 7f7c46200640 1 -- 172.21.15.40:0/2402948664 shutdown_connections 2024-04-26T23:02:13.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.173+0000 7f7c46200640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7c1c06c8f0 0x7f7c1c06edb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.173+0000 7f7c46200640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c400b59b0 0x7f7c4006d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.173+0000 7f7c46200640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c400b5500 0x7f7c400c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.173+0000 7f7c46200640 1 --2- 172.21.15.40:0/2402948664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c400b5120 0x7f7c401f4ad0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.173+0000 7f7c46200640 1 -- 172.21.15.40:0/2402948664 >> 172.21.15.40:0/2402948664 conn(0x7f7c400d4db0 msgr2=0x7f7c400b6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:13.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.173+0000 7f7c46200640 1 -- 172.21.15.40:0/2402948664 shutdown_connections 2024-04-26T23:02:13.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.173+0000 7f7c46200640 1 -- 172.21.15.40:0/2402948664 wait complete. 2024-04-26T23:02:13.224 INFO:tasks.cephfs_test_runner:Starting test: test_apply_tag (tasks.cephfs.test_multimds_misc.TestScrub2) 2024-04-26T23:02:13.224 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:13.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.441+0000 7f19f7e00640 1 Processor -- start 2024-04-26T23:02:13.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.441+0000 7f19f7e00640 1 -- start start 2024-04-26T23:02:13.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.442+0000 7f19f7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f19f80b6220 0x7f19f80d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:13.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.442+0000 7f19f7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f80d39f0 0x7f19f80b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:13.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.442+0000 7f19f7e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19f8162c40 con 0x7f19f80d08d0 2024-04-26T23:02:13.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.442+0000 7f19f7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f19f80d2d80 con 0x7f19f80b6220 2024-04-26T23:02:13.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.442+0000 7f19f7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f19f80b7c80 con 0x7f19f80d39f0 2024-04-26T23:02:13.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.443+0000 7f19f6a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f19f80b6220 0x7f19f80d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:13.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.443+0000 7f19f7400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f80d39f0 0x7f19f80b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:13.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.443+0000 7f19f6a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f19f80b6220 0x7f19f80d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:40652/0 (socket says 172.21.15.40:40652) 2024-04-26T23:02:13.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.443+0000 7f19f6a00640 1 -- 172.21.15.40:0/787844441 learned_addr learned my addr 172.21.15.40:0/787844441 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:13.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.444+0000 7f19f6a00640 1 -- 172.21.15.40:0/787844441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f80d39f0 msgr2=0x7f19f80b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:13.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.444+0000 7f19f6a00640 1 --2- 172.21.15.40:0/787844441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f80d39f0 0x7f19f80b8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.444+0000 7f19f6a00640 1 -- 172.21.15.40:0/787844441 >> v1:172.21.15.103:6789/0 conn(0x7f19f80d08d0 legacy=0x7f19f80d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:13.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.444+0000 7f19f6a00640 1 -- 172.21.15.40:0/787844441 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f80b7f60 con 0x7f19f80b6220 2024-04-26T23:02:13.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.445+0000 7f19f6a00640 1 --2- 172.21.15.40:0/787844441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f80b6220 0x7f19f80d0390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f19e8009870 tx=0x7f19e802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4668709acbd0eafa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:13.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.445+0000 7f19f5600640 1 -- 172.21.15.40:0/787844441 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e8037070 con 0x7f19f80b6220 2024-04-26T23:02:13.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.445+0000 7f19f5600640 1 -- 172.21.15.40:0/787844441 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19e8032440 con 0x7f19f80b6220 2024-04-26T23:02:13.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.445+0000 7f19f5600640 1 -- 172.21.15.40:0/787844441 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e8032810 con 0x7f19f80b6220 2024-04-26T23:02:13.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.446+0000 7f19f7e00640 1 -- 172.21.15.40:0/787844441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f80b6220 msgr2=0x7f19f80d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:13.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.446+0000 7f19f7e00640 1 --2- 172.21.15.40:0/787844441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f80b6220 0x7f19f80d0390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f19e8009870 tx=0x7f19e802a270 comp rx=0 tx=0).stop 2024-04-26T23:02:13.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.446+0000 7f19f7e00640 1 -- 172.21.15.40:0/787844441 shutdown_connections 2024-04-26T23:02:13.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.446+0000 7f19f7e00640 1 --2- 172.21.15.40:0/787844441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f80d39f0 0x7f19f80b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.446+0000 7f19f7e00640 1 --2- 172.21.15.40:0/787844441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f80b6220 0x7f19f80d0390 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.446+0000 7f19f7e00640 1 -- 172.21.15.40:0/787844441 >> 172.21.15.40:0/787844441 conn(0x7f19f80d4d90 msgr2=0x7f19f80d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:13.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.446+0000 7f19f7e00640 1 -- 172.21.15.40:0/787844441 shutdown_connections 2024-04-26T23:02:13.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.446+0000 7f19f7e00640 1 -- 172.21.15.40:0/787844441 wait complete. 2024-04-26T23:02:13.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.448+0000 7f19f7e00640 1 Processor -- start 2024-04-26T23:02:13.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.448+0000 7f19f7e00640 1 -- start start 2024-04-26T23:02:13.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.449+0000 7f19f7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f80b6220 0x7f19f80a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:13.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.449+0000 7f19f7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f80d08d0 0x7f19f815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:13.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.449+0000 7f19f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19f80d39f0 0x7f19f809ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:13.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.449+0000 7f19f7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f19f80b9260 con 0x7f19f80d39f0 2024-04-26T23:02:13.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.449+0000 7f19f7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f19f80b8520 con 0x7f19f80d08d0 2024-04-26T23:02:13.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.449+0000 7f19f7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f19f80b93e0 con 0x7f19f80b6220 2024-04-26T23:02:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.450+0000 7f19f7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19f80d39f0 0x7f19f809ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.450+0000 7f19f7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19f80d39f0 0x7f19f809ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36516/0 (socket says 172.21.15.40:36516) 2024-04-26T23:02:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.450+0000 7f19f7400640 1 -- 172.21.15.40:0/2464750556 learned_addr learned my addr 172.21.15.40:0/2464750556 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.450+0000 7f19f6000640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f80d08d0 0x7f19f815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.450+0000 7f19f6a00640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f80b6220 0x7f19f80a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.450+0000 7f19f7400640 1 -- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f80b6220 msgr2=0x7f19f80a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.450+0000 7f19f7400640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f80b6220 0x7f19f80a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.450+0000 7f19f7400640 1 -- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f80d08d0 msgr2=0x7f19f815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.450+0000 7f19f7400640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f80d08d0 0x7f19f815ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.450+0000 7f19f7400640 1 -- 172.21.15.40:0/2464750556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f809f200 con 0x7f19f80d39f0 2024-04-26T23:02:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.451+0000 7f19f7400640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19f80d39f0 0x7f19f809ea90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f19f000efc0 tx=0x7f19f000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.451+0000 7f19eea00640 1 -- 172.21.15.40:0/2464750556 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f0018020 con 0x7f19f80d39f0 2024-04-26T23:02:13.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.451+0000 7f19eea00640 1 -- 172.21.15.40:0/2464750556 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19f0014480 con 0x7f19f80d39f0 2024-04-26T23:02:13.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.451+0000 7f19eea00640 1 -- 172.21.15.40:0/2464750556 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f0009e60 con 0x7f19f80d39f0 2024-04-26T23:02:13.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.451+0000 7f19f7e00640 1 -- 172.21.15.40:0/2464750556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19f809f4e0 con 0x7f19f80d39f0 2024-04-26T23:02:13.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.451+0000 7f19f7e00640 1 -- 172.21.15.40:0/2464750556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19f809faa0 con 0x7f19f80d39f0 2024-04-26T23:02:13.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.453+0000 7f19f7e00640 1 -- 172.21.15.40:0/2464750556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19f8069370 con 0x7f19f80d39f0 2024-04-26T23:02:13.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.454+0000 7f19eea00640 1 -- 172.21.15.40:0/2464750556 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f19f0005d10 con 0x7f19f80d39f0 2024-04-26T23:02:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.455+0000 7f19eea00640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f19d406c8f0 0x7f19d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.455+0000 7f19eea00640 1 -- 172.21.15.40:0/2464750556 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f19f008cc30 con 0x7f19f80d39f0 2024-04-26T23:02:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.456+0000 7f19f6a00640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f19d406c8f0 0x7f19d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:13.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.457+0000 7f19f6a00640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f19d406c8f0 0x7f19d406edb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f19e8034000 tx=0x7f19e802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:13.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.460+0000 7f19eea00640 1 -- 172.21.15.40:0/2464750556 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f19f0022030 con 0x7f19f80d39f0 2024-04-26T23:02:13.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.666+0000 7f19f7e00640 1 -- 172.21.15.40:0/2464750556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19f80d30c0 con 0x7f19f80d39f0 2024-04-26T23:02:13.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.668+0000 7f19eea00640 1 -- 172.21.15.40:0/2464750556 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18812 (secure 0 0 0) 0x7f19f0010040 con 0x7f19f80d39f0 2024-04-26T23:02:13.668 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:13.669 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:46.975683+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.670+0000 7f19f7e00640 1 -- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f19d406c8f0 msgr2=0x7f19d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.670+0000 7f19f7e00640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f19d406c8f0 0x7f19d406edb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f19e8034000 tx=0x7f19e802e000 comp rx=0 tx=0).stop 2024-04-26T23:02:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.670+0000 7f19f7e00640 1 -- 172.21.15.40:0/2464750556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19f80d39f0 msgr2=0x7f19f809ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.670+0000 7f19f7e00640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19f80d39f0 0x7f19f809ea90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f19f000efc0 tx=0x7f19f000c490 comp rx=0 tx=0).stop 2024-04-26T23:02:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.671+0000 7f19f7e00640 1 -- 172.21.15.40:0/2464750556 shutdown_connections 2024-04-26T23:02:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.671+0000 7f19f7e00640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f19d406c8f0 0x7f19d406edb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.671+0000 7f19f7e00640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19f80d39f0 0x7f19f809ea90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.671+0000 7f19f7e00640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f80d08d0 0x7f19f815ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.671+0000 7f19f7e00640 1 --2- 172.21.15.40:0/2464750556 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f19f80b6220 0x7f19f80a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.671+0000 7f19f7e00640 1 -- 172.21.15.40:0/2464750556 >> 172.21.15.40:0/2464750556 conn(0x7f19f80d4d90 msgr2=0x7f19f80d32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:13.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.671+0000 7f19f7e00640 1 -- 172.21.15.40:0/2464750556 shutdown_connections 2024-04-26T23:02:13.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.671+0000 7f19f7e00640 1 -- 172.21.15.40:0/2464750556 wait complete. 2024-04-26T23:02:13.690 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:02:13.691 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:02:13.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.913+0000 7f97b5600640 1 Processor -- start 2024-04-26T23:02:13.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.914+0000 7f97b5600640 1 -- start start 2024-04-26T23:02:13.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.915+0000 7f97b5600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97b00b5500 0x7f97b00b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:13.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.915+0000 7f97b5600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f97b00b59b0 0x7f97b00b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:13.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.915+0000 7f97b5600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b00d3dc0 con 0x7f97b00b5120 2024-04-26T23:02:13.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.915+0000 7f97b5600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f97b00d03b0 con 0x7f97b00b59b0 2024-04-26T23:02:13.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.915+0000 7f97b5600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f97b00b75b0 con 0x7f97b00b5500 2024-04-26T23:02:13.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.916+0000 7f97b4c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f97b00b59b0 0x7f97b00b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:13.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.916+0000 7f97a7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97b00b5500 0x7f97b00b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:13.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.916+0000 7f97b4c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f97b00b59b0 0x7f97b00b6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:40668/0 (socket says 172.21.15.40:40668) 2024-04-26T23:02:13.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.916+0000 7f97b4c00640 1 -- 172.21.15.40:0/2947874985 learned_addr learned my addr 172.21.15.40:0/2947874985 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:13.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.917+0000 7f97b4c00640 1 -- 172.21.15.40:0/2947874985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97b00b5500 msgr2=0x7f97b00b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:13.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.917+0000 7f97b4c00640 1 --2- 172.21.15.40:0/2947874985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97b00b5500 0x7f97b00b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.917+0000 7f97b4c00640 1 -- 172.21.15.40:0/2947874985 >> v1:172.21.15.103:6789/0 conn(0x7f97b00b5120 legacy=0x7f97b00b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:13.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.917+0000 7f97b4c00640 1 -- 172.21.15.40:0/2947874985 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b00d0600 con 0x7f97b00b59b0 2024-04-26T23:02:13.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.918+0000 7f97b4c00640 1 --2- 172.21.15.40:0/2947874985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97b00b59b0 0x7f97b00b6f40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009e50 tx=0x7f97a802a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56ac01f392b98a63 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:13.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.919+0000 7f97af400640 1 -- 172.21.15.40:0/2947874985 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a8036070 con 0x7f97b00b59b0 2024-04-26T23:02:13.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.919+0000 7f97af400640 1 -- 172.21.15.40:0/2947874985 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a8031600 con 0x7f97b00b59b0 2024-04-26T23:02:13.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.919+0000 7f97af400640 1 -- 172.21.15.40:0/2947874985 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a80319d0 con 0x7f97b00b59b0 2024-04-26T23:02:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.919+0000 7f97b5600640 1 -- 172.21.15.40:0/2947874985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97b00b59b0 msgr2=0x7f97b00b6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.919+0000 7f97b5600640 1 --2- 172.21.15.40:0/2947874985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97b00b59b0 0x7f97b00b6f40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009e50 tx=0x7f97a802a5f0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.920+0000 7f97b5600640 1 -- 172.21.15.40:0/2947874985 shutdown_connections 2024-04-26T23:02:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.920+0000 7f97b5600640 1 --2- 172.21.15.40:0/2947874985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97b00b59b0 0x7f97b00b6f40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.920+0000 7f97b5600640 1 --2- 172.21.15.40:0/2947874985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97b00b5500 0x7f97b00b5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.920+0000 7f97b5600640 1 -- 172.21.15.40:0/2947874985 >> 172.21.15.40:0/2947874985 conn(0x7f97b00d4db0 msgr2=0x7f97b00d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.920+0000 7f97b5600640 1 -- 172.21.15.40:0/2947874985 shutdown_connections 2024-04-26T23:02:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.920+0000 7f97b5600640 1 -- 172.21.15.40:0/2947874985 wait complete. 2024-04-26T23:02:13.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.921+0000 7f97b5600640 1 Processor -- start 2024-04-26T23:02:13.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.922+0000 7f97b5600640 1 -- start start 2024-04-26T23:02:13.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.922+0000 7f97b5600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97b00b5120 0x7f97b00b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:13.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.923+0000 7f97b5600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97b00b5500 0x7f97b00b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:13.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.923+0000 7f97b5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97b00b59b0 0x7f97b00b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:13.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.923+0000 7f97b5600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f97b00b7c00 con 0x7f97b00b59b0 2024-04-26T23:02:13.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.923+0000 7f97b5600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f97b00d0bc0 con 0x7f97b00b5120 2024-04-26T23:02:13.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.923+0000 7f97b5600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f97b00b7d80 con 0x7f97b00b5500 2024-04-26T23:02:13.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.923+0000 7f97b4c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97b00b59b0 0x7f97b00b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:13.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.923+0000 7f97b4c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97b00b59b0 0x7f97b00b2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36520/0 (socket says 172.21.15.40:36520) 2024-04-26T23:02:13.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.923+0000 7f97b4c00640 1 -- 172.21.15.40:0/3264057468 learned_addr learned my addr 172.21.15.40:0/3264057468 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:13.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.923+0000 7f97a7e00640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97b00b5500 0x7f97b00b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:13.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.923+0000 7f97afe00640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97b00b5120 0x7f97b00b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:13.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.923+0000 7f97b4c00640 1 -- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97b00b5500 msgr2=0x7f97b00b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:13.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.923+0000 7f97b4c00640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97b00b5500 0x7f97b00b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.924+0000 7f97b4c00640 1 -- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97b00b5120 msgr2=0x7f97b00b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:13.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.924+0000 7f97b4c00640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97b00b5120 0x7f97b00b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:13.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.924+0000 7f97b4c00640 1 -- 172.21.15.40:0/3264057468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b009f6e0 con 0x7f97b00b59b0 2024-04-26T23:02:13.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.924+0000 7f97b4c00640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97b00b59b0 0x7f97b00b2f50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009e50 tx=0x7f97a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:13.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.924+0000 7f97acc00640 1 -- 172.21.15.40:0/3264057468 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a8041020 con 0x7f97b00b59b0 2024-04-26T23:02:13.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.924+0000 7f97acc00640 1 -- 172.21.15.40:0/3264057468 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a80077a0 con 0x7f97b00b59b0 2024-04-26T23:02:13.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.924+0000 7f97acc00640 1 -- 172.21.15.40:0/3264057468 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a8036040 con 0x7f97b00b59b0 2024-04-26T23:02:13.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.924+0000 7f97b5600640 1 -- 172.21.15.40:0/3264057468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b015f100 con 0x7f97b00b59b0 2024-04-26T23:02:13.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.924+0000 7f97b5600640 1 -- 172.21.15.40:0/3264057468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b009f9b0 con 0x7f97b00b59b0 2024-04-26T23:02:13.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.926+0000 7f97b5600640 1 -- 172.21.15.40:0/3264057468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b0069370 con 0x7f97b00b59b0 2024-04-26T23:02:13.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.927+0000 7f97acc00640 1 -- 172.21.15.40:0/3264057468 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f97a8044020 con 0x7f97b00b59b0 2024-04-26T23:02:13.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.928+0000 7f97acc00640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f978c06c8f0 0x7f978c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:13.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.928+0000 7f97acc00640 1 -- 172.21.15.40:0/3264057468 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f97a80acbf0 con 0x7f97b00b59b0 2024-04-26T23:02:13.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.928+0000 7f97afe00640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f978c06c8f0 0x7f978c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:13.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.930+0000 7f97afe00640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f978c06c8f0 0x7f978c06edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f97a0001640 tx=0x7f97a0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:13.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:13.932+0000 7f97acc00640 1 -- 172.21.15.40:0/3264057468 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f97a804b030 con 0x7f97b00b59b0 2024-04-26T23:02:14.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.136+0000 7f97b5600640 1 -- 172.21.15.40:0/3264057468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f97b0068db0 con 0x7f97b00b59b0 2024-04-26T23:02:14.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.137+0000 7f97acc00640 1 -- 172.21.15.40:0/3264057468 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f97b009f6e0 con 0x7f97b00b59b0 2024-04-26T23:02:14.137 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:02:14.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.139+0000 7f97b5600640 1 -- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f978c06c8f0 msgr2=0x7f978c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.139+0000 7f97b5600640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f978c06c8f0 0x7f978c06edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f97a0001640 tx=0x7f97a0009040 comp rx=0 tx=0).stop 2024-04-26T23:02:14.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.139+0000 7f97b5600640 1 -- 172.21.15.40:0/3264057468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97b00b59b0 msgr2=0x7f97b00b2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.139+0000 7f97b5600640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97b00b59b0 0x7f97b00b2f50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009e50 tx=0x7f97a8004a40 comp rx=0 tx=0).stop 2024-04-26T23:02:14.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.140+0000 7f97b5600640 1 -- 172.21.15.40:0/3264057468 shutdown_connections 2024-04-26T23:02:14.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.140+0000 7f97b5600640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f978c06c8f0 0x7f978c06edb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.140+0000 7f97b5600640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97b00b59b0 0x7f97b00b2f50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.140+0000 7f97b5600640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97b00b5500 0x7f97b00b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.140+0000 7f97b5600640 1 --2- 172.21.15.40:0/3264057468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97b00b5120 0x7f97b00b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.140+0000 7f97b5600640 1 -- 172.21.15.40:0/3264057468 >> 172.21.15.40:0/3264057468 conn(0x7f97b00d4db0 msgr2=0x7f97b006e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:14.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.140+0000 7f97b5600640 1 -- 172.21.15.40:0/3264057468 shutdown_connections 2024-04-26T23:02:14.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.140+0000 7f97b5600640 1 -- 172.21.15.40:0/3264057468 wait complete. 2024-04-26T23:02:14.159 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:02:14.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.366+0000 7f715de00640 1 Processor -- start 2024-04-26T23:02:14.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.367+0000 7f715de00640 1 -- start start 2024-04-26T23:02:14.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.368+0000 7f715de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b5120 0x7f71580b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.368+0000 7f715de00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71580d3dc0 con 0x7f71580b59b0 2024-04-26T23:02:14.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.368+0000 7f715de00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71580b75b0 con 0x7f71580b5500 2024-04-26T23:02:14.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.368+0000 7f715de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f71580d03b0 con 0x7f71580b5120 2024-04-26T23:02:14.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.369+0000 7f715d400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f71580b59b0 0x7f71580b7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52094/0 (socket says 172.21.15.40:52094) 2024-04-26T23:02:14.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.369+0000 7f715d400640 1 -- 172.21.15.40:0/3438166216 learned_addr learned my addr 172.21.15.40:0/3438166216 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:14.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.369+0000 7f715ca00640 1 --2- 172.21.15.40:0/3438166216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b5120 0x7f71580b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.369+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3343669884 0 0) 0x7f71580d3dc0 con 0x7f71580b59b0 2024-04-26T23:02:14.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f713c003520 con 0x7f71580b59b0 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 990439686 0 0) 0x7f713c003520 con 0x7f71580b59b0 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b5120 msgr2=0x7f71580b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 --2- 172.21.15.40:0/3438166216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b5120 0x7f71580b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 >> v1:172.21.15.103:6789/0 conn(0x7f71580b5500 legacy=0x7f71580b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71580d0600 con 0x7f71580b59b0 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f7150001670 con 0x7f71580b59b0 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 >> v1:172.21.15.40:6789/0 conn(0x7f71580b59b0 legacy=0x7f71580b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 --2- 172.21.15.40:0/3438166216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0039e0 0x7f713c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 --2- 172.21.15.40:0/3438166216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b5120 0x7f713c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 --2- 172.21.15.40:0/3438166216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f713c004d40 0x7f713c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f71580d03b0 con 0x7f713c0039e0 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f713c00baf0 con 0x7f713c004d40 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f713c00bcf0 con 0x7f71580b5120 2024-04-26T23:02:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.370+0000 7f715ca00640 1 --2- 172.21.15.40:0/3438166216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0039e0 0x7f713c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.371+0000 7f715d400640 1 --2- 172.21.15.40:0/3438166216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f713c004d40 0x7f713c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.371+0000 7f7157e00640 1 --2- 172.21.15.40:0/3438166216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b5120 0x7f713c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.371+0000 7f715ca00640 1 -- 172.21.15.40:0/3438166216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b5120 msgr2=0x7f713c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:14.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.371+0000 7f715ca00640 1 --2- 172.21.15.40:0/3438166216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b5120 0x7f713c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.371+0000 7f715ca00640 1 -- 172.21.15.40:0/3438166216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f713c004d40 msgr2=0x7f713c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:14.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.371+0000 7f715ca00640 1 --2- 172.21.15.40:0/3438166216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f713c004d40 0x7f713c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.371+0000 7f715ca00640 1 -- 172.21.15.40:0/3438166216 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713c003520 con 0x7f713c0039e0 2024-04-26T23:02:14.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.371+0000 7f7157e00640 1 --2- 172.21.15.40:0/3438166216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71580b5120 0x7f713c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:14.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.372+0000 7f715ca00640 1 --2- 172.21.15.40:0/3438166216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0039e0 0x7f713c003e00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7148009fc0 tx=0x7f713c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd574d44e7ca7af2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:14.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.372+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7148038070 con 0x7f713c0039e0 2024-04-26T23:02:14.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.372+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f714802f070 con 0x7f713c0039e0 2024-04-26T23:02:14.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.372+0000 7f7157400640 1 -- 172.21.15.40:0/3438166216 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7148009c10 con 0x7f713c0039e0 2024-04-26T23:02:14.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.373+0000 7f715de00640 1 -- 172.21.15.40:0/3438166216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0039e0 msgr2=0x7f713c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.373+0000 7f715de00640 1 --2- 172.21.15.40:0/3438166216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0039e0 0x7f713c003e00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7148009fc0 tx=0x7f713c005830 comp rx=0 tx=0).stop 2024-04-26T23:02:14.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.373+0000 7f715d400640 1 -- 172.21.15.40:0/3438166216 reap_dead start 2024-04-26T23:02:14.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.373+0000 7f715de00640 1 -- 172.21.15.40:0/3438166216 shutdown_connections 2024-04-26T23:02:14.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.373+0000 7f715de00640 1 -- 172.21.15.40:0/3438166216 >> 172.21.15.40:0/3438166216 conn(0x7f71580d4db0 msgr2=0x7f71580d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:14.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.373+0000 7f715de00640 1 -- 172.21.15.40:0/3438166216 shutdown_connections 2024-04-26T23:02:14.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.373+0000 7f715de00640 1 -- 172.21.15.40:0/3438166216 wait complete. 2024-04-26T23:02:14.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.375+0000 7f715de00640 1 Processor -- start 2024-04-26T23:02:14.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.375+0000 7f715de00640 1 -- start start 2024-04-26T23:02:14.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.376+0000 7f715de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f713c0039e0 0x7f71581f4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.376+0000 7f715de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71580b19a0 0x7f71580b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.376+0000 7f715de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71580cd000 0x7f71580cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.376+0000 7f715de00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f71580b5ee0 con 0x7f71580b19a0 2024-04-26T23:02:14.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.376+0000 7f715de00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f71580d0bc0 con 0x7f71580cd000 2024-04-26T23:02:14.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.376+0000 7f715de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f71580b6060 con 0x7f713c0039e0 2024-04-26T23:02:14.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.376+0000 7f7157e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71580b19a0 0x7f71580b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.376+0000 7f7157e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71580b19a0 0x7f71580b1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36536/0 (socket says 172.21.15.40:36536) 2024-04-26T23:02:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.376+0000 7f7157e00640 1 -- 172.21.15.40:0/2435122419 learned_addr learned my addr 172.21.15.40:0/2435122419 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.377+0000 7f715ca00640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f713c0039e0 0x7f71581f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.377+0000 7f715d400640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71580cd000 0x7f71580cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.377+0000 7f7157e00640 1 -- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f713c0039e0 msgr2=0x7f71581f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.377+0000 7f7157e00640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f713c0039e0 0x7f71581f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.377+0000 7f7157e00640 1 -- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71580cd000 msgr2=0x7f71580cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.377+0000 7f7157e00640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71580cd000 0x7f71580cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.377+0000 7f7157e00640 1 -- 172.21.15.40:0/2435122419 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71580cdd50 con 0x7f71580b19a0 2024-04-26T23:02:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.377+0000 7f715d400640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71580cd000 0x7f71580cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.377+0000 7f7157e00640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71580b19a0 0x7f71580b1e20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7140000f60 tx=0x7f714000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:14.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.378+0000 7f7154c00640 1 -- 172.21.15.40:0/2435122419 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7140005270 con 0x7f71580b19a0 2024-04-26T23:02:14.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.378+0000 7f7154c00640 1 -- 172.21.15.40:0/2435122419 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7140017070 con 0x7f71580b19a0 2024-04-26T23:02:14.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.378+0000 7f7154c00640 1 -- 172.21.15.40:0/2435122419 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714000b6b0 con 0x7f71580b19a0 2024-04-26T23:02:14.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.378+0000 7f715de00640 1 -- 172.21.15.40:0/2435122419 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71580ce030 con 0x7f71580b19a0 2024-04-26T23:02:14.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.378+0000 7f715de00640 1 -- 172.21.15.40:0/2435122419 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71580ce5f0 con 0x7f71580b19a0 2024-04-26T23:02:14.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.379+0000 7f715de00640 1 -- 172.21.15.40:0/2435122419 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7158069370 con 0x7f71580b19a0 2024-04-26T23:02:14.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.380+0000 7f7154c00640 1 -- 172.21.15.40:0/2435122419 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f714000b850 con 0x7f71580b19a0 2024-04-26T23:02:14.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.381+0000 7f7154c00640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f713406c920 0x7f713406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.381+0000 7f7154c00640 1 -- 172.21.15.40:0/2435122419 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f714008bd50 con 0x7f71580b19a0 2024-04-26T23:02:14.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.382+0000 7f715ca00640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f713406c920 0x7f713406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.383+0000 7f715ca00640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f713406c920 0x7f713406ede0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7148035000 tx=0x7f714802f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:14.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.386+0000 7f7154c00640 1 -- 172.21.15.40:0/2435122419 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7140019310 con 0x7f71580b19a0 2024-04-26T23:02:14.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.593+0000 7f715de00640 1 -- 172.21.15.40:0/2435122419 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f715815f500 con 0x7f71580b19a0 2024-04-26T23:02:14.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.594+0000 7f7154c00640 1 -- 172.21.15.40:0/2435122419 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f7140059d70 con 0x7f71580b19a0 2024-04-26T23:02:14.594 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.596+0000 7f715de00640 1 -- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f713406c920 msgr2=0x7f713406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.596+0000 7f715de00640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f713406c920 0x7f713406ede0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7148035000 tx=0x7f714802f000 comp rx=0 tx=0).stop 2024-04-26T23:02:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.597+0000 7f715de00640 1 -- 172.21.15.40:0/2435122419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71580b19a0 msgr2=0x7f71580b1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.597+0000 7f715de00640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71580b19a0 0x7f71580b1e20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7140000f60 tx=0x7f714000dc00 comp rx=0 tx=0).stop 2024-04-26T23:02:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.597+0000 7f715de00640 1 -- 172.21.15.40:0/2435122419 shutdown_connections 2024-04-26T23:02:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.597+0000 7f715de00640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f713406c920 0x7f713406ede0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.597+0000 7f715de00640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71580cd000 0x7f71580cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.597+0000 7f715de00640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71580b19a0 0x7f71580b1e20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.597+0000 7f715de00640 1 --2- 172.21.15.40:0/2435122419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f713c0039e0 0x7f71581f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.597+0000 7f715de00640 1 -- 172.21.15.40:0/2435122419 >> 172.21.15.40:0/2435122419 conn(0x7f71580d4db0 msgr2=0x7f71581611b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:14.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.597+0000 7f715de00640 1 -- 172.21.15.40:0/2435122419 shutdown_connections 2024-04-26T23:02:14.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.597+0000 7f715de00640 1 -- 172.21.15.40:0/2435122419 wait complete. 2024-04-26T23:02:14.616 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:02:14.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.831+0000 7f8bc1000640 1 Processor -- start 2024-04-26T23:02:14.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.832+0000 7f8bc1000640 1 -- start start 2024-04-26T23:02:14.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.833+0000 7f8bc1000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bbc0d2ae0 0x7f8bbc0b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.833+0000 7f8bc1000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bbc0b8bd0 con 0x7f8bbc15f1a0 2024-04-26T23:02:14.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.834+0000 7f8bc1000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bbc0b8eb0 con 0x7f8bbc0d39d0 2024-04-26T23:02:14.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.834+0000 7f8bc1000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8bbc0b9190 con 0x7f8bbc0d2ae0 2024-04-26T23:02:14.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.834+0000 7f8bbb400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8bbc15f1a0 0x7f8bbc0d32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52106/0 (socket says 172.21.15.40:52106) 2024-04-26T23:02:14.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.834+0000 7f8bbb400640 1 -- 172.21.15.40:0/1676377072 learned_addr learned my addr 172.21.15.40:0/1676377072 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:14.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.834+0000 7f8bbbe00640 1 --2- 172.21.15.40:0/1676377072 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bbc0d2ae0 0x7f8bbc0b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.834+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3726635720 0 0) 0x7f8bbc0b8bd0 con 0x7f8bbc15f1a0 2024-04-26T23:02:14.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ba0003520 con 0x7f8bbc15f1a0 2024-04-26T23:02:14.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4180926738 0 0) 0x7f8ba0003520 con 0x7f8bbc15f1a0 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bbc0d2ae0 msgr2=0x7f8bbc0b85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 --2- 172.21.15.40:0/1676377072 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bbc0d2ae0 0x7f8bbc0b85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 >> v1:172.21.15.103:6789/0 conn(0x7f8bbc0d39d0 legacy=0x7f8bbc0b9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bbc0b9470 con 0x7f8bbc15f1a0 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f8bac0019f0 con 0x7f8bbc15f1a0 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 >> v1:172.21.15.40:6789/0 conn(0x7f8bbc15f1a0 legacy=0x7f8bbc0d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 --2- 172.21.15.40:0/1676377072 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bbc0d2ae0 0x7f8ba00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 --2- 172.21.15.40:0/1676377072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ba0003fb0 0x7f8ba0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 --2- 172.21.15.40:0/1676377072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ba0004d40 0x7f8ba00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc0b9190 con 0x7f8ba0003fb0 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba000bb00 con 0x7f8ba0004d40 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba000bd00 con 0x7f8bbc0d2ae0 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.835+0000 7f8bbaa00640 1 --2- 172.21.15.40:0/1676377072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ba0003fb0 0x7f8ba0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.836+0000 7f8bbbe00640 1 --2- 172.21.15.40:0/1676377072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ba0004d40 0x7f8ba00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.836+0000 7f8bbb400640 1 --2- 172.21.15.40:0/1676377072 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bbc0d2ae0 0x7f8ba00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.836+0000 7f8bbaa00640 1 -- 172.21.15.40:0/1676377072 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bbc0d2ae0 msgr2=0x7f8ba00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.836+0000 7f8bbaa00640 1 --2- 172.21.15.40:0/1676377072 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bbc0d2ae0 0x7f8ba00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.836+0000 7f8bbaa00640 1 -- 172.21.15.40:0/1676377072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ba0004d40 msgr2=0x7f8ba00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.836+0000 7f8bbaa00640 1 --2- 172.21.15.40:0/1676377072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ba0004d40 0x7f8ba00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.836+0000 7f8bbaa00640 1 -- 172.21.15.40:0/1676377072 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba0003520 con 0x7f8ba0003fb0 2024-04-26T23:02:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.838+0000 7f8bbaa00640 1 --2- 172.21.15.40:0/1676377072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ba0003fb0 0x7f8ba0004800 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8001750 tx=0x7f8ba0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ed41808b1d0a1b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.838+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8001370 con 0x7f8ba0003fb0 2024-04-26T23:02:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.838+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba80052d0 con 0x7f8ba0003fb0 2024-04-26T23:02:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.838+0000 7f8bba000640 1 -- 172.21.15.40:0/1676377072 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba802ab20 con 0x7f8ba0003fb0 2024-04-26T23:02:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.838+0000 7f8bc1000640 1 -- 172.21.15.40:0/1676377072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ba0003fb0 msgr2=0x7f8ba0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.838+0000 7f8bc1000640 1 --2- 172.21.15.40:0/1676377072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ba0003fb0 0x7f8ba0004800 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8001750 tx=0x7f8ba0005830 comp rx=0 tx=0).stop 2024-04-26T23:02:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.838+0000 7f8bbbe00640 1 -- 172.21.15.40:0/1676377072 reap_dead start 2024-04-26T23:02:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.838+0000 7f8bc1000640 1 -- 172.21.15.40:0/1676377072 shutdown_connections 2024-04-26T23:02:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.838+0000 7f8bc1000640 1 -- 172.21.15.40:0/1676377072 >> 172.21.15.40:0/1676377072 conn(0x7f8bbc0d4d90 msgr2=0x7f8bbc0d5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.839+0000 7f8bc1000640 1 -- 172.21.15.40:0/1676377072 shutdown_connections 2024-04-26T23:02:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.839+0000 7f8bc1000640 1 -- 172.21.15.40:0/1676377072 wait complete. 2024-04-26T23:02:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.840+0000 7f8bc1000640 1 Processor -- start 2024-04-26T23:02:14.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.841+0000 7f8bc1000640 1 -- start start 2024-04-26T23:02:14.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.841+0000 7f8bc1000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bbc152150 0x7f8bbc161810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.841+0000 7f8bc1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bbc161d50 0x7f8bbc0d1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.842+0000 7f8bc1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bbc0d1d50 0x7f8bbc1605d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.842+0000 7f8bc1000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc0b85f0 con 0x7f8bbc161d50 2024-04-26T23:02:14.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.842+0000 7f8bc1000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc0d0bb0 con 0x7f8bbc0d1d50 2024-04-26T23:02:14.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.842+0000 7f8bc1000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8bbc0b8770 con 0x7f8bbc152150 2024-04-26T23:02:14.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.842+0000 7f8bbaa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bbc161d50 0x7f8bbc0d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.842+0000 7f8bbaa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bbc161d50 0x7f8bbc0d1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36562/0 (socket says 172.21.15.40:36562) 2024-04-26T23:02:14.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.842+0000 7f8bbaa00640 1 -- 172.21.15.40:0/1784343751 learned_addr learned my addr 172.21.15.40:0/1784343751 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:14.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.842+0000 7f8bbbe00640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bbc0d1d50 0x7f8bbc1605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.842+0000 7f8bbb400640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bbc152150 0x7f8bbc161810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.842+0000 7f8bbaa00640 1 -- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bbc152150 msgr2=0x7f8bbc161810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.842+0000 7f8bbaa00640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bbc152150 0x7f8bbc161810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.842+0000 7f8bbaa00640 1 -- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bbc0d1d50 msgr2=0x7f8bbc1605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:14.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.842+0000 7f8bbaa00640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bbc0d1d50 0x7f8bbc1605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:14.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.843+0000 7f8bbaa00640 1 -- 172.21.15.40:0/1784343751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bbc15b670 con 0x7f8bbc161d50 2024-04-26T23:02:14.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.843+0000 7f8bbaa00640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bbc161d50 0x7f8bbc0d1810 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8ba802a450 tx=0x7f8ba8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.843+0000 7f8bb3400640 1 -- 172.21.15.40:0/1784343751 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8007a30 con 0x7f8bbc161d50 2024-04-26T23:02:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.843+0000 7f8bb3400640 1 -- 172.21.15.40:0/1784343751 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba8040070 con 0x7f8bbc161d50 2024-04-26T23:02:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.843+0000 7f8bc1000640 1 -- 172.21.15.40:0/1784343751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bbc1f4ef0 con 0x7f8bbc161d50 2024-04-26T23:02:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.843+0000 7f8bb3400640 1 -- 172.21.15.40:0/1784343751 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8001370 con 0x7f8bbc161d50 2024-04-26T23:02:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.843+0000 7f8bc1000640 1 -- 172.21.15.40:0/1784343751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bbc09ef10 con 0x7f8bbc161d50 2024-04-26T23:02:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.844+0000 7f8bc1000640 1 -- 172.21.15.40:0/1784343751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bbc069370 con 0x7f8bbc161d50 2024-04-26T23:02:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.846+0000 7f8bb3400640 1 -- 172.21.15.40:0/1784343751 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8ba803a070 con 0x7f8bbc161d50 2024-04-26T23:02:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.846+0000 7f8bb3400640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8b9c06c8f0 0x7f8b9c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.847+0000 7f8bb3400640 1 -- 172.21.15.40:0/1784343751 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8ba80ab380 con 0x7f8bbc161d50 2024-04-26T23:02:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.847+0000 7f8bbb400640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8b9c06c8f0 0x7f8b9c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.848+0000 7f8bbb400640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8b9c06c8f0 0x7f8b9c06edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0013d0 tx=0x7f8bac009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:14.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:14.851+0000 7f8bb3400640 1 -- 172.21.15.40:0/1784343751 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8ba80793a0 con 0x7f8bbc161d50 2024-04-26T23:02:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.056+0000 7f8bc1000640 1 -- 172.21.15.40:0/1784343751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8bbc0d2130 con 0x7f8bbc161d50 2024-04-26T23:02:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.057+0000 7f8bb3400640 1 -- 172.21.15.40:0/1784343751 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f8ba8078ca0 con 0x7f8bbc161d50 2024-04-26T23:02:15.057 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:02:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.059+0000 7f8bc1000640 1 -- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8b9c06c8f0 msgr2=0x7f8b9c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.059+0000 7f8bc1000640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8b9c06c8f0 0x7f8b9c06edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0013d0 tx=0x7f8bac009040 comp rx=0 tx=0).stop 2024-04-26T23:02:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.059+0000 7f8bc1000640 1 -- 172.21.15.40:0/1784343751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bbc161d50 msgr2=0x7f8bbc0d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.059+0000 7f8bc1000640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bbc161d50 0x7f8bbc0d1810 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8ba802a450 tx=0x7f8ba8001120 comp rx=0 tx=0).stop 2024-04-26T23:02:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.060+0000 7f8bc1000640 1 -- 172.21.15.40:0/1784343751 shutdown_connections 2024-04-26T23:02:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.060+0000 7f8bc1000640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8b9c06c8f0 0x7f8b9c06edb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.060+0000 7f8bc1000640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bbc0d1d50 0x7f8bbc1605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.060+0000 7f8bc1000640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bbc161d50 0x7f8bbc0d1810 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.060+0000 7f8bc1000640 1 --2- 172.21.15.40:0/1784343751 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8bbc152150 0x7f8bbc161810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.060+0000 7f8bc1000640 1 -- 172.21.15.40:0/1784343751 >> 172.21.15.40:0/1784343751 conn(0x7f8bbc0d4d90 msgr2=0x7f8bbc0b6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.060+0000 7f8bc1000640 1 -- 172.21.15.40:0/1784343751 shutdown_connections 2024-04-26T23:02:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.060+0000 7f8bc1000640 1 -- 172.21.15.40:0/1784343751 wait complete. 2024-04-26T23:02:15.081 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:02:15.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.299+0000 7fb5ada00640 1 Processor -- start 2024-04-26T23:02:15.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.300+0000 7fb5ada00640 1 -- start start 2024-04-26T23:02:15.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.302+0000 7fb5ada00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a8073a50 0x7fb5a8072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.302+0000 7fb5ada00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5a8161880 con 0x7fb5a8160a50 2024-04-26T23:02:15.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.302+0000 7fb5ada00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5a80cff80 con 0x7fb5a80cfaf0 2024-04-26T23:02:15.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.302+0000 7fb5ada00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a8071ce0 con 0x7fb5a8073a50 2024-04-26T23:02:15.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.302+0000 7fb5a7e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb5a8160a50 0x7fb5a8160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52384/0 (socket says 172.21.15.40:52384) 2024-04-26T23:02:15.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.302+0000 7fb5a7e00640 1 -- 172.21.15.40:0/3938692840 learned_addr learned my addr 172.21.15.40:0/3938692840 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:15.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5ad000640 1 --2- 172.21.15.40:0/3938692840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a8073a50 0x7fb5a8072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2865613112 0 0) 0x7fb5a8161880 con 0x7fb5a8160a50 2024-04-26T23:02:15.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb58c003520 con 0x7fb5a8160a50 2024-04-26T23:02:15.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3778268762 0 0) 0x7fb58c003520 con 0x7fb5a8160a50 2024-04-26T23:02:15.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a8073a50 msgr2=0x7fb5a8072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:15.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 --2- 172.21.15.40:0/3938692840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a8073a50 0x7fb5a8072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 >> v1:172.21.15.103:6789/0 conn(0x7fb5a80cfaf0 legacy=0x7fb5a806dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:15.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a80d0260 con 0x7fb5a8160a50 2024-04-26T23:02:15.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fb598001670 con 0x7fb5a8160a50 2024-04-26T23:02:15.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 >> v1:172.21.15.40:6789/0 conn(0x7fb5a8160a50 legacy=0x7fb5a8160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:15.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 --2- 172.21.15.40:0/3938692840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb58c0039e0 0x7fb58c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 --2- 172.21.15.40:0/3938692840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb58c004800 0x7fb58c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 --2- 172.21.15.40:0/3938692840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a8073a50 0x7fb58c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a8071ce0 con 0x7fb58c004800 2024-04-26T23:02:15.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb58c00fee0 con 0x7fb58c0039e0 2024-04-26T23:02:15.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.303+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb58c0100e0 con 0x7fb5a8073a50 2024-04-26T23:02:15.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.304+0000 7fb5a7400640 1 --2- 172.21.15.40:0/3938692840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb58c004800 0x7fb58c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.304+0000 7fb5a7e00640 1 --2- 172.21.15.40:0/3938692840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb58c0039e0 0x7fb58c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.304+0000 7fb5ad000640 1 --2- 172.21.15.40:0/3938692840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a8073a50 0x7fb58c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.304+0000 7fb5a7400640 1 -- 172.21.15.40:0/3938692840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a8073a50 msgr2=0x7fb58c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:15.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.304+0000 7fb5a7400640 1 --2- 172.21.15.40:0/3938692840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a8073a50 0x7fb58c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.304+0000 7fb5a7400640 1 -- 172.21.15.40:0/3938692840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb58c0039e0 msgr2=0x7fb58c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:15.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.304+0000 7fb5a7400640 1 --2- 172.21.15.40:0/3938692840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb58c0039e0 0x7fb58c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.304+0000 7fb5a7400640 1 -- 172.21.15.40:0/3938692840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb58c003520 con 0x7fb58c004800 2024-04-26T23:02:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.305+0000 7fb5a7400640 1 --2- 172.21.15.40:0/3938692840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb58c004800 0x7fb58c004c80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb594001720 tx=0x7fb58c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c133e9256bf6a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.306+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb594001370 con 0x7fb58c004800 2024-04-26T23:02:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.306+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5940052d0 con 0x7fb58c004800 2024-04-26T23:02:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.306+0000 7fb5a6a00640 1 -- 172.21.15.40:0/3938692840 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59402ab90 con 0x7fb58c004800 2024-04-26T23:02:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.306+0000 7fb5ada00640 1 -- 172.21.15.40:0/3938692840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb58c004800 msgr2=0x7fb58c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.306+0000 7fb5ada00640 1 --2- 172.21.15.40:0/3938692840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb58c004800 0x7fb58c004c80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb594001720 tx=0x7fb58c005830 comp rx=0 tx=0).stop 2024-04-26T23:02:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.306+0000 7fb5ad000640 1 -- 172.21.15.40:0/3938692840 reap_dead start 2024-04-26T23:02:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.306+0000 7fb5ada00640 1 -- 172.21.15.40:0/3938692840 shutdown_connections 2024-04-26T23:02:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.306+0000 7fb5ada00640 1 -- 172.21.15.40:0/3938692840 >> 172.21.15.40:0/3938692840 conn(0x7fb5a80b2720 msgr2=0x7fb5a80b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.306+0000 7fb5ada00640 1 -- 172.21.15.40:0/3938692840 shutdown_connections 2024-04-26T23:02:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.307+0000 7fb5ada00640 1 -- 172.21.15.40:0/3938692840 wait complete. 2024-04-26T23:02:15.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.308+0000 7fb5ada00640 1 Processor -- start 2024-04-26T23:02:15.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.309+0000 7fb5ada00640 1 -- start start 2024-04-26T23:02:15.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5ada00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb58c004800 0x7fb5a80afcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5ada00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a81429b0 0x7fb5a8144fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5ada00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5a80b44a0 0x7fb5a80b4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5ada00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a8073f00 con 0x7fb5a81429b0 2024-04-26T23:02:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5ada00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a80722a0 con 0x7fb5a80b44a0 2024-04-26T23:02:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5ada00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a81468f0 con 0x7fb58c004800 2024-04-26T23:02:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5a7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a81429b0 0x7fb5a8144fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5a7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a81429b0 0x7fb5a8144fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51970/0 (socket says 172.21.15.40:51970) 2024-04-26T23:02:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5a7400640 1 -- 172.21.15.40:0/2327468358 learned_addr learned my addr 172.21.15.40:0/2327468358 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5a7e00640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb58c004800 0x7fb5a80afcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5ad000640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5a80b44a0 0x7fb5a80b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5a7400640 1 -- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb58c004800 msgr2=0x7fb5a80afcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5a7400640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb58c004800 0x7fb5a80afcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5a7400640 1 -- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5a80b44a0 msgr2=0x7fb5a80b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5a7400640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5a80b44a0 0x7fb5a80b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5a7400640 1 -- 172.21.15.40:0/2327468358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a813f260 con 0x7fb5a81429b0 2024-04-26T23:02:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.310+0000 7fb5a7e00640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb58c004800 0x7fb5a80afcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.311+0000 7fb5ad000640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5a80b44a0 0x7fb5a80b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.311+0000 7fb5a7400640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a81429b0 0x7fb5a8144fe0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb594000c00 tx=0x7fb594001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.311+0000 7fb59fe00640 1 -- 172.21.15.40:0/2327468358 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59403f030 con 0x7fb5a81429b0 2024-04-26T23:02:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.311+0000 7fb59fe00640 1 -- 172.21.15.40:0/2327468358 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb594005480 con 0x7fb5a81429b0 2024-04-26T23:02:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.311+0000 7fb59fe00640 1 -- 172.21.15.40:0/2327468358 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb594037040 con 0x7fb5a81429b0 2024-04-26T23:02:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.311+0000 7fb5ada00640 1 -- 172.21.15.40:0/2327468358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a813f540 con 0x7fb5a81429b0 2024-04-26T23:02:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.311+0000 7fb5ada00640 1 -- 172.21.15.40:0/2327468358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a813fb00 con 0x7fb5a81429b0 2024-04-26T23:02:15.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.313+0000 7fb5ada00640 1 -- 172.21.15.40:0/2327468358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a8069370 con 0x7fb5a81429b0 2024-04-26T23:02:15.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.314+0000 7fb59fe00640 1 -- 172.21.15.40:0/2327468358 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb594042070 con 0x7fb5a81429b0 2024-04-26T23:02:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.315+0000 7fb59fe00640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb58806c920 0x7fb58806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.315+0000 7fb59fe00640 1 -- 172.21.15.40:0/2327468358 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb5940ab570 con 0x7fb5a81429b0 2024-04-26T23:02:15.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.316+0000 7fb5a7e00640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb58806c920 0x7fb58806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.317+0000 7fb5a7e00640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb58806c920 0x7fb58806ede0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb598001050 tx=0x7fb598000f80 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.320+0000 7fb59fe00640 1 -- 172.21.15.40:0/2327468358 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fb5940433b0 con 0x7fb5a81429b0 2024-04-26T23:02:15.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.524+0000 7fb5ada00640 1 -- 172.21.15.40:0/2327468358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb5a806fa60 con 0x7fb5a81429b0 2024-04-26T23:02:15.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.525+0000 7fb59fe00640 1 -- 172.21.15.40:0/2327468358 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fb594079470 con 0x7fb5a81429b0 2024-04-26T23:02:15.526 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.528+0000 7fb5ada00640 1 -- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb58806c920 msgr2=0x7fb58806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.528+0000 7fb5ada00640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb58806c920 0x7fb58806ede0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb598001050 tx=0x7fb598000f80 comp rx=0 tx=0).stop 2024-04-26T23:02:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.528+0000 7fb5ada00640 1 -- 172.21.15.40:0/2327468358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a81429b0 msgr2=0x7fb5a8144fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.528+0000 7fb5ada00640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a81429b0 0x7fb5a8144fe0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb594000c00 tx=0x7fb594001120 comp rx=0 tx=0).stop 2024-04-26T23:02:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.528+0000 7fb5ada00640 1 -- 172.21.15.40:0/2327468358 shutdown_connections 2024-04-26T23:02:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.528+0000 7fb5ada00640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb58806c920 0x7fb58806ede0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.528+0000 7fb5ada00640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5a80b44a0 0x7fb5a80b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.528+0000 7fb5ada00640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a81429b0 0x7fb5a8144fe0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.528+0000 7fb5ada00640 1 --2- 172.21.15.40:0/2327468358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb58c004800 0x7fb5a80afcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.528+0000 7fb5ada00640 1 -- 172.21.15.40:0/2327468358 >> 172.21.15.40:0/2327468358 conn(0x7fb5a80b2720 msgr2=0x7fb5a80b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:15.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.528+0000 7fb5ada00640 1 -- 172.21.15.40:0/2327468358 shutdown_connections 2024-04-26T23:02:15.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.528+0000 7fb5ada00640 1 -- 172.21.15.40:0/2327468358 wait complete. 2024-04-26T23:02:15.548 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_apply_tag' 2024-04-26T23:02:15.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.761+0000 7f1911e00640 1 Processor -- start 2024-04-26T23:02:15.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.761+0000 7f1911e00640 1 -- start start 2024-04-26T23:02:15.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.763+0000 7f1911e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f190c0b5f30 0x7f190c0d2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.763+0000 7f1911e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f190c0cff60 0x7f190c0d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.763+0000 7f1911e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f190c0d5b90 con 0x7f190c160750 2024-04-26T23:02:15.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.763+0000 7f1911e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f190c160b30 con 0x7f190c0cff60 2024-04-26T23:02:15.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.763+0000 7f1911e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f190c0d5e70 con 0x7f190c0b5f30 2024-04-26T23:02:15.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.763+0000 7f190aa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f190c160750 0x7f190c0d1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52390/0 (socket says 172.21.15.40:52390) 2024-04-26T23:02:15.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.763+0000 7f190aa00640 1 -- 172.21.15.40:0/166211475 learned_addr learned my addr 172.21.15.40:0/166211475 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:15.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.763+0000 7f190b400640 1 --2- 172.21.15.40:0/166211475 >> v2:172.21.15.40:3300/0 conn(0x7f190c0cff60 0x7f190c0d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.764+0000 7f190a000640 1 --2- 172.21.15.40:0/166211475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f190c0b5f30 0x7f190c0d2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.764+0000 7f1909600640 1 -- 172.21.15.40:0/166211475 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1179429531 0 0) 0x7f190c0d5b90 con 0x7f190c160750 2024-04-26T23:02:15.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.764+0000 7f1909600640 1 -- 172.21.15.40:0/166211475 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18f0003520 con 0x7f190c160750 2024-04-26T23:02:15.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.764+0000 7f190b400640 1 -- 172.21.15.40:0/166211475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f190c0b5f30 msgr2=0x7f190c0d2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:15.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.764+0000 7f190b400640 1 --2- 172.21.15.40:0/166211475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f190c0b5f30 0x7f190c0d2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.764+0000 7f190b400640 1 -- 172.21.15.40:0/166211475 >> v1:172.21.15.40:6789/0 conn(0x7f190c160750 legacy=0x7f190c0d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:15.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.764+0000 7f190b400640 1 -- 172.21.15.40:0/166211475 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190c0d3930 con 0x7f190c0cff60 2024-04-26T23:02:15.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.765+0000 7f190b400640 1 --2- 172.21.15.40:0/166211475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f190c0cff60 0x7f190c0d03b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1904009d90 tx=0x7f190402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1aa1c08a79a93e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:15.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.765+0000 7f1909600640 1 -- 172.21.15.40:0/166211475 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1904031600 con 0x7f190c0cff60 2024-04-26T23:02:15.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.765+0000 7f1909600640 1 -- 172.21.15.40:0/166211475 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19040317a0 con 0x7f190c0cff60 2024-04-26T23:02:15.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.765+0000 7f1909600640 1 -- 172.21.15.40:0/166211475 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1904031b70 con 0x7f190c0cff60 2024-04-26T23:02:15.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.766+0000 7f1911e00640 1 -- 172.21.15.40:0/166211475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f190c0cff60 msgr2=0x7f190c0d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:15.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.766+0000 7f1911e00640 1 --2- 172.21.15.40:0/166211475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f190c0cff60 0x7f190c0d03b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1904009d90 tx=0x7f190402a5d0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.766+0000 7f1911e00640 1 -- 172.21.15.40:0/166211475 shutdown_connections 2024-04-26T23:02:15.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.766+0000 7f1911e00640 1 --2- 172.21.15.40:0/166211475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f190c0cff60 0x7f190c0d03b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.766+0000 7f1911e00640 1 --2- 172.21.15.40:0/166211475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f190c0b5f30 0x7f190c0d2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.766+0000 7f1911e00640 1 -- 172.21.15.40:0/166211475 >> 172.21.15.40:0/166211475 conn(0x7f190c0d5070 msgr2=0x7f190c0d5480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:15.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.766+0000 7f1911e00640 1 -- 172.21.15.40:0/166211475 shutdown_connections 2024-04-26T23:02:15.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.766+0000 7f1911e00640 1 -- 172.21.15.40:0/166211475 wait complete. 2024-04-26T23:02:15.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.768+0000 7f1911e00640 1 Processor -- start 2024-04-26T23:02:15.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.768+0000 7f1911e00640 1 -- start start 2024-04-26T23:02:15.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.769+0000 7f1911e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f190c0b5f30 0x7f190c153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.769+0000 7f1911e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f190c0cff60 0x7f190c0d1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.769+0000 7f1911e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f190c160750 0x7f190c0b9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.769+0000 7f1911e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f190c0d3ef0 con 0x7f190c0cff60 2024-04-26T23:02:15.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.769+0000 7f1911e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f190c0d3620 con 0x7f190c0b5f30 2024-04-26T23:02:15.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.769+0000 7f1911e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f190c160ef0 con 0x7f190c160750 2024-04-26T23:02:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.769+0000 7f190a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f190c0cff60 0x7f190c0d1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.769+0000 7f190a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f190c0cff60 0x7f190c0d1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51974/0 (socket says 172.21.15.40:51974) 2024-04-26T23:02:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.769+0000 7f190a000640 1 -- 172.21.15.40:0/782288330 learned_addr learned my addr 172.21.15.40:0/782288330 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.770+0000 7f190b400640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f190c160750 0x7f190c0b9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.770+0000 7f190aa00640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f190c0b5f30 0x7f190c153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.770+0000 7f190a000640 1 -- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f190c160750 msgr2=0x7f190c0b9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.770+0000 7f190a000640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f190c160750 0x7f190c0b9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.770+0000 7f190a000640 1 -- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f190c0b5f30 msgr2=0x7f190c153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.770+0000 7f190a000640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f190c0b5f30 0x7f190c153bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.770+0000 7f190a000640 1 -- 172.21.15.40:0/782288330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190c154110 con 0x7f190c0cff60 2024-04-26T23:02:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.770+0000 7f190a000640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f190c0cff60 0x7f190c0d1fd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f18f800d5f0 tx=0x7f18f800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.770+0000 7f1903400640 1 -- 172.21.15.40:0/782288330 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f80125f0 con 0x7f190c0cff60 2024-04-26T23:02:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.770+0000 7f1903400640 1 -- 172.21.15.40:0/782288330 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18f8019070 con 0x7f190c0cff60 2024-04-26T23:02:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.770+0000 7f1911e00640 1 -- 172.21.15.40:0/782288330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f190c0b89e0 con 0x7f190c0cff60 2024-04-26T23:02:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.770+0000 7f1903400640 1 -- 172.21.15.40:0/782288330 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f80129d0 con 0x7f190c0cff60 2024-04-26T23:02:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.770+0000 7f1911e00640 1 -- 172.21.15.40:0/782288330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f190c0b8fa0 con 0x7f190c0cff60 2024-04-26T23:02:15.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.771+0000 7f1911e00640 1 -- 172.21.15.40:0/782288330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f190c0694f0 con 0x7f190c0cff60 2024-04-26T23:02:15.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.773+0000 7f1903400640 1 -- 172.21.15.40:0/782288330 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f18f8012b70 con 0x7f190c0cff60 2024-04-26T23:02:15.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.774+0000 7f1903400640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f18ec06c8f0 0x7f18ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:15.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.774+0000 7f1903400640 1 -- 172.21.15.40:0/782288330 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f18f808c3b0 con 0x7f190c0cff60 2024-04-26T23:02:15.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.775+0000 7f190aa00640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f18ec06c8f0 0x7f18ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:15.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.776+0000 7f190aa00640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f18ec06c8f0 0x7f18ec06edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f18fc000f60 tx=0x7f18fc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:15.779+0000 7f1903400640 1 -- 172.21.15.40:0/782288330 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f190c154110 con 0x7f190c0cff60 2024-04-26T23:02:16.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.062+0000 7f1911e00640 1 -- 172.21.15.40:0/782288330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_apply_tag"]} v 0) -- 0x7f190c08ead0 con 0x7f190c0cff60 2024-04-26T23:02:16.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.142+0000 7f1903400640 1 -- 172.21.15.40:0/782288330 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_apply_tag"]}]=0 v84) ==== 139+0+0 (secure 0 0 0) 0x7f18f805a340 con 0x7f190c0cff60 2024-04-26T23:02:16.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.145+0000 7f1911e00640 1 -- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f18ec06c8f0 msgr2=0x7f18ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:16.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.145+0000 7f1911e00640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f18ec06c8f0 0x7f18ec06edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f18fc000f60 tx=0x7f18fc009040 comp rx=0 tx=0).stop 2024-04-26T23:02:16.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.145+0000 7f1911e00640 1 -- 172.21.15.40:0/782288330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f190c0cff60 msgr2=0x7f190c0d1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:16.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.145+0000 7f1911e00640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f190c0cff60 0x7f190c0d1fd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f18f800d5f0 tx=0x7f18f800dab0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.145+0000 7f1911e00640 1 -- 172.21.15.40:0/782288330 shutdown_connections 2024-04-26T23:02:16.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.145+0000 7f1911e00640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f18ec06c8f0 0x7f18ec06edb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.145+0000 7f1911e00640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f190c160750 0x7f190c0b9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.145+0000 7f1911e00640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f190c0cff60 0x7f190c0d1fd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.145+0000 7f1911e00640 1 --2- 172.21.15.40:0/782288330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f190c0b5f30 0x7f190c153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.145+0000 7f1911e00640 1 -- 172.21.15.40:0/782288330 >> 172.21.15.40:0/782288330 conn(0x7f190c0d5070 msgr2=0x7f190c1611b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:16.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.145+0000 7f1911e00640 1 -- 172.21.15.40:0/782288330 shutdown_connections 2024-04-26T23:02:16.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.145+0000 7f1911e00640 1 -- 172.21.15.40:0/782288330 wait complete. 2024-04-26T23:02:16.165 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-26T23:02:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.374+0000 7f8a52200640 1 Processor -- start 2024-04-26T23:02:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.374+0000 7f8a52200640 1 -- start start 2024-04-26T23:02:16.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.376+0000 7f8a52200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8a4c160a50 0x7f8a4c073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:16.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.376+0000 7f8a52200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c06dd40 0x7f8a4c072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:16.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.376+0000 7f8a52200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a4c161880 con 0x7f8a4c0cfaf0 2024-04-26T23:02:16.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.376+0000 7f8a52200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8a4c071ce0 con 0x7f8a4c160a50 2024-04-26T23:02:16.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.376+0000 7f8a52200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a4c0722a0 con 0x7f8a4c06dd40 2024-04-26T23:02:16.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.376+0000 7f8a4be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8a4c0cfaf0 0x7f8a4c160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52400/0 (socket says 172.21.15.40:52400) 2024-04-26T23:02:16.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.376+0000 7f8a4be00640 1 -- 172.21.15.40:0/2840039113 learned_addr learned my addr 172.21.15.40:0/2840039113 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:16.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.377+0000 7f8a50e00640 1 --2- 172.21.15.40:0/2840039113 >> v2:172.21.15.103:3300/0 conn(0x7f8a4c160a50 0x7f8a4c073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.377+0000 7f8a51800640 1 --2- 172.21.15.40:0/2840039113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c06dd40 0x7f8a4c072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.377+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3783444212 0 0) 0x7f8a4c161880 con 0x7f8a4c0cfaf0 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.377+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a34003520 con 0x7f8a4c0cfaf0 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.377+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1660327477 0 0) 0x7f8a34003520 con 0x7f8a4c0cfaf0 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.377+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c06dd40 msgr2=0x7f8a4c072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.377+0000 7f8a4b400640 1 --2- 172.21.15.40:0/2840039113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c06dd40 0x7f8a4c072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.377+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 >> v2:172.21.15.103:3300/0 conn(0x7f8a4c160a50 msgr2=0x7f8a4c073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.377+0000 7f8a4b400640 1 --2- 172.21.15.40:0/2840039113 >> v2:172.21.15.103:3300/0 conn(0x7f8a4c160a50 0x7f8a4c073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c0cff80 con 0x7f8a4c0cfaf0 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f8a3c001670 con 0x7f8a4c0cfaf0 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 >> v1:172.21.15.40:6789/0 conn(0x7f8a4c0cfaf0 legacy=0x7f8a4c160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a4b400640 1 --2- 172.21.15.40:0/2840039113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c06dd40 0x7f8a340039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a4b400640 1 --2- 172.21.15.40:0/2840039113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a34003f30 0x7f8a34004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:16.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a4b400640 1 --2- 172.21.15.40:0/2840039113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a34004d40 0x7f8a340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c071ce0 con 0x7f8a34004d40 2024-04-26T23:02:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c0722a0 con 0x7f8a34003f30 2024-04-26T23:02:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a3400b9c0 con 0x7f8a4c06dd40 2024-04-26T23:02:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a51800640 1 --2- 172.21.15.40:0/2840039113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a34004d40 0x7f8a340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a50e00640 1 --2- 172.21.15.40:0/2840039113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c06dd40 0x7f8a340039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a4be00640 1 --2- 172.21.15.40:0/2840039113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a34003f30 0x7f8a34004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a51800640 1 -- 172.21.15.40:0/2840039113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c06dd40 msgr2=0x7f8a340039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a51800640 1 --2- 172.21.15.40:0/2840039113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c06dd40 0x7f8a340039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a51800640 1 -- 172.21.15.40:0/2840039113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a34003f30 msgr2=0x7f8a34004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a51800640 1 --2- 172.21.15.40:0/2840039113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a34003f30 0x7f8a34004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a51800640 1 -- 172.21.15.40:0/2840039113 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a34003520 con 0x7f8a34004d40 2024-04-26T23:02:16.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.378+0000 7f8a50e00640 1 --2- 172.21.15.40:0/2840039113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c06dd40 0x7f8a340039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:16.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.380+0000 7f8a51800640 1 --2- 172.21.15.40:0/2840039113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a34004d40 0x7f8a340051c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009fc0 tx=0x7f8a34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e87b43aa4766627 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:16.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.380+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44009c10 con 0x7f8a34004d40 2024-04-26T23:02:16.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.380+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a44036070 con 0x7f8a34004d40 2024-04-26T23:02:16.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.380+0000 7f8a4b400640 1 -- 172.21.15.40:0/2840039113 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44031850 con 0x7f8a34004d40 2024-04-26T23:02:16.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.380+0000 7f8a52200640 1 -- 172.21.15.40:0/2840039113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a34004d40 msgr2=0x7f8a340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:16.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.380+0000 7f8a52200640 1 --2- 172.21.15.40:0/2840039113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a34004d40 0x7f8a340051c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009fc0 tx=0x7f8a34005830 comp rx=0 tx=0).stop 2024-04-26T23:02:16.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.380+0000 7f8a51800640 1 -- 172.21.15.40:0/2840039113 reap_dead start 2024-04-26T23:02:16.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.381+0000 7f8a52200640 1 -- 172.21.15.40:0/2840039113 shutdown_connections 2024-04-26T23:02:16.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.381+0000 7f8a52200640 1 -- 172.21.15.40:0/2840039113 >> 172.21.15.40:0/2840039113 conn(0x7f8a4c0b2720 msgr2=0x7f8a4c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:16.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.381+0000 7f8a52200640 1 -- 172.21.15.40:0/2840039113 shutdown_connections 2024-04-26T23:02:16.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.381+0000 7f8a52200640 1 -- 172.21.15.40:0/2840039113 wait complete. 2024-04-26T23:02:16.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.382+0000 7f8a52200640 1 Processor -- start 2024-04-26T23:02:16.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.383+0000 7f8a52200640 1 -- start start 2024-04-26T23:02:16.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.383+0000 7f8a52200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a34004d40 0x7f8a4c0aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:16.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.383+0000 7f8a52200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c1402e0 0x7f8a4c0abb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:16.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a52200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c144530 0x7f8a4c159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:16.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a52200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c071f30 con 0x7f8a4c144530 2024-04-26T23:02:16.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a52200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c073f90 con 0x7f8a34004d40 2024-04-26T23:02:16.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a52200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a4c06e1f0 con 0x7f8a4c1402e0 2024-04-26T23:02:16.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a51800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c144530 0x7f8a4c159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:16.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a51800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c144530 0x7f8a4c159ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52006/0 (socket says 172.21.15.40:52006) 2024-04-26T23:02:16.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a51800640 1 -- 172.21.15.40:0/4203908654 learned_addr learned my addr 172.21.15.40:0/4203908654 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:16.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a50e00640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a34004d40 0x7f8a4c0aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:16.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a4be00640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c1402e0 0x7f8a4c0abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:16.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a51800640 1 -- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c1402e0 msgr2=0x7f8a4c0abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:16.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a51800640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c1402e0 0x7f8a4c0abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a51800640 1 -- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a34004d40 msgr2=0x7f8a4c0aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:16.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a51800640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a34004d40 0x7f8a4c0aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.384+0000 7f8a51800640 1 -- 172.21.15.40:0/4203908654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c0aaa90 con 0x7f8a4c144530 2024-04-26T23:02:16.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.385+0000 7f8a51800640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c144530 0x7f8a4c159ee0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8a4402aeb0 tx=0x7f8a44009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:16.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.385+0000 7f8a48c00640 1 -- 172.21.15.40:0/4203908654 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44009c10 con 0x7f8a4c144530 2024-04-26T23:02:16.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.385+0000 7f8a48c00640 1 -- 172.21.15.40:0/4203908654 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a44003990 con 0x7f8a4c144530 2024-04-26T23:02:16.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.385+0000 7f8a48c00640 1 -- 172.21.15.40:0/4203908654 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44003d80 con 0x7f8a4c144530 2024-04-26T23:02:16.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.385+0000 7f8a52200640 1 -- 172.21.15.40:0/4203908654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4c0aad70 con 0x7f8a4c144530 2024-04-26T23:02:16.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.385+0000 7f8a52200640 1 -- 172.21.15.40:0/4203908654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a4c0ab330 con 0x7f8a4c144530 2024-04-26T23:02:16.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.387+0000 7f8a52200640 1 -- 172.21.15.40:0/4203908654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a4c069370 con 0x7f8a4c144530 2024-04-26T23:02:16.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.388+0000 7f8a48c00640 1 -- 172.21.15.40:0/4203908654 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8a44005d10 con 0x7f8a4c144530 2024-04-26T23:02:16.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.388+0000 7f8a48c00640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8a3006c920 0x7f8a3006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:16.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.389+0000 7f8a48c00640 1 -- 172.21.15.40:0/4203908654 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8a440ac890 con 0x7f8a4c144530 2024-04-26T23:02:16.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.389+0000 7f8a50e00640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8a3006c920 0x7f8a3006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:16.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.390+0000 7f8a50e00640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8a3006c920 0x7f8a3006ede0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8a40009b20 tx=0x7f8a40008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:16.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.393+0000 7f8a48c00640 1 -- 172.21.15.40:0/4203908654 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8a4404d020 con 0x7f8a4c144530 2024-04-26T23:02:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.606+0000 7f8a52200640 1 -- 172.21.15.40:0/4203908654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f8a4c06fa60 con 0x7f8a4c144530 2024-04-26T23:02:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.612+0000 7f8a48c00640 1 -- 172.21.15.40:0/4203908654 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f8a4402e050 con 0x7f8a4c144530 2024-04-26T23:02:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.615+0000 7f8a52200640 1 -- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8a3006c920 msgr2=0x7f8a3006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.615+0000 7f8a52200640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8a3006c920 0x7f8a3006ede0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8a40009b20 tx=0x7f8a40008040 comp rx=0 tx=0).stop 2024-04-26T23:02:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.615+0000 7f8a52200640 1 -- 172.21.15.40:0/4203908654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c144530 msgr2=0x7f8a4c159ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.615+0000 7f8a52200640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c144530 0x7f8a4c159ee0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8a4402aeb0 tx=0x7f8a44009750 comp rx=0 tx=0).stop 2024-04-26T23:02:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.615+0000 7f8a52200640 1 -- 172.21.15.40:0/4203908654 shutdown_connections 2024-04-26T23:02:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.615+0000 7f8a52200640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8a3006c920 0x7f8a3006ede0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.615+0000 7f8a52200640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c144530 0x7f8a4c159ee0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.615+0000 7f8a52200640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a4c1402e0 0x7f8a4c0abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.615+0000 7f8a52200640 1 --2- 172.21.15.40:0/4203908654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a34004d40 0x7f8a4c0aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.615+0000 7f8a52200640 1 -- 172.21.15.40:0/4203908654 >> 172.21.15.40:0/4203908654 conn(0x7f8a4c0b2720 msgr2=0x7f8a4c15e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.615+0000 7f8a52200640 1 -- 172.21.15.40:0/4203908654 shutdown_connections 2024-04-26T23:02:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:16.616+0000 7f8a52200640 1 -- 172.21.15.40:0/4203908654 wait complete. 2024-04-26T23:02:16.635 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-26T23:02:16.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:02:16.737 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:02:16.864 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-26T23:02:16.864 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:02:16.864 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:16.925 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:02:16.925 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:16.980 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:02:16.980 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:17.039 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:17.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:02:17.159 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:02:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.258+0000 7f9506c00640 1 Processor -- start 2024-04-26T23:02:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.258+0000 7f9506c00640 1 -- start start 2024-04-26T23:02:17.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.260+0000 7f9506c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f95000b6220 0x7f95000d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:17.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.260+0000 7f9506c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95000d39f0 0x7f95000b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:17.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.260+0000 7f9506c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9500162c40 con 0x7f95000d08d0 2024-04-26T23:02:17.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.260+0000 7f9506c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f95000d2d80 con 0x7f95000b6220 2024-04-26T23:02:17.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.260+0000 7f9506c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f95000b7c80 con 0x7f95000d39f0 2024-04-26T23:02:17.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.260+0000 7f9505800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f95000b6220 0x7f95000d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.260+0000 7f9504e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f95000d08d0 0x7f95000d32e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52416/0 (socket says 172.21.15.40:52416) 2024-04-26T23:02:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.260+0000 7f9504e00640 1 -- 172.21.15.40:0/2994327830 learned_addr learned my addr 172.21.15.40:0/2994327830 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.260+0000 7f9506200640 1 --2- 172.21.15.40:0/2994327830 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95000d39f0 0x7f95000b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.261+0000 7f9505800640 1 -- 172.21.15.40:0/2994327830 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95000d39f0 msgr2=0x7f95000b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.261+0000 7f9505800640 1 --2- 172.21.15.40:0/2994327830 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95000d39f0 0x7f95000b8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.261+0000 7f9505800640 1 -- 172.21.15.40:0/2994327830 >> v1:172.21.15.40:6789/0 conn(0x7f95000d08d0 legacy=0x7f95000d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:17.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.261+0000 7f9505800640 1 -- 172.21.15.40:0/2994327830 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95000b7f60 con 0x7f95000b6220 2024-04-26T23:02:17.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.261+0000 7f94ffe00640 1 -- 172.21.15.40:0/2994327830 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 129618796 0 0) 0x7f9500162c40 con 0x7f95000d08d0 2024-04-26T23:02:17.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.262+0000 7f9505800640 1 --2- 172.21.15.40:0/2994327830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95000b6220 0x7f95000d0390 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f94f40097a0 tx=0x7f94f402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3e635100df714 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:17.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.262+0000 7f94ffe00640 1 -- 172.21.15.40:0/2994327830 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4037070 con 0x7f95000b6220 2024-04-26T23:02:17.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.262+0000 7f94ffe00640 1 -- 172.21.15.40:0/2994327830 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94f4032440 con 0x7f95000b6220 2024-04-26T23:02:17.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.262+0000 7f94ffe00640 1 -- 172.21.15.40:0/2994327830 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4032810 con 0x7f95000b6220 2024-04-26T23:02:17.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.263+0000 7f9506c00640 1 -- 172.21.15.40:0/2994327830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95000b6220 msgr2=0x7f95000d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:17.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.263+0000 7f9506c00640 1 --2- 172.21.15.40:0/2994327830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95000b6220 0x7f95000d0390 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f94f40097a0 tx=0x7f94f402a180 comp rx=0 tx=0).stop 2024-04-26T23:02:17.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.263+0000 7f9506c00640 1 -- 172.21.15.40:0/2994327830 shutdown_connections 2024-04-26T23:02:17.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.263+0000 7f9506c00640 1 --2- 172.21.15.40:0/2994327830 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95000d39f0 0x7f95000b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.263+0000 7f9506c00640 1 --2- 172.21.15.40:0/2994327830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95000b6220 0x7f95000d0390 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.263+0000 7f9506c00640 1 -- 172.21.15.40:0/2994327830 >> 172.21.15.40:0/2994327830 conn(0x7f95000d4d90 msgr2=0x7f95000d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:17.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.263+0000 7f9506c00640 1 -- 172.21.15.40:0/2994327830 shutdown_connections 2024-04-26T23:02:17.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.263+0000 7f9506c00640 1 -- 172.21.15.40:0/2994327830 wait complete. 2024-04-26T23:02:17.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.265+0000 7f9506c00640 1 Processor -- start 2024-04-26T23:02:17.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.266+0000 7f9506c00640 1 -- start start 2024-04-26T23:02:17.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.267+0000 7f9506c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95000b6220 0x7f950015c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:17.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.267+0000 7f9506c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95000d08d0 0x7f95000b1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:17.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.267+0000 7f9506c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95000d39f0 0x7f95000b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:17.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.267+0000 7f9506c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f95000b9260 con 0x7f95000d08d0 2024-04-26T23:02:17.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.267+0000 7f9506c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f95000b8520 con 0x7f95000d39f0 2024-04-26T23:02:17.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.267+0000 7f9506c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f95000b93e0 con 0x7f95000b6220 2024-04-26T23:02:17.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.267+0000 7f9504e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95000d08d0 0x7f95000b1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:17.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.267+0000 7f9504e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95000d08d0 0x7f95000b1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52022/0 (socket says 172.21.15.40:52022) 2024-04-26T23:02:17.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.267+0000 7f9504e00640 1 -- 172.21.15.40:0/2785659460 learned_addr learned my addr 172.21.15.40:0/2785659460 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:17.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.267+0000 7f9505800640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95000b6220 0x7f950015c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:17.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.268+0000 7f9506200640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95000d39f0 0x7f95000b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:17.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.268+0000 7f9504e00640 1 -- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95000b6220 msgr2=0x7f950015c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:17.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.268+0000 7f9504e00640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95000b6220 0x7f950015c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.268+0000 7f9504e00640 1 -- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95000d39f0 msgr2=0x7f95000b4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:17.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.268+0000 7f9504e00640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95000d39f0 0x7f95000b4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.268+0000 7f9504e00640 1 -- 172.21.15.40:0/2785659460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950015de70 con 0x7f95000d08d0 2024-04-26T23:02:17.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.268+0000 7f9504e00640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95000d08d0 0x7f95000b1980 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00c500 tx=0x7f94ec00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:17.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.268+0000 7f94fd600640 1 -- 172.21.15.40:0/2785659460 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec018070 con 0x7f95000d08d0 2024-04-26T23:02:17.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.269+0000 7f94fd600640 1 -- 172.21.15.40:0/2785659460 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94ec007580 con 0x7f95000d08d0 2024-04-26T23:02:17.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.269+0000 7f94fd600640 1 -- 172.21.15.40:0/2785659460 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec004a20 con 0x7f95000d08d0 2024-04-26T23:02:17.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.269+0000 7f9506c00640 1 -- 172.21.15.40:0/2785659460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950015e110 con 0x7f95000d08d0 2024-04-26T23:02:17.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.269+0000 7f9506c00640 1 -- 172.21.15.40:0/2785659460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95000b5760 con 0x7f95000d08d0 2024-04-26T23:02:17.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.270+0000 7f9506c00640 1 -- 172.21.15.40:0/2785659460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f950015e2e0 con 0x7f95000d08d0 2024-04-26T23:02:17.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.271+0000 7f94fd600640 1 -- 172.21.15.40:0/2785659460 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f94ec004bc0 con 0x7f95000d08d0 2024-04-26T23:02:17.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.272+0000 7f94fd600640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f94e006c8f0 0x7f94e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:17.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.272+0000 7f94fd600640 1 -- 172.21.15.40:0/2785659460 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f94ec08b190 con 0x7f95000d08d0 2024-04-26T23:02:17.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.272+0000 7f9505800640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f94e006c8f0 0x7f94e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:17.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.273+0000 7f9505800640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f94e006c8f0 0x7f94e006edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f94f4034000 tx=0x7f94f402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:17.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.276+0000 7f94fd600640 1 -- 172.21.15.40:0/2785659460 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f94ec058b60 con 0x7f95000d08d0 2024-04-26T23:02:17.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.487+0000 7f9506c00640 1 -- 172.21.15.40:0/2785659460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9500069370 con 0x7f95000d08d0 2024-04-26T23:02:17.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.489+0000 7f94fd600640 1 -- 172.21.15.40:0/2785659460 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18812 (secure 0 0 0) 0x7f94ec05ce60 con 0x7f95000d08d0 2024-04-26T23:02:17.490 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:17.490 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:46.975683+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.492+0000 7f9506c00640 1 -- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f94e006c8f0 msgr2=0x7f94e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.492+0000 7f9506c00640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f94e006c8f0 0x7f94e006edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f94f4034000 tx=0x7f94f402e000 comp rx=0 tx=0).stop 2024-04-26T23:02:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.492+0000 7f9506c00640 1 -- 172.21.15.40:0/2785659460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95000d08d0 msgr2=0x7f95000b1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.492+0000 7f9506c00640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95000d08d0 0x7f95000b1980 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00c500 tx=0x7f94ec00c9c0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.492+0000 7f9506c00640 1 -- 172.21.15.40:0/2785659460 shutdown_connections 2024-04-26T23:02:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.492+0000 7f9506c00640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f94e006c8f0 0x7f94e006edb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.492+0000 7f9506c00640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95000d39f0 0x7f95000b4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.492+0000 7f9506c00640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95000d08d0 0x7f95000b1980 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.492+0000 7f9506c00640 1 --2- 172.21.15.40:0/2785659460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f95000b6220 0x7f950015c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.492+0000 7f9506c00640 1 -- 172.21.15.40:0/2785659460 >> 172.21.15.40:0/2785659460 conn(0x7f95000d4d90 msgr2=0x7f95000d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:17.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.492+0000 7f9506c00640 1 -- 172.21.15.40:0/2785659460 shutdown_connections 2024-04-26T23:02:17.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.492+0000 7f9506c00640 1 -- 172.21.15.40:0/2785659460 wait complete. 2024-04-26T23:02:17.512 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:02:17.513 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:02:17.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.757+0000 7f61ff200640 1 Processor -- start 2024-04-26T23:02:17.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.757+0000 7f61ff200640 1 -- start start 2024-04-26T23:02:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.759+0000 7f61ff200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f62000d0010 0x7f62000d1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.759+0000 7f61ff200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f620006d560 0x7f62000cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.759+0000 7f61ff200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f620015f670 con 0x7f620006dd60 2024-04-26T23:02:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.759+0000 7f61ff200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f62000d0fc0 con 0x7f62000d0010 2024-04-26T23:02:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.759+0000 7f61ff200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f62000d1b20 con 0x7f620006d560 2024-04-26T23:02:17.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.759+0000 7f61fd400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f62000d0010 0x7f62000d1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:17.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.759+0000 7f61fe800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f620006d560 0x7f62000cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:17.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.759+0000 7f61fd400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f62000d0010 0x7f62000d1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:39216/0 (socket says 172.21.15.40:39216) 2024-04-26T23:02:17.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.759+0000 7f61fd400640 1 -- 172.21.15.40:0/2870896801 learned_addr learned my addr 172.21.15.40:0/2870896801 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:17.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.760+0000 7f61fd400640 1 -- 172.21.15.40:0/2870896801 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f620006d560 msgr2=0x7f62000cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:17.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.760+0000 7f61fd400640 1 --2- 172.21.15.40:0/2870896801 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f620006d560 0x7f62000cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.760+0000 7f61fd400640 1 -- 172.21.15.40:0/2870896801 >> v1:172.21.15.103:6789/0 conn(0x7f620006dd60 legacy=0x7f620006e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.760+0000 7f61fd400640 1 -- 172.21.15.40:0/2870896801 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62000d0580 con 0x7f62000d0010 2024-04-26T23:02:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.761+0000 7f61fd400640 1 --2- 172.21.15.40:0/2870896801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62000d0010 0x7f62000d1fb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009870 tx=0x7f61e802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45a0413da1b6a24b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.762+0000 7f61fca00640 1 -- 172.21.15.40:0/2870896801 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8037070 con 0x7f62000d0010 2024-04-26T23:02:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.762+0000 7f61fca00640 1 -- 172.21.15.40:0/2870896801 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61e8032440 con 0x7f62000d0010 2024-04-26T23:02:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.762+0000 7f61fca00640 1 -- 172.21.15.40:0/2870896801 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8032810 con 0x7f62000d0010 2024-04-26T23:02:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.763+0000 7f61ff200640 1 -- 172.21.15.40:0/2870896801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62000d0010 msgr2=0x7f62000d1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.763+0000 7f61ff200640 1 --2- 172.21.15.40:0/2870896801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62000d0010 0x7f62000d1fb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009870 tx=0x7f61e802a270 comp rx=0 tx=0).stop 2024-04-26T23:02:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.763+0000 7f61ff200640 1 -- 172.21.15.40:0/2870896801 shutdown_connections 2024-04-26T23:02:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.763+0000 7f61ff200640 1 --2- 172.21.15.40:0/2870896801 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f620006d560 0x7f62000cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.763+0000 7f61ff200640 1 --2- 172.21.15.40:0/2870896801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62000d0010 0x7f62000d1fb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.763+0000 7f61ff200640 1 -- 172.21.15.40:0/2870896801 >> 172.21.15.40:0/2870896801 conn(0x7f62000d2d80 msgr2=0x7f62000d3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.763+0000 7f61ff200640 1 -- 172.21.15.40:0/2870896801 shutdown_connections 2024-04-26T23:02:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.763+0000 7f61ff200640 1 -- 172.21.15.40:0/2870896801 wait complete. 2024-04-26T23:02:17.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.765+0000 7f61ff200640 1 Processor -- start 2024-04-26T23:02:17.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.765+0000 7f61ff200640 1 -- start start 2024-04-26T23:02:17.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.766+0000 7f61ff200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f620006d560 0x7f62001f5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:17.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.766+0000 7f61ff200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620006dd60 0x7f620015c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:17.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.766+0000 7f61ff200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62000d0010 0x7f620014b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:17.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.766+0000 7f61ff200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f62000d17b0 con 0x7f62000d0010 2024-04-26T23:02:17.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.766+0000 7f61ff200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f62000d14d0 con 0x7f620006dd60 2024-04-26T23:02:17.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.766+0000 7f61ff200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f62000d29e0 con 0x7f620006d560 2024-04-26T23:02:17.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.766+0000 7f61fe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62000d0010 0x7f620014b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:17.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.766+0000 7f61fe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62000d0010 0x7f620014b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52038/0 (socket says 172.21.15.40:52038) 2024-04-26T23:02:17.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.766+0000 7f61fe800640 1 -- 172.21.15.40:0/3762910982 learned_addr learned my addr 172.21.15.40:0/3762910982 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:17.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.766+0000 7f61fd400640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620006dd60 0x7f620015c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:17.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.766+0000 7f61fde00640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f620006d560 0x7f62001f5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:17.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.767+0000 7f61fe800640 1 -- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f620006d560 msgr2=0x7f62001f5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:17.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.767+0000 7f61fe800640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f620006d560 0x7f62001f5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.767+0000 7f61fe800640 1 -- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620006dd60 msgr2=0x7f620015c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:17.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.767+0000 7f61fe800640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620006dd60 0x7f620015c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.767+0000 7f61fe800640 1 -- 172.21.15.40:0/3762910982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f620015cb00 con 0x7f62000d0010 2024-04-26T23:02:17.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.767+0000 7f61fe800640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62000d0010 0x7f620014b110 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f61f800efc0 tx=0x7f61f800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:17.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.767+0000 7f61f6000640 1 -- 172.21.15.40:0/3762910982 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f8018020 con 0x7f62000d0010 2024-04-26T23:02:17.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.767+0000 7f61f6000640 1 -- 172.21.15.40:0/3762910982 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61f8014480 con 0x7f62000d0010 2024-04-26T23:02:17.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.767+0000 7f61f6000640 1 -- 172.21.15.40:0/3762910982 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f8014870 con 0x7f62000d0010 2024-04-26T23:02:17.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.767+0000 7f61ff200640 1 -- 172.21.15.40:0/3762910982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f620015cdd0 con 0x7f62000d0010 2024-04-26T23:02:17.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.768+0000 7f61ff200640 1 -- 172.21.15.40:0/3762910982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f620015d390 con 0x7f62000d0010 2024-04-26T23:02:17.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.769+0000 7f61ff200640 1 -- 172.21.15.40:0/3762910982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6200069a80 con 0x7f62000d0010 2024-04-26T23:02:17.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.770+0000 7f61f6000640 1 -- 172.21.15.40:0/3762910982 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f61f8005d10 con 0x7f62000d0010 2024-04-26T23:02:17.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.771+0000 7f61f6000640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f61dc06c8f0 0x7f61dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:17.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.771+0000 7f61f6000640 1 -- 172.21.15.40:0/3762910982 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f61f808ccd0 con 0x7f62000d0010 2024-04-26T23:02:17.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.771+0000 7f61fde00640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f61dc06c8f0 0x7f61dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:17.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.773+0000 7f61fde00640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f61dc06c8f0 0x7f61dc06edb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6200159bc0 tx=0x7f61f0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:17.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.775+0000 7f61f6000640 1 -- 172.21.15.40:0/3762910982 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f61f8022030 con 0x7f62000d0010 2024-04-26T23:02:17.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.980+0000 7f61ff200640 1 -- 172.21.15.40:0/3762910982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f620006e950 con 0x7f62000d0010 2024-04-26T23:02:17.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.980+0000 7f61f6000640 1 -- 172.21.15.40:0/3762910982 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f61f8010040 con 0x7f62000d0010 2024-04-26T23:02:17.980 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:02:17.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.983+0000 7f61ff200640 1 -- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f61dc06c8f0 msgr2=0x7f61dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:17.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.983+0000 7f61ff200640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f61dc06c8f0 0x7f61dc06edb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6200159bc0 tx=0x7f61f0009040 comp rx=0 tx=0).stop 2024-04-26T23:02:17.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.983+0000 7f61ff200640 1 -- 172.21.15.40:0/3762910982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62000d0010 msgr2=0x7f620014b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:17.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.983+0000 7f61ff200640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62000d0010 0x7f620014b110 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f61f800efc0 tx=0x7f61f800c490 comp rx=0 tx=0).stop 2024-04-26T23:02:17.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.984+0000 7f61ff200640 1 -- 172.21.15.40:0/3762910982 shutdown_connections 2024-04-26T23:02:17.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.984+0000 7f61ff200640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f61dc06c8f0 0x7f61dc06edb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.984+0000 7f61ff200640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62000d0010 0x7f620014b110 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.984+0000 7f61ff200640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620006dd60 0x7f620015c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.984+0000 7f61ff200640 1 --2- 172.21.15.40:0/3762910982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f620006d560 0x7f62001f5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:17.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.984+0000 7f61ff200640 1 -- 172.21.15.40:0/3762910982 >> 172.21.15.40:0/3762910982 conn(0x7f62000d2d80 msgr2=0x7f6200160230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:17.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.984+0000 7f61ff200640 1 -- 172.21.15.40:0/3762910982 shutdown_connections 2024-04-26T23:02:17.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:17.984+0000 7f61ff200640 1 -- 172.21.15.40:0/3762910982 wait complete. 2024-04-26T23:02:18.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:02:18.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.200+0000 7fbd64e00640 1 Processor -- start 2024-04-26T23:02:18.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.201+0000 7fbd64e00640 1 -- start start 2024-04-26T23:02:18.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.202+0000 7fbd64e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbd600cc480 0x7fbd60072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:18.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.202+0000 7fbd64e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd60072780 0x7fbd60166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:18.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.202+0000 7fbd64e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd6015d7c0 con 0x7fbd60071e60 2024-04-26T23:02:18.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.202+0000 7fbd64e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fbd6015daa0 con 0x7fbd600cc480 2024-04-26T23:02:18.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.202+0000 7fbd64e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbd6015dd80 con 0x7fbd60072780 2024-04-26T23:02:18.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.203+0000 7fbd5de00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbd60071e60 0x7fbd6006f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52432/0 (socket says 172.21.15.40:52432) 2024-04-26T23:02:18.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.203+0000 7fbd5de00640 1 -- 172.21.15.40:0/2881001178 learned_addr learned my addr 172.21.15.40:0/2881001178 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:18.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.203+0000 7fbd5d400640 1 --2- 172.21.15.40:0/2881001178 >> v2:172.21.15.40:3300/0 conn(0x7fbd600cc480 0x7fbd60072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:18.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.203+0000 7fbd5e800640 1 --2- 172.21.15.40:0/2881001178 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd60072780 0x7fbd60166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:18.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.203+0000 7fbd5d400640 1 -- 172.21.15.40:0/2881001178 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd60072780 msgr2=0x7fbd60166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:18.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.203+0000 7fbd5d400640 1 --2- 172.21.15.40:0/2881001178 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd60072780 0x7fbd60166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.203+0000 7fbd5d400640 1 -- 172.21.15.40:0/2881001178 >> v1:172.21.15.40:6789/0 conn(0x7fbd60071e60 legacy=0x7fbd6006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:18.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.203+0000 7fbd5d400640 1 -- 172.21.15.40:0/2881001178 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6015e060 con 0x7fbd600cc480 2024-04-26T23:02:18.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.204+0000 7fbd5d400640 1 --2- 172.21.15.40:0/2881001178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd600cc480 0x7fbd60072240 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0097a0 tx=0x7fbd4c02a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45cb119516b3a242 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:18.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.205+0000 7fbd5ca00640 1 -- 172.21.15.40:0/2881001178 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c037070 con 0x7fbd600cc480 2024-04-26T23:02:18.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.205+0000 7fbd5ca00640 1 -- 172.21.15.40:0/2881001178 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd4c02ad10 con 0x7fbd600cc480 2024-04-26T23:02:18.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.205+0000 7fbd5ca00640 1 -- 172.21.15.40:0/2881001178 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c032660 con 0x7fbd600cc480 2024-04-26T23:02:18.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.205+0000 7fbd64e00640 1 -- 172.21.15.40:0/2881001178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd600cc480 msgr2=0x7fbd60072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:18.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.205+0000 7fbd64e00640 1 --2- 172.21.15.40:0/2881001178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd600cc480 0x7fbd60072240 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0097a0 tx=0x7fbd4c02a080 comp rx=0 tx=0).stop 2024-04-26T23:02:18.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.206+0000 7fbd64e00640 1 -- 172.21.15.40:0/2881001178 shutdown_connections 2024-04-26T23:02:18.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.206+0000 7fbd64e00640 1 --2- 172.21.15.40:0/2881001178 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd60072780 0x7fbd60166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.206+0000 7fbd64e00640 1 --2- 172.21.15.40:0/2881001178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd600cc480 0x7fbd60072240 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.206+0000 7fbd64e00640 1 -- 172.21.15.40:0/2881001178 >> 172.21.15.40:0/2881001178 conn(0x7fbd600d0be0 msgr2=0x7fbd600d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:18.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.206+0000 7fbd64e00640 1 -- 172.21.15.40:0/2881001178 shutdown_connections 2024-04-26T23:02:18.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.206+0000 7fbd64e00640 1 -- 172.21.15.40:0/2881001178 wait complete. 2024-04-26T23:02:18.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.208+0000 7fbd64e00640 1 Processor -- start 2024-04-26T23:02:18.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.208+0000 7fbd64e00640 1 -- start start 2024-04-26T23:02:18.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.209+0000 7fbd64e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd60071e60 0x7fbd600b6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.209+0000 7fbd64e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd60072780 0x7fbd600b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.209+0000 7fbd64e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd600cc480 0x7fbd600b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.209+0000 7fbd64e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6015d3f0 con 0x7fbd60072780 2024-04-26T23:02:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.209+0000 7fbd64e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbd60072c10 con 0x7fbd60071e60 2024-04-26T23:02:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.209+0000 7fbd64e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbd6015d570 con 0x7fbd600cc480 2024-04-26T23:02:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.210+0000 7fbd5d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd60072780 0x7fbd600b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.210+0000 7fbd5d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd60072780 0x7fbd600b13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52056/0 (socket says 172.21.15.40:52056) 2024-04-26T23:02:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.210+0000 7fbd5d400640 1 -- 172.21.15.40:0/3942242319 learned_addr learned my addr 172.21.15.40:0/3942242319 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.210+0000 7fbd5de00640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd60071e60 0x7fbd600b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.210+0000 7fbd5e800640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd600cc480 0x7fbd600b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:18.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.210+0000 7fbd5d400640 1 -- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd600cc480 msgr2=0x7fbd600b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:18.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.210+0000 7fbd5d400640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd600cc480 0x7fbd600b18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.210+0000 7fbd5d400640 1 -- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd60071e60 msgr2=0x7fbd600b6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:18.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.210+0000 7fbd5d400640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd60071e60 0x7fbd600b6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.210+0000 7fbd5d400640 1 -- 172.21.15.40:0/3942242319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd601f6130 con 0x7fbd60072780 2024-04-26T23:02:18.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.211+0000 7fbd5d400640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd60072780 0x7fbd600b13a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0098c0 tx=0x7fbd4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:18.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.211+0000 7fbd56000640 1 -- 172.21.15.40:0/3942242319 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c037040 con 0x7fbd60072780 2024-04-26T23:02:18.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.211+0000 7fbd56000640 1 -- 172.21.15.40:0/3942242319 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd4c004030 con 0x7fbd60072780 2024-04-26T23:02:18.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.211+0000 7fbd56000640 1 -- 172.21.15.40:0/3942242319 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c004420 con 0x7fbd60072780 2024-04-26T23:02:18.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.211+0000 7fbd64e00640 1 -- 172.21.15.40:0/3942242319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd601f53b0 con 0x7fbd60072780 2024-04-26T23:02:18.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.211+0000 7fbd64e00640 1 -- 172.21.15.40:0/3942242319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd601f5650 con 0x7fbd60072780 2024-04-26T23:02:18.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.213+0000 7fbd64e00640 1 -- 172.21.15.40:0/3942242319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd600694f0 con 0x7fbd60072780 2024-04-26T23:02:18.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.214+0000 7fbd56000640 1 -- 172.21.15.40:0/3942242319 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbd4c003610 con 0x7fbd60072780 2024-04-26T23:02:18.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.215+0000 7fbd56000640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd4006c8f0 0x7fbd4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:18.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.215+0000 7fbd56000640 1 -- 172.21.15.40:0/3942242319 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbd4c0ac7c0 con 0x7fbd60072780 2024-04-26T23:02:18.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.215+0000 7fbd5de00640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd4006c8f0 0x7fbd4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:18.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.216+0000 7fbd5de00640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd4006c8f0 0x7fbd4006edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbd6015f700 tx=0x7fbd50000bc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:18.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.219+0000 7fbd56000640 1 -- 172.21.15.40:0/3942242319 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fbd4c07a230 con 0x7fbd60072780 2024-04-26T23:02:18.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.424+0000 7fbd64e00640 1 -- 172.21.15.40:0/3942242319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbd600694f0 con 0x7fbd60072780 2024-04-26T23:02:18.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.425+0000 7fbd56000640 1 -- 172.21.15.40:0/3942242319 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fbd4c07e530 con 0x7fbd60072780 2024-04-26T23:02:18.425 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:18.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.428+0000 7fbd64e00640 1 -- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd4006c8f0 msgr2=0x7fbd4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:18.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.428+0000 7fbd64e00640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd4006c8f0 0x7fbd4006edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbd6015f700 tx=0x7fbd50000bc0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.428+0000 7fbd64e00640 1 -- 172.21.15.40:0/3942242319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd60072780 msgr2=0x7fbd600b13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:18.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.428+0000 7fbd64e00640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd60072780 0x7fbd600b13a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0098c0 tx=0x7fbd4c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.428+0000 7fbd64e00640 1 -- 172.21.15.40:0/3942242319 shutdown_connections 2024-04-26T23:02:18.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.428+0000 7fbd64e00640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd4006c8f0 0x7fbd4006edb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.428+0000 7fbd64e00640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd600cc480 0x7fbd600b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.428+0000 7fbd64e00640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd60072780 0x7fbd600b13a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.428+0000 7fbd64e00640 1 --2- 172.21.15.40:0/3942242319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd60071e60 0x7fbd600b6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.428+0000 7fbd64e00640 1 -- 172.21.15.40:0/3942242319 >> 172.21.15.40:0/3942242319 conn(0x7fbd600d0be0 msgr2=0x7fbd6015a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:18.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.428+0000 7fbd64e00640 1 -- 172.21.15.40:0/3942242319 shutdown_connections 2024-04-26T23:02:18.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.428+0000 7fbd64e00640 1 -- 172.21.15.40:0/3942242319 wait complete. 2024-04-26T23:02:18.446 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:02:18.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.652+0000 7fc233e00640 1 Processor -- start 2024-04-26T23:02:18.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.652+0000 7fc233e00640 1 -- start start 2024-04-26T23:02:18.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.654+0000 7fc233e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc23415f1a0 0x7fc2340b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:18.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.654+0000 7fc233e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc2340d2ae0 0x7fc23415f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:18.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.654+0000 7fc233e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340d32c0 0x7fc2340d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:18.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.654+0000 7fc233e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc2340b8bd0 con 0x7fc23415f1a0 2024-04-26T23:02:18.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.654+0000 7fc233e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc2340b8eb0 con 0x7fc2340d2ae0 2024-04-26T23:02:18.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.654+0000 7fc233e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc2340b9190 con 0x7fc2340d32c0 2024-04-26T23:02:18.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.654+0000 7fc232a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc23415f1a0 0x7fc2340b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:18.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.654+0000 7fc232a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc23415f1a0 0x7fc2340b85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52058/0 (socket says 172.21.15.40:52058) 2024-04-26T23:02:18.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.654+0000 7fc232a00640 1 -- 172.21.15.40:0/3391794610 learned_addr learned my addr 172.21.15.40:0/3391794610 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:18.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.655+0000 7fc233400640 1 --2- 172.21.15.40:0/3391794610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340d32c0 0x7fc2340d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:18.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.655+0000 7fc22be00640 1 --2- 172.21.15.40:0/3391794610 >> v2:172.21.15.103:3300/0 conn(0x7fc2340d2ae0 0x7fc23415f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:18.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.655+0000 7fc232a00640 1 -- 172.21.15.40:0/3391794610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340d32c0 msgr2=0x7fc2340d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:18.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.655+0000 7fc232a00640 1 --2- 172.21.15.40:0/3391794610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340d32c0 0x7fc2340d3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.655+0000 7fc232a00640 1 -- 172.21.15.40:0/3391794610 >> v2:172.21.15.103:3300/0 conn(0x7fc2340d2ae0 msgr2=0x7fc23415f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:18.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.655+0000 7fc232a00640 1 --2- 172.21.15.40:0/3391794610 >> v2:172.21.15.103:3300/0 conn(0x7fc2340d2ae0 0x7fc23415f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.655+0000 7fc232a00640 1 -- 172.21.15.40:0/3391794610 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2340b9470 con 0x7fc23415f1a0 2024-04-26T23:02:18.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.656+0000 7fc232a00640 1 --2- 172.21.15.40:0/3391794610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc23415f1a0 0x7fc2340b85f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc2340d0eb0 tx=0x7fc22402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78af6aa6a64a0d50 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:18.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.656+0000 7fc232000640 1 -- 172.21.15.40:0/3391794610 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224037070 con 0x7fc23415f1a0 2024-04-26T23:02:18.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.656+0000 7fc232000640 1 -- 172.21.15.40:0/3391794610 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc224032440 con 0x7fc23415f1a0 2024-04-26T23:02:18.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.656+0000 7fc232000640 1 -- 172.21.15.40:0/3391794610 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224032810 con 0x7fc23415f1a0 2024-04-26T23:02:18.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.657+0000 7fc233e00640 1 -- 172.21.15.40:0/3391794610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc23415f1a0 msgr2=0x7fc2340b85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:18.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.657+0000 7fc233e00640 1 --2- 172.21.15.40:0/3391794610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc23415f1a0 0x7fc2340b85f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc2340d0eb0 tx=0x7fc22402a1e0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.657+0000 7fc233e00640 1 -- 172.21.15.40:0/3391794610 shutdown_connections 2024-04-26T23:02:18.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.657+0000 7fc233e00640 1 --2- 172.21.15.40:0/3391794610 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340d32c0 0x7fc2340d3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.657+0000 7fc233e00640 1 --2- 172.21.15.40:0/3391794610 >> v2:172.21.15.103:3300/0 conn(0x7fc2340d2ae0 0x7fc23415f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.657+0000 7fc233e00640 1 --2- 172.21.15.40:0/3391794610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc23415f1a0 0x7fc2340b85f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.657+0000 7fc233e00640 1 -- 172.21.15.40:0/3391794610 >> 172.21.15.40:0/3391794610 conn(0x7fc2340d4d90 msgr2=0x7fc2340d5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:18.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.657+0000 7fc233e00640 1 -- 172.21.15.40:0/3391794610 shutdown_connections 2024-04-26T23:02:18.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.658+0000 7fc233e00640 1 -- 172.21.15.40:0/3391794610 wait complete. 2024-04-26T23:02:18.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.659+0000 7fc233e00640 1 Processor -- start 2024-04-26T23:02:18.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.660+0000 7fc233e00640 1 -- start start 2024-04-26T23:02:18.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.660+0000 7fc233e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2340d2ae0 0x7fc2341525b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.660+0000 7fc233e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340d32c0 0x7fc2340ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc233e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc23415f1a0 0x7fc234161780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc233e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc2340cf910 con 0x7fc2340d2ae0 2024-04-26T23:02:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc233e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc2340d0bb0 con 0x7fc23415f1a0 2024-04-26T23:02:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc233e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc2340cfa90 con 0x7fc2340d32c0 2024-04-26T23:02:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc232a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2340d2ae0 0x7fc2341525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc232a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2340d2ae0 0x7fc2341525b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52068/0 (socket says 172.21.15.40:52068) 2024-04-26T23:02:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc232a00640 1 -- 172.21.15.40:0/3906680378 learned_addr learned my addr 172.21.15.40:0/3906680378 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc22be00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340d32c0 0x7fc2340ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc233400640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc23415f1a0 0x7fc234161780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc232a00640 1 -- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340d32c0 msgr2=0x7fc2340ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc232a00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340d32c0 0x7fc2340ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc232a00640 1 -- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc23415f1a0 msgr2=0x7fc234161780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:18.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc232a00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc23415f1a0 0x7fc234161780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc232a00640 1 -- 172.21.15.40:0/3906680378 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23415ff90 con 0x7fc2340d2ae0 2024-04-26T23:02:18.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.661+0000 7fc22be00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340d32c0 0x7fc2340ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:02:18.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.662+0000 7fc232a00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2340d2ae0 0x7fc2341525b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc224000c00 tx=0x7fc2240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:18.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.662+0000 7fc22aa00640 1 -- 172.21.15.40:0/3906680378 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224037040 con 0x7fc2340d2ae0 2024-04-26T23:02:18.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.662+0000 7fc22aa00640 1 -- 172.21.15.40:0/3906680378 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc224042070 con 0x7fc2340d2ae0 2024-04-26T23:02:18.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.662+0000 7fc22aa00640 1 -- 172.21.15.40:0/3906680378 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224030880 con 0x7fc2340d2ae0 2024-04-26T23:02:18.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.662+0000 7fc233e00640 1 -- 172.21.15.40:0/3906680378 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc234160ad0 con 0x7fc2340d2ae0 2024-04-26T23:02:18.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.662+0000 7fc233e00640 1 -- 172.21.15.40:0/3906680378 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc234160510 con 0x7fc2340d2ae0 2024-04-26T23:02:18.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.663+0000 7fc233e00640 1 -- 172.21.15.40:0/3906680378 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc234069370 con 0x7fc2340d2ae0 2024-04-26T23:02:18.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.664+0000 7fc22aa00640 1 -- 172.21.15.40:0/3906680378 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc224030a20 con 0x7fc2340d2ae0 2024-04-26T23:02:18.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.665+0000 7fc22aa00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc21006c920 0x7fc21006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:18.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.665+0000 7fc22aa00640 1 -- 172.21.15.40:0/3906680378 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc2240ac860 con 0x7fc2340d2ae0 2024-04-26T23:02:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.666+0000 7fc22be00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc21006c920 0x7fc21006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:18.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.667+0000 7fc22be00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc21006c920 0x7fc21006ede0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc2341f8190 tx=0x7fc21c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:18.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.670+0000 7fc22aa00640 1 -- 172.21.15.40:0/3906680378 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc22407a7f0 con 0x7fc2340d2ae0 2024-04-26T23:02:18.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.870+0000 7fc233e00640 1 -- 172.21.15.40:0/3906680378 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc23415d650 con 0x7fc2340d2ae0 2024-04-26T23:02:18.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.871+0000 7fc22aa00640 1 -- 172.21.15.40:0/3906680378 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fc22407a0f0 con 0x7fc2340d2ae0 2024-04-26T23:02:18.872 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:02:18.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.874+0000 7fc233e00640 1 -- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc21006c920 msgr2=0x7fc21006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:18.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.874+0000 7fc233e00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc21006c920 0x7fc21006ede0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc2341f8190 tx=0x7fc21c008040 comp rx=0 tx=0).stop 2024-04-26T23:02:18.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.874+0000 7fc233e00640 1 -- 172.21.15.40:0/3906680378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2340d2ae0 msgr2=0x7fc2341525b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:18.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.874+0000 7fc233e00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2340d2ae0 0x7fc2341525b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc224000c00 tx=0x7fc2240046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.874+0000 7fc233e00640 1 -- 172.21.15.40:0/3906680378 shutdown_connections 2024-04-26T23:02:18.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.874+0000 7fc233e00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc21006c920 0x7fc21006ede0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.874+0000 7fc233e00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc23415f1a0 0x7fc234161780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.874+0000 7fc233e00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2340d32c0 0x7fc2340ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.874+0000 7fc233e00640 1 --2- 172.21.15.40:0/3906680378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2340d2ae0 0x7fc2341525b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:18.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.874+0000 7fc233e00640 1 -- 172.21.15.40:0/3906680378 >> 172.21.15.40:0/3906680378 conn(0x7fc2340d4d90 msgr2=0x7fc234154360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:18.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.874+0000 7fc233e00640 1 -- 172.21.15.40:0/3906680378 shutdown_connections 2024-04-26T23:02:18.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:18.874+0000 7fc233e00640 1 -- 172.21.15.40:0/3906680378 wait complete. 2024-04-26T23:02:18.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:02:19.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.098+0000 7f4d6e000640 1 Processor -- start 2024-04-26T23:02:19.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.099+0000 7f4d6e000640 1 -- start start 2024-04-26T23:02:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.100+0000 7f4d6e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d6815f440 0x7f4d680d1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.100+0000 7f4d6e000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4d680d1da0 0x7f4d680cab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.100+0000 7f4d6e000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d680d5c20 con 0x7f4d6815f060 2024-04-26T23:02:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.100+0000 7f4d6e000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4d680c99b0 con 0x7f4d680d1da0 2024-04-26T23:02:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.100+0000 7f4d6e000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4d680cfc40 con 0x7f4d6815f440 2024-04-26T23:02:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.101+0000 7f4d6d600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4d680d1da0 0x7f4d680cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.101+0000 7f4d67e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d6815f440 0x7f4d680d1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.101+0000 7f4d6d600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4d680d1da0 0x7f4d680cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:39252/0 (socket says 172.21.15.40:39252) 2024-04-26T23:02:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.101+0000 7f4d6d600640 1 -- 172.21.15.40:0/973187876 learned_addr learned my addr 172.21.15.40:0/973187876 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.102+0000 7f4d6d600640 1 -- 172.21.15.40:0/973187876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d6815f440 msgr2=0x7f4d680d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.102+0000 7f4d6d600640 1 --2- 172.21.15.40:0/973187876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d6815f440 0x7f4d680d1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.102+0000 7f4d6d600640 1 -- 172.21.15.40:0/973187876 >> v1:172.21.15.103:6789/0 conn(0x7f4d6815f060 legacy=0x7f4d680d3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.102+0000 7f4d6d600640 1 -- 172.21.15.40:0/973187876 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d680cff20 con 0x7f4d680d1da0 2024-04-26T23:02:19.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.103+0000 7f4d6d600640 1 --2- 172.21.15.40:0/973187876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d680d1da0 0x7f4d680cab70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4d60009e80 tx=0x7f4d6002a750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b49660c1270532b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:19.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.104+0000 7f4d67400640 1 -- 172.21.15.40:0/973187876 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d60036070 con 0x7f4d680d1da0 2024-04-26T23:02:19.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.104+0000 7f4d67400640 1 -- 172.21.15.40:0/973187876 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d60031600 con 0x7f4d680d1da0 2024-04-26T23:02:19.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.104+0000 7f4d67400640 1 -- 172.21.15.40:0/973187876 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d600319d0 con 0x7f4d680d1da0 2024-04-26T23:02:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.105+0000 7f4d6e000640 1 -- 172.21.15.40:0/973187876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d680d1da0 msgr2=0x7f4d680cab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.105+0000 7f4d6e000640 1 --2- 172.21.15.40:0/973187876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d680d1da0 0x7f4d680cab70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4d60009e80 tx=0x7f4d6002a750 comp rx=0 tx=0).stop 2024-04-26T23:02:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.105+0000 7f4d6e000640 1 -- 172.21.15.40:0/973187876 shutdown_connections 2024-04-26T23:02:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.105+0000 7f4d6e000640 1 --2- 172.21.15.40:0/973187876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d680d1da0 0x7f4d680cab70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.105+0000 7f4d6e000640 1 --2- 172.21.15.40:0/973187876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d6815f440 0x7f4d680d1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.105+0000 7f4d6e000640 1 -- 172.21.15.40:0/973187876 >> 172.21.15.40:0/973187876 conn(0x7f4d680d4e50 msgr2=0x7f4d680d5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.105+0000 7f4d6e000640 1 -- 172.21.15.40:0/973187876 shutdown_connections 2024-04-26T23:02:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.105+0000 7f4d6e000640 1 -- 172.21.15.40:0/973187876 wait complete. 2024-04-26T23:02:19.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.107+0000 7f4d6e000640 1 Processor -- start 2024-04-26T23:02:19.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.107+0000 7f4d6e000640 1 -- start start 2024-04-26T23:02:19.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.108+0000 7f4d6e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d680d1da0 0x7f4d6815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.108+0000 7f4d6e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6815f060 0x7f4d6815b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.108+0000 7f4d6e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d6815f440 0x7f4d680cec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.108+0000 7f4d6e000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4d680d0ce0 con 0x7f4d6815f060 2024-04-26T23:02:19.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.108+0000 7f4d6e000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4d680d04e0 con 0x7f4d680d1da0 2024-04-26T23:02:19.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.108+0000 7f4d6e000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4d6815b290 con 0x7f4d6815f440 2024-04-26T23:02:19.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.108+0000 7f4d67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6815f060 0x7f4d6815b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.108+0000 7f4d67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6815f060 0x7f4d6815b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52076/0 (socket says 172.21.15.40:52076) 2024-04-26T23:02:19.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.108+0000 7f4d67e00640 1 -- 172.21.15.40:0/48974512 learned_addr learned my addr 172.21.15.40:0/48974512 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:19.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.108+0000 7f4d6cc00640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d680d1da0 0x7f4d6815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.109+0000 7f4d6d600640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d6815f440 0x7f4d680cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.109+0000 7f4d67e00640 1 -- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d6815f440 msgr2=0x7f4d680cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:19.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.109+0000 7f4d67e00640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d6815f440 0x7f4d680cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.109+0000 7f4d67e00640 1 -- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d680d1da0 msgr2=0x7f4d6815ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:19.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.109+0000 7f4d67e00640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d680d1da0 0x7f4d6815ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.109+0000 7f4d67e00640 1 -- 172.21.15.40:0/48974512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d680cbf20 con 0x7f4d6815f060 2024-04-26T23:02:19.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.109+0000 7f4d67e00640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6815f060 0x7f4d6815b4e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400e870 tx=0x7f4d5400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.109+0000 7f4d64c00640 1 -- 172.21.15.40:0/48974512 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d54017070 con 0x7f4d6815f060 2024-04-26T23:02:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.110+0000 7f4d64c00640 1 -- 172.21.15.40:0/48974512 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d5400c940 con 0x7f4d6815f060 2024-04-26T23:02:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.110+0000 7f4d64c00640 1 -- 172.21.15.40:0/48974512 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5400cd30 con 0x7f4d6815f060 2024-04-26T23:02:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.110+0000 7f4d6e000640 1 -- 172.21.15.40:0/48974512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d680cc200 con 0x7f4d6815f060 2024-04-26T23:02:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.110+0000 7f4d6e000640 1 -- 172.21.15.40:0/48974512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d680cc7c0 con 0x7f4d6815f060 2024-04-26T23:02:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.111+0000 7f4d6e000640 1 -- 172.21.15.40:0/48974512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d68069a80 con 0x7f4d6815f060 2024-04-26T23:02:19.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.113+0000 7f4d64c00640 1 -- 172.21.15.40:0/48974512 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4d5401a070 con 0x7f4d6815f060 2024-04-26T23:02:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.114+0000 7f4d64c00640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4d4806c8f0 0x7f4d4806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.114+0000 7f4d64c00640 1 -- 172.21.15.40:0/48974512 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4d5408c1f0 con 0x7f4d6815f060 2024-04-26T23:02:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.115+0000 7f4d6cc00640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4d4806c8f0 0x7f4d4806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.116+0000 7f4d6cc00640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4d4806c8f0 0x7f4d4806edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c001640 tx=0x7f4d5c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:19.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.119+0000 7f4d64c00640 1 -- 172.21.15.40:0/48974512 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4d5405a180 con 0x7f4d6815f060 2024-04-26T23:02:19.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.321+0000 7f4d6e000640 1 -- 172.21.15.40:0/48974512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4d68068e00 con 0x7f4d6815f060 2024-04-26T23:02:19.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.322+0000 7f4d64c00640 1 -- 172.21.15.40:0/48974512 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f4d54059a80 con 0x7f4d6815f060 2024-04-26T23:02:19.323 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:19.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.325+0000 7f4d6e000640 1 -- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4d4806c8f0 msgr2=0x7f4d4806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:19.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.325+0000 7f4d6e000640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4d4806c8f0 0x7f4d4806edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c001640 tx=0x7f4d5c009040 comp rx=0 tx=0).stop 2024-04-26T23:02:19.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.325+0000 7f4d6e000640 1 -- 172.21.15.40:0/48974512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6815f060 msgr2=0x7f4d6815b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:19.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.325+0000 7f4d6e000640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6815f060 0x7f4d6815b4e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400e870 tx=0x7f4d5400ed30 comp rx=0 tx=0).stop 2024-04-26T23:02:19.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.325+0000 7f4d6e000640 1 -- 172.21.15.40:0/48974512 shutdown_connections 2024-04-26T23:02:19.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.325+0000 7f4d6e000640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4d4806c8f0 0x7f4d4806edb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.325+0000 7f4d6e000640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d6815f440 0x7f4d680cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.325+0000 7f4d6e000640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d6815f060 0x7f4d6815b4e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.325+0000 7f4d6e000640 1 --2- 172.21.15.40:0/48974512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d680d1da0 0x7f4d6815ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.325+0000 7f4d6e000640 1 -- 172.21.15.40:0/48974512 >> 172.21.15.40:0/48974512 conn(0x7f4d680d4e50 msgr2=0x7f4d680cb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:19.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.325+0000 7f4d6e000640 1 -- 172.21.15.40:0/48974512 shutdown_connections 2024-04-26T23:02:19.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.325+0000 7f4d6e000640 1 -- 172.21.15.40:0/48974512 wait complete. 2024-04-26T23:02:19.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:02:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.549+0000 7f329e600640 1 Processor -- start 2024-04-26T23:02:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.549+0000 7f329e600640 1 -- start start 2024-04-26T23:02:19.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.551+0000 7f329e600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f32980af7a0 0x7f32980afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.551+0000 7f329e600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32980d0820 con 0x7f3298071fe0 2024-04-26T23:02:19.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.551+0000 7f329e600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32981614c0 con 0x7f329815ce60 2024-04-26T23:02:19.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.551+0000 7f329e600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f32980cc450 con 0x7f32980af7a0 2024-04-26T23:02:19.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.551+0000 7f329dc00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3298071fe0 0x7f3298073760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52442/0 (socket says 172.21.15.40:52442) 2024-04-26T23:02:19.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.551+0000 7f329dc00640 1 -- 172.21.15.40:0/884058329 learned_addr learned my addr 172.21.15.40:0/884058329 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.551+0000 7f328fe00640 1 --2- 172.21.15.40:0/884058329 >> v2:172.21.15.103:3300/0 conn(0x7f32980af7a0 0x7f32980afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1052636525 0 0) 0x7f32980d0820 con 0x7f3298071fe0 2024-04-26T23:02:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3278003520 con 0x7f3298071fe0 2024-04-26T23:02:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 599066539 0 0) 0x7f3278003520 con 0x7f3298071fe0 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 >> v2:172.21.15.103:3300/0 conn(0x7f32980af7a0 msgr2=0x7f32980afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 --2- 172.21.15.40:0/884058329 >> v2:172.21.15.103:3300/0 conn(0x7f32980af7a0 0x7f32980afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 >> v1:172.21.15.103:6789/0 conn(0x7f329815ce60 legacy=0x7f32980ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3298161760 con 0x7f3298071fe0 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f32900019f0 con 0x7f3298071fe0 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 >> v1:172.21.15.40:6789/0 conn(0x7f3298071fe0 legacy=0x7f3298073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 --2- 172.21.15.40:0/884058329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3278003a00 0x7f3278003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 --2- 172.21.15.40:0/884058329 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3278004800 0x7f3278004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 --2- 172.21.15.40:0/884058329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3278005210 0x7f327800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32980cc450 con 0x7f3278005210 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f327800c160 con 0x7f3278003a00 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.552+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f327800c360 con 0x7f3278004800 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.553+0000 7f329dc00640 1 --2- 172.21.15.40:0/884058329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3278005210 0x7f327800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.553+0000 7f329d200640 1 --2- 172.21.15.40:0/884058329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3278003a00 0x7f3278003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.553+0000 7f328fe00640 1 --2- 172.21.15.40:0/884058329 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3278004800 0x7f3278004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.553+0000 7f329dc00640 1 -- 172.21.15.40:0/884058329 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3278004800 msgr2=0x7f3278004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.553+0000 7f329dc00640 1 --2- 172.21.15.40:0/884058329 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3278004800 0x7f3278004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.553+0000 7f329dc00640 1 -- 172.21.15.40:0/884058329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3278003a00 msgr2=0x7f3278003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.553+0000 7f329dc00640 1 --2- 172.21.15.40:0/884058329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3278003a00 0x7f3278003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.553+0000 7f329dc00640 1 -- 172.21.15.40:0/884058329 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3278003520 con 0x7f3278005210 2024-04-26T23:02:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.553+0000 7f328fe00640 1 --2- 172.21.15.40:0/884058329 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3278004800 0x7f3278004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.553+0000 7f329d200640 1 --2- 172.21.15.40:0/884058329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3278003a00 0x7f3278003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:02:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.554+0000 7f329dc00640 1 -- 172.21.15.40:0/884058329 reap_dead start 2024-04-26T23:02:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.554+0000 7f329dc00640 1 --2- 172.21.15.40:0/884058329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3278005210 0x7f327800b9c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f329000a780 tx=0x7f3278005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09c79231f3bb60f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.554+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329000a3d0 con 0x7f3278005210 2024-04-26T23:02:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.555+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f329002f070 con 0x7f3278005210 2024-04-26T23:02:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.555+0000 7f3297e00640 1 -- 172.21.15.40:0/884058329 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329002bbe0 con 0x7f3278005210 2024-04-26T23:02:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.555+0000 7f329e600640 1 -- 172.21.15.40:0/884058329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3278005210 msgr2=0x7f327800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:19.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.555+0000 7f329e600640 1 --2- 172.21.15.40:0/884058329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3278005210 0x7f327800b9c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f329000a780 tx=0x7f3278005830 comp rx=0 tx=0).stop 2024-04-26T23:02:19.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.555+0000 7f329e600640 1 -- 172.21.15.40:0/884058329 shutdown_connections 2024-04-26T23:02:19.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.555+0000 7f329e600640 1 --2- 172.21.15.40:0/884058329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3278005210 0x7f327800b9c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.555+0000 7f329e600640 1 -- 172.21.15.40:0/884058329 >> 172.21.15.40:0/884058329 conn(0x7f32980d0a10 msgr2=0x7f32980d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:19.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.555+0000 7f329e600640 1 -- 172.21.15.40:0/884058329 shutdown_connections 2024-04-26T23:02:19.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.555+0000 7f329e600640 1 -- 172.21.15.40:0/884058329 wait complete. 2024-04-26T23:02:19.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.557+0000 7f329e600640 1 Processor -- start 2024-04-26T23:02:19.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.557+0000 7f329e600640 1 -- start start 2024-04-26T23:02:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.558+0000 7f329e600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3278005210 0x7f32981563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.558+0000 7f329e600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32980c7ae0 0x7f32980c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.558+0000 7f329e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32980c7f60 0x7f32980c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.558+0000 7f329e600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32980afde0 con 0x7f32980c7f60 2024-04-26T23:02:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.558+0000 7f329e600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32980afc60 con 0x7f3278005210 2024-04-26T23:02:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.558+0000 7f329e600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f32980aff60 con 0x7f32980c7ae0 2024-04-26T23:02:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.559+0000 7f329dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32980c7f60 0x7f32980c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.559+0000 7f329dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32980c7f60 0x7f32980c6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52098/0 (socket says 172.21.15.40:52098) 2024-04-26T23:02:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.559+0000 7f329dc00640 1 -- 172.21.15.40:0/1260603261 learned_addr learned my addr 172.21.15.40:0/1260603261 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.559+0000 7f328fe00640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32980c7ae0 0x7f32980c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.559+0000 7f329d200640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3278005210 0x7f32981563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.559+0000 7f329dc00640 1 -- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32980c7ae0 msgr2=0x7f32980c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.559+0000 7f329dc00640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32980c7ae0 0x7f32980c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.559+0000 7f329dc00640 1 -- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3278005210 msgr2=0x7f32981563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.559+0000 7f329dc00640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3278005210 0x7f32981563f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.559+0000 7f329dc00640 1 -- 172.21.15.40:0/1260603261 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3298156930 con 0x7f32980c7f60 2024-04-26T23:02:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.560+0000 7f329dc00640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32980c7f60 0x7f32980c6680 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f329000a780 tx=0x7f329002d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.560+0000 7f3295600640 1 -- 172.21.15.40:0/1260603261 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329002b780 con 0x7f32980c7f60 2024-04-26T23:02:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.560+0000 7f3295600640 1 -- 172.21.15.40:0/1260603261 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f329002b920 con 0x7f32980c7f60 2024-04-26T23:02:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.560+0000 7f3295600640 1 -- 172.21.15.40:0/1260603261 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329002bcd0 con 0x7f32980c7f60 2024-04-26T23:02:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.560+0000 7f329e600640 1 -- 172.21.15.40:0/1260603261 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32980b4ea0 con 0x7f32980c7f60 2024-04-26T23:02:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.560+0000 7f329e600640 1 -- 172.21.15.40:0/1260603261 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32980b5460 con 0x7f32980c7f60 2024-04-26T23:02:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.561+0000 7f329e600640 1 -- 172.21.15.40:0/1260603261 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3298069370 con 0x7f32980c7f60 2024-04-26T23:02:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.562+0000 7f3295600640 1 -- 172.21.15.40:0/1260603261 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3290034470 con 0x7f32980c7f60 2024-04-26T23:02:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.563+0000 7f3295600640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f327406c920 0x7f327406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.563+0000 7f3295600640 1 -- 172.21.15.40:0/1260603261 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f32900ac130 con 0x7f32980c7f60 2024-04-26T23:02:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.564+0000 7f329d200640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f327406c920 0x7f327406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:19.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.565+0000 7f329d200640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f327406c920 0x7f327406ede0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3280000c00 tx=0x7f3280009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:19.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.568+0000 7f3295600640 1 -- 172.21.15.40:0/1260603261 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f329007a150 con 0x7f32980c7f60 2024-04-26T23:02:19.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.777+0000 7f329e600640 1 -- 172.21.15.40:0/1260603261 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f32980d01f0 con 0x7f32980c7f60 2024-04-26T23:02:19.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.778+0000 7f3295600640 1 -- 172.21.15.40:0/1260603261 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f3290079a50 con 0x7f32980c7f60 2024-04-26T23:02:19.778 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-04-26T23:02:19.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.780+0000 7f329e600640 1 -- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f327406c920 msgr2=0x7f327406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:19.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.780+0000 7f329e600640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f327406c920 0x7f327406ede0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3280000c00 tx=0x7f3280009040 comp rx=0 tx=0).stop 2024-04-26T23:02:19.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.780+0000 7f329e600640 1 -- 172.21.15.40:0/1260603261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32980c7f60 msgr2=0x7f32980c6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:19.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.780+0000 7f329e600640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32980c7f60 0x7f32980c6680 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f329000a780 tx=0x7f329002d070 comp rx=0 tx=0).stop 2024-04-26T23:02:19.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.780+0000 7f329e600640 1 -- 172.21.15.40:0/1260603261 shutdown_connections 2024-04-26T23:02:19.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.780+0000 7f329e600640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f327406c920 0x7f327406ede0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.780+0000 7f329e600640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32980c7f60 0x7f32980c6680 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.780+0000 7f329e600640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32980c7ae0 0x7f32980c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.780+0000 7f329e600640 1 --2- 172.21.15.40:0/1260603261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3278005210 0x7f32981563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:19.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.780+0000 7f329e600640 1 -- 172.21.15.40:0/1260603261 >> 172.21.15.40:0/1260603261 conn(0x7f32980d0a10 msgr2=0x7f3298160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:19.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.780+0000 7f329e600640 1 -- 172.21.15.40:0/1260603261 shutdown_connections 2024-04-26T23:02:19.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:19.780+0000 7f329e600640 1 -- 172.21.15.40:0/1260603261 wait complete. 2024-04-26T23:02:19.797 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T23:02:19.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:19.798 DEBUG:teuthology.orchestra.run.smithi040:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-04-26T23:02:19.887 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:02:19.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:19.888 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T23:02:19.937 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:02:19.937 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:19.996 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:02:19.996 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:20.045 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:02:20.045 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:20.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:20.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.360+0000 7fd70e800640 1 Processor -- start 2024-04-26T23:02:20.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.361+0000 7fd70e800640 1 -- start start 2024-04-26T23:02:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.362+0000 7fd70e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd70815f2d0 0x7fd7080afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:20.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.363+0000 7fd70e800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7080af7b0 con 0x7fd7080cc590 2024-04-26T23:02:20.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.363+0000 7fd70e800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7080afa90 con 0x7fd7080ce5e0 2024-04-26T23:02:20.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.363+0000 7fd70e800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd7080cbd90 con 0x7fd70815f2d0 2024-04-26T23:02:20.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.363+0000 7fd70de00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd7080cc590 0x7fd70806f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52444/0 (socket says 172.21.15.40:52444) 2024-04-26T23:02:20.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.363+0000 7fd70de00640 1 -- 172.21.15.40:0/4234820167 learned_addr learned my addr 172.21.15.40:0/4234820167 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:20.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.363+0000 7fd70d400640 1 --2- 172.21.15.40:0/4234820167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd70815f2d0 0x7fd7080afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:20.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 627172148 0 0) 0x7fd7080af7b0 con 0x7fd7080cc590 2024-04-26T23:02:20.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6f0005730 con 0x7fd7080cc590 2024-04-26T23:02:20.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2171913732 0 0) 0x7fd6f0005730 con 0x7fd7080cc590 2024-04-26T23:02:20.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd70815f2d0 msgr2=0x7fd7080afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:20.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 --2- 172.21.15.40:0/4234820167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd70815f2d0 0x7fd7080afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:20.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 >> v1:172.21.15.103:6789/0 conn(0x7fd7080ce5e0 legacy=0x7fd7080c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:20.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7080cc070 con 0x7fd7080cc590 2024-04-26T23:02:20.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fd7000019f0 con 0x7fd7080cc590 2024-04-26T23:02:20.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 >> v1:172.21.15.40:6789/0 conn(0x7fd7080cc590 legacy=0x7fd70806f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:20.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 --2- 172.21.15.40:0/4234820167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd70815f2d0 0x7fd6f0005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:20.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 --2- 172.21.15.40:0/4234820167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f0006130 0x7fd6f0006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:20.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 --2- 172.21.15.40:0/4234820167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0006f50 0x7fd6f00073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:20.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd7080cbd90 con 0x7fd6f0006130 2024-04-26T23:02:20.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f0006810 con 0x7fd6f0006f50 2024-04-26T23:02:20.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.364+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f0010020 con 0x7fd70815f2d0 2024-04-26T23:02:20.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.365+0000 7fd70ca00640 1 --2- 172.21.15.40:0/4234820167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f0006130 0x7fd6f0006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:20.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.365+0000 7fd70d400640 1 --2- 172.21.15.40:0/4234820167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd70815f2d0 0x7fd6f0005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:20.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.365+0000 7fd70de00640 1 --2- 172.21.15.40:0/4234820167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0006f50 0x7fd6f00073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:20.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.365+0000 7fd70ca00640 1 -- 172.21.15.40:0/4234820167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd70815f2d0 msgr2=0x7fd6f0005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:20.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.365+0000 7fd70ca00640 1 --2- 172.21.15.40:0/4234820167 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd70815f2d0 0x7fd6f0005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:20.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.365+0000 7fd70ca00640 1 -- 172.21.15.40:0/4234820167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0006f50 msgr2=0x7fd6f00073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:20.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.365+0000 7fd70ca00640 1 --2- 172.21.15.40:0/4234820167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0006f50 0x7fd6f00073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:20.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.365+0000 7fd70ca00640 1 -- 172.21.15.40:0/4234820167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f0005730 con 0x7fd6f0006130 2024-04-26T23:02:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.366+0000 7fd70ca00640 1 --2- 172.21.15.40:0/4234820167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f0006130 0x7fd6f0006a10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4001860 tx=0x7fd6f0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3476889fd262eb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.366+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4037070 con 0x7fd6f0006130 2024-04-26T23:02:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.366+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6f402abf0 con 0x7fd6f0006130 2024-04-26T23:02:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.366+0000 7fd707e00640 1 -- 172.21.15.40:0/4234820167 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4001430 con 0x7fd6f0006130 2024-04-26T23:02:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.367+0000 7fd70e800640 1 -- 172.21.15.40:0/4234820167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f0006130 msgr2=0x7fd6f0006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.367+0000 7fd70e800640 1 --2- 172.21.15.40:0/4234820167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f0006130 0x7fd6f0006a10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4001860 tx=0x7fd6f0007a40 comp rx=0 tx=0).stop 2024-04-26T23:02:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.367+0000 7fd70de00640 1 -- 172.21.15.40:0/4234820167 reap_dead start 2024-04-26T23:02:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.367+0000 7fd70e800640 1 -- 172.21.15.40:0/4234820167 shutdown_connections 2024-04-26T23:02:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.367+0000 7fd70e800640 1 -- 172.21.15.40:0/4234820167 >> 172.21.15.40:0/4234820167 conn(0x7fd7080d0a10 msgr2=0x7fd7080d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.367+0000 7fd70e800640 1 -- 172.21.15.40:0/4234820167 shutdown_connections 2024-04-26T23:02:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.367+0000 7fd70e800640 1 -- 172.21.15.40:0/4234820167 wait complete. 2024-04-26T23:02:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.369+0000 7fd70e800640 1 Processor -- start 2024-04-26T23:02:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.369+0000 7fd70e800640 1 -- start start 2024-04-26T23:02:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.370+0000 7fd70e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0006130 0x7fd7081f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.370+0000 7fd70e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7080c9e10 0x7fd70814c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.370+0000 7fd70e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd708159890 0x7fd708159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:20.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.370+0000 7fd70e800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd7080c5fc0 con 0x7fd7080c9e10 2024-04-26T23:02:20.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.370+0000 7fd70e800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd70806e270 con 0x7fd6f0006130 2024-04-26T23:02:20.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.370+0000 7fd70e800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd70815a230 con 0x7fd708159890 2024-04-26T23:02:20.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.370+0000 7fd70ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7080c9e10 0x7fd70814c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:20.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.370+0000 7fd70ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7080c9e10 0x7fd70814c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52106/0 (socket says 172.21.15.40:52106) 2024-04-26T23:02:20.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.370+0000 7fd70ca00640 1 -- 172.21.15.40:0/1528212498 learned_addr learned my addr 172.21.15.40:0/1528212498 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:20.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.371+0000 7fd70d400640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0006130 0x7fd7081f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:20.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.371+0000 7fd70de00640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd708159890 0x7fd708159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:20.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.371+0000 7fd70ca00640 1 -- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd708159890 msgr2=0x7fd708159cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:20.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.371+0000 7fd70ca00640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd708159890 0x7fd708159cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:20.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.371+0000 7fd70ca00640 1 -- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0006130 msgr2=0x7fd7081f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:20.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.371+0000 7fd70ca00640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0006130 0x7fd7081f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:20.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.371+0000 7fd70ca00640 1 -- 172.21.15.40:0/1528212498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7080b3730 con 0x7fd7080c9e10 2024-04-26T23:02:20.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.371+0000 7fd70de00640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd708159890 0x7fd708159cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:20.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.371+0000 7fd70ca00640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7080c9e10 0x7fd70814c3c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4001980 tx=0x7fd6f4001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:20.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.371+0000 7fd705600640 1 -- 172.21.15.40:0/1528212498 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f403f020 con 0x7fd7080c9e10 2024-04-26T23:02:20.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.371+0000 7fd705600640 1 -- 172.21.15.40:0/1528212498 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6f4005270 con 0x7fd7080c9e10 2024-04-26T23:02:20.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.371+0000 7fd705600640 1 -- 172.21.15.40:0/1528212498 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4037040 con 0x7fd7080c9e10 2024-04-26T23:02:20.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.372+0000 7fd70e800640 1 -- 172.21.15.40:0/1528212498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7081f46d0 con 0x7fd7080c9e10 2024-04-26T23:02:20.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.372+0000 7fd70e800640 1 -- 172.21.15.40:0/1528212498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7081f4c90 con 0x7fd7080c9e10 2024-04-26T23:02:20.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.374+0000 7fd705600640 1 -- 172.21.15.40:0/1528212498 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd6f4042020 con 0x7fd7080c9e10 2024-04-26T23:02:20.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.374+0000 7fd70e800640 1 -- 172.21.15.40:0/1528212498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd708069370 con 0x7fd7080c9e10 2024-04-26T23:02:20.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.375+0000 7fd705600640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd6e806c8f0 0x7fd6e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:20.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.375+0000 7fd705600640 1 -- 172.21.15.40:0/1528212498 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd6f40abef0 con 0x7fd7080c9e10 2024-04-26T23:02:20.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.375+0000 7fd70d400640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd6e806c8f0 0x7fd6e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:20.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.377+0000 7fd70d400640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd6e806c8f0 0x7fd6e806edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009a10 tx=0x7fd6fc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:20.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.384+0000 7fd705600640 1 -- 172.21.15.40:0/1528212498 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd6f4049030 con 0x7fd7080c9e10 2024-04-26T23:02:20.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.610+0000 7fd70e800640 1 -- 172.21.15.40:0/1528212498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd70806de10 con 0x7fd7080c9e10 2024-04-26T23:02:20.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.613+0000 7fd705600640 1 -- 172.21.15.40:0/1528212498 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18812 (secure 0 0 0) 0x7fd6f4079e90 con 0x7fd7080c9e10 2024-04-26T23:02:20.614 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:20.615 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:46.975683+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.616+0000 7fd70e800640 1 -- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd6e806c8f0 msgr2=0x7fd6e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.616+0000 7fd70e800640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd6e806c8f0 0x7fd6e806edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009a10 tx=0x7fd6fc008040 comp rx=0 tx=0).stop 2024-04-26T23:02:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.616+0000 7fd70e800640 1 -- 172.21.15.40:0/1528212498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7080c9e10 msgr2=0x7fd70814c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.616+0000 7fd70e800640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7080c9e10 0x7fd70814c3c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4001980 tx=0x7fd6f4001230 comp rx=0 tx=0).stop 2024-04-26T23:02:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.616+0000 7fd70e800640 1 -- 172.21.15.40:0/1528212498 shutdown_connections 2024-04-26T23:02:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.616+0000 7fd70e800640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd6e806c8f0 0x7fd6e806edb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.616+0000 7fd70e800640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd708159890 0x7fd708159cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.616+0000 7fd70e800640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7080c9e10 0x7fd70814c3c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.616+0000 7fd70e800640 1 --2- 172.21.15.40:0/1528212498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0006130 0x7fd7081f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.616+0000 7fd70e800640 1 -- 172.21.15.40:0/1528212498 >> 172.21.15.40:0/1528212498 conn(0x7fd7080d0a10 msgr2=0x7fd70815ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.617+0000 7fd70e800640 1 -- 172.21.15.40:0/1528212498 shutdown_connections 2024-04-26T23:02:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.617+0000 7fd70e800640 1 -- 172.21.15.40:0/1528212498 wait complete. 2024-04-26T23:02:20.636 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:02:20.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:02:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.852+0000 7fd626e00640 1 Processor -- start 2024-04-26T23:02:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.852+0000 7fd626e00640 1 -- start start 2024-04-26T23:02:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.853+0000 7fd626e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd620073600 0x7fd620072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.854+0000 7fd626e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6201604e0 con 0x7fd62015fe80 2024-04-26T23:02:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.854+0000 7fd626e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd620072200 con 0x7fd6200739e0 2024-04-26T23:02:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.854+0000 7fd626e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fd620073220 con 0x7fd620073600 2024-04-26T23:02:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.854+0000 7fd625a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd620073600 0x7fd620072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.854+0000 7fd626400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd62015fe80 0x7fd6200ae8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52448/0 (socket says 172.21.15.40:52448) 2024-04-26T23:02:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.854+0000 7fd626400640 1 -- 172.21.15.40:0/980925820 learned_addr learned my addr 172.21.15.40:0/980925820 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:20.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.854+0000 7fd625a00640 1 -- 172.21.15.40:0/980925820 >> v1:172.21.15.103:6789/0 conn(0x7fd6200739e0 legacy=0x7fd62015f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:20.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.854+0000 7fd625a00640 1 -- 172.21.15.40:0/980925820 >> v1:172.21.15.40:6789/0 conn(0x7fd62015fe80 legacy=0x7fd6200ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:20.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.854+0000 7fd625a00640 1 -- 172.21.15.40:0/980925820 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6200724e0 con 0x7fd620073600 2024-04-26T23:02:20.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.855+0000 7fd625a00640 1 --2- 172.21.15.40:0/980925820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd620073600 0x7fd620072ce0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd6140097a0 tx=0x7fd61402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed99d5a466fbd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:20.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.856+0000 7fd61fe00640 1 -- 172.21.15.40:0/980925820 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd614037070 con 0x7fd620073600 2024-04-26T23:02:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.856+0000 7fd61fe00640 1 -- 172.21.15.40:0/980925820 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd614032440 con 0x7fd620073600 2024-04-26T23:02:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.856+0000 7fd61fe00640 1 -- 172.21.15.40:0/980925820 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd614032810 con 0x7fd620073600 2024-04-26T23:02:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.856+0000 7fd626e00640 1 -- 172.21.15.40:0/980925820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd620073600 msgr2=0x7fd620072ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.856+0000 7fd626e00640 1 --2- 172.21.15.40:0/980925820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd620073600 0x7fd620072ce0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd6140097a0 tx=0x7fd61402a1e0 comp rx=0 tx=0).stop 2024-04-26T23:02:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.857+0000 7fd626e00640 1 -- 172.21.15.40:0/980925820 shutdown_connections 2024-04-26T23:02:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.857+0000 7fd626e00640 1 --2- 172.21.15.40:0/980925820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd620073600 0x7fd620072ce0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.857+0000 7fd626e00640 1 -- 172.21.15.40:0/980925820 >> 172.21.15.40:0/980925820 conn(0x7fd6200b2720 msgr2=0x7fd6200b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.857+0000 7fd626e00640 1 -- 172.21.15.40:0/980925820 shutdown_connections 2024-04-26T23:02:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.857+0000 7fd626e00640 1 -- 172.21.15.40:0/980925820 wait complete. 2024-04-26T23:02:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.858+0000 7fd626e00640 1 Processor -- start 2024-04-26T23:02:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.859+0000 7fd626e00640 1 -- start start 2024-04-26T23:02:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.859+0000 7fd626e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd620073600 0x7fd6201513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.859+0000 7fd626e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6200739e0 0x7fd620154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.860+0000 7fd626e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd62015fe80 0x7fd620155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.860+0000 7fd626e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd620160290 con 0x7fd62015fe80 2024-04-26T23:02:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.860+0000 7fd626e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd620072850 con 0x7fd620073600 2024-04-26T23:02:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.860+0000 7fd626e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd620071e20 con 0x7fd6200739e0 2024-04-26T23:02:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.860+0000 7fd626400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd62015fe80 0x7fd620155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.860+0000 7fd626400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd62015fe80 0x7fd620155530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52126/0 (socket says 172.21.15.40:52126) 2024-04-26T23:02:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.860+0000 7fd626400640 1 -- 172.21.15.40:0/647880107 learned_addr learned my addr 172.21.15.40:0/647880107 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.860+0000 7fd625a00640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd620073600 0x7fd6201513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.860+0000 7fd625000640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6200739e0 0x7fd620154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.860+0000 7fd626400640 1 -- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6200739e0 msgr2=0x7fd620154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.860+0000 7fd626400640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6200739e0 0x7fd620154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.860+0000 7fd626400640 1 -- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd620073600 msgr2=0x7fd6201513e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:20.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.860+0000 7fd626400640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd620073600 0x7fd6201513e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:20.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.861+0000 7fd626400640 1 -- 172.21.15.40:0/647880107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd620151920 con 0x7fd62015fe80 2024-04-26T23:02:20.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.861+0000 7fd626400640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd62015fe80 0x7fd620155530 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd61800cab0 tx=0x7fd61800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:20.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.861+0000 7fd61d600640 1 -- 172.21.15.40:0/647880107 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618017070 con 0x7fd62015fe80 2024-04-26T23:02:20.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.861+0000 7fd61d600640 1 -- 172.21.15.40:0/647880107 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd61800ac30 con 0x7fd62015fe80 2024-04-26T23:02:20.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.861+0000 7fd61d600640 1 -- 172.21.15.40:0/647880107 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618012400 con 0x7fd62015fe80 2024-04-26T23:02:20.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.861+0000 7fd626e00640 1 -- 172.21.15.40:0/647880107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd62015b570 con 0x7fd62015fe80 2024-04-26T23:02:20.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.861+0000 7fd626e00640 1 -- 172.21.15.40:0/647880107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd620152840 con 0x7fd62015fe80 2024-04-26T23:02:20.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.862+0000 7fd626e00640 1 -- 172.21.15.40:0/647880107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd620069370 con 0x7fd62015fe80 2024-04-26T23:02:20.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.863+0000 7fd61d600640 1 -- 172.21.15.40:0/647880107 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd61801a070 con 0x7fd62015fe80 2024-04-26T23:02:20.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.864+0000 7fd61d600640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd60006c8f0 0x7fd60006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:20.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.864+0000 7fd61d600640 1 -- 172.21.15.40:0/647880107 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd61808bb30 con 0x7fd62015fe80 2024-04-26T23:02:20.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.865+0000 7fd625a00640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd60006c8f0 0x7fd60006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:20.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.866+0000 7fd625a00640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd60006c8f0 0x7fd60006edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd614034000 tx=0x7fd61402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:20.868+0000 7fd61d600640 1 -- 172.21.15.40:0/647880107 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd618059500 con 0x7fd62015fe80 2024-04-26T23:02:21.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.070+0000 7fd626e00640 1 -- 172.21.15.40:0/647880107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd6200b3480 con 0x7fd62015fe80 2024-04-26T23:02:21.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.071+0000 7fd61d600640 1 -- 172.21.15.40:0/647880107 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fd61805d800 con 0x7fd62015fe80 2024-04-26T23:02:21.071 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:02:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.073+0000 7fd626e00640 1 -- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd60006c8f0 msgr2=0x7fd60006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.073+0000 7fd626e00640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd60006c8f0 0x7fd60006edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd614034000 tx=0x7fd61402e000 comp rx=0 tx=0).stop 2024-04-26T23:02:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.073+0000 7fd626e00640 1 -- 172.21.15.40:0/647880107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd62015fe80 msgr2=0x7fd620155530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.073+0000 7fd626e00640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd62015fe80 0x7fd620155530 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd61800cab0 tx=0x7fd61800cf70 comp rx=0 tx=0).stop 2024-04-26T23:02:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.074+0000 7fd626e00640 1 -- 172.21.15.40:0/647880107 shutdown_connections 2024-04-26T23:02:21.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.074+0000 7fd626e00640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd60006c8f0 0x7fd60006edb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.074+0000 7fd626e00640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd62015fe80 0x7fd620155530 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.074+0000 7fd626e00640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6200739e0 0x7fd620154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.074+0000 7fd626e00640 1 --2- 172.21.15.40:0/647880107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd620073600 0x7fd6201513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.074+0000 7fd626e00640 1 -- 172.21.15.40:0/647880107 >> 172.21.15.40:0/647880107 conn(0x7fd6200b2720 msgr2=0x7fd6200b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:21.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.074+0000 7fd626e00640 1 -- 172.21.15.40:0/647880107 shutdown_connections 2024-04-26T23:02:21.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.074+0000 7fd626e00640 1 -- 172.21.15.40:0/647880107 wait complete. 2024-04-26T23:02:21.095 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:02:21.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.306+0000 7fea76400640 1 Processor -- start 2024-04-26T23:02:21.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.306+0000 7fea76400640 1 -- start start 2024-04-26T23:02:21.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.308+0000 7fea76400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea70072ce0 0x7fea7015f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:21.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.308+0000 7fea76400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fea7015fcb0 0x7fea700ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:21.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.308+0000 7fea76400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea701604e0 con 0x7fea70073600 2024-04-26T23:02:21.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.308+0000 7fea76400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fea70072850 con 0x7fea7015fcb0 2024-04-26T23:02:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.308+0000 7fea76400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fea70072200 con 0x7fea70072ce0 2024-04-26T23:02:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.309+0000 7fea75000640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fea70073600 0x7fea700739e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:45516/0 (socket says 172.21.15.40:45516) 2024-04-26T23:02:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.309+0000 7fea6fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea70072ce0 0x7fea7015f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.309+0000 7fea75000640 1 -- 172.21.15.40:0/1480796618 learned_addr learned my addr 172.21.15.40:0/1480796618 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.309+0000 7fea75a00640 1 --2- 172.21.15.40:0/1480796618 >> v2:172.21.15.103:3300/0 conn(0x7fea7015fcb0 0x7fea700ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:21.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.310+0000 7fea6fe00640 1 -- 172.21.15.40:0/1480796618 >> v2:172.21.15.103:3300/0 conn(0x7fea7015fcb0 msgr2=0x7fea700ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:21.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.310+0000 7fea6fe00640 1 --2- 172.21.15.40:0/1480796618 >> v2:172.21.15.103:3300/0 conn(0x7fea7015fcb0 0x7fea700ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.310+0000 7fea6fe00640 1 -- 172.21.15.40:0/1480796618 >> v1:172.21.15.103:6789/0 conn(0x7fea70073600 legacy=0x7fea700739e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:21.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.310+0000 7fea6fe00640 1 -- 172.21.15.40:0/1480796618 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea700724e0 con 0x7fea70072ce0 2024-04-26T23:02:21.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.311+0000 7fea6fe00640 1 --2- 172.21.15.40:0/1480796618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea70072ce0 0x7fea7015f770 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fea5c0097a0 tx=0x7fea5c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3b553b25c41a0b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:21.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.312+0000 7fea6f400640 1 -- 172.21.15.40:0/1480796618 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5c037070 con 0x7fea70072ce0 2024-04-26T23:02:21.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.312+0000 7fea6f400640 1 -- 172.21.15.40:0/1480796618 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea5c032440 con 0x7fea70072ce0 2024-04-26T23:02:21.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.312+0000 7fea6f400640 1 -- 172.21.15.40:0/1480796618 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5c032810 con 0x7fea70072ce0 2024-04-26T23:02:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.312+0000 7fea76400640 1 -- 172.21.15.40:0/1480796618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea70072ce0 msgr2=0x7fea7015f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.312+0000 7fea76400640 1 --2- 172.21.15.40:0/1480796618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea70072ce0 0x7fea7015f770 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fea5c0097a0 tx=0x7fea5c02a180 comp rx=0 tx=0).stop 2024-04-26T23:02:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.313+0000 7fea76400640 1 -- 172.21.15.40:0/1480796618 shutdown_connections 2024-04-26T23:02:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.313+0000 7fea76400640 1 --2- 172.21.15.40:0/1480796618 >> v2:172.21.15.103:3300/0 conn(0x7fea7015fcb0 0x7fea700ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.313+0000 7fea76400640 1 --2- 172.21.15.40:0/1480796618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea70072ce0 0x7fea7015f770 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.313+0000 7fea76400640 1 -- 172.21.15.40:0/1480796618 >> 172.21.15.40:0/1480796618 conn(0x7fea700b2720 msgr2=0x7fea700b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.313+0000 7fea76400640 1 -- 172.21.15.40:0/1480796618 shutdown_connections 2024-04-26T23:02:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.313+0000 7fea76400640 1 -- 172.21.15.40:0/1480796618 wait complete. 2024-04-26T23:02:21.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.315+0000 7fea76400640 1 Processor -- start 2024-04-26T23:02:21.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.315+0000 7fea76400640 1 -- start start 2024-04-26T23:02:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.316+0000 7fea76400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea70072ce0 0x7fea70162120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.316+0000 7fea76400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea70073600 0x7fea701526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.316+0000 7fea76400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea7015fcb0 0x7fea7014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.316+0000 7fea76400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fea701602a0 con 0x7fea70073600 2024-04-26T23:02:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.316+0000 7fea76400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea70160120 con 0x7fea7015fcb0 2024-04-26T23:02:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.316+0000 7fea76400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fea70071e20 con 0x7fea70072ce0 2024-04-26T23:02:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.317+0000 7fea6fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea70073600 0x7fea701526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.317+0000 7fea6fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea70073600 0x7fea701526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52134/0 (socket says 172.21.15.40:52134) 2024-04-26T23:02:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.317+0000 7fea6fe00640 1 -- 172.21.15.40:0/2824210952 learned_addr learned my addr 172.21.15.40:0/2824210952 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.317+0000 7fea75000640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea70072ce0 0x7fea70162120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.317+0000 7fea75a00640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea7015fcb0 0x7fea7014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.317+0000 7fea6fe00640 1 -- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea70072ce0 msgr2=0x7fea70162120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.317+0000 7fea6fe00640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea70072ce0 0x7fea70162120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.317+0000 7fea6fe00640 1 -- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea7015fcb0 msgr2=0x7fea7014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.317+0000 7fea6fe00640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea7015fcb0 0x7fea7014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.317+0000 7fea6fe00640 1 -- 172.21.15.40:0/2824210952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea70152fd0 con 0x7fea70073600 2024-04-26T23:02:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.318+0000 7fea6fe00640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea70073600 0x7fea701526d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fea5c035040 tx=0x7fea5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.318+0000 7fea6cc00640 1 -- 172.21.15.40:0/2824210952 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5c037040 con 0x7fea70073600 2024-04-26T23:02:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.318+0000 7fea6cc00640 1 -- 172.21.15.40:0/2824210952 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea5c030500 con 0x7fea70073600 2024-04-26T23:02:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.318+0000 7fea6cc00640 1 -- 172.21.15.40:0/2824210952 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5c030910 con 0x7fea70073600 2024-04-26T23:02:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.318+0000 7fea76400640 1 -- 172.21.15.40:0/2824210952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea701532b0 con 0x7fea70073600 2024-04-26T23:02:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.318+0000 7fea76400640 1 -- 172.21.15.40:0/2824210952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea70153870 con 0x7fea70073600 2024-04-26T23:02:21.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.319+0000 7fea76400640 1 -- 172.21.15.40:0/2824210952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea70069370 con 0x7fea70073600 2024-04-26T23:02:21.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.320+0000 7fea6cc00640 1 -- 172.21.15.40:0/2824210952 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fea5c044020 con 0x7fea70073600 2024-04-26T23:02:21.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.321+0000 7fea6cc00640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea5006c8f0 0x7fea5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:21.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.321+0000 7fea6cc00640 1 -- 172.21.15.40:0/2824210952 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fea5c0ac290 con 0x7fea70073600 2024-04-26T23:02:21.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.322+0000 7fea75000640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea5006c8f0 0x7fea5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:21.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.323+0000 7fea75000640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea5006c8f0 0x7fea5006edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fea64000f60 tx=0x7fea64009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:21.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.325+0000 7fea6cc00640 1 -- 172.21.15.40:0/2824210952 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fea5c07a220 con 0x7fea70073600 2024-04-26T23:02:21.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.532+0000 7fea76400640 1 -- 172.21.15.40:0/2824210952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fea70151aa0 con 0x7fea70073600 2024-04-26T23:02:21.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.533+0000 7fea6cc00640 1 -- 172.21.15.40:0/2824210952 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fea5c079b20 con 0x7fea70073600 2024-04-26T23:02:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:21.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.535+0000 7fea76400640 1 -- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea5006c8f0 msgr2=0x7fea5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:21.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.535+0000 7fea76400640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea5006c8f0 0x7fea5006edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fea64000f60 tx=0x7fea64009040 comp rx=0 tx=0).stop 2024-04-26T23:02:21.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.535+0000 7fea76400640 1 -- 172.21.15.40:0/2824210952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea70073600 msgr2=0x7fea701526d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:21.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.535+0000 7fea76400640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea70073600 0x7fea701526d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fea5c035040 tx=0x7fea5c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.536+0000 7fea76400640 1 -- 172.21.15.40:0/2824210952 shutdown_connections 2024-04-26T23:02:21.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.536+0000 7fea76400640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea5006c8f0 0x7fea5006edb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.536+0000 7fea76400640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea7015fcb0 0x7fea7014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.536+0000 7fea76400640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea70073600 0x7fea701526d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.536+0000 7fea76400640 1 --2- 172.21.15.40:0/2824210952 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea70072ce0 0x7fea70162120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.536+0000 7fea76400640 1 -- 172.21.15.40:0/2824210952 >> 172.21.15.40:0/2824210952 conn(0x7fea700b2720 msgr2=0x7fea700b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:21.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.536+0000 7fea76400640 1 -- 172.21.15.40:0/2824210952 shutdown_connections 2024-04-26T23:02:21.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.536+0000 7fea76400640 1 -- 172.21.15.40:0/2824210952 wait complete. 2024-04-26T23:02:21.556 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:02:21.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.769+0000 7f117a800640 1 Processor -- start 2024-04-26T23:02:21.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.769+0000 7f117a800640 1 -- start start 2024-04-26T23:02:21.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.771+0000 7f117a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f11740c6f10 0x7f11740c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:21.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.771+0000 7f117a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11740c6010 0x7f1174158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:21.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.771+0000 7f117a800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11740cce10 con 0x7f11740cc980 2024-04-26T23:02:21.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.771+0000 7f117a800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f11740c72f0 con 0x7f11740c6f10 2024-04-26T23:02:21.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.771+0000 7f117a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f11740cb860 con 0x7f11740c6010 2024-04-26T23:02:21.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.772+0000 7f1179400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f11740c6f10 0x7f11740c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:21.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.772+0000 7f1179e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11740c6010 0x7f1174158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:21.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.772+0000 7f1179400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f11740c6f10 0x7f11740c5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:39342/0 (socket says 172.21.15.40:39342) 2024-04-26T23:02:21.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.772+0000 7f1179e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11740c6010 0x7f1174158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:60244/0 (socket says 172.21.15.40:60244) 2024-04-26T23:02:21.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.772+0000 7f1179400640 1 -- 172.21.15.40:0/3128057714 learned_addr learned my addr 172.21.15.40:0/3128057714 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:21.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.772+0000 7f1178a00640 1 --1- 172.21.15.40:0/3128057714 >> v1:172.21.15.103:6789/0 conn(0x7f11740cc980 0x7f11740cbcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:45520/0 (socket says 172.21.15.40:45520) 2024-04-26T23:02:21.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.773+0000 7f1179e00640 1 -- 172.21.15.40:0/3128057714 >> v2:172.21.15.103:3300/0 conn(0x7f11740c6f10 msgr2=0x7f11740c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:21.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.773+0000 7f1179e00640 1 --2- 172.21.15.40:0/3128057714 >> v2:172.21.15.103:3300/0 conn(0x7f11740c6f10 0x7f11740c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.773+0000 7f1179e00640 1 -- 172.21.15.40:0/3128057714 >> v1:172.21.15.103:6789/0 conn(0x7f11740cc980 legacy=0x7f11740cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:21.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.773+0000 7f1179e00640 1 -- 172.21.15.40:0/3128057714 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11740cd0f0 con 0x7f11740c6010 2024-04-26T23:02:21.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.773+0000 7f1179400640 1 --2- 172.21.15.40:0/3128057714 >> v2:172.21.15.103:3300/0 conn(0x7f11740c6f10 0x7f11740c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:02:21.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.774+0000 7f1179e00640 1 --2- 172.21.15.40:0/3128057714 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11740c6010 0x7f1174158cd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f116c009d90 tx=0x7f116c02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b674db8d8a23a7bf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:21.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.774+0000 7f1173e00640 1 -- 172.21.15.40:0/3128057714 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c031600 con 0x7f11740c6010 2024-04-26T23:02:21.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.774+0000 7f1173e00640 1 -- 172.21.15.40:0/3128057714 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f116c0317a0 con 0x7f11740c6010 2024-04-26T23:02:21.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.774+0000 7f1173e00640 1 -- 172.21.15.40:0/3128057714 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c031b70 con 0x7f11740c6010 2024-04-26T23:02:21.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.775+0000 7f117a800640 1 -- 172.21.15.40:0/3128057714 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11740c6010 msgr2=0x7f1174158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:21.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.775+0000 7f117a800640 1 --2- 172.21.15.40:0/3128057714 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11740c6010 0x7f1174158cd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f116c009d90 tx=0x7f116c02a5d0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.775+0000 7f117a800640 1 -- 172.21.15.40:0/3128057714 shutdown_connections 2024-04-26T23:02:21.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.775+0000 7f117a800640 1 --2- 172.21.15.40:0/3128057714 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11740c6010 0x7f1174158cd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.775+0000 7f117a800640 1 --2- 172.21.15.40:0/3128057714 >> v2:172.21.15.103:3300/0 conn(0x7f11740c6f10 0x7f11740c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.775+0000 7f117a800640 1 -- 172.21.15.40:0/3128057714 >> 172.21.15.40:0/3128057714 conn(0x7f11740d0a10 msgr2=0x7f11740d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:21.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.775+0000 7f117a800640 1 -- 172.21.15.40:0/3128057714 shutdown_connections 2024-04-26T23:02:21.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.776+0000 7f117a800640 1 -- 172.21.15.40:0/3128057714 wait complete. 2024-04-26T23:02:21.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.777+0000 7f117a800640 1 Processor -- start 2024-04-26T23:02:21.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.777+0000 7f117a800640 1 -- start start 2024-04-26T23:02:21.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.778+0000 7f117a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11740c6010 0x7f11741532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:21.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.778+0000 7f117a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11740c6f10 0x7f11740ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.778+0000 7f117a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f117406e860 0x7f117406ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.778+0000 7f117a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f11740cc400 con 0x7f11740c6010 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.778+0000 7f117a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f11740cb1a0 con 0x7f117406e860 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.778+0000 7f117a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f11740cc580 con 0x7f11740c6f10 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.779+0000 7f1179400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11740c6010 0x7f11741532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.779+0000 7f1179400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11740c6010 0x7f11741532a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52146/0 (socket says 172.21.15.40:52146) 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.779+0000 7f1179400640 1 -- 172.21.15.40:0/3740796140 learned_addr learned my addr 172.21.15.40:0/3740796140 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.779+0000 7f1178a00640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11740c6f10 0x7f11740ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.779+0000 7f1179e00640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f117406e860 0x7f117406ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.779+0000 7f1179400640 1 -- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11740c6f10 msgr2=0x7f11740ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.779+0000 7f1179400640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11740c6f10 0x7f11740ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.779+0000 7f1179400640 1 -- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f117406e860 msgr2=0x7f117406ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.779+0000 7f1179400640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f117406e860 0x7f117406ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.779+0000 7f1179400640 1 -- 172.21.15.40:0/3740796140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11740b38d0 con 0x7f11740c6010 2024-04-26T23:02:21.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.780+0000 7f1179400640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11740c6010 0x7f11741532a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f116800e9d0 tx=0x7f116800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:21.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.780+0000 7f1171600640 1 -- 172.21.15.40:0/3740796140 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1168017070 con 0x7f11740c6010 2024-04-26T23:02:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.780+0000 7f1171600640 1 -- 172.21.15.40:0/3740796140 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f116800ca60 con 0x7f11740c6010 2024-04-26T23:02:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.780+0000 7f1171600640 1 -- 172.21.15.40:0/3740796140 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116800ce10 con 0x7f11740c6010 2024-04-26T23:02:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.780+0000 7f117a800640 1 -- 172.21.15.40:0/3740796140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11740b3bb0 con 0x7f11740c6010 2024-04-26T23:02:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.780+0000 7f117a800640 1 -- 172.21.15.40:0/3740796140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11740ca770 con 0x7f11740c6010 2024-04-26T23:02:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.781+0000 7f117a800640 1 -- 172.21.15.40:0/3740796140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1174069370 con 0x7f11740c6010 2024-04-26T23:02:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.782+0000 7f1171600640 1 -- 172.21.15.40:0/3740796140 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f116801a070 con 0x7f11740c6010 2024-04-26T23:02:21.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.783+0000 7f1171600640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f115406c920 0x7f115406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:21.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.783+0000 7f1171600640 1 -- 172.21.15.40:0/3740796140 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f116808bef0 con 0x7f11740c6010 2024-04-26T23:02:21.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.784+0000 7f1178a00640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f115406c920 0x7f115406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.785+0000 7f1178a00640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f115406c920 0x7f115406ede0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f11741537e0 tx=0x7f1160007cc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:21.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.788+0000 7f1171600640 1 -- 172.21.15.40:0/3740796140 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1168059e80 con 0x7f11740c6010 2024-04-26T23:02:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.996+0000 7f117a800640 1 -- 172.21.15.40:0/3740796140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1174070020 con 0x7f11740c6010 2024-04-26T23:02:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:21.997+0000 7f1171600640 1 -- 172.21.15.40:0/3740796140 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f1168059780 con 0x7f11740c6010 2024-04-26T23:02:21.997 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:02:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.000+0000 7f117a800640 1 -- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f115406c920 msgr2=0x7f115406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.000+0000 7f117a800640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f115406c920 0x7f115406ede0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f11741537e0 tx=0x7f1160007cc0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.000+0000 7f117a800640 1 -- 172.21.15.40:0/3740796140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11740c6010 msgr2=0x7f11741532a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.000+0000 7f117a800640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11740c6010 0x7f11741532a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f116800e9d0 tx=0x7f116800ee90 comp rx=0 tx=0).stop 2024-04-26T23:02:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.000+0000 7f117a800640 1 -- 172.21.15.40:0/3740796140 shutdown_connections 2024-04-26T23:02:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.000+0000 7f117a800640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f115406c920 0x7f115406ede0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.000+0000 7f117a800640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f117406e860 0x7f117406ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.000+0000 7f117a800640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f11740c6f10 0x7f11740ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.000+0000 7f117a800640 1 --2- 172.21.15.40:0/3740796140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11740c6010 0x7f11741532a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.000+0000 7f117a800640 1 -- 172.21.15.40:0/3740796140 >> 172.21.15.40:0/3740796140 conn(0x7f11740d0a10 msgr2=0x7f1174153da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.000+0000 7f117a800640 1 -- 172.21.15.40:0/3740796140 shutdown_connections 2024-04-26T23:02:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.000+0000 7f117a800640 1 -- 172.21.15.40:0/3740796140 wait complete. 2024-04-26T23:02:22.019 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:02:22.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.239+0000 7ff596000640 1 Processor -- start 2024-04-26T23:02:22.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.239+0000 7ff596000640 1 -- start start 2024-04-26T23:02:22.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.240+0000 7ff596000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff5900b5500 0x7ff5900b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.240+0000 7ff596000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5900d3dc0 con 0x7ff5900b5120 2024-04-26T23:02:22.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.240+0000 7ff596000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5900b75b0 con 0x7ff5900b59b0 2024-04-26T23:02:22.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.240+0000 7ff596000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff5900d03b0 con 0x7ff5900b5500 2024-04-26T23:02:22.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.241+0000 7ff594c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff5900b5120 0x7ff5900b6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52454/0 (socket says 172.21.15.40:52454) 2024-04-26T23:02:22.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.241+0000 7ff594c00640 1 -- 172.21.15.40:0/4228098799 learned_addr learned my addr 172.21.15.40:0/4228098799 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.241+0000 7ff58fe00640 1 --2- 172.21.15.40:0/4228098799 >> v2:172.21.15.103:3300/0 conn(0x7ff5900b5500 0x7ff5900b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.241+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 341685025 0 0) 0x7ff5900d3dc0 con 0x7ff5900b5120 2024-04-26T23:02:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.241+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff574003520 con 0x7ff5900b5120 2024-04-26T23:02:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2631960078 0 0) 0x7ff574003520 con 0x7ff5900b5120 2024-04-26T23:02:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 >> v2:172.21.15.103:3300/0 conn(0x7ff5900b5500 msgr2=0x7ff5900b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff58f400640 1 --2- 172.21.15.40:0/4228098799 >> v2:172.21.15.103:3300/0 conn(0x7ff5900b5500 0x7ff5900b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 >> v1:172.21.15.103:6789/0 conn(0x7ff5900b59b0 legacy=0x7ff5900b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5900d0600 con 0x7ff5900b5120 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7ff578001670 con 0x7ff5900b5120 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 >> v1:172.21.15.40:6789/0 conn(0x7ff5900b5120 legacy=0x7ff5900b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff58f400640 1 --2- 172.21.15.40:0/4228098799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5740039e0 0x7ff574003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff58f400640 1 --2- 172.21.15.40:0/4228098799 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff574004800 0x7ff574004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff58f400640 1 --2- 172.21.15.40:0/4228098799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5740052e0 0x7ff57400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff5900d03b0 con 0x7ff5740039e0 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff57400c1e0 con 0x7ff5740052e0 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff57400c3e0 con 0x7ff574004800 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.242+0000 7ff594c00640 1 --2- 172.21.15.40:0/4228098799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5740039e0 0x7ff574003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.243+0000 7ff58fe00640 1 --2- 172.21.15.40:0/4228098799 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff574004800 0x7ff574004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.243+0000 7ff595600640 1 --2- 172.21.15.40:0/4228098799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5740052e0 0x7ff57400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.243+0000 7ff594c00640 1 -- 172.21.15.40:0/4228098799 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff574004800 msgr2=0x7ff574004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.243+0000 7ff594c00640 1 --2- 172.21.15.40:0/4228098799 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff574004800 0x7ff574004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.243+0000 7ff594c00640 1 -- 172.21.15.40:0/4228098799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5740052e0 msgr2=0x7ff57400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:22.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.243+0000 7ff594c00640 1 --2- 172.21.15.40:0/4228098799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5740052e0 0x7ff57400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.243+0000 7ff594c00640 1 -- 172.21.15.40:0/4228098799 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff574003520 con 0x7ff5740039e0 2024-04-26T23:02:22.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.243+0000 7ff595600640 1 --2- 172.21.15.40:0/4228098799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5740052e0 0x7ff57400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:22.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.243+0000 7ff595600640 1 -- 172.21.15.40:0/4228098799 reap_dead start 2024-04-26T23:02:22.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.244+0000 7ff594c00640 1 --2- 172.21.15.40:0/4228098799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5740039e0 0x7ff574003e00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c210 tx=0x7ff574005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827cc6f891e4feb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:22.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.244+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff578007d10 con 0x7ff5740039e0 2024-04-26T23:02:22.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.244+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5780058d0 con 0x7ff5740039e0 2024-04-26T23:02:22.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.244+0000 7ff58f400640 1 -- 172.21.15.40:0/4228098799 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff578005d60 con 0x7ff5740039e0 2024-04-26T23:02:22.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.245+0000 7ff596000640 1 -- 172.21.15.40:0/4228098799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5740039e0 msgr2=0x7ff574003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.245+0000 7ff596000640 1 --2- 172.21.15.40:0/4228098799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5740039e0 0x7ff574003e00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c210 tx=0x7ff574005830 comp rx=0 tx=0).stop 2024-04-26T23:02:22.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.245+0000 7ff596000640 1 -- 172.21.15.40:0/4228098799 shutdown_connections 2024-04-26T23:02:22.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.245+0000 7ff596000640 1 --2- 172.21.15.40:0/4228098799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5740039e0 0x7ff574003e00 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.245+0000 7ff596000640 1 -- 172.21.15.40:0/4228098799 >> 172.21.15.40:0/4228098799 conn(0x7ff5900d4db0 msgr2=0x7ff5900d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:22.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.245+0000 7ff596000640 1 -- 172.21.15.40:0/4228098799 shutdown_connections 2024-04-26T23:02:22.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.245+0000 7ff596000640 1 -- 172.21.15.40:0/4228098799 wait complete. 2024-04-26T23:02:22.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.247+0000 7ff596000640 1 Processor -- start 2024-04-26T23:02:22.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.247+0000 7ff596000640 1 -- start start 2024-04-26T23:02:22.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.248+0000 7ff596000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5740039e0 0x7ff5901513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.248+0000 7ff596000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5901f58e0 0x7ff5900cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.248+0000 7ff596000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5900c9f00 0x7ff5900ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.248+0000 7ff596000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff5900b5ee0 con 0x7ff5900c9f00 2024-04-26T23:02:22.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.248+0000 7ff596000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff5900d0bc0 con 0x7ff5740039e0 2024-04-26T23:02:22.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.248+0000 7ff596000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff5900b6060 con 0x7ff5901f58e0 2024-04-26T23:02:22.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.249+0000 7ff595600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5900c9f00 0x7ff5900ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.249+0000 7ff594c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5740039e0 0x7ff5901513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.249+0000 7ff595600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5900c9f00 0x7ff5900ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52162/0 (socket says 172.21.15.40:52162) 2024-04-26T23:02:22.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.249+0000 7ff595600640 1 -- 172.21.15.40:0/2102750473 learned_addr learned my addr 172.21.15.40:0/2102750473 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:22.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.249+0000 7ff58fe00640 1 --2- 172.21.15.40:0/2102750473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5901f58e0 0x7ff5900cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.249+0000 7ff595600640 1 -- 172.21.15.40:0/2102750473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5901f58e0 msgr2=0x7ff5900cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.249+0000 7ff595600640 1 --2- 172.21.15.40:0/2102750473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5901f58e0 0x7ff5900cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.249+0000 7ff595600640 1 -- 172.21.15.40:0/2102750473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5740039e0 msgr2=0x7ff5901513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.249+0000 7ff595600640 1 --2- 172.21.15.40:0/2102750473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5740039e0 0x7ff5901513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.249+0000 7ff595600640 1 -- 172.21.15.40:0/2102750473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5900c8840 con 0x7ff5900c9f00 2024-04-26T23:02:22.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.250+0000 7ff595600640 1 --2- 172.21.15.40:0/2102750473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5900c9f00 0x7ff5900ca3b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff5900d4170 tx=0x7ff58800d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:22.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.250+0000 7ff58cc00640 1 -- 172.21.15.40:0/2102750473 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff588018070 con 0x7ff5900c9f00 2024-04-26T23:02:22.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.250+0000 7ff58cc00640 1 -- 172.21.15.40:0/2102750473 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff588003da0 con 0x7ff5900c9f00 2024-04-26T23:02:22.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.250+0000 7ff58cc00640 1 -- 172.21.15.40:0/2102750473 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5880052b0 con 0x7ff5900c9f00 2024-04-26T23:02:22.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.250+0000 7ff596000640 1 -- 172.21.15.40:0/2102750473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5900c8b20 con 0x7ff5900c9f00 2024-04-26T23:02:22.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.250+0000 7ff596000640 1 -- 172.21.15.40:0/2102750473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5900c90e0 con 0x7ff5900c9f00 2024-04-26T23:02:22.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.251+0000 7ff596000640 1 -- 172.21.15.40:0/2102750473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff590069370 con 0x7ff5900c9f00 2024-04-26T23:02:22.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.252+0000 7ff58cc00640 1 -- 172.21.15.40:0/2102750473 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff58801b070 con 0x7ff5900c9f00 2024-04-26T23:02:22.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.253+0000 7ff58cc00640 1 --2- 172.21.15.40:0/2102750473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff56c06c920 0x7ff56c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.253+0000 7ff58cc00640 1 -- 172.21.15.40:0/2102750473 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ff58808b580 con 0x7ff5900c9f00 2024-04-26T23:02:22.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.254+0000 7ff594c00640 1 --2- 172.21.15.40:0/2102750473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff56c06c920 0x7ff56c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.255+0000 7ff594c00640 1 --2- 172.21.15.40:0/2102750473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff56c06c920 0x7ff56c06ede0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff578033000 tx=0x7ff57802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:22.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.258+0000 7ff58cc00640 1 -- 172.21.15.40:0/2102750473 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff588058ec0 con 0x7ff5900c9f00 2024-04-26T23:02:22.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.466+0000 7ff596000640 1 -- 172.21.15.40:0/2102750473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff59008e5a0 con 0x7ff5900c9f00 2024-04-26T23:02:22.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.467+0000 7ff58cc00640 1 -- 172.21.15.40:0/2102750473 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7ff58805d1c0 con 0x7ff5900c9f00 2024-04-26T23:02:22.468 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.470+0000 7ff596000640 1 -- 172.21.15.40:0/2102750473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff56c06c920 msgr2=0x7ff56c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.470+0000 7ff596000640 1 --2- 172.21.15.40:0/2102750473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff56c06c920 0x7ff56c06ede0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff578033000 tx=0x7ff57802e000 comp rx=0 tx=0).stop 2024-04-26T23:02:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.470+0000 7ff596000640 1 -- 172.21.15.40:0/2102750473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5900c9f00 msgr2=0x7ff5900ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.470+0000 7ff596000640 1 --2- 172.21.15.40:0/2102750473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5900c9f00 0x7ff5900ca3b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff5900d4170 tx=0x7ff58800d9f0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.470+0000 7ff596000640 1 -- 172.21.15.40:0/2102750473 shutdown_connections 2024-04-26T23:02:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.470+0000 7ff596000640 1 --2- 172.21.15.40:0/2102750473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff56c06c920 0x7ff56c06ede0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.470+0000 7ff596000640 1 --2- 172.21.15.40:0/2102750473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5900c9f00 0x7ff5900ca3b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.470+0000 7ff596000640 1 --2- 172.21.15.40:0/2102750473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5901f58e0 0x7ff5900cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.470+0000 7ff596000640 1 --2- 172.21.15.40:0/2102750473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5740039e0 0x7ff5901513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.470+0000 7ff596000640 1 -- 172.21.15.40:0/2102750473 >> 172.21.15.40:0/2102750473 conn(0x7ff5900d4db0 msgr2=0x7ff5900d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.470+0000 7ff596000640 1 -- 172.21.15.40:0/2102750473 shutdown_connections 2024-04-26T23:02:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.470+0000 7ff596000640 1 -- 172.21.15.40:0/2102750473 wait complete. 2024-04-26T23:02:22.489 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:02:22.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.706+0000 7f47b4a00640 1 Processor -- start 2024-04-26T23:02:22.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.706+0000 7f47b4a00640 1 -- start start 2024-04-26T23:02:22.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.708+0000 7f47b4a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f47b015ce60 0x7f47b00afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.708+0000 7f47b4a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f47b00af7a0 0x7f47b00ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.708+0000 7f47b4a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b00d0820 con 0x7f47b0071fe0 2024-04-26T23:02:22.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.708+0000 7f47b4a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f47b00cc450 con 0x7f47b00af7a0 2024-04-26T23:02:22.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.708+0000 7f47b4a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f47b01614c0 con 0x7f47b015ce60 2024-04-26T23:02:22.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.708+0000 7f47a7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f47b00af7a0 0x7f47b00ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.708+0000 7f47afe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f47b0071fe0 0x7f47b0073760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52464/0 (socket says 172.21.15.40:52464) 2024-04-26T23:02:22.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.708+0000 7f47afe00640 1 -- 172.21.15.40:0/3475551279 learned_addr learned my addr 172.21.15.40:0/3475551279 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:22.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.708+0000 7f47af400640 1 --2- 172.21.15.40:0/3475551279 >> v2:172.21.15.103:3300/0 conn(0x7f47b015ce60 0x7f47b00afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.709+0000 7f47a7e00640 1 -- 172.21.15.40:0/3475551279 >> v2:172.21.15.103:3300/0 conn(0x7f47b015ce60 msgr2=0x7f47b00afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:22.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.709+0000 7f47a7e00640 1 --2- 172.21.15.40:0/3475551279 >> v2:172.21.15.103:3300/0 conn(0x7f47b015ce60 0x7f47b00afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.709+0000 7f47aea00640 1 -- 172.21.15.40:0/3475551279 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 299787617 0 0) 0x7f47b00d0820 con 0x7f47b0071fe0 2024-04-26T23:02:22.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.709+0000 7f47a7e00640 1 -- 172.21.15.40:0/3475551279 >> v1:172.21.15.40:6789/0 conn(0x7f47b0071fe0 legacy=0x7f47b0073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.709+0000 7f47a7e00640 1 -- 172.21.15.40:0/3475551279 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b0161760 con 0x7f47b00af7a0 2024-04-26T23:02:22.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.709+0000 7f47af400640 1 --2- 172.21.15.40:0/3475551279 >> v2:172.21.15.103:3300/0 conn(0x7f47b015ce60 0x7f47b00afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:02:22.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.710+0000 7f47a7e00640 1 --2- 172.21.15.40:0/3475551279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47b00af7a0 0x7f47b00ceff0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f47b00741b0 tx=0x7f479802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae151c419f926f30 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:22.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.711+0000 7f47aea00640 1 -- 172.21.15.40:0/3475551279 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4798037070 con 0x7f47b00af7a0 2024-04-26T23:02:22.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.711+0000 7f47aea00640 1 -- 172.21.15.40:0/3475551279 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4798032440 con 0x7f47b00af7a0 2024-04-26T23:02:22.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.711+0000 7f47aea00640 1 -- 172.21.15.40:0/3475551279 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479802ae60 con 0x7f47b00af7a0 2024-04-26T23:02:22.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.711+0000 7f47b4a00640 1 -- 172.21.15.40:0/3475551279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47b00af7a0 msgr2=0x7f47b00ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.711+0000 7f47b4a00640 1 --2- 172.21.15.40:0/3475551279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47b00af7a0 0x7f47b00ceff0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f47b00741b0 tx=0x7f479802a110 comp rx=0 tx=0).stop 2024-04-26T23:02:22.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.711+0000 7f47b4a00640 1 -- 172.21.15.40:0/3475551279 shutdown_connections 2024-04-26T23:02:22.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.711+0000 7f47b4a00640 1 --2- 172.21.15.40:0/3475551279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47b00af7a0 0x7f47b00ceff0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.711+0000 7f47b4a00640 1 --2- 172.21.15.40:0/3475551279 >> v2:172.21.15.103:3300/0 conn(0x7f47b015ce60 0x7f47b00afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.711+0000 7f47b4a00640 1 -- 172.21.15.40:0/3475551279 >> 172.21.15.40:0/3475551279 conn(0x7f47b00d0a10 msgr2=0x7f47b00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:22.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.712+0000 7f47b4a00640 1 -- 172.21.15.40:0/3475551279 shutdown_connections 2024-04-26T23:02:22.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.712+0000 7f47b4a00640 1 -- 172.21.15.40:0/3475551279 wait complete. 2024-04-26T23:02:22.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.713+0000 7f47b4a00640 1 Processor -- start 2024-04-26T23:02:22.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.713+0000 7f47b4a00640 1 -- start start 2024-04-26T23:02:22.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.714+0000 7f47b4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47b0071fe0 0x7f47b0157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.714+0000 7f47b4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47b00af7a0 0x7f47b00ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.714+0000 7f47b4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47b015ce60 0x7f47b00aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.714+0000 7f47b4a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f47b0073e70 con 0x7f47b0071fe0 2024-04-26T23:02:22.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.714+0000 7f47b4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f47b00cf530 con 0x7f47b00af7a0 2024-04-26T23:02:22.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.714+0000 7f47b4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f47b0073ff0 con 0x7f47b015ce60 2024-04-26T23:02:22.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.715+0000 7f47af400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47b0071fe0 0x7f47b0157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.715+0000 7f47af400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47b0071fe0 0x7f47b0157470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52172/0 (socket says 172.21.15.40:52172) 2024-04-26T23:02:22.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.715+0000 7f47af400640 1 -- 172.21.15.40:0/1947533503 learned_addr learned my addr 172.21.15.40:0/1947533503 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:22.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.715+0000 7f47afe00640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47b015ce60 0x7f47b00aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.715+0000 7f47af400640 1 -- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47b015ce60 msgr2=0x7f47b00aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.715+0000 7f47a7e00640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47b00af7a0 0x7f47b00ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.715+0000 7f47af400640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47b015ce60 0x7f47b00aea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.715+0000 7f47af400640 1 -- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47b00af7a0 msgr2=0x7f47b00ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.715+0000 7f47af400640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47b00af7a0 0x7f47b00ca3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.715+0000 7f47af400640 1 -- 172.21.15.40:0/1947533503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b00c7f50 con 0x7f47b0071fe0 2024-04-26T23:02:22.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.715+0000 7f47af400640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47b0071fe0 0x7f47b0157470 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f47a000d7d0 tx=0x7f47a000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:22.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.716+0000 7f47a7400640 1 -- 172.21.15.40:0/1947533503 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a0004030 con 0x7f47b0071fe0 2024-04-26T23:02:22.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.716+0000 7f47a7400640 1 -- 172.21.15.40:0/1947533503 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47a00041d0 con 0x7f47b0071fe0 2024-04-26T23:02:22.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.716+0000 7f47a7400640 1 -- 172.21.15.40:0/1947533503 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a000b660 con 0x7f47b0071fe0 2024-04-26T23:02:22.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.716+0000 7f47b4a00640 1 -- 172.21.15.40:0/1947533503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b00c8230 con 0x7f47b0071fe0 2024-04-26T23:02:22.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.716+0000 7f47b4a00640 1 -- 172.21.15.40:0/1947533503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b00c87f0 con 0x7f47b0071fe0 2024-04-26T23:02:22.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.717+0000 7f47b4a00640 1 -- 172.21.15.40:0/1947533503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47b0069370 con 0x7f47b0071fe0 2024-04-26T23:02:22.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.718+0000 7f47a7400640 1 -- 172.21.15.40:0/1947533503 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f47a000b800 con 0x7f47b0071fe0 2024-04-26T23:02:22.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.719+0000 7f47a7400640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f478c06c8f0 0x7f478c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:22.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.719+0000 7f47a7400640 1 -- 172.21.15.40:0/1947533503 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f47a008c210 con 0x7f47b0071fe0 2024-04-26T23:02:22.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.720+0000 7f47a7e00640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f478c06c8f0 0x7f478c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:22.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.721+0000 7f47a7e00640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f478c06c8f0 0x7f478c06edb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4798034000 tx=0x7f479802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:22.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.723+0000 7f47a7400640 1 -- 172.21.15.40:0/1947533503 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f47a005a1a0 con 0x7f47b0071fe0 2024-04-26T23:02:22.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.943+0000 7f47b4a00640 1 -- 172.21.15.40:0/1947533503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f47b00d01f0 con 0x7f47b0071fe0 2024-04-26T23:02:22.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.944+0000 7f47a7400640 1 -- 172.21.15.40:0/1947533503 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f47a0059aa0 con 0x7f47b0071fe0 2024-04-26T23:02:22.944 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-04-26T23:02:22.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.947+0000 7f47b4a00640 1 -- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f478c06c8f0 msgr2=0x7f478c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.947+0000 7f47b4a00640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f478c06c8f0 0x7f478c06edb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4798034000 tx=0x7f479802e000 comp rx=0 tx=0).stop 2024-04-26T23:02:22.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.947+0000 7f47b4a00640 1 -- 172.21.15.40:0/1947533503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47b0071fe0 msgr2=0x7f47b0157470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:22.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.947+0000 7f47b4a00640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47b0071fe0 0x7f47b0157470 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f47a000d7d0 tx=0x7f47a000dc90 comp rx=0 tx=0).stop 2024-04-26T23:02:22.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.947+0000 7f47b4a00640 1 -- 172.21.15.40:0/1947533503 shutdown_connections 2024-04-26T23:02:22.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.947+0000 7f47b4a00640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f478c06c8f0 0x7f478c06edb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.947+0000 7f47b4a00640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47b015ce60 0x7f47b00aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.947+0000 7f47b4a00640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47b00af7a0 0x7f47b00ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.947+0000 7f47b4a00640 1 --2- 172.21.15.40:0/1947533503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47b0071fe0 0x7f47b0157470 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:22.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.947+0000 7f47b4a00640 1 -- 172.21.15.40:0/1947533503 >> 172.21.15.40:0/1947533503 conn(0x7f47b00d0a10 msgr2=0x7f47b00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:22.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.947+0000 7f47b4a00640 1 -- 172.21.15.40:0/1947533503 shutdown_connections 2024-04-26T23:02:22.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:22.947+0000 7f47b4a00640 1 -- 172.21.15.40:0/1947533503 wait complete. 2024-04-26T23:02:22.968 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-26T23:02:22.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:22.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-04-26T23:02:23.067 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T23:02:23.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:23.067 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-26T23:02:23.129 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:02:23.129 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:23.190 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:02:23.190 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:23.239 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:02:23.239 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:23.292 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.505+0000 7f74ada00640 1 Processor -- start 2024-04-26T23:02:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.505+0000 7f74ada00640 1 -- start start 2024-04-26T23:02:23.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.507+0000 7f74ada00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f74a815ce60 0x7f74a80afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.507+0000 7f74ada00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74a80af7a0 0x7f74a80ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.507+0000 7f74ada00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a80d0820 con 0x7f74a8071fe0 2024-04-26T23:02:23.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.507+0000 7f74ada00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f74a80cc450 con 0x7f74a815ce60 2024-04-26T23:02:23.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.507+0000 7f74ada00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f74a81614c0 con 0x7f74a80af7a0 2024-04-26T23:02:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.508+0000 7f74a7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f74a815ce60 0x7f74a80afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.508+0000 7f74a7400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74a80af7a0 0x7f74a80ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.508+0000 7f74a7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f74a815ce60 0x7f74a80afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:39410/0 (socket says 172.21.15.40:39410) 2024-04-26T23:02:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.508+0000 7f74a7e00640 1 -- 172.21.15.40:0/2438344002 learned_addr learned my addr 172.21.15.40:0/2438344002 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:23.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.509+0000 7f74a7400640 1 -- 172.21.15.40:0/2438344002 >> v2:172.21.15.103:3300/0 conn(0x7f74a815ce60 msgr2=0x7f74a80afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:23.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.509+0000 7f74a7400640 1 --2- 172.21.15.40:0/2438344002 >> v2:172.21.15.103:3300/0 conn(0x7f74a815ce60 0x7f74a80afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.509+0000 7f74a7400640 1 -- 172.21.15.40:0/2438344002 >> v1:172.21.15.103:6789/0 conn(0x7f74a8071fe0 legacy=0x7f74a8073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:23.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.509+0000 7f74a7400640 1 -- 172.21.15.40:0/2438344002 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a8161760 con 0x7f74a80af7a0 2024-04-26T23:02:23.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.509+0000 7f74a7e00640 1 --2- 172.21.15.40:0/2438344002 >> v2:172.21.15.103:3300/0 conn(0x7f74a815ce60 0x7f74a80afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:02:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.510+0000 7f74a7400640 1 --2- 172.21.15.40:0/2438344002 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74a80af7a0 0x7f74a80ceff0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f74940097a0 tx=0x7f749402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4344b05498178186 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.511+0000 7f74a6a00640 1 -- 172.21.15.40:0/2438344002 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494037070 con 0x7f74a80af7a0 2024-04-26T23:02:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.511+0000 7f74a6a00640 1 -- 172.21.15.40:0/2438344002 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f749402ae50 con 0x7f74a80af7a0 2024-04-26T23:02:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.511+0000 7f74a6a00640 1 -- 172.21.15.40:0/2438344002 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494032660 con 0x7f74a80af7a0 2024-04-26T23:02:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.511+0000 7f74ada00640 1 -- 172.21.15.40:0/2438344002 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74a80af7a0 msgr2=0x7f74a80ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.511+0000 7f74ada00640 1 --2- 172.21.15.40:0/2438344002 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74a80af7a0 0x7f74a80ceff0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f74940097a0 tx=0x7f749402a100 comp rx=0 tx=0).stop 2024-04-26T23:02:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.511+0000 7f74ada00640 1 -- 172.21.15.40:0/2438344002 shutdown_connections 2024-04-26T23:02:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.511+0000 7f74ada00640 1 --2- 172.21.15.40:0/2438344002 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74a80af7a0 0x7f74a80ceff0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.512+0000 7f74ada00640 1 --2- 172.21.15.40:0/2438344002 >> v2:172.21.15.103:3300/0 conn(0x7f74a815ce60 0x7f74a80afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.512+0000 7f74ada00640 1 -- 172.21.15.40:0/2438344002 >> 172.21.15.40:0/2438344002 conn(0x7f74a80d0a10 msgr2=0x7f74a80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.512+0000 7f74ada00640 1 -- 172.21.15.40:0/2438344002 shutdown_connections 2024-04-26T23:02:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.512+0000 7f74ada00640 1 -- 172.21.15.40:0/2438344002 wait complete. 2024-04-26T23:02:23.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.514+0000 7f74ada00640 1 Processor -- start 2024-04-26T23:02:23.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.514+0000 7f74ada00640 1 -- start start 2024-04-26T23:02:23.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.515+0000 7f74ada00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74a8071fe0 0x7f74a81f95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.515+0000 7f74ada00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a80af7a0 0x7f74a80ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.515+0000 7f74ada00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74a815ce60 0x7f74a815bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.516+0000 7f74ada00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f74a8073e70 con 0x7f74a8071fe0 2024-04-26T23:02:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.516+0000 7f74ada00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f74a80cf530 con 0x7f74a80af7a0 2024-04-26T23:02:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.516+0000 7f74ada00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f74a8073ff0 con 0x7f74a815ce60 2024-04-26T23:02:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.516+0000 7f74a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74a8071fe0 0x7f74a81f95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.516+0000 7f74a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74a8071fe0 0x7f74a81f95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52180/0 (socket says 172.21.15.40:52180) 2024-04-26T23:02:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.516+0000 7f74a7e00640 1 -- 172.21.15.40:0/1497834467 learned_addr learned my addr 172.21.15.40:0/1497834467 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.516+0000 7f74a7400640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a80af7a0 0x7f74a80ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.516+0000 7f74a7e00640 1 -- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74a815ce60 msgr2=0x7f74a815bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.516+0000 7f74a7e00640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74a815ce60 0x7f74a815bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.516+0000 7f74a7e00640 1 -- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a80af7a0 msgr2=0x7f74a80ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.516+0000 7f74a7e00640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a80af7a0 0x7f74a80ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.516+0000 7f74a7e00640 1 -- 172.21.15.40:0/1497834467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a815c1a0 con 0x7f74a8071fe0 2024-04-26T23:02:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.516+0000 7f74a7400640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a80af7a0 0x7f74a80ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:23.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.517+0000 7f74a7e00640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74a8071fe0 0x7f74a81f95e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f749800e9b0 tx=0x7f749800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:23.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.517+0000 7f749fe00640 1 -- 172.21.15.40:0/1497834467 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498017070 con 0x7f74a8071fe0 2024-04-26T23:02:23.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.517+0000 7f749fe00640 1 -- 172.21.15.40:0/1497834467 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f749800c9a0 con 0x7f74a8071fe0 2024-04-26T23:02:23.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.517+0000 7f74ada00640 1 -- 172.21.15.40:0/1497834467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a80c7d30 con 0x7f74a8071fe0 2024-04-26T23:02:23.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.517+0000 7f749fe00640 1 -- 172.21.15.40:0/1497834467 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749800cd50 con 0x7f74a8071fe0 2024-04-26T23:02:23.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.517+0000 7f74ada00640 1 -- 172.21.15.40:0/1497834467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a81f8b00 con 0x7f74a8071fe0 2024-04-26T23:02:23.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.518+0000 7f74ada00640 1 -- 172.21.15.40:0/1497834467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74a8069370 con 0x7f74a8071fe0 2024-04-26T23:02:23.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.519+0000 7f749fe00640 1 -- 172.21.15.40:0/1497834467 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f749801a070 con 0x7f74a8071fe0 2024-04-26T23:02:23.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.521+0000 7f749fe00640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f748806c8f0 0x7f748806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.521+0000 7f749fe00640 1 -- 172.21.15.40:0/1497834467 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f749808bf20 con 0x7f74a8071fe0 2024-04-26T23:02:23.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.522+0000 7f74a7400640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f748806c8f0 0x7f748806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:23.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.523+0000 7f74a7400640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f748806c8f0 0x7f748806edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7494034000 tx=0x7f749402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:23.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.526+0000 7f749fe00640 1 -- 172.21.15.40:0/1497834467 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7498059e30 con 0x7f74a8071fe0 2024-04-26T23:02:23.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.732+0000 7f74ada00640 1 -- 172.21.15.40:0/1497834467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74a80d01f0 con 0x7f74a8071fe0 2024-04-26T23:02:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.735+0000 7f749fe00640 1 -- 172.21.15.40:0/1497834467 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18812 (secure 0 0 0) 0x7f7498059730 con 0x7f74a8071fe0 2024-04-26T23:02:23.735 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:23.736 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:46.975683+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:23.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.738+0000 7f74ada00640 1 -- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f748806c8f0 msgr2=0x7f748806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:23.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.738+0000 7f74ada00640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f748806c8f0 0x7f748806edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7494034000 tx=0x7f749402e000 comp rx=0 tx=0).stop 2024-04-26T23:02:23.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.738+0000 7f74ada00640 1 -- 172.21.15.40:0/1497834467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74a8071fe0 msgr2=0x7f74a81f95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:23.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.738+0000 7f74ada00640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74a8071fe0 0x7f74a81f95e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f749800e9b0 tx=0x7f749800ee70 comp rx=0 tx=0).stop 2024-04-26T23:02:23.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.738+0000 7f74ada00640 1 -- 172.21.15.40:0/1497834467 shutdown_connections 2024-04-26T23:02:23.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.738+0000 7f74ada00640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f748806c8f0 0x7f748806edb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.738+0000 7f74ada00640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74a815ce60 0x7f74a815bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.738+0000 7f74ada00640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a80af7a0 0x7f74a80ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.738+0000 7f74ada00640 1 --2- 172.21.15.40:0/1497834467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74a8071fe0 0x7f74a81f95e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.738+0000 7f74ada00640 1 -- 172.21.15.40:0/1497834467 >> 172.21.15.40:0/1497834467 conn(0x7f74a80d0a10 msgr2=0x7f74a80cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:23.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.738+0000 7f74ada00640 1 -- 172.21.15.40:0/1497834467 shutdown_connections 2024-04-26T23:02:23.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.738+0000 7f74ada00640 1 -- 172.21.15.40:0/1497834467 wait complete. 2024-04-26T23:02:23.759 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:02:23.760 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:02:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.977+0000 7fde2d600640 1 Processor -- start 2024-04-26T23:02:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.977+0000 7fde2d600640 1 -- start start 2024-04-26T23:02:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.978+0000 7fde2d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde280cc480 0x7fde28072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.978+0000 7fde2d600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fde28072780 0x7fde28166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.978+0000 7fde2d600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde2815d7c0 con 0x7fde28071e60 2024-04-26T23:02:23.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.978+0000 7fde2d600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fde2815daa0 con 0x7fde28072780 2024-04-26T23:02:23.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.978+0000 7fde2d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fde2815dd80 con 0x7fde280cc480 2024-04-26T23:02:23.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.978+0000 7fde26a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fde28071e60 0x7fde2806f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52470/0 (socket says 172.21.15.40:52470) 2024-04-26T23:02:23.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde26a00640 1 -- 172.21.15.40:0/2767664836 learned_addr learned my addr 172.21.15.40:0/2767664836 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1141521169 0 0) 0x7fde2815d7c0 con 0x7fde28071e60 2024-04-26T23:02:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde08003520 con 0x7fde28071e60 2024-04-26T23:02:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde27400640 1 --2- 172.21.15.40:0/2767664836 >> v2:172.21.15.103:3300/0 conn(0x7fde28072780 0x7fde28166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde26000640 1 --2- 172.21.15.40:0/2767664836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde280cc480 0x7fde28072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2700520335 0 0) 0x7fde08003520 con 0x7fde28071e60 2024-04-26T23:02:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde280cc480 msgr2=0x7fde28072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde25600640 1 --2- 172.21.15.40:0/2767664836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde280cc480 0x7fde28072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 >> v2:172.21.15.103:3300/0 conn(0x7fde28072780 msgr2=0x7fde28166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde25600640 1 --2- 172.21.15.40:0/2767664836 >> v2:172.21.15.103:3300/0 conn(0x7fde28072780 0x7fde28166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2815e060 con 0x7fde28071e60 2024-04-26T23:02:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fde18001670 con 0x7fde28071e60 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 >> v1:172.21.15.40:6789/0 conn(0x7fde28071e60 legacy=0x7fde2806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.979+0000 7fde25600640 1 --2- 172.21.15.40:0/2767664836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde080039b0 0x7fde08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde25600640 1 --2- 172.21.15.40:0/2767664836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde08004800 0x7fde08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde25600640 1 --2- 172.21.15.40:0/2767664836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde280cc480 0x7fde080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde2815daa0 con 0x7fde08004800 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde2815dd80 con 0x7fde080039b0 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fde0800ba70 con 0x7fde280cc480 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde26000640 1 --2- 172.21.15.40:0/2767664836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde08004800 0x7fde08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde26a00640 1 --2- 172.21.15.40:0/2767664836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde080039b0 0x7fde08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde27400640 1 --2- 172.21.15.40:0/2767664836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde280cc480 0x7fde080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde26000640 1 -- 172.21.15.40:0/2767664836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde280cc480 msgr2=0x7fde080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde26000640 1 --2- 172.21.15.40:0/2767664836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde280cc480 0x7fde080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde26000640 1 -- 172.21.15.40:0/2767664836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde080039b0 msgr2=0x7fde08003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde26000640 1 --2- 172.21.15.40:0/2767664836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde080039b0 0x7fde08003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde26000640 1 -- 172.21.15.40:0/2767664836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde08003520 con 0x7fde08004800 2024-04-26T23:02:23.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.980+0000 7fde27400640 1 --2- 172.21.15.40:0/2767664836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde280cc480 0x7fde080051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.981+0000 7fde26000640 1 --2- 172.21.15.40:0/2767664836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde08004800 0x7fde08004c80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fde10007330 tx=0x7fde08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfce5b5caa6e7ff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.982+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10033070 con 0x7fde08004800 2024-04-26T23:02:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.982+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde1002f630 con 0x7fde08004800 2024-04-26T23:02:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.982+0000 7fde25600640 1 -- 172.21.15.40:0/2767664836 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10009be0 con 0x7fde08004800 2024-04-26T23:02:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.982+0000 7fde2d600640 1 -- 172.21.15.40:0/2767664836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde08004800 msgr2=0x7fde08004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.982+0000 7fde2d600640 1 --2- 172.21.15.40:0/2767664836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde08004800 0x7fde08004c80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fde10007330 tx=0x7fde08005830 comp rx=0 tx=0).stop 2024-04-26T23:02:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.982+0000 7fde27400640 1 -- 172.21.15.40:0/2767664836 reap_dead start 2024-04-26T23:02:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.982+0000 7fde2d600640 1 -- 172.21.15.40:0/2767664836 shutdown_connections 2024-04-26T23:02:23.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.982+0000 7fde2d600640 1 -- 172.21.15.40:0/2767664836 >> 172.21.15.40:0/2767664836 conn(0x7fde280d0be0 msgr2=0x7fde280d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:23.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.983+0000 7fde2d600640 1 -- 172.21.15.40:0/2767664836 shutdown_connections 2024-04-26T23:02:23.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.983+0000 7fde2d600640 1 -- 172.21.15.40:0/2767664836 wait complete. 2024-04-26T23:02:23.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.984+0000 7fde2d600640 1 Processor -- start 2024-04-26T23:02:23.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.984+0000 7fde2d600640 1 -- start start 2024-04-26T23:02:23.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.985+0000 7fde2d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde08004800 0x7fde2814aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.985+0000 7fde2d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde2815c240 0x7fde2815c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.985+0000 7fde2d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde2815e580 0x7fde2815ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.986+0000 7fde2d600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde2815d3f0 con 0x7fde2815c240 2024-04-26T23:02:23.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.986+0000 7fde2d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde28072c10 con 0x7fde2815e580 2024-04-26T23:02:23.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.986+0000 7fde2d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fde2815d570 con 0x7fde08004800 2024-04-26T23:02:23.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.986+0000 7fde26000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde2815c240 0x7fde2815c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:23.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.986+0000 7fde26000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde2815c240 0x7fde2815c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52196/0 (socket says 172.21.15.40:52196) 2024-04-26T23:02:23.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.986+0000 7fde26000640 1 -- 172.21.15.40:0/1765199903 learned_addr learned my addr 172.21.15.40:0/1765199903 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:23.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.986+0000 7fde26000640 1 -- 172.21.15.40:0/1765199903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde08004800 msgr2=0x7fde2814aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:23.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.986+0000 7fde26000640 1 --2- 172.21.15.40:0/1765199903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde08004800 0x7fde2814aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.986+0000 7fde26000640 1 -- 172.21.15.40:0/1765199903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde2815e580 msgr2=0x7fde2815ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:23.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.986+0000 7fde27400640 1 --2- 172.21.15.40:0/1765199903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde2815e580 0x7fde2815ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:23.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.986+0000 7fde26000640 1 --2- 172.21.15.40:0/1765199903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde2815e580 0x7fde2815ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:23.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.986+0000 7fde26000640 1 -- 172.21.15.40:0/1765199903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde281f1070 con 0x7fde2815c240 2024-04-26T23:02:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.987+0000 7fde26000640 1 --2- 172.21.15.40:0/1765199903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde2815c240 0x7fde2815c6c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fde10007450 tx=0x7fde10004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.987+0000 7fde1ea00640 1 -- 172.21.15.40:0/1765199903 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10033040 con 0x7fde2815c240 2024-04-26T23:02:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.987+0000 7fde1ea00640 1 -- 172.21.15.40:0/1765199903 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde1002f430 con 0x7fde2815c240 2024-04-26T23:02:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.987+0000 7fde1ea00640 1 -- 172.21.15.40:0/1765199903 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10009be0 con 0x7fde2815c240 2024-04-26T23:02:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.987+0000 7fde2d600640 1 -- 172.21.15.40:0/1765199903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde281f1350 con 0x7fde2815c240 2024-04-26T23:02:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.987+0000 7fde2d600640 1 -- 172.21.15.40:0/1765199903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde281f1910 con 0x7fde2815c240 2024-04-26T23:02:23.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.988+0000 7fde2d600640 1 -- 172.21.15.40:0/1765199903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde280694f0 con 0x7fde2815c240 2024-04-26T23:02:23.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.990+0000 7fde1ea00640 1 -- 172.21.15.40:0/1765199903 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fde10036070 con 0x7fde2815c240 2024-04-26T23:02:23.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.990+0000 7fde1ea00640 1 --2- 172.21.15.40:0/1765199903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fde0406c830 0x7fde0406ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:23.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.991+0000 7fde1ea00640 1 -- 172.21.15.40:0/1765199903 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fde100ac870 con 0x7fde2815c240 2024-04-26T23:02:23.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.991+0000 7fde26a00640 1 --2- 172.21.15.40:0/1765199903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fde0406c830 0x7fde0406ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:23.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.992+0000 7fde26a00640 1 --2- 172.21.15.40:0/1765199903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fde0406c830 0x7fde0406ecf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fde18001050 tx=0x7fde18001d90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:23.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:23.995+0000 7fde1ea00640 1 -- 172.21.15.40:0/1765199903 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fde1007a770 con 0x7fde2815c240 2024-04-26T23:02:24.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.203+0000 7fde2d600640 1 -- 172.21.15.40:0/1765199903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fde280cb010 con 0x7fde2815c240 2024-04-26T23:02:24.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.204+0000 7fde1ea00640 1 -- 172.21.15.40:0/1765199903 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fde1007a070 con 0x7fde2815c240 2024-04-26T23:02:24.204 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:02:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.206+0000 7fde2d600640 1 -- 172.21.15.40:0/1765199903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fde0406c830 msgr2=0x7fde0406ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.206+0000 7fde2d600640 1 --2- 172.21.15.40:0/1765199903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fde0406c830 0x7fde0406ecf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fde18001050 tx=0x7fde18001d90 comp rx=0 tx=0).stop 2024-04-26T23:02:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.206+0000 7fde2d600640 1 -- 172.21.15.40:0/1765199903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde2815c240 msgr2=0x7fde2815c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.206+0000 7fde2d600640 1 --2- 172.21.15.40:0/1765199903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde2815c240 0x7fde2815c6c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fde10007450 tx=0x7fde10004ab0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.207+0000 7fde2d600640 1 -- 172.21.15.40:0/1765199903 shutdown_connections 2024-04-26T23:02:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.207+0000 7fde2d600640 1 --2- 172.21.15.40:0/1765199903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fde0406c830 0x7fde0406ecf0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.207+0000 7fde2d600640 1 --2- 172.21.15.40:0/1765199903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde2815e580 0x7fde2815ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.207+0000 7fde2d600640 1 --2- 172.21.15.40:0/1765199903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde2815c240 0x7fde2815c6c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.207+0000 7fde2d600640 1 --2- 172.21.15.40:0/1765199903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde08004800 0x7fde2814aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.207+0000 7fde2d600640 1 -- 172.21.15.40:0/1765199903 >> 172.21.15.40:0/1765199903 conn(0x7fde280d0be0 msgr2=0x7fde280c7440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.207+0000 7fde2d600640 1 -- 172.21.15.40:0/1765199903 shutdown_connections 2024-04-26T23:02:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.207+0000 7fde2d600640 1 -- 172.21.15.40:0/1765199903 wait complete. 2024-04-26T23:02:24.227 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:02:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.439+0000 7f5ede800640 1 Processor -- start 2024-04-26T23:02:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.440+0000 7f5ede800640 1 -- start start 2024-04-26T23:02:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.441+0000 7f5ede800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ed806f790 0x7f5ed806fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.441+0000 7f5ede800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ed8072c50 0x7f5ed80730a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.441+0000 7f5ede800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed8156840 con 0x7f5ed8156460 2024-04-26T23:02:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.441+0000 7f5ede800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f5ed80cbf20 con 0x7f5ed806f790 2024-04-26T23:02:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.441+0000 7f5ede800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5ed80cc200 con 0x7f5ed8072c50 2024-04-26T23:02:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.442+0000 7f5ed7400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5ed8156460 0x7f5ed80cb4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52480/0 (socket says 172.21.15.40:52480) 2024-04-26T23:02:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.442+0000 7f5ed7400640 1 -- 172.21.15.40:0/3496597670 learned_addr learned my addr 172.21.15.40:0/3496597670 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.442+0000 7f5ecfe00640 1 --2- 172.21.15.40:0/3496597670 >> v2:172.21.15.40:3300/0 conn(0x7f5ed806f790 0x7f5ed806fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.442+0000 7f5ed7e00640 1 --2- 172.21.15.40:0/3496597670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ed8072c50 0x7f5ed80730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.442+0000 7f5ed6a00640 1 -- 172.21.15.40:0/3496597670 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1233694317 0 0) 0x7f5ed8156840 con 0x7f5ed8156460 2024-04-26T23:02:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.442+0000 7f5ed6a00640 1 -- 172.21.15.40:0/3496597670 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ebc003520 con 0x7f5ed8156460 2024-04-26T23:02:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.442+0000 7f5ecfe00640 1 -- 172.21.15.40:0/3496597670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ed8072c50 msgr2=0x7f5ed80730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.442+0000 7f5ecfe00640 1 --2- 172.21.15.40:0/3496597670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ed8072c50 0x7f5ed80730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.442+0000 7f5ecfe00640 1 -- 172.21.15.40:0/3496597670 >> v1:172.21.15.40:6789/0 conn(0x7f5ed8156460 legacy=0x7f5ed80cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.442+0000 7f5ecfe00640 1 -- 172.21.15.40:0/3496597670 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed80cc4e0 con 0x7f5ed806f790 2024-04-26T23:02:24.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.444+0000 7f5ecfe00640 1 --2- 172.21.15.40:0/3496597670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ed806f790 0x7f5ed806fc40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00097a0 tx=0x7f5ec002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e388ab2cb8c166 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:24.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.444+0000 7f5ed6a00640 1 -- 172.21.15.40:0/3496597670 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0037070 con 0x7f5ed806f790 2024-04-26T23:02:24.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.444+0000 7f5ed6a00640 1 -- 172.21.15.40:0/3496597670 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ec0032440 con 0x7f5ed806f790 2024-04-26T23:02:24.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.444+0000 7f5ed6a00640 1 -- 172.21.15.40:0/3496597670 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0032810 con 0x7f5ed806f790 2024-04-26T23:02:24.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.444+0000 7f5ede800640 1 -- 172.21.15.40:0/3496597670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ed806f790 msgr2=0x7f5ed806fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:24.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.444+0000 7f5ede800640 1 --2- 172.21.15.40:0/3496597670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ed806f790 0x7f5ed806fc40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00097a0 tx=0x7f5ec002a180 comp rx=0 tx=0).stop 2024-04-26T23:02:24.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.445+0000 7f5ede800640 1 -- 172.21.15.40:0/3496597670 shutdown_connections 2024-04-26T23:02:24.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.445+0000 7f5ede800640 1 --2- 172.21.15.40:0/3496597670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ed8072c50 0x7f5ed80730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.445+0000 7f5ede800640 1 --2- 172.21.15.40:0/3496597670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ed806f790 0x7f5ed806fc40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.445+0000 7f5ede800640 1 -- 172.21.15.40:0/3496597670 >> 172.21.15.40:0/3496597670 conn(0x7f5ed80d0be0 msgr2=0x7f5ed80d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.445+0000 7f5ede800640 1 -- 172.21.15.40:0/3496597670 shutdown_connections 2024-04-26T23:02:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.445+0000 7f5ede800640 1 -- 172.21.15.40:0/3496597670 wait complete. 2024-04-26T23:02:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.447+0000 7f5ede800640 1 Processor -- start 2024-04-26T23:02:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.447+0000 7f5ede800640 1 -- start start 2024-04-26T23:02:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.448+0000 7f5ede800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ed806f790 0x7f5ed8148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.448+0000 7f5ede800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ed8072c50 0x7f5ed80b4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.448+0000 7f5ede800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ed8156460 0x7f5ed814bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.448+0000 7f5ede800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed80cbbe0 con 0x7f5ed806f790 2024-04-26T23:02:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.448+0000 7f5ede800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed8072700 con 0x7f5ed8156460 2024-04-26T23:02:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.448+0000 7f5ede800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5ed806e260 con 0x7f5ed8072c50 2024-04-26T23:02:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.448+0000 7f5ed7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ed806f790 0x7f5ed8148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.448+0000 7f5ed7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ed806f790 0x7f5ed8148060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52224/0 (socket says 172.21.15.40:52224) 2024-04-26T23:02:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.448+0000 7f5ed7400640 1 -- 172.21.15.40:0/3254601289 learned_addr learned my addr 172.21.15.40:0/3254601289 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.449+0000 7f5ed7400640 1 -- 172.21.15.40:0/3254601289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ed8072c50 msgr2=0x7f5ed80b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.449+0000 7f5ed7400640 1 --2- 172.21.15.40:0/3254601289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ed8072c50 0x7f5ed80b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.449+0000 7f5ed7400640 1 -- 172.21.15.40:0/3254601289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ed8156460 msgr2=0x7f5ed814bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.449+0000 7f5ed7400640 1 --2- 172.21.15.40:0/3254601289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ed8156460 0x7f5ed814bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.449+0000 7f5ed7400640 1 -- 172.21.15.40:0/3254601289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed8168d10 con 0x7f5ed806f790 2024-04-26T23:02:24.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.449+0000 7f5ed7400640 1 --2- 172.21.15.40:0/3254601289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ed806f790 0x7f5ed8148060 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800c3c0 tx=0x7f5ec800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:24.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.449+0000 7f5ecf400640 1 -- 172.21.15.40:0/3254601289 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8017070 con 0x7f5ed806f790 2024-04-26T23:02:24.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.449+0000 7f5ecf400640 1 -- 172.21.15.40:0/3254601289 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ec8005ce0 con 0x7f5ed806f790 2024-04-26T23:02:24.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.449+0000 7f5ecf400640 1 -- 172.21.15.40:0/3254601289 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec80072f0 con 0x7f5ed806f790 2024-04-26T23:02:24.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.449+0000 7f5ede800640 1 -- 172.21.15.40:0/3254601289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed80b3860 con 0x7f5ed806f790 2024-04-26T23:02:24.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.450+0000 7f5ede800640 1 -- 172.21.15.40:0/3254601289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ed80b3e20 con 0x7f5ed806f790 2024-04-26T23:02:24.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.451+0000 7f5ede800640 1 -- 172.21.15.40:0/3254601289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ed806fc40 con 0x7f5ed806f790 2024-04-26T23:02:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.452+0000 7f5ecf400640 1 -- 172.21.15.40:0/3254601289 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5ec801a070 con 0x7f5ed806f790 2024-04-26T23:02:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.453+0000 7f5ecf400640 1 --2- 172.21.15.40:0/3254601289 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5eb406c8f0 0x7f5eb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.453+0000 7f5ecf400640 1 -- 172.21.15.40:0/3254601289 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f5ec808b430 con 0x7f5ed806f790 2024-04-26T23:02:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.454+0000 7f5ecfe00640 1 --2- 172.21.15.40:0/3254601289 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5eb406c8f0 0x7f5eb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:24.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.455+0000 7f5ecfe00640 1 --2- 172.21.15.40:0/3254601289 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5eb406c8f0 0x7f5eb406edb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0034000 tx=0x7f5ec002c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.458+0000 7f5ecf400640 1 -- 172.21.15.40:0/3254601289 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5ed806fc40 con 0x7f5ed806f790 2024-04-26T23:02:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.663+0000 7f5ede800640 1 -- 172.21.15.40:0/3254601289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5ed80694f0 con 0x7f5ed806f790 2024-04-26T23:02:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.664+0000 7f5ecf400640 1 -- 172.21.15.40:0/3254601289 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f5ec80593f0 con 0x7f5ed806f790 2024-04-26T23:02:24.664 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.666+0000 7f5ede800640 1 -- 172.21.15.40:0/3254601289 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5eb406c8f0 msgr2=0x7f5eb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.667+0000 7f5ede800640 1 --2- 172.21.15.40:0/3254601289 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5eb406c8f0 0x7f5eb406edb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0034000 tx=0x7f5ec002c000 comp rx=0 tx=0).stop 2024-04-26T23:02:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.667+0000 7f5ede800640 1 -- 172.21.15.40:0/3254601289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ed806f790 msgr2=0x7f5ed8148060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.667+0000 7f5ede800640 1 --2- 172.21.15.40:0/3254601289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ed806f790 0x7f5ed8148060 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800c3c0 tx=0x7f5ec800c880 comp rx=0 tx=0).stop 2024-04-26T23:02:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.667+0000 7f5ede800640 1 -- 172.21.15.40:0/3254601289 shutdown_connections 2024-04-26T23:02:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.667+0000 7f5ede800640 1 --2- 172.21.15.40:0/3254601289 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5eb406c8f0 0x7f5eb406edb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.667+0000 7f5ede800640 1 --2- 172.21.15.40:0/3254601289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ed8156460 0x7f5ed814bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.667+0000 7f5ede800640 1 --2- 172.21.15.40:0/3254601289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ed8072c50 0x7f5ed80b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.667+0000 7f5ede800640 1 --2- 172.21.15.40:0/3254601289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ed806f790 0x7f5ed8148060 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.667+0000 7f5ede800640 1 -- 172.21.15.40:0/3254601289 >> 172.21.15.40:0/3254601289 conn(0x7f5ed80d0be0 msgr2=0x7f5ed80d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.667+0000 7f5ede800640 1 -- 172.21.15.40:0/3254601289 shutdown_connections 2024-04-26T23:02:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.667+0000 7f5ede800640 1 -- 172.21.15.40:0/3254601289 wait complete. 2024-04-26T23:02:24.685 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:02:24.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.895+0000 7f1ff0a00640 1 Processor -- start 2024-04-26T23:02:24.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.896+0000 7f1ff0a00640 1 -- start start 2024-04-26T23:02:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.897+0000 7f1ff0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fec0c6f10 0x7f1fec0c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.897+0000 7f1ff0a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1fec0cc980 0x7f1fec0cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.897+0000 7f1ff0a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fec0cce10 con 0x7f1fec0c6010 2024-04-26T23:02:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.897+0000 7f1ff0a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1fec0c72f0 con 0x7f1fec0cc980 2024-04-26T23:02:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.897+0000 7f1ff0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1fec0cb860 con 0x7f1fec0c6f10 2024-04-26T23:02:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.898+0000 7f1fe3e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1fec0cc980 0x7f1fec0cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.898+0000 7f1feb400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fec0c6f10 0x7f1fec0c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.898+0000 7f1fe3e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1fec0cc980 0x7f1fec0cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50656/0 (socket says 172.21.15.40:50656) 2024-04-26T23:02:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.898+0000 7f1febe00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1fec0c6010 0x7f1fec158cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:33028/0 (socket says 172.21.15.40:33028) 2024-04-26T23:02:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.898+0000 7f1fe3e00640 1 -- 172.21.15.40:0/2959757651 learned_addr learned my addr 172.21.15.40:0/2959757651 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.899+0000 7f1feb400640 1 -- 172.21.15.40:0/2959757651 >> v2:172.21.15.103:3300/0 conn(0x7f1fec0cc980 msgr2=0x7f1fec0cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.899+0000 7f1feb400640 1 --2- 172.21.15.40:0/2959757651 >> v2:172.21.15.103:3300/0 conn(0x7f1fec0cc980 0x7f1fec0cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.899+0000 7f1feb400640 1 -- 172.21.15.40:0/2959757651 >> v1:172.21.15.103:6789/0 conn(0x7f1fec0c6010 legacy=0x7f1fec158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:24.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.899+0000 7f1feb400640 1 -- 172.21.15.40:0/2959757651 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fec0cd0f0 con 0x7f1fec0c6f10 2024-04-26T23:02:24.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.899+0000 7f1feaa00640 1 -- 172.21.15.40:0/2959757651 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2377156737 0 0) 0x7f1fec0cce10 con 0x7f1fec0c6010 2024-04-26T23:02:24.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.900+0000 7f1feb400640 1 --2- 172.21.15.40:0/2959757651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fec0c6f10 0x7f1fec0c5ad0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0097a0 tx=0x7f1fdc02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49fc51618095bd2c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.901+0000 7f1feaa00640 1 -- 172.21.15.40:0/2959757651 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc037070 con 0x7f1fec0c6f10 2024-04-26T23:02:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.901+0000 7f1feaa00640 1 -- 172.21.15.40:0/2959757651 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fdc02ae50 con 0x7f1fec0c6f10 2024-04-26T23:02:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.901+0000 7f1feaa00640 1 -- 172.21.15.40:0/2959757651 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc032660 con 0x7f1fec0c6f10 2024-04-26T23:02:24.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.901+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2959757651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fec0c6f10 msgr2=0x7f1fec0c5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:24.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.901+0000 7f1ff0a00640 1 --2- 172.21.15.40:0/2959757651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fec0c6f10 0x7f1fec0c5ad0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0097a0 tx=0x7f1fdc02a100 comp rx=0 tx=0).stop 2024-04-26T23:02:24.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.901+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2959757651 shutdown_connections 2024-04-26T23:02:24.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.901+0000 7f1ff0a00640 1 --2- 172.21.15.40:0/2959757651 >> v2:172.21.15.103:3300/0 conn(0x7f1fec0cc980 0x7f1fec0cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.901+0000 7f1ff0a00640 1 --2- 172.21.15.40:0/2959757651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fec0c6f10 0x7f1fec0c5ad0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.901+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2959757651 >> 172.21.15.40:0/2959757651 conn(0x7f1fec0d0a10 msgr2=0x7f1fec0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:24.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.902+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2959757651 shutdown_connections 2024-04-26T23:02:24.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.902+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2959757651 wait complete. 2024-04-26T23:02:24.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.903+0000 7f1ff0a00640 1 Processor -- start 2024-04-26T23:02:24.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.903+0000 7f1ff0a00640 1 -- start start 2024-04-26T23:02:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.904+0000 7f1ff0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec0c6010 0x7f1fec145d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.904+0000 7f1ff0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fec0c6f10 0x7f1fec158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.904+0000 7f1ff0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fec0c7ae0 0x7f1fec0ce570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.904+0000 7f1ff0a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec0c76a0 con 0x7f1fec0c6010 2024-04-26T23:02:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.904+0000 7f1ff0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec0cb1a0 con 0x7f1fec0c6f10 2024-04-26T23:02:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.904+0000 7f1ff0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1fec0c7820 con 0x7f1fec0c7ae0 2024-04-26T23:02:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.905+0000 7f1feb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec0c6010 0x7f1fec145d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.905+0000 7f1feb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec0c6010 0x7f1fec145d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45208/0 (socket says 172.21.15.40:45208) 2024-04-26T23:02:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.905+0000 7f1feb400640 1 -- 172.21.15.40:0/2130001123 learned_addr learned my addr 172.21.15.40:0/2130001123 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.905+0000 7f1fe3e00640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fec0c6f10 0x7f1fec158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.905+0000 7f1febe00640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fec0c7ae0 0x7f1fec0ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.905+0000 7f1feb400640 1 -- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fec0c7ae0 msgr2=0x7f1fec0ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.905+0000 7f1feb400640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fec0c7ae0 0x7f1fec0ce570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.905+0000 7f1feb400640 1 -- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fec0c6f10 msgr2=0x7f1fec158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.905+0000 7f1feb400640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fec0c6f10 0x7f1fec158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.905+0000 7f1feb400640 1 -- 172.21.15.40:0/2130001123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fec158bc0 con 0x7f1fec0c6010 2024-04-26T23:02:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.906+0000 7f1feb400640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec0c6010 0x7f1fec145d80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc02a6f0 tx=0x7f1fdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.906+0000 7f1fe3400640 1 -- 172.21.15.40:0/2130001123 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc042020 con 0x7f1fec0c6010 2024-04-26T23:02:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.906+0000 7f1fe3400640 1 -- 172.21.15.40:0/2130001123 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fdc03d690 con 0x7f1fec0c6010 2024-04-26T23:02:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.906+0000 7f1fe3400640 1 -- 172.21.15.40:0/2130001123 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc037040 con 0x7f1fec0c6010 2024-04-26T23:02:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.906+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2130001123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fec0b4730 con 0x7f1fec0c6010 2024-04-26T23:02:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.906+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2130001123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fec0b50f0 con 0x7f1fec0c6010 2024-04-26T23:02:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.907+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2130001123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fec069370 con 0x7f1fec0c6010 2024-04-26T23:02:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.908+0000 7f1fe3400640 1 -- 172.21.15.40:0/2130001123 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1fdc045020 con 0x7f1fec0c6010 2024-04-26T23:02:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.910+0000 7f1fe3400640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1fc806c8f0 0x7f1fc806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.910+0000 7f1fe3400640 1 -- 172.21.15.40:0/2130001123 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1fdc0ac370 con 0x7f1fec0c6010 2024-04-26T23:02:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.910+0000 7f1fe3e00640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1fc806c8f0 0x7f1fc806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.912+0000 7f1fe3e00640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1fc806c8f0 0x7f1fc806edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1fec157eb0 tx=0x7f1fd4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:24.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:24.914+0000 7f1fe3400640 1 -- 172.21.15.40:0/2130001123 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1fdc04c070 con 0x7f1fec0c6010 2024-04-26T23:02:25.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.119+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2130001123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1fec070020 con 0x7f1fec0c6010 2024-04-26T23:02:25.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.120+0000 7f1fe3400640 1 -- 172.21.15.40:0/2130001123 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f1fdc07a390 con 0x7f1fec0c6010 2024-04-26T23:02:25.120 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:02:25.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.123+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1fc806c8f0 msgr2=0x7f1fc806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:25.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.123+0000 7f1ff0a00640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1fc806c8f0 0x7f1fc806edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1fec157eb0 tx=0x7f1fd4008040 comp rx=0 tx=0).stop 2024-04-26T23:02:25.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.123+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2130001123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec0c6010 msgr2=0x7f1fec145d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:25.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.123+0000 7f1ff0a00640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec0c6010 0x7f1fec145d80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc02a6f0 tx=0x7f1fdc0046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.123+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2130001123 shutdown_connections 2024-04-26T23:02:25.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.123+0000 7f1ff0a00640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1fc806c8f0 0x7f1fc806edb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.123+0000 7f1ff0a00640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fec0c7ae0 0x7f1fec0ce570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.123+0000 7f1ff0a00640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fec0c6f10 0x7f1fec158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.123+0000 7f1ff0a00640 1 --2- 172.21.15.40:0/2130001123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec0c6010 0x7f1fec145d80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.123+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2130001123 >> 172.21.15.40:0/2130001123 conn(0x7f1fec0d0a10 msgr2=0x7f1fec0ae8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:25.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.123+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2130001123 shutdown_connections 2024-04-26T23:02:25.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.123+0000 7f1ff0a00640 1 -- 172.21.15.40:0/2130001123 wait complete. 2024-04-26T23:02:25.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:02:25.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.355+0000 7f41e5000640 1 Processor -- start 2024-04-26T23:02:25.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.355+0000 7f41e5000640 1 -- start start 2024-04-26T23:02:25.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.357+0000 7f41e5000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f41e00b5120 0x7f41e00b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:25.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.357+0000 7f41e5000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41e00b59b0 0x7f41e00b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:25.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.357+0000 7f41e5000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41e00d3dc0 con 0x7f41e00b5500 2024-04-26T23:02:25.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.357+0000 7f41e5000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f41e00d03b0 con 0x7f41e00b5120 2024-04-26T23:02:25.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.357+0000 7f41e5000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f41e00b75b0 con 0x7f41e00b59b0 2024-04-26T23:02:25.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.357+0000 7f41dea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f41e00b5500 0x7f41e00b6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35758/0 (socket says 172.21.15.40:35758) 2024-04-26T23:02:25.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.357+0000 7f41dea00640 1 -- 172.21.15.40:0/2296715091 learned_addr learned my addr 172.21.15.40:0/2296715091 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:25.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.357+0000 7f41df400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f41e00b5120 0x7f41e00b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:25.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.358+0000 7f41dfe00640 1 --2- 172.21.15.40:0/2296715091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41e00b59b0 0x7f41e00b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:25.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.358+0000 7f41df400640 1 -- 172.21.15.40:0/2296715091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41e00b59b0 msgr2=0x7f41e00b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:25.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.358+0000 7f41df400640 1 --2- 172.21.15.40:0/2296715091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41e00b59b0 0x7f41e00b6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.358+0000 7f41df400640 1 -- 172.21.15.40:0/2296715091 >> v1:172.21.15.40:6789/0 conn(0x7f41e00b5500 legacy=0x7f41e00b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:25.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.358+0000 7f41df400640 1 -- 172.21.15.40:0/2296715091 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e00d0600 con 0x7f41e00b5120 2024-04-26T23:02:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.359+0000 7f41df400640 1 --2- 172.21.15.40:0/2296715091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e00b5120 0x7f41e00b5ee0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f41d00097d0 tx=0x7f41d002a090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5178f73477750c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.359+0000 7f41de000640 1 -- 172.21.15.40:0/2296715091 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d0037070 con 0x7f41e00b5120 2024-04-26T23:02:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.359+0000 7f41de000640 1 -- 172.21.15.40:0/2296715091 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41d002ada0 con 0x7f41e00b5120 2024-04-26T23:02:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.359+0000 7f41de000640 1 -- 172.21.15.40:0/2296715091 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d0032660 con 0x7f41e00b5120 2024-04-26T23:02:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.360+0000 7f41e5000640 1 -- 172.21.15.40:0/2296715091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e00b5120 msgr2=0x7f41e00b5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.360+0000 7f41e5000640 1 --2- 172.21.15.40:0/2296715091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e00b5120 0x7f41e00b5ee0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f41d00097d0 tx=0x7f41d002a090 comp rx=0 tx=0).stop 2024-04-26T23:02:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.360+0000 7f41e5000640 1 -- 172.21.15.40:0/2296715091 shutdown_connections 2024-04-26T23:02:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.360+0000 7f41e5000640 1 --2- 172.21.15.40:0/2296715091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41e00b59b0 0x7f41e00b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.360+0000 7f41e5000640 1 --2- 172.21.15.40:0/2296715091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e00b5120 0x7f41e00b5ee0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.360+0000 7f41e5000640 1 -- 172.21.15.40:0/2296715091 >> 172.21.15.40:0/2296715091 conn(0x7f41e00d4db0 msgr2=0x7f41e00d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.360+0000 7f41e5000640 1 -- 172.21.15.40:0/2296715091 shutdown_connections 2024-04-26T23:02:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.360+0000 7f41e5000640 1 -- 172.21.15.40:0/2296715091 wait complete. 2024-04-26T23:02:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.362+0000 7f41e5000640 1 Processor -- start 2024-04-26T23:02:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.362+0000 7f41e5000640 1 -- start start 2024-04-26T23:02:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.363+0000 7f41e5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41e00b5120 0x7f41e015c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.363+0000 7f41e5000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41e00b5500 0x7f41e01f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.363+0000 7f41e5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e00b59b0 0x7f41e006d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.363+0000 7f41e5000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f41e00b7c00 con 0x7f41e00b59b0 2024-04-26T23:02:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.363+0000 7f41e5000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f41e00d0bc0 con 0x7f41e00b5120 2024-04-26T23:02:25.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.363+0000 7f41e5000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f41e00b7d80 con 0x7f41e00b5500 2024-04-26T23:02:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.364+0000 7f41dfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e00b59b0 0x7f41e006d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.364+0000 7f41dfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e00b59b0 0x7f41e006d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45240/0 (socket says 172.21.15.40:45240) 2024-04-26T23:02:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.364+0000 7f41dfe00640 1 -- 172.21.15.40:0/1563037136 learned_addr learned my addr 172.21.15.40:0/1563037136 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.364+0000 7f41df400640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41e00b5120 0x7f41e015c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.364+0000 7f41dea00640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41e00b5500 0x7f41e01f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.364+0000 7f41dfe00640 1 -- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41e00b5500 msgr2=0x7f41e01f4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.364+0000 7f41dfe00640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41e00b5500 0x7f41e01f4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.364+0000 7f41dfe00640 1 -- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41e00b5120 msgr2=0x7f41e015c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.364+0000 7f41dfe00640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41e00b5120 0x7f41e015c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.364+0000 7f41dfe00640 1 -- 172.21.15.40:0/1563037136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e01f5520 con 0x7f41e00b59b0 2024-04-26T23:02:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.365+0000 7f41dfe00640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e00b59b0 0x7f41e006d170 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f41d8007e60 tx=0x7f41d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.365+0000 7f41d7400640 1 -- 172.21.15.40:0/1563037136 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d8014420 con 0x7f41e00b59b0 2024-04-26T23:02:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.365+0000 7f41d7400640 1 -- 172.21.15.40:0/1563037136 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41d8019070 con 0x7f41e00b59b0 2024-04-26T23:02:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.365+0000 7f41d7400640 1 -- 172.21.15.40:0/1563037136 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d8014820 con 0x7f41e00b59b0 2024-04-26T23:02:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.365+0000 7f41e5000640 1 -- 172.21.15.40:0/1563037136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41e01f5800 con 0x7f41e00b59b0 2024-04-26T23:02:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.365+0000 7f41e5000640 1 -- 172.21.15.40:0/1563037136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41e01f5e10 con 0x7f41e00b59b0 2024-04-26T23:02:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.367+0000 7f41e5000640 1 -- 172.21.15.40:0/1563037136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41e0069370 con 0x7f41e00b59b0 2024-04-26T23:02:25.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.368+0000 7f41d7400640 1 -- 172.21.15.40:0/1563037136 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f41d80149c0 con 0x7f41e00b59b0 2024-04-26T23:02:25.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.369+0000 7f41d7400640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f41c006c8f0 0x7f41c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:25.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.369+0000 7f41d7400640 1 -- 172.21.15.40:0/1563037136 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f41d808c720 con 0x7f41e00b59b0 2024-04-26T23:02:25.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.369+0000 7f41df400640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f41c006c8f0 0x7f41c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:25.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.370+0000 7f41df400640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f41c006c8f0 0x7f41c006edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f41d0034000 tx=0x7f41d002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.373+0000 7f41d7400640 1 -- 172.21.15.40:0/1563037136 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f41d805a7e0 con 0x7f41e00b59b0 2024-04-26T23:02:25.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.579+0000 7f41e5000640 1 -- 172.21.15.40:0/1563037136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f41e00750f0 con 0x7f41e00b59b0 2024-04-26T23:02:25.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.580+0000 7f41d7400640 1 -- 172.21.15.40:0/1563037136 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f41d805a0e0 con 0x7f41e00b59b0 2024-04-26T23:02:25.580 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:25.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.582+0000 7f41e5000640 1 -- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f41c006c8f0 msgr2=0x7f41c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:25.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.582+0000 7f41e5000640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f41c006c8f0 0x7f41c006edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f41d0034000 tx=0x7f41d002e000 comp rx=0 tx=0).stop 2024-04-26T23:02:25.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.582+0000 7f41e5000640 1 -- 172.21.15.40:0/1563037136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e00b59b0 msgr2=0x7f41e006d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:25.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.582+0000 7f41e5000640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e00b59b0 0x7f41e006d170 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f41d8007e60 tx=0x7f41d800e4b0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.582+0000 7f41e5000640 1 -- 172.21.15.40:0/1563037136 shutdown_connections 2024-04-26T23:02:25.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.582+0000 7f41e5000640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f41c006c8f0 0x7f41c006edb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.582+0000 7f41e5000640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e00b59b0 0x7f41e006d170 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.582+0000 7f41e5000640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41e00b5500 0x7f41e01f4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.582+0000 7f41e5000640 1 --2- 172.21.15.40:0/1563037136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41e00b5120 0x7f41e015c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.582+0000 7f41e5000640 1 -- 172.21.15.40:0/1563037136 >> 172.21.15.40:0/1563037136 conn(0x7f41e00d4db0 msgr2=0x7f41e00b6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:25.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.582+0000 7f41e5000640 1 -- 172.21.15.40:0/1563037136 shutdown_connections 2024-04-26T23:02:25.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.582+0000 7f41e5000640 1 -- 172.21.15.40:0/1563037136 wait complete. 2024-04-26T23:02:25.601 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:02:25.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.815+0000 7fbeae800640 1 Processor -- start 2024-04-26T23:02:25.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.816+0000 7fbeae800640 1 -- start start 2024-04-26T23:02:25.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.817+0000 7fbeae800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbea80b6220 0x7fbea80d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:25.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.817+0000 7fbeae800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbea8162c40 con 0x7fbea80d39f0 2024-04-26T23:02:25.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.817+0000 7fbeae800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbea80d2d80 con 0x7fbea80d08d0 2024-04-26T23:02:25.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.817+0000 7fbeae800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fbea80b7c80 con 0x7fbea80b6220 2024-04-26T23:02:25.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.817+0000 7fbeade00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbea80d39f0 0x7fbea80b8d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35770/0 (socket says 172.21.15.40:35770) 2024-04-26T23:02:25.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.817+0000 7fbeade00640 1 -- 172.21.15.40:0/797716439 learned_addr learned my addr 172.21.15.40:0/797716439 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:25.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.817+0000 7fbead400640 1 --2- 172.21.15.40:0/797716439 >> v2:172.21.15.40:3300/0 conn(0x7fbea80b6220 0x7fbea80d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:25.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.818+0000 7fbea7e00640 1 -- 172.21.15.40:0/797716439 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1447615587 0 0) 0x7fbea8162c40 con 0x7fbea80d39f0 2024-04-26T23:02:25.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.818+0000 7fbea7e00640 1 -- 172.21.15.40:0/797716439 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe8c003520 con 0x7fbea80d39f0 2024-04-26T23:02:25.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.818+0000 7fbead400640 1 -- 172.21.15.40:0/797716439 >> v1:172.21.15.103:6789/0 conn(0x7fbea80d08d0 legacy=0x7fbea80d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:25.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.818+0000 7fbead400640 1 -- 172.21.15.40:0/797716439 >> v1:172.21.15.40:6789/0 conn(0x7fbea80d39f0 legacy=0x7fbea80b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:25.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.818+0000 7fbead400640 1 -- 172.21.15.40:0/797716439 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea80b7f60 con 0x7fbea80b6220 2024-04-26T23:02:25.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.819+0000 7fbead400640 1 --2- 172.21.15.40:0/797716439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbea80b6220 0x7fbea80d0390 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbe98009d90 tx=0x7fbe9802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48bfa63c19781290 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.820+0000 7fbea7e00640 1 -- 172.21.15.40:0/797716439 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe98007740 con 0x7fbea80b6220 2024-04-26T23:02:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.820+0000 7fbea7e00640 1 -- 172.21.15.40:0/797716439 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe980078e0 con 0x7fbea80b6220 2024-04-26T23:02:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.820+0000 7fbea7e00640 1 -- 172.21.15.40:0/797716439 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe98007cb0 con 0x7fbea80b6220 2024-04-26T23:02:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.820+0000 7fbeae800640 1 -- 172.21.15.40:0/797716439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbea80b6220 msgr2=0x7fbea80d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.820+0000 7fbeae800640 1 --2- 172.21.15.40:0/797716439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbea80b6220 0x7fbea80d0390 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbe98009d90 tx=0x7fbe9802a6f0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.820+0000 7fbeae800640 1 -- 172.21.15.40:0/797716439 shutdown_connections 2024-04-26T23:02:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.820+0000 7fbeae800640 1 --2- 172.21.15.40:0/797716439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbea80b6220 0x7fbea80d0390 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.820+0000 7fbeae800640 1 -- 172.21.15.40:0/797716439 >> 172.21.15.40:0/797716439 conn(0x7fbea80d4d90 msgr2=0x7fbea80d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.820+0000 7fbeae800640 1 -- 172.21.15.40:0/797716439 shutdown_connections 2024-04-26T23:02:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.820+0000 7fbeae800640 1 -- 172.21.15.40:0/797716439 wait complete. 2024-04-26T23:02:25.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.822+0000 7fbeae800640 1 Processor -- start 2024-04-26T23:02:25.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.822+0000 7fbeae800640 1 -- start start 2024-04-26T23:02:25.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.823+0000 7fbeae800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbea80b6220 0x7fbea8148010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:25.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.823+0000 7fbeae800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbea80d08d0 0x7fbea81f8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:25.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.823+0000 7fbeae800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbea80d39f0 0x7fbea8159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:25.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.823+0000 7fbeae800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbea80b9430 con 0x7fbea80d08d0 2024-04-26T23:02:25.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.823+0000 7fbeae800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbea80b8520 con 0x7fbea80d39f0 2024-04-26T23:02:25.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.823+0000 7fbeae800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbea80b95b0 con 0x7fbea80b6220 2024-04-26T23:02:25.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.823+0000 7fbeaca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbea80d08d0 0x7fbea81f8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.823+0000 7fbeaca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbea80d08d0 0x7fbea81f8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45272/0 (socket says 172.21.15.40:45272) 2024-04-26T23:02:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.823+0000 7fbeaca00640 1 -- 172.21.15.40:0/1383094586 learned_addr learned my addr 172.21.15.40:0/1383094586 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.824+0000 7fbead400640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbea80b6220 0x7fbea8148010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.824+0000 7fbeaca00640 1 -- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbea80b6220 msgr2=0x7fbea8148010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.824+0000 7fbeade00640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbea80d39f0 0x7fbea8159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.824+0000 7fbeaca00640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbea80b6220 0x7fbea8148010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.824+0000 7fbeaca00640 1 -- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbea80d39f0 msgr2=0x7fbea8159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.824+0000 7fbeaca00640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbea80d39f0 0x7fbea8159ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.824+0000 7fbeaca00640 1 -- 172.21.15.40:0/1383094586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea815a680 con 0x7fbea80d08d0 2024-04-26T23:02:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.824+0000 7fbead400640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbea80b6220 0x7fbea8148010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.824+0000 7fbeaca00640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbea80d08d0 0x7fbea81f8ac0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400c400 tx=0x7fbe9400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:25.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.825+0000 7fbea5600640 1 -- 172.21.15.40:0/1383094586 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe940052d0 con 0x7fbea80d08d0 2024-04-26T23:02:25.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.825+0000 7fbea5600640 1 -- 172.21.15.40:0/1383094586 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe94005490 con 0x7fbea80d08d0 2024-04-26T23:02:25.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.825+0000 7fbea5600640 1 -- 172.21.15.40:0/1383094586 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe94007260 con 0x7fbea80d08d0 2024-04-26T23:02:25.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.825+0000 7fbeae800640 1 -- 172.21.15.40:0/1383094586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbea80ccf50 con 0x7fbea80d08d0 2024-04-26T23:02:25.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.825+0000 7fbeae800640 1 -- 172.21.15.40:0/1383094586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbea80cd510 con 0x7fbea80d08d0 2024-04-26T23:02:25.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.826+0000 7fbeae800640 1 -- 172.21.15.40:0/1383094586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbea80cd710 con 0x7fbea80d08d0 2024-04-26T23:02:25.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.827+0000 7fbea5600640 1 -- 172.21.15.40:0/1383094586 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbe94007400 con 0x7fbea80d08d0 2024-04-26T23:02:25.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.828+0000 7fbea5600640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbe8806c8f0 0x7fbe8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:25.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.828+0000 7fbea5600640 1 -- 172.21.15.40:0/1383094586 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbe9408b4f0 con 0x7fbea80d08d0 2024-04-26T23:02:25.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.829+0000 7fbead400640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbe8806c8f0 0x7fbe8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:25.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.830+0000 7fbead400640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbe8806c8f0 0x7fbe8806edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbe98034000 tx=0x7fbe9802f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:25.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:25.832+0000 7fbea5600640 1 -- 172.21.15.40:0/1383094586 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fbea80cd710 con 0x7fbea80d08d0 2024-04-26T23:02:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.049+0000 7fbeae800640 1 -- 172.21.15.40:0/1383094586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbea8069370 con 0x7fbea80d08d0 2024-04-26T23:02:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.050+0000 7fbea5600640 1 -- 172.21.15.40:0/1383094586 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fbe94059510 con 0x7fbea80d08d0 2024-04-26T23:02:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-04-26T23:02:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.052+0000 7fbeae800640 1 -- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbe8806c8f0 msgr2=0x7fbe8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.052+0000 7fbeae800640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbe8806c8f0 0x7fbe8806edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbe98034000 tx=0x7fbe9802f000 comp rx=0 tx=0).stop 2024-04-26T23:02:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.052+0000 7fbeae800640 1 -- 172.21.15.40:0/1383094586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbea80d08d0 msgr2=0x7fbea81f8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.052+0000 7fbeae800640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbea80d08d0 0x7fbea81f8ac0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400c400 tx=0x7fbe9400c8c0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.053+0000 7fbeae800640 1 -- 172.21.15.40:0/1383094586 shutdown_connections 2024-04-26T23:02:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.053+0000 7fbeae800640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbe8806c8f0 0x7fbe8806edb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.053+0000 7fbeae800640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbea80d39f0 0x7fbea8159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.053+0000 7fbeae800640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbea80d08d0 0x7fbea81f8ac0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.053+0000 7fbeae800640 1 --2- 172.21.15.40:0/1383094586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbea80b6220 0x7fbea8148010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.053+0000 7fbeae800640 1 -- 172.21.15.40:0/1383094586 >> 172.21.15.40:0/1383094586 conn(0x7fbea80d4d90 msgr2=0x7fbea80b8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.053+0000 7fbeae800640 1 -- 172.21.15.40:0/1383094586 shutdown_connections 2024-04-26T23:02:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.053+0000 7fbeae800640 1 -- 172.21.15.40:0/1383094586 wait complete. 2024-04-26T23:02:26.073 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-04-26T23:02:26.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:26.074 DEBUG:teuthology.orchestra.run.smithi040:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-04-26T23:02:26.166 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:02:26.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:26.166 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-26T23:02:26.216 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:02:26.216 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:26.270 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:02:26.270 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:26.316 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:02:26.316 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:02:26.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.635+0000 7f3547200640 1 Processor -- start 2024-04-26T23:02:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.635+0000 7f3547200640 1 -- start start 2024-04-26T23:02:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.637+0000 7f3547200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35480b6220 0x7f35480d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.637+0000 7f3547200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35480d08d0 0x7f35480d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.637+0000 7f3547200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f35480d3820 0x7f35480b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.637+0000 7f3547200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f3548162c40 con 0x7f35480d08d0 2024-04-26T23:02:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.637+0000 7f3547200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f35480d2d80 con 0x7f35480d3820 2024-04-26T23:02:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.637+0000 7f3547200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f35480b7c80 con 0x7f35480b6220 2024-04-26T23:02:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.638+0000 7f353fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35480d08d0 0x7f35480d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.638+0000 7f353fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35480d08d0 0x7f35480d32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45274/0 (socket says 172.21.15.40:45274) 2024-04-26T23:02:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.638+0000 7f353fe00640 1 -- 172.21.15.40:0/3285131087 learned_addr learned my addr 172.21.15.40:0/3285131087 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.638+0000 7f3546800640 1 --2- 172.21.15.40:0/3285131087 >> v2:172.21.15.103:3300/0 conn(0x7f35480d3820 0x7f35480b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.638+0000 7f3545e00640 1 --2- 172.21.15.40:0/3285131087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35480b6220 0x7f35480d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.638+0000 7f353fe00640 1 -- 172.21.15.40:0/3285131087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35480b6220 msgr2=0x7f35480d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.638+0000 7f353fe00640 1 --2- 172.21.15.40:0/3285131087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35480b6220 0x7f35480d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.638+0000 7f353fe00640 1 -- 172.21.15.40:0/3285131087 >> v2:172.21.15.103:3300/0 conn(0x7f35480d3820 msgr2=0x7f35480b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.638+0000 7f353fe00640 1 --2- 172.21.15.40:0/3285131087 >> v2:172.21.15.103:3300/0 conn(0x7f35480d3820 0x7f35480b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.638+0000 7f353fe00640 1 -- 172.21.15.40:0/3285131087 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35480b7f60 con 0x7f35480d08d0 2024-04-26T23:02:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.639+0000 7f353fe00640 1 --2- 172.21.15.40:0/3285131087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35480d08d0 0x7f35480d32e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f35480b7920 tx=0x7f353002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558b34887620674d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.640+0000 7f3545400640 1 -- 172.21.15.40:0/3285131087 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530037070 con 0x7f35480d08d0 2024-04-26T23:02:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.640+0000 7f3545400640 1 -- 172.21.15.40:0/3285131087 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3530032440 con 0x7f35480d08d0 2024-04-26T23:02:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.640+0000 7f3545400640 1 -- 172.21.15.40:0/3285131087 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530032810 con 0x7f35480d08d0 2024-04-26T23:02:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.641+0000 7f3547200640 1 -- 172.21.15.40:0/3285131087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35480d08d0 msgr2=0x7f35480d32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.641+0000 7f3547200640 1 --2- 172.21.15.40:0/3285131087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35480d08d0 0x7f35480d32e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f35480b7920 tx=0x7f353002a1e0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.641+0000 7f3547200640 1 -- 172.21.15.40:0/3285131087 shutdown_connections 2024-04-26T23:02:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.641+0000 7f3547200640 1 --2- 172.21.15.40:0/3285131087 >> v2:172.21.15.103:3300/0 conn(0x7f35480d3820 0x7f35480b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.641+0000 7f3547200640 1 --2- 172.21.15.40:0/3285131087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35480d08d0 0x7f35480d32e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.641+0000 7f3547200640 1 --2- 172.21.15.40:0/3285131087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35480b6220 0x7f35480d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.641+0000 7f3547200640 1 -- 172.21.15.40:0/3285131087 >> 172.21.15.40:0/3285131087 conn(0x7f35480d4d90 msgr2=0x7f35480d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.641+0000 7f3547200640 1 -- 172.21.15.40:0/3285131087 shutdown_connections 2024-04-26T23:02:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.641+0000 7f3547200640 1 -- 172.21.15.40:0/3285131087 wait complete. 2024-04-26T23:02:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.642+0000 7f3547200640 1 Processor -- start 2024-04-26T23:02:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.643+0000 7f3547200640 1 -- start start 2024-04-26T23:02:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f3547200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35480b6220 0x7f35481f8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f3547200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35480d08d0 0x7f35480b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f3547200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35480d3820 0x7f3548161130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f3547200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f35480b6d50 con 0x7f35480d3820 2024-04-26T23:02:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f3547200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f35480b8520 con 0x7f35480b6220 2024-04-26T23:02:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f3547200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f35480b6ed0 con 0x7f35480d08d0 2024-04-26T23:02:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f3546800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35480d3820 0x7f3548161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f3546800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35480d3820 0x7f3548161130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45278/0 (socket says 172.21.15.40:45278) 2024-04-26T23:02:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f3546800640 1 -- 172.21.15.40:0/383838590 learned_addr learned my addr 172.21.15.40:0/383838590 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f3545e00640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35480b6220 0x7f35481f8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f353fe00640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35480d08d0 0x7f35480b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f3546800640 1 -- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35480d08d0 msgr2=0x7f35480b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f3546800640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35480d08d0 0x7f35480b5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.644+0000 7f3546800640 1 -- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35480b6220 msgr2=0x7f35481f8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.645+0000 7f3546800640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35480b6220 0x7f35481f8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.645+0000 7f3546800640 1 -- 172.21.15.40:0/383838590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35481f8f60 con 0x7f35480d3820 2024-04-26T23:02:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.645+0000 7f353fe00640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35480d08d0 0x7f35480b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.645+0000 7f3546800640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35480d3820 0x7f3548161130 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3540007e60 tx=0x7f354000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.645+0000 7f353e000640 1 -- 172.21.15.40:0/383838590 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3540014420 con 0x7f35480d3820 2024-04-26T23:02:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.645+0000 7f353e000640 1 -- 172.21.15.40:0/383838590 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3540019070 con 0x7f35480d3820 2024-04-26T23:02:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.645+0000 7f353e000640 1 -- 172.21.15.40:0/383838590 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3540014820 con 0x7f35480d3820 2024-04-26T23:02:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.645+0000 7f3547200640 1 -- 172.21.15.40:0/383838590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35481f86c0 con 0x7f35480d3820 2024-04-26T23:02:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.645+0000 7f3547200640 1 -- 172.21.15.40:0/383838590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35481f8c80 con 0x7f35480d3820 2024-04-26T23:02:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.646+0000 7f3547200640 1 -- 172.21.15.40:0/383838590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3548069370 con 0x7f35480d3820 2024-04-26T23:02:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.647+0000 7f353e000640 1 -- 172.21.15.40:0/383838590 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f35400149c0 con 0x7f35480d3820 2024-04-26T23:02:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.648+0000 7f353e000640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f352406c8f0 0x7f352406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.649+0000 7f353e000640 1 -- 172.21.15.40:0/383838590 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f354008c7c0 con 0x7f35480d3820 2024-04-26T23:02:26.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.649+0000 7f3545e00640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f352406c8f0 0x7f352406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:26.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.650+0000 7f3545e00640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f352406c8f0 0x7f352406edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3538006fd0 tx=0x7f3538008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:26.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.653+0000 7f353e000640 1 -- 172.21.15.40:0/383838590 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f354005a7e0 con 0x7f35480d3820 2024-04-26T23:02:26.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.857+0000 7f3547200640 1 -- 172.21.15.40:0/383838590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f35480d30c0 con 0x7f35480d3820 2024-04-26T23:02:26.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.860+0000 7f353e000640 1 -- 172.21.15.40:0/383838590 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18812 (secure 0 0 0) 0x7f354005a0e0 con 0x7f35480d3820 2024-04-26T23:02:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:26.861 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:46.975683+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:26.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.863+0000 7f3547200640 1 -- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f352406c8f0 msgr2=0x7f352406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:26.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.863+0000 7f3547200640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f352406c8f0 0x7f352406edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3538006fd0 tx=0x7f3538008040 comp rx=0 tx=0).stop 2024-04-26T23:02:26.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.863+0000 7f3547200640 1 -- 172.21.15.40:0/383838590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35480d3820 msgr2=0x7f3548161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:26.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.863+0000 7f3547200640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35480d3820 0x7f3548161130 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3540007e60 tx=0x7f354000e4b0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.863+0000 7f3547200640 1 -- 172.21.15.40:0/383838590 shutdown_connections 2024-04-26T23:02:26.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.863+0000 7f3547200640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f352406c8f0 0x7f352406edb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.863+0000 7f3547200640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35480d3820 0x7f3548161130 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.863+0000 7f3547200640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35480d08d0 0x7f35480b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.863+0000 7f3547200640 1 --2- 172.21.15.40:0/383838590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35480b6220 0x7f35481f8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:26.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.863+0000 7f3547200640 1 -- 172.21.15.40:0/383838590 >> 172.21.15.40:0/383838590 conn(0x7f35480d4d90 msgr2=0x7f354806dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:26.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.864+0000 7f3547200640 1 -- 172.21.15.40:0/383838590 shutdown_connections 2024-04-26T23:02:26.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:26.864+0000 7f3547200640 1 -- 172.21.15.40:0/383838590 wait complete. 2024-04-26T23:02:26.884 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:02:26.884 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:02:27.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.100+0000 7fbbefe00640 1 Processor -- start 2024-04-26T23:02:27.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.100+0000 7fbbefe00640 1 -- start start 2024-04-26T23:02:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.102+0000 7fbbefe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbbf00cfaf0 0x7fbbf0073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.102+0000 7fbbefe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbf0161880 con 0x7fbbf0160a50 2024-04-26T23:02:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.102+0000 7fbbefe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbf00cff80 con 0x7fbbf006dd40 2024-04-26T23:02:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.102+0000 7fbbefe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbbf0071ce0 con 0x7fbbf00cfaf0 2024-04-26T23:02:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.103+0000 7fbbeea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbbf0160a50 0x7fbbf0160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35774/0 (socket says 172.21.15.40:35774) 2024-04-26T23:02:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.103+0000 7fbbeea00640 1 -- 172.21.15.40:0/646482682 learned_addr learned my addr 172.21.15.40:0/646482682 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.103+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3569144384 0 0) 0x7fbbf0161880 con 0x7fbbf0160a50 2024-04-26T23:02:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.103+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbd4003520 con 0x7fbbf0160a50 2024-04-26T23:02:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.103+0000 7fbbe6000640 1 --2- 172.21.15.40:0/646482682 >> v2:172.21.15.103:3300/0 conn(0x7fbbf00cfaf0 0x7fbbf0073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3776536522 0 0) 0x7fbbd4003520 con 0x7fbbf0160a50 2024-04-26T23:02:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 >> v2:172.21.15.103:3300/0 conn(0x7fbbf00cfaf0 msgr2=0x7fbbf0073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbee000640 1 --2- 172.21.15.40:0/646482682 >> v2:172.21.15.103:3300/0 conn(0x7fbbf00cfaf0 0x7fbbf0073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 >> v1:172.21.15.103:6789/0 conn(0x7fbbf006dd40 legacy=0x7fbbf0072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf00d0260 con 0x7fbbf0160a50 2024-04-26T23:02:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fbbe0001670 con 0x7fbbf0160a50 2024-04-26T23:02:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 >> v1:172.21.15.40:6789/0 conn(0x7fbbf0160a50 legacy=0x7fbbf0160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbee000640 1 --2- 172.21.15.40:0/646482682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbd40039b0 0x7fbbd4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbee000640 1 --2- 172.21.15.40:0/646482682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbd4004800 0x7fbbd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbee000640 1 --2- 172.21.15.40:0/646482682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbd40051c0 0x7fbbd400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf0071ce0 con 0x7fbbd4004800 2024-04-26T23:02:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd400c110 con 0x7fbbd40039b0 2024-04-26T23:02:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbbd400c310 con 0x7fbbd40051c0 2024-04-26T23:02:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbe6000640 1 --2- 172.21.15.40:0/646482682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbd4004800 0x7fbbd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbe6000640 1 -- 172.21.15.40:0/646482682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbd40051c0 msgr2=0x7fbbd400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbeea00640 1 --2- 172.21.15.40:0/646482682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbd40039b0 0x7fbbd4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbe6000640 1 --2- 172.21.15.40:0/646482682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbd40051c0 0x7fbbd400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.104+0000 7fbbe6000640 1 -- 172.21.15.40:0/646482682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbd40039b0 msgr2=0x7fbbd4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.105+0000 7fbbe6000640 1 --2- 172.21.15.40:0/646482682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbd40039b0 0x7fbbd4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.105+0000 7fbbe6000640 1 -- 172.21.15.40:0/646482682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd4003520 con 0x7fbbd4004800 2024-04-26T23:02:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.105+0000 7fbbef400640 1 -- 172.21.15.40:0/646482682 reap_dead start 2024-04-26T23:02:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.105+0000 7fbbeea00640 1 --2- 172.21.15.40:0/646482682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbd40039b0 0x7fbbd4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.106+0000 7fbbe6000640 1 --2- 172.21.15.40:0/646482682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbd4004800 0x7fbbd4004c80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800ba30 tx=0x7fbbd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b3659bd2de0142 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.106+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8030070 con 0x7fbbd4004800 2024-04-26T23:02:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.106+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbd802ae00 con 0x7fbbd4004800 2024-04-26T23:02:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.106+0000 7fbbee000640 1 -- 172.21.15.40:0/646482682 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd800b600 con 0x7fbbd4004800 2024-04-26T23:02:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.107+0000 7fbbefe00640 1 -- 172.21.15.40:0/646482682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbd4004800 msgr2=0x7fbbd4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.107+0000 7fbbefe00640 1 --2- 172.21.15.40:0/646482682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbd4004800 0x7fbbd4004c80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800ba30 tx=0x7fbbd4005830 comp rx=0 tx=0).stop 2024-04-26T23:02:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.107+0000 7fbbefe00640 1 -- 172.21.15.40:0/646482682 shutdown_connections 2024-04-26T23:02:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.107+0000 7fbbefe00640 1 --2- 172.21.15.40:0/646482682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbd4004800 0x7fbbd4004c80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.107+0000 7fbbefe00640 1 -- 172.21.15.40:0/646482682 >> 172.21.15.40:0/646482682 conn(0x7fbbf00b2720 msgr2=0x7fbbf00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.107+0000 7fbbefe00640 1 -- 172.21.15.40:0/646482682 shutdown_connections 2024-04-26T23:02:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.107+0000 7fbbefe00640 1 -- 172.21.15.40:0/646482682 wait complete. 2024-04-26T23:02:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.109+0000 7fbbefe00640 1 Processor -- start 2024-04-26T23:02:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.109+0000 7fbbefe00640 1 -- start start 2024-04-26T23:02:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.110+0000 7fbbefe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbd4004800 0x7fbbf01342c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.110+0000 7fbbefe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbf013e510 0x7fbbf013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.110+0000 7fbbefe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf0144fb0 0x7fbbf0145430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.110+0000 7fbbefe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf006e1f0 con 0x7fbbd4004800 2024-04-26T23:02:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.110+0000 7fbbefe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf00722a0 con 0x7fbbf013e510 2024-04-26T23:02:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.110+0000 7fbbefe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf01468f0 con 0x7fbbf0144fb0 2024-04-26T23:02:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.110+0000 7fbbeea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbd4004800 0x7fbbf01342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.110+0000 7fbbeea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbd4004800 0x7fbbf01342c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45282/0 (socket says 172.21.15.40:45282) 2024-04-26T23:02:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.110+0000 7fbbeea00640 1 -- 172.21.15.40:0/1284143285 learned_addr learned my addr 172.21.15.40:0/1284143285 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.110+0000 7fbbeea00640 1 -- 172.21.15.40:0/1284143285 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf0144fb0 msgr2=0x7fbbf0145430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.111+0000 7fbbeea00640 1 --2- 172.21.15.40:0/1284143285 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf0144fb0 0x7fbbf0145430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.111+0000 7fbbeea00640 1 -- 172.21.15.40:0/1284143285 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbf013e510 msgr2=0x7fbbf013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.111+0000 7fbbeea00640 1 --2- 172.21.15.40:0/1284143285 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbf013e510 0x7fbbf013ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.111+0000 7fbbeea00640 1 -- 172.21.15.40:0/1284143285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf0134800 con 0x7fbbd4004800 2024-04-26T23:02:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.111+0000 7fbbeea00640 1 --2- 172.21.15.40:0/1284143285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbd4004800 0x7fbbf01342c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbbe000d7d0 tx=0x7fbbe000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.111+0000 7fbbe7400640 1 -- 172.21.15.40:0/1284143285 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0019070 con 0x7fbbd4004800 2024-04-26T23:02:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.111+0000 7fbbe7400640 1 -- 172.21.15.40:0/1284143285 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe00074e0 con 0x7fbbd4004800 2024-04-26T23:02:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.111+0000 7fbbe7400640 1 -- 172.21.15.40:0/1284143285 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0004a20 con 0x7fbbd4004800 2024-04-26T23:02:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.111+0000 7fbbefe00640 1 -- 172.21.15.40:0/1284143285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf013f520 con 0x7fbbd4004800 2024-04-26T23:02:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.111+0000 7fbbefe00640 1 -- 172.21.15.40:0/1284143285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf013fae0 con 0x7fbbd4004800 2024-04-26T23:02:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.112+0000 7fbbefe00640 1 -- 172.21.15.40:0/1284143285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbf0069370 con 0x7fbbd4004800 2024-04-26T23:02:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.114+0000 7fbbe7400640 1 -- 172.21.15.40:0/1284143285 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbbe0004bc0 con 0x7fbbd4004800 2024-04-26T23:02:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.114+0000 7fbbe7400640 1 --2- 172.21.15.40:0/1284143285 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbbcc06c8f0 0x7fbbcc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.115+0000 7fbbe7400640 1 -- 172.21.15.40:0/1284143285 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbbe008b9f0 con 0x7fbbd4004800 2024-04-26T23:02:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.115+0000 7fbbe6000640 1 --2- 172.21.15.40:0/1284143285 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbbcc06c8f0 0x7fbbcc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.116+0000 7fbbe6000640 1 --2- 172.21.15.40:0/1284143285 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbbcc06c8f0 0x7fbbcc06edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800ba00 tx=0x7fbbd802c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.119+0000 7fbbe7400640 1 -- 172.21.15.40:0/1284143285 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fbbe0059980 con 0x7fbbd4004800 2024-04-26T23:02:27.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.323+0000 7fbbefe00640 1 -- 172.21.15.40:0/1284143285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbbf006fa60 con 0x7fbbd4004800 2024-04-26T23:02:27.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.324+0000 7fbbe7400640 1 -- 172.21.15.40:0/1284143285 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fbbe0059280 con 0x7fbbd4004800 2024-04-26T23:02:27.324 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:02:27.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.326+0000 7fbbefe00640 1 -- 172.21.15.40:0/1284143285 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbbcc06c8f0 msgr2=0x7fbbcc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.326+0000 7fbbefe00640 1 --2- 172.21.15.40:0/1284143285 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbbcc06c8f0 0x7fbbcc06edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800ba00 tx=0x7fbbd802c000 comp rx=0 tx=0).stop 2024-04-26T23:02:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.326+0000 7fbbefe00640 1 -- 172.21.15.40:0/1284143285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbd4004800 msgr2=0x7fbbf01342c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.326+0000 7fbbefe00640 1 --2- 172.21.15.40:0/1284143285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbd4004800 0x7fbbf01342c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbbe000d7d0 tx=0x7fbbe000dc90 comp rx=0 tx=0).stop 2024-04-26T23:02:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.326+0000 7fbbefe00640 1 -- 172.21.15.40:0/1284143285 shutdown_connections 2024-04-26T23:02:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.326+0000 7fbbefe00640 1 --2- 172.21.15.40:0/1284143285 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbbcc06c8f0 0x7fbbcc06edb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.326+0000 7fbbefe00640 1 --2- 172.21.15.40:0/1284143285 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbbf0144fb0 0x7fbbf0145430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.326+0000 7fbbefe00640 1 --2- 172.21.15.40:0/1284143285 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbf013e510 0x7fbbf013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.326+0000 7fbbefe00640 1 --2- 172.21.15.40:0/1284143285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbd4004800 0x7fbbf01342c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.327+0000 7fbbefe00640 1 -- 172.21.15.40:0/1284143285 >> 172.21.15.40:0/1284143285 conn(0x7fbbf00b2720 msgr2=0x7fbbf006e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:27.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.327+0000 7fbbefe00640 1 -- 172.21.15.40:0/1284143285 shutdown_connections 2024-04-26T23:02:27.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.327+0000 7fbbefe00640 1 -- 172.21.15.40:0/1284143285 wait complete. 2024-04-26T23:02:27.346 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:02:27.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.549+0000 7f27a6a00640 1 Processor -- start 2024-04-26T23:02:27.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.549+0000 7f27a6a00640 1 -- start start 2024-04-26T23:02:27.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.551+0000 7f27a6a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f27a0073a50 0x7f27a0072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.551+0000 7f27a6a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27a0161880 con 0x7f27a00cfaf0 2024-04-26T23:02:27.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.551+0000 7f27a6a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27a00cff80 con 0x7f27a0160a50 2024-04-26T23:02:27.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.551+0000 7f27a6a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f27a0071ce0 con 0x7f27a0073a50 2024-04-26T23:02:27.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.552+0000 7f27a4c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f27a00cfaf0 0x7f27a006dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35782/0 (socket says 172.21.15.40:35782) 2024-04-26T23:02:27.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.552+0000 7f27a4c00640 1 -- 172.21.15.40:0/943157481 learned_addr learned my addr 172.21.15.40:0/943157481 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:27.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.552+0000 7f27a6000640 1 --2- 172.21.15.40:0/943157481 >> v2:172.21.15.103:3300/0 conn(0x7f27a0073a50 0x7f27a0072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:27.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.552+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1479898656 0 0) 0x7f27a0161880 con 0x7f27a00cfaf0 2024-04-26T23:02:27.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.552+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2788003520 con 0x7f27a00cfaf0 2024-04-26T23:02:27.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1986004150 0 0) 0x7f2788003520 con 0x7f27a00cfaf0 2024-04-26T23:02:27.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 >> v2:172.21.15.103:3300/0 conn(0x7f27a0073a50 msgr2=0x7f27a0072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f279fe00640 1 --2- 172.21.15.40:0/943157481 >> v2:172.21.15.103:3300/0 conn(0x7f27a0073a50 0x7f27a0072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 >> v1:172.21.15.103:6789/0 conn(0x7f27a0160a50 legacy=0x7f27a0160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a00d0260 con 0x7f27a00cfaf0 2024-04-26T23:02:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f278c001670 con 0x7f27a00cfaf0 2024-04-26T23:02:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 >> v1:172.21.15.40:6789/0 conn(0x7f27a00cfaf0 legacy=0x7f27a006dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f279fe00640 1 --2- 172.21.15.40:0/943157481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2788003a00 0x7f2788003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f279fe00640 1 --2- 172.21.15.40:0/943157481 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2788004800 0x7f2788004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f279fe00640 1 --2- 172.21.15.40:0/943157481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27880051c0 0x7f278800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f27a0071ce0 con 0x7f27880051c0 2024-04-26T23:02:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f278800c0c0 con 0x7f2788003a00 2024-04-26T23:02:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f278800c2c0 con 0x7f2788004800 2024-04-26T23:02:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.553+0000 7f27a6000640 1 --2- 172.21.15.40:0/943157481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27880051c0 0x7f278800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.554+0000 7f27a4c00640 1 --2- 172.21.15.40:0/943157481 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2788004800 0x7f2788004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:27.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.554+0000 7f27a5600640 1 --2- 172.21.15.40:0/943157481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2788003a00 0x7f2788003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:27.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.554+0000 7f27a6000640 1 -- 172.21.15.40:0/943157481 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2788004800 msgr2=0x7f2788004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:27.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.554+0000 7f27a6000640 1 --2- 172.21.15.40:0/943157481 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2788004800 0x7f2788004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.554+0000 7f27a6000640 1 -- 172.21.15.40:0/943157481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2788003a00 msgr2=0x7f2788003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:27.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.554+0000 7f27a6000640 1 --2- 172.21.15.40:0/943157481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2788003a00 0x7f2788003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.554+0000 7f27a6000640 1 -- 172.21.15.40:0/943157481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2788003520 con 0x7f27880051c0 2024-04-26T23:02:27.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.554+0000 7f27a5600640 1 --2- 172.21.15.40:0/943157481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2788003a00 0x7f2788003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:27.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.555+0000 7f27a6000640 1 -- 172.21.15.40:0/943157481 reap_dead start 2024-04-26T23:02:27.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.555+0000 7f27a6000640 1 --2- 172.21.15.40:0/943157481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27880051c0 0x7f278800b9c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2798009fc0 tx=0x7f2788005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6812a1a49e1b9a0c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:27.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.555+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2798035020 con 0x7f27880051c0 2024-04-26T23:02:27.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.555+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2798003bd0 con 0x7f27880051c0 2024-04-26T23:02:27.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.555+0000 7f279fe00640 1 -- 172.21.15.40:0/943157481 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2798009c10 con 0x7f27880051c0 2024-04-26T23:02:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.556+0000 7f27a6a00640 1 -- 172.21.15.40:0/943157481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27880051c0 msgr2=0x7f278800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.556+0000 7f27a6a00640 1 --2- 172.21.15.40:0/943157481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27880051c0 0x7f278800b9c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2798009fc0 tx=0x7f2788005830 comp rx=0 tx=0).stop 2024-04-26T23:02:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.556+0000 7f27a6a00640 1 -- 172.21.15.40:0/943157481 shutdown_connections 2024-04-26T23:02:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.556+0000 7f27a6a00640 1 --2- 172.21.15.40:0/943157481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27880051c0 0x7f278800b9c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.556+0000 7f27a6a00640 1 -- 172.21.15.40:0/943157481 >> 172.21.15.40:0/943157481 conn(0x7f27a00b2720 msgr2=0x7f27a00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.556+0000 7f27a6a00640 1 -- 172.21.15.40:0/943157481 shutdown_connections 2024-04-26T23:02:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.556+0000 7f27a6a00640 1 -- 172.21.15.40:0/943157481 wait complete. 2024-04-26T23:02:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.558+0000 7f27a6a00640 1 Processor -- start 2024-04-26T23:02:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.558+0000 7f27a6a00640 1 -- start start 2024-04-26T23:02:27.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.559+0000 7f27a6a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27880051c0 0x7f27a01402d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.559+0000 7f27a6a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a015dd70 0x7f27a0144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.559+0000 7f27a6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015e1f0 0x7f27a0144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.559+0000 7f27a6a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f27a0073f00 con 0x7f27a015e1f0 2024-04-26T23:02:27.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.559+0000 7f27a6a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f27a00722a0 con 0x7f27a015dd70 2024-04-26T23:02:27.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.559+0000 7f27a6a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f27a01468f0 con 0x7f27880051c0 2024-04-26T23:02:27.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.559+0000 7f27a6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015e1f0 0x7f27a0144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:27.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.560+0000 7f27a6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015e1f0 0x7f27a0144ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45306/0 (socket says 172.21.15.40:45306) 2024-04-26T23:02:27.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.560+0000 7f27a6000640 1 -- 172.21.15.40:0/3818288196 learned_addr learned my addr 172.21.15.40:0/3818288196 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:27.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.560+0000 7f27a5600640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27880051c0 0x7f27a01402d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:27.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.560+0000 7f27a4c00640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a015dd70 0x7f27a0144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:27.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.560+0000 7f27a6000640 1 -- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27880051c0 msgr2=0x7f27a01402d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.560+0000 7f27a6000640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27880051c0 0x7f27a01402d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.560+0000 7f27a6000640 1 -- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a015dd70 msgr2=0x7f27a0144960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.560+0000 7f27a6000640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a015dd70 0x7f27a0144960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.560+0000 7f27a6000640 1 -- 172.21.15.40:0/3818288196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a015e7c0 con 0x7f27a015e1f0 2024-04-26T23:02:27.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.560+0000 7f27a6000640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015e1f0 0x7f27a0144ea0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f279802af10 tx=0x7f2798009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:27.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.561+0000 7f279d600640 1 -- 172.21.15.40:0/3818288196 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2798035070 con 0x7f27a015e1f0 2024-04-26T23:02:27.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.561+0000 7f279d600640 1 -- 172.21.15.40:0/3818288196 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2798042070 con 0x7f27a015e1f0 2024-04-26T23:02:27.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.561+0000 7f279d600640 1 -- 172.21.15.40:0/3818288196 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279802d070 con 0x7f27a015e1f0 2024-04-26T23:02:27.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.561+0000 7f27a6a00640 1 -- 172.21.15.40:0/3818288196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a015eaa0 con 0x7f27a015e1f0 2024-04-26T23:02:27.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.561+0000 7f27a6a00640 1 -- 172.21.15.40:0/3818288196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a01428b0 con 0x7f27a015e1f0 2024-04-26T23:02:27.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.562+0000 7f27a6a00640 1 -- 172.21.15.40:0/3818288196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27a0142ab0 con 0x7f27a015e1f0 2024-04-26T23:02:27.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.563+0000 7f279d600640 1 -- 172.21.15.40:0/3818288196 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2798005d10 con 0x7f27a015e1f0 2024-04-26T23:02:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.564+0000 7f279d600640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f278006c8f0 0x7f278006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.564+0000 7f279d600640 1 -- 172.21.15.40:0/3818288196 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f27980acd70 con 0x7f27a015e1f0 2024-04-26T23:02:27.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.565+0000 7f27a5600640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f278006c8f0 0x7f278006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.567+0000 7f27a5600640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f278006c8f0 0x7f278006edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2794000c00 tx=0x7f2794009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:27.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.569+0000 7f279d600640 1 -- 172.21.15.40:0/3818288196 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f27a0142ab0 con 0x7f27a015e1f0 2024-04-26T23:02:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.771+0000 7f27a6a00640 1 -- 172.21.15.40:0/3818288196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f27a0069370 con 0x7f27a015e1f0 2024-04-26T23:02:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.771+0000 7f279d600640 1 -- 172.21.15.40:0/3818288196 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f279807ad00 con 0x7f27a015e1f0 2024-04-26T23:02:27.772 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:27.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.774+0000 7f27a6a00640 1 -- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f278006c8f0 msgr2=0x7f278006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.774+0000 7f27a6a00640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f278006c8f0 0x7f278006edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2794000c00 tx=0x7f2794009040 comp rx=0 tx=0).stop 2024-04-26T23:02:27.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.774+0000 7f27a6a00640 1 -- 172.21.15.40:0/3818288196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015e1f0 msgr2=0x7f27a0144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:27.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.774+0000 7f27a6a00640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015e1f0 0x7f27a0144ea0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f279802af10 tx=0x7f2798009750 comp rx=0 tx=0).stop 2024-04-26T23:02:27.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.774+0000 7f27a6a00640 1 -- 172.21.15.40:0/3818288196 shutdown_connections 2024-04-26T23:02:27.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.774+0000 7f27a6a00640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f278006c8f0 0x7f278006edb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.774+0000 7f27a6a00640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a015e1f0 0x7f27a0144ea0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.774+0000 7f27a6a00640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a015dd70 0x7f27a0144960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.774+0000 7f27a6a00640 1 --2- 172.21.15.40:0/3818288196 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f27880051c0 0x7f27a01402d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:27.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.774+0000 7f27a6a00640 1 -- 172.21.15.40:0/3818288196 >> 172.21.15.40:0/3818288196 conn(0x7f27a00b2720 msgr2=0x7f27a0140fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:27.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.774+0000 7f27a6a00640 1 -- 172.21.15.40:0/3818288196 shutdown_connections 2024-04-26T23:02:27.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:27.774+0000 7f27a6a00640 1 -- 172.21.15.40:0/3818288196 wait complete. 2024-04-26T23:02:27.794 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:02:28.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.013+0000 7f5d0ce00640 1 Processor -- start 2024-04-26T23:02:28.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.013+0000 7f5d0ce00640 1 -- start start 2024-04-26T23:02:28.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.014+0000 7f5d0ce00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d0806f790 0x7f5d0806fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.014+0000 7f5d0ce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d08072c50 0x7f5d080730a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.014+0000 7f5d0ce00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d08156840 con 0x7f5d08156460 2024-04-26T23:02:28.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.015+0000 7f5d0ce00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f5d080cbf20 con 0x7f5d0806f790 2024-04-26T23:02:28.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.015+0000 7f5d0ce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5d080cc200 con 0x7f5d08072c50 2024-04-26T23:02:28.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.015+0000 7f5d05400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d0806f790 0x7f5d0806fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.015+0000 7f5d05400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d0806f790 0x7f5d0806fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45316/0 (socket says 172.21.15.40:45316) 2024-04-26T23:02:28.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.015+0000 7f5d05400640 1 -- 172.21.15.40:0/3970793374 learned_addr learned my addr 172.21.15.40:0/3970793374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:28.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.015+0000 7f5d06800640 1 --2- 172.21.15.40:0/3970793374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d08072c50 0x7f5d080730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.015+0000 7f5d05400640 1 -- 172.21.15.40:0/3970793374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d08072c50 msgr2=0x7f5d080730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:28.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.015+0000 7f5d05400640 1 --2- 172.21.15.40:0/3970793374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d08072c50 0x7f5d080730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.015+0000 7f5d05400640 1 -- 172.21.15.40:0/3970793374 >> v1:172.21.15.103:6789/0 conn(0x7f5d08156460 legacy=0x7f5d080cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:28.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.015+0000 7f5d05400640 1 -- 172.21.15.40:0/3970793374 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d080cc4e0 con 0x7f5d0806f790 2024-04-26T23:02:28.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.015+0000 7f5d06800640 1 --2- 172.21.15.40:0/3970793374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d08072c50 0x7f5d080730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:28.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.016+0000 7f5d05400640 1 --2- 172.21.15.40:0/3970793374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d0806f790 0x7f5d0806fc40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009870 tx=0x7f5cf002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce2035001084aff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:28.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.017+0000 7f5d04a00640 1 -- 172.21.15.40:0/3970793374 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0037070 con 0x7f5d0806f790 2024-04-26T23:02:28.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.017+0000 7f5d04a00640 1 -- 172.21.15.40:0/3970793374 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf0032440 con 0x7f5d0806f790 2024-04-26T23:02:28.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.017+0000 7f5d04a00640 1 -- 172.21.15.40:0/3970793374 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0032810 con 0x7f5d0806f790 2024-04-26T23:02:28.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.017+0000 7f5d0ce00640 1 -- 172.21.15.40:0/3970793374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d0806f790 msgr2=0x7f5d0806fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.017+0000 7f5d0ce00640 1 --2- 172.21.15.40:0/3970793374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d0806f790 0x7f5d0806fc40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009870 tx=0x7f5cf002a200 comp rx=0 tx=0).stop 2024-04-26T23:02:28.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.017+0000 7f5d0ce00640 1 -- 172.21.15.40:0/3970793374 shutdown_connections 2024-04-26T23:02:28.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.017+0000 7f5d0ce00640 1 --2- 172.21.15.40:0/3970793374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d08072c50 0x7f5d080730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.017+0000 7f5d0ce00640 1 --2- 172.21.15.40:0/3970793374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d0806f790 0x7f5d0806fc40 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.017+0000 7f5d0ce00640 1 -- 172.21.15.40:0/3970793374 >> 172.21.15.40:0/3970793374 conn(0x7f5d080d0be0 msgr2=0x7f5d080d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:28.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.018+0000 7f5d0ce00640 1 -- 172.21.15.40:0/3970793374 shutdown_connections 2024-04-26T23:02:28.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.018+0000 7f5d0ce00640 1 -- 172.21.15.40:0/3970793374 wait complete. 2024-04-26T23:02:28.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.019+0000 7f5d0ce00640 1 Processor -- start 2024-04-26T23:02:28.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.019+0000 7f5d0ce00640 1 -- start start 2024-04-26T23:02:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.020+0000 7f5d0ce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0806f790 0x7f5d08153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.020+0000 7f5d0ce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08072c50 0x7f5d080b4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.020+0000 7f5d0ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08156460 0x7f5d080b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.020+0000 7f5d0ce00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5d080cbbe0 con 0x7f5d08156460 2024-04-26T23:02:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.020+0000 7f5d0ce00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5d08072700 con 0x7f5d08072c50 2024-04-26T23:02:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.020+0000 7f5d0ce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5d0806e260 con 0x7f5d0806f790 2024-04-26T23:02:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.021+0000 7f5d06800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08156460 0x7f5d080b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.021+0000 7f5d06800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08156460 0x7f5d080b3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45322/0 (socket says 172.21.15.40:45322) 2024-04-26T23:02:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.021+0000 7f5d06800640 1 -- 172.21.15.40:0/900900594 learned_addr learned my addr 172.21.15.40:0/900900594 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.021+0000 7f5d05e00640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0806f790 0x7f5d08153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.021+0000 7f5d06800640 1 -- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0806f790 msgr2=0x7f5d08153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.021+0000 7f5d06800640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0806f790 0x7f5d08153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.021+0000 7f5d05400640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08072c50 0x7f5d080b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.021+0000 7f5d06800640 1 -- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08072c50 msgr2=0x7f5d080b4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.021+0000 7f5d06800640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08072c50 0x7f5d080b4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.021+0000 7f5d06800640 1 -- 172.21.15.40:0/900900594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d080c8df0 con 0x7f5d08156460 2024-04-26T23:02:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.021+0000 7f5d05e00640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0806f790 0x7f5d08153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.021+0000 7f5d05400640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08072c50 0x7f5d080b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:28.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.022+0000 7f5d06800640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08156460 0x7f5d080b3860 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000c7f0 tx=0x7f5d0000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:28.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.022+0000 7f5cfe000640 1 -- 172.21.15.40:0/900900594 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d00015070 con 0x7f5d08156460 2024-04-26T23:02:28.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.022+0000 7f5cfe000640 1 -- 172.21.15.40:0/900900594 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d00007820 con 0x7f5d08156460 2024-04-26T23:02:28.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.022+0000 7f5cfe000640 1 -- 172.21.15.40:0/900900594 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d00007c50 con 0x7f5d08156460 2024-04-26T23:02:28.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.022+0000 7f5d0ce00640 1 -- 172.21.15.40:0/900900594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d080c90d0 con 0x7f5d08156460 2024-04-26T23:02:28.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.022+0000 7f5d0ce00640 1 -- 172.21.15.40:0/900900594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d080c9690 con 0x7f5d08156460 2024-04-26T23:02:28.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.023+0000 7f5d0ce00640 1 -- 172.21.15.40:0/900900594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d080694f0 con 0x7f5d08156460 2024-04-26T23:02:28.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.024+0000 7f5cfe000640 1 -- 172.21.15.40:0/900900594 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d00018070 con 0x7f5d08156460 2024-04-26T23:02:28.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.026+0000 7f5cfe000640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ce406c8f0 0x7f5ce406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.026+0000 7f5cfe000640 1 -- 172.21.15.40:0/900900594 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f5d0008c0d0 con 0x7f5d08156460 2024-04-26T23:02:28.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.026+0000 7f5d05e00640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ce406c8f0 0x7f5ce406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.028+0000 7f5d05e00640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ce406c8f0 0x7f5ce406edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8007e90 tx=0x7f5cf80073a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:28.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.030+0000 7f5cfe000640 1 -- 172.21.15.40:0/900900594 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5d0005a0f0 con 0x7f5d08156460 2024-04-26T23:02:28.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.235+0000 7f5d0ce00640 1 -- 172.21.15.40:0/900900594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5d0815ef50 con 0x7f5d08156460 2024-04-26T23:02:28.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.236+0000 7f5cfe000640 1 -- 172.21.15.40:0/900900594 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f5d000599f0 con 0x7f5d08156460 2024-04-26T23:02:28.236 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:02:28.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.238+0000 7f5d0ce00640 1 -- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ce406c8f0 msgr2=0x7f5ce406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.238+0000 7f5d0ce00640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ce406c8f0 0x7f5ce406edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8007e90 tx=0x7f5cf80073a0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.238+0000 7f5d0ce00640 1 -- 172.21.15.40:0/900900594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08156460 msgr2=0x7f5d080b3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.238+0000 7f5d0ce00640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08156460 0x7f5d080b3860 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000c7f0 tx=0x7f5d0000ccb0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.238+0000 7f5d0ce00640 1 -- 172.21.15.40:0/900900594 shutdown_connections 2024-04-26T23:02:28.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.238+0000 7f5d0ce00640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ce406c8f0 0x7f5ce406edb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.238+0000 7f5d0ce00640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08156460 0x7f5d080b3860 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.238+0000 7f5d0ce00640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08072c50 0x7f5d080b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.238+0000 7f5d0ce00640 1 --2- 172.21.15.40:0/900900594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d0806f790 0x7f5d08153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.238+0000 7f5d0ce00640 1 -- 172.21.15.40:0/900900594 >> 172.21.15.40:0/900900594 conn(0x7f5d080d0be0 msgr2=0x7f5d08155590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:28.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.238+0000 7f5d0ce00640 1 -- 172.21.15.40:0/900900594 shutdown_connections 2024-04-26T23:02:28.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.238+0000 7f5d0ce00640 1 -- 172.21.15.40:0/900900594 wait complete. 2024-04-26T23:02:28.257 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:02:28.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.462+0000 7f707d600640 1 Processor -- start 2024-04-26T23:02:28.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.463+0000 7f707d600640 1 -- start start 2024-04-26T23:02:28.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.464+0000 7f707d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7078160a50 0x7f7078073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.464+0000 7f707d600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f70780cfaf0 0x7f7078160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.465+0000 7f707d600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7078161880 con 0x7f707806dd40 2024-04-26T23:02:28.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.465+0000 7f707d600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7078071ce0 con 0x7f70780cfaf0 2024-04-26T23:02:28.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.465+0000 7f707d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f70780722a0 con 0x7f7078160a50 2024-04-26T23:02:28.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.465+0000 7f7077e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7078160a50 0x7f7078073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.465+0000 7f707cc00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f707806dd40 0x7f7078072730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:33078/0 (socket says 172.21.15.40:33078) 2024-04-26T23:02:28.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.466+0000 7f7077e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7078160a50 0x7f7078073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:50884/0 (socket says 172.21.15.40:50884) 2024-04-26T23:02:28.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.465+0000 7f706fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f70780cfaf0 0x7f7078160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.466+0000 7f707cc00640 1 -- 172.21.15.40:0/408140309 learned_addr learned my addr 172.21.15.40:0/408140309 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:28.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.466+0000 7f7077e00640 1 -- 172.21.15.40:0/408140309 >> v2:172.21.15.103:3300/0 conn(0x7f70780cfaf0 msgr2=0x7f7078160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:28.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.466+0000 7f7077e00640 1 --2- 172.21.15.40:0/408140309 >> v2:172.21.15.103:3300/0 conn(0x7f70780cfaf0 0x7f7078160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.466+0000 7f7077e00640 1 -- 172.21.15.40:0/408140309 >> v1:172.21.15.103:6789/0 conn(0x7f707806dd40 legacy=0x7f7078072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.466+0000 7f7077e00640 1 -- 172.21.15.40:0/408140309 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7078161370 con 0x7f7078160a50 2024-04-26T23:02:28.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.466+0000 7f706fe00640 1 --2- 172.21.15.40:0/408140309 >> v2:172.21.15.103:3300/0 conn(0x7f70780cfaf0 0x7f7078160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:02:28.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.468+0000 7f7077e00640 1 --2- 172.21.15.40:0/408140309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7078160a50 0x7f7078073a50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f70680097a0 tx=0x7f706802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fef8c3b9801457cb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:28.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.468+0000 7f7077400640 1 -- 172.21.15.40:0/408140309 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068037070 con 0x7f7078160a50 2024-04-26T23:02:28.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.468+0000 7f7077400640 1 -- 172.21.15.40:0/408140309 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7068032440 con 0x7f7078160a50 2024-04-26T23:02:28.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.468+0000 7f7077400640 1 -- 172.21.15.40:0/408140309 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068032810 con 0x7f7078160a50 2024-04-26T23:02:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.469+0000 7f707d600640 1 -- 172.21.15.40:0/408140309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7078160a50 msgr2=0x7f7078073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.469+0000 7f707d600640 1 --2- 172.21.15.40:0/408140309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7078160a50 0x7f7078073a50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f70680097a0 tx=0x7f706802a180 comp rx=0 tx=0).stop 2024-04-26T23:02:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.469+0000 7f707d600640 1 -- 172.21.15.40:0/408140309 shutdown_connections 2024-04-26T23:02:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.469+0000 7f707d600640 1 --2- 172.21.15.40:0/408140309 >> v2:172.21.15.103:3300/0 conn(0x7f70780cfaf0 0x7f7078160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.469+0000 7f707d600640 1 --2- 172.21.15.40:0/408140309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7078160a50 0x7f7078073a50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.469+0000 7f707d600640 1 -- 172.21.15.40:0/408140309 >> 172.21.15.40:0/408140309 conn(0x7f70780b2720 msgr2=0x7f70780b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.469+0000 7f707d600640 1 -- 172.21.15.40:0/408140309 shutdown_connections 2024-04-26T23:02:28.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.469+0000 7f707d600640 1 -- 172.21.15.40:0/408140309 wait complete. 2024-04-26T23:02:28.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.471+0000 7f707d600640 1 Processor -- start 2024-04-26T23:02:28.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.471+0000 7f707d600640 1 -- start start 2024-04-26T23:02:28.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.472+0000 7f707d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f707806dd40 0x7f707815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.472+0000 7f707d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7078160a50 0x7f70780b17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.472+0000 7f707d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f707813ab80 0x7f70780b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.472+0000 7f707d600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f70780d0260 con 0x7f7078160a50 2024-04-26T23:02:28.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.472+0000 7f707d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7078073f90 con 0x7f707813ab80 2024-04-26T23:02:28.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.472+0000 7f707d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7078071f30 con 0x7f707806dd40 2024-04-26T23:02:28.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.473+0000 7f706fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7078160a50 0x7f70780b17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.473+0000 7f706fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7078160a50 0x7f70780b17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45336/0 (socket says 172.21.15.40:45336) 2024-04-26T23:02:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.473+0000 7f706fe00640 1 -- 172.21.15.40:0/368325180 learned_addr learned my addr 172.21.15.40:0/368325180 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.473+0000 7f7077e00640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f707806dd40 0x7f707815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.473+0000 7f707cc00640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f707813ab80 0x7f70780b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.473+0000 7f706fe00640 1 -- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f707806dd40 msgr2=0x7f707815a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.473+0000 7f706fe00640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f707806dd40 0x7f707815a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.473+0000 7f706fe00640 1 -- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f707813ab80 msgr2=0x7f70780b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.473+0000 7f706fe00640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f707813ab80 0x7f70780b1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.473+0000 7f706fe00640 1 -- 172.21.15.40:0/368325180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f707815b3d0 con 0x7f7078160a50 2024-04-26T23:02:28.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.474+0000 7f706fe00640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7078160a50 0x7f70780b17f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f706000ebe0 tx=0x7f706000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:28.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.474+0000 7f7074c00640 1 -- 172.21.15.40:0/368325180 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7060017070 con 0x7f7078160a50 2024-04-26T23:02:28.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.474+0000 7f7074c00640 1 -- 172.21.15.40:0/368325180 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f706000cc30 con 0x7f7078160a50 2024-04-26T23:02:28.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.474+0000 7f7074c00640 1 -- 172.21.15.40:0/368325180 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7060013470 con 0x7f7078160a50 2024-04-26T23:02:28.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.474+0000 7f707d600640 1 -- 172.21.15.40:0/368325180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f707815b6b0 con 0x7f7078160a50 2024-04-26T23:02:28.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.474+0000 7f707d600640 1 -- 172.21.15.40:0/368325180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f707815bc70 con 0x7f7078160a50 2024-04-26T23:02:28.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.475+0000 7f707d600640 1 -- 172.21.15.40:0/368325180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7078069370 con 0x7f7078160a50 2024-04-26T23:02:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.476+0000 7f7074c00640 1 -- 172.21.15.40:0/368325180 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f706001a070 con 0x7f7078160a50 2024-04-26T23:02:28.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.477+0000 7f7074c00640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f705406c8f0 0x7f705406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.477+0000 7f7074c00640 1 -- 172.21.15.40:0/368325180 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f706008c250 con 0x7f7078160a50 2024-04-26T23:02:28.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.478+0000 7f7077e00640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f705406c8f0 0x7f705406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.480+0000 7f7077e00640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f705406c8f0 0x7f705406edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7068034000 tx=0x7f706802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.482+0000 7f7074c00640 1 -- 172.21.15.40:0/368325180 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f706005a270 con 0x7f7078160a50 2024-04-26T23:02:28.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.687+0000 7f707d600640 1 -- 172.21.15.40:0/368325180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f707815f7b0 con 0x7f7078160a50 2024-04-26T23:02:28.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.688+0000 7f7074c00640 1 -- 172.21.15.40:0/368325180 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f7060059b70 con 0x7f7078160a50 2024-04-26T23:02:28.688 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.691+0000 7f707d600640 1 -- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f705406c8f0 msgr2=0x7f705406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.691+0000 7f707d600640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f705406c8f0 0x7f705406edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7068034000 tx=0x7f706802e000 comp rx=0 tx=0).stop 2024-04-26T23:02:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.691+0000 7f707d600640 1 -- 172.21.15.40:0/368325180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7078160a50 msgr2=0x7f70780b17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.691+0000 7f707d600640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7078160a50 0x7f70780b17f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f706000ebe0 tx=0x7f706000c520 comp rx=0 tx=0).stop 2024-04-26T23:02:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.691+0000 7f707d600640 1 -- 172.21.15.40:0/368325180 shutdown_connections 2024-04-26T23:02:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.691+0000 7f707d600640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f705406c8f0 0x7f705406edb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.691+0000 7f707d600640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f707813ab80 0x7f70780b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.691+0000 7f707d600640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7078160a50 0x7f70780b17f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.691+0000 7f707d600640 1 --2- 172.21.15.40:0/368325180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f707806dd40 0x7f707815a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.691+0000 7f707d600640 1 -- 172.21.15.40:0/368325180 >> 172.21.15.40:0/368325180 conn(0x7f70780b2720 msgr2=0x7f707815dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:28.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.691+0000 7f707d600640 1 -- 172.21.15.40:0/368325180 shutdown_connections 2024-04-26T23:02:28.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.691+0000 7f707d600640 1 -- 172.21.15.40:0/368325180 wait complete. 2024-04-26T23:02:28.709 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:02:28.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.902+0000 7f06b9800640 1 Processor -- start 2024-04-26T23:02:28.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.902+0000 7f06b9800640 1 -- start start 2024-04-26T23:02:28.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.903+0000 7f06b9800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06b40d32e0 0x7f06b40d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.903+0000 7f06b9800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06b4162c40 con 0x7f06b40b8d20 2024-04-26T23:02:28.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.903+0000 7f06b9800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06b40d2d80 con 0x7f06b40b6220 2024-04-26T23:02:28.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.903+0000 7f06b9800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f06b40d3c80 con 0x7f06b40d32e0 2024-04-26T23:02:28.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.903+0000 7f06b8e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f06b40b8d20 0x7f06b40b91b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35798/0 (socket says 172.21.15.40:35798) 2024-04-26T23:02:28.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.903+0000 7f06b8e00640 1 -- 172.21.15.40:0/1167143498 learned_addr learned my addr 172.21.15.40:0/1167143498 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:28.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.904+0000 7f06abe00640 1 --2- 172.21.15.40:0/1167143498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06b40d32e0 0x7f06b40d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.904+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1409608807 0 0) 0x7f06b4162c40 con 0x7f06b40b8d20 2024-04-26T23:02:28.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.904+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0698003520 con 0x7f06b40b8d20 2024-04-26T23:02:28.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.904+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3903149264 0 0) 0x7f0698003520 con 0x7f06b40b8d20 2024-04-26T23:02:28.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.904+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06b40d32e0 msgr2=0x7f06b40d3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:28.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.904+0000 7f06b3400640 1 --2- 172.21.15.40:0/1167143498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06b40d32e0 0x7f06b40d3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.904+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 >> v1:172.21.15.103:6789/0 conn(0x7f06b40b6220 legacy=0x7f06b40d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.904+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b40b7c80 con 0x7f06b40b8d20 2024-04-26T23:02:28.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.904+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f06ac0019f0 con 0x7f06b40b8d20 2024-04-26T23:02:28.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.904+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 >> v1:172.21.15.40:6789/0 conn(0x7f06b40b8d20 legacy=0x7f06b40b91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06b3400640 1 --2- 172.21.15.40:0/1167143498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06980039e0 0x7f0698003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06b3400640 1 --2- 172.21.15.40:0/1167143498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06b40d32e0 0x7f0698004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06b3400640 1 --2- 172.21.15.40:0/1167143498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0698004d40 0x7f06980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f069800ba70 con 0x7f06980039e0 2024-04-26T23:02:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f069800bbf0 con 0x7f0698004d40 2024-04-26T23:02:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f069800bda0 con 0x7f06b40d32e0 2024-04-26T23:02:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06b3e00640 1 --2- 172.21.15.40:0/1167143498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06980039e0 0x7f0698003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06b8e00640 1 --2- 172.21.15.40:0/1167143498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0698004d40 0x7f06980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06abe00640 1 --2- 172.21.15.40:0/1167143498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06b40d32e0 0x7f0698004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06b3e00640 1 -- 172.21.15.40:0/1167143498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06b40d32e0 msgr2=0x7f0698004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06b3e00640 1 --2- 172.21.15.40:0/1167143498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06b40d32e0 0x7f0698004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06b3e00640 1 -- 172.21.15.40:0/1167143498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0698004d40 msgr2=0x7f06980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06b3e00640 1 --2- 172.21.15.40:0/1167143498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0698004d40 0x7f06980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.905+0000 7f06b3e00640 1 -- 172.21.15.40:0/1167143498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0698003520 con 0x7f06980039e0 2024-04-26T23:02:28.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.906+0000 7f06b3e00640 1 --2- 172.21.15.40:0/1167143498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06980039e0 0x7f0698003dc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f06a4001860 tx=0x7f0698005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da3c77e606dc61f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:28.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.907+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4037070 con 0x7f06980039e0 2024-04-26T23:02:28.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.907+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a402ac30 con 0x7f06980039e0 2024-04-26T23:02:28.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.907+0000 7f06b3400640 1 -- 172.21.15.40:0/1167143498 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4001430 con 0x7f06980039e0 2024-04-26T23:02:28.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.907+0000 7f06b9800640 1 -- 172.21.15.40:0/1167143498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06980039e0 msgr2=0x7f0698003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.907+0000 7f06b9800640 1 --2- 172.21.15.40:0/1167143498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06980039e0 0x7f0698003dc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f06a4001860 tx=0x7f0698005830 comp rx=0 tx=0).stop 2024-04-26T23:02:28.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.907+0000 7f06b8e00640 1 -- 172.21.15.40:0/1167143498 reap_dead start 2024-04-26T23:02:28.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.907+0000 7f06b9800640 1 -- 172.21.15.40:0/1167143498 shutdown_connections 2024-04-26T23:02:28.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.907+0000 7f06b9800640 1 -- 172.21.15.40:0/1167143498 >> 172.21.15.40:0/1167143498 conn(0x7f06b40d4d90 msgr2=0x7f06b40d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:28.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.908+0000 7f06b9800640 1 -- 172.21.15.40:0/1167143498 shutdown_connections 2024-04-26T23:02:28.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.908+0000 7f06b9800640 1 -- 172.21.15.40:0/1167143498 wait complete. 2024-04-26T23:02:28.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.909+0000 7f06b9800640 1 Processor -- start 2024-04-26T23:02:28.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.909+0000 7f06b9800640 1 -- start start 2024-04-26T23:02:28.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.910+0000 7f06b9800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06980039e0 0x7f06b414ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.910+0000 7f06b9800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b414e600 0x7f06b40ba0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.910+0000 7f06b9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06b415c9b0 0x7f06b415a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.910+0000 7f06b9800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f06b40b8520 con 0x7f06b415c9b0 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.910+0000 7f06b9800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f06b40b8240 con 0x7f06b414e600 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.910+0000 7f06b9800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f06b40d0aa0 con 0x7f06980039e0 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.911+0000 7f06b8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06b415c9b0 0x7f06b415a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.911+0000 7f06b8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06b415c9b0 0x7f06b415a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45354/0 (socket says 172.21.15.40:45354) 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.911+0000 7f06b8e00640 1 -- 172.21.15.40:0/2310910406 learned_addr learned my addr 172.21.15.40:0/2310910406 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.911+0000 7f06abe00640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b414e600 0x7f06b40ba0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.911+0000 7f06b3e00640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06980039e0 0x7f06b414ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.911+0000 7f06b8e00640 1 -- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06980039e0 msgr2=0x7f06b414ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.911+0000 7f06b8e00640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06980039e0 0x7f06b414ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.911+0000 7f06b8e00640 1 -- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b414e600 msgr2=0x7f06b40ba0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.911+0000 7f06b8e00640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b414e600 0x7f06b40ba0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.911+0000 7f06b8e00640 1 -- 172.21.15.40:0/2310910406 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b414b370 con 0x7f06b415c9b0 2024-04-26T23:02:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.912+0000 7f06b8e00640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06b415c9b0 0x7f06b415a4a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f06b40d5e00 tx=0x7f06ac007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:28.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.912+0000 7f06b0c00640 1 -- 172.21.15.40:0/2310910406 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac00eb00 con 0x7f06b415c9b0 2024-04-26T23:02:28.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.912+0000 7f06b0c00640 1 -- 172.21.15.40:0/2310910406 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06ac018070 con 0x7f06b415c9b0 2024-04-26T23:02:28.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.912+0000 7f06b0c00640 1 -- 172.21.15.40:0/2310910406 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac013420 con 0x7f06b415c9b0 2024-04-26T23:02:28.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.912+0000 7f06b9800640 1 -- 172.21.15.40:0/2310910406 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b414b650 con 0x7f06b415c9b0 2024-04-26T23:02:28.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.912+0000 7f06b9800640 1 -- 172.21.15.40:0/2310910406 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b415c1b0 con 0x7f06b415c9b0 2024-04-26T23:02:28.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.913+0000 7f06b9800640 1 -- 172.21.15.40:0/2310910406 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b4069370 con 0x7f06b415c9b0 2024-04-26T23:02:28.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.914+0000 7f06b0c00640 1 -- 172.21.15.40:0/2310910406 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f06ac0135c0 con 0x7f06b415c9b0 2024-04-26T23:02:28.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.915+0000 7f06b0c00640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f069006c920 0x7f069006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:28.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.915+0000 7f06b0c00640 1 -- 172.21.15.40:0/2310910406 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f06ac08c1d0 con 0x7f06b415c9b0 2024-04-26T23:02:28.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.916+0000 7f06b3e00640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f069006c920 0x7f069006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:28.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.917+0000 7f06b3e00640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f069006c920 0x7f069006ede0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f06a4034000 tx=0x7f06a4009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:28.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:28.920+0000 7f06b0c00640 1 -- 172.21.15.40:0/2310910406 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f06ac05a160 con 0x7f06b415c9b0 2024-04-26T23:02:29.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.135+0000 7f06b9800640 1 -- 172.21.15.40:0/2310910406 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f06b406e610 con 0x7f06b415c9b0 2024-04-26T23:02:29.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.136+0000 7f06b0c00640 1 -- 172.21.15.40:0/2310910406 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f06ac059a60 con 0x7f06b415c9b0 2024-04-26T23:02:29.136 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-04-26T23:02:29.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.139+0000 7f06b9800640 1 -- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f069006c920 msgr2=0x7f069006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:29.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.139+0000 7f06b9800640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f069006c920 0x7f069006ede0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f06a4034000 tx=0x7f06a4009030 comp rx=0 tx=0).stop 2024-04-26T23:02:29.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.139+0000 7f06b9800640 1 -- 172.21.15.40:0/2310910406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06b415c9b0 msgr2=0x7f06b415a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:29.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.139+0000 7f06b9800640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06b415c9b0 0x7f06b415a4a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f06b40d5e00 tx=0x7f06ac007bc0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.139+0000 7f06b9800640 1 -- 172.21.15.40:0/2310910406 shutdown_connections 2024-04-26T23:02:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.139+0000 7f06b9800640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f069006c920 0x7f069006ede0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.139+0000 7f06b9800640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06b415c9b0 0x7f06b415a4a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.139+0000 7f06b9800640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b414e600 0x7f06b40ba0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.139+0000 7f06b9800640 1 --2- 172.21.15.40:0/2310910406 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06980039e0 0x7f06b414ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.139+0000 7f06b9800640 1 -- 172.21.15.40:0/2310910406 >> 172.21.15.40:0/2310910406 conn(0x7f06b40d4d90 msgr2=0x7f06b40b9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.139+0000 7f06b9800640 1 -- 172.21.15.40:0/2310910406 shutdown_connections 2024-04-26T23:02:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.139+0000 7f06b9800640 1 -- 172.21.15.40:0/2310910406 wait complete. 2024-04-26T23:02:29.168 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-04-26T23:02:29.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:29.168 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-04-26T23:02:29.267 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T23:02:29.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:02:29.267 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-26T23:02:29.312 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-26T23:02:29.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:02:29.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.530+0000 7f13f4400640 1 Processor -- start 2024-04-26T23:02:29.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.531+0000 7f13f4400640 1 -- start start 2024-04-26T23:02:29.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.532+0000 7f13f4400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f13ec072950 0x7f13ec167ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:29.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.532+0000 7f13f4400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13ec15d7c0 con 0x7f13ec071e60 2024-04-26T23:02:29.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.532+0000 7f13f4400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13ec15daa0 con 0x7f13ec0cc480 2024-04-26T23:02:29.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.532+0000 7f13f4400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f13ec15dd80 con 0x7f13ec072950 2024-04-26T23:02:29.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.533+0000 7f13f1a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f13ec071e60 0x7f13ec06f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35812/0 (socket says 172.21.15.40:35812) 2024-04-26T23:02:29.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.533+0000 7f13f1a00640 1 -- 172.21.15.40:0/4260959587 learned_addr learned my addr 172.21.15.40:0/4260959587 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:29.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.533+0000 7f13f2400640 1 --2- 172.21.15.40:0/4260959587 >> v2:172.21.15.40:3300/0 conn(0x7f13ec072950 0x7f13ec167ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:29.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.533+0000 7f13ebe00640 1 -- 172.21.15.40:0/4260959587 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 293972627 0 0) 0x7f13ec15d7c0 con 0x7f13ec071e60 2024-04-26T23:02:29.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.533+0000 7f13ebe00640 1 -- 172.21.15.40:0/4260959587 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d4003520 con 0x7f13ec071e60 2024-04-26T23:02:29.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.533+0000 7f13f2400640 1 -- 172.21.15.40:0/4260959587 >> v1:172.21.15.103:6789/0 conn(0x7f13ec0cc480 legacy=0x7f13ec072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:29.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.533+0000 7f13f2400640 1 -- 172.21.15.40:0/4260959587 >> v1:172.21.15.40:6789/0 conn(0x7f13ec071e60 legacy=0x7f13ec06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:29.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.533+0000 7f13f2400640 1 -- 172.21.15.40:0/4260959587 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ec15e060 con 0x7f13ec072950 2024-04-26T23:02:29.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.535+0000 7f13f2400640 1 --2- 172.21.15.40:0/4260959587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13ec072950 0x7f13ec167ae0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f13e4009d90 tx=0x7f13e402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1f5cef019fed1b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:29.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.535+0000 7f13ebe00640 1 -- 172.21.15.40:0/4260959587 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e4007740 con 0x7f13ec072950 2024-04-26T23:02:29.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.535+0000 7f13ebe00640 1 -- 172.21.15.40:0/4260959587 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13e40078e0 con 0x7f13ec072950 2024-04-26T23:02:29.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.535+0000 7f13ebe00640 1 -- 172.21.15.40:0/4260959587 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e4007cb0 con 0x7f13ec072950 2024-04-26T23:02:29.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.535+0000 7f13f4400640 1 -- 172.21.15.40:0/4260959587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13ec072950 msgr2=0x7f13ec167ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:29.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.535+0000 7f13f4400640 1 --2- 172.21.15.40:0/4260959587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13ec072950 0x7f13ec167ae0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f13e4009d90 tx=0x7f13e402a6a0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.536+0000 7f13f4400640 1 -- 172.21.15.40:0/4260959587 shutdown_connections 2024-04-26T23:02:29.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.536+0000 7f13f4400640 1 --2- 172.21.15.40:0/4260959587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13ec072950 0x7f13ec167ae0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.536+0000 7f13f4400640 1 -- 172.21.15.40:0/4260959587 >> 172.21.15.40:0/4260959587 conn(0x7f13ec0d0be0 msgr2=0x7f13ec0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:29.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.536+0000 7f13f4400640 1 -- 172.21.15.40:0/4260959587 shutdown_connections 2024-04-26T23:02:29.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.536+0000 7f13f4400640 1 -- 172.21.15.40:0/4260959587 wait complete. 2024-04-26T23:02:29.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.538+0000 7f13f4400640 1 Processor -- start 2024-04-26T23:02:29.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.538+0000 7f13f4400640 1 -- start start 2024-04-26T23:02:29.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f4400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13ec071e60 0x7f13ec1f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:29.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f4400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec072950 0x7f13ec1f89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:29.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f4400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13ec0cc480 0x7f13ec1580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:29.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f4400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec15d570 con 0x7f13ec071e60 2024-04-26T23:02:29.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f4400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec15d3f0 con 0x7f13ec0cc480 2024-04-26T23:02:29.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f4400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f13ec06fff0 con 0x7f13ec072950 2024-04-26T23:02:29.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13ec071e60 0x7f13ec1f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:29.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13ec071e60 0x7f13ec1f49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45374/0 (socket says 172.21.15.40:45374) 2024-04-26T23:02:29.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f1a00640 1 -- 172.21.15.40:0/2775806881 learned_addr learned my addr 172.21.15.40:0/2775806881 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:29.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f1a00640 1 -- 172.21.15.40:0/2775806881 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec072950 msgr2=0x7f13ec1f89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:29.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f1a00640 1 --2- 172.21.15.40:0/2775806881 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec072950 0x7f13ec1f89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f1a00640 1 -- 172.21.15.40:0/2775806881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13ec0cc480 msgr2=0x7f13ec1580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:29.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f1a00640 1 --2- 172.21.15.40:0/2775806881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13ec0cc480 0x7f13ec1580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.539+0000 7f13f1a00640 1 -- 172.21.15.40:0/2775806881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ec1f4fa0 con 0x7f13ec071e60 2024-04-26T23:02:29.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.540+0000 7f13f1a00640 1 --2- 172.21.15.40:0/2775806881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13ec071e60 0x7f13ec1f49b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f13e000c430 tx=0x7f13e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:29.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.540+0000 7f13e9600640 1 -- 172.21.15.40:0/2775806881 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e0017070 con 0x7f13ec071e60 2024-04-26T23:02:29.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.540+0000 7f13e9600640 1 -- 172.21.15.40:0/2775806881 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13e0005ce0 con 0x7f13ec071e60 2024-04-26T23:02:29.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.540+0000 7f13e9600640 1 -- 172.21.15.40:0/2775806881 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e00072f0 con 0x7f13ec071e60 2024-04-26T23:02:29.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.540+0000 7f13f4400640 1 -- 172.21.15.40:0/2775806881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ec1f5280 con 0x7f13ec071e60 2024-04-26T23:02:29.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.540+0000 7f13f4400640 1 -- 172.21.15.40:0/2775806881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ec1f5840 con 0x7f13ec071e60 2024-04-26T23:02:29.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.541+0000 7f13f4400640 1 -- 172.21.15.40:0/2775806881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13ec0694f0 con 0x7f13ec071e60 2024-04-26T23:02:29.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.543+0000 7f13e9600640 1 -- 172.21.15.40:0/2775806881 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f13e001a070 con 0x7f13ec071e60 2024-04-26T23:02:29.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.543+0000 7f13e9600640 1 --2- 172.21.15.40:0/2775806881 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f13cc06c920 0x7f13cc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:29.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.544+0000 7f13e9600640 1 -- 172.21.15.40:0/2775806881 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f13e008b3f0 con 0x7f13ec071e60 2024-04-26T23:02:29.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.544+0000 7f13f1000640 1 --2- 172.21.15.40:0/2775806881 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f13cc06c920 0x7f13cc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:29.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.545+0000 7f13f1000640 1 --2- 172.21.15.40:0/2775806881 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f13cc06c920 0x7f13cc06ede0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f13d8006fd0 tx=0x7f13d8006bc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:29.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.548+0000 7f13e9600640 1 -- 172.21.15.40:0/2775806881 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f13e0059410 con 0x7f13ec071e60 2024-04-26T23:02:29.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.783+0000 7f13f4400640 1 -- 172.21.15.40:0/2775806881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13ec159ba0 con 0x7f13ec071e60 2024-04-26T23:02:29.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.784+0000 7f13e9600640 1 -- 172.21.15.40:0/2775806881 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f13e0058d10 con 0x7f13ec071e60 2024-04-26T23:02:29.784 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:29.785 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1253648754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1253648754},{"type":"v1","addr":"172.21.15.103:6836","nonce":1253648754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3239652611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":3239652611},{"type":"v1","addr":"172.21.15.103:6837","nonce":3239652611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3670144900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3670144900},{"type":"v1","addr":"172.21.15.40:6833","nonce":3670144900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:01:05.171276+0000","modified":"2024-04-26T23:01:33.399801+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3989190340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3989190340},{"type":"v1","addr":"172.21.15.40:6835","nonce":3989190340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-26T23:02:29.785 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 9 2024-04-26T23:02:29.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.787+0000 7f13f4400640 1 -- 172.21.15.40:0/2775806881 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f13cc06c920 msgr2=0x7f13cc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:29.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.787+0000 7f13f4400640 1 --2- 172.21.15.40:0/2775806881 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f13cc06c920 0x7f13cc06ede0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f13d8006fd0 tx=0x7f13d8006bc0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.787+0000 7f13f4400640 1 -- 172.21.15.40:0/2775806881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13ec071e60 msgr2=0x7f13ec1f49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:29.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.787+0000 7f13f4400640 1 --2- 172.21.15.40:0/2775806881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13ec071e60 0x7f13ec1f49b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f13e000c430 tx=0x7f13e000c8f0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.787+0000 7f13f4400640 1 -- 172.21.15.40:0/2775806881 shutdown_connections 2024-04-26T23:02:29.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.787+0000 7f13f4400640 1 --2- 172.21.15.40:0/2775806881 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f13cc06c920 0x7f13cc06ede0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.787+0000 7f13f4400640 1 --2- 172.21.15.40:0/2775806881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13ec0cc480 0x7f13ec1580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.787+0000 7f13f4400640 1 --2- 172.21.15.40:0/2775806881 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13ec072950 0x7f13ec1f89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.787+0000 7f13f4400640 1 --2- 172.21.15.40:0/2775806881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13ec071e60 0x7f13ec1f49b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:29.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.787+0000 7f13f4400640 1 -- 172.21.15.40:0/2775806881 >> 172.21.15.40:0/2775806881 conn(0x7f13ec0d0be0 msgr2=0x7f13ec0b0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:29.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.787+0000 7f13f4400640 1 -- 172.21.15.40:0/2775806881 shutdown_connections 2024-04-26T23:02:29.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:29.787+0000 7f13f4400640 1 -- 172.21.15.40:0/2775806881 wait complete. 2024-04-26T23:02:29.807 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.002+0000 7f4811600640 1 Processor -- start 2024-04-26T23:02:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.003+0000 7f4811600640 1 -- start start 2024-04-26T23:02:30.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.004+0000 7f4811600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f480c073b70 0x7f480c0aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.004+0000 7f4811600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f480c072c00 con 0x7f480c15f380 2024-04-26T23:02:30.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.004+0000 7f4811600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f480c072ee0 con 0x7f480c0cbb50 2024-04-26T23:02:30.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.004+0000 7f4811600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f480c0729b0 con 0x7f480c073b70 2024-04-26T23:02:30.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.004+0000 7f480aa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f480c15f380 0x7f480c0cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35818/0 (socket says 172.21.15.40:35818) 2024-04-26T23:02:30.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.005+0000 7f480aa00640 1 -- 172.21.15.40:0/1158703581 learned_addr learned my addr 172.21.15.40:0/1158703581 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:30.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.005+0000 7f480a000640 1 --2- 172.21.15.40:0/1158703581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f480c073b70 0x7f480c0aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.005+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503481011 0 0) 0x7f480c072c00 con 0x7f480c15f380 2024-04-26T23:02:30.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.005+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47f0003520 con 0x7f480c15f380 2024-04-26T23:02:30.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.005+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1074712944 0 0) 0x7f47f0003520 con 0x7f480c15f380 2024-04-26T23:02:30.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.005+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f480c073b70 msgr2=0x7f480c0aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:30.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.005+0000 7f4809600640 1 --2- 172.21.15.40:0/1158703581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f480c073b70 0x7f480c0aecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.005+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 >> v1:172.21.15.103:6789/0 conn(0x7f480c0cbb50 legacy=0x7f480c0af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.005+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480c0731c0 con 0x7f480c15f380 2024-04-26T23:02:30.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.005+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f47fc0019f0 con 0x7f480c15f380 2024-04-26T23:02:30.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 >> v1:172.21.15.40:6789/0 conn(0x7f480c15f380 legacy=0x7f480c0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f4809600640 1 --2- 172.21.15.40:0/1158703581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f00039e0 0x7f47f0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f4809600640 1 --2- 172.21.15.40:0/1158703581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f0004800 0x7f47f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f4809600640 1 --2- 172.21.15.40:0/1158703581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47f0005180 0x7f47f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f480c0729b0 con 0x7f47f00039e0 2024-04-26T23:02:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f47f000bf00 con 0x7f47f0005180 2024-04-26T23:02:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f47f000c080 con 0x7f47f0004800 2024-04-26T23:02:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f480aa00640 1 --2- 172.21.15.40:0/1158703581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f00039e0 0x7f47f0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f480a000640 1 --2- 172.21.15.40:0/1158703581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f0004800 0x7f47f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f480aa00640 1 -- 172.21.15.40:0/1158703581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f0004800 msgr2=0x7f47f0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f480aa00640 1 --2- 172.21.15.40:0/1158703581 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f0004800 0x7f47f0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f480aa00640 1 -- 172.21.15.40:0/1158703581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47f0005180 msgr2=0x7f47f000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f480aa00640 1 --2- 172.21.15.40:0/1158703581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47f0005180 0x7f47f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.006+0000 7f480aa00640 1 -- 172.21.15.40:0/1158703581 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47f0003520 con 0x7f47f00039e0 2024-04-26T23:02:30.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.008+0000 7f480aa00640 1 --2- 172.21.15.40:0/1158703581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f00039e0 0x7f47f0003e00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00a790 tx=0x7f47f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed339291f4ecd9a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:30.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.008+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc00a3b0 con 0x7f47f00039e0 2024-04-26T23:02:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.008+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47fc02f070 con 0x7f47f00039e0 2024-04-26T23:02:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.008+0000 7f4809600640 1 -- 172.21.15.40:0/1158703581 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc02bbf0 con 0x7f47f00039e0 2024-04-26T23:02:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.008+0000 7f4811600640 1 -- 172.21.15.40:0/1158703581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f00039e0 msgr2=0x7f47f0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.008+0000 7f4811600640 1 --2- 172.21.15.40:0/1158703581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47f00039e0 0x7f47f0003e00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00a790 tx=0x7f47f0005830 comp rx=0 tx=0).stop 2024-04-26T23:02:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.008+0000 7f480b400640 1 -- 172.21.15.40:0/1158703581 reap_dead start 2024-04-26T23:02:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.009+0000 7f4811600640 1 -- 172.21.15.40:0/1158703581 shutdown_connections 2024-04-26T23:02:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.009+0000 7f4811600640 1 -- 172.21.15.40:0/1158703581 >> 172.21.15.40:0/1158703581 conn(0x7f480c0d0be0 msgr2=0x7f480c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.009+0000 7f4811600640 1 -- 172.21.15.40:0/1158703581 shutdown_connections 2024-04-26T23:02:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.009+0000 7f4811600640 1 -- 172.21.15.40:0/1158703581 wait complete. 2024-04-26T23:02:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.010+0000 7f4811600640 1 Processor -- start 2024-04-26T23:02:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.011+0000 7f4811600640 1 -- start start 2024-04-26T23:02:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.011+0000 7f4811600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f00039e0 0x7f480c0b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.011+0000 7f4811600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f480c0b4760 0x7f480c1f43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f4811600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c0caa10 0x7f480c0b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f4811600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f480c073620 con 0x7f480c0caa10 2024-04-26T23:02:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f4811600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f480c0d0870 con 0x7f480c0b4760 2024-04-26T23:02:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f4811600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f480c0737a0 con 0x7f47f00039e0 2024-04-26T23:02:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f480b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c0caa10 0x7f480c0b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f480b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c0caa10 0x7f480c0b1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45398/0 (socket says 172.21.15.40:45398) 2024-04-26T23:02:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f480b400640 1 -- 172.21.15.40:0/153693838 learned_addr learned my addr 172.21.15.40:0/153693838 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f480a000640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f480c0b4760 0x7f480c1f43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f480aa00640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f00039e0 0x7f480c0b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f480b400640 1 -- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f00039e0 msgr2=0x7f480c0b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f480b400640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f00039e0 0x7f480c0b4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f480b400640 1 -- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f480c0b4760 msgr2=0x7f480c1f43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f480b400640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f480c0b4760 0x7f480c1f43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.012+0000 7f480b400640 1 -- 172.21.15.40:0/153693838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480c1f4e90 con 0x7f480c0caa10 2024-04-26T23:02:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.013+0000 7f480b400640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c0caa10 0x7f480c0b1560 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f480c0cb1e0 tx=0x7f480400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.013+0000 7f4802a00640 1 -- 172.21.15.40:0/153693838 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4804007340 con 0x7f480c0caa10 2024-04-26T23:02:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.013+0000 7f4802a00640 1 -- 172.21.15.40:0/153693838 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48040074e0 con 0x7f480c0caa10 2024-04-26T23:02:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.013+0000 7f4802a00640 1 -- 172.21.15.40:0/153693838 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4804004260 con 0x7f480c0caa10 2024-04-26T23:02:30.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.013+0000 7f4811600640 1 -- 172.21.15.40:0/153693838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f480c1f5170 con 0x7f480c0caa10 2024-04-26T23:02:30.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.013+0000 7f4811600640 1 -- 172.21.15.40:0/153693838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f480c0b3d00 con 0x7f480c0caa10 2024-04-26T23:02:30.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.015+0000 7f4811600640 1 -- 172.21.15.40:0/153693838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f480c0694f0 con 0x7f480c0caa10 2024-04-26T23:02:30.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.016+0000 7f4802a00640 1 -- 172.21.15.40:0/153693838 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4804011070 con 0x7f480c0caa10 2024-04-26T23:02:30.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.017+0000 7f4802a00640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f47e806c8f0 0x7f47e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.017+0000 7f4802a00640 1 -- 172.21.15.40:0/153693838 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f480408b100 con 0x7f480c0caa10 2024-04-26T23:02:30.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.018+0000 7f480aa00640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f47e806c8f0 0x7f47e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.019+0000 7f480aa00640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f47e806c8f0 0x7f47e806edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f47fc033000 tx=0x7f47fc007000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:30.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.022+0000 7f4802a00640 1 -- 172.21.15.40:0/153693838 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4804059090 con 0x7f480c0caa10 2024-04-26T23:02:30.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.222+0000 7f4811600640 1 -- 172.21.15.40:0/153693838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f480c0b4b40 con 0x7f480c0caa10 2024-04-26T23:02:30.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.225+0000 7f4802a00640 1 -- 172.21.15.40:0/153693838 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18812 (secure 0 0 0) 0x7f4804058990 con 0x7f480c0caa10 2024-04-26T23:02:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:30.225 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:46.975683+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:30.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.227+0000 7f4811600640 1 -- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f47e806c8f0 msgr2=0x7f47e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.227+0000 7f4811600640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f47e806c8f0 0x7f47e806edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f47fc033000 tx=0x7f47fc007000 comp rx=0 tx=0).stop 2024-04-26T23:02:30.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.227+0000 7f4811600640 1 -- 172.21.15.40:0/153693838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c0caa10 msgr2=0x7f480c0b1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.227+0000 7f4811600640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c0caa10 0x7f480c0b1560 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f480c0cb1e0 tx=0x7f480400c820 comp rx=0 tx=0).stop 2024-04-26T23:02:30.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.228+0000 7f4811600640 1 -- 172.21.15.40:0/153693838 shutdown_connections 2024-04-26T23:02:30.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.228+0000 7f4811600640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f47e806c8f0 0x7f47e806edb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.228+0000 7f4811600640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c0caa10 0x7f480c0b1560 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.228+0000 7f4811600640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f480c0b4760 0x7f480c1f43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.228+0000 7f4811600640 1 --2- 172.21.15.40:0/153693838 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47f00039e0 0x7f480c0b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.228+0000 7f4811600640 1 -- 172.21.15.40:0/153693838 >> 172.21.15.40:0/153693838 conn(0x7f480c0d0be0 msgr2=0x7f480c0cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:30.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.228+0000 7f4811600640 1 -- 172.21.15.40:0/153693838 shutdown_connections 2024-04-26T23:02:30.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.228+0000 7f4811600640 1 -- 172.21.15.40:0/153693838 wait complete. 2024-04-26T23:02:30.247 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:01:02.693456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:01:04.143797+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:01:07.764427+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:02:30.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:02:30.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.459+0000 7fbc1be00640 1 Processor -- start 2024-04-26T23:02:30.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.459+0000 7fbc1be00640 1 -- start start 2024-04-26T23:02:30.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.461+0000 7fbc1be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc1c160a50 0x7fbc1c073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.461+0000 7fbc1be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc1c0cfaf0 0x7fbc1c160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.461+0000 7fbc1be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc1c161880 con 0x7fbc1c06dd40 2024-04-26T23:02:30.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.461+0000 7fbc1be00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fbc1c071ce0 con 0x7fbc1c0cfaf0 2024-04-26T23:02:30.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.461+0000 7fbc1be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbc1c0722a0 con 0x7fbc1c160a50 2024-04-26T23:02:30.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.461+0000 7fbc1a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc1c0cfaf0 0x7fbc1c160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.461+0000 7fbc1a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc1c0cfaf0 0x7fbc1c160e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45410/0 (socket says 172.21.15.40:45410) 2024-04-26T23:02:30.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.461+0000 7fbc1a000640 1 -- 172.21.15.40:0/849391151 learned_addr learned my addr 172.21.15.40:0/849391151 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:30.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.461+0000 7fbc1aa00640 1 --2- 172.21.15.40:0/849391151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc1c160a50 0x7fbc1c073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.462+0000 7fbc1a000640 1 -- 172.21.15.40:0/849391151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc1c160a50 msgr2=0x7fbc1c073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:30.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.462+0000 7fbc1a000640 1 --2- 172.21.15.40:0/849391151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc1c160a50 0x7fbc1c073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.462+0000 7fbc1a000640 1 -- 172.21.15.40:0/849391151 >> v1:172.21.15.103:6789/0 conn(0x7fbc1c06dd40 legacy=0x7fbc1c072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:30.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.462+0000 7fbc1a000640 1 -- 172.21.15.40:0/849391151 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c161370 con 0x7fbc1c0cfaf0 2024-04-26T23:02:30.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.463+0000 7fbc1a000640 1 --2- 172.21.15.40:0/849391151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc1c0cfaf0 0x7fbc1c160e30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009870 tx=0x7fbc0802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eee22e74cb4fbfc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:30.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.463+0000 7fbc19600640 1 -- 172.21.15.40:0/849391151 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc08037070 con 0x7fbc1c0cfaf0 2024-04-26T23:02:30.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.463+0000 7fbc19600640 1 -- 172.21.15.40:0/849391151 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc08032440 con 0x7fbc1c0cfaf0 2024-04-26T23:02:30.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.463+0000 7fbc19600640 1 -- 172.21.15.40:0/849391151 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc08032810 con 0x7fbc1c0cfaf0 2024-04-26T23:02:30.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.464+0000 7fbc1be00640 1 -- 172.21.15.40:0/849391151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc1c0cfaf0 msgr2=0x7fbc1c160e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.464+0000 7fbc1be00640 1 --2- 172.21.15.40:0/849391151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc1c0cfaf0 0x7fbc1c160e30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009870 tx=0x7fbc0802a210 comp rx=0 tx=0).stop 2024-04-26T23:02:30.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.464+0000 7fbc1be00640 1 -- 172.21.15.40:0/849391151 shutdown_connections 2024-04-26T23:02:30.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.464+0000 7fbc1be00640 1 --2- 172.21.15.40:0/849391151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc1c0cfaf0 0x7fbc1c160e30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.464+0000 7fbc1be00640 1 --2- 172.21.15.40:0/849391151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc1c160a50 0x7fbc1c073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.464+0000 7fbc1be00640 1 -- 172.21.15.40:0/849391151 >> 172.21.15.40:0/849391151 conn(0x7fbc1c0b2720 msgr2=0x7fbc1c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:30.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.464+0000 7fbc1be00640 1 -- 172.21.15.40:0/849391151 shutdown_connections 2024-04-26T23:02:30.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.464+0000 7fbc1be00640 1 -- 172.21.15.40:0/849391151 wait complete. 2024-04-26T23:02:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.466+0000 7fbc1be00640 1 Processor -- start 2024-04-26T23:02:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.466+0000 7fbc1be00640 1 -- start start 2024-04-26T23:02:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.467+0000 7fbc1be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc1c06dd40 0x7fbc1c141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.467+0000 7fbc1be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc1c160a50 0x7fbc1c0af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.467+0000 7fbc1be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc1c0afee0 0x7fbc1c143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.467+0000 7fbc1be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c0d0260 con 0x7fbc1c06dd40 2024-04-26T23:02:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.467+0000 7fbc1be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c073f90 con 0x7fbc1c0afee0 2024-04-26T23:02:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.467+0000 7fbc1be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbc1c071f30 con 0x7fbc1c160a50 2024-04-26T23:02:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.468+0000 7fbc1aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc1c06dd40 0x7fbc1c141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.468+0000 7fbc1aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc1c06dd40 0x7fbc1c141220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45426/0 (socket says 172.21.15.40:45426) 2024-04-26T23:02:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.468+0000 7fbc1aa00640 1 -- 172.21.15.40:0/3741970098 learned_addr learned my addr 172.21.15.40:0/3741970098 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.468+0000 7fbc1a000640 1 --2- 172.21.15.40:0/3741970098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc1c160a50 0x7fbc1c0af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.468+0000 7fbc1aa00640 1 -- 172.21.15.40:0/3741970098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc1c160a50 msgr2=0x7fbc1c0af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.468+0000 7fbc1aa00640 1 --2- 172.21.15.40:0/3741970098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc1c160a50 0x7fbc1c0af9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.468+0000 7fbc1aa00640 1 -- 172.21.15.40:0/3741970098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc1c0afee0 msgr2=0x7fbc1c143ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.468+0000 7fbc1aa00640 1 --2- 172.21.15.40:0/3741970098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc1c0afee0 0x7fbc1c143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.468+0000 7fbc1aa00640 1 -- 172.21.15.40:0/3741970098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c144880 con 0x7fbc1c06dd40 2024-04-26T23:02:30.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.468+0000 7fbc1aa00640 1 --2- 172.21.15.40:0/3741970098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc1c06dd40 0x7fbc1c141220 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00c820 tx=0x7fbc0c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:30.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.469+0000 7fbc12a00640 1 -- 172.21.15.40:0/3741970098 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c015070 con 0x7fbc1c06dd40 2024-04-26T23:02:30.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.469+0000 7fbc12a00640 1 -- 172.21.15.40:0/3741970098 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc0c004020 con 0x7fbc1c06dd40 2024-04-26T23:02:30.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.469+0000 7fbc12a00640 1 -- 172.21.15.40:0/3741970098 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c0043f0 con 0x7fbc1c06dd40 2024-04-26T23:02:30.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.469+0000 7fbc1be00640 1 -- 172.21.15.40:0/3741970098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc1c144b60 con 0x7fbc1c06dd40 2024-04-26T23:02:30.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.469+0000 7fbc1be00640 1 -- 172.21.15.40:0/3741970098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc1c145120 con 0x7fbc1c06dd40 2024-04-26T23:02:30.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.470+0000 7fbc1be00640 1 -- 172.21.15.40:0/3741970098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc1c0b0300 con 0x7fbc1c06dd40 2024-04-26T23:02:30.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.471+0000 7fbc12a00640 1 -- 172.21.15.40:0/3741970098 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbc0c007830 con 0x7fbc1c06dd40 2024-04-26T23:02:30.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.472+0000 7fbc12a00640 1 --2- 172.21.15.40:0/3741970098 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbbfc06c8f0 0x7fbbfc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.472+0000 7fbc12a00640 1 -- 172.21.15.40:0/3741970098 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbc0c08bd10 con 0x7fbc1c06dd40 2024-04-26T23:02:30.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.473+0000 7fbc1a000640 1 --2- 172.21.15.40:0/3741970098 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbbfc06c8f0 0x7fbbfc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.474+0000 7fbc1a000640 1 --2- 172.21.15.40:0/3741970098 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbbfc06c8f0 0x7fbbfc06edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbc08034000 tx=0x7fbc0802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:30.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.476+0000 7fbc12a00640 1 -- 172.21.15.40:0/3741970098 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fbc0c091020 con 0x7fbc1c06dd40 2024-04-26T23:02:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.685+0000 7fbc1be00640 1 -- 172.21.15.40:0/3741970098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbc1c06fa60 con 0x7fbc1c06dd40 2024-04-26T23:02:30.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.686+0000 7fbc12a00640 1 -- 172.21.15.40:0/3741970098 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fbc0c059d30 con 0x7fbc1c06dd40 2024-04-26T23:02:30.686 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:02:30.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.689+0000 7fbc1be00640 1 -- 172.21.15.40:0/3741970098 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbbfc06c8f0 msgr2=0x7fbbfc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.689+0000 7fbc1be00640 1 --2- 172.21.15.40:0/3741970098 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbbfc06c8f0 0x7fbbfc06edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbc08034000 tx=0x7fbc0802e000 comp rx=0 tx=0).stop 2024-04-26T23:02:30.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.689+0000 7fbc1be00640 1 -- 172.21.15.40:0/3741970098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc1c06dd40 msgr2=0x7fbc1c141220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.689+0000 7fbc1be00640 1 --2- 172.21.15.40:0/3741970098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc1c06dd40 0x7fbc1c141220 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00c820 tx=0x7fbc0c00cce0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.689+0000 7fbc1be00640 1 -- 172.21.15.40:0/3741970098 shutdown_connections 2024-04-26T23:02:30.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.689+0000 7fbc1be00640 1 --2- 172.21.15.40:0/3741970098 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbbfc06c8f0 0x7fbbfc06edb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.689+0000 7fbc1be00640 1 --2- 172.21.15.40:0/3741970098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc1c0afee0 0x7fbc1c143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.689+0000 7fbc1be00640 1 --2- 172.21.15.40:0/3741970098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc1c160a50 0x7fbc1c0af9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.689+0000 7fbc1be00640 1 --2- 172.21.15.40:0/3741970098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc1c06dd40 0x7fbc1c141220 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.689+0000 7fbc1be00640 1 -- 172.21.15.40:0/3741970098 >> 172.21.15.40:0/3741970098 conn(0x7fbc1c0b2720 msgr2=0x7fbc1c145b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:30.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.690+0000 7fbc1be00640 1 -- 172.21.15.40:0/3741970098 shutdown_connections 2024-04-26T23:02:30.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.690+0000 7fbc1be00640 1 -- 172.21.15.40:0/3741970098 wait complete. 2024-04-26T23:02:30.709 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:02:30.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.921+0000 7fe540600640 1 Processor -- start 2024-04-26T23:02:30.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.921+0000 7fe540600640 1 -- start start 2024-04-26T23:02:30.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.922+0000 7fe540600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe538072950 0x7fe538167ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.922+0000 7fe540600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe53815d7c0 con 0x7fe5380cc480 2024-04-26T23:02:30.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.922+0000 7fe540600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe53815daa0 con 0x7fe538071e60 2024-04-26T23:02:30.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.922+0000 7fe540600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe53815dd80 con 0x7fe538072950 2024-04-26T23:02:30.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.923+0000 7fe53e600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe538072950 0x7fe538167ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.923+0000 7fe53e600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe538072950 0x7fe538167ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45442/0 (socket says 172.21.15.40:45442) 2024-04-26T23:02:30.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.923+0000 7fe53e600640 1 -- 172.21.15.40:0/2055790355 learned_addr learned my addr 172.21.15.40:0/2055790355 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:30.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.923+0000 7fe53e600640 1 -- 172.21.15.40:0/2055790355 >> v1:172.21.15.103:6789/0 conn(0x7fe538071e60 legacy=0x7fe53806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.923+0000 7fe53e600640 1 -- 172.21.15.40:0/2055790355 >> v1:172.21.15.40:6789/0 conn(0x7fe5380cc480 legacy=0x7fe538072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:30.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.924+0000 7fe53e600640 1 -- 172.21.15.40:0/2055790355 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe53815e060 con 0x7fe538072950 2024-04-26T23:02:30.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.925+0000 7fe53e600640 1 --2- 172.21.15.40:0/2055790355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe538072950 0x7fe538167ae0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe530009870 tx=0x7fe53002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce1d649b9efe0b5c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:30.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.925+0000 7fe537e00640 1 -- 172.21.15.40:0/2055790355 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530037070 con 0x7fe538072950 2024-04-26T23:02:30.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.925+0000 7fe537e00640 1 -- 172.21.15.40:0/2055790355 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe53002ae50 con 0x7fe538072950 2024-04-26T23:02:30.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.925+0000 7fe537e00640 1 -- 172.21.15.40:0/2055790355 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530032860 con 0x7fe538072950 2024-04-26T23:02:30.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.926+0000 7fe540600640 1 -- 172.21.15.40:0/2055790355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe538072950 msgr2=0x7fe538167ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.926+0000 7fe540600640 1 --2- 172.21.15.40:0/2055790355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe538072950 0x7fe538167ae0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe530009870 tx=0x7fe53002a300 comp rx=0 tx=0).stop 2024-04-26T23:02:30.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.926+0000 7fe540600640 1 -- 172.21.15.40:0/2055790355 shutdown_connections 2024-04-26T23:02:30.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.926+0000 7fe540600640 1 --2- 172.21.15.40:0/2055790355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe538072950 0x7fe538167ae0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.926+0000 7fe540600640 1 -- 172.21.15.40:0/2055790355 >> 172.21.15.40:0/2055790355 conn(0x7fe5380d0be0 msgr2=0x7fe5380d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:30.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.926+0000 7fe540600640 1 -- 172.21.15.40:0/2055790355 shutdown_connections 2024-04-26T23:02:30.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.926+0000 7fe540600640 1 -- 172.21.15.40:0/2055790355 wait complete. 2024-04-26T23:02:30.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.928+0000 7fe540600640 1 Processor -- start 2024-04-26T23:02:30.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.928+0000 7fe540600640 1 -- start start 2024-04-26T23:02:30.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe540600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe538071e60 0x7fe5381f54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe540600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe538072950 0x7fe538155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe540600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5380cc480 0x7fe538156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe540600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe53815d570 con 0x7fe538071e60 2024-04-26T23:02:30.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe540600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe53815d3f0 con 0x7fe538072950 2024-04-26T23:02:30.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe540600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe53806fff0 con 0x7fe5380cc480 2024-04-26T23:02:30.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe53dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe538071e60 0x7fe5381f54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe53dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe538071e60 0x7fe5381f54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45452/0 (socket says 172.21.15.40:45452) 2024-04-26T23:02:30.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe53dc00640 1 -- 172.21.15.40:0/3593151004 learned_addr learned my addr 172.21.15.40:0/3593151004 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe53d200640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe538072950 0x7fe538155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe53e600640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5380cc480 0x7fe538156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe53dc00640 1 -- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5380cc480 msgr2=0x7fe538156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe53dc00640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5380cc480 0x7fe538156d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe53dc00640 1 -- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe538072950 msgr2=0x7fe538155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe53dc00640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe538072950 0x7fe538155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.929+0000 7fe53dc00640 1 -- 172.21.15.40:0/3593151004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5381564e0 con 0x7fe538071e60 2024-04-26T23:02:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.930+0000 7fe53dc00640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe538071e60 0x7fe5381f54d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe52800c870 tx=0x7fe52800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.930+0000 7fe535600640 1 -- 172.21.15.40:0/3593151004 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe528003b70 con 0x7fe538071e60 2024-04-26T23:02:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.930+0000 7fe535600640 1 -- 172.21.15.40:0/3593151004 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe528003d30 con 0x7fe538071e60 2024-04-26T23:02:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.930+0000 7fe535600640 1 -- 172.21.15.40:0/3593151004 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52800aa90 con 0x7fe538071e60 2024-04-26T23:02:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.930+0000 7fe540600640 1 -- 172.21.15.40:0/3593151004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5381f5ac0 con 0x7fe538071e60 2024-04-26T23:02:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.930+0000 7fe540600640 1 -- 172.21.15.40:0/3593151004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5381f6080 con 0x7fe538071e60 2024-04-26T23:02:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.932+0000 7fe540600640 1 -- 172.21.15.40:0/3593151004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5380ccf70 con 0x7fe538071e60 2024-04-26T23:02:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.933+0000 7fe535600640 1 -- 172.21.15.40:0/3593151004 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe52800ac30 con 0x7fe538071e60 2024-04-26T23:02:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.934+0000 7fe535600640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe51806c8f0 0x7fe51806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.934+0000 7fe535600640 1 -- 172.21.15.40:0/3593151004 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe52808bfb0 con 0x7fe538071e60 2024-04-26T23:02:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.934+0000 7fe53d200640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe51806c8f0 0x7fe51806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.935+0000 7fe53d200640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe51806c8f0 0x7fe51806edb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe5380c7e60 tx=0x7fe524009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:30.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:30.938+0000 7fe535600640 1 -- 172.21.15.40:0/3593151004 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe5380ccf70 con 0x7fe538071e60 2024-04-26T23:02:31.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.141+0000 7fe540600640 1 -- 172.21.15.40:0/3593151004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe5380694f0 con 0x7fe538071e60 2024-04-26T23:02:31.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.142+0000 7fe535600640 1 -- 172.21.15.40:0/3593151004 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fe528059f40 con 0x7fe538071e60 2024-04-26T23:02:31.142 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:31.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.144+0000 7fe540600640 1 -- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe51806c8f0 msgr2=0x7fe51806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:31.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.144+0000 7fe540600640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe51806c8f0 0x7fe51806edb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe5380c7e60 tx=0x7fe524009040 comp rx=0 tx=0).stop 2024-04-26T23:02:31.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.144+0000 7fe540600640 1 -- 172.21.15.40:0/3593151004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe538071e60 msgr2=0x7fe5381f54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:31.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.144+0000 7fe540600640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe538071e60 0x7fe5381f54d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe52800c870 tx=0x7fe52800cd30 comp rx=0 tx=0).stop 2024-04-26T23:02:31.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.144+0000 7fe540600640 1 -- 172.21.15.40:0/3593151004 shutdown_connections 2024-04-26T23:02:31.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.144+0000 7fe540600640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe51806c8f0 0x7fe51806edb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.145+0000 7fe540600640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5380cc480 0x7fe538156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.145+0000 7fe540600640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe538072950 0x7fe538155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.145+0000 7fe540600640 1 --2- 172.21.15.40:0/3593151004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe538071e60 0x7fe5381f54d0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.145+0000 7fe540600640 1 -- 172.21.15.40:0/3593151004 >> 172.21.15.40:0/3593151004 conn(0x7fe5380d0be0 msgr2=0x7fe5380ceb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:31.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.145+0000 7fe540600640 1 -- 172.21.15.40:0/3593151004 shutdown_connections 2024-04-26T23:02:31.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.145+0000 7fe540600640 1 -- 172.21.15.40:0/3593151004 wait complete. 2024-04-26T23:02:31.164 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:02:31.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.389+0000 7f43ad800640 1 Processor -- start 2024-04-26T23:02:31.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.389+0000 7f43ad800640 1 -- start start 2024-04-26T23:02:31.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.390+0000 7f43ad800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a815ce60 0x7f43a80afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.390+0000 7f43ad800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43a80d0820 con 0x7f43a80af7a0 2024-04-26T23:02:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.391+0000 7f43ad800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43a81614c0 con 0x7f43a8071fe0 2024-04-26T23:02:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.391+0000 7f43ad800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f43a80cc450 con 0x7f43a815ce60 2024-04-26T23:02:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.391+0000 7f439fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f43a80af7a0 0x7f43a80ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35832/0 (socket says 172.21.15.40:35832) 2024-04-26T23:02:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.391+0000 7f439fe00640 1 -- 172.21.15.40:0/3111439827 learned_addr learned my addr 172.21.15.40:0/3111439827 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.391+0000 7f43a7e00640 1 --2- 172.21.15.40:0/3111439827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a815ce60 0x7f43a80afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2493846727 0 0) 0x7f43a80d0820 con 0x7f43a80af7a0 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f438c003520 con 0x7f43a80af7a0 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1649397899 0 0) 0x7f438c003520 con 0x7f43a80af7a0 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a815ce60 msgr2=0x7f43a80afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 --2- 172.21.15.40:0/3111439827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a815ce60 0x7f43a80afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 >> v1:172.21.15.103:6789/0 conn(0x7f43a8071fe0 legacy=0x7f43a8073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a8161760 con 0x7f43a80af7a0 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f4390001670 con 0x7f43a80af7a0 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 >> v1:172.21.15.40:6789/0 conn(0x7f43a80af7a0 legacy=0x7f43a80ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 --2- 172.21.15.40:0/3111439827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f438c0039e0 0x7f438c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 --2- 172.21.15.40:0/3111439827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a815ce60 0x7f438c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 --2- 172.21.15.40:0/3111439827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f438c004d40 0x7f438c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f43a80cc450 con 0x7f438c0039e0 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f438c00bb50 con 0x7f438c004d40 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f438c010160 con 0x7f43a815ce60 2024-04-26T23:02:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.392+0000 7f43a7e00640 1 --2- 172.21.15.40:0/3111439827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f438c0039e0 0x7f438c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.393+0000 7f439fe00640 1 --2- 172.21.15.40:0/3111439827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a815ce60 0x7f438c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.393+0000 7f43ace00640 1 --2- 172.21.15.40:0/3111439827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f438c004d40 0x7f438c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.393+0000 7f43a7e00640 1 -- 172.21.15.40:0/3111439827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a815ce60 msgr2=0x7f438c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.393+0000 7f43a7e00640 1 --2- 172.21.15.40:0/3111439827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a815ce60 0x7f438c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.393+0000 7f43a7e00640 1 -- 172.21.15.40:0/3111439827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f438c004d40 msgr2=0x7f438c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.393+0000 7f43a7e00640 1 --2- 172.21.15.40:0/3111439827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f438c004d40 0x7f438c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.393+0000 7f43a7e00640 1 -- 172.21.15.40:0/3111439827 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f438c003520 con 0x7f438c0039e0 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.393+0000 7f43ace00640 1 --2- 172.21.15.40:0/3111439827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f438c004d40 0x7f438c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.394+0000 7f43a7e00640 1 --2- 172.21.15.40:0/3111439827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f438c0039e0 0x7f438c003e00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4398009ac0 tx=0x7f438c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=251c666d42a00bdf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.394+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4398036070 con 0x7f438c0039e0 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.394+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4398031400 con 0x7f438c0039e0 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.394+0000 7f43a7400640 1 -- 172.21.15.40:0/3111439827 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4398009690 con 0x7f438c0039e0 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.395+0000 7f43ad800640 1 -- 172.21.15.40:0/3111439827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f438c0039e0 msgr2=0x7f438c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.395+0000 7f43ad800640 1 --2- 172.21.15.40:0/3111439827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f438c0039e0 0x7f438c003e00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4398009ac0 tx=0x7f438c005830 comp rx=0 tx=0).stop 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.395+0000 7f43ace00640 1 -- 172.21.15.40:0/3111439827 reap_dead start 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.395+0000 7f43ad800640 1 -- 172.21.15.40:0/3111439827 shutdown_connections 2024-04-26T23:02:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.395+0000 7f43ad800640 1 -- 172.21.15.40:0/3111439827 >> 172.21.15.40:0/3111439827 conn(0x7f43a80d0a10 msgr2=0x7f43a80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.395+0000 7f43ad800640 1 -- 172.21.15.40:0/3111439827 shutdown_connections 2024-04-26T23:02:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.395+0000 7f43ad800640 1 -- 172.21.15.40:0/3111439827 wait complete. 2024-04-26T23:02:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.397+0000 7f43ad800640 1 Processor -- start 2024-04-26T23:02:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.397+0000 7f43ad800640 1 -- start start 2024-04-26T23:02:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.398+0000 7f43ad800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f438c0039e0 0x7f43a815dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.398+0000 7f43ad800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a8154de0 0x7f43a80cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.398+0000 7f43ad800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43a80cf5f0 0x7f43a806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.398+0000 7f43ad800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f43a80cc1f0 con 0x7f438c0039e0 2024-04-26T23:02:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.398+0000 7f43ad800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f43a80cc070 con 0x7f43a80cf5f0 2024-04-26T23:02:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.398+0000 7f43ad800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f43a8073e70 con 0x7f43a8154de0 2024-04-26T23:02:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.398+0000 7f43a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f438c0039e0 0x7f43a815dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.398+0000 7f43a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f438c0039e0 0x7f43a815dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45458/0 (socket says 172.21.15.40:45458) 2024-04-26T23:02:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.398+0000 7f43a7e00640 1 -- 172.21.15.40:0/342148213 learned_addr learned my addr 172.21.15.40:0/342148213 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.399+0000 7f43a7e00640 1 -- 172.21.15.40:0/342148213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a8154de0 msgr2=0x7f43a80cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.399+0000 7f43a7e00640 1 --2- 172.21.15.40:0/342148213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a8154de0 0x7f43a80cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.399+0000 7f43a7e00640 1 -- 172.21.15.40:0/342148213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43a80cf5f0 msgr2=0x7f43a806ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.399+0000 7f43a7e00640 1 --2- 172.21.15.40:0/342148213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43a80cf5f0 0x7f43a806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.399+0000 7f43a7e00640 1 -- 172.21.15.40:0/342148213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a8155260 con 0x7f438c0039e0 2024-04-26T23:02:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.399+0000 7f43a7e00640 1 --2- 172.21.15.40:0/342148213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f438c0039e0 0x7f43a815dcb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4398034040 tx=0x7f4398004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.399+0000 7f43a4c00640 1 -- 172.21.15.40:0/342148213 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4398040020 con 0x7f438c0039e0 2024-04-26T23:02:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.399+0000 7f43a4c00640 1 -- 172.21.15.40:0/342148213 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4398003bc0 con 0x7f438c0039e0 2024-04-26T23:02:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.399+0000 7f43a4c00640 1 -- 172.21.15.40:0/342148213 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4398036040 con 0x7f438c0039e0 2024-04-26T23:02:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.399+0000 7f43ad800640 1 -- 172.21.15.40:0/342148213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a815e3b0 con 0x7f438c0039e0 2024-04-26T23:02:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.400+0000 7f43ad800640 1 -- 172.21.15.40:0/342148213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a815e970 con 0x7f438c0039e0 2024-04-26T23:02:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.401+0000 7f43ad800640 1 -- 172.21.15.40:0/342148213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43a80cfa10 con 0x7f438c0039e0 2024-04-26T23:02:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.402+0000 7f43a4c00640 1 -- 172.21.15.40:0/342148213 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4398043020 con 0x7f438c0039e0 2024-04-26T23:02:31.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.402+0000 7f43a4c00640 1 --2- 172.21.15.40:0/342148213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f438406c8f0 0x7f438406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.403+0000 7f43a4c00640 1 -- 172.21.15.40:0/342148213 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f43980ac2f0 con 0x7f438c0039e0 2024-04-26T23:02:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.403+0000 7f439fe00640 1 --2- 172.21.15.40:0/342148213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f438406c8f0 0x7f438406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:31.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.404+0000 7f439fe00640 1 --2- 172.21.15.40:0/342148213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f438406c8f0 0x7f438406edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f43900017e0 tx=0x7f4390009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:31.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.407+0000 7f43a4c00640 1 -- 172.21.15.40:0/342148213 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f439804a030 con 0x7f438c0039e0 2024-04-26T23:02:31.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.611+0000 7f43ad800640 1 -- 172.21.15.40:0/342148213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f43a8069370 con 0x7f438c0039e0 2024-04-26T23:02:31.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.612+0000 7f43a4c00640 1 -- 172.21.15.40:0/342148213 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f43980044e0 con 0x7f438c0039e0 2024-04-26T23:02:31.612 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:02:31.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.614+0000 7f43ad800640 1 -- 172.21.15.40:0/342148213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f438406c8f0 msgr2=0x7f438406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:31.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.614+0000 7f43ad800640 1 --2- 172.21.15.40:0/342148213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f438406c8f0 0x7f438406edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f43900017e0 tx=0x7f4390009040 comp rx=0 tx=0).stop 2024-04-26T23:02:31.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.614+0000 7f43ad800640 1 -- 172.21.15.40:0/342148213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f438c0039e0 msgr2=0x7f43a815dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:31.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.614+0000 7f43ad800640 1 --2- 172.21.15.40:0/342148213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f438c0039e0 0x7f43a815dcb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4398034040 tx=0x7f4398004730 comp rx=0 tx=0).stop 2024-04-26T23:02:31.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.614+0000 7f43ad800640 1 -- 172.21.15.40:0/342148213 shutdown_connections 2024-04-26T23:02:31.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.614+0000 7f43ad800640 1 --2- 172.21.15.40:0/342148213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f438406c8f0 0x7f438406edb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.614+0000 7f43ad800640 1 --2- 172.21.15.40:0/342148213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43a80cf5f0 0x7f43a806ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.614+0000 7f43ad800640 1 --2- 172.21.15.40:0/342148213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43a8154de0 0x7f43a80cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.614+0000 7f43ad800640 1 --2- 172.21.15.40:0/342148213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f438c0039e0 0x7f43a815dcb0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.614+0000 7f43ad800640 1 -- 172.21.15.40:0/342148213 >> 172.21.15.40:0/342148213 conn(0x7f43a80d0a10 msgr2=0x7f43a8155510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:31.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.615+0000 7f43ad800640 1 -- 172.21.15.40:0/342148213 shutdown_connections 2024-04-26T23:02:31.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.615+0000 7f43ad800640 1 -- 172.21.15.40:0/342148213 wait complete. 2024-04-26T23:02:31.634 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:02:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.851+0000 7f2d1ca00640 1 Processor -- start 2024-04-26T23:02:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.852+0000 7f2d1ca00640 1 -- start start 2024-04-26T23:02:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.853+0000 7f2d1ca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2d18073600 0x7f2d18072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.853+0000 7f2d1ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d1815fe80 0x7f2d180ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.853+0000 7f2d1ca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d181604e0 con 0x7f2d180739e0 2024-04-26T23:02:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.853+0000 7f2d1ca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2d18073220 con 0x7f2d18073600 2024-04-26T23:02:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.853+0000 7f2d1ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2d18072850 con 0x7f2d1815fe80 2024-04-26T23:02:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.854+0000 7f2d17200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d1815fe80 0x7f2d180ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.854+0000 7f2d17200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d1815fe80 0x7f2d180ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:51042/0 (socket says 172.21.15.40:51042) 2024-04-26T23:02:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.854+0000 7f2d17200640 1 -- 172.21.15.40:0/107213174 learned_addr learned my addr 172.21.15.40:0/107213174 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.854+0000 7f2d16800640 1 --2- 172.21.15.40:0/107213174 >> v2:172.21.15.103:3300/0 conn(0x7f2d18073600 0x7f2d18072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.855+0000 7f2d15e00640 1 -- 172.21.15.40:0/107213174 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 871543210 0 0) 0x7f2d181604e0 con 0x7f2d180739e0 2024-04-26T23:02:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.855+0000 7f2d15e00640 1 -- 172.21.15.40:0/107213174 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cfc003520 con 0x7f2d180739e0 2024-04-26T23:02:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.855+0000 7f2d16800640 1 -- 172.21.15.40:0/107213174 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d1815fe80 msgr2=0x7f2d180ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.855+0000 7f2d16800640 1 --2- 172.21.15.40:0/107213174 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d1815fe80 0x7f2d180ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.855+0000 7f2d16800640 1 -- 172.21.15.40:0/107213174 >> v1:172.21.15.103:6789/0 conn(0x7f2d180739e0 legacy=0x7f2d1815f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.855+0000 7f2d16800640 1 -- 172.21.15.40:0/107213174 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d18072200 con 0x7f2d18073600 2024-04-26T23:02:31.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.857+0000 7f2d16800640 1 --2- 172.21.15.40:0/107213174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d18073600 0x7f2d18072ce0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2d080097a0 tx=0x7f2d0802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd407bd73405e21c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:31.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.857+0000 7f2d15e00640 1 -- 172.21.15.40:0/107213174 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d08037070 con 0x7f2d18073600 2024-04-26T23:02:31.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.857+0000 7f2d15e00640 1 -- 172.21.15.40:0/107213174 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d08032440 con 0x7f2d18073600 2024-04-26T23:02:31.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.857+0000 7f2d15e00640 1 -- 172.21.15.40:0/107213174 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d08032810 con 0x7f2d18073600 2024-04-26T23:02:31.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.858+0000 7f2d1ca00640 1 -- 172.21.15.40:0/107213174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d18073600 msgr2=0x7f2d18072ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:31.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.858+0000 7f2d1ca00640 1 --2- 172.21.15.40:0/107213174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d18073600 0x7f2d18072ce0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2d080097a0 tx=0x7f2d0802a180 comp rx=0 tx=0).stop 2024-04-26T23:02:31.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.858+0000 7f2d1ca00640 1 -- 172.21.15.40:0/107213174 shutdown_connections 2024-04-26T23:02:31.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.858+0000 7f2d1ca00640 1 --2- 172.21.15.40:0/107213174 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d1815fe80 0x7f2d180ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.858+0000 7f2d1ca00640 1 --2- 172.21.15.40:0/107213174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d18073600 0x7f2d18072ce0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.858+0000 7f2d1ca00640 1 -- 172.21.15.40:0/107213174 >> 172.21.15.40:0/107213174 conn(0x7f2d180b2720 msgr2=0x7f2d180b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:31.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.858+0000 7f2d1ca00640 1 -- 172.21.15.40:0/107213174 shutdown_connections 2024-04-26T23:02:31.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.858+0000 7f2d1ca00640 1 -- 172.21.15.40:0/107213174 wait complete. 2024-04-26T23:02:31.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.860+0000 7f2d1ca00640 1 Processor -- start 2024-04-26T23:02:31.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.860+0000 7f2d1ca00640 1 -- start start 2024-04-26T23:02:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.861+0000 7f2d1ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d18073600 0x7f2d181551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.861+0000 7f2d1ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d180739e0 0x7f2d18157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.861+0000 7f2d1ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d1815fe80 0x7f2d18151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.861+0000 7f2d1ca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2d18071e20 con 0x7f2d18073600 2024-04-26T23:02:31.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.861+0000 7f2d1ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d18073e90 con 0x7f2d1815fe80 2024-04-26T23:02:31.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.861+0000 7f2d1ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2d18071fa0 con 0x7f2d180739e0 2024-04-26T23:02:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.862+0000 7f2d16800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d18073600 0x7f2d181551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.862+0000 7f2d16800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d18073600 0x7f2d181551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45460/0 (socket says 172.21.15.40:45460) 2024-04-26T23:02:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.862+0000 7f2d16800640 1 -- 172.21.15.40:0/655729085 learned_addr learned my addr 172.21.15.40:0/655729085 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.862+0000 7f2d17200640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d1815fe80 0x7f2d18151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.862+0000 7f2d0fe00640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d180739e0 0x7f2d18157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.862+0000 7f2d16800640 1 -- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d180739e0 msgr2=0x7f2d18157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.862+0000 7f2d16800640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d180739e0 0x7f2d18157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.862+0000 7f2d16800640 1 -- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d1815fe80 msgr2=0x7f2d18151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.862+0000 7f2d16800640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d1815fe80 0x7f2d18151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.862+0000 7f2d16800640 1 -- 172.21.15.40:0/655729085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d181556e0 con 0x7f2d18073600 2024-04-26T23:02:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.862+0000 7f2d0fe00640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d180739e0 0x7f2d18157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.863+0000 7f2d16800640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d18073600 0x7f2d181551a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2d08009890 tx=0x7f2d080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.863+0000 7f2d0ea00640 1 -- 172.21.15.40:0/655729085 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d08037050 con 0x7f2d18073600 2024-04-26T23:02:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.863+0000 7f2d0ea00640 1 -- 172.21.15.40:0/655729085 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d08043070 con 0x7f2d18073600 2024-04-26T23:02:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.863+0000 7f2d1ca00640 1 -- 172.21.15.40:0/655729085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d18156330 con 0x7f2d18073600 2024-04-26T23:02:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.863+0000 7f2d0ea00640 1 -- 172.21.15.40:0/655729085 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0803cb30 con 0x7f2d18073600 2024-04-26T23:02:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.863+0000 7f2d1ca00640 1 -- 172.21.15.40:0/655729085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d181568f0 con 0x7f2d18073600 2024-04-26T23:02:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.864+0000 7f2d1ca00640 1 -- 172.21.15.40:0/655729085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d18069370 con 0x7f2d18073600 2024-04-26T23:02:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.865+0000 7f2d0ea00640 1 -- 172.21.15.40:0/655729085 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d0803ccd0 con 0x7f2d18073600 2024-04-26T23:02:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.867+0000 7f2d0ea00640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2cf406c920 0x7f2cf406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.867+0000 7f2d0ea00640 1 -- 172.21.15.40:0/655729085 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2d080acba0 con 0x7f2d18073600 2024-04-26T23:02:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.867+0000 7f2d0fe00640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2cf406c920 0x7f2cf406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.868+0000 7f2d0fe00640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2cf406c920 0x7f2cf406ede0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2d18154600 tx=0x7f2d00009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:31.871+0000 7f2d0ea00640 1 -- 172.21.15.40:0/655729085 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2d08046360 con 0x7f2d18073600 2024-04-26T23:02:32.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.073+0000 7f2d1ca00640 1 -- 172.21.15.40:0/655729085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2d1806f170 con 0x7f2d18073600 2024-04-26T23:02:32.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.074+0000 7f2d0ea00640 1 -- 172.21.15.40:0/655729085 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f2d0807ace0 con 0x7f2d18073600 2024-04-26T23:02:32.075 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:02:32.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.077+0000 7f2d1ca00640 1 -- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2cf406c920 msgr2=0x7f2cf406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:32.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.077+0000 7f2d1ca00640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2cf406c920 0x7f2cf406ede0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2d18154600 tx=0x7f2d00009040 comp rx=0 tx=0).stop 2024-04-26T23:02:32.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.077+0000 7f2d1ca00640 1 -- 172.21.15.40:0/655729085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d18073600 msgr2=0x7f2d181551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:32.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.077+0000 7f2d1ca00640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d18073600 0x7f2d181551a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2d08009890 tx=0x7f2d080046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.077+0000 7f2d1ca00640 1 -- 172.21.15.40:0/655729085 shutdown_connections 2024-04-26T23:02:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.077+0000 7f2d1ca00640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2cf406c920 0x7f2cf406ede0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.077+0000 7f2d1ca00640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d1815fe80 0x7f2d18151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.077+0000 7f2d1ca00640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d180739e0 0x7f2d18157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.077+0000 7f2d1ca00640 1 --2- 172.21.15.40:0/655729085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d18073600 0x7f2d181551a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.077+0000 7f2d1ca00640 1 -- 172.21.15.40:0/655729085 >> 172.21.15.40:0/655729085 conn(0x7f2d180b2720 msgr2=0x7f2d180b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.077+0000 7f2d1ca00640 1 -- 172.21.15.40:0/655729085 shutdown_connections 2024-04-26T23:02:32.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.077+0000 7f2d1ca00640 1 -- 172.21.15.40:0/655729085 wait complete. 2024-04-26T23:02:32.097 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:02:32.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.299+0000 7f8e3fe00640 1 Processor -- start 2024-04-26T23:02:32.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.300+0000 7f8e3fe00640 1 -- start start 2024-04-26T23:02:32.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.301+0000 7f8e3fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e400b6220 0x7f8e400d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:32.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.301+0000 7f8e3fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e400d39f0 0x7f8e400b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:32.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.301+0000 7f8e3fe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e40162c40 con 0x7f8e400d08d0 2024-04-26T23:02:32.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.301+0000 7f8e3fe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f8e400d2d80 con 0x7f8e400b6220 2024-04-26T23:02:32.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.301+0000 7f8e3fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8e400b7c80 con 0x7f8e400d39f0 2024-04-26T23:02:32.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.302+0000 7f8e3ea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e400b6220 0x7f8e400d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:32.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.302+0000 7f8e3ea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e400b6220 0x7f8e400d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45476/0 (socket says 172.21.15.40:45476) 2024-04-26T23:02:32.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.302+0000 7f8e3ea00640 1 -- 172.21.15.40:0/3935131007 learned_addr learned my addr 172.21.15.40:0/3935131007 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:32.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.302+0000 7f8e3f400640 1 --2- 172.21.15.40:0/3935131007 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e400d39f0 0x7f8e400b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:32.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.302+0000 7f8e3ea00640 1 -- 172.21.15.40:0/3935131007 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e400d39f0 msgr2=0x7f8e400b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:32.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.302+0000 7f8e3ea00640 1 --2- 172.21.15.40:0/3935131007 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e400d39f0 0x7f8e400b8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.302+0000 7f8e3ea00640 1 -- 172.21.15.40:0/3935131007 >> v1:172.21.15.103:6789/0 conn(0x7f8e400d08d0 legacy=0x7f8e400d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:32.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.302+0000 7f8e3ea00640 1 -- 172.21.15.40:0/3935131007 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e400b7f60 con 0x7f8e400b6220 2024-04-26T23:02:32.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.304+0000 7f8e3ea00640 1 --2- 172.21.15.40:0/3935131007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e400b6220 0x7f8e400d0390 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8e30009870 tx=0x7f8e3002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e502e240aca02530 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:32.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.304+0000 7f8e3e000640 1 -- 172.21.15.40:0/3935131007 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30037070 con 0x7f8e400b6220 2024-04-26T23:02:32.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.304+0000 7f8e3e000640 1 -- 172.21.15.40:0/3935131007 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e30032440 con 0x7f8e400b6220 2024-04-26T23:02:32.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.304+0000 7f8e3e000640 1 -- 172.21.15.40:0/3935131007 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30032810 con 0x7f8e400b6220 2024-04-26T23:02:32.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.305+0000 7f8e3fe00640 1 -- 172.21.15.40:0/3935131007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e400b6220 msgr2=0x7f8e400d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:32.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.305+0000 7f8e3fe00640 1 --2- 172.21.15.40:0/3935131007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e400b6220 0x7f8e400d0390 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8e30009870 tx=0x7f8e3002a270 comp rx=0 tx=0).stop 2024-04-26T23:02:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.305+0000 7f8e3fe00640 1 -- 172.21.15.40:0/3935131007 shutdown_connections 2024-04-26T23:02:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.305+0000 7f8e3fe00640 1 --2- 172.21.15.40:0/3935131007 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e400d39f0 0x7f8e400b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.305+0000 7f8e3fe00640 1 --2- 172.21.15.40:0/3935131007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e400b6220 0x7f8e400d0390 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.305+0000 7f8e3fe00640 1 -- 172.21.15.40:0/3935131007 >> 172.21.15.40:0/3935131007 conn(0x7f8e400d4d90 msgr2=0x7f8e400d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.305+0000 7f8e3fe00640 1 -- 172.21.15.40:0/3935131007 shutdown_connections 2024-04-26T23:02:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.305+0000 7f8e3fe00640 1 -- 172.21.15.40:0/3935131007 wait complete. 2024-04-26T23:02:32.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.307+0000 7f8e3fe00640 1 Processor -- start 2024-04-26T23:02:32.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.307+0000 7f8e3fe00640 1 -- start start 2024-04-26T23:02:32.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.308+0000 7f8e3fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e400b6220 0x7f8e40155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:32.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.308+0000 7f8e3fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e400d08d0 0x7f8e40155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:32.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.308+0000 7f8e3fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e400d39f0 0x7f8e401f8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:32.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.308+0000 7f8e3fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8e400b9260 con 0x7f8e400b6220 2024-04-26T23:02:32.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.308+0000 7f8e3fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e400b8520 con 0x7f8e400d08d0 2024-04-26T23:02:32.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.308+0000 7f8e3fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8e400b93e0 con 0x7f8e400d39f0 2024-04-26T23:02:32.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.309+0000 7f8e3ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e400b6220 0x7f8e40155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:32.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.309+0000 7f8e3ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e400b6220 0x7f8e40155930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45484/0 (socket says 172.21.15.40:45484) 2024-04-26T23:02:32.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.309+0000 7f8e3ea00640 1 -- 172.21.15.40:0/993960612 learned_addr learned my addr 172.21.15.40:0/993960612 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:32.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.309+0000 7f8e3ea00640 1 -- 172.21.15.40:0/993960612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e400d39f0 msgr2=0x7f8e401f8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:32.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.309+0000 7f8e3ea00640 1 --2- 172.21.15.40:0/993960612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e400d39f0 0x7f8e401f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.309+0000 7f8e3ea00640 1 -- 172.21.15.40:0/993960612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e400d08d0 msgr2=0x7f8e40155e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:32.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.309+0000 7f8e3ea00640 1 --2- 172.21.15.40:0/993960612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e400d08d0 0x7f8e40155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.309+0000 7f8e3ea00640 1 -- 172.21.15.40:0/993960612 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4015b410 con 0x7f8e400b6220 2024-04-26T23:02:32.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.310+0000 7f8e3ea00640 1 --2- 172.21.15.40:0/993960612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e400b6220 0x7f8e40155930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8e3002a2b0 tx=0x7f8e300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:32.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.310+0000 7f8e37400640 1 -- 172.21.15.40:0/993960612 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30041020 con 0x7f8e400b6220 2024-04-26T23:02:32.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.310+0000 7f8e37400640 1 -- 172.21.15.40:0/993960612 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e300306c0 con 0x7f8e400b6220 2024-04-26T23:02:32.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.310+0000 7f8e3fe00640 1 -- 172.21.15.40:0/993960612 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e4015b6d0 con 0x7f8e400b6220 2024-04-26T23:02:32.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.310+0000 7f8e37400640 1 -- 172.21.15.40:0/993960612 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30037040 con 0x7f8e400b6220 2024-04-26T23:02:32.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.310+0000 7f8e3fe00640 1 -- 172.21.15.40:0/993960612 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e4015bc90 con 0x7f8e400b6220 2024-04-26T23:02:32.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.311+0000 7f8e3fe00640 1 -- 172.21.15.40:0/993960612 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e40069370 con 0x7f8e400b6220 2024-04-26T23:02:32.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.312+0000 7f8e37400640 1 -- 172.21.15.40:0/993960612 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8e30044020 con 0x7f8e400b6220 2024-04-26T23:02:32.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.313+0000 7f8e37400640 1 --2- 172.21.15.40:0/993960612 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8e1c06c8f0 0x7f8e1c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:32.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.313+0000 7f8e37400640 1 -- 172.21.15.40:0/993960612 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8e300ac770 con 0x7f8e400b6220 2024-04-26T23:02:32.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.314+0000 7f8e37e00640 1 --2- 172.21.15.40:0/993960612 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8e1c06c8f0 0x7f8e1c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:32.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.315+0000 7f8e37e00640 1 --2- 172.21.15.40:0/993960612 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8e1c06c8f0 0x7f8e1c06edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8e28000f60 tx=0x7f8e280079d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:32.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.320+0000 7f8e37400640 1 -- 172.21.15.40:0/993960612 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8e3004b030 con 0x7f8e400b6220 2024-04-26T23:02:32.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.558+0000 7f8e3fe00640 1 -- 172.21.15.40:0/993960612 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e400d30c0 con 0x7f8e400b6220 2024-04-26T23:02:32.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.559+0000 7f8e37400640 1 -- 172.21.15.40:0/993960612 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f8e3007a790 con 0x7f8e400b6220 2024-04-26T23:02:32.560 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:32.560 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1253648754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1253648754},{"type":"v1","addr":"172.21.15.103:6836","nonce":1253648754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3239652611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":3239652611},{"type":"v1","addr":"172.21.15.103:6837","nonce":3239652611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3670144900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3670144900},{"type":"v1","addr":"172.21.15.40:6833","nonce":3670144900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:01:05.171276+0000","modified":"2024-04-26T23:01:33.399801+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3989190340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3989190340},{"type":"v1","addr":"172.21.15.40:6835","nonce":3989190340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-26T23:02:32.560 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 9 2024-04-26T23:02:32.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.563+0000 7f8e3fe00640 1 -- 172.21.15.40:0/993960612 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8e1c06c8f0 msgr2=0x7f8e1c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:32.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.563+0000 7f8e3fe00640 1 --2- 172.21.15.40:0/993960612 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8e1c06c8f0 0x7f8e1c06edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8e28000f60 tx=0x7f8e280079d0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.563+0000 7f8e3fe00640 1 -- 172.21.15.40:0/993960612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e400b6220 msgr2=0x7f8e40155930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:32.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.563+0000 7f8e3fe00640 1 --2- 172.21.15.40:0/993960612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e400b6220 0x7f8e40155930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8e3002a2b0 tx=0x7f8e300046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.563+0000 7f8e3fe00640 1 -- 172.21.15.40:0/993960612 shutdown_connections 2024-04-26T23:02:32.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.563+0000 7f8e3fe00640 1 --2- 172.21.15.40:0/993960612 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8e1c06c8f0 0x7f8e1c06edb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.563+0000 7f8e3fe00640 1 --2- 172.21.15.40:0/993960612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e400d39f0 0x7f8e401f8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.563+0000 7f8e3fe00640 1 --2- 172.21.15.40:0/993960612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e400d08d0 0x7f8e40155e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.563+0000 7f8e3fe00640 1 --2- 172.21.15.40:0/993960612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e400b6220 0x7f8e40155930 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.563+0000 7f8e3fe00640 1 -- 172.21.15.40:0/993960612 >> 172.21.15.40:0/993960612 conn(0x7f8e400d4d90 msgr2=0x7f8e400d32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:32.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.563+0000 7f8e3fe00640 1 -- 172.21.15.40:0/993960612 shutdown_connections 2024-04-26T23:02:32.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.564+0000 7f8e3fe00640 1 -- 172.21.15.40:0/993960612 wait complete. 2024-04-26T23:02:32.581 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:32.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.786+0000 7f3e96a00640 1 Processor -- start 2024-04-26T23:02:32.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.786+0000 7f3e96a00640 1 -- start start 2024-04-26T23:02:32.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.788+0000 7f3e96a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3e90071fe0 0x7f3e900afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:32.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.788+0000 7f3e96a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e900d0820 con 0x7f3e900af7a0 2024-04-26T23:02:32.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.788+0000 7f3e96a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e901614c0 con 0x7f3e9015ce60 2024-04-26T23:02:32.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.788+0000 7f3e96a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f3e900cc450 con 0x7f3e90071fe0 2024-04-26T23:02:32.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.788+0000 7f3e94c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3e900af7a0 0x7f3e90073760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35836/0 (socket says 172.21.15.40:35836) 2024-04-26T23:02:32.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.788+0000 7f3e94c00640 1 -- 172.21.15.40:0/4191917646 learned_addr learned my addr 172.21.15.40:0/4191917646 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:32.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.788+0000 7f3e96000640 1 --2- 172.21.15.40:0/4191917646 >> v2:172.21.15.40:3300/0 conn(0x7f3e90071fe0 0x7f3e900afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:32.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.789+0000 7f3e96000640 1 -- 172.21.15.40:0/4191917646 >> v1:172.21.15.103:6789/0 conn(0x7f3e9015ce60 legacy=0x7f3e900ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:32.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.789+0000 7f3e8fe00640 1 -- 172.21.15.40:0/4191917646 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 68595370 0 0) 0x7f3e900d0820 con 0x7f3e900af7a0 2024-04-26T23:02:32.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.789+0000 7f3e96000640 1 -- 172.21.15.40:0/4191917646 >> v1:172.21.15.40:6789/0 conn(0x7f3e900af7a0 legacy=0x7f3e90073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:32.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.789+0000 7f3e96000640 1 -- 172.21.15.40:0/4191917646 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e90161760 con 0x7f3e90071fe0 2024-04-26T23:02:32.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.790+0000 7f3e96000640 1 --2- 172.21.15.40:0/4191917646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e90071fe0 0x7f3e900afc60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3e88009d90 tx=0x7f3e8802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbd05a9be8b05d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:32.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.790+0000 7f3e8fe00640 1 -- 172.21.15.40:0/4191917646 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e88007740 con 0x7f3e90071fe0 2024-04-26T23:02:32.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.790+0000 7f3e8fe00640 1 -- 172.21.15.40:0/4191917646 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e880078e0 con 0x7f3e90071fe0 2024-04-26T23:02:32.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.790+0000 7f3e8fe00640 1 -- 172.21.15.40:0/4191917646 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e88007cb0 con 0x7f3e90071fe0 2024-04-26T23:02:32.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.791+0000 7f3e96a00640 1 -- 172.21.15.40:0/4191917646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e90071fe0 msgr2=0x7f3e900afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:32.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.791+0000 7f3e96a00640 1 --2- 172.21.15.40:0/4191917646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e90071fe0 0x7f3e900afc60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3e88009d90 tx=0x7f3e8802a660 comp rx=0 tx=0).stop 2024-04-26T23:02:32.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.791+0000 7f3e96a00640 1 -- 172.21.15.40:0/4191917646 shutdown_connections 2024-04-26T23:02:32.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.791+0000 7f3e96a00640 1 --2- 172.21.15.40:0/4191917646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e90071fe0 0x7f3e900afc60 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.791+0000 7f3e96a00640 1 -- 172.21.15.40:0/4191917646 >> 172.21.15.40:0/4191917646 conn(0x7f3e900d0a10 msgr2=0x7f3e900d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:32.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.791+0000 7f3e96a00640 1 -- 172.21.15.40:0/4191917646 shutdown_connections 2024-04-26T23:02:32.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.791+0000 7f3e96a00640 1 -- 172.21.15.40:0/4191917646 wait complete. 2024-04-26T23:02:32.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.793+0000 7f3e96a00640 1 Processor -- start 2024-04-26T23:02:32.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.793+0000 7f3e96a00640 1 -- start start 2024-04-26T23:02:32.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.794+0000 7f3e96a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e90071fe0 0x7f3e9014c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:32.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.794+0000 7f3e96a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e900af7a0 0x7f3e900b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:32.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.794+0000 7f3e96a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9015ce60 0x7f3e901559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:32.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.794+0000 7f3e96a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3e900cc1f0 con 0x7f3e9015ce60 2024-04-26T23:02:32.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.794+0000 7f3e96a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3e900cc070 con 0x7f3e90071fe0 2024-04-26T23:02:32.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.794+0000 7f3e96a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3e90073e70 con 0x7f3e900af7a0 2024-04-26T23:02:32.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.794+0000 7f3e96000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9015ce60 0x7f3e901559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:32.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.794+0000 7f3e96000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9015ce60 0x7f3e901559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45490/0 (socket says 172.21.15.40:45490) 2024-04-26T23:02:32.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.794+0000 7f3e96000640 1 -- 172.21.15.40:0/4267572512 learned_addr learned my addr 172.21.15.40:0/4267572512 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.795+0000 7f3e94c00640 1 --2- 172.21.15.40:0/4267572512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e900af7a0 0x7f3e900b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.795+0000 7f3e96000640 1 -- 172.21.15.40:0/4267572512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e900af7a0 msgr2=0x7f3e900b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.795+0000 7f3e96000640 1 --2- 172.21.15.40:0/4267572512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e900af7a0 0x7f3e900b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.795+0000 7f3e96000640 1 -- 172.21.15.40:0/4267572512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e90071fe0 msgr2=0x7f3e9014c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.795+0000 7f3e96000640 1 --2- 172.21.15.40:0/4267572512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e90071fe0 0x7f3e9014c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.795+0000 7f3e96000640 1 -- 172.21.15.40:0/4267572512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9015d910 con 0x7f3e9015ce60 2024-04-26T23:02:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.795+0000 7f3e96000640 1 --2- 172.21.15.40:0/4267572512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9015ce60 0x7f3e901559c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3e88009d90 tx=0x7f3e88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.795+0000 7f3e8d600640 1 -- 172.21.15.40:0/4267572512 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e88040070 con 0x7f3e9015ce60 2024-04-26T23:02:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.796+0000 7f3e8d600640 1 -- 172.21.15.40:0/4267572512 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e88031c90 con 0x7f3e9015ce60 2024-04-26T23:02:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.796+0000 7f3e8d600640 1 -- 172.21.15.40:0/4267572512 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8803c640 con 0x7f3e9015ce60 2024-04-26T23:02:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.796+0000 7f3e96a00640 1 -- 172.21.15.40:0/4267572512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e9014cb30 con 0x7f3e9015ce60 2024-04-26T23:02:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.796+0000 7f3e96a00640 1 -- 172.21.15.40:0/4267572512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e900b42a0 con 0x7f3e9015ce60 2024-04-26T23:02:32.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.797+0000 7f3e96a00640 1 -- 172.21.15.40:0/4267572512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e90069370 con 0x7f3e9015ce60 2024-04-26T23:02:32.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.798+0000 7f3e8d600640 1 -- 172.21.15.40:0/4267572512 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3e88039040 con 0x7f3e9015ce60 2024-04-26T23:02:32.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.798+0000 7f3e8d600640 1 --2- 172.21.15.40:0/4267572512 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e7006c8f0 0x7f3e7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:32.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.799+0000 7f3e8d600640 1 -- 172.21.15.40:0/4267572512 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f3e880ad2a0 con 0x7f3e9015ce60 2024-04-26T23:02:32.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.799+0000 7f3e95600640 1 --2- 172.21.15.40:0/4267572512 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e7006c8f0 0x7f3e7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:32.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.800+0000 7f3e95600640 1 --2- 172.21.15.40:0/4267572512 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e7006c8f0 0x7f3e7006edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3e84000f60 tx=0x7f3e840079d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:32.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:32.803+0000 7f3e8d600640 1 -- 172.21.15.40:0/4267572512 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3e88039330 con 0x7f3e9015ce60 2024-04-26T23:02:33.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.008+0000 7f3e96a00640 1 -- 172.21.15.40:0/4267572512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e9015dae0 con 0x7f3e9015ce60 2024-04-26T23:02:33.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.011+0000 7f3e8d600640 1 -- 172.21.15.40:0/4267572512 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18812 (secure 0 0 0) 0x7f3e8807b2f0 con 0x7f3e9015ce60 2024-04-26T23:02:33.011 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:33.011 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:46.975683+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.013+0000 7f3e96a00640 1 -- 172.21.15.40:0/4267572512 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e7006c8f0 msgr2=0x7f3e7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.013+0000 7f3e96a00640 1 --2- 172.21.15.40:0/4267572512 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e7006c8f0 0x7f3e7006edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3e84000f60 tx=0x7f3e840079d0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.014+0000 7f3e96a00640 1 -- 172.21.15.40:0/4267572512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9015ce60 msgr2=0x7f3e901559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.014+0000 7f3e96a00640 1 --2- 172.21.15.40:0/4267572512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9015ce60 0x7f3e901559c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3e88009d90 tx=0x7f3e88004a40 comp rx=0 tx=0).stop 2024-04-26T23:02:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.014+0000 7f3e96a00640 1 -- 172.21.15.40:0/4267572512 shutdown_connections 2024-04-26T23:02:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.014+0000 7f3e96a00640 1 --2- 172.21.15.40:0/4267572512 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e7006c8f0 0x7f3e7006edb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.014+0000 7f3e96a00640 1 --2- 172.21.15.40:0/4267572512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9015ce60 0x7f3e901559c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.014+0000 7f3e96a00640 1 --2- 172.21.15.40:0/4267572512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e900af7a0 0x7f3e900b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.014+0000 7f3e96a00640 1 --2- 172.21.15.40:0/4267572512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e90071fe0 0x7f3e9014c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.014+0000 7f3e96a00640 1 -- 172.21.15.40:0/4267572512 >> 172.21.15.40:0/4267572512 conn(0x7f3e900d0a10 msgr2=0x7f3e900aefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.014+0000 7f3e96a00640 1 -- 172.21.15.40:0/4267572512 shutdown_connections 2024-04-26T23:02:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.014+0000 7f3e96a00640 1 -- 172.21.15.40:0/4267572512 wait complete. 2024-04-26T23:02:33.033 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T23:02:33.033 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:02:33.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.245+0000 7f067f000640 1 Processor -- start 2024-04-26T23:02:33.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.245+0000 7f067f000640 1 -- start start 2024-04-26T23:02:33.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.246+0000 7f067f000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0678156460 0x7f06780cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:33.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.246+0000 7f067f000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f06780cba10 0x7f067806f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:33.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.246+0000 7f067f000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0678156840 con 0x7f067806fcd0 2024-04-26T23:02:33.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.246+0000 7f067f000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f06780cbf20 con 0x7f06780cba10 2024-04-26T23:02:33.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.246+0000 7f067f000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f06780cc200 con 0x7f0678156460 2024-04-26T23:02:33.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.246+0000 7f0677400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f06780cba10 0x7f067806f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:33.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.246+0000 7f067d000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f067806fcd0 0x7f067815f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35846/0 (socket says 172.21.15.40:35846) 2024-04-26T23:02:33.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.246+0000 7f067d000640 1 -- 172.21.15.40:0/1114489823 learned_addr learned my addr 172.21.15.40:0/1114489823 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:33.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.246+0000 7f0677400640 1 -- 172.21.15.40:0/1114489823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0678156460 msgr2=0x7f06780cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:33.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.246+0000 7f0677400640 1 --2- 172.21.15.40:0/1114489823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0678156460 0x7f06780cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.246+0000 7f0677400640 1 -- 172.21.15.40:0/1114489823 >> v1:172.21.15.40:6789/0 conn(0x7f067806fcd0 legacy=0x7f067815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:33.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.247+0000 7f0677400640 1 -- 172.21.15.40:0/1114489823 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06780cc4e0 con 0x7f06780cba10 2024-04-26T23:02:33.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.247+0000 7f0676a00640 1 -- 172.21.15.40:0/1114489823 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 731013497 0 0) 0x7f0678156840 con 0x7f067806fcd0 2024-04-26T23:02:33.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.248+0000 7f0677400640 1 --2- 172.21.15.40:0/1114489823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06780cba10 0x7f067806f790 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f06640097a0 tx=0x7f066402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed7c00ff1a97bc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:33.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.248+0000 7f0676a00640 1 -- 172.21.15.40:0/1114489823 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664037070 con 0x7f06780cba10 2024-04-26T23:02:33.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.248+0000 7f0676a00640 1 -- 172.21.15.40:0/1114489823 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0664032440 con 0x7f06780cba10 2024-04-26T23:02:33.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.248+0000 7f0676a00640 1 -- 172.21.15.40:0/1114489823 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664032810 con 0x7f06780cba10 2024-04-26T23:02:33.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.249+0000 7f067f000640 1 -- 172.21.15.40:0/1114489823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06780cba10 msgr2=0x7f067806f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:33.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.249+0000 7f067f000640 1 --2- 172.21.15.40:0/1114489823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06780cba10 0x7f067806f790 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f06640097a0 tx=0x7f066402a180 comp rx=0 tx=0).stop 2024-04-26T23:02:33.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.249+0000 7f067f000640 1 -- 172.21.15.40:0/1114489823 shutdown_connections 2024-04-26T23:02:33.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.249+0000 7f067f000640 1 --2- 172.21.15.40:0/1114489823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06780cba10 0x7f067806f790 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.249+0000 7f067f000640 1 --2- 172.21.15.40:0/1114489823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0678156460 0x7f06780cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.249+0000 7f067f000640 1 -- 172.21.15.40:0/1114489823 >> 172.21.15.40:0/1114489823 conn(0x7f06780d0be0 msgr2=0x7f06780d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:33.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.249+0000 7f067f000640 1 -- 172.21.15.40:0/1114489823 shutdown_connections 2024-04-26T23:02:33.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.249+0000 7f067f000640 1 -- 172.21.15.40:0/1114489823 wait complete. 2024-04-26T23:02:33.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.251+0000 7f067f000640 1 Processor -- start 2024-04-26T23:02:33.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.251+0000 7f067f000640 1 -- start start 2024-04-26T23:02:33.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.253+0000 7f067f000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f067806fcd0 0x7f0678148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:33.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.253+0000 7f067f000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06780cba10 0x7f06780af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:33.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.253+0000 7f067f000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0678156460 0x7f06780c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:33.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.253+0000 7f067f000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f067806e260 con 0x7f06780cba10 2024-04-26T23:02:33.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.253+0000 7f067f000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0678072700 con 0x7f067806fcd0 2024-04-26T23:02:33.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.253+0000 7f067f000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f067806e3e0 con 0x7f0678156460 2024-04-26T23:02:33.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.253+0000 7f0677400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06780cba10 0x7f06780af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:33.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.253+0000 7f0677400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06780cba10 0x7f06780af2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45498/0 (socket says 172.21.15.40:45498) 2024-04-26T23:02:33.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.253+0000 7f0677400640 1 -- 172.21.15.40:0/2223494328 learned_addr learned my addr 172.21.15.40:0/2223494328 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:33.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.254+0000 7f0677400640 1 -- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0678156460 msgr2=0x7f06780c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:33.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.254+0000 7f0677400640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0678156460 0x7f06780c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.254+0000 7f0677400640 1 -- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f067806fcd0 msgr2=0x7f0678148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:33.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.254+0000 7f0677e00640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f067806fcd0 0x7f0678148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:33.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.254+0000 7f0677400640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f067806fcd0 0x7f0678148f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.254+0000 7f0677400640 1 -- 172.21.15.40:0/2223494328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06780afc10 con 0x7f06780cba10 2024-04-26T23:02:33.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.254+0000 7f0677e00640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f067806fcd0 0x7f0678148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:33.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.254+0000 7f0677400640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06780cba10 0x7f06780af2b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f06640098c0 tx=0x7f06640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:33.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.254+0000 7f066fe00640 1 -- 172.21.15.40:0/2223494328 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664041070 con 0x7f06780cba10 2024-04-26T23:02:33.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.254+0000 7f066fe00640 1 -- 172.21.15.40:0/2223494328 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f066403c930 con 0x7f06780cba10 2024-04-26T23:02:33.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.254+0000 7f066fe00640 1 -- 172.21.15.40:0/2223494328 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664037040 con 0x7f06780cba10 2024-04-26T23:02:33.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.254+0000 7f067f000640 1 -- 172.21.15.40:0/2223494328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06780afef0 con 0x7f06780cba10 2024-04-26T23:02:33.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.255+0000 7f067f000640 1 -- 172.21.15.40:0/2223494328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06781f5e70 con 0x7f06780cba10 2024-04-26T23:02:33.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.256+0000 7f067f000640 1 -- 172.21.15.40:0/2223494328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06780694f0 con 0x7f06780cba10 2024-04-26T23:02:33.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.256+0000 7f066fe00640 1 -- 172.21.15.40:0/2223494328 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0664044070 con 0x7f06780cba10 2024-04-26T23:02:33.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.257+0000 7f066fe00640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f065806c920 0x7f065806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:33.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.257+0000 7f066fe00640 1 -- 172.21.15.40:0/2223494328 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f06640acc00 con 0x7f06780cba10 2024-04-26T23:02:33.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.258+0000 7f0677e00640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f065806c920 0x7f065806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:33.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.259+0000 7f0677e00640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f065806c920 0x7f065806ede0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0668007eb0 tx=0x7f06680073c0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:33.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.262+0000 7f066fe00640 1 -- 172.21.15.40:0/2223494328 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f066407acb0 con 0x7f06780cba10 2024-04-26T23:02:33.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.496+0000 7f067f000640 1 -- 172.21.15.40:0/2223494328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0678151940 con 0x7f06780cba10 2024-04-26T23:02:33.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.497+0000 7f066fe00640 1 -- 172.21.15.40:0/2223494328 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f066407a5b0 con 0x7f06780cba10 2024-04-26T23:02:33.497 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:33.497 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1253648754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1253648754},{"type":"v1","addr":"172.21.15.103:6836","nonce":1253648754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3239652611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":3239652611},{"type":"v1","addr":"172.21.15.103:6837","nonce":3239652611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3670144900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3670144900},{"type":"v1","addr":"172.21.15.40:6833","nonce":3670144900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:01:05.171276+0000","modified":"2024-04-26T23:01:33.399801+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3989190340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3989190340},{"type":"v1","addr":"172.21.15.40:6835","nonce":3989190340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-26T23:02:33.497 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 9 2024-04-26T23:02:33.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.499+0000 7f067f000640 1 -- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f065806c920 msgr2=0x7f065806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:33.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.499+0000 7f067f000640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f065806c920 0x7f065806ede0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0668007eb0 tx=0x7f06680073c0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.499+0000 7f067f000640 1 -- 172.21.15.40:0/2223494328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06780cba10 msgr2=0x7f06780af2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:33.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.499+0000 7f067f000640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06780cba10 0x7f06780af2b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f06640098c0 tx=0x7f06640046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.499+0000 7f067f000640 1 -- 172.21.15.40:0/2223494328 shutdown_connections 2024-04-26T23:02:33.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.499+0000 7f067f000640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f065806c920 0x7f065806ede0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.499+0000 7f067f000640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0678156460 0x7f06780c86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.499+0000 7f067f000640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06780cba10 0x7f06780af2b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.499+0000 7f067f000640 1 --2- 172.21.15.40:0/2223494328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f067806fcd0 0x7f0678148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.499+0000 7f067f000640 1 -- 172.21.15.40:0/2223494328 >> 172.21.15.40:0/2223494328 conn(0x7f06780d0be0 msgr2=0x7f06780ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:33.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.499+0000 7f067f000640 1 -- 172.21.15.40:0/2223494328 shutdown_connections 2024-04-26T23:02:33.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.499+0000 7f067f000640 1 -- 172.21.15.40:0/2223494328 wait complete. 2024-04-26T23:02:33.516 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:02:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.718+0000 7f1462000640 1 Processor -- start 2024-04-26T23:02:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.719+0000 7f1462000640 1 -- start start 2024-04-26T23:02:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.720+0000 7f1462000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f145c0c6f10 0x7f145c0c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.720+0000 7f1462000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f145c0c6010 0x7f145c158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.721+0000 7f1462000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f145c0cce10 con 0x7f145c0cc980 2024-04-26T23:02:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.721+0000 7f1462000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f145c0c72f0 con 0x7f145c0c6f10 2024-04-26T23:02:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.721+0000 7f1462000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f145c0cb860 con 0x7f145c0c6010 2024-04-26T23:02:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.721+0000 7f1460c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f145c0c6f10 0x7f145c0c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.721+0000 7f1460c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f145c0c6f10 0x7f145c0c5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45502/0 (socket says 172.21.15.40:45502) 2024-04-26T23:02:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.721+0000 7f1460c00640 1 -- 172.21.15.40:0/1599669732 learned_addr learned my addr 172.21.15.40:0/1599669732 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:33.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.721+0000 7f1460c00640 1 -- 172.21.15.40:0/1599669732 >> v2:172.21.15.103:3300/0 conn(0x7f145c0c6010 msgr2=0x7f145c158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:33.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.721+0000 7f1461600640 1 --2- 172.21.15.40:0/1599669732 >> v2:172.21.15.103:3300/0 conn(0x7f145c0c6010 0x7f145c158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:33.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.721+0000 7f1460c00640 1 --2- 172.21.15.40:0/1599669732 >> v2:172.21.15.103:3300/0 conn(0x7f145c0c6010 0x7f145c158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.721+0000 7f1460c00640 1 -- 172.21.15.40:0/1599669732 >> v1:172.21.15.103:6789/0 conn(0x7f145c0cc980 legacy=0x7f145c0cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:33.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.721+0000 7f1460c00640 1 -- 172.21.15.40:0/1599669732 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145c0cd0f0 con 0x7f145c0c6f10 2024-04-26T23:02:33.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.723+0000 7f1460c00640 1 --2- 172.21.15.40:0/1599669732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f145c0c6f10 0x7f145c0c5ad0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f145c0c64a0 tx=0x7f145002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b87874eaf6ca2db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:33.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.723+0000 7f145b400640 1 -- 172.21.15.40:0/1599669732 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1450037070 con 0x7f145c0c6f10 2024-04-26T23:02:33.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.723+0000 7f145b400640 1 -- 172.21.15.40:0/1599669732 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1450032440 con 0x7f145c0c6f10 2024-04-26T23:02:33.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.723+0000 7f145b400640 1 -- 172.21.15.40:0/1599669732 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1450032810 con 0x7f145c0c6f10 2024-04-26T23:02:33.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.723+0000 7f1462000640 1 -- 172.21.15.40:0/1599669732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f145c0c6f10 msgr2=0x7f145c0c5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:33.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.723+0000 7f1462000640 1 --2- 172.21.15.40:0/1599669732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f145c0c6f10 0x7f145c0c5ad0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f145c0c64a0 tx=0x7f145002a1a0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.724+0000 7f1462000640 1 -- 172.21.15.40:0/1599669732 shutdown_connections 2024-04-26T23:02:33.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.724+0000 7f1462000640 1 --2- 172.21.15.40:0/1599669732 >> v2:172.21.15.103:3300/0 conn(0x7f145c0c6010 0x7f145c158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.724+0000 7f1462000640 1 --2- 172.21.15.40:0/1599669732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f145c0c6f10 0x7f145c0c5ad0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.724+0000 7f1462000640 1 -- 172.21.15.40:0/1599669732 >> 172.21.15.40:0/1599669732 conn(0x7f145c0d0a10 msgr2=0x7f145c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:33.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.724+0000 7f1462000640 1 -- 172.21.15.40:0/1599669732 shutdown_connections 2024-04-26T23:02:33.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.724+0000 7f1462000640 1 -- 172.21.15.40:0/1599669732 wait complete. 2024-04-26T23:02:33.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.726+0000 7f1462000640 1 Processor -- start 2024-04-26T23:02:33.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.726+0000 7f1462000640 1 -- start start 2024-04-26T23:02:33.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.727+0000 7f1462000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f145c0c6010 0x7f145c1f8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:33.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.727+0000 7f1462000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f145c0c6f10 0x7f145c157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:33.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.727+0000 7f1462000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f145c1576a0 0x7f145c0b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:33.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.727+0000 7f1462000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f145c0cc400 con 0x7f145c0c6010 2024-04-26T23:02:33.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.727+0000 7f1462000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f145c0cb1a0 con 0x7f145c1576a0 2024-04-26T23:02:33.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.727+0000 7f1462000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f145c0cc580 con 0x7f145c0c6f10 2024-04-26T23:02:33.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.727+0000 7f1460c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f145c0c6010 0x7f145c1f8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:33.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.727+0000 7f1460c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f145c0c6010 0x7f145c1f8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45516/0 (socket says 172.21.15.40:45516) 2024-04-26T23:02:33.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.727+0000 7f1460c00640 1 -- 172.21.15.40:0/625354545 learned_addr learned my addr 172.21.15.40:0/625354545 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:33.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.728+0000 7f145be00640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f145c0c6f10 0x7f145c157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:33.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.728+0000 7f1461600640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f145c1576a0 0x7f145c0b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:33.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.728+0000 7f1460c00640 1 -- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f145c0c6f10 msgr2=0x7f145c157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:33.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.728+0000 7f1460c00640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f145c0c6f10 0x7f145c157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.728+0000 7f1460c00640 1 -- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f145c1576a0 msgr2=0x7f145c0b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:33.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.728+0000 7f1460c00640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f145c1576a0 0x7f145c0b0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.728+0000 7f1460c00640 1 -- 172.21.15.40:0/625354545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145c0ca0b0 con 0x7f145c0c6010 2024-04-26T23:02:33.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.728+0000 7f1461600640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f145c1576a0 0x7f145c0b0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:33.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.728+0000 7f145be00640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f145c0c6f10 0x7f145c157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:33.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.728+0000 7f1460c00640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f145c0c6010 0x7f145c1f8d90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1450009960 tx=0x7f14500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:33.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.728+0000 7f1458c00640 1 -- 172.21.15.40:0/625354545 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1450041020 con 0x7f145c0c6010 2024-04-26T23:02:33.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.729+0000 7f1458c00640 1 -- 172.21.15.40:0/625354545 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1450030820 con 0x7f145c0c6010 2024-04-26T23:02:33.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.729+0000 7f1458c00640 1 -- 172.21.15.40:0/625354545 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1450037040 con 0x7f145c0c6010 2024-04-26T23:02:33.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.729+0000 7f1462000640 1 -- 172.21.15.40:0/625354545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f145c0ca350 con 0x7f145c0c6010 2024-04-26T23:02:33.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.729+0000 7f1462000640 1 -- 172.21.15.40:0/625354545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f145c0ca910 con 0x7f145c0c6010 2024-04-26T23:02:33.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.730+0000 7f1462000640 1 -- 172.21.15.40:0/625354545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f145c069370 con 0x7f145c0c6010 2024-04-26T23:02:33.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.731+0000 7f1458c00640 1 -- 172.21.15.40:0/625354545 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1450044020 con 0x7f145c0c6010 2024-04-26T23:02:33.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.732+0000 7f1458c00640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f143c06c8f0 0x7f143c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:33.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.732+0000 7f1458c00640 1 -- 172.21.15.40:0/625354545 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f14500ac840 con 0x7f145c0c6010 2024-04-26T23:02:33.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.733+0000 7f145be00640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f143c06c8f0 0x7f143c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:33.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.734+0000 7f145be00640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f143c06c8f0 0x7f143c06edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f145c0b1a00 tx=0x7f1448008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:33.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.737+0000 7f1458c00640 1 -- 172.21.15.40:0/625354545 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f145004b030 con 0x7f145c0c6010 2024-04-26T23:02:33.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.984+0000 7f1462000640 1 -- 172.21.15.40:0/625354545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f145c070020 con 0x7f145c0c6010 2024-04-26T23:02:33.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.985+0000 7f1458c00640 1 -- 172.21.15.40:0/625354545 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f145007a7d0 con 0x7f145c0c6010 2024-04-26T23:02:33.985 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:33.985 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1253648754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1253648754},{"type":"v1","addr":"172.21.15.103:6836","nonce":1253648754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3239652611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":3239652611},{"type":"v1","addr":"172.21.15.103:6837","nonce":3239652611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3670144900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3670144900},{"type":"v1","addr":"172.21.15.40:6833","nonce":3670144900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:01:05.171276+0000","modified":"2024-04-26T23:01:33.399801+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3989190340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3989190340},{"type":"v1","addr":"172.21.15.40:6835","nonce":3989190340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-26T23:02:33.986 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 9 2024-04-26T23:02:33.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.988+0000 7f1462000640 1 -- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f143c06c8f0 msgr2=0x7f143c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:33.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.988+0000 7f1462000640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f143c06c8f0 0x7f143c06edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f145c0b1a00 tx=0x7f1448008040 comp rx=0 tx=0).stop 2024-04-26T23:02:33.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.988+0000 7f1462000640 1 -- 172.21.15.40:0/625354545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f145c0c6010 msgr2=0x7f145c1f8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:33.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.988+0000 7f1462000640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f145c0c6010 0x7f145c1f8d90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1450009960 tx=0x7f14500046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.988+0000 7f1462000640 1 -- 172.21.15.40:0/625354545 shutdown_connections 2024-04-26T23:02:33.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.988+0000 7f1462000640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f143c06c8f0 0x7f143c06edb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.988+0000 7f1462000640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f145c1576a0 0x7f145c0b0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.988+0000 7f1462000640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f145c0c6f10 0x7f145c157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.988+0000 7f1462000640 1 --2- 172.21.15.40:0/625354545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f145c0c6010 0x7f145c1f8d90 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:33.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.988+0000 7f1462000640 1 -- 172.21.15.40:0/625354545 >> 172.21.15.40:0/625354545 conn(0x7f145c0d0a10 msgr2=0x7f145c0cbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:33.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.988+0000 7f1462000640 1 -- 172.21.15.40:0/625354545 shutdown_connections 2024-04-26T23:02:33.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:33.988+0000 7f1462000640 1 -- 172.21.15.40:0/625354545 wait complete. 2024-04-26T23:02:34.008 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:34.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.219+0000 7f9df6e00640 1 Processor -- start 2024-04-26T23:02:34.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.219+0000 7f9df6e00640 1 -- start start 2024-04-26T23:02:34.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.221+0000 7f9df6e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9df015ce60 0x7f9df00afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:34.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.221+0000 7f9df6e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9df00af7a0 0x7f9df00ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:34.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.221+0000 7f9df6e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df00d0820 con 0x7f9df0071fe0 2024-04-26T23:02:34.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.221+0000 7f9df6e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9df00cc450 con 0x7f9df00af7a0 2024-04-26T23:02:34.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.221+0000 7f9df6e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9df01614c0 con 0x7f9df015ce60 2024-04-26T23:02:34.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.222+0000 7f9df5a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9df015ce60 0x7f9df00afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:34.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.222+0000 7f9df5a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9df015ce60 0x7f9df00afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:51118/0 (socket says 172.21.15.40:51118) 2024-04-26T23:02:34.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.222+0000 7f9df5a00640 1 -- 172.21.15.40:0/1474630777 learned_addr learned my addr 172.21.15.40:0/1474630777 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:34.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.222+0000 7f9df5000640 1 --2- 172.21.15.40:0/1474630777 >> v2:172.21.15.103:3300/0 conn(0x7f9df00af7a0 0x7f9df00ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:34.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.222+0000 7f9df5a00640 1 -- 172.21.15.40:0/1474630777 >> v2:172.21.15.103:3300/0 conn(0x7f9df00af7a0 msgr2=0x7f9df00ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:34.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.222+0000 7f9df5a00640 1 --2- 172.21.15.40:0/1474630777 >> v2:172.21.15.103:3300/0 conn(0x7f9df00af7a0 0x7f9df00ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.222+0000 7f9df5a00640 1 -- 172.21.15.40:0/1474630777 >> v1:172.21.15.103:6789/0 conn(0x7f9df0071fe0 legacy=0x7f9df0073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:34.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.222+0000 7f9df5a00640 1 -- 172.21.15.40:0/1474630777 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df0161760 con 0x7f9df015ce60 2024-04-26T23:02:34.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.224+0000 7f9df5a00640 1 --2- 172.21.15.40:0/1474630777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9df015ce60 0x7f9df00afc60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009e50 tx=0x7f9de402a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39a0578db5283ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:34.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.224+0000 7f9defe00640 1 -- 172.21.15.40:0/1474630777 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4036070 con 0x7f9df015ce60 2024-04-26T23:02:34.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.224+0000 7f9defe00640 1 -- 172.21.15.40:0/1474630777 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de4031600 con 0x7f9df015ce60 2024-04-26T23:02:34.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.224+0000 7f9defe00640 1 -- 172.21.15.40:0/1474630777 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de40319d0 con 0x7f9df015ce60 2024-04-26T23:02:34.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.225+0000 7f9df6e00640 1 -- 172.21.15.40:0/1474630777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9df015ce60 msgr2=0x7f9df00afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:34.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.225+0000 7f9df6e00640 1 --2- 172.21.15.40:0/1474630777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9df015ce60 0x7f9df00afc60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009e50 tx=0x7f9de402a680 comp rx=0 tx=0).stop 2024-04-26T23:02:34.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.225+0000 7f9df6e00640 1 -- 172.21.15.40:0/1474630777 shutdown_connections 2024-04-26T23:02:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.225+0000 7f9df6e00640 1 --2- 172.21.15.40:0/1474630777 >> v2:172.21.15.103:3300/0 conn(0x7f9df00af7a0 0x7f9df00ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.225+0000 7f9df6e00640 1 --2- 172.21.15.40:0/1474630777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9df015ce60 0x7f9df00afc60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.225+0000 7f9df6e00640 1 -- 172.21.15.40:0/1474630777 >> 172.21.15.40:0/1474630777 conn(0x7f9df00d0a10 msgr2=0x7f9df00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.225+0000 7f9df6e00640 1 -- 172.21.15.40:0/1474630777 shutdown_connections 2024-04-26T23:02:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.225+0000 7f9df6e00640 1 -- 172.21.15.40:0/1474630777 wait complete. 2024-04-26T23:02:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.227+0000 7f9df6e00640 1 Processor -- start 2024-04-26T23:02:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.227+0000 7f9df6e00640 1 -- start start 2024-04-26T23:02:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.228+0000 7f9df6e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9df0071fe0 0x7f9df01f2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.228+0000 7f9df6e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df00af7a0 0x7f9df00b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.228+0000 7f9df6e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9df015ce60 0x7f9df006f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.228+0000 7f9df6e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9df0073e70 con 0x7f9df00af7a0 2024-04-26T23:02:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.228+0000 7f9df6e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9df00cf530 con 0x7f9df015ce60 2024-04-26T23:02:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.228+0000 7f9df6e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9df0073ff0 con 0x7f9df0071fe0 2024-04-26T23:02:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.228+0000 7f9df5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df00af7a0 0x7f9df00b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.228+0000 7f9df5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df00af7a0 0x7f9df00b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45528/0 (socket says 172.21.15.40:45528) 2024-04-26T23:02:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.228+0000 7f9df5000640 1 -- 172.21.15.40:0/229740440 learned_addr learned my addr 172.21.15.40:0/229740440 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:34.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.229+0000 7f9df5a00640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9df0071fe0 0x7f9df01f2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:34.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.229+0000 7f9df6400640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9df015ce60 0x7f9df006f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:34.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.229+0000 7f9df5000640 1 -- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9df0071fe0 msgr2=0x7f9df01f2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.229+0000 7f9df5000640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9df0071fe0 0x7f9df01f2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.229+0000 7f9df5000640 1 -- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9df015ce60 msgr2=0x7f9df006f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.229+0000 7f9df5000640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9df015ce60 0x7f9df006f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.229+0000 7f9df5000640 1 -- 172.21.15.40:0/229740440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df00b3b90 con 0x7f9df00af7a0 2024-04-26T23:02:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.229+0000 7f9df6400640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9df015ce60 0x7f9df006f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.229+0000 7f9df5000640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df00af7a0 0x7f9df00b3650 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00e9e0 tx=0x7f9ddc00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.229+0000 7f9ded600640 1 -- 172.21.15.40:0/229740440 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc017070 con 0x7f9df00af7a0 2024-04-26T23:02:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.230+0000 7f9ded600640 1 -- 172.21.15.40:0/229740440 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ddc00c9d0 con 0x7f9df00af7a0 2024-04-26T23:02:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.230+0000 7f9ded600640 1 -- 172.21.15.40:0/229740440 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc00cdc0 con 0x7f9df00af7a0 2024-04-26T23:02:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.230+0000 7f9df6e00640 1 -- 172.21.15.40:0/229740440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df01f3ca0 con 0x7f9df00af7a0 2024-04-26T23:02:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.230+0000 7f9df6e00640 1 -- 172.21.15.40:0/229740440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df01f3400 con 0x7f9df00af7a0 2024-04-26T23:02:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.231+0000 7f9df6e00640 1 -- 172.21.15.40:0/229740440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df0069370 con 0x7f9df00af7a0 2024-04-26T23:02:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.232+0000 7f9ded600640 1 -- 172.21.15.40:0/229740440 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9ddc01a070 con 0x7f9df00af7a0 2024-04-26T23:02:34.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.233+0000 7f9ded600640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9dd006c8f0 0x7f9dd006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:34.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.233+0000 7f9ded600640 1 -- 172.21.15.40:0/229740440 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f9ddc08bf60 con 0x7f9df00af7a0 2024-04-26T23:02:34.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.234+0000 7f9df5a00640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9dd006c8f0 0x7f9dd006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:34.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.235+0000 7f9df5a00640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9dd006c8f0 0x7f9dd006edb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9de4033000 tx=0x7f9de4009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:34.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.237+0000 7f9ded600640 1 -- 172.21.15.40:0/229740440 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9ddc059f80 con 0x7f9df00af7a0 2024-04-26T23:02:34.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.442+0000 7f9df6e00640 1 -- 172.21.15.40:0/229740440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9df015b350 con 0x7f9df00af7a0 2024-04-26T23:02:34.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.444+0000 7f9ded600640 1 -- 172.21.15.40:0/229740440 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18812 (secure 0 0 0) 0x7f9ddc059880 con 0x7f9df00af7a0 2024-04-26T23:02:34.445 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:34.445 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:01:46.975683+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:01:02.693456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:01:04.143797+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:01:07.764427+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.447+0000 7f9df6e00640 1 -- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9dd006c8f0 msgr2=0x7f9dd006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.447+0000 7f9df6e00640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9dd006c8f0 0x7f9dd006edb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9de4033000 tx=0x7f9de4009000 comp rx=0 tx=0).stop 2024-04-26T23:02:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.447+0000 7f9df6e00640 1 -- 172.21.15.40:0/229740440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df00af7a0 msgr2=0x7f9df00b3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.447+0000 7f9df6e00640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df00af7a0 0x7f9df00b3650 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00e9e0 tx=0x7f9ddc00eea0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.447+0000 7f9df6e00640 1 -- 172.21.15.40:0/229740440 shutdown_connections 2024-04-26T23:02:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.447+0000 7f9df6e00640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9dd006c8f0 0x7f9dd006edb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.447+0000 7f9df6e00640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9df015ce60 0x7f9df006f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.447+0000 7f9df6e00640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df00af7a0 0x7f9df00b3650 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.447+0000 7f9df6e00640 1 --2- 172.21.15.40:0/229740440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9df0071fe0 0x7f9df01f2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.447+0000 7f9df6e00640 1 -- 172.21.15.40:0/229740440 >> 172.21.15.40:0/229740440 conn(0x7f9df00d0a10 msgr2=0x7f9df00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.447+0000 7f9df6e00640 1 -- 172.21.15.40:0/229740440 shutdown_connections 2024-04-26T23:02:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.447+0000 7f9df6e00640 1 -- 172.21.15.40:0/229740440 wait complete. 2024-04-26T23:02:34.467 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-26T23:02:34.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.678+0000 7fe5c7200640 1 Processor -- start 2024-04-26T23:02:34.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.678+0000 7fe5c7200640 1 -- start start 2024-04-26T23:02:34.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.680+0000 7fe5c7200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe5c00c6f10 0x7fe5c00c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:34.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.680+0000 7fe5c7200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe5c00cc980 0x7fe5c00cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:34.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.680+0000 7fe5c7200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c00cce10 con 0x7fe5c00c6010 2024-04-26T23:02:34.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.680+0000 7fe5c7200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe5c00c72f0 con 0x7fe5c00cc980 2024-04-26T23:02:34.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.680+0000 7fe5c7200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe5c00cb860 con 0x7fe5c00c6f10 2024-04-26T23:02:34.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.680+0000 7fe5c5400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe5c00cc980 0x7fe5c00cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:34.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.680+0000 7fe5c5400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe5c00cc980 0x7fe5c00cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45532/0 (socket says 172.21.15.40:45532) 2024-04-26T23:02:34.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.680+0000 7fe5c6800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe5c00c6010 0x7fe5c0158cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35850/0 (socket says 172.21.15.40:35850) 2024-04-26T23:02:34.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.680+0000 7fe5c5400640 1 -- 172.21.15.40:0/436688399 learned_addr learned my addr 172.21.15.40:0/436688399 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:34.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.681+0000 7fe5c5e00640 1 --2- 172.21.15.40:0/436688399 >> v2:172.21.15.103:3300/0 conn(0x7fe5c00c6f10 0x7fe5c00c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:34.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.681+0000 7fe5c5400640 1 -- 172.21.15.40:0/436688399 >> v2:172.21.15.103:3300/0 conn(0x7fe5c00c6f10 msgr2=0x7fe5c00c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:34.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.681+0000 7fe5c5400640 1 --2- 172.21.15.40:0/436688399 >> v2:172.21.15.103:3300/0 conn(0x7fe5c00c6f10 0x7fe5c00c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.681+0000 7fe5c5400640 1 -- 172.21.15.40:0/436688399 >> v1:172.21.15.40:6789/0 conn(0x7fe5c00c6010 legacy=0x7fe5c0158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:34.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.681+0000 7fe5c5400640 1 -- 172.21.15.40:0/436688399 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c00cd0f0 con 0x7fe5c00cc980 2024-04-26T23:02:34.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.681+0000 7fe5c4a00640 1 -- 172.21.15.40:0/436688399 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2603343873 0 0) 0x7fe5c00cce10 con 0x7fe5c00c6010 2024-04-26T23:02:34.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.682+0000 7fe5c5400640 1 --2- 172.21.15.40:0/436688399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5c00cc980 0x7fe5c00cbcf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe5c00ce590 tx=0x7fe5b002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a99f0a236cbfa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:34.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.682+0000 7fe5c4a00640 1 -- 172.21.15.40:0/436688399 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0037070 con 0x7fe5c00cc980 2024-04-26T23:02:34.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.682+0000 7fe5c4a00640 1 -- 172.21.15.40:0/436688399 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5b0032440 con 0x7fe5c00cc980 2024-04-26T23:02:34.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.682+0000 7fe5c4a00640 1 -- 172.21.15.40:0/436688399 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0032810 con 0x7fe5c00cc980 2024-04-26T23:02:34.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.683+0000 7fe5c7200640 1 -- 172.21.15.40:0/436688399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5c00cc980 msgr2=0x7fe5c00cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:34.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.683+0000 7fe5c7200640 1 --2- 172.21.15.40:0/436688399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5c00cc980 0x7fe5c00cbcf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe5c00ce590 tx=0x7fe5b002a1e0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.683+0000 7fe5c7200640 1 -- 172.21.15.40:0/436688399 shutdown_connections 2024-04-26T23:02:34.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.683+0000 7fe5c7200640 1 --2- 172.21.15.40:0/436688399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5c00cc980 0x7fe5c00cbcf0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.683+0000 7fe5c7200640 1 --2- 172.21.15.40:0/436688399 >> v2:172.21.15.103:3300/0 conn(0x7fe5c00c6f10 0x7fe5c00c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.683+0000 7fe5c7200640 1 -- 172.21.15.40:0/436688399 >> 172.21.15.40:0/436688399 conn(0x7fe5c00d0a10 msgr2=0x7fe5c00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:34.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.683+0000 7fe5c7200640 1 -- 172.21.15.40:0/436688399 shutdown_connections 2024-04-26T23:02:34.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.683+0000 7fe5c7200640 1 -- 172.21.15.40:0/436688399 wait complete. 2024-04-26T23:02:34.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.685+0000 7fe5c7200640 1 Processor -- start 2024-04-26T23:02:34.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.685+0000 7fe5c7200640 1 -- start start 2024-04-26T23:02:34.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.686+0000 7fe5c7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5c00c6010 0x7fe5c01532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:34.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.686+0000 7fe5c7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5c00c6f10 0x7fe5c01598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:34.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.686+0000 7fe5c7200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5c00731f0 0x7fe5c0073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:34.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.686+0000 7fe5c7200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c00c76a0 con 0x7fe5c00731f0 2024-04-26T23:02:34.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.686+0000 7fe5c7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c00cb1a0 con 0x7fe5c00c6f10 2024-04-26T23:02:34.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.686+0000 7fe5c7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c00c7820 con 0x7fe5c00c6010 2024-04-26T23:02:34.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.686+0000 7fe5c6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5c00731f0 0x7fe5c0073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:34.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.686+0000 7fe5c6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5c00731f0 0x7fe5c0073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45548/0 (socket says 172.21.15.40:45548) 2024-04-26T23:02:34.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.686+0000 7fe5c6800640 1 -- 172.21.15.40:0/3192087905 learned_addr learned my addr 172.21.15.40:0/3192087905 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:34.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.687+0000 7fe5c6800640 1 -- 172.21.15.40:0/3192087905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5c00c6010 msgr2=0x7fe5c01532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:34.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.687+0000 7fe5c6800640 1 --2- 172.21.15.40:0/3192087905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5c00c6010 0x7fe5c01532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.687+0000 7fe5c6800640 1 -- 172.21.15.40:0/3192087905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5c00c6f10 msgr2=0x7fe5c01598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:34.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.687+0000 7fe5c5400640 1 --2- 172.21.15.40:0/3192087905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5c00c6f10 0x7fe5c01598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:34.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.687+0000 7fe5c6800640 1 --2- 172.21.15.40:0/3192087905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5c00c6f10 0x7fe5c01598f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.687+0000 7fe5c6800640 1 -- 172.21.15.40:0/3192087905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c0153800 con 0x7fe5c00731f0 2024-04-26T23:02:34.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.687+0000 7fe5c6800640 1 --2- 172.21.15.40:0/3192087905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5c00731f0 0x7fe5c0073680 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00c700 tx=0x7fe5bc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:34.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.687+0000 7fe5ba000640 1 -- 172.21.15.40:0/3192087905 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc018070 con 0x7fe5c00731f0 2024-04-26T23:02:34.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.687+0000 7fe5ba000640 1 -- 172.21.15.40:0/3192087905 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5bc00ce10 con 0x7fe5c00731f0 2024-04-26T23:02:34.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.687+0000 7fe5ba000640 1 -- 172.21.15.40:0/3192087905 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc00a880 con 0x7fe5c00731f0 2024-04-26T23:02:34.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.687+0000 7fe5c7200640 1 -- 172.21.15.40:0/3192087905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c0157ed0 con 0x7fe5c00731f0 2024-04-26T23:02:34.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.688+0000 7fe5c7200640 1 -- 172.21.15.40:0/3192087905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c0158490 con 0x7fe5c00731f0 2024-04-26T23:02:34.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.689+0000 7fe5c7200640 1 -- 172.21.15.40:0/3192087905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5c0069370 con 0x7fe5c00731f0 2024-04-26T23:02:34.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.690+0000 7fe5ba000640 1 -- 172.21.15.40:0/3192087905 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe5bc00aa20 con 0x7fe5c00731f0 2024-04-26T23:02:34.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.691+0000 7fe5ba000640 1 --2- 172.21.15.40:0/3192087905 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe5a406c8f0 0x7fe5a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:34.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.691+0000 7fe5ba000640 1 -- 172.21.15.40:0/3192087905 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe5bc08b4b0 con 0x7fe5c00731f0 2024-04-26T23:02:34.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.692+0000 7fe5c5e00640 1 --2- 172.21.15.40:0/3192087905 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe5a406c8f0 0x7fe5a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:34.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.693+0000 7fe5c5e00640 1 --2- 172.21.15.40:0/3192087905 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe5a406c8f0 0x7fe5a406edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40098f0 tx=0x7fe5b4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:34.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.696+0000 7fe5ba000640 1 -- 172.21.15.40:0/3192087905 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe5bc059570 con 0x7fe5c00731f0 2024-04-26T23:02:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.940+0000 7fe5c7200640 1 -- 172.21.15.40:0/3192087905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe5c0070020 con 0x7fe5c00731f0 2024-04-26T23:02:34.942 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:02:34.942+0000 7f36fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T23:02:34.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.946+0000 7fe5ba000640 1 -- 172.21.15.40:0/3192087905 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) ==== 161+0+0 (secure 0 0 0) 0x7fe5bc058e70 con 0x7fe5c00731f0 2024-04-26T23:02:34.946 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T23:02:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.950+0000 7fe5c7200640 1 -- 172.21.15.40:0/3192087905 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe5a406c8f0 msgr2=0x7fe5a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.950+0000 7fe5c7200640 1 --2- 172.21.15.40:0/3192087905 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe5a406c8f0 0x7fe5a406edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40098f0 tx=0x7fe5b4008040 comp rx=0 tx=0).stop 2024-04-26T23:02:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.950+0000 7fe5c7200640 1 -- 172.21.15.40:0/3192087905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5c00731f0 msgr2=0x7fe5c0073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:34.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.950+0000 7fe5c7200640 1 --2- 172.21.15.40:0/3192087905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5c00731f0 0x7fe5c0073680 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00c700 tx=0x7fe5bc00cbc0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.950+0000 7fe5c7200640 1 -- 172.21.15.40:0/3192087905 shutdown_connections 2024-04-26T23:02:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.950+0000 7fe5c7200640 1 --2- 172.21.15.40:0/3192087905 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe5a406c8f0 0x7fe5a406edb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.950+0000 7fe5c7200640 1 --2- 172.21.15.40:0/3192087905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5c00731f0 0x7fe5c0073680 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.950+0000 7fe5c7200640 1 --2- 172.21.15.40:0/3192087905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5c00c6f10 0x7fe5c01598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.950+0000 7fe5c7200640 1 --2- 172.21.15.40:0/3192087905 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5c00c6010 0x7fe5c01532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.950+0000 7fe5c7200640 1 -- 172.21.15.40:0/3192087905 >> 172.21.15.40:0/3192087905 conn(0x7fe5c00d0a10 msgr2=0x7fe5c006e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.952+0000 7fe5c7200640 1 -- 172.21.15.40:0/3192087905 shutdown_connections 2024-04-26T23:02:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:34.952+0000 7fe5c7200640 1 -- 172.21.15.40:0/3192087905 wait complete. 2024-04-26T23:02:34.967 INFO:tasks.ceph.mds.c.smithi040.stderr: -6786> 2024-04-26T23:01:10.653+0000 7fecfe400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T23:02:34.977 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T23:02:35.036 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:02:35.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.170+0000 7f1ed3600640 1 Processor -- start 2024-04-26T23:02:35.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.170+0000 7f1ed3600640 1 -- start start 2024-04-26T23:02:35.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.172+0000 7f1ed3600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1ecc071e60 0x7f1ecc06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:35.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.172+0000 7f1ed3600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1ecc06fe20 0x7f1ecc072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:35.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.172+0000 7f1ed3600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ecc072780 0x7f1ecc0cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:35.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.172+0000 7f1ed3600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1ecc15d7c0 con 0x7f1ecc071e60 2024-04-26T23:02:35.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.172+0000 7f1ed3600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1ecc15daa0 con 0x7f1ecc06fe20 2024-04-26T23:02:35.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.172+0000 7f1ed3600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1ecc15dd80 con 0x7f1ecc072780 2024-04-26T23:02:35.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.172+0000 7f1ed0c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1ecc071e60 0x7f1ecc06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:35.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.172+0000 7f1ed0c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1ecc071e60 0x7f1ecc06f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32784/0 (socket says 172.21.15.40:32784) 2024-04-26T23:02:35.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.172+0000 7f1ed0c00640 1 -- 172.21.15.40:0/941194916 learned_addr learned my addr 172.21.15.40:0/941194916 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:35.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.173+0000 7f1ed0c00640 1 -- 172.21.15.40:0/941194916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ecc072780 msgr2=0x7f1ecc0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:35.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.173+0000 7f1ed0c00640 1 --2- 172.21.15.40:0/941194916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ecc072780 0x7f1ecc0cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:35.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.173+0000 7f1ed0c00640 1 -- 172.21.15.40:0/941194916 >> v2:172.21.15.103:3300/0 conn(0x7f1ecc06fe20 msgr2=0x7f1ecc072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:35.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.173+0000 7f1ed0c00640 1 --2- 172.21.15.40:0/941194916 >> v2:172.21.15.103:3300/0 conn(0x7f1ecc06fe20 0x7f1ecc072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:35.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.173+0000 7f1ed0c00640 1 -- 172.21.15.40:0/941194916 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ecc15e060 con 0x7f1ecc071e60 2024-04-26T23:02:35.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.174+0000 7f1ed0c00640 1 --2- 172.21.15.40:0/941194916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc071e60 0x7f1ecc06f8e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0b3820 tx=0x7f1ec002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fdbfea9c8eb0aa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:35.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.174+0000 7f1ecb400640 1 -- 172.21.15.40:0/941194916 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0037070 con 0x7f1ecc071e60 2024-04-26T23:02:35.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.174+0000 7f1ecb400640 1 -- 172.21.15.40:0/941194916 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec0032440 con 0x7f1ecc071e60 2024-04-26T23:02:35.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.174+0000 7f1ecb400640 1 -- 172.21.15.40:0/941194916 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0032810 con 0x7f1ecc071e60 2024-04-26T23:02:35.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.175+0000 7f1ed3600640 1 -- 172.21.15.40:0/941194916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc071e60 msgr2=0x7f1ecc06f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:35.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.175+0000 7f1ed3600640 1 --2- 172.21.15.40:0/941194916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc071e60 0x7f1ecc06f8e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0b3820 tx=0x7f1ec002a1e0 comp rx=0 tx=0).stop 2024-04-26T23:02:35.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.175+0000 7f1ed3600640 1 -- 172.21.15.40:0/941194916 shutdown_connections 2024-04-26T23:02:35.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.175+0000 7f1ed3600640 1 --2- 172.21.15.40:0/941194916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ecc072780 0x7f1ecc0cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:35.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.175+0000 7f1ed3600640 1 --2- 172.21.15.40:0/941194916 >> v2:172.21.15.103:3300/0 conn(0x7f1ecc06fe20 0x7f1ecc072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:35.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.175+0000 7f1ed3600640 1 --2- 172.21.15.40:0/941194916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc071e60 0x7f1ecc06f8e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:35.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.175+0000 7f1ed3600640 1 -- 172.21.15.40:0/941194916 >> 172.21.15.40:0/941194916 conn(0x7f1ecc0d0be0 msgr2=0x7f1ecc0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:35.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.175+0000 7f1ed3600640 1 -- 172.21.15.40:0/941194916 shutdown_connections 2024-04-26T23:02:35.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.175+0000 7f1ed3600640 1 -- 172.21.15.40:0/941194916 wait complete. 2024-04-26T23:02:35.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.177+0000 7f1ed3600640 1 Processor -- start 2024-04-26T23:02:35.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.177+0000 7f1ed3600640 1 -- start start 2024-04-26T23:02:35.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.178+0000 7f1ed3600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc06fe20 0x7f1ecc0c8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:35.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.178+0000 7f1ed3600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ecc071e60 0x7f1ecc0b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:35.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.178+0000 7f1ed3600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ecc072780 0x7f1ecc0b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:35.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.178+0000 7f1ed3600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ecc15d3f0 con 0x7f1ecc06fe20 2024-04-26T23:02:35.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.178+0000 7f1ed3600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1ecc072c10 con 0x7f1ecc072780 2024-04-26T23:02:35.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.178+0000 7f1ed3600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1ecc15d570 con 0x7f1ecc071e60 2024-04-26T23:02:35.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.178+0000 7f1ed0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc06fe20 0x7f1ecc0c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:35.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.178+0000 7f1ed0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc06fe20 0x7f1ecc0c8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32798/0 (socket says 172.21.15.40:32798) 2024-04-26T23:02:35.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.178+0000 7f1ed0c00640 1 -- 172.21.15.40:0/2140830568 learned_addr learned my addr 172.21.15.40:0/2140830568 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:35.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.179+0000 7f1ed0c00640 1 -- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ecc071e60 msgr2=0x7f1ecc0b5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:35.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.179+0000 7f1ecbe00640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ecc071e60 0x7f1ecc0b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:35.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.179+0000 7f1ed1600640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ecc072780 0x7f1ecc0b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:35.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.179+0000 7f1ed0c00640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ecc071e60 0x7f1ecc0b5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:35.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.179+0000 7f1ed0c00640 1 -- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ecc072780 msgr2=0x7f1ecc0b5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:35.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.179+0000 7f1ed0c00640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ecc072780 0x7f1ecc0b5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:35.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.179+0000 7f1ed0c00640 1 -- 172.21.15.40:0/2140830568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ecc1f5900 con 0x7f1ecc06fe20 2024-04-26T23:02:35.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.179+0000 7f1ecbe00640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ecc071e60 0x7f1ecc0b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:35.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.179+0000 7f1ed0c00640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc06fe20 0x7f1ecc0c8090 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0000c00 tx=0x7f1ec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:35.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.180+0000 7f1ec8c00640 1 -- 172.21.15.40:0/2140830568 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0041070 con 0x7f1ecc06fe20 2024-04-26T23:02:35.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.180+0000 7f1ec8c00640 1 -- 172.21.15.40:0/2140830568 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec003c870 con 0x7f1ecc06fe20 2024-04-26T23:02:35.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.180+0000 7f1ed3600640 1 -- 172.21.15.40:0/2140830568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ecc14bc90 con 0x7f1ecc06fe20 2024-04-26T23:02:35.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.180+0000 7f1ec8c00640 1 -- 172.21.15.40:0/2140830568 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0037040 con 0x7f1ecc06fe20 2024-04-26T23:02:35.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.180+0000 7f1ed3600640 1 -- 172.21.15.40:0/2140830568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ecc1f4e20 con 0x7f1ecc06fe20 2024-04-26T23:02:35.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.181+0000 7f1ed3600640 1 -- 172.21.15.40:0/2140830568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ecc0694f0 con 0x7f1ecc06fe20 2024-04-26T23:02:35.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.182+0000 7f1ec8c00640 1 -- 172.21.15.40:0/2140830568 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1ec0044070 con 0x7f1ecc06fe20 2024-04-26T23:02:35.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.183+0000 7f1ec8c00640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1eac06c8f0 0x7f1eac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:35.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.183+0000 7f1ec8c00640 1 -- 172.21.15.40:0/2140830568 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6821+0+0 (secure 0 0 0) 0x7f1ec00acc40 con 0x7f1ecc06fe20 2024-04-26T23:02:35.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.184+0000 7f1ecbe00640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1eac06c8f0 0x7f1eac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:35.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.185+0000 7f1ecbe00640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1eac06c8f0 0x7f1eac06edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8006fd0 tx=0x7f1eb8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:35.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.188+0000 7f1ec8c00640 1 -- 172.21.15.40:0/2140830568 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1ec007ad20 con 0x7f1ecc06fe20 2024-04-26T23:02:35.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.426+0000 7f1ed3600640 1 -- 172.21.15.40:0/2140830568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1ecc0c9ab0 con 0x7f1ecc06fe20 2024-04-26T23:02:35.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.956+0000 7f1ec8c00640 1 -- 172.21.15.40:0/2140830568 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f1ec007a620 con 0x7f1ecc06fe20 2024-04-26T23:02:35.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.960+0000 7f1ed3600640 1 -- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1eac06c8f0 msgr2=0x7f1eac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:35.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.960+0000 7f1ed3600640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1eac06c8f0 0x7f1eac06edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8006fd0 tx=0x7f1eb8008040 comp rx=0 tx=0).stop 2024-04-26T23:02:35.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.960+0000 7f1ed3600640 1 -- 172.21.15.40:0/2140830568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc06fe20 msgr2=0x7f1ecc0c8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:35.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.960+0000 7f1ed3600640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc06fe20 0x7f1ecc0c8090 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0000c00 tx=0x7f1ec00046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:35.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.960+0000 7f1ed3600640 1 -- 172.21.15.40:0/2140830568 shutdown_connections 2024-04-26T23:02:35.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.960+0000 7f1ed3600640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1eac06c8f0 0x7f1eac06edb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:35.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.960+0000 7f1ed3600640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ecc072780 0x7f1ecc0b5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:35.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.960+0000 7f1ed3600640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ecc071e60 0x7f1ecc0b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:35.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.960+0000 7f1ed3600640 1 --2- 172.21.15.40:0/2140830568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc06fe20 0x7f1ecc0c8090 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:35.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.960+0000 7f1ed3600640 1 -- 172.21.15.40:0/2140830568 >> 172.21.15.40:0/2140830568 conn(0x7f1ecc0d0be0 msgr2=0x7f1ecc0b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:35.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.960+0000 7f1ed3600640 1 -- 172.21.15.40:0/2140830568 shutdown_connections 2024-04-26T23:02:35.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:35.961+0000 7f1ed3600640 1 -- 172.21.15.40:0/2140830568 wait complete. 2024-04-26T23:02:35.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T23:02:36.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.198+0000 7f6c17000640 1 Processor -- start 2024-04-26T23:02:36.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.198+0000 7f6c17000640 1 -- start start 2024-04-26T23:02:36.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.200+0000 7f6c17000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c100b5c00 0x7f6c100cded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:36.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.200+0000 7f6c17000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c100d04e0 con 0x7f6c10146790 2024-04-26T23:02:36.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.200+0000 7f6c17000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c100d07c0 con 0x7f6c100b5770 2024-04-26T23:02:36.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.200+0000 7f6c17000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6c100b6010 con 0x7f6c100b5c00 2024-04-26T23:02:36.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.200+0000 7f6c15c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6c10146790 0x7f6c1015dd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:54962/0 (socket says 172.21.15.40:54962) 2024-04-26T23:02:36.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.200+0000 7f6c15c00640 1 -- 172.21.15.40:0/1168285279 learned_addr learned my addr 172.21.15.40:0/1168285279 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.201+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3503851477 0 0) 0x7f6c100d04e0 con 0x7f6c10146790 2024-04-26T23:02:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.201+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bf8003520 con 0x7f6c10146790 2024-04-26T23:02:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.201+0000 7f6c16600640 1 --2- 172.21.15.40:0/1168285279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c100b5c00 0x7f6c100cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.201+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1988852250 0 0) 0x7f6bf8003520 con 0x7f6c10146790 2024-04-26T23:02:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.201+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c100b5c00 msgr2=0x7f6c100cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.201+0000 7f6c0fe00640 1 --2- 172.21.15.40:0/1168285279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c100b5c00 0x7f6c100cded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.201+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 >> v1:172.21.15.103:6789/0 conn(0x7f6c100b5770 legacy=0x7f6c1006de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.201+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c100b4c90 con 0x7f6c10146790 2024-04-26T23:02:36.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.201+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f6c040019f0 con 0x7f6c10146790 2024-04-26T23:02:36.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.201+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 >> v1:172.21.15.40:6789/0 conn(0x7f6c10146790 legacy=0x7f6c1015dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:36.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.202+0000 7f6c0fe00640 1 --2- 172.21.15.40:0/1168285279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf80039e0 0x7f6bf8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:36.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.202+0000 7f6c0fe00640 1 --2- 172.21.15.40:0/1168285279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c100b5c00 0x7f6bf8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:36.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.202+0000 7f6c0fe00640 1 --2- 172.21.15.40:0/1168285279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6bf8004e10 0x7f6bf8005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:36.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.202+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6c100b6010 con 0x7f6bf80039e0 2024-04-26T23:02:36.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.202+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf800bcf0 con 0x7f6bf8004e10 2024-04-26T23:02:36.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.202+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf800bef0 con 0x7f6c100b5c00 2024-04-26T23:02:36.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.202+0000 7f6c15c00640 1 --2- 172.21.15.40:0/1168285279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf80039e0 0x7f6bf8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.202+0000 7f6c15c00640 1 -- 172.21.15.40:0/1168285279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c100b5c00 msgr2=0x7f6bf8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.202+0000 7f6c15c00640 1 --2- 172.21.15.40:0/1168285279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c100b5c00 0x7f6bf8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.202+0000 7f6c15c00640 1 -- 172.21.15.40:0/1168285279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6bf8004e10 msgr2=0x7f6bf8005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.202+0000 7f6c15c00640 1 --2- 172.21.15.40:0/1168285279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6bf8004e10 0x7f6bf8005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.202+0000 7f6c15c00640 1 -- 172.21.15.40:0/1168285279 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf8003520 con 0x7f6bf80039e0 2024-04-26T23:02:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.203+0000 7f6c15c00640 1 --2- 172.21.15.40:0/1168285279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf80039e0 0x7f6bf8003e00 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400a7b0 tx=0x7f6bf8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3eee8988419f7c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.204+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0400a3d0 con 0x7f6bf80039e0 2024-04-26T23:02:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.204+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c0402f070 con 0x7f6bf80039e0 2024-04-26T23:02:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.204+0000 7f6c0fe00640 1 -- 172.21.15.40:0/1168285279 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0402bc40 con 0x7f6bf80039e0 2024-04-26T23:02:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.204+0000 7f6c17000640 1 -- 172.21.15.40:0/1168285279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf80039e0 msgr2=0x7f6bf8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.204+0000 7f6c17000640 1 --2- 172.21.15.40:0/1168285279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bf80039e0 0x7f6bf8003e00 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400a7b0 tx=0x7f6bf8005830 comp rx=0 tx=0).stop 2024-04-26T23:02:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.204+0000 7f6c16600640 1 -- 172.21.15.40:0/1168285279 reap_dead start 2024-04-26T23:02:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.204+0000 7f6c17000640 1 -- 172.21.15.40:0/1168285279 shutdown_connections 2024-04-26T23:02:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.204+0000 7f6c17000640 1 -- 172.21.15.40:0/1168285279 >> 172.21.15.40:0/1168285279 conn(0x7f6c10075f10 msgr2=0x7f6c10068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.204+0000 7f6c17000640 1 -- 172.21.15.40:0/1168285279 shutdown_connections 2024-04-26T23:02:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.204+0000 7f6c17000640 1 -- 172.21.15.40:0/1168285279 wait complete. 2024-04-26T23:02:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.206+0000 7f6c17000640 1 Processor -- start 2024-04-26T23:02:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.206+0000 7f6c17000640 1 -- start start 2024-04-26T23:02:36.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.207+0000 7f6c17000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c1013d270 0x7f6c1013de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:36.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.207+0000 7f6c17000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c1013e3c0 0x7f6c10072cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:36.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.207+0000 7f6c17000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c10073210 0x7f6c1013feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:36.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.207+0000 7f6c17000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1006e540 con 0x7f6c1013d270 2024-04-26T23:02:36.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.207+0000 7f6c17000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6c100b5250 con 0x7f6c10073210 2024-04-26T23:02:36.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.207+0000 7f6c17000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6c1006e6c0 con 0x7f6c1013e3c0 2024-04-26T23:02:36.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.208+0000 7f6c15c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c1013d270 0x7f6c1013de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:36.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.208+0000 7f6c15c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c1013d270 0x7f6c1013de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32818/0 (socket says 172.21.15.40:32818) 2024-04-26T23:02:36.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.208+0000 7f6c15c00640 1 -- 172.21.15.40:0/2358916445 learned_addr learned my addr 172.21.15.40:0/2358916445 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:36.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.208+0000 7f6c15200640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c1013e3c0 0x7f6c10072cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:36.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.208+0000 7f6c16600640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c10073210 0x7f6c1013feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:36.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.208+0000 7f6c15c00640 1 -- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c1013e3c0 msgr2=0x7f6c10072cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:36.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.208+0000 7f6c15c00640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c1013e3c0 0x7f6c10072cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:36.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.208+0000 7f6c15c00640 1 -- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c10073210 msgr2=0x7f6c1013feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:36.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.208+0000 7f6c15c00640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c10073210 0x7f6c1013feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:36.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.208+0000 7f6c15c00640 1 -- 172.21.15.40:0/2358916445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c100b10e0 con 0x7f6c1013d270 2024-04-26T23:02:36.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.208+0000 7f6c15200640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c1013e3c0 0x7f6c10072cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:36.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.208+0000 7f6c15c00640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c1013d270 0x7f6c1013de80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400a7b0 tx=0x7f6c04001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:36.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.209+0000 7f6c0d600640 1 -- 172.21.15.40:0/2358916445 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04044070 con 0x7f6c1013d270 2024-04-26T23:02:36.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.209+0000 7f6c0d600640 1 -- 172.21.15.40:0/2358916445 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c04001250 con 0x7f6c1013d270 2024-04-26T23:02:36.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.209+0000 7f6c0d600640 1 -- 172.21.15.40:0/2358916445 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0400a3d0 con 0x7f6c1013d270 2024-04-26T23:02:36.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.209+0000 7f6c17000640 1 -- 172.21.15.40:0/2358916445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c100b13c0 con 0x7f6c1013d270 2024-04-26T23:02:36.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.209+0000 7f6c17000640 1 -- 172.21.15.40:0/2358916445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c100b1980 con 0x7f6c1013d270 2024-04-26T23:02:36.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.210+0000 7f6c17000640 1 -- 172.21.15.40:0/2358916445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c1006e360 con 0x7f6c1013d270 2024-04-26T23:02:36.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.211+0000 7f6c0d600640 1 -- 172.21.15.40:0/2358916445 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6c04047070 con 0x7f6c1013d270 2024-04-26T23:02:36.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.212+0000 7f6c0d600640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6bf006c8f0 0x7f6bf006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:36.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.212+0000 7f6c0d600640 1 -- 172.21.15.40:0/2358916445 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6821+0+0 (secure 0 0 0) 0x7f6c040abbf0 con 0x7f6c1013d270 2024-04-26T23:02:36.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.212+0000 7f6c15200640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6bf006c8f0 0x7f6bf006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:36.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.213+0000 7f6c15200640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6bf006c8f0 0x7f6bf006edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6c100ccf50 tx=0x7f6bfc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.217+0000 7f6c0d600640 1 -- 172.21.15.40:0/2358916445 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f6c1006e360 con 0x7f6c1013d270 2024-04-26T23:02:36.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:36.423+0000 7f6c17000640 1 -- 172.21.15.40:0/2358916445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6c10069370 con 0x7f6c1013d270 2024-04-26T23:02:37.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.286+0000 7f6c0d600640 1 -- 172.21.15.40:0/2358916445 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) ==== 181+0+0 (secure 0 0 0) 0x7f6c04079bb0 con 0x7f6c1013d270 2024-04-26T23:02:37.287 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-04-26T23:02:37.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.290+0000 7f6c17000640 1 -- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6bf006c8f0 msgr2=0x7f6bf006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:37.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.290+0000 7f6c17000640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6bf006c8f0 0x7f6bf006edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6c100ccf50 tx=0x7f6bfc009040 comp rx=0 tx=0).stop 2024-04-26T23:02:37.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.290+0000 7f6c17000640 1 -- 172.21.15.40:0/2358916445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c1013d270 msgr2=0x7f6c1013de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:37.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.290+0000 7f6c17000640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c1013d270 0x7f6c1013de80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400a7b0 tx=0x7f6c04001cf0 comp rx=0 tx=0).stop 2024-04-26T23:02:37.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.292+0000 7f6c17000640 1 -- 172.21.15.40:0/2358916445 shutdown_connections 2024-04-26T23:02:37.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.292+0000 7f6c17000640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6bf006c8f0 0x7f6bf006edb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:37.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.292+0000 7f6c17000640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c10073210 0x7f6c1013feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:37.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.292+0000 7f6c17000640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c1013e3c0 0x7f6c10072cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:37.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.292+0000 7f6c17000640 1 --2- 172.21.15.40:0/2358916445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c1013d270 0x7f6c1013de80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:37.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.292+0000 7f6c17000640 1 -- 172.21.15.40:0/2358916445 >> 172.21.15.40:0/2358916445 conn(0x7f6c10075f10 msgr2=0x7f6c10142980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:37.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.292+0000 7f6c17000640 1 -- 172.21.15.40:0/2358916445 shutdown_connections 2024-04-26T23:02:37.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.292+0000 7f6c17000640 1 -- 172.21.15.40:0/2358916445 wait complete. 2024-04-26T23:02:37.321 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T23:02:37.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.536+0000 7fb2e7e00640 1 Processor -- start 2024-04-26T23:02:37.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.536+0000 7fb2e7e00640 1 -- start start 2024-04-26T23:02:37.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.537+0000 7fb2e7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb2e815f420 0x7fb2e80d0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:37.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.537+0000 7fb2e7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2e80d3b60 0x7fb2e80b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:37.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.537+0000 7fb2e7e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2e80d0d50 con 0x7fb2e80b3d90 2024-04-26T23:02:37.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.537+0000 7fb2e7e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb2e80d0a70 con 0x7fb2e80d3b60 2024-04-26T23:02:37.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.537+0000 7fb2e7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb2e80b1bf0 con 0x7fb2e815f420 2024-04-26T23:02:37.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.538+0000 7fb2e7400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2e80d3b60 0x7fb2e80b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:37.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.538+0000 7fb2e7400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2e80d3b60 0x7fb2e80b26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32824/0 (socket says 172.21.15.40:32824) 2024-04-26T23:02:37.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.538+0000 7fb2e7400640 1 -- 172.21.15.40:0/3253133351 learned_addr learned my addr 172.21.15.40:0/3253133351 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:37.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.538+0000 7fb2e7400640 1 -- 172.21.15.40:0/3253133351 >> v2:172.21.15.103:3300/0 conn(0x7fb2e815f420 msgr2=0x7fb2e80d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:37.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.538+0000 7fb2e7400640 1 --2- 172.21.15.40:0/3253133351 >> v2:172.21.15.103:3300/0 conn(0x7fb2e815f420 0x7fb2e80d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:37.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.538+0000 7fb2e7400640 1 -- 172.21.15.40:0/3253133351 >> v1:172.21.15.103:6789/0 conn(0x7fb2e80b3d90 legacy=0x7fb2e80d3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:37.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.538+0000 7fb2e7400640 1 -- 172.21.15.40:0/3253133351 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e80b1ed0 con 0x7fb2e80d3b60 2024-04-26T23:02:37.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.540+0000 7fb2e7400640 1 --2- 172.21.15.40:0/3253133351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e80d3b60 0x7fb2e80b26d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80d33a0 tx=0x7fb2e002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7dc7818373a1e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:37.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.540+0000 7fb2e5600640 1 -- 172.21.15.40:0/3253133351 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0037070 con 0x7fb2e80d3b60 2024-04-26T23:02:37.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.540+0000 7fb2e5600640 1 -- 172.21.15.40:0/3253133351 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2e0032440 con 0x7fb2e80d3b60 2024-04-26T23:02:37.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.540+0000 7fb2e5600640 1 -- 172.21.15.40:0/3253133351 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0032810 con 0x7fb2e80d3b60 2024-04-26T23:02:37.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.541+0000 7fb2e7e00640 1 -- 172.21.15.40:0/3253133351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e80d3b60 msgr2=0x7fb2e80b26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:37.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.541+0000 7fb2e7e00640 1 --2- 172.21.15.40:0/3253133351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e80d3b60 0x7fb2e80b26d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80d33a0 tx=0x7fb2e002a1a0 comp rx=0 tx=0).stop 2024-04-26T23:02:37.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.541+0000 7fb2e7e00640 1 -- 172.21.15.40:0/3253133351 shutdown_connections 2024-04-26T23:02:37.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.541+0000 7fb2e7e00640 1 --2- 172.21.15.40:0/3253133351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e80d3b60 0x7fb2e80b26d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:37.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.541+0000 7fb2e7e00640 1 --2- 172.21.15.40:0/3253133351 >> v2:172.21.15.103:3300/0 conn(0x7fb2e815f420 0x7fb2e80d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:37.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.541+0000 7fb2e7e00640 1 -- 172.21.15.40:0/3253133351 >> 172.21.15.40:0/3253133351 conn(0x7fb2e809e1d0 msgr2=0x7fb2e80d4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:37.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.541+0000 7fb2e7e00640 1 -- 172.21.15.40:0/3253133351 shutdown_connections 2024-04-26T23:02:37.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.541+0000 7fb2e7e00640 1 -- 172.21.15.40:0/3253133351 wait complete. 2024-04-26T23:02:37.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.542+0000 7fb2e7e00640 1 Processor -- start 2024-04-26T23:02:37.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.543+0000 7fb2e7e00640 1 -- start start 2024-04-26T23:02:37.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.543+0000 7fb2e7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2e80b3d90 0x7fb2e815ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:37.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.543+0000 7fb2e7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2e80d3b60 0x7fb2e81f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:37.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e815f420 0x7fb2e8150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:37.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e80b2c10 con 0x7fb2e815f420 2024-04-26T23:02:37.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e80d2e50 con 0x7fb2e80b3d90 2024-04-26T23:02:37.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb2e80b2d90 con 0x7fb2e80d3b60 2024-04-26T23:02:37.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e815f420 0x7fb2e8150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:37.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e815f420 0x7fb2e8150090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32828/0 (socket says 172.21.15.40:32828) 2024-04-26T23:02:37.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e7400640 1 -- 172.21.15.40:0/1725163447 learned_addr learned my addr 172.21.15.40:0/1725163447 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:37.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e6a00640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2e80b3d90 0x7fb2e815ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:37.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e7400640 1 -- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2e80d3b60 msgr2=0x7fb2e81f4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:37.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e6000640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2e80d3b60 0x7fb2e81f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:37.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e7400640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2e80d3b60 0x7fb2e81f4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:37.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e7400640 1 -- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2e80b3d90 msgr2=0x7fb2e815ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:37.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e7400640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2e80b3d90 0x7fb2e815ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:37.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.544+0000 7fb2e7400640 1 -- 172.21.15.40:0/1725163447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e815a230 con 0x7fb2e815f420 2024-04-26T23:02:37.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.545+0000 7fb2e7400640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e815f420 0x7fb2e8150090 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb2e002a1e0 tx=0x7fb2e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:37.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.545+0000 7fb2dea00640 1 -- 172.21.15.40:0/1725163447 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0041070 con 0x7fb2e815f420 2024-04-26T23:02:37.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.545+0000 7fb2dea00640 1 -- 172.21.15.40:0/1725163447 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2e003c950 con 0x7fb2e815f420 2024-04-26T23:02:37.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.545+0000 7fb2e7e00640 1 -- 172.21.15.40:0/1725163447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e80b9010 con 0x7fb2e815f420 2024-04-26T23:02:37.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.545+0000 7fb2dea00640 1 -- 172.21.15.40:0/1725163447 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0037040 con 0x7fb2e815f420 2024-04-26T23:02:37.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.545+0000 7fb2e7e00640 1 -- 172.21.15.40:0/1725163447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e80b95d0 con 0x7fb2e815f420 2024-04-26T23:02:37.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.547+0000 7fb2e7e00640 1 -- 172.21.15.40:0/1725163447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e8069370 con 0x7fb2e815f420 2024-04-26T23:02:37.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.547+0000 7fb2dea00640 1 -- 172.21.15.40:0/1725163447 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb2e0044070 con 0x7fb2e815f420 2024-04-26T23:02:37.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.548+0000 7fb2dea00640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb2c406c940 0x7fb2c406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:37.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.548+0000 7fb2dea00640 1 -- 172.21.15.40:0/1725163447 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6383+0+0 (secure 0 0 0) 0x7fb2e00ac970 con 0x7fb2e815f420 2024-04-26T23:02:37.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.549+0000 7fb2e6a00640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb2c406c940 0x7fb2c406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:37.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.550+0000 7fb2e6a00640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb2c406c940 0x7fb2c406ee00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800b810 tx=0x7fb2d8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:37.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.553+0000 7fb2dea00640 1 -- 172.21.15.40:0/1725163447 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fb2e007aa50 con 0x7fb2e815f420 2024-04-26T23:02:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:37.755+0000 7fb2e7e00640 1 -- 172.21.15.40:0/1725163447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb2e80b6040 con 0x7fb2e815f420 2024-04-26T23:02:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.290+0000 7fb2dea00640 1 -- 172.21.15.40:0/1725163447 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) ==== 169+0+0 (secure 0 0 0) 0x7fb2e007a350 con 0x7fb2e815f420 2024-04-26T23:02:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-04-26T23:02:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.294+0000 7fb2e7e00640 1 -- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb2c406c940 msgr2=0x7fb2c406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.294+0000 7fb2e7e00640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb2c406c940 0x7fb2c406ee00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800b810 tx=0x7fb2d8008040 comp rx=0 tx=0).stop 2024-04-26T23:02:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.294+0000 7fb2e7e00640 1 -- 172.21.15.40:0/1725163447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e815f420 msgr2=0x7fb2e8150090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.294+0000 7fb2e7e00640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e815f420 0x7fb2e8150090 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb2e002a1e0 tx=0x7fb2e00046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:38.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.295+0000 7fb2e7e00640 1 -- 172.21.15.40:0/1725163447 shutdown_connections 2024-04-26T23:02:38.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.295+0000 7fb2e7e00640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb2c406c940 0x7fb2c406ee00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:38.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.295+0000 7fb2e7e00640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e815f420 0x7fb2e8150090 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:38.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.295+0000 7fb2e7e00640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2e80d3b60 0x7fb2e81f4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:38.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.295+0000 7fb2e7e00640 1 --2- 172.21.15.40:0/1725163447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2e80b3d90 0x7fb2e815ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:38.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.295+0000 7fb2e7e00640 1 -- 172.21.15.40:0/1725163447 >> 172.21.15.40:0/1725163447 conn(0x7fb2e809e1d0 msgr2=0x7fb2e80d3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.299+0000 7fb2e7e00640 1 -- 172.21.15.40:0/1725163447 shutdown_connections 2024-04-26T23:02:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.299+0000 7fb2e7e00640 1 -- 172.21.15.40:0/1725163447 wait complete. 2024-04-26T23:02:38.327 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-26T23:02:38.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.538+0000 7ff88e600640 1 Processor -- start 2024-04-26T23:02:38.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.538+0000 7ff88e600640 1 -- start start 2024-04-26T23:02:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.540+0000 7ff88e600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff88806e700 0x7ff8880b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.540+0000 7ff88e600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff88806eae0 0x7ff8880ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.540+0000 7ff88e600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff88815a4a0 0x7ff88815a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.540+0000 7ff88e600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff88815ae50 con 0x7ff88806e700 2024-04-26T23:02:38.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.540+0000 7ff88e600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff88815d8d0 con 0x7ff88815a4a0 2024-04-26T23:02:38.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.540+0000 7ff88e600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff88815dbb0 con 0x7ff88806eae0 2024-04-26T23:02:38.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.541+0000 7ff887400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff88806e700 0x7ff8880b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:38.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.541+0000 7ff887400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff88806e700 0x7ff8880b3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32842/0 (socket says 172.21.15.40:32842) 2024-04-26T23:02:38.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.541+0000 7ff887400640 1 -- 172.21.15.40:0/3424487738 learned_addr learned my addr 172.21.15.40:0/3424487738 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:38.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.541+0000 7ff887e00640 1 --2- 172.21.15.40:0/3424487738 >> v2:172.21.15.103:3300/0 conn(0x7ff88815a4a0 0x7ff88815a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:38.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.541+0000 7ff886a00640 1 --2- 172.21.15.40:0/3424487738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff88806eae0 0x7ff8880ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:38.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.541+0000 7ff887400640 1 -- 172.21.15.40:0/3424487738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff88806eae0 msgr2=0x7ff8880ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:38.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.541+0000 7ff887400640 1 --2- 172.21.15.40:0/3424487738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff88806eae0 0x7ff8880ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:38.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.541+0000 7ff887400640 1 -- 172.21.15.40:0/3424487738 >> v2:172.21.15.103:3300/0 conn(0x7ff88815a4a0 msgr2=0x7ff88815a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:38.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.541+0000 7ff887400640 1 --2- 172.21.15.40:0/3424487738 >> v2:172.21.15.103:3300/0 conn(0x7ff88815a4a0 0x7ff88815a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:38.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.541+0000 7ff887400640 1 -- 172.21.15.40:0/3424487738 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88815de90 con 0x7ff88806e700 2024-04-26T23:02:38.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.541+0000 7ff886a00640 1 --2- 172.21.15.40:0/3424487738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff88806eae0 0x7ff8880ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:38.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.542+0000 7ff887400640 1 --2- 172.21.15.40:0/3424487738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff88806e700 0x7ff8880b3450 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff8880cf330 tx=0x7ff87802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25583b4af1d92e49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:38.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.542+0000 7ff886000640 1 -- 172.21.15.40:0/3424487738 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878036070 con 0x7ff88806e700 2024-04-26T23:02:38.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.543+0000 7ff886000640 1 -- 172.21.15.40:0/3424487738 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff878031600 con 0x7ff88806e700 2024-04-26T23:02:38.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.543+0000 7ff886000640 1 -- 172.21.15.40:0/3424487738 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8780319d0 con 0x7ff88806e700 2024-04-26T23:02:38.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.543+0000 7ff88e600640 1 -- 172.21.15.40:0/3424487738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff88806e700 msgr2=0x7ff8880b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:38.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.543+0000 7ff88e600640 1 --2- 172.21.15.40:0/3424487738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff88806e700 0x7ff8880b3450 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff8880cf330 tx=0x7ff87802a6a0 comp rx=0 tx=0).stop 2024-04-26T23:02:38.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.543+0000 7ff88e600640 1 -- 172.21.15.40:0/3424487738 shutdown_connections 2024-04-26T23:02:38.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.543+0000 7ff88e600640 1 --2- 172.21.15.40:0/3424487738 >> v2:172.21.15.103:3300/0 conn(0x7ff88815a4a0 0x7ff88815a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:38.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.543+0000 7ff88e600640 1 --2- 172.21.15.40:0/3424487738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff88806eae0 0x7ff8880ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:38.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.543+0000 7ff88e600640 1 --2- 172.21.15.40:0/3424487738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff88806e700 0x7ff8880b3450 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:38.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.543+0000 7ff88e600640 1 -- 172.21.15.40:0/3424487738 >> 172.21.15.40:0/3424487738 conn(0x7ff88806f310 msgr2=0x7ff88806f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:38.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.543+0000 7ff88e600640 1 -- 172.21.15.40:0/3424487738 shutdown_connections 2024-04-26T23:02:38.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.543+0000 7ff88e600640 1 -- 172.21.15.40:0/3424487738 wait complete. 2024-04-26T23:02:38.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.545+0000 7ff88e600640 1 Processor -- start 2024-04-26T23:02:38.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.545+0000 7ff88e600640 1 -- start start 2024-04-26T23:02:38.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.546+0000 7ff88e600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff88806e700 0x7ff88815eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:38.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.546+0000 7ff88e600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff88806eae0 0x7ff8881512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:38.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.546+0000 7ff88e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff88815a4a0 0x7ff88815b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:38.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.546+0000 7ff88e600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff8880d08c0 con 0x7ff88815a4a0 2024-04-26T23:02:38.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.546+0000 7ff88e600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff8880ce740 con 0x7ff88806e700 2024-04-26T23:02:38.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.546+0000 7ff88e600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff8880d0a40 con 0x7ff88806eae0 2024-04-26T23:02:38.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.546+0000 7ff887e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff88815a4a0 0x7ff88815b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:38.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.546+0000 7ff887e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff88815a4a0 0x7ff88815b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32844/0 (socket says 172.21.15.40:32844) 2024-04-26T23:02:38.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.546+0000 7ff887e00640 1 -- 172.21.15.40:0/3392139302 learned_addr learned my addr 172.21.15.40:0/3392139302 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:38.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.547+0000 7ff887400640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff88806e700 0x7ff88815eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:38.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.547+0000 7ff886a00640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff88806eae0 0x7ff8881512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:38.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.547+0000 7ff887e00640 1 -- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff88806eae0 msgr2=0x7ff8881512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:38.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.547+0000 7ff887e00640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff88806eae0 0x7ff8881512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:38.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.547+0000 7ff887e00640 1 -- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff88806e700 msgr2=0x7ff88815eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:38.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.547+0000 7ff887e00640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff88806e700 0x7ff88815eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:38.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.547+0000 7ff887e00640 1 -- 172.21.15.40:0/3392139302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8880b4380 con 0x7ff88815a4a0 2024-04-26T23:02:38.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.547+0000 7ff886a00640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff88806eae0 0x7ff8881512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:38.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.547+0000 7ff887400640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff88806e700 0x7ff88815eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:02:38.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.547+0000 7ff887e00640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff88815a4a0 0x7ff88815b620 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff88000d800 tx=0x7ff88000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:38.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.548+0000 7ff87f400640 1 -- 172.21.15.40:0/3392139302 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88000b6c0 con 0x7ff88815a4a0 2024-04-26T23:02:38.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.548+0000 7ff87f400640 1 -- 172.21.15.40:0/3392139302 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff880019070 con 0x7ff88815a4a0 2024-04-26T23:02:38.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.548+0000 7ff87f400640 1 -- 172.21.15.40:0/3392139302 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88000bac0 con 0x7ff88815a4a0 2024-04-26T23:02:38.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.548+0000 7ff88e600640 1 -- 172.21.15.40:0/3392139302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8880b4660 con 0x7ff88815a4a0 2024-04-26T23:02:38.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.548+0000 7ff88e600640 1 -- 172.21.15.40:0/3392139302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8880b4c20 con 0x7ff88815a4a0 2024-04-26T23:02:38.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.549+0000 7ff88e600640 1 -- 172.21.15.40:0/3392139302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff888069600 con 0x7ff88815a4a0 2024-04-26T23:02:38.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.550+0000 7ff87f400640 1 -- 172.21.15.40:0/3392139302 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff88000bc60 con 0x7ff88815a4a0 2024-04-26T23:02:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.551+0000 7ff87f400640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff86406c920 0x7ff86406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.551+0000 7ff87f400640 1 -- 172.21.15.40:0/3392139302 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5853+0+0 (secure 0 0 0) 0x7ff88008bfb0 con 0x7ff88815a4a0 2024-04-26T23:02:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.552+0000 7ff887400640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff86406c920 0x7ff86406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:38.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.553+0000 7ff887400640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff86406c920 0x7ff86406ede0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff878033000 tx=0x7ff878009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:38.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.561+0000 7ff87f400640 1 -- 172.21.15.40:0/3392139302 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff88005a340 con 0x7ff88815a4a0 2024-04-26T23:02:38.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:38.774+0000 7ff88e600640 1 -- 172.21.15.40:0/3392139302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff88815a9b0 con 0x7ff88815a4a0 2024-04-26T23:02:39.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.308+0000 7ff87f400640 1 -- 172.21.15.40:0/3392139302 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) ==== 178+0+0 (secure 0 0 0) 0x7ff880059c40 con 0x7ff88815a4a0 2024-04-26T23:02:39.308 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data_ec' removed 2024-04-26T23:02:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.312+0000 7ff88e600640 1 -- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff86406c920 msgr2=0x7ff86406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.312+0000 7ff88e600640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff86406c920 0x7ff86406ede0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff878033000 tx=0x7ff878009000 comp rx=0 tx=0).stop 2024-04-26T23:02:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.312+0000 7ff88e600640 1 -- 172.21.15.40:0/3392139302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff88815a4a0 msgr2=0x7ff88815b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.312+0000 7ff88e600640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff88815a4a0 0x7ff88815b620 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff88000d800 tx=0x7ff88000dcc0 comp rx=0 tx=0).stop 2024-04-26T23:02:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.312+0000 7ff88e600640 1 -- 172.21.15.40:0/3392139302 shutdown_connections 2024-04-26T23:02:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.312+0000 7ff88e600640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff86406c920 0x7ff86406ede0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.312+0000 7ff88e600640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff88815a4a0 0x7ff88815b620 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.312+0000 7ff88e600640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff88806eae0 0x7ff8881512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.312+0000 7ff88e600640 1 --2- 172.21.15.40:0/3392139302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff88806e700 0x7ff88815eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.312+0000 7ff88e600640 1 -- 172.21.15.40:0/3392139302 >> 172.21.15.40:0/3392139302 conn(0x7ff88806f310 msgr2=0x7ff8881598d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:39.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.314+0000 7ff88e600640 1 -- 172.21.15.40:0/3392139302 shutdown_connections 2024-04-26T23:02:39.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.314+0000 7ff88e600640 1 -- 172.21.15.40:0/3392139302 wait complete. 2024-04-26T23:02:39.339 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-26T23:02:39.339 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-26T23:02:39.339 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-26T23:02:39.340 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:02:39.340 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-26T23:02:39.340 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-26T23:02:39.340 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-26T23:02:39.340 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:02:39.340 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-26T23:02:39.340 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-26T23:02:39.341 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-26T23:02:39.341 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:02:39.341 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-26T23:02:39.341 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-26T23:02:39.341 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-26T23:02:39.341 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:02:39.362 INFO:tasks.ceph.mds.c:Stopped 2024-04-26T23:02:39.362 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-26T23:02:39.363 INFO:tasks.ceph.mds.a:Stopped 2024-04-26T23:02:39.363 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-26T23:02:39.364 INFO:tasks.ceph.mds.b:Stopped 2024-04-26T23:02:39.364 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-26T23:02:39.365 INFO:tasks.ceph.mds.d:Stopped 2024-04-26T23:02:39.365 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-26T23:02:39.411 INFO:tasks.ceph.mds.c:Started 2024-04-26T23:02:39.411 DEBUG:teuthology.parallel:result is None 2024-04-26T23:02:39.412 INFO:tasks.ceph.mds.b:Started 2024-04-26T23:02:39.412 DEBUG:teuthology.parallel:result is None 2024-04-26T23:02:39.417 INFO:tasks.ceph.mds.a:Started 2024-04-26T23:02:39.417 DEBUG:teuthology.parallel:result is None 2024-04-26T23:02:39.417 INFO:tasks.ceph.mds.d:Started 2024-04-26T23:02:39.417 DEBUG:teuthology.parallel:result is None 2024-04-26T23:02:39.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-26T23:02:39.572 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-26T23:02:39.574 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-26T23:02:39.579 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-04-26T23:02:39.580 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:02:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.612+0000 7f5003200640 1 Processor -- start 2024-04-26T23:02:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.612+0000 7f5003200640 1 -- start start 2024-04-26T23:02:39.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.613+0000 7f5003200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f50040b6220 0x7f50040d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:39.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.613+0000 7f5003200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50040d39f0 0x7f50040b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:39.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.613+0000 7f5003200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5004162c40 con 0x7f50040d08d0 2024-04-26T23:02:39.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.614+0000 7f5003200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f50040d2d80 con 0x7f50040b6220 2024-04-26T23:02:39.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.614+0000 7f5003200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f50040b7c80 con 0x7f50040d39f0 2024-04-26T23:02:39.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.614+0000 7f4ffbe00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f50040d08d0 0x7f50040d32e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:50970/0 (socket says 172.21.15.40:50970) 2024-04-26T23:02:39.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.614+0000 7f5001e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f50040b6220 0x7f50040d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:39.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.614+0000 7f4ffbe00640 1 -- 172.21.15.40:0/1949683691 learned_addr learned my addr 172.21.15.40:0/1949683691 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:39.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.614+0000 7f5002800640 1 --2- 172.21.15.40:0/1949683691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50040d39f0 0x7f50040b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:39.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.614+0000 7f5002800640 1 -- 172.21.15.40:0/1949683691 >> v2:172.21.15.40:3300/0 conn(0x7f50040b6220 msgr2=0x7f50040d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:39.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.614+0000 7f5002800640 1 --2- 172.21.15.40:0/1949683691 >> v2:172.21.15.40:3300/0 conn(0x7f50040b6220 0x7f50040d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:39.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.614+0000 7f5002800640 1 -- 172.21.15.40:0/1949683691 >> v1:172.21.15.103:6789/0 conn(0x7f50040d08d0 legacy=0x7f50040d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:39.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.614+0000 7f5002800640 1 -- 172.21.15.40:0/1949683691 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50040b7f60 con 0x7f50040d39f0 2024-04-26T23:02:39.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.614+0000 7f5001400640 1 -- 172.21.15.40:0/1949683691 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1621562172 0 0) 0x7f5004162c40 con 0x7f50040d08d0 2024-04-26T23:02:39.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.615+0000 7f5002800640 1 --2- 172.21.15.40:0/1949683691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50040d39f0 0x7f50040b8d20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc009d90 tx=0x7f4ffc02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd7bf302b3329e04 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.616+0000 7f5001400640 1 -- 172.21.15.40:0/1949683691 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc007740 con 0x7f50040d39f0 2024-04-26T23:02:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.616+0000 7f5001400640 1 -- 172.21.15.40:0/1949683691 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ffc0078e0 con 0x7f50040d39f0 2024-04-26T23:02:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.616+0000 7f5001400640 1 -- 172.21.15.40:0/1949683691 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc007cb0 con 0x7f50040d39f0 2024-04-26T23:02:39.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.617+0000 7f5003200640 1 -- 172.21.15.40:0/1949683691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50040d39f0 msgr2=0x7f50040b8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:39.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.617+0000 7f5003200640 1 --2- 172.21.15.40:0/1949683691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50040d39f0 0x7f50040b8d20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc009d90 tx=0x7f4ffc02a660 comp rx=0 tx=0).stop 2024-04-26T23:02:39.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.617+0000 7f5003200640 1 -- 172.21.15.40:0/1949683691 shutdown_connections 2024-04-26T23:02:39.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.617+0000 7f5003200640 1 --2- 172.21.15.40:0/1949683691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50040d39f0 0x7f50040b8d20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:39.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.617+0000 7f5003200640 1 --2- 172.21.15.40:0/1949683691 >> v2:172.21.15.40:3300/0 conn(0x7f50040b6220 0x7f50040d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:39.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.617+0000 7f5003200640 1 -- 172.21.15.40:0/1949683691 >> 172.21.15.40:0/1949683691 conn(0x7f50040d4d90 msgr2=0x7f50040d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:39.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.617+0000 7f5003200640 1 -- 172.21.15.40:0/1949683691 shutdown_connections 2024-04-26T23:02:39.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.617+0000 7f5003200640 1 -- 172.21.15.40:0/1949683691 wait complete. 2024-04-26T23:02:39.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.619+0000 7f5003200640 1 Processor -- start 2024-04-26T23:02:39.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.619+0000 7f5003200640 1 -- start start 2024-04-26T23:02:39.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.619+0000 7f5003200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50040b6220 0x7f500406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:39.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5003200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50040d08d0 0x7f50041f47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:39.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5003200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50040d39f0 0x7f500415f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:39.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5003200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f50040b9260 con 0x7f50040d39f0 2024-04-26T23:02:39.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5003200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f50040b8520 con 0x7f50040b6220 2024-04-26T23:02:39.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5003200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f50040b93e0 con 0x7f50040d08d0 2024-04-26T23:02:39.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5002800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50040d39f0 0x7f500415f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5002800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50040d39f0 0x7f500415f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32894/0 (socket says 172.21.15.40:32894) 2024-04-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5002800640 1 -- 172.21.15.40:0/2062450560 learned_addr learned my addr 172.21.15.40:0/2062450560 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f4ffbe00640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50040d08d0 0x7f50041f47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5001e00640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50040b6220 0x7f500406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5002800640 1 -- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50040d08d0 msgr2=0x7f50041f47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5002800640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50040d08d0 0x7f50041f47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5002800640 1 -- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50040b6220 msgr2=0x7f500406d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5002800640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50040b6220 0x7f500406d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.620+0000 7f5002800640 1 -- 172.21.15.40:0/2062450560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50041f5020 con 0x7f50040d39f0 2024-04-26T23:02:39.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.621+0000 7f5002800640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50040d39f0 0x7f500415f680 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc009eb0 tx=0x7f4ffc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.621+0000 7f4ffa000640 1 -- 172.21.15.40:0/2062450560 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc03f070 con 0x7f50040d39f0 2024-04-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.621+0000 7f4ffa000640 1 -- 172.21.15.40:0/2062450560 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ffc031bc0 con 0x7f50040d39f0 2024-04-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.621+0000 7f4ffa000640 1 -- 172.21.15.40:0/2062450560 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc009450 con 0x7f50040d39f0 2024-04-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.621+0000 7f5003200640 1 -- 172.21.15.40:0/2062450560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50041f5300 con 0x7f50040d39f0 2024-04-26T23:02:39.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.621+0000 7f5003200640 1 -- 172.21.15.40:0/2062450560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50041f4d20 con 0x7f50040d39f0 2024-04-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.623+0000 7f5003200640 1 -- 172.21.15.40:0/2062450560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50041f54d0 con 0x7f50040d39f0 2024-04-26T23:02:39.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.623+0000 7f4ffa000640 1 -- 172.21.15.40:0/2062450560 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4ffc03b410 con 0x7f50040d39f0 2024-04-26T23:02:39.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.624+0000 7f4ffa000640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4fe006c920 0x7f4fe006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.625+0000 7f4ffa000640 1 -- 172.21.15.40:0/2062450560 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5335+0+0 (secure 0 0 0) 0x7f4ffc0acfe0 con 0x7f50040d39f0 2024-04-26T23:02:39.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.625+0000 7f5001e00640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4fe006c920 0x7f4fe006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:39.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.626+0000 7f5001e00640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4fe006c920 0x7f4fe006ede0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4009b40 tx=0x7f4ff4009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:39.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.629+0000 7f4ffa000640 1 -- 172.21.15.40:0/2062450560 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4ffc07af60 con 0x7f50040d39f0 2024-04-26T23:02:39.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:39.846+0000 7f5003200640 1 -- 172.21.15.40:0/2062450560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f5004069370 con 0x7f50040d39f0 2024-04-26T23:02:40.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.345+0000 7f4ffa000640 1 -- 172.21.15.40:0/2062450560 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) ==== 97+0+0 (secure 0 0 0) 0x7f4ffc07f260 con 0x7f50040d39f0 2024-04-26T23:02:40.346 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-04-26T23:02:40.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.349+0000 7f5003200640 1 -- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4fe006c920 msgr2=0x7f4fe006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:40.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.349+0000 7f5003200640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4fe006c920 0x7f4fe006ede0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4009b40 tx=0x7f4ff4009190 comp rx=0 tx=0).stop 2024-04-26T23:02:40.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.350+0000 7f5003200640 1 -- 172.21.15.40:0/2062450560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50040d39f0 msgr2=0x7f500415f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:40.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.350+0000 7f5003200640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50040d39f0 0x7f500415f680 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc009eb0 tx=0x7f4ffc004a40 comp rx=0 tx=0).stop 2024-04-26T23:02:40.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.350+0000 7f5003200640 1 -- 172.21.15.40:0/2062450560 shutdown_connections 2024-04-26T23:02:40.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.350+0000 7f5003200640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4fe006c920 0x7f4fe006ede0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.350+0000 7f5003200640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50040d39f0 0x7f500415f680 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.350+0000 7f5003200640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50040d08d0 0x7f50041f47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.350+0000 7f5003200640 1 --2- 172.21.15.40:0/2062450560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50040b6220 0x7f500406d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.350+0000 7f5003200640 1 -- 172.21.15.40:0/2062450560 >> 172.21.15.40:0/2062450560 conn(0x7f50040d4d90 msgr2=0x7f50040cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:40.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.350+0000 7f5003200640 1 -- 172.21.15.40:0/2062450560 shutdown_connections 2024-04-26T23:02:40.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.350+0000 7f5003200640 1 -- 172.21.15.40:0/2062450560 wait complete. 2024-04-26T23:02:40.372 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-26T23:02:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.581+0000 7fc758e00640 1 Processor -- start 2024-04-26T23:02:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.581+0000 7fc758e00640 1 -- start start 2024-04-26T23:02:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.582+0000 7fc758e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc7540d17d0 0x7fc7540b6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.583+0000 7fc758e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7540d0c80 0x7fc7540b5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.583+0000 7fc758e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7540b66c0 con 0x7fc7540d2400 2024-04-26T23:02:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.583+0000 7fc758e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc7540d37b0 con 0x7fc7540d17d0 2024-04-26T23:02:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.583+0000 7fc758e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc7540d3d70 con 0x7fc7540d0c80 2024-04-26T23:02:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.583+0000 7fc752a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc7540d2400 0x7fc7540d1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:54978/0 (socket says 172.21.15.40:54978) 2024-04-26T23:02:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.583+0000 7fc752a00640 1 -- 172.21.15.40:0/3739909966 learned_addr learned my addr 172.21.15.40:0/3739909966 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:40.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.583+0000 7fc753400640 1 --2- 172.21.15.40:0/3739909966 >> v2:172.21.15.103:3300/0 conn(0x7fc7540d17d0 0x7fc7540b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:40.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.583+0000 7fc753e00640 1 --2- 172.21.15.40:0/3739909966 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7540d0c80 0x7fc7540b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:40.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.583+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2697829429 0 0) 0x7fc744001490 con 0x7fc7540d2400 2024-04-26T23:02:40.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc738003520 con 0x7fc7540d2400 2024-04-26T23:02:40.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3850483582 0 0) 0x7fc738003520 con 0x7fc7540d2400 2024-04-26T23:02:40.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7540d0c80 msgr2=0x7fc7540b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:40.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 --2- 172.21.15.40:0/3739909966 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7540d0c80 0x7fc7540b5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 >> v2:172.21.15.103:3300/0 conn(0x7fc7540d17d0 msgr2=0x7fc7540b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 --2- 172.21.15.40:0/3739909966 >> v2:172.21.15.103:3300/0 conn(0x7fc7540d17d0 0x7fc7540b6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75415f9c0 con 0x7fc7540d2400 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fc744001bc0 con 0x7fc7540d2400 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 >> v1:172.21.15.40:6789/0 conn(0x7fc7540d2400 legacy=0x7fc7540d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 --2- 172.21.15.40:0/3739909966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7380039b0 0x7fc738003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 --2- 172.21.15.40:0/3739909966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738004800 0x7fc738004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 --2- 172.21.15.40:0/3739909966 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7540d0c80 0x7fc7380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc7540d37b0 con 0x7fc7380039b0 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc7540d3d70 con 0x7fc738004800 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc73800b9c0 con 0x7fc7540d0c80 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.584+0000 7fc753400640 1 --2- 172.21.15.40:0/3739909966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7380039b0 0x7fc738003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.585+0000 7fc752a00640 1 --2- 172.21.15.40:0/3739909966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738004800 0x7fc738004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.585+0000 7fc753400640 1 -- 172.21.15.40:0/3739909966 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7540d0c80 msgr2=0x7fc7380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.585+0000 7fc753400640 1 --2- 172.21.15.40:0/3739909966 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7540d0c80 0x7fc7380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.585+0000 7fc753400640 1 -- 172.21.15.40:0/3739909966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738004800 msgr2=0x7fc738004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.585+0000 7fc753400640 1 --2- 172.21.15.40:0/3739909966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738004800 0x7fc738004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.585+0000 7fc753400640 1 -- 172.21.15.40:0/3739909966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc738003520 con 0x7fc7380039b0 2024-04-26T23:02:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.585+0000 7fc752a00640 1 --2- 172.21.15.40:0/3739909966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738004800 0x7fc738004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.586+0000 7fc753400640 1 --2- 172.21.15.40:0/3739909966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7380039b0 0x7fc738003dd0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc73c009af0 tx=0x7fc738005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d9ced7dbd0649b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.586+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c036070 con 0x7fc7380039b0 2024-04-26T23:02:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.586+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc73c031400 con 0x7fc7380039b0 2024-04-26T23:02:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.586+0000 7fc752000640 1 -- 172.21.15.40:0/3739909966 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c009690 con 0x7fc7380039b0 2024-04-26T23:02:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.587+0000 7fc758e00640 1 -- 172.21.15.40:0/3739909966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7380039b0 msgr2=0x7fc738003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.587+0000 7fc758e00640 1 --2- 172.21.15.40:0/3739909966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7380039b0 0x7fc738003dd0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc73c009af0 tx=0x7fc738005830 comp rx=0 tx=0).stop 2024-04-26T23:02:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.587+0000 7fc753e00640 1 -- 172.21.15.40:0/3739909966 reap_dead start 2024-04-26T23:02:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.587+0000 7fc758e00640 1 -- 172.21.15.40:0/3739909966 shutdown_connections 2024-04-26T23:02:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.587+0000 7fc758e00640 1 -- 172.21.15.40:0/3739909966 >> 172.21.15.40:0/3739909966 conn(0x7fc75409e210 msgr2=0x7fc7540d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.587+0000 7fc758e00640 1 -- 172.21.15.40:0/3739909966 shutdown_connections 2024-04-26T23:02:40.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.587+0000 7fc758e00640 1 -- 172.21.15.40:0/3739909966 wait complete. 2024-04-26T23:02:40.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.589+0000 7fc758e00640 1 Processor -- start 2024-04-26T23:02:40.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.589+0000 7fc758e00640 1 -- start start 2024-04-26T23:02:40.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.590+0000 7fc758e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7380039b0 0x7fc754162730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:40.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.590+0000 7fc758e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75414fc10 0x7fc75414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:40.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.590+0000 7fc758e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc75415fc50 0x7fc7540b9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:40.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.590+0000 7fc758e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc7540d3a00 con 0x7fc75414fc10 2024-04-26T23:02:40.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.590+0000 7fc758e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc7540d5a10 con 0x7fc75415fc50 2024-04-26T23:02:40.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.590+0000 7fc758e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc7540d1130 con 0x7fc7380039b0 2024-04-26T23:02:40.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.591+0000 7fc752a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75414fc10 0x7fc75414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:40.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.591+0000 7fc752a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75414fc10 0x7fc75414f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32912/0 (socket says 172.21.15.40:32912) 2024-04-26T23:02:40.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.591+0000 7fc752a00640 1 -- 172.21.15.40:0/1659106994 learned_addr learned my addr 172.21.15.40:0/1659106994 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:40.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.591+0000 7fc753400640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7380039b0 0x7fc754162730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:40.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.591+0000 7fc753e00640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc75415fc50 0x7fc7540b9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:40.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.591+0000 7fc752a00640 1 -- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7380039b0 msgr2=0x7fc754162730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:40.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.591+0000 7fc752a00640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7380039b0 0x7fc754162730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.591+0000 7fc752a00640 1 -- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc75415fc50 msgr2=0x7fc7540b9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:40.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.591+0000 7fc752a00640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc75415fc50 0x7fc7540b9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.591+0000 7fc752a00640 1 -- 172.21.15.40:0/1659106994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7540cd510 con 0x7fc75414fc10 2024-04-26T23:02:40.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.591+0000 7fc752a00640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75414fc10 0x7fc75414f3e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc744001bd0 tx=0x7fc74400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:40.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.592+0000 7fc74b400640 1 -- 172.21.15.40:0/1659106994 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744019070 con 0x7fc75414fc10 2024-04-26T23:02:40.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.592+0000 7fc74b400640 1 -- 172.21.15.40:0/1659106994 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc744004d60 con 0x7fc75414fc10 2024-04-26T23:02:40.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.592+0000 7fc74b400640 1 -- 172.21.15.40:0/1659106994 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7440072d0 con 0x7fc75414fc10 2024-04-26T23:02:40.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.592+0000 7fc758e00640 1 -- 172.21.15.40:0/1659106994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7540cd7f0 con 0x7fc75414fc10 2024-04-26T23:02:40.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.592+0000 7fc758e00640 1 -- 172.21.15.40:0/1659106994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7540cddb0 con 0x7fc75414fc10 2024-04-26T23:02:40.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.593+0000 7fc758e00640 1 -- 172.21.15.40:0/1659106994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc754069370 con 0x7fc75414fc10 2024-04-26T23:02:40.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.594+0000 7fc74b400640 1 -- 172.21.15.40:0/1659106994 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc74401c070 con 0x7fc75414fc10 2024-04-26T23:02:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.595+0000 7fc74b400640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc73006c940 0x7fc73006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.595+0000 7fc74b400640 1 -- 172.21.15.40:0/1659106994 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fc74408b9b0 con 0x7fc75414fc10 2024-04-26T23:02:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.596+0000 7fc753400640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc73006c940 0x7fc73006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.597+0000 7fc753400640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc73006c940 0x7fc73006ee00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc73c033000 tx=0x7fc73c008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:40.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.600+0000 7fc74b400640 1 -- 172.21.15.40:0/1659106994 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc7540b66c0 con 0x7fc75414fc10 2024-04-26T23:02:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.879+0000 7fc758e00640 1 -- 172.21.15.40:0/1659106994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fc7540d5db0 con 0x7fc75414fc10 2024-04-26T23:02:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.881+0000 7fc74b400640 1 -- 172.21.15.40:0/1659106994 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v8) ==== 80+0+6645 (secure 0 0 0) 0x7fc74400be20 con 0x7fc75414fc10 2024-04-26T23:02:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:40.881 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:02:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-04-26T23:02:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-04-26T23:02:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmF/UQHBAACdb8iskp2MuBMV6S5tYFJg==", 2024-04-26T23:02:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:02:40.881 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmniCyMxAAT/rVbIKj9EnvwDiblKMCeg==", 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmLGZhJxAAg/fsrRrNoF/Jx1PnBVjjeQ==", 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxms3DnAxAAxJj/rybrv/0cLK8PmQ619A==", 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.883 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDrMSxmbeONCxAATB67ujBDn2tVxpbMju7Mfg==", 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDuMSxmqE/PJhAAsNNu5ar8tHWQauxaXOvagA==", 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDyMSxmd2trBhAAbnl7EBldxgtOd1s0Ll0A9w==", 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:02:40.885 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD1MSxmqdtKEhAA9oxBxOu/musCKyladrJgYA==", 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-04-26T23:02:40.886 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD7MSxm+kl7IxAAlPAGIemdC8gcvp+koeULaQ==", 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD+MSxmzKiaLRAA2UPS1wwJHgMxQf/zLMjMVQ==", 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.887 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQACMixmluHrCxAAmbmb33f5MVSTWNuy1D0Zsg==", 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAFMixm6GQjJRAArmdnX7ioekT8GSdsbeB1Ow==", 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:02:40.888 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg==", 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-04-26T23:02:40.889 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmVhn4IhAA94XsDk1x5Oax5smXPoFQKg==", 2024-04-26T23:02:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:02:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:02:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:02:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:02:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-04-26T23:02:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxm3qhbGRAAK62blgPVShIYskDsMcaffQ==", 2024-04-26T23:02:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:02:40.890 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmxof/KxAAUmXVV4Vdp2DZnRjQzQ0Inw==", 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.891 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDlMSxmmVwcGRAAqgHymPPTPuS6XXC6LuOVPg==", 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.bootstrap-mds", 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYMixmNT+lABAAmVd4839OUENFGXLeHQEvXg==", 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile bootstrap-mds" 2024-04-26T23:02:40.892 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.bootstrap-mgr", 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYMixmeF6lABAAQHgM7pztQRIA9+r12SL+HA==", 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.bootstrap-osd", 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYMixm0nelABAAirrO2L0V++wbz378Aqy0mQ==", 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile bootstrap-osd" 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.bootstrap-rbd", 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYMixm1ZSlABAATz0ncfkAPuKI6MVmr9pBug==", 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYMixmVK6lABAAH8vp2tAIQaAkJ+2SPeM5bw==", 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.bootstrap-rgw", 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYMixm9MelABAAPZHwOf5yeu4fTeMju4/5TA==", 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmVf+9EhAAcpk8NgFqgCLXeHA2z7oHmA==", 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-04-26T23:02:40.895 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:02:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-04-26T23:02:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmjjpcCBAAjDQRUBn/TK9hF3T2WBObyw==", 2024-04-26T23:02:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:02:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:02:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-04-26T23:02:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:02:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-26T23:02:40.896 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:02:40.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.883+0000 7fc758e00640 1 -- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc73006c940 msgr2=0x7fc73006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:40.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.883+0000 7fc758e00640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc73006c940 0x7fc73006ee00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc73c033000 tx=0x7fc73c008030 comp rx=0 tx=0).stop 2024-04-26T23:02:40.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.883+0000 7fc758e00640 1 -- 172.21.15.40:0/1659106994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75414fc10 msgr2=0x7fc75414f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:40.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.883+0000 7fc758e00640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75414fc10 0x7fc75414f3e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc744001bd0 tx=0x7fc74400b4a0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.883+0000 7fc758e00640 1 -- 172.21.15.40:0/1659106994 shutdown_connections 2024-04-26T23:02:40.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.883+0000 7fc758e00640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc73006c940 0x7fc73006ee00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.883+0000 7fc758e00640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc75415fc50 0x7fc7540b9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.883+0000 7fc758e00640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc75414fc10 0x7fc75414f3e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.883+0000 7fc758e00640 1 --2- 172.21.15.40:0/1659106994 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc7380039b0 0x7fc754162730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:40.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.883+0000 7fc758e00640 1 -- 172.21.15.40:0/1659106994 >> 172.21.15.40:0/1659106994 conn(0x7fc75409e210 msgr2=0x7fc7540b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:40.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.884+0000 7fc758e00640 1 -- 172.21.15.40:0/1659106994 shutdown_connections 2024-04-26T23:02:40.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:40.884+0000 7fc758e00640 1 -- 172.21.15.40:0/1659106994 wait complete. 2024-04-26T23:02:40.903 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-26T23:02:41.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.096+0000 7fee8a800640 1 Processor -- start 2024-04-26T23:02:41.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.096+0000 7fee8a800640 1 -- start start 2024-04-26T23:02:41.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.097+0000 7fee8a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fee840c41d0 0x7fee840cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.098+0000 7fee8a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee841678f0 0x7fee840cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.098+0000 7fee8a800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee84163a50 con 0x7fee84073f30 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.098+0000 7fee8a800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fee840cf7a0 con 0x7fee840c41d0 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.098+0000 7fee8a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fee840ceb90 con 0x7fee841678f0 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.098+0000 7fee83400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fee840c41d0 0x7fee840cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.098+0000 7fee83400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fee840c41d0 0x7fee840cfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32918/0 (socket says 172.21.15.40:32918) 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.098+0000 7fee83400640 1 -- 172.21.15.40:0/2356396261 learned_addr learned my addr 172.21.15.40:0/2356396261 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.098+0000 7fee7be00640 1 --1- 172.21.15.40:0/2356396261 >> v1:172.21.15.40:6789/0 conn(0x7fee84073f30 0x7fee841671e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:54994/0 (socket says 172.21.15.40:54994) 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.098+0000 7fee83e00640 1 --2- 172.21.15.40:0/2356396261 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee841678f0 0x7fee840cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.099+0000 7fee83400640 1 -- 172.21.15.40:0/2356396261 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee841678f0 msgr2=0x7fee840cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.099+0000 7fee83400640 1 --2- 172.21.15.40:0/2356396261 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee841678f0 0x7fee840cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.099+0000 7fee83400640 1 -- 172.21.15.40:0/2356396261 >> v1:172.21.15.40:6789/0 conn(0x7fee84073f30 legacy=0x7fee841671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.099+0000 7fee82a00640 1 -- 172.21.15.40:0/2356396261 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1108715355 0 0) 0x7fee84163a50 con 0x7fee84073f30 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.099+0000 7fee83400640 1 -- 172.21.15.40:0/2356396261 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee840cee70 con 0x7fee840c41d0 2024-04-26T23:02:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.099+0000 7fee83e00640 1 --2- 172.21.15.40:0/2356396261 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee841678f0 0x7fee840cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.100+0000 7fee83400640 1 --2- 172.21.15.40:0/2356396261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee840c41d0 0x7fee840cfc30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fee74009870 tx=0x7fee7402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6510b04cc33e6d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.100+0000 7fee82a00640 1 -- 172.21.15.40:0/2356396261 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee74037070 con 0x7fee840c41d0 2024-04-26T23:02:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.100+0000 7fee82a00640 1 -- 172.21.15.40:0/2356396261 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee74032440 con 0x7fee840c41d0 2024-04-26T23:02:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.100+0000 7fee82a00640 1 -- 172.21.15.40:0/2356396261 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee74032810 con 0x7fee840c41d0 2024-04-26T23:02:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.100+0000 7fee8a800640 1 -- 172.21.15.40:0/2356396261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee840c41d0 msgr2=0x7fee840cfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.100+0000 7fee8a800640 1 --2- 172.21.15.40:0/2356396261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee840c41d0 0x7fee840cfc30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fee74009870 tx=0x7fee7402a250 comp rx=0 tx=0).stop 2024-04-26T23:02:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.101+0000 7fee8a800640 1 -- 172.21.15.40:0/2356396261 shutdown_connections 2024-04-26T23:02:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.101+0000 7fee8a800640 1 --2- 172.21.15.40:0/2356396261 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee841678f0 0x7fee840cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.101+0000 7fee8a800640 1 --2- 172.21.15.40:0/2356396261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee840c41d0 0x7fee840cfc30 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.101+0000 7fee8a800640 1 -- 172.21.15.40:0/2356396261 >> 172.21.15.40:0/2356396261 conn(0x7fee840d0be0 msgr2=0x7fee840d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.101+0000 7fee8a800640 1 -- 172.21.15.40:0/2356396261 shutdown_connections 2024-04-26T23:02:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.101+0000 7fee8a800640 1 -- 172.21.15.40:0/2356396261 wait complete. 2024-04-26T23:02:41.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.102+0000 7fee8a800640 1 Processor -- start 2024-04-26T23:02:41.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.103+0000 7fee8a800640 1 -- start start 2024-04-26T23:02:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.103+0000 7fee8a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee84073f30 0x7fee841653f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.103+0000 7fee8a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee840c41d0 0x7fee841fd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee8a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee841678f0 0x7fee840b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee8a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee840d0170 con 0x7fee84073f30 2024-04-26T23:02:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee8a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee840cf430 con 0x7fee841678f0 2024-04-26T23:02:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee8a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fee840ce7b0 con 0x7fee840c41d0 2024-04-26T23:02:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee83400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee84073f30 0x7fee841653f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee83400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee84073f30 0x7fee841653f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32922/0 (socket says 172.21.15.40:32922) 2024-04-26T23:02:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee83400640 1 -- 172.21.15.40:0/1303224580 learned_addr learned my addr 172.21.15.40:0/1303224580 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee7be00640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee840c41d0 0x7fee841fd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:41.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee83400640 1 -- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee840c41d0 msgr2=0x7fee841fd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:41.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee83400640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee840c41d0 0x7fee841fd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee83e00640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee841678f0 0x7fee840b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee83400640 1 -- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee841678f0 msgr2=0x7fee840b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee83400640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee841678f0 0x7fee840b5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.104+0000 7fee83400640 1 -- 172.21.15.40:0/1303224580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee840ca1b0 con 0x7fee84073f30 2024-04-26T23:02:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.105+0000 7fee83400640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee84073f30 0x7fee841653f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fee74009870 tx=0x7fee740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.105+0000 7fee7b400640 1 -- 172.21.15.40:0/1303224580 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee74037040 con 0x7fee84073f30 2024-04-26T23:02:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.105+0000 7fee7b400640 1 -- 172.21.15.40:0/1303224580 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee74004250 con 0x7fee84073f30 2024-04-26T23:02:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.105+0000 7fee8a800640 1 -- 172.21.15.40:0/1303224580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee840ca430 con 0x7fee84073f30 2024-04-26T23:02:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.105+0000 7fee7b400640 1 -- 172.21.15.40:0/1303224580 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7403c730 con 0x7fee84073f30 2024-04-26T23:02:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.105+0000 7fee8a800640 1 -- 172.21.15.40:0/1303224580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee840ca9f0 con 0x7fee84073f30 2024-04-26T23:02:41.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.106+0000 7fee8a800640 1 -- 172.21.15.40:0/1303224580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee840c7540 con 0x7fee84073f30 2024-04-26T23:02:41.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.108+0000 7fee7b400640 1 -- 172.21.15.40:0/1303224580 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fee7403a070 con 0x7fee84073f30 2024-04-26T23:02:41.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.108+0000 7fee7b400640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee6006c8f0 0x7fee6006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:41.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.109+0000 7fee7b400640 1 -- 172.21.15.40:0/1303224580 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fee740ac300 con 0x7fee84073f30 2024-04-26T23:02:41.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.109+0000 7fee7be00640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee6006c8f0 0x7fee6006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:41.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.110+0000 7fee7be00640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee6006c8f0 0x7fee6006edb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fee841678c0 tx=0x7fee6c007b50 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:41.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.113+0000 7fee7b400640 1 -- 172.21.15.40:0/1303224580 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fee840c7540 con 0x7fee84073f30 2024-04-26T23:02:41.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.401+0000 7fee8a800640 1 -- 172.21.15.40:0/1303224580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7fee840694f0 con 0x7fee84073f30 2024-04-26T23:02:41.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.406+0000 7fee7b400640 1 -- 172.21.15.40:0/1303224580 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v9) ==== 90+0+0 (secure 0 0 0) 0x7fee840ca1b0 con 0x7fee84073f30 2024-04-26T23:02:41.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.409+0000 7fee8a800640 1 -- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee6006c8f0 msgr2=0x7fee6006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:41.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.409+0000 7fee8a800640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee6006c8f0 0x7fee6006edb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fee841678c0 tx=0x7fee6c007b50 comp rx=0 tx=0).stop 2024-04-26T23:02:41.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.409+0000 7fee8a800640 1 -- 172.21.15.40:0/1303224580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee84073f30 msgr2=0x7fee841653f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:41.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.409+0000 7fee8a800640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee84073f30 0x7fee841653f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fee74009870 tx=0x7fee740046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.409+0000 7fee8a800640 1 -- 172.21.15.40:0/1303224580 shutdown_connections 2024-04-26T23:02:41.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.409+0000 7fee8a800640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee6006c8f0 0x7fee6006edb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.409+0000 7fee8a800640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee841678f0 0x7fee840b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.409+0000 7fee8a800640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee840c41d0 0x7fee841fd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.409+0000 7fee8a800640 1 --2- 172.21.15.40:0/1303224580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee84073f30 0x7fee841653f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.409+0000 7fee8a800640 1 -- 172.21.15.40:0/1303224580 >> 172.21.15.40:0/1303224580 conn(0x7fee840d0be0 msgr2=0x7fee840d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:41.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.409+0000 7fee8a800640 1 -- 172.21.15.40:0/1303224580 shutdown_connections 2024-04-26T23:02:41.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.409+0000 7fee8a800640 1 -- 172.21.15.40:0/1303224580 wait complete. 2024-04-26T23:02:41.427 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-26T23:02:41.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.638+0000 7f7e46e00640 1 Processor -- start 2024-04-26T23:02:41.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.638+0000 7f7e46e00640 1 -- start start 2024-04-26T23:02:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.640+0000 7f7e46e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7e40145970 0x7f7e401612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.640+0000 7f7e46e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e40146060 0x7f7e400b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.640+0000 7f7e46e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7e400b3fb0 0x7f7e400d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.640+0000 7f7e46e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f7e400ceed0 con 0x7f7e40145970 2024-04-26T23:02:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.640+0000 7f7e46e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7e400b59f0 con 0x7f7e400b3fb0 2024-04-26T23:02:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.640+0000 7f7e46e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7e401617f0 con 0x7f7e40146060 2024-04-26T23:02:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.640+0000 7f7e45a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7e40145970 0x7f7e401612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.640+0000 7f7e45a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7e40145970 0x7f7e401612b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32934/0 (socket says 172.21.15.40:32934) 2024-04-26T23:02:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.640+0000 7f7e45a00640 1 -- 172.21.15.40:0/2280832916 learned_addr learned my addr 172.21.15.40:0/2280832916 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.641+0000 7f7e45a00640 1 -- 172.21.15.40:0/2280832916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e40146060 msgr2=0x7f7e400b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:41.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.641+0000 7f7e45a00640 1 --2- 172.21.15.40:0/2280832916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e40146060 0x7f7e400b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.641+0000 7f7e46400640 1 --2- 172.21.15.40:0/2280832916 >> v2:172.21.15.103:3300/0 conn(0x7f7e400b3fb0 0x7f7e400d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:41.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.641+0000 7f7e45a00640 1 -- 172.21.15.40:0/2280832916 >> v2:172.21.15.103:3300/0 conn(0x7f7e400b3fb0 msgr2=0x7f7e400d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:41.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.641+0000 7f7e45a00640 1 --2- 172.21.15.40:0/2280832916 >> v2:172.21.15.103:3300/0 conn(0x7f7e400b3fb0 0x7f7e400d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.641+0000 7f7e45a00640 1 -- 172.21.15.40:0/2280832916 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e400b5f40 con 0x7f7e40145970 2024-04-26T23:02:41.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.642+0000 7f7e45a00640 1 --2- 172.21.15.40:0/2280832916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e40145970 0x7f7e401612b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7e34009870 tx=0x7f7e3402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6e5a8ee441210b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:41.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.642+0000 7f7e3fe00640 1 -- 172.21.15.40:0/2280832916 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34037070 con 0x7f7e40145970 2024-04-26T23:02:41.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.642+0000 7f7e3fe00640 1 -- 172.21.15.40:0/2280832916 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e34032440 con 0x7f7e40145970 2024-04-26T23:02:41.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.642+0000 7f7e3fe00640 1 -- 172.21.15.40:0/2280832916 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34032810 con 0x7f7e40145970 2024-04-26T23:02:41.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.643+0000 7f7e46e00640 1 -- 172.21.15.40:0/2280832916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e40145970 msgr2=0x7f7e401612b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:41.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.643+0000 7f7e46e00640 1 --2- 172.21.15.40:0/2280832916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e40145970 0x7f7e401612b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7e34009870 tx=0x7f7e3402a210 comp rx=0 tx=0).stop 2024-04-26T23:02:41.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.643+0000 7f7e46e00640 1 -- 172.21.15.40:0/2280832916 shutdown_connections 2024-04-26T23:02:41.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.643+0000 7f7e46e00640 1 --2- 172.21.15.40:0/2280832916 >> v2:172.21.15.103:3300/0 conn(0x7f7e400b3fb0 0x7f7e400d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.643+0000 7f7e46e00640 1 --2- 172.21.15.40:0/2280832916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e40146060 0x7f7e400b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.643+0000 7f7e46e00640 1 --2- 172.21.15.40:0/2280832916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e40145970 0x7f7e401612b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.643+0000 7f7e46e00640 1 -- 172.21.15.40:0/2280832916 >> 172.21.15.40:0/2280832916 conn(0x7f7e40075eb0 msgr2=0x7f7e40068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:41.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.643+0000 7f7e46e00640 1 -- 172.21.15.40:0/2280832916 shutdown_connections 2024-04-26T23:02:41.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.643+0000 7f7e46e00640 1 -- 172.21.15.40:0/2280832916 wait complete. 2024-04-26T23:02:41.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.645+0000 7f7e46e00640 1 Processor -- start 2024-04-26T23:02:41.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.645+0000 7f7e46e00640 1 -- start start 2024-04-26T23:02:41.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.646+0000 7f7e46e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e40145970 0x7f7e40159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:41.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.646+0000 7f7e46e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e40146060 0x7f7e40159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.646+0000 7f7e46e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e4013d610 0x7f7e40157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.646+0000 7f7e46e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e400cf3e0 con 0x7f7e4013d610 2024-04-26T23:02:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.646+0000 7f7e46e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7e400cf100 con 0x7f7e40145970 2024-04-26T23:02:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.646+0000 7f7e46e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7e40145d50 con 0x7f7e40146060 2024-04-26T23:02:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.646+0000 7f7e46400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e4013d610 0x7f7e40157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.647+0000 7f7e46400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e4013d610 0x7f7e40157210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32936/0 (socket says 172.21.15.40:32936) 2024-04-26T23:02:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.647+0000 7f7e46400640 1 -- 172.21.15.40:0/1268148671 learned_addr learned my addr 172.21.15.40:0/1268148671 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:41.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.647+0000 7f7e45a00640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e40145970 0x7f7e40159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:41.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.647+0000 7f7e46400640 1 -- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e40146060 msgr2=0x7f7e40159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:41.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.647+0000 7f7e46400640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e40146060 0x7f7e40159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.647+0000 7f7e46400640 1 -- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e40145970 msgr2=0x7f7e40159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:41.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.647+0000 7f7e46400640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e40145970 0x7f7e40159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.647+0000 7f7e46400640 1 -- 172.21.15.40:0/1268148671 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4015b890 con 0x7f7e4013d610 2024-04-26T23:02:41.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.647+0000 7f7e45a00640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e40145970 0x7f7e40159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:41.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.647+0000 7f7e46400640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e4013d610 0x7f7e40157210 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f7e3800cd20 tx=0x7f7e3800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:41.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.648+0000 7f7e3d600640 1 -- 172.21.15.40:0/1268148671 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e38003b90 con 0x7f7e4013d610 2024-04-26T23:02:41.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.648+0000 7f7e3d600640 1 -- 172.21.15.40:0/1268148671 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e38003d30 con 0x7f7e4013d610 2024-04-26T23:02:41.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.648+0000 7f7e3d600640 1 -- 172.21.15.40:0/1268148671 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e38004240 con 0x7f7e4013d610 2024-04-26T23:02:41.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.648+0000 7f7e46e00640 1 -- 172.21.15.40:0/1268148671 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e4015bb30 con 0x7f7e4013d610 2024-04-26T23:02:41.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.648+0000 7f7e46e00640 1 -- 172.21.15.40:0/1268148671 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e4015c0f0 con 0x7f7e4013d610 2024-04-26T23:02:41.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.649+0000 7f7e46e00640 1 -- 172.21.15.40:0/1268148671 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e400cdec0 con 0x7f7e4013d610 2024-04-26T23:02:41.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.650+0000 7f7e3d600640 1 -- 172.21.15.40:0/1268148671 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7e380043e0 con 0x7f7e4013d610 2024-04-26T23:02:41.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.651+0000 7f7e3d600640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e2006c920 0x7f7e2006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:41.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.651+0000 7f7e3d600640 1 -- 172.21.15.40:0/1268148671 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f7e3808c500 con 0x7f7e4013d610 2024-04-26T23:02:41.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.652+0000 7f7e45a00640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e2006c920 0x7f7e2006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:41.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.653+0000 7f7e45a00640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e2006c920 0x7f7e2006ede0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7e34034000 tx=0x7f7e3402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:41.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.655+0000 7f7e3d600640 1 -- 172.21.15.40:0/1268148671 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7e400cdec0 con 0x7f7e4013d610 2024-04-26T23:02:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.937+0000 7f7e46e00640 1 -- 172.21.15.40:0/1268148671 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f7e401469f0 con 0x7f7e4013d610 2024-04-26T23:02:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.941+0000 7f7e3d600640 1 -- 172.21.15.40:0/1268148671 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f7e401469f0 con 0x7f7e4013d610 2024-04-26T23:02:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.944+0000 7f7e46e00640 1 -- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e2006c920 msgr2=0x7f7e2006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.944+0000 7f7e46e00640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e2006c920 0x7f7e2006ede0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7e34034000 tx=0x7f7e3402e000 comp rx=0 tx=0).stop 2024-04-26T23:02:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.944+0000 7f7e46e00640 1 -- 172.21.15.40:0/1268148671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e4013d610 msgr2=0x7f7e40157210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.944+0000 7f7e46e00640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e4013d610 0x7f7e40157210 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f7e3800cd20 tx=0x7f7e3800a660 comp rx=0 tx=0).stop 2024-04-26T23:02:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.944+0000 7f7e46e00640 1 -- 172.21.15.40:0/1268148671 shutdown_connections 2024-04-26T23:02:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.944+0000 7f7e46e00640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e2006c920 0x7f7e2006ede0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.944+0000 7f7e46e00640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e4013d610 0x7f7e40157210 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.944+0000 7f7e46e00640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e40146060 0x7f7e40159f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.944+0000 7f7e46e00640 1 --2- 172.21.15.40:0/1268148671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e40145970 0x7f7e40159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.944+0000 7f7e46e00640 1 -- 172.21.15.40:0/1268148671 >> 172.21.15.40:0/1268148671 conn(0x7f7e40075eb0 msgr2=0x7f7e40140110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.945+0000 7f7e46e00640 1 -- 172.21.15.40:0/1268148671 shutdown_connections 2024-04-26T23:02:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:41.945+0000 7f7e46e00640 1 -- 172.21.15.40:0/1268148671 wait complete. 2024-04-26T23:02:41.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-26T23:02:42.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.179+0000 7f2508600640 1 Processor -- start 2024-04-26T23:02:42.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.179+0000 7f2508600640 1 -- start start 2024-04-26T23:02:42.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.180+0000 7f2508600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2500073b70 0x7f25000aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.180+0000 7f2508600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f25000cbb50 0x7f25000af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.180+0000 7f2508600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2500072c00 con 0x7f250015f380 2024-04-26T23:02:42.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.180+0000 7f2508600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f25000729b0 con 0x7f25000cbb50 2024-04-26T23:02:42.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.180+0000 7f2508600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2500072ee0 con 0x7f2500073b70 2024-04-26T23:02:42.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.181+0000 7f2505c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f250015f380 0x7f25000cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55004/0 (socket says 172.21.15.40:55004) 2024-04-26T23:02:42.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.181+0000 7f2505c00640 1 -- 172.21.15.40:0/3176236937 learned_addr learned my addr 172.21.15.40:0/3176236937 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:42.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.181+0000 7f2506600640 1 --2- 172.21.15.40:0/3176236937 >> v2:172.21.15.103:3300/0 conn(0x7f25000cbb50 0x7f25000af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.181+0000 7f2505200640 1 --2- 172.21.15.40:0/3176236937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2500073b70 0x7f25000aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3031671820 0 0) 0x7f2500072c00 con 0x7f250015f380 2024-04-26T23:02:42.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24e8003520 con 0x7f250015f380 2024-04-26T23:02:42.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 115380961 0 0) 0x7f24e8003520 con 0x7f250015f380 2024-04-26T23:02:42.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2500073b70 msgr2=0x7f25000aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:42.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 --2- 172.21.15.40:0/3176236937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2500073b70 0x7f25000aecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 >> v2:172.21.15.103:3300/0 conn(0x7f25000cbb50 msgr2=0x7f25000af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:42.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 --2- 172.21.15.40:0/3176236937 >> v2:172.21.15.103:3300/0 conn(0x7f25000cbb50 0x7f25000af1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25000731c0 con 0x7f250015f380 2024-04-26T23:02:42.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f24f4001670 con 0x7f250015f380 2024-04-26T23:02:42.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 >> v1:172.21.15.40:6789/0 conn(0x7f250015f380 legacy=0x7f25000cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:42.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 --2- 172.21.15.40:0/3176236937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24e80039b0 0x7f24e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 --2- 172.21.15.40:0/3176236937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24e8004800 0x7f24e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 --2- 172.21.15.40:0/3176236937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24e80051c0 0x7f24e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25000729b0 con 0x7f24e8004800 2024-04-26T23:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2500072ee0 con 0x7f24e80039b0 2024-04-26T23:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.182+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f24e80045f0 con 0x7f24e80051c0 2024-04-26T23:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.183+0000 7f2505200640 1 --2- 172.21.15.40:0/3176236937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24e8004800 0x7f24e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.183+0000 7f2505c00640 1 --2- 172.21.15.40:0/3176236937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24e80039b0 0x7f24e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.183+0000 7f2505200640 1 -- 172.21.15.40:0/3176236937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24e80051c0 msgr2=0x7f24e800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.183+0000 7f2505200640 1 --2- 172.21.15.40:0/3176236937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24e80051c0 0x7f24e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.183+0000 7f2505200640 1 -- 172.21.15.40:0/3176236937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24e80039b0 msgr2=0x7f24e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.183+0000 7f2505200640 1 --2- 172.21.15.40:0/3176236937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24e80039b0 0x7f24e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.183+0000 7f2505200640 1 -- 172.21.15.40:0/3176236937 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e8003520 con 0x7f24e8004800 2024-04-26T23:02:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.184+0000 7f2505200640 1 --2- 172.21.15.40:0/3176236937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24e8004800 0x7f24e8004c80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f24ec009af0 tx=0x7f24e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15cd4e0adc488832 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.184+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec036070 con 0x7f24e8004800 2024-04-26T23:02:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.184+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24ec031400 con 0x7f24e8004800 2024-04-26T23:02:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.184+0000 7f24ffe00640 1 -- 172.21.15.40:0/3176236937 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec009690 con 0x7f24e8004800 2024-04-26T23:02:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.185+0000 7f2508600640 1 -- 172.21.15.40:0/3176236937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24e8004800 msgr2=0x7f24e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.185+0000 7f2508600640 1 --2- 172.21.15.40:0/3176236937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24e8004800 0x7f24e8004c80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f24ec009af0 tx=0x7f24e8005830 comp rx=0 tx=0).stop 2024-04-26T23:02:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.185+0000 7f2506600640 1 -- 172.21.15.40:0/3176236937 reap_dead start 2024-04-26T23:02:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.185+0000 7f2508600640 1 -- 172.21.15.40:0/3176236937 shutdown_connections 2024-04-26T23:02:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.185+0000 7f2508600640 1 -- 172.21.15.40:0/3176236937 >> 172.21.15.40:0/3176236937 conn(0x7f25000d0be0 msgr2=0x7f25000d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.185+0000 7f2508600640 1 -- 172.21.15.40:0/3176236937 shutdown_connections 2024-04-26T23:02:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.185+0000 7f2508600640 1 -- 172.21.15.40:0/3176236937 wait complete. 2024-04-26T23:02:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.187+0000 7f2508600640 1 Processor -- start 2024-04-26T23:02:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.187+0000 7f2508600640 1 -- start start 2024-04-26T23:02:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.188+0000 7f2508600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24e8004800 0x7f25001f9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.188+0000 7f2508600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250015aaa0 0x7f25000c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.188+0000 7f2508600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25000c81f0 0x7f25000c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.188+0000 7f2508600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2500073620 con 0x7f250015aaa0 2024-04-26T23:02:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.188+0000 7f2508600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f25000d0830 con 0x7f24e8004800 2024-04-26T23:02:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.188+0000 7f2508600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f25000737a0 con 0x7f25000c81f0 2024-04-26T23:02:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.189+0000 7f2505200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250015aaa0 0x7f25000c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.189+0000 7f2505200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250015aaa0 0x7f25000c7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32946/0 (socket says 172.21.15.40:32946) 2024-04-26T23:02:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.189+0000 7f2505200640 1 -- 172.21.15.40:0/225073215 learned_addr learned my addr 172.21.15.40:0/225073215 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.189+0000 7f2505c00640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24e8004800 0x7f25001f9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.189+0000 7f2505200640 1 -- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25000c81f0 msgr2=0x7f25000c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.189+0000 7f2506600640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25000c81f0 0x7f25000c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.189+0000 7f2505200640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25000c81f0 0x7f25000c9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.189+0000 7f2505200640 1 -- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24e8004800 msgr2=0x7f25001f9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.189+0000 7f2505200640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24e8004800 0x7f25001f9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.189+0000 7f2505200640 1 -- 172.21.15.40:0/225073215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25000c4840 con 0x7f250015aaa0 2024-04-26T23:02:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.189+0000 7f2505c00640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24e8004800 0x7f25001f9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.190+0000 7f2505200640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250015aaa0 0x7f25000c7cb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f24ec034040 tx=0x7f24ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.190+0000 7f24fd600640 1 -- 172.21.15.40:0/225073215 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec041020 con 0x7f250015aaa0 2024-04-26T23:02:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.190+0000 7f24fd600640 1 -- 172.21.15.40:0/225073215 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24ec02ade0 con 0x7f250015aaa0 2024-04-26T23:02:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.190+0000 7f24fd600640 1 -- 172.21.15.40:0/225073215 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec036040 con 0x7f250015aaa0 2024-04-26T23:02:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.190+0000 7f2508600640 1 -- 172.21.15.40:0/225073215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25000c9e40 con 0x7f250015aaa0 2024-04-26T23:02:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.190+0000 7f2508600640 1 -- 172.21.15.40:0/225073215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25000c4dc0 con 0x7f250015aaa0 2024-04-26T23:02:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.191+0000 7f2508600640 1 -- 172.21.15.40:0/225073215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25000c8600 con 0x7f250015aaa0 2024-04-26T23:02:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.192+0000 7f24fd600640 1 -- 172.21.15.40:0/225073215 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f24ec044020 con 0x7f250015aaa0 2024-04-26T23:02:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.193+0000 7f24fd600640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f24e006c8f0 0x7f24e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.193+0000 7f24fd600640 1 -- 172.21.15.40:0/225073215 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f24ec0abef0 con 0x7f250015aaa0 2024-04-26T23:02:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.194+0000 7f2505c00640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f24e006c8f0 0x7f24e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.195+0000 7f2505c00640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f24e006c8f0 0x7f24e006edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f24f4000fb0 tx=0x7f24f4001e70 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:42.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.198+0000 7f24fd600640 1 -- 172.21.15.40:0/225073215 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f25000c8600 con 0x7f250015aaa0 2024-04-26T23:02:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.471+0000 7f2508600640 1 -- 172.21.15.40:0/225073215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f25000ccb70 con 0x7f250015aaa0 2024-04-26T23:02:42.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.476+0000 7f24fd600640 1 -- 172.21.15.40:0/225073215 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v11) ==== 90+0+0 (secure 0 0 0) 0x7f24ec04b030 con 0x7f250015aaa0 2024-04-26T23:02:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.478+0000 7f2508600640 1 -- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f24e006c8f0 msgr2=0x7f24e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.478+0000 7f2508600640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f24e006c8f0 0x7f24e006edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f24f4000fb0 tx=0x7f24f4001e70 comp rx=0 tx=0).stop 2024-04-26T23:02:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.478+0000 7f2508600640 1 -- 172.21.15.40:0/225073215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250015aaa0 msgr2=0x7f25000c7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.478+0000 7f2508600640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250015aaa0 0x7f25000c7cb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f24ec034040 tx=0x7f24ec004730 comp rx=0 tx=0).stop 2024-04-26T23:02:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.479+0000 7f2508600640 1 -- 172.21.15.40:0/225073215 shutdown_connections 2024-04-26T23:02:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.479+0000 7f2508600640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f24e006c8f0 0x7f24e006edb0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.479+0000 7f2508600640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f25000c81f0 0x7f25000c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.479+0000 7f2508600640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250015aaa0 0x7f25000c7cb0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.479+0000 7f2508600640 1 --2- 172.21.15.40:0/225073215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24e8004800 0x7f25001f9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.479+0000 7f2508600640 1 -- 172.21.15.40:0/225073215 >> 172.21.15.40:0/225073215 conn(0x7f25000d0be0 msgr2=0x7f25000b1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.479+0000 7f2508600640 1 -- 172.21.15.40:0/225073215 shutdown_connections 2024-04-26T23:02:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.479+0000 7f2508600640 1 -- 172.21.15.40:0/225073215 wait complete. 2024-04-26T23:02:42.496 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-26T23:02:42.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.700+0000 7f1edfe00640 1 Processor -- start 2024-04-26T23:02:42.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.700+0000 7f1edfe00640 1 -- start start 2024-04-26T23:02:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.702+0000 7f1edfe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ee0155f20 0x7f1ee00d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.702+0000 7f1edfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1ee015f490 0x7f1ee006e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.702+0000 7f1edfe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ee015fca0 con 0x7f1ee006eaa0 2024-04-26T23:02:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.702+0000 7f1edfe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1ee015ff80 con 0x7f1ee015f490 2024-04-26T23:02:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.702+0000 7f1edfe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1ee0160260 con 0x7f1ee0155f20 2024-04-26T23:02:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.703+0000 7f1ede000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1ee015f490 0x7f1ee006e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.703+0000 7f1edf400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1ee006eaa0 0x7f1ee00b5500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55020/0 (socket says 172.21.15.40:55020) 2024-04-26T23:02:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.703+0000 7f1edf400640 1 -- 172.21.15.40:0/3571127848 learned_addr learned my addr 172.21.15.40:0/3571127848 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:42.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.703+0000 7f1edea00640 1 --2- 172.21.15.40:0/3571127848 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ee0155f20 0x7f1ee00d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.703+0000 7f1ede000640 1 -- 172.21.15.40:0/3571127848 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ee0155f20 msgr2=0x7f1ee00d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:42.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.703+0000 7f1ede000640 1 --2- 172.21.15.40:0/3571127848 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ee0155f20 0x7f1ee00d11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.703+0000 7f1ede000640 1 -- 172.21.15.40:0/3571127848 >> v1:172.21.15.40:6789/0 conn(0x7f1ee006eaa0 legacy=0x7f1ee00b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:42.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.703+0000 7f1ede000640 1 -- 172.21.15.40:0/3571127848 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee006e040 con 0x7f1ee015f490 2024-04-26T23:02:42.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.703+0000 7f1edd600640 1 -- 172.21.15.40:0/3571127848 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4247250435 0 0) 0x7f1ee015fca0 con 0x7f1ee006eaa0 2024-04-26T23:02:42.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.703+0000 7f1edea00640 1 --2- 172.21.15.40:0/3571127848 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ee0155f20 0x7f1ee00d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:42.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.704+0000 7f1ede000640 1 --2- 172.21.15.40:0/3571127848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee015f490 0x7f1ee006e560 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0097a0 tx=0x7f1ecc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=279129d3b85dfd85 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:42.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.704+0000 7f1edd600640 1 -- 172.21.15.40:0/3571127848 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc037070 con 0x7f1ee015f490 2024-04-26T23:02:42.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.704+0000 7f1edd600640 1 -- 172.21.15.40:0/3571127848 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ecc032440 con 0x7f1ee015f490 2024-04-26T23:02:42.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.704+0000 7f1edd600640 1 -- 172.21.15.40:0/3571127848 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc032810 con 0x7f1ee015f490 2024-04-26T23:02:42.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.705+0000 7f1edfe00640 1 -- 172.21.15.40:0/3571127848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee015f490 msgr2=0x7f1ee006e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:42.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.705+0000 7f1edfe00640 1 --2- 172.21.15.40:0/3571127848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee015f490 0x7f1ee006e560 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc0097a0 tx=0x7f1ecc02a180 comp rx=0 tx=0).stop 2024-04-26T23:02:42.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.705+0000 7f1edfe00640 1 -- 172.21.15.40:0/3571127848 shutdown_connections 2024-04-26T23:02:42.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.705+0000 7f1edfe00640 1 --2- 172.21.15.40:0/3571127848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee015f490 0x7f1ee006e560 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.705+0000 7f1edfe00640 1 --2- 172.21.15.40:0/3571127848 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ee0155f20 0x7f1ee00d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.705+0000 7f1edfe00640 1 -- 172.21.15.40:0/3571127848 >> 172.21.15.40:0/3571127848 conn(0x7f1ee0075eb0 msgr2=0x7f1ee0068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:42.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.705+0000 7f1edfe00640 1 -- 172.21.15.40:0/3571127848 shutdown_connections 2024-04-26T23:02:42.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.705+0000 7f1edfe00640 1 -- 172.21.15.40:0/3571127848 wait complete. 2024-04-26T23:02:42.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.708+0000 7f1edfe00640 1 Processor -- start 2024-04-26T23:02:42.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.708+0000 7f1edfe00640 1 -- start start 2024-04-26T23:02:42.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.709+0000 7f1edfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ee006eaa0 0x7f1ee00b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.709+0000 7f1edfe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ee0155f20 0x7f1ee00b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.709+0000 7f1edfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee015f490 0x7f1ee00b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.709+0000 7f1edfe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee015fa50 con 0x7f1ee015f490 2024-04-26T23:02:42.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.709+0000 7f1edfe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee015f8d0 con 0x7f1ee006eaa0 2024-04-26T23:02:42.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.709+0000 7f1edfe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1ee00b5c10 con 0x7f1ee0155f20 2024-04-26T23:02:42.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.709+0000 7f1edf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee015f490 0x7f1ee00b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.709+0000 7f1edf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee015f490 0x7f1ee00b0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32960/0 (socket says 172.21.15.40:32960) 2024-04-26T23:02:42.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.709+0000 7f1edf400640 1 -- 172.21.15.40:0/1833126000 learned_addr learned my addr 172.21.15.40:0/1833126000 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:42.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.710+0000 7f1edea00640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ee006eaa0 0x7f1ee00b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.710+0000 7f1ede000640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ee0155f20 0x7f1ee00b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.710+0000 7f1edf400640 1 -- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ee0155f20 msgr2=0x7f1ee00b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:42.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.710+0000 7f1edf400640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ee0155f20 0x7f1ee00b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.710+0000 7f1edf400640 1 -- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ee006eaa0 msgr2=0x7f1ee00b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:42.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.710+0000 7f1edf400640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ee006eaa0 0x7f1ee00b26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:42.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.710+0000 7f1edf400640 1 -- 172.21.15.40:0/1833126000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee00b1600 con 0x7f1ee015f490 2024-04-26T23:02:42.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.710+0000 7f1edf400640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee015f490 0x7f1ee00b0bd0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800c7a0 tx=0x7f1ed800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:42.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.710+0000 7f1ed6a00640 1 -- 172.21.15.40:0/1833126000 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8018070 con 0x7f1ee015f490 2024-04-26T23:02:42.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.711+0000 7f1ed6a00640 1 -- 172.21.15.40:0/1833126000 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed800a640 con 0x7f1ee015f490 2024-04-26T23:02:42.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.711+0000 7f1ed6a00640 1 -- 172.21.15.40:0/1833126000 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed800aab0 con 0x7f1ee015f490 2024-04-26T23:02:42.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.711+0000 7f1edfe00640 1 -- 172.21.15.40:0/1833126000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ee00b18e0 con 0x7f1ee015f490 2024-04-26T23:02:42.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.711+0000 7f1edfe00640 1 -- 172.21.15.40:0/1833126000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ee00b1ea0 con 0x7f1ee015f490 2024-04-26T23:02:42.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.712+0000 7f1edfe00640 1 -- 172.21.15.40:0/1833126000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ee0069370 con 0x7f1ee015f490 2024-04-26T23:02:42.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.713+0000 7f1ed6a00640 1 -- 172.21.15.40:0/1833126000 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1ed801b070 con 0x7f1ee015f490 2024-04-26T23:02:42.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.714+0000 7f1ed6a00640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ec006c8f0 0x7f1ec006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:42.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.714+0000 7f1ed6a00640 1 -- 172.21.15.40:0/1833126000 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f1ed808af80 con 0x7f1ee015f490 2024-04-26T23:02:42.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.715+0000 7f1edea00640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ec006c8f0 0x7f1ec006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:42.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.716+0000 7f1edea00640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ec006c8f0 0x7f1ec006edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000a990 tx=0x7f1ed0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:42.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.718+0000 7f1ed6a00640 1 -- 172.21.15.40:0/1833126000 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1ed8059570 con 0x7f1ee015f490 2024-04-26T23:02:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.994+0000 7f1edfe00640 1 -- 172.21.15.40:0/1833126000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f1ee015fd30 con 0x7f1ee015f490 2024-04-26T23:02:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:42.998+0000 7f1ed6a00640 1 -- 172.21.15.40:0/1833126000 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v12) ==== 90+0+0 (secure 0 0 0) 0x7f1ed8058e70 con 0x7f1ee015f490 2024-04-26T23:02:43.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.001+0000 7f1edfe00640 1 -- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ec006c8f0 msgr2=0x7f1ec006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:43.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.001+0000 7f1edfe00640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ec006c8f0 0x7f1ec006edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000a990 tx=0x7f1ed0008040 comp rx=0 tx=0).stop 2024-04-26T23:02:43.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.001+0000 7f1edfe00640 1 -- 172.21.15.40:0/1833126000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee015f490 msgr2=0x7f1ee00b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:43.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.001+0000 7f1edfe00640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee015f490 0x7f1ee00b0bd0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800c7a0 tx=0x7f1ed800cc60 comp rx=0 tx=0).stop 2024-04-26T23:02:43.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.001+0000 7f1edfe00640 1 -- 172.21.15.40:0/1833126000 shutdown_connections 2024-04-26T23:02:43.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.001+0000 7f1edfe00640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ec006c8f0 0x7f1ec006edb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.001+0000 7f1edfe00640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ee015f490 0x7f1ee00b0bd0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.001+0000 7f1edfe00640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ee0155f20 0x7f1ee00b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.001+0000 7f1edfe00640 1 --2- 172.21.15.40:0/1833126000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ee006eaa0 0x7f1ee00b26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.001+0000 7f1edfe00640 1 -- 172.21.15.40:0/1833126000 >> 172.21.15.40:0/1833126000 conn(0x7f1ee0075eb0 msgr2=0x7f1ee00d0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:43.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.001+0000 7f1edfe00640 1 -- 172.21.15.40:0/1833126000 shutdown_connections 2024-04-26T23:02:43.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.001+0000 7f1edfe00640 1 -- 172.21.15.40:0/1833126000 wait complete. 2024-04-26T23:02:43.019 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-26T23:02:43.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.213+0000 7f6d1fe00640 1 Processor -- start 2024-04-26T23:02:43.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.213+0000 7f6d1fe00640 1 -- start start 2024-04-26T23:02:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.215+0000 7f6d1fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d2015c160 0x7f6d200b4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.215+0000 7f6d1fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6d2015c5c0 0x7f6d200b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.215+0000 7f6d1fe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d2015cb90 con 0x7f6d20073e10 2024-04-26T23:02:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.215+0000 7f6d1fe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f6d20153040 con 0x7f6d2015c5c0 2024-04-26T23:02:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.215+0000 7f6d1fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6d2015fc40 con 0x7f6d2015c160 2024-04-26T23:02:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.215+0000 7f6d1f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6d2015c5c0 0x7f6d200b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.215+0000 7f6d1f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6d2015c5c0 0x7f6d200b4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32966/0 (socket says 172.21.15.40:32966) 2024-04-26T23:02:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.215+0000 7f6d1f400640 1 -- 172.21.15.40:0/1970406612 learned_addr learned my addr 172.21.15.40:0/1970406612 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.216+0000 7f6d1f400640 1 -- 172.21.15.40:0/1970406612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d2015c160 msgr2=0x7f6d200b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.216+0000 7f6d1f400640 1 --2- 172.21.15.40:0/1970406612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d2015c160 0x7f6d200b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.216+0000 7f6d1f400640 1 -- 172.21.15.40:0/1970406612 >> v1:172.21.15.103:6789/0 conn(0x7f6d20073e10 legacy=0x7f6d201600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.216+0000 7f6d1f400640 1 -- 172.21.15.40:0/1970406612 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2015ce70 con 0x7f6d2015c5c0 2024-04-26T23:02:43.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.217+0000 7f6d1f400640 1 --2- 172.21.15.40:0/1970406612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d2015c5c0 0x7f6d200b4d80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009870 tx=0x7f6d1802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a50576077fb27c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:43.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.217+0000 7f6d1e000640 1 -- 172.21.15.40:0/1970406612 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d18037070 con 0x7f6d2015c5c0 2024-04-26T23:02:43.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.217+0000 7f6d1e000640 1 -- 172.21.15.40:0/1970406612 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d18032440 con 0x7f6d2015c5c0 2024-04-26T23:02:43.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.217+0000 7f6d1e000640 1 -- 172.21.15.40:0/1970406612 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d18032810 con 0x7f6d2015c5c0 2024-04-26T23:02:43.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.218+0000 7f6d1fe00640 1 -- 172.21.15.40:0/1970406612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d2015c5c0 msgr2=0x7f6d200b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:43.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.218+0000 7f6d1fe00640 1 --2- 172.21.15.40:0/1970406612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d2015c5c0 0x7f6d200b4d80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009870 tx=0x7f6d1802a210 comp rx=0 tx=0).stop 2024-04-26T23:02:43.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.218+0000 7f6d1fe00640 1 -- 172.21.15.40:0/1970406612 shutdown_connections 2024-04-26T23:02:43.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.218+0000 7f6d1fe00640 1 --2- 172.21.15.40:0/1970406612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d2015c160 0x7f6d200b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.218+0000 7f6d1fe00640 1 --2- 172.21.15.40:0/1970406612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d2015c5c0 0x7f6d200b4d80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.218+0000 7f6d1fe00640 1 -- 172.21.15.40:0/1970406612 >> 172.21.15.40:0/1970406612 conn(0x7f6d20075eb0 msgr2=0x7f6d20068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:43.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.218+0000 7f6d1fe00640 1 -- 172.21.15.40:0/1970406612 shutdown_connections 2024-04-26T23:02:43.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.218+0000 7f6d1fe00640 1 -- 172.21.15.40:0/1970406612 wait complete. 2024-04-26T23:02:43.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.220+0000 7f6d1fe00640 1 Processor -- start 2024-04-26T23:02:43.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.220+0000 7f6d1fe00640 1 -- start start 2024-04-26T23:02:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.221+0000 7f6d1fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d20073e10 0x7f6d2014a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.221+0000 7f6d1fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d2015c160 0x7f6d201531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.221+0000 7f6d1fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d2015c5c0 0x7f6d20144e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.221+0000 7f6d1fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d20073540 con 0x7f6d2015c5c0 2024-04-26T23:02:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.221+0000 7f6d1fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6d200733c0 con 0x7f6d2015c160 2024-04-26T23:02:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.221+0000 7f6d1fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6d2015d150 con 0x7f6d20073e10 2024-04-26T23:02:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.221+0000 7f6d1f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d2015c5c0 0x7f6d20144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:43.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.221+0000 7f6d1f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d2015c5c0 0x7f6d20144e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32980/0 (socket says 172.21.15.40:32980) 2024-04-26T23:02:43.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.221+0000 7f6d1f400640 1 -- 172.21.15.40:0/3855099686 learned_addr learned my addr 172.21.15.40:0/3855099686 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.221+0000 7f6d1ea00640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d20073e10 0x7f6d2014a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.222+0000 7f6d1f400640 1 -- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d20073e10 msgr2=0x7f6d2014a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.222+0000 7f6d16000640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d2015c160 0x7f6d201531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.222+0000 7f6d1f400640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d20073e10 0x7f6d2014a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.222+0000 7f6d1f400640 1 -- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d2015c160 msgr2=0x7f6d201531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.222+0000 7f6d1f400640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d2015c160 0x7f6d201531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.222+0000 7f6d1f400640 1 -- 172.21.15.40:0/3855099686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d20153910 con 0x7f6d2015c5c0 2024-04-26T23:02:43.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.222+0000 7f6d1ea00640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d20073e10 0x7f6d2014a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:43.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.222+0000 7f6d16000640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d2015c160 0x7f6d201531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:43.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.222+0000 7f6d1f400640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d2015c5c0 0x7f6d20144e10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009870 tx=0x7f6d180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:43.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.222+0000 7f6d17e00640 1 -- 172.21.15.40:0/3855099686 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d18041070 con 0x7f6d2015c5c0 2024-04-26T23:02:43.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.222+0000 7f6d17e00640 1 -- 172.21.15.40:0/3855099686 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d1803c950 con 0x7f6d2015c5c0 2024-04-26T23:02:43.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.222+0000 7f6d17e00640 1 -- 172.21.15.40:0/3855099686 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d18037040 con 0x7f6d2015c5c0 2024-04-26T23:02:43.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.222+0000 7f6d1fe00640 1 -- 172.21.15.40:0/3855099686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d20153bf0 con 0x7f6d2015c5c0 2024-04-26T23:02:43.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.223+0000 7f6d1fe00640 1 -- 172.21.15.40:0/3855099686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d201541b0 con 0x7f6d2015c5c0 2024-04-26T23:02:43.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.224+0000 7f6d1fe00640 1 -- 172.21.15.40:0/3855099686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d20069370 con 0x7f6d2015c5c0 2024-04-26T23:02:43.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.225+0000 7f6d17e00640 1 -- 172.21.15.40:0/3855099686 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6d18044070 con 0x7f6d2015c5c0 2024-04-26T23:02:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.226+0000 7f6d17e00640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6cfc06c8f0 0x7f6cfc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.226+0000 7f6d17e00640 1 -- 172.21.15.40:0/3855099686 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f6d180ac470 con 0x7f6d2015c5c0 2024-04-26T23:02:43.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.227+0000 7f6d1ea00640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6cfc06c8f0 0x7f6cfc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:43.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.228+0000 7f6d1ea00640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6cfc06c8f0 0x7f6cfc06edb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6d2014af70 tx=0x7f6d10009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:43.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.232+0000 7f6d17e00640 1 -- 172.21.15.40:0/3855099686 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f6d1807aa50 con 0x7f6d2015c5c0 2024-04-26T23:02:43.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.512+0000 7f6d1fe00640 1 -- 172.21.15.40:0/3855099686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f6d2006f4e0 con 0x7f6d2015c5c0 2024-04-26T23:02:43.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.517+0000 7f6d17e00640 1 -- 172.21.15.40:0/3855099686 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v13) ==== 97+0+0 (secure 0 0 0) 0x7f6d1807a350 con 0x7f6d2015c5c0 2024-04-26T23:02:43.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.520+0000 7f6d1fe00640 1 -- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6cfc06c8f0 msgr2=0x7f6cfc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:43.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.520+0000 7f6d1fe00640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6cfc06c8f0 0x7f6cfc06edb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6d2014af70 tx=0x7f6d10009040 comp rx=0 tx=0).stop 2024-04-26T23:02:43.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.520+0000 7f6d1fe00640 1 -- 172.21.15.40:0/3855099686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d2015c5c0 msgr2=0x7f6d20144e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:43.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.520+0000 7f6d1fe00640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d2015c5c0 0x7f6d20144e10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009870 tx=0x7f6d180046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.520+0000 7f6d1fe00640 1 -- 172.21.15.40:0/3855099686 shutdown_connections 2024-04-26T23:02:43.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.520+0000 7f6d1fe00640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6cfc06c8f0 0x7f6cfc06edb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.520+0000 7f6d1fe00640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d2015c5c0 0x7f6d20144e10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.520+0000 7f6d1fe00640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d2015c160 0x7f6d201531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.520+0000 7f6d1fe00640 1 --2- 172.21.15.40:0/3855099686 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d20073e10 0x7f6d2014a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.520+0000 7f6d1fe00640 1 -- 172.21.15.40:0/3855099686 >> 172.21.15.40:0/3855099686 conn(0x7f6d20075eb0 msgr2=0x7f6d2015b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:43.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.520+0000 7f6d1fe00640 1 -- 172.21.15.40:0/3855099686 shutdown_connections 2024-04-26T23:02:43.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.520+0000 7f6d1fe00640 1 -- 172.21.15.40:0/3855099686 wait complete. 2024-04-26T23:02:43.540 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-26T23:02:43.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.739+0000 7f5c3b200640 1 Processor -- start 2024-04-26T23:02:43.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.740+0000 7f5c3b200640 1 -- start start 2024-04-26T23:02:43.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.741+0000 7f5c3b200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5c3415c160 0x7f5c340b4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:43.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.741+0000 7f5c3b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c3415c5c0 0x7f5c340b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:43.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.741+0000 7f5c3b200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3415cb90 con 0x7f5c34073e10 2024-04-26T23:02:43.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.741+0000 7f5c3b200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f5c34153040 con 0x7f5c3415c160 2024-04-26T23:02:43.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.741+0000 7f5c3b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5c3415fc40 con 0x7f5c3415c5c0 2024-04-26T23:02:43.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.742+0000 7f5c39e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5c34073e10 0x7f5c341600d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55032/0 (socket says 172.21.15.40:55032) 2024-04-26T23:02:43.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.742+0000 7f5c39e00640 1 -- 172.21.15.40:0/255583532 learned_addr learned my addr 172.21.15.40:0/255583532 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:43.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.742+0000 7f5c39400640 1 --2- 172.21.15.40:0/255583532 >> v2:172.21.15.40:3300/0 conn(0x7f5c3415c160 0x7f5c340b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.742+0000 7f5c3a800640 1 --2- 172.21.15.40:0/255583532 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c3415c5c0 0x7f5c340b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.742+0000 7f5c39400640 1 -- 172.21.15.40:0/255583532 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c3415c5c0 msgr2=0x7f5c340b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.742+0000 7f5c39400640 1 --2- 172.21.15.40:0/255583532 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c3415c5c0 0x7f5c340b4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.742+0000 7f5c39400640 1 -- 172.21.15.40:0/255583532 >> v1:172.21.15.40:6789/0 conn(0x7f5c34073e10 legacy=0x7f5c341600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.742+0000 7f5c39400640 1 -- 172.21.15.40:0/255583532 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3415ce70 con 0x7f5c3415c160 2024-04-26T23:02:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.742+0000 7f5c38a00640 1 -- 172.21.15.40:0/255583532 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4155649871 0 0) 0x7f5c3415cb90 con 0x7f5c34073e10 2024-04-26T23:02:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.743+0000 7f5c39400640 1 --2- 172.21.15.40:0/255583532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3415c160 0x7f5c340b4840 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5c240097a0 tx=0x7f5c2402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=650e172f6bf1b222 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.744+0000 7f5c38a00640 1 -- 172.21.15.40:0/255583532 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24037070 con 0x7f5c3415c160 2024-04-26T23:02:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.744+0000 7f5c38a00640 1 -- 172.21.15.40:0/255583532 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c24032440 con 0x7f5c3415c160 2024-04-26T23:02:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.744+0000 7f5c38a00640 1 -- 172.21.15.40:0/255583532 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24032810 con 0x7f5c3415c160 2024-04-26T23:02:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.744+0000 7f5c3b200640 1 -- 172.21.15.40:0/255583532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3415c160 msgr2=0x7f5c340b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.744+0000 7f5c3b200640 1 --2- 172.21.15.40:0/255583532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3415c160 0x7f5c340b4840 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5c240097a0 tx=0x7f5c2402a180 comp rx=0 tx=0).stop 2024-04-26T23:02:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.744+0000 7f5c3b200640 1 -- 172.21.15.40:0/255583532 shutdown_connections 2024-04-26T23:02:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.744+0000 7f5c3b200640 1 --2- 172.21.15.40:0/255583532 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c3415c5c0 0x7f5c340b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.744+0000 7f5c3b200640 1 --2- 172.21.15.40:0/255583532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3415c160 0x7f5c340b4840 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.744+0000 7f5c3b200640 1 -- 172.21.15.40:0/255583532 >> 172.21.15.40:0/255583532 conn(0x7f5c34075eb0 msgr2=0x7f5c34068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.745+0000 7f5c3b200640 1 -- 172.21.15.40:0/255583532 shutdown_connections 2024-04-26T23:02:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.745+0000 7f5c3b200640 1 -- 172.21.15.40:0/255583532 wait complete. 2024-04-26T23:02:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.746+0000 7f5c3b200640 1 Processor -- start 2024-04-26T23:02:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.747+0000 7f5c3b200640 1 -- start start 2024-04-26T23:02:43.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.747+0000 7f5c3b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c34073e10 0x7f5c340ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:43.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.747+0000 7f5c3b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3415c160 0x7f5c340cf720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:43.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.747+0000 7f5c3b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c3415c5c0 0x7f5c34146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:43.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.747+0000 7f5c3b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5c34073540 con 0x7f5c3415c160 2024-04-26T23:02:43.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.747+0000 7f5c3b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5c340733c0 con 0x7f5c3415c5c0 2024-04-26T23:02:43.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.747+0000 7f5c3b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5c3415d150 con 0x7f5c34073e10 2024-04-26T23:02:43.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.748+0000 7f5c39400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3415c160 0x7f5c340cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:43.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.748+0000 7f5c39400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3415c160 0x7f5c340cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32998/0 (socket says 172.21.15.40:32998) 2024-04-26T23:02:43.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.748+0000 7f5c39400640 1 -- 172.21.15.40:0/3721473296 learned_addr learned my addr 172.21.15.40:0/3721473296 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:43.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.748+0000 7f5c39400640 1 -- 172.21.15.40:0/3721473296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c34073e10 msgr2=0x7f5c340ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:43.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.748+0000 7f5c39400640 1 --2- 172.21.15.40:0/3721473296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c34073e10 0x7f5c340ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.748+0000 7f5c39400640 1 -- 172.21.15.40:0/3721473296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c3415c5c0 msgr2=0x7f5c34146ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:43.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.748+0000 7f5c3a800640 1 --2- 172.21.15.40:0/3721473296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c3415c5c0 0x7f5c34146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:43.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.748+0000 7f5c39400640 1 --2- 172.21.15.40:0/3721473296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c3415c5c0 0x7f5c34146ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:43.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.748+0000 7f5c39400640 1 -- 172.21.15.40:0/3721473296 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c340cb390 con 0x7f5c3415c160 2024-04-26T23:02:43.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.749+0000 7f5c39400640 1 --2- 172.21.15.40:0/3721473296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3415c160 0x7f5c340cf720 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5c24035040 tx=0x7f5c240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:43.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.749+0000 7f5c2e000640 1 -- 172.21.15.40:0/3721473296 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24037040 con 0x7f5c3415c160 2024-04-26T23:02:43.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.749+0000 7f5c2e000640 1 -- 172.21.15.40:0/3721473296 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c24030770 con 0x7f5c3415c160 2024-04-26T23:02:43.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.749+0000 7f5c2e000640 1 -- 172.21.15.40:0/3721473296 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24030b80 con 0x7f5c3415c160 2024-04-26T23:02:43.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.749+0000 7f5c3b200640 1 -- 172.21.15.40:0/3721473296 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c340cb670 con 0x7f5c3415c160 2024-04-26T23:02:43.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.749+0000 7f5c3b200640 1 -- 172.21.15.40:0/3721473296 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c340cbf10 con 0x7f5c3415c160 2024-04-26T23:02:43.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.750+0000 7f5c3b200640 1 -- 172.21.15.40:0/3721473296 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c34152e10 con 0x7f5c3415c160 2024-04-26T23:02:43.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.751+0000 7f5c2e000640 1 -- 172.21.15.40:0/3721473296 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5c24044020 con 0x7f5c3415c160 2024-04-26T23:02:43.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.752+0000 7f5c2e000640 1 --2- 172.21.15.40:0/3721473296 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5c1806c8f0 0x7f5c1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:43.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.752+0000 7f5c2e000640 1 -- 172.21.15.40:0/3721473296 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f5c240abee0 con 0x7f5c3415c160 2024-04-26T23:02:43.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.753+0000 7f5c39e00640 1 --2- 172.21.15.40:0/3721473296 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5c1806c8f0 0x7f5c1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:43.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.754+0000 7f5c39e00640 1 --2- 172.21.15.40:0/3721473296 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5c1806c8f0 0x7f5c1806edb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5c28007fd0 tx=0x7f5c28007920 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:43.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:43.756+0000 7f5c2e000640 1 -- 172.21.15.40:0/3721473296 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5c34152e10 con 0x7f5c3415c160 2024-04-26T23:02:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.024+0000 7f5c3b200640 1 -- 172.21.15.40:0/3721473296 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f5c340cb840 con 0x7f5c3415c160 2024-04-26T23:02:44.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.029+0000 7f5c2e000640 1 -- 172.21.15.40:0/3721473296 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v14) ==== 90+0+0 (secure 0 0 0) 0x7f5c24032e20 con 0x7f5c3415c160 2024-04-26T23:02:44.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.032+0000 7f5c3b200640 1 -- 172.21.15.40:0/3721473296 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5c1806c8f0 msgr2=0x7f5c1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.032+0000 7f5c3b200640 1 --2- 172.21.15.40:0/3721473296 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5c1806c8f0 0x7f5c1806edb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5c28007fd0 tx=0x7f5c28007920 comp rx=0 tx=0).stop 2024-04-26T23:02:44.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.032+0000 7f5c3b200640 1 -- 172.21.15.40:0/3721473296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3415c160 msgr2=0x7f5c340cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.032+0000 7f5c3b200640 1 --2- 172.21.15.40:0/3721473296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3415c160 0x7f5c340cf720 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5c24035040 tx=0x7f5c240046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.032+0000 7f5c3b200640 1 -- 172.21.15.40:0/3721473296 shutdown_connections 2024-04-26T23:02:44.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.032+0000 7f5c3b200640 1 --2- 172.21.15.40:0/3721473296 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5c1806c8f0 0x7f5c1806edb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.032+0000 7f5c3b200640 1 --2- 172.21.15.40:0/3721473296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c3415c5c0 0x7f5c34146ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.032+0000 7f5c3b200640 1 --2- 172.21.15.40:0/3721473296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3415c160 0x7f5c340cf720 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.032+0000 7f5c3b200640 1 --2- 172.21.15.40:0/3721473296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5c34073e10 0x7f5c340ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.032+0000 7f5c3b200640 1 -- 172.21.15.40:0/3721473296 >> 172.21.15.40:0/3721473296 conn(0x7f5c34075eb0 msgr2=0x7f5c3415acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.032+0000 7f5c3b200640 1 -- 172.21.15.40:0/3721473296 shutdown_connections 2024-04-26T23:02:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.032+0000 7f5c3b200640 1 -- 172.21.15.40:0/3721473296 wait complete. 2024-04-26T23:02:44.049 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:44.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.250+0000 7fcea7200640 1 Processor -- start 2024-04-26T23:02:44.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.250+0000 7fcea7200640 1 -- start start 2024-04-26T23:02:44.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.252+0000 7fcea7200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcea80cc980 0x7fcea80cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.252+0000 7fcea7200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcea80cc230 0x7fcea8158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.252+0000 7fcea7200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcea80cce10 con 0x7fcea80c6f10 2024-04-26T23:02:44.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.252+0000 7fcea7200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fcea80c72f0 con 0x7fcea80cc230 2024-04-26T23:02:44.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.252+0000 7fcea7200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcea80cb860 con 0x7fcea80cc980 2024-04-26T23:02:44.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.252+0000 7fcea5e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcea80c6f10 0x7fcea80c5ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55034/0 (socket says 172.21.15.40:55034) 2024-04-26T23:02:44.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.252+0000 7fcea5e00640 1 -- 172.21.15.40:0/423398384 learned_addr learned my addr 172.21.15.40:0/423398384 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.252+0000 7fcea6800640 1 --2- 172.21.15.40:0/423398384 >> v2:172.21.15.40:3300/0 conn(0x7fcea80cc230 0x7fcea8158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.253+0000 7fcea5400640 1 --2- 172.21.15.40:0/423398384 >> v2:172.21.15.103:3300/0 conn(0x7fcea80cc980 0x7fcea80cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.253+0000 7fcea4a00640 1 -- 172.21.15.40:0/423398384 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2791976797 0 0) 0x7fcea80cce10 con 0x7fcea80c6f10 2024-04-26T23:02:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.253+0000 7fcea4a00640 1 -- 172.21.15.40:0/423398384 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce8c003520 con 0x7fcea80c6f10 2024-04-26T23:02:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.253+0000 7fcea6800640 1 -- 172.21.15.40:0/423398384 >> v2:172.21.15.103:3300/0 conn(0x7fcea80cc980 msgr2=0x7fcea80cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.253+0000 7fcea6800640 1 --2- 172.21.15.40:0/423398384 >> v2:172.21.15.103:3300/0 conn(0x7fcea80cc980 0x7fcea80cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.253+0000 7fcea6800640 1 -- 172.21.15.40:0/423398384 >> v1:172.21.15.40:6789/0 conn(0x7fcea80c6f10 legacy=0x7fcea80c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.253+0000 7fcea6800640 1 -- 172.21.15.40:0/423398384 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea80cd0f0 con 0x7fcea80cc230 2024-04-26T23:02:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.253+0000 7fcea5400640 1 --2- 172.21.15.40:0/423398384 >> v2:172.21.15.103:3300/0 conn(0x7fcea80cc980 0x7fcea80cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:44.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.254+0000 7fcea6800640 1 --2- 172.21.15.40:0/423398384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcea80cc230 0x7fcea8158cd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fcea80b0460 tx=0x7fcea002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65a8722e954b0f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:44.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.254+0000 7fcea4a00640 1 -- 172.21.15.40:0/423398384 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea0007740 con 0x7fcea80cc230 2024-04-26T23:02:44.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.254+0000 7fcea4a00640 1 -- 172.21.15.40:0/423398384 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcea00078e0 con 0x7fcea80cc230 2024-04-26T23:02:44.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.254+0000 7fcea4a00640 1 -- 172.21.15.40:0/423398384 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea0007cb0 con 0x7fcea80cc230 2024-04-26T23:02:44.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.255+0000 7fcea7200640 1 -- 172.21.15.40:0/423398384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcea80cc230 msgr2=0x7fcea8158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.255+0000 7fcea7200640 1 --2- 172.21.15.40:0/423398384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcea80cc230 0x7fcea8158cd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fcea80b0460 tx=0x7fcea002a5f0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.255+0000 7fcea7200640 1 -- 172.21.15.40:0/423398384 shutdown_connections 2024-04-26T23:02:44.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.255+0000 7fcea7200640 1 --2- 172.21.15.40:0/423398384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcea80cc230 0x7fcea8158cd0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.255+0000 7fcea7200640 1 --2- 172.21.15.40:0/423398384 >> v2:172.21.15.103:3300/0 conn(0x7fcea80cc980 0x7fcea80cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.255+0000 7fcea7200640 1 -- 172.21.15.40:0/423398384 >> 172.21.15.40:0/423398384 conn(0x7fcea80d0a10 msgr2=0x7fcea80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:44.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.255+0000 7fcea7200640 1 -- 172.21.15.40:0/423398384 shutdown_connections 2024-04-26T23:02:44.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.255+0000 7fcea7200640 1 -- 172.21.15.40:0/423398384 wait complete. 2024-04-26T23:02:44.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.256+0000 7fcea7200640 1 Processor -- start 2024-04-26T23:02:44.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.257+0000 7fcea7200640 1 -- start start 2024-04-26T23:02:44.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.257+0000 7fcea7200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcea80c6f10 0x7fcea80b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.257+0000 7fcea7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcea80cc230 0x7fcea8161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.257+0000 7fcea7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcea81524e0 0x7fcea80c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.257+0000 7fcea7200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcea80c61e0 con 0x7fcea80c6f10 2024-04-26T23:02:44.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.257+0000 7fcea7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcea80cb1a0 con 0x7fcea81524e0 2024-04-26T23:02:44.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.257+0000 7fcea7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcea80c6360 con 0x7fcea80cc230 2024-04-26T23:02:44.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.258+0000 7fcea5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcea80c6f10 0x7fcea80b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:44.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.258+0000 7fcea5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcea80c6f10 0x7fcea80b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33008/0 (socket says 172.21.15.40:33008) 2024-04-26T23:02:44.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.258+0000 7fcea5e00640 1 -- 172.21.15.40:0/1151710975 learned_addr learned my addr 172.21.15.40:0/1151710975 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:44.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.258+0000 7fcea5e00640 1 -- 172.21.15.40:0/1151710975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcea80cc230 msgr2=0x7fcea8161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.258+0000 7fcea5e00640 1 --2- 172.21.15.40:0/1151710975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcea80cc230 0x7fcea8161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.258+0000 7fcea5e00640 1 -- 172.21.15.40:0/1151710975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcea81524e0 msgr2=0x7fcea80c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.258+0000 7fcea5e00640 1 --2- 172.21.15.40:0/1151710975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcea81524e0 0x7fcea80c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.258+0000 7fcea5e00640 1 -- 172.21.15.40:0/1151710975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea8162210 con 0x7fcea80c6f10 2024-04-26T23:02:44.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.258+0000 7fcea5e00640 1 --2- 172.21.15.40:0/1151710975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcea80c6f10 0x7fcea80b3730 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fce9800c460 tx=0x7fce9800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:44.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.259+0000 7fce9e000640 1 -- 172.21.15.40:0/1151710975 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce98017070 con 0x7fcea80c6f10 2024-04-26T23:02:44.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.259+0000 7fce9e000640 1 -- 172.21.15.40:0/1151710975 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce98005ce0 con 0x7fcea80c6f10 2024-04-26T23:02:44.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.259+0000 7fce9e000640 1 -- 172.21.15.40:0/1151710975 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce98007290 con 0x7fcea80c6f10 2024-04-26T23:02:44.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.259+0000 7fcea7200640 1 -- 172.21.15.40:0/1151710975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea80b3da0 con 0x7fcea80c6f10 2024-04-26T23:02:44.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.259+0000 7fcea7200640 1 -- 172.21.15.40:0/1151710975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea80b4360 con 0x7fcea80c6f10 2024-04-26T23:02:44.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.260+0000 7fcea7200640 1 -- 172.21.15.40:0/1151710975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcea8069370 con 0x7fcea80c6f10 2024-04-26T23:02:44.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.261+0000 7fce9e000640 1 -- 172.21.15.40:0/1151710975 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fce9801a070 con 0x7fcea80c6f10 2024-04-26T23:02:44.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.262+0000 7fce9e000640 1 --2- 172.21.15.40:0/1151710975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fce8806c8f0 0x7fce8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.263+0000 7fce9e000640 1 -- 172.21.15.40:0/1151710975 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fce9808ad20 con 0x7fcea80c6f10 2024-04-26T23:02:44.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.263+0000 7fcea5400640 1 --2- 172.21.15.40:0/1151710975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fce8806c8f0 0x7fce8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:44.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.264+0000 7fcea5400640 1 --2- 172.21.15.40:0/1151710975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fce8806c8f0 0x7fce8806edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fce94006fd0 tx=0x7fce94008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:44.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.267+0000 7fce9e000640 1 -- 172.21.15.40:0/1151710975 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fce98059310 con 0x7fcea80c6f10 2024-04-26T23:02:44.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.470+0000 7fcea7200640 1 -- 172.21.15.40:0/1151710975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcea8070020 con 0x7fcea80c6f10 2024-04-26T23:02:44.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.471+0000 7fce9e000640 1 -- 172.21.15.40:0/1151710975 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+13128 (secure 0 0 0) 0x7fce98058c10 con 0x7fcea80c6f10 2024-04-26T23:02:44.472 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:44.472 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":35,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:02:40.360507+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:44.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.474+0000 7fcea7200640 1 -- 172.21.15.40:0/1151710975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fce8806c8f0 msgr2=0x7fce8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.474+0000 7fcea7200640 1 --2- 172.21.15.40:0/1151710975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fce8806c8f0 0x7fce8806edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fce94006fd0 tx=0x7fce94008040 comp rx=0 tx=0).stop 2024-04-26T23:02:44.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.474+0000 7fcea7200640 1 -- 172.21.15.40:0/1151710975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcea80c6f10 msgr2=0x7fcea80b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.474+0000 7fcea7200640 1 --2- 172.21.15.40:0/1151710975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcea80c6f10 0x7fcea80b3730 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fce9800c460 tx=0x7fce9800c920 comp rx=0 tx=0).stop 2024-04-26T23:02:44.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.474+0000 7fcea7200640 1 -- 172.21.15.40:0/1151710975 shutdown_connections 2024-04-26T23:02:44.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.474+0000 7fcea7200640 1 --2- 172.21.15.40:0/1151710975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fce8806c8f0 0x7fce8806edb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.474+0000 7fcea7200640 1 --2- 172.21.15.40:0/1151710975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcea81524e0 0x7fcea80c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.474+0000 7fcea7200640 1 --2- 172.21.15.40:0/1151710975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcea80cc230 0x7fcea8161cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.474+0000 7fcea7200640 1 --2- 172.21.15.40:0/1151710975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcea80c6f10 0x7fcea80b3730 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.474+0000 7fcea7200640 1 -- 172.21.15.40:0/1151710975 >> 172.21.15.40:0/1151710975 conn(0x7fcea80d0a10 msgr2=0x7fcea8162480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:44.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.474+0000 7fcea7200640 1 -- 172.21.15.40:0/1151710975 shutdown_connections 2024-04-26T23:02:44.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.474+0000 7fcea7200640 1 -- 172.21.15.40:0/1151710975 wait complete. 2024-04-26T23:02:44.493 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:02:44.494 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:02:44.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.702+0000 7f1704c00640 1 Processor -- start 2024-04-26T23:02:44.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.702+0000 7f1704c00640 1 -- start start 2024-04-26T23:02:44.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.702+0000 7f1704c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1700160a50 0x7f1700073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.702+0000 7f1704c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f170006dd40 0x7f1700072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.702+0000 7f1704c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1700161880 con 0x7f17000cfaf0 2024-04-26T23:02:44.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.702+0000 7f1704c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1700071ce0 con 0x7f1700160a50 2024-04-26T23:02:44.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.702+0000 7f1704c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f17000722a0 con 0x7f170006dd40 2024-04-26T23:02:44.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.703+0000 7f16fea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f17000cfaf0 0x7f1700160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55046/0 (socket says 172.21.15.40:55046) 2024-04-26T23:02:44.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.703+0000 7f16fea00640 1 -- 172.21.15.40:0/2266137204 learned_addr learned my addr 172.21.15.40:0/2266137204 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:44.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.703+0000 7f16ff400640 1 --2- 172.21.15.40:0/2266137204 >> v2:172.21.15.103:3300/0 conn(0x7f1700160a50 0x7f1700073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:44.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.703+0000 7f16ffe00640 1 --2- 172.21.15.40:0/2266137204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f170006dd40 0x7f1700072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:44.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.703+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 677740309 0 0) 0x7f1700161880 con 0x7f17000cfaf0 2024-04-26T23:02:44.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16e4003520 con 0x7f17000cfaf0 2024-04-26T23:02:44.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3154747569 0 0) 0x7f16e4003520 con 0x7f17000cfaf0 2024-04-26T23:02:44.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f170006dd40 msgr2=0x7f1700072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:44.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 --2- 172.21.15.40:0/2266137204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f170006dd40 0x7f1700072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 >> v2:172.21.15.103:3300/0 conn(0x7f1700160a50 msgr2=0x7f1700073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:44.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 --2- 172.21.15.40:0/2266137204 >> v2:172.21.15.103:3300/0 conn(0x7f1700160a50 0x7f1700073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17000cff80 con 0x7f17000cfaf0 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f16ec001670 con 0x7f17000cfaf0 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 >> v1:172.21.15.40:6789/0 conn(0x7f17000cfaf0 legacy=0x7f1700160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 --2- 172.21.15.40:0/2266137204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16e40039b0 0x7f16e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 --2- 172.21.15.40:0/2266137204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f170006dd40 0x7f16e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 --2- 172.21.15.40:0/2266137204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16e4004d40 0x7f16e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1700071ce0 con 0x7f16e4004d40 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17000722a0 con 0x7f16e40039b0 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f16e400b9c0 con 0x7f170006dd40 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.704+0000 7f16ffe00640 1 --2- 172.21.15.40:0/2266137204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16e4004d40 0x7f16e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.705+0000 7f16ff400640 1 --2- 172.21.15.40:0/2266137204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16e40039b0 0x7f16e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.705+0000 7f16ffe00640 1 -- 172.21.15.40:0/2266137204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f170006dd40 msgr2=0x7f16e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.705+0000 7f16ffe00640 1 --2- 172.21.15.40:0/2266137204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f170006dd40 0x7f16e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.705+0000 7f16ffe00640 1 -- 172.21.15.40:0/2266137204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16e40039b0 msgr2=0x7f16e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.705+0000 7f16ffe00640 1 --2- 172.21.15.40:0/2266137204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16e40039b0 0x7f16e4003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.705+0000 7f16ffe00640 1 -- 172.21.15.40:0/2266137204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e4003520 con 0x7f16e4004d40 2024-04-26T23:02:44.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.707+0000 7f16ffe00640 1 --2- 172.21.15.40:0/2266137204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16e4004d40 0x7f16e40051c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f16f800e170 tx=0x7f16e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f687d0e0c33fa28 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:44.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.707+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f8034020 con 0x7f16e4004d40 2024-04-26T23:02:44.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.707+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16f8030430 con 0x7f16e4004d40 2024-04-26T23:02:44.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.707+0000 7f16fe000640 1 -- 172.21.15.40:0/2266137204 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f8009c10 con 0x7f16e4004d40 2024-04-26T23:02:44.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.707+0000 7f1704c00640 1 -- 172.21.15.40:0/2266137204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16e4004d40 msgr2=0x7f16e40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.707+0000 7f1704c00640 1 --2- 172.21.15.40:0/2266137204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16e4004d40 0x7f16e40051c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f16f800e170 tx=0x7f16e4005830 comp rx=0 tx=0).stop 2024-04-26T23:02:44.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.707+0000 7f16ffe00640 1 -- 172.21.15.40:0/2266137204 reap_dead start 2024-04-26T23:02:44.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.707+0000 7f1704c00640 1 -- 172.21.15.40:0/2266137204 shutdown_connections 2024-04-26T23:02:44.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.707+0000 7f1704c00640 1 -- 172.21.15.40:0/2266137204 >> 172.21.15.40:0/2266137204 conn(0x7f17000b2720 msgr2=0x7f17000b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:44.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.707+0000 7f1704c00640 1 -- 172.21.15.40:0/2266137204 shutdown_connections 2024-04-26T23:02:44.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.707+0000 7f1704c00640 1 -- 172.21.15.40:0/2266137204 wait complete. 2024-04-26T23:02:44.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.709+0000 7f1704c00640 1 Processor -- start 2024-04-26T23:02:44.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.709+0000 7f1704c00640 1 -- start start 2024-04-26T23:02:44.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.710+0000 7f1704c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16e4004d40 0x7f170015a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.710+0000 7f1704c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1700142910 0x7f1700142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.710+0000 7f1704c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17001433f0 0x7f17000b16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.710+0000 7f1704c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1700071f30 con 0x7f16e4004d40 2024-04-26T23:02:44.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.710+0000 7f1704c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1700073f90 con 0x7f17001433f0 2024-04-26T23:02:44.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.710+0000 7f1704c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f170006e1f0 con 0x7f1700142910 2024-04-26T23:02:44.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.710+0000 7f16ff400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16e4004d40 0x7f170015a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:44.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.710+0000 7f16ff400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16e4004d40 0x7f170015a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33024/0 (socket says 172.21.15.40:33024) 2024-04-26T23:02:44.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.710+0000 7f16ff400640 1 -- 172.21.15.40:0/3506711543 learned_addr learned my addr 172.21.15.40:0/3506711543 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:44.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.711+0000 7f16ff400640 1 -- 172.21.15.40:0/3506711543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1700142910 msgr2=0x7f1700142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.711+0000 7f16ff400640 1 --2- 172.21.15.40:0/3506711543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1700142910 0x7f1700142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.711+0000 7f16ff400640 1 -- 172.21.15.40:0/3506711543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17001433f0 msgr2=0x7f17000b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.711+0000 7f16ffe00640 1 --2- 172.21.15.40:0/3506711543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17001433f0 0x7f17000b16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:44.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.711+0000 7f16ff400640 1 --2- 172.21.15.40:0/3506711543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17001433f0 0x7f17000b16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.711+0000 7f16ff400640 1 -- 172.21.15.40:0/3506711543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170015b120 con 0x7f16e4004d40 2024-04-26T23:02:44.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.711+0000 7f16ff400640 1 --2- 172.21.15.40:0/3506711543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16e4004d40 0x7f170015a420 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f16f00108e0 tx=0x7f16f0010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:44.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.711+0000 7f16f7400640 1 -- 172.21.15.40:0/3506711543 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f000f040 con 0x7f16e4004d40 2024-04-26T23:02:44.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.711+0000 7f16f7400640 1 -- 172.21.15.40:0/3506711543 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16f000eb60 con 0x7f16e4004d40 2024-04-26T23:02:44.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.711+0000 7f16f7400640 1 -- 172.21.15.40:0/3506711543 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f001c400 con 0x7f16e4004d40 2024-04-26T23:02:44.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.711+0000 7f1704c00640 1 -- 172.21.15.40:0/3506711543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170015b3c0 con 0x7f16e4004d40 2024-04-26T23:02:44.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.711+0000 7f1704c00640 1 -- 172.21.15.40:0/3506711543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f170015b980 con 0x7f16e4004d40 2024-04-26T23:02:44.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.712+0000 7f1704c00640 1 -- 172.21.15.40:0/3506711543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1700069370 con 0x7f16e4004d40 2024-04-26T23:02:44.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.713+0000 7f16f7400640 1 -- 172.21.15.40:0/3506711543 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f16f001c5a0 con 0x7f16e4004d40 2024-04-26T23:02:44.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.714+0000 7f16f7400640 1 --2- 172.21.15.40:0/3506711543 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f16e006c920 0x7f16e006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:44.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.714+0000 7f16f7400640 1 -- 172.21.15.40:0/3506711543 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f16f008c1c0 con 0x7f16e4004d40 2024-04-26T23:02:44.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.715+0000 7f16fea00640 1 --2- 172.21.15.40:0/3506711543 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f16e006c920 0x7f16e006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:44.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.716+0000 7f16fea00640 1 --2- 172.21.15.40:0/3506711543 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f16e006c920 0x7f16e006ede0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f17001f8630 tx=0x7f16ec001530 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:44.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.720+0000 7f16f7400640 1 -- 172.21.15.40:0/3506711543 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f16f0018030 con 0x7f16e4004d40 2024-04-26T23:02:44.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.922+0000 7f1704c00640 1 -- 172.21.15.40:0/3506711543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1700140b90 con 0x7f16e4004d40 2024-04-26T23:02:44.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.923+0000 7f16f7400640 1 -- 172.21.15.40:0/3506711543 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) ==== 93+0+10 (secure 0 0 0) 0x7f16f005a710 con 0x7f16e4004d40 2024-04-26T23:02:44.923 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:02:44.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.926+0000 7f1704c00640 1 -- 172.21.15.40:0/3506711543 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f16e006c920 msgr2=0x7f16e006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.926+0000 7f1704c00640 1 --2- 172.21.15.40:0/3506711543 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f16e006c920 0x7f16e006ede0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f17001f8630 tx=0x7f16ec001530 comp rx=0 tx=0).stop 2024-04-26T23:02:44.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.926+0000 7f1704c00640 1 -- 172.21.15.40:0/3506711543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16e4004d40 msgr2=0x7f170015a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:44.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.926+0000 7f1704c00640 1 --2- 172.21.15.40:0/3506711543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16e4004d40 0x7f170015a420 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f16f00108e0 tx=0x7f16f0010da0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.926+0000 7f1704c00640 1 -- 172.21.15.40:0/3506711543 shutdown_connections 2024-04-26T23:02:44.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.926+0000 7f1704c00640 1 --2- 172.21.15.40:0/3506711543 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f16e006c920 0x7f16e006ede0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.926+0000 7f1704c00640 1 --2- 172.21.15.40:0/3506711543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17001433f0 0x7f17000b16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.926+0000 7f1704c00640 1 --2- 172.21.15.40:0/3506711543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1700142910 0x7f1700142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.926+0000 7f1704c00640 1 --2- 172.21.15.40:0/3506711543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16e4004d40 0x7f170015a420 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:44.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.926+0000 7f1704c00640 1 -- 172.21.15.40:0/3506711543 >> 172.21.15.40:0/3506711543 conn(0x7f17000b2720 msgr2=0x7f170006f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:44.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.926+0000 7f1704c00640 1 -- 172.21.15.40:0/3506711543 shutdown_connections 2024-04-26T23:02:44.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:44.926+0000 7f1704c00640 1 -- 172.21.15.40:0/3506711543 wait complete. 2024-04-26T23:02:44.946 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T23:02:45.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.156+0000 7ff601200640 1 Processor -- start 2024-04-26T23:02:45.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.156+0000 7ff601200640 1 -- start start 2024-04-26T23:02:45.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.157+0000 7ff601200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5fc155f20 0x7ff5fc0d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:45.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.157+0000 7ff601200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5fc15f490 0x7ff5fc06e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:45.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.157+0000 7ff601200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5fc15fca0 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.157+0000 7ff601200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff5fc15ff80 con 0x7ff5fc155f20 2024-04-26T23:02:45.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.157+0000 7ff601200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff5fc160260 con 0x7ff5fc15f490 2024-04-26T23:02:45.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.157+0000 7ff5fb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5fc155f20 0x7ff5fc0d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:45.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.158+0000 7ff5fb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5fc155f20 0x7ff5fc0d11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42498/0 (socket says 172.21.15.40:42498) 2024-04-26T23:02:45.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.158+0000 7ff5fb400640 1 -- 172.21.15.40:0/3219766365 learned_addr learned my addr 172.21.15.40:0/3219766365 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:45.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.158+0000 7ff5fbe00640 1 --1- 172.21.15.40:0/3219766365 >> v1:172.21.15.40:6789/0 conn(0x7ff5fc06eaa0 0x7ff5fc0b5500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49880/0 (socket says 172.21.15.40:49880) 2024-04-26T23:02:45.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.158+0000 7ff5faa00640 1 --2- 172.21.15.40:0/3219766365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5fc15f490 0x7ff5fc06e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:45.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.158+0000 7ff5fb400640 1 -- 172.21.15.40:0/3219766365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5fc15f490 msgr2=0x7ff5fc06e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:45.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.158+0000 7ff5fb400640 1 --2- 172.21.15.40:0/3219766365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5fc15f490 0x7ff5fc06e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.158+0000 7ff5fb400640 1 -- 172.21.15.40:0/3219766365 >> v1:172.21.15.40:6789/0 conn(0x7ff5fc06eaa0 legacy=0x7ff5fc0b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:45.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.158+0000 7ff5fb400640 1 -- 172.21.15.40:0/3219766365 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5fc06e040 con 0x7ff5fc155f20 2024-04-26T23:02:45.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.160+0000 7ff5fb400640 1 --2- 172.21.15.40:0/3219766365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5fc155f20 0x7ff5fc0d11a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009870 tx=0x7ff5ec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16094d5ba1468734 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:45.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.160+0000 7ff5fa000640 1 -- 172.21.15.40:0/3219766365 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec037070 con 0x7ff5fc155f20 2024-04-26T23:02:45.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.160+0000 7ff5fa000640 1 -- 172.21.15.40:0/3219766365 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5ec032440 con 0x7ff5fc155f20 2024-04-26T23:02:45.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.160+0000 7ff5fa000640 1 -- 172.21.15.40:0/3219766365 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec032810 con 0x7ff5fc155f20 2024-04-26T23:02:45.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.160+0000 7ff601200640 1 -- 172.21.15.40:0/3219766365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5fc155f20 msgr2=0x7ff5fc0d11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:45.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.160+0000 7ff601200640 1 --2- 172.21.15.40:0/3219766365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5fc155f20 0x7ff5fc0d11a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009870 tx=0x7ff5ec02a270 comp rx=0 tx=0).stop 2024-04-26T23:02:45.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.161+0000 7ff601200640 1 -- 172.21.15.40:0/3219766365 shutdown_connections 2024-04-26T23:02:45.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.161+0000 7ff601200640 1 --2- 172.21.15.40:0/3219766365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5fc15f490 0x7ff5fc06e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.161+0000 7ff601200640 1 --2- 172.21.15.40:0/3219766365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5fc155f20 0x7ff5fc0d11a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.161+0000 7ff601200640 1 -- 172.21.15.40:0/3219766365 >> 172.21.15.40:0/3219766365 conn(0x7ff5fc075eb0 msgr2=0x7ff5fc068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:45.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.161+0000 7ff601200640 1 -- 172.21.15.40:0/3219766365 shutdown_connections 2024-04-26T23:02:45.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.161+0000 7ff601200640 1 -- 172.21.15.40:0/3219766365 wait complete. 2024-04-26T23:02:45.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.162+0000 7ff601200640 1 Processor -- start 2024-04-26T23:02:45.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.163+0000 7ff601200640 1 -- start start 2024-04-26T23:02:45.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.163+0000 7ff601200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5fc06eaa0 0x7ff5fc150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:45.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.163+0000 7ff601200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5fc155f20 0x7ff5fc0b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:45.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.164+0000 7ff601200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5fc15f490 0x7ff5fc14b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:45.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.164+0000 7ff601200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff5fc15fa50 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.164+0000 7ff601200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff5fc15f8d0 con 0x7ff5fc155f20 2024-04-26T23:02:45.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.164+0000 7ff601200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff5fc0b5c10 con 0x7ff5fc15f490 2024-04-26T23:02:45.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.164+0000 7ff5fb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5fc06eaa0 0x7ff5fc150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:45.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.164+0000 7ff5fb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5fc06eaa0 0x7ff5fc150f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42510/0 (socket says 172.21.15.40:42510) 2024-04-26T23:02:45.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.164+0000 7ff5fb400640 1 -- 172.21.15.40:0/2852153837 learned_addr learned my addr 172.21.15.40:0/2852153837 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:45.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.164+0000 7ff5fb400640 1 -- 172.21.15.40:0/2852153837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5fc15f490 msgr2=0x7ff5fc14b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:45.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.164+0000 7ff5fb400640 1 --2- 172.21.15.40:0/2852153837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5fc15f490 0x7ff5fc14b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.164+0000 7ff5fb400640 1 -- 172.21.15.40:0/2852153837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5fc155f20 msgr2=0x7ff5fc0b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:45.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.164+0000 7ff5fb400640 1 --2- 172.21.15.40:0/2852153837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5fc155f20 0x7ff5fc0b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.164+0000 7ff5fb400640 1 -- 172.21.15.40:0/2852153837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5fc14e6e0 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.165+0000 7ff5fb400640 1 --2- 172.21.15.40:0/2852153837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5fc06eaa0 0x7ff5fc150f00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec000c00 tx=0x7ff5ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:45.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.165+0000 7ff5f3400640 1 -- 172.21.15.40:0/2852153837 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec041020 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.165+0000 7ff5f3400640 1 -- 172.21.15.40:0/2852153837 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5ec0306c0 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.165+0000 7ff601200640 1 -- 172.21.15.40:0/2852153837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5fc14e9c0 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.165+0000 7ff5f3400640 1 -- 172.21.15.40:0/2852153837 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec037040 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.165+0000 7ff601200640 1 -- 172.21.15.40:0/2852153837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5fc0b25e0 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.166+0000 7ff601200640 1 -- 172.21.15.40:0/2852153837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5fc162420 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.167+0000 7ff5f3400640 1 -- 172.21.15.40:0/2852153837 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff5ec044020 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.170+0000 7ff5f3400640 1 --2- 172.21.15.40:0/2852153837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5dc06c920 0x7ff5dc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:45.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.170+0000 7ff5f3400640 1 -- 172.21.15.40:0/2852153837 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7ff5ec0ac1a0 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.170+0000 7ff5faa00640 1 --2- 172.21.15.40:0/2852153837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5dc06c920 0x7ff5dc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:45.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.170+0000 7ff5faa00640 1 --2- 172.21.15.40:0/2852153837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5dc06c920 0x7ff5dc06ede0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80070b0 tx=0x7ff5e8007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:45.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.174+0000 7ff5f3400640 1 -- 172.21.15.40:0/2852153837 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff5fc162420 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.382+0000 7ff601200640 1 -- 172.21.15.40:0/2852153837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff5fc069370 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.382+0000 7ff5f3400640 1 -- 172.21.15.40:0/2852153837 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) ==== 84+0+65 (secure 0 0 0) 0x7ff5ec04b030 con 0x7ff5fc06eaa0 2024-04-26T23:02:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:45.382 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-04-26T23:02:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:02:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-04-26T23:02:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-04-26T23:02:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:02:45.382 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-04-26T23:02:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.384+0000 7ff601200640 1 -- 172.21.15.40:0/2852153837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5dc06c920 msgr2=0x7ff5dc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.384+0000 7ff601200640 1 --2- 172.21.15.40:0/2852153837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5dc06c920 0x7ff5dc06ede0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80070b0 tx=0x7ff5e8007040 comp rx=0 tx=0).stop 2024-04-26T23:02:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.384+0000 7ff601200640 1 -- 172.21.15.40:0/2852153837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5fc06eaa0 msgr2=0x7ff5fc150f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.384+0000 7ff601200640 1 --2- 172.21.15.40:0/2852153837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5fc06eaa0 0x7ff5fc150f00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec000c00 tx=0x7ff5ec0046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.384+0000 7ff601200640 1 -- 172.21.15.40:0/2852153837 shutdown_connections 2024-04-26T23:02:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.384+0000 7ff601200640 1 --2- 172.21.15.40:0/2852153837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5dc06c920 0x7ff5dc06ede0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.384+0000 7ff601200640 1 --2- 172.21.15.40:0/2852153837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5fc15f490 0x7ff5fc14b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.384+0000 7ff601200640 1 --2- 172.21.15.40:0/2852153837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5fc155f20 0x7ff5fc0b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.384+0000 7ff601200640 1 --2- 172.21.15.40:0/2852153837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5fc06eaa0 0x7ff5fc150f00 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.384+0000 7ff601200640 1 -- 172.21.15.40:0/2852153837 >> 172.21.15.40:0/2852153837 conn(0x7ff5fc075eb0 msgr2=0x7ff5fc068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.384+0000 7ff601200640 1 -- 172.21.15.40:0/2852153837 shutdown_connections 2024-04-26T23:02:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.384+0000 7ff601200640 1 -- 172.21.15.40:0/2852153837 wait complete. 2024-04-26T23:02:45.404 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T23:02:45.405 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T23:02:45.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.625+0000 7f84f4800640 1 Processor -- start 2024-04-26T23:02:45.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.625+0000 7f84f4800640 1 -- start start 2024-04-26T23:02:45.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.626+0000 7f84f4800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84ec0d3ef0 0x7f84ec0d1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:45.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.626+0000 7f84f4800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f84ec0d1eb0 0x7f84ec0b7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:45.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.626+0000 7f84f4800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84ec0d2e50 con 0x7f84ec15f2b0 2024-04-26T23:02:45.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.626+0000 7f84f4800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f84ec0d2c00 con 0x7f84ec0d1eb0 2024-04-26T23:02:45.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.626+0000 7f84f4800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f84ec0d3a60 con 0x7f84ec0d3ef0 2024-04-26T23:02:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.627+0000 7f84f2800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f84ec0d1eb0 0x7f84ec0b7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.627+0000 7f84f2800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f84ec0d1eb0 0x7f84ec0b7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:44880/0 (socket says 172.21.15.40:44880) 2024-04-26T23:02:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.627+0000 7f84f2800640 1 -- 172.21.15.40:0/3132318194 learned_addr learned my addr 172.21.15.40:0/3132318194 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.627+0000 7f84f1400640 1 --2- 172.21.15.40:0/3132318194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84ec0d3ef0 0x7f84ec0d1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:45.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.628+0000 7f84f2800640 1 -- 172.21.15.40:0/3132318194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84ec0d3ef0 msgr2=0x7f84ec0d1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:45.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.628+0000 7f84f2800640 1 --2- 172.21.15.40:0/3132318194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84ec0d3ef0 0x7f84ec0d1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.628+0000 7f84f2800640 1 -- 172.21.15.40:0/3132318194 >> v1:172.21.15.103:6789/0 conn(0x7f84ec15f2b0 legacy=0x7f84ec0d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:45.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.628+0000 7f84f2800640 1 -- 172.21.15.40:0/3132318194 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ec0d3130 con 0x7f84ec0d1eb0 2024-04-26T23:02:45.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.628+0000 7f84f0a00640 1 -- 172.21.15.40:0/3132318194 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2392891491 0 0) 0x7f84ec0d2e50 con 0x7f84ec15f2b0 2024-04-26T23:02:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.630+0000 7f84f2800640 1 --2- 172.21.15.40:0/3132318194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84ec0d1eb0 0x7f84ec0b7e80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f84e8009e50 tx=0x7f84e802a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b901adcf5679b47 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:45.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.630+0000 7f84f0a00640 1 -- 172.21.15.40:0/3132318194 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8036070 con 0x7f84ec0d1eb0 2024-04-26T23:02:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.630+0000 7f84f0a00640 1 -- 172.21.15.40:0/3132318194 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84e8031600 con 0x7f84ec0d1eb0 2024-04-26T23:02:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.630+0000 7f84f0a00640 1 -- 172.21.15.40:0/3132318194 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e80319d0 con 0x7f84ec0d1eb0 2024-04-26T23:02:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.630+0000 7f84f4800640 1 -- 172.21.15.40:0/3132318194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84ec0d1eb0 msgr2=0x7f84ec0b7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.630+0000 7f84f4800640 1 --2- 172.21.15.40:0/3132318194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84ec0d1eb0 0x7f84ec0b7e80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f84e8009e50 tx=0x7f84e802a710 comp rx=0 tx=0).stop 2024-04-26T23:02:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.630+0000 7f84f4800640 1 -- 172.21.15.40:0/3132318194 shutdown_connections 2024-04-26T23:02:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.630+0000 7f84f4800640 1 --2- 172.21.15.40:0/3132318194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84ec0d1eb0 0x7f84ec0b7e80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.630+0000 7f84f4800640 1 --2- 172.21.15.40:0/3132318194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84ec0d3ef0 0x7f84ec0d1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.630+0000 7f84f4800640 1 -- 172.21.15.40:0/3132318194 >> 172.21.15.40:0/3132318194 conn(0x7f84ec0d5030 msgr2=0x7f84ec0d5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.630+0000 7f84f4800640 1 -- 172.21.15.40:0/3132318194 shutdown_connections 2024-04-26T23:02:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.631+0000 7f84f4800640 1 -- 172.21.15.40:0/3132318194 wait complete. 2024-04-26T23:02:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.632+0000 7f84f4800640 1 Processor -- start 2024-04-26T23:02:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.632+0000 7f84f4800640 1 -- start start 2024-04-26T23:02:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.633+0000 7f84f4800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84ec0d1eb0 0x7f84ec0b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.633+0000 7f84f4800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84ec0d3ef0 0x7f84ec0cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.633+0000 7f84f4800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84ec15f2b0 0x7f84ec0cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.633+0000 7f84f4800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec15f690 con 0x7f84ec15f2b0 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.633+0000 7f84f4800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec0d36f0 con 0x7f84ec0d1eb0 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.633+0000 7f84f4800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f84ec0d0e60 con 0x7f84ec0d3ef0 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.634+0000 7f84f2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84ec15f2b0 0x7f84ec0cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.634+0000 7f84f2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84ec15f2b0 0x7f84ec0cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42520/0 (socket says 172.21.15.40:42520) 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.634+0000 7f84f2800640 1 -- 172.21.15.40:0/3519041464 learned_addr learned my addr 172.21.15.40:0/3519041464 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.634+0000 7f84f1400640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84ec0d3ef0 0x7f84ec0cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.634+0000 7f84f1e00640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84ec0d1eb0 0x7f84ec0b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.634+0000 7f84f2800640 1 -- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84ec0d3ef0 msgr2=0x7f84ec0cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.634+0000 7f84f2800640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84ec0d3ef0 0x7f84ec0cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.634+0000 7f84f2800640 1 -- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84ec0d1eb0 msgr2=0x7f84ec0b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.634+0000 7f84f2800640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84ec0d1eb0 0x7f84ec0b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.634+0000 7f84f2800640 1 -- 172.21.15.40:0/3519041464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ec1557b0 con 0x7f84ec15f2b0 2024-04-26T23:02:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.635+0000 7f84f2800640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84ec15f2b0 0x7f84ec0cd690 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f84e8009e50 tx=0x7f84e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.635+0000 7f84e6000640 1 -- 172.21.15.40:0/3519041464 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8036050 con 0x7f84ec15f2b0 2024-04-26T23:02:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.635+0000 7f84e6000640 1 -- 172.21.15.40:0/3519041464 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84e8042070 con 0x7f84ec15f2b0 2024-04-26T23:02:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.635+0000 7f84f4800640 1 -- 172.21.15.40:0/3519041464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84ec1529e0 con 0x7f84ec15f2b0 2024-04-26T23:02:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.635+0000 7f84e6000640 1 -- 172.21.15.40:0/3519041464 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8009510 con 0x7f84ec15f2b0 2024-04-26T23:02:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.635+0000 7f84f4800640 1 -- 172.21.15.40:0/3519041464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84ec152fa0 con 0x7f84ec15f2b0 2024-04-26T23:02:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.637+0000 7f84f4800640 1 -- 172.21.15.40:0/3519041464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84ec069600 con 0x7f84ec15f2b0 2024-04-26T23:02:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.639+0000 7f84e6000640 1 -- 172.21.15.40:0/3519041464 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f84e8007bf0 con 0x7f84ec15f2b0 2024-04-26T23:02:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.639+0000 7f84e6000640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f84d006c8f0 0x7f84d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.640+0000 7f84e6000640 1 -- 172.21.15.40:0/3519041464 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f84e80ac810 con 0x7f84ec15f2b0 2024-04-26T23:02:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.640+0000 7f84f1e00640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f84d006c8f0 0x7f84d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.641+0000 7f84f1e00640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f84d006c8f0 0x7f84d006edb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f84e0007fd0 tx=0x7f84e0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.644+0000 7f84e6000640 1 -- 172.21.15.40:0/3519041464 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f84e807ad60 con 0x7f84ec15f2b0 2024-04-26T23:02:45.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:45.848+0000 7f84f4800640 1 -- 172.21.15.40:0/3519041464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f84ec08eb50 con 0x7f84ec15f2b0 2024-04-26T23:02:46.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.375+0000 7f84e6000640 1 -- 172.21.15.40:0/3519041464 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) ==== 138+0+0 (secure 0 0 0) 0x7f84e807a660 con 0x7f84ec15f2b0 2024-04-26T23:02:46.376 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-04-26T23:02:46.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.379+0000 7f84f4800640 1 -- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f84d006c8f0 msgr2=0x7f84d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:46.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.379+0000 7f84f4800640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f84d006c8f0 0x7f84d006edb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f84e0007fd0 tx=0x7f84e0009040 comp rx=0 tx=0).stop 2024-04-26T23:02:46.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.379+0000 7f84f4800640 1 -- 172.21.15.40:0/3519041464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84ec15f2b0 msgr2=0x7f84ec0cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:46.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.379+0000 7f84f4800640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84ec15f2b0 0x7f84ec0cd690 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f84e8009e50 tx=0x7f84e8004a40 comp rx=0 tx=0).stop 2024-04-26T23:02:46.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.379+0000 7f84f4800640 1 -- 172.21.15.40:0/3519041464 shutdown_connections 2024-04-26T23:02:46.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.379+0000 7f84f4800640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f84d006c8f0 0x7f84d006edb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:46.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.379+0000 7f84f4800640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84ec15f2b0 0x7f84ec0cd690 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:46.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.379+0000 7f84f4800640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84ec0d3ef0 0x7f84ec0cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:46.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.379+0000 7f84f4800640 1 --2- 172.21.15.40:0/3519041464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84ec0d1eb0 0x7f84ec0b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:46.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.379+0000 7f84f4800640 1 -- 172.21.15.40:0/3519041464 >> 172.21.15.40:0/3519041464 conn(0x7f84ec0d5030 msgr2=0x7f84ec06e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:46.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.380+0000 7f84f4800640 1 -- 172.21.15.40:0/3519041464 shutdown_connections 2024-04-26T23:02:46.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.380+0000 7f84f4800640 1 -- 172.21.15.40:0/3519041464 wait complete. 2024-04-26T23:02:46.411 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T23:02:46.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.624+0000 7f5ac7e00640 1 Processor -- start 2024-04-26T23:02:46.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.624+0000 7f5ac7e00640 1 -- start start 2024-04-26T23:02:46.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.625+0000 7f5ac7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ac815f4f0 0x7f5ac80d34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:46.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.625+0000 7f5ac7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5ac80d39e0 0x7f5ac80b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:46.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.625+0000 7f5ac7e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ac80b7df0 con 0x7f5ac815f110 2024-04-26T23:02:46.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.625+0000 7f5ac7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac80d3e70 con 0x7f5ac80d39e0 2024-04-26T23:02:46.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.625+0000 7f5ac7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac80b80d0 con 0x7f5ac815f4f0 2024-04-26T23:02:46.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.626+0000 7f5ac6a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5ac815f110 0x7f5ac80d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49886/0 (socket says 172.21.15.40:49886) 2024-04-26T23:02:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.626+0000 7f5ac6a00640 1 -- 172.21.15.40:0/2893729146 learned_addr learned my addr 172.21.15.40:0/2893729146 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.626+0000 7f5ac6000640 1 --2- 172.21.15.40:0/2893729146 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ac815f4f0 0x7f5ac80d34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.626+0000 7f5ac7400640 1 --2- 172.21.15.40:0/2893729146 >> v2:172.21.15.103:3300/0 conn(0x7f5ac80d39e0 0x7f5ac80b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.626+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3037757597 0 0) 0x7f5ac80b7df0 con 0x7f5ac815f110 2024-04-26T23:02:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5aac003520 con 0x7f5ac815f110 2024-04-26T23:02:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2713156269 0 0) 0x7f5aac003520 con 0x7f5ac815f110 2024-04-26T23:02:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ac815f4f0 msgr2=0x7f5ac80d34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 --2- 172.21.15.40:0/2893729146 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ac815f4f0 0x7f5ac80d34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 >> v2:172.21.15.103:3300/0 conn(0x7f5ac80d39e0 msgr2=0x7f5ac80b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 --2- 172.21.15.40:0/2893729146 >> v2:172.21.15.103:3300/0 conn(0x7f5ac80d39e0 0x7f5ac80b5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:46.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac80b83b0 con 0x7f5ac815f110 2024-04-26T23:02:46.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f5ab80019f0 con 0x7f5ac815f110 2024-04-26T23:02:46.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 >> v1:172.21.15.40:6789/0 conn(0x7f5ac815f110 legacy=0x7f5ac80d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:46.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 --2- 172.21.15.40:0/2893729146 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ac815f4f0 0x7f5aac0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:46.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 --2- 172.21.15.40:0/2893729146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aac003f30 0x7f5aac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:46.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 --2- 172.21.15.40:0/2893729146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aac004d40 0x7f5aac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:46.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac80d3e70 con 0x7f5aac003f30 2024-04-26T23:02:46.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac80b80d0 con 0x7f5aac004d40 2024-04-26T23:02:46.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5aac00b9c0 con 0x7f5ac815f4f0 2024-04-26T23:02:46.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.627+0000 7f5ac6000640 1 --2- 172.21.15.40:0/2893729146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aac003f30 0x7f5aac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:46.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.628+0000 7f5ac6a00640 1 --2- 172.21.15.40:0/2893729146 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ac815f4f0 0x7f5aac0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:46.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.628+0000 7f5ac7400640 1 --2- 172.21.15.40:0/2893729146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aac004d40 0x7f5aac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:46.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.628+0000 7f5ac6000640 1 -- 172.21.15.40:0/2893729146 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ac815f4f0 msgr2=0x7f5aac0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:46.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.628+0000 7f5ac6000640 1 --2- 172.21.15.40:0/2893729146 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ac815f4f0 0x7f5aac0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:46.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.628+0000 7f5ac6000640 1 -- 172.21.15.40:0/2893729146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aac004d40 msgr2=0x7f5aac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:46.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.628+0000 7f5ac6000640 1 --2- 172.21.15.40:0/2893729146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aac004d40 0x7f5aac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:46.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.628+0000 7f5ac6000640 1 -- 172.21.15.40:0/2893729146 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aac003520 con 0x7f5aac003f30 2024-04-26T23:02:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.629+0000 7f5ac6000640 1 --2- 172.21.15.40:0/2893729146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aac003f30 0x7f5aac004800 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0009ac0 tx=0x7f5aac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cd37fed0fb0b52 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.629+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0036070 con 0x7f5aac003f30 2024-04-26T23:02:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.629+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab0031400 con 0x7f5aac003f30 2024-04-26T23:02:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.629+0000 7f5ac5600640 1 -- 172.21.15.40:0/2893729146 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0009690 con 0x7f5aac003f30 2024-04-26T23:02:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.630+0000 7f5ac7e00640 1 -- 172.21.15.40:0/2893729146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aac003f30 msgr2=0x7f5aac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.630+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/2893729146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aac003f30 0x7f5aac004800 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0009ac0 tx=0x7f5aac005830 comp rx=0 tx=0).stop 2024-04-26T23:02:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.630+0000 7f5ac7400640 1 -- 172.21.15.40:0/2893729146 reap_dead start 2024-04-26T23:02:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.630+0000 7f5ac7e00640 1 -- 172.21.15.40:0/2893729146 shutdown_connections 2024-04-26T23:02:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.630+0000 7f5ac7e00640 1 -- 172.21.15.40:0/2893729146 >> 172.21.15.40:0/2893729146 conn(0x7f5ac809e210 msgr2=0x7f5ac80d4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.630+0000 7f5ac7e00640 1 -- 172.21.15.40:0/2893729146 shutdown_connections 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.630+0000 7f5ac7e00640 1 -- 172.21.15.40:0/2893729146 wait complete. 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.632+0000 7f5ac7e00640 1 Processor -- start 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.632+0000 7f5ac7e00640 1 -- start start 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.633+0000 7f5ac7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aac003f30 0x7f5ac815c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.633+0000 7f5ac7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ac809ff00 0x7f5ac80cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.633+0000 7f5ac7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ac80cb6d0 0x7f5ac80ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.633+0000 7f5ac7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac80d2fa0 con 0x7f5ac80cb6d0 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.633+0000 7f5ac7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac80b8970 con 0x7f5aac003f30 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.633+0000 7f5ac7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac80d0c30 con 0x7f5ac809ff00 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.633+0000 7f5ac7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ac80cb6d0 0x7f5ac80ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.633+0000 7f5ac7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ac80cb6d0 0x7f5ac80ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42542/0 (socket says 172.21.15.40:42542) 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.633+0000 7f5ac7400640 1 -- 172.21.15.40:0/423650649 learned_addr learned my addr 172.21.15.40:0/423650649 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.634+0000 7f5ac6000640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ac809ff00 0x7f5ac80cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.634+0000 7f5ac6a00640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aac003f30 0x7f5ac815c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:46.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.634+0000 7f5ac7400640 1 -- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ac809ff00 msgr2=0x7f5ac80cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:46.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.634+0000 7f5ac7400640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ac809ff00 0x7f5ac80cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:46.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.634+0000 7f5ac7400640 1 -- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aac003f30 msgr2=0x7f5ac815c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:46.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.634+0000 7f5ac7400640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aac003f30 0x7f5ac815c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:46.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.634+0000 7f5ac7400640 1 -- 172.21.15.40:0/423650649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac80ccda0 con 0x7f5ac80cb6d0 2024-04-26T23:02:46.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.634+0000 7f5ac7400640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ac80cb6d0 0x7f5ac80ce420 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8162e90 tx=0x7f5ac0011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:46.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.634+0000 7f5abea00640 1 -- 172.21.15.40:0/423650649 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0019020 con 0x7f5ac80cb6d0 2024-04-26T23:02:46.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.635+0000 7f5abea00640 1 -- 172.21.15.40:0/423650649 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac0003610 con 0x7f5ac80cb6d0 2024-04-26T23:02:46.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.635+0000 7f5ac7e00640 1 -- 172.21.15.40:0/423650649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac80cd080 con 0x7f5ac80cb6d0 2024-04-26T23:02:46.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.635+0000 7f5abea00640 1 -- 172.21.15.40:0/423650649 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0003a40 con 0x7f5ac80cb6d0 2024-04-26T23:02:46.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.635+0000 7f5ac7e00640 1 -- 172.21.15.40:0/423650649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac80cc990 con 0x7f5ac80cb6d0 2024-04-26T23:02:46.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.636+0000 7f5ac7e00640 1 -- 172.21.15.40:0/423650649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac8069370 con 0x7f5ac80cb6d0 2024-04-26T23:02:46.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.637+0000 7f5abea00640 1 -- 172.21.15.40:0/423650649 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5ac001c020 con 0x7f5ac80cb6d0 2024-04-26T23:02:46.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.638+0000 7f5abea00640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5aa406c8f0 0x7f5aa406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:46.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.638+0000 7f5abea00640 1 -- 172.21.15.40:0/423650649 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5619+0+0 (secure 0 0 0) 0x7f5ac008be30 con 0x7f5ac80cb6d0 2024-04-26T23:02:46.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.639+0000 7f5ac6a00640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5aa406c8f0 0x7f5aa406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:46.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.640+0000 7f5ac6a00640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5aa406c8f0 0x7f5aa406edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80013d0 tx=0x7f5ab8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:46.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.643+0000 7f5abea00640 1 -- 172.21.15.40:0/423650649 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5ac0023030 con 0x7f5ac80cb6d0 2024-04-26T23:02:46.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:46.854+0000 7f5ac7e00640 1 -- 172.21.15.40:0/423650649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f5ac80d2420 con 0x7f5ac80cb6d0 2024-04-26T23:02:47.272 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:02:47.281 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:02:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.387+0000 7f5abea00640 1 -- 172.21.15.40:0/423650649 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) ==== 170+0+0 (secure 0 0 0) 0x7f5ac005a2a0 con 0x7f5ac80cb6d0 2024-04-26T23:02:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-04-26T23:02:47.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.392+0000 7f5ac7e00640 1 -- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5aa406c8f0 msgr2=0x7f5aa406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:47.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.392+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5aa406c8f0 0x7f5aa406edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80013d0 tx=0x7f5ab8009040 comp rx=0 tx=0).stop 2024-04-26T23:02:47.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.392+0000 7f5ac7e00640 1 -- 172.21.15.40:0/423650649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ac80cb6d0 msgr2=0x7f5ac80ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:47.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.392+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ac80cb6d0 0x7f5ac80ce420 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8162e90 tx=0x7f5ac0011920 comp rx=0 tx=0).stop 2024-04-26T23:02:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.392+0000 7f5ac7e00640 1 -- 172.21.15.40:0/423650649 shutdown_connections 2024-04-26T23:02:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.392+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5aa406c8f0 0x7f5aa406edb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.392+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ac80cb6d0 0x7f5ac80ce420 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.392+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ac809ff00 0x7f5ac80cdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.392+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/423650649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aac003f30 0x7f5ac815c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.392+0000 7f5ac7e00640 1 -- 172.21.15.40:0/423650649 >> 172.21.15.40:0/423650649 conn(0x7f5ac809e210 msgr2=0x7f5ac8161cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.392+0000 7f5ac7e00640 1 -- 172.21.15.40:0/423650649 shutdown_connections 2024-04-26T23:02:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.392+0000 7f5ac7e00640 1 -- 172.21.15.40:0/423650649 wait complete. 2024-04-26T23:02:47.429 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T23:02:47.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.660+0000 7fbc7be00640 1 Processor -- start 2024-04-26T23:02:47.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.660+0000 7fbc7be00640 1 -- start start 2024-04-26T23:02:47.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.661+0000 7fbc7be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740cc480 0x7fbc74072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:47.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.661+0000 7fbc7be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbc74072780 0x7fbc74166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.661+0000 7fbc7be00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc7415d7c0 con 0x7fbc74071e60 2024-04-26T23:02:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.661+0000 7fbc7be00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbc7415daa0 con 0x7fbc74072780 2024-04-26T23:02:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.661+0000 7fbc7be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7415dd80 con 0x7fbc740cc480 2024-04-26T23:02:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.662+0000 7fbc79400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbc74071e60 0x7fbc7406f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49888/0 (socket says 172.21.15.40:49888) 2024-04-26T23:02:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.662+0000 7fbc79400640 1 -- 172.21.15.40:0/1768461092 learned_addr learned my addr 172.21.15.40:0/1768461092 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.662+0000 7fbc78a00640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740cc480 0x7fbc74072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.662+0000 7fbc79e00640 1 --2- 172.21.15.40:0/1768461092 >> v2:172.21.15.103:3300/0 conn(0x7fbc74072780 0x7fbc74166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.662+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3495140679 0 0) 0x7fbc7415d7c0 con 0x7fbc74071e60 2024-04-26T23:02:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.662+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc58003520 con 0x7fbc74071e60 2024-04-26T23:02:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3126030722 0 0) 0x7fbc58003520 con 0x7fbc74071e60 2024-04-26T23:02:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740cc480 msgr2=0x7fbc74072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740cc480 0x7fbc74072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 >> v2:172.21.15.103:3300/0 conn(0x7fbc74072780 msgr2=0x7fbc74166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 --2- 172.21.15.40:0/1768461092 >> v2:172.21.15.103:3300/0 conn(0x7fbc74072780 0x7fbc74166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7415e060 con 0x7fbc74071e60 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fbc640019f0 con 0x7fbc74071e60 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 >> v1:172.21.15.40:6789/0 conn(0x7fbc74071e60 legacy=0x7fbc7406f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740cc480 0x7fbc580039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc58003f30 0x7fbc58004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc58004d40 0x7fbc580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7415daa0 con 0x7fbc58004d40 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7415dd80 con 0x7fbc58003f30 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbc5800b9c0 con 0x7fbc740cc480 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc79e00640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc58004d40 0x7fbc580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc79400640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740cc480 0x7fbc580039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc78a00640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc58003f30 0x7fbc58004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc79e00640 1 -- 172.21.15.40:0/1768461092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740cc480 msgr2=0x7fbc580039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc79e00640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740cc480 0x7fbc580039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.663+0000 7fbc79e00640 1 -- 172.21.15.40:0/1768461092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc58003f30 msgr2=0x7fbc58004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.664+0000 7fbc79e00640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc58003f30 0x7fbc58004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.664+0000 7fbc79e00640 1 -- 172.21.15.40:0/1768461092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc58003520 con 0x7fbc58004d40 2024-04-26T23:02:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.664+0000 7fbc79400640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740cc480 0x7fbc580039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.664+0000 7fbc78a00640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc58003f30 0x7fbc58004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.665+0000 7fbc79e00640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc58004d40 0x7fbc580051c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009ac0 tx=0x7fbc58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89fdaa16b1945e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.665+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c036070 con 0x7fbc58004d40 2024-04-26T23:02:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.665+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc6c031400 con 0x7fbc58004d40 2024-04-26T23:02:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.665+0000 7fbc73e00640 1 -- 172.21.15.40:0/1768461092 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c009690 con 0x7fbc58004d40 2024-04-26T23:02:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.665+0000 7fbc7be00640 1 -- 172.21.15.40:0/1768461092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc58004d40 msgr2=0x7fbc580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.665+0000 7fbc7be00640 1 --2- 172.21.15.40:0/1768461092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc58004d40 0x7fbc580051c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009ac0 tx=0x7fbc58005830 comp rx=0 tx=0).stop 2024-04-26T23:02:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.665+0000 7fbc79e00640 1 -- 172.21.15.40:0/1768461092 reap_dead start 2024-04-26T23:02:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.666+0000 7fbc7be00640 1 -- 172.21.15.40:0/1768461092 shutdown_connections 2024-04-26T23:02:47.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.666+0000 7fbc7be00640 1 -- 172.21.15.40:0/1768461092 >> 172.21.15.40:0/1768461092 conn(0x7fbc740d0be0 msgr2=0x7fbc740d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:47.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.666+0000 7fbc7be00640 1 -- 172.21.15.40:0/1768461092 shutdown_connections 2024-04-26T23:02:47.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.666+0000 7fbc7be00640 1 -- 172.21.15.40:0/1768461092 wait complete. 2024-04-26T23:02:47.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.667+0000 7fbc7be00640 1 Processor -- start 2024-04-26T23:02:47.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.668+0000 7fbc7be00640 1 -- start start 2024-04-26T23:02:47.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.668+0000 7fbc7be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc58004d40 0x7fbc7414e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:47.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.668+0000 7fbc7be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740c9e80 0x7fbc740ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:47.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc7be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc741591b0 0x7fbc74159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:47.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc7be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7415d3f0 con 0x7fbc741591b0 2024-04-26T23:02:47.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc7be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbc74072c10 con 0x7fbc58004d40 2024-04-26T23:02:47.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc7be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7415d570 con 0x7fbc740c9e80 2024-04-26T23:02:47.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc79e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc741591b0 0x7fbc74159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:47.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc79e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc741591b0 0x7fbc74159630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42564/0 (socket says 172.21.15.40:42564) 2024-04-26T23:02:47.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc79e00640 1 -- 172.21.15.40:0/1929260131 learned_addr learned my addr 172.21.15.40:0/1929260131 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:47.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc79400640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc58004d40 0x7fbc7414e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:47.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc78a00640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740c9e80 0x7fbc740ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc79e00640 1 -- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740c9e80 msgr2=0x7fbc740ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc79e00640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740c9e80 0x7fbc740ca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc79e00640 1 -- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc58004d40 msgr2=0x7fbc7414e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc79e00640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc58004d40 0x7fbc7414e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.669+0000 7fbc79e00640 1 -- 172.21.15.40:0/1929260131 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc74159ea0 con 0x7fbc741591b0 2024-04-26T23:02:47.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.670+0000 7fbc79e00640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc741591b0 0x7fbc74159630 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009be0 tx=0x7fbc6c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:47.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.670+0000 7fbc71600640 1 -- 172.21.15.40:0/1929260131 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c036040 con 0x7fbc741591b0 2024-04-26T23:02:47.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.670+0000 7fbc71600640 1 -- 172.21.15.40:0/1929260131 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc6c041070 con 0x7fbc741591b0 2024-04-26T23:02:47.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.670+0000 7fbc7be00640 1 -- 172.21.15.40:0/1929260131 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7415a180 con 0x7fbc741591b0 2024-04-26T23:02:47.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.670+0000 7fbc71600640 1 -- 172.21.15.40:0/1929260131 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c003a90 con 0x7fbc741591b0 2024-04-26T23:02:47.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.670+0000 7fbc7be00640 1 -- 172.21.15.40:0/1929260131 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7415a740 con 0x7fbc741591b0 2024-04-26T23:02:47.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.671+0000 7fbc7be00640 1 -- 172.21.15.40:0/1929260131 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc740694f0 con 0x7fbc741591b0 2024-04-26T23:02:47.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.672+0000 7fbc71600640 1 -- 172.21.15.40:0/1929260131 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbc6c039070 con 0x7fbc741591b0 2024-04-26T23:02:47.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.673+0000 7fbc71600640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbc5406c8f0 0x7fbc5406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:47.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.674+0000 7fbc71600640 1 -- 172.21.15.40:0/1929260131 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6001+0+0 (secure 0 0 0) 0x7fbc6c0ab890 con 0x7fbc741591b0 2024-04-26T23:02:47.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.674+0000 7fbc79400640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbc5406c8f0 0x7fbc5406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:47.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.675+0000 7fbc79400640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbc5406c8f0 0x7fbc5406edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbc64001b60 tx=0x7fbc64009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:47.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.679+0000 7fbc71600640 1 -- 172.21.15.40:0/1929260131 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fbc6c079b80 con 0x7fbc741591b0 2024-04-26T23:02:47.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:47.923+0000 7fbc7be00640 1 -- 172.21.15.40:0/1929260131 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fbc74069d30 con 0x7fbc741591b0 2024-04-26T23:02:48.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.392+0000 7fbc71600640 1 -- 172.21.15.40:0/1929260131 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) ==== 172+0+0 (secure 0 0 0) 0x7fbc6c079480 con 0x7fbc741591b0 2024-04-26T23:02:48.392 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 5 and data pool 6 2024-04-26T23:02:48.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.395+0000 7fbc7be00640 1 -- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbc5406c8f0 msgr2=0x7fbc5406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:48.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.395+0000 7fbc7be00640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbc5406c8f0 0x7fbc5406edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbc64001b60 tx=0x7fbc64009040 comp rx=0 tx=0).stop 2024-04-26T23:02:48.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.395+0000 7fbc7be00640 1 -- 172.21.15.40:0/1929260131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc741591b0 msgr2=0x7fbc74159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:48.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.395+0000 7fbc7be00640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc741591b0 0x7fbc74159630 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009be0 tx=0x7fbc6c004730 comp rx=0 tx=0).stop 2024-04-26T23:02:48.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.396+0000 7fbc7be00640 1 -- 172.21.15.40:0/1929260131 shutdown_connections 2024-04-26T23:02:48.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.396+0000 7fbc7be00640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbc5406c8f0 0x7fbc5406edb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.396+0000 7fbc7be00640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc741591b0 0x7fbc74159630 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.396+0000 7fbc7be00640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740c9e80 0x7fbc740ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.396+0000 7fbc7be00640 1 --2- 172.21.15.40:0/1929260131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc58004d40 0x7fbc7414e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.396+0000 7fbc7be00640 1 -- 172.21.15.40:0/1929260131 >> 172.21.15.40:0/1929260131 conn(0x7fbc740d0be0 msgr2=0x7fbc7415fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:48.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.396+0000 7fbc7be00640 1 -- 172.21.15.40:0/1929260131 shutdown_connections 2024-04-26T23:02:48.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.397+0000 7fbc7be00640 1 -- 172.21.15.40:0/1929260131 wait complete. 2024-04-26T23:02:48.435 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:48.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.607+0000 7f73df200640 1 Processor -- start 2024-04-26T23:02:48.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.607+0000 7f73df200640 1 -- start start 2024-04-26T23:02:48.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.608+0000 7f73df200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73d80c72d0 0x7f73d8167670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:48.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.608+0000 7f73df200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f73d8167bb0 0x7f73d80ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:48.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.608+0000 7f73df200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73d8163810 con 0x7f73d8163430 2024-04-26T23:02:48.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.608+0000 7f73df200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f73d80cb400 con 0x7f73d8167bb0 2024-04-26T23:02:48.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.608+0000 7f73df200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f73d80ccc60 con 0x7f73d80c72d0 2024-04-26T23:02:48.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.609+0000 7f73dde00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f73d8163430 0x7f73d8166f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49900/0 (socket says 172.21.15.40:49900) 2024-04-26T23:02:48.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.609+0000 7f73dde00640 1 -- 172.21.15.40:0/3157009629 learned_addr learned my addr 172.21.15.40:0/3157009629 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:48.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.609+0000 7f73dd400640 1 --2- 172.21.15.40:0/3157009629 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73d80c72d0 0x7f73d8167670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:48.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.609+0000 7f73de800640 1 --2- 172.21.15.40:0/3157009629 >> v2:172.21.15.103:3300/0 conn(0x7f73d8167bb0 0x7f73d80ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:48.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.609+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1362448523 0 0) 0x7f73d8163810 con 0x7f73d8163430 2024-04-26T23:02:48.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.609+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73c0003520 con 0x7f73d8163430 2024-04-26T23:02:48.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1879023748 0 0) 0x7f73c0003520 con 0x7f73d8163430 2024-04-26T23:02:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73d80c72d0 msgr2=0x7f73d8167670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 --2- 172.21.15.40:0/3157009629 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73d80c72d0 0x7f73d8167670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 >> v2:172.21.15.103:3300/0 conn(0x7f73d8167bb0 msgr2=0x7f73d80ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 --2- 172.21.15.40:0/3157009629 >> v2:172.21.15.103:3300/0 conn(0x7f73d8167bb0 0x7f73d80ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d80cb100 con 0x7f73d8163430 2024-04-26T23:02:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f73cc001670 con 0x7f73d8163430 2024-04-26T23:02:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 >> v1:172.21.15.40:6789/0 conn(0x7f73d8163430 legacy=0x7f73d8166f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 --2- 172.21.15.40:0/3157009629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73c00039b0 0x7f73c0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 --2- 172.21.15.40:0/3157009629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73c0004800 0x7f73c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 --2- 172.21.15.40:0/3157009629 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73d80c72d0 0x7f73c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:48.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f73d80cb400 con 0x7f73c0004800 2024-04-26T23:02:48.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f73d80ccc60 con 0x7f73c00039b0 2024-04-26T23:02:48.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f73c000b9c0 con 0x7f73d80c72d0 2024-04-26T23:02:48.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.610+0000 7f73dd400640 1 --2- 172.21.15.40:0/3157009629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73c0004800 0x7f73c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:48.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.611+0000 7f73de800640 1 --2- 172.21.15.40:0/3157009629 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73d80c72d0 0x7f73c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:48.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.611+0000 7f73dde00640 1 --2- 172.21.15.40:0/3157009629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73c00039b0 0x7f73c0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:48.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.611+0000 7f73dd400640 1 -- 172.21.15.40:0/3157009629 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73d80c72d0 msgr2=0x7f73c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:48.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.611+0000 7f73dd400640 1 --2- 172.21.15.40:0/3157009629 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73d80c72d0 0x7f73c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.611+0000 7f73dd400640 1 -- 172.21.15.40:0/3157009629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73c00039b0 msgr2=0x7f73c0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:48.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.611+0000 7f73dd400640 1 --2- 172.21.15.40:0/3157009629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73c00039b0 0x7f73c0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.611+0000 7f73dd400640 1 -- 172.21.15.40:0/3157009629 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c0003520 con 0x7f73c0004800 2024-04-26T23:02:48.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.612+0000 7f73dd400640 1 --2- 172.21.15.40:0/3157009629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73c0004800 0x7f73c0004c80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f73c8009a90 tx=0x7f73c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdae4419213bddb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:48.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.612+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c8036070 con 0x7f73c0004800 2024-04-26T23:02:48.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.612+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73c8031400 con 0x7f73c0004800 2024-04-26T23:02:48.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.612+0000 7f73dca00640 1 -- 172.21.15.40:0/3157009629 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c8009690 con 0x7f73c0004800 2024-04-26T23:02:48.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.613+0000 7f73df200640 1 -- 172.21.15.40:0/3157009629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73c0004800 msgr2=0x7f73c0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:48.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.613+0000 7f73df200640 1 --2- 172.21.15.40:0/3157009629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73c0004800 0x7f73c0004c80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f73c8009a90 tx=0x7f73c0005830 comp rx=0 tx=0).stop 2024-04-26T23:02:48.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.613+0000 7f73de800640 1 -- 172.21.15.40:0/3157009629 reap_dead start 2024-04-26T23:02:48.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.613+0000 7f73df200640 1 -- 172.21.15.40:0/3157009629 shutdown_connections 2024-04-26T23:02:48.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.613+0000 7f73df200640 1 -- 172.21.15.40:0/3157009629 >> 172.21.15.40:0/3157009629 conn(0x7f73d80d0a10 msgr2=0x7f73d80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:48.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.613+0000 7f73df200640 1 -- 172.21.15.40:0/3157009629 shutdown_connections 2024-04-26T23:02:48.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.613+0000 7f73df200640 1 -- 172.21.15.40:0/3157009629 wait complete. 2024-04-26T23:02:48.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.615+0000 7f73df200640 1 Processor -- start 2024-04-26T23:02:48.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.615+0000 7f73df200640 1 -- start start 2024-04-26T23:02:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.616+0000 7f73df200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73c0004800 0x7f73d815e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.616+0000 7f73df200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80b3650 0x7f73d8164810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.616+0000 7f73df200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73d815f550 0x7f73d80b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.616+0000 7f73df200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f73d80cd1e0 con 0x7f73d815f550 2024-04-26T23:02:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.616+0000 7f73df200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f73d80afe80 con 0x7f73d80b3650 2024-04-26T23:02:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.616+0000 7f73df200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f73d8167670 con 0x7f73c0004800 2024-04-26T23:02:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.616+0000 7f73de800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73d815f550 0x7f73d80b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.616+0000 7f73de800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73d815f550 0x7f73d80b3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42568/0 (socket says 172.21.15.40:42568) 2024-04-26T23:02:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.616+0000 7f73de800640 1 -- 172.21.15.40:0/1201060726 learned_addr learned my addr 172.21.15.40:0/1201060726 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73dd400640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80b3650 0x7f73d8164810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73dde00640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73c0004800 0x7f73d815e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73de800640 1 -- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73c0004800 msgr2=0x7f73d815e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73de800640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73c0004800 0x7f73d815e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73de800640 1 -- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80b3650 msgr2=0x7f73d8164810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73de800640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80b3650 0x7f73d8164810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73de800640 1 -- 172.21.15.40:0/1201060726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d80b3b30 con 0x7f73d815f550 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73dde00640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73c0004800 0x7f73d815e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73dd400640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80b3650 0x7f73d8164810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73de800640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73d815f550 0x7f73d80b3e10 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f73d80b6b80 tx=0x7f73d40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73d2000640 1 -- 172.21.15.40:0/1201060726 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4011e40 con 0x7f73d815f550 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73d2000640 1 -- 172.21.15.40:0/1201060726 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d401b070 con 0x7f73d815f550 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73d2000640 1 -- 172.21.15.40:0/1201060726 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4003990 con 0x7f73d815f550 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73df200640 1 -- 172.21.15.40:0/1201060726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73d80b61b0 con 0x7f73d815f550 2024-04-26T23:02:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.617+0000 7f73df200640 1 -- 172.21.15.40:0/1201060726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73d815ebd0 con 0x7f73d815f550 2024-04-26T23:02:48.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.618+0000 7f73df200640 1 -- 172.21.15.40:0/1201060726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73d80b67c0 con 0x7f73d815f550 2024-04-26T23:02:48.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.620+0000 7f73d2000640 1 -- 172.21.15.40:0/1201060726 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f73d4005d10 con 0x7f73d815f550 2024-04-26T23:02:48.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.621+0000 7f73d2000640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f73bc06c8f0 0x7f73bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:48.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.621+0000 7f73d2000640 1 -- 172.21.15.40:0/1201060726 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f73d408c3d0 con 0x7f73d815f550 2024-04-26T23:02:48.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.622+0000 7f73dde00640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f73bc06c8f0 0x7f73bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.623+0000 7f73dde00640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f73bc06c8f0 0x7f73bc06edb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f73cc001eb0 tx=0x7f73cc001ba0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:48.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.626+0000 7f73d2000640 1 -- 172.21.15.40:0/1201060726 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f73d4025020 con 0x7f73d815f550 2024-04-26T23:02:48.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.829+0000 7f73df200640 1 -- 172.21.15.40:0/1201060726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73d8069370 con 0x7f73d815f550 2024-04-26T23:02:48.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.832+0000 7f73d2000640 1 -- 172.21.15.40:0/1201060726 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16919 (secure 0 0 0) 0x7f73d4014350 con 0x7f73d815f550 2024-04-26T23:02:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:48.833 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":38,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:02:47.924769+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:02:45.864120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-26T23:02:46.869290+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.834+0000 7f73df200640 1 -- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f73bc06c8f0 msgr2=0x7f73bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.834+0000 7f73df200640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f73bc06c8f0 0x7f73bc06edb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f73cc001eb0 tx=0x7f73cc001ba0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.835+0000 7f73df200640 1 -- 172.21.15.40:0/1201060726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73d815f550 msgr2=0x7f73d80b3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.835+0000 7f73df200640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73d815f550 0x7f73d80b3e10 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f73d80b6b80 tx=0x7f73d40115a0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.835+0000 7f73df200640 1 -- 172.21.15.40:0/1201060726 shutdown_connections 2024-04-26T23:02:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.835+0000 7f73df200640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f73bc06c8f0 0x7f73bc06edb0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.835+0000 7f73df200640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73d815f550 0x7f73d80b3e10 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.835+0000 7f73df200640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80b3650 0x7f73d8164810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.835+0000 7f73df200640 1 --2- 172.21.15.40:0/1201060726 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73c0004800 0x7f73d815e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:48.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.835+0000 7f73df200640 1 -- 172.21.15.40:0/1201060726 >> 172.21.15.40:0/1201060726 conn(0x7f73d80d0a10 msgr2=0x7f73d80b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:48.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.835+0000 7f73df200640 1 -- 172.21.15.40:0/1201060726 shutdown_connections 2024-04-26T23:02:48.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:48.835+0000 7f73df200640 1 -- 172.21.15.40:0/1201060726 wait complete. 2024-04-26T23:02:48.855 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:49.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.069+0000 7f8a77200640 1 Processor -- start 2024-04-26T23:02:49.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.069+0000 7f8a77200640 1 -- start start 2024-04-26T23:02:49.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.070+0000 7f8a77200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780b6220 0x7f8a780d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.070+0000 7f8a77200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a78162c40 con 0x7f8a780d39f0 2024-04-26T23:02:49.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.070+0000 7f8a77200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a780d2d80 con 0x7f8a780d08d0 2024-04-26T23:02:49.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.070+0000 7f8a77200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a780b7c80 con 0x7f8a780b6220 2024-04-26T23:02:49.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.070+0000 7f8a76800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8a780d39f0 0x7f8a780b8d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49916/0 (socket says 172.21.15.40:49916) 2024-04-26T23:02:49.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.070+0000 7f8a76800640 1 -- 172.21.15.40:0/1232025106 learned_addr learned my addr 172.21.15.40:0/1232025106 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:49.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.071+0000 7f8a75e00640 1 --2- 172.21.15.40:0/1232025106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780b6220 0x7f8a780d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.071+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750642550 0 0) 0x7f8a78162c40 con 0x7f8a780d39f0 2024-04-26T23:02:49.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.071+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a58003520 con 0x7f8a780d39f0 2024-04-26T23:02:49.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.071+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2069475026 0 0) 0x7f8a58003520 con 0x7f8a780d39f0 2024-04-26T23:02:49.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.071+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780b6220 msgr2=0x7f8a780d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:49.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.071+0000 7f8a74a00640 1 --2- 172.21.15.40:0/1232025106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780b6220 0x7f8a780d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.071+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 >> v1:172.21.15.103:6789/0 conn(0x7f8a780d08d0 legacy=0x7f8a780d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:49.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.071+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a780b7f60 con 0x7f8a780d39f0 2024-04-26T23:02:49.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.071+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f8a70001670 con 0x7f8a780d39f0 2024-04-26T23:02:49.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.071+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 >> v1:172.21.15.40:6789/0 conn(0x7f8a780d39f0 legacy=0x7f8a780b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:49.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.071+0000 7f8a74a00640 1 --2- 172.21.15.40:0/1232025106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a580039e0 0x7f8a58003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.071+0000 7f8a74a00640 1 --2- 172.21.15.40:0/1232025106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a58004800 0x7f8a58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.072+0000 7f8a74a00640 1 --2- 172.21.15.40:0/1232025106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780b6220 0x7f8a580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.072+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a780b7c80 con 0x7f8a58004800 2024-04-26T23:02:49.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.072+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5800ba70 con 0x7f8a580039e0 2024-04-26T23:02:49.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.072+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a5800bc70 con 0x7f8a780b6220 2024-04-26T23:02:49.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.072+0000 7f8a75400640 1 --2- 172.21.15.40:0/1232025106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a58004800 0x7f8a58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.072+0000 7f8a75e00640 1 --2- 172.21.15.40:0/1232025106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a580039e0 0x7f8a58003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.072+0000 7f8a76800640 1 --2- 172.21.15.40:0/1232025106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780b6220 0x7f8a580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.072+0000 7f8a75400640 1 -- 172.21.15.40:0/1232025106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780b6220 msgr2=0x7f8a580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:49.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.072+0000 7f8a75400640 1 --2- 172.21.15.40:0/1232025106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780b6220 0x7f8a580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.072+0000 7f8a75400640 1 -- 172.21.15.40:0/1232025106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a580039e0 msgr2=0x7f8a58003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:49.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.072+0000 7f8a75400640 1 --2- 172.21.15.40:0/1232025106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a580039e0 0x7f8a58003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.072+0000 7f8a75400640 1 -- 172.21.15.40:0/1232025106 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a58003520 con 0x7f8a58004800 2024-04-26T23:02:49.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.072+0000 7f8a76800640 1 --2- 172.21.15.40:0/1232025106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780b6220 0x7f8a580051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:49.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.073+0000 7f8a75400640 1 --2- 172.21.15.40:0/1232025106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a58004800 0x7f8a58004c80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8a60001e20 tx=0x7f8a58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442bae150eb65d98 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:49.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.074+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60001a70 con 0x7f8a58004800 2024-04-26T23:02:49.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.074+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a6002ae10 con 0x7f8a58004800 2024-04-26T23:02:49.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.074+0000 7f8a74a00640 1 -- 172.21.15.40:0/1232025106 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60003bb0 con 0x7f8a58004800 2024-04-26T23:02:49.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.074+0000 7f8a77200640 1 -- 172.21.15.40:0/1232025106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a58004800 msgr2=0x7f8a58004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:49.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.074+0000 7f8a77200640 1 --2- 172.21.15.40:0/1232025106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a58004800 0x7f8a58004c80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8a60001e20 tx=0x7f8a58005830 comp rx=0 tx=0).stop 2024-04-26T23:02:49.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.074+0000 7f8a76800640 1 -- 172.21.15.40:0/1232025106 reap_dead start 2024-04-26T23:02:49.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.074+0000 7f8a77200640 1 -- 172.21.15.40:0/1232025106 shutdown_connections 2024-04-26T23:02:49.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.074+0000 7f8a77200640 1 -- 172.21.15.40:0/1232025106 >> 172.21.15.40:0/1232025106 conn(0x7f8a780d4d90 msgr2=0x7f8a780d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:49.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.074+0000 7f8a77200640 1 -- 172.21.15.40:0/1232025106 shutdown_connections 2024-04-26T23:02:49.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.074+0000 7f8a77200640 1 -- 172.21.15.40:0/1232025106 wait complete. 2024-04-26T23:02:49.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.076+0000 7f8a77200640 1 Processor -- start 2024-04-26T23:02:49.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.076+0000 7f8a77200640 1 -- start start 2024-04-26T23:02:49.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.077+0000 7f8a77200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a58004800 0x7f8a78159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.077+0000 7f8a77200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780a2540 0x7f8a7815d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.077+0000 7f8a77200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a7815acb0 0x7f8a780cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.077+0000 7f8a77200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8a780d0390 con 0x7f8a58004800 2024-04-26T23:02:49.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.077+0000 7f8a77200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a780b8520 con 0x7f8a7815acb0 2024-04-26T23:02:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.077+0000 7f8a77200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a780d0510 con 0x7f8a780a2540 2024-04-26T23:02:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.077+0000 7f8a75e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a58004800 0x7f8a78159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.078+0000 7f8a75e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a58004800 0x7f8a78159b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42584/0 (socket says 172.21.15.40:42584) 2024-04-26T23:02:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.078+0000 7f8a75e00640 1 -- 172.21.15.40:0/3993979039 learned_addr learned my addr 172.21.15.40:0/3993979039 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.078+0000 7f8a76800640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a7815acb0 0x7f8a780cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.078+0000 7f8a75400640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780a2540 0x7f8a7815d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.078+0000 7f8a75e00640 1 -- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780a2540 msgr2=0x7f8a7815d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.078+0000 7f8a75e00640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780a2540 0x7f8a7815d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.078+0000 7f8a75e00640 1 -- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a7815acb0 msgr2=0x7f8a780cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.078+0000 7f8a75e00640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a7815acb0 0x7f8a780cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.078+0000 7f8a75e00640 1 -- 172.21.15.40:0/3993979039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7815e050 con 0x7f8a58004800 2024-04-26T23:02:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.078+0000 7f8a76800640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a7815acb0 0x7f8a780cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.078+0000 7f8a75400640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780a2540 0x7f8a7815d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:49.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.078+0000 7f8a75e00640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a58004800 0x7f8a78159b40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8a68011600 tx=0x7f8a68011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:49.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.079+0000 7f8a6e000640 1 -- 172.21.15.40:0/3993979039 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800f920 con 0x7f8a58004800 2024-04-26T23:02:49.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.079+0000 7f8a6e000640 1 -- 172.21.15.40:0/3993979039 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a6800fac0 con 0x7f8a58004800 2024-04-26T23:02:49.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.079+0000 7f8a6e000640 1 -- 172.21.15.40:0/3993979039 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800fe70 con 0x7f8a58004800 2024-04-26T23:02:49.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.079+0000 7f8a77200640 1 -- 172.21.15.40:0/3993979039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a7815e330 con 0x7f8a58004800 2024-04-26T23:02:49.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.079+0000 7f8a77200640 1 -- 172.21.15.40:0/3993979039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a78159620 con 0x7f8a58004800 2024-04-26T23:02:49.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.080+0000 7f8a77200640 1 -- 172.21.15.40:0/3993979039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a78069370 con 0x7f8a58004800 2024-04-26T23:02:49.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.081+0000 7f8a6e000640 1 -- 172.21.15.40:0/3993979039 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8a68015460 con 0x7f8a58004800 2024-04-26T23:02:49.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.082+0000 7f8a6e000640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8a5406c8f0 0x7f8a5406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.082+0000 7f8a6e000640 1 -- 172.21.15.40:0/3993979039 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f8a6808c6c0 con 0x7f8a58004800 2024-04-26T23:02:49.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.083+0000 7f8a75400640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8a5406c8f0 0x7f8a5406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.084+0000 7f8a75400640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8a5406c8f0 0x7f8a5406edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8a60032000 tx=0x7f8a6002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:49.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.086+0000 7f8a6e000640 1 -- 172.21.15.40:0/3993979039 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8a68019030 con 0x7f8a58004800 2024-04-26T23:02:49.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.292+0000 7f8a77200640 1 -- 172.21.15.40:0/3993979039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a780d30c0 con 0x7f8a58004800 2024-04-26T23:02:49.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.295+0000 7f8a6e000640 1 -- 172.21.15.40:0/3993979039 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16919 (secure 0 0 0) 0x7f8a6805a950 con 0x7f8a58004800 2024-04-26T23:02:49.295 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:49.296 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":38,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:02:47.924769+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:02:45.864120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-26T23:02:46.869290+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:49.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.297+0000 7f8a77200640 1 -- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8a5406c8f0 msgr2=0x7f8a5406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:49.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.297+0000 7f8a77200640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8a5406c8f0 0x7f8a5406edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8a60032000 tx=0x7f8a6002e000 comp rx=0 tx=0).stop 2024-04-26T23:02:49.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.297+0000 7f8a77200640 1 -- 172.21.15.40:0/3993979039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a58004800 msgr2=0x7f8a78159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:49.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.297+0000 7f8a77200640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a58004800 0x7f8a78159b40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8a68011600 tx=0x7f8a68011ac0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.297+0000 7f8a77200640 1 -- 172.21.15.40:0/3993979039 shutdown_connections 2024-04-26T23:02:49.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.297+0000 7f8a77200640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8a5406c8f0 0x7f8a5406edb0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.297+0000 7f8a77200640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a7815acb0 0x7f8a780cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.297+0000 7f8a77200640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a780a2540 0x7f8a7815d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.297+0000 7f8a77200640 1 --2- 172.21.15.40:0/3993979039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a58004800 0x7f8a78159b40 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.297+0000 7f8a77200640 1 -- 172.21.15.40:0/3993979039 >> 172.21.15.40:0/3993979039 conn(0x7f8a780d4d90 msgr2=0x7f8a7815fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:49.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.298+0000 7f8a77200640 1 -- 172.21.15.40:0/3993979039 shutdown_connections 2024-04-26T23:02:49.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.298+0000 7f8a77200640 1 -- 172.21.15.40:0/3993979039 wait complete. 2024-04-26T23:02:49.319 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T23:02:49.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.532+0000 7f4badc00640 1 Processor -- start 2024-04-26T23:02:49.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.533+0000 7f4badc00640 1 -- start start 2024-04-26T23:02:49.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.534+0000 7f4badc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ba80b5120 0x7f4ba80b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.534+0000 7f4badc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4ba80b5500 0x7f4ba80b6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.534+0000 7f4badc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ba80b59b0 0x7f4ba80b6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.534+0000 7f4badc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f4ba80d03b0 con 0x7f4ba80b5120 2024-04-26T23:02:49.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.534+0000 7f4badc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4ba80d3dc0 con 0x7f4ba80b5500 2024-04-26T23:02:49.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.534+0000 7f4badc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ba80b75b0 con 0x7f4ba80b59b0 2024-04-26T23:02:49.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.534+0000 7f4ba7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ba80b5120 0x7f4ba80b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.534+0000 7f4ba7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ba80b5120 0x7f4ba80b5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42596/0 (socket says 172.21.15.40:42596) 2024-04-26T23:02:49.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.534+0000 7f4ba7e00640 1 -- 172.21.15.40:0/826064398 learned_addr learned my addr 172.21.15.40:0/826064398 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:49.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.535+0000 7f4ba7400640 1 --2- 172.21.15.40:0/826064398 >> v2:172.21.15.103:3300/0 conn(0x7f4ba80b5500 0x7f4ba80b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.535+0000 7f4ba7e00640 1 -- 172.21.15.40:0/826064398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ba80b59b0 msgr2=0x7f4ba80b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:49.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.535+0000 7f4bad200640 1 --2- 172.21.15.40:0/826064398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ba80b59b0 0x7f4ba80b6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.535+0000 7f4ba7e00640 1 --2- 172.21.15.40:0/826064398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ba80b59b0 0x7f4ba80b6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.535+0000 7f4ba7e00640 1 -- 172.21.15.40:0/826064398 >> v2:172.21.15.103:3300/0 conn(0x7f4ba80b5500 msgr2=0x7f4ba80b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.535+0000 7f4ba7e00640 1 --2- 172.21.15.40:0/826064398 >> v2:172.21.15.103:3300/0 conn(0x7f4ba80b5500 0x7f4ba80b6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.535+0000 7f4ba7e00640 1 -- 172.21.15.40:0/826064398 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba80d0600 con 0x7f4ba80b5120 2024-04-26T23:02:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.535+0000 7f4ba7400640 1 --2- 172.21.15.40:0/826064398 >> v2:172.21.15.103:3300/0 conn(0x7f4ba80b5500 0x7f4ba80b6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.536+0000 7f4ba7e00640 1 --2- 172.21.15.40:0/826064398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ba80b5120 0x7f4ba80b5ee0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009e50 tx=0x7f4b9802a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b0944062ea4e91 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.536+0000 7f4ba6a00640 1 -- 172.21.15.40:0/826064398 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b98036070 con 0x7f4ba80b5120 2024-04-26T23:02:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.536+0000 7f4ba6a00640 1 -- 172.21.15.40:0/826064398 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b98031600 con 0x7f4ba80b5120 2024-04-26T23:02:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.536+0000 7f4ba6a00640 1 -- 172.21.15.40:0/826064398 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b980319d0 con 0x7f4ba80b5120 2024-04-26T23:02:49.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.537+0000 7f4badc00640 1 -- 172.21.15.40:0/826064398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ba80b5120 msgr2=0x7f4ba80b5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:49.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.537+0000 7f4badc00640 1 --2- 172.21.15.40:0/826064398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ba80b5120 0x7f4ba80b5ee0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009e50 tx=0x7f4b9802a640 comp rx=0 tx=0).stop 2024-04-26T23:02:49.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.537+0000 7f4badc00640 1 -- 172.21.15.40:0/826064398 shutdown_connections 2024-04-26T23:02:49.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.537+0000 7f4badc00640 1 --2- 172.21.15.40:0/826064398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ba80b59b0 0x7f4ba80b6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.537+0000 7f4badc00640 1 --2- 172.21.15.40:0/826064398 >> v2:172.21.15.103:3300/0 conn(0x7f4ba80b5500 0x7f4ba80b6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.537+0000 7f4badc00640 1 --2- 172.21.15.40:0/826064398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ba80b5120 0x7f4ba80b5ee0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.537+0000 7f4badc00640 1 -- 172.21.15.40:0/826064398 >> 172.21.15.40:0/826064398 conn(0x7f4ba80d4db0 msgr2=0x7f4ba80d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:49.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.537+0000 7f4badc00640 1 -- 172.21.15.40:0/826064398 shutdown_connections 2024-04-26T23:02:49.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.537+0000 7f4badc00640 1 -- 172.21.15.40:0/826064398 wait complete. 2024-04-26T23:02:49.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.539+0000 7f4badc00640 1 Processor -- start 2024-04-26T23:02:49.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.539+0000 7f4badc00640 1 -- start start 2024-04-26T23:02:49.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.540+0000 7f4badc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ba80b5120 0x7f4ba81f5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.540+0000 7f4badc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ba80b5500 0x7f4ba81f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.540+0000 7f4badc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ba80b59b0 0x7f4ba80cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.540+0000 7f4badc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba80b72b0 con 0x7f4ba80b59b0 2024-04-26T23:02:49.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.540+0000 7f4badc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba80d0bc0 con 0x7f4ba80b5120 2024-04-26T23:02:49.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.540+0000 7f4badc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ba80d2b00 con 0x7f4ba80b5500 2024-04-26T23:02:49.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.540+0000 7f4bad200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ba80b59b0 0x7f4ba80cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.540+0000 7f4bad200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ba80b59b0 0x7f4ba80cda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42602/0 (socket says 172.21.15.40:42602) 2024-04-26T23:02:49.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.540+0000 7f4bad200640 1 -- 172.21.15.40:0/104331073 learned_addr learned my addr 172.21.15.40:0/104331073 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:49.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.540+0000 7f4ba7400640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ba80b5500 0x7f4ba81f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.540+0000 7f4ba7e00640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ba80b5120 0x7f4ba81f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.541+0000 7f4bad200640 1 -- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ba80b5500 msgr2=0x7f4ba81f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:49.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.541+0000 7f4bad200640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ba80b5500 0x7f4ba81f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.541+0000 7f4bad200640 1 -- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ba80b5120 msgr2=0x7f4ba81f5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:49.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.541+0000 7f4bad200640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ba80b5120 0x7f4ba81f5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:49.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.541+0000 7f4bad200640 1 -- 172.21.15.40:0/104331073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba81f5b80 con 0x7f4ba80b59b0 2024-04-26T23:02:49.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.541+0000 7f4bad200640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ba80b59b0 0x7f4ba80cda40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000c550 tx=0x7f4ba000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:49.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.541+0000 7f4b9fe00640 1 -- 172.21.15.40:0/104331073 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0015070 con 0x7f4ba80b59b0 2024-04-26T23:02:49.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.541+0000 7f4badc00640 1 -- 172.21.15.40:0/104331073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ba81f25e0 con 0x7f4ba80b59b0 2024-04-26T23:02:49.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.541+0000 7f4b9fe00640 1 -- 172.21.15.40:0/104331073 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ba0004220 con 0x7f4ba80b59b0 2024-04-26T23:02:49.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.542+0000 7f4b9fe00640 1 -- 172.21.15.40:0/104331073 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0003610 con 0x7f4ba80b59b0 2024-04-26T23:02:49.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.542+0000 7f4badc00640 1 -- 172.21.15.40:0/104331073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ba80cc570 con 0x7f4ba80b59b0 2024-04-26T23:02:49.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.543+0000 7f4badc00640 1 -- 172.21.15.40:0/104331073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ba8069370 con 0x7f4ba80b59b0 2024-04-26T23:02:49.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.544+0000 7f4b9fe00640 1 -- 172.21.15.40:0/104331073 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4ba00037b0 con 0x7f4ba80b59b0 2024-04-26T23:02:49.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.545+0000 7f4b9fe00640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4b8406c8f0 0x7f4b8406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:49.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.545+0000 7f4b9fe00640 1 -- 172.21.15.40:0/104331073 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f4ba008bb70 con 0x7f4ba80b59b0 2024-04-26T23:02:49.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.546+0000 7f4ba7e00640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4b8406c8f0 0x7f4b8406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:49.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.547+0000 7f4ba7e00640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4b8406c8f0 0x7f4b8406edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4b98033000 tx=0x7f4b98009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:49.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.550+0000 7f4b9fe00640 1 -- 172.21.15.40:0/104331073 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4ba81f5b80 con 0x7f4ba80b59b0 2024-04-26T23:02:49.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:49.784+0000 7f4badc00640 1 -- 172.21.15.40:0/104331073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f4ba8161cd0 con 0x7f4ba80b59b0 2024-04-26T23:02:50.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.422+0000 7f4b9fe00640 1 -- 172.21.15.40:0/104331073 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f4ba0059cf0 con 0x7f4ba80b59b0 2024-04-26T23:02:50.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.426+0000 7f4badc00640 1 -- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4b8406c8f0 msgr2=0x7f4b8406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:50.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.426+0000 7f4badc00640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4b8406c8f0 0x7f4b8406edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4b98033000 tx=0x7f4b98009000 comp rx=0 tx=0).stop 2024-04-26T23:02:50.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.426+0000 7f4badc00640 1 -- 172.21.15.40:0/104331073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ba80b59b0 msgr2=0x7f4ba80cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:50.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.426+0000 7f4badc00640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ba80b59b0 0x7f4ba80cda40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000c550 tx=0x7f4ba000ca10 comp rx=0 tx=0).stop 2024-04-26T23:02:50.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.427+0000 7f4badc00640 1 -- 172.21.15.40:0/104331073 shutdown_connections 2024-04-26T23:02:50.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.427+0000 7f4badc00640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4b8406c8f0 0x7f4b8406edb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:50.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.427+0000 7f4badc00640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ba80b59b0 0x7f4ba80cda40 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:50.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.427+0000 7f4badc00640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ba80b5500 0x7f4ba81f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:50.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.427+0000 7f4badc00640 1 --2- 172.21.15.40:0/104331073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ba80b5120 0x7f4ba81f5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:50.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.427+0000 7f4badc00640 1 -- 172.21.15.40:0/104331073 >> 172.21.15.40:0/104331073 conn(0x7f4ba80d4db0 msgr2=0x7f4ba80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:50.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.427+0000 7f4badc00640 1 -- 172.21.15.40:0/104331073 shutdown_connections 2024-04-26T23:02:50.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.427+0000 7f4badc00640 1 -- 172.21.15.40:0/104331073 wait complete. 2024-04-26T23:02:50.452 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-26T23:02:50.452 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-26T23:02:50.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.672+0000 7ff44e200640 1 Processor -- start 2024-04-26T23:02:50.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.672+0000 7ff44e200640 1 -- start start 2024-04-26T23:02:50.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.674+0000 7ff44e200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4480c41d0 0x7ff4480cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:50.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.674+0000 7ff44e200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff448163a50 con 0x7ff4481678f0 2024-04-26T23:02:50.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.674+0000 7ff44e200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4480ceb90 con 0x7ff448073f30 2024-04-26T23:02:50.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.674+0000 7ff44e200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff4480cf7a0 con 0x7ff4480c41d0 2024-04-26T23:02:50.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.674+0000 7ff447e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff4481678f0 0x7ff44806ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49920/0 (socket says 172.21.15.40:49920) 2024-04-26T23:02:50.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.674+0000 7ff447e00640 1 -- 172.21.15.40:0/539620968 learned_addr learned my addr 172.21.15.40:0/539620968 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:50.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff447400640 1 --2- 172.21.15.40:0/539620968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4480c41d0 0x7ff4480cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:50.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4169690100 0 0) 0x7ff448163a50 con 0x7ff4481678f0 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff42c003520 con 0x7ff4481678f0 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4144012075 0 0) 0x7ff42c003520 con 0x7ff4481678f0 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4480c41d0 msgr2=0x7ff4480cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 --2- 172.21.15.40:0/539620968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4480c41d0 0x7ff4480cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 >> v1:172.21.15.103:6789/0 conn(0x7ff448073f30 legacy=0x7ff4481671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4480cee70 con 0x7ff4481678f0 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7ff4400019f0 con 0x7ff4481678f0 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 >> v1:172.21.15.40:6789/0 conn(0x7ff4481678f0 legacy=0x7ff44806ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 --2- 172.21.15.40:0/539620968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff42c0039e0 0x7ff42c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 --2- 172.21.15.40:0/539620968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c004800 0x7ff42c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 --2- 172.21.15.40:0/539620968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4480c41d0 0x7ff42c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff4480cf7a0 con 0x7ff42c004800 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c00ba70 con 0x7ff42c0039e0 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.675+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff42c00bc20 con 0x7ff4480c41d0 2024-04-26T23:02:50.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.676+0000 7ff43ea00640 1 --2- 172.21.15.40:0/539620968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c004800 0x7ff42c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:50.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.676+0000 7ff447400640 1 --2- 172.21.15.40:0/539620968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff42c0039e0 0x7ff42c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:50.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.676+0000 7ff43ea00640 1 -- 172.21.15.40:0/539620968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4480c41d0 msgr2=0x7ff42c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:50.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.676+0000 7ff43ea00640 1 --2- 172.21.15.40:0/539620968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4480c41d0 0x7ff42c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:50.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.676+0000 7ff43ea00640 1 -- 172.21.15.40:0/539620968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff42c0039e0 msgr2=0x7ff42c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:50.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.676+0000 7ff43ea00640 1 --2- 172.21.15.40:0/539620968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff42c0039e0 0x7ff42c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:50.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.676+0000 7ff43ea00640 1 -- 172.21.15.40:0/539620968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42c003520 con 0x7ff42c004800 2024-04-26T23:02:50.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.677+0000 7ff43ea00640 1 --2- 172.21.15.40:0/539620968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c004800 0x7ff42c004c80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff448074310 tx=0x7ff42c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d011e06f4b0a26d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:50.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.678+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430037070 con 0x7ff42c004800 2024-04-26T23:02:50.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.678+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff43002ab30 con 0x7ff42c004800 2024-04-26T23:02:50.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.678+0000 7ff446a00640 1 -- 172.21.15.40:0/539620968 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430001430 con 0x7ff42c004800 2024-04-26T23:02:50.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.678+0000 7ff44e200640 1 -- 172.21.15.40:0/539620968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c004800 msgr2=0x7ff42c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:50.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.678+0000 7ff44e200640 1 --2- 172.21.15.40:0/539620968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c004800 0x7ff42c004c80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff448074310 tx=0x7ff42c005830 comp rx=0 tx=0).stop 2024-04-26T23:02:50.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.678+0000 7ff447e00640 1 -- 172.21.15.40:0/539620968 reap_dead start 2024-04-26T23:02:50.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.678+0000 7ff44e200640 1 -- 172.21.15.40:0/539620968 shutdown_connections 2024-04-26T23:02:50.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.678+0000 7ff44e200640 1 -- 172.21.15.40:0/539620968 >> 172.21.15.40:0/539620968 conn(0x7ff4480d0be0 msgr2=0x7ff4480d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:50.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.678+0000 7ff44e200640 1 -- 172.21.15.40:0/539620968 shutdown_connections 2024-04-26T23:02:50.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.678+0000 7ff44e200640 1 -- 172.21.15.40:0/539620968 wait complete. 2024-04-26T23:02:50.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.680+0000 7ff44e200640 1 Processor -- start 2024-04-26T23:02:50.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.680+0000 7ff44e200640 1 -- start start 2024-04-26T23:02:50.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.681+0000 7ff44e200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c004800 0x7ff448158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:50.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.681+0000 7ff44e200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff448164b80 0x7ff448164fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:50.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.681+0000 7ff44e200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4481f91f0 0x7ff4481f9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:50.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.681+0000 7ff44e200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff4480ce7b0 con 0x7ff42c004800 2024-04-26T23:02:50.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.681+0000 7ff44e200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff4480cf430 con 0x7ff4481f91f0 2024-04-26T23:02:50.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.681+0000 7ff44e200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff4480ce930 con 0x7ff448164b80 2024-04-26T23:02:50.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.682+0000 7ff447400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c004800 0x7ff448158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:50.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.682+0000 7ff447400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c004800 0x7ff448158540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42634/0 (socket says 172.21.15.40:42634) 2024-04-26T23:02:50.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.682+0000 7ff447400640 1 -- 172.21.15.40:0/1311702576 learned_addr learned my addr 172.21.15.40:0/1311702576 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.682+0000 7ff447e00640 1 --2- 172.21.15.40:0/1311702576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4481f91f0 0x7ff4481f9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.682+0000 7ff447400640 1 -- 172.21.15.40:0/1311702576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff448164b80 msgr2=0x7ff448164fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.682+0000 7ff447400640 1 --2- 172.21.15.40:0/1311702576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff448164b80 0x7ff448164fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.682+0000 7ff447400640 1 -- 172.21.15.40:0/1311702576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4481f91f0 msgr2=0x7ff4481f9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.682+0000 7ff447400640 1 --2- 172.21.15.40:0/1311702576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4481f91f0 0x7ff4481f9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:50.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.682+0000 7ff447400640 1 -- 172.21.15.40:0/1311702576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4480b3800 con 0x7ff42c004800 2024-04-26T23:02:50.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.682+0000 7ff447400640 1 --2- 172.21.15.40:0/1311702576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c004800 0x7ff448158540 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff438011600 tx=0x7ff438011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:50.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.683+0000 7ff43fe00640 1 -- 172.21.15.40:0/1311702576 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43800f920 con 0x7ff42c004800 2024-04-26T23:02:50.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.683+0000 7ff43fe00640 1 -- 172.21.15.40:0/1311702576 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff43800fac0 con 0x7ff42c004800 2024-04-26T23:02:50.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.683+0000 7ff44e200640 1 -- 172.21.15.40:0/1311702576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4480b3a70 con 0x7ff42c004800 2024-04-26T23:02:50.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.683+0000 7ff43fe00640 1 -- 172.21.15.40:0/1311702576 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43800fe70 con 0x7ff42c004800 2024-04-26T23:02:50.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.683+0000 7ff44e200640 1 -- 172.21.15.40:0/1311702576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4480b4030 con 0x7ff42c004800 2024-04-26T23:02:50.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.684+0000 7ff44e200640 1 -- 172.21.15.40:0/1311702576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4480694f0 con 0x7ff42c004800 2024-04-26T23:02:50.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.685+0000 7ff43fe00640 1 -- 172.21.15.40:0/1311702576 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff438015460 con 0x7ff42c004800 2024-04-26T23:02:50.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.686+0000 7ff43fe00640 1 --2- 172.21.15.40:0/1311702576 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff42406c8f0 0x7ff42406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:50.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.686+0000 7ff43fe00640 1 -- 172.21.15.40:0/1311702576 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7ff43808c740 con 0x7ff42c004800 2024-04-26T23:02:50.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.687+0000 7ff43ea00640 1 --2- 172.21.15.40:0/1311702576 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff42406c8f0 0x7ff42406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:50.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.688+0000 7ff43ea00640 1 --2- 172.21.15.40:0/1311702576 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff42406c8f0 0x7ff42406edb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff430034000 tx=0x7ff43002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:50.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.690+0000 7ff43fe00640 1 -- 172.21.15.40:0/1311702576 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff438019030 con 0x7ff42c004800 2024-04-26T23:02:50.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:50.929+0000 7ff44e200640 1 -- 172.21.15.40:0/1311702576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7ff4480c6da0 con 0x7ff42c004800 2024-04-26T23:02:51.356 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-04-26T23:02:51.355+0000 7f0c37200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T23:02:51.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.441+0000 7ff43fe00640 1 -- 172.21.15.40:0/1311702576 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7ff43805a940 con 0x7ff42c004800 2024-04-26T23:02:51.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.445+0000 7ff44e200640 1 -- 172.21.15.40:0/1311702576 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff42406c8f0 msgr2=0x7ff42406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:51.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.445+0000 7ff44e200640 1 --2- 172.21.15.40:0/1311702576 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff42406c8f0 0x7ff42406edb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff430034000 tx=0x7ff43002e000 comp rx=0 tx=0).stop 2024-04-26T23:02:51.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.445+0000 7ff44e200640 1 -- 172.21.15.40:0/1311702576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c004800 msgr2=0x7ff448158540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:51.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.445+0000 7ff44e200640 1 --2- 172.21.15.40:0/1311702576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c004800 0x7ff448158540 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff438011600 tx=0x7ff438011ac0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.445+0000 7ff44e200640 1 -- 172.21.15.40:0/1311702576 shutdown_connections 2024-04-26T23:02:51.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.445+0000 7ff44e200640 1 --2- 172.21.15.40:0/1311702576 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff42406c8f0 0x7ff42406edb0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.445+0000 7ff44e200640 1 --2- 172.21.15.40:0/1311702576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4481f91f0 0x7ff4481f9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.445+0000 7ff44e200640 1 --2- 172.21.15.40:0/1311702576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff448164b80 0x7ff448164fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.445+0000 7ff44e200640 1 --2- 172.21.15.40:0/1311702576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c004800 0x7ff448158540 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.445+0000 7ff44e200640 1 -- 172.21.15.40:0/1311702576 >> 172.21.15.40:0/1311702576 conn(0x7ff4480d0be0 msgr2=0x7ff4480c61f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:51.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.445+0000 7ff44e200640 1 -- 172.21.15.40:0/1311702576 shutdown_connections 2024-04-26T23:02:51.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.445+0000 7ff44e200640 1 -- 172.21.15.40:0/1311702576 wait complete. 2024-04-26T23:02:51.469 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-26T23:02:51.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:02:51.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.684+0000 7f32a0200640 1 Processor -- start 2024-04-26T23:02:51.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.684+0000 7f32a0200640 1 -- start start 2024-04-26T23:02:51.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.685+0000 7f32a0200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3298071e60 0x7f329806f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:51.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.685+0000 7f32a0200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f329806fe20 0x7f3298072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:51.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.685+0000 7f32a0200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3298072780 0x7f32980cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:51.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.685+0000 7f32a0200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f329815d7c0 con 0x7f3298072780 2024-04-26T23:02:51.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.685+0000 7f32a0200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f329815daa0 con 0x7f3298071e60 2024-04-26T23:02:51.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.685+0000 7f32a0200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f329815dd80 con 0x7f329806fe20 2024-04-26T23:02:51.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.686+0000 7f329e200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3298072780 0x7f32980cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:51.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.686+0000 7f329e200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3298072780 0x7f32980cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42642/0 (socket says 172.21.15.40:42642) 2024-04-26T23:02:51.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.686+0000 7f329e200640 1 -- 172.21.15.40:0/1214642287 learned_addr learned my addr 172.21.15.40:0/1214642287 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:51.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.686+0000 7f329e200640 1 -- 172.21.15.40:0/1214642287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f329806fe20 msgr2=0x7f3298072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:51.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.686+0000 7f329e200640 1 --2- 172.21.15.40:0/1214642287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f329806fe20 0x7f3298072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.686+0000 7f329e200640 1 -- 172.21.15.40:0/1214642287 >> v2:172.21.15.103:3300/0 conn(0x7f3298071e60 msgr2=0x7f329806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:51.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.686+0000 7f329e200640 1 --2- 172.21.15.40:0/1214642287 >> v2:172.21.15.103:3300/0 conn(0x7f3298071e60 0x7f329806f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.686+0000 7f329e200640 1 -- 172.21.15.40:0/1214642287 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f329815e060 con 0x7f3298072780 2024-04-26T23:02:51.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.687+0000 7f329e200640 1 --2- 172.21.15.40:0/1214642287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298072780 0x7f32980cb470 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f32980b3700 tx=0x7f329002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cda03f0673fb02 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:51.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.687+0000 7f3297e00640 1 -- 172.21.15.40:0/1214642287 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3290036070 con 0x7f3298072780 2024-04-26T23:02:51.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.688+0000 7f3297e00640 1 -- 172.21.15.40:0/1214642287 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3290031600 con 0x7f3298072780 2024-04-26T23:02:51.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.688+0000 7f3297e00640 1 -- 172.21.15.40:0/1214642287 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32900319d0 con 0x7f3298072780 2024-04-26T23:02:51.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.688+0000 7f32a0200640 1 -- 172.21.15.40:0/1214642287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298072780 msgr2=0x7f32980cb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:51.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.688+0000 7f32a0200640 1 --2- 172.21.15.40:0/1214642287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298072780 0x7f32980cb470 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f32980b3700 tx=0x7f329002a6a0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.688+0000 7f32a0200640 1 -- 172.21.15.40:0/1214642287 shutdown_connections 2024-04-26T23:02:51.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.688+0000 7f32a0200640 1 --2- 172.21.15.40:0/1214642287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298072780 0x7f32980cb470 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.688+0000 7f32a0200640 1 --2- 172.21.15.40:0/1214642287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f329806fe20 0x7f3298072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.688+0000 7f32a0200640 1 --2- 172.21.15.40:0/1214642287 >> v2:172.21.15.103:3300/0 conn(0x7f3298071e60 0x7f329806f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.688+0000 7f32a0200640 1 -- 172.21.15.40:0/1214642287 >> 172.21.15.40:0/1214642287 conn(0x7f32980d0be0 msgr2=0x7f32980d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:51.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.688+0000 7f32a0200640 1 -- 172.21.15.40:0/1214642287 shutdown_connections 2024-04-26T23:02:51.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.688+0000 7f32a0200640 1 -- 172.21.15.40:0/1214642287 wait complete. 2024-04-26T23:02:51.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.690+0000 7f32a0200640 1 Processor -- start 2024-04-26T23:02:51.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.690+0000 7f32a0200640 1 -- start start 2024-04-26T23:02:51.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.691+0000 7f32a0200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f329806fe20 0x7f32980b5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:51.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.691+0000 7f32a0200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298071e60 0x7f32981f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:51.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.691+0000 7f32a0200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3298072780 0x7f32981f3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:51.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.691+0000 7f32a0200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f329815d3f0 con 0x7f3298071e60 2024-04-26T23:02:51.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.691+0000 7f32a0200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3298072c10 con 0x7f329806fe20 2024-04-26T23:02:51.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.691+0000 7f32a0200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f329815d570 con 0x7f3298072780 2024-04-26T23:02:51.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.691+0000 7f329ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298071e60 0x7f32981f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:51.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.691+0000 7f329ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298071e60 0x7f32981f5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42652/0 (socket says 172.21.15.40:42652) 2024-04-26T23:02:51.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.691+0000 7f329ce00640 1 -- 172.21.15.40:0/1327618852 learned_addr learned my addr 172.21.15.40:0/1327618852 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:51.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.692+0000 7f329ce00640 1 -- 172.21.15.40:0/1327618852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3298072780 msgr2=0x7f32981f3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:51.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.692+0000 7f329ce00640 1 --2- 172.21.15.40:0/1327618852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3298072780 0x7f32981f3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.692+0000 7f329ce00640 1 -- 172.21.15.40:0/1327618852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f329806fe20 msgr2=0x7f32980b5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:51.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.692+0000 7f329ce00640 1 --2- 172.21.15.40:0/1327618852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f329806fe20 0x7f32980b5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.692+0000 7f329ce00640 1 -- 172.21.15.40:0/1327618852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32981f3650 con 0x7f3298071e60 2024-04-26T23:02:51.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.692+0000 7f329ce00640 1 --2- 172.21.15.40:0/1327618852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298071e60 0x7f32981f5550 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f328400b4a0 tx=0x7f328400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:51.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.692+0000 7f3295600640 1 -- 172.21.15.40:0/1327618852 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3284014070 con 0x7f3298071e60 2024-04-26T23:02:51.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.692+0000 7f3295600640 1 -- 172.21.15.40:0/1327618852 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3284010610 con 0x7f3298071e60 2024-04-26T23:02:51.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.692+0000 7f3295600640 1 -- 172.21.15.40:0/1327618852 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3284010a00 con 0x7f3298071e60 2024-04-26T23:02:51.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.692+0000 7f32a0200640 1 -- 172.21.15.40:0/1327618852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32980b5880 con 0x7f3298071e60 2024-04-26T23:02:51.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.692+0000 7f32a0200640 1 -- 172.21.15.40:0/1327618852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32980b5e40 con 0x7f3298071e60 2024-04-26T23:02:51.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.693+0000 7f32a0200640 1 -- 172.21.15.40:0/1327618852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32980694f0 con 0x7f3298071e60 2024-04-26T23:02:51.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.694+0000 7f3295600640 1 -- 172.21.15.40:0/1327618852 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3284017070 con 0x7f3298071e60 2024-04-26T23:02:51.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.695+0000 7f3295600640 1 --2- 172.21.15.40:0/1327618852 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f327806c8f0 0x7f327806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:51.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.695+0000 7f3295600640 1 -- 172.21.15.40:0/1327618852 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f328408b800 con 0x7f3298071e60 2024-04-26T23:02:51.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.696+0000 7f329d800640 1 --2- 172.21.15.40:0/1327618852 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f327806c8f0 0x7f327806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:51.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.697+0000 7f329d800640 1 --2- 172.21.15.40:0/1327618852 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f327806c8f0 0x7f327806edb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f328c009710 tx=0x7f328c006c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:51.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.701+0000 7f3295600640 1 -- 172.21.15.40:0/1327618852 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f32981f3650 con 0x7f3298071e60 2024-04-26T23:02:51.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.946+0000 7f32a0200640 1 -- 172.21.15.40:0/1327618852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f329815ab80 con 0x7f3298071e60 2024-04-26T23:02:51.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.947+0000 7f3295600640 1 -- 172.21.15.40:0/1327618852 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5178 (secure 0 0 0) 0x7f3284059a00 con 0x7f3298071e60 2024-04-26T23:02:51.948 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:51.948 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1979956236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1979956236},{"type":"v1","addr":"172.21.15.103:6837","nonce":1979956236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/603576312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":603576312},{"type":"v1","addr":"172.21.15.103:6836","nonce":603576312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/716981348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":716981348},{"type":"v1","addr":"172.21.15.40:6834","nonce":716981348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:02:47.924696+0000","modified":"2024-04-26T23:02:50.931247+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5334},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5334":{"gid":5334,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.40:6835/299800334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":299800334},{"type":"v1","addr":"172.21.15.40:6835","nonce":299800334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5334,"qdb_cluster":[5334]},"id":2}]} 2024-04-26T23:02:51.949 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 18 2024-04-26T23:02:51.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.949+0000 7f32a0200640 1 -- 172.21.15.40:0/1327618852 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f327806c8f0 msgr2=0x7f327806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:51.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.949+0000 7f32a0200640 1 --2- 172.21.15.40:0/1327618852 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f327806c8f0 0x7f327806edb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f328c009710 tx=0x7f328c006c90 comp rx=0 tx=0).stop 2024-04-26T23:02:51.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.949+0000 7f32a0200640 1 -- 172.21.15.40:0/1327618852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298071e60 msgr2=0x7f32981f5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:51.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.949+0000 7f32a0200640 1 --2- 172.21.15.40:0/1327618852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298071e60 0x7f32981f5550 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f328400b4a0 tx=0x7f328400b960 comp rx=0 tx=0).stop 2024-04-26T23:02:51.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.949+0000 7f32a0200640 1 -- 172.21.15.40:0/1327618852 shutdown_connections 2024-04-26T23:02:51.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.949+0000 7f32a0200640 1 --2- 172.21.15.40:0/1327618852 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f327806c8f0 0x7f327806edb0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.949+0000 7f32a0200640 1 --2- 172.21.15.40:0/1327618852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3298072780 0x7f32981f3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.949+0000 7f32a0200640 1 --2- 172.21.15.40:0/1327618852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298071e60 0x7f32981f5550 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.949+0000 7f32a0200640 1 --2- 172.21.15.40:0/1327618852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f329806fe20 0x7f32980b5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.949+0000 7f32a0200640 1 -- 172.21.15.40:0/1327618852 >> 172.21.15.40:0/1327618852 conn(0x7f32980d0be0 msgr2=0x7f32980cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.950+0000 7f32a0200640 1 -- 172.21.15.40:0/1327618852 shutdown_connections 2024-04-26T23:02:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:51.950+0000 7f32a0200640 1 -- 172.21.15.40:0/1327618852 wait complete. 2024-04-26T23:02:51.969 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:02:52.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.177+0000 7f08cbe00640 1 Processor -- start 2024-04-26T23:02:52.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.177+0000 7f08cbe00640 1 -- start start 2024-04-26T23:02:52.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.179+0000 7f08cbe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f08cc0af7a0 0x7f08cc0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.179+0000 7f08cbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08cc071fe0 0x7f08cc073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.179+0000 7f08cbe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08cc0d0820 con 0x7f08cc15ce60 2024-04-26T23:02:52.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.179+0000 7f08cbe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f08cc0cc450 con 0x7f08cc0af7a0 2024-04-26T23:02:52.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.179+0000 7f08cbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f08cc1614c0 con 0x7f08cc071fe0 2024-04-26T23:02:52.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.180+0000 7f08c3e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f08cc0af7a0 0x7f08cc0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.180+0000 7f08cb400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08cc071fe0 0x7f08cc073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.180+0000 7f08c3e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f08cc0af7a0 0x7f08cc0afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:45048/0 (socket says 172.21.15.40:45048) 2024-04-26T23:02:52.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.180+0000 7f08c3e00640 1 -- 172.21.15.40:0/1841230324 learned_addr learned my addr 172.21.15.40:0/1841230324 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:52.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.180+0000 7f08caa00640 1 --1- 172.21.15.40:0/1841230324 >> v1:172.21.15.103:6789/0 conn(0x7f08cc15ce60 0x7f08cc0ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:59456/0 (socket says 172.21.15.40:59456) 2024-04-26T23:02:52.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.181+0000 7f08c3e00640 1 -- 172.21.15.40:0/1841230324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08cc071fe0 msgr2=0x7f08cc073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:52.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.181+0000 7f08c3e00640 1 --2- 172.21.15.40:0/1841230324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08cc071fe0 0x7f08cc073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.181+0000 7f08c3e00640 1 -- 172.21.15.40:0/1841230324 >> v1:172.21.15.103:6789/0 conn(0x7f08cc15ce60 legacy=0x7f08cc0ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:52.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.181+0000 7f08c3e00640 1 -- 172.21.15.40:0/1841230324 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08cc161760 con 0x7f08cc0af7a0 2024-04-26T23:02:52.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.182+0000 7f08c3e00640 1 --2- 172.21.15.40:0/1841230324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc0af7a0 0x7f08cc0afc60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009870 tx=0x7f08b402a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2528f21cb1e89ef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:52.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.183+0000 7f08ca000640 1 -- 172.21.15.40:0/1841230324 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b4037070 con 0x7f08cc0af7a0 2024-04-26T23:02:52.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.183+0000 7f08ca000640 1 -- 172.21.15.40:0/1841230324 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b4032440 con 0x7f08cc0af7a0 2024-04-26T23:02:52.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.183+0000 7f08ca000640 1 -- 172.21.15.40:0/1841230324 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b4032810 con 0x7f08cc0af7a0 2024-04-26T23:02:52.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.183+0000 7f08cbe00640 1 -- 172.21.15.40:0/1841230324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc0af7a0 msgr2=0x7f08cc0afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:52.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.183+0000 7f08cbe00640 1 --2- 172.21.15.40:0/1841230324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc0af7a0 0x7f08cc0afc60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009870 tx=0x7f08b402a1e0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.183+0000 7f08cbe00640 1 -- 172.21.15.40:0/1841230324 shutdown_connections 2024-04-26T23:02:52.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.183+0000 7f08cbe00640 1 --2- 172.21.15.40:0/1841230324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08cc071fe0 0x7f08cc073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.183+0000 7f08cbe00640 1 --2- 172.21.15.40:0/1841230324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc0af7a0 0x7f08cc0afc60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.184+0000 7f08cbe00640 1 -- 172.21.15.40:0/1841230324 >> 172.21.15.40:0/1841230324 conn(0x7f08cc0d0a10 msgr2=0x7f08cc0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:52.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.184+0000 7f08cbe00640 1 -- 172.21.15.40:0/1841230324 shutdown_connections 2024-04-26T23:02:52.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.184+0000 7f08cbe00640 1 -- 172.21.15.40:0/1841230324 wait complete. 2024-04-26T23:02:52.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.185+0000 7f08cbe00640 1 Processor -- start 2024-04-26T23:02:52.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.185+0000 7f08cbe00640 1 -- start start 2024-04-26T23:02:52.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.186+0000 7f08cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08cc071fe0 0x7f08cc15f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.186+0000 7f08cbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08cc0af7a0 0x7f08cc15fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.186+0000 7f08cbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc15ce60 0x7f08cc15d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.186+0000 7f08cbe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc0cc1f0 con 0x7f08cc071fe0 2024-04-26T23:02:52.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.186+0000 7f08cbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc0cc070 con 0x7f08cc15ce60 2024-04-26T23:02:52.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.186+0000 7f08cbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f08cc073ca0 con 0x7f08cc0af7a0 2024-04-26T23:02:52.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.187+0000 7f08caa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08cc071fe0 0x7f08cc15f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.187+0000 7f08caa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08cc071fe0 0x7f08cc15f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42664/0 (socket says 172.21.15.40:42664) 2024-04-26T23:02:52.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.187+0000 7f08caa00640 1 -- 172.21.15.40:0/4154241524 learned_addr learned my addr 172.21.15.40:0/4154241524 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:52.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.187+0000 7f08c3e00640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08cc0af7a0 0x7f08cc15fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.187+0000 7f08cb400640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc15ce60 0x7f08cc15d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.187+0000 7f08caa00640 1 -- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08cc0af7a0 msgr2=0x7f08cc15fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:52.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.187+0000 7f08caa00640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08cc0af7a0 0x7f08cc15fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.187+0000 7f08caa00640 1 -- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc15ce60 msgr2=0x7f08cc15d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:52.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.187+0000 7f08caa00640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc15ce60 0x7f08cc15d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.187+0000 7f08caa00640 1 -- 172.21.15.40:0/4154241524 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08cc15b9d0 con 0x7f08cc071fe0 2024-04-26T23:02:52.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.187+0000 7f08cb400640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc15ce60 0x7f08cc15d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:02:52.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.188+0000 7f08caa00640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08cc071fe0 0x7f08cc15f650 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00c3f0 tx=0x7f08bc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:52.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.188+0000 7f08c2a00640 1 -- 172.21.15.40:0/4154241524 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc0074e0 con 0x7f08cc071fe0 2024-04-26T23:02:52.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.188+0000 7f08c2a00640 1 -- 172.21.15.40:0/4154241524 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08bc007680 con 0x7f08cc071fe0 2024-04-26T23:02:52.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.188+0000 7f08c2a00640 1 -- 172.21.15.40:0/4154241524 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc004a00 con 0x7f08cc071fe0 2024-04-26T23:02:52.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.188+0000 7f08cbe00640 1 -- 172.21.15.40:0/4154241524 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08cc15bcb0 con 0x7f08cc071fe0 2024-04-26T23:02:52.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.188+0000 7f08cbe00640 1 -- 172.21.15.40:0/4154241524 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08cc14f380 con 0x7f08cc071fe0 2024-04-26T23:02:52.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.189+0000 7f08cbe00640 1 -- 172.21.15.40:0/4154241524 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08cc0b4120 con 0x7f08cc071fe0 2024-04-26T23:02:52.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.190+0000 7f08c2a00640 1 -- 172.21.15.40:0/4154241524 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f08bc01a020 con 0x7f08cc071fe0 2024-04-26T23:02:52.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.191+0000 7f08c2a00640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f08a806c920 0x7f08a806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.191+0000 7f08c2a00640 1 -- 172.21.15.40:0/4154241524 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f08bc08b560 con 0x7f08cc071fe0 2024-04-26T23:02:52.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.192+0000 7f08c3e00640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f08a806c920 0x7f08a806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.193+0000 7f08c3e00640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f08a806c920 0x7f08a806ede0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f08b4034000 tx=0x7f08b402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:52.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.196+0000 7f08c2a00640 1 -- 172.21.15.40:0/4154241524 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f08bc010300 con 0x7f08cc071fe0 2024-04-26T23:02:52.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.403+0000 7f08cbe00640 1 -- 172.21.15.40:0/4154241524 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f08cc069370 con 0x7f08cc071fe0 2024-04-26T23:02:52.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.405+0000 7f08c2a00640 1 -- 172.21.15.40:0/4154241524 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16906 (secure 0 0 0) 0x7f08bc059760 con 0x7f08cc071fe0 2024-04-26T23:02:52.405 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:52.406 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":39,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:02:49.409138+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:02:45.864120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-26T23:02:46.869290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:02:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.408+0000 7f08cbe00640 1 -- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f08a806c920 msgr2=0x7f08a806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.408+0000 7f08cbe00640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f08a806c920 0x7f08a806ede0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f08b4034000 tx=0x7f08b402e000 comp rx=0 tx=0).stop 2024-04-26T23:02:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.408+0000 7f08cbe00640 1 -- 172.21.15.40:0/4154241524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08cc071fe0 msgr2=0x7f08cc15f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.408+0000 7f08cbe00640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08cc071fe0 0x7f08cc15f650 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00c3f0 tx=0x7f08bc00c8b0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.408+0000 7f08cbe00640 1 -- 172.21.15.40:0/4154241524 shutdown_connections 2024-04-26T23:02:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.408+0000 7f08cbe00640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f08a806c920 0x7f08a806ede0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.408+0000 7f08cbe00640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc15ce60 0x7f08cc15d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.408+0000 7f08cbe00640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08cc0af7a0 0x7f08cc15fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.408+0000 7f08cbe00640 1 --2- 172.21.15.40:0/4154241524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08cc071fe0 0x7f08cc15f650 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.408+0000 7f08cbe00640 1 -- 172.21.15.40:0/4154241524 >> 172.21.15.40:0/4154241524 conn(0x7f08cc0d0a10 msgr2=0x7f08cc0ce6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.408+0000 7f08cbe00640 1 -- 172.21.15.40:0/4154241524 shutdown_connections 2024-04-26T23:02:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.408+0000 7f08cbe00640 1 -- 172.21.15.40:0/4154241524 wait complete. 2024-04-26T23:02:52.429 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T23:02:52.429 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:02:52.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.645+0000 7f15db800640 1 Processor -- start 2024-04-26T23:02:52.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.645+0000 7f15db800640 1 -- start start 2024-04-26T23:02:52.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.646+0000 7f15db800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15d40cbb50 0x7f15d40af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.646+0000 7f15db800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15d4072c00 con 0x7f15d415f380 2024-04-26T23:02:52.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.646+0000 7f15db800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15d4072ee0 con 0x7f15d4073b70 2024-04-26T23:02:52.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.646+0000 7f15db800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f15d40729b0 con 0x7f15d40cbb50 2024-04-26T23:02:52.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.647+0000 7f15d8e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f15d415f380 0x7f15d40cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49924/0 (socket says 172.21.15.40:49924) 2024-04-26T23:02:52.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.647+0000 7f15d8e00640 1 -- 172.21.15.40:0/1051146902 learned_addr learned my addr 172.21.15.40:0/1051146902 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:52.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.647+0000 7f15d9800640 1 --2- 172.21.15.40:0/1051146902 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15d40cbb50 0x7f15d40af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.647+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 18856956 0 0) 0x7f15d4072c00 con 0x7f15d415f380 2024-04-26T23:02:52.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.647+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15bc003520 con 0x7f15d415f380 2024-04-26T23:02:52.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2728074379 0 0) 0x7f15bc003520 con 0x7f15d415f380 2024-04-26T23:02:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15d40cbb50 msgr2=0x7f15d40af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3400640 1 --2- 172.21.15.40:0/1051146902 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15d40cbb50 0x7f15d40af3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 >> v1:172.21.15.103:6789/0 conn(0x7f15d4073b70 legacy=0x7f15d40aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15d40731c0 con 0x7f15d415f380 2024-04-26T23:02:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f15c8001670 con 0x7f15d415f380 2024-04-26T23:02:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 >> v1:172.21.15.40:6789/0 conn(0x7f15d415f380 legacy=0x7f15d40cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3400640 1 --2- 172.21.15.40:0/1051146902 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15d40cbb50 0x7f15bc003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3400640 1 --2- 172.21.15.40:0/1051146902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15bc003f80 0x7f15bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3400640 1 --2- 172.21.15.40:0/1051146902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15bc004d40 0x7f15bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15d40729b0 con 0x7f15bc003f80 2024-04-26T23:02:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc00bad0 con 0x7f15bc004d40 2024-04-26T23:02:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f15bc00bcd0 con 0x7f15d40cbb50 2024-04-26T23:02:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.648+0000 7f15d3e00640 1 --2- 172.21.15.40:0/1051146902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15bc003f80 0x7f15bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.649+0000 7f15d9800640 1 --2- 172.21.15.40:0/1051146902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15bc004d40 0x7f15bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.649+0000 7f15d8e00640 1 --2- 172.21.15.40:0/1051146902 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15d40cbb50 0x7f15bc003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.649+0000 7f15d3e00640 1 -- 172.21.15.40:0/1051146902 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15d40cbb50 msgr2=0x7f15bc003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.649+0000 7f15d3e00640 1 --2- 172.21.15.40:0/1051146902 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15d40cbb50 0x7f15bc003a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.649+0000 7f15d3e00640 1 -- 172.21.15.40:0/1051146902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15bc004d40 msgr2=0x7f15bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.649+0000 7f15d3e00640 1 --2- 172.21.15.40:0/1051146902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15bc004d40 0x7f15bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.649+0000 7f15d3e00640 1 -- 172.21.15.40:0/1051146902 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15bc003520 con 0x7f15bc003f80 2024-04-26T23:02:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.650+0000 7f15d3e00640 1 --2- 172.21.15.40:0/1051146902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15bc003f80 0x7f15bc004800 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f15c0001720 tx=0x7f15bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=defa86f7a77f60e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.650+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c0001370 con 0x7f15bc003f80 2024-04-26T23:02:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.650+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15c00052d0 con 0x7f15bc003f80 2024-04-26T23:02:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.650+0000 7f15d3400640 1 -- 172.21.15.40:0/1051146902 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c002abc0 con 0x7f15bc003f80 2024-04-26T23:02:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.651+0000 7f15db800640 1 -- 172.21.15.40:0/1051146902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15bc003f80 msgr2=0x7f15bc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.651+0000 7f15db800640 1 --2- 172.21.15.40:0/1051146902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15bc003f80 0x7f15bc004800 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f15c0001720 tx=0x7f15bc005830 comp rx=0 tx=0).stop 2024-04-26T23:02:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.651+0000 7f15d9800640 1 -- 172.21.15.40:0/1051146902 reap_dead start 2024-04-26T23:02:52.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.651+0000 7f15db800640 1 -- 172.21.15.40:0/1051146902 shutdown_connections 2024-04-26T23:02:52.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.651+0000 7f15db800640 1 -- 172.21.15.40:0/1051146902 >> 172.21.15.40:0/1051146902 conn(0x7f15d40d0be0 msgr2=0x7f15d40d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:52.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.651+0000 7f15db800640 1 -- 172.21.15.40:0/1051146902 shutdown_connections 2024-04-26T23:02:52.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.651+0000 7f15db800640 1 -- 172.21.15.40:0/1051146902 wait complete. 2024-04-26T23:02:52.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.653+0000 7f15db800640 1 Processor -- start 2024-04-26T23:02:52.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.653+0000 7f15db800640 1 -- start start 2024-04-26T23:02:52.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.654+0000 7f15db800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15bc003f80 0x7f15d40c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.654+0000 7f15db800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15d40b4ef0 0x7f15d40b5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.654+0000 7f15db800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d40b17c0 0x7f15d40b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.654+0000 7f15db800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15d4073620 con 0x7f15d40b17c0 2024-04-26T23:02:52.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.654+0000 7f15db800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f15d40d0870 con 0x7f15bc003f80 2024-04-26T23:02:52.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.654+0000 7f15db800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f15d40737a0 con 0x7f15d40b4ef0 2024-04-26T23:02:52.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.654+0000 7f15d9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d40b17c0 0x7f15d40b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.654+0000 7f15d9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d40b17c0 0x7f15d40b1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42686/0 (socket says 172.21.15.40:42686) 2024-04-26T23:02:52.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.654+0000 7f15d9800640 1 -- 172.21.15.40:0/2252303125 learned_addr learned my addr 172.21.15.40:0/2252303125 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:52.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.655+0000 7f15d3e00640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15d40b4ef0 0x7f15d40b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.655+0000 7f15d8e00640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15bc003f80 0x7f15d40c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.655+0000 7f15d9800640 1 -- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15d40b4ef0 msgr2=0x7f15d40b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:52.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.655+0000 7f15d9800640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15d40b4ef0 0x7f15d40b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.655+0000 7f15d9800640 1 -- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15bc003f80 msgr2=0x7f15d40c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:52.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.655+0000 7f15d9800640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15bc003f80 0x7f15d40c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.655+0000 7f15d9800640 1 -- 172.21.15.40:0/2252303125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15d40b5a60 con 0x7f15d40b17c0 2024-04-26T23:02:52.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.655+0000 7f15d8e00640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15bc003f80 0x7f15d40c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:52.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.655+0000 7f15d9800640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d40b17c0 0x7f15d40b1c20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f15d40cb1c0 tx=0x7f15cc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:52.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.655+0000 7f15d0c00640 1 -- 172.21.15.40:0/2252303125 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc011e40 con 0x7f15d40b17c0 2024-04-26T23:02:52.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.655+0000 7f15d0c00640 1 -- 172.21.15.40:0/2252303125 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15cc01b070 con 0x7f15d40b17c0 2024-04-26T23:02:52.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.655+0000 7f15d0c00640 1 -- 172.21.15.40:0/2252303125 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc003990 con 0x7f15d40b17c0 2024-04-26T23:02:52.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.655+0000 7f15db800640 1 -- 172.21.15.40:0/2252303125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15d40b5d40 con 0x7f15d40b17c0 2024-04-26T23:02:52.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.656+0000 7f15db800640 1 -- 172.21.15.40:0/2252303125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15d40b6300 con 0x7f15d40b17c0 2024-04-26T23:02:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.657+0000 7f15db800640 1 -- 172.21.15.40:0/2252303125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15d40b66e0 con 0x7f15d40b17c0 2024-04-26T23:02:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.658+0000 7f15d0c00640 1 -- 172.21.15.40:0/2252303125 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f15cc005d10 con 0x7f15d40b17c0 2024-04-26T23:02:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.659+0000 7f15d0c00640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f15b406c8f0 0x7f15b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.659+0000 7f15d0c00640 1 -- 172.21.15.40:0/2252303125 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f15cc08c400 con 0x7f15d40b17c0 2024-04-26T23:02:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.660+0000 7f15d8e00640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f15b406c8f0 0x7f15b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:52.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.661+0000 7f15d8e00640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f15b406c8f0 0x7f15b406edb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f15c8001200 tx=0x7f15c8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:52.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.663+0000 7f15d0c00640 1 -- 172.21.15.40:0/2252303125 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f15d40b66e0 con 0x7f15d40b17c0 2024-04-26T23:02:52.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.862+0000 7f15db800640 1 -- 172.21.15.40:0/2252303125 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f15d40c7690 con 0x7f15b406c8f0 2024-04-26T23:02:52.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.883+0000 7f15d0c00640 1 -- 172.21.15.40:0/2252303125 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246635 (secure 0 0 0) 0x7f15d40c7690 con 0x7f15b406c8f0 2024-04-26T23:02:52.888 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:02:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.888+0000 7f15db800640 1 -- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f15b406c8f0 msgr2=0x7f15b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.888+0000 7f15db800640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f15b406c8f0 0x7f15b406edb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f15c8001200 tx=0x7f15c8009040 comp rx=0 tx=0).stop 2024-04-26T23:02:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.888+0000 7f15db800640 1 -- 172.21.15.40:0/2252303125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d40b17c0 msgr2=0x7f15d40b1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.888+0000 7f15db800640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d40b17c0 0x7f15d40b1c20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f15d40cb1c0 tx=0x7f15cc0115a0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.888+0000 7f15db800640 1 -- 172.21.15.40:0/2252303125 shutdown_connections 2024-04-26T23:02:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.888+0000 7f15db800640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f15b406c8f0 0x7f15b406edb0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.888+0000 7f15db800640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15d40b17c0 0x7f15d40b1c20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.888+0000 7f15db800640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f15d40b4ef0 0x7f15d40b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.888+0000 7f15db800640 1 --2- 172.21.15.40:0/2252303125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15bc003f80 0x7f15d40c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:52.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.888+0000 7f15db800640 1 -- 172.21.15.40:0/2252303125 >> 172.21.15.40:0/2252303125 conn(0x7f15d40d0be0 msgr2=0x7f15d40cede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:52.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.888+0000 7f15db800640 1 -- 172.21.15.40:0/2252303125 shutdown_connections 2024-04-26T23:02:52.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:52.888+0000 7f15db800640 1 -- 172.21.15.40:0/2252303125 wait complete. 2024-04-26T23:02:52.908 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-04-26T23:02:51.182377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220620,"kb_used_data":4452,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754100,"statfs":{"total":773094113280,"available":772868198400,"internally_reserved":0,"allocated":4558848,"data_stored":2627119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.803272"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409049+0000","last_change":"2024-04-26T23:02:47.409049+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409049+0000","last_undegraded":"2024-04-26T23:02:47.409049+0000","last_fullsized":"2024-04-26T23:02:47.409049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409471+0000","last_change":"2024-04-26T23:02:47.409471+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409471+0000","last_undegraded":"2024-04-26T23:02:47.409471+0000","last_fullsized":"2024-04-26T23:02:47.409471+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410825+0000","last_change":"2024-04-26T23:02:47.410825+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410825+0000","last_undegraded":"2024-04-26T23:02:47.410825+0000","last_fullsized":"2024-04-26T23:02:47.410825+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.402588+0000","last_change":"2024-04-26T23:02:47.402588+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.402588+0000","last_undegraded":"2024-04-26T23:02:47.402588+0000","last_fullsized":"2024-04-26T23:02:47.402588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400758+0000","last_change":"2024-04-26T23:02:47.400758+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400758+0000","last_undegraded":"2024-04-26T23:02:47.400758+0000","last_fullsized":"2024-04-26T23:02:47.400758+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407437+0000","last_change":"2024-04-26T23:02:47.407437+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407437+0000","last_undegraded":"2024-04-26T23:02:47.407437+0000","last_fullsized":"2024-04-26T23:02:47.407437+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409272+0000","last_change":"2024-04-26T23:02:47.409272+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409272+0000","last_undegraded":"2024-04-26T23:02:47.409272+0000","last_fullsized":"2024-04-26T23:02:47.409272+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408667+0000","last_change":"2024-04-26T23:02:47.408667+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408667+0000","last_undegraded":"2024-04-26T23:02:47.408667+0000","last_fullsized":"2024-04-26T23:02:47.408667+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.406089+0000","last_change":"2024-04-26T23:02:47.406089+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.406089+0000","last_undegraded":"2024-04-26T23:02:47.406089+0000","last_fullsized":"2024-04-26T23:02:47.406089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409393+0000","last_change":"2024-04-26T23:02:47.409393+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409393+0000","last_undegraded":"2024-04-26T23:02:47.409393+0000","last_fullsized":"2024-04-26T23:02:47.409393+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408476+0000","last_change":"2024-04-26T23:02:47.408476+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408476+0000","last_undegraded":"2024-04-26T23:02:47.408476+0000","last_fullsized":"2024-04-26T23:02:47.408476+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411813+0000","last_change":"2024-04-26T23:02:47.411813+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411813+0000","last_undegraded":"2024-04-26T23:02:47.411813+0000","last_fullsized":"2024-04-26T23:02:47.411813+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412901+0000","last_change":"2024-04-26T23:02:47.412901+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412901+0000","last_undegraded":"2024-04-26T23:02:47.412901+0000","last_fullsized":"2024-04-26T23:02:47.412901+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404645+0000","last_change":"2024-04-26T23:02:47.404645+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404645+0000","last_undegraded":"2024-04-26T23:02:47.404645+0000","last_fullsized":"2024-04-26T23:02:47.404645+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408580+0000","last_change":"2024-04-26T23:02:47.408580+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408580+0000","last_undegraded":"2024-04-26T23:02:47.408580+0000","last_fullsized":"2024-04-26T23:02:47.408580+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412732+0000","last_change":"2024-04-26T23:02:47.412732+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412732+0000","last_undegraded":"2024-04-26T23:02:47.412732+0000","last_fullsized":"2024-04-26T23:02:47.412732+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.417759+0000","last_change":"2024-04-26T23:02:47.417759+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.417759+0000","last_undegraded":"2024-04-26T23:02:47.417759+0000","last_fullsized":"2024-04-26T23:02:47.417759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.401677+0000","last_change":"2024-04-26T23:02:47.401677+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.401677+0000","last_undegraded":"2024-04-26T23:02:47.401677+0000","last_fullsized":"2024-04-26T23:02:47.401677+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400720+0000","last_change":"2024-04-26T23:02:47.400720+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400720+0000","last_undegraded":"2024-04-26T23:02:47.400720+0000","last_fullsized":"2024-04-26T23:02:47.400720+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.401112+0000","last_change":"2024-04-26T23:02:47.401112+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.401112+0000","last_undegraded":"2024-04-26T23:02:47.401112+0000","last_fullsized":"2024-04-26T23:02:47.401112+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409722+0000","last_change":"2024-04-26T23:02:47.409722+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409722+0000","last_undegraded":"2024-04-26T23:02:47.409722+0000","last_fullsized":"2024-04-26T23:02:47.409722+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.403188+0000","last_change":"2024-04-26T23:02:47.403188+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.403188+0000","last_undegraded":"2024-04-26T23:02:47.403188+0000","last_fullsized":"2024-04-26T23:02:47.403188+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411186+0000","last_change":"2024-04-26T23:02:47.411186+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411186+0000","last_undegraded":"2024-04-26T23:02:47.411186+0000","last_fullsized":"2024-04-26T23:02:47.411186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400306+0000","last_change":"2024-04-26T23:02:47.400306+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400306+0000","last_undegraded":"2024-04-26T23:02:47.400306+0000","last_fullsized":"2024-04-26T23:02:47.400306+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404031+0000","last_change":"2024-04-26T23:02:47.404031+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404031+0000","last_undegraded":"2024-04-26T23:02:47.404031+0000","last_fullsized":"2024-04-26T23:02:47.404031+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408158+0000","last_change":"2024-04-26T23:02:47.408158+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408158+0000","last_undegraded":"2024-04-26T23:02:47.408158+0000","last_fullsized":"2024-04-26T23:02:47.408158+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407444+0000","last_change":"2024-04-26T23:02:47.407444+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407444+0000","last_undegraded":"2024-04-26T23:02:47.407444+0000","last_fullsized":"2024-04-26T23:02:47.407444+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408555+0000","last_change":"2024-04-26T23:02:47.408555+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408555+0000","last_undegraded":"2024-04-26T23:02:47.408555+0000","last_fullsized":"2024-04-26T23:02:47.408555+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400673+0000","last_change":"2024-04-26T23:02:47.400673+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400673+0000","last_undegraded":"2024-04-26T23:02:47.400673+0000","last_fullsized":"2024-04-26T23:02:47.400673+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.399159+0000","last_change":"2024-04-26T23:02:47.399159+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.399159+0000","last_undegraded":"2024-04-26T23:02:47.399159+0000","last_fullsized":"2024-04-26T23:02:47.399159+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410706+0000","last_change":"2024-04-26T23:02:47.410706+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410706+0000","last_undegraded":"2024-04-26T23:02:47.410706+0000","last_fullsized":"2024-04-26T23:02:47.410706+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404746+0000","last_change":"2024-04-26T23:02:47.404746+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404746+0000","last_undegraded":"2024-04-26T23:02:47.404746+0000","last_fullsized":"2024-04-26T23:02:47.404746+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.419764+0000","last_change":"2024-04-26T23:02:47.419764+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.419764+0000","last_undegraded":"2024-04-26T23:02:47.419764+0000","last_fullsized":"2024-04-26T23:02:47.419764+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.422997+0000","last_change":"2024-04-26T23:02:47.416086+0000","last_active":"2024-04-26T23:02:47.422997+0000","last_peered":"2024-04-26T23:02:47.422997+0000","last_clean":"2024-04-26T23:02:47.422997+0000","last_became_active":"2024-04-26T23:02:47.415794+0000","last_became_peered":"2024-04-26T23:02:47.415794+0000","last_unstale":"2024-04-26T23:02:47.422997+0000","last_undegraded":"2024-04-26T23:02:47.422997+0000","last_fullsized":"2024-04-26T23:02:47.422997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:54:53.447069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410136+0000","last_change":"2024-04-26T23:02:47.410136+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410136+0000","last_undegraded":"2024-04-26T23:02:47.410136+0000","last_fullsized":"2024-04-26T23:02:47.410136+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428834+0000","last_change":"2024-04-26T23:02:47.422501+0000","last_active":"2024-04-26T23:02:47.428834+0000","last_peered":"2024-04-26T23:02:47.428834+0000","last_clean":"2024-04-26T23:02:47.428834+0000","last_became_active":"2024-04-26T23:02:47.422240+0000","last_became_peered":"2024-04-26T23:02:47.422240+0000","last_unstale":"2024-04-26T23:02:47.428834+0000","last_undegraded":"2024-04-26T23:02:47.428834+0000","last_fullsized":"2024-04-26T23:02:47.428834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.398614+0000","last_change":"2024-04-26T23:02:47.398614+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.398614+0000","last_undegraded":"2024-04-26T23:02:47.398614+0000","last_fullsized":"2024-04-26T23:02:47.398614+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.432856+0000","last_change":"2024-04-26T23:02:47.427135+0000","last_active":"2024-04-26T23:02:47.432856+0000","last_peered":"2024-04-26T23:02:47.432856+0000","last_clean":"2024-04-26T23:02:47.432856+0000","last_became_active":"2024-04-26T23:02:47.425928+0000","last_became_peered":"2024-04-26T23:02:47.425928+0000","last_unstale":"2024-04-26T23:02:47.432856+0000","last_undegraded":"2024-04-26T23:02:47.432856+0000","last_fullsized":"2024-04-26T23:02:47.432856+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409517+0000","last_change":"2024-04-26T23:02:47.409517+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409517+0000","last_undegraded":"2024-04-26T23:02:47.409517+0000","last_fullsized":"2024-04-26T23:02:47.409517+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.427084+0000","last_change":"2024-04-26T23:02:47.423448+0000","last_active":"2024-04-26T23:02:47.427084+0000","last_peered":"2024-04-26T23:02:47.427084+0000","last_clean":"2024-04-26T23:02:47.427084+0000","last_became_active":"2024-04-26T23:02:47.423145+0000","last_became_peered":"2024-04-26T23:02:47.423145+0000","last_unstale":"2024-04-26T23:02:47.427084+0000","last_undegraded":"2024-04-26T23:02:47.427084+0000","last_fullsized":"2024-04-26T23:02:47.427084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.415281+0000","last_change":"2024-04-26T23:02:47.415281+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.415281+0000","last_undegraded":"2024-04-26T23:02:47.415281+0000","last_fullsized":"2024-04-26T23:02:47.415281+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431170+0000","last_change":"2024-04-26T23:02:47.429506+0000","last_active":"2024-04-26T23:02:47.431170+0000","last_peered":"2024-04-26T23:02:47.431170+0000","last_clean":"2024-04-26T23:02:47.431170+0000","last_became_active":"2024-04-26T23:02:47.429131+0000","last_became_peered":"2024-04-26T23:02:47.429131+0000","last_unstale":"2024-04-26T23:02:47.431170+0000","last_undegraded":"2024-04-26T23:02:47.431170+0000","last_fullsized":"2024-04-26T23:02:47.431170+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:28.822626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407967+0000","last_change":"2024-04-26T23:02:47.407967+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407967+0000","last_undegraded":"2024-04-26T23:02:47.407967+0000","last_fullsized":"2024-04-26T23:02:47.407967+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430956+0000","last_change":"2024-04-26T23:02:47.428296+0000","last_active":"2024-04-26T23:02:47.430956+0000","last_peered":"2024-04-26T23:02:47.430956+0000","last_clean":"2024-04-26T23:02:47.430956+0000","last_became_active":"2024-04-26T23:02:47.427980+0000","last_became_peered":"2024-04-26T23:02:47.427980+0000","last_unstale":"2024-04-26T23:02:47.430956+0000","last_undegraded":"2024-04-26T23:02:47.430956+0000","last_fullsized":"2024-04-26T23:02:47.430956+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:24:26.243323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.397241+0000","last_change":"2024-04-26T23:02:47.397241+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.397241+0000","last_undegraded":"2024-04-26T23:02:47.397241+0000","last_fullsized":"2024-04-26T23:02:47.397241+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.427966+0000","last_change":"2024-04-26T23:02:47.422927+0000","last_active":"2024-04-26T23:02:47.427966+0000","last_peered":"2024-04-26T23:02:47.427966+0000","last_clean":"2024-04-26T23:02:47.427966+0000","last_became_active":"2024-04-26T23:02:47.422616+0000","last_became_peered":"2024-04-26T23:02:47.422616+0000","last_unstale":"2024-04-26T23:02:47.427966+0000","last_undegraded":"2024-04-26T23:02:47.427966+0000","last_fullsized":"2024-04-26T23:02:47.427966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:23.080779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411122+0000","last_change":"2024-04-26T23:02:47.411122+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411122+0000","last_undegraded":"2024-04-26T23:02:47.411122+0000","last_fullsized":"2024-04-26T23:02:47.411122+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.429074+0000","last_change":"2024-04-26T23:02:47.426665+0000","last_active":"2024-04-26T23:02:47.429074+0000","last_peered":"2024-04-26T23:02:47.429074+0000","last_clean":"2024-04-26T23:02:47.429074+0000","last_became_active":"2024-04-26T23:02:47.426408+0000","last_became_peered":"2024-04-26T23:02:47.426408+0000","last_unstale":"2024-04-26T23:02:47.429074+0000","last_undegraded":"2024-04-26T23:02:47.429074+0000","last_fullsized":"2024-04-26T23:02:47.429074+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:28.822626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400911+0000","last_change":"2024-04-26T23:02:47.400911+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400911+0000","last_undegraded":"2024-04-26T23:02:47.400911+0000","last_fullsized":"2024-04-26T23:02:47.400911+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.433081+0000","last_change":"2024-04-26T23:02:47.429805+0000","last_active":"2024-04-26T23:02:47.433081+0000","last_peered":"2024-04-26T23:02:47.433081+0000","last_clean":"2024-04-26T23:02:47.433081+0000","last_became_active":"2024-04-26T23:02:47.429541+0000","last_became_peered":"2024-04-26T23:02:47.429541+0000","last_unstale":"2024-04-26T23:02:47.433081+0000","last_undegraded":"2024-04-26T23:02:47.433081+0000","last_fullsized":"2024-04-26T23:02:47.433081+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:01.704078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.420803+0000","last_change":"2024-04-26T23:02:47.420803+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.420803+0000","last_undegraded":"2024-04-26T23:02:47.420803+0000","last_fullsized":"2024-04-26T23:02:47.420803+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.425979+0000","last_change":"2024-04-26T23:02:47.424076+0000","last_active":"2024-04-26T23:02:47.425979+0000","last_peered":"2024-04-26T23:02:47.425979+0000","last_clean":"2024-04-26T23:02:47.425979+0000","last_became_active":"2024-04-26T23:02:47.420944+0000","last_became_peered":"2024-04-26T23:02:47.420944+0000","last_unstale":"2024-04-26T23:02:47.425979+0000","last_undegraded":"2024-04-26T23:02:47.425979+0000","last_fullsized":"2024-04-26T23:02:47.425979+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409689+0000","last_change":"2024-04-26T23:02:47.409689+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409689+0000","last_undegraded":"2024-04-26T23:02:47.409689+0000","last_fullsized":"2024-04-26T23:02:47.409689+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430549+0000","last_change":"2024-04-26T23:02:47.426825+0000","last_active":"2024-04-26T23:02:47.430549+0000","last_peered":"2024-04-26T23:02:47.430549+0000","last_clean":"2024-04-26T23:02:47.430549+0000","last_became_active":"2024-04-26T23:02:47.426475+0000","last_became_peered":"2024-04-26T23:02:47.426475+0000","last_unstale":"2024-04-26T23:02:47.430549+0000","last_undegraded":"2024-04-26T23:02:47.430549+0000","last_fullsized":"2024-04-26T23:02:47.430549+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.397867+0000","last_change":"2024-04-26T23:02:47.397867+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.397867+0000","last_undegraded":"2024-04-26T23:02:47.397867+0000","last_fullsized":"2024-04-26T23:02:47.397867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428260+0000","last_change":"2024-04-26T23:02:47.425906+0000","last_active":"2024-04-26T23:02:47.428260+0000","last_peered":"2024-04-26T23:02:47.428260+0000","last_clean":"2024-04-26T23:02:47.428260+0000","last_became_active":"2024-04-26T23:02:47.425612+0000","last_became_peered":"2024-04-26T23:02:47.425612+0000","last_unstale":"2024-04-26T23:02:47.428260+0000","last_undegraded":"2024-04-26T23:02:47.428260+0000","last_fullsized":"2024-04-26T23:02:47.428260+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:56:06.218292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412607+0000","last_change":"2024-04-26T23:02:47.412607+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412607+0000","last_undegraded":"2024-04-26T23:02:47.412607+0000","last_fullsized":"2024-04-26T23:02:47.412607+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.433025+0000","last_change":"2024-04-26T23:02:47.429056+0000","last_active":"2024-04-26T23:02:47.433025+0000","last_peered":"2024-04-26T23:02:47.433025+0000","last_clean":"2024-04-26T23:02:47.433025+0000","last_became_active":"2024-04-26T23:02:47.428144+0000","last_became_peered":"2024-04-26T23:02:47.428144+0000","last_unstale":"2024-04-26T23:02:47.433025+0000","last_undegraded":"2024-04-26T23:02:47.433025+0000","last_fullsized":"2024-04-26T23:02:47.433025+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:25.938366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408705+0000","last_change":"2024-04-26T23:02:47.408705+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408705+0000","last_undegraded":"2024-04-26T23:02:47.408705+0000","last_fullsized":"2024-04-26T23:02:47.408705+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431539+0000","last_change":"2024-04-26T23:02:47.426625+0000","last_active":"2024-04-26T23:02:47.431539+0000","last_peered":"2024-04-26T23:02:47.431539+0000","last_clean":"2024-04-26T23:02:47.431539+0000","last_became_active":"2024-04-26T23:02:47.426351+0000","last_became_peered":"2024-04-26T23:02:47.426351+0000","last_unstale":"2024-04-26T23:02:47.431539+0000","last_undegraded":"2024-04-26T23:02:47.431539+0000","last_fullsized":"2024-04-26T23:02:47.431539+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.413966+0000","last_change":"2024-04-26T23:02:47.413966+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.413966+0000","last_undegraded":"2024-04-26T23:02:47.413966+0000","last_fullsized":"2024-04-26T23:02:47.413966+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.423709+0000","last_change":"2024-04-26T23:02:47.418555+0000","last_active":"2024-04-26T23:02:47.423709+0000","last_peered":"2024-04-26T23:02:47.423709+0000","last_clean":"2024-04-26T23:02:47.423709+0000","last_became_active":"2024-04-26T23:02:47.418241+0000","last_became_peered":"2024-04-26T23:02:47.418241+0000","last_unstale":"2024-04-26T23:02:47.423709+0000","last_undegraded":"2024-04-26T23:02:47.423709+0000","last_fullsized":"2024-04-26T23:02:47.423709+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:18:53.091762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410588+0000","last_change":"2024-04-26T23:02:47.410588+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410588+0000","last_undegraded":"2024-04-26T23:02:47.410588+0000","last_fullsized":"2024-04-26T23:02:47.410588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430778+0000","last_change":"2024-04-26T23:02:47.426236+0000","last_active":"2024-04-26T23:02:47.430778+0000","last_peered":"2024-04-26T23:02:47.430778+0000","last_clean":"2024-04-26T23:02:47.430778+0000","last_became_active":"2024-04-26T23:02:47.425508+0000","last_became_peered":"2024-04-26T23:02:47.425508+0000","last_unstale":"2024-04-26T23:02:47.430778+0000","last_undegraded":"2024-04-26T23:02:47.430778+0000","last_fullsized":"2024-04-26T23:02:47.430778+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:37:06.151991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411859+0000","last_change":"2024-04-26T23:02:47.411859+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411859+0000","last_undegraded":"2024-04-26T23:02:47.411859+0000","last_fullsized":"2024-04-26T23:02:47.411859+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.425942+0000","last_change":"2024-04-26T23:02:47.422531+0000","last_active":"2024-04-26T23:02:47.425942+0000","last_peered":"2024-04-26T23:02:47.425942+0000","last_clean":"2024-04-26T23:02:47.425942+0000","last_became_active":"2024-04-26T23:02:47.422263+0000","last_became_peered":"2024-04-26T23:02:47.422263+0000","last_unstale":"2024-04-26T23:02:47.425942+0000","last_undegraded":"2024-04-26T23:02:47.425942+0000","last_fullsized":"2024-04-26T23:02:47.425942+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:23.080779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407338+0000","last_change":"2024-04-26T23:02:47.407338+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407338+0000","last_undegraded":"2024-04-26T23:02:47.407338+0000","last_fullsized":"2024-04-26T23:02:47.407338+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428543+0000","last_change":"2024-04-26T23:02:47.425145+0000","last_active":"2024-04-26T23:02:47.428543+0000","last_peered":"2024-04-26T23:02:47.428543+0000","last_clean":"2024-04-26T23:02:47.428543+0000","last_became_active":"2024-04-26T23:02:47.424860+0000","last_became_peered":"2024-04-26T23:02:47.424860+0000","last_unstale":"2024-04-26T23:02:47.428543+0000","last_undegraded":"2024-04-26T23:02:47.428543+0000","last_fullsized":"2024-04-26T23:02:47.428543+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.402648+0000","last_change":"2024-04-26T23:02:47.402648+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.402648+0000","last_undegraded":"2024-04-26T23:02:47.402648+0000","last_fullsized":"2024-04-26T23:02:47.402648+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431674+0000","last_change":"2024-04-26T23:02:47.427171+0000","last_active":"2024-04-26T23:02:47.431674+0000","last_peered":"2024-04-26T23:02:47.431674+0000","last_clean":"2024-04-26T23:02:47.431674+0000","last_became_active":"2024-04-26T23:02:47.426664+0000","last_became_peered":"2024-04-26T23:02:47.426664+0000","last_unstale":"2024-04-26T23:02:47.431674+0000","last_undegraded":"2024-04-26T23:02:47.431674+0000","last_fullsized":"2024-04-26T23:02:47.431674+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:35.059231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.399491+0000","last_change":"2024-04-26T23:02:47.399491+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.399491+0000","last_undegraded":"2024-04-26T23:02:47.399491+0000","last_fullsized":"2024-04-26T23:02:47.399491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431454+0000","last_change":"2024-04-26T23:02:47.425522+0000","last_active":"2024-04-26T23:02:47.431454+0000","last_peered":"2024-04-26T23:02:47.431454+0000","last_clean":"2024-04-26T23:02:47.431454+0000","last_became_active":"2024-04-26T23:02:47.425249+0000","last_became_peered":"2024-04-26T23:02:47.425249+0000","last_unstale":"2024-04-26T23:02:47.431454+0000","last_undegraded":"2024-04-26T23:02:47.431454+0000","last_fullsized":"2024-04-26T23:02:47.431454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412312+0000","last_change":"2024-04-26T23:02:47.412312+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412312+0000","last_undegraded":"2024-04-26T23:02:47.412312+0000","last_fullsized":"2024-04-26T23:02:47.412312+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.419715+0000","last_change":"2024-04-26T23:02:47.414839+0000","last_active":"2024-04-26T23:02:47.419715+0000","last_peered":"2024-04-26T23:02:47.419715+0000","last_clean":"2024-04-26T23:02:47.419715+0000","last_became_active":"2024-04-26T23:02:47.414531+0000","last_became_peered":"2024-04-26T23:02:47.414531+0000","last_unstale":"2024-04-26T23:02:47.419715+0000","last_undegraded":"2024-04-26T23:02:47.419715+0000","last_fullsized":"2024-04-26T23:02:47.419715+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:47:26.056049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409298+0000","last_change":"2024-04-26T23:02:47.409298+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409298+0000","last_undegraded":"2024-04-26T23:02:47.409298+0000","last_fullsized":"2024-04-26T23:02:47.409298+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.420473+0000","last_change":"2024-04-26T23:02:47.417644+0000","last_active":"2024-04-26T23:02:47.420473+0000","last_peered":"2024-04-26T23:02:47.420473+0000","last_clean":"2024-04-26T23:02:47.420473+0000","last_became_active":"2024-04-26T23:02:47.417390+0000","last_became_peered":"2024-04-26T23:02:47.417390+0000","last_unstale":"2024-04-26T23:02:47.420473+0000","last_undegraded":"2024-04-26T23:02:47.420473+0000","last_fullsized":"2024-04-26T23:02:47.420473+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:48:34.048591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409401+0000","last_change":"2024-04-26T23:02:47.409401+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409401+0000","last_undegraded":"2024-04-26T23:02:47.409401+0000","last_fullsized":"2024-04-26T23:02:47.409401+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.429682+0000","last_change":"2024-04-26T23:02:47.425458+0000","last_active":"2024-04-26T23:02:47.429682+0000","last_peered":"2024-04-26T23:02:47.429682+0000","last_clean":"2024-04-26T23:02:47.429682+0000","last_became_active":"2024-04-26T23:02:47.424961+0000","last_became_peered":"2024-04-26T23:02:47.424961+0000","last_unstale":"2024-04-26T23:02:47.429682+0000","last_undegraded":"2024-04-26T23:02:47.429682+0000","last_fullsized":"2024-04-26T23:02:47.429682+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:18:53.091762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409951+0000","last_change":"2024-04-26T23:02:47.409951+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409951+0000","last_undegraded":"2024-04-26T23:02:47.409951+0000","last_fullsized":"2024-04-26T23:02:47.409951+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.427504+0000","last_change":"2024-04-26T23:02:47.422918+0000","last_active":"2024-04-26T23:02:47.427504+0000","last_peered":"2024-04-26T23:02:47.427504+0000","last_clean":"2024-04-26T23:02:47.427504+0000","last_became_active":"2024-04-26T23:02:47.422578+0000","last_became_peered":"2024-04-26T23:02:47.422578+0000","last_unstale":"2024-04-26T23:02:47.427504+0000","last_undegraded":"2024-04-26T23:02:47.427504+0000","last_fullsized":"2024-04-26T23:02:47.427504+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:02:18.489647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407819+0000","last_change":"2024-04-26T23:02:47.407819+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407819+0000","last_undegraded":"2024-04-26T23:02:47.407819+0000","last_fullsized":"2024-04-26T23:02:47.407819+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430568+0000","last_change":"2024-04-26T23:02:47.427276+0000","last_active":"2024-04-26T23:02:47.430568+0000","last_peered":"2024-04-26T23:02:47.430568+0000","last_clean":"2024-04-26T23:02:47.430568+0000","last_became_active":"2024-04-26T23:02:47.426167+0000","last_became_peered":"2024-04-26T23:02:47.426167+0000","last_unstale":"2024-04-26T23:02:47.430568+0000","last_undegraded":"2024-04-26T23:02:47.430568+0000","last_fullsized":"2024-04-26T23:02:47.430568+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.420271+0000","last_change":"2024-04-26T23:02:47.420271+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.420271+0000","last_undegraded":"2024-04-26T23:02:47.420271+0000","last_fullsized":"2024-04-26T23:02:47.420271+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.419691+0000","last_change":"2024-04-26T23:02:47.415885+0000","last_active":"2024-04-26T23:02:47.419691+0000","last_peered":"2024-04-26T23:02:47.419691+0000","last_clean":"2024-04-26T23:02:47.419691+0000","last_became_active":"2024-04-26T23:02:47.415613+0000","last_became_peered":"2024-04-26T23:02:47.415613+0000","last_unstale":"2024-04-26T23:02:47.419691+0000","last_undegraded":"2024-04-26T23:02:47.419691+0000","last_fullsized":"2024-04-26T23:02:47.419691+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:46:24.430302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404728+0000","last_change":"2024-04-26T23:02:47.404728+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404728+0000","last_undegraded":"2024-04-26T23:02:47.404728+0000","last_fullsized":"2024-04-26T23:02:47.404728+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428564+0000","last_change":"2024-04-26T23:02:47.423354+0000","last_active":"2024-04-26T23:02:47.428564+0000","last_peered":"2024-04-26T23:02:47.428564+0000","last_clean":"2024-04-26T23:02:47.428564+0000","last_became_active":"2024-04-26T23:02:47.423058+0000","last_became_peered":"2024-04-26T23:02:47.423058+0000","last_unstale":"2024-04-26T23:02:47.428564+0000","last_undegraded":"2024-04-26T23:02:47.428564+0000","last_fullsized":"2024-04-26T23:02:47.428564+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411594+0000","last_change":"2024-04-26T23:02:47.411594+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411594+0000","last_undegraded":"2024-04-26T23:02:47.411594+0000","last_fullsized":"2024-04-26T23:02:47.411594+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428596+0000","last_change":"2024-04-26T23:02:47.421824+0000","last_active":"2024-04-26T23:02:47.428596+0000","last_peered":"2024-04-26T23:02:47.428596+0000","last_clean":"2024-04-26T23:02:47.428596+0000","last_became_active":"2024-04-26T23:02:47.421537+0000","last_became_peered":"2024-04-26T23:02:47.421537+0000","last_unstale":"2024-04-26T23:02:47.428596+0000","last_undegraded":"2024-04-26T23:02:47.428596+0000","last_fullsized":"2024-04-26T23:02:47.428596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:23.080779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.406234+0000","last_change":"2024-04-26T23:02:47.406234+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.406234+0000","last_undegraded":"2024-04-26T23:02:47.406234+0000","last_fullsized":"2024-04-26T23:02:47.406234+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.425058+0000","last_change":"2024-04-26T23:02:47.422919+0000","last_active":"2024-04-26T23:02:47.425058+0000","last_peered":"2024-04-26T23:02:47.425058+0000","last_clean":"2024-04-26T23:02:47.425058+0000","last_became_active":"2024-04-26T23:02:47.421694+0000","last_became_peered":"2024-04-26T23:02:47.421694+0000","last_unstale":"2024-04-26T23:02:47.425058+0000","last_undegraded":"2024-04-26T23:02:47.425058+0000","last_fullsized":"2024-04-26T23:02:47.425058+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:37:06.151991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.403814+0000","last_change":"2024-04-26T23:02:47.403814+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.403814+0000","last_undegraded":"2024-04-26T23:02:47.403814+0000","last_fullsized":"2024-04-26T23:02:47.403814+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.423560+0000","last_change":"2024-04-26T23:02:47.419127+0000","last_active":"2024-04-26T23:02:47.423560+0000","last_peered":"2024-04-26T23:02:47.423560+0000","last_clean":"2024-04-26T23:02:47.423560+0000","last_became_active":"2024-04-26T23:02:47.417694+0000","last_became_peered":"2024-04-26T23:02:47.417694+0000","last_unstale":"2024-04-26T23:02:47.423560+0000","last_undegraded":"2024-04-26T23:02:47.423560+0000","last_fullsized":"2024-04-26T23:02:47.423560+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.415211+0000","last_change":"2024-04-26T23:02:47.415211+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.415211+0000","last_undegraded":"2024-04-26T23:02:47.415211+0000","last_fullsized":"2024-04-26T23:02:47.415211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.419342+0000","last_change":"2024-04-26T23:02:47.416825+0000","last_active":"2024-04-26T23:02:47.419342+0000","last_peered":"2024-04-26T23:02:47.419342+0000","last_clean":"2024-04-26T23:02:47.419342+0000","last_became_active":"2024-04-26T23:02:47.416571+0000","last_became_peered":"2024-04-26T23:02:47.416571+0000","last_unstale":"2024-04-26T23:02:47.419342+0000","last_undegraded":"2024-04-26T23:02:47.419342+0000","last_fullsized":"2024-04-26T23:02:47.419342+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:25:15.376546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.406994+0000","last_change":"2024-04-26T23:02:47.406994+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.406994+0000","last_undegraded":"2024-04-26T23:02:47.406994+0000","last_fullsized":"2024-04-26T23:02:47.406994+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.433433+0000","last_change":"2024-04-26T23:02:47.432253+0000","last_active":"2024-04-26T23:02:47.433433+0000","last_peered":"2024-04-26T23:02:47.433433+0000","last_clean":"2024-04-26T23:02:47.433433+0000","last_became_active":"2024-04-26T23:02:47.431955+0000","last_became_peered":"2024-04-26T23:02:47.431955+0000","last_unstale":"2024-04-26T23:02:47.433433+0000","last_undegraded":"2024-04-26T23:02:47.433433+0000","last_fullsized":"2024-04-26T23:02:47.433433+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:27.125135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":203,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.413362+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:02:47.413362+0000","last_peered":"2024-04-26T23:02:47.413362+0000","last_clean":"2024-04-26T23:02:47.413362+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:02:47.413362+0000","last_undegraded":"2024-04-26T23:02:47.413362+0000","last_fullsized":"2024-04-26T23:02:47.413362+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":766999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1019904,"data_stored":773666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:02:52.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:02:53.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.166+0000 7fb749200640 1 Processor -- start 2024-04-26T23:02:53.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.167+0000 7fb749200640 1 -- start start 2024-04-26T23:02:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.168+0000 7fb749200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb7440b6220 0x7fb7440d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.168+0000 7fb749200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb7440d08d0 0x7fb7440d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.168+0000 7fb749200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb744162c40 con 0x7fb7440d3820 2024-04-26T23:02:53.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.168+0000 7fb749200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb7440d3c60 con 0x7fb7440d08d0 2024-04-26T23:02:53.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.168+0000 7fb749200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb7440d2d80 con 0x7fb7440b6220 2024-04-26T23:02:53.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.169+0000 7fb742a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb7440d08d0 0x7fb7440d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:53.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.169+0000 7fb743e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb7440d3820 0x7fb7440b6810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49930/0 (socket says 172.21.15.40:49930) 2024-04-26T23:02:53.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.169+0000 7fb743e00640 1 -- 172.21.15.40:0/2148778086 learned_addr learned my addr 172.21.15.40:0/2148778086 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:53.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.169+0000 7fb743400640 1 --2- 172.21.15.40:0/2148778086 >> v2:172.21.15.103:3300/0 conn(0x7fb7440b6220 0x7fb7440d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:53.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.169+0000 7fb742a00640 1 -- 172.21.15.40:0/2148778086 >> v2:172.21.15.103:3300/0 conn(0x7fb7440b6220 msgr2=0x7fb7440d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:53.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.169+0000 7fb742a00640 1 --2- 172.21.15.40:0/2148778086 >> v2:172.21.15.103:3300/0 conn(0x7fb7440b6220 0x7fb7440d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.169+0000 7fb742a00640 1 -- 172.21.15.40:0/2148778086 >> v1:172.21.15.40:6789/0 conn(0x7fb7440d3820 legacy=0x7fb7440b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:53.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.169+0000 7fb742000640 1 -- 172.21.15.40:0/2148778086 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2727511252 0 0) 0x7fb744162c40 con 0x7fb7440d3820 2024-04-26T23:02:53.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.169+0000 7fb742a00640 1 -- 172.21.15.40:0/2148778086 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7440b7c80 con 0x7fb7440d08d0 2024-04-26T23:02:53.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.169+0000 7fb743400640 1 --2- 172.21.15.40:0/2148778086 >> v2:172.21.15.103:3300/0 conn(0x7fb7440b6220 0x7fb7440d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:53.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.171+0000 7fb742a00640 1 --2- 172.21.15.40:0/2148778086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7440d08d0 0x7fb7440d32e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb7440d4cf0 tx=0x7fb73002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e65d741e6d73a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:53.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.171+0000 7fb742000640 1 -- 172.21.15.40:0/2148778086 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb730037070 con 0x7fb7440d08d0 2024-04-26T23:02:53.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.171+0000 7fb742000640 1 -- 172.21.15.40:0/2148778086 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb730032440 con 0x7fb7440d08d0 2024-04-26T23:02:53.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.171+0000 7fb742000640 1 -- 172.21.15.40:0/2148778086 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb730032810 con 0x7fb7440d08d0 2024-04-26T23:02:53.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.171+0000 7fb749200640 1 -- 172.21.15.40:0/2148778086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7440d08d0 msgr2=0x7fb7440d32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:53.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.171+0000 7fb749200640 1 --2- 172.21.15.40:0/2148778086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7440d08d0 0x7fb7440d32e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb7440d4cf0 tx=0x7fb73002a1a0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.172+0000 7fb749200640 1 -- 172.21.15.40:0/2148778086 shutdown_connections 2024-04-26T23:02:53.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.172+0000 7fb749200640 1 --2- 172.21.15.40:0/2148778086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7440d08d0 0x7fb7440d32e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.172+0000 7fb749200640 1 --2- 172.21.15.40:0/2148778086 >> v2:172.21.15.103:3300/0 conn(0x7fb7440b6220 0x7fb7440d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.172+0000 7fb749200640 1 -- 172.21.15.40:0/2148778086 >> 172.21.15.40:0/2148778086 conn(0x7fb7440d4d90 msgr2=0x7fb7440d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:53.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.172+0000 7fb749200640 1 -- 172.21.15.40:0/2148778086 shutdown_connections 2024-04-26T23:02:53.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.172+0000 7fb749200640 1 -- 172.21.15.40:0/2148778086 wait complete. 2024-04-26T23:02:53.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.173+0000 7fb749200640 1 Processor -- start 2024-04-26T23:02:53.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.174+0000 7fb749200640 1 -- start start 2024-04-26T23:02:53.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.174+0000 7fb749200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7440b6220 0x7fb74414e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:53.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.174+0000 7fb749200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7440d08d0 0x7fb744150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:53.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.174+0000 7fb749200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7440d3820 0x7fb7440b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:53.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.174+0000 7fb749200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7440b8520 con 0x7fb7440b6220 2024-04-26T23:02:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.174+0000 7fb749200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb7440b8240 con 0x7fb7440d3820 2024-04-26T23:02:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.174+0000 7fb749200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb7440b9d30 con 0x7fb7440d08d0 2024-04-26T23:02:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.175+0000 7fb743400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7440b6220 0x7fb74414e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.175+0000 7fb743400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7440b6220 0x7fb74414e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42692/0 (socket says 172.21.15.40:42692) 2024-04-26T23:02:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.175+0000 7fb743400640 1 -- 172.21.15.40:0/3668803074 learned_addr learned my addr 172.21.15.40:0/3668803074 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.175+0000 7fb742a00640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7440d08d0 0x7fb744150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.175+0000 7fb743e00640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7440d3820 0x7fb7440b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:53.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.175+0000 7fb743400640 1 -- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7440d08d0 msgr2=0x7fb744150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:53.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.175+0000 7fb743400640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7440d08d0 0x7fb744150ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.175+0000 7fb743400640 1 -- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7440d3820 msgr2=0x7fb7440b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:53.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.175+0000 7fb743400640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7440d3820 0x7fb7440b55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.175+0000 7fb743400640 1 -- 172.21.15.40:0/3668803074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74414e7a0 con 0x7fb7440b6220 2024-04-26T23:02:53.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.175+0000 7fb742a00640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7440d08d0 0x7fb744150ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:53.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.175+0000 7fb743e00640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7440d3820 0x7fb7440b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:53.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.176+0000 7fb743400640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7440b6220 0x7fb74414e0e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb73400c7f0 tx=0x7fb73400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:53.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.176+0000 7fb73b400640 1 -- 172.21.15.40:0/3668803074 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb734015070 con 0x7fb7440b6220 2024-04-26T23:02:53.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.176+0000 7fb73b400640 1 -- 172.21.15.40:0/3668803074 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb734007a30 con 0x7fb7440b6220 2024-04-26T23:02:53.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.176+0000 7fb73b400640 1 -- 172.21.15.40:0/3668803074 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb734007e00 con 0x7fb7440b6220 2024-04-26T23:02:53.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.176+0000 7fb749200640 1 -- 172.21.15.40:0/3668803074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb74414ea80 con 0x7fb7440b6220 2024-04-26T23:02:53.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.176+0000 7fb749200640 1 -- 172.21.15.40:0/3668803074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb74414f040 con 0x7fb7440b6220 2024-04-26T23:02:53.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.177+0000 7fb749200640 1 -- 172.21.15.40:0/3668803074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb744069370 con 0x7fb7440b6220 2024-04-26T23:02:53.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.178+0000 7fb73b400640 1 -- 172.21.15.40:0/3668803074 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb734018070 con 0x7fb7440b6220 2024-04-26T23:02:53.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.179+0000 7fb73b400640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb72406c8f0 0x7fb72406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:53.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.179+0000 7fb73b400640 1 -- 172.21.15.40:0/3668803074 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fb73408bcb0 con 0x7fb7440b6220 2024-04-26T23:02:53.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.180+0000 7fb742a00640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb72406c8f0 0x7fb72406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:53.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.181+0000 7fb742a00640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb72406c8f0 0x7fb72406edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb730034000 tx=0x7fb73002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:53.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.184+0000 7fb73b400640 1 -- 172.21.15.40:0/3668803074 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fb734059f40 con 0x7fb7440b6220 2024-04-26T23:02:53.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.388+0000 7fb749200640 1 -- 172.21.15.40:0/3668803074 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb7441510e0 con 0x7fb72406c8f0 2024-04-26T23:02:53.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.407+0000 7fb73b400640 1 -- 172.21.15.40:0/3668803074 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246635 (secure 0 0 0) 0x7fb7300326f0 con 0x7fb72406c8f0 2024-04-26T23:02:53.412 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:53.413 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:02:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.413+0000 7fb749200640 1 -- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb72406c8f0 msgr2=0x7fb72406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.413+0000 7fb749200640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb72406c8f0 0x7fb72406edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb730034000 tx=0x7fb73002e000 comp rx=0 tx=0).stop 2024-04-26T23:02:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.413+0000 7fb749200640 1 -- 172.21.15.40:0/3668803074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7440b6220 msgr2=0x7fb74414e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.413+0000 7fb749200640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7440b6220 0x7fb74414e0e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb73400c7f0 tx=0x7fb73400cbf0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.414+0000 7fb749200640 1 -- 172.21.15.40:0/3668803074 shutdown_connections 2024-04-26T23:02:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.414+0000 7fb749200640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb72406c8f0 0x7fb72406edb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.414+0000 7fb749200640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7440d3820 0x7fb7440b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.414+0000 7fb749200640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7440d08d0 0x7fb744150ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.414+0000 7fb749200640 1 --2- 172.21.15.40:0/3668803074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7440b6220 0x7fb74414e0e0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.414+0000 7fb749200640 1 -- 172.21.15.40:0/3668803074 >> 172.21.15.40:0/3668803074 conn(0x7fb7440d4d90 msgr2=0x7fb7440cf860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.414+0000 7fb749200640 1 -- 172.21.15.40:0/3668803074 shutdown_connections 2024-04-26T23:02:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.414+0000 7fb749200640 1 -- 172.21.15.40:0/3668803074 wait complete. 2024-04-26T23:02:53.433 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-04-26T23:02:53.182976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220620,"kb_used_data":4452,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754100,"statfs":{"total":773094113280,"available":772868198400,"internally_reserved":0,"allocated":4558848,"data_stored":2627119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001782"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409049+0000","last_change":"2024-04-26T23:02:47.409049+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409049+0000","last_undegraded":"2024-04-26T23:02:47.409049+0000","last_fullsized":"2024-04-26T23:02:47.409049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409471+0000","last_change":"2024-04-26T23:02:47.409471+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409471+0000","last_undegraded":"2024-04-26T23:02:47.409471+0000","last_fullsized":"2024-04-26T23:02:47.409471+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410825+0000","last_change":"2024-04-26T23:02:47.410825+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410825+0000","last_undegraded":"2024-04-26T23:02:47.410825+0000","last_fullsized":"2024-04-26T23:02:47.410825+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.402588+0000","last_change":"2024-04-26T23:02:47.402588+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.402588+0000","last_undegraded":"2024-04-26T23:02:47.402588+0000","last_fullsized":"2024-04-26T23:02:47.402588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400758+0000","last_change":"2024-04-26T23:02:47.400758+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400758+0000","last_undegraded":"2024-04-26T23:02:47.400758+0000","last_fullsized":"2024-04-26T23:02:47.400758+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407437+0000","last_change":"2024-04-26T23:02:47.407437+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407437+0000","last_undegraded":"2024-04-26T23:02:47.407437+0000","last_fullsized":"2024-04-26T23:02:47.407437+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409272+0000","last_change":"2024-04-26T23:02:47.409272+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409272+0000","last_undegraded":"2024-04-26T23:02:47.409272+0000","last_fullsized":"2024-04-26T23:02:47.409272+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408667+0000","last_change":"2024-04-26T23:02:47.408667+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408667+0000","last_undegraded":"2024-04-26T23:02:47.408667+0000","last_fullsized":"2024-04-26T23:02:47.408667+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.406089+0000","last_change":"2024-04-26T23:02:47.406089+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.406089+0000","last_undegraded":"2024-04-26T23:02:47.406089+0000","last_fullsized":"2024-04-26T23:02:47.406089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409393+0000","last_change":"2024-04-26T23:02:47.409393+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409393+0000","last_undegraded":"2024-04-26T23:02:47.409393+0000","last_fullsized":"2024-04-26T23:02:47.409393+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408476+0000","last_change":"2024-04-26T23:02:47.408476+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408476+0000","last_undegraded":"2024-04-26T23:02:47.408476+0000","last_fullsized":"2024-04-26T23:02:47.408476+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411813+0000","last_change":"2024-04-26T23:02:47.411813+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411813+0000","last_undegraded":"2024-04-26T23:02:47.411813+0000","last_fullsized":"2024-04-26T23:02:47.411813+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412901+0000","last_change":"2024-04-26T23:02:47.412901+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412901+0000","last_undegraded":"2024-04-26T23:02:47.412901+0000","last_fullsized":"2024-04-26T23:02:47.412901+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404645+0000","last_change":"2024-04-26T23:02:47.404645+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404645+0000","last_undegraded":"2024-04-26T23:02:47.404645+0000","last_fullsized":"2024-04-26T23:02:47.404645+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408580+0000","last_change":"2024-04-26T23:02:47.408580+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408580+0000","last_undegraded":"2024-04-26T23:02:47.408580+0000","last_fullsized":"2024-04-26T23:02:47.408580+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412732+0000","last_change":"2024-04-26T23:02:47.412732+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412732+0000","last_undegraded":"2024-04-26T23:02:47.412732+0000","last_fullsized":"2024-04-26T23:02:47.412732+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.417759+0000","last_change":"2024-04-26T23:02:47.417759+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.417759+0000","last_undegraded":"2024-04-26T23:02:47.417759+0000","last_fullsized":"2024-04-26T23:02:47.417759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.401677+0000","last_change":"2024-04-26T23:02:47.401677+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.401677+0000","last_undegraded":"2024-04-26T23:02:47.401677+0000","last_fullsized":"2024-04-26T23:02:47.401677+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400720+0000","last_change":"2024-04-26T23:02:47.400720+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400720+0000","last_undegraded":"2024-04-26T23:02:47.400720+0000","last_fullsized":"2024-04-26T23:02:47.400720+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.401112+0000","last_change":"2024-04-26T23:02:47.401112+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.401112+0000","last_undegraded":"2024-04-26T23:02:47.401112+0000","last_fullsized":"2024-04-26T23:02:47.401112+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409722+0000","last_change":"2024-04-26T23:02:47.409722+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409722+0000","last_undegraded":"2024-04-26T23:02:47.409722+0000","last_fullsized":"2024-04-26T23:02:47.409722+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.403188+0000","last_change":"2024-04-26T23:02:47.403188+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.403188+0000","last_undegraded":"2024-04-26T23:02:47.403188+0000","last_fullsized":"2024-04-26T23:02:47.403188+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411186+0000","last_change":"2024-04-26T23:02:47.411186+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411186+0000","last_undegraded":"2024-04-26T23:02:47.411186+0000","last_fullsized":"2024-04-26T23:02:47.411186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400306+0000","last_change":"2024-04-26T23:02:47.400306+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400306+0000","last_undegraded":"2024-04-26T23:02:47.400306+0000","last_fullsized":"2024-04-26T23:02:47.400306+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404031+0000","last_change":"2024-04-26T23:02:47.404031+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404031+0000","last_undegraded":"2024-04-26T23:02:47.404031+0000","last_fullsized":"2024-04-26T23:02:47.404031+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408158+0000","last_change":"2024-04-26T23:02:47.408158+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408158+0000","last_undegraded":"2024-04-26T23:02:47.408158+0000","last_fullsized":"2024-04-26T23:02:47.408158+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407444+0000","last_change":"2024-04-26T23:02:47.407444+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407444+0000","last_undegraded":"2024-04-26T23:02:47.407444+0000","last_fullsized":"2024-04-26T23:02:47.407444+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408555+0000","last_change":"2024-04-26T23:02:47.408555+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408555+0000","last_undegraded":"2024-04-26T23:02:47.408555+0000","last_fullsized":"2024-04-26T23:02:47.408555+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400673+0000","last_change":"2024-04-26T23:02:47.400673+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400673+0000","last_undegraded":"2024-04-26T23:02:47.400673+0000","last_fullsized":"2024-04-26T23:02:47.400673+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.399159+0000","last_change":"2024-04-26T23:02:47.399159+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.399159+0000","last_undegraded":"2024-04-26T23:02:47.399159+0000","last_fullsized":"2024-04-26T23:02:47.399159+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410706+0000","last_change":"2024-04-26T23:02:47.410706+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410706+0000","last_undegraded":"2024-04-26T23:02:47.410706+0000","last_fullsized":"2024-04-26T23:02:47.410706+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404746+0000","last_change":"2024-04-26T23:02:47.404746+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404746+0000","last_undegraded":"2024-04-26T23:02:47.404746+0000","last_fullsized":"2024-04-26T23:02:47.404746+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.419764+0000","last_change":"2024-04-26T23:02:47.419764+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.419764+0000","last_undegraded":"2024-04-26T23:02:47.419764+0000","last_fullsized":"2024-04-26T23:02:47.419764+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.422997+0000","last_change":"2024-04-26T23:02:47.416086+0000","last_active":"2024-04-26T23:02:47.422997+0000","last_peered":"2024-04-26T23:02:47.422997+0000","last_clean":"2024-04-26T23:02:47.422997+0000","last_became_active":"2024-04-26T23:02:47.415794+0000","last_became_peered":"2024-04-26T23:02:47.415794+0000","last_unstale":"2024-04-26T23:02:47.422997+0000","last_undegraded":"2024-04-26T23:02:47.422997+0000","last_fullsized":"2024-04-26T23:02:47.422997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:54:53.447069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410136+0000","last_change":"2024-04-26T23:02:47.410136+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410136+0000","last_undegraded":"2024-04-26T23:02:47.410136+0000","last_fullsized":"2024-04-26T23:02:47.410136+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428834+0000","last_change":"2024-04-26T23:02:47.422501+0000","last_active":"2024-04-26T23:02:47.428834+0000","last_peered":"2024-04-26T23:02:47.428834+0000","last_clean":"2024-04-26T23:02:47.428834+0000","last_became_active":"2024-04-26T23:02:47.422240+0000","last_became_peered":"2024-04-26T23:02:47.422240+0000","last_unstale":"2024-04-26T23:02:47.428834+0000","last_undegraded":"2024-04-26T23:02:47.428834+0000","last_fullsized":"2024-04-26T23:02:47.428834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.398614+0000","last_change":"2024-04-26T23:02:47.398614+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.398614+0000","last_undegraded":"2024-04-26T23:02:47.398614+0000","last_fullsized":"2024-04-26T23:02:47.398614+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.432856+0000","last_change":"2024-04-26T23:02:47.427135+0000","last_active":"2024-04-26T23:02:47.432856+0000","last_peered":"2024-04-26T23:02:47.432856+0000","last_clean":"2024-04-26T23:02:47.432856+0000","last_became_active":"2024-04-26T23:02:47.425928+0000","last_became_peered":"2024-04-26T23:02:47.425928+0000","last_unstale":"2024-04-26T23:02:47.432856+0000","last_undegraded":"2024-04-26T23:02:47.432856+0000","last_fullsized":"2024-04-26T23:02:47.432856+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409517+0000","last_change":"2024-04-26T23:02:47.409517+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409517+0000","last_undegraded":"2024-04-26T23:02:47.409517+0000","last_fullsized":"2024-04-26T23:02:47.409517+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.427084+0000","last_change":"2024-04-26T23:02:47.423448+0000","last_active":"2024-04-26T23:02:47.427084+0000","last_peered":"2024-04-26T23:02:47.427084+0000","last_clean":"2024-04-26T23:02:47.427084+0000","last_became_active":"2024-04-26T23:02:47.423145+0000","last_became_peered":"2024-04-26T23:02:47.423145+0000","last_unstale":"2024-04-26T23:02:47.427084+0000","last_undegraded":"2024-04-26T23:02:47.427084+0000","last_fullsized":"2024-04-26T23:02:47.427084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.415281+0000","last_change":"2024-04-26T23:02:47.415281+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.415281+0000","last_undegraded":"2024-04-26T23:02:47.415281+0000","last_fullsized":"2024-04-26T23:02:47.415281+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431170+0000","last_change":"2024-04-26T23:02:47.429506+0000","last_active":"2024-04-26T23:02:47.431170+0000","last_peered":"2024-04-26T23:02:47.431170+0000","last_clean":"2024-04-26T23:02:47.431170+0000","last_became_active":"2024-04-26T23:02:47.429131+0000","last_became_peered":"2024-04-26T23:02:47.429131+0000","last_unstale":"2024-04-26T23:02:47.431170+0000","last_undegraded":"2024-04-26T23:02:47.431170+0000","last_fullsized":"2024-04-26T23:02:47.431170+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:28.822626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407967+0000","last_change":"2024-04-26T23:02:47.407967+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407967+0000","last_undegraded":"2024-04-26T23:02:47.407967+0000","last_fullsized":"2024-04-26T23:02:47.407967+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430956+0000","last_change":"2024-04-26T23:02:47.428296+0000","last_active":"2024-04-26T23:02:47.430956+0000","last_peered":"2024-04-26T23:02:47.430956+0000","last_clean":"2024-04-26T23:02:47.430956+0000","last_became_active":"2024-04-26T23:02:47.427980+0000","last_became_peered":"2024-04-26T23:02:47.427980+0000","last_unstale":"2024-04-26T23:02:47.430956+0000","last_undegraded":"2024-04-26T23:02:47.430956+0000","last_fullsized":"2024-04-26T23:02:47.430956+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:24:26.243323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.397241+0000","last_change":"2024-04-26T23:02:47.397241+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.397241+0000","last_undegraded":"2024-04-26T23:02:47.397241+0000","last_fullsized":"2024-04-26T23:02:47.397241+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.427966+0000","last_change":"2024-04-26T23:02:47.422927+0000","last_active":"2024-04-26T23:02:47.427966+0000","last_peered":"2024-04-26T23:02:47.427966+0000","last_clean":"2024-04-26T23:02:47.427966+0000","last_became_active":"2024-04-26T23:02:47.422616+0000","last_became_peered":"2024-04-26T23:02:47.422616+0000","last_unstale":"2024-04-26T23:02:47.427966+0000","last_undegraded":"2024-04-26T23:02:47.427966+0000","last_fullsized":"2024-04-26T23:02:47.427966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:23.080779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411122+0000","last_change":"2024-04-26T23:02:47.411122+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411122+0000","last_undegraded":"2024-04-26T23:02:47.411122+0000","last_fullsized":"2024-04-26T23:02:47.411122+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.429074+0000","last_change":"2024-04-26T23:02:47.426665+0000","last_active":"2024-04-26T23:02:47.429074+0000","last_peered":"2024-04-26T23:02:47.429074+0000","last_clean":"2024-04-26T23:02:47.429074+0000","last_became_active":"2024-04-26T23:02:47.426408+0000","last_became_peered":"2024-04-26T23:02:47.426408+0000","last_unstale":"2024-04-26T23:02:47.429074+0000","last_undegraded":"2024-04-26T23:02:47.429074+0000","last_fullsized":"2024-04-26T23:02:47.429074+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:28.822626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400911+0000","last_change":"2024-04-26T23:02:47.400911+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400911+0000","last_undegraded":"2024-04-26T23:02:47.400911+0000","last_fullsized":"2024-04-26T23:02:47.400911+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.433081+0000","last_change":"2024-04-26T23:02:47.429805+0000","last_active":"2024-04-26T23:02:47.433081+0000","last_peered":"2024-04-26T23:02:47.433081+0000","last_clean":"2024-04-26T23:02:47.433081+0000","last_became_active":"2024-04-26T23:02:47.429541+0000","last_became_peered":"2024-04-26T23:02:47.429541+0000","last_unstale":"2024-04-26T23:02:47.433081+0000","last_undegraded":"2024-04-26T23:02:47.433081+0000","last_fullsized":"2024-04-26T23:02:47.433081+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:01.704078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.420803+0000","last_change":"2024-04-26T23:02:47.420803+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.420803+0000","last_undegraded":"2024-04-26T23:02:47.420803+0000","last_fullsized":"2024-04-26T23:02:47.420803+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.425979+0000","last_change":"2024-04-26T23:02:47.424076+0000","last_active":"2024-04-26T23:02:47.425979+0000","last_peered":"2024-04-26T23:02:47.425979+0000","last_clean":"2024-04-26T23:02:47.425979+0000","last_became_active":"2024-04-26T23:02:47.420944+0000","last_became_peered":"2024-04-26T23:02:47.420944+0000","last_unstale":"2024-04-26T23:02:47.425979+0000","last_undegraded":"2024-04-26T23:02:47.425979+0000","last_fullsized":"2024-04-26T23:02:47.425979+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409689+0000","last_change":"2024-04-26T23:02:47.409689+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409689+0000","last_undegraded":"2024-04-26T23:02:47.409689+0000","last_fullsized":"2024-04-26T23:02:47.409689+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430549+0000","last_change":"2024-04-26T23:02:47.426825+0000","last_active":"2024-04-26T23:02:47.430549+0000","last_peered":"2024-04-26T23:02:47.430549+0000","last_clean":"2024-04-26T23:02:47.430549+0000","last_became_active":"2024-04-26T23:02:47.426475+0000","last_became_peered":"2024-04-26T23:02:47.426475+0000","last_unstale":"2024-04-26T23:02:47.430549+0000","last_undegraded":"2024-04-26T23:02:47.430549+0000","last_fullsized":"2024-04-26T23:02:47.430549+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.397867+0000","last_change":"2024-04-26T23:02:47.397867+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.397867+0000","last_undegraded":"2024-04-26T23:02:47.397867+0000","last_fullsized":"2024-04-26T23:02:47.397867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428260+0000","last_change":"2024-04-26T23:02:47.425906+0000","last_active":"2024-04-26T23:02:47.428260+0000","last_peered":"2024-04-26T23:02:47.428260+0000","last_clean":"2024-04-26T23:02:47.428260+0000","last_became_active":"2024-04-26T23:02:47.425612+0000","last_became_peered":"2024-04-26T23:02:47.425612+0000","last_unstale":"2024-04-26T23:02:47.428260+0000","last_undegraded":"2024-04-26T23:02:47.428260+0000","last_fullsized":"2024-04-26T23:02:47.428260+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:56:06.218292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412607+0000","last_change":"2024-04-26T23:02:47.412607+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412607+0000","last_undegraded":"2024-04-26T23:02:47.412607+0000","last_fullsized":"2024-04-26T23:02:47.412607+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.433025+0000","last_change":"2024-04-26T23:02:47.429056+0000","last_active":"2024-04-26T23:02:47.433025+0000","last_peered":"2024-04-26T23:02:47.433025+0000","last_clean":"2024-04-26T23:02:47.433025+0000","last_became_active":"2024-04-26T23:02:47.428144+0000","last_became_peered":"2024-04-26T23:02:47.428144+0000","last_unstale":"2024-04-26T23:02:47.433025+0000","last_undegraded":"2024-04-26T23:02:47.433025+0000","last_fullsized":"2024-04-26T23:02:47.433025+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:25.938366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408705+0000","last_change":"2024-04-26T23:02:47.408705+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408705+0000","last_undegraded":"2024-04-26T23:02:47.408705+0000","last_fullsized":"2024-04-26T23:02:47.408705+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431539+0000","last_change":"2024-04-26T23:02:47.426625+0000","last_active":"2024-04-26T23:02:47.431539+0000","last_peered":"2024-04-26T23:02:47.431539+0000","last_clean":"2024-04-26T23:02:47.431539+0000","last_became_active":"2024-04-26T23:02:47.426351+0000","last_became_peered":"2024-04-26T23:02:47.426351+0000","last_unstale":"2024-04-26T23:02:47.431539+0000","last_undegraded":"2024-04-26T23:02:47.431539+0000","last_fullsized":"2024-04-26T23:02:47.431539+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.413966+0000","last_change":"2024-04-26T23:02:47.413966+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.413966+0000","last_undegraded":"2024-04-26T23:02:47.413966+0000","last_fullsized":"2024-04-26T23:02:47.413966+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.423709+0000","last_change":"2024-04-26T23:02:47.418555+0000","last_active":"2024-04-26T23:02:47.423709+0000","last_peered":"2024-04-26T23:02:47.423709+0000","last_clean":"2024-04-26T23:02:47.423709+0000","last_became_active":"2024-04-26T23:02:47.418241+0000","last_became_peered":"2024-04-26T23:02:47.418241+0000","last_unstale":"2024-04-26T23:02:47.423709+0000","last_undegraded":"2024-04-26T23:02:47.423709+0000","last_fullsized":"2024-04-26T23:02:47.423709+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:18:53.091762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410588+0000","last_change":"2024-04-26T23:02:47.410588+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410588+0000","last_undegraded":"2024-04-26T23:02:47.410588+0000","last_fullsized":"2024-04-26T23:02:47.410588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430778+0000","last_change":"2024-04-26T23:02:47.426236+0000","last_active":"2024-04-26T23:02:47.430778+0000","last_peered":"2024-04-26T23:02:47.430778+0000","last_clean":"2024-04-26T23:02:47.430778+0000","last_became_active":"2024-04-26T23:02:47.425508+0000","last_became_peered":"2024-04-26T23:02:47.425508+0000","last_unstale":"2024-04-26T23:02:47.430778+0000","last_undegraded":"2024-04-26T23:02:47.430778+0000","last_fullsized":"2024-04-26T23:02:47.430778+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:37:06.151991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411859+0000","last_change":"2024-04-26T23:02:47.411859+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411859+0000","last_undegraded":"2024-04-26T23:02:47.411859+0000","last_fullsized":"2024-04-26T23:02:47.411859+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.425942+0000","last_change":"2024-04-26T23:02:47.422531+0000","last_active":"2024-04-26T23:02:47.425942+0000","last_peered":"2024-04-26T23:02:47.425942+0000","last_clean":"2024-04-26T23:02:47.425942+0000","last_became_active":"2024-04-26T23:02:47.422263+0000","last_became_peered":"2024-04-26T23:02:47.422263+0000","last_unstale":"2024-04-26T23:02:47.425942+0000","last_undegraded":"2024-04-26T23:02:47.425942+0000","last_fullsized":"2024-04-26T23:02:47.425942+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:23.080779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407338+0000","last_change":"2024-04-26T23:02:47.407338+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407338+0000","last_undegraded":"2024-04-26T23:02:47.407338+0000","last_fullsized":"2024-04-26T23:02:47.407338+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428543+0000","last_change":"2024-04-26T23:02:47.425145+0000","last_active":"2024-04-26T23:02:47.428543+0000","last_peered":"2024-04-26T23:02:47.428543+0000","last_clean":"2024-04-26T23:02:47.428543+0000","last_became_active":"2024-04-26T23:02:47.424860+0000","last_became_peered":"2024-04-26T23:02:47.424860+0000","last_unstale":"2024-04-26T23:02:47.428543+0000","last_undegraded":"2024-04-26T23:02:47.428543+0000","last_fullsized":"2024-04-26T23:02:47.428543+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.402648+0000","last_change":"2024-04-26T23:02:47.402648+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.402648+0000","last_undegraded":"2024-04-26T23:02:47.402648+0000","last_fullsized":"2024-04-26T23:02:47.402648+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431674+0000","last_change":"2024-04-26T23:02:47.427171+0000","last_active":"2024-04-26T23:02:47.431674+0000","last_peered":"2024-04-26T23:02:47.431674+0000","last_clean":"2024-04-26T23:02:47.431674+0000","last_became_active":"2024-04-26T23:02:47.426664+0000","last_became_peered":"2024-04-26T23:02:47.426664+0000","last_unstale":"2024-04-26T23:02:47.431674+0000","last_undegraded":"2024-04-26T23:02:47.431674+0000","last_fullsized":"2024-04-26T23:02:47.431674+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:35.059231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.399491+0000","last_change":"2024-04-26T23:02:47.399491+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.399491+0000","last_undegraded":"2024-04-26T23:02:47.399491+0000","last_fullsized":"2024-04-26T23:02:47.399491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431454+0000","last_change":"2024-04-26T23:02:47.425522+0000","last_active":"2024-04-26T23:02:47.431454+0000","last_peered":"2024-04-26T23:02:47.431454+0000","last_clean":"2024-04-26T23:02:47.431454+0000","last_became_active":"2024-04-26T23:02:47.425249+0000","last_became_peered":"2024-04-26T23:02:47.425249+0000","last_unstale":"2024-04-26T23:02:47.431454+0000","last_undegraded":"2024-04-26T23:02:47.431454+0000","last_fullsized":"2024-04-26T23:02:47.431454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412312+0000","last_change":"2024-04-26T23:02:47.412312+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412312+0000","last_undegraded":"2024-04-26T23:02:47.412312+0000","last_fullsized":"2024-04-26T23:02:47.412312+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.419715+0000","last_change":"2024-04-26T23:02:47.414839+0000","last_active":"2024-04-26T23:02:47.419715+0000","last_peered":"2024-04-26T23:02:47.419715+0000","last_clean":"2024-04-26T23:02:47.419715+0000","last_became_active":"2024-04-26T23:02:47.414531+0000","last_became_peered":"2024-04-26T23:02:47.414531+0000","last_unstale":"2024-04-26T23:02:47.419715+0000","last_undegraded":"2024-04-26T23:02:47.419715+0000","last_fullsized":"2024-04-26T23:02:47.419715+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:47:26.056049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409298+0000","last_change":"2024-04-26T23:02:47.409298+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409298+0000","last_undegraded":"2024-04-26T23:02:47.409298+0000","last_fullsized":"2024-04-26T23:02:47.409298+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.420473+0000","last_change":"2024-04-26T23:02:47.417644+0000","last_active":"2024-04-26T23:02:47.420473+0000","last_peered":"2024-04-26T23:02:47.420473+0000","last_clean":"2024-04-26T23:02:47.420473+0000","last_became_active":"2024-04-26T23:02:47.417390+0000","last_became_peered":"2024-04-26T23:02:47.417390+0000","last_unstale":"2024-04-26T23:02:47.420473+0000","last_undegraded":"2024-04-26T23:02:47.420473+0000","last_fullsized":"2024-04-26T23:02:47.420473+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:48:34.048591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409401+0000","last_change":"2024-04-26T23:02:47.409401+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409401+0000","last_undegraded":"2024-04-26T23:02:47.409401+0000","last_fullsized":"2024-04-26T23:02:47.409401+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.429682+0000","last_change":"2024-04-26T23:02:47.425458+0000","last_active":"2024-04-26T23:02:47.429682+0000","last_peered":"2024-04-26T23:02:47.429682+0000","last_clean":"2024-04-26T23:02:47.429682+0000","last_became_active":"2024-04-26T23:02:47.424961+0000","last_became_peered":"2024-04-26T23:02:47.424961+0000","last_unstale":"2024-04-26T23:02:47.429682+0000","last_undegraded":"2024-04-26T23:02:47.429682+0000","last_fullsized":"2024-04-26T23:02:47.429682+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:18:53.091762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409951+0000","last_change":"2024-04-26T23:02:47.409951+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409951+0000","last_undegraded":"2024-04-26T23:02:47.409951+0000","last_fullsized":"2024-04-26T23:02:47.409951+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.427504+0000","last_change":"2024-04-26T23:02:47.422918+0000","last_active":"2024-04-26T23:02:47.427504+0000","last_peered":"2024-04-26T23:02:47.427504+0000","last_clean":"2024-04-26T23:02:47.427504+0000","last_became_active":"2024-04-26T23:02:47.422578+0000","last_became_peered":"2024-04-26T23:02:47.422578+0000","last_unstale":"2024-04-26T23:02:47.427504+0000","last_undegraded":"2024-04-26T23:02:47.427504+0000","last_fullsized":"2024-04-26T23:02:47.427504+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:02:18.489647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407819+0000","last_change":"2024-04-26T23:02:47.407819+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407819+0000","last_undegraded":"2024-04-26T23:02:47.407819+0000","last_fullsized":"2024-04-26T23:02:47.407819+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430568+0000","last_change":"2024-04-26T23:02:47.427276+0000","last_active":"2024-04-26T23:02:47.430568+0000","last_peered":"2024-04-26T23:02:47.430568+0000","last_clean":"2024-04-26T23:02:47.430568+0000","last_became_active":"2024-04-26T23:02:47.426167+0000","last_became_peered":"2024-04-26T23:02:47.426167+0000","last_unstale":"2024-04-26T23:02:47.430568+0000","last_undegraded":"2024-04-26T23:02:47.430568+0000","last_fullsized":"2024-04-26T23:02:47.430568+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.420271+0000","last_change":"2024-04-26T23:02:47.420271+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.420271+0000","last_undegraded":"2024-04-26T23:02:47.420271+0000","last_fullsized":"2024-04-26T23:02:47.420271+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.419691+0000","last_change":"2024-04-26T23:02:47.415885+0000","last_active":"2024-04-26T23:02:47.419691+0000","last_peered":"2024-04-26T23:02:47.419691+0000","last_clean":"2024-04-26T23:02:47.419691+0000","last_became_active":"2024-04-26T23:02:47.415613+0000","last_became_peered":"2024-04-26T23:02:47.415613+0000","last_unstale":"2024-04-26T23:02:47.419691+0000","last_undegraded":"2024-04-26T23:02:47.419691+0000","last_fullsized":"2024-04-26T23:02:47.419691+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:46:24.430302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404728+0000","last_change":"2024-04-26T23:02:47.404728+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404728+0000","last_undegraded":"2024-04-26T23:02:47.404728+0000","last_fullsized":"2024-04-26T23:02:47.404728+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428564+0000","last_change":"2024-04-26T23:02:47.423354+0000","last_active":"2024-04-26T23:02:47.428564+0000","last_peered":"2024-04-26T23:02:47.428564+0000","last_clean":"2024-04-26T23:02:47.428564+0000","last_became_active":"2024-04-26T23:02:47.423058+0000","last_became_peered":"2024-04-26T23:02:47.423058+0000","last_unstale":"2024-04-26T23:02:47.428564+0000","last_undegraded":"2024-04-26T23:02:47.428564+0000","last_fullsized":"2024-04-26T23:02:47.428564+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411594+0000","last_change":"2024-04-26T23:02:47.411594+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411594+0000","last_undegraded":"2024-04-26T23:02:47.411594+0000","last_fullsized":"2024-04-26T23:02:47.411594+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428596+0000","last_change":"2024-04-26T23:02:47.421824+0000","last_active":"2024-04-26T23:02:47.428596+0000","last_peered":"2024-04-26T23:02:47.428596+0000","last_clean":"2024-04-26T23:02:47.428596+0000","last_became_active":"2024-04-26T23:02:47.421537+0000","last_became_peered":"2024-04-26T23:02:47.421537+0000","last_unstale":"2024-04-26T23:02:47.428596+0000","last_undegraded":"2024-04-26T23:02:47.428596+0000","last_fullsized":"2024-04-26T23:02:47.428596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:23.080779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.406234+0000","last_change":"2024-04-26T23:02:47.406234+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.406234+0000","last_undegraded":"2024-04-26T23:02:47.406234+0000","last_fullsized":"2024-04-26T23:02:47.406234+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.425058+0000","last_change":"2024-04-26T23:02:47.422919+0000","last_active":"2024-04-26T23:02:47.425058+0000","last_peered":"2024-04-26T23:02:47.425058+0000","last_clean":"2024-04-26T23:02:47.425058+0000","last_became_active":"2024-04-26T23:02:47.421694+0000","last_became_peered":"2024-04-26T23:02:47.421694+0000","last_unstale":"2024-04-26T23:02:47.425058+0000","last_undegraded":"2024-04-26T23:02:47.425058+0000","last_fullsized":"2024-04-26T23:02:47.425058+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:37:06.151991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.403814+0000","last_change":"2024-04-26T23:02:47.403814+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.403814+0000","last_undegraded":"2024-04-26T23:02:47.403814+0000","last_fullsized":"2024-04-26T23:02:47.403814+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.423560+0000","last_change":"2024-04-26T23:02:47.419127+0000","last_active":"2024-04-26T23:02:47.423560+0000","last_peered":"2024-04-26T23:02:47.423560+0000","last_clean":"2024-04-26T23:02:47.423560+0000","last_became_active":"2024-04-26T23:02:47.417694+0000","last_became_peered":"2024-04-26T23:02:47.417694+0000","last_unstale":"2024-04-26T23:02:47.423560+0000","last_undegraded":"2024-04-26T23:02:47.423560+0000","last_fullsized":"2024-04-26T23:02:47.423560+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.415211+0000","last_change":"2024-04-26T23:02:47.415211+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.415211+0000","last_undegraded":"2024-04-26T23:02:47.415211+0000","last_fullsized":"2024-04-26T23:02:47.415211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.419342+0000","last_change":"2024-04-26T23:02:47.416825+0000","last_active":"2024-04-26T23:02:47.419342+0000","last_peered":"2024-04-26T23:02:47.419342+0000","last_clean":"2024-04-26T23:02:47.419342+0000","last_became_active":"2024-04-26T23:02:47.416571+0000","last_became_peered":"2024-04-26T23:02:47.416571+0000","last_unstale":"2024-04-26T23:02:47.419342+0000","last_undegraded":"2024-04-26T23:02:47.419342+0000","last_fullsized":"2024-04-26T23:02:47.419342+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:25:15.376546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.406994+0000","last_change":"2024-04-26T23:02:47.406994+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.406994+0000","last_undegraded":"2024-04-26T23:02:47.406994+0000","last_fullsized":"2024-04-26T23:02:47.406994+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.433433+0000","last_change":"2024-04-26T23:02:47.432253+0000","last_active":"2024-04-26T23:02:47.433433+0000","last_peered":"2024-04-26T23:02:47.433433+0000","last_clean":"2024-04-26T23:02:47.433433+0000","last_became_active":"2024-04-26T23:02:47.431955+0000","last_became_peered":"2024-04-26T23:02:47.431955+0000","last_unstale":"2024-04-26T23:02:47.433433+0000","last_undegraded":"2024-04-26T23:02:47.433433+0000","last_fullsized":"2024-04-26T23:02:47.433433+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:27.125135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":203,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.413362+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:02:47.413362+0000","last_peered":"2024-04-26T23:02:47.413362+0000","last_clean":"2024-04-26T23:02:47.413362+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:02:47.413362+0000","last_undegraded":"2024-04-26T23:02:47.413362+0000","last_fullsized":"2024-04-26T23:02:47.413362+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":766999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1019904,"data_stored":773666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:02:53.437 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:02:53.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.691+0000 7f8e55c00640 1 Processor -- start 2024-04-26T23:02:53.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.691+0000 7f8e55c00640 1 -- start start 2024-04-26T23:02:53.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.693+0000 7f8e55c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e500af7a0 0x7f8e500afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:53.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.693+0000 7f8e55c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e50071fe0 0x7f8e50073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:53.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.693+0000 7f8e55c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e500d0820 con 0x7f8e5015ce60 2024-04-26T23:02:53.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.693+0000 7f8e55c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f8e500cc450 con 0x7f8e500af7a0 2024-04-26T23:02:53.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.693+0000 7f8e55c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8e501614c0 con 0x7f8e50071fe0 2024-04-26T23:02:53.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.693+0000 7f8e47400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e500af7a0 0x7f8e500afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.694+0000 7f8e47400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e500af7a0 0x7f8e500afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42694/0 (socket says 172.21.15.40:42694) 2024-04-26T23:02:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.694+0000 7f8e47400640 1 -- 172.21.15.40:0/4029840464 learned_addr learned my addr 172.21.15.40:0/4029840464 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.694+0000 7f8e55200640 1 --2- 172.21.15.40:0/4029840464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e50071fe0 0x7f8e50073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.694+0000 7f8e47400640 1 -- 172.21.15.40:0/4029840464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e50071fe0 msgr2=0x7f8e50073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.694+0000 7f8e47400640 1 --2- 172.21.15.40:0/4029840464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e50071fe0 0x7f8e50073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.694+0000 7f8e47400640 1 -- 172.21.15.40:0/4029840464 >> v1:172.21.15.103:6789/0 conn(0x7f8e5015ce60 legacy=0x7f8e500ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.694+0000 7f8e47400640 1 -- 172.21.15.40:0/4029840464 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e50161760 con 0x7f8e500af7a0 2024-04-26T23:02:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.694+0000 7f8e55200640 1 --2- 172.21.15.40:0/4029840464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e50071fe0 0x7f8e50073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:53.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.695+0000 7f8e47400640 1 --2- 172.21.15.40:0/4029840464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e500af7a0 0x7f8e500afc60 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009e50 tx=0x7f8e3802a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88dfd1e9191f404b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:53.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.696+0000 7f8e4f400640 1 -- 172.21.15.40:0/4029840464 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e38036070 con 0x7f8e500af7a0 2024-04-26T23:02:53.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.696+0000 7f8e4f400640 1 -- 172.21.15.40:0/4029840464 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e38031600 con 0x7f8e500af7a0 2024-04-26T23:02:53.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.696+0000 7f8e4f400640 1 -- 172.21.15.40:0/4029840464 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e380319d0 con 0x7f8e500af7a0 2024-04-26T23:02:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.696+0000 7f8e55c00640 1 -- 172.21.15.40:0/4029840464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e500af7a0 msgr2=0x7f8e500afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.696+0000 7f8e55c00640 1 --2- 172.21.15.40:0/4029840464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e500af7a0 0x7f8e500afc60 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009e50 tx=0x7f8e3802a640 comp rx=0 tx=0).stop 2024-04-26T23:02:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.696+0000 7f8e55c00640 1 -- 172.21.15.40:0/4029840464 shutdown_connections 2024-04-26T23:02:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.696+0000 7f8e55c00640 1 --2- 172.21.15.40:0/4029840464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e50071fe0 0x7f8e50073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.696+0000 7f8e55c00640 1 --2- 172.21.15.40:0/4029840464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e500af7a0 0x7f8e500afc60 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.696+0000 7f8e55c00640 1 -- 172.21.15.40:0/4029840464 >> 172.21.15.40:0/4029840464 conn(0x7f8e500d0a10 msgr2=0x7f8e500d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.696+0000 7f8e55c00640 1 -- 172.21.15.40:0/4029840464 shutdown_connections 2024-04-26T23:02:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.697+0000 7f8e55c00640 1 -- 172.21.15.40:0/4029840464 wait complete. 2024-04-26T23:02:53.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.699+0000 7f8e55c00640 1 Processor -- start 2024-04-26T23:02:53.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.699+0000 7f8e55c00640 1 -- start start 2024-04-26T23:02:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.700+0000 7f8e55c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e50071fe0 0x7f8e501f7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.700+0000 7f8e55c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e500af7a0 0x7f8e500c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.700+0000 7f8e55c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e5015ce60 0x7f8e5014bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.700+0000 7f8e55c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8e500cc1f0 con 0x7f8e5015ce60 2024-04-26T23:02:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.700+0000 7f8e55c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e500cc070 con 0x7f8e50071fe0 2024-04-26T23:02:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.700+0000 7f8e55c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8e50073ca0 con 0x7f8e500af7a0 2024-04-26T23:02:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.700+0000 7f8e55200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e5015ce60 0x7f8e5014bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.701+0000 7f8e55200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e5015ce60 0x7f8e5014bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42710/0 (socket says 172.21.15.40:42710) 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.701+0000 7f8e55200640 1 -- 172.21.15.40:0/2050667092 learned_addr learned my addr 172.21.15.40:0/2050667092 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.701+0000 7f8e4fe00640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e50071fe0 0x7f8e501f7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.701+0000 7f8e47400640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e500af7a0 0x7f8e500c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.701+0000 7f8e55200640 1 -- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e500af7a0 msgr2=0x7f8e500c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.701+0000 7f8e55200640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e500af7a0 0x7f8e500c7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.701+0000 7f8e55200640 1 -- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e50071fe0 msgr2=0x7f8e501f7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.701+0000 7f8e55200640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e50071fe0 0x7f8e501f7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.701+0000 7f8e55200640 1 -- 172.21.15.40:0/2050667092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e500c8310 con 0x7f8e5015ce60 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.701+0000 7f8e4fe00640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e50071fe0 0x7f8e501f7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.701+0000 7f8e55200640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e5015ce60 0x7f8e5014bc00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800c7f0 tx=0x7f8e4800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.702+0000 7f8e4cc00640 1 -- 172.21.15.40:0/2050667092 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e48015070 con 0x7f8e5015ce60 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.702+0000 7f8e4cc00640 1 -- 172.21.15.40:0/2050667092 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e480077c0 con 0x7f8e5015ce60 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.702+0000 7f8e4cc00640 1 -- 172.21.15.40:0/2050667092 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e48007bf0 con 0x7f8e5015ce60 2024-04-26T23:02:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.702+0000 7f8e55c00640 1 -- 172.21.15.40:0/2050667092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e500c85f0 con 0x7f8e5015ce60 2024-04-26T23:02:53.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.702+0000 7f8e55c00640 1 -- 172.21.15.40:0/2050667092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e501f6b40 con 0x7f8e5015ce60 2024-04-26T23:02:53.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.703+0000 7f8e55c00640 1 -- 172.21.15.40:0/2050667092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e500cd2b0 con 0x7f8e5015ce60 2024-04-26T23:02:53.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.704+0000 7f8e4cc00640 1 -- 172.21.15.40:0/2050667092 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8e48018070 con 0x7f8e5015ce60 2024-04-26T23:02:53.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.705+0000 7f8e4cc00640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8e2c06c8f0 0x7f8e2c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:53.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.705+0000 7f8e4cc00640 1 -- 172.21.15.40:0/2050667092 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f8e4808be80 con 0x7f8e5015ce60 2024-04-26T23:02:53.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.706+0000 7f8e4fe00640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8e2c06c8f0 0x7f8e2c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:53.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.707+0000 7f8e4fe00640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8e2c06c8f0 0x7f8e2c06edb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8e4000a830 tx=0x7f8e40008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:53.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.710+0000 7f8e4cc00640 1 -- 172.21.15.40:0/2050667092 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8e500cd2b0 con 0x7f8e5015ce60 2024-04-26T23:02:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.914+0000 7f8e55c00640 1 -- 172.21.15.40:0/2050667092 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8e50069370 con 0x7f8e2c06c8f0 2024-04-26T23:02:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.933+0000 7f8e4cc00640 1 -- 172.21.15.40:0/2050667092 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246635 (secure 0 0 0) 0x7f8e50069370 con 0x7f8e2c06c8f0 2024-04-26T23:02:53.937 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:02:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.940+0000 7f8e55c00640 1 -- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8e2c06c8f0 msgr2=0x7f8e2c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.940+0000 7f8e55c00640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8e2c06c8f0 0x7f8e2c06edb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8e4000a830 tx=0x7f8e40008040 comp rx=0 tx=0).stop 2024-04-26T23:02:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.940+0000 7f8e55c00640 1 -- 172.21.15.40:0/2050667092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e5015ce60 msgr2=0x7f8e5014bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.940+0000 7f8e55c00640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e5015ce60 0x7f8e5014bc00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800c7f0 tx=0x7f8e4800ccb0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.940+0000 7f8e55c00640 1 -- 172.21.15.40:0/2050667092 shutdown_connections 2024-04-26T23:02:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.940+0000 7f8e55c00640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8e2c06c8f0 0x7f8e2c06edb0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.940+0000 7f8e55c00640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e5015ce60 0x7f8e5014bc00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.940+0000 7f8e55c00640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e500af7a0 0x7f8e500c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.940+0000 7f8e55c00640 1 --2- 172.21.15.40:0/2050667092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e50071fe0 0x7f8e501f7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.940+0000 7f8e55c00640 1 -- 172.21.15.40:0/2050667092 >> 172.21.15.40:0/2050667092 conn(0x7f8e500d0a10 msgr2=0x7f8e500ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.940+0000 7f8e55c00640 1 -- 172.21.15.40:0/2050667092 shutdown_connections 2024-04-26T23:02:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:53.940+0000 7f8e55c00640 1 -- 172.21.15.40:0/2050667092 wait complete. 2024-04-26T23:02:53.959 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-04-26T23:02:53.182976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220620,"kb_used_data":4452,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754100,"statfs":{"total":773094113280,"available":772868198400,"internally_reserved":0,"allocated":4558848,"data_stored":2627119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001782"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409049+0000","last_change":"2024-04-26T23:02:47.409049+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409049+0000","last_undegraded":"2024-04-26T23:02:47.409049+0000","last_fullsized":"2024-04-26T23:02:47.409049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409471+0000","last_change":"2024-04-26T23:02:47.409471+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409471+0000","last_undegraded":"2024-04-26T23:02:47.409471+0000","last_fullsized":"2024-04-26T23:02:47.409471+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410825+0000","last_change":"2024-04-26T23:02:47.410825+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410825+0000","last_undegraded":"2024-04-26T23:02:47.410825+0000","last_fullsized":"2024-04-26T23:02:47.410825+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.402588+0000","last_change":"2024-04-26T23:02:47.402588+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.402588+0000","last_undegraded":"2024-04-26T23:02:47.402588+0000","last_fullsized":"2024-04-26T23:02:47.402588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400758+0000","last_change":"2024-04-26T23:02:47.400758+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400758+0000","last_undegraded":"2024-04-26T23:02:47.400758+0000","last_fullsized":"2024-04-26T23:02:47.400758+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407437+0000","last_change":"2024-04-26T23:02:47.407437+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407437+0000","last_undegraded":"2024-04-26T23:02:47.407437+0000","last_fullsized":"2024-04-26T23:02:47.407437+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409272+0000","last_change":"2024-04-26T23:02:47.409272+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409272+0000","last_undegraded":"2024-04-26T23:02:47.409272+0000","last_fullsized":"2024-04-26T23:02:47.409272+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408667+0000","last_change":"2024-04-26T23:02:47.408667+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408667+0000","last_undegraded":"2024-04-26T23:02:47.408667+0000","last_fullsized":"2024-04-26T23:02:47.408667+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.406089+0000","last_change":"2024-04-26T23:02:47.406089+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.406089+0000","last_undegraded":"2024-04-26T23:02:47.406089+0000","last_fullsized":"2024-04-26T23:02:47.406089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409393+0000","last_change":"2024-04-26T23:02:47.409393+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409393+0000","last_undegraded":"2024-04-26T23:02:47.409393+0000","last_fullsized":"2024-04-26T23:02:47.409393+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408476+0000","last_change":"2024-04-26T23:02:47.408476+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408476+0000","last_undegraded":"2024-04-26T23:02:47.408476+0000","last_fullsized":"2024-04-26T23:02:47.408476+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411813+0000","last_change":"2024-04-26T23:02:47.411813+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411813+0000","last_undegraded":"2024-04-26T23:02:47.411813+0000","last_fullsized":"2024-04-26T23:02:47.411813+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412901+0000","last_change":"2024-04-26T23:02:47.412901+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412901+0000","last_undegraded":"2024-04-26T23:02:47.412901+0000","last_fullsized":"2024-04-26T23:02:47.412901+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404645+0000","last_change":"2024-04-26T23:02:47.404645+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404645+0000","last_undegraded":"2024-04-26T23:02:47.404645+0000","last_fullsized":"2024-04-26T23:02:47.404645+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408580+0000","last_change":"2024-04-26T23:02:47.408580+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408580+0000","last_undegraded":"2024-04-26T23:02:47.408580+0000","last_fullsized":"2024-04-26T23:02:47.408580+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412732+0000","last_change":"2024-04-26T23:02:47.412732+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412732+0000","last_undegraded":"2024-04-26T23:02:47.412732+0000","last_fullsized":"2024-04-26T23:02:47.412732+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.417759+0000","last_change":"2024-04-26T23:02:47.417759+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.417759+0000","last_undegraded":"2024-04-26T23:02:47.417759+0000","last_fullsized":"2024-04-26T23:02:47.417759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.401677+0000","last_change":"2024-04-26T23:02:47.401677+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.401677+0000","last_undegraded":"2024-04-26T23:02:47.401677+0000","last_fullsized":"2024-04-26T23:02:47.401677+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400720+0000","last_change":"2024-04-26T23:02:47.400720+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400720+0000","last_undegraded":"2024-04-26T23:02:47.400720+0000","last_fullsized":"2024-04-26T23:02:47.400720+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.401112+0000","last_change":"2024-04-26T23:02:47.401112+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.401112+0000","last_undegraded":"2024-04-26T23:02:47.401112+0000","last_fullsized":"2024-04-26T23:02:47.401112+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409722+0000","last_change":"2024-04-26T23:02:47.409722+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409722+0000","last_undegraded":"2024-04-26T23:02:47.409722+0000","last_fullsized":"2024-04-26T23:02:47.409722+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.403188+0000","last_change":"2024-04-26T23:02:47.403188+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.403188+0000","last_undegraded":"2024-04-26T23:02:47.403188+0000","last_fullsized":"2024-04-26T23:02:47.403188+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411186+0000","last_change":"2024-04-26T23:02:47.411186+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411186+0000","last_undegraded":"2024-04-26T23:02:47.411186+0000","last_fullsized":"2024-04-26T23:02:47.411186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400306+0000","last_change":"2024-04-26T23:02:47.400306+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400306+0000","last_undegraded":"2024-04-26T23:02:47.400306+0000","last_fullsized":"2024-04-26T23:02:47.400306+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404031+0000","last_change":"2024-04-26T23:02:47.404031+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404031+0000","last_undegraded":"2024-04-26T23:02:47.404031+0000","last_fullsized":"2024-04-26T23:02:47.404031+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408158+0000","last_change":"2024-04-26T23:02:47.408158+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408158+0000","last_undegraded":"2024-04-26T23:02:47.408158+0000","last_fullsized":"2024-04-26T23:02:47.408158+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407444+0000","last_change":"2024-04-26T23:02:47.407444+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407444+0000","last_undegraded":"2024-04-26T23:02:47.407444+0000","last_fullsized":"2024-04-26T23:02:47.407444+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408555+0000","last_change":"2024-04-26T23:02:47.408555+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408555+0000","last_undegraded":"2024-04-26T23:02:47.408555+0000","last_fullsized":"2024-04-26T23:02:47.408555+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400673+0000","last_change":"2024-04-26T23:02:47.400673+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400673+0000","last_undegraded":"2024-04-26T23:02:47.400673+0000","last_fullsized":"2024-04-26T23:02:47.400673+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.399159+0000","last_change":"2024-04-26T23:02:47.399159+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.399159+0000","last_undegraded":"2024-04-26T23:02:47.399159+0000","last_fullsized":"2024-04-26T23:02:47.399159+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410706+0000","last_change":"2024-04-26T23:02:47.410706+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410706+0000","last_undegraded":"2024-04-26T23:02:47.410706+0000","last_fullsized":"2024-04-26T23:02:47.410706+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404746+0000","last_change":"2024-04-26T23:02:47.404746+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404746+0000","last_undegraded":"2024-04-26T23:02:47.404746+0000","last_fullsized":"2024-04-26T23:02:47.404746+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.419764+0000","last_change":"2024-04-26T23:02:47.419764+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.419764+0000","last_undegraded":"2024-04-26T23:02:47.419764+0000","last_fullsized":"2024-04-26T23:02:47.419764+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.422997+0000","last_change":"2024-04-26T23:02:47.416086+0000","last_active":"2024-04-26T23:02:47.422997+0000","last_peered":"2024-04-26T23:02:47.422997+0000","last_clean":"2024-04-26T23:02:47.422997+0000","last_became_active":"2024-04-26T23:02:47.415794+0000","last_became_peered":"2024-04-26T23:02:47.415794+0000","last_unstale":"2024-04-26T23:02:47.422997+0000","last_undegraded":"2024-04-26T23:02:47.422997+0000","last_fullsized":"2024-04-26T23:02:47.422997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:54:53.447069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410136+0000","last_change":"2024-04-26T23:02:47.410136+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410136+0000","last_undegraded":"2024-04-26T23:02:47.410136+0000","last_fullsized":"2024-04-26T23:02:47.410136+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428834+0000","last_change":"2024-04-26T23:02:47.422501+0000","last_active":"2024-04-26T23:02:47.428834+0000","last_peered":"2024-04-26T23:02:47.428834+0000","last_clean":"2024-04-26T23:02:47.428834+0000","last_became_active":"2024-04-26T23:02:47.422240+0000","last_became_peered":"2024-04-26T23:02:47.422240+0000","last_unstale":"2024-04-26T23:02:47.428834+0000","last_undegraded":"2024-04-26T23:02:47.428834+0000","last_fullsized":"2024-04-26T23:02:47.428834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.398614+0000","last_change":"2024-04-26T23:02:47.398614+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.398614+0000","last_undegraded":"2024-04-26T23:02:47.398614+0000","last_fullsized":"2024-04-26T23:02:47.398614+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.432856+0000","last_change":"2024-04-26T23:02:47.427135+0000","last_active":"2024-04-26T23:02:47.432856+0000","last_peered":"2024-04-26T23:02:47.432856+0000","last_clean":"2024-04-26T23:02:47.432856+0000","last_became_active":"2024-04-26T23:02:47.425928+0000","last_became_peered":"2024-04-26T23:02:47.425928+0000","last_unstale":"2024-04-26T23:02:47.432856+0000","last_undegraded":"2024-04-26T23:02:47.432856+0000","last_fullsized":"2024-04-26T23:02:47.432856+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409517+0000","last_change":"2024-04-26T23:02:47.409517+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409517+0000","last_undegraded":"2024-04-26T23:02:47.409517+0000","last_fullsized":"2024-04-26T23:02:47.409517+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.427084+0000","last_change":"2024-04-26T23:02:47.423448+0000","last_active":"2024-04-26T23:02:47.427084+0000","last_peered":"2024-04-26T23:02:47.427084+0000","last_clean":"2024-04-26T23:02:47.427084+0000","last_became_active":"2024-04-26T23:02:47.423145+0000","last_became_peered":"2024-04-26T23:02:47.423145+0000","last_unstale":"2024-04-26T23:02:47.427084+0000","last_undegraded":"2024-04-26T23:02:47.427084+0000","last_fullsized":"2024-04-26T23:02:47.427084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.415281+0000","last_change":"2024-04-26T23:02:47.415281+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.415281+0000","last_undegraded":"2024-04-26T23:02:47.415281+0000","last_fullsized":"2024-04-26T23:02:47.415281+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431170+0000","last_change":"2024-04-26T23:02:47.429506+0000","last_active":"2024-04-26T23:02:47.431170+0000","last_peered":"2024-04-26T23:02:47.431170+0000","last_clean":"2024-04-26T23:02:47.431170+0000","last_became_active":"2024-04-26T23:02:47.429131+0000","last_became_peered":"2024-04-26T23:02:47.429131+0000","last_unstale":"2024-04-26T23:02:47.431170+0000","last_undegraded":"2024-04-26T23:02:47.431170+0000","last_fullsized":"2024-04-26T23:02:47.431170+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:28.822626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407967+0000","last_change":"2024-04-26T23:02:47.407967+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407967+0000","last_undegraded":"2024-04-26T23:02:47.407967+0000","last_fullsized":"2024-04-26T23:02:47.407967+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430956+0000","last_change":"2024-04-26T23:02:47.428296+0000","last_active":"2024-04-26T23:02:47.430956+0000","last_peered":"2024-04-26T23:02:47.430956+0000","last_clean":"2024-04-26T23:02:47.430956+0000","last_became_active":"2024-04-26T23:02:47.427980+0000","last_became_peered":"2024-04-26T23:02:47.427980+0000","last_unstale":"2024-04-26T23:02:47.430956+0000","last_undegraded":"2024-04-26T23:02:47.430956+0000","last_fullsized":"2024-04-26T23:02:47.430956+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:24:26.243323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.397241+0000","last_change":"2024-04-26T23:02:47.397241+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.397241+0000","last_undegraded":"2024-04-26T23:02:47.397241+0000","last_fullsized":"2024-04-26T23:02:47.397241+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.427966+0000","last_change":"2024-04-26T23:02:47.422927+0000","last_active":"2024-04-26T23:02:47.427966+0000","last_peered":"2024-04-26T23:02:47.427966+0000","last_clean":"2024-04-26T23:02:47.427966+0000","last_became_active":"2024-04-26T23:02:47.422616+0000","last_became_peered":"2024-04-26T23:02:47.422616+0000","last_unstale":"2024-04-26T23:02:47.427966+0000","last_undegraded":"2024-04-26T23:02:47.427966+0000","last_fullsized":"2024-04-26T23:02:47.427966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:23.080779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411122+0000","last_change":"2024-04-26T23:02:47.411122+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411122+0000","last_undegraded":"2024-04-26T23:02:47.411122+0000","last_fullsized":"2024-04-26T23:02:47.411122+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.429074+0000","last_change":"2024-04-26T23:02:47.426665+0000","last_active":"2024-04-26T23:02:47.429074+0000","last_peered":"2024-04-26T23:02:47.429074+0000","last_clean":"2024-04-26T23:02:47.429074+0000","last_became_active":"2024-04-26T23:02:47.426408+0000","last_became_peered":"2024-04-26T23:02:47.426408+0000","last_unstale":"2024-04-26T23:02:47.429074+0000","last_undegraded":"2024-04-26T23:02:47.429074+0000","last_fullsized":"2024-04-26T23:02:47.429074+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:28.822626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400911+0000","last_change":"2024-04-26T23:02:47.400911+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400911+0000","last_undegraded":"2024-04-26T23:02:47.400911+0000","last_fullsized":"2024-04-26T23:02:47.400911+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.433081+0000","last_change":"2024-04-26T23:02:47.429805+0000","last_active":"2024-04-26T23:02:47.433081+0000","last_peered":"2024-04-26T23:02:47.433081+0000","last_clean":"2024-04-26T23:02:47.433081+0000","last_became_active":"2024-04-26T23:02:47.429541+0000","last_became_peered":"2024-04-26T23:02:47.429541+0000","last_unstale":"2024-04-26T23:02:47.433081+0000","last_undegraded":"2024-04-26T23:02:47.433081+0000","last_fullsized":"2024-04-26T23:02:47.433081+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:01.704078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.420803+0000","last_change":"2024-04-26T23:02:47.420803+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.420803+0000","last_undegraded":"2024-04-26T23:02:47.420803+0000","last_fullsized":"2024-04-26T23:02:47.420803+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.425979+0000","last_change":"2024-04-26T23:02:47.424076+0000","last_active":"2024-04-26T23:02:47.425979+0000","last_peered":"2024-04-26T23:02:47.425979+0000","last_clean":"2024-04-26T23:02:47.425979+0000","last_became_active":"2024-04-26T23:02:47.420944+0000","last_became_peered":"2024-04-26T23:02:47.420944+0000","last_unstale":"2024-04-26T23:02:47.425979+0000","last_undegraded":"2024-04-26T23:02:47.425979+0000","last_fullsized":"2024-04-26T23:02:47.425979+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409689+0000","last_change":"2024-04-26T23:02:47.409689+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409689+0000","last_undegraded":"2024-04-26T23:02:47.409689+0000","last_fullsized":"2024-04-26T23:02:47.409689+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430549+0000","last_change":"2024-04-26T23:02:47.426825+0000","last_active":"2024-04-26T23:02:47.430549+0000","last_peered":"2024-04-26T23:02:47.430549+0000","last_clean":"2024-04-26T23:02:47.430549+0000","last_became_active":"2024-04-26T23:02:47.426475+0000","last_became_peered":"2024-04-26T23:02:47.426475+0000","last_unstale":"2024-04-26T23:02:47.430549+0000","last_undegraded":"2024-04-26T23:02:47.430549+0000","last_fullsized":"2024-04-26T23:02:47.430549+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.397867+0000","last_change":"2024-04-26T23:02:47.397867+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.397867+0000","last_undegraded":"2024-04-26T23:02:47.397867+0000","last_fullsized":"2024-04-26T23:02:47.397867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428260+0000","last_change":"2024-04-26T23:02:47.425906+0000","last_active":"2024-04-26T23:02:47.428260+0000","last_peered":"2024-04-26T23:02:47.428260+0000","last_clean":"2024-04-26T23:02:47.428260+0000","last_became_active":"2024-04-26T23:02:47.425612+0000","last_became_peered":"2024-04-26T23:02:47.425612+0000","last_unstale":"2024-04-26T23:02:47.428260+0000","last_undegraded":"2024-04-26T23:02:47.428260+0000","last_fullsized":"2024-04-26T23:02:47.428260+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:56:06.218292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412607+0000","last_change":"2024-04-26T23:02:47.412607+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412607+0000","last_undegraded":"2024-04-26T23:02:47.412607+0000","last_fullsized":"2024-04-26T23:02:47.412607+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.433025+0000","last_change":"2024-04-26T23:02:47.429056+0000","last_active":"2024-04-26T23:02:47.433025+0000","last_peered":"2024-04-26T23:02:47.433025+0000","last_clean":"2024-04-26T23:02:47.433025+0000","last_became_active":"2024-04-26T23:02:47.428144+0000","last_became_peered":"2024-04-26T23:02:47.428144+0000","last_unstale":"2024-04-26T23:02:47.433025+0000","last_undegraded":"2024-04-26T23:02:47.433025+0000","last_fullsized":"2024-04-26T23:02:47.433025+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:25.938366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408705+0000","last_change":"2024-04-26T23:02:47.408705+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408705+0000","last_undegraded":"2024-04-26T23:02:47.408705+0000","last_fullsized":"2024-04-26T23:02:47.408705+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431539+0000","last_change":"2024-04-26T23:02:47.426625+0000","last_active":"2024-04-26T23:02:47.431539+0000","last_peered":"2024-04-26T23:02:47.431539+0000","last_clean":"2024-04-26T23:02:47.431539+0000","last_became_active":"2024-04-26T23:02:47.426351+0000","last_became_peered":"2024-04-26T23:02:47.426351+0000","last_unstale":"2024-04-26T23:02:47.431539+0000","last_undegraded":"2024-04-26T23:02:47.431539+0000","last_fullsized":"2024-04-26T23:02:47.431539+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.413966+0000","last_change":"2024-04-26T23:02:47.413966+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.413966+0000","last_undegraded":"2024-04-26T23:02:47.413966+0000","last_fullsized":"2024-04-26T23:02:47.413966+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.423709+0000","last_change":"2024-04-26T23:02:47.418555+0000","last_active":"2024-04-26T23:02:47.423709+0000","last_peered":"2024-04-26T23:02:47.423709+0000","last_clean":"2024-04-26T23:02:47.423709+0000","last_became_active":"2024-04-26T23:02:47.418241+0000","last_became_peered":"2024-04-26T23:02:47.418241+0000","last_unstale":"2024-04-26T23:02:47.423709+0000","last_undegraded":"2024-04-26T23:02:47.423709+0000","last_fullsized":"2024-04-26T23:02:47.423709+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:18:53.091762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410588+0000","last_change":"2024-04-26T23:02:47.410588+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410588+0000","last_undegraded":"2024-04-26T23:02:47.410588+0000","last_fullsized":"2024-04-26T23:02:47.410588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430778+0000","last_change":"2024-04-26T23:02:47.426236+0000","last_active":"2024-04-26T23:02:47.430778+0000","last_peered":"2024-04-26T23:02:47.430778+0000","last_clean":"2024-04-26T23:02:47.430778+0000","last_became_active":"2024-04-26T23:02:47.425508+0000","last_became_peered":"2024-04-26T23:02:47.425508+0000","last_unstale":"2024-04-26T23:02:47.430778+0000","last_undegraded":"2024-04-26T23:02:47.430778+0000","last_fullsized":"2024-04-26T23:02:47.430778+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:37:06.151991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411859+0000","last_change":"2024-04-26T23:02:47.411859+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411859+0000","last_undegraded":"2024-04-26T23:02:47.411859+0000","last_fullsized":"2024-04-26T23:02:47.411859+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.425942+0000","last_change":"2024-04-26T23:02:47.422531+0000","last_active":"2024-04-26T23:02:47.425942+0000","last_peered":"2024-04-26T23:02:47.425942+0000","last_clean":"2024-04-26T23:02:47.425942+0000","last_became_active":"2024-04-26T23:02:47.422263+0000","last_became_peered":"2024-04-26T23:02:47.422263+0000","last_unstale":"2024-04-26T23:02:47.425942+0000","last_undegraded":"2024-04-26T23:02:47.425942+0000","last_fullsized":"2024-04-26T23:02:47.425942+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:23.080779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407338+0000","last_change":"2024-04-26T23:02:47.407338+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407338+0000","last_undegraded":"2024-04-26T23:02:47.407338+0000","last_fullsized":"2024-04-26T23:02:47.407338+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428543+0000","last_change":"2024-04-26T23:02:47.425145+0000","last_active":"2024-04-26T23:02:47.428543+0000","last_peered":"2024-04-26T23:02:47.428543+0000","last_clean":"2024-04-26T23:02:47.428543+0000","last_became_active":"2024-04-26T23:02:47.424860+0000","last_became_peered":"2024-04-26T23:02:47.424860+0000","last_unstale":"2024-04-26T23:02:47.428543+0000","last_undegraded":"2024-04-26T23:02:47.428543+0000","last_fullsized":"2024-04-26T23:02:47.428543+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.402648+0000","last_change":"2024-04-26T23:02:47.402648+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.402648+0000","last_undegraded":"2024-04-26T23:02:47.402648+0000","last_fullsized":"2024-04-26T23:02:47.402648+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431674+0000","last_change":"2024-04-26T23:02:47.427171+0000","last_active":"2024-04-26T23:02:47.431674+0000","last_peered":"2024-04-26T23:02:47.431674+0000","last_clean":"2024-04-26T23:02:47.431674+0000","last_became_active":"2024-04-26T23:02:47.426664+0000","last_became_peered":"2024-04-26T23:02:47.426664+0000","last_unstale":"2024-04-26T23:02:47.431674+0000","last_undegraded":"2024-04-26T23:02:47.431674+0000","last_fullsized":"2024-04-26T23:02:47.431674+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:35.059231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.399491+0000","last_change":"2024-04-26T23:02:47.399491+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.399491+0000","last_undegraded":"2024-04-26T23:02:47.399491+0000","last_fullsized":"2024-04-26T23:02:47.399491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431454+0000","last_change":"2024-04-26T23:02:47.425522+0000","last_active":"2024-04-26T23:02:47.431454+0000","last_peered":"2024-04-26T23:02:47.431454+0000","last_clean":"2024-04-26T23:02:47.431454+0000","last_became_active":"2024-04-26T23:02:47.425249+0000","last_became_peered":"2024-04-26T23:02:47.425249+0000","last_unstale":"2024-04-26T23:02:47.431454+0000","last_undegraded":"2024-04-26T23:02:47.431454+0000","last_fullsized":"2024-04-26T23:02:47.431454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412312+0000","last_change":"2024-04-26T23:02:47.412312+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412312+0000","last_undegraded":"2024-04-26T23:02:47.412312+0000","last_fullsized":"2024-04-26T23:02:47.412312+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.419715+0000","last_change":"2024-04-26T23:02:47.414839+0000","last_active":"2024-04-26T23:02:47.419715+0000","last_peered":"2024-04-26T23:02:47.419715+0000","last_clean":"2024-04-26T23:02:47.419715+0000","last_became_active":"2024-04-26T23:02:47.414531+0000","last_became_peered":"2024-04-26T23:02:47.414531+0000","last_unstale":"2024-04-26T23:02:47.419715+0000","last_undegraded":"2024-04-26T23:02:47.419715+0000","last_fullsized":"2024-04-26T23:02:47.419715+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:47:26.056049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409298+0000","last_change":"2024-04-26T23:02:47.409298+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409298+0000","last_undegraded":"2024-04-26T23:02:47.409298+0000","last_fullsized":"2024-04-26T23:02:47.409298+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.420473+0000","last_change":"2024-04-26T23:02:47.417644+0000","last_active":"2024-04-26T23:02:47.420473+0000","last_peered":"2024-04-26T23:02:47.420473+0000","last_clean":"2024-04-26T23:02:47.420473+0000","last_became_active":"2024-04-26T23:02:47.417390+0000","last_became_peered":"2024-04-26T23:02:47.417390+0000","last_unstale":"2024-04-26T23:02:47.420473+0000","last_undegraded":"2024-04-26T23:02:47.420473+0000","last_fullsized":"2024-04-26T23:02:47.420473+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:48:34.048591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409401+0000","last_change":"2024-04-26T23:02:47.409401+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409401+0000","last_undegraded":"2024-04-26T23:02:47.409401+0000","last_fullsized":"2024-04-26T23:02:47.409401+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.429682+0000","last_change":"2024-04-26T23:02:47.425458+0000","last_active":"2024-04-26T23:02:47.429682+0000","last_peered":"2024-04-26T23:02:47.429682+0000","last_clean":"2024-04-26T23:02:47.429682+0000","last_became_active":"2024-04-26T23:02:47.424961+0000","last_became_peered":"2024-04-26T23:02:47.424961+0000","last_unstale":"2024-04-26T23:02:47.429682+0000","last_undegraded":"2024-04-26T23:02:47.429682+0000","last_fullsized":"2024-04-26T23:02:47.429682+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:18:53.091762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409951+0000","last_change":"2024-04-26T23:02:47.409951+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409951+0000","last_undegraded":"2024-04-26T23:02:47.409951+0000","last_fullsized":"2024-04-26T23:02:47.409951+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.427504+0000","last_change":"2024-04-26T23:02:47.422918+0000","last_active":"2024-04-26T23:02:47.427504+0000","last_peered":"2024-04-26T23:02:47.427504+0000","last_clean":"2024-04-26T23:02:47.427504+0000","last_became_active":"2024-04-26T23:02:47.422578+0000","last_became_peered":"2024-04-26T23:02:47.422578+0000","last_unstale":"2024-04-26T23:02:47.427504+0000","last_undegraded":"2024-04-26T23:02:47.427504+0000","last_fullsized":"2024-04-26T23:02:47.427504+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:02:18.489647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407819+0000","last_change":"2024-04-26T23:02:47.407819+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407819+0000","last_undegraded":"2024-04-26T23:02:47.407819+0000","last_fullsized":"2024-04-26T23:02:47.407819+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430568+0000","last_change":"2024-04-26T23:02:47.427276+0000","last_active":"2024-04-26T23:02:47.430568+0000","last_peered":"2024-04-26T23:02:47.430568+0000","last_clean":"2024-04-26T23:02:47.430568+0000","last_became_active":"2024-04-26T23:02:47.426167+0000","last_became_peered":"2024-04-26T23:02:47.426167+0000","last_unstale":"2024-04-26T23:02:47.430568+0000","last_undegraded":"2024-04-26T23:02:47.430568+0000","last_fullsized":"2024-04-26T23:02:47.430568+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.420271+0000","last_change":"2024-04-26T23:02:47.420271+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.420271+0000","last_undegraded":"2024-04-26T23:02:47.420271+0000","last_fullsized":"2024-04-26T23:02:47.420271+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.419691+0000","last_change":"2024-04-26T23:02:47.415885+0000","last_active":"2024-04-26T23:02:47.419691+0000","last_peered":"2024-04-26T23:02:47.419691+0000","last_clean":"2024-04-26T23:02:47.419691+0000","last_became_active":"2024-04-26T23:02:47.415613+0000","last_became_peered":"2024-04-26T23:02:47.415613+0000","last_unstale":"2024-04-26T23:02:47.419691+0000","last_undegraded":"2024-04-26T23:02:47.419691+0000","last_fullsized":"2024-04-26T23:02:47.419691+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:46:24.430302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404728+0000","last_change":"2024-04-26T23:02:47.404728+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404728+0000","last_undegraded":"2024-04-26T23:02:47.404728+0000","last_fullsized":"2024-04-26T23:02:47.404728+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428564+0000","last_change":"2024-04-26T23:02:47.423354+0000","last_active":"2024-04-26T23:02:47.428564+0000","last_peered":"2024-04-26T23:02:47.428564+0000","last_clean":"2024-04-26T23:02:47.428564+0000","last_became_active":"2024-04-26T23:02:47.423058+0000","last_became_peered":"2024-04-26T23:02:47.423058+0000","last_unstale":"2024-04-26T23:02:47.428564+0000","last_undegraded":"2024-04-26T23:02:47.428564+0000","last_fullsized":"2024-04-26T23:02:47.428564+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411594+0000","last_change":"2024-04-26T23:02:47.411594+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411594+0000","last_undegraded":"2024-04-26T23:02:47.411594+0000","last_fullsized":"2024-04-26T23:02:47.411594+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428596+0000","last_change":"2024-04-26T23:02:47.421824+0000","last_active":"2024-04-26T23:02:47.428596+0000","last_peered":"2024-04-26T23:02:47.428596+0000","last_clean":"2024-04-26T23:02:47.428596+0000","last_became_active":"2024-04-26T23:02:47.421537+0000","last_became_peered":"2024-04-26T23:02:47.421537+0000","last_unstale":"2024-04-26T23:02:47.428596+0000","last_undegraded":"2024-04-26T23:02:47.428596+0000","last_fullsized":"2024-04-26T23:02:47.428596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:23.080779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.406234+0000","last_change":"2024-04-26T23:02:47.406234+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.406234+0000","last_undegraded":"2024-04-26T23:02:47.406234+0000","last_fullsized":"2024-04-26T23:02:47.406234+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.425058+0000","last_change":"2024-04-26T23:02:47.422919+0000","last_active":"2024-04-26T23:02:47.425058+0000","last_peered":"2024-04-26T23:02:47.425058+0000","last_clean":"2024-04-26T23:02:47.425058+0000","last_became_active":"2024-04-26T23:02:47.421694+0000","last_became_peered":"2024-04-26T23:02:47.421694+0000","last_unstale":"2024-04-26T23:02:47.425058+0000","last_undegraded":"2024-04-26T23:02:47.425058+0000","last_fullsized":"2024-04-26T23:02:47.425058+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:37:06.151991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.403814+0000","last_change":"2024-04-26T23:02:47.403814+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.403814+0000","last_undegraded":"2024-04-26T23:02:47.403814+0000","last_fullsized":"2024-04-26T23:02:47.403814+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.423560+0000","last_change":"2024-04-26T23:02:47.419127+0000","last_active":"2024-04-26T23:02:47.423560+0000","last_peered":"2024-04-26T23:02:47.423560+0000","last_clean":"2024-04-26T23:02:47.423560+0000","last_became_active":"2024-04-26T23:02:47.417694+0000","last_became_peered":"2024-04-26T23:02:47.417694+0000","last_unstale":"2024-04-26T23:02:47.423560+0000","last_undegraded":"2024-04-26T23:02:47.423560+0000","last_fullsized":"2024-04-26T23:02:47.423560+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.415211+0000","last_change":"2024-04-26T23:02:47.415211+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.415211+0000","last_undegraded":"2024-04-26T23:02:47.415211+0000","last_fullsized":"2024-04-26T23:02:47.415211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.419342+0000","last_change":"2024-04-26T23:02:47.416825+0000","last_active":"2024-04-26T23:02:47.419342+0000","last_peered":"2024-04-26T23:02:47.419342+0000","last_clean":"2024-04-26T23:02:47.419342+0000","last_became_active":"2024-04-26T23:02:47.416571+0000","last_became_peered":"2024-04-26T23:02:47.416571+0000","last_unstale":"2024-04-26T23:02:47.419342+0000","last_undegraded":"2024-04-26T23:02:47.419342+0000","last_fullsized":"2024-04-26T23:02:47.419342+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:25:15.376546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.406994+0000","last_change":"2024-04-26T23:02:47.406994+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.406994+0000","last_undegraded":"2024-04-26T23:02:47.406994+0000","last_fullsized":"2024-04-26T23:02:47.406994+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.433433+0000","last_change":"2024-04-26T23:02:47.432253+0000","last_active":"2024-04-26T23:02:47.433433+0000","last_peered":"2024-04-26T23:02:47.433433+0000","last_clean":"2024-04-26T23:02:47.433433+0000","last_became_active":"2024-04-26T23:02:47.431955+0000","last_became_peered":"2024-04-26T23:02:47.431955+0000","last_unstale":"2024-04-26T23:02:47.433433+0000","last_undegraded":"2024-04-26T23:02:47.433433+0000","last_fullsized":"2024-04-26T23:02:47.433433+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:27.125135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":203,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.413362+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:02:47.413362+0000","last_peered":"2024-04-26T23:02:47.413362+0000","last_clean":"2024-04-26T23:02:47.413362+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:02:47.413362+0000","last_undegraded":"2024-04-26T23:02:47.413362+0000","last_fullsized":"2024-04-26T23:02:47.413362+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":766999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1019904,"data_stored":773666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:02:53.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2024-04-26T23:02:53.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.409049+0000', 'last_change': '2024-04-26T23:02:47.409049+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.409049+0000', 'last_undegraded': '2024-04-26T23:02:47.409049+0000', 'last_fullsized': '2024-04-26T23:02:47.409049+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T23:02:53.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2024-04-26T23:02:53.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.409471+0000', 'last_change': '2024-04-26T23:02:47.409471+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.409471+0000', 'last_undegraded': '2024-04-26T23:02:47.409471+0000', 'last_fullsized': '2024-04-26T23:02:47.409471+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T23:02:53.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2024-04-26T23:02:53.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.410825+0000', 'last_change': '2024-04-26T23:02:47.410825+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.410825+0000', 'last_undegraded': '2024-04-26T23:02:47.410825+0000', 'last_fullsized': '2024-04-26T23:02:47.410825+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T23:02:53.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2024-04-26T23:02:53.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.402588+0000', 'last_change': '2024-04-26T23:02:47.402588+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.402588+0000', 'last_undegraded': '2024-04-26T23:02:47.402588+0000', 'last_fullsized': '2024-04-26T23:02:47.402588+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T23:02:53.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2024-04-26T23:02:53.964 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.400758+0000', 'last_change': '2024-04-26T23:02:47.400758+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.400758+0000', 'last_undegraded': '2024-04-26T23:02:47.400758+0000', 'last_fullsized': '2024-04-26T23:02:47.400758+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T23:02:53.964 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3a is not active+clean 2024-04-26T23:02:53.964 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.407437+0000', 'last_change': '2024-04-26T23:02:47.407437+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.407437+0000', 'last_undegraded': '2024-04-26T23:02:47.407437+0000', 'last_fullsized': '2024-04-26T23:02:47.407437+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-26T23:02:53.964 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2024-04-26T23:02:53.964 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.409272+0000', 'last_change': '2024-04-26T23:02:47.409272+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.409272+0000', 'last_undegraded': '2024-04-26T23:02:47.409272+0000', 'last_fullsized': '2024-04-26T23:02:47.409272+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-26T23:02:53.964 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2024-04-26T23:02:53.964 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.408667+0000', 'last_change': '2024-04-26T23:02:47.408667+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.408667+0000', 'last_undegraded': '2024-04-26T23:02:47.408667+0000', 'last_fullsized': '2024-04-26T23:02:47.408667+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T23:02:53.964 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2024-04-26T23:02:53.964 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.406089+0000', 'last_change': '2024-04-26T23:02:47.406089+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.406089+0000', 'last_undegraded': '2024-04-26T23:02:47.406089+0000', 'last_fullsized': '2024-04-26T23:02:47.406089+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T23:02:53.964 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2024-04-26T23:02:53.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.409393+0000', 'last_change': '2024-04-26T23:02:47.409393+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.409393+0000', 'last_undegraded': '2024-04-26T23:02:47.409393+0000', 'last_fullsized': '2024-04-26T23:02:47.409393+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-26T23:02:53.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2024-04-26T23:02:53.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.408476+0000', 'last_change': '2024-04-26T23:02:47.408476+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.408476+0000', 'last_undegraded': '2024-04-26T23:02:47.408476+0000', 'last_fullsized': '2024-04-26T23:02:47.408476+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-26T23:02:53.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2024-04-26T23:02:53.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.411813+0000', 'last_change': '2024-04-26T23:02:47.411813+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.411813+0000', 'last_undegraded': '2024-04-26T23:02:47.411813+0000', 'last_fullsized': '2024-04-26T23:02:47.411813+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T23:02:53.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2024-04-26T23:02:53.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.412901+0000', 'last_change': '2024-04-26T23:02:47.412901+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.412901+0000', 'last_undegraded': '2024-04-26T23:02:47.412901+0000', 'last_fullsized': '2024-04-26T23:02:47.412901+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-26T23:02:53.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2024-04-26T23:02:53.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.404645+0000', 'last_change': '2024-04-26T23:02:47.404645+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.404645+0000', 'last_undegraded': '2024-04-26T23:02:47.404645+0000', 'last_fullsized': '2024-04-26T23:02:47.404645+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-26T23:02:53.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2024-04-26T23:02:53.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.408580+0000', 'last_change': '2024-04-26T23:02:47.408580+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.408580+0000', 'last_undegraded': '2024-04-26T23:02:47.408580+0000', 'last_fullsized': '2024-04-26T23:02:47.408580+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-26T23:02:53.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2024-04-26T23:02:53.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.412732+0000', 'last_change': '2024-04-26T23:02:47.412732+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.412732+0000', 'last_undegraded': '2024-04-26T23:02:47.412732+0000', 'last_fullsized': '2024-04-26T23:02:47.412732+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-26T23:02:53.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2024-04-26T23:02:53.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.417759+0000', 'last_change': '2024-04-26T23:02:47.417759+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.417759+0000', 'last_undegraded': '2024-04-26T23:02:47.417759+0000', 'last_fullsized': '2024-04-26T23:02:47.417759+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-26T23:02:53.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2024-04-26T23:02:53.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.401677+0000', 'last_change': '2024-04-26T23:02:47.401677+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.401677+0000', 'last_undegraded': '2024-04-26T23:02:47.401677+0000', 'last_fullsized': '2024-04-26T23:02:47.401677+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-26T23:02:53.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2024-04-26T23:02:53.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.400720+0000', 'last_change': '2024-04-26T23:02:47.400720+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.400720+0000', 'last_undegraded': '2024-04-26T23:02:47.400720+0000', 'last_fullsized': '2024-04-26T23:02:47.400720+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T23:02:53.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2024-04-26T23:02:53.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.401112+0000', 'last_change': '2024-04-26T23:02:47.401112+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.401112+0000', 'last_undegraded': '2024-04-26T23:02:47.401112+0000', 'last_fullsized': '2024-04-26T23:02:47.401112+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-26T23:02:53.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2024-04-26T23:02:53.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.409722+0000', 'last_change': '2024-04-26T23:02:47.409722+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.409722+0000', 'last_undegraded': '2024-04-26T23:02:47.409722+0000', 'last_fullsized': '2024-04-26T23:02:47.409722+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-26T23:02:53.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2024-04-26T23:02:53.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.403188+0000', 'last_change': '2024-04-26T23:02:47.403188+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.403188+0000', 'last_undegraded': '2024-04-26T23:02:47.403188+0000', 'last_fullsized': '2024-04-26T23:02:47.403188+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-26T23:02:53.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2024-04-26T23:02:53.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.411186+0000', 'last_change': '2024-04-26T23:02:47.411186+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.411186+0000', 'last_undegraded': '2024-04-26T23:02:47.411186+0000', 'last_fullsized': '2024-04-26T23:02:47.411186+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-26T23:02:53.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2024-04-26T23:02:53.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.400306+0000', 'last_change': '2024-04-26T23:02:47.400306+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.400306+0000', 'last_undegraded': '2024-04-26T23:02:47.400306+0000', 'last_fullsized': '2024-04-26T23:02:47.400306+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T23:02:53.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2024-04-26T23:02:53.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.404031+0000', 'last_change': '2024-04-26T23:02:47.404031+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.404031+0000', 'last_undegraded': '2024-04-26T23:02:47.404031+0000', 'last_fullsized': '2024-04-26T23:02:47.404031+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-26T23:02:53.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.26 is not active+clean 2024-04-26T23:02:53.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.408158+0000', 'last_change': '2024-04-26T23:02:47.408158+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.408158+0000', 'last_undegraded': '2024-04-26T23:02:47.408158+0000', 'last_fullsized': '2024-04-26T23:02:47.408158+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-26T23:02:53.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2024-04-26T23:02:53.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.407444+0000', 'last_change': '2024-04-26T23:02:47.407444+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.407444+0000', 'last_undegraded': '2024-04-26T23:02:47.407444+0000', 'last_fullsized': '2024-04-26T23:02:47.407444+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-26T23:02:53.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2024-04-26T23:02:53.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.408555+0000', 'last_change': '2024-04-26T23:02:47.408555+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.408555+0000', 'last_undegraded': '2024-04-26T23:02:47.408555+0000', 'last_fullsized': '2024-04-26T23:02:47.408555+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-26T23:02:53.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2024-04-26T23:02:53.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.400673+0000', 'last_change': '2024-04-26T23:02:47.400673+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.400673+0000', 'last_undegraded': '2024-04-26T23:02:47.400673+0000', 'last_fullsized': '2024-04-26T23:02:47.400673+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-26T23:02:53.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2024-04-26T23:02:53.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.399159+0000', 'last_change': '2024-04-26T23:02:47.399159+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.399159+0000', 'last_undegraded': '2024-04-26T23:02:47.399159+0000', 'last_fullsized': '2024-04-26T23:02:47.399159+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-26T23:02:53.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2024-04-26T23:02:53.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.410706+0000', 'last_change': '2024-04-26T23:02:47.410706+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.410706+0000', 'last_undegraded': '2024-04-26T23:02:47.410706+0000', 'last_fullsized': '2024-04-26T23:02:47.410706+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-26T23:02:53.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.20 is not active+clean 2024-04-26T23:02:53.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.404746+0000', 'last_change': '2024-04-26T23:02:47.404746+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.404746+0000', 'last_undegraded': '2024-04-26T23:02:47.404746+0000', 'last_fullsized': '2024-04-26T23:02:47.404746+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-26T23:02:53.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2024-04-26T23:02:53.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.419764+0000', 'last_change': '2024-04-26T23:02:47.419764+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.419764+0000', 'last_undegraded': '2024-04-26T23:02:47.419764+0000', 'last_fullsized': '2024-04-26T23:02:47.419764+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-26T23:02:53.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2024-04-26T23:02:53.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.410136+0000', 'last_change': '2024-04-26T23:02:47.410136+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.410136+0000', 'last_undegraded': '2024-04-26T23:02:47.410136+0000', 'last_fullsized': '2024-04-26T23:02:47.410136+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-26T23:02:53.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2024-04-26T23:02:53.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.398614+0000', 'last_change': '2024-04-26T23:02:47.398614+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.398614+0000', 'last_undegraded': '2024-04-26T23:02:47.398614+0000', 'last_fullsized': '2024-04-26T23:02:47.398614+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-26T23:02:53.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2024-04-26T23:02:53.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.409517+0000', 'last_change': '2024-04-26T23:02:47.409517+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.409517+0000', 'last_undegraded': '2024-04-26T23:02:47.409517+0000', 'last_fullsized': '2024-04-26T23:02:47.409517+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T23:02:53.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2024-04-26T23:02:53.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.415281+0000', 'last_change': '2024-04-26T23:02:47.415281+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.415281+0000', 'last_undegraded': '2024-04-26T23:02:47.415281+0000', 'last_fullsized': '2024-04-26T23:02:47.415281+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T23:02:53.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2024-04-26T23:02:53.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.407967+0000', 'last_change': '2024-04-26T23:02:47.407967+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.407967+0000', 'last_undegraded': '2024-04-26T23:02:47.407967+0000', 'last_fullsized': '2024-04-26T23:02:47.407967+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-26T23:02:53.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2024-04-26T23:02:53.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.397241+0000', 'last_change': '2024-04-26T23:02:47.397241+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.397241+0000', 'last_undegraded': '2024-04-26T23:02:47.397241+0000', 'last_fullsized': '2024-04-26T23:02:47.397241+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-26T23:02:53.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2024-04-26T23:02:53.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.411122+0000', 'last_change': '2024-04-26T23:02:47.411122+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.411122+0000', 'last_undegraded': '2024-04-26T23:02:47.411122+0000', 'last_fullsized': '2024-04-26T23:02:47.411122+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T23:02:53.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2024-04-26T23:02:53.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.400911+0000', 'last_change': '2024-04-26T23:02:47.400911+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.400911+0000', 'last_undegraded': '2024-04-26T23:02:47.400911+0000', 'last_fullsized': '2024-04-26T23:02:47.400911+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T23:02:53.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2024-04-26T23:02:53.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.420803+0000', 'last_change': '2024-04-26T23:02:47.420803+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.420803+0000', 'last_undegraded': '2024-04-26T23:02:47.420803+0000', 'last_fullsized': '2024-04-26T23:02:47.420803+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-26T23:02:53.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2024-04-26T23:02:53.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.409689+0000', 'last_change': '2024-04-26T23:02:47.409689+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.409689+0000', 'last_undegraded': '2024-04-26T23:02:47.409689+0000', 'last_fullsized': '2024-04-26T23:02:47.409689+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T23:02:53.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2024-04-26T23:02:53.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.397867+0000', 'last_change': '2024-04-26T23:02:47.397867+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.397867+0000', 'last_undegraded': '2024-04-26T23:02:47.397867+0000', 'last_fullsized': '2024-04-26T23:02:47.397867+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-26T23:02:53.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2024-04-26T23:02:53.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.412607+0000', 'last_change': '2024-04-26T23:02:47.412607+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.412607+0000', 'last_undegraded': '2024-04-26T23:02:47.412607+0000', 'last_fullsized': '2024-04-26T23:02:47.412607+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T23:02:53.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2024-04-26T23:02:53.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.408705+0000', 'last_change': '2024-04-26T23:02:47.408705+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.408705+0000', 'last_undegraded': '2024-04-26T23:02:47.408705+0000', 'last_fullsized': '2024-04-26T23:02:47.408705+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T23:02:53.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2024-04-26T23:02:53.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.413966+0000', 'last_change': '2024-04-26T23:02:47.413966+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.413966+0000', 'last_undegraded': '2024-04-26T23:02:47.413966+0000', 'last_fullsized': '2024-04-26T23:02:47.413966+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-26T23:02:53.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2024-04-26T23:02:53.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.410588+0000', 'last_change': '2024-04-26T23:02:47.410588+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.410588+0000', 'last_undegraded': '2024-04-26T23:02:47.410588+0000', 'last_fullsized': '2024-04-26T23:02:47.410588+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T23:02:53.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2024-04-26T23:02:53.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.411859+0000', 'last_change': '2024-04-26T23:02:47.411859+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.411859+0000', 'last_undegraded': '2024-04-26T23:02:47.411859+0000', 'last_fullsized': '2024-04-26T23:02:47.411859+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T23:02:53.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2024-04-26T23:02:53.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.407338+0000', 'last_change': '2024-04-26T23:02:47.407338+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.407338+0000', 'last_undegraded': '2024-04-26T23:02:47.407338+0000', 'last_fullsized': '2024-04-26T23:02:47.407338+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-26T23:02:53.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2024-04-26T23:02:53.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.402648+0000', 'last_change': '2024-04-26T23:02:47.402648+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.402648+0000', 'last_undegraded': '2024-04-26T23:02:47.402648+0000', 'last_fullsized': '2024-04-26T23:02:47.402648+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-26T23:02:53.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2024-04-26T23:02:53.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.399491+0000', 'last_change': '2024-04-26T23:02:47.399491+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.399491+0000', 'last_undegraded': '2024-04-26T23:02:47.399491+0000', 'last_fullsized': '2024-04-26T23:02:47.399491+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-26T23:02:53.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2024-04-26T23:02:53.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.412312+0000', 'last_change': '2024-04-26T23:02:47.412312+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.412312+0000', 'last_undegraded': '2024-04-26T23:02:47.412312+0000', 'last_fullsized': '2024-04-26T23:02:47.412312+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-26T23:02:53.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2024-04-26T23:02:53.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.409298+0000', 'last_change': '2024-04-26T23:02:47.409298+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.409298+0000', 'last_undegraded': '2024-04-26T23:02:47.409298+0000', 'last_fullsized': '2024-04-26T23:02:47.409298+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T23:02:53.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2024-04-26T23:02:53.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.409401+0000', 'last_change': '2024-04-26T23:02:47.409401+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.409401+0000', 'last_undegraded': '2024-04-26T23:02:47.409401+0000', 'last_fullsized': '2024-04-26T23:02:47.409401+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-26T23:02:53.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2024-04-26T23:02:53.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.409951+0000', 'last_change': '2024-04-26T23:02:47.409951+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.409951+0000', 'last_undegraded': '2024-04-26T23:02:47.409951+0000', 'last_fullsized': '2024-04-26T23:02:47.409951+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T23:02:53.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2024-04-26T23:02:53.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.407819+0000', 'last_change': '2024-04-26T23:02:47.407819+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.407819+0000', 'last_undegraded': '2024-04-26T23:02:47.407819+0000', 'last_fullsized': '2024-04-26T23:02:47.407819+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-26T23:02:53.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2024-04-26T23:02:53.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.420271+0000', 'last_change': '2024-04-26T23:02:47.420271+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.420271+0000', 'last_undegraded': '2024-04-26T23:02:47.420271+0000', 'last_fullsized': '2024-04-26T23:02:47.420271+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-26T23:02:53.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2024-04-26T23:02:53.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.404728+0000', 'last_change': '2024-04-26T23:02:47.404728+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.404728+0000', 'last_undegraded': '2024-04-26T23:02:47.404728+0000', 'last_fullsized': '2024-04-26T23:02:47.404728+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-26T23:02:53.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2024-04-26T23:02:53.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.411594+0000', 'last_change': '2024-04-26T23:02:47.411594+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.411594+0000', 'last_undegraded': '2024-04-26T23:02:47.411594+0000', 'last_fullsized': '2024-04-26T23:02:47.411594+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-26T23:02:53.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2024-04-26T23:02:53.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.406234+0000', 'last_change': '2024-04-26T23:02:47.406234+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.406234+0000', 'last_undegraded': '2024-04-26T23:02:47.406234+0000', 'last_fullsized': '2024-04-26T23:02:47.406234+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-26T23:02:53.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2024-04-26T23:02:53.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.403814+0000', 'last_change': '2024-04-26T23:02:47.403814+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.403814+0000', 'last_undegraded': '2024-04-26T23:02:47.403814+0000', 'last_fullsized': '2024-04-26T23:02:47.403814+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-26T23:02:53.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2024-04-26T23:02:53.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.415211+0000', 'last_change': '2024-04-26T23:02:47.415211+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.415211+0000', 'last_undegraded': '2024-04-26T23:02:47.415211+0000', 'last_fullsized': '2024-04-26T23:02:47.415211+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-26T23:02:53.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2024-04-26T23:02:53.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-26T23:02:47.406994+0000', 'last_change': '2024-04-26T23:02:47.406994+0000', 'last_active': '2024-04-26T23:02:47.377264+0000', 'last_peered': '2024-04-26T23:02:47.377264+0000', 'last_clean': '2024-04-26T23:02:47.377264+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-26T23:02:47.406994+0000', 'last_undegraded': '2024-04-26T23:02:47.406994+0000', 'last_fullsized': '2024-04-26T23:02:47.406994+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'last_clean_scrub_stamp': '2024-04-26T23:02:47.377264+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-26T23:02:53.977 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:02:54.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.214+0000 7f2914a00640 1 Processor -- start 2024-04-26T23:02:54.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.214+0000 7f2914a00640 1 -- start start 2024-04-26T23:02:54.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.215+0000 7f2914a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910145970 0x7f29101612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.215+0000 7f2914a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29100b59f0 con 0x7f2910146060 2024-04-26T23:02:54.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.215+0000 7f2914a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29101617f0 con 0x7f29100d06b0 2024-04-26T23:02:54.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.215+0000 7f2914a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f29100ceed0 con 0x7f2910145970 2024-04-26T23:02:54.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.216+0000 7f290de00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2910146060 0x7f29100b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49942/0 (socket says 172.21.15.40:49942) 2024-04-26T23:02:54.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.216+0000 7f290de00640 1 -- 172.21.15.40:0/32060039 learned_addr learned my addr 172.21.15.40:0/32060039 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.216+0000 7f290e800640 1 --2- 172.21.15.40:0/32060039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910145970 0x7f29101612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.216+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3528565974 0 0) 0x7f29100b59f0 con 0x7f2910146060 2024-04-26T23:02:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.216+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28f4003520 con 0x7f2910146060 2024-04-26T23:02:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.216+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1064861224 0 0) 0x7f28f4003520 con 0x7f2910146060 2024-04-26T23:02:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.216+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910145970 msgr2=0x7f29101612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290d400640 1 --2- 172.21.15.40:0/32060039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910145970 0x7f29101612b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 >> v1:172.21.15.103:6789/0 conn(0x7f29100d06b0 legacy=0x7f29100cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29100b4180 con 0x7f2910146060 2024-04-26T23:02:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f2900001670 con 0x7f2910146060 2024-04-26T23:02:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 >> v1:172.21.15.40:6789/0 conn(0x7f2910146060 legacy=0x7f29100b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290d400640 1 --2- 172.21.15.40:0/32060039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910145970 0x7f28f40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290d400640 1 --2- 172.21.15.40:0/32060039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28f4003f60 0x7f28f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290d400640 1 --2- 172.21.15.40:0/32060039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28f4004d40 0x7f28f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f28f400ba70 con 0x7f28f4004d40 2024-04-26T23:02:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f28f400bc20 con 0x7f28f4003f60 2024-04-26T23:02:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f28f400be20 con 0x7f2910145970 2024-04-26T23:02:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290f200640 1 --2- 172.21.15.40:0/32060039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28f4004d40 0x7f28f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290e800640 1 --2- 172.21.15.40:0/32060039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910145970 0x7f28f40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.217+0000 7f290de00640 1 --2- 172.21.15.40:0/32060039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28f4003f60 0x7f28f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.218+0000 7f290f200640 1 -- 172.21.15.40:0/32060039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910145970 msgr2=0x7f28f40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.218+0000 7f290f200640 1 --2- 172.21.15.40:0/32060039 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910145970 0x7f28f40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.218+0000 7f290f200640 1 -- 172.21.15.40:0/32060039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28f4003f60 msgr2=0x7f28f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.218+0000 7f290f200640 1 --2- 172.21.15.40:0/32060039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28f4003f60 0x7f28f4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.218+0000 7f290f200640 1 -- 172.21.15.40:0/32060039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f4003520 con 0x7f28f4004d40 2024-04-26T23:02:54.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.219+0000 7f290f200640 1 --2- 172.21.15.40:0/32060039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28f4004d40 0x7f28f40051c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2908001e50 tx=0x7f28f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec02f9e3bee4db3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:54.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.219+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908001a70 con 0x7f28f4004d40 2024-04-26T23:02:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.219+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2908003970 con 0x7f28f4004d40 2024-04-26T23:02:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.219+0000 7f290d400640 1 -- 172.21.15.40:0/32060039 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908003d60 con 0x7f28f4004d40 2024-04-26T23:02:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.219+0000 7f2914a00640 1 -- 172.21.15.40:0/32060039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28f4004d40 msgr2=0x7f28f40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.219+0000 7f2914a00640 1 --2- 172.21.15.40:0/32060039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28f4004d40 0x7f28f40051c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2908001e50 tx=0x7f28f4005830 comp rx=0 tx=0).stop 2024-04-26T23:02:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.219+0000 7f290f200640 1 -- 172.21.15.40:0/32060039 reap_dead start 2024-04-26T23:02:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.220+0000 7f2914a00640 1 -- 172.21.15.40:0/32060039 shutdown_connections 2024-04-26T23:02:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.220+0000 7f2914a00640 1 -- 172.21.15.40:0/32060039 >> 172.21.15.40:0/32060039 conn(0x7f2910075eb0 msgr2=0x7f2910068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.220+0000 7f2914a00640 1 -- 172.21.15.40:0/32060039 shutdown_connections 2024-04-26T23:02:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.220+0000 7f2914a00640 1 -- 172.21.15.40:0/32060039 wait complete. 2024-04-26T23:02:54.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.221+0000 7f2914a00640 1 Processor -- start 2024-04-26T23:02:54.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.222+0000 7f2914a00640 1 -- start start 2024-04-26T23:02:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f2914a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28f4004d40 0x7f291015a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f2914a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910072d10 0x7f29100b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f2914a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29100b34c0 0x7f291015beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f2914a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f29100cf100 con 0x7f29100b34c0 2024-04-26T23:02:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f2914a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f29100b5c60 con 0x7f28f4004d40 2024-04-26T23:02:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f2914a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f29100cf3e0 con 0x7f2910072d10 2024-04-26T23:02:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f290f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29100b34c0 0x7f291015beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f290f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29100b34c0 0x7f291015beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42734/0 (socket says 172.21.15.40:42734) 2024-04-26T23:02:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f290f200640 1 -- 172.21.15.40:0/3407991453 learned_addr learned my addr 172.21.15.40:0/3407991453 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f290de00640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910072d10 0x7f29100b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f290e800640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28f4004d40 0x7f291015a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f290f200640 1 -- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910072d10 msgr2=0x7f29100b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f290f200640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910072d10 0x7f29100b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f290f200640 1 -- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28f4004d40 msgr2=0x7f291015a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f290f200640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28f4004d40 0x7f291015a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f290f200640 1 -- 172.21.15.40:0/3407991453 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291015a610 con 0x7f29100b34c0 2024-04-26T23:02:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.223+0000 7f290de00640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910072d10 0x7f29100b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.224+0000 7f290f200640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29100b34c0 0x7f291015beb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f290802a920 tx=0x7f29080014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.224+0000 7f2906a00640 1 -- 172.21.15.40:0/3407991453 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908037040 con 0x7f29100b34c0 2024-04-26T23:02:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.224+0000 7f2906a00640 1 -- 172.21.15.40:0/3407991453 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2908003e50 con 0x7f29100b34c0 2024-04-26T23:02:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.224+0000 7f2906a00640 1 -- 172.21.15.40:0/3407991453 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908001a70 con 0x7f29100b34c0 2024-04-26T23:02:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.224+0000 7f2914a00640 1 -- 172.21.15.40:0/3407991453 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2910071f50 con 0x7f29100b34c0 2024-04-26T23:02:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.224+0000 7f2914a00640 1 -- 172.21.15.40:0/3407991453 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2910072510 con 0x7f29100b34c0 2024-04-26T23:02:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.225+0000 7f2914a00640 1 -- 172.21.15.40:0/3407991453 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2910069370 con 0x7f29100b34c0 2024-04-26T23:02:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.227+0000 7f2906a00640 1 -- 172.21.15.40:0/3407991453 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2908042020 con 0x7f29100b34c0 2024-04-26T23:02:54.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.228+0000 7f2906a00640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f28ec06c8f0 0x7f28ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.228+0000 7f2906a00640 1 -- 172.21.15.40:0/3407991453 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f29080ab220 con 0x7f29100b34c0 2024-04-26T23:02:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.228+0000 7f290e800640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f28ec06c8f0 0x7f28ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.229+0000 7f290e800640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f28ec06c8f0 0x7f28ec06edb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f28f8009950 tx=0x7f28f8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:54.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.232+0000 7f2906a00640 1 -- 172.21.15.40:0/3407991453 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f29080795a0 con 0x7f29100b34c0 2024-04-26T23:02:54.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.515+0000 7f2914a00640 1 -- 172.21.15.40:0/3407991453 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f29100d1400 con 0x7f29100b34c0 2024-04-26T23:02:54.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.516+0000 7f2906a00640 1 -- 172.21.15.40:0/3407991453 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1034 (secure 0 0 0) 0x7f2908078ea0 con 0x7f29100b34c0 2024-04-26T23:02:54.516 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:54.516 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":134,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":225914880,"bytes_avail":772868198400,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":18,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5334}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-04-26T23:02:11.167223+0000","services":{}},"progress_events":{}} 2024-04-26T23:02:54.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.519+0000 7f2914a00640 1 -- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f28ec06c8f0 msgr2=0x7f28ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:54.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.519+0000 7f2914a00640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f28ec06c8f0 0x7f28ec06edb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f28f8009950 tx=0x7f28f8008040 comp rx=0 tx=0).stop 2024-04-26T23:02:54.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.519+0000 7f2914a00640 1 -- 172.21.15.40:0/3407991453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29100b34c0 msgr2=0x7f291015beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:54.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.519+0000 7f2914a00640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29100b34c0 0x7f291015beb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f290802a920 tx=0x7f29080014a0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.519+0000 7f2914a00640 1 -- 172.21.15.40:0/3407991453 shutdown_connections 2024-04-26T23:02:54.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.519+0000 7f2914a00640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f28ec06c8f0 0x7f28ec06edb0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.519+0000 7f2914a00640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29100b34c0 0x7f291015beb0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.519+0000 7f2914a00640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2910072d10 0x7f29100b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.519+0000 7f2914a00640 1 --2- 172.21.15.40:0/3407991453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28f4004d40 0x7f291015a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.519+0000 7f2914a00640 1 -- 172.21.15.40:0/3407991453 >> 172.21.15.40:0/3407991453 conn(0x7f2910075eb0 msgr2=0x7f2910144ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:54.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.519+0000 7f2914a00640 1 -- 172.21.15.40:0/3407991453 shutdown_connections 2024-04-26T23:02:54.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.519+0000 7f2914a00640 1 -- 172.21.15.40:0/3407991453 wait complete. 2024-04-26T23:02:54.539 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-26T23:02:54.539 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:02:54.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.752+0000 7f1e98800640 1 Processor -- start 2024-04-26T23:02:54.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.752+0000 7f1e98800640 1 -- start start 2024-04-26T23:02:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.753+0000 7f1e98800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1e90071e60 0x7f1e9006f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.753+0000 7f1e98800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e9006fe20 0x7f1e90072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.753+0000 7f1e98800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e9015d7c0 con 0x7f1e90072780 2024-04-26T23:02:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.753+0000 7f1e98800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1e9015daa0 con 0x7f1e90071e60 2024-04-26T23:02:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.753+0000 7f1e98800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9015dd80 con 0x7f1e9006fe20 2024-04-26T23:02:54.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.754+0000 7f1e95400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e9006fe20 0x7f1e90072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.754+0000 7f1e95e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1e90071e60 0x7f1e9006f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.754+0000 7f1e95400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e9006fe20 0x7f1e90072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:36472/0 (socket says 172.21.15.40:36472) 2024-04-26T23:02:54.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.754+0000 7f1e95400640 1 -- 172.21.15.40:0/3944663626 learned_addr learned my addr 172.21.15.40:0/3944663626 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:54.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.755+0000 7f1e95400640 1 -- 172.21.15.40:0/3944663626 >> v2:172.21.15.103:3300/0 conn(0x7f1e90071e60 msgr2=0x7f1e9006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:54.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.755+0000 7f1e95400640 1 --2- 172.21.15.40:0/3944663626 >> v2:172.21.15.103:3300/0 conn(0x7f1e90071e60 0x7f1e9006f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.755+0000 7f1e95400640 1 -- 172.21.15.40:0/3944663626 >> v1:172.21.15.103:6789/0 conn(0x7f1e90072780 legacy=0x7f1e900cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:54.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.755+0000 7f1e95400640 1 -- 172.21.15.40:0/3944663626 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9015e060 con 0x7f1e9006fe20 2024-04-26T23:02:54.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.755+0000 7f1e95e00640 1 --2- 172.21.15.40:0/3944663626 >> v2:172.21.15.103:3300/0 conn(0x7f1e90071e60 0x7f1e9006f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:02:54.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.757+0000 7f1e95400640 1 --2- 172.21.15.40:0/3944663626 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e9006fe20 0x7f1e90072240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1e80009870 tx=0x7f1e8002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4e3e015858d212a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:54.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.757+0000 7f1e94a00640 1 -- 172.21.15.40:0/3944663626 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e80037070 con 0x7f1e9006fe20 2024-04-26T23:02:54.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.757+0000 7f1e94a00640 1 -- 172.21.15.40:0/3944663626 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e80032440 con 0x7f1e9006fe20 2024-04-26T23:02:54.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.757+0000 7f1e94a00640 1 -- 172.21.15.40:0/3944663626 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e80032810 con 0x7f1e9006fe20 2024-04-26T23:02:54.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.758+0000 7f1e98800640 1 -- 172.21.15.40:0/3944663626 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e9006fe20 msgr2=0x7f1e90072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:54.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.758+0000 7f1e98800640 1 --2- 172.21.15.40:0/3944663626 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e9006fe20 0x7f1e90072240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1e80009870 tx=0x7f1e8002a250 comp rx=0 tx=0).stop 2024-04-26T23:02:54.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.758+0000 7f1e98800640 1 -- 172.21.15.40:0/3944663626 shutdown_connections 2024-04-26T23:02:54.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.758+0000 7f1e98800640 1 --2- 172.21.15.40:0/3944663626 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e9006fe20 0x7f1e90072240 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.758+0000 7f1e98800640 1 --2- 172.21.15.40:0/3944663626 >> v2:172.21.15.103:3300/0 conn(0x7f1e90071e60 0x7f1e9006f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.758+0000 7f1e98800640 1 -- 172.21.15.40:0/3944663626 >> 172.21.15.40:0/3944663626 conn(0x7f1e900d0be0 msgr2=0x7f1e900d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:54.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.758+0000 7f1e98800640 1 -- 172.21.15.40:0/3944663626 shutdown_connections 2024-04-26T23:02:54.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.758+0000 7f1e98800640 1 -- 172.21.15.40:0/3944663626 wait complete. 2024-04-26T23:02:54.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.760+0000 7f1e98800640 1 Processor -- start 2024-04-26T23:02:54.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.760+0000 7f1e98800640 1 -- start start 2024-04-26T23:02:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.761+0000 7f1e98800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e9006fe20 0x7f1e9015c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.761+0000 7f1e98800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e90071e60 0x7f1e9015b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.761+0000 7f1e98800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e90072780 0x7f1e90153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.761+0000 7f1e98800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9015d3f0 con 0x7f1e90072780 2024-04-26T23:02:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.761+0000 7f1e98800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e90072c10 con 0x7f1e9006fe20 2024-04-26T23:02:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.761+0000 7f1e98800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9015d570 con 0x7f1e90071e60 2024-04-26T23:02:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.761+0000 7f1e96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e90072780 0x7f1e90153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.762+0000 7f1e96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e90072780 0x7f1e90153eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42742/0 (socket says 172.21.15.40:42742) 2024-04-26T23:02:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.762+0000 7f1e96800640 1 -- 172.21.15.40:0/2163577293 learned_addr learned my addr 172.21.15.40:0/2163577293 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.762+0000 7f1e95e00640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e9006fe20 0x7f1e9015c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.762+0000 7f1e95400640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e90071e60 0x7f1e9015b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.762+0000 7f1e96800640 1 -- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e90071e60 msgr2=0x7f1e9015b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.762+0000 7f1e96800640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e90071e60 0x7f1e9015b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.762+0000 7f1e96800640 1 -- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e9006fe20 msgr2=0x7f1e9015c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.762+0000 7f1e96800640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e9006fe20 0x7f1e9015c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.762+0000 7f1e96800640 1 -- 172.21.15.40:0/2163577293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e901f8230 con 0x7f1e90072780 2024-04-26T23:02:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.763+0000 7f1e96800640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e90072780 0x7f1e90153eb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00c6b0 tx=0x7f1e8c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.763+0000 7f1e8a000640 1 -- 172.21.15.40:0/2163577293 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c003b70 con 0x7f1e90072780 2024-04-26T23:02:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.763+0000 7f1e8a000640 1 -- 172.21.15.40:0/2163577293 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e8c003d30 con 0x7f1e90072780 2024-04-26T23:02:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.763+0000 7f1e8a000640 1 -- 172.21.15.40:0/2163577293 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c005310 con 0x7f1e90072780 2024-04-26T23:02:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.763+0000 7f1e98800640 1 -- 172.21.15.40:0/2163577293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e901f8510 con 0x7f1e90072780 2024-04-26T23:02:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.763+0000 7f1e98800640 1 -- 172.21.15.40:0/2163577293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e901f8ad0 con 0x7f1e90072780 2024-04-26T23:02:54.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.765+0000 7f1e98800640 1 -- 172.21.15.40:0/2163577293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e900694f0 con 0x7f1e90072780 2024-04-26T23:02:54.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.765+0000 7f1e8a000640 1 -- 172.21.15.40:0/2163577293 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1e8c0054b0 con 0x7f1e90072780 2024-04-26T23:02:54.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.766+0000 7f1e8a000640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7406c8f0 0x7f1e7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:54.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.766+0000 7f1e8a000640 1 -- 172.21.15.40:0/2163577293 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f1e8c08ba90 con 0x7f1e90072780 2024-04-26T23:02:54.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.767+0000 7f1e95e00640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7406c8f0 0x7f1e7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:54.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.768+0000 7f1e95e00640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7406c8f0 0x7f1e7406edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009560 tx=0x7f1e84009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:54.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.771+0000 7f1e8a000640 1 -- 172.21.15.40:0/2163577293 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1e8c059d30 con 0x7f1e90072780 2024-04-26T23:02:54.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.977+0000 7f1e98800640 1 -- 172.21.15.40:0/2163577293 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1e9006ce60 con 0x7f1e7406c8f0 2024-04-26T23:02:54.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:54.996+0000 7f1e8a000640 1 -- 172.21.15.40:0/2163577293 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246635 (secure 0 0 0) 0x7f1e9006ce60 con 0x7f1e7406c8f0 2024-04-26T23:02:55.001 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:55.002 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:02:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:55.001+0000 7f1e98800640 1 -- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7406c8f0 msgr2=0x7f1e7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:55.001+0000 7f1e98800640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7406c8f0 0x7f1e7406edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009560 tx=0x7f1e84009190 comp rx=0 tx=0).stop 2024-04-26T23:02:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:55.001+0000 7f1e98800640 1 -- 172.21.15.40:0/2163577293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e90072780 msgr2=0x7f1e90153eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:55.001+0000 7f1e98800640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e90072780 0x7f1e90153eb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00c6b0 tx=0x7f1e8c00cb70 comp rx=0 tx=0).stop 2024-04-26T23:02:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:55.001+0000 7f1e98800640 1 -- 172.21.15.40:0/2163577293 shutdown_connections 2024-04-26T23:02:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:55.001+0000 7f1e98800640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7406c8f0 0x7f1e7406edb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:55.002+0000 7f1e98800640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e90072780 0x7f1e90153eb0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:55.002+0000 7f1e98800640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e90071e60 0x7f1e9015b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:55.002+0000 7f1e98800640 1 --2- 172.21.15.40:0/2163577293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e9006fe20 0x7f1e9015c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:55.002+0000 7f1e98800640 1 -- 172.21.15.40:0/2163577293 >> 172.21.15.40:0/2163577293 conn(0x7f1e900d0be0 msgr2=0x7f1e900cb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:55.002+0000 7f1e98800640 1 -- 172.21.15.40:0/2163577293 shutdown_connections 2024-04-26T23:02:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:55.002+0000 7f1e98800640 1 -- 172.21.15.40:0/2163577293 wait complete. 2024-04-26T23:02:55.022 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-04-26T23:02:53.182976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220620,"kb_used_data":4452,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754100,"statfs":{"total":773094113280,"available":772868198400,"internally_reserved":0,"allocated":4558848,"data_stored":2627119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001782"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409049+0000","last_change":"2024-04-26T23:02:47.409049+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409049+0000","last_undegraded":"2024-04-26T23:02:47.409049+0000","last_fullsized":"2024-04-26T23:02:47.409049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409471+0000","last_change":"2024-04-26T23:02:47.409471+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409471+0000","last_undegraded":"2024-04-26T23:02:47.409471+0000","last_fullsized":"2024-04-26T23:02:47.409471+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410825+0000","last_change":"2024-04-26T23:02:47.410825+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410825+0000","last_undegraded":"2024-04-26T23:02:47.410825+0000","last_fullsized":"2024-04-26T23:02:47.410825+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.402588+0000","last_change":"2024-04-26T23:02:47.402588+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.402588+0000","last_undegraded":"2024-04-26T23:02:47.402588+0000","last_fullsized":"2024-04-26T23:02:47.402588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400758+0000","last_change":"2024-04-26T23:02:47.400758+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400758+0000","last_undegraded":"2024-04-26T23:02:47.400758+0000","last_fullsized":"2024-04-26T23:02:47.400758+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407437+0000","last_change":"2024-04-26T23:02:47.407437+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407437+0000","last_undegraded":"2024-04-26T23:02:47.407437+0000","last_fullsized":"2024-04-26T23:02:47.407437+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409272+0000","last_change":"2024-04-26T23:02:47.409272+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409272+0000","last_undegraded":"2024-04-26T23:02:47.409272+0000","last_fullsized":"2024-04-26T23:02:47.409272+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408667+0000","last_change":"2024-04-26T23:02:47.408667+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408667+0000","last_undegraded":"2024-04-26T23:02:47.408667+0000","last_fullsized":"2024-04-26T23:02:47.408667+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.406089+0000","last_change":"2024-04-26T23:02:47.406089+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.406089+0000","last_undegraded":"2024-04-26T23:02:47.406089+0000","last_fullsized":"2024-04-26T23:02:47.406089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409393+0000","last_change":"2024-04-26T23:02:47.409393+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409393+0000","last_undegraded":"2024-04-26T23:02:47.409393+0000","last_fullsized":"2024-04-26T23:02:47.409393+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408476+0000","last_change":"2024-04-26T23:02:47.408476+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408476+0000","last_undegraded":"2024-04-26T23:02:47.408476+0000","last_fullsized":"2024-04-26T23:02:47.408476+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411813+0000","last_change":"2024-04-26T23:02:47.411813+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411813+0000","last_undegraded":"2024-04-26T23:02:47.411813+0000","last_fullsized":"2024-04-26T23:02:47.411813+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412901+0000","last_change":"2024-04-26T23:02:47.412901+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412901+0000","last_undegraded":"2024-04-26T23:02:47.412901+0000","last_fullsized":"2024-04-26T23:02:47.412901+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404645+0000","last_change":"2024-04-26T23:02:47.404645+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404645+0000","last_undegraded":"2024-04-26T23:02:47.404645+0000","last_fullsized":"2024-04-26T23:02:47.404645+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408580+0000","last_change":"2024-04-26T23:02:47.408580+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408580+0000","last_undegraded":"2024-04-26T23:02:47.408580+0000","last_fullsized":"2024-04-26T23:02:47.408580+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412732+0000","last_change":"2024-04-26T23:02:47.412732+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412732+0000","last_undegraded":"2024-04-26T23:02:47.412732+0000","last_fullsized":"2024-04-26T23:02:47.412732+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.417759+0000","last_change":"2024-04-26T23:02:47.417759+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.417759+0000","last_undegraded":"2024-04-26T23:02:47.417759+0000","last_fullsized":"2024-04-26T23:02:47.417759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.401677+0000","last_change":"2024-04-26T23:02:47.401677+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.401677+0000","last_undegraded":"2024-04-26T23:02:47.401677+0000","last_fullsized":"2024-04-26T23:02:47.401677+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400720+0000","last_change":"2024-04-26T23:02:47.400720+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400720+0000","last_undegraded":"2024-04-26T23:02:47.400720+0000","last_fullsized":"2024-04-26T23:02:47.400720+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.401112+0000","last_change":"2024-04-26T23:02:47.401112+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.401112+0000","last_undegraded":"2024-04-26T23:02:47.401112+0000","last_fullsized":"2024-04-26T23:02:47.401112+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409722+0000","last_change":"2024-04-26T23:02:47.409722+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409722+0000","last_undegraded":"2024-04-26T23:02:47.409722+0000","last_fullsized":"2024-04-26T23:02:47.409722+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.403188+0000","last_change":"2024-04-26T23:02:47.403188+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.403188+0000","last_undegraded":"2024-04-26T23:02:47.403188+0000","last_fullsized":"2024-04-26T23:02:47.403188+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411186+0000","last_change":"2024-04-26T23:02:47.411186+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411186+0000","last_undegraded":"2024-04-26T23:02:47.411186+0000","last_fullsized":"2024-04-26T23:02:47.411186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400306+0000","last_change":"2024-04-26T23:02:47.400306+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400306+0000","last_undegraded":"2024-04-26T23:02:47.400306+0000","last_fullsized":"2024-04-26T23:02:47.400306+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404031+0000","last_change":"2024-04-26T23:02:47.404031+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404031+0000","last_undegraded":"2024-04-26T23:02:47.404031+0000","last_fullsized":"2024-04-26T23:02:47.404031+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408158+0000","last_change":"2024-04-26T23:02:47.408158+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408158+0000","last_undegraded":"2024-04-26T23:02:47.408158+0000","last_fullsized":"2024-04-26T23:02:47.408158+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407444+0000","last_change":"2024-04-26T23:02:47.407444+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407444+0000","last_undegraded":"2024-04-26T23:02:47.407444+0000","last_fullsized":"2024-04-26T23:02:47.407444+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408555+0000","last_change":"2024-04-26T23:02:47.408555+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408555+0000","last_undegraded":"2024-04-26T23:02:47.408555+0000","last_fullsized":"2024-04-26T23:02:47.408555+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400673+0000","last_change":"2024-04-26T23:02:47.400673+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400673+0000","last_undegraded":"2024-04-26T23:02:47.400673+0000","last_fullsized":"2024-04-26T23:02:47.400673+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.399159+0000","last_change":"2024-04-26T23:02:47.399159+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.399159+0000","last_undegraded":"2024-04-26T23:02:47.399159+0000","last_fullsized":"2024-04-26T23:02:47.399159+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410706+0000","last_change":"2024-04-26T23:02:47.410706+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410706+0000","last_undegraded":"2024-04-26T23:02:47.410706+0000","last_fullsized":"2024-04-26T23:02:47.410706+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404746+0000","last_change":"2024-04-26T23:02:47.404746+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404746+0000","last_undegraded":"2024-04-26T23:02:47.404746+0000","last_fullsized":"2024-04-26T23:02:47.404746+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.419764+0000","last_change":"2024-04-26T23:02:47.419764+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.419764+0000","last_undegraded":"2024-04-26T23:02:47.419764+0000","last_fullsized":"2024-04-26T23:02:47.419764+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.422997+0000","last_change":"2024-04-26T23:02:47.416086+0000","last_active":"2024-04-26T23:02:47.422997+0000","last_peered":"2024-04-26T23:02:47.422997+0000","last_clean":"2024-04-26T23:02:47.422997+0000","last_became_active":"2024-04-26T23:02:47.415794+0000","last_became_peered":"2024-04-26T23:02:47.415794+0000","last_unstale":"2024-04-26T23:02:47.422997+0000","last_undegraded":"2024-04-26T23:02:47.422997+0000","last_fullsized":"2024-04-26T23:02:47.422997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:54:53.447069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410136+0000","last_change":"2024-04-26T23:02:47.410136+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410136+0000","last_undegraded":"2024-04-26T23:02:47.410136+0000","last_fullsized":"2024-04-26T23:02:47.410136+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428834+0000","last_change":"2024-04-26T23:02:47.422501+0000","last_active":"2024-04-26T23:02:47.428834+0000","last_peered":"2024-04-26T23:02:47.428834+0000","last_clean":"2024-04-26T23:02:47.428834+0000","last_became_active":"2024-04-26T23:02:47.422240+0000","last_became_peered":"2024-04-26T23:02:47.422240+0000","last_unstale":"2024-04-26T23:02:47.428834+0000","last_undegraded":"2024-04-26T23:02:47.428834+0000","last_fullsized":"2024-04-26T23:02:47.428834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.398614+0000","last_change":"2024-04-26T23:02:47.398614+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.398614+0000","last_undegraded":"2024-04-26T23:02:47.398614+0000","last_fullsized":"2024-04-26T23:02:47.398614+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.432856+0000","last_change":"2024-04-26T23:02:47.427135+0000","last_active":"2024-04-26T23:02:47.432856+0000","last_peered":"2024-04-26T23:02:47.432856+0000","last_clean":"2024-04-26T23:02:47.432856+0000","last_became_active":"2024-04-26T23:02:47.425928+0000","last_became_peered":"2024-04-26T23:02:47.425928+0000","last_unstale":"2024-04-26T23:02:47.432856+0000","last_undegraded":"2024-04-26T23:02:47.432856+0000","last_fullsized":"2024-04-26T23:02:47.432856+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409517+0000","last_change":"2024-04-26T23:02:47.409517+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409517+0000","last_undegraded":"2024-04-26T23:02:47.409517+0000","last_fullsized":"2024-04-26T23:02:47.409517+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.427084+0000","last_change":"2024-04-26T23:02:47.423448+0000","last_active":"2024-04-26T23:02:47.427084+0000","last_peered":"2024-04-26T23:02:47.427084+0000","last_clean":"2024-04-26T23:02:47.427084+0000","last_became_active":"2024-04-26T23:02:47.423145+0000","last_became_peered":"2024-04-26T23:02:47.423145+0000","last_unstale":"2024-04-26T23:02:47.427084+0000","last_undegraded":"2024-04-26T23:02:47.427084+0000","last_fullsized":"2024-04-26T23:02:47.427084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.415281+0000","last_change":"2024-04-26T23:02:47.415281+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.415281+0000","last_undegraded":"2024-04-26T23:02:47.415281+0000","last_fullsized":"2024-04-26T23:02:47.415281+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431170+0000","last_change":"2024-04-26T23:02:47.429506+0000","last_active":"2024-04-26T23:02:47.431170+0000","last_peered":"2024-04-26T23:02:47.431170+0000","last_clean":"2024-04-26T23:02:47.431170+0000","last_became_active":"2024-04-26T23:02:47.429131+0000","last_became_peered":"2024-04-26T23:02:47.429131+0000","last_unstale":"2024-04-26T23:02:47.431170+0000","last_undegraded":"2024-04-26T23:02:47.431170+0000","last_fullsized":"2024-04-26T23:02:47.431170+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:28.822626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407967+0000","last_change":"2024-04-26T23:02:47.407967+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407967+0000","last_undegraded":"2024-04-26T23:02:47.407967+0000","last_fullsized":"2024-04-26T23:02:47.407967+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430956+0000","last_change":"2024-04-26T23:02:47.428296+0000","last_active":"2024-04-26T23:02:47.430956+0000","last_peered":"2024-04-26T23:02:47.430956+0000","last_clean":"2024-04-26T23:02:47.430956+0000","last_became_active":"2024-04-26T23:02:47.427980+0000","last_became_peered":"2024-04-26T23:02:47.427980+0000","last_unstale":"2024-04-26T23:02:47.430956+0000","last_undegraded":"2024-04-26T23:02:47.430956+0000","last_fullsized":"2024-04-26T23:02:47.430956+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:24:26.243323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.397241+0000","last_change":"2024-04-26T23:02:47.397241+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.397241+0000","last_undegraded":"2024-04-26T23:02:47.397241+0000","last_fullsized":"2024-04-26T23:02:47.397241+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.427966+0000","last_change":"2024-04-26T23:02:47.422927+0000","last_active":"2024-04-26T23:02:47.427966+0000","last_peered":"2024-04-26T23:02:47.427966+0000","last_clean":"2024-04-26T23:02:47.427966+0000","last_became_active":"2024-04-26T23:02:47.422616+0000","last_became_peered":"2024-04-26T23:02:47.422616+0000","last_unstale":"2024-04-26T23:02:47.427966+0000","last_undegraded":"2024-04-26T23:02:47.427966+0000","last_fullsized":"2024-04-26T23:02:47.427966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:23.080779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411122+0000","last_change":"2024-04-26T23:02:47.411122+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411122+0000","last_undegraded":"2024-04-26T23:02:47.411122+0000","last_fullsized":"2024-04-26T23:02:47.411122+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.429074+0000","last_change":"2024-04-26T23:02:47.426665+0000","last_active":"2024-04-26T23:02:47.429074+0000","last_peered":"2024-04-26T23:02:47.429074+0000","last_clean":"2024-04-26T23:02:47.429074+0000","last_became_active":"2024-04-26T23:02:47.426408+0000","last_became_peered":"2024-04-26T23:02:47.426408+0000","last_unstale":"2024-04-26T23:02:47.429074+0000","last_undegraded":"2024-04-26T23:02:47.429074+0000","last_fullsized":"2024-04-26T23:02:47.429074+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:28.822626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.400911+0000","last_change":"2024-04-26T23:02:47.400911+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.400911+0000","last_undegraded":"2024-04-26T23:02:47.400911+0000","last_fullsized":"2024-04-26T23:02:47.400911+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.433081+0000","last_change":"2024-04-26T23:02:47.429805+0000","last_active":"2024-04-26T23:02:47.433081+0000","last_peered":"2024-04-26T23:02:47.433081+0000","last_clean":"2024-04-26T23:02:47.433081+0000","last_became_active":"2024-04-26T23:02:47.429541+0000","last_became_peered":"2024-04-26T23:02:47.429541+0000","last_unstale":"2024-04-26T23:02:47.433081+0000","last_undegraded":"2024-04-26T23:02:47.433081+0000","last_fullsized":"2024-04-26T23:02:47.433081+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:01.704078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.420803+0000","last_change":"2024-04-26T23:02:47.420803+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.420803+0000","last_undegraded":"2024-04-26T23:02:47.420803+0000","last_fullsized":"2024-04-26T23:02:47.420803+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.425979+0000","last_change":"2024-04-26T23:02:47.424076+0000","last_active":"2024-04-26T23:02:47.425979+0000","last_peered":"2024-04-26T23:02:47.425979+0000","last_clean":"2024-04-26T23:02:47.425979+0000","last_became_active":"2024-04-26T23:02:47.420944+0000","last_became_peered":"2024-04-26T23:02:47.420944+0000","last_unstale":"2024-04-26T23:02:47.425979+0000","last_undegraded":"2024-04-26T23:02:47.425979+0000","last_fullsized":"2024-04-26T23:02:47.425979+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409689+0000","last_change":"2024-04-26T23:02:47.409689+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409689+0000","last_undegraded":"2024-04-26T23:02:47.409689+0000","last_fullsized":"2024-04-26T23:02:47.409689+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430549+0000","last_change":"2024-04-26T23:02:47.426825+0000","last_active":"2024-04-26T23:02:47.430549+0000","last_peered":"2024-04-26T23:02:47.430549+0000","last_clean":"2024-04-26T23:02:47.430549+0000","last_became_active":"2024-04-26T23:02:47.426475+0000","last_became_peered":"2024-04-26T23:02:47.426475+0000","last_unstale":"2024-04-26T23:02:47.430549+0000","last_undegraded":"2024-04-26T23:02:47.430549+0000","last_fullsized":"2024-04-26T23:02:47.430549+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.397867+0000","last_change":"2024-04-26T23:02:47.397867+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.397867+0000","last_undegraded":"2024-04-26T23:02:47.397867+0000","last_fullsized":"2024-04-26T23:02:47.397867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428260+0000","last_change":"2024-04-26T23:02:47.425906+0000","last_active":"2024-04-26T23:02:47.428260+0000","last_peered":"2024-04-26T23:02:47.428260+0000","last_clean":"2024-04-26T23:02:47.428260+0000","last_became_active":"2024-04-26T23:02:47.425612+0000","last_became_peered":"2024-04-26T23:02:47.425612+0000","last_unstale":"2024-04-26T23:02:47.428260+0000","last_undegraded":"2024-04-26T23:02:47.428260+0000","last_fullsized":"2024-04-26T23:02:47.428260+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:56:06.218292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412607+0000","last_change":"2024-04-26T23:02:47.412607+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412607+0000","last_undegraded":"2024-04-26T23:02:47.412607+0000","last_fullsized":"2024-04-26T23:02:47.412607+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.433025+0000","last_change":"2024-04-26T23:02:47.429056+0000","last_active":"2024-04-26T23:02:47.433025+0000","last_peered":"2024-04-26T23:02:47.433025+0000","last_clean":"2024-04-26T23:02:47.433025+0000","last_became_active":"2024-04-26T23:02:47.428144+0000","last_became_peered":"2024-04-26T23:02:47.428144+0000","last_unstale":"2024-04-26T23:02:47.433025+0000","last_undegraded":"2024-04-26T23:02:47.433025+0000","last_fullsized":"2024-04-26T23:02:47.433025+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:25.938366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.408705+0000","last_change":"2024-04-26T23:02:47.408705+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.408705+0000","last_undegraded":"2024-04-26T23:02:47.408705+0000","last_fullsized":"2024-04-26T23:02:47.408705+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431539+0000","last_change":"2024-04-26T23:02:47.426625+0000","last_active":"2024-04-26T23:02:47.431539+0000","last_peered":"2024-04-26T23:02:47.431539+0000","last_clean":"2024-04-26T23:02:47.431539+0000","last_became_active":"2024-04-26T23:02:47.426351+0000","last_became_peered":"2024-04-26T23:02:47.426351+0000","last_unstale":"2024-04-26T23:02:47.431539+0000","last_undegraded":"2024-04-26T23:02:47.431539+0000","last_fullsized":"2024-04-26T23:02:47.431539+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.413966+0000","last_change":"2024-04-26T23:02:47.413966+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.413966+0000","last_undegraded":"2024-04-26T23:02:47.413966+0000","last_fullsized":"2024-04-26T23:02:47.413966+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.423709+0000","last_change":"2024-04-26T23:02:47.418555+0000","last_active":"2024-04-26T23:02:47.423709+0000","last_peered":"2024-04-26T23:02:47.423709+0000","last_clean":"2024-04-26T23:02:47.423709+0000","last_became_active":"2024-04-26T23:02:47.418241+0000","last_became_peered":"2024-04-26T23:02:47.418241+0000","last_unstale":"2024-04-26T23:02:47.423709+0000","last_undegraded":"2024-04-26T23:02:47.423709+0000","last_fullsized":"2024-04-26T23:02:47.423709+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:18:53.091762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.410588+0000","last_change":"2024-04-26T23:02:47.410588+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.410588+0000","last_undegraded":"2024-04-26T23:02:47.410588+0000","last_fullsized":"2024-04-26T23:02:47.410588+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430778+0000","last_change":"2024-04-26T23:02:47.426236+0000","last_active":"2024-04-26T23:02:47.430778+0000","last_peered":"2024-04-26T23:02:47.430778+0000","last_clean":"2024-04-26T23:02:47.430778+0000","last_became_active":"2024-04-26T23:02:47.425508+0000","last_became_peered":"2024-04-26T23:02:47.425508+0000","last_unstale":"2024-04-26T23:02:47.430778+0000","last_undegraded":"2024-04-26T23:02:47.430778+0000","last_fullsized":"2024-04-26T23:02:47.430778+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:37:06.151991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411859+0000","last_change":"2024-04-26T23:02:47.411859+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411859+0000","last_undegraded":"2024-04-26T23:02:47.411859+0000","last_fullsized":"2024-04-26T23:02:47.411859+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.425942+0000","last_change":"2024-04-26T23:02:47.422531+0000","last_active":"2024-04-26T23:02:47.425942+0000","last_peered":"2024-04-26T23:02:47.425942+0000","last_clean":"2024-04-26T23:02:47.425942+0000","last_became_active":"2024-04-26T23:02:47.422263+0000","last_became_peered":"2024-04-26T23:02:47.422263+0000","last_unstale":"2024-04-26T23:02:47.425942+0000","last_undegraded":"2024-04-26T23:02:47.425942+0000","last_fullsized":"2024-04-26T23:02:47.425942+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:23.080779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407338+0000","last_change":"2024-04-26T23:02:47.407338+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407338+0000","last_undegraded":"2024-04-26T23:02:47.407338+0000","last_fullsized":"2024-04-26T23:02:47.407338+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428543+0000","last_change":"2024-04-26T23:02:47.425145+0000","last_active":"2024-04-26T23:02:47.428543+0000","last_peered":"2024-04-26T23:02:47.428543+0000","last_clean":"2024-04-26T23:02:47.428543+0000","last_became_active":"2024-04-26T23:02:47.424860+0000","last_became_peered":"2024-04-26T23:02:47.424860+0000","last_unstale":"2024-04-26T23:02:47.428543+0000","last_undegraded":"2024-04-26T23:02:47.428543+0000","last_fullsized":"2024-04-26T23:02:47.428543+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.402648+0000","last_change":"2024-04-26T23:02:47.402648+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.402648+0000","last_undegraded":"2024-04-26T23:02:47.402648+0000","last_fullsized":"2024-04-26T23:02:47.402648+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431674+0000","last_change":"2024-04-26T23:02:47.427171+0000","last_active":"2024-04-26T23:02:47.431674+0000","last_peered":"2024-04-26T23:02:47.431674+0000","last_clean":"2024-04-26T23:02:47.431674+0000","last_became_active":"2024-04-26T23:02:47.426664+0000","last_became_peered":"2024-04-26T23:02:47.426664+0000","last_unstale":"2024-04-26T23:02:47.431674+0000","last_undegraded":"2024-04-26T23:02:47.431674+0000","last_fullsized":"2024-04-26T23:02:47.431674+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:35.059231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.399491+0000","last_change":"2024-04-26T23:02:47.399491+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.399491+0000","last_undegraded":"2024-04-26T23:02:47.399491+0000","last_fullsized":"2024-04-26T23:02:47.399491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.431454+0000","last_change":"2024-04-26T23:02:47.425522+0000","last_active":"2024-04-26T23:02:47.431454+0000","last_peered":"2024-04-26T23:02:47.431454+0000","last_clean":"2024-04-26T23:02:47.431454+0000","last_became_active":"2024-04-26T23:02:47.425249+0000","last_became_peered":"2024-04-26T23:02:47.425249+0000","last_unstale":"2024-04-26T23:02:47.431454+0000","last_undegraded":"2024-04-26T23:02:47.431454+0000","last_fullsized":"2024-04-26T23:02:47.431454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.412312+0000","last_change":"2024-04-26T23:02:47.412312+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.412312+0000","last_undegraded":"2024-04-26T23:02:47.412312+0000","last_fullsized":"2024-04-26T23:02:47.412312+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.419715+0000","last_change":"2024-04-26T23:02:47.414839+0000","last_active":"2024-04-26T23:02:47.419715+0000","last_peered":"2024-04-26T23:02:47.419715+0000","last_clean":"2024-04-26T23:02:47.419715+0000","last_became_active":"2024-04-26T23:02:47.414531+0000","last_became_peered":"2024-04-26T23:02:47.414531+0000","last_unstale":"2024-04-26T23:02:47.419715+0000","last_undegraded":"2024-04-26T23:02:47.419715+0000","last_fullsized":"2024-04-26T23:02:47.419715+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:47:26.056049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409298+0000","last_change":"2024-04-26T23:02:47.409298+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409298+0000","last_undegraded":"2024-04-26T23:02:47.409298+0000","last_fullsized":"2024-04-26T23:02:47.409298+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.420473+0000","last_change":"2024-04-26T23:02:47.417644+0000","last_active":"2024-04-26T23:02:47.420473+0000","last_peered":"2024-04-26T23:02:47.420473+0000","last_clean":"2024-04-26T23:02:47.420473+0000","last_became_active":"2024-04-26T23:02:47.417390+0000","last_became_peered":"2024-04-26T23:02:47.417390+0000","last_unstale":"2024-04-26T23:02:47.420473+0000","last_undegraded":"2024-04-26T23:02:47.420473+0000","last_fullsized":"2024-04-26T23:02:47.420473+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:48:34.048591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409401+0000","last_change":"2024-04-26T23:02:47.409401+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409401+0000","last_undegraded":"2024-04-26T23:02:47.409401+0000","last_fullsized":"2024-04-26T23:02:47.409401+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.429682+0000","last_change":"2024-04-26T23:02:47.425458+0000","last_active":"2024-04-26T23:02:47.429682+0000","last_peered":"2024-04-26T23:02:47.429682+0000","last_clean":"2024-04-26T23:02:47.429682+0000","last_became_active":"2024-04-26T23:02:47.424961+0000","last_became_peered":"2024-04-26T23:02:47.424961+0000","last_unstale":"2024-04-26T23:02:47.429682+0000","last_undegraded":"2024-04-26T23:02:47.429682+0000","last_fullsized":"2024-04-26T23:02:47.429682+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:18:53.091762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.409951+0000","last_change":"2024-04-26T23:02:47.409951+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.409951+0000","last_undegraded":"2024-04-26T23:02:47.409951+0000","last_fullsized":"2024-04-26T23:02:47.409951+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.427504+0000","last_change":"2024-04-26T23:02:47.422918+0000","last_active":"2024-04-26T23:02:47.427504+0000","last_peered":"2024-04-26T23:02:47.427504+0000","last_clean":"2024-04-26T23:02:47.427504+0000","last_became_active":"2024-04-26T23:02:47.422578+0000","last_became_peered":"2024-04-26T23:02:47.422578+0000","last_unstale":"2024-04-26T23:02:47.427504+0000","last_undegraded":"2024-04-26T23:02:47.427504+0000","last_fullsized":"2024-04-26T23:02:47.427504+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:02:18.489647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.407819+0000","last_change":"2024-04-26T23:02:47.407819+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.407819+0000","last_undegraded":"2024-04-26T23:02:47.407819+0000","last_fullsized":"2024-04-26T23:02:47.407819+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.430568+0000","last_change":"2024-04-26T23:02:47.427276+0000","last_active":"2024-04-26T23:02:47.430568+0000","last_peered":"2024-04-26T23:02:47.430568+0000","last_clean":"2024-04-26T23:02:47.430568+0000","last_became_active":"2024-04-26T23:02:47.426167+0000","last_became_peered":"2024-04-26T23:02:47.426167+0000","last_unstale":"2024-04-26T23:02:47.430568+0000","last_undegraded":"2024-04-26T23:02:47.430568+0000","last_fullsized":"2024-04-26T23:02:47.430568+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.420271+0000","last_change":"2024-04-26T23:02:47.420271+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.420271+0000","last_undegraded":"2024-04-26T23:02:47.420271+0000","last_fullsized":"2024-04-26T23:02:47.420271+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.419691+0000","last_change":"2024-04-26T23:02:47.415885+0000","last_active":"2024-04-26T23:02:47.419691+0000","last_peered":"2024-04-26T23:02:47.419691+0000","last_clean":"2024-04-26T23:02:47.419691+0000","last_became_active":"2024-04-26T23:02:47.415613+0000","last_became_peered":"2024-04-26T23:02:47.415613+0000","last_unstale":"2024-04-26T23:02:47.419691+0000","last_undegraded":"2024-04-26T23:02:47.419691+0000","last_fullsized":"2024-04-26T23:02:47.419691+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:46:24.430302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.404728+0000","last_change":"2024-04-26T23:02:47.404728+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.404728+0000","last_undegraded":"2024-04-26T23:02:47.404728+0000","last_fullsized":"2024-04-26T23:02:47.404728+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428564+0000","last_change":"2024-04-26T23:02:47.423354+0000","last_active":"2024-04-26T23:02:47.428564+0000","last_peered":"2024-04-26T23:02:47.428564+0000","last_clean":"2024-04-26T23:02:47.428564+0000","last_became_active":"2024-04-26T23:02:47.423058+0000","last_became_peered":"2024-04-26T23:02:47.423058+0000","last_unstale":"2024-04-26T23:02:47.428564+0000","last_undegraded":"2024-04-26T23:02:47.428564+0000","last_fullsized":"2024-04-26T23:02:47.428564+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.411594+0000","last_change":"2024-04-26T23:02:47.411594+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.411594+0000","last_undegraded":"2024-04-26T23:02:47.411594+0000","last_fullsized":"2024-04-26T23:02:47.411594+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.428596+0000","last_change":"2024-04-26T23:02:47.421824+0000","last_active":"2024-04-26T23:02:47.428596+0000","last_peered":"2024-04-26T23:02:47.428596+0000","last_clean":"2024-04-26T23:02:47.428596+0000","last_became_active":"2024-04-26T23:02:47.421537+0000","last_became_peered":"2024-04-26T23:02:47.421537+0000","last_unstale":"2024-04-26T23:02:47.428596+0000","last_undegraded":"2024-04-26T23:02:47.428596+0000","last_fullsized":"2024-04-26T23:02:47.428596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:23.080779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.406234+0000","last_change":"2024-04-26T23:02:47.406234+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.406234+0000","last_undegraded":"2024-04-26T23:02:47.406234+0000","last_fullsized":"2024-04-26T23:02:47.406234+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.425058+0000","last_change":"2024-04-26T23:02:47.422919+0000","last_active":"2024-04-26T23:02:47.425058+0000","last_peered":"2024-04-26T23:02:47.425058+0000","last_clean":"2024-04-26T23:02:47.425058+0000","last_became_active":"2024-04-26T23:02:47.421694+0000","last_became_peered":"2024-04-26T23:02:47.421694+0000","last_unstale":"2024-04-26T23:02:47.425058+0000","last_undegraded":"2024-04-26T23:02:47.425058+0000","last_fullsized":"2024-04-26T23:02:47.425058+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:37:06.151991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.403814+0000","last_change":"2024-04-26T23:02:47.403814+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.403814+0000","last_undegraded":"2024-04-26T23:02:47.403814+0000","last_fullsized":"2024-04-26T23:02:47.403814+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.423560+0000","last_change":"2024-04-26T23:02:47.419127+0000","last_active":"2024-04-26T23:02:47.423560+0000","last_peered":"2024-04-26T23:02:47.423560+0000","last_clean":"2024-04-26T23:02:47.423560+0000","last_became_active":"2024-04-26T23:02:47.417694+0000","last_became_peered":"2024-04-26T23:02:47.417694+0000","last_unstale":"2024-04-26T23:02:47.423560+0000","last_undegraded":"2024-04-26T23:02:47.423560+0000","last_fullsized":"2024-04-26T23:02:47.423560+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.415211+0000","last_change":"2024-04-26T23:02:47.415211+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.415211+0000","last_undegraded":"2024-04-26T23:02:47.415211+0000","last_fullsized":"2024-04-26T23:02:47.415211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.419342+0000","last_change":"2024-04-26T23:02:47.416825+0000","last_active":"2024-04-26T23:02:47.419342+0000","last_peered":"2024-04-26T23:02:47.419342+0000","last_clean":"2024-04-26T23:02:47.419342+0000","last_became_active":"2024-04-26T23:02:47.416571+0000","last_became_peered":"2024-04-26T23:02:47.416571+0000","last_unstale":"2024-04-26T23:02:47.419342+0000","last_undegraded":"2024-04-26T23:02:47.419342+0000","last_fullsized":"2024-04-26T23:02:47.419342+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:25:15.376546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-26T23:02:47.406994+0000","last_change":"2024-04-26T23:02:47.406994+0000","last_active":"2024-04-26T23:02:47.377264+0000","last_peered":"2024-04-26T23:02:47.377264+0000","last_clean":"2024-04-26T23:02:47.377264+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:02:47.406994+0000","last_undegraded":"2024-04-26T23:02:47.406994+0000","last_fullsized":"2024-04-26T23:02:47.406994+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.433433+0000","last_change":"2024-04-26T23:02:47.432253+0000","last_active":"2024-04-26T23:02:47.433433+0000","last_peered":"2024-04-26T23:02:47.433433+0000","last_clean":"2024-04-26T23:02:47.433433+0000","last_became_active":"2024-04-26T23:02:47.431955+0000","last_became_peered":"2024-04-26T23:02:47.431955+0000","last_unstale":"2024-04-26T23:02:47.433433+0000","last_undegraded":"2024-04-26T23:02:47.433433+0000","last_fullsized":"2024-04-26T23:02:47.433433+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:27.125135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":203,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-26T23:02:47.413362+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:02:47.413362+0000","last_peered":"2024-04-26T23:02:47.413362+0000","last_clean":"2024-04-26T23:02:47.413362+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:02:47.413362+0000","last_undegraded":"2024-04-26T23:02:47.413362+0000","last_fullsized":"2024-04-26T23:02:47.413362+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":766999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":183298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1019904,"data_stored":773666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:02:58.026 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:02:58.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.252+0000 7fa9eec00640 1 Processor -- start 2024-04-26T23:02:58.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.252+0000 7fa9eec00640 1 -- start start 2024-04-26T23:02:58.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.254+0000 7fa9eec00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa9e80d32e0 0x7fa9e80d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:58.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.254+0000 7fa9eec00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9e8162c40 con 0x7fa9e80b8d20 2024-04-26T23:02:58.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.254+0000 7fa9eec00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9e80d2d80 con 0x7fa9e80b6220 2024-04-26T23:02:58.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.254+0000 7fa9eec00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa9e80d3c80 con 0x7fa9e80d32e0 2024-04-26T23:02:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.254+0000 7fa9ece00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa9e80d32e0 0x7fa9e80d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.254+0000 7fa9ece00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa9e80d32e0 0x7fa9e80d3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36726/0 (socket says 172.21.15.40:36726) 2024-04-26T23:02:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.254+0000 7fa9ee200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa9e80b8d20 0x7fa9e80b9160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42880/0 (socket says 172.21.15.40:42880) 2024-04-26T23:02:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.254+0000 7fa9ece00640 1 -- 172.21.15.40:0/3327250692 learned_addr learned my addr 172.21.15.40:0/3327250692 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.255+0000 7fa9ece00640 1 -- 172.21.15.40:0/3327250692 >> v1:172.21.15.103:6789/0 conn(0x7fa9e80b6220 legacy=0x7fa9e80d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.255+0000 7fa9ece00640 1 -- 172.21.15.40:0/3327250692 >> v1:172.21.15.40:6789/0 conn(0x7fa9e80b8d20 legacy=0x7fa9e80b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.255+0000 7fa9ece00640 1 -- 172.21.15.40:0/3327250692 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e80b7c80 con 0x7fa9e80d32e0 2024-04-26T23:02:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.255+0000 7fa9e7e00640 1 -- 172.21.15.40:0/3327250692 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2902169616 0 0) 0x7fa9e8162c40 con 0x7fa9e80b8d20 2024-04-26T23:02:58.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.256+0000 7fa9ece00640 1 --2- 172.21.15.40:0/3327250692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9e80d32e0 0x7fa9e80d3740 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009870 tx=0x7fa9d402a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda2c5763c1bd987 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:58.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.256+0000 7fa9e7e00640 1 -- 172.21.15.40:0/3327250692 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4037070 con 0x7fa9e80d32e0 2024-04-26T23:02:58.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.256+0000 7fa9e7e00640 1 -- 172.21.15.40:0/3327250692 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9d4032440 con 0x7fa9e80d32e0 2024-04-26T23:02:58.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.256+0000 7fa9e7e00640 1 -- 172.21.15.40:0/3327250692 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4032810 con 0x7fa9e80d32e0 2024-04-26T23:02:58.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.257+0000 7fa9eec00640 1 -- 172.21.15.40:0/3327250692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9e80d32e0 msgr2=0x7fa9e80d3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:58.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.257+0000 7fa9eec00640 1 --2- 172.21.15.40:0/3327250692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9e80d32e0 0x7fa9e80d3740 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009870 tx=0x7fa9d402a2a0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.257+0000 7fa9eec00640 1 -- 172.21.15.40:0/3327250692 shutdown_connections 2024-04-26T23:02:58.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.257+0000 7fa9eec00640 1 --2- 172.21.15.40:0/3327250692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9e80d32e0 0x7fa9e80d3740 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.257+0000 7fa9eec00640 1 -- 172.21.15.40:0/3327250692 >> 172.21.15.40:0/3327250692 conn(0x7fa9e80d4d90 msgr2=0x7fa9e80d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:58.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.257+0000 7fa9eec00640 1 -- 172.21.15.40:0/3327250692 shutdown_connections 2024-04-26T23:02:58.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.257+0000 7fa9eec00640 1 -- 172.21.15.40:0/3327250692 wait complete. 2024-04-26T23:02:58.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.259+0000 7fa9eec00640 1 Processor -- start 2024-04-26T23:02:58.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.259+0000 7fa9eec00640 1 -- start start 2024-04-26T23:02:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.260+0000 7fa9eec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9e80b6220 0x7fa9e809f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.260+0000 7fa9eec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9e80b8d20 0x7fa9e81f83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.260+0000 7fa9eec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e80d32e0 0x7fa9e81f8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.260+0000 7fa9eec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e80b8520 con 0x7fa9e80b8d20 2024-04-26T23:02:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.260+0000 7fa9eec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e80b8240 con 0x7fa9e80d32e0 2024-04-26T23:02:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.260+0000 7fa9eec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e80d0aa0 con 0x7fa9e80b6220 2024-04-26T23:02:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.260+0000 7fa9ece00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9e80b8d20 0x7fa9e81f83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.260+0000 7fa9ece00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9e80b8d20 0x7fa9e81f83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36730/0 (socket says 172.21.15.40:36730) 2024-04-26T23:02:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.260+0000 7fa9ece00640 1 -- 172.21.15.40:0/278928014 learned_addr learned my addr 172.21.15.40:0/278928014 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:58.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.261+0000 7fa9ed800640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9e80b6220 0x7fa9e809f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:58.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.261+0000 7fa9ee200640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e80d32e0 0x7fa9e81f8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:58.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.261+0000 7fa9ece00640 1 -- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9e80b6220 msgr2=0x7fa9e809f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:58.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.261+0000 7fa9ece00640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9e80b6220 0x7fa9e809f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.261+0000 7fa9ece00640 1 -- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e80d32e0 msgr2=0x7fa9e81f8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:58.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.261+0000 7fa9ece00640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e80d32e0 0x7fa9e81f8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.261+0000 7fa9ece00640 1 -- 172.21.15.40:0/278928014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e8159d50 con 0x7fa9e80b8d20 2024-04-26T23:02:58.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.261+0000 7fa9ed800640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9e80b6220 0x7fa9e809f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:58.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.261+0000 7fa9ece00640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9e80b8d20 0x7fa9e81f83c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40090b0 tx=0x7fa9d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:58.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.261+0000 7fa9e5600640 1 -- 172.21.15.40:0/278928014 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4042020 con 0x7fa9e80b8d20 2024-04-26T23:02:58.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.262+0000 7fa9e5600640 1 -- 172.21.15.40:0/278928014 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9d4030990 con 0x7fa9e80b8d20 2024-04-26T23:02:58.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.262+0000 7fa9e5600640 1 -- 172.21.15.40:0/278928014 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4037040 con 0x7fa9e80b8d20 2024-04-26T23:02:58.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.262+0000 7fa9eec00640 1 -- 172.21.15.40:0/278928014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e80a01a0 con 0x7fa9e80b8d20 2024-04-26T23:02:58.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.262+0000 7fa9eec00640 1 -- 172.21.15.40:0/278928014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e80a0760 con 0x7fa9e80b8d20 2024-04-26T23:02:58.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.263+0000 7fa9eec00640 1 -- 172.21.15.40:0/278928014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e80d1220 con 0x7fa9e80b8d20 2024-04-26T23:02:58.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.264+0000 7fa9e5600640 1 -- 172.21.15.40:0/278928014 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa9d4045020 con 0x7fa9e80b8d20 2024-04-26T23:02:58.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.265+0000 7fa9e5600640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa9c806c920 0x7fa9c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:58.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.265+0000 7fa9e5600640 1 -- 172.21.15.40:0/278928014 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fa9d40ac9a0 con 0x7fa9e80b8d20 2024-04-26T23:02:58.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.266+0000 7fa9ed800640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa9c806c920 0x7fa9c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:58.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.267+0000 7fa9ed800640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa9c806c920 0x7fa9c806ede0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc003eb0 tx=0x7fa9dc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:58.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.270+0000 7fa9e5600640 1 -- 172.21.15.40:0/278928014 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa9e80d1220 con 0x7fa9e80b8d20 2024-04-26T23:02:58.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.469+0000 7fa9eec00640 1 -- 172.21.15.40:0/278928014 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa9e8069370 con 0x7fa9c806c920 2024-04-26T23:02:58.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.491+0000 7fa9e5600640 1 -- 172.21.15.40:0/278928014 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251963 (secure 0 0 0) 0x7fa9dc055850 con 0x7fa9c806c920 2024-04-26T23:02:58.495 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:58.496 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:02:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.498+0000 7fa9eec00640 1 -- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa9c806c920 msgr2=0x7fa9c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.498+0000 7fa9eec00640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa9c806c920 0x7fa9c806ede0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc003eb0 tx=0x7fa9dc009040 comp rx=0 tx=0).stop 2024-04-26T23:02:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.498+0000 7fa9eec00640 1 -- 172.21.15.40:0/278928014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9e80b8d20 msgr2=0x7fa9e81f83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.498+0000 7fa9eec00640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9e80b8d20 0x7fa9e81f83c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40090b0 tx=0x7fa9d40046c0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.498+0000 7fa9eec00640 1 -- 172.21.15.40:0/278928014 shutdown_connections 2024-04-26T23:02:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.498+0000 7fa9eec00640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa9c806c920 0x7fa9c806ede0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.498+0000 7fa9eec00640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e80d32e0 0x7fa9e81f8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.498+0000 7fa9eec00640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9e80b8d20 0x7fa9e81f83c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.498+0000 7fa9eec00640 1 --2- 172.21.15.40:0/278928014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa9e80b6220 0x7fa9e809f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.498+0000 7fa9eec00640 1 -- 172.21.15.40:0/278928014 >> 172.21.15.40:0/278928014 conn(0x7fa9e80d4d90 msgr2=0x7fa9e806e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.498+0000 7fa9eec00640 1 -- 172.21.15.40:0/278928014 shutdown_connections 2024-04-26T23:02:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.498+0000 7fa9eec00640 1 -- 172.21.15.40:0/278928014 wait complete. 2024-04-26T23:02:58.518 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-04-26T23:02:57.184584+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220936,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753784,"statfs":{"total":773094113280,"available":772867874816,"internally_reserved":0,"allocated":4857856,"data_stored":2762812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.786833"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.424469+0000","last_change":"2024-04-26T23:02:48.418508+0000","last_active":"2024-04-26T23:02:49.424469+0000","last_peered":"2024-04-26T23:02:49.424469+0000","last_clean":"2024-04-26T23:02:49.424469+0000","last_became_active":"2024-04-26T23:02:48.418119+0000","last_became_peered":"2024-04-26T23:02:48.418119+0000","last_unstale":"2024-04-26T23:02:49.424469+0000","last_undegraded":"2024-04-26T23:02:49.424469+0000","last_fullsized":"2024-04-26T23:02:49.424469+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:23:43.783054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.424757+0000","last_change":"2024-04-26T23:02:48.417290+0000","last_active":"2024-04-26T23:02:49.424757+0000","last_peered":"2024-04-26T23:02:49.424757+0000","last_clean":"2024-04-26T23:02:49.424757+0000","last_became_active":"2024-04-26T23:02:48.416981+0000","last_became_peered":"2024-04-26T23:02:48.416981+0000","last_unstale":"2024-04-26T23:02:49.424757+0000","last_undegraded":"2024-04-26T23:02:49.424757+0000","last_fullsized":"2024-04-26T23:02:49.424757+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:27.032255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.061734+0000","last_change":"2024-04-26T23:02:48.438667+0000","last_active":"2024-04-26T23:02:50.061734+0000","last_peered":"2024-04-26T23:02:50.061734+0000","last_clean":"2024-04-26T23:02:50.061734+0000","last_became_active":"2024-04-26T23:02:48.438327+0000","last_became_peered":"2024-04-26T23:02:48.438327+0000","last_unstale":"2024-04-26T23:02:50.061734+0000","last_undegraded":"2024-04-26T23:02:50.061734+0000","last_fullsized":"2024-04-26T23:02:50.061734+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:29:39.493442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178941+0000","last_change":"2024-04-26T23:02:48.418051+0000","last_active":"2024-04-26T23:02:50.178941+0000","last_peered":"2024-04-26T23:02:50.178941+0000","last_clean":"2024-04-26T23:02:50.178941+0000","last_became_active":"2024-04-26T23:02:48.417762+0000","last_became_peered":"2024-04-26T23:02:48.417762+0000","last_unstale":"2024-04-26T23:02:50.178941+0000","last_undegraded":"2024-04-26T23:02:50.178941+0000","last_fullsized":"2024-04-26T23:02:50.178941+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:14.613748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178337+0000","last_change":"2024-04-26T23:02:48.418903+0000","last_active":"2024-04-26T23:02:50.178337+0000","last_peered":"2024-04-26T23:02:50.178337+0000","last_clean":"2024-04-26T23:02:50.178337+0000","last_became_active":"2024-04-26T23:02:48.418708+0000","last_became_peered":"2024-04-26T23:02:48.418708+0000","last_unstale":"2024-04-26T23:02:50.178337+0000","last_undegraded":"2024-04-26T23:02:50.178337+0000","last_fullsized":"2024-04-26T23:02:50.178337+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:24.096704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.176438+0000","last_change":"2024-04-26T23:02:48.420454+0000","last_active":"2024-04-26T23:02:50.176438+0000","last_peered":"2024-04-26T23:02:50.176438+0000","last_clean":"2024-04-26T23:02:50.176438+0000","last_became_active":"2024-04-26T23:02:48.420059+0000","last_became_peered":"2024-04-26T23:02:48.420059+0000","last_unstale":"2024-04-26T23:02:50.176438+0000","last_undegraded":"2024-04-26T23:02:50.176438+0000","last_fullsized":"2024-04-26T23:02:50.176438+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:03:38.777288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.176842+0000","last_change":"2024-04-26T23:02:48.434270+0000","last_active":"2024-04-26T23:02:50.176842+0000","last_peered":"2024-04-26T23:02:50.176842+0000","last_clean":"2024-04-26T23:02:50.176842+0000","last_became_active":"2024-04-26T23:02:48.433406+0000","last_became_peered":"2024-04-26T23:02:48.433406+0000","last_unstale":"2024-04-26T23:02:50.176842+0000","last_undegraded":"2024-04-26T23:02:50.176842+0000","last_fullsized":"2024-04-26T23:02:50.176842+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:11.262736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.423695+0000","last_change":"2024-04-26T23:02:48.434102+0000","last_active":"2024-04-26T23:02:49.423695+0000","last_peered":"2024-04-26T23:02:49.423695+0000","last_clean":"2024-04-26T23:02:49.423695+0000","last_became_active":"2024-04-26T23:02:48.433780+0000","last_became_peered":"2024-04-26T23:02:48.433780+0000","last_unstale":"2024-04-26T23:02:49.423695+0000","last_undegraded":"2024-04-26T23:02:49.423695+0000","last_fullsized":"2024-04-26T23:02:49.423695+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:04.420466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.062116+0000","last_change":"2024-04-26T23:02:48.443600+0000","last_active":"2024-04-26T23:02:50.062116+0000","last_peered":"2024-04-26T23:02:50.062116+0000","last_clean":"2024-04-26T23:02:50.062116+0000","last_became_active":"2024-04-26T23:02:48.443373+0000","last_became_peered":"2024-04-26T23:02:48.443373+0000","last_unstale":"2024-04-26T23:02:50.062116+0000","last_undegraded":"2024-04-26T23:02:50.062116+0000","last_fullsized":"2024-04-26T23:02:50.062116+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:11.262736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177683+0000","last_change":"2024-04-26T23:02:48.447679+0000","last_active":"2024-04-26T23:02:50.177683+0000","last_peered":"2024-04-26T23:02:50.177683+0000","last_clean":"2024-04-26T23:02:50.177683+0000","last_became_active":"2024-04-26T23:02:48.445804+0000","last_became_peered":"2024-04-26T23:02:48.445804+0000","last_unstale":"2024-04-26T23:02:50.177683+0000","last_undegraded":"2024-04-26T23:02:50.177683+0000","last_fullsized":"2024-04-26T23:02:50.177683+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:04.420466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177666+0000","last_change":"2024-04-26T23:02:48.437436+0000","last_active":"2024-04-26T23:02:50.177666+0000","last_peered":"2024-04-26T23:02:50.177666+0000","last_clean":"2024-04-26T23:02:50.177666+0000","last_became_active":"2024-04-26T23:02:48.432731+0000","last_became_peered":"2024-04-26T23:02:48.432731+0000","last_unstale":"2024-04-26T23:02:50.177666+0000","last_undegraded":"2024-04-26T23:02:50.177666+0000","last_fullsized":"2024-04-26T23:02:50.177666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:16:21.734848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.061951+0000","last_change":"2024-04-26T23:02:48.422151+0000","last_active":"2024-04-26T23:02:50.061951+0000","last_peered":"2024-04-26T23:02:50.061951+0000","last_clean":"2024-04-26T23:02:50.061951+0000","last_became_active":"2024-04-26T23:02:48.421824+0000","last_became_peered":"2024-04-26T23:02:48.421824+0000","last_unstale":"2024-04-26T23:02:50.061951+0000","last_undegraded":"2024-04-26T23:02:50.061951+0000","last_fullsized":"2024-04-26T23:02:50.061951+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:46.637432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177626+0000","last_change":"2024-04-26T23:02:48.442483+0000","last_active":"2024-04-26T23:02:50.177626+0000","last_peered":"2024-04-26T23:02:50.177626+0000","last_clean":"2024-04-26T23:02:50.177626+0000","last_became_active":"2024-04-26T23:02:48.442197+0000","last_became_peered":"2024-04-26T23:02:48.442197+0000","last_unstale":"2024-04-26T23:02:50.177626+0000","last_undegraded":"2024-04-26T23:02:50.177626+0000","last_fullsized":"2024-04-26T23:02:50.177626+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:03:44.629813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.610648+0000","last_change":"2024-04-26T23:02:48.425051+0000","last_active":"2024-04-26T23:02:49.610648+0000","last_peered":"2024-04-26T23:02:49.610648+0000","last_clean":"2024-04-26T23:02:49.610648+0000","last_became_active":"2024-04-26T23:02:48.424635+0000","last_became_peered":"2024-04-26T23:02:48.424635+0000","last_unstale":"2024-04-26T23:02:49.610648+0000","last_undegraded":"2024-04-26T23:02:49.610648+0000","last_fullsized":"2024-04-26T23:02:49.610648+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:29.486859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177096+0000","last_change":"2024-04-26T23:02:48.438533+0000","last_active":"2024-04-26T23:02:50.177096+0000","last_peered":"2024-04-26T23:02:50.177096+0000","last_clean":"2024-04-26T23:02:50.177096+0000","last_became_active":"2024-04-26T23:02:48.437740+0000","last_became_peered":"2024-04-26T23:02:48.437740+0000","last_unstale":"2024-04-26T23:02:50.177096+0000","last_undegraded":"2024-04-26T23:02:50.177096+0000","last_fullsized":"2024-04-26T23:02:50.177096+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:46.899581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.427591+0000","last_change":"2024-04-26T23:02:48.448948+0000","last_active":"2024-04-26T23:02:49.427591+0000","last_peered":"2024-04-26T23:02:49.427591+0000","last_clean":"2024-04-26T23:02:49.427591+0000","last_became_active":"2024-04-26T23:02:48.448567+0000","last_became_peered":"2024-04-26T23:02:48.448567+0000","last_unstale":"2024-04-26T23:02:49.427591+0000","last_undegraded":"2024-04-26T23:02:49.427591+0000","last_fullsized":"2024-04-26T23:02:49.427591+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:08:40.788946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.429800+0000","last_change":"2024-04-26T23:02:48.449617+0000","last_active":"2024-04-26T23:02:49.429800+0000","last_peered":"2024-04-26T23:02:49.429800+0000","last_clean":"2024-04-26T23:02:49.429800+0000","last_became_active":"2024-04-26T23:02:48.449205+0000","last_became_peered":"2024-04-26T23:02:48.449205+0000","last_unstale":"2024-04-26T23:02:49.429800+0000","last_undegraded":"2024-04-26T23:02:49.429800+0000","last_fullsized":"2024-04-26T23:02:49.429800+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:46.637432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.428647+0000","last_change":"2024-04-26T23:02:48.428830+0000","last_active":"2024-04-26T23:02:49.428647+0000","last_peered":"2024-04-26T23:02:49.428647+0000","last_clean":"2024-04-26T23:02:49.428647+0000","last_became_active":"2024-04-26T23:02:48.427638+0000","last_became_peered":"2024-04-26T23:02:48.427638+0000","last_unstale":"2024-04-26T23:02:49.428647+0000","last_undegraded":"2024-04-26T23:02:49.428647+0000","last_fullsized":"2024-04-26T23:02:49.428647+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:26.954291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.179160+0000","last_change":"2024-04-26T23:02:48.412991+0000","last_active":"2024-04-26T23:02:50.179160+0000","last_peered":"2024-04-26T23:02:50.179160+0000","last_clean":"2024-04-26T23:02:50.179160+0000","last_became_active":"2024-04-26T23:02:48.412643+0000","last_became_peered":"2024-04-26T23:02:48.412643+0000","last_unstale":"2024-04-26T23:02:50.179160+0000","last_undegraded":"2024-04-26T23:02:50.179160+0000","last_fullsized":"2024-04-26T23:02:50.179160+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:29.838551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.427762+0000","last_change":"2024-04-26T23:02:48.430191+0000","last_active":"2024-04-26T23:02:49.427762+0000","last_peered":"2024-04-26T23:02:49.427762+0000","last_clean":"2024-04-26T23:02:49.427762+0000","last_became_active":"2024-04-26T23:02:48.429874+0000","last_became_peered":"2024-04-26T23:02:48.429874+0000","last_unstale":"2024-04-26T23:02:49.427762+0000","last_undegraded":"2024-04-26T23:02:49.427762+0000","last_fullsized":"2024-04-26T23:02:49.427762+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:49:24.223830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178118+0000","last_change":"2024-04-26T23:02:48.435556+0000","last_active":"2024-04-26T23:02:50.178118+0000","last_peered":"2024-04-26T23:02:50.178118+0000","last_clean":"2024-04-26T23:02:50.178118+0000","last_became_active":"2024-04-26T23:02:48.435195+0000","last_became_peered":"2024-04-26T23:02:48.435195+0000","last_unstale":"2024-04-26T23:02:50.178118+0000","last_undegraded":"2024-04-26T23:02:50.178118+0000","last_fullsized":"2024-04-26T23:02:50.178118+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:09:16.546571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.611331+0000","last_change":"2024-04-26T23:02:48.423009+0000","last_active":"2024-04-26T23:02:49.611331+0000","last_peered":"2024-04-26T23:02:49.611331+0000","last_clean":"2024-04-26T23:02:49.611331+0000","last_became_active":"2024-04-26T23:02:48.422115+0000","last_became_peered":"2024-04-26T23:02:48.422115+0000","last_unstale":"2024-04-26T23:02:49.611331+0000","last_undegraded":"2024-04-26T23:02:49.611331+0000","last_fullsized":"2024-04-26T23:02:49.611331+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:46.637432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.429617+0000","last_change":"2024-04-26T23:02:48.444743+0000","last_active":"2024-04-26T23:02:49.429617+0000","last_peered":"2024-04-26T23:02:49.429617+0000","last_clean":"2024-04-26T23:02:49.429617+0000","last_became_active":"2024-04-26T23:02:48.443743+0000","last_became_peered":"2024-04-26T23:02:48.443743+0000","last_unstale":"2024-04-26T23:02:49.429617+0000","last_undegraded":"2024-04-26T23:02:49.429617+0000","last_fullsized":"2024-04-26T23:02:49.429617+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:29.486859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177226+0000","last_change":"2024-04-26T23:02:48.434344+0000","last_active":"2024-04-26T23:02:50.177226+0000","last_peered":"2024-04-26T23:02:50.177226+0000","last_clean":"2024-04-26T23:02:50.177226+0000","last_became_active":"2024-04-26T23:02:48.433188+0000","last_became_peered":"2024-04-26T23:02:48.433188+0000","last_unstale":"2024-04-26T23:02:50.177226+0000","last_undegraded":"2024-04-26T23:02:50.177226+0000","last_fullsized":"2024-04-26T23:02:50.177226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:28.141060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.428982+0000","last_change":"2024-04-26T23:02:48.417918+0000","last_active":"2024-04-26T23:02:49.428982+0000","last_peered":"2024-04-26T23:02:49.428982+0000","last_clean":"2024-04-26T23:02:49.428982+0000","last_became_active":"2024-04-26T23:02:48.417340+0000","last_became_peered":"2024-04-26T23:02:48.417340+0000","last_unstale":"2024-04-26T23:02:49.428982+0000","last_undegraded":"2024-04-26T23:02:49.428982+0000","last_fullsized":"2024-04-26T23:02:49.428982+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:25:02.720003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177496+0000","last_change":"2024-04-26T23:02:48.444276+0000","last_active":"2024-04-26T23:02:50.177496+0000","last_peered":"2024-04-26T23:02:50.177496+0000","last_clean":"2024-04-26T23:02:50.177496+0000","last_became_active":"2024-04-26T23:02:48.443987+0000","last_became_peered":"2024-04-26T23:02:48.443987+0000","last_unstale":"2024-04-26T23:02:50.177496+0000","last_undegraded":"2024-04-26T23:02:50.177496+0000","last_fullsized":"2024-04-26T23:02:50.177496+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:29:39.493442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.428029+0000","last_change":"2024-04-26T23:02:48.426976+0000","last_active":"2024-04-26T23:02:49.428029+0000","last_peered":"2024-04-26T23:02:49.428029+0000","last_clean":"2024-04-26T23:02:49.428029+0000","last_became_active":"2024-04-26T23:02:48.424925+0000","last_became_peered":"2024-04-26T23:02:48.424925+0000","last_unstale":"2024-04-26T23:02:49.428029+0000","last_undegraded":"2024-04-26T23:02:49.428029+0000","last_fullsized":"2024-04-26T23:02:49.428029+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:25:16.392471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177573+0000","last_change":"2024-04-26T23:02:48.437287+0000","last_active":"2024-04-26T23:02:50.177573+0000","last_peered":"2024-04-26T23:02:50.177573+0000","last_clean":"2024-04-26T23:02:50.177573+0000","last_became_active":"2024-04-26T23:02:48.436893+0000","last_became_peered":"2024-04-26T23:02:48.436893+0000","last_unstale":"2024-04-26T23:02:50.177573+0000","last_undegraded":"2024-04-26T23:02:50.177573+0000","last_fullsized":"2024-04-26T23:02:50.177573+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:47:29.052371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.428491+0000","last_change":"2024-04-26T23:02:48.422527+0000","last_active":"2024-04-26T23:02:49.428491+0000","last_peered":"2024-04-26T23:02:49.428491+0000","last_clean":"2024-04-26T23:02:49.428491+0000","last_became_active":"2024-04-26T23:02:48.420616+0000","last_became_peered":"2024-04-26T23:02:48.420616+0000","last_unstale":"2024-04-26T23:02:49.428491+0000","last_undegraded":"2024-04-26T23:02:49.428491+0000","last_fullsized":"2024-04-26T23:02:49.428491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:56:07.234217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.427552+0000","last_change":"2024-04-26T23:02:48.428597+0000","last_active":"2024-04-26T23:02:49.427552+0000","last_peered":"2024-04-26T23:02:49.427552+0000","last_clean":"2024-04-26T23:02:49.427552+0000","last_became_active":"2024-04-26T23:02:48.425326+0000","last_became_peered":"2024-04-26T23:02:48.425326+0000","last_unstale":"2024-04-26T23:02:49.427552+0000","last_undegraded":"2024-04-26T23:02:49.427552+0000","last_fullsized":"2024-04-26T23:02:49.427552+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:00:20.199701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.430026+0000","last_change":"2024-04-26T23:02:48.443448+0000","last_active":"2024-04-26T23:02:49.430026+0000","last_peered":"2024-04-26T23:02:49.430026+0000","last_clean":"2024-04-26T23:02:49.430026+0000","last_became_active":"2024-04-26T23:02:48.443176+0000","last_became_peered":"2024-04-26T23:02:48.443176+0000","last_unstale":"2024-04-26T23:02:49.430026+0000","last_undegraded":"2024-04-26T23:02:49.430026+0000","last_fullsized":"2024-04-26T23:02:49.430026+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:11.262736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.176525+0000","last_change":"2024-04-26T23:02:48.410465+0000","last_active":"2024-04-26T23:02:50.176525+0000","last_peered":"2024-04-26T23:02:50.176525+0000","last_clean":"2024-04-26T23:02:50.176525+0000","last_became_active":"2024-04-26T23:02:48.410189+0000","last_became_peered":"2024-04-26T23:02:48.410189+0000","last_unstale":"2024-04-26T23:02:50.176525+0000","last_undegraded":"2024-04-26T23:02:50.176525+0000","last_fullsized":"2024-04-26T23:02:50.176525+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:08:40.788946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.429171+0000","last_change":"2024-04-26T23:02:48.449712+0000","last_active":"2024-04-26T23:02:49.429171+0000","last_peered":"2024-04-26T23:02:49.429171+0000","last_clean":"2024-04-26T23:02:49.429171+0000","last_became_active":"2024-04-26T23:02:48.448732+0000","last_became_peered":"2024-04-26T23:02:48.448732+0000","last_unstale":"2024-04-26T23:02:49.429171+0000","last_undegraded":"2024-04-26T23:02:49.429171+0000","last_fullsized":"2024-04-26T23:02:49.429171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:33:33.799026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.429220+0000","last_change":"2024-04-26T23:02:47.416086+0000","last_active":"2024-04-26T23:02:49.429220+0000","last_peered":"2024-04-26T23:02:49.429220+0000","last_clean":"2024-04-26T23:02:49.429220+0000","last_became_active":"2024-04-26T23:02:47.415794+0000","last_became_peered":"2024-04-26T23:02:47.415794+0000","last_unstale":"2024-04-26T23:02:49.429220+0000","last_undegraded":"2024-04-26T23:02:49.429220+0000","last_fullsized":"2024-04-26T23:02:49.429220+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:27.125135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177576+0000","last_change":"2024-04-26T23:02:48.439160+0000","last_active":"2024-04-26T23:02:50.177576+0000","last_peered":"2024-04-26T23:02:50.177576+0000","last_clean":"2024-04-26T23:02:50.177576+0000","last_became_active":"2024-04-26T23:02:48.438458+0000","last_became_peered":"2024-04-26T23:02:48.438458+0000","last_unstale":"2024-04-26T23:02:50.177576+0000","last_undegraded":"2024-04-26T23:02:50.177576+0000","last_fullsized":"2024-04-26T23:02:50.177576+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:46:46.637432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.061849+0000","last_change":"2024-04-26T23:02:47.422501+0000","last_active":"2024-04-26T23:02:50.061849+0000","last_peered":"2024-04-26T23:02:50.061849+0000","last_clean":"2024-04-26T23:02:50.061849+0000","last_became_active":"2024-04-26T23:02:47.422240+0000","last_became_peered":"2024-04-26T23:02:47.422240+0000","last_unstale":"2024-04-26T23:02:50.061849+0000","last_undegraded":"2024-04-26T23:02:50.061849+0000","last_fullsized":"2024-04-26T23:02:50.061849+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:28.822626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.611933+0000","last_change":"2024-04-26T23:02:48.431395+0000","last_active":"2024-04-26T23:02:49.611933+0000","last_peered":"2024-04-26T23:02:49.611933+0000","last_clean":"2024-04-26T23:02:49.611933+0000","last_became_active":"2024-04-26T23:02:48.431055+0000","last_became_peered":"2024-04-26T23:02:48.431055+0000","last_unstale":"2024-04-26T23:02:49.611933+0000","last_undegraded":"2024-04-26T23:02:49.611933+0000","last_fullsized":"2024-04-26T23:02:49.611933+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:24:14.649118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177681+0000","last_change":"2024-04-26T23:02:47.427135+0000","last_active":"2024-04-26T23:02:50.177681+0000","last_peered":"2024-04-26T23:02:50.177681+0000","last_clean":"2024-04-26T23:02:50.177681+0000","last_became_active":"2024-04-26T23:02:47.425928+0000","last_became_peered":"2024-04-26T23:02:47.425928+0000","last_unstale":"2024-04-26T23:02:50.177681+0000","last_undegraded":"2024-04-26T23:02:50.177681+0000","last_fullsized":"2024-04-26T23:02:50.177681+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:49:23.207905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.424301+0000","last_change":"2024-04-26T23:02:48.431562+0000","last_active":"2024-04-26T23:02:49.424301+0000","last_peered":"2024-04-26T23:02:49.424301+0000","last_clean":"2024-04-26T23:02:49.424301+0000","last_became_active":"2024-04-26T23:02:48.431246+0000","last_became_peered":"2024-04-26T23:02:48.431246+0000","last_unstale":"2024-04-26T23:02:49.424301+0000","last_undegraded":"2024-04-26T23:02:49.424301+0000","last_fullsized":"2024-04-26T23:02:49.424301+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:41.139886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.611474+0000","last_change":"2024-04-26T23:02:47.423448+0000","last_active":"2024-04-26T23:02:49.611474+0000","last_peered":"2024-04-26T23:02:49.611474+0000","last_clean":"2024-04-26T23:02:49.611474+0000","last_became_active":"2024-04-26T23:02:47.423145+0000","last_became_peered":"2024-04-26T23:02:47.423145+0000","last_unstale":"2024-04-26T23:02:49.611474+0000","last_undegraded":"2024-04-26T23:02:49.611474+0000","last_fullsized":"2024-04-26T23:02:49.611474+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:42:43.072612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.061192+0000","last_change":"2024-04-26T23:02:48.415579+0000","last_active":"2024-04-26T23:02:50.061192+0000","last_peered":"2024-04-26T23:02:50.061192+0000","last_clean":"2024-04-26T23:02:50.061192+0000","last_became_active":"2024-04-26T23:02:48.415263+0000","last_became_peered":"2024-04-26T23:02:48.415263+0000","last_unstale":"2024-04-26T23:02:50.061192+0000","last_undegraded":"2024-04-26T23:02:50.061192+0000","last_fullsized":"2024-04-26T23:02:50.061192+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:08:40.788946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177917+0000","last_change":"2024-04-26T23:02:47.429506+0000","last_active":"2024-04-26T23:02:50.177917+0000","last_peered":"2024-04-26T23:02:50.177917+0000","last_clean":"2024-04-26T23:02:50.177917+0000","last_became_active":"2024-04-26T23:02:47.429131+0000","last_became_peered":"2024-04-26T23:02:47.429131+0000","last_unstale":"2024-04-26T23:02:50.177917+0000","last_undegraded":"2024-04-26T23:02:50.177917+0000","last_fullsized":"2024-04-26T23:02:50.177917+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:56:06.218292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177341+0000","last_change":"2024-04-26T23:02:48.441472+0000","last_active":"2024-04-26T23:02:50.177341+0000","last_peered":"2024-04-26T23:02:50.177341+0000","last_clean":"2024-04-26T23:02:50.177341+0000","last_became_active":"2024-04-26T23:02:48.437087+0000","last_became_peered":"2024-04-26T23:02:48.437087+0000","last_unstale":"2024-04-26T23:02:50.177341+0000","last_undegraded":"2024-04-26T23:02:50.177341+0000","last_fullsized":"2024-04-26T23:02:50.177341+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:24:14.649118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.430823+0000","last_change":"2024-04-26T23:02:47.428296+0000","last_active":"2024-04-26T23:02:49.430823+0000","last_peered":"2024-04-26T23:02:49.430823+0000","last_clean":"2024-04-26T23:02:49.430823+0000","last_became_active":"2024-04-26T23:02:47.427980+0000","last_became_peered":"2024-04-26T23:02:47.427980+0000","last_unstale":"2024-04-26T23:02:49.430823+0000","last_undegraded":"2024-04-26T23:02:49.430823+0000","last_fullsized":"2024-04-26T23:02:49.430823+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:13.597822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.611508+0000","last_change":"2024-04-26T23:02:48.435107+0000","last_active":"2024-04-26T23:02:49.611508+0000","last_peered":"2024-04-26T23:02:49.611508+0000","last_clean":"2024-04-26T23:02:49.611508+0000","last_became_active":"2024-04-26T23:02:48.434403+0000","last_became_peered":"2024-04-26T23:02:48.434403+0000","last_unstale":"2024-04-26T23:02:49.611508+0000","last_undegraded":"2024-04-26T23:02:49.611508+0000","last_fullsized":"2024-04-26T23:02:49.611508+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:11.262736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.176567+0000","last_change":"2024-04-26T23:02:47.422927+0000","last_active":"2024-04-26T23:02:50.176567+0000","last_peered":"2024-04-26T23:02:50.176567+0000","last_clean":"2024-04-26T23:02:50.176567+0000","last_became_active":"2024-04-26T23:02:47.422616+0000","last_became_peered":"2024-04-26T23:02:47.422616+0000","last_unstale":"2024-04-26T23:02:50.176567+0000","last_undegraded":"2024-04-26T23:02:50.176567+0000","last_fullsized":"2024-04-26T23:02:50.176567+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:16:16.728950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.425115+0000","last_change":"2024-04-26T23:02:48.442214+0000","last_active":"2024-04-26T23:02:49.425115+0000","last_peered":"2024-04-26T23:02:49.425115+0000","last_clean":"2024-04-26T23:02:49.425115+0000","last_became_active":"2024-04-26T23:02:48.441897+0000","last_became_peered":"2024-04-26T23:02:48.441897+0000","last_unstale":"2024-04-26T23:02:49.425115+0000","last_undegraded":"2024-04-26T23:02:49.425115+0000","last_fullsized":"2024-04-26T23:02:49.425115+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:37:14.150665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.610549+0000","last_change":"2024-04-26T23:02:47.426665+0000","last_active":"2024-04-26T23:02:49.610549+0000","last_peered":"2024-04-26T23:02:49.610549+0000","last_clean":"2024-04-26T23:02:49.610549+0000","last_became_active":"2024-04-26T23:02:47.426408+0000","last_became_peered":"2024-04-26T23:02:47.426408+0000","last_unstale":"2024-04-26T23:02:49.610549+0000","last_undegraded":"2024-04-26T23:02:49.610549+0000","last_fullsized":"2024-04-26T23:02:49.610549+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:16:16.728950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178704+0000","last_change":"2024-04-26T23:02:48.415074+0000","last_active":"2024-04-26T23:02:50.178704+0000","last_peered":"2024-04-26T23:02:50.178704+0000","last_clean":"2024-04-26T23:02:50.178704+0000","last_became_active":"2024-04-26T23:02:48.414773+0000","last_became_peered":"2024-04-26T23:02:48.414773+0000","last_unstale":"2024-04-26T23:02:50.178704+0000","last_undegraded":"2024-04-26T23:02:50.178704+0000","last_fullsized":"2024-04-26T23:02:50.178704+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:41.920015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.424426+0000","last_change":"2024-04-26T23:02:47.429805+0000","last_active":"2024-04-26T23:02:49.424426+0000","last_peered":"2024-04-26T23:02:49.424426+0000","last_clean":"2024-04-26T23:02:49.424426+0000","last_became_active":"2024-04-26T23:02:47.429541+0000","last_became_peered":"2024-04-26T23:02:47.429541+0000","last_unstale":"2024-04-26T23:02:49.424426+0000","last_undegraded":"2024-04-26T23:02:49.424426+0000","last_fullsized":"2024-04-26T23:02:49.424426+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:04:05.633056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.429288+0000","last_change":"2024-04-26T23:02:48.447632+0000","last_active":"2024-04-26T23:02:49.429288+0000","last_peered":"2024-04-26T23:02:49.429288+0000","last_clean":"2024-04-26T23:02:49.429288+0000","last_became_active":"2024-04-26T23:02:48.446953+0000","last_became_peered":"2024-04-26T23:02:48.446953+0000","last_unstale":"2024-04-26T23:02:49.429288+0000","last_undegraded":"2024-04-26T23:02:49.429288+0000","last_fullsized":"2024-04-26T23:02:49.429288+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:03:38.777288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177999+0000","last_change":"2024-04-26T23:02:47.424076+0000","last_active":"2024-04-26T23:02:50.177999+0000","last_peered":"2024-04-26T23:02:50.177999+0000","last_clean":"2024-04-26T23:02:50.177999+0000","last_became_active":"2024-04-26T23:02:47.420944+0000","last_became_peered":"2024-04-26T23:02:47.420944+0000","last_unstale":"2024-04-26T23:02:50.177999+0000","last_undegraded":"2024-04-26T23:02:50.177999+0000","last_fullsized":"2024-04-26T23:02:50.177999+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:42:43.072612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.062307+0000","last_change":"2024-04-26T23:02:48.441294+0000","last_active":"2024-04-26T23:02:50.062307+0000","last_peered":"2024-04-26T23:02:50.062307+0000","last_clean":"2024-04-26T23:02:50.062307+0000","last_became_active":"2024-04-26T23:02:48.440989+0000","last_became_peered":"2024-04-26T23:02:48.440989+0000","last_unstale":"2024-04-26T23:02:50.062307+0000","last_undegraded":"2024-04-26T23:02:50.062307+0000","last_fullsized":"2024-04-26T23:02:50.062307+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:46.899581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178160+0000","last_change":"2024-04-26T23:02:47.426825+0000","last_active":"2024-04-26T23:02:50.178160+0000","last_peered":"2024-04-26T23:02:50.178160+0000","last_clean":"2024-04-26T23:02:50.178160+0000","last_became_active":"2024-04-26T23:02:47.426475+0000","last_became_peered":"2024-04-26T23:02:47.426475+0000","last_unstale":"2024-04-26T23:02:50.178160+0000","last_undegraded":"2024-04-26T23:02:50.178160+0000","last_fullsized":"2024-04-26T23:02:50.178160+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:16:16.728950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.611939+0000","last_change":"2024-04-26T23:02:48.417610+0000","last_active":"2024-04-26T23:02:49.611939+0000","last_peered":"2024-04-26T23:02:49.611939+0000","last_clean":"2024-04-26T23:02:49.611939+0000","last_became_active":"2024-04-26T23:02:48.415255+0000","last_became_peered":"2024-04-26T23:02:48.415255+0000","last_unstale":"2024-04-26T23:02:49.611939+0000","last_undegraded":"2024-04-26T23:02:49.611939+0000","last_fullsized":"2024-04-26T23:02:49.611939+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:47:29.052371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.424337+0000","last_change":"2024-04-26T23:02:47.425906+0000","last_active":"2024-04-26T23:02:49.424337+0000","last_peered":"2024-04-26T23:02:49.424337+0000","last_clean":"2024-04-26T23:02:49.424337+0000","last_became_active":"2024-04-26T23:02:47.425612+0000","last_became_peered":"2024-04-26T23:02:47.425612+0000","last_unstale":"2024-04-26T23:02:49.424337+0000","last_undegraded":"2024-04-26T23:02:49.424337+0000","last_fullsized":"2024-04-26T23:02:49.424337+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:33:32.783100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.061200+0000","last_change":"2024-04-26T23:02:48.425723+0000","last_active":"2024-04-26T23:02:50.061200+0000","last_peered":"2024-04-26T23:02:50.061200+0000","last_clean":"2024-04-26T23:02:50.061200+0000","last_became_active":"2024-04-26T23:02:48.425457+0000","last_became_peered":"2024-04-26T23:02:48.425457+0000","last_unstale":"2024-04-26T23:02:50.061200+0000","last_undegraded":"2024-04-26T23:02:50.061200+0000","last_fullsized":"2024-04-26T23:02:50.061200+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:33:33.799026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.424497+0000","last_change":"2024-04-26T23:02:47.429056+0000","last_active":"2024-04-26T23:02:49.424497+0000","last_peered":"2024-04-26T23:02:49.424497+0000","last_clean":"2024-04-26T23:02:49.424497+0000","last_became_active":"2024-04-26T23:02:47.428144+0000","last_became_peered":"2024-04-26T23:02:47.428144+0000","last_unstale":"2024-04-26T23:02:49.424497+0000","last_undegraded":"2024-04-26T23:02:49.424497+0000","last_fullsized":"2024-04-26T23:02:49.424497+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:08:39.773021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.423401+0000","last_change":"2024-04-26T23:02:48.417787+0000","last_active":"2024-04-26T23:02:49.423401+0000","last_peered":"2024-04-26T23:02:49.423401+0000","last_clean":"2024-04-26T23:02:49.423401+0000","last_became_active":"2024-04-26T23:02:48.417517+0000","last_became_peered":"2024-04-26T23:02:48.417517+0000","last_unstale":"2024-04-26T23:02:49.423401+0000","last_undegraded":"2024-04-26T23:02:49.423401+0000","last_fullsized":"2024-04-26T23:02:49.423401+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:35.782775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.176519+0000","last_change":"2024-04-26T23:02:47.426625+0000","last_active":"2024-04-26T23:02:50.176519+0000","last_peered":"2024-04-26T23:02:50.176519+0000","last_clean":"2024-04-26T23:02:50.176519+0000","last_became_active":"2024-04-26T23:02:47.426351+0000","last_became_peered":"2024-04-26T23:02:47.426351+0000","last_unstale":"2024-04-26T23:02:50.176519+0000","last_undegraded":"2024-04-26T23:02:50.176519+0000","last_fullsized":"2024-04-26T23:02:50.176519+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:42:43.072612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.430628+0000","last_change":"2024-04-26T23:02:48.440829+0000","last_active":"2024-04-26T23:02:49.430628+0000","last_peered":"2024-04-26T23:02:49.430628+0000","last_clean":"2024-04-26T23:02:49.430628+0000","last_became_active":"2024-04-26T23:02:48.440401+0000","last_became_peered":"2024-04-26T23:02:48.440401+0000","last_unstale":"2024-04-26T23:02:49.430628+0000","last_undegraded":"2024-04-26T23:02:49.430628+0000","last_fullsized":"2024-04-26T23:02:49.430628+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:46.899581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.427061+0000","last_change":"2024-04-26T23:02:47.418555+0000","last_active":"2024-04-26T23:02:49.427061+0000","last_peered":"2024-04-26T23:02:49.427061+0000","last_clean":"2024-04-26T23:02:49.427061+0000","last_became_active":"2024-04-26T23:02:47.418241+0000","last_became_peered":"2024-04-26T23:02:47.418241+0000","last_unstale":"2024-04-26T23:02:49.427061+0000","last_undegraded":"2024-04-26T23:02:49.427061+0000","last_fullsized":"2024-04-26T23:02:49.427061+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:23.080779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.423972+0000","last_change":"2024-04-26T23:02:48.419482+0000","last_active":"2024-04-26T23:02:49.423972+0000","last_peered":"2024-04-26T23:02:49.423972+0000","last_clean":"2024-04-26T23:02:49.423972+0000","last_became_active":"2024-04-26T23:02:48.419183+0000","last_became_peered":"2024-04-26T23:02:48.419183+0000","last_unstale":"2024-04-26T23:02:49.423972+0000","last_undegraded":"2024-04-26T23:02:49.423972+0000","last_fullsized":"2024-04-26T23:02:49.423972+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:38:01.616813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.430430+0000","last_change":"2024-04-26T23:02:47.426236+0000","last_active":"2024-04-26T23:02:49.430430+0000","last_peered":"2024-04-26T23:02:49.430430+0000","last_clean":"2024-04-26T23:02:49.430430+0000","last_became_active":"2024-04-26T23:02:47.425508+0000","last_became_peered":"2024-04-26T23:02:47.425508+0000","last_unstale":"2024-04-26T23:02:49.430430+0000","last_undegraded":"2024-04-26T23:02:49.430430+0000","last_fullsized":"2024-04-26T23:02:49.430430+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:00:40.904090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.424193+0000","last_change":"2024-04-26T23:02:48.434102+0000","last_active":"2024-04-26T23:02:49.424193+0000","last_peered":"2024-04-26T23:02:49.424193+0000","last_clean":"2024-04-26T23:02:49.424193+0000","last_became_active":"2024-04-26T23:02:48.433834+0000","last_became_peered":"2024-04-26T23:02:48.433834+0000","last_unstale":"2024-04-26T23:02:49.424193+0000","last_undegraded":"2024-04-26T23:02:49.424193+0000","last_fullsized":"2024-04-26T23:02:49.424193+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:09:16.546571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178283+0000","last_change":"2024-04-26T23:02:47.422531+0000","last_active":"2024-04-26T23:02:50.178283+0000","last_peered":"2024-04-26T23:02:50.178283+0000","last_clean":"2024-04-26T23:02:50.178283+0000","last_became_active":"2024-04-26T23:02:47.422263+0000","last_became_peered":"2024-04-26T23:02:47.422263+0000","last_unstale":"2024-04-26T23:02:50.178283+0000","last_undegraded":"2024-04-26T23:02:50.178283+0000","last_fullsized":"2024-04-26T23:02:50.178283+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:00:19.183776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178134+0000","last_change":"2024-04-26T23:02:48.437858+0000","last_active":"2024-04-26T23:02:50.178134+0000","last_peered":"2024-04-26T23:02:50.178134+0000","last_clean":"2024-04-26T23:02:50.178134+0000","last_became_active":"2024-04-26T23:02:48.437054+0000","last_became_peered":"2024-04-26T23:02:48.437054+0000","last_unstale":"2024-04-26T23:02:50.178134+0000","last_undegraded":"2024-04-26T23:02:50.178134+0000","last_fullsized":"2024-04-26T23:02:50.178134+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:38:01.616813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.611977+0000","last_change":"2024-04-26T23:02:47.425145+0000","last_active":"2024-04-26T23:02:49.611977+0000","last_peered":"2024-04-26T23:02:49.611977+0000","last_clean":"2024-04-26T23:02:49.611977+0000","last_became_active":"2024-04-26T23:02:47.424860+0000","last_became_peered":"2024-04-26T23:02:47.424860+0000","last_unstale":"2024-04-26T23:02:49.611977+0000","last_undegraded":"2024-04-26T23:02:49.611977+0000","last_fullsized":"2024-04-26T23:02:49.611977+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:49:23.207905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.611778+0000","last_change":"2024-04-26T23:02:48.424133+0000","last_active":"2024-04-26T23:02:49.611778+0000","last_peered":"2024-04-26T23:02:49.611778+0000","last_clean":"2024-04-26T23:02:49.611778+0000","last_became_active":"2024-04-26T23:02:48.423757+0000","last_became_peered":"2024-04-26T23:02:48.423757+0000","last_unstale":"2024-04-26T23:02:49.611778+0000","last_undegraded":"2024-04-26T23:02:49.611778+0000","last_fullsized":"2024-04-26T23:02:49.611778+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:46.899581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178673+0000","last_change":"2024-04-26T23:02:47.427171+0000","last_active":"2024-04-26T23:02:50.178673+0000","last_peered":"2024-04-26T23:02:50.178673+0000","last_clean":"2024-04-26T23:02:50.178673+0000","last_became_active":"2024-04-26T23:02:47.426664+0000","last_became_peered":"2024-04-26T23:02:47.426664+0000","last_unstale":"2024-04-26T23:02:50.178673+0000","last_undegraded":"2024-04-26T23:02:50.178673+0000","last_fullsized":"2024-04-26T23:02:50.178673+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:06:28.137216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178954+0000","last_change":"2024-04-26T23:02:48.422054+0000","last_active":"2024-04-26T23:02:50.178954+0000","last_peered":"2024-04-26T23:02:50.178954+0000","last_clean":"2024-04-26T23:02:50.178954+0000","last_became_active":"2024-04-26T23:02:48.421710+0000","last_became_peered":"2024-04-26T23:02:48.421710+0000","last_unstale":"2024-04-26T23:02:50.178954+0000","last_undegraded":"2024-04-26T23:02:50.178954+0000","last_fullsized":"2024-04-26T23:02:50.178954+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:33.269612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177323+0000","last_change":"2024-04-26T23:02:47.425522+0000","last_active":"2024-04-26T23:02:50.177323+0000","last_peered":"2024-04-26T23:02:50.177323+0000","last_clean":"2024-04-26T23:02:50.177323+0000","last_became_active":"2024-04-26T23:02:47.425249+0000","last_became_peered":"2024-04-26T23:02:47.425249+0000","last_unstale":"2024-04-26T23:02:50.177323+0000","last_undegraded":"2024-04-26T23:02:50.177323+0000","last_fullsized":"2024-04-26T23:02:50.177323+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:00:19.183776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.430956+0000","last_change":"2024-04-26T23:02:48.443801+0000","last_active":"2024-04-26T23:02:49.430956+0000","last_peered":"2024-04-26T23:02:49.430956+0000","last_clean":"2024-04-26T23:02:49.430956+0000","last_became_active":"2024-04-26T23:02:48.443337+0000","last_became_peered":"2024-04-26T23:02:48.443337+0000","last_unstale":"2024-04-26T23:02:49.430956+0000","last_undegraded":"2024-04-26T23:02:49.430956+0000","last_fullsized":"2024-04-26T23:02:49.430956+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:24:14.649118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178147+0000","last_change":"2024-04-26T23:02:47.414839+0000","last_active":"2024-04-26T23:02:50.178147+0000","last_peered":"2024-04-26T23:02:50.178147+0000","last_clean":"2024-04-26T23:02:50.178147+0000","last_became_active":"2024-04-26T23:02:47.414531+0000","last_became_peered":"2024-04-26T23:02:47.414531+0000","last_unstale":"2024-04-26T23:02:50.178147+0000","last_undegraded":"2024-04-26T23:02:50.178147+0000","last_fullsized":"2024-04-26T23:02:50.178147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:14.373025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.061286+0000","last_change":"2024-04-26T23:02:48.439667+0000","last_active":"2024-04-26T23:02:50.061286+0000","last_peered":"2024-04-26T23:02:50.061286+0000","last_clean":"2024-04-26T23:02:50.061286+0000","last_became_active":"2024-04-26T23:02:48.439378+0000","last_became_peered":"2024-04-26T23:02:48.439378+0000","last_unstale":"2024-04-26T23:02:50.061286+0000","last_undegraded":"2024-04-26T23:02:50.061286+0000","last_fullsized":"2024-04-26T23:02:50.061286+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:03:38.777288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178572+0000","last_change":"2024-04-26T23:02:47.417644+0000","last_active":"2024-04-26T23:02:50.178572+0000","last_peered":"2024-04-26T23:02:50.178572+0000","last_clean":"2024-04-26T23:02:50.178572+0000","last_became_active":"2024-04-26T23:02:47.417390+0000","last_became_peered":"2024-04-26T23:02:47.417390+0000","last_unstale":"2024-04-26T23:02:50.178572+0000","last_undegraded":"2024-04-26T23:02:50.178572+0000","last_fullsized":"2024-04-26T23:02:50.178572+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:58:52.055775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177668+0000","last_change":"2024-04-26T23:02:48.444525+0000","last_active":"2024-04-26T23:02:50.177668+0000","last_peered":"2024-04-26T23:02:50.177668+0000","last_clean":"2024-04-26T23:02:50.177668+0000","last_became_active":"2024-04-26T23:02:48.443761+0000","last_became_peered":"2024-04-26T23:02:48.443761+0000","last_unstale":"2024-04-26T23:02:50.177668+0000","last_undegraded":"2024-04-26T23:02:50.177668+0000","last_fullsized":"2024-04-26T23:02:50.177668+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:35.782775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.430232+0000","last_change":"2024-04-26T23:02:47.425458+0000","last_active":"2024-04-26T23:02:49.430232+0000","last_peered":"2024-04-26T23:02:49.430232+0000","last_clean":"2024-04-26T23:02:49.430232+0000","last_became_active":"2024-04-26T23:02:47.424961+0000","last_became_peered":"2024-04-26T23:02:47.424961+0000","last_unstale":"2024-04-26T23:02:49.430232+0000","last_undegraded":"2024-04-26T23:02:49.430232+0000","last_fullsized":"2024-04-26T23:02:49.430232+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:28.822626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.424345+0000","last_change":"2024-04-26T23:02:48.433362+0000","last_active":"2024-04-26T23:02:49.424345+0000","last_peered":"2024-04-26T23:02:49.424345+0000","last_clean":"2024-04-26T23:02:49.424345+0000","last_became_active":"2024-04-26T23:02:48.433063+0000","last_became_peered":"2024-04-26T23:02:48.433063+0000","last_unstale":"2024-04-26T23:02:49.424345+0000","last_undegraded":"2024-04-26T23:02:49.424345+0000","last_fullsized":"2024-04-26T23:02:49.424345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:35:56.159564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177482+0000","last_change":"2024-04-26T23:02:47.422918+0000","last_active":"2024-04-26T23:02:50.177482+0000","last_peered":"2024-04-26T23:02:50.177482+0000","last_clean":"2024-04-26T23:02:50.177482+0000","last_became_active":"2024-04-26T23:02:47.422578+0000","last_became_peered":"2024-04-26T23:02:47.422578+0000","last_unstale":"2024-04-26T23:02:50.177482+0000","last_undegraded":"2024-04-26T23:02:50.177482+0000","last_fullsized":"2024-04-26T23:02:50.177482+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:45:54.221462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.177345+0000","last_change":"2024-04-26T23:02:48.443424+0000","last_active":"2024-04-26T23:02:50.177345+0000","last_peered":"2024-04-26T23:02:50.177345+0000","last_clean":"2024-04-26T23:02:50.177345+0000","last_became_active":"2024-04-26T23:02:48.443104+0000","last_became_peered":"2024-04-26T23:02:48.443104+0000","last_unstale":"2024-04-26T23:02:50.177345+0000","last_undegraded":"2024-04-26T23:02:50.177345+0000","last_fullsized":"2024-04-26T23:02:50.177345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:29.486859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178555+0000","last_change":"2024-04-26T23:02:47.427276+0000","last_active":"2024-04-26T23:02:50.178555+0000","last_peered":"2024-04-26T23:02:50.178555+0000","last_clean":"2024-04-26T23:02:50.178555+0000","last_became_active":"2024-04-26T23:02:47.426167+0000","last_became_peered":"2024-04-26T23:02:47.426167+0000","last_unstale":"2024-04-26T23:02:50.178555+0000","last_undegraded":"2024-04-26T23:02:50.178555+0000","last_fullsized":"2024-04-26T23:02:50.178555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:49:23.207905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.429730+0000","last_change":"2024-04-26T23:02:48.447911+0000","last_active":"2024-04-26T23:02:49.429730+0000","last_peered":"2024-04-26T23:02:49.429730+0000","last_clean":"2024-04-26T23:02:49.429730+0000","last_became_active":"2024-04-26T23:02:48.447345+0000","last_became_peered":"2024-04-26T23:02:48.447345+0000","last_unstale":"2024-04-26T23:02:49.429730+0000","last_undegraded":"2024-04-26T23:02:49.429730+0000","last_fullsized":"2024-04-26T23:02:49.429730+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:29:39.493442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178628+0000","last_change":"2024-04-26T23:02:47.415885+0000","last_active":"2024-04-26T23:02:50.178628+0000","last_peered":"2024-04-26T23:02:50.178628+0000","last_clean":"2024-04-26T23:02:50.178628+0000","last_became_active":"2024-04-26T23:02:47.415613+0000","last_became_peered":"2024-04-26T23:02:47.415613+0000","last_unstale":"2024-04-26T23:02:50.178628+0000","last_undegraded":"2024-04-26T23:02:50.178628+0000","last_fullsized":"2024-04-26T23:02:50.178628+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:16:53.767458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.424919+0000","last_change":"2024-04-26T23:02:48.430505+0000","last_active":"2024-04-26T23:02:49.424919+0000","last_peered":"2024-04-26T23:02:49.424919+0000","last_clean":"2024-04-26T23:02:49.424919+0000","last_became_active":"2024-04-26T23:02:48.430261+0000","last_became_peered":"2024-04-26T23:02:48.430261+0000","last_unstale":"2024-04-26T23:02:49.424919+0000","last_undegraded":"2024-04-26T23:02:49.424919+0000","last_fullsized":"2024-04-26T23:02:49.424919+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:17:38.620831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.061563+0000","last_change":"2024-04-26T23:02:47.423354+0000","last_active":"2024-04-26T23:02:50.061563+0000","last_peered":"2024-04-26T23:02:50.061563+0000","last_clean":"2024-04-26T23:02:50.061563+0000","last_became_active":"2024-04-26T23:02:47.423058+0000","last_became_peered":"2024-04-26T23:02:47.423058+0000","last_unstale":"2024-04-26T23:02:50.061563+0000","last_undegraded":"2024-04-26T23:02:50.061563+0000","last_fullsized":"2024-04-26T23:02:50.061563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:16:16.728950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.061637+0000","last_change":"2024-04-26T23:02:48.421639+0000","last_active":"2024-04-26T23:02:50.061637+0000","last_peered":"2024-04-26T23:02:50.061637+0000","last_clean":"2024-04-26T23:02:50.061637+0000","last_became_active":"2024-04-26T23:02:48.421339+0000","last_became_peered":"2024-04-26T23:02:48.421339+0000","last_unstale":"2024-04-26T23:02:50.061637+0000","last_undegraded":"2024-04-26T23:02:50.061637+0000","last_fullsized":"2024-04-26T23:02:50.061637+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:29.486859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.061426+0000","last_change":"2024-04-26T23:02:47.421824+0000","last_active":"2024-04-26T23:02:50.061426+0000","last_peered":"2024-04-26T23:02:50.061426+0000","last_clean":"2024-04-26T23:02:50.061426+0000","last_became_active":"2024-04-26T23:02:47.421537+0000","last_became_peered":"2024-04-26T23:02:47.421537+0000","last_unstale":"2024-04-26T23:02:50.061426+0000","last_undegraded":"2024-04-26T23:02:50.061426+0000","last_fullsized":"2024-04-26T23:02:50.061426+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.176636+0000","last_change":"2024-04-26T23:02:48.423689+0000","last_active":"2024-04-26T23:02:50.176636+0000","last_peered":"2024-04-26T23:02:50.176636+0000","last_clean":"2024-04-26T23:02:50.176636+0000","last_became_active":"2024-04-26T23:02:48.423395+0000","last_became_peered":"2024-04-26T23:02:48.423395+0000","last_unstale":"2024-04-26T23:02:50.176636+0000","last_undegraded":"2024-04-26T23:02:50.176636+0000","last_fullsized":"2024-04-26T23:02:50.176636+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:33:33.799026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.427828+0000","last_change":"2024-04-26T23:02:47.422919+0000","last_active":"2024-04-26T23:02:49.427828+0000","last_peered":"2024-04-26T23:02:49.427828+0000","last_clean":"2024-04-26T23:02:49.427828+0000","last_became_active":"2024-04-26T23:02:47.421694+0000","last_became_peered":"2024-04-26T23:02:47.421694+0000","last_unstale":"2024-04-26T23:02:49.427828+0000","last_undegraded":"2024-04-26T23:02:49.427828+0000","last_fullsized":"2024-04-26T23:02:49.427828+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:24:26.243323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.610787+0000","last_change":"2024-04-26T23:02:48.422824+0000","last_active":"2024-04-26T23:02:49.610787+0000","last_peered":"2024-04-26T23:02:49.610787+0000","last_clean":"2024-04-26T23:02:49.610787+0000","last_became_active":"2024-04-26T23:02:48.422532+0000","last_became_peered":"2024-04-26T23:02:48.422532+0000","last_unstale":"2024-04-26T23:02:49.610787+0000","last_undegraded":"2024-04-26T23:02:49.610787+0000","last_fullsized":"2024-04-26T23:02:49.610787+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:29:39.493442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.610503+0000","last_change":"2024-04-26T23:02:47.419127+0000","last_active":"2024-04-26T23:02:49.610503+0000","last_peered":"2024-04-26T23:02:49.610503+0000","last_clean":"2024-04-26T23:02:49.610503+0000","last_became_active":"2024-04-26T23:02:47.417694+0000","last_became_peered":"2024-04-26T23:02:47.417694+0000","last_unstale":"2024-04-26T23:02:49.610503+0000","last_undegraded":"2024-04-26T23:02:49.610503+0000","last_fullsized":"2024-04-26T23:02:49.610503+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:00:19.183776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.431435+0000","last_change":"2024-04-26T23:02:48.446963+0000","last_active":"2024-04-26T23:02:49.431435+0000","last_peered":"2024-04-26T23:02:49.431435+0000","last_clean":"2024-04-26T23:02:49.431435+0000","last_became_active":"2024-04-26T23:02:48.446646+0000","last_became_peered":"2024-04-26T23:02:48.446646+0000","last_unstale":"2024-04-26T23:02:49.431435+0000","last_undegraded":"2024-04-26T23:02:49.431435+0000","last_fullsized":"2024-04-26T23:02:49.431435+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:47:29.052371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.423209+0000","last_change":"2024-04-26T23:02:47.416825+0000","last_active":"2024-04-26T23:02:49.423209+0000","last_peered":"2024-04-26T23:02:49.423209+0000","last_clean":"2024-04-26T23:02:49.423209+0000","last_became_active":"2024-04-26T23:02:47.416571+0000","last_became_peered":"2024-04-26T23:02:47.416571+0000","last_unstale":"2024-04-26T23:02:49.423209+0000","last_undegraded":"2024-04-26T23:02:49.423209+0000","last_fullsized":"2024-04-26T23:02:49.423209+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:47:45.095849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:50.178079+0000","last_change":"2024-04-26T23:02:48.443710+0000","last_active":"2024-04-26T23:02:50.178079+0000","last_peered":"2024-04-26T23:02:50.178079+0000","last_clean":"2024-04-26T23:02:50.178079+0000","last_became_active":"2024-04-26T23:02:48.443432+0000","last_became_peered":"2024-04-26T23:02:48.443432+0000","last_unstale":"2024-04-26T23:02:50.178079+0000","last_undegraded":"2024-04-26T23:02:50.178079+0000","last_fullsized":"2024-04-26T23:02:50.178079+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:47.377264+0000","last_clean_scrub_stamp":"2024-04-26T23:02:47.377264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:41.139886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.429651+0000","last_change":"2024-04-26T23:02:47.432253+0000","last_active":"2024-04-26T23:02:49.429651+0000","last_peered":"2024-04-26T23:02:49.429651+0000","last_clean":"2024-04-26T23:02:49.429651+0000","last_became_active":"2024-04-26T23:02:47.431955+0000","last_became_peered":"2024-04-26T23:02:47.431955+0000","last_unstale":"2024-04-26T23:02:49.429651+0000","last_undegraded":"2024-04-26T23:02:49.429651+0000","last_fullsized":"2024-04-26T23:02:49.429651+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:02:46.361339+0000","last_clean_scrub_stamp":"2024-04-26T23:02:46.361339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:32.253687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":207,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-26T23:02:49.431082+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:02:49.431082+0000","last_peered":"2024-04-26T23:02:49.431082+0000","last_clean":"2024-04-26T23:02:49.431082+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:02:49.431082+0000","last_undegraded":"2024-04-26T23:02:49.431082+0000","last_fullsized":"2024-04-26T23:02:49.431082+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":789207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":197269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":458752,"data_stored":197761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":466944,"data_stored":198803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":197227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":197247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1048576,"data_stored":787559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:02:58.522 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T23:02:58.522 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-26T23:02:58.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.777+0000 7fb701800640 1 Processor -- start 2024-04-26T23:02:58.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.777+0000 7fb701800640 1 -- start start 2024-04-26T23:02:58.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.778+0000 7fb701800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb6fc0d1a00 0x7fb6fc0d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:58.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.778+0000 7fb701800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb6fc0b40d0 0x7fb6fc0b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:58.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.778+0000 7fb701800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6fc0d0a10 con 0x7fb6fc0d2320 2024-04-26T23:02:58.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.779+0000 7fb701800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb6fc0d0cf0 con 0x7fb6fc0b40d0 2024-04-26T23:02:58.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.779+0000 7fb701800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb6fc0d0fd0 con 0x7fb6fc0d1a00 2024-04-26T23:02:58.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.779+0000 7fb6fb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb6fc0b40d0 0x7fb6fc0b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:58.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.779+0000 7fb6fa000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb6fc0d2320 0x7fb6fc1625f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42884/0 (socket says 172.21.15.40:42884) 2024-04-26T23:02:58.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.779+0000 7fb6fa000640 1 -- 172.21.15.40:0/213781262 learned_addr learned my addr 172.21.15.40:0/213781262 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:02:58.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.779+0000 7fb6faa00640 1 --2- 172.21.15.40:0/213781262 >> v2:172.21.15.103:3300/0 conn(0x7fb6fc0d1a00 0x7fb6fc0d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:58.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.779+0000 7fb6f9600640 1 -- 172.21.15.40:0/213781262 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1727398495 0 0) 0x7fb6fc0d0a10 con 0x7fb6fc0d2320 2024-04-26T23:02:58.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.779+0000 7fb6f9600640 1 -- 172.21.15.40:0/213781262 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6e0003520 con 0x7fb6fc0d2320 2024-04-26T23:02:58.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.780+0000 7fb6fb400640 1 -- 172.21.15.40:0/213781262 >> v2:172.21.15.103:3300/0 conn(0x7fb6fc0d1a00 msgr2=0x7fb6fc0d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:58.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.780+0000 7fb6fb400640 1 --2- 172.21.15.40:0/213781262 >> v2:172.21.15.103:3300/0 conn(0x7fb6fc0d1a00 0x7fb6fc0d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.780+0000 7fb6fb400640 1 -- 172.21.15.40:0/213781262 >> v1:172.21.15.40:6789/0 conn(0x7fb6fc0d2320 legacy=0x7fb6fc1625f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:58.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.780+0000 7fb6fb400640 1 -- 172.21.15.40:0/213781262 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6fc0d40a0 con 0x7fb6fc0b40d0 2024-04-26T23:02:58.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.781+0000 7fb6fb400640 1 --2- 172.21.15.40:0/213781262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6fc0b40d0 0x7fb6fc0b4b00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc06c650 tx=0x7fb6f402a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6248e66536f9f541 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:58.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.781+0000 7fb6f9600640 1 -- 172.21.15.40:0/213781262 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4007740 con 0x7fb6fc0b40d0 2024-04-26T23:02:58.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.781+0000 7fb6f9600640 1 -- 172.21.15.40:0/213781262 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6f40078e0 con 0x7fb6fc0b40d0 2024-04-26T23:02:58.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.781+0000 7fb6f9600640 1 -- 172.21.15.40:0/213781262 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4007cb0 con 0x7fb6fc0b40d0 2024-04-26T23:02:58.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.782+0000 7fb701800640 1 -- 172.21.15.40:0/213781262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6fc0b40d0 msgr2=0x7fb6fc0b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:58.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.782+0000 7fb701800640 1 --2- 172.21.15.40:0/213781262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6fc0b40d0 0x7fb6fc0b4b00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc06c650 tx=0x7fb6f402a550 comp rx=0 tx=0).stop 2024-04-26T23:02:58.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.782+0000 7fb701800640 1 -- 172.21.15.40:0/213781262 shutdown_connections 2024-04-26T23:02:58.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.782+0000 7fb701800640 1 --2- 172.21.15.40:0/213781262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6fc0b40d0 0x7fb6fc0b4b00 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.782+0000 7fb701800640 1 --2- 172.21.15.40:0/213781262 >> v2:172.21.15.103:3300/0 conn(0x7fb6fc0d1a00 0x7fb6fc0d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.782+0000 7fb701800640 1 -- 172.21.15.40:0/213781262 >> 172.21.15.40:0/213781262 conn(0x7fb6fc0d4fd0 msgr2=0x7fb6fc0d53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:58.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.782+0000 7fb701800640 1 -- 172.21.15.40:0/213781262 shutdown_connections 2024-04-26T23:02:58.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.782+0000 7fb701800640 1 -- 172.21.15.40:0/213781262 wait complete. 2024-04-26T23:02:58.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.784+0000 7fb701800640 1 Processor -- start 2024-04-26T23:02:58.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.784+0000 7fb701800640 1 -- start start 2024-04-26T23:02:58.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.785+0000 7fb701800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6fc0b40d0 0x7fb6fc15e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:58.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.785+0000 7fb701800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6fc0d1a00 0x7fb6fc0ba4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:58.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.785+0000 7fb701800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6fc0d2320 0x7fb6fc0c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:58.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.785+0000 7fb701800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc0b5040 con 0x7fb6fc0d1a00 2024-04-26T23:02:58.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.785+0000 7fb701800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc162f50 con 0x7fb6fc0b40d0 2024-04-26T23:02:58.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.785+0000 7fb701800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6fc0b51c0 con 0x7fb6fc0d2320 2024-04-26T23:02:58.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.785+0000 7fb6fa000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6fc0d1a00 0x7fb6fc0ba4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:58.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.785+0000 7fb6fa000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6fc0d1a00 0x7fb6fc0ba4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36742/0 (socket says 172.21.15.40:36742) 2024-04-26T23:02:58.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.785+0000 7fb6fa000640 1 -- 172.21.15.40:0/1282403137 learned_addr learned my addr 172.21.15.40:0/1282403137 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:58.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.786+0000 7fb6faa00640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6fc0b40d0 0x7fb6fc15e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:58.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.786+0000 7fb6fa000640 1 -- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6fc0d2320 msgr2=0x7fb6fc0c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:58.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.786+0000 7fb6fa000640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6fc0d2320 0x7fb6fc0c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.786+0000 7fb6fa000640 1 -- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6fc0b40d0 msgr2=0x7fb6fc15e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:58.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.786+0000 7fb6fa000640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6fc0b40d0 0x7fb6fc15e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:58.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.786+0000 7fb6fa000640 1 -- 172.21.15.40:0/1282403137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6fc0b7eb0 con 0x7fb6fc0d1a00 2024-04-26T23:02:58.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.786+0000 7fb6faa00640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6fc0b40d0 0x7fb6fc15e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:58.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.786+0000 7fb6fa000640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6fc0d1a00 0x7fb6fc0ba4c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800c390 tx=0x7fb6e800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:58.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.786+0000 7fb6f2a00640 1 -- 172.21.15.40:0/1282403137 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8017070 con 0x7fb6fc0d1a00 2024-04-26T23:02:58.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.786+0000 7fb6f2a00640 1 -- 172.21.15.40:0/1282403137 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6e8005ce0 con 0x7fb6fc0d1a00 2024-04-26T23:02:58.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.786+0000 7fb701800640 1 -- 172.21.15.40:0/1282403137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6fc0b8190 con 0x7fb6fc0d1a00 2024-04-26T23:02:58.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.787+0000 7fb6f2a00640 1 -- 172.21.15.40:0/1282403137 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e80072f0 con 0x7fb6fc0d1a00 2024-04-26T23:02:58.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.787+0000 7fb701800640 1 -- 172.21.15.40:0/1282403137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6fc0b8750 con 0x7fb6fc0d1a00 2024-04-26T23:02:58.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.788+0000 7fb701800640 1 -- 172.21.15.40:0/1282403137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6fc0694f0 con 0x7fb6fc0d1a00 2024-04-26T23:02:58.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.789+0000 7fb6f2a00640 1 -- 172.21.15.40:0/1282403137 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb6e801a070 con 0x7fb6fc0d1a00 2024-04-26T23:02:58.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.790+0000 7fb6f2a00640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb6dc06c8f0 0x7fb6dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:58.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.790+0000 7fb6f2a00640 1 -- 172.21.15.40:0/1282403137 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fb6e808b1a0 con 0x7fb6fc0d1a00 2024-04-26T23:02:58.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.791+0000 7fb6faa00640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb6dc06c8f0 0x7fb6dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:58.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.792+0000 7fb6faa00640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb6dc06c8f0 0x7fb6dc06edb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec006fd0 tx=0x7fb6ec008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:58.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:58.794+0000 7fb6f2a00640 1 -- 172.21.15.40:0/1282403137 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fb6e80593a0 con 0x7fb6fc0d1a00 2024-04-26T23:02:59.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.074+0000 7fb701800640 1 -- 172.21.15.40:0/1282403137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fb6fc0b4dc0 con 0x7fb6fc0d1a00 2024-04-26T23:02:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.080+0000 7fb6f2a00640 1 -- 172.21.15.40:0/1282403137 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7fb6e8058ca0 con 0x7fb6fc0d1a00 2024-04-26T23:02:59.080 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-04-26T23:02:59.080 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg== 2024-04-26T23:02:59.080 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-04-26T23:02:59.080 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-04-26T23:02:59.081 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-26T23:02:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-04-26T23:02:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.083+0000 7fb701800640 1 -- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb6dc06c8f0 msgr2=0x7fb6dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.083+0000 7fb701800640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb6dc06c8f0 0x7fb6dc06edb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec006fd0 tx=0x7fb6ec008040 comp rx=0 tx=0).stop 2024-04-26T23:02:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.083+0000 7fb701800640 1 -- 172.21.15.40:0/1282403137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6fc0d1a00 msgr2=0x7fb6fc0ba4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.083+0000 7fb701800640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6fc0d1a00 0x7fb6fc0ba4c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800c390 tx=0x7fb6e800c850 comp rx=0 tx=0).stop 2024-04-26T23:02:59.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.083+0000 7fb701800640 1 -- 172.21.15.40:0/1282403137 shutdown_connections 2024-04-26T23:02:59.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.083+0000 7fb701800640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb6dc06c8f0 0x7fb6dc06edb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.083+0000 7fb701800640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6fc0d2320 0x7fb6fc0c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.083+0000 7fb701800640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6fc0d1a00 0x7fb6fc0ba4c0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.083+0000 7fb701800640 1 --2- 172.21.15.40:0/1282403137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6fc0b40d0 0x7fb6fc15e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.083+0000 7fb701800640 1 -- 172.21.15.40:0/1282403137 >> 172.21.15.40:0/1282403137 conn(0x7fb6fc0d4fd0 msgr2=0x7fb6fc06cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:59.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.083+0000 7fb701800640 1 -- 172.21.15.40:0/1282403137 shutdown_connections 2024-04-26T23:02:59.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.083+0000 7fb701800640 1 -- 172.21.15.40:0/1282403137 wait complete. 2024-04-26T23:02:59.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:02:59.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.311+0000 7f5138e00640 1 Processor -- start 2024-04-26T23:02:59.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.312+0000 7f5138e00640 1 -- start start 2024-04-26T23:02:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.313+0000 7f5138e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f513415f380 0x7f51340cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.313+0000 7f5138e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5134073b70 0x7f51340cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.314+0000 7f5138e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5134072c00 con 0x7f51340d0620 2024-04-26T23:02:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.314+0000 7f5138e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f51340729b0 con 0x7f5134073b70 2024-04-26T23:02:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.314+0000 7f5138e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5134072ee0 con 0x7f513415f380 2024-04-26T23:02:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.314+0000 7f5131400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5134073b70 0x7f51340cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.314+0000 7f5131400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5134073b70 0x7f51340cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36754/0 (socket says 172.21.15.40:36754) 2024-04-26T23:02:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.314+0000 7f5131400640 1 -- 172.21.15.40:0/41809674 learned_addr learned my addr 172.21.15.40:0/41809674 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.314+0000 7f5131e00640 1 --2- 172.21.15.40:0/41809674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f513415f380 0x7f51340cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.314+0000 7f5131400640 1 -- 172.21.15.40:0/41809674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f513415f380 msgr2=0x7f51340cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.314+0000 7f5131400640 1 --2- 172.21.15.40:0/41809674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f513415f380 0x7f51340cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.314+0000 7f5131400640 1 -- 172.21.15.40:0/41809674 >> v1:172.21.15.103:6789/0 conn(0x7f51340d0620 legacy=0x7f51340cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:02:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.314+0000 7f5131400640 1 -- 172.21.15.40:0/41809674 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51340731c0 con 0x7f5134073b70 2024-04-26T23:02:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.314+0000 7f5131e00640 1 --2- 172.21.15.40:0/41809674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f513415f380 0x7f51340cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:02:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.316+0000 7f5131400640 1 --2- 172.21.15.40:0/41809674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5134073b70 0x7f51340cb980 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5120009870 tx=0x7f512002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6989f6b79b22ec0b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.316+0000 7f5130a00640 1 -- 172.21.15.40:0/41809674 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120037070 con 0x7f5134073b70 2024-04-26T23:02:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.316+0000 7f5130a00640 1 -- 172.21.15.40:0/41809674 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5120032440 con 0x7f5134073b70 2024-04-26T23:02:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.316+0000 7f5130a00640 1 -- 172.21.15.40:0/41809674 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120032810 con 0x7f5134073b70 2024-04-26T23:02:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.317+0000 7f5138e00640 1 -- 172.21.15.40:0/41809674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5134073b70 msgr2=0x7f51340cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.317+0000 7f5138e00640 1 --2- 172.21.15.40:0/41809674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5134073b70 0x7f51340cb980 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5120009870 tx=0x7f512002a210 comp rx=0 tx=0).stop 2024-04-26T23:02:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.317+0000 7f5138e00640 1 -- 172.21.15.40:0/41809674 shutdown_connections 2024-04-26T23:02:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.317+0000 7f5138e00640 1 --2- 172.21.15.40:0/41809674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5134073b70 0x7f51340cb980 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.317+0000 7f5138e00640 1 --2- 172.21.15.40:0/41809674 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f513415f380 0x7f51340cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.317+0000 7f5138e00640 1 -- 172.21.15.40:0/41809674 >> 172.21.15.40:0/41809674 conn(0x7f51340d0be0 msgr2=0x7f51340d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.317+0000 7f5138e00640 1 -- 172.21.15.40:0/41809674 shutdown_connections 2024-04-26T23:02:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.317+0000 7f5138e00640 1 -- 172.21.15.40:0/41809674 wait complete. 2024-04-26T23:02:59.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.320+0000 7f5138e00640 1 Processor -- start 2024-04-26T23:02:59.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.320+0000 7f5138e00640 1 -- start start 2024-04-26T23:02:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.321+0000 7f5138e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51340d0620 0x7f513414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.321+0000 7f5138e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f513415f380 0x7f51340c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.321+0000 7f5138e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51340ca200 0x7f51340b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.321+0000 7f5138e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5134073620 con 0x7f51340ca200 2024-04-26T23:02:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.321+0000 7f5138e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f51340aff10 con 0x7f513415f380 2024-04-26T23:02:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.321+0000 7f5138e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f51340737a0 con 0x7f51340d0620 2024-04-26T23:02:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.322+0000 7f5132800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51340ca200 0x7f51340b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.322+0000 7f5132800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51340ca200 0x7f51340b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36768/0 (socket says 172.21.15.40:36768) 2024-04-26T23:02:59.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.322+0000 7f5132800640 1 -- 172.21.15.40:0/3040005123 learned_addr learned my addr 172.21.15.40:0/3040005123 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:02:59.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.322+0000 7f5131e00640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51340d0620 0x7f513414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:59.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.322+0000 7f5131400640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f513415f380 0x7f51340c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.322+0000 7f5132800640 1 -- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51340d0620 msgr2=0x7f513414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.322+0000 7f5132800640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51340d0620 0x7f513414eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.322+0000 7f5132800640 1 -- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f513415f380 msgr2=0x7f51340c9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.322+0000 7f5132800640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f513415f380 0x7f51340c9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.322+0000 7f5132800640 1 -- 172.21.15.40:0/3040005123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51340b63b0 con 0x7f51340ca200 2024-04-26T23:02:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.323+0000 7f5132800640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51340ca200 0x7f51340b5640 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f512c00c700 tx=0x7f512c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.323+0000 7f512a000640 1 -- 172.21.15.40:0/3040005123 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c016020 con 0x7f51340ca200 2024-04-26T23:02:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.323+0000 7f512a000640 1 -- 172.21.15.40:0/3040005123 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f512c00aa50 con 0x7f51340ca200 2024-04-26T23:02:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.323+0000 7f5138e00640 1 -- 172.21.15.40:0/3040005123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51340b6690 con 0x7f51340ca200 2024-04-26T23:02:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.323+0000 7f512a000640 1 -- 172.21.15.40:0/3040005123 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c00ae40 con 0x7f51340ca200 2024-04-26T23:02:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.323+0000 7f5138e00640 1 -- 172.21.15.40:0/3040005123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f513414f3e0 con 0x7f51340ca200 2024-04-26T23:02:59.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.325+0000 7f5138e00640 1 -- 172.21.15.40:0/3040005123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51340694f0 con 0x7f51340ca200 2024-04-26T23:02:59.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.325+0000 7f512a000640 1 -- 172.21.15.40:0/3040005123 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f512c019020 con 0x7f51340ca200 2024-04-26T23:02:59.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.326+0000 7f512a000640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f511406c8f0 0x7f511406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:02:59.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.327+0000 7f512a000640 1 -- 172.21.15.40:0/3040005123 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f512c08bfb0 con 0x7f51340ca200 2024-04-26T23:02:59.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.327+0000 7f5131e00640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f511406c8f0 0x7f511406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:02:59.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.328+0000 7f5131e00640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f511406c8f0 0x7f511406edb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f512400aac0 tx=0x7f5124008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:02:59.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.331+0000 7f512a000640 1 -- 172.21.15.40:0/3040005123 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f512c020020 con 0x7f51340ca200 2024-04-26T23:02:59.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.585+0000 7f5138e00640 1 -- 172.21.15.40:0/3040005123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51340b6860 con 0x7f51340ca200 2024-04-26T23:02:59.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.586+0000 7f512a000640 1 -- 172.21.15.40:0/3040005123 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5178 (secure 0 0 0) 0x7f512c059ad0 con 0x7f51340ca200 2024-04-26T23:02:59.587 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:02:59.587 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1979956236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1979956236},{"type":"v1","addr":"172.21.15.103:6837","nonce":1979956236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/603576312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":603576312},{"type":"v1","addr":"172.21.15.103:6836","nonce":603576312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5331,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/716981348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":716981348},{"type":"v1","addr":"172.21.15.40:6834","nonce":716981348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:02:47.924696+0000","modified":"2024-04-26T23:02:50.931247+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5334},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5334":{"gid":5334,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.40:6835/299800334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":299800334},{"type":"v1","addr":"172.21.15.40:6835","nonce":299800334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5334,"qdb_cluster":[5334]},"id":2}]} 2024-04-26T23:02:59.587 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 18 2024-04-26T23:02:59.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.589+0000 7f5138e00640 1 -- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f511406c8f0 msgr2=0x7f511406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:59.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.589+0000 7f5138e00640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f511406c8f0 0x7f511406edb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f512400aac0 tx=0x7f5124008040 comp rx=0 tx=0).stop 2024-04-26T23:02:59.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.589+0000 7f5138e00640 1 -- 172.21.15.40:0/3040005123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51340ca200 msgr2=0x7f51340b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:02:59.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.589+0000 7f5138e00640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51340ca200 0x7f51340b5640 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f512c00c700 tx=0x7f512c00cbc0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.589+0000 7f5138e00640 1 -- 172.21.15.40:0/3040005123 shutdown_connections 2024-04-26T23:02:59.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.589+0000 7f5138e00640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f511406c8f0 0x7f511406edb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.589+0000 7f5138e00640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51340ca200 0x7f51340b5640 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.589+0000 7f5138e00640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f513415f380 0x7f51340c9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.589+0000 7f5138e00640 1 --2- 172.21.15.40:0/3040005123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51340d0620 0x7f513414eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:02:59.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.589+0000 7f5138e00640 1 -- 172.21.15.40:0/3040005123 >> 172.21.15.40:0/3040005123 conn(0x7f51340d0be0 msgr2=0x7f51340cd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:02:59.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.590+0000 7f5138e00640 1 -- 172.21.15.40:0/3040005123 shutdown_connections 2024-04-26T23:02:59.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:02:59.590+0000 7f5138e00640 1 -- 172.21.15.40:0/3040005123 wait complete. 2024-04-26T23:03:00.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:02:47.924696+0000', 'modified': '2024-04-26T23:02:50.931247+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5334}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5334': {'gid': 5334, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.40:6835/299800334', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 299800334}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 299800334}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5334, 'qdb_cluster': [5334]} 2024-04-26T23:03:00.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-26T23:03:00.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T23:03:00.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.237+0000 7f67c3200640 1 Processor -- start 2024-04-26T23:03:00.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.237+0000 7f67c3200640 1 -- start start 2024-04-26T23:03:00.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.239+0000 7f67c3200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f67c40ce5e0 0x7f67c40afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.239+0000 7f67c3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67c40cc590 0x7f67c40cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.239+0000 7f67c3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67c40af7b0 con 0x7f67c415f2d0 2024-04-26T23:03:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.239+0000 7f67c3200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f67c40afa90 con 0x7f67c40ce5e0 2024-04-26T23:03:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.239+0000 7f67c3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f67c40cbd90 con 0x7f67c40cc590 2024-04-26T23:03:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.240+0000 7f67c1400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f67c40ce5e0 0x7f67c40afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.240+0000 7f67c1400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f67c40ce5e0 0x7f67c40afdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36782/0 (socket says 172.21.15.40:36782) 2024-04-26T23:03:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.240+0000 7f67c1400640 1 -- 172.21.15.40:0/2737860257 learned_addr learned my addr 172.21.15.40:0/2737860257 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.240+0000 7f67c2800640 1 --2- 172.21.15.40:0/2737860257 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67c40cc590 0x7f67c40cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.240+0000 7f67c1400640 1 -- 172.21.15.40:0/2737860257 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67c40cc590 msgr2=0x7f67c40cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.240+0000 7f67c1400640 1 --2- 172.21.15.40:0/2737860257 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67c40cc590 0x7f67c40cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.240+0000 7f67c1400640 1 -- 172.21.15.40:0/2737860257 >> v1:172.21.15.103:6789/0 conn(0x7f67c415f2d0 legacy=0x7f67c40c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.240+0000 7f67c1400640 1 -- 172.21.15.40:0/2737860257 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c40cc070 con 0x7f67c40ce5e0 2024-04-26T23:03:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.240+0000 7f67c2800640 1 --2- 172.21.15.40:0/2737860257 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67c40cc590 0x7f67c40cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.241+0000 7f67c1400640 1 --2- 172.21.15.40:0/2737860257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c40ce5e0 0x7f67c40afdf0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f67b000bb10 tx=0x7f67b002c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c20762b2dda7369c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:00.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.242+0000 7f67c0a00640 1 -- 172.21.15.40:0/2737860257 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b0032790 con 0x7f67c40ce5e0 2024-04-26T23:03:00.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.242+0000 7f67c0a00640 1 -- 172.21.15.40:0/2737860257 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67b0038070 con 0x7f67c40ce5e0 2024-04-26T23:03:00.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.242+0000 7f67c0a00640 1 -- 172.21.15.40:0/2737860257 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b0032b50 con 0x7f67c40ce5e0 2024-04-26T23:03:00.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.243+0000 7f67c3200640 1 -- 172.21.15.40:0/2737860257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c40ce5e0 msgr2=0x7f67c40afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.243+0000 7f67c3200640 1 --2- 172.21.15.40:0/2737860257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c40ce5e0 0x7f67c40afdf0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f67b000bb10 tx=0x7f67b002c8d0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.243+0000 7f67c3200640 1 -- 172.21.15.40:0/2737860257 shutdown_connections 2024-04-26T23:03:00.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.243+0000 7f67c3200640 1 --2- 172.21.15.40:0/2737860257 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67c40cc590 0x7f67c40cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.243+0000 7f67c3200640 1 --2- 172.21.15.40:0/2737860257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c40ce5e0 0x7f67c40afdf0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.243+0000 7f67c3200640 1 -- 172.21.15.40:0/2737860257 >> 172.21.15.40:0/2737860257 conn(0x7f67c40d0a10 msgr2=0x7f67c40d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:00.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.243+0000 7f67c3200640 1 -- 172.21.15.40:0/2737860257 shutdown_connections 2024-04-26T23:03:00.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.243+0000 7f67c3200640 1 -- 172.21.15.40:0/2737860257 wait complete. 2024-04-26T23:03:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.245+0000 7f67c3200640 1 Processor -- start 2024-04-26T23:03:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.246+0000 7f67c3200640 1 -- start start 2024-04-26T23:03:00.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.247+0000 7f67c3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67c40cc590 0x7f67c41505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.247+0000 7f67c3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67c40ce5e0 0x7f67c415e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.247+0000 7f67c3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c415f2d0 0x7f67c415e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.247+0000 7f67c3200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f67c40c5fc0 con 0x7f67c415f2d0 2024-04-26T23:03:00.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.247+0000 7f67c3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f67c406e270 con 0x7f67c40ce5e0 2024-04-26T23:03:00.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.247+0000 7f67c3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f67c40b0440 con 0x7f67c40cc590 2024-04-26T23:03:00.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.247+0000 7f67c2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c415f2d0 0x7f67c415e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.247+0000 7f67c2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c415f2d0 0x7f67c415e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36796/0 (socket says 172.21.15.40:36796) 2024-04-26T23:03:00.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.247+0000 7f67c2800640 1 -- 172.21.15.40:0/2751746127 learned_addr learned my addr 172.21.15.40:0/2751746127 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:00.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.248+0000 7f67c1400640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67c40ce5e0 0x7f67c415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.248+0000 7f67c1e00640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67c40cc590 0x7f67c41505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.248+0000 7f67c2800640 1 -- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67c40cc590 msgr2=0x7f67c41505c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.248+0000 7f67c2800640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67c40cc590 0x7f67c41505c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.248+0000 7f67c2800640 1 -- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67c40ce5e0 msgr2=0x7f67c415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.248+0000 7f67c2800640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67c40ce5e0 0x7f67c415e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.248+0000 7f67c2800640 1 -- 172.21.15.40:0/2751746127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c40b0720 con 0x7f67c415f2d0 2024-04-26T23:03:00.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.248+0000 7f67c1400640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67c40ce5e0 0x7f67c415e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:03:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.248+0000 7f67c2800640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c415f2d0 0x7f67c415e7c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00cef0 tx=0x7f67bc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.249+0000 7f67ba000640 1 -- 172.21.15.40:0/2751746127 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc01a070 con 0x7f67c415f2d0 2024-04-26T23:03:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.249+0000 7f67ba000640 1 -- 172.21.15.40:0/2751746127 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67bc00d860 con 0x7f67c415f2d0 2024-04-26T23:03:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.249+0000 7f67ba000640 1 -- 172.21.15.40:0/2751746127 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc00dc50 con 0x7f67c415f2d0 2024-04-26T23:03:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.249+0000 7f67c3200640 1 -- 172.21.15.40:0/2751746127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67c40b0a00 con 0x7f67c415f2d0 2024-04-26T23:03:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.249+0000 7f67c3200640 1 -- 172.21.15.40:0/2751746127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67c40b0fc0 con 0x7f67c415f2d0 2024-04-26T23:03:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.251+0000 7f67ba000640 1 -- 172.21.15.40:0/2751746127 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f67bc01d070 con 0x7f67c415f2d0 2024-04-26T23:03:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.252+0000 7f67ba000640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f67a406c920 0x7f67a406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.252+0000 7f67ba000640 1 -- 172.21.15.40:0/2751746127 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f67bc08c580 con 0x7f67c415f2d0 2024-04-26T23:03:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.253+0000 7f67cbe5e740 1 Processor -- start 2024-04-26T23:03:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.253+0000 7f67c1e00640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f67a406c920 0x7f67a406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.253+0000 7f67cbe5e740 1 -- start start 2024-04-26T23:03:00.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.254+0000 7f67cbe5e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e593e87b0 0x564e593ea910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.254+0000 7f67cbe5e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e593e8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.254+0000 7f67cbe5e740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e59413270 0x564e594136f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.254+0000 7f67cbe5e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564e5934c600 con 0x564e59424e80 2024-04-26T23:03:00.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.254+0000 7f67cbe5e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564e592e9100 con 0x564e593e87b0 2024-04-26T23:03:00.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.254+0000 7f67cbe5e740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564e593485c0 con 0x564e59413270 2024-04-26T23:03:00.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.254+0000 7f67c1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e593e8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.254+0000 7f67c1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e593e8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36804/0 (socket says 172.21.15.40:36804) 2024-04-26T23:03:00.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.254+0000 7f67c1400640 1 -- 172.21.15.40:0/2842633956 learned_addr learned my addr 172.21.15.40:0/2842633956 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:00.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.254+0000 7f67c1e00640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f67a406c920 0x7f67a406ede0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f67b4006fd0 tx=0x7f67b4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:00.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.254+0000 7f67c1e00640 1 --2- 172.21.15.40:0/2842633956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e593e87b0 0x564e593ea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.255+0000 7f67c1400640 1 -- 172.21.15.40:0/2842633956 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e59413270 msgr2=0x564e594136f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:00.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.255+0000 7f67c1400640 1 --2- 172.21.15.40:0/2842633956 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e59413270 0x564e594136f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.255+0000 7f67c1400640 1 -- 172.21.15.40:0/2842633956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e593e87b0 msgr2=0x564e593ea910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:00.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.255+0000 7f67c1400640 1 --2- 172.21.15.40:0/2842633956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e593e87b0 0x564e593ea910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.255+0000 7f67c1400640 1 -- 172.21.15.40:0/2842633956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e5905e2d0 con 0x564e59424e80 2024-04-26T23:03:00.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.255+0000 7f67c1400640 1 --2- 172.21.15.40:0/2842633956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e593e8b90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f67b0000c00 tx=0x7f67b0031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9e63eb72ed22d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:00.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.255+0000 7f67aea00640 1 -- 172.21.15.40:0/2842633956 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b0032bd0 con 0x564e59424e80 2024-04-26T23:03:00.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.255+0000 7f67aea00640 1 -- 172.21.15.40:0/2842633956 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67b0038040 con 0x564e59424e80 2024-04-26T23:03:00.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.255+0000 7f67aea00640 1 -- 172.21.15.40:0/2842633956 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b0033420 con 0x564e59424e80 2024-04-26T23:03:00.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.256+0000 7f67cbe5e740 1 -- 172.21.15.40:0/2842633956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 msgr2=0x564e593e8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.256+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/2842633956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e593e8b90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f67b0000c00 tx=0x7f67b0031000 comp rx=0 tx=0).stop 2024-04-26T23:03:00.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.256+0000 7f67cbe5e740 1 -- 172.21.15.40:0/2842633956 shutdown_connections 2024-04-26T23:03:00.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.256+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/2842633956 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e59413270 0x564e594136f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.256+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/2842633956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e593e8b90 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.256+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/2842633956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e593e87b0 0x564e593ea910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.256+0000 7f67cbe5e740 1 -- 172.21.15.40:0/2842633956 >> 172.21.15.40:0/2842633956 conn(0x564e5937cd10 msgr2=0x564e5942e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:00.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.256+0000 7f67cbe5e740 1 -- 172.21.15.40:0/2842633956 shutdown_connections 2024-04-26T23:03:00.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.256+0000 7f67cbe5e740 1 -- 172.21.15.40:0/2842633956 wait complete. 2024-04-26T23:03:00.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.257+0000 7f67cbe5e740 1 -- start start 2024-04-26T23:03:00.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.257+0000 7f67cbe5e740 1 Processor -- start 2024-04-26T23:03:00.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.258+0000 7f67cbe5e740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e593e87b0 0x564e59414420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.258+0000 7f67cbe5e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e59413270 0x564e59426b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.258+0000 7f67cbe5e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e59400290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.258+0000 7f67cbe5e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564e592d3f70 con 0x564e59424e80 2024-04-26T23:03:00.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.258+0000 7f67cbe5e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564e59410dc0 con 0x564e59413270 2024-04-26T23:03:00.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.258+0000 7f67cbe5e740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564e59427040 con 0x564e593e87b0 2024-04-26T23:03:00.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.258+0000 7f67cbe5e740 10 client.0 _ll_register_callbacks cb 0x564e59342480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:00.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.258+0000 7f67c2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e59400290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.258+0000 7f67c2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e59400290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36816/0 (socket says 172.21.15.40:36816) 2024-04-26T23:03:00.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.259+0000 7f67c2800640 1 -- 172.21.15.40:0/2541548237 learned_addr learned my addr 172.21.15.40:0/2541548237 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:00.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.259+0000 7f67c1400640 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e59413270 0x564e59426b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.259+0000 7f67c1e00640 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e593e87b0 0x564e59414420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.259+0000 7f67c2800640 1 -- 172.21.15.40:0/2541548237 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e593e87b0 msgr2=0x564e59414420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:00.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.259+0000 7f67c2800640 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e593e87b0 0x564e59414420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.259+0000 7f67c2800640 1 -- 172.21.15.40:0/2541548237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e59413270 msgr2=0x564e59426b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:00.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.259+0000 7f67c2800640 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e59413270 0x564e59426b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.259+0000 7f67c2800640 1 -- 172.21.15.40:0/2541548237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e591286b0 con 0x564e59424e80 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.259+0000 7f67c1400640 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e59413270 0x564e59426b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.259+0000 7f67c2800640 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e59400290 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f67bc05a160 tx=0x7f67bc05a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4613977b482f0908 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.259+0000 7f67a3400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.259+0000 7f67a3400640 1 -- 172.21.15.40:0/2541548237 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc05e120 con 0x564e59424e80 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.260+0000 7f67cbe5e740 1 -- 172.21.15.40:0/2541548237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67b0038040 con 0x564e59424e80 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.260+0000 7f67a3400640 1 -- 172.21.15.40:0/2541548237 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f67bc05e2c0 con 0x564e59424e80 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.260+0000 7f67a3400640 1 -- 172.21.15.40:0/2541548237 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f67bc062920 con 0x564e59424e80 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.260+0000 7f67cbe5e740 10 client.5529 fetch_fsmap learned FSMap version 18 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.260+0000 7f67cbe5e740 1 -- 172.21.15.40:0/2541548237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x564e59426350 con 0x564e59424e80 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.260+0000 7f67a3400640 1 -- 172.21.15.40:0/2541548237 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67b0038040 con 0x564e59424e80 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.260+0000 7f67a3400640 1 -- 172.21.15.40:0/2541548237 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7f67bc08ace0 con 0x564e59424e80 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.260+0000 7f67cbe5e740 10 client.5529 fetch_fsmap finished waiting for FSMap version 18 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.260+0000 7f67cbe5e740 10 client.5529 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.260+0000 7f67cbe5e740 20 client.5529 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:00.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.260+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x564e594007d0 0x564e59415310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.260+0000 7f67cbe5e740 4 client.5529 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:03:00.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.260+0000 7f67cbe5e740 1 -- 172.21.15.40:0/2541548237 --> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e5935a210 con 0x564e594007d0 2024-04-26T23:03:00.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.261+0000 7f67c1e00640 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x564e594007d0 0x564e59415310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.261+0000 7f67c1e00640 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x564e594007d0 0x564e59415310 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68bd30cb96a9f101 server_cookie=1e61b430c77389fb in_seq=0 out_seq=0 2024-04-26T23:03:00.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.261+0000 7f67a3400640 10 client.5529 ms_handle_connect on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:00.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.262+0000 7f67a3400640 1 -- 172.21.15.40:0/2541548237 <== mds.0 v2:172.21.15.40:6833/299800334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e5935a210 con 0x564e594007d0 2024-04-26T23:03:00.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.262+0000 7f67a3400640 10 client.5529 handle_command_reply: tid=0 2024-04-26T23:03:00.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.262+0000 7f67cbe5e740 1 client.5529 shutdown 2024-04-26T23:03:00.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.263+0000 7f67cbe5e740 1 -- 172.21.15.40:0/2541548237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 msgr2=0x564e59400290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.263+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e59400290 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f67bc05a160 tx=0x7f67bc05a1d0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.263+0000 7f67cbe5e740 1 -- 172.21.15.40:0/2541548237 shutdown_connections 2024-04-26T23:03:00.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.263+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x564e594007d0 0x564e59415310 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.263+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e59400290 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.263+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e59413270 0x564e59426b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.263+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/2541548237 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e593e87b0 0x564e59414420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.263+0000 7f67cbe5e740 1 -- 172.21.15.40:0/2541548237 >> 172.21.15.40:0/2541548237 conn(0x564e5937cd10 msgr2=0x564e594111c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.263+0000 7f67a3400640 0 client.5529 ms_handle_reset on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.263+0000 7f67cbe5e740 1 -- 172.21.15.40:0/2541548237 shutdown_connections 2024-04-26T23:03:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.263+0000 7f67cbe5e740 1 -- 172.21.15.40:0/2541548237 wait complete. 2024-04-26T23:03:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.263+0000 7f67cbe5e740 20 client.5529 trim_cache size 0 max 16384 2024-04-26T23:03:00.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.283+0000 7f67cbe5e740 1 Processor -- start 2024-04-26T23:03:00.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.283+0000 7f67cbe5e740 1 -- start start 2024-04-26T23:03:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.284+0000 7f67cbe5e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e593e87b0 0x564e594268c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.284+0000 7f67cbe5e740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e59413270 0x564e59440ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.284+0000 7f67cbe5e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e593e26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.284+0000 7f67cbe5e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f67c4200ae0 con 0x564e59424e80 2024-04-26T23:03:00.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.284+0000 7f67cbe5e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f67c415d870 con 0x564e593e87b0 2024-04-26T23:03:00.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.284+0000 7f67cbe5e740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f67c415dec0 con 0x564e59413270 2024-04-26T23:03:00.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.284+0000 7f67c2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e593e26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.284+0000 7f67c2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e593e26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36826/0 (socket says 172.21.15.40:36826) 2024-04-26T23:03:00.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.284+0000 7f67c2800640 1 -- 172.21.15.40:0/340427355 learned_addr learned my addr 172.21.15.40:0/340427355 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:00.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.285+0000 7f67c2800640 1 -- 172.21.15.40:0/340427355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e59413270 msgr2=0x564e59440ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:00.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.285+0000 7f67c2800640 1 --2- 172.21.15.40:0/340427355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e59413270 0x564e59440ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.285+0000 7f67c2800640 1 -- 172.21.15.40:0/340427355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e593e87b0 msgr2=0x564e594268c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:00.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.285+0000 7f67c2800640 1 --2- 172.21.15.40:0/340427355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e593e87b0 0x564e594268c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.285+0000 7f67c2800640 1 -- 172.21.15.40:0/340427355 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e593df460 con 0x564e59424e80 2024-04-26T23:03:00.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.285+0000 7f67c2800640 1 --2- 172.21.15.40:0/340427355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e593e26d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f67bc058480 tx=0x7f67bc08c050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f694d575f044e52d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:00.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.285+0000 7f67ae000640 1 -- 172.21.15.40:0/340427355 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc059b30 con 0x564e59424e80 2024-04-26T23:03:00.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.285+0000 7f67ae000640 1 -- 172.21.15.40:0/340427355 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67bc05e120 con 0x564e59424e80 2024-04-26T23:03:00.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.285+0000 7f67ae000640 1 -- 172.21.15.40:0/340427355 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc08c0c0 con 0x564e59424e80 2024-04-26T23:03:00.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.286+0000 7f67cbe5e740 1 -- 172.21.15.40:0/340427355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 msgr2=0x564e593e26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.286+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/340427355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e593e26d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f67bc058480 tx=0x7f67bc08c050 comp rx=0 tx=0).stop 2024-04-26T23:03:00.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.286+0000 7f67cbe5e740 1 -- 172.21.15.40:0/340427355 shutdown_connections 2024-04-26T23:03:00.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.286+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/340427355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e593e26d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.286+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/340427355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e59413270 0x564e59440ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.286+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/340427355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e593e87b0 0x564e594268c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.286+0000 7f67cbe5e740 1 -- 172.21.15.40:0/340427355 >> 172.21.15.40:0/340427355 conn(0x564e5937cd10 msgr2=0x564e593fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:00.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.286+0000 7f67cbe5e740 1 -- 172.21.15.40:0/340427355 shutdown_connections 2024-04-26T23:03:00.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.286+0000 7f67cbe5e740 1 -- 172.21.15.40:0/340427355 wait complete. 2024-04-26T23:03:00.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.287+0000 7f67cbe5e740 1 -- start start 2024-04-26T23:03:00.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.287+0000 7f67cbe5e740 1 Processor -- start 2024-04-26T23:03:00.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.288+0000 7f67cbe5e740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e593e87b0 0x564e59431f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.288+0000 7f67cbe5e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e59413270 0x564e594a7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.288+0000 7f67cbe5e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e594a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.288+0000 7f67cbe5e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564e593ffd00 con 0x564e59424e80 2024-04-26T23:03:00.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.288+0000 7f67cbe5e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564e59400d70 con 0x564e59413270 2024-04-26T23:03:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.288+0000 7f67cbe5e740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564e5940fb30 con 0x564e593e87b0 2024-04-26T23:03:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.288+0000 7f67cbe5e740 10 client.0 _ll_register_callbacks cb 0x7f67bc056470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.288+0000 7f67c2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e594a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67c2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e594a79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36842/0 (socket says 172.21.15.40:36842) 2024-04-26T23:03:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67c2800640 1 -- 172.21.15.40:0/1509089888 learned_addr learned my addr 172.21.15.40:0/1509089888 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67c1e00640 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e593e87b0 0x564e59431f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67c2800640 1 -- 172.21.15.40:0/1509089888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e593e87b0 msgr2=0x564e59431f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67c1400640 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e59413270 0x564e594a7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67c2800640 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e593e87b0 0x564e59431f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67c2800640 1 -- 172.21.15.40:0/1509089888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e59413270 msgr2=0x564e594a7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67c2800640 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e59413270 0x564e594a7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67c2800640 1 -- 172.21.15.40:0/1509089888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e59423ca0 con 0x564e59424e80 2024-04-26T23:03:00.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67c1e00640 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e593e87b0 0x564e59431f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:00.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67c1400640 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e59413270 0x564e594a7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:00.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67c2800640 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e594a79d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f67bc058720 tx=0x7f67bc021070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af4b69f8430c5e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:00.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67a3400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:00.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.289+0000 7f67a3400640 1 -- 172.21.15.40:0/1509089888 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc062ac0 con 0x564e59424e80 2024-04-26T23:03:00.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.290+0000 7f67a3400640 1 -- 172.21.15.40:0/1509089888 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f67bc059830 con 0x564e59424e80 2024-04-26T23:03:00.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.290+0000 7f67cbe5e740 1 -- 172.21.15.40:0/1509089888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67bc05e120 con 0x564e59424e80 2024-04-26T23:03:00.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.290+0000 7f67a3400640 1 -- 172.21.15.40:0/1509089888 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f67bc06a050 con 0x564e59424e80 2024-04-26T23:03:00.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.290+0000 7f67cbe5e740 10 client.5535 fetch_fsmap learned FSMap version 18 2024-04-26T23:03:00.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.290+0000 7f67cbe5e740 1 -- 172.21.15.40:0/1509089888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x564e59423ca0 con 0x564e59424e80 2024-04-26T23:03:00.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.290+0000 7f67a3400640 1 -- 172.21.15.40:0/1509089888 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67bc05e120 con 0x564e59424e80 2024-04-26T23:03:00.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.290+0000 7f67a3400640 1 -- 172.21.15.40:0/1509089888 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7f67bc06d020 con 0x564e59424e80 2024-04-26T23:03:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.290+0000 7f67cbe5e740 10 client.5535 fetch_fsmap finished waiting for FSMap version 18 2024-04-26T23:03:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.290+0000 7f67cbe5e740 10 client.5535 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:03:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.290+0000 7f67cbe5e740 20 client.5535 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.290+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x564e593e7630 0x564e59413900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.290+0000 7f67cbe5e740 4 client.5535 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:03:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.290+0000 7f67cbe5e740 1 -- 172.21.15.40:0/1509089888 --> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] -- command(tid 0: {"prefix": "status"}) -- 0x564e59410e50 con 0x564e593e7630 2024-04-26T23:03:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.291+0000 7f67c1e00640 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x564e593e7630 0x564e59413900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.291+0000 7f67c1e00640 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x564e593e7630 0x564e59413900 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d78ef71df7cb3645 server_cookie=43c8e4ece5387dfd in_seq=0 out_seq=0 2024-04-26T23:03:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.291+0000 7f67a3400640 10 client.5535 ms_handle_connect on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67a3400640 1 -- 172.21.15.40:0/1509089888 <== mds.0 v2:172.21.15.40:6833/299800334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x564e59410e50 con 0x564e593e7630 2024-04-26T23:03:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67a3400640 10 client.5535 handle_command_reply: tid=0 2024-04-26T23:03:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67cbe5e740 1 client.5535 shutdown 2024-04-26T23:03:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67cbe5e740 1 -- 172.21.15.40:0/1509089888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 msgr2=0x564e594a79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e594a79d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f67bc058720 tx=0x7f67bc021070 comp rx=0 tx=0).stop 2024-04-26T23:03:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67cbe5e740 1 -- 172.21.15.40:0/1509089888 shutdown_connections 2024-04-26T23:03:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x564e593e7630 0x564e59413900 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564e59424e80 0x564e594a79d0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564e59413270 0x564e594a7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67cbe5e740 1 --2- 172.21.15.40:0/1509089888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564e593e87b0 0x564e59431f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67a3400640 0 client.5535 ms_handle_reset on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67cbe5e740 1 -- 172.21.15.40:0/1509089888 >> 172.21.15.40:0/1509089888 conn(0x564e5937cd10 msgr2=0x564e59426e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67cbe5e740 1 -- 172.21.15.40:0/1509089888 shutdown_connections 2024-04-26T23:03:00.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67cbe5e740 1 -- 172.21.15.40:0/1509089888 wait complete. 2024-04-26T23:03:00.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.292+0000 7f67cbe5e740 20 client.5535 trim_cache size 0 max 16384 2024-04-26T23:03:00.297 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:03:00.297 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:03:00.297 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:03:00.297 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5334, 2024-04-26T23:03:00.297 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:03:00.297 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:03:00.298 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:03:00.298 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 11.898288452999999, 2024-04-26T23:03:00.298 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 18, 2024-04-26T23:03:00.298 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 39, 2024-04-26T23:03:00.298 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 39, 2024-04-26T23:03:00.298 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 20.705804364999999 2024-04-26T23:03:00.298 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:03:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.293+0000 7f67c3200640 1 -- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f67a406c920 msgr2=0x7f67a406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.293+0000 7f67c3200640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f67a406c920 0x7f67a406ede0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f67b4006fd0 tx=0x7f67b4008040 comp rx=0 tx=0).stop 2024-04-26T23:03:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.293+0000 7f67c3200640 1 -- 172.21.15.40:0/2751746127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c415f2d0 msgr2=0x7f67c415e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.293+0000 7f67c3200640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c415f2d0 0x7f67c415e7c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00cef0 tx=0x7f67bc00a710 comp rx=0 tx=0).stop 2024-04-26T23:03:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.293+0000 7f67c3200640 1 -- 172.21.15.40:0/2751746127 shutdown_connections 2024-04-26T23:03:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.293+0000 7f67c3200640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f67a406c920 0x7f67a406ede0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.293+0000 7f67c3200640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67c415f2d0 0x7f67c415e7c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.293+0000 7f67c3200640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67c40ce5e0 0x7f67c415e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.293+0000 7f67c3200640 1 --2- 172.21.15.40:0/2751746127 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67c40cc590 0x7f67c41505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.293+0000 7f67c3200640 1 -- 172.21.15.40:0/2751746127 >> 172.21.15.40:0/2751746127 conn(0x7f67c40d0a10 msgr2=0x7f67c40c62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:00.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.293+0000 7f67c3200640 1 -- 172.21.15.40:0/2751746127 shutdown_connections 2024-04-26T23:03:00.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.293+0000 7f67c3200640 1 -- 172.21.15.40:0/2751746127 wait complete. 2024-04-26T23:03:00.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-26T23:03:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.519+0000 7efc5be00640 1 Processor -- start 2024-04-26T23:03:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.519+0000 7efc5be00640 1 -- start start 2024-04-26T23:03:00.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.520+0000 7efc5be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efc5c0d2400 0x7efc5c0d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.520+0000 7efc5be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc5c0d0c80 0x7efc5c0b5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.520+0000 7efc5be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc5c0d20f0 con 0x7efc5c0d17d0 2024-04-26T23:03:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.520+0000 7efc5be00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7efc5c0d37b0 con 0x7efc5c0d0c80 2024-04-26T23:03:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.520+0000 7efc5be00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7efc5c0d3d70 con 0x7efc5c0d2400 2024-04-26T23:03:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.521+0000 7efc5b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc5c0d0c80 0x7efc5c0b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.521+0000 7efc5b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc5c0d0c80 0x7efc5c0b5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36844/0 (socket says 172.21.15.40:36844) 2024-04-26T23:03:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.521+0000 7efc5b400640 1 -- 172.21.15.40:0/1746222278 learned_addr learned my addr 172.21.15.40:0/1746222278 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.521+0000 7efc5a000640 1 --2- 172.21.15.40:0/1746222278 >> v2:172.21.15.103:3300/0 conn(0x7efc5c0d2400 0x7efc5c0d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.521+0000 7efc5b400640 1 -- 172.21.15.40:0/1746222278 >> v2:172.21.15.103:3300/0 conn(0x7efc5c0d2400 msgr2=0x7efc5c0d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.521+0000 7efc5b400640 1 --2- 172.21.15.40:0/1746222278 >> v2:172.21.15.103:3300/0 conn(0x7efc5c0d2400 0x7efc5c0d1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.521+0000 7efc5b400640 1 -- 172.21.15.40:0/1746222278 >> v1:172.21.15.103:6789/0 conn(0x7efc5c0d17d0 legacy=0x7efc5c0b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:00.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.521+0000 7efc5b400640 1 -- 172.21.15.40:0/1746222278 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc5c15f9c0 con 0x7efc5c0d0c80 2024-04-26T23:03:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.522+0000 7efc5b400640 1 --2- 172.21.15.40:0/1746222278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc5c0d0c80 0x7efc5c0b5210 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0b6920 tx=0x7efc5402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5dc957f1260340 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.523+0000 7efc59600640 1 -- 172.21.15.40:0/1746222278 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc54036070 con 0x7efc5c0d0c80 2024-04-26T23:03:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.523+0000 7efc59600640 1 -- 172.21.15.40:0/1746222278 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc54031600 con 0x7efc5c0d0c80 2024-04-26T23:03:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.523+0000 7efc59600640 1 -- 172.21.15.40:0/1746222278 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc540319d0 con 0x7efc5c0d0c80 2024-04-26T23:03:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.523+0000 7efc5be00640 1 -- 172.21.15.40:0/1746222278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc5c0d0c80 msgr2=0x7efc5c0b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.523+0000 7efc5be00640 1 --2- 172.21.15.40:0/1746222278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc5c0d0c80 0x7efc5c0b5210 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0b6920 tx=0x7efc5402a660 comp rx=0 tx=0).stop 2024-04-26T23:03:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.523+0000 7efc5be00640 1 -- 172.21.15.40:0/1746222278 shutdown_connections 2024-04-26T23:03:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.523+0000 7efc5be00640 1 --2- 172.21.15.40:0/1746222278 >> v2:172.21.15.103:3300/0 conn(0x7efc5c0d2400 0x7efc5c0d1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.523+0000 7efc5be00640 1 --2- 172.21.15.40:0/1746222278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc5c0d0c80 0x7efc5c0b5210 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.523+0000 7efc5be00640 1 -- 172.21.15.40:0/1746222278 >> 172.21.15.40:0/1746222278 conn(0x7efc5c09e210 msgr2=0x7efc5c0d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:00.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.524+0000 7efc5be00640 1 -- 172.21.15.40:0/1746222278 shutdown_connections 2024-04-26T23:03:00.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.524+0000 7efc5be00640 1 -- 172.21.15.40:0/1746222278 wait complete. 2024-04-26T23:03:00.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.525+0000 7efc5be00640 1 Processor -- start 2024-04-26T23:03:00.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.526+0000 7efc5be00640 1 -- start start 2024-04-26T23:03:00.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.526+0000 7efc5be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc5c0d0c80 0x7efc5c15e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.526+0000 7efc5be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc5c0d17d0 0x7efc5c14ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.527+0000 7efc5be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc5c0d2400 0x7efc5c14d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.527+0000 7efc5be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c0d3a00 con 0x7efc5c0d0c80 2024-04-26T23:03:00.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.527+0000 7efc5be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c0d5a10 con 0x7efc5c0d17d0 2024-04-26T23:03:00.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.527+0000 7efc5be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efc5c0d10d0 con 0x7efc5c0d2400 2024-04-26T23:03:00.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.527+0000 7efc5aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc5c0d0c80 0x7efc5c15e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.527+0000 7efc5aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc5c0d0c80 0x7efc5c15e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36848/0 (socket says 172.21.15.40:36848) 2024-04-26T23:03:00.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.527+0000 7efc5aa00640 1 -- 172.21.15.40:0/1461741308 learned_addr learned my addr 172.21.15.40:0/1461741308 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:00.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.527+0000 7efc5b400640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc5c0d2400 0x7efc5c14d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.527+0000 7efc5a000640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc5c0d17d0 0x7efc5c14ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.528+0000 7efc5aa00640 1 -- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc5c0d2400 msgr2=0x7efc5c14d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.528+0000 7efc5aa00640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc5c0d2400 0x7efc5c14d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.528+0000 7efc5aa00640 1 -- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc5c0d17d0 msgr2=0x7efc5c14ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.528+0000 7efc5aa00640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc5c0d17d0 0x7efc5c14ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.528+0000 7efc5aa00640 1 -- 172.21.15.40:0/1461741308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc5c1f9650 con 0x7efc5c0d0c80 2024-04-26T23:03:00.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.528+0000 7efc5aa00640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc5c0d0c80 0x7efc5c15e240 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00c430 tx=0x7efc4c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:00.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.528+0000 7efc52a00640 1 -- 172.21.15.40:0/1461741308 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4c011790 con 0x7efc5c0d0c80 2024-04-26T23:03:00.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.528+0000 7efc52a00640 1 -- 172.21.15.40:0/1461741308 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc4c017070 con 0x7efc5c0d0c80 2024-04-26T23:03:00.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.528+0000 7efc52a00640 1 -- 172.21.15.40:0/1461741308 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4c011bb0 con 0x7efc5c0d0c80 2024-04-26T23:03:00.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.528+0000 7efc5be00640 1 -- 172.21.15.40:0/1461741308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc5c0b7bf0 con 0x7efc5c0d0c80 2024-04-26T23:03:00.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.529+0000 7efc5be00640 1 -- 172.21.15.40:0/1461741308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc5c15e780 con 0x7efc5c0d0c80 2024-04-26T23:03:00.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.530+0000 7efc5be00640 1 -- 172.21.15.40:0/1461741308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc5c0d0a10 con 0x7efc5c0d0c80 2024-04-26T23:03:00.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.531+0000 7efc52a00640 1 -- 172.21.15.40:0/1461741308 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efc4c01c460 con 0x7efc5c0d0c80 2024-04-26T23:03:00.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.532+0000 7efc52a00640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc3806c920 0x7efc3806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:00.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.532+0000 7efc52a00640 1 -- 172.21.15.40:0/1461741308 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7efc4c08baf0 con 0x7efc5c0d0c80 2024-04-26T23:03:00.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.533+0000 7efc5a000640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc3806c920 0x7efc3806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:00.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.534+0000 7efc5a000640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc3806c920 0x7efc3806ede0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efc5c162bd0 tx=0x7efc44008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:00.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.537+0000 7efc52a00640 1 -- 172.21.15.40:0/1461741308 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7efc5c0d0a10 con 0x7efc5c0d0c80 2024-04-26T23:03:00.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.782+0000 7efc5be00640 1 -- 172.21.15.40:0/1461741308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7efc5c069370 con 0x7efc5c0d0c80 2024-04-26T23:03:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.783+0000 7efc52a00640 1 -- 172.21.15.40:0/1461741308 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+73 (secure 0 0 0) 0x7efc4c059d90 con 0x7efc5c0d0c80 2024-04-26T23:03:00.784 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-26T23:03:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.786+0000 7efc5be00640 1 -- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc3806c920 msgr2=0x7efc3806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.786+0000 7efc5be00640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc3806c920 0x7efc3806ede0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efc5c162bd0 tx=0x7efc44008040 comp rx=0 tx=0).stop 2024-04-26T23:03:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.786+0000 7efc5be00640 1 -- 172.21.15.40:0/1461741308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc5c0d0c80 msgr2=0x7efc5c15e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.786+0000 7efc5be00640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc5c0d0c80 0x7efc5c15e240 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00c430 tx=0x7efc4c00c8f0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.786+0000 7efc5be00640 1 -- 172.21.15.40:0/1461741308 shutdown_connections 2024-04-26T23:03:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.786+0000 7efc5be00640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc3806c920 0x7efc3806ede0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.786+0000 7efc5be00640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc5c0d2400 0x7efc5c14d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.786+0000 7efc5be00640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc5c0d17d0 0x7efc5c14ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.786+0000 7efc5be00640 1 --2- 172.21.15.40:0/1461741308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc5c0d0c80 0x7efc5c15e240 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.786+0000 7efc5be00640 1 -- 172.21.15.40:0/1461741308 >> 172.21.15.40:0/1461741308 conn(0x7efc5c09e210 msgr2=0x7efc5c06df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.786+0000 7efc5be00640 1 -- 172.21.15.40:0/1461741308 shutdown_connections 2024-04-26T23:03:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:00.786+0000 7efc5be00640 1 -- 172.21.15.40:0/1461741308 wait complete. 2024-04-26T23:03:00.805 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-26T23:03:00.805 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:03:00.805 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-04-26T23:03:00.806 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T23:03:00.806 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T23:03:00.806 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-26T23:03:00.806 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-26T23:03:00.806 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:03:00.806 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:03:00.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:00.806 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-04-26T23:03:00.857 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:03:00.857 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:03:00.857 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:03:00.857 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:03:00.857 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-26T23:03:00.858 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:03:00.858 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:03:00.858 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:03:00.858 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-04-26T23:03:00.858 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:03:00.858 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:03:00.858 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-04-26T23:03:00.858 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:03:00.858 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-04-26T23:03:00.858 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-04-26T23:03:00.858 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T23:03:00.858 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42533sec preferred_lft 42533sec 2024-04-26T23:03:00.858 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 8e:8e:79:d7:cf:5d brd ff:ff:ff:ff:ff:ff 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::4ce7:9bff:fe51:b1/64 scope link 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether be:f1:28:e1:99:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::bcf1:28ff:fee1:9921/64 scope link 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:03:00.859 INFO:teuthology.orchestra.run.smithi040.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:03:00.860 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 8e:8e:79:d7:cf:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-26T23:03:00.860 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::8c8e:79ff:fed7:cf5d/64 scope link 2024-04-26T23:03:00.860 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:03:00.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:00.860 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:03:00.944 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-04-26T23:03:00.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:00.949 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-04-26T23:03:01.001 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-04-26T23:03:01.002 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:03:01.002 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T23:03:01.002 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T23:03:01.002 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:03:01.002 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:03:01.002 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:03:01.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T23:03:01.003 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T23:03:01.003 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T23:03:01.003 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:03:01.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:03:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:03:01.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:01.225 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:03:01.276 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T23:03:01.276 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-26T23:03:01.277 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:03:01.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:01.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-04-26T23:03:01.366 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:03:01.366 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T23:03:01.424 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:03:01.425 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:03:01.425 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:03:01.480 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-26T23:03:01.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:01.480 DEBUG:teuthology.orchestra.run.smithi040:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:03:01.615 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:03:01.615 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:03:01.616 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:03:01.637 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:03:01.637 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:03:01.638 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:03:01.638 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:03:01.638 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:03:01.665 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:03:01.665 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:03:01.665 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:03:01.665 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:03:01.665 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:03:01.665 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-26T23:03:01.665 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-26T23:03:01.665 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=815936ab-d4ac-471e-b069-1a0dc70078ee" will pass to kernel 2024-04-26T23:03:01.667 INFO:teuthology.orchestra.run.smithi040.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T23:03:01.667 INFO:teuthology.orchestra.run.smithi040.stdout: You just mounted a file system that supports labels which does not 2024-04-26T23:03:01.667 INFO:teuthology.orchestra.run.smithi040.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T23:03:01.667 INFO:teuthology.orchestra.run.smithi040.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T23:03:01.667 INFO:teuthology.orchestra.run.smithi040.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T23:03:01.671 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T23:03:01.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:01.671 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:03:01.752 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:03:01.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:03:01.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:03:01.884 INFO:teuthology.orchestra.run.smithi040.stdout:client5544 2024-04-26T23:03:01.884 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:03:01.884 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:03:01.938 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-26T23:03:02.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.195+0000 7f4a8be00640 1 Processor -- start 2024-04-26T23:03:02.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.195+0000 7f4a8be00640 1 -- start start 2024-04-26T23:03:02.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.197+0000 7f4a8be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a8c160a50 0x7f4a8c073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.197+0000 7f4a8be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a8c06dd40 0x7f4a8c072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.197+0000 7f4a8be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a8c161880 con 0x7f4a8c0cfaf0 2024-04-26T23:03:02.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.197+0000 7f4a8be00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f4a8c071ce0 con 0x7f4a8c160a50 2024-04-26T23:03:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.197+0000 7f4a8be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4a8c0722a0 con 0x7f4a8c06dd40 2024-04-26T23:03:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.197+0000 7f4a8aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a8c160a50 0x7f4a8c073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.197+0000 7f4a8aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a8c160a50 0x7f4a8c073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36864/0 (socket says 172.21.15.40:36864) 2024-04-26T23:03:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.197+0000 7f4a8aa00640 1 -- 172.21.15.40:0/3934595150 learned_addr learned my addr 172.21.15.40:0/3934595150 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.198+0000 7f4a8b400640 1 --2- 172.21.15.40:0/3934595150 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a8c06dd40 0x7f4a8c072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:02.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.198+0000 7f4a8aa00640 1 -- 172.21.15.40:0/3934595150 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a8c06dd40 msgr2=0x7f4a8c072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.198+0000 7f4a8aa00640 1 --2- 172.21.15.40:0/3934595150 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a8c06dd40 0x7f4a8c072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.198+0000 7f4a8aa00640 1 -- 172.21.15.40:0/3934595150 >> v1:172.21.15.103:6789/0 conn(0x7f4a8c0cfaf0 legacy=0x7f4a8c160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.198+0000 7f4a8aa00640 1 -- 172.21.15.40:0/3934595150 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c0cff80 con 0x7f4a8c160a50 2024-04-26T23:03:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.199+0000 7f4a8aa00640 1 --2- 172.21.15.40:0/3934595150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a8c160a50 0x7f4a8c073a50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c009870 tx=0x7f4a7c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151370a60a43d941 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.199+0000 7f4a89600640 1 -- 172.21.15.40:0/3934595150 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c037070 con 0x7f4a8c160a50 2024-04-26T23:03:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.199+0000 7f4a89600640 1 -- 172.21.15.40:0/3934595150 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a7c032440 con 0x7f4a8c160a50 2024-04-26T23:03:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.199+0000 7f4a89600640 1 -- 172.21.15.40:0/3934595150 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c032810 con 0x7f4a8c160a50 2024-04-26T23:03:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.200+0000 7f4a8be00640 1 -- 172.21.15.40:0/3934595150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a8c160a50 msgr2=0x7f4a8c073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.200+0000 7f4a8be00640 1 --2- 172.21.15.40:0/3934595150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a8c160a50 0x7f4a8c073a50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c009870 tx=0x7f4a7c02a210 comp rx=0 tx=0).stop 2024-04-26T23:03:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.200+0000 7f4a8be00640 1 -- 172.21.15.40:0/3934595150 shutdown_connections 2024-04-26T23:03:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.200+0000 7f4a8be00640 1 --2- 172.21.15.40:0/3934595150 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a8c06dd40 0x7f4a8c072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.200+0000 7f4a8be00640 1 --2- 172.21.15.40:0/3934595150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a8c160a50 0x7f4a8c073a50 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.200+0000 7f4a8be00640 1 -- 172.21.15.40:0/3934595150 >> 172.21.15.40:0/3934595150 conn(0x7f4a8c0b2720 msgr2=0x7f4a8c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.200+0000 7f4a8be00640 1 -- 172.21.15.40:0/3934595150 shutdown_connections 2024-04-26T23:03:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.201+0000 7f4a8be00640 1 -- 172.21.15.40:0/3934595150 wait complete. 2024-04-26T23:03:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.202+0000 7f4a8be00640 1 Processor -- start 2024-04-26T23:03:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.202+0000 7f4a8be00640 1 -- start start 2024-04-26T23:03:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.203+0000 7f4a8be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a8c06dd40 0x7f4a8c143480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.203+0000 7f4a8be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a8c160a50 0x7f4a8c15a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.203+0000 7f4a8be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a8c15c4c0 0x7f4a8c15c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.203+0000 7f4a8be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c071f30 con 0x7f4a8c160a50 2024-04-26T23:03:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.203+0000 7f4a8be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c073f90 con 0x7f4a8c15c4c0 2024-04-26T23:03:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.203+0000 7f4a8be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4a8c06e1f0 con 0x7f4a8c06dd40 2024-04-26T23:03:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.204+0000 7f4a8a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a8c160a50 0x7f4a8c15a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.204+0000 7f4a8a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a8c160a50 0x7f4a8c15a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36872/0 (socket says 172.21.15.40:36872) 2024-04-26T23:03:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.204+0000 7f4a8a000640 1 -- 172.21.15.40:0/3770872042 learned_addr learned my addr 172.21.15.40:0/3770872042 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.204+0000 7f4a8aa00640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a8c06dd40 0x7f4a8c143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.204+0000 7f4a8b400640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a8c15c4c0 0x7f4a8c15c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.204+0000 7f4a8a000640 1 -- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a8c06dd40 msgr2=0x7f4a8c143480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.204+0000 7f4a8a000640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a8c06dd40 0x7f4a8c143480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.204+0000 7f4a8a000640 1 -- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a8c15c4c0 msgr2=0x7f4a8c15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.204+0000 7f4a8a000640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a8c15c4c0 0x7f4a8c15c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.204+0000 7f4a8a000640 1 -- 172.21.15.40:0/3770872042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c1439c0 con 0x7f4a8c160a50 2024-04-26T23:03:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.205+0000 7f4a8a000640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a8c160a50 0x7f4a8c15a610 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800c430 tx=0x7f4a7800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:02.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.205+0000 7f4a82a00640 1 -- 172.21.15.40:0/3770872042 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a78011790 con 0x7f4a8c160a50 2024-04-26T23:03:02.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.205+0000 7f4a82a00640 1 -- 172.21.15.40:0/3770872042 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a78017070 con 0x7f4a8c160a50 2024-04-26T23:03:02.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.205+0000 7f4a82a00640 1 -- 172.21.15.40:0/3770872042 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a78011bb0 con 0x7f4a8c160a50 2024-04-26T23:03:02.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.205+0000 7f4a8be00640 1 -- 172.21.15.40:0/3770872042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8c0b47a0 con 0x7f4a8c160a50 2024-04-26T23:03:02.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.205+0000 7f4a8be00640 1 -- 172.21.15.40:0/3770872042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a8c1429a0 con 0x7f4a8c160a50 2024-04-26T23:03:02.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.206+0000 7f4a8be00640 1 -- 172.21.15.40:0/3770872042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a8c069370 con 0x7f4a8c160a50 2024-04-26T23:03:02.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.207+0000 7f4a82a00640 1 -- 172.21.15.40:0/3770872042 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4a7801c460 con 0x7f4a8c160a50 2024-04-26T23:03:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.209+0000 7f4a82a00640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a6c06c920 0x7f4a6c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.209+0000 7f4a82a00640 1 -- 172.21.15.40:0/3770872042 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f4a7808bbd0 con 0x7f4a8c160a50 2024-04-26T23:03:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.210+0000 7f4a8aa00640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a6c06c920 0x7f4a6c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:02.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.211+0000 7f4a8aa00640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a6c06c920 0x7f4a6c06ede0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c034000 tx=0x7f4a7c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:02.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.214+0000 7f4a82a00640 1 -- 172.21.15.40:0/3770872042 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4a78059dd0 con 0x7f4a8c160a50 2024-04-26T23:03:02.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.467+0000 7f4a8be00640 1 -- 172.21.15.40:0/3770872042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f4a8c06fa60 con 0x7f4a8c160a50 2024-04-26T23:03:02.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.508+0000 7f4a82a00640 1 -- 172.21.15.40:0/3770872042 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f4a780596d0 con 0x7f4a8c160a50 2024-04-26T23:03:02.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.512+0000 7f4a8be00640 1 -- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a6c06c920 msgr2=0x7f4a6c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:02.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.512+0000 7f4a8be00640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a6c06c920 0x7f4a6c06ede0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c034000 tx=0x7f4a7c02e000 comp rx=0 tx=0).stop 2024-04-26T23:03:02.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.512+0000 7f4a8be00640 1 -- 172.21.15.40:0/3770872042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a8c160a50 msgr2=0x7f4a8c15a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:02.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.512+0000 7f4a8be00640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a8c160a50 0x7f4a8c15a610 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800c430 tx=0x7f4a7800c8f0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.513+0000 7f4a8be00640 1 -- 172.21.15.40:0/3770872042 shutdown_connections 2024-04-26T23:03:02.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.513+0000 7f4a8be00640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a6c06c920 0x7f4a6c06ede0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.513+0000 7f4a8be00640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a8c15c4c0 0x7f4a8c15c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.513+0000 7f4a8be00640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a8c160a50 0x7f4a8c15a610 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.513+0000 7f4a8be00640 1 --2- 172.21.15.40:0/3770872042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a8c06dd40 0x7f4a8c143480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.513+0000 7f4a8be00640 1 -- 172.21.15.40:0/3770872042 >> 172.21.15.40:0/3770872042 conn(0x7f4a8c0b2720 msgr2=0x7f4a8c074ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:02.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.513+0000 7f4a8be00640 1 -- 172.21.15.40:0/3770872042 shutdown_connections 2024-04-26T23:03:02.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.513+0000 7f4a8be00640 1 -- 172.21.15.40:0/3770872042 wait complete. 2024-04-26T23:03:02.538 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:03:02.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.714+0000 7f4330200640 1 Processor -- start 2024-04-26T23:03:02.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.715+0000 7f4330200640 1 -- start start 2024-04-26T23:03:02.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.716+0000 7f4330200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4328073f30 0x7f43280cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.716+0000 7f4330200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43281671e0 0x7f4328167670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.717+0000 7f4330200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4328163a50 con 0x7f43280c41d0 2024-04-26T23:03:02.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.717+0000 7f4330200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f43280cf7a0 con 0x7f4328073f30 2024-04-26T23:03:02.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.717+0000 7f4330200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f43280ceb90 con 0x7f43281671e0 2024-04-26T23:03:02.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.717+0000 7f432d800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f43280c41d0 0x7f43280cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42886/0 (socket says 172.21.15.40:42886) 2024-04-26T23:03:02.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.717+0000 7f432d800640 1 -- 172.21.15.40:0/306304562 learned_addr learned my addr 172.21.15.40:0/306304562 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.717+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1714755420 0 0) 0x7f4328163a50 con 0x7f43280c41d0 2024-04-26T23:03:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.717+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4310003520 con 0x7f43280c41d0 2024-04-26T23:03:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.717+0000 7f432e200640 1 --2- 172.21.15.40:0/306304562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43281671e0 0x7f4328167670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.717+0000 7f432ce00640 1 --2- 172.21.15.40:0/306304562 >> v2:172.21.15.103:3300/0 conn(0x7f4328073f30 0x7f43280cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1829055015 0 0) 0x7f4310003520 con 0x7f43280c41d0 2024-04-26T23:03:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43281671e0 msgr2=0x7f4328167670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 --2- 172.21.15.40:0/306304562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43281671e0 0x7f4328167670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 >> v2:172.21.15.103:3300/0 conn(0x7f4328073f30 msgr2=0x7f43280cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 --2- 172.21.15.40:0/306304562 >> v2:172.21.15.103:3300/0 conn(0x7f4328073f30 0x7f43280cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43280cee70 con 0x7f43280c41d0 2024-04-26T23:03:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f431c001690 con 0x7f43280c41d0 2024-04-26T23:03:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 >> v1:172.21.15.40:6789/0 conn(0x7f43280c41d0 legacy=0x7f43280cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 --2- 172.21.15.40:0/306304562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43100039b0 0x7f4310003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 --2- 172.21.15.40:0/306304562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4310004800 0x7f4310004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 --2- 172.21.15.40:0/306304562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43281671e0 0x7f43100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f43280cf7a0 con 0x7f4310004800 2024-04-26T23:03:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f43280ceb90 con 0x7f43100039b0 2024-04-26T23:03:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f431000ba70 con 0x7f43281671e0 2024-04-26T23:03:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f432ce00640 1 --2- 172.21.15.40:0/306304562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4310004800 0x7f4310004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f432ce00640 1 -- 172.21.15.40:0/306304562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43281671e0 msgr2=0x7f43100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.718+0000 7f432ce00640 1 --2- 172.21.15.40:0/306304562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43281671e0 0x7f43100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.719+0000 7f432ce00640 1 -- 172.21.15.40:0/306304562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43100039b0 msgr2=0x7f4310003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:02.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.719+0000 7f432ce00640 1 --2- 172.21.15.40:0/306304562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43100039b0 0x7f4310003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.719+0000 7f432ce00640 1 -- 172.21.15.40:0/306304562 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4310003520 con 0x7f4310004800 2024-04-26T23:03:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.720+0000 7f432ce00640 1 --2- 172.21.15.40:0/306304562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4310004800 0x7f4310004c80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4328074310 tx=0x7f4310005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8022080ef3f80eaa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.720+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4318030070 con 0x7f4310004800 2024-04-26T23:03:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.720+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f431802ada0 con 0x7f4310004800 2024-04-26T23:03:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.720+0000 7f4327e00640 1 -- 172.21.15.40:0/306304562 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431800b610 con 0x7f4310004800 2024-04-26T23:03:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.720+0000 7f4330200640 1 -- 172.21.15.40:0/306304562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4310004800 msgr2=0x7f4310004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.720+0000 7f4330200640 1 --2- 172.21.15.40:0/306304562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4310004800 0x7f4310004c80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4328074310 tx=0x7f4310005830 comp rx=0 tx=0).stop 2024-04-26T23:03:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.720+0000 7f432e200640 1 -- 172.21.15.40:0/306304562 reap_dead start 2024-04-26T23:03:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.721+0000 7f4330200640 1 -- 172.21.15.40:0/306304562 shutdown_connections 2024-04-26T23:03:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.721+0000 7f4330200640 1 -- 172.21.15.40:0/306304562 >> 172.21.15.40:0/306304562 conn(0x7f43280d0be0 msgr2=0x7f43280d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:02.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.721+0000 7f4330200640 1 -- 172.21.15.40:0/306304562 shutdown_connections 2024-04-26T23:03:02.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.721+0000 7f4330200640 1 -- 172.21.15.40:0/306304562 wait complete. 2024-04-26T23:03:02.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.722+0000 7f4330200640 1 Processor -- start 2024-04-26T23:03:02.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.723+0000 7f4330200640 1 -- start start 2024-04-26T23:03:02.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.723+0000 7f4330200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4310004800 0x7f432815eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f4330200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4328161360 0x7f4328151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f4330200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4328160650 0x7f4328160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f4330200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f43280ce7b0 con 0x7f4328160650 2024-04-26T23:03:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f4330200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f43280cf430 con 0x7f4310004800 2024-04-26T23:03:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f4330200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f43280ce930 con 0x7f4328161360 2024-04-26T23:03:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f432e200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4328160650 0x7f4328160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f432e200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4328160650 0x7f4328160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36888/0 (socket says 172.21.15.40:36888) 2024-04-26T23:03:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f432e200640 1 -- 172.21.15.40:0/1009203587 learned_addr learned my addr 172.21.15.40:0/1009203587 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f432ce00640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4328161360 0x7f4328151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f432d800640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4310004800 0x7f432815eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f432e200640 1 -- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4328161360 msgr2=0x7f4328151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f432e200640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4328161360 0x7f4328151010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f432e200640 1 -- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4310004800 msgr2=0x7f432815eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f432e200640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4310004800 0x7f432815eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.724+0000 7f432e200640 1 -- 172.21.15.40:0/1009203587 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432815f0e0 con 0x7f4328160650 2024-04-26T23:03:02.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.725+0000 7f432e200640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4328160650 0x7f4328160ad0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f43280b6d50 tx=0x7f432000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:02.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.725+0000 7f4325600640 1 -- 172.21.15.40:0/1009203587 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432000eb90 con 0x7f4328160650 2024-04-26T23:03:02.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.725+0000 7f4325600640 1 -- 172.21.15.40:0/1009203587 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f432000ed50 con 0x7f4328160650 2024-04-26T23:03:02.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.725+0000 7f4325600640 1 -- 172.21.15.40:0/1009203587 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320003bd0 con 0x7f4328160650 2024-04-26T23:03:02.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.725+0000 7f4330200640 1 -- 172.21.15.40:0/1009203587 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43281f9a20 con 0x7f4328160650 2024-04-26T23:03:02.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.725+0000 7f4330200640 1 -- 172.21.15.40:0/1009203587 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43280b48b0 con 0x7f4328160650 2024-04-26T23:03:02.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.726+0000 7f4330200640 1 -- 172.21.15.40:0/1009203587 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43280764d0 con 0x7f4328160650 2024-04-26T23:03:02.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.727+0000 7f4325600640 1 -- 172.21.15.40:0/1009203587 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f432001b020 con 0x7f4328160650 2024-04-26T23:03:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.729+0000 7f4325600640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f430c06c8f0 0x7f430c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.729+0000 7f4325600640 1 -- 172.21.15.40:0/1009203587 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f432008bab0 con 0x7f4328160650 2024-04-26T23:03:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.729+0000 7f432d800640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f430c06c8f0 0x7f430c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.731+0000 7f432d800640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f430c06c8f0 0x7f430c06edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f432806c410 tx=0x7f431c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:02.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.733+0000 7f4325600640 1 -- 172.21.15.40:0/1009203587 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f43280764d0 con 0x7f4328160650 2024-04-26T23:03:02.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.980+0000 7f4330200640 1 -- 172.21.15.40:0/1009203587 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43280c71b0 con 0x7f4328160650 2024-04-26T23:03:02.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.982+0000 7f4325600640 1 -- 172.21.15.40:0/1009203587 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5194 (secure 0 0 0) 0x7f4320059cb0 con 0x7f4328160650 2024-04-26T23:03:02.982 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:03:02.982 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1979956236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1979956236},{"type":"v1","addr":"172.21.15.103:6837","nonce":1979956236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4975,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/603576312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":603576312},{"type":"v1","addr":"172.21.15.103:6836","nonce":603576312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:02:47.924696+0000","modified":"2024-04-26T23:03:02.512885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1],"up":{"mds_0":5334,"mds_1":5331},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5331":{"gid":5331,"name":"c","rank":1,"incarnation":20,"state":"up:creating","state_seq":1,"addr":"172.21.15.40:6834/716981348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":716981348},{"type":"v1","addr":"172.21.15.40:6834","nonce":716981348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5334":{"gid":5334,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.40:6835/299800334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":299800334},{"type":"v1","addr":"172.21.15.40:6835","nonce":299800334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5334,"qdb_cluster":[5334]},"id":2}]} 2024-04-26T23:03:02.983 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 20 2024-04-26T23:03:02.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.984+0000 7f4330200640 1 -- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f430c06c8f0 msgr2=0x7f430c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:02.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.984+0000 7f4330200640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f430c06c8f0 0x7f430c06edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f432806c410 tx=0x7f431c009040 comp rx=0 tx=0).stop 2024-04-26T23:03:02.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.984+0000 7f4330200640 1 -- 172.21.15.40:0/1009203587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4328160650 msgr2=0x7f4328160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:02.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.984+0000 7f4330200640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4328160650 0x7f4328160ad0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f43280b6d50 tx=0x7f432000e4b0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.985+0000 7f4330200640 1 -- 172.21.15.40:0/1009203587 shutdown_connections 2024-04-26T23:03:02.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.985+0000 7f4330200640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f430c06c8f0 0x7f430c06edb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.985+0000 7f4330200640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4328160650 0x7f4328160ad0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.985+0000 7f4330200640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4328161360 0x7f4328151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.985+0000 7f4330200640 1 --2- 172.21.15.40:0/1009203587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4310004800 0x7f432815eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:02.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.985+0000 7f4330200640 1 -- 172.21.15.40:0/1009203587 >> 172.21.15.40:0/1009203587 conn(0x7f43280d0be0 msgr2=0x7f432806f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:02.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.985+0000 7f4330200640 1 -- 172.21.15.40:0/1009203587 shutdown_connections 2024-04-26T23:03:02.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:02.985+0000 7f4330200640 1 -- 172.21.15.40:0/1009203587 wait complete. 2024-04-26T23:03:03.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:02:47.924696+0000', 'modified': '2024-04-26T23:03:02.512885+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 5334, 'mds_1': 5331}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5331': {'gid': 5331, 'name': 'c', 'rank': 1, 'incarnation': 20, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.40:6834/716981348', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 716981348}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 716981348}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_5334': {'gid': 5334, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.40:6835/299800334', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 299800334}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 299800334}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5334, 'qdb_cluster': [5334]} 2024-04-26T23:03:03.407 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5331:up:creating 2024-04-26T23:03:04.409 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:03:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.614+0000 7ff5e7200640 1 Processor -- start 2024-04-26T23:03:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.614+0000 7ff5e7200640 1 -- start start 2024-04-26T23:03:04.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.616+0000 7ff5e7200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff5e80b6220 0x7ff5e80d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:04.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.616+0000 7ff5e7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5e80d08d0 0x7ff5e80d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:04.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.616+0000 7ff5e7200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5e8162c40 con 0x7ff5e80d3820 2024-04-26T23:03:04.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.616+0000 7ff5e7200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff5e80d2d80 con 0x7ff5e80b6220 2024-04-26T23:03:04.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.616+0000 7ff5e7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e80b7c80 con 0x7ff5e80d08d0 2024-04-26T23:03:04.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.616+0000 7ff5e6800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff5e80d3820 0x7ff5e80b6810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42898/0 (socket says 172.21.15.40:42898) 2024-04-26T23:03:04.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.616+0000 7ff5e6800640 1 -- 172.21.15.40:0/3355351359 learned_addr learned my addr 172.21.15.40:0/3355351359 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:04.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.617+0000 7ff5e5400640 1 --2- 172.21.15.40:0/3355351359 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5e80d08d0 0x7ff5e80d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:04.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.617+0000 7ff5e5e00640 1 --2- 172.21.15.40:0/3355351359 >> v2:172.21.15.103:3300/0 conn(0x7ff5e80b6220 0x7ff5e80d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:04.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.617+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1611733399 0 0) 0x7ff5e8162c40 con 0x7ff5e80d3820 2024-04-26T23:03:04.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.617+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5cc003520 con 0x7ff5e80d3820 2024-04-26T23:03:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.617+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2032585652 0 0) 0x7ff5cc003520 con 0x7ff5e80d3820 2024-04-26T23:03:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.617+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5e80d08d0 msgr2=0x7ff5e80d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.617+0000 7ff5e4a00640 1 --2- 172.21.15.40:0/3355351359 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5e80d08d0 0x7ff5e80d32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.617+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 >> v2:172.21.15.103:3300/0 conn(0x7ff5e80b6220 msgr2=0x7ff5e80d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.617+0000 7ff5e4a00640 1 --2- 172.21.15.40:0/3355351359 >> v2:172.21.15.103:3300/0 conn(0x7ff5e80b6220 0x7ff5e80d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.617+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e80b7f60 con 0x7ff5e80d3820 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.617+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7ff5e00019f0 con 0x7ff5e80d3820 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.617+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 >> v1:172.21.15.40:6789/0 conn(0x7ff5e80d3820 legacy=0x7ff5e80b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e4a00640 1 --2- 172.21.15.40:0/3355351359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5cc0039b0 0x7ff5cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e4a00640 1 --2- 172.21.15.40:0/3355351359 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5e80d08d0 0x7ff5cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e4a00640 1 --2- 172.21.15.40:0/3355351359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5cc004d40 0x7ff5cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e80d2d80 con 0x7ff5cc0039b0 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e80b7c80 con 0x7ff5cc004d40 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff5cc00b9c0 con 0x7ff5e80d08d0 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e5e00640 1 --2- 172.21.15.40:0/3355351359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5cc0039b0 0x7ff5cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e6800640 1 --2- 172.21.15.40:0/3355351359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5cc004d40 0x7ff5cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e5400640 1 --2- 172.21.15.40:0/3355351359 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5e80d08d0 0x7ff5cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e5e00640 1 -- 172.21.15.40:0/3355351359 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5e80d08d0 msgr2=0x7ff5cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e5e00640 1 --2- 172.21.15.40:0/3355351359 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5e80d08d0 0x7ff5cc004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e5e00640 1 -- 172.21.15.40:0/3355351359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5cc004d40 msgr2=0x7ff5cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e5e00640 1 --2- 172.21.15.40:0/3355351359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5cc004d40 0x7ff5cc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.618+0000 7ff5e5e00640 1 -- 172.21.15.40:0/3355351359 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5cc003520 con 0x7ff5cc0039b0 2024-04-26T23:03:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.619+0000 7ff5e5e00640 1 --2- 172.21.15.40:0/3355351359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5cc0039b0 0x7ff5cc003dd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8009ac0 tx=0x7ff5cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2522fc54a33cd751 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.620+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8036070 con 0x7ff5cc0039b0 2024-04-26T23:03:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.620+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5d8031400 con 0x7ff5cc0039b0 2024-04-26T23:03:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.620+0000 7ff5e4a00640 1 -- 172.21.15.40:0/3355351359 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8009690 con 0x7ff5cc0039b0 2024-04-26T23:03:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.620+0000 7ff5e7200640 1 -- 172.21.15.40:0/3355351359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5cc0039b0 msgr2=0x7ff5cc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.620+0000 7ff5e7200640 1 --2- 172.21.15.40:0/3355351359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5cc0039b0 0x7ff5cc003dd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8009ac0 tx=0x7ff5cc005830 comp rx=0 tx=0).stop 2024-04-26T23:03:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.620+0000 7ff5e6800640 1 -- 172.21.15.40:0/3355351359 reap_dead start 2024-04-26T23:03:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.620+0000 7ff5e7200640 1 -- 172.21.15.40:0/3355351359 shutdown_connections 2024-04-26T23:03:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.620+0000 7ff5e7200640 1 -- 172.21.15.40:0/3355351359 >> 172.21.15.40:0/3355351359 conn(0x7ff5e80d4d90 msgr2=0x7ff5e80d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.620+0000 7ff5e7200640 1 -- 172.21.15.40:0/3355351359 shutdown_connections 2024-04-26T23:03:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.620+0000 7ff5e7200640 1 -- 172.21.15.40:0/3355351359 wait complete. 2024-04-26T23:03:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.622+0000 7ff5e7200640 1 Processor -- start 2024-04-26T23:03:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.622+0000 7ff5e7200640 1 -- start start 2024-04-26T23:03:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.623+0000 7ff5e7200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5cc0039b0 0x7ff5e8154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.623+0000 7ff5e7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5e8151810 0x7ff5e80b8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5e8152e70 0x7ff5e8154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e7200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e80b9d30 con 0x7ff5cc0039b0 2024-04-26T23:03:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e80b8520 con 0x7ff5e8151810 2024-04-26T23:03:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e80b9eb0 con 0x7ff5e8152e70 2024-04-26T23:03:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5cc0039b0 0x7ff5e8154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5cc0039b0 0x7ff5e8154260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36908/0 (socket says 172.21.15.40:36908) 2024-04-26T23:03:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e5e00640 1 -- 172.21.15.40:0/3446823892 learned_addr learned my addr 172.21.15.40:0/3446823892 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e5400640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5e8151810 0x7ff5e80b8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e6800640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5e8152e70 0x7ff5e8154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e5e00640 1 -- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5e8152e70 msgr2=0x7ff5e8154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e5e00640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5e8152e70 0x7ff5e8154d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e5e00640 1 -- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5e8151810 msgr2=0x7ff5e80b8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e5e00640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5e8151810 0x7ff5e80b8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.624+0000 7ff5e5e00640 1 -- 172.21.15.40:0/3446823892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e81547a0 con 0x7ff5cc0039b0 2024-04-26T23:03:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.625+0000 7ff5e5e00640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5cc0039b0 0x7ff5e8154260 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8034040 tx=0x7ff5d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.625+0000 7ff5de000640 1 -- 172.21.15.40:0/3446823892 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8040020 con 0x7ff5cc0039b0 2024-04-26T23:03:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.625+0000 7ff5de000640 1 -- 172.21.15.40:0/3446823892 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5d8003950 con 0x7ff5cc0039b0 2024-04-26T23:03:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.625+0000 7ff5e7200640 1 -- 172.21.15.40:0/3446823892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e8151dd0 con 0x7ff5cc0039b0 2024-04-26T23:03:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.625+0000 7ff5de000640 1 -- 172.21.15.40:0/3446823892 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8036040 con 0x7ff5cc0039b0 2024-04-26T23:03:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.625+0000 7ff5e7200640 1 -- 172.21.15.40:0/3446823892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e8152390 con 0x7ff5cc0039b0 2024-04-26T23:03:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.627+0000 7ff5e7200640 1 -- 172.21.15.40:0/3446823892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e8069370 con 0x7ff5cc0039b0 2024-04-26T23:03:04.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.628+0000 7ff5de000640 1 -- 172.21.15.40:0/3446823892 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff5d8043020 con 0x7ff5cc0039b0 2024-04-26T23:03:04.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.628+0000 7ff5de000640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5c806c920 0x7ff5c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:04.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.629+0000 7ff5de000640 1 -- 172.21.15.40:0/3446823892 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7ff5d80ac030 con 0x7ff5cc0039b0 2024-04-26T23:03:04.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.629+0000 7ff5e5400640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5c806c920 0x7ff5c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:04.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.630+0000 7ff5e5400640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5c806c920 0x7ff5c806ede0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80a26d0 tx=0x7ff5d4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:04.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.637+0000 7ff5de000640 1 -- 172.21.15.40:0/3446823892 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff5d804a030 con 0x7ff5cc0039b0 2024-04-26T23:03:04.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.876+0000 7ff5e7200640 1 -- 172.21.15.40:0/3446823892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5e8074360 con 0x7ff5cc0039b0 2024-04-26T23:03:04.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.877+0000 7ff5de000640 1 -- 172.21.15.40:0/3446823892 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5216 (secure 0 0 0) 0x7ff5d807a360 con 0x7ff5cc0039b0 2024-04-26T23:03:04.877 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:03:04.877 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1979956236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1979956236},{"type":"v1","addr":"172.21.15.103:6837","nonce":1979956236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:02:47.924696+0000","modified":"2024-04-26T23:03:04.527692+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5334,"mds_1":5331,"mds_2":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"b","rank":2,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.103:6836/603576312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":603576312},{"type":"v1","addr":"172.21.15.103:6836","nonce":603576312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5331":{"gid":5331,"name":"c","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6834/716981348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":716981348},{"type":"v1","addr":"172.21.15.40:6834","nonce":716981348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5334":{"gid":5334,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.40:6835/299800334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":299800334},{"type":"v1","addr":"172.21.15.40:6835","nonce":299800334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5334,"qdb_cluster":[4975,5331,5334]},"id":2}]} 2024-04-26T23:03:04.877 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 23 2024-04-26T23:03:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.879+0000 7ff5e7200640 1 -- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5c806c920 msgr2=0x7ff5c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.879+0000 7ff5e7200640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5c806c920 0x7ff5c806ede0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80a26d0 tx=0x7ff5d4008040 comp rx=0 tx=0).stop 2024-04-26T23:03:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.879+0000 7ff5e7200640 1 -- 172.21.15.40:0/3446823892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5cc0039b0 msgr2=0x7ff5e8154260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.879+0000 7ff5e7200640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5cc0039b0 0x7ff5e8154260 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8034040 tx=0x7ff5d8004730 comp rx=0 tx=0).stop 2024-04-26T23:03:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.879+0000 7ff5e7200640 1 -- 172.21.15.40:0/3446823892 shutdown_connections 2024-04-26T23:03:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.879+0000 7ff5e7200640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff5c806c920 0x7ff5c806ede0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.879+0000 7ff5e7200640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5e8152e70 0x7ff5e8154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.879+0000 7ff5e7200640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5e8151810 0x7ff5e80b8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.879+0000 7ff5e7200640 1 --2- 172.21.15.40:0/3446823892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5cc0039b0 0x7ff5e8154260 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.879+0000 7ff5e7200640 1 -- 172.21.15.40:0/3446823892 >> 172.21.15.40:0/3446823892 conn(0x7ff5e80d4d90 msgr2=0x7ff5e80d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.879+0000 7ff5e7200640 1 -- 172.21.15.40:0/3446823892 shutdown_connections 2024-04-26T23:03:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:04.879+0000 7ff5e7200640 1 -- 172.21.15.40:0/3446823892 wait complete. 2024-04-26T23:03:05.301 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:02:47.924696+0000', 'modified': '2024-04-26T23:03:04.527692+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 5334, 'mds_1': 5331, 'mds_2': 4975}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4975': {'gid': 4975, 'name': 'b', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.103:6836/603576312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6834', 'nonce': 603576312}, {'type': 'v1', 'addr': '172.21.15.103:6836', 'nonce': 603576312}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_5331': {'gid': 5331, 'name': 'c', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.40:6834/716981348', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 716981348}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 716981348}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_5334': {'gid': 5334, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.40:6835/299800334', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 299800334}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 299800334}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5334, 'qdb_cluster': [4975, 5331, 5334]} 2024-04-26T23:03:05.301 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T23:03:05.301 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T23:03:05.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.499+0000 7f2387200640 1 Processor -- start 2024-04-26T23:03:05.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.499+0000 7f2387200640 1 -- start start 2024-04-26T23:03:05.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.501+0000 7f2387200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f23880b6220 0x7f23880d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.501+0000 7f2387200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23880d39f0 0x7f23880b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.501+0000 7f2387200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2388162c40 con 0x7f23880d08d0 2024-04-26T23:03:05.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.501+0000 7f2387200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f23880d2d80 con 0x7f23880b6220 2024-04-26T23:03:05.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.501+0000 7f2387200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f23880b7c80 con 0x7f23880d39f0 2024-04-26T23:03:05.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.501+0000 7f2385400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f23880d08d0 0x7f23880d32e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37672/0 (socket says 172.21.15.40:37672) 2024-04-26T23:03:05.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.501+0000 7f2385400640 1 -- 172.21.15.40:0/3590188578 learned_addr learned my addr 172.21.15.40:0/3590188578 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:05.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.501+0000 7f2385e00640 1 --2- 172.21.15.40:0/3590188578 >> v2:172.21.15.40:3300/0 conn(0x7f23880b6220 0x7f23880d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.502+0000 7f2386800640 1 --2- 172.21.15.40:0/3590188578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23880d39f0 0x7f23880b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.502+0000 7f2385e00640 1 -- 172.21.15.40:0/3590188578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23880d39f0 msgr2=0x7f23880b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.502+0000 7f2385e00640 1 --2- 172.21.15.40:0/3590188578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23880d39f0 0x7f23880b8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.502+0000 7f2385e00640 1 -- 172.21.15.40:0/3590188578 >> v1:172.21.15.40:6789/0 conn(0x7f23880d08d0 legacy=0x7f23880d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.502+0000 7f2385e00640 1 -- 172.21.15.40:0/3590188578 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23880b7f60 con 0x7f23880b6220 2024-04-26T23:03:05.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.502+0000 7f2384a00640 1 -- 172.21.15.40:0/3590188578 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2915214568 0 0) 0x7f2388162c40 con 0x7f23880d08d0 2024-04-26T23:03:05.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.503+0000 7f2385e00640 1 --2- 172.21.15.40:0/3590188578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23880b6220 0x7f23880d0390 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f23780097a0 tx=0x7f237802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40dce397c88c52f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.503+0000 7f2384a00640 1 -- 172.21.15.40:0/3590188578 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378037070 con 0x7f23880b6220 2024-04-26T23:03:05.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.503+0000 7f2384a00640 1 -- 172.21.15.40:0/3590188578 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2378032440 con 0x7f23880b6220 2024-04-26T23:03:05.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.503+0000 7f2384a00640 1 -- 172.21.15.40:0/3590188578 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378032810 con 0x7f23880b6220 2024-04-26T23:03:05.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.504+0000 7f2387200640 1 -- 172.21.15.40:0/3590188578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23880b6220 msgr2=0x7f23880d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.504+0000 7f2387200640 1 --2- 172.21.15.40:0/3590188578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23880b6220 0x7f23880d0390 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f23780097a0 tx=0x7f237802a180 comp rx=0 tx=0).stop 2024-04-26T23:03:05.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.504+0000 7f2387200640 1 -- 172.21.15.40:0/3590188578 shutdown_connections 2024-04-26T23:03:05.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.504+0000 7f2387200640 1 --2- 172.21.15.40:0/3590188578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23880d39f0 0x7f23880b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.504+0000 7f2387200640 1 --2- 172.21.15.40:0/3590188578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23880b6220 0x7f23880d0390 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.504+0000 7f2387200640 1 -- 172.21.15.40:0/3590188578 >> 172.21.15.40:0/3590188578 conn(0x7f23880d4d90 msgr2=0x7f23880d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:05.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.504+0000 7f2387200640 1 -- 172.21.15.40:0/3590188578 shutdown_connections 2024-04-26T23:03:05.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.504+0000 7f2387200640 1 -- 172.21.15.40:0/3590188578 wait complete. 2024-04-26T23:03:05.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.506+0000 7f2387200640 1 Processor -- start 2024-04-26T23:03:05.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.506+0000 7f2387200640 1 -- start start 2024-04-26T23:03:05.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.507+0000 7f2387200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23880b6220 0x7f23880b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.507+0000 7f2387200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23880d08d0 0x7f2388152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.507+0000 7f2387200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23880d39f0 0x7f23880b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.507+0000 7f2387200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f23880b9260 con 0x7f23880d39f0 2024-04-26T23:03:05.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.507+0000 7f2387200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f23880b8520 con 0x7f23880b6220 2024-04-26T23:03:05.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.507+0000 7f2387200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f23880b93e0 con 0x7f23880d08d0 2024-04-26T23:03:05.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.507+0000 7f2386800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23880d39f0 0x7f23880b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.507+0000 7f2386800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23880d39f0 0x7f23880b6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51508/0 (socket says 172.21.15.40:51508) 2024-04-26T23:03:05.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.507+0000 7f2386800640 1 -- 172.21.15.40:0/2524958886 learned_addr learned my addr 172.21.15.40:0/2524958886 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:05.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.508+0000 7f2385400640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23880d08d0 0x7f2388152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.508+0000 7f2385e00640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23880b6220 0x7f23880b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.508+0000 7f2386800640 1 -- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23880d08d0 msgr2=0x7f2388152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.508+0000 7f2386800640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23880d08d0 0x7f2388152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.508+0000 7f2386800640 1 -- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23880b6220 msgr2=0x7f23880b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.508+0000 7f2386800640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23880b6220 0x7f23880b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.508+0000 7f2386800640 1 -- 172.21.15.40:0/2524958886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238815f860 con 0x7f23880d39f0 2024-04-26T23:03:05.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.508+0000 7f2386800640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23880d39f0 0x7f23880b6810 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2380007d80 tx=0x7f238000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.509+0000 7f237e000640 1 -- 172.21.15.40:0/2524958886 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2380014420 con 0x7f23880d39f0 2024-04-26T23:03:05.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.509+0000 7f237e000640 1 -- 172.21.15.40:0/2524958886 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2380019070 con 0x7f23880d39f0 2024-04-26T23:03:05.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.509+0000 7f237e000640 1 -- 172.21.15.40:0/2524958886 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2380014800 con 0x7f23880d39f0 2024-04-26T23:03:05.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.509+0000 7f2387200640 1 -- 172.21.15.40:0/2524958886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23880b1bd0 con 0x7f23880d39f0 2024-04-26T23:03:05.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.509+0000 7f2387200640 1 -- 172.21.15.40:0/2524958886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23880b2190 con 0x7f23880d39f0 2024-04-26T23:03:05.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.512+0000 7f237e000640 1 -- 172.21.15.40:0/2524958886 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f23800149a0 con 0x7f23880d39f0 2024-04-26T23:03:05.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.513+0000 7f237e000640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f236806c8f0 0x7f236806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.513+0000 7f237e000640 1 -- 172.21.15.40:0/2524958886 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f238008c560 con 0x7f23880d39f0 2024-04-26T23:03:05.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.513+0000 7f2390171740 1 Processor -- start 2024-04-26T23:03:05.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.513+0000 7f2390171740 1 -- start start 2024-04-26T23:03:05.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.514+0000 7f2385e00640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f236806c8f0 0x7f236806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.514+0000 7f2390171740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f7ea910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.514+0000 7f2390171740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f824e80 0x55d70f7e8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.514+0000 7f2390171740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f813270 0x55d70f8136f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.514+0000 7f2390171740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d70f74c600 con 0x55d70f7e87b0 2024-04-26T23:03:05.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.514+0000 7f2390171740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d70f6e9100 con 0x55d70f824e80 2024-04-26T23:03:05.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.514+0000 7f2390171740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d70f7485c0 con 0x55d70f813270 2024-04-26T23:03:05.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.515+0000 7f2385e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f7ea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.515+0000 7f2385e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f7ea910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51524/0 (socket says 172.21.15.40:51524) 2024-04-26T23:03:05.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.515+0000 7f2385e00640 1 -- 172.21.15.40:0/4165328985 learned_addr learned my addr 172.21.15.40:0/4165328985 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:05.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.515+0000 7f2386800640 1 --2- 172.21.15.40:0/4165328985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f813270 0x55d70f8136f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.515+0000 7f2385400640 1 --2- 172.21.15.40:0/4165328985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f824e80 0x55d70f7e8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.515+0000 7f2385e00640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f236806c8f0 0x7f236806edb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2378034000 tx=0x7f237802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.515+0000 7f2385e00640 1 -- 172.21.15.40:0/4165328985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f813270 msgr2=0x55d70f8136f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.515+0000 7f2385e00640 1 --2- 172.21.15.40:0/4165328985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f813270 0x55d70f8136f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.515+0000 7f2385e00640 1 -- 172.21.15.40:0/4165328985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f824e80 msgr2=0x55d70f7e8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.515+0000 7f2385e00640 1 --2- 172.21.15.40:0/4165328985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f824e80 0x55d70f7e8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.515+0000 7f2385e00640 1 -- 172.21.15.40:0/4165328985 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d70f45e2d0 con 0x55d70f7e87b0 2024-04-26T23:03:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.515+0000 7f2385400640 1 --2- 172.21.15.40:0/4165328985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f824e80 0x55d70f7e8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:03:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.516+0000 7f2385e00640 1 --2- 172.21.15.40:0/4165328985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f7ea910 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f23780097a0 tx=0x7f2378008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d402c2e0b4fca6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.516+0000 7f2372a00640 1 -- 172.21.15.40:0/4165328985 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237804f070 con 0x55d70f7e87b0 2024-04-26T23:03:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.516+0000 7f2372a00640 1 -- 172.21.15.40:0/4165328985 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2378038040 con 0x55d70f7e87b0 2024-04-26T23:03:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.516+0000 7f2372a00640 1 -- 172.21.15.40:0/4165328985 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378037050 con 0x55d70f7e87b0 2024-04-26T23:03:05.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.517+0000 7f2390171740 1 -- 172.21.15.40:0/4165328985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 msgr2=0x55d70f7ea910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.517+0000 7f2390171740 1 --2- 172.21.15.40:0/4165328985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f7ea910 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f23780097a0 tx=0x7f2378008030 comp rx=0 tx=0).stop 2024-04-26T23:03:05.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.517+0000 7f2390171740 1 -- 172.21.15.40:0/4165328985 shutdown_connections 2024-04-26T23:03:05.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.517+0000 7f2390171740 1 --2- 172.21.15.40:0/4165328985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f813270 0x55d70f8136f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.517+0000 7f2390171740 1 --2- 172.21.15.40:0/4165328985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f824e80 0x55d70f7e8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.517+0000 7f2390171740 1 --2- 172.21.15.40:0/4165328985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f7ea910 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.517+0000 7f2390171740 1 -- 172.21.15.40:0/4165328985 >> 172.21.15.40:0/4165328985 conn(0x55d70f77cd10 msgr2=0x55d70f82e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:05.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.517+0000 7f2390171740 1 -- 172.21.15.40:0/4165328985 shutdown_connections 2024-04-26T23:03:05.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.517+0000 7f2390171740 1 -- 172.21.15.40:0/4165328985 wait complete. 2024-04-26T23:03:05.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.517+0000 7f2390171740 1 -- start start 2024-04-26T23:03:05.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.518+0000 7f2390171740 1 Processor -- start 2024-04-26T23:03:05.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2390171740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f802b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2390171740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f813270 0x55d70f816870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2390171740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f824e80 0x55d70f812370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2390171740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d70f6d3f70 con 0x55d70f7e87b0 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2390171740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d70f810dc0 con 0x55d70f813270 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2390171740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d70f83f120 con 0x55d70f824e80 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2390171740 10 client.0 _ll_register_callbacks cb 0x55d70f742480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2385e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f802b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2385e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f802b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51528/0 (socket says 172.21.15.40:51528) 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2385e00640 1 -- 172.21.15.40:0/3585332556 learned_addr learned my addr 172.21.15.40:0/3585332556 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2385e00640 1 -- 172.21.15.40:0/3585332556 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f824e80 msgr2=0x55d70f812370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2385e00640 1 --2- 172.21.15.40:0/3585332556 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f824e80 0x55d70f812370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2385e00640 1 -- 172.21.15.40:0/3585332556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f813270 msgr2=0x55d70f816870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2385e00640 1 --2- 172.21.15.40:0/3585332556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f813270 0x55d70f816870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.519+0000 7f2385e00640 1 -- 172.21.15.40:0/3585332556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d70f5286b0 con 0x55d70f7e87b0 2024-04-26T23:03:05.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.520+0000 7f2385e00640 1 --2- 172.21.15.40:0/3585332556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f802b00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f23780098c0 tx=0x7f2378004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea165a711ac1b9ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.520+0000 7f2367400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.520+0000 7f2367400640 1 -- 172.21.15.40:0/3585332556 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378058060 con 0x55d70f7e87b0 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.520+0000 7f2367400640 1 -- 172.21.15.40:0/3585332556 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f2378032850 con 0x55d70f7e87b0 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.520+0000 7f2390171740 1 -- 172.21.15.40:0/3585332556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2378038040 con 0x55d70f7e87b0 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.520+0000 7f2367400640 1 -- 172.21.15.40:0/3585332556 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f237805e050 con 0x55d70f7e87b0 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.520+0000 7f2390171740 10 client.5577 fetch_fsmap learned FSMap version 23 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.520+0000 7f2390171740 1 -- 172.21.15.40:0/3585332556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55d70f8128b0 con 0x55d70f7e87b0 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.521+0000 7f2367400640 1 -- 172.21.15.40:0/3585332556 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2378038040 con 0x55d70f7e87b0 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.521+0000 7f2367400640 1 -- 172.21.15.40:0/3585332556 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) ==== 3534+0+0 (secure 0 0 0) 0x7f2378061020 con 0x55d70f7e87b0 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.521+0000 7f2390171740 10 client.5577 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.521+0000 7f2390171740 10 client.5577 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.521+0000 7f2390171740 20 client.5577 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.521+0000 7f2390171740 1 --2- 172.21.15.40:0/3585332556 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55d70f80f5b0 0x55d70f83df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.521+0000 7f2390171740 4 client.5577 mds_command: new command op to 4975 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:03:05.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.521+0000 7f2390171740 1 -- 172.21.15.40:0/3585332556 --> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d70f75a210 con 0x55d70f80f5b0 2024-04-26T23:03:05.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.522+0000 7f2385400640 1 --2- 172.21.15.40:0/3585332556 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55d70f80f5b0 0x55d70f83df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.523+0000 7f2385400640 1 --2- 172.21.15.40:0/3585332556 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55d70f80f5b0 0x55d70f83df70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ce65a4332c6af3f1 server_cookie=630612dd3b6a38b3 in_seq=0 out_seq=0 2024-04-26T23:03:05.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.523+0000 7f2367400640 10 client.5577 ms_handle_connect on v2:172.21.15.103:6834/603576312 2024-04-26T23:03:05.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.524+0000 7f2367400640 1 -- 172.21.15.40:0/3585332556 <== mds.2 v2:172.21.15.103:6834/603576312 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d70f75a210 con 0x55d70f80f5b0 2024-04-26T23:03:05.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.524+0000 7f2367400640 10 client.5577 handle_command_reply: tid=0 2024-04-26T23:03:05.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.525+0000 7f2390171740 1 client.5577 shutdown 2024-04-26T23:03:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.525+0000 7f2390171740 1 -- 172.21.15.40:0/3585332556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 msgr2=0x55d70f802b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.525+0000 7f2390171740 1 --2- 172.21.15.40:0/3585332556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f802b00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f23780098c0 tx=0x7f2378004730 comp rx=0 tx=0).stop 2024-04-26T23:03:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.525+0000 7f2390171740 1 -- 172.21.15.40:0/3585332556 shutdown_connections 2024-04-26T23:03:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.525+0000 7f2390171740 1 --2- 172.21.15.40:0/3585332556 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55d70f80f5b0 0x55d70f83df70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.525+0000 7f2390171740 1 --2- 172.21.15.40:0/3585332556 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f824e80 0x55d70f812370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.525+0000 7f2390171740 1 --2- 172.21.15.40:0/3585332556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f813270 0x55d70f816870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.525+0000 7f2390171740 1 --2- 172.21.15.40:0/3585332556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f802b00 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.525+0000 7f2367400640 0 client.5577 ms_handle_reset on v2:172.21.15.103:6834/603576312 2024-04-26T23:03:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.525+0000 7f2390171740 1 -- 172.21.15.40:0/3585332556 >> 172.21.15.40:0/3585332556 conn(0x55d70f77cd10 msgr2=0x55d70f810fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.525+0000 7f2390171740 1 -- 172.21.15.40:0/3585332556 shutdown_connections 2024-04-26T23:03:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.525+0000 7f2390171740 1 -- 172.21.15.40:0/3585332556 wait complete. 2024-04-26T23:03:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.525+0000 7f2390171740 20 client.5577 trim_cache size 0 max 16384 2024-04-26T23:03:05.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.545+0000 7f2390171740 1 Processor -- start 2024-04-26T23:03:05.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.545+0000 7f2390171740 1 -- start start 2024-04-26T23:03:05.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2390171740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f801ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2390171740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f813270 0x55d70f83d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2390171740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f824e80 0x55d70f815c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2390171740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2388200c00 con 0x55d70f7e87b0 2024-04-26T23:03:05.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2390171740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f23882012d0 con 0x55d70f813270 2024-04-26T23:03:05.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2390171740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f23881ffc90 con 0x55d70f824e80 2024-04-26T23:03:05.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2385e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f801ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2385e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f801ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51536/0 (socket says 172.21.15.40:51536) 2024-04-26T23:03:05.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2385e00640 1 -- 172.21.15.40:0/3953653700 learned_addr learned my addr 172.21.15.40:0/3953653700 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:05.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2385400640 1 --2- 172.21.15.40:0/3953653700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f813270 0x55d70f83d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2385e00640 1 -- 172.21.15.40:0/3953653700 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f824e80 msgr2=0x55d70f815c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2386800640 1 --2- 172.21.15.40:0/3953653700 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f824e80 0x55d70f815c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2385e00640 1 --2- 172.21.15.40:0/3953653700 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f824e80 0x55d70f815c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2385e00640 1 -- 172.21.15.40:0/3953653700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f813270 msgr2=0x55d70f83d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2385e00640 1 --2- 172.21.15.40:0/3953653700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f813270 0x55d70f83d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.546+0000 7f2385e00640 1 -- 172.21.15.40:0/3953653700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d70f830480 con 0x55d70f7e87b0 2024-04-26T23:03:05.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.547+0000 7f2385e00640 1 --2- 172.21.15.40:0/3953653700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f801ca0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f23780096a0 tx=0x7f2378041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2880281349b809b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.547+0000 7f2372000640 1 -- 172.21.15.40:0/3953653700 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237804f070 con 0x55d70f7e87b0 2024-04-26T23:03:05.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.547+0000 7f2372000640 1 -- 172.21.15.40:0/3953653700 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2378032de0 con 0x55d70f7e87b0 2024-04-26T23:03:05.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.547+0000 7f2372000640 1 -- 172.21.15.40:0/3953653700 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378037040 con 0x55d70f7e87b0 2024-04-26T23:03:05.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.548+0000 7f2390171740 1 -- 172.21.15.40:0/3953653700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 msgr2=0x55d70f801ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.548+0000 7f2390171740 1 --2- 172.21.15.40:0/3953653700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f801ca0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f23780096a0 tx=0x7f2378041040 comp rx=0 tx=0).stop 2024-04-26T23:03:05.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.548+0000 7f2390171740 1 -- 172.21.15.40:0/3953653700 shutdown_connections 2024-04-26T23:03:05.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.548+0000 7f2390171740 1 --2- 172.21.15.40:0/3953653700 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f824e80 0x55d70f815c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.548+0000 7f2390171740 1 --2- 172.21.15.40:0/3953653700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f813270 0x55d70f83d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.548+0000 7f2390171740 1 --2- 172.21.15.40:0/3953653700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f801ca0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.548+0000 7f2390171740 1 -- 172.21.15.40:0/3953653700 >> 172.21.15.40:0/3953653700 conn(0x55d70f77cd10 msgr2=0x55d70f83def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:05.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.548+0000 7f2390171740 1 -- 172.21.15.40:0/3953653700 shutdown_connections 2024-04-26T23:03:05.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.548+0000 7f2390171740 1 -- 172.21.15.40:0/3953653700 wait complete. 2024-04-26T23:03:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.549+0000 7f2390171740 1 -- start start 2024-04-26T23:03:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.549+0000 7f2390171740 1 Processor -- start 2024-04-26T23:03:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2390171740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f800840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2390171740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f813270 0x55d70f812290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2390171740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f824e80 0x55d70f7e6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2390171740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d70f80f430 con 0x55d70f7e87b0 2024-04-26T23:03:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2390171740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d70f6d3f70 con 0x55d70f813270 2024-04-26T23:03:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2390171740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d70f80fa70 con 0x55d70f824e80 2024-04-26T23:03:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2390171740 10 client.0 _ll_register_callbacks cb 0x7f237804d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2385e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f800840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2385e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f800840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51546/0 (socket says 172.21.15.40:51546) 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2385e00640 1 -- 172.21.15.40:0/1651031335 learned_addr learned my addr 172.21.15.40:0/1651031335 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2385e00640 1 -- 172.21.15.40:0/1651031335 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f824e80 msgr2=0x55d70f7e6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2385e00640 1 --2- 172.21.15.40:0/1651031335 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f824e80 0x55d70f7e6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2385e00640 1 -- 172.21.15.40:0/1651031335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f813270 msgr2=0x55d70f812290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2385e00640 1 --2- 172.21.15.40:0/1651031335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f813270 0x55d70f812290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.550+0000 7f2385e00640 1 -- 172.21.15.40:0/1651031335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d70f8012f0 con 0x55d70f7e87b0 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.551+0000 7f2385e00640 1 --2- 172.21.15.40:0/1651031335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f800840 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f23780098c0 tx=0x7f2378004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9c89f953dbe5db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.551+0000 7f2367400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.551+0000 7f2367400640 1 -- 172.21.15.40:0/1651031335 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378037040 con 0x55d70f7e87b0 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.551+0000 7f2367400640 1 -- 172.21.15.40:0/1651031335 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f2378032a60 con 0x55d70f7e87b0 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.551+0000 7f2390171740 1 -- 172.21.15.40:0/1651031335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2378032de0 con 0x55d70f7e87b0 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.551+0000 7f2367400640 1 -- 172.21.15.40:0/1651031335 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2378006040 con 0x55d70f7e87b0 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.551+0000 7f2390171740 10 client.5583 fetch_fsmap learned FSMap version 23 2024-04-26T23:03:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.551+0000 7f2390171740 1 -- 172.21.15.40:0/1651031335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55d70f7e7570 con 0x55d70f7e87b0 2024-04-26T23:03:05.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.552+0000 7f2367400640 1 -- 172.21.15.40:0/1651031335 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2378032de0 con 0x55d70f7e87b0 2024-04-26T23:03:05.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.552+0000 7f2367400640 1 -- 172.21.15.40:0/1651031335 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) ==== 3534+0+0 (secure 0 0 0) 0x7f237805b020 con 0x55d70f7e87b0 2024-04-26T23:03:05.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.552+0000 7f2390171740 10 client.5583 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:03:05.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.552+0000 7f2390171740 10 client.5583 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:03:05.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.552+0000 7f2390171740 20 client.5583 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:05.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.552+0000 7f2390171740 1 --2- 172.21.15.40:0/1651031335 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55d70f815e20 0x55d70f816200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.552+0000 7f2390171740 4 client.5583 mds_command: new command op to 4975 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:03:05.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.552+0000 7f2390171740 1 -- 172.21.15.40:0/1651031335 --> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] -- command(tid 0: {"prefix": "status"}) -- 0x55d70f80df20 con 0x55d70f815e20 2024-04-26T23:03:05.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.552+0000 7f2385400640 1 --2- 172.21.15.40:0/1651031335 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55d70f815e20 0x55d70f816200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.553+0000 7f2385400640 1 --2- 172.21.15.40:0/1651031335 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55d70f815e20 0x55d70f816200 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=34af967d14c43079 server_cookie=9d6b4ffdeaf68670 in_seq=0 out_seq=0 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.553+0000 7f2367400640 10 client.5583 ms_handle_connect on v2:172.21.15.103:6834/603576312 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2367400640 1 -- 172.21.15.40:0/1651031335 <== mds.2 v2:172.21.15.103:6834/603576312 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55d70f80df20 con 0x55d70f815e20 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2367400640 10 client.5583 handle_command_reply: tid=0 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2390171740 1 client.5583 shutdown 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2390171740 1 -- 172.21.15.40:0/1651031335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 msgr2=0x55d70f800840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2390171740 1 --2- 172.21.15.40:0/1651031335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f800840 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f23780098c0 tx=0x7f2378004730 comp rx=0 tx=0).stop 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2390171740 1 -- 172.21.15.40:0/1651031335 shutdown_connections 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2390171740 1 --2- 172.21.15.40:0/1651031335 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55d70f815e20 0x55d70f816200 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2390171740 1 --2- 172.21.15.40:0/1651031335 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d70f824e80 0x55d70f7e6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2390171740 1 --2- 172.21.15.40:0/1651031335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d70f813270 0x55d70f812290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2367400640 0 client.5583 ms_handle_reset on v2:172.21.15.103:6834/603576312 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2390171740 1 --2- 172.21.15.40:0/1651031335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d70f7e87b0 0x55d70f800840 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2390171740 1 -- 172.21.15.40:0/1651031335 >> 172.21.15.40:0/1651031335 conn(0x55d70f77cd10 msgr2=0x55d70f82fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:05.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2390171740 1 -- 172.21.15.40:0/1651031335 shutdown_connections 2024-04-26T23:03:05.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.554+0000 7f2390171740 1 -- 172.21.15.40:0/1651031335 wait complete. 2024-04-26T23:03:05.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.555+0000 7f2390171740 20 client.5583 trim_cache size 0 max 16384 2024-04-26T23:03:05.557 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:03:05.557 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:03:05.557 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-04-26T23:03:05.557 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4975, 2024-04-26T23:03:05.557 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:03:05.557 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:03:05.557 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:03:05.558 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 2.0245213679999998, 2024-04-26T23:03:05.558 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 23, 2024-04-26T23:03:05.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 39, 2024-04-26T23:03:05.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 39, 2024-04-26T23:03:05.558 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 25.977131286999999 2024-04-26T23:03:05.558 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:03:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.555+0000 7f2387200640 1 -- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f236806c8f0 msgr2=0x7f236806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.555+0000 7f2387200640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f236806c8f0 0x7f236806edb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2378034000 tx=0x7f237802e000 comp rx=0 tx=0).stop 2024-04-26T23:03:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.555+0000 7f2387200640 1 -- 172.21.15.40:0/2524958886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23880d39f0 msgr2=0x7f23880b6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.555+0000 7f2387200640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23880d39f0 0x7f23880b6810 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2380007d80 tx=0x7f238000e4b0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.555+0000 7f2387200640 1 -- 172.21.15.40:0/2524958886 shutdown_connections 2024-04-26T23:03:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.555+0000 7f2387200640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f236806c8f0 0x7f236806edb0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.555+0000 7f2387200640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23880d39f0 0x7f23880b6810 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.555+0000 7f2387200640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23880d08d0 0x7f2388152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.555+0000 7f2387200640 1 --2- 172.21.15.40:0/2524958886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23880b6220 0x7f23880b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.555+0000 7f2387200640 1 -- 172.21.15.40:0/2524958886 >> 172.21.15.40:0/2524958886 conn(0x7f23880d4d90 msgr2=0x7f23880d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.555+0000 7f2387200640 1 -- 172.21.15.40:0/2524958886 shutdown_connections 2024-04-26T23:03:05.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.555+0000 7f2387200640 1 -- 172.21.15.40:0/2524958886 wait complete. 2024-04-26T23:03:05.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T23:03:05.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.766+0000 7f2661a00640 1 Processor -- start 2024-04-26T23:03:05.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.767+0000 7f2661a00640 1 -- start start 2024-04-26T23:03:05.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.768+0000 7f2661a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f265c0d3740 0x7f265c0b9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.768+0000 7f2661a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f265c162c40 con 0x7f265c0b6220 2024-04-26T23:03:05.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.768+0000 7f2661a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f265c0d2d80 con 0x7f265c0d32e0 2024-04-26T23:03:05.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.768+0000 7f2661a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f265c0b7c80 con 0x7f265c0d3740 2024-04-26T23:03:05.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.769+0000 7f265be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f265c0b6220 0x7f265c0d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37676/0 (socket says 172.21.15.40:37676) 2024-04-26T23:03:05.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.769+0000 7f265be00640 1 -- 172.21.15.40:0/3738285541 learned_addr learned my addr 172.21.15.40:0/3738285541 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.769+0000 7f2661000640 1 --2- 172.21.15.40:0/3738285541 >> v2:172.21.15.40:3300/0 conn(0x7f265c0d3740 0x7f265c0b9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.769+0000 7f265aa00640 1 -- 172.21.15.40:0/3738285541 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182942172 0 0) 0x7f265c162c40 con 0x7f265c0b6220 2024-04-26T23:03:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.769+0000 7f265aa00640 1 -- 172.21.15.40:0/3738285541 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2640003520 con 0x7f265c0b6220 2024-04-26T23:03:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.769+0000 7f2661000640 1 -- 172.21.15.40:0/3738285541 >> v1:172.21.15.103:6789/0 conn(0x7f265c0d32e0 legacy=0x7f265c0b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.769+0000 7f265aa00640 1 -- 172.21.15.40:0/3738285541 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3522964342 0 0) 0x7f2640003520 con 0x7f265c0b6220 2024-04-26T23:03:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.769+0000 7f2661000640 1 -- 172.21.15.40:0/3738285541 >> v1:172.21.15.40:6789/0 conn(0x7f265c0b6220 legacy=0x7f265c0d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.770+0000 7f2661000640 1 -- 172.21.15.40:0/3738285541 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265c0b7f60 con 0x7f265c0d3740 2024-04-26T23:03:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.771+0000 7f2661000640 1 --2- 172.21.15.40:0/3738285541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f265c0d3740 0x7f265c0b9430 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f26540097a0 tx=0x7f265402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188a95c882b54048 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.771+0000 7f265aa00640 1 -- 172.21.15.40:0/3738285541 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2654037070 con 0x7f265c0d3740 2024-04-26T23:03:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.771+0000 7f265aa00640 1 -- 172.21.15.40:0/3738285541 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f265402ac50 con 0x7f265c0d3740 2024-04-26T23:03:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.771+0000 7f265aa00640 1 -- 172.21.15.40:0/3738285541 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c0019f0 con 0x7f265c0d3740 2024-04-26T23:03:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.771+0000 7f2661a00640 1 -- 172.21.15.40:0/3738285541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f265c0d3740 msgr2=0x7f265c0b9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.771+0000 7f2661a00640 1 --2- 172.21.15.40:0/3738285541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f265c0d3740 0x7f265c0b9430 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f26540097a0 tx=0x7f265402a1c0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.772+0000 7f2661a00640 1 -- 172.21.15.40:0/3738285541 shutdown_connections 2024-04-26T23:03:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.772+0000 7f2661a00640 1 --2- 172.21.15.40:0/3738285541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f265c0d3740 0x7f265c0b9430 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.772+0000 7f2661a00640 1 -- 172.21.15.40:0/3738285541 >> 172.21.15.40:0/3738285541 conn(0x7f265c0d4d90 msgr2=0x7f265c0d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.772+0000 7f2661a00640 1 -- 172.21.15.40:0/3738285541 shutdown_connections 2024-04-26T23:03:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.772+0000 7f2661a00640 1 -- 172.21.15.40:0/3738285541 wait complete. 2024-04-26T23:03:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.773+0000 7f2661a00640 1 Processor -- start 2024-04-26T23:03:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.774+0000 7f2661a00640 1 -- start start 2024-04-26T23:03:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.774+0000 7f2661a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f265c0b6220 0x7f265c151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.774+0000 7f2661a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f265c0d32e0 0x7f265c0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.774+0000 7f2661a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f265c0d3740 0x7f265c0b5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.774+0000 7f2661a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f265c0d3b80 con 0x7f265c0b6220 2024-04-26T23:03:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.774+0000 7f2661a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f265c0b8520 con 0x7f265c0d3740 2024-04-26T23:03:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.774+0000 7f2661a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f265c0d0aa0 con 0x7f265c0d32e0 2024-04-26T23:03:05.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.775+0000 7f265be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f265c0b6220 0x7f265c151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.775+0000 7f265be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f265c0b6220 0x7f265c151180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51574/0 (socket says 172.21.15.40:51574) 2024-04-26T23:03:05.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.775+0000 7f265be00640 1 -- 172.21.15.40:0/2904859365 learned_addr learned my addr 172.21.15.40:0/2904859365 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:05.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.775+0000 7f265be00640 1 -- 172.21.15.40:0/2904859365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f265c0d32e0 msgr2=0x7f265c0b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.775+0000 7f265be00640 1 --2- 172.21.15.40:0/2904859365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f265c0d32e0 0x7f265c0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.775+0000 7f2661000640 1 --2- 172.21.15.40:0/2904859365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f265c0d3740 0x7f265c0b5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.775+0000 7f265be00640 1 -- 172.21.15.40:0/2904859365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f265c0d3740 msgr2=0x7f265c0b5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.775+0000 7f265be00640 1 --2- 172.21.15.40:0/2904859365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f265c0d3740 0x7f265c0b5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.775+0000 7f265be00640 1 -- 172.21.15.40:0/2904859365 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265c0b39e0 con 0x7f265c0b6220 2024-04-26T23:03:05.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.776+0000 7f265be00640 1 --2- 172.21.15.40:0/2904859365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f265c0b6220 0x7f265c151180 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f264c001fb0 tx=0x7f264c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.776+0000 7f2653e00640 1 -- 172.21.15.40:0/2904859365 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c017070 con 0x7f265c0b6220 2024-04-26T23:03:05.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.776+0000 7f2653e00640 1 -- 172.21.15.40:0/2904859365 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f264c012400 con 0x7f265c0b6220 2024-04-26T23:03:05.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.776+0000 7f2653e00640 1 -- 172.21.15.40:0/2904859365 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c012830 con 0x7f265c0b6220 2024-04-26T23:03:05.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.776+0000 7f2661a00640 1 -- 172.21.15.40:0/2904859365 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f265c0b3cc0 con 0x7f265c0b6220 2024-04-26T23:03:05.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.776+0000 7f2661a00640 1 -- 172.21.15.40:0/2904859365 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f265c0b4280 con 0x7f265c0b6220 2024-04-26T23:03:05.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.778+0000 7f2653e00640 1 -- 172.21.15.40:0/2904859365 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f264c01a070 con 0x7f265c0b6220 2024-04-26T23:03:05.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.779+0000 7f2653e00640 1 --2- 172.21.15.40:0/2904859365 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f263806c8f0 0x7f263806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.779+0000 7f2653e00640 1 -- 172.21.15.40:0/2904859365 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f264c08bdb0 con 0x7f265c0b6220 2024-04-26T23:03:05.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.780+0000 7f2665e4c740 1 Processor -- start 2024-04-26T23:03:05.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.780+0000 7f2665e4c740 1 -- start start 2024-04-26T23:03:05.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.780+0000 7f265b400640 1 --2- 172.21.15.40:0/2904859365 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f263806c8f0 0x7f263806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.781+0000 7f2665e4c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eecb7b0 0x55966eecd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.781+0000 7f2665e4c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 0x55966eecbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.781+0000 7f2665e4c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eef6270 0x55966eef66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.781+0000 7f2665e4c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55966ee2f600 con 0x55966ef07e80 2024-04-26T23:03:05.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.781+0000 7f2665e4c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55966edcc100 con 0x55966eecb7b0 2024-04-26T23:03:05.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.781+0000 7f2665e4c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55966ee2b5c0 con 0x55966eef6270 2024-04-26T23:03:05.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.781+0000 7f265b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 0x55966eecbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.781+0000 7f265b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 0x55966eecbb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51576/0 (socket says 172.21.15.40:51576) 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.781+0000 7f265b400640 1 -- 172.21.15.40:0/3174525623 learned_addr learned my addr 172.21.15.40:0/3174525623 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.781+0000 7f265b400640 1 --2- 172.21.15.40:0/2904859365 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f263806c8f0 0x7f263806edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2644000f60 tx=0x7f26440079d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.781+0000 7f2661000640 1 --2- 172.21.15.40:0/3174525623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eef6270 0x55966eef66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.781+0000 7f265be00640 1 --2- 172.21.15.40:0/3174525623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eecb7b0 0x55966eecd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.781+0000 7f265b400640 1 -- 172.21.15.40:0/3174525623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eef6270 msgr2=0x55966eef66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.782+0000 7f265b400640 1 --2- 172.21.15.40:0/3174525623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eef6270 0x55966eef66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.782+0000 7f265b400640 1 -- 172.21.15.40:0/3174525623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eecb7b0 msgr2=0x55966eecd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.782+0000 7f265b400640 1 --2- 172.21.15.40:0/3174525623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eecb7b0 0x55966eecd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.782+0000 7f265b400640 1 -- 172.21.15.40:0/3174525623 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55966eb412d0 con 0x55966ef07e80 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.782+0000 7f265be00640 1 --2- 172.21.15.40:0/3174525623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eecb7b0 0x55966eecd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.782+0000 7f265b400640 1 --2- 172.21.15.40:0/3174525623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 0x55966eecbb90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f26440168b0 tx=0x7f264400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=543bf47c07717cd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.782+0000 7f2650c00640 1 -- 172.21.15.40:0/3174525623 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2644016e30 con 0x55966ef07e80 2024-04-26T23:03:05.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.782+0000 7f2650c00640 1 -- 172.21.15.40:0/3174525623 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26440050d0 con 0x55966ef07e80 2024-04-26T23:03:05.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.782+0000 7f2650c00640 1 -- 172.21.15.40:0/3174525623 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26440138b0 con 0x55966ef07e80 2024-04-26T23:03:05.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.783+0000 7f2665e4c740 1 -- 172.21.15.40:0/3174525623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 msgr2=0x55966eecbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.783+0000 7f2665e4c740 1 --2- 172.21.15.40:0/3174525623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 0x55966eecbb90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f26440168b0 tx=0x7f264400f000 comp rx=0 tx=0).stop 2024-04-26T23:03:05.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.783+0000 7f2665e4c740 1 -- 172.21.15.40:0/3174525623 shutdown_connections 2024-04-26T23:03:05.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.783+0000 7f2665e4c740 1 --2- 172.21.15.40:0/3174525623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eef6270 0x55966eef66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.783+0000 7f2665e4c740 1 --2- 172.21.15.40:0/3174525623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 0x55966eecbb90 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.783+0000 7f2665e4c740 1 --2- 172.21.15.40:0/3174525623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eecb7b0 0x55966eecd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.783+0000 7f2665e4c740 1 -- 172.21.15.40:0/3174525623 >> 172.21.15.40:0/3174525623 conn(0x55966ee5fd10 msgr2=0x55966ef11270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.783+0000 7f2665e4c740 1 -- 172.21.15.40:0/3174525623 shutdown_connections 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.783+0000 7f2665e4c740 1 -- 172.21.15.40:0/3174525623 wait complete. 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.784+0000 7f2665e4c740 1 -- start start 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.784+0000 7f2665e4c740 1 Processor -- start 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.785+0000 7f2665e4c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eecb7b0 0x55966eee2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.785+0000 7f2665e4c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eef6270 0x55966ef082c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.785+0000 7f2665e4c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966ef07e80 0x55966eee3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.785+0000 7f2665e4c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55966edb6f70 con 0x55966eef6270 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.785+0000 7f2665e4c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55966eef3dc0 con 0x55966ef07e80 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.785+0000 7f2665e4c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55966eee4090 con 0x55966eecb7b0 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.785+0000 7f2665e4c740 10 client.0 _ll_register_callbacks cb 0x55966ee25480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.785+0000 7f265b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eef6270 0x55966ef082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.785+0000 7f265b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eef6270 0x55966ef082c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51578/0 (socket says 172.21.15.40:51578) 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.785+0000 7f265b400640 1 -- 172.21.15.40:0/1823694127 learned_addr learned my addr 172.21.15.40:0/1823694127 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.786+0000 7f265b400640 1 -- 172.21.15.40:0/1823694127 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eecb7b0 msgr2=0x55966eee2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.786+0000 7f265b400640 1 --2- 172.21.15.40:0/1823694127 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eecb7b0 0x55966eee2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.786+0000 7f2661000640 1 --2- 172.21.15.40:0/1823694127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966ef07e80 0x55966eee3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.786+0000 7f265b400640 1 -- 172.21.15.40:0/1823694127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966ef07e80 msgr2=0x55966eee3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.786+0000 7f265b400640 1 --2- 172.21.15.40:0/1823694127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966ef07e80 0x55966eee3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.786+0000 7f265b400640 1 -- 172.21.15.40:0/1823694127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55966ec0b6b0 con 0x55966eef6270 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.786+0000 7f265b400640 1 --2- 172.21.15.40:0/1823694127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eef6270 0x55966ef082c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2644013780 tx=0x7f2644001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726feeb15b819170 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.786+0000 7f2649600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.786+0000 7f2649600640 1 -- 172.21.15.40:0/1823694127 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2644016e30 con 0x55966eef6270 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.786+0000 7f2665e4c740 1 -- 172.21.15.40:0/1823694127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26440050d0 con 0x55966eef6270 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.786+0000 7f2649600640 1 -- 172.21.15.40:0/1823694127 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f264401c040 con 0x55966eef6270 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.787+0000 7f2649600640 1 -- 172.21.15.40:0/1823694127 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2644031050 con 0x55966eef6270 2024-04-26T23:03:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.787+0000 7f2665e4c740 10 client.5598 fetch_fsmap learned FSMap version 23 2024-04-26T23:03:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.787+0000 7f2665e4c740 1 -- 172.21.15.40:0/1823694127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55966ef095b0 con 0x55966eef6270 2024-04-26T23:03:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.787+0000 7f2649600640 1 -- 172.21.15.40:0/1823694127 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26440050d0 con 0x55966eef6270 2024-04-26T23:03:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.787+0000 7f2649600640 1 -- 172.21.15.40:0/1823694127 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) ==== 3534+0+0 (secure 0 0 0) 0x7f2644034020 con 0x55966eef6270 2024-04-26T23:03:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.787+0000 7f2665e4c740 10 client.5598 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:03:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.787+0000 7f2665e4c740 10 client.5598 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:03:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.787+0000 7f2665e4c740 20 client.5598 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.787+0000 7f2665e4c740 1 --2- 172.21.15.40:0/1823694127 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x55966eee3950 0x55966eef6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.787+0000 7f2665e4c740 4 client.5598 mds_command: new command op to 5331 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:03:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.787+0000 7f2665e4c740 1 -- 172.21.15.40:0/1823694127 --> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55966ee3d210 con 0x55966eee3950 2024-04-26T23:03:05.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.787+0000 7f265be00640 1 --2- 172.21.15.40:0/1823694127 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x55966eee3950 0x55966eef6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.788+0000 7f265be00640 1 --2- 172.21.15.40:0/1823694127 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x55966eee3950 0x55966eef6760 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=645383f33cab2c65 server_cookie=a6b762f308a7ab71 in_seq=0 out_seq=0 2024-04-26T23:03:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.788+0000 7f2649600640 10 client.5598 ms_handle_connect on v2:172.21.15.40:6832/716981348 2024-04-26T23:03:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2649600640 1 -- 172.21.15.40:0/1823694127 <== mds.1 v2:172.21.15.40:6832/716981348 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55966ee3d210 con 0x55966eee3950 2024-04-26T23:03:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2649600640 10 client.5598 handle_command_reply: tid=0 2024-04-26T23:03:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2665e4c740 1 client.5598 shutdown 2024-04-26T23:03:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2665e4c740 1 -- 172.21.15.40:0/1823694127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eef6270 msgr2=0x55966ef082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2665e4c740 1 --2- 172.21.15.40:0/1823694127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eef6270 0x55966ef082c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2644013780 tx=0x7f2644001120 comp rx=0 tx=0).stop 2024-04-26T23:03:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2665e4c740 1 -- 172.21.15.40:0/1823694127 shutdown_connections 2024-04-26T23:03:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2665e4c740 1 --2- 172.21.15.40:0/1823694127 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x55966eee3950 0x55966eef6760 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2665e4c740 1 --2- 172.21.15.40:0/1823694127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966ef07e80 0x55966eee3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2665e4c740 1 --2- 172.21.15.40:0/1823694127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eef6270 0x55966ef082c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2665e4c740 1 --2- 172.21.15.40:0/1823694127 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eecb7b0 0x55966eee2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2649600640 0 client.5598 ms_handle_reset on v2:172.21.15.40:6832/716981348 2024-04-26T23:03:05.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2665e4c740 1 -- 172.21.15.40:0/1823694127 >> 172.21.15.40:0/1823694127 conn(0x55966ee5fd10 msgr2=0x55966eef41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:05.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2665e4c740 1 -- 172.21.15.40:0/1823694127 shutdown_connections 2024-04-26T23:03:05.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.789+0000 7f2665e4c740 1 -- 172.21.15.40:0/1823694127 wait complete. 2024-04-26T23:03:05.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.790+0000 7f2665e4c740 20 client.5598 trim_cache size 0 max 16384 2024-04-26T23:03:05.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.809+0000 7f2665e4c740 1 Processor -- start 2024-04-26T23:03:05.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.809+0000 7f2665e4c740 1 -- start start 2024-04-26T23:03:05.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.810+0000 7f2665e4c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eecb7b0 0x55966eecd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.810+0000 7f2665e4c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eef6270 0x55966eee2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.810+0000 7f2665e4c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966ef07e80 0x55966ef1f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.810+0000 7f2665e4c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f265c200c00 con 0x55966eecb7b0 2024-04-26T23:03:05.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.810+0000 7f2665e4c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f265c2012d0 con 0x55966eef6270 2024-04-26T23:03:05.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.810+0000 7f2665e4c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f265c152c80 con 0x55966ef07e80 2024-04-26T23:03:05.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.810+0000 7f265be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eecb7b0 0x55966eecd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.810+0000 7f265be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eecb7b0 0x55966eecd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51594/0 (socket says 172.21.15.40:51594) 2024-04-26T23:03:05.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.810+0000 7f265be00640 1 -- 172.21.15.40:0/226272414 learned_addr learned my addr 172.21.15.40:0/226272414 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:05.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.811+0000 7f265be00640 1 -- 172.21.15.40:0/226272414 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966ef07e80 msgr2=0x55966ef1f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.811+0000 7f265be00640 1 --2- 172.21.15.40:0/226272414 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966ef07e80 0x55966ef1f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.811+0000 7f265be00640 1 -- 172.21.15.40:0/226272414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eef6270 msgr2=0x55966eee2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.811+0000 7f265be00640 1 --2- 172.21.15.40:0/226272414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eef6270 0x55966eee2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.811+0000 7f265be00640 1 -- 172.21.15.40:0/226272414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55966eec2680 con 0x55966eecb7b0 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.811+0000 7f265be00640 1 --2- 172.21.15.40:0/226272414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eecb7b0 0x55966eecd790 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f264c00a930 tx=0x7f264c001db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4756cb0c88ff66e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.811+0000 7f264be00640 1 -- 172.21.15.40:0/226272414 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c08a170 con 0x55966eecb7b0 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.811+0000 7f264be00640 1 -- 172.21.15.40:0/226272414 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f264c055fb0 con 0x55966eecb7b0 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.811+0000 7f264be00640 1 -- 172.21.15.40:0/226272414 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c062090 con 0x55966eecb7b0 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.812+0000 7f2665e4c740 1 -- 172.21.15.40:0/226272414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eecb7b0 msgr2=0x55966eecd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.812+0000 7f2665e4c740 1 --2- 172.21.15.40:0/226272414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eecb7b0 0x55966eecd790 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f264c00a930 tx=0x7f264c001db0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.812+0000 7f2665e4c740 1 -- 172.21.15.40:0/226272414 shutdown_connections 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.812+0000 7f2665e4c740 1 --2- 172.21.15.40:0/226272414 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966ef07e80 0x55966ef1f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.812+0000 7f2665e4c740 1 --2- 172.21.15.40:0/226272414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eef6270 0x55966eee2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.812+0000 7f2665e4c740 1 --2- 172.21.15.40:0/226272414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966eecb7b0 0x55966eecd790 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.812+0000 7f2665e4c740 1 -- 172.21.15.40:0/226272414 >> 172.21.15.40:0/226272414 conn(0x55966ee5fd10 msgr2=0x55966eef13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:05.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.812+0000 7f2665e4c740 1 -- 172.21.15.40:0/226272414 shutdown_connections 2024-04-26T23:03:05.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.812+0000 7f2665e4c740 1 -- 172.21.15.40:0/226272414 wait complete. 2024-04-26T23:03:05.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.813+0000 7f2665e4c740 1 -- start start 2024-04-26T23:03:05.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.813+0000 7f2665e4c740 1 Processor -- start 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.814+0000 7f2665e4c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eecb7b0 0x55966ef23230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.814+0000 7f2665e4c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eef6270 0x55966ef14310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.814+0000 7f2665e4c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 0x55966ef14850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.814+0000 7f2665e4c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55966eee37d0 con 0x55966ef07e80 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.814+0000 7f2665e4c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55966eee3dd0 con 0x55966eef6270 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.814+0000 7f2665e4c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55966eef2af0 con 0x55966eecb7b0 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.814+0000 7f2665e4c740 10 client.0 _ll_register_callbacks cb 0x7f2644016750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.814+0000 7f2661000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 0x55966ef14850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.814+0000 7f2661000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 0x55966ef14850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51610/0 (socket says 172.21.15.40:51610) 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.814+0000 7f2661000640 1 -- 172.21.15.40:0/3759569594 learned_addr learned my addr 172.21.15.40:0/3759569594 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.815+0000 7f265be00640 1 --2- 172.21.15.40:0/3759569594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eecb7b0 0x55966ef23230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.815+0000 7f2661000640 1 -- 172.21.15.40:0/3759569594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eecb7b0 msgr2=0x55966ef23230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.815+0000 7f2661000640 1 --2- 172.21.15.40:0/3759569594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eecb7b0 0x55966ef23230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.815+0000 7f2661000640 1 -- 172.21.15.40:0/3759569594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eef6270 msgr2=0x55966ef14310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:05.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.815+0000 7f2661000640 1 --2- 172.21.15.40:0/3759569594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eef6270 0x55966ef14310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.815+0000 7f2661000640 1 -- 172.21.15.40:0/3759569594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55966ef23b10 con 0x55966ef07e80 2024-04-26T23:03:05.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.815+0000 7f2661000640 1 --2- 172.21.15.40:0/3759569594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 0x55966ef14850 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2654035040 tx=0x7f2654004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e7d03820919dea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:05.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.815+0000 7f2649600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:05.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2649600640 1 -- 172.21.15.40:0/3759569594 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2654043070 con 0x55966ef07e80 2024-04-26T23:03:05.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2649600640 1 -- 172.21.15.40:0/3759569594 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f2654032b70 con 0x55966ef07e80 2024-04-26T23:03:05.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2665e4c740 1 -- 172.21.15.40:0/3759569594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f264c055fb0 con 0x55966ef07e80 2024-04-26T23:03:05.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2649600640 1 -- 172.21.15.40:0/3759569594 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f265404e050 con 0x55966ef07e80 2024-04-26T23:03:05.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2665e4c740 10 client.5604 fetch_fsmap learned FSMap version 23 2024-04-26T23:03:05.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2665e4c740 1 -- 172.21.15.40:0/3759569594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55966eef0c20 con 0x55966ef07e80 2024-04-26T23:03:05.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2649600640 1 -- 172.21.15.40:0/3759569594 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f264c055fb0 con 0x55966ef07e80 2024-04-26T23:03:05.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2649600640 1 -- 172.21.15.40:0/3759569594 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) ==== 3534+0+0 (secure 0 0 0) 0x7f2654051020 con 0x55966ef07e80 2024-04-26T23:03:05.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2665e4c740 10 client.5604 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:03:05.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2665e4c740 10 client.5604 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2665e4c740 20 client.5604 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2665e4c740 1 --2- 172.21.15.40:0/3759569594 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x55966ef06c00 0x55966ef06fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2665e4c740 4 client.5604 mds_command: new command op to 5331 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.816+0000 7f2665e4c740 1 -- 172.21.15.40:0/3759569594 --> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] -- command(tid 0: {"prefix": "status"}) -- 0x55966ee37080 con 0x55966ef06c00 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.817+0000 7f265be00640 1 --2- 172.21.15.40:0/3759569594 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x55966ef06c00 0x55966ef06fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.817+0000 7f265be00640 1 --2- 172.21.15.40:0/3759569594 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x55966ef06c00 0x55966ef06fe0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1644d400a86223c9 server_cookie=e6f800322bcd6381 in_seq=0 out_seq=0 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.817+0000 7f2649600640 10 client.5604 ms_handle_connect on v2:172.21.15.40:6832/716981348 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.817+0000 7f2649600640 1 -- 172.21.15.40:0/3759569594 <== mds.1 v2:172.21.15.40:6832/716981348 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55966ee37080 con 0x55966ef06c00 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.817+0000 7f2649600640 10 client.5604 handle_command_reply: tid=0 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.817+0000 7f2665e4c740 1 client.5604 shutdown 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.818+0000 7f2665e4c740 1 -- 172.21.15.40:0/3759569594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 msgr2=0x55966ef14850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.818+0000 7f2665e4c740 1 --2- 172.21.15.40:0/3759569594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 0x55966ef14850 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2654035040 tx=0x7f2654004730 comp rx=0 tx=0).stop 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.818+0000 7f2665e4c740 1 -- 172.21.15.40:0/3759569594 shutdown_connections 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.818+0000 7f2665e4c740 1 --2- 172.21.15.40:0/3759569594 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x55966ef06c00 0x55966ef06fe0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.818+0000 7f2665e4c740 1 --2- 172.21.15.40:0/3759569594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55966ef07e80 0x55966ef14850 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.818+0000 7f2665e4c740 1 --2- 172.21.15.40:0/3759569594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55966eef6270 0x55966ef14310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.818+0000 7f2665e4c740 1 --2- 172.21.15.40:0/3759569594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55966eecb7b0 0x55966ef23230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.818+0000 7f2649600640 0 client.5604 ms_handle_reset on v2:172.21.15.40:6832/716981348 2024-04-26T23:03:05.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.818+0000 7f2665e4c740 1 -- 172.21.15.40:0/3759569594 >> 172.21.15.40:0/3759569594 conn(0x55966ee5fd10 msgr2=0x55966eec1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:05.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.818+0000 7f2665e4c740 1 -- 172.21.15.40:0/3759569594 shutdown_connections 2024-04-26T23:03:05.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.818+0000 7f2665e4c740 1 -- 172.21.15.40:0/3759569594 wait complete. 2024-04-26T23:03:05.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.818+0000 7f2665e4c740 20 client.5604 trim_cache size 0 max 16384 2024-04-26T23:03:05.821 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:03:05.821 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:03:05.821 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-04-26T23:03:05.821 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5331, 2024-04-26T23:03:05.821 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:03:05.821 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:03:05.822 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:03:05.822 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.3015150919999998, 2024-04-26T23:03:05.822 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 23, 2024-04-26T23:03:05.822 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 39, 2024-04-26T23:03:05.822 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 39, 2024-04-26T23:03:05.822 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 26.232463264 2024-04-26T23:03:05.822 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:03:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.819+0000 7f2661a00640 1 -- 172.21.15.40:0/2904859365 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f263806c8f0 msgr2=0x7f263806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.819+0000 7f2661a00640 1 --2- 172.21.15.40:0/2904859365 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f263806c8f0 0x7f263806edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2644000f60 tx=0x7f26440079d0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.819+0000 7f2661a00640 1 -- 172.21.15.40:0/2904859365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f265c0b6220 msgr2=0x7f265c151180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.819+0000 7f2661a00640 1 --2- 172.21.15.40:0/2904859365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f265c0b6220 0x7f265c151180 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f264c001fb0 tx=0x7f264c00a430 comp rx=0 tx=0).stop 2024-04-26T23:03:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.819+0000 7f2661a00640 1 -- 172.21.15.40:0/2904859365 shutdown_connections 2024-04-26T23:03:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.819+0000 7f2661a00640 1 --2- 172.21.15.40:0/2904859365 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f263806c8f0 0x7f263806edb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.819+0000 7f2661a00640 1 --2- 172.21.15.40:0/2904859365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f265c0d3740 0x7f265c0b5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.819+0000 7f2661a00640 1 --2- 172.21.15.40:0/2904859365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f265c0d32e0 0x7f265c0b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.819+0000 7f2661a00640 1 --2- 172.21.15.40:0/2904859365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f265c0b6220 0x7f265c151180 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:05.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.819+0000 7f2661a00640 1 -- 172.21.15.40:0/2904859365 >> 172.21.15.40:0/2904859365 conn(0x7f265c0d4d90 msgr2=0x7f265c0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:05.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.819+0000 7f2661a00640 1 -- 172.21.15.40:0/2904859365 shutdown_connections 2024-04-26T23:03:05.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:05.819+0000 7f2661a00640 1 -- 172.21.15.40:0/2904859365 wait complete. 2024-04-26T23:03:05.835 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T23:03:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.024+0000 7f29a2800640 1 Processor -- start 2024-04-26T23:03:06.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.024+0000 7f29a2800640 1 -- start start 2024-04-26T23:03:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.026+0000 7f29a2800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f299c0d3740 0x7f299c0b9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.026+0000 7f29a2800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f299c162c40 con 0x7f299c0d32e0 2024-04-26T23:03:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.026+0000 7f29a2800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f299c0d2d80 con 0x7f299c0b6220 2024-04-26T23:03:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.026+0000 7f29a2800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f299c0b7c80 con 0x7f299c0d3740 2024-04-26T23:03:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.026+0000 7f29a0a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f299c0d32e0 0x7f299c0b8d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37688/0 (socket says 172.21.15.40:37688) 2024-04-26T23:03:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.026+0000 7f29a0a00640 1 -- 172.21.15.40:0/2317806121 learned_addr learned my addr 172.21.15.40:0/2317806121 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.026+0000 7f29a1e00640 1 --2- 172.21.15.40:0/2317806121 >> v2:172.21.15.40:3300/0 conn(0x7f299c0d3740 0x7f299c0b9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.027+0000 7f299be00640 1 -- 172.21.15.40:0/2317806121 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3450136878 0 0) 0x7f299c162c40 con 0x7f299c0d32e0 2024-04-26T23:03:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.027+0000 7f299be00640 1 -- 172.21.15.40:0/2317806121 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2984003520 con 0x7f299c0d32e0 2024-04-26T23:03:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.027+0000 7f29a1e00640 1 -- 172.21.15.40:0/2317806121 >> v1:172.21.15.103:6789/0 conn(0x7f299c0b6220 legacy=0x7f299c0d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.027+0000 7f29a1e00640 1 -- 172.21.15.40:0/2317806121 >> v1:172.21.15.40:6789/0 conn(0x7f299c0d32e0 legacy=0x7f299c0b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.027+0000 7f29a1e00640 1 -- 172.21.15.40:0/2317806121 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299c0b7f60 con 0x7f299c0d3740 2024-04-26T23:03:06.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.028+0000 7f29a1e00640 1 --2- 172.21.15.40:0/2317806121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c0d3740 0x7f299c0b9430 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2994009d90 tx=0x7f299402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1b2edef4d805e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:06.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.028+0000 7f299be00640 1 -- 172.21.15.40:0/2317806121 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2994007740 con 0x7f299c0d3740 2024-04-26T23:03:06.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.028+0000 7f299be00640 1 -- 172.21.15.40:0/2317806121 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29940078e0 con 0x7f299c0d3740 2024-04-26T23:03:06.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.028+0000 7f299be00640 1 -- 172.21.15.40:0/2317806121 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2994007cb0 con 0x7f299c0d3740 2024-04-26T23:03:06.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.029+0000 7f29a2800640 1 -- 172.21.15.40:0/2317806121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c0d3740 msgr2=0x7f299c0b9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.029+0000 7f29a2800640 1 --2- 172.21.15.40:0/2317806121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c0d3740 0x7f299c0b9430 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2994009d90 tx=0x7f299402a6f0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.029+0000 7f29a2800640 1 -- 172.21.15.40:0/2317806121 shutdown_connections 2024-04-26T23:03:06.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.029+0000 7f29a2800640 1 --2- 172.21.15.40:0/2317806121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c0d3740 0x7f299c0b9430 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.029+0000 7f29a2800640 1 -- 172.21.15.40:0/2317806121 >> 172.21.15.40:0/2317806121 conn(0x7f299c0d4d90 msgr2=0x7f299c0d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:06.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.029+0000 7f29a2800640 1 -- 172.21.15.40:0/2317806121 shutdown_connections 2024-04-26T23:03:06.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.029+0000 7f29a2800640 1 -- 172.21.15.40:0/2317806121 wait complete. 2024-04-26T23:03:06.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.031+0000 7f29a2800640 1 Processor -- start 2024-04-26T23:03:06.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.031+0000 7f29a2800640 1 -- start start 2024-04-26T23:03:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.032+0000 7f29a2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c0b6220 0x7f299c1585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.032+0000 7f29a2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299c0d32e0 0x7f299c09ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.032+0000 7f29a2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f299c0d3740 0x7f299c160890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.032+0000 7f29a2800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f299c0d3b80 con 0x7f299c0b6220 2024-04-26T23:03:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.032+0000 7f29a2800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f299c0b8520 con 0x7f299c0d32e0 2024-04-26T23:03:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.032+0000 7f29a2800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f299c0d0aa0 con 0x7f299c0d3740 2024-04-26T23:03:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.032+0000 7f29a1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c0b6220 0x7f299c1585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.032+0000 7f29a1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c0b6220 0x7f299c1585f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51628/0 (socket says 172.21.15.40:51628) 2024-04-26T23:03:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.032+0000 7f29a1400640 1 -- 172.21.15.40:0/4189047950 learned_addr learned my addr 172.21.15.40:0/4189047950 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.033+0000 7f29a0a00640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299c0d32e0 0x7f299c09ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.033+0000 7f29a1400640 1 -- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f299c0d3740 msgr2=0x7f299c160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.033+0000 7f29a1e00640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f299c0d3740 0x7f299c160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.033+0000 7f29a1400640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f299c0d3740 0x7f299c160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.033+0000 7f29a1400640 1 -- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299c0d32e0 msgr2=0x7f299c09ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.033+0000 7f29a1400640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299c0d32e0 0x7f299c09ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.033+0000 7f29a1400640 1 -- 172.21.15.40:0/4189047950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299c158f50 con 0x7f299c0b6220 2024-04-26T23:03:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.033+0000 7f29a0a00640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299c0d32e0 0x7f299c09ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.033+0000 7f29a1400640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c0b6220 0x7f299c1585f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f299000c450 tx=0x7f299000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.033+0000 7f2999600640 1 -- 172.21.15.40:0/4189047950 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29900052d0 con 0x7f299c0b6220 2024-04-26T23:03:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.034+0000 7f2999600640 1 -- 172.21.15.40:0/4189047950 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2990005470 con 0x7f299c0b6220 2024-04-26T23:03:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.034+0000 7f29a2800640 1 -- 172.21.15.40:0/4189047950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299c159230 con 0x7f299c0b6220 2024-04-26T23:03:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.034+0000 7f2999600640 1 -- 172.21.15.40:0/4189047950 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990012870 con 0x7f299c0b6220 2024-04-26T23:03:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.034+0000 7f29a2800640 1 -- 172.21.15.40:0/4189047950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f299c1597f0 con 0x7f299c0b6220 2024-04-26T23:03:06.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.036+0000 7f2999600640 1 -- 172.21.15.40:0/4189047950 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2990012a10 con 0x7f299c0b6220 2024-04-26T23:03:06.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.037+0000 7f2999600640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f297c06c8f0 0x7f297c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.037+0000 7f2999600640 1 -- 172.21.15.40:0/4189047950 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f299008b8f0 con 0x7f299c0b6220 2024-04-26T23:03:06.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.037+0000 7f29a6ded740 1 Processor -- start 2024-04-26T23:03:06.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.038+0000 7f29a6ded740 1 -- start start 2024-04-26T23:03:06.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.038+0000 7f29a0a00640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f297c06c8f0 0x7f297c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.038+0000 7f29a6ded740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 0x555723975910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.038+0000 7f29a6ded740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5557239afe80 0x555723973b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.038+0000 7f29a6ded740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55572399e270 0x55572399e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.038+0000 7f29a6ded740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5557238d7600 con 0x5557239737b0 2024-04-26T23:03:06.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.038+0000 7f29a6ded740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555723874100 con 0x55572399e270 2024-04-26T23:03:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.038+0000 7f29a6ded740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5557238d35c0 con 0x5557239afe80 2024-04-26T23:03:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.039+0000 7f29a1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 0x555723975910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.039+0000 7f29a1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 0x555723975910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51642/0 (socket says 172.21.15.40:51642) 2024-04-26T23:03:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.039+0000 7f29a1400640 1 -- 172.21.15.40:0/3708121883 learned_addr learned my addr 172.21.15.40:0/3708121883 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.039+0000 7f29a0a00640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f297c06c8f0 0x7f297c06edb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f299c15f780 tx=0x7f2988009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.039+0000 7f29a0a00640 1 --2- 172.21.15.40:0/3708121883 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5557239afe80 0x555723973b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.039+0000 7f29a1e00640 1 --2- 172.21.15.40:0/3708121883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55572399e270 0x55572399e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.039+0000 7f29a1400640 1 -- 172.21.15.40:0/3708121883 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5557239afe80 msgr2=0x555723973b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.039+0000 7f29a1400640 1 --2- 172.21.15.40:0/3708121883 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5557239afe80 0x555723973b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.039+0000 7f29a1400640 1 -- 172.21.15.40:0/3708121883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55572399e270 msgr2=0x55572399e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.039+0000 7f29a1400640 1 --2- 172.21.15.40:0/3708121883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55572399e270 0x55572399e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.039+0000 7f29a1400640 1 -- 172.21.15.40:0/3708121883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557235e92d0 con 0x5557239737b0 2024-04-26T23:03:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.040+0000 7f29a1400640 1 --2- 172.21.15.40:0/3708121883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 0x555723975910 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f2990089bf0 tx=0x7f2990059480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca5a6fe4c574293 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.040+0000 7f298dc00640 1 -- 172.21.15.40:0/3708121883 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990057780 con 0x5557239737b0 2024-04-26T23:03:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.040+0000 7f298dc00640 1 -- 172.21.15.40:0/3708121883 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2990012d00 con 0x5557239737b0 2024-04-26T23:03:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.040+0000 7f298dc00640 1 -- 172.21.15.40:0/3708121883 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990061bd0 con 0x5557239737b0 2024-04-26T23:03:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.040+0000 7f29a6ded740 1 -- 172.21.15.40:0/3708121883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 msgr2=0x555723975910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.040+0000 7f29a6ded740 1 --2- 172.21.15.40:0/3708121883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 0x555723975910 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f2990089bf0 tx=0x7f2990059480 comp rx=0 tx=0).stop 2024-04-26T23:03:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.041+0000 7f29a6ded740 1 -- 172.21.15.40:0/3708121883 shutdown_connections 2024-04-26T23:03:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.041+0000 7f29a6ded740 1 --2- 172.21.15.40:0/3708121883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55572399e270 0x55572399e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.041+0000 7f29a6ded740 1 --2- 172.21.15.40:0/3708121883 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5557239afe80 0x555723973b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.041+0000 7f29a6ded740 1 --2- 172.21.15.40:0/3708121883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 0x555723975910 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.041+0000 7f29a6ded740 1 -- 172.21.15.40:0/3708121883 >> 172.21.15.40:0/3708121883 conn(0x555723907d10 msgr2=0x5557239b9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.041+0000 7f29a6ded740 1 -- 172.21.15.40:0/3708121883 shutdown_connections 2024-04-26T23:03:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.041+0000 7f29a6ded740 1 -- 172.21.15.40:0/3708121883 wait complete. 2024-04-26T23:03:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.041+0000 7f29a6ded740 1 -- start start 2024-04-26T23:03:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.042+0000 7f29a6ded740 1 Processor -- start 2024-04-26T23:03:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.042+0000 7f29a6ded740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5557239737b0 0x55572398ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a6ded740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55572399e270 0x5557239a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a6ded740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 0x5557239b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a6ded740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55572385ef70 con 0x5557239afe80 2024-04-26T23:03:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a6ded740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55572399bdc0 con 0x5557239737b0 2024-04-26T23:03:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a6ded740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5557239b6c30 con 0x55572399e270 2024-04-26T23:03:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a6ded740 10 client.0 _ll_register_callbacks cb 0x5557238cd480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 0x5557239b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 0x5557239b44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51658/0 (socket says 172.21.15.40:51658) 2024-04-26T23:03:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a1e00640 1 -- 172.21.15.40:0/2786869487 learned_addr learned my addr 172.21.15.40:0/2786869487 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a1400640 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5557239737b0 0x55572398ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a0a00640 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55572399e270 0x5557239a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a1e00640 1 -- 172.21.15.40:0/2786869487 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55572399e270 msgr2=0x5557239a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a1e00640 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55572399e270 0x5557239a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a1e00640 1 -- 172.21.15.40:0/2786869487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5557239737b0 msgr2=0x55572398ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a1e00640 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5557239737b0 0x55572398ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a1e00640 1 -- 172.21.15.40:0/2786869487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557236b36b0 con 0x5557239afe80 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a1400640 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5557239737b0 0x55572398ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.043+0000 7f29a0a00640 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55572399e270 0x5557239a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.044+0000 7f29a1e00640 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 0x5557239b44e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2994009eb0 tx=0x7f2994004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1b49bf16a1b4ef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.044+0000 7f2982000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.044+0000 7f2982000640 1 -- 172.21.15.40:0/2786869487 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2994009450 con 0x5557239afe80 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.044+0000 7f2982000640 1 -- 172.21.15.40:0/2786869487 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f2994046480 con 0x5557239afe80 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.044+0000 7f29a6ded740 1 -- 172.21.15.40:0/2786869487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2990012d00 con 0x5557239afe80 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.044+0000 7f2982000640 1 -- 172.21.15.40:0/2786869487 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f299404e050 con 0x5557239afe80 2024-04-26T23:03:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.044+0000 7f29a6ded740 10 client.5619 fetch_fsmap learned FSMap version 23 2024-04-26T23:03:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.044+0000 7f29a6ded740 1 -- 172.21.15.40:0/2786869487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x5557239b6780 con 0x5557239afe80 2024-04-26T23:03:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.045+0000 7f2982000640 1 -- 172.21.15.40:0/2786869487 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2990012d00 con 0x5557239afe80 2024-04-26T23:03:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.045+0000 7f2982000640 1 -- 172.21.15.40:0/2786869487 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) ==== 3534+0+0 (secure 0 0 0) 0x7f2994051020 con 0x5557239afe80 2024-04-26T23:03:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.045+0000 7f29a6ded740 10 client.5619 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:03:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.045+0000 7f29a6ded740 10 client.5619 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:03:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.045+0000 7f29a6ded740 20 client.5619 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.045+0000 7f29a6ded740 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x5557239cc850 0x5557239ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.045+0000 7f29a6ded740 4 client.5619 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:03:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.045+0000 7f29a6ded740 1 -- 172.21.15.40:0/2786869487 --> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557238e5210 con 0x5557239cc850 2024-04-26T23:03:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.045+0000 7f29a1400640 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x5557239cc850 0x5557239ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.046+0000 7f29a1400640 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x5557239cc850 0x5557239ccc30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bb6815206ef07d7 server_cookie=b8cfbc9839915858 in_seq=0 out_seq=0 2024-04-26T23:03:06.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.046+0000 7f2982000640 10 client.5619 ms_handle_connect on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:06.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.046+0000 7f2982000640 1 -- 172.21.15.40:0/2786869487 <== mds.0 v2:172.21.15.40:6833/299800334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5557238e5210 con 0x5557239cc850 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.046+0000 7f2982000640 10 client.5619 handle_command_reply: tid=0 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.046+0000 7f29a6ded740 1 client.5619 shutdown 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.047+0000 7f29a6ded740 1 -- 172.21.15.40:0/2786869487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 msgr2=0x5557239b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.047+0000 7f29a6ded740 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 0x5557239b44e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2994009eb0 tx=0x7f2994004ab0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.047+0000 7f29a6ded740 1 -- 172.21.15.40:0/2786869487 shutdown_connections 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.047+0000 7f29a6ded740 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x5557239cc850 0x5557239ccc30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.047+0000 7f29a6ded740 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 0x5557239b44e0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.047+0000 7f29a6ded740 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55572399e270 0x5557239a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.047+0000 7f29a6ded740 1 --2- 172.21.15.40:0/2786869487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5557239737b0 0x55572398ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.047+0000 7f2982000640 0 client.5619 ms_handle_reset on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.047+0000 7f29a6ded740 1 -- 172.21.15.40:0/2786869487 >> 172.21.15.40:0/2786869487 conn(0x555723907d10 msgr2=0x5557239bc760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.047+0000 7f29a6ded740 1 -- 172.21.15.40:0/2786869487 shutdown_connections 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.047+0000 7f29a6ded740 1 -- 172.21.15.40:0/2786869487 wait complete. 2024-04-26T23:03:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.047+0000 7f29a6ded740 20 client.5619 trim_cache size 0 max 16384 2024-04-26T23:03:06.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.067+0000 7f29a6ded740 1 Processor -- start 2024-04-26T23:03:06.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.067+0000 7f29a6ded740 1 -- start start 2024-04-26T23:03:06.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.068+0000 7f29a6ded740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 0x5557239dce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.068+0000 7f29a6ded740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55572399e270 0x55572399bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.068+0000 7f29a6ded740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5557239afe80 0x5557239b4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.068+0000 7f29a6ded740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f299c200d90 con 0x5557239737b0 2024-04-26T23:03:06.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.068+0000 7f29a6ded740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f299c201460 con 0x5557239afe80 2024-04-26T23:03:06.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.068+0000 7f29a6ded740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f299c15ce10 con 0x55572399e270 2024-04-26T23:03:06.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.068+0000 7f29a1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 0x5557239dce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.068+0000 7f29a1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 0x5557239dce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51662/0 (socket says 172.21.15.40:51662) 2024-04-26T23:03:06.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.068+0000 7f29a1400640 1 -- 172.21.15.40:0/2376212207 learned_addr learned my addr 172.21.15.40:0/2376212207 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:06.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.069+0000 7f29a1e00640 1 --2- 172.21.15.40:0/2376212207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5557239afe80 0x5557239b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.069+0000 7f29a0a00640 1 --2- 172.21.15.40:0/2376212207 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55572399e270 0x55572399bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.069+0000 7f29a1400640 1 -- 172.21.15.40:0/2376212207 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55572399e270 msgr2=0x55572399bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:06.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.069+0000 7f29a1400640 1 --2- 172.21.15.40:0/2376212207 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55572399e270 0x55572399bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.069+0000 7f29a1400640 1 -- 172.21.15.40:0/2376212207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5557239afe80 msgr2=0x5557239b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.069+0000 7f29a1400640 1 --2- 172.21.15.40:0/2376212207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5557239afe80 0x5557239b4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.069+0000 7f29a1400640 1 -- 172.21.15.40:0/2376212207 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555723971b10 con 0x5557239737b0 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.069+0000 7f29a1400640 1 --2- 172.21.15.40:0/2376212207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 0x5557239dce40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2990059410 tx=0x7f299005d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7639fa63d7e54e6c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.069+0000 7f298d200640 1 -- 172.21.15.40:0/2376212207 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990077070 con 0x5557239737b0 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.069+0000 7f298d200640 1 -- 172.21.15.40:0/2376212207 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2990012d00 con 0x5557239737b0 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.069+0000 7f298d200640 1 -- 172.21.15.40:0/2376212207 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990057780 con 0x5557239737b0 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.070+0000 7f29a6ded740 1 -- 172.21.15.40:0/2376212207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 msgr2=0x5557239dce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.070+0000 7f29a6ded740 1 --2- 172.21.15.40:0/2376212207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 0x5557239dce40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2990059410 tx=0x7f299005d000 comp rx=0 tx=0).stop 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.070+0000 7f29a6ded740 1 -- 172.21.15.40:0/2376212207 shutdown_connections 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.070+0000 7f29a6ded740 1 --2- 172.21.15.40:0/2376212207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5557239afe80 0x5557239b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.070+0000 7f29a6ded740 1 --2- 172.21.15.40:0/2376212207 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55572399e270 0x55572399bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.070+0000 7f29a6ded740 1 --2- 172.21.15.40:0/2376212207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239737b0 0x5557239dce40 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.070+0000 7f29a6ded740 1 -- 172.21.15.40:0/2376212207 >> 172.21.15.40:0/2376212207 conn(0x555723907d10 msgr2=0x555723969ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:06.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.070+0000 7f29a6ded740 1 -- 172.21.15.40:0/2376212207 shutdown_connections 2024-04-26T23:03:06.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.070+0000 7f29a6ded740 1 -- 172.21.15.40:0/2376212207 wait complete. 2024-04-26T23:03:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.071+0000 7f29a6ded740 1 -- start start 2024-04-26T23:03:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.071+0000 7f29a6ded740 1 Processor -- start 2024-04-26T23:03:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.072+0000 7f29a6ded740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5557239737b0 0x555723972b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.072+0000 7f29a6ded740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55572399e270 0x5557239b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.072+0000 7f29a6ded740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 0x5557239b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.072+0000 7f29a6ded740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5557239b4cf0 con 0x5557239afe80 2024-04-26T23:03:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.072+0000 7f29a6ded740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5557239b6a70 con 0x55572399e270 2024-04-26T23:03:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.072+0000 7f29a6ded740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55572399f960 con 0x5557239737b0 2024-04-26T23:03:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.072+0000 7f29a6ded740 10 client.0 _ll_register_callbacks cb 0x7f2994007920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.072+0000 7f29a1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 0x5557239b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.072+0000 7f29a1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 0x5557239b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51666/0 (socket says 172.21.15.40:51666) 2024-04-26T23:03:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.072+0000 7f29a1e00640 1 -- 172.21.15.40:0/4214109056 learned_addr learned my addr 172.21.15.40:0/4214109056 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:06.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.073+0000 7f29a0a00640 1 --2- 172.21.15.40:0/4214109056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55572399e270 0x5557239b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.073+0000 7f29a1400640 1 --2- 172.21.15.40:0/4214109056 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5557239737b0 0x555723972b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.073+0000 7f29a1e00640 1 -- 172.21.15.40:0/4214109056 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5557239737b0 msgr2=0x555723972b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.073+0000 7f29a1e00640 1 --2- 172.21.15.40:0/4214109056 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5557239737b0 0x555723972b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.073+0000 7f29a1e00640 1 -- 172.21.15.40:0/4214109056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55572399e270 msgr2=0x5557239b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.073+0000 7f29a1e00640 1 --2- 172.21.15.40:0/4214109056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55572399e270 0x5557239b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.073+0000 7f29a1e00640 1 -- 172.21.15.40:0/4214109056 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557239dd1a0 con 0x5557239afe80 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.073+0000 7f29a1e00640 1 --2- 172.21.15.40:0/4214109056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 0x5557239b2f30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x5557239cd7a0 tx=0x7f2994030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5249a10b76b2cad4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.073+0000 7f2982000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.073+0000 7f2982000640 1 -- 172.21.15.40:0/4214109056 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299402ad70 con 0x5557239afe80 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.074+0000 7f2982000640 1 -- 172.21.15.40:0/4214109056 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f2994003ba0 con 0x5557239afe80 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.074+0000 7f29a6ded740 1 -- 172.21.15.40:0/4214109056 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2990012d00 con 0x5557239afe80 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.074+0000 7f2982000640 1 -- 172.21.15.40:0/4214109056 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f299405f050 con 0x5557239afe80 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.074+0000 7f29a6ded740 10 client.5625 fetch_fsmap learned FSMap version 23 2024-04-26T23:03:06.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.074+0000 7f29a6ded740 1 -- 172.21.15.40:0/4214109056 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x5557239726c0 con 0x5557239afe80 2024-04-26T23:03:06.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.074+0000 7f2982000640 1 -- 172.21.15.40:0/4214109056 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2990012d00 con 0x5557239afe80 2024-04-26T23:03:06.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.074+0000 7f2982000640 1 -- 172.21.15.40:0/4214109056 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) ==== 3534+0+0 (secure 0 0 0) 0x7f2994051030 con 0x5557239afe80 2024-04-26T23:03:06.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.074+0000 7f29a6ded740 10 client.5625 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:03:06.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.074+0000 7f29a6ded740 10 client.5625 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:03:06.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.074+0000 7f29a6ded740 20 client.5625 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:06.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.074+0000 7f29a6ded740 1 --2- 172.21.15.40:0/4214109056 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55572399a350 0x55572399a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.074+0000 7f29a6ded740 4 client.5625 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:03:06.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.074+0000 7f29a6ded740 1 -- 172.21.15.40:0/4214109056 --> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] -- command(tid 0: {"prefix": "status"}) -- 0x55572396f740 con 0x55572399a350 2024-04-26T23:03:06.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.075+0000 7f29a1400640 1 --2- 172.21.15.40:0/4214109056 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55572399a350 0x55572399a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.075+0000 7f29a1400640 1 --2- 172.21.15.40:0/4214109056 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55572399a350 0x55572399a730 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=620ba641cd73403f server_cookie=4a5486acbd5a8634 in_seq=0 out_seq=0 2024-04-26T23:03:06.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.075+0000 7f2982000640 10 client.5625 ms_handle_connect on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.075+0000 7f2982000640 1 -- 172.21.15.40:0/4214109056 <== mds.0 v2:172.21.15.40:6833/299800334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55572396f740 con 0x55572399a350 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.075+0000 7f2982000640 10 client.5625 handle_command_reply: tid=0 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.075+0000 7f29a6ded740 1 client.5625 shutdown 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.076+0000 7f29a6ded740 1 -- 172.21.15.40:0/4214109056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 msgr2=0x5557239b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.076+0000 7f29a6ded740 1 --2- 172.21.15.40:0/4214109056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 0x5557239b2f30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x5557239cd7a0 tx=0x7f2994030000 comp rx=0 tx=0).stop 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.076+0000 7f29a6ded740 1 -- 172.21.15.40:0/4214109056 shutdown_connections 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.076+0000 7f29a6ded740 1 --2- 172.21.15.40:0/4214109056 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55572399a350 0x55572399a730 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.076+0000 7f29a6ded740 1 --2- 172.21.15.40:0/4214109056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5557239afe80 0x5557239b2f30 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.076+0000 7f29a6ded740 1 --2- 172.21.15.40:0/4214109056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55572399e270 0x5557239b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.076+0000 7f29a6ded740 1 --2- 172.21.15.40:0/4214109056 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5557239737b0 0x555723972b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.076+0000 7f2982000640 0 client.5625 ms_handle_reset on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.076+0000 7f29a6ded740 1 -- 172.21.15.40:0/4214109056 >> 172.21.15.40:0/4214109056 conn(0x555723907d10 msgr2=0x5557239b4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.076+0000 7f29a6ded740 1 -- 172.21.15.40:0/4214109056 shutdown_connections 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.076+0000 7f29a6ded740 1 -- 172.21.15.40:0/4214109056 wait complete. 2024-04-26T23:03:06.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.076+0000 7f29a6ded740 20 client.5625 trim_cache size 0 max 16384 2024-04-26T23:03:06.080 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:03:06.080 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:03:06.080 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:03:06.080 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5334, 2024-04-26T23:03:06.080 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:03:06.080 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:03:06.080 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:03:06.080 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 17.682191510999999, 2024-04-26T23:03:06.080 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 23, 2024-04-26T23:03:06.080 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 39, 2024-04-26T23:03:06.081 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 39, 2024-04-26T23:03:06.081 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 26.489707453000001 2024-04-26T23:03:06.081 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:03:06.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.077+0000 7f29a2800640 1 -- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f297c06c8f0 msgr2=0x7f297c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.077+0000 7f29a2800640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f297c06c8f0 0x7f297c06edb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f299c15f780 tx=0x7f2988009040 comp rx=0 tx=0).stop 2024-04-26T23:03:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.077+0000 7f29a2800640 1 -- 172.21.15.40:0/4189047950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c0b6220 msgr2=0x7f299c1585f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.077+0000 7f29a2800640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c0b6220 0x7f299c1585f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f299000c450 tx=0x7f299000c910 comp rx=0 tx=0).stop 2024-04-26T23:03:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.077+0000 7f29a2800640 1 -- 172.21.15.40:0/4189047950 shutdown_connections 2024-04-26T23:03:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.077+0000 7f29a2800640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f297c06c8f0 0x7f297c06edb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.077+0000 7f29a2800640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f299c0d3740 0x7f299c160890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.077+0000 7f29a2800640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299c0d32e0 0x7f299c09ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.077+0000 7f29a2800640 1 --2- 172.21.15.40:0/4189047950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c0b6220 0x7f299c1585f0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.077+0000 7f29a2800640 1 -- 172.21.15.40:0/4189047950 >> 172.21.15.40:0/4189047950 conn(0x7f299c0d4d90 msgr2=0x7f299c0cf860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.077+0000 7f29a2800640 1 -- 172.21.15.40:0/4189047950 shutdown_connections 2024-04-26T23:03:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.077+0000 7f29a2800640 1 -- 172.21.15.40:0/4189047950 wait complete. 2024-04-26T23:03:06.094 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:03:06.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.290+0000 7f236f200640 1 Processor -- start 2024-04-26T23:03:06.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.291+0000 7f236f200640 1 -- start start 2024-04-26T23:03:06.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.292+0000 7f236f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f237015ce60 0x7f23700afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.292+0000 7f236f200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f23700af7a0 0x7f23700ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.292+0000 7f236f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2370071fe0 0x7f2370073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.292+0000 7f236f200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f23700cc450 con 0x7f23700af7a0 2024-04-26T23:03:06.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.292+0000 7f236f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f23700d0820 con 0x7f237015ce60 2024-04-26T23:03:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.292+0000 7f236f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f23701614c0 con 0x7f2370071fe0 2024-04-26T23:03:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.293+0000 7f236d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f23700af7a0 0x7f23700ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.293+0000 7f236d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f23700af7a0 0x7f23700ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51678/0 (socket says 172.21.15.40:51678) 2024-04-26T23:03:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.293+0000 7f236d400640 1 -- 172.21.15.40:0/1445267870 learned_addr learned my addr 172.21.15.40:0/1445267870 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.293+0000 7f236de00640 1 --2- 172.21.15.40:0/1445267870 >> v2:172.21.15.103:3300/0 conn(0x7f237015ce60 0x7f23700afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.293+0000 7f236e800640 1 --2- 172.21.15.40:0/1445267870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2370071fe0 0x7f2370073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.293+0000 7f236d400640 1 -- 172.21.15.40:0/1445267870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2370071fe0 msgr2=0x7f2370073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.293+0000 7f236d400640 1 --2- 172.21.15.40:0/1445267870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2370071fe0 0x7f2370073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.293+0000 7f236d400640 1 -- 172.21.15.40:0/1445267870 >> v2:172.21.15.103:3300/0 conn(0x7f237015ce60 msgr2=0x7f23700afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.293+0000 7f236d400640 1 --2- 172.21.15.40:0/1445267870 >> v2:172.21.15.103:3300/0 conn(0x7f237015ce60 0x7f23700afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.293+0000 7f236d400640 1 -- 172.21.15.40:0/1445267870 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2370161760 con 0x7f23700af7a0 2024-04-26T23:03:06.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.294+0000 7f236d400640 1 --2- 172.21.15.40:0/1445267870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23700af7a0 0x7f23700ceff0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f23700741b0 tx=0x7f235802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd7366a35115e52 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:06.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.295+0000 7f236ca00640 1 -- 172.21.15.40:0/1445267870 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358037070 con 0x7f23700af7a0 2024-04-26T23:03:06.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.295+0000 7f236ca00640 1 -- 172.21.15.40:0/1445267870 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2358032440 con 0x7f23700af7a0 2024-04-26T23:03:06.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.295+0000 7f236ca00640 1 -- 172.21.15.40:0/1445267870 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358032810 con 0x7f23700af7a0 2024-04-26T23:03:06.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.296+0000 7f236f200640 1 -- 172.21.15.40:0/1445267870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23700af7a0 msgr2=0x7f23700ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.296+0000 7f236f200640 1 --2- 172.21.15.40:0/1445267870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23700af7a0 0x7f23700ceff0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f23700741b0 tx=0x7f235802a140 comp rx=0 tx=0).stop 2024-04-26T23:03:06.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.296+0000 7f236f200640 1 -- 172.21.15.40:0/1445267870 shutdown_connections 2024-04-26T23:03:06.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.296+0000 7f236f200640 1 --2- 172.21.15.40:0/1445267870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2370071fe0 0x7f2370073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.296+0000 7f236f200640 1 --2- 172.21.15.40:0/1445267870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23700af7a0 0x7f23700ceff0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.296+0000 7f236f200640 1 --2- 172.21.15.40:0/1445267870 >> v2:172.21.15.103:3300/0 conn(0x7f237015ce60 0x7f23700afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.296+0000 7f236f200640 1 -- 172.21.15.40:0/1445267870 >> 172.21.15.40:0/1445267870 conn(0x7f23700d0a10 msgr2=0x7f23700d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:06.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.296+0000 7f236f200640 1 -- 172.21.15.40:0/1445267870 shutdown_connections 2024-04-26T23:03:06.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.296+0000 7f236f200640 1 -- 172.21.15.40:0/1445267870 wait complete. 2024-04-26T23:03:06.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.298+0000 7f236f200640 1 Processor -- start 2024-04-26T23:03:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.298+0000 7f236f200640 1 -- start start 2024-04-26T23:03:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2370071fe0 0x7f237015e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23700af7a0 0x7f23701543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f237015ce60 0x7f23700c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2370073ca0 con 0x7f23700af7a0 2024-04-26T23:03:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f23700cf530 con 0x7f237015ce60 2024-04-26T23:03:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2370073e20 con 0x7f2370071fe0 2024-04-26T23:03:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23700af7a0 0x7f23701543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23700af7a0 0x7f23701543b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51686/0 (socket says 172.21.15.40:51686) 2024-04-26T23:03:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236d400640 1 -- 172.21.15.40:0/380509617 learned_addr learned my addr 172.21.15.40:0/380509617 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236d400640 1 -- 172.21.15.40:0/380509617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2370071fe0 msgr2=0x7f237015e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236d400640 1 --2- 172.21.15.40:0/380509617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2370071fe0 0x7f237015e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236d400640 1 -- 172.21.15.40:0/380509617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f237015ce60 msgr2=0x7f23700c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236e800640 1 --2- 172.21.15.40:0/380509617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f237015ce60 0x7f23700c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236d400640 1 --2- 172.21.15.40:0/380509617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f237015ce60 0x7f23700c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.299+0000 7f236d400640 1 -- 172.21.15.40:0/380509617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f237015d610 con 0x7f23700af7a0 2024-04-26T23:03:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.300+0000 7f236d400640 1 --2- 172.21.15.40:0/380509617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23700af7a0 0x7f23701543b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2358000c00 tx=0x7f23580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.300+0000 7f2366000640 1 -- 172.21.15.40:0/380509617 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358041020 con 0x7f23700af7a0 2024-04-26T23:03:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.300+0000 7f2366000640 1 -- 172.21.15.40:0/380509617 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23580306c0 con 0x7f23700af7a0 2024-04-26T23:03:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.300+0000 7f2366000640 1 -- 172.21.15.40:0/380509617 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358037040 con 0x7f23700af7a0 2024-04-26T23:03:06.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.300+0000 7f236f200640 1 -- 172.21.15.40:0/380509617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f237015d8f0 con 0x7f23700af7a0 2024-04-26T23:03:06.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.300+0000 7f236f200640 1 -- 172.21.15.40:0/380509617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f237015deb0 con 0x7f23700af7a0 2024-04-26T23:03:06.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.301+0000 7f236f200640 1 -- 172.21.15.40:0/380509617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2370069370 con 0x7f23700af7a0 2024-04-26T23:03:06.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.302+0000 7f2366000640 1 -- 172.21.15.40:0/380509617 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2358044020 con 0x7f23700af7a0 2024-04-26T23:03:06.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.303+0000 7f2366000640 1 --2- 172.21.15.40:0/380509617 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f234c06c8f0 0x7f234c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:06.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.304+0000 7f2366000640 1 -- 172.21.15.40:0/380509617 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f23580ac580 con 0x7f23700af7a0 2024-04-26T23:03:06.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.304+0000 7f236de00640 1 --2- 172.21.15.40:0/380509617 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f234c06c8f0 0x7f234c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:06.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.305+0000 7f236de00640 1 --2- 172.21.15.40:0/380509617 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f234c06c8f0 0x7f234c06edb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2360009dd0 tx=0x7f2360009450 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:06.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.308+0000 7f2366000640 1 -- 172.21.15.40:0/380509617 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f235804b030 con 0x7f23700af7a0 2024-04-26T23:03:06.357 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-04-26T23:03:06.356+0000 7f0c37200640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T23:03:06.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.556+0000 7f236f200640 1 -- 172.21.15.40:0/380509617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2370069c80 con 0x7f23700af7a0 2024-04-26T23:03:06.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.557+0000 7f2366000640 1 -- 172.21.15.40:0/380509617 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5216 (secure 0 0 0) 0x7f235807a780 con 0x7f23700af7a0 2024-04-26T23:03:06.559 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:03:06.559 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1979956236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1979956236},{"type":"v1","addr":"172.21.15.103:6837","nonce":1979956236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:02:47.924696+0000","modified":"2024-04-26T23:03:04.527692+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5334,"mds_1":5331,"mds_2":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"b","rank":2,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.103:6836/603576312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":603576312},{"type":"v1","addr":"172.21.15.103:6836","nonce":603576312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5331":{"gid":5331,"name":"c","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6834/716981348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":716981348},{"type":"v1","addr":"172.21.15.40:6834","nonce":716981348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5334":{"gid":5334,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.40:6835/299800334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":299800334},{"type":"v1","addr":"172.21.15.40:6835","nonce":299800334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5334,"qdb_cluster":[4975,5331,5334]},"id":2}]} 2024-04-26T23:03:06.559 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 23 2024-04-26T23:03:06.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.560+0000 7f236f200640 1 -- 172.21.15.40:0/380509617 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f234c06c8f0 msgr2=0x7f234c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.560+0000 7f236f200640 1 --2- 172.21.15.40:0/380509617 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f234c06c8f0 0x7f234c06edb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2360009dd0 tx=0x7f2360009450 comp rx=0 tx=0).stop 2024-04-26T23:03:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.560+0000 7f236f200640 1 -- 172.21.15.40:0/380509617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23700af7a0 msgr2=0x7f23701543b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.560+0000 7f236f200640 1 --2- 172.21.15.40:0/380509617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23700af7a0 0x7f23701543b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2358000c00 tx=0x7f23580046c0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.560+0000 7f236f200640 1 -- 172.21.15.40:0/380509617 shutdown_connections 2024-04-26T23:03:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.560+0000 7f236f200640 1 --2- 172.21.15.40:0/380509617 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f234c06c8f0 0x7f234c06edb0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.560+0000 7f236f200640 1 --2- 172.21.15.40:0/380509617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f237015ce60 0x7f23700c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.560+0000 7f236f200640 1 --2- 172.21.15.40:0/380509617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23700af7a0 0x7f23701543b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.560+0000 7f236f200640 1 --2- 172.21.15.40:0/380509617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2370071fe0 0x7f237015e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.560+0000 7f236f200640 1 -- 172.21.15.40:0/380509617 >> 172.21.15.40:0/380509617 conn(0x7f23700d0a10 msgr2=0x7f2370160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.560+0000 7f236f200640 1 -- 172.21.15.40:0/380509617 shutdown_connections 2024-04-26T23:03:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:06.560+0000 7f236f200640 1 -- 172.21.15.40:0/380509617 wait complete. 2024-04-26T23:03:06.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:06.581 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p d1/d2/d3/d4/d5/d6/d7/d8) 2024-04-26T23:03:06.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:06.666 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec sync d1/d2/d3/d4/d5/d6/d7/d8) 2024-04-26T23:03:06.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:06.737 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 d1/d2) 2024-04-26T23:03:06.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:06.791 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 d1/d2/d3/d4) 2024-04-26T23:03:06.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:06.842 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 2 d1/d2/d3/d4/d5/d6) 2024-04-26T23:03:06.900 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:03:07.135 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"5","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:03:07.136 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:03:07.136 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:03:09.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:03:09.378 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"5","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:03:09.378 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:03:09.379 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:03:11.380 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:03:11.628 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"7","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:03:11.628 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:03:11.628 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:03:13.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:03:13.869 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"7","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:03:13.869 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:03:13.870 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:03:15.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:03:16.111 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"13","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/d1/d2","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-04-26T23:03:16.111 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000001", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/d1/d2/d3/d4", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-04-26T23:03:16.111 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:03:16.112 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:03:16.343 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:03:16.344 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:03:16.344 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:03:17.413 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:03:17.421 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:03:18.345 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:03:18.587 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:03:18.587 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:03:18.587 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:03:20.588 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:03:20.807 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1,"2":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":2,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/d1/d2/d3/d4/d5/d6","dirfrag":"0x10000000005","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-04-26T23:03:20.808 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1, "2": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "2", "dirfrag": "0x10000000005", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/d1/d2/d3/d4/d5/d6", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "6", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 2, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false } ] 2024-04-26T23:03:20.808 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:03:20.809 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-04-26T23:03:21.040 INFO:teuthology.orchestra.run.smithi103.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"0","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dnwaiter":0,"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":2,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/d1/d2/d3/d4/d5/d6","dirfrag":"0x10000000005","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:03:21.041 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dnwaiter": 0, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "2", "dirfrag": "0x10000000005", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4/d5/d6", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 2, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:03:21.041 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:03:21.042 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 flush journal 2024-04-26T23:03:21.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.268+0000 7f2787200640 1 Processor -- start 2024-04-26T23:03:21.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.269+0000 7f2787200640 1 -- start start 2024-04-26T23:03:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.270+0000 7f2787200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f27880b6470 0x7f27880cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.270+0000 7f2787200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27880d5d30 con 0x7f27880d1770 2024-04-26T23:03:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.270+0000 7f2787200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27880d5a70 con 0x7f278815f680 2024-04-26T23:03:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.270+0000 7f2787200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f27880d3760 con 0x7f27880b6470 2024-04-26T23:03:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.271+0000 7f2785e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f27880d1770 0x7f278815ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49402/0 (socket says 172.21.15.40:49402) 2024-04-26T23:03:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.271+0000 7f2785e00640 1 -- 172.21.15.40:0/2867075304 learned_addr learned my addr 172.21.15.40:0/2867075304 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.271+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1796151866 0 0) 0x7f27880d5d30 con 0x7f27880d1770 2024-04-26T23:03:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.271+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2768003520 con 0x7f27880d1770 2024-04-26T23:03:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.271+0000 7f2785400640 1 --2- 172.21.15.40:0/2867075304 >> v2:172.21.15.103:3300/0 conn(0x7f27880b6470 0x7f27880cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2661620373 0 0) 0x7f2768003520 con 0x7f27880d1770 2024-04-26T23:03:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 >> v2:172.21.15.103:3300/0 conn(0x7f27880b6470 msgr2=0x7f27880cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2784a00640 1 --2- 172.21.15.40:0/2867075304 >> v2:172.21.15.103:3300/0 conn(0x7f27880b6470 0x7f27880cf9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 >> v1:172.21.15.103:6789/0 conn(0x7f278815f680 legacy=0x7f27880cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27880b5b00 con 0x7f27880d1770 2024-04-26T23:03:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f2778001670 con 0x7f27880d1770 2024-04-26T23:03:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 >> v1:172.21.15.40:6789/0 conn(0x7f27880d1770 legacy=0x7f278815ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2784a00640 1 --2- 172.21.15.40:0/2867075304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27680039b0 0x7f2768003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2784a00640 1 --2- 172.21.15.40:0/2867075304 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2768004800 0x7f2768004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2784a00640 1 --2- 172.21.15.40:0/2867075304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27680052e0 0x7f276800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f27880d3760 con 0x7f27680039b0 2024-04-26T23:03:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f276800c1e0 con 0x7f27680052e0 2024-04-26T23:03:21.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f276800c3e0 con 0x7f2768004800 2024-04-26T23:03:21.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2785e00640 1 --2- 172.21.15.40:0/2867075304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27680039b0 0x7f2768003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2786800640 1 --2- 172.21.15.40:0/2867075304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27680052e0 0x7f276800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2785e00640 1 -- 172.21.15.40:0/2867075304 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2768004800 msgr2=0x7f2768004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.272+0000 7f2785e00640 1 --2- 172.21.15.40:0/2867075304 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2768004800 0x7f2768004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.273+0000 7f2785e00640 1 -- 172.21.15.40:0/2867075304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27680052e0 msgr2=0x7f276800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.273+0000 7f2785e00640 1 --2- 172.21.15.40:0/2867075304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27680052e0 0x7f276800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.273+0000 7f2785e00640 1 -- 172.21.15.40:0/2867075304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2768003520 con 0x7f27680039b0 2024-04-26T23:03:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.273+0000 7f2786800640 1 --2- 172.21.15.40:0/2867075304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27680052e0 0x7f276800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.273+0000 7f2786800640 1 -- 172.21.15.40:0/2867075304 reap_dead start 2024-04-26T23:03:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.274+0000 7f2785e00640 1 --2- 172.21.15.40:0/2867075304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27680039b0 0x7f2768003dd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f277800c210 tx=0x7f2768005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b000b75f3e2681c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.274+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778007d30 con 0x7f27680039b0 2024-04-26T23:03:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.274+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2778005ad0 con 0x7f27680039b0 2024-04-26T23:03:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.274+0000 7f2784a00640 1 -- 172.21.15.40:0/2867075304 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778007070 con 0x7f27680039b0 2024-04-26T23:03:21.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.274+0000 7f2787200640 1 -- 172.21.15.40:0/2867075304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27680039b0 msgr2=0x7f2768003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.274+0000 7f2787200640 1 --2- 172.21.15.40:0/2867075304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27680039b0 0x7f2768003dd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f277800c210 tx=0x7f2768005830 comp rx=0 tx=0).stop 2024-04-26T23:03:21.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.275+0000 7f2787200640 1 -- 172.21.15.40:0/2867075304 shutdown_connections 2024-04-26T23:03:21.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.275+0000 7f2787200640 1 --2- 172.21.15.40:0/2867075304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27680039b0 0x7f2768003dd0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.275+0000 7f2787200640 1 -- 172.21.15.40:0/2867075304 >> 172.21.15.40:0/2867075304 conn(0x7f27880d4dd0 msgr2=0x7f27880d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.275+0000 7f2787200640 1 -- 172.21.15.40:0/2867075304 shutdown_connections 2024-04-26T23:03:21.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.275+0000 7f2787200640 1 -- 172.21.15.40:0/2867075304 wait complete. 2024-04-26T23:03:21.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.276+0000 7f2787200640 1 Processor -- start 2024-04-26T23:03:21.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.277+0000 7f2787200640 1 -- start start 2024-04-26T23:03:21.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.277+0000 7f2787200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27680039b0 0x7f27881f5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.277+0000 7f2787200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f278815c950 0x7f27881f7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2787200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2788161e80 0x7f2788162330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2787200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f27880d3d70 con 0x7f27680039b0 2024-04-26T23:03:21.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2787200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f27880d3bf0 con 0x7f2788161e80 2024-04-26T23:03:21.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2787200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f27880d1b50 con 0x7f278815c950 2024-04-26T23:03:21.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27680039b0 0x7f27881f5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27680039b0 0x7f27881f5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39590/0 (socket says 172.21.15.40:39590) 2024-04-26T23:03:21.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2785e00640 1 -- 172.21.15.40:0/3847217514 learned_addr learned my addr 172.21.15.40:0/3847217514 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2785400640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f278815c950 0x7f27881f7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2785e00640 1 -- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f278815c950 msgr2=0x7f27881f7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2785e00640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f278815c950 0x7f27881f7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2785e00640 1 -- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2788161e80 msgr2=0x7f2788162330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2785e00640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2788161e80 0x7f2788162330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2785e00640 1 -- 172.21.15.40:0/3847217514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27880ba590 con 0x7f27680039b0 2024-04-26T23:03:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.278+0000 7f2785400640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f278815c950 0x7f27881f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.279+0000 7f2785e00640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27680039b0 0x7f27881f5040 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2778034040 tx=0x7f2778001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.279+0000 7f277e000640 1 -- 172.21.15.40:0/3847217514 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778041070 con 0x7f27680039b0 2024-04-26T23:03:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.279+0000 7f277e000640 1 -- 172.21.15.40:0/3847217514 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2778000ed0 con 0x7f27680039b0 2024-04-26T23:03:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.279+0000 7f277e000640 1 -- 172.21.15.40:0/3847217514 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778007270 con 0x7f27680039b0 2024-04-26T23:03:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.279+0000 7f2787200640 1 -- 172.21.15.40:0/3847217514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27880ccbf0 con 0x7f27680039b0 2024-04-26T23:03:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.279+0000 7f2787200640 1 -- 172.21.15.40:0/3847217514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2788162870 con 0x7f27680039b0 2024-04-26T23:03:21.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.282+0000 7f277e000640 1 -- 172.21.15.40:0/3847217514 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2778044070 con 0x7f27680039b0 2024-04-26T23:03:21.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.283+0000 7f277e000640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f276406c920 0x7f276406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.283+0000 7f277e000640 1 -- 172.21.15.40:0/3847217514 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f27780ac400 con 0x7f27680039b0 2024-04-26T23:03:21.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.284+0000 7f278f87d740 1 Processor -- start 2024-04-26T23:03:21.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.284+0000 7f278f87d740 1 -- start start 2024-04-26T23:03:21.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.284+0000 7f2785400640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f276406c920 0x7f276406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f278f87d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b69caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f278f87d740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b6b9fb0 0x562a2b6ba500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f278f87d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b6723f0 0x562a2b6b8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f278f87d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562a2b582030 con 0x562a2b69c710 2024-04-26T23:03:21.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f278f87d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562a2b59d7f0 con 0x562a2b6723f0 2024-04-26T23:03:21.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f278f87d740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x562a2b5587e0 con 0x562a2b6b9fb0 2024-04-26T23:03:21.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f2785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b69caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f2785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b69caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39598/0 (socket says 172.21.15.40:39598) 2024-04-26T23:03:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f2785e00640 1 -- 172.21.15.40:0/3433324223 learned_addr learned my addr 172.21.15.40:0/3433324223 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f2785400640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f276406c920 0x7f276406ede0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f27880b82f0 tx=0x7f2770006e60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f2785e00640 1 -- 172.21.15.40:0/3433324223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b6b9fb0 msgr2=0x562a2b6ba500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f2785e00640 1 --2- 172.21.15.40:0/3433324223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b6b9fb0 0x562a2b6ba500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f2785e00640 1 -- 172.21.15.40:0/3433324223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b6723f0 msgr2=0x562a2b6b8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f2785e00640 1 --2- 172.21.15.40:0/3433324223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b6723f0 0x562a2b6b8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.285+0000 7f2785e00640 1 -- 172.21.15.40:0/3433324223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a2b564050 con 0x562a2b69c710 2024-04-26T23:03:21.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.286+0000 7f2785e00640 1 --2- 172.21.15.40:0/3433324223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b69caf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2778082640 tx=0x7f2778031f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea2ef14fb71131d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.286+0000 7f2776600640 1 -- 172.21.15.40:0/3433324223 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277807e340 con 0x562a2b69c710 2024-04-26T23:03:21.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.286+0000 7f2776600640 1 -- 172.21.15.40:0/3433324223 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2778082b70 con 0x562a2b69c710 2024-04-26T23:03:21.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.286+0000 7f2776600640 1 -- 172.21.15.40:0/3433324223 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27780aa3d0 con 0x562a2b69c710 2024-04-26T23:03:21.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.286+0000 7f278f87d740 1 -- 172.21.15.40:0/3433324223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 msgr2=0x562a2b69caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.286+0000 7f278f87d740 1 --2- 172.21.15.40:0/3433324223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b69caf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2778082640 tx=0x7f2778031f40 comp rx=0 tx=0).stop 2024-04-26T23:03:21.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.286+0000 7f278f87d740 1 -- 172.21.15.40:0/3433324223 shutdown_connections 2024-04-26T23:03:21.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.286+0000 7f278f87d740 1 --2- 172.21.15.40:0/3433324223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b6723f0 0x562a2b6b8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.286+0000 7f278f87d740 1 --2- 172.21.15.40:0/3433324223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b6b9fb0 0x562a2b6ba500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.286+0000 7f278f87d740 1 --2- 172.21.15.40:0/3433324223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b69caf0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.287+0000 7f278f87d740 1 -- 172.21.15.40:0/3433324223 >> 172.21.15.40:0/3433324223 conn(0x562a2b66ed20 msgr2=0x562a2b6b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.287+0000 7f278f87d740 1 -- 172.21.15.40:0/3433324223 shutdown_connections 2024-04-26T23:03:21.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.287+0000 7f278f87d740 1 -- 172.21.15.40:0/3433324223 wait complete. 2024-04-26T23:03:21.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.287+0000 7f278f87d740 1 -- start start 2024-04-26T23:03:21.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.287+0000 7f278f87d740 1 Processor -- start 2024-04-26T23:03:21.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.288+0000 7f278f87d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b691190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f278f87d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b6b9fb0 0x562a2b694450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f278f87d740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b696680 0x562a2b696b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f278f87d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562a2b5da200 con 0x562a2b69c710 2024-04-26T23:03:21.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f278f87d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562a2b5d4ee0 con 0x562a2b6b9fb0 2024-04-26T23:03:21.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f278f87d740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x562a2b5e1e70 con 0x562a2b696680 2024-04-26T23:03:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f278f87d740 10 client.0 _ll_register_callbacks cb 0x562a2b5cb270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f2785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b691190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f2785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b691190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39600/0 (socket says 172.21.15.40:39600) 2024-04-26T23:03:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f2785e00640 1 -- 172.21.15.40:0/3072383071 learned_addr learned my addr 172.21.15.40:0/3072383071 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f2785e00640 1 -- 172.21.15.40:0/3072383071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b696680 msgr2=0x562a2b696b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f2785e00640 1 --2- 172.21.15.40:0/3072383071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b696680 0x562a2b696b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f2785e00640 1 -- 172.21.15.40:0/3072383071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b6b9fb0 msgr2=0x562a2b694450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f2785e00640 1 --2- 172.21.15.40:0/3072383071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b6b9fb0 0x562a2b694450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f2785e00640 1 -- 172.21.15.40:0/3072383071 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a2b3b17d0 con 0x562a2b69c710 2024-04-26T23:03:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.289+0000 7f2785e00640 1 --2- 172.21.15.40:0/3072383071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b691190 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f27780763f0 tx=0x7f27780ac050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1303bb966a6500 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.290+0000 7f276ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.290+0000 7f276ea00640 1 -- 172.21.15.40:0/3072383071 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778097030 con 0x562a2b69c710 2024-04-26T23:03:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.290+0000 7f276ea00640 1 -- 172.21.15.40:0/3072383071 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f2778078290 con 0x562a2b69c710 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.290+0000 7f278f87d740 1 -- 172.21.15.40:0/3072383071 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2778082b70 con 0x562a2b69c710 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.290+0000 7f276ea00640 1 -- 172.21.15.40:0/3072383071 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f277807e340 con 0x562a2b69c710 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.290+0000 7f278f87d740 10 client.5643 fetch_fsmap learned FSMap version 25 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.290+0000 7f278f87d740 1 -- 172.21.15.40:0/3072383071 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x562a2b2e73f0 con 0x562a2b69c710 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.290+0000 7f276ea00640 1 -- 172.21.15.40:0/3072383071 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2778082b70 con 0x562a2b69c710 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.290+0000 7f276ea00640 1 -- 172.21.15.40:0/3072383071 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 3546+0+0 (secure 0 0 0) 0x7f277804e060 con 0x562a2b69c710 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.290+0000 7f278f87d740 10 client.5643 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.290+0000 7f278f87d740 10 client.5643 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.290+0000 7f278f87d740 20 client.5643 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.291+0000 7f278f87d740 1 --2- 172.21.15.40:0/3072383071 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x562a2b695250 0x562a2b6956c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.291+0000 7f278f87d740 4 client.5643 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.291+0000 7f278f87d740 1 -- 172.21.15.40:0/3072383071 --> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a2b5e37e0 con 0x562a2b695250 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.291+0000 7f2785400640 1 --2- 172.21.15.40:0/3072383071 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x562a2b695250 0x562a2b6956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.291+0000 7f2785400640 1 --2- 172.21.15.40:0/3072383071 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x562a2b695250 0x562a2b6956c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33260d1cc1b65926 server_cookie=fb0af61e2d2093f6 in_seq=0 out_seq=0 2024-04-26T23:03:21.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.291+0000 7f276ea00640 10 client.5643 ms_handle_connect on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.292+0000 7f276ea00640 1 -- 172.21.15.40:0/3072383071 <== mds.0 v2:172.21.15.40:6833/299800334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a2b5e37e0 con 0x562a2b695250 2024-04-26T23:03:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.292+0000 7f276ea00640 10 client.5643 handle_command_reply: tid=0 2024-04-26T23:03:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.292+0000 7f278f87d740 1 client.5643 shutdown 2024-04-26T23:03:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.292+0000 7f278f87d740 1 -- 172.21.15.40:0/3072383071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 msgr2=0x562a2b691190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.292+0000 7f278f87d740 1 --2- 172.21.15.40:0/3072383071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b691190 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f27780763f0 tx=0x7f27780ac050 comp rx=0 tx=0).stop 2024-04-26T23:03:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.292+0000 7f278f87d740 1 -- 172.21.15.40:0/3072383071 shutdown_connections 2024-04-26T23:03:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.292+0000 7f278f87d740 1 --2- 172.21.15.40:0/3072383071 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x562a2b695250 0x562a2b6956c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.292+0000 7f278f87d740 1 --2- 172.21.15.40:0/3072383071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b696680 0x562a2b696b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.292+0000 7f278f87d740 1 --2- 172.21.15.40:0/3072383071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b6b9fb0 0x562a2b694450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.292+0000 7f278f87d740 1 --2- 172.21.15.40:0/3072383071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b691190 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.293+0000 7f278f87d740 1 -- 172.21.15.40:0/3072383071 >> 172.21.15.40:0/3072383071 conn(0x562a2b6723f0 msgr2=0x562a2b69bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.293+0000 7f276ea00640 0 client.5643 ms_handle_reset on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.293+0000 7f278f87d740 1 -- 172.21.15.40:0/3072383071 shutdown_connections 2024-04-26T23:03:21.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.293+0000 7f278f87d740 1 -- 172.21.15.40:0/3072383071 wait complete. 2024-04-26T23:03:21.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.293+0000 7f278f87d740 20 client.5643 trim_cache size 0 max 16384 2024-04-26T23:03:21.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.310+0000 7f278f87d740 1 Processor -- start 2024-04-26T23:03:21.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.310+0000 7f278f87d740 1 -- start start 2024-04-26T23:03:21.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.310+0000 7f278f87d740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b6723f0 0x562a2b671a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.310+0000 7f278f87d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b696680 0x562a2b69dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f278f87d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b69e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f278f87d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f27882026b0 con 0x562a2b69c710 2024-04-26T23:03:21.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f278f87d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2788201880 con 0x562a2b696680 2024-04-26T23:03:21.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f278f87d740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2788201f80 con 0x562a2b6723f0 2024-04-26T23:03:21.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f2786800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b69e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f2786800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b69e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39606/0 (socket says 172.21.15.40:39606) 2024-04-26T23:03:21.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f2786800640 1 -- 172.21.15.40:0/3727792639 learned_addr learned my addr 172.21.15.40:0/3727792639 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f2785e00640 1 --2- 172.21.15.40:0/3727792639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b6723f0 0x562a2b671a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f2785400640 1 --2- 172.21.15.40:0/3727792639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b696680 0x562a2b69dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f2786800640 1 -- 172.21.15.40:0/3727792639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b6723f0 msgr2=0x562a2b671a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f2786800640 1 --2- 172.21.15.40:0/3727792639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b6723f0 0x562a2b671a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f2786800640 1 -- 172.21.15.40:0/3727792639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b696680 msgr2=0x562a2b69dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f2786800640 1 --2- 172.21.15.40:0/3727792639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b696680 0x562a2b69dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f2786800640 1 -- 172.21.15.40:0/3727792639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a2b69ce60 con 0x562a2b69c710 2024-04-26T23:03:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.311+0000 7f2785400640 1 --2- 172.21.15.40:0/3727792639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b696680 0x562a2b69dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.312+0000 7f2786800640 1 --2- 172.21.15.40:0/3727792639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b69e520 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f278806f480 tx=0x7f2780001f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84034dc9d9c4ebb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.312+0000 7f2775c00640 1 -- 172.21.15.40:0/3727792639 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2780018070 con 0x562a2b69c710 2024-04-26T23:03:21.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.312+0000 7f2775c00640 1 -- 172.21.15.40:0/3727792639 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2780003da0 con 0x562a2b69c710 2024-04-26T23:03:21.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.312+0000 7f2775c00640 1 -- 172.21.15.40:0/3727792639 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2780005310 con 0x562a2b69c710 2024-04-26T23:03:21.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.313+0000 7f278f87d740 1 -- 172.21.15.40:0/3727792639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 msgr2=0x562a2b69e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.313+0000 7f278f87d740 1 --2- 172.21.15.40:0/3727792639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b69e520 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f278806f480 tx=0x7f2780001f20 comp rx=0 tx=0).stop 2024-04-26T23:03:21.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.313+0000 7f278f87d740 1 -- 172.21.15.40:0/3727792639 shutdown_connections 2024-04-26T23:03:21.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.313+0000 7f278f87d740 1 --2- 172.21.15.40:0/3727792639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b69c710 0x562a2b69e520 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.313+0000 7f278f87d740 1 --2- 172.21.15.40:0/3727792639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b696680 0x562a2b69dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.313+0000 7f278f87d740 1 --2- 172.21.15.40:0/3727792639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b6723f0 0x562a2b671a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.313+0000 7f278f87d740 1 -- 172.21.15.40:0/3727792639 >> 172.21.15.40:0/3727792639 conn(0x562a2b69c140 msgr2=0x562a2b66d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.313+0000 7f278f87d740 1 -- 172.21.15.40:0/3727792639 shutdown_connections 2024-04-26T23:03:21.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.313+0000 7f278f87d740 1 -- 172.21.15.40:0/3727792639 wait complete. 2024-04-26T23:03:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.314+0000 7f278f87d740 1 -- start start 2024-04-26T23:03:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.314+0000 7f278f87d740 1 Processor -- start 2024-04-26T23:03:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.315+0000 7f278f87d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b6723f0 0x562a2b66b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.315+0000 7f278f87d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b696680 0x562a2b66bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.315+0000 7f278f87d740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b69c710 0x562a2b69f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.315+0000 7f278f87d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562a2b5e2fd0 con 0x562a2b6723f0 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.315+0000 7f278f87d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f27880ce3a0 con 0x562a2b696680 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.315+0000 7f278f87d740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x562a2b6b9510 con 0x562a2b69c710 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.315+0000 7f278f87d740 10 client.0 _ll_register_callbacks cb 0x7f2778079b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.315+0000 7f2785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b6723f0 0x562a2b66b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.315+0000 7f2785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b6723f0 0x562a2b66b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39610/0 (socket says 172.21.15.40:39610) 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.315+0000 7f2785e00640 1 -- 172.21.15.40:0/1297781 learned_addr learned my addr 172.21.15.40:0/1297781 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.316+0000 7f2785400640 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b696680 0x562a2b66bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.316+0000 7f2786800640 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b69c710 0x562a2b69f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.316+0000 7f2785e00640 1 -- 172.21.15.40:0/1297781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b69c710 msgr2=0x562a2b69f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.316+0000 7f2785e00640 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b69c710 0x562a2b69f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.316+0000 7f2785e00640 1 -- 172.21.15.40:0/1297781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b696680 msgr2=0x562a2b66bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.316+0000 7f2785e00640 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b696680 0x562a2b66bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.316+0000 7f2785e00640 1 -- 172.21.15.40:0/1297781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a2b2a70f0 con 0x562a2b6723f0 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.316+0000 7f2785400640 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b696680 0x562a2b66bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.316+0000 7f2785e00640 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b6723f0 0x562a2b66b960 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f27780aac00 tx=0x7f27780762e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44fb38347e626a08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.316+0000 7f276ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.316+0000 7f276ea00640 1 -- 172.21.15.40:0/1297781 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778097070 con 0x562a2b6723f0 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.316+0000 7f276ea00640 1 -- 172.21.15.40:0/1297781 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f27780ac0c0 con 0x562a2b6723f0 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.316+0000 7f278f87d740 1 -- 172.21.15.40:0/1297781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2780003da0 con 0x562a2b6723f0 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.317+0000 7f276ea00640 1 -- 172.21.15.40:0/1297781 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f27780798f0 con 0x562a2b6723f0 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.317+0000 7f278f87d740 10 client.5649 fetch_fsmap learned FSMap version 25 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.317+0000 7f278f87d740 1 -- 172.21.15.40:0/1297781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x562a2b59da30 con 0x562a2b6723f0 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.317+0000 7f276ea00640 1 -- 172.21.15.40:0/1297781 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2780003da0 con 0x562a2b6723f0 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.317+0000 7f276ea00640 1 -- 172.21.15.40:0/1297781 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 3546+0+0 (secure 0 0 0) 0x7f27780826e0 con 0x562a2b6723f0 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.317+0000 7f278f87d740 10 client.5649 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.317+0000 7f278f87d740 10 client.5649 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.317+0000 7f278f87d740 20 client.5649 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:21.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.317+0000 7f278f87d740 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x562a2b6b9fb0 0x562a2b69f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.317+0000 7f278f87d740 4 client.5649 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:03:21.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.317+0000 7f278f87d740 1 -- 172.21.15.40:0/1297781 --> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a2b5dd650 con 0x562a2b6b9fb0 2024-04-26T23:03:21.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.317+0000 7f2785400640 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x562a2b6b9fb0 0x562a2b69f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.318+0000 7f2785400640 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x562a2b6b9fb0 0x562a2b69f5c0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b32935b7e0697427 server_cookie=44280f24b3555a8f in_seq=0 out_seq=0 2024-04-26T23:03:21.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.318+0000 7f276ea00640 10 client.5649 ms_handle_connect on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:21.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.344+0000 7f276ea00640 1 -- 172.21.15.40:0/1297781 <== mds.0 v2:172.21.15.40:6833/299800334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562a2b5dd650 con 0x562a2b6b9fb0 2024-04-26T23:03:21.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.344+0000 7f276ea00640 10 client.5649 handle_command_reply: tid=0 2024-04-26T23:03:21.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.344+0000 7f278f87d740 1 client.5649 shutdown 2024-04-26T23:03:21.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.345+0000 7f278f87d740 1 -- 172.21.15.40:0/1297781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b6723f0 msgr2=0x562a2b66b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.345+0000 7f278f87d740 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b6723f0 0x562a2b66b960 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f27780aac00 tx=0x7f27780762e0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.345+0000 7f278f87d740 1 -- 172.21.15.40:0/1297781 shutdown_connections 2024-04-26T23:03:21.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.345+0000 7f278f87d740 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x562a2b6b9fb0 0x562a2b69f5c0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.345+0000 7f278f87d740 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a2b69c710 0x562a2b69f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.345+0000 7f278f87d740 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a2b696680 0x562a2b66bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.345+0000 7f278f87d740 1 --2- 172.21.15.40:0/1297781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a2b6723f0 0x562a2b66b960 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.345+0000 7f276ea00640 0 client.5649 ms_handle_reset on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:21.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.345+0000 7f278f87d740 1 -- 172.21.15.40:0/1297781 >> 172.21.15.40:0/1297781 conn(0x562a2b69c140 msgr2=0x562a2b6ba390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.345+0000 7f278f87d740 1 -- 172.21.15.40:0/1297781 shutdown_connections 2024-04-26T23:03:21.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.345+0000 7f278f87d740 1 -- 172.21.15.40:0/1297781 wait complete. 2024-04-26T23:03:21.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.345+0000 7f278f87d740 20 client.5649 trim_cache size 0 max 16384 2024-04-26T23:03:21.346 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:03:21.347 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:03:21.347 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:03:21.347 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:03:21.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.346+0000 7f2787200640 1 -- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f276406c920 msgr2=0x7f276406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.346+0000 7f2787200640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f276406c920 0x7f276406ede0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f27880b82f0 tx=0x7f2770006e60 comp rx=0 tx=0).stop 2024-04-26T23:03:21.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.346+0000 7f2787200640 1 -- 172.21.15.40:0/3847217514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27680039b0 msgr2=0x7f27881f5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.346+0000 7f2787200640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27680039b0 0x7f27881f5040 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2778034040 tx=0x7f2778001a40 comp rx=0 tx=0).stop 2024-04-26T23:03:21.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.346+0000 7f2787200640 1 -- 172.21.15.40:0/3847217514 shutdown_connections 2024-04-26T23:03:21.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.346+0000 7f2787200640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f276406c920 0x7f276406ede0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.346+0000 7f2787200640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2788161e80 0x7f2788162330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.346+0000 7f2787200640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f278815c950 0x7f27881f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.346+0000 7f2787200640 1 --2- 172.21.15.40:0/3847217514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27680039b0 0x7f27881f5040 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.346+0000 7f2787200640 1 -- 172.21.15.40:0/3847217514 >> 172.21.15.40:0/3847217514 conn(0x7f27880d4dd0 msgr2=0x7f278815e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.346+0000 7f2787200640 1 -- 172.21.15.40:0/3847217514 shutdown_connections 2024-04-26T23:03:21.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.346+0000 7f2787200640 1 -- 172.21.15.40:0/3847217514 wait complete. 2024-04-26T23:03:21.363 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:1 flush journal 2024-04-26T23:03:21.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.560+0000 7fe6c2400640 1 Processor -- start 2024-04-26T23:03:21.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.560+0000 7fe6c2400640 1 -- start start 2024-04-26T23:03:21.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.562+0000 7fe6c2400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe6bc160a50 0x7fe6bc073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.562+0000 7fe6c2400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6bc0cfaf0 0x7fe6bc160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.562+0000 7fe6c2400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6bc161880 con 0x7fe6bc06dd40 2024-04-26T23:03:21.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.562+0000 7fe6c2400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe6bc071ce0 con 0x7fe6bc0cfaf0 2024-04-26T23:03:21.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.562+0000 7fe6c2400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe6bc0722a0 con 0x7fe6bc160a50 2024-04-26T23:03:21.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.562+0000 7fe6b3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6bc0cfaf0 0x7fe6bc160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.562+0000 7fe6b3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6bc0cfaf0 0x7fe6bc160e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39612/0 (socket says 172.21.15.40:39612) 2024-04-26T23:03:21.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.562+0000 7fe6b3e00640 1 -- 172.21.15.40:0/2060147920 learned_addr learned my addr 172.21.15.40:0/2060147920 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.562+0000 7fe6c1000640 1 --2- 172.21.15.40:0/2060147920 >> v2:172.21.15.103:3300/0 conn(0x7fe6bc160a50 0x7fe6bc073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.562+0000 7fe6b3e00640 1 -- 172.21.15.40:0/2060147920 >> v2:172.21.15.103:3300/0 conn(0x7fe6bc160a50 msgr2=0x7fe6bc073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.562+0000 7fe6b3e00640 1 --2- 172.21.15.40:0/2060147920 >> v2:172.21.15.103:3300/0 conn(0x7fe6bc160a50 0x7fe6bc073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.562+0000 7fe6b3e00640 1 -- 172.21.15.40:0/2060147920 >> v1:172.21.15.103:6789/0 conn(0x7fe6bc06dd40 legacy=0x7fe6bc072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.562+0000 7fe6b3e00640 1 -- 172.21.15.40:0/2060147920 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6bc161370 con 0x7fe6bc0cfaf0 2024-04-26T23:03:21.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.563+0000 7fe6b3e00640 1 --2- 172.21.15.40:0/2060147920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6bc0cfaf0 0x7fe6bc160e30 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc161b70 tx=0x7fe6a402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c235e633b889ef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.564+0000 7fe6bbe00640 1 -- 172.21.15.40:0/2060147920 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4037070 con 0x7fe6bc0cfaf0 2024-04-26T23:03:21.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.564+0000 7fe6bbe00640 1 -- 172.21.15.40:0/2060147920 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6a4032440 con 0x7fe6bc0cfaf0 2024-04-26T23:03:21.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.564+0000 7fe6bbe00640 1 -- 172.21.15.40:0/2060147920 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4032810 con 0x7fe6bc0cfaf0 2024-04-26T23:03:21.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.564+0000 7fe6c2400640 1 -- 172.21.15.40:0/2060147920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6bc0cfaf0 msgr2=0x7fe6bc160e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.564+0000 7fe6c2400640 1 --2- 172.21.15.40:0/2060147920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6bc0cfaf0 0x7fe6bc160e30 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc161b70 tx=0x7fe6a402a230 comp rx=0 tx=0).stop 2024-04-26T23:03:21.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.564+0000 7fe6c2400640 1 -- 172.21.15.40:0/2060147920 shutdown_connections 2024-04-26T23:03:21.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.564+0000 7fe6c2400640 1 --2- 172.21.15.40:0/2060147920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6bc0cfaf0 0x7fe6bc160e30 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.564+0000 7fe6c2400640 1 --2- 172.21.15.40:0/2060147920 >> v2:172.21.15.103:3300/0 conn(0x7fe6bc160a50 0x7fe6bc073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.564+0000 7fe6c2400640 1 -- 172.21.15.40:0/2060147920 >> 172.21.15.40:0/2060147920 conn(0x7fe6bc0b2720 msgr2=0x7fe6bc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.565+0000 7fe6c2400640 1 -- 172.21.15.40:0/2060147920 shutdown_connections 2024-04-26T23:03:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.565+0000 7fe6c2400640 1 -- 172.21.15.40:0/2060147920 wait complete. 2024-04-26T23:03:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.566+0000 7fe6c2400640 1 Processor -- start 2024-04-26T23:03:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.566+0000 7fe6c2400640 1 -- start start 2024-04-26T23:03:21.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.567+0000 7fe6c2400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6bc06dd40 0x7fe6bc15a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.567+0000 7fe6c2400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe6bc160a50 0x7fe6bc0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.567+0000 7fe6c2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6bc0b1170 0x7fe6bc144050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.567+0000 7fe6c2400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc0d0260 con 0x7fe6bc0b1170 2024-04-26T23:03:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.567+0000 7fe6c2400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc073f90 con 0x7fe6bc06dd40 2024-04-26T23:03:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.567+0000 7fe6c2400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe6bc071f30 con 0x7fe6bc160a50 2024-04-26T23:03:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.568+0000 7fe6c1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6bc0b1170 0x7fe6bc144050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.568+0000 7fe6c1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6bc0b1170 0x7fe6bc144050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39620/0 (socket says 172.21.15.40:39620) 2024-04-26T23:03:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.568+0000 7fe6c1a00640 1 -- 172.21.15.40:0/2898709712 learned_addr learned my addr 172.21.15.40:0/2898709712 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.568+0000 7fe6c1000640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6bc06dd40 0x7fe6bc15a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.568+0000 7fe6c1a00640 1 -- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe6bc160a50 msgr2=0x7fe6bc0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.568+0000 7fe6c1a00640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe6bc160a50 0x7fe6bc0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.568+0000 7fe6c1a00640 1 -- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6bc06dd40 msgr2=0x7fe6bc15a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.568+0000 7fe6c1a00640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6bc06dd40 0x7fe6bc15a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.568+0000 7fe6c1a00640 1 -- 172.21.15.40:0/2898709712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6bc142f90 con 0x7fe6bc0b1170 2024-04-26T23:03:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.568+0000 7fe6c1000640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6bc06dd40 0x7fe6bc15a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.568+0000 7fe6c1a00640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6bc0b1170 0x7fe6bc144050 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400c3c0 tx=0x7fe6b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.569+0000 7fe6b9600640 1 -- 172.21.15.40:0/2898709712 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4017070 con 0x7fe6bc0b1170 2024-04-26T23:03:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.569+0000 7fe6b9600640 1 -- 172.21.15.40:0/2898709712 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b40035f0 con 0x7fe6bc0b1170 2024-04-26T23:03:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.569+0000 7fe6b9600640 1 -- 172.21.15.40:0/2898709712 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40039e0 con 0x7fe6bc0b1170 2024-04-26T23:03:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.569+0000 7fe6c2400640 1 -- 172.21.15.40:0/2898709712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6bc143270 con 0x7fe6bc0b1170 2024-04-26T23:03:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.569+0000 7fe6c2400640 1 -- 172.21.15.40:0/2898709712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6bc143830 con 0x7fe6bc0b1170 2024-04-26T23:03:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.571+0000 7fe6b9600640 1 -- 172.21.15.40:0/2898709712 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe6b401a070 con 0x7fe6bc0b1170 2024-04-26T23:03:21.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.572+0000 7fe6b9600640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe69806c920 0x7fe69806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.572+0000 7fe6b9600640 1 -- 172.21.15.40:0/2898709712 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fe6b408b0d0 con 0x7fe6bc0b1170 2024-04-26T23:03:21.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.573+0000 7fe6c1000640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe69806c920 0x7fe69806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.573+0000 7fe6c69a6740 1 Processor -- start 2024-04-26T23:03:21.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.573+0000 7fe6c69a6740 1 -- start start 2024-04-26T23:03:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.574+0000 7fe6c69a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 0x5649eaaf5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.574+0000 7fe6c69a6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 0x5649eab13500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.574+0000 7fe6c69a6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaacb3f0 0x5649eab11670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.574+0000 7fe6c69a6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5649ea9db030 con 0x5649eab12fb0 2024-04-26T23:03:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.574+0000 7fe6c69a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5649ea9f67f0 con 0x5649eaaf5710 2024-04-26T23:03:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.574+0000 7fe6c69a6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5649ea9b17e0 con 0x5649eaacb3f0 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.574+0000 7fe6c1000640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe69806c920 0x7fe69806ede0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac007eb0 tx=0x7fe6ac007400 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.574+0000 7fe6b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 0x5649eab13500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.574+0000 7fe6b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 0x5649eab13500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39624/0 (socket says 172.21.15.40:39624) 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.574+0000 7fe6b3e00640 1 -- 172.21.15.40:0/1538935214 learned_addr learned my addr 172.21.15.40:0/1538935214 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.575+0000 7fe6c1a00640 1 --2- 172.21.15.40:0/1538935214 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaacb3f0 0x5649eab11670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.575+0000 7fe6b3e00640 1 -- 172.21.15.40:0/1538935214 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaacb3f0 msgr2=0x5649eab11670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.575+0000 7fe6b3e00640 1 --2- 172.21.15.40:0/1538935214 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaacb3f0 0x5649eab11670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.575+0000 7fe6b3e00640 1 -- 172.21.15.40:0/1538935214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 msgr2=0x5649eaaf5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.575+0000 7fe6b3e00640 1 --2- 172.21.15.40:0/1538935214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 0x5649eaaf5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.575+0000 7fe6b3e00640 1 -- 172.21.15.40:0/1538935214 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649ea9bd050 con 0x5649eab12fb0 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.575+0000 7fe6b3e00640 1 --2- 172.21.15.40:0/1538935214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 0x5649eab13500 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40090b0 tx=0x7fe6a402c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769f8fa3f0a23354 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.575+0000 7fe6b1600640 1 -- 172.21.15.40:0/1538935214 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4037040 con 0x5649eab12fb0 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.575+0000 7fe6b1600640 1 -- 172.21.15.40:0/1538935214 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6a4042070 con 0x5649eab12fb0 2024-04-26T23:03:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.575+0000 7fe6b1600640 1 -- 172.21.15.40:0/1538935214 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a40328e0 con 0x5649eab12fb0 2024-04-26T23:03:21.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.576+0000 7fe6c69a6740 1 -- 172.21.15.40:0/1538935214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 msgr2=0x5649eab13500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.576+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/1538935214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 0x5649eab13500 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40090b0 tx=0x7fe6a402c070 comp rx=0 tx=0).stop 2024-04-26T23:03:21.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.576+0000 7fe6c69a6740 1 -- 172.21.15.40:0/1538935214 shutdown_connections 2024-04-26T23:03:21.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.576+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/1538935214 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaacb3f0 0x5649eab11670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.576+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/1538935214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 0x5649eab13500 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.576+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/1538935214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 0x5649eaaf5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.576+0000 7fe6c69a6740 1 -- 172.21.15.40:0/1538935214 >> 172.21.15.40:0/1538935214 conn(0x5649eaac7d20 msgr2=0x5649eab10520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.576+0000 7fe6c69a6740 1 -- 172.21.15.40:0/1538935214 shutdown_connections 2024-04-26T23:03:21.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.576+0000 7fe6c69a6740 1 -- 172.21.15.40:0/1538935214 wait complete. 2024-04-26T23:03:21.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.577+0000 7fe6c69a6740 1 -- start start 2024-04-26T23:03:21.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.577+0000 7fe6c69a6740 1 Processor -- start 2024-04-26T23:03:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.578+0000 7fe6c69a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 0x5649eaaf66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.578+0000 7fe6c69a6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eab12fb0 0x5649eab201e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.578+0000 7fe6c69a6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 0x5649eaaf81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.578+0000 7fe6c69a6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5649eaa33200 con 0x5649eaae3110 2024-04-26T23:03:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.578+0000 7fe6c69a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5649eaa2dee0 con 0x5649eaaf5710 2024-04-26T23:03:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.578+0000 7fe6c69a6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5649eaa3ae70 con 0x5649eab12fb0 2024-04-26T23:03:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.578+0000 7fe6c69a6740 10 client.0 _ll_register_callbacks cb 0x5649eaa24270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.578+0000 7fe6c1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 0x5649eaaf81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.578+0000 7fe6c1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 0x5649eaaf81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39640/0 (socket says 172.21.15.40:39640) 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.578+0000 7fe6c1a00640 1 -- 172.21.15.40:0/4176719208 learned_addr learned my addr 172.21.15.40:0/4176719208 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.579+0000 7fe6c1a00640 1 -- 172.21.15.40:0/4176719208 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eab12fb0 msgr2=0x5649eab201e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.579+0000 7fe6c1a00640 1 --2- 172.21.15.40:0/4176719208 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eab12fb0 0x5649eab201e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.579+0000 7fe6c1a00640 1 -- 172.21.15.40:0/4176719208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 msgr2=0x5649eaaf66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.579+0000 7fe6c1a00640 1 --2- 172.21.15.40:0/4176719208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 0x5649eaaf66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.579+0000 7fe6c1a00640 1 -- 172.21.15.40:0/4176719208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649ea80a7d0 con 0x5649eaae3110 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.579+0000 7fe6c1a00640 1 --2- 172.21.15.40:0/4176719208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 0x5649eaaf81e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4055030 tx=0x7fe6b4058d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54ebc3573153341 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.579+0000 7fe6aa000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.579+0000 7fe6aa000640 1 -- 172.21.15.40:0/4176719208 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4056f90 con 0x5649eaae3110 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.579+0000 7fe6aa000640 1 -- 172.21.15.40:0/4176719208 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fe6b4058020 con 0x5649eaae3110 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.579+0000 7fe6c69a6740 1 -- 172.21.15.40:0/4176719208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6a4042070 con 0x5649eaae3110 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.579+0000 7fe6aa000640 1 -- 172.21.15.40:0/4176719208 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe6b4071180 con 0x5649eaae3110 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.579+0000 7fe6c69a6740 10 client.5661 fetch_fsmap learned FSMap version 25 2024-04-26T23:03:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.579+0000 7fe6c69a6740 1 -- 172.21.15.40:0/4176719208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x5649ea7403f0 con 0x5649eaae3110 2024-04-26T23:03:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.580+0000 7fe6aa000640 1 -- 172.21.15.40:0/4176719208 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6a4042070 con 0x5649eaae3110 2024-04-26T23:03:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.580+0000 7fe6aa000640 1 -- 172.21.15.40:0/4176719208 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 3546+0+0 (secure 0 0 0) 0x7fe6b405f180 con 0x5649eaae3110 2024-04-26T23:03:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.580+0000 7fe6c69a6740 10 client.5661 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T23:03:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.580+0000 7fe6c69a6740 10 client.5661 resolve_mds: resolved 2:1 to role '2:1' aka daemon mds.c 2024-04-26T23:03:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.580+0000 7fe6c69a6740 20 client.5661 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.580+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/4176719208 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x5649eab13f70 0x5649eaaf0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.580+0000 7fe6c69a6740 4 client.5661 mds_command: new command op to 5331 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:03:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.580+0000 7fe6c69a6740 1 -- 172.21.15.40:0/4176719208 --> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649eaa3c7e0 con 0x5649eab13f70 2024-04-26T23:03:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.580+0000 7fe6c1000640 1 --2- 172.21.15.40:0/4176719208 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x5649eab13f70 0x5649eaaf0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.581+0000 7fe6c1000640 1 --2- 172.21.15.40:0/4176719208 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x5649eab13f70 0x5649eaaf0d30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cfd72aedaf2d56d5 server_cookie=8e8f67a71780c90c in_seq=0 out_seq=0 2024-04-26T23:03:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.581+0000 7fe6aa000640 10 client.5661 ms_handle_connect on v2:172.21.15.40:6832/716981348 2024-04-26T23:03:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.581+0000 7fe6aa000640 1 -- 172.21.15.40:0/4176719208 <== mds.1 v2:172.21.15.40:6832/716981348 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5649eaa3c7e0 con 0x5649eab13f70 2024-04-26T23:03:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.581+0000 7fe6aa000640 10 client.5661 handle_command_reply: tid=0 2024-04-26T23:03:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.581+0000 7fe6c69a6740 1 client.5661 shutdown 2024-04-26T23:03:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.582+0000 7fe6c69a6740 1 -- 172.21.15.40:0/4176719208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 msgr2=0x5649eaaf81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.582+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/4176719208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 0x5649eaaf81e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4055030 tx=0x7fe6b4058d10 comp rx=0 tx=0).stop 2024-04-26T23:03:21.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.582+0000 7fe6c69a6740 1 -- 172.21.15.40:0/4176719208 shutdown_connections 2024-04-26T23:03:21.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.582+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/4176719208 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x5649eab13f70 0x5649eaaf0d30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.582+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/4176719208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 0x5649eaaf81e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.582+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/4176719208 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eab12fb0 0x5649eab201e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.582+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/4176719208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 0x5649eaaf66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.582+0000 7fe6aa000640 0 client.5661 ms_handle_reset on v2:172.21.15.40:6832/716981348 2024-04-26T23:03:21.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.582+0000 7fe6c69a6740 1 -- 172.21.15.40:0/4176719208 >> 172.21.15.40:0/4176719208 conn(0x5649eaacb3f0 msgr2=0x5649eaaca930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.582+0000 7fe6c69a6740 1 -- 172.21.15.40:0/4176719208 shutdown_connections 2024-04-26T23:03:21.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.582+0000 7fe6c69a6740 1 -- 172.21.15.40:0/4176719208 wait complete. 2024-04-26T23:03:21.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.582+0000 7fe6c69a6740 20 client.5661 trim_cache size 0 max 16384 2024-04-26T23:03:21.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.598+0000 7fe6c69a6740 1 Processor -- start 2024-04-26T23:03:21.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.598+0000 7fe6c69a6740 1 -- start start 2024-04-26T23:03:21.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.599+0000 7fe6c69a6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 0x5649eaaf17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.599+0000 7fe6c69a6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaaf5710 0x5649eaaca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.599+0000 7fe6c69a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eab12fb0 0x5649eab05eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.599+0000 7fe6c69a6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc201cb0 con 0x5649eaae3110 2024-04-26T23:03:21.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.599+0000 7fe6c69a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc200e80 con 0x5649eab12fb0 2024-04-26T23:03:21.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.599+0000 7fe6c69a6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe6bc201580 con 0x5649eaaf5710 2024-04-26T23:03:21.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.599+0000 7fe6c1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 0x5649eaaf17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.600+0000 7fe6c1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 0x5649eaaf17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39656/0 (socket says 172.21.15.40:39656) 2024-04-26T23:03:21.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.600+0000 7fe6c1000640 1 -- 172.21.15.40:0/3462190396 learned_addr learned my addr 172.21.15.40:0/3462190396 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.600+0000 7fe6c1000640 1 -- 172.21.15.40:0/3462190396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaaf5710 msgr2=0x5649eaaca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.600+0000 7fe6c1000640 1 --2- 172.21.15.40:0/3462190396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaaf5710 0x5649eaaca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.600+0000 7fe6c1000640 1 -- 172.21.15.40:0/3462190396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eab12fb0 msgr2=0x5649eab05eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.600+0000 7fe6c1a00640 1 --2- 172.21.15.40:0/3462190396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eab12fb0 0x5649eab05eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.600+0000 7fe6c1000640 1 --2- 172.21.15.40:0/3462190396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eab12fb0 0x5649eab05eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.600+0000 7fe6c1000640 1 -- 172.21.15.40:0/3462190396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649ea7000f0 con 0x5649eaae3110 2024-04-26T23:03:21.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.600+0000 7fe6c1000640 1 --2- 172.21.15.40:0/3462190396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 0x5649eaaf17f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac010480 tx=0x7fe6ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfdd4127ce4fb345 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.600+0000 7fe6b0c00640 1 -- 172.21.15.40:0/3462190396 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac018040 con 0x5649eaae3110 2024-04-26T23:03:21.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.601+0000 7fe6b0c00640 1 -- 172.21.15.40:0/3462190396 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6ac022e50 con 0x5649eaae3110 2024-04-26T23:03:21.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.601+0000 7fe6b0c00640 1 -- 172.21.15.40:0/3462190396 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac007b20 con 0x5649eaae3110 2024-04-26T23:03:21.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.601+0000 7fe6c69a6740 1 -- 172.21.15.40:0/3462190396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 msgr2=0x5649eaaf17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.601+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/3462190396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 0x5649eaaf17f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac010480 tx=0x7fe6ac004a40 comp rx=0 tx=0).stop 2024-04-26T23:03:21.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.601+0000 7fe6c69a6740 1 -- 172.21.15.40:0/3462190396 shutdown_connections 2024-04-26T23:03:21.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.601+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/3462190396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eab12fb0 0x5649eab05eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.601+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/3462190396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaaf5710 0x5649eaaca9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.601+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/3462190396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eaae3110 0x5649eaaf17f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.601+0000 7fe6c69a6740 1 -- 172.21.15.40:0/3462190396 >> 172.21.15.40:0/3462190396 conn(0x5649eaacb3f0 msgr2=0x5649eaaf4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.601+0000 7fe6c69a6740 1 -- 172.21.15.40:0/3462190396 shutdown_connections 2024-04-26T23:03:21.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.601+0000 7fe6c69a6740 1 -- 172.21.15.40:0/3462190396 wait complete. 2024-04-26T23:03:21.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.602+0000 7fe6c69a6740 1 -- start start 2024-04-26T23:03:21.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.602+0000 7fe6c69a6740 1 Processor -- start 2024-04-26T23:03:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.603+0000 7fe6c69a6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaae3110 0x5649eab72cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.603+0000 7fe6c69a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 0x5649eaaf4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.603+0000 7fe6c69a6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 0x5649eaaf4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.603+0000 7fe6c69a6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5649eaa3bfd0 con 0x5649eab12fb0 2024-04-26T23:03:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.603+0000 7fe6c69a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc1f5e30 con 0x5649eaaf5710 2024-04-26T23:03:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.603+0000 7fe6c69a6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5649eaac88c0 con 0x5649eaae3110 2024-04-26T23:03:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.603+0000 7fe6c69a6740 10 client.0 _ll_register_callbacks cb 0x7fe6b4054ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.603+0000 7fe6c1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 0x5649eaaf4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.603+0000 7fe6c1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 0x5649eaaf4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39658/0 (socket says 172.21.15.40:39658) 2024-04-26T23:03:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.603+0000 7fe6c1a00640 1 -- 172.21.15.40:0/1390943179 learned_addr learned my addr 172.21.15.40:0/1390943179 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.604+0000 7fe6b3e00640 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 0x5649eaaf4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.604+0000 7fe6c1000640 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaae3110 0x5649eab72cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.604+0000 7fe6c1a00640 1 -- 172.21.15.40:0/1390943179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaae3110 msgr2=0x5649eab72cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.604+0000 7fe6c1a00640 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaae3110 0x5649eab72cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.604+0000 7fe6c1a00640 1 -- 172.21.15.40:0/1390943179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 msgr2=0x5649eaaf4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.604+0000 7fe6c1a00640 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 0x5649eaaf4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.604+0000 7fe6c1a00640 1 -- 172.21.15.40:0/1390943179 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649ea9f6a30 con 0x5649eab12fb0 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.604+0000 7fe6b3e00640 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 0x5649eaaf4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.604+0000 7fe6c1a00640 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 0x5649eaaf4cb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400cd80 tx=0x7fe6b40892a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe084243a876000 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.604+0000 7fe6aa000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.604+0000 7fe6aa000640 1 -- 172.21.15.40:0/1390943179 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4058220 con 0x5649eab12fb0 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.604+0000 7fe6aa000640 1 -- 172.21.15.40:0/1390943179 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fe6b4073c40 con 0x5649eab12fb0 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.604+0000 7fe6c69a6740 1 -- 172.21.15.40:0/1390943179 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6ac022e50 con 0x5649eab12fb0 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.605+0000 7fe6aa000640 1 -- 172.21.15.40:0/1390943179 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe6b4056f90 con 0x5649eab12fb0 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.605+0000 7fe6c69a6740 10 client.5667 fetch_fsmap learned FSMap version 25 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.605+0000 7fe6c69a6740 1 -- 172.21.15.40:0/1390943179 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x5649eab8d640 con 0x5649eab12fb0 2024-04-26T23:03:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.605+0000 7fe6aa000640 1 -- 172.21.15.40:0/1390943179 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6ac022e50 con 0x5649eab12fb0 2024-04-26T23:03:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.605+0000 7fe6aa000640 1 -- 172.21.15.40:0/1390943179 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 3546+0+0 (secure 0 0 0) 0x7fe6b4054920 con 0x5649eab12fb0 2024-04-26T23:03:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.605+0000 7fe6c69a6740 10 client.5667 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T23:03:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.605+0000 7fe6c69a6740 10 client.5667 resolve_mds: resolved 2:1 to role '2:1' aka daemon mds.c 2024-04-26T23:03:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.605+0000 7fe6c69a6740 20 client.5667 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.605+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x5649eaac5d70 0x5649eab1eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.605+0000 7fe6c69a6740 4 client.5667 mds_command: new command op to 5331 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:03:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.605+0000 7fe6c69a6740 1 -- 172.21.15.40:0/1390943179 --> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5649eaa36650 con 0x5649eaac5d70 2024-04-26T23:03:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.605+0000 7fe6c1000640 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x5649eaac5d70 0x5649eab1eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.606+0000 7fe6c1000640 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x5649eaac5d70 0x5649eab1eaa0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=944d8b2a8ca35d73 server_cookie=67cf0ccc75680b57 in_seq=0 out_seq=0 2024-04-26T23:03:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.606+0000 7fe6aa000640 10 client.5667 ms_handle_connect on v2:172.21.15.40:6832/716981348 2024-04-26T23:03:21.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.625+0000 7fe6aa000640 1 -- 172.21.15.40:0/1390943179 <== mds.1 v2:172.21.15.40:6832/716981348 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5649eaa36650 con 0x5649eaac5d70 2024-04-26T23:03:21.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.625+0000 7fe6aa000640 10 client.5667 handle_command_reply: tid=0 2024-04-26T23:03:21.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.625+0000 7fe6c69a6740 1 client.5667 shutdown 2024-04-26T23:03:21.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.625+0000 7fe6c69a6740 1 -- 172.21.15.40:0/1390943179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 msgr2=0x5649eaaf4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.625+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 0x5649eaaf4cb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400cd80 tx=0x7fe6b40892a0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.625+0000 7fe6c69a6740 1 -- 172.21.15.40:0/1390943179 shutdown_connections 2024-04-26T23:03:21.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.625+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.40:6832/716981348,v1:172.21.15.40:6834/716981348] conn(0x5649eaac5d70 0x5649eab1eaa0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.625+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5649eab12fb0 0x5649eaaf4cb0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.625+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5649eaaf5710 0x5649eaaf4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.625+0000 7fe6c69a6740 1 --2- 172.21.15.40:0/1390943179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5649eaae3110 0x5649eab72cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.625+0000 7fe6aa000640 0 client.5667 ms_handle_reset on v2:172.21.15.40:6832/716981348 2024-04-26T23:03:21.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.625+0000 7fe6c69a6740 1 -- 172.21.15.40:0/1390943179 >> 172.21.15.40:0/1390943179 conn(0x5649eaacb3f0 msgr2=0x5649eaaca9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.625+0000 7fe6c69a6740 1 -- 172.21.15.40:0/1390943179 shutdown_connections 2024-04-26T23:03:21.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.626+0000 7fe6c69a6740 1 -- 172.21.15.40:0/1390943179 wait complete. 2024-04-26T23:03:21.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.626+0000 7fe6c69a6740 20 client.5667 trim_cache size 0 max 16384 2024-04-26T23:03:21.627 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:03:21.627 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:03:21.627 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:03:21.627 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:03:21.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.627+0000 7fe6c2400640 1 -- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe69806c920 msgr2=0x7fe69806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.627+0000 7fe6c2400640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe69806c920 0x7fe69806ede0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac007eb0 tx=0x7fe6ac007400 comp rx=0 tx=0).stop 2024-04-26T23:03:21.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.627+0000 7fe6c2400640 1 -- 172.21.15.40:0/2898709712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6bc0b1170 msgr2=0x7fe6bc144050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.627+0000 7fe6c2400640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6bc0b1170 0x7fe6bc144050 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400c3c0 tx=0x7fe6b400c880 comp rx=0 tx=0).stop 2024-04-26T23:03:21.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.627+0000 7fe6c2400640 1 -- 172.21.15.40:0/2898709712 shutdown_connections 2024-04-26T23:03:21.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.627+0000 7fe6c2400640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe69806c920 0x7fe69806ede0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.627+0000 7fe6c2400640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6bc0b1170 0x7fe6bc144050 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.627+0000 7fe6c2400640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe6bc160a50 0x7fe6bc0b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.627+0000 7fe6c2400640 1 --2- 172.21.15.40:0/2898709712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6bc06dd40 0x7fe6bc15a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.627+0000 7fe6c2400640 1 -- 172.21.15.40:0/2898709712 >> 172.21.15.40:0/2898709712 conn(0x7fe6bc0b2720 msgr2=0x7fe6bc0d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.627+0000 7fe6c2400640 1 -- 172.21.15.40:0/2898709712 shutdown_connections 2024-04-26T23:03:21.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.627+0000 7fe6c2400640 1 -- 172.21.15.40:0/2898709712 wait complete. 2024-04-26T23:03:21.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:2 flush journal 2024-04-26T23:03:21.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.851+0000 7f7beea00640 1 Processor -- start 2024-04-26T23:03:21.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.852+0000 7f7beea00640 1 -- start start 2024-04-26T23:03:21.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.853+0000 7f7beea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7be80ccd00 0x7f7be80b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.853+0000 7f7beea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7be80ce450 0x7f7be81611b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.853+0000 7f7beea00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be80cd0e0 con 0x7f7be81616f0 2024-04-26T23:03:21.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.853+0000 7f7beea00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f7be8160a40 con 0x7f7be80ccd00 2024-04-26T23:03:21.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.853+0000 7f7beea00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7be815d300 con 0x7f7be80ce450 2024-04-26T23:03:21.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.854+0000 7f7bee000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7be81616f0 0x7f7be80cb920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49406/0 (socket says 172.21.15.40:49406) 2024-04-26T23:03:21.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.854+0000 7f7bee000640 1 -- 172.21.15.40:0/2053180978 learned_addr learned my addr 172.21.15.40:0/2053180978 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:21.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.854+0000 7f7bed600640 1 --2- 172.21.15.40:0/2053180978 >> v2:172.21.15.40:3300/0 conn(0x7f7be80ccd00 0x7f7be80b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.854+0000 7f7becc00640 1 --2- 172.21.15.40:0/2053180978 >> v2:172.21.15.103:3300/0 conn(0x7f7be80ce450 0x7f7be81611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.854+0000 7f7be7e00640 1 -- 172.21.15.40:0/2053180978 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 926297856 0 0) 0x7f7be0001490 con 0x7f7be81616f0 2024-04-26T23:03:21.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.854+0000 7f7be7e00640 1 -- 172.21.15.40:0/2053180978 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bd0003520 con 0x7f7be81616f0 2024-04-26T23:03:21.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.855+0000 7f7bed600640 1 -- 172.21.15.40:0/2053180978 >> v2:172.21.15.103:3300/0 conn(0x7f7be80ce450 msgr2=0x7f7be81611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.855+0000 7f7bed600640 1 --2- 172.21.15.40:0/2053180978 >> v2:172.21.15.103:3300/0 conn(0x7f7be80ce450 0x7f7be81611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.855+0000 7f7bed600640 1 -- 172.21.15.40:0/2053180978 >> v1:172.21.15.40:6789/0 conn(0x7f7be81616f0 legacy=0x7f7be80cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.855+0000 7f7bed600640 1 -- 172.21.15.40:0/2053180978 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be80b25d0 con 0x7f7be80ccd00 2024-04-26T23:03:21.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.856+0000 7f7bed600640 1 --2- 172.21.15.40:0/2053180978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be80ccd00 0x7f7be80b1ef0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7be80d2390 tx=0x7f7bdc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0337d32881daa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.856+0000 7f7be7e00640 1 -- 172.21.15.40:0/2053180978 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc037070 con 0x7f7be80ccd00 2024-04-26T23:03:21.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.856+0000 7f7be7e00640 1 -- 172.21.15.40:0/2053180978 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bdc032440 con 0x7f7be80ccd00 2024-04-26T23:03:21.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.856+0000 7f7be7e00640 1 -- 172.21.15.40:0/2053180978 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc032810 con 0x7f7be80ccd00 2024-04-26T23:03:21.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.856+0000 7f7beea00640 1 -- 172.21.15.40:0/2053180978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be80ccd00 msgr2=0x7f7be80b1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.856+0000 7f7beea00640 1 --2- 172.21.15.40:0/2053180978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be80ccd00 0x7f7be80b1ef0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7be80d2390 tx=0x7f7bdc02a1a0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.857+0000 7f7beea00640 1 -- 172.21.15.40:0/2053180978 shutdown_connections 2024-04-26T23:03:21.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.857+0000 7f7beea00640 1 --2- 172.21.15.40:0/2053180978 >> v2:172.21.15.103:3300/0 conn(0x7f7be80ce450 0x7f7be81611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.857+0000 7f7beea00640 1 --2- 172.21.15.40:0/2053180978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be80ccd00 0x7f7be80b1ef0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.857+0000 7f7beea00640 1 -- 172.21.15.40:0/2053180978 >> 172.21.15.40:0/2053180978 conn(0x7f7be80b2ee0 msgr2=0x7f7be80b32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.857+0000 7f7beea00640 1 -- 172.21.15.40:0/2053180978 shutdown_connections 2024-04-26T23:03:21.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.857+0000 7f7beea00640 1 -- 172.21.15.40:0/2053180978 wait complete. 2024-04-26T23:03:21.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.858+0000 7f7beea00640 1 Processor -- start 2024-04-26T23:03:21.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.859+0000 7f7beea00640 1 -- start start 2024-04-26T23:03:21.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7beea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be80ccd00 0x7f7be80ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7beea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7be80ce450 0x7f7be815f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7beea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be81f6ab0 0x7f7be80b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7beea00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7be8160c90 con 0x7f7be80ccd00 2024-04-26T23:03:21.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7beea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7be80b2b90 con 0x7f7be81f6ab0 2024-04-26T23:03:21.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7beea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7be80cc030 con 0x7f7be80ce450 2024-04-26T23:03:21.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7bed600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be80ccd00 0x7f7be80ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7bed600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be80ccd00 0x7f7be80ca8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39680/0 (socket says 172.21.15.40:39680) 2024-04-26T23:03:21.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7bed600640 1 -- 172.21.15.40:0/3923248889 learned_addr learned my addr 172.21.15.40:0/3923248889 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7bed600640 1 -- 172.21.15.40:0/3923248889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7be80ce450 msgr2=0x7f7be815f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7bed600640 1 --2- 172.21.15.40:0/3923248889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7be80ce450 0x7f7be815f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7bed600640 1 -- 172.21.15.40:0/3923248889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be81f6ab0 msgr2=0x7f7be80b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7bed600640 1 --2- 172.21.15.40:0/3923248889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be81f6ab0 0x7f7be80b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.860+0000 7f7bed600640 1 -- 172.21.15.40:0/3923248889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be8151b10 con 0x7f7be80ccd00 2024-04-26T23:03:21.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.861+0000 7f7bed600640 1 --2- 172.21.15.40:0/3923248889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be80ccd00 0x7f7be80ca8e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc000c00 tx=0x7f7bdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.861+0000 7f7be5600640 1 -- 172.21.15.40:0/3923248889 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc037040 con 0x7f7be80ccd00 2024-04-26T23:03:21.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.861+0000 7f7be5600640 1 -- 172.21.15.40:0/3923248889 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bdc042070 con 0x7f7be80ccd00 2024-04-26T23:03:21.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.861+0000 7f7be5600640 1 -- 172.21.15.40:0/3923248889 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc030960 con 0x7f7be80ccd00 2024-04-26T23:03:21.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.861+0000 7f7beea00640 1 -- 172.21.15.40:0/3923248889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be80c9840 con 0x7f7be80ccd00 2024-04-26T23:03:21.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.861+0000 7f7beea00640 1 -- 172.21.15.40:0/3923248889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be80c9e00 con 0x7f7be80ccd00 2024-04-26T23:03:21.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.864+0000 7f7be5600640 1 -- 172.21.15.40:0/3923248889 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7bdc030b00 con 0x7f7be80ccd00 2024-04-26T23:03:21.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.865+0000 7f7be5600640 1 --2- 172.21.15.40:0/3923248889 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7bc806c940 0x7f7bc806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.865+0000 7f7be5600640 1 -- 172.21.15.40:0/3923248889 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f7bdc0ac690 con 0x7f7be80ccd00 2024-04-26T23:03:21.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.866+0000 7f7becc00640 1 --2- 172.21.15.40:0/3923248889 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7bc806c940 0x7f7bc806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.866+0000 7f7bf2db5740 1 Processor -- start 2024-04-26T23:03:21.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.866+0000 7f7bf2db5740 1 -- start start 2024-04-26T23:03:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.867+0000 7f7bf2db5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f9763cc0 0x55f7f973c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.867+0000 7f7bf2db5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f9766520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.867+0000 7f7bf2db5740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f9766a60 0x55f7f973a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.867+0000 7f7bf2db5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f7f964c030 con 0x55f7f97640a0 2024-04-26T23:03:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.867+0000 7f7bf2db5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f7f96677f0 con 0x55f7f9763cc0 2024-04-26T23:03:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.867+0000 7f7bf2db5740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55f7f96227e0 con 0x55f7f9766a60 2024-04-26T23:03:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.867+0000 7f7becc00640 1 --2- 172.21.15.40:0/3923248889 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7bc806c940 0x7f7bc806ee00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7be8160680 tx=0x7f7bd4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.868+0000 7f7becc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f9766520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.868+0000 7f7becc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f9766520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39686/0 (socket says 172.21.15.40:39686) 2024-04-26T23:03:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.868+0000 7f7becc00640 1 -- 172.21.15.40:0/3744183612 learned_addr learned my addr 172.21.15.40:0/3744183612 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.868+0000 7f7bee000640 1 --2- 172.21.15.40:0/3744183612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f9766a60 0x55f7f973a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.868+0000 7f7bed600640 1 --2- 172.21.15.40:0/3744183612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f9763cc0 0x55f7f973c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.868+0000 7f7becc00640 1 -- 172.21.15.40:0/3744183612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f9766a60 msgr2=0x55f7f973a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.868+0000 7f7becc00640 1 --2- 172.21.15.40:0/3744183612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f9766a60 0x55f7f973a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.868+0000 7f7becc00640 1 -- 172.21.15.40:0/3744183612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f9763cc0 msgr2=0x55f7f973c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.868+0000 7f7becc00640 1 --2- 172.21.15.40:0/3744183612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f9763cc0 0x55f7f973c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.868+0000 7f7becc00640 1 -- 172.21.15.40:0/3744183612 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7f962e050 con 0x55f7f97640a0 2024-04-26T23:03:21.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.868+0000 7f7becc00640 1 --2- 172.21.15.40:0/3744183612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f9766520 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400d890 tx=0x7f7bd4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de1677c7b1cffca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.869+0000 7f7bd9c00640 1 -- 172.21.15.40:0/3744183612 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4024070 con 0x55f7f97640a0 2024-04-26T23:03:21.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.869+0000 7f7bd9c00640 1 -- 172.21.15.40:0/3744183612 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd400de10 con 0x55f7f97640a0 2024-04-26T23:03:21.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.869+0000 7f7bd9c00640 1 -- 172.21.15.40:0/3744183612 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4014dc0 con 0x55f7f97640a0 2024-04-26T23:03:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.869+0000 7f7bf2db5740 1 -- 172.21.15.40:0/3744183612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 msgr2=0x55f7f9766520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.869+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/3744183612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f9766520 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400d890 tx=0x7f7bd4010000 comp rx=0 tx=0).stop 2024-04-26T23:03:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.869+0000 7f7bf2db5740 1 -- 172.21.15.40:0/3744183612 shutdown_connections 2024-04-26T23:03:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.869+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/3744183612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f9766a60 0x55f7f973a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.869+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/3744183612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f9766520 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.869+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/3744183612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f9763cc0 0x55f7f973c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.869+0000 7f7bf2db5740 1 -- 172.21.15.40:0/3744183612 >> 172.21.15.40:0/3744183612 conn(0x55f7f9738d20 msgr2=0x55f7f9781520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.870+0000 7f7bf2db5740 1 -- 172.21.15.40:0/3744183612 shutdown_connections 2024-04-26T23:03:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.870+0000 7f7bf2db5740 1 -- 172.21.15.40:0/3744183612 wait complete. 2024-04-26T23:03:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.870+0000 7f7bf2db5740 1 -- start start 2024-04-26T23:03:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.870+0000 7f7bf2db5740 1 Processor -- start 2024-04-26T23:03:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.871+0000 7f7bf2db5740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f97640a0 0x55f7f9780700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.871+0000 7f7bf2db5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f9766a60 0x55f7f9780c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.871+0000 7f7bf2db5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f976a550 0x55f7f9781180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.871+0000 7f7bf2db5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f7f96a4200 con 0x55f7f9766a60 2024-04-26T23:03:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.871+0000 7f7bf2db5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f7f969eee0 con 0x55f7f976a550 2024-04-26T23:03:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.871+0000 7f7bf2db5740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55f7f96abe70 con 0x55f7f97640a0 2024-04-26T23:03:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.872+0000 7f7bf2db5740 10 client.0 _ll_register_callbacks cb 0x55f7f9695270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.872+0000 7f7becc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f9766a60 0x55f7f9780c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.872+0000 7f7becc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f9766a60 0x55f7f9780c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39698/0 (socket says 172.21.15.40:39698) 2024-04-26T23:03:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.872+0000 7f7becc00640 1 -- 172.21.15.40:0/143923048 learned_addr learned my addr 172.21.15.40:0/143923048 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.872+0000 7f7bed600640 1 --2- 172.21.15.40:0/143923048 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f97640a0 0x55f7f9780700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.872+0000 7f7bee000640 1 --2- 172.21.15.40:0/143923048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f976a550 0x55f7f9781180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.872+0000 7f7becc00640 1 -- 172.21.15.40:0/143923048 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f97640a0 msgr2=0x55f7f9780700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.872+0000 7f7becc00640 1 --2- 172.21.15.40:0/143923048 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f97640a0 0x55f7f9780700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.872+0000 7f7becc00640 1 -- 172.21.15.40:0/143923048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f976a550 msgr2=0x55f7f9781180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.872+0000 7f7becc00640 1 --2- 172.21.15.40:0/143923048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f976a550 0x55f7f9781180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.872+0000 7f7becc00640 1 -- 172.21.15.40:0/143923048 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7f947b7d0 con 0x55f7f9766a60 2024-04-26T23:03:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.873+0000 7f7becc00640 1 --2- 172.21.15.40:0/143923048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f9766a60 0x55f7f9780c40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4017040 tx=0x7f7bd4006e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b233e9be163940 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.873+0000 7f7bce000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.873+0000 7f7bce000640 1 -- 172.21.15.40:0/143923048 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4024040 con 0x55f7f9766a60 2024-04-26T23:03:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.873+0000 7f7bce000640 1 -- 172.21.15.40:0/143923048 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f7bd401f740 con 0x55f7f9766a60 2024-04-26T23:03:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.873+0000 7f7bf2db5740 1 -- 172.21.15.40:0/143923048 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bd400de10 con 0x55f7f9766a60 2024-04-26T23:03:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.873+0000 7f7bce000640 1 -- 172.21.15.40:0/143923048 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7bd401f940 con 0x55f7f9766a60 2024-04-26T23:03:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.873+0000 7f7bf2db5740 10 client.5682 fetch_fsmap learned FSMap version 25 2024-04-26T23:03:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.874+0000 7f7bf2db5740 1 -- 172.21.15.40:0/143923048 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55f7f93b13f0 con 0x55f7f9766a60 2024-04-26T23:03:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.874+0000 7f7bce000640 1 -- 172.21.15.40:0/143923048 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd400de10 con 0x55f7f9766a60 2024-04-26T23:03:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.874+0000 7f7bce000640 1 -- 172.21.15.40:0/143923048 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 3546+0+0 (secure 0 0 0) 0x7f7bd4039020 con 0x55f7f9766a60 2024-04-26T23:03:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.874+0000 7f7bf2db5740 10 client.5682 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T23:03:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.874+0000 7f7bf2db5740 10 client.5682 resolve_mds: resolved 2:2 to role '2:2' aka daemon mds.b 2024-04-26T23:03:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.874+0000 7f7bf2db5740 20 client.5682 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.874+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/143923048 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55f7f9751020 0x55f7f97691a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.874+0000 7f7bf2db5740 4 client.5682 mds_command: new command op to 4975 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:03:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.874+0000 7f7bf2db5740 1 -- 172.21.15.40:0/143923048 --> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7f96ad7e0 con 0x55f7f9751020 2024-04-26T23:03:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.875+0000 7f7bed600640 1 --2- 172.21.15.40:0/143923048 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55f7f9751020 0x55f7f97691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.876+0000 7f7bed600640 1 --2- 172.21.15.40:0/143923048 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55f7f9751020 0x55f7f97691a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=60fcae0ad61b5a43 server_cookie=5b982c6458b5a71a in_seq=0 out_seq=0 2024-04-26T23:03:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.876+0000 7f7bce000640 10 client.5682 ms_handle_connect on v2:172.21.15.103:6834/603576312 2024-04-26T23:03:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.877+0000 7f7bce000640 1 -- 172.21.15.40:0/143923048 <== mds.2 v2:172.21.15.103:6834/603576312 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7f96ad7e0 con 0x55f7f9751020 2024-04-26T23:03:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.877+0000 7f7bce000640 10 client.5682 handle_command_reply: tid=0 2024-04-26T23:03:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.878+0000 7f7bf2db5740 1 client.5682 shutdown 2024-04-26T23:03:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.878+0000 7f7bf2db5740 1 -- 172.21.15.40:0/143923048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f9766a60 msgr2=0x55f7f9780c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.878+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/143923048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f9766a60 0x55f7f9780c40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4017040 tx=0x7f7bd4006e10 comp rx=0 tx=0).stop 2024-04-26T23:03:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.878+0000 7f7bf2db5740 1 -- 172.21.15.40:0/143923048 shutdown_connections 2024-04-26T23:03:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.878+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/143923048 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55f7f9751020 0x55f7f97691a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.878+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/143923048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f976a550 0x55f7f9781180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.878+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/143923048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f9766a60 0x55f7f9780c40 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.878+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/143923048 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f97640a0 0x55f7f9780700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.878+0000 7f7bce000640 0 client.5682 ms_handle_reset on v2:172.21.15.103:6834/603576312 2024-04-26T23:03:21.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.878+0000 7f7bf2db5740 1 -- 172.21.15.40:0/143923048 >> 172.21.15.40:0/143923048 conn(0x55f7f9763cc0 msgr2=0x55f7f973a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.878+0000 7f7bf2db5740 1 -- 172.21.15.40:0/143923048 shutdown_connections 2024-04-26T23:03:21.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.878+0000 7f7bf2db5740 1 -- 172.21.15.40:0/143923048 wait complete. 2024-04-26T23:03:21.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.878+0000 7f7bf2db5740 20 client.5682 trim_cache size 0 max 16384 2024-04-26T23:03:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.895+0000 7f7bf2db5740 1 Processor -- start 2024-04-26T23:03:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.895+0000 7f7bf2db5740 1 -- start start 2024-04-26T23:03:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bf2db5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f973a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bf2db5740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f9766a60 0x55f7f973a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bf2db5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f976a550 0x55f7f9783630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bf2db5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f7f97672c0 con 0x55f7f97640a0 2024-04-26T23:03:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bf2db5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7be81fd7e0 con 0x55f7f976a550 2024-04-26T23:03:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bf2db5740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7be80c7c50 con 0x55f7f9766a60 2024-04-26T23:03:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bed600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f973a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bed600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f973a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39710/0 (socket says 172.21.15.40:39710) 2024-04-26T23:03:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bed600640 1 -- 172.21.15.40:0/2646692096 learned_addr learned my addr 172.21.15.40:0/2646692096 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bed600640 1 -- 172.21.15.40:0/2646692096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f9766a60 msgr2=0x55f7f973a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bed600640 1 --2- 172.21.15.40:0/2646692096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f9766a60 0x55f7f973a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bed600640 1 -- 172.21.15.40:0/2646692096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f976a550 msgr2=0x55f7f9783630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bed600640 1 --2- 172.21.15.40:0/2646692096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f976a550 0x55f7f9783630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.896+0000 7f7bed600640 1 -- 172.21.15.40:0/2646692096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7f93710f0 con 0x55f7f97640a0 2024-04-26T23:03:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.897+0000 7f7bed600640 1 --2- 172.21.15.40:0/2646692096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f973a2c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc079f10 tx=0x7f7bdc049040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472983d6d0782f18 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.897+0000 7f7bd9200640 1 -- 172.21.15.40:0/2646692096 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc07a210 con 0x55f7f97640a0 2024-04-26T23:03:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.897+0000 7f7bd9200640 1 -- 172.21.15.40:0/2646692096 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bdc07e5c0 con 0x55f7f97640a0 2024-04-26T23:03:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.897+0000 7f7bd9200640 1 -- 172.21.15.40:0/2646692096 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc082df0 con 0x55f7f97640a0 2024-04-26T23:03:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.898+0000 7f7bf2db5740 1 -- 172.21.15.40:0/2646692096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 msgr2=0x55f7f973a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.898+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/2646692096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f973a2c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc079f10 tx=0x7f7bdc049040 comp rx=0 tx=0).stop 2024-04-26T23:03:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.898+0000 7f7bf2db5740 1 -- 172.21.15.40:0/2646692096 shutdown_connections 2024-04-26T23:03:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.898+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/2646692096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f976a550 0x55f7f9783630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.898+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/2646692096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f9766a60 0x55f7f973a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.898+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/2646692096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f973a2c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.898+0000 7f7bf2db5740 1 -- 172.21.15.40:0/2646692096 >> 172.21.15.40:0/2646692096 conn(0x55f7f9763cc0 msgr2=0x55f7f9735d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.898+0000 7f7bf2db5740 1 -- 172.21.15.40:0/2646692096 shutdown_connections 2024-04-26T23:03:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.898+0000 7f7bf2db5740 1 -- 172.21.15.40:0/2646692096 wait complete. 2024-04-26T23:03:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.898+0000 7f7bf2db5740 1 -- start start 2024-04-26T23:03:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.899+0000 7f7bf2db5740 1 Processor -- start 2024-04-26T23:03:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bf2db5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f973a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bf2db5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f9766a60 0x55f7f973a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bf2db5740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f976a550 0x55f7f9760870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bf2db5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7be80b68d0 con 0x55f7f97640a0 2024-04-26T23:03:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bf2db5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7be80b6f20 con 0x55f7f9766a60 2024-04-26T23:03:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bf2db5740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55f7f96acfd0 con 0x55f7f976a550 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bf2db5740 10 client.0 _ll_register_callbacks cb 0x7f7bd4009190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bed600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f973a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bed600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f973a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39724/0 (socket says 172.21.15.40:39724) 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bed600640 1 -- 172.21.15.40:0/1496171729 learned_addr learned my addr 172.21.15.40:0/1496171729 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bed600640 1 -- 172.21.15.40:0/1496171729 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f976a550 msgr2=0x55f7f9760870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bed600640 1 --2- 172.21.15.40:0/1496171729 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f976a550 0x55f7f9760870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bed600640 1 -- 172.21.15.40:0/1496171729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f9766a60 msgr2=0x55f7f973a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bed600640 1 --2- 172.21.15.40:0/1496171729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f9766a60 0x55f7f973a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.900+0000 7f7bed600640 1 -- 172.21.15.40:0/1496171729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7f9667a30 con 0x55f7f97640a0 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.901+0000 7f7bed600640 1 --2- 172.21.15.40:0/1496171729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f973a2c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc079e00 tx=0x7f7bd4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a41d243e5a148e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.901+0000 7f7bce000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.901+0000 7f7bce000640 1 -- 172.21.15.40:0/1496171729 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc082df0 con 0x55f7f97640a0 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.901+0000 7f7bce000640 1 -- 172.21.15.40:0/1496171729 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f7bdc0782b0 con 0x55f7f97640a0 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.901+0000 7f7bf2db5740 1 -- 172.21.15.40:0/1496171729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bdc07e5c0 con 0x55f7f97640a0 2024-04-26T23:03:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.901+0000 7f7bce000640 1 -- 172.21.15.40:0/1496171729 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7bdc078510 con 0x55f7f97640a0 2024-04-26T23:03:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.901+0000 7f7bf2db5740 10 client.5688 fetch_fsmap learned FSMap version 25 2024-04-26T23:03:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.901+0000 7f7bf2db5740 1 -- 172.21.15.40:0/1496171729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55f7f977ede0 con 0x55f7f97640a0 2024-04-26T23:03:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.902+0000 7f7bce000640 1 -- 172.21.15.40:0/1496171729 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bdc07e5c0 con 0x55f7f97640a0 2024-04-26T23:03:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.902+0000 7f7bce000640 1 -- 172.21.15.40:0/1496171729 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 3546+0+0 (secure 0 0 0) 0x7f7bdc03a070 con 0x55f7f97640a0 2024-04-26T23:03:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.902+0000 7f7bf2db5740 10 client.5688 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T23:03:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.902+0000 7f7bf2db5740 10 client.5688 resolve_mds: resolved 2:2 to role '2:2' aka daemon mds.b 2024-04-26T23:03:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.902+0000 7f7bf2db5740 20 client.5688 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.902+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/1496171729 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55f7f9760db0 0x55f7f97611d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.902+0000 7f7bf2db5740 4 client.5688 mds_command: new command op to 4975 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:03:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.902+0000 7f7bf2db5740 1 -- 172.21.15.40:0/1496171729 --> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f7f96a7650 con 0x55f7f9760db0 2024-04-26T23:03:21.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.902+0000 7f7becc00640 1 --2- 172.21.15.40:0/1496171729 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55f7f9760db0 0x55f7f97611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:21.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.903+0000 7f7becc00640 1 --2- 172.21.15.40:0/1496171729 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55f7f9760db0 0x55f7f97611d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=307b3a319266dbd6 server_cookie=144d916e8fd7ebeb in_seq=0 out_seq=0 2024-04-26T23:03:21.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.903+0000 7f7bce000640 10 client.5688 ms_handle_connect on v2:172.21.15.103:6834/603576312 2024-04-26T23:03:21.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.925+0000 7f7bce000640 1 -- 172.21.15.40:0/1496171729 <== mds.2 v2:172.21.15.103:6834/603576312 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f7f96a7650 con 0x55f7f9760db0 2024-04-26T23:03:21.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.925+0000 7f7bce000640 10 client.5688 handle_command_reply: tid=0 2024-04-26T23:03:21.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.925+0000 7f7bf2db5740 1 client.5688 shutdown 2024-04-26T23:03:21.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.925+0000 7f7bf2db5740 1 -- 172.21.15.40:0/1496171729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 msgr2=0x55f7f973a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.925+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/1496171729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f973a2c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc079e00 tx=0x7f7bd4012000 comp rx=0 tx=0).stop 2024-04-26T23:03:21.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.925+0000 7f7bf2db5740 1 -- 172.21.15.40:0/1496171729 shutdown_connections 2024-04-26T23:03:21.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.925+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/1496171729 >> [v2:172.21.15.103:6834/603576312,v1:172.21.15.103:6836/603576312] conn(0x55f7f9760db0 0x55f7f97611d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.925+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/1496171729 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f7f976a550 0x55f7f9760870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.925+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/1496171729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f7f9766a60 0x55f7f973a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.925+0000 7f7bf2db5740 1 --2- 172.21.15.40:0/1496171729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f7f97640a0 0x55f7f973a2c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.925+0000 7f7bf2db5740 1 -- 172.21.15.40:0/1496171729 >> 172.21.15.40:0/1496171729 conn(0x55f7f9763cc0 msgr2=0x55f7f9737b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.925+0000 7f7bce000640 0 client.5688 ms_handle_reset on v2:172.21.15.103:6834/603576312 2024-04-26T23:03:21.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.926+0000 7f7bf2db5740 1 -- 172.21.15.40:0/1496171729 shutdown_connections 2024-04-26T23:03:21.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.926+0000 7f7bf2db5740 1 -- 172.21.15.40:0/1496171729 wait complete. 2024-04-26T23:03:21.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.926+0000 7f7bf2db5740 20 client.5688 trim_cache size 0 max 16384 2024-04-26T23:03:21.927 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:03:21.927 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:03:21.927 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:03:21.928 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:03:21.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.927+0000 7f7beea00640 1 -- 172.21.15.40:0/3923248889 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7bc806c940 msgr2=0x7f7bc806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.927+0000 7f7beea00640 1 --2- 172.21.15.40:0/3923248889 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7bc806c940 0x7f7bc806ee00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7be8160680 tx=0x7f7bd4008040 comp rx=0 tx=0).stop 2024-04-26T23:03:21.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.927+0000 7f7beea00640 1 -- 172.21.15.40:0/3923248889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be80ccd00 msgr2=0x7f7be80ca8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:21.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.927+0000 7f7beea00640 1 --2- 172.21.15.40:0/3923248889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be80ccd00 0x7f7be80ca8e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc000c00 tx=0x7f7bdc0046c0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.927+0000 7f7beea00640 1 -- 172.21.15.40:0/3923248889 shutdown_connections 2024-04-26T23:03:21.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.927+0000 7f7beea00640 1 --2- 172.21.15.40:0/3923248889 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7bc806c940 0x7f7bc806ee00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.927+0000 7f7beea00640 1 --2- 172.21.15.40:0/3923248889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be81f6ab0 0x7f7be80b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.927+0000 7f7beea00640 1 --2- 172.21.15.40:0/3923248889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7be80ce450 0x7f7be815f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.927+0000 7f7beea00640 1 --2- 172.21.15.40:0/3923248889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be80ccd00 0x7f7be80ca8e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:21.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.927+0000 7f7beea00640 1 -- 172.21.15.40:0/3923248889 >> 172.21.15.40:0/3923248889 conn(0x7f7be80b2ee0 msgr2=0x7f7be815b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:21.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.927+0000 7f7beea00640 1 -- 172.21.15.40:0/3923248889 shutdown_connections 2024-04-26T23:03:21.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:21.927+0000 7f7beea00640 1 -- 172.21.15.40:0/3923248889 wait complete. 2024-04-26T23:03:21.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:21.949 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec find d1/d2/d3/) 2024-04-26T23:03:22.014 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/ 2024-04-26T23:03:22.014 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4 2024-04-26T23:03:22.015 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5 2024-04-26T23:03:22.015 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6 2024-04-26T23:03:22.015 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6/d7 2024-04-26T23:03:22.015 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6/d7/d8 2024-04-26T23:03:22.016 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:03:22.016 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:03:22.016 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:03:22.016 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:03:22.016 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/").st_ino) 2024-04-26T23:03:22.016 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:03:22.158 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627778 2024-04-26T23:03:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:03:22.172 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:03:22.173 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:03:22.173 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:03:22.173 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:03:22.173 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4").st_ino) 2024-04-26T23:03:22.173 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:03:22.366 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627779 2024-04-26T23:03:22.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:03:22.381 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:03:22.381 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:03:22.381 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:03:22.381 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:03:22.381 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5").st_ino) 2024-04-26T23:03:22.381 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:03:22.574 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627780 2024-04-26T23:03:22.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:03:22.588 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:03:22.588 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:03:22.588 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:03:22.588 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:03:22.588 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6").st_ino) 2024-04-26T23:03:22.588 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:03:22.769 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627781 2024-04-26T23:03:22.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:03:22.783 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:03:22.783 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:03:22.783 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:03:22.783 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:03:22.783 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6/d7").st_ino) 2024-04-26T23:03:22.784 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:03:22.978 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627782 2024-04-26T23:03:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:03:22.993 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:03:22.993 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:03:22.993 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:03:22.993 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:03:22.993 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6/d7/d8").st_ino) 2024-04-26T23:03:22.993 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:03:23.173 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627783 2024-04-26T23:03:23.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:03:23.187 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 tag path /d1/d2/d3 tag123 2024-04-26T23:03:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.453+0000 7fdf7f000640 1 Processor -- start 2024-04-26T23:03:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.453+0000 7fdf7f000640 1 -- start start 2024-04-26T23:03:23.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.454+0000 7fdf7f000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf7815f420 0x7fdf780d0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.454+0000 7fdf7f000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf780d0d50 con 0x7fdf780d3b60 2024-04-26T23:03:23.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.454+0000 7fdf7f000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf780d0a70 con 0x7fdf780b3d90 2024-04-26T23:03:23.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.454+0000 7fdf7f000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fdf780b1bf0 con 0x7fdf7815f420 2024-04-26T23:03:23.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.455+0000 7fdf7e600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdf780d3b60 0x7fdf780b26d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49416/0 (socket says 172.21.15.40:49416) 2024-04-26T23:03:23.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.455+0000 7fdf7e600640 1 -- 172.21.15.40:0/3094067986 learned_addr learned my addr 172.21.15.40:0/3094067986 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:23.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.455+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/3094067986 >> v2:172.21.15.40:3300/0 conn(0x7fdf7815f420 0x7fdf780d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.455+0000 7fdf7dc00640 1 -- 172.21.15.40:0/3094067986 >> v1:172.21.15.103:6789/0 conn(0x7fdf780b3d90 legacy=0x7fdf780d3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.455+0000 7fdf7dc00640 1 -- 172.21.15.40:0/3094067986 >> v1:172.21.15.40:6789/0 conn(0x7fdf780d3b60 legacy=0x7fdf780b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.455+0000 7fdf7dc00640 1 -- 172.21.15.40:0/3094067986 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf780b1ed0 con 0x7fdf7815f420 2024-04-26T23:03:23.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.456+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/3094067986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7815f420 0x7fdf780d0440 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fdf780d33a0 tx=0x7fdf6c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93eca606db950b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.457+0000 7fdf77e00640 1 -- 172.21.15.40:0/3094067986 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c037070 con 0x7fdf7815f420 2024-04-26T23:03:23.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.457+0000 7fdf77e00640 1 -- 172.21.15.40:0/3094067986 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf6c02ae20 con 0x7fdf7815f420 2024-04-26T23:03:23.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.457+0000 7fdf77e00640 1 -- 172.21.15.40:0/3094067986 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c032660 con 0x7fdf7815f420 2024-04-26T23:03:23.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.457+0000 7fdf7f000640 1 -- 172.21.15.40:0/3094067986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7815f420 msgr2=0x7fdf780d0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.457+0000 7fdf7f000640 1 --2- 172.21.15.40:0/3094067986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7815f420 0x7fdf780d0440 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fdf780d33a0 tx=0x7fdf6c02a110 comp rx=0 tx=0).stop 2024-04-26T23:03:23.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.458+0000 7fdf7f000640 1 -- 172.21.15.40:0/3094067986 shutdown_connections 2024-04-26T23:03:23.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.458+0000 7fdf7f000640 1 --2- 172.21.15.40:0/3094067986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7815f420 0x7fdf780d0440 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.458+0000 7fdf7f000640 1 -- 172.21.15.40:0/3094067986 >> 172.21.15.40:0/3094067986 conn(0x7fdf7809e1d0 msgr2=0x7fdf780d4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:23.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.458+0000 7fdf7f000640 1 -- 172.21.15.40:0/3094067986 shutdown_connections 2024-04-26T23:03:23.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.458+0000 7fdf7f000640 1 -- 172.21.15.40:0/3094067986 wait complete. 2024-04-26T23:03:23.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.459+0000 7fdf7f000640 1 Processor -- start 2024-04-26T23:03:23.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.459+0000 7fdf7f000640 1 -- start start 2024-04-26T23:03:23.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.460+0000 7fdf7f000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf780b3d90 0x7fdf781f9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.460+0000 7fdf7f000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf780d3b60 0x7fdf78159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.460+0000 7fdf7f000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7815f420 0x7fdf780b4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.460+0000 7fdf7f000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf780b2de0 con 0x7fdf7815f420 2024-04-26T23:03:23.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.460+0000 7fdf7f000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf780d2e50 con 0x7fdf780d3b60 2024-04-26T23:03:23.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.460+0000 7fdf7f000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdf780b2f60 con 0x7fdf780b3d90 2024-04-26T23:03:23.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.461+0000 7fdf7e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7815f420 0x7fdf780b4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.461+0000 7fdf7e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7815f420 0x7fdf780b4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39744/0 (socket says 172.21.15.40:39744) 2024-04-26T23:03:23.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.461+0000 7fdf7e600640 1 -- 172.21.15.40:0/2406873323 learned_addr learned my addr 172.21.15.40:0/2406873323 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.461+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf780b3d90 0x7fdf781f9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.461+0000 7fdf7e600640 1 -- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf780b3d90 msgr2=0x7fdf781f9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.461+0000 7fdf7e600640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf780b3d90 0x7fdf781f9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.461+0000 7fdf7e600640 1 -- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf780d3b60 msgr2=0x7fdf78159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.461+0000 7fdf7d200640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf780d3b60 0x7fdf78159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.461+0000 7fdf7e600640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf780d3b60 0x7fdf78159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.461+0000 7fdf7e600640 1 -- 172.21.15.40:0/2406873323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf781f9660 con 0x7fdf7815f420 2024-04-26T23:03:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.461+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf780b3d90 0x7fdf781f9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.461+0000 7fdf7d200640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf780d3b60 0x7fdf78159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.462+0000 7fdf7e600640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7815f420 0x7fdf780b4e80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000caf0 tx=0x7fdf7000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.462+0000 7fdf75600640 1 -- 172.21.15.40:0/2406873323 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70017070 con 0x7fdf7815f420 2024-04-26T23:03:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.462+0000 7fdf75600640 1 -- 172.21.15.40:0/2406873323 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf7000ac40 con 0x7fdf7815f420 2024-04-26T23:03:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.462+0000 7fdf75600640 1 -- 172.21.15.40:0/2406873323 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70012400 con 0x7fdf7815f420 2024-04-26T23:03:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.462+0000 7fdf7f000640 1 -- 172.21.15.40:0/2406873323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf78162310 con 0x7fdf7815f420 2024-04-26T23:03:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.462+0000 7fdf7f000640 1 -- 172.21.15.40:0/2406873323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf780cbfe0 con 0x7fdf7815f420 2024-04-26T23:03:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.464+0000 7fdf75600640 1 -- 172.21.15.40:0/2406873323 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdf7001a070 con 0x7fdf7815f420 2024-04-26T23:03:23.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.465+0000 7fdf75600640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdf5806c920 0x7fdf5806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.465+0000 7fdf75600640 1 -- 172.21.15.40:0/2406873323 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fdf7008b8b0 con 0x7fdf7815f420 2024-04-26T23:03:23.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.466+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdf5806c920 0x7fdf5806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.466+0000 7fdf833f8740 1 Processor -- start 2024-04-26T23:03:23.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.466+0000 7fdf833f8740 1 -- start start 2024-04-26T23:03:23.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.467+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdf5806c920 0x7fdf5806ede0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c034000 tx=0x7fdf6c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.467+0000 7fdf833f8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c018b0 0x556d32bb44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.467+0000 7fdf833f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 0x556d32be3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.467+0000 7fdf833f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be4290 0x556d32bb90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.467+0000 7fdf833f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556d32b1ddb0 con 0x556d32be4290 2024-04-26T23:03:23.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.467+0000 7fdf833f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556d32aba000 con 0x556d32be49d0 2024-04-26T23:03:23.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.467+0000 7fdf833f8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556d32b19d70 con 0x556d32c018b0 2024-04-26T23:03:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.468+0000 7fdf7e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be4290 0x556d32bb90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.468+0000 7fdf7e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be4290 0x556d32bb90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39754/0 (socket says 172.21.15.40:39754) 2024-04-26T23:03:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.468+0000 7fdf7e600640 1 -- 172.21.15.40:0/3488009575 learned_addr learned my addr 172.21.15.40:0/3488009575 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.468+0000 7fdf7d200640 1 --2- 172.21.15.40:0/3488009575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 0x556d32be3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.468+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/3488009575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c018b0 0x556d32bb44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.468+0000 7fdf7e600640 1 -- 172.21.15.40:0/3488009575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c018b0 msgr2=0x556d32bb44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.468+0000 7fdf7e600640 1 --2- 172.21.15.40:0/3488009575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c018b0 0x556d32bb44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.468+0000 7fdf7e600640 1 -- 172.21.15.40:0/3488009575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 msgr2=0x556d32be3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.468+0000 7fdf7e600640 1 --2- 172.21.15.40:0/3488009575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 0x556d32be3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.468+0000 7fdf7e600640 1 -- 172.21.15.40:0/3488009575 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d328fba20 con 0x556d32be4290 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.468+0000 7fdf7d200640 1 --2- 172.21.15.40:0/3488009575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 0x556d32be3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.468+0000 7fdf7e600640 1 --2- 172.21.15.40:0/3488009575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be4290 0x556d32bb90a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fdf70089bb0 tx=0x7fdf70062020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08d57d6bcd0d961 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.469+0000 7fdf6a000640 1 -- 172.21.15.40:0/3488009575 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70055b60 con 0x556d32be4290 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.469+0000 7fdf6a000640 1 -- 172.21.15.40:0/3488009575 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf7006de50 con 0x556d32be4290 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.469+0000 7fdf6a000640 1 -- 172.21.15.40:0/3488009575 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70055d00 con 0x556d32be4290 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.469+0000 7fdf833f8740 1 -- 172.21.15.40:0/3488009575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be4290 msgr2=0x556d32bb90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.469+0000 7fdf833f8740 1 --2- 172.21.15.40:0/3488009575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be4290 0x556d32bb90a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fdf70089bb0 tx=0x7fdf70062020 comp rx=0 tx=0).stop 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.469+0000 7fdf833f8740 1 -- 172.21.15.40:0/3488009575 shutdown_connections 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.469+0000 7fdf833f8740 1 --2- 172.21.15.40:0/3488009575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be4290 0x556d32bb90a0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.469+0000 7fdf833f8740 1 --2- 172.21.15.40:0/3488009575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 0x556d32be3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.469+0000 7fdf833f8740 1 --2- 172.21.15.40:0/3488009575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c018b0 0x556d32bb44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.469+0000 7fdf833f8740 1 -- 172.21.15.40:0/3488009575 >> 172.21.15.40:0/3488009575 conn(0x556d32bb6740 msgr2=0x556d32bfed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.469+0000 7fdf833f8740 1 -- 172.21.15.40:0/3488009575 shutdown_connections 2024-04-26T23:03:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.469+0000 7fdf833f8740 1 -- 172.21.15.40:0/3488009575 wait complete. 2024-04-26T23:03:23.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.470+0000 7fdf833f8740 1 -- start start 2024-04-26T23:03:23.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.470+0000 7fdf833f8740 1 Processor -- start 2024-04-26T23:03:23.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.471+0000 7fdf833f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 0x556d32bfc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.471+0000 7fdf833f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c018b0 0x556d32bb93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.471+0000 7fdf833f8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c0ec20 0x556d32bdf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.471+0000 7fdf833f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556d32bb8660 con 0x556d32c018b0 2024-04-26T23:03:23.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.471+0000 7fdf833f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556d32aa4c50 con 0x556d32be49d0 2024-04-26T23:03:23.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.471+0000 7fdf833f8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556d32bb7a50 con 0x556d32c0ec20 2024-04-26T23:03:23.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.471+0000 7fdf833f8740 10 client.0 _ll_register_callbacks cb 0x556d32b14fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:23.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.472+0000 7fdf7d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c018b0 0x556d32bb93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.472+0000 7fdf7d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c018b0 0x556d32bb93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39766/0 (socket says 172.21.15.40:39766) 2024-04-26T23:03:23.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.472+0000 7fdf7d200640 1 -- 172.21.15.40:0/1010219150 learned_addr learned my addr 172.21.15.40:0/1010219150 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:23.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.472+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/1010219150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 0x556d32bfc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.472+0000 7fdf7e600640 1 --2- 172.21.15.40:0/1010219150 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c0ec20 0x556d32bdf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.472+0000 7fdf7d200640 1 -- 172.21.15.40:0/1010219150 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c0ec20 msgr2=0x556d32bdf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.472+0000 7fdf7d200640 1 --2- 172.21.15.40:0/1010219150 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c0ec20 0x556d32bdf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.472+0000 7fdf7d200640 1 -- 172.21.15.40:0/1010219150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 msgr2=0x556d32bfc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.472+0000 7fdf7d200640 1 --2- 172.21.15.40:0/1010219150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 0x556d32bfc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.472+0000 7fdf7d200640 1 -- 172.21.15.40:0/1010219150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d327ef310 con 0x556d32c018b0 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.472+0000 7fdf7d200640 1 --2- 172.21.15.40:0/1010219150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c018b0 0x556d32bb93d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fdf640055d0 tx=0x7fdf6400e650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770c8b5e54f3672d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.472+0000 7fdf5ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.473+0000 7fdf5ea00640 1 -- 172.21.15.40:0/1010219150 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf64007230 con 0x556d32c018b0 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.473+0000 7fdf5ea00640 1 -- 172.21.15.40:0/1010219150 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fdf640073d0 con 0x556d32c018b0 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.473+0000 7fdf833f8740 1 -- 172.21.15.40:0/1010219150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf7006de50 con 0x556d32c018b0 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.473+0000 7fdf5ea00640 1 -- 172.21.15.40:0/1010219150 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdf64020050 con 0x556d32c018b0 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.473+0000 7fdf833f8740 10 client.5703 fetch_fsmap learned FSMap version 25 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.473+0000 7fdf833f8740 1 -- 172.21.15.40:0/1010219150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x556d3295c1b0 con 0x556d32c018b0 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.473+0000 7fdf5ea00640 1 -- 172.21.15.40:0/1010219150 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf7006de50 con 0x556d32c018b0 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.473+0000 7fdf5ea00640 1 -- 172.21.15.40:0/1010219150 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 3546+0+0 (secure 0 0 0) 0x7fdf64023020 con 0x556d32c018b0 2024-04-26T23:03:23.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.473+0000 7fdf833f8740 10 client.5703 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T23:03:23.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.473+0000 7fdf833f8740 10 client.5703 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-26T23:03:23.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.473+0000 7fdf833f8740 20 client.5703 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:23.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.474+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1010219150 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x556d32be4ec0 0x556d32be66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.474+0000 7fdf833f8740 4 client.5703 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:03:23.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.474+0000 7fdf833f8740 1 -- 172.21.15.40:0/1010219150 --> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d32b2b9c0 con 0x556d32be4ec0 2024-04-26T23:03:23.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.474+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/1010219150 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x556d32be4ec0 0x556d32be66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.474+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/1010219150 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x556d32be4ec0 0x556d32be66b0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8eb16f340b607c2 server_cookie=68280e4b9ae67768 in_seq=0 out_seq=0 2024-04-26T23:03:23.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.474+0000 7fdf5ea00640 10 client.5703 ms_handle_connect on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:23.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.475+0000 7fdf5ea00640 1 -- 172.21.15.40:0/1010219150 <== mds.0 v2:172.21.15.40:6833/299800334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556d32b2b9c0 con 0x556d32be4ec0 2024-04-26T23:03:23.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.475+0000 7fdf5ea00640 10 client.5703 handle_command_reply: tid=0 2024-04-26T23:03:23.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.475+0000 7fdf833f8740 1 client.5703 shutdown 2024-04-26T23:03:23.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.475+0000 7fdf833f8740 1 -- 172.21.15.40:0/1010219150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c018b0 msgr2=0x556d32bb93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.475+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1010219150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c018b0 0x556d32bb93d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fdf640055d0 tx=0x7fdf6400e650 comp rx=0 tx=0).stop 2024-04-26T23:03:23.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.475+0000 7fdf833f8740 1 -- 172.21.15.40:0/1010219150 shutdown_connections 2024-04-26T23:03:23.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.475+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1010219150 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x556d32be4ec0 0x556d32be66b0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.475+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1010219150 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c0ec20 0x556d32bdf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.475+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1010219150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c018b0 0x556d32bb93d0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.475+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1010219150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 0x556d32bfc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.475+0000 7fdf5ea00640 0 client.5703 ms_handle_reset on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:23.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.475+0000 7fdf833f8740 1 -- 172.21.15.40:0/1010219150 >> 172.21.15.40:0/1010219150 conn(0x556d32be4290 msgr2=0x556d32bfed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:23.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.476+0000 7fdf833f8740 1 -- 172.21.15.40:0/1010219150 shutdown_connections 2024-04-26T23:03:23.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.476+0000 7fdf833f8740 1 -- 172.21.15.40:0/1010219150 wait complete. 2024-04-26T23:03:23.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.476+0000 7fdf833f8740 20 client.5703 trim_cache size 0 max 16384 2024-04-26T23:03:23.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.496+0000 7fdf833f8740 1 Processor -- start 2024-04-26T23:03:23.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.496+0000 7fdf833f8740 1 -- start start 2024-04-26T23:03:23.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.497+0000 7fdf833f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be49d0 0x556d32bb5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.497+0000 7fdf833f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32c018b0 0x556d32c02210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.497+0000 7fdf833f8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c0ec20 0x556d32bb3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.497+0000 7fdf833f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf78201dd0 con 0x556d32be49d0 2024-04-26T23:03:23.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.497+0000 7fdf833f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf78200fa0 con 0x556d32c018b0 2024-04-26T23:03:23.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.497+0000 7fdf833f8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdf782016a0 con 0x556d32c0ec20 2024-04-26T23:03:23.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.498+0000 7fdf7dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be49d0 0x556d32bb5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.498+0000 7fdf7dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be49d0 0x556d32bb5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39778/0 (socket says 172.21.15.40:39778) 2024-04-26T23:03:23.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.498+0000 7fdf7dc00640 1 -- 172.21.15.40:0/1695890748 learned_addr learned my addr 172.21.15.40:0/1695890748 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:23.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.498+0000 7fdf7dc00640 1 -- 172.21.15.40:0/1695890748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c0ec20 msgr2=0x556d32bb3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.498+0000 7fdf7e600640 1 --2- 172.21.15.40:0/1695890748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c0ec20 0x556d32bb3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.498+0000 7fdf7d200640 1 --2- 172.21.15.40:0/1695890748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32c018b0 0x556d32c02210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.498+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/1695890748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c0ec20 0x556d32bb3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.498+0000 7fdf7dc00640 1 -- 172.21.15.40:0/1695890748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32c018b0 msgr2=0x556d32c02210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.498+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/1695890748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32c018b0 0x556d32c02210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.498+0000 7fdf7dc00640 1 -- 172.21.15.40:0/1695890748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d32bb75f0 con 0x556d32be49d0 2024-04-26T23:03:23.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.498+0000 7fdf7e600640 1 --2- 172.21.15.40:0/1695890748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c0ec20 0x556d32bb3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:23.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.498+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/1695890748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be49d0 0x556d32bb5210 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c000c00 tx=0x7fdf6c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f54ec7003e242 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.499+0000 7fdf69600640 1 -- 172.21.15.40:0/1695890748 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c054070 con 0x556d32be49d0 2024-04-26T23:03:23.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.499+0000 7fdf69600640 1 -- 172.21.15.40:0/1695890748 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf6c04e3c0 con 0x556d32be49d0 2024-04-26T23:03:23.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.499+0000 7fdf69600640 1 -- 172.21.15.40:0/1695890748 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c037040 con 0x556d32be49d0 2024-04-26T23:03:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.499+0000 7fdf833f8740 1 -- 172.21.15.40:0/1695890748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be49d0 msgr2=0x556d32bb5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.499+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1695890748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be49d0 0x556d32bb5210 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c000c00 tx=0x7fdf6c02c000 comp rx=0 tx=0).stop 2024-04-26T23:03:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.500+0000 7fdf833f8740 1 -- 172.21.15.40:0/1695890748 shutdown_connections 2024-04-26T23:03:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.500+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1695890748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c0ec20 0x556d32bb3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.500+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1695890748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32c018b0 0x556d32c02210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.500+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1695890748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32be49d0 0x556d32bb5210 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.500+0000 7fdf833f8740 1 -- 172.21.15.40:0/1695890748 >> 172.21.15.40:0/1695890748 conn(0x556d32be4290 msgr2=0x556d32be66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.500+0000 7fdf833f8740 1 -- 172.21.15.40:0/1695890748 shutdown_connections 2024-04-26T23:03:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.500+0000 7fdf833f8740 1 -- 172.21.15.40:0/1695890748 wait complete. 2024-04-26T23:03:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.500+0000 7fdf833f8740 1 -- start start 2024-04-26T23:03:23.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.501+0000 7fdf833f8740 1 Processor -- start 2024-04-26T23:03:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.502+0000 7fdf833f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 0x556d32be1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.502+0000 7fdf833f8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c018b0 0x556d32c80ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.502+0000 7fdf833f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c0ec20 0x556d32bb3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.502+0000 7fdf833f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556d32c01520 con 0x556d32c0ec20 2024-04-26T23:03:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.502+0000 7fdf833f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf780cd4f0 con 0x556d32be49d0 2024-04-26T23:03:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.502+0000 7fdf833f8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556d32bfec50 con 0x556d32c018b0 2024-04-26T23:03:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.502+0000 7fdf833f8740 10 client.0 _ll_register_callbacks cb 0x7fdf64007db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.502+0000 7fdf7e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c0ec20 0x556d32bb3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.502+0000 7fdf7e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c0ec20 0x556d32bb3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39780/0 (socket says 172.21.15.40:39780) 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.502+0000 7fdf7e600640 1 -- 172.21.15.40:0/1682123816 learned_addr learned my addr 172.21.15.40:0/1682123816 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.502+0000 7fdf7d200640 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c018b0 0x556d32c80ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.502+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 0x556d32be1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.502+0000 7fdf7e600640 1 -- 172.21.15.40:0/1682123816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c018b0 msgr2=0x556d32c80ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.503+0000 7fdf7e600640 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c018b0 0x556d32c80ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.503+0000 7fdf7e600640 1 -- 172.21.15.40:0/1682123816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 msgr2=0x556d32be1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.503+0000 7fdf7e600640 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 0x556d32be1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.503+0000 7fdf7e600640 1 -- 172.21.15.40:0/1682123816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d327c8490 con 0x556d32c0ec20 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.503+0000 7fdf7d200640 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c018b0 0x556d32c80ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.503+0000 7fdf7e600640 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c0ec20 0x556d32bb3f60 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fdf70061af0 tx=0x7fdf70001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238337f6e991f87a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.503+0000 7fdf5ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.503+0000 7fdf5ea00640 1 -- 172.21.15.40:0/1682123816 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7006de50 con 0x556d32c0ec20 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.503+0000 7fdf5ea00640 1 -- 172.21.15.40:0/1682123816 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fdf70004b60 con 0x556d32c0ec20 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.503+0000 7fdf833f8740 1 -- 172.21.15.40:0/1682123816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf6c04e3c0 con 0x556d32c0ec20 2024-04-26T23:03:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.503+0000 7fdf5ea00640 1 -- 172.21.15.40:0/1682123816 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdf70004d60 con 0x556d32c0ec20 2024-04-26T23:03:23.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.503+0000 7fdf833f8740 10 client.5709 fetch_fsmap learned FSMap version 25 2024-04-26T23:03:23.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.503+0000 7fdf833f8740 1 -- 172.21.15.40:0/1682123816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x556d32be0870 con 0x556d32c0ec20 2024-04-26T23:03:23.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.504+0000 7fdf5ea00640 1 -- 172.21.15.40:0/1682123816 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf6c04e3c0 con 0x556d32c0ec20 2024-04-26T23:03:23.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.504+0000 7fdf5ea00640 1 -- 172.21.15.40:0/1682123816 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 3546+0+0 (secure 0 0 0) 0x7fdf70058940 con 0x556d32c0ec20 2024-04-26T23:03:23.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.504+0000 7fdf833f8740 10 client.5709 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T23:03:23.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.504+0000 7fdf833f8740 10 client.5709 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-26T23:03:23.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.504+0000 7fdf833f8740 20 client.5709 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:23.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.504+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x556d32c6a100 0x556d32bb44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.504+0000 7fdf833f8740 4 client.5709 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "tag path", "path": "/d1/d2/d3", "tag": "tag123"}] 2024-04-26T23:03:23.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.504+0000 7fdf833f8740 1 -- 172.21.15.40:0/1682123816 --> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] -- command(tid 0: {"prefix": "tag path", "path": "/d1/d2/d3", "tag": "tag123"}) -- 0x556d32b25830 con 0x556d32c6a100 2024-04-26T23:03:23.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.504+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x556d32c6a100 0x556d32bb44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.505+0000 7fdf7dc00640 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x556d32c6a100 0x556d32bb44a0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc255843bef9c1f6 server_cookie=3c0608c2703ad097 in_seq=0 out_seq=0 2024-04-26T23:03:23.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.505+0000 7fdf5ea00640 10 client.5709 ms_handle_connect on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:23.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.505+0000 7fdf5ea00640 1 -- 172.21.15.40:0/1682123816 <== mds.0 v2:172.21.15.40:6833/299800334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x556d32b25830 con 0x556d32c6a100 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.505+0000 7fdf5ea00640 10 client.5709 handle_command_reply: tid=0 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.505+0000 7fdf833f8740 1 client.5709 shutdown 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.506+0000 7fdf833f8740 1 -- 172.21.15.40:0/1682123816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c0ec20 msgr2=0x556d32bb3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.506+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c0ec20 0x556d32bb3f60 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fdf70061af0 tx=0x7fdf70001250 comp rx=0 tx=0).stop 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.506+0000 7fdf833f8740 1 -- 172.21.15.40:0/1682123816 shutdown_connections 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.506+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x556d32c6a100 0x556d32bb44a0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.506+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d32c0ec20 0x556d32bb3f60 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.506+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556d32c018b0 0x556d32c80ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.506+0000 7fdf833f8740 1 --2- 172.21.15.40:0/1682123816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d32be49d0 0x556d32be1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.506+0000 7fdf5ea00640 0 client.5709 ms_handle_reset on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.506+0000 7fdf833f8740 1 -- 172.21.15.40:0/1682123816 >> 172.21.15.40:0/1682123816 conn(0x556d32be4290 msgr2=0x556d32c02a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.506+0000 7fdf833f8740 1 -- 172.21.15.40:0/1682123816 shutdown_connections 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.506+0000 7fdf833f8740 1 -- 172.21.15.40:0/1682123816 wait complete. 2024-04-26T23:03:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.507+0000 7fdf833f8740 20 client.5709 trim_cache size 0 max 16384 2024-04-26T23:03:23.509 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:03:23.509 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-04-26T23:03:23.509 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "tag123", 2024-04-26T23:03:23.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-04-26T23:03:23.509 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:03:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.508+0000 7fdf7f000640 1 -- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdf5806c920 msgr2=0x7fdf5806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.508+0000 7fdf7f000640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdf5806c920 0x7fdf5806ede0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c034000 tx=0x7fdf6c02e000 comp rx=0 tx=0).stop 2024-04-26T23:03:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.508+0000 7fdf7f000640 1 -- 172.21.15.40:0/2406873323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7815f420 msgr2=0x7fdf780b4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.508+0000 7fdf7f000640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7815f420 0x7fdf780b4e80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000caf0 tx=0x7fdf7000a060 comp rx=0 tx=0).stop 2024-04-26T23:03:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.508+0000 7fdf7f000640 1 -- 172.21.15.40:0/2406873323 shutdown_connections 2024-04-26T23:03:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.508+0000 7fdf7f000640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdf5806c920 0x7fdf5806ede0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.508+0000 7fdf7f000640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7815f420 0x7fdf780b4e80 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.508+0000 7fdf7f000640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf780d3b60 0x7fdf78159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.508+0000 7fdf7f000640 1 --2- 172.21.15.40:0/2406873323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf780b3d90 0x7fdf781f9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.508+0000 7fdf7f000640 1 -- 172.21.15.40:0/2406873323 >> 172.21.15.40:0/2406873323 conn(0x7fdf7809e1d0 msgr2=0x7fdf780b5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.509+0000 7fdf7f000640 1 -- 172.21.15.40:0/2406873323 shutdown_connections 2024-04-26T23:03:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.509+0000 7fdf7f000640 1 -- 172.21.15.40:0/2406873323 wait complete. 2024-04-26T23:03:23.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:23.530 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-04-26T23:03:23.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.738+0000 7ff103000640 1 Processor -- start 2024-04-26T23:03:23.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.738+0000 7ff103000640 1 -- start start 2024-04-26T23:03:23.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.740+0000 7ff103000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0fc0d1770 0x7ff0fc15ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.740+0000 7ff103000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0fc0d5d30 con 0x7ff0fc0d1b50 2024-04-26T23:03:23.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.740+0000 7ff103000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0fc0b5550 con 0x7ff0fc0b6470 2024-04-26T23:03:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.740+0000 7ff103000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff0fc0d3760 con 0x7ff0fc0d1770 2024-04-26T23:03:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.740+0000 7ff101c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0fc0d1770 0x7ff0fc15ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.741+0000 7ff101c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0fc0d1770 0x7ff0fc15ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39792/0 (socket says 172.21.15.40:39792) 2024-04-26T23:03:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.741+0000 7ff101c00640 1 -- 172.21.15.40:0/1617683010 learned_addr learned my addr 172.21.15.40:0/1617683010 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.741+0000 7ff101c00640 1 -- 172.21.15.40:0/1617683010 >> v1:172.21.15.103:6789/0 conn(0x7ff0fc0b6470 legacy=0x7ff0fc0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.741+0000 7ff101c00640 1 -- 172.21.15.40:0/1617683010 >> v1:172.21.15.40:6789/0 conn(0x7ff0fc0d1b50 legacy=0x7ff0fc0b4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.741+0000 7ff101c00640 1 -- 172.21.15.40:0/1617683010 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc0d09f0 con 0x7ff0fc0d1770 2024-04-26T23:03:23.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.742+0000 7ff101c00640 1 --2- 172.21.15.40:0/1617683010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0fc0d1770 0x7ff0fc15ef70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009870 tx=0x7ff0f002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4968433612896181 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.742+0000 7ff0fbe00640 1 -- 172.21.15.40:0/1617683010 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0037070 con 0x7ff0fc0d1770 2024-04-26T23:03:23.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.743+0000 7ff0fbe00640 1 -- 172.21.15.40:0/1617683010 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f0032440 con 0x7ff0fc0d1770 2024-04-26T23:03:23.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.743+0000 7ff0fbe00640 1 -- 172.21.15.40:0/1617683010 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0032810 con 0x7ff0fc0d1770 2024-04-26T23:03:23.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.743+0000 7ff103000640 1 -- 172.21.15.40:0/1617683010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0fc0d1770 msgr2=0x7ff0fc15ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.743+0000 7ff103000640 1 --2- 172.21.15.40:0/1617683010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0fc0d1770 0x7ff0fc15ef70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009870 tx=0x7ff0f002a210 comp rx=0 tx=0).stop 2024-04-26T23:03:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.743+0000 7ff103000640 1 -- 172.21.15.40:0/1617683010 shutdown_connections 2024-04-26T23:03:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.743+0000 7ff103000640 1 --2- 172.21.15.40:0/1617683010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0fc0d1770 0x7ff0fc15ef70 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.743+0000 7ff103000640 1 -- 172.21.15.40:0/1617683010 >> 172.21.15.40:0/1617683010 conn(0x7ff0fc0d4dd0 msgr2=0x7ff0fc0d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.743+0000 7ff103000640 1 -- 172.21.15.40:0/1617683010 shutdown_connections 2024-04-26T23:03:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.743+0000 7ff103000640 1 -- 172.21.15.40:0/1617683010 wait complete. 2024-04-26T23:03:23.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.745+0000 7ff103000640 1 Processor -- start 2024-04-26T23:03:23.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.745+0000 7ff103000640 1 -- start start 2024-04-26T23:03:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.746+0000 7ff103000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0fc0b6470 0x7ff0fc1e5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.746+0000 7ff103000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0fc0d1770 0x7ff0fc0b9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.746+0000 7ff103000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0fc0d1b50 0x7ff0fc0b9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.746+0000 7ff103000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc0d2670 con 0x7ff0fc0d1b50 2024-04-26T23:03:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.746+0000 7ff103000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc0b5b00 con 0x7ff0fc0d1770 2024-04-26T23:03:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.746+0000 7ff103000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff0fc0d3bf0 con 0x7ff0fc0b6470 2024-04-26T23:03:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.746+0000 7ff102600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0fc0d1b50 0x7ff0fc0b9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.747+0000 7ff102600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0fc0d1b50 0x7ff0fc0b9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39804/0 (socket says 172.21.15.40:39804) 2024-04-26T23:03:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.747+0000 7ff102600640 1 -- 172.21.15.40:0/3010786351 learned_addr learned my addr 172.21.15.40:0/3010786351 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.747+0000 7ff101200640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0fc0d1770 0x7ff0fc0b9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.747+0000 7ff102600640 1 -- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0fc0b6470 msgr2=0x7ff0fc1e5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.747+0000 7ff102600640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0fc0b6470 0x7ff0fc1e5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.747+0000 7ff102600640 1 -- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0fc0d1770 msgr2=0x7ff0fc0b9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.747+0000 7ff102600640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0fc0d1770 0x7ff0fc0b9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.747+0000 7ff102600640 1 -- 172.21.15.40:0/3010786351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc15d980 con 0x7ff0fc0d1b50 2024-04-26T23:03:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.747+0000 7ff101200640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0fc0d1770 0x7ff0fc0b9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.747+0000 7ff102600640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0fc0d1b50 0x7ff0fc0b9c90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400c7d0 tx=0x7ff0f400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.747+0000 7ff0f9600640 1 -- 172.21.15.40:0/3010786351 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4003b70 con 0x7ff0fc0d1b50 2024-04-26T23:03:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.748+0000 7ff0f9600640 1 -- 172.21.15.40:0/3010786351 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f4003d30 con 0x7ff0fc0d1b50 2024-04-26T23:03:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.748+0000 7ff0f9600640 1 -- 172.21.15.40:0/3010786351 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f400aa90 con 0x7ff0fc0d1b50 2024-04-26T23:03:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.748+0000 7ff103000640 1 -- 172.21.15.40:0/3010786351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0fc15dc60 con 0x7ff0fc0d1b50 2024-04-26T23:03:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.748+0000 7ff103000640 1 -- 172.21.15.40:0/3010786351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0fc1e3f80 con 0x7ff0fc0d1b50 2024-04-26T23:03:23.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.750+0000 7ff0f9600640 1 -- 172.21.15.40:0/3010786351 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0f400ac30 con 0x7ff0fc0d1b50 2024-04-26T23:03:23.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.751+0000 7ff0f9600640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff0dc06c8f0 0x7ff0dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.751+0000 7ff0f9600640 1 -- 172.21.15.40:0/3010786351 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7ff0f408bbe0 con 0x7ff0fc0d1b50 2024-04-26T23:03:23.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.752+0000 7ff101c00640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff0dc06c8f0 0x7ff0dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.752+0000 7ff107439740 1 Processor -- start 2024-04-26T23:03:23.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.752+0000 7ff107439740 1 -- start start 2024-04-26T23:03:23.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.753+0000 7ff101c00640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff0dc06c8f0 0x7ff0dc06edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0034000 tx=0x7ff0f002c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.753+0000 7ff107439740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccad8710 0x55a6ccad8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.753+0000 7ff107439740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaf5fb0 0x55a6ccaf6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.753+0000 7ff107439740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccaae3f0 0x55a6ccaf4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.753+0000 7ff107439740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a6cc9be030 con 0x55a6ccaf5fb0 2024-04-26T23:03:23.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.753+0000 7ff107439740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a6cc9d97f0 con 0x55a6ccad8710 2024-04-26T23:03:23.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.753+0000 7ff107439740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a6cc9947e0 con 0x55a6ccaae3f0 2024-04-26T23:03:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.753+0000 7ff101200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaf5fb0 0x55a6ccaf6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.753+0000 7ff101200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaf5fb0 0x55a6ccaf6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39810/0 (socket says 172.21.15.40:39810) 2024-04-26T23:03:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.753+0000 7ff101200640 1 -- 172.21.15.40:0/3347519839 learned_addr learned my addr 172.21.15.40:0/3347519839 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.754+0000 7ff101c00640 1 --2- 172.21.15.40:0/3347519839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccad8710 0x55a6ccad8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.754+0000 7ff102600640 1 --2- 172.21.15.40:0/3347519839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccaae3f0 0x55a6ccaf4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.754+0000 7ff101200640 1 -- 172.21.15.40:0/3347519839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccaae3f0 msgr2=0x55a6ccaf4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.754+0000 7ff101200640 1 --2- 172.21.15.40:0/3347519839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccaae3f0 0x55a6ccaf4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.754+0000 7ff101200640 1 -- 172.21.15.40:0/3347519839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccad8710 msgr2=0x55a6ccad8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.754+0000 7ff101200640 1 --2- 172.21.15.40:0/3347519839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccad8710 0x55a6ccad8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.754+0000 7ff101200640 1 -- 172.21.15.40:0/3347519839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6cc9a0050 con 0x55a6ccaf5fb0 2024-04-26T23:03:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.754+0000 7ff101200640 1 --2- 172.21.15.40:0/3347519839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaf5fb0 0x55a6ccaf6500 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800a850 tx=0x7ff0e800ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d4d8214222bc0e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.754+0000 7ff0ee000640 1 -- 172.21.15.40:0/3347519839 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8015070 con 0x55a6ccaf5fb0 2024-04-26T23:03:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.755+0000 7ff0ee000640 1 -- 172.21.15.40:0/3347519839 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e8004240 con 0x55a6ccaf5fb0 2024-04-26T23:03:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.755+0000 7ff0ee000640 1 -- 172.21.15.40:0/3347519839 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e800c940 con 0x55a6ccaf5fb0 2024-04-26T23:03:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.755+0000 7ff107439740 1 -- 172.21.15.40:0/3347519839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaf5fb0 msgr2=0x55a6ccaf6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.755+0000 7ff107439740 1 --2- 172.21.15.40:0/3347519839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaf5fb0 0x55a6ccaf6500 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800a850 tx=0x7ff0e800ad10 comp rx=0 tx=0).stop 2024-04-26T23:03:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.755+0000 7ff107439740 1 -- 172.21.15.40:0/3347519839 shutdown_connections 2024-04-26T23:03:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.755+0000 7ff107439740 1 --2- 172.21.15.40:0/3347519839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccaae3f0 0x55a6ccaf4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.755+0000 7ff107439740 1 --2- 172.21.15.40:0/3347519839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaf5fb0 0x55a6ccaf6500 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.755+0000 7ff107439740 1 --2- 172.21.15.40:0/3347519839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccad8710 0x55a6ccad8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.755+0000 7ff107439740 1 -- 172.21.15.40:0/3347519839 >> 172.21.15.40:0/3347519839 conn(0x55a6ccaaad20 msgr2=0x55a6ccaf3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.755+0000 7ff107439740 1 -- 172.21.15.40:0/3347519839 shutdown_connections 2024-04-26T23:03:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.755+0000 7ff107439740 1 -- 172.21.15.40:0/3347519839 wait complete. 2024-04-26T23:03:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.756+0000 7ff107439740 1 -- start start 2024-04-26T23:03:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.756+0000 7ff107439740 1 Processor -- start 2024-04-26T23:03:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.757+0000 7ff107439740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccad8710 0x55a6ccaeb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.757+0000 7ff107439740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccaf5fb0 0x55a6ccaa7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff107439740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccaa7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff107439740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a6cca16200 con 0x55a6ccaebd70 2024-04-26T23:03:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff107439740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a6cca10ee0 con 0x55a6ccaf5fb0 2024-04-26T23:03:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff107439740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a6cca1de70 con 0x55a6ccad8710 2024-04-26T23:03:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff107439740 10 client.0 _ll_register_callbacks cb 0x55a6cca07270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff102600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccaa7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff102600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccaa7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39818/0 (socket says 172.21.15.40:39818) 2024-04-26T23:03:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff102600640 1 -- 172.21.15.40:0/2652238494 learned_addr learned my addr 172.21.15.40:0/2652238494 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff101c00640 1 --2- 172.21.15.40:0/2652238494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccad8710 0x55a6ccaeb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff101200640 1 --2- 172.21.15.40:0/2652238494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccaf5fb0 0x55a6ccaa7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff102600640 1 -- 172.21.15.40:0/2652238494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccad8710 msgr2=0x55a6ccaeb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff102600640 1 --2- 172.21.15.40:0/2652238494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccad8710 0x55a6ccaeb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff102600640 1 -- 172.21.15.40:0/2652238494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccaf5fb0 msgr2=0x55a6ccaa7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff102600640 1 --2- 172.21.15.40:0/2652238494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccaf5fb0 0x55a6ccaa7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.758+0000 7ff102600640 1 -- 172.21.15.40:0/2652238494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6cc7ed7d0 con 0x55a6ccaebd70 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.759+0000 7ff102600640 1 --2- 172.21.15.40:0/2652238494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccaa7ea0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400a430 tx=0x7ff0f40590e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbbb7fef36afa66 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.759+0000 7ff0e2a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.759+0000 7ff0e2a00640 1 -- 172.21.15.40:0/2652238494 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f405db30 con 0x55a6ccaebd70 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.759+0000 7ff0e2a00640 1 -- 172.21.15.40:0/2652238494 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7ff0f4055df0 con 0x55a6ccaebd70 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.759+0000 7ff107439740 1 -- 172.21.15.40:0/2652238494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0e8004240 con 0x55a6ccaebd70 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.759+0000 7ff0e2a00640 1 -- 172.21.15.40:0/2652238494 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff0f4062360 con 0x55a6ccaebd70 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.759+0000 7ff107439740 10 client.5724 fetch_fsmap learned FSMap version 25 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.759+0000 7ff107439740 1 -- 172.21.15.40:0/2652238494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55a6cc7233f0 con 0x55a6ccaebd70 2024-04-26T23:03:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.759+0000 7ff0e2a00640 1 -- 172.21.15.40:0/2652238494 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e8004240 con 0x55a6ccaebd70 2024-04-26T23:03:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.760+0000 7ff0e2a00640 1 -- 172.21.15.40:0/2652238494 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 3546+0+0 (secure 0 0 0) 0x7ff0f4058bd0 con 0x55a6ccaebd70 2024-04-26T23:03:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.760+0000 7ff107439740 10 client.5724 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T23:03:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.760+0000 7ff107439740 10 client.5724 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-26T23:03:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.760+0000 7ff107439740 20 client.5724 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.760+0000 7ff107439740 1 --2- 172.21.15.40:0/2652238494 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55a6ccaf30e0 0x55a6ccad9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.760+0000 7ff107439740 4 client.5724 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:03:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.760+0000 7ff107439740 1 -- 172.21.15.40:0/2652238494 --> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6cca1f7e0 con 0x55a6ccaf30e0 2024-04-26T23:03:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.760+0000 7ff101c00640 1 --2- 172.21.15.40:0/2652238494 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55a6ccaf30e0 0x55a6ccad9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.761+0000 7ff101c00640 1 --2- 172.21.15.40:0/2652238494 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55a6ccaf30e0 0x55a6ccad9b90 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33d3114bfcdd9c6d server_cookie=f01beff1b3dfb8bc in_seq=0 out_seq=0 2024-04-26T23:03:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.761+0000 7ff0e2a00640 10 client.5724 ms_handle_connect on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.761+0000 7ff0e2a00640 1 -- 172.21.15.40:0/2652238494 <== mds.0 v2:172.21.15.40:6833/299800334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a6cca1f7e0 con 0x55a6ccaf30e0 2024-04-26T23:03:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.761+0000 7ff0e2a00640 10 client.5724 handle_command_reply: tid=0 2024-04-26T23:03:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.761+0000 7ff107439740 1 client.5724 shutdown 2024-04-26T23:03:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.762+0000 7ff107439740 1 -- 172.21.15.40:0/2652238494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 msgr2=0x55a6ccaa7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.762+0000 7ff107439740 1 --2- 172.21.15.40:0/2652238494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccaa7ea0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400a430 tx=0x7ff0f40590e0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.762+0000 7ff107439740 1 -- 172.21.15.40:0/2652238494 shutdown_connections 2024-04-26T23:03:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.762+0000 7ff107439740 1 --2- 172.21.15.40:0/2652238494 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55a6ccaf30e0 0x55a6ccad9b90 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.762+0000 7ff107439740 1 --2- 172.21.15.40:0/2652238494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccaa7ea0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.762+0000 7ff107439740 1 --2- 172.21.15.40:0/2652238494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccaf5fb0 0x55a6ccaa7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.762+0000 7ff107439740 1 --2- 172.21.15.40:0/2652238494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccad8710 0x55a6ccaeb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.762+0000 7ff0e2a00640 0 client.5724 ms_handle_reset on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.762+0000 7ff107439740 1 -- 172.21.15.40:0/2652238494 >> 172.21.15.40:0/2652238494 conn(0x55a6ccaae3f0 msgr2=0x55a6ccad7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.762+0000 7ff107439740 1 -- 172.21.15.40:0/2652238494 shutdown_connections 2024-04-26T23:03:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.762+0000 7ff107439740 1 -- 172.21.15.40:0/2652238494 wait complete. 2024-04-26T23:03:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.762+0000 7ff107439740 20 client.5724 trim_cache size 0 max 16384 2024-04-26T23:03:23.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.781+0000 7ff107439740 1 Processor -- start 2024-04-26T23:03:23.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.782+0000 7ff107439740 1 -- start start 2024-04-26T23:03:23.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.782+0000 7ff107439740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccad8710 0x55a6ccaeb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.782+0000 7ff107439740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccb0d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.782+0000 7ff107439740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccaf5fb0 0x55a6ccaa7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.782+0000 7ff107439740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a6ccaa9060 con 0x55a6ccaebd70 2024-04-26T23:03:23.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.782+0000 7ff107439740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc201e70 con 0x55a6ccaf5fb0 2024-04-26T23:03:23.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.782+0000 7ff107439740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff0fc201040 con 0x55a6ccad8710 2024-04-26T23:03:23.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.783+0000 7ff101200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccb0d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.783+0000 7ff101200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccb0d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39832/0 (socket says 172.21.15.40:39832) 2024-04-26T23:03:23.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.783+0000 7ff101200640 1 -- 172.21.15.40:0/1090282713 learned_addr learned my addr 172.21.15.40:0/1090282713 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:23.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.783+0000 7ff101c00640 1 --2- 172.21.15.40:0/1090282713 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccad8710 0x55a6ccaeb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.783+0000 7ff101200640 1 -- 172.21.15.40:0/1090282713 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccad8710 msgr2=0x55a6ccaeb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.783+0000 7ff101200640 1 --2- 172.21.15.40:0/1090282713 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccad8710 0x55a6ccaeb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.783+0000 7ff101200640 1 -- 172.21.15.40:0/1090282713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccaf5fb0 msgr2=0x55a6ccaa7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.783+0000 7ff101200640 1 --2- 172.21.15.40:0/1090282713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccaf5fb0 0x55a6ccaa7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.783+0000 7ff101200640 1 -- 172.21.15.40:0/1090282713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6cc6e30f0 con 0x55a6ccaebd70 2024-04-26T23:03:23.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.783+0000 7ff101c00640 1 --2- 172.21.15.40:0/1090282713 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccad8710 0x55a6ccaeb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:23.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.784+0000 7ff101200640 1 --2- 172.21.15.40:0/1090282713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccb0d370 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8008010 tx=0x7ff0e8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136e97ebb5b603ce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.784+0000 7ff0ed600640 1 -- 172.21.15.40:0/1090282713 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8029070 con 0x55a6ccaebd70 2024-04-26T23:03:23.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.784+0000 7ff0ed600640 1 -- 172.21.15.40:0/1090282713 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e800cb40 con 0x55a6ccaebd70 2024-04-26T23:03:23.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.784+0000 7ff0ed600640 1 -- 172.21.15.40:0/1090282713 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8015040 con 0x55a6ccaebd70 2024-04-26T23:03:23.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.785+0000 7ff107439740 1 -- 172.21.15.40:0/1090282713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 msgr2=0x55a6ccb0d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.785+0000 7ff107439740 1 --2- 172.21.15.40:0/1090282713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccb0d370 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8008010 tx=0x7ff0e8012000 comp rx=0 tx=0).stop 2024-04-26T23:03:23.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.785+0000 7ff107439740 1 -- 172.21.15.40:0/1090282713 shutdown_connections 2024-04-26T23:03:23.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.785+0000 7ff107439740 1 --2- 172.21.15.40:0/1090282713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccaf5fb0 0x55a6ccaa7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.785+0000 7ff107439740 1 --2- 172.21.15.40:0/1090282713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccb0d370 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.785+0000 7ff107439740 1 --2- 172.21.15.40:0/1090282713 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccad8710 0x55a6ccaeb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.785+0000 7ff107439740 1 -- 172.21.15.40:0/1090282713 >> 172.21.15.40:0/1090282713 conn(0x55a6ccaae3f0 msgr2=0x55a6ccaf43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:23.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.785+0000 7ff107439740 1 -- 172.21.15.40:0/1090282713 shutdown_connections 2024-04-26T23:03:23.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.785+0000 7ff107439740 1 -- 172.21.15.40:0/1090282713 wait complete. 2024-04-26T23:03:23.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.786+0000 7ff107439740 1 -- start start 2024-04-26T23:03:23.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.786+0000 7ff107439740 1 Processor -- start 2024-04-26T23:03:23.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.787+0000 7ff107439740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccad8710 0x55a6ccb6d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.787+0000 7ff107439740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccaea770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.787+0000 7ff107439740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccaf5fb0 0x55a6ccaeacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.787+0000 7ff107439740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc1e51c0 con 0x55a6ccaebd70 2024-04-26T23:03:23.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.787+0000 7ff107439740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc201740 con 0x55a6ccad8710 2024-04-26T23:03:23.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.787+0000 7ff107439740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a6cca1efd0 con 0x55a6ccaf5fb0 2024-04-26T23:03:23.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.787+0000 7ff107439740 10 client.0 _ll_register_callbacks cb 0x7ff0f4001830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:23.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.787+0000 7ff101200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccaea770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.787+0000 7ff101200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccaea770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39846/0 (socket says 172.21.15.40:39846) 2024-04-26T23:03:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.787+0000 7ff101200640 1 -- 172.21.15.40:0/132210771 learned_addr learned my addr 172.21.15.40:0/132210771 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.788+0000 7ff101c00640 1 --2- 172.21.15.40:0/132210771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccad8710 0x55a6ccb6d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.788+0000 7ff101200640 1 -- 172.21.15.40:0/132210771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccaf5fb0 msgr2=0x55a6ccaeacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.788+0000 7ff101200640 1 --2- 172.21.15.40:0/132210771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccaf5fb0 0x55a6ccaeacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.788+0000 7ff101200640 1 -- 172.21.15.40:0/132210771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccad8710 msgr2=0x55a6ccb6d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.788+0000 7ff101200640 1 --2- 172.21.15.40:0/132210771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccad8710 0x55a6ccb6d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.788+0000 7ff101200640 1 -- 172.21.15.40:0/132210771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6cc9d9a30 con 0x55a6ccaebd70 2024-04-26T23:03:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.788+0000 7ff101c00640 1 --2- 172.21.15.40:0/132210771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccad8710 0x55a6ccb6d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.788+0000 7ff101200640 1 --2- 172.21.15.40:0/132210771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccaea770 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8006fd0 tx=0x7ff0e8004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d925160cbf285cd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.788+0000 7ff0e2a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:23.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.788+0000 7ff0e2a00640 1 -- 172.21.15.40:0/132210771 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8029040 con 0x55a6ccaebd70 2024-04-26T23:03:23.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.788+0000 7ff0e2a00640 1 -- 172.21.15.40:0/132210771 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7ff0e80042f0 con 0x55a6ccaebd70 2024-04-26T23:03:23.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.788+0000 7ff107439740 1 -- 172.21.15.40:0/132210771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0e800cb40 con 0x55a6ccaebd70 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.789+0000 7ff0e2a00640 1 -- 172.21.15.40:0/132210771 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff0e8015040 con 0x55a6ccaebd70 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.789+0000 7ff107439740 10 client.5730 fetch_fsmap learned FSMap version 25 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.789+0000 7ff107439740 1 -- 172.21.15.40:0/132210771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55a6ccaaceb0 con 0x55a6ccaebd70 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.789+0000 7ff0e2a00640 1 -- 172.21.15.40:0/132210771 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e800cb40 con 0x55a6ccaebd70 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.789+0000 7ff0e2a00640 1 -- 172.21.15.40:0/132210771 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 25) ==== 3546+0+0 (secure 0 0 0) 0x7ff0e8023070 con 0x55a6ccaebd70 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.789+0000 7ff107439740 10 client.5730 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.789+0000 7ff107439740 10 client.5730 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.789+0000 7ff107439740 20 client.5730 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.789+0000 7ff107439740 1 --2- 172.21.15.40:0/132210771 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55a6ccb03620 0x55a6ccaf6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.789+0000 7ff107439740 4 client.5730 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.789+0000 7ff107439740 1 -- 172.21.15.40:0/132210771 --> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a6cca19650 con 0x55a6ccb03620 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.790+0000 7ff101c00640 1 --2- 172.21.15.40:0/132210771 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55a6ccb03620 0x55a6ccaf6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.790+0000 7ff101c00640 1 --2- 172.21.15.40:0/132210771 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55a6ccb03620 0x55a6ccaf6d00 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2bd10f64a5883d2 server_cookie=2be6dc85d3950111 in_seq=0 out_seq=0 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.790+0000 7ff0e2a00640 10 client.5730 ms_handle_connect on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:23.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.790+0000 7ff0e2a00640 1 -- 172.21.15.40:0/132210771 <== mds.0 v2:172.21.15.40:6833/299800334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+207 (crc 0 0 0) 0x55a6cca19650 con 0x55a6ccb03620 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.790+0000 7ff0e2a00640 10 client.5730 handle_command_reply: tid=0 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.790+0000 7ff107439740 1 client.5730 shutdown 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.790+0000 7ff107439740 1 -- 172.21.15.40:0/132210771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 msgr2=0x55a6ccaea770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.790+0000 7ff107439740 1 --2- 172.21.15.40:0/132210771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccaea770 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8006fd0 tx=0x7ff0e8004680 comp rx=0 tx=0).stop 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.791+0000 7ff107439740 1 -- 172.21.15.40:0/132210771 shutdown_connections 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.791+0000 7ff107439740 1 --2- 172.21.15.40:0/132210771 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55a6ccb03620 0x55a6ccaf6d00 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.791+0000 7ff107439740 1 --2- 172.21.15.40:0/132210771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a6ccaf5fb0 0x55a6ccaeacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.791+0000 7ff107439740 1 --2- 172.21.15.40:0/132210771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6ccaebd70 0x55a6ccaea770 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.791+0000 7ff107439740 1 --2- 172.21.15.40:0/132210771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6ccad8710 0x55a6ccb6d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.791+0000 7ff107439740 1 -- 172.21.15.40:0/132210771 >> 172.21.15.40:0/132210771 conn(0x55a6ccaae3f0 msgr2=0x55a6ccad4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.791+0000 7ff0e2a00640 0 client.5730 ms_handle_reset on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.791+0000 7ff107439740 1 -- 172.21.15.40:0/132210771 shutdown_connections 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.791+0000 7ff107439740 1 -- 172.21.15.40:0/132210771 wait complete. 2024-04-26T23:03:23.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.791+0000 7ff107439740 20 client.5730 trim_cache size 0 max 16384 2024-04-26T23:03:23.794 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:03:23.794 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-26T23:03:23.794 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:03:23.794 INFO:teuthology.orchestra.run.smithi040.stdout: "tag123": { 2024-04-26T23:03:23.794 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:03:23.794 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "tag123", 2024-04-26T23:03:23.794 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:03:23.794 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:03:23.794 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:03:23.794 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:03:23.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.792+0000 7ff103000640 1 -- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff0dc06c8f0 msgr2=0x7ff0dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.792+0000 7ff103000640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff0dc06c8f0 0x7ff0dc06edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0034000 tx=0x7ff0f002c000 comp rx=0 tx=0).stop 2024-04-26T23:03:23.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.792+0000 7ff103000640 1 -- 172.21.15.40:0/3010786351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0fc0d1b50 msgr2=0x7ff0fc0b9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:23.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.792+0000 7ff103000640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0fc0d1b50 0x7ff0fc0b9c90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400c7d0 tx=0x7ff0f400cc90 comp rx=0 tx=0).stop 2024-04-26T23:03:23.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.792+0000 7ff103000640 1 -- 172.21.15.40:0/3010786351 shutdown_connections 2024-04-26T23:03:23.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.792+0000 7ff103000640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff0dc06c8f0 0x7ff0dc06edb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.792+0000 7ff103000640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0fc0d1b50 0x7ff0fc0b9c90 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.792+0000 7ff103000640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0fc0d1770 0x7ff0fc0b9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.792+0000 7ff103000640 1 --2- 172.21.15.40:0/3010786351 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff0fc0b6470 0x7ff0fc1e5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:23.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.792+0000 7ff103000640 1 -- 172.21.15.40:0/3010786351 >> 172.21.15.40:0/3010786351 conn(0x7ff0fc0d4dd0 msgr2=0x7ff0fc0b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:23.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.792+0000 7ff103000640 1 -- 172.21.15.40:0/3010786351 shutdown_connections 2024-04-26T23:03:23.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:23.792+0000 7ff103000640 1 -- 172.21.15.40:0/3010786351 wait complete. 2024-04-26T23:03:23.809 INFO:tasks.cephfs.filesystem:scrub status for tag:tag123 - {'path': '/d1/d2/d3', 'tag': 'tag123', 'options': 'recursive,force'} 2024-04-26T23:03:47.549 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:03:47.558 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:03:53.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:53.810 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-04-26T23:03:54.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.035+0000 7ff896000640 1 Processor -- start 2024-04-26T23:03:54.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.035+0000 7ff896000640 1 -- start start 2024-04-26T23:03:54.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.037+0000 7ff896000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff890073600 0x7ff890072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.037+0000 7ff896000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff8900739e0 0x7ff89015f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.037+0000 7ff896000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8901604e0 con 0x7ff89015fcb0 2024-04-26T23:03:54.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.037+0000 7ff896000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff890073220 con 0x7ff890073600 2024-04-26T23:03:54.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.037+0000 7ff896000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff890072850 con 0x7ff8900739e0 2024-04-26T23:03:54.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.037+0000 7ff894c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff890073600 0x7ff890072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.037+0000 7ff895600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff89015fcb0 0x7ff8900ae8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38354/0 (socket says 172.21.15.40:38354) 2024-04-26T23:03:54.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.037+0000 7ff895600640 1 -- 172.21.15.40:0/3423541692 learned_addr learned my addr 172.21.15.40:0/3423541692 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:54.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.038+0000 7ff88fe00640 1 --2- 172.21.15.40:0/3423541692 >> v2:172.21.15.103:3300/0 conn(0x7ff8900739e0 0x7ff89015f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.038+0000 7ff894c00640 1 -- 172.21.15.40:0/3423541692 >> v2:172.21.15.103:3300/0 conn(0x7ff8900739e0 msgr2=0x7ff89015f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.038+0000 7ff894c00640 1 --2- 172.21.15.40:0/3423541692 >> v2:172.21.15.103:3300/0 conn(0x7ff8900739e0 0x7ff89015f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.038+0000 7ff894c00640 1 -- 172.21.15.40:0/3423541692 >> v1:172.21.15.40:6789/0 conn(0x7ff89015fcb0 legacy=0x7ff8900ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.038+0000 7ff894c00640 1 -- 172.21.15.40:0/3423541692 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff890072200 con 0x7ff890073600 2024-04-26T23:03:54.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.038+0000 7ff88f400640 1 -- 172.21.15.40:0/3423541692 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355404291 0 0) 0x7ff8901604e0 con 0x7ff89015fcb0 2024-04-26T23:03:54.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.038+0000 7ff88fe00640 1 --2- 172.21.15.40:0/3423541692 >> v2:172.21.15.103:3300/0 conn(0x7ff8900739e0 0x7ff89015f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:54.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.039+0000 7ff894c00640 1 --2- 172.21.15.40:0/3423541692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff890073600 0x7ff890072ce0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff8900733c0 tx=0x7ff88402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f35f9ef04b67c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.039+0000 7ff88f400640 1 -- 172.21.15.40:0/3423541692 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff884037070 con 0x7ff890073600 2024-04-26T23:03:54.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.040+0000 7ff88f400640 1 -- 172.21.15.40:0/3423541692 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff884032440 con 0x7ff890073600 2024-04-26T23:03:54.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.040+0000 7ff88f400640 1 -- 172.21.15.40:0/3423541692 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff884032810 con 0x7ff890073600 2024-04-26T23:03:54.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.040+0000 7ff896000640 1 -- 172.21.15.40:0/3423541692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff890073600 msgr2=0x7ff890072ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.040+0000 7ff896000640 1 --2- 172.21.15.40:0/3423541692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff890073600 0x7ff890072ce0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff8900733c0 tx=0x7ff88402a1a0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.040+0000 7ff896000640 1 -- 172.21.15.40:0/3423541692 shutdown_connections 2024-04-26T23:03:54.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.040+0000 7ff896000640 1 --2- 172.21.15.40:0/3423541692 >> v2:172.21.15.103:3300/0 conn(0x7ff8900739e0 0x7ff89015f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.040+0000 7ff896000640 1 --2- 172.21.15.40:0/3423541692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff890073600 0x7ff890072ce0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.040+0000 7ff896000640 1 -- 172.21.15.40:0/3423541692 >> 172.21.15.40:0/3423541692 conn(0x7ff8900b2720 msgr2=0x7ff8900b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.040+0000 7ff896000640 1 -- 172.21.15.40:0/3423541692 shutdown_connections 2024-04-26T23:03:54.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.040+0000 7ff896000640 1 -- 172.21.15.40:0/3423541692 wait complete. 2024-04-26T23:03:54.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.042+0000 7ff896000640 1 Processor -- start 2024-04-26T23:03:54.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.042+0000 7ff896000640 1 -- start start 2024-04-26T23:03:54.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.043+0000 7ff896000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff890073600 0x7ff890147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.043+0000 7ff896000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8900739e0 0x7ff89015a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.043+0000 7ff896000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff89015fcb0 0x7ff89015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.043+0000 7ff896000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff890071e20 con 0x7ff890073600 2024-04-26T23:03:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.043+0000 7ff896000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff890073e90 con 0x7ff89015fcb0 2024-04-26T23:03:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.043+0000 7ff896000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff890071fa0 con 0x7ff8900739e0 2024-04-26T23:03:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.044+0000 7ff894c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff890073600 0x7ff890147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.044+0000 7ff894c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff890073600 0x7ff890147470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49160/0 (socket says 172.21.15.40:49160) 2024-04-26T23:03:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.044+0000 7ff894c00640 1 -- 172.21.15.40:0/2500799144 learned_addr learned my addr 172.21.15.40:0/2500799144 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.044+0000 7ff88fe00640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8900739e0 0x7ff89015a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.044+0000 7ff895600640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff89015fcb0 0x7ff89015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.044+0000 7ff894c00640 1 -- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8900739e0 msgr2=0x7ff89015a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.044+0000 7ff894c00640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8900739e0 0x7ff89015a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.044+0000 7ff894c00640 1 -- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff89015fcb0 msgr2=0x7ff89015a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.044+0000 7ff894c00640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff89015fcb0 0x7ff89015a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.044+0000 7ff894c00640 1 -- 172.21.15.40:0/2500799144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff890157900 con 0x7ff890073600 2024-04-26T23:03:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.044+0000 7ff88fe00640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8900739e0 0x7ff89015a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:54.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.044+0000 7ff895600640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff89015fcb0 0x7ff89015a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:54.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.045+0000 7ff894c00640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff890073600 0x7ff890147470 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff884000c00 tx=0x7ff8840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.045+0000 7ff88cc00640 1 -- 172.21.15.40:0/2500799144 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff884037050 con 0x7ff890073600 2024-04-26T23:03:54.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.045+0000 7ff88cc00640 1 -- 172.21.15.40:0/2500799144 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff884038040 con 0x7ff890073600 2024-04-26T23:03:54.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.045+0000 7ff88cc00640 1 -- 172.21.15.40:0/2500799144 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff884004450 con 0x7ff890073600 2024-04-26T23:03:54.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.045+0000 7ff896000640 1 -- 172.21.15.40:0/2500799144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff890156d80 con 0x7ff890073600 2024-04-26T23:03:54.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.045+0000 7ff896000640 1 -- 172.21.15.40:0/2500799144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff890157340 con 0x7ff890073600 2024-04-26T23:03:54.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.048+0000 7ff88cc00640 1 -- 172.21.15.40:0/2500799144 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff884030980 con 0x7ff890073600 2024-04-26T23:03:54.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.049+0000 7ff88cc00640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff87006c920 0x7ff87006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.049+0000 7ff88cc00640 1 -- 172.21.15.40:0/2500799144 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7ff8840acb00 con 0x7ff890073600 2024-04-26T23:03:54.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.050+0000 7ff89a477740 1 Processor -- start 2024-04-26T23:03:54.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.050+0000 7ff88fe00640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff87006c920 0x7ff87006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.050+0000 7ff89a477740 1 -- start start 2024-04-26T23:03:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.051+0000 7ff89a477740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558ac710 0x55d1558acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.051+0000 7ff89a477740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558c9fb0 0x55d1558ca500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.051+0000 7ff89a477740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558823f0 0x55d1558c8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.051+0000 7ff89a477740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d155792030 con 0x55d1558823f0 2024-04-26T23:03:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.051+0000 7ff89a477740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d1557ad7f0 con 0x55d1558ac710 2024-04-26T23:03:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.051+0000 7ff89a477740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d1557687e0 con 0x55d1558c9fb0 2024-04-26T23:03:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.051+0000 7ff88fe00640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff87006c920 0x7ff87006ede0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff8900d06d0 tx=0x7ff87c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.051+0000 7ff895600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558823f0 0x55d1558c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.051+0000 7ff895600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558823f0 0x55d1558c8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49166/0 (socket says 172.21.15.40:49166) 2024-04-26T23:03:54.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.051+0000 7ff895600640 1 -- 172.21.15.40:0/88262666 learned_addr learned my addr 172.21.15.40:0/88262666 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.052+0000 7ff894c00640 1 --2- 172.21.15.40:0/88262666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558ac710 0x55d1558acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.052+0000 7ff88fe00640 1 --2- 172.21.15.40:0/88262666 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558c9fb0 0x55d1558ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.052+0000 7ff895600640 1 -- 172.21.15.40:0/88262666 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558c9fb0 msgr2=0x55d1558ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.052+0000 7ff895600640 1 --2- 172.21.15.40:0/88262666 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558c9fb0 0x55d1558ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.052+0000 7ff895600640 1 -- 172.21.15.40:0/88262666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558ac710 msgr2=0x55d1558acaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.052+0000 7ff895600640 1 --2- 172.21.15.40:0/88262666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558ac710 0x55d1558acaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.052+0000 7ff895600640 1 -- 172.21.15.40:0/88262666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d155774050 con 0x55d1558823f0 2024-04-26T23:03:54.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.052+0000 7ff895600640 1 --2- 172.21.15.40:0/88262666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558823f0 0x55d1558c8670 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff888001de0 tx=0x7ff88800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b81c76d1490a571a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.052+0000 7ff881200640 1 -- 172.21.15.40:0/88262666 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88800de30 con 0x55d1558823f0 2024-04-26T23:03:54.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.052+0000 7ff881200640 1 -- 172.21.15.40:0/88262666 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff888004b60 con 0x55d1558823f0 2024-04-26T23:03:54.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.052+0000 7ff881200640 1 -- 172.21.15.40:0/88262666 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8880070d0 con 0x55d1558823f0 2024-04-26T23:03:54.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.053+0000 7ff89a477740 1 -- 172.21.15.40:0/88262666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558823f0 msgr2=0x55d1558c8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.053+0000 7ff89a477740 1 --2- 172.21.15.40:0/88262666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558823f0 0x55d1558c8670 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff888001de0 tx=0x7ff88800d520 comp rx=0 tx=0).stop 2024-04-26T23:03:54.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.053+0000 7ff89a477740 1 -- 172.21.15.40:0/88262666 shutdown_connections 2024-04-26T23:03:54.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.053+0000 7ff89a477740 1 --2- 172.21.15.40:0/88262666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558823f0 0x55d1558c8670 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.053+0000 7ff89a477740 1 --2- 172.21.15.40:0/88262666 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558c9fb0 0x55d1558ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.053+0000 7ff89a477740 1 --2- 172.21.15.40:0/88262666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558ac710 0x55d1558acaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.053+0000 7ff89a477740 1 -- 172.21.15.40:0/88262666 >> 172.21.15.40:0/88262666 conn(0x55d15587ed20 msgr2=0x55d1558c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.053+0000 7ff89a477740 1 -- 172.21.15.40:0/88262666 shutdown_connections 2024-04-26T23:03:54.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.053+0000 7ff89a477740 1 -- 172.21.15.40:0/88262666 wait complete. 2024-04-26T23:03:54.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.054+0000 7ff89a477740 1 -- start start 2024-04-26T23:03:54.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.054+0000 7ff89a477740 1 Processor -- start 2024-04-26T23:03:54.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.055+0000 7ff89a477740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558ac710 0x55d1558a8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.055+0000 7ff89a477740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558c9fb0 0x55d1558a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.055+0000 7ff89a477740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.055+0000 7ff89a477740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d1557ea200 con 0x55d1558c6f50 2024-04-26T23:03:54.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.055+0000 7ff89a477740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d1557e4ee0 con 0x55d1558c9fb0 2024-04-26T23:03:54.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.055+0000 7ff89a477740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d1557f1e70 con 0x55d1558ac710 2024-04-26T23:03:54.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.055+0000 7ff89a477740 10 client.0 _ll_register_callbacks cb 0x55d1557db270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:54.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.055+0000 7ff895600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.055+0000 7ff895600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558c84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49180/0 (socket says 172.21.15.40:49180) 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.055+0000 7ff895600640 1 -- 172.21.15.40:0/2011221223 learned_addr learned my addr 172.21.15.40:0/2011221223 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.056+0000 7ff894c00640 1 --2- 172.21.15.40:0/2011221223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558ac710 0x55d1558a8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.056+0000 7ff895600640 1 -- 172.21.15.40:0/2011221223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558ac710 msgr2=0x55d1558a8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.056+0000 7ff895600640 1 --2- 172.21.15.40:0/2011221223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558ac710 0x55d1558a8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.056+0000 7ff895600640 1 -- 172.21.15.40:0/2011221223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558c9fb0 msgr2=0x55d1558a4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.056+0000 7ff895600640 1 --2- 172.21.15.40:0/2011221223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558c9fb0 0x55d1558a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.056+0000 7ff895600640 1 -- 172.21.15.40:0/2011221223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1555c17d0 con 0x55d1558c6f50 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.056+0000 7ff895600640 1 --2- 172.21.15.40:0/2011221223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558c84a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff88800da20 tx=0x7ff888007cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=debfbbca00881b5d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.056+0000 7ff875600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.056+0000 7ff875600640 1 -- 172.21.15.40:0/2011221223 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff888023030 con 0x55d1558c6f50 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.056+0000 7ff875600640 1 -- 172.21.15.40:0/2011221223 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7ff88801a040 con 0x55d1558c6f50 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.057+0000 7ff89a477740 1 -- 172.21.15.40:0/2011221223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff888004b60 con 0x55d1558c6f50 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.057+0000 7ff875600640 1 -- 172.21.15.40:0/2011221223 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff8880072f0 con 0x55d1558c6f50 2024-04-26T23:03:54.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.057+0000 7ff89a477740 10 client.5745 fetch_fsmap learned FSMap version 26 2024-04-26T23:03:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.057+0000 7ff89a477740 1 -- 172.21.15.40:0/2011221223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) -- 0x55d1554f73f0 con 0x55d1558c6f50 2024-04-26T23:03:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.057+0000 7ff875600640 1 -- 172.21.15.40:0/2011221223 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff888004b60 con 0x55d1558c6f50 2024-04-26T23:03:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.057+0000 7ff875600640 1 -- 172.21.15.40:0/2011221223 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) ==== 3550+0+0 (secure 0 0 0) 0x7ff888007490 con 0x55d1558c6f50 2024-04-26T23:03:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.057+0000 7ff89a477740 10 client.5745 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T23:03:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.057+0000 7ff89a477740 10 client.5745 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-26T23:03:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.057+0000 7ff89a477740 20 client.5745 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.057+0000 7ff89a477740 1 --2- 172.21.15.40:0/2011221223 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55d1558dad40 0x55d1558db190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.057+0000 7ff89a477740 4 client.5745 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:03:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.057+0000 7ff89a477740 1 -- 172.21.15.40:0/2011221223 --> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1557f37e0 con 0x55d1558dad40 2024-04-26T23:03:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.058+0000 7ff894c00640 1 --2- 172.21.15.40:0/2011221223 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55d1558dad40 0x55d1558db190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.058+0000 7ff894c00640 1 --2- 172.21.15.40:0/2011221223 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55d1558dad40 0x55d1558db190 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=992ccd94b42914bc server_cookie=f888c724d57550c4 in_seq=0 out_seq=0 2024-04-26T23:03:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.058+0000 7ff875600640 10 client.5745 ms_handle_connect on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.059+0000 7ff875600640 1 -- 172.21.15.40:0/2011221223 <== mds.0 v2:172.21.15.40:6833/299800334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d1557f37e0 con 0x55d1558dad40 2024-04-26T23:03:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.059+0000 7ff875600640 10 client.5745 handle_command_reply: tid=0 2024-04-26T23:03:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.059+0000 7ff89a477740 1 client.5745 shutdown 2024-04-26T23:03:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.059+0000 7ff89a477740 1 -- 172.21.15.40:0/2011221223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 msgr2=0x55d1558c84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.059+0000 7ff89a477740 1 --2- 172.21.15.40:0/2011221223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558c84a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff88800da20 tx=0x7ff888007cc0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.059+0000 7ff89a477740 1 -- 172.21.15.40:0/2011221223 shutdown_connections 2024-04-26T23:03:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.059+0000 7ff89a477740 1 --2- 172.21.15.40:0/2011221223 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55d1558dad40 0x55d1558db190 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.059+0000 7ff89a477740 1 --2- 172.21.15.40:0/2011221223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558c84a0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.059+0000 7ff89a477740 1 --2- 172.21.15.40:0/2011221223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558c9fb0 0x55d1558a4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.059+0000 7ff89a477740 1 --2- 172.21.15.40:0/2011221223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558ac710 0x55d1558a8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.059+0000 7ff89a477740 1 -- 172.21.15.40:0/2011221223 >> 172.21.15.40:0/2011221223 conn(0x55d1558823f0 msgr2=0x55d15587b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.059+0000 7ff875600640 0 client.5745 ms_handle_reset on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.060+0000 7ff89a477740 1 -- 172.21.15.40:0/2011221223 shutdown_connections 2024-04-26T23:03:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.060+0000 7ff89a477740 1 -- 172.21.15.40:0/2011221223 wait complete. 2024-04-26T23:03:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.060+0000 7ff89a477740 20 client.5745 trim_cache size 0 max 16384 2024-04-26T23:03:54.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.079+0000 7ff89a477740 1 Processor -- start 2024-04-26T23:03:54.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.079+0000 7ff89a477740 1 -- start start 2024-04-26T23:03:54.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.080+0000 7ff89a477740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558ac710 0x55d1558c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.080+0000 7ff89a477740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558c7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.080+0000 7ff89a477740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558c9fb0 0x55d1558e1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.080+0000 7ff89a477740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff890201c30 con 0x55d1558c6f50 2024-04-26T23:03:54.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.080+0000 7ff89a477740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff890200dd0 con 0x55d1558ac710 2024-04-26T23:03:54.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.080+0000 7ff89a477740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff890201500 con 0x55d1558c9fb0 2024-04-26T23:03:54.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.081+0000 7ff88fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.081+0000 7ff88fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49184/0 (socket says 172.21.15.40:49184) 2024-04-26T23:03:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.081+0000 7ff88fe00640 1 -- 172.21.15.40:0/1019543552 learned_addr learned my addr 172.21.15.40:0/1019543552 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.081+0000 7ff894c00640 1 --2- 172.21.15.40:0/1019543552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558ac710 0x55d1558c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.081+0000 7ff895600640 1 --2- 172.21.15.40:0/1019543552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558c9fb0 0x55d1558e1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.081+0000 7ff88fe00640 1 -- 172.21.15.40:0/1019543552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558c9fb0 msgr2=0x55d1558e1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.081+0000 7ff88fe00640 1 --2- 172.21.15.40:0/1019543552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558c9fb0 0x55d1558e1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.081+0000 7ff88fe00640 1 -- 172.21.15.40:0/1019543552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558ac710 msgr2=0x55d1558c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.081+0000 7ff88fe00640 1 --2- 172.21.15.40:0/1019543552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558ac710 0x55d1558c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.081+0000 7ff88fe00640 1 -- 172.21.15.40:0/1019543552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1554b70f0 con 0x55d1558c6f50 2024-04-26T23:03:54.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.082+0000 7ff88fe00640 1 --2- 172.21.15.40:0/1019543552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558c7870 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00dad0 tx=0x7ff87c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c151c8aebbe361 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.082+0000 7ff877e00640 1 -- 172.21.15.40:0/1019543552 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c019070 con 0x55d1558c6f50 2024-04-26T23:03:54.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.082+0000 7ff877e00640 1 -- 172.21.15.40:0/1019543552 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff87c024070 con 0x55d1558c6f50 2024-04-26T23:03:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.082+0000 7ff877e00640 1 -- 172.21.15.40:0/1019543552 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c012c30 con 0x55d1558c6f50 2024-04-26T23:03:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.082+0000 7ff89a477740 1 -- 172.21.15.40:0/1019543552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 msgr2=0x55d1558c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.082+0000 7ff89a477740 1 --2- 172.21.15.40:0/1019543552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558c7870 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00dad0 tx=0x7ff87c010000 comp rx=0 tx=0).stop 2024-04-26T23:03:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.082+0000 7ff89a477740 1 -- 172.21.15.40:0/1019543552 shutdown_connections 2024-04-26T23:03:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.082+0000 7ff89a477740 1 --2- 172.21.15.40:0/1019543552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558c9fb0 0x55d1558e1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.082+0000 7ff89a477740 1 --2- 172.21.15.40:0/1019543552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558c7870 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.082+0000 7ff89a477740 1 --2- 172.21.15.40:0/1019543552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558ac710 0x55d1558c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.082+0000 7ff89a477740 1 -- 172.21.15.40:0/1019543552 >> 172.21.15.40:0/1019543552 conn(0x55d1558823f0 msgr2=0x55d155878080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.083+0000 7ff89a477740 1 -- 172.21.15.40:0/1019543552 shutdown_connections 2024-04-26T23:03:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.083+0000 7ff89a477740 1 -- 172.21.15.40:0/1019543552 wait complete. 2024-04-26T23:03:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.083+0000 7ff89a477740 1 -- start start 2024-04-26T23:03:54.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.084+0000 7ff89a477740 1 Processor -- start 2024-04-26T23:03:54.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.084+0000 7ff89a477740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558ac710 0x55d1558af920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff89a477740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558eab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff89a477740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558c9fb0 0x55d1558c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff89a477740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d1557f2fd0 con 0x55d1558c6f50 2024-04-26T23:03:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff89a477740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff8901f5f70 con 0x55d1558c9fb0 2024-04-26T23:03:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff89a477740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d1558c9230 con 0x55d1558ac710 2024-04-26T23:03:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff89a477740 10 client.0 _ll_register_callbacks cb 0x7ff888001c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:03:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff88fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558eab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff88fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558eab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49196/0 (socket says 172.21.15.40:49196) 2024-04-26T23:03:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff88fe00640 1 -- 172.21.15.40:0/693212034 learned_addr learned my addr 172.21.15.40:0/693212034 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff894c00640 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558ac710 0x55d1558af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff895600640 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558c9fb0 0x55d1558c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff88fe00640 1 -- 172.21.15.40:0/693212034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558ac710 msgr2=0x55d1558af920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff88fe00640 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558ac710 0x55d1558af920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff88fe00640 1 -- 172.21.15.40:0/693212034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558c9fb0 msgr2=0x55d1558c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff88fe00640 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558c9fb0 0x55d1558c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff88fe00640 1 -- 172.21.15.40:0/693212034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1557ada30 con 0x55d1558c6f50 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.085+0000 7ff895600640 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558c9fb0 0x55d1558c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.086+0000 7ff88fe00640 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558eab00 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff87c008010 tx=0x7ff87c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aca950e28d747e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.086+0000 7ff875600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.086+0000 7ff875600640 1 -- 172.21.15.40:0/693212034 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c012e40 con 0x55d1558c6f50 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.086+0000 7ff875600640 1 -- 172.21.15.40:0/693212034 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7ff87c033c90 con 0x55d1558c6f50 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.086+0000 7ff89a477740 1 -- 172.21.15.40:0/693212034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff87c024070 con 0x55d1558c6f50 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.086+0000 7ff875600640 1 -- 172.21.15.40:0/693212034 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff87c028420 con 0x55d1558c6f50 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.086+0000 7ff89a477740 10 client.5751 fetch_fsmap learned FSMap version 26 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.086+0000 7ff89a477740 1 -- 172.21.15.40:0/693212034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) -- 0x55d15587ea00 con 0x55d1558c6f50 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.087+0000 7ff875600640 1 -- 172.21.15.40:0/693212034 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff87c024070 con 0x55d1558c6f50 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.087+0000 7ff875600640 1 -- 172.21.15.40:0/693212034 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) ==== 3550+0+0 (secure 0 0 0) 0x7ff87c03a050 con 0x55d1558c6f50 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.087+0000 7ff89a477740 10 client.5751 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T23:03:54.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.087+0000 7ff89a477740 10 client.5751 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-26T23:03:54.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.087+0000 7ff89a477740 20 client.5751 populate_metadata read hostname 'smithi040' 2024-04-26T23:03:54.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.087+0000 7ff89a477740 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55d1559413f0 0x55d1558c7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.087+0000 7ff89a477740 4 client.5751 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:03:54.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.087+0000 7ff89a477740 1 -- 172.21.15.40:0/693212034 --> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d1557ed650 con 0x55d1559413f0 2024-04-26T23:03:54.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.087+0000 7ff894c00640 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55d1559413f0 0x55d1558c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff894c00640 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55d1559413f0 0x55d1558c7870 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cdda35b22aa1ebf server_cookie=6960378db23f2bbc in_seq=0 out_seq=0 2024-04-26T23:03:54.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff875600640 10 client.5751 ms_handle_connect on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:54.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff875600640 1 -- 172.21.15.40:0/693212034 <== mds.0 v2:172.21.15.40:6833/299800334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d1557ed650 con 0x55d1559413f0 2024-04-26T23:03:54.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff875600640 10 client.5751 handle_command_reply: tid=0 2024-04-26T23:03:54.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff89a477740 1 client.5751 shutdown 2024-04-26T23:03:54.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff89a477740 1 -- 172.21.15.40:0/693212034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 msgr2=0x55d1558eab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff89a477740 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558eab00 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff87c008010 tx=0x7ff87c004730 comp rx=0 tx=0).stop 2024-04-26T23:03:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff89a477740 1 -- 172.21.15.40:0/693212034 shutdown_connections 2024-04-26T23:03:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff89a477740 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.40:6833/299800334,v1:172.21.15.40:6835/299800334] conn(0x55d1559413f0 0x55d1558c7870 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff89a477740 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1558c9fb0 0x55d1558c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff89a477740 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1558c6f50 0x55d1558eab00 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff89a477740 1 --2- 172.21.15.40:0/693212034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1558ac710 0x55d1558af920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff875600640 0 client.5751 ms_handle_reset on v2:172.21.15.40:6833/299800334 2024-04-26T23:03:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff89a477740 1 -- 172.21.15.40:0/693212034 >> 172.21.15.40:0/693212034 conn(0x55d1558823f0 msgr2=0x55d1558abc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.088+0000 7ff89a477740 1 -- 172.21.15.40:0/693212034 shutdown_connections 2024-04-26T23:03:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.089+0000 7ff89a477740 1 -- 172.21.15.40:0/693212034 wait complete. 2024-04-26T23:03:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.089+0000 7ff89a477740 20 client.5751 trim_cache size 0 max 16384 2024-04-26T23:03:54.093 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:03:54.093 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-04-26T23:03:54.093 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-04-26T23:03:54.093 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:03:54.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.089+0000 7ff896000640 1 -- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff87006c920 msgr2=0x7ff87006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.089+0000 7ff896000640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff87006c920 0x7ff87006ede0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff8900d06d0 tx=0x7ff87c008040 comp rx=0 tx=0).stop 2024-04-26T23:03:54.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.089+0000 7ff896000640 1 -- 172.21.15.40:0/2500799144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff890073600 msgr2=0x7ff890147470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.089+0000 7ff896000640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff890073600 0x7ff890147470 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff884000c00 tx=0x7ff8840046c0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.089+0000 7ff896000640 1 -- 172.21.15.40:0/2500799144 shutdown_connections 2024-04-26T23:03:54.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.089+0000 7ff896000640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff87006c920 0x7ff87006ede0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.089+0000 7ff896000640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff89015fcb0 0x7ff89015a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.089+0000 7ff896000640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8900739e0 0x7ff89015a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.089+0000 7ff896000640 1 --2- 172.21.15.40:0/2500799144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff890073600 0x7ff890147470 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.089+0000 7ff896000640 1 -- 172.21.15.40:0/2500799144 >> 172.21.15.40:0/2500799144 conn(0x7ff8900b2720 msgr2=0x7ff8900b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.090+0000 7ff896000640 1 -- 172.21.15.40:0/2500799144 shutdown_connections 2024-04-26T23:03:54.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.090+0000 7ff896000640 1 -- 172.21.15.40:0/2500799144 wait complete. 2024-04-26T23:03:54.108 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:03:54.108 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000002.00000000 scrub_tag 2024-04-26T23:03:54.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.220+0000 7f8a0bdc4f40 1 Processor -- start 2024-04-26T23:03:54.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.220+0000 7f8a0bdc4f40 1 -- start start 2024-04-26T23:03:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.221+0000 7f8a0bdc4f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55add8a2fe90 0x55add8a2d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.221+0000 7f8a0bdc4f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55add8a31070 0x55add8ad2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.221+0000 7f8a0bdc4f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55add8ad87c0 0x55add8a4a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.221+0000 7f8a0bdc4f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55add8ad6310 con 0x55add8a31070 2024-04-26T23:03:54.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.221+0000 7f8a0bdc4f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55add8ad6030 con 0x55add8a2fe90 2024-04-26T23:03:54.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.221+0000 7f8a0bdc4f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55add8ad65f0 con 0x55add8ad87c0 2024-04-26T23:03:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.222+0000 7f8a08600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55add8a31070 0x55add8ad2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.222+0000 7f8a08600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55add8a31070 0x55add8ad2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49212/0 (socket says 172.21.15.40:49212) 2024-04-26T23:03:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.222+0000 7f8a08600640 1 -- 172.21.15.40:0/3420879849 learned_addr learned my addr 172.21.15.40:0/3420879849 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.222+0000 7f8a09000640 1 --2- 172.21.15.40:0/3420879849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55add8a2fe90 0x55add8a2d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.222+0000 7f8a09a00640 1 --2- 172.21.15.40:0/3420879849 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55add8ad87c0 0x55add8a4a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.222+0000 7f8a08600640 1 -- 172.21.15.40:0/3420879849 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55add8ad87c0 msgr2=0x55add8a4a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.222+0000 7f8a08600640 1 --2- 172.21.15.40:0/3420879849 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55add8ad87c0 0x55add8a4a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.222+0000 7f8a08600640 1 -- 172.21.15.40:0/3420879849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55add8a2fe90 msgr2=0x55add8a2d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.222+0000 7f8a08600640 1 --2- 172.21.15.40:0/3420879849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55add8a2fe90 0x55add8a2d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.222+0000 7f8a08600640 1 -- 172.21.15.40:0/3420879849 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55add8928b30 con 0x55add8a31070 2024-04-26T23:03:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.223+0000 7f8a08600640 1 --2- 172.21.15.40:0/3420879849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55add8a31070 0x55add8ad2210 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f89f400a770 tx=0x7f89f400a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253f521c2b52b999 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.223+0000 7f8a07200640 1 -- 172.21.15.40:0/3420879849 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4018070 con 0x55add8a31070 2024-04-26T23:03:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.223+0000 7f8a07200640 1 -- 172.21.15.40:0/3420879849 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89f4007600 con 0x55add8a31070 2024-04-26T23:03:54.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.223+0000 7f8a07200640 1 -- 172.21.15.40:0/3420879849 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f40079d0 con 0x55add8a31070 2024-04-26T23:03:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.223+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/3420879849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55add8a31070 msgr2=0x55add8ad2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.223+0000 7f8a0bdc4f40 1 --2- 172.21.15.40:0/3420879849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55add8a31070 0x55add8ad2210 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f89f400a770 tx=0x7f89f400a7a0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.224+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/3420879849 shutdown_connections 2024-04-26T23:03:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.224+0000 7f8a0bdc4f40 1 --2- 172.21.15.40:0/3420879849 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55add8ad87c0 0x55add8a4a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.224+0000 7f8a0bdc4f40 1 --2- 172.21.15.40:0/3420879849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55add8a31070 0x55add8ad2210 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.224+0000 7f8a0bdc4f40 1 --2- 172.21.15.40:0/3420879849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55add8a2fe90 0x55add8a2d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.224+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/3420879849 >> 172.21.15.40:0/3420879849 conn(0x55add8a4bdf0 msgr2=0x55add8a2c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.224+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/3420879849 shutdown_connections 2024-04-26T23:03:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.224+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/3420879849 wait complete. 2024-04-26T23:03:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.224+0000 7f8a0bdc4f40 1 Processor -- start 2024-04-26T23:03:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.225+0000 7f8a0bdc4f40 1 -- start start 2024-04-26T23:03:54.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a0bdc4f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55add8a2fe90 0x55add8ad7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a0bdc4f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55add8a31070 0x55add8ac9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a0bdc4f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55add8ad87c0 0x55add8ac9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a0bdc4f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55add8ad2750 con 0x55add8a31070 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a0bdc4f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55add89feb70 con 0x55add8ad87c0 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a0bdc4f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55add8a2ca20 con 0x55add8a2fe90 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a08600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55add8a31070 0x55add8ac9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a08600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55add8a31070 0x55add8ac9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49220/0 (socket says 172.21.15.40:49220) 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a08600640 1 -- 172.21.15.40:0/916712993 learned_addr learned my addr 172.21.15.40:0/916712993 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a09000640 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55add8a2fe90 0x55add8ad7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a09a00640 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55add8ad87c0 0x55add8ac9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a08600640 1 -- 172.21.15.40:0/916712993 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55add8a2fe90 msgr2=0x55add8ad7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a08600640 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55add8a2fe90 0x55add8ad7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a08600640 1 -- 172.21.15.40:0/916712993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55add8ad87c0 msgr2=0x55add8ac9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a08600640 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55add8ad87c0 0x55add8ac9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a08600640 1 -- 172.21.15.40:0/916712993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55add8acaa70 con 0x55add8a31070 2024-04-26T23:03:54.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.226+0000 7f8a09a00640 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55add8ad87c0 0x55add8ac9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:54.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.227+0000 7f8a08600640 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55add8a31070 0x55add8ac9420 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f89f400a7e0 tx=0x7f89f4003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.227+0000 7f8a05e00640 1 -- 172.21.15.40:0/916712993 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4021020 con 0x55add8a31070 2024-04-26T23:03:54.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.227+0000 7f8a05e00640 1 -- 172.21.15.40:0/916712993 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89f4019040 con 0x55add8a31070 2024-04-26T23:03:54.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.227+0000 7f8a05e00640 1 -- 172.21.15.40:0/916712993 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4018070 con 0x55add8a31070 2024-04-26T23:03:54.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.227+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/916712993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55add8ac63e0 con 0x55add8a31070 2024-04-26T23:03:54.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.227+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/916712993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55add8acf2d0 con 0x55add8a31070 2024-04-26T23:03:54.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.229+0000 7f8a05e00640 1 -- 172.21.15.40:0/916712993 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f89f4024020 con 0x55add8a31070 2024-04-26T23:03:54.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.230+0000 7f8a05e00640 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f89e806b300 0x7f89e806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.230+0000 7f8a05e00640 1 -- 172.21.15.40:0/916712993 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f89f408b7d0 con 0x55add8a31070 2024-04-26T23:03:54.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.231+0000 7f8a0bdc4f40 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x55add8ad7e50 0x55add8ad4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.231+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/916712993 --> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] -- osd_op(unknown.0.0:1 5.12 5:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) -- 0x55add8ac6ce0 con 0x55add8ad7e50 2024-04-26T23:03:54.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.231+0000 7f8a09a00640 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x55add8ad7e50 0x55add8ad4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.231+0000 7f8a09000640 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f89e806b300 0x7f89e806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.232+0000 7f8a09a00640 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x55add8ad7e50 0x55add8ad4230 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.232+0000 7f8a09a00640 1 -- 172.21.15.40:0/916712993 <== osd.1 v2:172.21.15.40:6815/2323014490 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr (10) out=10b] v0'0 uv8 ondisk = 0) ==== 164+0+10 (crc 0 0 0) 0x7f8a00051860 con 0x55add8ad7e50 2024-04-26T23:03:54.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.232+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/916712993 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x55add8ad7e50 msgr2=0x55add8ad4230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.232+0000 7f8a0bdc4f40 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x55add8ad7e50 0x55add8ad4230 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.232+0000 7f8a09000640 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f89e806b300 0x7f89e806d7c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x55add8a49730 tx=0x7f89f80058c0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.232+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/916712993 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f89e806b300 msgr2=0x7f89e806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.232+0000 7f8a0bdc4f40 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f89e806b300 0x7f89e806d7c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x55add8a49730 tx=0x7f89f80058c0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.232+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/916712993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55add8a31070 msgr2=0x55add8ac9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.232+0000 7f8a0bdc4f40 1 --2- 172.21.15.40:0/916712993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55add8a31070 0x55add8ac9420 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f89f400a7e0 tx=0x7f89f4003020 comp rx=0 tx=0).stop 2024-04-26T23:03:54.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.232+0000 7f8a09a00640 1 -- 172.21.15.40:0/916712993 reap_dead start 2024-04-26T23:03:54.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.232+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/916712993 shutdown_connections 2024-04-26T23:03:54.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.232+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/916712993 >> 172.21.15.40:0/916712993 conn(0x55add8a4bdf0 msgr2=0x55add8a2c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.232+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/916712993 shutdown_connections 2024-04-26T23:03:54.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.233+0000 7f8a0bdc4f40 1 -- 172.21.15.40:0/916712993 wait complete. 2024-04-26T23:03:54.237 INFO:teuthology.orchestra.run.smithi040.stdout:tag123 2024-04-26T23:03:54.237 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000003.00000000 scrub_tag 2024-04-26T23:03:54.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.326+0000 7f7706e20f40 1 Processor -- start 2024-04-26T23:03:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.327+0000 7f7706e20f40 1 -- start start 2024-04-26T23:03:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.327+0000 7f7706e20f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5614e13e5e90 0x5614e147fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.327+0000 7f7706e20f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614e13e7070 0x5614e13b2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.327+0000 7f7706e20f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5614e14016e0 0x5614e13b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.327+0000 7f7706e20f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5614e148c030 con 0x5614e13e7070 2024-04-26T23:03:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.327+0000 7f7706e20f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5614e135bd20 con 0x5614e14016e0 2024-04-26T23:03:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.327+0000 7f7706e20f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5614e148c310 con 0x5614e13e5e90 2024-04-26T23:03:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.328+0000 7f7703600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614e13e7070 0x5614e13b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.328+0000 7f7703600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614e13e7070 0x5614e13b2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49236/0 (socket says 172.21.15.40:49236) 2024-04-26T23:03:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.328+0000 7f7703600640 1 -- 172.21.15.40:0/2289191916 learned_addr learned my addr 172.21.15.40:0/2289191916 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.328+0000 7f7704000640 1 --2- 172.21.15.40:0/2289191916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5614e13e5e90 0x5614e147fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.328+0000 7f7703600640 1 -- 172.21.15.40:0/2289191916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5614e13e5e90 msgr2=0x5614e147fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.328+0000 7f7703600640 1 --2- 172.21.15.40:0/2289191916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5614e13e5e90 0x5614e147fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.328+0000 7f7704a00640 1 --2- 172.21.15.40:0/2289191916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5614e14016e0 0x5614e13b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.328+0000 7f7703600640 1 -- 172.21.15.40:0/2289191916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5614e14016e0 msgr2=0x5614e13b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.328+0000 7f7703600640 1 --2- 172.21.15.40:0/2289191916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5614e14016e0 0x5614e13b26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.328+0000 7f7703600640 1 -- 172.21.15.40:0/2289191916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614e148d280 con 0x5614e13e7070 2024-04-26T23:03:54.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.329+0000 7f7703600640 1 --2- 172.21.15.40:0/2289191916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614e13e7070 0x5614e13b2190 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00a770 tx=0x7f76ec00a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edfecbec2e1f699 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.329+0000 7f7702200640 1 -- 172.21.15.40:0/2289191916 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec018070 con 0x5614e13e7070 2024-04-26T23:03:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.329+0000 7f7702200640 1 -- 172.21.15.40:0/2289191916 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76ec007600 con 0x5614e13e7070 2024-04-26T23:03:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.329+0000 7f7702200640 1 -- 172.21.15.40:0/2289191916 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec0079f0 con 0x5614e13e7070 2024-04-26T23:03:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.329+0000 7f7706e20f40 1 -- 172.21.15.40:0/2289191916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614e13e7070 msgr2=0x5614e13b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.329+0000 7f7706e20f40 1 --2- 172.21.15.40:0/2289191916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614e13e7070 0x5614e13b2190 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00a770 tx=0x7f76ec00a7a0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.329+0000 7f7706e20f40 1 -- 172.21.15.40:0/2289191916 shutdown_connections 2024-04-26T23:03:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.329+0000 7f7706e20f40 1 --2- 172.21.15.40:0/2289191916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5614e14016e0 0x5614e13b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.329+0000 7f7706e20f40 1 --2- 172.21.15.40:0/2289191916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614e13e7070 0x5614e13b2190 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.329+0000 7f7706e20f40 1 --2- 172.21.15.40:0/2289191916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5614e13e5e90 0x5614e147fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.329+0000 7f7706e20f40 1 -- 172.21.15.40:0/2289191916 >> 172.21.15.40:0/2289191916 conn(0x5614e1401df0 msgr2=0x5614e14021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.330+0000 7f7706e20f40 1 -- 172.21.15.40:0/2289191916 shutdown_connections 2024-04-26T23:03:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.330+0000 7f7706e20f40 1 -- 172.21.15.40:0/2289191916 wait complete. 2024-04-26T23:03:54.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.330+0000 7f7706e20f40 1 Processor -- start 2024-04-26T23:03:54.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.330+0000 7f7706e20f40 1 -- start start 2024-04-26T23:03:54.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.331+0000 7f7706e20f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5614e13e5e90 0x5614e147d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.331+0000 7f7706e20f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5614e13e7070 0x5614e148e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.331+0000 7f7706e20f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614e14016e0 0x5614e148e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.331+0000 7f7706e20f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5614e148d520 con 0x5614e14016e0 2024-04-26T23:03:54.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.331+0000 7f7706e20f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5614e13e5cc0 con 0x5614e13e5e90 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.331+0000 7f7706e20f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5614e1483b80 con 0x5614e13e7070 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.332+0000 7f7704a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614e14016e0 0x5614e148e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.332+0000 7f7704a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614e14016e0 0x5614e148e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49240/0 (socket says 172.21.15.40:49240) 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.332+0000 7f7704a00640 1 -- 172.21.15.40:0/4270549284 learned_addr learned my addr 172.21.15.40:0/4270549284 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.332+0000 7f7704000640 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5614e13e5e90 0x5614e147d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.332+0000 7f7704a00640 1 -- 172.21.15.40:0/4270549284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5614e13e7070 msgr2=0x5614e148e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.332+0000 7f7703600640 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5614e13e7070 0x5614e148e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.332+0000 7f7704a00640 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5614e13e7070 0x5614e148e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.332+0000 7f7704a00640 1 -- 172.21.15.40:0/4270549284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5614e13e5e90 msgr2=0x5614e147d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.332+0000 7f7704a00640 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5614e13e5e90 0x5614e147d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.332+0000 7f7704a00640 1 -- 172.21.15.40:0/4270549284 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614e1483350 con 0x5614e14016e0 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.332+0000 7f7704000640 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5614e13e5e90 0x5614e147d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.332+0000 7f7704a00640 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614e14016e0 0x5614e148e9c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00cc90 tx=0x7f76fc00a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.333+0000 7f7700e00640 1 -- 172.21.15.40:0/4270549284 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc008040 con 0x5614e14016e0 2024-04-26T23:03:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.333+0000 7f7700e00640 1 -- 172.21.15.40:0/4270549284 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76fc014020 con 0x5614e14016e0 2024-04-26T23:03:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.333+0000 7f7700e00640 1 -- 172.21.15.40:0/4270549284 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc00f400 con 0x5614e14016e0 2024-04-26T23:03:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.333+0000 7f7706e20f40 1 -- 172.21.15.40:0/4270549284 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5614e13e21a0 con 0x5614e14016e0 2024-04-26T23:03:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.333+0000 7f7706e20f40 1 -- 172.21.15.40:0/4270549284 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614e13e1ea0 con 0x5614e14016e0 2024-04-26T23:03:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.335+0000 7f7700e00640 1 -- 172.21.15.40:0/4270549284 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f76fc00f5a0 con 0x5614e14016e0 2024-04-26T23:03:54.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.336+0000 7f7700e00640 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f76e006b2d0 0x7f76e006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.336+0000 7f7700e00640 1 -- 172.21.15.40:0/4270549284 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f76fc08a950 con 0x5614e14016e0 2024-04-26T23:03:54.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.337+0000 7f7706e20f40 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5614e147e7b0 0x5614e1516950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.337+0000 7f7706e20f40 1 -- 172.21.15.40:0/4270549284 --> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] -- osd_op(unknown.0.0:1 5.12 5:4def72d4:::10000000003.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) -- 0x5614e1516e90 con 0x5614e147e7b0 2024-04-26T23:03:54.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.337+0000 7f7704000640 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f76e006b2d0 0x7f76e006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.337+0000 7f7703600640 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5614e147e7b0 0x5614e1516950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.337+0000 7f7703600640 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5614e147e7b0 0x5614e1516950 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.338+0000 7f7703600640 1 -- 172.21.15.40:0/4270549284 <== osd.1 v2:172.21.15.40:6815/2323014490 1 ==== osd_op_reply(1 10000000003.00000000 [getxattr (10) out=10b] v0'0 uv9 ondisk = 0) ==== 164+0+10 (crc 0 0 0) 0x7f76ec007b30 con 0x5614e147e7b0 2024-04-26T23:03:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.338+0000 7f7706e20f40 1 -- 172.21.15.40:0/4270549284 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5614e147e7b0 msgr2=0x5614e1516950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.338+0000 7f7706e20f40 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5614e147e7b0 0x5614e1516950 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.338+0000 7f7706e20f40 1 -- 172.21.15.40:0/4270549284 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f76e006b2d0 msgr2=0x7f76e006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.338+0000 7f7706e20f40 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f76e006b2d0 0x7f76e006d790 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f76f4057000 tx=0x7f76f4007030 comp rx=0 tx=0).stop 2024-04-26T23:03:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.338+0000 7f7706e20f40 1 -- 172.21.15.40:0/4270549284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614e14016e0 msgr2=0x5614e148e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.338+0000 7f7706e20f40 1 --2- 172.21.15.40:0/4270549284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5614e14016e0 0x5614e148e9c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00cc90 tx=0x7f76fc00a5d0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.338+0000 7f7704a00640 1 -- 172.21.15.40:0/4270549284 reap_dead start 2024-04-26T23:03:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.338+0000 7f7706e20f40 1 -- 172.21.15.40:0/4270549284 shutdown_connections 2024-04-26T23:03:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.338+0000 7f7706e20f40 1 -- 172.21.15.40:0/4270549284 >> 172.21.15.40:0/4270549284 conn(0x5614e1401df0 msgr2=0x5614e14021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.338+0000 7f7706e20f40 1 -- 172.21.15.40:0/4270549284 shutdown_connections 2024-04-26T23:03:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.338+0000 7f7706e20f40 1 -- 172.21.15.40:0/4270549284 wait complete. 2024-04-26T23:03:54.343 INFO:teuthology.orchestra.run.smithi040.stdout:tag123 2024-04-26T23:03:54.343 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000004.00000000 scrub_tag 2024-04-26T23:03:54.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.450+0000 7fb8f5ce7f40 1 Processor -- start 2024-04-26T23:03:54.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.450+0000 7fb8f5ce7f40 1 -- start start 2024-04-26T23:03:54.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.451+0000 7fb8f5ce7f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5602a1a910a0 0x5602a1aa97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.451+0000 7fb8f5ce7f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602a1a91480 0x5602a1b3a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.451+0000 7fb8f5ce7f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602a1aab290 0x5602a1b3a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.451+0000 7fb8f5ce7f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5602a1aa6880 con 0x5602a1aab290 2024-04-26T23:03:54.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.451+0000 7fb8f5ce7f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5602a1a08d20 con 0x5602a1a91480 2024-04-26T23:03:54.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.451+0000 7fb8f5ce7f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5602a1aa6b20 con 0x5602a1a910a0 2024-04-26T23:03:54.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.452+0000 7fb8f3800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602a1aab290 0x5602a1b3a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.452+0000 7fb8f3800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602a1aab290 0x5602a1b3a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49256/0 (socket says 172.21.15.40:49256) 2024-04-26T23:03:54.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.452+0000 7fb8f3800640 1 -- 172.21.15.40:0/3465843704 learned_addr learned my addr 172.21.15.40:0/3465843704 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.452+0000 7fb8f2e00640 1 --2- 172.21.15.40:0/3465843704 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5602a1a910a0 0x5602a1aa97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.452+0000 7fb8ea400640 1 --2- 172.21.15.40:0/3465843704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602a1a91480 0x5602a1b3a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.452+0000 7fb8f3800640 1 -- 172.21.15.40:0/3465843704 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5602a1a910a0 msgr2=0x5602a1aa97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.452+0000 7fb8f3800640 1 --2- 172.21.15.40:0/3465843704 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5602a1a910a0 0x5602a1aa97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.452+0000 7fb8f3800640 1 -- 172.21.15.40:0/3465843704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602a1a91480 msgr2=0x5602a1b3a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.452+0000 7fb8f3800640 1 --2- 172.21.15.40:0/3465843704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602a1a91480 0x5602a1b3a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.452+0000 7fb8f3800640 1 -- 172.21.15.40:0/3465843704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602a1a64d10 con 0x5602a1aab290 2024-04-26T23:03:54.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.452+0000 7fb8ea400640 1 --2- 172.21.15.40:0/3465843704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602a1a91480 0x5602a1b3a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:54.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.452+0000 7fb8f3800640 1 --2- 172.21.15.40:0/3465843704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602a1aab290 0x5602a1b3a830 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec025b50 tx=0x7fb8ec051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d40f82c73c7447e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.453+0000 7fb8f1a00640 1 -- 172.21.15.40:0/3465843704 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec062070 con 0x5602a1aab290 2024-04-26T23:03:54.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.453+0000 7fb8f1a00640 1 -- 172.21.15.40:0/3465843704 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8ec054b50 con 0x5602a1aab290 2024-04-26T23:03:54.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.453+0000 7fb8f1a00640 1 -- 172.21.15.40:0/3465843704 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec0545c0 con 0x5602a1aab290 2024-04-26T23:03:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.453+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/3465843704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602a1aab290 msgr2=0x5602a1b3a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.453+0000 7fb8f5ce7f40 1 --2- 172.21.15.40:0/3465843704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602a1aab290 0x5602a1b3a830 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec025b50 tx=0x7fb8ec051000 comp rx=0 tx=0).stop 2024-04-26T23:03:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.454+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/3465843704 shutdown_connections 2024-04-26T23:03:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.454+0000 7fb8f5ce7f40 1 --2- 172.21.15.40:0/3465843704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602a1aab290 0x5602a1b3a830 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.454+0000 7fb8f5ce7f40 1 --2- 172.21.15.40:0/3465843704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602a1a91480 0x5602a1b3a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.454+0000 7fb8f5ce7f40 1 --2- 172.21.15.40:0/3465843704 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5602a1a910a0 0x5602a1aa97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.454+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/3465843704 >> 172.21.15.40:0/3465843704 conn(0x5602a1a8ec00 msgr2=0x5602a1a8e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.454+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/3465843704 shutdown_connections 2024-04-26T23:03:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.454+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/3465843704 wait complete. 2024-04-26T23:03:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.454+0000 7fb8f5ce7f40 1 Processor -- start 2024-04-26T23:03:54.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.455+0000 7fb8f5ce7f40 1 -- start start 2024-04-26T23:03:54.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.455+0000 7fb8f5ce7f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602a1a910a0 0x5602a1b18c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.455+0000 7fb8f5ce7f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5602a1a91480 0x5602a1a62f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.455+0000 7fb8f5ce7f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602a1aab290 0x5602a1b34660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.455+0000 7fb8f5ce7f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5602a1a8f080 con 0x5602a1aab290 2024-04-26T23:03:54.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.455+0000 7fb8f5ce7f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5602a1a64a30 con 0x5602a1a910a0 2024-04-26T23:03:54.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.455+0000 7fb8f5ce7f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5602a1a8f360 con 0x5602a1a91480 2024-04-26T23:03:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.456+0000 7fb8f3800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602a1aab290 0x5602a1b34660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.456+0000 7fb8f3800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602a1aab290 0x5602a1b34660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49268/0 (socket says 172.21.15.40:49268) 2024-04-26T23:03:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.456+0000 7fb8f3800640 1 -- 172.21.15.40:0/2897621414 learned_addr learned my addr 172.21.15.40:0/2897621414 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.456+0000 7fb8ea400640 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5602a1a91480 0x5602a1a62f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.456+0000 7fb8f2e00640 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602a1a910a0 0x5602a1b18c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.456+0000 7fb8f3800640 1 -- 172.21.15.40:0/2897621414 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5602a1a91480 msgr2=0x5602a1a62f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.456+0000 7fb8f3800640 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5602a1a91480 0x5602a1a62f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.456+0000 7fb8f3800640 1 -- 172.21.15.40:0/2897621414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602a1a910a0 msgr2=0x5602a1b18c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.456+0000 7fb8f3800640 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602a1a910a0 0x5602a1b18c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.456+0000 7fb8f3800640 1 -- 172.21.15.40:0/2897621414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602a1a63680 con 0x5602a1aab290 2024-04-26T23:03:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.457+0000 7fb8f3800640 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602a1aab290 0x5602a1b34660 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00a5a0 tx=0x7fb8ec002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.457+0000 7fb8ebe00640 1 -- 172.21.15.40:0/2897621414 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec070020 con 0x5602a1aab290 2024-04-26T23:03:54.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.457+0000 7fb8ebe00640 1 -- 172.21.15.40:0/2897621414 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8ec067450 con 0x5602a1aab290 2024-04-26T23:03:54.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.457+0000 7fb8ebe00640 1 -- 172.21.15.40:0/2897621414 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec062040 con 0x5602a1aab290 2024-04-26T23:03:54.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.457+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/2897621414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5602a1a63960 con 0x5602a1aab290 2024-04-26T23:03:54.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.458+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/2897621414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602a1b19360 con 0x5602a1aab290 2024-04-26T23:03:54.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.459+0000 7fb8ebe00640 1 -- 172.21.15.40:0/2897621414 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb8ec073020 con 0x5602a1aab290 2024-04-26T23:03:54.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.460+0000 7fb8ebe00640 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb8d006b320 0x7fb8d006d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.460+0000 7fb8ebe00640 1 -- 172.21.15.40:0/2897621414 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fb8ec0cf170 con 0x5602a1aab290 2024-04-26T23:03:54.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.461+0000 7fb8f5ce7f40 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5602a1b34ba0 0x5602a1a60bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.461+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/2897621414 --> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] -- osd_op(unknown.0.0:1 5.5 5:a367a3e8:::10000000004.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) -- 0x5602a1b1a3a0 con 0x5602a1b34ba0 2024-04-26T23:03:54.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.461+0000 7fb8ea400640 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5602a1b34ba0 0x5602a1a60bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.461+0000 7fb8f2e00640 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb8d006b320 0x7fb8d006d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.462+0000 7fb8ea400640 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5602a1b34ba0 0x5602a1a60bd0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.462+0000 7fb8ea400640 1 -- 172.21.15.40:0/2897621414 <== osd.0 v2:172.21.15.40:6800/2897145808 1 ==== osd_op_reply(1 10000000004.00000000 [getxattr (10) out=10b] v0'0 uv6 ondisk = 0) ==== 164+0+10 (crc 0 0 0) 0x7fb8dc00e590 con 0x5602a1b34ba0 2024-04-26T23:03:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.462+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/2897621414 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5602a1b34ba0 msgr2=0x5602a1a60bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.462+0000 7fb8f5ce7f40 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5602a1b34ba0 0x5602a1a60bd0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.462+0000 7fb8f2e00640 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb8d006b320 0x7fb8d006d7e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x5602a1b351d0 tx=0x7fb8e4007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.462+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/2897621414 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb8d006b320 msgr2=0x7fb8d006d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.462+0000 7fb8f5ce7f40 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb8d006b320 0x7fb8d006d7e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x5602a1b351d0 tx=0x7fb8e4007040 comp rx=0 tx=0).stop 2024-04-26T23:03:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.462+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/2897621414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602a1aab290 msgr2=0x5602a1b34660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.462+0000 7fb8f5ce7f40 1 --2- 172.21.15.40:0/2897621414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5602a1aab290 0x5602a1b34660 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00a5a0 tx=0x7fb8ec002fb0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.462+0000 7fb8f3800640 1 -- 172.21.15.40:0/2897621414 reap_dead start 2024-04-26T23:03:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.462+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/2897621414 shutdown_connections 2024-04-26T23:03:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.462+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/2897621414 >> 172.21.15.40:0/2897621414 conn(0x5602a1a8ec00 msgr2=0x5602a1aa98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.463+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/2897621414 shutdown_connections 2024-04-26T23:03:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.463+0000 7fb8f5ce7f40 1 -- 172.21.15.40:0/2897621414 wait complete. 2024-04-26T23:03:54.468 INFO:teuthology.orchestra.run.smithi040.stdout:tag123 2024-04-26T23:03:54.468 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000005.00000000 scrub_tag 2024-04-26T23:03:54.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.563+0000 7f0573c5ef40 1 Processor -- start 2024-04-26T23:03:54.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.563+0000 7f0573c5ef40 1 -- start start 2024-04-26T23:03:54.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.564+0000 7f0573c5ef40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5620a5c9fb60 0x5620a5ca5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.564+0000 7f0573c5ef40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620a5c9ff40 0x5620a5ca0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.564+0000 7f0573c5ef40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620a5ca08b0 0x5620a5bce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.564+0000 7f0573c5ef40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5620a5b77d20 con 0x5620a5c9ff40 2024-04-26T23:03:54.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.564+0000 7f0573c5ef40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5620a5b7bd60 con 0x5620a5ca08b0 2024-04-26T23:03:54.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.564+0000 7f0573c5ef40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5620a5ca85f0 con 0x5620a5c9fb60 2024-04-26T23:03:54.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.564+0000 7f0570400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620a5c9ff40 0x5620a5ca0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.564+0000 7f0570400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620a5c9ff40 0x5620a5ca0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49286/0 (socket says 172.21.15.40:49286) 2024-04-26T23:03:54.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.564+0000 7f0570400640 1 -- 172.21.15.40:0/2156542820 learned_addr learned my addr 172.21.15.40:0/2156542820 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.565+0000 7f0570400640 1 -- 172.21.15.40:0/2156542820 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5620a5c9fb60 msgr2=0x5620a5ca5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.565+0000 7f0570400640 1 --2- 172.21.15.40:0/2156542820 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5620a5c9fb60 0x5620a5ca5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.565+0000 7f0570400640 1 -- 172.21.15.40:0/2156542820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620a5ca08b0 msgr2=0x5620a5bce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.565+0000 7f0570400640 1 --2- 172.21.15.40:0/2156542820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620a5ca08b0 0x5620a5bce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.565+0000 7f0570400640 1 -- 172.21.15.40:0/2156542820 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620a5c016e0 con 0x5620a5c9ff40 2024-04-26T23:03:54.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.565+0000 7f0570400640 1 --2- 172.21.15.40:0/2156542820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620a5c9ff40 0x5620a5ca0370 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f055c0097e0 tx=0x7f055c005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65655415a952fef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.565+0000 7f056f000640 1 -- 172.21.15.40:0/2156542820 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c0078c0 con 0x5620a5c9ff40 2024-04-26T23:03:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.565+0000 7f056f000640 1 -- 172.21.15.40:0/2156542820 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f055c007a60 con 0x5620a5c9ff40 2024-04-26T23:03:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.565+0000 7f056f000640 1 -- 172.21.15.40:0/2156542820 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c007e10 con 0x5620a5c9ff40 2024-04-26T23:03:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.566+0000 7f0573c5ef40 1 -- 172.21.15.40:0/2156542820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620a5c9ff40 msgr2=0x5620a5ca0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.566+0000 7f0573c5ef40 1 --2- 172.21.15.40:0/2156542820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620a5c9ff40 0x5620a5ca0370 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f055c0097e0 tx=0x7f055c005a00 comp rx=0 tx=0).stop 2024-04-26T23:03:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.566+0000 7f0573c5ef40 1 -- 172.21.15.40:0/2156542820 shutdown_connections 2024-04-26T23:03:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.566+0000 7f0573c5ef40 1 --2- 172.21.15.40:0/2156542820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620a5ca08b0 0x5620a5bce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.566+0000 7f0573c5ef40 1 --2- 172.21.15.40:0/2156542820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620a5c9ff40 0x5620a5ca0370 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.566+0000 7f0573c5ef40 1 --2- 172.21.15.40:0/2156542820 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5620a5c9fb60 0x5620a5ca5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.566+0000 7f0573c5ef40 1 -- 172.21.15.40:0/2156542820 >> 172.21.15.40:0/2156542820 conn(0x5620a5c1ddf0 msgr2=0x5620a5c02270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.566+0000 7f0573c5ef40 1 -- 172.21.15.40:0/2156542820 shutdown_connections 2024-04-26T23:03:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.566+0000 7f0573c5ef40 1 -- 172.21.15.40:0/2156542820 wait complete. 2024-04-26T23:03:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.567+0000 7f0573c5ef40 1 Processor -- start 2024-04-26T23:03:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.567+0000 7f0573c5ef40 1 -- start start 2024-04-26T23:03:54.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.568+0000 7f0573c5ef40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620a5c9fb60 0x5620a5ca6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.568+0000 7f0573c5ef40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620a5c9ff40 0x5620a5ca3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.568+0000 7f0573c5ef40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5620a5ca08b0 0x5620a5c99640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.568+0000 7f0573c5ef40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5620a5ca0f40 con 0x5620a5c9ff40 2024-04-26T23:03:54.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.568+0000 7f0573c5ef40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5620a5caadf0 con 0x5620a5c9fb60 2024-04-26T23:03:54.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.568+0000 7f0573c5ef40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5620a5caaa80 con 0x5620a5ca08b0 2024-04-26T23:03:54.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.568+0000 7f0570400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620a5c9ff40 0x5620a5ca3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.568+0000 7f0570400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620a5c9ff40 0x5620a5ca3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49300/0 (socket says 172.21.15.40:49300) 2024-04-26T23:03:54.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.568+0000 7f0570400640 1 -- 172.21.15.40:0/3957980420 learned_addr learned my addr 172.21.15.40:0/3957980420 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.568+0000 7f0570e00640 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620a5c9fb60 0x5620a5ca6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.568+0000 7f0571800640 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5620a5ca08b0 0x5620a5c99640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.569+0000 7f0570400640 1 -- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5620a5ca08b0 msgr2=0x5620a5c99640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.569+0000 7f0570400640 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5620a5ca08b0 0x5620a5c99640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.569+0000 7f0570400640 1 -- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620a5c9fb60 msgr2=0x5620a5ca6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.569+0000 7f0570400640 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620a5c9fb60 0x5620a5ca6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.569+0000 7f0570400640 1 -- 172.21.15.40:0/3957980420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620a5ca2c10 con 0x5620a5c9ff40 2024-04-26T23:03:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.569+0000 7f0570400640 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620a5c9ff40 0x5620a5ca3c90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x5620a5bd1290 tx=0x7f055c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.569+0000 7f056dc00640 1 -- 172.21.15.40:0/3957980420 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c01e820 con 0x5620a5c9ff40 2024-04-26T23:03:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.569+0000 7f056dc00640 1 -- 172.21.15.40:0/3957980420 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f055c01e9c0 con 0x5620a5c9ff40 2024-04-26T23:03:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.569+0000 7f056dc00640 1 -- 172.21.15.40:0/3957980420 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c01ed90 con 0x5620a5c9ff40 2024-04-26T23:03:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.569+0000 7f0573c5ef40 1 -- 172.21.15.40:0/3957980420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5620a5ca2ef0 con 0x5620a5c9ff40 2024-04-26T23:03:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.570+0000 7f0573c5ef40 1 -- 172.21.15.40:0/3957980420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620a5ca34b0 con 0x5620a5c9ff40 2024-04-26T23:03:54.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.572+0000 7f056dc00640 1 -- 172.21.15.40:0/3957980420 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f055c013070 con 0x5620a5c9ff40 2024-04-26T23:03:54.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.572+0000 7f056dc00640 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f055006b320 0x7f055006d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.573+0000 7f056dc00640 1 -- 172.21.15.40:0/3957980420 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f055c08c430 con 0x5620a5c9ff40 2024-04-26T23:03:54.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.573+0000 7f0573c5ef40 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x5620a5ca7300 0x5620a5d429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.573+0000 7f0573c5ef40 1 -- 172.21.15.40:0/3957980420 --> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] -- osd_op(unknown.0.0:1 5.9 5:9258b67d:::10000000005.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) -- 0x5620a5c9c170 con 0x5620a5ca7300 2024-04-26T23:03:54.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.573+0000 7f0570e00640 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f055006b320 0x7f055006d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.574+0000 7f0571800640 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x5620a5ca7300 0x5620a5d429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.574+0000 7f0570e00640 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f055006b320 0x7f055006d7e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0560001370 tx=0x7f0560008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.575+0000 7f0571800640 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x5620a5ca7300 0x5620a5d429d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.576+0000 7f0571800640 1 -- 172.21.15.40:0/3957980420 <== osd.7 v2:172.21.15.103:6819/3769540391 1 ==== osd_op_reply(1 10000000005.00000000 [getxattr (10) out=10b] v0'0 uv6 ondisk = 0) ==== 164+0+10 (crc 0 0 0) 0x7f05680547c0 con 0x5620a5ca7300 2024-04-26T23:03:54.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.576+0000 7f0573c5ef40 1 -- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x5620a5ca7300 msgr2=0x5620a5d429d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.576+0000 7f0573c5ef40 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x5620a5ca7300 0x5620a5d429d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.576+0000 7f0573c5ef40 1 -- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f055006b320 msgr2=0x7f055006d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.576+0000 7f0573c5ef40 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f055006b320 0x7f055006d7e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0560001370 tx=0x7f0560008040 comp rx=0 tx=0).stop 2024-04-26T23:03:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.576+0000 7f0573c5ef40 1 -- 172.21.15.40:0/3957980420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620a5c9ff40 msgr2=0x5620a5ca3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.576+0000 7f0573c5ef40 1 --2- 172.21.15.40:0/3957980420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620a5c9ff40 0x5620a5ca3c90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x5620a5bd1290 tx=0x7f055c002fb0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.576+0000 7f0571800640 1 -- 172.21.15.40:0/3957980420 reap_dead start 2024-04-26T23:03:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.576+0000 7f0573c5ef40 1 -- 172.21.15.40:0/3957980420 shutdown_connections 2024-04-26T23:03:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.576+0000 7f0573c5ef40 1 -- 172.21.15.40:0/3957980420 >> 172.21.15.40:0/3957980420 conn(0x5620a5c1ddf0 msgr2=0x5620a5ca8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.576+0000 7f0573c5ef40 1 -- 172.21.15.40:0/3957980420 shutdown_connections 2024-04-26T23:03:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.576+0000 7f0573c5ef40 1 -- 172.21.15.40:0/3957980420 wait complete. 2024-04-26T23:03:54.581 INFO:teuthology.orchestra.run.smithi040.stdout:tag123 2024-04-26T23:03:54.581 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000006.00000000 scrub_tag 2024-04-26T23:03:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.685+0000 7f8ccf320f40 1 Processor -- start 2024-04-26T23:03:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.685+0000 7f8ccf320f40 1 -- start start 2024-04-26T23:03:54.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.686+0000 7f8ccf320f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640fdbd9080 0x5640fdca6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.686+0000 7f8ccf320f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640fdc07530 0x5640fdc96620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.686+0000 7f8ccf320f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5640fdcaa280 0x5640fdca8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.686+0000 7f8ccf320f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5640fdcacec0 con 0x5640fdc07530 2024-04-26T23:03:54.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.686+0000 7f8ccf320f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5640fdb7ed20 con 0x5640fdbd9080 2024-04-26T23:03:54.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.686+0000 7f8ccf320f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5640fdcad420 con 0x5640fdcaa280 2024-04-26T23:03:54.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.686+0000 7f8cc3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640fdc07530 0x5640fdc96620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.687+0000 7f8cc3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640fdc07530 0x5640fdc96620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49304/0 (socket says 172.21.15.40:49304) 2024-04-26T23:03:54.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.687+0000 7f8cc3e00640 1 -- 172.21.15.40:0/2091350347 learned_addr learned my addr 172.21.15.40:0/2091350347 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.687+0000 7f8ccbc00640 1 --2- 172.21.15.40:0/2091350347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640fdbd9080 0x5640fdca6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.687+0000 7f8ccc600640 1 --2- 172.21.15.40:0/2091350347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5640fdcaa280 0x5640fdca8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.687+0000 7f8cc3e00640 1 -- 172.21.15.40:0/2091350347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5640fdcaa280 msgr2=0x5640fdca8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.687+0000 7f8cc3e00640 1 --2- 172.21.15.40:0/2091350347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5640fdcaa280 0x5640fdca8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.687+0000 7f8cc3e00640 1 -- 172.21.15.40:0/2091350347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640fdbd9080 msgr2=0x5640fdca6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.687+0000 7f8cc3e00640 1 --2- 172.21.15.40:0/2091350347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640fdbd9080 0x5640fdca6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.687+0000 7f8cc3e00640 1 -- 172.21.15.40:0/2091350347 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640fdca42c0 con 0x5640fdc07530 2024-04-26T23:03:54.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.687+0000 7f8cc3e00640 1 --2- 172.21.15.40:0/2091350347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640fdc07530 0x5640fdc96620 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400b2e0 tx=0x7f8cb400b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11688fd209d25c36 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.688+0000 7f8cca800640 1 -- 172.21.15.40:0/2091350347 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4008070 con 0x5640fdc07530 2024-04-26T23:03:54.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.688+0000 7f8cca800640 1 -- 172.21.15.40:0/2091350347 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cb400d500 con 0x5640fdc07530 2024-04-26T23:03:54.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.688+0000 7f8cca800640 1 -- 172.21.15.40:0/2091350347 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb400d8f0 con 0x5640fdc07530 2024-04-26T23:03:54.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.688+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2091350347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640fdc07530 msgr2=0x5640fdc96620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.688+0000 7f8ccf320f40 1 --2- 172.21.15.40:0/2091350347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640fdc07530 0x5640fdc96620 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400b2e0 tx=0x7f8cb400b7a0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.688+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2091350347 shutdown_connections 2024-04-26T23:03:54.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.688+0000 7f8ccf320f40 1 --2- 172.21.15.40:0/2091350347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5640fdcaa280 0x5640fdca8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.688+0000 7f8ccf320f40 1 --2- 172.21.15.40:0/2091350347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640fdc07530 0x5640fdc96620 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.688+0000 7f8ccf320f40 1 --2- 172.21.15.40:0/2091350347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640fdbd9080 0x5640fdca6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.688+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2091350347 >> 172.21.15.40:0/2091350347 conn(0x5640fdc04c00 msgr2=0x5640fdcac070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.688+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2091350347 shutdown_connections 2024-04-26T23:03:54.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.688+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2091350347 wait complete. 2024-04-26T23:03:54.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.689+0000 7f8ccf320f40 1 Processor -- start 2024-04-26T23:03:54.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.689+0000 7f8ccf320f40 1 -- start start 2024-04-26T23:03:54.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.690+0000 7f8ccf320f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640fdbd9080 0x5640fdc91970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.690+0000 7f8ccf320f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5640fdc07530 0x5640fdc1ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.690+0000 7f8ccf320f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640fdcaa280 0x5640fdc1cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.690+0000 7f8ccf320f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5640fdcaa700 con 0x5640fdcaa280 2024-04-26T23:03:54.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.690+0000 7f8ccf320f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5640fdc96b60 con 0x5640fdbd9080 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.690+0000 7f8ccf320f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5640fdca98d0 con 0x5640fdc07530 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.690+0000 7f8ccc600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640fdcaa280 0x5640fdc1cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.691+0000 7f8ccc600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640fdcaa280 0x5640fdc1cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49310/0 (socket says 172.21.15.40:49310) 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.691+0000 7f8ccc600640 1 -- 172.21.15.40:0/2173305475 learned_addr learned my addr 172.21.15.40:0/2173305475 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.691+0000 7f8cc3e00640 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5640fdc07530 0x5640fdc1ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.691+0000 7f8ccbc00640 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640fdbd9080 0x5640fdc91970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.691+0000 7f8ccc600640 1 -- 172.21.15.40:0/2173305475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5640fdc07530 msgr2=0x5640fdc1ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.691+0000 7f8ccc600640 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5640fdc07530 0x5640fdc1ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.691+0000 7f8ccc600640 1 -- 172.21.15.40:0/2173305475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640fdbd9080 msgr2=0x5640fdc91970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.691+0000 7f8ccc600640 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640fdbd9080 0x5640fdc91970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.691+0000 7f8ccc600640 1 -- 172.21.15.40:0/2173305475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640fdc959d0 con 0x5640fdcaa280 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.691+0000 7f8ccc600640 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640fdcaa280 0x5640fdc1cfa0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400c270 tx=0x7f8cc400c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.692+0000 7f8cc9400640 1 -- 172.21.15.40:0/2173305475 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4017020 con 0x5640fdcaa280 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.692+0000 7f8cc9400640 1 -- 172.21.15.40:0/2173305475 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cc4013670 con 0x5640fdcaa280 2024-04-26T23:03:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.692+0000 7f8cc9400640 1 -- 172.21.15.40:0/2173305475 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4013a40 con 0x5640fdcaa280 2024-04-26T23:03:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.692+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2173305475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5640fdca72d0 con 0x5640fdcaa280 2024-04-26T23:03:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.692+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2173305475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640fdc20550 con 0x5640fdcaa280 2024-04-26T23:03:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.694+0000 7f8cc9400640 1 -- 172.21.15.40:0/2173305475 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8cc401a020 con 0x5640fdcaa280 2024-04-26T23:03:54.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.695+0000 7f8cc9400640 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8ca806b300 0x7f8ca806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.695+0000 7f8cc9400640 1 -- 172.21.15.40:0/2173305475 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f8cc408ad00 con 0x5640fdcaa280 2024-04-26T23:03:54.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.695+0000 7f8ccf320f40 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5640fdbdb080 0x5640fdc942a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.695+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2173305475 --> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] -- osd_op(unknown.0.0:1 5.5 5:a5feb6f3:::10000000006.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) -- 0x5640fdc96720 con 0x5640fdbdb080 2024-04-26T23:03:54.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.696+0000 7f8ccbc00640 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8ca806b300 0x7f8ca806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.696+0000 7f8cc3e00640 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5640fdbdb080 0x5640fdc942a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.696+0000 7f8cc3e00640 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5640fdbdb080 0x5640fdc942a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.697+0000 7f8cc3e00640 1 -- 172.21.15.40:0/2173305475 <== osd.0 v2:172.21.15.40:6800/2897145808 1 ==== osd_op_reply(1 10000000006.00000000 [getxattr (10) out=10b] v0'0 uv7 ondisk = 0) ==== 164+0+10 (crc 0 0 0) 0x7f8cb400d830 con 0x5640fdbdb080 2024-04-26T23:03:54.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.697+0000 7f8ccbc00640 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8ca806b300 0x7f8ca806d7c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc056000 tx=0x7f8cbc050000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.697+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2173305475 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5640fdbdb080 msgr2=0x5640fdc942a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.697+0000 7f8ccf320f40 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5640fdbdb080 0x5640fdc942a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.697+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2173305475 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8ca806b300 msgr2=0x7f8ca806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.697+0000 7f8ccf320f40 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8ca806b300 0x7f8ca806d7c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc056000 tx=0x7f8cbc050000 comp rx=0 tx=0).stop 2024-04-26T23:03:54.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.697+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2173305475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640fdcaa280 msgr2=0x5640fdc1cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.697+0000 7f8ccf320f40 1 --2- 172.21.15.40:0/2173305475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5640fdcaa280 0x5640fdc1cfa0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400c270 tx=0x7f8cc400c730 comp rx=0 tx=0).stop 2024-04-26T23:03:54.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.697+0000 7f8ccc600640 1 -- 172.21.15.40:0/2173305475 reap_dead start 2024-04-26T23:03:54.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.697+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2173305475 shutdown_connections 2024-04-26T23:03:54.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.697+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2173305475 >> 172.21.15.40:0/2173305475 conn(0x5640fdc04c00 msgr2=0x5640fdcaafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.697+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2173305475 shutdown_connections 2024-04-26T23:03:54.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.697+0000 7f8ccf320f40 1 -- 172.21.15.40:0/2173305475 wait complete. 2024-04-26T23:03:54.702 INFO:teuthology.orchestra.run.smithi040.stdout:tag123 2024-04-26T23:03:54.702 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000007.00000000 scrub_tag 2024-04-26T23:03:54.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.789+0000 7fe20d60cf40 1 Processor -- start 2024-04-26T23:03:54.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.789+0000 7fe20d60cf40 1 -- start start 2024-04-26T23:03:54.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.790+0000 7fe20d60cf40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636c4663080 0x5636c4734940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.790+0000 7fe20d60cf40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636c4691360 0x5636c472dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.790+0000 7fe20d60cf40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636c4734280 0x5636c46a9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.790+0000 7fe20d60cf40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5636c46917d0 con 0x5636c4734280 2024-04-26T23:03:54.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.790+0000 7fe20d60cf40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5636c4608d20 con 0x5636c4691360 2024-04-26T23:03:54.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.790+0000 7fe20d60cf40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5636c46aca60 con 0x5636c4663080 2024-04-26T23:03:54.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.790+0000 7fe20a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636c4734280 0x5636c46a9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.790+0000 7fe20a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636c4734280 0x5636c46a9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49318/0 (socket says 172.21.15.40:49318) 2024-04-26T23:03:54.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.790+0000 7fe20a600640 1 -- 172.21.15.40:0/659359439 learned_addr learned my addr 172.21.15.40:0/659359439 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.790+0000 7fe209200640 1 --2- 172.21.15.40:0/659359439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636c4691360 0x5636c472dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.790+0000 7fe209c00640 1 --2- 172.21.15.40:0/659359439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636c4663080 0x5636c4734940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.791+0000 7fe20a600640 1 -- 172.21.15.40:0/659359439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636c4663080 msgr2=0x5636c4734940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.791+0000 7fe20a600640 1 --2- 172.21.15.40:0/659359439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636c4663080 0x5636c4734940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.791+0000 7fe20a600640 1 -- 172.21.15.40:0/659359439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636c4691360 msgr2=0x5636c472dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.791+0000 7fe20a600640 1 --2- 172.21.15.40:0/659359439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636c4691360 0x5636c472dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.791+0000 7fe20a600640 1 -- 172.21.15.40:0/659359439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636c468d340 con 0x5636c4734280 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.791+0000 7fe209200640 1 --2- 172.21.15.40:0/659359439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636c4691360 0x5636c472dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.791+0000 7fe209c00640 1 --2- 172.21.15.40:0/659359439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636c4663080 0x5636c4734940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.791+0000 7fe20a600640 1 --2- 172.21.15.40:0/659359439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636c4734280 0x5636c46a9580 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc0098a0 tx=0x7fe1fc005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9fa01f1b14631e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.791+0000 7fe203400640 1 -- 172.21.15.40:0/659359439 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc008930 con 0x5636c4734280 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.791+0000 7fe203400640 1 -- 172.21.15.40:0/659359439 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1fc018070 con 0x5636c4734280 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.791+0000 7fe203400640 1 -- 172.21.15.40:0/659359439 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc008d10 con 0x5636c4734280 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.792+0000 7fe20d60cf40 1 -- 172.21.15.40:0/659359439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636c4734280 msgr2=0x5636c46a9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.792+0000 7fe20d60cf40 1 --2- 172.21.15.40:0/659359439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636c4734280 0x5636c46a9580 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc0098a0 tx=0x7fe1fc005a00 comp rx=0 tx=0).stop 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.792+0000 7fe20d60cf40 1 -- 172.21.15.40:0/659359439 shutdown_connections 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.792+0000 7fe20d60cf40 1 --2- 172.21.15.40:0/659359439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636c4734280 0x5636c46a9580 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.792+0000 7fe20d60cf40 1 --2- 172.21.15.40:0/659359439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636c4691360 0x5636c472dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.792+0000 7fe20d60cf40 1 --2- 172.21.15.40:0/659359439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636c4663080 0x5636c4734940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.792+0000 7fe20d60cf40 1 -- 172.21.15.40:0/659359439 >> 172.21.15.40:0/659359439 conn(0x5636c468ec00 msgr2=0x5636c46ac350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.792+0000 7fe20d60cf40 1 -- 172.21.15.40:0/659359439 shutdown_connections 2024-04-26T23:03:54.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.792+0000 7fe20d60cf40 1 -- 172.21.15.40:0/659359439 wait complete. 2024-04-26T23:03:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.793+0000 7fe20d60cf40 1 Processor -- start 2024-04-26T23:03:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.793+0000 7fe20d60cf40 1 -- start start 2024-04-26T23:03:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.794+0000 7fe20d60cf40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636c4663080 0x5636c4660210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.794+0000 7fe20d60cf40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636c4691360 0x5636c4729c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.794+0000 7fe20d60cf40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636c4734280 0x5636c472a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.794+0000 7fe20d60cf40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5636c4732d10 con 0x5636c4663080 2024-04-26T23:03:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.794+0000 7fe20d60cf40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5636c4732ff0 con 0x5636c4734280 2024-04-26T23:03:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.794+0000 7fe20d60cf40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5636c47351e0 con 0x5636c4691360 2024-04-26T23:03:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.794+0000 7fe209c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636c4663080 0x5636c4660210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.794+0000 7fe209c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636c4663080 0x5636c4660210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49334/0 (socket says 172.21.15.40:49334) 2024-04-26T23:03:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.794+0000 7fe209c00640 1 -- 172.21.15.40:0/1796844062 learned_addr learned my addr 172.21.15.40:0/1796844062 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.794+0000 7fe209200640 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636c4691360 0x5636c4729c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.794+0000 7fe20a600640 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636c4734280 0x5636c472a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.794+0000 7fe209c00640 1 -- 172.21.15.40:0/1796844062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636c4691360 msgr2=0x5636c4729c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.795+0000 7fe209c00640 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636c4691360 0x5636c4729c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.795+0000 7fe209c00640 1 -- 172.21.15.40:0/1796844062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636c4734280 msgr2=0x5636c472a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.795+0000 7fe209c00640 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636c4734280 0x5636c472a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.795+0000 7fe209c00640 1 -- 172.21.15.40:0/1796844062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636c468f380 con 0x5636c4663080 2024-04-26T23:03:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.795+0000 7fe20a600640 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636c4734280 0x5636c472a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.795+0000 7fe209c00640 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636c4663080 0x5636c4660210 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe204006f90 tx=0x7fe20400c5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.795+0000 7fe202000640 1 -- 172.21.15.40:0/1796844062 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe204011800 con 0x5636c4663080 2024-04-26T23:03:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.795+0000 7fe202000640 1 -- 172.21.15.40:0/1796844062 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2040119a0 con 0x5636c4663080 2024-04-26T23:03:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.795+0000 7fe202000640 1 -- 172.21.15.40:0/1796844062 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe204011d90 con 0x5636c4663080 2024-04-26T23:03:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.795+0000 7fe20d60cf40 1 -- 172.21.15.40:0/1796844062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5636c472fd90 con 0x5636c4663080 2024-04-26T23:03:54.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.795+0000 7fe20d60cf40 1 -- 172.21.15.40:0/1796844062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636c47318b0 con 0x5636c4663080 2024-04-26T23:03:54.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.797+0000 7fe202000640 1 -- 172.21.15.40:0/1796844062 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe20401a400 con 0x5636c4663080 2024-04-26T23:03:54.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.798+0000 7fe202000640 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1e806b2d0 0x7fe1e806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.798+0000 7fe202000640 1 -- 172.21.15.40:0/1796844062 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fe20408afa0 con 0x5636c4663080 2024-04-26T23:03:54.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.799+0000 7fe20d60cf40 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5636c472d080 0x5636c4730590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:54.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.799+0000 7fe20d60cf40 1 -- 172.21.15.40:0/1796844062 --> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] -- osd_op(unknown.0.0:1 5.1d 5:bbf5850a:::10000000007.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) -- 0x5636c4660750 con 0x5636c472d080 2024-04-26T23:03:54.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.799+0000 7fe20a600640 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5636c472d080 0x5636c4730590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.799+0000 7fe209200640 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1e806b2d0 0x7fe1e806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:54.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.800+0000 7fe20a600640 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5636c472d080 0x5636c4730590 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.800+0000 7fe20a600640 1 -- 172.21.15.40:0/1796844062 <== osd.1 v2:172.21.15.40:6815/2323014490 1 ==== osd_op_reply(1 10000000007.00000000 [getxattr (10) out=10b] v0'0 uv7 ondisk = 0) ==== 164+0+10 (crc 0 0 0) 0x7fe1fc0088d0 con 0x5636c472d080 2024-04-26T23:03:54.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.800+0000 7fe209200640 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1e806b2d0 0x7fe1e806d790 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4057000 tx=0x7fe1f4052000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:54.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.800+0000 7fe20d60cf40 1 -- 172.21.15.40:0/1796844062 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5636c472d080 msgr2=0x5636c4730590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.800+0000 7fe20d60cf40 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5636c472d080 0x5636c4730590 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.800+0000 7fe20d60cf40 1 -- 172.21.15.40:0/1796844062 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1e806b2d0 msgr2=0x7fe1e806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.800+0000 7fe20d60cf40 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1e806b2d0 0x7fe1e806d790 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4057000 tx=0x7fe1f4052000 comp rx=0 tx=0).stop 2024-04-26T23:03:54.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.800+0000 7fe20d60cf40 1 -- 172.21.15.40:0/1796844062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636c4663080 msgr2=0x5636c4660210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:54.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.800+0000 7fe20d60cf40 1 --2- 172.21.15.40:0/1796844062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636c4663080 0x5636c4660210 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe204006f90 tx=0x7fe20400c5e0 comp rx=0 tx=0).stop 2024-04-26T23:03:54.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.800+0000 7fe20a600640 1 -- 172.21.15.40:0/1796844062 reap_dead start 2024-04-26T23:03:54.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.801+0000 7fe20d60cf40 1 -- 172.21.15.40:0/1796844062 shutdown_connections 2024-04-26T23:03:54.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.801+0000 7fe20d60cf40 1 -- 172.21.15.40:0/1796844062 >> 172.21.15.40:0/1796844062 conn(0x5636c468ec00 msgr2=0x5636c4734940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:54.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.801+0000 7fe20d60cf40 1 -- 172.21.15.40:0/1796844062 shutdown_connections 2024-04-26T23:03:54.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:54.801+0000 7fe20d60cf40 1 -- 172.21.15.40:0/1796844062 wait complete. 2024-04-26T23:03:54.805 INFO:teuthology.orchestra.run.smithi040.stdout:tag123 2024-04-26T23:03:54.805 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-26T23:03:54.806 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:03:54.893 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:03:55.015 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-26T23:03:55.017 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:03:55.018 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:03:55.018 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:03:55.079 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:03:55.079 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:03:55.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:03:55.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.353+0000 7fc572400640 1 Processor -- start 2024-04-26T23:03:55.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.354+0000 7fc572400640 1 -- start start 2024-04-26T23:03:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.355+0000 7fc572400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc56c0cbb50 0x7fc56c0af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.355+0000 7fc572400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc56c072c00 con 0x7fc56c15f380 2024-04-26T23:03:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.355+0000 7fc572400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc56c072ee0 con 0x7fc56c073b70 2024-04-26T23:03:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.355+0000 7fc572400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc56c0729b0 con 0x7fc56c0cbb50 2024-04-26T23:03:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.356+0000 7fc56b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc56c15f380 0x7fc56c0cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45680/0 (socket says 172.21.15.40:45680) 2024-04-26T23:03:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.356+0000 7fc56b400640 1 -- 172.21.15.40:0/3453373368 learned_addr learned my addr 172.21.15.40:0/3453373368 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.356+0000 7fc56be00640 1 --2- 172.21.15.40:0/3453373368 >> v2:172.21.15.40:3300/0 conn(0x7fc56c0cbb50 0x7fc56c0af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.356+0000 7fc56be00640 1 -- 172.21.15.40:0/3453373368 >> v1:172.21.15.103:6789/0 conn(0x7fc56c073b70 legacy=0x7fc56c0aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.356+0000 7fc56be00640 1 -- 172.21.15.40:0/3453373368 >> v1:172.21.15.40:6789/0 conn(0x7fc56c15f380 legacy=0x7fc56c0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.356+0000 7fc56be00640 1 -- 172.21.15.40:0/3453373368 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56c0731c0 con 0x7fc56c0cbb50 2024-04-26T23:03:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.357+0000 7fc56be00640 1 --2- 172.21.15.40:0/3453373368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc56c0cbb50 0x7fc56c0af3c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc564009d90 tx=0x7fc56402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb81bff5790f61 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.357+0000 7fc56a000640 1 -- 172.21.15.40:0/3453373368 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564007740 con 0x7fc56c0cbb50 2024-04-26T23:03:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.358+0000 7fc56a000640 1 -- 172.21.15.40:0/3453373368 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5640078e0 con 0x7fc56c0cbb50 2024-04-26T23:03:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.358+0000 7fc56a000640 1 -- 172.21.15.40:0/3453373368 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564007cb0 con 0x7fc56c0cbb50 2024-04-26T23:03:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.358+0000 7fc572400640 1 -- 172.21.15.40:0/3453373368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc56c0cbb50 msgr2=0x7fc56c0af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.358+0000 7fc572400640 1 --2- 172.21.15.40:0/3453373368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc56c0cbb50 0x7fc56c0af3c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc564009d90 tx=0x7fc56402a6a0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.358+0000 7fc572400640 1 -- 172.21.15.40:0/3453373368 shutdown_connections 2024-04-26T23:03:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.358+0000 7fc572400640 1 --2- 172.21.15.40:0/3453373368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc56c0cbb50 0x7fc56c0af3c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.358+0000 7fc572400640 1 -- 172.21.15.40:0/3453373368 >> 172.21.15.40:0/3453373368 conn(0x7fc56c0d0be0 msgr2=0x7fc56c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.358+0000 7fc572400640 1 -- 172.21.15.40:0/3453373368 shutdown_connections 2024-04-26T23:03:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.358+0000 7fc572400640 1 -- 172.21.15.40:0/3453373368 wait complete. 2024-04-26T23:03:55.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.360+0000 7fc572400640 1 Processor -- start 2024-04-26T23:03:55.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.360+0000 7fc572400640 1 -- start start 2024-04-26T23:03:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.361+0000 7fc572400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc56c0cbb50 0x7fc56c14f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.361+0000 7fc572400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc56c15f380 0x7fc56c0c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.361+0000 7fc572400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc56c0c70b0 0x7fc56c0c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.361+0000 7fc572400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c073620 con 0x7fc56c0cbb50 2024-04-26T23:03:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.361+0000 7fc572400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c0d08f0 con 0x7fc56c0c70b0 2024-04-26T23:03:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.361+0000 7fc572400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc56c0737a0 con 0x7fc56c15f380 2024-04-26T23:03:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.362+0000 7fc56b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc56c0cbb50 0x7fc56c14f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.362+0000 7fc56b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc56c0cbb50 0x7fc56c14f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36542/0 (socket says 172.21.15.40:36542) 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.362+0000 7fc56b400640 1 -- 172.21.15.40:0/1737126461 learned_addr learned my addr 172.21.15.40:0/1737126461 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.362+0000 7fc56be00640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc56c0c70b0 0x7fc56c0c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.362+0000 7fc56aa00640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc56c15f380 0x7fc56c0c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.362+0000 7fc56b400640 1 -- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc56c15f380 msgr2=0x7fc56c0c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.362+0000 7fc56b400640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc56c15f380 0x7fc56c0c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.362+0000 7fc56b400640 1 -- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc56c0c70b0 msgr2=0x7fc56c0c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.362+0000 7fc56b400640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc56c0c70b0 0x7fc56c0c7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.362+0000 7fc56b400640 1 -- 172.21.15.40:0/1737126461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56c0c88c0 con 0x7fc56c0cbb50 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.362+0000 7fc56aa00640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc56c15f380 0x7fc56c0c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.363+0000 7fc56b400640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc56c0cbb50 0x7fc56c14f0f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00c430 tx=0x7fc55c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.363+0000 7fc563400640 1 -- 172.21.15.40:0/1737126461 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c017070 con 0x7fc56c0cbb50 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.363+0000 7fc563400640 1 -- 172.21.15.40:0/1737126461 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc55c005ce0 con 0x7fc56c0cbb50 2024-04-26T23:03:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.363+0000 7fc563400640 1 -- 172.21.15.40:0/1737126461 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c0072f0 con 0x7fc56c0cbb50 2024-04-26T23:03:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.363+0000 7fc572400640 1 -- 172.21.15.40:0/1737126461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56c0c8ba0 con 0x7fc56c0cbb50 2024-04-26T23:03:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.363+0000 7fc572400640 1 -- 172.21.15.40:0/1737126461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc56c0c9160 con 0x7fc56c0cbb50 2024-04-26T23:03:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.364+0000 7fc572400640 1 -- 172.21.15.40:0/1737126461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc56c0c74c0 con 0x7fc56c0cbb50 2024-04-26T23:03:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.365+0000 7fc563400640 1 -- 172.21.15.40:0/1737126461 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc55c01a070 con 0x7fc56c0cbb50 2024-04-26T23:03:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.366+0000 7fc563400640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc54806c8f0 0x7fc54806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.366+0000 7fc563400640 1 -- 172.21.15.40:0/1737126461 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fc55c08b180 con 0x7fc56c0cbb50 2024-04-26T23:03:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.367+0000 7fc56aa00640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc54806c8f0 0x7fc54806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:55.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.368+0000 7fc56aa00640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc54806c8f0 0x7fc54806edb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc56c0af720 tx=0x7fc554008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:55.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.371+0000 7fc563400640 1 -- 172.21.15.40:0/1737126461 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc56c0c74c0 con 0x7fc56c0cbb50 2024-04-26T23:03:55.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.578+0000 7fc572400640 1 -- 172.21.15.40:0/1737126461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc56c0694f0 con 0x7fc56c0cbb50 2024-04-26T23:03:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.580+0000 7fc563400640 1 -- 172.21.15.40:0/1737126461 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+16955 (secure 0 0 0) 0x7fc55c059360 con 0x7fc56c0cbb50 2024-04-26T23:03:55.580 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:03:55.581 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":41,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:03:47.741302+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:02:45.864120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-26T23:02:46.869290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:03:55.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.583+0000 7fc572400640 1 -- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc54806c8f0 msgr2=0x7fc54806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.583+0000 7fc572400640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc54806c8f0 0x7fc54806edb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc56c0af720 tx=0x7fc554008040 comp rx=0 tx=0).stop 2024-04-26T23:03:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.583+0000 7fc572400640 1 -- 172.21.15.40:0/1737126461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc56c0cbb50 msgr2=0x7fc56c14f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.583+0000 7fc572400640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc56c0cbb50 0x7fc56c14f0f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00c430 tx=0x7fc55c00c8f0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.583+0000 7fc572400640 1 -- 172.21.15.40:0/1737126461 shutdown_connections 2024-04-26T23:03:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.583+0000 7fc572400640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc54806c8f0 0x7fc54806edb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.583+0000 7fc572400640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc56c0c70b0 0x7fc56c0c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.583+0000 7fc572400640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc56c15f380 0x7fc56c0c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.583+0000 7fc572400640 1 --2- 172.21.15.40:0/1737126461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc56c0cbb50 0x7fc56c14f0f0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.583+0000 7fc572400640 1 -- 172.21.15.40:0/1737126461 >> 172.21.15.40:0/1737126461 conn(0x7fc56c0d0be0 msgr2=0x7fc56c0aecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.583+0000 7fc572400640 1 -- 172.21.15.40:0/1737126461 shutdown_connections 2024-04-26T23:03:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.583+0000 7fc572400640 1 -- 172.21.15.40:0/1737126461 wait complete. 2024-04-26T23:03:55.602 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:02:45.864120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:02:46.869290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:03:55.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:03:55.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.819+0000 7fbd23e00640 1 Processor -- start 2024-04-26T23:03:55.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.819+0000 7fbd23e00640 1 -- start start 2024-04-26T23:03:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.820+0000 7fbd23e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbd2415bc70 0x7fbd240cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.820+0000 7fbd23e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbd24071ef0 0x7fbd24072990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.820+0000 7fbd23e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd2406dde0 con 0x7fbd24072440 2024-04-26T23:03:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.820+0000 7fbd23e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fbd2406e3a0 con 0x7fbd24071ef0 2024-04-26T23:03:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.820+0000 7fbd23e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbd240b4840 con 0x7fbd2415bc70 2024-04-26T23:03:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.821+0000 7fbd22000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbd24071ef0 0x7fbd24072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.821+0000 7fbd23400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbd24072440 0x7fbd2415f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45696/0 (socket says 172.21.15.40:45696) 2024-04-26T23:03:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.821+0000 7fbd23400640 1 -- 172.21.15.40:0/2929126031 learned_addr learned my addr 172.21.15.40:0/2929126031 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.821+0000 7fbd22a00640 1 --2- 172.21.15.40:0/2929126031 >> v2:172.21.15.103:3300/0 conn(0x7fbd2415bc70 0x7fbd240cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.821+0000 7fbd22000640 1 -- 172.21.15.40:0/2929126031 >> v2:172.21.15.103:3300/0 conn(0x7fbd2415bc70 msgr2=0x7fbd240cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.821+0000 7fbd22000640 1 --2- 172.21.15.40:0/2929126031 >> v2:172.21.15.103:3300/0 conn(0x7fbd2415bc70 0x7fbd240cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.821+0000 7fbd22000640 1 -- 172.21.15.40:0/2929126031 >> v1:172.21.15.40:6789/0 conn(0x7fbd24072440 legacy=0x7fbd2415f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.821+0000 7fbd22000640 1 -- 172.21.15.40:0/2929126031 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd2406e0c0 con 0x7fbd24071ef0 2024-04-26T23:03:55.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.823+0000 7fbd22000640 1 --2- 172.21.15.40:0/2929126031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd24071ef0 0x7fbd24072990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fbd240b0640 tx=0x7fbd0c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799a4b191b0986da server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:55.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.823+0000 7fbd21600640 1 -- 172.21.15.40:0/2929126031 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c037070 con 0x7fbd24071ef0 2024-04-26T23:03:55.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.823+0000 7fbd21600640 1 -- 172.21.15.40:0/2929126031 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd0c032440 con 0x7fbd24071ef0 2024-04-26T23:03:55.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.823+0000 7fbd21600640 1 -- 172.21.15.40:0/2929126031 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c032810 con 0x7fbd24071ef0 2024-04-26T23:03:55.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.823+0000 7fbd23e00640 1 -- 172.21.15.40:0/2929126031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd24071ef0 msgr2=0x7fbd24072990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:55.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.823+0000 7fbd23e00640 1 --2- 172.21.15.40:0/2929126031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd24071ef0 0x7fbd24072990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fbd240b0640 tx=0x7fbd0c02a1a0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.824+0000 7fbd23e00640 1 -- 172.21.15.40:0/2929126031 shutdown_connections 2024-04-26T23:03:55.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.824+0000 7fbd23e00640 1 --2- 172.21.15.40:0/2929126031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd24071ef0 0x7fbd24072990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.824+0000 7fbd23e00640 1 --2- 172.21.15.40:0/2929126031 >> v2:172.21.15.103:3300/0 conn(0x7fbd2415bc70 0x7fbd240cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.824+0000 7fbd23e00640 1 -- 172.21.15.40:0/2929126031 >> 172.21.15.40:0/2929126031 conn(0x7fbd240b2720 msgr2=0x7fbd240b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:55.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.824+0000 7fbd23e00640 1 -- 172.21.15.40:0/2929126031 shutdown_connections 2024-04-26T23:03:55.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.824+0000 7fbd23e00640 1 -- 172.21.15.40:0/2929126031 wait complete. 2024-04-26T23:03:55.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.825+0000 7fbd23e00640 1 Processor -- start 2024-04-26T23:03:55.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.826+0000 7fbd23e00640 1 -- start start 2024-04-26T23:03:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd23e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd24071ef0 0x7fbd241f5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd23e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd24072440 0x7fbd2414d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd23e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2415bc70 0x7fbd2414eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd23e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2415c050 con 0x7fbd24071ef0 2024-04-26T23:03:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd23e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2406fad0 con 0x7fbd2415bc70 2024-04-26T23:03:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd23e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbd2415c1d0 con 0x7fbd24072440 2024-04-26T23:03:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd22a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd24071ef0 0x7fbd241f5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd22a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd24071ef0 0x7fbd241f5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36566/0 (socket says 172.21.15.40:36566) 2024-04-26T23:03:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd22a00640 1 -- 172.21.15.40:0/1916421367 learned_addr learned my addr 172.21.15.40:0/1916421367 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd22000640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd24072440 0x7fbd2414d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd23400640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2415bc70 0x7fbd2414eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd22a00640 1 -- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd24072440 msgr2=0x7fbd2414d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd22a00640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd24072440 0x7fbd2414d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd22a00640 1 -- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2415bc70 msgr2=0x7fbd2414eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.827+0000 7fbd22a00640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2415bc70 0x7fbd2414eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.828+0000 7fbd22a00640 1 -- 172.21.15.40:0/1916421367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd241f5e80 con 0x7fbd24071ef0 2024-04-26T23:03:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.828+0000 7fbd22000640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd24072440 0x7fbd2414d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.828+0000 7fbd22a00640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd24071ef0 0x7fbd241f5940 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fbd1400d830 tx=0x7fbd1400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.828+0000 7fbd1b400640 1 -- 172.21.15.40:0/1916421367 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1400b6c0 con 0x7fbd24071ef0 2024-04-26T23:03:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.828+0000 7fbd1b400640 1 -- 172.21.15.40:0/1916421367 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd14019070 con 0x7fbd24071ef0 2024-04-26T23:03:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.828+0000 7fbd1b400640 1 -- 172.21.15.40:0/1916421367 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1400ba80 con 0x7fbd24071ef0 2024-04-26T23:03:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.828+0000 7fbd23e00640 1 -- 172.21.15.40:0/1916421367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd2415af40 con 0x7fbd24071ef0 2024-04-26T23:03:55.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.828+0000 7fbd23e00640 1 -- 172.21.15.40:0/1916421367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd240ae9f0 con 0x7fbd24071ef0 2024-04-26T23:03:55.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.830+0000 7fbd23e00640 1 -- 172.21.15.40:0/1916421367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd24069370 con 0x7fbd24071ef0 2024-04-26T23:03:55.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.831+0000 7fbd1b400640 1 -- 172.21.15.40:0/1916421367 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbd1401e460 con 0x7fbd24071ef0 2024-04-26T23:03:55.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.831+0000 7fbd1b400640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd0006c8f0 0x7fbd0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:55.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.832+0000 7fbd1b400640 1 -- 172.21.15.40:0/1916421367 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fbd1408c3f0 con 0x7fbd24071ef0 2024-04-26T23:03:55.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.832+0000 7fbd22000640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd0006c8f0 0x7fbd0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:55.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.833+0000 7fbd22000640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd0006c8f0 0x7fbd0006edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c034000 tx=0x7fbd0c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:55.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:55.836+0000 7fbd1b400640 1 -- 172.21.15.40:0/1916421367 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fbd241f5e80 con 0x7fbd24071ef0 2024-04-26T23:03:56.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.039+0000 7fbd23e00640 1 -- 172.21.15.40:0/1916421367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbd240cd1f0 con 0x7fbd24071ef0 2024-04-26T23:03:56.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.040+0000 7fbd1b400640 1 -- 172.21.15.40:0/1916421367 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7fbd1405a660 con 0x7fbd24071ef0 2024-04-26T23:03:56.042 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:03:56.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.044+0000 7fbd23e00640 1 -- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd0006c8f0 msgr2=0x7fbd0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:56.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.044+0000 7fbd23e00640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd0006c8f0 0x7fbd0006edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c034000 tx=0x7fbd0c02e000 comp rx=0 tx=0).stop 2024-04-26T23:03:56.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.044+0000 7fbd23e00640 1 -- 172.21.15.40:0/1916421367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd24071ef0 msgr2=0x7fbd241f5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:56.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.044+0000 7fbd23e00640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd24071ef0 0x7fbd241f5940 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fbd1400d830 tx=0x7fbd1400dcf0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.044+0000 7fbd23e00640 1 -- 172.21.15.40:0/1916421367 shutdown_connections 2024-04-26T23:03:56.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.044+0000 7fbd23e00640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd0006c8f0 0x7fbd0006edb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.044+0000 7fbd23e00640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2415bc70 0x7fbd2414eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.044+0000 7fbd23e00640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd24072440 0x7fbd2414d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.044+0000 7fbd23e00640 1 --2- 172.21.15.40:0/1916421367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd24071ef0 0x7fbd241f5940 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.044+0000 7fbd23e00640 1 -- 172.21.15.40:0/1916421367 >> 172.21.15.40:0/1916421367 conn(0x7fbd240b2720 msgr2=0x7fbd240b37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:56.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.044+0000 7fbd23e00640 1 -- 172.21.15.40:0/1916421367 shutdown_connections 2024-04-26T23:03:56.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.044+0000 7fbd23e00640 1 -- 172.21.15.40:0/1916421367 wait complete. 2024-04-26T23:03:56.063 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:03:56.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.276+0000 7f7292a00640 1 Processor -- start 2024-04-26T23:03:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.276+0000 7f7292a00640 1 -- start start 2024-04-26T23:03:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.278+0000 7f7292a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728c0b5120 0x7f728c0b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.278+0000 7f7292a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f728c0b5500 0x7f728c0b6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.278+0000 7f7292a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f728c0b59b0 0x7f728c0b6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.278+0000 7f7292a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f728c0d03b0 con 0x7f728c0b5500 2024-04-26T23:03:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.278+0000 7f7292a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f728c0d3dc0 con 0x7f728c0b59b0 2024-04-26T23:03:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.278+0000 7f7292a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f728c0b75b0 con 0x7f728c0b5120 2024-04-26T23:03:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.278+0000 7f7290c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f728c0b5500 0x7f728c0b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.278+0000 7f7290c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f728c0b5500 0x7f728c0b6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36572/0 (socket says 172.21.15.40:36572) 2024-04-26T23:03:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.278+0000 7f7290c00640 1 -- 172.21.15.40:0/1533142266 learned_addr learned my addr 172.21.15.40:0/1533142266 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.279+0000 7f7291600640 1 --2- 172.21.15.40:0/1533142266 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728c0b5120 0x7f728c0b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.279+0000 7f7292000640 1 --2- 172.21.15.40:0/1533142266 >> v2:172.21.15.103:3300/0 conn(0x7f728c0b59b0 0x7f728c0b6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.279+0000 7f7290c00640 1 -- 172.21.15.40:0/1533142266 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728c0b5120 msgr2=0x7f728c0b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.279+0000 7f7290c00640 1 --2- 172.21.15.40:0/1533142266 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728c0b5120 0x7f728c0b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.279+0000 7f7290c00640 1 -- 172.21.15.40:0/1533142266 >> v2:172.21.15.103:3300/0 conn(0x7f728c0b59b0 msgr2=0x7f728c0b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.279+0000 7f7290c00640 1 --2- 172.21.15.40:0/1533142266 >> v2:172.21.15.103:3300/0 conn(0x7f728c0b59b0 0x7f728c0b6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.279+0000 7f7290c00640 1 -- 172.21.15.40:0/1533142266 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728c0d0600 con 0x7f728c0b5500 2024-04-26T23:03:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.280+0000 7f7290c00640 1 --2- 172.21.15.40:0/1533142266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f728c0b5500 0x7f728c0b6830 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f728c08f5c0 tx=0x7f727802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59625f732d81c08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.280+0000 7f728be00640 1 -- 172.21.15.40:0/1533142266 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7278037070 con 0x7f728c0b5500 2024-04-26T23:03:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.280+0000 7f728be00640 1 -- 172.21.15.40:0/1533142266 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7278032440 con 0x7f728c0b5500 2024-04-26T23:03:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.280+0000 7f728be00640 1 -- 172.21.15.40:0/1533142266 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7278032810 con 0x7f728c0b5500 2024-04-26T23:03:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.281+0000 7f7292a00640 1 -- 172.21.15.40:0/1533142266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f728c0b5500 msgr2=0x7f728c0b6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.281+0000 7f7292a00640 1 --2- 172.21.15.40:0/1533142266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f728c0b5500 0x7f728c0b6830 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f728c08f5c0 tx=0x7f727802a140 comp rx=0 tx=0).stop 2024-04-26T23:03:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.281+0000 7f7292a00640 1 -- 172.21.15.40:0/1533142266 shutdown_connections 2024-04-26T23:03:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.281+0000 7f7292a00640 1 --2- 172.21.15.40:0/1533142266 >> v2:172.21.15.103:3300/0 conn(0x7f728c0b59b0 0x7f728c0b6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.281+0000 7f7292a00640 1 --2- 172.21.15.40:0/1533142266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f728c0b5500 0x7f728c0b6830 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.281+0000 7f7292a00640 1 --2- 172.21.15.40:0/1533142266 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728c0b5120 0x7f728c0b5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.281+0000 7f7292a00640 1 -- 172.21.15.40:0/1533142266 >> 172.21.15.40:0/1533142266 conn(0x7f728c0d4db0 msgr2=0x7f728c0d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.281+0000 7f7292a00640 1 -- 172.21.15.40:0/1533142266 shutdown_connections 2024-04-26T23:03:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.281+0000 7f7292a00640 1 -- 172.21.15.40:0/1533142266 wait complete. 2024-04-26T23:03:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.283+0000 7f7292a00640 1 Processor -- start 2024-04-26T23:03:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.284+0000 7f7292a00640 1 -- start start 2024-04-26T23:03:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.285+0000 7f7292a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f728c0b5120 0x7f728c0a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.285+0000 7f7292a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c0b5500 0x7f728c156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.285+0000 7f7292a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728c0b59b0 0x7f728c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.285+0000 7f7292a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f728c0b72b0 con 0x7f728c0b5120 2024-04-26T23:03:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.285+0000 7f7292a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f728c0d0bc0 con 0x7f728c0b5500 2024-04-26T23:03:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.285+0000 7f7292a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f728c0d2b00 con 0x7f728c0b59b0 2024-04-26T23:03:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.285+0000 7f7291600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f728c0b5120 0x7f728c0a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.285+0000 7f7291600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f728c0b5120 0x7f728c0a1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36584/0 (socket says 172.21.15.40:36584) 2024-04-26T23:03:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.285+0000 7f7291600640 1 -- 172.21.15.40:0/2777295132 learned_addr learned my addr 172.21.15.40:0/2777295132 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.286+0000 7f7291600640 1 -- 172.21.15.40:0/2777295132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728c0b59b0 msgr2=0x7f728c15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.286+0000 7f7291600640 1 --2- 172.21.15.40:0/2777295132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728c0b59b0 0x7f728c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.286+0000 7f7291600640 1 -- 172.21.15.40:0/2777295132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c0b5500 msgr2=0x7f728c156260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.286+0000 7f7291600640 1 --2- 172.21.15.40:0/2777295132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c0b5500 0x7f728c156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.286+0000 7f7291600640 1 -- 172.21.15.40:0/2777295132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728c1567a0 con 0x7f728c0b5120 2024-04-26T23:03:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.286+0000 7f7291600640 1 --2- 172.21.15.40:0/2777295132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f728c0b5120 0x7f728c0a1470 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f728000d800 tx=0x7f728000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.286+0000 7f7289600640 1 -- 172.21.15.40:0/2777295132 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7280018020 con 0x7f728c0b5120 2024-04-26T23:03:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.286+0000 7f7289600640 1 -- 172.21.15.40:0/2777295132 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72800096a0 con 0x7f728c0b5120 2024-04-26T23:03:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.286+0000 7f7292a00640 1 -- 172.21.15.40:0/2777295132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f728c0a1ae0 con 0x7f728c0b5120 2024-04-26T23:03:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.286+0000 7f7289600640 1 -- 172.21.15.40:0/2777295132 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7280009a70 con 0x7f728c0b5120 2024-04-26T23:03:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.286+0000 7f7292a00640 1 -- 172.21.15.40:0/2777295132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f728c0a20a0 con 0x7f728c0b5120 2024-04-26T23:03:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.287+0000 7f7292a00640 1 -- 172.21.15.40:0/2777295132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f728c069370 con 0x7f728c0b5120 2024-04-26T23:03:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.288+0000 7f7289600640 1 -- 172.21.15.40:0/2777295132 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f728001b020 con 0x7f728c0b5120 2024-04-26T23:03:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.290+0000 7f7289600640 1 --2- 172.21.15.40:0/2777295132 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f726c06c8f0 0x7f726c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.290+0000 7f7289600640 1 -- 172.21.15.40:0/2777295132 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f728008bf00 con 0x7f728c0b5120 2024-04-26T23:03:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.291+0000 7f7290c00640 1 --2- 172.21.15.40:0/2777295132 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f726c06c8f0 0x7f726c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.292+0000 7f7290c00640 1 --2- 172.21.15.40:0/2777295132 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f726c06c8f0 0x7f726c06edb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7278034000 tx=0x7f727802c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:56.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.295+0000 7f7289600640 1 -- 172.21.15.40:0/2777295132 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7280022030 con 0x7f728c0b5120 2024-04-26T23:03:56.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.500+0000 7f7292a00640 1 -- 172.21.15.40:0/2777295132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f728c0b5ee0 con 0x7f728c0b5120 2024-04-26T23:03:56.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.500+0000 7f7289600640 1 -- 172.21.15.40:0/2777295132 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) ==== 104+0+11 (secure 0 0 0) 0x7f728c1567a0 con 0x7f728c0b5120 2024-04-26T23:03:56.501 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:03:56.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.503+0000 7f7292a00640 1 -- 172.21.15.40:0/2777295132 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f726c06c8f0 msgr2=0x7f726c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:56.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.503+0000 7f7292a00640 1 --2- 172.21.15.40:0/2777295132 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f726c06c8f0 0x7f726c06edb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7278034000 tx=0x7f727802c000 comp rx=0 tx=0).stop 2024-04-26T23:03:56.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.503+0000 7f7292a00640 1 -- 172.21.15.40:0/2777295132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f728c0b5120 msgr2=0x7f728c0a1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:56.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.504+0000 7f7292a00640 1 --2- 172.21.15.40:0/2777295132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f728c0b5120 0x7f728c0a1470 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f728000d800 tx=0x7f728000dcc0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.504+0000 7f7292a00640 1 -- 172.21.15.40:0/2777295132 shutdown_connections 2024-04-26T23:03:56.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.504+0000 7f7292a00640 1 --2- 172.21.15.40:0/2777295132 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f726c06c8f0 0x7f726c06edb0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.504+0000 7f7292a00640 1 --2- 172.21.15.40:0/2777295132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f728c0b59b0 0x7f728c15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.504+0000 7f7292a00640 1 --2- 172.21.15.40:0/2777295132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c0b5500 0x7f728c156260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.504+0000 7f7292a00640 1 --2- 172.21.15.40:0/2777295132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f728c0b5120 0x7f728c0a1470 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.504+0000 7f7292a00640 1 -- 172.21.15.40:0/2777295132 >> 172.21.15.40:0/2777295132 conn(0x7f728c0d4db0 msgr2=0x7f728c0d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:56.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.504+0000 7f7292a00640 1 -- 172.21.15.40:0/2777295132 shutdown_connections 2024-04-26T23:03:56.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.504+0000 7f7292a00640 1 -- 172.21.15.40:0/2777295132 wait complete. 2024-04-26T23:03:56.522 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:03:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.726+0000 7fd6c3e00640 1 Processor -- start 2024-04-26T23:03:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.726+0000 7fd6c3e00640 1 -- start start 2024-04-26T23:03:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.727+0000 7fd6c3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6c40ce5e0 0x7fd6c40afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.728+0000 7fd6c3e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c40af7b0 con 0x7fd6c40cc590 2024-04-26T23:03:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.728+0000 7fd6c3e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c40afa90 con 0x7fd6c415f2d0 2024-04-26T23:03:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.728+0000 7fd6c3e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fd6c40cbd90 con 0x7fd6c40ce5e0 2024-04-26T23:03:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.728+0000 7fd6c2000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6c40ce5e0 0x7fd6c40afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.728+0000 7fd6c3400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd6c40cc590 0x7fd6c406f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45712/0 (socket says 172.21.15.40:45712) 2024-04-26T23:03:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.728+0000 7fd6c3400640 1 -- 172.21.15.40:0/3682087079 learned_addr learned my addr 172.21.15.40:0/3682087079 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.729+0000 7fd6c2000640 1 -- 172.21.15.40:0/3682087079 >> v1:172.21.15.103:6789/0 conn(0x7fd6c415f2d0 legacy=0x7fd6c40c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.729+0000 7fd6c2000640 1 -- 172.21.15.40:0/3682087079 >> v1:172.21.15.40:6789/0 conn(0x7fd6c40cc590 legacy=0x7fd6c406f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.729+0000 7fd6c2000640 1 -- 172.21.15.40:0/3682087079 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c40cc070 con 0x7fd6c40ce5e0 2024-04-26T23:03:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.729+0000 7fd6c1600640 1 -- 172.21.15.40:0/3682087079 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1877413374 0 0) 0x7fd6c40af7b0 con 0x7fd6c40cc590 2024-04-26T23:03:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.730+0000 7fd6c2000640 1 --2- 172.21.15.40:0/3682087079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40ce5e0 0x7fd6c40afdf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0097a0 tx=0x7fd6ac02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=745a462bd403515 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.730+0000 7fd6c1600640 1 -- 172.21.15.40:0/3682087079 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac037070 con 0x7fd6c40ce5e0 2024-04-26T23:03:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.730+0000 7fd6c1600640 1 -- 172.21.15.40:0/3682087079 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6ac032440 con 0x7fd6c40ce5e0 2024-04-26T23:03:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.730+0000 7fd6c1600640 1 -- 172.21.15.40:0/3682087079 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac032810 con 0x7fd6c40ce5e0 2024-04-26T23:03:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.731+0000 7fd6c3e00640 1 -- 172.21.15.40:0/3682087079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40ce5e0 msgr2=0x7fd6c40afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.731+0000 7fd6c3e00640 1 --2- 172.21.15.40:0/3682087079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40ce5e0 0x7fd6c40afdf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0097a0 tx=0x7fd6ac02a230 comp rx=0 tx=0).stop 2024-04-26T23:03:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.731+0000 7fd6c3e00640 1 -- 172.21.15.40:0/3682087079 shutdown_connections 2024-04-26T23:03:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.731+0000 7fd6c3e00640 1 --2- 172.21.15.40:0/3682087079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40ce5e0 0x7fd6c40afdf0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.731+0000 7fd6c3e00640 1 -- 172.21.15.40:0/3682087079 >> 172.21.15.40:0/3682087079 conn(0x7fd6c40d0a10 msgr2=0x7fd6c40d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.731+0000 7fd6c3e00640 1 -- 172.21.15.40:0/3682087079 shutdown_connections 2024-04-26T23:03:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.731+0000 7fd6c3e00640 1 -- 172.21.15.40:0/3682087079 wait complete. 2024-04-26T23:03:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.733+0000 7fd6c3e00640 1 Processor -- start 2024-04-26T23:03:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.733+0000 7fd6c3e00640 1 -- start start 2024-04-26T23:03:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.734+0000 7fd6c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40cc590 0x7fd6c415d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.734+0000 7fd6c3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6c40ce5e0 0x7fd6c415b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.734+0000 7fd6c3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6c415f2d0 0x7fd6c4151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.734+0000 7fd6c3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c40c5fc0 con 0x7fd6c40cc590 2024-04-26T23:03:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.734+0000 7fd6c3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c406e270 con 0x7fd6c40ce5e0 2024-04-26T23:03:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.734+0000 7fd6c3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd6c415d870 con 0x7fd6c415f2d0 2024-04-26T23:03:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.734+0000 7fd6c2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40cc590 0x7fd6c415d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.734+0000 7fd6c2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40cc590 0x7fd6c415d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36608/0 (socket says 172.21.15.40:36608) 2024-04-26T23:03:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.734+0000 7fd6c2a00640 1 -- 172.21.15.40:0/2934118123 learned_addr learned my addr 172.21.15.40:0/2934118123 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.735+0000 7fd6c2a00640 1 -- 172.21.15.40:0/2934118123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6c415f2d0 msgr2=0x7fd6c4151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.735+0000 7fd6c2a00640 1 --2- 172.21.15.40:0/2934118123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6c415f2d0 0x7fd6c4151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.735+0000 7fd6c2a00640 1 -- 172.21.15.40:0/2934118123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6c40ce5e0 msgr2=0x7fd6c415b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.735+0000 7fd6c2a00640 1 --2- 172.21.15.40:0/2934118123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6c40ce5e0 0x7fd6c415b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.735+0000 7fd6c2a00640 1 -- 172.21.15.40:0/2934118123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c415b5b0 con 0x7fd6c40cc590 2024-04-26T23:03:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.735+0000 7fd6c2a00640 1 --2- 172.21.15.40:0/2934118123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40cc590 0x7fd6c415d270 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400c400 tx=0x7fd6b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.735+0000 7fd6baa00640 1 -- 172.21.15.40:0/2934118123 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4017070 con 0x7fd6c40cc590 2024-04-26T23:03:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.735+0000 7fd6baa00640 1 -- 172.21.15.40:0/2934118123 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6b40052d0 con 0x7fd6c40cc590 2024-04-26T23:03:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.735+0000 7fd6baa00640 1 -- 172.21.15.40:0/2934118123 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4003830 con 0x7fd6c40cc590 2024-04-26T23:03:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.735+0000 7fd6c3e00640 1 -- 172.21.15.40:0/2934118123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c4159470 con 0x7fd6c40cc590 2024-04-26T23:03:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.735+0000 7fd6c3e00640 1 -- 172.21.15.40:0/2934118123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c4152430 con 0x7fd6c40cc590 2024-04-26T23:03:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.736+0000 7fd6c3e00640 1 -- 172.21.15.40:0/2934118123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6c4069370 con 0x7fd6c40cc590 2024-04-26T23:03:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.738+0000 7fd6baa00640 1 -- 172.21.15.40:0/2934118123 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd6b401a070 con 0x7fd6c40cc590 2024-04-26T23:03:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.739+0000 7fd6baa00640 1 --2- 172.21.15.40:0/2934118123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd6a006c8f0 0x7fd6a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.739+0000 7fd6baa00640 1 -- 172.21.15.40:0/2934118123 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fd6b408aff0 con 0x7fd6c40cc590 2024-04-26T23:03:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.739+0000 7fd6c2000640 1 --2- 172.21.15.40:0/2934118123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd6a006c8f0 0x7fd6a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.741+0000 7fd6c2000640 1 --2- 172.21.15.40:0/2934118123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd6a006c8f0 0x7fd6a006edb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac034000 tx=0x7fd6ac02c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.743+0000 7fd6baa00640 1 -- 172.21.15.40:0/2934118123 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd6c415b5b0 con 0x7fd6c40cc590 2024-04-26T23:03:56.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.948+0000 7fd6c3e00640 1 -- 172.21.15.40:0/2934118123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd6c40cebe0 con 0x7fd6c40cc590 2024-04-26T23:03:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.948+0000 7fd6baa00640 1 -- 172.21.15.40:0/2934118123 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) ==== 100+0+11 (secure 0 0 0) 0x7fd6b40592c0 con 0x7fd6c40cc590 2024-04-26T23:03:56.949 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:03:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.951+0000 7fd6c3e00640 1 -- 172.21.15.40:0/2934118123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd6a006c8f0 msgr2=0x7fd6a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.951+0000 7fd6c3e00640 1 --2- 172.21.15.40:0/2934118123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd6a006c8f0 0x7fd6a006edb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac034000 tx=0x7fd6ac02c000 comp rx=0 tx=0).stop 2024-04-26T23:03:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.951+0000 7fd6c3e00640 1 -- 172.21.15.40:0/2934118123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40cc590 msgr2=0x7fd6c415d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.951+0000 7fd6c3e00640 1 --2- 172.21.15.40:0/2934118123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40cc590 0x7fd6c415d270 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400c400 tx=0x7fd6b400c8c0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.951+0000 7fd6c3e00640 1 -- 172.21.15.40:0/2934118123 shutdown_connections 2024-04-26T23:03:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.951+0000 7fd6c3e00640 1 --2- 172.21.15.40:0/2934118123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd6a006c8f0 0x7fd6a006edb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.951+0000 7fd6c3e00640 1 --2- 172.21.15.40:0/2934118123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd6c415f2d0 0x7fd6c4151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.951+0000 7fd6c3e00640 1 --2- 172.21.15.40:0/2934118123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6c40ce5e0 0x7fd6c415b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.951+0000 7fd6c3e00640 1 --2- 172.21.15.40:0/2934118123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40cc590 0x7fd6c415d270 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.951+0000 7fd6c3e00640 1 -- 172.21.15.40:0/2934118123 >> 172.21.15.40:0/2934118123 conn(0x7fd6c40d0a10 msgr2=0x7fd6c406f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.951+0000 7fd6c3e00640 1 -- 172.21.15.40:0/2934118123 shutdown_connections 2024-04-26T23:03:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:56.951+0000 7fd6c3e00640 1 -- 172.21.15.40:0/2934118123 wait complete. 2024-04-26T23:03:56.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:03:57.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.180+0000 7eff02400640 1 Processor -- start 2024-04-26T23:03:57.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.181+0000 7eff02400640 1 -- start start 2024-04-26T23:03:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.182+0000 7eff02400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efefc0af7a0 0x7efefc0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.182+0000 7eff02400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efefc071fe0 0x7efefc073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.182+0000 7eff02400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efefc0d0820 con 0x7efefc15ce60 2024-04-26T23:03:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.182+0000 7eff02400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7efefc0cc450 con 0x7efefc071fe0 2024-04-26T23:03:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.182+0000 7eff02400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7efefc1614c0 con 0x7efefc0af7a0 2024-04-26T23:03:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.183+0000 7eff01000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efefc15ce60 0x7efefc0ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45726/0 (socket says 172.21.15.40:45726) 2024-04-26T23:03:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.183+0000 7eff01000640 1 -- 172.21.15.40:0/372363995 learned_addr learned my addr 172.21.15.40:0/372363995 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:57.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.183+0000 7eff01a00640 1 --2- 172.21.15.40:0/372363995 >> v2:172.21.15.40:3300/0 conn(0x7efefc071fe0 0x7efefc073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:57.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.183+0000 7efefbe00640 1 --2- 172.21.15.40:0/372363995 >> v2:172.21.15.103:3300/0 conn(0x7efefc0af7a0 0x7efefc0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:57.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.183+0000 7efefb400640 1 -- 172.21.15.40:0/372363995 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3893735961 0 0) 0x7efefc0d0820 con 0x7efefc15ce60 2024-04-26T23:03:57.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.183+0000 7eff01a00640 1 -- 172.21.15.40:0/372363995 >> v2:172.21.15.103:3300/0 conn(0x7efefc0af7a0 msgr2=0x7efefc0afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:57.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.183+0000 7eff01a00640 1 --2- 172.21.15.40:0/372363995 >> v2:172.21.15.103:3300/0 conn(0x7efefc0af7a0 0x7efefc0afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.183+0000 7eff01a00640 1 -- 172.21.15.40:0/372363995 >> v1:172.21.15.40:6789/0 conn(0x7efefc15ce60 legacy=0x7efefc0ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:57.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.183+0000 7eff01a00640 1 -- 172.21.15.40:0/372363995 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efefc161760 con 0x7efefc071fe0 2024-04-26T23:03:57.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.185+0000 7eff01a00640 1 --2- 172.21.15.40:0/372363995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efefc071fe0 0x7efefc073760 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7efefc0d1940 tx=0x7efef402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34eec89f7de6721 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:57.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.185+0000 7efefb400640 1 -- 172.21.15.40:0/372363995 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4007740 con 0x7efefc071fe0 2024-04-26T23:03:57.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.185+0000 7efefb400640 1 -- 172.21.15.40:0/372363995 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef40078e0 con 0x7efefc071fe0 2024-04-26T23:03:57.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.185+0000 7efefb400640 1 -- 172.21.15.40:0/372363995 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4007cb0 con 0x7efefc071fe0 2024-04-26T23:03:57.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.185+0000 7eff02400640 1 -- 172.21.15.40:0/372363995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efefc071fe0 msgr2=0x7efefc073760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:57.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.185+0000 7eff02400640 1 --2- 172.21.15.40:0/372363995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efefc071fe0 0x7efefc073760 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7efefc0d1940 tx=0x7efef402a5f0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.186+0000 7eff02400640 1 -- 172.21.15.40:0/372363995 shutdown_connections 2024-04-26T23:03:57.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.186+0000 7eff02400640 1 --2- 172.21.15.40:0/372363995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efefc071fe0 0x7efefc073760 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.186+0000 7eff02400640 1 --2- 172.21.15.40:0/372363995 >> v2:172.21.15.103:3300/0 conn(0x7efefc0af7a0 0x7efefc0afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.186+0000 7eff02400640 1 -- 172.21.15.40:0/372363995 >> 172.21.15.40:0/372363995 conn(0x7efefc0d0a10 msgr2=0x7efefc0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:57.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.186+0000 7eff02400640 1 -- 172.21.15.40:0/372363995 shutdown_connections 2024-04-26T23:03:57.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.186+0000 7eff02400640 1 -- 172.21.15.40:0/372363995 wait complete. 2024-04-26T23:03:57.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.187+0000 7eff02400640 1 Processor -- start 2024-04-26T23:03:57.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.187+0000 7eff02400640 1 -- start start 2024-04-26T23:03:57.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.188+0000 7eff02400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efefc071fe0 0x7efefc0ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:57.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.188+0000 7eff02400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efefc0af7a0 0x7efefc15e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:57.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.188+0000 7eff02400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efefc15ce60 0x7efefc06e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:57.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.188+0000 7eff02400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efefc0cc1f0 con 0x7efefc0af7a0 2024-04-26T23:03:57.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.188+0000 7eff02400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efefc0cc070 con 0x7efefc071fe0 2024-04-26T23:03:57.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.188+0000 7eff02400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efefc073ca0 con 0x7efefc15ce60 2024-04-26T23:03:57.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.189+0000 7efefbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efefc0af7a0 0x7efefc15e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:57.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.189+0000 7efefbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efefc0af7a0 0x7efefc15e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36624/0 (socket says 172.21.15.40:36624) 2024-04-26T23:03:57.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.189+0000 7efefbe00640 1 -- 172.21.15.40:0/1942601045 learned_addr learned my addr 172.21.15.40:0/1942601045 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:57.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.189+0000 7eff01000640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efefc071fe0 0x7efefc0ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.189+0000 7eff01a00640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efefc15ce60 0x7efefc06e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.189+0000 7efefbe00640 1 -- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efefc15ce60 msgr2=0x7efefc06e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.189+0000 7efefbe00640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efefc15ce60 0x7efefc06e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.189+0000 7efefbe00640 1 -- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efefc071fe0 msgr2=0x7efefc0ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.189+0000 7efefbe00640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efefc071fe0 0x7efefc0ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.189+0000 7efefbe00640 1 -- 172.21.15.40:0/1942601045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efefc0c9500 con 0x7efefc0af7a0 2024-04-26T23:03:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.189+0000 7eff01000640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efefc071fe0 0x7efefc0ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.190+0000 7efefbe00640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efefc0af7a0 0x7efefc15e7a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7efee800d800 tx=0x7efee800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.190+0000 7efef8c00640 1 -- 172.21.15.40:0/1942601045 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee800b660 con 0x7efefc0af7a0 2024-04-26T23:03:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.190+0000 7efef8c00640 1 -- 172.21.15.40:0/1942601045 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee8019070 con 0x7efefc0af7a0 2024-04-26T23:03:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.190+0000 7efef8c00640 1 -- 172.21.15.40:0/1942601045 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee800ba20 con 0x7efefc0af7a0 2024-04-26T23:03:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.190+0000 7eff02400640 1 -- 172.21.15.40:0/1942601045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efefc06fad0 con 0x7efefc0af7a0 2024-04-26T23:03:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.190+0000 7eff02400640 1 -- 172.21.15.40:0/1942601045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efefc06de80 con 0x7efefc0af7a0 2024-04-26T23:03:57.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.191+0000 7eff02400640 1 -- 172.21.15.40:0/1942601045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efefc073a60 con 0x7efefc0af7a0 2024-04-26T23:03:57.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.192+0000 7efef8c00640 1 -- 172.21.15.40:0/1942601045 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efee800bbc0 con 0x7efefc0af7a0 2024-04-26T23:03:57.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.194+0000 7efef8c00640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efedc06c8f0 0x7efedc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:57.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.194+0000 7efef8c00640 1 -- 172.21.15.40:0/1942601045 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7efee808c0c0 con 0x7efefc0af7a0 2024-04-26T23:03:57.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.195+0000 7eff01000640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efedc06c8f0 0x7efedc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:57.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.196+0000 7eff01000640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efedc06c8f0 0x7efedc06edb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efef0000f60 tx=0x7efef0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:57.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.198+0000 7efef8c00640 1 -- 172.21.15.40:0/1942601045 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7efefc073a60 con 0x7efefc0af7a0 2024-04-26T23:03:57.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.416+0000 7eff02400640 1 -- 172.21.15.40:0/1942601045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efefc0afe40 con 0x7efefc0af7a0 2024-04-26T23:03:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.417+0000 7efef8c00640 1 -- 172.21.15.40:0/1942601045 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v41) ==== 80+0+0 (secure 0 0 0) 0x7efefc0afe40 con 0x7efefc0af7a0 2024-04-26T23:03:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-04-26T23:03:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.419+0000 7eff02400640 1 -- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efedc06c8f0 msgr2=0x7efedc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.419+0000 7eff02400640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efedc06c8f0 0x7efedc06edb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efef0000f60 tx=0x7efef0009040 comp rx=0 tx=0).stop 2024-04-26T23:03:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.419+0000 7eff02400640 1 -- 172.21.15.40:0/1942601045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efefc0af7a0 msgr2=0x7efefc15e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.419+0000 7eff02400640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efefc0af7a0 0x7efefc15e7a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7efee800d800 tx=0x7efee800dcc0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.419+0000 7eff02400640 1 -- 172.21.15.40:0/1942601045 shutdown_connections 2024-04-26T23:03:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.419+0000 7eff02400640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efedc06c8f0 0x7efedc06edb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.419+0000 7eff02400640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efefc15ce60 0x7efefc06e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.419+0000 7eff02400640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efefc0af7a0 0x7efefc15e7a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.419+0000 7eff02400640 1 --2- 172.21.15.40:0/1942601045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efefc071fe0 0x7efefc0ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.419+0000 7eff02400640 1 -- 172.21.15.40:0/1942601045 >> 172.21.15.40:0/1942601045 conn(0x7efefc0d0a10 msgr2=0x7efefc0cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:57.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.420+0000 7eff02400640 1 -- 172.21.15.40:0/1942601045 shutdown_connections 2024-04-26T23:03:57.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.420+0000 7eff02400640 1 -- 172.21.15.40:0/1942601045 wait complete. 2024-04-26T23:03:57.439 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T23:03:57.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:57.439 DEBUG:teuthology.orchestra.run.smithi040:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-26T23:03:57.539 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:03:57.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:03:57.539 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T23:03:57.590 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:03:57.590 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:03:57.590 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:03:57.646 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:03:57.646 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:03:57.646 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:03:57.646 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:03:57.703 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:03:57.703 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:03:57.703 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:03:57.703 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:03:57.761 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:03:57.761 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:03:57.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.974+0000 7f353a800640 1 Processor -- start 2024-04-26T23:03:57.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.974+0000 7f353a800640 1 -- start start 2024-04-26T23:03:57.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.976+0000 7f353a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3534163430 0x7f3534166f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:57.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.976+0000 7f353a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35340cf810 0x7f3534167950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:57.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.976+0000 7f353a800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3534163810 con 0x7f35341674a0 2024-04-26T23:03:57.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.976+0000 7f353a800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f3534167e90 con 0x7f3534163430 2024-04-26T23:03:57.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.976+0000 7f353a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f35340cb400 con 0x7f35340cf810 2024-04-26T23:03:57.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.976+0000 7f3539400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3534163430 0x7f3534166f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:57.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.977+0000 7f3539400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3534163430 0x7f3534166f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36628/0 (socket says 172.21.15.40:36628) 2024-04-26T23:03:57.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.977+0000 7f3539400640 1 -- 172.21.15.40:0/1370730562 learned_addr learned my addr 172.21.15.40:0/1370730562 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:57.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.977+0000 7f3539e00640 1 --2- 172.21.15.40:0/1370730562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35340cf810 0x7f3534167950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:57.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.977+0000 7f3539400640 1 -- 172.21.15.40:0/1370730562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35340cf810 msgr2=0x7f3534167950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:57.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.977+0000 7f3539400640 1 --2- 172.21.15.40:0/1370730562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35340cf810 0x7f3534167950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.977+0000 7f3539400640 1 -- 172.21.15.40:0/1370730562 >> v1:172.21.15.103:6789/0 conn(0x7f35341674a0 legacy=0x7f35340c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:57.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.977+0000 7f3539400640 1 -- 172.21.15.40:0/1370730562 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35340cb100 con 0x7f3534163430 2024-04-26T23:03:57.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.979+0000 7f3539400640 1 --2- 172.21.15.40:0/1370730562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3534163430 0x7f3534166f60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f3528009870 tx=0x7f352802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf59f9377a8cd5b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:57.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.979+0000 7f3533e00640 1 -- 172.21.15.40:0/1370730562 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528037070 con 0x7f3534163430 2024-04-26T23:03:57.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.979+0000 7f3533e00640 1 -- 172.21.15.40:0/1370730562 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3528032440 con 0x7f3534163430 2024-04-26T23:03:57.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.979+0000 7f3533e00640 1 -- 172.21.15.40:0/1370730562 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528032810 con 0x7f3534163430 2024-04-26T23:03:57.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.979+0000 7f353a800640 1 -- 172.21.15.40:0/1370730562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3534163430 msgr2=0x7f3534166f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:57.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.979+0000 7f353a800640 1 --2- 172.21.15.40:0/1370730562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3534163430 0x7f3534166f60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f3528009870 tx=0x7f352802a270 comp rx=0 tx=0).stop 2024-04-26T23:03:57.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.980+0000 7f353a800640 1 -- 172.21.15.40:0/1370730562 shutdown_connections 2024-04-26T23:03:57.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.980+0000 7f353a800640 1 --2- 172.21.15.40:0/1370730562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35340cf810 0x7f3534167950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.980+0000 7f353a800640 1 --2- 172.21.15.40:0/1370730562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3534163430 0x7f3534166f60 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.980+0000 7f353a800640 1 -- 172.21.15.40:0/1370730562 >> 172.21.15.40:0/1370730562 conn(0x7f35340d0a10 msgr2=0x7f35340d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:57.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.980+0000 7f353a800640 1 -- 172.21.15.40:0/1370730562 shutdown_connections 2024-04-26T23:03:57.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.980+0000 7f353a800640 1 -- 172.21.15.40:0/1370730562 wait complete. 2024-04-26T23:03:57.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.981+0000 7f353a800640 1 Processor -- start 2024-04-26T23:03:57.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.982+0000 7f353a800640 1 -- start start 2024-04-26T23:03:57.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.982+0000 7f353a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35340cf810 0x7f3534150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:57.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.982+0000 7f353a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3534163430 0x7f3534165250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:57.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.982+0000 7f353a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35341674a0 0x7f35340ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:57.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.983+0000 7f353a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f35340afe80 con 0x7f35340cf810 2024-04-26T23:03:57.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.983+0000 7f353a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f35340ccc60 con 0x7f3534163430 2024-04-26T23:03:57.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.983+0000 7f353a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f35340cd1e0 con 0x7f35341674a0 2024-04-26T23:03:57.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.983+0000 7f3539400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35340cf810 0x7f3534150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:57.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.983+0000 7f3539400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35340cf810 0x7f3534150f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36642/0 (socket says 172.21.15.40:36642) 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.983+0000 7f3539400640 1 -- 172.21.15.40:0/3073861547 learned_addr learned my addr 172.21.15.40:0/3073861547 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.983+0000 7f3538a00640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3534163430 0x7f3534165250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.983+0000 7f3539400640 1 -- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35341674a0 msgr2=0x7f35340ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.983+0000 7f3539400640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35341674a0 0x7f35340ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.983+0000 7f3539400640 1 -- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3534163430 msgr2=0x7f3534165250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.983+0000 7f3539400640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3534163430 0x7f3534165250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.983+0000 7f3539400640 1 -- 172.21.15.40:0/3073861547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3534165bc0 con 0x7f35340cf810 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.983+0000 7f3538a00640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3534163430 0x7f3534165250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.984+0000 7f3539400640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35340cf810 0x7f3534150f50 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f352802a2b0 tx=0x7f35280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.984+0000 7f3531600640 1 -- 172.21.15.40:0/3073861547 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528041020 con 0x7f35340cf810 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.984+0000 7f3531600640 1 -- 172.21.15.40:0/3073861547 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35280306c0 con 0x7f35340cf810 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.984+0000 7f3531600640 1 -- 172.21.15.40:0/3073861547 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528037040 con 0x7f35340cf810 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.984+0000 7f353a800640 1 -- 172.21.15.40:0/3073861547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3534165ea0 con 0x7f35340cf810 2024-04-26T23:03:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.984+0000 7f353a800640 1 -- 172.21.15.40:0/3073861547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35340b36f0 con 0x7f35340cf810 2024-04-26T23:03:57.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.985+0000 7f353a800640 1 -- 172.21.15.40:0/3073861547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3534069370 con 0x7f35340cf810 2024-04-26T23:03:57.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.986+0000 7f3531600640 1 -- 172.21.15.40:0/3073861547 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3528044020 con 0x7f35340cf810 2024-04-26T23:03:57.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.987+0000 7f3531600640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f351406c8f0 0x7f351406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:57.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.987+0000 7f3531600640 1 -- 172.21.15.40:0/3073861547 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f35280ac5b0 con 0x7f35340cf810 2024-04-26T23:03:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.988+0000 7f3538a00640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f351406c8f0 0x7f351406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:57.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.989+0000 7f3538a00640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f351406c8f0 0x7f351406edb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3534167d60 tx=0x7f3520009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:57.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:57.991+0000 7f3531600640 1 -- 172.21.15.40:0/3073861547 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f352804b030 con 0x7f35340cf810 2024-04-26T23:03:58.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.233+0000 7f353a800640 1 -- 172.21.15.40:0/3073861547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3534162be0 con 0x7f35340cf810 2024-04-26T23:03:58.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.234+0000 7f3531600640 1 -- 172.21.15.40:0/3073861547 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5222 (secure 0 0 0) 0x7f352807a790 con 0x7f35340cf810 2024-04-26T23:03:58.235 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:03:58.235 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":26,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1979956236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1979956236},{"type":"v1","addr":"172.21.15.103:6837","nonce":1979956236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:02:47.924696+0000","modified":"2024-04-26T23:03:24.591161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5334,"mds_1":5331,"mds_2":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"b","rank":2,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.103:6836/603576312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":603576312},{"type":"v1","addr":"172.21.15.103:6836","nonce":603576312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5331":{"gid":5331,"name":"c","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6834/716981348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":716981348},{"type":"v1","addr":"172.21.15.40:6834","nonce":716981348}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5334":{"gid":5334,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.40:6835/299800334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":299800334},{"type":"v1","addr":"172.21.15.40:6835","nonce":299800334}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5334,"qdb_cluster":[5334,5331,4975]},"id":2}]} 2024-04-26T23:03:58.235 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 26 2024-04-26T23:03:58.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.237+0000 7f353a800640 1 -- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f351406c8f0 msgr2=0x7f351406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:58.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.237+0000 7f353a800640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f351406c8f0 0x7f351406edb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3534167d60 tx=0x7f3520009040 comp rx=0 tx=0).stop 2024-04-26T23:03:58.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.237+0000 7f353a800640 1 -- 172.21.15.40:0/3073861547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35340cf810 msgr2=0x7f3534150f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:58.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.237+0000 7f353a800640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35340cf810 0x7f3534150f50 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f352802a2b0 tx=0x7f35280046c0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.237+0000 7f353a800640 1 -- 172.21.15.40:0/3073861547 shutdown_connections 2024-04-26T23:03:58.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.237+0000 7f353a800640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f351406c8f0 0x7f351406edb0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.237+0000 7f353a800640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35341674a0 0x7f35340ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.237+0000 7f353a800640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3534163430 0x7f3534165250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.237+0000 7f353a800640 1 --2- 172.21.15.40:0/3073861547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35340cf810 0x7f3534150f50 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.237+0000 7f353a800640 1 -- 172.21.15.40:0/3073861547 >> 172.21.15.40:0/3073861547 conn(0x7f35340d0a10 msgr2=0x7f353406f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:58.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.237+0000 7f353a800640 1 -- 172.21.15.40:0/3073861547 shutdown_connections 2024-04-26T23:03:58.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.237+0000 7f353a800640 1 -- 172.21.15.40:0/3073861547 wait complete. 2024-04-26T23:03:58.257 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:03:58.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.471+0000 7f0f9a800640 1 Processor -- start 2024-04-26T23:03:58.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.472+0000 7f0f9a800640 1 -- start start 2024-04-26T23:03:58.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.473+0000 7f0f9a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0f940cc480 0x7f0f94072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.473+0000 7f0f9a800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f9415d7c0 con 0x7f0f94071e60 2024-04-26T23:03:58.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.473+0000 7f0f9a800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f9415daa0 con 0x7f0f94072780 2024-04-26T23:03:58.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.473+0000 7f0f9a800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0f9415dd80 con 0x7f0f940cc480 2024-04-26T23:03:58.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.473+0000 7f0f93400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0f94071e60 0x7f0f9406f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45738/0 (socket says 172.21.15.40:45738) 2024-04-26T23:03:58.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.473+0000 7f0f93400640 1 -- 172.21.15.40:0/2474353939 learned_addr learned my addr 172.21.15.40:0/2474353939 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:58.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.474+0000 7f0f92a00640 1 --2- 172.21.15.40:0/2474353939 >> v2:172.21.15.103:3300/0 conn(0x7f0f940cc480 0x7f0f94072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:58.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.474+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1122619373 0 0) 0x7f0f9415d7c0 con 0x7f0f94071e60 2024-04-26T23:03:58.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.474+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f78003520 con 0x7f0f94071e60 2024-04-26T23:03:58.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.474+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 445574141 0 0) 0x7f0f78003520 con 0x7f0f94071e60 2024-04-26T23:03:58.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.474+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 >> v2:172.21.15.103:3300/0 conn(0x7f0f940cc480 msgr2=0x7f0f94072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:58.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.474+0000 7f0f92000640 1 --2- 172.21.15.40:0/2474353939 >> v2:172.21.15.103:3300/0 conn(0x7f0f940cc480 0x7f0f94072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.474+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 >> v1:172.21.15.103:6789/0 conn(0x7f0f94072780 legacy=0x7f0f94166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:58.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.474+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f9415e060 con 0x7f0f94071e60 2024-04-26T23:03:58.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.474+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f0f84001670 con 0x7f0f94071e60 2024-04-26T23:03:58.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.474+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 >> v1:172.21.15.40:6789/0 conn(0x7f0f94071e60 legacy=0x7f0f9406f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:58.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f92000640 1 --2- 172.21.15.40:0/2474353939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f780039b0 0x7f0f78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f92000640 1 --2- 172.21.15.40:0/2474353939 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f78004800 0x7f0f78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f92000640 1 --2- 172.21.15.40:0/2474353939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f780051c0 0x7f0f7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9415dd80 con 0x7f0f780039b0 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7800c060 con 0x7f0f780051c0 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0f7800c260 con 0x7f0f78004800 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f93400640 1 --2- 172.21.15.40:0/2474353939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f780039b0 0x7f0f78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f92a00640 1 --2- 172.21.15.40:0/2474353939 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f78004800 0x7f0f78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f93e00640 1 --2- 172.21.15.40:0/2474353939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f780051c0 0x7f0f7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f93400640 1 -- 172.21.15.40:0/2474353939 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f78004800 msgr2=0x7f0f78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f93400640 1 --2- 172.21.15.40:0/2474353939 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f78004800 0x7f0f78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f93400640 1 -- 172.21.15.40:0/2474353939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f780051c0 msgr2=0x7f0f7800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f93400640 1 --2- 172.21.15.40:0/2474353939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f780051c0 0x7f0f7800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f93400640 1 -- 172.21.15.40:0/2474353939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f78003520 con 0x7f0f780039b0 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f93e00640 1 --2- 172.21.15.40:0/2474353939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f780051c0 0x7f0f7800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f92a00640 1 --2- 172.21.15.40:0/2474353939 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f78004800 0x7f0f78004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:58.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.475+0000 7f0f93e00640 1 -- 172.21.15.40:0/2474353939 reap_dead start 2024-04-26T23:03:58.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.477+0000 7f0f93400640 1 --2- 172.21.15.40:0/2474353939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f780039b0 0x7f0f78003dd0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400c240 tx=0x7f0f78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5751961a7d665353 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.477+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f84007cf0 con 0x7f0f780039b0 2024-04-26T23:03:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.477+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f840058d0 con 0x7f0f780039b0 2024-04-26T23:03:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.477+0000 7f0f92000640 1 -- 172.21.15.40:0/2474353939 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f84005d20 con 0x7f0f780039b0 2024-04-26T23:03:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.477+0000 7f0f9a800640 1 -- 172.21.15.40:0/2474353939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f780039b0 msgr2=0x7f0f78003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.477+0000 7f0f9a800640 1 --2- 172.21.15.40:0/2474353939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f780039b0 0x7f0f78003dd0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400c240 tx=0x7f0f78005830 comp rx=0 tx=0).stop 2024-04-26T23:03:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.477+0000 7f0f9a800640 1 -- 172.21.15.40:0/2474353939 shutdown_connections 2024-04-26T23:03:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.477+0000 7f0f9a800640 1 --2- 172.21.15.40:0/2474353939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f780039b0 0x7f0f78003dd0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.477+0000 7f0f9a800640 1 -- 172.21.15.40:0/2474353939 >> 172.21.15.40:0/2474353939 conn(0x7f0f940d0be0 msgr2=0x7f0f940d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.478+0000 7f0f9a800640 1 -- 172.21.15.40:0/2474353939 shutdown_connections 2024-04-26T23:03:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.478+0000 7f0f9a800640 1 -- 172.21.15.40:0/2474353939 wait complete. 2024-04-26T23:03:58.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.479+0000 7f0f9a800640 1 Processor -- start 2024-04-26T23:03:58.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.479+0000 7f0f9a800640 1 -- start start 2024-04-26T23:03:58.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.480+0000 7f0f9a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f780039b0 0x7f0f940b5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.480+0000 7f0f9a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f94154a00 0x7f0f9415bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.480+0000 7f0f9a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f9415c410 0x7f0f94157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.480+0000 7f0f9a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9415d3f0 con 0x7f0f9415c410 2024-04-26T23:03:58.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.480+0000 7f0f9a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f94072bd0 con 0x7f0f780039b0 2024-04-26T23:03:58.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.480+0000 7f0f9a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0f9415d570 con 0x7f0f94154a00 2024-04-26T23:03:58.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.481+0000 7f0f93e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f9415c410 0x7f0f94157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:58.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.481+0000 7f0f93e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f9415c410 0x7f0f94157b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36654/0 (socket says 172.21.15.40:36654) 2024-04-26T23:03:58.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.481+0000 7f0f93e00640 1 -- 172.21.15.40:0/4275609982 learned_addr learned my addr 172.21.15.40:0/4275609982 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:58.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.481+0000 7f0f92a00640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f94154a00 0x7f0f9415bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:58.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.481+0000 7f0f93400640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f780039b0 0x7f0f940b5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:58.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.481+0000 7f0f93e00640 1 -- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f94154a00 msgr2=0x7f0f9415bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:58.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.481+0000 7f0f93e00640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f94154a00 0x7f0f9415bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.481+0000 7f0f93e00640 1 -- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f780039b0 msgr2=0x7f0f940b5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:58.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.481+0000 7f0f93e00640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f780039b0 0x7f0f940b5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.481+0000 7f0f93e00640 1 -- 172.21.15.40:0/4275609982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f940b3c60 con 0x7f0f9415c410 2024-04-26T23:03:58.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.482+0000 7f0f93e00640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f9415c410 0x7f0f94157b20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0f940cc910 tx=0x7f0f8c00d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:58.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.482+0000 7f0f8b400640 1 -- 172.21.15.40:0/4275609982 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c003d80 con 0x7f0f9415c410 2024-04-26T23:03:58.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.482+0000 7f0f8b400640 1 -- 172.21.15.40:0/4275609982 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f8c017070 con 0x7f0f9415c410 2024-04-26T23:03:58.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.482+0000 7f0f9a800640 1 -- 172.21.15.40:0/4275609982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f940b3f40 con 0x7f0f9415c410 2024-04-26T23:03:58.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.482+0000 7f0f8b400640 1 -- 172.21.15.40:0/4275609982 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0052b0 con 0x7f0f9415c410 2024-04-26T23:03:58.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.482+0000 7f0f9a800640 1 -- 172.21.15.40:0/4275609982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f940b4500 con 0x7f0f9415c410 2024-04-26T23:03:58.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.483+0000 7f0f9a800640 1 -- 172.21.15.40:0/4275609982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f940694f0 con 0x7f0f9415c410 2024-04-26T23:03:58.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.484+0000 7f0f8b400640 1 -- 172.21.15.40:0/4275609982 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0f8c005450 con 0x7f0f9415c410 2024-04-26T23:03:58.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.486+0000 7f0f8b400640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0f7406c920 0x7f0f7406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.486+0000 7f0f8b400640 1 -- 172.21.15.40:0/4275609982 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f0f8c08b900 con 0x7f0f9415c410 2024-04-26T23:03:58.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.487+0000 7f0f93400640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0f7406c920 0x7f0f7406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:58.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.488+0000 7f0f93400640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0f7406c920 0x7f0f7406ede0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0f84033000 tx=0x7f0f84009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:58.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.490+0000 7f0f8b400640 1 -- 172.21.15.40:0/4275609982 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0f8c00f2f0 con 0x7f0f9415c410 2024-04-26T23:03:58.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.693+0000 7f0f9a800640 1 -- 172.21.15.40:0/4275609982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0f9415f020 con 0x7f0f9415c410 2024-04-26T23:03:58.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.695+0000 7f0f8b400640 1 -- 172.21.15.40:0/4275609982 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+16955 (secure 0 0 0) 0x7f0f8c059b70 con 0x7f0f9415c410 2024-04-26T23:03:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:03:58.696 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":41,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:03:47.741302+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:02:45.864120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-26T23:02:46.869290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:03:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.697+0000 7f0f9a800640 1 -- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0f7406c920 msgr2=0x7f0f7406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.697+0000 7f0f9a800640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0f7406c920 0x7f0f7406ede0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0f84033000 tx=0x7f0f84009030 comp rx=0 tx=0).stop 2024-04-26T23:03:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.697+0000 7f0f9a800640 1 -- 172.21.15.40:0/4275609982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f9415c410 msgr2=0x7f0f94157b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.697+0000 7f0f9a800640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f9415c410 0x7f0f94157b20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0f940cc910 tx=0x7f0f8c00d9d0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.697+0000 7f0f9a800640 1 -- 172.21.15.40:0/4275609982 shutdown_connections 2024-04-26T23:03:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.697+0000 7f0f9a800640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0f7406c920 0x7f0f7406ede0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.697+0000 7f0f9a800640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f9415c410 0x7f0f94157b20 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.697+0000 7f0f9a800640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f94154a00 0x7f0f9415bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.697+0000 7f0f9a800640 1 --2- 172.21.15.40:0/4275609982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f780039b0 0x7f0f940b5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.698+0000 7f0f9a800640 1 -- 172.21.15.40:0/4275609982 >> 172.21.15.40:0/4275609982 conn(0x7f0f940d0be0 msgr2=0x7f0f9415f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:58.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.698+0000 7f0f9a800640 1 -- 172.21.15.40:0/4275609982 shutdown_connections 2024-04-26T23:03:58.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.698+0000 7f0f9a800640 1 -- 172.21.15.40:0/4275609982 wait complete. 2024-04-26T23:03:58.715 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:02:45.864120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:02:46.869290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:03:58.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:03:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.926+0000 7fd443e00640 1 Processor -- start 2024-04-26T23:03:58.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.927+0000 7fd443e00640 1 -- start start 2024-04-26T23:03:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.928+0000 7fd443e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd444073a50 0x7fd444072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.929+0000 7fd443e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd444161880 con 0x7fd444160a50 2024-04-26T23:03:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.929+0000 7fd443e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4440cff80 con 0x7fd4440cfaf0 2024-04-26T23:03:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.929+0000 7fd443e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd444071ce0 con 0x7fd444073a50 2024-04-26T23:03:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.929+0000 7fd442a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd444160a50 0x7fd444160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45744/0 (socket says 172.21.15.40:45744) 2024-04-26T23:03:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.929+0000 7fd442a00640 1 -- 172.21.15.40:0/3156825485 learned_addr learned my addr 172.21.15.40:0/3156825485 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:03:58.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.929+0000 7fd443400640 1 --2- 172.21.15.40:0/3156825485 >> v2:172.21.15.103:3300/0 conn(0x7fd444073a50 0x7fd444072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:58.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.929+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1273039228 0 0) 0x7fd444161880 con 0x7fd444160a50 2024-04-26T23:03:58.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.929+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd428003520 con 0x7fd444160a50 2024-04-26T23:03:58.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 25314888 0 0) 0x7fd428003520 con 0x7fd444160a50 2024-04-26T23:03:58.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 >> v2:172.21.15.103:3300/0 conn(0x7fd444073a50 msgr2=0x7fd444072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:58.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd441600640 1 --2- 172.21.15.40:0/3156825485 >> v2:172.21.15.103:3300/0 conn(0x7fd444073a50 0x7fd444072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 >> v1:172.21.15.103:6789/0 conn(0x7fd4440cfaf0 legacy=0x7fd44406dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4440d0260 con 0x7fd444160a50 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fd434001670 con 0x7fd444160a50 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 >> v1:172.21.15.40:6789/0 conn(0x7fd444160a50 legacy=0x7fd444160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd441600640 1 --2- 172.21.15.40:0/3156825485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd428003a00 0x7fd428003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd441600640 1 --2- 172.21.15.40:0/3156825485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd428004800 0x7fd428004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd441600640 1 --2- 172.21.15.40:0/3156825485 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd428005210 0x7fd42800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd444071ce0 con 0x7fd428003a00 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd42800c140 con 0x7fd428004800 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd42800c340 con 0x7fd428005210 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.930+0000 7fd442a00640 1 --2- 172.21.15.40:0/3156825485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd428003a00 0x7fd428003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.931+0000 7fd442a00640 1 -- 172.21.15.40:0/3156825485 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd428005210 msgr2=0x7fd42800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.931+0000 7fd442a00640 1 --2- 172.21.15.40:0/3156825485 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd428005210 0x7fd42800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.931+0000 7fd442a00640 1 -- 172.21.15.40:0/3156825485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd428004800 msgr2=0x7fd428004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.931+0000 7fd442a00640 1 --2- 172.21.15.40:0/3156825485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd428004800 0x7fd428004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.931+0000 7fd442a00640 1 -- 172.21.15.40:0/3156825485 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd428003520 con 0x7fd428003a00 2024-04-26T23:03:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.931+0000 7fd443400640 1 -- 172.21.15.40:0/3156825485 reap_dead start 2024-04-26T23:03:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.932+0000 7fd442a00640 1 --2- 172.21.15.40:0/3156825485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd428003a00 0x7fd428003e00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd434001eb0 tx=0x7fd428005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a718b56c71fdaf74 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.932+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd434007c80 con 0x7fd428003a00 2024-04-26T23:03:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.932+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd434005af0 con 0x7fd428003a00 2024-04-26T23:03:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.932+0000 7fd441600640 1 -- 172.21.15.40:0/3156825485 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd434007070 con 0x7fd428003a00 2024-04-26T23:03:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.933+0000 7fd443e00640 1 -- 172.21.15.40:0/3156825485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd428003a00 msgr2=0x7fd428003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.933+0000 7fd443e00640 1 --2- 172.21.15.40:0/3156825485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd428003a00 0x7fd428003e00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd434001eb0 tx=0x7fd428005830 comp rx=0 tx=0).stop 2024-04-26T23:03:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.933+0000 7fd443e00640 1 -- 172.21.15.40:0/3156825485 shutdown_connections 2024-04-26T23:03:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.933+0000 7fd443e00640 1 --2- 172.21.15.40:0/3156825485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd428003a00 0x7fd428003e00 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.933+0000 7fd443e00640 1 -- 172.21.15.40:0/3156825485 >> 172.21.15.40:0/3156825485 conn(0x7fd4440b2720 msgr2=0x7fd4440b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.933+0000 7fd443e00640 1 -- 172.21.15.40:0/3156825485 shutdown_connections 2024-04-26T23:03:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.933+0000 7fd443e00640 1 -- 172.21.15.40:0/3156825485 wait complete. 2024-04-26T23:03:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.935+0000 7fd443e00640 1 Processor -- start 2024-04-26T23:03:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.935+0000 7fd443e00640 1 -- start start 2024-04-26T23:03:58.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.936+0000 7fd443e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd428003a00 0x7fd44414dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.936+0000 7fd443e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44415da30 0x7fd44406ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.936+0000 7fd443e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd44415deb0 0x7fd44406efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.936+0000 7fd443e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd444073f00 con 0x7fd44415da30 2024-04-26T23:03:58.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.936+0000 7fd443e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd4440722a0 con 0x7fd44415deb0 2024-04-26T23:03:58.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.936+0000 7fd443e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd4441468f0 con 0x7fd428003a00 2024-04-26T23:03:58.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.936+0000 7fd442000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44415da30 0x7fd44406ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:58.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.936+0000 7fd442000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44415da30 0x7fd44406ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36670/0 (socket says 172.21.15.40:36670) 2024-04-26T23:03:58.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.936+0000 7fd442000640 1 -- 172.21.15.40:0/2471461222 learned_addr learned my addr 172.21.15.40:0/2471461222 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:58.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.937+0000 7fd443400640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd44415deb0 0x7fd44406efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:58.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.937+0000 7fd442000640 1 -- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd428003a00 msgr2=0x7fd44414dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:58.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.937+0000 7fd442000640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd428003a00 0x7fd44414dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.937+0000 7fd442000640 1 -- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd44415deb0 msgr2=0x7fd44406efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:58.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.937+0000 7fd442000640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd44415deb0 0x7fd44406efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:58.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.937+0000 7fd442000640 1 -- 172.21.15.40:0/2471461222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44414e650 con 0x7fd44415da30 2024-04-26T23:03:58.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.937+0000 7fd443400640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd44415deb0 0x7fd44406efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:03:58.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.937+0000 7fd442000640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44415da30 0x7fd44406ea60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd43000c3c0 tx=0x7fd43000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:58.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.937+0000 7fd43aa00640 1 -- 172.21.15.40:0/2471461222 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd430011790 con 0x7fd44415da30 2024-04-26T23:03:58.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.937+0000 7fd43aa00640 1 -- 172.21.15.40:0/2471461222 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd430017070 con 0x7fd44415da30 2024-04-26T23:03:58.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.937+0000 7fd43aa00640 1 -- 172.21.15.40:0/2471461222 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd430011bb0 con 0x7fd44415da30 2024-04-26T23:03:58.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.937+0000 7fd443e00640 1 -- 172.21.15.40:0/2471461222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd44414e930 con 0x7fd44415da30 2024-04-26T23:03:58.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.938+0000 7fd443e00640 1 -- 172.21.15.40:0/2471461222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd44414eef0 con 0x7fd44415da30 2024-04-26T23:03:58.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.939+0000 7fd443e00640 1 -- 172.21.15.40:0/2471461222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd444069370 con 0x7fd44415da30 2024-04-26T23:03:58.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.940+0000 7fd43aa00640 1 -- 172.21.15.40:0/2471461222 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd43001c460 con 0x7fd44415da30 2024-04-26T23:03:58.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.941+0000 7fd43aa00640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd42406c8f0 0x7fd42406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:58.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.941+0000 7fd43aa00640 1 -- 172.21.15.40:0/2471461222 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fd43008bc10 con 0x7fd44415da30 2024-04-26T23:03:58.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.942+0000 7fd442a00640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd42406c8f0 0x7fd42406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.943+0000 7fd442a00640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd42406c8f0 0x7fd42406edb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd434032000 tx=0x7fd434009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:58.946+0000 7fd43aa00640 1 -- 172.21.15.40:0/2471461222 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd430059df0 con 0x7fd44415da30 2024-04-26T23:03:59.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.147+0000 7fd443e00640 1 -- 172.21.15.40:0/2471461222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd44406fa60 con 0x7fd44415da30 2024-04-26T23:03:59.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.148+0000 7fd43aa00640 1 -- 172.21.15.40:0/2471461222 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7fd4300596f0 con 0x7fd44415da30 2024-04-26T23:03:59.148 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:03:59.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.150+0000 7fd443e00640 1 -- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd42406c8f0 msgr2=0x7fd42406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:59.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.150+0000 7fd443e00640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd42406c8f0 0x7fd42406edb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd434032000 tx=0x7fd434009030 comp rx=0 tx=0).stop 2024-04-26T23:03:59.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.150+0000 7fd443e00640 1 -- 172.21.15.40:0/2471461222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44415da30 msgr2=0x7fd44406ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:59.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.151+0000 7fd443e00640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44415da30 0x7fd44406ea60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd43000c3c0 tx=0x7fd43000c7c0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.151+0000 7fd443e00640 1 -- 172.21.15.40:0/2471461222 shutdown_connections 2024-04-26T23:03:59.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.151+0000 7fd443e00640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd42406c8f0 0x7fd42406edb0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.151+0000 7fd443e00640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd44415deb0 0x7fd44406efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.151+0000 7fd443e00640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44415da30 0x7fd44406ea60 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.151+0000 7fd443e00640 1 --2- 172.21.15.40:0/2471461222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd428003a00 0x7fd44414dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.151+0000 7fd443e00640 1 -- 172.21.15.40:0/2471461222 >> 172.21.15.40:0/2471461222 conn(0x7fd4440b2720 msgr2=0x7fd4440d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:59.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.151+0000 7fd443e00640 1 -- 172.21.15.40:0/2471461222 shutdown_connections 2024-04-26T23:03:59.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.151+0000 7fd443e00640 1 -- 172.21.15.40:0/2471461222 wait complete. 2024-04-26T23:03:59.169 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:03:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.372+0000 7fa634a00640 1 Processor -- start 2024-04-26T23:03:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.372+0000 7fa634a00640 1 -- start start 2024-04-26T23:03:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.375+0000 7fa634a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa6300cfaf0 0x7fa630073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.375+0000 7fa634a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa63006dd40 0x7fa630072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.375+0000 7fa634a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa630161880 con 0x7fa630160a50 2024-04-26T23:03:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.375+0000 7fa634a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa630071ce0 con 0x7fa6300cfaf0 2024-04-26T23:03:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.375+0000 7fa634a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa6300722a0 con 0x7fa63006dd40 2024-04-26T23:03:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.375+0000 7fa627e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa6300cfaf0 0x7fa630073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.375+0000 7fa62f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa63006dd40 0x7fa630072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.375+0000 7fa62f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa63006dd40 0x7fa630072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:36782/0 (socket says 172.21.15.40:36782) 2024-04-26T23:03:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.375+0000 7fa627e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa6300cfaf0 0x7fa630073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50074/0 (socket says 172.21.15.40:50074) 2024-04-26T23:03:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.375+0000 7fa62f200640 1 -- 172.21.15.40:0/826237197 learned_addr learned my addr 172.21.15.40:0/826237197 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.375+0000 7fa62e800640 1 --1- 172.21.15.40:0/826237197 >> v1:172.21.15.103:6789/0 conn(0x7fa630160a50 0x7fa630160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:34474/0 (socket says 172.21.15.40:34474) 2024-04-26T23:03:59.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.376+0000 7fa627e00640 1 -- 172.21.15.40:0/826237197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa63006dd40 msgr2=0x7fa630072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:59.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.376+0000 7fa627e00640 1 --2- 172.21.15.40:0/826237197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa63006dd40 0x7fa630072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.376+0000 7fa627e00640 1 -- 172.21.15.40:0/826237197 >> v1:172.21.15.103:6789/0 conn(0x7fa630160a50 legacy=0x7fa630160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:59.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.376+0000 7fa627e00640 1 -- 172.21.15.40:0/826237197 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6300cff80 con 0x7fa6300cfaf0 2024-04-26T23:03:59.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.376+0000 7fa62f200640 1 --2- 172.21.15.40:0/826237197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa63006dd40 0x7fa630072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:03:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.378+0000 7fa627e00640 1 --2- 172.21.15.40:0/826237197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6300cfaf0 0x7fa630073a50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa6180097a0 tx=0x7fa61802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bfa89a90adf0d44 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.378+0000 7fa62de00640 1 -- 172.21.15.40:0/826237197 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa618037070 con 0x7fa6300cfaf0 2024-04-26T23:03:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.379+0000 7fa62de00640 1 -- 172.21.15.40:0/826237197 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa618032440 con 0x7fa6300cfaf0 2024-04-26T23:03:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.379+0000 7fa62de00640 1 -- 172.21.15.40:0/826237197 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa618032810 con 0x7fa6300cfaf0 2024-04-26T23:03:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.379+0000 7fa634a00640 1 -- 172.21.15.40:0/826237197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6300cfaf0 msgr2=0x7fa630073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.379+0000 7fa634a00640 1 --2- 172.21.15.40:0/826237197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6300cfaf0 0x7fa630073a50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa6180097a0 tx=0x7fa61802a180 comp rx=0 tx=0).stop 2024-04-26T23:03:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.379+0000 7fa634a00640 1 -- 172.21.15.40:0/826237197 shutdown_connections 2024-04-26T23:03:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.379+0000 7fa634a00640 1 --2- 172.21.15.40:0/826237197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa63006dd40 0x7fa630072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.379+0000 7fa634a00640 1 --2- 172.21.15.40:0/826237197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6300cfaf0 0x7fa630073a50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.379+0000 7fa634a00640 1 -- 172.21.15.40:0/826237197 >> 172.21.15.40:0/826237197 conn(0x7fa6300b2720 msgr2=0x7fa6300b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.379+0000 7fa634a00640 1 -- 172.21.15.40:0/826237197 shutdown_connections 2024-04-26T23:03:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.379+0000 7fa634a00640 1 -- 172.21.15.40:0/826237197 wait complete. 2024-04-26T23:03:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.380+0000 7fa634a00640 1 Processor -- start 2024-04-26T23:03:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.380+0000 7fa634a00640 1 -- start start 2024-04-26T23:03:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.381+0000 7fa634a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa63006dd40 0x7fa63013e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.381+0000 7fa634a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa630160a50 0x7fa630158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.381+0000 7fa634a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa63015f5e0 0x7fa6301590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.381+0000 7fa634a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa630071f30 con 0x7fa63015f5e0 2024-04-26T23:03:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.381+0000 7fa634a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa630073f90 con 0x7fa630160a50 2024-04-26T23:03:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.381+0000 7fa634a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa63006e1f0 con 0x7fa63006dd40 2024-04-26T23:03:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.382+0000 7fa62f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa63015f5e0 0x7fa6301590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.382+0000 7fa62f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa63015f5e0 0x7fa6301590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36672/0 (socket says 172.21.15.40:36672) 2024-04-26T23:03:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.382+0000 7fa62f200640 1 -- 172.21.15.40:0/1965106910 learned_addr learned my addr 172.21.15.40:0/1965106910 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.382+0000 7fa627e00640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa630160a50 0x7fa630158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.382+0000 7fa62e800640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa63006dd40 0x7fa63013e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.382+0000 7fa62f200640 1 -- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa63006dd40 msgr2=0x7fa63013e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.382+0000 7fa62f200640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa63006dd40 0x7fa63013e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.382+0000 7fa62f200640 1 -- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa630160a50 msgr2=0x7fa630158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.382+0000 7fa62f200640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa630160a50 0x7fa630158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.382+0000 7fa62f200640 1 -- 172.21.15.40:0/1965106910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa630158e50 con 0x7fa63015f5e0 2024-04-26T23:03:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.383+0000 7fa62f200640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa63015f5e0 0x7fa6301590a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa628009e70 tx=0x7fa62800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.383+0000 7fa626a00640 1 -- 172.21.15.40:0/1965106910 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62800cbe0 con 0x7fa63015f5e0 2024-04-26T23:03:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.383+0000 7fa626a00640 1 -- 172.21.15.40:0/1965106910 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa62800cd80 con 0x7fa63015f5e0 2024-04-26T23:03:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.383+0000 7fa634a00640 1 -- 172.21.15.40:0/1965106910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa63013d2c0 con 0x7fa63015f5e0 2024-04-26T23:03:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.383+0000 7fa626a00640 1 -- 172.21.15.40:0/1965106910 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628003bb0 con 0x7fa63015f5e0 2024-04-26T23:03:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.383+0000 7fa634a00640 1 -- 172.21.15.40:0/1965106910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa63013d880 con 0x7fa63015f5e0 2024-04-26T23:03:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.384+0000 7fa634a00640 1 -- 172.21.15.40:0/1965106910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa630069370 con 0x7fa63015f5e0 2024-04-26T23:03:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.385+0000 7fa626a00640 1 -- 172.21.15.40:0/1965106910 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa62801b030 con 0x7fa63015f5e0 2024-04-26T23:03:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.386+0000 7fa626a00640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa60c06c920 0x7fa60c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.386+0000 7fa626a00640 1 -- 172.21.15.40:0/1965106910 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fa62808c150 con 0x7fa63015f5e0 2024-04-26T23:03:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.387+0000 7fa62e800640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa60c06c920 0x7fa60c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.389+0000 7fa62e800640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa60c06c920 0x7fa60c06ede0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa620000f60 tx=0x7fa620007cc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.391+0000 7fa626a00640 1 -- 172.21.15.40:0/1965106910 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa628022020 con 0x7fa63015f5e0 2024-04-26T23:03:59.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.604+0000 7fa634a00640 1 -- 172.21.15.40:0/1965106910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa63006fa60 con 0x7fa63015f5e0 2024-04-26T23:03:59.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.605+0000 7fa626a00640 1 -- 172.21.15.40:0/1965106910 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) ==== 104+0+11 (secure 0 0 0) 0x7fa628010040 con 0x7fa63015f5e0 2024-04-26T23:03:59.606 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:03:59.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.608+0000 7fa634a00640 1 -- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa60c06c920 msgr2=0x7fa60c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:59.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.608+0000 7fa634a00640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa60c06c920 0x7fa60c06ede0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa620000f60 tx=0x7fa620007cc0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.608+0000 7fa634a00640 1 -- 172.21.15.40:0/1965106910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa63015f5e0 msgr2=0x7fa6301590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:59.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.608+0000 7fa634a00640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa63015f5e0 0x7fa6301590a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa628009e70 tx=0x7fa62800c490 comp rx=0 tx=0).stop 2024-04-26T23:03:59.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.609+0000 7fa634a00640 1 -- 172.21.15.40:0/1965106910 shutdown_connections 2024-04-26T23:03:59.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.609+0000 7fa634a00640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa60c06c920 0x7fa60c06ede0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.609+0000 7fa634a00640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa63015f5e0 0x7fa6301590a0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.609+0000 7fa634a00640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa630160a50 0x7fa630158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.609+0000 7fa634a00640 1 --2- 172.21.15.40:0/1965106910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa63006dd40 0x7fa63013e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.609+0000 7fa634a00640 1 -- 172.21.15.40:0/1965106910 >> 172.21.15.40:0/1965106910 conn(0x7fa6300b2720 msgr2=0x7fa63015d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:59.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.609+0000 7fa634a00640 1 -- 172.21.15.40:0/1965106910 shutdown_connections 2024-04-26T23:03:59.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.609+0000 7fa634a00640 1 -- 172.21.15.40:0/1965106910 wait complete. 2024-04-26T23:03:59.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:03:59.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.839+0000 7fea50400640 1 Processor -- start 2024-04-26T23:03:59.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.840+0000 7fea50400640 1 -- start start 2024-04-26T23:03:59.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.841+0000 7fea50400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea4815f550 0x7fea480d5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:59.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.841+0000 7fea50400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fea480d3bc0 0x7fea480d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:59.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.841+0000 7fea50400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea480d1040 con 0x7fea4815f170 2024-04-26T23:03:59.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.841+0000 7fea50400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fea480c9430 con 0x7fea480d3bc0 2024-04-26T23:03:59.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.841+0000 7fea50400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fea480c9680 con 0x7fea4815f550 2024-04-26T23:03:59.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.842+0000 7fea4e400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fea480d3bc0 0x7fea480d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:59.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.842+0000 7fea4e400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fea480d3bc0 0x7fea480d2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36674/0 (socket says 172.21.15.40:36674) 2024-04-26T23:03:59.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.842+0000 7fea4e400640 1 -- 172.21.15.40:0/3038265271 learned_addr learned my addr 172.21.15.40:0/3038265271 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:59.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.842+0000 7fea4d000640 1 --2- 172.21.15.40:0/3038265271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea4815f550 0x7fea480d5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:59.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.842+0000 7fea4e400640 1 -- 172.21.15.40:0/3038265271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea4815f550 msgr2=0x7fea480d5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:59.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.842+0000 7fea4e400640 1 --2- 172.21.15.40:0/3038265271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea4815f550 0x7fea480d5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.842+0000 7fea4e400640 1 -- 172.21.15.40:0/3038265271 >> v1:172.21.15.103:6789/0 conn(0x7fea4815f170 legacy=0x7fea480d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:03:59.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.842+0000 7fea4e400640 1 -- 172.21.15.40:0/3038265271 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea480c9960 con 0x7fea480d3bc0 2024-04-26T23:03:59.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.843+0000 7fea4e400640 1 --2- 172.21.15.40:0/3038265271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea480d3bc0 0x7fea480d2cb0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fea40009870 tx=0x7fea4002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e18f7da1a0c7aeb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:59.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.843+0000 7fea47e00640 1 -- 172.21.15.40:0/3038265271 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40037070 con 0x7fea480d3bc0 2024-04-26T23:03:59.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.844+0000 7fea47e00640 1 -- 172.21.15.40:0/3038265271 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea40032440 con 0x7fea480d3bc0 2024-04-26T23:03:59.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.844+0000 7fea47e00640 1 -- 172.21.15.40:0/3038265271 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40032810 con 0x7fea480d3bc0 2024-04-26T23:03:59.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.844+0000 7fea50400640 1 -- 172.21.15.40:0/3038265271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea480d3bc0 msgr2=0x7fea480d2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:59.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.844+0000 7fea50400640 1 --2- 172.21.15.40:0/3038265271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea480d3bc0 0x7fea480d2cb0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fea40009870 tx=0x7fea4002a210 comp rx=0 tx=0).stop 2024-04-26T23:03:59.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.844+0000 7fea50400640 1 -- 172.21.15.40:0/3038265271 shutdown_connections 2024-04-26T23:03:59.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.844+0000 7fea50400640 1 --2- 172.21.15.40:0/3038265271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea4815f550 0x7fea480d5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.844+0000 7fea50400640 1 --2- 172.21.15.40:0/3038265271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea480d3bc0 0x7fea480d2cb0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.844+0000 7fea50400640 1 -- 172.21.15.40:0/3038265271 >> 172.21.15.40:0/3038265271 conn(0x7fea480d4f60 msgr2=0x7fea480d5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:03:59.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.844+0000 7fea50400640 1 -- 172.21.15.40:0/3038265271 shutdown_connections 2024-04-26T23:03:59.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.844+0000 7fea50400640 1 -- 172.21.15.40:0/3038265271 wait complete. 2024-04-26T23:03:59.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.846+0000 7fea50400640 1 Processor -- start 2024-04-26T23:03:59.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.846+0000 7fea50400640 1 -- start start 2024-04-26T23:03:59.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.847+0000 7fea50400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea480d3bc0 0x7fea480cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:59.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.847+0000 7fea50400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4815f170 0x7fea480ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:59.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.847+0000 7fea50400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea480cbfa0 0x7fea4814ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:59.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.847+0000 7fea50400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fea480ca5c0 con 0x7fea480d3bc0 2024-04-26T23:03:59.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.847+0000 7fea50400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea480ca440 con 0x7fea4815f170 2024-04-26T23:03:59.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.847+0000 7fea50400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fea480ca740 con 0x7fea480cbfa0 2024-04-26T23:03:59.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.847+0000 7fea4da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea480d3bc0 0x7fea480cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:59.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.848+0000 7fea4da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea480d3bc0 0x7fea480cd500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36684/0 (socket says 172.21.15.40:36684) 2024-04-26T23:03:59.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.848+0000 7fea4da00640 1 -- 172.21.15.40:0/1495242182 learned_addr learned my addr 172.21.15.40:0/1495242182 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:03:59.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.848+0000 7fea4da00640 1 -- 172.21.15.40:0/1495242182 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea480cbfa0 msgr2=0x7fea4814ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:59.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.848+0000 7fea4da00640 1 --2- 172.21.15.40:0/1495242182 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea480cbfa0 0x7fea4814ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.848+0000 7fea4da00640 1 -- 172.21.15.40:0/1495242182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4815f170 msgr2=0x7fea480ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:03:59.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.848+0000 7fea4da00640 1 --2- 172.21.15.40:0/1495242182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4815f170 0x7fea480ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:03:59.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.848+0000 7fea4da00640 1 -- 172.21.15.40:0/1495242182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea480cc460 con 0x7fea480d3bc0 2024-04-26T23:03:59.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.848+0000 7fea4da00640 1 --2- 172.21.15.40:0/1495242182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea480d3bc0 0x7fea480cd500 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fea3800c7c0 tx=0x7fea3800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:59.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.848+0000 7fea45600640 1 -- 172.21.15.40:0/1495242182 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38016020 con 0x7fea480d3bc0 2024-04-26T23:03:59.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.848+0000 7fea45600640 1 -- 172.21.15.40:0/1495242182 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea3800aa50 con 0x7fea480d3bc0 2024-04-26T23:03:59.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.849+0000 7fea50400640 1 -- 172.21.15.40:0/1495242182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea480cc740 con 0x7fea480d3bc0 2024-04-26T23:03:59.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.849+0000 7fea45600640 1 -- 172.21.15.40:0/1495242182 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38012420 con 0x7fea480d3bc0 2024-04-26T23:03:59.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.849+0000 7fea50400640 1 -- 172.21.15.40:0/1495242182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea480ccd00 con 0x7fea480d3bc0 2024-04-26T23:03:59.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.851+0000 7fea50400640 1 -- 172.21.15.40:0/1495242182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea480694f0 con 0x7fea480d3bc0 2024-04-26T23:03:59.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.851+0000 7fea45600640 1 -- 172.21.15.40:0/1495242182 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fea38019020 con 0x7fea480d3bc0 2024-04-26T23:03:59.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.853+0000 7fea45600640 1 --2- 172.21.15.40:0/1495242182 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea2806c8f0 0x7fea2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:03:59.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.853+0000 7fea45600640 1 -- 172.21.15.40:0/1495242182 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fea3808bb10 con 0x7fea480d3bc0 2024-04-26T23:03:59.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.853+0000 7fea4d000640 1 --2- 172.21.15.40:0/1495242182 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea2806c8f0 0x7fea2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:03:59.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.854+0000 7fea4d000640 1 --2- 172.21.15.40:0/1495242182 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea2806c8f0 0x7fea2806edb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fea480a2e80 tx=0x7fea34008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:03:59.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:03:59.857+0000 7fea45600640 1 -- 172.21.15.40:0/1495242182 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fea38020070 con 0x7fea480d3bc0 2024-04-26T23:04:00.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.061+0000 7fea50400640 1 -- 172.21.15.40:0/1495242182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fea480ba9b0 con 0x7fea480d3bc0 2024-04-26T23:04:00.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.062+0000 7fea45600640 1 -- 172.21.15.40:0/1495242182 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) ==== 100+0+11 (secure 0 0 0) 0x7fea38059cf0 con 0x7fea480d3bc0 2024-04-26T23:04:00.062 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:04:00.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.064+0000 7fea50400640 1 -- 172.21.15.40:0/1495242182 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea2806c8f0 msgr2=0x7fea2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:00.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.064+0000 7fea50400640 1 --2- 172.21.15.40:0/1495242182 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea2806c8f0 0x7fea2806edb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fea480a2e80 tx=0x7fea34008040 comp rx=0 tx=0).stop 2024-04-26T23:04:00.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.064+0000 7fea50400640 1 -- 172.21.15.40:0/1495242182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea480d3bc0 msgr2=0x7fea480cd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:00.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.064+0000 7fea50400640 1 --2- 172.21.15.40:0/1495242182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea480d3bc0 0x7fea480cd500 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fea3800c7c0 tx=0x7fea3800cc80 comp rx=0 tx=0).stop 2024-04-26T23:04:00.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.065+0000 7fea50400640 1 -- 172.21.15.40:0/1495242182 shutdown_connections 2024-04-26T23:04:00.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.065+0000 7fea50400640 1 --2- 172.21.15.40:0/1495242182 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea2806c8f0 0x7fea2806edb0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.065+0000 7fea50400640 1 --2- 172.21.15.40:0/1495242182 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea480cbfa0 0x7fea4814ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.065+0000 7fea50400640 1 --2- 172.21.15.40:0/1495242182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4815f170 0x7fea480ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.065+0000 7fea50400640 1 --2- 172.21.15.40:0/1495242182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea480d3bc0 0x7fea480cd500 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.065+0000 7fea50400640 1 -- 172.21.15.40:0/1495242182 >> 172.21.15.40:0/1495242182 conn(0x7fea480d4f60 msgr2=0x7fea4806d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:00.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.065+0000 7fea50400640 1 -- 172.21.15.40:0/1495242182 shutdown_connections 2024-04-26T23:04:00.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.065+0000 7fea50400640 1 -- 172.21.15.40:0/1495242182 wait complete. 2024-04-26T23:04:00.084 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:04:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.294+0000 7f7952a00640 1 Processor -- start 2024-04-26T23:04:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.294+0000 7f7952a00640 1 -- start start 2024-04-26T23:04:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.296+0000 7f7952a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794c15ce60 0x7f794c0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.296+0000 7f7952a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f794c071fe0 0x7f794c073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.296+0000 7f7952a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794c0d0820 con 0x7f794c0af7a0 2024-04-26T23:04:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.296+0000 7f7952a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f794c0cc450 con 0x7f794c071fe0 2024-04-26T23:04:00.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.296+0000 7f7952a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f794c1614c0 con 0x7f794c15ce60 2024-04-26T23:04:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.297+0000 7f7952000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f794c071fe0 0x7f794c073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.297+0000 7f7951600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794c15ce60 0x7f794c0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.297+0000 7f7952000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f794c071fe0 0x7f794c073760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50106/0 (socket says 172.21.15.40:50106) 2024-04-26T23:04:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.297+0000 7f7952000640 1 -- 172.21.15.40:0/880302874 learned_addr learned my addr 172.21.15.40:0/880302874 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:00.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.297+0000 7f7952000640 1 -- 172.21.15.40:0/880302874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794c15ce60 msgr2=0x7f794c0afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:00.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.297+0000 7f7952000640 1 --2- 172.21.15.40:0/880302874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794c15ce60 0x7f794c0afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.297+0000 7f7952000640 1 -- 172.21.15.40:0/880302874 >> v1:172.21.15.103:6789/0 conn(0x7f794c0af7a0 legacy=0x7f794c0ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:00.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.297+0000 7f7952000640 1 -- 172.21.15.40:0/880302874 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794c161760 con 0x7f794c071fe0 2024-04-26T23:04:00.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.297+0000 7f7951600640 1 --2- 172.21.15.40:0/880302874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794c15ce60 0x7f794c0afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:04:00.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.298+0000 7f7952000640 1 --2- 172.21.15.40:0/880302874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c071fe0 0x7f794c073760 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7944009e50 tx=0x7f794402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb66ebec567d0548 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:00.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.298+0000 7f794be00640 1 -- 172.21.15.40:0/880302874 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7944036070 con 0x7f794c071fe0 2024-04-26T23:04:00.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.298+0000 7f794be00640 1 -- 172.21.15.40:0/880302874 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7944031600 con 0x7f794c071fe0 2024-04-26T23:04:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.298+0000 7f794be00640 1 -- 172.21.15.40:0/880302874 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79440319d0 con 0x7f794c071fe0 2024-04-26T23:04:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.299+0000 7f7952a00640 1 -- 172.21.15.40:0/880302874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c071fe0 msgr2=0x7f794c073760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.299+0000 7f7952a00640 1 --2- 172.21.15.40:0/880302874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c071fe0 0x7f794c073760 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7944009e50 tx=0x7f794402a680 comp rx=0 tx=0).stop 2024-04-26T23:04:00.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.299+0000 7f7952a00640 1 -- 172.21.15.40:0/880302874 shutdown_connections 2024-04-26T23:04:00.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.299+0000 7f7952a00640 1 --2- 172.21.15.40:0/880302874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c071fe0 0x7f794c073760 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.299+0000 7f7952a00640 1 --2- 172.21.15.40:0/880302874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794c15ce60 0x7f794c0afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.299+0000 7f7952a00640 1 -- 172.21.15.40:0/880302874 >> 172.21.15.40:0/880302874 conn(0x7f794c0d0a10 msgr2=0x7f794c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:00.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.299+0000 7f7952a00640 1 -- 172.21.15.40:0/880302874 shutdown_connections 2024-04-26T23:04:00.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.299+0000 7f7952a00640 1 -- 172.21.15.40:0/880302874 wait complete. 2024-04-26T23:04:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.301+0000 7f7952a00640 1 Processor -- start 2024-04-26T23:04:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.301+0000 7f7952a00640 1 -- start start 2024-04-26T23:04:00.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.302+0000 7f7952a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794c071fe0 0x7f794c152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:00.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.302+0000 7f7952a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c0af7a0 0x7f794c0c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:00.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.302+0000 7f7952a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f794c15ce60 0x7f794c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:00.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.302+0000 7f7952a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f794c0cc1f0 con 0x7f794c15ce60 2024-04-26T23:04:00.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.302+0000 7f7952a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f794c0cc070 con 0x7f794c0af7a0 2024-04-26T23:04:00.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.302+0000 7f7952a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f794c073ca0 con 0x7f794c071fe0 2024-04-26T23:04:00.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.303+0000 7f7952000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f794c15ce60 0x7f794c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:00.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.303+0000 7f7952000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f794c15ce60 0x7f794c06ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36692/0 (socket says 172.21.15.40:36692) 2024-04-26T23:04:00.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.303+0000 7f7952000640 1 -- 172.21.15.40:0/867377034 learned_addr learned my addr 172.21.15.40:0/867377034 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:00.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.303+0000 7f7951600640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794c071fe0 0x7f794c152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:00.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.303+0000 7f7950c00640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c0af7a0 0x7f794c0c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:00.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.303+0000 7f7952000640 1 -- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794c071fe0 msgr2=0x7f794c152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:00.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.303+0000 7f7952000640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794c071fe0 0x7f794c152900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.303+0000 7f7952000640 1 -- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c0af7a0 msgr2=0x7f794c0c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:00.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.303+0000 7f7952000640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c0af7a0 0x7f794c0c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.303+0000 7f7952000640 1 -- 172.21.15.40:0/867377034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794c152e40 con 0x7f794c15ce60 2024-04-26T23:04:00.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.303+0000 7f7952000640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f794c15ce60 0x7f794c06ee80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7944009f70 tx=0x7f7944004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:00.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.304+0000 7f7949600640 1 -- 172.21.15.40:0/867377034 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794403f020 con 0x7f794c15ce60 2024-04-26T23:04:00.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.304+0000 7f7949600640 1 -- 172.21.15.40:0/867377034 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7944037040 con 0x7f794c15ce60 2024-04-26T23:04:00.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.304+0000 7f7949600640 1 -- 172.21.15.40:0/867377034 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7944036070 con 0x7f794c15ce60 2024-04-26T23:04:00.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.304+0000 7f7952a00640 1 -- 172.21.15.40:0/867377034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f794c0b4230 con 0x7f794c15ce60 2024-04-26T23:04:00.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.304+0000 7f7952a00640 1 -- 172.21.15.40:0/867377034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f794c0b47f0 con 0x7f794c15ce60 2024-04-26T23:04:00.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.305+0000 7f7952a00640 1 -- 172.21.15.40:0/867377034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f794c069370 con 0x7f794c15ce60 2024-04-26T23:04:00.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.306+0000 7f7949600640 1 -- 172.21.15.40:0/867377034 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7944042020 con 0x7f794c15ce60 2024-04-26T23:04:00.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.308+0000 7f7949600640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f792c06c8f0 0x7f792c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:00.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.308+0000 7f7949600640 1 -- 172.21.15.40:0/867377034 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f79440ac210 con 0x7f794c15ce60 2024-04-26T23:04:00.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.308+0000 7f7951600640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f792c06c8f0 0x7f792c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:00.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.309+0000 7f7951600640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f792c06c8f0 0x7f792c06edb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f794c0b4120 tx=0x7f7940008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:00.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.313+0000 7f7949600640 1 -- 172.21.15.40:0/867377034 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7944049030 con 0x7f794c15ce60 2024-04-26T23:04:00.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.561+0000 7f7952a00640 1 -- 172.21.15.40:0/867377034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f794c0b10a0 con 0x7f794c15ce60 2024-04-26T23:04:00.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.564+0000 7f7949600640 1 -- 172.21.15.40:0/867377034 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5222 (secure 0 0 0) 0x7f794407a3f0 con 0x7f794c15ce60 2024-04-26T23:04:00.563 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:00.563 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":26,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1979956236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1979956236},{"type":"v1","addr":"172.21.15.103:6837","nonce":1979956236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:02:47.924696+0000","modified":"2024-04-26T23:03:24.591161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5334,"mds_1":5331,"mds_2":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"b","rank":2,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.103:6836/603576312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":603576312},{"type":"v1","addr":"172.21.15.103:6836","nonce":603576312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5331":{"gid":5331,"name":"c","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6834/716981348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":716981348},{"type":"v1","addr":"172.21.15.40:6834","nonce":716981348}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5334":{"gid":5334,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.40:6835/299800334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":299800334},{"type":"v1","addr":"172.21.15.40:6835","nonce":299800334}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5334,"qdb_cluster":[5334,5331,4975]},"id":2}]} 2024-04-26T23:04:00.563 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 26 2024-04-26T23:04:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.565+0000 7f7952a00640 1 -- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f792c06c8f0 msgr2=0x7f792c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.565+0000 7f7952a00640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f792c06c8f0 0x7f792c06edb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f794c0b4120 tx=0x7f7940008040 comp rx=0 tx=0).stop 2024-04-26T23:04:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.565+0000 7f7952a00640 1 -- 172.21.15.40:0/867377034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f794c15ce60 msgr2=0x7f794c06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.565+0000 7f7952a00640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f794c15ce60 0x7f794c06ee80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7944009f70 tx=0x7f7944004a40 comp rx=0 tx=0).stop 2024-04-26T23:04:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.565+0000 7f7952a00640 1 -- 172.21.15.40:0/867377034 shutdown_connections 2024-04-26T23:04:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.565+0000 7f7952a00640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f792c06c8f0 0x7f792c06edb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.565+0000 7f7952a00640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f794c15ce60 0x7f794c06ee80 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.565+0000 7f7952a00640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c0af7a0 0x7f794c0c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.565+0000 7f7952a00640 1 --2- 172.21.15.40:0/867377034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794c071fe0 0x7f794c152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.565+0000 7f7952a00640 1 -- 172.21.15.40:0/867377034 >> 172.21.15.40:0/867377034 conn(0x7f794c0d0a10 msgr2=0x7f794c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.565+0000 7f7952a00640 1 -- 172.21.15.40:0/867377034 shutdown_connections 2024-04-26T23:04:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.565+0000 7f7952a00640 1 -- 172.21.15.40:0/867377034 wait complete. 2024-04-26T23:04:00.583 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:04:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.797+0000 7f0f0f000640 1 Processor -- start 2024-04-26T23:04:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.797+0000 7f0f0f000640 1 -- start start 2024-04-26T23:04:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.798+0000 7f0f0f000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0f080af7a0 0x7f0f080afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.798+0000 7f0f0f000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0f08071fe0 0x7f0f08073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.798+0000 7f0f0f000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f080d0820 con 0x7f0f0815ce60 2024-04-26T23:04:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.798+0000 7f0f0f000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f0f080cc450 con 0x7f0f080af7a0 2024-04-26T23:04:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.798+0000 7f0f0f000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0f081614c0 con 0x7f0f08071fe0 2024-04-26T23:04:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.798+0000 7f0f0d200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0f080af7a0 0x7f0f080afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.798+0000 7f0f0d200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0f080af7a0 0x7f0f080afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36696/0 (socket says 172.21.15.40:36696) 2024-04-26T23:04:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.799+0000 7f0f0d200640 1 -- 172.21.15.40:0/3921190280 learned_addr learned my addr 172.21.15.40:0/3921190280 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.799+0000 7f0f0e600640 1 --2- 172.21.15.40:0/3921190280 >> v2:172.21.15.103:3300/0 conn(0x7f0f08071fe0 0x7f0f08073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.799+0000 7f0f0d200640 1 -- 172.21.15.40:0/3921190280 >> v2:172.21.15.103:3300/0 conn(0x7f0f08071fe0 msgr2=0x7f0f08073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.799+0000 7f0f0d200640 1 --2- 172.21.15.40:0/3921190280 >> v2:172.21.15.103:3300/0 conn(0x7f0f08071fe0 0x7f0f08073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.799+0000 7f0f0d200640 1 -- 172.21.15.40:0/3921190280 >> v1:172.21.15.103:6789/0 conn(0x7f0f0815ce60 legacy=0x7f0f080ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.799+0000 7f0f0d200640 1 -- 172.21.15.40:0/3921190280 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f08161760 con 0x7f0f080af7a0 2024-04-26T23:04:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.799+0000 7f0f0e600640 1 --2- 172.21.15.40:0/3921190280 >> v2:172.21.15.103:3300/0 conn(0x7f0f08071fe0 0x7f0f08073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.800+0000 7f0f0d200640 1 --2- 172.21.15.40:0/3921190280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af7a0 0x7f0f080afc60 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0f080d1af0 tx=0x7f0ef402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c27bd2c594a95592 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.800+0000 7f0f07e00640 1 -- 172.21.15.40:0/3921190280 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4037070 con 0x7f0f080af7a0 2024-04-26T23:04:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.800+0000 7f0f07e00640 1 -- 172.21.15.40:0/3921190280 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef4032440 con 0x7f0f080af7a0 2024-04-26T23:04:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.800+0000 7f0f07e00640 1 -- 172.21.15.40:0/3921190280 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4032810 con 0x7f0f080af7a0 2024-04-26T23:04:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.801+0000 7f0f0f000640 1 -- 172.21.15.40:0/3921190280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af7a0 msgr2=0x7f0f080afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.801+0000 7f0f0f000640 1 --2- 172.21.15.40:0/3921190280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af7a0 0x7f0f080afc60 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0f080d1af0 tx=0x7f0ef402a1a0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.801+0000 7f0f0f000640 1 -- 172.21.15.40:0/3921190280 shutdown_connections 2024-04-26T23:04:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.801+0000 7f0f0f000640 1 --2- 172.21.15.40:0/3921190280 >> v2:172.21.15.103:3300/0 conn(0x7f0f08071fe0 0x7f0f08073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.801+0000 7f0f0f000640 1 --2- 172.21.15.40:0/3921190280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af7a0 0x7f0f080afc60 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.801+0000 7f0f0f000640 1 -- 172.21.15.40:0/3921190280 >> 172.21.15.40:0/3921190280 conn(0x7f0f080d0a10 msgr2=0x7f0f080d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.801+0000 7f0f0f000640 1 -- 172.21.15.40:0/3921190280 shutdown_connections 2024-04-26T23:04:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.801+0000 7f0f0f000640 1 -- 172.21.15.40:0/3921190280 wait complete. 2024-04-26T23:04:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.803+0000 7f0f0f000640 1 Processor -- start 2024-04-26T23:04:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.803+0000 7f0f0f000640 1 -- start start 2024-04-26T23:04:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.804+0000 7f0f0f000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f08071fe0 0x7f0f080c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.804+0000 7f0f0f000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af7a0 0x7f0f08159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.804+0000 7f0f0f000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f0815ce60 0x7f0f081572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.804+0000 7f0f0f000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0f080cc1f0 con 0x7f0f080af7a0 2024-04-26T23:04:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.804+0000 7f0f0f000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f080cc070 con 0x7f0f08071fe0 2024-04-26T23:04:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.804+0000 7f0f0f000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0f08073ca0 con 0x7f0f0815ce60 2024-04-26T23:04:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.804+0000 7f0f0d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af7a0 0x7f0f08159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.804+0000 7f0f0d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af7a0 0x7f0f08159470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36704/0 (socket says 172.21.15.40:36704) 2024-04-26T23:04:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.804+0000 7f0f0d200640 1 -- 172.21.15.40:0/259639693 learned_addr learned my addr 172.21.15.40:0/259639693 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.804+0000 7f0f0dc00640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f08071fe0 0x7f0f080c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f0d200640 1 -- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f0815ce60 msgr2=0x7f0f081572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f0e600640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f0815ce60 0x7f0f081572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f0d200640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f0815ce60 0x7f0f081572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f0d200640 1 -- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f08071fe0 msgr2=0x7f0f080c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f0d200640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f08071fe0 0x7f0f080c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f0d200640 1 -- 172.21.15.40:0/259639693 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f08147680 con 0x7f0f080af7a0 2024-04-26T23:04:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f0dc00640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f08071fe0 0x7f0f080c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f0e600640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f0815ce60 0x7f0f081572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f0d200640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af7a0 0x7f0f08159470 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4000c00 tx=0x7f0ef40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f05600640 1 -- 172.21.15.40:0/259639693 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4041020 con 0x7f0f080af7a0 2024-04-26T23:04:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f0f000640 1 -- 172.21.15.40:0/259639693 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f08158320 con 0x7f0f080af7a0 2024-04-26T23:04:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f05600640 1 -- 172.21.15.40:0/259639693 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef4030970 con 0x7f0f080af7a0 2024-04-26T23:04:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f05600640 1 -- 172.21.15.40:0/259639693 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4037040 con 0x7f0f080af7a0 2024-04-26T23:04:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.805+0000 7f0f0f000640 1 -- 172.21.15.40:0/259639693 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f081588e0 con 0x7f0f080af7a0 2024-04-26T23:04:00.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.807+0000 7f0f0f000640 1 -- 172.21.15.40:0/259639693 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f08156f80 con 0x7f0f080af7a0 2024-04-26T23:04:00.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.808+0000 7f0f05600640 1 -- 172.21.15.40:0/259639693 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ef4044020 con 0x7f0f080af7a0 2024-04-26T23:04:00.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.808+0000 7f0f05600640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ee806c920 0x7f0ee806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:00.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.809+0000 7f0f05600640 1 -- 172.21.15.40:0/259639693 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f0ef40ac750 con 0x7f0f080af7a0 2024-04-26T23:04:00.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.809+0000 7f0f0dc00640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ee806c920 0x7f0ee806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:00.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.810+0000 7f0f0dc00640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ee806c920 0x7f0ee806ede0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0f08159dc0 tx=0x7f0efc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:00.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:00.813+0000 7f0f05600640 1 -- 172.21.15.40:0/259639693 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0ef404b030 con 0x7f0f080af7a0 2024-04-26T23:04:01.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.018+0000 7f0f0f000640 1 -- 172.21.15.40:0/259639693 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0f08069370 con 0x7f0f080af7a0 2024-04-26T23:04:01.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.020+0000 7f0f05600640 1 -- 172.21.15.40:0/259639693 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+16955 (secure 0 0 0) 0x7f0ef407a9c0 con 0x7f0f080af7a0 2024-04-26T23:04:01.021 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:01.021 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":41,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:03:47.741302+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:02:45.864120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-26T23:02:46.869290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:04:01.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.023+0000 7f0f0f000640 1 -- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ee806c920 msgr2=0x7f0ee806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:01.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.023+0000 7f0f0f000640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ee806c920 0x7f0ee806ede0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0f08159dc0 tx=0x7f0efc008040 comp rx=0 tx=0).stop 2024-04-26T23:04:01.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.023+0000 7f0f0f000640 1 -- 172.21.15.40:0/259639693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af7a0 msgr2=0x7f0f08159470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:01.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.023+0000 7f0f0f000640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af7a0 0x7f0f08159470 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4000c00 tx=0x7f0ef40046c0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.023+0000 7f0f0f000640 1 -- 172.21.15.40:0/259639693 shutdown_connections 2024-04-26T23:04:01.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.023+0000 7f0f0f000640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ee806c920 0x7f0ee806ede0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.023+0000 7f0f0f000640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f0815ce60 0x7f0f081572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.023+0000 7f0f0f000640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af7a0 0x7f0f08159470 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.023+0000 7f0f0f000640 1 --2- 172.21.15.40:0/259639693 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f08071fe0 0x7f0f080c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.023+0000 7f0f0f000640 1 -- 172.21.15.40:0/259639693 >> 172.21.15.40:0/259639693 conn(0x7f0f080d0a10 msgr2=0x7f0f0814d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:01.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.023+0000 7f0f0f000640 1 -- 172.21.15.40:0/259639693 shutdown_connections 2024-04-26T23:04:01.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.024+0000 7f0f0f000640 1 -- 172.21.15.40:0/259639693 wait complete. 2024-04-26T23:04:01.043 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T23:04:01.043 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:04:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.254+0000 7f2e9f200640 1 Processor -- start 2024-04-26T23:04:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.255+0000 7f2e9f200640 1 -- start start 2024-04-26T23:04:01.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.256+0000 7f2e9f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2ea00d32e0 0x7f2ea00d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.256+0000 7f2e9f200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea0162c40 con 0x7f2ea00b6220 2024-04-26T23:04:01.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.256+0000 7f2e9f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea00d2d80 con 0x7f2ea00b8d20 2024-04-26T23:04:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.256+0000 7f2e9f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2ea00d3c80 con 0x7f2ea00d32e0 2024-04-26T23:04:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.257+0000 7f2e9de00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2ea00b6220 0x7f2ea00d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45746/0 (socket says 172.21.15.40:45746) 2024-04-26T23:04:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.257+0000 7f2e9de00640 1 -- 172.21.15.40:0/3486384260 learned_addr learned my addr 172.21.15.40:0/3486384260 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.257+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3363985569 0 0) 0x7f2ea0162c40 con 0x7f2ea00b6220 2024-04-26T23:04:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.257+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e84003520 con 0x7f2ea00b6220 2024-04-26T23:04:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.257+0000 7f2e9d400640 1 --2- 172.21.15.40:0/3486384260 >> v2:172.21.15.103:3300/0 conn(0x7f2ea00d32e0 0x7f2ea00d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 407494160 0 0) 0x7f2e84003520 con 0x7f2ea00b6220 2024-04-26T23:04:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 >> v2:172.21.15.103:3300/0 conn(0x7f2ea00d32e0 msgr2=0x7f2ea00d3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9ca00640 1 --2- 172.21.15.40:0/3486384260 >> v2:172.21.15.103:3300/0 conn(0x7f2ea00d32e0 0x7f2ea00d3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 >> v1:172.21.15.103:6789/0 conn(0x7f2ea00b8d20 legacy=0x7f2ea00b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea00b7c80 con 0x7f2ea00b6220 2024-04-26T23:04:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f2e88001670 con 0x7f2ea00b6220 2024-04-26T23:04:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 >> v1:172.21.15.40:6789/0 conn(0x7f2ea00b6220 legacy=0x7f2ea00d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9ca00640 1 --2- 172.21.15.40:0/3486384260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e840039b0 0x7f2e84003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9ca00640 1 --2- 172.21.15.40:0/3486384260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e84004800 0x7f2e84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9ca00640 1 --2- 172.21.15.40:0/3486384260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e840051c0 0x7f2e8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8400c090 con 0x7f2e840051c0 2024-04-26T23:04:01.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8400c240 con 0x7f2e840039b0 2024-04-26T23:04:01.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2e8400c440 con 0x7f2e84004800 2024-04-26T23:04:01.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9e800640 1 --2- 172.21.15.40:0/3486384260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e840051c0 0x7f2e8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:01.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.258+0000 7f2e9de00640 1 --2- 172.21.15.40:0/3486384260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e840039b0 0x7f2e84003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:01.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.259+0000 7f2e9d400640 1 --2- 172.21.15.40:0/3486384260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e84004800 0x7f2e84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:01.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.259+0000 7f2e9e800640 1 -- 172.21.15.40:0/3486384260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e84004800 msgr2=0x7f2e84004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:01.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.259+0000 7f2e9e800640 1 --2- 172.21.15.40:0/3486384260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e84004800 0x7f2e84004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.259+0000 7f2e9e800640 1 -- 172.21.15.40:0/3486384260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e840039b0 msgr2=0x7f2e84003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:01.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.259+0000 7f2e9e800640 1 --2- 172.21.15.40:0/3486384260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e840039b0 0x7f2e84003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.259+0000 7f2e9e800640 1 -- 172.21.15.40:0/3486384260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea00b7c80 con 0x7f2e840051c0 2024-04-26T23:04:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.260+0000 7f2e9e800640 1 -- 172.21.15.40:0/3486384260 reap_dead start 2024-04-26T23:04:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.260+0000 7f2e9e800640 1 --2- 172.21.15.40:0/3486384260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e840051c0 0x7f2e8400b9c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2e98003db0 tx=0x7f2e84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6536edd45d95eb58 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.260+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e98003980 con 0x7f2e840051c0 2024-04-26T23:04:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.260+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e98007240 con 0x7f2e840051c0 2024-04-26T23:04:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.260+0000 7f2e9ca00640 1 -- 172.21.15.40:0/3486384260 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e98007630 con 0x7f2e840051c0 2024-04-26T23:04:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.261+0000 7f2e9f200640 1 -- 172.21.15.40:0/3486384260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e840051c0 msgr2=0x7f2e8400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.261+0000 7f2e9f200640 1 --2- 172.21.15.40:0/3486384260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e840051c0 0x7f2e8400b9c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2e98003db0 tx=0x7f2e84005830 comp rx=0 tx=0).stop 2024-04-26T23:04:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.261+0000 7f2e9f200640 1 -- 172.21.15.40:0/3486384260 shutdown_connections 2024-04-26T23:04:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.261+0000 7f2e9f200640 1 --2- 172.21.15.40:0/3486384260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e840051c0 0x7f2e8400b9c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.261+0000 7f2e9f200640 1 -- 172.21.15.40:0/3486384260 >> 172.21.15.40:0/3486384260 conn(0x7f2ea00d4d90 msgr2=0x7f2ea00d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.261+0000 7f2e9f200640 1 -- 172.21.15.40:0/3486384260 shutdown_connections 2024-04-26T23:04:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.261+0000 7f2e9f200640 1 -- 172.21.15.40:0/3486384260 wait complete. 2024-04-26T23:04:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.263+0000 7f2e9f200640 1 Processor -- start 2024-04-26T23:04:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.263+0000 7f2e9f200640 1 -- start start 2024-04-26T23:04:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.264+0000 7f2e9f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e840051c0 0x7f2ea015c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.264+0000 7f2e9f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea00c9590 0x7f2ea00cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.264+0000 7f2e9f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ea01f3cb0 0x7f2ea01f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.264+0000 7f2e9f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea00b8520 con 0x7f2ea00c9590 2024-04-26T23:04:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.264+0000 7f2e9f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea00b8240 con 0x7f2e840051c0 2024-04-26T23:04:01.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.264+0000 7f2e9f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea00d3740 con 0x7f2ea01f3cb0 2024-04-26T23:04:01.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.264+0000 7f2e9d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea00c9590 0x7f2ea00cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:01.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.264+0000 7f2e9d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea00c9590 0x7f2ea00cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36724/0 (socket says 172.21.15.40:36724) 2024-04-26T23:04:01.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.264+0000 7f2e9d400640 1 -- 172.21.15.40:0/3792373115 learned_addr learned my addr 172.21.15.40:0/3792373115 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:01.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.265+0000 7f2e9e800640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ea01f3cb0 0x7f2ea01f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:01.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.265+0000 7f2e9de00640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e840051c0 0x7f2ea015c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:01.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.265+0000 7f2e9d400640 1 -- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ea01f3cb0 msgr2=0x7f2ea01f4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:01.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.265+0000 7f2e9d400640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ea01f3cb0 0x7f2ea01f4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.265+0000 7f2e9d400640 1 -- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e840051c0 msgr2=0x7f2ea015c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.265+0000 7f2e9d400640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e840051c0 0x7f2ea015c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.265+0000 7f2e9d400640 1 -- 172.21.15.40:0/3792373115 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea015ce50 con 0x7f2ea00c9590 2024-04-26T23:04:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.265+0000 7f2e9e800640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ea01f3cb0 0x7f2ea01f4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.265+0000 7f2e9de00640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e840051c0 0x7f2ea015c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.265+0000 7f2e9d400640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea00c9590 0x7f2ea00cd3d0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009cb0 tx=0x7f2e90007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.265+0000 7f2e96000640 1 -- 172.21.15.40:0/3792373115 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90014610 con 0x7f2ea00c9590 2024-04-26T23:04:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.265+0000 7f2e96000640 1 -- 172.21.15.40:0/3792373115 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e90019070 con 0x7f2ea00c9590 2024-04-26T23:04:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.266+0000 7f2e96000640 1 -- 172.21.15.40:0/3792373115 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90014a10 con 0x7f2ea00c9590 2024-04-26T23:04:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.266+0000 7f2e9f200640 1 -- 172.21.15.40:0/3792373115 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea00c9a20 con 0x7f2ea00c9590 2024-04-26T23:04:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.266+0000 7f2e9f200640 1 -- 172.21.15.40:0/3792373115 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea00c9fe0 con 0x7f2ea00c9590 2024-04-26T23:04:01.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.267+0000 7f2e9f200640 1 -- 172.21.15.40:0/3792373115 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ea00bab60 con 0x7f2ea00c9590 2024-04-26T23:04:01.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.268+0000 7f2e96000640 1 -- 172.21.15.40:0/3792373115 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2e90014bb0 con 0x7f2ea00c9590 2024-04-26T23:04:01.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.269+0000 7f2e96000640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2e7c06c8f0 0x7f2e7c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.269+0000 7f2e96000640 1 -- 172.21.15.40:0/3792373115 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f2e9008c710 con 0x7f2ea00c9590 2024-04-26T23:04:01.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.269+0000 7f2e9de00640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2e7c06c8f0 0x7f2e7c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:01.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.271+0000 7f2e9de00640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2e7c06c8f0 0x7f2e7c06edb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2e88001200 tx=0x7f2e88000ed0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:01.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.274+0000 7f2e96000640 1 -- 172.21.15.40:0/3792373115 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2ea00bab60 con 0x7f2ea00c9590 2024-04-26T23:04:01.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.516+0000 7f2e9f200640 1 -- 172.21.15.40:0/3792373115 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ea0069370 con 0x7f2ea00c9590 2024-04-26T23:04:01.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.517+0000 7f2e96000640 1 -- 172.21.15.40:0/3792373115 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5222 (secure 0 0 0) 0x7f2e9005a8f0 con 0x7f2ea00c9590 2024-04-26T23:04:01.518 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:01.518 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":26,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1979956236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1979956236},{"type":"v1","addr":"172.21.15.103:6837","nonce":1979956236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:02:47.924696+0000","modified":"2024-04-26T23:03:24.591161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5334,"mds_1":5331,"mds_2":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"b","rank":2,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.103:6836/603576312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":603576312},{"type":"v1","addr":"172.21.15.103:6836","nonce":603576312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5331":{"gid":5331,"name":"c","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6834/716981348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":716981348},{"type":"v1","addr":"172.21.15.40:6834","nonce":716981348}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5334":{"gid":5334,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.40:6835/299800334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":299800334},{"type":"v1","addr":"172.21.15.40:6835","nonce":299800334}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5334,"qdb_cluster":[5334,5331,4975]},"id":2}]} 2024-04-26T23:04:01.518 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 26 2024-04-26T23:04:01.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.519+0000 7f2e9f200640 1 -- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2e7c06c8f0 msgr2=0x7f2e7c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:01.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.519+0000 7f2e9f200640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2e7c06c8f0 0x7f2e7c06edb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2e88001200 tx=0x7f2e88000ed0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.520+0000 7f2e9f200640 1 -- 172.21.15.40:0/3792373115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea00c9590 msgr2=0x7f2ea00cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:01.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.520+0000 7f2e9f200640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea00c9590 0x7f2ea00cd3d0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009cb0 tx=0x7f2e90007700 comp rx=0 tx=0).stop 2024-04-26T23:04:01.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.520+0000 7f2e9f200640 1 -- 172.21.15.40:0/3792373115 shutdown_connections 2024-04-26T23:04:01.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.520+0000 7f2e9f200640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2e7c06c8f0 0x7f2e7c06edb0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.520+0000 7f2e9f200640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2ea01f3cb0 0x7f2ea01f4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.520+0000 7f2e9f200640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea00c9590 0x7f2ea00cd3d0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.520+0000 7f2e9f200640 1 --2- 172.21.15.40:0/3792373115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e840051c0 0x7f2ea015c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.520+0000 7f2e9f200640 1 -- 172.21.15.40:0/3792373115 >> 172.21.15.40:0/3792373115 conn(0x7f2ea00d4d90 msgr2=0x7f2ea00b6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:01.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.520+0000 7f2e9f200640 1 -- 172.21.15.40:0/3792373115 shutdown_connections 2024-04-26T23:04:01.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.520+0000 7f2e9f200640 1 -- 172.21.15.40:0/3792373115 wait complete. 2024-04-26T23:04:01.538 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:04:01.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.740+0000 7f0af5600640 1 Processor -- start 2024-04-26T23:04:01.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.741+0000 7f0af5600640 1 -- start start 2024-04-26T23:04:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.742+0000 7f0af5600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0af0071fe0 0x7f0af00afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.742+0000 7f0af5600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0af00d0820 con 0x7f0af015ce60 2024-04-26T23:04:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.742+0000 7f0af5600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0af01614c0 con 0x7f0af00af7a0 2024-04-26T23:04:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.742+0000 7f0af5600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0af00cc450 con 0x7f0af0071fe0 2024-04-26T23:04:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.743+0000 7f0aefe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0af015ce60 0x7f0af00ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45748/0 (socket says 172.21.15.40:45748) 2024-04-26T23:04:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.743+0000 7f0aefe00640 1 -- 172.21.15.40:0/878176188 learned_addr learned my addr 172.21.15.40:0/878176188 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:01.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.743+0000 7f0af4c00640 1 --2- 172.21.15.40:0/878176188 >> v2:172.21.15.103:3300/0 conn(0x7f0af0071fe0 0x7f0af00afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:01.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.743+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3241810689 0 0) 0x7f0af00d0820 con 0x7f0af015ce60 2024-04-26T23:04:01.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.743+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ad4003520 con 0x7f0af015ce60 2024-04-26T23:04:01.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.743+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1644811058 0 0) 0x7f0ad4003520 con 0x7f0af015ce60 2024-04-26T23:04:01.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 >> v2:172.21.15.103:3300/0 conn(0x7f0af0071fe0 msgr2=0x7f0af00afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:01.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aef400640 1 --2- 172.21.15.40:0/878176188 >> v2:172.21.15.103:3300/0 conn(0x7f0af0071fe0 0x7f0af00afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 >> v1:172.21.15.103:6789/0 conn(0x7f0af00af7a0 legacy=0x7f0af0073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:01.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af0161760 con 0x7f0af015ce60 2024-04-26T23:04:01.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f0ae0001670 con 0x7f0af015ce60 2024-04-26T23:04:01.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 >> v1:172.21.15.40:6789/0 conn(0x7f0af015ce60 legacy=0x7f0af00ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:01.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aef400640 1 --2- 172.21.15.40:0/878176188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ad40039b0 0x7f0ad4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aef400640 1 --2- 172.21.15.40:0/878176188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ad4004800 0x7f0ad4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aef400640 1 --2- 172.21.15.40:0/878176188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ad40051c0 0x7f0ad400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0af00cc450 con 0x7f0ad40039b0 2024-04-26T23:04:01.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad400c160 con 0x7f0ad40051c0 2024-04-26T23:04:01.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0ad400c360 con 0x7f0ad4004800 2024-04-26T23:04:01.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aefe00640 1 --2- 172.21.15.40:0/878176188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ad40039b0 0x7f0ad4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aefe00640 1 -- 172.21.15.40:0/878176188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ad4004800 msgr2=0x7f0ad4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aefe00640 1 --2- 172.21.15.40:0/878176188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ad4004800 0x7f0ad4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aefe00640 1 -- 172.21.15.40:0/878176188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ad40051c0 msgr2=0x7f0ad400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aefe00640 1 --2- 172.21.15.40:0/878176188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ad40051c0 0x7f0ad400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0aefe00640 1 -- 172.21.15.40:0/878176188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad4003520 con 0x7f0ad40039b0 2024-04-26T23:04:01.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.744+0000 7f0af4c00640 1 -- 172.21.15.40:0/878176188 reap_dead start 2024-04-26T23:04:01.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.745+0000 7f0aefe00640 1 --2- 172.21.15.40:0/878176188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ad40039b0 0x7f0ad4003dd0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f0ae000c210 tx=0x7f0ad4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc4543835d949cd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:01.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.746+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0007d30 con 0x7f0ad40039b0 2024-04-26T23:04:01.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.746+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae0005ad0 con 0x7f0ad40039b0 2024-04-26T23:04:01.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.746+0000 7f0aef400640 1 -- 172.21.15.40:0/878176188 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0007070 con 0x7f0ad40039b0 2024-04-26T23:04:01.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.746+0000 7f0af5600640 1 -- 172.21.15.40:0/878176188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ad40039b0 msgr2=0x7f0ad4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.746+0000 7f0af5600640 1 --2- 172.21.15.40:0/878176188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ad40039b0 0x7f0ad4003dd0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f0ae000c210 tx=0x7f0ad4005830 comp rx=0 tx=0).stop 2024-04-26T23:04:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.746+0000 7f0af5600640 1 -- 172.21.15.40:0/878176188 shutdown_connections 2024-04-26T23:04:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.746+0000 7f0af5600640 1 --2- 172.21.15.40:0/878176188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ad40039b0 0x7f0ad4003dd0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.746+0000 7f0af5600640 1 -- 172.21.15.40:0/878176188 >> 172.21.15.40:0/878176188 conn(0x7f0af00d0a10 msgr2=0x7f0af00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.746+0000 7f0af5600640 1 -- 172.21.15.40:0/878176188 shutdown_connections 2024-04-26T23:04:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.746+0000 7f0af5600640 1 -- 172.21.15.40:0/878176188 wait complete. 2024-04-26T23:04:01.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.748+0000 7f0af5600640 1 Processor -- start 2024-04-26T23:04:01.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.748+0000 7f0af5600640 1 -- start start 2024-04-26T23:04:01.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.749+0000 7f0af5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ad40039b0 0x7f0af00c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.749+0000 7f0af5600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0af00c84f0 0x7f0af015fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.749+0000 7f0af5600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0af00c6810 0x7f0af00ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.749+0000 7f0af5600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0af00afde0 con 0x7f0ad40039b0 2024-04-26T23:04:01.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.749+0000 7f0af5600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0af00afc60 con 0x7f0af00c6810 2024-04-26T23:04:01.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.749+0000 7f0af5600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0af00aff60 con 0x7f0af00c84f0 2024-04-26T23:04:01.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.749+0000 7f0aefe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ad40039b0 0x7f0af00c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:01.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.749+0000 7f0aefe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ad40039b0 0x7f0af00c98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36736/0 (socket says 172.21.15.40:36736) 2024-04-26T23:04:01.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.749+0000 7f0aefe00640 1 -- 172.21.15.40:0/4126803985 learned_addr learned my addr 172.21.15.40:0/4126803985 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:01.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.750+0000 7f0aefe00640 1 -- 172.21.15.40:0/4126803985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0af00c84f0 msgr2=0x7f0af015fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:01.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.750+0000 7f0aefe00640 1 --2- 172.21.15.40:0/4126803985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0af00c84f0 0x7f0af015fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.750+0000 7f0aefe00640 1 -- 172.21.15.40:0/4126803985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0af00c6810 msgr2=0x7f0af00ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:01.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.750+0000 7f0aefe00640 1 --2- 172.21.15.40:0/4126803985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0af00c6810 0x7f0af00ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:01.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.750+0000 7f0aefe00640 1 -- 172.21.15.40:0/4126803985 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af00c5a50 con 0x7f0ad40039b0 2024-04-26T23:04:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.750+0000 7f0aefe00640 1 --2- 172.21.15.40:0/4126803985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ad40039b0 0x7f0af00c98f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0ae000c210 tx=0x7f0ae0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.750+0000 7f0aecc00640 1 -- 172.21.15.40:0/4126803985 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0007d30 con 0x7f0ad40039b0 2024-04-26T23:04:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.751+0000 7f0aecc00640 1 -- 172.21.15.40:0/4126803985 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae0042070 con 0x7f0ad40039b0 2024-04-26T23:04:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.751+0000 7f0aecc00640 1 -- 172.21.15.40:0/4126803985 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0000ed0 con 0x7f0ad40039b0 2024-04-26T23:04:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.751+0000 7f0af5600640 1 -- 172.21.15.40:0/4126803985 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0af00c5d30 con 0x7f0ad40039b0 2024-04-26T23:04:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.751+0000 7f0af5600640 1 -- 172.21.15.40:0/4126803985 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0af00c62f0 con 0x7f0ad40039b0 2024-04-26T23:04:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.752+0000 7f0af5600640 1 -- 172.21.15.40:0/4126803985 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0af0069370 con 0x7f0ad40039b0 2024-04-26T23:04:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.753+0000 7f0aecc00640 1 -- 172.21.15.40:0/4126803985 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ae003b040 con 0x7f0ad40039b0 2024-04-26T23:04:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.754+0000 7f0aecc00640 1 --2- 172.21.15.40:0/4126803985 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0acc06cac0 0x7f0acc06ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.754+0000 7f0aecc00640 1 -- 172.21.15.40:0/4126803985 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f0ae00abd40 con 0x7f0ad40039b0 2024-04-26T23:04:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.755+0000 7f0ae7400640 1 --2- 172.21.15.40:0/4126803985 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0acc06cac0 0x7f0acc06ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.756+0000 7f0ae7400640 1 --2- 172.21.15.40:0/4126803985 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0acc06cac0 0x7f0acc06ef80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0af015d3c0 tx=0x7f0ad80079d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:01.759+0000 7f0aecc00640 1 -- 172.21.15.40:0/4126803985 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0ae0079fb0 con 0x7f0ad40039b0 2024-04-26T23:04:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.009+0000 7f0af5600640 1 -- 172.21.15.40:0/4126803985 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0af00d01f0 con 0x7f0ad40039b0 2024-04-26T23:04:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.010+0000 7f0aecc00640 1 -- 172.21.15.40:0/4126803985 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5222 (secure 0 0 0) 0x7f0ae00798b0 con 0x7f0ad40039b0 2024-04-26T23:04:02.011 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:02.011 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":26,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1979956236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1979956236},{"type":"v1","addr":"172.21.15.103:6837","nonce":1979956236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:02:47.924696+0000","modified":"2024-04-26T23:03:24.591161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5334,"mds_1":5331,"mds_2":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"b","rank":2,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.103:6836/603576312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":603576312},{"type":"v1","addr":"172.21.15.103:6836","nonce":603576312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5331":{"gid":5331,"name":"c","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6834/716981348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":716981348},{"type":"v1","addr":"172.21.15.40:6834","nonce":716981348}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5334":{"gid":5334,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.40:6835/299800334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":299800334},{"type":"v1","addr":"172.21.15.40:6835","nonce":299800334}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5334,"qdb_cluster":[5334,5331,4975]},"id":2}]} 2024-04-26T23:04:02.012 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 26 2024-04-26T23:04:02.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.013+0000 7f0af5600640 1 -- 172.21.15.40:0/4126803985 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0acc06cac0 msgr2=0x7f0acc06ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.013+0000 7f0af5600640 1 --2- 172.21.15.40:0/4126803985 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0acc06cac0 0x7f0acc06ef80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0af015d3c0 tx=0x7f0ad80079d0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.013+0000 7f0af5600640 1 -- 172.21.15.40:0/4126803985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ad40039b0 msgr2=0x7f0af00c98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.013+0000 7f0af5600640 1 --2- 172.21.15.40:0/4126803985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ad40039b0 0x7f0af00c98f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0ae000c210 tx=0x7f0ae0001a40 comp rx=0 tx=0).stop 2024-04-26T23:04:02.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.013+0000 7f0af5600640 1 -- 172.21.15.40:0/4126803985 shutdown_connections 2024-04-26T23:04:02.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.013+0000 7f0af5600640 1 --2- 172.21.15.40:0/4126803985 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0acc06cac0 0x7f0acc06ef80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.013+0000 7f0af5600640 1 --2- 172.21.15.40:0/4126803985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0af00c6810 0x7f0af00ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.013+0000 7f0af5600640 1 --2- 172.21.15.40:0/4126803985 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0af00c84f0 0x7f0af015fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.013+0000 7f0af5600640 1 --2- 172.21.15.40:0/4126803985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ad40039b0 0x7f0af00c98f0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.013+0000 7f0af5600640 1 -- 172.21.15.40:0/4126803985 >> 172.21.15.40:0/4126803985 conn(0x7f0af00d0a10 msgr2=0x7f0af00cd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:02.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.013+0000 7f0af5600640 1 -- 172.21.15.40:0/4126803985 shutdown_connections 2024-04-26T23:04:02.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.014+0000 7f0af5600640 1 -- 172.21.15.40:0/4126803985 wait complete. 2024-04-26T23:04:02.032 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:04:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.234+0000 7fb047e00640 1 Processor -- start 2024-04-26T23:04:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.234+0000 7fb047e00640 1 -- start start 2024-04-26T23:04:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.236+0000 7fb047e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb0480cc980 0x7fb0480cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.236+0000 7fb047e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0480cce10 con 0x7fb0480c6f10 2024-04-26T23:04:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.236+0000 7fb047e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0480cd0f0 con 0x7fb0480cc230 2024-04-26T23:04:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.236+0000 7fb047e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb0480c72f0 con 0x7fb0480cc980 2024-04-26T23:04:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.236+0000 7fb046000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb0480cc980 0x7fb0480cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.236+0000 7fb046a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb0480c6f10 0x7fb0480c5ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45760/0 (socket says 172.21.15.40:45760) 2024-04-26T23:04:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.236+0000 7fb046a00640 1 -- 172.21.15.40:0/103849908 learned_addr learned my addr 172.21.15.40:0/103849908 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.237+0000 7fb046000640 1 -- 172.21.15.40:0/103849908 >> v1:172.21.15.103:6789/0 conn(0x7fb0480cc230 legacy=0x7fb048158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.237+0000 7fb046000640 1 -- 172.21.15.40:0/103849908 >> v1:172.21.15.40:6789/0 conn(0x7fb0480c6f10 legacy=0x7fb0480c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.237+0000 7fb045600640 1 -- 172.21.15.40:0/103849908 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3247369734 0 0) 0x7fb0480cce10 con 0x7fb0480c6f10 2024-04-26T23:04:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.237+0000 7fb046000640 1 -- 172.21.15.40:0/103849908 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0480cb4f0 con 0x7fb0480cc980 2024-04-26T23:04:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.238+0000 7fb046000640 1 --2- 172.21.15.40:0/103849908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0480cc980 0x7fb0480cbcf0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb0480c7470 tx=0x7fb03402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291f35885b6abeca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.239+0000 7fb045600640 1 -- 172.21.15.40:0/103849908 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034037070 con 0x7fb0480cc980 2024-04-26T23:04:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.239+0000 7fb045600640 1 -- 172.21.15.40:0/103849908 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb034032440 con 0x7fb0480cc980 2024-04-26T23:04:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.239+0000 7fb045600640 1 -- 172.21.15.40:0/103849908 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034032810 con 0x7fb0480cc980 2024-04-26T23:04:02.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.239+0000 7fb047e00640 1 -- 172.21.15.40:0/103849908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0480cc980 msgr2=0x7fb0480cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.239+0000 7fb047e00640 1 --2- 172.21.15.40:0/103849908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0480cc980 0x7fb0480cbcf0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb0480c7470 tx=0x7fb03402a180 comp rx=0 tx=0).stop 2024-04-26T23:04:02.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.239+0000 7fb047e00640 1 -- 172.21.15.40:0/103849908 shutdown_connections 2024-04-26T23:04:02.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.239+0000 7fb047e00640 1 --2- 172.21.15.40:0/103849908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0480cc980 0x7fb0480cbcf0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.239+0000 7fb047e00640 1 -- 172.21.15.40:0/103849908 >> 172.21.15.40:0/103849908 conn(0x7fb0480d0a10 msgr2=0x7fb0480d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:02.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.239+0000 7fb047e00640 1 -- 172.21.15.40:0/103849908 shutdown_connections 2024-04-26T23:04:02.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.239+0000 7fb047e00640 1 -- 172.21.15.40:0/103849908 wait complete. 2024-04-26T23:04:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.241+0000 7fb047e00640 1 Processor -- start 2024-04-26T23:04:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.241+0000 7fb047e00640 1 -- start start 2024-04-26T23:04:02.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.242+0000 7fb047e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0480c6f10 0x7fb048152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:02.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.242+0000 7fb047e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0480cc230 0x7fb04815d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:02.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.242+0000 7fb047e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0481534e0 0x7fb0480b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.242+0000 7fb047e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb0480c61e0 con 0x7fb0481534e0 2024-04-26T23:04:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.242+0000 7fb047e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb0480cb860 con 0x7fb0480c6f10 2024-04-26T23:04:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.242+0000 7fb047e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb0480c6360 con 0x7fb0480cc230 2024-04-26T23:04:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.242+0000 7fb047400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0481534e0 0x7fb0480b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.242+0000 7fb047400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0481534e0 0x7fb0480b4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36756/0 (socket says 172.21.15.40:36756) 2024-04-26T23:04:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.242+0000 7fb047400640 1 -- 172.21.15.40:0/942013543 learned_addr learned my addr 172.21.15.40:0/942013543 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.243+0000 7fb047400640 1 -- 172.21.15.40:0/942013543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0480cc230 msgr2=0x7fb04815d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.243+0000 7fb047400640 1 --2- 172.21.15.40:0/942013543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0480cc230 0x7fb04815d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.243+0000 7fb047400640 1 -- 172.21.15.40:0/942013543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0480c6f10 msgr2=0x7fb048152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.243+0000 7fb047400640 1 --2- 172.21.15.40:0/942013543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0480c6f10 0x7fb048152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.243+0000 7fb047400640 1 -- 172.21.15.40:0/942013543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb048153d30 con 0x7fb0481534e0 2024-04-26T23:04:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.243+0000 7fb047400640 1 --2- 172.21.15.40:0/942013543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0481534e0 0x7fb0480b4e60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb04000c7a0 tx=0x7fb04000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.243+0000 7fb03ea00640 1 -- 172.21.15.40:0/942013543 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb040003b70 con 0x7fb0481534e0 2024-04-26T23:04:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.243+0000 7fb03ea00640 1 -- 172.21.15.40:0/942013543 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb040003d30 con 0x7fb0481534e0 2024-04-26T23:04:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.243+0000 7fb03ea00640 1 -- 172.21.15.40:0/942013543 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04000aa90 con 0x7fb0481534e0 2024-04-26T23:04:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.243+0000 7fb047e00640 1 -- 172.21.15.40:0/942013543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0480b4a20 con 0x7fb0481534e0 2024-04-26T23:04:02.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.244+0000 7fb047e00640 1 -- 172.21.15.40:0/942013543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0480c8e70 con 0x7fb0481534e0 2024-04-26T23:04:02.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.245+0000 7fb047e00640 1 -- 172.21.15.40:0/942013543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb048069370 con 0x7fb0481534e0 2024-04-26T23:04:02.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.246+0000 7fb03ea00640 1 -- 172.21.15.40:0/942013543 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb04000ac30 con 0x7fb0481534e0 2024-04-26T23:04:02.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.247+0000 7fb03ea00640 1 --2- 172.21.15.40:0/942013543 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb02806c8f0 0x7fb02806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:02.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.247+0000 7fb03ea00640 1 -- 172.21.15.40:0/942013543 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fb04008bca0 con 0x7fb0481534e0 2024-04-26T23:04:02.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.247+0000 7fb046a00640 1 --2- 172.21.15.40:0/942013543 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb02806c8f0 0x7fb02806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:02.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.248+0000 7fb046a00640 1 --2- 172.21.15.40:0/942013543 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb02806c8f0 0x7fb02806edb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb038007fd0 tx=0x7fb038007900 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:02.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.251+0000 7fb03ea00640 1 -- 172.21.15.40:0/942013543 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fb040059f20 con 0x7fb0481534e0 2024-04-26T23:04:02.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.445+0000 7fb047e00640 1 -- 172.21.15.40:0/942013543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb0480cae30 con 0x7fb0481534e0 2024-04-26T23:04:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.448+0000 7fb03ea00640 1 -- 172.21.15.40:0/942013543 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+16955 (secure 0 0 0) 0x7fb040059820 con 0x7fb0481534e0 2024-04-26T23:04:02.448 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:02.449 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":41,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:03:47.741302+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:02:45.864120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-26T23:02:46.869290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:04:02.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.450+0000 7fb047e00640 1 -- 172.21.15.40:0/942013543 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb02806c8f0 msgr2=0x7fb02806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.450+0000 7fb047e00640 1 --2- 172.21.15.40:0/942013543 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb02806c8f0 0x7fb02806edb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb038007fd0 tx=0x7fb038007900 comp rx=0 tx=0).stop 2024-04-26T23:04:02.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.450+0000 7fb047e00640 1 -- 172.21.15.40:0/942013543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0481534e0 msgr2=0x7fb0480b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.450+0000 7fb047e00640 1 --2- 172.21.15.40:0/942013543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0481534e0 0x7fb0480b4e60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb04000c7a0 tx=0x7fb04000cc60 comp rx=0 tx=0).stop 2024-04-26T23:04:02.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.450+0000 7fb047e00640 1 -- 172.21.15.40:0/942013543 shutdown_connections 2024-04-26T23:04:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.450+0000 7fb047e00640 1 --2- 172.21.15.40:0/942013543 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb02806c8f0 0x7fb02806edb0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.450+0000 7fb047e00640 1 --2- 172.21.15.40:0/942013543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0481534e0 0x7fb0480b4e60 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.450+0000 7fb047e00640 1 --2- 172.21.15.40:0/942013543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb0480cc230 0x7fb04815d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.450+0000 7fb047e00640 1 --2- 172.21.15.40:0/942013543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0480c6f10 0x7fb048152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.450+0000 7fb047e00640 1 -- 172.21.15.40:0/942013543 >> 172.21.15.40:0/942013543 conn(0x7fb0480d0a10 msgr2=0x7fb04815d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.450+0000 7fb047e00640 1 -- 172.21.15.40:0/942013543 shutdown_connections 2024-04-26T23:04:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.451+0000 7fb047e00640 1 -- 172.21.15.40:0/942013543 wait complete. 2024-04-26T23:04:02.471 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-26T23:04:02.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.675+0000 7f71bd400640 1 Processor -- start 2024-04-26T23:04:02.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.676+0000 7f71bd400640 1 -- start start 2024-04-26T23:04:02.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.677+0000 7f71bd400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f71b80b6220 0x7f71b80d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:02.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.677+0000 7f71bd400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f71b80d08d0 0x7f71b80d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:02.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.677+0000 7f71bd400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71b8162c40 con 0x7f71b80d3820 2024-04-26T23:04:02.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.677+0000 7f71bd400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f71b80d3c60 con 0x7f71b80b6220 2024-04-26T23:04:02.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.677+0000 7f71bd400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f71b80d2d80 con 0x7f71b80d08d0 2024-04-26T23:04:02.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.677+0000 7f71b7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f71b80b6220 0x7f71b80d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:02.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.678+0000 7f71b7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f71b80b6220 0x7f71b80d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36770/0 (socket says 172.21.15.40:36770) 2024-04-26T23:04:02.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.678+0000 7f71b7e00640 1 -- 172.21.15.40:0/1645935050 learned_addr learned my addr 172.21.15.40:0/1645935050 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:02.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.678+0000 7f71b7400640 1 --2- 172.21.15.40:0/1645935050 >> v2:172.21.15.103:3300/0 conn(0x7f71b80d08d0 0x7f71b80d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:02.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.678+0000 7f71b7e00640 1 -- 172.21.15.40:0/1645935050 >> v2:172.21.15.103:3300/0 conn(0x7f71b80d08d0 msgr2=0x7f71b80d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:02.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.678+0000 7f71b7e00640 1 --2- 172.21.15.40:0/1645935050 >> v2:172.21.15.103:3300/0 conn(0x7f71b80d08d0 0x7f71b80d32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.678+0000 7f71b7e00640 1 -- 172.21.15.40:0/1645935050 >> v1:172.21.15.40:6789/0 conn(0x7f71b80d3820 legacy=0x7f71b80b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.678+0000 7f71b7e00640 1 -- 172.21.15.40:0/1645935050 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b80b7c80 con 0x7f71b80b6220 2024-04-26T23:04:02.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.678+0000 7f71b7400640 1 --2- 172.21.15.40:0/1645935050 >> v2:172.21.15.103:3300/0 conn(0x7f71b80d08d0 0x7f71b80d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:02.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.678+0000 7f71b6a00640 1 -- 172.21.15.40:0/1645935050 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3712701491 0 0) 0x7f71b8162c40 con 0x7f71b80d3820 2024-04-26T23:04:02.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.679+0000 7f71b7e00640 1 --2- 172.21.15.40:0/1645935050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71b80b6220 0x7f71b80d0390 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f71b808f300 tx=0x7f71a802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f04fd688529e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:02.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.679+0000 7f71b6a00640 1 -- 172.21.15.40:0/1645935050 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a8037070 con 0x7f71b80b6220 2024-04-26T23:04:02.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.680+0000 7f71b6a00640 1 -- 172.21.15.40:0/1645935050 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71a8032440 con 0x7f71b80b6220 2024-04-26T23:04:02.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.680+0000 7f71b6a00640 1 -- 172.21.15.40:0/1645935050 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a8032810 con 0x7f71b80b6220 2024-04-26T23:04:02.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.680+0000 7f71bd400640 1 -- 172.21.15.40:0/1645935050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71b80b6220 msgr2=0x7f71b80d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.680+0000 7f71bd400640 1 --2- 172.21.15.40:0/1645935050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71b80b6220 0x7f71b80d0390 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f71b808f300 tx=0x7f71a802a270 comp rx=0 tx=0).stop 2024-04-26T23:04:02.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.680+0000 7f71bd400640 1 -- 172.21.15.40:0/1645935050 shutdown_connections 2024-04-26T23:04:02.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.680+0000 7f71bd400640 1 --2- 172.21.15.40:0/1645935050 >> v2:172.21.15.103:3300/0 conn(0x7f71b80d08d0 0x7f71b80d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.680+0000 7f71bd400640 1 --2- 172.21.15.40:0/1645935050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71b80b6220 0x7f71b80d0390 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.680+0000 7f71bd400640 1 -- 172.21.15.40:0/1645935050 >> 172.21.15.40:0/1645935050 conn(0x7f71b80d4d90 msgr2=0x7f71b80d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:02.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.680+0000 7f71bd400640 1 -- 172.21.15.40:0/1645935050 shutdown_connections 2024-04-26T23:04:02.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.680+0000 7f71bd400640 1 -- 172.21.15.40:0/1645935050 wait complete. 2024-04-26T23:04:02.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.682+0000 7f71bd400640 1 Processor -- start 2024-04-26T23:04:02.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.682+0000 7f71bd400640 1 -- start start 2024-04-26T23:04:02.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.683+0000 7f71bd400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71b80b6220 0x7f71b80a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:02.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.683+0000 7f71bd400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71b80d08d0 0x7f71b8157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:02.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.683+0000 7f71bd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71b80d3820 0x7f71b8158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:02.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.683+0000 7f71bd400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f71b80b8520 con 0x7f71b80d3820 2024-04-26T23:04:02.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.683+0000 7f71bd400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f71b80b8240 con 0x7f71b80b6220 2024-04-26T23:04:02.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.683+0000 7f71bd400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f71b80b9d30 con 0x7f71b80d08d0 2024-04-26T23:04:02.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.683+0000 7f71bca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71b80d3820 0x7f71b8158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:02.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.683+0000 7f71bca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71b80d3820 0x7f71b8158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36786/0 (socket says 172.21.15.40:36786) 2024-04-26T23:04:02.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.683+0000 7f71bca00640 1 -- 172.21.15.40:0/550051060 learned_addr learned my addr 172.21.15.40:0/550051060 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:02.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.684+0000 7f71bca00640 1 -- 172.21.15.40:0/550051060 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71b80d08d0 msgr2=0x7f71b8157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.684+0000 7f71bca00640 1 --2- 172.21.15.40:0/550051060 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71b80d08d0 0x7f71b8157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.684+0000 7f71bca00640 1 -- 172.21.15.40:0/550051060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71b80b6220 msgr2=0x7f71b80a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.684+0000 7f71bca00640 1 --2- 172.21.15.40:0/550051060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71b80b6220 0x7f71b80a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.684+0000 7f71bca00640 1 -- 172.21.15.40:0/550051060 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b80b5320 con 0x7f71b80d3820 2024-04-26T23:04:02.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.684+0000 7f71bca00640 1 --2- 172.21.15.40:0/550051060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71b80d3820 0x7f71b8158370 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f71b000c3e0 tx=0x7f71b000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:02.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.684+0000 7f71afe00640 1 -- 172.21.15.40:0/550051060 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0017070 con 0x7f71b80d3820 2024-04-26T23:04:02.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.684+0000 7f71afe00640 1 -- 172.21.15.40:0/550051060 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71b00035f0 con 0x7f71b80d3820 2024-04-26T23:04:02.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.684+0000 7f71afe00640 1 -- 172.21.15.40:0/550051060 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b00039e0 con 0x7f71b80d3820 2024-04-26T23:04:02.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.684+0000 7f71bd400640 1 -- 172.21.15.40:0/550051060 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b809f0e0 con 0x7f71b80d3820 2024-04-26T23:04:02.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.684+0000 7f71bd400640 1 -- 172.21.15.40:0/550051060 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b80a1e80 con 0x7f71b80d3820 2024-04-26T23:04:02.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.685+0000 7f71bd400640 1 -- 172.21.15.40:0/550051060 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71b8069370 con 0x7f71b80d3820 2024-04-26T23:04:02.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.686+0000 7f71afe00640 1 -- 172.21.15.40:0/550051060 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f71b001a070 con 0x7f71b80d3820 2024-04-26T23:04:02.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.687+0000 7f71afe00640 1 --2- 172.21.15.40:0/550051060 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f719406c920 0x7f719406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:02.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.687+0000 7f71afe00640 1 -- 172.21.15.40:0/550051060 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f71b008af80 con 0x7f71b80d3820 2024-04-26T23:04:02.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.688+0000 7f71b7e00640 1 --2- 172.21.15.40:0/550051060 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f719406c920 0x7f719406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:02.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.689+0000 7f71b7e00640 1 --2- 172.21.15.40:0/550051060 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f719406c920 0x7f719406ede0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f71a8034000 tx=0x7f71a802c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:02.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.692+0000 7f71afe00640 1 -- 172.21.15.40:0/550051060 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f71b00591f0 con 0x7f71b80d3820 2024-04-26T23:04:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.930+0000 7f71bd400640 1 -- 172.21.15.40:0/550051060 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f71b80d30c0 con 0x7f71b80d3820 2024-04-26T23:04:02.932 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:04:02.932+0000 7f36fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T23:04:02.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.936+0000 7f71afe00640 1 -- 172.21.15.40:0/550051060 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) ==== 161+0+0 (secure 0 0 0) 0x7f71b0058af0 con 0x7f71b80d3820 2024-04-26T23:04:02.936 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T23:04:02.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.938+0000 7f71bd400640 1 -- 172.21.15.40:0/550051060 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f719406c920 msgr2=0x7f719406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.938+0000 7f71bd400640 1 --2- 172.21.15.40:0/550051060 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f719406c920 0x7f719406ede0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f71a8034000 tx=0x7f71a802c000 comp rx=0 tx=0).stop 2024-04-26T23:04:02.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.938+0000 7f71bd400640 1 -- 172.21.15.40:0/550051060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71b80d3820 msgr2=0x7f71b8158370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:02.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.938+0000 7f71bd400640 1 --2- 172.21.15.40:0/550051060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71b80d3820 0x7f71b8158370 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f71b000c3e0 tx=0x7f71b000c8a0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.939+0000 7f71bd400640 1 -- 172.21.15.40:0/550051060 shutdown_connections 2024-04-26T23:04:02.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.939+0000 7f71bd400640 1 --2- 172.21.15.40:0/550051060 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f719406c920 0x7f719406ede0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.939+0000 7f71bd400640 1 --2- 172.21.15.40:0/550051060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71b80d3820 0x7f71b8158370 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.939+0000 7f71bd400640 1 --2- 172.21.15.40:0/550051060 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71b80d08d0 0x7f71b8157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.939+0000 7f71bd400640 1 --2- 172.21.15.40:0/550051060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71b80b6220 0x7f71b80a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:02.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.939+0000 7f71bd400640 1 -- 172.21.15.40:0/550051060 >> 172.21.15.40:0/550051060 conn(0x7f71b80d4d90 msgr2=0x7f71b80cf860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:02.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.940+0000 7f71bd400640 1 -- 172.21.15.40:0/550051060 shutdown_connections 2024-04-26T23:04:02.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:02.940+0000 7f71bd400640 1 -- 172.21.15.40:0/550051060 wait complete. 2024-04-26T23:04:02.961 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T23:04:03.019 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:04:03.037 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-04-26T23:04:03.046 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-26T23:04:03.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.140+0000 7fcff9000640 1 Processor -- start 2024-04-26T23:04:03.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.141+0000 7fcff9000640 1 -- start start 2024-04-26T23:04:03.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.142+0000 7fcff9000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcff4160a50 0x7fcff4073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:03.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.142+0000 7fcff9000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcff4161880 con 0x7fcff406dd40 2024-04-26T23:04:03.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.142+0000 7fcff9000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcff40cff80 con 0x7fcff40cfaf0 2024-04-26T23:04:03.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.142+0000 7fcff9000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcff4071ce0 con 0x7fcff4160a50 2024-04-26T23:04:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.142+0000 7fcff3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcff406dd40 0x7fcff4072730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45774/0 (socket says 172.21.15.40:45774) 2024-04-26T23:04:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.142+0000 7fcff3e00640 1 -- 172.21.15.40:0/155997536 learned_addr learned my addr 172.21.15.40:0/155997536 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2636574465 0 0) 0x7fcff4161880 con 0x7fcff406dd40 2024-04-26T23:04:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfd8003520 con 0x7fcff406dd40 2024-04-26T23:04:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff3400640 1 --2- 172.21.15.40:0/155997536 >> v2:172.21.15.103:3300/0 conn(0x7fcff4160a50 0x7fcff4073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1047577940 0 0) 0x7fcfd8003520 con 0x7fcff406dd40 2024-04-26T23:04:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 >> v2:172.21.15.103:3300/0 conn(0x7fcff4160a50 msgr2=0x7fcff4073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 --2- 172.21.15.40:0/155997536 >> v2:172.21.15.103:3300/0 conn(0x7fcff4160a50 0x7fcff4073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 >> v1:172.21.15.103:6789/0 conn(0x7fcff40cfaf0 legacy=0x7fcff4160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff40d0260 con 0x7fcff406dd40 2024-04-26T23:04:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fcfec001670 con 0x7fcff406dd40 2024-04-26T23:04:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 >> v1:172.21.15.40:6789/0 conn(0x7fcff406dd40 legacy=0x7fcff4072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 --2- 172.21.15.40:0/155997536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfd8003a00 0x7fcfd8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 --2- 172.21.15.40:0/155997536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd8004800 0x7fcfd8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 --2- 172.21.15.40:0/155997536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfd8005210 0x7fcfd800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcff4071ce0 con 0x7fcfd8004800 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd800c160 con 0x7fcfd8005210 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.143+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd800c360 con 0x7fcfd8003a00 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.144+0000 7fcff2a00640 1 --2- 172.21.15.40:0/155997536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd8004800 0x7fcfd8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.144+0000 7fcff3e00640 1 --2- 172.21.15.40:0/155997536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfd8005210 0x7fcfd800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.144+0000 7fcff3400640 1 --2- 172.21.15.40:0/155997536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfd8003a00 0x7fcfd8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.144+0000 7fcff2a00640 1 -- 172.21.15.40:0/155997536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfd8003a00 msgr2=0x7fcfd8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.144+0000 7fcff2a00640 1 --2- 172.21.15.40:0/155997536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfd8003a00 0x7fcfd8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.144+0000 7fcff2a00640 1 -- 172.21.15.40:0/155997536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfd8005210 msgr2=0x7fcfd800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.144+0000 7fcff2a00640 1 --2- 172.21.15.40:0/155997536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfd8005210 0x7fcfd800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.144+0000 7fcff3e00640 1 -- 172.21.15.40:0/155997536 reap_dead start 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.144+0000 7fcff2a00640 1 -- 172.21.15.40:0/155997536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd8003520 con 0x7fcfd8004800 2024-04-26T23:04:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.144+0000 7fcff3400640 1 --2- 172.21.15.40:0/155997536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfd8003a00 0x7fcfd8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:03.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.145+0000 7fcff2a00640 1 --2- 172.21.15.40:0/155997536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd8004800 0x7fcfd8004c80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc001750 tx=0x7fcfd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c490d4e86ed3ded3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:03.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.145+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc001370 con 0x7fcfd8004800 2024-04-26T23:04:03.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.146+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfdc0052d0 con 0x7fcfd8004800 2024-04-26T23:04:03.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.146+0000 7fcff2000640 1 -- 172.21.15.40:0/155997536 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc02abc0 con 0x7fcfd8004800 2024-04-26T23:04:03.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.146+0000 7fcff9000640 1 -- 172.21.15.40:0/155997536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd8004800 msgr2=0x7fcfd8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:03.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.146+0000 7fcff9000640 1 --2- 172.21.15.40:0/155997536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd8004800 0x7fcfd8004c80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc001750 tx=0x7fcfd8005830 comp rx=0 tx=0).stop 2024-04-26T23:04:03.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.146+0000 7fcff9000640 1 -- 172.21.15.40:0/155997536 shutdown_connections 2024-04-26T23:04:03.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.146+0000 7fcff9000640 1 --2- 172.21.15.40:0/155997536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd8004800 0x7fcfd8004c80 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:03.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.146+0000 7fcff9000640 1 -- 172.21.15.40:0/155997536 >> 172.21.15.40:0/155997536 conn(0x7fcff40b2720 msgr2=0x7fcff40b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:03.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.146+0000 7fcff9000640 1 -- 172.21.15.40:0/155997536 shutdown_connections 2024-04-26T23:04:03.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.146+0000 7fcff9000640 1 -- 172.21.15.40:0/155997536 wait complete. 2024-04-26T23:04:03.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.148+0000 7fcff9000640 1 Processor -- start 2024-04-26T23:04:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.148+0000 7fcff9000640 1 -- start start 2024-04-26T23:04:03.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff9000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd8004800 0x7fcff40b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:03.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff9000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcff40b5010 0x7fcff40ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:03.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff9000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcff40aee00 0x7fcff40af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:03.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff9000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcff406e1f0 con 0x7fcfd8004800 2024-04-26T23:04:03.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff9000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcff40722a0 con 0x7fcff40b5010 2024-04-26T23:04:03.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff9000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcff41468f0 con 0x7fcff40aee00 2024-04-26T23:04:03.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd8004800 0x7fcff40b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:03.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd8004800 0x7fcff40b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36862/0 (socket says 172.21.15.40:36862) 2024-04-26T23:04:03.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff3400640 1 -- 172.21.15.40:0/3640641855 learned_addr learned my addr 172.21.15.40:0/3640641855 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:03.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff3400640 1 -- 172.21.15.40:0/3640641855 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcff40aee00 msgr2=0x7fcff40af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:03.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff3400640 1 --2- 172.21.15.40:0/3640641855 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcff40aee00 0x7fcff40af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:03.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff3400640 1 -- 172.21.15.40:0/3640641855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcff40b5010 msgr2=0x7fcff40ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:03.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff3400640 1 --2- 172.21.15.40:0/3640641855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcff40b5010 0x7fcff40ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:03.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.149+0000 7fcff3400640 1 -- 172.21.15.40:0/3640641855 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff415d490 con 0x7fcfd8004800 2024-04-26T23:04:03.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.150+0000 7fcff3400640 1 --2- 172.21.15.40:0/3640641855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd8004800 0x7fcff40b4ad0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4007e20 tx=0x7fcfe400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:03.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.150+0000 7fcfeb400640 1 -- 172.21.15.40:0/3640641855 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4018020 con 0x7fcfd8004800 2024-04-26T23:04:03.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.150+0000 7fcfeb400640 1 -- 172.21.15.40:0/3640641855 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfe40145f0 con 0x7fcfd8004800 2024-04-26T23:04:03.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.150+0000 7fcfeb400640 1 -- 172.21.15.40:0/3640641855 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe40149a0 con 0x7fcfd8004800 2024-04-26T23:04:03.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.150+0000 7fcff9000640 1 -- 172.21.15.40:0/3640641855 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff415d730 con 0x7fcfd8004800 2024-04-26T23:04:03.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.150+0000 7fcff9000640 1 -- 172.21.15.40:0/3640641855 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff415dcf0 con 0x7fcfd8004800 2024-04-26T23:04:03.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.151+0000 7fcff9000640 1 -- 172.21.15.40:0/3640641855 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcff4069370 con 0x7fcfd8004800 2024-04-26T23:04:03.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.152+0000 7fcfeb400640 1 -- 172.21.15.40:0/3640641855 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcfe401b020 con 0x7fcfd8004800 2024-04-26T23:04:03.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.153+0000 7fcfeb400640 1 --2- 172.21.15.40:0/3640641855 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcfd006c920 0x7fcfd006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:03.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.153+0000 7fcfeb400640 1 -- 172.21.15.40:0/3640641855 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6388+0+0 (secure 0 0 0) 0x7fcfe408c850 con 0x7fcfd8004800 2024-04-26T23:04:03.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.154+0000 7fcff2a00640 1 --2- 172.21.15.40:0/3640641855 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcfd006c920 0x7fcfd006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:03.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.155+0000 7fcff2a00640 1 --2- 172.21.15.40:0/3640641855 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcfd006c920 0x7fcfd006ede0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc032000 tx=0x7fcfdc02c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:03.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.157+0000 7fcfeb400640 1 -- 172.21.15.40:0/3640641855 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fcfe4022020 con 0x7fcfd8004800 2024-04-26T23:04:03.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.405+0000 7fcff9000640 1 -- 172.21.15.40:0/3640641855 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fcff4160420 con 0x7fcfd8004800 2024-04-26T23:04:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.939+0000 7fcfeb400640 1 -- 172.21.15.40:0/3640641855 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) ==== 104+0+0 (secure 0 0 0) 0x7fcfe405a9c0 con 0x7fcfd8004800 2024-04-26T23:04:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.942+0000 7fcff9000640 1 -- 172.21.15.40:0/3640641855 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcfd006c920 msgr2=0x7fcfd006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.942+0000 7fcff9000640 1 --2- 172.21.15.40:0/3640641855 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcfd006c920 0x7fcfd006ede0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc032000 tx=0x7fcfdc02c000 comp rx=0 tx=0).stop 2024-04-26T23:04:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.943+0000 7fcff9000640 1 -- 172.21.15.40:0/3640641855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd8004800 msgr2=0x7fcff40b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.943+0000 7fcff9000640 1 --2- 172.21.15.40:0/3640641855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd8004800 0x7fcff40b4ad0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4007e20 tx=0x7fcfe400f520 comp rx=0 tx=0).stop 2024-04-26T23:04:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.943+0000 7fcff9000640 1 -- 172.21.15.40:0/3640641855 shutdown_connections 2024-04-26T23:04:03.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.943+0000 7fcff9000640 1 --2- 172.21.15.40:0/3640641855 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcfd006c920 0x7fcfd006ede0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.943+0000 7fcff9000640 1 --2- 172.21.15.40:0/3640641855 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcff40aee00 0x7fcff40af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.943+0000 7fcff9000640 1 --2- 172.21.15.40:0/3640641855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcff40b5010 0x7fcff40ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.943+0000 7fcff9000640 1 --2- 172.21.15.40:0/3640641855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd8004800 0x7fcff40b4ad0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.943+0000 7fcff9000640 1 -- 172.21.15.40:0/3640641855 >> 172.21.15.40:0/3640641855 conn(0x7fcff40b2720 msgr2=0x7fcff40b2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.943+0000 7fcff9000640 1 -- 172.21.15.40:0/3640641855 shutdown_connections 2024-04-26T23:04:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:03.943+0000 7fcff9000640 1 -- 172.21.15.40:0/3640641855 wait complete. 2024-04-26T23:04:03.967 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T23:04:04.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.170+0000 7fcf0d600640 1 Processor -- start 2024-04-26T23:04:04.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.170+0000 7fcf0d600640 1 -- start start 2024-04-26T23:04:04.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.172+0000 7fcf0d600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf0806e700 0x7fcf080b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:04.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.172+0000 7fcf0d600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf0815d8d0 con 0x7fcf0815a4a0 2024-04-26T23:04:04.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.172+0000 7fcf0d600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf0815dbb0 con 0x7fcf0806eae0 2024-04-26T23:04:04.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.172+0000 7fcf0d600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fcf0815de90 con 0x7fcf0806e700 2024-04-26T23:04:04.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.172+0000 7fcf06a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf0806e700 0x7fcf080b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:04.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.172+0000 7fcf07400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcf0815a4a0 0x7fcf08160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45782/0 (socket says 172.21.15.40:45782) 2024-04-26T23:04:04.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.172+0000 7fcf07400640 1 -- 172.21.15.40:0/1656078243 learned_addr learned my addr 172.21.15.40:0/1656078243 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:04.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.173+0000 7fcf05600640 1 -- 172.21.15.40:0/1656078243 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1254247466 0 0) 0x7fcf0815d8d0 con 0x7fcf0815a4a0 2024-04-26T23:04:04.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.173+0000 7fcf05600640 1 -- 172.21.15.40:0/1656078243 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceec003520 con 0x7fcf0815a4a0 2024-04-26T23:04:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.173+0000 7fcf06a00640 1 -- 172.21.15.40:0/1656078243 >> v1:172.21.15.103:6789/0 conn(0x7fcf0806eae0 legacy=0x7fcf080ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.173+0000 7fcf06a00640 1 -- 172.21.15.40:0/1656078243 >> v1:172.21.15.40:6789/0 conn(0x7fcf0815a4a0 legacy=0x7fcf08160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.173+0000 7fcf06a00640 1 -- 172.21.15.40:0/1656078243 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0815e170 con 0x7fcf0806e700 2024-04-26T23:04:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.175+0000 7fcf06a00640 1 --2- 172.21.15.40:0/1656078243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf0806e700 0x7fcf080b3450 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fcef8009dc0 tx=0x7fcef802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca9aa11bb15e273 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:04.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.175+0000 7fcf05600640 1 -- 172.21.15.40:0/1656078243 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef8007740 con 0x7fcf0806e700 2024-04-26T23:04:04.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.175+0000 7fcf05600640 1 -- 172.21.15.40:0/1656078243 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef80078e0 con 0x7fcf0806e700 2024-04-26T23:04:04.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.175+0000 7fcf05600640 1 -- 172.21.15.40:0/1656078243 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef8007cb0 con 0x7fcf0806e700 2024-04-26T23:04:04.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.176+0000 7fcf0d600640 1 -- 172.21.15.40:0/1656078243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf0806e700 msgr2=0x7fcf080b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:04.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.176+0000 7fcf0d600640 1 --2- 172.21.15.40:0/1656078243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf0806e700 0x7fcf080b3450 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fcef8009dc0 tx=0x7fcef802a730 comp rx=0 tx=0).stop 2024-04-26T23:04:04.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.176+0000 7fcf0d600640 1 -- 172.21.15.40:0/1656078243 shutdown_connections 2024-04-26T23:04:04.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.176+0000 7fcf0d600640 1 --2- 172.21.15.40:0/1656078243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf0806e700 0x7fcf080b3450 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:04.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.176+0000 7fcf0d600640 1 -- 172.21.15.40:0/1656078243 >> 172.21.15.40:0/1656078243 conn(0x7fcf0806f310 msgr2=0x7fcf0806f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:04.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.176+0000 7fcf0d600640 1 -- 172.21.15.40:0/1656078243 shutdown_connections 2024-04-26T23:04:04.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.176+0000 7fcf0d600640 1 -- 172.21.15.40:0/1656078243 wait complete. 2024-04-26T23:04:04.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.178+0000 7fcf0d600640 1 Processor -- start 2024-04-26T23:04:04.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.178+0000 7fcf0d600640 1 -- start start 2024-04-26T23:04:04.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.179+0000 7fcf0d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf0806e700 0x7fcf08156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:04.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.179+0000 7fcf0d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf0806eae0 0x7fcf0815b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:04.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.179+0000 7fcf0d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcf0815a4a0 0x7fcf0815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:04.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.179+0000 7fcf0d600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0815d600 con 0x7fcf0806eae0 2024-04-26T23:04:04.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.179+0000 7fcf0d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0815d480 con 0x7fcf0806e700 2024-04-26T23:04:04.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.179+0000 7fcf0d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcf08161540 con 0x7fcf0815a4a0 2024-04-26T23:04:04.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.179+0000 7fcf06000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf0806eae0 0x7fcf0815b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:04.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.180+0000 7fcf06000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf0806eae0 0x7fcf0815b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36880/0 (socket says 172.21.15.40:36880) 2024-04-26T23:04:04.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.180+0000 7fcf06000640 1 -- 172.21.15.40:0/247784318 learned_addr learned my addr 172.21.15.40:0/247784318 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:04.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.180+0000 7fcf07400640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcf0815a4a0 0x7fcf0815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:04.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.180+0000 7fcf06a00640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf0806e700 0x7fcf08156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:04.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.180+0000 7fcf06000640 1 -- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcf0815a4a0 msgr2=0x7fcf0815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:04.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.180+0000 7fcf06000640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcf0815a4a0 0x7fcf0815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:04.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.180+0000 7fcf06000640 1 -- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf0806e700 msgr2=0x7fcf08156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:04.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.180+0000 7fcf06000640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf0806e700 0x7fcf08156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:04.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.180+0000 7fcf06000640 1 -- 172.21.15.40:0/247784318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf081574a0 con 0x7fcf0806eae0 2024-04-26T23:04:04.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.180+0000 7fcf07400640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcf0815a4a0 0x7fcf0815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:04:04.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.180+0000 7fcf06000640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf0806eae0 0x7fcf0815b9a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcef400c400 tx=0x7fcef400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:04.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.181+0000 7fcefea00640 1 -- 172.21.15.40:0/247784318 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef40052d0 con 0x7fcf0806eae0 2024-04-26T23:04:04.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.181+0000 7fcefea00640 1 -- 172.21.15.40:0/247784318 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef4005490 con 0x7fcf0806eae0 2024-04-26T23:04:04.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.181+0000 7fcefea00640 1 -- 172.21.15.40:0/247784318 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4007260 con 0x7fcf0806eae0 2024-04-26T23:04:04.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.181+0000 7fcf0d600640 1 -- 172.21.15.40:0/247784318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf080c61d0 con 0x7fcf0806eae0 2024-04-26T23:04:04.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.181+0000 7fcf0d600640 1 -- 172.21.15.40:0/247784318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf080b17b0 con 0x7fcf0806eae0 2024-04-26T23:04:04.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.182+0000 7fcf0d600640 1 -- 172.21.15.40:0/247784318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf080cfc40 con 0x7fcf0806eae0 2024-04-26T23:04:04.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.184+0000 7fcefea00640 1 -- 172.21.15.40:0/247784318 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcef4007400 con 0x7fcf0806eae0 2024-04-26T23:04:04.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.185+0000 7fcefea00640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcee806c8f0 0x7fcee806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:04.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.185+0000 7fcefea00640 1 -- 172.21.15.40:0/247784318 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6388+0+0 (secure 0 0 0) 0x7fcef408b430 con 0x7fcf0806eae0 2024-04-26T23:04:04.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.185+0000 7fcf06a00640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcee806c8f0 0x7fcee806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.186+0000 7fcf06a00640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcee806c8f0 0x7fcee806edb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcef8034000 tx=0x7fcef802f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:04.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.189+0000 7fcefea00640 1 -- 172.21.15.40:0/247784318 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fcf080cfc40 con 0x7fcf0806eae0 2024-04-26T23:04:04.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.392+0000 7fcf0d600640 1 -- 172.21.15.40:0/247784318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fcf08161900 con 0x7fcf0806eae0 2024-04-26T23:04:04.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.952+0000 7fcefea00640 1 -- 172.21.15.40:0/247784318 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v43) ==== 181+0+0 (secure 0 0 0) 0x7fcf08161900 con 0x7fcf0806eae0 2024-04-26T23:04:04.952 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-04-26T23:04:04.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.955+0000 7fcf0d600640 1 -- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcee806c8f0 msgr2=0x7fcee806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:04.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.955+0000 7fcf0d600640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcee806c8f0 0x7fcee806edb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcef8034000 tx=0x7fcef802f000 comp rx=0 tx=0).stop 2024-04-26T23:04:04.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.955+0000 7fcf0d600640 1 -- 172.21.15.40:0/247784318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf0806eae0 msgr2=0x7fcf0815b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:04.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.955+0000 7fcf0d600640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf0806eae0 0x7fcf0815b9a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcef400c400 tx=0x7fcef400c8c0 comp rx=0 tx=0).stop 2024-04-26T23:04:04.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.956+0000 7fcf0d600640 1 -- 172.21.15.40:0/247784318 shutdown_connections 2024-04-26T23:04:04.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.956+0000 7fcf0d600640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcee806c8f0 0x7fcee806edb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:04.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.956+0000 7fcf0d600640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcf0815a4a0 0x7fcf0815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:04.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.956+0000 7fcf0d600640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf0806eae0 0x7fcf0815b9a0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:04.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.956+0000 7fcf0d600640 1 --2- 172.21.15.40:0/247784318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf0806e700 0x7fcf08156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:04.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.956+0000 7fcf0d600640 1 -- 172.21.15.40:0/247784318 >> 172.21.15.40:0/247784318 conn(0x7fcf0806f310 msgr2=0x7fcf080afa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:04.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.956+0000 7fcf0d600640 1 -- 172.21.15.40:0/247784318 shutdown_connections 2024-04-26T23:04:04.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:04.956+0000 7fcf0d600640 1 -- 172.21.15.40:0/247784318 wait complete. 2024-04-26T23:04:04.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T23:04:05.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.198+0000 7f6fdc600640 1 Processor -- start 2024-04-26T23:04:05.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.198+0000 7f6fdc600640 1 -- start start 2024-04-26T23:04:05.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.200+0000 7f6fdc600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6fd40b3190 0x7f6fd4166d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:05.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.200+0000 7f6fdc600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd40cfd70 con 0x7f6fd4145630 2024-04-26T23:04:05.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.200+0000 7f6fdc600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd40d0050 con 0x7f6fd415f980 2024-04-26T23:04:05.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.200+0000 7f6fdc600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f6fd40d0330 con 0x7f6fd40b3190 2024-04-26T23:04:05.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.200+0000 7f6fda600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6fd40b3190 0x7f6fd4166d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:05.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.200+0000 7f6fda600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6fd40b3190 0x7f6fd4166d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60778/0 (socket says 172.21.15.40:60778) 2024-04-26T23:04:05.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.200+0000 7f6fda600640 1 -- 172.21.15.40:0/1623143749 learned_addr learned my addr 172.21.15.40:0/1623143749 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:05.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.201+0000 7f6fda600640 1 -- 172.21.15.40:0/1623143749 >> v1:172.21.15.103:6789/0 conn(0x7f6fd415f980 legacy=0x7f6fd40b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:05.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.201+0000 7f6fda600640 1 -- 172.21.15.40:0/1623143749 >> v1:172.21.15.40:6789/0 conn(0x7f6fd4145630 legacy=0x7f6fd40cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:05.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.201+0000 7f6fda600640 1 -- 172.21.15.40:0/1623143749 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd40d0610 con 0x7f6fd40b3190 2024-04-26T23:04:05.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.201+0000 7f6fd3e00640 1 -- 172.21.15.40:0/1623143749 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3747058743 0 0) 0x7f6fd40cfd70 con 0x7f6fd4145630 2024-04-26T23:04:05.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.202+0000 7f6fda600640 1 --2- 172.21.15.40:0/1623143749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd40b3190 0x7f6fd4166d10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc009870 tx=0x7f6fcc02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655edae55770256b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.202+0000 7f6fd3e00640 1 -- 172.21.15.40:0/1623143749 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc037070 con 0x7f6fd40b3190 2024-04-26T23:04:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.202+0000 7f6fd3e00640 1 -- 172.21.15.40:0/1623143749 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fcc032440 con 0x7f6fd40b3190 2024-04-26T23:04:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.203+0000 7f6fd3e00640 1 -- 172.21.15.40:0/1623143749 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc032810 con 0x7f6fd40b3190 2024-04-26T23:04:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.203+0000 7f6fdc600640 1 -- 172.21.15.40:0/1623143749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd40b3190 msgr2=0x7f6fd4166d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.203+0000 7f6fdc600640 1 --2- 172.21.15.40:0/1623143749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd40b3190 0x7f6fd4166d10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc009870 tx=0x7f6fcc02a2a0 comp rx=0 tx=0).stop 2024-04-26T23:04:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.203+0000 7f6fdc600640 1 -- 172.21.15.40:0/1623143749 shutdown_connections 2024-04-26T23:04:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.203+0000 7f6fdc600640 1 --2- 172.21.15.40:0/1623143749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd40b3190 0x7f6fd4166d10 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.203+0000 7f6fdc600640 1 -- 172.21.15.40:0/1623143749 >> 172.21.15.40:0/1623143749 conn(0x7f6fd406f310 msgr2=0x7f6fd406f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.203+0000 7f6fdc600640 1 -- 172.21.15.40:0/1623143749 shutdown_connections 2024-04-26T23:04:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.203+0000 7f6fdc600640 1 -- 172.21.15.40:0/1623143749 wait complete. 2024-04-26T23:04:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.205+0000 7f6fdc600640 1 Processor -- start 2024-04-26T23:04:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.205+0000 7f6fdc600640 1 -- start start 2024-04-26T23:04:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.206+0000 7f6fdc600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fd40b3190 0x7f6fd4156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.206+0000 7f6fdc600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd4145630 0x7f6fd412ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.206+0000 7f6fdc600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fd415f980 0x7f6fd413c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.206+0000 7f6fdc600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd4145b90 con 0x7f6fd4145630 2024-04-26T23:04:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.206+0000 7f6fdc600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd4145a10 con 0x7f6fd415f980 2024-04-26T23:04:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.206+0000 7f6fdc600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6fd40b4f00 con 0x7f6fd40b3190 2024-04-26T23:04:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.207+0000 7f6fd9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd4145630 0x7f6fd412ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:05.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.207+0000 7f6fd9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd4145630 0x7f6fd412ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60788/0 (socket says 172.21.15.40:60788) 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.207+0000 7f6fd9200640 1 -- 172.21.15.40:0/1881079553 learned_addr learned my addr 172.21.15.40:0/1881079553 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.207+0000 7f6fda600640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fd415f980 0x7f6fd413c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.207+0000 7f6fd9c00640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fd40b3190 0x7f6fd4156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.207+0000 7f6fd9200640 1 -- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fd40b3190 msgr2=0x7f6fd4156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.207+0000 7f6fd9200640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fd40b3190 0x7f6fd4156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.207+0000 7f6fd9200640 1 -- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fd415f980 msgr2=0x7f6fd413c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.207+0000 7f6fd9200640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fd415f980 0x7f6fd413c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.207+0000 7f6fd9200640 1 -- 172.21.15.40:0/1881079553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd413cce0 con 0x7f6fd4145630 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.207+0000 7f6fd9c00640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fd40b3190 0x7f6fd4156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.207+0000 7f6fd9200640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd4145630 0x7f6fd412ed20 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000b3f0 tx=0x7f6fc000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.208+0000 7f6fd1600640 1 -- 172.21.15.40:0/1881079553 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0015020 con 0x7f6fd4145630 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.208+0000 7f6fd1600640 1 -- 172.21.15.40:0/1881079553 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fc0011830 con 0x7f6fd4145630 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.208+0000 7f6fd1600640 1 -- 172.21.15.40:0/1881079553 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0011c20 con 0x7f6fd4145630 2024-04-26T23:04:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.208+0000 7f6fdc600640 1 -- 172.21.15.40:0/1881079553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd413dbf0 con 0x7f6fd4145630 2024-04-26T23:04:05.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.208+0000 7f6fdc600640 1 -- 172.21.15.40:0/1881079553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fd413e1b0 con 0x7f6fd4145630 2024-04-26T23:04:05.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.209+0000 7f6fdc600640 1 -- 172.21.15.40:0/1881079553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fd4069600 con 0x7f6fd4145630 2024-04-26T23:04:05.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.210+0000 7f6fd1600640 1 -- 172.21.15.40:0/1881079553 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6fc0018020 con 0x7f6fd4145630 2024-04-26T23:04:05.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.211+0000 7f6fd1600640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6fb406c920 0x7f6fb406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:05.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.211+0000 7f6fd1600640 1 -- 172.21.15.40:0/1881079553 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5950+0+0 (secure 0 0 0) 0x7f6fc008b920 con 0x7f6fd4145630 2024-04-26T23:04:05.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.212+0000 7f6fd9c00640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6fb406c920 0x7f6fb406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.213+0000 7f6fd9c00640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6fb406c920 0x7f6fb406ede0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6fc80012e0 tx=0x7f6fc8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.216+0000 7f6fd1600640 1 -- 172.21.15.40:0/1881079553 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f6fc001f030 con 0x7f6fd4145630 2024-04-26T23:04:05.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.446+0000 7f6fdc600640 1 -- 172.21.15.40:0/1881079553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6fd40b55b0 con 0x7f6fd4145630 2024-04-26T23:04:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.962+0000 7f6fd1600640 1 -- 172.21.15.40:0/1881079553 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v44) ==== 169+0+0 (secure 0 0 0) 0x7f6fc00172c0 con 0x7f6fd4145630 2024-04-26T23:04:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-04-26T23:04:05.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.966+0000 7f6fdc600640 1 -- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6fb406c920 msgr2=0x7f6fb406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:05.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.966+0000 7f6fdc600640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6fb406c920 0x7f6fb406ede0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6fc80012e0 tx=0x7f6fc8009040 comp rx=0 tx=0).stop 2024-04-26T23:04:05.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.966+0000 7f6fdc600640 1 -- 172.21.15.40:0/1881079553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd4145630 msgr2=0x7f6fd412ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:05.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.966+0000 7f6fdc600640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd4145630 0x7f6fd412ed20 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000b3f0 tx=0x7f6fc000b8b0 comp rx=0 tx=0).stop 2024-04-26T23:04:05.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.966+0000 7f6fdc600640 1 -- 172.21.15.40:0/1881079553 shutdown_connections 2024-04-26T23:04:05.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.966+0000 7f6fdc600640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6fb406c920 0x7f6fb406ede0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:05.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.966+0000 7f6fdc600640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fd415f980 0x7f6fd413c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:05.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.966+0000 7f6fdc600640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd4145630 0x7f6fd412ed20 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:05.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.966+0000 7f6fdc600640 1 --2- 172.21.15.40:0/1881079553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fd40b3190 0x7f6fd4156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:05.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.966+0000 7f6fdc600640 1 -- 172.21.15.40:0/1881079553 >> 172.21.15.40:0/1881079553 conn(0x7f6fd406f310 msgr2=0x7f6fd406f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:05.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.968+0000 7f6fdc600640 1 -- 172.21.15.40:0/1881079553 shutdown_connections 2024-04-26T23:04:05.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:05.969+0000 7f6fdc600640 1 -- 172.21.15.40:0/1881079553 wait complete. 2024-04-26T23:04:05.995 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-26T23:04:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.215+0000 7f2a65000640 1 Processor -- start 2024-04-26T23:04:06.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.215+0000 7f2a65000640 1 -- start start 2024-04-26T23:04:06.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.216+0000 7f2a65000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2a600c41d0 0x7f2a600cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:06.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.216+0000 7f2a65000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a60073f30 0x7f2a601671e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:06.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.216+0000 7f2a65000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a60167720 0x7f2a6006ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.217+0000 7f2a65000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f2a600cf7a0 con 0x7f2a60167720 2024-04-26T23:04:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.217+0000 7f2a65000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2a60163a50 con 0x7f2a600c41d0 2024-04-26T23:04:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.217+0000 7f2a65000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2a600ceb90 con 0x7f2a60073f30 2024-04-26T23:04:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.217+0000 7f2a5e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a60167720 0x7f2a6006ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.217+0000 7f2a5e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a60167720 0x7f2a6006ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60802/0 (socket says 172.21.15.40:60802) 2024-04-26T23:04:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.217+0000 7f2a5e800640 1 -- 172.21.15.40:0/1495753499 learned_addr learned my addr 172.21.15.40:0/1495753499 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.217+0000 7f2a55400640 1 --2- 172.21.15.40:0/1495753499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a60073f30 0x7f2a601671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.217+0000 7f2a5de00640 1 --2- 172.21.15.40:0/1495753499 >> v2:172.21.15.103:3300/0 conn(0x7f2a600c41d0 0x7f2a600cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.217+0000 7f2a5e800640 1 -- 172.21.15.40:0/1495753499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a60073f30 msgr2=0x7f2a601671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.217+0000 7f2a5e800640 1 --2- 172.21.15.40:0/1495753499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a60073f30 0x7f2a601671e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.217+0000 7f2a5e800640 1 -- 172.21.15.40:0/1495753499 >> v2:172.21.15.103:3300/0 conn(0x7f2a600c41d0 msgr2=0x7f2a600cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.217+0000 7f2a5e800640 1 --2- 172.21.15.40:0/1495753499 >> v2:172.21.15.103:3300/0 conn(0x7f2a600c41d0 0x7f2a600cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.217+0000 7f2a5e800640 1 -- 172.21.15.40:0/1495753499 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a600cee70 con 0x7f2a60167720 2024-04-26T23:04:06.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.219+0000 7f2a5e800640 1 --2- 172.21.15.40:0/1495753499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a60167720 0x7f2a6006ddf0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2a600b3730 tx=0x7f2a5802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85a87cc5dac5507 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:06.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.219+0000 7f2a5d400640 1 -- 172.21.15.40:0/1495753499 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a58036070 con 0x7f2a60167720 2024-04-26T23:04:06.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.219+0000 7f2a5d400640 1 -- 172.21.15.40:0/1495753499 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a58031600 con 0x7f2a60167720 2024-04-26T23:04:06.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.219+0000 7f2a5d400640 1 -- 172.21.15.40:0/1495753499 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a580319d0 con 0x7f2a60167720 2024-04-26T23:04:06.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.219+0000 7f2a65000640 1 -- 172.21.15.40:0/1495753499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a60167720 msgr2=0x7f2a6006ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:06.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.219+0000 7f2a65000640 1 --2- 172.21.15.40:0/1495753499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a60167720 0x7f2a6006ddf0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2a600b3730 tx=0x7f2a5802a6a0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.220+0000 7f2a65000640 1 -- 172.21.15.40:0/1495753499 shutdown_connections 2024-04-26T23:04:06.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.220+0000 7f2a65000640 1 --2- 172.21.15.40:0/1495753499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a60167720 0x7f2a6006ddf0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.220+0000 7f2a65000640 1 --2- 172.21.15.40:0/1495753499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a60073f30 0x7f2a601671e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.220+0000 7f2a65000640 1 --2- 172.21.15.40:0/1495753499 >> v2:172.21.15.103:3300/0 conn(0x7f2a600c41d0 0x7f2a600cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.220+0000 7f2a65000640 1 -- 172.21.15.40:0/1495753499 >> 172.21.15.40:0/1495753499 conn(0x7f2a600d0be0 msgr2=0x7f2a600d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:06.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.220+0000 7f2a65000640 1 -- 172.21.15.40:0/1495753499 shutdown_connections 2024-04-26T23:04:06.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.220+0000 7f2a65000640 1 -- 172.21.15.40:0/1495753499 wait complete. 2024-04-26T23:04:06.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.222+0000 7f2a65000640 1 Processor -- start 2024-04-26T23:04:06.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.222+0000 7f2a65000640 1 -- start start 2024-04-26T23:04:06.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a65000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a60073f30 0x7f2a60158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:06.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a65000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a600c41d0 0x7f2a600b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:06.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a65000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a60167720 0x7f2a600b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:06.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a65000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2a600d0170 con 0x7f2a60073f30 2024-04-26T23:04:06.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a65000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2a600cf430 con 0x7f2a600c41d0 2024-04-26T23:04:06.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a65000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2a600ce7b0 con 0x7f2a60167720 2024-04-26T23:04:06.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a5de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a60073f30 0x7f2a60158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:06.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a5de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a60073f30 0x7f2a60158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60812/0 (socket says 172.21.15.40:60812) 2024-04-26T23:04:06.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a5de00640 1 -- 172.21.15.40:0/4252425843 learned_addr learned my addr 172.21.15.40:0/4252425843 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:06.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a5de00640 1 -- 172.21.15.40:0/4252425843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a60167720 msgr2=0x7f2a600b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:06.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a55400640 1 --2- 172.21.15.40:0/4252425843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a600c41d0 0x7f2a600b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:06.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a5de00640 1 --2- 172.21.15.40:0/4252425843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a60167720 0x7f2a600b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a5de00640 1 -- 172.21.15.40:0/4252425843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a600c41d0 msgr2=0x7f2a600b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:06.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a5de00640 1 --2- 172.21.15.40:0/4252425843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a600c41d0 0x7f2a600b37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.223+0000 7f2a5de00640 1 -- 172.21.15.40:0/4252425843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a600b4410 con 0x7f2a60073f30 2024-04-26T23:04:06.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.224+0000 7f2a5de00640 1 --2- 172.21.15.40:0/4252425843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a60073f30 0x7f2a60158060 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000d800 tx=0x7f2a5000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:06.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.224+0000 7f2a56a00640 1 -- 172.21.15.40:0/4252425843 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5000b6c0 con 0x7f2a60073f30 2024-04-26T23:04:06.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.224+0000 7f2a56a00640 1 -- 172.21.15.40:0/4252425843 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a50019070 con 0x7f2a60073f30 2024-04-26T23:04:06.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.224+0000 7f2a56a00640 1 -- 172.21.15.40:0/4252425843 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5000ba80 con 0x7f2a60073f30 2024-04-26T23:04:06.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.224+0000 7f2a65000640 1 -- 172.21.15.40:0/4252425843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a600b46f0 con 0x7f2a60073f30 2024-04-26T23:04:06.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.224+0000 7f2a65000640 1 -- 172.21.15.40:0/4252425843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a600b4cb0 con 0x7f2a60073f30 2024-04-26T23:04:06.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.226+0000 7f2a65000640 1 -- 172.21.15.40:0/4252425843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a600694f0 con 0x7f2a60073f30 2024-04-26T23:04:06.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.227+0000 7f2a56a00640 1 -- 172.21.15.40:0/4252425843 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a5000bc20 con 0x7f2a60073f30 2024-04-26T23:04:06.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.227+0000 7f2a56a00640 1 --2- 172.21.15.40:0/4252425843 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2a3c06c920 0x7f2a3c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:06.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.228+0000 7f2a56a00640 1 -- 172.21.15.40:0/4252425843 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5507+0+0 (secure 0 0 0) 0x7f2a5008bfe0 con 0x7f2a60073f30 2024-04-26T23:04:06.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.228+0000 7f2a55400640 1 --2- 172.21.15.40:0/4252425843 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2a3c06c920 0x7f2a3c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:06.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.229+0000 7f2a55400640 1 --2- 172.21.15.40:0/4252425843 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2a3c06c920 0x7f2a3c06ede0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2a6015c3a0 tx=0x7f2a48008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:06.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.232+0000 7f2a56a00640 1 -- 172.21.15.40:0/4252425843 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2a5005a430 con 0x7f2a60073f30 2024-04-26T23:04:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.437+0000 7f2a65000640 1 -- 172.21.15.40:0/4252425843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f2a600d0430 con 0x7f2a60073f30 2024-04-26T23:04:06.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.441+0000 7f2a56a00640 1 -- 172.21.15.40:0/4252425843 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f2a50059d30 con 0x7f2a60073f30 2024-04-26T23:04:06.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.444+0000 7f2a65000640 1 -- 172.21.15.40:0/4252425843 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2a3c06c920 msgr2=0x7f2a3c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:06.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.444+0000 7f2a65000640 1 --2- 172.21.15.40:0/4252425843 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2a3c06c920 0x7f2a3c06ede0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2a6015c3a0 tx=0x7f2a48008040 comp rx=0 tx=0).stop 2024-04-26T23:04:06.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.444+0000 7f2a65000640 1 -- 172.21.15.40:0/4252425843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a60073f30 msgr2=0x7f2a60158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:06.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.444+0000 7f2a65000640 1 --2- 172.21.15.40:0/4252425843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a60073f30 0x7f2a60158060 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000d800 tx=0x7f2a5000dcc0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.445+0000 7f2a65000640 1 -- 172.21.15.40:0/4252425843 shutdown_connections 2024-04-26T23:04:06.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.445+0000 7f2a65000640 1 --2- 172.21.15.40:0/4252425843 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2a3c06c920 0x7f2a3c06ede0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.445+0000 7f2a65000640 1 --2- 172.21.15.40:0/4252425843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2a60167720 0x7f2a600b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.445+0000 7f2a65000640 1 --2- 172.21.15.40:0/4252425843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a600c41d0 0x7f2a600b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.445+0000 7f2a65000640 1 --2- 172.21.15.40:0/4252425843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a60073f30 0x7f2a60158060 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.445+0000 7f2a65000640 1 -- 172.21.15.40:0/4252425843 >> 172.21.15.40:0/4252425843 conn(0x7f2a600d0be0 msgr2=0x7f2a601589c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:06.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.445+0000 7f2a65000640 1 -- 172.21.15.40:0/4252425843 shutdown_connections 2024-04-26T23:04:06.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.445+0000 7f2a65000640 1 -- 172.21.15.40:0/4252425843 wait complete. 2024-04-26T23:04:06.466 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_apply_tag' 2024-04-26T23:04:06.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.667+0000 7fa7fb200640 1 Processor -- start 2024-04-26T23:04:06.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.668+0000 7fa7fb200640 1 -- start start 2024-04-26T23:04:06.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.669+0000 7fa7fb200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7fc0d06b0 0x7fa7fc0d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:06.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.669+0000 7fa7fb200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7fc0b59f0 con 0x7fa7fc146060 2024-04-26T23:04:06.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.669+0000 7fa7fb200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7fc0b4180 con 0x7fa7fc145970 2024-04-26T23:04:06.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.669+0000 7fa7fb200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa7fc0ceed0 con 0x7fa7fc0d06b0 2024-04-26T23:04:06.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.669+0000 7fa7fa800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7fc0d06b0 0x7fa7fc0d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:06.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.669+0000 7fa7f3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa7fc146060 0x7fa7fc0b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37782/0 (socket says 172.21.15.40:37782) 2024-04-26T23:04:06.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.669+0000 7fa7f3e00640 1 -- 172.21.15.40:0/2582914630 learned_addr learned my addr 172.21.15.40:0/2582914630 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:06.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.670+0000 7fa7fa800640 1 -- 172.21.15.40:0/2582914630 >> v1:172.21.15.103:6789/0 conn(0x7fa7fc145970 legacy=0x7fa7fc1612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:06.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.670+0000 7fa7fa800640 1 -- 172.21.15.40:0/2582914630 >> v1:172.21.15.40:6789/0 conn(0x7fa7fc146060 legacy=0x7fa7fc0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:06.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.670+0000 7fa7fa800640 1 -- 172.21.15.40:0/2582914630 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7fc0b5f40 con 0x7fa7fc0d06b0 2024-04-26T23:04:06.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.670+0000 7fa7f9400640 1 -- 172.21.15.40:0/2582914630 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 651920713 0 0) 0x7fa7fc0b59f0 con 0x7fa7fc146060 2024-04-26T23:04:06.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.671+0000 7fa7fa800640 1 --2- 172.21.15.40:0/2582914630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7fc0d06b0 0x7fa7fc0d0ae0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009dc0 tx=0x7fa7f402a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e665d073331cd8b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:06.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.671+0000 7fa7f9400640 1 -- 172.21.15.40:0/2582914630 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4007740 con 0x7fa7fc0d06b0 2024-04-26T23:04:06.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.671+0000 7fa7f9400640 1 -- 172.21.15.40:0/2582914630 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7f40078e0 con 0x7fa7fc0d06b0 2024-04-26T23:04:06.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.671+0000 7fa7f9400640 1 -- 172.21.15.40:0/2582914630 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4007cb0 con 0x7fa7fc0d06b0 2024-04-26T23:04:06.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.672+0000 7fa7fb200640 1 -- 172.21.15.40:0/2582914630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7fc0d06b0 msgr2=0x7fa7fc0d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:06.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.672+0000 7fa7fb200640 1 --2- 172.21.15.40:0/2582914630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7fc0d06b0 0x7fa7fc0d0ae0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009dc0 tx=0x7fa7f402a6d0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.672+0000 7fa7fb200640 1 -- 172.21.15.40:0/2582914630 shutdown_connections 2024-04-26T23:04:06.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.672+0000 7fa7fb200640 1 --2- 172.21.15.40:0/2582914630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7fc0d06b0 0x7fa7fc0d0ae0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.672+0000 7fa7fb200640 1 -- 172.21.15.40:0/2582914630 >> 172.21.15.40:0/2582914630 conn(0x7fa7fc075eb0 msgr2=0x7fa7fc068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:06.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.672+0000 7fa7fb200640 1 -- 172.21.15.40:0/2582914630 shutdown_connections 2024-04-26T23:04:06.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.672+0000 7fa7fb200640 1 -- 172.21.15.40:0/2582914630 wait complete. 2024-04-26T23:04:06.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.674+0000 7fa7fb200640 1 Processor -- start 2024-04-26T23:04:06.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.674+0000 7fa7fb200640 1 -- start start 2024-04-26T23:04:06.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.675+0000 7fa7fb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7fc0d06b0 0x7fa7fc159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:06.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.675+0000 7fa7fb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7fc145970 0x7fa7fc06e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:06.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.675+0000 7fa7fb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7fc146060 0x7fa7fc1f5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:06.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.675+0000 7fa7fb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa7fc0cf100 con 0x7fa7fc146060 2024-04-26T23:04:06.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.675+0000 7fa7fb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa7fc0d1020 con 0x7fa7fc145970 2024-04-26T23:04:06.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.675+0000 7fa7fb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa7fc0cf3e0 con 0x7fa7fc0d06b0 2024-04-26T23:04:06.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.675+0000 7fa7fa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7fc146060 0x7fa7fc1f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:06.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.676+0000 7fa7fa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7fc146060 0x7fa7fc1f5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60838/0 (socket says 172.21.15.40:60838) 2024-04-26T23:04:06.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.676+0000 7fa7fa800640 1 -- 172.21.15.40:0/3633981268 learned_addr learned my addr 172.21.15.40:0/3633981268 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:06.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.676+0000 7fa7f9e00640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7fc0d06b0 0x7fa7fc159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:06.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.676+0000 7fa7f3e00640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7fc145970 0x7fa7fc06e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:06.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.676+0000 7fa7fa800640 1 -- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7fc0d06b0 msgr2=0x7fa7fc159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:06.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.676+0000 7fa7fa800640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7fc0d06b0 0x7fa7fc159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.676+0000 7fa7fa800640 1 -- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7fc145970 msgr2=0x7fa7fc06e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:06.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.676+0000 7fa7fa800640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7fc145970 0x7fa7fc06e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:06.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.676+0000 7fa7fa800640 1 -- 172.21.15.40:0/3633981268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7fc1573c0 con 0x7fa7fc146060 2024-04-26T23:04:06.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.676+0000 7fa7f9e00640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7fc0d06b0 0x7fa7fc159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:04:06.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.676+0000 7fa7fa800640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7fc146060 0x7fa7fc1f5690 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009dc0 tx=0x7fa7f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:06.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.677+0000 7fa7f2000640 1 -- 172.21.15.40:0/3633981268 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4040070 con 0x7fa7fc146060 2024-04-26T23:04:06.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.677+0000 7fa7f2000640 1 -- 172.21.15.40:0/3633981268 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7f4031c40 con 0x7fa7fc146060 2024-04-26T23:04:06.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.677+0000 7fa7f2000640 1 -- 172.21.15.40:0/3633981268 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4003990 con 0x7fa7fc146060 2024-04-26T23:04:06.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.677+0000 7fa7fb200640 1 -- 172.21.15.40:0/3633981268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7fc1576a0 con 0x7fa7fc146060 2024-04-26T23:04:06.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.677+0000 7fa7fb200640 1 -- 172.21.15.40:0/3633981268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7fc157c60 con 0x7fa7fc146060 2024-04-26T23:04:06.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.678+0000 7fa7fb200640 1 -- 172.21.15.40:0/3633981268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7fc069370 con 0x7fa7fc146060 2024-04-26T23:04:06.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.679+0000 7fa7f2000640 1 -- 172.21.15.40:0/3633981268 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa7f4039040 con 0x7fa7fc146060 2024-04-26T23:04:06.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.680+0000 7fa7f2000640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7d806c940 0x7fa7d806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:06.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.680+0000 7fa7f2000640 1 -- 172.21.15.40:0/3633981268 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5507+0+0 (secure 0 0 0) 0x7fa7f40ac9f0 con 0x7fa7fc146060 2024-04-26T23:04:06.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.681+0000 7fa7f9e00640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7d806c940 0x7fa7d806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:06.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.682+0000 7fa7f9e00640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7d806c940 0x7fa7d806ee00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec003eb0 tx=0x7fa7ec009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:06.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.685+0000 7fa7f2000640 1 -- 172.21.15.40:0/3633981268 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa7f407afc0 con 0x7fa7fc146060 2024-04-26T23:04:06.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:06.968+0000 7fa7fb200640 1 -- 172.21.15.40:0/3633981268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_apply_tag"]} v 0) -- 0x7fa7fc0d1400 con 0x7fa7fc146060 2024-04-26T23:04:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:07.975+0000 7fa7f2000640 1 -- 172.21.15.40:0/3633981268 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_apply_tag"]}]=0 v175) ==== 136+0+0 (secure 0 0 0) 0x7fa7f407a8c0 con 0x7fa7fc146060 2024-04-26T23:04:07.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:07.979+0000 7fa7fb200640 1 -- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7d806c940 msgr2=0x7fa7d806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:07.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:07.979+0000 7fa7fb200640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7d806c940 0x7fa7d806ee00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec003eb0 tx=0x7fa7ec009040 comp rx=0 tx=0).stop 2024-04-26T23:04:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:07.979+0000 7fa7fb200640 1 -- 172.21.15.40:0/3633981268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7fc146060 msgr2=0x7fa7fc1f5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:07.979+0000 7fa7fb200640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7fc146060 0x7fa7fc1f5690 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009dc0 tx=0x7fa7f4004a40 comp rx=0 tx=0).stop 2024-04-26T23:04:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:07.980+0000 7fa7fb200640 1 -- 172.21.15.40:0/3633981268 shutdown_connections 2024-04-26T23:04:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:07.980+0000 7fa7fb200640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7d806c940 0x7fa7d806ee00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:07.980+0000 7fa7fb200640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7fc146060 0x7fa7fc1f5690 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:07.980+0000 7fa7fb200640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7fc145970 0x7fa7fc06e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:07.980+0000 7fa7fb200640 1 --2- 172.21.15.40:0/3633981268 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7fc0d06b0 0x7fa7fc159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:07.980+0000 7fa7fb200640 1 -- 172.21.15.40:0/3633981268 >> 172.21.15.40:0/3633981268 conn(0x7fa7fc075eb0 msgr2=0x7fa7fc0b5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:07.980+0000 7fa7fb200640 1 -- 172.21.15.40:0/3633981268 shutdown_connections 2024-04-26T23:04:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:07.980+0000 7fa7fb200640 1 -- 172.21.15.40:0/3633981268 wait complete. 2024-04-26T23:04:08.004 INFO:tasks.cephfs_test_runner:test_apply_tag (tasks.cephfs.test_multimds_misc.TestScrub2) ... ok 2024-04-26T23:04:08.005 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_abort_mds0 (tasks.cephfs.test_multimds_misc.TestScrub2) 2024-04-26T23:04:08.005 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:04:08.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.220+0000 7fa41b400640 1 Processor -- start 2024-04-26T23:04:08.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.220+0000 7fa41b400640 1 -- start start 2024-04-26T23:04:08.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.222+0000 7fa41b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41415f380 0x7fa4140cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.222+0000 7fa41b400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa414073b70 0x7fa4140cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.222+0000 7fa41b400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa414072c00 con 0x7fa4140d0620 2024-04-26T23:04:08.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.222+0000 7fa41b400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa4140729b0 con 0x7fa414073b70 2024-04-26T23:04:08.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.222+0000 7fa41b400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa414072ee0 con 0x7fa41415f380 2024-04-26T23:04:08.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.222+0000 7fa419400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa4140d0620 0x7fa4140cede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37784/0 (socket says 172.21.15.40:37784) 2024-04-26T23:04:08.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.222+0000 7fa419400640 1 -- 172.21.15.40:0/1290874219 learned_addr learned my addr 172.21.15.40:0/1290874219 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:08.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.222+0000 7fa413e00640 1 --2- 172.21.15.40:0/1290874219 >> v2:172.21.15.103:3300/0 conn(0x7fa414073b70 0x7fa4140cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.222+0000 7fa418a00640 1 --2- 172.21.15.40:0/1290874219 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41415f380 0x7fa4140cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3385746654 0 0) 0x7fa414072c00 con 0x7fa4140d0620 2024-04-26T23:04:08.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3fc003520 con 0x7fa4140d0620 2024-04-26T23:04:08.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 403702969 0 0) 0x7fa3fc003520 con 0x7fa4140d0620 2024-04-26T23:04:08.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41415f380 msgr2=0x7fa4140cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:08.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 --2- 172.21.15.40:0/1290874219 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41415f380 0x7fa4140cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 >> v2:172.21.15.103:3300/0 conn(0x7fa414073b70 msgr2=0x7fa4140cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:08.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 --2- 172.21.15.40:0/1290874219 >> v2:172.21.15.103:3300/0 conn(0x7fa414073b70 0x7fa4140cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4140731c0 con 0x7fa4140d0620 2024-04-26T23:04:08.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fa40c0019f0 con 0x7fa4140d0620 2024-04-26T23:04:08.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 >> v1:172.21.15.40:6789/0 conn(0x7fa4140d0620 legacy=0x7fa4140cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:08.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 --2- 172.21.15.40:0/1290874219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3fc0039b0 0x7fa3fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 --2- 172.21.15.40:0/1290874219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3fc004800 0x7fa3fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 --2- 172.21.15.40:0/1290874219 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41415f380 0x7fa3fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa4140729b0 con 0x7fa3fc0039b0 2024-04-26T23:04:08.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa414072ee0 con 0x7fa3fc004800 2024-04-26T23:04:08.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.223+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa3fc00b9c0 con 0x7fa41415f380 2024-04-26T23:04:08.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.224+0000 7fa418a00640 1 --2- 172.21.15.40:0/1290874219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3fc0039b0 0x7fa3fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.224+0000 7fa413e00640 1 --2- 172.21.15.40:0/1290874219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3fc004800 0x7fa3fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.224+0000 7fa419400640 1 --2- 172.21.15.40:0/1290874219 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41415f380 0x7fa3fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.224+0000 7fa418a00640 1 -- 172.21.15.40:0/1290874219 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41415f380 msgr2=0x7fa3fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:08.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.224+0000 7fa418a00640 1 --2- 172.21.15.40:0/1290874219 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41415f380 0x7fa3fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.224+0000 7fa418a00640 1 -- 172.21.15.40:0/1290874219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3fc004800 msgr2=0x7fa3fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:08.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.224+0000 7fa418a00640 1 --2- 172.21.15.40:0/1290874219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3fc004800 0x7fa3fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.224+0000 7fa418a00640 1 -- 172.21.15.40:0/1290874219 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3fc003520 con 0x7fa3fc0039b0 2024-04-26T23:04:08.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.225+0000 7fa418a00640 1 --2- 172.21.15.40:0/1290874219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3fc0039b0 0x7fa3fc003dd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa408009ac0 tx=0x7fa3fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b83cffec78bb5b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:08.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.225+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408036070 con 0x7fa3fc0039b0 2024-04-26T23:04:08.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.226+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa408031400 con 0x7fa3fc0039b0 2024-04-26T23:04:08.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.226+0000 7fa413400640 1 -- 172.21.15.40:0/1290874219 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408009690 con 0x7fa3fc0039b0 2024-04-26T23:04:08.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.226+0000 7fa41b400640 1 -- 172.21.15.40:0/1290874219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3fc0039b0 msgr2=0x7fa3fc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:08.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.226+0000 7fa41b400640 1 --2- 172.21.15.40:0/1290874219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3fc0039b0 0x7fa3fc003dd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa408009ac0 tx=0x7fa3fc005830 comp rx=0 tx=0).stop 2024-04-26T23:04:08.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.226+0000 7fa419400640 1 -- 172.21.15.40:0/1290874219 reap_dead start 2024-04-26T23:04:08.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.226+0000 7fa41b400640 1 -- 172.21.15.40:0/1290874219 shutdown_connections 2024-04-26T23:04:08.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.226+0000 7fa41b400640 1 -- 172.21.15.40:0/1290874219 >> 172.21.15.40:0/1290874219 conn(0x7fa4140d0be0 msgr2=0x7fa4140d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:08.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.226+0000 7fa41b400640 1 -- 172.21.15.40:0/1290874219 shutdown_connections 2024-04-26T23:04:08.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.226+0000 7fa41b400640 1 -- 172.21.15.40:0/1290874219 wait complete. 2024-04-26T23:04:08.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.228+0000 7fa41b400640 1 Processor -- start 2024-04-26T23:04:08.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.228+0000 7fa41b400640 1 -- start start 2024-04-26T23:04:08.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.229+0000 7fa41b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3fc0039b0 0x7fa4140c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.229+0000 7fa41b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41414be00 0x7fa4141f8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.229+0000 7fa41b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4140c9c00 0x7fa4141f87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.229+0000 7fa41b400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa414073620 con 0x7fa3fc0039b0 2024-04-26T23:04:08.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.229+0000 7fa41b400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4140aff10 con 0x7fa41414be00 2024-04-26T23:04:08.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.229+0000 7fa41b400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa4140737a0 con 0x7fa4140c9c00 2024-04-26T23:04:08.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.230+0000 7fa418a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3fc0039b0 0x7fa4140c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.230+0000 7fa418a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3fc0039b0 0x7fa4140c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60858/0 (socket says 172.21.15.40:60858) 2024-04-26T23:04:08.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.230+0000 7fa418a00640 1 -- 172.21.15.40:0/2969390654 learned_addr learned my addr 172.21.15.40:0/2969390654 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:08.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.230+0000 7fa413e00640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41414be00 0x7fa4141f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.230+0000 7fa419400640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4140c9c00 0x7fa4141f87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.230+0000 7fa418a00640 1 -- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4140c9c00 msgr2=0x7fa4141f87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:08.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.230+0000 7fa418a00640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4140c9c00 0x7fa4141f87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.230+0000 7fa418a00640 1 -- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41414be00 msgr2=0x7fa4141f8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:08.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.230+0000 7fa418a00640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41414be00 0x7fa4141f8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.230+0000 7fa418a00640 1 -- 172.21.15.40:0/2969390654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4140ca1d0 con 0x7fa3fc0039b0 2024-04-26T23:04:08.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.231+0000 7fa418a00640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3fc0039b0 0x7fa4140c96c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa408034040 tx=0x7fa408004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.231+0000 7fa410c00640 1 -- 172.21.15.40:0/2969390654 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408040020 con 0x7fa3fc0039b0 2024-04-26T23:04:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.231+0000 7fa410c00640 1 -- 172.21.15.40:0/2969390654 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa408003940 con 0x7fa3fc0039b0 2024-04-26T23:04:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.231+0000 7fa41b400640 1 -- 172.21.15.40:0/2969390654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4140ca4b0 con 0x7fa3fc0039b0 2024-04-26T23:04:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.231+0000 7fa410c00640 1 -- 172.21.15.40:0/2969390654 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408036040 con 0x7fa3fc0039b0 2024-04-26T23:04:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.231+0000 7fa41b400640 1 -- 172.21.15.40:0/2969390654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa41414c4a0 con 0x7fa3fc0039b0 2024-04-26T23:04:08.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.233+0000 7fa41b400640 1 -- 172.21.15.40:0/2969390654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4140694f0 con 0x7fa3fc0039b0 2024-04-26T23:04:08.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.233+0000 7fa410c00640 1 -- 172.21.15.40:0/2969390654 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa408043020 con 0x7fa3fc0039b0 2024-04-26T23:04:08.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.234+0000 7fa410c00640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa3f406c8f0 0x7fa3f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.235+0000 7fa410c00640 1 -- 172.21.15.40:0/2969390654 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5507+0+0 (secure 0 0 0) 0x7fa4080abcf0 con 0x7fa3fc0039b0 2024-04-26T23:04:08.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.235+0000 7fa413e00640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa3f406c8f0 0x7fa3f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.236+0000 7fa413e00640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa3f406c8f0 0x7fa3f406edb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa4141e59e0 tx=0x7fa400008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:08.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.239+0000 7fa410c00640 1 -- 172.21.15.40:0/2969390654 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa40804a070 con 0x7fa3fc0039b0 2024-04-26T23:04:08.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.446+0000 7fa41b400640 1 -- 172.21.15.40:0/2969390654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa4140cfd20 con 0x7fa3fc0039b0 2024-04-26T23:04:08.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.447+0000 7fa410c00640 1 -- 172.21.15.40:0/2969390654 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+13513 (secure 0 0 0) 0x7fa40807a1d0 con 0x7fa3fc0039b0 2024-04-26T23:04:08.447 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:08.447 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":44,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:04:05.953443+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6836/603576312":"2024-04-27T23:04:02.932415+0000","172.21.15.103:6834/603576312":"2024-04-27T23:04:02.932415+0000","172.21.15.40:6834/716981348":"2024-04-27T23:04:02.932443+0000","172.21.15.40:6832/716981348":"2024-04-27T23:04:02.932443+0000","172.21.15.40:6835/299800334":"2024-04-27T23:04:02.932461+0000","172.21.15.40:6833/299800334":"2024-04-27T23:04:02.932461+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:04:08.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.450+0000 7fa41b400640 1 -- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa3f406c8f0 msgr2=0x7fa3f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:08.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.450+0000 7fa41b400640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa3f406c8f0 0x7fa3f406edb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa4141e59e0 tx=0x7fa400008040 comp rx=0 tx=0).stop 2024-04-26T23:04:08.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.450+0000 7fa41b400640 1 -- 172.21.15.40:0/2969390654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3fc0039b0 msgr2=0x7fa4140c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:08.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.450+0000 7fa41b400640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3fc0039b0 0x7fa4140c96c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa408034040 tx=0x7fa408004730 comp rx=0 tx=0).stop 2024-04-26T23:04:08.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.450+0000 7fa41b400640 1 -- 172.21.15.40:0/2969390654 shutdown_connections 2024-04-26T23:04:08.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.450+0000 7fa41b400640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa3f406c8f0 0x7fa3f406edb0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.450+0000 7fa41b400640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4140c9c00 0x7fa4141f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.450+0000 7fa41b400640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41414be00 0x7fa4141f8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.450+0000 7fa41b400640 1 --2- 172.21.15.40:0/2969390654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3fc0039b0 0x7fa4140c96c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.450+0000 7fa41b400640 1 -- 172.21.15.40:0/2969390654 >> 172.21.15.40:0/2969390654 conn(0x7fa4140d0be0 msgr2=0x7fa4140b0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:08.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.450+0000 7fa41b400640 1 -- 172.21.15.40:0/2969390654 shutdown_connections 2024-04-26T23:04:08.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.450+0000 7fa41b400640 1 -- 172.21.15.40:0/2969390654 wait complete. 2024-04-26T23:04:08.471 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:04:08.472 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:04:08.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.684+0000 7fac67e00640 1 Processor -- start 2024-04-26T23:04:08.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.684+0000 7fac67e00640 1 -- start start 2024-04-26T23:04:08.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.685+0000 7fac67e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac68072ce0 0x7fac6815f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.686+0000 7fac67e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fac6815fcb0 0x7fac680ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.686+0000 7fac67e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac681604e0 con 0x7fac68073600 2024-04-26T23:04:08.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.686+0000 7fac67e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fac68072850 con 0x7fac6815fcb0 2024-04-26T23:04:08.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.686+0000 7fac67e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fac68072200 con 0x7fac68072ce0 2024-04-26T23:04:08.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.686+0000 7fac66a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fac68073600 0x7fac680739e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37796/0 (socket says 172.21.15.40:37796) 2024-04-26T23:04:08.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.686+0000 7fac66a00640 1 -- 172.21.15.40:0/3146116795 learned_addr learned my addr 172.21.15.40:0/3146116795 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:08.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.686+0000 7fac5fe00640 1 --2- 172.21.15.40:0/3146116795 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac68072ce0 0x7fac6815f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.686+0000 7fac67400640 1 --2- 172.21.15.40:0/3146116795 >> v2:172.21.15.103:3300/0 conn(0x7fac6815fcb0 0x7fac680ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.686+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 297687868 0 0) 0x7fac681604e0 con 0x7fac68073600 2024-04-26T23:04:08.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac4c003520 con 0x7fac68073600 2024-04-26T23:04:08.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1148731512 0 0) 0x7fac4c003520 con 0x7fac68073600 2024-04-26T23:04:08.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac68072ce0 msgr2=0x7fac6815f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 --2- 172.21.15.40:0/3146116795 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac68072ce0 0x7fac6815f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 >> v2:172.21.15.103:3300/0 conn(0x7fac6815fcb0 msgr2=0x7fac680ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 --2- 172.21.15.40:0/3146116795 >> v2:172.21.15.103:3300/0 conn(0x7fac6815fcb0 0x7fac680ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac680724e0 con 0x7fac68073600 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fac58001670 con 0x7fac68073600 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 >> v1:172.21.15.40:6789/0 conn(0x7fac68073600 legacy=0x7fac680739e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 --2- 172.21.15.40:0/3146116795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac4c0039b0 0x7fac4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 --2- 172.21.15.40:0/3146116795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c004800 0x7fac4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 --2- 172.21.15.40:0/3146116795 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac68072ce0 0x7fac4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fac68072850 con 0x7fac4c004800 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fac68072200 con 0x7fac4c0039b0 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fac4c00b9c0 con 0x7fac68072ce0 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.687+0000 7fac5fe00640 1 --2- 172.21.15.40:0/3146116795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c004800 0x7fac4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.688+0000 7fac66a00640 1 --2- 172.21.15.40:0/3146116795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac4c0039b0 0x7fac4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.688+0000 7fac67400640 1 --2- 172.21.15.40:0/3146116795 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac68072ce0 0x7fac4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.688+0000 7fac5fe00640 1 -- 172.21.15.40:0/3146116795 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac68072ce0 msgr2=0x7fac4c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:08.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.688+0000 7fac5fe00640 1 --2- 172.21.15.40:0/3146116795 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac68072ce0 0x7fac4c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.688+0000 7fac5fe00640 1 -- 172.21.15.40:0/3146116795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac4c0039b0 msgr2=0x7fac4c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:08.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.688+0000 7fac5fe00640 1 --2- 172.21.15.40:0/3146116795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac4c0039b0 0x7fac4c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.688+0000 7fac5fe00640 1 -- 172.21.15.40:0/3146116795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c003520 con 0x7fac4c004800 2024-04-26T23:04:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.689+0000 7fac5fe00640 1 --2- 172.21.15.40:0/3146116795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c004800 0x7fac4c004c80 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fac50009ac0 tx=0x7fac4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=973a1e61f45b54fe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.690+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50036070 con 0x7fac4c004800 2024-04-26T23:04:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.690+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac50031400 con 0x7fac4c004800 2024-04-26T23:04:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.690+0000 7fac66000640 1 -- 172.21.15.40:0/3146116795 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50009690 con 0x7fac4c004800 2024-04-26T23:04:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.690+0000 7fac67e00640 1 -- 172.21.15.40:0/3146116795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c004800 msgr2=0x7fac4c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.690+0000 7fac67e00640 1 --2- 172.21.15.40:0/3146116795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c004800 0x7fac4c004c80 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fac50009ac0 tx=0x7fac4c005830 comp rx=0 tx=0).stop 2024-04-26T23:04:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.690+0000 7fac67400640 1 -- 172.21.15.40:0/3146116795 reap_dead start 2024-04-26T23:04:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.690+0000 7fac67e00640 1 -- 172.21.15.40:0/3146116795 shutdown_connections 2024-04-26T23:04:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.690+0000 7fac67e00640 1 -- 172.21.15.40:0/3146116795 >> 172.21.15.40:0/3146116795 conn(0x7fac680b2720 msgr2=0x7fac680b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.690+0000 7fac67e00640 1 -- 172.21.15.40:0/3146116795 shutdown_connections 2024-04-26T23:04:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.690+0000 7fac67e00640 1 -- 172.21.15.40:0/3146116795 wait complete. 2024-04-26T23:04:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.692+0000 7fac67e00640 1 Processor -- start 2024-04-26T23:04:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.692+0000 7fac67e00640 1 -- start start 2024-04-26T23:04:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.693+0000 7fac67e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac4c004800 0x7fac681607c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.693+0000 7fac67e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac68160d00 0x7fac6815a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.693+0000 7fac67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac681517c0 0x7fac6815a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.693+0000 7fac67e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fac681602a0 con 0x7fac681517c0 2024-04-26T23:04:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.693+0000 7fac67e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fac68160120 con 0x7fac4c004800 2024-04-26T23:04:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.693+0000 7fac67e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fac68071e20 con 0x7fac68160d00 2024-04-26T23:04:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.694+0000 7fac67400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac681517c0 0x7fac6815a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.694+0000 7fac67400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac681517c0 0x7fac6815a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60878/0 (socket says 172.21.15.40:60878) 2024-04-26T23:04:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.694+0000 7fac67400640 1 -- 172.21.15.40:0/2084574818 learned_addr learned my addr 172.21.15.40:0/2084574818 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.694+0000 7fac5fe00640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac68160d00 0x7fac6815a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.694+0000 7fac66a00640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac4c004800 0x7fac681607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.694+0000 7fac67400640 1 -- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac68160d00 msgr2=0x7fac6815a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.694+0000 7fac67400640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac68160d00 0x7fac6815a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.694+0000 7fac67400640 1 -- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac4c004800 msgr2=0x7fac681607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.694+0000 7fac67400640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac4c004800 0x7fac681607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.694+0000 7fac67400640 1 -- 172.21.15.40:0/2084574818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac68161210 con 0x7fac681517c0 2024-04-26T23:04:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.694+0000 7fac67400640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac681517c0 0x7fac6815a660 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fac60009fd0 tx=0x7fac6000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.695+0000 7fac5ea00640 1 -- 172.21.15.40:0/2084574818 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac60016620 con 0x7fac681517c0 2024-04-26T23:04:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.695+0000 7fac5ea00640 1 -- 172.21.15.40:0/2084574818 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac6001b070 con 0x7fac681517c0 2024-04-26T23:04:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.695+0000 7fac5ea00640 1 -- 172.21.15.40:0/2084574818 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac60016a20 con 0x7fac681517c0 2024-04-26T23:04:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.695+0000 7fac67e00640 1 -- 172.21.15.40:0/2084574818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac681614f0 con 0x7fac681517c0 2024-04-26T23:04:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.695+0000 7fac67e00640 1 -- 172.21.15.40:0/2084574818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac68152880 con 0x7fac681517c0 2024-04-26T23:04:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.696+0000 7fac67e00640 1 -- 172.21.15.40:0/2084574818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac681604e0 con 0x7fac681517c0 2024-04-26T23:04:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.697+0000 7fac5ea00640 1 -- 172.21.15.40:0/2084574818 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fac60005d10 con 0x7fac681517c0 2024-04-26T23:04:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.698+0000 7fac5ea00640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fac4406c920 0x7fac4406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.698+0000 7fac5ea00640 1 -- 172.21.15.40:0/2084574818 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5507+0+0 (secure 0 0 0) 0x7fac6008c4c0 con 0x7fac681517c0 2024-04-26T23:04:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.699+0000 7fac66a00640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fac4406c920 0x7fac4406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:08.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.700+0000 7fac66a00640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fac4406c920 0x7fac4406ede0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fac580017d0 tx=0x7fac58000f50 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:08.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.702+0000 7fac5ea00640 1 -- 172.21.15.40:0/2084574818 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fac681604e0 con 0x7fac681517c0 2024-04-26T23:04:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.903+0000 7fac67e00640 1 -- 172.21.15.40:0/2084574818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fac68069370 con 0x7fac681517c0 2024-04-26T23:04:08.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.904+0000 7fac5ea00640 1 -- 172.21.15.40:0/2084574818 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) ==== 93+0+10 (secure 0 0 0) 0x7fac6005a910 con 0x7fac681517c0 2024-04-26T23:04:08.905 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:04:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.906+0000 7fac67e00640 1 -- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fac4406c920 msgr2=0x7fac4406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.906+0000 7fac67e00640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fac4406c920 0x7fac4406ede0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fac580017d0 tx=0x7fac58000f50 comp rx=0 tx=0).stop 2024-04-26T23:04:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.906+0000 7fac67e00640 1 -- 172.21.15.40:0/2084574818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac681517c0 msgr2=0x7fac6815a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.906+0000 7fac67e00640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac681517c0 0x7fac6815a660 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fac60009fd0 tx=0x7fac6000f520 comp rx=0 tx=0).stop 2024-04-26T23:04:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.907+0000 7fac67e00640 1 -- 172.21.15.40:0/2084574818 shutdown_connections 2024-04-26T23:04:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.907+0000 7fac67e00640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fac4406c920 0x7fac4406ede0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.907+0000 7fac67e00640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac681517c0 0x7fac6815a660 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.907+0000 7fac67e00640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fac68160d00 0x7fac6815a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.907+0000 7fac67e00640 1 --2- 172.21.15.40:0/2084574818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac4c004800 0x7fac681607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.907+0000 7fac67e00640 1 -- 172.21.15.40:0/2084574818 >> 172.21.15.40:0/2084574818 conn(0x7fac680b2720 msgr2=0x7fac6815f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.907+0000 7fac67e00640 1 -- 172.21.15.40:0/2084574818 shutdown_connections 2024-04-26T23:04:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:08.907+0000 7fac67e00640 1 -- 172.21.15.40:0/2084574818 wait complete. 2024-04-26T23:04:08.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_abort_mds0' 2024-04-26T23:04:09.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.137+0000 7f46b2400640 1 Processor -- start 2024-04-26T23:04:09.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.137+0000 7f46b2400640 1 -- start start 2024-04-26T23:04:09.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.138+0000 7f46b2400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac0cc480 0x7f46ac072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:09.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.138+0000 7f46b2400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46ac15d7c0 con 0x7f46ac072780 2024-04-26T23:04:09.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.139+0000 7f46b2400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46ac15daa0 con 0x7f46ac071e60 2024-04-26T23:04:09.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.139+0000 7f46b2400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f46ac15dd80 con 0x7f46ac0cc480 2024-04-26T23:04:09.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.139+0000 7f46abe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f46ac072780 0x7f46ac166ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37812/0 (socket says 172.21.15.40:37812) 2024-04-26T23:04:09.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.139+0000 7f46abe00640 1 -- 172.21.15.40:0/3692064899 learned_addr learned my addr 172.21.15.40:0/3692064899 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:09.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.139+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4177729123 0 0) 0x7f46ac15d7c0 con 0x7f46ac072780 2024-04-26T23:04:09.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.139+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4690003520 con 0x7f46ac072780 2024-04-26T23:04:09.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aaa00640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac0cc480 0x7f46ac072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:09.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 308642158 0 0) 0x7f4690003520 con 0x7f46ac072780 2024-04-26T23:04:09.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac0cc480 msgr2=0x7f46ac072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:09.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aa000640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac0cc480 0x7f46ac072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:09.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 >> v1:172.21.15.103:6789/0 conn(0x7f46ac071e60 legacy=0x7f46ac06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:09.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ac15e060 con 0x7f46ac072780 2024-04-26T23:04:09.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f46a4001670 con 0x7f46ac072780 2024-04-26T23:04:09.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 >> v1:172.21.15.40:6789/0 conn(0x7f46ac072780 legacy=0x7f46ac166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:09.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aa000640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900039e0 0x7f4690003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:09.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aa000640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4690004800 0x7f4690004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:09.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aa000640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac0cc480 0x7f4690005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:09.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac15dd80 con 0x7f4690004800 2024-04-26T23:04:09.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f469000ff00 con 0x7f46900039e0 2024-04-26T23:04:09.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4690010100 con 0x7f46ac0cc480 2024-04-26T23:04:09.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.140+0000 7f46aaa00640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4690004800 0x7f4690004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:09.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.141+0000 7f46abe00640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac0cc480 0x7f4690005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:09.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.141+0000 7f46ab400640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900039e0 0x7f4690003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:09.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.141+0000 7f46aaa00640 1 -- 172.21.15.40:0/3692064899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac0cc480 msgr2=0x7f4690005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:09.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.141+0000 7f46aaa00640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac0cc480 0x7f4690005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:09.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.141+0000 7f46aaa00640 1 -- 172.21.15.40:0/3692064899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900039e0 msgr2=0x7f4690003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:09.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.141+0000 7f46aaa00640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900039e0 0x7f4690003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:09.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.141+0000 7f46aaa00640 1 -- 172.21.15.40:0/3692064899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4690003520 con 0x7f4690004800 2024-04-26T23:04:09.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.141+0000 7f46ab400640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46900039e0 0x7f4690003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:09.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.141+0000 7f46abe00640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac0cc480 0x7f4690005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.142+0000 7f46aaa00640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4690004800 0x7f4690004c80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f469400ba30 tx=0x7f4690005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b9a24765b4ee1c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.142+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694030070 con 0x7f4690004800 2024-04-26T23:04:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.142+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f469402add0 con 0x7f4690004800 2024-04-26T23:04:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.142+0000 7f46aa000640 1 -- 172.21.15.40:0/3692064899 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469400b600 con 0x7f4690004800 2024-04-26T23:04:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.143+0000 7f46b2400640 1 -- 172.21.15.40:0/3692064899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4690004800 msgr2=0x7f4690004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.143+0000 7f46b2400640 1 --2- 172.21.15.40:0/3692064899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4690004800 0x7f4690004c80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f469400ba30 tx=0x7f4690005830 comp rx=0 tx=0).stop 2024-04-26T23:04:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.143+0000 7f46abe00640 1 -- 172.21.15.40:0/3692064899 reap_dead start 2024-04-26T23:04:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.143+0000 7f46b2400640 1 -- 172.21.15.40:0/3692064899 shutdown_connections 2024-04-26T23:04:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.143+0000 7f46b2400640 1 -- 172.21.15.40:0/3692064899 >> 172.21.15.40:0/3692064899 conn(0x7f46ac0d0be0 msgr2=0x7f46ac0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:09.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.143+0000 7f46b2400640 1 -- 172.21.15.40:0/3692064899 shutdown_connections 2024-04-26T23:04:09.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.143+0000 7f46b2400640 1 -- 172.21.15.40:0/3692064899 wait complete. 2024-04-26T23:04:09.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.145+0000 7f46b2400640 1 Processor -- start 2024-04-26T23:04:09.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.145+0000 7f46b2400640 1 -- start start 2024-04-26T23:04:09.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.146+0000 7f46b2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4690004800 0x7f46ac156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:09.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.146+0000 7f46b2400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46ac14ac60 0x7f46ac15b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:09.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.146+0000 7f46b2400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac15bb10 0x7f46ac1f0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:09.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.146+0000 7f46b2400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac15d3f0 con 0x7f4690004800 2024-04-26T23:04:09.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.146+0000 7f46b2400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac072c10 con 0x7f46ac14ac60 2024-04-26T23:04:09.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.146+0000 7f46b2400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f46ac15d570 con 0x7f46ac15bb10 2024-04-26T23:04:09.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.146+0000 7f46ab400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4690004800 0x7f46ac156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:09.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.146+0000 7f46ab400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4690004800 0x7f46ac156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60898/0 (socket says 172.21.15.40:60898) 2024-04-26T23:04:09.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.146+0000 7f46ab400640 1 -- 172.21.15.40:0/2662424422 learned_addr learned my addr 172.21.15.40:0/2662424422 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:09.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.146+0000 7f46abe00640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac15bb10 0x7f46ac1f0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:09.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.147+0000 7f46ab400640 1 -- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac15bb10 msgr2=0x7f46ac1f0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:09.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.147+0000 7f46ab400640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac15bb10 0x7f46ac1f0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:09.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.147+0000 7f46aaa00640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46ac14ac60 0x7f46ac15b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:09.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.147+0000 7f46ab400640 1 -- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46ac14ac60 msgr2=0x7f46ac15b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:09.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.147+0000 7f46ab400640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46ac14ac60 0x7f46ac15b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:09.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.147+0000 7f46ab400640 1 -- 172.21.15.40:0/2662424422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ac14b0c0 con 0x7f4690004800 2024-04-26T23:04:09.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.147+0000 7f46abe00640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac15bb10 0x7f46ac1f0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:09.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.147+0000 7f46ab400640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4690004800 0x7f46ac156300 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f469c00db70 tx=0x7f469c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:09.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.147+0000 7f46a3400640 1 -- 172.21.15.40:0/2662424422 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c00de70 con 0x7f4690004800 2024-04-26T23:04:09.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.147+0000 7f46a3400640 1 -- 172.21.15.40:0/2662424422 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f469c00bbb0 con 0x7f4690004800 2024-04-26T23:04:09.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.147+0000 7f46a3400640 1 -- 172.21.15.40:0/2662424422 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c006b50 con 0x7f4690004800 2024-04-26T23:04:09.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.147+0000 7f46b2400640 1 -- 172.21.15.40:0/2662424422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46ac0b6770 con 0x7f4690004800 2024-04-26T23:04:09.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.147+0000 7f46b2400640 1 -- 172.21.15.40:0/2662424422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46ac15a6c0 con 0x7f4690004800 2024-04-26T23:04:09.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.149+0000 7f46b2400640 1 -- 172.21.15.40:0/2662424422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46ac0694f0 con 0x7f4690004800 2024-04-26T23:04:09.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.149+0000 7f46a3400640 1 -- 172.21.15.40:0/2662424422 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f469c01a020 con 0x7f4690004800 2024-04-26T23:04:09.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.150+0000 7f46a3400640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f468806c940 0x7f468806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:09.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.150+0000 7f46a3400640 1 -- 172.21.15.40:0/2662424422 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5507+0+0 (secure 0 0 0) 0x7f469c08b680 con 0x7f4690004800 2024-04-26T23:04:09.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.151+0000 7f46aaa00640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f468806c940 0x7f468806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:09.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.152+0000 7f46aaa00640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f468806c940 0x7f468806ee00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f469400ba00 tx=0x7f4694030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:09.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.155+0000 7f46a3400640 1 -- 172.21.15.40:0/2662424422 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f46ac14b0c0 con 0x7f4690004800 2024-04-26T23:04:09.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.435+0000 7f46b2400640 1 -- 172.21.15.40:0/2662424422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_abort_mds0"]} v 0) -- 0x7f46ac15bef0 con 0x7f4690004800 2024-04-26T23:04:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.988+0000 7f46a3400640 1 -- 172.21.15.40:0/2662424422 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_abort_mds0"]}]=0 v177) ==== 146+0+0 (secure 0 0 0) 0x7f469c059b50 con 0x7f4690004800 2024-04-26T23:04:09.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.992+0000 7f46b2400640 1 -- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f468806c940 msgr2=0x7f468806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:09.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.992+0000 7f46b2400640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f468806c940 0x7f468806ee00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f469400ba00 tx=0x7f4694030000 comp rx=0 tx=0).stop 2024-04-26T23:04:09.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.992+0000 7f46b2400640 1 -- 172.21.15.40:0/2662424422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4690004800 msgr2=0x7f46ac156300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:09.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.992+0000 7f46b2400640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4690004800 0x7f46ac156300 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f469c00db70 tx=0x7f469c00b4a0 comp rx=0 tx=0).stop 2024-04-26T23:04:09.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.993+0000 7f46b2400640 1 -- 172.21.15.40:0/2662424422 shutdown_connections 2024-04-26T23:04:09.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.993+0000 7f46b2400640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f468806c940 0x7f468806ee00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:09.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.993+0000 7f46b2400640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46ac15bb10 0x7f46ac1f0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:09.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.993+0000 7f46b2400640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46ac14ac60 0x7f46ac15b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:09.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.993+0000 7f46b2400640 1 --2- 172.21.15.40:0/2662424422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4690004800 0x7f46ac156300 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:09.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.993+0000 7f46b2400640 1 -- 172.21.15.40:0/2662424422 >> 172.21.15.40:0/2662424422 conn(0x7f46ac0d0be0 msgr2=0x7f46ac06fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:09.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.993+0000 7f46b2400640 1 -- 172.21.15.40:0/2662424422 shutdown_connections 2024-04-26T23:04:09.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:09.993+0000 7f46b2400640 1 -- 172.21.15.40:0/2662424422 wait complete. 2024-04-26T23:04:10.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-26T23:04:10.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.226+0000 7f3eb0200640 1 Processor -- start 2024-04-26T23:04:10.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.226+0000 7f3eb0200640 1 -- start start 2024-04-26T23:04:10.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.227+0000 7f3eb0200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ea80cbb50 0x7f3ea80af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:10.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.227+0000 7f3eb0200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ea8072c00 con 0x7f3ea815f380 2024-04-26T23:04:10.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.227+0000 7f3eb0200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ea8072ee0 con 0x7f3ea8073b70 2024-04-26T23:04:10.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.227+0000 7f3eb0200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3ea80729b0 con 0x7f3ea80cbb50 2024-04-26T23:04:10.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.228+0000 7f3ead800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3ea815f380 0x7f3ea80cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37828/0 (socket says 172.21.15.40:37828) 2024-04-26T23:04:10.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.228+0000 7f3ead800640 1 -- 172.21.15.40:0/4267076829 learned_addr learned my addr 172.21.15.40:0/4267076829 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:10.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.228+0000 7f3eae200640 1 --2- 172.21.15.40:0/4267076829 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ea80cbb50 0x7f3ea80af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:10.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.228+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3684285304 0 0) 0x7f3ea8072c00 con 0x7f3ea815f380 2024-04-26T23:04:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.228+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e90003520 con 0x7f3ea815f380 2024-04-26T23:04:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1299504173 0 0) 0x7f3e90003520 con 0x7f3ea815f380 2024-04-26T23:04:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ea80cbb50 msgr2=0x7f3ea80af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3ea7e00640 1 --2- 172.21.15.40:0/4267076829 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ea80cbb50 0x7f3ea80af3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 >> v1:172.21.15.103:6789/0 conn(0x7f3ea8073b70 legacy=0x7f3ea80aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea80731c0 con 0x7f3ea815f380 2024-04-26T23:04:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f3e9c001670 con 0x7f3ea815f380 2024-04-26T23:04:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 >> v1:172.21.15.40:6789/0 conn(0x7f3ea815f380 legacy=0x7f3ea80cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3ea7e00640 1 --2- 172.21.15.40:0/4267076829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e900039e0 0x7f3e90003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3ea7e00640 1 --2- 172.21.15.40:0/4267076829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e90004800 0x7f3e90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3ea7e00640 1 --2- 172.21.15.40:0/4267076829 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ea80cbb50 0x7f3e900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea80729b0 con 0x7f3e90004800 2024-04-26T23:04:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9000baf0 con 0x7f3e900039e0 2024-04-26T23:04:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3e9000bcf0 con 0x7f3ea80cbb50 2024-04-26T23:04:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.229+0000 7f3eace00640 1 --2- 172.21.15.40:0/4267076829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e90004800 0x7f3e90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.230+0000 7f3ead800640 1 --2- 172.21.15.40:0/4267076829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e900039e0 0x7f3e90003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.230+0000 7f3eace00640 1 -- 172.21.15.40:0/4267076829 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ea80cbb50 msgr2=0x7f3e900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.230+0000 7f3eace00640 1 --2- 172.21.15.40:0/4267076829 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ea80cbb50 0x7f3e900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.230+0000 7f3eace00640 1 -- 172.21.15.40:0/4267076829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e900039e0 msgr2=0x7f3e90003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.230+0000 7f3eace00640 1 --2- 172.21.15.40:0/4267076829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e900039e0 0x7f3e90003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.230+0000 7f3eace00640 1 -- 172.21.15.40:0/4267076829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e90003520 con 0x7f3e90004800 2024-04-26T23:04:10.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.231+0000 7f3eace00640 1 --2- 172.21.15.40:0/4267076829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e90004800 0x7f3e90004c80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3e98001e50 tx=0x7f3e90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7c170aa5acce45 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:10.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.231+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e98001a70 con 0x7f3e90004800 2024-04-26T23:04:10.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.231+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e98003970 con 0x7f3e90004800 2024-04-26T23:04:10.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.231+0000 7f3ea7e00640 1 -- 172.21.15.40:0/4267076829 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e98003d60 con 0x7f3e90004800 2024-04-26T23:04:10.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.232+0000 7f3eb0200640 1 -- 172.21.15.40:0/4267076829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e90004800 msgr2=0x7f3e90004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:10.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.232+0000 7f3eb0200640 1 --2- 172.21.15.40:0/4267076829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e90004800 0x7f3e90004c80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3e98001e50 tx=0x7f3e90005830 comp rx=0 tx=0).stop 2024-04-26T23:04:10.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.232+0000 7f3eae200640 1 -- 172.21.15.40:0/4267076829 reap_dead start 2024-04-26T23:04:10.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.232+0000 7f3eb0200640 1 -- 172.21.15.40:0/4267076829 shutdown_connections 2024-04-26T23:04:10.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.232+0000 7f3eb0200640 1 -- 172.21.15.40:0/4267076829 >> 172.21.15.40:0/4267076829 conn(0x7f3ea80d0be0 msgr2=0x7f3ea80d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:10.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.232+0000 7f3eb0200640 1 -- 172.21.15.40:0/4267076829 shutdown_connections 2024-04-26T23:04:10.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.232+0000 7f3eb0200640 1 -- 172.21.15.40:0/4267076829 wait complete. 2024-04-26T23:04:10.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.234+0000 7f3eb0200640 1 Processor -- start 2024-04-26T23:04:10.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.234+0000 7f3eb0200640 1 -- start start 2024-04-26T23:04:10.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.235+0000 7f3eb0200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e90004800 0x7f3ea80c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:10.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.235+0000 7f3eb0200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ea80b4290 0x7f3ea8151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:10.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.235+0000 7f3eb0200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ea80c94c0 0x7f3ea80c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:10.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.235+0000 7f3eb0200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea8073620 con 0x7f3ea80c94c0 2024-04-26T23:04:10.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.235+0000 7f3eb0200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea80d08f0 con 0x7f3ea80b4290 2024-04-26T23:04:10.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.235+0000 7f3eb0200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3ea80737a0 con 0x7f3e90004800 2024-04-26T23:04:10.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.235+0000 7f3eae200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ea80c94c0 0x7f3ea80c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:10.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.235+0000 7f3eae200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ea80c94c0 0x7f3ea80c9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60902/0 (socket says 172.21.15.40:60902) 2024-04-26T23:04:10.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.235+0000 7f3eae200640 1 -- 172.21.15.40:0/2754015807 learned_addr learned my addr 172.21.15.40:0/2754015807 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:10.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.236+0000 7f3ead800640 1 --2- 172.21.15.40:0/2754015807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e90004800 0x7f3ea80c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:10.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.236+0000 7f3eae200640 1 -- 172.21.15.40:0/2754015807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e90004800 msgr2=0x7f3ea80c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:10.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.236+0000 7f3eae200640 1 --2- 172.21.15.40:0/2754015807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e90004800 0x7f3ea80c8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:10.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.236+0000 7f3eae200640 1 -- 172.21.15.40:0/2754015807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ea80b4290 msgr2=0x7f3ea8151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:10.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.236+0000 7f3eae200640 1 --2- 172.21.15.40:0/2754015807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ea80b4290 0x7f3ea8151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:10.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.236+0000 7f3eae200640 1 -- 172.21.15.40:0/2754015807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea80c4d80 con 0x7f3ea80c94c0 2024-04-26T23:04:10.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.236+0000 7f3eae200640 1 --2- 172.21.15.40:0/2754015807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ea80c94c0 0x7f3ea80c9940 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8071b50 tx=0x7f3ea000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:10.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.236+0000 7f3ea5600640 1 -- 172.21.15.40:0/2754015807 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea000c620 con 0x7f3ea80c94c0 2024-04-26T23:04:10.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.236+0000 7f3ea5600640 1 -- 172.21.15.40:0/2754015807 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ea001a070 con 0x7f3ea80c94c0 2024-04-26T23:04:10.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.237+0000 7f3eb0200640 1 -- 172.21.15.40:0/2754015807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ea8151550 con 0x7f3ea80c94c0 2024-04-26T23:04:10.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.237+0000 7f3ea5600640 1 -- 172.21.15.40:0/2754015807 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea000ca00 con 0x7f3ea80c94c0 2024-04-26T23:04:10.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.237+0000 7f3eb0200640 1 -- 172.21.15.40:0/2754015807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ea80b4d60 con 0x7f3ea80c94c0 2024-04-26T23:04:10.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.238+0000 7f3eb0200640 1 -- 172.21.15.40:0/2754015807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ea80694f0 con 0x7f3ea80c94c0 2024-04-26T23:04:10.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.239+0000 7f3ea5600640 1 -- 172.21.15.40:0/2754015807 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3ea000cba0 con 0x7f3ea80c94c0 2024-04-26T23:04:10.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.240+0000 7f3ea5600640 1 --2- 172.21.15.40:0/2754015807 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e8c06c940 0x7f3e8c06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:10.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.240+0000 7f3ea5600640 1 -- 172.21.15.40:0/2754015807 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5507+0+0 (secure 0 0 0) 0x7f3ea008b170 con 0x7f3ea80c94c0 2024-04-26T23:04:10.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.241+0000 7f3ead800640 1 --2- 172.21.15.40:0/2754015807 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e8c06c940 0x7f3e8c06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:10.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.242+0000 7f3ead800640 1 --2- 172.21.15.40:0/2754015807 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e8c06c940 0x7f3e8c06ee00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c0017b0 tx=0x7f3e9c000ed0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:10.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.245+0000 7f3ea5600640 1 -- 172.21.15.40:0/2754015807 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3ea0023020 con 0x7f3ea80c94c0 2024-04-26T23:04:10.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.454+0000 7f3eb0200640 1 -- 172.21.15.40:0/2754015807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f3ea80cfa00 con 0x7f3ea80c94c0 2024-04-26T23:04:10.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.460+0000 7f3ea5600640 1 -- 172.21.15.40:0/2754015807 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f3ea0059740 con 0x7f3ea80c94c0 2024-04-26T23:04:10.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.463+0000 7f3eb0200640 1 -- 172.21.15.40:0/2754015807 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e8c06c940 msgr2=0x7f3e8c06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:10.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.463+0000 7f3eb0200640 1 --2- 172.21.15.40:0/2754015807 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e8c06c940 0x7f3e8c06ee00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c0017b0 tx=0x7f3e9c000ed0 comp rx=0 tx=0).stop 2024-04-26T23:04:10.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.463+0000 7f3eb0200640 1 -- 172.21.15.40:0/2754015807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ea80c94c0 msgr2=0x7f3ea80c9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:10.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.463+0000 7f3eb0200640 1 --2- 172.21.15.40:0/2754015807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ea80c94c0 0x7f3ea80c9940 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8071b50 tx=0x7f3ea000ec00 comp rx=0 tx=0).stop 2024-04-26T23:04:10.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.464+0000 7f3eb0200640 1 -- 172.21.15.40:0/2754015807 shutdown_connections 2024-04-26T23:04:10.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.464+0000 7f3eb0200640 1 --2- 172.21.15.40:0/2754015807 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e8c06c940 0x7f3e8c06ee00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:10.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.464+0000 7f3eb0200640 1 --2- 172.21.15.40:0/2754015807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ea80c94c0 0x7f3ea80c9940 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:10.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.464+0000 7f3eb0200640 1 --2- 172.21.15.40:0/2754015807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ea80b4290 0x7f3ea8151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:10.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.464+0000 7f3eb0200640 1 --2- 172.21.15.40:0/2754015807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e90004800 0x7f3ea80c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:10.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.464+0000 7f3eb0200640 1 -- 172.21.15.40:0/2754015807 >> 172.21.15.40:0/2754015807 conn(0x7f3ea80d0be0 msgr2=0x7f3ea8071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:10.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.464+0000 7f3eb0200640 1 -- 172.21.15.40:0/2754015807 shutdown_connections 2024-04-26T23:04:10.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:10.464+0000 7f3eb0200640 1 -- 172.21.15.40:0/2754015807 wait complete. 2024-04-26T23:04:10.483 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-26T23:04:10.483 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:04:10.584 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:04:10.701 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-26T23:04:10.701 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:04:10.701 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:04:10.763 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:04:10.763 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:04:10.763 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:04:10.822 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:04:10.822 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:04:10.823 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:04:10.883 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:04:10.883 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:04:10.884 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:04:10.941 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:04:10.941 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-26T23:04:10.942 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:04:11.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.160+0000 7fd065a00640 1 Processor -- start 2024-04-26T23:04:11.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.160+0000 7fd065a00640 1 -- start start 2024-04-26T23:04:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.162+0000 7fd065a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0600cc980 0x7fd0600cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.162+0000 7fd065a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0600cce10 con 0x7fd0600c6f10 2024-04-26T23:04:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.162+0000 7fd065a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0600cd0f0 con 0x7fd0600cc230 2024-04-26T23:04:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.162+0000 7fd065a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd0600c72f0 con 0x7fd0600cc980 2024-04-26T23:04:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.162+0000 7fd05fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd0600c6f10 0x7fd0600c5ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37836/0 (socket says 172.21.15.40:37836) 2024-04-26T23:04:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.162+0000 7fd05fe00640 1 -- 172.21.15.40:0/3446800221 learned_addr learned my addr 172.21.15.40:0/3446800221 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.163+0000 7fd057400640 1 --2- 172.21.15.40:0/3446800221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0600cc980 0x7fd0600cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.163+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 233058191 0 0) 0x7fd0600cce10 con 0x7fd0600c6f10 2024-04-26T23:04:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.163+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd044003520 con 0x7fd0600c6f10 2024-04-26T23:04:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.163+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2304613483 0 0) 0x7fd044003520 con 0x7fd0600c6f10 2024-04-26T23:04:11.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.163+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0600cc980 msgr2=0x7fd0600cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:11.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.163+0000 7fd05f400640 1 --2- 172.21.15.40:0/3446800221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0600cc980 0x7fd0600cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.163+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 >> v1:172.21.15.103:6789/0 conn(0x7fd0600cc230 legacy=0x7fd060158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:11.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.163+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0600cb4f0 con 0x7fd0600c6f10 2024-04-26T23:04:11.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.163+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fd050001670 con 0x7fd0600c6f10 2024-04-26T23:04:11.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.163+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 >> v1:172.21.15.40:6789/0 conn(0x7fd0600c6f10 legacy=0x7fd0600c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:11.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd05f400640 1 --2- 172.21.15.40:0/3446800221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0440039e0 0x7fd044003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd05f400640 1 --2- 172.21.15.40:0/3446800221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd044004390 0x7fd044004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd05f400640 1 --2- 172.21.15.40:0/3446800221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd044005060 0x7fd04400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd0600c72f0 con 0x7fd044004390 2024-04-26T23:04:11.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd04400bf80 con 0x7fd044005060 2024-04-26T23:04:11.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd04400c180 con 0x7fd0440039e0 2024-04-26T23:04:11.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd057400640 1 --2- 172.21.15.40:0/3446800221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd044004390 0x7fd044004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd05fe00640 1 --2- 172.21.15.40:0/3446800221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0440039e0 0x7fd044003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd065000640 1 --2- 172.21.15.40:0/3446800221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd044005060 0x7fd04400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd057400640 1 -- 172.21.15.40:0/3446800221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0440039e0 msgr2=0x7fd044003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:11.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd057400640 1 --2- 172.21.15.40:0/3446800221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0440039e0 0x7fd044003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd057400640 1 -- 172.21.15.40:0/3446800221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd044005060 msgr2=0x7fd04400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:11.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd057400640 1 --2- 172.21.15.40:0/3446800221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd044005060 0x7fd04400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.164+0000 7fd057400640 1 -- 172.21.15.40:0/3446800221 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd044003520 con 0x7fd044004390 2024-04-26T23:04:11.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.165+0000 7fd065000640 1 --2- 172.21.15.40:0/3446800221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd044005060 0x7fd04400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:11.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.166+0000 7fd057400640 1 --2- 172.21.15.40:0/3446800221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd044004390 0x7fd044004860 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd048009ac0 tx=0x7fd044005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5829b251524ea117 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:11.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.166+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048036070 con 0x7fd044004390 2024-04-26T23:04:11.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.166+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd048031400 con 0x7fd044004390 2024-04-26T23:04:11.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.166+0000 7fd05f400640 1 -- 172.21.15.40:0/3446800221 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048009690 con 0x7fd044004390 2024-04-26T23:04:11.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.167+0000 7fd065a00640 1 -- 172.21.15.40:0/3446800221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd044004390 msgr2=0x7fd044004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:11.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.167+0000 7fd065a00640 1 --2- 172.21.15.40:0/3446800221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd044004390 0x7fd044004860 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd048009ac0 tx=0x7fd044005830 comp rx=0 tx=0).stop 2024-04-26T23:04:11.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.167+0000 7fd065000640 1 -- 172.21.15.40:0/3446800221 reap_dead start 2024-04-26T23:04:11.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.167+0000 7fd065a00640 1 -- 172.21.15.40:0/3446800221 shutdown_connections 2024-04-26T23:04:11.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.167+0000 7fd065a00640 1 -- 172.21.15.40:0/3446800221 >> 172.21.15.40:0/3446800221 conn(0x7fd0600d0a10 msgr2=0x7fd0600d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:11.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.167+0000 7fd065a00640 1 -- 172.21.15.40:0/3446800221 shutdown_connections 2024-04-26T23:04:11.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.167+0000 7fd065a00640 1 -- 172.21.15.40:0/3446800221 wait complete. 2024-04-26T23:04:11.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.170+0000 7fd065a00640 1 Processor -- start 2024-04-26T23:04:11.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.170+0000 7fd065a00640 1 -- start start 2024-04-26T23:04:11.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.171+0000 7fd065a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd044004390 0x7fd0600b5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.171+0000 7fd065a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0601511a0 0x7fd060151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.171+0000 7fd065a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd060071ce0 0x7fd060072140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.171+0000 7fd065a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd0600c61e0 con 0x7fd060071ce0 2024-04-26T23:04:11.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.171+0000 7fd065a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd0600cb860 con 0x7fd0601511a0 2024-04-26T23:04:11.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.171+0000 7fd065a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd0600c6360 con 0x7fd044004390 2024-04-26T23:04:11.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.171+0000 7fd065000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd060071ce0 0x7fd060072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.172+0000 7fd065000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd060071ce0 0x7fd060072140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60908/0 (socket says 172.21.15.40:60908) 2024-04-26T23:04:11.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.172+0000 7fd065000640 1 -- 172.21.15.40:0/2713991492 learned_addr learned my addr 172.21.15.40:0/2713991492 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:11.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.172+0000 7fd05fe00640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd044004390 0x7fd0600b5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.172+0000 7fd065000640 1 -- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd044004390 msgr2=0x7fd0600b5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:11.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.172+0000 7fd065000640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd044004390 0x7fd0600b5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.172+0000 7fd065000640 1 -- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0601511a0 msgr2=0x7fd060151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:11.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.172+0000 7fd057400640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0601511a0 0x7fd060151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.172+0000 7fd065000640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0601511a0 0x7fd060151620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.172+0000 7fd065000640 1 -- 172.21.15.40:0/2713991492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd060151b60 con 0x7fd060071ce0 2024-04-26T23:04:11.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.172+0000 7fd065000640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd060071ce0 0x7fd060072140 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd0600cb7d0 tx=0x7fd05800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:11.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.173+0000 7fd05cc00640 1 -- 172.21.15.40:0/2713991492 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd058006d50 con 0x7fd060071ce0 2024-04-26T23:04:11.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.173+0000 7fd05cc00640 1 -- 172.21.15.40:0/2713991492 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd058003970 con 0x7fd060071ce0 2024-04-26T23:04:11.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.173+0000 7fd05cc00640 1 -- 172.21.15.40:0/2713991492 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd058003da0 con 0x7fd060071ce0 2024-04-26T23:04:11.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.173+0000 7fd065a00640 1 -- 172.21.15.40:0/2713991492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0600729d0 con 0x7fd060071ce0 2024-04-26T23:04:11.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.173+0000 7fd065a00640 1 -- 172.21.15.40:0/2713991492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd060072f90 con 0x7fd060071ce0 2024-04-26T23:04:11.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.174+0000 7fd065a00640 1 -- 172.21.15.40:0/2713991492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd060069370 con 0x7fd060071ce0 2024-04-26T23:04:11.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.175+0000 7fd05cc00640 1 -- 172.21.15.40:0/2713991492 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd05801b020 con 0x7fd060071ce0 2024-04-26T23:04:11.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.176+0000 7fd05cc00640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd03c06c8f0 0x7fd03c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.177+0000 7fd05cc00640 1 -- 172.21.15.40:0/2713991492 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5507+0+0 (secure 0 0 0) 0x7fd05808b740 con 0x7fd060071ce0 2024-04-26T23:04:11.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.177+0000 7fd05fe00640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd03c06c8f0 0x7fd03c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.178+0000 7fd05fe00640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd03c06c8f0 0x7fd03c06edb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd050001050 tx=0x7fd050009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:11.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.181+0000 7fd05cc00640 1 -- 172.21.15.40:0/2713991492 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd058059c20 con 0x7fd060071ce0 2024-04-26T23:04:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.425+0000 7fd065a00640 1 -- 172.21.15.40:0/2713991492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0600c58b0 con 0x7fd060071ce0 2024-04-26T23:04:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.426+0000 7fd05cc00640 1 -- 172.21.15.40:0/2713991492 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+3872 (secure 0 0 0) 0x7fd058059520 con 0x7fd060071ce0 2024-04-26T23:04:11.427 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:11.427 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1979956236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":1979956236},{"type":"v1","addr":"172.21.15.103:6837","nonce":1979956236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/491467283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":491467283},{"type":"v1","addr":"172.21.15.103:6836","nonce":491467283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1749141671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1749141671},{"type":"v1","addr":"172.21.15.40:6833","nonce":1749141671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5937,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1898841398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1898841398},{"type":"v1","addr":"172.21.15.40:6835","nonce":1898841398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[]} 2024-04-26T23:04:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 28 2024-04-26T23:04:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.429+0000 7fd065a00640 1 -- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd03c06c8f0 msgr2=0x7fd03c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.429+0000 7fd065a00640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd03c06c8f0 0x7fd03c06edb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd050001050 tx=0x7fd050009040 comp rx=0 tx=0).stop 2024-04-26T23:04:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.429+0000 7fd065a00640 1 -- 172.21.15.40:0/2713991492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd060071ce0 msgr2=0x7fd060072140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.429+0000 7fd065a00640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd060071ce0 0x7fd060072140 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd0600cb7d0 tx=0x7fd05800dc00 comp rx=0 tx=0).stop 2024-04-26T23:04:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.429+0000 7fd065a00640 1 -- 172.21.15.40:0/2713991492 shutdown_connections 2024-04-26T23:04:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.429+0000 7fd065a00640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd03c06c8f0 0x7fd03c06edb0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.429+0000 7fd065a00640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd060071ce0 0x7fd060072140 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.429+0000 7fd065a00640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0601511a0 0x7fd060151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.429+0000 7fd065a00640 1 --2- 172.21.15.40:0/2713991492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd044004390 0x7fd0600b5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.429+0000 7fd065a00640 1 -- 172.21.15.40:0/2713991492 >> 172.21.15.40:0/2713991492 conn(0x7fd0600d0a10 msgr2=0x7fd0600ce570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.429+0000 7fd065a00640 1 -- 172.21.15.40:0/2713991492 shutdown_connections 2024-04-26T23:04:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.429+0000 7fd065a00640 1 -- 172.21.15.40:0/2713991492 wait complete. 2024-04-26T23:04:11.449 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-26T23:04:11.449 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-26T23:04:11.450 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-26T23:04:11.450 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:04:11.450 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-26T23:04:11.450 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-26T23:04:11.450 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-26T23:04:11.450 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:04:11.451 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-26T23:04:11.451 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-26T23:04:11.451 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-26T23:04:11.451 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:04:11.451 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-26T23:04:11.451 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-26T23:04:11.451 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-26T23:04:11.451 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:04:11.469 INFO:tasks.ceph.mds.a:Stopped 2024-04-26T23:04:11.470 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-26T23:04:11.471 INFO:tasks.ceph.mds.c:Stopped 2024-04-26T23:04:11.471 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-26T23:04:11.474 INFO:tasks.ceph.mds.d:Stopped 2024-04-26T23:04:11.474 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-26T23:04:11.475 INFO:tasks.ceph.mds.b:Stopped 2024-04-26T23:04:11.475 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-26T23:04:11.516 INFO:tasks.ceph.mds.a:Started 2024-04-26T23:04:11.516 DEBUG:teuthology.parallel:result is None 2024-04-26T23:04:11.521 INFO:tasks.ceph.mds.c:Started 2024-04-26T23:04:11.521 DEBUG:teuthology.parallel:result is None 2024-04-26T23:04:11.523 INFO:tasks.ceph.mds.d:Started 2024-04-26T23:04:11.524 DEBUG:teuthology.parallel:result is None 2024-04-26T23:04:11.529 INFO:tasks.ceph.mds.b:Started 2024-04-26T23:04:11.529 DEBUG:teuthology.parallel:result is None 2024-04-26T23:04:11.530 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-26T23:04:11.669 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-04-26T23:04:11.669 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:04:11.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-04-26T23:04:11.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-04-26T23:04:11.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-04-26T23:04:11.678 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-26T23:04:11.681 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-26T23:04:11.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.713+0000 7f7bd5400640 1 Processor -- start 2024-04-26T23:04:11.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.713+0000 7f7bd5400640 1 -- start start 2024-04-26T23:04:11.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.714+0000 7f7bd5400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7bd00c41d0 0x7f7bd00cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.714+0000 7f7bd5400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7bd0073f30 0x7f7bd01671e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.714+0000 7f7bd5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bd0167720 0x7f7bd006ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.714+0000 7f7bd5400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f7bd00cf7a0 con 0x7f7bd00c41d0 2024-04-26T23:04:11.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.714+0000 7f7bd5400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7bd0163a50 con 0x7f7bd0073f30 2024-04-26T23:04:11.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.714+0000 7f7bd5400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7bd00ceb90 con 0x7f7bd0167720 2024-04-26T23:04:11.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.714+0000 7f7bce800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7bd00c41d0 0x7f7bd00cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.714+0000 7f7bce800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7bd00c41d0 0x7f7bd00cfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60970/0 (socket says 172.21.15.40:60970) 2024-04-26T23:04:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.714+0000 7f7bce800640 1 -- 172.21.15.40:0/3745938906 learned_addr learned my addr 172.21.15.40:0/3745938906 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.715+0000 7f7bc5e00640 1 --2- 172.21.15.40:0/3745938906 >> v2:172.21.15.103:3300/0 conn(0x7f7bd0073f30 0x7f7bd01671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.715+0000 7f7bcf200640 1 --2- 172.21.15.40:0/3745938906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bd0167720 0x7f7bd006ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.715+0000 7f7bce800640 1 -- 172.21.15.40:0/3745938906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bd0167720 msgr2=0x7f7bd006ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.715+0000 7f7bce800640 1 --2- 172.21.15.40:0/3745938906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bd0167720 0x7f7bd006ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.715+0000 7f7bce800640 1 -- 172.21.15.40:0/3745938906 >> v2:172.21.15.103:3300/0 conn(0x7f7bd0073f30 msgr2=0x7f7bd01671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.715+0000 7f7bce800640 1 --2- 172.21.15.40:0/3745938906 >> v2:172.21.15.103:3300/0 conn(0x7f7bd0073f30 0x7f7bd01671e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.715+0000 7f7bce800640 1 -- 172.21.15.40:0/3745938906 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd00cee70 con 0x7f7bd00c41d0 2024-04-26T23:04:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.716+0000 7f7bce800640 1 --2- 172.21.15.40:0/3745938906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bd00c41d0 0x7f7bd00cfc30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00d0700 tx=0x7f7bc002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c1b67ca263ccb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.716+0000 7f7bcde00640 1 -- 172.21.15.40:0/3745938906 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc0036070 con 0x7f7bd00c41d0 2024-04-26T23:04:11.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.716+0000 7f7bcde00640 1 -- 172.21.15.40:0/3745938906 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bc0031600 con 0x7f7bd00c41d0 2024-04-26T23:04:11.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.716+0000 7f7bcde00640 1 -- 172.21.15.40:0/3745938906 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc00319d0 con 0x7f7bd00c41d0 2024-04-26T23:04:11.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.717+0000 7f7bd5400640 1 -- 172.21.15.40:0/3745938906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bd00c41d0 msgr2=0x7f7bd00cfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:11.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.717+0000 7f7bd5400640 1 --2- 172.21.15.40:0/3745938906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bd00c41d0 0x7f7bd00cfc30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00d0700 tx=0x7f7bc002a6a0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.717+0000 7f7bd5400640 1 -- 172.21.15.40:0/3745938906 shutdown_connections 2024-04-26T23:04:11.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.717+0000 7f7bd5400640 1 --2- 172.21.15.40:0/3745938906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bd0167720 0x7f7bd006ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.717+0000 7f7bd5400640 1 --2- 172.21.15.40:0/3745938906 >> v2:172.21.15.103:3300/0 conn(0x7f7bd0073f30 0x7f7bd01671e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.717+0000 7f7bd5400640 1 --2- 172.21.15.40:0/3745938906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bd00c41d0 0x7f7bd00cfc30 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.717+0000 7f7bd5400640 1 -- 172.21.15.40:0/3745938906 >> 172.21.15.40:0/3745938906 conn(0x7f7bd00d0be0 msgr2=0x7f7bd00d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:11.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.717+0000 7f7bd5400640 1 -- 172.21.15.40:0/3745938906 shutdown_connections 2024-04-26T23:04:11.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.717+0000 7f7bd5400640 1 -- 172.21.15.40:0/3745938906 wait complete. 2024-04-26T23:04:11.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.719+0000 7f7bd5400640 1 Processor -- start 2024-04-26T23:04:11.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.719+0000 7f7bd5400640 1 -- start start 2024-04-26T23:04:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.720+0000 7f7bd5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bd0073f30 0x7f7bd015ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.720+0000 7f7bd5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd00c41d0 0x7f7bd01581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.720+0000 7f7bd5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bd0167720 0x7f7bd00b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.720+0000 7f7bd5400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd00d0170 con 0x7f7bd0073f30 2024-04-26T23:04:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.720+0000 7f7bd5400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd00cf430 con 0x7f7bd00c41d0 2024-04-26T23:04:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.720+0000 7f7bd5400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7bd00ce7b0 con 0x7f7bd0167720 2024-04-26T23:04:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.720+0000 7f7bce800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bd0073f30 0x7f7bd015ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.720+0000 7f7bce800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bd0073f30 0x7f7bd015ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60978/0 (socket says 172.21.15.40:60978) 2024-04-26T23:04:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.720+0000 7f7bce800640 1 -- 172.21.15.40:0/1052294435 learned_addr learned my addr 172.21.15.40:0/1052294435 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:11.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.721+0000 7f7bcf200640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bd0167720 0x7f7bd00b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.721+0000 7f7bc5e00640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd00c41d0 0x7f7bd01581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.721+0000 7f7bce800640 1 -- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bd0167720 msgr2=0x7f7bd00b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:11.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.721+0000 7f7bce800640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bd0167720 0x7f7bd00b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.721+0000 7f7bce800640 1 -- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd00c41d0 msgr2=0x7f7bd01581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:11.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.721+0000 7f7bce800640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd00c41d0 0x7f7bd01581f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:11.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.721+0000 7f7bce800640 1 -- 172.21.15.40:0/1052294435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd00b4090 con 0x7f7bd0073f30 2024-04-26T23:04:11.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.721+0000 7f7bce800640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bd0073f30 0x7f7bd015ee30 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0034040 tx=0x7f7bc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.722+0000 7f7bc7400640 1 -- 172.21.15.40:0/1052294435 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc0036050 con 0x7f7bd0073f30 2024-04-26T23:04:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.722+0000 7f7bc7400640 1 -- 172.21.15.40:0/1052294435 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bc0042070 con 0x7f7bd0073f30 2024-04-26T23:04:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.722+0000 7f7bc7400640 1 -- 172.21.15.40:0/1052294435 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc0009510 con 0x7f7bd0073f30 2024-04-26T23:04:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.722+0000 7f7bd5400640 1 -- 172.21.15.40:0/1052294435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd015f410 con 0x7f7bd0073f30 2024-04-26T23:04:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.722+0000 7f7bd5400640 1 -- 172.21.15.40:0/1052294435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd015f9d0 con 0x7f7bd0073f30 2024-04-26T23:04:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.723+0000 7f7bd5400640 1 -- 172.21.15.40:0/1052294435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bd00694f0 con 0x7f7bd0073f30 2024-04-26T23:04:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.724+0000 7f7bc7400640 1 -- 172.21.15.40:0/1052294435 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7bc0007c50 con 0x7f7bd0073f30 2024-04-26T23:04:11.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.725+0000 7f7bc7400640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7bac06c940 0x7f7bac06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:11.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.725+0000 7f7bc7400640 1 -- 172.21.15.40:0/1052294435 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5507+0+0 (secure 0 0 0) 0x7f7bc00ac6c0 con 0x7f7bd0073f30 2024-04-26T23:04:11.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.725+0000 7f7bc5e00640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7bac06c940 0x7f7bac06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:11.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.726+0000 7f7bc5e00640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7bac06c940 0x7f7bac06ee00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7bd01672d0 tx=0x7f7bb8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:11.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.729+0000 7f7bc7400640 1 -- 172.21.15.40:0/1052294435 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7bc007ac20 con 0x7f7bd0073f30 2024-04-26T23:04:11.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:11.947+0000 7f7bd5400640 1 -- 172.21.15.40:0/1052294435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f7bd00d0430 con 0x7f7bd0073f30 2024-04-26T23:04:12.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.011+0000 7f7bc7400640 1 -- 172.21.15.40:0/1052294435 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v46) ==== 97+0+0 (secure 0 0 0) 0x7f7bc007a520 con 0x7f7bd0073f30 2024-04-26T23:04:12.011 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-04-26T23:04:12.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.015+0000 7f7bd5400640 1 -- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7bac06c940 msgr2=0x7f7bac06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.015+0000 7f7bd5400640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7bac06c940 0x7f7bac06ee00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7bd01672d0 tx=0x7f7bb8008040 comp rx=0 tx=0).stop 2024-04-26T23:04:12.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.015+0000 7f7bd5400640 1 -- 172.21.15.40:0/1052294435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bd0073f30 msgr2=0x7f7bd015ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.015+0000 7f7bd5400640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bd0073f30 0x7f7bd015ee30 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0034040 tx=0x7f7bc0004a40 comp rx=0 tx=0).stop 2024-04-26T23:04:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.015+0000 7f7bd5400640 1 -- 172.21.15.40:0/1052294435 shutdown_connections 2024-04-26T23:04:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.015+0000 7f7bd5400640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7bac06c940 0x7f7bac06ee00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.015+0000 7f7bd5400640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bd0167720 0x7f7bd00b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.015+0000 7f7bd5400640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bd00c41d0 0x7f7bd01581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.015+0000 7f7bd5400640 1 --2- 172.21.15.40:0/1052294435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bd0073f30 0x7f7bd015ee30 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.015+0000 7f7bd5400640 1 -- 172.21.15.40:0/1052294435 >> 172.21.15.40:0/1052294435 conn(0x7f7bd00d0be0 msgr2=0x7f7bd00c4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.015+0000 7f7bd5400640 1 -- 172.21.15.40:0/1052294435 shutdown_connections 2024-04-26T23:04:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.015+0000 7f7bd5400640 1 -- 172.21.15.40:0/1052294435 wait complete. 2024-04-26T23:04:12.038 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-26T23:04:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.227+0000 7f3ab5800640 1 Processor -- start 2024-04-26T23:04:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.228+0000 7f3ab5800640 1 -- start start 2024-04-26T23:04:12.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.229+0000 7f3ab5800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ab00d17d0 0x7f3ab00b6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.229+0000 7f3ab5800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ab00d0c80 0x7f3ab00b5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.229+0000 7f3ab5800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ab00b66c0 con 0x7f3ab00d2400 2024-04-26T23:04:12.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.229+0000 7f3ab5800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f3ab00d37b0 con 0x7f3ab00d17d0 2024-04-26T23:04:12.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.229+0000 7f3ab5800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab00d3d70 con 0x7f3ab00d0c80 2024-04-26T23:04:12.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.230+0000 7f3aafe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ab00d17d0 0x7f3ab00b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:12.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.230+0000 7f3aafe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ab00d17d0 0x7f3ab00b6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60980/0 (socket says 172.21.15.40:60980) 2024-04-26T23:04:12.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.230+0000 7f3aafe00640 1 -- 172.21.15.40:0/2325429743 learned_addr learned my addr 172.21.15.40:0/2325429743 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:12.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.230+0000 7f3ab4e00640 1 --2- 172.21.15.40:0/2325429743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ab00d0c80 0x7f3ab00b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:12.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.230+0000 7f3aafe00640 1 -- 172.21.15.40:0/2325429743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ab00d0c80 msgr2=0x7f3ab00b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:12.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.230+0000 7f3aafe00640 1 --2- 172.21.15.40:0/2325429743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ab00d0c80 0x7f3ab00b5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.230+0000 7f3aafe00640 1 -- 172.21.15.40:0/2325429743 >> v1:172.21.15.103:6789/0 conn(0x7f3ab00d2400 legacy=0x7f3ab00d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.231+0000 7f3aafe00640 1 -- 172.21.15.40:0/2325429743 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab015f9c0 con 0x7f3ab00d17d0 2024-04-26T23:04:12.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.232+0000 7f3aafe00640 1 --2- 172.21.15.40:0/2325429743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ab00d17d0 0x7f3ab00b6180 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3aa00098a0 tx=0x7f3aa002a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22296dda1330f55b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:12.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.232+0000 7f3aaf400640 1 -- 172.21.15.40:0/2325429743 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0037070 con 0x7f3ab00d17d0 2024-04-26T23:04:12.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.232+0000 7f3aaf400640 1 -- 172.21.15.40:0/2325429743 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aa0032440 con 0x7f3ab00d17d0 2024-04-26T23:04:12.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.232+0000 7f3aaf400640 1 -- 172.21.15.40:0/2325429743 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0032810 con 0x7f3ab00d17d0 2024-04-26T23:04:12.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.232+0000 7f3ab5800640 1 -- 172.21.15.40:0/2325429743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ab00d17d0 msgr2=0x7f3ab00b6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.232+0000 7f3ab5800640 1 --2- 172.21.15.40:0/2325429743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ab00d17d0 0x7f3ab00b6180 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3aa00098a0 tx=0x7f3aa002a1f0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.233+0000 7f3ab5800640 1 -- 172.21.15.40:0/2325429743 shutdown_connections 2024-04-26T23:04:12.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.233+0000 7f3ab5800640 1 --2- 172.21.15.40:0/2325429743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ab00d0c80 0x7f3ab00b5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.233+0000 7f3ab5800640 1 --2- 172.21.15.40:0/2325429743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ab00d17d0 0x7f3ab00b6180 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.233+0000 7f3ab5800640 1 -- 172.21.15.40:0/2325429743 >> 172.21.15.40:0/2325429743 conn(0x7f3ab009e210 msgr2=0x7f3ab00d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:12.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.233+0000 7f3ab5800640 1 -- 172.21.15.40:0/2325429743 shutdown_connections 2024-04-26T23:04:12.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.233+0000 7f3ab5800640 1 -- 172.21.15.40:0/2325429743 wait complete. 2024-04-26T23:04:12.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.234+0000 7f3ab5800640 1 Processor -- start 2024-04-26T23:04:12.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.235+0000 7f3ab5800640 1 -- start start 2024-04-26T23:04:12.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.235+0000 7f3ab5800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ab00d0c80 0x7f3ab00cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3ab5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ab00d17d0 0x7f3ab01f88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3ab5800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ab00d2400 0x7f3ab01f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3ab5800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab00d3a00 con 0x7f3ab00d17d0 2024-04-26T23:04:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3ab5800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab00d5a10 con 0x7f3ab00d2400 2024-04-26T23:04:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3ab5800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab00d1130 con 0x7f3ab00d0c80 2024-04-26T23:04:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3aa7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ab00d17d0 0x7f3ab01f88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3aa7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ab00d17d0 0x7f3ab01f88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60984/0 (socket says 172.21.15.40:60984) 2024-04-26T23:04:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3aa7e00640 1 -- 172.21.15.40:0/4154548389 learned_addr learned my addr 172.21.15.40:0/4154548389 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3aafe00640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ab00d0c80 0x7f3ab00cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3ab4e00640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ab00d2400 0x7f3ab01f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3aa7e00640 1 -- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ab00d0c80 msgr2=0x7f3ab00cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3aa7e00640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ab00d0c80 0x7f3ab00cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3aa7e00640 1 -- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ab00d2400 msgr2=0x7f3ab01f8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3aa7e00640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ab00d2400 0x7f3ab01f8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.236+0000 7f3aa7e00640 1 -- 172.21.15.40:0/4154548389 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab00cc3c0 con 0x7f3ab00d17d0 2024-04-26T23:04:12.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.237+0000 7f3aa7e00640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ab00d17d0 0x7f3ab01f88f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800c440 tx=0x7f3a98001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:12.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.237+0000 7f3aacc00640 1 -- 172.21.15.40:0/4154548389 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a980052d0 con 0x7f3ab00d17d0 2024-04-26T23:04:12.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.237+0000 7f3aacc00640 1 -- 172.21.15.40:0/4154548389 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a98005470 con 0x7f3ab00d17d0 2024-04-26T23:04:12.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.237+0000 7f3aacc00640 1 -- 172.21.15.40:0/4154548389 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98007260 con 0x7f3ab00d17d0 2024-04-26T23:04:12.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.237+0000 7f3ab5800640 1 -- 172.21.15.40:0/4154548389 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab00cc6a0 con 0x7f3ab00d17d0 2024-04-26T23:04:12.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.237+0000 7f3ab5800640 1 -- 172.21.15.40:0/4154548389 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab00ccc60 con 0x7f3ab00d17d0 2024-04-26T23:04:12.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.239+0000 7f3ab5800640 1 -- 172.21.15.40:0/4154548389 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ab0069370 con 0x7f3ab00d17d0 2024-04-26T23:04:12.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.239+0000 7f3aacc00640 1 -- 172.21.15.40:0/4154548389 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3a98007400 con 0x7f3ab00d17d0 2024-04-26T23:04:12.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.240+0000 7f3aacc00640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3a8c06c8f0 0x7f3a8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.240+0000 7f3aacc00640 1 -- 172.21.15.40:0/4154548389 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7f3a9808ae70 con 0x7f3ab00d17d0 2024-04-26T23:04:12.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.241+0000 7f3aafe00640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3a8c06c8f0 0x7f3a8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:12.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.242+0000 7f3aafe00640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3a8c06c8f0 0x7f3a8c06edb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0034000 tx=0x7f3aa002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:12.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.245+0000 7f3aacc00640 1 -- 172.21.15.40:0/4154548389 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3ab00b66c0 con 0x7f3ab00d17d0 2024-04-26T23:04:12.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.519+0000 7f3ab5800640 1 -- 172.21.15.40:0/4154548389 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f3ab00ced20 con 0x7f3ab00d17d0 2024-04-26T23:04:12.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.521+0000 7f3aacc00640 1 -- 172.21.15.40:0/4154548389 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) ==== 80+0+5304 (secure 0 0 0) 0x7f3a980594d0 con 0x7f3ab00d17d0 2024-04-26T23:04:12.521 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:12.521 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:04:12.521 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-04-26T23:04:12.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.521 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-04-26T23:04:12.521 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmF/UQHBAACdb8iskp2MuBMV6S5tYFJg==", 2024-04-26T23:04:12.522 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:04:12.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:04:12.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:04:12.522 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.522 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.522 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-04-26T23:04:12.522 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmniCyMxAAT/rVbIKj9EnvwDiblKMCeg==", 2024-04-26T23:04:12.522 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:04:12.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:04:12.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:04:12.523 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.523 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.523 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-04-26T23:04:12.524 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmLGZhJxAAg/fsrRrNoF/Jx1PnBVjjeQ==", 2024-04-26T23:04:12.524 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:04:12.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:04:12.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:04:12.524 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.524 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.524 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-04-26T23:04:12.524 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxms3DnAxAAxJj/rybrv/0cLK8PmQ619A==", 2024-04-26T23:04:12.524 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:04:12.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:04:12.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:04:12.525 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.525 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.525 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-04-26T23:04:12.525 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDrMSxmbeONCxAATB67ujBDn2tVxpbMju7Mfg==", 2024-04-26T23:04:12.525 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:04:12.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:04:12.525 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.525 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.526 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-04-26T23:04:12.526 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDuMSxmqE/PJhAAsNNu5ar8tHWQauxaXOvagA==", 2024-04-26T23:04:12.526 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:04:12.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:04:12.526 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.526 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.527 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-04-26T23:04:12.527 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDyMSxmd2trBhAAbnl7EBldxgtOd1s0Ll0A9w==", 2024-04-26T23:04:12.527 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:04:12.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:04:12.527 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.527 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.528 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-04-26T23:04:12.528 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD1MSxmqdtKEhAA9oxBxOu/musCKyladrJgYA==", 2024-04-26T23:04:12.528 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:04:12.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:04:12.528 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.528 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.528 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-04-26T23:04:12.529 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD7MSxm+kl7IxAAlPAGIemdC8gcvp+koeULaQ==", 2024-04-26T23:04:12.529 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:04:12.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:04:12.529 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.529 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.529 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-04-26T23:04:12.529 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD+MSxmzKiaLRAA2UPS1wwJHgMxQf/zLMjMVQ==", 2024-04-26T23:04:12.529 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:04:12.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:04:12.530 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.530 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.530 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-04-26T23:04:12.530 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQACMixmluHrCxAAmbmb33f5MVSTWNuy1D0Zsg==", 2024-04-26T23:04:12.530 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:04:12.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:04:12.530 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.531 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.531 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-04-26T23:04:12.531 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAFMixm6GQjJRAArmdnX7ioekT8GSdsbeB1Ow==", 2024-04-26T23:04:12.531 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.531 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:04:12.531 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:04:12.531 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.531 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.531 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-04-26T23:04:12.532 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg==", 2024-04-26T23:04:12.532 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.532 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:04:12.532 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow r", 2024-04-26T23:04:12.532 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-26T23:04:12.532 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.532 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-04-26T23:04:12.532 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmVhn4IhAA94XsDk1x5Oax5smXPoFQKg==", 2024-04-26T23:04:12.532 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.532 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:04:12.533 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:04:12.533 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:04:12.533 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:04:12.533 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.533 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-04-26T23:04:12.533 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxm3qhbGRAAK62blgPVShIYskDsMcaffQ==", 2024-04-26T23:04:12.533 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.533 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:04:12.533 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:04:12.533 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:04:12.534 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:04:12.534 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.534 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-04-26T23:04:12.534 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmxof/KxAAUmXVV4Vdp2DZnRjQzQ0Inw==", 2024-04-26T23:04:12.534 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.534 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:04:12.534 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:04:12.534 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:04:12.534 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:04:12.535 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.535 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-04-26T23:04:12.535 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDlMSxmmVwcGRAAqgHymPPTPuS6XXC6LuOVPg==", 2024-04-26T23:04:12.535 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.535 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:04:12.535 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:04:12.535 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:04:12.535 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.535 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.536 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-04-26T23:04:12.536 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmVf+9EhAAcpk8NgFqgCLXeHA2z7oHmA==", 2024-04-26T23:04:12.536 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.536 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:04:12.536 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-04-26T23:04:12.536 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.536 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:04:12.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:12.536 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-04-26T23:04:12.536 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmjjpcCBAAjDQRUBn/TK9hF3T2WBObyw==", 2024-04-26T23:04:12.537 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:04:12.537 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:04:12.537 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-04-26T23:04:12.537 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:04:12.537 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.537 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:12.537 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-26T23:04:12.537 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:04:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.523+0000 7f3ab5800640 1 -- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3a8c06c8f0 msgr2=0x7f3a8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.523+0000 7f3ab5800640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3a8c06c8f0 0x7f3a8c06edb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0034000 tx=0x7f3aa002e000 comp rx=0 tx=0).stop 2024-04-26T23:04:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.523+0000 7f3ab5800640 1 -- 172.21.15.40:0/4154548389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ab00d17d0 msgr2=0x7f3ab01f88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.523+0000 7f3ab5800640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ab00d17d0 0x7f3ab01f88f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800c440 tx=0x7f3a98001e70 comp rx=0 tx=0).stop 2024-04-26T23:04:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.523+0000 7f3ab5800640 1 -- 172.21.15.40:0/4154548389 shutdown_connections 2024-04-26T23:04:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.523+0000 7f3ab5800640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3a8c06c8f0 0x7f3a8c06edb0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.523+0000 7f3ab5800640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ab00d2400 0x7f3ab01f8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.523+0000 7f3ab5800640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ab00d17d0 0x7f3ab01f88f0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.523+0000 7f3ab5800640 1 --2- 172.21.15.40:0/4154548389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ab00d0c80 0x7f3ab00cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.523+0000 7f3ab5800640 1 -- 172.21.15.40:0/4154548389 >> 172.21.15.40:0/4154548389 conn(0x7f3ab009e210 msgr2=0x7f3ab015eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.524+0000 7f3ab5800640 1 -- 172.21.15.40:0/4154548389 shutdown_connections 2024-04-26T23:04:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.524+0000 7f3ab5800640 1 -- 172.21.15.40:0/4154548389 wait complete. 2024-04-26T23:04:12.542 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:04:12.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.751+0000 7f6357200640 1 Processor -- start 2024-04-26T23:04:12.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.751+0000 7f6357200640 1 -- start start 2024-04-26T23:04:12.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.753+0000 7f6357200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f635815ce60 0x7f63580afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.753+0000 7f6357200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f63580af7a0 0x7f63580ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.753+0000 7f6357200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63580d0820 con 0x7f6358071fe0 2024-04-26T23:04:12.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.753+0000 7f6357200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f63580cc450 con 0x7f63580af7a0 2024-04-26T23:04:12.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.753+0000 7f6357200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f63581614c0 con 0x7f635815ce60 2024-04-26T23:04:12.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.753+0000 7f6356800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6358071fe0 0x7f6358073760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37866/0 (socket says 172.21.15.40:37866) 2024-04-26T23:04:12.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.753+0000 7f6356800640 1 -- 172.21.15.40:0/769720387 learned_addr learned my addr 172.21.15.40:0/769720387 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:12.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.753+0000 7f6355400640 1 --2- 172.21.15.40:0/769720387 >> v2:172.21.15.103:3300/0 conn(0x7f63580af7a0 0x7f63580ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:12.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.753+0000 7f6355e00640 1 --2- 172.21.15.40:0/769720387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f635815ce60 0x7f63580afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:12.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3041383294 0 0) 0x7f63580d0820 con 0x7f6358071fe0 2024-04-26T23:04:12.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f633c003520 con 0x7f6358071fe0 2024-04-26T23:04:12.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 870490911 0 0) 0x7f633c003520 con 0x7f6358071fe0 2024-04-26T23:04:12.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f635815ce60 msgr2=0x7f63580afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:12.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 --2- 172.21.15.40:0/769720387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f635815ce60 0x7f63580afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 >> v2:172.21.15.103:3300/0 conn(0x7f63580af7a0 msgr2=0x7f63580ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:12.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 --2- 172.21.15.40:0/769720387 >> v2:172.21.15.103:3300/0 conn(0x7f63580af7a0 0x7f63580ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6358161760 con 0x7f6358071fe0 2024-04-26T23:04:12.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f63500019f0 con 0x7f6358071fe0 2024-04-26T23:04:12.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 >> v1:172.21.15.40:6789/0 conn(0x7f6358071fe0 legacy=0x7f6358073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 --2- 172.21.15.40:0/769720387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f635815ce60 0x7f633c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 --2- 172.21.15.40:0/769720387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c003f30 0x7f633c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 --2- 172.21.15.40:0/769720387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f633c004d40 0x7f633c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63580cc450 con 0x7f633c003f30 2024-04-26T23:04:12.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63581614c0 con 0x7f633c004d40 2024-04-26T23:04:12.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f633c00ba10 con 0x7f635815ce60 2024-04-26T23:04:12.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.754+0000 7f6355400640 1 --2- 172.21.15.40:0/769720387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c003f30 0x7f633c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:12.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.755+0000 7f6355400640 1 -- 172.21.15.40:0/769720387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f635815ce60 msgr2=0x7f633c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:12.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.755+0000 7f6355400640 1 --2- 172.21.15.40:0/769720387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f635815ce60 0x7f633c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.755+0000 7f6355400640 1 -- 172.21.15.40:0/769720387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f633c004d40 msgr2=0x7f633c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:12.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.755+0000 7f6355400640 1 --2- 172.21.15.40:0/769720387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f633c004d40 0x7f633c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.755+0000 7f6355400640 1 -- 172.21.15.40:0/769720387 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f633c003520 con 0x7f633c003f30 2024-04-26T23:04:12.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.756+0000 7f6355400640 1 --2- 172.21.15.40:0/769720387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c003f30 0x7f633c004800 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f6344009af0 tx=0x7f633c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e5270717373bba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:12.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.756+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6344036070 con 0x7f633c003f30 2024-04-26T23:04:12.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.756+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6344031400 con 0x7f633c003f30 2024-04-26T23:04:12.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.756+0000 7f6354a00640 1 -- 172.21.15.40:0/769720387 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6344009690 con 0x7f633c003f30 2024-04-26T23:04:12.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.757+0000 7f6357200640 1 -- 172.21.15.40:0/769720387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c003f30 msgr2=0x7f633c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.757+0000 7f6357200640 1 --2- 172.21.15.40:0/769720387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c003f30 0x7f633c004800 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f6344009af0 tx=0x7f633c005830 comp rx=0 tx=0).stop 2024-04-26T23:04:12.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.757+0000 7f6356800640 1 -- 172.21.15.40:0/769720387 reap_dead start 2024-04-26T23:04:12.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.757+0000 7f6357200640 1 -- 172.21.15.40:0/769720387 shutdown_connections 2024-04-26T23:04:12.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.757+0000 7f6357200640 1 -- 172.21.15.40:0/769720387 >> 172.21.15.40:0/769720387 conn(0x7f63580d0a10 msgr2=0x7f63580d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:12.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.757+0000 7f6357200640 1 -- 172.21.15.40:0/769720387 shutdown_connections 2024-04-26T23:04:12.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.757+0000 7f6357200640 1 -- 172.21.15.40:0/769720387 wait complete. 2024-04-26T23:04:12.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.759+0000 7f6357200640 1 Processor -- start 2024-04-26T23:04:12.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.759+0000 7f6357200640 1 -- start start 2024-04-26T23:04:12.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.760+0000 7f6357200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c003f30 0x7f63581514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.760+0000 7f6357200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63580b5940 0x7f63580aec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.760+0000 7f6357200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63580b5e90 0x7f63580af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.760+0000 7f6357200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6358073e70 con 0x7f633c003f30 2024-04-26T23:04:12.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.760+0000 7f6357200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63580cf530 con 0x7f63580b5e90 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.760+0000 7f6357200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6358073ff0 con 0x7f63580b5940 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.760+0000 7f6355e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c003f30 0x7f63581514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.760+0000 7f6355e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c003f30 0x7f63581514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32772/0 (socket says 172.21.15.40:32772) 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.760+0000 7f6355e00640 1 -- 172.21.15.40:0/3618812387 learned_addr learned my addr 172.21.15.40:0/3618812387 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.760+0000 7f6355e00640 1 -- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63580b5940 msgr2=0x7f63580aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.760+0000 7f6355e00640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63580b5940 0x7f63580aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.760+0000 7f6355e00640 1 -- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63580b5e90 msgr2=0x7f63580af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.760+0000 7f6356800640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63580b5e90 0x7f63580af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.761+0000 7f6355e00640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63580b5e90 0x7f63580af180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.761+0000 7f6355e00640 1 -- 172.21.15.40:0/3618812387 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63580c65f0 con 0x7f633c003f30 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.761+0000 7f6356800640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63580b5e90 0x7f63580af180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.761+0000 7f6355e00640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c003f30 0x7f63581514b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f634800e7c0 tx=0x7f634800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.761+0000 7f634e000640 1 -- 172.21.15.40:0/3618812387 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6348017070 con 0x7f633c003f30 2024-04-26T23:04:12.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.761+0000 7f634e000640 1 -- 172.21.15.40:0/3618812387 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f634800c940 con 0x7f633c003f30 2024-04-26T23:04:12.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.761+0000 7f6357200640 1 -- 172.21.15.40:0/3618812387 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63580c68d0 con 0x7f633c003f30 2024-04-26T23:04:12.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.761+0000 7f634e000640 1 -- 172.21.15.40:0/3618812387 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634800cd10 con 0x7f633c003f30 2024-04-26T23:04:12.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.761+0000 7f6357200640 1 -- 172.21.15.40:0/3618812387 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63580c6e90 con 0x7f633c003f30 2024-04-26T23:04:12.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.762+0000 7f6357200640 1 -- 172.21.15.40:0/3618812387 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6358069370 con 0x7f633c003f30 2024-04-26T23:04:12.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.764+0000 7f634e000640 1 -- 172.21.15.40:0/3618812387 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f634801a070 con 0x7f633c003f30 2024-04-26T23:04:12.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.765+0000 7f634e000640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f633806c8f0 0x7f633806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:12.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.765+0000 7f634e000640 1 -- 172.21.15.40:0/3618812387 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7f634808bb20 con 0x7f633c003f30 2024-04-26T23:04:12.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.765+0000 7f6355400640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f633806c8f0 0x7f633806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:12.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.766+0000 7f6355400640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f633806c8f0 0x7f633806edb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6344033000 tx=0x7f634402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:12.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.769+0000 7f634e000640 1 -- 172.21.15.40:0/3618812387 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f634805a100 con 0x7f633c003f30 2024-04-26T23:04:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.966+0000 7f6357200640 1 -- 172.21.15.40:0/3618812387 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f63580d01f0 con 0x7f633c003f30 2024-04-26T23:04:12.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.968+0000 7f634e000640 1 -- 172.21.15.40:0/3618812387 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+13128 (secure 0 0 0) 0x7f6348059a00 con 0x7f633c003f30 2024-04-26T23:04:12.969 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:12.969 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":48,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:04:12.025912+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:04:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.971+0000 7f6357200640 1 -- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f633806c8f0 msgr2=0x7f633806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.971+0000 7f6357200640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f633806c8f0 0x7f633806edb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6344033000 tx=0x7f634402e000 comp rx=0 tx=0).stop 2024-04-26T23:04:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.971+0000 7f6357200640 1 -- 172.21.15.40:0/3618812387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c003f30 msgr2=0x7f63581514b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.971+0000 7f6357200640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c003f30 0x7f63581514b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f634800e7c0 tx=0x7f634800ec80 comp rx=0 tx=0).stop 2024-04-26T23:04:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.971+0000 7f6357200640 1 -- 172.21.15.40:0/3618812387 shutdown_connections 2024-04-26T23:04:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.971+0000 7f6357200640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f633806c8f0 0x7f633806edb0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.971+0000 7f6357200640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63580b5e90 0x7f63580af180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.971+0000 7f6357200640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63580b5940 0x7f63580aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.971+0000 7f6357200640 1 --2- 172.21.15.40:0/3618812387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f633c003f30 0x7f63581514b0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.971+0000 7f6357200640 1 -- 172.21.15.40:0/3618812387 >> 172.21.15.40:0/3618812387 conn(0x7f63580d0a10 msgr2=0x7f6358156dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.971+0000 7f6357200640 1 -- 172.21.15.40:0/3618812387 shutdown_connections 2024-04-26T23:04:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:12.971+0000 7f6357200640 1 -- 172.21.15.40:0/3618812387 wait complete. 2024-04-26T23:04:12.989 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:04:12.989 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:04:13.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.184+0000 7f2317000640 1 Processor -- start 2024-04-26T23:04:13.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.185+0000 7f2317000640 1 -- start start 2024-04-26T23:04:13.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.186+0000 7f2317000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f23100b5120 0x7f23100b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.186+0000 7f2317000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f23100b59b0 0x7f23100b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.186+0000 7f2317000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23100d3dc0 con 0x7f23100b5500 2024-04-26T23:04:13.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.186+0000 7f2317000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f23100d03b0 con 0x7f23100b5120 2024-04-26T23:04:13.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.186+0000 7f2317000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f23100b75b0 con 0x7f23100b59b0 2024-04-26T23:04:13.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.186+0000 7f2315c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f23100b5120 0x7f23100b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:13.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.186+0000 7f2315c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f23100b5120 0x7f23100b5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32774/0 (socket says 172.21.15.40:32774) 2024-04-26T23:04:13.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.186+0000 7f2315c00640 1 -- 172.21.15.40:0/3623501568 learned_addr learned my addr 172.21.15.40:0/3623501568 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:13.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.187+0000 7f2316600640 1 --2- 172.21.15.40:0/3623501568 >> v2:172.21.15.103:3300/0 conn(0x7f23100b59b0 0x7f23100b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:13.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.187+0000 7f2315c00640 1 -- 172.21.15.40:0/3623501568 >> v2:172.21.15.103:3300/0 conn(0x7f23100b59b0 msgr2=0x7f23100b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:13.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.187+0000 7f2315c00640 1 --2- 172.21.15.40:0/3623501568 >> v2:172.21.15.103:3300/0 conn(0x7f23100b59b0 0x7f23100b6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.187+0000 7f2315c00640 1 -- 172.21.15.40:0/3623501568 >> v1:172.21.15.103:6789/0 conn(0x7f23100b5500 legacy=0x7f23100b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:13.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.187+0000 7f2315c00640 1 -- 172.21.15.40:0/3623501568 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23100d0600 con 0x7f23100b5120 2024-04-26T23:04:13.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.188+0000 7f2315c00640 1 --2- 172.21.15.40:0/3623501568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23100b5120 0x7f23100b5ee0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2300009870 tx=0x7f230002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1032aef71e2f4aaa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:13.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.188+0000 7f230fe00640 1 -- 172.21.15.40:0/3623501568 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300037070 con 0x7f23100b5120 2024-04-26T23:04:13.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.188+0000 7f230fe00640 1 -- 172.21.15.40:0/3623501568 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f230002ae40 con 0x7f23100b5120 2024-04-26T23:04:13.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.188+0000 7f230fe00640 1 -- 172.21.15.40:0/3623501568 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300032660 con 0x7f23100b5120 2024-04-26T23:04:13.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.189+0000 7f2317000640 1 -- 172.21.15.40:0/3623501568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23100b5120 msgr2=0x7f23100b5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:13.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.189+0000 7f2317000640 1 --2- 172.21.15.40:0/3623501568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23100b5120 0x7f23100b5ee0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2300009870 tx=0x7f230002a130 comp rx=0 tx=0).stop 2024-04-26T23:04:13.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.189+0000 7f2317000640 1 -- 172.21.15.40:0/3623501568 shutdown_connections 2024-04-26T23:04:13.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.189+0000 7f2317000640 1 --2- 172.21.15.40:0/3623501568 >> v2:172.21.15.103:3300/0 conn(0x7f23100b59b0 0x7f23100b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.189+0000 7f2317000640 1 --2- 172.21.15.40:0/3623501568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23100b5120 0x7f23100b5ee0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.189+0000 7f2317000640 1 -- 172.21.15.40:0/3623501568 >> 172.21.15.40:0/3623501568 conn(0x7f23100d4db0 msgr2=0x7f23100d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:13.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.189+0000 7f2317000640 1 -- 172.21.15.40:0/3623501568 shutdown_connections 2024-04-26T23:04:13.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.189+0000 7f2317000640 1 -- 172.21.15.40:0/3623501568 wait complete. 2024-04-26T23:04:13.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.191+0000 7f2317000640 1 Processor -- start 2024-04-26T23:04:13.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.191+0000 7f2317000640 1 -- start start 2024-04-26T23:04:13.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.192+0000 7f2317000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23100b5120 0x7f23100ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.192+0000 7f2317000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23100b5500 0x7f23100b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.192+0000 7f2317000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23100b59b0 0x7f23100ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.192+0000 7f2317000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f23100b7c00 con 0x7f23100b5120 2024-04-26T23:04:13.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.192+0000 7f2317000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f23100d0bc0 con 0x7f23100b59b0 2024-04-26T23:04:13.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.192+0000 7f2317000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f23100b7d80 con 0x7f23100b5500 2024-04-26T23:04:13.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.192+0000 7f2315c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23100b5120 0x7f23100ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:13.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.192+0000 7f2315c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23100b5120 0x7f23100ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32786/0 (socket says 172.21.15.40:32786) 2024-04-26T23:04:13.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.192+0000 7f2315c00640 1 -- 172.21.15.40:0/3212704558 learned_addr learned my addr 172.21.15.40:0/3212704558 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:13.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.193+0000 7f2315c00640 1 -- 172.21.15.40:0/3212704558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23100b5500 msgr2=0x7f23100b89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:13.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.193+0000 7f2315c00640 1 --2- 172.21.15.40:0/3212704558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23100b5500 0x7f23100b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.193+0000 7f2315c00640 1 -- 172.21.15.40:0/3212704558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23100b59b0 msgr2=0x7f23100ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:13.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.193+0000 7f2315c00640 1 --2- 172.21.15.40:0/3212704558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23100b59b0 0x7f23100ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.193+0000 7f2315c00640 1 -- 172.21.15.40:0/3212704558 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23100cd890 con 0x7f23100b5120 2024-04-26T23:04:13.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.193+0000 7f2315c00640 1 --2- 172.21.15.40:0/3212704558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23100b5120 0x7f23100ce930 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2300000c00 tx=0x7f23000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:13.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.193+0000 7f230d600640 1 -- 172.21.15.40:0/3212704558 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300037040 con 0x7f23100b5120 2024-04-26T23:04:13.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.193+0000 7f230d600640 1 -- 172.21.15.40:0/3212704558 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2300004030 con 0x7f23100b5120 2024-04-26T23:04:13.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.193+0000 7f230d600640 1 -- 172.21.15.40:0/3212704558 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300004200 con 0x7f23100b5120 2024-04-26T23:04:13.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.194+0000 7f2317000640 1 -- 172.21.15.40:0/3212704558 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23100cdb70 con 0x7f23100b5120 2024-04-26T23:04:13.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.194+0000 7f2317000640 1 -- 172.21.15.40:0/3212704558 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23100ce130 con 0x7f23100b5120 2024-04-26T23:04:13.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.195+0000 7f2317000640 1 -- 172.21.15.40:0/3212704558 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2310069370 con 0x7f23100b5120 2024-04-26T23:04:13.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.196+0000 7f230d600640 1 -- 172.21.15.40:0/3212704558 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f230003a070 con 0x7f23100b5120 2024-04-26T23:04:13.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.197+0000 7f230d600640 1 --2- 172.21.15.40:0/3212704558 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f22f006c920 0x7f22f006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.197+0000 7f230d600640 1 -- 172.21.15.40:0/3212704558 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7f23000abf60 con 0x7f23100b5120 2024-04-26T23:04:13.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.197+0000 7f2315200640 1 --2- 172.21.15.40:0/3212704558 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f22f006c920 0x7f22f006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:13.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.198+0000 7f2315200640 1 --2- 172.21.15.40:0/3212704558 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f22f006c920 0x7f22f006ede0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2310160210 tx=0x7f22fc0079b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:13.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.201+0000 7f230d600640 1 -- 172.21.15.40:0/3212704558 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2300041020 con 0x7f23100b5120 2024-04-26T23:04:13.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.401+0000 7f2317000640 1 -- 172.21.15.40:0/3212704558 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f23100b7260 con 0x7f23100b5120 2024-04-26T23:04:13.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.402+0000 7f230d600640 1 -- 172.21.15.40:0/3212704558 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7f230007a420 con 0x7f23100b5120 2024-04-26T23:04:13.402 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:04:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.405+0000 7f2317000640 1 -- 172.21.15.40:0/3212704558 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f22f006c920 msgr2=0x7f22f006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.405+0000 7f2317000640 1 --2- 172.21.15.40:0/3212704558 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f22f006c920 0x7f22f006ede0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2310160210 tx=0x7f22fc0079b0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.405+0000 7f2317000640 1 -- 172.21.15.40:0/3212704558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23100b5120 msgr2=0x7f23100ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.405+0000 7f2317000640 1 --2- 172.21.15.40:0/3212704558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23100b5120 0x7f23100ce930 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2300000c00 tx=0x7f23000046c0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.405+0000 7f2317000640 1 -- 172.21.15.40:0/3212704558 shutdown_connections 2024-04-26T23:04:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.405+0000 7f2317000640 1 --2- 172.21.15.40:0/3212704558 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f22f006c920 0x7f22f006ede0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.405+0000 7f2317000640 1 --2- 172.21.15.40:0/3212704558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23100b59b0 0x7f23100ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.405+0000 7f2317000640 1 --2- 172.21.15.40:0/3212704558 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23100b5500 0x7f23100b89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.405+0000 7f2317000640 1 --2- 172.21.15.40:0/3212704558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23100b5120 0x7f23100ce930 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.405+0000 7f2317000640 1 -- 172.21.15.40:0/3212704558 >> 172.21.15.40:0/3212704558 conn(0x7f23100d4db0 msgr2=0x7f231006ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.405+0000 7f2317000640 1 -- 172.21.15.40:0/3212704558 shutdown_connections 2024-04-26T23:04:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.405+0000 7f2317000640 1 -- 172.21.15.40:0/3212704558 wait complete. 2024-04-26T23:04:13.424 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T23:04:13.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.634+0000 7f927aa00640 1 Processor -- start 2024-04-26T23:04:13.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.634+0000 7f927aa00640 1 -- start start 2024-04-26T23:04:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.635+0000 7f927aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9274072950 0x7f9274167ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.635+0000 7f927aa00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f927415d7c0 con 0x7f92740cc480 2024-04-26T23:04:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.635+0000 7f927aa00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f927415daa0 con 0x7f9274071e60 2024-04-26T23:04:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.636+0000 7f927aa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f927415dd80 con 0x7f9274072950 2024-04-26T23:04:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.636+0000 7f926b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f92740cc480 0x7f9274072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37870/0 (socket says 172.21.15.40:37870) 2024-04-26T23:04:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.636+0000 7f926b400640 1 -- 172.21.15.40:0/3156078347 learned_addr learned my addr 172.21.15.40:0/3156078347 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.636+0000 7f9278a00640 1 --2- 172.21.15.40:0/3156078347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9274072950 0x7f9274167ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.636+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458340625 0 0) 0x7f927415d7c0 con 0x7f92740cc480 2024-04-26T23:04:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.636+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9258003520 con 0x7f92740cc480 2024-04-26T23:04:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2223825386 0 0) 0x7f9258003520 con 0x7f92740cc480 2024-04-26T23:04:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9274072950 msgr2=0x7f9274167ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f9273400640 1 --2- 172.21.15.40:0/3156078347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9274072950 0x7f9274167ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 >> v1:172.21.15.103:6789/0 conn(0x7f9274071e60 legacy=0x7f927406f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f927415e060 con 0x7f92740cc480 2024-04-26T23:04:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f925c001670 con 0x7f92740cc480 2024-04-26T23:04:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 >> v1:172.21.15.40:6789/0 conn(0x7f92740cc480 legacy=0x7f9274072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f9273400640 1 --2- 172.21.15.40:0/3156078347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9274072950 0x7f92580039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f9273400640 1 --2- 172.21.15.40:0/3156078347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9258003f60 0x7f9258004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f9273400640 1 --2- 172.21.15.40:0/3156078347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9258004d40 0x7f92580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f927415dd80 con 0x7f9258003f60 2024-04-26T23:04:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f925800bb50 con 0x7f9258004d40 2024-04-26T23:04:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f925800bd50 con 0x7f9274072950 2024-04-26T23:04:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.637+0000 7f926b400640 1 --2- 172.21.15.40:0/3156078347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9258003f60 0x7f9258004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.638+0000 7f9273e00640 1 --2- 172.21.15.40:0/3156078347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9274072950 0x7f92580039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:13.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.638+0000 7f9278a00640 1 --2- 172.21.15.40:0/3156078347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9258004d40 0x7f92580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:13.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.638+0000 7f926b400640 1 -- 172.21.15.40:0/3156078347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9274072950 msgr2=0x7f92580039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:13.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.638+0000 7f926b400640 1 --2- 172.21.15.40:0/3156078347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9274072950 0x7f92580039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.638+0000 7f926b400640 1 -- 172.21.15.40:0/3156078347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9258004d40 msgr2=0x7f92580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:13.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.638+0000 7f926b400640 1 --2- 172.21.15.40:0/3156078347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9258004d40 0x7f92580051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.638+0000 7f926b400640 1 -- 172.21.15.40:0/3156078347 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9258003520 con 0x7f9258003f60 2024-04-26T23:04:13.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.639+0000 7f926b400640 1 --2- 172.21.15.40:0/3156078347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9258003f60 0x7f9258004800 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f925c00c240 tx=0x7f9258005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a2401047b261e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:13.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.639+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c007d10 con 0x7f9258003f60 2024-04-26T23:04:13.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.639+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f925c005ad0 con 0x7f9258003f60 2024-04-26T23:04:13.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.639+0000 7f9273400640 1 -- 172.21.15.40:0/3156078347 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c007070 con 0x7f9258003f60 2024-04-26T23:04:13.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.640+0000 7f927aa00640 1 -- 172.21.15.40:0/3156078347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9258003f60 msgr2=0x7f9258004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:13.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.640+0000 7f927aa00640 1 --2- 172.21.15.40:0/3156078347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9258003f60 0x7f9258004800 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f925c00c240 tx=0x7f9258005830 comp rx=0 tx=0).stop 2024-04-26T23:04:13.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.640+0000 7f9278a00640 1 -- 172.21.15.40:0/3156078347 reap_dead start 2024-04-26T23:04:13.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.640+0000 7f927aa00640 1 -- 172.21.15.40:0/3156078347 shutdown_connections 2024-04-26T23:04:13.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.640+0000 7f927aa00640 1 -- 172.21.15.40:0/3156078347 >> 172.21.15.40:0/3156078347 conn(0x7f92740d0be0 msgr2=0x7f92740d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:13.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.640+0000 7f927aa00640 1 -- 172.21.15.40:0/3156078347 shutdown_connections 2024-04-26T23:04:13.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.640+0000 7f927aa00640 1 -- 172.21.15.40:0/3156078347 wait complete. 2024-04-26T23:04:13.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.642+0000 7f927aa00640 1 Processor -- start 2024-04-26T23:04:13.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.642+0000 7f927aa00640 1 -- start start 2024-04-26T23:04:13.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.643+0000 7f927aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9258003f60 0x7f9274158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.643+0000 7f927aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92740b60b0 0x7f92740b6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.643+0000 7f927aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f927415ae70 0x7f927415b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.643+0000 7f927aa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f927415d570 con 0x7f92740b60b0 2024-04-26T23:04:13.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.643+0000 7f927aa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f927415d3f0 con 0x7f927415ae70 2024-04-26T23:04:13.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.643+0000 7f927aa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f927406fff0 con 0x7f9258003f60 2024-04-26T23:04:13.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.643+0000 7f926b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92740b60b0 0x7f92740b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:13.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.643+0000 7f926b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92740b60b0 0x7f92740b6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32810/0 (socket says 172.21.15.40:32810) 2024-04-26T23:04:13.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.643+0000 7f926b400640 1 -- 172.21.15.40:0/1983465395 learned_addr learned my addr 172.21.15.40:0/1983465395 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:13.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.643+0000 7f9278a00640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f927415ae70 0x7f927415b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:13.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.643+0000 7f9273e00640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9258003f60 0x7f9274158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:13.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.644+0000 7f926b400640 1 -- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9258003f60 msgr2=0x7f9274158280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:13.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.644+0000 7f926b400640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9258003f60 0x7f9274158280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.644+0000 7f926b400640 1 -- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f927415ae70 msgr2=0x7f927415b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:13.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.644+0000 7f926b400640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f927415ae70 0x7f927415b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.644+0000 7f926b400640 1 -- 172.21.15.40:0/1983465395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92740c8190 con 0x7f92740b60b0 2024-04-26T23:04:13.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.644+0000 7f926b400640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92740b60b0 0x7f92740b6530 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f925c02ae20 tx=0x7f927415d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:13.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.644+0000 7f9270c00640 1 -- 172.21.15.40:0/1983465395 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c007d10 con 0x7f92740b60b0 2024-04-26T23:04:13.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.644+0000 7f9270c00640 1 -- 172.21.15.40:0/1983465395 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f925c043070 con 0x7f92740b60b0 2024-04-26T23:04:13.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.644+0000 7f9270c00640 1 -- 172.21.15.40:0/1983465395 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c004df0 con 0x7f92740b60b0 2024-04-26T23:04:13.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.644+0000 7f927aa00640 1 -- 172.21.15.40:0/1983465395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92740c8470 con 0x7f92740b60b0 2024-04-26T23:04:13.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.645+0000 7f927aa00640 1 -- 172.21.15.40:0/1983465395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92740c8a30 con 0x7f92740b60b0 2024-04-26T23:04:13.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.646+0000 7f927aa00640 1 -- 172.21.15.40:0/1983465395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92740764d0 con 0x7f92740b60b0 2024-04-26T23:04:13.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.647+0000 7f9270c00640 1 -- 172.21.15.40:0/1983465395 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f925c007070 con 0x7f92740b60b0 2024-04-26T23:04:13.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.648+0000 7f9270c00640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f925006c8f0 0x7f925006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:13.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.648+0000 7f9270c00640 1 -- 172.21.15.40:0/1983465395 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7f925c0abdd0 con 0x7f92740b60b0 2024-04-26T23:04:13.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.649+0000 7f9273e00640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f925006c8f0 0x7f925006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:13.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.650+0000 7f9273e00640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f925006c8f0 0x7f925006edb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9264001710 tx=0x7f9264009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:13.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.653+0000 7f9270c00640 1 -- 172.21.15.40:0/1983465395 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f92740764d0 con 0x7f92740b60b0 2024-04-26T23:04:13.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.862+0000 7f927aa00640 1 -- 172.21.15.40:0/1983465395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f92740694f0 con 0x7f92740b60b0 2024-04-26T23:04:13.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.863+0000 7f9270c00640 1 -- 172.21.15.40:0/1983465395 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v48) ==== 84+0+65 (secure 0 0 0) 0x7f925c07a320 con 0x7f92740b60b0 2024-04-26T23:04:13.864 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:13.864 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-04-26T23:04:13.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:04:13.864 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-04-26T23:04:13.864 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-04-26T23:04:13.864 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:13.864 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-04-26T23:04:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.866+0000 7f927aa00640 1 -- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f925006c8f0 msgr2=0x7f925006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.866+0000 7f927aa00640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f925006c8f0 0x7f925006edb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9264001710 tx=0x7f9264009040 comp rx=0 tx=0).stop 2024-04-26T23:04:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.866+0000 7f927aa00640 1 -- 172.21.15.40:0/1983465395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92740b60b0 msgr2=0x7f92740b6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.866+0000 7f927aa00640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92740b60b0 0x7f92740b6530 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f925c02ae20 tx=0x7f927415d990 comp rx=0 tx=0).stop 2024-04-26T23:04:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.866+0000 7f927aa00640 1 -- 172.21.15.40:0/1983465395 shutdown_connections 2024-04-26T23:04:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.866+0000 7f927aa00640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f925006c8f0 0x7f925006edb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.866+0000 7f927aa00640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f927415ae70 0x7f927415b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.866+0000 7f927aa00640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92740b60b0 0x7f92740b6530 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.866+0000 7f927aa00640 1 --2- 172.21.15.40:0/1983465395 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9258003f60 0x7f9274158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.866+0000 7f927aa00640 1 -- 172.21.15.40:0/1983465395 >> 172.21.15.40:0/1983465395 conn(0x7f92740d0be0 msgr2=0x7f9274072e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.866+0000 7f927aa00640 1 -- 172.21.15.40:0/1983465395 shutdown_connections 2024-04-26T23:04:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:13.867+0000 7f927aa00640 1 -- 172.21.15.40:0/1983465395 wait complete. 2024-04-26T23:04:13.884 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T23:04:13.885 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T23:04:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.099+0000 7f3bc7c00640 1 Processor -- start 2024-04-26T23:04:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.099+0000 7f3bc7c00640 1 -- start start 2024-04-26T23:04:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.100+0000 7f3bc7c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3bc0145630 0x7f3bc00cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.100+0000 7f3bc7c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bc015f980 0x7f3bc00b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.100+0000 7f3bc7c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc00b33e0 con 0x7f3bc00b2fc0 2024-04-26T23:04:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.100+0000 7f3bc7c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f3bc00cfd70 con 0x7f3bc0145630 2024-04-26T23:04:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.100+0000 7f3bc7c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc00d0050 con 0x7f3bc015f980 2024-04-26T23:04:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.101+0000 7f3bc5200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3bc0145630 0x7f3bc00cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.101+0000 7f3bc5c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3bc00b2fc0 0x7f3bc00730d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37872/0 (socket says 172.21.15.40:37872) 2024-04-26T23:04:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.101+0000 7f3bc5c00640 1 -- 172.21.15.40:0/3093901008 learned_addr learned my addr 172.21.15.40:0/3093901008 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.101+0000 7f3bbfe00640 1 --2- 172.21.15.40:0/3093901008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bc015f980 0x7f3bc00b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.101+0000 7f3bbf400640 1 -- 172.21.15.40:0/3093901008 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3371327519 0 0) 0x7f3bc00b33e0 con 0x7f3bc00b2fc0 2024-04-26T23:04:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.102+0000 7f3bbf400640 1 -- 172.21.15.40:0/3093901008 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ba8005730 con 0x7f3bc00b2fc0 2024-04-26T23:04:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.102+0000 7f3bc5200640 1 -- 172.21.15.40:0/3093901008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bc015f980 msgr2=0x7f3bc00b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.102+0000 7f3bc5200640 1 --2- 172.21.15.40:0/3093901008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bc015f980 0x7f3bc00b2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.102+0000 7f3bc5200640 1 -- 172.21.15.40:0/3093901008 >> v1:172.21.15.40:6789/0 conn(0x7f3bc00b2fc0 legacy=0x7f3bc00730d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.102+0000 7f3bc5200640 1 -- 172.21.15.40:0/3093901008 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc00d0330 con 0x7f3bc0145630 2024-04-26T23:04:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.103+0000 7f3bc5200640 1 --2- 172.21.15.40:0/3093901008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bc0145630 0x7f3bc00cf500 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40097a0 tx=0x7f3bb402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c57e082896423b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.103+0000 7f3bbf400640 1 -- 172.21.15.40:0/3093901008 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4037070 con 0x7f3bc0145630 2024-04-26T23:04:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.103+0000 7f3bbf400640 1 -- 172.21.15.40:0/3093901008 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bb4032440 con 0x7f3bc0145630 2024-04-26T23:04:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.103+0000 7f3bbf400640 1 -- 172.21.15.40:0/3093901008 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4032810 con 0x7f3bc0145630 2024-04-26T23:04:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.104+0000 7f3bc7c00640 1 -- 172.21.15.40:0/3093901008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bc0145630 msgr2=0x7f3bc00cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.104+0000 7f3bc7c00640 1 --2- 172.21.15.40:0/3093901008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bc0145630 0x7f3bc00cf500 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40097a0 tx=0x7f3bb402a180 comp rx=0 tx=0).stop 2024-04-26T23:04:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.104+0000 7f3bc7c00640 1 -- 172.21.15.40:0/3093901008 shutdown_connections 2024-04-26T23:04:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.104+0000 7f3bc7c00640 1 --2- 172.21.15.40:0/3093901008 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bc015f980 0x7f3bc00b2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.104+0000 7f3bc7c00640 1 --2- 172.21.15.40:0/3093901008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bc0145630 0x7f3bc00cf500 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.104+0000 7f3bc7c00640 1 -- 172.21.15.40:0/3093901008 >> 172.21.15.40:0/3093901008 conn(0x7f3bc006f310 msgr2=0x7f3bc006f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.104+0000 7f3bc7c00640 1 -- 172.21.15.40:0/3093901008 shutdown_connections 2024-04-26T23:04:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.104+0000 7f3bc7c00640 1 -- 172.21.15.40:0/3093901008 wait complete. 2024-04-26T23:04:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.106+0000 7f3bc7c00640 1 Processor -- start 2024-04-26T23:04:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.106+0000 7f3bc7c00640 1 -- start start 2024-04-26T23:04:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.107+0000 7f3bc7c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bc00b2fc0 0x7f3bc01f36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.107+0000 7f3bc7c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bc0145630 0x7f3bc00b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.107+0000 7f3bc7c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bc015f980 0x7f3bc00b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.107+0000 7f3bc7c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc00cfa40 con 0x7f3bc0145630 2024-04-26T23:04:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.107+0000 7f3bc7c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc00d08f0 con 0x7f3bc00b2fc0 2024-04-26T23:04:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.107+0000 7f3bc7c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc0145a10 con 0x7f3bc015f980 2024-04-26T23:04:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.107+0000 7f3bbfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bc0145630 0x7f3bc00b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.107+0000 7f3bbfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bc0145630 0x7f3bc00b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32830/0 (socket says 172.21.15.40:32830) 2024-04-26T23:04:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.107+0000 7f3bbfe00640 1 -- 172.21.15.40:0/4175858191 learned_addr learned my addr 172.21.15.40:0/4175858191 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.108+0000 7f3bc5200640 1 --2- 172.21.15.40:0/4175858191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bc00b2fc0 0x7f3bc01f36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.108+0000 7f3bbfe00640 1 -- 172.21.15.40:0/4175858191 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bc015f980 msgr2=0x7f3bc00b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.108+0000 7f3bbfe00640 1 --2- 172.21.15.40:0/4175858191 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bc015f980 0x7f3bc00b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.108+0000 7f3bbfe00640 1 -- 172.21.15.40:0/4175858191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bc00b2fc0 msgr2=0x7f3bc01f36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.108+0000 7f3bbfe00640 1 --2- 172.21.15.40:0/4175858191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bc00b2fc0 0x7f3bc01f36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.108+0000 7f3bbfe00640 1 -- 172.21.15.40:0/4175858191 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc01fa870 con 0x7f3bc0145630 2024-04-26T23:04:14.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.108+0000 7f3bbfe00640 1 --2- 172.21.15.40:0/4175858191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bc0145630 0x7f3bc00b45e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3bac00e8a0 tx=0x7f3bac00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:14.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.108+0000 7f3bbcc00640 1 -- 172.21.15.40:0/4175858191 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac017070 con 0x7f3bc0145630 2024-04-26T23:04:14.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.108+0000 7f3bbcc00640 1 -- 172.21.15.40:0/4175858191 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bac00c940 con 0x7f3bc0145630 2024-04-26T23:04:14.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.108+0000 7f3bc7c00640 1 -- 172.21.15.40:0/4175858191 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc00b51a0 con 0x7f3bc0145630 2024-04-26T23:04:14.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.109+0000 7f3bbcc00640 1 -- 172.21.15.40:0/4175858191 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac00cd30 con 0x7f3bc0145630 2024-04-26T23:04:14.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.109+0000 7f3bc7c00640 1 -- 172.21.15.40:0/4175858191 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc00ab4e0 con 0x7f3bc0145630 2024-04-26T23:04:14.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.110+0000 7f3bc7c00640 1 -- 172.21.15.40:0/4175858191 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc0069600 con 0x7f3bc0145630 2024-04-26T23:04:14.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.111+0000 7f3bbcc00640 1 -- 172.21.15.40:0/4175858191 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3bac01a070 con 0x7f3bc0145630 2024-04-26T23:04:14.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.112+0000 7f3bbcc00640 1 --2- 172.21.15.40:0/4175858191 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3ba006c8f0 0x7f3ba006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:14.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.112+0000 7f3bbcc00640 1 -- 172.21.15.40:0/4175858191 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5249+0+0 (secure 0 0 0) 0x7f3bac08b830 con 0x7f3bc0145630 2024-04-26T23:04:14.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.113+0000 7f3bc5200640 1 --2- 172.21.15.40:0/4175858191 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3ba006c8f0 0x7f3ba006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:14.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.114+0000 7f3bc5200640 1 --2- 172.21.15.40:0/4175858191 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3ba006c8f0 0x7f3ba006edb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4034000 tx=0x7f3bb402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.116+0000 7f3bbcc00640 1 -- 172.21.15.40:0/4175858191 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3bac059e10 con 0x7f3bc0145630 2024-04-26T23:04:14.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:14.321+0000 7f3bc7c00640 1 -- 172.21.15.40:0/4175858191 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3bc00b2b20 con 0x7f3bc0145630 2024-04-26T23:04:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.040+0000 7f3bbcc00640 1 -- 172.21.15.40:0/4175858191 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v49) ==== 138+0+0 (secure 0 0 0) 0x7f3bac059710 con 0x7f3bc0145630 2024-04-26T23:04:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-04-26T23:04:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.045+0000 7f3bc7c00640 1 -- 172.21.15.40:0/4175858191 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3ba006c8f0 msgr2=0x7f3ba006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.045+0000 7f3bc7c00640 1 --2- 172.21.15.40:0/4175858191 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3ba006c8f0 0x7f3ba006edb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4034000 tx=0x7f3bb402e000 comp rx=0 tx=0).stop 2024-04-26T23:04:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.045+0000 7f3bc7c00640 1 -- 172.21.15.40:0/4175858191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bc0145630 msgr2=0x7f3bc00b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.045+0000 7f3bc7c00640 1 --2- 172.21.15.40:0/4175858191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bc0145630 0x7f3bc00b45e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3bac00e8a0 tx=0x7f3bac00ed60 comp rx=0 tx=0).stop 2024-04-26T23:04:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.045+0000 7f3bc7c00640 1 -- 172.21.15.40:0/4175858191 shutdown_connections 2024-04-26T23:04:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.045+0000 7f3bc7c00640 1 --2- 172.21.15.40:0/4175858191 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3ba006c8f0 0x7f3ba006edb0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.045+0000 7f3bc7c00640 1 --2- 172.21.15.40:0/4175858191 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3bc015f980 0x7f3bc00b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.045+0000 7f3bc7c00640 1 --2- 172.21.15.40:0/4175858191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bc0145630 0x7f3bc00b45e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.045+0000 7f3bc7c00640 1 --2- 172.21.15.40:0/4175858191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3bc00b2fc0 0x7f3bc01f36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.045+0000 7f3bc7c00640 1 -- 172.21.15.40:0/4175858191 >> 172.21.15.40:0/4175858191 conn(0x7f3bc006f310 msgr2=0x7f3bc0073930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.046+0000 7f3bc7c00640 1 -- 172.21.15.40:0/4175858191 shutdown_connections 2024-04-26T23:04:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.046+0000 7f3bc7c00640 1 -- 172.21.15.40:0/4175858191 wait complete. 2024-04-26T23:04:15.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T23:04:15.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.280+0000 7f4426600640 1 Processor -- start 2024-04-26T23:04:15.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.281+0000 7f4426600640 1 -- start start 2024-04-26T23:04:15.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.282+0000 7f4426600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f442006c240 0x7f442006c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:15.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.282+0000 7f4426600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f44200cca40 0x7f44200cce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:15.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.282+0000 7f4426600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44200c79a0 con 0x7f442015f0d0 2024-04-26T23:04:15.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.282+0000 7f4426600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f442006cbf0 con 0x7f44200cca40 2024-04-26T23:04:15.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.282+0000 7f4426600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f44200c7c80 con 0x7f442006c240 2024-04-26T23:04:15.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.283+0000 7f4425c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f44200cca40 0x7f44200cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:15.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.283+0000 7f4425c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f44200cca40 0x7f44200cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42240/0 (socket says 172.21.15.40:42240) 2024-04-26T23:04:15.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.283+0000 7f4425c00640 1 -- 172.21.15.40:0/869529515 learned_addr learned my addr 172.21.15.40:0/869529515 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:15.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.283+0000 7f441fe00640 1 --2- 172.21.15.40:0/869529515 >> v2:172.21.15.103:3300/0 conn(0x7f442006c240 0x7f442006c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:15.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.283+0000 7f4425c00640 1 -- 172.21.15.40:0/869529515 >> v2:172.21.15.103:3300/0 conn(0x7f442006c240 msgr2=0x7f442006c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:15.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.283+0000 7f4425c00640 1 --2- 172.21.15.40:0/869529515 >> v2:172.21.15.103:3300/0 conn(0x7f442006c240 0x7f442006c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:15.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.283+0000 7f4425c00640 1 -- 172.21.15.40:0/869529515 >> v1:172.21.15.103:6789/0 conn(0x7f442015f0d0 legacy=0x7f44200c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:15.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.283+0000 7f4425c00640 1 -- 172.21.15.40:0/869529515 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44200cc240 con 0x7f44200cca40 2024-04-26T23:04:15.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.285+0000 7f4425c00640 1 --2- 172.21.15.40:0/869529515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44200cca40 0x7f44200cce90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f44200729b0 tx=0x7f441802a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8881702431cf2df8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:15.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.285+0000 7f441f400640 1 -- 172.21.15.40:0/869529515 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418037070 con 0x7f44200cca40 2024-04-26T23:04:15.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.285+0000 7f441f400640 1 -- 172.21.15.40:0/869529515 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4418032440 con 0x7f44200cca40 2024-04-26T23:04:15.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.285+0000 7f441f400640 1 -- 172.21.15.40:0/869529515 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418032810 con 0x7f44200cca40 2024-04-26T23:04:15.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.285+0000 7f4426600640 1 -- 172.21.15.40:0/869529515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44200cca40 msgr2=0x7f44200cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:15.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.285+0000 7f4426600640 1 --2- 172.21.15.40:0/869529515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44200cca40 0x7f44200cce90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f44200729b0 tx=0x7f441802a2c0 comp rx=0 tx=0).stop 2024-04-26T23:04:15.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.286+0000 7f4426600640 1 -- 172.21.15.40:0/869529515 shutdown_connections 2024-04-26T23:04:15.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.286+0000 7f4426600640 1 --2- 172.21.15.40:0/869529515 >> v2:172.21.15.103:3300/0 conn(0x7f442006c240 0x7f442006c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:15.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.286+0000 7f4426600640 1 --2- 172.21.15.40:0/869529515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44200cca40 0x7f44200cce90 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:15.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.286+0000 7f4426600640 1 -- 172.21.15.40:0/869529515 >> 172.21.15.40:0/869529515 conn(0x7f44200d1b50 msgr2=0x7f44200d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:15.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.286+0000 7f4426600640 1 -- 172.21.15.40:0/869529515 shutdown_connections 2024-04-26T23:04:15.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.286+0000 7f4426600640 1 -- 172.21.15.40:0/869529515 wait complete. 2024-04-26T23:04:15.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.288+0000 7f4426600640 1 Processor -- start 2024-04-26T23:04:15.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.288+0000 7f4426600640 1 -- start start 2024-04-26T23:04:15.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4426600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f442006c240 0x7f4420161a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:15.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4426600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44200cca40 0x7f44201568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:15.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4426600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f442015f0d0 0x7f44200ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:15.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4426600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f44200ce1b0 con 0x7f442006c240 2024-04-26T23:04:15.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4426600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f44200c7f60 con 0x7f442015f0d0 2024-04-26T23:04:15.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4426600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f44200c9000 con 0x7f44200cca40 2024-04-26T23:04:15.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4425200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f442006c240 0x7f4420161a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:15.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4425200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f442006c240 0x7f4420161a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42244/0 (socket says 172.21.15.40:42244) 2024-04-26T23:04:15.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4425200640 1 -- 172.21.15.40:0/108294869 learned_addr learned my addr 172.21.15.40:0/108294869 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:15.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f441fe00640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44200cca40 0x7f44201568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:15.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4425200640 1 -- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44200cca40 msgr2=0x7f44201568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:15.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4425200640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44200cca40 0x7f44201568a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:15.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4425200640 1 -- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f442015f0d0 msgr2=0x7f44200ca0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:15.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4425200640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f442015f0d0 0x7f44200ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:15.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f4425200640 1 -- 172.21.15.40:0/108294869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44200c92e0 con 0x7f442006c240 2024-04-26T23:04:15.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.289+0000 7f441fe00640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44200cca40 0x7f44201568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:15.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.290+0000 7f4425200640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f442006c240 0x7f4420161a90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f440800b3f0 tx=0x7f440800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:15.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.290+0000 7f441cc00640 1 -- 172.21.15.40:0/108294869 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408015020 con 0x7f442006c240 2024-04-26T23:04:15.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.290+0000 7f441cc00640 1 -- 172.21.15.40:0/108294869 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4408011830 con 0x7f442006c240 2024-04-26T23:04:15.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.290+0000 7f441cc00640 1 -- 172.21.15.40:0/108294869 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408011c20 con 0x7f442006c240 2024-04-26T23:04:15.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.290+0000 7f4426600640 1 -- 172.21.15.40:0/108294869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44200c95c0 con 0x7f442006c240 2024-04-26T23:04:15.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.290+0000 7f4426600640 1 -- 172.21.15.40:0/108294869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44200c9b80 con 0x7f442006c240 2024-04-26T23:04:15.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.292+0000 7f4426600640 1 -- 172.21.15.40:0/108294869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4420069a80 con 0x7f442006c240 2024-04-26T23:04:15.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.293+0000 7f441cc00640 1 -- 172.21.15.40:0/108294869 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4408018020 con 0x7f442006c240 2024-04-26T23:04:15.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.294+0000 7f441cc00640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f43fc06c8f0 0x7f43fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:15.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.294+0000 7f441cc00640 1 -- 172.21.15.40:0/108294869 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5619+0+0 (secure 0 0 0) 0x7f440808b840 con 0x7f442006c240 2024-04-26T23:04:15.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.295+0000 7f441fe00640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f43fc06c8f0 0x7f43fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:15.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.296+0000 7f441fe00640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f43fc06c8f0 0x7f43fc06edb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4420070f70 tx=0x7f441000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:15.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.299+0000 7f441cc00640 1 -- 172.21.15.40:0/108294869 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f440801f030 con 0x7f442006c240 2024-04-26T23:04:15.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:15.513+0000 7f4426600640 1 -- 172.21.15.40:0/108294869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f4420162030 con 0x7f442006c240 2024-04-26T23:04:16.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.059+0000 7f441cc00640 1 -- 172.21.15.40:0/108294869 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v50) ==== 170+0+0 (secure 0 0 0) 0x7f44080172c0 con 0x7f442006c240 2024-04-26T23:04:16.060 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-04-26T23:04:16.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.063+0000 7f4426600640 1 -- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f43fc06c8f0 msgr2=0x7f43fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:16.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.063+0000 7f4426600640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f43fc06c8f0 0x7f43fc06edb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4420070f70 tx=0x7f441000b040 comp rx=0 tx=0).stop 2024-04-26T23:04:16.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.063+0000 7f4426600640 1 -- 172.21.15.40:0/108294869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f442006c240 msgr2=0x7f4420161a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:16.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.063+0000 7f4426600640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f442006c240 0x7f4420161a90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f440800b3f0 tx=0x7f440800b8b0 comp rx=0 tx=0).stop 2024-04-26T23:04:16.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.063+0000 7f4426600640 1 -- 172.21.15.40:0/108294869 shutdown_connections 2024-04-26T23:04:16.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.063+0000 7f4426600640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f43fc06c8f0 0x7f43fc06edb0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:16.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.063+0000 7f4426600640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f442015f0d0 0x7f44200ca0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:16.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.063+0000 7f4426600640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44200cca40 0x7f44201568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:16.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.063+0000 7f4426600640 1 --2- 172.21.15.40:0/108294869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f442006c240 0x7f4420161a90 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:16.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.063+0000 7f4426600640 1 -- 172.21.15.40:0/108294869 >> 172.21.15.40:0/108294869 conn(0x7f44200d1b50 msgr2=0x7f4420073930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:16.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.064+0000 7f4426600640 1 -- 172.21.15.40:0/108294869 shutdown_connections 2024-04-26T23:04:16.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.065+0000 7f4426600640 1 -- 172.21.15.40:0/108294869 wait complete. 2024-04-26T23:04:16.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T23:04:16.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.290+0000 7ff8e7e00640 1 Processor -- start 2024-04-26T23:04:16.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.290+0000 7ff8e7e00640 1 -- start start 2024-04-26T23:04:16.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.291+0000 7ff8e7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b59b0 0x7ff8e80b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:16.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.291+0000 7ff8e7e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8e80d3dc0 con 0x7ff8e80b5120 2024-04-26T23:04:16.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.292+0000 7ff8e7e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8e80b75b0 con 0x7ff8e80b5500 2024-04-26T23:04:16.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.292+0000 7ff8e7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff8e80d03b0 con 0x7ff8e80b59b0 2024-04-26T23:04:16.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.292+0000 7ff8e6a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff8e80b5120 0x7ff8e80b6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57936/0 (socket says 172.21.15.40:57936) 2024-04-26T23:04:16.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.292+0000 7ff8e6a00640 1 -- 172.21.15.40:0/1324166819 learned_addr learned my addr 172.21.15.40:0/1324166819 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:16.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.292+0000 7ff8e7400640 1 --2- 172.21.15.40:0/1324166819 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b59b0 0x7ff8e80b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:16.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.292+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1705314335 0 0) 0x7ff8e80d3dc0 con 0x7ff8e80b5120 2024-04-26T23:04:16.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8cc003520 con 0x7ff8e80b5120 2024-04-26T23:04:16.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2236679636 0 0) 0x7ff8cc003520 con 0x7ff8e80b5120 2024-04-26T23:04:16.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b59b0 msgr2=0x7ff8e80b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:16.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 --2- 172.21.15.40:0/1324166819 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b59b0 0x7ff8e80b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:16.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 >> v1:172.21.15.103:6789/0 conn(0x7ff8e80b5500 legacy=0x7ff8e80b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:16.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e80d0600 con 0x7ff8e80b5120 2024-04-26T23:04:16.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7ff8d8001670 con 0x7ff8e80b5120 2024-04-26T23:04:16.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 >> v1:172.21.15.40:6789/0 conn(0x7ff8e80b5120 legacy=0x7ff8e80b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:16.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 --2- 172.21.15.40:0/1324166819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8cc0039e0 0x7ff8cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:16.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 --2- 172.21.15.40:0/1324166819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8cc004800 0x7ff8cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:16.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 --2- 172.21.15.40:0/1324166819 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b59b0 0x7ff8cc005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:16.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e80d03b0 con 0x7ff8cc004800 2024-04-26T23:04:16.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc00bb80 con 0x7ff8cc0039e0 2024-04-26T23:04:16.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff8cc00bd80 con 0x7ff8e80b59b0 2024-04-26T23:04:16.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.293+0000 7ff8e6000640 1 --2- 172.21.15.40:0/1324166819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8cc004800 0x7ff8cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:16.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.294+0000 7ff8e6a00640 1 --2- 172.21.15.40:0/1324166819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8cc0039e0 0x7ff8cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:16.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.294+0000 7ff8e6000640 1 -- 172.21.15.40:0/1324166819 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b59b0 msgr2=0x7ff8cc005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:16.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.294+0000 7ff8e7400640 1 --2- 172.21.15.40:0/1324166819 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b59b0 0x7ff8cc005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:16.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.294+0000 7ff8e6000640 1 --2- 172.21.15.40:0/1324166819 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b59b0 0x7ff8cc005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:16.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.294+0000 7ff8e6000640 1 -- 172.21.15.40:0/1324166819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8cc0039e0 msgr2=0x7ff8cc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:16.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.294+0000 7ff8e6000640 1 --2- 172.21.15.40:0/1324166819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8cc0039e0 0x7ff8cc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:16.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.294+0000 7ff8e6000640 1 -- 172.21.15.40:0/1324166819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8cc003520 con 0x7ff8cc004800 2024-04-26T23:04:16.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.294+0000 7ff8e7400640 1 --2- 172.21.15.40:0/1324166819 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b59b0 0x7ff8cc005220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:16.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.295+0000 7ff8e6000640 1 --2- 172.21.15.40:0/1324166819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8cc004800 0x7ff8cc004c80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4001750 tx=0x7ff8cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed64c17a2e2520e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:16.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.295+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4001370 con 0x7ff8cc004800 2024-04-26T23:04:16.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.295+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d40052d0 con 0x7ff8cc004800 2024-04-26T23:04:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.295+0000 7ff8e5600640 1 -- 172.21.15.40:0/1324166819 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d402abe0 con 0x7ff8cc004800 2024-04-26T23:04:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.296+0000 7ff8e7e00640 1 -- 172.21.15.40:0/1324166819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8cc004800 msgr2=0x7ff8cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.296+0000 7ff8e7e00640 1 --2- 172.21.15.40:0/1324166819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8cc004800 0x7ff8cc004c80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4001750 tx=0x7ff8cc005830 comp rx=0 tx=0).stop 2024-04-26T23:04:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.296+0000 7ff8e7400640 1 -- 172.21.15.40:0/1324166819 reap_dead start 2024-04-26T23:04:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.296+0000 7ff8e7e00640 1 -- 172.21.15.40:0/1324166819 shutdown_connections 2024-04-26T23:04:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.296+0000 7ff8e7e00640 1 -- 172.21.15.40:0/1324166819 >> 172.21.15.40:0/1324166819 conn(0x7ff8e80d4db0 msgr2=0x7ff8e80d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.296+0000 7ff8e7e00640 1 -- 172.21.15.40:0/1324166819 shutdown_connections 2024-04-26T23:04:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.296+0000 7ff8e7e00640 1 -- 172.21.15.40:0/1324166819 wait complete. 2024-04-26T23:04:16.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.298+0000 7ff8e7e00640 1 Processor -- start 2024-04-26T23:04:16.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.298+0000 7ff8e7e00640 1 -- start start 2024-04-26T23:04:16.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.299+0000 7ff8e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8cc004800 0x7ff8e80ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:16.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.299+0000 7ff8e7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8e80b9440 0x7ff8e815bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:16.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.299+0000 7ff8e7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b98c0 0x7ff8e815c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:16.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.299+0000 7ff8e7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e80b5ee0 con 0x7ff8cc004800 2024-04-26T23:04:16.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.299+0000 7ff8e7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e80d0bc0 con 0x7ff8e80b9440 2024-04-26T23:04:16.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.299+0000 7ff8e7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff8e80b6060 con 0x7ff8e80b98c0 2024-04-26T23:04:16.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.299+0000 7ff8e6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8cc004800 0x7ff8e80ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:16.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.299+0000 7ff8e6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8cc004800 0x7ff8e80ca4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42262/0 (socket says 172.21.15.40:42262) 2024-04-26T23:04:16.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.299+0000 7ff8e6a00640 1 -- 172.21.15.40:0/789530364 learned_addr learned my addr 172.21.15.40:0/789530364 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:16.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.300+0000 7ff8e6000640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8e80b9440 0x7ff8e815bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:16.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.300+0000 7ff8e7400640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b98c0 0x7ff8e815c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:16.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.300+0000 7ff8e6a00640 1 -- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b98c0 msgr2=0x7ff8e815c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:16.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.300+0000 7ff8e6a00640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b98c0 0x7ff8e815c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:16.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.300+0000 7ff8e6a00640 1 -- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8e80b9440 msgr2=0x7ff8e815bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:16.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.300+0000 7ff8e6a00640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8e80b9440 0x7ff8e815bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:16.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.300+0000 7ff8e6a00640 1 -- 172.21.15.40:0/789530364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e80c9410 con 0x7ff8cc004800 2024-04-26T23:04:16.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.300+0000 7ff8e7400640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b98c0 0x7ff8e815c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:16.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.300+0000 7ff8e6a00640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8cc004800 0x7ff8e80ca4b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800e730 tx=0x7ff8d800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:16.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.301+0000 7ff8df400640 1 -- 172.21.15.40:0/789530364 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d800ee40 con 0x7ff8cc004800 2024-04-26T23:04:16.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.301+0000 7ff8df400640 1 -- 172.21.15.40:0/789530364 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d8018070 con 0x7ff8cc004800 2024-04-26T23:04:16.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.301+0000 7ff8df400640 1 -- 172.21.15.40:0/789530364 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d800cab0 con 0x7ff8cc004800 2024-04-26T23:04:16.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.301+0000 7ff8e7e00640 1 -- 172.21.15.40:0/789530364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e80c96f0 con 0x7ff8cc004800 2024-04-26T23:04:16.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.301+0000 7ff8e7e00640 1 -- 172.21.15.40:0/789530364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e80c9cb0 con 0x7ff8cc004800 2024-04-26T23:04:16.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.302+0000 7ff8e7e00640 1 -- 172.21.15.40:0/789530364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e8069370 con 0x7ff8cc004800 2024-04-26T23:04:16.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.303+0000 7ff8df400640 1 -- 172.21.15.40:0/789530364 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff8d800cc50 con 0x7ff8cc004800 2024-04-26T23:04:16.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.304+0000 7ff8df400640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8c806c940 0x7ff8c806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:16.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.304+0000 7ff8df400640 1 -- 172.21.15.40:0/789530364 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6001+0+0 (secure 0 0 0) 0x7ff8d808c4a0 con 0x7ff8cc004800 2024-04-26T23:04:16.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.305+0000 7ff8e6000640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8c806c940 0x7ff8c806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:16.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.306+0000 7ff8e6000640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8c806c940 0x7ff8c806ee00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4032000 tx=0x7ff8d402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:16.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.309+0000 7ff8df400640 1 -- 172.21.15.40:0/789530364 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff8d805a880 con 0x7ff8cc004800 2024-04-26T23:04:16.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:16.556+0000 7ff8e7e00640 1 -- 172.21.15.40:0/789530364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff8e806da60 con 0x7ff8cc004800 2024-04-26T23:04:17.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.073+0000 7ff8df400640 1 -- 172.21.15.40:0/789530364 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v31) ==== 172+0+0 (secure 0 0 0) 0x7ff8d805a180 con 0x7ff8cc004800 2024-04-26T23:04:17.079 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 7 and data pool 8 2024-04-26T23:04:17.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.085+0000 7ff8e7e00640 1 -- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8c806c940 msgr2=0x7ff8c806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:17.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.085+0000 7ff8e7e00640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8c806c940 0x7ff8c806ee00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4032000 tx=0x7ff8d402e000 comp rx=0 tx=0).stop 2024-04-26T23:04:17.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.085+0000 7ff8e7e00640 1 -- 172.21.15.40:0/789530364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8cc004800 msgr2=0x7ff8e80ca4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:17.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.085+0000 7ff8e7e00640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8cc004800 0x7ff8e80ca4b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800e730 tx=0x7ff8d800ebf0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.088+0000 7ff8e7e00640 1 -- 172.21.15.40:0/789530364 shutdown_connections 2024-04-26T23:04:17.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.088+0000 7ff8e7e00640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8c806c940 0x7ff8c806ee00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.088+0000 7ff8e7e00640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8e80b98c0 0x7ff8e815c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.088+0000 7ff8e7e00640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8e80b9440 0x7ff8e815bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.088+0000 7ff8e7e00640 1 --2- 172.21.15.40:0/789530364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8cc004800 0x7ff8e80ca4b0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.088+0000 7ff8e7e00640 1 -- 172.21.15.40:0/789530364 >> 172.21.15.40:0/789530364 conn(0x7ff8e80d4db0 msgr2=0x7ff8e80b7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:17.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.089+0000 7ff8e7e00640 1 -- 172.21.15.40:0/789530364 shutdown_connections 2024-04-26T23:04:17.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.089+0000 7ff8e7e00640 1 -- 172.21.15.40:0/789530364 wait complete. 2024-04-26T23:04:17.122 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:04:17.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.299+0000 7f1379600640 1 Processor -- start 2024-04-26T23:04:17.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.299+0000 7f1379600640 1 -- start start 2024-04-26T23:04:17.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.300+0000 7f1379600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1374071e60 0x7f137406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:17.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.301+0000 7f1379600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f137406fe20 0x7f1374072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:17.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.301+0000 7f1379600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f137415d7c0 con 0x7f1374072780 2024-04-26T23:04:17.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.301+0000 7f1379600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f137415daa0 con 0x7f1374071e60 2024-04-26T23:04:17.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.301+0000 7f1379600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f137415dd80 con 0x7f137406fe20 2024-04-26T23:04:17.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.301+0000 7f1372a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1374071e60 0x7f137406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:17.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.301+0000 7f1372a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1374071e60 0x7f137406f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42272/0 (socket says 172.21.15.40:42272) 2024-04-26T23:04:17.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.301+0000 7f1372a00640 1 -- 172.21.15.40:0/1106432337 learned_addr learned my addr 172.21.15.40:0/1106432337 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:17.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.301+0000 7f1373400640 1 --1- 172.21.15.40:0/1106432337 >> v1:172.21.15.40:6789/0 conn(0x7f1374072780 0x7f13740cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57946/0 (socket says 172.21.15.40:57946) 2024-04-26T23:04:17.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.301+0000 7f1372a00640 1 -- 172.21.15.40:0/1106432337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f137406fe20 msgr2=0x7f1374072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:17.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.301+0000 7f1372a00640 1 --2- 172.21.15.40:0/1106432337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f137406fe20 0x7f1374072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.301+0000 7f1372a00640 1 -- 172.21.15.40:0/1106432337 >> v1:172.21.15.40:6789/0 conn(0x7f1374072780 legacy=0x7f13740cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:17.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.301+0000 7f1372a00640 1 -- 172.21.15.40:0/1106432337 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137415e060 con 0x7f1374071e60 2024-04-26T23:04:17.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.302+0000 7f1372a00640 1 --2- 172.21.15.40:0/1106432337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1374071e60 0x7f137406f8e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f1364009870 tx=0x7f136402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b156f930c4aade43 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:17.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.303+0000 7f1372000640 1 -- 172.21.15.40:0/1106432337 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364037070 con 0x7f1374071e60 2024-04-26T23:04:17.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.303+0000 7f1372000640 1 -- 172.21.15.40:0/1106432337 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1364032440 con 0x7f1374071e60 2024-04-26T23:04:17.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.303+0000 7f1372000640 1 -- 172.21.15.40:0/1106432337 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364032810 con 0x7f1374071e60 2024-04-26T23:04:17.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.303+0000 7f1379600640 1 -- 172.21.15.40:0/1106432337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1374071e60 msgr2=0x7f137406f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:17.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.303+0000 7f1379600640 1 --2- 172.21.15.40:0/1106432337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1374071e60 0x7f137406f8e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f1364009870 tx=0x7f136402a250 comp rx=0 tx=0).stop 2024-04-26T23:04:17.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.304+0000 7f1379600640 1 -- 172.21.15.40:0/1106432337 shutdown_connections 2024-04-26T23:04:17.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.304+0000 7f1379600640 1 --2- 172.21.15.40:0/1106432337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f137406fe20 0x7f1374072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.304+0000 7f1379600640 1 --2- 172.21.15.40:0/1106432337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1374071e60 0x7f137406f8e0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.304+0000 7f1379600640 1 -- 172.21.15.40:0/1106432337 >> 172.21.15.40:0/1106432337 conn(0x7f13740d0be0 msgr2=0x7f13740d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:17.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.304+0000 7f1379600640 1 -- 172.21.15.40:0/1106432337 shutdown_connections 2024-04-26T23:04:17.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.304+0000 7f1379600640 1 -- 172.21.15.40:0/1106432337 wait complete. 2024-04-26T23:04:17.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.306+0000 7f1379600640 1 Processor -- start 2024-04-26T23:04:17.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.306+0000 7f1379600640 1 -- start start 2024-04-26T23:04:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.307+0000 7f1379600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f137406fe20 0x7f13740b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.307+0000 7f1379600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1374071e60 0x7f13740c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.307+0000 7f1379600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1374072780 0x7f137414bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.307+0000 7f1379600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f137415d3f0 con 0x7f1374072780 2024-04-26T23:04:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.307+0000 7f1379600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1374072c10 con 0x7f1374071e60 2024-04-26T23:04:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.307+0000 7f1379600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f137415d570 con 0x7f137406fe20 2024-04-26T23:04:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.307+0000 7f1373400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1374072780 0x7f137414bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.307+0000 7f1373400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1374072780 0x7f137414bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42286/0 (socket says 172.21.15.40:42286) 2024-04-26T23:04:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.307+0000 7f1373400640 1 -- 172.21.15.40:0/4144289895 learned_addr learned my addr 172.21.15.40:0/4144289895 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.308+0000 7f136a000640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1374071e60 0x7f13740c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.308+0000 7f1372a00640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f137406fe20 0x7f13740b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.308+0000 7f1373400640 1 -- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f137406fe20 msgr2=0x7f13740b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.308+0000 7f1373400640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f137406fe20 0x7f13740b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.308+0000 7f1373400640 1 -- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1374071e60 msgr2=0x7f13740c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.308+0000 7f1373400640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1374071e60 0x7f13740c9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.308+0000 7f1373400640 1 -- 172.21.15.40:0/4144289895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13740ca660 con 0x7f1374072780 2024-04-26T23:04:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.308+0000 7f136a000640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1374071e60 0x7f13740c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:04:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.308+0000 7f1373400640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1374072780 0x7f137414bb90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f136c00c730 tx=0x7f136c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:17.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.308+0000 7f136b400640 1 -- 172.21.15.40:0/4144289895 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c017070 con 0x7f1374072780 2024-04-26T23:04:17.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.308+0000 7f136b400640 1 -- 172.21.15.40:0/4144289895 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f136c00ce40 con 0x7f1374072780 2024-04-26T23:04:17.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.308+0000 7f136b400640 1 -- 172.21.15.40:0/4144289895 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c00a7e0 con 0x7f1374072780 2024-04-26T23:04:17.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.308+0000 7f1379600640 1 -- 172.21.15.40:0/4144289895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13740ca940 con 0x7f1374072780 2024-04-26T23:04:17.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.309+0000 7f1379600640 1 -- 172.21.15.40:0/4144289895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13740caf00 con 0x7f1374072780 2024-04-26T23:04:17.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.310+0000 7f1379600640 1 -- 172.21.15.40:0/4144289895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f137406ec80 con 0x7f1374072780 2024-04-26T23:04:17.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.311+0000 7f136b400640 1 -- 172.21.15.40:0/4144289895 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f136c01a070 con 0x7f1374072780 2024-04-26T23:04:17.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.312+0000 7f136b400640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f135006c8f0 0x7f135006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:17.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.312+0000 7f136b400640 1 -- 172.21.15.40:0/4144289895 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f136c08b2d0 con 0x7f1374072780 2024-04-26T23:04:17.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.313+0000 7f1372a00640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f135006c8f0 0x7f135006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:17.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.314+0000 7f1372a00640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f135006c8f0 0x7f135006edb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1364034000 tx=0x7f136402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:17.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.317+0000 7f136b400640 1 -- 172.21.15.40:0/4144289895 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f137406ec80 con 0x7f1374072780 2024-04-26T23:04:17.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.519+0000 7f1379600640 1 -- 172.21.15.40:0/4144289895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f13740694f0 con 0x7f1374072780 2024-04-26T23:04:17.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.522+0000 7f136b400640 1 -- 172.21.15.40:0/4144289895 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+16919 (secure 0 0 0) 0x7f136c058df0 con 0x7f1374072780 2024-04-26T23:04:17.522 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:17.522 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":51,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:04:16.558666+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:04:14.334803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-26T23:04:15.535793+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:04:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.524+0000 7f1379600640 1 -- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f135006c8f0 msgr2=0x7f135006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.524+0000 7f1379600640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f135006c8f0 0x7f135006edb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1364034000 tx=0x7f136402e000 comp rx=0 tx=0).stop 2024-04-26T23:04:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.524+0000 7f1379600640 1 -- 172.21.15.40:0/4144289895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1374072780 msgr2=0x7f137414bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.524+0000 7f1379600640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1374072780 0x7f137414bb90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f136c00c730 tx=0x7f136c00cbf0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.524+0000 7f1379600640 1 -- 172.21.15.40:0/4144289895 shutdown_connections 2024-04-26T23:04:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.524+0000 7f1379600640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f135006c8f0 0x7f135006edb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.524+0000 7f1379600640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1374072780 0x7f137414bb90 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.524+0000 7f1379600640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1374071e60 0x7f13740c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.524+0000 7f1379600640 1 --2- 172.21.15.40:0/4144289895 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f137406fe20 0x7f13740b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.525+0000 7f1379600640 1 -- 172.21.15.40:0/4144289895 >> 172.21.15.40:0/4144289895 conn(0x7f13740d0be0 msgr2=0x7f13740b06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:17.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.525+0000 7f1379600640 1 -- 172.21.15.40:0/4144289895 shutdown_connections 2024-04-26T23:04:17.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.525+0000 7f1379600640 1 -- 172.21.15.40:0/4144289895 wait complete. 2024-04-26T23:04:17.543 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:04:17.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:04:17.688 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:04:17.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.751+0000 7fadfbe00640 1 Processor -- start 2024-04-26T23:04:17.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.751+0000 7fadfbe00640 1 -- start start 2024-04-26T23:04:17.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.753+0000 7fadfbe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fadfc0b6220 0x7fadfc0d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:17.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.753+0000 7fadfbe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fadfc0d08d0 0x7fadfc0d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:17.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.753+0000 7fadfbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadfc0d3820 0x7fadfc0b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:17.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.753+0000 7fadfbe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fadfc162c40 con 0x7fadfc0d08d0 2024-04-26T23:04:17.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.753+0000 7fadfbe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fadfc0d2d80 con 0x7fadfc0b6220 2024-04-26T23:04:17.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.753+0000 7fadfbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fadfc0b7c80 con 0x7fadfc0d3820 2024-04-26T23:04:17.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.753+0000 7fadfa000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fadfc0d08d0 0x7fadfc0d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:17.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.754+0000 7fadfa000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fadfc0d08d0 0x7fadfc0d32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42296/0 (socket says 172.21.15.40:42296) 2024-04-26T23:04:17.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.754+0000 7fadfa000640 1 -- 172.21.15.40:0/2388889531 learned_addr learned my addr 172.21.15.40:0/2388889531 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:17.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.754+0000 7fadfaa00640 1 --2- 172.21.15.40:0/2388889531 >> v2:172.21.15.103:3300/0 conn(0x7fadfc0b6220 0x7fadfc0d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:17.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.754+0000 7fadfb400640 1 --2- 172.21.15.40:0/2388889531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadfc0d3820 0x7fadfc0b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:17.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.754+0000 7fadfa000640 1 -- 172.21.15.40:0/2388889531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadfc0d3820 msgr2=0x7fadfc0b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:17.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.754+0000 7fadfa000640 1 --2- 172.21.15.40:0/2388889531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadfc0d3820 0x7fadfc0b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.754+0000 7fadfa000640 1 -- 172.21.15.40:0/2388889531 >> v2:172.21.15.103:3300/0 conn(0x7fadfc0b6220 msgr2=0x7fadfc0d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:17.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.754+0000 7fadfa000640 1 --2- 172.21.15.40:0/2388889531 >> v2:172.21.15.103:3300/0 conn(0x7fadfc0b6220 0x7fadfc0d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.754+0000 7fadfa000640 1 -- 172.21.15.40:0/2388889531 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadfc0b7f60 con 0x7fadfc0d08d0 2024-04-26T23:04:17.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.755+0000 7fadfa000640 1 --2- 172.21.15.40:0/2388889531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc0d08d0 0x7fadfc0d32e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fadfc0d4cf0 tx=0x7fade802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366447d509bf7da9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:17.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.756+0000 7fadf9600640 1 -- 172.21.15.40:0/2388889531 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8037070 con 0x7fadfc0d08d0 2024-04-26T23:04:17.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.756+0000 7fadf9600640 1 -- 172.21.15.40:0/2388889531 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fade8032440 con 0x7fadfc0d08d0 2024-04-26T23:04:17.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.756+0000 7fadf9600640 1 -- 172.21.15.40:0/2388889531 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8032810 con 0x7fadfc0d08d0 2024-04-26T23:04:17.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.757+0000 7fadfbe00640 1 -- 172.21.15.40:0/2388889531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc0d08d0 msgr2=0x7fadfc0d32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:17.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.757+0000 7fadfbe00640 1 --2- 172.21.15.40:0/2388889531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc0d08d0 0x7fadfc0d32e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fadfc0d4cf0 tx=0x7fade802a1e0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.757+0000 7fadfbe00640 1 -- 172.21.15.40:0/2388889531 shutdown_connections 2024-04-26T23:04:17.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.757+0000 7fadfbe00640 1 --2- 172.21.15.40:0/2388889531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadfc0d3820 0x7fadfc0b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.757+0000 7fadfbe00640 1 --2- 172.21.15.40:0/2388889531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc0d08d0 0x7fadfc0d32e0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.757+0000 7fadfbe00640 1 --2- 172.21.15.40:0/2388889531 >> v2:172.21.15.103:3300/0 conn(0x7fadfc0b6220 0x7fadfc0d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.757+0000 7fadfbe00640 1 -- 172.21.15.40:0/2388889531 >> 172.21.15.40:0/2388889531 conn(0x7fadfc0d4d90 msgr2=0x7fadfc0d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:17.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.757+0000 7fadfbe00640 1 -- 172.21.15.40:0/2388889531 shutdown_connections 2024-04-26T23:04:17.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.757+0000 7fadfbe00640 1 -- 172.21.15.40:0/2388889531 wait complete. 2024-04-26T23:04:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.759+0000 7fadfbe00640 1 Processor -- start 2024-04-26T23:04:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.759+0000 7fadfbe00640 1 -- start start 2024-04-26T23:04:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.760+0000 7fadfbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadfc0b6220 0x7fadfc0caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.760+0000 7fadfbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadfc0d08d0 0x7fadfc15a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.760+0000 7fadfbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc0d3820 0x7fadfc159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.760+0000 7fadfbe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fadfc0b6d50 con 0x7fadfc0d3820 2024-04-26T23:04:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.760+0000 7fadfbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fadfc0b8520 con 0x7fadfc0b6220 2024-04-26T23:04:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.760+0000 7fadfbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fadfc0b6ed0 con 0x7fadfc0d08d0 2024-04-26T23:04:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.760+0000 7fadfb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc0d3820 0x7fadfc159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.761+0000 7fadfb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc0d3820 0x7fadfc159520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42308/0 (socket says 172.21.15.40:42308) 2024-04-26T23:04:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.761+0000 7fadfb400640 1 -- 172.21.15.40:0/114424320 learned_addr learned my addr 172.21.15.40:0/114424320 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.761+0000 7fadfaa00640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadfc0b6220 0x7fadfc0caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.761+0000 7fadfa000640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadfc0d08d0 0x7fadfc15a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.761+0000 7fadfb400640 1 -- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadfc0d08d0 msgr2=0x7fadfc15a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.761+0000 7fadfb400640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadfc0d08d0 0x7fadfc15a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.761+0000 7fadfb400640 1 -- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadfc0b6220 msgr2=0x7fadfc0caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.761+0000 7fadfb400640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadfc0b6220 0x7fadfc0caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.761+0000 7fadfb400640 1 -- 172.21.15.40:0/114424320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadfc0cafe0 con 0x7fadfc0d3820 2024-04-26T23:04:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.761+0000 7fadfb400640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc0d3820 0x7fadfc159520 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fadf4007e60 tx=0x7fadf400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.762+0000 7fadf2a00640 1 -- 172.21.15.40:0/114424320 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf4014420 con 0x7fadfc0d3820 2024-04-26T23:04:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.762+0000 7fadf2a00640 1 -- 172.21.15.40:0/114424320 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadf4019070 con 0x7fadfc0d3820 2024-04-26T23:04:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.762+0000 7fadfbe00640 1 -- 172.21.15.40:0/114424320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadfc15ad20 con 0x7fadfc0d3820 2024-04-26T23:04:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.762+0000 7fadf2a00640 1 -- 172.21.15.40:0/114424320 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf4014820 con 0x7fadfc0d3820 2024-04-26T23:04:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.762+0000 7fadfbe00640 1 -- 172.21.15.40:0/114424320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadfc15f280 con 0x7fadfc0d3820 2024-04-26T23:04:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.763+0000 7fadfbe00640 1 -- 172.21.15.40:0/114424320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadfc069370 con 0x7fadfc0d3820 2024-04-26T23:04:17.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.764+0000 7fadf2a00640 1 -- 172.21.15.40:0/114424320 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fadf40149c0 con 0x7fadfc0d3820 2024-04-26T23:04:17.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.765+0000 7fadf2a00640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faddc06c8f0 0x7faddc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:17.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.765+0000 7fadf2a00640 1 -- 172.21.15.40:0/114424320 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fadf408c440 con 0x7fadfc0d3820 2024-04-26T23:04:17.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.765+0000 7fadfaa00640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faddc06c8f0 0x7faddc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:17.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.766+0000 7fadfaa00640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faddc06c8f0 0x7faddc06edb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fadec009ac0 tx=0x7fadec009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:17.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.770+0000 7fadf2a00640 1 -- 172.21.15.40:0/114424320 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fadfc0cafe0 con 0x7fadfc0d3820 2024-04-26T23:04:17.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.984+0000 7fadfbe00640 1 -- 172.21.15.40:0/114424320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fadfc0d30c0 con 0x7fadfc0d3820 2024-04-26T23:04:17.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.985+0000 7fadf2a00640 1 -- 172.21.15.40:0/114424320 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+16919 (secure 0 0 0) 0x7fadf4059ff0 con 0x7fadfc0d3820 2024-04-26T23:04:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:17.986 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":51,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:04:16.558666+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:04:14.334803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-26T23:04:15.535793+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:04:17.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.988+0000 7fadfbe00640 1 -- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faddc06c8f0 msgr2=0x7faddc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:17.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.989+0000 7fadfbe00640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faddc06c8f0 0x7faddc06edb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fadec009ac0 tx=0x7fadec009190 comp rx=0 tx=0).stop 2024-04-26T23:04:17.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.989+0000 7fadfbe00640 1 -- 172.21.15.40:0/114424320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc0d3820 msgr2=0x7fadfc159520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.989+0000 7fadfbe00640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc0d3820 0x7fadfc159520 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fadf4007e60 tx=0x7fadf400e4b0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.989+0000 7fadfbe00640 1 -- 172.21.15.40:0/114424320 shutdown_connections 2024-04-26T23:04:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.989+0000 7fadfbe00640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faddc06c8f0 0x7faddc06edb0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.989+0000 7fadfbe00640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc0d3820 0x7fadfc159520 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.989+0000 7fadfbe00640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fadfc0d08d0 0x7fadfc15a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.989+0000 7fadfbe00640 1 --2- 172.21.15.40:0/114424320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadfc0b6220 0x7fadfc0caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.989+0000 7fadfbe00640 1 -- 172.21.15.40:0/114424320 >> 172.21.15.40:0/114424320 conn(0x7fadfc0d4d90 msgr2=0x7fadfc1623b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.989+0000 7fadfbe00640 1 -- 172.21.15.40:0/114424320 shutdown_connections 2024-04-26T23:04:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:17.989+0000 7fadfbe00640 1 -- 172.21.15.40:0/114424320 wait complete. 2024-04-26T23:04:18.008 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T23:04:18.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.222+0000 7f91c5800640 1 Processor -- start 2024-04-26T23:04:18.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.222+0000 7f91c5800640 1 -- start start 2024-04-26T23:04:18.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.224+0000 7f91c5800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f91c00c41d0 0x7f91c00cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:18.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.224+0000 7f91c5800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91c01678f0 0x7f91c00cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:18.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.224+0000 7f91c5800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91c0163a50 con 0x7f91c0073f30 2024-04-26T23:04:18.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.224+0000 7f91c5800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f91c00cf7a0 con 0x7f91c00c41d0 2024-04-26T23:04:18.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.224+0000 7f91c5800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f91c00ceb90 con 0x7f91c01678f0 2024-04-26T23:04:18.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.224+0000 7f91bea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f91c00c41d0 0x7f91c00cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:18.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.224+0000 7f91bea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f91c00c41d0 0x7f91c00cfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42322/0 (socket says 172.21.15.40:42322) 2024-04-26T23:04:18.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.224+0000 7f91bea00640 1 -- 172.21.15.40:0/3516974628 learned_addr learned my addr 172.21.15.40:0/3516974628 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:18.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.224+0000 7f91b6000640 1 --1- 172.21.15.40:0/3516974628 >> v1:172.21.15.40:6789/0 conn(0x7f91c0073f30 0x7f91c01671e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57956/0 (socket says 172.21.15.40:57956) 2024-04-26T23:04:18.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.225+0000 7f91bf400640 1 --2- 172.21.15.40:0/3516974628 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91c01678f0 0x7f91c00cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:18.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.225+0000 7f91bea00640 1 -- 172.21.15.40:0/3516974628 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91c01678f0 msgr2=0x7f91c00cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:18.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.225+0000 7f91bea00640 1 --2- 172.21.15.40:0/3516974628 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91c01678f0 0x7f91c00cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:18.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.225+0000 7f91bea00640 1 -- 172.21.15.40:0/3516974628 >> v1:172.21.15.40:6789/0 conn(0x7f91c0073f30 legacy=0x7f91c01671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:18.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.225+0000 7f91be000640 1 -- 172.21.15.40:0/3516974628 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3395694184 0 0) 0x7f91c0163a50 con 0x7f91c0073f30 2024-04-26T23:04:18.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.225+0000 7f91bea00640 1 -- 172.21.15.40:0/3516974628 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c00cee70 con 0x7f91c00c41d0 2024-04-26T23:04:18.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.226+0000 7f91bea00640 1 --2- 172.21.15.40:0/3516974628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91c00c41d0 0x7f91c00cfc30 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009870 tx=0x7f91b002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc18518f70dfc07 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:18.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.226+0000 7f91be000640 1 -- 172.21.15.40:0/3516974628 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0037070 con 0x7f91c00c41d0 2024-04-26T23:04:18.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.227+0000 7f91be000640 1 -- 172.21.15.40:0/3516974628 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91b0032440 con 0x7f91c00c41d0 2024-04-26T23:04:18.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.227+0000 7f91be000640 1 -- 172.21.15.40:0/3516974628 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0032810 con 0x7f91c00c41d0 2024-04-26T23:04:18.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.227+0000 7f91c5800640 1 -- 172.21.15.40:0/3516974628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91c00c41d0 msgr2=0x7f91c00cfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:18.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.227+0000 7f91c5800640 1 --2- 172.21.15.40:0/3516974628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91c00c41d0 0x7f91c00cfc30 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009870 tx=0x7f91b002a270 comp rx=0 tx=0).stop 2024-04-26T23:04:18.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.227+0000 7f91c5800640 1 -- 172.21.15.40:0/3516974628 shutdown_connections 2024-04-26T23:04:18.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.227+0000 7f91c5800640 1 --2- 172.21.15.40:0/3516974628 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91c01678f0 0x7f91c00cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:18.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.227+0000 7f91c5800640 1 --2- 172.21.15.40:0/3516974628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91c00c41d0 0x7f91c00cfc30 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:18.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.227+0000 7f91c5800640 1 -- 172.21.15.40:0/3516974628 >> 172.21.15.40:0/3516974628 conn(0x7f91c00d0be0 msgr2=0x7f91c00d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:18.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.227+0000 7f91c5800640 1 -- 172.21.15.40:0/3516974628 shutdown_connections 2024-04-26T23:04:18.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.227+0000 7f91c5800640 1 -- 172.21.15.40:0/3516974628 wait complete. 2024-04-26T23:04:18.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.229+0000 7f91c5800640 1 Processor -- start 2024-04-26T23:04:18.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.229+0000 7f91c5800640 1 -- start start 2024-04-26T23:04:18.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.230+0000 7f91c5800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91c0073f30 0x7f91c00c3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:18.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.230+0000 7f91c5800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91c00c41d0 0x7f91c0166740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:18.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.230+0000 7f91c5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91c01678f0 0x7f91c00bf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:18.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.230+0000 7f91c5800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f91c00d0170 con 0x7f91c01678f0 2024-04-26T23:04:18.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.230+0000 7f91c5800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f91c00cf430 con 0x7f91c00c41d0 2024-04-26T23:04:18.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.230+0000 7f91c5800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f91c00ce7b0 con 0x7f91c0073f30 2024-04-26T23:04:18.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.231+0000 7f91bf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91c01678f0 0x7f91c00bf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:18.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.231+0000 7f91bf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91c01678f0 0x7f91c00bf9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42328/0 (socket says 172.21.15.40:42328) 2024-04-26T23:04:18.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.231+0000 7f91bf400640 1 -- 172.21.15.40:0/1974129841 learned_addr learned my addr 172.21.15.40:0/1974129841 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:18.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.231+0000 7f91bea00640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91c0073f30 0x7f91c00c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:18.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.231+0000 7f91b6000640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91c00c41d0 0x7f91c0166740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:18.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.231+0000 7f91bf400640 1 -- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91c0073f30 msgr2=0x7f91c00c3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:18.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.231+0000 7f91bf400640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91c0073f30 0x7f91c00c3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:18.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.231+0000 7f91bf400640 1 -- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91c00c41d0 msgr2=0x7f91c0166740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:18.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.231+0000 7f91bf400640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91c00c41d0 0x7f91c0166740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:18.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.231+0000 7f91bf400640 1 -- 172.21.15.40:0/1974129841 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c00b5390 con 0x7f91c01678f0 2024-04-26T23:04:18.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.231+0000 7f91bf400640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91c01678f0 0x7f91c00bf9d0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009fd0 tx=0x7f91b800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:18.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.232+0000 7f91b7400640 1 -- 172.21.15.40:0/1974129841 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b8014420 con 0x7f91c01678f0 2024-04-26T23:04:18.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.232+0000 7f91b7400640 1 -- 172.21.15.40:0/1974129841 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91b8019070 con 0x7f91c01678f0 2024-04-26T23:04:18.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.232+0000 7f91b7400640 1 -- 172.21.15.40:0/1974129841 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b8014800 con 0x7f91c01678f0 2024-04-26T23:04:18.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.232+0000 7f91c5800640 1 -- 172.21.15.40:0/1974129841 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91c00b5670 con 0x7f91c01678f0 2024-04-26T23:04:18.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.232+0000 7f91c5800640 1 -- 172.21.15.40:0/1974129841 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91c00b5c30 con 0x7f91c01678f0 2024-04-26T23:04:18.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.233+0000 7f91c5800640 1 -- 172.21.15.40:0/1974129841 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91c00694f0 con 0x7f91c01678f0 2024-04-26T23:04:18.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.235+0000 7f91b7400640 1 -- 172.21.15.40:0/1974129841 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f91b80149a0 con 0x7f91c01678f0 2024-04-26T23:04:18.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.236+0000 7f91b7400640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f919c06c8f0 0x7f919c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:18.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.236+0000 7f91b7400640 1 -- 172.21.15.40:0/1974129841 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f91b808c5b0 con 0x7f91c01678f0 2024-04-26T23:04:18.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.237+0000 7f91bea00640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f919c06c8f0 0x7f919c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:18.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.238+0000 7f91bea00640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f919c06c8f0 0x7f919c06edb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f91b0034000 tx=0x7f91b002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:18.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.241+0000 7f91b7400640 1 -- 172.21.15.40:0/1974129841 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f91b805a7b0 con 0x7f91c01678f0 2024-04-26T23:04:18.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-04-26T23:04:18.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-04-26T23:04:18.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-04-26T23:04:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:18.476+0000 7f91c5800640 1 -- 172.21.15.40:0/1974129841 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f91c00d0430 con 0x7f91c01678f0 2024-04-26T23:04:19.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.097+0000 7f91b7400640 1 -- 172.21.15.40:0/1974129841 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v34) ==== 118+0+0 (secure 0 0 0) 0x7f91b805a0b0 con 0x7f91c01678f0 2024-04-26T23:04:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.101+0000 7f91c5800640 1 -- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f919c06c8f0 msgr2=0x7f919c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.101+0000 7f91c5800640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f919c06c8f0 0x7f919c06edb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f91b0034000 tx=0x7f91b002e000 comp rx=0 tx=0).stop 2024-04-26T23:04:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.101+0000 7f91c5800640 1 -- 172.21.15.40:0/1974129841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91c01678f0 msgr2=0x7f91c00bf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.101+0000 7f91c5800640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91c01678f0 0x7f91c00bf9d0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009fd0 tx=0x7f91b800e4b0 comp rx=0 tx=0).stop 2024-04-26T23:04:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.101+0000 7f91c5800640 1 -- 172.21.15.40:0/1974129841 shutdown_connections 2024-04-26T23:04:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.101+0000 7f91c5800640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f919c06c8f0 0x7f919c06edb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.101+0000 7f91c5800640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91c01678f0 0x7f91c00bf9d0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.101+0000 7f91c5800640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91c00c41d0 0x7f91c0166740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.101+0000 7f91c5800640 1 --2- 172.21.15.40:0/1974129841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91c0073f30 0x7f91c00c3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.101+0000 7f91c5800640 1 -- 172.21.15.40:0/1974129841 >> 172.21.15.40:0/1974129841 conn(0x7f91c00d0be0 msgr2=0x7f91c00c6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.101+0000 7f91c5800640 1 -- 172.21.15.40:0/1974129841 shutdown_connections 2024-04-26T23:04:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.101+0000 7f91c5800640 1 -- 172.21.15.40:0/1974129841 wait complete. 2024-04-26T23:04:19.121 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-26T23:04:19.122 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-26T23:04:19.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.344+0000 7f139d400640 1 Processor -- start 2024-04-26T23:04:19.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.345+0000 7f139d400640 1 -- start start 2024-04-26T23:04:19.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.346+0000 7f139d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f13980b5500 0x7f13980b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:19.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.346+0000 7f139d400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13980d3dc0 con 0x7f13980b5120 2024-04-26T23:04:19.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.346+0000 7f139d400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13980b75b0 con 0x7f13980b59b0 2024-04-26T23:04:19.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.346+0000 7f139d400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f13980d03b0 con 0x7f13980b5500 2024-04-26T23:04:19.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.346+0000 7f1397400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f13980b5500 0x7f13980b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:19.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.346+0000 7f1397400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f13980b5500 0x7f13980b5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42342/0 (socket says 172.21.15.40:42342) 2024-04-26T23:04:19.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.346+0000 7f1397400640 1 -- 172.21.15.40:0/1363362608 learned_addr learned my addr 172.21.15.40:0/1363362608 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:19.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.347+0000 7f1397400640 1 -- 172.21.15.40:0/1363362608 >> v1:172.21.15.103:6789/0 conn(0x7f13980b59b0 legacy=0x7f13980b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:19.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.347+0000 7f1397400640 1 -- 172.21.15.40:0/1363362608 >> v1:172.21.15.40:6789/0 conn(0x7f13980b5120 legacy=0x7f13980b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:19.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.347+0000 7f1397400640 1 -- 172.21.15.40:0/1363362608 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13980d0600 con 0x7f13980b5500 2024-04-26T23:04:19.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.348+0000 7f1397400640 1 --2- 172.21.15.40:0/1363362608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13980b5500 0x7f13980b5ee0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f13840098a0 tx=0x7f138402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c839cba573f7ec8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:19.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.348+0000 7f1396a00640 1 -- 172.21.15.40:0/1363362608 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384037070 con 0x7f13980b5500 2024-04-26T23:04:19.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.348+0000 7f1396a00640 1 -- 172.21.15.40:0/1363362608 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1384032440 con 0x7f13980b5500 2024-04-26T23:04:19.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.348+0000 7f1396a00640 1 -- 172.21.15.40:0/1363362608 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384032810 con 0x7f13980b5500 2024-04-26T23:04:19.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.349+0000 7f139d400640 1 -- 172.21.15.40:0/1363362608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13980b5500 msgr2=0x7f13980b5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:19.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.349+0000 7f139d400640 1 --2- 172.21.15.40:0/1363362608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13980b5500 0x7f13980b5ee0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f13840098a0 tx=0x7f138402a1c0 comp rx=0 tx=0).stop 2024-04-26T23:04:19.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.349+0000 7f139d400640 1 -- 172.21.15.40:0/1363362608 shutdown_connections 2024-04-26T23:04:19.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.349+0000 7f139d400640 1 --2- 172.21.15.40:0/1363362608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13980b5500 0x7f13980b5ee0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:19.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.349+0000 7f139d400640 1 -- 172.21.15.40:0/1363362608 >> 172.21.15.40:0/1363362608 conn(0x7f13980d4db0 msgr2=0x7f13980d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:19.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.349+0000 7f139d400640 1 -- 172.21.15.40:0/1363362608 shutdown_connections 2024-04-26T23:04:19.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.349+0000 7f139d400640 1 -- 172.21.15.40:0/1363362608 wait complete. 2024-04-26T23:04:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.351+0000 7f139d400640 1 Processor -- start 2024-04-26T23:04:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.351+0000 7f139d400640 1 -- start start 2024-04-26T23:04:19.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.352+0000 7f139d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13980b5120 0x7f1398158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:19.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.352+0000 7f139d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13980b59b0 0x7f139815b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:19.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.352+0000 7f139d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f139815bc60 0x7f13980b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:19.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.352+0000 7f139d400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f13980b6f40 con 0x7f13980b5120 2024-04-26T23:04:19.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.352+0000 7f139d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13980d0bc0 con 0x7f139815bc60 2024-04-26T23:04:19.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.352+0000 7f139d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f13980b70c0 con 0x7f13980b59b0 2024-04-26T23:04:19.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.353+0000 7f1397e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13980b5120 0x7f1398158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:19.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.353+0000 7f1397e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13980b5120 0x7f1398158760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42354/0 (socket says 172.21.15.40:42354) 2024-04-26T23:04:19.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.353+0000 7f1397e00640 1 -- 172.21.15.40:0/231984671 learned_addr learned my addr 172.21.15.40:0/231984671 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:19.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.353+0000 7f1397400640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13980b59b0 0x7f139815b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:19.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.353+0000 7f139ca00640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f139815bc60 0x7f13980b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:19.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.353+0000 7f1397e00640 1 -- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13980b59b0 msgr2=0x7f139815b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:19.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.353+0000 7f1397e00640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13980b59b0 0x7f139815b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:19.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.353+0000 7f1397e00640 1 -- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f139815bc60 msgr2=0x7f13980b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:19.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.353+0000 7f1397e00640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f139815bc60 0x7f13980b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:19.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.353+0000 7f1397e00640 1 -- 172.21.15.40:0/231984671 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13981597a0 con 0x7f13980b5120 2024-04-26T23:04:19.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.354+0000 7f1397e00640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13980b5120 0x7f1398158760 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f138800c400 tx=0x7f138800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:19.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.354+0000 7f138fe00640 1 -- 172.21.15.40:0/231984671 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13880052d0 con 0x7f13980b5120 2024-04-26T23:04:19.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.354+0000 7f138fe00640 1 -- 172.21.15.40:0/231984671 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1388005490 con 0x7f13980b5120 2024-04-26T23:04:19.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.354+0000 7f138fe00640 1 -- 172.21.15.40:0/231984671 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388007260 con 0x7f13980b5120 2024-04-26T23:04:19.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.354+0000 7f139d400640 1 -- 172.21.15.40:0/231984671 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1398159a80 con 0x7f13980b5120 2024-04-26T23:04:19.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.354+0000 7f139d400640 1 -- 172.21.15.40:0/231984671 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f139815a040 con 0x7f13980b5120 2024-04-26T23:04:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.355+0000 7f139d400640 1 -- 172.21.15.40:0/231984671 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1398069370 con 0x7f13980b5120 2024-04-26T23:04:19.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.356+0000 7f138fe00640 1 -- 172.21.15.40:0/231984671 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1388007400 con 0x7f13980b5120 2024-04-26T23:04:19.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.357+0000 7f138fe00640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f137806c920 0x7f137806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:19.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.357+0000 7f138fe00640 1 -- 172.21.15.40:0/231984671 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f138808b270 con 0x7f13980b5120 2024-04-26T23:04:19.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.358+0000 7f1397400640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f137806c920 0x7f137806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:19.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.359+0000 7f1397400640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f137806c920 0x7f137806ede0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1384034000 tx=0x7f138402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:19.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.362+0000 7f138fe00640 1 -- 172.21.15.40:0/231984671 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1388059470 con 0x7f13980b5120 2024-04-26T23:04:19.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:19.611+0000 7f139d400640 1 -- 172.21.15.40:0/231984671 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f13980d2470 con 0x7f13980b5120 2024-04-26T23:04:20.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.107+0000 7f138fe00640 1 -- 172.21.15.40:0/231984671 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v35) ==== 122+0+0 (secure 0 0 0) 0x7f1388058d70 con 0x7f13980b5120 2024-04-26T23:04:20.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.110+0000 7f139d400640 1 -- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f137806c920 msgr2=0x7f137806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:20.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.110+0000 7f139d400640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f137806c920 0x7f137806ede0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1384034000 tx=0x7f138402e000 comp rx=0 tx=0).stop 2024-04-26T23:04:20.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.110+0000 7f139d400640 1 -- 172.21.15.40:0/231984671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13980b5120 msgr2=0x7f1398158760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:20.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.110+0000 7f139d400640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13980b5120 0x7f1398158760 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f138800c400 tx=0x7f138800c8c0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.110+0000 7f139d400640 1 -- 172.21.15.40:0/231984671 shutdown_connections 2024-04-26T23:04:20.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.110+0000 7f139d400640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f137806c920 0x7f137806ede0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.110+0000 7f139d400640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f139815bc60 0x7f13980b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.110+0000 7f139d400640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13980b59b0 0x7f139815b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.110+0000 7f139d400640 1 --2- 172.21.15.40:0/231984671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13980b5120 0x7f1398158760 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.110+0000 7f139d400640 1 -- 172.21.15.40:0/231984671 >> 172.21.15.40:0/231984671 conn(0x7f13980d4db0 msgr2=0x7f13980b8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:20.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.110+0000 7f139d400640 1 -- 172.21.15.40:0/231984671 shutdown_connections 2024-04-26T23:04:20.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.111+0000 7f139d400640 1 -- 172.21.15.40:0/231984671 wait complete. 2024-04-26T23:04:20.134 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-26T23:04:20.135 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:04:20.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.351+0000 7f3e69200640 1 Processor -- start 2024-04-26T23:04:20.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.352+0000 7f3e69200640 1 -- start start 2024-04-26T23:04:20.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.353+0000 7f3e69200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3e640b6220 0x7f3e640d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:20.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.353+0000 7f3e69200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e640d39f0 0x7f3e640b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:20.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.353+0000 7f3e69200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e64162c40 con 0x7f3e640d08d0 2024-04-26T23:04:20.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.353+0000 7f3e69200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f3e640d2d80 con 0x7f3e640b6220 2024-04-26T23:04:20.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.353+0000 7f3e69200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3e640b7c80 con 0x7f3e640d39f0 2024-04-26T23:04:20.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.353+0000 7f3e63400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3e640b6220 0x7f3e640d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:20.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.354+0000 7f3e62a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3e640d08d0 0x7f3e640d32e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57960/0 (socket says 172.21.15.40:57960) 2024-04-26T23:04:20.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.354+0000 7f3e62a00640 1 -- 172.21.15.40:0/4219602816 learned_addr learned my addr 172.21.15.40:0/4219602816 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.354+0000 7f3e63e00640 1 --2- 172.21.15.40:0/4219602816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e640d39f0 0x7f3e640b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.354+0000 7f3e63400640 1 -- 172.21.15.40:0/4219602816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e640d39f0 msgr2=0x7f3e640b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.354+0000 7f3e63400640 1 --2- 172.21.15.40:0/4219602816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e640d39f0 0x7f3e640b8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.354+0000 7f3e63400640 1 -- 172.21.15.40:0/4219602816 >> v1:172.21.15.40:6789/0 conn(0x7f3e640d08d0 legacy=0x7f3e640d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.354+0000 7f3e63400640 1 -- 172.21.15.40:0/4219602816 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e640b7f60 con 0x7f3e640b6220 2024-04-26T23:04:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.354+0000 7f3e62000640 1 -- 172.21.15.40:0/4219602816 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3916833816 0 0) 0x7f3e64162c40 con 0x7f3e640d08d0 2024-04-26T23:04:20.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.356+0000 7f3e63400640 1 --2- 172.21.15.40:0/4219602816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e640b6220 0x7f3e640d0390 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3e540097a0 tx=0x7f3e5402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f338bc165a5c74 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:20.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.356+0000 7f3e62000640 1 -- 172.21.15.40:0/4219602816 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e54037070 con 0x7f3e640b6220 2024-04-26T23:04:20.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.356+0000 7f3e62000640 1 -- 172.21.15.40:0/4219602816 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e54032440 con 0x7f3e640b6220 2024-04-26T23:04:20.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.356+0000 7f3e62000640 1 -- 172.21.15.40:0/4219602816 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e54032810 con 0x7f3e640b6220 2024-04-26T23:04:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.357+0000 7f3e69200640 1 -- 172.21.15.40:0/4219602816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e640b6220 msgr2=0x7f3e640d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.357+0000 7f3e69200640 1 --2- 172.21.15.40:0/4219602816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e640b6220 0x7f3e640d0390 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3e540097a0 tx=0x7f3e5402a180 comp rx=0 tx=0).stop 2024-04-26T23:04:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.357+0000 7f3e69200640 1 -- 172.21.15.40:0/4219602816 shutdown_connections 2024-04-26T23:04:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.357+0000 7f3e69200640 1 --2- 172.21.15.40:0/4219602816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e640d39f0 0x7f3e640b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.357+0000 7f3e69200640 1 --2- 172.21.15.40:0/4219602816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e640b6220 0x7f3e640d0390 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.357+0000 7f3e69200640 1 -- 172.21.15.40:0/4219602816 >> 172.21.15.40:0/4219602816 conn(0x7f3e640d4d90 msgr2=0x7f3e640d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.357+0000 7f3e69200640 1 -- 172.21.15.40:0/4219602816 shutdown_connections 2024-04-26T23:04:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.357+0000 7f3e69200640 1 -- 172.21.15.40:0/4219602816 wait complete. 2024-04-26T23:04:20.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.359+0000 7f3e69200640 1 Processor -- start 2024-04-26T23:04:20.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.359+0000 7f3e69200640 1 -- start start 2024-04-26T23:04:20.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.360+0000 7f3e69200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e640b6220 0x7f3e6415c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:20.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.360+0000 7f3e69200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e640d08d0 0x7f3e640b1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:20.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.360+0000 7f3e69200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e640d39f0 0x7f3e640b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:20.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.360+0000 7f3e69200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3e640b9260 con 0x7f3e640d39f0 2024-04-26T23:04:20.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.360+0000 7f3e69200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3e640b8520 con 0x7f3e640b6220 2024-04-26T23:04:20.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.360+0000 7f3e69200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3e640b93e0 con 0x7f3e640d08d0 2024-04-26T23:04:20.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.360+0000 7f3e63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e640d39f0 0x7f3e640b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:20.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.360+0000 7f3e63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e640d39f0 0x7f3e640b4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42386/0 (socket says 172.21.15.40:42386) 2024-04-26T23:04:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.360+0000 7f3e63e00640 1 -- 172.21.15.40:0/63377091 learned_addr learned my addr 172.21.15.40:0/63377091 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.361+0000 7f3e62a00640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e640d08d0 0x7f3e640b1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.361+0000 7f3e63400640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e640b6220 0x7f3e6415c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.361+0000 7f3e63e00640 1 -- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e640d08d0 msgr2=0x7f3e640b1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.361+0000 7f3e63e00640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e640d08d0 0x7f3e640b1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.361+0000 7f3e63e00640 1 -- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e640b6220 msgr2=0x7f3e6415c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.361+0000 7f3e63e00640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e640b6220 0x7f3e6415c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.361+0000 7f3e63e00640 1 -- 172.21.15.40:0/63377091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e6415de70 con 0x7f3e640d39f0 2024-04-26T23:04:20.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.361+0000 7f3e63e00640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e640d39f0 0x7f3e640b4be0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c007d80 tx=0x7f3e5c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:20.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.362+0000 7f3e5b400640 1 -- 172.21.15.40:0/63377091 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c014420 con 0x7f3e640d39f0 2024-04-26T23:04:20.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.362+0000 7f3e5b400640 1 -- 172.21.15.40:0/63377091 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e5c019070 con 0x7f3e640d39f0 2024-04-26T23:04:20.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.362+0000 7f3e5b400640 1 -- 172.21.15.40:0/63377091 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c014800 con 0x7f3e640d39f0 2024-04-26T23:04:20.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.362+0000 7f3e69200640 1 -- 172.21.15.40:0/63377091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e6415e110 con 0x7f3e640d39f0 2024-04-26T23:04:20.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.362+0000 7f3e69200640 1 -- 172.21.15.40:0/63377091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e640b5760 con 0x7f3e640d39f0 2024-04-26T23:04:20.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.363+0000 7f3e69200640 1 -- 172.21.15.40:0/63377091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e6415e2e0 con 0x7f3e640d39f0 2024-04-26T23:04:20.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.364+0000 7f3e5b400640 1 -- 172.21.15.40:0/63377091 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3e5c0149a0 con 0x7f3e640d39f0 2024-04-26T23:04:20.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.365+0000 7f3e5b400640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e4006c8f0 0x7f3e4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:20.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.366+0000 7f3e5b400640 1 -- 172.21.15.40:0/63377091 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f3e5c08c5b0 con 0x7f3e640d39f0 2024-04-26T23:04:20.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.366+0000 7f3e63400640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e4006c8f0 0x7f3e4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:20.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.367+0000 7f3e63400640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e4006c8f0 0x7f3e4006edb0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3e54034000 tx=0x7f3e5402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.370+0000 7f3e5b400640 1 -- 172.21.15.40:0/63377091 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3e5c05a840 con 0x7f3e640d39f0 2024-04-26T23:04:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.618+0000 7f3e69200640 1 -- 172.21.15.40:0/63377091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e64069370 con 0x7f3e640d39f0 2024-04-26T23:04:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.619+0000 7f3e5b400640 1 -- 172.21.15.40:0/63377091 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5178 (secure 0 0 0) 0x7f3e5c05a140 con 0x7f3e640d39f0 2024-04-26T23:04:20.619 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:20.620 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/189004193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":189004193},{"type":"v1","addr":"172.21.15.103:6835","nonce":189004193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5284,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/670995452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":670995452},{"type":"v1","addr":"172.21.15.103:6837","nonce":670995452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":6027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/2304583125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2304583125},{"type":"v1","addr":"172.21.15.40:6834","nonce":2304583125}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:04:16.558591+0000","modified":"2024-04-26T23:04:19.612962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6036":{"gid":6036,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/618018110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":618018110},{"type":"v1","addr":"172.21.15.40:6835","nonce":618018110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6036,"qdb_cluster":[6036]},"id":3}]} 2024-04-26T23:04:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 35 2024-04-26T23:04:20.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.621+0000 7f3e69200640 1 -- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e4006c8f0 msgr2=0x7f3e4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:20.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.621+0000 7f3e69200640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e4006c8f0 0x7f3e4006edb0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3e54034000 tx=0x7f3e5402e000 comp rx=0 tx=0).stop 2024-04-26T23:04:20.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.621+0000 7f3e69200640 1 -- 172.21.15.40:0/63377091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e640d39f0 msgr2=0x7f3e640b4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:20.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.621+0000 7f3e69200640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e640d39f0 0x7f3e640b4be0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c007d80 tx=0x7f3e5c00e4b0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.621+0000 7f3e69200640 1 -- 172.21.15.40:0/63377091 shutdown_connections 2024-04-26T23:04:20.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.621+0000 7f3e69200640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e4006c8f0 0x7f3e4006edb0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.621+0000 7f3e69200640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e640d39f0 0x7f3e640b4be0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.621+0000 7f3e69200640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e640d08d0 0x7f3e640b1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.621+0000 7f3e69200640 1 --2- 172.21.15.40:0/63377091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e640b6220 0x7f3e6415c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.621+0000 7f3e69200640 1 -- 172.21.15.40:0/63377091 >> 172.21.15.40:0/63377091 conn(0x7f3e640d4d90 msgr2=0x7f3e640d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:20.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.621+0000 7f3e69200640 1 -- 172.21.15.40:0/63377091 shutdown_connections 2024-04-26T23:04:20.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.621+0000 7f3e69200640 1 -- 172.21.15.40:0/63377091 wait complete. 2024-04-26T23:04:20.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:04:20.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.842+0000 7f8f38a00640 1 Processor -- start 2024-04-26T23:04:20.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.842+0000 7f8f38a00640 1 -- start start 2024-04-26T23:04:20.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.844+0000 7f8f38a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f340ccca0 0x7f8f340afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:20.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.844+0000 7f8f38a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f340af7b0 con 0x7f8f3415f2d0 2024-04-26T23:04:20.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.844+0000 7f8f38a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f340afa90 con 0x7f8f340ce5e0 2024-04-26T23:04:20.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.844+0000 7f8f38a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f8f340cbd90 con 0x7f8f340ccca0 2024-04-26T23:04:20.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.844+0000 7f8f32800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8f3415f2d0 0x7f8f340c58b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57970/0 (socket says 172.21.15.40:57970) 2024-04-26T23:04:20.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.844+0000 7f8f32800640 1 -- 172.21.15.40:0/2125479450 learned_addr learned my addr 172.21.15.40:0/2125479450 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:20.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.845+0000 7f8f33200640 1 --2- 172.21.15.40:0/2125479450 >> v2:172.21.15.40:3300/0 conn(0x7f8f340ccca0 0x7f8f340afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:20.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.845+0000 7f8f31400640 1 -- 172.21.15.40:0/2125479450 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3568594957 0 0) 0x7f8f340af7b0 con 0x7f8f3415f2d0 2024-04-26T23:04:20.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.845+0000 7f8f31400640 1 -- 172.21.15.40:0/2125479450 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f18003520 con 0x7f8f3415f2d0 2024-04-26T23:04:20.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.845+0000 7f8f33200640 1 -- 172.21.15.40:0/2125479450 >> v1:172.21.15.103:6789/0 conn(0x7f8f340ce5e0 legacy=0x7f8f340cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:20.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.845+0000 7f8f33200640 1 -- 172.21.15.40:0/2125479450 >> v1:172.21.15.40:6789/0 conn(0x7f8f3415f2d0 legacy=0x7f8f340c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:20.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.845+0000 7f8f33200640 1 -- 172.21.15.40:0/2125479450 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f340cc070 con 0x7f8f340ccca0 2024-04-26T23:04:20.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.846+0000 7f8f33200640 1 --2- 172.21.15.40:0/2125479450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f340ccca0 0x7f8f340afdf0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009dc0 tx=0x7f8f2c02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b7c507aef36dea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.847+0000 7f8f31400640 1 -- 172.21.15.40:0/2125479450 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c007740 con 0x7f8f340ccca0 2024-04-26T23:04:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.847+0000 7f8f31400640 1 -- 172.21.15.40:0/2125479450 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f2c0078e0 con 0x7f8f340ccca0 2024-04-26T23:04:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.847+0000 7f8f31400640 1 -- 172.21.15.40:0/2125479450 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c007c90 con 0x7f8f340ccca0 2024-04-26T23:04:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.847+0000 7f8f38a00640 1 -- 172.21.15.40:0/2125479450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f340ccca0 msgr2=0x7f8f340afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.847+0000 7f8f38a00640 1 --2- 172.21.15.40:0/2125479450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f340ccca0 0x7f8f340afdf0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009dc0 tx=0x7f8f2c02a730 comp rx=0 tx=0).stop 2024-04-26T23:04:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.847+0000 7f8f38a00640 1 -- 172.21.15.40:0/2125479450 shutdown_connections 2024-04-26T23:04:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.847+0000 7f8f38a00640 1 --2- 172.21.15.40:0/2125479450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f340ccca0 0x7f8f340afdf0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.848+0000 7f8f38a00640 1 -- 172.21.15.40:0/2125479450 >> 172.21.15.40:0/2125479450 conn(0x7f8f340d0a10 msgr2=0x7f8f340d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.848+0000 7f8f38a00640 1 -- 172.21.15.40:0/2125479450 shutdown_connections 2024-04-26T23:04:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.848+0000 7f8f38a00640 1 -- 172.21.15.40:0/2125479450 wait complete. 2024-04-26T23:04:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.849+0000 7f8f38a00640 1 Processor -- start 2024-04-26T23:04:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.850+0000 7f8f38a00640 1 -- start start 2024-04-26T23:04:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.851+0000 7f8f38a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8f340ccca0 0x7f8f3414a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.851+0000 7f8f38a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f340ce5e0 0x7f8f341567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.851+0000 7f8f38a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3415f2d0 0x7f8f3415e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.851+0000 7f8f38a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8f340c5fc0 con 0x7f8f3415f2d0 2024-04-26T23:04:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.851+0000 7f8f38a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3406e270 con 0x7f8f340ce5e0 2024-04-26T23:04:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.851+0000 7f8f38a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8f340c7e30 con 0x7f8f340ccca0 2024-04-26T23:04:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.851+0000 7f8f33200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3415f2d0 0x7f8f3415e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.851+0000 7f8f33200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3415f2d0 0x7f8f3415e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42408/0 (socket says 172.21.15.40:42408) 2024-04-26T23:04:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.851+0000 7f8f33200640 1 -- 172.21.15.40:0/1194502213 learned_addr learned my addr 172.21.15.40:0/1194502213 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.851+0000 7f8f32800640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8f340ccca0 0x7f8f3414a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.851+0000 7f8f31e00640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f340ce5e0 0x7f8f341567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.852+0000 7f8f33200640 1 -- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8f340ccca0 msgr2=0x7f8f3414a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.852+0000 7f8f33200640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8f340ccca0 0x7f8f3414a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.852+0000 7f8f33200640 1 -- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f340ce5e0 msgr2=0x7f8f341567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.852+0000 7f8f33200640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f340ce5e0 0x7f8f341567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.852+0000 7f8f33200640 1 -- 172.21.15.40:0/1194502213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f34157ca0 con 0x7f8f3415f2d0 2024-04-26T23:04:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.852+0000 7f8f33200640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3415f2d0 0x7f8f3415e060 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009ee0 tx=0x7f8f2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.852+0000 7f8f2aa00640 1 -- 172.21.15.40:0/1194502213 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c009450 con 0x7f8f3415f2d0 2024-04-26T23:04:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.852+0000 7f8f2aa00640 1 -- 172.21.15.40:0/1194502213 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f2c004240 con 0x7f8f3415f2d0 2024-04-26T23:04:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.852+0000 7f8f2aa00640 1 -- 172.21.15.40:0/1194502213 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c004610 con 0x7f8f3415f2d0 2024-04-26T23:04:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.852+0000 7f8f38a00640 1 -- 172.21.15.40:0/1194502213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f34157120 con 0x7f8f3415f2d0 2024-04-26T23:04:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.853+0000 7f8f38a00640 1 -- 172.21.15.40:0/1194502213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f341576e0 con 0x7f8f3415f2d0 2024-04-26T23:04:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.854+0000 7f8f38a00640 1 -- 172.21.15.40:0/1194502213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f34069370 con 0x7f8f3415f2d0 2024-04-26T23:04:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.855+0000 7f8f2aa00640 1 -- 172.21.15.40:0/1194502213 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8f2c039040 con 0x7f8f3415f2d0 2024-04-26T23:04:20.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.856+0000 7f8f2aa00640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f1406c8f0 0x7f8f1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:20.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.857+0000 7f8f2aa00640 1 -- 172.21.15.40:0/1194502213 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f8f2c0acce0 con 0x7f8f3415f2d0 2024-04-26T23:04:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.857+0000 7f8f32800640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f1406c8f0 0x7f8f1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.858+0000 7f8f32800640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f1406c8f0 0x7f8f1406edb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8f240011e0 tx=0x7f8f24009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:20.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:20.862+0000 7f8f2aa00640 1 -- 172.21.15.40:0/1194502213 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8f2c040030 con 0x7f8f3415f2d0 2024-04-26T23:04:21.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.066+0000 7f8f38a00640 1 -- 172.21.15.40:0/1194502213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8f340c8030 con 0x7f8f3415f2d0 2024-04-26T23:04:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.068+0000 7f8f2aa00640 1 -- 172.21.15.40:0/1194502213 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+16906 (secure 0 0 0) 0x7f8f2c07af70 con 0x7f8f3415f2d0 2024-04-26T23:04:21.069 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:21.070 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":52,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:04:18.071156+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:04:14.334803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-26T23:04:15.535793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:04:21.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.071+0000 7f8f38a00640 1 -- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f1406c8f0 msgr2=0x7f8f1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.071+0000 7f8f38a00640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f1406c8f0 0x7f8f1406edb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8f240011e0 tx=0x7f8f24009040 comp rx=0 tx=0).stop 2024-04-26T23:04:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.071+0000 7f8f38a00640 1 -- 172.21.15.40:0/1194502213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3415f2d0 msgr2=0x7f8f3415e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.071+0000 7f8f38a00640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3415f2d0 0x7f8f3415e060 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009ee0 tx=0x7f8f2c004a40 comp rx=0 tx=0).stop 2024-04-26T23:04:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.071+0000 7f8f38a00640 1 -- 172.21.15.40:0/1194502213 shutdown_connections 2024-04-26T23:04:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.071+0000 7f8f38a00640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f1406c8f0 0x7f8f1406edb0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.071+0000 7f8f38a00640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3415f2d0 0x7f8f3415e060 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.071+0000 7f8f38a00640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f340ce5e0 0x7f8f341567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.071+0000 7f8f38a00640 1 --2- 172.21.15.40:0/1194502213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8f340ccca0 0x7f8f3414a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.071+0000 7f8f38a00640 1 -- 172.21.15.40:0/1194502213 >> 172.21.15.40:0/1194502213 conn(0x7f8f340d0a10 msgr2=0x7f8f340b0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.071+0000 7f8f38a00640 1 -- 172.21.15.40:0/1194502213 shutdown_connections 2024-04-26T23:04:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.071+0000 7f8f38a00640 1 -- 172.21.15.40:0/1194502213 wait complete. 2024-04-26T23:04:21.089 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T23:04:21.090 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:04:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.299+0000 7f2db2200640 1 Processor -- start 2024-04-26T23:04:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.299+0000 7f2db2200640 1 -- start start 2024-04-26T23:04:21.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.300+0000 7f2db2200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dac161850 0x7f2dac0d1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:21.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.300+0000 7f2db2200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dac0b63e0 0x7f2dac161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:21.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.300+0000 7f2db2200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dac0b4110 con 0x7f2dac0b4770 2024-04-26T23:04:21.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.300+0000 7f2db2200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f2dac0d3850 con 0x7f2dac0b63e0 2024-04-26T23:04:21.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.300+0000 7f2db2200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2dac0d3b30 con 0x7f2dac161850 2024-04-26T23:04:21.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.300+0000 7f2dabe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2dac0b4770 0x7f2dac0d1980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57978/0 (socket says 172.21.15.40:57978) 2024-04-26T23:04:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.300+0000 7f2dabe00640 1 -- 172.21.15.40:0/113038232 learned_addr learned my addr 172.21.15.40:0/113038232 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.300+0000 7f2db1800640 1 --2- 172.21.15.40:0/113038232 >> v2:172.21.15.40:3300/0 conn(0x7f2dac0b63e0 0x7f2dac161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.301+0000 7f2db0e00640 1 --2- 172.21.15.40:0/113038232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dac161850 0x7f2dac0d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.301+0000 7f2db1800640 1 -- 172.21.15.40:0/113038232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dac161850 msgr2=0x7f2dac0d1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.301+0000 7f2db1800640 1 --2- 172.21.15.40:0/113038232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dac161850 0x7f2dac0d1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.301+0000 7f2db1800640 1 -- 172.21.15.40:0/113038232 >> v1:172.21.15.40:6789/0 conn(0x7f2dac0b4770 legacy=0x7f2dac0d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.301+0000 7f2db1800640 1 -- 172.21.15.40:0/113038232 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dac0b3d70 con 0x7f2dac0b63e0 2024-04-26T23:04:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.301+0000 7f2dab400640 1 -- 172.21.15.40:0/113038232 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3437870243 0 0) 0x7f2dac0b4110 con 0x7f2dac0b4770 2024-04-26T23:04:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.302+0000 7f2db1800640 1 --2- 172.21.15.40:0/113038232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac0b63e0 0x7f2dac161c30 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009d90 tx=0x7f2da402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b435b8b5984803 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.302+0000 7f2dab400640 1 -- 172.21.15.40:0/113038232 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da4031600 con 0x7f2dac0b63e0 2024-04-26T23:04:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.302+0000 7f2dab400640 1 -- 172.21.15.40:0/113038232 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da40317a0 con 0x7f2dac0b63e0 2024-04-26T23:04:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.302+0000 7f2dab400640 1 -- 172.21.15.40:0/113038232 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da4031b70 con 0x7f2dac0b63e0 2024-04-26T23:04:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.303+0000 7f2db2200640 1 -- 172.21.15.40:0/113038232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac0b63e0 msgr2=0x7f2dac161c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.303+0000 7f2db2200640 1 --2- 172.21.15.40:0/113038232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac0b63e0 0x7f2dac161c30 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009d90 tx=0x7f2da402a5d0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.303+0000 7f2db2200640 1 -- 172.21.15.40:0/113038232 shutdown_connections 2024-04-26T23:04:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.303+0000 7f2db2200640 1 --2- 172.21.15.40:0/113038232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac0b63e0 0x7f2dac161c30 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.303+0000 7f2db2200640 1 --2- 172.21.15.40:0/113038232 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dac161850 0x7f2dac0d1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.303+0000 7f2db2200640 1 -- 172.21.15.40:0/113038232 >> 172.21.15.40:0/113038232 conn(0x7f2dac0d4e10 msgr2=0x7f2dac0d5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.303+0000 7f2db2200640 1 -- 172.21.15.40:0/113038232 shutdown_connections 2024-04-26T23:04:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.303+0000 7f2db2200640 1 -- 172.21.15.40:0/113038232 wait complete. 2024-04-26T23:04:21.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.305+0000 7f2db2200640 1 Processor -- start 2024-04-26T23:04:21.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.305+0000 7f2db2200640 1 -- start start 2024-04-26T23:04:21.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.306+0000 7f2db2200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dac0b4770 0x7f2dac1501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:21.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.306+0000 7f2db2200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac0b63e0 0x7f2dac0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:21.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.306+0000 7f2db2200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dac161850 0x7f2dac0cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:21.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.306+0000 7f2db2200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac0d2210 con 0x7f2dac0b63e0 2024-04-26T23:04:21.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.306+0000 7f2db2200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac0d2090 con 0x7f2dac0b4770 2024-04-26T23:04:21.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.306+0000 7f2db2200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2dac162170 con 0x7f2dac161850 2024-04-26T23:04:21.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.307+0000 7f2dabe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac0b63e0 0x7f2dac0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:21.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.307+0000 7f2dabe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac0b63e0 0x7f2dac0cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42434/0 (socket says 172.21.15.40:42434) 2024-04-26T23:04:21.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.307+0000 7f2dabe00640 1 -- 172.21.15.40:0/2417754220 learned_addr learned my addr 172.21.15.40:0/2417754220 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:21.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.307+0000 7f2db1800640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dac161850 0x7f2dac0cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.307+0000 7f2db0e00640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dac0b4770 0x7f2dac1501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.307+0000 7f2dabe00640 1 -- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dac161850 msgr2=0x7f2dac0cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.307+0000 7f2dabe00640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dac161850 0x7f2dac0cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.307+0000 7f2dabe00640 1 -- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dac0b4770 msgr2=0x7f2dac1501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.307+0000 7f2dabe00640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dac0b4770 0x7f2dac1501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.307+0000 7f2dabe00640 1 -- 172.21.15.40:0/2417754220 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dac150f00 con 0x7f2dac0b63e0 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.307+0000 7f2db1800640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dac161850 0x7f2dac0cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.307+0000 7f2db0e00640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dac0b4770 0x7f2dac1501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.307+0000 7f2dabe00640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac0b63e0 0x7f2dac0cd030 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400c4e0 tx=0x7f2d9400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.308+0000 7f2da8c00640 1 -- 172.21.15.40:0/2417754220 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94018070 con 0x7f2dac0b63e0 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.308+0000 7f2da8c00640 1 -- 172.21.15.40:0/2417754220 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d94007500 con 0x7f2dac0b63e0 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.308+0000 7f2da8c00640 1 -- 172.21.15.40:0/2417754220 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94004a20 con 0x7f2dac0b63e0 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.308+0000 7f2db2200640 1 -- 172.21.15.40:0/2417754220 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dac0cdab0 con 0x7f2dac0b63e0 2024-04-26T23:04:21.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.308+0000 7f2db2200640 1 -- 172.21.15.40:0/2417754220 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dac151480 con 0x7f2dac0b63e0 2024-04-26T23:04:21.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.309+0000 7f2db2200640 1 -- 172.21.15.40:0/2417754220 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dac069a80 con 0x7f2dac0b63e0 2024-04-26T23:04:21.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.310+0000 7f2da8c00640 1 -- 172.21.15.40:0/2417754220 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d9401b070 con 0x7f2dac0b63e0 2024-04-26T23:04:21.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.311+0000 7f2da8c00640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d8806c940 0x7f2d8806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:21.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.312+0000 7f2da8c00640 1 -- 172.21.15.40:0/2417754220 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f2d9408b140 con 0x7f2dac0b63e0 2024-04-26T23:04:21.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.312+0000 7f2db0e00640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d8806c940 0x7f2d8806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:21.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.313+0000 7f2db0e00640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d8806c940 0x7f2d8806ee00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c006fd0 tx=0x7f2d9c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:21.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.316+0000 7f2da8c00640 1 -- 172.21.15.40:0/2417754220 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2d94059510 con 0x7f2dac0b63e0 2024-04-26T23:04:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.519+0000 7f2db2200640 1 -- 172.21.15.40:0/2417754220 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2dac0008d0 con 0x7f2d8806c940 2024-04-26T23:04:21.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.538+0000 7f2da8c00640 1 -- 172.21.15.40:0/2417754220 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251939 (secure 0 0 0) 0x7f2dac0008d0 con 0x7f2d8806c940 2024-04-26T23:04:21.542 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:21.543 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:04:21.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.543+0000 7f2db2200640 1 -- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d8806c940 msgr2=0x7f2d8806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:21.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.543+0000 7f2db2200640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d8806c940 0x7f2d8806ee00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c006fd0 tx=0x7f2d9c008040 comp rx=0 tx=0).stop 2024-04-26T23:04:21.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.543+0000 7f2db2200640 1 -- 172.21.15.40:0/2417754220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac0b63e0 msgr2=0x7f2dac0cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:21.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.543+0000 7f2db2200640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac0b63e0 0x7f2dac0cd030 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400c4e0 tx=0x7f2d9400c9a0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.543+0000 7f2db2200640 1 -- 172.21.15.40:0/2417754220 shutdown_connections 2024-04-26T23:04:21.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.543+0000 7f2db2200640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d8806c940 0x7f2d8806ee00 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.543+0000 7f2db2200640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dac161850 0x7f2dac0cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.543+0000 7f2db2200640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac0b63e0 0x7f2dac0cd030 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.543+0000 7f2db2200640 1 --2- 172.21.15.40:0/2417754220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dac0b4770 0x7f2dac1501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.543+0000 7f2db2200640 1 -- 172.21.15.40:0/2417754220 >> 172.21.15.40:0/2417754220 conn(0x7f2dac0d4e10 msgr2=0x7f2dac0b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:21.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.544+0000 7f2db2200640 1 -- 172.21.15.40:0/2417754220 shutdown_connections 2024-04-26T23:04:21.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.544+0000 7f2db2200640 1 -- 172.21.15.40:0/2417754220 wait complete. 2024-04-26T23:04:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-04-26T23:04:21.215706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":230492,"kb_used_data":5972,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754744228,"statfs":{"total":773094113280,"available":772858089472,"internally_reserved":0,"allocated":6115328,"data_stored":3366240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":229756466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002164"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087916+0000","last_change":"2024-04-26T23:04:17.108756+0000","last_active":"2024-04-26T23:04:18.087916+0000","last_peered":"2024-04-26T23:04:18.087916+0000","last_clean":"2024-04-26T23:04:18.087916+0000","last_became_active":"2024-04-26T23:04:17.108423+0000","last_became_peered":"2024-04-26T23:04:17.108423+0000","last_unstale":"2024-04-26T23:04:18.087916+0000","last_undegraded":"2024-04-26T23:04:18.087916+0000","last_fullsized":"2024-04-26T23:04:18.087916+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:52:58.602974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101034+0000","last_change":"2024-04-26T23:04:17.086315+0000","last_active":"2024-04-26T23:04:18.101034+0000","last_peered":"2024-04-26T23:04:18.101034+0000","last_clean":"2024-04-26T23:04:18.101034+0000","last_became_active":"2024-04-26T23:04:17.086039+0000","last_became_peered":"2024-04-26T23:04:17.086039+0000","last_unstale":"2024-04-26T23:04:18.101034+0000","last_undegraded":"2024-04-26T23:04:18.101034+0000","last_fullsized":"2024-04-26T23:04:18.101034+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:38:49.132739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093341+0000","last_change":"2024-04-26T23:04:17.097344+0000","last_active":"2024-04-26T23:04:18.093341+0000","last_peered":"2024-04-26T23:04:18.093341+0000","last_clean":"2024-04-26T23:04:18.093341+0000","last_became_active":"2024-04-26T23:04:17.091514+0000","last_became_peered":"2024-04-26T23:04:17.091514+0000","last_unstale":"2024-04-26T23:04:18.093341+0000","last_undegraded":"2024-04-26T23:04:18.093341+0000","last_fullsized":"2024-04-26T23:04:18.093341+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:05.987818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101168+0000","last_change":"2024-04-26T23:04:17.094548+0000","last_active":"2024-04-26T23:04:18.101168+0000","last_peered":"2024-04-26T23:04:18.101168+0000","last_clean":"2024-04-26T23:04:18.101168+0000","last_became_active":"2024-04-26T23:04:17.094062+0000","last_became_peered":"2024-04-26T23:04:17.094062+0000","last_unstale":"2024-04-26T23:04:18.101168+0000","last_undegraded":"2024-04-26T23:04:18.101168+0000","last_fullsized":"2024-04-26T23:04:18.101168+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:06.221700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.098205+0000","last_change":"2024-04-26T23:04:17.090776+0000","last_active":"2024-04-26T23:04:17.098205+0000","last_peered":"2024-04-26T23:04:17.098205+0000","last_clean":"2024-04-26T23:04:17.098205+0000","last_became_active":"2024-04-26T23:04:17.089579+0000","last_became_peered":"2024-04-26T23:04:17.089579+0000","last_unstale":"2024-04-26T23:04:17.098205+0000","last_undegraded":"2024-04-26T23:04:17.098205+0000","last_fullsized":"2024-04-26T23:04:17.098205+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:49:36.443734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087287+0000","last_change":"2024-04-26T23:04:17.093394+0000","last_active":"2024-04-26T23:04:18.087287+0000","last_peered":"2024-04-26T23:04:18.087287+0000","last_clean":"2024-04-26T23:04:18.087287+0000","last_became_active":"2024-04-26T23:04:17.093123+0000","last_became_peered":"2024-04-26T23:04:17.093123+0000","last_unstale":"2024-04-26T23:04:18.087287+0000","last_undegraded":"2024-04-26T23:04:18.087287+0000","last_fullsized":"2024-04-26T23:04:18.087287+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:59:44.819074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093073+0000","last_change":"2024-04-26T23:04:17.089724+0000","last_active":"2024-04-26T23:04:18.093073+0000","last_peered":"2024-04-26T23:04:18.093073+0000","last_clean":"2024-04-26T23:04:18.093073+0000","last_became_active":"2024-04-26T23:04:17.089424+0000","last_became_peered":"2024-04-26T23:04:17.089424+0000","last_unstale":"2024-04-26T23:04:18.093073+0000","last_undegraded":"2024-04-26T23:04:18.093073+0000","last_fullsized":"2024-04-26T23:04:18.093073+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:44.792411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.111093+0000","last_change":"2024-04-26T23:04:17.108239+0000","last_active":"2024-04-26T23:04:17.111093+0000","last_peered":"2024-04-26T23:04:17.111093+0000","last_clean":"2024-04-26T23:04:17.111093+0000","last_became_active":"2024-04-26T23:04:17.107492+0000","last_became_peered":"2024-04-26T23:04:17.107492+0000","last_unstale":"2024-04-26T23:04:17.111093+0000","last_undegraded":"2024-04-26T23:04:17.111093+0000","last_fullsized":"2024-04-26T23:04:17.111093+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:14:26.705521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.088053+0000","last_change":"2024-04-26T23:04:17.085761+0000","last_active":"2024-04-26T23:04:17.088053+0000","last_peered":"2024-04-26T23:04:17.088053+0000","last_clean":"2024-04-26T23:04:17.088053+0000","last_became_active":"2024-04-26T23:04:17.085411+0000","last_became_peered":"2024-04-26T23:04:17.085411+0000","last_unstale":"2024-04-26T23:04:17.088053+0000","last_undegraded":"2024-04-26T23:04:17.088053+0000","last_fullsized":"2024-04-26T23:04:17.088053+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:38:49.132739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087874+0000","last_change":"2024-04-26T23:04:17.107215+0000","last_active":"2024-04-26T23:04:18.087874+0000","last_peered":"2024-04-26T23:04:18.087874+0000","last_clean":"2024-04-26T23:04:18.087874+0000","last_became_active":"2024-04-26T23:04:17.106918+0000","last_became_peered":"2024-04-26T23:04:17.106918+0000","last_unstale":"2024-04-26T23:04:18.087874+0000","last_undegraded":"2024-04-26T23:04:18.087874+0000","last_fullsized":"2024-04-26T23:04:18.087874+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:25.066755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.238045+0000","last_change":"2024-04-26T23:04:17.089834+0000","last_active":"2024-04-26T23:04:18.238045+0000","last_peered":"2024-04-26T23:04:18.238045+0000","last_clean":"2024-04-26T23:04:18.238045+0000","last_became_active":"2024-04-26T23:04:17.089565+0000","last_became_peered":"2024-04-26T23:04:17.089565+0000","last_unstale":"2024-04-26T23:04:18.238045+0000","last_undegraded":"2024-04-26T23:04:18.238045+0000","last_fullsized":"2024-04-26T23:04:18.238045+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:21.424428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.101623+0000","last_change":"2024-04-26T23:04:17.091262+0000","last_active":"2024-04-26T23:04:17.101623+0000","last_peered":"2024-04-26T23:04:17.101623+0000","last_clean":"2024-04-26T23:04:17.101623+0000","last_became_active":"2024-04-26T23:04:17.090829+0000","last_became_peered":"2024-04-26T23:04:17.090829+0000","last_unstale":"2024-04-26T23:04:17.101623+0000","last_undegraded":"2024-04-26T23:04:17.101623+0000","last_fullsized":"2024-04-26T23:04:17.101623+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:05:41.081684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087374+0000","last_change":"2024-04-26T23:04:17.107886+0000","last_active":"2024-04-26T23:04:18.087374+0000","last_peered":"2024-04-26T23:04:18.087374+0000","last_clean":"2024-04-26T23:04:18.087374+0000","last_became_active":"2024-04-26T23:04:17.106514+0000","last_became_peered":"2024-04-26T23:04:17.106514+0000","last_unstale":"2024-04-26T23:04:18.087374+0000","last_undegraded":"2024-04-26T23:04:18.087374+0000","last_fullsized":"2024-04-26T23:04:18.087374+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:18:31.829572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093330+0000","last_change":"2024-04-26T23:04:17.095553+0000","last_active":"2024-04-26T23:04:18.093330+0000","last_peered":"2024-04-26T23:04:18.093330+0000","last_clean":"2024-04-26T23:04:18.093330+0000","last_became_active":"2024-04-26T23:04:17.091689+0000","last_became_peered":"2024-04-26T23:04:17.091689+0000","last_unstale":"2024-04-26T23:04:18.093330+0000","last_undegraded":"2024-04-26T23:04:18.093330+0000","last_fullsized":"2024-04-26T23:04:18.093330+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:52:37.327132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.099835+0000","last_change":"2024-04-26T23:04:17.087714+0000","last_active":"2024-04-26T23:04:18.099835+0000","last_peered":"2024-04-26T23:04:18.099835+0000","last_clean":"2024-04-26T23:04:18.099835+0000","last_became_active":"2024-04-26T23:04:17.087424+0000","last_became_peered":"2024-04-26T23:04:17.087424+0000","last_unstale":"2024-04-26T23:04:18.099835+0000","last_undegraded":"2024-04-26T23:04:18.099835+0000","last_fullsized":"2024-04-26T23:04:18.099835+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:05:24.474044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087000+0000","last_change":"2024-04-26T23:04:17.096307+0000","last_active":"2024-04-26T23:04:18.087000+0000","last_peered":"2024-04-26T23:04:18.087000+0000","last_clean":"2024-04-26T23:04:18.087000+0000","last_became_active":"2024-04-26T23:04:17.096038+0000","last_became_peered":"2024-04-26T23:04:17.096038+0000","last_unstale":"2024-04-26T23:04:18.087000+0000","last_undegraded":"2024-04-26T23:04:18.087000+0000","last_fullsized":"2024-04-26T23:04:18.087000+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:38:25.040887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.094190+0000","last_change":"2024-04-26T23:04:17.090691+0000","last_active":"2024-04-26T23:04:18.094190+0000","last_peered":"2024-04-26T23:04:18.094190+0000","last_clean":"2024-04-26T23:04:18.094190+0000","last_became_active":"2024-04-26T23:04:17.083165+0000","last_became_peered":"2024-04-26T23:04:17.083165+0000","last_unstale":"2024-04-26T23:04:18.094190+0000","last_undegraded":"2024-04-26T23:04:18.094190+0000","last_fullsized":"2024-04-26T23:04:18.094190+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:26:49.475479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101349+0000","last_change":"2024-04-26T23:04:17.086481+0000","last_active":"2024-04-26T23:04:18.101349+0000","last_peered":"2024-04-26T23:04:18.101349+0000","last_clean":"2024-04-26T23:04:18.101349+0000","last_became_active":"2024-04-26T23:04:17.086196+0000","last_became_peered":"2024-04-26T23:04:17.086196+0000","last_unstale":"2024-04-26T23:04:18.101349+0000","last_undegraded":"2024-04-26T23:04:18.101349+0000","last_fullsized":"2024-04-26T23:04:18.101349+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:33.055716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093514+0000","last_change":"2024-04-26T23:04:17.097764+0000","last_active":"2024-04-26T23:04:18.093514+0000","last_peered":"2024-04-26T23:04:18.093514+0000","last_clean":"2024-04-26T23:04:18.093514+0000","last_became_active":"2024-04-26T23:04:17.097476+0000","last_became_peered":"2024-04-26T23:04:17.097476+0000","last_unstale":"2024-04-26T23:04:18.093514+0000","last_undegraded":"2024-04-26T23:04:18.093514+0000","last_fullsized":"2024-04-26T23:04:18.093514+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:57.457992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092519+0000","last_change":"2024-04-26T23:04:17.087958+0000","last_active":"2024-04-26T23:04:18.092519+0000","last_peered":"2024-04-26T23:04:18.092519+0000","last_clean":"2024-04-26T23:04:18.092519+0000","last_became_active":"2024-04-26T23:04:17.087650+0000","last_became_peered":"2024-04-26T23:04:17.087650+0000","last_unstale":"2024-04-26T23:04:18.092519+0000","last_undegraded":"2024-04-26T23:04:18.092519+0000","last_fullsized":"2024-04-26T23:04:18.092519+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:51:06.254495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093223+0000","last_change":"2024-04-26T23:04:17.089714+0000","last_active":"2024-04-26T23:04:18.093223+0000","last_peered":"2024-04-26T23:04:18.093223+0000","last_clean":"2024-04-26T23:04:18.093223+0000","last_became_active":"2024-04-26T23:04:17.088110+0000","last_became_peered":"2024-04-26T23:04:17.088110+0000","last_unstale":"2024-04-26T23:04:18.093223+0000","last_undegraded":"2024-04-26T23:04:18.093223+0000","last_fullsized":"2024-04-26T23:04:18.093223+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:59:44.819074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.103518+0000","last_change":"2024-04-26T23:04:17.097312+0000","last_active":"2024-04-26T23:04:17.103518+0000","last_peered":"2024-04-26T23:04:17.103518+0000","last_clean":"2024-04-26T23:04:17.103518+0000","last_became_active":"2024-04-26T23:04:17.097067+0000","last_became_peered":"2024-04-26T23:04:17.097067+0000","last_unstale":"2024-04-26T23:04:17.103518+0000","last_undegraded":"2024-04-26T23:04:17.103518+0000","last_fullsized":"2024-04-26T23:04:17.103518+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:52:37.327132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.238084+0000","last_change":"2024-04-26T23:04:17.087678+0000","last_active":"2024-04-26T23:04:18.238084+0000","last_peered":"2024-04-26T23:04:18.238084+0000","last_clean":"2024-04-26T23:04:18.238084+0000","last_became_active":"2024-04-26T23:04:17.087399+0000","last_became_peered":"2024-04-26T23:04:17.087399+0000","last_unstale":"2024-04-26T23:04:18.238084+0000","last_undegraded":"2024-04-26T23:04:18.238084+0000","last_fullsized":"2024-04-26T23:04:18.238084+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:59.965974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092918+0000","last_change":"2024-04-26T23:04:17.082519+0000","last_active":"2024-04-26T23:04:18.092918+0000","last_peered":"2024-04-26T23:04:18.092918+0000","last_clean":"2024-04-26T23:04:18.092918+0000","last_became_active":"2024-04-26T23:04:17.081193+0000","last_became_peered":"2024-04-26T23:04:17.081193+0000","last_unstale":"2024-04-26T23:04:18.092918+0000","last_undegraded":"2024-04-26T23:04:18.092918+0000","last_fullsized":"2024-04-26T23:04:18.092918+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:05:41.081684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.094024+0000","last_change":"2024-04-26T23:04:17.096363+0000","last_active":"2024-04-26T23:04:18.094024+0000","last_peered":"2024-04-26T23:04:18.094024+0000","last_clean":"2024-04-26T23:04:18.094024+0000","last_became_active":"2024-04-26T23:04:17.091540+0000","last_became_peered":"2024-04-26T23:04:17.091540+0000","last_unstale":"2024-04-26T23:04:18.094024+0000","last_undegraded":"2024-04-26T23:04:18.094024+0000","last_fullsized":"2024-04-26T23:04:18.094024+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:18:50.045304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.107069+0000","last_change":"2024-04-26T23:04:17.102799+0000","last_active":"2024-04-26T23:04:17.107069+0000","last_peered":"2024-04-26T23:04:17.107069+0000","last_clean":"2024-04-26T23:04:17.107069+0000","last_became_active":"2024-04-26T23:04:17.102433+0000","last_became_peered":"2024-04-26T23:04:17.102433+0000","last_unstale":"2024-04-26T23:04:17.107069+0000","last_undegraded":"2024-04-26T23:04:17.107069+0000","last_fullsized":"2024-04-26T23:04:17.107069+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:34.886245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087996+0000","last_change":"2024-04-26T23:04:17.108342+0000","last_active":"2024-04-26T23:04:18.087996+0000","last_peered":"2024-04-26T23:04:18.087996+0000","last_clean":"2024-04-26T23:04:18.087996+0000","last_became_active":"2024-04-26T23:04:17.107942+0000","last_became_peered":"2024-04-26T23:04:17.107942+0000","last_unstale":"2024-04-26T23:04:18.087996+0000","last_undegraded":"2024-04-26T23:04:18.087996+0000","last_fullsized":"2024-04-26T23:04:18.087996+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:00.840289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092938+0000","last_change":"2024-04-26T23:04:17.081008+0000","last_active":"2024-04-26T23:04:18.092938+0000","last_peered":"2024-04-26T23:04:18.092938+0000","last_clean":"2024-04-26T23:04:18.092938+0000","last_became_active":"2024-04-26T23:04:17.080710+0000","last_became_peered":"2024-04-26T23:04:17.080710+0000","last_unstale":"2024-04-26T23:04:18.092938+0000","last_undegraded":"2024-04-26T23:04:18.092938+0000","last_fullsized":"2024-04-26T23:04:18.092938+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:38:25.040887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092867+0000","last_change":"2024-04-26T23:04:17.087561+0000","last_active":"2024-04-26T23:04:18.092867+0000","last_peered":"2024-04-26T23:04:18.092867+0000","last_clean":"2024-04-26T23:04:18.092867+0000","last_became_active":"2024-04-26T23:04:17.087254+0000","last_became_peered":"2024-04-26T23:04:17.087254+0000","last_unstale":"2024-04-26T23:04:18.092867+0000","last_undegraded":"2024-04-26T23:04:18.092867+0000","last_fullsized":"2024-04-26T23:04:18.092867+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:35:15.713183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.100313+0000","last_change":"2024-04-26T23:04:17.107516+0000","last_active":"2024-04-26T23:04:18.100313+0000","last_peered":"2024-04-26T23:04:18.100313+0000","last_clean":"2024-04-26T23:04:18.100313+0000","last_became_active":"2024-04-26T23:04:17.107267+0000","last_became_peered":"2024-04-26T23:04:17.107267+0000","last_unstale":"2024-04-26T23:04:18.100313+0000","last_undegraded":"2024-04-26T23:04:18.100313+0000","last_fullsized":"2024-04-26T23:04:18.100313+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:52:58.602974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093205+0000","last_change":"2024-04-26T23:04:17.083089+0000","last_active":"2024-04-26T23:04:18.093205+0000","last_peered":"2024-04-26T23:04:18.093205+0000","last_clean":"2024-04-26T23:04:18.093205+0000","last_became_active":"2024-04-26T23:04:17.081306+0000","last_became_peered":"2024-04-26T23:04:17.081306+0000","last_unstale":"2024-04-26T23:04:18.093205+0000","last_undegraded":"2024-04-26T23:04:18.093205+0000","last_fullsized":"2024-04-26T23:04:18.093205+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:41.757362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.090486+0000","last_change":"2024-04-26T23:04:17.095460+0000","last_active":"2024-04-26T23:04:18.090486+0000","last_peered":"2024-04-26T23:04:18.090486+0000","last_clean":"2024-04-26T23:04:18.090486+0000","last_became_active":"2024-04-26T23:04:17.094048+0000","last_became_peered":"2024-04-26T23:04:17.094048+0000","last_unstale":"2024-04-26T23:04:18.090486+0000","last_undegraded":"2024-04-26T23:04:18.090486+0000","last_fullsized":"2024-04-26T23:04:18.090486+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:00.840289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.100205+0000","last_change":"2024-04-26T23:04:17.086930+0000","last_active":"2024-04-26T23:04:18.100205+0000","last_peered":"2024-04-26T23:04:18.100205+0000","last_clean":"2024-04-26T23:04:18.100205+0000","last_became_active":"2024-04-26T23:04:17.085984+0000","last_became_peered":"2024-04-26T23:04:17.085984+0000","last_unstale":"2024-04-26T23:04:18.100205+0000","last_undegraded":"2024-04-26T23:04:18.100205+0000","last_fullsized":"2024-04-26T23:04:18.100205+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:25.066755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":18,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093485+0000","last_change":"2024-04-26T23:04:16.089299+0000","last_active":"2024-04-26T23:04:18.093485+0000","last_peered":"2024-04-26T23:04:18.093485+0000","last_clean":"2024-04-26T23:04:18.093485+0000","last_became_active":"2024-04-26T23:04:16.088966+0000","last_became_peered":"2024-04-26T23:04:16.088966+0000","last_unstale":"2024-04-26T23:04:18.093485+0000","last_undegraded":"2024-04-26T23:04:18.093485+0000","last_fullsized":"2024-04-26T23:04:18.093485+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:25:44.106018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.091140+0000","last_change":"2024-04-26T23:04:17.088848+0000","last_active":"2024-04-26T23:04:18.091140+0000","last_peered":"2024-04-26T23:04:18.091140+0000","last_clean":"2024-04-26T23:04:18.091140+0000","last_became_active":"2024-04-26T23:04:17.087714+0000","last_became_peered":"2024-04-26T23:04:17.087714+0000","last_unstale":"2024-04-26T23:04:18.091140+0000","last_undegraded":"2024-04-26T23:04:18.091140+0000","last_fullsized":"2024-04-26T23:04:18.091140+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:02.222941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087806+0000","last_change":"2024-04-26T23:04:16.099694+0000","last_active":"2024-04-26T23:04:18.087806+0000","last_peered":"2024-04-26T23:04:18.087806+0000","last_clean":"2024-04-26T23:04:18.087806+0000","last_became_active":"2024-04-26T23:04:16.099200+0000","last_became_peered":"2024-04-26T23:04:16.099200+0000","last_unstale":"2024-04-26T23:04:18.087806+0000","last_undegraded":"2024-04-26T23:04:18.087806+0000","last_fullsized":"2024-04-26T23:04:18.087806+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:11.272237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092958+0000","last_change":"2024-04-26T23:04:17.091407+0000","last_active":"2024-04-26T23:04:18.092958+0000","last_peered":"2024-04-26T23:04:18.092958+0000","last_clean":"2024-04-26T23:04:18.092958+0000","last_became_active":"2024-04-26T23:04:17.088002+0000","last_became_peered":"2024-04-26T23:04:17.088002+0000","last_unstale":"2024-04-26T23:04:18.092958+0000","last_undegraded":"2024-04-26T23:04:18.092958+0000","last_fullsized":"2024-04-26T23:04:18.092958+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:57:35.423782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.088153+0000","last_change":"2024-04-26T23:04:16.091143+0000","last_active":"2024-04-26T23:04:18.088153+0000","last_peered":"2024-04-26T23:04:18.088153+0000","last_clean":"2024-04-26T23:04:18.088153+0000","last_became_active":"2024-04-26T23:04:16.090657+0000","last_became_peered":"2024-04-26T23:04:16.090657+0000","last_unstale":"2024-04-26T23:04:18.088153+0000","last_undegraded":"2024-04-26T23:04:18.088153+0000","last_fullsized":"2024-04-26T23:04:18.088153+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:25:44.106018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.105066+0000","last_change":"2024-04-26T23:04:17.092804+0000","last_active":"2024-04-26T23:04:17.105066+0000","last_peered":"2024-04-26T23:04:17.105066+0000","last_clean":"2024-04-26T23:04:17.105066+0000","last_became_active":"2024-04-26T23:04:17.092393+0000","last_became_peered":"2024-04-26T23:04:17.092393+0000","last_unstale":"2024-04-26T23:04:17.105066+0000","last_undegraded":"2024-04-26T23:04:17.105066+0000","last_fullsized":"2024-04-26T23:04:17.105066+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:41.757362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.091583+0000","last_change":"2024-04-26T23:04:16.099288+0000","last_active":"2024-04-26T23:04:18.091583+0000","last_peered":"2024-04-26T23:04:18.091583+0000","last_clean":"2024-04-26T23:04:18.091583+0000","last_became_active":"2024-04-26T23:04:16.099003+0000","last_became_peered":"2024-04-26T23:04:16.099003+0000","last_unstale":"2024-04-26T23:04:18.091583+0000","last_undegraded":"2024-04-26T23:04:18.091583+0000","last_fullsized":"2024-04-26T23:04:18.091583+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:51:05.238308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.104652+0000","last_change":"2024-04-26T23:04:17.103015+0000","last_active":"2024-04-26T23:04:17.104652+0000","last_peered":"2024-04-26T23:04:17.104652+0000","last_clean":"2024-04-26T23:04:17.104652+0000","last_became_active":"2024-04-26T23:04:17.102694+0000","last_became_peered":"2024-04-26T23:04:17.102694+0000","last_unstale":"2024-04-26T23:04:17.104652+0000","last_undegraded":"2024-04-26T23:04:17.104652+0000","last_fullsized":"2024-04-26T23:04:17.104652+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:10.801318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092994+0000","last_change":"2024-04-26T23:04:16.093076+0000","last_active":"2024-04-26T23:04:18.092994+0000","last_peered":"2024-04-26T23:04:18.092994+0000","last_clean":"2024-04-26T23:04:18.092994+0000","last_became_active":"2024-04-26T23:04:16.092763+0000","last_became_peered":"2024-04-26T23:04:16.092763+0000","last_unstale":"2024-04-26T23:04:18.092994+0000","last_undegraded":"2024-04-26T23:04:18.092994+0000","last_fullsized":"2024-04-26T23:04:18.092994+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:49:35.427546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.100802+0000","last_change":"2024-04-26T23:04:17.110748+0000","last_active":"2024-04-26T23:04:18.100802+0000","last_peered":"2024-04-26T23:04:18.100802+0000","last_clean":"2024-04-26T23:04:18.100802+0000","last_became_active":"2024-04-26T23:04:17.110535+0000","last_became_peered":"2024-04-26T23:04:17.110535+0000","last_unstale":"2024-04-26T23:04:18.100802+0000","last_undegraded":"2024-04-26T23:04:18.100802+0000","last_fullsized":"2024-04-26T23:04:18.100802+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:00.840289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.114619+0000","last_change":"2024-04-26T23:04:16.098768+0000","last_active":"2024-04-26T23:04:17.114619+0000","last_peered":"2024-04-26T23:04:17.114619+0000","last_clean":"2024-04-26T23:04:17.114619+0000","last_became_active":"2024-04-26T23:04:16.098447+0000","last_became_peered":"2024-04-26T23:04:16.098447+0000","last_unstale":"2024-04-26T23:04:17.114619+0000","last_undegraded":"2024-04-26T23:04:17.114619+0000","last_fullsized":"2024-04-26T23:04:17.114619+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:35.744533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.101019+0000","last_change":"2024-04-26T23:04:17.089918+0000","last_active":"2024-04-26T23:04:17.101019+0000","last_peered":"2024-04-26T23:04:17.101019+0000","last_clean":"2024-04-26T23:04:17.101019+0000","last_became_active":"2024-04-26T23:04:17.089657+0000","last_became_peered":"2024-04-26T23:04:17.089657+0000","last_unstale":"2024-04-26T23:04:17.101019+0000","last_undegraded":"2024-04-26T23:04:17.101019+0000","last_fullsized":"2024-04-26T23:04:17.101019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:33.055716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092554+0000","last_change":"2024-04-26T23:04:16.094715+0000","last_active":"2024-04-26T23:04:18.092554+0000","last_peered":"2024-04-26T23:04:18.092554+0000","last_clean":"2024-04-26T23:04:18.092554+0000","last_became_active":"2024-04-26T23:04:16.094400+0000","last_became_peered":"2024-04-26T23:04:16.094400+0000","last_unstale":"2024-04-26T23:04:18.092554+0000","last_undegraded":"2024-04-26T23:04:18.092554+0000","last_fullsized":"2024-04-26T23:04:18.092554+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:11.272237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101060+0000","last_change":"2024-04-26T23:04:17.082328+0000","last_active":"2024-04-26T23:04:18.101060+0000","last_peered":"2024-04-26T23:04:18.101060+0000","last_clean":"2024-04-26T23:04:18.101060+0000","last_became_active":"2024-04-26T23:04:17.082039+0000","last_became_peered":"2024-04-26T23:04:17.082039+0000","last_unstale":"2024-04-26T23:04:18.101060+0000","last_undegraded":"2024-04-26T23:04:18.101060+0000","last_fullsized":"2024-04-26T23:04:18.101060+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:02.222941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092660+0000","last_change":"2024-04-26T23:04:16.087222+0000","last_active":"2024-04-26T23:04:18.092660+0000","last_peered":"2024-04-26T23:04:18.092660+0000","last_clean":"2024-04-26T23:04:18.092660+0000","last_became_active":"2024-04-26T23:04:16.086896+0000","last_became_peered":"2024-04-26T23:04:16.086896+0000","last_unstale":"2024-04-26T23:04:18.092660+0000","last_undegraded":"2024-04-26T23:04:18.092660+0000","last_fullsized":"2024-04-26T23:04:18.092660+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:35.744533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.099464+0000","last_change":"2024-04-26T23:04:17.097851+0000","last_active":"2024-04-26T23:04:17.099464+0000","last_peered":"2024-04-26T23:04:17.099464+0000","last_clean":"2024-04-26T23:04:17.099464+0000","last_became_active":"2024-04-26T23:04:17.097577+0000","last_became_peered":"2024-04-26T23:04:17.097577+0000","last_unstale":"2024-04-26T23:04:17.099464+0000","last_undegraded":"2024-04-26T23:04:17.099464+0000","last_fullsized":"2024-04-26T23:04:17.099464+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:34.013250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101228+0000","last_change":"2024-04-26T23:04:16.102075+0000","last_active":"2024-04-26T23:04:18.101228+0000","last_peered":"2024-04-26T23:04:18.101228+0000","last_clean":"2024-04-26T23:04:18.101228+0000","last_became_active":"2024-04-26T23:04:16.101000+0000","last_became_peered":"2024-04-26T23:04:16.101000+0000","last_unstale":"2024-04-26T23:04:18.101228+0000","last_undegraded":"2024-04-26T23:04:18.101228+0000","last_fullsized":"2024-04-26T23:04:18.101228+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:05:35.157689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.090353+0000","last_change":"2024-04-26T23:04:17.093421+0000","last_active":"2024-04-26T23:04:18.090353+0000","last_peered":"2024-04-26T23:04:18.090353+0000","last_clean":"2024-04-26T23:04:18.090353+0000","last_became_active":"2024-04-26T23:04:17.089057+0000","last_became_peered":"2024-04-26T23:04:17.089057+0000","last_unstale":"2024-04-26T23:04:18.090353+0000","last_undegraded":"2024-04-26T23:04:18.090353+0000","last_fullsized":"2024-04-26T23:04:18.090353+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:25.066755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.083925+0000","last_change":"2024-04-26T23:04:16.089350+0000","last_active":"2024-04-26T23:04:17.083925+0000","last_peered":"2024-04-26T23:04:17.083925+0000","last_clean":"2024-04-26T23:04:17.083925+0000","last_became_active":"2024-04-26T23:04:16.089062+0000","last_became_peered":"2024-04-26T23:04:16.089062+0000","last_unstale":"2024-04-26T23:04:17.083925+0000","last_undegraded":"2024-04-26T23:04:17.083925+0000","last_fullsized":"2024-04-26T23:04:17.083925+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:28:08.155712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092873+0000","last_change":"2024-04-26T23:04:17.091188+0000","last_active":"2024-04-26T23:04:18.092873+0000","last_peered":"2024-04-26T23:04:18.092873+0000","last_clean":"2024-04-26T23:04:18.092873+0000","last_became_active":"2024-04-26T23:04:17.089833+0000","last_became_peered":"2024-04-26T23:04:17.089833+0000","last_unstale":"2024-04-26T23:04:18.092873+0000","last_undegraded":"2024-04-26T23:04:18.092873+0000","last_fullsized":"2024-04-26T23:04:18.092873+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:30:47.888626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":17,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101415+0000","last_change":"2024-04-26T23:04:16.103455+0000","last_active":"2024-04-26T23:04:18.101415+0000","last_peered":"2024-04-26T23:04:18.101415+0000","last_clean":"2024-04-26T23:04:18.101415+0000","last_became_active":"2024-04-26T23:04:16.103184+0000","last_became_peered":"2024-04-26T23:04:16.103184+0000","last_unstale":"2024-04-26T23:04:18.101415+0000","last_undegraded":"2024-04-26T23:04:18.101415+0000","last_fullsized":"2024-04-26T23:04:18.101415+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:49:35.427546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.098402+0000","last_change":"2024-04-26T23:04:17.096503+0000","last_active":"2024-04-26T23:04:17.098402+0000","last_peered":"2024-04-26T23:04:17.098402+0000","last_clean":"2024-04-26T23:04:17.098402+0000","last_became_active":"2024-04-26T23:04:17.093367+0000","last_became_peered":"2024-04-26T23:04:17.093367+0000","last_unstale":"2024-04-26T23:04:17.098402+0000","last_undegraded":"2024-04-26T23:04:17.098402+0000","last_fullsized":"2024-04-26T23:04:17.098402+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:50:17.848964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":18,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.119345+0000","last_change":"2024-04-26T23:04:16.089752+0000","last_active":"2024-04-26T23:04:17.119345+0000","last_peered":"2024-04-26T23:04:17.119345+0000","last_clean":"2024-04-26T23:04:17.119345+0000","last_became_active":"2024-04-26T23:04:16.089465+0000","last_became_peered":"2024-04-26T23:04:16.089465+0000","last_unstale":"2024-04-26T23:04:17.119345+0000","last_undegraded":"2024-04-26T23:04:17.119345+0000","last_fullsized":"2024-04-26T23:04:17.119345+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:30:46.872439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.091135+0000","last_change":"2024-04-26T23:04:17.094537+0000","last_active":"2024-04-26T23:04:18.091135+0000","last_peered":"2024-04-26T23:04:18.091135+0000","last_clean":"2024-04-26T23:04:18.091135+0000","last_became_active":"2024-04-26T23:04:17.093102+0000","last_became_peered":"2024-04-26T23:04:17.093102+0000","last_unstale":"2024-04-26T23:04:18.091135+0000","last_undegraded":"2024-04-26T23:04:18.091135+0000","last_fullsized":"2024-04-26T23:04:18.091135+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:38:49.132739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093112+0000","last_change":"2024-04-26T23:04:16.094717+0000","last_active":"2024-04-26T23:04:18.093112+0000","last_peered":"2024-04-26T23:04:18.093112+0000","last_clean":"2024-04-26T23:04:18.093112+0000","last_became_active":"2024-04-26T23:04:16.094365+0000","last_became_peered":"2024-04-26T23:04:16.094365+0000","last_unstale":"2024-04-26T23:04:18.093112+0000","last_undegraded":"2024-04-26T23:04:18.093112+0000","last_fullsized":"2024-04-26T23:04:18.093112+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:32.997062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093554+0000","last_change":"2024-04-26T23:04:17.112073+0000","last_active":"2024-04-26T23:04:18.093554+0000","last_peered":"2024-04-26T23:04:18.093554+0000","last_clean":"2024-04-26T23:04:18.093554+0000","last_became_active":"2024-04-26T23:04:17.111658+0000","last_became_peered":"2024-04-26T23:04:17.111658+0000","last_unstale":"2024-04-26T23:04:18.093554+0000","last_undegraded":"2024-04-26T23:04:18.093554+0000","last_fullsized":"2024-04-26T23:04:18.093554+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:14:26.705521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092834+0000","last_change":"2024-04-26T23:04:16.080398+0000","last_active":"2024-04-26T23:04:18.092834+0000","last_peered":"2024-04-26T23:04:18.092834+0000","last_clean":"2024-04-26T23:04:18.092834+0000","last_became_active":"2024-04-26T23:04:16.079680+0000","last_became_peered":"2024-04-26T23:04:16.079680+0000","last_unstale":"2024-04-26T23:04:18.092834+0000","last_undegraded":"2024-04-26T23:04:18.092834+0000","last_fullsized":"2024-04-26T23:04:18.092834+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:36.222215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.102974+0000","last_change":"2024-04-26T23:04:17.098083+0000","last_active":"2024-04-26T23:04:17.102974+0000","last_peered":"2024-04-26T23:04:17.102974+0000","last_clean":"2024-04-26T23:04:17.102974+0000","last_became_active":"2024-04-26T23:04:17.097737+0000","last_became_peered":"2024-04-26T23:04:17.097737+0000","last_unstale":"2024-04-26T23:04:17.102974+0000","last_undegraded":"2024-04-26T23:04:17.102974+0000","last_fullsized":"2024-04-26T23:04:17.102974+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:05.987818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.099917+0000","last_change":"2024-04-26T23:04:16.101294+0000","last_active":"2024-04-26T23:04:18.099917+0000","last_peered":"2024-04-26T23:04:18.099917+0000","last_clean":"2024-04-26T23:04:18.099917+0000","last_became_active":"2024-04-26T23:04:16.100730+0000","last_became_peered":"2024-04-26T23:04:16.100730+0000","last_unstale":"2024-04-26T23:04:18.099917+0000","last_undegraded":"2024-04-26T23:04:18.099917+0000","last_fullsized":"2024-04-26T23:04:18.099917+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:35.744533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093766+0000","last_change":"2024-04-26T23:04:17.096063+0000","last_active":"2024-04-26T23:04:18.093766+0000","last_peered":"2024-04-26T23:04:18.093766+0000","last_clean":"2024-04-26T23:04:18.093766+0000","last_became_active":"2024-04-26T23:04:17.092943+0000","last_became_peered":"2024-04-26T23:04:17.092943+0000","last_unstale":"2024-04-26T23:04:18.093766+0000","last_undegraded":"2024-04-26T23:04:18.093766+0000","last_fullsized":"2024-04-26T23:04:18.093766+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:59.263980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":20,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.115391+0000","last_change":"2024-04-26T23:04:16.095953+0000","last_active":"2024-04-26T23:04:17.115391+0000","last_peered":"2024-04-26T23:04:17.115391+0000","last_clean":"2024-04-26T23:04:17.115391+0000","last_became_active":"2024-04-26T23:04:16.089330+0000","last_became_peered":"2024-04-26T23:04:16.089330+0000","last_unstale":"2024-04-26T23:04:17.115391+0000","last_undegraded":"2024-04-26T23:04:17.115391+0000","last_fullsized":"2024-04-26T23:04:17.115391+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:51:44.403748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.090742+0000","last_change":"2024-04-26T23:04:17.087779+0000","last_active":"2024-04-26T23:04:17.090742+0000","last_peered":"2024-04-26T23:04:17.090742+0000","last_clean":"2024-04-26T23:04:17.090742+0000","last_became_active":"2024-04-26T23:04:17.087322+0000","last_became_peered":"2024-04-26T23:04:17.087322+0000","last_unstale":"2024-04-26T23:04:17.090742+0000","last_undegraded":"2024-04-26T23:04:17.090742+0000","last_fullsized":"2024-04-26T23:04:17.090742+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:02.222941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.094135+0000","last_change":"2024-04-26T23:04:16.097363+0000","last_active":"2024-04-26T23:04:18.094135+0000","last_peered":"2024-04-26T23:04:18.094135+0000","last_clean":"2024-04-26T23:04:18.094135+0000","last_became_active":"2024-04-26T23:04:16.097089+0000","last_became_peered":"2024-04-26T23:04:16.097089+0000","last_unstale":"2024-04-26T23:04:18.094135+0000","last_undegraded":"2024-04-26T23:04:18.094135+0000","last_fullsized":"2024-04-26T23:04:18.094135+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:09:50.040264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087026+0000","last_change":"2024-04-26T23:04:17.092477+0000","last_active":"2024-04-26T23:04:18.087026+0000","last_peered":"2024-04-26T23:04:18.087026+0000","last_clean":"2024-04-26T23:04:18.087026+0000","last_became_active":"2024-04-26T23:04:17.092173+0000","last_became_peered":"2024-04-26T23:04:17.092173+0000","last_unstale":"2024-04-26T23:04:18.087026+0000","last_undegraded":"2024-04-26T23:04:18.087026+0000","last_fullsized":"2024-04-26T23:04:18.087026+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:57:35.423782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.238346+0000","last_change":"2024-04-26T23:04:16.094647+0000","last_active":"2024-04-26T23:04:18.238346+0000","last_peered":"2024-04-26T23:04:18.238346+0000","last_clean":"2024-04-26T23:04:18.238346+0000","last_became_active":"2024-04-26T23:04:16.094375+0000","last_became_peered":"2024-04-26T23:04:16.094375+0000","last_unstale":"2024-04-26T23:04:18.238346+0000","last_undegraded":"2024-04-26T23:04:18.238346+0000","last_fullsized":"2024-04-26T23:04:18.238346+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:51.393053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.104717+0000","last_change":"2024-04-26T23:04:17.103652+0000","last_active":"2024-04-26T23:04:17.104717+0000","last_peered":"2024-04-26T23:04:17.104717+0000","last_clean":"2024-04-26T23:04:17.104717+0000","last_became_active":"2024-04-26T23:04:17.103320+0000","last_became_peered":"2024-04-26T23:04:17.103320+0000","last_unstale":"2024-04-26T23:04:17.104717+0000","last_undegraded":"2024-04-26T23:04:17.104717+0000","last_fullsized":"2024-04-26T23:04:17.104717+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:51:06.254495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093434+0000","last_change":"2024-04-26T23:04:16.088844+0000","last_active":"2024-04-26T23:04:18.093434+0000","last_peered":"2024-04-26T23:04:18.093434+0000","last_clean":"2024-04-26T23:04:18.093434+0000","last_became_active":"2024-04-26T23:04:16.086941+0000","last_became_peered":"2024-04-26T23:04:16.086941+0000","last_unstale":"2024-04-26T23:04:18.093434+0000","last_undegraded":"2024-04-26T23:04:18.093434+0000","last_fullsized":"2024-04-26T23:04:18.093434+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:40:29.162642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087638+0000","last_change":"2024-04-26T23:04:17.109317+0000","last_active":"2024-04-26T23:04:18.087638+0000","last_peered":"2024-04-26T23:04:18.087638+0000","last_clean":"2024-04-26T23:04:18.087638+0000","last_became_active":"2024-04-26T23:04:17.109032+0000","last_became_peered":"2024-04-26T23:04:17.109032+0000","last_unstale":"2024-04-26T23:04:18.087638+0000","last_undegraded":"2024-04-26T23:04:18.087638+0000","last_fullsized":"2024-04-26T23:04:18.087638+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:05:24.474044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093188+0000","last_change":"2024-04-26T23:04:16.098205+0000","last_active":"2024-04-26T23:04:18.093188+0000","last_peered":"2024-04-26T23:04:18.093188+0000","last_clean":"2024-04-26T23:04:18.093188+0000","last_became_active":"2024-04-26T23:04:16.097872+0000","last_became_peered":"2024-04-26T23:04:16.097872+0000","last_unstale":"2024-04-26T23:04:18.093188+0000","last_undegraded":"2024-04-26T23:04:18.093188+0000","last_fullsized":"2024-04-26T23:04:18.093188+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:43:17.495520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.091514+0000","last_change":"2024-04-26T23:04:17.090023+0000","last_active":"2024-04-26T23:04:18.091514+0000","last_peered":"2024-04-26T23:04:18.091514+0000","last_clean":"2024-04-26T23:04:18.091514+0000","last_became_active":"2024-04-26T23:04:17.089182+0000","last_became_peered":"2024-04-26T23:04:17.089182+0000","last_unstale":"2024-04-26T23:04:18.091514+0000","last_undegraded":"2024-04-26T23:04:18.091514+0000","last_fullsized":"2024-04-26T23:04:18.091514+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:06.221700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087627+0000","last_change":"2024-04-26T23:04:16.097882+0000","last_active":"2024-04-26T23:04:18.087627+0000","last_peered":"2024-04-26T23:04:18.087627+0000","last_clean":"2024-04-26T23:04:18.087627+0000","last_became_active":"2024-04-26T23:04:16.093755+0000","last_became_peered":"2024-04-26T23:04:16.093755+0000","last_unstale":"2024-04-26T23:04:18.087627+0000","last_undegraded":"2024-04-26T23:04:18.087627+0000","last_fullsized":"2024-04-26T23:04:18.087627+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:08.895545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093398+0000","last_change":"2024-04-26T23:04:17.094180+0000","last_active":"2024-04-26T23:04:18.093398+0000","last_peered":"2024-04-26T23:04:18.093398+0000","last_clean":"2024-04-26T23:04:18.093398+0000","last_became_active":"2024-04-26T23:04:17.093889+0000","last_became_peered":"2024-04-26T23:04:17.093889+0000","last_unstale":"2024-04-26T23:04:18.093398+0000","last_undegraded":"2024-04-26T23:04:18.093398+0000","last_fullsized":"2024-04-26T23:04:18.093398+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:11:33.823168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.090450+0000","last_change":"2024-04-26T23:04:16.088890+0000","last_active":"2024-04-26T23:04:17.090450+0000","last_peered":"2024-04-26T23:04:17.090450+0000","last_clean":"2024-04-26T23:04:17.090450+0000","last_became_active":"2024-04-26T23:04:16.084413+0000","last_became_peered":"2024-04-26T23:04:16.084413+0000","last_unstale":"2024-04-26T23:04:17.090450+0000","last_undegraded":"2024-04-26T23:04:17.090450+0000","last_fullsized":"2024-04-26T23:04:17.090450+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:25.929664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093628+0000","last_change":"2024-04-26T23:04:17.089079+0000","last_active":"2024-04-26T23:04:18.093628+0000","last_peered":"2024-04-26T23:04:18.093628+0000","last_clean":"2024-04-26T23:04:18.093628+0000","last_became_active":"2024-04-26T23:04:17.088017+0000","last_became_peered":"2024-04-26T23:04:17.088017+0000","last_unstale":"2024-04-26T23:04:18.093628+0000","last_undegraded":"2024-04-26T23:04:18.093628+0000","last_fullsized":"2024-04-26T23:04:18.093628+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:55.287002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093398+0000","last_change":"2024-04-26T23:04:16.094914+0000","last_active":"2024-04-26T23:04:18.093398+0000","last_peered":"2024-04-26T23:04:18.093398+0000","last_clean":"2024-04-26T23:04:18.093398+0000","last_became_active":"2024-04-26T23:04:16.093879+0000","last_became_peered":"2024-04-26T23:04:16.093879+0000","last_unstale":"2024-04-26T23:04:18.093398+0000","last_undegraded":"2024-04-26T23:04:18.093398+0000","last_fullsized":"2024-04-26T23:04:18.093398+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:50:16.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.088235+0000","last_change":"2024-04-26T23:04:17.092499+0000","last_active":"2024-04-26T23:04:18.088235+0000","last_peered":"2024-04-26T23:04:18.088235+0000","last_clean":"2024-04-26T23:04:18.088235+0000","last_became_active":"2024-04-26T23:04:17.092196+0000","last_became_peered":"2024-04-26T23:04:17.092196+0000","last_unstale":"2024-04-26T23:04:18.088235+0000","last_undegraded":"2024-04-26T23:04:18.088235+0000","last_fullsized":"2024-04-26T23:04:18.088235+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:38:49.132739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.238678+0000","last_change":"2024-04-26T23:04:16.101090+0000","last_active":"2024-04-26T23:04:18.238678+0000","last_peered":"2024-04-26T23:04:18.238678+0000","last_clean":"2024-04-26T23:04:18.238678+0000","last_became_active":"2024-04-26T23:04:16.100161+0000","last_became_peered":"2024-04-26T23:04:16.100161+0000","last_unstale":"2024-04-26T23:04:18.238678+0000","last_undegraded":"2024-04-26T23:04:18.238678+0000","last_fullsized":"2024-04-26T23:04:18.238678+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:38:41.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.090427+0000","last_change":"2024-04-26T23:04:17.095748+0000","last_active":"2024-04-26T23:04:18.090427+0000","last_peered":"2024-04-26T23:04:18.090427+0000","last_clean":"2024-04-26T23:04:18.090427+0000","last_became_active":"2024-04-26T23:04:17.093979+0000","last_became_peered":"2024-04-26T23:04:17.093979+0000","last_unstale":"2024-04-26T23:04:18.090427+0000","last_undegraded":"2024-04-26T23:04:18.090427+0000","last_fullsized":"2024-04-26T23:04:18.090427+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:52:58.602974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093494+0000","last_change":"2024-04-26T23:04:16.085825+0000","last_active":"2024-04-26T23:04:18.093494+0000","last_peered":"2024-04-26T23:04:18.093494+0000","last_clean":"2024-04-26T23:04:18.093494+0000","last_became_active":"2024-04-26T23:04:16.085389+0000","last_became_peered":"2024-04-26T23:04:16.085389+0000","last_unstale":"2024-04-26T23:04:18.093494+0000","last_undegraded":"2024-04-26T23:04:18.093494+0000","last_fullsized":"2024-04-26T23:04:18.093494+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:08.895545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087312+0000","last_change":"2024-04-26T23:04:17.098246+0000","last_active":"2024-04-26T23:04:18.087312+0000","last_peered":"2024-04-26T23:04:18.087312+0000","last_clean":"2024-04-26T23:04:18.087312+0000","last_became_active":"2024-04-26T23:04:17.095967+0000","last_became_peered":"2024-04-26T23:04:17.095967+0000","last_unstale":"2024-04-26T23:04:18.087312+0000","last_undegraded":"2024-04-26T23:04:18.087312+0000","last_fullsized":"2024-04-26T23:04:18.087312+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:11:33.823168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101237+0000","last_change":"2024-04-26T23:04:16.101742+0000","last_active":"2024-04-26T23:04:18.101237+0000","last_peered":"2024-04-26T23:04:18.101237+0000","last_clean":"2024-04-26T23:04:18.101237+0000","last_became_active":"2024-04-26T23:04:16.101385+0000","last_became_peered":"2024-04-26T23:04:16.101385+0000","last_unstale":"2024-04-26T23:04:18.101237+0000","last_undegraded":"2024-04-26T23:04:18.101237+0000","last_fullsized":"2024-04-26T23:04:18.101237+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:50:16.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.105040+0000","last_change":"2024-04-26T23:04:17.089678+0000","last_active":"2024-04-26T23:04:17.105040+0000","last_peered":"2024-04-26T23:04:17.105040+0000","last_clean":"2024-04-26T23:04:17.105040+0000","last_became_active":"2024-04-26T23:04:17.089187+0000","last_became_peered":"2024-04-26T23:04:17.089187+0000","last_unstale":"2024-04-26T23:04:17.105040+0000","last_undegraded":"2024-04-26T23:04:17.105040+0000","last_fullsized":"2024-04-26T23:04:17.105040+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:06.221700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093568+0000","last_change":"2024-04-26T23:04:16.087592+0000","last_active":"2024-04-26T23:04:18.093568+0000","last_peered":"2024-04-26T23:04:18.093568+0000","last_clean":"2024-04-26T23:04:18.093568+0000","last_became_active":"2024-04-26T23:04:16.087278+0000","last_became_peered":"2024-04-26T23:04:16.087278+0000","last_unstale":"2024-04-26T23:04:18.093568+0000","last_undegraded":"2024-04-26T23:04:18.093568+0000","last_fullsized":"2024-04-26T23:04:18.093568+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:51:05.238308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093426+0000","last_change":"2024-04-26T23:04:17.095993+0000","last_active":"2024-04-26T23:04:18.093426+0000","last_peered":"2024-04-26T23:04:18.093426+0000","last_clean":"2024-04-26T23:04:18.093426+0000","last_became_active":"2024-04-26T23:04:17.092887+0000","last_became_peered":"2024-04-26T23:04:17.092887+0000","last_unstale":"2024-04-26T23:04:18.093426+0000","last_undegraded":"2024-04-26T23:04:18.093426+0000","last_fullsized":"2024-04-26T23:04:18.093426+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:34.886245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.092762+0000","last_change":"2024-04-26T23:04:16.091433+0000","last_active":"2024-04-26T23:04:17.092762+0000","last_peered":"2024-04-26T23:04:17.092762+0000","last_clean":"2024-04-26T23:04:17.092762+0000","last_became_active":"2024-04-26T23:04:16.089397+0000","last_became_peered":"2024-04-26T23:04:16.089397+0000","last_unstale":"2024-04-26T23:04:17.092762+0000","last_undegraded":"2024-04-26T23:04:17.092762+0000","last_fullsized":"2024-04-26T23:04:17.092762+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:43.776224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.091747+0000","last_change":"2024-04-26T23:04:17.096767+0000","last_active":"2024-04-26T23:04:18.091747+0000","last_peered":"2024-04-26T23:04:18.091747+0000","last_clean":"2024-04-26T23:04:18.091747+0000","last_became_active":"2024-04-26T23:04:17.094004+0000","last_became_peered":"2024-04-26T23:04:17.094004+0000","last_unstale":"2024-04-26T23:04:18.091747+0000","last_undegraded":"2024-04-26T23:04:18.091747+0000","last_fullsized":"2024-04-26T23:04:18.091747+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:33.055716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.090691+0000","last_change":"2024-04-26T23:04:16.091306+0000","last_active":"2024-04-26T23:04:17.090691+0000","last_peered":"2024-04-26T23:04:17.090691+0000","last_clean":"2024-04-26T23:04:17.090691+0000","last_became_active":"2024-04-26T23:04:16.091047+0000","last_became_peered":"2024-04-26T23:04:16.091047+0000","last_unstale":"2024-04-26T23:04:17.090691+0000","last_undegraded":"2024-04-26T23:04:17.090691+0000","last_fullsized":"2024-04-26T23:04:17.090691+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:35:14.696995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093833+0000","last_change":"2024-04-26T23:04:17.091332+0000","last_active":"2024-04-26T23:04:18.093833+0000","last_peered":"2024-04-26T23:04:18.093833+0000","last_clean":"2024-04-26T23:04:18.093833+0000","last_became_active":"2024-04-26T23:04:17.090638+0000","last_became_peered":"2024-04-26T23:04:17.090638+0000","last_unstale":"2024-04-26T23:04:18.093833+0000","last_undegraded":"2024-04-26T23:04:18.093833+0000","last_fullsized":"2024-04-26T23:04:18.093833+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:44:44.675774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.084291+0000","last_change":"2024-04-26T23:04:16.097479+0000","last_active":"2024-04-26T23:04:17.084291+0000","last_peered":"2024-04-26T23:04:17.084291+0000","last_clean":"2024-04-26T23:04:17.084291+0000","last_became_active":"2024-04-26T23:04:16.097258+0000","last_became_peered":"2024-04-26T23:04:16.097258+0000","last_unstale":"2024-04-26T23:04:17.084291+0000","last_undegraded":"2024-04-26T23:04:17.084291+0000","last_fullsized":"2024-04-26T23:04:17.084291+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:05:35.157689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.106900+0000","last_change":"2024-04-26T23:04:17.102835+0000","last_active":"2024-04-26T23:04:17.106900+0000","last_peered":"2024-04-26T23:04:17.106900+0000","last_clean":"2024-04-26T23:04:17.106900+0000","last_became_active":"2024-04-26T23:04:17.102491+0000","last_became_peered":"2024-04-26T23:04:17.102491+0000","last_unstale":"2024-04-26T23:04:17.106900+0000","last_undegraded":"2024-04-26T23:04:17.106900+0000","last_fullsized":"2024-04-26T23:04:17.106900+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:57.457992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.090688+0000","last_change":"2024-04-26T23:04:16.096428+0000","last_active":"2024-04-26T23:04:18.090688+0000","last_peered":"2024-04-26T23:04:18.090688+0000","last_clean":"2024-04-26T23:04:18.090688+0000","last_became_active":"2024-04-26T23:04:16.096138+0000","last_became_peered":"2024-04-26T23:04:16.096138+0000","last_unstale":"2024-04-26T23:04:18.090688+0000","last_undegraded":"2024-04-26T23:04:18.090688+0000","last_fullsized":"2024-04-26T23:04:18.090688+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:09.785131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092585+0000","last_change":"2024-04-26T23:04:17.085064+0000","last_active":"2024-04-26T23:04:18.092585+0000","last_peered":"2024-04-26T23:04:18.092585+0000","last_clean":"2024-04-26T23:04:18.092585+0000","last_became_active":"2024-04-26T23:04:17.084698+0000","last_became_peered":"2024-04-26T23:04:17.084698+0000","last_unstale":"2024-04-26T23:04:18.092585+0000","last_undegraded":"2024-04-26T23:04:18.092585+0000","last_fullsized":"2024-04-26T23:04:18.092585+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:09:51.056451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.097713+0000","last_change":"2024-04-26T23:04:16.095445+0000","last_active":"2024-04-26T23:04:18.097713+0000","last_peered":"2024-04-26T23:04:18.097713+0000","last_clean":"2024-04-26T23:04:18.097713+0000","last_became_active":"2024-04-26T23:04:16.095153+0000","last_became_peered":"2024-04-26T23:04:16.095153+0000","last_unstale":"2024-04-26T23:04:18.097713+0000","last_undegraded":"2024-04-26T23:04:18.097713+0000","last_fullsized":"2024-04-26T23:04:18.097713+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:09.785131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":232,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.099502+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:04:18.099502+0000","last_peered":"2024-04-26T23:04:18.099502+0000","last_clean":"2024-04-26T23:04:18.099502+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:04:18.099502+0000","last_undegraded":"2024-04-26T23:04:18.099502+0000","last_fullsized":"2024-04-26T23:04:18.099502+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1208320,"data_stored":862241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":272401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":606208,"data_stored":271749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":272331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":622592,"data_stored":273383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574893,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":618496,"data_stored":273361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":614400,"data_stored":271909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1220608,"data_stored":868865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:04:21.568 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:04:21.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.810+0000 7f6a75e00640 1 Processor -- start 2024-04-26T23:04:21.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.811+0000 7f6a75e00640 1 -- start start 2024-04-26T23:04:21.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.812+0000 7f6a75e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a7015d1e0 0x7f6a700d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:21.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.812+0000 7f6a75e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6a700d0f20 0x7f6a700d1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:21.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.812+0000 7f6a75e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a700d18c0 0x7f6a700cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:21.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.812+0000 7f6a75e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f6a700cede0 con 0x7f6a7015d1e0 2024-04-26T23:04:21.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.812+0000 7f6a75e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6a70070650 con 0x7f6a700d0f20 2024-04-26T23:04:21.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.812+0000 7f6a75e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6a700ce840 con 0x7f6a700d18c0 2024-04-26T23:04:21.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.813+0000 7f6a6ea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a7015d1e0 0x7f6a700d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:21.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.813+0000 7f6a6ea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a7015d1e0 0x7f6a700d09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42438/0 (socket says 172.21.15.40:42438) 2024-04-26T23:04:21.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.813+0000 7f6a6ea00640 1 -- 172.21.15.40:0/1891588826 learned_addr learned my addr 172.21.15.40:0/1891588826 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:21.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.813+0000 7f6a6e000640 1 --2- 172.21.15.40:0/1891588826 >> v2:172.21.15.103:3300/0 conn(0x7f6a700d0f20 0x7f6a700d1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:21.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.813+0000 7f6a6ea00640 1 -- 172.21.15.40:0/1891588826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a700d18c0 msgr2=0x7f6a700cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:21.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.813+0000 7f6a6ea00640 1 --2- 172.21.15.40:0/1891588826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a700d18c0 0x7f6a700cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.813+0000 7f6a6ea00640 1 -- 172.21.15.40:0/1891588826 >> v2:172.21.15.103:3300/0 conn(0x7f6a700d0f20 msgr2=0x7f6a700d1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:21.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.813+0000 7f6a6ea00640 1 --2- 172.21.15.40:0/1891588826 >> v2:172.21.15.103:3300/0 conn(0x7f6a700d0f20 0x7f6a700d1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.813+0000 7f6a6ea00640 1 -- 172.21.15.40:0/1891588826 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a700ceb00 con 0x7f6a7015d1e0 2024-04-26T23:04:21.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.814+0000 7f6a6ea00640 1 --2- 172.21.15.40:0/1891588826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a7015d1e0 0x7f6a700d09e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6a700c64b0 tx=0x7f6a6002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3271714c21c55b02 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.815+0000 7f6a6d600640 1 -- 172.21.15.40:0/1891588826 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60037070 con 0x7f6a7015d1e0 2024-04-26T23:04:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.815+0000 7f6a6d600640 1 -- 172.21.15.40:0/1891588826 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a60032440 con 0x7f6a7015d1e0 2024-04-26T23:04:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.815+0000 7f6a6d600640 1 -- 172.21.15.40:0/1891588826 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60032810 con 0x7f6a7015d1e0 2024-04-26T23:04:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.815+0000 7f6a75e00640 1 -- 172.21.15.40:0/1891588826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a7015d1e0 msgr2=0x7f6a700d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.815+0000 7f6a75e00640 1 --2- 172.21.15.40:0/1891588826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a7015d1e0 0x7f6a700d09e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6a700c64b0 tx=0x7f6a6002a1e0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.815+0000 7f6a75e00640 1 -- 172.21.15.40:0/1891588826 shutdown_connections 2024-04-26T23:04:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.815+0000 7f6a75e00640 1 --2- 172.21.15.40:0/1891588826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a700d18c0 0x7f6a700cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.815+0000 7f6a75e00640 1 --2- 172.21.15.40:0/1891588826 >> v2:172.21.15.103:3300/0 conn(0x7f6a700d0f20 0x7f6a700d1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.815+0000 7f6a75e00640 1 --2- 172.21.15.40:0/1891588826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a7015d1e0 0x7f6a700d09e0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.815+0000 7f6a75e00640 1 -- 172.21.15.40:0/1891588826 >> 172.21.15.40:0/1891588826 conn(0x7f6a700d2da0 msgr2=0x7f6a700d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.815+0000 7f6a75e00640 1 -- 172.21.15.40:0/1891588826 shutdown_connections 2024-04-26T23:04:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.815+0000 7f6a75e00640 1 -- 172.21.15.40:0/1891588826 wait complete. 2024-04-26T23:04:21.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.817+0000 7f6a75e00640 1 Processor -- start 2024-04-26T23:04:21.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.817+0000 7f6a75e00640 1 -- start start 2024-04-26T23:04:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.818+0000 7f6a75e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a700d0f20 0x7f6a70154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.818+0000 7f6a75e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a700d18c0 0x7f6a700b1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:21.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.818+0000 7f6a75e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a7015d1e0 0x7f6a700b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:21.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.818+0000 7f6a75e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6a700cfb20 con 0x7f6a7015d1e0 2024-04-26T23:04:21.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.818+0000 7f6a75e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a700ce4c0 con 0x7f6a700d18c0 2024-04-26T23:04:21.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.818+0000 7f6a75e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6a700cfca0 con 0x7f6a700d0f20 2024-04-26T23:04:21.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.818+0000 7f6a6f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a7015d1e0 0x7f6a700b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:21.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.819+0000 7f6a6f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a7015d1e0 0x7f6a700b1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42452/0 (socket says 172.21.15.40:42452) 2024-04-26T23:04:21.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.819+0000 7f6a6f400640 1 -- 172.21.15.40:0/3761919199 learned_addr learned my addr 172.21.15.40:0/3761919199 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.819+0000 7f6a6ea00640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a700d0f20 0x7f6a70154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.819+0000 7f6a6e000640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a700d18c0 0x7f6a700b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.819+0000 7f6a6f400640 1 -- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a700d0f20 msgr2=0x7f6a70154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.819+0000 7f6a6f400640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a700d0f20 0x7f6a70154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.819+0000 7f6a6f400640 1 -- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a700d18c0 msgr2=0x7f6a700b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.819+0000 7f6a6f400640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a700d18c0 0x7f6a700b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.819+0000 7f6a6f400640 1 -- 172.21.15.40:0/3761919199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a7015ac60 con 0x7f6a7015d1e0 2024-04-26T23:04:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.819+0000 7f6a6ea00640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a700d0f20 0x7f6a70154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:21.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.819+0000 7f6a6e000640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a700d18c0 0x7f6a700b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:21.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.819+0000 7f6a6f400640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a7015d1e0 0x7f6a700b1b90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009fd0 tx=0x7f6a6800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:21.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.820+0000 7f6a67400640 1 -- 172.21.15.40:0/3761919199 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68011830 con 0x7f6a7015d1e0 2024-04-26T23:04:21.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.820+0000 7f6a67400640 1 -- 172.21.15.40:0/3761919199 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a68016070 con 0x7f6a7015d1e0 2024-04-26T23:04:21.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.820+0000 7f6a67400640 1 -- 172.21.15.40:0/3761919199 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68011bf0 con 0x7f6a7015d1e0 2024-04-26T23:04:21.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.820+0000 7f6a75e00640 1 -- 172.21.15.40:0/3761919199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a700cbe00 con 0x7f6a7015d1e0 2024-04-26T23:04:21.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.820+0000 7f6a75e00640 1 -- 172.21.15.40:0/3761919199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a700cc3c0 con 0x7f6a7015d1e0 2024-04-26T23:04:21.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.821+0000 7f6a75e00640 1 -- 172.21.15.40:0/3761919199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a700694f0 con 0x7f6a7015d1e0 2024-04-26T23:04:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.822+0000 7f6a67400640 1 -- 172.21.15.40:0/3761919199 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6a6801b460 con 0x7f6a7015d1e0 2024-04-26T23:04:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.823+0000 7f6a67400640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6a4c06c8f0 0x7f6a4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.823+0000 7f6a67400640 1 -- 172.21.15.40:0/3761919199 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f6a6808ba50 con 0x7f6a7015d1e0 2024-04-26T23:04:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.824+0000 7f6a6ea00640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6a4c06c8f0 0x7f6a4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:21.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.825+0000 7f6a6ea00640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6a4c06c8f0 0x7f6a4c06edb0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6a60034000 tx=0x7f6a6002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:21.827+0000 7f6a67400640 1 -- 172.21.15.40:0/3761919199 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f6a7015ac60 con 0x7f6a7015d1e0 2024-04-26T23:04:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.029+0000 7f6a75e00640 1 -- 172.21.15.40:0/3761919199 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6a700cbbb0 con 0x7f6a4c06c8f0 2024-04-26T23:04:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.048+0000 7f6a67400640 1 -- 172.21.15.40:0/3761919199 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251939 (secure 0 0 0) 0x7f6a700cbbb0 con 0x7f6a4c06c8f0 2024-04-26T23:04:22.052 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:04:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.054+0000 7f6a75e00640 1 -- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6a4c06c8f0 msgr2=0x7f6a4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.054+0000 7f6a75e00640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6a4c06c8f0 0x7f6a4c06edb0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6a60034000 tx=0x7f6a6002e000 comp rx=0 tx=0).stop 2024-04-26T23:04:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.054+0000 7f6a75e00640 1 -- 172.21.15.40:0/3761919199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a7015d1e0 msgr2=0x7f6a700b1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.054+0000 7f6a75e00640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a7015d1e0 0x7f6a700b1b90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009fd0 tx=0x7f6a6800b8e0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.055+0000 7f6a75e00640 1 -- 172.21.15.40:0/3761919199 shutdown_connections 2024-04-26T23:04:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.055+0000 7f6a75e00640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6a4c06c8f0 0x7f6a4c06edb0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.055+0000 7f6a75e00640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a7015d1e0 0x7f6a700b1b90 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.055+0000 7f6a75e00640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a700d18c0 0x7f6a700b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.055+0000 7f6a75e00640 1 --2- 172.21.15.40:0/3761919199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6a700d0f20 0x7f6a70154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.055+0000 7f6a75e00640 1 -- 172.21.15.40:0/3761919199 >> 172.21.15.40:0/3761919199 conn(0x7f6a700d2da0 msgr2=0x7f6a700c8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.055+0000 7f6a75e00640 1 -- 172.21.15.40:0/3761919199 shutdown_connections 2024-04-26T23:04:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.055+0000 7f6a75e00640 1 -- 172.21.15.40:0/3761919199 wait complete. 2024-04-26T23:04:22.077 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-04-26T23:04:21.215706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":230492,"kb_used_data":5972,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754744228,"statfs":{"total":773094113280,"available":772858089472,"internally_reserved":0,"allocated":6115328,"data_stored":3366240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":229756466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002164"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087916+0000","last_change":"2024-04-26T23:04:17.108756+0000","last_active":"2024-04-26T23:04:18.087916+0000","last_peered":"2024-04-26T23:04:18.087916+0000","last_clean":"2024-04-26T23:04:18.087916+0000","last_became_active":"2024-04-26T23:04:17.108423+0000","last_became_peered":"2024-04-26T23:04:17.108423+0000","last_unstale":"2024-04-26T23:04:18.087916+0000","last_undegraded":"2024-04-26T23:04:18.087916+0000","last_fullsized":"2024-04-26T23:04:18.087916+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:52:58.602974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101034+0000","last_change":"2024-04-26T23:04:17.086315+0000","last_active":"2024-04-26T23:04:18.101034+0000","last_peered":"2024-04-26T23:04:18.101034+0000","last_clean":"2024-04-26T23:04:18.101034+0000","last_became_active":"2024-04-26T23:04:17.086039+0000","last_became_peered":"2024-04-26T23:04:17.086039+0000","last_unstale":"2024-04-26T23:04:18.101034+0000","last_undegraded":"2024-04-26T23:04:18.101034+0000","last_fullsized":"2024-04-26T23:04:18.101034+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:38:49.132739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093341+0000","last_change":"2024-04-26T23:04:17.097344+0000","last_active":"2024-04-26T23:04:18.093341+0000","last_peered":"2024-04-26T23:04:18.093341+0000","last_clean":"2024-04-26T23:04:18.093341+0000","last_became_active":"2024-04-26T23:04:17.091514+0000","last_became_peered":"2024-04-26T23:04:17.091514+0000","last_unstale":"2024-04-26T23:04:18.093341+0000","last_undegraded":"2024-04-26T23:04:18.093341+0000","last_fullsized":"2024-04-26T23:04:18.093341+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:05.987818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101168+0000","last_change":"2024-04-26T23:04:17.094548+0000","last_active":"2024-04-26T23:04:18.101168+0000","last_peered":"2024-04-26T23:04:18.101168+0000","last_clean":"2024-04-26T23:04:18.101168+0000","last_became_active":"2024-04-26T23:04:17.094062+0000","last_became_peered":"2024-04-26T23:04:17.094062+0000","last_unstale":"2024-04-26T23:04:18.101168+0000","last_undegraded":"2024-04-26T23:04:18.101168+0000","last_fullsized":"2024-04-26T23:04:18.101168+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:06.221700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.098205+0000","last_change":"2024-04-26T23:04:17.090776+0000","last_active":"2024-04-26T23:04:17.098205+0000","last_peered":"2024-04-26T23:04:17.098205+0000","last_clean":"2024-04-26T23:04:17.098205+0000","last_became_active":"2024-04-26T23:04:17.089579+0000","last_became_peered":"2024-04-26T23:04:17.089579+0000","last_unstale":"2024-04-26T23:04:17.098205+0000","last_undegraded":"2024-04-26T23:04:17.098205+0000","last_fullsized":"2024-04-26T23:04:17.098205+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:49:36.443734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087287+0000","last_change":"2024-04-26T23:04:17.093394+0000","last_active":"2024-04-26T23:04:18.087287+0000","last_peered":"2024-04-26T23:04:18.087287+0000","last_clean":"2024-04-26T23:04:18.087287+0000","last_became_active":"2024-04-26T23:04:17.093123+0000","last_became_peered":"2024-04-26T23:04:17.093123+0000","last_unstale":"2024-04-26T23:04:18.087287+0000","last_undegraded":"2024-04-26T23:04:18.087287+0000","last_fullsized":"2024-04-26T23:04:18.087287+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:59:44.819074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093073+0000","last_change":"2024-04-26T23:04:17.089724+0000","last_active":"2024-04-26T23:04:18.093073+0000","last_peered":"2024-04-26T23:04:18.093073+0000","last_clean":"2024-04-26T23:04:18.093073+0000","last_became_active":"2024-04-26T23:04:17.089424+0000","last_became_peered":"2024-04-26T23:04:17.089424+0000","last_unstale":"2024-04-26T23:04:18.093073+0000","last_undegraded":"2024-04-26T23:04:18.093073+0000","last_fullsized":"2024-04-26T23:04:18.093073+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:44.792411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.111093+0000","last_change":"2024-04-26T23:04:17.108239+0000","last_active":"2024-04-26T23:04:17.111093+0000","last_peered":"2024-04-26T23:04:17.111093+0000","last_clean":"2024-04-26T23:04:17.111093+0000","last_became_active":"2024-04-26T23:04:17.107492+0000","last_became_peered":"2024-04-26T23:04:17.107492+0000","last_unstale":"2024-04-26T23:04:17.111093+0000","last_undegraded":"2024-04-26T23:04:17.111093+0000","last_fullsized":"2024-04-26T23:04:17.111093+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:14:26.705521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.088053+0000","last_change":"2024-04-26T23:04:17.085761+0000","last_active":"2024-04-26T23:04:17.088053+0000","last_peered":"2024-04-26T23:04:17.088053+0000","last_clean":"2024-04-26T23:04:17.088053+0000","last_became_active":"2024-04-26T23:04:17.085411+0000","last_became_peered":"2024-04-26T23:04:17.085411+0000","last_unstale":"2024-04-26T23:04:17.088053+0000","last_undegraded":"2024-04-26T23:04:17.088053+0000","last_fullsized":"2024-04-26T23:04:17.088053+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:38:49.132739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087874+0000","last_change":"2024-04-26T23:04:17.107215+0000","last_active":"2024-04-26T23:04:18.087874+0000","last_peered":"2024-04-26T23:04:18.087874+0000","last_clean":"2024-04-26T23:04:18.087874+0000","last_became_active":"2024-04-26T23:04:17.106918+0000","last_became_peered":"2024-04-26T23:04:17.106918+0000","last_unstale":"2024-04-26T23:04:18.087874+0000","last_undegraded":"2024-04-26T23:04:18.087874+0000","last_fullsized":"2024-04-26T23:04:18.087874+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:25.066755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.238045+0000","last_change":"2024-04-26T23:04:17.089834+0000","last_active":"2024-04-26T23:04:18.238045+0000","last_peered":"2024-04-26T23:04:18.238045+0000","last_clean":"2024-04-26T23:04:18.238045+0000","last_became_active":"2024-04-26T23:04:17.089565+0000","last_became_peered":"2024-04-26T23:04:17.089565+0000","last_unstale":"2024-04-26T23:04:18.238045+0000","last_undegraded":"2024-04-26T23:04:18.238045+0000","last_fullsized":"2024-04-26T23:04:18.238045+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:12:21.424428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.101623+0000","last_change":"2024-04-26T23:04:17.091262+0000","last_active":"2024-04-26T23:04:17.101623+0000","last_peered":"2024-04-26T23:04:17.101623+0000","last_clean":"2024-04-26T23:04:17.101623+0000","last_became_active":"2024-04-26T23:04:17.090829+0000","last_became_peered":"2024-04-26T23:04:17.090829+0000","last_unstale":"2024-04-26T23:04:17.101623+0000","last_undegraded":"2024-04-26T23:04:17.101623+0000","last_fullsized":"2024-04-26T23:04:17.101623+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:05:41.081684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087374+0000","last_change":"2024-04-26T23:04:17.107886+0000","last_active":"2024-04-26T23:04:18.087374+0000","last_peered":"2024-04-26T23:04:18.087374+0000","last_clean":"2024-04-26T23:04:18.087374+0000","last_became_active":"2024-04-26T23:04:17.106514+0000","last_became_peered":"2024-04-26T23:04:17.106514+0000","last_unstale":"2024-04-26T23:04:18.087374+0000","last_undegraded":"2024-04-26T23:04:18.087374+0000","last_fullsized":"2024-04-26T23:04:18.087374+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:18:31.829572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093330+0000","last_change":"2024-04-26T23:04:17.095553+0000","last_active":"2024-04-26T23:04:18.093330+0000","last_peered":"2024-04-26T23:04:18.093330+0000","last_clean":"2024-04-26T23:04:18.093330+0000","last_became_active":"2024-04-26T23:04:17.091689+0000","last_became_peered":"2024-04-26T23:04:17.091689+0000","last_unstale":"2024-04-26T23:04:18.093330+0000","last_undegraded":"2024-04-26T23:04:18.093330+0000","last_fullsized":"2024-04-26T23:04:18.093330+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:52:37.327132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.099835+0000","last_change":"2024-04-26T23:04:17.087714+0000","last_active":"2024-04-26T23:04:18.099835+0000","last_peered":"2024-04-26T23:04:18.099835+0000","last_clean":"2024-04-26T23:04:18.099835+0000","last_became_active":"2024-04-26T23:04:17.087424+0000","last_became_peered":"2024-04-26T23:04:17.087424+0000","last_unstale":"2024-04-26T23:04:18.099835+0000","last_undegraded":"2024-04-26T23:04:18.099835+0000","last_fullsized":"2024-04-26T23:04:18.099835+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:05:24.474044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087000+0000","last_change":"2024-04-26T23:04:17.096307+0000","last_active":"2024-04-26T23:04:18.087000+0000","last_peered":"2024-04-26T23:04:18.087000+0000","last_clean":"2024-04-26T23:04:18.087000+0000","last_became_active":"2024-04-26T23:04:17.096038+0000","last_became_peered":"2024-04-26T23:04:17.096038+0000","last_unstale":"2024-04-26T23:04:18.087000+0000","last_undegraded":"2024-04-26T23:04:18.087000+0000","last_fullsized":"2024-04-26T23:04:18.087000+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:38:25.040887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.094190+0000","last_change":"2024-04-26T23:04:17.090691+0000","last_active":"2024-04-26T23:04:18.094190+0000","last_peered":"2024-04-26T23:04:18.094190+0000","last_clean":"2024-04-26T23:04:18.094190+0000","last_became_active":"2024-04-26T23:04:17.083165+0000","last_became_peered":"2024-04-26T23:04:17.083165+0000","last_unstale":"2024-04-26T23:04:18.094190+0000","last_undegraded":"2024-04-26T23:04:18.094190+0000","last_fullsized":"2024-04-26T23:04:18.094190+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:26:49.475479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101349+0000","last_change":"2024-04-26T23:04:17.086481+0000","last_active":"2024-04-26T23:04:18.101349+0000","last_peered":"2024-04-26T23:04:18.101349+0000","last_clean":"2024-04-26T23:04:18.101349+0000","last_became_active":"2024-04-26T23:04:17.086196+0000","last_became_peered":"2024-04-26T23:04:17.086196+0000","last_unstale":"2024-04-26T23:04:18.101349+0000","last_undegraded":"2024-04-26T23:04:18.101349+0000","last_fullsized":"2024-04-26T23:04:18.101349+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:33.055716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093514+0000","last_change":"2024-04-26T23:04:17.097764+0000","last_active":"2024-04-26T23:04:18.093514+0000","last_peered":"2024-04-26T23:04:18.093514+0000","last_clean":"2024-04-26T23:04:18.093514+0000","last_became_active":"2024-04-26T23:04:17.097476+0000","last_became_peered":"2024-04-26T23:04:17.097476+0000","last_unstale":"2024-04-26T23:04:18.093514+0000","last_undegraded":"2024-04-26T23:04:18.093514+0000","last_fullsized":"2024-04-26T23:04:18.093514+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:57.457992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092519+0000","last_change":"2024-04-26T23:04:17.087958+0000","last_active":"2024-04-26T23:04:18.092519+0000","last_peered":"2024-04-26T23:04:18.092519+0000","last_clean":"2024-04-26T23:04:18.092519+0000","last_became_active":"2024-04-26T23:04:17.087650+0000","last_became_peered":"2024-04-26T23:04:17.087650+0000","last_unstale":"2024-04-26T23:04:18.092519+0000","last_undegraded":"2024-04-26T23:04:18.092519+0000","last_fullsized":"2024-04-26T23:04:18.092519+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:51:06.254495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093223+0000","last_change":"2024-04-26T23:04:17.089714+0000","last_active":"2024-04-26T23:04:18.093223+0000","last_peered":"2024-04-26T23:04:18.093223+0000","last_clean":"2024-04-26T23:04:18.093223+0000","last_became_active":"2024-04-26T23:04:17.088110+0000","last_became_peered":"2024-04-26T23:04:17.088110+0000","last_unstale":"2024-04-26T23:04:18.093223+0000","last_undegraded":"2024-04-26T23:04:18.093223+0000","last_fullsized":"2024-04-26T23:04:18.093223+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:59:44.819074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.103518+0000","last_change":"2024-04-26T23:04:17.097312+0000","last_active":"2024-04-26T23:04:17.103518+0000","last_peered":"2024-04-26T23:04:17.103518+0000","last_clean":"2024-04-26T23:04:17.103518+0000","last_became_active":"2024-04-26T23:04:17.097067+0000","last_became_peered":"2024-04-26T23:04:17.097067+0000","last_unstale":"2024-04-26T23:04:17.103518+0000","last_undegraded":"2024-04-26T23:04:17.103518+0000","last_fullsized":"2024-04-26T23:04:17.103518+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:52:37.327132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.238084+0000","last_change":"2024-04-26T23:04:17.087678+0000","last_active":"2024-04-26T23:04:18.238084+0000","last_peered":"2024-04-26T23:04:18.238084+0000","last_clean":"2024-04-26T23:04:18.238084+0000","last_became_active":"2024-04-26T23:04:17.087399+0000","last_became_peered":"2024-04-26T23:04:17.087399+0000","last_unstale":"2024-04-26T23:04:18.238084+0000","last_undegraded":"2024-04-26T23:04:18.238084+0000","last_fullsized":"2024-04-26T23:04:18.238084+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:59.965974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092918+0000","last_change":"2024-04-26T23:04:17.082519+0000","last_active":"2024-04-26T23:04:18.092918+0000","last_peered":"2024-04-26T23:04:18.092918+0000","last_clean":"2024-04-26T23:04:18.092918+0000","last_became_active":"2024-04-26T23:04:17.081193+0000","last_became_peered":"2024-04-26T23:04:17.081193+0000","last_unstale":"2024-04-26T23:04:18.092918+0000","last_undegraded":"2024-04-26T23:04:18.092918+0000","last_fullsized":"2024-04-26T23:04:18.092918+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:05:41.081684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.094024+0000","last_change":"2024-04-26T23:04:17.096363+0000","last_active":"2024-04-26T23:04:18.094024+0000","last_peered":"2024-04-26T23:04:18.094024+0000","last_clean":"2024-04-26T23:04:18.094024+0000","last_became_active":"2024-04-26T23:04:17.091540+0000","last_became_peered":"2024-04-26T23:04:17.091540+0000","last_unstale":"2024-04-26T23:04:18.094024+0000","last_undegraded":"2024-04-26T23:04:18.094024+0000","last_fullsized":"2024-04-26T23:04:18.094024+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:18:50.045304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.107069+0000","last_change":"2024-04-26T23:04:17.102799+0000","last_active":"2024-04-26T23:04:17.107069+0000","last_peered":"2024-04-26T23:04:17.107069+0000","last_clean":"2024-04-26T23:04:17.107069+0000","last_became_active":"2024-04-26T23:04:17.102433+0000","last_became_peered":"2024-04-26T23:04:17.102433+0000","last_unstale":"2024-04-26T23:04:17.107069+0000","last_undegraded":"2024-04-26T23:04:17.107069+0000","last_fullsized":"2024-04-26T23:04:17.107069+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:34.886245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087996+0000","last_change":"2024-04-26T23:04:17.108342+0000","last_active":"2024-04-26T23:04:18.087996+0000","last_peered":"2024-04-26T23:04:18.087996+0000","last_clean":"2024-04-26T23:04:18.087996+0000","last_became_active":"2024-04-26T23:04:17.107942+0000","last_became_peered":"2024-04-26T23:04:17.107942+0000","last_unstale":"2024-04-26T23:04:18.087996+0000","last_undegraded":"2024-04-26T23:04:18.087996+0000","last_fullsized":"2024-04-26T23:04:18.087996+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:00.840289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092938+0000","last_change":"2024-04-26T23:04:17.081008+0000","last_active":"2024-04-26T23:04:18.092938+0000","last_peered":"2024-04-26T23:04:18.092938+0000","last_clean":"2024-04-26T23:04:18.092938+0000","last_became_active":"2024-04-26T23:04:17.080710+0000","last_became_peered":"2024-04-26T23:04:17.080710+0000","last_unstale":"2024-04-26T23:04:18.092938+0000","last_undegraded":"2024-04-26T23:04:18.092938+0000","last_fullsized":"2024-04-26T23:04:18.092938+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:38:25.040887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092867+0000","last_change":"2024-04-26T23:04:17.087561+0000","last_active":"2024-04-26T23:04:18.092867+0000","last_peered":"2024-04-26T23:04:18.092867+0000","last_clean":"2024-04-26T23:04:18.092867+0000","last_became_active":"2024-04-26T23:04:17.087254+0000","last_became_peered":"2024-04-26T23:04:17.087254+0000","last_unstale":"2024-04-26T23:04:18.092867+0000","last_undegraded":"2024-04-26T23:04:18.092867+0000","last_fullsized":"2024-04-26T23:04:18.092867+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:35:15.713183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.100313+0000","last_change":"2024-04-26T23:04:17.107516+0000","last_active":"2024-04-26T23:04:18.100313+0000","last_peered":"2024-04-26T23:04:18.100313+0000","last_clean":"2024-04-26T23:04:18.100313+0000","last_became_active":"2024-04-26T23:04:17.107267+0000","last_became_peered":"2024-04-26T23:04:17.107267+0000","last_unstale":"2024-04-26T23:04:18.100313+0000","last_undegraded":"2024-04-26T23:04:18.100313+0000","last_fullsized":"2024-04-26T23:04:18.100313+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:52:58.602974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093205+0000","last_change":"2024-04-26T23:04:17.083089+0000","last_active":"2024-04-26T23:04:18.093205+0000","last_peered":"2024-04-26T23:04:18.093205+0000","last_clean":"2024-04-26T23:04:18.093205+0000","last_became_active":"2024-04-26T23:04:17.081306+0000","last_became_peered":"2024-04-26T23:04:17.081306+0000","last_unstale":"2024-04-26T23:04:18.093205+0000","last_undegraded":"2024-04-26T23:04:18.093205+0000","last_fullsized":"2024-04-26T23:04:18.093205+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:41.757362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.090486+0000","last_change":"2024-04-26T23:04:17.095460+0000","last_active":"2024-04-26T23:04:18.090486+0000","last_peered":"2024-04-26T23:04:18.090486+0000","last_clean":"2024-04-26T23:04:18.090486+0000","last_became_active":"2024-04-26T23:04:17.094048+0000","last_became_peered":"2024-04-26T23:04:17.094048+0000","last_unstale":"2024-04-26T23:04:18.090486+0000","last_undegraded":"2024-04-26T23:04:18.090486+0000","last_fullsized":"2024-04-26T23:04:18.090486+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:00.840289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.100205+0000","last_change":"2024-04-26T23:04:17.086930+0000","last_active":"2024-04-26T23:04:18.100205+0000","last_peered":"2024-04-26T23:04:18.100205+0000","last_clean":"2024-04-26T23:04:18.100205+0000","last_became_active":"2024-04-26T23:04:17.085984+0000","last_became_peered":"2024-04-26T23:04:17.085984+0000","last_unstale":"2024-04-26T23:04:18.100205+0000","last_undegraded":"2024-04-26T23:04:18.100205+0000","last_fullsized":"2024-04-26T23:04:18.100205+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:25.066755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":18,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093485+0000","last_change":"2024-04-26T23:04:16.089299+0000","last_active":"2024-04-26T23:04:18.093485+0000","last_peered":"2024-04-26T23:04:18.093485+0000","last_clean":"2024-04-26T23:04:18.093485+0000","last_became_active":"2024-04-26T23:04:16.088966+0000","last_became_peered":"2024-04-26T23:04:16.088966+0000","last_unstale":"2024-04-26T23:04:18.093485+0000","last_undegraded":"2024-04-26T23:04:18.093485+0000","last_fullsized":"2024-04-26T23:04:18.093485+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:25:44.106018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.091140+0000","last_change":"2024-04-26T23:04:17.088848+0000","last_active":"2024-04-26T23:04:18.091140+0000","last_peered":"2024-04-26T23:04:18.091140+0000","last_clean":"2024-04-26T23:04:18.091140+0000","last_became_active":"2024-04-26T23:04:17.087714+0000","last_became_peered":"2024-04-26T23:04:17.087714+0000","last_unstale":"2024-04-26T23:04:18.091140+0000","last_undegraded":"2024-04-26T23:04:18.091140+0000","last_fullsized":"2024-04-26T23:04:18.091140+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:02.222941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087806+0000","last_change":"2024-04-26T23:04:16.099694+0000","last_active":"2024-04-26T23:04:18.087806+0000","last_peered":"2024-04-26T23:04:18.087806+0000","last_clean":"2024-04-26T23:04:18.087806+0000","last_became_active":"2024-04-26T23:04:16.099200+0000","last_became_peered":"2024-04-26T23:04:16.099200+0000","last_unstale":"2024-04-26T23:04:18.087806+0000","last_undegraded":"2024-04-26T23:04:18.087806+0000","last_fullsized":"2024-04-26T23:04:18.087806+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:11.272237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092958+0000","last_change":"2024-04-26T23:04:17.091407+0000","last_active":"2024-04-26T23:04:18.092958+0000","last_peered":"2024-04-26T23:04:18.092958+0000","last_clean":"2024-04-26T23:04:18.092958+0000","last_became_active":"2024-04-26T23:04:17.088002+0000","last_became_peered":"2024-04-26T23:04:17.088002+0000","last_unstale":"2024-04-26T23:04:18.092958+0000","last_undegraded":"2024-04-26T23:04:18.092958+0000","last_fullsized":"2024-04-26T23:04:18.092958+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:57:35.423782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.088153+0000","last_change":"2024-04-26T23:04:16.091143+0000","last_active":"2024-04-26T23:04:18.088153+0000","last_peered":"2024-04-26T23:04:18.088153+0000","last_clean":"2024-04-26T23:04:18.088153+0000","last_became_active":"2024-04-26T23:04:16.090657+0000","last_became_peered":"2024-04-26T23:04:16.090657+0000","last_unstale":"2024-04-26T23:04:18.088153+0000","last_undegraded":"2024-04-26T23:04:18.088153+0000","last_fullsized":"2024-04-26T23:04:18.088153+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:25:44.106018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.105066+0000","last_change":"2024-04-26T23:04:17.092804+0000","last_active":"2024-04-26T23:04:17.105066+0000","last_peered":"2024-04-26T23:04:17.105066+0000","last_clean":"2024-04-26T23:04:17.105066+0000","last_became_active":"2024-04-26T23:04:17.092393+0000","last_became_peered":"2024-04-26T23:04:17.092393+0000","last_unstale":"2024-04-26T23:04:17.105066+0000","last_undegraded":"2024-04-26T23:04:17.105066+0000","last_fullsized":"2024-04-26T23:04:17.105066+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:41.757362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.091583+0000","last_change":"2024-04-26T23:04:16.099288+0000","last_active":"2024-04-26T23:04:18.091583+0000","last_peered":"2024-04-26T23:04:18.091583+0000","last_clean":"2024-04-26T23:04:18.091583+0000","last_became_active":"2024-04-26T23:04:16.099003+0000","last_became_peered":"2024-04-26T23:04:16.099003+0000","last_unstale":"2024-04-26T23:04:18.091583+0000","last_undegraded":"2024-04-26T23:04:18.091583+0000","last_fullsized":"2024-04-26T23:04:18.091583+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:51:05.238308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.104652+0000","last_change":"2024-04-26T23:04:17.103015+0000","last_active":"2024-04-26T23:04:17.104652+0000","last_peered":"2024-04-26T23:04:17.104652+0000","last_clean":"2024-04-26T23:04:17.104652+0000","last_became_active":"2024-04-26T23:04:17.102694+0000","last_became_peered":"2024-04-26T23:04:17.102694+0000","last_unstale":"2024-04-26T23:04:17.104652+0000","last_undegraded":"2024-04-26T23:04:17.104652+0000","last_fullsized":"2024-04-26T23:04:17.104652+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:10.801318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092994+0000","last_change":"2024-04-26T23:04:16.093076+0000","last_active":"2024-04-26T23:04:18.092994+0000","last_peered":"2024-04-26T23:04:18.092994+0000","last_clean":"2024-04-26T23:04:18.092994+0000","last_became_active":"2024-04-26T23:04:16.092763+0000","last_became_peered":"2024-04-26T23:04:16.092763+0000","last_unstale":"2024-04-26T23:04:18.092994+0000","last_undegraded":"2024-04-26T23:04:18.092994+0000","last_fullsized":"2024-04-26T23:04:18.092994+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:49:35.427546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.100802+0000","last_change":"2024-04-26T23:04:17.110748+0000","last_active":"2024-04-26T23:04:18.100802+0000","last_peered":"2024-04-26T23:04:18.100802+0000","last_clean":"2024-04-26T23:04:18.100802+0000","last_became_active":"2024-04-26T23:04:17.110535+0000","last_became_peered":"2024-04-26T23:04:17.110535+0000","last_unstale":"2024-04-26T23:04:18.100802+0000","last_undegraded":"2024-04-26T23:04:18.100802+0000","last_fullsized":"2024-04-26T23:04:18.100802+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:00.840289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":13,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.114619+0000","last_change":"2024-04-26T23:04:16.098768+0000","last_active":"2024-04-26T23:04:17.114619+0000","last_peered":"2024-04-26T23:04:17.114619+0000","last_clean":"2024-04-26T23:04:17.114619+0000","last_became_active":"2024-04-26T23:04:16.098447+0000","last_became_peered":"2024-04-26T23:04:16.098447+0000","last_unstale":"2024-04-26T23:04:17.114619+0000","last_undegraded":"2024-04-26T23:04:17.114619+0000","last_fullsized":"2024-04-26T23:04:17.114619+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:35.744533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.101019+0000","last_change":"2024-04-26T23:04:17.089918+0000","last_active":"2024-04-26T23:04:17.101019+0000","last_peered":"2024-04-26T23:04:17.101019+0000","last_clean":"2024-04-26T23:04:17.101019+0000","last_became_active":"2024-04-26T23:04:17.089657+0000","last_became_peered":"2024-04-26T23:04:17.089657+0000","last_unstale":"2024-04-26T23:04:17.101019+0000","last_undegraded":"2024-04-26T23:04:17.101019+0000","last_fullsized":"2024-04-26T23:04:17.101019+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:33.055716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092554+0000","last_change":"2024-04-26T23:04:16.094715+0000","last_active":"2024-04-26T23:04:18.092554+0000","last_peered":"2024-04-26T23:04:18.092554+0000","last_clean":"2024-04-26T23:04:18.092554+0000","last_became_active":"2024-04-26T23:04:16.094400+0000","last_became_peered":"2024-04-26T23:04:16.094400+0000","last_unstale":"2024-04-26T23:04:18.092554+0000","last_undegraded":"2024-04-26T23:04:18.092554+0000","last_fullsized":"2024-04-26T23:04:18.092554+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:11.272237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101060+0000","last_change":"2024-04-26T23:04:17.082328+0000","last_active":"2024-04-26T23:04:18.101060+0000","last_peered":"2024-04-26T23:04:18.101060+0000","last_clean":"2024-04-26T23:04:18.101060+0000","last_became_active":"2024-04-26T23:04:17.082039+0000","last_became_peered":"2024-04-26T23:04:17.082039+0000","last_unstale":"2024-04-26T23:04:18.101060+0000","last_undegraded":"2024-04-26T23:04:18.101060+0000","last_fullsized":"2024-04-26T23:04:18.101060+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:02.222941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092660+0000","last_change":"2024-04-26T23:04:16.087222+0000","last_active":"2024-04-26T23:04:18.092660+0000","last_peered":"2024-04-26T23:04:18.092660+0000","last_clean":"2024-04-26T23:04:18.092660+0000","last_became_active":"2024-04-26T23:04:16.086896+0000","last_became_peered":"2024-04-26T23:04:16.086896+0000","last_unstale":"2024-04-26T23:04:18.092660+0000","last_undegraded":"2024-04-26T23:04:18.092660+0000","last_fullsized":"2024-04-26T23:04:18.092660+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:35.744533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.099464+0000","last_change":"2024-04-26T23:04:17.097851+0000","last_active":"2024-04-26T23:04:17.099464+0000","last_peered":"2024-04-26T23:04:17.099464+0000","last_clean":"2024-04-26T23:04:17.099464+0000","last_became_active":"2024-04-26T23:04:17.097577+0000","last_became_peered":"2024-04-26T23:04:17.097577+0000","last_unstale":"2024-04-26T23:04:17.099464+0000","last_undegraded":"2024-04-26T23:04:17.099464+0000","last_fullsized":"2024-04-26T23:04:17.099464+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:34.013250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101228+0000","last_change":"2024-04-26T23:04:16.102075+0000","last_active":"2024-04-26T23:04:18.101228+0000","last_peered":"2024-04-26T23:04:18.101228+0000","last_clean":"2024-04-26T23:04:18.101228+0000","last_became_active":"2024-04-26T23:04:16.101000+0000","last_became_peered":"2024-04-26T23:04:16.101000+0000","last_unstale":"2024-04-26T23:04:18.101228+0000","last_undegraded":"2024-04-26T23:04:18.101228+0000","last_fullsized":"2024-04-26T23:04:18.101228+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:05:35.157689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.090353+0000","last_change":"2024-04-26T23:04:17.093421+0000","last_active":"2024-04-26T23:04:18.090353+0000","last_peered":"2024-04-26T23:04:18.090353+0000","last_clean":"2024-04-26T23:04:18.090353+0000","last_became_active":"2024-04-26T23:04:17.089057+0000","last_became_peered":"2024-04-26T23:04:17.089057+0000","last_unstale":"2024-04-26T23:04:18.090353+0000","last_undegraded":"2024-04-26T23:04:18.090353+0000","last_fullsized":"2024-04-26T23:04:18.090353+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:25.066755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.083925+0000","last_change":"2024-04-26T23:04:16.089350+0000","last_active":"2024-04-26T23:04:17.083925+0000","last_peered":"2024-04-26T23:04:17.083925+0000","last_clean":"2024-04-26T23:04:17.083925+0000","last_became_active":"2024-04-26T23:04:16.089062+0000","last_became_peered":"2024-04-26T23:04:16.089062+0000","last_unstale":"2024-04-26T23:04:17.083925+0000","last_undegraded":"2024-04-26T23:04:17.083925+0000","last_fullsized":"2024-04-26T23:04:17.083925+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:28:08.155712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092873+0000","last_change":"2024-04-26T23:04:17.091188+0000","last_active":"2024-04-26T23:04:18.092873+0000","last_peered":"2024-04-26T23:04:18.092873+0000","last_clean":"2024-04-26T23:04:18.092873+0000","last_became_active":"2024-04-26T23:04:17.089833+0000","last_became_peered":"2024-04-26T23:04:17.089833+0000","last_unstale":"2024-04-26T23:04:18.092873+0000","last_undegraded":"2024-04-26T23:04:18.092873+0000","last_fullsized":"2024-04-26T23:04:18.092873+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:30:47.888626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":17,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101415+0000","last_change":"2024-04-26T23:04:16.103455+0000","last_active":"2024-04-26T23:04:18.101415+0000","last_peered":"2024-04-26T23:04:18.101415+0000","last_clean":"2024-04-26T23:04:18.101415+0000","last_became_active":"2024-04-26T23:04:16.103184+0000","last_became_peered":"2024-04-26T23:04:16.103184+0000","last_unstale":"2024-04-26T23:04:18.101415+0000","last_undegraded":"2024-04-26T23:04:18.101415+0000","last_fullsized":"2024-04-26T23:04:18.101415+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:49:35.427546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.098402+0000","last_change":"2024-04-26T23:04:17.096503+0000","last_active":"2024-04-26T23:04:17.098402+0000","last_peered":"2024-04-26T23:04:17.098402+0000","last_clean":"2024-04-26T23:04:17.098402+0000","last_became_active":"2024-04-26T23:04:17.093367+0000","last_became_peered":"2024-04-26T23:04:17.093367+0000","last_unstale":"2024-04-26T23:04:17.098402+0000","last_undegraded":"2024-04-26T23:04:17.098402+0000","last_fullsized":"2024-04-26T23:04:17.098402+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:50:17.848964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":18,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.119345+0000","last_change":"2024-04-26T23:04:16.089752+0000","last_active":"2024-04-26T23:04:17.119345+0000","last_peered":"2024-04-26T23:04:17.119345+0000","last_clean":"2024-04-26T23:04:17.119345+0000","last_became_active":"2024-04-26T23:04:16.089465+0000","last_became_peered":"2024-04-26T23:04:16.089465+0000","last_unstale":"2024-04-26T23:04:17.119345+0000","last_undegraded":"2024-04-26T23:04:17.119345+0000","last_fullsized":"2024-04-26T23:04:17.119345+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:30:46.872439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.091135+0000","last_change":"2024-04-26T23:04:17.094537+0000","last_active":"2024-04-26T23:04:18.091135+0000","last_peered":"2024-04-26T23:04:18.091135+0000","last_clean":"2024-04-26T23:04:18.091135+0000","last_became_active":"2024-04-26T23:04:17.093102+0000","last_became_peered":"2024-04-26T23:04:17.093102+0000","last_unstale":"2024-04-26T23:04:18.091135+0000","last_undegraded":"2024-04-26T23:04:18.091135+0000","last_fullsized":"2024-04-26T23:04:18.091135+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:38:49.132739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093112+0000","last_change":"2024-04-26T23:04:16.094717+0000","last_active":"2024-04-26T23:04:18.093112+0000","last_peered":"2024-04-26T23:04:18.093112+0000","last_clean":"2024-04-26T23:04:18.093112+0000","last_became_active":"2024-04-26T23:04:16.094365+0000","last_became_peered":"2024-04-26T23:04:16.094365+0000","last_unstale":"2024-04-26T23:04:18.093112+0000","last_undegraded":"2024-04-26T23:04:18.093112+0000","last_fullsized":"2024-04-26T23:04:18.093112+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:32.997062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093554+0000","last_change":"2024-04-26T23:04:17.112073+0000","last_active":"2024-04-26T23:04:18.093554+0000","last_peered":"2024-04-26T23:04:18.093554+0000","last_clean":"2024-04-26T23:04:18.093554+0000","last_became_active":"2024-04-26T23:04:17.111658+0000","last_became_peered":"2024-04-26T23:04:17.111658+0000","last_unstale":"2024-04-26T23:04:18.093554+0000","last_undegraded":"2024-04-26T23:04:18.093554+0000","last_fullsized":"2024-04-26T23:04:18.093554+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:14:26.705521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092834+0000","last_change":"2024-04-26T23:04:16.080398+0000","last_active":"2024-04-26T23:04:18.092834+0000","last_peered":"2024-04-26T23:04:18.092834+0000","last_clean":"2024-04-26T23:04:18.092834+0000","last_became_active":"2024-04-26T23:04:16.079680+0000","last_became_peered":"2024-04-26T23:04:16.079680+0000","last_unstale":"2024-04-26T23:04:18.092834+0000","last_undegraded":"2024-04-26T23:04:18.092834+0000","last_fullsized":"2024-04-26T23:04:18.092834+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:36.222215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.102974+0000","last_change":"2024-04-26T23:04:17.098083+0000","last_active":"2024-04-26T23:04:17.102974+0000","last_peered":"2024-04-26T23:04:17.102974+0000","last_clean":"2024-04-26T23:04:17.102974+0000","last_became_active":"2024-04-26T23:04:17.097737+0000","last_became_peered":"2024-04-26T23:04:17.097737+0000","last_unstale":"2024-04-26T23:04:17.102974+0000","last_undegraded":"2024-04-26T23:04:17.102974+0000","last_fullsized":"2024-04-26T23:04:17.102974+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:05.987818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.099917+0000","last_change":"2024-04-26T23:04:16.101294+0000","last_active":"2024-04-26T23:04:18.099917+0000","last_peered":"2024-04-26T23:04:18.099917+0000","last_clean":"2024-04-26T23:04:18.099917+0000","last_became_active":"2024-04-26T23:04:16.100730+0000","last_became_peered":"2024-04-26T23:04:16.100730+0000","last_unstale":"2024-04-26T23:04:18.099917+0000","last_undegraded":"2024-04-26T23:04:18.099917+0000","last_fullsized":"2024-04-26T23:04:18.099917+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:35.744533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093766+0000","last_change":"2024-04-26T23:04:17.096063+0000","last_active":"2024-04-26T23:04:18.093766+0000","last_peered":"2024-04-26T23:04:18.093766+0000","last_clean":"2024-04-26T23:04:18.093766+0000","last_became_active":"2024-04-26T23:04:17.092943+0000","last_became_peered":"2024-04-26T23:04:17.092943+0000","last_unstale":"2024-04-26T23:04:18.093766+0000","last_undegraded":"2024-04-26T23:04:18.093766+0000","last_fullsized":"2024-04-26T23:04:18.093766+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:59.263980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":20,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.115391+0000","last_change":"2024-04-26T23:04:16.095953+0000","last_active":"2024-04-26T23:04:17.115391+0000","last_peered":"2024-04-26T23:04:17.115391+0000","last_clean":"2024-04-26T23:04:17.115391+0000","last_became_active":"2024-04-26T23:04:16.089330+0000","last_became_peered":"2024-04-26T23:04:16.089330+0000","last_unstale":"2024-04-26T23:04:17.115391+0000","last_undegraded":"2024-04-26T23:04:17.115391+0000","last_fullsized":"2024-04-26T23:04:17.115391+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:51:44.403748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.090742+0000","last_change":"2024-04-26T23:04:17.087779+0000","last_active":"2024-04-26T23:04:17.090742+0000","last_peered":"2024-04-26T23:04:17.090742+0000","last_clean":"2024-04-26T23:04:17.090742+0000","last_became_active":"2024-04-26T23:04:17.087322+0000","last_became_peered":"2024-04-26T23:04:17.087322+0000","last_unstale":"2024-04-26T23:04:17.090742+0000","last_undegraded":"2024-04-26T23:04:17.090742+0000","last_fullsized":"2024-04-26T23:04:17.090742+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:02.222941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.094135+0000","last_change":"2024-04-26T23:04:16.097363+0000","last_active":"2024-04-26T23:04:18.094135+0000","last_peered":"2024-04-26T23:04:18.094135+0000","last_clean":"2024-04-26T23:04:18.094135+0000","last_became_active":"2024-04-26T23:04:16.097089+0000","last_became_peered":"2024-04-26T23:04:16.097089+0000","last_unstale":"2024-04-26T23:04:18.094135+0000","last_undegraded":"2024-04-26T23:04:18.094135+0000","last_fullsized":"2024-04-26T23:04:18.094135+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:09:50.040264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087026+0000","last_change":"2024-04-26T23:04:17.092477+0000","last_active":"2024-04-26T23:04:18.087026+0000","last_peered":"2024-04-26T23:04:18.087026+0000","last_clean":"2024-04-26T23:04:18.087026+0000","last_became_active":"2024-04-26T23:04:17.092173+0000","last_became_peered":"2024-04-26T23:04:17.092173+0000","last_unstale":"2024-04-26T23:04:18.087026+0000","last_undegraded":"2024-04-26T23:04:18.087026+0000","last_fullsized":"2024-04-26T23:04:18.087026+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:57:35.423782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.238346+0000","last_change":"2024-04-26T23:04:16.094647+0000","last_active":"2024-04-26T23:04:18.238346+0000","last_peered":"2024-04-26T23:04:18.238346+0000","last_clean":"2024-04-26T23:04:18.238346+0000","last_became_active":"2024-04-26T23:04:16.094375+0000","last_became_peered":"2024-04-26T23:04:16.094375+0000","last_unstale":"2024-04-26T23:04:18.238346+0000","last_undegraded":"2024-04-26T23:04:18.238346+0000","last_fullsized":"2024-04-26T23:04:18.238346+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:51.393053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.104717+0000","last_change":"2024-04-26T23:04:17.103652+0000","last_active":"2024-04-26T23:04:17.104717+0000","last_peered":"2024-04-26T23:04:17.104717+0000","last_clean":"2024-04-26T23:04:17.104717+0000","last_became_active":"2024-04-26T23:04:17.103320+0000","last_became_peered":"2024-04-26T23:04:17.103320+0000","last_unstale":"2024-04-26T23:04:17.104717+0000","last_undegraded":"2024-04-26T23:04:17.104717+0000","last_fullsized":"2024-04-26T23:04:17.104717+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:51:06.254495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093434+0000","last_change":"2024-04-26T23:04:16.088844+0000","last_active":"2024-04-26T23:04:18.093434+0000","last_peered":"2024-04-26T23:04:18.093434+0000","last_clean":"2024-04-26T23:04:18.093434+0000","last_became_active":"2024-04-26T23:04:16.086941+0000","last_became_peered":"2024-04-26T23:04:16.086941+0000","last_unstale":"2024-04-26T23:04:18.093434+0000","last_undegraded":"2024-04-26T23:04:18.093434+0000","last_fullsized":"2024-04-26T23:04:18.093434+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:40:29.162642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087638+0000","last_change":"2024-04-26T23:04:17.109317+0000","last_active":"2024-04-26T23:04:18.087638+0000","last_peered":"2024-04-26T23:04:18.087638+0000","last_clean":"2024-04-26T23:04:18.087638+0000","last_became_active":"2024-04-26T23:04:17.109032+0000","last_became_peered":"2024-04-26T23:04:17.109032+0000","last_unstale":"2024-04-26T23:04:18.087638+0000","last_undegraded":"2024-04-26T23:04:18.087638+0000","last_fullsized":"2024-04-26T23:04:18.087638+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:05:24.474044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093188+0000","last_change":"2024-04-26T23:04:16.098205+0000","last_active":"2024-04-26T23:04:18.093188+0000","last_peered":"2024-04-26T23:04:18.093188+0000","last_clean":"2024-04-26T23:04:18.093188+0000","last_became_active":"2024-04-26T23:04:16.097872+0000","last_became_peered":"2024-04-26T23:04:16.097872+0000","last_unstale":"2024-04-26T23:04:18.093188+0000","last_undegraded":"2024-04-26T23:04:18.093188+0000","last_fullsized":"2024-04-26T23:04:18.093188+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:43:17.495520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.091514+0000","last_change":"2024-04-26T23:04:17.090023+0000","last_active":"2024-04-26T23:04:18.091514+0000","last_peered":"2024-04-26T23:04:18.091514+0000","last_clean":"2024-04-26T23:04:18.091514+0000","last_became_active":"2024-04-26T23:04:17.089182+0000","last_became_peered":"2024-04-26T23:04:17.089182+0000","last_unstale":"2024-04-26T23:04:18.091514+0000","last_undegraded":"2024-04-26T23:04:18.091514+0000","last_fullsized":"2024-04-26T23:04:18.091514+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:06.221700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087627+0000","last_change":"2024-04-26T23:04:16.097882+0000","last_active":"2024-04-26T23:04:18.087627+0000","last_peered":"2024-04-26T23:04:18.087627+0000","last_clean":"2024-04-26T23:04:18.087627+0000","last_became_active":"2024-04-26T23:04:16.093755+0000","last_became_peered":"2024-04-26T23:04:16.093755+0000","last_unstale":"2024-04-26T23:04:18.087627+0000","last_undegraded":"2024-04-26T23:04:18.087627+0000","last_fullsized":"2024-04-26T23:04:18.087627+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:08.895545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093398+0000","last_change":"2024-04-26T23:04:17.094180+0000","last_active":"2024-04-26T23:04:18.093398+0000","last_peered":"2024-04-26T23:04:18.093398+0000","last_clean":"2024-04-26T23:04:18.093398+0000","last_became_active":"2024-04-26T23:04:17.093889+0000","last_became_peered":"2024-04-26T23:04:17.093889+0000","last_unstale":"2024-04-26T23:04:18.093398+0000","last_undegraded":"2024-04-26T23:04:18.093398+0000","last_fullsized":"2024-04-26T23:04:18.093398+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:11:33.823168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.090450+0000","last_change":"2024-04-26T23:04:16.088890+0000","last_active":"2024-04-26T23:04:17.090450+0000","last_peered":"2024-04-26T23:04:17.090450+0000","last_clean":"2024-04-26T23:04:17.090450+0000","last_became_active":"2024-04-26T23:04:16.084413+0000","last_became_peered":"2024-04-26T23:04:16.084413+0000","last_unstale":"2024-04-26T23:04:17.090450+0000","last_undegraded":"2024-04-26T23:04:17.090450+0000","last_fullsized":"2024-04-26T23:04:17.090450+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:25.929664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093628+0000","last_change":"2024-04-26T23:04:17.089079+0000","last_active":"2024-04-26T23:04:18.093628+0000","last_peered":"2024-04-26T23:04:18.093628+0000","last_clean":"2024-04-26T23:04:18.093628+0000","last_became_active":"2024-04-26T23:04:17.088017+0000","last_became_peered":"2024-04-26T23:04:17.088017+0000","last_unstale":"2024-04-26T23:04:18.093628+0000","last_undegraded":"2024-04-26T23:04:18.093628+0000","last_fullsized":"2024-04-26T23:04:18.093628+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:55.287002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093398+0000","last_change":"2024-04-26T23:04:16.094914+0000","last_active":"2024-04-26T23:04:18.093398+0000","last_peered":"2024-04-26T23:04:18.093398+0000","last_clean":"2024-04-26T23:04:18.093398+0000","last_became_active":"2024-04-26T23:04:16.093879+0000","last_became_peered":"2024-04-26T23:04:16.093879+0000","last_unstale":"2024-04-26T23:04:18.093398+0000","last_undegraded":"2024-04-26T23:04:18.093398+0000","last_fullsized":"2024-04-26T23:04:18.093398+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:50:16.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.088235+0000","last_change":"2024-04-26T23:04:17.092499+0000","last_active":"2024-04-26T23:04:18.088235+0000","last_peered":"2024-04-26T23:04:18.088235+0000","last_clean":"2024-04-26T23:04:18.088235+0000","last_became_active":"2024-04-26T23:04:17.092196+0000","last_became_peered":"2024-04-26T23:04:17.092196+0000","last_unstale":"2024-04-26T23:04:18.088235+0000","last_undegraded":"2024-04-26T23:04:18.088235+0000","last_fullsized":"2024-04-26T23:04:18.088235+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:38:49.132739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.238678+0000","last_change":"2024-04-26T23:04:16.101090+0000","last_active":"2024-04-26T23:04:18.238678+0000","last_peered":"2024-04-26T23:04:18.238678+0000","last_clean":"2024-04-26T23:04:18.238678+0000","last_became_active":"2024-04-26T23:04:16.100161+0000","last_became_peered":"2024-04-26T23:04:16.100161+0000","last_unstale":"2024-04-26T23:04:18.238678+0000","last_undegraded":"2024-04-26T23:04:18.238678+0000","last_fullsized":"2024-04-26T23:04:18.238678+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:38:41.800210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.090427+0000","last_change":"2024-04-26T23:04:17.095748+0000","last_active":"2024-04-26T23:04:18.090427+0000","last_peered":"2024-04-26T23:04:18.090427+0000","last_clean":"2024-04-26T23:04:18.090427+0000","last_became_active":"2024-04-26T23:04:17.093979+0000","last_became_peered":"2024-04-26T23:04:17.093979+0000","last_unstale":"2024-04-26T23:04:18.090427+0000","last_undegraded":"2024-04-26T23:04:18.090427+0000","last_fullsized":"2024-04-26T23:04:18.090427+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:52:58.602974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093494+0000","last_change":"2024-04-26T23:04:16.085825+0000","last_active":"2024-04-26T23:04:18.093494+0000","last_peered":"2024-04-26T23:04:18.093494+0000","last_clean":"2024-04-26T23:04:18.093494+0000","last_became_active":"2024-04-26T23:04:16.085389+0000","last_became_peered":"2024-04-26T23:04:16.085389+0000","last_unstale":"2024-04-26T23:04:18.093494+0000","last_undegraded":"2024-04-26T23:04:18.093494+0000","last_fullsized":"2024-04-26T23:04:18.093494+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:08.895545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.087312+0000","last_change":"2024-04-26T23:04:17.098246+0000","last_active":"2024-04-26T23:04:18.087312+0000","last_peered":"2024-04-26T23:04:18.087312+0000","last_clean":"2024-04-26T23:04:18.087312+0000","last_became_active":"2024-04-26T23:04:17.095967+0000","last_became_peered":"2024-04-26T23:04:17.095967+0000","last_unstale":"2024-04-26T23:04:18.087312+0000","last_undegraded":"2024-04-26T23:04:18.087312+0000","last_fullsized":"2024-04-26T23:04:18.087312+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:11:33.823168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.101237+0000","last_change":"2024-04-26T23:04:16.101742+0000","last_active":"2024-04-26T23:04:18.101237+0000","last_peered":"2024-04-26T23:04:18.101237+0000","last_clean":"2024-04-26T23:04:18.101237+0000","last_became_active":"2024-04-26T23:04:16.101385+0000","last_became_peered":"2024-04-26T23:04:16.101385+0000","last_unstale":"2024-04-26T23:04:18.101237+0000","last_undegraded":"2024-04-26T23:04:18.101237+0000","last_fullsized":"2024-04-26T23:04:18.101237+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:50:16.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.105040+0000","last_change":"2024-04-26T23:04:17.089678+0000","last_active":"2024-04-26T23:04:17.105040+0000","last_peered":"2024-04-26T23:04:17.105040+0000","last_clean":"2024-04-26T23:04:17.105040+0000","last_became_active":"2024-04-26T23:04:17.089187+0000","last_became_peered":"2024-04-26T23:04:17.089187+0000","last_unstale":"2024-04-26T23:04:17.105040+0000","last_undegraded":"2024-04-26T23:04:17.105040+0000","last_fullsized":"2024-04-26T23:04:17.105040+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:06.221700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093568+0000","last_change":"2024-04-26T23:04:16.087592+0000","last_active":"2024-04-26T23:04:18.093568+0000","last_peered":"2024-04-26T23:04:18.093568+0000","last_clean":"2024-04-26T23:04:18.093568+0000","last_became_active":"2024-04-26T23:04:16.087278+0000","last_became_peered":"2024-04-26T23:04:16.087278+0000","last_unstale":"2024-04-26T23:04:18.093568+0000","last_undegraded":"2024-04-26T23:04:18.093568+0000","last_fullsized":"2024-04-26T23:04:18.093568+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:51:05.238308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093426+0000","last_change":"2024-04-26T23:04:17.095993+0000","last_active":"2024-04-26T23:04:18.093426+0000","last_peered":"2024-04-26T23:04:18.093426+0000","last_clean":"2024-04-26T23:04:18.093426+0000","last_became_active":"2024-04-26T23:04:17.092887+0000","last_became_peered":"2024-04-26T23:04:17.092887+0000","last_unstale":"2024-04-26T23:04:18.093426+0000","last_undegraded":"2024-04-26T23:04:18.093426+0000","last_fullsized":"2024-04-26T23:04:18.093426+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:34.886245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.092762+0000","last_change":"2024-04-26T23:04:16.091433+0000","last_active":"2024-04-26T23:04:17.092762+0000","last_peered":"2024-04-26T23:04:17.092762+0000","last_clean":"2024-04-26T23:04:17.092762+0000","last_became_active":"2024-04-26T23:04:16.089397+0000","last_became_peered":"2024-04-26T23:04:16.089397+0000","last_unstale":"2024-04-26T23:04:17.092762+0000","last_undegraded":"2024-04-26T23:04:17.092762+0000","last_fullsized":"2024-04-26T23:04:17.092762+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:43.776224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.091747+0000","last_change":"2024-04-26T23:04:17.096767+0000","last_active":"2024-04-26T23:04:18.091747+0000","last_peered":"2024-04-26T23:04:18.091747+0000","last_clean":"2024-04-26T23:04:18.091747+0000","last_became_active":"2024-04-26T23:04:17.094004+0000","last_became_peered":"2024-04-26T23:04:17.094004+0000","last_unstale":"2024-04-26T23:04:18.091747+0000","last_undegraded":"2024-04-26T23:04:18.091747+0000","last_fullsized":"2024-04-26T23:04:18.091747+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:33.055716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.090691+0000","last_change":"2024-04-26T23:04:16.091306+0000","last_active":"2024-04-26T23:04:17.090691+0000","last_peered":"2024-04-26T23:04:17.090691+0000","last_clean":"2024-04-26T23:04:17.090691+0000","last_became_active":"2024-04-26T23:04:16.091047+0000","last_became_peered":"2024-04-26T23:04:16.091047+0000","last_unstale":"2024-04-26T23:04:17.090691+0000","last_undegraded":"2024-04-26T23:04:17.090691+0000","last_fullsized":"2024-04-26T23:04:17.090691+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:35:14.696995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.093833+0000","last_change":"2024-04-26T23:04:17.091332+0000","last_active":"2024-04-26T23:04:18.093833+0000","last_peered":"2024-04-26T23:04:18.093833+0000","last_clean":"2024-04-26T23:04:18.093833+0000","last_became_active":"2024-04-26T23:04:17.090638+0000","last_became_peered":"2024-04-26T23:04:17.090638+0000","last_unstale":"2024-04-26T23:04:18.093833+0000","last_undegraded":"2024-04-26T23:04:18.093833+0000","last_fullsized":"2024-04-26T23:04:18.093833+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:44:44.675774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.084291+0000","last_change":"2024-04-26T23:04:16.097479+0000","last_active":"2024-04-26T23:04:17.084291+0000","last_peered":"2024-04-26T23:04:17.084291+0000","last_clean":"2024-04-26T23:04:17.084291+0000","last_became_active":"2024-04-26T23:04:16.097258+0000","last_became_peered":"2024-04-26T23:04:16.097258+0000","last_unstale":"2024-04-26T23:04:17.084291+0000","last_undegraded":"2024-04-26T23:04:17.084291+0000","last_fullsized":"2024-04-26T23:04:17.084291+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:05:35.157689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-26T23:04:17.106900+0000","last_change":"2024-04-26T23:04:17.102835+0000","last_active":"2024-04-26T23:04:17.106900+0000","last_peered":"2024-04-26T23:04:17.106900+0000","last_clean":"2024-04-26T23:04:17.106900+0000","last_became_active":"2024-04-26T23:04:17.102491+0000","last_became_peered":"2024-04-26T23:04:17.102491+0000","last_unstale":"2024-04-26T23:04:17.106900+0000","last_undegraded":"2024-04-26T23:04:17.106900+0000","last_fullsized":"2024-04-26T23:04:17.106900+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:57.457992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.090688+0000","last_change":"2024-04-26T23:04:16.096428+0000","last_active":"2024-04-26T23:04:18.090688+0000","last_peered":"2024-04-26T23:04:18.090688+0000","last_clean":"2024-04-26T23:04:18.090688+0000","last_became_active":"2024-04-26T23:04:16.096138+0000","last_became_peered":"2024-04-26T23:04:16.096138+0000","last_unstale":"2024-04-26T23:04:18.090688+0000","last_undegraded":"2024-04-26T23:04:18.090688+0000","last_fullsized":"2024-04-26T23:04:18.090688+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:09.785131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.092585+0000","last_change":"2024-04-26T23:04:17.085064+0000","last_active":"2024-04-26T23:04:18.092585+0000","last_peered":"2024-04-26T23:04:18.092585+0000","last_clean":"2024-04-26T23:04:18.092585+0000","last_became_active":"2024-04-26T23:04:17.084698+0000","last_became_peered":"2024-04-26T23:04:17.084698+0000","last_unstale":"2024-04-26T23:04:18.092585+0000","last_undegraded":"2024-04-26T23:04:18.092585+0000","last_fullsized":"2024-04-26T23:04:18.092585+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:16.042996+0000","last_clean_scrub_stamp":"2024-04-26T23:04:16.042996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:09:51.056451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.097713+0000","last_change":"2024-04-26T23:04:16.095445+0000","last_active":"2024-04-26T23:04:18.097713+0000","last_peered":"2024-04-26T23:04:18.097713+0000","last_clean":"2024-04-26T23:04:18.097713+0000","last_became_active":"2024-04-26T23:04:16.095153+0000","last_became_peered":"2024-04-26T23:04:16.095153+0000","last_unstale":"2024-04-26T23:04:18.097713+0000","last_undegraded":"2024-04-26T23:04:18.097713+0000","last_fullsized":"2024-04-26T23:04:18.097713+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:04:15.026809+0000","last_clean_scrub_stamp":"2024-04-26T23:04:15.026809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:09.785131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":232,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-26T23:04:18.099502+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:04:18.099502+0000","last_peered":"2024-04-26T23:04:18.099502+0000","last_clean":"2024-04-26T23:04:18.099502+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:04:18.099502+0000","last_undegraded":"2024-04-26T23:04:18.099502+0000","last_fullsized":"2024-04-26T23:04:18.099502+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1208320,"data_stored":862241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":272401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":606208,"data_stored":271749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":272331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":622592,"data_stored":273383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574893,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":618496,"data_stored":273361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":614400,"data_stored":271909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1220608,"data_stored":868865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:04:22.081 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T23:04:22.081 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-26T23:04:22.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.346+0000 7fcdb2c00640 1 Processor -- start 2024-04-26T23:04:22.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.346+0000 7fcdb2c00640 1 -- start start 2024-04-26T23:04:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.348+0000 7fcdb2c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac0ce2b0 0x7fcdac169e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.348+0000 7fcdb2c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdac06e170 con 0x7fcdac0b31c0 2024-04-26T23:04:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.348+0000 7fcdb2c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdac0b40d0 con 0x7fcdac0b3970 2024-04-26T23:04:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.348+0000 7fcdb2c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcdac0d17c0 con 0x7fcdac0ce2b0 2024-04-26T23:04:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.348+0000 7fcdb1800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcdac0b31c0 0x7fcdac06d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57988/0 (socket says 172.21.15.40:57988) 2024-04-26T23:04:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.348+0000 7fcdb1800640 1 -- 172.21.15.40:0/318118493 learned_addr learned my addr 172.21.15.40:0/318118493 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:22.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdb2200640 1 --2- 172.21.15.40:0/318118493 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac0ce2b0 0x7fcdac169e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 828605288 0 0) 0x7fcdac06e170 con 0x7fcdac0b31c0 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd94003520 con 0x7fcdac0b31c0 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3281035315 0 0) 0x7fcd94003520 con 0x7fcdac0b31c0 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac0ce2b0 msgr2=0x7fcdac169e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 --2- 172.21.15.40:0/318118493 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac0ce2b0 0x7fcdac169e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 >> v1:172.21.15.103:6789/0 conn(0x7fcdac0b3970 legacy=0x7fcdac06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdac0b43b0 con 0x7fcdac0b31c0 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fcda0001670 con 0x7fcdac0b31c0 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 >> v1:172.21.15.40:6789/0 conn(0x7fcdac0b31c0 legacy=0x7fcdac06d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 --2- 172.21.15.40:0/318118493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940039e0 0x7fcd94003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 --2- 172.21.15.40:0/318118493 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac0ce2b0 0x7fcd94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 --2- 172.21.15.40:0/318118493 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd94004d40 0x7fcd940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcdac0d17c0 con 0x7fcd940039e0 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9400baa0 con 0x7fcd94004d40 2024-04-26T23:04:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.349+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcd9400bca0 con 0x7fcdac0ce2b0 2024-04-26T23:04:22.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.350+0000 7fcdb1800640 1 --2- 172.21.15.40:0/318118493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940039e0 0x7fcd94003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.350+0000 7fcdb0e00640 1 --2- 172.21.15.40:0/318118493 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac0ce2b0 0x7fcd94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.350+0000 7fcdb2200640 1 --2- 172.21.15.40:0/318118493 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd94004d40 0x7fcd940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.350+0000 7fcdb1800640 1 -- 172.21.15.40:0/318118493 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac0ce2b0 msgr2=0x7fcd94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:22.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.350+0000 7fcdb1800640 1 --2- 172.21.15.40:0/318118493 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac0ce2b0 0x7fcd94004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.350+0000 7fcdb1800640 1 -- 172.21.15.40:0/318118493 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd94004d40 msgr2=0x7fcd940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:22.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.350+0000 7fcdb1800640 1 --2- 172.21.15.40:0/318118493 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd94004d40 0x7fcd940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.350+0000 7fcdb1800640 1 -- 172.21.15.40:0/318118493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd94003520 con 0x7fcd940039e0 2024-04-26T23:04:22.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.350+0000 7fcdb2200640 1 --2- 172.21.15.40:0/318118493 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd94004d40 0x7fcd940051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:22.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.351+0000 7fcdb1800640 1 --2- 172.21.15.40:0/318118493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940039e0 0x7fcd94003e00 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fcda0001eb0 tx=0x7fcd94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65f233286866011 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:22.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.352+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda0007d30 con 0x7fcd940039e0 2024-04-26T23:04:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.352+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcda00058d0 con 0x7fcd940039e0 2024-04-26T23:04:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.352+0000 7fcdabe00640 1 -- 172.21.15.40:0/318118493 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda0005ce0 con 0x7fcd940039e0 2024-04-26T23:04:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.353+0000 7fcdb2c00640 1 -- 172.21.15.40:0/318118493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940039e0 msgr2=0x7fcd94003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.353+0000 7fcdb2c00640 1 --2- 172.21.15.40:0/318118493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940039e0 0x7fcd94003e00 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fcda0001eb0 tx=0x7fcd94005830 comp rx=0 tx=0).stop 2024-04-26T23:04:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.353+0000 7fcdb2200640 1 -- 172.21.15.40:0/318118493 reap_dead start 2024-04-26T23:04:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.353+0000 7fcdb2c00640 1 -- 172.21.15.40:0/318118493 shutdown_connections 2024-04-26T23:04:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.353+0000 7fcdb2c00640 1 -- 172.21.15.40:0/318118493 >> 172.21.15.40:0/318118493 conn(0x7fcdac0b4760 msgr2=0x7fcdac0b4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.353+0000 7fcdb2c00640 1 -- 172.21.15.40:0/318118493 shutdown_connections 2024-04-26T23:04:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.353+0000 7fcdb2c00640 1 -- 172.21.15.40:0/318118493 wait complete. 2024-04-26T23:04:22.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.355+0000 7fcdb2c00640 1 Processor -- start 2024-04-26T23:04:22.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.355+0000 7fcdb2c00640 1 -- start start 2024-04-26T23:04:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.356+0000 7fcdb2c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940039e0 0x7fcdac161910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.356+0000 7fcdb2c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdac1f8b70 0x7fcdac1f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.356+0000 7fcdb2c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac14f6e0 0x7fcdac06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.356+0000 7fcdb2c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcdac0b35a0 con 0x7fcd940039e0 2024-04-26T23:04:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.356+0000 7fcdb2c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcdac0d1ce0 con 0x7fcdac1f8b70 2024-04-26T23:04:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.356+0000 7fcdb2c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcdac0b3720 con 0x7fcdac14f6e0 2024-04-26T23:04:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.356+0000 7fcdb1800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940039e0 0x7fcdac161910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.356+0000 7fcdb1800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940039e0 0x7fcdac161910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42484/0 (socket says 172.21.15.40:42484) 2024-04-26T23:04:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.356+0000 7fcdb1800640 1 -- 172.21.15.40:0/1907508321 learned_addr learned my addr 172.21.15.40:0/1907508321 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.356+0000 7fcdb2200640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac14f6e0 0x7fcdac06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.356+0000 7fcdb0e00640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdac1f8b70 0x7fcdac1f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.356+0000 7fcdb1800640 1 -- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac14f6e0 msgr2=0x7fcdac06d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.356+0000 7fcdb1800640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac14f6e0 0x7fcdac06d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.357+0000 7fcdb1800640 1 -- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdac1f8b70 msgr2=0x7fcdac1f42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.357+0000 7fcdb1800640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdac1f8b70 0x7fcdac1f42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.357+0000 7fcdb1800640 1 -- 172.21.15.40:0/1907508321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdac1f4830 con 0x7fcd940039e0 2024-04-26T23:04:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.357+0000 7fcdb2200640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac14f6e0 0x7fcdac06d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.357+0000 7fcdb1800640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940039e0 0x7fcdac161910 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fcda000c300 tx=0x7fcda0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:22.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.357+0000 7fcda9600640 1 -- 172.21.15.40:0/1907508321 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda0042020 con 0x7fcd940039e0 2024-04-26T23:04:22.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.357+0000 7fcda9600640 1 -- 172.21.15.40:0/1907508321 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcda0000ed0 con 0x7fcd940039e0 2024-04-26T23:04:22.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.357+0000 7fcda9600640 1 -- 172.21.15.40:0/1907508321 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda00074a0 con 0x7fcd940039e0 2024-04-26T23:04:22.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.357+0000 7fcdb2c00640 1 -- 172.21.15.40:0/1907508321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdac1f80d0 con 0x7fcd940039e0 2024-04-26T23:04:22.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.357+0000 7fcdb2c00640 1 -- 172.21.15.40:0/1907508321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdac1f8370 con 0x7fcd940039e0 2024-04-26T23:04:22.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.359+0000 7fcdb2c00640 1 -- 172.21.15.40:0/1907508321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdac069370 con 0x7fcd940039e0 2024-04-26T23:04:22.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.360+0000 7fcda9600640 1 -- 172.21.15.40:0/1907508321 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcda0045020 con 0x7fcd940039e0 2024-04-26T23:04:22.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.361+0000 7fcda9600640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd8c06c920 0x7fcd8c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.361+0000 7fcda9600640 1 -- 172.21.15.40:0/1907508321 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fcda00ac060 con 0x7fcd940039e0 2024-04-26T23:04:22.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.362+0000 7fcdb0e00640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd8c06c920 0x7fcd8c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.363+0000 7fcdb0e00640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd8c06c920 0x7fcd8c06ede0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcd98007fd0 tx=0x7fcd98009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:22.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.366+0000 7fcda9600640 1 -- 172.21.15.40:0/1907508321 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fcdac1f4830 con 0x7fcd940039e0 2024-04-26T23:04:22.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.642+0000 7fcdb2c00640 1 -- 172.21.15.40:0/1907508321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fcdac0d1190 con 0x7fcd940039e0 2024-04-26T23:04:22.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.647+0000 7fcda9600640 1 -- 172.21.15.40:0/1907508321 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) ==== 190+0+147 (secure 0 0 0) 0x7fcda007a2f0 con 0x7fcd940039e0 2024-04-26T23:04:22.647 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-04-26T23:04:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg== 2024-04-26T23:04:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-04-26T23:04:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-04-26T23:04:22.648 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-26T23:04:22.648 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-04-26T23:04:22.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.650+0000 7fcdb2c00640 1 -- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd8c06c920 msgr2=0x7fcd8c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:22.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.650+0000 7fcdb2c00640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd8c06c920 0x7fcd8c06ede0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcd98007fd0 tx=0x7fcd98009040 comp rx=0 tx=0).stop 2024-04-26T23:04:22.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.650+0000 7fcdb2c00640 1 -- 172.21.15.40:0/1907508321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940039e0 msgr2=0x7fcdac161910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:22.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.650+0000 7fcdb2c00640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940039e0 0x7fcdac161910 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fcda000c300 tx=0x7fcda0001a40 comp rx=0 tx=0).stop 2024-04-26T23:04:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.650+0000 7fcdb2c00640 1 -- 172.21.15.40:0/1907508321 shutdown_connections 2024-04-26T23:04:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.650+0000 7fcdb2c00640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd8c06c920 0x7fcd8c06ede0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.650+0000 7fcdb2c00640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdac14f6e0 0x7fcdac06d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.650+0000 7fcdb2c00640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdac1f8b70 0x7fcdac1f42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.650+0000 7fcdb2c00640 1 --2- 172.21.15.40:0/1907508321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940039e0 0x7fcdac161910 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.650+0000 7fcdb2c00640 1 -- 172.21.15.40:0/1907508321 >> 172.21.15.40:0/1907508321 conn(0x7fcdac0b4760 msgr2=0x7fcdac0b4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.650+0000 7fcdb2c00640 1 -- 172.21.15.40:0/1907508321 shutdown_connections 2024-04-26T23:04:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.650+0000 7fcdb2c00640 1 -- 172.21.15.40:0/1907508321 wait complete. 2024-04-26T23:04:22.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:04:22.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.881+0000 7fa565200640 1 Processor -- start 2024-04-26T23:04:22.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.881+0000 7fa565200640 1 -- start start 2024-04-26T23:04:22.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.882+0000 7fa565200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa560070a80 0x7fa560163610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.882+0000 7fa565200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5600d2c30 0x7fa5600d30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.882+0000 7fa565200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5600701e0 con 0x7fa56015f1d0 2024-04-26T23:04:22.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.883+0000 7fa565200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa56015f5b0 con 0x7fa560070a80 2024-04-26T23:04:22.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.883+0000 7fa565200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa56006da80 con 0x7fa5600d2c30 2024-04-26T23:04:22.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.883+0000 7fa55f400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa56015f1d0 0x7fa5600d0e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:55302/0 (socket says 172.21.15.40:55302) 2024-04-26T23:04:22.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.883+0000 7fa55f400640 1 -- 172.21.15.40:0/2973095160 learned_addr learned my addr 172.21.15.40:0/2973095160 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:22.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.883+0000 7fa55fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5600d2c30 0x7fa5600d30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.884+0000 7fa55ea00640 1 --2- 172.21.15.40:0/2973095160 >> v2:172.21.15.103:3300/0 conn(0x7fa560070a80 0x7fa560163610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.884+0000 7fa55e000640 1 -- 172.21.15.40:0/2973095160 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 763013572 0 0) 0x7fa5600701e0 con 0x7fa56015f1d0 2024-04-26T23:04:22.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.884+0000 7fa55e000640 1 -- 172.21.15.40:0/2973095160 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa540003520 con 0x7fa56015f1d0 2024-04-26T23:04:22.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.884+0000 7fa55fe00640 1 -- 172.21.15.40:0/2973095160 >> v2:172.21.15.103:3300/0 conn(0x7fa560070a80 msgr2=0x7fa560163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:22.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.884+0000 7fa55fe00640 1 --2- 172.21.15.40:0/2973095160 >> v2:172.21.15.103:3300/0 conn(0x7fa560070a80 0x7fa560163610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.884+0000 7fa55fe00640 1 -- 172.21.15.40:0/2973095160 >> v1:172.21.15.103:6789/0 conn(0x7fa56015f1d0 legacy=0x7fa5600d0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:22.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.884+0000 7fa55fe00640 1 -- 172.21.15.40:0/2973095160 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5600704c0 con 0x7fa5600d2c30 2024-04-26T23:04:22.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.886+0000 7fa55fe00640 1 --2- 172.21.15.40:0/2973095160 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5600d2c30 0x7fa5600d30c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa56006dc00 tx=0x7fa55802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31708d2562c3b35a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:22.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.886+0000 7fa55e000640 1 -- 172.21.15.40:0/2973095160 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558037070 con 0x7fa5600d2c30 2024-04-26T23:04:22.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.886+0000 7fa55e000640 1 -- 172.21.15.40:0/2973095160 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa55802ae50 con 0x7fa5600d2c30 2024-04-26T23:04:22.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.886+0000 7fa55e000640 1 -- 172.21.15.40:0/2973095160 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558032660 con 0x7fa5600d2c30 2024-04-26T23:04:22.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.887+0000 7fa565200640 1 -- 172.21.15.40:0/2973095160 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5600d2c30 msgr2=0x7fa5600d30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:22.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.887+0000 7fa565200640 1 --2- 172.21.15.40:0/2973095160 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5600d2c30 0x7fa5600d30c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa56006dc00 tx=0x7fa55802a100 comp rx=0 tx=0).stop 2024-04-26T23:04:22.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.887+0000 7fa565200640 1 -- 172.21.15.40:0/2973095160 shutdown_connections 2024-04-26T23:04:22.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.887+0000 7fa565200640 1 --2- 172.21.15.40:0/2973095160 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5600d2c30 0x7fa5600d30c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.887+0000 7fa565200640 1 --2- 172.21.15.40:0/2973095160 >> v2:172.21.15.103:3300/0 conn(0x7fa560070a80 0x7fa560163610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.887+0000 7fa565200640 1 -- 172.21.15.40:0/2973095160 >> 172.21.15.40:0/2973095160 conn(0x7fa56009ba50 msgr2=0x7fa56009be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:22.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.887+0000 7fa565200640 1 -- 172.21.15.40:0/2973095160 shutdown_connections 2024-04-26T23:04:22.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.887+0000 7fa565200640 1 -- 172.21.15.40:0/2973095160 wait complete. 2024-04-26T23:04:22.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.889+0000 7fa565200640 1 Processor -- start 2024-04-26T23:04:22.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.889+0000 7fa565200640 1 -- start start 2024-04-26T23:04:22.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.890+0000 7fa565200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa560070a80 0x7fa560157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.890+0000 7fa565200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5600d2c30 0x7fa5601596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.890+0000 7fa565200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa56015f1d0 0x7fa5601e9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.890+0000 7fa565200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa56006dcd0 con 0x7fa560070a80 2024-04-26T23:04:22.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.890+0000 7fa565200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa560163b50 con 0x7fa5600d2c30 2024-04-26T23:04:22.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.890+0000 7fa565200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa5600d1580 con 0x7fa56015f1d0 2024-04-26T23:04:22.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.890+0000 7fa55f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa560070a80 0x7fa560157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.890+0000 7fa55f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa560070a80 0x7fa560157eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42494/0 (socket says 172.21.15.40:42494) 2024-04-26T23:04:22.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.890+0000 7fa55f400640 1 -- 172.21.15.40:0/545150421 learned_addr learned my addr 172.21.15.40:0/545150421 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:22.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.891+0000 7fa55fe00640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa56015f1d0 0x7fa5601e9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.891+0000 7fa55ea00640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5600d2c30 0x7fa5601596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.891+0000 7fa55f400640 1 -- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa56015f1d0 msgr2=0x7fa5601e9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:22.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.891+0000 7fa55f400640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa56015f1d0 0x7fa5601e9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.891+0000 7fa55f400640 1 -- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5600d2c30 msgr2=0x7fa5601596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:22.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.891+0000 7fa55f400640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5600d2c30 0x7fa5601596e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:22.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.891+0000 7fa55f400640 1 -- 172.21.15.40:0/545150421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5601f8a20 con 0x7fa560070a80 2024-04-26T23:04:22.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.891+0000 7fa55f400640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa560070a80 0x7fa560157eb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fa55000c430 tx=0x7fa55000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:22.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.892+0000 7fa557400640 1 -- 172.21.15.40:0/545150421 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550017070 con 0x7fa560070a80 2024-04-26T23:04:22.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.892+0000 7fa557400640 1 -- 172.21.15.40:0/545150421 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa550005ce0 con 0x7fa560070a80 2024-04-26T23:04:22.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.892+0000 7fa557400640 1 -- 172.21.15.40:0/545150421 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5500072f0 con 0x7fa560070a80 2024-04-26T23:04:22.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.892+0000 7fa565200640 1 -- 172.21.15.40:0/545150421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5601f8d00 con 0x7fa560070a80 2024-04-26T23:04:22.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.892+0000 7fa565200640 1 -- 172.21.15.40:0/545150421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5601f92c0 con 0x7fa560070a80 2024-04-26T23:04:22.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.893+0000 7fa565200640 1 -- 172.21.15.40:0/545150421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa560069a80 con 0x7fa560070a80 2024-04-26T23:04:22.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.894+0000 7fa557400640 1 -- 172.21.15.40:0/545150421 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa55001a070 con 0x7fa560070a80 2024-04-26T23:04:22.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.895+0000 7fa557400640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa53c06c8f0 0x7fa53c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:22.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.895+0000 7fa557400640 1 -- 172.21.15.40:0/545150421 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fa55008b260 con 0x7fa560070a80 2024-04-26T23:04:22.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.896+0000 7fa55ea00640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa53c06c8f0 0x7fa53c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:22.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.897+0000 7fa55ea00640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa53c06c8f0 0x7fa53c06edb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa5600d2580 tx=0x7fa54800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:22.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:22.900+0000 7fa557400640 1 -- 172.21.15.40:0/545150421 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa550059460 con 0x7fa560070a80 2024-04-26T23:04:23.024 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:04:23.024+0000 7f47d7c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T23:04:23.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.146+0000 7fa565200640 1 -- 172.21.15.40:0/545150421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa560158a90 con 0x7fa560070a80 2024-04-26T23:04:23.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.147+0000 7fa557400640 1 -- 172.21.15.40:0/545150421 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5178 (secure 0 0 0) 0x7fa550058d60 con 0x7fa560070a80 2024-04-26T23:04:23.147 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:23.148 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/189004193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":189004193},{"type":"v1","addr":"172.21.15.103:6835","nonce":189004193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5284,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/670995452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":670995452},{"type":"v1","addr":"172.21.15.103:6837","nonce":670995452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":6027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/2304583125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2304583125},{"type":"v1","addr":"172.21.15.40:6834","nonce":2304583125}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:04:16.558591+0000","modified":"2024-04-26T23:04:19.612962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6036},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6036":{"gid":6036,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/618018110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":618018110},{"type":"v1","addr":"172.21.15.40:6835","nonce":618018110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6036,"qdb_cluster":[6036]},"id":3}]} 2024-04-26T23:04:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 35 2024-04-26T23:04:23.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.150+0000 7fa565200640 1 -- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa53c06c8f0 msgr2=0x7fa53c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:23.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.150+0000 7fa565200640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa53c06c8f0 0x7fa53c06edb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa5600d2580 tx=0x7fa54800b040 comp rx=0 tx=0).stop 2024-04-26T23:04:23.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.150+0000 7fa565200640 1 -- 172.21.15.40:0/545150421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa560070a80 msgr2=0x7fa560157eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:23.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.150+0000 7fa565200640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa560070a80 0x7fa560157eb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fa55000c430 tx=0x7fa55000c8f0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.150+0000 7fa565200640 1 -- 172.21.15.40:0/545150421 shutdown_connections 2024-04-26T23:04:23.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.150+0000 7fa565200640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa53c06c8f0 0x7fa53c06edb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.150+0000 7fa565200640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa56015f1d0 0x7fa5601e9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.150+0000 7fa565200640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5600d2c30 0x7fa5601596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.150+0000 7fa565200640 1 --2- 172.21.15.40:0/545150421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa560070a80 0x7fa560157eb0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.150+0000 7fa565200640 1 -- 172.21.15.40:0/545150421 >> 172.21.15.40:0/545150421 conn(0x7fa56009ba50 msgr2=0x7fa5600b4940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:23.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.150+0000 7fa565200640 1 -- 172.21.15.40:0/545150421 shutdown_connections 2024-04-26T23:04:23.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.150+0000 7fa565200640 1 -- 172.21.15.40:0/545150421 wait complete. 2024-04-26T23:04:23.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:04:16.558591+0000', 'modified': '2024-04-26T23:04:19.612962+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6036}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6036': {'gid': 6036, 'name': 'c', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/618018110', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 618018110}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 618018110}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6036, 'qdb_cluster': [6036]} 2024-04-26T23:04:23.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-26T23:04:23.574 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T23:04:23.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.797+0000 7efef9c00640 1 Processor -- start 2024-04-26T23:04:23.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.798+0000 7efef9c00640 1 -- start start 2024-04-26T23:04:23.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.799+0000 7efef9c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef40b6220 0x7efef40d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.799+0000 7efef9c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efef40d39f0 0x7efef40b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.799+0000 7efef9c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef4162c40 con 0x7efef40d08d0 2024-04-26T23:04:23.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.799+0000 7efef9c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7efef40d2d80 con 0x7efef40d39f0 2024-04-26T23:04:23.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.799+0000 7efef9c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efef40b7c80 con 0x7efef40b6220 2024-04-26T23:04:23.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.800+0000 7efef9200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efef40d39f0 0x7efef40b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.800+0000 7efef9200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efef40d39f0 0x7efef40b8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42496/0 (socket says 172.21.15.40:42496) 2024-04-26T23:04:23.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.800+0000 7efef9200640 1 -- 172.21.15.40:0/1521884181 learned_addr learned my addr 172.21.15.40:0/1521884181 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:23.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.800+0000 7efef3e00640 1 --2- 172.21.15.40:0/1521884181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef40b6220 0x7efef40d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.800+0000 7efef9200640 1 -- 172.21.15.40:0/1521884181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef40b6220 msgr2=0x7efef40d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:23.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.801+0000 7efef9200640 1 --2- 172.21.15.40:0/1521884181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef40b6220 0x7efef40d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.801+0000 7efef9200640 1 -- 172.21.15.40:0/1521884181 >> v1:172.21.15.40:6789/0 conn(0x7efef40d08d0 legacy=0x7efef40d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:23.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.801+0000 7efef9200640 1 -- 172.21.15.40:0/1521884181 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef40b7f60 con 0x7efef40d39f0 2024-04-26T23:04:23.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.802+0000 7efef9200640 1 --2- 172.21.15.40:0/1521884181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef40d39f0 0x7efef40b8d20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7efeec009e50 tx=0x7efeec02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9128fb15ee91f05 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:23.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.802+0000 7efef3400640 1 -- 172.21.15.40:0/1521884181 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec036070 con 0x7efef40d39f0 2024-04-26T23:04:23.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.802+0000 7efef3400640 1 -- 172.21.15.40:0/1521884181 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeec031600 con 0x7efef40d39f0 2024-04-26T23:04:23.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.802+0000 7efef3400640 1 -- 172.21.15.40:0/1521884181 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec0319d0 con 0x7efef40d39f0 2024-04-26T23:04:23.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.803+0000 7efef9c00640 1 -- 172.21.15.40:0/1521884181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef40d39f0 msgr2=0x7efef40b8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:23.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.803+0000 7efef9c00640 1 --2- 172.21.15.40:0/1521884181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef40d39f0 0x7efef40b8d20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7efeec009e50 tx=0x7efeec02a710 comp rx=0 tx=0).stop 2024-04-26T23:04:23.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.803+0000 7efef9c00640 1 -- 172.21.15.40:0/1521884181 shutdown_connections 2024-04-26T23:04:23.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.803+0000 7efef9c00640 1 --2- 172.21.15.40:0/1521884181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef40d39f0 0x7efef40b8d20 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.803+0000 7efef9c00640 1 --2- 172.21.15.40:0/1521884181 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef40b6220 0x7efef40d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.803+0000 7efef9c00640 1 -- 172.21.15.40:0/1521884181 >> 172.21.15.40:0/1521884181 conn(0x7efef40d4d90 msgr2=0x7efef40d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:23.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.803+0000 7efef9c00640 1 -- 172.21.15.40:0/1521884181 shutdown_connections 2024-04-26T23:04:23.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.803+0000 7efef9c00640 1 -- 172.21.15.40:0/1521884181 wait complete. 2024-04-26T23:04:23.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.805+0000 7efef9c00640 1 Processor -- start 2024-04-26T23:04:23.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.805+0000 7efef9c00640 1 -- start start 2024-04-26T23:04:23.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.806+0000 7efef9c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef40b6220 0x7efef415c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.806+0000 7efef9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef40d08d0 0x7efef415cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.806+0000 7efef9c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efef40d39f0 0x7efef415eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.806+0000 7efef9c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efef40b9260 con 0x7efef40d08d0 2024-04-26T23:04:23.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.806+0000 7efef9c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efef40b8520 con 0x7efef40d39f0 2024-04-26T23:04:23.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.806+0000 7efef9c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efef40b93e0 con 0x7efef40b6220 2024-04-26T23:04:23.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.806+0000 7efeebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef40d08d0 0x7efef415cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.806+0000 7efeebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef40d08d0 0x7efef415cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42500/0 (socket says 172.21.15.40:42500) 2024-04-26T23:04:23.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.806+0000 7efeebe00640 1 -- 172.21.15.40:0/1323646047 learned_addr learned my addr 172.21.15.40:0/1323646047 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:23.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.807+0000 7efef9200640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efef40d39f0 0x7efef415eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.807+0000 7efef3e00640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef40b6220 0x7efef415c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.807+0000 7efeebe00640 1 -- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef40b6220 msgr2=0x7efef415c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:23.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.807+0000 7efeebe00640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef40b6220 0x7efef415c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.807+0000 7efeebe00640 1 -- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efef40d39f0 msgr2=0x7efef415eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:23.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.807+0000 7efeebe00640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efef40d39f0 0x7efef415eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.807+0000 7efeebe00640 1 -- 172.21.15.40:0/1323646047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef415cb90 con 0x7efef40d08d0 2024-04-26T23:04:23.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.807+0000 7efef9200640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efef40d39f0 0x7efef415eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:23.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.807+0000 7efeebe00640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef40d08d0 0x7efef415cd80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7efedc00c430 tx=0x7efedc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:23.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.807+0000 7efef0c00640 1 -- 172.21.15.40:0/1323646047 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc0052d0 con 0x7efef40d08d0 2024-04-26T23:04:23.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.808+0000 7efef0c00640 1 -- 172.21.15.40:0/1323646047 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efedc005490 con 0x7efef40d08d0 2024-04-26T23:04:23.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.808+0000 7efef0c00640 1 -- 172.21.15.40:0/1323646047 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc007260 con 0x7efef40d08d0 2024-04-26T23:04:23.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.808+0000 7efef9c00640 1 -- 172.21.15.40:0/1323646047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef415e3c0 con 0x7efef40d08d0 2024-04-26T23:04:23.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.808+0000 7efef9c00640 1 -- 172.21.15.40:0/1323646047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef415db20 con 0x7efef40d08d0 2024-04-26T23:04:23.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.810+0000 7efef0c00640 1 -- 172.21.15.40:0/1323646047 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efedc007400 con 0x7efef40d08d0 2024-04-26T23:04:23.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.811+0000 7efef0c00640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efed006c8f0 0x7efed006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.811+0000 7efef0c00640 1 -- 172.21.15.40:0/1323646047 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7efedc08b2e0 con 0x7efef40d08d0 2024-04-26T23:04:23.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.812+0000 7efefe045740 1 Processor -- start 2024-04-26T23:04:23.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.812+0000 7efef3e00640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efed006c8f0 0x7efed006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.812+0000 7efefe045740 1 -- start start 2024-04-26T23:04:23.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.812+0000 7efefe045740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd557b0 0x5638dcd57910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efefe045740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd91e80 0x5638dcd55b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efefe045740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd80270 0x5638dcd806f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efefe045740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5638dccb9600 con 0x5638dcd91e80 2024-04-26T23:04:23.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efefe045740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5638dcc56100 con 0x5638dcd557b0 2024-04-26T23:04:23.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efefe045740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5638dccb55c0 con 0x5638dcd80270 2024-04-26T23:04:23.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efeebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd91e80 0x5638dcd55b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efeebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd91e80 0x5638dcd55b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42510/0 (socket says 172.21.15.40:42510) 2024-04-26T23:04:23.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efeebe00640 1 -- 172.21.15.40:0/3724410355 learned_addr learned my addr 172.21.15.40:0/3724410355 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:23.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efef3e00640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efed006c8f0 0x7efed006edb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7efee4006fd0 tx=0x7efee4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:23.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efef3e00640 1 --2- 172.21.15.40:0/3724410355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd557b0 0x5638dcd57910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efef9200640 1 --2- 172.21.15.40:0/3724410355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd80270 0x5638dcd806f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efeebe00640 1 -- 172.21.15.40:0/3724410355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd80270 msgr2=0x5638dcd806f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:23.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efeebe00640 1 --2- 172.21.15.40:0/3724410355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd80270 0x5638dcd806f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efeebe00640 1 -- 172.21.15.40:0/3724410355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd557b0 msgr2=0x5638dcd57910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:23.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efeebe00640 1 --2- 172.21.15.40:0/3724410355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd557b0 0x5638dcd57910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.813+0000 7efeebe00640 1 -- 172.21.15.40:0/3724410355 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638dc9cb2d0 con 0x5638dcd91e80 2024-04-26T23:04:23.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.814+0000 7efeebe00640 1 --2- 172.21.15.40:0/3724410355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd91e80 0x5638dcd55b90 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7efedc089550 tx=0x7efedc00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec8b5303abeb059 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:23.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.814+0000 7efee8c00640 1 -- 172.21.15.40:0/3724410355 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc08b0c0 con 0x5638dcd91e80 2024-04-26T23:04:23.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.814+0000 7efee8c00640 1 -- 172.21.15.40:0/3724410355 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efedc0570e0 con 0x5638dcd91e80 2024-04-26T23:04:23.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.814+0000 7efee8c00640 1 -- 172.21.15.40:0/3724410355 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc071170 con 0x5638dcd91e80 2024-04-26T23:04:23.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.815+0000 7efefe045740 1 -- 172.21.15.40:0/3724410355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd91e80 msgr2=0x5638dcd55b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.815+0000 7efefe045740 1 --2- 172.21.15.40:0/3724410355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd91e80 0x5638dcd55b90 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7efedc089550 tx=0x7efedc00a000 comp rx=0 tx=0).stop 2024-04-26T23:04:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.815+0000 7efefe045740 1 -- 172.21.15.40:0/3724410355 shutdown_connections 2024-04-26T23:04:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.815+0000 7efefe045740 1 --2- 172.21.15.40:0/3724410355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd80270 0x5638dcd806f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.815+0000 7efefe045740 1 --2- 172.21.15.40:0/3724410355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd91e80 0x5638dcd55b90 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.815+0000 7efefe045740 1 --2- 172.21.15.40:0/3724410355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd557b0 0x5638dcd57910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.815+0000 7efefe045740 1 -- 172.21.15.40:0/3724410355 >> 172.21.15.40:0/3724410355 conn(0x5638dcce9d10 msgr2=0x5638dcd9b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.815+0000 7efefe045740 1 -- 172.21.15.40:0/3724410355 shutdown_connections 2024-04-26T23:04:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.815+0000 7efefe045740 1 -- 172.21.15.40:0/3724410355 wait complete. 2024-04-26T23:04:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.816+0000 7efefe045740 1 -- start start 2024-04-26T23:04:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.816+0000 7efefe045740 1 Processor -- start 2024-04-26T23:04:23.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.817+0000 7efefe045740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 0x5638dcd70b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.817+0000 7efefe045740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd80270 0x5638dcd81060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.817+0000 7efefe045740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd91e80 0x5638dcd82ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.817+0000 7efefe045740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5638dcc40f70 con 0x5638dcd557b0 2024-04-26T23:04:23.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.817+0000 7efefe045740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5638dcd7ddc0 con 0x5638dcd80270 2024-04-26T23:04:23.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.817+0000 7efefe045740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5638dcdac620 con 0x5638dcd91e80 2024-04-26T23:04:23.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.817+0000 7efefe045740 10 client.0 _ll_register_callbacks cb 0x5638dccaf480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:23.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.817+0000 7efef3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 0x5638dcd70b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.817+0000 7efef3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 0x5638dcd70b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42524/0 (socket says 172.21.15.40:42524) 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.817+0000 7efef3e00640 1 -- 172.21.15.40:0/4170871197 learned_addr learned my addr 172.21.15.40:0/4170871197 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.818+0000 7efef3e00640 1 -- 172.21.15.40:0/4170871197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd91e80 msgr2=0x5638dcd82ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.818+0000 7efef3e00640 1 --2- 172.21.15.40:0/4170871197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd91e80 0x5638dcd82ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.818+0000 7efeebe00640 1 --2- 172.21.15.40:0/4170871197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd80270 0x5638dcd81060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.818+0000 7efef3e00640 1 -- 172.21.15.40:0/4170871197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd80270 msgr2=0x5638dcd81060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.818+0000 7efef3e00640 1 --2- 172.21.15.40:0/4170871197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd80270 0x5638dcd81060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.818+0000 7efef3e00640 1 -- 172.21.15.40:0/4170871197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638dca956b0 con 0x5638dcd557b0 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.818+0000 7efef3e00640 1 --2- 172.21.15.40:0/4170871197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 0x5638dcd70b10 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7efee400ef40 tx=0x7efee4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd6caaefdce60f76 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.818+0000 7efee1600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.818+0000 7efee1600640 1 -- 172.21.15.40:0/4170871197 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee400f040 con 0x5638dcd557b0 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.818+0000 7efee1600640 1 -- 172.21.15.40:0/4170871197 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7efee4015b50 con 0x5638dcd557b0 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.818+0000 7efefe045740 1 -- 172.21.15.40:0/4170871197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efedc0570e0 con 0x5638dcd557b0 2024-04-26T23:04:23.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.819+0000 7efee1600640 1 -- 172.21.15.40:0/4170871197 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efee402e050 con 0x5638dcd557b0 2024-04-26T23:04:23.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.819+0000 7efefe045740 10 client.6183 fetch_fsmap learned FSMap version 35 2024-04-26T23:04:23.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.819+0000 7efefe045740 1 -- 172.21.15.40:0/4170871197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x5638dcdac420 con 0x5638dcd557b0 2024-04-26T23:04:23.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.819+0000 7efee1600640 1 -- 172.21.15.40:0/4170871197 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efedc0570e0 con 0x5638dcd557b0 2024-04-26T23:04:23.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.819+0000 7efee1600640 1 -- 172.21.15.40:0/4170871197 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7efee4031020 con 0x5638dcd557b0 2024-04-26T23:04:23.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.819+0000 7efefe045740 10 client.6183 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T23:04:23.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.819+0000 7efefe045740 10 client.6183 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:04:23.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.819+0000 7efefe045740 20 client.6183 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:23.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.819+0000 7efefe045740 1 --2- 172.21.15.40:0/4170871197 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5638dcd92a00 0x5638dcdab600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.819+0000 7efefe045740 4 client.6183 mds_command: new command op to 6036 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:04:23.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.819+0000 7efefe045740 1 -- 172.21.15.40:0/4170871197 --> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638dccc7210 con 0x5638dcd92a00 2024-04-26T23:04:23.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.820+0000 7efeebe00640 1 --2- 172.21.15.40:0/4170871197 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5638dcd92a00 0x5638dcdab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.820+0000 7efeebe00640 1 --2- 172.21.15.40:0/4170871197 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5638dcd92a00 0x5638dcdab600 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e1ddff25e3d2a53 server_cookie=ddb21e69bb107c89 in_seq=0 out_seq=0 2024-04-26T23:04:23.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.820+0000 7efee1600640 10 client.6183 ms_handle_connect on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:23.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.821+0000 7efee1600640 1 -- 172.21.15.40:0/4170871197 <== mds.0 v2:172.21.15.40:6833/618018110 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638dccc7210 con 0x5638dcd92a00 2024-04-26T23:04:23.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.821+0000 7efee1600640 10 client.6183 handle_command_reply: tid=0 2024-04-26T23:04:23.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.821+0000 7efefe045740 1 client.6183 shutdown 2024-04-26T23:04:23.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.821+0000 7efefe045740 1 -- 172.21.15.40:0/4170871197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 msgr2=0x5638dcd70b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:23.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.821+0000 7efefe045740 1 --2- 172.21.15.40:0/4170871197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 0x5638dcd70b10 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7efee400ef40 tx=0x7efee4004730 comp rx=0 tx=0).stop 2024-04-26T23:04:23.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.821+0000 7efefe045740 1 -- 172.21.15.40:0/4170871197 shutdown_connections 2024-04-26T23:04:23.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.821+0000 7efefe045740 1 --2- 172.21.15.40:0/4170871197 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5638dcd92a00 0x5638dcdab600 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.821+0000 7efefe045740 1 --2- 172.21.15.40:0/4170871197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd91e80 0x5638dcd82ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.821+0000 7efefe045740 1 --2- 172.21.15.40:0/4170871197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd80270 0x5638dcd81060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.821+0000 7efee1600640 0 client.6183 ms_handle_reset on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:23.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.821+0000 7efefe045740 1 --2- 172.21.15.40:0/4170871197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 0x5638dcd70b10 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.821+0000 7efefe045740 1 -- 172.21.15.40:0/4170871197 >> 172.21.15.40:0/4170871197 conn(0x5638dcce9d10 msgr2=0x5638dcd9e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:23.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.822+0000 7efefe045740 1 -- 172.21.15.40:0/4170871197 shutdown_connections 2024-04-26T23:04:23.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.822+0000 7efefe045740 1 -- 172.21.15.40:0/4170871197 wait complete. 2024-04-26T23:04:23.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.822+0000 7efefe045740 20 client.6183 trim_cache size 0 max 16384 2024-04-26T23:04:23.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.842+0000 7efefe045740 1 Processor -- start 2024-04-26T23:04:23.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.842+0000 7efefe045740 1 -- start start 2024-04-26T23:04:23.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.843+0000 7efefe045740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 0x5638dcd6fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.843+0000 7efefe045740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd80270 0x5638dcdbe6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.843+0000 7efefe045740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd91e80 0x5638dcdbec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.843+0000 7efefe045740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efef4200d30 con 0x5638dcd557b0 2024-04-26T23:04:23.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.843+0000 7efefe045740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efef40c9880 con 0x5638dcd91e80 2024-04-26T23:04:23.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.843+0000 7efefe045740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efef42007e0 con 0x5638dcd80270 2024-04-26T23:04:23.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.843+0000 7efef3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 0x5638dcd6fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.843+0000 7efef3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 0x5638dcd6fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42534/0 (socket says 172.21.15.40:42534) 2024-04-26T23:04:23.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.843+0000 7efef3e00640 1 -- 172.21.15.40:0/3432379660 learned_addr learned my addr 172.21.15.40:0/3432379660 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:23.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.844+0000 7efeebe00640 1 --2- 172.21.15.40:0/3432379660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd80270 0x5638dcdbe6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.844+0000 7efef3e00640 1 -- 172.21.15.40:0/3432379660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd80270 msgr2=0x5638dcdbe6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:23.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.844+0000 7efef9200640 1 --2- 172.21.15.40:0/3432379660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd91e80 0x5638dcdbec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.844+0000 7efef3e00640 1 --2- 172.21.15.40:0/3432379660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd80270 0x5638dcdbe6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.844+0000 7efef3e00640 1 -- 172.21.15.40:0/3432379660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd91e80 msgr2=0x5638dcdbec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:23.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.844+0000 7efef3e00640 1 --2- 172.21.15.40:0/3432379660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd91e80 0x5638dcdbec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.844+0000 7efef3e00640 1 -- 172.21.15.40:0/3432379660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638dcd51a70 con 0x5638dcd557b0 2024-04-26T23:04:23.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.844+0000 7efeebe00640 1 --2- 172.21.15.40:0/3432379660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd80270 0x5638dcdbe6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:23.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.844+0000 7efef9200640 1 --2- 172.21.15.40:0/3432379660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd91e80 0x5638dcdbec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:23.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.844+0000 7efef3e00640 1 --2- 172.21.15.40:0/3432379660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 0x5638dcd6fd90 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7efee4009670 tx=0x7efee401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b8f670959430a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:23.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.844+0000 7efee3e00640 1 -- 172.21.15.40:0/3432379660 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee400eb00 con 0x5638dcd557b0 2024-04-26T23:04:23.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.844+0000 7efee3e00640 1 -- 172.21.15.40:0/3432379660 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee4026070 con 0x5638dcd557b0 2024-04-26T23:04:23.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.844+0000 7efee3e00640 1 -- 172.21.15.40:0/3432379660 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee40216c0 con 0x5638dcd557b0 2024-04-26T23:04:23.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.845+0000 7efefe045740 1 -- 172.21.15.40:0/3432379660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 msgr2=0x5638dcd6fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:23.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.845+0000 7efefe045740 1 --2- 172.21.15.40:0/3432379660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 0x5638dcd6fd90 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7efee4009670 tx=0x7efee401f000 comp rx=0 tx=0).stop 2024-04-26T23:04:23.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.845+0000 7efefe045740 1 -- 172.21.15.40:0/3432379660 shutdown_connections 2024-04-26T23:04:23.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.845+0000 7efefe045740 1 --2- 172.21.15.40:0/3432379660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd91e80 0x5638dcdbec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.845+0000 7efefe045740 1 --2- 172.21.15.40:0/3432379660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd80270 0x5638dcdbe6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.845+0000 7efefe045740 1 --2- 172.21.15.40:0/3432379660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd557b0 0x5638dcd6fd90 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.845+0000 7efefe045740 1 -- 172.21.15.40:0/3432379660 >> 172.21.15.40:0/3432379660 conn(0x5638dcce9d10 msgr2=0x5638dcd94c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:23.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.845+0000 7efefe045740 1 -- 172.21.15.40:0/3432379660 shutdown_connections 2024-04-26T23:04:23.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.845+0000 7efefe045740 1 -- 172.21.15.40:0/3432379660 wait complete. 2024-04-26T23:04:23.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.846+0000 7efefe045740 1 -- start start 2024-04-26T23:04:23.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.846+0000 7efefe045740 1 Processor -- start 2024-04-26T23:04:23.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.847+0000 7efefe045740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd557b0 0x5638dce14b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.847+0000 7efefe045740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd80270 0x5638dcd9ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.847+0000 7efefe045740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd91e80 0x5638dcd7ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.847+0000 7efefe045740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5638dcd92880 con 0x5638dcd80270 2024-04-26T23:04:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.847+0000 7efefe045740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5638dcd83540 con 0x5638dcd557b0 2024-04-26T23:04:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.847+0000 7efefe045740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5638dcd92e50 con 0x5638dcd91e80 2024-04-26T23:04:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.847+0000 7efefe045740 10 client.0 _ll_register_callbacks cb 0x7efee4009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.847+0000 7efeebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd80270 0x5638dcd9ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.847+0000 7efeebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd80270 0x5638dcd9ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42546/0 (socket says 172.21.15.40:42546) 2024-04-26T23:04:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.848+0000 7efeebe00640 1 -- 172.21.15.40:0/1766766448 learned_addr learned my addr 172.21.15.40:0/1766766448 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.848+0000 7efeebe00640 1 -- 172.21.15.40:0/1766766448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd91e80 msgr2=0x5638dcd7ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.848+0000 7efeebe00640 1 --2- 172.21.15.40:0/1766766448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd91e80 0x5638dcd7ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.848+0000 7efeebe00640 1 -- 172.21.15.40:0/1766766448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd557b0 msgr2=0x5638dce14b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.848+0000 7efeebe00640 1 --2- 172.21.15.40:0/1766766448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd557b0 0x5638dce14b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.848+0000 7efeebe00640 1 -- 172.21.15.40:0/1766766448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638dcd83130 con 0x5638dcd80270 2024-04-26T23:04:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.848+0000 7efeebe00640 1 --2- 172.21.15.40:0/1766766448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd80270 0x5638dcd9ece0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7efedc00cdf0 tx=0x7efedc01d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc422715c5225ca8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.848+0000 7efee1600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.848+0000 7efee1600640 1 -- 172.21.15.40:0/1766766448 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc076060 con 0x5638dcd80270 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.848+0000 7efee1600640 1 -- 172.21.15.40:0/1766766448 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7efedc0570e0 con 0x5638dcd80270 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.848+0000 7efefe045740 1 -- 172.21.15.40:0/1766766448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efee4026070 con 0x5638dcd80270 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.849+0000 7efee1600640 1 -- 172.21.15.40:0/1766766448 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efedc07d050 con 0x5638dcd80270 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.849+0000 7efefe045740 10 client.6189 fetch_fsmap learned FSMap version 35 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.849+0000 7efefe045740 1 -- 172.21.15.40:0/1766766448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x5638dcdad8b0 con 0x5638dcd80270 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.849+0000 7efee1600640 1 -- 172.21.15.40:0/1766766448 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee4026070 con 0x5638dcd80270 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.849+0000 7efee1600640 1 -- 172.21.15.40:0/1766766448 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7efedc0782b0 con 0x5638dcd80270 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.849+0000 7efefe045740 10 client.6189 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.849+0000 7efefe045740 10 client.6189 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.849+0000 7efefe045740 20 client.6189 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.849+0000 7efefe045740 1 --2- 172.21.15.40:0/1766766448 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5638dcda99b0 0x5638dcda9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.849+0000 7efefe045740 4 client.6189 mds_command: new command op to 6036 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:04:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.849+0000 7efefe045740 1 -- 172.21.15.40:0/1766766448 --> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] -- command(tid 0: {"prefix": "status"}) -- 0x5638dcd962e0 con 0x5638dcda99b0 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.849+0000 7efef3e00640 1 --2- 172.21.15.40:0/1766766448 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5638dcda99b0 0x5638dcda9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.850+0000 7efef3e00640 1 --2- 172.21.15.40:0/1766766448 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5638dcda99b0 0x5638dcda9d90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7302cd7aadf6d320 server_cookie=614b16ef0c729a21 in_seq=0 out_seq=0 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.850+0000 7efee1600640 10 client.6189 ms_handle_connect on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.850+0000 7efee1600640 1 -- 172.21.15.40:0/1766766448 <== mds.0 v2:172.21.15.40:6833/618018110 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5638dcd962e0 con 0x5638dcda99b0 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.850+0000 7efee1600640 10 client.6189 handle_command_reply: tid=0 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.850+0000 7efefe045740 1 client.6189 shutdown 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.850+0000 7efefe045740 1 -- 172.21.15.40:0/1766766448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd80270 msgr2=0x5638dcd9ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.850+0000 7efefe045740 1 --2- 172.21.15.40:0/1766766448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd80270 0x5638dcd9ece0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7efedc00cdf0 tx=0x7efedc01d040 comp rx=0 tx=0).stop 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efefe045740 1 -- 172.21.15.40:0/1766766448 shutdown_connections 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efefe045740 1 --2- 172.21.15.40:0/1766766448 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5638dcda99b0 0x5638dcda9d90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efefe045740 1 --2- 172.21.15.40:0/1766766448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5638dcd91e80 0x5638dcd7ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efefe045740 1 --2- 172.21.15.40:0/1766766448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dcd80270 0x5638dcd9ece0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efefe045740 1 --2- 172.21.15.40:0/1766766448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638dcd557b0 0x5638dce14b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efefe045740 1 -- 172.21.15.40:0/1766766448 >> 172.21.15.40:0/1766766448 conn(0x5638dcce9d10 msgr2=0x5638dcd6ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:23.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efee1600640 0 client.6189 ms_handle_reset on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:23.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efefe045740 1 -- 172.21.15.40:0/1766766448 shutdown_connections 2024-04-26T23:04:23.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efefe045740 1 -- 172.21.15.40:0/1766766448 wait complete. 2024-04-26T23:04:23.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efefe045740 20 client.6189 trim_cache size 0 max 16384 2024-04-26T23:04:23.857 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:04:23.857 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:04:23.857 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:04:23.857 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6036, 2024-04-26T23:04:23.857 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:04:23.857 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:04:23.858 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:04:23.858 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 6.7765530590000003, 2024-04-26T23:04:23.858 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 35, 2024-04-26T23:04:23.858 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 52, 2024-04-26T23:04:23.858 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 52, 2024-04-26T23:04:23.858 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 12.174859641999999 2024-04-26T23:04:23.858 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:04:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efef9c00640 1 -- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efed006c8f0 msgr2=0x7efed006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efef9c00640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efed006c8f0 0x7efed006edb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7efee4006fd0 tx=0x7efee4008040 comp rx=0 tx=0).stop 2024-04-26T23:04:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efef9c00640 1 -- 172.21.15.40:0/1323646047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef40d08d0 msgr2=0x7efef415cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.851+0000 7efef9c00640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef40d08d0 0x7efef415cd80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7efedc00c430 tx=0x7efedc00c8f0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.852+0000 7efef9c00640 1 -- 172.21.15.40:0/1323646047 shutdown_connections 2024-04-26T23:04:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.852+0000 7efef9c00640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efed006c8f0 0x7efed006edb0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.852+0000 7efef9c00640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efef40d39f0 0x7efef415eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.852+0000 7efef9c00640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef40d08d0 0x7efef415cd80 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.852+0000 7efef9c00640 1 --2- 172.21.15.40:0/1323646047 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efef40b6220 0x7efef415c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.852+0000 7efef9c00640 1 -- 172.21.15.40:0/1323646047 >> 172.21.15.40:0/1323646047 conn(0x7efef40d4d90 msgr2=0x7efef40b97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.852+0000 7efef9c00640 1 -- 172.21.15.40:0/1323646047 shutdown_connections 2024-04-26T23:04:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:23.852+0000 7efef9c00640 1 -- 172.21.15.40:0/1323646047 wait complete. 2024-04-26T23:04:23.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-26T23:04:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.074+0000 7f21fa800640 1 Processor -- start 2024-04-26T23:04:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.074+0000 7f21fa800640 1 -- start start 2024-04-26T23:04:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.076+0000 7f21fa800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21f40cffa0 0x7f21f40d1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.076+0000 7f21fa800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21f40d0f20 con 0x7f21f40d0a90 2024-04-26T23:04:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.076+0000 7f21fa800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21f40cde20 con 0x7f21f41608f0 2024-04-26T23:04:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.076+0000 7f21fa800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f21f40cea30 con 0x7f21f40cffa0 2024-04-26T23:04:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.076+0000 7f21f3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f21f40d0a90 0x7f21f40b2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58000/0 (socket says 172.21.15.40:58000) 2024-04-26T23:04:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.076+0000 7f21f3e00640 1 -- 172.21.15.40:0/1181454349 learned_addr learned my addr 172.21.15.40:0/1181454349 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.076+0000 7f21f2a00640 1 --2- 172.21.15.40:0/1181454349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21f40cffa0 0x7f21f40d1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.076+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3014056323 0 0) 0x7f21f40d0f20 con 0x7f21f40d0a90 2024-04-26T23:04:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21d8003520 con 0x7f21f40d0a90 2024-04-26T23:04:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3379446534 0 0) 0x7f21d8003520 con 0x7f21f40d0a90 2024-04-26T23:04:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21f40cffa0 msgr2=0x7f21f40d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 --2- 172.21.15.40:0/1181454349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21f40cffa0 0x7f21f40d1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 >> v1:172.21.15.103:6789/0 conn(0x7f21f41608f0 legacy=0x7f21f40cf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f40ce100 con 0x7f21f40d0a90 2024-04-26T23:04:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f21ec001670 con 0x7f21f40d0a90 2024-04-26T23:04:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 >> v1:172.21.15.40:6789/0 conn(0x7f21f40d0a90 legacy=0x7f21f40b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 --2- 172.21.15.40:0/1181454349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d80039e0 0x7f21d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 --2- 172.21.15.40:0/1181454349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21f40cffa0 0x7f21d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 --2- 172.21.15.40:0/1181454349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21d8004d40 0x7f21d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21f40cea30 con 0x7f21d80039e0 2024-04-26T23:04:24.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f21d800ba70 con 0x7f21d8004d40 2024-04-26T23:04:24.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f21d800bc70 con 0x7f21f40cffa0 2024-04-26T23:04:24.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.077+0000 7f21f3400640 1 --2- 172.21.15.40:0/1181454349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d80039e0 0x7f21d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:24.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.078+0000 7f21f3e00640 1 --2- 172.21.15.40:0/1181454349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21d8004d40 0x7f21d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:24.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.078+0000 7f21f2a00640 1 --2- 172.21.15.40:0/1181454349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21f40cffa0 0x7f21d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:24.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.078+0000 7f21f3400640 1 -- 172.21.15.40:0/1181454349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21f40cffa0 msgr2=0x7f21d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:24.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.078+0000 7f21f3400640 1 --2- 172.21.15.40:0/1181454349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21f40cffa0 0x7f21d8004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:24.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.078+0000 7f21f3400640 1 -- 172.21.15.40:0/1181454349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21d8004d40 msgr2=0x7f21d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:24.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.078+0000 7f21f3400640 1 --2- 172.21.15.40:0/1181454349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21d8004d40 0x7f21d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:24.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.078+0000 7f21f3400640 1 -- 172.21.15.40:0/1181454349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d8003520 con 0x7f21d80039e0 2024-04-26T23:04:24.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.079+0000 7f21f3400640 1 --2- 172.21.15.40:0/1181454349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d80039e0 0x7f21d8003e00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f21e4001e20 tx=0x7f21d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ee8989e758cbd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:24.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.079+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4001a70 con 0x7f21d80039e0 2024-04-26T23:04:24.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.079+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21e402ae40 con 0x7f21d80039e0 2024-04-26T23:04:24.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.079+0000 7f21f2000640 1 -- 172.21.15.40:0/1181454349 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4003bb0 con 0x7f21d80039e0 2024-04-26T23:04:24.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.080+0000 7f21fa800640 1 -- 172.21.15.40:0/1181454349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d80039e0 msgr2=0x7f21d8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:24.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.080+0000 7f21fa800640 1 --2- 172.21.15.40:0/1181454349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d80039e0 0x7f21d8003e00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f21e4001e20 tx=0x7f21d8005830 comp rx=0 tx=0).stop 2024-04-26T23:04:24.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.080+0000 7f21f3e00640 1 -- 172.21.15.40:0/1181454349 reap_dead start 2024-04-26T23:04:24.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.080+0000 7f21fa800640 1 -- 172.21.15.40:0/1181454349 shutdown_connections 2024-04-26T23:04:24.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.080+0000 7f21fa800640 1 -- 172.21.15.40:0/1181454349 >> 172.21.15.40:0/1181454349 conn(0x7f21f40d2ec0 msgr2=0x7f21f40d32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:24.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.080+0000 7f21fa800640 1 -- 172.21.15.40:0/1181454349 shutdown_connections 2024-04-26T23:04:24.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.080+0000 7f21fa800640 1 -- 172.21.15.40:0/1181454349 wait complete. 2024-04-26T23:04:24.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.082+0000 7f21fa800640 1 Processor -- start 2024-04-26T23:04:24.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.082+0000 7f21fa800640 1 -- start start 2024-04-26T23:04:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.083+0000 7f21fa800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21d80039e0 0x7f21f4158390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.083+0000 7f21fa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f41f4ff0 0x7f21f40d1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.083+0000 7f21fa800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f40d1670 0x7f21f40c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.083+0000 7f21fa800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21f40ceec0 con 0x7f21f41f4ff0 2024-04-26T23:04:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.083+0000 7f21fa800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f21f40ce6c0 con 0x7f21f40d1670 2024-04-26T23:04:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.083+0000 7f21fa800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f21f40cda40 con 0x7f21d80039e0 2024-04-26T23:04:24.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.083+0000 7f21f2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f41f4ff0 0x7f21f40d1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:24.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.083+0000 7f21f2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f41f4ff0 0x7f21f40d1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42560/0 (socket says 172.21.15.40:42560) 2024-04-26T23:04:24.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.083+0000 7f21f2a00640 1 -- 172.21.15.40:0/1076396762 learned_addr learned my addr 172.21.15.40:0/1076396762 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:24.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.084+0000 7f21f3400640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21d80039e0 0x7f21f4158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:24.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.084+0000 7f21f3e00640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f40d1670 0x7f21f40c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:24.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.084+0000 7f21f2a00640 1 -- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21d80039e0 msgr2=0x7f21f4158390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:24.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.084+0000 7f21f2a00640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21d80039e0 0x7f21f4158390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:24.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.084+0000 7f21f2a00640 1 -- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f40d1670 msgr2=0x7f21f40c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:24.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.084+0000 7f21f2a00640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f40d1670 0x7f21f40c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:24.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.084+0000 7f21f2a00640 1 -- 172.21.15.40:0/1076396762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f41588d0 con 0x7f21f41f4ff0 2024-04-26T23:04:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.084+0000 7f21f3e00640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f40d1670 0x7f21f40c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.084+0000 7f21f2a00640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f41f4ff0 0x7f21f40d1130 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f21e0010860 tx=0x7f21e0010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.085+0000 7f21eb400640 1 -- 172.21.15.40:0/1076396762 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e000f040 con 0x7f21f41f4ff0 2024-04-26T23:04:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.085+0000 7f21eb400640 1 -- 172.21.15.40:0/1076396762 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21e000eb60 con 0x7f21f41f4ff0 2024-04-26T23:04:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.085+0000 7f21eb400640 1 -- 172.21.15.40:0/1076396762 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e001c400 con 0x7f21f41f4ff0 2024-04-26T23:04:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.085+0000 7f21fa800640 1 -- 172.21.15.40:0/1076396762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f406d650 con 0x7f21f41f4ff0 2024-04-26T23:04:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.085+0000 7f21fa800640 1 -- 172.21.15.40:0/1076396762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f406dc10 con 0x7f21f41f4ff0 2024-04-26T23:04:24.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.086+0000 7f21fa800640 1 -- 172.21.15.40:0/1076396762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f40694f0 con 0x7f21f41f4ff0 2024-04-26T23:04:24.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.087+0000 7f21eb400640 1 -- 172.21.15.40:0/1076396762 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f21e001c5a0 con 0x7f21f41f4ff0 2024-04-26T23:04:24.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.088+0000 7f21eb400640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f21d406c920 0x7f21d406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:24.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.088+0000 7f21eb400640 1 -- 172.21.15.40:0/1076396762 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f21e008c2e0 con 0x7f21f41f4ff0 2024-04-26T23:04:24.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.089+0000 7f21f3400640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f21d406c920 0x7f21d406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.090+0000 7f21f3400640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f21d406c920 0x7f21d406ede0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f21e4032000 tx=0x7f21e402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:24.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.094+0000 7f21eb400640 1 -- 172.21.15.40:0/1076396762 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f21f41588d0 con 0x7f21f41f4ff0 2024-04-26T23:04:24.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.346+0000 7f21fa800640 1 -- 172.21.15.40:0/1076396762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f21f40c6600 con 0x7f21f41f4ff0 2024-04-26T23:04:24.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.347+0000 7f21eb400640 1 -- 172.21.15.40:0/1076396762 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) ==== 53+0+73 (secure 0 0 0) 0x7f21e0018030 con 0x7f21f41f4ff0 2024-04-26T23:04:24.347 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-26T23:04:24.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.350+0000 7f21fa800640 1 -- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f21d406c920 msgr2=0x7f21d406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.350+0000 7f21fa800640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f21d406c920 0x7f21d406ede0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f21e4032000 tx=0x7f21e402e000 comp rx=0 tx=0).stop 2024-04-26T23:04:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.350+0000 7f21fa800640 1 -- 172.21.15.40:0/1076396762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f41f4ff0 msgr2=0x7f21f40d1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.350+0000 7f21fa800640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f41f4ff0 0x7f21f40d1130 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f21e0010860 tx=0x7f21e0010d20 comp rx=0 tx=0).stop 2024-04-26T23:04:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.350+0000 7f21fa800640 1 -- 172.21.15.40:0/1076396762 shutdown_connections 2024-04-26T23:04:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.350+0000 7f21fa800640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f21d406c920 0x7f21d406ede0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.350+0000 7f21fa800640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21f40d1670 0x7f21f40c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.350+0000 7f21fa800640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21f41f4ff0 0x7f21f40d1130 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.350+0000 7f21fa800640 1 --2- 172.21.15.40:0/1076396762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21d80039e0 0x7f21f4158390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.350+0000 7f21fa800640 1 -- 172.21.15.40:0/1076396762 >> 172.21.15.40:0/1076396762 conn(0x7f21f40d2ec0 msgr2=0x7f21f415f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.350+0000 7f21fa800640 1 -- 172.21.15.40:0/1076396762 shutdown_connections 2024-04-26T23:04:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24.350+0000 7f21fa800640 1 -- 172.21.15.40:0/1076396762 wait complete. 2024-04-26T23:04:24.371 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-26T23:04:24.371 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:04:24.372 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-04-26T23:04:24.372 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T23:04:24.372 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T23:04:24.372 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-26T23:04:24.372 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-26T23:04:24.372 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:04:24.372 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:04:24.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:24.372 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-04-26T23:04:24.427 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:04:24.427 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:04:24.428 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:04:24.428 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:04:24.428 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-26T23:04:24.428 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:04:24.428 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:04:24.428 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:04:24.428 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-04-26T23:04:24.428 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:04:24.428 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:04:24.428 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-04-26T23:04:24.428 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:04:24.429 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-04-26T23:04:24.429 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-04-26T23:04:24.429 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T23:04:24.429 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42450sec preferred_lft 42450sec 2024-04-26T23:04:24.429 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-26T23:04:24.429 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:04:24.429 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:04:24.429 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-04-26T23:04:24.429 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-04-26T23:04:24.429 INFO:teuthology.orchestra.run.smithi040.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-26T23:04:24.429 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 8e:8e:79:d7:cf:5d brd ff:ff:ff:ff:ff:ff 2024-04-26T23:04:24.429 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-26T23:04:24.429 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:04:24.430 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::4ce7:9bff:fe51:b1/64 scope link 2024-04-26T23:04:24.430 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:04:24.430 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:04:24.430 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether be:f1:28:e1:99:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T23:04:24.430 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::bcf1:28ff:fee1:9921/64 scope link 2024-04-26T23:04:24.430 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:04:24.430 INFO:teuthology.orchestra.run.smithi040.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:04:24.430 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 8e:8e:79:d7:cf:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-26T23:04:24.430 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::8c8e:79ff:fed7:cf5d/64 scope link 2024-04-26T23:04:24.430 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:04:24.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:24.432 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:04:24.517 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-04-26T23:04:24.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:24.521 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-04-26T23:04:24.577 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-04-26T23:04:24.577 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:04:24.577 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T23:04:24.578 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T23:04:24.578 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:04:24.578 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:04:24.579 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:04:24.579 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T23:04:24.579 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T23:04:24.579 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T23:04:24.579 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:04:24.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:04:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:04:24.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:24.809 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:04:24.859 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T23:04:24.859 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-26T23:04:24.860 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:04:24.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:24.861 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-04-26T23:04:24.947 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:04:24.947 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T23:04:25.003 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:04:25.003 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:04:25.004 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:04:25.060 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-26T23:04:25.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:25.061 DEBUG:teuthology.orchestra.run.smithi040:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:04:25.198 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:04:25.198 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:04:25.199 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:04:25.220 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:04:25.220 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:04:25.220 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:04:25.220 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:04:25.220 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:04:25.247 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:04:25.247 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:04:25.247 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:04:25.247 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:04:25.247 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:04:25.247 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-26T23:04:25.248 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-26T23:04:25.248 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=815936ab-d4ac-471e-b069-1a0dc70078ee" will pass to kernel 2024-04-26T23:04:25.250 INFO:teuthology.orchestra.run.smithi040.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T23:04:25.250 INFO:teuthology.orchestra.run.smithi040.stdout: You just mounted a file system that supports labels which does not 2024-04-26T23:04:25.250 INFO:teuthology.orchestra.run.smithi040.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T23:04:25.251 INFO:teuthology.orchestra.run.smithi040.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T23:04:25.251 INFO:teuthology.orchestra.run.smithi040.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T23:04:25.254 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T23:04:25.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:25.255 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:04:25.332 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:04:25.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:04:25.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:04:25.467 INFO:teuthology.orchestra.run.smithi040.stdout:client5356 2024-04-26T23:04:25.468 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:04:25.468 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:04:25.521 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-26T23:04:25.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.730+0000 7fea90800640 1 Processor -- start 2024-04-26T23:04:25.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.730+0000 7fea90800640 1 -- start start 2024-04-26T23:04:25.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.732+0000 7fea90800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea88156460 0x7fea880cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:25.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.732+0000 7fea90800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fea880cba10 0x7fea8806f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:25.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.732+0000 7fea90800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fea8806fcd0 0x7fea8815f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:25.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.732+0000 7fea90800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fea88156840 con 0x7fea8806fcd0 2024-04-26T23:04:25.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.732+0000 7fea90800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fea880cbf20 con 0x7fea880cba10 2024-04-26T23:04:25.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.732+0000 7fea90800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fea880cc200 con 0x7fea88156460 2024-04-26T23:04:25.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.733+0000 7fea8e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fea8806fcd0 0x7fea8815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:25.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.733+0000 7fea8e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fea8806fcd0 0x7fea8815f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56710/0 (socket says 172.21.15.40:56710) 2024-04-26T23:04:25.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.733+0000 7fea8e800640 1 -- 172.21.15.40:0/3558558570 learned_addr learned my addr 172.21.15.40:0/3558558570 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:25.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.733+0000 7fea8d400640 1 --2- 172.21.15.40:0/3558558570 >> v2:172.21.15.103:3300/0 conn(0x7fea880cba10 0x7fea8806f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:25.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.733+0000 7fea8de00640 1 --2- 172.21.15.40:0/3558558570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea88156460 0x7fea880cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:25.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.733+0000 7fea8e800640 1 -- 172.21.15.40:0/3558558570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea88156460 msgr2=0x7fea880cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:25.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.733+0000 7fea8e800640 1 --2- 172.21.15.40:0/3558558570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea88156460 0x7fea880cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:25.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.733+0000 7fea8e800640 1 -- 172.21.15.40:0/3558558570 >> v2:172.21.15.103:3300/0 conn(0x7fea880cba10 msgr2=0x7fea8806f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:25.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.733+0000 7fea8e800640 1 --2- 172.21.15.40:0/3558558570 >> v2:172.21.15.103:3300/0 conn(0x7fea880cba10 0x7fea8806f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:25.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.733+0000 7fea8e800640 1 -- 172.21.15.40:0/3558558570 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea880cc4e0 con 0x7fea8806fcd0 2024-04-26T23:04:25.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.735+0000 7fea8e800640 1 --2- 172.21.15.40:0/3558558570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea8806fcd0 0x7fea8815f980 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fea88070320 tx=0x7fea8402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368d368387b7c7c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:25.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.735+0000 7fea8ca00640 1 -- 172.21.15.40:0/3558558570 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea84036070 con 0x7fea8806fcd0 2024-04-26T23:04:25.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.735+0000 7fea8ca00640 1 -- 172.21.15.40:0/3558558570 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea84031600 con 0x7fea8806fcd0 2024-04-26T23:04:25.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.735+0000 7fea8ca00640 1 -- 172.21.15.40:0/3558558570 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea840319d0 con 0x7fea8806fcd0 2024-04-26T23:04:25.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.736+0000 7fea90800640 1 -- 172.21.15.40:0/3558558570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea8806fcd0 msgr2=0x7fea8815f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.736+0000 7fea90800640 1 --2- 172.21.15.40:0/3558558570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea8806fcd0 0x7fea8815f980 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fea88070320 tx=0x7fea8402a6a0 comp rx=0 tx=0).stop 2024-04-26T23:04:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.736+0000 7fea90800640 1 -- 172.21.15.40:0/3558558570 shutdown_connections 2024-04-26T23:04:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.736+0000 7fea90800640 1 --2- 172.21.15.40:0/3558558570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea8806fcd0 0x7fea8815f980 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.736+0000 7fea90800640 1 --2- 172.21.15.40:0/3558558570 >> v2:172.21.15.103:3300/0 conn(0x7fea880cba10 0x7fea8806f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.736+0000 7fea90800640 1 --2- 172.21.15.40:0/3558558570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea88156460 0x7fea880cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.736+0000 7fea90800640 1 -- 172.21.15.40:0/3558558570 >> 172.21.15.40:0/3558558570 conn(0x7fea880d0be0 msgr2=0x7fea880d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.736+0000 7fea90800640 1 -- 172.21.15.40:0/3558558570 shutdown_connections 2024-04-26T23:04:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.736+0000 7fea90800640 1 -- 172.21.15.40:0/3558558570 wait complete. 2024-04-26T23:04:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.738+0000 7fea90800640 1 Processor -- start 2024-04-26T23:04:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.738+0000 7fea90800640 1 -- start start 2024-04-26T23:04:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.739+0000 7fea90800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea8806fcd0 0x7fea8814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.739+0000 7fea90800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea880cba10 0x7fea88150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.739+0000 7fea90800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea88156460 0x7fea880c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.739+0000 7fea90800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fea8806e260 con 0x7fea880cba10 2024-04-26T23:04:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.739+0000 7fea90800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea88072700 con 0x7fea88156460 2024-04-26T23:04:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.739+0000 7fea90800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fea8806e3e0 con 0x7fea8806fcd0 2024-04-26T23:04:25.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.740+0000 7fea8d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea880cba10 0x7fea88150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:25.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.740+0000 7fea8d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea880cba10 0x7fea88150b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56718/0 (socket says 172.21.15.40:56718) 2024-04-26T23:04:25.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.740+0000 7fea8d400640 1 -- 172.21.15.40:0/1874985688 learned_addr learned my addr 172.21.15.40:0/1874985688 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:25.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.740+0000 7fea8e800640 1 --2- 172.21.15.40:0/1874985688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea88156460 0x7fea880c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:25.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.740+0000 7fea8d400640 1 -- 172.21.15.40:0/1874985688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea8806fcd0 msgr2=0x7fea8814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:25.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.740+0000 7fea8d400640 1 --2- 172.21.15.40:0/1874985688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea8806fcd0 0x7fea8814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:25.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.740+0000 7fea8d400640 1 -- 172.21.15.40:0/1874985688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea88156460 msgr2=0x7fea880c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:25.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.740+0000 7fea8d400640 1 --2- 172.21.15.40:0/1874985688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea88156460 0x7fea880c9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:25.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.740+0000 7fea8d400640 1 -- 172.21.15.40:0/1874985688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8814faa0 con 0x7fea880cba10 2024-04-26T23:04:25.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.740+0000 7fea8d400640 1 --2- 172.21.15.40:0/1874985688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea880cba10 0x7fea88150b00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fea7800e860 tx=0x7fea7800ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:25.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.741+0000 7fea82000640 1 -- 172.21.15.40:0/1874985688 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea78017070 con 0x7fea880cba10 2024-04-26T23:04:25.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.741+0000 7fea82000640 1 -- 172.21.15.40:0/1874985688 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea7800c940 con 0x7fea880cba10 2024-04-26T23:04:25.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.741+0000 7fea82000640 1 -- 172.21.15.40:0/1874985688 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7800cd10 con 0x7fea880cba10 2024-04-26T23:04:25.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.741+0000 7fea90800640 1 -- 172.21.15.40:0/1874985688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea8814fd40 con 0x7fea880cba10 2024-04-26T23:04:25.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.741+0000 7fea90800640 1 -- 172.21.15.40:0/1874985688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea88150300 con 0x7fea880cba10 2024-04-26T23:04:25.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.742+0000 7fea90800640 1 -- 172.21.15.40:0/1874985688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea880694f0 con 0x7fea880cba10 2024-04-26T23:04:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.743+0000 7fea82000640 1 -- 172.21.15.40:0/1874985688 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fea7801a070 con 0x7fea880cba10 2024-04-26T23:04:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.744+0000 7fea82000640 1 --2- 172.21.15.40:0/1874985688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea6c06c8f0 0x7fea6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.744+0000 7fea82000640 1 -- 172.21.15.40:0/1874985688 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fea7808c100 con 0x7fea880cba10 2024-04-26T23:04:25.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.745+0000 7fea8de00640 1 --2- 172.21.15.40:0/1874985688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea6c06c8f0 0x7fea6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:25.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.746+0000 7fea8de00640 1 --2- 172.21.15.40:0/1874985688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea6c06c8f0 0x7fea6c06edb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fea7c006a90 tx=0x7fea7c006a20 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:25.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.749+0000 7fea82000640 1 -- 172.21.15.40:0/1874985688 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fea7805a300 con 0x7fea880cba10 2024-04-26T23:04:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:25.997+0000 7fea90800640 1 -- 172.21.15.40:0/1874985688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fea880b0980 con 0x7fea880cba10 2024-04-26T23:04:26.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.139+0000 7fea82000640 1 -- 172.21.15.40:0/1874985688 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v36) ==== 105+0+0 (secure 0 0 0) 0x7fea78059c00 con 0x7fea880cba10 2024-04-26T23:04:26.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.143+0000 7fea90800640 1 -- 172.21.15.40:0/1874985688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea6c06c8f0 msgr2=0x7fea6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:26.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.143+0000 7fea90800640 1 --2- 172.21.15.40:0/1874985688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea6c06c8f0 0x7fea6c06edb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fea7c006a90 tx=0x7fea7c006a20 comp rx=0 tx=0).stop 2024-04-26T23:04:26.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.143+0000 7fea90800640 1 -- 172.21.15.40:0/1874985688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea880cba10 msgr2=0x7fea88150b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:26.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.143+0000 7fea90800640 1 --2- 172.21.15.40:0/1874985688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea880cba10 0x7fea88150b00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fea7800e860 tx=0x7fea7800ed20 comp rx=0 tx=0).stop 2024-04-26T23:04:26.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.143+0000 7fea90800640 1 -- 172.21.15.40:0/1874985688 shutdown_connections 2024-04-26T23:04:26.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.143+0000 7fea90800640 1 --2- 172.21.15.40:0/1874985688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fea6c06c8f0 0x7fea6c06edb0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:26.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.143+0000 7fea90800640 1 --2- 172.21.15.40:0/1874985688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea88156460 0x7fea880c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:26.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.143+0000 7fea90800640 1 --2- 172.21.15.40:0/1874985688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea880cba10 0x7fea88150b00 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:26.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.143+0000 7fea90800640 1 --2- 172.21.15.40:0/1874985688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea8806fcd0 0x7fea8814efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:26.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.143+0000 7fea90800640 1 -- 172.21.15.40:0/1874985688 >> 172.21.15.40:0/1874985688 conn(0x7fea880d0be0 msgr2=0x7fea88072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:26.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.144+0000 7fea90800640 1 -- 172.21.15.40:0/1874985688 shutdown_connections 2024-04-26T23:04:26.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.144+0000 7fea90800640 1 -- 172.21.15.40:0/1874985688 wait complete. 2024-04-26T23:04:26.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:04:26.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.361+0000 7f63bf200640 1 Processor -- start 2024-04-26T23:04:26.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.361+0000 7f63bf200640 1 -- start start 2024-04-26T23:04:26.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.363+0000 7f63bf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63c00af7a0 0x7f63c00afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:26.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.363+0000 7f63bf200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c00d0820 con 0x7f63c0071fe0 2024-04-26T23:04:26.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.363+0000 7f63bf200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c01614c0 con 0x7f63c015ce60 2024-04-26T23:04:26.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.363+0000 7f63bf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f63c00cc450 con 0x7f63c00af7a0 2024-04-26T23:04:26.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.363+0000 7f63be800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f63c0071fe0 0x7f63c0073760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58570/0 (socket says 172.21.15.40:58570) 2024-04-26T23:04:26.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.363+0000 7f63be800640 1 -- 172.21.15.40:0/3126530667 learned_addr learned my addr 172.21.15.40:0/3126530667 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:26.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.363+0000 7f63b5400640 1 --2- 172.21.15.40:0/3126530667 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63c00af7a0 0x7f63c00afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:26.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 158162316 0 0) 0x7f63c00d0820 con 0x7f63c0071fe0 2024-04-26T23:04:26.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63a4003520 con 0x7f63c0071fe0 2024-04-26T23:04:26.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 449014006 0 0) 0x7f63a4003520 con 0x7f63c0071fe0 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63c00af7a0 msgr2=0x7f63c00afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 --2- 172.21.15.40:0/3126530667 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63c00af7a0 0x7f63c00afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 >> v1:172.21.15.103:6789/0 conn(0x7f63c015ce60 legacy=0x7f63c00ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c0161760 con 0x7f63c0071fe0 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f63b8001670 con 0x7f63c0071fe0 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 >> v1:172.21.15.40:6789/0 conn(0x7f63c0071fe0 legacy=0x7f63c0073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 --2- 172.21.15.40:0/3126530667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63a40039e0 0x7f63a4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 --2- 172.21.15.40:0/3126530667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63a4004800 0x7f63a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 --2- 172.21.15.40:0/3126530667 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63c00af7a0 0x7f63a4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63c00cc450 con 0x7f63a4004800 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63a400baf0 con 0x7f63a40039e0 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.364+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f63a400bcf0 con 0x7f63c00af7a0 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.365+0000 7f63b5400640 1 --2- 172.21.15.40:0/3126530667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63a4004800 0x7f63a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.365+0000 7f63bde00640 1 --2- 172.21.15.40:0/3126530667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63a40039e0 0x7f63a4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:26.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.365+0000 7f63b5400640 1 -- 172.21.15.40:0/3126530667 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63c00af7a0 msgr2=0x7f63a4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:26.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.365+0000 7f63be800640 1 --2- 172.21.15.40:0/3126530667 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63c00af7a0 0x7f63a4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:26.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.365+0000 7f63b5400640 1 --2- 172.21.15.40:0/3126530667 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63c00af7a0 0x7f63a4005200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:26.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.365+0000 7f63b5400640 1 -- 172.21.15.40:0/3126530667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63a40039e0 msgr2=0x7f63a4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:26.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.365+0000 7f63b5400640 1 --2- 172.21.15.40:0/3126530667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63a40039e0 0x7f63a4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:26.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.365+0000 7f63b5400640 1 -- 172.21.15.40:0/3126530667 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a4003520 con 0x7f63a4004800 2024-04-26T23:04:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.366+0000 7f63b5400640 1 --2- 172.21.15.40:0/3126530667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63a4004800 0x7f63a4004c80 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009fc0 tx=0x7f63a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6e242c4a5ba557 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.367+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a8009c10 con 0x7f63a4004800 2024-04-26T23:04:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.367+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63a8036070 con 0x7f63a4004800 2024-04-26T23:04:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.367+0000 7f63bd400640 1 -- 172.21.15.40:0/3126530667 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a8031880 con 0x7f63a4004800 2024-04-26T23:04:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.367+0000 7f63bf200640 1 -- 172.21.15.40:0/3126530667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63a4004800 msgr2=0x7f63a4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.367+0000 7f63bf200640 1 --2- 172.21.15.40:0/3126530667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63a4004800 0x7f63a4004c80 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009fc0 tx=0x7f63a4005830 comp rx=0 tx=0).stop 2024-04-26T23:04:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.367+0000 7f63be800640 1 -- 172.21.15.40:0/3126530667 reap_dead start 2024-04-26T23:04:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.367+0000 7f63bf200640 1 -- 172.21.15.40:0/3126530667 shutdown_connections 2024-04-26T23:04:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.367+0000 7f63bf200640 1 -- 172.21.15.40:0/3126530667 >> 172.21.15.40:0/3126530667 conn(0x7f63c00d0a10 msgr2=0x7f63c00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.367+0000 7f63bf200640 1 -- 172.21.15.40:0/3126530667 shutdown_connections 2024-04-26T23:04:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.367+0000 7f63bf200640 1 -- 172.21.15.40:0/3126530667 wait complete. 2024-04-26T23:04:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.369+0000 7f63bf200640 1 Processor -- start 2024-04-26T23:04:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.369+0000 7f63bf200640 1 -- start start 2024-04-26T23:04:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.370+0000 7f63bf200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63a4004800 0x7f63c01510f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.370+0000 7f63bf200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c00c84f0 0x7f63c00c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.370+0000 7f63bf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63c015b180 0x7f63c015b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.370+0000 7f63bf200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63c00cc1f0 con 0x7f63c00c84f0 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.370+0000 7f63bf200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63c00cc070 con 0x7f63a4004800 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.370+0000 7f63bf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f63c0073e70 con 0x7f63c015b180 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.371+0000 7f63b5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c00c84f0 0x7f63c00c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.371+0000 7f63b5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c00c84f0 0x7f63c00c93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56736/0 (socket says 172.21.15.40:56736) 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.371+0000 7f63b5400640 1 -- 172.21.15.40:0/2032259486 learned_addr learned my addr 172.21.15.40:0/2032259486 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.371+0000 7f63bde00640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63a4004800 0x7f63c01510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.371+0000 7f63be800640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63c015b180 0x7f63c015b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.371+0000 7f63b5400640 1 -- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63c015b180 msgr2=0x7f63c015b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.371+0000 7f63b5400640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63c015b180 0x7f63c015b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.371+0000 7f63b5400640 1 -- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63a4004800 msgr2=0x7f63c01510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.371+0000 7f63b5400640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63a4004800 0x7f63c01510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.371+0000 7f63b5400640 1 -- 172.21.15.40:0/2032259486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c00c8cd0 con 0x7f63c00c84f0 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.371+0000 7f63bde00640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63a4004800 0x7f63c01510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.372+0000 7f63b5400640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c00c84f0 0x7f63c00c93a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009f60 tx=0x7f63a8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.372+0000 7f63b6a00640 1 -- 172.21.15.40:0/2032259486 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a8009c10 con 0x7f63c00c84f0 2024-04-26T23:04:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.372+0000 7f63b6a00640 1 -- 172.21.15.40:0/2032259486 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63a8003990 con 0x7f63c00c84f0 2024-04-26T23:04:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.372+0000 7f63b6a00640 1 -- 172.21.15.40:0/2032259486 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a8003d60 con 0x7f63c00c84f0 2024-04-26T23:04:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.372+0000 7f63bf200640 1 -- 172.21.15.40:0/2032259486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c015a6a0 con 0x7f63c00c84f0 2024-04-26T23:04:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.372+0000 7f63bf200640 1 -- 172.21.15.40:0/2032259486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c015bb40 con 0x7f63c00c84f0 2024-04-26T23:04:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.373+0000 7f63bf200640 1 -- 172.21.15.40:0/2032259486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c015a490 con 0x7f63c00c84f0 2024-04-26T23:04:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.374+0000 7f63b6a00640 1 -- 172.21.15.40:0/2032259486 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f63a8005d10 con 0x7f63c00c84f0 2024-04-26T23:04:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.375+0000 7f63b6a00640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f639c06c8f0 0x7f639c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.375+0000 7f63b6a00640 1 -- 172.21.15.40:0/2032259486 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f63a80acf50 con 0x7f63c00c84f0 2024-04-26T23:04:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.376+0000 7f63bde00640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f639c06c8f0 0x7f639c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.377+0000 7f63bde00640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f639c06c8f0 0x7f639c06edb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f63b00016b0 tx=0x7f63b0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:26.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.380+0000 7f63b6a00640 1 -- 172.21.15.40:0/2032259486 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f63a80443e0 con 0x7f63c00c84f0 2024-04-26T23:04:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.623+0000 7f63bf200640 1 -- 172.21.15.40:0/2032259486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63c0069370 con 0x7f63c00c84f0 2024-04-26T23:04:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.624+0000 7f63b6a00640 1 -- 172.21.15.40:0/2032259486 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5194 (secure 0 0 0) 0x7f63a807b1e0 con 0x7f63c00c84f0 2024-04-26T23:04:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:26.625 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/189004193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":189004193},{"type":"v1","addr":"172.21.15.103:6835","nonce":189004193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5284,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/670995452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":670995452},{"type":"v1","addr":"172.21.15.103:6837","nonce":670995452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:04:16.558591+0000","modified":"2024-04-26T23:04:26.146703+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1],"up":{"mds_0":6036,"mds_1":6027},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6027":{"gid":6027,"name":"a","rank":1,"incarnation":37,"state":"up:creating","state_seq":1,"addr":"172.21.15.40:6834/2304583125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2304583125},{"type":"v1","addr":"172.21.15.40:6834","nonce":2304583125}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6036":{"gid":6036,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/618018110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":618018110},{"type":"v1","addr":"172.21.15.40:6835","nonce":618018110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6036,"qdb_cluster":[6036]},"id":3}]} 2024-04-26T23:04:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 37 2024-04-26T23:04:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.626+0000 7f63bf200640 1 -- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f639c06c8f0 msgr2=0x7f639c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.626+0000 7f63bf200640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f639c06c8f0 0x7f639c06edb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f63b00016b0 tx=0x7f63b0009040 comp rx=0 tx=0).stop 2024-04-26T23:04:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.626+0000 7f63bf200640 1 -- 172.21.15.40:0/2032259486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c00c84f0 msgr2=0x7f63c00c93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.626+0000 7f63bf200640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c00c84f0 0x7f63c00c93a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009f60 tx=0x7f63a8009750 comp rx=0 tx=0).stop 2024-04-26T23:04:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.627+0000 7f63bf200640 1 -- 172.21.15.40:0/2032259486 shutdown_connections 2024-04-26T23:04:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.627+0000 7f63bf200640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f639c06c8f0 0x7f639c06edb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.627+0000 7f63bf200640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63c015b180 0x7f63c015b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.627+0000 7f63bf200640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c00c84f0 0x7f63c00c93a0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.627+0000 7f63bf200640 1 --2- 172.21.15.40:0/2032259486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63a4004800 0x7f63c01510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.627+0000 7f63bf200640 1 -- 172.21.15.40:0/2032259486 >> 172.21.15.40:0/2032259486 conn(0x7f63c00d0a10 msgr2=0x7f63c00b1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.627+0000 7f63bf200640 1 -- 172.21.15.40:0/2032259486 shutdown_connections 2024-04-26T23:04:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:26.627+0000 7f63bf200640 1 -- 172.21.15.40:0/2032259486 wait complete. 2024-04-26T23:04:27.047 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:04:16.558591+0000', 'modified': '2024-04-26T23:04:26.146703+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 6036, 'mds_1': 6027}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6027': {'gid': 6027, 'name': 'a', 'rank': 1, 'incarnation': 37, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.40:6834/2304583125', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 2304583125}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 2304583125}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_6036': {'gid': 6036, 'name': 'c', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/618018110', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 618018110}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 618018110}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6036, 'qdb_cluster': [6036]} 2024-04-26T23:04:27.047 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6027:up:creating 2024-04-26T23:04:28.024 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:04:28.024+0000 7f47d7c00640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T23:04:28.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:04:28.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.248+0000 7f5062400640 1 Processor -- start 2024-04-26T23:04:28.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.248+0000 7f5062400640 1 -- start start 2024-04-26T23:04:28.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.250+0000 7f5062400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f505c15d250 0x7f505c075b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.250+0000 7f5062400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f505c0d0480 0x7f505c0d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.250+0000 7f5062400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f505c0cb8b0 0x7f505c0cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.250+0000 7f5062400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f505c0761d0 con 0x7f505c0cb8b0 2024-04-26T23:04:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.250+0000 7f5062400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f505c0d0e00 con 0x7f505c0d0480 2024-04-26T23:04:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.250+0000 7f5062400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f505c0cca40 con 0x7f505c15d250 2024-04-26T23:04:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.250+0000 7f5061a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f505c0cb8b0 0x7f505c0cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.250+0000 7f5061a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f505c0cb8b0 0x7f505c0cbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56752/0 (socket says 172.21.15.40:56752) 2024-04-26T23:04:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.250+0000 7f5061a00640 1 -- 172.21.15.40:0/2884048034 learned_addr learned my addr 172.21.15.40:0/2884048034 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.251+0000 7f5061a00640 1 -- 172.21.15.40:0/2884048034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f505c15d250 msgr2=0x7f505c075b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.251+0000 7f5061a00640 1 --2- 172.21.15.40:0/2884048034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f505c15d250 0x7f505c075b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.251+0000 7f5061a00640 1 -- 172.21.15.40:0/2884048034 >> v2:172.21.15.103:3300/0 conn(0x7f505c0d0480 msgr2=0x7f505c0d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.251+0000 7f5061a00640 1 --2- 172.21.15.40:0/2884048034 >> v2:172.21.15.103:3300/0 conn(0x7f505c0d0480 0x7f505c0d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.251+0000 7f5061a00640 1 -- 172.21.15.40:0/2884048034 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505c0ccd20 con 0x7f505c0cb8b0 2024-04-26T23:04:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.252+0000 7f5061a00640 1 --2- 172.21.15.40:0/2884048034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f505c0cb8b0 0x7f505c0cbd40 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f5054009e80 tx=0x7f505402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228b296ab26c1f04 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.252+0000 7f505b400640 1 -- 172.21.15.40:0/2884048034 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5054036070 con 0x7f505c0cb8b0 2024-04-26T23:04:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.252+0000 7f505b400640 1 -- 172.21.15.40:0/2884048034 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5054031600 con 0x7f505c0cb8b0 2024-04-26T23:04:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.252+0000 7f505b400640 1 -- 172.21.15.40:0/2884048034 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50540319d0 con 0x7f505c0cb8b0 2024-04-26T23:04:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.252+0000 7f5062400640 1 -- 172.21.15.40:0/2884048034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f505c0cb8b0 msgr2=0x7f505c0cbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.252+0000 7f5062400640 1 --2- 172.21.15.40:0/2884048034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f505c0cb8b0 0x7f505c0cbd40 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f5054009e80 tx=0x7f505402a6f0 comp rx=0 tx=0).stop 2024-04-26T23:04:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.252+0000 7f5062400640 1 -- 172.21.15.40:0/2884048034 shutdown_connections 2024-04-26T23:04:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.252+0000 7f5062400640 1 --2- 172.21.15.40:0/2884048034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f505c0cb8b0 0x7f505c0cbd40 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.252+0000 7f5062400640 1 --2- 172.21.15.40:0/2884048034 >> v2:172.21.15.103:3300/0 conn(0x7f505c0d0480 0x7f505c0d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.252+0000 7f5062400640 1 --2- 172.21.15.40:0/2884048034 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f505c15d250 0x7f505c075b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.252+0000 7f5062400640 1 -- 172.21.15.40:0/2884048034 >> 172.21.15.40:0/2884048034 conn(0x7f505c0b2e70 msgr2=0x7f505c0b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.253+0000 7f5062400640 1 -- 172.21.15.40:0/2884048034 shutdown_connections 2024-04-26T23:04:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.253+0000 7f5062400640 1 -- 172.21.15.40:0/2884048034 wait complete. 2024-04-26T23:04:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.254+0000 7f5062400640 1 Processor -- start 2024-04-26T23:04:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.254+0000 7f5062400640 1 -- start start 2024-04-26T23:04:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.255+0000 7f5062400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f505c0cb8b0 0x7f505c0b6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.255+0000 7f5062400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f505c0d0480 0x7f505c0c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.255+0000 7f5062400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f505c15d250 0x7f505c159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.255+0000 7f5062400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f505c0b4170 con 0x7f505c0d0480 2024-04-26T23:04:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.255+0000 7f5062400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f505c0cd2e0 con 0x7f505c15d250 2024-04-26T23:04:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.255+0000 7f5062400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f505c06cfb0 con 0x7f505c0cb8b0 2024-04-26T23:04:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.256+0000 7f505be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f505c0d0480 0x7f505c0c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:28.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.256+0000 7f505be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f505c0d0480 0x7f505c0c8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56760/0 (socket says 172.21.15.40:56760) 2024-04-26T23:04:28.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.256+0000 7f505be00640 1 -- 172.21.15.40:0/811851119 learned_addr learned my addr 172.21.15.40:0/811851119 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:28.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.256+0000 7f5061000640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f505c0cb8b0 0x7f505c0b6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:28.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.256+0000 7f5061a00640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f505c15d250 0x7f505c159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:28.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.256+0000 7f505be00640 1 -- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f505c0cb8b0 msgr2=0x7f505c0b6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:28.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.256+0000 7f505be00640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f505c0cb8b0 0x7f505c0b6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:28.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.256+0000 7f505be00640 1 -- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f505c15d250 msgr2=0x7f505c159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:28.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.256+0000 7f505be00640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f505c15d250 0x7f505c159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:28.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.256+0000 7f505be00640 1 -- 172.21.15.40:0/811851119 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505c158800 con 0x7f505c0d0480 2024-04-26T23:04:28.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.256+0000 7f505be00640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f505c0d0480 0x7f505c0c8510 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f504800b4d0 tx=0x7f504800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:28.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.257+0000 7f5058c00640 1 -- 172.21.15.40:0/811851119 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048014070 con 0x7f505c0d0480 2024-04-26T23:04:28.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.257+0000 7f5058c00640 1 -- 172.21.15.40:0/811851119 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5048010870 con 0x7f505c0d0480 2024-04-26T23:04:28.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.257+0000 7f5058c00640 1 -- 172.21.15.40:0/811851119 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048010c60 con 0x7f505c0d0480 2024-04-26T23:04:28.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.257+0000 7f5062400640 1 -- 172.21.15.40:0/811851119 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f505c158ae0 con 0x7f505c0d0480 2024-04-26T23:04:28.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.257+0000 7f5062400640 1 -- 172.21.15.40:0/811851119 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f505c1590a0 con 0x7f505c0d0480 2024-04-26T23:04:28.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.258+0000 7f5062400640 1 -- 172.21.15.40:0/811851119 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f505c069a80 con 0x7f505c0d0480 2024-04-26T23:04:28.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.259+0000 7f5058c00640 1 -- 172.21.15.40:0/811851119 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5048017070 con 0x7f505c0d0480 2024-04-26T23:04:28.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.260+0000 7f5058c00640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f503c06c8f0 0x7f503c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:28.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.260+0000 7f5058c00640 1 -- 172.21.15.40:0/811851119 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f504808bad0 con 0x7f505c0d0480 2024-04-26T23:04:28.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.261+0000 7f5061000640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f503c06c8f0 0x7f503c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:28.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.262+0000 7f5061000640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f503c06c8f0 0x7f503c06edb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f505c15c160 tx=0x7f5050009060 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.268+0000 7f5058c00640 1 -- 172.21.15.40:0/811851119 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5048059680 con 0x7f505c0d0480 2024-04-26T23:04:28.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.515+0000 7f5062400640 1 -- 172.21.15.40:0/811851119 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f505c1594a0 con 0x7f505c0d0480 2024-04-26T23:04:28.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.516+0000 7f5058c00640 1 -- 172.21.15.40:0/811851119 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+5216 (secure 0 0 0) 0x7f504805d980 con 0x7f505c0d0480 2024-04-26T23:04:28.516 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:28.516 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/189004193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":189004193},{"type":"v1","addr":"172.21.15.103:6835","nonce":189004193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:04:16.558591+0000","modified":"2024-04-26T23:04:28.159788+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6036,"mds_1":6027,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"b","rank":2,"incarnation":39,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/670995452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":670995452},{"type":"v1","addr":"172.21.15.103:6837","nonce":670995452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6027":{"gid":6027,"name":"a","rank":1,"incarnation":37,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/2304583125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2304583125},{"type":"v1","addr":"172.21.15.40:6834","nonce":2304583125}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6036":{"gid":6036,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/618018110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":618018110},{"type":"v1","addr":"172.21.15.40:6835","nonce":618018110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6036,"qdb_cluster":[5284,6027,6036]},"id":3}]} 2024-04-26T23:04:28.517 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 40 2024-04-26T23:04:28.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.519+0000 7f5062400640 1 -- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f503c06c8f0 msgr2=0x7f503c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:28.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.519+0000 7f5062400640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f503c06c8f0 0x7f503c06edb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f505c15c160 tx=0x7f5050009060 comp rx=0 tx=0).stop 2024-04-26T23:04:28.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.519+0000 7f5062400640 1 -- 172.21.15.40:0/811851119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f505c0d0480 msgr2=0x7f505c0c8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:28.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.519+0000 7f5062400640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f505c0d0480 0x7f505c0c8510 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f504800b4d0 tx=0x7f504800b990 comp rx=0 tx=0).stop 2024-04-26T23:04:28.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.519+0000 7f5062400640 1 -- 172.21.15.40:0/811851119 shutdown_connections 2024-04-26T23:04:28.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.519+0000 7f5062400640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f503c06c8f0 0x7f503c06edb0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:28.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.519+0000 7f5062400640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f505c15d250 0x7f505c159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:28.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.519+0000 7f5062400640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f505c0d0480 0x7f505c0c8510 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:28.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.519+0000 7f5062400640 1 --2- 172.21.15.40:0/811851119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f505c0cb8b0 0x7f505c0b6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:28.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.519+0000 7f5062400640 1 -- 172.21.15.40:0/811851119 >> 172.21.15.40:0/811851119 conn(0x7f505c0b2e70 msgr2=0x7f505c15c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:28.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.519+0000 7f5062400640 1 -- 172.21.15.40:0/811851119 shutdown_connections 2024-04-26T23:04:28.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:28.519+0000 7f5062400640 1 -- 172.21.15.40:0/811851119 wait complete. 2024-04-26T23:04:28.940 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:04:16.558591+0000', 'modified': '2024-04-26T23:04:28.159788+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 6036, 'mds_1': 6027, 'mds_2': 5284}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5284': {'gid': 5284, 'name': 'b', 'rank': 2, 'incarnation': 39, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.103:6837/670995452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6836', 'nonce': 670995452}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 670995452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_6027': {'gid': 6027, 'name': 'a', 'rank': 1, 'incarnation': 37, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.40:6834/2304583125', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 2304583125}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 2304583125}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_6036': {'gid': 6036, 'name': 'c', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/618018110', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 618018110}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 618018110}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6036, 'qdb_cluster': [5284, 6027, 6036]} 2024-04-26T23:04:28.940 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T23:04:28.941 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T23:04:29.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.130+0000 7f3c97200640 1 Processor -- start 2024-04-26T23:04:29.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.130+0000 7f3c97200640 1 -- start start 2024-04-26T23:04:29.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.132+0000 7f3c97200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3c980d32e0 0x7f3c980d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.132+0000 7f3c97200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c98162c40 con 0x7f3c980b6220 2024-04-26T23:04:29.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.132+0000 7f3c97200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c980d2d80 con 0x7f3c980b8d20 2024-04-26T23:04:29.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.132+0000 7f3c97200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f3c980d3c80 con 0x7f3c980d32e0 2024-04-26T23:04:29.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.132+0000 7f3c95e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3c980b6220 0x7f3c980d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58576/0 (socket says 172.21.15.40:58576) 2024-04-26T23:04:29.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.132+0000 7f3c95e00640 1 -- 172.21.15.40:0/1611953957 learned_addr learned my addr 172.21.15.40:0/1611953957 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:29.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.132+0000 7f3c8fe00640 1 --2- 172.21.15.40:0/1611953957 >> v2:172.21.15.40:3300/0 conn(0x7f3c980d32e0 0x7f3c980d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.133+0000 7f3c8fe00640 1 -- 172.21.15.40:0/1611953957 >> v1:172.21.15.103:6789/0 conn(0x7f3c980b8d20 legacy=0x7f3c980b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.133+0000 7f3c8fe00640 1 -- 172.21.15.40:0/1611953957 >> v1:172.21.15.40:6789/0 conn(0x7f3c980b6220 legacy=0x7f3c980d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.133+0000 7f3c95400640 1 -- 172.21.15.40:0/1611953957 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 781247187 0 0) 0x7f3c98162c40 con 0x7f3c980b6220 2024-04-26T23:04:29.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.133+0000 7f3c8fe00640 1 -- 172.21.15.40:0/1611953957 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c980b7c80 con 0x7f3c980d32e0 2024-04-26T23:04:29.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.134+0000 7f3c8fe00640 1 --2- 172.21.15.40:0/1611953957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c980d32e0 0x7f3c980d3740 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f3c800097a0 tx=0x7f3c8002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f769c4fca4787a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.134+0000 7f3c95400640 1 -- 172.21.15.40:0/1611953957 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80037070 con 0x7f3c980d32e0 2024-04-26T23:04:29.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.134+0000 7f3c95400640 1 -- 172.21.15.40:0/1611953957 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c80032440 con 0x7f3c980d32e0 2024-04-26T23:04:29.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.134+0000 7f3c95400640 1 -- 172.21.15.40:0/1611953957 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80032810 con 0x7f3c980d32e0 2024-04-26T23:04:29.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.134+0000 7f3c97200640 1 -- 172.21.15.40:0/1611953957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c980d32e0 msgr2=0x7f3c980d3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.134+0000 7f3c97200640 1 --2- 172.21.15.40:0/1611953957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c980d32e0 0x7f3c980d3740 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f3c800097a0 tx=0x7f3c8002a1e0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.134+0000 7f3c97200640 1 -- 172.21.15.40:0/1611953957 shutdown_connections 2024-04-26T23:04:29.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.134+0000 7f3c97200640 1 --2- 172.21.15.40:0/1611953957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c980d32e0 0x7f3c980d3740 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.134+0000 7f3c97200640 1 -- 172.21.15.40:0/1611953957 >> 172.21.15.40:0/1611953957 conn(0x7f3c980d4d90 msgr2=0x7f3c980d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.135+0000 7f3c97200640 1 -- 172.21.15.40:0/1611953957 shutdown_connections 2024-04-26T23:04:29.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.135+0000 7f3c97200640 1 -- 172.21.15.40:0/1611953957 wait complete. 2024-04-26T23:04:29.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.136+0000 7f3c97200640 1 Processor -- start 2024-04-26T23:04:29.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.136+0000 7f3c97200640 1 -- start start 2024-04-26T23:04:29.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.137+0000 7f3c97200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c980b6220 0x7f3c9815d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.137+0000 7f3c97200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c980b8d20 0x7f3c980ce210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.137+0000 7f3c97200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c980d32e0 0x7f3c98152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.137+0000 7f3c97200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c980b8520 con 0x7f3c980b6220 2024-04-26T23:04:29.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.137+0000 7f3c97200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c980b8240 con 0x7f3c980b8d20 2024-04-26T23:04:29.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.137+0000 7f3c97200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c980d0aa0 con 0x7f3c980d32e0 2024-04-26T23:04:29.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.137+0000 7f3c95e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c980b6220 0x7f3c9815d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.138+0000 7f3c95e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c980b6220 0x7f3c9815d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56770/0 (socket says 172.21.15.40:56770) 2024-04-26T23:04:29.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.138+0000 7f3c95e00640 1 -- 172.21.15.40:0/3765196999 learned_addr learned my addr 172.21.15.40:0/3765196999 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.138+0000 7f3c8fe00640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c980b8d20 0x7f3c980ce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.138+0000 7f3c96800640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c980d32e0 0x7f3c98152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.138+0000 7f3c95e00640 1 -- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c980d32e0 msgr2=0x7f3c98152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.138+0000 7f3c95e00640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c980d32e0 0x7f3c98152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.138+0000 7f3c95e00640 1 -- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c980b8d20 msgr2=0x7f3c980ce210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.138+0000 7f3c95e00640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c980b8d20 0x7f3c980ce210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.138+0000 7f3c95e00640 1 -- 172.21.15.40:0/3765196999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c981f83b0 con 0x7f3c980b6220 2024-04-26T23:04:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.138+0000 7f3c95e00640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c980b6220 0x7f3c9815d9a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800c4d0 tx=0x7f3c8800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.139+0000 7f3c8e000640 1 -- 172.21.15.40:0/3765196999 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c88018070 con 0x7f3c980b6220 2024-04-26T23:04:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.139+0000 7f3c8e000640 1 -- 172.21.15.40:0/3765196999 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c88007500 con 0x7f3c980b6220 2024-04-26T23:04:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.139+0000 7f3c8e000640 1 -- 172.21.15.40:0/3765196999 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c88004a20 con 0x7f3c980b6220 2024-04-26T23:04:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.139+0000 7f3c97200640 1 -- 172.21.15.40:0/3765196999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c981f8690 con 0x7f3c980b6220 2024-04-26T23:04:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.139+0000 7f3c97200640 1 -- 172.21.15.40:0/3765196999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c981f8c50 con 0x7f3c980b6220 2024-04-26T23:04:29.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.141+0000 7f3c8e000640 1 -- 172.21.15.40:0/3765196999 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3c88004bc0 con 0x7f3c980b6220 2024-04-26T23:04:29.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.142+0000 7f3c8e000640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c7406c920 0x7f3c7406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.142+0000 7f3c8e000640 1 -- 172.21.15.40:0/3765196999 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f3c8808af90 con 0x7f3c980b6220 2024-04-26T23:04:29.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.143+0000 7f3c9f863740 1 Processor -- start 2024-04-26T23:04:29.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.143+0000 7f3c8fe00640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c7406c920 0x7f3c7406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.143+0000 7f3c9f863740 1 -- start start 2024-04-26T23:04:29.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.143+0000 7f3c9f863740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178a57b0 0x5615178a7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.143+0000 7f3c9f863740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178e1e80 0x5615178a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.143+0000 7f3c9f863740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178d0270 0x5615178d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.143+0000 7f3c9f863740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561517809600 con 0x5615178a57b0 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.144+0000 7f3c9f863740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5615177a6100 con 0x5615178d0270 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.144+0000 7f3c9f863740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5615178055c0 con 0x5615178e1e80 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.144+0000 7f3c95e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178a57b0 0x5615178a7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.144+0000 7f3c8fe00640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c7406c920 0x7f3c7406ede0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3c80034000 tx=0x7f3c8002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.144+0000 7f3c95e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178a57b0 0x5615178a7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56786/0 (socket says 172.21.15.40:56786) 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.144+0000 7f3c95e00640 1 -- 172.21.15.40:0/777374682 learned_addr learned my addr 172.21.15.40:0/777374682 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.144+0000 7f3c95e00640 1 -- 172.21.15.40:0/777374682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178e1e80 msgr2=0x5615178a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.144+0000 7f3c95e00640 1 --2- 172.21.15.40:0/777374682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178e1e80 0x5615178a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.144+0000 7f3c96800640 1 --2- 172.21.15.40:0/777374682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178d0270 0x5615178d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.144+0000 7f3c95e00640 1 -- 172.21.15.40:0/777374682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178d0270 msgr2=0x5615178d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.144+0000 7f3c95e00640 1 --2- 172.21.15.40:0/777374682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178d0270 0x5615178d06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.144+0000 7f3c95e00640 1 -- 172.21.15.40:0/777374682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56151751b2d0 con 0x5615178a57b0 2024-04-26T23:04:29.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.144+0000 7f3c95e00640 1 --2- 172.21.15.40:0/777374682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178a57b0 0x5615178a7910 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3c88089260 tx=0x7f3c88058a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=911457df58db2577 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.145+0000 7f3c86600640 1 -- 172.21.15.40:0/777374682 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8805ae60 con 0x5615178a57b0 2024-04-26T23:04:29.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.145+0000 7f3c86600640 1 -- 172.21.15.40:0/777374682 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c88061700 con 0x5615178a57b0 2024-04-26T23:04:29.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.145+0000 7f3c86600640 1 -- 172.21.15.40:0/777374682 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c88055190 con 0x5615178a57b0 2024-04-26T23:04:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.145+0000 7f3c9f863740 1 -- 172.21.15.40:0/777374682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178a57b0 msgr2=0x5615178a7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.145+0000 7f3c9f863740 1 --2- 172.21.15.40:0/777374682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178a57b0 0x5615178a7910 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3c88089260 tx=0x7f3c88058a20 comp rx=0 tx=0).stop 2024-04-26T23:04:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.145+0000 7f3c9f863740 1 -- 172.21.15.40:0/777374682 shutdown_connections 2024-04-26T23:04:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.145+0000 7f3c9f863740 1 --2- 172.21.15.40:0/777374682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178d0270 0x5615178d06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.145+0000 7f3c9f863740 1 --2- 172.21.15.40:0/777374682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178e1e80 0x5615178a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.145+0000 7f3c9f863740 1 --2- 172.21.15.40:0/777374682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178a57b0 0x5615178a7910 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.145+0000 7f3c9f863740 1 -- 172.21.15.40:0/777374682 >> 172.21.15.40:0/777374682 conn(0x561517839d10 msgr2=0x5615178eb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.146+0000 7f3c9f863740 1 -- 172.21.15.40:0/777374682 shutdown_connections 2024-04-26T23:04:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.146+0000 7f3c9f863740 1 -- 172.21.15.40:0/777374682 wait complete. 2024-04-26T23:04:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.146+0000 7f3c9f863740 1 -- start start 2024-04-26T23:04:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.146+0000 7f3c9f863740 1 Processor -- start 2024-04-26T23:04:29.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.147+0000 7f3c9f863740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178a57b0 0x5615178bb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.147+0000 7f3c9f863740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178d0270 0x5615178e4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.147+0000 7f3c9f863740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178e1e80 0x5615178e4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.147+0000 7f3c9f863740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561517790f70 con 0x5615178d0270 2024-04-26T23:04:29.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.147+0000 7f3c9f863740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5615178cddc0 con 0x5615178e1e80 2024-04-26T23:04:29.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.147+0000 7f3c9f863740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5615178e5200 con 0x5615178a57b0 2024-04-26T23:04:29.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.148+0000 7f3c9f863740 10 client.0 _ll_register_callbacks cb 0x5615177ff480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:29.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.148+0000 7f3c8fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178d0270 0x5615178e4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.148+0000 7f3c8fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178d0270 0x5615178e4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56792/0 (socket says 172.21.15.40:56792) 2024-04-26T23:04:29.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.148+0000 7f3c8fe00640 1 -- 172.21.15.40:0/554960455 learned_addr learned my addr 172.21.15.40:0/554960455 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.148+0000 7f3c95e00640 1 --2- 172.21.15.40:0/554960455 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178a57b0 0x5615178bb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.148+0000 7f3c8fe00640 1 -- 172.21.15.40:0/554960455 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178a57b0 msgr2=0x5615178bb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.148+0000 7f3c8fe00640 1 --2- 172.21.15.40:0/554960455 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178a57b0 0x5615178bb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.148+0000 7f3c96800640 1 --2- 172.21.15.40:0/554960455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178e1e80 0x5615178e4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.148+0000 7f3c8fe00640 1 -- 172.21.15.40:0/554960455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178e1e80 msgr2=0x5615178e4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.148+0000 7f3c8fe00640 1 --2- 172.21.15.40:0/554960455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178e1e80 0x5615178e4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.148+0000 7f3c8fe00640 1 -- 172.21.15.40:0/554960455 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615175e56b0 con 0x5615178d0270 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.148+0000 7f3c8fe00640 1 --2- 172.21.15.40:0/554960455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178d0270 0x5615178e4650 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3c800098c0 tx=0x7f3c800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9b02f5f7561f79 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.148+0000 7f3c7aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.149+0000 7f3c7aa00640 1 -- 172.21.15.40:0/554960455 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80037040 con 0x5615178d0270 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.149+0000 7f3c7aa00640 1 -- 172.21.15.40:0/554960455 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f3c8003d850 con 0x5615178d0270 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.149+0000 7f3c9f863740 1 -- 172.21.15.40:0/554960455 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c88061700 con 0x5615178d0270 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.149+0000 7f3c7aa00640 1 -- 172.21.15.40:0/554960455 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f3c80055050 con 0x5615178d0270 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.149+0000 7f3c9f863740 10 client.6228 fetch_fsmap learned FSMap version 40 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.149+0000 7f3c9f863740 1 -- 172.21.15.40:0/554960455 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) -- 0x5615178d1900 con 0x5615178d0270 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.149+0000 7f3c7aa00640 1 -- 172.21.15.40:0/554960455 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c88061700 con 0x5615178d0270 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.149+0000 7f3c7aa00640 1 -- 172.21.15.40:0/554960455 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 3534+0+0 (secure 0 0 0) 0x7f3c80058020 con 0x5615178d0270 2024-04-26T23:04:29.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.149+0000 7f3c9f863740 10 client.6228 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T23:04:29.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.150+0000 7f3c9f863740 10 client.6228 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:04:29.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.150+0000 7f3c9f863740 20 client.6228 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:29.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.150+0000 7f3c9f863740 1 --2- 172.21.15.40:0/554960455 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x5615178bddd0 0x5615178bcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.150+0000 7f3c9f863740 4 client.6228 mds_command: new command op to 5284 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:04:29.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.150+0000 7f3c9f863740 1 -- 172.21.15.40:0/554960455 --> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561517817210 con 0x5615178bddd0 2024-04-26T23:04:29.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.150+0000 7f3c95e00640 1 --2- 172.21.15.40:0/554960455 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x5615178bddd0 0x5615178bcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.151+0000 7f3c95e00640 1 --2- 172.21.15.40:0/554960455 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x5615178bddd0 0x5615178bcd40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c62f744e923f25f8 server_cookie=7e659caa088c52af in_seq=0 out_seq=0 2024-04-26T23:04:29.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.151+0000 7f3c7aa00640 10 client.6228 ms_handle_connect on v2:172.21.15.103:6836/670995452 2024-04-26T23:04:29.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.153+0000 7f3c7aa00640 1 -- 172.21.15.40:0/554960455 <== mds.2 v2:172.21.15.103:6836/670995452 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561517817210 con 0x5615178bddd0 2024-04-26T23:04:29.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.153+0000 7f3c7aa00640 10 client.6228 handle_command_reply: tid=0 2024-04-26T23:04:29.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.153+0000 7f3c9f863740 1 client.6228 shutdown 2024-04-26T23:04:29.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.154+0000 7f3c9f863740 1 -- 172.21.15.40:0/554960455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178d0270 msgr2=0x5615178e4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.154+0000 7f3c9f863740 1 --2- 172.21.15.40:0/554960455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178d0270 0x5615178e4650 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3c800098c0 tx=0x7f3c800046c0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.154+0000 7f3c9f863740 1 -- 172.21.15.40:0/554960455 shutdown_connections 2024-04-26T23:04:29.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.154+0000 7f3c9f863740 1 --2- 172.21.15.40:0/554960455 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x5615178bddd0 0x5615178bcd40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.154+0000 7f3c9f863740 1 --2- 172.21.15.40:0/554960455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178e1e80 0x5615178e4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.154+0000 7f3c9f863740 1 --2- 172.21.15.40:0/554960455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178d0270 0x5615178e4650 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.154+0000 7f3c9f863740 1 --2- 172.21.15.40:0/554960455 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178a57b0 0x5615178bb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.154+0000 7f3c7aa00640 0 client.6228 ms_handle_reset on v2:172.21.15.103:6836/670995452 2024-04-26T23:04:29.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.154+0000 7f3c9f863740 1 -- 172.21.15.40:0/554960455 >> 172.21.15.40:0/554960455 conn(0x561517839d10 msgr2=0x5615178cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.154+0000 7f3c9f863740 1 -- 172.21.15.40:0/554960455 shutdown_connections 2024-04-26T23:04:29.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.154+0000 7f3c9f863740 1 -- 172.21.15.40:0/554960455 wait complete. 2024-04-26T23:04:29.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.154+0000 7f3c9f863740 20 client.6228 trim_cache size 0 max 16384 2024-04-26T23:04:29.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.174+0000 7f3c9f863740 1 Processor -- start 2024-04-26T23:04:29.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.175+0000 7f3c9f863740 1 -- start start 2024-04-26T23:04:29.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.175+0000 7f3c9f863740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178a57b0 0x5615178cc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.175+0000 7f3c9f863740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178e1e80 0x5615178fdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.175+0000 7f3c9f863740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178fe0e0 0x5615178ee660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.175+0000 7f3c9f863740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c98200c90 con 0x5615178fe0e0 2024-04-26T23:04:29.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.175+0000 7f3c9f863740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c98201360 con 0x5615178e1e80 2024-04-26T23:04:29.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.175+0000 7f3c9f863740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3c9815b5b0 con 0x5615178a57b0 2024-04-26T23:04:29.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.176+0000 7f3c96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178fe0e0 0x5615178ee660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.176+0000 7f3c96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178fe0e0 0x5615178ee660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56806/0 (socket says 172.21.15.40:56806) 2024-04-26T23:04:29.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.176+0000 7f3c96800640 1 -- 172.21.15.40:0/1872917792 learned_addr learned my addr 172.21.15.40:0/1872917792 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.176+0000 7f3c96800640 1 -- 172.21.15.40:0/1872917792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178a57b0 msgr2=0x5615178cc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.176+0000 7f3c96800640 1 --2- 172.21.15.40:0/1872917792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178a57b0 0x5615178cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.176+0000 7f3c96800640 1 -- 172.21.15.40:0/1872917792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178e1e80 msgr2=0x5615178fdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.176+0000 7f3c8fe00640 1 --2- 172.21.15.40:0/1872917792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178e1e80 0x5615178fdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.176+0000 7f3c96800640 1 --2- 172.21.15.40:0/1872917792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178e1e80 0x5615178fdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.176+0000 7f3c96800640 1 -- 172.21.15.40:0/1872917792 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615177be570 con 0x5615178fe0e0 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.176+0000 7f3c96800640 1 --2- 172.21.15.40:0/1872917792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178fe0e0 0x5615178ee660 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3c90001a90 tx=0x7f3c90001f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2860bb1c2bb616cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.177+0000 7f3c85c00640 1 -- 172.21.15.40:0/1872917792 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c90003d80 con 0x5615178fe0e0 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.177+0000 7f3c85c00640 1 -- 172.21.15.40:0/1872917792 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c90018070 con 0x5615178fe0e0 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.177+0000 7f3c85c00640 1 -- 172.21.15.40:0/1872917792 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9000fa30 con 0x5615178fe0e0 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.177+0000 7f3c9f863740 1 -- 172.21.15.40:0/1872917792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178fe0e0 msgr2=0x5615178ee660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.177+0000 7f3c9f863740 1 --2- 172.21.15.40:0/1872917792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178fe0e0 0x5615178ee660 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3c90001a90 tx=0x7f3c90001f50 comp rx=0 tx=0).stop 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.177+0000 7f3c9f863740 1 -- 172.21.15.40:0/1872917792 shutdown_connections 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.177+0000 7f3c9f863740 1 --2- 172.21.15.40:0/1872917792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178fe0e0 0x5615178ee660 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.177+0000 7f3c9f863740 1 --2- 172.21.15.40:0/1872917792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178e1e80 0x5615178fdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.177+0000 7f3c9f863740 1 --2- 172.21.15.40:0/1872917792 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178a57b0 0x5615178cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.177+0000 7f3c9f863740 1 -- 172.21.15.40:0/1872917792 >> 172.21.15.40:0/1872917792 conn(0x561517839d10 msgr2=0x5615178e3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.177+0000 7f3c9f863740 1 -- 172.21.15.40:0/1872917792 shutdown_connections 2024-04-26T23:04:29.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.177+0000 7f3c9f863740 1 -- 172.21.15.40:0/1872917792 wait complete. 2024-04-26T23:04:29.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.178+0000 7f3c9f863740 1 -- start start 2024-04-26T23:04:29.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.178+0000 7f3c9f863740 1 Processor -- start 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.179+0000 7f3c9f863740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178a57b0 0x56151794e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.179+0000 7f3c9f863740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178e1e80 0x5615178cc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.179+0000 7f3c9f863740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178fe0e0 0x5615178e0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.179+0000 7f3c9f863740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5615178bdc50 con 0x5615178e1e80 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.179+0000 7f3c9f863740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5615178e5e70 con 0x5615178a57b0 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.179+0000 7f3c9f863740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5615178be2a0 con 0x5615178fe0e0 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.179+0000 7f3c9f863740 10 client.0 _ll_register_callbacks cb 0x7f3c80035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.180+0000 7f3c8fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178e1e80 0x5615178cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.180+0000 7f3c8fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178e1e80 0x5615178cc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56810/0 (socket says 172.21.15.40:56810) 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.180+0000 7f3c8fe00640 1 -- 172.21.15.40:0/2914804645 learned_addr learned my addr 172.21.15.40:0/2914804645 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.180+0000 7f3c8fe00640 1 -- 172.21.15.40:0/2914804645 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178fe0e0 msgr2=0x5615178e0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.180+0000 7f3c8fe00640 1 --2- 172.21.15.40:0/2914804645 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178fe0e0 0x5615178e0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.180+0000 7f3c8fe00640 1 -- 172.21.15.40:0/2914804645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178a57b0 msgr2=0x56151794e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.180+0000 7f3c8fe00640 1 --2- 172.21.15.40:0/2914804645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178a57b0 0x56151794e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.180+0000 7f3c8fe00640 1 -- 172.21.15.40:0/2914804645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56151790e680 con 0x5615178e1e80 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.180+0000 7f3c8fe00640 1 --2- 172.21.15.40:0/2914804645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178e1e80 0x5615178cc070 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x5615178e3480 tx=0x7f3c80008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c52efb680a609bc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.180+0000 7f3c7aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.180+0000 7f3c7aa00640 1 -- 172.21.15.40:0/2914804645 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80037040 con 0x5615178e1e80 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.181+0000 7f3c7aa00640 1 -- 172.21.15.40:0/2914804645 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f3c8003dae0 con 0x5615178e1e80 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.181+0000 7f3c9f863740 1 -- 172.21.15.40:0/2914804645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c90018070 con 0x5615178e1e80 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.181+0000 7f3c7aa00640 1 -- 172.21.15.40:0/2914804645 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f3c80003610 con 0x5615178e1e80 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.181+0000 7f3c9f863740 10 client.6234 fetch_fsmap learned FSMap version 40 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.181+0000 7f3c9f863740 1 -- 172.21.15.40:0/2914804645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) -- 0x5615178bac80 con 0x5615178e1e80 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.181+0000 7f3c7aa00640 1 -- 172.21.15.40:0/2914804645 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c90018070 con 0x5615178e1e80 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.181+0000 7f3c7aa00640 1 -- 172.21.15.40:0/2914804645 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 3534+0+0 (secure 0 0 0) 0x7f3c80058070 con 0x5615178e1e80 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.181+0000 7f3c9f863740 10 client.6234 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.181+0000 7f3c9f863740 10 client.6234 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:04:29.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.181+0000 7f3c9f863740 20 client.6234 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:29.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.181+0000 7f3c9f863740 1 --2- 172.21.15.40:0/2914804645 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x5615178b9c00 0x5615178e1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.181+0000 7f3c9f863740 4 client.6234 mds_command: new command op to 5284 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:04:29.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.181+0000 7f3c9f863740 1 -- 172.21.15.40:0/2914804645 --> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] -- command(tid 0: {"prefix": "status"}) -- 0x561517811080 con 0x5615178b9c00 2024-04-26T23:04:29.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.182+0000 7f3c95e00640 1 --2- 172.21.15.40:0/2914804645 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x5615178b9c00 0x5615178e1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.183+0000 7f3c95e00640 1 --2- 172.21.15.40:0/2914804645 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x5615178b9c00 0x5615178e1140 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=add22f79cd0fe6c1 server_cookie=a892a0174250c5c0 in_seq=0 out_seq=0 2024-04-26T23:04:29.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.183+0000 7f3c7aa00640 10 client.6234 ms_handle_connect on v2:172.21.15.103:6836/670995452 2024-04-26T23:04:29.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.183+0000 7f3c7aa00640 1 -- 172.21.15.40:0/2914804645 <== mds.2 v2:172.21.15.103:6836/670995452 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x561517811080 con 0x5615178b9c00 2024-04-26T23:04:29.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.183+0000 7f3c7aa00640 10 client.6234 handle_command_reply: tid=0 2024-04-26T23:04:29.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.183+0000 7f3c9f863740 1 client.6234 shutdown 2024-04-26T23:04:29.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.184+0000 7f3c9f863740 1 -- 172.21.15.40:0/2914804645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178e1e80 msgr2=0x5615178cc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.184+0000 7f3c9f863740 1 --2- 172.21.15.40:0/2914804645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178e1e80 0x5615178cc070 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x5615178e3480 tx=0x7f3c80008030 comp rx=0 tx=0).stop 2024-04-26T23:04:29.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.184+0000 7f3c9f863740 1 -- 172.21.15.40:0/2914804645 shutdown_connections 2024-04-26T23:04:29.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.184+0000 7f3c9f863740 1 --2- 172.21.15.40:0/2914804645 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x5615178b9c00 0x5615178e1140 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.184+0000 7f3c9f863740 1 --2- 172.21.15.40:0/2914804645 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5615178fe0e0 0x5615178e0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.184+0000 7f3c9f863740 1 --2- 172.21.15.40:0/2914804645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615178e1e80 0x5615178cc070 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.184+0000 7f3c9f863740 1 --2- 172.21.15.40:0/2914804645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5615178a57b0 0x56151794e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.184+0000 7f3c7aa00640 0 client.6234 ms_handle_reset on v2:172.21.15.103:6836/670995452 2024-04-26T23:04:29.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.184+0000 7f3c9f863740 1 -- 172.21.15.40:0/2914804645 >> 172.21.15.40:0/2914804645 conn(0x561517839d10 msgr2=0x5615178ebac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.184+0000 7f3c9f863740 1 -- 172.21.15.40:0/2914804645 shutdown_connections 2024-04-26T23:04:29.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.184+0000 7f3c9f863740 1 -- 172.21.15.40:0/2914804645 wait complete. 2024-04-26T23:04:29.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.184+0000 7f3c9f863740 20 client.6234 trim_cache size 0 max 16384 2024-04-26T23:04:29.187 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:04:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:04:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-04-26T23:04:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5284, 2024-04-26T23:04:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:04:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:04:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:04:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 2.0212666600000002, 2024-04-26T23:04:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 40, 2024-04-26T23:04:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 52, 2024-04-26T23:04:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 52, 2024-04-26T23:04:29.188 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 17.497436938 2024-04-26T23:04:29.188 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:04:29.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.185+0000 7f3c97200640 1 -- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c7406c920 msgr2=0x7f3c7406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.185+0000 7f3c97200640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c7406c920 0x7f3c7406ede0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3c80034000 tx=0x7f3c8002e000 comp rx=0 tx=0).stop 2024-04-26T23:04:29.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.185+0000 7f3c97200640 1 -- 172.21.15.40:0/3765196999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c980b6220 msgr2=0x7f3c9815d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.185+0000 7f3c97200640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c980b6220 0x7f3c9815d9a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800c4d0 tx=0x7f3c8800c990 comp rx=0 tx=0).stop 2024-04-26T23:04:29.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.185+0000 7f3c97200640 1 -- 172.21.15.40:0/3765196999 shutdown_connections 2024-04-26T23:04:29.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.185+0000 7f3c97200640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c7406c920 0x7f3c7406ede0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.185+0000 7f3c97200640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3c980d32e0 0x7f3c98152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.185+0000 7f3c97200640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c980b8d20 0x7f3c980ce210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.185+0000 7f3c97200640 1 --2- 172.21.15.40:0/3765196999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c980b6220 0x7f3c9815d9a0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.185+0000 7f3c97200640 1 -- 172.21.15.40:0/3765196999 >> 172.21.15.40:0/3765196999 conn(0x7f3c980d4d90 msgr2=0x7f3c9806d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.185+0000 7f3c97200640 1 -- 172.21.15.40:0/3765196999 shutdown_connections 2024-04-26T23:04:29.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.185+0000 7f3c97200640 1 -- 172.21.15.40:0/3765196999 wait complete. 2024-04-26T23:04:29.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T23:04:29.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.390+0000 7fc5c6400640 1 Processor -- start 2024-04-26T23:04:29.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.391+0000 7fc5c6400640 1 -- start start 2024-04-26T23:04:29.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.392+0000 7fc5c6400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5c0071220 0x7fc5c0168260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.392+0000 7fc5c6400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c0068e60 con 0x7fc5c00b6eb0 2024-04-26T23:04:29.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.392+0000 7fc5c6400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c00b11d0 con 0x7fc5c015d010 2024-04-26T23:04:29.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.392+0000 7fc5c6400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c015d3f0 con 0x7fc5c0071220 2024-04-26T23:04:29.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.393+0000 7fc5bfe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc5c00b6eb0 0x7fc5c0164b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58592/0 (socket says 172.21.15.40:58592) 2024-04-26T23:04:29.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.393+0000 7fc5bfe00640 1 -- 172.21.15.40:0/3749963780 learned_addr learned my addr 172.21.15.40:0/3749963780 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:29.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.393+0000 7fc5c5a00640 1 --2- 172.21.15.40:0/3749963780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5c0071220 0x7fc5c0168260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.393+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1119495430 0 0) 0x7fc5c0068e60 con 0x7fc5c00b6eb0 2024-04-26T23:04:29.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.393+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5a8003520 con 0x7fc5c00b6eb0 2024-04-26T23:04:29.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3871667178 0 0) 0x7fc5a8003520 con 0x7fc5c00b6eb0 2024-04-26T23:04:29.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5c0071220 msgr2=0x7fc5c0168260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5bf400640 1 --2- 172.21.15.40:0/3749963780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5c0071220 0x7fc5c0168260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 >> v1:172.21.15.103:6789/0 conn(0x7fc5c015d010 legacy=0x7fc5c00b67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c00b14b0 con 0x7fc5c00b6eb0 2024-04-26T23:04:29.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fc5ac001670 con 0x7fc5c00b6eb0 2024-04-26T23:04:29.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 >> v1:172.21.15.40:6789/0 conn(0x7fc5c00b6eb0 legacy=0x7fc5c0164b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5bf400640 1 --2- 172.21.15.40:0/3749963780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5a80039e0 0x7fc5a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5bf400640 1 --2- 172.21.15.40:0/3749963780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5c0071220 0x7fc5a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5bf400640 1 --2- 172.21.15.40:0/3749963780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5a8004d40 0x7fc5a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c015d3f0 con 0x7fc5a80039e0 2024-04-26T23:04:29.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a800bb50 con 0x7fc5a8004d40 2024-04-26T23:04:29.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc5a800bd50 con 0x7fc5c0071220 2024-04-26T23:04:29.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5c5000640 1 --2- 172.21.15.40:0/3749963780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5a80039e0 0x7fc5a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5c5a00640 1 --2- 172.21.15.40:0/3749963780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5a8004d40 0x7fc5a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.394+0000 7fc5c5000640 1 -- 172.21.15.40:0/3749963780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5c0071220 msgr2=0x7fc5a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.395+0000 7fc5c5000640 1 --2- 172.21.15.40:0/3749963780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5c0071220 0x7fc5a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.395+0000 7fc5c5000640 1 -- 172.21.15.40:0/3749963780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5a8004d40 msgr2=0x7fc5a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.395+0000 7fc5c5000640 1 --2- 172.21.15.40:0/3749963780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5a8004d40 0x7fc5a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.395+0000 7fc5c5000640 1 -- 172.21.15.40:0/3749963780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a8003520 con 0x7fc5a80039e0 2024-04-26T23:04:29.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.395+0000 7fc5c5a00640 1 --2- 172.21.15.40:0/3749963780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5a8004d40 0x7fc5a80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:29.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.396+0000 7fc5c5000640 1 --2- 172.21.15.40:0/3749963780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5a80039e0 0x7fc5a8003e00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4001750 tx=0x7fc5a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ed2257af1f4d95 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.396+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4001370 con 0x7fc5a80039e0 2024-04-26T23:04:29.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.396+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b40052d0 con 0x7fc5a80039e0 2024-04-26T23:04:29.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.396+0000 7fc5bf400640 1 -- 172.21.15.40:0/3749963780 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b402abe0 con 0x7fc5a80039e0 2024-04-26T23:04:29.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.397+0000 7fc5c6400640 1 -- 172.21.15.40:0/3749963780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5a80039e0 msgr2=0x7fc5a8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.397+0000 7fc5c6400640 1 --2- 172.21.15.40:0/3749963780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5a80039e0 0x7fc5a8003e00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4001750 tx=0x7fc5a8005830 comp rx=0 tx=0).stop 2024-04-26T23:04:29.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.397+0000 7fc5c5a00640 1 -- 172.21.15.40:0/3749963780 reap_dead start 2024-04-26T23:04:29.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.397+0000 7fc5c6400640 1 -- 172.21.15.40:0/3749963780 shutdown_connections 2024-04-26T23:04:29.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.397+0000 7fc5c6400640 1 -- 172.21.15.40:0/3749963780 >> 172.21.15.40:0/3749963780 conn(0x7fc5c009ba50 msgr2=0x7fc5c009be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.397+0000 7fc5c6400640 1 -- 172.21.15.40:0/3749963780 shutdown_connections 2024-04-26T23:04:29.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.397+0000 7fc5c6400640 1 -- 172.21.15.40:0/3749963780 wait complete. 2024-04-26T23:04:29.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.399+0000 7fc5c6400640 1 Processor -- start 2024-04-26T23:04:29.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.399+0000 7fc5c6400640 1 -- start start 2024-04-26T23:04:29.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c6400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5a80039e0 0x7fc5c014a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c6400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5c006dd30 0x7fc5c006e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5c014cfc0 0x7fc5c014d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c6400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c00b1d50 con 0x7fc5c014cfc0 2024-04-26T23:04:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c6400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c00b1a70 con 0x7fc5c006dd30 2024-04-26T23:04:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c6400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c00b0e60 con 0x7fc5a80039e0 2024-04-26T23:04:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5c014cfc0 0x7fc5c014d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5c014cfc0 0x7fc5c014d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56818/0 (socket says 172.21.15.40:56818) 2024-04-26T23:04:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c5a00640 1 -- 172.21.15.40:0/1013540888 learned_addr learned my addr 172.21.15.40:0/1013540888 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c5000640 1 --2- 172.21.15.40:0/1013540888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5a80039e0 0x7fc5c014a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c5a00640 1 -- 172.21.15.40:0/1013540888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5a80039e0 msgr2=0x7fc5c014a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c5a00640 1 --2- 172.21.15.40:0/1013540888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5a80039e0 0x7fc5c014a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c5a00640 1 -- 172.21.15.40:0/1013540888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5c006dd30 msgr2=0x7fc5c006e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c5a00640 1 --2- 172.21.15.40:0/1013540888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5c006dd30 0x7fc5c006e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.400+0000 7fc5c5a00640 1 -- 172.21.15.40:0/1013540888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c006e820 con 0x7fc5c014cfc0 2024-04-26T23:04:29.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.401+0000 7fc5c5a00640 1 --2- 172.21.15.40:0/1013540888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5c014cfc0 0x7fc5c014d420 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0074b10 tx=0x7fc5b800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.401+0000 7fc5bcc00640 1 -- 172.21.15.40:0/1013540888 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b801c070 con 0x7fc5c014cfc0 2024-04-26T23:04:29.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.401+0000 7fc5bcc00640 1 -- 172.21.15.40:0/1013540888 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b8017400 con 0x7fc5c014cfc0 2024-04-26T23:04:29.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.401+0000 7fc5bcc00640 1 -- 172.21.15.40:0/1013540888 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b80177d0 con 0x7fc5c014cfc0 2024-04-26T23:04:29.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.401+0000 7fc5c6400640 1 -- 172.21.15.40:0/1013540888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c006eb00 con 0x7fc5c014cfc0 2024-04-26T23:04:29.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.401+0000 7fc5c6400640 1 -- 172.21.15.40:0/1013540888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c014abe0 con 0x7fc5c014cfc0 2024-04-26T23:04:29.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.403+0000 7fc5bcc00640 1 -- 172.21.15.40:0/1013540888 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc5b801f070 con 0x7fc5c014cfc0 2024-04-26T23:04:29.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.405+0000 7fc5bcc00640 1 --2- 172.21.15.40:0/1013540888 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc5a006c8f0 0x7fc5a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.405+0000 7fc5bcc00640 1 -- 172.21.15.40:0/1013540888 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fc5b808bcc0 con 0x7fc5c014cfc0 2024-04-26T23:04:29.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.406+0000 7fc5ca9d8740 1 Processor -- start 2024-04-26T23:04:29.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.406+0000 7fc5c5000640 1 --2- 172.21.15.40:0/1013540888 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc5a006c8f0 0x7fc5a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.406+0000 7fc5ca9d8740 1 -- start start 2024-04-26T23:04:29.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.407+0000 7fc5ca9d8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed28467b0 0x561ed2848910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.407+0000 7fc5ca9d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2882e80 0x561ed2846b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.407+0000 7fc5ca9d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed2871270 0x561ed28716f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.407+0000 7fc5ca9d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561ed27aa600 con 0x561ed2882e80 2024-04-26T23:04:29.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.407+0000 7fc5ca9d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561ed2747100 con 0x561ed2871270 2024-04-26T23:04:29.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.407+0000 7fc5ca9d8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561ed27a65c0 con 0x561ed28467b0 2024-04-26T23:04:29.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.407+0000 7fc5c5000640 1 --2- 172.21.15.40:0/1013540888 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc5a006c8f0 0x7fc5a006edb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4032000 tx=0x7fc5b402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.407+0000 7fc5bfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2882e80 0x561ed2846b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.407+0000 7fc5bfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2882e80 0x561ed2846b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56824/0 (socket says 172.21.15.40:56824) 2024-04-26T23:04:29.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.407+0000 7fc5bfe00640 1 -- 172.21.15.40:0/1691817385 learned_addr learned my addr 172.21.15.40:0/1691817385 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.407+0000 7fc5c5a00640 1 --2- 172.21.15.40:0/1691817385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed2871270 0x561ed28716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.407+0000 7fc5bfe00640 1 -- 172.21.15.40:0/1691817385 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed28467b0 msgr2=0x561ed2848910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.407+0000 7fc5bfe00640 1 --2- 172.21.15.40:0/1691817385 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed28467b0 0x561ed2848910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.408+0000 7fc5bfe00640 1 -- 172.21.15.40:0/1691817385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed2871270 msgr2=0x561ed28716f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.408+0000 7fc5bfe00640 1 --2- 172.21.15.40:0/1691817385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed2871270 0x561ed28716f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.408+0000 7fc5bfe00640 1 -- 172.21.15.40:0/1691817385 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ed24bc2d0 con 0x561ed2882e80 2024-04-26T23:04:29.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.408+0000 7fc5bfe00640 1 --2- 172.21.15.40:0/1691817385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2882e80 0x561ed2846b90 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0079b0 tx=0x7fc5ac007e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708072fc3ffa2e22 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.408+0000 7fc5b1600640 1 -- 172.21.15.40:0/1691817385 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0074e0 con 0x561ed2882e80 2024-04-26T23:04:29.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.408+0000 7fc5b1600640 1 -- 172.21.15.40:0/1691817385 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ac0047e0 con 0x561ed2882e80 2024-04-26T23:04:29.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.408+0000 7fc5b1600640 1 -- 172.21.15.40:0/1691817385 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac004bd0 con 0x561ed2882e80 2024-04-26T23:04:29.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.409+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/1691817385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2882e80 msgr2=0x561ed2846b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.409+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/1691817385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2882e80 0x561ed2846b90 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0079b0 tx=0x7fc5ac007e70 comp rx=0 tx=0).stop 2024-04-26T23:04:29.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.409+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/1691817385 shutdown_connections 2024-04-26T23:04:29.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.409+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/1691817385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed2871270 0x561ed28716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.409+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/1691817385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2882e80 0x561ed2846b90 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.409+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/1691817385 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed28467b0 0x561ed2848910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.409+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/1691817385 >> 172.21.15.40:0/1691817385 conn(0x561ed27dad10 msgr2=0x561ed288c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.409+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/1691817385 shutdown_connections 2024-04-26T23:04:29.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.409+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/1691817385 wait complete. 2024-04-26T23:04:29.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.410+0000 7fc5ca9d8740 1 -- start start 2024-04-26T23:04:29.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.410+0000 7fc5ca9d8740 1 Processor -- start 2024-04-26T23:04:29.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.411+0000 7fc5ca9d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 0x561ed2884e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.411+0000 7fc5ca9d8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed2871270 0x561ed288e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.411+0000 7fc5ca9d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed2882e80 0x561ed2883280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.411+0000 7fc5ca9d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561ed2731f70 con 0x561ed28467b0 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.411+0000 7fc5ca9d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561ed286edc0 con 0x561ed2882e80 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.411+0000 7fc5ca9d8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561ed285f230 con 0x561ed2871270 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.411+0000 7fc5ca9d8740 10 client.0 _ll_register_callbacks cb 0x561ed27a0480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.411+0000 7fc5c5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 0x561ed2884e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.411+0000 7fc5c5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 0x561ed2884e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56834/0 (socket says 172.21.15.40:56834) 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.411+0000 7fc5c5000640 1 -- 172.21.15.40:0/1668321916 learned_addr learned my addr 172.21.15.40:0/1668321916 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.412+0000 7fc5c5000640 1 -- 172.21.15.40:0/1668321916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed2871270 msgr2=0x561ed288e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.412+0000 7fc5c5000640 1 --2- 172.21.15.40:0/1668321916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed2871270 0x561ed288e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.412+0000 7fc5c5000640 1 -- 172.21.15.40:0/1668321916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed2882e80 msgr2=0x561ed2883280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.412+0000 7fc5c5000640 1 --2- 172.21.15.40:0/1668321916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed2882e80 0x561ed2883280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.412+0000 7fc5c5000640 1 -- 172.21.15.40:0/1668321916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ed25866b0 con 0x561ed28467b0 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.412+0000 7fc5c5000640 1 --2- 172.21.15.40:0/1668321916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 0x561ed2884e70 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fc5b402a590 tx=0x7fc5b4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11623e3e4c404d0d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.412+0000 7fc5a6000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.412+0000 7fc5a6000640 1 -- 172.21.15.40:0/1668321916 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b404d060 con 0x561ed28467b0 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.412+0000 7fc5a6000640 1 -- 172.21.15.40:0/1668321916 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fc5b40067d0 con 0x561ed28467b0 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.412+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/1668321916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5ac0047e0 con 0x561ed28467b0 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.412+0000 7fc5a6000640 1 -- 172.21.15.40:0/1668321916 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc5b40076a0 con 0x561ed28467b0 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.413+0000 7fc5ca9d8740 10 client.6246 fetch_fsmap learned FSMap version 40 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.413+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/1668321916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) -- 0x561ed285ef70 con 0x561ed28467b0 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.413+0000 7fc5a6000640 1 -- 172.21.15.40:0/1668321916 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ac0047e0 con 0x561ed28467b0 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.413+0000 7fc5a6000640 1 -- 172.21.15.40:0/1668321916 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 3534+0+0 (secure 0 0 0) 0x7fc5b40069d0 con 0x561ed28467b0 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.413+0000 7fc5ca9d8740 10 client.6246 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.413+0000 7fc5ca9d8740 10 client.6246 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.413+0000 7fc5ca9d8740 20 client.6246 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.413+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/1668321916 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x561ed289fa70 0x561ed289fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.413+0000 7fc5ca9d8740 4 client.6246 mds_command: new command op to 6027 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.413+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/1668321916 --> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ed27b8210 con 0x561ed289fa70 2024-04-26T23:04:29.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.413+0000 7fc5bfe00640 1 --2- 172.21.15.40:0/1668321916 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x561ed289fa70 0x561ed289fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.414+0000 7fc5bfe00640 1 --2- 172.21.15.40:0/1668321916 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x561ed289fa70 0x561ed289fe50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=936f93f1b6f7fcbb server_cookie=b63d3ba6833c39c0 in_seq=0 out_seq=0 2024-04-26T23:04:29.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.414+0000 7fc5a6000640 10 client.6246 ms_handle_connect on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:29.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.415+0000 7fc5a6000640 1 -- 172.21.15.40:0/1668321916 <== mds.1 v2:172.21.15.40:6832/2304583125 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ed27b8210 con 0x561ed289fa70 2024-04-26T23:04:29.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.415+0000 7fc5a6000640 10 client.6246 handle_command_reply: tid=0 2024-04-26T23:04:29.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.415+0000 7fc5ca9d8740 1 client.6246 shutdown 2024-04-26T23:04:29.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.415+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/1668321916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 msgr2=0x561ed2884e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.415+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/1668321916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 0x561ed2884e70 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fc5b402a590 tx=0x7fc5b4001120 comp rx=0 tx=0).stop 2024-04-26T23:04:29.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.416+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/1668321916 shutdown_connections 2024-04-26T23:04:29.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.416+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/1668321916 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x561ed289fa70 0x561ed289fe50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.416+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/1668321916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed2882e80 0x561ed2883280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.416+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/1668321916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed2871270 0x561ed288e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.416+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/1668321916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 0x561ed2884e70 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.416+0000 7fc5a6000640 0 client.6246 ms_handle_reset on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:29.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.416+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/1668321916 >> 172.21.15.40:0/1668321916 conn(0x561ed27dad10 msgr2=0x561ed286c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.416+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/1668321916 shutdown_connections 2024-04-26T23:04:29.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.416+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/1668321916 wait complete. 2024-04-26T23:04:29.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.416+0000 7fc5ca9d8740 20 client.6246 trim_cache size 0 max 16384 2024-04-26T23:04:29.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.436+0000 7fc5ca9d8740 1 Processor -- start 2024-04-26T23:04:29.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.436+0000 7fc5ca9d8740 1 -- start start 2024-04-26T23:04:29.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.436+0000 7fc5ca9d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed28467b0 0x561ed28739f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.436+0000 7fc5ca9d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2871270 0x561ed288d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.436+0000 7fc5ca9d8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed2882e80 0x561ed288d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.436+0000 7fc5ca9d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c0201620 con 0x561ed2871270 2024-04-26T23:04:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.436+0000 7fc5ca9d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c0201cf0 con 0x561ed28467b0 2024-04-26T23:04:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.436+0000 7fc5ca9d8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c02006a0 con 0x561ed2882e80 2024-04-26T23:04:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.437+0000 7fc5bfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2871270 0x561ed288d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.437+0000 7fc5bfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2871270 0x561ed288d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56838/0 (socket says 172.21.15.40:56838) 2024-04-26T23:04:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.437+0000 7fc5bfe00640 1 -- 172.21.15.40:0/3657584108 learned_addr learned my addr 172.21.15.40:0/3657584108 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.437+0000 7fc5bfe00640 1 -- 172.21.15.40:0/3657584108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed2882e80 msgr2=0x561ed288d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.437+0000 7fc5bfe00640 1 --2- 172.21.15.40:0/3657584108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed2882e80 0x561ed288d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.437+0000 7fc5c5000640 1 --2- 172.21.15.40:0/3657584108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed28467b0 0x561ed28739f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.437+0000 7fc5bfe00640 1 -- 172.21.15.40:0/3657584108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed28467b0 msgr2=0x561ed28739f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.437+0000 7fc5bfe00640 1 --2- 172.21.15.40:0/3657584108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed28467b0 0x561ed28739f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.437+0000 7fc5bfe00640 1 -- 172.21.15.40:0/3657584108 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ed2844b10 con 0x561ed2871270 2024-04-26T23:04:29.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.438+0000 7fc5bfe00640 1 --2- 172.21.15.40:0/3657584108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2871270 0x561ed288d3a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac010000 tx=0x7fc5ac00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8873fe4c4caabd7a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.438+0000 7fc5b0c00640 1 -- 172.21.15.40:0/3657584108 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac028070 con 0x561ed2871270 2024-04-26T23:04:29.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.438+0000 7fc5b0c00640 1 -- 172.21.15.40:0/3657584108 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ac00b030 con 0x561ed2871270 2024-04-26T23:04:29.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.438+0000 7fc5b0c00640 1 -- 172.21.15.40:0/3657584108 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac019040 con 0x561ed2871270 2024-04-26T23:04:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.438+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/3657584108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2871270 msgr2=0x561ed288d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.438+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/3657584108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2871270 0x561ed288d3a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac010000 tx=0x7fc5ac00f000 comp rx=0 tx=0).stop 2024-04-26T23:04:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.439+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/3657584108 shutdown_connections 2024-04-26T23:04:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.439+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/3657584108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed2882e80 0x561ed288d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.439+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/3657584108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed2871270 0x561ed288d3a0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.439+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/3657584108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed28467b0 0x561ed28739f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.439+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/3657584108 >> 172.21.15.40:0/3657584108 conn(0x561ed27dad10 msgr2=0x561ed2847690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.439+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/3657584108 shutdown_connections 2024-04-26T23:04:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.439+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/3657584108 wait complete. 2024-04-26T23:04:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.439+0000 7fc5ca9d8740 1 -- start start 2024-04-26T23:04:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.439+0000 7fc5ca9d8740 1 Processor -- start 2024-04-26T23:04:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.440+0000 7fc5ca9d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 0x561ed290c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.440+0000 7fc5ca9d8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed2871270 0x561ed289e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.440+0000 7fc5ca9d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed2882e80 0x561ed290f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.440+0000 7fc5ca9d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561ed285ea70 con 0x561ed28467b0 2024-04-26T23:04:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.440+0000 7fc5ca9d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561ed28837c0 con 0x561ed2882e80 2024-04-26T23:04:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.440+0000 7fc5ca9d8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561ed2871810 con 0x561ed2871270 2024-04-26T23:04:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.440+0000 7fc5ca9d8740 10 client.0 _ll_register_callbacks cb 0x7fc5a80045d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.441+0000 7fc5c5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 0x561ed290c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.441+0000 7fc5c5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 0x561ed290c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56852/0 (socket says 172.21.15.40:56852) 2024-04-26T23:04:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.441+0000 7fc5c5000640 1 -- 172.21.15.40:0/2392204962 learned_addr learned my addr 172.21.15.40:0/2392204962 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.441+0000 7fc5c5000640 1 -- 172.21.15.40:0/2392204962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed2871270 msgr2=0x561ed289e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.441+0000 7fc5c5000640 1 --2- 172.21.15.40:0/2392204962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed2871270 0x561ed289e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.441+0000 7fc5c5000640 1 -- 172.21.15.40:0/2392204962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed2882e80 msgr2=0x561ed290f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.441+0000 7fc5c5000640 1 --2- 172.21.15.40:0/2392204962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed2882e80 0x561ed290f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.441+0000 7fc5c5000640 1 -- 172.21.15.40:0/2392204962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ed289e970 con 0x561ed28467b0 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.441+0000 7fc5c5000640 1 --2- 172.21.15.40:0/2392204962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 0x561ed290c3b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4008fd0 tx=0x7fc5b4009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d2166e64376eb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.441+0000 7fc5a6000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.441+0000 7fc5a6000640 1 -- 172.21.15.40:0/2392204962 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b405c060 con 0x561ed28467b0 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.441+0000 7fc5a6000640 1 -- 172.21.15.40:0/2392204962 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fc5b4004dc0 con 0x561ed28467b0 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.442+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/2392204962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5ac00b030 con 0x561ed28467b0 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.442+0000 7fc5a6000640 1 -- 172.21.15.40:0/2392204962 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc5b4062050 con 0x561ed28467b0 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.442+0000 7fc5ca9d8740 10 client.6252 fetch_fsmap learned FSMap version 40 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.442+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/2392204962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) -- 0x561ed288c590 con 0x561ed28467b0 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.442+0000 7fc5a6000640 1 -- 172.21.15.40:0/2392204962 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ac00b030 con 0x561ed28467b0 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.442+0000 7fc5a6000640 1 -- 172.21.15.40:0/2392204962 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 3534+0+0 (secure 0 0 0) 0x7fc5b4065020 con 0x561ed28467b0 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.442+0000 7fc5ca9d8740 10 client.6252 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.442+0000 7fc5ca9d8740 10 client.6252 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.442+0000 7fc5ca9d8740 20 client.6252 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.442+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/2392204962 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x561ed28af0d0 0x561ed28af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.442+0000 7fc5ca9d8740 4 client.6252 mds_command: new command op to 6027 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.442+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/2392204962 --> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] -- command(tid 0: {"prefix": "status"}) -- 0x561ed27b2080 con 0x561ed28af0d0 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.442+0000 7fc5bfe00640 1 --2- 172.21.15.40:0/2392204962 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x561ed28af0d0 0x561ed28af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5bfe00640 1 --2- 172.21.15.40:0/2392204962 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x561ed28af0d0 0x561ed28af4b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9b1f4833d8ffe5b1 server_cookie=7e930fe657a75635 in_seq=0 out_seq=0 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5a6000640 10 client.6252 ms_handle_connect on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5a6000640 1 -- 172.21.15.40:0/2392204962 <== mds.1 v2:172.21.15.40:6832/2304583125 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x561ed27b2080 con 0x561ed28af0d0 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5a6000640 10 client.6252 handle_command_reply: tid=0 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5ca9d8740 1 client.6252 shutdown 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/2392204962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 msgr2=0x561ed290c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/2392204962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 0x561ed290c3b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4008fd0 tx=0x7fc5b4009030 comp rx=0 tx=0).stop 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/2392204962 shutdown_connections 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/2392204962 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x561ed28af0d0 0x561ed28af4b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/2392204962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561ed2882e80 0x561ed290f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/2392204962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561ed2871270 0x561ed289e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5ca9d8740 1 --2- 172.21.15.40:0/2392204962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561ed28467b0 0x561ed290c3b0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5a6000640 0 client.6252 ms_handle_reset on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:29.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.443+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/2392204962 >> 172.21.15.40:0/2392204962 conn(0x561ed27dad10 msgr2=0x561ed28b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.444+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/2392204962 shutdown_connections 2024-04-26T23:04:29.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.444+0000 7fc5ca9d8740 1 -- 172.21.15.40:0/2392204962 wait complete. 2024-04-26T23:04:29.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.444+0000 7fc5ca9d8740 20 client.6252 trim_cache size 0 max 16384 2024-04-26T23:04:29.447 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:04:29.447 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:04:29.447 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-04-26T23:04:29.447 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6027, 2024-04-26T23:04:29.447 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:04:29.448 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:04:29.448 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:04:29.448 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.2941340060000002, 2024-04-26T23:04:29.448 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 40, 2024-04-26T23:04:29.448 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 52, 2024-04-26T23:04:29.448 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 52, 2024-04-26T23:04:29.448 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 17.768662826 2024-04-26T23:04:29.448 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:04:29.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.444+0000 7fc5c6400640 1 -- 172.21.15.40:0/1013540888 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc5a006c8f0 msgr2=0x7fc5a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.444+0000 7fc5c6400640 1 --2- 172.21.15.40:0/1013540888 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc5a006c8f0 0x7fc5a006edb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4032000 tx=0x7fc5b402e000 comp rx=0 tx=0).stop 2024-04-26T23:04:29.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.444+0000 7fc5c6400640 1 -- 172.21.15.40:0/1013540888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5c014cfc0 msgr2=0x7fc5c014d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.444+0000 7fc5c6400640 1 --2- 172.21.15.40:0/1013540888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5c014cfc0 0x7fc5c014d420 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0074b10 tx=0x7fc5b800f4a0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.444+0000 7fc5c6400640 1 -- 172.21.15.40:0/1013540888 shutdown_connections 2024-04-26T23:04:29.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.444+0000 7fc5c6400640 1 --2- 172.21.15.40:0/1013540888 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc5a006c8f0 0x7fc5a006edb0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.444+0000 7fc5c6400640 1 --2- 172.21.15.40:0/1013540888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5c014cfc0 0x7fc5c014d420 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.444+0000 7fc5c6400640 1 --2- 172.21.15.40:0/1013540888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5c006dd30 0x7fc5c006e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.444+0000 7fc5c6400640 1 --2- 172.21.15.40:0/1013540888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5a80039e0 0x7fc5c014a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.444+0000 7fc5c6400640 1 -- 172.21.15.40:0/1013540888 >> 172.21.15.40:0/1013540888 conn(0x7fc5c009ba50 msgr2=0x7fc5c00b0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.445+0000 7fc5c6400640 1 -- 172.21.15.40:0/1013540888 shutdown_connections 2024-04-26T23:04:29.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.445+0000 7fc5c6400640 1 -- 172.21.15.40:0/1013540888 wait complete. 2024-04-26T23:04:29.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T23:04:29.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.654+0000 7f4b3a000640 1 Processor -- start 2024-04-26T23:04:29.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.654+0000 7f4b3a000640 1 -- start start 2024-04-26T23:04:29.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.655+0000 7f4b3a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4b340c6f10 0x7f4b340c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b3a000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4b340cc980 0x7f4b340cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b3a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4b340c6010 0x7f4b340cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b3a000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f4b340c72f0 con 0x7f4b340c6f10 2024-04-26T23:04:29.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b3a000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4b340cb860 con 0x7f4b340cc980 2024-04-26T23:04:29.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b3a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4b340cce10 con 0x7f4b340c6010 2024-04-26T23:04:29.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b38c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4b340c6f10 0x7f4b340c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b38c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4b340c6f10 0x7f4b340c5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56862/0 (socket says 172.21.15.40:56862) 2024-04-26T23:04:29.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b38c00640 1 -- 172.21.15.40:0/625365651 learned_addr learned my addr 172.21.15.40:0/625365651 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b33e00640 1 --2- 172.21.15.40:0/625365651 >> v2:172.21.15.103:3300/0 conn(0x7f4b340cc980 0x7f4b340cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b38c00640 1 -- 172.21.15.40:0/625365651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4b340c6010 msgr2=0x7f4b340cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b38c00640 1 --2- 172.21.15.40:0/625365651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4b340c6010 0x7f4b340cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b38c00640 1 -- 172.21.15.40:0/625365651 >> v2:172.21.15.103:3300/0 conn(0x7f4b340cc980 msgr2=0x7f4b340cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b38c00640 1 --2- 172.21.15.40:0/625365651 >> v2:172.21.15.103:3300/0 conn(0x7f4b340cc980 0x7f4b340cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b38c00640 1 -- 172.21.15.40:0/625365651 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b340cd0f0 con 0x7f4b340c6f10 2024-04-26T23:04:29.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.656+0000 7f4b33e00640 1 --2- 172.21.15.40:0/625365651 >> v2:172.21.15.103:3300/0 conn(0x7f4b340cc980 0x7f4b340cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:29.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.658+0000 7f4b38c00640 1 --2- 172.21.15.40:0/625365651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b340c6f10 0x7f4b340c5ad0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4b340b0460 tx=0x7f4b2402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63254426bae3165 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.658+0000 7f4b33400640 1 -- 172.21.15.40:0/625365651 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b24037070 con 0x7f4b340c6f10 2024-04-26T23:04:29.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.658+0000 7f4b33400640 1 -- 172.21.15.40:0/625365651 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b24032440 con 0x7f4b340c6f10 2024-04-26T23:04:29.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.658+0000 7f4b33400640 1 -- 172.21.15.40:0/625365651 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b24032810 con 0x7f4b340c6f10 2024-04-26T23:04:29.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.659+0000 7f4b3a000640 1 -- 172.21.15.40:0/625365651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b340c6f10 msgr2=0x7f4b340c5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.659+0000 7f4b3a000640 1 --2- 172.21.15.40:0/625365651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b340c6f10 0x7f4b340c5ad0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4b340b0460 tx=0x7f4b2402a140 comp rx=0 tx=0).stop 2024-04-26T23:04:29.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.659+0000 7f4b3a000640 1 -- 172.21.15.40:0/625365651 shutdown_connections 2024-04-26T23:04:29.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.659+0000 7f4b3a000640 1 --2- 172.21.15.40:0/625365651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4b340c6010 0x7f4b340cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.659+0000 7f4b3a000640 1 --2- 172.21.15.40:0/625365651 >> v2:172.21.15.103:3300/0 conn(0x7f4b340cc980 0x7f4b340cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.659+0000 7f4b3a000640 1 --2- 172.21.15.40:0/625365651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b340c6f10 0x7f4b340c5ad0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.659+0000 7f4b3a000640 1 -- 172.21.15.40:0/625365651 >> 172.21.15.40:0/625365651 conn(0x7f4b340d0a10 msgr2=0x7f4b340d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.659+0000 7f4b3a000640 1 -- 172.21.15.40:0/625365651 shutdown_connections 2024-04-26T23:04:29.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.659+0000 7f4b3a000640 1 -- 172.21.15.40:0/625365651 wait complete. 2024-04-26T23:04:29.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.661+0000 7f4b3a000640 1 Processor -- start 2024-04-26T23:04:29.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.661+0000 7f4b3a000640 1 -- start start 2024-04-26T23:04:29.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.662+0000 7f4b3a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b340c6010 0x7f4b340b6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.662+0000 7f4b3a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b340c6f10 0x7f4b3415bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.662+0000 7f4b3a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4b3415c2d0 0x7f4b340c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.662+0000 7f4b3a000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b340c76a0 con 0x7f4b340c6010 2024-04-26T23:04:29.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.662+0000 7f4b3a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4b340cb1a0 con 0x7f4b340c6f10 2024-04-26T23:04:29.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.662+0000 7f4b3a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4b340c7820 con 0x7f4b3415c2d0 2024-04-26T23:04:29.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.662+0000 7f4b38c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b340c6010 0x7f4b340b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.662+0000 7f4b38c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b340c6010 0x7f4b340b6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56864/0 (socket says 172.21.15.40:56864) 2024-04-26T23:04:29.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.662+0000 7f4b38c00640 1 -- 172.21.15.40:0/4036156592 learned_addr learned my addr 172.21.15.40:0/4036156592 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.663+0000 7f4b38c00640 1 -- 172.21.15.40:0/4036156592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4b3415c2d0 msgr2=0x7f4b340c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.663+0000 7f4b38c00640 1 --2- 172.21.15.40:0/4036156592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4b3415c2d0 0x7f4b340c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.663+0000 7f4b38c00640 1 -- 172.21.15.40:0/4036156592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b340c6f10 msgr2=0x7f4b3415bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.663+0000 7f4b33e00640 1 --2- 172.21.15.40:0/4036156592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b340c6f10 0x7f4b3415bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.663+0000 7f4b38c00640 1 --2- 172.21.15.40:0/4036156592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b340c6f10 0x7f4b3415bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.663+0000 7f4b38c00640 1 -- 172.21.15.40:0/4036156592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b3415c7a0 con 0x7f4b340c6010 2024-04-26T23:04:29.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.663+0000 7f4b38c00640 1 --2- 172.21.15.40:0/4036156592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b340c6010 0x7f4b340b6380 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f4b24000c00 tx=0x7f4b240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.664+0000 7f4b30c00640 1 -- 172.21.15.40:0/4036156592 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b24041020 con 0x7f4b340c6010 2024-04-26T23:04:29.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.664+0000 7f4b30c00640 1 -- 172.21.15.40:0/4036156592 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b24030970 con 0x7f4b340c6010 2024-04-26T23:04:29.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.664+0000 7f4b3a000640 1 -- 172.21.15.40:0/4036156592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b3415ca80 con 0x7f4b340c6010 2024-04-26T23:04:29.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.664+0000 7f4b30c00640 1 -- 172.21.15.40:0/4036156592 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b24037040 con 0x7f4b340c6010 2024-04-26T23:04:29.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.664+0000 7f4b3a000640 1 -- 172.21.15.40:0/4036156592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b340b42c0 con 0x7f4b340c6010 2024-04-26T23:04:29.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.666+0000 7f4b30c00640 1 -- 172.21.15.40:0/4036156592 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4b24044020 con 0x7f4b340c6010 2024-04-26T23:04:29.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.667+0000 7f4b30c00640 1 --2- 172.21.15.40:0/4036156592 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4b1406c8f0 0x7f4b1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.667+0000 7f4b30c00640 1 -- 172.21.15.40:0/4036156592 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f4b240ac7a0 con 0x7f4b340c6010 2024-04-26T23:04:29.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.667+0000 7f4b3e3bf740 1 Processor -- start 2024-04-26T23:04:29.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.668+0000 7f4b33e00640 1 --2- 172.21.15.40:0/4036156592 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4b1406c8f0 0x7f4b1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.668+0000 7f4b3e3bf740 1 -- start start 2024-04-26T23:04:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b3e3bf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 0x555ac96d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b3e3bf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac970be80 0x555ac96cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b3e3bf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac96fa270 0x555ac96fa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b3e3bf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555ac9633600 con 0x555ac970be80 2024-04-26T23:04:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b3e3bf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555ac95d0100 con 0x555ac96fa270 2024-04-26T23:04:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b3e3bf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x555ac962f5c0 con 0x555ac96cf7b0 2024-04-26T23:04:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b33e00640 1 --2- 172.21.15.40:0/4036156592 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4b1406c8f0 0x7f4b1406edb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4b340b3d00 tx=0x7f4b20008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b33e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac970be80 0x555ac96cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b33e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac970be80 0x555ac96cfb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56868/0 (socket says 172.21.15.40:56868) 2024-04-26T23:04:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b33e00640 1 -- 172.21.15.40:0/737608893 learned_addr learned my addr 172.21.15.40:0/737608893 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b38c00640 1 --2- 172.21.15.40:0/737608893 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 0x555ac96d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b39600640 1 --2- 172.21.15.40:0/737608893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac96fa270 0x555ac96fa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b33e00640 1 -- 172.21.15.40:0/737608893 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 msgr2=0x555ac96d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b33e00640 1 --2- 172.21.15.40:0/737608893 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 0x555ac96d1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b33e00640 1 -- 172.21.15.40:0/737608893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac96fa270 msgr2=0x555ac96fa6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.669+0000 7f4b33e00640 1 --2- 172.21.15.40:0/737608893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac96fa270 0x555ac96fa6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.670+0000 7f4b33e00640 1 -- 172.21.15.40:0/737608893 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ac93452d0 con 0x555ac970be80 2024-04-26T23:04:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.670+0000 7f4b33e00640 1 --2- 172.21.15.40:0/737608893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac970be80 0x555ac96cfb90 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f4b2000db60 tx=0x7f4b20010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ad1315bccbab5c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.670+0000 7f4b29200640 1 -- 172.21.15.40:0/737608893 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b20019050 con 0x555ac970be80 2024-04-26T23:04:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.670+0000 7f4b29200640 1 -- 172.21.15.40:0/737608893 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b20023070 con 0x555ac970be80 2024-04-26T23:04:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.670+0000 7f4b29200640 1 -- 172.21.15.40:0/737608893 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2000de60 con 0x555ac970be80 2024-04-26T23:04:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.671+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/737608893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac970be80 msgr2=0x555ac96cfb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.671+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/737608893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac970be80 0x555ac96cfb90 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f4b2000db60 tx=0x7f4b20010000 comp rx=0 tx=0).stop 2024-04-26T23:04:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.671+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/737608893 shutdown_connections 2024-04-26T23:04:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.671+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/737608893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac96fa270 0x555ac96fa6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.671+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/737608893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac970be80 0x555ac96cfb90 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.671+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/737608893 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 0x555ac96d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.671+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/737608893 >> 172.21.15.40:0/737608893 conn(0x555ac9663d10 msgr2=0x555ac9715270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.671+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/737608893 shutdown_connections 2024-04-26T23:04:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.671+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/737608893 wait complete. 2024-04-26T23:04:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.672+0000 7f4b3e3bf740 1 -- start start 2024-04-26T23:04:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.672+0000 7f4b3e3bf740 1 Processor -- start 2024-04-26T23:04:29.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.673+0000 7f4b3e3bf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 0x555ac970cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.673+0000 7f4b3e3bf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 0x555ac96fcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.673+0000 7f4b3e3bf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac970be80 0x555ac96e5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.673+0000 7f4b3e3bf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555ac95baf70 con 0x555ac96fa270 2024-04-26T23:04:29.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.673+0000 7f4b3e3bf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555ac96f7dc0 con 0x555ac970be80 2024-04-26T23:04:29.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.673+0000 7f4b3e3bf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x555ac96e6890 con 0x555ac96cf7b0 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.673+0000 7f4b3e3bf740 10 client.0 _ll_register_callbacks cb 0x555ac9629480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.673+0000 7f4b33e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 0x555ac96fcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.673+0000 7f4b33e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 0x555ac96fcac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56880/0 (socket says 172.21.15.40:56880) 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.673+0000 7f4b33e00640 1 -- 172.21.15.40:0/900554124 learned_addr learned my addr 172.21.15.40:0/900554124 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.674+0000 7f4b38c00640 1 --2- 172.21.15.40:0/900554124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 0x555ac970cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.674+0000 7f4b33e00640 1 -- 172.21.15.40:0/900554124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 msgr2=0x555ac970cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.674+0000 7f4b33e00640 1 --2- 172.21.15.40:0/900554124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 0x555ac970cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.674+0000 7f4b33e00640 1 -- 172.21.15.40:0/900554124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac970be80 msgr2=0x555ac96e5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.674+0000 7f4b39600640 1 --2- 172.21.15.40:0/900554124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac970be80 0x555ac96e5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.674+0000 7f4b33e00640 1 --2- 172.21.15.40:0/900554124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac970be80 0x555ac96e5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.674+0000 7f4b33e00640 1 -- 172.21.15.40:0/900554124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ac940f6b0 con 0x555ac96fa270 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.674+0000 7f4b33e00640 1 --2- 172.21.15.40:0/900554124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 0x555ac96fcac0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4b20008010 tx=0x7f4b20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=433e43c94cb0beac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.674+0000 7f4b1d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.674+0000 7f4b1d600640 1 -- 172.21.15.40:0/900554124 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2000de60 con 0x555ac96fa270 2024-04-26T23:04:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.675+0000 7f4b1d600640 1 -- 172.21.15.40:0/900554124 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f4b20033c70 con 0x555ac96fa270 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.675+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/900554124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b20023070 con 0x555ac96fa270 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.675+0000 7f4b1d600640 1 -- 172.21.15.40:0/900554124 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f4b20003610 con 0x555ac96fa270 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.675+0000 7f4b3e3bf740 10 client.6264 fetch_fsmap learned FSMap version 40 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.675+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/900554124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) -- 0x555ac96fc2c0 con 0x555ac96fa270 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.675+0000 7f4b1d600640 1 -- 172.21.15.40:0/900554124 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b20023070 con 0x555ac96fa270 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.675+0000 7f4b1d600640 1 -- 172.21.15.40:0/900554124 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 3534+0+0 (secure 0 0 0) 0x7f4b2003a050 con 0x555ac96fa270 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.675+0000 7f4b3e3bf740 10 client.6264 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.675+0000 7f4b3e3bf740 10 client.6264 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.675+0000 7f4b3e3bf740 20 client.6264 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.675+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/900554124 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x555ac96e7f70 0x555ac96e8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.675+0000 7f4b3e3bf740 4 client.6264 mds_command: new command op to 6036 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.675+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/900554124 --> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ac9641210 con 0x555ac96e7f70 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.676+0000 7f4b38c00640 1 --2- 172.21.15.40:0/900554124 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x555ac96e7f70 0x555ac96e8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.676+0000 7f4b38c00640 1 --2- 172.21.15.40:0/900554124 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x555ac96e7f70 0x555ac96e8350 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2af59da01aacb6c server_cookie=ec3722a07eabe854 in_seq=0 out_seq=0 2024-04-26T23:04:29.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.676+0000 7f4b1d600640 10 client.6264 ms_handle_connect on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:29.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.677+0000 7f4b1d600640 1 -- 172.21.15.40:0/900554124 <== mds.0 v2:172.21.15.40:6833/618018110 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555ac9641210 con 0x555ac96e7f70 2024-04-26T23:04:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.677+0000 7f4b1d600640 10 client.6264 handle_command_reply: tid=0 2024-04-26T23:04:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.677+0000 7f4b3e3bf740 1 client.6264 shutdown 2024-04-26T23:04:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.677+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/900554124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 msgr2=0x555ac96fcac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.677+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/900554124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 0x555ac96fcac0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4b20008010 tx=0x7f4b20004730 comp rx=0 tx=0).stop 2024-04-26T23:04:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.677+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/900554124 shutdown_connections 2024-04-26T23:04:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.677+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/900554124 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x555ac96e7f70 0x555ac96e8350 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.677+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/900554124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac970be80 0x555ac96e5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.677+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/900554124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 0x555ac96fcac0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.677+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/900554124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 0x555ac970cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.677+0000 7f4b1d600640 0 client.6264 ms_handle_reset on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.677+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/900554124 >> 172.21.15.40:0/900554124 conn(0x555ac9663d10 msgr2=0x555ac96f7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.678+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/900554124 shutdown_connections 2024-04-26T23:04:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.678+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/900554124 wait complete. 2024-04-26T23:04:29.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.678+0000 7f4b3e3bf740 20 client.6264 trim_cache size 0 max 16384 2024-04-26T23:04:29.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.697+0000 7f4b3e3bf740 1 Processor -- start 2024-04-26T23:04:29.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.697+0000 7f4b3e3bf740 1 -- start start 2024-04-26T23:04:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.698+0000 7f4b3e3bf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96cf7b0 0x555ac96f9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.698+0000 7f4b3e3bf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac96fa270 0x555ac9789b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.698+0000 7f4b3e3bf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac970be80 0x555ac978a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.698+0000 7f4b3e3bf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b34200e20 con 0x555ac96cf7b0 2024-04-26T23:04:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.698+0000 7f4b3e3bf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4b342014f0 con 0x555ac96fa270 2024-04-26T23:04:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.698+0000 7f4b3e3bf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4b340b55a0 con 0x555ac970be80 2024-04-26T23:04:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.698+0000 7f4b38c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96cf7b0 0x555ac96f9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.698+0000 7f4b38c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96cf7b0 0x555ac96f9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56882/0 (socket says 172.21.15.40:56882) 2024-04-26T23:04:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.698+0000 7f4b38c00640 1 -- 172.21.15.40:0/3989255814 learned_addr learned my addr 172.21.15.40:0/3989255814 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.699+0000 7f4b38c00640 1 -- 172.21.15.40:0/3989255814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac970be80 msgr2=0x555ac978a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.699+0000 7f4b38c00640 1 --2- 172.21.15.40:0/3989255814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac970be80 0x555ac978a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.699+0000 7f4b38c00640 1 -- 172.21.15.40:0/3989255814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac96fa270 msgr2=0x555ac9789b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.699+0000 7f4b38c00640 1 --2- 172.21.15.40:0/3989255814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac96fa270 0x555ac9789b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.699+0000 7f4b38c00640 1 -- 172.21.15.40:0/3989255814 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ac96e57b0 con 0x555ac96cf7b0 2024-04-26T23:04:29.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.699+0000 7f4b38c00640 1 --2- 172.21.15.40:0/3989255814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96cf7b0 0x555ac96f9800 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4b2407a230 tx=0x7f4b240aa970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6af38f3b904997 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.699+0000 7f4b1fe00640 1 -- 172.21.15.40:0/3989255814 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b24082e80 con 0x555ac96cf7b0 2024-04-26T23:04:29.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.699+0000 7f4b1fe00640 1 -- 172.21.15.40:0/3989255814 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b240aabb0 con 0x555ac96cf7b0 2024-04-26T23:04:29.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.699+0000 7f4b1fe00640 1 -- 172.21.15.40:0/3989255814 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b24030d40 con 0x555ac96cf7b0 2024-04-26T23:04:29.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.700+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/3989255814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96cf7b0 msgr2=0x555ac96f9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.700+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/3989255814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96cf7b0 0x555ac96f9800 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4b2407a230 tx=0x7f4b240aa970 comp rx=0 tx=0).stop 2024-04-26T23:04:29.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.700+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/3989255814 shutdown_connections 2024-04-26T23:04:29.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.700+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/3989255814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac970be80 0x555ac978a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.700+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/3989255814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac96fa270 0x555ac9789b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.700+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/3989255814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96cf7b0 0x555ac96f9800 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.700+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/3989255814 >> 172.21.15.40:0/3989255814 conn(0x555ac9663d10 msgr2=0x555ac96fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.700+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/3989255814 shutdown_connections 2024-04-26T23:04:29.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.700+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/3989255814 wait complete. 2024-04-26T23:04:29.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.701+0000 7f4b3e3bf740 1 -- start start 2024-04-26T23:04:29.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.701+0000 7f4b3e3bf740 1 Processor -- start 2024-04-26T23:04:29.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.702+0000 7f4b3e3bf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 0x555ac9723990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.702+0000 7f4b3e3bf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 0x555ac97717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.702+0000 7f4b3e3bf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac970be80 0x555ac970c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.702+0000 7f4b3e3bf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555ac96e59a0 con 0x555ac96fa270 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.702+0000 7f4b3e3bf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555ac96fc5f0 con 0x555ac970be80 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.702+0000 7f4b3e3bf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x555ac970f780 con 0x555ac96cf7b0 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.702+0000 7f4b3e3bf740 10 client.0 _ll_register_callbacks cb 0x7f4b2000da30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.702+0000 7f4b33e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 0x555ac97717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.702+0000 7f4b33e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 0x555ac97717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56884/0 (socket says 172.21.15.40:56884) 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.702+0000 7f4b33e00640 1 -- 172.21.15.40:0/4213379095 learned_addr learned my addr 172.21.15.40:0/4213379095 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.703+0000 7f4b33e00640 1 -- 172.21.15.40:0/4213379095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 msgr2=0x555ac9723990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.703+0000 7f4b33e00640 1 --2- 172.21.15.40:0/4213379095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 0x555ac9723990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.703+0000 7f4b39600640 1 --2- 172.21.15.40:0/4213379095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac970be80 0x555ac970c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.703+0000 7f4b33e00640 1 -- 172.21.15.40:0/4213379095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac970be80 msgr2=0x555ac970c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.703+0000 7f4b33e00640 1 --2- 172.21.15.40:0/4213379095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac970be80 0x555ac970c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.703+0000 7f4b33e00640 1 -- 172.21.15.40:0/4213379095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ac9724290 con 0x555ac96fa270 2024-04-26T23:04:29.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.703+0000 7f4b33e00640 1 --2- 172.21.15.40:0/4213379095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 0x555ac97717a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f4b2000d780 tx=0x7f4b200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7db0d3f9d1d12d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.703+0000 7f4b1d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:29.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.703+0000 7f4b1d600640 1 -- 172.21.15.40:0/4213379095 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b20019070 con 0x555ac96fa270 2024-04-26T23:04:29.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.703+0000 7f4b1d600640 1 -- 172.21.15.40:0/4213379095 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f4b200039c0 con 0x555ac96fa270 2024-04-26T23:04:29.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.703+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/4213379095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b240aabb0 con 0x555ac96fa270 2024-04-26T23:04:29.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.704+0000 7f4b1d600640 1 -- 172.21.15.40:0/4213379095 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f4b20048050 con 0x555ac96fa270 2024-04-26T23:04:29.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.704+0000 7f4b3e3bf740 10 client.6270 fetch_fsmap learned FSMap version 40 2024-04-26T23:04:29.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.704+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/4213379095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) -- 0x555ac9774b10 con 0x555ac96fa270 2024-04-26T23:04:29.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.704+0000 7f4b1d600640 1 -- 172.21.15.40:0/4213379095 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b240aabb0 con 0x555ac96fa270 2024-04-26T23:04:29.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.704+0000 7f4b1d600640 1 -- 172.21.15.40:0/4213379095 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 3534+0+0 (secure 0 0 0) 0x7f4b2004b020 con 0x555ac96fa270 2024-04-26T23:04:29.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.704+0000 7f4b3e3bf740 10 client.6270 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T23:04:29.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.704+0000 7f4b3e3bf740 10 client.6270 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:04:29.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.704+0000 7f4b3e3bf740 20 client.6270 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:29.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.704+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/4213379095 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x555ac9774630 0x555ac9727250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.704+0000 7f4b3e3bf740 4 client.6270 mds_command: new command op to 6036 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.704+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/4213379095 --> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] -- command(tid 0: {"prefix": "status"}) -- 0x555ac963b080 con 0x555ac9774630 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.704+0000 7f4b38c00640 1 --2- 172.21.15.40:0/4213379095 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x555ac9774630 0x555ac9727250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.705+0000 7f4b38c00640 1 --2- 172.21.15.40:0/4213379095 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x555ac9774630 0x555ac9727250 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1c5af722bef6434 server_cookie=7d2a367e481844b3 in_seq=0 out_seq=0 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.705+0000 7f4b1d600640 10 client.6270 ms_handle_connect on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.705+0000 7f4b1d600640 1 -- 172.21.15.40:0/4213379095 <== mds.0 v2:172.21.15.40:6833/618018110 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x555ac963b080 con 0x555ac9774630 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.705+0000 7f4b1d600640 10 client.6270 handle_command_reply: tid=0 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.705+0000 7f4b3e3bf740 1 client.6270 shutdown 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.705+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/4213379095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 msgr2=0x555ac97717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.705+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/4213379095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 0x555ac97717a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f4b2000d780 tx=0x7f4b200046c0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.705+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/4213379095 shutdown_connections 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.705+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/4213379095 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x555ac9774630 0x555ac9727250 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.705+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/4213379095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ac970be80 0x555ac970c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.705+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/4213379095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ac96fa270 0x555ac97717a0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.705+0000 7f4b1d600640 0 client.6270 ms_handle_reset on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:29.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.705+0000 7f4b3e3bf740 1 --2- 172.21.15.40:0/4213379095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ac96cf7b0 0x555ac9723990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.706+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/4213379095 >> 172.21.15.40:0/4213379095 conn(0x555ac9663d10 msgr2=0x555ac9718eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.706+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/4213379095 shutdown_connections 2024-04-26T23:04:29.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.706+0000 7f4b3e3bf740 1 -- 172.21.15.40:0/4213379095 wait complete. 2024-04-26T23:04:29.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.706+0000 7f4b3e3bf740 20 client.6270 trim_cache size 0 max 16384 2024-04-26T23:04:29.709 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:04:29.709 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:04:29.709 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:04:29.709 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6036, 2024-04-26T23:04:29.709 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:04:29.709 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:04:29.710 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:04:29.710 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 12.63155147, 2024-04-26T23:04:29.710 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 40, 2024-04-26T23:04:29.710 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 52, 2024-04-26T23:04:29.710 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 52, 2024-04-26T23:04:29.710 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 18.029861136000001 2024-04-26T23:04:29.710 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:04:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.706+0000 7f4b3a000640 1 -- 172.21.15.40:0/4036156592 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4b1406c8f0 msgr2=0x7f4b1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.706+0000 7f4b3a000640 1 --2- 172.21.15.40:0/4036156592 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4b1406c8f0 0x7f4b1406edb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4b340b3d00 tx=0x7f4b20008040 comp rx=0 tx=0).stop 2024-04-26T23:04:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.706+0000 7f4b3a000640 1 -- 172.21.15.40:0/4036156592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b340c6010 msgr2=0x7f4b340b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.706+0000 7f4b3a000640 1 --2- 172.21.15.40:0/4036156592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b340c6010 0x7f4b340b6380 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f4b24000c00 tx=0x7f4b240046c0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.707+0000 7f4b3a000640 1 -- 172.21.15.40:0/4036156592 shutdown_connections 2024-04-26T23:04:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.707+0000 7f4b3a000640 1 --2- 172.21.15.40:0/4036156592 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4b1406c8f0 0x7f4b1406edb0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.707+0000 7f4b3a000640 1 --2- 172.21.15.40:0/4036156592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4b3415c2d0 0x7f4b340c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.707+0000 7f4b3a000640 1 --2- 172.21.15.40:0/4036156592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b340c6f10 0x7f4b3415bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.707+0000 7f4b3a000640 1 --2- 172.21.15.40:0/4036156592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b340c6010 0x7f4b340b6380 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.707+0000 7f4b3a000640 1 -- 172.21.15.40:0/4036156592 >> 172.21.15.40:0/4036156592 conn(0x7f4b340d0a10 msgr2=0x7f4b3406ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.707+0000 7f4b3a000640 1 -- 172.21.15.40:0/4036156592 shutdown_connections 2024-04-26T23:04:29.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.707+0000 7f4b3a000640 1 -- 172.21.15.40:0/4036156592 wait complete. 2024-04-26T23:04:29.724 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:04:29.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.929+0000 7f5e00a00640 1 Processor -- start 2024-04-26T23:04:29.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.929+0000 7f5e00a00640 1 -- start start 2024-04-26T23:04:29.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.930+0000 7f5e00a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5dfc071e60 0x7f5dfc06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.930+0000 7f5e00a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dfc0cc480 0x7f5dfc166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.931+0000 7f5e00a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dfc15d7c0 con 0x7f5dfc06fe20 2024-04-26T23:04:29.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.931+0000 7f5e00a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f5dfc15daa0 con 0x7f5dfc071e60 2024-04-26T23:04:29.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.931+0000 7f5e00a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5dfc15dd80 con 0x7f5dfc0cc480 2024-04-26T23:04:29.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.931+0000 7f5df9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5dfc071e60 0x7f5dfc06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.931+0000 7f5df9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5dfc071e60 0x7f5dfc06f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56888/0 (socket says 172.21.15.40:56888) 2024-04-26T23:04:29.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.931+0000 7f5df9e00640 1 -- 172.21.15.40:0/2811014823 learned_addr learned my addr 172.21.15.40:0/2811014823 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.931+0000 7f5dfa800640 1 --2- 172.21.15.40:0/2811014823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dfc0cc480 0x7f5dfc166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.932+0000 7f5df9e00640 1 -- 172.21.15.40:0/2811014823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dfc0cc480 msgr2=0x7f5dfc166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:29.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.932+0000 7f5df9e00640 1 --2- 172.21.15.40:0/2811014823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dfc0cc480 0x7f5dfc166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.932+0000 7f5df9e00640 1 -- 172.21.15.40:0/2811014823 >> v1:172.21.15.40:6789/0 conn(0x7f5dfc06fe20 legacy=0x7f5dfc072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.932+0000 7f5df9e00640 1 -- 172.21.15.40:0/2811014823 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dfc15e060 con 0x7f5dfc071e60 2024-04-26T23:04:29.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.932+0000 7f5df9400640 1 -- 172.21.15.40:0/2811014823 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2024288983 0 0) 0x7f5dfc15d7c0 con 0x7f5dfc06fe20 2024-04-26T23:04:29.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.933+0000 7f5df9e00640 1 --2- 172.21.15.40:0/2811014823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dfc071e60 0x7f5dfc06f8e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f5dec009870 tx=0x7f5dec02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79048bf54ef0d9d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.933+0000 7f5df9400640 1 -- 172.21.15.40:0/2811014823 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec037070 con 0x7f5dfc071e60 2024-04-26T23:04:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.933+0000 7f5df9400640 1 -- 172.21.15.40:0/2811014823 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dec032440 con 0x7f5dfc071e60 2024-04-26T23:04:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.933+0000 7f5df9400640 1 -- 172.21.15.40:0/2811014823 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec032810 con 0x7f5dfc071e60 2024-04-26T23:04:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.934+0000 7f5e00a00640 1 -- 172.21.15.40:0/2811014823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dfc071e60 msgr2=0x7f5dfc06f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.934+0000 7f5e00a00640 1 --2- 172.21.15.40:0/2811014823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dfc071e60 0x7f5dfc06f8e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f5dec009870 tx=0x7f5dec02a250 comp rx=0 tx=0).stop 2024-04-26T23:04:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.934+0000 7f5e00a00640 1 -- 172.21.15.40:0/2811014823 shutdown_connections 2024-04-26T23:04:29.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.934+0000 7f5e00a00640 1 --2- 172.21.15.40:0/2811014823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dfc0cc480 0x7f5dfc166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.934+0000 7f5e00a00640 1 --2- 172.21.15.40:0/2811014823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dfc071e60 0x7f5dfc06f8e0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.934+0000 7f5e00a00640 1 -- 172.21.15.40:0/2811014823 >> 172.21.15.40:0/2811014823 conn(0x7f5dfc0d0be0 msgr2=0x7f5dfc0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:29.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.934+0000 7f5e00a00640 1 -- 172.21.15.40:0/2811014823 shutdown_connections 2024-04-26T23:04:29.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.934+0000 7f5e00a00640 1 -- 172.21.15.40:0/2811014823 wait complete. 2024-04-26T23:04:29.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.936+0000 7f5e00a00640 1 Processor -- start 2024-04-26T23:04:29.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.936+0000 7f5e00a00640 1 -- start start 2024-04-26T23:04:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.937+0000 7f5e00a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dfc06fe20 0x7f5dfc151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.937+0000 7f5e00a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dfc071e60 0x7f5dfc0b5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.937+0000 7f5e00a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dfc0cc480 0x7f5dfc159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.937+0000 7f5e00a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc15d570 con 0x7f5dfc0cc480 2024-04-26T23:04:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.937+0000 7f5e00a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc15d3f0 con 0x7f5dfc071e60 2024-04-26T23:04:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.937+0000 7f5e00a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5dfc072950 con 0x7f5dfc06fe20 2024-04-26T23:04:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.937+0000 7f5dfa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dfc0cc480 0x7f5dfc159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.937+0000 7f5dfa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dfc0cc480 0x7f5dfc159d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56900/0 (socket says 172.21.15.40:56900) 2024-04-26T23:04:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.937+0000 7f5dfa800640 1 -- 172.21.15.40:0/3637514036 learned_addr learned my addr 172.21.15.40:0/3637514036 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.937+0000 7f5dfa800640 1 -- 172.21.15.40:0/3637514036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dfc06fe20 msgr2=0x7f5dfc151c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.937+0000 7f5dfa800640 1 --2- 172.21.15.40:0/3637514036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dfc06fe20 0x7f5dfc151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.937+0000 7f5dfa800640 1 -- 172.21.15.40:0/3637514036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dfc071e60 msgr2=0x7f5dfc0b5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.937+0000 7f5df3e00640 1 --2- 172.21.15.40:0/3637514036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dfc071e60 0x7f5dfc0b5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.938+0000 7f5dfa800640 1 --2- 172.21.15.40:0/3637514036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dfc071e60 0x7f5dfc0b5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.938+0000 7f5dfa800640 1 -- 172.21.15.40:0/3637514036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dfc0b4330 con 0x7f5dfc0cc480 2024-04-26T23:04:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.938+0000 7f5dfa800640 1 --2- 172.21.15.40:0/3637514036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dfc0cc480 0x7f5dfc159d50 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009fd0 tx=0x7f5df400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.938+0000 7f5df2000640 1 -- 172.21.15.40:0/3637514036 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4014420 con 0x7f5dfc0cc480 2024-04-26T23:04:29.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.938+0000 7f5df2000640 1 -- 172.21.15.40:0/3637514036 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5df4019070 con 0x7f5dfc0cc480 2024-04-26T23:04:29.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.938+0000 7f5df2000640 1 -- 172.21.15.40:0/3637514036 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4014800 con 0x7f5dfc0cc480 2024-04-26T23:04:29.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.938+0000 7f5e00a00640 1 -- 172.21.15.40:0/3637514036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dfc0c9f80 con 0x7f5dfc0cc480 2024-04-26T23:04:29.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.938+0000 7f5e00a00640 1 -- 172.21.15.40:0/3637514036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dfc0b64b0 con 0x7f5dfc0cc480 2024-04-26T23:04:29.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.939+0000 7f5e00a00640 1 -- 172.21.15.40:0/3637514036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dfc0694f0 con 0x7f5dfc0cc480 2024-04-26T23:04:29.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.940+0000 7f5df2000640 1 -- 172.21.15.40:0/3637514036 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5df40149a0 con 0x7f5dfc0cc480 2024-04-26T23:04:29.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.941+0000 7f5df2000640 1 --2- 172.21.15.40:0/3637514036 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5dd806c8f0 0x7f5dd806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:29.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.941+0000 7f5df2000640 1 -- 172.21.15.40:0/3637514036 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f5df408c550 con 0x7f5dfc0cc480 2024-04-26T23:04:29.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.942+0000 7f5df9e00640 1 --2- 172.21.15.40:0/3637514036 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5dd806c8f0 0x7f5dd806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:29.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.943+0000 7f5df9e00640 1 --2- 172.21.15.40:0/3637514036 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5dd806c8f0 0x7f5dd806edb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5dec034000 tx=0x7f5dec02c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:29.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:29.946+0000 7f5df2000640 1 -- 172.21.15.40:0/3637514036 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5df405a7e0 con 0x7f5dfc0cc480 2024-04-26T23:04:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.183+0000 7f5e00a00640 1 -- 172.21.15.40:0/3637514036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5dfc0ceb60 con 0x7f5dfc0cc480 2024-04-26T23:04:30.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.185+0000 7f5df2000640 1 -- 172.21.15.40:0/3637514036 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+5216 (secure 0 0 0) 0x7f5df405a0e0 con 0x7f5dfc0cc480 2024-04-26T23:04:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:04:30.185 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/189004193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":189004193},{"type":"v1","addr":"172.21.15.103:6835","nonce":189004193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:04:16.558591+0000","modified":"2024-04-26T23:04:28.159788+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6036,"mds_1":6027,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"b","rank":2,"incarnation":39,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/670995452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":670995452},{"type":"v1","addr":"172.21.15.103:6837","nonce":670995452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6027":{"gid":6027,"name":"a","rank":1,"incarnation":37,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/2304583125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2304583125},{"type":"v1","addr":"172.21.15.40:6834","nonce":2304583125}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6036":{"gid":6036,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/618018110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":618018110},{"type":"v1","addr":"172.21.15.40:6835","nonce":618018110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6036,"qdb_cluster":[5284,6027,6036]},"id":3}]} 2024-04-26T23:04:30.185 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 40 2024-04-26T23:04:30.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.187+0000 7f5e00a00640 1 -- 172.21.15.40:0/3637514036 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5dd806c8f0 msgr2=0x7f5dd806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:30.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.187+0000 7f5e00a00640 1 --2- 172.21.15.40:0/3637514036 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5dd806c8f0 0x7f5dd806edb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5dec034000 tx=0x7f5dec02c000 comp rx=0 tx=0).stop 2024-04-26T23:04:30.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.187+0000 7f5e00a00640 1 -- 172.21.15.40:0/3637514036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dfc0cc480 msgr2=0x7f5dfc159d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:30.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.187+0000 7f5e00a00640 1 --2- 172.21.15.40:0/3637514036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dfc0cc480 0x7f5dfc159d50 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009fd0 tx=0x7f5df400e4b0 comp rx=0 tx=0).stop 2024-04-26T23:04:30.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.187+0000 7f5e00a00640 1 -- 172.21.15.40:0/3637514036 shutdown_connections 2024-04-26T23:04:30.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.187+0000 7f5e00a00640 1 --2- 172.21.15.40:0/3637514036 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5dd806c8f0 0x7f5dd806edb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:30.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.187+0000 7f5e00a00640 1 --2- 172.21.15.40:0/3637514036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dfc0cc480 0x7f5dfc159d50 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:30.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.187+0000 7f5e00a00640 1 --2- 172.21.15.40:0/3637514036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dfc071e60 0x7f5dfc0b5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:30.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.187+0000 7f5e00a00640 1 --2- 172.21.15.40:0/3637514036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5dfc06fe20 0x7f5dfc151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:30.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.187+0000 7f5e00a00640 1 -- 172.21.15.40:0/3637514036 >> 172.21.15.40:0/3637514036 conn(0x7f5dfc0d0be0 msgr2=0x7f5dfc072d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:30.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.187+0000 7f5e00a00640 1 -- 172.21.15.40:0/3637514036 shutdown_connections 2024-04-26T23:04:30.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:30.187+0000 7f5e00a00640 1 -- 172.21.15.40:0/3637514036 wait complete. 2024-04-26T23:04:30.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:30.206 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p d1/d2/d3/d4/d5/d6/d7/d8) 2024-04-26T23:04:30.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:30.278 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec sync d1/d2/d3/d4/d5/d6/d7/d8) 2024-04-26T23:04:30.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:30.339 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 d1/d2) 2024-04-26T23:04:30.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:30.393 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 d1/d2/d3/d4) 2024-04-26T23:04:30.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:30.445 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 2 d1/d2/d3/d4/d5/d6) 2024-04-26T23:04:30.496 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:04:30.726 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"5","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:04:30.726 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:04:30.726 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:04:32.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:04:32.970 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"7","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:04:32.970 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:04:32.970 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:04:33.025 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:04:33.025+0000 7f47d7c00640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T23:04:34.972 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:04:35.212 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"7","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:04:35.213 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:04:35.213 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:04:37.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:04:37.462 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"13","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/d1/d2","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-04-26T23:04:37.462 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000001", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/d1/d2/d3/d4", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-04-26T23:04:37.463 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:04:37.463 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:04:37.691 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:04:37.692 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:04:37.692 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:04:39.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:04:39.942 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:04:39.943 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:04:39.943 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:04:41.945 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:04:42.194 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:04:42.194 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:04:42.194 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:04:44.196 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:04:44.434 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1,"2":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":2,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/d1/d2/d3/d4/d5/d6","dirfrag":"0x10000000005","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-04-26T23:04:44.434 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1, "2": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "2", "dirfrag": "0x10000000005", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/d1/d2/d3/d4/d5/d6", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "6", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 2, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false } ] 2024-04-26T23:04:44.435 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:04:44.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-04-26T23:04:44.664 INFO:teuthology.orchestra.run.smithi103.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"0","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dnwaiter":0,"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":2,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/d1/d2/d3/d4/d5/d6","dirfrag":"0x10000000005","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:04:44.664 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dnwaiter": 0, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "2", "dirfrag": "0x10000000005", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4/d5/d6", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 2, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:04:44.665 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:04:44.665 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 flush journal 2024-04-26T23:04:44.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.892+0000 7fda0fe00640 1 Processor -- start 2024-04-26T23:04:44.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.892+0000 7fda0fe00640 1 -- start start 2024-04-26T23:04:44.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.894+0000 7fda0fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fda100cfaf0 0x7fda10073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.894+0000 7fda0fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fda1006dd40 0x7fda10072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.894+0000 7fda0fe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda10161880 con 0x7fda10160a50 2024-04-26T23:04:44.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.894+0000 7fda0fe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fda10071ce0 con 0x7fda100cfaf0 2024-04-26T23:04:44.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.894+0000 7fda0fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fda100722a0 con 0x7fda1006dd40 2024-04-26T23:04:44.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.894+0000 7fda0e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fda100cfaf0 0x7fda10073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.894+0000 7fda0e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fda100cfaf0 0x7fda10073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47728/0 (socket says 172.21.15.40:47728) 2024-04-26T23:04:44.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.894+0000 7fda0e000640 1 -- 172.21.15.40:0/2818732123 learned_addr learned my addr 172.21.15.40:0/2818732123 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:44.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.895+0000 7fda0f400640 1 --2- 172.21.15.40:0/2818732123 >> v2:172.21.15.103:3300/0 conn(0x7fda1006dd40 0x7fda10072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.895+0000 7fda0e000640 1 -- 172.21.15.40:0/2818732123 >> v2:172.21.15.103:3300/0 conn(0x7fda1006dd40 msgr2=0x7fda10072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:44.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.895+0000 7fda0e000640 1 --2- 172.21.15.40:0/2818732123 >> v2:172.21.15.103:3300/0 conn(0x7fda1006dd40 0x7fda10072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.895+0000 7fda0e000640 1 -- 172.21.15.40:0/2818732123 >> v1:172.21.15.103:6789/0 conn(0x7fda10160a50 legacy=0x7fda10160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:44.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.895+0000 7fda0e000640 1 -- 172.21.15.40:0/2818732123 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda100cff80 con 0x7fda100cfaf0 2024-04-26T23:04:44.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.895+0000 7fda0f400640 1 --2- 172.21.15.40:0/2818732123 >> v2:172.21.15.103:3300/0 conn(0x7fda1006dd40 0x7fda10072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:44.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.896+0000 7fda0e000640 1 --2- 172.21.15.40:0/2818732123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda100cfaf0 0x7fda10073a50 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fda10146c70 tx=0x7fd9fc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd7ef30ffd49acc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.896+0000 7fda0d600640 1 -- 172.21.15.40:0/2818732123 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc037070 con 0x7fda100cfaf0 2024-04-26T23:04:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.896+0000 7fda0d600640 1 -- 172.21.15.40:0/2818732123 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9fc032440 con 0x7fda100cfaf0 2024-04-26T23:04:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.896+0000 7fda0d600640 1 -- 172.21.15.40:0/2818732123 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc032810 con 0x7fda100cfaf0 2024-04-26T23:04:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.897+0000 7fda0fe00640 1 -- 172.21.15.40:0/2818732123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda100cfaf0 msgr2=0x7fda10073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.897+0000 7fda0fe00640 1 --2- 172.21.15.40:0/2818732123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda100cfaf0 0x7fda10073a50 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fda10146c70 tx=0x7fd9fc02a230 comp rx=0 tx=0).stop 2024-04-26T23:04:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.897+0000 7fda0fe00640 1 -- 172.21.15.40:0/2818732123 shutdown_connections 2024-04-26T23:04:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.897+0000 7fda0fe00640 1 --2- 172.21.15.40:0/2818732123 >> v2:172.21.15.103:3300/0 conn(0x7fda1006dd40 0x7fda10072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.897+0000 7fda0fe00640 1 --2- 172.21.15.40:0/2818732123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda100cfaf0 0x7fda10073a50 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.897+0000 7fda0fe00640 1 -- 172.21.15.40:0/2818732123 >> 172.21.15.40:0/2818732123 conn(0x7fda100b2720 msgr2=0x7fda100b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.897+0000 7fda0fe00640 1 -- 172.21.15.40:0/2818732123 shutdown_connections 2024-04-26T23:04:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.897+0000 7fda0fe00640 1 -- 172.21.15.40:0/2818732123 wait complete. 2024-04-26T23:04:44.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.899+0000 7fda0fe00640 1 Processor -- start 2024-04-26T23:04:44.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.899+0000 7fda0fe00640 1 -- start start 2024-04-26T23:04:44.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.900+0000 7fda0fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda1006dd40 0x7fda10158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.900+0000 7fda0fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda10160a50 0x7fda10158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.900+0000 7fda0fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda10147890 0x7fda10146cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.900+0000 7fda0fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fda10071f30 con 0x7fda10147890 2024-04-26T23:04:44.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.900+0000 7fda0fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda10073f90 con 0x7fda1006dd40 2024-04-26T23:04:44.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.900+0000 7fda0fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda1006e1f0 con 0x7fda10160a50 2024-04-26T23:04:44.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.901+0000 7fda0f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda10147890 0x7fda10146cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.901+0000 7fda0f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda10147890 0x7fda10146cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47736/0 (socket says 172.21.15.40:47736) 2024-04-26T23:04:44.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.901+0000 7fda0f400640 1 -- 172.21.15.40:0/3680978241 learned_addr learned my addr 172.21.15.40:0/3680978241 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:44.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.901+0000 7fda0ea00640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda1006dd40 0x7fda10158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.901+0000 7fda0e000640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda10160a50 0x7fda10158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.901+0000 7fda0f400640 1 -- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda10160a50 msgr2=0x7fda10158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.901+0000 7fda0f400640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda10160a50 0x7fda10158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.901+0000 7fda0f400640 1 -- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda1006dd40 msgr2=0x7fda10158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.901+0000 7fda0f400640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda1006dd40 0x7fda10158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.901+0000 7fda0f400640 1 -- 172.21.15.40:0/3680978241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda100b16b0 con 0x7fda10147890 2024-04-26T23:04:44.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.901+0000 7fda0f400640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda10147890 0x7fda10146cf0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fda0800cef0 tx=0x7fda0800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:44.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.902+0000 7fda06a00640 1 -- 172.21.15.40:0/3680978241 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0801a070 con 0x7fda10147890 2024-04-26T23:04:44.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.902+0000 7fda06a00640 1 -- 172.21.15.40:0/3680978241 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda0800d860 con 0x7fda10147890 2024-04-26T23:04:44.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.902+0000 7fda06a00640 1 -- 172.21.15.40:0/3680978241 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0800dc50 con 0x7fda10147890 2024-04-26T23:04:44.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.902+0000 7fda0fe00640 1 -- 172.21.15.40:0/3680978241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda100aa700 con 0x7fda10147890 2024-04-26T23:04:44.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.902+0000 7fda0fe00640 1 -- 172.21.15.40:0/3680978241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda100d0650 con 0x7fda10147890 2024-04-26T23:04:44.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.904+0000 7fda06a00640 1 -- 172.21.15.40:0/3680978241 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fda0801d070 con 0x7fda10147890 2024-04-26T23:04:44.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.905+0000 7fda06a00640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd9f006c8f0 0x7fd9f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.905+0000 7fda06a00640 1 -- 172.21.15.40:0/3680978241 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fda0808c5c0 con 0x7fda10147890 2024-04-26T23:04:44.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.906+0000 7fda18a44740 1 Processor -- start 2024-04-26T23:04:44.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.906+0000 7fda18a44740 1 -- start start 2024-04-26T23:04:44.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.906+0000 7fda0ea00640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd9f006c8f0 0x7fd9f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.907+0000 7fda18a44740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e524134af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.907+0000 7fda18a44740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 0x55e524152500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.907+0000 7fda18a44740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e52410a3f0 0x55e524150670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.907+0000 7fda18a44740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e52401a030 con 0x55e524134710 2024-04-26T23:04:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.907+0000 7fda18a44740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e5240357f0 con 0x55e52410a3f0 2024-04-26T23:04:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.907+0000 7fda18a44740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e523ff07e0 con 0x55e524151fb0 2024-04-26T23:04:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.907+0000 7fda0ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e524134af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.907+0000 7fda0ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e524134af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47752/0 (socket says 172.21.15.40:47752) 2024-04-26T23:04:44.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.907+0000 7fda0ea00640 1 -- 172.21.15.40:0/103811510 learned_addr learned my addr 172.21.15.40:0/103811510 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:44.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.907+0000 7fda0ea00640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd9f006c8f0 0x7fd9f006edb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fda00006fd0 tx=0x7fda00008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:44.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.908+0000 7fda0ea00640 1 -- 172.21.15.40:0/103811510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 msgr2=0x55e524152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:44.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.908+0000 7fda0ea00640 1 --2- 172.21.15.40:0/103811510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 0x55e524152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.908+0000 7fda0ea00640 1 -- 172.21.15.40:0/103811510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e52410a3f0 msgr2=0x55e524150670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:44.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.908+0000 7fda0ea00640 1 --2- 172.21.15.40:0/103811510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e52410a3f0 0x55e524150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.908+0000 7fda0ea00640 1 -- 172.21.15.40:0/103811510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e523ffc050 con 0x55e524134710 2024-04-26T23:04:44.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.908+0000 7fda0ea00640 1 --2- 172.21.15.40:0/103811510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e524134af0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fda0000d8b0 tx=0x7fda00010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec17a79c59c333e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:44.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.908+0000 7fd9fb400640 1 -- 172.21.15.40:0/103811510 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00024070 con 0x55e524134710 2024-04-26T23:04:44.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.908+0000 7fd9fb400640 1 -- 172.21.15.40:0/103811510 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda0000de30 con 0x55e524134710 2024-04-26T23:04:44.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.908+0000 7fd9fb400640 1 -- 172.21.15.40:0/103811510 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00014de0 con 0x55e524134710 2024-04-26T23:04:44.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.909+0000 7fda18a44740 1 -- 172.21.15.40:0/103811510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 msgr2=0x55e524134af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.909+0000 7fda18a44740 1 --2- 172.21.15.40:0/103811510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e524134af0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fda0000d8b0 tx=0x7fda00010000 comp rx=0 tx=0).stop 2024-04-26T23:04:44.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.909+0000 7fda18a44740 1 -- 172.21.15.40:0/103811510 shutdown_connections 2024-04-26T23:04:44.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.909+0000 7fda18a44740 1 --2- 172.21.15.40:0/103811510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e52410a3f0 0x55e524150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.909+0000 7fda18a44740 1 --2- 172.21.15.40:0/103811510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 0x55e524152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.909+0000 7fda18a44740 1 --2- 172.21.15.40:0/103811510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e524134af0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.909+0000 7fda18a44740 1 -- 172.21.15.40:0/103811510 >> 172.21.15.40:0/103811510 conn(0x55e524106d20 msgr2=0x55e52414f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:44.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.909+0000 7fda18a44740 1 -- 172.21.15.40:0/103811510 shutdown_connections 2024-04-26T23:04:44.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.909+0000 7fda18a44740 1 -- 172.21.15.40:0/103811510 wait complete. 2024-04-26T23:04:44.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.910+0000 7fda18a44740 1 -- start start 2024-04-26T23:04:44.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.910+0000 7fda18a44740 1 Processor -- start 2024-04-26T23:04:44.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.911+0000 7fda18a44740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e5241301c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.911+0000 7fda18a44740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 0x55e5241471a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.911+0000 7fda18a44740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5241476e0 0x55e524132ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.911+0000 7fda18a44740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e524072200 con 0x55e524134710 2024-04-26T23:04:44.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.911+0000 7fda18a44740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e52406cee0 con 0x55e5241476e0 2024-04-26T23:04:44.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.911+0000 7fda18a44740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e524079e70 con 0x55e524151fb0 2024-04-26T23:04:44.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.911+0000 7fda18a44740 10 client.0 _ll_register_callbacks cb 0x55e524063270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:44.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.911+0000 7fda0ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e5241301c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.912+0000 7fda0ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e5241301c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47768/0 (socket says 172.21.15.40:47768) 2024-04-26T23:04:44.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.912+0000 7fda0ea00640 1 -- 172.21.15.40:0/3662528198 learned_addr learned my addr 172.21.15.40:0/3662528198 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:44.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.912+0000 7fda0e000640 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 0x55e5241471a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.912+0000 7fda0f400640 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5241476e0 0x55e524132ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.912+0000 7fda0ea00640 1 -- 172.21.15.40:0/3662528198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 msgr2=0x55e5241471a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:44.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.912+0000 7fda0ea00640 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 0x55e5241471a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.912+0000 7fda0ea00640 1 -- 172.21.15.40:0/3662528198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5241476e0 msgr2=0x55e524132ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:44.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.912+0000 7fda0ea00640 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5241476e0 0x55e524132ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.912+0000 7fda0ea00640 1 -- 172.21.15.40:0/3662528198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e523e497d0 con 0x55e524134710 2024-04-26T23:04:44.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.912+0000 7fda0e000640 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 0x55e5241471a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:44.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.912+0000 7fda0f400640 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5241476e0 0x55e524132ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:44.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.912+0000 7fda0ea00640 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e5241301c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fda00014e20 tx=0x7fda00014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a988e5bc115362 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:44.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.912+0000 7fd9efe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:44.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fd9efe00640 1 -- 172.21.15.40:0/3662528198 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00024050 con 0x55e524134710 2024-04-26T23:04:44.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fda18a44740 1 -- 172.21.15.40:0/3662528198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda0000de30 con 0x55e524134710 2024-04-26T23:04:44.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fd9efe00640 1 -- 172.21.15.40:0/3662528198 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fda0001f8d0 con 0x55e524134710 2024-04-26T23:04:44.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fd9efe00640 1 -- 172.21.15.40:0/3662528198 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fda0001fad0 con 0x55e524134710 2024-04-26T23:04:44.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fda18a44740 10 client.6291 fetch_fsmap learned FSMap version 42 2024-04-26T23:04:44.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fda18a44740 1 -- 172.21.15.40:0/3662528198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) -- 0x55e523d7f3f0 con 0x55e524134710 2024-04-26T23:04:44.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fd9efe00640 1 -- 172.21.15.40:0/3662528198 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda0000de30 con 0x55e524134710 2024-04-26T23:04:44.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fd9efe00640 1 -- 172.21.15.40:0/3662528198 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) ==== 3546+0+0 (secure 0 0 0) 0x7fda00028070 con 0x55e524134710 2024-04-26T23:04:44.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fda18a44740 10 client.6291 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T23:04:44.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fda18a44740 10 client.6291 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-04-26T23:04:44.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fda18a44740 20 client.6291 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:44.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fda18a44740 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55e524135420 0x55e524135830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fda18a44740 4 client.6291 mds_command: new command op to 6036 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:04:44.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.913+0000 7fda18a44740 1 -- 172.21.15.40:0/3662528198 --> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e52407b7e0 con 0x55e524135420 2024-04-26T23:04:44.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.914+0000 7fda0e000640 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55e524135420 0x55e524135830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.914+0000 7fda0e000640 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55e524135420 0x55e524135830 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebe560d790e008e6 server_cookie=6a312395d76bc088 in_seq=0 out_seq=0 2024-04-26T23:04:44.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.914+0000 7fd9efe00640 10 client.6291 ms_handle_connect on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:44.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.915+0000 7fd9efe00640 1 -- 172.21.15.40:0/3662528198 <== mds.0 v2:172.21.15.40:6833/618018110 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e52407b7e0 con 0x55e524135420 2024-04-26T23:04:44.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.915+0000 7fd9efe00640 10 client.6291 handle_command_reply: tid=0 2024-04-26T23:04:44.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.915+0000 7fda18a44740 1 client.6291 shutdown 2024-04-26T23:04:44.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.915+0000 7fda18a44740 1 -- 172.21.15.40:0/3662528198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 msgr2=0x55e5241301c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.915+0000 7fda18a44740 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e5241301c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fda00014e20 tx=0x7fda00014000 comp rx=0 tx=0).stop 2024-04-26T23:04:44.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.915+0000 7fda18a44740 1 -- 172.21.15.40:0/3662528198 shutdown_connections 2024-04-26T23:04:44.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.915+0000 7fda18a44740 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55e524135420 0x55e524135830 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.915+0000 7fda18a44740 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5241476e0 0x55e524132ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.915+0000 7fda18a44740 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 0x55e5241471a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.915+0000 7fd9efe00640 0 client.6291 ms_handle_reset on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:44.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.915+0000 7fda18a44740 1 --2- 172.21.15.40:0/3662528198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e5241301c0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.915+0000 7fda18a44740 1 -- 172.21.15.40:0/3662528198 >> 172.21.15.40:0/3662528198 conn(0x55e52410a3f0 msgr2=0x55e524109970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:44.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.916+0000 7fda18a44740 1 -- 172.21.15.40:0/3662528198 shutdown_connections 2024-04-26T23:04:44.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.916+0000 7fda18a44740 1 -- 172.21.15.40:0/3662528198 wait complete. 2024-04-26T23:04:44.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.916+0000 7fda18a44740 20 client.6291 trim_cache size 0 max 16384 2024-04-26T23:04:44.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.932+0000 7fda18a44740 1 Processor -- start 2024-04-26T23:04:44.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.932+0000 7fda18a44740 1 -- start start 2024-04-26T23:04:44.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.933+0000 7fda18a44740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e524133c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.933+0000 7fda18a44740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5241476e0 0x55e524109970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.933+0000 7fda18a44740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 0x55e5241465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.933+0000 7fda18a44740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fda10201c40 con 0x55e524134710 2024-04-26T23:04:44.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.933+0000 7fda18a44740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda10200e10 con 0x55e5241476e0 2024-04-26T23:04:44.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.933+0000 7fda18a44740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda10201510 con 0x55e524151fb0 2024-04-26T23:04:44.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.933+0000 7fda0ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e524133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.933+0000 7fda0ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e524133c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47780/0 (socket says 172.21.15.40:47780) 2024-04-26T23:04:44.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.933+0000 7fda0ea00640 1 -- 172.21.15.40:0/1164217948 learned_addr learned my addr 172.21.15.40:0/1164217948 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:44.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.934+0000 7fda0ea00640 1 -- 172.21.15.40:0/1164217948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 msgr2=0x55e5241465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:44.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.934+0000 7fda0ea00640 1 --2- 172.21.15.40:0/1164217948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 0x55e5241465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.934+0000 7fda0ea00640 1 -- 172.21.15.40:0/1164217948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5241476e0 msgr2=0x55e524109970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:44.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.934+0000 7fda0ea00640 1 --2- 172.21.15.40:0/1164217948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5241476e0 0x55e524109970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.934+0000 7fda0ea00640 1 -- 172.21.15.40:0/1164217948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e524147510 con 0x55e524134710 2024-04-26T23:04:44.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.934+0000 7fda0ea00640 1 --2- 172.21.15.40:0/1164217948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e524133c70 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fda0000d680 tx=0x7fda00012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e367854fb90e86 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:44.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.934+0000 7fd9faa00640 1 -- 172.21.15.40:0/1164217948 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00024070 con 0x55e524134710 2024-04-26T23:04:44.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.934+0000 7fd9faa00640 1 -- 172.21.15.40:0/1164217948 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda0002e070 con 0x55e524134710 2024-04-26T23:04:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.934+0000 7fd9faa00640 1 -- 172.21.15.40:0/1164217948 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00004020 con 0x55e524134710 2024-04-26T23:04:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.935+0000 7fda18a44740 1 -- 172.21.15.40:0/1164217948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 msgr2=0x55e524133c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.935+0000 7fda18a44740 1 --2- 172.21.15.40:0/1164217948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e524133c70 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fda0000d680 tx=0x7fda00012000 comp rx=0 tx=0).stop 2024-04-26T23:04:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.935+0000 7fda18a44740 1 -- 172.21.15.40:0/1164217948 shutdown_connections 2024-04-26T23:04:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.935+0000 7fda18a44740 1 --2- 172.21.15.40:0/1164217948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524151fb0 0x55e5241465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.935+0000 7fda18a44740 1 --2- 172.21.15.40:0/1164217948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e5241476e0 0x55e524109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.935+0000 7fda18a44740 1 --2- 172.21.15.40:0/1164217948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e524134710 0x55e524133c70 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.935+0000 7fda18a44740 1 -- 172.21.15.40:0/1164217948 >> 172.21.15.40:0/1164217948 conn(0x55e52410a3f0 msgr2=0x55e524148890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.935+0000 7fda18a44740 1 -- 172.21.15.40:0/1164217948 shutdown_connections 2024-04-26T23:04:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.935+0000 7fda18a44740 1 -- 172.21.15.40:0/1164217948 wait complete. 2024-04-26T23:04:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.936+0000 7fda18a44740 1 -- start start 2024-04-26T23:04:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.936+0000 7fda18a44740 1 Processor -- start 2024-04-26T23:04:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.937+0000 7fda18a44740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524134710 0x55e52414d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.937+0000 7fda18a44740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5241476e0 0x55e5241d0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.937+0000 7fda18a44740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e524151fb0 0x55e5241d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.937+0000 7fda18a44740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e52407afd0 con 0x55e5241476e0 2024-04-26T23:04:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.937+0000 7fda18a44740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda101f61b0 con 0x55e524151fb0 2024-04-26T23:04:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.937+0000 7fda18a44740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e524109eb0 con 0x55e524134710 2024-04-26T23:04:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.937+0000 7fda18a44740 10 client.0 _ll_register_callbacks cb 0x7fda00022070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.937+0000 7fda0e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5241476e0 0x55e5241d0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.937+0000 7fda0e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5241476e0 0x55e5241d0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47796/0 (socket says 172.21.15.40:47796) 2024-04-26T23:04:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.937+0000 7fda0e000640 1 -- 172.21.15.40:0/2939598554 learned_addr learned my addr 172.21.15.40:0/2939598554 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.938+0000 7fda0e000640 1 -- 172.21.15.40:0/2939598554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524134710 msgr2=0x55e52414d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.938+0000 7fda0e000640 1 --2- 172.21.15.40:0/2939598554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524134710 0x55e52414d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.938+0000 7fda0e000640 1 -- 172.21.15.40:0/2939598554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e524151fb0 msgr2=0x55e5241d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.938+0000 7fda0e000640 1 --2- 172.21.15.40:0/2939598554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e524151fb0 0x55e5241d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.938+0000 7fda0e000640 1 -- 172.21.15.40:0/2939598554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e524132f10 con 0x55e5241476e0 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.938+0000 7fda0e000640 1 --2- 172.21.15.40:0/2939598554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5241476e0 0x55e5241d0920 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc02a820 tx=0x7fd9fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d318204c600c6e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.938+0000 7fd9efe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.938+0000 7fd9efe00640 1 -- 172.21.15.40:0/2939598554 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc037040 con 0x55e5241476e0 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.938+0000 7fd9efe00640 1 -- 172.21.15.40:0/2939598554 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fd9fc032920 con 0x55e5241476e0 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.938+0000 7fda18a44740 1 -- 172.21.15.40:0/2939598554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda0002e070 con 0x55e5241476e0 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.939+0000 7fd9efe00640 1 -- 172.21.15.40:0/2939598554 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fd9fc04f050 con 0x55e5241476e0 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.939+0000 7fda18a44740 10 client.6297 fetch_fsmap learned FSMap version 42 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.939+0000 7fda18a44740 1 -- 172.21.15.40:0/2939598554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) -- 0x55e523d3f0f0 con 0x55e5241476e0 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.939+0000 7fd9efe00640 1 -- 172.21.15.40:0/2939598554 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda0002e070 con 0x55e5241476e0 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.939+0000 7fd9efe00640 1 -- 172.21.15.40:0/2939598554 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) ==== 3546+0+0 (secure 0 0 0) 0x7fd9fc052020 con 0x55e5241476e0 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.939+0000 7fda18a44740 10 client.6297 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.939+0000 7fda18a44740 10 client.6297 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-04-26T23:04:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.939+0000 7fda18a44740 20 client.6297 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:44.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.939+0000 7fda18a44740 1 --2- 172.21.15.40:0/2939598554 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55e524104b40 0x55e5241d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:44.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.939+0000 7fda18a44740 4 client.6297 mds_command: new command op to 6036 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:04:44.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.939+0000 7fda18a44740 1 -- 172.21.15.40:0/2939598554 --> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e524075650 con 0x55e524104b40 2024-04-26T23:04:44.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.939+0000 7fda0ea00640 1 --2- 172.21.15.40:0/2939598554 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55e524104b40 0x55e5241d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:44.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.940+0000 7fda0ea00640 1 --2- 172.21.15.40:0/2939598554 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55e524104b40 0x55e5241d4a20 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2c0dbab3c2bbb28 server_cookie=f9d25e96dd3a6f0a in_seq=0 out_seq=0 2024-04-26T23:04:44.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.940+0000 7fd9efe00640 10 client.6297 ms_handle_connect on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:44.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.956+0000 7fd9efe00640 1 -- 172.21.15.40:0/2939598554 <== mds.0 v2:172.21.15.40:6833/618018110 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e524075650 con 0x55e524104b40 2024-04-26T23:04:44.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.956+0000 7fd9efe00640 10 client.6297 handle_command_reply: tid=0 2024-04-26T23:04:44.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.956+0000 7fda18a44740 1 client.6297 shutdown 2024-04-26T23:04:44.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.957+0000 7fda18a44740 1 -- 172.21.15.40:0/2939598554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5241476e0 msgr2=0x55e5241d0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.957+0000 7fda18a44740 1 --2- 172.21.15.40:0/2939598554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5241476e0 0x55e5241d0920 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc02a820 tx=0x7fd9fc0046c0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.957+0000 7fda18a44740 1 -- 172.21.15.40:0/2939598554 shutdown_connections 2024-04-26T23:04:44.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.957+0000 7fda18a44740 1 --2- 172.21.15.40:0/2939598554 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55e524104b40 0x55e5241d4a20 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.957+0000 7fda18a44740 1 --2- 172.21.15.40:0/2939598554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e524151fb0 0x55e5241d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.957+0000 7fda18a44740 1 --2- 172.21.15.40:0/2939598554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5241476e0 0x55e5241d0920 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.957+0000 7fda18a44740 1 --2- 172.21.15.40:0/2939598554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e524134710 0x55e52414d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.957+0000 7fd9efe00640 0 client.6297 ms_handle_reset on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.957+0000 7fda18a44740 1 -- 172.21.15.40:0/2939598554 >> 172.21.15.40:0/2939598554 conn(0x55e52410a3f0 msgr2=0x55e52414ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.957+0000 7fda18a44740 1 -- 172.21.15.40:0/2939598554 shutdown_connections 2024-04-26T23:04:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.957+0000 7fda18a44740 1 -- 172.21.15.40:0/2939598554 wait complete. 2024-04-26T23:04:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.957+0000 7fda18a44740 20 client.6297 trim_cache size 0 max 16384 2024-04-26T23:04:44.958 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:04:44.958 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:04:44.959 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:04:44.959 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:04:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.958+0000 7fda0fe00640 1 -- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd9f006c8f0 msgr2=0x7fd9f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.958+0000 7fda0fe00640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd9f006c8f0 0x7fd9f006edb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fda00006fd0 tx=0x7fda00008040 comp rx=0 tx=0).stop 2024-04-26T23:04:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.958+0000 7fda0fe00640 1 -- 172.21.15.40:0/3680978241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda10147890 msgr2=0x7fda10146cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.958+0000 7fda0fe00640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda10147890 0x7fda10146cf0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fda0800cef0 tx=0x7fda0800a710 comp rx=0 tx=0).stop 2024-04-26T23:04:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.958+0000 7fda0fe00640 1 -- 172.21.15.40:0/3680978241 shutdown_connections 2024-04-26T23:04:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.958+0000 7fda0fe00640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd9f006c8f0 0x7fd9f006edb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.958+0000 7fda0fe00640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda10147890 0x7fda10146cf0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.958+0000 7fda0fe00640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda10160a50 0x7fda10158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.958+0000 7fda0fe00640 1 --2- 172.21.15.40:0/3680978241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda1006dd40 0x7fda10158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.958+0000 7fda0fe00640 1 -- 172.21.15.40:0/3680978241 >> 172.21.15.40:0/3680978241 conn(0x7fda100b2720 msgr2=0x7fda1013b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.958+0000 7fda0fe00640 1 -- 172.21.15.40:0/3680978241 shutdown_connections 2024-04-26T23:04:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:44.958+0000 7fda0fe00640 1 -- 172.21.15.40:0/3680978241 wait complete. 2024-04-26T23:04:44.977 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:1 flush journal 2024-04-26T23:04:45.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.173+0000 7f496e000640 1 Processor -- start 2024-04-26T23:04:45.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.173+0000 7f496e000640 1 -- start start 2024-04-26T23:04:45.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.174+0000 7f496e000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4968162880 0x7f49680d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.175+0000 7f496e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49680d3bc0 0x7f49680b8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.175+0000 7f496e000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49680b7e50 con 0x7f49680b5f20 2024-04-26T23:04:45.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.175+0000 7f496e000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f49680d0580 con 0x7f4968162880 2024-04-26T23:04:45.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.175+0000 7f496e000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f49680d1800 con 0x7f49680d3bc0 2024-04-26T23:04:45.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.175+0000 7f4967400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4968162880 0x7f49680d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.175+0000 7f4967e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49680d3bc0 0x7f49680b8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.176+0000 7f4967e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49680d3bc0 0x7f49680b8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:44688/0 (socket says 172.21.15.40:44688) 2024-04-26T23:04:45.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.176+0000 7f4967400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4968162880 0x7f49680d2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47156/0 (socket says 172.21.15.40:47156) 2024-04-26T23:04:45.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.176+0000 7f4967e00640 1 -- 172.21.15.40:0/1953722293 learned_addr learned my addr 172.21.15.40:0/1953722293 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:45.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.176+0000 7f4966a00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f49680b5f20 0x7f49680d34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:40180/0 (socket says 172.21.15.40:40180) 2024-04-26T23:04:45.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.176+0000 7f4967e00640 1 -- 172.21.15.40:0/1953722293 >> v2:172.21.15.103:3300/0 conn(0x7f4968162880 msgr2=0x7f49680d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.176+0000 7f4967e00640 1 --2- 172.21.15.40:0/1953722293 >> v2:172.21.15.103:3300/0 conn(0x7f4968162880 0x7f49680d2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.177+0000 7f4967e00640 1 -- 172.21.15.40:0/1953722293 >> v1:172.21.15.103:6789/0 conn(0x7f49680b5f20 legacy=0x7f49680d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.177+0000 7f4967e00640 1 -- 172.21.15.40:0/1953722293 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49680d07d0 con 0x7f49680d3bc0 2024-04-26T23:04:45.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.178+0000 7f4967e00640 1 --2- 172.21.15.40:0/1953722293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49680d3bc0 0x7f49680b8300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4960009870 tx=0x7f496002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faa737c785e50969 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.178+0000 7f4966000640 1 -- 172.21.15.40:0/1953722293 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960037070 con 0x7f49680d3bc0 2024-04-26T23:04:45.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.178+0000 7f4966000640 1 -- 172.21.15.40:0/1953722293 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4960032440 con 0x7f49680d3bc0 2024-04-26T23:04:45.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.178+0000 7f4966000640 1 -- 172.21.15.40:0/1953722293 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960032810 con 0x7f49680d3bc0 2024-04-26T23:04:45.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.179+0000 7f496e000640 1 -- 172.21.15.40:0/1953722293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49680d3bc0 msgr2=0x7f49680b8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.179+0000 7f496e000640 1 --2- 172.21.15.40:0/1953722293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49680d3bc0 0x7f49680b8300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4960009870 tx=0x7f496002a250 comp rx=0 tx=0).stop 2024-04-26T23:04:45.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.179+0000 7f496e000640 1 -- 172.21.15.40:0/1953722293 shutdown_connections 2024-04-26T23:04:45.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.179+0000 7f496e000640 1 --2- 172.21.15.40:0/1953722293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f49680d3bc0 0x7f49680b8300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.179+0000 7f496e000640 1 --2- 172.21.15.40:0/1953722293 >> v2:172.21.15.103:3300/0 conn(0x7f4968162880 0x7f49680d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.179+0000 7f496e000640 1 -- 172.21.15.40:0/1953722293 >> 172.21.15.40:0/1953722293 conn(0x7f49680d4f60 msgr2=0x7f49680d5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.179+0000 7f496e000640 1 -- 172.21.15.40:0/1953722293 shutdown_connections 2024-04-26T23:04:45.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.179+0000 7f496e000640 1 -- 172.21.15.40:0/1953722293 wait complete. 2024-04-26T23:04:45.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.181+0000 7f496e000640 1 Processor -- start 2024-04-26T23:04:45.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.181+0000 7f496e000640 1 -- start start 2024-04-26T23:04:45.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.182+0000 7f496e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49680b5f20 0x7f4968159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.182+0000 7f496e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49680d3bc0 0x7f496806cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.182+0000 7f496e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4968162880 0x7f496806d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.182+0000 7f496e000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4968162c60 con 0x7f49680d3bc0 2024-04-26T23:04:45.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.182+0000 7f496e000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f49680d0d90 con 0x7f49680b5f20 2024-04-26T23:04:45.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.182+0000 7f496e000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4968162de0 con 0x7f4968162880 2024-04-26T23:04:45.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.183+0000 7f4966a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49680d3bc0 0x7f496806cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.183+0000 7f4966a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49680d3bc0 0x7f496806cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47812/0 (socket says 172.21.15.40:47812) 2024-04-26T23:04:45.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.183+0000 7f4966a00640 1 -- 172.21.15.40:0/222070425 learned_addr learned my addr 172.21.15.40:0/222070425 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:45.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.183+0000 7f4967400640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49680b5f20 0x7f4968159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.183+0000 7f4967e00640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4968162880 0x7f496806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.183+0000 7f4966a00640 1 -- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4968162880 msgr2=0x7f496806d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.183+0000 7f4966a00640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4968162880 0x7f496806d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.183+0000 7f4966a00640 1 -- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49680b5f20 msgr2=0x7f4968159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.183+0000 7f4966a00640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49680b5f20 0x7f4968159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.183+0000 7f4966a00640 1 -- 172.21.15.40:0/222070425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49681f5bf0 con 0x7f49680d3bc0 2024-04-26T23:04:45.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.184+0000 7f4966a00640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49680d3bc0 0x7f496806cd10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f495800c730 tx=0x7f495800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.184+0000 7f495f400640 1 -- 172.21.15.40:0/222070425 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4958017070 con 0x7f49680d3bc0 2024-04-26T23:04:45.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.184+0000 7f495f400640 1 -- 172.21.15.40:0/222070425 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f495800ce40 con 0x7f49680d3bc0 2024-04-26T23:04:45.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.184+0000 7f496e000640 1 -- 172.21.15.40:0/222070425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49681f5910 con 0x7f49680d3bc0 2024-04-26T23:04:45.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.184+0000 7f495f400640 1 -- 172.21.15.40:0/222070425 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495800a860 con 0x7f49680d3bc0 2024-04-26T23:04:45.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.184+0000 7f496e000640 1 -- 172.21.15.40:0/222070425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49680cc320 con 0x7f49680d3bc0 2024-04-26T23:04:45.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.187+0000 7f495f400640 1 -- 172.21.15.40:0/222070425 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f495801a070 con 0x7f49680d3bc0 2024-04-26T23:04:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.188+0000 7f495f400640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f494406c8f0 0x7f494406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.188+0000 7f495f400640 1 -- 172.21.15.40:0/222070425 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f495808b2d0 con 0x7f49680d3bc0 2024-04-26T23:04:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.189+0000 7f4970d87740 1 Processor -- start 2024-04-26T23:04:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.189+0000 7f4967400640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f494406c8f0 0x7f494406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.189+0000 7f4970d87740 1 -- start start 2024-04-26T23:04:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4970d87740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 0x55b5bee3f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4970d87740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee5cde0 0x55b5bee5d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4970d87740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee15220 0x55b5bee5b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4970d87740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b5bed25030 con 0x55b5bee3f540 2024-04-26T23:04:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4970d87740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b5bed407f0 con 0x55b5bee15220 2024-04-26T23:04:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4970d87740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b5becfb7e0 con 0x55b5bee5cde0 2024-04-26T23:04:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4967400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 0x55b5bee3f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4967400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 0x55b5bee3f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47824/0 (socket says 172.21.15.40:47824) 2024-04-26T23:04:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4967400640 1 -- 172.21.15.40:0/3377335775 learned_addr learned my addr 172.21.15.40:0/3377335775 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4967400640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f494406c8f0 0x7f494406edb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4950009ad0 tx=0x7f4950008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4967400640 1 -- 172.21.15.40:0/3377335775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee5cde0 msgr2=0x55b5bee5d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4967400640 1 --2- 172.21.15.40:0/3377335775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee5cde0 0x55b5bee5d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4967400640 1 -- 172.21.15.40:0/3377335775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee15220 msgr2=0x55b5bee5b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4967400640 1 --2- 172.21.15.40:0/3377335775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee15220 0x55b5bee5b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.190+0000 7f4967400640 1 -- 172.21.15.40:0/3377335775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5bed07050 con 0x55b5bee3f540 2024-04-26T23:04:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.191+0000 7f4967400640 1 --2- 172.21.15.40:0/3377335775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 0x55b5bee3f920 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4950011d90 tx=0x7f4950009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b30e5b5a1ae83adf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.191+0000 7f4957400640 1 -- 172.21.15.40:0/3377335775 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950011dc0 con 0x55b5bee3f540 2024-04-26T23:04:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.191+0000 7f4957400640 1 -- 172.21.15.40:0/3377335775 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4950026070 con 0x55b5bee3f540 2024-04-26T23:04:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.191+0000 7f4957400640 1 -- 172.21.15.40:0/3377335775 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950021660 con 0x55b5bee3f540 2024-04-26T23:04:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.192+0000 7f4970d87740 1 -- 172.21.15.40:0/3377335775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 msgr2=0x55b5bee3f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.192+0000 7f4970d87740 1 --2- 172.21.15.40:0/3377335775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 0x55b5bee3f920 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4950011d90 tx=0x7f4950009f90 comp rx=0 tx=0).stop 2024-04-26T23:04:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.192+0000 7f4970d87740 1 -- 172.21.15.40:0/3377335775 shutdown_connections 2024-04-26T23:04:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.192+0000 7f4970d87740 1 --2- 172.21.15.40:0/3377335775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee15220 0x55b5bee5b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.192+0000 7f4970d87740 1 --2- 172.21.15.40:0/3377335775 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee5cde0 0x55b5bee5d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.192+0000 7f4970d87740 1 --2- 172.21.15.40:0/3377335775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 0x55b5bee3f920 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.192+0000 7f4970d87740 1 -- 172.21.15.40:0/3377335775 >> 172.21.15.40:0/3377335775 conn(0x55b5bee11b50 msgr2=0x55b5bee5a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.192+0000 7f4970d87740 1 -- 172.21.15.40:0/3377335775 shutdown_connections 2024-04-26T23:04:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.192+0000 7f4970d87740 1 -- 172.21.15.40:0/3377335775 wait complete. 2024-04-26T23:04:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.193+0000 7f4970d87740 1 -- start start 2024-04-26T23:04:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.193+0000 7f4970d87740 1 Processor -- start 2024-04-26T23:04:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.194+0000 7f4970d87740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee3f540 0x55b5bee54400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.194+0000 7f4970d87740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee5cde0 0x55b5bee3e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.194+0000 7f4970d87740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee55970 0x55b5bee54d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.194+0000 7f4970d87740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b5bed7d200 con 0x55b5bee55970 2024-04-26T23:04:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.194+0000 7f4970d87740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b5bed77ee0 con 0x55b5bee3f540 2024-04-26T23:04:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.194+0000 7f4970d87740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b5bed84e70 con 0x55b5bee5cde0 2024-04-26T23:04:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.194+0000 7f4970d87740 10 client.0 _ll_register_callbacks cb 0x55b5bed6e270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.194+0000 7f4967e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee55970 0x55b5bee54d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.194+0000 7f4967e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee55970 0x55b5bee54d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47834/0 (socket says 172.21.15.40:47834) 2024-04-26T23:04:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.194+0000 7f4967e00640 1 -- 172.21.15.40:0/4023598442 learned_addr learned my addr 172.21.15.40:0/4023598442 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.195+0000 7f4967400640 1 --2- 172.21.15.40:0/4023598442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee3f540 0x55b5bee54400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.195+0000 7f4967e00640 1 -- 172.21.15.40:0/4023598442 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee5cde0 msgr2=0x55b5bee3e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.195+0000 7f4967e00640 1 --2- 172.21.15.40:0/4023598442 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee5cde0 0x55b5bee3e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.195+0000 7f4967e00640 1 -- 172.21.15.40:0/4023598442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee3f540 msgr2=0x55b5bee54400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.195+0000 7f4967e00640 1 --2- 172.21.15.40:0/4023598442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee3f540 0x55b5bee54400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.195+0000 7f4967e00640 1 -- 172.21.15.40:0/4023598442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5beb547d0 con 0x55b5bee55970 2024-04-26T23:04:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.195+0000 7f4967400640 1 --2- 172.21.15.40:0/4023598442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee3f540 0x55b5bee54400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.195+0000 7f4967e00640 1 --2- 172.21.15.40:0/4023598442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee55970 0x55b5bee54d60 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f4960009990 tx=0x7f49600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5696e998415dfdb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.195+0000 7f494be00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.196+0000 7f494be00640 1 -- 172.21.15.40:0/4023598442 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960037040 con 0x55b5bee55970 2024-04-26T23:04:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.196+0000 7f494be00640 1 -- 172.21.15.40:0/4023598442 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f4960032cc0 con 0x55b5bee55970 2024-04-26T23:04:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.196+0000 7f4970d87740 1 -- 172.21.15.40:0/4023598442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4950026070 con 0x55b5bee55970 2024-04-26T23:04:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.196+0000 7f494be00640 1 -- 172.21.15.40:0/4023598442 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f496003c480 con 0x55b5bee55970 2024-04-26T23:04:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.196+0000 7f4970d87740 10 client.6306 fetch_fsmap learned FSMap version 42 2024-04-26T23:04:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.196+0000 7f4970d87740 1 -- 172.21.15.40:0/4023598442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) -- 0x55b5bea8a3f0 con 0x55b5bee55970 2024-04-26T23:04:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.196+0000 7f494be00640 1 -- 172.21.15.40:0/4023598442 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4950026070 con 0x55b5bee55970 2024-04-26T23:04:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.196+0000 7f494be00640 1 -- 172.21.15.40:0/4023598442 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) ==== 3546+0+0 (secure 0 0 0) 0x7f496003c620 con 0x55b5bee55970 2024-04-26T23:04:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.196+0000 7f4970d87740 10 client.6306 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T23:04:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.196+0000 7f4970d87740 10 client.6306 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.a 2024-04-26T23:04:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.196+0000 7f4970d87740 20 client.6306 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.197+0000 7f4970d87740 1 --2- 172.21.15.40:0/4023598442 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55b5bee567b0 0x55b5bee56bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.197+0000 7f4970d87740 4 client.6306 mds_command: new command op to 6027 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:04:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.197+0000 7f4970d87740 1 -- 172.21.15.40:0/4023598442 --> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5bed867e0 con 0x55b5bee567b0 2024-04-26T23:04:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.197+0000 7f4967400640 1 --2- 172.21.15.40:0/4023598442 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55b5bee567b0 0x55b5bee56bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.198+0000 7f4967400640 1 --2- 172.21.15.40:0/4023598442 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55b5bee567b0 0x55b5bee56bf0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c40c4f807841163b server_cookie=f88bf595f7249071 in_seq=0 out_seq=0 2024-04-26T23:04:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.198+0000 7f494be00640 10 client.6306 ms_handle_connect on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.198+0000 7f494be00640 1 -- 172.21.15.40:0/4023598442 <== mds.1 v2:172.21.15.40:6832/2304583125 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b5bed867e0 con 0x55b5bee567b0 2024-04-26T23:04:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.198+0000 7f494be00640 10 client.6306 handle_command_reply: tid=0 2024-04-26T23:04:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.198+0000 7f4970d87740 1 client.6306 shutdown 2024-04-26T23:04:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.199+0000 7f4970d87740 1 -- 172.21.15.40:0/4023598442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee55970 msgr2=0x55b5bee54d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.199+0000 7f4970d87740 1 --2- 172.21.15.40:0/4023598442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee55970 0x55b5bee54d60 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f4960009990 tx=0x7f49600046c0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.199+0000 7f4970d87740 1 -- 172.21.15.40:0/4023598442 shutdown_connections 2024-04-26T23:04:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.199+0000 7f4970d87740 1 --2- 172.21.15.40:0/4023598442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee55970 0x55b5bee54d60 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.199+0000 7f4970d87740 1 --2- 172.21.15.40:0/4023598442 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee5cde0 0x55b5bee3e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.199+0000 7f4970d87740 1 --2- 172.21.15.40:0/4023598442 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55b5bee567b0 0x55b5bee56bf0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.199+0000 7f4970d87740 1 --2- 172.21.15.40:0/4023598442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee3f540 0x55b5bee54400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.199+0000 7f4970d87740 1 -- 172.21.15.40:0/4023598442 >> 172.21.15.40:0/4023598442 conn(0x55b5bee15220 msgr2=0x55b5bee5d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.199+0000 7f494be00640 0 client.6306 ms_handle_reset on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:45.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.199+0000 7f4970d87740 1 -- 172.21.15.40:0/4023598442 shutdown_connections 2024-04-26T23:04:45.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.199+0000 7f4970d87740 1 -- 172.21.15.40:0/4023598442 wait complete. 2024-04-26T23:04:45.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.199+0000 7f4970d87740 20 client.6306 trim_cache size 0 max 16384 2024-04-26T23:04:45.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.216+0000 7f4970d87740 1 Processor -- start 2024-04-26T23:04:45.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.216+0000 7f4970d87740 1 -- start start 2024-04-26T23:04:45.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.217+0000 7f4970d87740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee3f540 0x55b5bee53380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.217+0000 7f4970d87740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee55970 0x55b5bee57810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.217+0000 7f4970d87740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee5cde0 0x55b5bee53d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.217+0000 7f4970d87740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4968201f00 con 0x55b5bee5cde0 2024-04-26T23:04:45.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.217+0000 7f4970d87740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f49682010f0 con 0x55b5bee55970 2024-04-26T23:04:45.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.217+0000 7f4970d87740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f49682017d0 con 0x55b5bee3f540 2024-04-26T23:04:45.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.217+0000 7f4967e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee5cde0 0x55b5bee53d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.217+0000 7f4967e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee5cde0 0x55b5bee53d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47844/0 (socket says 172.21.15.40:47844) 2024-04-26T23:04:45.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.217+0000 7f4967e00640 1 -- 172.21.15.40:0/2278820660 learned_addr learned my addr 172.21.15.40:0/2278820660 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:45.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.218+0000 7f4966a00640 1 --2- 172.21.15.40:0/2278820660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee55970 0x55b5bee57810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.218+0000 7f4967400640 1 --2- 172.21.15.40:0/2278820660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee3f540 0x55b5bee53380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.218+0000 7f4967e00640 1 -- 172.21.15.40:0/2278820660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee3f540 msgr2=0x55b5bee53380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.218+0000 7f4967e00640 1 --2- 172.21.15.40:0/2278820660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee3f540 0x55b5bee53380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.218+0000 7f4967e00640 1 -- 172.21.15.40:0/2278820660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee55970 msgr2=0x55b5bee57810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.218+0000 7f4967e00640 1 --2- 172.21.15.40:0/2278820660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee55970 0x55b5bee57810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.218+0000 7f4967e00640 1 -- 172.21.15.40:0/2278820660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5bea4a0f0 con 0x55b5bee5cde0 2024-04-26T23:04:45.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.218+0000 7f4966a00640 1 --2- 172.21.15.40:0/2278820660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee55970 0x55b5bee57810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:45.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.218+0000 7f4967e00640 1 --2- 172.21.15.40:0/2278820660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee5cde0 0x55b5bee53d10 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f496002a290 tx=0x7f496003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ffba3cc03916e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.218+0000 7f4956a00640 1 -- 172.21.15.40:0/2278820660 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960037070 con 0x55b5bee5cde0 2024-04-26T23:04:45.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.218+0000 7f4956a00640 1 -- 172.21.15.40:0/2278820660 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f496003cb20 con 0x55b5bee5cde0 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.218+0000 7f4956a00640 1 -- 172.21.15.40:0/2278820660 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960049020 con 0x55b5bee5cde0 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.219+0000 7f4970d87740 1 -- 172.21.15.40:0/2278820660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee5cde0 msgr2=0x55b5bee53d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.219+0000 7f4970d87740 1 --2- 172.21.15.40:0/2278820660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee5cde0 0x55b5bee53d10 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f496002a290 tx=0x7f496003c000 comp rx=0 tx=0).stop 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.219+0000 7f4970d87740 1 -- 172.21.15.40:0/2278820660 shutdown_connections 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.219+0000 7f4970d87740 1 --2- 172.21.15.40:0/2278820660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee5cde0 0x55b5bee53d10 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.219+0000 7f4970d87740 1 --2- 172.21.15.40:0/2278820660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee55970 0x55b5bee57810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.219+0000 7f4970d87740 1 --2- 172.21.15.40:0/2278820660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee3f540 0x55b5bee53380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.219+0000 7f4970d87740 1 -- 172.21.15.40:0/2278820660 >> 172.21.15.40:0/2278820660 conn(0x55b5bee15220 msgr2=0x55b5bee55fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.219+0000 7f4970d87740 1 -- 172.21.15.40:0/2278820660 shutdown_connections 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.219+0000 7f4970d87740 1 -- 172.21.15.40:0/2278820660 wait complete. 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.220+0000 7f4970d87740 1 -- start start 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.220+0000 7f4970d87740 1 Processor -- start 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.221+0000 7f4970d87740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 0x55b5beedaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.221+0000 7f4970d87740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee55970 0x55b5bee4fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.221+0000 7f4970d87740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee5cde0 0x55b5bee5d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.221+0000 7f4970d87740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b5bed85fd0 con 0x55b5bee3f540 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.221+0000 7f4970d87740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f49680cdd20 con 0x55b5bee55970 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.221+0000 7f4970d87740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b5bee57240 con 0x55b5bee5cde0 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.221+0000 7f4970d87740 10 client.0 _ll_register_callbacks cb 0x7f4960035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.221+0000 7f4967400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 0x55b5beedaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.221+0000 7f4967400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 0x55b5beedaa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47860/0 (socket says 172.21.15.40:47860) 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.221+0000 7f4967400640 1 -- 172.21.15.40:0/1004904342 learned_addr learned my addr 172.21.15.40:0/1004904342 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.222+0000 7f4967e00640 1 --2- 172.21.15.40:0/1004904342 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee5cde0 0x55b5bee5d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.222+0000 7f4967400640 1 -- 172.21.15.40:0/1004904342 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee5cde0 msgr2=0x55b5bee5d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.222+0000 7f4967400640 1 --2- 172.21.15.40:0/1004904342 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee5cde0 0x55b5bee5d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.222+0000 7f4967400640 1 -- 172.21.15.40:0/1004904342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee55970 msgr2=0x55b5bee4fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.222+0000 7f4967400640 1 --2- 172.21.15.40:0/1004904342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee55970 0x55b5bee4fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.222+0000 7f4967400640 1 -- 172.21.15.40:0/1004904342 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5bed40a30 con 0x55b5bee3f540 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.222+0000 7f4967400640 1 --2- 172.21.15.40:0/1004904342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 0x55b5beedaa60 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f4950022000 tx=0x7f49500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c76400564f1777df server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.222+0000 7f494be00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.222+0000 7f494be00640 1 -- 172.21.15.40:0/1004904342 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49500119a0 con 0x55b5bee3f540 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.222+0000 7f494be00640 1 -- 172.21.15.40:0/1004904342 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f4950036020 con 0x55b5bee3f540 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.222+0000 7f4970d87740 1 -- 172.21.15.40:0/1004904342 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f496003cb20 con 0x55b5bee3f540 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.223+0000 7f494be00640 1 -- 172.21.15.40:0/1004904342 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4950036bd0 con 0x55b5bee3f540 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.223+0000 7f4970d87740 10 client.6312 fetch_fsmap learned FSMap version 42 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.223+0000 7f4970d87740 1 -- 172.21.15.40:0/1004904342 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) -- 0x55b5bee50460 con 0x55b5bee3f540 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.223+0000 7f494be00640 1 -- 172.21.15.40:0/1004904342 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f496003cb20 con 0x55b5bee3f540 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.223+0000 7f494be00640 1 -- 172.21.15.40:0/1004904342 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) ==== 3546+0+0 (secure 0 0 0) 0x7f4950036220 con 0x55b5bee3f540 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.223+0000 7f4970d87740 10 client.6312 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.223+0000 7f4970d87740 10 client.6312 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.a 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.223+0000 7f4970d87740 20 client.6312 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:45.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.223+0000 7f4970d87740 1 --2- 172.21.15.40:0/1004904342 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55b5bee5a590 0x55b5bee53ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.223+0000 7f4970d87740 4 client.6312 mds_command: new command op to 6027 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:04:45.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.223+0000 7f4970d87740 1 -- 172.21.15.40:0/1004904342 --> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b5bed80650 con 0x55b5bee5a590 2024-04-26T23:04:45.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.223+0000 7f4966a00640 1 --2- 172.21.15.40:0/1004904342 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55b5bee5a590 0x55b5bee53ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.224+0000 7f4966a00640 1 --2- 172.21.15.40:0/1004904342 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55b5bee5a590 0x55b5bee53ba0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fbd0e19eadd96966 server_cookie=959dcabb7e5cf1b6 in_seq=0 out_seq=0 2024-04-26T23:04:45.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.224+0000 7f494be00640 10 client.6312 ms_handle_connect on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:45.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.241+0000 7f494be00640 1 -- 172.21.15.40:0/1004904342 <== mds.1 v2:172.21.15.40:6832/2304583125 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b5bed80650 con 0x55b5bee5a590 2024-04-26T23:04:45.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.241+0000 7f494be00640 10 client.6312 handle_command_reply: tid=0 2024-04-26T23:04:45.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.241+0000 7f4970d87740 1 client.6312 shutdown 2024-04-26T23:04:45.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.241+0000 7f4970d87740 1 -- 172.21.15.40:0/1004904342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 msgr2=0x55b5beedaa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.241+0000 7f4970d87740 1 --2- 172.21.15.40:0/1004904342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 0x55b5beedaa60 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f4950022000 tx=0x7f49500046c0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.241+0000 7f4970d87740 1 -- 172.21.15.40:0/1004904342 shutdown_connections 2024-04-26T23:04:45.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.241+0000 7f4970d87740 1 --2- 172.21.15.40:0/1004904342 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b5bee5cde0 0x55b5bee5d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.241+0000 7f4970d87740 1 --2- 172.21.15.40:0/1004904342 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55b5bee5a590 0x55b5bee53ba0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.241+0000 7f4970d87740 1 --2- 172.21.15.40:0/1004904342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5bee55970 0x55b5bee4fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.241+0000 7f4970d87740 1 --2- 172.21.15.40:0/1004904342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5bee3f540 0x55b5beedaa60 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.241+0000 7f4970d87740 1 -- 172.21.15.40:0/1004904342 >> 172.21.15.40:0/1004904342 conn(0x55b5bee15220 msgr2=0x55b5bee3e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.241+0000 7f494be00640 0 client.6312 ms_handle_reset on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:45.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.242+0000 7f4970d87740 1 -- 172.21.15.40:0/1004904342 shutdown_connections 2024-04-26T23:04:45.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.242+0000 7f4970d87740 1 -- 172.21.15.40:0/1004904342 wait complete. 2024-04-26T23:04:45.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.242+0000 7f4970d87740 20 client.6312 trim_cache size 0 max 16384 2024-04-26T23:04:45.244 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:04:45.244 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:04:45.244 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:04:45.244 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:04:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.243+0000 7f496e000640 1 -- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f494406c8f0 msgr2=0x7f494406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.243+0000 7f496e000640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f494406c8f0 0x7f494406edb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4950009ad0 tx=0x7f4950008040 comp rx=0 tx=0).stop 2024-04-26T23:04:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.243+0000 7f496e000640 1 -- 172.21.15.40:0/222070425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49680d3bc0 msgr2=0x7f496806cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.243+0000 7f496e000640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49680d3bc0 0x7f496806cd10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f495800c730 tx=0x7f495800cbf0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.243+0000 7f496e000640 1 -- 172.21.15.40:0/222070425 shutdown_connections 2024-04-26T23:04:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.243+0000 7f496e000640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f494406c8f0 0x7f494406edb0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.243+0000 7f496e000640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4968162880 0x7f496806d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.243+0000 7f496e000640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49680d3bc0 0x7f496806cd10 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.243+0000 7f496e000640 1 --2- 172.21.15.40:0/222070425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49680b5f20 0x7f4968159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.243+0000 7f496e000640 1 -- 172.21.15.40:0/222070425 >> 172.21.15.40:0/222070425 conn(0x7f49680d4f60 msgr2=0x7f49680d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.243+0000 7f496e000640 1 -- 172.21.15.40:0/222070425 shutdown_connections 2024-04-26T23:04:45.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.243+0000 7f496e000640 1 -- 172.21.15.40:0/222070425 wait complete. 2024-04-26T23:04:45.262 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:2 flush journal 2024-04-26T23:04:45.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.472+0000 7f9475a00640 1 Processor -- start 2024-04-26T23:04:45.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.472+0000 7f9475a00640 1 -- start start 2024-04-26T23:04:45.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.473+0000 7f9475a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94700d1770 0x7f947015ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.473+0000 7f9475a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94700d5d30 con 0x7f94700b6470 2024-04-26T23:04:45.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.473+0000 7f9475a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94700b5550 con 0x7f94700d1b50 2024-04-26T23:04:45.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.473+0000 7f9475a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f94700d3760 con 0x7f94700d1770 2024-04-26T23:04:45.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.474+0000 7f9467400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f94700b6470 0x7f94700b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35756/0 (socket says 172.21.15.40:35756) 2024-04-26T23:04:45.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.474+0000 7f9467400640 1 -- 172.21.15.40:0/507155394 learned_addr learned my addr 172.21.15.40:0/507155394 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:45.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.474+0000 7f946fe00640 1 --2- 172.21.15.40:0/507155394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94700d1770 0x7f947015ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.474+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1980700518 0 0) 0x7f94700d5d30 con 0x7f94700b6470 2024-04-26T23:04:45.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.474+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9454003520 con 0x7f94700b6470 2024-04-26T23:04:45.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 429086962 0 0) 0x7f9454003520 con 0x7f94700b6470 2024-04-26T23:04:45.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94700d1770 msgr2=0x7f947015ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f946f400640 1 --2- 172.21.15.40:0/507155394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94700d1770 0x7f947015ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 >> v1:172.21.15.103:6789/0 conn(0x7f94700d1b50 legacy=0x7f94700b4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94700d09f0 con 0x7f94700b6470 2024-04-26T23:04:45.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f9458001670 con 0x7f94700b6470 2024-04-26T23:04:45.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 >> v1:172.21.15.40:6789/0 conn(0x7f94700b6470 legacy=0x7f94700b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f946f400640 1 --2- 172.21.15.40:0/507155394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94540039e0 0x7f9454003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f946f400640 1 --2- 172.21.15.40:0/507155394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9454004800 0x7f9454004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f946f400640 1 --2- 172.21.15.40:0/507155394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94700d1770 0x7f94540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f94700d3760 con 0x7f9454004800 2024-04-26T23:04:45.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f945400ba70 con 0x7f94540039e0 2024-04-26T23:04:45.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f945400bc20 con 0x7f94700d1770 2024-04-26T23:04:45.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.475+0000 7f9467400640 1 --2- 172.21.15.40:0/507155394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9454004800 0x7f9454004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.476+0000 7f946fe00640 1 --2- 172.21.15.40:0/507155394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94540039e0 0x7f9454003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.476+0000 7f9475000640 1 --2- 172.21.15.40:0/507155394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94700d1770 0x7f94540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.476+0000 7f9467400640 1 -- 172.21.15.40:0/507155394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94700d1770 msgr2=0x7f94540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.476+0000 7f9467400640 1 --2- 172.21.15.40:0/507155394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94700d1770 0x7f94540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.476+0000 7f9467400640 1 -- 172.21.15.40:0/507155394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94540039e0 msgr2=0x7f9454003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.476+0000 7f9467400640 1 --2- 172.21.15.40:0/507155394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94540039e0 0x7f9454003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.476+0000 7f9467400640 1 -- 172.21.15.40:0/507155394 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9454003520 con 0x7f9454004800 2024-04-26T23:04:45.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.477+0000 7f9467400640 1 --2- 172.21.15.40:0/507155394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9454004800 0x7f9454004c80 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9458001eb0 tx=0x7f9454005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8622bfc50e2f9314 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.477+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458007c80 con 0x7f9454004800 2024-04-26T23:04:45.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.477+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9458005af0 con 0x7f9454004800 2024-04-26T23:04:45.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.477+0000 7f946f400640 1 -- 172.21.15.40:0/507155394 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458007070 con 0x7f9454004800 2024-04-26T23:04:45.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.478+0000 7f9475a00640 1 -- 172.21.15.40:0/507155394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9454004800 msgr2=0x7f9454004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.478+0000 7f9475a00640 1 --2- 172.21.15.40:0/507155394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9454004800 0x7f9454004c80 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9458001eb0 tx=0x7f9454005830 comp rx=0 tx=0).stop 2024-04-26T23:04:45.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.478+0000 7f9475000640 1 -- 172.21.15.40:0/507155394 reap_dead start 2024-04-26T23:04:45.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.478+0000 7f9475a00640 1 -- 172.21.15.40:0/507155394 shutdown_connections 2024-04-26T23:04:45.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.478+0000 7f9475a00640 1 -- 172.21.15.40:0/507155394 >> 172.21.15.40:0/507155394 conn(0x7f94700d4dd0 msgr2=0x7f94700d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.478+0000 7f9475a00640 1 -- 172.21.15.40:0/507155394 shutdown_connections 2024-04-26T23:04:45.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.478+0000 7f9475a00640 1 -- 172.21.15.40:0/507155394 wait complete. 2024-04-26T23:04:45.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.480+0000 7f9475a00640 1 Processor -- start 2024-04-26T23:04:45.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.480+0000 7f9475a00640 1 -- start start 2024-04-26T23:04:45.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.482+0000 7f9475a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9454004800 0x7f94701576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.482+0000 7f9475a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94700cc890 0x7f9470157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.482+0000 7f9475a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94701583d0 0x7f94701f82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.482+0000 7f9475a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f94700d2670 con 0x7f94701583d0 2024-04-26T23:04:45.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.482+0000 7f9475a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f94700b5b00 con 0x7f9454004800 2024-04-26T23:04:45.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.482+0000 7f9475a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f94700d3bf0 con 0x7f94700cc890 2024-04-26T23:04:45.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.482+0000 7f9475000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94701583d0 0x7f94701f82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.482+0000 7f9475000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94701583d0 0x7f94701f82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47876/0 (socket says 172.21.15.40:47876) 2024-04-26T23:04:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.482+0000 7f9475000640 1 -- 172.21.15.40:0/232556194 learned_addr learned my addr 172.21.15.40:0/232556194 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.482+0000 7f946fe00640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9454004800 0x7f94701576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.482+0000 7f9467400640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94700cc890 0x7f9470157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.483+0000 7f9475000640 1 -- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94700cc890 msgr2=0x7f9470157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.483+0000 7f9475000640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94700cc890 0x7f9470157e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.483+0000 7f9475000640 1 -- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9454004800 msgr2=0x7f94701576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.483+0000 7f9475000640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9454004800 0x7f94701576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.483+0000 7f9475000640 1 -- 172.21.15.40:0/232556194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94701f5c10 con 0x7f94701583d0 2024-04-26T23:04:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.483+0000 7f9475000640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94701583d0 0x7f94701f82f0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f947008d490 tx=0x7f946800e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.483+0000 7f946cc00640 1 -- 172.21.15.40:0/232556194 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9468019070 con 0x7f94701583d0 2024-04-26T23:04:45.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.483+0000 7f946cc00640 1 -- 172.21.15.40:0/232556194 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94680052e0 con 0x7f94701583d0 2024-04-26T23:04:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.483+0000 7f946cc00640 1 -- 172.21.15.40:0/232556194 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9468003810 con 0x7f94701583d0 2024-04-26T23:04:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.483+0000 7f9475a00640 1 -- 172.21.15.40:0/232556194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94700cd5c0 con 0x7f94701583d0 2024-04-26T23:04:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.484+0000 7f9475a00640 1 -- 172.21.15.40:0/232556194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94700cdb80 con 0x7f94701583d0 2024-04-26T23:04:45.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.487+0000 7f946cc00640 1 -- 172.21.15.40:0/232556194 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f946801c070 con 0x7f94701583d0 2024-04-26T23:04:45.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.488+0000 7f946cc00640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f944c06c8f0 0x7f944c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.488+0000 7f946cc00640 1 -- 172.21.15.40:0/232556194 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f946808b270 con 0x7f94701583d0 2024-04-26T23:04:45.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.488+0000 7f9479d91740 1 Processor -- start 2024-04-26T23:04:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.488+0000 7f9479d91740 1 -- start start 2024-04-26T23:04:45.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.489+0000 7f946fe00640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f944c06c8f0 0x7f944c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.489+0000 7f9479d91740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 0x55c0fdb69af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.489+0000 7f9479d91740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb86fb0 0x55c0fdb87500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f9479d91740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb3f3f0 0x55c0fdb85670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f9479d91740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c0fda4f030 con 0x55c0fdb86fb0 2024-04-26T23:04:45.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f9479d91740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c0fda6a7f0 con 0x55c0fdb69710 2024-04-26T23:04:45.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f9479d91740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c0fda257e0 con 0x55c0fdb3f3f0 2024-04-26T23:04:45.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f9467400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb86fb0 0x55c0fdb87500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f9467400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb86fb0 0x55c0fdb87500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47880/0 (socket says 172.21.15.40:47880) 2024-04-26T23:04:45.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f9467400640 1 -- 172.21.15.40:0/200292890 learned_addr learned my addr 172.21.15.40:0/200292890 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:45.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f946fe00640 1 --2- 172.21.15.40:0/200292890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 0x55c0fdb69af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f9475000640 1 --2- 172.21.15.40:0/200292890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb3f3f0 0x55c0fdb85670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f946fe00640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f944c06c8f0 0x7f944c06edb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9470162f20 tx=0x7f9460009750 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f9467400640 1 -- 172.21.15.40:0/200292890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb3f3f0 msgr2=0x55c0fdb85670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f9467400640 1 --2- 172.21.15.40:0/200292890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb3f3f0 0x55c0fdb85670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f9467400640 1 -- 172.21.15.40:0/200292890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 msgr2=0x55c0fdb69af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f9467400640 1 --2- 172.21.15.40:0/200292890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 0x55c0fdb69af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.490+0000 7f9467400640 1 -- 172.21.15.40:0/200292890 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0fda31050 con 0x55c0fdb86fb0 2024-04-26T23:04:45.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.491+0000 7f9467400640 1 --2- 172.21.15.40:0/200292890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb86fb0 0x55c0fdb87500 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f945800c220 tx=0x7f9458009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a232e7acf296fd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.491+0000 7f9464c00640 1 -- 172.21.15.40:0/200292890 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458001430 con 0x55c0fdb86fb0 2024-04-26T23:04:45.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.491+0000 7f9464c00640 1 -- 172.21.15.40:0/200292890 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9458044070 con 0x55c0fdb86fb0 2024-04-26T23:04:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.491+0000 7f9464c00640 1 -- 172.21.15.40:0/200292890 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94580047e0 con 0x55c0fdb86fb0 2024-04-26T23:04:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.492+0000 7f9479d91740 1 -- 172.21.15.40:0/200292890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb86fb0 msgr2=0x55c0fdb87500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.492+0000 7f9479d91740 1 --2- 172.21.15.40:0/200292890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb86fb0 0x55c0fdb87500 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f945800c220 tx=0x7f9458009030 comp rx=0 tx=0).stop 2024-04-26T23:04:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.492+0000 7f9479d91740 1 -- 172.21.15.40:0/200292890 shutdown_connections 2024-04-26T23:04:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.492+0000 7f9479d91740 1 --2- 172.21.15.40:0/200292890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb3f3f0 0x55c0fdb85670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.492+0000 7f9479d91740 1 --2- 172.21.15.40:0/200292890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb86fb0 0x55c0fdb87500 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.492+0000 7f9479d91740 1 --2- 172.21.15.40:0/200292890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 0x55c0fdb69af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.492+0000 7f9479d91740 1 -- 172.21.15.40:0/200292890 >> 172.21.15.40:0/200292890 conn(0x55c0fdb3bd20 msgr2=0x55c0fdb84520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.492+0000 7f9479d91740 1 -- 172.21.15.40:0/200292890 shutdown_connections 2024-04-26T23:04:45.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.492+0000 7f9479d91740 1 -- 172.21.15.40:0/200292890 wait complete. 2024-04-26T23:04:45.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.493+0000 7f9479d91740 1 -- start start 2024-04-26T23:04:45.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.493+0000 7f9479d91740 1 Processor -- start 2024-04-26T23:04:45.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.494+0000 7f9479d91740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 0x55c0fdb55f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.494+0000 7f9479d91740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb86fb0 0x55c0fdb97850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.494+0000 7f9479d91740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 0x55c0fdb87aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.494+0000 7f9479d91740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c0fdaa7200 con 0x55c0fdb97d90 2024-04-26T23:04:45.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.494+0000 7f9479d91740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c0fdaa1ee0 con 0x55c0fdb69710 2024-04-26T23:04:45.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.494+0000 7f9479d91740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c0fdaaee70 con 0x55c0fdb86fb0 2024-04-26T23:04:45.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.494+0000 7f9479d91740 10 client.0 _ll_register_callbacks cb 0x55c0fda98270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:45.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.494+0000 7f9475000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 0x55c0fdb87aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.494+0000 7f9475000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 0x55c0fdb87aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47894/0 (socket says 172.21.15.40:47894) 2024-04-26T23:04:45.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.494+0000 7f9475000640 1 -- 172.21.15.40:0/221942093 learned_addr learned my addr 172.21.15.40:0/221942093 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:45.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f946fe00640 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 0x55c0fdb55f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f9467400640 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb86fb0 0x55c0fdb97850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f9475000640 1 -- 172.21.15.40:0/221942093 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb86fb0 msgr2=0x55c0fdb97850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f9475000640 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb86fb0 0x55c0fdb97850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f9475000640 1 -- 172.21.15.40:0/221942093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 msgr2=0x55c0fdb55f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f9475000640 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 0x55c0fdb55f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f9475000640 1 -- 172.21.15.40:0/221942093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0fd87e7d0 con 0x55c0fdb97d90 2024-04-26T23:04:45.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f946fe00640 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 0x55c0fdb55f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:45.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f9475000640 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 0x55c0fdb87aa0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f9468058d90 tx=0x7f9468007eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c08ac5d8ab8171f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f945d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f945d600640 1 -- 172.21.15.40:0/221942093 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9468087060 con 0x55c0fdb97d90 2024-04-26T23:04:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f945d600640 1 -- 172.21.15.40:0/221942093 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f9468071100 con 0x55c0fdb97d90 2024-04-26T23:04:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f9479d91740 1 -- 172.21.15.40:0/221942093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9458044070 con 0x55c0fdb97d90 2024-04-26T23:04:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f945d600640 1 -- 172.21.15.40:0/221942093 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f9468076050 con 0x55c0fdb97d90 2024-04-26T23:04:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f9479d91740 10 client.6324 fetch_fsmap learned FSMap version 42 2024-04-26T23:04:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.495+0000 7f9479d91740 1 -- 172.21.15.40:0/221942093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) -- 0x55c0fd7b43f0 con 0x55c0fdb97d90 2024-04-26T23:04:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.496+0000 7f945d600640 1 -- 172.21.15.40:0/221942093 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9458044070 con 0x55c0fdb97d90 2024-04-26T23:04:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.496+0000 7f945d600640 1 -- 172.21.15.40:0/221942093 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) ==== 3546+0+0 (secure 0 0 0) 0x7f94680782e0 con 0x55c0fdb97d90 2024-04-26T23:04:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.496+0000 7f9479d91740 10 client.6324 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T23:04:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.496+0000 7f9479d91740 10 client.6324 resolve_mds: resolved 3:2 to role '3:2' aka daemon mds.b 2024-04-26T23:04:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.496+0000 7f9479d91740 20 client.6324 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.496+0000 7f9479d91740 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x55c0fdb38960 0x55c0fdb38db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.496+0000 7f9479d91740 4 client.6324 mds_command: new command op to 5284 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:04:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.496+0000 7f9479d91740 1 -- 172.21.15.40:0/221942093 --> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0fdab07e0 con 0x55c0fdb38960 2024-04-26T23:04:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.497+0000 7f946fe00640 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x55c0fdb38960 0x55c0fdb38db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.498+0000 7f946fe00640 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x55c0fdb38960 0x55c0fdb38db0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e0e8a46b8f664269 server_cookie=811defe0a4698861 in_seq=0 out_seq=0 2024-04-26T23:04:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.498+0000 7f945d600640 10 client.6324 ms_handle_connect on v2:172.21.15.103:6836/670995452 2024-04-26T23:04:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.499+0000 7f945d600640 1 -- 172.21.15.40:0/221942093 <== mds.2 v2:172.21.15.103:6836/670995452 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0fdab07e0 con 0x55c0fdb38960 2024-04-26T23:04:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.499+0000 7f945d600640 10 client.6324 handle_command_reply: tid=0 2024-04-26T23:04:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.500+0000 7f9479d91740 1 client.6324 shutdown 2024-04-26T23:04:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.500+0000 7f9479d91740 1 -- 172.21.15.40:0/221942093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 msgr2=0x55c0fdb87aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.500+0000 7f9479d91740 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 0x55c0fdb87aa0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f9468058d90 tx=0x7f9468007eb0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.500+0000 7f9479d91740 1 -- 172.21.15.40:0/221942093 shutdown_connections 2024-04-26T23:04:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.500+0000 7f9479d91740 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x55c0fdb38960 0x55c0fdb38db0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.500+0000 7f9479d91740 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 0x55c0fdb87aa0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.500+0000 7f9479d91740 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb86fb0 0x55c0fdb97850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.500+0000 7f9479d91740 1 --2- 172.21.15.40:0/221942093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 0x55c0fdb55f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.500+0000 7f945d600640 0 client.6324 ms_handle_reset on v2:172.21.15.103:6836/670995452 2024-04-26T23:04:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.500+0000 7f9479d91740 1 -- 172.21.15.40:0/221942093 >> 172.21.15.40:0/221942093 conn(0x55c0fdb3f3f0 msgr2=0x55c0fdb83ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.500+0000 7f9479d91740 1 -- 172.21.15.40:0/221942093 shutdown_connections 2024-04-26T23:04:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.500+0000 7f9479d91740 1 -- 172.21.15.40:0/221942093 wait complete. 2024-04-26T23:04:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.500+0000 7f9479d91740 20 client.6324 trim_cache size 0 max 16384 2024-04-26T23:04:45.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.517+0000 7f9479d91740 1 Processor -- start 2024-04-26T23:04:45.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.517+0000 7f9479d91740 1 -- start start 2024-04-26T23:04:45.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.517+0000 7f9479d91740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 0x55c0fdb87460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9479d91740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb86fb0 0x55c0fdb879a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9479d91740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 0x55c0fdb844f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9479d91740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9470202250 con 0x55c0fdb97d90 2024-04-26T23:04:45.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9479d91740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9470201420 con 0x55c0fdb69710 2024-04-26T23:04:45.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9479d91740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9470201b20 con 0x55c0fdb86fb0 2024-04-26T23:04:45.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9475000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 0x55c0fdb844f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9475000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 0x55c0fdb844f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47908/0 (socket says 172.21.15.40:47908) 2024-04-26T23:04:45.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9475000640 1 -- 172.21.15.40:0/2483077579 learned_addr learned my addr 172.21.15.40:0/2483077579 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:45.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9467400640 1 --2- 172.21.15.40:0/2483077579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb86fb0 0x55c0fdb879a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9475000640 1 -- 172.21.15.40:0/2483077579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb86fb0 msgr2=0x55c0fdb879a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f946fe00640 1 --2- 172.21.15.40:0/2483077579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 0x55c0fdb87460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9475000640 1 --2- 172.21.15.40:0/2483077579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb86fb0 0x55c0fdb879a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9475000640 1 -- 172.21.15.40:0/2483077579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 msgr2=0x55c0fdb87460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9475000640 1 --2- 172.21.15.40:0/2483077579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 0x55c0fdb87460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9475000640 1 -- 172.21.15.40:0/2483077579 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0fd7740f0 con 0x55c0fdb97d90 2024-04-26T23:04:45.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.518+0000 7f9467400640 1 --2- 172.21.15.40:0/2483077579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb86fb0 0x55c0fdb879a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:45.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.519+0000 7f9475000640 1 --2- 172.21.15.40:0/2483077579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 0x55c0fdb844f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f94680570b0 tx=0x7f946806e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120186977c50c26a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.519+0000 7f945fe00640 1 -- 172.21.15.40:0/2483077579 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94680619e0 con 0x55c0fdb97d90 2024-04-26T23:04:45.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.519+0000 7f945fe00640 1 -- 172.21.15.40:0/2483077579 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94680782e0 con 0x55c0fdb97d90 2024-04-26T23:04:45.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.519+0000 7f945fe00640 1 -- 172.21.15.40:0/2483077579 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9468071100 con 0x55c0fdb97d90 2024-04-26T23:04:45.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.520+0000 7f9479d91740 1 -- 172.21.15.40:0/2483077579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 msgr2=0x55c0fdb844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.520+0000 7f9479d91740 1 --2- 172.21.15.40:0/2483077579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 0x55c0fdb844f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f94680570b0 tx=0x7f946806e000 comp rx=0 tx=0).stop 2024-04-26T23:04:45.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.520+0000 7f9479d91740 1 -- 172.21.15.40:0/2483077579 shutdown_connections 2024-04-26T23:04:45.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.520+0000 7f9479d91740 1 --2- 172.21.15.40:0/2483077579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb97d90 0x55c0fdb844f0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.520+0000 7f9479d91740 1 --2- 172.21.15.40:0/2483077579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb86fb0 0x55c0fdb879a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.520+0000 7f9479d91740 1 --2- 172.21.15.40:0/2483077579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb69710 0x55c0fdb87460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.520+0000 7f9479d91740 1 -- 172.21.15.40:0/2483077579 >> 172.21.15.40:0/2483077579 conn(0x55c0fdb3f3f0 msgr2=0x55c0fdb35080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.520+0000 7f9479d91740 1 -- 172.21.15.40:0/2483077579 shutdown_connections 2024-04-26T23:04:45.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.520+0000 7f9479d91740 1 -- 172.21.15.40:0/2483077579 wait complete. 2024-04-26T23:04:45.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.521+0000 7f9479d91740 1 -- start start 2024-04-26T23:04:45.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.521+0000 7f9479d91740 1 Processor -- start 2024-04-26T23:04:45.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f9479d91740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb69710 0x55c0fdbfd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f9479d91740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb86fb0 0x55c0fdb6c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f9479d91740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb97d90 0x55c0fdb87460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f9479d91740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c0fdaaffd0 con 0x55c0fdb69710 2024-04-26T23:04:45.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f9479d91740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c0fda44260 con 0x55c0fdb86fb0 2024-04-26T23:04:45.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f9479d91740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c0fdb62680 con 0x55c0fdb97d90 2024-04-26T23:04:45.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f9479d91740 10 client.0 _ll_register_callbacks cb 0x7f94680550a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:45.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f946fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb69710 0x55c0fdbfd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f946fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb69710 0x55c0fdbfd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47912/0 (socket says 172.21.15.40:47912) 2024-04-26T23:04:45.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f946fe00640 1 -- 172.21.15.40:0/3825695396 learned_addr learned my addr 172.21.15.40:0/3825695396 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:45.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f946fe00640 1 -- 172.21.15.40:0/3825695396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb97d90 msgr2=0x55c0fdb87460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f946fe00640 1 --2- 172.21.15.40:0/3825695396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb97d90 0x55c0fdb87460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f946fe00640 1 -- 172.21.15.40:0/3825695396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb86fb0 msgr2=0x55c0fdb6c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:45.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f946fe00640 1 --2- 172.21.15.40:0/3825695396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb86fb0 0x55c0fdb6c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.522+0000 7f946fe00640 1 -- 172.21.15.40:0/3825695396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0fda6aa30 con 0x55c0fdb69710 2024-04-26T23:04:45.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.523+0000 7f946fe00640 1 --2- 172.21.15.40:0/3825695396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb69710 0x55c0fdbfd0d0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f9460007da0 tx=0x7f9460004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315755b07418cc6e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:45.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.523+0000 7f945d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.523+0000 7f945d600640 1 -- 172.21.15.40:0/3825695396 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9460003da0 con 0x55c0fdb69710 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.523+0000 7f945d600640 1 -- 172.21.15.40:0/3825695396 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f94600258d0 con 0x55c0fdb69710 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.523+0000 7f9479d91740 1 -- 172.21.15.40:0/3825695396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94680782e0 con 0x55c0fdb69710 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.523+0000 7f945d600640 1 -- 172.21.15.40:0/3825695396 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f9460037050 con 0x55c0fdb69710 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.523+0000 7f9479d91740 10 client.6330 fetch_fsmap learned FSMap version 42 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.523+0000 7f9479d91740 1 -- 172.21.15.40:0/3825695396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) -- 0x55c0fdb86850 con 0x55c0fdb69710 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.524+0000 7f945d600640 1 -- 172.21.15.40:0/3825695396 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94680782e0 con 0x55c0fdb69710 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.524+0000 7f945d600640 1 -- 172.21.15.40:0/3825695396 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) ==== 3546+0+0 (secure 0 0 0) 0x7f946003a020 con 0x55c0fdb69710 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.524+0000 7f9479d91740 10 client.6330 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.524+0000 7f9479d91740 10 client.6330 resolve_mds: resolved 3:2 to role '3:2' aka daemon mds.b 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.524+0000 7f9479d91740 20 client.6330 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.524+0000 7f9479d91740 1 --2- 172.21.15.40:0/3825695396 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x55c0fdb879a0 0x55c0fdb64b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.524+0000 7f9479d91740 4 client.6330 mds_command: new command op to 5284 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.524+0000 7f9479d91740 1 -- 172.21.15.40:0/3825695396 --> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c0fdaaa650 con 0x55c0fdb879a0 2024-04-26T23:04:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.525+0000 7f9467400640 1 --2- 172.21.15.40:0/3825695396 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x55c0fdb879a0 0x55c0fdb64b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.526+0000 7f9467400640 1 --2- 172.21.15.40:0/3825695396 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x55c0fdb879a0 0x55c0fdb64b20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d5ace651bdd9f55f server_cookie=ed19435076c36c1c in_seq=0 out_seq=0 2024-04-26T23:04:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.526+0000 7f945d600640 10 client.6330 ms_handle_connect on v2:172.21.15.103:6836/670995452 2024-04-26T23:04:45.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.546+0000 7f945d600640 1 -- 172.21.15.40:0/3825695396 <== mds.2 v2:172.21.15.103:6836/670995452 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c0fdaaa650 con 0x55c0fdb879a0 2024-04-26T23:04:45.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.546+0000 7f945d600640 10 client.6330 handle_command_reply: tid=0 2024-04-26T23:04:45.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.546+0000 7f9479d91740 1 client.6330 shutdown 2024-04-26T23:04:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.547+0000 7f9479d91740 1 -- 172.21.15.40:0/3825695396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb69710 msgr2=0x55c0fdbfd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.547+0000 7f9479d91740 1 --2- 172.21.15.40:0/3825695396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb69710 0x55c0fdbfd0d0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f9460007da0 tx=0x7f9460004a40 comp rx=0 tx=0).stop 2024-04-26T23:04:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.547+0000 7f9479d91740 1 -- 172.21.15.40:0/3825695396 shutdown_connections 2024-04-26T23:04:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.547+0000 7f9479d91740 1 --2- 172.21.15.40:0/3825695396 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x55c0fdb879a0 0x55c0fdb64b20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.547+0000 7f9479d91740 1 --2- 172.21.15.40:0/3825695396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c0fdb97d90 0x55c0fdb87460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.547+0000 7f9479d91740 1 --2- 172.21.15.40:0/3825695396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0fdb86fb0 0x55c0fdb6c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.547+0000 7f945d600640 0 client.6330 ms_handle_reset on v2:172.21.15.103:6836/670995452 2024-04-26T23:04:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.547+0000 7f9479d91740 1 --2- 172.21.15.40:0/3825695396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0fdb69710 0x55c0fdbfd0d0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.547+0000 7f9479d91740 1 -- 172.21.15.40:0/3825695396 >> 172.21.15.40:0/3825695396 conn(0x55c0fdb3f3f0 msgr2=0x55c0fdb6cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.547+0000 7f9479d91740 1 -- 172.21.15.40:0/3825695396 shutdown_connections 2024-04-26T23:04:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.547+0000 7f9479d91740 1 -- 172.21.15.40:0/3825695396 wait complete. 2024-04-26T23:04:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.547+0000 7f9479d91740 20 client.6330 trim_cache size 0 max 16384 2024-04-26T23:04:45.549 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:04:45.549 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:04:45.549 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:04:45.549 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:04:45.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.548+0000 7f9475a00640 1 -- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f944c06c8f0 msgr2=0x7f944c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.548+0000 7f9475a00640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f944c06c8f0 0x7f944c06edb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9470162f20 tx=0x7f9460009750 comp rx=0 tx=0).stop 2024-04-26T23:04:45.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.548+0000 7f9475a00640 1 -- 172.21.15.40:0/232556194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94701583d0 msgr2=0x7f94701f82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.548+0000 7f9475a00640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94701583d0 0x7f94701f82f0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f947008d490 tx=0x7f946800e820 comp rx=0 tx=0).stop 2024-04-26T23:04:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.548+0000 7f9475a00640 1 -- 172.21.15.40:0/232556194 shutdown_connections 2024-04-26T23:04:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.548+0000 7f9475a00640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f944c06c8f0 0x7f944c06edb0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.548+0000 7f9475a00640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94701583d0 0x7f94701f82f0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.548+0000 7f9475a00640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94700cc890 0x7f9470157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.548+0000 7f9475a00640 1 --2- 172.21.15.40:0/232556194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9454004800 0x7f94701576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.548+0000 7f9475a00640 1 -- 172.21.15.40:0/232556194 >> 172.21.15.40:0/232556194 conn(0x7f94700d4dd0 msgr2=0x7f947006e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.549+0000 7f9475a00640 1 -- 172.21.15.40:0/232556194 shutdown_connections 2024-04-26T23:04:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45.549+0000 7f9475a00640 1 -- 172.21.15.40:0/232556194 wait complete. 2024-04-26T23:04:45.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:45.567 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec find d1/d2/d3/) 2024-04-26T23:04:45.638 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/ 2024-04-26T23:04:45.638 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4 2024-04-26T23:04:45.638 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5 2024-04-26T23:04:45.638 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6 2024-04-26T23:04:45.638 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6/d7 2024-04-26T23:04:45.638 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6/d7/d8 2024-04-26T23:04:45.639 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:04:45.639 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:04:45.639 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:04:45.639 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:04:45.640 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/").st_ino) 2024-04-26T23:04:45.640 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:04:45.788 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627778 2024-04-26T23:04:45.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:04:45.802 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:04:45.802 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:04:45.803 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:04:45.803 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:04:45.803 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4").st_ino) 2024-04-26T23:04:45.803 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:04:45.999 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627779 2024-04-26T23:04:46.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:04:46.014 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:04:46.014 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:04:46.014 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:04:46.014 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:04:46.014 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5").st_ino) 2024-04-26T23:04:46.014 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:04:46.206 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627780 2024-04-26T23:04:46.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:04:46.221 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:04:46.221 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:04:46.221 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:04:46.221 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:04:46.221 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6").st_ino) 2024-04-26T23:04:46.221 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:04:46.415 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627781 2024-04-26T23:04:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:04:46.430 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:04:46.430 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:04:46.430 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:04:46.430 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:04:46.430 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6/d7").st_ino) 2024-04-26T23:04:46.430 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:04:46.621 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627782 2024-04-26T23:04:46.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:04:46.636 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:04:46.636 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:04:46.636 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:04:46.636 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:04:46.636 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6/d7/d8").st_ino) 2024-04-26T23:04:46.636 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:04:46.830 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627783 2024-04-26T23:04:46.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:04:46.844 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000002.00000000 parent 2024-04-26T23:04:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.004+0000 7f5d71e27f40 1 Processor -- start 2024-04-26T23:04:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.005+0000 7f5d71e27f40 1 -- start start 2024-04-26T23:04:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.005+0000 7f5d71e27f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556c5f31a080 0x556c5f344620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.005+0000 7f5d71e27f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c5f348360 0x556c5f361380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.005+0000 7f5d71e27f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c5f3eb280 0x556c5f3618c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.005+0000 7f5d71e27f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556c5f3487d0 con 0x556c5f3eb280 2024-04-26T23:04:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.005+0000 7f5d71e27f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556c5f2bfd20 con 0x556c5f348360 2024-04-26T23:04:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.005+0000 7f5d71e27f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556c5f363a60 con 0x556c5f31a080 2024-04-26T23:04:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.006+0000 7f5d6fa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c5f3eb280 0x556c5f3618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.006+0000 7f5d6fa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c5f3eb280 0x556c5f3618c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47924/0 (socket says 172.21.15.40:47924) 2024-04-26T23:04:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.006+0000 7f5d6fa00640 1 -- 172.21.15.40:0/2174139542 learned_addr learned my addr 172.21.15.40:0/2174139542 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.006+0000 7f5d66600640 1 --2- 172.21.15.40:0/2174139542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c5f348360 0x556c5f361380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.006+0000 7f5d6fa00640 1 -- 172.21.15.40:0/2174139542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556c5f31a080 msgr2=0x556c5f344620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.006+0000 7f5d6fa00640 1 --2- 172.21.15.40:0/2174139542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556c5f31a080 0x556c5f344620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.006+0000 7f5d6fa00640 1 -- 172.21.15.40:0/2174139542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c5f348360 msgr2=0x556c5f361380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.006+0000 7f5d6fa00640 1 --2- 172.21.15.40:0/2174139542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c5f348360 0x556c5f361380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.006+0000 7f5d6fa00640 1 -- 172.21.15.40:0/2174139542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c5f344b60 con 0x556c5f3eb280 2024-04-26T23:04:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.007+0000 7f5d6fa00640 1 --2- 172.21.15.40:0/2174139542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c5f3eb280 0x556c5f3618c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800a770 tx=0x7f5d6800a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9e32c423b7b7ef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.007+0000 7f5d6dc00640 1 -- 172.21.15.40:0/2174139542 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68019070 con 0x556c5f3eb280 2024-04-26T23:04:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.007+0000 7f5d6dc00640 1 -- 172.21.15.40:0/2174139542 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d68007600 con 0x556c5f3eb280 2024-04-26T23:04:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.007+0000 7f5d6dc00640 1 -- 172.21.15.40:0/2174139542 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68007a10 con 0x556c5f3eb280 2024-04-26T23:04:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.007+0000 7f5d71e27f40 1 -- 172.21.15.40:0/2174139542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c5f3eb280 msgr2=0x556c5f3618c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.007+0000 7f5d71e27f40 1 --2- 172.21.15.40:0/2174139542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c5f3eb280 0x556c5f3618c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800a770 tx=0x7f5d6800a860 comp rx=0 tx=0).stop 2024-04-26T23:04:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.007+0000 7f5d71e27f40 1 -- 172.21.15.40:0/2174139542 shutdown_connections 2024-04-26T23:04:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.007+0000 7f5d71e27f40 1 --2- 172.21.15.40:0/2174139542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c5f3eb280 0x556c5f3618c0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.007+0000 7f5d71e27f40 1 --2- 172.21.15.40:0/2174139542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c5f348360 0x556c5f361380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.007+0000 7f5d71e27f40 1 --2- 172.21.15.40:0/2174139542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556c5f31a080 0x556c5f344620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.007+0000 7f5d71e27f40 1 -- 172.21.15.40:0/2174139542 >> 172.21.15.40:0/2174139542 conn(0x556c5f345c00 msgr2=0x556c5f363350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.008+0000 7f5d71e27f40 1 -- 172.21.15.40:0/2174139542 shutdown_connections 2024-04-26T23:04:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.008+0000 7f5d71e27f40 1 -- 172.21.15.40:0/2174139542 wait complete. 2024-04-26T23:04:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.008+0000 7f5d71e27f40 1 Processor -- start 2024-04-26T23:04:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.009+0000 7f5d71e27f40 1 -- start start 2024-04-26T23:04:47.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.009+0000 7f5d71e27f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c5f31a080 0x556c5f363350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.009+0000 7f5d71e27f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556c5f348360 0x556c5f3e2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.009+0000 7f5d71e27f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c5f3eb280 0x556c5f3e6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.009+0000 7f5d71e27f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556c5f3ea490 con 0x556c5f3eb280 2024-04-26T23:04:47.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.009+0000 7f5d71e27f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556c5f343eb0 con 0x556c5f31a080 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.009+0000 7f5d71e27f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556c5f346080 con 0x556c5f348360 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.010+0000 7f5d6fa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c5f3eb280 0x556c5f3e6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.010+0000 7f5d6fa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c5f3eb280 0x556c5f3e6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47930/0 (socket says 172.21.15.40:47930) 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.010+0000 7f5d6fa00640 1 -- 172.21.15.40:0/3418965098 learned_addr learned my addr 172.21.15.40:0/3418965098 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.010+0000 7f5d6f000640 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c5f31a080 0x556c5f363350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.010+0000 7f5d66600640 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556c5f348360 0x556c5f3e2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.010+0000 7f5d6fa00640 1 -- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556c5f348360 msgr2=0x556c5f3e2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.010+0000 7f5d6fa00640 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556c5f348360 0x556c5f3e2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.010+0000 7f5d6fa00640 1 -- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c5f31a080 msgr2=0x556c5f363350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.010+0000 7f5d6fa00640 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c5f31a080 0x556c5f363350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.010+0000 7f5d6fa00640 1 -- 172.21.15.40:0/3418965098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c5f342e60 con 0x556c5f3eb280 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.011+0000 7f5d6fa00640 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c5f3eb280 0x556c5f3e6f80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5d68013000 tx=0x7f5d68003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.011+0000 7f5d67e00640 1 -- 172.21.15.40:0/3418965098 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68023020 con 0x556c5f3eb280 2024-04-26T23:04:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.011+0000 7f5d67e00640 1 -- 172.21.15.40:0/3418965098 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d6801f690 con 0x556c5f3eb280 2024-04-26T23:04:47.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.011+0000 7f5d71e27f40 1 -- 172.21.15.40:0/3418965098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556c5f3e36f0 con 0x556c5f3eb280 2024-04-26T23:04:47.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.011+0000 7f5d67e00640 1 -- 172.21.15.40:0/3418965098 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68019040 con 0x556c5f3eb280 2024-04-26T23:04:47.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.011+0000 7f5d71e27f40 1 -- 172.21.15.40:0/3418965098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c5f3e8af0 con 0x556c5f3eb280 2024-04-26T23:04:47.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.014+0000 7f5d67e00640 1 -- 172.21.15.40:0/3418965098 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d68026020 con 0x556c5f3eb280 2024-04-26T23:04:47.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.014+0000 7f5d67e00640 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5d4c06b2d0 0x7f5d4c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.015+0000 7f5d67e00640 1 -- 172.21.15.40:0/3418965098 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f5d6808c280 con 0x556c5f3eb280 2024-04-26T23:04:47.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.015+0000 7f5d71e27f40 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x556c5f3e2bd0 0x556c5f3e74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.015+0000 7f5d71e27f40 1 -- 172.21.15.40:0/3418965098 --> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] -- osd_op(unknown.0.0:1 7.12 7:4bec3886:::10000000002.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52) -- 0x556c5f3e50f0 con 0x556c5f3e2bd0 2024-04-26T23:04:47.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.015+0000 7f5d6f000640 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5d4c06b2d0 0x7f5d4c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.016+0000 7f5d66600640 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x556c5f3e2bd0 0x556c5f3e74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.016+0000 7f5d6f000640 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5d4c06b2d0 0x7f5d4c06d790 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000c470 tx=0x7f5d60007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.017+0000 7f5d66600640 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x556c5f3e2bd0 0x556c5f3e74c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.020+0000 7f5d66600640 1 -- 172.21.15.40:0/3418965098 <== osd.6 v2:172.21.15.103:6816/293480649 1 ==== osd_op_reply(1 10000000002.00000000 [rmxattr] v52'8 uv8 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5d58054870 con 0x556c5f3e2bd0 2024-04-26T23:04:47.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.020+0000 7f5d71e27f40 1 -- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x556c5f3e2bd0 msgr2=0x556c5f3e74c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.020+0000 7f5d71e27f40 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x556c5f3e2bd0 0x556c5f3e74c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.020+0000 7f5d71e27f40 1 -- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5d4c06b2d0 msgr2=0x7f5d4c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.020+0000 7f5d71e27f40 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5d4c06b2d0 0x7f5d4c06d790 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000c470 tx=0x7f5d60007040 comp rx=0 tx=0).stop 2024-04-26T23:04:47.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.020+0000 7f5d71e27f40 1 -- 172.21.15.40:0/3418965098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c5f3eb280 msgr2=0x556c5f3e6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.020+0000 7f5d71e27f40 1 --2- 172.21.15.40:0/3418965098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c5f3eb280 0x556c5f3e6f80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5d68013000 tx=0x7f5d68003020 comp rx=0 tx=0).stop 2024-04-26T23:04:47.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.020+0000 7f5d6fa00640 1 -- 172.21.15.40:0/3418965098 reap_dead start 2024-04-26T23:04:47.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.020+0000 7f5d71e27f40 1 -- 172.21.15.40:0/3418965098 shutdown_connections 2024-04-26T23:04:47.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.020+0000 7f5d71e27f40 1 -- 172.21.15.40:0/3418965098 >> 172.21.15.40:0/3418965098 conn(0x556c5f345c00 msgr2=0x556c5f3e4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.020+0000 7f5d71e27f40 1 -- 172.21.15.40:0/3418965098 shutdown_connections 2024-04-26T23:04:47.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.020+0000 7f5d71e27f40 1 -- 172.21.15.40:0/3418965098 wait complete. 2024-04-26T23:04:47.025 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000003.00000000 parent 2024-04-26T23:04:47.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.129+0000 7fc489b74f40 1 Processor -- start 2024-04-26T23:04:47.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.129+0000 7fc489b74f40 1 -- start start 2024-04-26T23:04:47.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.130+0000 7fc489b74f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc46c0022d0 0x557b39c5b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.130+0000 7fc489b74f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b39c5bc50 0x557b39c56a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.130+0000 7fc489b74f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b39c54de0 0x557b39bcf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.130+0000 7fc489b74f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557b39b2ed20 con 0x557b39c54de0 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.130+0000 7fc489b74f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557b39b32d60 con 0x7fc46c0022d0 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.130+0000 7fc489b74f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x557b39bd2a60 con 0x557b39c5bc50 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.130+0000 7fc487800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b39c54de0 0x557b39bcf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.131+0000 7fc487800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b39c54de0 0x557b39bcf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47952/0 (socket says 172.21.15.40:47952) 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.131+0000 7fc487800640 1 -- 172.21.15.40:0/1034895624 learned_addr learned my addr 172.21.15.40:0/1034895624 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.131+0000 7fc486e00640 1 --2- 172.21.15.40:0/1034895624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc46c0022d0 0x557b39c5b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.131+0000 7fc487800640 1 -- 172.21.15.40:0/1034895624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b39c5bc50 msgr2=0x557b39c56a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.131+0000 7fc486400640 1 --2- 172.21.15.40:0/1034895624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b39c5bc50 0x557b39c56a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.131+0000 7fc487800640 1 --2- 172.21.15.40:0/1034895624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b39c5bc50 0x557b39c56a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.131+0000 7fc487800640 1 -- 172.21.15.40:0/1034895624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc46c0022d0 msgr2=0x557b39c5b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.131+0000 7fc487800640 1 --2- 172.21.15.40:0/1034895624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc46c0022d0 0x557b39c5b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.131+0000 7fc487800640 1 -- 172.21.15.40:0/1034895624 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b39bb52b0 con 0x557b39c54de0 2024-04-26T23:04:47.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.131+0000 7fc486e00640 1 --2- 172.21.15.40:0/1034895624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc46c0022d0 0x557b39c5b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:47.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.131+0000 7fc487800640 1 --2- 172.21.15.40:0/1034895624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b39c54de0 0x557b39bcf370 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc4800025f0 tx=0x7fc480002ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b07c968d4510afa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.132+0000 7fc485000640 1 -- 172.21.15.40:0/1034895624 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480016020 con 0x557b39c54de0 2024-04-26T23:04:47.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.132+0000 7fc485000640 1 -- 172.21.15.40:0/1034895624 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc480012680 con 0x557b39c54de0 2024-04-26T23:04:47.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.132+0000 7fc485000640 1 -- 172.21.15.40:0/1034895624 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480012a70 con 0x557b39c54de0 2024-04-26T23:04:47.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.132+0000 7fc489b74f40 1 -- 172.21.15.40:0/1034895624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b39c54de0 msgr2=0x557b39bcf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.132+0000 7fc489b74f40 1 --2- 172.21.15.40:0/1034895624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b39c54de0 0x557b39bcf370 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc4800025f0 tx=0x7fc480002ab0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.132+0000 7fc489b74f40 1 -- 172.21.15.40:0/1034895624 shutdown_connections 2024-04-26T23:04:47.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.132+0000 7fc489b74f40 1 --2- 172.21.15.40:0/1034895624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b39c54de0 0x557b39bcf370 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.132+0000 7fc489b74f40 1 --2- 172.21.15.40:0/1034895624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b39c5bc50 0x557b39c56a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.132+0000 7fc489b74f40 1 --2- 172.21.15.40:0/1034895624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc46c0022d0 0x557b39c5b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.132+0000 7fc489b74f40 1 -- 172.21.15.40:0/1034895624 >> 172.21.15.40:0/1034895624 conn(0x557b39bb4c00 msgr2=0x557b39b8af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.133+0000 7fc489b74f40 1 -- 172.21.15.40:0/1034895624 shutdown_connections 2024-04-26T23:04:47.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.133+0000 7fc489b74f40 1 -- 172.21.15.40:0/1034895624 wait complete. 2024-04-26T23:04:47.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.133+0000 7fc489b74f40 1 Processor -- start 2024-04-26T23:04:47.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.134+0000 7fc489b74f40 1 -- start start 2024-04-26T23:04:47.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc489b74f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b39c54de0 0x557b39bc9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc489b74f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b39c5bc50 0x557b39c434d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc489b74f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc46c0022d0 0x557b39c58330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc489b74f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557b39c54c10 con 0x557b39c54de0 2024-04-26T23:04:47.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc489b74f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557b39bd2550 con 0x557b39c5bc50 2024-04-26T23:04:47.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc489b74f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x557b39c5c3c0 con 0x7fc46c0022d0 2024-04-26T23:04:47.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc486e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b39c54de0 0x557b39bc9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc486e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b39c54de0 0x557b39bc9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47960/0 (socket says 172.21.15.40:47960) 2024-04-26T23:04:47.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc486e00640 1 -- 172.21.15.40:0/3554396057 learned_addr learned my addr 172.21.15.40:0/3554396057 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc486400640 1 --2- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b39c5bc50 0x557b39c434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc486e00640 1 -- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc46c0022d0 msgr2=0x557b39c58330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc486e00640 1 --2- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc46c0022d0 0x557b39c58330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc486e00640 1 -- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b39c5bc50 msgr2=0x557b39c434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc486e00640 1 --2- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b39c5bc50 0x557b39c434d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.135+0000 7fc486e00640 1 -- 172.21.15.40:0/3554396057 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b39bc8690 con 0x557b39c54de0 2024-04-26T23:04:47.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.136+0000 7fc486e00640 1 --2- 172.21.15.40:0/3554396057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b39c54de0 0x557b39bc9b80 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc4780258a0 tx=0x7fc478001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.136+0000 7fc47f400640 1 -- 172.21.15.40:0/3554396057 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc478061070 con 0x557b39c54de0 2024-04-26T23:04:47.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.136+0000 7fc47f400640 1 -- 172.21.15.40:0/3554396057 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc478000ed0 con 0x557b39c54de0 2024-04-26T23:04:47.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.136+0000 7fc47f400640 1 -- 172.21.15.40:0/3554396057 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4780014e0 con 0x557b39c54de0 2024-04-26T23:04:47.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.136+0000 7fc489b74f40 1 -- 172.21.15.40:0/3554396057 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557b39bc8970 con 0x557b39c54de0 2024-04-26T23:04:47.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.136+0000 7fc489b74f40 1 -- 172.21.15.40:0/3554396057 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b39bc8f30 con 0x557b39c54de0 2024-04-26T23:04:47.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.138+0000 7fc47f400640 1 -- 172.21.15.40:0/3554396057 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc478064070 con 0x557b39c54de0 2024-04-26T23:04:47.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.139+0000 7fc47f400640 1 --2- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc46806b2d0 0x7fc46806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.139+0000 7fc47f400640 1 -- 172.21.15.40:0/3554396057 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7fc4780cdd80 con 0x557b39c54de0 2024-04-26T23:04:47.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.140+0000 7fc489b74f40 1 --2- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x557b39c577f0 0x557b39bcbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.140+0000 7fc489b74f40 1 -- 172.21.15.40:0/3554396057 --> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] -- osd_op(unknown.0.0:1 7.12 7:4def72d4:::10000000003.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52) -- 0x557b39bcdf90 con 0x557b39c577f0 2024-04-26T23:04:47.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.140+0000 7fc486400640 1 --2- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc46806b2d0 0x7fc46806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.140+0000 7fc487800640 1 --2- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x557b39c577f0 0x557b39bcbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.141+0000 7fc486400640 1 --2- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc46806b2d0 0x7fc46806d790 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc4740017e0 tx=0x7fc474001430 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.141+0000 7fc487800640 1 --2- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x557b39c577f0 0x557b39bcbb40 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.144+0000 7fc487800640 1 -- 172.21.15.40:0/3554396057 <== osd.6 v2:172.21.15.103:6816/293480649 1 ==== osd_op_reply(1 10000000003.00000000 [rmxattr] v52'9 uv9 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc480012a70 con 0x557b39c577f0 2024-04-26T23:04:47.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.144+0000 7fc489b74f40 1 -- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x557b39c577f0 msgr2=0x557b39bcbb40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.144+0000 7fc489b74f40 1 --2- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x557b39c577f0 0x557b39bcbb40 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.145+0000 7fc489b74f40 1 -- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc46806b2d0 msgr2=0x7fc46806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.145+0000 7fc489b74f40 1 --2- 172.21.15.40:0/3554396057 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc46806b2d0 0x7fc46806d790 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc4740017e0 tx=0x7fc474001430 comp rx=0 tx=0).stop 2024-04-26T23:04:47.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.145+0000 7fc489b74f40 1 -- 172.21.15.40:0/3554396057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b39c54de0 msgr2=0x557b39bc9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.145+0000 7fc489b74f40 1 --2- 172.21.15.40:0/3554396057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b39c54de0 0x557b39bc9b80 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc4780258a0 tx=0x7fc478001120 comp rx=0 tx=0).stop 2024-04-26T23:04:47.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.145+0000 7fc487800640 1 -- 172.21.15.40:0/3554396057 reap_dead start 2024-04-26T23:04:47.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.145+0000 7fc489b74f40 1 -- 172.21.15.40:0/3554396057 shutdown_connections 2024-04-26T23:04:47.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.145+0000 7fc489b74f40 1 -- 172.21.15.40:0/3554396057 >> 172.21.15.40:0/3554396057 conn(0x557b39bb4c00 msgr2=0x557b39c57c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.145+0000 7fc489b74f40 1 -- 172.21.15.40:0/3554396057 shutdown_connections 2024-04-26T23:04:47.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.145+0000 7fc489b74f40 1 -- 172.21.15.40:0/3554396057 wait complete. 2024-04-26T23:04:47.149 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000004.00000000 parent 2024-04-26T23:04:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.258+0000 7f9186e49f40 1 Processor -- start 2024-04-26T23:04:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.258+0000 7f9186e49f40 1 -- start start 2024-04-26T23:04:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.259+0000 7f9186e49f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558ac82e5e90 0x558ac8300590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.259+0000 7f9186e49f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ac82e7070 0x558ac83852e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.259+0000 7f9186e49f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ac82fe240 0x558ac838a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.259+0000 7f9186e49f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558ac838c030 con 0x558ac82e7070 2024-04-26T23:04:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.259+0000 7f9186e49f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558ac825bd20 con 0x558ac82fe240 2024-04-26T23:04:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.259+0000 7f9186e49f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558ac838c310 con 0x558ac82e5e90 2024-04-26T23:04:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.259+0000 7f917b600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ac82e7070 0x558ac83852e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.259+0000 7f917b600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ac82e7070 0x558ac83852e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47968/0 (socket says 172.21.15.40:47968) 2024-04-26T23:04:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.259+0000 7f917b600640 1 -- 172.21.15.40:0/2817690136 learned_addr learned my addr 172.21.15.40:0/2817690136 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.260+0000 7f917b600640 1 -- 172.21.15.40:0/2817690136 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558ac82e5e90 msgr2=0x558ac8300590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.260+0000 7f917b600640 1 --2- 172.21.15.40:0/2817690136 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558ac82e5e90 0x558ac8300590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.260+0000 7f917b600640 1 -- 172.21.15.40:0/2817690136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ac82fe240 msgr2=0x558ac838a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.260+0000 7f9184a00640 1 --2- 172.21.15.40:0/2817690136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ac82fe240 0x558ac838a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.260+0000 7f917b600640 1 --2- 172.21.15.40:0/2817690136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ac82fe240 0x558ac838a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.260+0000 7f917b600640 1 -- 172.21.15.40:0/2817690136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ac82e6590 con 0x558ac82e7070 2024-04-26T23:04:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.260+0000 7f9184a00640 1 --2- 172.21.15.40:0/2817690136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ac82fe240 0x558ac838a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.260+0000 7f917b600640 1 --2- 172.21.15.40:0/2817690136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ac82e7070 0x558ac83852e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f916c000c00 tx=0x7f916c051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea1140a07d3fb44 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.260+0000 7f9182c00640 1 -- 172.21.15.40:0/2817690136 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c056030 con 0x558ac82e7070 2024-04-26T23:04:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.261+0000 7f9182c00640 1 -- 172.21.15.40:0/2817690136 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f916c052ab0 con 0x558ac82e7070 2024-04-26T23:04:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.261+0000 7f9182c00640 1 -- 172.21.15.40:0/2817690136 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c05b450 con 0x558ac82e7070 2024-04-26T23:04:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.261+0000 7f9186e49f40 1 -- 172.21.15.40:0/2817690136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ac82e7070 msgr2=0x558ac83852e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.261+0000 7f9186e49f40 1 --2- 172.21.15.40:0/2817690136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ac82e7070 0x558ac83852e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f916c000c00 tx=0x7f916c051000 comp rx=0 tx=0).stop 2024-04-26T23:04:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.261+0000 7f9186e49f40 1 -- 172.21.15.40:0/2817690136 shutdown_connections 2024-04-26T23:04:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.261+0000 7f9186e49f40 1 --2- 172.21.15.40:0/2817690136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ac82fe240 0x558ac838a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.261+0000 7f9186e49f40 1 --2- 172.21.15.40:0/2817690136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ac82e7070 0x558ac83852e0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.261+0000 7f9186e49f40 1 --2- 172.21.15.40:0/2817690136 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558ac82e5e90 0x558ac8300590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.261+0000 7f9186e49f40 1 -- 172.21.15.40:0/2817690136 >> 172.21.15.40:0/2817690136 conn(0x558ac8301df0 msgr2=0x558ac838b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.261+0000 7f9186e49f40 1 -- 172.21.15.40:0/2817690136 shutdown_connections 2024-04-26T23:04:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.261+0000 7f9186e49f40 1 -- 172.21.15.40:0/2817690136 wait complete. 2024-04-26T23:04:47.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.262+0000 7f9186e49f40 1 Processor -- start 2024-04-26T23:04:47.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.262+0000 7f9186e49f40 1 -- start start 2024-04-26T23:04:47.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.263+0000 7f9186e49f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ac82e5e90 0x558ac84154b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.263+0000 7f9186e49f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ac82e7070 0x558ac8418510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.263+0000 7f9186e49f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558ac82fe240 0x558ac8418a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.263+0000 7f9186e49f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558ac8385820 con 0x558ac82e5e90 2024-04-26T23:04:47.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.263+0000 7f9186e49f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558ac838ebb0 con 0x558ac82e7070 2024-04-26T23:04:47.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.263+0000 7f9186e49f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558ac838e280 con 0x558ac82fe240 2024-04-26T23:04:47.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.263+0000 7f9184000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ac82e5e90 0x558ac84154b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.264+0000 7f9184000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ac82e5e90 0x558ac84154b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47976/0 (socket says 172.21.15.40:47976) 2024-04-26T23:04:47.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.264+0000 7f9184000640 1 -- 172.21.15.40:0/1554707650 learned_addr learned my addr 172.21.15.40:0/1554707650 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.264+0000 7f917b600640 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ac82e7070 0x558ac8418510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.264+0000 7f9184a00640 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558ac82fe240 0x558ac8418a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.264+0000 7f9184000640 1 -- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558ac82fe240 msgr2=0x558ac8418a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.264+0000 7f9184000640 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558ac82fe240 0x558ac8418a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.264+0000 7f9184000640 1 -- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ac82e7070 msgr2=0x558ac8418510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.264+0000 7f9184000640 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ac82e7070 0x558ac8418510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.264+0000 7f9184000640 1 -- 172.21.15.40:0/1554707650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ac8385410 con 0x558ac82e5e90 2024-04-26T23:04:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.264+0000 7f9184a00640 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558ac82fe240 0x558ac8418a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.264+0000 7f9184000640 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ac82e5e90 0x558ac84154b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9174009d00 tx=0x7f9174006750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.265+0000 7f9181800640 1 -- 172.21.15.40:0/1554707650 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9174017070 con 0x558ac82e5e90 2024-04-26T23:04:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.265+0000 7f9181800640 1 -- 172.21.15.40:0/1554707650 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91740090d0 con 0x558ac82e5e90 2024-04-26T23:04:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.265+0000 7f9181800640 1 -- 172.21.15.40:0/1554707650 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9174012650 con 0x558ac82e5e90 2024-04-26T23:04:47.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.265+0000 7f9186e49f40 1 -- 172.21.15.40:0/1554707650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558ac82e2c80 con 0x558ac82e5e90 2024-04-26T23:04:47.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.265+0000 7f9186e49f40 1 -- 172.21.15.40:0/1554707650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ac84159f0 con 0x558ac82e5e90 2024-04-26T23:04:47.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.267+0000 7f9181800640 1 -- 172.21.15.40:0/1554707650 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f917401a070 con 0x558ac82e5e90 2024-04-26T23:04:47.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.268+0000 7f9181800640 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f916006b2d0 0x7f916006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.268+0000 7f9181800640 1 -- 172.21.15.40:0/1554707650 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6101+0+0 (secure 0 0 0) 0x7f9174089a90 con 0x558ac82e5e90 2024-04-26T23:04:47.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.268+0000 7f9186e49f40 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x558ac84194a0 0x558ac83827b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.268+0000 7f9186e49f40 1 -- 172.21.15.40:0/1554707650 --> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] -- osd_op(unknown.0.0:1 7.5 7:a367a3e8:::10000000004.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52) -- 0x558ac8382cf0 con 0x558ac84194a0 2024-04-26T23:04:47.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.269+0000 7f917b600640 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f916006b2d0 0x7f916006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.269+0000 7f9184a00640 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x558ac84194a0 0x558ac83827b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.270+0000 7f917b600640 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f916006b2d0 0x7f916006d790 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f916c045ae0 tx=0x7f916c053000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.270+0000 7f9184a00640 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x558ac84194a0 0x558ac83827b0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.273+0000 7f9184a00640 1 -- 172.21.15.40:0/1554707650 <== osd.7 v2:172.21.15.103:6819/3769540391 1 ==== osd_op_reply(1 10000000004.00000000 [rmxattr] v52'6 uv6 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f917c00c640 con 0x558ac84194a0 2024-04-26T23:04:47.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.273+0000 7f9186e49f40 1 -- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x558ac84194a0 msgr2=0x558ac83827b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.273+0000 7f9186e49f40 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x558ac84194a0 0x558ac83827b0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.273+0000 7f9186e49f40 1 -- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f916006b2d0 msgr2=0x7f916006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.274+0000 7f9186e49f40 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f916006b2d0 0x7f916006d790 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f916c045ae0 tx=0x7f916c053000 comp rx=0 tx=0).stop 2024-04-26T23:04:47.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.274+0000 7f9186e49f40 1 -- 172.21.15.40:0/1554707650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ac82e5e90 msgr2=0x558ac84154b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.274+0000 7f9186e49f40 1 --2- 172.21.15.40:0/1554707650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ac82e5e90 0x558ac84154b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9174009d00 tx=0x7f9174006750 comp rx=0 tx=0).stop 2024-04-26T23:04:47.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.274+0000 7f9184a00640 1 -- 172.21.15.40:0/1554707650 reap_dead start 2024-04-26T23:04:47.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.274+0000 7f9186e49f40 1 -- 172.21.15.40:0/1554707650 shutdown_connections 2024-04-26T23:04:47.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.274+0000 7f9186e49f40 1 -- 172.21.15.40:0/1554707650 >> 172.21.15.40:0/1554707650 conn(0x558ac8301df0 msgr2=0x558ac838d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.274+0000 7f9186e49f40 1 -- 172.21.15.40:0/1554707650 shutdown_connections 2024-04-26T23:04:47.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.274+0000 7f9186e49f40 1 -- 172.21.15.40:0/1554707650 wait complete. 2024-04-26T23:04:47.279 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000005.00000000 parent 2024-04-26T23:04:47.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.395+0000 7f7415108f40 1 Processor -- start 2024-04-26T23:04:47.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.395+0000 7f7415108f40 1 -- start start 2024-04-26T23:04:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.396+0000 7f7415108f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80022a0 0x555cf897f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.396+0000 7f7415108f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555cf8976bb0 0x555cf88d98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.396+0000 7f7415108f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cf88d9e10 0x555cf897ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.396+0000 7f7415108f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555cf8851d20 con 0x7f73f80022a0 2024-04-26T23:04:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.396+0000 7f7415108f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555cf8855d60 con 0x555cf88d9e10 2024-04-26T23:04:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.396+0000 7f7415108f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x555cf8980420 con 0x555cf8976bb0 2024-04-26T23:04:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.397+0000 7f7412400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80022a0 0x555cf897f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.397+0000 7f7412400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80022a0 0x555cf897f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47994/0 (socket says 172.21.15.40:47994) 2024-04-26T23:04:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.397+0000 7f7412400640 1 -- 172.21.15.40:0/3618130575 learned_addr learned my addr 172.21.15.40:0/3618130575 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.397+0000 7f7412400640 1 -- 172.21.15.40:0/3618130575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555cf8976bb0 msgr2=0x555cf88d98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.397+0000 7f7411a00640 1 --2- 172.21.15.40:0/3618130575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555cf8976bb0 0x555cf88d98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.397+0000 7f7412400640 1 --2- 172.21.15.40:0/3618130575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555cf8976bb0 0x555cf88d98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.397+0000 7f7412e00640 1 --2- 172.21.15.40:0/3618130575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cf88d9e10 0x555cf897ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.397+0000 7f7412400640 1 -- 172.21.15.40:0/3618130575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cf88d9e10 msgr2=0x555cf897ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.397+0000 7f7412400640 1 --2- 172.21.15.40:0/3618130575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cf88d9e10 0x555cf897ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.397+0000 7f7412400640 1 -- 172.21.15.40:0/3618130575 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cf88f3d40 con 0x7f73f80022a0 2024-04-26T23:04:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.398+0000 7f7412400640 1 --2- 172.21.15.40:0/3618130575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80022a0 0x555cf897f100 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f7404057040 tx=0x7f7404050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff7de4af46a087e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.398+0000 7f740be00640 1 -- 172.21.15.40:0/3618130575 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404061450 con 0x7f73f80022a0 2024-04-26T23:04:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.398+0000 7f740be00640 1 -- 172.21.15.40:0/3618130575 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7404066070 con 0x7f73f80022a0 2024-04-26T23:04:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.398+0000 7f740be00640 1 -- 172.21.15.40:0/3618130575 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404061810 con 0x7f73f80022a0 2024-04-26T23:04:47.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.398+0000 7f7415108f40 1 -- 172.21.15.40:0/3618130575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80022a0 msgr2=0x555cf897f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.398+0000 7f7415108f40 1 --2- 172.21.15.40:0/3618130575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80022a0 0x555cf897f100 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f7404057040 tx=0x7f7404050000 comp rx=0 tx=0).stop 2024-04-26T23:04:47.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.398+0000 7f7415108f40 1 -- 172.21.15.40:0/3618130575 shutdown_connections 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.398+0000 7f7415108f40 1 --2- 172.21.15.40:0/3618130575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cf88d9e10 0x555cf897ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.398+0000 7f7415108f40 1 --2- 172.21.15.40:0/3618130575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555cf8976bb0 0x555cf88d98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.398+0000 7f7415108f40 1 --2- 172.21.15.40:0/3618130575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73f80022a0 0x555cf897f100 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.398+0000 7f7415108f40 1 -- 172.21.15.40:0/3618130575 >> 172.21.15.40:0/3618130575 conn(0x555cf88d7c00 msgr2=0x555cf88d6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.399+0000 7f7415108f40 1 -- 172.21.15.40:0/3618130575 shutdown_connections 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.399+0000 7f7415108f40 1 -- 172.21.15.40:0/3618130575 wait complete. 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.399+0000 7f7415108f40 1 Processor -- start 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.399+0000 7f7415108f40 1 -- start start 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.400+0000 7f7415108f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cf88d9e10 0x555cf897f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.400+0000 7f7415108f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555cf8976bb0 0x555cf88f06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.400+0000 7f7415108f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73f80022a0 0x555cf8974600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.400+0000 7f7415108f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555cf897f640 con 0x555cf88d9e10 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.400+0000 7f7415108f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555cf897d880 con 0x7f73f80022a0 2024-04-26T23:04:47.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.400+0000 7f7415108f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x555cf89770c0 con 0x555cf8976bb0 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.401+0000 7f7412400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cf88d9e10 0x555cf897f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.401+0000 7f7412400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cf88d9e10 0x555cf897f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48006/0 (socket says 172.21.15.40:48006) 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.401+0000 7f7412400640 1 -- 172.21.15.40:0/3387308815 learned_addr learned my addr 172.21.15.40:0/3387308815 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.401+0000 7f7412e00640 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73f80022a0 0x555cf8974600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.401+0000 7f7411a00640 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555cf8976bb0 0x555cf88f06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.401+0000 7f7412400640 1 -- 172.21.15.40:0/3387308815 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555cf8976bb0 msgr2=0x555cf88f06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.401+0000 7f7412400640 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555cf8976bb0 0x555cf88f06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.401+0000 7f7412400640 1 -- 172.21.15.40:0/3387308815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73f80022a0 msgr2=0x555cf8974600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.401+0000 7f7412400640 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73f80022a0 0x555cf8974600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.401+0000 7f7412400640 1 -- 172.21.15.40:0/3387308815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cf88d9a20 con 0x555cf88d9e10 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.401+0000 7f7412400640 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cf88d9e10 0x555cf897f0e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f74040263c0 tx=0x7f7404001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.402+0000 7f740aa00640 1 -- 172.21.15.40:0/3387308815 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404061790 con 0x555cf88d9e10 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.402+0000 7f740aa00640 1 -- 172.21.15.40:0/3387308815 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7404075070 con 0x555cf88d9e10 2024-04-26T23:04:47.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.402+0000 7f7415108f40 1 -- 172.21.15.40:0/3387308815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555cf897b630 con 0x555cf88d9e10 2024-04-26T23:04:47.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.402+0000 7f740aa00640 1 -- 172.21.15.40:0/3387308815 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404061b70 con 0x555cf88d9e10 2024-04-26T23:04:47.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.402+0000 7f7415108f40 1 -- 172.21.15.40:0/3387308815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cf897bbf0 con 0x555cf88d9e10 2024-04-26T23:04:47.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.404+0000 7f740aa00640 1 -- 172.21.15.40:0/3387308815 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f740405e050 con 0x555cf88d9e10 2024-04-26T23:04:47.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.405+0000 7f740aa00640 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f73f406b300 0x7f73f406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.405+0000 7f740aa00640 1 -- 172.21.15.40:0/3387308815 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6159+0+0 (secure 0 0 0) 0x7f74040d0db0 con 0x555cf88d9e10 2024-04-26T23:04:47.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.405+0000 7f7415108f40 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x555cf88ed900 0x555cf88edd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.405+0000 7f7415108f40 1 -- 172.21.15.40:0/3387308815 --> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] -- osd_op(unknown.0.0:1 7.9 7:9258b67d:::10000000005.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) -- 0x555cf88ee250 con 0x555cf88ed900 2024-04-26T23:04:47.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.405+0000 7f7412e00640 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x555cf88ed900 0x555cf88edd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.406+0000 7f7411a00640 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f73f406b300 0x7f73f406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.406+0000 7f7412e00640 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x555cf88ed900 0x555cf88edd10 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.407+0000 7f7411a00640 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f73f406b300 0x7f73f406d7c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7400000c00 tx=0x7f7400002410 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.408+0000 7f7412e00640 1 -- 172.21.15.40:0/3387308815 <== osd.0 v2:172.21.15.40:6800/2897145808 1 ==== osd_op_reply(1 10000000005.00000000 [rmxattr] v53'6 uv6 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f740c011040 con 0x555cf88ed900 2024-04-26T23:04:47.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.408+0000 7f7415108f40 1 -- 172.21.15.40:0/3387308815 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x555cf88ed900 msgr2=0x555cf88edd10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.408+0000 7f7415108f40 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x555cf88ed900 0x555cf88edd10 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.408+0000 7f7415108f40 1 -- 172.21.15.40:0/3387308815 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f73f406b300 msgr2=0x7f73f406d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.408+0000 7f7415108f40 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f73f406b300 0x7f73f406d7c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7400000c00 tx=0x7f7400002410 comp rx=0 tx=0).stop 2024-04-26T23:04:47.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.408+0000 7f7415108f40 1 -- 172.21.15.40:0/3387308815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cf88d9e10 msgr2=0x555cf897f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.408+0000 7f7415108f40 1 --2- 172.21.15.40:0/3387308815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cf88d9e10 0x555cf897f0e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f74040263c0 tx=0x7f7404001a40 comp rx=0 tx=0).stop 2024-04-26T23:04:47.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.408+0000 7f7412e00640 1 -- 172.21.15.40:0/3387308815 reap_dead start 2024-04-26T23:04:47.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.408+0000 7f7415108f40 1 -- 172.21.15.40:0/3387308815 shutdown_connections 2024-04-26T23:04:47.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.408+0000 7f7415108f40 1 -- 172.21.15.40:0/3387308815 >> 172.21.15.40:0/3387308815 conn(0x555cf88d7c00 msgr2=0x555cf88ab420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.408+0000 7f7415108f40 1 -- 172.21.15.40:0/3387308815 shutdown_connections 2024-04-26T23:04:47.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.408+0000 7f7415108f40 1 -- 172.21.15.40:0/3387308815 wait complete. 2024-04-26T23:04:47.413 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000006.00000000 parent 2024-04-26T23:04:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.496+0000 7fd164928f40 1 Processor -- start 2024-04-26T23:04:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.497+0000 7fd164928f40 1 -- start start 2024-04-26T23:04:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.498+0000 7fd164928f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b382d9be90 0x55b382e404c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.498+0000 7fd164928f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b382d9d070 0x55b382e3ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.498+0000 7fd164928f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b382db43d0 0x55b382e3e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.498+0000 7fd164928f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b382e42030 con 0x55b382d9d070 2024-04-26T23:04:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.498+0000 7fd164928f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b382d11d20 con 0x55b382d9be90 2024-04-26T23:04:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.498+0000 7fd164928f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b382e42310 con 0x55b382db43d0 2024-04-26T23:04:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.498+0000 7fd15be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b382d9d070 0x55b382e3ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.498+0000 7fd15be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b382d9d070 0x55b382e3ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48034/0 (socket says 172.21.15.40:48034) 2024-04-26T23:04:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.498+0000 7fd15be00640 1 -- 172.21.15.40:0/622079100 learned_addr learned my addr 172.21.15.40:0/622079100 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.499+0000 7fd161c00640 1 --2- 172.21.15.40:0/622079100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b382d9be90 0x55b382e404c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.499+0000 7fd15be00640 1 -- 172.21.15.40:0/622079100 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b382db43d0 msgr2=0x55b382e3e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.499+0000 7fd15be00640 1 --2- 172.21.15.40:0/622079100 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b382db43d0 0x55b382e3e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.499+0000 7fd15be00640 1 -- 172.21.15.40:0/622079100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b382d9be90 msgr2=0x55b382e404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.499+0000 7fd15be00640 1 --2- 172.21.15.40:0/622079100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b382d9be90 0x55b382e404c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.499+0000 7fd15be00640 1 -- 172.21.15.40:0/622079100 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b382d6a930 con 0x55b382d9d070 2024-04-26T23:04:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.499+0000 7fd15be00640 1 --2- 172.21.15.40:0/622079100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b382d9d070 0x55b382e3ee80 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd14c025bc0 tx=0x7fd14c04e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5fd7c0c91ff6d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.499+0000 7fd15b400640 1 -- 172.21.15.40:0/622079100 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c059040 con 0x55b382d9d070 2024-04-26T23:04:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.499+0000 7fd15b400640 1 -- 172.21.15.40:0/622079100 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd14c064070 con 0x55b382d9d070 2024-04-26T23:04:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.499+0000 7fd15b400640 1 -- 172.21.15.40:0/622079100 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c054bf0 con 0x55b382d9d070 2024-04-26T23:04:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.500+0000 7fd164928f40 1 -- 172.21.15.40:0/622079100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b382d9d070 msgr2=0x55b382e3ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.500+0000 7fd164928f40 1 --2- 172.21.15.40:0/622079100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b382d9d070 0x55b382e3ee80 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd14c025bc0 tx=0x7fd14c04e000 comp rx=0 tx=0).stop 2024-04-26T23:04:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.500+0000 7fd164928f40 1 -- 172.21.15.40:0/622079100 shutdown_connections 2024-04-26T23:04:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.500+0000 7fd164928f40 1 --2- 172.21.15.40:0/622079100 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b382db43d0 0x55b382e3e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.500+0000 7fd164928f40 1 --2- 172.21.15.40:0/622079100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b382d9d070 0x55b382e3ee80 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.500+0000 7fd164928f40 1 --2- 172.21.15.40:0/622079100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b382d9be90 0x55b382e404c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.500+0000 7fd164928f40 1 -- 172.21.15.40:0/622079100 >> 172.21.15.40:0/622079100 conn(0x55b382db7df0 msgr2=0x55b382d9b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.500+0000 7fd164928f40 1 -- 172.21.15.40:0/622079100 shutdown_connections 2024-04-26T23:04:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.500+0000 7fd164928f40 1 -- 172.21.15.40:0/622079100 wait complete. 2024-04-26T23:04:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.501+0000 7fd164928f40 1 Processor -- start 2024-04-26T23:04:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.502+0000 7fd164928f40 1 -- start start 2024-04-26T23:04:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.503+0000 7fd164928f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b382d9be90 0x55b382e3af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.503+0000 7fd164928f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b382d9d070 0x55b382e38230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.503+0000 7fd164928f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b382db43d0 0x55b382e36c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.503+0000 7fd164928f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b382e41200 con 0x55b382d9d070 2024-04-26T23:04:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.503+0000 7fd164928f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b382e41450 con 0x55b382db43d0 2024-04-26T23:04:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.503+0000 7fd164928f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b382e3f3c0 con 0x55b382d9be90 2024-04-26T23:04:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.503+0000 7fd15be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b382d9d070 0x55b382e38230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.503+0000 7fd15be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b382d9d070 0x55b382e38230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48050/0 (socket says 172.21.15.40:48050) 2024-04-26T23:04:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.503+0000 7fd15be00640 1 -- 172.21.15.40:0/2738225596 learned_addr learned my addr 172.21.15.40:0/2738225596 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.504+0000 7fd161c00640 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b382d9be90 0x55b382e3af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.504+0000 7fd15be00640 1 -- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b382d9be90 msgr2=0x55b382e3af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.504+0000 7fd15be00640 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b382d9be90 0x55b382e3af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.504+0000 7fd15be00640 1 -- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b382db43d0 msgr2=0x55b382e36c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.504+0000 7fd162600640 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b382db43d0 0x55b382e36c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.504+0000 7fd15be00640 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b382db43d0 0x55b382e36c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.504+0000 7fd15be00640 1 -- 172.21.15.40:0/2738225596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b382e3fc90 con 0x55b382d9d070 2024-04-26T23:04:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.504+0000 7fd161c00640 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b382d9be90 0x55b382e3af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.504+0000 7fd162600640 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b382db43d0 0x55b382e36c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.504+0000 7fd15be00640 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b382d9d070 0x55b382e38230 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fd14c061040 tx=0x7fd14c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.505+0000 7fd15a000640 1 -- 172.21.15.40:0/2738225596 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c06e070 con 0x55b382d9d070 2024-04-26T23:04:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.505+0000 7fd15a000640 1 -- 172.21.15.40:0/2738225596 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd14c054da0 con 0x55b382d9d070 2024-04-26T23:04:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.505+0000 7fd15a000640 1 -- 172.21.15.40:0/2738225596 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c059050 con 0x55b382d9d070 2024-04-26T23:04:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.505+0000 7fd164928f40 1 -- 172.21.15.40:0/2738225596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b382e375f0 con 0x55b382d9d070 2024-04-26T23:04:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.505+0000 7fd164928f40 1 -- 172.21.15.40:0/2738225596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b382e3e870 con 0x55b382d9d070 2024-04-26T23:04:47.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.508+0000 7fd15a000640 1 -- 172.21.15.40:0/2738225596 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd14c05f6b0 con 0x55b382d9d070 2024-04-26T23:04:47.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.509+0000 7fd15a000640 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd14006b2d0 0x7fd14006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.509+0000 7fd15a000640 1 -- 172.21.15.40:0/2738225596 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6159+0+0 (secure 0 0 0) 0x7fd14c0cf420 con 0x55b382d9d070 2024-04-26T23:04:47.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.509+0000 7fd164928f40 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x55b382ed04c0 0x55b382ed08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.509+0000 7fd164928f40 1 -- 172.21.15.40:0/2738225596 --> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] -- osd_op(unknown.0.0:1 7.5 7:a5feb6f3:::10000000006.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) -- 0x55b382ed0e10 con 0x55b382ed04c0 2024-04-26T23:04:47.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.509+0000 7fd161c00640 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd14006b2d0 0x7fd14006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.510+0000 7fd162600640 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x55b382ed04c0 0x55b382ed08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.510+0000 7fd161c00640 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd14006b2d0 0x7fd14006d790 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd154005610 tx=0x7fd154005560 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.511+0000 7fd162600640 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x55b382ed04c0 0x55b382ed08d0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.514+0000 7fd162600640 1 -- 172.21.15.40:0/2738225596 <== osd.7 v2:172.21.15.103:6819/3769540391 1 ==== osd_op_reply(1 10000000006.00000000 [rmxattr] v53'7 uv7 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd15c001750 con 0x55b382ed04c0 2024-04-26T23:04:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.514+0000 7fd164928f40 1 -- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x55b382ed04c0 msgr2=0x55b382ed08d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.514+0000 7fd164928f40 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x55b382ed04c0 0x55b382ed08d0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.514+0000 7fd164928f40 1 -- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd14006b2d0 msgr2=0x7fd14006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.514+0000 7fd164928f40 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd14006b2d0 0x7fd14006d790 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd154005610 tx=0x7fd154005560 comp rx=0 tx=0).stop 2024-04-26T23:04:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.514+0000 7fd164928f40 1 -- 172.21.15.40:0/2738225596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b382d9d070 msgr2=0x55b382e38230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.514+0000 7fd164928f40 1 --2- 172.21.15.40:0/2738225596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b382d9d070 0x55b382e38230 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fd14c061040 tx=0x7fd14c002fb0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.514+0000 7fd162600640 1 -- 172.21.15.40:0/2738225596 reap_dead start 2024-04-26T23:04:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.514+0000 7fd164928f40 1 -- 172.21.15.40:0/2738225596 shutdown_connections 2024-04-26T23:04:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.514+0000 7fd164928f40 1 -- 172.21.15.40:0/2738225596 >> 172.21.15.40:0/2738225596 conn(0x55b382db7df0 msgr2=0x55b382e40290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.515+0000 7fd164928f40 1 -- 172.21.15.40:0/2738225596 shutdown_connections 2024-04-26T23:04:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.515+0000 7fd164928f40 1 -- 172.21.15.40:0/2738225596 wait complete. 2024-04-26T23:04:47.520 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000007.00000000 parent 2024-04-26T23:04:47.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.625+0000 7f1a6b830f40 1 Processor -- start 2024-04-26T23:04:47.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.625+0000 7f1a6b830f40 1 -- start start 2024-04-26T23:04:47.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.626+0000 7f1a6b830f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648f2f08e90 0x5648f2fb02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.626+0000 7f1a6b830f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648f2f0a070 0x5648f3040b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.626+0000 7f1a6b830f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5648f2f23590 0x5648f30410d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.626+0000 7f1a6b830f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5648f2f21ef0 con 0x5648f2f08e90 2024-04-26T23:04:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.626+0000 7f1a6b830f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5648f2e7ed20 con 0x5648f2f0a070 2024-04-26T23:04:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.626+0000 7f1a6b830f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5648f2faf030 con 0x5648f2f23590 2024-04-26T23:04:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.626+0000 7f1a68a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648f2f08e90 0x5648f2fb02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.626+0000 7f1a68a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648f2f08e90 0x5648f2fb02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48056/0 (socket says 172.21.15.40:48056) 2024-04-26T23:04:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.626+0000 7f1a68a00640 1 -- 172.21.15.40:0/994579444 learned_addr learned my addr 172.21.15.40:0/994579444 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.627+0000 7f1a63e00640 1 --2- 172.21.15.40:0/994579444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648f2f0a070 0x5648f3040b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.627+0000 7f1a69400640 1 --2- 172.21.15.40:0/994579444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5648f2f23590 0x5648f30410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.627+0000 7f1a68a00640 1 -- 172.21.15.40:0/994579444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5648f2f23590 msgr2=0x5648f30410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.627+0000 7f1a68a00640 1 --2- 172.21.15.40:0/994579444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5648f2f23590 0x5648f30410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.627+0000 7f1a68a00640 1 -- 172.21.15.40:0/994579444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648f2f0a070 msgr2=0x5648f3040b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.627+0000 7f1a68a00640 1 --2- 172.21.15.40:0/994579444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648f2f0a070 0x5648f3040b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.627+0000 7f1a68a00640 1 -- 172.21.15.40:0/994579444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648f2f04c10 con 0x5648f2f08e90 2024-04-26T23:04:47.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.627+0000 7f1a68a00640 1 --2- 172.21.15.40:0/994579444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648f2f08e90 0x5648f2fb02b0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009750 tx=0x7f1a5c005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd76f7c7c5321969 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.627+0000 7f1a62a00640 1 -- 172.21.15.40:0/994579444 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c019070 con 0x5648f2f08e90 2024-04-26T23:04:47.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.627+0000 7f1a62a00640 1 -- 172.21.15.40:0/994579444 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5c008890 con 0x5648f2f08e90 2024-04-26T23:04:47.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.627+0000 7f1a62a00640 1 -- 172.21.15.40:0/994579444 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c008c60 con 0x5648f2f08e90 2024-04-26T23:04:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.628+0000 7f1a6b830f40 1 -- 172.21.15.40:0/994579444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648f2f08e90 msgr2=0x5648f2fb02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.628+0000 7f1a6b830f40 1 --2- 172.21.15.40:0/994579444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648f2f08e90 0x5648f2fb02b0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009750 tx=0x7f1a5c005a00 comp rx=0 tx=0).stop 2024-04-26T23:04:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.628+0000 7f1a6b830f40 1 -- 172.21.15.40:0/994579444 shutdown_connections 2024-04-26T23:04:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.628+0000 7f1a6b830f40 1 --2- 172.21.15.40:0/994579444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5648f2f23590 0x5648f30410d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.628+0000 7f1a6b830f40 1 --2- 172.21.15.40:0/994579444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648f2f0a070 0x5648f3040b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.628+0000 7f1a6b830f40 1 --2- 172.21.15.40:0/994579444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648f2f08e90 0x5648f2fb02b0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.628+0000 7f1a6b830f40 1 -- 172.21.15.40:0/994579444 >> 172.21.15.40:0/994579444 conn(0x5648f2f24df0 msgr2=0x5648f2f21340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.628+0000 7f1a6b830f40 1 -- 172.21.15.40:0/994579444 shutdown_connections 2024-04-26T23:04:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.628+0000 7f1a6b830f40 1 -- 172.21.15.40:0/994579444 wait complete. 2024-04-26T23:04:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.629+0000 7f1a6b830f40 1 Processor -- start 2024-04-26T23:04:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.629+0000 7f1a6b830f40 1 -- start start 2024-04-26T23:04:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.630+0000 7f1a6b830f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648f2f08e90 0x5648f2ed7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.630+0000 7f1a6b830f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648f2f0a070 0x5648f3039ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.630+0000 7f1a6b830f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5648f2f23590 0x5648f303b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.630+0000 7f1a6b830f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5648f2f09c60 con 0x5648f2f08e90 2024-04-26T23:04:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.630+0000 7f1a6b830f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5648f2f09eb0 con 0x5648f2f0a070 2024-04-26T23:04:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.630+0000 7f1a6b830f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5648f2fb1920 con 0x5648f2f23590 2024-04-26T23:04:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.630+0000 7f1a68a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648f2f08e90 0x5648f2ed7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.630+0000 7f1a68a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648f2f08e90 0x5648f2ed7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48070/0 (socket says 172.21.15.40:48070) 2024-04-26T23:04:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.630+0000 7f1a68a00640 1 -- 172.21.15.40:0/221137304 learned_addr learned my addr 172.21.15.40:0/221137304 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.631+0000 7f1a63e00640 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648f2f0a070 0x5648f3039ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.631+0000 7f1a69400640 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5648f2f23590 0x5648f303b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.631+0000 7f1a68a00640 1 -- 172.21.15.40:0/221137304 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5648f2f23590 msgr2=0x5648f303b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.631+0000 7f1a68a00640 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5648f2f23590 0x5648f303b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.631+0000 7f1a68a00640 1 -- 172.21.15.40:0/221137304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648f2f0a070 msgr2=0x5648f3039ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.631+0000 7f1a68a00640 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648f2f0a070 0x5648f3039ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.631+0000 7f1a68a00640 1 -- 172.21.15.40:0/221137304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648f2fabc40 con 0x5648f2f08e90 2024-04-26T23:04:47.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.631+0000 7f1a63e00640 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648f2f0a070 0x5648f3039ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:47.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.631+0000 7f1a68a00640 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648f2f08e90 0x5648f2ed7b60 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c000c00 tx=0x7f1a5c003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.632+0000 7f1a61600640 1 -- 172.21.15.40:0/221137304 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c019040 con 0x5648f2f08e90 2024-04-26T23:04:47.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.632+0000 7f1a61600640 1 -- 172.21.15.40:0/221137304 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5c023070 con 0x5648f2f08e90 2024-04-26T23:04:47.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.632+0000 7f1a6b830f40 1 -- 172.21.15.40:0/221137304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5648f2fabee0 con 0x5648f2f08e90 2024-04-26T23:04:47.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.632+0000 7f1a61600640 1 -- 172.21.15.40:0/221137304 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c0146a0 con 0x5648f2f08e90 2024-04-26T23:04:47.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.632+0000 7f1a6b830f40 1 -- 172.21.15.40:0/221137304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648f2faa640 con 0x5648f2f08e90 2024-04-26T23:04:47.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.634+0000 7f1a61600640 1 -- 172.21.15.40:0/221137304 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1a5c014840 con 0x5648f2f08e90 2024-04-26T23:04:47.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.635+0000 7f1a61600640 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a4806b300 0x7f1a4806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.635+0000 7f1a61600640 1 -- 172.21.15.40:0/221137304 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6159+0+0 (secure 0 0 0) 0x7f1a5c08c0c0 con 0x5648f2f08e90 2024-04-26T23:04:47.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.635+0000 7f1a6b830f40 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5648f303c990 0x5648f2fb2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.635+0000 7f1a6b830f40 1 -- 172.21.15.40:0/221137304 --> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] -- osd_op(unknown.0.0:1 7.1d 7:bbf5850a:::10000000007.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) -- 0x5648f2faab60 con 0x5648f303c990 2024-04-26T23:04:47.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.635+0000 7f1a63e00640 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a4806b300 0x7f1a4806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.635+0000 7f1a69400640 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5648f303c990 0x5648f2fb2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.636+0000 7f1a69400640 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5648f303c990 0x5648f2fb2ca0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.638+0000 7f1a63e00640 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a4806b300 0x7f1a4806d7c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x5648f2ed63a0 tx=0x7f1a540025b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.640+0000 7f1a69400640 1 -- 172.21.15.40:0/221137304 <== osd.1 v2:172.21.15.40:6815/2323014490 1 ==== osd_op_reply(1 10000000007.00000000 [rmxattr] v53'7 uv7 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1a640518f0 con 0x5648f303c990 2024-04-26T23:04:47.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.640+0000 7f1a6b830f40 1 -- 172.21.15.40:0/221137304 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5648f303c990 msgr2=0x5648f2fb2ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.640+0000 7f1a6b830f40 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x5648f303c990 0x5648f2fb2ca0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.640+0000 7f1a6b830f40 1 -- 172.21.15.40:0/221137304 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a4806b300 msgr2=0x7f1a4806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.640+0000 7f1a6b830f40 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a4806b300 0x7f1a4806d7c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x5648f2ed63a0 tx=0x7f1a540025b0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.640+0000 7f1a6b830f40 1 -- 172.21.15.40:0/221137304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648f2f08e90 msgr2=0x5648f2ed7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.640+0000 7f1a6b830f40 1 --2- 172.21.15.40:0/221137304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648f2f08e90 0x5648f2ed7b60 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c000c00 tx=0x7f1a5c003020 comp rx=0 tx=0).stop 2024-04-26T23:04:47.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.640+0000 7f1a69400640 1 -- 172.21.15.40:0/221137304 reap_dead start 2024-04-26T23:04:47.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.640+0000 7f1a6b830f40 1 -- 172.21.15.40:0/221137304 shutdown_connections 2024-04-26T23:04:47.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.640+0000 7f1a6b830f40 1 -- 172.21.15.40:0/221137304 >> 172.21.15.40:0/221137304 conn(0x5648f2f24df0 msgr2=0x5648f2fa73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.640+0000 7f1a6b830f40 1 -- 172.21.15.40:0/221137304 shutdown_connections 2024-04-26T23:04:47.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.640+0000 7f1a6b830f40 1 -- 172.21.15.40:0/221137304 wait complete. 2024-04-26T23:04:47.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:47.645 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub start /d1/d2/d3 recursive,force 2024-04-26T23:04:47.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.835+0000 7f98ee600640 1 Processor -- start 2024-04-26T23:04:47.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.835+0000 7f98ee600640 1 -- start start 2024-04-26T23:04:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.837+0000 7f98ee600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f98e80d3aa0 0x7f98e80b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.837+0000 7f98ee600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98e8163ab0 0x7f98e80b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.837+0000 7f98ee600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e80b7880 con 0x7f98e80b6b40 2024-04-26T23:04:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.837+0000 7f98ee600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f98e80d0bb0 con 0x7f98e80d3aa0 2024-04-26T23:04:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.837+0000 7f98ee600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f98e80d05a0 con 0x7f98e8163ab0 2024-04-26T23:04:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.838+0000 7f98e7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f98e80d3aa0 0x7f98e80b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.838+0000 7f98e7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f98e80d3aa0 0x7f98e80b6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48082/0 (socket says 172.21.15.40:48082) 2024-04-26T23:04:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.838+0000 7f98ed200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f98e80b6b40 0x7f98e81633a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35788/0 (socket says 172.21.15.40:35788) 2024-04-26T23:04:47.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.838+0000 7f98e7e00640 1 -- 172.21.15.40:0/1563008004 learned_addr learned my addr 172.21.15.40:0/1563008004 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.838+0000 7f98edc00640 1 --2- 172.21.15.40:0/1563008004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98e8163ab0 0x7f98e80b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.838+0000 7f98e7e00640 1 -- 172.21.15.40:0/1563008004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98e8163ab0 msgr2=0x7f98e80b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.838+0000 7f98e7e00640 1 --2- 172.21.15.40:0/1563008004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98e8163ab0 0x7f98e80b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.838+0000 7f98e7e00640 1 -- 172.21.15.40:0/1563008004 >> v1:172.21.15.40:6789/0 conn(0x7f98e80b6b40 legacy=0x7f98e81633a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.838+0000 7f98e7e00640 1 -- 172.21.15.40:0/1563008004 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e80b75a0 con 0x7f98e80d3aa0 2024-04-26T23:04:47.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.838+0000 7f98edc00640 1 --2- 172.21.15.40:0/1563008004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98e8163ab0 0x7f98e80b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:47.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.840+0000 7f98e7e00640 1 --2- 172.21.15.40:0/1563008004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98e80d3aa0 0x7f98e80b6f20 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f98d4009870 tx=0x7f98d402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6983ebbe01c61366 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.840+0000 7f98e7400640 1 -- 172.21.15.40:0/1563008004 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d4037070 con 0x7f98e80d3aa0 2024-04-26T23:04:47.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.840+0000 7f98e7400640 1 -- 172.21.15.40:0/1563008004 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98d4032440 con 0x7f98e80d3aa0 2024-04-26T23:04:47.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.840+0000 7f98e7400640 1 -- 172.21.15.40:0/1563008004 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d4032810 con 0x7f98e80d3aa0 2024-04-26T23:04:47.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.841+0000 7f98ee600640 1 -- 172.21.15.40:0/1563008004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98e80d3aa0 msgr2=0x7f98e80b6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.841+0000 7f98ee600640 1 --2- 172.21.15.40:0/1563008004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98e80d3aa0 0x7f98e80b6f20 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f98d4009870 tx=0x7f98d402a1e0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.841+0000 7f98ee600640 1 -- 172.21.15.40:0/1563008004 shutdown_connections 2024-04-26T23:04:47.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.841+0000 7f98ee600640 1 --2- 172.21.15.40:0/1563008004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98e8163ab0 0x7f98e80b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.841+0000 7f98ee600640 1 --2- 172.21.15.40:0/1563008004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98e80d3aa0 0x7f98e80b6f20 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.841+0000 7f98ee600640 1 -- 172.21.15.40:0/1563008004 >> 172.21.15.40:0/1563008004 conn(0x7f98e80d4e50 msgr2=0x7f98e80d5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.841+0000 7f98ee600640 1 -- 172.21.15.40:0/1563008004 shutdown_connections 2024-04-26T23:04:47.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.841+0000 7f98ee600640 1 -- 172.21.15.40:0/1563008004 wait complete. 2024-04-26T23:04:47.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.844+0000 7f98ee600640 1 Processor -- start 2024-04-26T23:04:47.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.844+0000 7f98ee600640 1 -- start start 2024-04-26T23:04:47.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.845+0000 7f98ee600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98e80b6b40 0x7f98e80cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.845+0000 7f98ee600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98e80d3aa0 0x7f98e81542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.845+0000 7f98ee600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98e8163ab0 0x7f98e80b9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.845+0000 7f98ee600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f98e80b6300 con 0x7f98e80b6b40 2024-04-26T23:04:47.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.845+0000 7f98ee600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f98e80b6180 con 0x7f98e80d3aa0 2024-04-26T23:04:47.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.845+0000 7f98ee600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f98e8163f60 con 0x7f98e8163ab0 2024-04-26T23:04:47.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.846+0000 7f98ed200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98e80b6b40 0x7f98e80cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.846+0000 7f98ed200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98e80b6b40 0x7f98e80cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48086/0 (socket says 172.21.15.40:48086) 2024-04-26T23:04:47.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.846+0000 7f98ed200640 1 -- 172.21.15.40:0/757374916 learned_addr learned my addr 172.21.15.40:0/757374916 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.846+0000 7f98edc00640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98e8163ab0 0x7f98e80b9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.846+0000 7f98ed200640 1 -- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98e8163ab0 msgr2=0x7f98e80b9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.846+0000 7f98ed200640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98e8163ab0 0x7f98e80b9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.846+0000 7f98e7e00640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98e80d3aa0 0x7f98e81542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.846+0000 7f98ed200640 1 -- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98e80d3aa0 msgr2=0x7f98e81542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.846+0000 7f98ed200640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98e80d3aa0 0x7f98e81542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.846+0000 7f98ed200640 1 -- 172.21.15.40:0/757374916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e80b85b0 con 0x7f98e80b6b40 2024-04-26T23:04:47.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.846+0000 7f98edc00640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98e8163ab0 0x7f98e80b9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:47.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.846+0000 7f98e7e00640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98e80d3aa0 0x7f98e81542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:47.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.847+0000 7f98ed200640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98e80b6b40 0x7f98e80cb470 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00c430 tx=0x7f98dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.847+0000 7f98e4c00640 1 -- 172.21.15.40:0/757374916 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc0074e0 con 0x7f98e80b6b40 2024-04-26T23:04:47.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.847+0000 7f98e4c00640 1 -- 172.21.15.40:0/757374916 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98dc007680 con 0x7f98e80b6b40 2024-04-26T23:04:47.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.847+0000 7f98e4c00640 1 -- 172.21.15.40:0/757374916 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc004a00 con 0x7f98e80b6b40 2024-04-26T23:04:47.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.847+0000 7f98ee600640 1 -- 172.21.15.40:0/757374916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e80b8890 con 0x7f98e80b6b40 2024-04-26T23:04:47.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.847+0000 7f98ee600640 1 -- 172.21.15.40:0/757374916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e80b8e50 con 0x7f98e80b6b40 2024-04-26T23:04:47.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.850+0000 7f98e4c00640 1 -- 172.21.15.40:0/757374916 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f98dc01a020 con 0x7f98e80b6b40 2024-04-26T23:04:47.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.851+0000 7f98e4c00640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f98c806c8f0 0x7f98c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.851+0000 7f98e4c00640 1 -- 172.21.15.40:0/757374916 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6159+0+0 (secure 0 0 0) 0x7f98dc08b550 con 0x7f98e80b6b40 2024-04-26T23:04:47.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.852+0000 7f98f29f6740 1 Processor -- start 2024-04-26T23:04:47.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.852+0000 7f98f29f6740 1 -- start start 2024-04-26T23:04:47.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.852+0000 7f98e7e00640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f98c806c8f0 0x7f98c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98f29f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9922010 0x55f4a98d49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98f29f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 0x55f4a99044b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98f29f6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a99049f0 0x55f4a98d9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98f29f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f4a983eb80 con 0x55f4a9922010 2024-04-26T23:04:47.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98f29f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f4a97db420 con 0x55f4a9905130 2024-04-26T23:04:47.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98f29f6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55f4a983ab40 con 0x55f4a99049f0 2024-04-26T23:04:47.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98ed200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9922010 0x55f4a98d49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98ed200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9922010 0x55f4a98d49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48090/0 (socket says 172.21.15.40:48090) 2024-04-26T23:04:47.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98ed200640 1 -- 172.21.15.40:0/3781653403 learned_addr learned my addr 172.21.15.40:0/3781653403 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98e7e00640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f98c806c8f0 0x7f98c806edb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f98d4034000 tx=0x7f98d402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98ed200640 1 -- 172.21.15.40:0/3781653403 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a99049f0 msgr2=0x55f4a98d9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98ed200640 1 --2- 172.21.15.40:0/3781653403 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a99049f0 0x55f4a98d9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98ed200640 1 -- 172.21.15.40:0/3781653403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 msgr2=0x55f4a99044b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98e7e00640 1 --2- 172.21.15.40:0/3781653403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 0x55f4a99044b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98ed200640 1 --2- 172.21.15.40:0/3781653403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 0x55f4a99044b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.853+0000 7f98ed200640 1 -- 172.21.15.40:0/3781653403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4a9618a50 con 0x55f4a9922010 2024-04-26T23:04:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.854+0000 7f98ed200640 1 --2- 172.21.15.40:0/3781653403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9922010 0x55f4a98d49a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f98dc059130 tx=0x7f98dc059220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd4c69f4127d884 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.854+0000 7f98d9600640 1 -- 172.21.15.40:0/3781653403 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc0573b0 con 0x55f4a9922010 2024-04-26T23:04:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.854+0000 7f98d9600640 1 -- 172.21.15.40:0/3781653403 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98dc019da0 con 0x55f4a9922010 2024-04-26T23:04:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.854+0000 7f98d9600640 1 -- 172.21.15.40:0/3781653403 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc05d080 con 0x55f4a9922010 2024-04-26T23:04:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.855+0000 7f98f29f6740 1 -- 172.21.15.40:0/3781653403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9922010 msgr2=0x55f4a98d49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.855+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3781653403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9922010 0x55f4a98d49a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f98dc059130 tx=0x7f98dc059220 comp rx=0 tx=0).stop 2024-04-26T23:04:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.855+0000 7f98f29f6740 1 -- 172.21.15.40:0/3781653403 shutdown_connections 2024-04-26T23:04:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.855+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3781653403 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a99049f0 0x55f4a98d9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.855+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3781653403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 0x55f4a99044b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.855+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3781653403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9922010 0x55f4a98d49a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.855+0000 7f98f29f6740 1 -- 172.21.15.40:0/3781653403 >> 172.21.15.40:0/3781653403 conn(0x55f4a98d6c10 msgr2=0x55f4a991f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.855+0000 7f98f29f6740 1 -- 172.21.15.40:0/3781653403 shutdown_connections 2024-04-26T23:04:47.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.855+0000 7f98f29f6740 1 -- 172.21.15.40:0/3781653403 wait complete. 2024-04-26T23:04:47.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.855+0000 7f98f29f6740 1 -- start start 2024-04-26T23:04:47.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.856+0000 7f98f29f6740 1 Processor -- start 2024-04-26T23:04:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.857+0000 7f98f29f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 0x55f4a9907d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.857+0000 7f98f29f6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a9922010 0x55f4a99205e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.857+0000 7f98f29f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 0x55f4a9920b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.857+0000 7f98f29f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f4a98d8dc0 con 0x55f4a98ffbe0 2024-04-26T23:04:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.857+0000 7f98f29f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f4a97c6370 con 0x55f4a9905130 2024-04-26T23:04:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.857+0000 7f98f29f6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55f4a98d81b0 con 0x55f4a9922010 2024-04-26T23:04:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.857+0000 7f98f29f6740 10 client.0 _ll_register_callbacks cb 0x55f4a9835890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.857+0000 7f98edc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 0x55f4a9920b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.857+0000 7f98edc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 0x55f4a9920b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48106/0 (socket says 172.21.15.40:48106) 2024-04-26T23:04:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.857+0000 7f98edc00640 1 -- 172.21.15.40:0/1534423418 learned_addr learned my addr 172.21.15.40:0/1534423418 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.857+0000 7f98ed200640 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 0x55f4a9907d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.857+0000 7f98e7e00640 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a9922010 0x55f4a99205e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.858+0000 7f98edc00640 1 -- 172.21.15.40:0/1534423418 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a9922010 msgr2=0x55f4a99205e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.858+0000 7f98edc00640 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a9922010 0x55f4a99205e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.858+0000 7f98edc00640 1 -- 172.21.15.40:0/1534423418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 msgr2=0x55f4a9907d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.858+0000 7f98edc00640 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 0x55f4a9907d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.858+0000 7f98edc00640 1 -- 172.21.15.40:0/1534423418 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4a95103e0 con 0x55f4a98ffbe0 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.858+0000 7f98e7e00640 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a9922010 0x55f4a99205e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.858+0000 7f98ed200640 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 0x55f4a9907d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.858+0000 7f98edc00640 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 0x55f4a9920b20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f98e000da40 tx=0x7f98e000df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b586ae0064a140 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.858+0000 7f98ce000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.858+0000 7f98ce000640 1 -- 172.21.15.40:0/1534423418 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e000bd40 con 0x55f4a98ffbe0 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.858+0000 7f98ce000640 1 -- 172.21.15.40:0/1534423418 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(53..53 src has 1..53) ==== 6159+0+0 (secure 0 0 0) 0x7f98e000f450 con 0x55f4a98ffbe0 2024-04-26T23:04:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.858+0000 7f98f29f6740 1 -- 172.21.15.40:0/1534423418 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98dc019da0 con 0x55f4a98ffbe0 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.859+0000 7f98ce000640 1 -- 172.21.15.40:0/1534423418 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f98e0021050 con 0x55f4a98ffbe0 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.859+0000 7f98f29f6740 10 client.6399 fetch_fsmap learned FSMap version 43 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.859+0000 7f98f29f6740 1 -- 172.21.15.40:0/1534423418 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) -- 0x55f4a96791e0 con 0x55f4a98ffbe0 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.859+0000 7f98ce000640 1 -- 172.21.15.40:0/1534423418 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98dc019da0 con 0x55f4a98ffbe0 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.859+0000 7f98ce000640 1 -- 172.21.15.40:0/1534423418 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 43) ==== 3550+0+0 (secure 0 0 0) 0x7f98e0024020 con 0x55f4a98ffbe0 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.859+0000 7f98f29f6740 10 client.6399 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.859+0000 7f98f29f6740 10 client.6399 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.859+0000 7f98f29f6740 20 client.6399 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.859+0000 7f98f29f6740 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55f4a9930340 0x55f4a9930750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.859+0000 7f98f29f6740 4 client.6399 mds_command: new command op to 6036 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.859+0000 7f98f29f6740 1 -- 172.21.15.40:0/1534423418 --> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4a984c790 con 0x55f4a9930340 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.859+0000 7f98ed200640 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55f4a9930340 0x55f4a9930750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.860+0000 7f98ed200640 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55f4a9930340 0x55f4a9930750 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d22b61d6f3a7f09 server_cookie=69871bb9a43fa49c in_seq=0 out_seq=0 2024-04-26T23:04:47.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.860+0000 7f98ce000640 10 client.6399 ms_handle_connect on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98ce000640 1 -- 172.21.15.40:0/1534423418 <== mds.0 v2:172.21.15.40:6833/618018110 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f4a984c790 con 0x55f4a9930340 2024-04-26T23:04:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98ce000640 10 client.6399 handle_command_reply: tid=0 2024-04-26T23:04:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98f29f6740 1 client.6399 shutdown 2024-04-26T23:04:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98f29f6740 1 -- 172.21.15.40:0/1534423418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 msgr2=0x55f4a9920b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98f29f6740 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 0x55f4a9920b20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f98e000da40 tx=0x7f98e000df00 comp rx=0 tx=0).stop 2024-04-26T23:04:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98f29f6740 1 -- 172.21.15.40:0/1534423418 shutdown_connections 2024-04-26T23:04:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98f29f6740 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55f4a9930340 0x55f4a9930750 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98f29f6740 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 0x55f4a9920b20 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98f29f6740 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a9922010 0x55f4a99205e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98f29f6740 1 --2- 172.21.15.40:0/1534423418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 0x55f4a9907d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98f29f6740 1 -- 172.21.15.40:0/1534423418 >> 172.21.15.40:0/1534423418 conn(0x55f4a99049f0 msgr2=0x55f4a98d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98ce000640 0 client.6399 ms_handle_reset on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98f29f6740 1 -- 172.21.15.40:0/1534423418 shutdown_connections 2024-04-26T23:04:47.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98f29f6740 1 -- 172.21.15.40:0/1534423418 wait complete. 2024-04-26T23:04:47.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.861+0000 7f98f29f6740 20 client.6399 trim_cache size 0 max 16384 2024-04-26T23:04:47.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.881+0000 7f98f29f6740 1 Processor -- start 2024-04-26T23:04:47.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.881+0000 7f98f29f6740 1 -- start start 2024-04-26T23:04:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.882+0000 7f98f29f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 0x55f4a991e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.882+0000 7f98f29f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 0x55f4a99212f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.882+0000 7f98f29f6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a9922010 0x55f4a99228d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.882+0000 7f98f29f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f98e8201e70 con 0x55f4a98ffbe0 2024-04-26T23:04:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.882+0000 7f98f29f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f98e8201010 con 0x55f4a9905130 2024-04-26T23:04:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.882+0000 7f98f29f6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f98e8201740 con 0x55f4a9922010 2024-04-26T23:04:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.882+0000 7f98ed200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 0x55f4a991e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.882+0000 7f98ed200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 0x55f4a991e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48116/0 (socket says 172.21.15.40:48116) 2024-04-26T23:04:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.882+0000 7f98ed200640 1 -- 172.21.15.40:0/3356559123 learned_addr learned my addr 172.21.15.40:0/3356559123 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.882+0000 7f98e7e00640 1 --2- 172.21.15.40:0/3356559123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 0x55f4a99212f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.882+0000 7f98edc00640 1 --2- 172.21.15.40:0/3356559123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a9922010 0x55f4a99228d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.883+0000 7f98ed200640 1 -- 172.21.15.40:0/3356559123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a9922010 msgr2=0x55f4a99228d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.883+0000 7f98ed200640 1 --2- 172.21.15.40:0/3356559123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a9922010 0x55f4a99228d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.883+0000 7f98ed200640 1 -- 172.21.15.40:0/3356559123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 msgr2=0x55f4a99212f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.883+0000 7f98ed200640 1 --2- 172.21.15.40:0/3356559123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 0x55f4a99212f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.883+0000 7f98ed200640 1 -- 172.21.15.40:0/3356559123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4a94e94a0 con 0x55f4a98ffbe0 2024-04-26T23:04:47.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.883+0000 7f98ed200640 1 --2- 172.21.15.40:0/3356559123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 0x55f4a991e740 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f98dc059130 tx=0x7f98dc01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4e1ec254563e9c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.883+0000 7f98d8c00640 1 -- 172.21.15.40:0/3356559123 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc019da0 con 0x55f4a98ffbe0 2024-04-26T23:04:47.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.883+0000 7f98d8c00640 1 -- 172.21.15.40:0/3356559123 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98dc055720 con 0x55f4a98ffbe0 2024-04-26T23:04:47.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.883+0000 7f98d8c00640 1 -- 172.21.15.40:0/3356559123 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc08b0e0 con 0x55f4a98ffbe0 2024-04-26T23:04:47.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.884+0000 7f98f29f6740 1 -- 172.21.15.40:0/3356559123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 msgr2=0x55f4a991e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.884+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3356559123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 0x55f4a991e740 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f98dc059130 tx=0x7f98dc01d000 comp rx=0 tx=0).stop 2024-04-26T23:04:47.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.884+0000 7f98f29f6740 1 -- 172.21.15.40:0/3356559123 shutdown_connections 2024-04-26T23:04:47.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.884+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3356559123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a9922010 0x55f4a99228d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.884+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3356559123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9905130 0x55f4a99212f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.884+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3356559123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a98ffbe0 0x55f4a991e740 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.884+0000 7f98f29f6740 1 -- 172.21.15.40:0/3356559123 >> 172.21.15.40:0/3356559123 conn(0x55f4a99049f0 msgr2=0x55f4a9919540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.884+0000 7f98f29f6740 1 -- 172.21.15.40:0/3356559123 shutdown_connections 2024-04-26T23:04:47.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.884+0000 7f98f29f6740 1 -- 172.21.15.40:0/3356559123 wait complete. 2024-04-26T23:04:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.885+0000 7f98f29f6740 1 -- start start 2024-04-26T23:04:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.885+0000 7f98f29f6740 1 Processor -- start 2024-04-26T23:04:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.886+0000 7f98f29f6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a98ffbe0 0x55f4a999a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.886+0000 7f98f29f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9905130 0x55f4a991eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.886+0000 7f98f29f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9922010 0x55f4a9917210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.886+0000 7f98f29f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f4a9923980 con 0x55f4a9905130 2024-04-26T23:04:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.886+0000 7f98f29f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f98e81f98b0 con 0x55f4a9922010 2024-04-26T23:04:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.886+0000 7f98f29f6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55f4a9907630 con 0x55f4a98ffbe0 2024-04-26T23:04:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.886+0000 7f98f29f6740 10 client.0 _ll_register_callbacks cb 0x7f98e0007640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.886+0000 7f98e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9905130 0x55f4a991eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.886+0000 7f98e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9905130 0x55f4a991eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48118/0 (socket says 172.21.15.40:48118) 2024-04-26T23:04:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.886+0000 7f98e7e00640 1 -- 172.21.15.40:0/3677742247 learned_addr learned my addr 172.21.15.40:0/3677742247 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.886+0000 7f98edc00640 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9922010 0x55f4a9917210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.886+0000 7f98ed200640 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a98ffbe0 0x55f4a999a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.887+0000 7f98e7e00640 1 -- 172.21.15.40:0/3677742247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a98ffbe0 msgr2=0x55f4a999a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.887+0000 7f98e7e00640 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a98ffbe0 0x55f4a999a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.887+0000 7f98e7e00640 1 -- 172.21.15.40:0/3677742247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9922010 msgr2=0x55f4a9917210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.887+0000 7f98e7e00640 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9922010 0x55f4a9917210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.887+0000 7f98e7e00640 1 -- 172.21.15.40:0/3677742247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4a991f080 con 0x55f4a9905130 2024-04-26T23:04:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.887+0000 7f98edc00640 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9922010 0x55f4a9917210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.887+0000 7f98ed200640 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a98ffbe0 0x55f4a999a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.887+0000 7f98e7e00640 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9905130 0x55f4a991eb40 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f98d4009990 tx=0x7f98d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16f9d7967db74d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.887+0000 7f98ce000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.887+0000 7f98ce000640 1 -- 172.21.15.40:0/3677742247 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d4037040 con 0x55f4a9905130 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.887+0000 7f98ce000640 1 -- 172.21.15.40:0/3677742247 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(53..53 src has 1..53) ==== 6159+0+0 (secure 0 0 0) 0x7f98d403cbb0 con 0x55f4a9905130 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.887+0000 7f98f29f6740 1 -- 172.21.15.40:0/3677742247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98dc055720 con 0x55f4a9905130 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.888+0000 7f98ce000640 1 -- 172.21.15.40:0/3677742247 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f98d403cdb0 con 0x55f4a9905130 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.888+0000 7f98f29f6740 10 client.6405 fetch_fsmap learned FSMap version 43 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.888+0000 7f98f29f6740 1 -- 172.21.15.40:0/3677742247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) -- 0x55f4a9917a40 con 0x55f4a9905130 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.888+0000 7f98ce000640 1 -- 172.21.15.40:0/3677742247 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98dc055720 con 0x55f4a9905130 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.888+0000 7f98ce000640 1 -- 172.21.15.40:0/3677742247 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 43) ==== 3550+0+0 (secure 0 0 0) 0x7f98d4046070 con 0x55f4a9905130 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.888+0000 7f98f29f6740 10 client.6405 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.888+0000 7f98f29f6740 10 client.6405 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.888+0000 7f98f29f6740 20 client.6405 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.888+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55f4a9939b90 0x55f4a9996910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.888+0000 7f98f29f6740 4 client.6405 mds_command: new command op to 6036 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/d1/d2/d3", "scrubops": ["recursive", "force"]}] 2024-04-26T23:04:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.888+0000 7f98f29f6740 1 -- 172.21.15.40:0/3677742247 --> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] -- command(tid 0: {"prefix": "scrub start", "path": "/d1/d2/d3", "scrubops": ["recursive", "force"]}) -- 0x55f4a9846600 con 0x55f4a9939b90 2024-04-26T23:04:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.888+0000 7f98ed200640 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55f4a9939b90 0x55f4a9996910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.889+0000 7f98ed200640 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55f4a9939b90 0x55f4a9996910 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7003c4964eec8d5 server_cookie=dd4994cc9e6d5dbc in_seq=0 out_seq=0 2024-04-26T23:04:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.889+0000 7f98ce000640 10 client.6405 ms_handle_connect on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:47.893 INFO:tasks.ceph.mds.b.smithi103.stderr:2024-04-26T23:04:47.892+0000 7fec41e00640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000007 [...2,head] /d1/d2/d3/d4/d5/d6/d7/d8/ auth v2 ap=1 f() n(v0 1=0+1) (inest lock) (ifile excl) caps={5356=pAsLsXsFsx/AsLsXsFsx@2},l=5356 | importingcaps=0 dirfrag=0 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x55b6f108e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000007:[<0x10000000006/d8 v2>,<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 1=0+1)","error_str":""},"return_code":-61} 2024-04-26T23:04:47.893 INFO:tasks.ceph.mds.b.smithi103.stderr:2024-04-26T23:04:47.893+0000 7fec41e00640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000006 [...2,head] /d1/d2/d3/d4/d5/d6/d7/ auth v4 ap=1 f(v0 m2024-04-26T23:04:30.274604+0000 1=0+1) n(v0 rc2024-04-26T23:04:30.274604+0000 2=0+2) (inest lock) (ifile excl) caps={5356=pAsLsXsFsx/pAsLsXsFsxcral@4},l=5356 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x55b6f108e000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000006:[<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:04:30.274604+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:04:30.274604+0000 2=0+2)","memory_value.dirstat":"f(v0 m2024-04-26T23:04:30.274604+0000 1=0+1)","memory_value.rstat":"n(v0 rc2024-04-26T23:04:30.274604+0000 2=0+2)","error_str":""},"return_code":-61} 2024-04-26T23:04:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f98ce000640 1 -- 172.21.15.40:0/3677742247 <== mds.0 v2:172.21.15.40:6833/618018110 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f4a9846600 con 0x55f4a9939b90 2024-04-26T23:04:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f98ce000640 10 client.6405 handle_command_reply: tid=0 2024-04-26T23:04:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f98f29f6740 1 client.6405 shutdown 2024-04-26T23:04:47.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f98f29f6740 1 -- 172.21.15.40:0/3677742247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9905130 msgr2=0x55f4a991eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9905130 0x55f4a991eb40 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f98d4009990 tx=0x7f98d4004730 comp rx=0 tx=0).stop 2024-04-26T23:04:47.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f98f29f6740 1 -- 172.21.15.40:0/3677742247 shutdown_connections 2024-04-26T23:04:47.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x55f4a9939b90 0x55f4a9996910 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4a9922010 0x55f4a9917210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4a9905130 0x55f4a991eb40 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f98f29f6740 1 --2- 172.21.15.40:0/3677742247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4a98ffbe0 0x55f4a999a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f98f29f6740 1 -- 172.21.15.40:0/3677742247 >> 172.21.15.40:0/3677742247 conn(0x55f4a99049f0 msgr2=0x55f4a98fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f98ce000640 0 client.6405 ms_handle_reset on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:47.895 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f47d2c00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000002 [...2,head] /d1/d2/d3/ auth{1=1,2=1} v19 ap=1 f(v0 m2024-04-26T23:04:30.268604+0000 1=0+1) n(v3 rc2024-04-26T23:04:30.494601+0000 6=0+6) (inest mix dirty) (ifile excl) caps={5356=pAsLsXsFsx/pAsLsXsFsxcral@4},l=5356 | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 0x55e20976b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000002:[<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:04:30.268604+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:04:30.494601+0000 6=0+6)","memory_value.dirstat":"f(v0 m2024-04-26T23:04:30.268604+0000 1=0+1)","memory_value.rstat":"n(v3 rc2024-04-26T23:04:30.494601+0000 6=0+6)","error_str":""},"return_code":-61} 2024-04-26T23:04:47.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.890+0000 7f98f29f6740 1 -- 172.21.15.40:0/3677742247 shutdown_connections 2024-04-26T23:04:47.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.891+0000 7f98f29f6740 1 -- 172.21.15.40:0/3677742247 wait complete. 2024-04-26T23:04:47.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.891+0000 7f98f29f6740 20 client.6405 trim_cache size 0 max 16384 2024-04-26T23:04:47.896 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:04:47.896 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-04-26T23:04:47.896 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "b07480c4-ced7-4267-91ee-1b8bac05bf74", 2024-04-26T23:04:47.896 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-04-26T23:04:47.896 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:04:47.896 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-04-26T23:04:47.891+0000 7ff189400640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000004 [...2,head] /d1/d2/d3/d4/d5/ auth{0=1,2=1} v15 ap=1 f(v0 m2024-04-26T23:04:30.271604+0000 1=0+1) n(v2 rc2024-04-26T23:04:30.494601+0000 4=0+4) (inest lock) (ifile excl) caps={5356=pAsLsXsFsx/pAsLsXsFsxcral@4},l=5356 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 authpin=1 scrubqueue=0 0x561a6cc18b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000004:[<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:04:30.271604+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:04:30.494601+0000 4=0+4)","memory_value.dirstat":"f(v0 m2024-04-26T23:04:30.271604+0000 1=0+1)","memory_value.rstat":"n(v2 rc2024-04-26T23:04:30.494601+0000 4=0+4)","error_str":""},"return_code":-61} 2024-04-26T23:04:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.891+0000 7f98ee600640 1 -- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f98c806c8f0 msgr2=0x7f98c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.891+0000 7f98ee600640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f98c806c8f0 0x7f98c806edb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f98d4034000 tx=0x7f98d402e000 comp rx=0 tx=0).stop 2024-04-26T23:04:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.891+0000 7f98ee600640 1 -- 172.21.15.40:0/757374916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98e80b6b40 msgr2=0x7f98e80cb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.891+0000 7f98ee600640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98e80b6b40 0x7f98e80cb470 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00c430 tx=0x7f98dc00c8f0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.891+0000 7f98ee600640 1 -- 172.21.15.40:0/757374916 shutdown_connections 2024-04-26T23:04:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.891+0000 7f98ee600640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f98c806c8f0 0x7f98c806edb0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.891+0000 7f98ee600640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f98e8163ab0 0x7f98e80b9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.891+0000 7f98ee600640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98e80d3aa0 0x7f98e81542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.891+0000 7f98ee600640 1 --2- 172.21.15.40:0/757374916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98e80b6b40 0x7f98e80cb470 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:47.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.891+0000 7f98ee600640 1 -- 172.21.15.40:0/757374916 >> 172.21.15.40:0/757374916 conn(0x7f98e80d4e50 msgr2=0x7f98e80d5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:47.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.892+0000 7f98ee600640 1 -- 172.21.15.40:0/757374916 shutdown_connections 2024-04-26T23:04:47.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:47.892+0000 7f98ee600640 1 -- 172.21.15.40:0/757374916 wait complete. 2024-04-26T23:04:47.898 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:04:47.893+0000 7f47dae00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000003 [...2,head] /d1/d2/d3/d4/ auth{1=1,2=1} v19 ap=2 f(v0 m2024-04-26T23:04:30.269604+0000 1=0+1) n(v4 rc2024-04-26T23:04:30.494601+0000 5=0+5) (inest sync->mix g=1,2) caps={5356=pAsLsXsFs/pAsLsXsFsxcral@8} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 export_pin=1 0x55e20976ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000003:[<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:04:30.269604+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:04:30.494601+0000 5=0+5)","memory_value.dirstat":"f(v0 m2024-04-26T23:04:30.269604+0000 1=0+1)","memory_value.rstat":"n(v4 rc2024-04-26T23:04:30.494601+0000 5=0+5)","error_str":""},"return_code":-61} 2024-04-26T23:04:47.898 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-04-26T23:04:47.894+0000 7ff191600640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000005 [...2,head] /d1/d2/d3/d4/d5/d6/ auth{0=1,2=1} v11 ap=2 f(v0 m2024-04-26T23:04:30.272604+0000 1=0+1) n(v1 rc2024-04-26T23:04:30.494601+0000 3=0+3) (inest sync->mix g=0,2 scatter_wanted) caps={5356=pAsLsXsFs/pAsLsXsFsxcral@7} | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 export_pin=2 0x561a6cc18000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000005:[<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:04:30.272604+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:04:30.274604+0000 3=0+3)","memory_value.dirstat":"f(v0 m2024-04-26T23:04:30.272604+0000 1=0+1)","memory_value.rstat":"n(v1 rc2024-04-26T23:04:30.494601+0000 3=0+3)","error_str":""},"return_code":-61} 2024-04-26T23:04:47.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:04:47.911 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub abort 2024-04-26T23:04:47.946 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:04:47.961 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:04:48.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.115+0000 7fae9d400640 1 Processor -- start 2024-04-26T23:04:48.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.115+0000 7fae9d400640 1 -- start start 2024-04-26T23:04:48.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.117+0000 7fae9d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980d06f0 0x7fae980d1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.117+0000 7fae9d400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fae980b2850 0x7fae980b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.117+0000 7fae9d400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae980d0b70 con 0x7fae9815f3a0 2024-04-26T23:04:48.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.117+0000 7fae9d400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fae980d0e20 con 0x7fae980b2850 2024-04-26T23:04:48.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.117+0000 7fae9d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fae980b1d70 con 0x7fae980d06f0 2024-04-26T23:04:48.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.117+0000 7fae96800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fae9815f3a0 0x7fae980d1770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35794/0 (socket says 172.21.15.40:35794) 2024-04-26T23:04:48.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.117+0000 7fae96800640 1 -- 172.21.15.40:0/3114673618 learned_addr learned my addr 172.21.15.40:0/3114673618 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:48.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae8de00640 1 --2- 172.21.15.40:0/3114673618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980d06f0 0x7fae980d1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737769696 0 0) 0x7fae980d0b70 con 0x7fae9815f3a0 2024-04-26T23:04:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae97200640 1 --2- 172.21.15.40:0/3114673618 >> v2:172.21.15.103:3300/0 conn(0x7fae980b2850 0x7fae980b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae7c003520 con 0x7fae9815f3a0 2024-04-26T23:04:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2831196383 0 0) 0x7fae7c003520 con 0x7fae9815f3a0 2024-04-26T23:04:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980d06f0 msgr2=0x7fae980d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 --2- 172.21.15.40:0/3114673618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980d06f0 0x7fae980d1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 >> v2:172.21.15.103:3300/0 conn(0x7fae980b2850 msgr2=0x7fae980b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 --2- 172.21.15.40:0/3114673618 >> v2:172.21.15.103:3300/0 conn(0x7fae980b2850 0x7fae980b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae980b2050 con 0x7fae9815f3a0 2024-04-26T23:04:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fae80001670 con 0x7fae9815f3a0 2024-04-26T23:04:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 >> v1:172.21.15.40:6789/0 conn(0x7fae9815f3a0 legacy=0x7fae980d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 --2- 172.21.15.40:0/3114673618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae7c0039b0 0x7fae7c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 --2- 172.21.15.40:0/3114673618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980d06f0 0x7fae7c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 --2- 172.21.15.40:0/3114673618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae7c004d40 0x7fae7c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fae980d0e20 con 0x7fae7c004d40 2024-04-26T23:04:48.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fae980b1d70 con 0x7fae7c0039b0 2024-04-26T23:04:48.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fae7c00ba70 con 0x7fae980d06f0 2024-04-26T23:04:48.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.118+0000 7fae97200640 1 --2- 172.21.15.40:0/3114673618 >> v2:172.21.15.103:3300/0 conn(0x7fae980b2850 0x7fae980b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:04:48.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.119+0000 7fae97200640 1 --2- 172.21.15.40:0/3114673618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae7c004d40 0x7fae7c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.119+0000 7fae96800640 1 --2- 172.21.15.40:0/3114673618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae7c0039b0 0x7fae7c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.119+0000 7fae8de00640 1 --2- 172.21.15.40:0/3114673618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980d06f0 0x7fae7c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.119+0000 7fae97200640 1 -- 172.21.15.40:0/3114673618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980d06f0 msgr2=0x7fae7c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.119+0000 7fae97200640 1 --2- 172.21.15.40:0/3114673618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980d06f0 0x7fae7c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.119+0000 7fae97200640 1 -- 172.21.15.40:0/3114673618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae7c0039b0 msgr2=0x7fae7c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.119+0000 7fae97200640 1 --2- 172.21.15.40:0/3114673618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae7c0039b0 0x7fae7c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.119+0000 7fae97200640 1 -- 172.21.15.40:0/3114673618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae7c003520 con 0x7fae7c004d40 2024-04-26T23:04:48.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.120+0000 7fae97200640 1 --2- 172.21.15.40:0/3114673618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae7c004d40 0x7fae7c0051c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fae90007410 tx=0x7fae7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704e9f33b2331ad4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.120+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90009c70 con 0x7fae7c004d40 2024-04-26T23:04:48.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.120+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae9002c6c0 con 0x7fae7c004d40 2024-04-26T23:04:48.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.120+0000 7fae95e00640 1 -- 172.21.15.40:0/3114673618 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9002ca70 con 0x7fae7c004d40 2024-04-26T23:04:48.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.121+0000 7fae9d400640 1 -- 172.21.15.40:0/3114673618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae7c004d40 msgr2=0x7fae7c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.121+0000 7fae9d400640 1 --2- 172.21.15.40:0/3114673618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae7c004d40 0x7fae7c0051c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fae90007410 tx=0x7fae7c005830 comp rx=0 tx=0).stop 2024-04-26T23:04:48.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.121+0000 7fae97200640 1 -- 172.21.15.40:0/3114673618 reap_dead start 2024-04-26T23:04:48.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.121+0000 7fae9d400640 1 -- 172.21.15.40:0/3114673618 shutdown_connections 2024-04-26T23:04:48.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.121+0000 7fae9d400640 1 -- 172.21.15.40:0/3114673618 >> 172.21.15.40:0/3114673618 conn(0x7fae980d4f90 msgr2=0x7fae980d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.121+0000 7fae9d400640 1 -- 172.21.15.40:0/3114673618 shutdown_connections 2024-04-26T23:04:48.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.121+0000 7fae9d400640 1 -- 172.21.15.40:0/3114673618 wait complete. 2024-04-26T23:04:48.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.123+0000 7fae9d400640 1 Processor -- start 2024-04-26T23:04:48.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.123+0000 7fae9d400640 1 -- start start 2024-04-26T23:04:48.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.124+0000 7fae9d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae7c004d40 0x7fae98154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.124+0000 7fae9d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980caa00 0x7fae980cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.124+0000 7fae9d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae980ccf50 0x7fae980cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.124+0000 7fae9d400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fae9815f900 con 0x7fae980ccf50 2024-04-26T23:04:48.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.124+0000 7fae9d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fae9815f780 con 0x7fae7c004d40 2024-04-26T23:04:48.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.124+0000 7fae9d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fae980d1e80 con 0x7fae980caa00 2024-04-26T23:04:48.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.124+0000 7fae97200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae980ccf50 0x7fae980cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.124+0000 7fae97200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae980ccf50 0x7fae980cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48144/0 (socket says 172.21.15.40:48144) 2024-04-26T23:04:48.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.124+0000 7fae97200640 1 -- 172.21.15.40:0/2898246735 learned_addr learned my addr 172.21.15.40:0/2898246735 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:48.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae96800640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae7c004d40 0x7fae98154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae8de00640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980caa00 0x7fae980cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae97200640 1 -- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980caa00 msgr2=0x7fae980cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae97200640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980caa00 0x7fae980cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae97200640 1 -- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae7c004d40 msgr2=0x7fae98154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae97200640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae7c004d40 0x7fae98154490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae97200640 1 -- 172.21.15.40:0/2898246735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae980b8590 con 0x7fae980ccf50 2024-04-26T23:04:48.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae8de00640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980caa00 0x7fae980cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:48.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae96800640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae7c004d40 0x7fae98154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:48.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae97200640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae980ccf50 0x7fae980cd3d0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fae90007410 tx=0x7fae90004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae8f400640 1 -- 172.21.15.40:0/2898246735 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90009c70 con 0x7fae980ccf50 2024-04-26T23:04:48.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae8f400640 1 -- 172.21.15.40:0/2898246735 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae90046070 con 0x7fae980ccf50 2024-04-26T23:04:48.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae8f400640 1 -- 172.21.15.40:0/2898246735 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90034b00 con 0x7fae980ccf50 2024-04-26T23:04:48.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.125+0000 7fae9d400640 1 -- 172.21.15.40:0/2898246735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae980b8870 con 0x7fae980ccf50 2024-04-26T23:04:48.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.126+0000 7fae9d400640 1 -- 172.21.15.40:0/2898246735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae980b8e30 con 0x7fae980ccf50 2024-04-26T23:04:48.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.128+0000 7fae8f400640 1 -- 172.21.15.40:0/2898246735 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fae90034ca0 con 0x7fae980ccf50 2024-04-26T23:04:48.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.129+0000 7fae8f400640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fae7406c920 0x7fae7406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.129+0000 7fae8f400640 1 -- 172.21.15.40:0/2898246735 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6159+0+0 (secure 0 0 0) 0x7fae900acf00 con 0x7fae980ccf50 2024-04-26T23:04:48.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.129+0000 7fae96800640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fae7406c920 0x7fae7406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.131+0000 7fae96800640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fae7406c920 0x7fae7406ede0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fae980ba410 tx=0x7fae80000f90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.131+0000 7faea01e7740 1 Processor -- start 2024-04-26T23:04:48.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.131+0000 7faea01e7740 1 -- start start 2024-04-26T23:04:48.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7faea01e7740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a5057ef50 0x564a5057f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7faea01e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 0x564a5059cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7faea01e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a50554c30 0x564a5059aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7faea01e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564a504b8900 con 0x564a5059c7f0 2024-04-26T23:04:48.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7faea01e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564a504551d0 con 0x564a50554c30 2024-04-26T23:04:48.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7faea01e7740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564a504b48c0 con 0x564a5057ef50 2024-04-26T23:04:48.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7fae8de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 0x564a5059cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7fae8de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 0x564a5059cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48160/0 (socket says 172.21.15.40:48160) 2024-04-26T23:04:48.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7fae8de00640 1 -- 172.21.15.40:0/1047096963 learned_addr learned my addr 172.21.15.40:0/1047096963 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:48.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7fae96800640 1 --2- 172.21.15.40:0/1047096963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a5057ef50 0x564a5057f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7fae8de00640 1 -- 172.21.15.40:0/1047096963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a5057ef50 msgr2=0x564a5057f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7fae8de00640 1 --2- 172.21.15.40:0/1047096963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a5057ef50 0x564a5057f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7fae8de00640 1 -- 172.21.15.40:0/1047096963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a50554c30 msgr2=0x564a5059aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7fae8de00640 1 --2- 172.21.15.40:0/1047096963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a50554c30 0x564a5059aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.133+0000 7fae8de00640 1 -- 172.21.15.40:0/1047096963 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a502947d0 con 0x564a5059c7f0 2024-04-26T23:04:48.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.134+0000 7fae8de00640 1 --2- 172.21.15.40:0/1047096963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 0x564a5059cd40 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fae8800e9d0 tx=0x7fae8800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7bb96d07a690303 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.134+0000 7fae87000640 1 -- 172.21.15.40:0/1047096963 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8800f040 con 0x564a5059c7f0 2024-04-26T23:04:48.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.134+0000 7fae87000640 1 -- 172.21.15.40:0/1047096963 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae88010d00 con 0x564a5059c7f0 2024-04-26T23:04:48.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.134+0000 7fae87000640 1 -- 172.21.15.40:0/1047096963 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8801c600 con 0x564a5059c7f0 2024-04-26T23:04:48.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.135+0000 7faea01e7740 1 -- 172.21.15.40:0/1047096963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 msgr2=0x564a5059cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.135+0000 7faea01e7740 1 --2- 172.21.15.40:0/1047096963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 0x564a5059cd40 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fae8800e9d0 tx=0x7fae8800ee90 comp rx=0 tx=0).stop 2024-04-26T23:04:48.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.135+0000 7faea01e7740 1 -- 172.21.15.40:0/1047096963 shutdown_connections 2024-04-26T23:04:48.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.135+0000 7faea01e7740 1 --2- 172.21.15.40:0/1047096963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a50554c30 0x564a5059aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.135+0000 7faea01e7740 1 --2- 172.21.15.40:0/1047096963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 0x564a5059cd40 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.135+0000 7faea01e7740 1 --2- 172.21.15.40:0/1047096963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a5057ef50 0x564a5057f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.135+0000 7faea01e7740 1 -- 172.21.15.40:0/1047096963 >> 172.21.15.40:0/1047096963 conn(0x564a50551560 msgr2=0x564a50599d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.135+0000 7faea01e7740 1 -- 172.21.15.40:0/1047096963 shutdown_connections 2024-04-26T23:04:48.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.135+0000 7faea01e7740 1 -- 172.21.15.40:0/1047096963 wait complete. 2024-04-26T23:04:48.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.136+0000 7faea01e7740 1 -- start start 2024-04-26T23:04:48.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.136+0000 7faea01e7740 1 Processor -- start 2024-04-26T23:04:48.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7faea01e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a5057ef50 0x564a5057b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7faea01e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 0x564a50590e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7faea01e7740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a505aa100 0x564a5059ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7faea01e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564a505559d0 con 0x564a5059c7f0 2024-04-26T23:04:48.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7faea01e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564a50440120 con 0x564a5057ef50 2024-04-26T23:04:48.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7faea01e7740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564a5054f490 con 0x564a505aa100 2024-04-26T23:04:48.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7faea01e7740 10 client.0 _ll_register_callbacks cb 0x564a50478750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7fae8de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 0x564a50590e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7fae8de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 0x564a50590e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48166/0 (socket says 172.21.15.40:48166) 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7fae8de00640 1 -- 172.21.15.40:0/1782926441 learned_addr learned my addr 172.21.15.40:0/1782926441 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7fae8de00640 1 -- 172.21.15.40:0/1782926441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a505aa100 msgr2=0x564a5059ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7fae8de00640 1 --2- 172.21.15.40:0/1782926441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a505aa100 0x564a5059ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7fae8de00640 1 -- 172.21.15.40:0/1782926441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a5057ef50 msgr2=0x564a5057b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7fae8de00640 1 --2- 172.21.15.40:0/1782926441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a5057ef50 0x564a5057b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.137+0000 7fae8de00640 1 -- 172.21.15.40:0/1782926441 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a5018a0f0 con 0x564a5059c7f0 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.138+0000 7fae8de00640 1 --2- 172.21.15.40:0/1782926441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 0x564a50590e10 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fae88018000 tx=0x7fae880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b3cf77bf282dd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.138+0000 7fae7b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.138+0000 7fae7b400640 1 -- 172.21.15.40:0/1782926441 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8800e5a0 con 0x564a5059c7f0 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.138+0000 7fae7b400640 1 -- 172.21.15.40:0/1782926441 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(53..53 src has 1..53) ==== 6159+0+0 (secure 0 0 0) 0x7fae88010b00 con 0x564a5059c7f0 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.138+0000 7faea01e7740 1 -- 172.21.15.40:0/1782926441 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae88010d00 con 0x564a5059c7f0 2024-04-26T23:04:48.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.138+0000 7fae7b400640 1 -- 172.21.15.40:0/1782926441 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fae8801cdf0 con 0x564a5059c7f0 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.138+0000 7faea01e7740 10 client.6417 fetch_fsmap learned FSMap version 43 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.138+0000 7faea01e7740 1 -- 172.21.15.40:0/1782926441 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) -- 0x564a502f4f60 con 0x564a5059c7f0 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.139+0000 7fae7b400640 1 -- 172.21.15.40:0/1782926441 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae88010d00 con 0x564a5059c7f0 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.139+0000 7fae7b400640 1 -- 172.21.15.40:0/1782926441 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 43) ==== 3550+0+0 (secure 0 0 0) 0x7fae88030080 con 0x564a5059c7f0 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.139+0000 7faea01e7740 10 client.6417 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.139+0000 7faea01e7740 10 client.6417 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.139+0000 7faea01e7740 20 client.6417 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.139+0000 7faea01e7740 1 --2- 172.21.15.40:0/1782926441 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x564a50582cd0 0x564a5057ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.139+0000 7faea01e7740 4 client.6417 mds_command: new command op to 6036 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.139+0000 7faea01e7740 1 -- 172.21.15.40:0/1782926441 --> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a504c6510 con 0x564a50582cd0 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.139+0000 7fae96800640 1 --2- 172.21.15.40:0/1782926441 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x564a50582cd0 0x564a5057ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.140+0000 7fae96800640 1 --2- 172.21.15.40:0/1782926441 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x564a50582cd0 0x564a5057ff00 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94791c336472e53a server_cookie=8caef7a1334d5363 in_seq=0 out_seq=0 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.140+0000 7fae7b400640 10 client.6417 ms_handle_connect on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.140+0000 7fae7b400640 1 -- 172.21.15.40:0/1782926441 <== mds.0 v2:172.21.15.40:6833/618018110 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a504c6510 con 0x564a50582cd0 2024-04-26T23:04:48.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.140+0000 7fae7b400640 10 client.6417 handle_command_reply: tid=0 2024-04-26T23:04:48.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.140+0000 7faea01e7740 1 client.6417 shutdown 2024-04-26T23:04:48.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.141+0000 7faea01e7740 1 -- 172.21.15.40:0/1782926441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 msgr2=0x564a50590e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.141+0000 7faea01e7740 1 --2- 172.21.15.40:0/1782926441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 0x564a50590e10 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fae88018000 tx=0x7fae880046c0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.141+0000 7faea01e7740 1 -- 172.21.15.40:0/1782926441 shutdown_connections 2024-04-26T23:04:48.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.141+0000 7faea01e7740 1 --2- 172.21.15.40:0/1782926441 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x564a50582cd0 0x564a5057ff00 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.141+0000 7faea01e7740 1 --2- 172.21.15.40:0/1782926441 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a505aa100 0x564a5059ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.141+0000 7faea01e7740 1 --2- 172.21.15.40:0/1782926441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5059c7f0 0x564a50590e10 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.141+0000 7faea01e7740 1 --2- 172.21.15.40:0/1782926441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a5057ef50 0x564a5057b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.141+0000 7faea01e7740 1 -- 172.21.15.40:0/1782926441 >> 172.21.15.40:0/1782926441 conn(0x564a50554c30 msgr2=0x564a5059d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.141+0000 7fae7b400640 0 client.6417 ms_handle_reset on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:48.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.141+0000 7faea01e7740 1 -- 172.21.15.40:0/1782926441 shutdown_connections 2024-04-26T23:04:48.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.141+0000 7faea01e7740 1 -- 172.21.15.40:0/1782926441 wait complete. 2024-04-26T23:04:48.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.141+0000 7faea01e7740 20 client.6417 trim_cache size 0 max 16384 2024-04-26T23:04:48.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.160+0000 7faea01e7740 1 Processor -- start 2024-04-26T23:04:48.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.160+0000 7faea01e7740 1 -- start start 2024-04-26T23:04:48.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.160+0000 7faea01e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 0x564a505bde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.160+0000 7faea01e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a5059c7f0 0x564a50591910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.160+0000 7faea01e7740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a505aa100 0x564a50591e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.160+0000 7faea01e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fae982026d0 con 0x564a5057ef50 2024-04-26T23:04:48.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.160+0000 7faea01e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fae98201870 con 0x564a5059c7f0 2024-04-26T23:04:48.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.160+0000 7faea01e7740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fae98201fa0 con 0x564a505aa100 2024-04-26T23:04:48.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.161+0000 7fae96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 0x564a505bde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.161+0000 7fae96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 0x564a505bde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48174/0 (socket says 172.21.15.40:48174) 2024-04-26T23:04:48.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.161+0000 7fae96800640 1 -- 172.21.15.40:0/2938223575 learned_addr learned my addr 172.21.15.40:0/2938223575 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.161+0000 7fae96800640 1 -- 172.21.15.40:0/2938223575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a505aa100 msgr2=0x564a50591e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.161+0000 7fae96800640 1 --2- 172.21.15.40:0/2938223575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a505aa100 0x564a50591e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.161+0000 7fae96800640 1 -- 172.21.15.40:0/2938223575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a5059c7f0 msgr2=0x564a50591910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.161+0000 7fae96800640 1 --2- 172.21.15.40:0/2938223575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a5059c7f0 0x564a50591910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.161+0000 7fae96800640 1 -- 172.21.15.40:0/2938223575 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a501631b0 con 0x564a5057ef50 2024-04-26T23:04:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.161+0000 7fae96800640 1 --2- 172.21.15.40:0/2938223575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 0x564a505bde10 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fae8000f540 tx=0x7fae80012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7152966a0595f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.162+0000 7fae86600640 1 -- 172.21.15.40:0/2938223575 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80010040 con 0x564a5057ef50 2024-04-26T23:04:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.162+0000 7fae86600640 1 -- 172.21.15.40:0/2938223575 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae80029070 con 0x564a5057ef50 2024-04-26T23:04:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.162+0000 7fae86600640 1 -- 172.21.15.40:0/2938223575 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80015e00 con 0x564a5057ef50 2024-04-26T23:04:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.162+0000 7faea01e7740 1 -- 172.21.15.40:0/2938223575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 msgr2=0x564a505bde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.162+0000 7faea01e7740 1 --2- 172.21.15.40:0/2938223575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 0x564a505bde10 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fae8000f540 tx=0x7fae80012000 comp rx=0 tx=0).stop 2024-04-26T23:04:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.162+0000 7faea01e7740 1 -- 172.21.15.40:0/2938223575 shutdown_connections 2024-04-26T23:04:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.162+0000 7faea01e7740 1 --2- 172.21.15.40:0/2938223575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a505aa100 0x564a50591e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.162+0000 7faea01e7740 1 --2- 172.21.15.40:0/2938223575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a5059c7f0 0x564a50591910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.162+0000 7faea01e7740 1 --2- 172.21.15.40:0/2938223575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 0x564a505bde10 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.162+0000 7faea01e7740 1 -- 172.21.15.40:0/2938223575 >> 172.21.15.40:0/2938223575 conn(0x564a50554c30 msgr2=0x564a5059bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.162+0000 7faea01e7740 1 -- 172.21.15.40:0/2938223575 shutdown_connections 2024-04-26T23:04:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.163+0000 7faea01e7740 1 -- 172.21.15.40:0/2938223575 wait complete. 2024-04-26T23:04:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.163+0000 7faea01e7740 1 -- start start 2024-04-26T23:04:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.163+0000 7faea01e7740 1 Processor -- start 2024-04-26T23:04:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.164+0000 7faea01e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 0x564a505bd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.164+0000 7faea01e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a5059c7f0 0x564a505b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.164+0000 7faea01e7740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a505aa100 0x564a50613640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.164+0000 7faea01e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564a50581420 con 0x564a5057ef50 2024-04-26T23:04:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.164+0000 7faea01e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fae980ca1a0 con 0x564a5059c7f0 2024-04-26T23:04:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.164+0000 7faea01e7740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564a505559e0 con 0x564a505aa100 2024-04-26T23:04:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.164+0000 7faea01e7740 10 client.0 _ll_register_callbacks cb 0x7fae8800e820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.165+0000 7fae96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 0x564a505bd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.165+0000 7fae96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 0x564a505bd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48182/0 (socket says 172.21.15.40:48182) 2024-04-26T23:04:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.165+0000 7fae96800640 1 -- 172.21.15.40:0/3200701683 learned_addr learned my addr 172.21.15.40:0/3200701683 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.165+0000 7fae96800640 1 -- 172.21.15.40:0/3200701683 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a505aa100 msgr2=0x564a50613640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.165+0000 7fae96800640 1 --2- 172.21.15.40:0/3200701683 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a505aa100 0x564a50613640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.165+0000 7fae96800640 1 -- 172.21.15.40:0/3200701683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a5059c7f0 msgr2=0x564a505b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.165+0000 7fae8de00640 1 --2- 172.21.15.40:0/3200701683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a5059c7f0 0x564a505b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.165+0000 7fae96800640 1 --2- 172.21.15.40:0/3200701683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a5059c7f0 0x564a505b40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.165+0000 7fae96800640 1 -- 172.21.15.40:0/3200701683 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a50612bc0 con 0x564a5057ef50 2024-04-26T23:04:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.165+0000 7fae96800640 1 --2- 172.21.15.40:0/3200701683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 0x564a505bd3f0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fae80026040 tx=0x7fae80001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be57131613f2d870 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.165+0000 7fae7b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7fae7b400640 1 -- 172.21.15.40:0/3200701683 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80024050 con 0x564a5057ef50 2024-04-26T23:04:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7fae7b400640 1 -- 172.21.15.40:0/3200701683 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(53..53 src has 1..53) ==== 6159+0+0 (secure 0 0 0) 0x7fae800241f0 con 0x564a5057ef50 2024-04-26T23:04:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7faea01e7740 1 -- 172.21.15.40:0/3200701683 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae80029070 con 0x564a5057ef50 2024-04-26T23:04:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7fae7b400640 1 -- 172.21.15.40:0/3200701683 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fae80040050 con 0x564a5057ef50 2024-04-26T23:04:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7faea01e7740 10 client.6423 fetch_fsmap learned FSMap version 43 2024-04-26T23:04:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7faea01e7740 1 -- 172.21.15.40:0/3200701683 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) -- 0x564a50612ea0 con 0x564a5057ef50 2024-04-26T23:04:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7fae7b400640 1 -- 172.21.15.40:0/3200701683 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae80029070 con 0x564a5057ef50 2024-04-26T23:04:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7fae7b400640 1 -- 172.21.15.40:0/3200701683 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 43) ==== 3550+0+0 (secure 0 0 0) 0x7fae80043020 con 0x564a5057ef50 2024-04-26T23:04:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7faea01e7740 10 client.6423 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T23:04:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7faea01e7740 10 client.6423 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-04-26T23:04:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7faea01e7740 20 client.6423 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7faea01e7740 1 --2- 172.21.15.40:0/3200701683 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x564a5058f6f0 0x564a5058fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7faea01e7740 4 client.6423 mds_command: new command op to 6036 tid=0 multi_id=0 [{"prefix": "scrub abort"}] 2024-04-26T23:04:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.166+0000 7faea01e7740 1 -- 172.21.15.40:0/3200701683 --> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] -- command(tid 0: {"prefix": "scrub abort"}) -- 0x564a504c0380 con 0x564a5058f6f0 2024-04-26T23:04:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.167+0000 7fae8de00640 1 --2- 172.21.15.40:0/3200701683 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x564a5058f6f0 0x564a5058fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.167+0000 7fae8de00640 1 --2- 172.21.15.40:0/3200701683 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x564a5058f6f0 0x564a5058fb20 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=176fb2836ea33e34 server_cookie=ef970a450bee0ed7 in_seq=0 out_seq=0 2024-04-26T23:04:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.167+0000 7fae7b400640 10 client.6423 ms_handle_connect on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7fae7b400640 1 -- 172.21.15.40:0/3200701683 <== mds.0 v2:172.21.15.40:6833/618018110 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x564a504c0380 con 0x564a5058f6f0 2024-04-26T23:04:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7fae7b400640 10 client.6423 handle_command_reply: tid=0 2024-04-26T23:04:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7faea01e7740 1 client.6423 shutdown 2024-04-26T23:04:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7faea01e7740 1 -- 172.21.15.40:0/3200701683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 msgr2=0x564a505bd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7faea01e7740 1 --2- 172.21.15.40:0/3200701683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 0x564a505bd3f0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fae80026040 tx=0x7fae80001ab0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7faea01e7740 1 -- 172.21.15.40:0/3200701683 shutdown_connections 2024-04-26T23:04:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7faea01e7740 1 --2- 172.21.15.40:0/3200701683 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x564a5058f6f0 0x564a5058fb20 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7faea01e7740 1 --2- 172.21.15.40:0/3200701683 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564a505aa100 0x564a50613640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7faea01e7740 1 --2- 172.21.15.40:0/3200701683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564a5059c7f0 0x564a505b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7faea01e7740 1 --2- 172.21.15.40:0/3200701683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564a5057ef50 0x564a505bd3f0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7fae7b400640 0 client.6423 ms_handle_reset on v2:172.21.15.40:6833/618018110 2024-04-26T23:04:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7faea01e7740 1 -- 172.21.15.40:0/3200701683 >> 172.21.15.40:0/3200701683 conn(0x564a50554c30 msgr2=0x564a5059bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7faea01e7740 1 -- 172.21.15.40:0/3200701683 shutdown_connections 2024-04-26T23:04:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7faea01e7740 1 -- 172.21.15.40:0/3200701683 wait complete. 2024-04-26T23:04:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.168+0000 7faea01e7740 20 client.6423 trim_cache size 0 max 16384 2024-04-26T23:04:48.172 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:04:48.172 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:04:48.172 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:04:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.169+0000 7fae9d400640 1 -- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fae7406c920 msgr2=0x7fae7406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.169+0000 7fae9d400640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fae7406c920 0x7fae7406ede0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fae980ba410 tx=0x7fae80000f90 comp rx=0 tx=0).stop 2024-04-26T23:04:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.169+0000 7fae9d400640 1 -- 172.21.15.40:0/2898246735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae980ccf50 msgr2=0x7fae980cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.169+0000 7fae9d400640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae980ccf50 0x7fae980cd3d0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fae90007410 tx=0x7fae90004ab0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.169+0000 7fae9d400640 1 -- 172.21.15.40:0/2898246735 shutdown_connections 2024-04-26T23:04:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.169+0000 7fae9d400640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fae7406c920 0x7fae7406ede0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.169+0000 7fae9d400640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae980ccf50 0x7fae980cd3d0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.169+0000 7fae9d400640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fae980caa00 0x7fae980cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.169+0000 7fae9d400640 1 --2- 172.21.15.40:0/2898246735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae7c004d40 0x7fae98154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.169+0000 7fae9d400640 1 -- 172.21.15.40:0/2898246735 >> 172.21.15.40:0/2898246735 conn(0x7fae980d4f90 msgr2=0x7fae98074f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.169+0000 7fae9d400640 1 -- 172.21.15.40:0/2898246735 shutdown_connections 2024-04-26T23:04:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.169+0000 7fae9d400640 1 -- 172.21.15.40:0/2898246735 wait complete. 2024-04-26T23:04:48.191 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:04:48.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.3:1 scrub status 2024-04-26T23:04:48.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.376+0000 7f04c9000640 1 Processor -- start 2024-04-26T23:04:48.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.377+0000 7f04c9000640 1 -- start start 2024-04-26T23:04:48.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.378+0000 7f04c9000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f04c40cfd30 0x7f04c40c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.378+0000 7f04c9000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04c40c68c0 con 0x7f04c40d36f0 2024-04-26T23:04:48.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.378+0000 7f04c9000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04c40c6b60 con 0x7f04c40d11b0 2024-04-26T23:04:48.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.378+0000 7f04c9000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f04c40d3b80 con 0x7f04c40cfd30 2024-04-26T23:04:48.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.378+0000 7f04c2a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f04c40d36f0 0x7f04c40c7640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35798/0 (socket says 172.21.15.40:35798) 2024-04-26T23:04:48.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.378+0000 7f04c2a00640 1 -- 172.21.15.40:0/2781296690 learned_addr learned my addr 172.21.15.40:0/2781296690 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:48.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.379+0000 7f04c3e00640 1 --2- 172.21.15.40:0/2781296690 >> v2:172.21.15.103:3300/0 conn(0x7f04c40cfd30 0x7f04c40c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.379+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3284417714 0 0) 0x7f04c40c68c0 con 0x7f04c40d36f0 2024-04-26T23:04:48.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.379+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04a8003520 con 0x7f04c40d36f0 2024-04-26T23:04:48.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.379+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2528121149 0 0) 0x7f04a8003520 con 0x7f04c40d36f0 2024-04-26T23:04:48.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.379+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 >> v2:172.21.15.103:3300/0 conn(0x7f04c40cfd30 msgr2=0x7f04c40c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.379+0000 7f04c2000640 1 --2- 172.21.15.40:0/2781296690 >> v2:172.21.15.103:3300/0 conn(0x7f04c40cfd30 0x7f04c40c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.379+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 >> v1:172.21.15.103:6789/0 conn(0x7f04c40d11b0 legacy=0x7f04c40d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.379+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c40c6e40 con 0x7f04c40d36f0 2024-04-26T23:04:48.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.379+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f04b0001690 con 0x7f04c40d36f0 2024-04-26T23:04:48.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.379+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 >> v1:172.21.15.40:6789/0 conn(0x7f04c40d36f0 legacy=0x7f04c40c7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.379+0000 7f04c2000640 1 --2- 172.21.15.40:0/2781296690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a80039e0 0x7f04a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.380+0000 7f04c2000640 1 --2- 172.21.15.40:0/2781296690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04a8004800 0x7f04a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.380+0000 7f04c2000640 1 --2- 172.21.15.40:0/2781296690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04a8005210 0x7f04a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.380+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f04c40d3b80 con 0x7f04a80039e0 2024-04-26T23:04:48.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.380+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04a800c140 con 0x7f04a8005210 2024-04-26T23:04:48.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.380+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f04a800c340 con 0x7f04a8004800 2024-04-26T23:04:48.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.380+0000 7f04c3400640 1 --2- 172.21.15.40:0/2781296690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a80039e0 0x7f04a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.380+0000 7f04c3e00640 1 --2- 172.21.15.40:0/2781296690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04a8005210 0x7f04a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.380+0000 7f04c3400640 1 -- 172.21.15.40:0/2781296690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04a8004800 msgr2=0x7f04a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.380+0000 7f04c3400640 1 --2- 172.21.15.40:0/2781296690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04a8004800 0x7f04a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.380+0000 7f04c3400640 1 -- 172.21.15.40:0/2781296690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04a8005210 msgr2=0x7f04a800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.380+0000 7f04c3400640 1 --2- 172.21.15.40:0/2781296690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04a8005210 0x7f04a800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.380+0000 7f04c3400640 1 -- 172.21.15.40:0/2781296690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a8003520 con 0x7f04a80039e0 2024-04-26T23:04:48.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.380+0000 7f04c3e00640 1 -- 172.21.15.40:0/2781296690 reap_dead start 2024-04-26T23:04:48.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.381+0000 7f04c3400640 1 --2- 172.21.15.40:0/2781296690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a80039e0 0x7f04a8003e00 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f04b4001750 tx=0x7f04a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be614d34d2ba480 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.381+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b4001370 con 0x7f04a80039e0 2024-04-26T23:04:48.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.381+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b40052d0 con 0x7f04a80039e0 2024-04-26T23:04:48.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.381+0000 7f04c2000640 1 -- 172.21.15.40:0/2781296690 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b402abc0 con 0x7f04a80039e0 2024-04-26T23:04:48.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.382+0000 7f04c9000640 1 -- 172.21.15.40:0/2781296690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a80039e0 msgr2=0x7f04a8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.382+0000 7f04c9000640 1 --2- 172.21.15.40:0/2781296690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a80039e0 0x7f04a8003e00 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f04b4001750 tx=0x7f04a8005830 comp rx=0 tx=0).stop 2024-04-26T23:04:48.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.382+0000 7f04c9000640 1 -- 172.21.15.40:0/2781296690 shutdown_connections 2024-04-26T23:04:48.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.382+0000 7f04c9000640 1 --2- 172.21.15.40:0/2781296690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a80039e0 0x7f04a8003e00 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.382+0000 7f04c9000640 1 -- 172.21.15.40:0/2781296690 >> 172.21.15.40:0/2781296690 conn(0x7f04c40d2bd0 msgr2=0x7f04c40d2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.382+0000 7f04c9000640 1 -- 172.21.15.40:0/2781296690 shutdown_connections 2024-04-26T23:04:48.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.382+0000 7f04c9000640 1 -- 172.21.15.40:0/2781296690 wait complete. 2024-04-26T23:04:48.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.384+0000 7f04c9000640 1 Processor -- start 2024-04-26T23:04:48.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.384+0000 7f04c9000640 1 -- start start 2024-04-26T23:04:48.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.385+0000 7f04c9000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a80039e0 0x7f04c4152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.385+0000 7f04c9000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c40afdc0 0x7f04c40b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.385+0000 7f04c9000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04c415a9d0 0x7f04c415ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.385+0000 7f04c9000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f04c40b44c0 con 0x7f04a80039e0 2024-04-26T23:04:48.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.385+0000 7f04c9000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04c40b4340 con 0x7f04c40afdc0 2024-04-26T23:04:48.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.385+0000 7f04c9000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f04c40d09b0 con 0x7f04c415a9d0 2024-04-26T23:04:48.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.385+0000 7f04c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a80039e0 0x7f04c4152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.385+0000 7f04c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a80039e0 0x7f04c4152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48194/0 (socket says 172.21.15.40:48194) 2024-04-26T23:04:48.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.385+0000 7f04c3400640 1 -- 172.21.15.40:0/645562552 learned_addr learned my addr 172.21.15.40:0/645562552 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:48.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.386+0000 7f04c3400640 1 -- 172.21.15.40:0/645562552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04c415a9d0 msgr2=0x7f04c415ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.386+0000 7f04c3400640 1 --2- 172.21.15.40:0/645562552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04c415a9d0 0x7f04c415ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.386+0000 7f04c3400640 1 -- 172.21.15.40:0/645562552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c40afdc0 msgr2=0x7f04c40b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.386+0000 7f04c3400640 1 --2- 172.21.15.40:0/645562552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c40afdc0 0x7f04c40b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.386+0000 7f04c3400640 1 -- 172.21.15.40:0/645562552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c40b2950 con 0x7f04a80039e0 2024-04-26T23:04:48.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.386+0000 7f04c3400640 1 --2- 172.21.15.40:0/645562552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a80039e0 0x7f04c4152060 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f04b4005040 tx=0x7f04b4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.386+0000 7f04bb400640 1 -- 172.21.15.40:0/645562552 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b4037040 con 0x7f04a80039e0 2024-04-26T23:04:48.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.386+0000 7f04bb400640 1 -- 172.21.15.40:0/645562552 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b4040070 con 0x7f04a80039e0 2024-04-26T23:04:48.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.386+0000 7f04bb400640 1 -- 172.21.15.40:0/645562552 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b4001370 con 0x7f04a80039e0 2024-04-26T23:04:48.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.386+0000 7f04c9000640 1 -- 172.21.15.40:0/645562552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c40b2c30 con 0x7f04a80039e0 2024-04-26T23:04:48.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.387+0000 7f04c9000640 1 -- 172.21.15.40:0/645562552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c41525a0 con 0x7f04a80039e0 2024-04-26T23:04:48.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.389+0000 7f04bb400640 1 -- 172.21.15.40:0/645562552 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f04b40052d0 con 0x7f04a80039e0 2024-04-26T23:04:48.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.390+0000 7f04bb400640 1 --2- 172.21.15.40:0/645562552 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f04a406c8f0 0x7f04a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.390+0000 7f04bb400640 1 -- 172.21.15.40:0/645562552 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f04b40ab9c0 con 0x7f04a80039e0 2024-04-26T23:04:48.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.390+0000 7f04cd5a2740 1 Processor -- start 2024-04-26T23:04:48.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.390+0000 7f04cd5a2740 1 -- start start 2024-04-26T23:04:48.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.391+0000 7f04c2a00640 1 --2- 172.21.15.40:0/645562552 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f04a406c8f0 0x7f04a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.391+0000 7f04cd5a2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c52710 0x55d454c52af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.391+0000 7f04cd5a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c6ffb0 0x55d454c70500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.391+0000 7f04cd5a2740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c283f0 0x55d454c6e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.391+0000 7f04cd5a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d454b38030 con 0x55d454c6ffb0 2024-04-26T23:04:48.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.391+0000 7f04cd5a2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d454b537f0 con 0x55d454c52710 2024-04-26T23:04:48.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.391+0000 7f04cd5a2740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d454b0e7e0 con 0x55d454c283f0 2024-04-26T23:04:48.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.391+0000 7f04c2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c6ffb0 0x55d454c70500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.391+0000 7f04c2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c6ffb0 0x55d454c70500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48202/0 (socket says 172.21.15.40:48202) 2024-04-26T23:04:48.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.391+0000 7f04c2a00640 1 -- 172.21.15.40:0/4024897024 learned_addr learned my addr 172.21.15.40:0/4024897024 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.392+0000 7f04c2a00640 1 -- 172.21.15.40:0/4024897024 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c283f0 msgr2=0x55d454c6e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.392+0000 7f04c2a00640 1 --2- 172.21.15.40:0/4024897024 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c283f0 0x55d454c6e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.392+0000 7f04c2a00640 1 -- 172.21.15.40:0/4024897024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c52710 msgr2=0x55d454c52af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.392+0000 7f04c2a00640 1 --2- 172.21.15.40:0/4024897024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c52710 0x55d454c52af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.392+0000 7f04c2a00640 1 -- 172.21.15.40:0/4024897024 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d454b1a050 con 0x55d454c6ffb0 2024-04-26T23:04:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.392+0000 7f04c2a00640 1 --2- 172.21.15.40:0/645562552 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f04a406c8f0 0x7f04a406edb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f04b00011e0 tx=0x7f04b00014f0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.392+0000 7f04c2a00640 1 --2- 172.21.15.40:0/4024897024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c6ffb0 0x55d454c70500 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f04b0000ff0 tx=0x7f04b0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb6afef3cd77f6c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.392+0000 7f04afe00640 1 -- 172.21.15.40:0/4024897024 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b00074e0 con 0x55d454c6ffb0 2024-04-26T23:04:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.392+0000 7f04afe00640 1 -- 172.21.15.40:0/4024897024 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b00048a0 con 0x55d454c6ffb0 2024-04-26T23:04:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.392+0000 7f04afe00640 1 -- 172.21.15.40:0/4024897024 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b0004c50 con 0x55d454c6ffb0 2024-04-26T23:04:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.393+0000 7f04cd5a2740 1 -- 172.21.15.40:0/4024897024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c6ffb0 msgr2=0x55d454c70500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.393+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/4024897024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c6ffb0 0x55d454c70500 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f04b0000ff0 tx=0x7f04b0019000 comp rx=0 tx=0).stop 2024-04-26T23:04:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.393+0000 7f04cd5a2740 1 -- 172.21.15.40:0/4024897024 shutdown_connections 2024-04-26T23:04:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.393+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/4024897024 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c283f0 0x55d454c6e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.393+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/4024897024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c6ffb0 0x55d454c70500 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.393+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/4024897024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c52710 0x55d454c52af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.393+0000 7f04cd5a2740 1 -- 172.21.15.40:0/4024897024 >> 172.21.15.40:0/4024897024 conn(0x55d454c24d20 msgr2=0x55d454c6d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.393+0000 7f04cd5a2740 1 -- 172.21.15.40:0/4024897024 shutdown_connections 2024-04-26T23:04:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.393+0000 7f04cd5a2740 1 -- 172.21.15.40:0/4024897024 wait complete. 2024-04-26T23:04:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.394+0000 7f04cd5a2740 1 -- start start 2024-04-26T23:04:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.394+0000 7f04cd5a2740 1 Processor -- start 2024-04-26T23:04:48.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.395+0000 7f04cd5a2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c52710 0x55d454c64660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.395+0000 7f04cd5a2740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c6ffb0 0x55d454c51790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.395+0000 7f04cd5a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 0x55d454c51cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.395+0000 7f04cd5a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d454b90200 con 0x55d454c7d180 2024-04-26T23:04:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.395+0000 7f04cd5a2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d454b8aee0 con 0x55d454c52710 2024-04-26T23:04:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.395+0000 7f04cd5a2740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d454b97e70 con 0x55d454c6ffb0 2024-04-26T23:04:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.395+0000 7f04cd5a2740 10 client.0 _ll_register_callbacks cb 0x55d454b81270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.395+0000 7f04c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 0x55d454c51cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.395+0000 7f04c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 0x55d454c51cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48206/0 (socket says 172.21.15.40:48206) 2024-04-26T23:04:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.395+0000 7f04c3e00640 1 -- 172.21.15.40:0/1040459578 learned_addr learned my addr 172.21.15.40:0/1040459578 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.396+0000 7f04c2a00640 1 --2- 172.21.15.40:0/1040459578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c6ffb0 0x55d454c51790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.396+0000 7f04c3400640 1 --2- 172.21.15.40:0/1040459578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c52710 0x55d454c64660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.396+0000 7f04c3e00640 1 -- 172.21.15.40:0/1040459578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c6ffb0 msgr2=0x55d454c51790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.396+0000 7f04c3e00640 1 --2- 172.21.15.40:0/1040459578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c6ffb0 0x55d454c51790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.396+0000 7f04c3e00640 1 -- 172.21.15.40:0/1040459578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c52710 msgr2=0x55d454c64660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.396+0000 7f04c3e00640 1 --2- 172.21.15.40:0/1040459578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c52710 0x55d454c64660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.396+0000 7f04c3e00640 1 -- 172.21.15.40:0/1040459578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4549677d0 con 0x55d454c7d180 2024-04-26T23:04:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.396+0000 7f04c3e00640 1 --2- 172.21.15.40:0/1040459578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 0x55d454c51cd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f04c40d1db0 tx=0x7f04bc007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae9e8c3d9b9e181 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.396+0000 7f04acc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.396+0000 7f04acc00640 1 -- 172.21.15.40:0/1040459578 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc00fb00 con 0x55d454c7d180 2024-04-26T23:04:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.396+0000 7f04acc00640 1 -- 172.21.15.40:0/1040459578 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f04bc00fca0 con 0x55d454c7d180 2024-04-26T23:04:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.396+0000 7f04cd5a2740 1 -- 172.21.15.40:0/1040459578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04b00048a0 con 0x55d454c7d180 2024-04-26T23:04:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.397+0000 7f04acc00640 1 -- 172.21.15.40:0/1040459578 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f04bc00fea0 con 0x55d454c7d180 2024-04-26T23:04:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.397+0000 7f04cd5a2740 10 client.6435 fetch_fsmap learned FSMap version 43 2024-04-26T23:04:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.397+0000 7f04cd5a2740 1 -- 172.21.15.40:0/1040459578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) -- 0x55d45489d3f0 con 0x55d454c7d180 2024-04-26T23:04:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.397+0000 7f04acc00640 1 -- 172.21.15.40:0/1040459578 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b00048a0 con 0x55d454c7d180 2024-04-26T23:04:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.397+0000 7f04acc00640 1 -- 172.21.15.40:0/1040459578 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 43) ==== 3550+0+0 (secure 0 0 0) 0x7f04bc026080 con 0x55d454c7d180 2024-04-26T23:04:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.397+0000 7f04cd5a2740 10 client.6435 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T23:04:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.397+0000 7f04cd5a2740 10 client.6435 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.a 2024-04-26T23:04:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.397+0000 7f04cd5a2740 20 client.6435 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.397+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/1040459578 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55d454c4d550 0x55d454c55da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.397+0000 7f04cd5a2740 4 client.6435 mds_command: new command op to 6027 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:04:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.397+0000 7f04cd5a2740 1 -- 172.21.15.40:0/1040459578 --> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d454b997e0 con 0x55d454c4d550 2024-04-26T23:04:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.397+0000 7f04c3400640 1 --2- 172.21.15.40:0/1040459578 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55d454c4d550 0x55d454c55da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.398+0000 7f04c3400640 1 --2- 172.21.15.40:0/1040459578 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55d454c4d550 0x55d454c55da0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=249e56361be33678 server_cookie=568f3995cfdfc6bf in_seq=0 out_seq=0 2024-04-26T23:04:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.398+0000 7f04acc00640 10 client.6435 ms_handle_connect on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04acc00640 1 -- 172.21.15.40:0/1040459578 <== mds.1 v2:172.21.15.40:6832/2304583125 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d454b997e0 con 0x55d454c4d550 2024-04-26T23:04:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04acc00640 10 client.6435 handle_command_reply: tid=0 2024-04-26T23:04:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04cd5a2740 1 client.6435 shutdown 2024-04-26T23:04:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04cd5a2740 1 -- 172.21.15.40:0/1040459578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 msgr2=0x55d454c51cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/1040459578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 0x55d454c51cd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f04c40d1db0 tx=0x7f04bc007dd0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04cd5a2740 1 -- 172.21.15.40:0/1040459578 shutdown_connections 2024-04-26T23:04:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/1040459578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 0x55d454c51cd0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/1040459578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c6ffb0 0x55d454c51790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/1040459578 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55d454c4d550 0x55d454c55da0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/1040459578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c52710 0x55d454c64660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04cd5a2740 1 -- 172.21.15.40:0/1040459578 >> 172.21.15.40:0/1040459578 conn(0x55d454c283f0 msgr2=0x55d454c278e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04acc00640 0 client.6435 ms_handle_reset on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:48.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04cd5a2740 1 -- 172.21.15.40:0/1040459578 shutdown_connections 2024-04-26T23:04:48.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04cd5a2740 1 -- 172.21.15.40:0/1040459578 wait complete. 2024-04-26T23:04:48.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.399+0000 7f04cd5a2740 20 client.6435 trim_cache size 0 max 16384 2024-04-26T23:04:48.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.418+0000 7f04cd5a2740 1 Processor -- start 2024-04-26T23:04:48.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.418+0000 7f04cd5a2740 1 -- start start 2024-04-26T23:04:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.419+0000 7f04cd5a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c52710 0x55d454c56680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.419+0000 7f04cd5a2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c6ffb0 0x55d454c4c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.419+0000 7f04cd5a2740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c7d180 0x55d454c90ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.419+0000 7f04cd5a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f04c4202640 con 0x55d454c52710 2024-04-26T23:04:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.419+0000 7f04cd5a2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04c42017e0 con 0x55d454c6ffb0 2024-04-26T23:04:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.419+0000 7f04cd5a2740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f04c4201f10 con 0x55d454c7d180 2024-04-26T23:04:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.419+0000 7f04c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c52710 0x55d454c56680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.419+0000 7f04c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c52710 0x55d454c56680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48222/0 (socket says 172.21.15.40:48222) 2024-04-26T23:04:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.419+0000 7f04c3400640 1 -- 172.21.15.40:0/2479028634 learned_addr learned my addr 172.21.15.40:0/2479028634 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.420+0000 7f04c3400640 1 -- 172.21.15.40:0/2479028634 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c7d180 msgr2=0x55d454c90ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.420+0000 7f04c3400640 1 --2- 172.21.15.40:0/2479028634 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c7d180 0x55d454c90ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.420+0000 7f04c3400640 1 -- 172.21.15.40:0/2479028634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c6ffb0 msgr2=0x55d454c4c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.420+0000 7f04c3400640 1 --2- 172.21.15.40:0/2479028634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c6ffb0 0x55d454c4c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.420+0000 7f04c3400640 1 -- 172.21.15.40:0/2479028634 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d454c52e60 con 0x55d454c52710 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.420+0000 7f04c3400640 1 --2- 172.21.15.40:0/2479028634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c52710 0x55d454c56680 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f04b402a4f0 tx=0x7f04b402a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6eeac8d0362de6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.420+0000 7f04af400640 1 -- 172.21.15.40:0/2479028634 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b402be30 con 0x55d454c52710 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.420+0000 7f04af400640 1 -- 172.21.15.40:0/2479028634 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b4096070 con 0x55d454c52710 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.420+0000 7f04af400640 1 -- 172.21.15.40:0/2479028634 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b4079b90 con 0x55d454c52710 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.421+0000 7f04cd5a2740 1 -- 172.21.15.40:0/2479028634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c52710 msgr2=0x55d454c56680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.421+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/2479028634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c52710 0x55d454c56680 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f04b402a4f0 tx=0x7f04b402a480 comp rx=0 tx=0).stop 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.421+0000 7f04cd5a2740 1 -- 172.21.15.40:0/2479028634 shutdown_connections 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.421+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/2479028634 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c7d180 0x55d454c90ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.421+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/2479028634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c6ffb0 0x55d454c4c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.421+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/2479028634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c52710 0x55d454c56680 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.421+0000 7f04cd5a2740 1 -- 172.21.15.40:0/2479028634 >> 172.21.15.40:0/2479028634 conn(0x55d454c283f0 msgr2=0x55d454c21c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.421+0000 7f04cd5a2740 1 -- 172.21.15.40:0/2479028634 shutdown_connections 2024-04-26T23:04:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.421+0000 7f04cd5a2740 1 -- 172.21.15.40:0/2479028634 wait complete. 2024-04-26T23:04:48.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.422+0000 7f04cd5a2740 1 -- start start 2024-04-26T23:04:48.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.422+0000 7f04cd5a2740 1 Processor -- start 2024-04-26T23:04:48.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04cd5a2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c52710 0x55d454c6b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04cd5a2740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c6ffb0 0x55d454c51420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04cd5a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 0x55d454c51960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04cd5a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d454b98fd0 con 0x55d454c7d180 2024-04-26T23:04:48.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04cd5a2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04c4200830 con 0x55d454c52710 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04cd5a2740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d454c27110 con 0x55d454c6ffb0 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04cd5a2740 10 client.0 _ll_register_callbacks cb 0x7f04bc009270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 0x55d454c51960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 0x55d454c51960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48226/0 (socket says 172.21.15.40:48226) 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04c3e00640 1 -- 172.21.15.40:0/361270928 learned_addr learned my addr 172.21.15.40:0/361270928 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04c3e00640 1 -- 172.21.15.40:0/361270928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c6ffb0 msgr2=0x55d454c51420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04c3e00640 1 --2- 172.21.15.40:0/361270928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c6ffb0 0x55d454c51420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04c3e00640 1 -- 172.21.15.40:0/361270928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c52710 msgr2=0x55d454c6b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04c3e00640 1 --2- 172.21.15.40:0/361270928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c52710 0x55d454c6b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.423+0000 7f04c3e00640 1 -- 172.21.15.40:0/361270928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d45485d0f0 con 0x55d454c7d180 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.424+0000 7f04c3e00640 1 --2- 172.21.15.40:0/361270928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 0x55d454c51960 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f04bc000f80 tx=0x7f04bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf3ef0c07d6edda server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.424+0000 7f04acc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.424+0000 7f04acc00640 1 -- 172.21.15.40:0/361270928 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc0167c0 con 0x55d454c7d180 2024-04-26T23:04:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.424+0000 7f04acc00640 1 -- 172.21.15.40:0/361270928 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f04bc020c40 con 0x55d454c7d180 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.424+0000 7f04cd5a2740 1 -- 172.21.15.40:0/361270928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04b4096070 con 0x55d454c7d180 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.424+0000 7f04acc00640 1 -- 172.21.15.40:0/361270928 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f04bc020e40 con 0x55d454c7d180 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.424+0000 7f04cd5a2740 10 client.6441 fetch_fsmap learned FSMap version 43 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.424+0000 7f04cd5a2740 1 -- 172.21.15.40:0/361270928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) -- 0x55d454b53a30 con 0x55d454c7d180 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.424+0000 7f04acc00640 1 -- 172.21.15.40:0/361270928 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b4096070 con 0x55d454c7d180 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.425+0000 7f04acc00640 1 -- 172.21.15.40:0/361270928 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 43) ==== 3550+0+0 (secure 0 0 0) 0x7f04bc016960 con 0x55d454c7d180 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.425+0000 7f04cd5a2740 10 client.6441 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.425+0000 7f04cd5a2740 10 client.6441 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.a 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.425+0000 7f04cd5a2740 20 client.6441 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.425+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/361270928 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55d454ce6150 0x55d454c262c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.425+0000 7f04cd5a2740 4 client.6441 mds_command: new command op to 6027 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.425+0000 7f04cd5a2740 1 -- 172.21.15.40:0/361270928 --> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d454b93650 con 0x55d454ce6150 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.425+0000 7f04c3400640 1 --2- 172.21.15.40:0/361270928 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55d454ce6150 0x55d454c262c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:48.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.425+0000 7f04c3400640 1 --2- 172.21.15.40:0/361270928 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55d454ce6150 0x55d454c262c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1f0fc3d18bc37238 server_cookie=872992add81584d6 in_seq=0 out_seq=0 2024-04-26T23:04:48.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.425+0000 7f04acc00640 10 client.6441 ms_handle_connect on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:48.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04acc00640 1 -- 172.21.15.40:0/361270928 <== mds.1 v2:172.21.15.40:6832/2304583125 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55d454b93650 con 0x55d454ce6150 2024-04-26T23:04:48.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04acc00640 10 client.6441 handle_command_reply: tid=0 2024-04-26T23:04:48.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04cd5a2740 1 client.6441 shutdown 2024-04-26T23:04:48.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04cd5a2740 1 -- 172.21.15.40:0/361270928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 msgr2=0x55d454c51960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/361270928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 0x55d454c51960 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f04bc000f80 tx=0x7f04bc00b040 comp rx=0 tx=0).stop 2024-04-26T23:04:48.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04cd5a2740 1 -- 172.21.15.40:0/361270928 shutdown_connections 2024-04-26T23:04:48.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/361270928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d454c7d180 0x55d454c51960 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/361270928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d454c6ffb0 0x55d454c51420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/361270928 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55d454ce6150 0x55d454c262c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04cd5a2740 1 --2- 172.21.15.40:0/361270928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d454c52710 0x55d454c6b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04cd5a2740 1 -- 172.21.15.40:0/361270928 >> 172.21.15.40:0/361270928 conn(0x55d454c283f0 msgr2=0x55d454c277d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04acc00640 0 client.6441 ms_handle_reset on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:48.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04cd5a2740 1 -- 172.21.15.40:0/361270928 shutdown_connections 2024-04-26T23:04:48.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04cd5a2740 1 -- 172.21.15.40:0/361270928 wait complete. 2024-04-26T23:04:48.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.426+0000 7f04cd5a2740 20 client.6441 trim_cache size 0 max 16384 2024-04-26T23:04:48.430 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:04:48.430 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-26T23:04:48.430 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:04:48.430 INFO:teuthology.orchestra.run.smithi040.stdout: "b07480c4-ced7-4267-91ee-1b8bac05bf74": { 2024-04-26T23:04:48.430 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:04:48.430 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "b07480c4-ced7-4267-91ee-1b8bac05bf74", 2024-04-26T23:04:48.430 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:04:48.430 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:48.431 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:48.431 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:04:48.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.427+0000 7f04c9000640 1 -- 172.21.15.40:0/645562552 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f04a406c8f0 msgr2=0x7f04a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.427+0000 7f04c9000640 1 --2- 172.21.15.40:0/645562552 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f04a406c8f0 0x7f04a406edb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f04b00011e0 tx=0x7f04b00014f0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.427+0000 7f04c9000640 1 -- 172.21.15.40:0/645562552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a80039e0 msgr2=0x7f04c4152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:48.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.427+0000 7f04c9000640 1 --2- 172.21.15.40:0/645562552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a80039e0 0x7f04c4152060 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f04b4005040 tx=0x7f04b4001120 comp rx=0 tx=0).stop 2024-04-26T23:04:48.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.427+0000 7f04c9000640 1 -- 172.21.15.40:0/645562552 shutdown_connections 2024-04-26T23:04:48.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.427+0000 7f04c9000640 1 --2- 172.21.15.40:0/645562552 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f04a406c8f0 0x7f04a406edb0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.427+0000 7f04c9000640 1 --2- 172.21.15.40:0/645562552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04c415a9d0 0x7f04c415ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.427+0000 7f04c9000640 1 --2- 172.21.15.40:0/645562552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c40afdc0 0x7f04c40b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.427+0000 7f04c9000640 1 --2- 172.21.15.40:0/645562552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a80039e0 0x7f04c4152060 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:48.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.427+0000 7f04c9000640 1 -- 172.21.15.40:0/645562552 >> 172.21.15.40:0/645562552 conn(0x7f04c40d2bd0 msgr2=0x7f04c4071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:48.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.427+0000 7f04c9000640 1 -- 172.21.15.40:0/645562552 shutdown_connections 2024-04-26T23:04:48.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:48.427+0000 7f04c9000640 1 -- 172.21.15.40:0/645562552 wait complete. 2024-04-26T23:04:53.386 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:04:53.386+0000 7f36ffc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-26T23:04:53.443 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:04:53.443 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.3:1 scrub status 2024-04-26T23:04:53.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.648+0000 7fa5ae600640 1 Processor -- start 2024-04-26T23:04:53.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.648+0000 7fa5ae600640 1 -- start start 2024-04-26T23:04:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.649+0000 7fa5ae600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa5a815f3a0 0x7fa5a80d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.649+0000 7fa5ae600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5a80d0e20 con 0x7fa5a80d1770 2024-04-26T23:04:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.650+0000 7fa5ae600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5a80b1d70 con 0x7fa5a80d2330 2024-04-26T23:04:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.650+0000 7fa5ae600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa5a80b2050 con 0x7fa5a815f3a0 2024-04-26T23:04:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.650+0000 7fa5a6a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa5a80d1770 0x7fa5a80d1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35802/0 (socket says 172.21.15.40:35802) 2024-04-26T23:04:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.650+0000 7fa5a6a00640 1 -- 172.21.15.40:0/3514082061 learned_addr learned my addr 172.21.15.40:0/3514082061 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:04:53.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.650+0000 7fa5a7400640 1 --2- 172.21.15.40:0/3514082061 >> v2:172.21.15.103:3300/0 conn(0x7fa5a815f3a0 0x7fa5a80d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.650+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1449463921 0 0) 0x7fa5a80d0e20 con 0x7fa5a80d1770 2024-04-26T23:04:53.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.650+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa58c003520 con 0x7fa5a80d1770 2024-04-26T23:04:53.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2957229800 0 0) 0x7fa58c003520 con 0x7fa5a80d1770 2024-04-26T23:04:53.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 >> v2:172.21.15.103:3300/0 conn(0x7fa5a815f3a0 msgr2=0x7fa5a80d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:53.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a6000640 1 --2- 172.21.15.40:0/3514082061 >> v2:172.21.15.103:3300/0 conn(0x7fa5a815f3a0 0x7fa5a80d06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 >> v1:172.21.15.103:6789/0 conn(0x7fa5a80d2330 legacy=0x7fa5a80b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:53.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a80b2330 con 0x7fa5a80d1770 2024-04-26T23:04:53.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fa594001690 con 0x7fa5a80d1770 2024-04-26T23:04:53.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 >> v1:172.21.15.40:6789/0 conn(0x7fa5a80d1770 legacy=0x7fa5a80d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:53.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a6000640 1 --2- 172.21.15.40:0/3514082061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa58c0039b0 0x7fa58c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a6000640 1 --2- 172.21.15.40:0/3514082061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa58c004800 0x7fa58c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a6000640 1 --2- 172.21.15.40:0/3514082061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa58c0051c0 0x7fa58c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a80b2050 con 0x7fa58c0039b0 2024-04-26T23:04:53.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa58c00c060 con 0x7fa58c004800 2024-04-26T23:04:53.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa58c00c260 con 0x7fa58c0051c0 2024-04-26T23:04:53.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.651+0000 7fa5a7400640 1 --2- 172.21.15.40:0/3514082061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa58c0039b0 0x7fa58c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.652+0000 7fa5a6a00640 1 --2- 172.21.15.40:0/3514082061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa58c004800 0x7fa58c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.652+0000 7fa5a7400640 1 -- 172.21.15.40:0/3514082061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa58c0051c0 msgr2=0x7fa58c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:53.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.652+0000 7fa5a7400640 1 --2- 172.21.15.40:0/3514082061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa58c0051c0 0x7fa58c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.652+0000 7fa5a7400640 1 -- 172.21.15.40:0/3514082061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa58c004800 msgr2=0x7fa58c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:53.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.652+0000 7fa5a7400640 1 --2- 172.21.15.40:0/3514082061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa58c004800 0x7fa58c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.652+0000 7fa5a7400640 1 -- 172.21.15.40:0/3514082061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa58c003520 con 0x7fa58c0039b0 2024-04-26T23:04:53.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.652+0000 7fa5a6a00640 1 --2- 172.21.15.40:0/3514082061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa58c004800 0x7fa58c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:53.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.652+0000 7fa5a7e00640 1 -- 172.21.15.40:0/3514082061 reap_dead start 2024-04-26T23:04:53.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.653+0000 7fa5a7400640 1 --2- 172.21.15.40:0/3514082061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa58c0039b0 0x7fa58c003dd0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa598009fc0 tx=0x7fa58c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d3c18935411061 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:53.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.653+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa598009c10 con 0x7fa58c0039b0 2024-04-26T23:04:53.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.653+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa598036070 con 0x7fa58c0039b0 2024-04-26T23:04:53.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.653+0000 7fa5a6000640 1 -- 172.21.15.40:0/3514082061 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59802d070 con 0x7fa58c0039b0 2024-04-26T23:04:53.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.654+0000 7fa5ae600640 1 -- 172.21.15.40:0/3514082061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa58c0039b0 msgr2=0x7fa58c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:53.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.654+0000 7fa5ae600640 1 --2- 172.21.15.40:0/3514082061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa58c0039b0 0x7fa58c003dd0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa598009fc0 tx=0x7fa58c005830 comp rx=0 tx=0).stop 2024-04-26T23:04:53.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.654+0000 7fa5ae600640 1 -- 172.21.15.40:0/3514082061 shutdown_connections 2024-04-26T23:04:53.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.654+0000 7fa5ae600640 1 --2- 172.21.15.40:0/3514082061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa58c0039b0 0x7fa58c003dd0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.654+0000 7fa5ae600640 1 -- 172.21.15.40:0/3514082061 >> 172.21.15.40:0/3514082061 conn(0x7fa5a80d4f90 msgr2=0x7fa5a80d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.654+0000 7fa5ae600640 1 -- 172.21.15.40:0/3514082061 shutdown_connections 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.654+0000 7fa5ae600640 1 -- 172.21.15.40:0/3514082061 wait complete. 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.656+0000 7fa5ae600640 1 Processor -- start 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.656+0000 7fa5ae600640 1 -- start start 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5ae600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa58c0039b0 0x7fa5a80cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5ae600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5a80cd370 0x7fa5a81f53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5ae600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5a81f5930 0x7fa5a815cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5ae600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a80d0870 con 0x7fa58c0039b0 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5ae600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a80d06f0 con 0x7fa5a80cd370 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5ae600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa5a80d09f0 con 0x7fa5a81f5930 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5a7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa58c0039b0 0x7fa5a80cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5a7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa58c0039b0 0x7fa5a80cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48256/0 (socket says 172.21.15.40:48256) 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5a7400640 1 -- 172.21.15.40:0/1931167875 learned_addr learned my addr 172.21.15.40:0/1931167875 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5a6a00640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5a80cd370 0x7fa5a81f53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5a7400640 1 -- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5a81f5930 msgr2=0x7fa5a815cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:53.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5a7400640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5a81f5930 0x7fa5a815cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5a7400640 1 -- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5a80cd370 msgr2=0x7fa5a81f53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:53.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5a7400640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5a80cd370 0x7fa5a81f53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5a7400640 1 -- 172.21.15.40:0/1931167875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a80cd810 con 0x7fa58c0039b0 2024-04-26T23:04:53.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.657+0000 7fa5a6a00640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5a80cd370 0x7fa5a81f53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:53.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.658+0000 7fa5a7400640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa58c0039b0 0x7fa5a80cce30 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa59800e230 tx=0x7fa598009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:53.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.658+0000 7fa59f400640 1 -- 172.21.15.40:0/1931167875 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa598009c10 con 0x7fa58c0039b0 2024-04-26T23:04:53.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.658+0000 7fa59f400640 1 -- 172.21.15.40:0/1931167875 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa598003bb0 con 0x7fa58c0039b0 2024-04-26T23:04:53.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.658+0000 7fa5ae600640 1 -- 172.21.15.40:0/1931167875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a80b7dd0 con 0x7fa58c0039b0 2024-04-26T23:04:53.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.658+0000 7fa59f400640 1 -- 172.21.15.40:0/1931167875 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa598004020 con 0x7fa58c0039b0 2024-04-26T23:04:53.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.658+0000 7fa5ae600640 1 -- 172.21.15.40:0/1931167875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5a80b8390 con 0x7fa58c0039b0 2024-04-26T23:04:53.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.660+0000 7fa59f400640 1 -- 172.21.15.40:0/1931167875 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa598005d10 con 0x7fa58c0039b0 2024-04-26T23:04:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.662+0000 7fa5b15a6740 1 Processor -- start 2024-04-26T23:04:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.662+0000 7fa5b15a6740 1 -- start start 2024-04-26T23:04:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.662+0000 7fa59f400640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa58806c8f0 0x7fa58806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.662+0000 7fa59f400640 1 -- 172.21.15.40:0/1931167875 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7fa5980ad350 con 0x7fa58c0039b0 2024-04-26T23:04:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5a7e00640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa58806c8f0 0x7fa58806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5b15a6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685d4540 0x557b685d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5b15a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685f1de0 0x557b685f2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5b15a6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685aa220 0x557b685f04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5b15a6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557b684ba030 con 0x557b685d4540 2024-04-26T23:04:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5b15a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557b684d57f0 con 0x557b685f1de0 2024-04-26T23:04:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5b15a6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x557b684907e0 con 0x557b685aa220 2024-04-26T23:04:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5a7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685d4540 0x557b685d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5a7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685d4540 0x557b685d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48272/0 (socket says 172.21.15.40:48272) 2024-04-26T23:04:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5a7400640 1 -- 172.21.15.40:0/2796465996 learned_addr learned my addr 172.21.15.40:0/2796465996 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5a6a00640 1 --2- 172.21.15.40:0/2796465996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685f1de0 0x557b685f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5a7400640 1 -- 172.21.15.40:0/2796465996 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685aa220 msgr2=0x557b685f04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5a7400640 1 --2- 172.21.15.40:0/2796465996 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685aa220 0x557b685f04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5a7400640 1 -- 172.21.15.40:0/2796465996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685f1de0 msgr2=0x557b685f2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5a7400640 1 --2- 172.21.15.40:0/2796465996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685f1de0 0x557b685f2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.663+0000 7fa5a7400640 1 -- 172.21.15.40:0/2796465996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b6849c050 con 0x557b685d4540 2024-04-26T23:04:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.664+0000 7fa5a7400640 1 --2- 172.21.15.40:0/2796465996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685d4540 0x557b685d4920 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa5980ab650 tx=0x7fa598004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a90638d70af827c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.664+0000 7fa593e00640 1 -- 172.21.15.40:0/2796465996 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59807f260 con 0x557b685d4540 2024-04-26T23:04:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.664+0000 7fa593e00640 1 -- 172.21.15.40:0/2796465996 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa598036020 con 0x557b685d4540 2024-04-26T23:04:53.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.664+0000 7fa593e00640 1 -- 172.21.15.40:0/2796465996 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa598083a90 con 0x557b685d4540 2024-04-26T23:04:53.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.665+0000 7fa5b15a6740 1 -- 172.21.15.40:0/2796465996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685d4540 msgr2=0x557b685d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:53.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.665+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/2796465996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685d4540 0x557b685d4920 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa5980ab650 tx=0x7fa598004280 comp rx=0 tx=0).stop 2024-04-26T23:04:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.665+0000 7fa5a7e00640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa58806c8f0 0x7fa58806edb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa5a808d470 tx=0x7fa5a0007cc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.665+0000 7fa5b15a6740 1 -- 172.21.15.40:0/2796465996 shutdown_connections 2024-04-26T23:04:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.665+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/2796465996 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685aa220 0x557b685f04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.665+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/2796465996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685f1de0 0x557b685f2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.665+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/2796465996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685d4540 0x557b685d4920 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.665+0000 7fa5b15a6740 1 -- 172.21.15.40:0/2796465996 >> 172.21.15.40:0/2796465996 conn(0x557b685a6b50 msgr2=0x557b685ef350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.665+0000 7fa5b15a6740 1 -- 172.21.15.40:0/2796465996 shutdown_connections 2024-04-26T23:04:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.665+0000 7fa5b15a6740 1 -- 172.21.15.40:0/2796465996 wait complete. 2024-04-26T23:04:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.665+0000 7fa5b15a6740 1 -- start start 2024-04-26T23:04:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.666+0000 7fa5b15a6740 1 Processor -- start 2024-04-26T23:04:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5b15a6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685d4540 0x557b685d6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5b15a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685f1de0 0x557b685d84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5b15a6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685c9300 0x557b685f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5b15a6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557b68512200 con 0x557b685c9300 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5b15a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557b6850cee0 con 0x557b685f1de0 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5b15a6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x557b68519e70 con 0x557b685d4540 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5b15a6740 10 client.0 _ll_register_callbacks cb 0x557b68503270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685c9300 0x557b685f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685c9300 0x557b685f02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48280/0 (socket says 172.21.15.40:48280) 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5a7e00640 1 -- 172.21.15.40:0/1343889362 learned_addr learned my addr 172.21.15.40:0/1343889362 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5a7400640 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685d4540 0x557b685d6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5a6a00640 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685f1de0 0x557b685d84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5a7e00640 1 -- 172.21.15.40:0/1343889362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685d4540 msgr2=0x557b685d6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5a7e00640 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685d4540 0x557b685d6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5a7e00640 1 -- 172.21.15.40:0/1343889362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685f1de0 msgr2=0x557b685d84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5a7e00640 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685f1de0 0x557b685d84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.667+0000 7fa5a7e00640 1 -- 172.21.15.40:0/1343889362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b682e97d0 con 0x557b685c9300 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.668+0000 7fa5a6a00640 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685f1de0 0x557b685d84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.668+0000 7fa5a7e00640 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685c9300 0x557b685f02d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0006a00 tx=0x7fa5a0001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6398239324302fe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.668+0000 7fa590c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.668+0000 7fa590c00640 1 -- 172.21.15.40:0/1343889362 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0024060 con 0x557b685c9300 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.668+0000 7fa590c00640 1 -- 172.21.15.40:0/1343889362 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7fa5a0016a80 con 0x557b685c9300 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.668+0000 7fa5b15a6740 1 -- 172.21.15.40:0/1343889362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa598036020 con 0x557b685c9300 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.668+0000 7fa590c00640 1 -- 172.21.15.40:0/1343889362 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa5a002a050 con 0x557b685c9300 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.668+0000 7fa5b15a6740 10 client.6453 fetch_fsmap learned FSMap version 44 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.668+0000 7fa5b15a6740 1 -- 172.21.15.40:0/1343889362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x557b6821f3f0 con 0x557b685c9300 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.669+0000 7fa590c00640 1 -- 172.21.15.40:0/1343889362 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa598036020 con 0x557b685c9300 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.669+0000 7fa590c00640 1 -- 172.21.15.40:0/1343889362 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 44) ==== 3550+0+0 (secure 0 0 0) 0x7fa5a002d020 con 0x557b685c9300 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.669+0000 7fa5b15a6740 10 client.6453 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T23:04:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.669+0000 7fa5b15a6740 10 client.6453 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.a 2024-04-26T23:04:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.669+0000 7fa5b15a6740 20 client.6453 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.669+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x557b685f08b0 0x557b685cf520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.669+0000 7fa5b15a6740 4 client.6453 mds_command: new command op to 6027 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:04:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.669+0000 7fa5b15a6740 1 -- 172.21.15.40:0/1343889362 --> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b6851b7e0 con 0x557b685f08b0 2024-04-26T23:04:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.669+0000 7fa5a7400640 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x557b685f08b0 0x557b685cf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.670+0000 7fa5a7400640 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x557b685f08b0 0x557b685cf520 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ab349868e7348628 server_cookie=e37465fc4fd92a4a in_seq=0 out_seq=0 2024-04-26T23:04:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.670+0000 7fa590c00640 10 client.6453 ms_handle_connect on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.670+0000 7fa590c00640 1 -- 172.21.15.40:0/1343889362 <== mds.1 v2:172.21.15.40:6832/2304583125 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b6851b7e0 con 0x557b685f08b0 2024-04-26T23:04:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.670+0000 7fa590c00640 10 client.6453 handle_command_reply: tid=0 2024-04-26T23:04:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.670+0000 7fa5b15a6740 1 client.6453 shutdown 2024-04-26T23:04:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.671+0000 7fa5b15a6740 1 -- 172.21.15.40:0/1343889362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685c9300 msgr2=0x557b685f02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.671+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685c9300 0x557b685f02d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0006a00 tx=0x7fa5a0001190 comp rx=0 tx=0).stop 2024-04-26T23:04:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.671+0000 7fa5b15a6740 1 -- 172.21.15.40:0/1343889362 shutdown_connections 2024-04-26T23:04:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.671+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685c9300 0x557b685f02d0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.671+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x557b685f08b0 0x557b685cf520 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.671+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685f1de0 0x557b685d84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.671+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/1343889362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685d4540 0x557b685d6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.671+0000 7fa5b15a6740 1 -- 172.21.15.40:0/1343889362 >> 172.21.15.40:0/1343889362 conn(0x557b685aa220 msgr2=0x557b685f28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.671+0000 7fa590c00640 0 client.6453 ms_handle_reset on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.671+0000 7fa5b15a6740 1 -- 172.21.15.40:0/1343889362 shutdown_connections 2024-04-26T23:04:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.671+0000 7fa5b15a6740 1 -- 172.21.15.40:0/1343889362 wait complete. 2024-04-26T23:04:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.671+0000 7fa5b15a6740 20 client.6453 trim_cache size 0 max 16384 2024-04-26T23:04:53.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.707+0000 7fa5b15a6740 1 Processor -- start 2024-04-26T23:04:53.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.707+0000 7fa5b15a6740 1 -- start start 2024-04-26T23:04:53.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5b15a6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685c9300 0x557b685f1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5b15a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685d4540 0x557b685cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5b15a6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 0x557b685aa600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5b15a6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a8202920 con 0x557b685f1de0 2024-04-26T23:04:53.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5b15a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a8201ac0 con 0x557b685d4540 2024-04-26T23:04:53.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5b15a6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa5a82021f0 con 0x557b685c9300 2024-04-26T23:04:53.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 0x557b685aa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 0x557b685aa600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48286/0 (socket says 172.21.15.40:48286) 2024-04-26T23:04:53.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5a7e00640 1 -- 172.21.15.40:0/3319190372 learned_addr learned my addr 172.21.15.40:0/3319190372 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:53.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5a7e00640 1 -- 172.21.15.40:0/3319190372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685c9300 msgr2=0x557b685f1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:53.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5a7400640 1 --2- 172.21.15.40:0/3319190372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685c9300 0x557b685f1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5a6a00640 1 --2- 172.21.15.40:0/3319190372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685d4540 0x557b685cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5a7e00640 1 --2- 172.21.15.40:0/3319190372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685c9300 0x557b685f1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5a7e00640 1 -- 172.21.15.40:0/3319190372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685d4540 msgr2=0x557b685cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:53.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5a7e00640 1 --2- 172.21.15.40:0/3319190372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685d4540 0x557b685cbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5a7e00640 1 -- 172.21.15.40:0/3319190372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b681df0f0 con 0x557b685f1de0 2024-04-26T23:04:53.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.709+0000 7fa5a7400640 1 --2- 172.21.15.40:0/3319190372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685c9300 0x557b685f1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.710+0000 7fa5a7e00640 1 --2- 172.21.15.40:0/3319190372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 0x557b685aa600 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa5a001f460 tx=0x7fa5a001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6278b8ee3d88ec0c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.710+0000 7fa593400640 1 -- 172.21.15.40:0/3319190372 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0033070 con 0x557b685f1de0 2024-04-26T23:04:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.710+0000 7fa593400640 1 -- 172.21.15.40:0/3319190372 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5a0016a80 con 0x557b685f1de0 2024-04-26T23:04:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.710+0000 7fa593400640 1 -- 172.21.15.40:0/3319190372 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0010070 con 0x557b685f1de0 2024-04-26T23:04:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.711+0000 7fa5b15a6740 1 -- 172.21.15.40:0/3319190372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 msgr2=0x557b685aa600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.711+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/3319190372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 0x557b685aa600 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa5a001f460 tx=0x7fa5a001b000 comp rx=0 tx=0).stop 2024-04-26T23:04:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.711+0000 7fa5b15a6740 1 -- 172.21.15.40:0/3319190372 shutdown_connections 2024-04-26T23:04:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.711+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/3319190372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 0x557b685aa600 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.711+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/3319190372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685d4540 0x557b685cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.711+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/3319190372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685c9300 0x557b685f1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.711+0000 7fa5b15a6740 1 -- 172.21.15.40:0/3319190372 >> 172.21.15.40:0/3319190372 conn(0x557b685aa220 msgr2=0x557b685cd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.711+0000 7fa5b15a6740 1 -- 172.21.15.40:0/3319190372 shutdown_connections 2024-04-26T23:04:53.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.711+0000 7fa5b15a6740 1 -- 172.21.15.40:0/3319190372 wait complete. 2024-04-26T23:04:53.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.712+0000 7fa5b15a6740 1 -- start start 2024-04-26T23:04:53.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.712+0000 7fa5b15a6740 1 Processor -- start 2024-04-26T23:04:53.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5b15a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685c9300 0x557b685ee250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5b15a6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685d4540 0x557b685ee790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5b15a6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 0x557b68611ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5b15a6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557b6851afd0 con 0x557b685f1de0 2024-04-26T23:04:53.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5b15a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a80b9e60 con 0x557b685c9300 2024-04-26T23:04:53.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5b15a6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x557b685f1640 con 0x557b685d4540 2024-04-26T23:04:53.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5b15a6740 10 client.0 _ll_register_callbacks cb 0x7fa58c004700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:53.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 0x557b68611ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 0x557b68611ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48302/0 (socket says 172.21.15.40:48302) 2024-04-26T23:04:53.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5a7e00640 1 -- 172.21.15.40:0/3094188025 learned_addr learned my addr 172.21.15.40:0/3094188025 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5a7400640 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685c9300 0x557b685ee250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5a6a00640 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685d4540 0x557b685ee790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5a7e00640 1 -- 172.21.15.40:0/3094188025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685d4540 msgr2=0x557b685ee790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5a7e00640 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685d4540 0x557b685ee790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5a7e00640 1 -- 172.21.15.40:0/3094188025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685c9300 msgr2=0x557b685ee250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5a7e00640 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685c9300 0x557b685ee250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5a7e00640 1 -- 172.21.15.40:0/3094188025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b684d5a30 con 0x557b685f1de0 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5a7400640 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685c9300 0x557b685ee250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.714+0000 7fa5a6a00640 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685d4540 0x557b685ee790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.715+0000 7fa5a7e00640 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 0x557b68611ee0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0031040 tx=0x7fa5a0007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa686951d054016 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.715+0000 7fa590c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.715+0000 7fa590c00640 1 -- 172.21.15.40:0/3094188025 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0024070 con 0x557b685f1de0 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.715+0000 7fa590c00640 1 -- 172.21.15.40:0/3094188025 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7fa5a001fd50 con 0x557b685f1de0 2024-04-26T23:04:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.715+0000 7fa5b15a6740 1 -- 172.21.15.40:0/3094188025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5a0016a80 con 0x557b685f1de0 2024-04-26T23:04:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.715+0000 7fa590c00640 1 -- 172.21.15.40:0/3094188025 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa5a0033040 con 0x557b685f1de0 2024-04-26T23:04:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.715+0000 7fa5b15a6740 10 client.6459 fetch_fsmap learned FSMap version 44 2024-04-26T23:04:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.715+0000 7fa5b15a6740 1 -- 172.21.15.40:0/3094188025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x557b68608420 con 0x557b685f1de0 2024-04-26T23:04:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.716+0000 7fa590c00640 1 -- 172.21.15.40:0/3094188025 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5a0016a80 con 0x557b685f1de0 2024-04-26T23:04:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.716+0000 7fa590c00640 1 -- 172.21.15.40:0/3094188025 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 44) ==== 3550+0+0 (secure 0 0 0) 0x7fa5a0036070 con 0x557b685f1de0 2024-04-26T23:04:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.716+0000 7fa5b15a6740 10 client.6459 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T23:04:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.716+0000 7fa5b15a6740 10 client.6459 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.a 2024-04-26T23:04:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.716+0000 7fa5b15a6740 20 client.6459 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.716+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x557b685cd890 0x557b68612420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.716+0000 7fa5b15a6740 4 client.6459 mds_command: new command op to 6027 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:04:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.716+0000 7fa5b15a6740 1 -- 172.21.15.40:0/3094188025 --> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557b685d41a0 con 0x557b685cd890 2024-04-26T23:04:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.716+0000 7fa5a7400640 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x557b685cd890 0x557b68612420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.716+0000 7fa5a7400640 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x557b685cd890 0x557b68612420 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cd468f13f19ffad5 server_cookie=9b7e8e16354d010c in_seq=0 out_seq=0 2024-04-26T23:04:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.716+0000 7fa590c00640 10 client.6459 ms_handle_connect on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa590c00640 1 -- 172.21.15.40:0/3094188025 <== mds.1 v2:172.21.15.40:6832/2304583125 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x557b685d41a0 con 0x557b685cd890 2024-04-26T23:04:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa590c00640 10 client.6459 handle_command_reply: tid=0 2024-04-26T23:04:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa5b15a6740 1 client.6459 shutdown 2024-04-26T23:04:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa5b15a6740 1 -- 172.21.15.40:0/3094188025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 msgr2=0x557b68611ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 0x557b68611ee0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0031040 tx=0x7fa5a0007000 comp rx=0 tx=0).stop 2024-04-26T23:04:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa5b15a6740 1 -- 172.21.15.40:0/3094188025 shutdown_connections 2024-04-26T23:04:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557b685f1de0 0x557b68611ee0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557b685d4540 0x557b685ee790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x557b685cd890 0x557b68612420 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa5b15a6740 1 --2- 172.21.15.40:0/3094188025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557b685c9300 0x557b685ee250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa5b15a6740 1 -- 172.21.15.40:0/3094188025 >> 172.21.15.40:0/3094188025 conn(0x557b685aa220 msgr2=0x557b685cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa590c00640 0 client.6459 ms_handle_reset on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa5b15a6740 1 -- 172.21.15.40:0/3094188025 shutdown_connections 2024-04-26T23:04:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa5b15a6740 1 -- 172.21.15.40:0/3094188025 wait complete. 2024-04-26T23:04:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.717+0000 7fa5b15a6740 20 client.6459 trim_cache size 0 max 16384 2024-04-26T23:04:53.721 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:04:53.721 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-26T23:04:53.721 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:04:53.721 INFO:teuthology.orchestra.run.smithi040.stdout: "b07480c4-ced7-4267-91ee-1b8bac05bf74": { 2024-04-26T23:04:53.721 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:04:53.722 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "b07480c4-ced7-4267-91ee-1b8bac05bf74", 2024-04-26T23:04:53.722 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:04:53.722 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:53.722 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:53.722 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:04:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.718+0000 7fa5ae600640 1 -- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa58806c8f0 msgr2=0x7fa58806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.718+0000 7fa5ae600640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa58806c8f0 0x7fa58806edb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa5a808d470 tx=0x7fa5a0007cc0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.718+0000 7fa5ae600640 1 -- 172.21.15.40:0/1931167875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa58c0039b0 msgr2=0x7fa5a80cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.718+0000 7fa5ae600640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa58c0039b0 0x7fa5a80cce30 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa59800e230 tx=0x7fa598009750 comp rx=0 tx=0).stop 2024-04-26T23:04:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.718+0000 7fa5ae600640 1 -- 172.21.15.40:0/1931167875 shutdown_connections 2024-04-26T23:04:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.718+0000 7fa5ae600640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa58806c8f0 0x7fa58806edb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.718+0000 7fa5ae600640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5a81f5930 0x7fa5a815cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.718+0000 7fa5ae600640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5a80cd370 0x7fa5a81f53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.718+0000 7fa5ae600640 1 --2- 172.21.15.40:0/1931167875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa58c0039b0 0x7fa5a80cce30 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.718+0000 7fa5ae600640 1 -- 172.21.15.40:0/1931167875 >> 172.21.15.40:0/1931167875 conn(0x7fa5a80d4f90 msgr2=0x7fa5a8074ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.718+0000 7fa5ae600640 1 -- 172.21.15.40:0/1931167875 shutdown_connections 2024-04-26T23:04:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:53.718+0000 7fa5ae600640 1 -- 172.21.15.40:0/1931167875 wait complete. 2024-04-26T23:04:58.737 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:04:58.737 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.3:1 scrub status 2024-04-26T23:04:58.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.962+0000 7f34d3200640 1 Processor -- start 2024-04-26T23:04:58.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.962+0000 7f34d3200640 1 -- start start 2024-04-26T23:04:58.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.964+0000 7f34d3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f34cc160a50 0x7f34cc073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:58.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.964+0000 7f34d3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34cc0cfaf0 0x7f34cc160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:58.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.964+0000 7f34d3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34cc161880 con 0x7f34cc06dd40 2024-04-26T23:04:58.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.964+0000 7f34d3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f34cc071ce0 con 0x7f34cc160a50 2024-04-26T23:04:58.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.964+0000 7f34d3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f34cc0722a0 con 0x7f34cc0cfaf0 2024-04-26T23:04:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.964+0000 7f34d1e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f34cc160a50 0x7f34cc073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.964+0000 7f34d1400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34cc0cfaf0 0x7f34cc160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.965+0000 7f34d1e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f34cc160a50 0x7f34cc073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50050/0 (socket says 172.21.15.40:50050) 2024-04-26T23:04:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.965+0000 7f34d1e00640 1 -- 172.21.15.40:0/2290694337 learned_addr learned my addr 172.21.15.40:0/2290694337 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.965+0000 7f34d1e00640 1 -- 172.21.15.40:0/2290694337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34cc0cfaf0 msgr2=0x7f34cc160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.965+0000 7f34d1e00640 1 --2- 172.21.15.40:0/2290694337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34cc0cfaf0 0x7f34cc160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.965+0000 7f34d1e00640 1 -- 172.21.15.40:0/2290694337 >> v1:172.21.15.103:6789/0 conn(0x7f34cc06dd40 legacy=0x7f34cc072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:58.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.965+0000 7f34d1e00640 1 -- 172.21.15.40:0/2290694337 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34cc161370 con 0x7f34cc160a50 2024-04-26T23:04:58.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.965+0000 7f34d1400640 1 --2- 172.21.15.40:0/2290694337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34cc0cfaf0 0x7f34cc160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:04:58.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.967+0000 7f34d1e00640 1 --2- 172.21.15.40:0/2290694337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34cc160a50 0x7f34cc073a50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009870 tx=0x7f34c002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=834d6b169232de1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:58.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.967+0000 7f34d0a00640 1 -- 172.21.15.40:0/2290694337 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c0037070 con 0x7f34cc160a50 2024-04-26T23:04:58.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.967+0000 7f34d0a00640 1 -- 172.21.15.40:0/2290694337 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c0032440 con 0x7f34cc160a50 2024-04-26T23:04:58.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.967+0000 7f34d0a00640 1 -- 172.21.15.40:0/2290694337 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c0032810 con 0x7f34cc160a50 2024-04-26T23:04:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.968+0000 7f34d3200640 1 -- 172.21.15.40:0/2290694337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34cc160a50 msgr2=0x7f34cc073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.968+0000 7f34d3200640 1 --2- 172.21.15.40:0/2290694337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34cc160a50 0x7f34cc073a50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009870 tx=0x7f34c002a250 comp rx=0 tx=0).stop 2024-04-26T23:04:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.968+0000 7f34d3200640 1 -- 172.21.15.40:0/2290694337 shutdown_connections 2024-04-26T23:04:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.968+0000 7f34d3200640 1 --2- 172.21.15.40:0/2290694337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34cc0cfaf0 0x7f34cc160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.968+0000 7f34d3200640 1 --2- 172.21.15.40:0/2290694337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34cc160a50 0x7f34cc073a50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.968+0000 7f34d3200640 1 -- 172.21.15.40:0/2290694337 >> 172.21.15.40:0/2290694337 conn(0x7f34cc0b2720 msgr2=0x7f34cc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.968+0000 7f34d3200640 1 -- 172.21.15.40:0/2290694337 shutdown_connections 2024-04-26T23:04:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.968+0000 7f34d3200640 1 -- 172.21.15.40:0/2290694337 wait complete. 2024-04-26T23:04:58.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.970+0000 7f34d3200640 1 Processor -- start 2024-04-26T23:04:58.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.970+0000 7f34d3200640 1 -- start start 2024-04-26T23:04:58.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.971+0000 7f34d3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34cc06dd40 0x7f34cc140210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:58.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.971+0000 7f34d3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34cc160a50 0x7f34cc0b54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:58.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.971+0000 7f34d3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34cc13af60 0x7f34cc139540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:58.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.971+0000 7f34d3200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc0d0260 con 0x7f34cc06dd40 2024-04-26T23:04:58.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.971+0000 7f34d3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc073f90 con 0x7f34cc160a50 2024-04-26T23:04:58.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.971+0000 7f34d3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f34cc071f30 con 0x7f34cc13af60 2024-04-26T23:04:58.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.971+0000 7f34d1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34cc06dd40 0x7f34cc140210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:58.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.971+0000 7f34d1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34cc06dd40 0x7f34cc140210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44304/0 (socket says 172.21.15.40:44304) 2024-04-26T23:04:58.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.971+0000 7f34d1e00640 1 -- 172.21.15.40:0/627939381 learned_addr learned my addr 172.21.15.40:0/627939381 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:58.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.971+0000 7f34d1e00640 1 -- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34cc13af60 msgr2=0x7f34cc139540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:58.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.971+0000 7f34d1400640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34cc160a50 0x7f34cc0b54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:58.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.971+0000 7f34d2800640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34cc13af60 0x7f34cc139540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:58.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.972+0000 7f34d1e00640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34cc13af60 0x7f34cc139540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.972+0000 7f34d1e00640 1 -- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34cc160a50 msgr2=0x7f34cc0b54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:58.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.972+0000 7f34d1e00640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34cc160a50 0x7f34cc0b54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.972+0000 7f34d1e00640 1 -- 172.21.15.40:0/627939381 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34cc13a210 con 0x7f34cc06dd40 2024-04-26T23:04:58.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.972+0000 7f34d1e00640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34cc06dd40 0x7f34cc140210 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f34c00097c0 tx=0x7f34c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:58.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.972+0000 7f34c6000640 1 -- 172.21.15.40:0/627939381 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c0037040 con 0x7f34cc06dd40 2024-04-26T23:04:58.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.972+0000 7f34c6000640 1 -- 172.21.15.40:0/627939381 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c0042070 con 0x7f34cc06dd40 2024-04-26T23:04:58.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.972+0000 7f34c6000640 1 -- 172.21.15.40:0/627939381 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c0030940 con 0x7f34cc06dd40 2024-04-26T23:04:58.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.972+0000 7f34d3200640 1 -- 172.21.15.40:0/627939381 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34cc13a4f0 con 0x7f34cc06dd40 2024-04-26T23:04:58.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.972+0000 7f34d3200640 1 -- 172.21.15.40:0/627939381 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34cc13aab0 con 0x7f34cc06dd40 2024-04-26T23:04:58.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.975+0000 7f34c6000640 1 -- 172.21.15.40:0/627939381 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f34c0030ae0 con 0x7f34cc06dd40 2024-04-26T23:04:58.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.976+0000 7f34c6000640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f34b006c920 0x7f34b006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:58.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.976+0000 7f34c6000640 1 -- 172.21.15.40:0/627939381 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f34c00ac5f0 con 0x7f34cc06dd40 2024-04-26T23:04:58.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.976+0000 7f34d1400640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f34b006c920 0x7f34b006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:58.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.976+0000 7f34d7765740 1 Processor -- start 2024-04-26T23:04:58.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.976+0000 7f34d7765740 1 -- start start 2024-04-26T23:04:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.977+0000 7f34d1400640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f34b006c920 0x7f34b006ede0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009560 tx=0x7f34bc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.977+0000 7f34d7765740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56461515a710 0x56461515aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.977+0000 7f34d7765740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564615177fb0 0x564615178500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.977+0000 7f34d7765740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646151303f0 0x564615176670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.977+0000 7f34d7765740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564615040030 con 0x564615177fb0 2024-04-26T23:04:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.977+0000 7f34d7765740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56461505b7f0 con 0x5646151303f0 2024-04-26T23:04:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.977+0000 7f34d7765740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5646150167e0 con 0x56461515a710 2024-04-26T23:04:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.978+0000 7f34d1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564615177fb0 0x564615178500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.978+0000 7f34d1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564615177fb0 0x564615178500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44318/0 (socket says 172.21.15.40:44318) 2024-04-26T23:04:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.978+0000 7f34d1400640 1 -- 172.21.15.40:0/1093861580 learned_addr learned my addr 172.21.15.40:0/1093861580 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.978+0000 7f34d1e00640 1 --2- 172.21.15.40:0/1093861580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56461515a710 0x56461515aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.978+0000 7f34d2800640 1 --2- 172.21.15.40:0/1093861580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646151303f0 0x564615176670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.978+0000 7f34d1400640 1 -- 172.21.15.40:0/1093861580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56461515a710 msgr2=0x56461515aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.978+0000 7f34d1400640 1 --2- 172.21.15.40:0/1093861580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56461515a710 0x56461515aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.978+0000 7f34d1400640 1 -- 172.21.15.40:0/1093861580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646151303f0 msgr2=0x564615176670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.978+0000 7f34d1400640 1 --2- 172.21.15.40:0/1093861580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646151303f0 0x564615176670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.978+0000 7f34d1400640 1 -- 172.21.15.40:0/1093861580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564615022050 con 0x564615177fb0 2024-04-26T23:04:58.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.979+0000 7f34d1400640 1 --2- 172.21.15.40:0/1093861580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564615177fb0 0x564615178500 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f34bc006a70 tx=0x7f34bc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc2460499fde022 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:58.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.979+0000 7f34ba600640 1 -- 172.21.15.40:0/1093861580 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc01c050 con 0x564615177fb0 2024-04-26T23:04:58.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.979+0000 7f34ba600640 1 -- 172.21.15.40:0/1093861580 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34bc026070 con 0x564615177fb0 2024-04-26T23:04:58.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.979+0000 7f34ba600640 1 -- 172.21.15.40:0/1093861580 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc00f040 con 0x564615177fb0 2024-04-26T23:04:58.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.980+0000 7f34d7765740 1 -- 172.21.15.40:0/1093861580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564615177fb0 msgr2=0x564615178500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:58.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.980+0000 7f34d7765740 1 --2- 172.21.15.40:0/1093861580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564615177fb0 0x564615178500 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f34bc006a70 tx=0x7f34bc013000 comp rx=0 tx=0).stop 2024-04-26T23:04:58.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.980+0000 7f34d7765740 1 -- 172.21.15.40:0/1093861580 shutdown_connections 2024-04-26T23:04:58.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.980+0000 7f34d7765740 1 --2- 172.21.15.40:0/1093861580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646151303f0 0x564615176670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.980+0000 7f34d7765740 1 --2- 172.21.15.40:0/1093861580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564615177fb0 0x564615178500 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.980+0000 7f34d7765740 1 --2- 172.21.15.40:0/1093861580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56461515a710 0x56461515aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.980+0000 7f34d7765740 1 -- 172.21.15.40:0/1093861580 >> 172.21.15.40:0/1093861580 conn(0x56461512cd20 msgr2=0x564615175520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:58.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.980+0000 7f34d7765740 1 -- 172.21.15.40:0/1093861580 shutdown_connections 2024-04-26T23:04:58.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.980+0000 7f34d7765740 1 -- 172.21.15.40:0/1093861580 wait complete. 2024-04-26T23:04:58.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.981+0000 7f34d7765740 1 -- start start 2024-04-26T23:04:58.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.981+0000 7f34d7765740 1 Processor -- start 2024-04-26T23:04:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.982+0000 7f34d7765740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56461515a710 0x56461515dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.982+0000 7f34d7765740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564615177fb0 0x56461516f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.982+0000 7f34d7765740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 0x564615149920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.982+0000 7f34d7765740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564615098200 con 0x56461516d2a0 2024-04-26T23:04:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.982+0000 7f34d7765740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564615092ee0 con 0x56461515a710 2024-04-26T23:04:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.982+0000 7f34d7765740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x56461509fe70 con 0x564615177fb0 2024-04-26T23:04:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.982+0000 7f34d7765740 10 client.0 _ll_register_callbacks cb 0x564615089270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.982+0000 7f34d2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 0x564615149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.982+0000 7f34d2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 0x564615149920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44324/0 (socket says 172.21.15.40:44324) 2024-04-26T23:04:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.982+0000 7f34d2800640 1 -- 172.21.15.40:0/2274262895 learned_addr learned my addr 172.21.15.40:0/2274262895 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.982+0000 7f34d1e00640 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56461515a710 0x56461515dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.982+0000 7f34d1400640 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564615177fb0 0x56461516f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.983+0000 7f34d2800640 1 -- 172.21.15.40:0/2274262895 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564615177fb0 msgr2=0x56461516f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.983+0000 7f34d2800640 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564615177fb0 0x56461516f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.983+0000 7f34d2800640 1 -- 172.21.15.40:0/2274262895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56461515a710 msgr2=0x56461515dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.983+0000 7f34d2800640 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56461515a710 0x56461515dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.983+0000 7f34d2800640 1 -- 172.21.15.40:0/2274262895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564614e6f7d0 con 0x56461516d2a0 2024-04-26T23:04:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.983+0000 7f34d1400640 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564615177fb0 0x56461516f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.983+0000 7f34d2800640 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 0x564615149920 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f34c800f4e0 tx=0x7f34c800f9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ce1281170c1287 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.983+0000 7f34aea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.983+0000 7f34aea00640 1 -- 172.21.15.40:0/2274262895 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c8017020 con 0x56461516d2a0 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.983+0000 7f34aea00640 1 -- 172.21.15.40:0/2274262895 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f34c8003b80 con 0x56461516d2a0 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.983+0000 7f34d7765740 1 -- 172.21.15.40:0/2274262895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34bc026070 con 0x56461516d2a0 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.984+0000 7f34aea00640 1 -- 172.21.15.40:0/2274262895 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f34c8021050 con 0x56461516d2a0 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.984+0000 7f34d7765740 10 client.6468 fetch_fsmap learned FSMap version 46 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.984+0000 7f34d7765740 1 -- 172.21.15.40:0/2274262895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x564614da53f0 con 0x56461516d2a0 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.984+0000 7f34aea00640 1 -- 172.21.15.40:0/2274262895 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34bc026070 con 0x56461516d2a0 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.984+0000 7f34aea00640 1 -- 172.21.15.40:0/2274262895 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 46) ==== 3550+0+0 (secure 0 0 0) 0x7f34c8024020 con 0x56461516d2a0 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.984+0000 7f34d7765740 10 client.6468 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.984+0000 7f34d7765740 10 client.6468 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.a 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.984+0000 7f34d7765740 20 client.6468 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.984+0000 7f34d7765740 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x56461514a680 0x56461514aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.984+0000 7f34d7765740 4 client.6468 mds_command: new command op to 6027 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:04:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.984+0000 7f34d7765740 1 -- 172.21.15.40:0/2274262895 --> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646150a17e0 con 0x56461514a680 2024-04-26T23:04:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.985+0000 7f34d1e00640 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x56461514a680 0x56461514aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.985+0000 7f34d1e00640 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x56461514a680 0x56461514aa90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6a134e848ac434da server_cookie=1fd3f7862aefc2ae in_seq=0 out_seq=0 2024-04-26T23:04:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.985+0000 7f34aea00640 10 client.6468 ms_handle_connect on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34aea00640 1 -- 172.21.15.40:0/2274262895 <== mds.1 v2:172.21.15.40:6832/2304583125 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5646150a17e0 con 0x56461514a680 2024-04-26T23:04:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34aea00640 10 client.6468 handle_command_reply: tid=0 2024-04-26T23:04:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34d7765740 1 client.6468 shutdown 2024-04-26T23:04:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34d7765740 1 -- 172.21.15.40:0/2274262895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 msgr2=0x564615149920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34d7765740 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 0x564615149920 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f34c800f4e0 tx=0x7f34c800f9a0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34d7765740 1 -- 172.21.15.40:0/2274262895 shutdown_connections 2024-04-26T23:04:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34d7765740 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 0x564615149920 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34d7765740 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564615177fb0 0x56461516f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34d7765740 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x56461514a680 0x56461514aa90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34d7765740 1 --2- 172.21.15.40:0/2274262895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56461515a710 0x56461515dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34d7765740 1 -- 172.21.15.40:0/2274262895 >> 172.21.15.40:0/2274262895 conn(0x5646151303f0 msgr2=0x564615174ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34aea00640 0 client.6468 ms_handle_reset on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34d7765740 1 -- 172.21.15.40:0/2274262895 shutdown_connections 2024-04-26T23:04:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.986+0000 7f34d7765740 1 -- 172.21.15.40:0/2274262895 wait complete. 2024-04-26T23:04:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:58.987+0000 7f34d7765740 20 client.6468 trim_cache size 0 max 16384 2024-04-26T23:04:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.006+0000 7f34d7765740 1 Processor -- start 2024-04-26T23:04:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.006+0000 7f34d7765740 1 -- start start 2024-04-26T23:04:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d7765740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461515a710 0x56461515e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d7765740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56461516d2a0 0x564615149890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d7765740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564615177fb0 0x564615178650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d7765740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc201e00 con 0x56461515a710 2024-04-26T23:04:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d7765740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc200ff0 con 0x564615177fb0 2024-04-26T23:04:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d7765740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f34cc2016d0 con 0x56461516d2a0 2024-04-26T23:04:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461515a710 0x56461515e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461515a710 0x56461515e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44330/0 (socket says 172.21.15.40:44330) 2024-04-26T23:04:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d1e00640 1 -- 172.21.15.40:0/2573982902 learned_addr learned my addr 172.21.15.40:0/2573982902 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d1e00640 1 -- 172.21.15.40:0/2573982902 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56461516d2a0 msgr2=0x564615149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d1e00640 1 --2- 172.21.15.40:0/2573982902 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56461516d2a0 0x564615149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d1e00640 1 -- 172.21.15.40:0/2573982902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564615177fb0 msgr2=0x564615178650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d1e00640 1 --2- 172.21.15.40:0/2573982902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564615177fb0 0x564615178650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.007+0000 7f34d1e00640 1 -- 172.21.15.40:0/2573982902 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564614d650f0 con 0x56461515a710 2024-04-26T23:04:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.008+0000 7f34d1e00640 1 --2- 172.21.15.40:0/2573982902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461515a710 0x56461515e4c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f34c0079dc0 tx=0x7f34c00aa790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25836dbd1927201 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.008+0000 7f34b9c00640 1 -- 172.21.15.40:0/2573982902 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c007a0c0 con 0x56461515a710 2024-04-26T23:04:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.008+0000 7f34b9c00640 1 -- 172.21.15.40:0/2573982902 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c0092070 con 0x56461515a710 2024-04-26T23:04:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.008+0000 7f34b9c00640 1 -- 172.21.15.40:0/2573982902 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c007e470 con 0x56461515a710 2024-04-26T23:04:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.009+0000 7f34d7765740 1 -- 172.21.15.40:0/2573982902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461515a710 msgr2=0x56461515e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.009+0000 7f34d7765740 1 --2- 172.21.15.40:0/2573982902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461515a710 0x56461515e4c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f34c0079dc0 tx=0x7f34c00aa790 comp rx=0 tx=0).stop 2024-04-26T23:04:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.009+0000 7f34d7765740 1 -- 172.21.15.40:0/2573982902 shutdown_connections 2024-04-26T23:04:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.009+0000 7f34d7765740 1 --2- 172.21.15.40:0/2573982902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564615177fb0 0x564615178650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.009+0000 7f34d7765740 1 --2- 172.21.15.40:0/2573982902 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56461516d2a0 0x564615149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.009+0000 7f34d7765740 1 --2- 172.21.15.40:0/2573982902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461515a710 0x56461515e4c0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.009+0000 7f34d7765740 1 -- 172.21.15.40:0/2573982902 >> 172.21.15.40:0/2573982902 conn(0x5646151303f0 msgr2=0x56461515da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.009+0000 7f34d7765740 1 -- 172.21.15.40:0/2573982902 shutdown_connections 2024-04-26T23:04:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.009+0000 7f34d7765740 1 -- 172.21.15.40:0/2573982902 wait complete. 2024-04-26T23:04:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.010+0000 7f34d7765740 1 -- start start 2024-04-26T23:04:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.010+0000 7f34d7765740 1 Processor -- start 2024-04-26T23:04:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d7765740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56461515a710 0x5646151ef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d7765740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 0x564615158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d7765740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564615177fb0 0x564615159230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d7765740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5646150a0fd0 con 0x56461516d2a0 2024-04-26T23:04:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d7765740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc0b4770 con 0x564615177fb0 2024-04-26T23:04:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d7765740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5646151efc40 con 0x56461515a710 2024-04-26T23:04:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d7765740 10 client.0 _ll_register_callbacks cb 0x7f34c8001540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:04:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 0x564615158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 0x564615158cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44336/0 (socket says 172.21.15.40:44336) 2024-04-26T23:04:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d1400640 1 -- 172.21.15.40:0/3087924596 learned_addr learned my addr 172.21.15.40:0/3087924596 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:04:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d1400640 1 -- 172.21.15.40:0/3087924596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56461515a710 msgr2=0x5646151ef0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d1400640 1 --2- 172.21.15.40:0/3087924596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56461515a710 0x5646151ef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d1400640 1 -- 172.21.15.40:0/3087924596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564615177fb0 msgr2=0x564615159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.011+0000 7f34d2800640 1 --2- 172.21.15.40:0/3087924596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564615177fb0 0x564615159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.012+0000 7f34d1400640 1 --2- 172.21.15.40:0/3087924596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564615177fb0 0x564615159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.012+0000 7f34d1400640 1 -- 172.21.15.40:0/3087924596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56461505ba30 con 0x56461516d2a0 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.012+0000 7f34d2800640 1 --2- 172.21.15.40:0/3087924596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564615177fb0 0x564615159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.012+0000 7f34d1400640 1 --2- 172.21.15.40:0/3087924596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 0x564615158cf0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f34bc022000 tx=0x7f34bc009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925dd9aed587db44 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.012+0000 7f34aea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.012+0000 7f34aea00640 1 -- 172.21.15.40:0/3087924596 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc003810 con 0x56461516d2a0 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.012+0000 7f34aea00640 1 -- 172.21.15.40:0/3087924596 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f34bc0039b0 con 0x56461516d2a0 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.012+0000 7f34d7765740 1 -- 172.21.15.40:0/3087924596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34c0092070 con 0x56461516d2a0 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.013+0000 7f34aea00640 1 -- 172.21.15.40:0/3087924596 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f34bc01c020 con 0x56461516d2a0 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.013+0000 7f34d7765740 10 client.6474 fetch_fsmap learned FSMap version 46 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.013+0000 7f34d7765740 1 -- 172.21.15.40:0/3087924596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x5646151ef9e0 con 0x56461516d2a0 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.013+0000 7f34aea00640 1 -- 172.21.15.40:0/3087924596 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c0092070 con 0x56461516d2a0 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.013+0000 7f34aea00640 1 -- 172.21.15.40:0/3087924596 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 46) ==== 3550+0+0 (secure 0 0 0) 0x7f34bc003bb0 con 0x56461516d2a0 2024-04-26T23:04:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.013+0000 7f34d7765740 10 client.6474 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T23:04:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.013+0000 7f34d7765740 10 client.6474 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.a 2024-04-26T23:04:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.013+0000 7f34d7765740 20 client.6474 populate_metadata read hostname 'smithi040' 2024-04-26T23:04:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.013+0000 7f34d7765740 1 --2- 172.21.15.40:0/3087924596 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x564615174350 0x5646151747a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:04:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.013+0000 7f34d7765740 4 client.6474 mds_command: new command op to 6027 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:04:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.013+0000 7f34d7765740 1 -- 172.21.15.40:0/3087924596 --> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56461509b650 con 0x564615174350 2024-04-26T23:04:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.013+0000 7f34d1e00640 1 --2- 172.21.15.40:0/3087924596 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x564615174350 0x5646151747a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:04:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34d1e00640 1 --2- 172.21.15.40:0/3087924596 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x564615174350 0x5646151747a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7aabd6e664a5d834 server_cookie=8e892b1664b555c1 in_seq=0 out_seq=0 2024-04-26T23:04:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34aea00640 10 client.6474 ms_handle_connect on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34aea00640 1 -- 172.21.15.40:0/3087924596 <== mds.1 v2:172.21.15.40:6832/2304583125 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x56461509b650 con 0x564615174350 2024-04-26T23:04:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34aea00640 10 client.6474 handle_command_reply: tid=0 2024-04-26T23:04:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34d7765740 1 client.6474 shutdown 2024-04-26T23:04:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34d7765740 1 -- 172.21.15.40:0/3087924596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 msgr2=0x564615158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34d7765740 1 --2- 172.21.15.40:0/3087924596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 0x564615158cf0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f34bc022000 tx=0x7f34bc009190 comp rx=0 tx=0).stop 2024-04-26T23:04:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34d7765740 1 -- 172.21.15.40:0/3087924596 shutdown_connections 2024-04-26T23:04:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34d7765740 1 --2- 172.21.15.40:0/3087924596 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x564615174350 0x5646151747a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34d7765740 1 --2- 172.21.15.40:0/3087924596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564615177fb0 0x564615159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34d7765740 1 --2- 172.21.15.40:0/3087924596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56461516d2a0 0x564615158cf0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34d7765740 1 --2- 172.21.15.40:0/3087924596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56461515a710 0x5646151ef0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34aea00640 0 client.6474 ms_handle_reset on v2:172.21.15.40:6832/2304583125 2024-04-26T23:04:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.014+0000 7f34d7765740 1 -- 172.21.15.40:0/3087924596 >> 172.21.15.40:0/3087924596 conn(0x5646151303f0 msgr2=0x56461512fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.015+0000 7f34d7765740 1 -- 172.21.15.40:0/3087924596 shutdown_connections 2024-04-26T23:04:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.015+0000 7f34d7765740 1 -- 172.21.15.40:0/3087924596 wait complete. 2024-04-26T23:04:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.015+0000 7f34d7765740 20 client.6474 trim_cache size 0 max 16384 2024-04-26T23:04:59.018 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:04:59.019 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-26T23:04:59.019 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:04:59.019 INFO:teuthology.orchestra.run.smithi040.stdout: "b07480c4-ced7-4267-91ee-1b8bac05bf74": { 2024-04-26T23:04:59.019 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:04:59.019 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "b07480c4-ced7-4267-91ee-1b8bac05bf74", 2024-04-26T23:04:59.019 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:04:59.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:59.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:04:59.019 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:04:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.015+0000 7f34d3200640 1 -- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f34b006c920 msgr2=0x7f34b006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.015+0000 7f34d3200640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f34b006c920 0x7f34b006ede0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009560 tx=0x7f34bc008040 comp rx=0 tx=0).stop 2024-04-26T23:04:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.015+0000 7f34d3200640 1 -- 172.21.15.40:0/627939381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34cc06dd40 msgr2=0x7f34cc140210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:04:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.015+0000 7f34d3200640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34cc06dd40 0x7f34cc140210 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f34c00097c0 tx=0x7f34c00046c0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.016+0000 7f34d3200640 1 -- 172.21.15.40:0/627939381 shutdown_connections 2024-04-26T23:04:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.016+0000 7f34d3200640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f34b006c920 0x7f34b006ede0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.016+0000 7f34d3200640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34cc13af60 0x7f34cc139540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.016+0000 7f34d3200640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34cc160a50 0x7f34cc0b54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.016+0000 7f34d3200640 1 --2- 172.21.15.40:0/627939381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34cc06dd40 0x7f34cc140210 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:04:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.016+0000 7f34d3200640 1 -- 172.21.15.40:0/627939381 >> 172.21.15.40:0/627939381 conn(0x7f34cc0b2720 msgr2=0x7f34cc15d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:04:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.016+0000 7f34d3200640 1 -- 172.21.15.40:0/627939381 shutdown_connections 2024-04-26T23:04:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:04:59.016+0000 7f34d3200640 1 -- 172.21.15.40:0/627939381 wait complete. 2024-04-26T23:05:00.007 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:05:00.007+0000 7f36ffc00640 -1 log_channel(cluster) log [ERR] : Health check update: 3 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-26T23:05:04.034 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:05:04.034 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.3:1 scrub status 2024-04-26T23:05:04.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.266+0000 7f458b200640 1 Processor -- start 2024-04-26T23:05:04.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.266+0000 7f458b200640 1 -- start start 2024-04-26T23:05:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.268+0000 7f458b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f458c0cfaf0 0x7f458c073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.268+0000 7f458b200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f458c161880 con 0x7f458c06dd40 2024-04-26T23:05:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.268+0000 7f458b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f458c0cff80 con 0x7f458c160a50 2024-04-26T23:05:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.268+0000 7f458b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f458c071ce0 con 0x7f458c0cfaf0 2024-04-26T23:05:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.268+0000 7f458a800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f458c06dd40 0x7f458c072730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35516/0 (socket says 172.21.15.40:35516) 2024-04-26T23:05:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.268+0000 7f458a800640 1 -- 172.21.15.40:0/662832040 learned_addr learned my addr 172.21.15.40:0/662832040 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.268+0000 7f4589400640 1 --2- 172.21.15.40:0/662832040 >> v2:172.21.15.103:3300/0 conn(0x7f458c0cfaf0 0x7f458c073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418624555 0 0) 0x7f458c161880 con 0x7f458c06dd40 2024-04-26T23:05:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4570003520 con 0x7f458c06dd40 2024-04-26T23:05:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2964732941 0 0) 0x7f4570003520 con 0x7f458c06dd40 2024-04-26T23:05:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 >> v2:172.21.15.103:3300/0 conn(0x7f458c0cfaf0 msgr2=0x7f458c073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 --2- 172.21.15.40:0/662832040 >> v2:172.21.15.103:3300/0 conn(0x7f458c0cfaf0 0x7f458c073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 >> v1:172.21.15.103:6789/0 conn(0x7f458c160a50 legacy=0x7f458c160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458c0d0260 con 0x7f458c06dd40 2024-04-26T23:05:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f45840019f0 con 0x7f458c06dd40 2024-04-26T23:05:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 >> v1:172.21.15.40:6789/0 conn(0x7f458c06dd40 legacy=0x7f458c072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 --2- 172.21.15.40:0/662832040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45700039b0 0x7f4570003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 --2- 172.21.15.40:0/662832040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4570004800 0x7f4570004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 --2- 172.21.15.40:0/662832040 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45700051c0 0x7f457000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f458c071ce0 con 0x7f4570004800 2024-04-26T23:05:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f457000c060 con 0x7f45700039b0 2024-04-26T23:05:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.269+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f457000c260 con 0x7f45700051c0 2024-04-26T23:05:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.270+0000 7f4589400640 1 --2- 172.21.15.40:0/662832040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4570004800 0x7f4570004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.270+0000 7f4589e00640 1 --2- 172.21.15.40:0/662832040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45700039b0 0x7f4570003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.270+0000 7f458a800640 1 --2- 172.21.15.40:0/662832040 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45700051c0 0x7f457000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.270+0000 7f4589400640 1 -- 172.21.15.40:0/662832040 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45700051c0 msgr2=0x7f457000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.270+0000 7f4589400640 1 --2- 172.21.15.40:0/662832040 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45700051c0 0x7f457000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.270+0000 7f4589400640 1 -- 172.21.15.40:0/662832040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45700039b0 msgr2=0x7f4570003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.270+0000 7f4589400640 1 --2- 172.21.15.40:0/662832040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45700039b0 0x7f4570003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.270+0000 7f4589400640 1 -- 172.21.15.40:0/662832040 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4570003520 con 0x7f4570004800 2024-04-26T23:05:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.270+0000 7f458a800640 1 -- 172.21.15.40:0/662832040 reap_dead start 2024-04-26T23:05:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.271+0000 7f4589400640 1 --2- 172.21.15.40:0/662832040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4570004800 0x7f4570004c80 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f4578009ac0 tx=0x7f4570005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e860dcecbd7a4a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.272+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4578036070 con 0x7f4570004800 2024-04-26T23:05:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.272+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4578031400 con 0x7f4570004800 2024-04-26T23:05:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.272+0000 7f4588a00640 1 -- 172.21.15.40:0/662832040 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4578009690 con 0x7f4570004800 2024-04-26T23:05:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.272+0000 7f458b200640 1 -- 172.21.15.40:0/662832040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4570004800 msgr2=0x7f4570004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.272+0000 7f458b200640 1 --2- 172.21.15.40:0/662832040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4570004800 0x7f4570004c80 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f4578009ac0 tx=0x7f4570005830 comp rx=0 tx=0).stop 2024-04-26T23:05:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.272+0000 7f458b200640 1 -- 172.21.15.40:0/662832040 shutdown_connections 2024-04-26T23:05:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.272+0000 7f458b200640 1 --2- 172.21.15.40:0/662832040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4570004800 0x7f4570004c80 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.272+0000 7f458b200640 1 -- 172.21.15.40:0/662832040 >> 172.21.15.40:0/662832040 conn(0x7f458c0b2720 msgr2=0x7f458c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.272+0000 7f458b200640 1 -- 172.21.15.40:0/662832040 shutdown_connections 2024-04-26T23:05:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.272+0000 7f458b200640 1 -- 172.21.15.40:0/662832040 wait complete. 2024-04-26T23:05:04.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.274+0000 7f458b200640 1 Processor -- start 2024-04-26T23:05:04.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.274+0000 7f458b200640 1 -- start start 2024-04-26T23:05:04.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.275+0000 7f458b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4570004800 0x7f458c15e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.275+0000 7f458b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f458c0b4950 0x7f458c15bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.275+0000 7f458b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f458c0ae8c0 0x7f458c0aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.275+0000 7f458b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f458c06e1f0 con 0x7f458c0b4950 2024-04-26T23:05:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.275+0000 7f458b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f458c0722a0 con 0x7f4570004800 2024-04-26T23:05:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.275+0000 7f458b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f458c1468f0 con 0x7f458c0ae8c0 2024-04-26T23:05:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.275+0000 7f4589400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f458c0b4950 0x7f458c15bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.276+0000 7f4589400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f458c0b4950 0x7f458c15bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44348/0 (socket says 172.21.15.40:44348) 2024-04-26T23:05:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.276+0000 7f4589400640 1 -- 172.21.15.40:0/1361892278 learned_addr learned my addr 172.21.15.40:0/1361892278 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.276+0000 7f4589e00640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4570004800 0x7f458c15e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.276+0000 7f4589400640 1 -- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f458c0ae8c0 msgr2=0x7f458c0aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.276+0000 7f458a800640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f458c0ae8c0 0x7f458c0aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.276+0000 7f4589400640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f458c0ae8c0 0x7f458c0aed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.276+0000 7f4589400640 1 -- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4570004800 msgr2=0x7f458c15e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.276+0000 7f4589400640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4570004800 0x7f458c15e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.276+0000 7f4589400640 1 -- 172.21.15.40:0/1361892278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458c0b4ec0 con 0x7f458c0b4950 2024-04-26T23:05:04.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.276+0000 7f4589e00640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4570004800 0x7f458c15e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:04.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.276+0000 7f4589400640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f458c0b4950 0x7f458c15bbf0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f4578009be0 tx=0x7f4578004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.277+0000 7f4582000640 1 -- 172.21.15.40:0/1361892278 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4578041030 con 0x7f458c0b4950 2024-04-26T23:05:04.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.277+0000 7f4582000640 1 -- 172.21.15.40:0/1361892278 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45780044c0 con 0x7f458c0b4950 2024-04-26T23:05:04.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.277+0000 7f4582000640 1 -- 172.21.15.40:0/1361892278 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4578036040 con 0x7f458c0b4950 2024-04-26T23:05:04.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.277+0000 7f458b200640 1 -- 172.21.15.40:0/1361892278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f458c0b51a0 con 0x7f458c0b4950 2024-04-26T23:05:04.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.277+0000 7f458b200640 1 -- 172.21.15.40:0/1361892278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f458c0b5760 con 0x7f458c0b4950 2024-04-26T23:05:04.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.279+0000 7f4582000640 1 -- 172.21.15.40:0/1361892278 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4578044070 con 0x7f458c0b4950 2024-04-26T23:05:04.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.280+0000 7f4582000640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f456c06c8f0 0x7f456c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.280+0000 7f4582000640 1 -- 172.21.15.40:0/1361892278 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f45780abb50 con 0x7f458c0b4950 2024-04-26T23:05:04.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.281+0000 7f4593e5e740 1 Processor -- start 2024-04-26T23:05:04.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.281+0000 7f4589e00640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f456c06c8f0 0x7f456c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.281+0000 7f4593e5e740 1 -- start start 2024-04-26T23:05:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.282+0000 7f4593e5e740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec4c710 0x55769ec4caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.282+0000 7f4593e5e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec69fb0 0x55769ec6a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.282+0000 7f4593e5e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec223f0 0x55769ec68670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.282+0000 7f4593e5e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55769eb32030 con 0x55769ec223f0 2024-04-26T23:05:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.282+0000 7f4593e5e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55769eb4d7f0 con 0x55769ec69fb0 2024-04-26T23:05:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.282+0000 7f4593e5e740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55769eb087e0 con 0x55769ec4c710 2024-04-26T23:05:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.282+0000 7f458a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec223f0 0x55769ec68670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.282+0000 7f458a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec223f0 0x55769ec68670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44360/0 (socket says 172.21.15.40:44360) 2024-04-26T23:05:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.282+0000 7f458a800640 1 -- 172.21.15.40:0/131724629 learned_addr learned my addr 172.21.15.40:0/131724629 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.282+0000 7f4589e00640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f456c06c8f0 0x7f456c06edb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f458c06b230 tx=0x7f457c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.283+0000 7f4589e00640 1 --2- 172.21.15.40:0/131724629 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec4c710 0x55769ec4caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.283+0000 7f4589400640 1 --2- 172.21.15.40:0/131724629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec69fb0 0x55769ec6a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.283+0000 7f458a800640 1 -- 172.21.15.40:0/131724629 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec4c710 msgr2=0x55769ec4caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.283+0000 7f458a800640 1 --2- 172.21.15.40:0/131724629 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec4c710 0x55769ec4caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.283+0000 7f458a800640 1 -- 172.21.15.40:0/131724629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec69fb0 msgr2=0x55769ec6a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.283+0000 7f458a800640 1 --2- 172.21.15.40:0/131724629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec69fb0 0x55769ec6a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.283+0000 7f458a800640 1 -- 172.21.15.40:0/131724629 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55769eb14050 con 0x55769ec223f0 2024-04-26T23:05:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.283+0000 7f4589400640 1 --2- 172.21.15.40:0/131724629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec69fb0 0x55769ec6a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.283+0000 7f458a800640 1 --2- 172.21.15.40:0/131724629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec223f0 0x55769ec68670 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f458c0cedd0 tx=0x7f4584005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0106d63ecbcdec2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.283+0000 7f4576a00640 1 -- 172.21.15.40:0/131724629 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584019070 con 0x55769ec223f0 2024-04-26T23:05:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.284+0000 7f4576a00640 1 -- 172.21.15.40:0/131724629 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4584007900 con 0x55769ec223f0 2024-04-26T23:05:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.284+0000 7f4576a00640 1 -- 172.21.15.40:0/131724629 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584010040 con 0x55769ec223f0 2024-04-26T23:05:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.284+0000 7f4593e5e740 1 -- 172.21.15.40:0/131724629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec223f0 msgr2=0x55769ec68670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.284+0000 7f4593e5e740 1 --2- 172.21.15.40:0/131724629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec223f0 0x55769ec68670 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f458c0cedd0 tx=0x7f4584005f40 comp rx=0 tx=0).stop 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.284+0000 7f4593e5e740 1 -- 172.21.15.40:0/131724629 shutdown_connections 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.284+0000 7f4593e5e740 1 --2- 172.21.15.40:0/131724629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec223f0 0x55769ec68670 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.284+0000 7f4593e5e740 1 --2- 172.21.15.40:0/131724629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec69fb0 0x55769ec6a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.284+0000 7f4593e5e740 1 --2- 172.21.15.40:0/131724629 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec4c710 0x55769ec4caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.284+0000 7f4593e5e740 1 -- 172.21.15.40:0/131724629 >> 172.21.15.40:0/131724629 conn(0x55769ec1ed20 msgr2=0x55769ec67520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.284+0000 7f4593e5e740 1 -- 172.21.15.40:0/131724629 shutdown_connections 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.284+0000 7f4593e5e740 1 -- 172.21.15.40:0/131724629 wait complete. 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.285+0000 7f4593e5e740 1 -- start start 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.285+0000 7f4593e5e740 1 Processor -- start 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.286+0000 7f4593e5e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec4c710 0x55769ec5f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.286+0000 7f4593e5e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 0x55769ec6aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.286+0000 7f4593e5e740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec60b90 0x55769ec68bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.286+0000 7f4593e5e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55769eb8a200 con 0x55769ec69fb0 2024-04-26T23:05:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f4593e5e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55769eb84ee0 con 0x55769ec4c710 2024-04-26T23:05:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f4593e5e740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55769eb91e70 con 0x55769ec60b90 2024-04-26T23:05:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f4593e5e740 10 client.0 _ll_register_callbacks cb 0x55769eb7b270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f4589400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 0x55769ec6aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f4589400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 0x55769ec6aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44366/0 (socket says 172.21.15.40:44366) 2024-04-26T23:05:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f4589400640 1 -- 172.21.15.40:0/2486983005 learned_addr learned my addr 172.21.15.40:0/2486983005 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f458a800640 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec60b90 0x55769ec68bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f4589e00640 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec4c710 0x55769ec5f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f4589400640 1 -- 172.21.15.40:0/2486983005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec60b90 msgr2=0x55769ec68bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f4589400640 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec60b90 0x55769ec68bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f4589400640 1 -- 172.21.15.40:0/2486983005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec4c710 msgr2=0x55769ec5f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f4589400640 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec4c710 0x55769ec5f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f4589400640 1 -- 172.21.15.40:0/2486983005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55769e9617d0 con 0x55769ec69fb0 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.287+0000 7f458a800640 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec60b90 0x55769ec68bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f4589400640 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 0x55769ec6aea0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f4578079840 tx=0x7f457807dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a4cbbe53e87d38 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f456b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f456b400640 1 -- 172.21.15.40:0/2486983005 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4578003e80 con 0x55769ec69fb0 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f456b400640 1 -- 172.21.15.40:0/2486983005 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f4578081f50 con 0x55769ec69fb0 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f4593e5e740 1 -- 172.21.15.40:0/2486983005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4584007900 con 0x55769ec69fb0 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f456b400640 1 -- 172.21.15.40:0/2486983005 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f4578082150 con 0x55769ec69fb0 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f4593e5e740 10 client.6486 fetch_fsmap learned FSMap version 46 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f4593e5e740 1 -- 172.21.15.40:0/2486983005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x55769e8973f0 con 0x55769ec69fb0 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f456b400640 1 -- 172.21.15.40:0/2486983005 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4584007900 con 0x55769ec69fb0 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f456b400640 1 -- 172.21.15.40:0/2486983005 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 46) ==== 3550+0+0 (secure 0 0 0) 0x7f457807ab30 con 0x55769ec69fb0 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f4593e5e740 10 client.6486 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T23:05:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f4593e5e740 10 client.6486 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.a 2024-04-26T23:05:04.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f4593e5e740 20 client.6486 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:04.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.288+0000 7f4593e5e740 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55769ec49a80 0x55769ec76ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.289+0000 7f4593e5e740 4 client.6486 mds_command: new command op to 6027 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:05:04.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.289+0000 7f4593e5e740 1 -- 172.21.15.40:0/2486983005 --> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55769eb937e0 con 0x55769ec49a80 2024-04-26T23:05:04.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.289+0000 7f4589e00640 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55769ec49a80 0x55769ec76ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.289+0000 7f4589e00640 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55769ec49a80 0x55769ec76ed0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=776a32e55cbfe993 server_cookie=e361b4eeb9d813d0 in_seq=0 out_seq=0 2024-04-26T23:05:04.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.289+0000 7f456b400640 10 client.6486 ms_handle_connect on v2:172.21.15.40:6832/2304583125 2024-04-26T23:05:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.290+0000 7f456b400640 1 -- 172.21.15.40:0/2486983005 <== mds.1 v2:172.21.15.40:6832/2304583125 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55769eb937e0 con 0x55769ec49a80 2024-04-26T23:05:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.290+0000 7f456b400640 10 client.6486 handle_command_reply: tid=0 2024-04-26T23:05:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.290+0000 7f4593e5e740 1 client.6486 shutdown 2024-04-26T23:05:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.290+0000 7f4593e5e740 1 -- 172.21.15.40:0/2486983005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 msgr2=0x55769ec6aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.290+0000 7f4593e5e740 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 0x55769ec6aea0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f4578079840 tx=0x7f457807dc70 comp rx=0 tx=0).stop 2024-04-26T23:05:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.291+0000 7f4593e5e740 1 -- 172.21.15.40:0/2486983005 shutdown_connections 2024-04-26T23:05:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.291+0000 7f4593e5e740 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec60b90 0x55769ec68bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.291+0000 7f4593e5e740 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 0x55769ec6aea0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.291+0000 7f4593e5e740 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55769ec49a80 0x55769ec76ed0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.291+0000 7f4593e5e740 1 --2- 172.21.15.40:0/2486983005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec4c710 0x55769ec5f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.291+0000 7f4593e5e740 1 -- 172.21.15.40:0/2486983005 >> 172.21.15.40:0/2486983005 conn(0x55769ec223f0 msgr2=0x55769ec1c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.291+0000 7f456b400640 0 client.6486 ms_handle_reset on v2:172.21.15.40:6832/2304583125 2024-04-26T23:05:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.291+0000 7f4593e5e740 1 -- 172.21.15.40:0/2486983005 shutdown_connections 2024-04-26T23:05:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.291+0000 7f4593e5e740 1 -- 172.21.15.40:0/2486983005 wait complete. 2024-04-26T23:05:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.291+0000 7f4593e5e740 20 client.6486 trim_cache size 0 max 16384 2024-04-26T23:05:04.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.310+0000 7f4593e5e740 1 Processor -- start 2024-04-26T23:05:04.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.310+0000 7f4593e5e740 1 -- start start 2024-04-26T23:05:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.311+0000 7f4593e5e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec4c710 0x55769ec1d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.311+0000 7f4593e5e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec60b90 0x55769ec6ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.311+0000 7f4593e5e740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec69fb0 0x55769ec4f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.311+0000 7f4593e5e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f458c202570 con 0x55769ec4c710 2024-04-26T23:05:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.311+0000 7f4593e5e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f458c201710 con 0x55769ec60b90 2024-04-26T23:05:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.311+0000 7f4593e5e740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f458c201e40 con 0x55769ec69fb0 2024-04-26T23:05:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.312+0000 7f4589e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec4c710 0x55769ec1d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.312+0000 7f4589e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec4c710 0x55769ec1d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44380/0 (socket says 172.21.15.40:44380) 2024-04-26T23:05:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.312+0000 7f4589e00640 1 -- 172.21.15.40:0/3263704779 learned_addr learned my addr 172.21.15.40:0/3263704779 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.312+0000 7f458a800640 1 --2- 172.21.15.40:0/3263704779 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec69fb0 0x55769ec4f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.312+0000 7f4589e00640 1 -- 172.21.15.40:0/3263704779 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec69fb0 msgr2=0x55769ec4f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.312+0000 7f4589400640 1 --2- 172.21.15.40:0/3263704779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec60b90 0x55769ec6ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.312+0000 7f4589e00640 1 --2- 172.21.15.40:0/3263704779 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec69fb0 0x55769ec4f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.312+0000 7f4589e00640 1 -- 172.21.15.40:0/3263704779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec60b90 msgr2=0x55769ec6ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.312+0000 7f4589e00640 1 --2- 172.21.15.40:0/3263704779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec60b90 0x55769ec6ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.312+0000 7f4589e00640 1 -- 172.21.15.40:0/3263704779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55769e8570f0 con 0x55769ec4c710 2024-04-26T23:05:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.312+0000 7f458a800640 1 --2- 172.21.15.40:0/3263704779 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec69fb0 0x55769ec4f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.312+0000 7f4589e00640 1 --2- 172.21.15.40:0/3263704779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec4c710 0x55769ec1d7d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x55769ec49fa0 tx=0x7f457c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e6f571a45fc544 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.313+0000 7f4576000640 1 -- 172.21.15.40:0/3263704779 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c027210 con 0x55769ec4c710 2024-04-26T23:05:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.313+0000 7f4576000640 1 -- 172.21.15.40:0/3263704779 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f457c02d070 con 0x55769ec4c710 2024-04-26T23:05:04.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.313+0000 7f4576000640 1 -- 172.21.15.40:0/3263704779 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c0275b0 con 0x55769ec4c710 2024-04-26T23:05:04.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.313+0000 7f4593e5e740 1 -- 172.21.15.40:0/3263704779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec4c710 msgr2=0x55769ec1d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.313+0000 7f4593e5e740 1 --2- 172.21.15.40:0/3263704779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec4c710 0x55769ec1d7d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x55769ec49fa0 tx=0x7f457c012000 comp rx=0 tx=0).stop 2024-04-26T23:05:04.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.313+0000 7f4593e5e740 1 -- 172.21.15.40:0/3263704779 shutdown_connections 2024-04-26T23:05:04.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.313+0000 7f4593e5e740 1 --2- 172.21.15.40:0/3263704779 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec69fb0 0x55769ec4f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.314+0000 7f4593e5e740 1 --2- 172.21.15.40:0/3263704779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec60b90 0x55769ec6ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.314+0000 7f4593e5e740 1 --2- 172.21.15.40:0/3263704779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec4c710 0x55769ec1d7d0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.314+0000 7f4593e5e740 1 -- 172.21.15.40:0/3263704779 >> 172.21.15.40:0/3263704779 conn(0x55769ec223f0 msgr2=0x55769ec18080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.314+0000 7f4593e5e740 1 -- 172.21.15.40:0/3263704779 shutdown_connections 2024-04-26T23:05:04.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.314+0000 7f4593e5e740 1 -- 172.21.15.40:0/3263704779 wait complete. 2024-04-26T23:05:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.314+0000 7f4593e5e740 1 -- start start 2024-04-26T23:05:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.315+0000 7f4593e5e740 1 Processor -- start 2024-04-26T23:05:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f4593e5e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec4c710 0x55769ece05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f4593e5e740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec60b90 0x55769ec8a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f4593e5e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 0x55769ec5ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f4593e5e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55769eb92fd0 con 0x55769ec69fb0 2024-04-26T23:05:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f4593e5e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f458c143d80 con 0x55769ec4c710 2024-04-26T23:05:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f4593e5e740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55769ec1f8c0 con 0x55769ec60b90 2024-04-26T23:05:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f4593e5e740 10 client.0 _ll_register_callbacks cb 0x7f4578079680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f458a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 0x55769ec5ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f458a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 0x55769ec5ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44392/0 (socket says 172.21.15.40:44392) 2024-04-26T23:05:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f458a800640 1 -- 172.21.15.40:0/1137546780 learned_addr learned my addr 172.21.15.40:0/1137546780 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f4589e00640 1 --2- 172.21.15.40:0/1137546780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec4c710 0x55769ece05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f458a800640 1 -- 172.21.15.40:0/1137546780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec60b90 msgr2=0x55769ec8a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f458a800640 1 --2- 172.21.15.40:0/1137546780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec60b90 0x55769ec8a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.316+0000 7f458a800640 1 -- 172.21.15.40:0/1137546780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec4c710 msgr2=0x55769ece05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.317+0000 7f458a800640 1 --2- 172.21.15.40:0/1137546780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec4c710 0x55769ece05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.317+0000 7f458a800640 1 -- 172.21.15.40:0/1137546780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55769eb4da30 con 0x55769ec69fb0 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.317+0000 7f4589e00640 1 --2- 172.21.15.40:0/1137546780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec4c710 0x55769ece05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.317+0000 7f458a800640 1 --2- 172.21.15.40:0/1137546780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 0x55769ec5ceb0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f458c06fd10 tx=0x7f4584001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b281579ac93a6819 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.317+0000 7f456b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.317+0000 7f456b400640 1 -- 172.21.15.40:0/1137546780 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584024070 con 0x55769ec69fb0 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.317+0000 7f456b400640 1 -- 172.21.15.40:0/1137546780 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f4584014630 con 0x55769ec69fb0 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.317+0000 7f4593e5e740 1 -- 172.21.15.40:0/1137546780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f457c02d070 con 0x55769ec69fb0 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.317+0000 7f456b400640 1 -- 172.21.15.40:0/1137546780 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f458402f050 con 0x55769ec69fb0 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.317+0000 7f4593e5e740 10 client.6492 fetch_fsmap learned FSMap version 46 2024-04-26T23:05:04.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.317+0000 7f4593e5e740 1 -- 172.21.15.40:0/1137546780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x55769ec661e0 con 0x55769ec69fb0 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.318+0000 7f456b400640 1 -- 172.21.15.40:0/1137546780 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f457c02d070 con 0x55769ec69fb0 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.318+0000 7f456b400640 1 -- 172.21.15.40:0/1137546780 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 46) ==== 3550+0+0 (secure 0 0 0) 0x7f4584032020 con 0x55769ec69fb0 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.318+0000 7f4593e5e740 10 client.6492 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.318+0000 7f4593e5e740 10 client.6492 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.a 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.318+0000 7f4593e5e740 20 client.6492 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.318+0000 7f4593e5e740 1 --2- 172.21.15.40:0/1137546780 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55769ec5d3f0 0x55769ec67520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.318+0000 7f4593e5e740 4 client.6492 mds_command: new command op to 6027 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.318+0000 7f4593e5e740 1 -- 172.21.15.40:0/1137546780 --> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55769ec7a240 con 0x55769ec5d3f0 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.318+0000 7f4589e00640 1 --2- 172.21.15.40:0/1137546780 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55769ec5d3f0 0x55769ec67520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f4589e00640 1 --2- 172.21.15.40:0/1137546780 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55769ec5d3f0 0x55769ec67520 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=137a24a0220ccbe1 server_cookie=69a1e346601a7a7a in_seq=0 out_seq=0 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f456b400640 10 client.6492 ms_handle_connect on v2:172.21.15.40:6832/2304583125 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f456b400640 1 -- 172.21.15.40:0/1137546780 <== mds.1 v2:172.21.15.40:6832/2304583125 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55769ec7a240 con 0x55769ec5d3f0 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f456b400640 10 client.6492 handle_command_reply: tid=0 2024-04-26T23:05:04.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f4593e5e740 1 client.6492 shutdown 2024-04-26T23:05:04.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f4593e5e740 1 -- 172.21.15.40:0/1137546780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 msgr2=0x55769ec5ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f4593e5e740 1 --2- 172.21.15.40:0/1137546780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 0x55769ec5ceb0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f458c06fd10 tx=0x7f4584001dc0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f4593e5e740 1 -- 172.21.15.40:0/1137546780 shutdown_connections 2024-04-26T23:05:04.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f4593e5e740 1 --2- 172.21.15.40:0/1137546780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55769ec69fb0 0x55769ec5ceb0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f4593e5e740 1 --2- 172.21.15.40:0/1137546780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55769ec60b90 0x55769ec8a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f4593e5e740 1 --2- 172.21.15.40:0/1137546780 >> [v2:172.21.15.40:6832/2304583125,v1:172.21.15.40:6834/2304583125] conn(0x55769ec5d3f0 0x55769ec67520 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f4593e5e740 1 --2- 172.21.15.40:0/1137546780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55769ec4c710 0x55769ece05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f4593e5e740 1 -- 172.21.15.40:0/1137546780 >> 172.21.15.40:0/1137546780 conn(0x55769ec223f0 msgr2=0x55769ec18080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.319+0000 7f456b400640 0 client.6492 ms_handle_reset on v2:172.21.15.40:6832/2304583125 2024-04-26T23:05:04.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.320+0000 7f4593e5e740 1 -- 172.21.15.40:0/1137546780 shutdown_connections 2024-04-26T23:05:04.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.320+0000 7f4593e5e740 1 -- 172.21.15.40:0/1137546780 wait complete. 2024-04-26T23:05:04.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.320+0000 7f4593e5e740 20 client.6492 trim_cache size 0 max 16384 2024-04-26T23:05:04.325 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:05:04.325 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-04-26T23:05:04.325 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-04-26T23:05:04.325 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:05:04.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.320+0000 7f458b200640 1 -- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f456c06c8f0 msgr2=0x7f456c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.320+0000 7f458b200640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f456c06c8f0 0x7f456c06edb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f458c06b230 tx=0x7f457c009040 comp rx=0 tx=0).stop 2024-04-26T23:05:04.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.320+0000 7f458b200640 1 -- 172.21.15.40:0/1361892278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f458c0b4950 msgr2=0x7f458c15bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.320+0000 7f458b200640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f458c0b4950 0x7f458c15bbf0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f4578009be0 tx=0x7f4578004730 comp rx=0 tx=0).stop 2024-04-26T23:05:04.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.321+0000 7f458b200640 1 -- 172.21.15.40:0/1361892278 shutdown_connections 2024-04-26T23:05:04.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.321+0000 7f458b200640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f456c06c8f0 0x7f456c06edb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.321+0000 7f458b200640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f458c0ae8c0 0x7f458c0aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.321+0000 7f458b200640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f458c0b4950 0x7f458c15bbf0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.321+0000 7f458b200640 1 --2- 172.21.15.40:0/1361892278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4570004800 0x7f458c15e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.321+0000 7f458b200640 1 -- 172.21.15.40:0/1361892278 >> 172.21.15.40:0/1361892278 conn(0x7f458c0b2720 msgr2=0x7f458c141190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.321+0000 7f458b200640 1 -- 172.21.15.40:0/1361892278 shutdown_connections 2024-04-26T23:05:04.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.321+0000 7f458b200640 1 -- 172.21.15.40:0/1361892278 wait complete. 2024-04-26T23:05:04.341 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:05:04.342 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:05:04.342 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.3:2 scrub status 2024-04-26T23:05:04.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.549+0000 7f2d35800640 1 Processor -- start 2024-04-26T23:05:04.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.550+0000 7f2d35800640 1 -- start start 2024-04-26T23:05:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.551+0000 7f2d35800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2d3006f790 0x7f2d3006fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.551+0000 7f2d35800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d30072c50 0x7f2d300730a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.551+0000 7f2d35800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d30156840 con 0x7f2d30156460 2024-04-26T23:05:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.551+0000 7f2d35800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2d300cbf20 con 0x7f2d3006f790 2024-04-26T23:05:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.551+0000 7f2d35800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2d300cc200 con 0x7f2d30072c50 2024-04-26T23:05:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.552+0000 7f2d2e000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2d3006f790 0x7f2d3006fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.552+0000 7f2d2f400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d30072c50 0x7f2d300730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.552+0000 7f2d2f400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d30072c50 0x7f2d300730a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:57430/0 (socket says 172.21.15.40:57430) 2024-04-26T23:05:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.552+0000 7f2d2f400640 1 -- 172.21.15.40:0/2813930114 learned_addr learned my addr 172.21.15.40:0/2813930114 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.553+0000 7f2d2f400640 1 -- 172.21.15.40:0/2813930114 >> v2:172.21.15.103:3300/0 conn(0x7f2d3006f790 msgr2=0x7f2d3006fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.553+0000 7f2d2f400640 1 --2- 172.21.15.40:0/2813930114 >> v2:172.21.15.103:3300/0 conn(0x7f2d3006f790 0x7f2d3006fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.553+0000 7f2d2f400640 1 -- 172.21.15.40:0/2813930114 >> v1:172.21.15.103:6789/0 conn(0x7f2d30156460 legacy=0x7f2d300cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.553+0000 7f2d2f400640 1 -- 172.21.15.40:0/2813930114 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d300cc4e0 con 0x7f2d30072c50 2024-04-26T23:05:04.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.555+0000 7f2d2f400640 1 --2- 172.21.15.40:0/2813930114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d30072c50 0x7f2d300730a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800d150 tx=0x7f2d2802c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f30dd3365519c9f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.555+0000 7f2d2d600640 1 -- 172.21.15.40:0/2813930114 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d28009600 con 0x7f2d30072c50 2024-04-26T23:05:04.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.556+0000 7f2d2d600640 1 -- 172.21.15.40:0/2813930114 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d28037070 con 0x7f2d30072c50 2024-04-26T23:05:04.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.556+0000 7f2d2d600640 1 -- 172.21.15.40:0/2813930114 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d280099c0 con 0x7f2d30072c50 2024-04-26T23:05:04.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.556+0000 7f2d35800640 1 -- 172.21.15.40:0/2813930114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d30072c50 msgr2=0x7f2d300730a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.556+0000 7f2d35800640 1 --2- 172.21.15.40:0/2813930114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d30072c50 0x7f2d300730a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800d150 tx=0x7f2d2802c930 comp rx=0 tx=0).stop 2024-04-26T23:05:04.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.556+0000 7f2d35800640 1 -- 172.21.15.40:0/2813930114 shutdown_connections 2024-04-26T23:05:04.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.556+0000 7f2d35800640 1 --2- 172.21.15.40:0/2813930114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d30072c50 0x7f2d300730a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.556+0000 7f2d35800640 1 --2- 172.21.15.40:0/2813930114 >> v2:172.21.15.103:3300/0 conn(0x7f2d3006f790 0x7f2d3006fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.556+0000 7f2d35800640 1 -- 172.21.15.40:0/2813930114 >> 172.21.15.40:0/2813930114 conn(0x7f2d300d0be0 msgr2=0x7f2d300d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.556+0000 7f2d35800640 1 -- 172.21.15.40:0/2813930114 shutdown_connections 2024-04-26T23:05:04.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.557+0000 7f2d35800640 1 -- 172.21.15.40:0/2813930114 wait complete. 2024-04-26T23:05:04.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.559+0000 7f2d35800640 1 Processor -- start 2024-04-26T23:05:04.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.559+0000 7f2d35800640 1 -- start start 2024-04-26T23:05:04.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.560+0000 7f2d35800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3006f790 0x7f2d300b5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.560+0000 7f2d35800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d30072c50 0x7f2d301f4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.560+0000 7f2d35800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d30156460 0x7f2d301526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.560+0000 7f2d35800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2d300cbbe0 con 0x7f2d30072c50 2024-04-26T23:05:04.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.560+0000 7f2d35800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d30072700 con 0x7f2d30156460 2024-04-26T23:05:04.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.560+0000 7f2d35800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3006e260 con 0x7f2d3006f790 2024-04-26T23:05:04.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.561+0000 7f2d2e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d30072c50 0x7f2d301f4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.561+0000 7f2d2e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d30072c50 0x7f2d301f4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44404/0 (socket says 172.21.15.40:44404) 2024-04-26T23:05:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.561+0000 7f2d2e000640 1 -- 172.21.15.40:0/2483480260 learned_addr learned my addr 172.21.15.40:0/2483480260 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.561+0000 7f2d2f400640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d30156460 0x7f2d301526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.561+0000 7f2d2ea00640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3006f790 0x7f2d300b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.561+0000 7f2d2e000640 1 -- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3006f790 msgr2=0x7f2d300b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.561+0000 7f2d2e000640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3006f790 0x7f2d300b5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.561+0000 7f2d2e000640 1 -- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d30156460 msgr2=0x7f2d301526b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.561+0000 7f2d2e000640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d30156460 0x7f2d301526b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.561+0000 7f2d2e000640 1 -- 172.21.15.40:0/2483480260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d300b5b00 con 0x7f2d30072c50 2024-04-26T23:05:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.562+0000 7f2d2e000640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d30072c50 0x7f2d301f4e40 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00e9b0 tx=0x7f2d1c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.562+0000 7f2d26a00640 1 -- 172.21.15.40:0/2483480260 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c017070 con 0x7f2d30072c50 2024-04-26T23:05:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.562+0000 7f2d26a00640 1 -- 172.21.15.40:0/2483480260 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d1c00c9a0 con 0x7f2d30072c50 2024-04-26T23:05:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.562+0000 7f2d35800640 1 -- 172.21.15.40:0/2483480260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d300b5de0 con 0x7f2d30072c50 2024-04-26T23:05:04.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.562+0000 7f2d26a00640 1 -- 172.21.15.40:0/2483480260 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c00cd90 con 0x7f2d30072c50 2024-04-26T23:05:04.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.562+0000 7f2d35800640 1 -- 172.21.15.40:0/2483480260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d300b6680 con 0x7f2d30072c50 2024-04-26T23:05:04.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.565+0000 7f2d26a00640 1 -- 172.21.15.40:0/2483480260 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d1c01a070 con 0x7f2d30072c50 2024-04-26T23:05:04.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.566+0000 7f2d26a00640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d1006c8f0 0x7f2d1006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.566+0000 7f2d26a00640 1 -- 172.21.15.40:0/2483480260 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f2d1c08c150 con 0x7f2d30072c50 2024-04-26T23:05:04.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.566+0000 7f2d385b5740 1 Processor -- start 2024-04-26T23:05:04.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.566+0000 7f2d385b5740 1 -- start start 2024-04-26T23:05:04.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.566+0000 7f2d2ea00640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d1006c8f0 0x7f2d1006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.567+0000 7f2d385b5740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 0x556446b67920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.567+0000 7f2d385b5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b85330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.567+0000 7f2d385b5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b3d220 0x556446b834a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.567+0000 7f2d385b5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556446a4d030 con 0x556446b84de0 2024-04-26T23:05:04.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.567+0000 7f2d385b5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556446a687f0 con 0x556446b3d220 2024-04-26T23:05:04.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.567+0000 7f2d385b5740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556446a237e0 con 0x556446b67540 2024-04-26T23:05:04.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.568+0000 7f2d2e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b85330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.568+0000 7f2d2ea00640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d1006c8f0 0x7f2d1006edb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2d20007fd0 tx=0x7f2d20009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.568+0000 7f2d2e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b85330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44412/0 (socket says 172.21.15.40:44412) 2024-04-26T23:05:04.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.568+0000 7f2d2e000640 1 -- 172.21.15.40:0/3598475370 learned_addr learned my addr 172.21.15.40:0/3598475370 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.568+0000 7f2d2ea00640 1 --2- 172.21.15.40:0/3598475370 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 0x556446b67920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.568+0000 7f2d2f400640 1 --2- 172.21.15.40:0/3598475370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b3d220 0x556446b834a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.568+0000 7f2d2e000640 1 -- 172.21.15.40:0/3598475370 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 msgr2=0x556446b67920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.568+0000 7f2d2e000640 1 --2- 172.21.15.40:0/3598475370 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 0x556446b67920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.568+0000 7f2d2e000640 1 -- 172.21.15.40:0/3598475370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b3d220 msgr2=0x556446b834a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.568+0000 7f2d2e000640 1 --2- 172.21.15.40:0/3598475370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b3d220 0x556446b834a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.568+0000 7f2d2e000640 1 -- 172.21.15.40:0/3598475370 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556446a2f050 con 0x556446b84de0 2024-04-26T23:05:04.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.568+0000 7f2d2e000640 1 --2- 172.21.15.40:0/3598475370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b85330 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c059c40 tx=0x7f2d1c059cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de388d122c8cb348 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.569+0000 7f2d1b400640 1 -- 172.21.15.40:0/3598475370 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c021040 con 0x556446b84de0 2024-04-26T23:05:04.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.569+0000 7f2d1b400640 1 -- 172.21.15.40:0/3598475370 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d1c0721c0 con 0x556446b84de0 2024-04-26T23:05:04.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.569+0000 7f2d1b400640 1 -- 172.21.15.40:0/3598475370 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c056320 con 0x556446b84de0 2024-04-26T23:05:04.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.569+0000 7f2d385b5740 1 -- 172.21.15.40:0/3598475370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 msgr2=0x556446b85330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.569+0000 7f2d385b5740 1 --2- 172.21.15.40:0/3598475370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b85330 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c059c40 tx=0x7f2d1c059cb0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.569+0000 7f2d385b5740 1 -- 172.21.15.40:0/3598475370 shutdown_connections 2024-04-26T23:05:04.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.569+0000 7f2d385b5740 1 --2- 172.21.15.40:0/3598475370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b3d220 0x556446b834a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.570+0000 7f2d385b5740 1 --2- 172.21.15.40:0/3598475370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b85330 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.570+0000 7f2d385b5740 1 --2- 172.21.15.40:0/3598475370 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 0x556446b67920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.570+0000 7f2d385b5740 1 -- 172.21.15.40:0/3598475370 >> 172.21.15.40:0/3598475370 conn(0x556446b39b50 msgr2=0x556446b82350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.570+0000 7f2d385b5740 1 -- 172.21.15.40:0/3598475370 shutdown_connections 2024-04-26T23:05:04.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.570+0000 7f2d385b5740 1 -- 172.21.15.40:0/3598475370 wait complete. 2024-04-26T23:05:04.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.570+0000 7f2d385b5740 1 -- start start 2024-04-26T23:05:04.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.571+0000 7f2d385b5740 1 Processor -- start 2024-04-26T23:05:04.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.571+0000 7f2d385b5740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 0x556446b56be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d385b5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b7b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d385b5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b79410 0x556446b574b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d385b5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556446aa5200 con 0x556446b84de0 2024-04-26T23:05:04.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d385b5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556446a9fee0 con 0x556446b79410 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d385b5740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556446aace70 con 0x556446b67540 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d385b5740 10 client.0 _ll_register_callbacks cb 0x556446a96270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d2e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b7b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d2e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b7b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44426/0 (socket says 172.21.15.40:44426) 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d2e000640 1 -- 172.21.15.40:0/1822205942 learned_addr learned my addr 172.21.15.40:0/1822205942 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d2ea00640 1 --2- 172.21.15.40:0/1822205942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 0x556446b56be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d2e000640 1 -- 172.21.15.40:0/1822205942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 msgr2=0x556446b56be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d2e000640 1 --2- 172.21.15.40:0/1822205942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 0x556446b56be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d2e000640 1 -- 172.21.15.40:0/1822205942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b79410 msgr2=0x556446b574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d2e000640 1 --2- 172.21.15.40:0/1822205942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b79410 0x556446b574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.572+0000 7f2d2e000640 1 -- 172.21.15.40:0/1822205942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55644687c7d0 con 0x556446b84de0 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.573+0000 7f2d2e000640 1 --2- 172.21.15.40:0/1822205942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b7b020 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0599b0 tx=0x7f2d1c00efa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60068e661cc5063a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.573+0000 7f2d0fe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:04.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.573+0000 7f2d0fe00640 1 -- 172.21.15.40:0/1822205942 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c087020 con 0x556446b84de0 2024-04-26T23:05:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.573+0000 7f2d385b5740 1 -- 172.21.15.40:0/1822205942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d1c0721c0 con 0x556446b84de0 2024-04-26T23:05:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.573+0000 7f2d0fe00640 1 -- 172.21.15.40:0/1822205942 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f2d1c059300 con 0x556446b84de0 2024-04-26T23:05:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.573+0000 7f2d0fe00640 1 -- 172.21.15.40:0/1822205942 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f2d1c079050 con 0x556446b84de0 2024-04-26T23:05:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.573+0000 7f2d385b5740 10 client.6501 fetch_fsmap learned FSMap version 46 2024-04-26T23:05:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.573+0000 7f2d385b5740 1 -- 172.21.15.40:0/1822205942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x5564467b23f0 con 0x556446b84de0 2024-04-26T23:05:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.573+0000 7f2d0fe00640 1 -- 172.21.15.40:0/1822205942 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d1c0721c0 con 0x556446b84de0 2024-04-26T23:05:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.573+0000 7f2d0fe00640 1 -- 172.21.15.40:0/1822205942 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 46) ==== 3550+0+0 (secure 0 0 0) 0x7f2d1c060260 con 0x556446b84de0 2024-04-26T23:05:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.574+0000 7f2d385b5740 10 client.6501 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T23:05:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.574+0000 7f2d385b5740 10 client.6501 resolve_mds: resolved 3:2 to role '3:2' aka daemon mds.b 2024-04-26T23:05:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.574+0000 7f2d385b5740 20 client.6501 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.574+0000 7f2d385b5740 1 --2- 172.21.15.40:0/1822205942 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x556446b79830 0x556446b36c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.574+0000 7f2d385b5740 4 client.6501 mds_command: new command op to 5284 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:05:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.574+0000 7f2d385b5740 1 -- 172.21.15.40:0/1822205942 --> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556446aae7e0 con 0x556446b79830 2024-04-26T23:05:04.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.574+0000 7f2d2ea00640 1 --2- 172.21.15.40:0/1822205942 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x556446b79830 0x556446b36c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.575+0000 7f2d2ea00640 1 --2- 172.21.15.40:0/1822205942 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x556446b79830 0x556446b36c80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b2ca738547bf603 server_cookie=afbe2f041f0b649d in_seq=0 out_seq=0 2024-04-26T23:05:04.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.575+0000 7f2d0fe00640 10 client.6501 ms_handle_connect on v2:172.21.15.103:6836/670995452 2024-04-26T23:05:04.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.577+0000 7f2d0fe00640 1 -- 172.21.15.40:0/1822205942 <== mds.2 v2:172.21.15.103:6836/670995452 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556446aae7e0 con 0x556446b79830 2024-04-26T23:05:04.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.577+0000 7f2d0fe00640 10 client.6501 handle_command_reply: tid=0 2024-04-26T23:05:04.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.577+0000 7f2d385b5740 1 client.6501 shutdown 2024-04-26T23:05:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.578+0000 7f2d385b5740 1 -- 172.21.15.40:0/1822205942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 msgr2=0x556446b7b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.578+0000 7f2d385b5740 1 --2- 172.21.15.40:0/1822205942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b7b020 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0599b0 tx=0x7f2d1c00efa0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.578+0000 7f2d385b5740 1 -- 172.21.15.40:0/1822205942 shutdown_connections 2024-04-26T23:05:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.578+0000 7f2d385b5740 1 --2- 172.21.15.40:0/1822205942 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x556446b79830 0x556446b36c80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.578+0000 7f2d385b5740 1 --2- 172.21.15.40:0/1822205942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b79410 0x556446b574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.578+0000 7f2d385b5740 1 --2- 172.21.15.40:0/1822205942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b7b020 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.578+0000 7f2d385b5740 1 --2- 172.21.15.40:0/1822205942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 0x556446b56be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.578+0000 7f2d0fe00640 0 client.6501 ms_handle_reset on v2:172.21.15.103:6836/670995452 2024-04-26T23:05:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.578+0000 7f2d385b5740 1 -- 172.21.15.40:0/1822205942 >> 172.21.15.40:0/1822205942 conn(0x556446b3d220 msgr2=0x556446b3c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.578+0000 7f2d385b5740 1 -- 172.21.15.40:0/1822205942 shutdown_connections 2024-04-26T23:05:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.578+0000 7f2d385b5740 1 -- 172.21.15.40:0/1822205942 wait complete. 2024-04-26T23:05:04.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.578+0000 7f2d385b5740 20 client.6501 trim_cache size 0 max 16384 2024-04-26T23:05:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.597+0000 7f2d385b5740 1 Processor -- start 2024-04-26T23:05:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.598+0000 7f2d385b5740 1 -- start start 2024-04-26T23:05:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.598+0000 7f2d385b5740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 0x556446ba5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.598+0000 7f2d385b5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b79410 0x556446b64540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.598+0000 7f2d385b5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b84de0 0x556446b64a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.598+0000 7f2d385b5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2d30202030 con 0x556446b79410 2024-04-26T23:05:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.598+0000 7f2d385b5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d302011d0 con 0x556446b84de0 2024-04-26T23:05:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.598+0000 7f2d385b5740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2d30201900 con 0x556446b67540 2024-04-26T23:05:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.599+0000 7f2d2e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b79410 0x556446b64540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.599+0000 7f2d2e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b79410 0x556446b64540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44428/0 (socket says 172.21.15.40:44428) 2024-04-26T23:05:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.599+0000 7f2d2e000640 1 -- 172.21.15.40:0/1286247124 learned_addr learned my addr 172.21.15.40:0/1286247124 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.599+0000 7f2d2e000640 1 -- 172.21.15.40:0/1286247124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 msgr2=0x556446ba5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.599+0000 7f2d2e000640 1 --2- 172.21.15.40:0/1286247124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 0x556446ba5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.599+0000 7f2d2e000640 1 -- 172.21.15.40:0/1286247124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b84de0 msgr2=0x556446b64a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.599+0000 7f2d2e000640 1 --2- 172.21.15.40:0/1286247124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b84de0 0x556446b64a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.599+0000 7f2d2e000640 1 -- 172.21.15.40:0/1286247124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564467720f0 con 0x556446b79410 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.600+0000 7f2d2e000640 1 --2- 172.21.15.40:0/1286247124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b79410 0x556446b64540 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c05a560 tx=0x7f2d1c070040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ddd23380b2e9ab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.600+0000 7f2d1aa00640 1 -- 172.21.15.40:0/1286247124 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c087070 con 0x556446b79410 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.600+0000 7f2d1aa00640 1 -- 172.21.15.40:0/1286247124 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d1c0791f0 con 0x556446b79410 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.600+0000 7f2d1aa00640 1 -- 172.21.15.40:0/1286247124 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c021070 con 0x556446b79410 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.600+0000 7f2d385b5740 1 -- 172.21.15.40:0/1286247124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b79410 msgr2=0x556446b64540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.600+0000 7f2d385b5740 1 --2- 172.21.15.40:0/1286247124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b79410 0x556446b64540 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c05a560 tx=0x7f2d1c070040 comp rx=0 tx=0).stop 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.600+0000 7f2d385b5740 1 -- 172.21.15.40:0/1286247124 shutdown_connections 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.600+0000 7f2d385b5740 1 --2- 172.21.15.40:0/1286247124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b84de0 0x556446b64a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.600+0000 7f2d385b5740 1 --2- 172.21.15.40:0/1286247124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b79410 0x556446b64540 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.600+0000 7f2d385b5740 1 --2- 172.21.15.40:0/1286247124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b67540 0x556446ba5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.600+0000 7f2d385b5740 1 -- 172.21.15.40:0/1286247124 >> 172.21.15.40:0/1286247124 conn(0x556446b3d220 msgr2=0x556446b3e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.601+0000 7f2d385b5740 1 -- 172.21.15.40:0/1286247124 shutdown_connections 2024-04-26T23:05:04.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.601+0000 7f2d385b5740 1 -- 172.21.15.40:0/1286247124 wait complete. 2024-04-26T23:05:04.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.601+0000 7f2d385b5740 1 -- start start 2024-04-26T23:05:04.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.601+0000 7f2d385b5740 1 Processor -- start 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.602+0000 7f2d385b5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b67540 0x556446b83110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.602+0000 7f2d385b5740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b79410 0x556446b77ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.602+0000 7f2d385b5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b6a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.602+0000 7f2d385b5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556446aadfd0 con 0x556446b84de0 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.602+0000 7f2d385b5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d300af820 con 0x556446b67540 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.602+0000 7f2d385b5740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556446b6b1b0 con 0x556446b79410 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.602+0000 7f2d385b5740 10 client.0 _ll_register_callbacks cb 0x7f2d1c0598c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.603+0000 7f2d2f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b6a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.603+0000 7f2d2f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b6a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44444/0 (socket says 172.21.15.40:44444) 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.603+0000 7f2d2f400640 1 -- 172.21.15.40:0/910718918 learned_addr learned my addr 172.21.15.40:0/910718918 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.603+0000 7f2d2ea00640 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b67540 0x556446b83110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.603+0000 7f2d2e000640 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b79410 0x556446b77ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.603+0000 7f2d2f400640 1 -- 172.21.15.40:0/910718918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b79410 msgr2=0x556446b77ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.603+0000 7f2d2f400640 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b79410 0x556446b77ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.603+0000 7f2d2f400640 1 -- 172.21.15.40:0/910718918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b67540 msgr2=0x556446b83110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.603+0000 7f2d2f400640 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b67540 0x556446b83110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.603+0000 7f2d2f400640 1 -- 172.21.15.40:0/910718918 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556446a68a30 con 0x556446b84de0 2024-04-26T23:05:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.603+0000 7f2d2ea00640 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b67540 0x556446b83110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.604+0000 7f2d2f400640 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b6a4d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2d28034040 tx=0x7f2d2800b510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d169941f8f67929 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.604+0000 7f2d0fe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:04.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.604+0000 7f2d0fe00640 1 -- 172.21.15.40:0/910718918 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d28032970 con 0x556446b84de0 2024-04-26T23:05:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.604+0000 7f2d0fe00640 1 -- 172.21.15.40:0/910718918 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f2d28032b10 con 0x556446b84de0 2024-04-26T23:05:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.604+0000 7f2d385b5740 1 -- 172.21.15.40:0/910718918 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d1c0791f0 con 0x556446b84de0 2024-04-26T23:05:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.604+0000 7f2d0fe00640 1 -- 172.21.15.40:0/910718918 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f2d2804f050 con 0x556446b84de0 2024-04-26T23:05:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.604+0000 7f2d385b5740 10 client.6507 fetch_fsmap learned FSMap version 46 2024-04-26T23:05:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.604+0000 7f2d385b5740 1 -- 172.21.15.40:0/910718918 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x556446b82910 con 0x556446b84de0 2024-04-26T23:05:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.604+0000 7f2d0fe00640 1 -- 172.21.15.40:0/910718918 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d1c0791f0 con 0x556446b84de0 2024-04-26T23:05:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.605+0000 7f2d0fe00640 1 -- 172.21.15.40:0/910718918 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 46) ==== 3550+0+0 (secure 0 0 0) 0x7f2d28052020 con 0x556446b84de0 2024-04-26T23:05:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.605+0000 7f2d385b5740 10 client.6507 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T23:05:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.605+0000 7f2d385b5740 10 client.6507 resolve_mds: resolved 3:2 to role '3:2' aka daemon mds.b 2024-04-26T23:05:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.605+0000 7f2d385b5740 20 client.6507 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.605+0000 7f2d385b5740 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x556446be95d0 0x556446b3d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.605+0000 7f2d385b5740 4 client.6507 mds_command: new command op to 5284 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:05:04.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.605+0000 7f2d385b5740 1 -- 172.21.15.40:0/910718918 --> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556446aa8650 con 0x556446be95d0 2024-04-26T23:05:04.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.605+0000 7f2d2ea00640 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x556446be95d0 0x556446b3d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.606+0000 7f2d2ea00640 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x556446be95d0 0x556446b3d600 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=155dad001c9b5991 server_cookie=e3521040b40d6f74 in_seq=0 out_seq=0 2024-04-26T23:05:04.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.606+0000 7f2d0fe00640 10 client.6507 ms_handle_connect on v2:172.21.15.103:6836/670995452 2024-04-26T23:05:04.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d0fe00640 1 -- 172.21.15.40:0/910718918 <== mds.2 v2:172.21.15.103:6836/670995452 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556446aa8650 con 0x556446be95d0 2024-04-26T23:05:04.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d0fe00640 10 client.6507 handle_command_reply: tid=0 2024-04-26T23:05:04.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d385b5740 1 client.6507 shutdown 2024-04-26T23:05:04.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d385b5740 1 -- 172.21.15.40:0/910718918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 msgr2=0x556446b6a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d385b5740 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b6a4d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2d28034040 tx=0x7f2d2800b510 comp rx=0 tx=0).stop 2024-04-26T23:05:04.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d385b5740 1 -- 172.21.15.40:0/910718918 shutdown_connections 2024-04-26T23:05:04.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d385b5740 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.103:6836/670995452,v1:172.21.15.103:6837/670995452] conn(0x556446be95d0 0x556446b3d600 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d385b5740 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556446b84de0 0x556446b6a4d0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d385b5740 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556446b79410 0x556446b77ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d385b5740 1 --2- 172.21.15.40:0/910718918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556446b67540 0x556446b83110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d0fe00640 0 client.6507 ms_handle_reset on v2:172.21.15.103:6836/670995452 2024-04-26T23:05:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d385b5740 1 -- 172.21.15.40:0/910718918 >> 172.21.15.40:0/910718918 conn(0x556446b3d220 msgr2=0x556446b81810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d385b5740 1 -- 172.21.15.40:0/910718918 shutdown_connections 2024-04-26T23:05:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d385b5740 1 -- 172.21.15.40:0/910718918 wait complete. 2024-04-26T23:05:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.607+0000 7f2d385b5740 20 client.6507 trim_cache size 0 max 16384 2024-04-26T23:05:04.610 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:05:04.610 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-04-26T23:05:04.610 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-04-26T23:05:04.610 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:05:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.608+0000 7f2d35800640 1 -- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d1006c8f0 msgr2=0x7f2d1006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.608+0000 7f2d35800640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d1006c8f0 0x7f2d1006edb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2d20007fd0 tx=0x7f2d20009040 comp rx=0 tx=0).stop 2024-04-26T23:05:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.608+0000 7f2d35800640 1 -- 172.21.15.40:0/2483480260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d30072c50 msgr2=0x7f2d301f4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.608+0000 7f2d35800640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d30072c50 0x7f2d301f4e40 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00e9b0 tx=0x7f2d1c00ee70 comp rx=0 tx=0).stop 2024-04-26T23:05:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.608+0000 7f2d35800640 1 -- 172.21.15.40:0/2483480260 shutdown_connections 2024-04-26T23:05:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.608+0000 7f2d35800640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d1006c8f0 0x7f2d1006edb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.608+0000 7f2d35800640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d30156460 0x7f2d301526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.608+0000 7f2d35800640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d30072c50 0x7f2d301f4e40 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.608+0000 7f2d35800640 1 --2- 172.21.15.40:0/2483480260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3006f790 0x7f2d300b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.608+0000 7f2d35800640 1 -- 172.21.15.40:0/2483480260 >> 172.21.15.40:0/2483480260 conn(0x7f2d300d0be0 msgr2=0x7f2d301517d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.608+0000 7f2d35800640 1 -- 172.21.15.40:0/2483480260 shutdown_connections 2024-04-26T23:05:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.608+0000 7f2d35800640 1 -- 172.21.15.40:0/2483480260 wait complete. 2024-04-26T23:05:04.626 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:05:04.627 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:05:04.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.3:0 scrub status 2024-04-26T23:05:04.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.832+0000 7f1497400640 1 Processor -- start 2024-04-26T23:05:04.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.832+0000 7f1497400640 1 -- start start 2024-04-26T23:05:04.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.834+0000 7f1497400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f149006c950 0x7f14900ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.834+0000 7f1497400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14900cfd80 con 0x7f14900b0380 2024-04-26T23:05:04.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.834+0000 7f1497400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14900d0020 con 0x7f14900b10b0 2024-04-26T23:05:04.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.834+0000 7f1497400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f14900af8e0 con 0x7f149006c950 2024-04-26T23:05:04.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.834+0000 7f1495400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f149006c950 0x7f14900ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.835+0000 7f1495400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f149006c950 0x7f14900ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44446/0 (socket says 172.21.15.40:44446) 2024-04-26T23:05:04.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.835+0000 7f1495400640 1 -- 172.21.15.40:0/1395864001 learned_addr learned my addr 172.21.15.40:0/1395864001 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.835+0000 7f1495400640 1 -- 172.21.15.40:0/1395864001 >> v1:172.21.15.103:6789/0 conn(0x7f14900b10b0 legacy=0x7f149015e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.835+0000 7f1495400640 1 -- 172.21.15.40:0/1395864001 >> v1:172.21.15.40:6789/0 conn(0x7f14900b0380 legacy=0x7f149006c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.835+0000 7f1495400640 1 -- 172.21.15.40:0/1395864001 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14900d0300 con 0x7f149006c950 2024-04-26T23:05:04.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.836+0000 7f1495400640 1 --2- 172.21.15.40:0/1395864001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f149006c950 0x7f14900ccd50 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1488009e50 tx=0x7f148802a720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175a86959309c152 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.837+0000 7f148f400640 1 -- 172.21.15.40:0/1395864001 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488036070 con 0x7f149006c950 2024-04-26T23:05:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.837+0000 7f148f400640 1 -- 172.21.15.40:0/1395864001 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1488031600 con 0x7f149006c950 2024-04-26T23:05:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.837+0000 7f148f400640 1 -- 172.21.15.40:0/1395864001 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14880319d0 con 0x7f149006c950 2024-04-26T23:05:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.837+0000 7f1497400640 1 -- 172.21.15.40:0/1395864001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f149006c950 msgr2=0x7f14900ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.837+0000 7f1497400640 1 --2- 172.21.15.40:0/1395864001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f149006c950 0x7f14900ccd50 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1488009e50 tx=0x7f148802a720 comp rx=0 tx=0).stop 2024-04-26T23:05:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.837+0000 7f1497400640 1 -- 172.21.15.40:0/1395864001 shutdown_connections 2024-04-26T23:05:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.837+0000 7f1497400640 1 --2- 172.21.15.40:0/1395864001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f149006c950 0x7f14900ccd50 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.837+0000 7f1497400640 1 -- 172.21.15.40:0/1395864001 >> 172.21.15.40:0/1395864001 conn(0x7f14900d1cf0 msgr2=0x7f14900d20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.838+0000 7f1497400640 1 -- 172.21.15.40:0/1395864001 shutdown_connections 2024-04-26T23:05:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.838+0000 7f1497400640 1 -- 172.21.15.40:0/1395864001 wait complete. 2024-04-26T23:05:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.839+0000 7f1497400640 1 Processor -- start 2024-04-26T23:05:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.839+0000 7f1497400640 1 -- start start 2024-04-26T23:05:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.840+0000 7f1497400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f149006c950 0x7f1490150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.840+0000 7f1497400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14900b0380 0x7f14900c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.840+0000 7f1497400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14900b10b0 0x7f14900b7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.840+0000 7f1497400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f14900b0760 con 0x7f149006c950 2024-04-26T23:05:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.840+0000 7f1497400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f14900d1980 con 0x7f14900b0380 2024-04-26T23:05:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.840+0000 7f1497400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f14900d15c0 con 0x7f14900b10b0 2024-04-26T23:05:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.841+0000 7f1494a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f149006c950 0x7f1490150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.841+0000 7f1494a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f149006c950 0x7f1490150850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44456/0 (socket says 172.21.15.40:44456) 2024-04-26T23:05:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.841+0000 7f1494a00640 1 -- 172.21.15.40:0/1465488484 learned_addr learned my addr 172.21.15.40:0/1465488484 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.841+0000 7f148fe00640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14900b0380 0x7f14900c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.841+0000 7f1495400640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14900b10b0 0x7f14900b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.841+0000 7f1494a00640 1 -- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14900b10b0 msgr2=0x7f14900b7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.841+0000 7f1494a00640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14900b10b0 0x7f14900b7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.841+0000 7f1494a00640 1 -- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14900b0380 msgr2=0x7f14900c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.841+0000 7f1494a00640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14900b0380 0x7f14900c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.841+0000 7f1494a00640 1 -- 172.21.15.40:0/1465488484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1490150d90 con 0x7f149006c950 2024-04-26T23:05:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.842+0000 7f1494a00640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f149006c950 0x7f1490150850 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f148400c4d0 tx=0x7f148400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.842+0000 7f148cc00640 1 -- 172.21.15.40:0/1465488484 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14840052d0 con 0x7f149006c950 2024-04-26T23:05:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.842+0000 7f148cc00640 1 -- 172.21.15.40:0/1465488484 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1484005470 con 0x7f149006c950 2024-04-26T23:05:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.842+0000 7f148cc00640 1 -- 172.21.15.40:0/1465488484 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1484012850 con 0x7f149006c950 2024-04-26T23:05:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.842+0000 7f1497400640 1 -- 172.21.15.40:0/1465488484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14901f5b70 con 0x7f149006c950 2024-04-26T23:05:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.842+0000 7f1497400640 1 -- 172.21.15.40:0/1465488484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14900b6b10 con 0x7f149006c950 2024-04-26T23:05:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.844+0000 7f148cc00640 1 -- 172.21.15.40:0/1465488484 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f14840129f0 con 0x7f149006c950 2024-04-26T23:05:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.846+0000 7f148cc00640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f147006c8f0 0x7f147006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.846+0000 7f148cc00640 1 -- 172.21.15.40:0/1465488484 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f148408b9a0 con 0x7f149006c950 2024-04-26T23:05:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.847+0000 7f149a234740 1 Processor -- start 2024-04-26T23:05:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.847+0000 7f148fe00640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f147006c8f0 0x7f147006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.847+0000 7f149a234740 1 -- start start 2024-04-26T23:05:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.848+0000 7f149a234740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e7540 0x5591f84e7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.848+0000 7f149a234740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 0x5591f8505330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.848+0000 7f149a234740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84bd220 0x5591f85034a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.848+0000 7f149a234740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5591f83cd030 con 0x5591f84bd220 2024-04-26T23:05:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.848+0000 7f149a234740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5591f83e87f0 con 0x5591f84e7540 2024-04-26T23:05:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.848+0000 7f149a234740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5591f83a37e0 con 0x5591f8504de0 2024-04-26T23:05:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.848+0000 7f148fe00640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f147006c8f0 0x7f147006edb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f149006f100 tx=0x7f147c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.848+0000 7f1495400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84bd220 0x5591f85034a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.848+0000 7f1495400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84bd220 0x5591f85034a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44458/0 (socket says 172.21.15.40:44458) 2024-04-26T23:05:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.848+0000 7f1495400640 1 -- 172.21.15.40:0/1090998437 learned_addr learned my addr 172.21.15.40:0/1090998437 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.849+0000 7f1494a00640 1 --2- 172.21.15.40:0/1090998437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e7540 0x5591f84e7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.849+0000 7f148fe00640 1 --2- 172.21.15.40:0/1090998437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 0x5591f8505330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.849+0000 7f1495400640 1 -- 172.21.15.40:0/1090998437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 msgr2=0x5591f8505330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.849+0000 7f1495400640 1 --2- 172.21.15.40:0/1090998437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 0x5591f8505330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.849+0000 7f1495400640 1 -- 172.21.15.40:0/1090998437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e7540 msgr2=0x5591f84e7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.849+0000 7f1495400640 1 --2- 172.21.15.40:0/1090998437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e7540 0x5591f84e7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.849+0000 7f1495400640 1 -- 172.21.15.40:0/1090998437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591f83af050 con 0x5591f84bd220 2024-04-26T23:05:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.849+0000 7f1494a00640 1 --2- 172.21.15.40:0/1090998437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e7540 0x5591f84e7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.849+0000 7f1495400640 1 --2- 172.21.15.40:0/1090998437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84bd220 0x5591f85034a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f1488009f70 tx=0x7f148802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72956b0514f63c39 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.850+0000 7f1481200640 1 -- 172.21.15.40:0/1090998437 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488036040 con 0x5591f84bd220 2024-04-26T23:05:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.850+0000 7f1481200640 1 -- 172.21.15.40:0/1090998437 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1488042070 con 0x5591f84bd220 2024-04-26T23:05:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.850+0000 7f1481200640 1 -- 172.21.15.40:0/1090998437 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148803c8b0 con 0x5591f84bd220 2024-04-26T23:05:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.850+0000 7f149a234740 1 -- 172.21.15.40:0/1090998437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84bd220 msgr2=0x5591f85034a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.850+0000 7f149a234740 1 --2- 172.21.15.40:0/1090998437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84bd220 0x5591f85034a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f1488009f70 tx=0x7f148802e000 comp rx=0 tx=0).stop 2024-04-26T23:05:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.850+0000 7f149a234740 1 -- 172.21.15.40:0/1090998437 shutdown_connections 2024-04-26T23:05:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.850+0000 7f149a234740 1 --2- 172.21.15.40:0/1090998437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84bd220 0x5591f85034a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.850+0000 7f149a234740 1 --2- 172.21.15.40:0/1090998437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 0x5591f8505330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.850+0000 7f149a234740 1 --2- 172.21.15.40:0/1090998437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e7540 0x5591f84e7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.850+0000 7f149a234740 1 -- 172.21.15.40:0/1090998437 >> 172.21.15.40:0/1090998437 conn(0x5591f84b9b50 msgr2=0x5591f8502350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.850+0000 7f149a234740 1 -- 172.21.15.40:0/1090998437 shutdown_connections 2024-04-26T23:05:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.850+0000 7f149a234740 1 -- 172.21.15.40:0/1090998437 wait complete. 2024-04-26T23:05:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.851+0000 7f149a234740 1 -- start start 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.851+0000 7f149a234740 1 Processor -- start 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.852+0000 7f149a234740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e7540 0x5591f8505cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.852+0000 7f149a234740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 0x5591f84eaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.852+0000 7f149a234740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e2b40 0x5591f84e2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.852+0000 7f149a234740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5591f8425200 con 0x5591f84e7540 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.852+0000 7f149a234740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5591f841fee0 con 0x5591f84e2b40 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.852+0000 7f149a234740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5591f842ce70 con 0x5591f8504de0 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.852+0000 7f149a234740 10 client.0 _ll_register_callbacks cb 0x5591f8416270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.853+0000 7f1494a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e7540 0x5591f8505cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.853+0000 7f1494a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e7540 0x5591f8505cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44466/0 (socket says 172.21.15.40:44466) 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.853+0000 7f1494a00640 1 -- 172.21.15.40:0/2607958708 learned_addr learned my addr 172.21.15.40:0/2607958708 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.853+0000 7f1495400640 1 --2- 172.21.15.40:0/2607958708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e2b40 0x5591f84e2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.853+0000 7f1494a00640 1 -- 172.21.15.40:0/2607958708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 msgr2=0x5591f84eaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.853+0000 7f148fe00640 1 --2- 172.21.15.40:0/2607958708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 0x5591f84eaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.853+0000 7f1494a00640 1 --2- 172.21.15.40:0/2607958708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 0x5591f84eaa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.853+0000 7f1494a00640 1 -- 172.21.15.40:0/2607958708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e2b40 msgr2=0x5591f84e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.853+0000 7f1494a00640 1 --2- 172.21.15.40:0/2607958708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e2b40 0x5591f84e2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.853+0000 7f1494a00640 1 -- 172.21.15.40:0/2607958708 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591f81fc7d0 con 0x5591f84e7540 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.853+0000 7f1494a00640 1 --2- 172.21.15.40:0/2607958708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e7540 0x5591f8505cd0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f14840558d0 tx=0x7f148405d820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30994b6e7dea0e97 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.853+0000 7f1475600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f1475600640 1 -- 172.21.15.40:0/2607958708 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1484055ae0 con 0x5591f84e7540 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f1475600640 1 -- 172.21.15.40:0/2607958708 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f1484061bc0 con 0x5591f84e7540 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f149a234740 1 -- 172.21.15.40:0/2607958708 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1488042070 con 0x5591f84e7540 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f1475600640 1 -- 172.21.15.40:0/2607958708 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f1484055c80 con 0x5591f84e7540 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f149a234740 10 client.6522 fetch_fsmap learned FSMap version 46 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f149a234740 1 -- 172.21.15.40:0/2607958708 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x5591f81323f0 con 0x5591f84e7540 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f1475600640 1 -- 172.21.15.40:0/2607958708 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1488042070 con 0x5591f84e7540 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f1475600640 1 -- 172.21.15.40:0/2607958708 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 46) ==== 3550+0+0 (secure 0 0 0) 0x7f14840588c0 con 0x5591f84e7540 2024-04-26T23:05:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f149a234740 10 client.6522 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T23:05:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f149a234740 10 client.6522 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-04-26T23:05:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f149a234740 20 client.6522 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f149a234740 1 --2- 172.21.15.40:0/2607958708 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5591f8511f70 0x5591f84fa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f149a234740 4 client.6522 mds_command: new command op to 6036 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:05:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.854+0000 7f149a234740 1 -- 172.21.15.40:0/2607958708 --> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591f842e7e0 con 0x5591f8511f70 2024-04-26T23:05:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.855+0000 7f148fe00640 1 --2- 172.21.15.40:0/2607958708 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5591f8511f70 0x5591f84fa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.855+0000 7f148fe00640 1 --2- 172.21.15.40:0/2607958708 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5591f8511f70 0x5591f84fa170 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e244565e5fff41d server_cookie=d23c5c5328b49121 in_seq=0 out_seq=0 2024-04-26T23:05:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.855+0000 7f1475600640 10 client.6522 ms_handle_connect on v2:172.21.15.40:6833/618018110 2024-04-26T23:05:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.856+0000 7f1475600640 1 -- 172.21.15.40:0/2607958708 <== mds.0 v2:172.21.15.40:6833/618018110 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5591f842e7e0 con 0x5591f8511f70 2024-04-26T23:05:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.856+0000 7f1475600640 10 client.6522 handle_command_reply: tid=0 2024-04-26T23:05:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.856+0000 7f149a234740 1 client.6522 shutdown 2024-04-26T23:05:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.856+0000 7f149a234740 1 -- 172.21.15.40:0/2607958708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e7540 msgr2=0x5591f8505cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.856+0000 7f149a234740 1 --2- 172.21.15.40:0/2607958708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e7540 0x5591f8505cd0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f14840558d0 tx=0x7f148405d820 comp rx=0 tx=0).stop 2024-04-26T23:05:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.856+0000 7f149a234740 1 -- 172.21.15.40:0/2607958708 shutdown_connections 2024-04-26T23:05:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.856+0000 7f149a234740 1 --2- 172.21.15.40:0/2607958708 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5591f8511f70 0x5591f84fa170 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.856+0000 7f149a234740 1 --2- 172.21.15.40:0/2607958708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e2b40 0x5591f84e2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.856+0000 7f149a234740 1 --2- 172.21.15.40:0/2607958708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 0x5591f84eaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.856+0000 7f1475600640 0 client.6522 ms_handle_reset on v2:172.21.15.40:6833/618018110 2024-04-26T23:05:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.856+0000 7f149a234740 1 --2- 172.21.15.40:0/2607958708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e7540 0x5591f8505cd0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.856+0000 7f149a234740 1 -- 172.21.15.40:0/2607958708 >> 172.21.15.40:0/2607958708 conn(0x5591f84bd220 msgr2=0x5591f84b70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.857+0000 7f149a234740 1 -- 172.21.15.40:0/2607958708 shutdown_connections 2024-04-26T23:05:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.857+0000 7f149a234740 1 -- 172.21.15.40:0/2607958708 wait complete. 2024-04-26T23:05:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.857+0000 7f149a234740 20 client.6522 trim_cache size 0 max 16384 2024-04-26T23:05:04.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.876+0000 7f149a234740 1 Processor -- start 2024-04-26T23:05:04.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.876+0000 7f149a234740 1 -- start start 2024-04-26T23:05:04.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.877+0000 7f149a234740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 0x5591f84b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.877+0000 7f149a234740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f84e7540 0x5591f84e3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.877+0000 7f149a234740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f8504de0 0x5591f8500f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.877+0000 7f149a234740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1490202060 con 0x5591f84e2b40 2024-04-26T23:05:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.877+0000 7f149a234740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1490201200 con 0x5591f8504de0 2024-04-26T23:05:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.877+0000 7f149a234740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1490201930 con 0x5591f84e7540 2024-04-26T23:05:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.878+0000 7f1494a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 0x5591f84b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.878+0000 7f1494a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 0x5591f84b32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37122/0 (socket says 172.21.15.40:37122) 2024-04-26T23:05:04.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.878+0000 7f1494a00640 1 -- 172.21.15.40:0/613601562 learned_addr learned my addr 172.21.15.40:0/613601562 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.878+0000 7f148fe00640 1 --2- 172.21.15.40:0/613601562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f84e7540 0x5591f84e3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.878+0000 7f1495400640 1 --2- 172.21.15.40:0/613601562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f8504de0 0x5591f8500f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.878+0000 7f1494a00640 1 -- 172.21.15.40:0/613601562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f84e7540 msgr2=0x5591f84e3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.878+0000 7f1494a00640 1 --2- 172.21.15.40:0/613601562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f84e7540 0x5591f84e3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.878+0000 7f1494a00640 1 -- 172.21.15.40:0/613601562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f8504de0 msgr2=0x5591f8500f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.878+0000 7f1494a00640 1 --2- 172.21.15.40:0/613601562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f8504de0 0x5591f8500f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.878+0000 7f1494a00640 1 -- 172.21.15.40:0/613601562 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591f80f20f0 con 0x5591f84e2b40 2024-04-26T23:05:04.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.878+0000 7f1495400640 1 --2- 172.21.15.40:0/613601562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f8504de0 0x5591f8500f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:04.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.878+0000 7f1494a00640 1 --2- 172.21.15.40:0/613601562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 0x5591f84b32f0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1484062210 tx=0x7f148405c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34104d6b6488246 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.879+0000 7f1477e00640 1 -- 172.21.15.40:0/613601562 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1484055710 con 0x5591f84e2b40 2024-04-26T23:05:04.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.879+0000 7f1477e00640 1 -- 172.21.15.40:0/613601562 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1484012ce0 con 0x5591f84e2b40 2024-04-26T23:05:04.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.879+0000 7f1477e00640 1 -- 172.21.15.40:0/613601562 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1484061e00 con 0x5591f84e2b40 2024-04-26T23:05:04.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.879+0000 7f149a234740 1 -- 172.21.15.40:0/613601562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 msgr2=0x5591f84b32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.879+0000 7f149a234740 1 --2- 172.21.15.40:0/613601562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 0x5591f84b32f0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1484062210 tx=0x7f148405c000 comp rx=0 tx=0).stop 2024-04-26T23:05:04.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.879+0000 7f149a234740 1 -- 172.21.15.40:0/613601562 shutdown_connections 2024-04-26T23:05:04.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.879+0000 7f149a234740 1 --2- 172.21.15.40:0/613601562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f8504de0 0x5591f8500f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.879+0000 7f149a234740 1 --2- 172.21.15.40:0/613601562 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f84e7540 0x5591f84e3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.879+0000 7f149a234740 1 --2- 172.21.15.40:0/613601562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 0x5591f84b32f0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.879+0000 7f149a234740 1 -- 172.21.15.40:0/613601562 >> 172.21.15.40:0/613601562 conn(0x5591f84bd220 msgr2=0x5591f84b70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.880+0000 7f149a234740 1 -- 172.21.15.40:0/613601562 shutdown_connections 2024-04-26T23:05:04.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.880+0000 7f149a234740 1 -- 172.21.15.40:0/613601562 wait complete. 2024-04-26T23:05:04.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.880+0000 7f149a234740 1 -- start start 2024-04-26T23:05:04.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.880+0000 7f149a234740 1 Processor -- start 2024-04-26T23:05:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.881+0000 7f149a234740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 0x5591f8500cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.881+0000 7f149a234740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e7540 0x5591f857b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.881+0000 7f149a234740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 0x5591f85147a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.881+0000 7f149a234740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5591f842dfd0 con 0x5591f84e2b40 2024-04-26T23:05:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.881+0000 7f149a234740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f14900b54e0 con 0x5591f84e7540 2024-04-26T23:05:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.881+0000 7f149a234740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5591f857bc30 con 0x5591f8504de0 2024-04-26T23:05:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.881+0000 7f149a234740 10 client.0 _ll_register_callbacks cb 0x7f1484059240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.882+0000 7f1494a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 0x5591f8500cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.882+0000 7f1494a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 0x5591f8500cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37126/0 (socket says 172.21.15.40:37126) 2024-04-26T23:05:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.882+0000 7f1494a00640 1 -- 172.21.15.40:0/4279929326 learned_addr learned my addr 172.21.15.40:0/4279929326 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.882+0000 7f148fe00640 1 --2- 172.21.15.40:0/4279929326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e7540 0x5591f857b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.882+0000 7f1494a00640 1 -- 172.21.15.40:0/4279929326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 msgr2=0x5591f85147a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.882+0000 7f1495400640 1 --2- 172.21.15.40:0/4279929326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 0x5591f85147a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.882+0000 7f1494a00640 1 --2- 172.21.15.40:0/4279929326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 0x5591f85147a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.882+0000 7f1494a00640 1 -- 172.21.15.40:0/4279929326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e7540 msgr2=0x5591f857b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.882+0000 7f1494a00640 1 --2- 172.21.15.40:0/4279929326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e7540 0x5591f857b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.882+0000 7f1494a00640 1 -- 172.21.15.40:0/4279929326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591f83e8a30 con 0x5591f84e2b40 2024-04-26T23:05:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f1494a00640 1 --2- 172.21.15.40:0/4279929326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 0x5591f8500cd0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f1484062120 tx=0x7f1484089b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d46452e0bc306e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f1475600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f1475600640 1 -- 172.21.15.40:0/4279929326 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1484055710 con 0x5591f84e2b40 2024-04-26T23:05:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f1475600640 1 -- 172.21.15.40:0/4279929326 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f14840553d0 con 0x5591f84e2b40 2024-04-26T23:05:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f149a234740 1 -- 172.21.15.40:0/4279929326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1484012ce0 con 0x5591f84e2b40 2024-04-26T23:05:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f1475600640 1 -- 172.21.15.40:0/4279929326 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f14840830d0 con 0x5591f84e2b40 2024-04-26T23:05:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f149a234740 10 client.6528 fetch_fsmap learned FSMap version 46 2024-04-26T23:05:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f149a234740 1 -- 172.21.15.40:0/4279929326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x5591f83e8a30 con 0x5591f84e2b40 2024-04-26T23:05:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f1475600640 1 -- 172.21.15.40:0/4279929326 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1484012ce0 con 0x5591f84e2b40 2024-04-26T23:05:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f1475600640 1 -- 172.21.15.40:0/4279929326 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 46) ==== 3550+0+0 (secure 0 0 0) 0x7f1484064050 con 0x5591f84e2b40 2024-04-26T23:05:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f149a234740 10 client.6528 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T23:05:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f149a234740 10 client.6528 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-04-26T23:05:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f149a234740 20 client.6528 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f149a234740 1 --2- 172.21.15.40:0/4279929326 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5591f84eb600 0x5591f8514ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f149a234740 4 client.6528 mds_command: new command op to 6036 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:05:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.883+0000 7f149a234740 1 -- 172.21.15.40:0/4279929326 --> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5591f8428650 con 0x5591f84eb600 2024-04-26T23:05:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.884+0000 7f148fe00640 1 --2- 172.21.15.40:0/4279929326 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5591f84eb600 0x5591f8514ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.884+0000 7f148fe00640 1 --2- 172.21.15.40:0/4279929326 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5591f84eb600 0x5591f8514ce0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ade728198e809fb server_cookie=a4f85a07c5de9313 in_seq=0 out_seq=0 2024-04-26T23:05:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.884+0000 7f1475600640 10 client.6528 ms_handle_connect on v2:172.21.15.40:6833/618018110 2024-04-26T23:05:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f1475600640 1 -- 172.21.15.40:0/4279929326 <== mds.0 v2:172.21.15.40:6833/618018110 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5591f8428650 con 0x5591f84eb600 2024-04-26T23:05:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f1475600640 10 client.6528 handle_command_reply: tid=0 2024-04-26T23:05:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f149a234740 1 client.6528 shutdown 2024-04-26T23:05:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f149a234740 1 -- 172.21.15.40:0/4279929326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 msgr2=0x5591f8500cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f149a234740 1 --2- 172.21.15.40:0/4279929326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 0x5591f8500cd0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f1484062120 tx=0x7f1484089b40 comp rx=0 tx=0).stop 2024-04-26T23:05:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f149a234740 1 -- 172.21.15.40:0/4279929326 shutdown_connections 2024-04-26T23:05:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f149a234740 1 --2- 172.21.15.40:0/4279929326 >> [v2:172.21.15.40:6833/618018110,v1:172.21.15.40:6835/618018110] conn(0x5591f84eb600 0x5591f8514ce0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f149a234740 1 --2- 172.21.15.40:0/4279929326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5591f8504de0 0x5591f85147a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f149a234740 1 --2- 172.21.15.40:0/4279929326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5591f84e7540 0x5591f857b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f149a234740 1 --2- 172.21.15.40:0/4279929326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5591f84e2b40 0x5591f8500cd0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f149a234740 1 -- 172.21.15.40:0/4279929326 >> 172.21.15.40:0/4279929326 conn(0x5591f84bd220 msgr2=0x5591f84b6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f1475600640 0 client.6528 ms_handle_reset on v2:172.21.15.40:6833/618018110 2024-04-26T23:05:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f149a234740 1 -- 172.21.15.40:0/4279929326 shutdown_connections 2024-04-26T23:05:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f149a234740 1 -- 172.21.15.40:0/4279929326 wait complete. 2024-04-26T23:05:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.885+0000 7f149a234740 20 client.6528 trim_cache size 0 max 16384 2024-04-26T23:05:04.892 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:05:04.892 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-04-26T23:05:04.892 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-04-26T23:05:04.892 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:05:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.886+0000 7f1497400640 1 -- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f147006c8f0 msgr2=0x7f147006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.886+0000 7f1497400640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f147006c8f0 0x7f147006edb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f149006f100 tx=0x7f147c009040 comp rx=0 tx=0).stop 2024-04-26T23:05:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.886+0000 7f1497400640 1 -- 172.21.15.40:0/1465488484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f149006c950 msgr2=0x7f1490150850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.886+0000 7f1497400640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f149006c950 0x7f1490150850 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f148400c4d0 tx=0x7f148400c990 comp rx=0 tx=0).stop 2024-04-26T23:05:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.886+0000 7f1497400640 1 -- 172.21.15.40:0/1465488484 shutdown_connections 2024-04-26T23:05:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.886+0000 7f1497400640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f147006c8f0 0x7f147006edb0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.886+0000 7f1497400640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14900b10b0 0x7f14900b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.886+0000 7f1497400640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14900b0380 0x7f14900c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.886+0000 7f1497400640 1 --2- 172.21.15.40:0/1465488484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f149006c950 0x7f1490150850 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.886+0000 7f1497400640 1 -- 172.21.15.40:0/1465488484 >> 172.21.15.40:0/1465488484 conn(0x7f14900d1cf0 msgr2=0x7f1490072f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.886+0000 7f1497400640 1 -- 172.21.15.40:0/1465488484 shutdown_connections 2024-04-26T23:05:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:04.886+0000 7f1497400640 1 -- 172.21.15.40:0/1465488484 wait complete. 2024-04-26T23:05:04.903 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:05:04.903 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:05:05.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.115+0000 7f46eac00640 1 Processor -- start 2024-04-26T23:05:05.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.115+0000 7f46eac00640 1 -- start start 2024-04-26T23:05:05.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.117+0000 7f46eac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46e4163430 0x7f46e4166f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.117+0000 7f46eac00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f46e41674a0 0x7f46e4167920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.117+0000 7f46eac00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46e4163810 con 0x7f46e40c62c0 2024-04-26T23:05:05.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.117+0000 7f46eac00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f46e40cb400 con 0x7f46e41674a0 2024-04-26T23:05:05.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.117+0000 7f46eac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f46e4167e60 con 0x7f46e4163430 2024-04-26T23:05:05.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.118+0000 7f46e8e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f46e41674a0 0x7f46e4167920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:05.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.118+0000 7f46e9800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46e4163430 0x7f46e4166f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:05.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.118+0000 7f46e8e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f46e41674a0 0x7f46e4167920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46834/0 (socket says 172.21.15.40:46834) 2024-04-26T23:05:05.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.118+0000 7f46e8e00640 1 -- 172.21.15.40:0/1087518813 learned_addr learned my addr 172.21.15.40:0/1087518813 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:05.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.118+0000 7f46e8e00640 1 -- 172.21.15.40:0/1087518813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46e4163430 msgr2=0x7f46e4166f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:05.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.118+0000 7f46e8e00640 1 --2- 172.21.15.40:0/1087518813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46e4163430 0x7f46e4166f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.118+0000 7f46e8e00640 1 -- 172.21.15.40:0/1087518813 >> v1:172.21.15.103:6789/0 conn(0x7f46e40c62c0 legacy=0x7f46e40c6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:05.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.118+0000 7f46e8e00640 1 -- 172.21.15.40:0/1087518813 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e40cb100 con 0x7f46e41674a0 2024-04-26T23:05:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.120+0000 7f46e8e00640 1 --2- 172.21.15.40:0/1087518813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46e41674a0 0x7f46e4167920 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f46e40cb580 tx=0x7f46d002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2e514e6efed0477 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.120+0000 7f46e3e00640 1 -- 172.21.15.40:0/1087518813 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d0037070 con 0x7f46e41674a0 2024-04-26T23:05:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.120+0000 7f46e3e00640 1 -- 172.21.15.40:0/1087518813 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46d0032440 con 0x7f46e41674a0 2024-04-26T23:05:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.120+0000 7f46e3e00640 1 -- 172.21.15.40:0/1087518813 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d0032810 con 0x7f46e41674a0 2024-04-26T23:05:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.121+0000 7f46eac00640 1 -- 172.21.15.40:0/1087518813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46e41674a0 msgr2=0x7f46e4167920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.121+0000 7f46eac00640 1 --2- 172.21.15.40:0/1087518813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46e41674a0 0x7f46e4167920 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f46e40cb580 tx=0x7f46d002a250 comp rx=0 tx=0).stop 2024-04-26T23:05:05.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.121+0000 7f46eac00640 1 -- 172.21.15.40:0/1087518813 shutdown_connections 2024-04-26T23:05:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.121+0000 7f46eac00640 1 --2- 172.21.15.40:0/1087518813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46e41674a0 0x7f46e4167920 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.121+0000 7f46eac00640 1 --2- 172.21.15.40:0/1087518813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46e4163430 0x7f46e4166f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.121+0000 7f46eac00640 1 -- 172.21.15.40:0/1087518813 >> 172.21.15.40:0/1087518813 conn(0x7f46e40d0a10 msgr2=0x7f46e40d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.121+0000 7f46eac00640 1 -- 172.21.15.40:0/1087518813 shutdown_connections 2024-04-26T23:05:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.121+0000 7f46eac00640 1 -- 172.21.15.40:0/1087518813 wait complete. 2024-04-26T23:05:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.123+0000 7f46eac00640 1 Processor -- start 2024-04-26T23:05:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.124+0000 7f46eac00640 1 -- start start 2024-04-26T23:05:05.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.125+0000 7f46eac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46e40c62c0 0x7f46e40cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.125+0000 7f46eac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46e4163430 0x7f46e414f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.125+0000 7f46eac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46e41674a0 0x7f46e414f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.125+0000 7f46eac00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f46e40afe80 con 0x7f46e4163430 2024-04-26T23:05:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.125+0000 7f46eac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f46e40ccc60 con 0x7f46e40c62c0 2024-04-26T23:05:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.125+0000 7f46eac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f46e40cd1e0 con 0x7f46e41674a0 2024-04-26T23:05:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.125+0000 7f46e8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46e4163430 0x7f46e414f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.125+0000 7f46e8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46e4163430 0x7f46e414f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37136/0 (socket says 172.21.15.40:37136) 2024-04-26T23:05:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.125+0000 7f46e8e00640 1 -- 172.21.15.40:0/3294179508 learned_addr learned my addr 172.21.15.40:0/3294179508 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.126+0000 7f46ea200640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46e41674a0 0x7f46e414f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.126+0000 7f46e9800640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46e40c62c0 0x7f46e40cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.126+0000 7f46e8e00640 1 -- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46e41674a0 msgr2=0x7f46e414f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.126+0000 7f46e8e00640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46e41674a0 0x7f46e414f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.126+0000 7f46e8e00640 1 -- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46e40c62c0 msgr2=0x7f46e40cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.126+0000 7f46e8e00640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46e40c62c0 0x7f46e40cd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.126+0000 7f46e8e00640 1 -- 172.21.15.40:0/3294179508 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e414ffb0 con 0x7f46e4163430 2024-04-26T23:05:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.126+0000 7f46ea200640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46e41674a0 0x7f46e414f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:05:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.126+0000 7f46e9800640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46e40c62c0 0x7f46e40cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:05:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.126+0000 7f46e8e00640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46e4163430 0x7f46e414f1a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f46d0000c00 tx=0x7f46d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.127+0000 7f46e1600640 1 -- 172.21.15.40:0/3294179508 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d0037040 con 0x7f46e4163430 2024-04-26T23:05:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.127+0000 7f46e1600640 1 -- 172.21.15.40:0/3294179508 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46d0042070 con 0x7f46e4163430 2024-04-26T23:05:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.127+0000 7f46eac00640 1 -- 172.21.15.40:0/3294179508 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e4150290 con 0x7f46e4163430 2024-04-26T23:05:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.127+0000 7f46e1600640 1 -- 172.21.15.40:0/3294179508 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d0030940 con 0x7f46e4163430 2024-04-26T23:05:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.127+0000 7f46eac00640 1 -- 172.21.15.40:0/3294179508 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e40cdba0 con 0x7f46e4163430 2024-04-26T23:05:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.128+0000 7f46eac00640 1 -- 172.21.15.40:0/3294179508 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46e40cdda0 con 0x7f46e4163430 2024-04-26T23:05:05.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.129+0000 7f46e1600640 1 -- 172.21.15.40:0/3294179508 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f46d0030ae0 con 0x7f46e4163430 2024-04-26T23:05:05.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.130+0000 7f46e1600640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f46c406c8f0 0x7f46c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.130+0000 7f46e1600640 1 -- 172.21.15.40:0/3294179508 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f46d00ac540 con 0x7f46e4163430 2024-04-26T23:05:05.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.131+0000 7f46e9800640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f46c406c8f0 0x7f46c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:05.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.132+0000 7f46e9800640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f46c406c8f0 0x7f46c406edb0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009b90 tx=0x7f46d8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.135+0000 7f46e1600640 1 -- 172.21.15.40:0/3294179508 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f46e40cdda0 con 0x7f46e4163430 2024-04-26T23:05:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.379+0000 7f46eac00640 1 -- 172.21.15.40:0/3294179508 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46e4069370 con 0x7f46e4163430 2024-04-26T23:05:05.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.381+0000 7f46e1600640 1 -- 172.21.15.40:0/3294179508 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5225 (secure 0 0 0) 0x7f46d007a7a0 con 0x7f46e4163430 2024-04-26T23:05:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:05.381 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":46,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/189004193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":189004193},{"type":"v1","addr":"172.21.15.103:6835","nonce":189004193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:04:16.558591+0000","modified":"2024-04-26T23:04:55.194872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6036,"mds_1":6027,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"b","rank":2,"incarnation":39,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/670995452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":670995452},{"type":"v1","addr":"172.21.15.103:6837","nonce":670995452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6027":{"gid":6027,"name":"a","rank":1,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6834/2304583125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2304583125},{"type":"v1","addr":"172.21.15.40:6834","nonce":2304583125}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6036":{"gid":6036,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/618018110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":618018110},{"type":"v1","addr":"172.21.15.40:6835","nonce":618018110}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6036,"qdb_cluster":[5284,6027,6036]},"id":3}]} 2024-04-26T23:05:05.381 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 46 2024-04-26T23:05:05.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.383+0000 7f46eac00640 1 -- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f46c406c8f0 msgr2=0x7f46c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:05.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.383+0000 7f46eac00640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f46c406c8f0 0x7f46c406edb0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009b90 tx=0x7f46d8008040 comp rx=0 tx=0).stop 2024-04-26T23:05:05.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.383+0000 7f46eac00640 1 -- 172.21.15.40:0/3294179508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46e4163430 msgr2=0x7f46e414f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:05.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.383+0000 7f46eac00640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46e4163430 0x7f46e414f1a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f46d0000c00 tx=0x7f46d00046c0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.383+0000 7f46eac00640 1 -- 172.21.15.40:0/3294179508 shutdown_connections 2024-04-26T23:05:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.383+0000 7f46eac00640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f46c406c8f0 0x7f46c406edb0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.383+0000 7f46eac00640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46e41674a0 0x7f46e414f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.383+0000 7f46eac00640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46e4163430 0x7f46e414f1a0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.383+0000 7f46eac00640 1 --2- 172.21.15.40:0/3294179508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46e40c62c0 0x7f46e40cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.383+0000 7f46eac00640 1 -- 172.21.15.40:0/3294179508 >> 172.21.15.40:0/3294179508 conn(0x7f46e40d0a10 msgr2=0x7f46e406e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.384+0000 7f46eac00640 1 -- 172.21.15.40:0/3294179508 shutdown_connections 2024-04-26T23:05:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.384+0000 7f46eac00640 1 -- 172.21.15.40:0/3294179508 wait complete. 2024-04-26T23:05:05.403 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'b'] 2024-04-26T23:05:05.404 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:05:05.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.618+0000 7f7007200640 1 Processor -- start 2024-04-26T23:05:05.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.619+0000 7f7007200640 1 -- start start 2024-04-26T23:05:05.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.620+0000 7f7007200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f70080d17d0 0x7f70080b6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.620+0000 7f7007200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70080b66c0 con 0x7f70080d0c80 2024-04-26T23:05:05.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.620+0000 7f7007200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f700815f9c0 con 0x7f70080d2400 2024-04-26T23:05:05.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.620+0000 7f7007200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f70080d37b0 con 0x7f70080d17d0 2024-04-26T23:05:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.621+0000 7f7006800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f70080d0c80 0x7f70080b5210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58166/0 (socket says 172.21.15.40:58166) 2024-04-26T23:05:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.621+0000 7f7006800640 1 -- 172.21.15.40:0/2109759732 learned_addr learned my addr 172.21.15.40:0/2109759732 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.621+0000 7f7005e00640 1 --2- 172.21.15.40:0/2109759732 >> v2:172.21.15.103:3300/0 conn(0x7f70080d17d0 0x7f70080b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:05.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.621+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2386300803 0 0) 0x7f7000001490 con 0x7f70080d0c80 2024-04-26T23:05:05.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.621+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fec003520 con 0x7f70080d0c80 2024-04-26T23:05:05.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.621+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1864381389 0 0) 0x7f6fec003520 con 0x7f70080d0c80 2024-04-26T23:05:05.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 >> v2:172.21.15.103:3300/0 conn(0x7f70080d17d0 msgr2=0x7f70080b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:05.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7004a00640 1 --2- 172.21.15.40:0/2109759732 >> v2:172.21.15.103:3300/0 conn(0x7f70080d17d0 0x7f70080b6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 >> v1:172.21.15.103:6789/0 conn(0x7f70080d2400 legacy=0x7f70080d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:05.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70080d5730 con 0x7f70080d0c80 2024-04-26T23:05:05.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f7000001bc0 con 0x7f70080d0c80 2024-04-26T23:05:05.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 >> v1:172.21.15.40:6789/0 conn(0x7f70080d0c80 legacy=0x7f70080b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7004a00640 1 --2- 172.21.15.40:0/2109759732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec0039b0 0x7f6fec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7004a00640 1 --2- 172.21.15.40:0/2109759732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fec004800 0x7f6fec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7004a00640 1 --2- 172.21.15.40:0/2109759732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fec0051c0 0x7f6fec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f70080d37b0 con 0x7f6fec0039b0 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec00c160 con 0x7f6fec004800 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6fec00c360 con 0x7f6fec0051c0 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7005e00640 1 --2- 172.21.15.40:0/2109759732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec0039b0 0x7f6fec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7005400640 1 --2- 172.21.15.40:0/2109759732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fec004800 0x7f6fec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7005e00640 1 -- 172.21.15.40:0/2109759732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fec0051c0 msgr2=0x7f6fec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7005e00640 1 --2- 172.21.15.40:0/2109759732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6fec0051c0 0x7f6fec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7005e00640 1 -- 172.21.15.40:0/2109759732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fec004800 msgr2=0x7f6fec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7005e00640 1 --2- 172.21.15.40:0/2109759732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6fec004800 0x7f6fec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7005e00640 1 -- 172.21.15.40:0/2109759732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fec003520 con 0x7f6fec0039b0 2024-04-26T23:05:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.622+0000 7f7006800640 1 -- 172.21.15.40:0/2109759732 reap_dead start 2024-04-26T23:05:05.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.623+0000 7f7005e00640 1 --2- 172.21.15.40:0/2109759732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec0039b0 0x7f6fec003dd0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009ac0 tx=0x7f6fec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326ef1c259c920c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:05.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.624+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8036070 con 0x7f6fec0039b0 2024-04-26T23:05:05.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.624+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff8031400 con 0x7f6fec0039b0 2024-04-26T23:05:05.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.624+0000 7f7004a00640 1 -- 172.21.15.40:0/2109759732 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8009690 con 0x7f6fec0039b0 2024-04-26T23:05:05.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.624+0000 7f7007200640 1 -- 172.21.15.40:0/2109759732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec0039b0 msgr2=0x7f6fec003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:05.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.624+0000 7f7007200640 1 --2- 172.21.15.40:0/2109759732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec0039b0 0x7f6fec003dd0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009ac0 tx=0x7f6fec005830 comp rx=0 tx=0).stop 2024-04-26T23:05:05.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.624+0000 7f7007200640 1 -- 172.21.15.40:0/2109759732 shutdown_connections 2024-04-26T23:05:05.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.624+0000 7f7007200640 1 --2- 172.21.15.40:0/2109759732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec0039b0 0x7f6fec003dd0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.624+0000 7f7007200640 1 -- 172.21.15.40:0/2109759732 >> 172.21.15.40:0/2109759732 conn(0x7f700809e210 msgr2=0x7f70080d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:05.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.625+0000 7f7007200640 1 -- 172.21.15.40:0/2109759732 shutdown_connections 2024-04-26T23:05:05.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.625+0000 7f7007200640 1 -- 172.21.15.40:0/2109759732 wait complete. 2024-04-26T23:05:05.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.626+0000 7f7007200640 1 Processor -- start 2024-04-26T23:05:05.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.626+0000 7f7007200640 1 -- start start 2024-04-26T23:05:05.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.627+0000 7f7007200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec0039b0 0x7f70080ce880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.627+0000 7f7007200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70080b8e70 0x7f70080b2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.627+0000 7f7007200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70080cd220 0x7f7008152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.627+0000 7f7007200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f70080d3a00 con 0x7f6fec0039b0 2024-04-26T23:05:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.627+0000 7f7007200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f70080d3d70 con 0x7f70080b8e70 2024-04-26T23:05:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.627+0000 7f7007200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f70080d10f0 con 0x7f70080cd220 2024-04-26T23:05:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.628+0000 7f7005e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec0039b0 0x7f70080ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.628+0000 7f7005e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec0039b0 0x7f70080ce880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37154/0 (socket says 172.21.15.40:37154) 2024-04-26T23:05:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.628+0000 7f7005e00640 1 -- 172.21.15.40:0/1113115989 learned_addr learned my addr 172.21.15.40:0/1113115989 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:05.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.628+0000 7f7006800640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70080cd220 0x7f7008152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.628+0000 7f7005400640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70080b8e70 0x7f70080b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.628+0000 7f7005e00640 1 -- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70080cd220 msgr2=0x7f7008152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.628+0000 7f7005e00640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70080cd220 0x7f7008152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.628+0000 7f7005e00640 1 -- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70080b8e70 msgr2=0x7f70080b2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.628+0000 7f7005e00640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70080b8e70 0x7f70080b2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.628+0000 7f7005e00640 1 -- 172.21.15.40:0/1113115989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70080cd7e0 con 0x7f6fec0039b0 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.628+0000 7f7005400640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70080b8e70 0x7f70080b2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.629+0000 7f7005e00640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec0039b0 0x7f70080ce880 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f6ff802abb0 tx=0x7f6ff8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.629+0000 7f6ffe000640 1 -- 172.21.15.40:0/1113115989 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff803f020 con 0x7f6fec0039b0 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.629+0000 7f6ffe000640 1 -- 172.21.15.40:0/1113115989 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff8037040 con 0x7f6fec0039b0 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.629+0000 7f6ffe000640 1 -- 172.21.15.40:0/1113115989 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8036070 con 0x7f6fec0039b0 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.629+0000 7f7007200640 1 -- 172.21.15.40:0/1113115989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70080cdac0 con 0x7f6fec0039b0 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.629+0000 7f7007200640 1 -- 172.21.15.40:0/1113115989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70080ce080 con 0x7f6fec0039b0 2024-04-26T23:05:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.630+0000 7f7007200640 1 -- 172.21.15.40:0/1113115989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7008069370 con 0x7f6fec0039b0 2024-04-26T23:05:05.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.631+0000 7f6ffe000640 1 -- 172.21.15.40:0/1113115989 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ff8042020 con 0x7f6fec0039b0 2024-04-26T23:05:05.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.632+0000 7f6ffe000640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6fe406c8f0 0x7f6fe406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:05.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.633+0000 7f6ffe000640 1 -- 172.21.15.40:0/1113115989 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f6ff80abe10 con 0x7f6fec0039b0 2024-04-26T23:05:05.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.633+0000 7f7005400640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6fe406c8f0 0x7f6fe406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:05.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.634+0000 7f7005400640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6fe406c8f0 0x7f6fe406edb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0000c00 tx=0x7f6ff0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:05.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.637+0000 7f6ffe000640 1 -- 172.21.15.40:0/1113115989 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f6ff80af030 con 0x7f6fec0039b0 2024-04-26T23:05:05.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.918+0000 7f7007200640 1 -- 172.21.15.40:0/1113115989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f70080d5db0 con 0x7f6fec0039b0 2024-04-26T23:05:05.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.919+0000 7f6ffe000640 1 -- 172.21.15.40:0/1113115989 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1965 (secure 0 0 0) 0x7f6ff8049030 con 0x7f6fec0039b0 2024-04-26T23:05:05.919 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:05.919 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":265,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":712040,"bytes_used":236589056,"bytes_avail":772857524224,"bytes_total":773094113280,"write_bytes_sec":1279,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":46,"id":3,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":3,"rank":2,"name":"b","status":"up:active","gid":5284},{"filesystem_id":3,"rank":1,"name":"a","status":"up:active","gid":6027},{"filesystem_id":3,"rank":0,"name":"c","status":"up:active","gid":6036}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-04-26T23:04:53.227936+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T23:05:05.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.922+0000 7f7007200640 1 -- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6fe406c8f0 msgr2=0x7f6fe406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:05.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.922+0000 7f7007200640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6fe406c8f0 0x7f6fe406edb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0000c00 tx=0x7f6ff0009040 comp rx=0 tx=0).stop 2024-04-26T23:05:05.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.922+0000 7f7007200640 1 -- 172.21.15.40:0/1113115989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec0039b0 msgr2=0x7f70080ce880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:05.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.922+0000 7f7007200640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec0039b0 0x7f70080ce880 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f6ff802abb0 tx=0x7f6ff8004730 comp rx=0 tx=0).stop 2024-04-26T23:05:05.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.922+0000 7f7007200640 1 -- 172.21.15.40:0/1113115989 shutdown_connections 2024-04-26T23:05:05.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.922+0000 7f7007200640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6fe406c8f0 0x7f6fe406edb0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.922+0000 7f7007200640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f70080cd220 0x7f7008152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.922+0000 7f7007200640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70080b8e70 0x7f70080b2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.922+0000 7f7007200640 1 --2- 172.21.15.40:0/1113115989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fec0039b0 0x7f70080ce880 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:05.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.922+0000 7f7007200640 1 -- 172.21.15.40:0/1113115989 >> 172.21.15.40:0/1113115989 conn(0x7f700809e210 msgr2=0x7f70080b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:05.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.922+0000 7f7007200640 1 -- 172.21.15.40:0/1113115989 shutdown_connections 2024-04-26T23:05:05.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:05.922+0000 7f7007200640 1 -- 172.21.15.40:0/1113115989 wait complete. 2024-04-26T23:05:05.942 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'idle+waiting paths [/d1/d2/d3]', 'b': 'idle+waiting paths [/d1/d2/d3]', 'c': 'idle+waiting paths [/d1/d2/d3]'} 2024-04-26T23:05:06.943 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:05:07.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.163+0000 7fa196e00640 1 Processor -- start 2024-04-26T23:05:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.164+0000 7fa196e00640 1 -- start start 2024-04-26T23:05:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.165+0000 7fa196e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa190073f30 0x7fa1900cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.165+0000 7fa196e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1901671e0 0x7fa190167670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.165+0000 7fa196e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa190163a50 con 0x7fa1900c41d0 2024-04-26T23:05:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.165+0000 7fa196e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa1900cf7a0 con 0x7fa190073f30 2024-04-26T23:05:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.165+0000 7fa196e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa1900ceb90 con 0x7fa1901671e0 2024-04-26T23:05:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.165+0000 7fa18f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa190073f30 0x7fa1900cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.165+0000 7fa18fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa1900c41d0 0x7fa1900cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58172/0 (socket says 172.21.15.40:58172) 2024-04-26T23:05:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.165+0000 7fa18f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa190073f30 0x7fa1900cc240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37170/0 (socket says 172.21.15.40:37170) 2024-04-26T23:05:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.165+0000 7fa18fe00640 1 -- 172.21.15.40:0/3366523067 learned_addr learned my addr 172.21.15.40:0/3366523067 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.166+0000 7fa194e00640 1 --2- 172.21.15.40:0/3366523067 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1901671e0 0x7fa190167670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.166+0000 7fa18f400640 1 -- 172.21.15.40:0/3366523067 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1901671e0 msgr2=0x7fa190167670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.166+0000 7fa18f400640 1 --2- 172.21.15.40:0/3366523067 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1901671e0 0x7fa190167670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.166+0000 7fa18f400640 1 -- 172.21.15.40:0/3366523067 >> v1:172.21.15.40:6789/0 conn(0x7fa1900c41d0 legacy=0x7fa1900cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:07.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.166+0000 7fa18ea00640 1 -- 172.21.15.40:0/3366523067 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3685616328 0 0) 0x7fa190163a50 con 0x7fa1900c41d0 2024-04-26T23:05:07.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.166+0000 7fa18f400640 1 -- 172.21.15.40:0/3366523067 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1900cee70 con 0x7fa190073f30 2024-04-26T23:05:07.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.167+0000 7fa18f400640 1 --2- 172.21.15.40:0/3366523067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa190073f30 0x7fa1900cc240 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0097a0 tx=0x7fa17c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95f7af1498327ecc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:07.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.168+0000 7fa18ea00640 1 -- 172.21.15.40:0/3366523067 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c037070 con 0x7fa190073f30 2024-04-26T23:05:07.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.168+0000 7fa18ea00640 1 -- 172.21.15.40:0/3366523067 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa17c032440 con 0x7fa190073f30 2024-04-26T23:05:07.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.168+0000 7fa18ea00640 1 -- 172.21.15.40:0/3366523067 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c032810 con 0x7fa190073f30 2024-04-26T23:05:07.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.168+0000 7fa196e00640 1 -- 172.21.15.40:0/3366523067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa190073f30 msgr2=0x7fa1900cc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:07.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.168+0000 7fa196e00640 1 --2- 172.21.15.40:0/3366523067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa190073f30 0x7fa1900cc240 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0097a0 tx=0x7fa17c02a180 comp rx=0 tx=0).stop 2024-04-26T23:05:07.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.169+0000 7fa196e00640 1 -- 172.21.15.40:0/3366523067 shutdown_connections 2024-04-26T23:05:07.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.169+0000 7fa196e00640 1 --2- 172.21.15.40:0/3366523067 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1901671e0 0x7fa190167670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.169+0000 7fa196e00640 1 --2- 172.21.15.40:0/3366523067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa190073f30 0x7fa1900cc240 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.169+0000 7fa196e00640 1 -- 172.21.15.40:0/3366523067 >> 172.21.15.40:0/3366523067 conn(0x7fa1900d0be0 msgr2=0x7fa1900d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:07.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.169+0000 7fa196e00640 1 -- 172.21.15.40:0/3366523067 shutdown_connections 2024-04-26T23:05:07.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.169+0000 7fa196e00640 1 -- 172.21.15.40:0/3366523067 wait complete. 2024-04-26T23:05:07.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.170+0000 7fa196e00640 1 Processor -- start 2024-04-26T23:05:07.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.171+0000 7fa196e00640 1 -- start start 2024-04-26T23:05:07.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa196e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa190073f30 0x7fa1900cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:07.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa196e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1900c41d0 0x7fa1900bff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:07.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa196e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1901671e0 0x7fa190161310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:07.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa196e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa1900ce7b0 con 0x7fa1900c41d0 2024-04-26T23:05:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa196e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1900cf430 con 0x7fa190073f30 2024-04-26T23:05:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa196e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa1900ce930 con 0x7fa1901671e0 2024-04-26T23:05:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa18f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1900c41d0 0x7fa1900bff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa18f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1900c41d0 0x7fa1900bff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37180/0 (socket says 172.21.15.40:37180) 2024-04-26T23:05:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa18f400640 1 -- 172.21.15.40:0/1957465498 learned_addr learned my addr 172.21.15.40:0/1957465498 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa18f400640 1 -- 172.21.15.40:0/1957465498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1901671e0 msgr2=0x7fa190161310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa18f400640 1 --2- 172.21.15.40:0/1957465498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1901671e0 0x7fa190161310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa18f400640 1 -- 172.21.15.40:0/1957465498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa190073f30 msgr2=0x7fa1900cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa18f400640 1 --2- 172.21.15.40:0/1957465498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa190073f30 0x7fa1900cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.172+0000 7fa18f400640 1 -- 172.21.15.40:0/1957465498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1900c0460 con 0x7fa1900c41d0 2024-04-26T23:05:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.173+0000 7fa18f400640 1 --2- 172.21.15.40:0/1957465498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1900c41d0 0x7fa1900bff20 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0098c0 tx=0x7fa17c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.173+0000 7fa187e00640 1 -- 172.21.15.40:0/1957465498 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c037040 con 0x7fa1900c41d0 2024-04-26T23:05:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.173+0000 7fa187e00640 1 -- 172.21.15.40:0/1957465498 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa17c030770 con 0x7fa1900c41d0 2024-04-26T23:05:07.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.173+0000 7fa187e00640 1 -- 172.21.15.40:0/1957465498 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c030b40 con 0x7fa1900c41d0 2024-04-26T23:05:07.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.173+0000 7fa196e00640 1 -- 172.21.15.40:0/1957465498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa190165710 con 0x7fa1900c41d0 2024-04-26T23:05:07.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.173+0000 7fa196e00640 1 -- 172.21.15.40:0/1957465498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa190165cd0 con 0x7fa1900c41d0 2024-04-26T23:05:07.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.174+0000 7fa196e00640 1 -- 172.21.15.40:0/1957465498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1900694f0 con 0x7fa1900c41d0 2024-04-26T23:05:07.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.176+0000 7fa187e00640 1 -- 172.21.15.40:0/1957465498 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa17c044020 con 0x7fa1900c41d0 2024-04-26T23:05:07.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.176+0000 7fa187e00640 1 --2- 172.21.15.40:0/1957465498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa17006c8f0 0x7fa17006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:07.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.177+0000 7fa187e00640 1 -- 172.21.15.40:0/1957465498 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7fa17c0ac2c0 con 0x7fa1900c41d0 2024-04-26T23:05:07.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.177+0000 7fa18fe00640 1 --2- 172.21.15.40:0/1957465498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa17006c8f0 0x7fa17006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:07.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.179+0000 7fa18fe00640 1 --2- 172.21.15.40:0/1957465498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa17006c8f0 0x7fa17006edb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa180005880 tx=0x7fa180007950 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:07.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.181+0000 7fa187e00640 1 -- 172.21.15.40:0/1957465498 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa17c032e20 con 0x7fa1900c41d0 2024-04-26T23:05:07.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.430+0000 7fa196e00640 1 -- 172.21.15.40:0/1957465498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1900d0430 con 0x7fa1900c41d0 2024-04-26T23:05:07.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.431+0000 7fa187e00640 1 -- 172.21.15.40:0/1957465498 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5225 (secure 0 0 0) 0x7fa17c07a490 con 0x7fa1900c41d0 2024-04-26T23:05:07.432 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:07.432 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":46,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/189004193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":189004193},{"type":"v1","addr":"172.21.15.103:6835","nonce":189004193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:04:16.558591+0000","modified":"2024-04-26T23:04:55.194872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6036,"mds_1":6027,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"b","rank":2,"incarnation":39,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/670995452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":670995452},{"type":"v1","addr":"172.21.15.103:6837","nonce":670995452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6027":{"gid":6027,"name":"a","rank":1,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6834/2304583125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2304583125},{"type":"v1","addr":"172.21.15.40:6834","nonce":2304583125}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6036":{"gid":6036,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/618018110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":618018110},{"type":"v1","addr":"172.21.15.40:6835","nonce":618018110}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6036,"qdb_cluster":[5284,6027,6036]},"id":3}]} 2024-04-26T23:05:07.432 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 46 2024-04-26T23:05:07.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.434+0000 7fa196e00640 1 -- 172.21.15.40:0/1957465498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa17006c8f0 msgr2=0x7fa17006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:07.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.434+0000 7fa196e00640 1 --2- 172.21.15.40:0/1957465498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa17006c8f0 0x7fa17006edb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa180005880 tx=0x7fa180007950 comp rx=0 tx=0).stop 2024-04-26T23:05:07.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.434+0000 7fa196e00640 1 -- 172.21.15.40:0/1957465498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1900c41d0 msgr2=0x7fa1900bff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:07.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.434+0000 7fa196e00640 1 --2- 172.21.15.40:0/1957465498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1900c41d0 0x7fa1900bff20 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0098c0 tx=0x7fa17c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.434+0000 7fa196e00640 1 -- 172.21.15.40:0/1957465498 shutdown_connections 2024-04-26T23:05:07.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.434+0000 7fa196e00640 1 --2- 172.21.15.40:0/1957465498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa17006c8f0 0x7fa17006edb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.434+0000 7fa196e00640 1 --2- 172.21.15.40:0/1957465498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1901671e0 0x7fa190161310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.434+0000 7fa196e00640 1 --2- 172.21.15.40:0/1957465498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1900c41d0 0x7fa1900bff20 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.434+0000 7fa196e00640 1 --2- 172.21.15.40:0/1957465498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa190073f30 0x7fa1900cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.434+0000 7fa196e00640 1 -- 172.21.15.40:0/1957465498 >> 172.21.15.40:0/1957465498 conn(0x7fa1900d0be0 msgr2=0x7fa1901631c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:07.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.435+0000 7fa196e00640 1 -- 172.21.15.40:0/1957465498 shutdown_connections 2024-04-26T23:05:07.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.435+0000 7fa196e00640 1 -- 172.21.15.40:0/1957465498 wait complete. 2024-04-26T23:05:07.456 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'b'] 2024-04-26T23:05:07.456 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:05:07.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.667+0000 7f3365200640 1 Processor -- start 2024-04-26T23:05:07.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.667+0000 7f3365200640 1 -- start start 2024-04-26T23:05:07.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.668+0000 7f3365200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f336015c160 0x7f33600b4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:07.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.668+0000 7f3365200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f336015c5c0 0x7f33600b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:07.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.668+0000 7f3365200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f336015cb90 con 0x7f3360073e10 2024-04-26T23:05:07.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.668+0000 7f3365200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3360153040 con 0x7f336015c160 2024-04-26T23:05:07.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.668+0000 7f3365200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f336015fc40 con 0x7f336015c5c0 2024-04-26T23:05:07.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.669+0000 7f335ea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f336015c160 0x7f33600b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:07.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.669+0000 7f335fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f336015c5c0 0x7f33600b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:07.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.669+0000 7f335ea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f336015c160 0x7f33600b4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46850/0 (socket says 172.21.15.40:46850) 2024-04-26T23:05:07.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.669+0000 7f335ea00640 1 -- 172.21.15.40:0/3172291717 learned_addr learned my addr 172.21.15.40:0/3172291717 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:07.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.669+0000 7f335f400640 1 --1- 172.21.15.40:0/3172291717 >> v1:172.21.15.103:6789/0 conn(0x7f3360073e10 0x7f33601600d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:51044/0 (socket says 172.21.15.40:51044) 2024-04-26T23:05:07.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.670+0000 7f335e000640 1 -- 172.21.15.40:0/3172291717 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147276202 0 0) 0x7f336015cb90 con 0x7f3360073e10 2024-04-26T23:05:07.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.670+0000 7f335e000640 1 -- 172.21.15.40:0/3172291717 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3344005730 con 0x7f3360073e10 2024-04-26T23:05:07.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.670+0000 7f335fe00640 1 -- 172.21.15.40:0/3172291717 >> v2:172.21.15.103:3300/0 conn(0x7f336015c160 msgr2=0x7f33600b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:07.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.670+0000 7f335fe00640 1 --2- 172.21.15.40:0/3172291717 >> v2:172.21.15.103:3300/0 conn(0x7f336015c160 0x7f33600b4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.670+0000 7f335fe00640 1 -- 172.21.15.40:0/3172291717 >> v1:172.21.15.103:6789/0 conn(0x7f3360073e10 legacy=0x7f33601600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:07.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.670+0000 7f335fe00640 1 -- 172.21.15.40:0/3172291717 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f336015ce70 con 0x7f336015c5c0 2024-04-26T23:05:07.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.671+0000 7f335fe00640 1 --2- 172.21.15.40:0/3172291717 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f336015c5c0 0x7f33600b4d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f33580097a0 tx=0x7f335802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73ef424cf218506c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:07.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.672+0000 7f335e000640 1 -- 172.21.15.40:0/3172291717 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3358037070 con 0x7f336015c5c0 2024-04-26T23:05:07.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.672+0000 7f335e000640 1 -- 172.21.15.40:0/3172291717 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3358032440 con 0x7f336015c5c0 2024-04-26T23:05:07.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.672+0000 7f335e000640 1 -- 172.21.15.40:0/3172291717 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3358032810 con 0x7f336015c5c0 2024-04-26T23:05:07.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.673+0000 7f3365200640 1 -- 172.21.15.40:0/3172291717 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f336015c5c0 msgr2=0x7f33600b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:07.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.673+0000 7f3365200640 1 --2- 172.21.15.40:0/3172291717 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f336015c5c0 0x7f33600b4d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f33580097a0 tx=0x7f335802a180 comp rx=0 tx=0).stop 2024-04-26T23:05:07.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.673+0000 7f3365200640 1 -- 172.21.15.40:0/3172291717 shutdown_connections 2024-04-26T23:05:07.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.673+0000 7f3365200640 1 --2- 172.21.15.40:0/3172291717 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f336015c5c0 0x7f33600b4d80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.673+0000 7f3365200640 1 --2- 172.21.15.40:0/3172291717 >> v2:172.21.15.103:3300/0 conn(0x7f336015c160 0x7f33600b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.673+0000 7f3365200640 1 -- 172.21.15.40:0/3172291717 >> 172.21.15.40:0/3172291717 conn(0x7f3360075eb0 msgr2=0x7f3360068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:07.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.673+0000 7f3365200640 1 -- 172.21.15.40:0/3172291717 shutdown_connections 2024-04-26T23:05:07.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.673+0000 7f3365200640 1 -- 172.21.15.40:0/3172291717 wait complete. 2024-04-26T23:05:07.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.675+0000 7f3365200640 1 Processor -- start 2024-04-26T23:05:07.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.675+0000 7f3365200640 1 -- start start 2024-04-26T23:05:07.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.676+0000 7f3365200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3360073e10 0x7f336014df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:07.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.676+0000 7f3365200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f336015c160 0x7f3360156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:07.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.676+0000 7f3365200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f336015c5c0 0x7f336014ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:07.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.676+0000 7f3365200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3360073540 con 0x7f336015c160 2024-04-26T23:05:07.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.676+0000 7f3365200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33600733c0 con 0x7f336015c5c0 2024-04-26T23:05:07.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.676+0000 7f3365200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f336015d150 con 0x7f3360073e10 2024-04-26T23:05:07.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.676+0000 7f335ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f336015c160 0x7f3360156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:07.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.677+0000 7f335ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f336015c160 0x7f3360156b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37182/0 (socket says 172.21.15.40:37182) 2024-04-26T23:05:07.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.677+0000 7f335ea00640 1 -- 172.21.15.40:0/1403604338 learned_addr learned my addr 172.21.15.40:0/1403604338 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:07.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.677+0000 7f335f400640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3360073e10 0x7f336014df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:07.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.677+0000 7f335fe00640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f336015c5c0 0x7f336014ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:07.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.677+0000 7f335ea00640 1 -- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3360073e10 msgr2=0x7f336014df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:07.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.677+0000 7f335ea00640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3360073e10 0x7f336014df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.677+0000 7f335ea00640 1 -- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f336015c5c0 msgr2=0x7f336014ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:07.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.677+0000 7f335ea00640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f336015c5c0 0x7f336014ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.677+0000 7f335ea00640 1 -- 172.21.15.40:0/1403604338 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3360153290 con 0x7f336015c160 2024-04-26T23:05:07.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.677+0000 7f335ea00640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f336015c160 0x7f3360156b00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3348009fd0 tx=0x7f334800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:07.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.678+0000 7f3357400640 1 -- 172.21.15.40:0/1403604338 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3348017070 con 0x7f336015c160 2024-04-26T23:05:07.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.678+0000 7f3357400640 1 -- 172.21.15.40:0/1403604338 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f334800c940 con 0x7f336015c160 2024-04-26T23:05:07.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.678+0000 7f3357400640 1 -- 172.21.15.40:0/1403604338 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334800cd10 con 0x7f336015c160 2024-04-26T23:05:07.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.678+0000 7f3365200640 1 -- 172.21.15.40:0/1403604338 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f336014ea50 con 0x7f336015c160 2024-04-26T23:05:07.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.678+0000 7f3365200640 1 -- 172.21.15.40:0/1403604338 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3360153500 con 0x7f336015c160 2024-04-26T23:05:07.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.679+0000 7f3365200640 1 -- 172.21.15.40:0/1403604338 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3360069370 con 0x7f336015c160 2024-04-26T23:05:07.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.680+0000 7f3357400640 1 -- 172.21.15.40:0/1403604338 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f334801a070 con 0x7f336015c160 2024-04-26T23:05:07.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.681+0000 7f3357400640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f333c06c8f0 0x7f333c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:07.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.681+0000 7f3357400640 1 -- 172.21.15.40:0/1403604338 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f334808bcf0 con 0x7f336015c160 2024-04-26T23:05:07.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.682+0000 7f335f400640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f333c06c8f0 0x7f333c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:07.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.684+0000 7f335f400640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f333c06c8f0 0x7f333c06edb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3350005fd0 tx=0x7f3350007e90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:07.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.686+0000 7f3357400640 1 -- 172.21.15.40:0/1403604338 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f336014ea50 con 0x7f336015c160 2024-04-26T23:05:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.962+0000 7f3365200640 1 -- 172.21.15.40:0/1403604338 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f336006f160 con 0x7f336015c160 2024-04-26T23:05:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.962+0000 7f3357400640 1 -- 172.21.15.40:0/1403604338 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1917 (secure 0 0 0) 0x7f3348059ec0 con 0x7f336015c160 2024-04-26T23:05:07.963 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:07.963 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":267,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":712040,"bytes_used":236589056,"bytes_avail":772857524224,"bytes_total":773094113280,"write_bytes_sec":597,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":46,"id":3,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":3,"rank":2,"name":"b","status":"up:active","gid":5284},{"filesystem_id":3,"rank":1,"name":"a","status":"up:active","gid":6027},{"filesystem_id":3,"rank":0,"name":"c","status":"up:active","gid":6036}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-04-26T23:05:07.233598+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T23:05:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.965+0000 7f3365200640 1 -- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f333c06c8f0 msgr2=0x7f333c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.965+0000 7f3365200640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f333c06c8f0 0x7f333c06edb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3350005fd0 tx=0x7f3350007e90 comp rx=0 tx=0).stop 2024-04-26T23:05:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.965+0000 7f3365200640 1 -- 172.21.15.40:0/1403604338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f336015c160 msgr2=0x7f3360156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.965+0000 7f3365200640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f336015c160 0x7f3360156b00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3348009fd0 tx=0x7f334800ed40 comp rx=0 tx=0).stop 2024-04-26T23:05:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.965+0000 7f3365200640 1 -- 172.21.15.40:0/1403604338 shutdown_connections 2024-04-26T23:05:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.965+0000 7f3365200640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f333c06c8f0 0x7f333c06edb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.965+0000 7f3365200640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f336015c5c0 0x7f336014ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.965+0000 7f3365200640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f336015c160 0x7f3360156b00 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.965+0000 7f3365200640 1 --2- 172.21.15.40:0/1403604338 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3360073e10 0x7f336014df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:07.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.965+0000 7f3365200640 1 -- 172.21.15.40:0/1403604338 >> 172.21.15.40:0/1403604338 conn(0x7f3360075eb0 msgr2=0x7f336015b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:07.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.965+0000 7f3365200640 1 -- 172.21.15.40:0/1403604338 shutdown_connections 2024-04-26T23:05:07.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:07.965+0000 7f3365200640 1 -- 172.21.15.40:0/1403604338 wait complete. 2024-04-26T23:05:07.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'b': 'idle+waiting paths [/d1/d2/d3]', 'c': 'idle+waiting paths [/d1/d2/d3]'} 2024-04-26T23:05:08.985 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:05:09.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.213+0000 7ff13be00640 1 Processor -- start 2024-04-26T23:05:09.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.213+0000 7ff13be00640 1 -- start start 2024-04-26T23:05:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.214+0000 7ff13be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff13c0b6220 0x7ff13c0d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.214+0000 7ff13be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff13c0d08d0 0x7ff13c0d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.214+0000 7ff13be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff13c0d3820 0x7ff13c0b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.214+0000 7ff13be00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff13c162c40 con 0x7ff13c0d08d0 2024-04-26T23:05:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.214+0000 7ff13be00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff13c0d2d80 con 0x7ff13c0b6220 2024-04-26T23:05:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.214+0000 7ff13be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff13c0b7c80 con 0x7ff13c0d3820 2024-04-26T23:05:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.215+0000 7ff13a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff13c0d08d0 0x7ff13c0d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.215+0000 7ff13a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff13c0d08d0 0x7ff13c0d32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37194/0 (socket says 172.21.15.40:37194) 2024-04-26T23:05:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.215+0000 7ff13a000640 1 -- 172.21.15.40:0/2709748042 learned_addr learned my addr 172.21.15.40:0/2709748042 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.215+0000 7ff13a000640 1 -- 172.21.15.40:0/2709748042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff13c0d3820 msgr2=0x7ff13c0b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.215+0000 7ff13a000640 1 --2- 172.21.15.40:0/2709748042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff13c0d3820 0x7ff13c0b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.215+0000 7ff13a000640 1 -- 172.21.15.40:0/2709748042 >> v2:172.21.15.103:3300/0 conn(0x7ff13c0b6220 msgr2=0x7ff13c0d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.215+0000 7ff13aa00640 1 --2- 172.21.15.40:0/2709748042 >> v2:172.21.15.103:3300/0 conn(0x7ff13c0b6220 0x7ff13c0d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.215+0000 7ff13a000640 1 --2- 172.21.15.40:0/2709748042 >> v2:172.21.15.103:3300/0 conn(0x7ff13c0b6220 0x7ff13c0d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.215+0000 7ff13a000640 1 -- 172.21.15.40:0/2709748042 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13c0b7f60 con 0x7ff13c0d08d0 2024-04-26T23:05:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.216+0000 7ff13a000640 1 --2- 172.21.15.40:0/2709748042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13c0d08d0 0x7ff13c0d32e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0d4cf0 tx=0x7ff12c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175cb011aee474d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.217+0000 7ff139600640 1 -- 172.21.15.40:0/2709748042 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c037070 con 0x7ff13c0d08d0 2024-04-26T23:05:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.217+0000 7ff139600640 1 -- 172.21.15.40:0/2709748042 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff12c032440 con 0x7ff13c0d08d0 2024-04-26T23:05:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.217+0000 7ff139600640 1 -- 172.21.15.40:0/2709748042 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c032810 con 0x7ff13c0d08d0 2024-04-26T23:05:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.218+0000 7ff13be00640 1 -- 172.21.15.40:0/2709748042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13c0d08d0 msgr2=0x7ff13c0d32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.218+0000 7ff13be00640 1 --2- 172.21.15.40:0/2709748042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13c0d08d0 0x7ff13c0d32e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0d4cf0 tx=0x7ff12c02a1e0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.218+0000 7ff13be00640 1 -- 172.21.15.40:0/2709748042 shutdown_connections 2024-04-26T23:05:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.218+0000 7ff13be00640 1 --2- 172.21.15.40:0/2709748042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff13c0d3820 0x7ff13c0b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.218+0000 7ff13be00640 1 --2- 172.21.15.40:0/2709748042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13c0d08d0 0x7ff13c0d32e0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.218+0000 7ff13be00640 1 --2- 172.21.15.40:0/2709748042 >> v2:172.21.15.103:3300/0 conn(0x7ff13c0b6220 0x7ff13c0d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.218+0000 7ff13be00640 1 -- 172.21.15.40:0/2709748042 >> 172.21.15.40:0/2709748042 conn(0x7ff13c0d4d90 msgr2=0x7ff13c0d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.218+0000 7ff13be00640 1 -- 172.21.15.40:0/2709748042 shutdown_connections 2024-04-26T23:05:09.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.218+0000 7ff13be00640 1 -- 172.21.15.40:0/2709748042 wait complete. 2024-04-26T23:05:09.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.220+0000 7ff13be00640 1 Processor -- start 2024-04-26T23:05:09.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.220+0000 7ff13be00640 1 -- start start 2024-04-26T23:05:09.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.222+0000 7ff13be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13c0b6220 0x7ff13c1540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:09.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.222+0000 7ff13be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff13c0d08d0 0x7ff13c1f8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:09.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.222+0000 7ff13be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff13c0d3820 0x7ff13c0b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:09.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.222+0000 7ff13be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c0b6d50 con 0x7ff13c0b6220 2024-04-26T23:05:09.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.222+0000 7ff13be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c0b8520 con 0x7ff13c0d3820 2024-04-26T23:05:09.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.222+0000 7ff13be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff13c0b6ed0 con 0x7ff13c0d08d0 2024-04-26T23:05:09.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.222+0000 7ff13aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13c0b6220 0x7ff13c1540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:09.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.222+0000 7ff13aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13c0b6220 0x7ff13c1540f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37196/0 (socket says 172.21.15.40:37196) 2024-04-26T23:05:09.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.222+0000 7ff13aa00640 1 -- 172.21.15.40:0/2787278851 learned_addr learned my addr 172.21.15.40:0/2787278851 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:09.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.223+0000 7ff13a000640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff13c0d08d0 0x7ff13c1f8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:09.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.223+0000 7ff13b400640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff13c0d3820 0x7ff13c0b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:09.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.223+0000 7ff13aa00640 1 -- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff13c0d08d0 msgr2=0x7ff13c1f8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:09.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.223+0000 7ff13aa00640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff13c0d08d0 0x7ff13c1f8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.223+0000 7ff13aa00640 1 -- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff13c0d3820 msgr2=0x7ff13c0b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:09.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.223+0000 7ff13aa00640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff13c0d3820 0x7ff13c0b51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.223+0000 7ff13aa00640 1 -- 172.21.15.40:0/2787278851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13c1f8b50 con 0x7ff13c0b6220 2024-04-26T23:05:09.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.223+0000 7ff13aa00640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13c0b6220 0x7ff13c1540f0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ff12400c7f0 tx=0x7ff12400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:09.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.223+0000 7ff132a00640 1 -- 172.21.15.40:0/2787278851 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124015070 con 0x7ff13c0b6220 2024-04-26T23:05:09.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.223+0000 7ff132a00640 1 -- 172.21.15.40:0/2787278851 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff124004240 con 0x7ff13c0b6220 2024-04-26T23:05:09.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.223+0000 7ff132a00640 1 -- 172.21.15.40:0/2787278851 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124007620 con 0x7ff13c0b6220 2024-04-26T23:05:09.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.223+0000 7ff13be00640 1 -- 172.21.15.40:0/2787278851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13c1f8e30 con 0x7ff13c0b6220 2024-04-26T23:05:09.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.224+0000 7ff13be00640 1 -- 172.21.15.40:0/2787278851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff13c0b49f0 con 0x7ff13c0b6220 2024-04-26T23:05:09.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.225+0000 7ff13be00640 1 -- 172.21.15.40:0/2787278851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff13c069370 con 0x7ff13c0b6220 2024-04-26T23:05:09.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.226+0000 7ff132a00640 1 -- 172.21.15.40:0/2787278851 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff1240079d0 con 0x7ff13c0b6220 2024-04-26T23:05:09.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.227+0000 7ff132a00640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff11806c940 0x7ff11806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:09.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.227+0000 7ff132a00640 1 -- 172.21.15.40:0/2787278851 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7ff12408c120 con 0x7ff13c0b6220 2024-04-26T23:05:09.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.227+0000 7ff13a000640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff11806c940 0x7ff11806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:09.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.229+0000 7ff13a000640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff11806c940 0x7ff11806ee00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff12c034000 tx=0x7ff12c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:09.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.231+0000 7ff132a00640 1 -- 172.21.15.40:0/2787278851 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff12405a400 con 0x7ff13c0b6220 2024-04-26T23:05:09.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.467+0000 7ff13be00640 1 -- 172.21.15.40:0/2787278851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff13c0d30c0 con 0x7ff13c0b6220 2024-04-26T23:05:09.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.468+0000 7ff132a00640 1 -- 172.21.15.40:0/2787278851 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5225 (secure 0 0 0) 0x7ff124059d00 con 0x7ff13c0b6220 2024-04-26T23:05:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:09.468 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":46,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/189004193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":189004193},{"type":"v1","addr":"172.21.15.103:6835","nonce":189004193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:04:16.558591+0000","modified":"2024-04-26T23:04:55.194872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6036,"mds_1":6027,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"b","rank":2,"incarnation":39,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/670995452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":670995452},{"type":"v1","addr":"172.21.15.103:6837","nonce":670995452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6027":{"gid":6027,"name":"a","rank":1,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6834/2304583125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2304583125},{"type":"v1","addr":"172.21.15.40:6834","nonce":2304583125}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6036":{"gid":6036,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/618018110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":618018110},{"type":"v1","addr":"172.21.15.40:6835","nonce":618018110}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6036,"qdb_cluster":[5284,6027,6036]},"id":3}]} 2024-04-26T23:05:09.469 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 46 2024-04-26T23:05:09.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.470+0000 7ff13be00640 1 -- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff11806c940 msgr2=0x7ff11806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:09.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.470+0000 7ff13be00640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff11806c940 0x7ff11806ee00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff12c034000 tx=0x7ff12c02e000 comp rx=0 tx=0).stop 2024-04-26T23:05:09.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.470+0000 7ff13be00640 1 -- 172.21.15.40:0/2787278851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13c0b6220 msgr2=0x7ff13c1540f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:09.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.470+0000 7ff13be00640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13c0b6220 0x7ff13c1540f0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ff12400c7f0 tx=0x7ff12400ccb0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.471+0000 7ff13be00640 1 -- 172.21.15.40:0/2787278851 shutdown_connections 2024-04-26T23:05:09.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.471+0000 7ff13be00640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff11806c940 0x7ff11806ee00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.471+0000 7ff13be00640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff13c0d3820 0x7ff13c0b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.471+0000 7ff13be00640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff13c0d08d0 0x7ff13c1f8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.471+0000 7ff13be00640 1 --2- 172.21.15.40:0/2787278851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13c0b6220 0x7ff13c1540f0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.471+0000 7ff13be00640 1 -- 172.21.15.40:0/2787278851 >> 172.21.15.40:0/2787278851 conn(0x7ff13c0d4d90 msgr2=0x7ff13c1623b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:09.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.471+0000 7ff13be00640 1 -- 172.21.15.40:0/2787278851 shutdown_connections 2024-04-26T23:05:09.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.471+0000 7ff13be00640 1 -- 172.21.15.40:0/2787278851 wait complete. 2024-04-26T23:05:09.489 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'b'] 2024-04-26T23:05:09.489 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:05:09.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.683+0000 7fcd63200640 1 Processor -- start 2024-04-26T23:05:09.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.684+0000 7fcd63200640 1 -- start start 2024-04-26T23:05:09.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.685+0000 7fcd63200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcd5c0d17d0 0x7fcd5c0b6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:09.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.685+0000 7fcd63200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd5c0b66c0 con 0x7fcd5c0d0c80 2024-04-26T23:05:09.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.685+0000 7fcd63200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd5c15f9c0 con 0x7fcd5c0d2400 2024-04-26T23:05:09.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.685+0000 7fcd63200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fcd5c0d37b0 con 0x7fcd5c0d17d0 2024-04-26T23:05:09.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.686+0000 7fcd61e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcd5c0d17d0 0x7fcd5c0b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:09.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.686+0000 7fcd61e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcd5c0d17d0 0x7fcd5c0b6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37208/0 (socket says 172.21.15.40:37208) 2024-04-26T23:05:09.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.686+0000 7fcd61e00640 1 -- 172.21.15.40:0/535077189 learned_addr learned my addr 172.21.15.40:0/535077189 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:09.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.686+0000 7fcd61e00640 1 -- 172.21.15.40:0/535077189 >> v1:172.21.15.103:6789/0 conn(0x7fcd5c0d2400 legacy=0x7fcd5c0d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:09.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.686+0000 7fcd61e00640 1 -- 172.21.15.40:0/535077189 >> v1:172.21.15.40:6789/0 conn(0x7fcd5c0d0c80 legacy=0x7fcd5c0b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:09.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.686+0000 7fcd61e00640 1 -- 172.21.15.40:0/535077189 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5c0d5730 con 0x7fcd5c0d17d0 2024-04-26T23:05:09.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.687+0000 7fcd61e00640 1 --2- 172.21.15.40:0/535077189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd5c0d17d0 0x7fcd5c0b6180 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fcd54009e80 tx=0x7fcd5402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac596d8714e87c00 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:09.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.688+0000 7fcd60a00640 1 -- 172.21.15.40:0/535077189 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd54036070 con 0x7fcd5c0d17d0 2024-04-26T23:05:09.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.688+0000 7fcd60a00640 1 -- 172.21.15.40:0/535077189 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd54031600 con 0x7fcd5c0d17d0 2024-04-26T23:05:09.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.688+0000 7fcd60a00640 1 -- 172.21.15.40:0/535077189 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd540319d0 con 0x7fcd5c0d17d0 2024-04-26T23:05:09.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.688+0000 7fcd63200640 1 -- 172.21.15.40:0/535077189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd5c0d17d0 msgr2=0x7fcd5c0b6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:09.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.688+0000 7fcd63200640 1 --2- 172.21.15.40:0/535077189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd5c0d17d0 0x7fcd5c0b6180 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fcd54009e80 tx=0x7fcd5402a710 comp rx=0 tx=0).stop 2024-04-26T23:05:09.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.688+0000 7fcd63200640 1 -- 172.21.15.40:0/535077189 shutdown_connections 2024-04-26T23:05:09.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.688+0000 7fcd63200640 1 --2- 172.21.15.40:0/535077189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd5c0d17d0 0x7fcd5c0b6180 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.688+0000 7fcd63200640 1 -- 172.21.15.40:0/535077189 >> 172.21.15.40:0/535077189 conn(0x7fcd5c09e210 msgr2=0x7fcd5c0d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:09.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.688+0000 7fcd63200640 1 -- 172.21.15.40:0/535077189 shutdown_connections 2024-04-26T23:05:09.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.688+0000 7fcd63200640 1 -- 172.21.15.40:0/535077189 wait complete. 2024-04-26T23:05:09.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.690+0000 7fcd63200640 1 Processor -- start 2024-04-26T23:05:09.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.690+0000 7fcd63200640 1 -- start start 2024-04-26T23:05:09.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.691+0000 7fcd63200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd5c0d0c80 0x7fcd5c1522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:09.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.691+0000 7fcd63200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd5c0d17d0 0x7fcd5c1f56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:09.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.691+0000 7fcd63200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd5c0d2400 0x7fcd5c15bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:09.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.691+0000 7fcd63200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c0d3a00 con 0x7fcd5c0d2400 2024-04-26T23:05:09.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.691+0000 7fcd63200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c0d3d70 con 0x7fcd5c0d0c80 2024-04-26T23:05:09.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.691+0000 7fcd63200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcd5c0d10f0 con 0x7fcd5c0d17d0 2024-04-26T23:05:09.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.691+0000 7fcd62800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd5c0d2400 0x7fcd5c15bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:09.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.692+0000 7fcd62800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd5c0d2400 0x7fcd5c15bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37218/0 (socket says 172.21.15.40:37218) 2024-04-26T23:05:09.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.692+0000 7fcd62800640 1 -- 172.21.15.40:0/606335662 learned_addr learned my addr 172.21.15.40:0/606335662 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:09.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.692+0000 7fcd61e00640 1 --2- 172.21.15.40:0/606335662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd5c0d0c80 0x7fcd5c1522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:09.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.692+0000 7fcd62800640 1 -- 172.21.15.40:0/606335662 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd5c0d17d0 msgr2=0x7fcd5c1f56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:09.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.692+0000 7fcd62800640 1 --2- 172.21.15.40:0/606335662 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd5c0d17d0 0x7fcd5c1f56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.692+0000 7fcd62800640 1 -- 172.21.15.40:0/606335662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd5c0d0c80 msgr2=0x7fcd5c1522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.692+0000 7fcd62800640 1 --2- 172.21.15.40:0/606335662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd5c0d0c80 0x7fcd5c1522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.692+0000 7fcd62800640 1 -- 172.21.15.40:0/606335662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5c0cbf70 con 0x7fcd5c0d2400 2024-04-26T23:05:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.692+0000 7fcd62800640 1 --2- 172.21.15.40:0/606335662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd5c0d2400 0x7fcd5c15bd40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800c3c0 tx=0x7fcd5800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.692+0000 7fcd52000640 1 -- 172.21.15.40:0/606335662 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd580052d0 con 0x7fcd5c0d2400 2024-04-26T23:05:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.693+0000 7fcd52000640 1 -- 172.21.15.40:0/606335662 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd58005470 con 0x7fcd5c0d2400 2024-04-26T23:05:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.693+0000 7fcd63200640 1 -- 172.21.15.40:0/606335662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd5c0cc250 con 0x7fcd5c0d2400 2024-04-26T23:05:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.693+0000 7fcd52000640 1 -- 172.21.15.40:0/606335662 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd58007240 con 0x7fcd5c0d2400 2024-04-26T23:05:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.693+0000 7fcd63200640 1 -- 172.21.15.40:0/606335662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd5c0cc810 con 0x7fcd5c0d2400 2024-04-26T23:05:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.694+0000 7fcd63200640 1 -- 172.21.15.40:0/606335662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd5c069370 con 0x7fcd5c0d2400 2024-04-26T23:05:09.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.695+0000 7fcd52000640 1 -- 172.21.15.40:0/606335662 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcd580073e0 con 0x7fcd5c0d2400 2024-04-26T23:05:09.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.696+0000 7fcd52000640 1 --2- 172.21.15.40:0/606335662 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd4006c8f0 0x7fcd4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:09.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.696+0000 7fcd52000640 1 -- 172.21.15.40:0/606335662 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7fcd5808b0e0 con 0x7fcd5c0d2400 2024-04-26T23:05:09.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.696+0000 7fcd61e00640 1 --2- 172.21.15.40:0/606335662 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd4006c8f0 0x7fcd4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.698+0000 7fcd61e00640 1 --2- 172.21.15.40:0/606335662 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd4006c8f0 0x7fcd4006edb0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcd54033000 tx=0x7fcd54009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.700+0000 7fcd52000640 1 -- 172.21.15.40:0/606335662 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fcd5c0b66c0 con 0x7fcd5c0d2400 2024-04-26T23:05:09.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.975+0000 7fcd63200640 1 -- 172.21.15.40:0/606335662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fcd5c0d5db0 con 0x7fcd5c0d2400 2024-04-26T23:05:09.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.976+0000 7fcd52000640 1 -- 172.21.15.40:0/606335662 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1823 (secure 0 0 0) 0x7fcd58059340 con 0x7fcd5c0d2400 2024-04-26T23:05:09.976 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:09.976 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":269,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":712040,"bytes_used":236589056,"bytes_avail":772857524224,"bytes_total":773094113280,"write_bytes_sec":597,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":46,"id":3,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":3,"rank":2,"name":"b","status":"up:active","gid":5284},{"filesystem_id":3,"rank":1,"name":"a","status":"up:active","gid":6027},{"filesystem_id":3,"rank":0,"name":"c","status":"up:active","gid":6036}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-04-26T23:05:09.234606+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T23:05:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.979+0000 7fcd63200640 1 -- 172.21.15.40:0/606335662 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd4006c8f0 msgr2=0x7fcd4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.979+0000 7fcd63200640 1 --2- 172.21.15.40:0/606335662 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd4006c8f0 0x7fcd4006edb0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcd54033000 tx=0x7fcd54009000 comp rx=0 tx=0).stop 2024-04-26T23:05:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.979+0000 7fcd63200640 1 -- 172.21.15.40:0/606335662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd5c0d2400 msgr2=0x7fcd5c15bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.979+0000 7fcd63200640 1 --2- 172.21.15.40:0/606335662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd5c0d2400 0x7fcd5c15bd40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800c3c0 tx=0x7fcd5800c880 comp rx=0 tx=0).stop 2024-04-26T23:05:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.979+0000 7fcd63200640 1 -- 172.21.15.40:0/606335662 shutdown_connections 2024-04-26T23:05:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.979+0000 7fcd63200640 1 --2- 172.21.15.40:0/606335662 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd4006c8f0 0x7fcd4006edb0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.979+0000 7fcd63200640 1 --2- 172.21.15.40:0/606335662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd5c0d2400 0x7fcd5c15bd40 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.979+0000 7fcd63200640 1 --2- 172.21.15.40:0/606335662 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd5c0d17d0 0x7fcd5c1f56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.979+0000 7fcd63200640 1 --2- 172.21.15.40:0/606335662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd5c0d0c80 0x7fcd5c1522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.979+0000 7fcd63200640 1 -- 172.21.15.40:0/606335662 >> 172.21.15.40:0/606335662 conn(0x7fcd5c09e210 msgr2=0x7fcd5c0b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.979+0000 7fcd63200640 1 -- 172.21.15.40:0/606335662 shutdown_connections 2024-04-26T23:05:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:09.979+0000 7fcd63200640 1 -- 172.21.15.40:0/606335662 wait complete. 2024-04-26T23:05:09.998 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-26T23:05:09.999 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-26T23:05:09.999 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:05:10.109 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:05:10.230 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-26T23:05:10.230 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:05:10.230 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:05:10.230 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:05:10.289 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:05:10.290 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:05:10.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:05:10.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.559+0000 7f607fe00640 1 Processor -- start 2024-04-26T23:05:10.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.559+0000 7f607fe00640 1 -- start start 2024-04-26T23:05:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.561+0000 7f607fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f60800af7a0 0x7f60800afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.561+0000 7f607fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071fe0 0x7f6080073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.561+0000 7f607fe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60800d0820 con 0x7f608015ce60 2024-04-26T23:05:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.561+0000 7f607fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f60800cc450 con 0x7f60800af7a0 2024-04-26T23:05:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.561+0000 7f607fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f60801614c0 con 0x7f6080071fe0 2024-04-26T23:05:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.561+0000 7f607ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f608015ce60 0x7f60800ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58198/0 (socket says 172.21.15.40:58198) 2024-04-26T23:05:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.561+0000 7f607ea00640 1 -- 172.21.15.40:0/402987177 learned_addr learned my addr 172.21.15.40:0/402987177 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.562+0000 7f607f400640 1 --2- 172.21.15.40:0/402987177 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071fe0 0x7f6080073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.562+0000 7f607e000640 1 --2- 172.21.15.40:0/402987177 >> v2:172.21.15.103:3300/0 conn(0x7f60800af7a0 0x7f60800afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.562+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2793231327 0 0) 0x7f60800d0820 con 0x7f608015ce60 2024-04-26T23:05:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.562+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6064003520 con 0x7f608015ce60 2024-04-26T23:05:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.562+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3638782271 0 0) 0x7f6064003520 con 0x7f608015ce60 2024-04-26T23:05:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.562+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071fe0 msgr2=0x7f6080073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.562+0000 7f607d600640 1 --2- 172.21.15.40:0/402987177 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071fe0 0x7f6080073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.562+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 >> v2:172.21.15.103:3300/0 conn(0x7f60800af7a0 msgr2=0x7f60800afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.562+0000 7f607d600640 1 --2- 172.21.15.40:0/402987177 >> v2:172.21.15.103:3300/0 conn(0x7f60800af7a0 0x7f60800afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.562+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6080161760 con 0x7f608015ce60 2024-04-26T23:05:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.562+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f6070001670 con 0x7f608015ce60 2024-04-26T23:05:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.562+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 >> v1:172.21.15.40:6789/0 conn(0x7f608015ce60 legacy=0x7f60800ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607d600640 1 --2- 172.21.15.40:0/402987177 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071fe0 0x7f60640039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607d600640 1 --2- 172.21.15.40:0/402987177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6064003f30 0x7f6064004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607d600640 1 --2- 172.21.15.40:0/402987177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6064004d40 0x7f60640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f60800cc450 con 0x7f6064004d40 2024-04-26T23:05:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60801614c0 con 0x7f6064003f30 2024-04-26T23:05:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f606400ba10 con 0x7f6080071fe0 2024-04-26T23:05:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607f400640 1 --2- 172.21.15.40:0/402987177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6064004d40 0x7f60640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607e000640 1 --2- 172.21.15.40:0/402987177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6064003f30 0x7f6064004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607ea00640 1 --2- 172.21.15.40:0/402987177 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071fe0 0x7f60640039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607f400640 1 -- 172.21.15.40:0/402987177 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071fe0 msgr2=0x7f60640039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607f400640 1 --2- 172.21.15.40:0/402987177 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071fe0 0x7f60640039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607f400640 1 -- 172.21.15.40:0/402987177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6064003f30 msgr2=0x7f6064004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:10.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607f400640 1 --2- 172.21.15.40:0/402987177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6064003f30 0x7f6064004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:10.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.563+0000 7f607f400640 1 -- 172.21.15.40:0/402987177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6064003520 con 0x7f6064004d40 2024-04-26T23:05:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.565+0000 7f607f400640 1 --2- 172.21.15.40:0/402987177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6064004d40 0x7f60640051c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f607800e170 tx=0x7f6064005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d4df9973f58c7c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.565+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078034020 con 0x7f6064004d40 2024-04-26T23:05:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.565+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60780305f0 con 0x7f6064004d40 2024-04-26T23:05:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.565+0000 7f607d600640 1 -- 172.21.15.40:0/402987177 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078009c10 con 0x7f6064004d40 2024-04-26T23:05:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.566+0000 7f607fe00640 1 -- 172.21.15.40:0/402987177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6064004d40 msgr2=0x7f60640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.566+0000 7f607fe00640 1 --2- 172.21.15.40:0/402987177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6064004d40 0x7f60640051c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f607800e170 tx=0x7f6064005830 comp rx=0 tx=0).stop 2024-04-26T23:05:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.566+0000 7f607f400640 1 -- 172.21.15.40:0/402987177 reap_dead start 2024-04-26T23:05:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.566+0000 7f607fe00640 1 -- 172.21.15.40:0/402987177 shutdown_connections 2024-04-26T23:05:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.566+0000 7f607fe00640 1 -- 172.21.15.40:0/402987177 >> 172.21.15.40:0/402987177 conn(0x7f60800d0a10 msgr2=0x7f60800d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.566+0000 7f607fe00640 1 -- 172.21.15.40:0/402987177 shutdown_connections 2024-04-26T23:05:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.566+0000 7f607fe00640 1 -- 172.21.15.40:0/402987177 wait complete. 2024-04-26T23:05:10.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.568+0000 7f607fe00640 1 Processor -- start 2024-04-26T23:05:10.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.568+0000 7f607fe00640 1 -- start start 2024-04-26T23:05:10.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.569+0000 7f607fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6080153cb0 0x7f60801f87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:10.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.569+0000 7f607fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080154480 0x7f60800ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:10.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.569+0000 7f607fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800aee00 0x7f6080156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:10.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.569+0000 7f607fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f60800cc1f0 con 0x7f60800aee00 2024-04-26T23:05:10.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.569+0000 7f607fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60800cc070 con 0x7f6080153cb0 2024-04-26T23:05:10.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.569+0000 7f607fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6080073760 con 0x7f6080154480 2024-04-26T23:05:10.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.569+0000 7f607f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800aee00 0x7f6080156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:10.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.569+0000 7f607f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800aee00 0x7f6080156230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37236/0 (socket says 172.21.15.40:37236) 2024-04-26T23:05:10.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.569+0000 7f607f400640 1 -- 172.21.15.40:0/1449866044 learned_addr learned my addr 172.21.15.40:0/1449866044 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:10.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.570+0000 7f607ea00640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6080153cb0 0x7f60801f87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:10.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.570+0000 7f607e000640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080154480 0x7f60800ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:10.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.570+0000 7f607f400640 1 -- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080154480 msgr2=0x7f60800ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:10.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.570+0000 7f607f400640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080154480 0x7f60800ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:10.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.570+0000 7f607f400640 1 -- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6080153cb0 msgr2=0x7f60801f87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:10.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.570+0000 7f607f400640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6080153cb0 0x7f60801f87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:10.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.570+0000 7f607f400640 1 -- 172.21.15.40:0/1449866044 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60801f8db0 con 0x7f60800aee00 2024-04-26T23:05:10.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.570+0000 7f607ea00640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6080153cb0 0x7f60801f87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:10.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.570+0000 7f607f400640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800aee00 0x7f6080156230 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f607800e290 tx=0x7f6078009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.570+0000 7f6076a00640 1 -- 172.21.15.40:0/1449866044 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078034030 con 0x7f60800aee00 2024-04-26T23:05:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.570+0000 7f6076a00640 1 -- 172.21.15.40:0/1449866044 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6078030e50 con 0x7f60800aee00 2024-04-26T23:05:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.570+0000 7f6076a00640 1 -- 172.21.15.40:0/1449866044 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078009c10 con 0x7f60800aee00 2024-04-26T23:05:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.570+0000 7f607fe00640 1 -- 172.21.15.40:0/1449866044 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60801f9090 con 0x7f60800aee00 2024-04-26T23:05:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.571+0000 7f607fe00640 1 -- 172.21.15.40:0/1449866044 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60801f9650 con 0x7f60800aee00 2024-04-26T23:05:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.572+0000 7f607fe00640 1 -- 172.21.15.40:0/1449866044 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6080069370 con 0x7f60800aee00 2024-04-26T23:05:10.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.573+0000 7f6076a00640 1 -- 172.21.15.40:0/1449866044 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6078037020 con 0x7f60800aee00 2024-04-26T23:05:10.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.574+0000 7f6076a00640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f606006c8f0 0x7f606006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:10.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.574+0000 7f6076a00640 1 -- 172.21.15.40:0/1449866044 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f60780abff0 con 0x7f60800aee00 2024-04-26T23:05:10.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.575+0000 7f607ea00640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f606006c8f0 0x7f606006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:10.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.576+0000 7f607ea00640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f606006c8f0 0x7f606006edb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6070005a50 tx=0x7f6070001db0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.579+0000 7f6076a00640 1 -- 172.21.15.40:0/1449866044 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f607807a190 con 0x7f60800aee00 2024-04-26T23:05:10.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.786+0000 7f607fe00640 1 -- 172.21.15.40:0/1449866044 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f60800d01f0 con 0x7f60800aee00 2024-04-26T23:05:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.788+0000 7f6076a00640 1 -- 172.21.15.40:0/1449866044 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+17002 (secure 0 0 0) 0x7f6078079a90 con 0x7f60800aee00 2024-04-26T23:05:10.788 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:10.789 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":54,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:04:48.353789+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:04:14.334803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-26T23:04:15.535793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1a","mappings":[{"from":0,"to":1}]},{"pgid":"8.27","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:05:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.791+0000 7f607fe00640 1 -- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f606006c8f0 msgr2=0x7f606006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.791+0000 7f607fe00640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f606006c8f0 0x7f606006edb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6070005a50 tx=0x7f6070001db0 comp rx=0 tx=0).stop 2024-04-26T23:05:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.791+0000 7f607fe00640 1 -- 172.21.15.40:0/1449866044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800aee00 msgr2=0x7f6080156230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.791+0000 7f607fe00640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800aee00 0x7f6080156230 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f607800e290 tx=0x7f6078009750 comp rx=0 tx=0).stop 2024-04-26T23:05:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.791+0000 7f607fe00640 1 -- 172.21.15.40:0/1449866044 shutdown_connections 2024-04-26T23:05:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.791+0000 7f607fe00640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f606006c8f0 0x7f606006edb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.791+0000 7f607fe00640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800aee00 0x7f6080156230 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.791+0000 7f607fe00640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080154480 0x7f60800ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.791+0000 7f607fe00640 1 --2- 172.21.15.40:0/1449866044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6080153cb0 0x7f60801f87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.791+0000 7f607fe00640 1 -- 172.21.15.40:0/1449866044 >> 172.21.15.40:0/1449866044 conn(0x7f60800d0a10 msgr2=0x7f6080160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.791+0000 7f607fe00640 1 -- 172.21.15.40:0/1449866044 shutdown_connections 2024-04-26T23:05:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:10.791+0000 7f607fe00640 1 -- 172.21.15.40:0/1449866044 wait complete. 2024-04-26T23:05:10.810 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:04:14.334803+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:04:15.535793+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:05:10.810 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:05:11.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.029+0000 7f55a7e00640 1 Processor -- start 2024-04-26T23:05:11.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.029+0000 7f55a7e00640 1 -- start start 2024-04-26T23:05:11.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.031+0000 7f55a7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55a8160a50 0x7f55a8073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.031+0000 7f55a7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f55a80cfaf0 0x7f55a8160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.031+0000 7f55a7e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55a8161880 con 0x7f55a806dd40 2024-04-26T23:05:11.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.031+0000 7f55a7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f55a8071ce0 con 0x7f55a80cfaf0 2024-04-26T23:05:11.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.031+0000 7f55a7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f55a80722a0 con 0x7f55a8160a50 2024-04-26T23:05:11.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.032+0000 7f55a7400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f55a806dd40 0x7f55a8072730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58214/0 (socket says 172.21.15.40:58214) 2024-04-26T23:05:11.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.032+0000 7f55a7400640 1 -- 172.21.15.40:0/2911970235 learned_addr learned my addr 172.21.15.40:0/2911970235 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:11.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.032+0000 7f55a6a00640 1 --2- 172.21.15.40:0/2911970235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55a8160a50 0x7f55a8073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.032+0000 7f55a6000640 1 --2- 172.21.15.40:0/2911970235 >> v2:172.21.15.103:3300/0 conn(0x7f55a80cfaf0 0x7f55a8160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.032+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1228934585 0 0) 0x7f55a8161880 con 0x7f55a806dd40 2024-04-26T23:05:11.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.032+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f558c003520 con 0x7f55a806dd40 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.032+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 58600004 0 0) 0x7f558c003520 con 0x7f55a806dd40 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55a8160a50 msgr2=0x7f55a8073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a5600640 1 --2- 172.21.15.40:0/2911970235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55a8160a50 0x7f55a8073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 >> v2:172.21.15.103:3300/0 conn(0x7f55a80cfaf0 msgr2=0x7f55a8160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a5600640 1 --2- 172.21.15.40:0/2911970235 >> v2:172.21.15.103:3300/0 conn(0x7f55a80cfaf0 0x7f55a8160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a8161370 con 0x7f55a806dd40 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f55a00019f0 con 0x7f55a806dd40 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 >> v1:172.21.15.40:6789/0 conn(0x7f55a806dd40 legacy=0x7f55a8072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a5600640 1 --2- 172.21.15.40:0/2911970235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f558c0039b0 0x7f558c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a5600640 1 --2- 172.21.15.40:0/2911970235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55a8160a50 0x7f558c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a5600640 1 --2- 172.21.15.40:0/2911970235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f558c004d40 0x7f558c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f55a8071ce0 con 0x7f558c004d40 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f55a80722a0 con 0x7f558c0039b0 2024-04-26T23:05:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f558c00b9c0 con 0x7f55a8160a50 2024-04-26T23:05:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a7400640 1 --2- 172.21.15.40:0/2911970235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f558c004d40 0x7f558c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a7400640 1 -- 172.21.15.40:0/2911970235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55a8160a50 msgr2=0x7f558c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a7400640 1 --2- 172.21.15.40:0/2911970235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f55a8160a50 0x7f558c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.033+0000 7f55a6a00640 1 --2- 172.21.15.40:0/2911970235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f558c0039b0 0x7f558c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.034+0000 7f55a7400640 1 -- 172.21.15.40:0/2911970235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f558c0039b0 msgr2=0x7f558c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.034+0000 7f55a7400640 1 --2- 172.21.15.40:0/2911970235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f558c0039b0 0x7f558c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.034+0000 7f55a7400640 1 -- 172.21.15.40:0/2911970235 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558c003520 con 0x7f558c004d40 2024-04-26T23:05:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.035+0000 7f55a7400640 1 --2- 172.21.15.40:0/2911970235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f558c004d40 0x7f558c0051c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f55a000a8a0 tx=0x7f558c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0818461cc5868d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.035+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a000a4c0 con 0x7f558c004d40 2024-04-26T23:05:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.035+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55a002f070 con 0x7f558c004d40 2024-04-26T23:05:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.035+0000 7f55a5600640 1 -- 172.21.15.40:0/2911970235 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a002bd30 con 0x7f558c004d40 2024-04-26T23:05:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.035+0000 7f55a7e00640 1 -- 172.21.15.40:0/2911970235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f558c004d40 msgr2=0x7f558c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.035+0000 7f55a7e00640 1 --2- 172.21.15.40:0/2911970235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f558c004d40 0x7f558c0051c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f55a000a8a0 tx=0x7f558c005830 comp rx=0 tx=0).stop 2024-04-26T23:05:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.035+0000 7f55a7400640 1 -- 172.21.15.40:0/2911970235 reap_dead start 2024-04-26T23:05:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.036+0000 7f55a7e00640 1 -- 172.21.15.40:0/2911970235 shutdown_connections 2024-04-26T23:05:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.036+0000 7f55a7e00640 1 -- 172.21.15.40:0/2911970235 >> 172.21.15.40:0/2911970235 conn(0x7f55a80b2720 msgr2=0x7f55a80b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.036+0000 7f55a7e00640 1 -- 172.21.15.40:0/2911970235 shutdown_connections 2024-04-26T23:05:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.036+0000 7f55a7e00640 1 -- 172.21.15.40:0/2911970235 wait complete. 2024-04-26T23:05:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.037+0000 7f55a7e00640 1 Processor -- start 2024-04-26T23:05:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.038+0000 7f55a7e00640 1 -- start start 2024-04-26T23:05:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f558c004d40 0x7f55a813dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a813b4f0 0x7f55a8141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55a813a750 0x7f55a81415b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f55a8071f30 con 0x7f55a813a750 2024-04-26T23:05:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f55a80d0260 con 0x7f55a813b4f0 2024-04-26T23:05:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f55a806e1f0 con 0x7f558c004d40 2024-04-26T23:05:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55a813a750 0x7f55a81415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55a813a750 0x7f55a81415b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37250/0 (socket says 172.21.15.40:37250) 2024-04-26T23:05:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7400640 1 -- 172.21.15.40:0/2131266132 learned_addr learned my addr 172.21.15.40:0/2131266132 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7400640 1 -- 172.21.15.40:0/2131266132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f558c004d40 msgr2=0x7f55a813dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7400640 1 --2- 172.21.15.40:0/2131266132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f558c004d40 0x7f55a813dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7400640 1 -- 172.21.15.40:0/2131266132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a813b4f0 msgr2=0x7f55a8141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7400640 1 --2- 172.21.15.40:0/2131266132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a813b4f0 0x7f55a8141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.039+0000 7f55a7400640 1 -- 172.21.15.40:0/2131266132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a813b9f0 con 0x7f55a813a750 2024-04-26T23:05:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.040+0000 7f55a7400640 1 --2- 172.21.15.40:0/2131266132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55a813a750 0x7f55a81415b0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f55a000a8a0 tx=0x7f55a0001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.040+0000 7f559ea00640 1 -- 172.21.15.40:0/2131266132 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0044070 con 0x7f55a813a750 2024-04-26T23:05:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.040+0000 7f559ea00640 1 -- 172.21.15.40:0/2131266132 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55a0034470 con 0x7f55a813a750 2024-04-26T23:05:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.040+0000 7f559ea00640 1 -- 172.21.15.40:0/2131266132 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a000a4c0 con 0x7f55a813a750 2024-04-26T23:05:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.040+0000 7f55a7e00640 1 -- 172.21.15.40:0/2131266132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a80b3ca0 con 0x7f55a813a750 2024-04-26T23:05:11.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.040+0000 7f55a7e00640 1 -- 172.21.15.40:0/2131266132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a815c970 con 0x7f55a813a750 2024-04-26T23:05:11.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.041+0000 7f55a7e00640 1 -- 172.21.15.40:0/2131266132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a8069370 con 0x7f55a813a750 2024-04-26T23:05:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.042+0000 7f559ea00640 1 -- 172.21.15.40:0/2131266132 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f55a0047070 con 0x7f55a813a750 2024-04-26T23:05:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.043+0000 7f559ea00640 1 --2- 172.21.15.40:0/2131266132 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f558806c920 0x7f558806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.044+0000 7f559ea00640 1 -- 172.21.15.40:0/2131266132 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f55a00ac070 con 0x7f55a813a750 2024-04-26T23:05:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.044+0000 7f55a6a00640 1 --2- 172.21.15.40:0/2131266132 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f558806c920 0x7f558806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.045+0000 7f55a6a00640 1 --2- 172.21.15.40:0/2131266132 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f558806c920 0x7f558806ede0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5598009970 tx=0x7f5598008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:11.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.048+0000 7f559ea00640 1 -- 172.21.15.40:0/2131266132 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f55a007a240 con 0x7f55a813a750 2024-04-26T23:05:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.250+0000 7f55a7e00640 1 -- 172.21.15.40:0/2131266132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f55a806fa60 con 0x7f55a813a750 2024-04-26T23:05:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.251+0000 7f559ea00640 1 -- 172.21.15.40:0/2131266132 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) ==== 93+0+10 (secure 0 0 0) 0x7f55a0079b40 con 0x7f55a813a750 2024-04-26T23:05:11.251 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:05:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.253+0000 7f55a7e00640 1 -- 172.21.15.40:0/2131266132 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f558806c920 msgr2=0x7f558806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.253+0000 7f55a7e00640 1 --2- 172.21.15.40:0/2131266132 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f558806c920 0x7f558806ede0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5598009970 tx=0x7f5598008040 comp rx=0 tx=0).stop 2024-04-26T23:05:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.253+0000 7f55a7e00640 1 -- 172.21.15.40:0/2131266132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55a813a750 msgr2=0x7f55a81415b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.253+0000 7f55a7e00640 1 --2- 172.21.15.40:0/2131266132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55a813a750 0x7f55a81415b0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f55a000a8a0 tx=0x7f55a0001dc0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.254+0000 7f55a7e00640 1 -- 172.21.15.40:0/2131266132 shutdown_connections 2024-04-26T23:05:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.254+0000 7f55a7e00640 1 --2- 172.21.15.40:0/2131266132 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f558806c920 0x7f558806ede0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.254+0000 7f55a7e00640 1 --2- 172.21.15.40:0/2131266132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55a813a750 0x7f55a81415b0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.254+0000 7f55a7e00640 1 --2- 172.21.15.40:0/2131266132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a813b4f0 0x7f55a8141070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.254+0000 7f55a7e00640 1 --2- 172.21.15.40:0/2131266132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f558c004d40 0x7f55a813dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.254+0000 7f55a7e00640 1 -- 172.21.15.40:0/2131266132 >> 172.21.15.40:0/2131266132 conn(0x7f55a80b2720 msgr2=0x7f55a806e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.254+0000 7f55a7e00640 1 -- 172.21.15.40:0/2131266132 shutdown_connections 2024-04-26T23:05:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.254+0000 7f55a7e00640 1 -- 172.21.15.40:0/2131266132 wait complete. 2024-04-26T23:05:11.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:05:11.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.491+0000 7fc38d000640 1 Processor -- start 2024-04-26T23:05:11.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.491+0000 7fc38d000640 1 -- start start 2024-04-26T23:05:11.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.493+0000 7fc38d000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc388073600 0x7fc388072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.493+0000 7fc38d000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3881604e0 con 0x7fc38815fe80 2024-04-26T23:05:11.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.493+0000 7fc38d000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc388072200 con 0x7fc3880739e0 2024-04-26T23:05:11.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.493+0000 7fc38d000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc388073220 con 0x7fc388073600 2024-04-26T23:05:11.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.493+0000 7fc387400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc388073600 0x7fc388072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.493+0000 7fc387400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc388073600 0x7fc388072ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37266/0 (socket says 172.21.15.40:37266) 2024-04-26T23:05:11.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.493+0000 7fc387400640 1 -- 172.21.15.40:0/3796830177 learned_addr learned my addr 172.21.15.40:0/3796830177 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:11.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.494+0000 7fc387400640 1 -- 172.21.15.40:0/3796830177 >> v1:172.21.15.103:6789/0 conn(0x7fc3880739e0 legacy=0x7fc38815f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:11.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.494+0000 7fc387400640 1 -- 172.21.15.40:0/3796830177 >> v1:172.21.15.40:6789/0 conn(0x7fc38815fe80 legacy=0x7fc3880ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.494+0000 7fc387400640 1 -- 172.21.15.40:0/3796830177 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3880724e0 con 0x7fc388073600 2024-04-26T23:05:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.495+0000 7fc387400640 1 --2- 172.21.15.40:0/3796830177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc388073600 0x7fc388072ce0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc378009e50 tx=0x7fc37802a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d56b41b393c3d53 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.495+0000 7fc386000640 1 -- 172.21.15.40:0/3796830177 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc378036070 con 0x7fc388073600 2024-04-26T23:05:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.495+0000 7fc386000640 1 -- 172.21.15.40:0/3796830177 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc378031600 con 0x7fc388073600 2024-04-26T23:05:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.495+0000 7fc386000640 1 -- 172.21.15.40:0/3796830177 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3780319d0 con 0x7fc388073600 2024-04-26T23:05:11.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.496+0000 7fc38d000640 1 -- 172.21.15.40:0/3796830177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc388073600 msgr2=0x7fc388072ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.496+0000 7fc38d000640 1 --2- 172.21.15.40:0/3796830177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc388073600 0x7fc388072ce0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc378009e50 tx=0x7fc37802a760 comp rx=0 tx=0).stop 2024-04-26T23:05:11.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.496+0000 7fc38d000640 1 -- 172.21.15.40:0/3796830177 shutdown_connections 2024-04-26T23:05:11.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.496+0000 7fc38d000640 1 --2- 172.21.15.40:0/3796830177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc388073600 0x7fc388072ce0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.496+0000 7fc38d000640 1 -- 172.21.15.40:0/3796830177 >> 172.21.15.40:0/3796830177 conn(0x7fc3880b2720 msgr2=0x7fc3880b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:11.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.496+0000 7fc38d000640 1 -- 172.21.15.40:0/3796830177 shutdown_connections 2024-04-26T23:05:11.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.496+0000 7fc38d000640 1 -- 172.21.15.40:0/3796830177 wait complete. 2024-04-26T23:05:11.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.498+0000 7fc38d000640 1 Processor -- start 2024-04-26T23:05:11.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.498+0000 7fc38d000640 1 -- start start 2024-04-26T23:05:11.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.499+0000 7fc38d000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc388073600 0x7fc3881513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.499+0000 7fc38d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3880739e0 0x7fc388154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.499+0000 7fc38d000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc38815fe80 0x7fc388155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.499+0000 7fc38d000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc388160290 con 0x7fc3880739e0 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.499+0000 7fc38d000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc388072850 con 0x7fc38815fe80 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.499+0000 7fc38d000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc388071e20 con 0x7fc388073600 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.499+0000 7fc386a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3880739e0 0x7fc388154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.499+0000 7fc386a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3880739e0 0x7fc388154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37280/0 (socket says 172.21.15.40:37280) 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.499+0000 7fc386a00640 1 -- 172.21.15.40:0/2554407646 learned_addr learned my addr 172.21.15.40:0/2554407646 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc387e00640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc38815fe80 0x7fc388155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc387400640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc388073600 0x7fc3881513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc386a00640 1 -- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc388073600 msgr2=0x7fc3881513e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc386a00640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc388073600 0x7fc3881513e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc386a00640 1 -- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc38815fe80 msgr2=0x7fc388155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc386a00640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc38815fe80 0x7fc388155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc386a00640 1 -- 172.21.15.40:0/2554407646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc388151920 con 0x7fc3880739e0 2024-04-26T23:05:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc387400640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc388073600 0x7fc3881513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc387e00640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc38815fe80 0x7fc388155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc386a00640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3880739e0 0x7fc388154ff0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fc37400c440 tx=0x7fc37400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc37f400640 1 -- 172.21.15.40:0/2554407646 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc374017070 con 0x7fc3880739e0 2024-04-26T23:05:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc37f400640 1 -- 172.21.15.40:0/2554407646 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3740035f0 con 0x7fc3880739e0 2024-04-26T23:05:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc37f400640 1 -- 172.21.15.40:0/2554407646 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3740039e0 con 0x7fc3880739e0 2024-04-26T23:05:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.500+0000 7fc38d000640 1 -- 172.21.15.40:0/2554407646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc38815b570 con 0x7fc3880739e0 2024-04-26T23:05:11.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.501+0000 7fc38d000640 1 -- 172.21.15.40:0/2554407646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc388152840 con 0x7fc3880739e0 2024-04-26T23:05:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.502+0000 7fc38d000640 1 -- 172.21.15.40:0/2554407646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc388069370 con 0x7fc3880739e0 2024-04-26T23:05:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.503+0000 7fc37f400640 1 -- 172.21.15.40:0/2554407646 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc374003b80 con 0x7fc3880739e0 2024-04-26T23:05:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.504+0000 7fc37f400640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc36806c8f0 0x7fc36806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.504+0000 7fc37f400640 1 -- 172.21.15.40:0/2554407646 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7fc37408aff0 con 0x7fc3880739e0 2024-04-26T23:05:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.505+0000 7fc387400640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc36806c8f0 0x7fc36806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.506+0000 7fc387400640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc36806c8f0 0x7fc36806edb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc378033000 tx=0x7fc378009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:11.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.509+0000 7fc37f400640 1 -- 172.21.15.40:0/2554407646 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc3740592b0 con 0x7fc3880739e0 2024-04-26T23:05:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.719+0000 7fc38d000640 1 -- 172.21.15.40:0/2554407646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc3880b3480 con 0x7fc3880739e0 2024-04-26T23:05:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.720+0000 7fc37f400640 1 -- 172.21.15.40:0/2554407646 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) ==== 104+0+11 (secure 0 0 0) 0x7fc374058bb0 con 0x7fc3880739e0 2024-04-26T23:05:11.720 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:05:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.723+0000 7fc38d000640 1 -- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc36806c8f0 msgr2=0x7fc36806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.723+0000 7fc38d000640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc36806c8f0 0x7fc36806edb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc378033000 tx=0x7fc378009000 comp rx=0 tx=0).stop 2024-04-26T23:05:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.723+0000 7fc38d000640 1 -- 172.21.15.40:0/2554407646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3880739e0 msgr2=0x7fc388154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.723+0000 7fc38d000640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3880739e0 0x7fc388154ff0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fc37400c440 tx=0x7fc37400c900 comp rx=0 tx=0).stop 2024-04-26T23:05:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.723+0000 7fc38d000640 1 -- 172.21.15.40:0/2554407646 shutdown_connections 2024-04-26T23:05:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.723+0000 7fc38d000640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc36806c8f0 0x7fc36806edb0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.723+0000 7fc38d000640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc38815fe80 0x7fc388155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.723+0000 7fc38d000640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3880739e0 0x7fc388154ff0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.723+0000 7fc38d000640 1 --2- 172.21.15.40:0/2554407646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc388073600 0x7fc3881513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.723+0000 7fc38d000640 1 -- 172.21.15.40:0/2554407646 >> 172.21.15.40:0/2554407646 conn(0x7fc3880b2720 msgr2=0x7fc3880b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.723+0000 7fc38d000640 1 -- 172.21.15.40:0/2554407646 shutdown_connections 2024-04-26T23:05:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.723+0000 7fc38d000640 1 -- 172.21.15.40:0/2554407646 wait complete. 2024-04-26T23:05:11.743 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:05:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.955+0000 7f0eee800640 1 Processor -- start 2024-04-26T23:05:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.955+0000 7f0eee800640 1 -- start start 2024-04-26T23:05:11.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.957+0000 7f0eee800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0ee80c6f10 0x7f0ee80c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.957+0000 7f0eee800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ee80c6010 0x7f0ee8158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.957+0000 7f0eee800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee80cce10 con 0x7f0ee80cc980 2024-04-26T23:05:11.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.957+0000 7f0eee800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee80c72f0 con 0x7f0ee80c6f10 2024-04-26T23:05:11.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.957+0000 7f0eee800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee80cb860 con 0x7f0ee80c6010 2024-04-26T23:05:11.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.958+0000 7f0eed400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0ee80c6f10 0x7f0ee80c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.958+0000 7f0eed400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0ee80c6f10 0x7f0ee80c5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37292/0 (socket says 172.21.15.40:37292) 2024-04-26T23:05:11.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.958+0000 7f0eed400640 1 -- 172.21.15.40:0/1452409943 learned_addr learned my addr 172.21.15.40:0/1452409943 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:11.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.958+0000 7f0eede00640 1 --2- 172.21.15.40:0/1452409943 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ee80c6010 0x7f0ee8158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.958+0000 7f0eed400640 1 -- 172.21.15.40:0/1452409943 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ee80c6010 msgr2=0x7f0ee8158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:11.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.958+0000 7f0eed400640 1 --2- 172.21.15.40:0/1452409943 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ee80c6010 0x7f0ee8158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.958+0000 7f0eed400640 1 -- 172.21.15.40:0/1452409943 >> v1:172.21.15.103:6789/0 conn(0x7f0ee80cc980 legacy=0x7f0ee80cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:11.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.958+0000 7f0eed400640 1 -- 172.21.15.40:0/1452409943 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee80cd0f0 con 0x7f0ee80c6f10 2024-04-26T23:05:11.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.959+0000 7f0eed400640 1 --2- 172.21.15.40:0/1452409943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ee80c6f10 0x7f0ee80c5ad0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009870 tx=0x7f0edc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8051ae1de3b7bc31 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:11.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.960+0000 7f0ee7e00640 1 -- 172.21.15.40:0/1452409943 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc037070 con 0x7f0ee80c6f10 2024-04-26T23:05:11.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.960+0000 7f0ee7e00640 1 -- 172.21.15.40:0/1452409943 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0edc032440 con 0x7f0ee80c6f10 2024-04-26T23:05:11.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.960+0000 7f0ee7e00640 1 -- 172.21.15.40:0/1452409943 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc032810 con 0x7f0ee80c6f10 2024-04-26T23:05:11.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.960+0000 7f0eee800640 1 -- 172.21.15.40:0/1452409943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ee80c6f10 msgr2=0x7f0ee80c5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.960+0000 7f0eee800640 1 --2- 172.21.15.40:0/1452409943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ee80c6f10 0x7f0ee80c5ad0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009870 tx=0x7f0edc02a180 comp rx=0 tx=0).stop 2024-04-26T23:05:11.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.960+0000 7f0eee800640 1 -- 172.21.15.40:0/1452409943 shutdown_connections 2024-04-26T23:05:11.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.960+0000 7f0eee800640 1 --2- 172.21.15.40:0/1452409943 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ee80c6010 0x7f0ee8158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.960+0000 7f0eee800640 1 --2- 172.21.15.40:0/1452409943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ee80c6f10 0x7f0ee80c5ad0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.960+0000 7f0eee800640 1 -- 172.21.15.40:0/1452409943 >> 172.21.15.40:0/1452409943 conn(0x7f0ee80d0a10 msgr2=0x7f0ee80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:11.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.960+0000 7f0eee800640 1 -- 172.21.15.40:0/1452409943 shutdown_connections 2024-04-26T23:05:11.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.961+0000 7f0eee800640 1 -- 172.21.15.40:0/1452409943 wait complete. 2024-04-26T23:05:11.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.962+0000 7f0eee800640 1 Processor -- start 2024-04-26T23:05:11.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.962+0000 7f0eee800640 1 -- start start 2024-04-26T23:05:11.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.963+0000 7f0eee800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ee80c6010 0x7f0ee8156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.963+0000 7f0eee800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ee80c6f10 0x7f0ee80c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.963+0000 7f0eee800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ee80c9a90 0x7f0ee80b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.963+0000 7f0eee800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee80cc400 con 0x7f0ee80c6010 2024-04-26T23:05:11.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.963+0000 7f0eee800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee80cb1a0 con 0x7f0ee80c9a90 2024-04-26T23:05:11.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.963+0000 7f0eee800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee80cc580 con 0x7f0ee80c6f10 2024-04-26T23:05:11.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.964+0000 7f0eed400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ee80c6010 0x7f0ee8156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.964+0000 7f0eed400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ee80c6010 0x7f0ee8156fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37300/0 (socket says 172.21.15.40:37300) 2024-04-26T23:05:11.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.964+0000 7f0eed400640 1 -- 172.21.15.40:0/872206738 learned_addr learned my addr 172.21.15.40:0/872206738 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:11.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.964+0000 7f0eeca00640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ee80c6f10 0x7f0ee80c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.964+0000 7f0eed400640 1 -- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ee80c6f10 msgr2=0x7f0ee80c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.964+0000 7f0eede00640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ee80c9a90 0x7f0ee80b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.964+0000 7f0eed400640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ee80c6f10 0x7f0ee80c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.964+0000 7f0eed400640 1 -- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ee80c9a90 msgr2=0x7f0ee80b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:11.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.964+0000 7f0eed400640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ee80c9a90 0x7f0ee80b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:11.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.964+0000 7f0eed400640 1 -- 172.21.15.40:0/872206738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee80b38d0 con 0x7f0ee80c6010 2024-04-26T23:05:11.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.964+0000 7f0eeca00640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ee80c6f10 0x7f0ee80c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:11.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.965+0000 7f0eed400640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ee80c6010 0x7f0ee8156fa0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009870 tx=0x7f0edc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:11.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.965+0000 7f0ee5600640 1 -- 172.21.15.40:0/872206738 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc041020 con 0x7f0ee80c6010 2024-04-26T23:05:11.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.965+0000 7f0ee5600640 1 -- 172.21.15.40:0/872206738 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0edc0306c0 con 0x7f0ee80c6010 2024-04-26T23:05:11.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.965+0000 7f0ee5600640 1 -- 172.21.15.40:0/872206738 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc037040 con 0x7f0ee80c6010 2024-04-26T23:05:11.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.965+0000 7f0eee800640 1 -- 172.21.15.40:0/872206738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee80b3bb0 con 0x7f0ee80c6010 2024-04-26T23:05:11.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.965+0000 7f0eee800640 1 -- 172.21.15.40:0/872206738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee80b4170 con 0x7f0ee80c6010 2024-04-26T23:05:11.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.966+0000 7f0eee800640 1 -- 172.21.15.40:0/872206738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ee80c9ea0 con 0x7f0ee80c6010 2024-04-26T23:05:11.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.967+0000 7f0ee5600640 1 -- 172.21.15.40:0/872206738 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0edc044020 con 0x7f0ee80c6010 2024-04-26T23:05:11.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.968+0000 7f0ee5600640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ec806c8f0 0x7f0ec806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:11.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.968+0000 7f0ee5600640 1 -- 172.21.15.40:0/872206738 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f0edc0ac5e0 con 0x7f0ee80c6010 2024-04-26T23:05:11.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.969+0000 7f0eeca00640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ec806c8f0 0x7f0ec806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:11.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.970+0000 7f0eeca00640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ec806c8f0 0x7f0ec806edb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0ee815cf50 tx=0x7f0ed4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:11.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:11.972+0000 7f0ee5600640 1 -- 172.21.15.40:0/872206738 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0ee80c9ea0 con 0x7f0ee80c6010 2024-04-26T23:05:12.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.179+0000 7f0eee800640 1 -- 172.21.15.40:0/872206738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0ee8069370 con 0x7f0ee80c6010 2024-04-26T23:05:12.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.180+0000 7f0ee5600640 1 -- 172.21.15.40:0/872206738 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) ==== 100+0+11 (secure 0 0 0) 0x7f0edc04b030 con 0x7f0ee80c6010 2024-04-26T23:05:12.180 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:05:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.183+0000 7f0eee800640 1 -- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ec806c8f0 msgr2=0x7f0ec806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.183+0000 7f0eee800640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ec806c8f0 0x7f0ec806edb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0ee815cf50 tx=0x7f0ed4008040 comp rx=0 tx=0).stop 2024-04-26T23:05:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.183+0000 7f0eee800640 1 -- 172.21.15.40:0/872206738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ee80c6010 msgr2=0x7f0ee8156fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.183+0000 7f0eee800640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ee80c6010 0x7f0ee8156fa0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009870 tx=0x7f0edc0046c0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.183+0000 7f0eee800640 1 -- 172.21.15.40:0/872206738 shutdown_connections 2024-04-26T23:05:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.183+0000 7f0eee800640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ec806c8f0 0x7f0ec806edb0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.183+0000 7f0eee800640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ee80c9a90 0x7f0ee80b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.183+0000 7f0eee800640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0ee80c6f10 0x7f0ee80c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.183+0000 7f0eee800640 1 --2- 172.21.15.40:0/872206738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ee80c6010 0x7f0ee8156fa0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.183+0000 7f0eee800640 1 -- 172.21.15.40:0/872206738 >> 172.21.15.40:0/872206738 conn(0x7f0ee80d0a10 msgr2=0x7f0ee815d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.184+0000 7f0eee800640 1 -- 172.21.15.40:0/872206738 shutdown_connections 2024-04-26T23:05:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.184+0000 7f0eee800640 1 -- 172.21.15.40:0/872206738 wait complete. 2024-04-26T23:05:12.202 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:05:12.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.406+0000 7f878a000640 1 Processor -- start 2024-04-26T23:05:12.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.406+0000 7f878a000640 1 -- start start 2024-04-26T23:05:12.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.407+0000 7f878a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87840b6220 0x7f87840d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:12.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.408+0000 7f878a000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8784162c40 con 0x7f87840d08d0 2024-04-26T23:05:12.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.408+0000 7f878a000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87840d2d80 con 0x7f87840d39f0 2024-04-26T23:05:12.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.408+0000 7f878a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f87840b7c80 con 0x7f87840b6220 2024-04-26T23:05:12.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.408+0000 7f8783e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f87840d08d0 0x7f87840d32e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58234/0 (socket says 172.21.15.40:58234) 2024-04-26T23:05:12.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.408+0000 7f8783e00640 1 -- 172.21.15.40:0/2307072783 learned_addr learned my addr 172.21.15.40:0/2307072783 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:12.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.408+0000 7f8788c00640 1 --2- 172.21.15.40:0/2307072783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87840b6220 0x7f87840d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:12.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.408+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2493193668 0 0) 0x7f8784162c40 con 0x7f87840d08d0 2024-04-26T23:05:12.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f876c003520 con 0x7f87840d08d0 2024-04-26T23:05:12.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 65616583 0 0) 0x7f876c003520 con 0x7f87840d08d0 2024-04-26T23:05:12.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87840b6220 msgr2=0x7f87840d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:12.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 --2- 172.21.15.40:0/2307072783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87840b6220 0x7f87840d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 >> v1:172.21.15.103:6789/0 conn(0x7f87840d39f0 legacy=0x7f87840b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:12.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87840b7f60 con 0x7f87840d08d0 2024-04-26T23:05:12.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f8770001670 con 0x7f87840d08d0 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 >> v1:172.21.15.40:6789/0 conn(0x7f87840d08d0 legacy=0x7f87840d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 --2- 172.21.15.40:0/2307072783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87840b6220 0x7f876c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 --2- 172.21.15.40:0/2307072783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f876c003fb0 0x7f876c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 --2- 172.21.15.40:0/2307072783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f876c004d40 0x7f876c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f87840b7c80 con 0x7f876c004d40 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f876c00bb00 con 0x7f876c003fb0 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f876c00bd00 con 0x7f87840b6220 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.409+0000 7f8789600640 1 --2- 172.21.15.40:0/2307072783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f876c004d40 0x7f876c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.410+0000 7f8783e00640 1 --2- 172.21.15.40:0/2307072783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f876c003fb0 0x7f876c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.410+0000 7f8788c00640 1 --2- 172.21.15.40:0/2307072783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87840b6220 0x7f876c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.410+0000 7f8789600640 1 -- 172.21.15.40:0/2307072783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87840b6220 msgr2=0x7f876c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.410+0000 7f8789600640 1 --2- 172.21.15.40:0/2307072783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87840b6220 0x7f876c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.410+0000 7f8789600640 1 -- 172.21.15.40:0/2307072783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f876c003fb0 msgr2=0x7f876c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.410+0000 7f8789600640 1 --2- 172.21.15.40:0/2307072783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f876c003fb0 0x7f876c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.410+0000 7f8789600640 1 -- 172.21.15.40:0/2307072783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876c003520 con 0x7f876c004d40 2024-04-26T23:05:12.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.411+0000 7f8789600640 1 --2- 172.21.15.40:0/2307072783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f876c004d40 0x7f876c0051c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f877c001e20 tx=0x7f876c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6731d41d53f3d97a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:12.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.411+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c001a70 con 0x7f876c004d40 2024-04-26T23:05:12.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.411+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f877c02ae40 con 0x7f876c004d40 2024-04-26T23:05:12.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.411+0000 7f8783400640 1 -- 172.21.15.40:0/2307072783 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c003bb0 con 0x7f876c004d40 2024-04-26T23:05:12.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.412+0000 7f878a000640 1 -- 172.21.15.40:0/2307072783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f876c004d40 msgr2=0x7f876c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:12.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.412+0000 7f878a000640 1 --2- 172.21.15.40:0/2307072783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f876c004d40 0x7f876c0051c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f877c001e20 tx=0x7f876c005830 comp rx=0 tx=0).stop 2024-04-26T23:05:12.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.412+0000 7f8789600640 1 -- 172.21.15.40:0/2307072783 reap_dead start 2024-04-26T23:05:12.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.412+0000 7f878a000640 1 -- 172.21.15.40:0/2307072783 shutdown_connections 2024-04-26T23:05:12.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.412+0000 7f878a000640 1 -- 172.21.15.40:0/2307072783 >> 172.21.15.40:0/2307072783 conn(0x7f87840d4d90 msgr2=0x7f87840d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:12.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.412+0000 7f878a000640 1 -- 172.21.15.40:0/2307072783 shutdown_connections 2024-04-26T23:05:12.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.412+0000 7f878a000640 1 -- 172.21.15.40:0/2307072783 wait complete. 2024-04-26T23:05:12.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.414+0000 7f878a000640 1 Processor -- start 2024-04-26T23:05:12.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.414+0000 7f878a000640 1 -- start start 2024-04-26T23:05:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.415+0000 7f878a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f876c004d40 0x7f878409ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.415+0000 7f878a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878414cd30 0x7f878415a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.415+0000 7f878a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f878414b460 0x7f878414d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.415+0000 7f878a000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f87840d0390 con 0x7f876c004d40 2024-04-26T23:05:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.415+0000 7f878a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87840b8520 con 0x7f878414b460 2024-04-26T23:05:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.415+0000 7f878a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f87840d0510 con 0x7f878414cd30 2024-04-26T23:05:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.415+0000 7f8788c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f876c004d40 0x7f878409ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.415+0000 7f8788c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f876c004d40 0x7f878409ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37312/0 (socket says 172.21.15.40:37312) 2024-04-26T23:05:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.415+0000 7f8788c00640 1 -- 172.21.15.40:0/2608008151 learned_addr learned my addr 172.21.15.40:0/2608008151 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:12.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.416+0000 7f8783e00640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878414cd30 0x7f878415a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.416+0000 7f8788c00640 1 -- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878414cd30 msgr2=0x7f878415a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.416+0000 7f8789600640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f878414b460 0x7f878414d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.416+0000 7f8788c00640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878414cd30 0x7f878415a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.416+0000 7f8788c00640 1 -- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f878414b460 msgr2=0x7f878414d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.416+0000 7f8788c00640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f878414b460 0x7f878414d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.416+0000 7f8788c00640 1 -- 172.21.15.40:0/2608008151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878414bc90 con 0x7f876c004d40 2024-04-26T23:05:12.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.416+0000 7f8788c00640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f876c004d40 0x7f878409ede0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8778011660 tx=0x7f8778011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:12.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.416+0000 7f8780c00640 1 -- 172.21.15.40:0/2608008151 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877800f960 con 0x7f876c004d40 2024-04-26T23:05:12.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.416+0000 7f8780c00640 1 -- 172.21.15.40:0/2608008151 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f877800fb00 con 0x7f876c004d40 2024-04-26T23:05:12.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.416+0000 7f8780c00640 1 -- 172.21.15.40:0/2608008151 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8778015460 con 0x7f876c004d40 2024-04-26T23:05:12.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.416+0000 7f878a000640 1 -- 172.21.15.40:0/2608008151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f878414bf70 con 0x7f876c004d40 2024-04-26T23:05:12.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.417+0000 7f878a000640 1 -- 172.21.15.40:0/2608008151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f878414c530 con 0x7f876c004d40 2024-04-26T23:05:12.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.418+0000 7f878a000640 1 -- 172.21.15.40:0/2608008151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8784069370 con 0x7f876c004d40 2024-04-26T23:05:12.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.419+0000 7f8780c00640 1 -- 172.21.15.40:0/2608008151 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8778015600 con 0x7f876c004d40 2024-04-26T23:05:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.420+0000 7f8780c00640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f876406c8f0 0x7f876406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.420+0000 7f8780c00640 1 -- 172.21.15.40:0/2608008151 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f877808c730 con 0x7f876c004d40 2024-04-26T23:05:12.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.420+0000 7f8783e00640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f876406c8f0 0x7f876406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:12.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.422+0000 7f8783e00640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f876406c8f0 0x7f876406edb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8770001bf0 tx=0x7f8770000f70 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:12.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.424+0000 7f8780c00640 1 -- 172.21.15.40:0/2608008151 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8778019030 con 0x7f876c004d40 2024-04-26T23:05:12.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.641+0000 7f878a000640 1 -- 172.21.15.40:0/2608008151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f87840d30c0 con 0x7f876c004d40 2024-04-26T23:05:12.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.642+0000 7f8780c00640 1 -- 172.21.15.40:0/2608008151 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v54) ==== 80+0+0 (secure 0 0 0) 0x7f877808ae20 con 0x7f876c004d40 2024-04-26T23:05:12.642 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-04-26T23:05:12.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.645+0000 7f878a000640 1 -- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f876406c8f0 msgr2=0x7f876406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:12.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.645+0000 7f878a000640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f876406c8f0 0x7f876406edb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8770001bf0 tx=0x7f8770000f70 comp rx=0 tx=0).stop 2024-04-26T23:05:12.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.645+0000 7f878a000640 1 -- 172.21.15.40:0/2608008151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f876c004d40 msgr2=0x7f878409ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:12.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.645+0000 7f878a000640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f876c004d40 0x7f878409ede0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8778011660 tx=0x7f8778011b20 comp rx=0 tx=0).stop 2024-04-26T23:05:12.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.645+0000 7f878a000640 1 -- 172.21.15.40:0/2608008151 shutdown_connections 2024-04-26T23:05:12.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.645+0000 7f878a000640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f876406c8f0 0x7f876406edb0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.645+0000 7f878a000640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f878414b460 0x7f878414d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.645+0000 7f878a000640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878414cd30 0x7f878415a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.645+0000 7f878a000640 1 --2- 172.21.15.40:0/2608008151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f876c004d40 0x7f878409ede0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:12.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.645+0000 7f878a000640 1 -- 172.21.15.40:0/2608008151 >> 172.21.15.40:0/2608008151 conn(0x7f87840d4d90 msgr2=0x7f878415fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:12.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.645+0000 7f878a000640 1 -- 172.21.15.40:0/2608008151 shutdown_connections 2024-04-26T23:05:12.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:12.645+0000 7f878a000640 1 -- 172.21.15.40:0/2608008151 wait complete. 2024-04-26T23:05:12.664 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T23:05:12.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:12.665 DEBUG:teuthology.orchestra.run.smithi040:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-26T23:05:12.761 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:05:12.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:12.761 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T23:05:12.813 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:05:12.813 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:05:12.813 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:05:12.872 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:05:12.872 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:05:12.872 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:05:12.872 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:05:12.934 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:05:12.934 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:05:12.934 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:05:12.934 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:05:12.993 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:05:12.994 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:05:13.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.215+0000 7fc906400640 1 Processor -- start 2024-04-26T23:05:13.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.215+0000 7fc906400640 1 -- start start 2024-04-26T23:05:13.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc906400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc9000b6220 0x7fc9000d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:13.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc906400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9000d08d0 0x7fc9000d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:13.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc906400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc900162c40 con 0x7fc9000d3820 2024-04-26T23:05:13.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc906400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc9000d3c60 con 0x7fc9000d08d0 2024-04-26T23:05:13.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc906400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc9000d2d80 con 0x7fc9000b6220 2024-04-26T23:05:13.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc8ffe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9000d08d0 0x7fc9000d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:13.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc8ffe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9000d08d0 0x7fc9000d32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37314/0 (socket says 172.21.15.40:37314) 2024-04-26T23:05:13.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc8ffe00640 1 -- 172.21.15.40:0/1351237070 learned_addr learned my addr 172.21.15.40:0/1351237070 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc905000640 1 --2- 172.21.15.40:0/1351237070 >> v2:172.21.15.103:3300/0 conn(0x7fc9000b6220 0x7fc9000d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc8ffe00640 1 -- 172.21.15.40:0/1351237070 >> v2:172.21.15.103:3300/0 conn(0x7fc9000b6220 msgr2=0x7fc9000d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc8ffe00640 1 --2- 172.21.15.40:0/1351237070 >> v2:172.21.15.103:3300/0 conn(0x7fc9000b6220 0x7fc9000d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc8ffe00640 1 -- 172.21.15.40:0/1351237070 >> v1:172.21.15.40:6789/0 conn(0x7fc9000d3820 legacy=0x7fc9000b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc8ffe00640 1 -- 172.21.15.40:0/1351237070 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9000b7c80 con 0x7fc9000d08d0 2024-04-26T23:05:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.217+0000 7fc8ff400640 1 -- 172.21.15.40:0/1351237070 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2265648857 0 0) 0x7fc900162c40 con 0x7fc9000d3820 2024-04-26T23:05:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.218+0000 7fc8ffe00640 1 --2- 172.21.15.40:0/1351237070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9000d08d0 0x7fc9000d32e0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fc9000d4cf0 tx=0x7fc8f002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2585219ffb4e91 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.219+0000 7fc8ff400640 1 -- 172.21.15.40:0/1351237070 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0036070 con 0x7fc9000d08d0 2024-04-26T23:05:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.219+0000 7fc8ff400640 1 -- 172.21.15.40:0/1351237070 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8f0031600 con 0x7fc9000d08d0 2024-04-26T23:05:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.219+0000 7fc8ff400640 1 -- 172.21.15.40:0/1351237070 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f00319d0 con 0x7fc9000d08d0 2024-04-26T23:05:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.219+0000 7fc906400640 1 -- 172.21.15.40:0/1351237070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9000d08d0 msgr2=0x7fc9000d32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.219+0000 7fc906400640 1 --2- 172.21.15.40:0/1351237070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9000d08d0 0x7fc9000d32e0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fc9000d4cf0 tx=0x7fc8f002a730 comp rx=0 tx=0).stop 2024-04-26T23:05:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.219+0000 7fc906400640 1 -- 172.21.15.40:0/1351237070 shutdown_connections 2024-04-26T23:05:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.219+0000 7fc906400640 1 --2- 172.21.15.40:0/1351237070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9000d08d0 0x7fc9000d32e0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.219+0000 7fc906400640 1 --2- 172.21.15.40:0/1351237070 >> v2:172.21.15.103:3300/0 conn(0x7fc9000b6220 0x7fc9000d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.219+0000 7fc906400640 1 -- 172.21.15.40:0/1351237070 >> 172.21.15.40:0/1351237070 conn(0x7fc9000d4d90 msgr2=0x7fc9000d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.220+0000 7fc906400640 1 -- 172.21.15.40:0/1351237070 shutdown_connections 2024-04-26T23:05:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.220+0000 7fc906400640 1 -- 172.21.15.40:0/1351237070 wait complete. 2024-04-26T23:05:13.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.221+0000 7fc906400640 1 Processor -- start 2024-04-26T23:05:13.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.221+0000 7fc906400640 1 -- start start 2024-04-26T23:05:13.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.222+0000 7fc906400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9000b6220 0x7fc9000cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:13.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.222+0000 7fc906400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9000d08d0 0x7fc9000b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:13.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.222+0000 7fc906400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9000d3820 0x7fc9000b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:13.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.222+0000 7fc906400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc9000b8520 con 0x7fc9000b6220 2024-04-26T23:05:13.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.222+0000 7fc906400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9000b8240 con 0x7fc9000d08d0 2024-04-26T23:05:13.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.222+0000 7fc906400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc9000b9d30 con 0x7fc9000d3820 2024-04-26T23:05:13.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.223+0000 7fc905000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9000b6220 0x7fc9000cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:13.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.223+0000 7fc905000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9000b6220 0x7fc9000cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37326/0 (socket says 172.21.15.40:37326) 2024-04-26T23:05:13.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.223+0000 7fc905000640 1 -- 172.21.15.40:0/178605035 learned_addr learned my addr 172.21.15.40:0/178605035 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:13.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.223+0000 7fc8ffe00640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9000d08d0 0x7fc9000b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:13.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.223+0000 7fc905000640 1 -- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9000d3820 msgr2=0x7fc9000b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:13.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.223+0000 7fc905000640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9000d3820 0x7fc9000b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.223+0000 7fc905000640 1 -- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9000d08d0 msgr2=0x7fc9000b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.223+0000 7fc905000640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9000d08d0 0x7fc9000b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.223+0000 7fc905000640 1 -- 172.21.15.40:0/178605035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc900152120 con 0x7fc9000b6220 2024-04-26T23:05:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.223+0000 7fc8ffe00640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9000d08d0 0x7fc9000b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.223+0000 7fc905000640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9000b6220 0x7fc9000cb4e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800d800 tx=0x7fc8e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.224+0000 7fc8fcc00640 1 -- 172.21.15.40:0/178605035 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e800b6c0 con 0x7fc9000b6220 2024-04-26T23:05:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.224+0000 7fc8fcc00640 1 -- 172.21.15.40:0/178605035 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8e8019070 con 0x7fc9000b6220 2024-04-26T23:05:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.224+0000 7fc8fcc00640 1 -- 172.21.15.40:0/178605035 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e800ba80 con 0x7fc9000b6220 2024-04-26T23:05:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.224+0000 7fc906400640 1 -- 172.21.15.40:0/178605035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9000cba20 con 0x7fc9000b6220 2024-04-26T23:05:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.224+0000 7fc906400640 1 -- 172.21.15.40:0/178605035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9001523c0 con 0x7fc9000b6220 2024-04-26T23:05:13.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.225+0000 7fc906400640 1 -- 172.21.15.40:0/178605035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc900069370 con 0x7fc9000b6220 2024-04-26T23:05:13.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.226+0000 7fc8fcc00640 1 -- 172.21.15.40:0/178605035 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc8e800bc20 con 0x7fc9000b6220 2024-04-26T23:05:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.227+0000 7fc8fcc00640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc8dc06c8f0 0x7fc8dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.227+0000 7fc8fcc00640 1 -- 172.21.15.40:0/178605035 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7fc8e808c2a0 con 0x7fc9000b6220 2024-04-26T23:05:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.227+0000 7fc8ffe00640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc8dc06c8f0 0x7fc8dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:13.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.228+0000 7fc8ffe00640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc8dc06c8f0 0x7fc8dc06edb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0033000 tx=0x7fc8f0009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:13.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.231+0000 7fc8fcc00640 1 -- 172.21.15.40:0/178605035 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc8e801c360 con 0x7fc9000b6220 2024-04-26T23:05:13.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.480+0000 7fc906400640 1 -- 172.21.15.40:0/178605035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9000d30c0 con 0x7fc9000b6220 2024-04-26T23:05:13.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.481+0000 7fc8fcc00640 1 -- 172.21.15.40:0/178605035 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5225 (secure 0 0 0) 0x7fc8e805a470 con 0x7fc9000b6220 2024-04-26T23:05:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:13.482 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":46,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/189004193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":189004193},{"type":"v1","addr":"172.21.15.103:6835","nonce":189004193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:04:16.558591+0000","modified":"2024-04-26T23:04:55.194872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6036,"mds_1":6027,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"b","rank":2,"incarnation":39,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/670995452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":670995452},{"type":"v1","addr":"172.21.15.103:6837","nonce":670995452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6027":{"gid":6027,"name":"a","rank":1,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6834/2304583125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2304583125},{"type":"v1","addr":"172.21.15.40:6834","nonce":2304583125}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6036":{"gid":6036,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/618018110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":618018110},{"type":"v1","addr":"172.21.15.40:6835","nonce":618018110}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6036,"qdb_cluster":[5284,6027,6036]},"id":3}]} 2024-04-26T23:05:13.482 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 46 2024-04-26T23:05:13.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.484+0000 7fc906400640 1 -- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc8dc06c8f0 msgr2=0x7fc8dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:13.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.484+0000 7fc906400640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc8dc06c8f0 0x7fc8dc06edb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0033000 tx=0x7fc8f0009000 comp rx=0 tx=0).stop 2024-04-26T23:05:13.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.484+0000 7fc906400640 1 -- 172.21.15.40:0/178605035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9000b6220 msgr2=0x7fc9000cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:13.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.484+0000 7fc906400640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9000b6220 0x7fc9000cb4e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800d800 tx=0x7fc8e800dcc0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.484+0000 7fc906400640 1 -- 172.21.15.40:0/178605035 shutdown_connections 2024-04-26T23:05:13.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.484+0000 7fc906400640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc8dc06c8f0 0x7fc8dc06edb0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.484+0000 7fc906400640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9000d3820 0x7fc9000b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.484+0000 7fc906400640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9000d08d0 0x7fc9000b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.484+0000 7fc906400640 1 --2- 172.21.15.40:0/178605035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9000b6220 0x7fc9000cb4e0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.484+0000 7fc906400640 1 -- 172.21.15.40:0/178605035 >> 172.21.15.40:0/178605035 conn(0x7fc9000d4d90 msgr2=0x7fc9000ba0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:13.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.484+0000 7fc906400640 1 -- 172.21.15.40:0/178605035 shutdown_connections 2024-04-26T23:05:13.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.484+0000 7fc906400640 1 -- 172.21.15.40:0/178605035 wait complete. 2024-04-26T23:05:13.502 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:05:13.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.714+0000 7f5333e00640 1 Processor -- start 2024-04-26T23:05:13.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.714+0000 7f5333e00640 1 -- start start 2024-04-26T23:05:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.715+0000 7f5333e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53340cc980 0x7f53340cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.715+0000 7f5333e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53340cc230 0x7f5334158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.715+0000 7f5333e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53340cce10 con 0x7f53340c6f10 2024-04-26T23:05:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.715+0000 7f5333e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f53340c72f0 con 0x7f53340cc980 2024-04-26T23:05:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.715+0000 7f5333e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f53340cb860 con 0x7f53340cc230 2024-04-26T23:05:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.715+0000 7f5332000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53340cc980 0x7f53340cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.715+0000 7f5332000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53340cc980 0x7f53340cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37332/0 (socket says 172.21.15.40:37332) 2024-04-26T23:05:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.715+0000 7f5332000640 1 -- 172.21.15.40:0/3775621245 learned_addr learned my addr 172.21.15.40:0/3775621245 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.716+0000 7f5333400640 1 --2- 172.21.15.40:0/3775621245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53340cc230 0x7f5334158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.716+0000 7f5332000640 1 -- 172.21.15.40:0/3775621245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53340cc230 msgr2=0x7f5334158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.716+0000 7f5332000640 1 --2- 172.21.15.40:0/3775621245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53340cc230 0x7f5334158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.716+0000 7f5332000640 1 -- 172.21.15.40:0/3775621245 >> v1:172.21.15.40:6789/0 conn(0x7f53340c6f10 legacy=0x7f53340c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.716+0000 7f5332000640 1 -- 172.21.15.40:0/3775621245 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53340cd0f0 con 0x7f53340cc980 2024-04-26T23:05:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.716+0000 7f5333400640 1 --2- 172.21.15.40:0/3775621245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53340cc230 0x7f5334158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.717+0000 7f5332000640 1 --2- 172.21.15.40:0/3775621245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53340cc980 0x7f53340cbcf0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f5320009870 tx=0x7f532002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34adda6031013abb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:13.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.717+0000 7f5331600640 1 -- 172.21.15.40:0/3775621245 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5320037070 con 0x7f53340cc980 2024-04-26T23:05:13.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.717+0000 7f5331600640 1 -- 172.21.15.40:0/3775621245 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5320032440 con 0x7f53340cc980 2024-04-26T23:05:13.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.717+0000 7f5331600640 1 -- 172.21.15.40:0/3775621245 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5320032810 con 0x7f53340cc980 2024-04-26T23:05:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.718+0000 7f5333e00640 1 -- 172.21.15.40:0/3775621245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53340cc980 msgr2=0x7f53340cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.718+0000 7f5333e00640 1 --2- 172.21.15.40:0/3775621245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53340cc980 0x7f53340cbcf0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f5320009870 tx=0x7f532002a1c0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.718+0000 7f5333e00640 1 -- 172.21.15.40:0/3775621245 shutdown_connections 2024-04-26T23:05:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.718+0000 7f5333e00640 1 --2- 172.21.15.40:0/3775621245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53340cc230 0x7f5334158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.718+0000 7f5333e00640 1 --2- 172.21.15.40:0/3775621245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53340cc980 0x7f53340cbcf0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.718+0000 7f5333e00640 1 -- 172.21.15.40:0/3775621245 >> 172.21.15.40:0/3775621245 conn(0x7f53340d0a10 msgr2=0x7f53340d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.718+0000 7f5333e00640 1 -- 172.21.15.40:0/3775621245 shutdown_connections 2024-04-26T23:05:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.718+0000 7f5333e00640 1 -- 172.21.15.40:0/3775621245 wait complete. 2024-04-26T23:05:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.722+0000 7f5333e00640 1 Processor -- start 2024-04-26T23:05:13.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.722+0000 7f5333e00640 1 -- start start 2024-04-26T23:05:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53340c6f10 0x7f53340b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53340cc230 0x7f53340b1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f533414da90 0x7f533414a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f53340c61e0 con 0x7f533414da90 2024-04-26T23:05:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f53340cb1a0 con 0x7f53340cc230 2024-04-26T23:05:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f53340c6360 con 0x7f53340c6f10 2024-04-26T23:05:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f533414da90 0x7f533414a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f533414da90 0x7f533414a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37346/0 (socket says 172.21.15.40:37346) 2024-04-26T23:05:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333400640 1 -- 172.21.15.40:0/6397238 learned_addr learned my addr 172.21.15.40:0/6397238 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5332a00640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53340c6f10 0x7f53340b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5332000640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53340cc230 0x7f53340b1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333400640 1 -- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53340c6f10 msgr2=0x7f53340b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333400640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53340c6f10 0x7f53340b53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333400640 1 -- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53340cc230 msgr2=0x7f53340b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333400640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53340cc230 0x7f53340b1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.723+0000 7f5333400640 1 -- 172.21.15.40:0/6397238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53340b5ca0 con 0x7f533414da90 2024-04-26T23:05:13.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.724+0000 7f5333400640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f533414da90 0x7f533414a140 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f532c00cef0 tx=0x7f532c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:13.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.724+0000 7f532aa00640 1 -- 172.21.15.40:0/6397238 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c01a070 con 0x7f533414da90 2024-04-26T23:05:13.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.724+0000 7f532aa00640 1 -- 172.21.15.40:0/6397238 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f532c00d860 con 0x7f533414da90 2024-04-26T23:05:13.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.724+0000 7f532aa00640 1 -- 172.21.15.40:0/6397238 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c00dc50 con 0x7f533414da90 2024-04-26T23:05:13.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.724+0000 7f5333e00640 1 -- 172.21.15.40:0/6397238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53340b5f80 con 0x7f533414da90 2024-04-26T23:05:13.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.724+0000 7f5333e00640 1 -- 172.21.15.40:0/6397238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53340b6540 con 0x7f533414da90 2024-04-26T23:05:13.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.725+0000 7f5333e00640 1 -- 172.21.15.40:0/6397238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53340cfb40 con 0x7f533414da90 2024-04-26T23:05:13.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.726+0000 7f532aa00640 1 -- 172.21.15.40:0/6397238 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f532c01d070 con 0x7f533414da90 2024-04-26T23:05:13.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.727+0000 7f532aa00640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f531406c8f0 0x7f531406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:13.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.727+0000 7f532aa00640 1 -- 172.21.15.40:0/6397238 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f532c08c640 con 0x7f533414da90 2024-04-26T23:05:13.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.728+0000 7f5332a00640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f531406c8f0 0x7f531406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:13.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.729+0000 7f5332a00640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f531406c8f0 0x7f531406edb0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5324007fd0 tx=0x7f5324009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:13.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.732+0000 7f532aa00640 1 -- 172.21.15.40:0/6397238 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f53340cfb40 con 0x7f533414da90 2024-04-26T23:05:13.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.935+0000 7f5333e00640 1 -- 172.21.15.40:0/6397238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5334070020 con 0x7f533414da90 2024-04-26T23:05:13.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.937+0000 7f532aa00640 1 -- 172.21.15.40:0/6397238 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+17002 (secure 0 0 0) 0x7f532c05a8d0 con 0x7f533414da90 2024-04-26T23:05:13.937 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:13.937 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":54,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:04:48.353789+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:04:14.334803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-26T23:04:15.535793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1a","mappings":[{"from":0,"to":1}]},{"pgid":"8.27","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:05:13.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.940+0000 7f5333e00640 1 -- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f531406c8f0 msgr2=0x7f531406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:13.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.940+0000 7f5333e00640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f531406c8f0 0x7f531406edb0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5324007fd0 tx=0x7f5324009040 comp rx=0 tx=0).stop 2024-04-26T23:05:13.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.940+0000 7f5333e00640 1 -- 172.21.15.40:0/6397238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f533414da90 msgr2=0x7f533414a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:13.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.940+0000 7f5333e00640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f533414da90 0x7f533414a140 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f532c00cef0 tx=0x7f532c00a710 comp rx=0 tx=0).stop 2024-04-26T23:05:13.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.940+0000 7f5333e00640 1 -- 172.21.15.40:0/6397238 shutdown_connections 2024-04-26T23:05:13.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.940+0000 7f5333e00640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f531406c8f0 0x7f531406edb0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.940+0000 7f5333e00640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f533414da90 0x7f533414a140 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.940+0000 7f5333e00640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53340cc230 0x7f53340b1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.940+0000 7f5333e00640 1 --2- 172.21.15.40:0/6397238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53340c6f10 0x7f53340b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:13.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.940+0000 7f5333e00640 1 -- 172.21.15.40:0/6397238 >> 172.21.15.40:0/6397238 conn(0x7f53340d0a10 msgr2=0x7f53340c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:13.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.940+0000 7f5333e00640 1 -- 172.21.15.40:0/6397238 shutdown_connections 2024-04-26T23:05:13.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:13.940+0000 7f5333e00640 1 -- 172.21.15.40:0/6397238 wait complete. 2024-04-26T23:05:13.959 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:04:14.334803+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:04:15.535793+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:05:13.959 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:05:14.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.174+0000 7f0c87e00640 1 Processor -- start 2024-04-26T23:05:14.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.175+0000 7f0c87e00640 1 -- start start 2024-04-26T23:05:14.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.176+0000 7f0c87e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0c88073a50 0x7f0c88072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:14.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.176+0000 7f0c87e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c88161880 con 0x7f0c88160a50 2024-04-26T23:05:14.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.176+0000 7f0c87e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c880cff80 con 0x7f0c880cfaf0 2024-04-26T23:05:14.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.176+0000 7f0c87e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f0c88071ce0 con 0x7f0c88073a50 2024-04-26T23:05:14.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.176+0000 7f0c86a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0c88160a50 0x7f0c88160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58258/0 (socket says 172.21.15.40:58258) 2024-04-26T23:05:14.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.176+0000 7f0c86a00640 1 -- 172.21.15.40:0/2220428919 learned_addr learned my addr 172.21.15.40:0/2220428919 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:14.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.176+0000 7f0c87400640 1 --2- 172.21.15.40:0/2220428919 >> v2:172.21.15.40:3300/0 conn(0x7f0c88073a50 0x7f0c88072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:14.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.176+0000 7f0c85600640 1 -- 172.21.15.40:0/2220428919 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2490126826 0 0) 0x7f0c88161880 con 0x7f0c88160a50 2024-04-26T23:05:14.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.176+0000 7f0c85600640 1 -- 172.21.15.40:0/2220428919 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c6c003520 con 0x7f0c88160a50 2024-04-26T23:05:14.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.176+0000 7f0c87400640 1 -- 172.21.15.40:0/2220428919 >> v1:172.21.15.103:6789/0 conn(0x7f0c880cfaf0 legacy=0x7f0c8806dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:14.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.176+0000 7f0c87400640 1 -- 172.21.15.40:0/2220428919 >> v1:172.21.15.40:6789/0 conn(0x7f0c88160a50 legacy=0x7f0c88160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:14.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.176+0000 7f0c87400640 1 -- 172.21.15.40:0/2220428919 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c880d0260 con 0x7f0c88073a50 2024-04-26T23:05:14.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.178+0000 7f0c87400640 1 --2- 172.21.15.40:0/2220428919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c88073a50 0x7f0c88072730 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f0c80009d90 tx=0x7f0c8002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501659ea167a1463 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:14.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.178+0000 7f0c85600640 1 -- 172.21.15.40:0/2220428919 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c80007740 con 0x7f0c88073a50 2024-04-26T23:05:14.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.178+0000 7f0c85600640 1 -- 172.21.15.40:0/2220428919 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c800078e0 con 0x7f0c88073a50 2024-04-26T23:05:14.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.178+0000 7f0c85600640 1 -- 172.21.15.40:0/2220428919 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c80007cb0 con 0x7f0c88073a50 2024-04-26T23:05:14.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.178+0000 7f0c87e00640 1 -- 172.21.15.40:0/2220428919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c88073a50 msgr2=0x7f0c88072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:14.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.178+0000 7f0c87e00640 1 --2- 172.21.15.40:0/2220428919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c88073a50 0x7f0c88072730 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f0c80009d90 tx=0x7f0c8002a6f0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.179+0000 7f0c87e00640 1 -- 172.21.15.40:0/2220428919 shutdown_connections 2024-04-26T23:05:14.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.179+0000 7f0c87e00640 1 --2- 172.21.15.40:0/2220428919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c88073a50 0x7f0c88072730 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.179+0000 7f0c87e00640 1 -- 172.21.15.40:0/2220428919 >> 172.21.15.40:0/2220428919 conn(0x7f0c880b2720 msgr2=0x7f0c880b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:14.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.179+0000 7f0c87e00640 1 -- 172.21.15.40:0/2220428919 shutdown_connections 2024-04-26T23:05:14.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.179+0000 7f0c87e00640 1 -- 172.21.15.40:0/2220428919 wait complete. 2024-04-26T23:05:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.180+0000 7f0c87e00640 1 Processor -- start 2024-04-26T23:05:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.181+0000 7f0c87e00640 1 -- start start 2024-04-26T23:05:14.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c87e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c88073a50 0x7f0c88143030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:14.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c87e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c88160a50 0x7f0c880b1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:14.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c87e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c8815a300 0x7f0c881596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:14.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c87e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0c88073f00 con 0x7f0c88160a50 2024-04-26T23:05:14.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c87e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0c880722a0 con 0x7f0c88073a50 2024-04-26T23:05:14.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c87e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0c881468f0 con 0x7f0c8815a300 2024-04-26T23:05:14.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c86000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c88160a50 0x7f0c880b1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:14.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c86000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c88160a50 0x7f0c880b1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37354/0 (socket says 172.21.15.40:37354) 2024-04-26T23:05:14.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c86000640 1 -- 172.21.15.40:0/1042277688 learned_addr learned my addr 172.21.15.40:0/1042277688 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:14.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c86a00640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c88073a50 0x7f0c88143030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:14.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c86000640 1 -- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c8815a300 msgr2=0x7f0c881596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:14.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c87400640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c8815a300 0x7f0c881596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:14.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c86000640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c8815a300 0x7f0c881596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c86000640 1 -- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c88073a50 msgr2=0x7f0c88143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:14.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c86000640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c88073a50 0x7f0c88143030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c86000640 1 -- 172.21.15.40:0/1042277688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c88143570 con 0x7f0c88160a50 2024-04-26T23:05:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c86a00640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c88073a50 0x7f0c88143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c86000640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c88160a50 0x7f0c880b1a80 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400c450 tx=0x7f0c7400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c7ea00640 1 -- 172.21.15.40:0/1042277688 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c740052d0 con 0x7f0c88160a50 2024-04-26T23:05:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c7ea00640 1 -- 172.21.15.40:0/1042277688 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c74005470 con 0x7f0c88160a50 2024-04-26T23:05:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c87e00640 1 -- 172.21.15.40:0/1042277688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c8815e5e0 con 0x7f0c88160a50 2024-04-26T23:05:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c7ea00640 1 -- 172.21.15.40:0/1042277688 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c74012870 con 0x7f0c88160a50 2024-04-26T23:05:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.183+0000 7f0c87e00640 1 -- 172.21.15.40:0/1042277688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c8815ee80 con 0x7f0c88160a50 2024-04-26T23:05:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.184+0000 7f0c87e00640 1 -- 172.21.15.40:0/1042277688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c8815e9d0 con 0x7f0c88160a50 2024-04-26T23:05:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.186+0000 7f0c7ea00640 1 -- 172.21.15.40:0/1042277688 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0c74012a10 con 0x7f0c88160a50 2024-04-26T23:05:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.186+0000 7f0c7ea00640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0c6806c8f0 0x7f0c6806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.187+0000 7f0c7ea00640 1 -- 172.21.15.40:0/1042277688 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f0c7408b9b0 con 0x7f0c88160a50 2024-04-26T23:05:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.187+0000 7f0c86a00640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0c6806c8f0 0x7f0c6806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:14.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.188+0000 7f0c86a00640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0c6806c8f0 0x7f0c6806edb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0c780011e0 tx=0x7f0c78009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:14.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.191+0000 7f0c7ea00640 1 -- 172.21.15.40:0/1042277688 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0c88143570 con 0x7f0c88160a50 2024-04-26T23:05:14.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.395+0000 7f0c87e00640 1 -- 172.21.15.40:0/1042277688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0c88069370 con 0x7f0c88160a50 2024-04-26T23:05:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.396+0000 7f0c7ea00640 1 -- 172.21.15.40:0/1042277688 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) ==== 93+0+10 (secure 0 0 0) 0x7f0c74059b80 con 0x7f0c88160a50 2024-04-26T23:05:14.396 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:05:14.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.400+0000 7f0c87e00640 1 -- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0c6806c8f0 msgr2=0x7f0c6806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:14.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.400+0000 7f0c87e00640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0c6806c8f0 0x7f0c6806edb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0c780011e0 tx=0x7f0c78009040 comp rx=0 tx=0).stop 2024-04-26T23:05:14.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.400+0000 7f0c87e00640 1 -- 172.21.15.40:0/1042277688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c88160a50 msgr2=0x7f0c880b1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:14.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.400+0000 7f0c87e00640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c88160a50 0x7f0c880b1a80 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400c450 tx=0x7f0c7400c910 comp rx=0 tx=0).stop 2024-04-26T23:05:14.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.400+0000 7f0c87e00640 1 -- 172.21.15.40:0/1042277688 shutdown_connections 2024-04-26T23:05:14.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.400+0000 7f0c87e00640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0c6806c8f0 0x7f0c6806edb0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.400+0000 7f0c87e00640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c8815a300 0x7f0c881596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.400+0000 7f0c87e00640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c88160a50 0x7f0c880b1a80 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.400+0000 7f0c87e00640 1 --2- 172.21.15.40:0/1042277688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c88073a50 0x7f0c88143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.400+0000 7f0c87e00640 1 -- 172.21.15.40:0/1042277688 >> 172.21.15.40:0/1042277688 conn(0x7f0c880b2720 msgr2=0x7f0c880aea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:14.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.400+0000 7f0c87e00640 1 -- 172.21.15.40:0/1042277688 shutdown_connections 2024-04-26T23:05:14.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.400+0000 7f0c87e00640 1 -- 172.21.15.40:0/1042277688 wait complete. 2024-04-26T23:05:14.419 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:05:14.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.638+0000 7f34f2000640 1 Processor -- start 2024-04-26T23:05:14.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.638+0000 7f34f2000640 1 -- start start 2024-04-26T23:05:14.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.640+0000 7f34f2000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f34ec073a50 0x7f34ec072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:14.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.640+0000 7f34f2000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34ec161880 con 0x7f34ec0cfaf0 2024-04-26T23:05:14.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.640+0000 7f34f2000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34ec0cff80 con 0x7f34ec160a50 2024-04-26T23:05:14.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.640+0000 7f34f2000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f34ec071ce0 con 0x7f34ec073a50 2024-04-26T23:05:14.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.640+0000 7f34e3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f34ec0cfaf0 0x7f34ec06dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58274/0 (socket says 172.21.15.40:58274) 2024-04-26T23:05:14.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.640+0000 7f34f1600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f34ec073a50 0x7f34ec072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:14.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.640+0000 7f34e3e00640 1 -- 172.21.15.40:0/1025219614 learned_addr learned my addr 172.21.15.40:0/1025219614 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:14.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.641+0000 7f34f1600640 1 -- 172.21.15.40:0/1025219614 >> v1:172.21.15.103:6789/0 conn(0x7f34ec160a50 legacy=0x7f34ec160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:14.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.641+0000 7f34f1600640 1 -- 172.21.15.40:0/1025219614 >> v1:172.21.15.40:6789/0 conn(0x7f34ec0cfaf0 legacy=0x7f34ec06dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:14.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.641+0000 7f34f1600640 1 -- 172.21.15.40:0/1025219614 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ec0d0260 con 0x7f34ec073a50 2024-04-26T23:05:14.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.642+0000 7f34f1600640 1 --2- 172.21.15.40:0/1025219614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34ec073a50 0x7f34ec072730 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009d90 tx=0x7f34e402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede94ea9a8b53fab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:14.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.642+0000 7f34ebe00640 1 -- 172.21.15.40:0/1025219614 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4007740 con 0x7f34ec073a50 2024-04-26T23:05:14.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.643+0000 7f34ebe00640 1 -- 172.21.15.40:0/1025219614 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e40078e0 con 0x7f34ec073a50 2024-04-26T23:05:14.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.643+0000 7f34ebe00640 1 -- 172.21.15.40:0/1025219614 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4007cb0 con 0x7f34ec073a50 2024-04-26T23:05:14.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.643+0000 7f34f2000640 1 -- 172.21.15.40:0/1025219614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34ec073a50 msgr2=0x7f34ec072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:14.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.643+0000 7f34f2000640 1 --2- 172.21.15.40:0/1025219614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34ec073a50 0x7f34ec072730 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009d90 tx=0x7f34e402a6a0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.643+0000 7f34f2000640 1 -- 172.21.15.40:0/1025219614 shutdown_connections 2024-04-26T23:05:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.643+0000 7f34f2000640 1 --2- 172.21.15.40:0/1025219614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34ec073a50 0x7f34ec072730 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.643+0000 7f34f2000640 1 -- 172.21.15.40:0/1025219614 >> 172.21.15.40:0/1025219614 conn(0x7f34ec0b2720 msgr2=0x7f34ec0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.643+0000 7f34f2000640 1 -- 172.21.15.40:0/1025219614 shutdown_connections 2024-04-26T23:05:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.643+0000 7f34f2000640 1 -- 172.21.15.40:0/1025219614 wait complete. 2024-04-26T23:05:14.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.645+0000 7f34f2000640 1 Processor -- start 2024-04-26T23:05:14.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.645+0000 7f34f2000640 1 -- start start 2024-04-26T23:05:14.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.646+0000 7f34f2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34ec073a50 0x7f34ec15a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:14.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.646+0000 7f34f2000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34ec160a50 0x7f34ec0b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:14.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.646+0000 7f34f2000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34ec142ee0 0x7f34ec1434a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:14.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.646+0000 7f34f2000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec073f00 con 0x7f34ec073a50 2024-04-26T23:05:14.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.646+0000 7f34f2000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec0722a0 con 0x7f34ec142ee0 2024-04-26T23:05:14.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.646+0000 7f34f2000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f34ec1468f0 con 0x7f34ec160a50 2024-04-26T23:05:14.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.647+0000 7f34f0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34ec073a50 0x7f34ec15a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:14.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.647+0000 7f34f0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34ec073a50 0x7f34ec15a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37384/0 (socket says 172.21.15.40:37384) 2024-04-26T23:05:14.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.647+0000 7f34f0c00640 1 -- 172.21.15.40:0/3055993285 learned_addr learned my addr 172.21.15.40:0/3055993285 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:14.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.647+0000 7f34f0c00640 1 -- 172.21.15.40:0/3055993285 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34ec160a50 msgr2=0x7f34ec0b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:14.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.647+0000 7f34f0c00640 1 --2- 172.21.15.40:0/3055993285 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34ec160a50 0x7f34ec0b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.647+0000 7f34f0c00640 1 -- 172.21.15.40:0/3055993285 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34ec142ee0 msgr2=0x7f34ec1434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:14.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.647+0000 7f34f0c00640 1 --2- 172.21.15.40:0/3055993285 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34ec142ee0 0x7f34ec1434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.647+0000 7f34f0c00640 1 -- 172.21.15.40:0/3055993285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ec0b49c0 con 0x7f34ec073a50 2024-04-26T23:05:14.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.647+0000 7f34f0c00640 1 --2- 172.21.15.40:0/3055993285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34ec073a50 0x7f34ec15a610 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f34d400b3f0 tx=0x7f34d400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:14.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.648+0000 7f34e9600640 1 -- 172.21.15.40:0/3055993285 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4015020 con 0x7f34ec073a50 2024-04-26T23:05:14.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.648+0000 7f34e9600640 1 -- 172.21.15.40:0/3055993285 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d4011830 con 0x7f34ec073a50 2024-04-26T23:05:14.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.648+0000 7f34f2000640 1 -- 172.21.15.40:0/3055993285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34ec0b4ca0 con 0x7f34ec073a50 2024-04-26T23:05:14.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.648+0000 7f34e9600640 1 -- 172.21.15.40:0/3055993285 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4011c20 con 0x7f34ec073a50 2024-04-26T23:05:14.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.648+0000 7f34f2000640 1 -- 172.21.15.40:0/3055993285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34ec0b5260 con 0x7f34ec073a50 2024-04-26T23:05:14.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.649+0000 7f34f2000640 1 -- 172.21.15.40:0/3055993285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34ec069370 con 0x7f34ec073a50 2024-04-26T23:05:14.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.650+0000 7f34e9600640 1 -- 172.21.15.40:0/3055993285 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f34d4018020 con 0x7f34ec073a50 2024-04-26T23:05:14.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.651+0000 7f34e9600640 1 --2- 172.21.15.40:0/3055993285 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f34c806c970 0x7f34c806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:14.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.651+0000 7f34e9600640 1 -- 172.21.15.40:0/3055993285 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f34d408b8e0 con 0x7f34ec073a50 2024-04-26T23:05:14.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.652+0000 7f34e3e00640 1 --2- 172.21.15.40:0/3055993285 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f34c806c970 0x7f34c806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:14.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.653+0000 7f34e3e00640 1 --2- 172.21.15.40:0/3055993285 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f34c806c970 0x7f34c806ee30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f34dc000f60 tx=0x7f34dc007a80 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:14.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.656+0000 7f34e9600640 1 -- 172.21.15.40:0/3055993285 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f34d401f030 con 0x7f34ec073a50 2024-04-26T23:05:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.857+0000 7f34f2000640 1 -- 172.21.15.40:0/3055993285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f34ec06f170 con 0x7f34ec073a50 2024-04-26T23:05:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.858+0000 7f34e9600640 1 -- 172.21.15.40:0/3055993285 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) ==== 104+0+11 (secure 0 0 0) 0x7f34d40172c0 con 0x7f34ec073a50 2024-04-26T23:05:14.858 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:05:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.860+0000 7f34f2000640 1 -- 172.21.15.40:0/3055993285 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f34c806c970 msgr2=0x7f34c806ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.860+0000 7f34f2000640 1 --2- 172.21.15.40:0/3055993285 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f34c806c970 0x7f34c806ee30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f34dc000f60 tx=0x7f34dc007a80 comp rx=0 tx=0).stop 2024-04-26T23:05:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.860+0000 7f34f2000640 1 -- 172.21.15.40:0/3055993285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34ec073a50 msgr2=0x7f34ec15a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.860+0000 7f34f2000640 1 --2- 172.21.15.40:0/3055993285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34ec073a50 0x7f34ec15a610 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f34d400b3f0 tx=0x7f34d400b8b0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.861+0000 7f34f2000640 1 -- 172.21.15.40:0/3055993285 shutdown_connections 2024-04-26T23:05:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.861+0000 7f34f2000640 1 --2- 172.21.15.40:0/3055993285 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f34c806c970 0x7f34c806ee30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.861+0000 7f34f2000640 1 --2- 172.21.15.40:0/3055993285 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34ec142ee0 0x7f34ec1434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.861+0000 7f34f2000640 1 --2- 172.21.15.40:0/3055993285 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f34ec160a50 0x7f34ec0b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.861+0000 7f34f2000640 1 --2- 172.21.15.40:0/3055993285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34ec073a50 0x7f34ec15a610 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.861+0000 7f34f2000640 1 -- 172.21.15.40:0/3055993285 >> 172.21.15.40:0/3055993285 conn(0x7f34ec0b2720 msgr2=0x7f34ec15dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.861+0000 7f34f2000640 1 -- 172.21.15.40:0/3055993285 shutdown_connections 2024-04-26T23:05:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:14.861+0000 7f34f2000640 1 -- 172.21.15.40:0/3055993285 wait complete. 2024-04-26T23:05:14.881 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:05:15.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.093+0000 7f0387200640 1 Processor -- start 2024-04-26T23:05:15.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.094+0000 7f0387200640 1 -- start start 2024-04-26T23:05:15.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.095+0000 7f0387200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f03800af7a0 0x7f03800afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:15.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.095+0000 7f0387200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0380071fe0 0x7f0380073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:15.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.095+0000 7f0387200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03800d0820 con 0x7f038015ce60 2024-04-26T23:05:15.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.095+0000 7f0387200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f03800cc450 con 0x7f03800af7a0 2024-04-26T23:05:15.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.095+0000 7f0387200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f03801614c0 con 0x7f0380071fe0 2024-04-26T23:05:15.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.096+0000 7f0386800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0380071fe0 0x7f0380073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:15.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.096+0000 7f0385400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f03800af7a0 0x7f03800afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:15.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.096+0000 7f0386800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0380071fe0 0x7f0380073760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:58768/0 (socket says 172.21.15.40:58768) 2024-04-26T23:05:15.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.096+0000 7f0386800640 1 -- 172.21.15.40:0/1932884924 learned_addr learned my addr 172.21.15.40:0/1932884924 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:15.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.097+0000 7f0386800640 1 -- 172.21.15.40:0/1932884924 >> v2:172.21.15.103:3300/0 conn(0x7f03800af7a0 msgr2=0x7f03800afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:15.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.097+0000 7f0386800640 1 --2- 172.21.15.40:0/1932884924 >> v2:172.21.15.103:3300/0 conn(0x7f03800af7a0 0x7f03800afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.097+0000 7f0386800640 1 -- 172.21.15.40:0/1932884924 >> v1:172.21.15.103:6789/0 conn(0x7f038015ce60 legacy=0x7f03800ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:15.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.097+0000 7f0386800640 1 -- 172.21.15.40:0/1932884924 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0380161760 con 0x7f0380071fe0 2024-04-26T23:05:15.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.099+0000 7f0386800640 1 --2- 172.21.15.40:0/1932884924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0380071fe0 0x7f0380073760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f037c009e50 tx=0x7f037c02a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fac6c83e36052e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:15.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.099+0000 7f0384a00640 1 -- 172.21.15.40:0/1932884924 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c036070 con 0x7f0380071fe0 2024-04-26T23:05:15.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.099+0000 7f0384a00640 1 -- 172.21.15.40:0/1932884924 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f037c031600 con 0x7f0380071fe0 2024-04-26T23:05:15.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.099+0000 7f0384a00640 1 -- 172.21.15.40:0/1932884924 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c0319d0 con 0x7f0380071fe0 2024-04-26T23:05:15.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.100+0000 7f0387200640 1 -- 172.21.15.40:0/1932884924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0380071fe0 msgr2=0x7f0380073760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:15.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.100+0000 7f0387200640 1 --2- 172.21.15.40:0/1932884924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0380071fe0 0x7f0380073760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f037c009e50 tx=0x7f037c02a680 comp rx=0 tx=0).stop 2024-04-26T23:05:15.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.100+0000 7f0387200640 1 -- 172.21.15.40:0/1932884924 shutdown_connections 2024-04-26T23:05:15.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.100+0000 7f0387200640 1 --2- 172.21.15.40:0/1932884924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0380071fe0 0x7f0380073760 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.100+0000 7f0387200640 1 --2- 172.21.15.40:0/1932884924 >> v2:172.21.15.103:3300/0 conn(0x7f03800af7a0 0x7f03800afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.100+0000 7f0387200640 1 -- 172.21.15.40:0/1932884924 >> 172.21.15.40:0/1932884924 conn(0x7f03800d0a10 msgr2=0x7f03800d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:15.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.100+0000 7f0387200640 1 -- 172.21.15.40:0/1932884924 shutdown_connections 2024-04-26T23:05:15.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.100+0000 7f0387200640 1 -- 172.21.15.40:0/1932884924 wait complete. 2024-04-26T23:05:15.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.102+0000 7f0387200640 1 Processor -- start 2024-04-26T23:05:15.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.103+0000 7f0387200640 1 -- start start 2024-04-26T23:05:15.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.104+0000 7f0387200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0380071fe0 0x7f038014b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:15.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.104+0000 7f0387200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03800af7a0 0x7f038006ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:15.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.104+0000 7f0387200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f038015ce60 0x7f038006f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:15.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.104+0000 7f0387200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f03800cc1f0 con 0x7f038015ce60 2024-04-26T23:05:15.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.104+0000 7f0387200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f03800cc070 con 0x7f03800af7a0 2024-04-26T23:05:15.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.104+0000 7f0387200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0380073ca0 con 0x7f0380071fe0 2024-04-26T23:05:15.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.104+0000 7f0386800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f038015ce60 0x7f038006f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:15.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.104+0000 7f0386800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f038015ce60 0x7f038006f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44746/0 (socket says 172.21.15.40:44746) 2024-04-26T23:05:15.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.104+0000 7f0386800640 1 -- 172.21.15.40:0/3072753787 learned_addr learned my addr 172.21.15.40:0/3072753787 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:15.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.105+0000 7f0385e00640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0380071fe0 0x7f038014b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:15.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.105+0000 7f0385400640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03800af7a0 0x7f038006ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:15.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.105+0000 7f0386800640 1 -- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0380071fe0 msgr2=0x7f038014b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:15.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.105+0000 7f0386800640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0380071fe0 0x7f038014b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.105+0000 7f0386800640 1 -- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03800af7a0 msgr2=0x7f038006ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:15.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.105+0000 7f0386800640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03800af7a0 0x7f038006ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.105+0000 7f0386800640 1 -- 172.21.15.40:0/3072753787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03800c6a30 con 0x7f038015ce60 2024-04-26T23:05:15.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.105+0000 7f0385400640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03800af7a0 0x7f038006ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:15.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.105+0000 7f0386800640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f038015ce60 0x7f038006f450 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f037c009f70 tx=0x7f037c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:15.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.105+0000 7f037a000640 1 -- 172.21.15.40:0/3072753787 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c041030 con 0x7f038015ce60 2024-04-26T23:05:15.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.105+0000 7f037a000640 1 -- 172.21.15.40:0/3072753787 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f037c031400 con 0x7f038015ce60 2024-04-26T23:05:15.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.105+0000 7f037a000640 1 -- 172.21.15.40:0/3072753787 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c036040 con 0x7f038015ce60 2024-04-26T23:05:15.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.106+0000 7f0387200640 1 -- 172.21.15.40:0/3072753787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03800c6cd0 con 0x7f038015ce60 2024-04-26T23:05:15.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.106+0000 7f0387200640 1 -- 172.21.15.40:0/3072753787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03800c7290 con 0x7f038015ce60 2024-04-26T23:05:15.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.107+0000 7f0387200640 1 -- 172.21.15.40:0/3072753787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0380069370 con 0x7f038015ce60 2024-04-26T23:05:15.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.108+0000 7f037a000640 1 -- 172.21.15.40:0/3072753787 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f037c044070 con 0x7f038015ce60 2024-04-26T23:05:15.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.109+0000 7f037a000640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f036406c920 0x7f036406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:15.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.109+0000 7f037a000640 1 -- 172.21.15.40:0/3072753787 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f037c0ac3e0 con 0x7f038015ce60 2024-04-26T23:05:15.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.110+0000 7f0385e00640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f036406c920 0x7f036406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:15.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.111+0000 7f0385e00640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f036406c920 0x7f036406ede0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0374000f60 tx=0x7f0374009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:15.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.114+0000 7f037a000640 1 -- 172.21.15.40:0/3072753787 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f037c04b070 con 0x7f038015ce60 2024-04-26T23:05:15.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.316+0000 7f0387200640 1 -- 172.21.15.40:0/3072753787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f03800d01f0 con 0x7f038015ce60 2024-04-26T23:05:15.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.316+0000 7f037a000640 1 -- 172.21.15.40:0/3072753787 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) ==== 100+0+11 (secure 0 0 0) 0x7f037c07a6e0 con 0x7f038015ce60 2024-04-26T23:05:15.317 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:05:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.319+0000 7f0387200640 1 -- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f036406c920 msgr2=0x7f036406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.319+0000 7f0387200640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f036406c920 0x7f036406ede0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0374000f60 tx=0x7f0374009040 comp rx=0 tx=0).stop 2024-04-26T23:05:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.319+0000 7f0387200640 1 -- 172.21.15.40:0/3072753787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f038015ce60 msgr2=0x7f038006f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.319+0000 7f0387200640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f038015ce60 0x7f038006f450 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f037c009f70 tx=0x7f037c004a40 comp rx=0 tx=0).stop 2024-04-26T23:05:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.319+0000 7f0387200640 1 -- 172.21.15.40:0/3072753787 shutdown_connections 2024-04-26T23:05:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.319+0000 7f0387200640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f036406c920 0x7f036406ede0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.319+0000 7f0387200640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f038015ce60 0x7f038006f450 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.319+0000 7f0387200640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03800af7a0 0x7f038006ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.319+0000 7f0387200640 1 --2- 172.21.15.40:0/3072753787 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0380071fe0 0x7f038014b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.319+0000 7f0387200640 1 -- 172.21.15.40:0/3072753787 >> 172.21.15.40:0/3072753787 conn(0x7f03800d0a10 msgr2=0x7f0380155ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.320+0000 7f0387200640 1 -- 172.21.15.40:0/3072753787 shutdown_connections 2024-04-26T23:05:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.320+0000 7f0387200640 1 -- 172.21.15.40:0/3072753787 wait complete. 2024-04-26T23:05:15.340 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:05:15.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.546+0000 7f2027000640 1 Processor -- start 2024-04-26T23:05:15.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.546+0000 7f2027000640 1 -- start start 2024-04-26T23:05:15.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.548+0000 7f2027000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f20200af7a0 0x7f20200afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:15.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.548+0000 7f2027000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20200d0820 con 0x7f202015ce60 2024-04-26T23:05:15.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.548+0000 7f2027000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20201614c0 con 0x7f2020071fe0 2024-04-26T23:05:15.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.548+0000 7f2027000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f20200cc450 con 0x7f20200af7a0 2024-04-26T23:05:15.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.548+0000 7f2025200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f20200af7a0 0x7f20200afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:15.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.548+0000 7f2025c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f202015ce60 0x7f20200ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50564/0 (socket says 172.21.15.40:50564) 2024-04-26T23:05:15.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.548+0000 7f2025c00640 1 -- 172.21.15.40:0/3595282852 learned_addr learned my addr 172.21.15.40:0/3595282852 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:15.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.549+0000 7f201fe00640 1 -- 172.21.15.40:0/3595282852 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2436965670 0 0) 0x7f20200d0820 con 0x7f202015ce60 2024-04-26T23:05:15.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.549+0000 7f201fe00640 1 -- 172.21.15.40:0/3595282852 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2008003520 con 0x7f202015ce60 2024-04-26T23:05:15.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.549+0000 7f2025200640 1 -- 172.21.15.40:0/3595282852 >> v1:172.21.15.103:6789/0 conn(0x7f2020071fe0 legacy=0x7f2020073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:15.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.549+0000 7f2025200640 1 -- 172.21.15.40:0/3595282852 >> v1:172.21.15.40:6789/0 conn(0x7f202015ce60 legacy=0x7f20200ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:15.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.549+0000 7f2025200640 1 -- 172.21.15.40:0/3595282852 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2020161760 con 0x7f20200af7a0 2024-04-26T23:05:15.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.550+0000 7f2025200640 1 --2- 172.21.15.40:0/3595282852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20200af7a0 0x7f20200afc60 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f200c0097a0 tx=0x7f200c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc942051ef5eb00a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:15.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.551+0000 7f201fe00640 1 -- 172.21.15.40:0/3595282852 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c037070 con 0x7f20200af7a0 2024-04-26T23:05:15.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.551+0000 7f201fe00640 1 -- 172.21.15.40:0/3595282852 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f200c032440 con 0x7f20200af7a0 2024-04-26T23:05:15.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.551+0000 7f201fe00640 1 -- 172.21.15.40:0/3595282852 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c0327f0 con 0x7f20200af7a0 2024-04-26T23:05:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.551+0000 7f2027000640 1 -- 172.21.15.40:0/3595282852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20200af7a0 msgr2=0x7f20200afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.551+0000 7f2027000640 1 --2- 172.21.15.40:0/3595282852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20200af7a0 0x7f20200afc60 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f200c0097a0 tx=0x7f200c02a1a0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.552+0000 7f2027000640 1 -- 172.21.15.40:0/3595282852 shutdown_connections 2024-04-26T23:05:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.552+0000 7f2027000640 1 --2- 172.21.15.40:0/3595282852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20200af7a0 0x7f20200afc60 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.552+0000 7f2027000640 1 -- 172.21.15.40:0/3595282852 >> 172.21.15.40:0/3595282852 conn(0x7f20200d0a10 msgr2=0x7f20200d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.552+0000 7f2027000640 1 -- 172.21.15.40:0/3595282852 shutdown_connections 2024-04-26T23:05:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.552+0000 7f2027000640 1 -- 172.21.15.40:0/3595282852 wait complete. 2024-04-26T23:05:15.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.553+0000 7f2027000640 1 Processor -- start 2024-04-26T23:05:15.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.554+0000 7f2027000640 1 -- start start 2024-04-26T23:05:15.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.554+0000 7f2027000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2020071fe0 0x7f202014d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:15.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.554+0000 7f2027000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20200af7a0 0x7f20200b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:15.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.554+0000 7f2027000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f202015ce60 0x7f20200b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:15.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2027000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f20200cc1f0 con 0x7f202015ce60 2024-04-26T23:05:15.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2027000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f20200cc070 con 0x7f20200af7a0 2024-04-26T23:05:15.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2027000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2020073e70 con 0x7f2020071fe0 2024-04-26T23:05:15.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2026600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f202015ce60 0x7f20200b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:15.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2026600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f202015ce60 0x7f20200b5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44762/0 (socket says 172.21.15.40:44762) 2024-04-26T23:05:15.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2026600640 1 -- 172.21.15.40:0/3651740322 learned_addr learned my addr 172.21.15.40:0/3651740322 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:15.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2025200640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20200af7a0 0x7f20200b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:15.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2025c00640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2020071fe0 0x7f202014d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:15.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2026600640 1 -- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2020071fe0 msgr2=0x7f202014d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:15.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2026600640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2020071fe0 0x7f202014d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2026600640 1 -- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20200af7a0 msgr2=0x7f20200b55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:15.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2026600640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20200af7a0 0x7f20200b55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2026600640 1 -- 172.21.15.40:0/3651740322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20200b4380 con 0x7f202015ce60 2024-04-26T23:05:15.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.555+0000 7f2025200640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20200af7a0 0x7f20200b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:15.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.556+0000 7f2026600640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f202015ce60 0x7f20200b5af0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f201800c7a0 tx=0x7f201800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:15.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.556+0000 7f201d600640 1 -- 172.21.15.40:0/3651740322 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018003b70 con 0x7f202015ce60 2024-04-26T23:05:15.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.556+0000 7f201d600640 1 -- 172.21.15.40:0/3651740322 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2018003d30 con 0x7f202015ce60 2024-04-26T23:05:15.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.556+0000 7f201d600640 1 -- 172.21.15.40:0/3651740322 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201800aa90 con 0x7f202015ce60 2024-04-26T23:05:15.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.556+0000 7f2027000640 1 -- 172.21.15.40:0/3651740322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20200b4660 con 0x7f202015ce60 2024-04-26T23:05:15.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.556+0000 7f2027000640 1 -- 172.21.15.40:0/3651740322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20200b4c20 con 0x7f202015ce60 2024-04-26T23:05:15.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.557+0000 7f2027000640 1 -- 172.21.15.40:0/3651740322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2020069370 con 0x7f202015ce60 2024-04-26T23:05:15.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.559+0000 7f201d600640 1 -- 172.21.15.40:0/3651740322 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f201800ac30 con 0x7f202015ce60 2024-04-26T23:05:15.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.559+0000 7f201d600640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f200006c8f0 0x7f200006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:15.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.559+0000 7f201d600640 1 -- 172.21.15.40:0/3651740322 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f201808bd50 con 0x7f202015ce60 2024-04-26T23:05:15.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.560+0000 7f2025c00640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f200006c8f0 0x7f200006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:15.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.562+0000 7f2025c00640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f200006c8f0 0x7f200006edb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f20140011e0 tx=0x7f2014009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:15.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.565+0000 7f201d600640 1 -- 172.21.15.40:0/3651740322 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2018059f20 con 0x7f202015ce60 2024-04-26T23:05:15.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.813+0000 7f2027000640 1 -- 172.21.15.40:0/3651740322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20200d01f0 con 0x7f202015ce60 2024-04-26T23:05:15.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.814+0000 7f201d600640 1 -- 172.21.15.40:0/3651740322 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5225 (secure 0 0 0) 0x7f2018059820 con 0x7f202015ce60 2024-04-26T23:05:15.814 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:15.815 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":46,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/189004193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":189004193},{"type":"v1","addr":"172.21.15.103:6835","nonce":189004193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:04:16.558591+0000","modified":"2024-04-26T23:04:55.194872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6036,"mds_1":6027,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"b","rank":2,"incarnation":39,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/670995452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":670995452},{"type":"v1","addr":"172.21.15.103:6837","nonce":670995452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6027":{"gid":6027,"name":"a","rank":1,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6834/2304583125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2304583125},{"type":"v1","addr":"172.21.15.40:6834","nonce":2304583125}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6036":{"gid":6036,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/618018110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":618018110},{"type":"v1","addr":"172.21.15.40:6835","nonce":618018110}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6036,"qdb_cluster":[5284,6027,6036]},"id":3}]} 2024-04-26T23:05:15.815 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 46 2024-04-26T23:05:15.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.816+0000 7f2027000640 1 -- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f200006c8f0 msgr2=0x7f200006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:15.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.816+0000 7f2027000640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f200006c8f0 0x7f200006edb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f20140011e0 tx=0x7f2014009040 comp rx=0 tx=0).stop 2024-04-26T23:05:15.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.816+0000 7f2027000640 1 -- 172.21.15.40:0/3651740322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f202015ce60 msgr2=0x7f20200b5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:15.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.816+0000 7f2027000640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f202015ce60 0x7f20200b5af0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f201800c7a0 tx=0x7f201800cc60 comp rx=0 tx=0).stop 2024-04-26T23:05:15.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.816+0000 7f2027000640 1 -- 172.21.15.40:0/3651740322 shutdown_connections 2024-04-26T23:05:15.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.816+0000 7f2027000640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f200006c8f0 0x7f200006edb0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.816+0000 7f2027000640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f202015ce60 0x7f20200b5af0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.816+0000 7f2027000640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20200af7a0 0x7f20200b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.816+0000 7f2027000640 1 --2- 172.21.15.40:0/3651740322 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2020071fe0 0x7f202014d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:15.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.816+0000 7f2027000640 1 -- 172.21.15.40:0/3651740322 >> 172.21.15.40:0/3651740322 conn(0x7f20200d0a10 msgr2=0x7f20200b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:15.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.816+0000 7f2027000640 1 -- 172.21.15.40:0/3651740322 shutdown_connections 2024-04-26T23:05:15.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:15.817+0000 7f2027000640 1 -- 172.21.15.40:0/3651740322 wait complete. 2024-04-26T23:05:15.836 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:05:16.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.053+0000 7fe4f0800640 1 Processor -- start 2024-04-26T23:05:16.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.053+0000 7fe4f0800640 1 -- start start 2024-04-26T23:05:16.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.054+0000 7fe4f0800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe4e8073b70 0x7fe4e80aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.054+0000 7fe4f0800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4e8072c00 con 0x7fe4e815f380 2024-04-26T23:05:16.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.055+0000 7fe4f0800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4e8072ee0 con 0x7fe4e80cbb50 2024-04-26T23:05:16.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.055+0000 7fe4f0800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe4e80729b0 con 0x7fe4e8073b70 2024-04-26T23:05:16.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.055+0000 7fe4ede00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe4e815f380 0x7fe4e80cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50568/0 (socket says 172.21.15.40:50568) 2024-04-26T23:05:16.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.055+0000 7fe4ede00640 1 -- 172.21.15.40:0/1218566481 learned_addr learned my addr 172.21.15.40:0/1218566481 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:16.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.055+0000 7fe4ed400640 1 --2- 172.21.15.40:0/1218566481 >> v2:172.21.15.103:3300/0 conn(0x7fe4e8073b70 0x7fe4e80aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.055+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2827810135 0 0) 0x7fe4e8072c00 con 0x7fe4e815f380 2024-04-26T23:05:16.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.055+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4d0003520 con 0x7fe4e815f380 2024-04-26T23:05:16.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1982530906 0 0) 0x7fe4d0003520 con 0x7fe4e815f380 2024-04-26T23:05:16.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 >> v2:172.21.15.103:3300/0 conn(0x7fe4e8073b70 msgr2=0x7fe4e80aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:16.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4eca00640 1 --2- 172.21.15.40:0/1218566481 >> v2:172.21.15.103:3300/0 conn(0x7fe4e8073b70 0x7fe4e80aecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 >> v1:172.21.15.103:6789/0 conn(0x7fe4e80cbb50 legacy=0x7fe4e80af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e80731c0 con 0x7fe4e815f380 2024-04-26T23:05:16.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fe4dc001670 con 0x7fe4e815f380 2024-04-26T23:05:16.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 >> v1:172.21.15.40:6789/0 conn(0x7fe4e815f380 legacy=0x7fe4e80cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4eca00640 1 --2- 172.21.15.40:0/1218566481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4d0003a00 0x7fe4d0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4eca00640 1 --2- 172.21.15.40:0/1218566481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d0004800 0x7fe4d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4eca00640 1 --2- 172.21.15.40:0/1218566481 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4d0005210 0x7fe4d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e80729b0 con 0x7fe4d0004800 2024-04-26T23:05:16.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d000c160 con 0x7fe4d0003a00 2024-04-26T23:05:16.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe4d000c360 con 0x7fe4d0005210 2024-04-26T23:05:16.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.056+0000 7fe4ed400640 1 --2- 172.21.15.40:0/1218566481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d0004800 0x7fe4d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.057+0000 7fe4ede00640 1 --2- 172.21.15.40:0/1218566481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4d0003a00 0x7fe4d0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.057+0000 7fe4ee800640 1 --2- 172.21.15.40:0/1218566481 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4d0005210 0x7fe4d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.057+0000 7fe4ed400640 1 -- 172.21.15.40:0/1218566481 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4d0005210 msgr2=0x7fe4d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:16.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.057+0000 7fe4ed400640 1 --2- 172.21.15.40:0/1218566481 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4d0005210 0x7fe4d000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.057+0000 7fe4ed400640 1 -- 172.21.15.40:0/1218566481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4d0003a00 msgr2=0x7fe4d0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:16.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.057+0000 7fe4ed400640 1 --2- 172.21.15.40:0/1218566481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4d0003a00 0x7fe4d0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.057+0000 7fe4ed400640 1 -- 172.21.15.40:0/1218566481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d0003520 con 0x7fe4d0004800 2024-04-26T23:05:16.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.057+0000 7fe4ede00640 1 --2- 172.21.15.40:0/1218566481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4d0003a00 0x7fe4d0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:05:16.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.057+0000 7fe4ee800640 1 -- 172.21.15.40:0/1218566481 reap_dead start 2024-04-26T23:05:16.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.058+0000 7fe4ed400640 1 --2- 172.21.15.40:0/1218566481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d0004800 0x7fe4d0004c80 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800ba30 tx=0x7fe4d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38260b5a32cf548e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:16.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.059+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8030070 con 0x7fe4d0004800 2024-04-26T23:05:16.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.059+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4d802ade0 con 0x7fe4d0004800 2024-04-26T23:05:16.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.059+0000 7fe4eca00640 1 -- 172.21.15.40:0/1218566481 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d800b600 con 0x7fe4d0004800 2024-04-26T23:05:16.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.059+0000 7fe4f0800640 1 -- 172.21.15.40:0/1218566481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d0004800 msgr2=0x7fe4d0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.059+0000 7fe4f0800640 1 --2- 172.21.15.40:0/1218566481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d0004800 0x7fe4d0004c80 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800ba30 tx=0x7fe4d0005830 comp rx=0 tx=0).stop 2024-04-26T23:05:16.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.060+0000 7fe4f0800640 1 -- 172.21.15.40:0/1218566481 shutdown_connections 2024-04-26T23:05:16.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.060+0000 7fe4f0800640 1 --2- 172.21.15.40:0/1218566481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d0004800 0x7fe4d0004c80 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.060+0000 7fe4f0800640 1 -- 172.21.15.40:0/1218566481 >> 172.21.15.40:0/1218566481 conn(0x7fe4e80d0be0 msgr2=0x7fe4e80d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:16.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.060+0000 7fe4f0800640 1 -- 172.21.15.40:0/1218566481 shutdown_connections 2024-04-26T23:05:16.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.060+0000 7fe4f0800640 1 -- 172.21.15.40:0/1218566481 wait complete. 2024-04-26T23:05:16.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.062+0000 7fe4f0800640 1 Processor -- start 2024-04-26T23:05:16.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.062+0000 7fe4f0800640 1 -- start start 2024-04-26T23:05:16.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.063+0000 7fe4f0800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4d0004800 0x7fe4e80c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.063+0000 7fe4f0800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4e80c5a50 0x7fe4e80b5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.063+0000 7fe4f0800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80b6040 0x7fe4e81f5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.063+0000 7fe4f0800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e8073620 con 0x7fe4e80c5a50 2024-04-26T23:05:16.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.063+0000 7fe4f0800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e80d0830 con 0x7fe4d0004800 2024-04-26T23:05:16.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.063+0000 7fe4f0800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe4e80737a0 con 0x7fe4e80b6040 2024-04-26T23:05:16.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.063+0000 7fe4ed400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4e80c5a50 0x7fe4e80b5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.063+0000 7fe4ed400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4e80c5a50 0x7fe4e80b5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44788/0 (socket says 172.21.15.40:44788) 2024-04-26T23:05:16.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.063+0000 7fe4ed400640 1 -- 172.21.15.40:0/4157570476 learned_addr learned my addr 172.21.15.40:0/4157570476 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:16.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.063+0000 7fe4ee800640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80b6040 0x7fe4e81f5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.064+0000 7fe4ed400640 1 -- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80b6040 msgr2=0x7fe4e81f5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.064+0000 7fe4ede00640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4d0004800 0x7fe4e80c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.064+0000 7fe4ed400640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80b6040 0x7fe4e81f5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.064+0000 7fe4ed400640 1 -- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4d0004800 msgr2=0x7fe4e80c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.064+0000 7fe4ed400640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4d0004800 0x7fe4e80c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.064+0000 7fe4ed400640 1 -- 172.21.15.40:0/4157570476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e81f60e0 con 0x7fe4e80c5a50 2024-04-26T23:05:16.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.064+0000 7fe4ee800640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80b6040 0x7fe4e81f5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:16.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.064+0000 7fe4ed400640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4e80c5a50 0x7fe4e80b5b00 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fe4d802a8d0 tx=0x7fe4d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:16.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.064+0000 7fe4e2000640 1 -- 172.21.15.40:0/4157570476 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8043030 con 0x7fe4e80c5a50 2024-04-26T23:05:16.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.064+0000 7fe4e2000640 1 -- 172.21.15.40:0/4157570476 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4d802abe0 con 0x7fe4e80c5a50 2024-04-26T23:05:16.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.064+0000 7fe4e2000640 1 -- 172.21.15.40:0/4157570476 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d800b600 con 0x7fe4e80c5a50 2024-04-26T23:05:16.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.064+0000 7fe4f0800640 1 -- 172.21.15.40:0/4157570476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e80c8fd0 con 0x7fe4e80c5a50 2024-04-26T23:05:16.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.064+0000 7fe4f0800640 1 -- 172.21.15.40:0/4157570476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e80c9590 con 0x7fe4e80c5a50 2024-04-26T23:05:16.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.066+0000 7fe4f0800640 1 -- 172.21.15.40:0/4157570476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4e80694f0 con 0x7fe4e80c5a50 2024-04-26T23:05:16.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.067+0000 7fe4e2000640 1 -- 172.21.15.40:0/4157570476 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe4d8046070 con 0x7fe4e80c5a50 2024-04-26T23:05:16.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.067+0000 7fe4e2000640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe4cc06c920 0x7fe4cc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.068+0000 7fe4e2000640 1 -- 172.21.15.40:0/4157570476 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7fe4d80ac7c0 con 0x7fe4e80c5a50 2024-04-26T23:05:16.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.068+0000 7fe4ede00640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe4cc06c920 0x7fe4cc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.069+0000 7fe4ede00640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe4cc06c920 0x7fe4cc06ede0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc000ff0 tx=0x7fe4dc001a00 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:16.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.072+0000 7fe4e2000640 1 -- 172.21.15.40:0/4157570476 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe4d80333b0 con 0x7fe4e80c5a50 2024-04-26T23:05:16.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.279+0000 7fe4f0800640 1 -- 172.21.15.40:0/4157570476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe4e80b6420 con 0x7fe4e80c5a50 2024-04-26T23:05:16.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.282+0000 7fe4e2000640 1 -- 172.21.15.40:0/4157570476 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+17002 (secure 0 0 0) 0x7fe4d807aa20 con 0x7fe4e80c5a50 2024-04-26T23:05:16.283 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:16.283 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":54,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:04:48.353789+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:04:14.334803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-26T23:04:15.535793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1a","mappings":[{"from":0,"to":1}]},{"pgid":"8.27","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:05:16.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.285+0000 7fe4f0800640 1 -- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe4cc06c920 msgr2=0x7fe4cc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.285+0000 7fe4f0800640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe4cc06c920 0x7fe4cc06ede0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc000ff0 tx=0x7fe4dc001a00 comp rx=0 tx=0).stop 2024-04-26T23:05:16.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.285+0000 7fe4f0800640 1 -- 172.21.15.40:0/4157570476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4e80c5a50 msgr2=0x7fe4e80b5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.285+0000 7fe4f0800640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4e80c5a50 0x7fe4e80b5b00 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fe4d802a8d0 tx=0x7fe4d80046c0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.285+0000 7fe4f0800640 1 -- 172.21.15.40:0/4157570476 shutdown_connections 2024-04-26T23:05:16.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.285+0000 7fe4f0800640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe4cc06c920 0x7fe4cc06ede0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.285+0000 7fe4f0800640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4e80b6040 0x7fe4e81f5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.285+0000 7fe4f0800640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4e80c5a50 0x7fe4e80b5b00 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.285+0000 7fe4f0800640 1 --2- 172.21.15.40:0/4157570476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4d0004800 0x7fe4e80c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.285+0000 7fe4f0800640 1 -- 172.21.15.40:0/4157570476 >> 172.21.15.40:0/4157570476 conn(0x7fe4e80d0be0 msgr2=0x7fe4e80b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:16.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.286+0000 7fe4f0800640 1 -- 172.21.15.40:0/4157570476 shutdown_connections 2024-04-26T23:05:16.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.286+0000 7fe4f0800640 1 -- 172.21.15.40:0/4157570476 wait complete. 2024-04-26T23:05:16.307 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T23:05:16.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:05:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.515+0000 7f1a69000640 1 Processor -- start 2024-04-26T23:05:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.516+0000 7f1a69000640 1 -- start start 2024-04-26T23:05:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.517+0000 7f1a69000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1a64071fe0 0x7f1a640afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.517+0000 7f1a69000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a640d0820 con 0x7f1a6415ce60 2024-04-26T23:05:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.517+0000 7f1a69000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a641614c0 con 0x7f1a640af7a0 2024-04-26T23:05:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.517+0000 7f1a69000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1a640cc450 con 0x7f1a64071fe0 2024-04-26T23:05:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.518+0000 7f1a63400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1a6415ce60 0x7f1a640ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50570/0 (socket says 172.21.15.40:50570) 2024-04-26T23:05:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.518+0000 7f1a63400640 1 -- 172.21.15.40:0/153966220 learned_addr learned my addr 172.21.15.40:0/153966220 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:16.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.518+0000 7f1a63e00640 1 --2- 172.21.15.40:0/153966220 >> v2:172.21.15.103:3300/0 conn(0x7f1a64071fe0 0x7f1a640afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.518+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1864138898 0 0) 0x7f1a640d0820 con 0x7f1a6415ce60 2024-04-26T23:05:16.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.518+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a48003520 con 0x7f1a6415ce60 2024-04-26T23:05:16.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4101900523 0 0) 0x7f1a48003520 con 0x7f1a6415ce60 2024-04-26T23:05:16.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 >> v2:172.21.15.103:3300/0 conn(0x7f1a64071fe0 msgr2=0x7f1a640afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:16.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a62a00640 1 --2- 172.21.15.40:0/153966220 >> v2:172.21.15.103:3300/0 conn(0x7f1a64071fe0 0x7f1a640afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 >> v1:172.21.15.103:6789/0 conn(0x7f1a640af7a0 legacy=0x7f1a64073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a64161760 con 0x7f1a6415ce60 2024-04-26T23:05:16.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f1a54001670 con 0x7f1a6415ce60 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 >> v1:172.21.15.40:6789/0 conn(0x7f1a6415ce60 legacy=0x7f1a640ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a62a00640 1 --2- 172.21.15.40:0/153966220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a480039b0 0x7f1a48003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a62a00640 1 --2- 172.21.15.40:0/153966220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a48004800 0x7f1a48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a62a00640 1 --2- 172.21.15.40:0/153966220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a480051c0 0x7f1a4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a640cc450 con 0x7f1a480051c0 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4800c0f0 con 0x7f1a480039b0 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1a4800c2f0 con 0x7f1a48004800 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a63e00640 1 --2- 172.21.15.40:0/153966220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a480051c0 0x7f1a4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.519+0000 7f1a63400640 1 --2- 172.21.15.40:0/153966220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a480039b0 0x7f1a48003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.520+0000 7f1a5be00640 1 --2- 172.21.15.40:0/153966220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a48004800 0x7f1a48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.520+0000 7f1a63e00640 1 -- 172.21.15.40:0/153966220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a48004800 msgr2=0x7f1a48004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.520+0000 7f1a63e00640 1 --2- 172.21.15.40:0/153966220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a48004800 0x7f1a48004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.520+0000 7f1a63e00640 1 -- 172.21.15.40:0/153966220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a480039b0 msgr2=0x7f1a48003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.520+0000 7f1a63e00640 1 --2- 172.21.15.40:0/153966220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a480039b0 0x7f1a48003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.520+0000 7f1a63e00640 1 -- 172.21.15.40:0/153966220 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a48003520 con 0x7f1a480051c0 2024-04-26T23:05:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.521+0000 7f1a63e00640 1 -- 172.21.15.40:0/153966220 reap_dead start 2024-04-26T23:05:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.521+0000 7f1a63e00640 1 --2- 172.21.15.40:0/153966220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a480051c0 0x7f1a4800b9c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009fc0 tx=0x7f1a48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92c70c0648e77fd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.521+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c035020 con 0x7f1a480051c0 2024-04-26T23:05:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.521+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5c003bd0 con 0x7f1a480051c0 2024-04-26T23:05:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.521+0000 7f1a62a00640 1 -- 172.21.15.40:0/153966220 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c009c10 con 0x7f1a480051c0 2024-04-26T23:05:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.522+0000 7f1a69000640 1 -- 172.21.15.40:0/153966220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a480051c0 msgr2=0x7f1a4800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.522+0000 7f1a69000640 1 --2- 172.21.15.40:0/153966220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a480051c0 0x7f1a4800b9c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009fc0 tx=0x7f1a48005830 comp rx=0 tx=0).stop 2024-04-26T23:05:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.522+0000 7f1a69000640 1 -- 172.21.15.40:0/153966220 shutdown_connections 2024-04-26T23:05:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.522+0000 7f1a69000640 1 --2- 172.21.15.40:0/153966220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a480051c0 0x7f1a4800b9c0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.522+0000 7f1a69000640 1 -- 172.21.15.40:0/153966220 >> 172.21.15.40:0/153966220 conn(0x7f1a640d0a10 msgr2=0x7f1a640d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.522+0000 7f1a69000640 1 -- 172.21.15.40:0/153966220 shutdown_connections 2024-04-26T23:05:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.522+0000 7f1a69000640 1 -- 172.21.15.40:0/153966220 wait complete. 2024-04-26T23:05:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.524+0000 7f1a69000640 1 Processor -- start 2024-04-26T23:05:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.524+0000 7f1a69000640 1 -- start start 2024-04-26T23:05:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.525+0000 7f1a69000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a480051c0 0x7f1a6415fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.525+0000 7f1a69000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a640c7c00 0x7f1a64141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.525+0000 7f1a69000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a641f3d50 0x7f1a641f41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.525+0000 7f1a69000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a640afde0 con 0x7f1a641f3d50 2024-04-26T23:05:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.525+0000 7f1a69000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a640afc60 con 0x7f1a480051c0 2024-04-26T23:05:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.525+0000 7f1a69000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1a640aff60 con 0x7f1a640c7c00 2024-04-26T23:05:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.525+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a641f3d50 0x7f1a641f41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.525+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a641f3d50 0x7f1a641f41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44806/0 (socket says 172.21.15.40:44806) 2024-04-26T23:05:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.525+0000 7f1a63e00640 1 -- 172.21.15.40:0/3633063148 learned_addr learned my addr 172.21.15.40:0/3633063148 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.526+0000 7f1a5be00640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a640c7c00 0x7f1a64141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.526+0000 7f1a63400640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a480051c0 0x7f1a6415fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.526+0000 7f1a63e00640 1 -- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a640c7c00 msgr2=0x7f1a64141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.526+0000 7f1a63e00640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a640c7c00 0x7f1a64141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.526+0000 7f1a63e00640 1 -- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a480051c0 msgr2=0x7f1a6415fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.526+0000 7f1a63e00640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a480051c0 0x7f1a6415fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.526+0000 7f1a63e00640 1 -- 172.21.15.40:0/3633063148 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a64156390 con 0x7f1a641f3d50 2024-04-26T23:05:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.526+0000 7f1a63e00640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a641f3d50 0x7f1a641f41d0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00e230 tx=0x7f1a5c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.526+0000 7f1a5b400640 1 -- 172.21.15.40:0/3633063148 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c035070 con 0x7f1a641f3d50 2024-04-26T23:05:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.526+0000 7f1a5b400640 1 -- 172.21.15.40:0/3633063148 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5c041070 con 0x7f1a641f3d50 2024-04-26T23:05:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.526+0000 7f1a5b400640 1 -- 172.21.15.40:0/3633063148 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c02d070 con 0x7f1a641f3d50 2024-04-26T23:05:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.526+0000 7f1a69000640 1 -- 172.21.15.40:0/3633063148 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a64156670 con 0x7f1a641f3d50 2024-04-26T23:05:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.527+0000 7f1a69000640 1 -- 172.21.15.40:0/3633063148 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a64156c30 con 0x7f1a641f3d50 2024-04-26T23:05:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.528+0000 7f1a69000640 1 -- 172.21.15.40:0/3633063148 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a64156e30 con 0x7f1a641f3d50 2024-04-26T23:05:16.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.529+0000 7f1a5b400640 1 -- 172.21.15.40:0/3633063148 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1a5c005d10 con 0x7f1a641f3d50 2024-04-26T23:05:16.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.529+0000 7f1a5b400640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a4006c8f0 0x7f1a4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.530+0000 7f1a5b400640 1 -- 172.21.15.40:0/3633063148 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f1a5c0acbe0 con 0x7f1a641f3d50 2024-04-26T23:05:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.530+0000 7f1a63400640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a4006c8f0 0x7f1a4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:16.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.531+0000 7f1a63400640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a4006c8f0 0x7f1a4006edb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1a54001050 tx=0x7f1a54001150 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:16.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.534+0000 7f1a5b400640 1 -- 172.21.15.40:0/3633063148 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1a64156e30 con 0x7f1a641f3d50 2024-04-26T23:05:16.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.791+0000 7f1a69000640 1 -- 172.21.15.40:0/3633063148 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a64069370 con 0x7f1a641f3d50 2024-04-26T23:05:16.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.792+0000 7f1a5b400640 1 -- 172.21.15.40:0/3633063148 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5225 (secure 0 0 0) 0x7f1a5c03a360 con 0x7f1a641f3d50 2024-04-26T23:05:16.793 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:16.793 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":46,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/189004193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":189004193},{"type":"v1","addr":"172.21.15.103:6835","nonce":189004193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:04:16.558591+0000","modified":"2024-04-26T23:04:55.194872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6036,"mds_1":6027,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"b","rank":2,"incarnation":39,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/670995452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":670995452},{"type":"v1","addr":"172.21.15.103:6837","nonce":670995452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6027":{"gid":6027,"name":"a","rank":1,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6834/2304583125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2304583125},{"type":"v1","addr":"172.21.15.40:6834","nonce":2304583125}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6036":{"gid":6036,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/618018110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":618018110},{"type":"v1","addr":"172.21.15.40:6835","nonce":618018110}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6036,"qdb_cluster":[5284,6027,6036]},"id":3}]} 2024-04-26T23:05:16.793 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 46 2024-04-26T23:05:16.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.795+0000 7f1a69000640 1 -- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a4006c8f0 msgr2=0x7f1a4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.795+0000 7f1a69000640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a4006c8f0 0x7f1a4006edb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1a54001050 tx=0x7f1a54001150 comp rx=0 tx=0).stop 2024-04-26T23:05:16.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.795+0000 7f1a69000640 1 -- 172.21.15.40:0/3633063148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a641f3d50 msgr2=0x7f1a641f41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:16.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.795+0000 7f1a69000640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a641f3d50 0x7f1a641f41d0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00e230 tx=0x7f1a5c009750 comp rx=0 tx=0).stop 2024-04-26T23:05:16.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.795+0000 7f1a69000640 1 -- 172.21.15.40:0/3633063148 shutdown_connections 2024-04-26T23:05:16.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.795+0000 7f1a69000640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a4006c8f0 0x7f1a4006edb0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.795+0000 7f1a69000640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a641f3d50 0x7f1a641f41d0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.795+0000 7f1a69000640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a640c7c00 0x7f1a64141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.795+0000 7f1a69000640 1 --2- 172.21.15.40:0/3633063148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a480051c0 0x7f1a6415fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:16.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.795+0000 7f1a69000640 1 -- 172.21.15.40:0/3633063148 >> 172.21.15.40:0/3633063148 conn(0x7f1a640d0a10 msgr2=0x7f1a640b06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:16.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.795+0000 7f1a69000640 1 -- 172.21.15.40:0/3633063148 shutdown_connections 2024-04-26T23:05:16.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:16.796+0000 7f1a69000640 1 -- 172.21.15.40:0/3633063148 wait complete. 2024-04-26T23:05:16.816 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:05:17.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.029+0000 7f03d9a00640 1 Processor -- start 2024-04-26T23:05:17.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.030+0000 7f03d9a00640 1 -- start start 2024-04-26T23:05:17.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.031+0000 7f03d9a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03d4071e60 0x7f03d406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.031+0000 7f03d9a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f03d406fe20 0x7f03d4072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.031+0000 7f03d9a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03d415d7c0 con 0x7f03d4072780 2024-04-26T23:05:17.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.031+0000 7f03d9a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f03d415daa0 con 0x7f03d4071e60 2024-04-26T23:05:17.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.031+0000 7f03d9a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f03d415dd80 con 0x7f03d406fe20 2024-04-26T23:05:17.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.032+0000 7f03d2a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03d4071e60 0x7f03d406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.032+0000 7f03d2a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03d4071e60 0x7f03d406f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44810/0 (socket says 172.21.15.40:44810) 2024-04-26T23:05:17.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.032+0000 7f03d3400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f03d4072780 0x7f03d40cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50572/0 (socket says 172.21.15.40:50572) 2024-04-26T23:05:17.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.032+0000 7f03d2a00640 1 -- 172.21.15.40:0/1593904157 learned_addr learned my addr 172.21.15.40:0/1593904157 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:17.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.032+0000 7f03d2000640 1 --2- 172.21.15.40:0/1593904157 >> v2:172.21.15.103:3300/0 conn(0x7f03d406fe20 0x7f03d4072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.032+0000 7f03d2a00640 1 -- 172.21.15.40:0/1593904157 >> v2:172.21.15.103:3300/0 conn(0x7f03d406fe20 msgr2=0x7f03d4072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:17.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.032+0000 7f03d2a00640 1 --2- 172.21.15.40:0/1593904157 >> v2:172.21.15.103:3300/0 conn(0x7f03d406fe20 0x7f03d4072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.032+0000 7f03d2a00640 1 -- 172.21.15.40:0/1593904157 >> v1:172.21.15.40:6789/0 conn(0x7f03d4072780 legacy=0x7f03d40cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.032+0000 7f03d2a00640 1 -- 172.21.15.40:0/1593904157 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d415e060 con 0x7f03d4071e60 2024-04-26T23:05:17.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.033+0000 7f03d2a00640 1 --2- 172.21.15.40:0/1593904157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d4071e60 0x7f03d406f8e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f03d40b3820 tx=0x7f03c402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eabcd2ee7a88c75c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:17.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.034+0000 7f03d1600640 1 -- 172.21.15.40:0/1593904157 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c4036070 con 0x7f03d4071e60 2024-04-26T23:05:17.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.034+0000 7f03d1600640 1 -- 172.21.15.40:0/1593904157 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03c4031600 con 0x7f03d4071e60 2024-04-26T23:05:17.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.034+0000 7f03d1600640 1 -- 172.21.15.40:0/1593904157 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c40319d0 con 0x7f03d4071e60 2024-04-26T23:05:17.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.035+0000 7f03d9a00640 1 -- 172.21.15.40:0/1593904157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d4071e60 msgr2=0x7f03d406f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.035+0000 7f03d9a00640 1 --2- 172.21.15.40:0/1593904157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d4071e60 0x7f03d406f8e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f03d40b3820 tx=0x7f03c402a730 comp rx=0 tx=0).stop 2024-04-26T23:05:17.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.035+0000 7f03d9a00640 1 -- 172.21.15.40:0/1593904157 shutdown_connections 2024-04-26T23:05:17.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.035+0000 7f03d9a00640 1 --2- 172.21.15.40:0/1593904157 >> v2:172.21.15.103:3300/0 conn(0x7f03d406fe20 0x7f03d4072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.035+0000 7f03d9a00640 1 --2- 172.21.15.40:0/1593904157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d4071e60 0x7f03d406f8e0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.035+0000 7f03d9a00640 1 -- 172.21.15.40:0/1593904157 >> 172.21.15.40:0/1593904157 conn(0x7f03d40d0be0 msgr2=0x7f03d40d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:17.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.035+0000 7f03d9a00640 1 -- 172.21.15.40:0/1593904157 shutdown_connections 2024-04-26T23:05:17.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.035+0000 7f03d9a00640 1 -- 172.21.15.40:0/1593904157 wait complete. 2024-04-26T23:05:17.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.037+0000 7f03d9a00640 1 Processor -- start 2024-04-26T23:05:17.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.037+0000 7f03d9a00640 1 -- start start 2024-04-26T23:05:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.038+0000 7f03d9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d406fe20 0x7f03d4146970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.038+0000 7f03d9a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d4071e60 0x7f03d40b5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.038+0000 7f03d9a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03d4072780 0x7f03d40c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.038+0000 7f03d9a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f03d415d3f0 con 0x7f03d406fe20 2024-04-26T23:05:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.038+0000 7f03d9a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f03d4072bd0 con 0x7f03d4071e60 2024-04-26T23:05:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.038+0000 7f03d9a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f03d415d570 con 0x7f03d4072780 2024-04-26T23:05:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.038+0000 7f03d2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d406fe20 0x7f03d4146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.038+0000 7f03d2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d406fe20 0x7f03d4146970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44814/0 (socket says 172.21.15.40:44814) 2024-04-26T23:05:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.038+0000 7f03d2a00640 1 -- 172.21.15.40:0/3407578050 learned_addr learned my addr 172.21.15.40:0/3407578050 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.039+0000 7f03d3400640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03d4072780 0x7f03d40c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.039+0000 7f03d2a00640 1 -- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03d4072780 msgr2=0x7f03d40c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.039+0000 7f03d2000640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d4071e60 0x7f03d40b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.039+0000 7f03d2a00640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03d4072780 0x7f03d40c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.039+0000 7f03d2a00640 1 -- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d4071e60 msgr2=0x7f03d40b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.039+0000 7f03d2a00640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d4071e60 0x7f03d40b5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.039+0000 7f03d2a00640 1 -- 172.21.15.40:0/3407578050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d40b6890 con 0x7f03d406fe20 2024-04-26T23:05:17.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.039+0000 7f03d3400640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03d4072780 0x7f03d40c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:17.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.039+0000 7f03d2a00640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d406fe20 0x7f03d4146970 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f03c4034040 tx=0x7f03c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:17.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.040+0000 7f03caa00640 1 -- 172.21.15.40:0/3407578050 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c4036050 con 0x7f03d406fe20 2024-04-26T23:05:17.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.040+0000 7f03caa00640 1 -- 172.21.15.40:0/3407578050 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03c4042070 con 0x7f03d406fe20 2024-04-26T23:05:17.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.040+0000 7f03caa00640 1 -- 172.21.15.40:0/3407578050 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c4009510 con 0x7f03d406fe20 2024-04-26T23:05:17.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.040+0000 7f03d9a00640 1 -- 172.21.15.40:0/3407578050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03d40b5d10 con 0x7f03d406fe20 2024-04-26T23:05:17.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.040+0000 7f03d9a00640 1 -- 172.21.15.40:0/3407578050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03d40b62d0 con 0x7f03d406fe20 2024-04-26T23:05:17.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.041+0000 7f03d9a00640 1 -- 172.21.15.40:0/3407578050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03d40694f0 con 0x7f03d406fe20 2024-04-26T23:05:17.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.042+0000 7f03caa00640 1 -- 172.21.15.40:0/3407578050 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f03c4007c50 con 0x7f03d406fe20 2024-04-26T23:05:17.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.043+0000 7f03caa00640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f03b006c8f0 0x7f03b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.043+0000 7f03caa00640 1 -- 172.21.15.40:0/3407578050 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f03c40acaf0 con 0x7f03d406fe20 2024-04-26T23:05:17.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.044+0000 7f03d2000640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f03b006c8f0 0x7f03b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.045+0000 7f03d2000640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f03b006c8f0 0x7f03b006edb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f03d4143e30 tx=0x7f03bc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:17.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.048+0000 7f03caa00640 1 -- 172.21.15.40:0/3407578050 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f03c404c020 con 0x7f03d406fe20 2024-04-26T23:05:17.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.291+0000 7f03d9a00640 1 -- 172.21.15.40:0/3407578050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03d40cc370 con 0x7f03d406fe20 2024-04-26T23:05:17.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.292+0000 7f03caa00640 1 -- 172.21.15.40:0/3407578050 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5225 (secure 0 0 0) 0x7f03c407acc0 con 0x7f03d406fe20 2024-04-26T23:05:17.292 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:17.293 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":46,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/189004193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":189004193},{"type":"v1","addr":"172.21.15.103:6835","nonce":189004193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:04:16.558591+0000","modified":"2024-04-26T23:04:55.194872+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6036,"mds_1":6027,"mds_2":5284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"b","rank":2,"incarnation":39,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/670995452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":670995452},{"type":"v1","addr":"172.21.15.103:6837","nonce":670995452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6027":{"gid":6027,"name":"a","rank":1,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6834/2304583125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2304583125},{"type":"v1","addr":"172.21.15.40:6834","nonce":2304583125}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6036":{"gid":6036,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/618018110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":618018110},{"type":"v1","addr":"172.21.15.40:6835","nonce":618018110}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6036,"qdb_cluster":[5284,6027,6036]},"id":3}]} 2024-04-26T23:05:17.293 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 46 2024-04-26T23:05:17.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.295+0000 7f03d9a00640 1 -- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f03b006c8f0 msgr2=0x7f03b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.295+0000 7f03d9a00640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f03b006c8f0 0x7f03b006edb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f03d4143e30 tx=0x7f03bc008040 comp rx=0 tx=0).stop 2024-04-26T23:05:17.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.295+0000 7f03d9a00640 1 -- 172.21.15.40:0/3407578050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d406fe20 msgr2=0x7f03d4146970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.295+0000 7f03d9a00640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d406fe20 0x7f03d4146970 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f03c4034040 tx=0x7f03c4004a40 comp rx=0 tx=0).stop 2024-04-26T23:05:17.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.295+0000 7f03d9a00640 1 -- 172.21.15.40:0/3407578050 shutdown_connections 2024-04-26T23:05:17.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.295+0000 7f03d9a00640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f03b006c8f0 0x7f03b006edb0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.295+0000 7f03d9a00640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03d4072780 0x7f03d40c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.295+0000 7f03d9a00640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d4071e60 0x7f03d40b5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.295+0000 7f03d9a00640 1 --2- 172.21.15.40:0/3407578050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03d406fe20 0x7f03d4146970 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.295+0000 7f03d9a00640 1 -- 172.21.15.40:0/3407578050 >> 172.21.15.40:0/3407578050 conn(0x7f03d40d0be0 msgr2=0x7f03d40cb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:17.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.295+0000 7f03d9a00640 1 -- 172.21.15.40:0/3407578050 shutdown_connections 2024-04-26T23:05:17.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.295+0000 7f03d9a00640 1 -- 172.21.15.40:0/3407578050 wait complete. 2024-04-26T23:05:17.314 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:05:17.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.527+0000 7f2dccc00640 1 Processor -- start 2024-04-26T23:05:17.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.528+0000 7f2dccc00640 1 -- start start 2024-04-26T23:05:17.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.529+0000 7f2dccc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dc815bde0 0x7f2dc80d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.529+0000 7f2dccc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dc8072f70 0x7f2dc80733e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.529+0000 7f2dccc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dc815c1c0 con 0x7f2dc80cd570 2024-04-26T23:05:17.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.529+0000 7f2dccc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f2dc8073920 con 0x7f2dc815bde0 2024-04-26T23:05:17.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.529+0000 7f2dccc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2dc815c7d0 con 0x7f2dc8072f70 2024-04-26T23:05:17.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.530+0000 7f2dc5e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dc815bde0 0x7f2dc80d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.530+0000 7f2dc5e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dc815bde0 0x7f2dc80d02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44824/0 (socket says 172.21.15.40:44824) 2024-04-26T23:05:17.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.530+0000 7f2dc5e00640 1 -- 172.21.15.40:0/1907603583 learned_addr learned my addr 172.21.15.40:0/1907603583 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:17.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.530+0000 7f2dc6800640 1 --2- 172.21.15.40:0/1907603583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dc8072f70 0x7f2dc80733e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.530+0000 7f2dc5e00640 1 -- 172.21.15.40:0/1907603583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dc8072f70 msgr2=0x7f2dc80733e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:17.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.530+0000 7f2dc5e00640 1 --2- 172.21.15.40:0/1907603583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dc8072f70 0x7f2dc80733e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.530+0000 7f2dc5e00640 1 -- 172.21.15.40:0/1907603583 >> v1:172.21.15.103:6789/0 conn(0x7f2dc80cd570 legacy=0x7f2dc815cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.530+0000 7f2dc5e00640 1 -- 172.21.15.40:0/1907603583 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc80cdb30 con 0x7f2dc815bde0 2024-04-26T23:05:17.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.532+0000 7f2dc5e00640 1 --2- 172.21.15.40:0/1907603583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dc815bde0 0x7f2dc80d02d0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009870 tx=0x7f2db802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb2769edc4aff4f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:17.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.532+0000 7f2dc5400640 1 -- 172.21.15.40:0/1907603583 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db8037070 con 0x7f2dc815bde0 2024-04-26T23:05:17.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.532+0000 7f2dc5400640 1 -- 172.21.15.40:0/1907603583 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2db8032440 con 0x7f2dc815bde0 2024-04-26T23:05:17.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.532+0000 7f2dc5400640 1 -- 172.21.15.40:0/1907603583 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db8032810 con 0x7f2dc815bde0 2024-04-26T23:05:17.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.533+0000 7f2dccc00640 1 -- 172.21.15.40:0/1907603583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dc815bde0 msgr2=0x7f2dc80d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.533+0000 7f2dccc00640 1 --2- 172.21.15.40:0/1907603583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dc815bde0 0x7f2dc80d02d0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009870 tx=0x7f2db802a1c0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.533+0000 7f2dccc00640 1 -- 172.21.15.40:0/1907603583 shutdown_connections 2024-04-26T23:05:17.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.533+0000 7f2dccc00640 1 --2- 172.21.15.40:0/1907603583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dc8072f70 0x7f2dc80733e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.533+0000 7f2dccc00640 1 --2- 172.21.15.40:0/1907603583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dc815bde0 0x7f2dc80d02d0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.533+0000 7f2dccc00640 1 -- 172.21.15.40:0/1907603583 >> 172.21.15.40:0/1907603583 conn(0x7f2dc80d1cf0 msgr2=0x7f2dc80d20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:17.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.533+0000 7f2dccc00640 1 -- 172.21.15.40:0/1907603583 shutdown_connections 2024-04-26T23:05:17.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.533+0000 7f2dccc00640 1 -- 172.21.15.40:0/1907603583 wait complete. 2024-04-26T23:05:17.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.535+0000 7f2dccc00640 1 Processor -- start 2024-04-26T23:05:17.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.535+0000 7f2dccc00640 1 -- start start 2024-04-26T23:05:17.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.536+0000 7f2dccc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dc8072f70 0x7f2dc81e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.536+0000 7f2dccc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dc80cd570 0x7f2dc80c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.536+0000 7f2dccc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dc815bde0 0x7f2dc815add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.536+0000 7f2dccc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc80ce0f0 con 0x7f2dc80cd570 2024-04-26T23:05:17.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.536+0000 7f2dccc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc815c460 con 0x7f2dc815bde0 2024-04-26T23:05:17.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.536+0000 7f2dccc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2dc80ce270 con 0x7f2dc8072f70 2024-04-26T23:05:17.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.536+0000 7f2dbfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dc80cd570 0x7f2dc80c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.536+0000 7f2dbfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dc80cd570 0x7f2dc80c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44826/0 (socket says 172.21.15.40:44826) 2024-04-26T23:05:17.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.536+0000 7f2dbfe00640 1 -- 172.21.15.40:0/2431007424 learned_addr learned my addr 172.21.15.40:0/2431007424 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:17.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.537+0000 7f2dc5e00640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dc8072f70 0x7f2dc81e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.537+0000 7f2dc6800640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dc815bde0 0x7f2dc815add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.537+0000 7f2dbfe00640 1 -- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dc8072f70 msgr2=0x7f2dc81e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.537+0000 7f2dbfe00640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dc8072f70 0x7f2dc81e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.537+0000 7f2dbfe00640 1 -- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dc815bde0 msgr2=0x7f2dc815add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.537+0000 7f2dbfe00640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dc815bde0 0x7f2dc815add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.537+0000 7f2dbfe00640 1 -- 172.21.15.40:0/2431007424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc80c9520 con 0x7f2dc80cd570 2024-04-26T23:05:17.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.537+0000 7f2dbfe00640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dc80cd570 0x7f2dc80c8dc0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f2db000c400 tx=0x7f2db000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:17.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.537+0000 7f2dbe000640 1 -- 172.21.15.40:0/2431007424 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db00052d0 con 0x7f2dc80cd570 2024-04-26T23:05:17.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.537+0000 7f2dbe000640 1 -- 172.21.15.40:0/2431007424 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2db0005490 con 0x7f2dc80cd570 2024-04-26T23:05:17.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.537+0000 7f2dbe000640 1 -- 172.21.15.40:0/2431007424 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db0007260 con 0x7f2dc80cd570 2024-04-26T23:05:17.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.538+0000 7f2dccc00640 1 -- 172.21.15.40:0/2431007424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dc80c9800 con 0x7f2dc80cd570 2024-04-26T23:05:17.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.538+0000 7f2dccc00640 1 -- 172.21.15.40:0/2431007424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dc80c9dc0 con 0x7f2dc80cd570 2024-04-26T23:05:17.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.539+0000 7f2dccc00640 1 -- 172.21.15.40:0/2431007424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dc80694f0 con 0x7f2dc80cd570 2024-04-26T23:05:17.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.540+0000 7f2dbe000640 1 -- 172.21.15.40:0/2431007424 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2db0007400 con 0x7f2dc80cd570 2024-04-26T23:05:17.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.541+0000 7f2dbe000640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2da406c8f0 0x7f2da406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.541+0000 7f2dbe000640 1 -- 172.21.15.40:0/2431007424 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f2db008b2b0 con 0x7f2dc80cd570 2024-04-26T23:05:17.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.541+0000 7f2dc5e00640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2da406c8f0 0x7f2da406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.543+0000 7f2dc5e00640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2da406c8f0 0x7f2da406edb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2db8034000 tx=0x7f2db802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:17.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.545+0000 7f2dbe000640 1 -- 172.21.15.40:0/2431007424 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2db0059510 con 0x7f2dc80cd570 2024-04-26T23:05:17.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.752+0000 7f2dccc00640 1 -- 172.21.15.40:0/2431007424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2dc80cd350 con 0x7f2dc80cd570 2024-04-26T23:05:17.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.755+0000 7f2dbe000640 1 -- 172.21.15.40:0/2431007424 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+17002 (secure 0 0 0) 0x7f2db0058e10 con 0x7f2dc80cd570 2024-04-26T23:05:17.756 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:17.756 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":54,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:04:48.353789+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:04:14.334803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-26T23:04:15.535793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1a","mappings":[{"from":0,"to":1}]},{"pgid":"8.27","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:05:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.758+0000 7f2dccc00640 1 -- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2da406c8f0 msgr2=0x7f2da406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.758+0000 7f2dccc00640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2da406c8f0 0x7f2da406edb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2db8034000 tx=0x7f2db802e000 comp rx=0 tx=0).stop 2024-04-26T23:05:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.758+0000 7f2dccc00640 1 -- 172.21.15.40:0/2431007424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dc80cd570 msgr2=0x7f2dc80c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.758+0000 7f2dccc00640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dc80cd570 0x7f2dc80c8dc0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f2db000c400 tx=0x7f2db000c8c0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.758+0000 7f2dccc00640 1 -- 172.21.15.40:0/2431007424 shutdown_connections 2024-04-26T23:05:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.758+0000 7f2dccc00640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2da406c8f0 0x7f2da406edb0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.758+0000 7f2dccc00640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dc815bde0 0x7f2dc815add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.758+0000 7f2dccc00640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dc80cd570 0x7f2dc80c8dc0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.758+0000 7f2dccc00640 1 --2- 172.21.15.40:0/2431007424 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2dc8072f70 0x7f2dc81e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.758+0000 7f2dccc00640 1 -- 172.21.15.40:0/2431007424 >> 172.21.15.40:0/2431007424 conn(0x7f2dc80d1cf0 msgr2=0x7f2dc815e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:17.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.758+0000 7f2dccc00640 1 -- 172.21.15.40:0/2431007424 shutdown_connections 2024-04-26T23:05:17.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.758+0000 7f2dccc00640 1 -- 172.21.15.40:0/2431007424 wait complete. 2024-04-26T23:05:17.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-26T23:05:17.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.983+0000 7f21b3e00640 1 Processor -- start 2024-04-26T23:05:17.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.983+0000 7f21b3e00640 1 -- start start 2024-04-26T23:05:17.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.984+0000 7f21b3e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f21b40d32e0 0x7f21b40d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.984+0000 7f21b3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b40b8d20 0x7f21b40b91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.984+0000 7f21b3e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21b4162c40 con 0x7f21b40b6220 2024-04-26T23:05:17.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.984+0000 7f21b3e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f21b40d3c80 con 0x7f21b40d32e0 2024-04-26T23:05:17.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.984+0000 7f21b3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f21b40d2d80 con 0x7f21b40b8d20 2024-04-26T23:05:17.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.984+0000 7f21b2a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f21b40b6220 0x7f21b40d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50580/0 (socket says 172.21.15.40:50580) 2024-04-26T23:05:17.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.984+0000 7f21b2a00640 1 -- 172.21.15.40:0/1436945999 learned_addr learned my addr 172.21.15.40:0/1436945999 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.985+0000 7f21b3400640 1 --2- 172.21.15.40:0/1436945999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b40b8d20 0x7f21b40b91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.985+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 564704012 0 0) 0x7f21b4162c40 con 0x7f21b40b6220 2024-04-26T23:05:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.985+0000 7f21b2000640 1 --2- 172.21.15.40:0/1436945999 >> v2:172.21.15.103:3300/0 conn(0x7f21b40d32e0 0x7f21b40d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.985+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2198003520 con 0x7f21b40b6220 2024-04-26T23:05:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.985+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1634913545 0 0) 0x7f2198003520 con 0x7f21b40b6220 2024-04-26T23:05:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.985+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b40b8d20 msgr2=0x7f21b40b91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.985+0000 7f21b1600640 1 --2- 172.21.15.40:0/1436945999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b40b8d20 0x7f21b40b91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.985+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 >> v2:172.21.15.103:3300/0 conn(0x7f21b40d32e0 msgr2=0x7f21b40d3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.985+0000 7f21b1600640 1 --2- 172.21.15.40:0/1436945999 >> v2:172.21.15.103:3300/0 conn(0x7f21b40d32e0 0x7f21b40d3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.985+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b40b7c80 con 0x7f21b40b6220 2024-04-26T23:05:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.985+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f21a4001a10 con 0x7f21b40b6220 2024-04-26T23:05:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.985+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 >> v1:172.21.15.40:6789/0 conn(0x7f21b40b6220 legacy=0x7f21b40d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b1600640 1 --2- 172.21.15.40:0/1436945999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21980039b0 0x7f2198003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b1600640 1 --2- 172.21.15.40:0/1436945999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2198004800 0x7f2198004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b1600640 1 --2- 172.21.15.40:0/1436945999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b40b8d20 0x7f21980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21b40d2d80 con 0x7f21980039b0 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f219800b9c0 con 0x7f2198004800 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f219800bb40 con 0x7f21b40b8d20 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b2a00640 1 --2- 172.21.15.40:0/1436945999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21980039b0 0x7f2198003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b2000640 1 --2- 172.21.15.40:0/1436945999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2198004800 0x7f2198004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b3400640 1 --2- 172.21.15.40:0/1436945999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b40b8d20 0x7f21980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b2a00640 1 -- 172.21.15.40:0/1436945999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b40b8d20 msgr2=0x7f21980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b2a00640 1 --2- 172.21.15.40:0/1436945999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b40b8d20 0x7f21980051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b2a00640 1 -- 172.21.15.40:0/1436945999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2198004800 msgr2=0x7f2198004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b2a00640 1 --2- 172.21.15.40:0/1436945999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2198004800 0x7f2198004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b2a00640 1 -- 172.21.15.40:0/1436945999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2198003520 con 0x7f21980039b0 2024-04-26T23:05:17.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.986+0000 7f21b2000640 1 --2- 172.21.15.40:0/1436945999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2198004800 0x7f2198004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:17.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.987+0000 7f21b2a00640 1 --2- 172.21.15.40:0/1436945999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21980039b0 0x7f2198003dd0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f21b40d5e00 tx=0x7f2198005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d76b3d482567b45 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.988+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a400a410 con 0x7f21980039b0 2024-04-26T23:05:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.988+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a402f070 con 0x7f21980039b0 2024-04-26T23:05:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.988+0000 7f21b1600640 1 -- 172.21.15.40:0/1436945999 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a402bbd0 con 0x7f21980039b0 2024-04-26T23:05:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.988+0000 7f21b3e00640 1 -- 172.21.15.40:0/1436945999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21980039b0 msgr2=0x7f2198003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.988+0000 7f21b3e00640 1 --2- 172.21.15.40:0/1436945999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21980039b0 0x7f2198003dd0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f21b40d5e00 tx=0x7f2198005830 comp rx=0 tx=0).stop 2024-04-26T23:05:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.988+0000 7f21b3400640 1 -- 172.21.15.40:0/1436945999 reap_dead start 2024-04-26T23:05:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.988+0000 7f21b3e00640 1 -- 172.21.15.40:0/1436945999 shutdown_connections 2024-04-26T23:05:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.988+0000 7f21b3e00640 1 -- 172.21.15.40:0/1436945999 >> 172.21.15.40:0/1436945999 conn(0x7f21b40d4d90 msgr2=0x7f21b40d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.989+0000 7f21b3e00640 1 -- 172.21.15.40:0/1436945999 shutdown_connections 2024-04-26T23:05:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.989+0000 7f21b3e00640 1 -- 172.21.15.40:0/1436945999 wait complete. 2024-04-26T23:05:17.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.990+0000 7f21b3e00640 1 Processor -- start 2024-04-26T23:05:17.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.990+0000 7f21b3e00640 1 -- start start 2024-04-26T23:05:17.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.991+0000 7f21b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21b414f160 0x7f21b40b9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.991+0000 7f21b3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b40b9670 0x7f21b415ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.991+0000 7f21b3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b415d380 0x7f21b40cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.991+0000 7f21b3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21b40b8520 con 0x7f21b414f160 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.991+0000 7f21b3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f21b40b8240 con 0x7f21b415d380 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.991+0000 7f21b3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f21b40d0aa0 con 0x7f21b40b9670 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.992+0000 7f21b2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21b414f160 0x7f21b40b9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.992+0000 7f21b2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21b414f160 0x7f21b40b9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44852/0 (socket says 172.21.15.40:44852) 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.992+0000 7f21b2a00640 1 -- 172.21.15.40:0/2233486621 learned_addr learned my addr 172.21.15.40:0/2233486621 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.992+0000 7f21b2000640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b40b9670 0x7f21b415ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.992+0000 7f21b3400640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b415d380 0x7f21b40cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.992+0000 7f21b2a00640 1 -- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b40b9670 msgr2=0x7f21b415ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.992+0000 7f21b2a00640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b40b9670 0x7f21b415ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.992+0000 7f21b2a00640 1 -- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b415d380 msgr2=0x7f21b40cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.992+0000 7f21b2a00640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b415d380 0x7f21b40cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.992+0000 7f21b2a00640 1 -- 172.21.15.40:0/2233486621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b41f81c0 con 0x7f21b414f160 2024-04-26T23:05:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.992+0000 7f21b3400640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b415d380 0x7f21b40cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:17.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.993+0000 7f21b2a00640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21b414f160 0x7f21b40b9130 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f21a402b440 tx=0x7f21b40b8800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:17.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.993+0000 7f21aaa00640 1 -- 172.21.15.40:0/2233486621 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a4001a20 con 0x7f21b414f160 2024-04-26T23:05:17.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.993+0000 7f21aaa00640 1 -- 172.21.15.40:0/2233486621 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a402bd50 con 0x7f21b414f160 2024-04-26T23:05:17.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.993+0000 7f21b3e00640 1 -- 172.21.15.40:0/2233486621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b41f84a0 con 0x7f21b414f160 2024-04-26T23:05:17.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.993+0000 7f21aaa00640 1 -- 172.21.15.40:0/2233486621 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a40346b0 con 0x7f21b414f160 2024-04-26T23:05:17.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.993+0000 7f21b3e00640 1 -- 172.21.15.40:0/2233486621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b41f8a60 con 0x7f21b414f160 2024-04-26T23:05:17.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.994+0000 7f21b3e00640 1 -- 172.21.15.40:0/2233486621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21b4069370 con 0x7f21b414f160 2024-04-26T23:05:17.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.995+0000 7f21aaa00640 1 -- 172.21.15.40:0/2233486621 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f21a4034850 con 0x7f21b414f160 2024-04-26T23:05:17.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.996+0000 7f21aaa00640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f219406c920 0x7f219406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:17.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.996+0000 7f21aaa00640 1 -- 172.21.15.40:0/2233486621 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6159+0+0 (secure 0 0 0) 0x7f21a40abfb0 con 0x7f21b414f160 2024-04-26T23:05:17.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.997+0000 7f21b2000640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f219406c920 0x7f219406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:17.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:17.998+0000 7f21b2000640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f219406c920 0x7f219406ede0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f21b406ce00 tx=0x7f21a0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:18.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.001+0000 7f21aaa00640 1 -- 172.21.15.40:0/2233486621 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f21a407a180 con 0x7f21b414f160 2024-04-26T23:05:18.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:05:18.083 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:05:18.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.252+0000 7f21b3e00640 1 -- 172.21.15.40:0/2233486621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f21b40d30c0 con 0x7f21b414f160 2024-04-26T23:05:18.253 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:05:18.253+0000 7f36fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T23:05:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.257+0000 7f21aaa00640 1 -- 172.21.15.40:0/2233486621 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v47) ==== 161+0+0 (secure 0 0 0) 0x7f21a400a4b0 con 0x7f21b414f160 2024-04-26T23:05:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T23:05:18.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.260+0000 7f21b3e00640 1 -- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f219406c920 msgr2=0x7f219406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:18.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.260+0000 7f21b3e00640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f219406c920 0x7f219406ede0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f21b406ce00 tx=0x7f21a0008040 comp rx=0 tx=0).stop 2024-04-26T23:05:18.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.260+0000 7f21b3e00640 1 -- 172.21.15.40:0/2233486621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21b414f160 msgr2=0x7f21b40b9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:18.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.260+0000 7f21b3e00640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21b414f160 0x7f21b40b9130 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f21a402b440 tx=0x7f21b40b8800 comp rx=0 tx=0).stop 2024-04-26T23:05:18.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.261+0000 7f21b3e00640 1 -- 172.21.15.40:0/2233486621 shutdown_connections 2024-04-26T23:05:18.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.261+0000 7f21b3e00640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f219406c920 0x7f219406ede0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:18.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.261+0000 7f21b3e00640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b415d380 0x7f21b40cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:18.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.261+0000 7f21b3e00640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b40b9670 0x7f21b415ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:18.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.261+0000 7f21b3e00640 1 --2- 172.21.15.40:0/2233486621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21b414f160 0x7f21b40b9130 secure :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f21a402b440 tx=0x7f21b40b8800 comp rx=0 tx=0).stop 2024-04-26T23:05:18.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.261+0000 7f21b3e00640 1 -- 172.21.15.40:0/2233486621 >> 172.21.15.40:0/2233486621 conn(0x7f21b40d4d90 msgr2=0x7f21b40d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:18.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.261+0000 7f21b3e00640 1 -- 172.21.15.40:0/2233486621 shutdown_connections 2024-04-26T23:05:18.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.261+0000 7f21b3e00640 1 -- 172.21.15.40:0/2233486621 wait complete. 2024-04-26T23:05:18.285 INFO:tasks.ceph.mds.a.smithi040.stderr: -1589> 2024-04-26T23:04:47.891+0000 7ff189400640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000004 [...2,head] /d1/d2/d3/d4/d5/ auth{0=1,2=1} v15 ap=1 f(v0 m2024-04-26T23:04:30.271604+0000 1=0+1) n(v2 rc2024-04-26T23:04:30.494601+0000 4=0+4) (inest lock) (ifile excl) caps={5356=pAsLsXsFsx/pAsLsXsFsxcral@4},l=5356 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 authpin=1 scrubqueue=0 0x561a6cc18b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000004:[<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:04:30.271604+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:04:30.494601+0000 4=0+4)","memory_value.dirstat":"f(v0 m2024-04-26T23:04:30.271604+0000 1=0+1)","memory_value.rstat":"n(v2 rc2024-04-26T23:04:30.494601+0000 4=0+4)","error_str":""},"return_code":-61} 2024-04-26T23:05:18.286 INFO:tasks.ceph.mds.a.smithi040.stderr: -1440> 2024-04-26T23:04:47.894+0000 7ff191600640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000005 [...2,head] /d1/d2/d3/d4/d5/d6/ auth{0=1,2=1} v11 ap=2 f(v0 m2024-04-26T23:04:30.272604+0000 1=0+1) n(v1 rc2024-04-26T23:04:30.494601+0000 3=0+3) (inest sync->mix g=0,2 scatter_wanted) caps={5356=pAsLsXsFs/pAsLsXsFsxcral@7} | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 export_pin=2 0x561a6cc18000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000005:[<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:04:30.272604+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:04:30.274604+0000 3=0+3)","memory_value.dirstat":"f(v0 m2024-04-26T23:04:30.272604+0000 1=0+1)","memory_value.rstat":"n(v1 rc2024-04-26T23:04:30.494601+0000 3=0+3)","error_str":""},"return_code":-61} 2024-04-26T23:05:18.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T23:05:18.300 INFO:tasks.ceph.mds.b.smithi103.stderr: -1369> 2024-04-26T23:04:47.892+0000 7fec41e00640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000007 [...2,head] /d1/d2/d3/d4/d5/d6/d7/d8/ auth v2 ap=1 f() n(v0 1=0+1) (inest lock) (ifile excl) caps={5356=pAsLsXsFsx/AsLsXsFsx@2},l=5356 | importingcaps=0 dirfrag=0 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x55b6f108e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000007:[<0x10000000006/d8 v2>,<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 1=0+1)","error_str":""},"return_code":-61} 2024-04-26T23:05:18.300 INFO:tasks.ceph.mds.b.smithi103.stderr: -1355> 2024-04-26T23:04:47.893+0000 7fec41e00640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000006 [...2,head] /d1/d2/d3/d4/d5/d6/d7/ auth v4 ap=1 f(v0 m2024-04-26T23:04:30.274604+0000 1=0+1) n(v0 rc2024-04-26T23:04:30.274604+0000 2=0+2) (inest lock) (ifile excl) caps={5356=pAsLsXsFsx/pAsLsXsFsxcral@4},l=5356 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x55b6f108e000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000006:[<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:04:30.274604+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:04:30.274604+0000 2=0+2)","memory_value.dirstat":"f(v0 m2024-04-26T23:04:30.274604+0000 1=0+1)","memory_value.rstat":"n(v0 rc2024-04-26T23:04:30.274604+0000 2=0+2)","error_str":""},"return_code":-61} 2024-04-26T23:05:18.302 INFO:tasks.ceph.mds.c.smithi040.stderr: -7774> 2024-04-26T23:04:33.025+0000 7f47d7c00640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T23:05:18.315 INFO:tasks.ceph.mds.c.smithi040.stderr: -3185> 2024-04-26T23:04:47.890+0000 7f47d2c00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000002 [...2,head] /d1/d2/d3/ auth{1=1,2=1} v19 ap=1 f(v0 m2024-04-26T23:04:30.268604+0000 1=0+1) n(v3 rc2024-04-26T23:04:30.494601+0000 6=0+6) (inest mix dirty) (ifile excl) caps={5356=pAsLsXsFsx/pAsLsXsFsxcral@4},l=5356 | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 0x55e20976b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000002:[<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:04:30.268604+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:04:30.494601+0000 6=0+6)","memory_value.dirstat":"f(v0 m2024-04-26T23:04:30.268604+0000 1=0+1)","memory_value.rstat":"n(v3 rc2024-04-26T23:04:30.494601+0000 6=0+6)","error_str":""},"return_code":-61} 2024-04-26T23:05:18.316 INFO:tasks.ceph.mds.c.smithi040.stderr: -3061> 2024-04-26T23:04:47.893+0000 7f47dae00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000003 [...2,head] /d1/d2/d3/d4/ auth{1=1,2=1} v19 ap=2 f(v0 m2024-04-26T23:04:30.269604+0000 1=0+1) n(v4 rc2024-04-26T23:04:30.494601+0000 5=0+5) (inest sync->mix g=1,2) caps={5356=pAsLsXsFs/pAsLsXsFsxcral@8} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 export_pin=1 0x55e20976ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000003:[<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:04:30.269604+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:04:30.494601+0000 5=0+5)","memory_value.dirstat":"f(v0 m2024-04-26T23:04:30.269604+0000 1=0+1)","memory_value.rstat":"n(v4 rc2024-04-26T23:04:30.494601+0000 5=0+5)","error_str":""},"return_code":-61} 2024-04-26T23:05:18.333 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-04-26T23:05:18.367 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-26T23:05:18.377 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:05:18.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.472+0000 7fa3b0a00640 1 Processor -- start 2024-04-26T23:05:18.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.472+0000 7fa3b0a00640 1 -- start start 2024-04-26T23:05:18.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.473+0000 7fa3b0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3ac06f790 0x7fa3ac06fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:18.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.473+0000 7fa3b0a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa3ac072c50 0x7fa3ac0730c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:18.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.473+0000 7fa3b0a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ac156840 con 0x7fa3ac156460 2024-04-26T23:05:18.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.473+0000 7fa3b0a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa3ac0cbf20 con 0x7fa3ac072c50 2024-04-26T23:05:18.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.473+0000 7fa3b0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa3ac0cc200 con 0x7fa3ac06f790 2024-04-26T23:05:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.474+0000 7fa3a9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3ac06f790 0x7fa3ac06fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.474+0000 7fa3aa800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa3ac072c50 0x7fa3ac0730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.474+0000 7fa3a9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3ac06f790 0x7fa3ac06fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:58898/0 (socket says 172.21.15.40:58898) 2024-04-26T23:05:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.474+0000 7fa3a9400640 1 -- 172.21.15.40:0/304255917 learned_addr learned my addr 172.21.15.40:0/304255917 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.475+0000 7fa3a9400640 1 -- 172.21.15.40:0/304255917 >> v2:172.21.15.103:3300/0 conn(0x7fa3ac072c50 msgr2=0x7fa3ac0730c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.475+0000 7fa3a9400640 1 --2- 172.21.15.40:0/304255917 >> v2:172.21.15.103:3300/0 conn(0x7fa3ac072c50 0x7fa3ac0730c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.475+0000 7fa3a9400640 1 -- 172.21.15.40:0/304255917 >> v1:172.21.15.103:6789/0 conn(0x7fa3ac156460 legacy=0x7fa3ac0cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.475+0000 7fa3a9400640 1 -- 172.21.15.40:0/304255917 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ac0cc4e0 con 0x7fa3ac06f790 2024-04-26T23:05:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.476+0000 7fa3a9400640 1 --2- 172.21.15.40:0/304255917 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3ac06f790 0x7fa3ac06fc10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa398009870 tx=0x7fa39802c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1248e9dd18c0054a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:18.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.477+0000 7fa3a8a00640 1 -- 172.21.15.40:0/304255917 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398038070 con 0x7fa3ac06f790 2024-04-26T23:05:18.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.477+0000 7fa3a8a00640 1 -- 172.21.15.40:0/304255917 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa398031710 con 0x7fa3ac06f790 2024-04-26T23:05:18.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.477+0000 7fa3a8a00640 1 -- 172.21.15.40:0/304255917 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398031ae0 con 0x7fa3ac06f790 2024-04-26T23:05:18.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.478+0000 7fa3b0a00640 1 -- 172.21.15.40:0/304255917 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3ac06f790 msgr2=0x7fa3ac06fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:18.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.478+0000 7fa3b0a00640 1 --2- 172.21.15.40:0/304255917 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3ac06f790 0x7fa3ac06fc10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa398009870 tx=0x7fa39802c460 comp rx=0 tx=0).stop 2024-04-26T23:05:18.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.478+0000 7fa3b0a00640 1 -- 172.21.15.40:0/304255917 shutdown_connections 2024-04-26T23:05:18.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.478+0000 7fa3b0a00640 1 --2- 172.21.15.40:0/304255917 >> v2:172.21.15.103:3300/0 conn(0x7fa3ac072c50 0x7fa3ac0730c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:18.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.478+0000 7fa3b0a00640 1 --2- 172.21.15.40:0/304255917 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3ac06f790 0x7fa3ac06fc10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:18.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.478+0000 7fa3b0a00640 1 -- 172.21.15.40:0/304255917 >> 172.21.15.40:0/304255917 conn(0x7fa3ac0d0be0 msgr2=0x7fa3ac0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:18.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.478+0000 7fa3b0a00640 1 -- 172.21.15.40:0/304255917 shutdown_connections 2024-04-26T23:05:18.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.478+0000 7fa3b0a00640 1 -- 172.21.15.40:0/304255917 wait complete. 2024-04-26T23:05:18.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.480+0000 7fa3b0a00640 1 Processor -- start 2024-04-26T23:05:18.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.480+0000 7fa3b0a00640 1 -- start start 2024-04-26T23:05:18.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.481+0000 7fa3b0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3ac06f790 0x7fa3ac1f5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:18.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.481+0000 7fa3b0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3ac072c50 0x7fa3ac1f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:18.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.481+0000 7fa3b0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3ac156460 0x7fa3ac0afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:18.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.481+0000 7fa3b0a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ac0cbbe0 con 0x7fa3ac156460 2024-04-26T23:05:18.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.481+0000 7fa3b0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ac072700 con 0x7fa3ac072c50 2024-04-26T23:05:18.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.481+0000 7fa3b0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa3ac06e260 con 0x7fa3ac06f790 2024-04-26T23:05:18.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.481+0000 7fa3aa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3ac156460 0x7fa3ac0afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:18.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.481+0000 7fa3aa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3ac156460 0x7fa3ac0afaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44890/0 (socket says 172.21.15.40:44890) 2024-04-26T23:05:18.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.481+0000 7fa3aa800640 1 -- 172.21.15.40:0/1944101989 learned_addr learned my addr 172.21.15.40:0/1944101989 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:18.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.482+0000 7fa3a9e00640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3ac06f790 0x7fa3ac1f5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:18.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.482+0000 7fa3a9400640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3ac072c50 0x7fa3ac1f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:18.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.482+0000 7fa3aa800640 1 -- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3ac06f790 msgr2=0x7fa3ac1f5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:18.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.482+0000 7fa3aa800640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3ac06f790 0x7fa3ac1f5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:18.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.482+0000 7fa3aa800640 1 -- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3ac072c50 msgr2=0x7fa3ac1f4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:18.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.482+0000 7fa3aa800640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3ac072c50 0x7fa3ac1f4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:18.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.482+0000 7fa3aa800640 1 -- 172.21.15.40:0/1944101989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ac0b0260 con 0x7fa3ac156460 2024-04-26T23:05:18.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.482+0000 7fa3aa800640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3ac156460 0x7fa3ac0afaf0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400efc0 tx=0x7fa3a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:18.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.483+0000 7fa3a2000640 1 -- 172.21.15.40:0/1944101989 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4018020 con 0x7fa3ac156460 2024-04-26T23:05:18.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.483+0000 7fa3a2000640 1 -- 172.21.15.40:0/1944101989 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a4014480 con 0x7fa3ac156460 2024-04-26T23:05:18.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.483+0000 7fa3a2000640 1 -- 172.21.15.40:0/1944101989 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4014870 con 0x7fa3ac156460 2024-04-26T23:05:18.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.483+0000 7fa3b0a00640 1 -- 172.21.15.40:0/1944101989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3ac0c7cb0 con 0x7fa3ac156460 2024-04-26T23:05:18.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.483+0000 7fa3b0a00640 1 -- 172.21.15.40:0/1944101989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3ac0c8270 con 0x7fa3ac156460 2024-04-26T23:05:18.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.484+0000 7fa3b0a00640 1 -- 172.21.15.40:0/1944101989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3ac0694f0 con 0x7fa3ac156460 2024-04-26T23:05:18.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.485+0000 7fa3a2000640 1 -- 172.21.15.40:0/1944101989 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa3a401b020 con 0x7fa3ac156460 2024-04-26T23:05:18.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.486+0000 7fa3a2000640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa38c06c8f0 0x7fa38c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:18.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.487+0000 7fa3a2000640 1 -- 172.21.15.40:0/1944101989 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6417+0+0 (secure 0 0 0) 0x7fa3a408ca40 con 0x7fa3ac156460 2024-04-26T23:05:18.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.487+0000 7fa3a9e00640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa38c06c8f0 0x7fa38c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:18.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.488+0000 7fa3a9e00640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa38c06c8f0 0x7fa38c06edb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa39c007fd0 tx=0x7fa39c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:18.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.491+0000 7fa3a2000640 1 -- 172.21.15.40:0/1944101989 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa3a4022030 con 0x7fa3ac156460 2024-04-26T23:05:18.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:18.739+0000 7fa3b0a00640 1 -- 172.21.15.40:0/1944101989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa3ac06de80 con 0x7fa3ac156460 2024-04-26T23:05:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.260+0000 7fa3a2000640 1 -- 172.21.15.40:0/1944101989 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v48) ==== 104+0+0 (secure 0 0 0) 0x7fa3a4010040 con 0x7fa3ac156460 2024-04-26T23:05:19.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.264+0000 7fa3b0a00640 1 -- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa38c06c8f0 msgr2=0x7fa38c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:19.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.264+0000 7fa3b0a00640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa38c06c8f0 0x7fa38c06edb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa39c007fd0 tx=0x7fa39c009040 comp rx=0 tx=0).stop 2024-04-26T23:05:19.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.264+0000 7fa3b0a00640 1 -- 172.21.15.40:0/1944101989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3ac156460 msgr2=0x7fa3ac0afaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:19.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.264+0000 7fa3b0a00640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3ac156460 0x7fa3ac0afaf0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400efc0 tx=0x7fa3a400c490 comp rx=0 tx=0).stop 2024-04-26T23:05:19.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.265+0000 7fa3b0a00640 1 -- 172.21.15.40:0/1944101989 shutdown_connections 2024-04-26T23:05:19.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.265+0000 7fa3b0a00640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa38c06c8f0 0x7fa38c06edb0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:19.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.265+0000 7fa3b0a00640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3ac156460 0x7fa3ac0afaf0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:19.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.265+0000 7fa3b0a00640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3ac072c50 0x7fa3ac1f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:19.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.265+0000 7fa3b0a00640 1 --2- 172.21.15.40:0/1944101989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa3ac06f790 0x7fa3ac1f5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:19.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.265+0000 7fa3b0a00640 1 -- 172.21.15.40:0/1944101989 >> 172.21.15.40:0/1944101989 conn(0x7fa3ac0d0be0 msgr2=0x7fa3ac06e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:19.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.265+0000 7fa3b0a00640 1 -- 172.21.15.40:0/1944101989 shutdown_connections 2024-04-26T23:05:19.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.265+0000 7fa3b0a00640 1 -- 172.21.15.40:0/1944101989 wait complete. 2024-04-26T23:05:19.288 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T23:05:19.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.549+0000 7f1c3bc00640 1 Processor -- start 2024-04-26T23:05:19.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.549+0000 7f1c3bc00640 1 -- start start 2024-04-26T23:05:19.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.551+0000 7f1c3bc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c341489a0 0x7f1c340710e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:19.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.551+0000 7f1c3bc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1c340b04b0 0x7f1c340b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:19.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.551+0000 7f1c3bc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c34148d80 con 0x7f1c3406c240 2024-04-26T23:05:19.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.551+0000 7f1c3bc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1c340b11d0 con 0x7f1c340b04b0 2024-04-26T23:05:19.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.551+0000 7f1c3bc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c340b0f10 con 0x7f1c341489a0 2024-04-26T23:05:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.552+0000 7f1c39200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c341489a0 0x7f1c340710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.552+0000 7f1c39200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c341489a0 0x7f1c340710e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:58912/0 (socket says 172.21.15.40:58912) 2024-04-26T23:05:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.552+0000 7f1c39200640 1 -- 172.21.15.40:0/611114062 learned_addr learned my addr 172.21.15.40:0/611114062 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.552+0000 7f1c39c00640 1 --1- 172.21.15.40:0/611114062 >> v1:172.21.15.103:6789/0 conn(0x7f1c3406c240 0x7f1c340ae8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:43552/0 (socket says 172.21.15.40:43552) 2024-04-26T23:05:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.552+0000 7f1c33e00640 1 --2- 172.21.15.40:0/611114062 >> v2:172.21.15.103:3300/0 conn(0x7f1c340b04b0 0x7f1c340b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.552+0000 7f1c39200640 1 -- 172.21.15.40:0/611114062 >> v2:172.21.15.103:3300/0 conn(0x7f1c340b04b0 msgr2=0x7f1c340b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.552+0000 7f1c39200640 1 --2- 172.21.15.40:0/611114062 >> v2:172.21.15.103:3300/0 conn(0x7f1c340b04b0 0x7f1c340b08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.552+0000 7f1c39200640 1 -- 172.21.15.40:0/611114062 >> v1:172.21.15.103:6789/0 conn(0x7f1c3406c240 legacy=0x7f1c340ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.552+0000 7f1c39200640 1 -- 172.21.15.40:0/611114062 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c34070600 con 0x7f1c341489a0 2024-04-26T23:05:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.554+0000 7f1c39200640 1 --2- 172.21.15.40:0/611114062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c341489a0 0x7f1c340710e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c24009870 tx=0x7f1c2402a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed44129b6b36b8f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.554+0000 7f1c33400640 1 -- 172.21.15.40:0/611114062 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c24037070 con 0x7f1c341489a0 2024-04-26T23:05:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.554+0000 7f1c33400640 1 -- 172.21.15.40:0/611114062 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c24032440 con 0x7f1c341489a0 2024-04-26T23:05:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.554+0000 7f1c33400640 1 -- 172.21.15.40:0/611114062 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c24032810 con 0x7f1c341489a0 2024-04-26T23:05:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.555+0000 7f1c3bc00640 1 -- 172.21.15.40:0/611114062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c341489a0 msgr2=0x7f1c340710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.555+0000 7f1c3bc00640 1 --2- 172.21.15.40:0/611114062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c341489a0 0x7f1c340710e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c24009870 tx=0x7f1c2402a250 comp rx=0 tx=0).stop 2024-04-26T23:05:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.555+0000 7f1c3bc00640 1 -- 172.21.15.40:0/611114062 shutdown_connections 2024-04-26T23:05:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.555+0000 7f1c3bc00640 1 --2- 172.21.15.40:0/611114062 >> v2:172.21.15.103:3300/0 conn(0x7f1c340b04b0 0x7f1c340b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.555+0000 7f1c3bc00640 1 --2- 172.21.15.40:0/611114062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c341489a0 0x7f1c340710e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.555+0000 7f1c3bc00640 1 -- 172.21.15.40:0/611114062 >> 172.21.15.40:0/611114062 conn(0x7f1c340b3be0 msgr2=0x7f1c340b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.555+0000 7f1c3bc00640 1 -- 172.21.15.40:0/611114062 shutdown_connections 2024-04-26T23:05:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.555+0000 7f1c3bc00640 1 -- 172.21.15.40:0/611114062 wait complete. 2024-04-26T23:05:19.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.558+0000 7f1c3bc00640 1 Processor -- start 2024-04-26T23:05:19.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.558+0000 7f1c3bc00640 1 -- start start 2024-04-26T23:05:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.559+0000 7f1c3bc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c3406c240 0x7f1c341ea6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.559+0000 7f1c3bc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c340b04b0 0x7f1c340b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.559+0000 7f1c3bc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c341489a0 0x7f1c340b7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.559+0000 7f1c3bc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1c34071620 con 0x7f1c340b04b0 2024-04-26T23:05:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.559+0000 7f1c3bc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c34070bc0 con 0x7f1c341489a0 2024-04-26T23:05:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.559+0000 7f1c3bc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c341604d0 con 0x7f1c3406c240 2024-04-26T23:05:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.560+0000 7f1c33e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c340b04b0 0x7f1c340b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.560+0000 7f1c33e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c340b04b0 0x7f1c340b7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44902/0 (socket says 172.21.15.40:44902) 2024-04-26T23:05:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.560+0000 7f1c33e00640 1 -- 172.21.15.40:0/438548101 learned_addr learned my addr 172.21.15.40:0/438548101 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.560+0000 7f1c39200640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c3406c240 0x7f1c341ea6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.560+0000 7f1c39c00640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c341489a0 0x7f1c340b7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.560+0000 7f1c33e00640 1 -- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c3406c240 msgr2=0x7f1c341ea6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.560+0000 7f1c33e00640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c3406c240 0x7f1c341ea6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.560+0000 7f1c33e00640 1 -- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c341489a0 msgr2=0x7f1c340b7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.560+0000 7f1c33e00640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c341489a0 0x7f1c340b7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.560+0000 7f1c33e00640 1 -- 172.21.15.40:0/438548101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c340ce920 con 0x7f1c340b04b0 2024-04-26T23:05:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.560+0000 7f1c39c00640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c341489a0 0x7f1c340b7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.560+0000 7f1c33e00640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c340b04b0 0x7f1c340b7030 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000efc0 tx=0x7f1c2000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.561+0000 7f1c30c00640 1 -- 172.21.15.40:0/438548101 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20018020 con 0x7f1c340b04b0 2024-04-26T23:05:19.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.561+0000 7f1c30c00640 1 -- 172.21.15.40:0/438548101 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c20014480 con 0x7f1c340b04b0 2024-04-26T23:05:19.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.561+0000 7f1c30c00640 1 -- 172.21.15.40:0/438548101 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20014850 con 0x7f1c340b04b0 2024-04-26T23:05:19.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.561+0000 7f1c3bc00640 1 -- 172.21.15.40:0/438548101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c341eac10 con 0x7f1c340b04b0 2024-04-26T23:05:19.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.561+0000 7f1c3bc00640 1 -- 172.21.15.40:0/438548101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c340ceec0 con 0x7f1c340b04b0 2024-04-26T23:05:19.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.562+0000 7f1c3bc00640 1 -- 172.21.15.40:0/438548101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c34069600 con 0x7f1c340b04b0 2024-04-26T23:05:19.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.563+0000 7f1c30c00640 1 -- 172.21.15.40:0/438548101 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1c20005d10 con 0x7f1c340b04b0 2024-04-26T23:05:19.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.564+0000 7f1c30c00640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c1406c8f0 0x7f1c1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:19.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.564+0000 7f1c30c00640 1 -- 172.21.15.40:0/438548101 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6417+0+0 (secure 0 0 0) 0x7f1c2008c980 con 0x7f1c340b04b0 2024-04-26T23:05:19.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.565+0000 7f1c39200640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c1406c8f0 0x7f1c1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:19.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.566+0000 7f1c39200640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c1406c8f0 0x7f1c1406edb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1c24034000 tx=0x7f1c2402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:19.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.570+0000 7f1c30c00640 1 -- 172.21.15.40:0/438548101 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1c20022070 con 0x7f1c340b04b0 2024-04-26T23:05:19.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:19.781+0000 7f1c3bc00640 1 -- 172.21.15.40:0/438548101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1c340b49e0 con 0x7f1c340b04b0 2024-04-26T23:05:20.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.612+0000 7f1c30c00640 1 -- 172.21.15.40:0/438548101 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v56) ==== 181+0+0 (secure 0 0 0) 0x7f1c20010040 con 0x7f1c340b04b0 2024-04-26T23:05:20.613 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-04-26T23:05:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.616+0000 7f1c3bc00640 1 -- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c1406c8f0 msgr2=0x7f1c1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.616+0000 7f1c3bc00640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c1406c8f0 0x7f1c1406edb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1c24034000 tx=0x7f1c2402e000 comp rx=0 tx=0).stop 2024-04-26T23:05:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.616+0000 7f1c3bc00640 1 -- 172.21.15.40:0/438548101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c340b04b0 msgr2=0x7f1c340b7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.616+0000 7f1c3bc00640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c340b04b0 0x7f1c340b7030 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000efc0 tx=0x7f1c2000c490 comp rx=0 tx=0).stop 2024-04-26T23:05:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.616+0000 7f1c3bc00640 1 -- 172.21.15.40:0/438548101 shutdown_connections 2024-04-26T23:05:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.616+0000 7f1c3bc00640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c1406c8f0 0x7f1c1406edb0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.617+0000 7f1c3bc00640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c341489a0 0x7f1c340b7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.617+0000 7f1c3bc00640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c340b04b0 0x7f1c340b7030 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.617+0000 7f1c3bc00640 1 --2- 172.21.15.40:0/438548101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c3406c240 0x7f1c341ea6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.617+0000 7f1c3bc00640 1 -- 172.21.15.40:0/438548101 >> 172.21.15.40:0/438548101 conn(0x7f1c340b3be0 msgr2=0x7f1c341629f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.617+0000 7f1c3bc00640 1 -- 172.21.15.40:0/438548101 shutdown_connections 2024-04-26T23:05:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.617+0000 7f1c3bc00640 1 -- 172.21.15.40:0/438548101 wait complete. 2024-04-26T23:05:20.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T23:05:20.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.866+0000 7fe175400640 1 Processor -- start 2024-04-26T23:05:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.867+0000 7fe175400640 1 -- start start 2024-04-26T23:05:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.868+0000 7fe175400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe1700b6b40 0x7fe1700b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.868+0000 7fe175400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1700b7880 con 0x7fe170163f40 2024-04-26T23:05:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.868+0000 7fe175400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1700d05a0 con 0x7fe1701633a0 2024-04-26T23:05:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.868+0000 7fe175400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe1700b5ed0 con 0x7fe1700b6b40 2024-04-26T23:05:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.869+0000 7fe174a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe170163f40 0x7fe1700b47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50606/0 (socket says 172.21.15.40:50606) 2024-04-26T23:05:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.869+0000 7fe174a00640 1 -- 172.21.15.40:0/171361136 learned_addr learned my addr 172.21.15.40:0/171361136 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.869+0000 7fe16fe00640 1 --2- 172.21.15.40:0/171361136 >> v2:172.21.15.103:3300/0 conn(0x7fe1700b6b40 0x7fe1700b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.869+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2336994239 0 0) 0x7fe1700b7880 con 0x7fe170163f40 2024-04-26T23:05:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.869+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe150003520 con 0x7fe170163f40 2024-04-26T23:05:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1625013204 0 0) 0x7fe150003520 con 0x7fe170163f40 2024-04-26T23:05:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 >> v2:172.21.15.103:3300/0 conn(0x7fe1700b6b40 msgr2=0x7fe1700b6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16ea00640 1 --2- 172.21.15.40:0/171361136 >> v2:172.21.15.103:3300/0 conn(0x7fe1700b6b40 0x7fe1700b6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 >> v1:172.21.15.103:6789/0 conn(0x7fe1701633a0 legacy=0x7fe170163830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1700b75a0 con 0x7fe170163f40 2024-04-26T23:05:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fe168001670 con 0x7fe170163f40 2024-04-26T23:05:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 >> v1:172.21.15.40:6789/0 conn(0x7fe170163f40 legacy=0x7fe1700b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16ea00640 1 --2- 172.21.15.40:0/171361136 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe150003a00 0x7fe150003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16ea00640 1 --2- 172.21.15.40:0/171361136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe150004800 0x7fe150004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16ea00640 1 --2- 172.21.15.40:0/171361136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe150005210 0x7fe15000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe1700b5ed0 con 0x7fe150004800 2024-04-26T23:05:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe15000c160 con 0x7fe150005210 2024-04-26T23:05:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe15000c360 con 0x7fe150003a00 2024-04-26T23:05:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16f400640 1 --2- 172.21.15.40:0/171361136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe150004800 0x7fe150004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16fe00640 1 --2- 172.21.15.40:0/171361136 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe150003a00 0x7fe150003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16f400640 1 -- 172.21.15.40:0/171361136 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe150003a00 msgr2=0x7fe150003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16f400640 1 --2- 172.21.15.40:0/171361136 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe150003a00 0x7fe150003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16f400640 1 -- 172.21.15.40:0/171361136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe150005210 msgr2=0x7fe15000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16f400640 1 --2- 172.21.15.40:0/171361136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe150005210 0x7fe15000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe16f400640 1 -- 172.21.15.40:0/171361136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe150003520 con 0x7fe150004800 2024-04-26T23:05:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.870+0000 7fe174a00640 1 -- 172.21.15.40:0/171361136 reap_dead start 2024-04-26T23:05:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.871+0000 7fe16f400640 1 --2- 172.21.15.40:0/171361136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe150004800 0x7fe150004c80 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe158001e20 tx=0x7fe150005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c672a4a207aa10 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.872+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe158001a70 con 0x7fe150004800 2024-04-26T23:05:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.872+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe15802ae40 con 0x7fe150004800 2024-04-26T23:05:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.872+0000 7fe16ea00640 1 -- 172.21.15.40:0/171361136 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe158003bb0 con 0x7fe150004800 2024-04-26T23:05:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.872+0000 7fe175400640 1 -- 172.21.15.40:0/171361136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe150004800 msgr2=0x7fe150004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.872+0000 7fe175400640 1 --2- 172.21.15.40:0/171361136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe150004800 0x7fe150004c80 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe158001e20 tx=0x7fe150005830 comp rx=0 tx=0).stop 2024-04-26T23:05:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.872+0000 7fe175400640 1 -- 172.21.15.40:0/171361136 shutdown_connections 2024-04-26T23:05:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.872+0000 7fe175400640 1 --2- 172.21.15.40:0/171361136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe150004800 0x7fe150004c80 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.872+0000 7fe175400640 1 -- 172.21.15.40:0/171361136 >> 172.21.15.40:0/171361136 conn(0x7fe1700d4e50 msgr2=0x7fe1700d5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.873+0000 7fe175400640 1 -- 172.21.15.40:0/171361136 shutdown_connections 2024-04-26T23:05:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.873+0000 7fe175400640 1 -- 172.21.15.40:0/171361136 wait complete. 2024-04-26T23:05:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.875+0000 7fe175400640 1 Processor -- start 2024-04-26T23:05:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.875+0000 7fe175400640 1 -- start start 2024-04-26T23:05:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.876+0000 7fe175400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe150004800 0x7fe1700ce920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.876+0000 7fe175400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1701f8390 0x7fe170156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.876+0000 7fe175400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1700b2c40 0x7fe1700b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.876+0000 7fe175400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe1700b6180 con 0x7fe150004800 2024-04-26T23:05:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.876+0000 7fe175400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe1700d0bb0 con 0x7fe1700b2c40 2024-04-26T23:05:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.876+0000 7fe175400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe1700b6300 con 0x7fe1701f8390 2024-04-26T23:05:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.876+0000 7fe16fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe150004800 0x7fe1700ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.876+0000 7fe16fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe150004800 0x7fe1700ce920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44918/0 (socket says 172.21.15.40:44918) 2024-04-26T23:05:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.876+0000 7fe16fe00640 1 -- 172.21.15.40:0/1767236639 learned_addr learned my addr 172.21.15.40:0/1767236639 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.876+0000 7fe16fe00640 1 -- 172.21.15.40:0/1767236639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1701f8390 msgr2=0x7fe170156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.876+0000 7fe16fe00640 1 --2- 172.21.15.40:0/1767236639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1701f8390 0x7fe170156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.876+0000 7fe16fe00640 1 -- 172.21.15.40:0/1767236639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1700b2c40 msgr2=0x7fe1700b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.876+0000 7fe16fe00640 1 --2- 172.21.15.40:0/1767236639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1700b2c40 0x7fe1700b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.877+0000 7fe16fe00640 1 -- 172.21.15.40:0/1767236639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1700cee60 con 0x7fe150004800 2024-04-26T23:05:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.877+0000 7fe16fe00640 1 --2- 172.21.15.40:0/1767236639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe150004800 0x7fe1700ce920 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe1600108e0 tx=0x7fe160010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.877+0000 7fe167e00640 1 -- 172.21.15.40:0/1767236639 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16000f040 con 0x7fe150004800 2024-04-26T23:05:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.877+0000 7fe167e00640 1 -- 172.21.15.40:0/1767236639 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16000eb60 con 0x7fe150004800 2024-04-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.877+0000 7fe167e00640 1 -- 172.21.15.40:0/1767236639 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16001c400 con 0x7fe150004800 2024-04-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.877+0000 7fe175400640 1 -- 172.21.15.40:0/1767236639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe170157040 con 0x7fe150004800 2024-04-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.877+0000 7fe175400640 1 -- 172.21.15.40:0/1767236639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1701f8820 con 0x7fe150004800 2024-04-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.878+0000 7fe175400640 1 -- 172.21.15.40:0/1767236639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1700c9940 con 0x7fe150004800 2024-04-26T23:05:20.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.879+0000 7fe167e00640 1 -- 172.21.15.40:0/1767236639 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe16001c5a0 con 0x7fe150004800 2024-04-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.880+0000 7fe167e00640 1 --2- 172.21.15.40:0/1767236639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe14c06c920 0x7fe14c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:20.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.880+0000 7fe167e00640 1 -- 172.21.15.40:0/1767236639 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5979+0+0 (secure 0 0 0) 0x7fe16008c470 con 0x7fe150004800 2024-04-26T23:05:20.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.881+0000 7fe16f400640 1 --2- 172.21.15.40:0/1767236639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe14c06c920 0x7fe14c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:20.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.882+0000 7fe16f400640 1 --2- 172.21.15.40:0/1767236639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe14c06c920 0x7fe14c06ede0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe158032000 tx=0x7fe15802c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:20.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:20.885+0000 7fe167e00640 1 -- 172.21.15.40:0/1767236639 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe160018030 con 0x7fe150004800 2024-04-26T23:05:21.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.100+0000 7fe175400640 1 -- 172.21.15.40:0/1767236639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe170069a80 con 0x7fe150004800 2024-04-26T23:05:21.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.623+0000 7fe167e00640 1 -- 172.21.15.40:0/1767236639 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v57) ==== 169+0+0 (secure 0 0 0) 0x7fe16005a130 con 0x7fe150004800 2024-04-26T23:05:21.758 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-04-26T23:05:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.626+0000 7fe175400640 1 -- 172.21.15.40:0/1767236639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe14c06c920 msgr2=0x7fe14c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.626+0000 7fe175400640 1 --2- 172.21.15.40:0/1767236639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe14c06c920 0x7fe14c06ede0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe158032000 tx=0x7fe15802c000 comp rx=0 tx=0).stop 2024-04-26T23:05:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.626+0000 7fe175400640 1 -- 172.21.15.40:0/1767236639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe150004800 msgr2=0x7fe1700ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.626+0000 7fe175400640 1 --2- 172.21.15.40:0/1767236639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe150004800 0x7fe1700ce920 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe1600108e0 tx=0x7fe160010da0 comp rx=0 tx=0).stop 2024-04-26T23:05:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.627+0000 7fe175400640 1 -- 172.21.15.40:0/1767236639 shutdown_connections 2024-04-26T23:05:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.627+0000 7fe175400640 1 --2- 172.21.15.40:0/1767236639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe14c06c920 0x7fe14c06ede0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.627+0000 7fe175400640 1 --2- 172.21.15.40:0/1767236639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1700b2c40 0x7fe1700b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.627+0000 7fe175400640 1 --2- 172.21.15.40:0/1767236639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1701f8390 0x7fe170156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.627+0000 7fe175400640 1 --2- 172.21.15.40:0/1767236639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe150004800 0x7fe1700ce920 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.627+0000 7fe175400640 1 -- 172.21.15.40:0/1767236639 >> 172.21.15.40:0/1767236639 conn(0x7fe1700d4e50 msgr2=0x7fe17015dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:21.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.627+0000 7fe175400640 1 -- 172.21.15.40:0/1767236639 shutdown_connections 2024-04-26T23:05:21.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.627+0000 7fe175400640 1 -- 172.21.15.40:0/1767236639 wait complete. 2024-04-26T23:05:21.761 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-26T23:05:21.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.985+0000 7f4ff7e00640 1 Processor -- start 2024-04-26T23:05:21.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.985+0000 7f4ff7e00640 1 -- start start 2024-04-26T23:05:21.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.986+0000 7f4ff7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff80d32e0 0x7f4ff80d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:21.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.986+0000 7f4ff7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ff80b8d20 0x7f4ff80b91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:21.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.986+0000 7f4ff7e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ff8162c40 con 0x7f4ff80b6220 2024-04-26T23:05:21.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.986+0000 7f4ff7e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f4ff80d3c80 con 0x7f4ff80b8d20 2024-04-26T23:05:21.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.986+0000 7f4ff7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff80d2d80 con 0x7f4ff80d32e0 2024-04-26T23:05:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.987+0000 7f4ff7400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ff80b8d20 0x7f4ff80b91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.987+0000 7f4ff6a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4ff80b6220 0x7f4ff80d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50608/0 (socket says 172.21.15.40:50608) 2024-04-26T23:05:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.987+0000 7f4ff6a00640 1 -- 172.21.15.40:0/3898171512 learned_addr learned my addr 172.21.15.40:0/3898171512 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.988+0000 7f4ff6000640 1 --2- 172.21.15.40:0/3898171512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff80d32e0 0x7f4ff80d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.988+0000 7f4ff7400640 1 -- 172.21.15.40:0/3898171512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff80d32e0 msgr2=0x7f4ff80d3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.988+0000 7f4ff7400640 1 --2- 172.21.15.40:0/3898171512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff80d32e0 0x7f4ff80d3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.988+0000 7f4ff5600640 1 -- 172.21.15.40:0/3898171512 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2531036082 0 0) 0x7f4ff8162c40 con 0x7f4ff80b6220 2024-04-26T23:05:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.988+0000 7f4ff7400640 1 -- 172.21.15.40:0/3898171512 >> v1:172.21.15.40:6789/0 conn(0x7f4ff80b6220 legacy=0x7f4ff80d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.988+0000 7f4ff7400640 1 -- 172.21.15.40:0/3898171512 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff80b7c80 con 0x7f4ff80b8d20 2024-04-26T23:05:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.988+0000 7f4ff6000640 1 --2- 172.21.15.40:0/3898171512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff80d32e0 0x7f4ff80d3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.989+0000 7f4ff7400640 1 --2- 172.21.15.40:0/3898171512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff80b8d20 0x7f4ff80b91b0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009d90 tx=0x7f4ff002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb443e200d88291f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.989+0000 7f4ff5600640 1 -- 172.21.15.40:0/3898171512 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0007740 con 0x7f4ff80b8d20 2024-04-26T23:05:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.989+0000 7f4ff5600640 1 -- 172.21.15.40:0/3898171512 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ff00078e0 con 0x7f4ff80b8d20 2024-04-26T23:05:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.989+0000 7f4ff5600640 1 -- 172.21.15.40:0/3898171512 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0007cb0 con 0x7f4ff80b8d20 2024-04-26T23:05:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.990+0000 7f4ff7e00640 1 -- 172.21.15.40:0/3898171512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff80b8d20 msgr2=0x7f4ff80b91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.990+0000 7f4ff7e00640 1 --2- 172.21.15.40:0/3898171512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff80b8d20 0x7f4ff80b91b0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009d90 tx=0x7f4ff002a660 comp rx=0 tx=0).stop 2024-04-26T23:05:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.990+0000 7f4ff7e00640 1 -- 172.21.15.40:0/3898171512 shutdown_connections 2024-04-26T23:05:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.990+0000 7f4ff7e00640 1 --2- 172.21.15.40:0/3898171512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff80b8d20 0x7f4ff80b91b0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.990+0000 7f4ff7e00640 1 --2- 172.21.15.40:0/3898171512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff80d32e0 0x7f4ff80d3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.990+0000 7f4ff7e00640 1 -- 172.21.15.40:0/3898171512 >> 172.21.15.40:0/3898171512 conn(0x7f4ff80d4d90 msgr2=0x7f4ff80d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.990+0000 7f4ff7e00640 1 -- 172.21.15.40:0/3898171512 shutdown_connections 2024-04-26T23:05:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.990+0000 7f4ff7e00640 1 -- 172.21.15.40:0/3898171512 wait complete. 2024-04-26T23:05:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.992+0000 7f4ff7e00640 1 Processor -- start 2024-04-26T23:05:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.992+0000 7f4ff7e00640 1 -- start start 2024-04-26T23:05:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.993+0000 7f4ff7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff80b6220 0x7f4ff814f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.993+0000 7f4ff7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff80b8d20 0x7f4ff81f4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.993+0000 7f4ff7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ff80d32e0 0x7f4ff815f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.993+0000 7f4ff7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff80b8520 con 0x7f4ff80b6220 2024-04-26T23:05:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.993+0000 7f4ff7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff80b8240 con 0x7f4ff80d32e0 2024-04-26T23:05:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.993+0000 7f4ff7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff80d0aa0 con 0x7f4ff80b8d20 2024-04-26T23:05:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.993+0000 7f4ff6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff80b6220 0x7f4ff814f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.993+0000 7f4ff6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff80b6220 0x7f4ff814f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44932/0 (socket says 172.21.15.40:44932) 2024-04-26T23:05:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.993+0000 7f4ff6a00640 1 -- 172.21.15.40:0/1637730952 learned_addr learned my addr 172.21.15.40:0/1637730952 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.994+0000 7f4ff6000640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff80b8d20 0x7f4ff81f4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.994+0000 7f4ff7400640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ff80d32e0 0x7f4ff815f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.994+0000 7f4ff6a00640 1 -- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff80b8d20 msgr2=0x7f4ff81f4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.994+0000 7f4ff6a00640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff80b8d20 0x7f4ff81f4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.994+0000 7f4ff6a00640 1 -- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ff80d32e0 msgr2=0x7f4ff815f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.994+0000 7f4ff6a00640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ff80d32e0 0x7f4ff815f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.994+0000 7f4ff6a00640 1 -- 172.21.15.40:0/1637730952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff815a1d0 con 0x7f4ff80b6220 2024-04-26T23:05:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.994+0000 7f4ff7400640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ff80d32e0 0x7f4ff815f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.994+0000 7f4ff6a00640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff80b6220 0x7f4ff814f030 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800c460 tx=0x7f4fe800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.995+0000 7f4feea00640 1 -- 172.21.15.40:0/1637730952 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8018070 con 0x7f4ff80b6220 2024-04-26T23:05:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.995+0000 7f4feea00640 1 -- 172.21.15.40:0/1637730952 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe8007500 con 0x7f4ff80b6220 2024-04-26T23:05:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.995+0000 7f4feea00640 1 -- 172.21.15.40:0/1637730952 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8004a20 con 0x7f4ff80b6220 2024-04-26T23:05:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.995+0000 7f4ff7e00640 1 -- 172.21.15.40:0/1637730952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff815a4b0 con 0x7f4ff80b6220 2024-04-26T23:05:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.995+0000 7f4ff7e00640 1 -- 172.21.15.40:0/1637730952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff814f570 con 0x7f4ff80b6220 2024-04-26T23:05:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.996+0000 7f4ff7e00640 1 -- 172.21.15.40:0/1637730952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ff8069370 con 0x7f4ff80b6220 2024-04-26T23:05:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.997+0000 7f4feea00640 1 -- 172.21.15.40:0/1637730952 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4fe801b070 con 0x7f4ff80b6220 2024-04-26T23:05:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.998+0000 7f4feea00640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4fd406c8f0 0x7f4fd406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.998+0000 7f4feea00640 1 -- 172.21.15.40:0/1637730952 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5507+0+0 (secure 0 0 0) 0x7f4fe808ad50 con 0x7f4ff80b6220 2024-04-26T23:05:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:21.999+0000 7f4ff6000640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4fd406c8f0 0x7f4fd406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.000+0000 7f4ff6000640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4fd406c8f0 0x7f4fd406edb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8158a50 tx=0x7f4fe0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:22.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.004+0000 7f4feea00640 1 -- 172.21.15.40:0/1637730952 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4fe8059230 con 0x7f4ff80b6220 2024-04-26T23:05:22.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.206+0000 7f4ff7e00640 1 -- 172.21.15.40:0/1637730952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f4ff80d30c0 con 0x7f4ff80b6220 2024-04-26T23:05:22.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.214+0000 7f4feea00640 1 -- 172.21.15.40:0/1637730952 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7f4fe8058b30 con 0x7f4ff80b6220 2024-04-26T23:05:22.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.217+0000 7f4ff7e00640 1 -- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4fd406c8f0 msgr2=0x7f4fd406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:22.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.217+0000 7f4ff7e00640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4fd406c8f0 0x7f4fd406edb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8158a50 tx=0x7f4fe0008040 comp rx=0 tx=0).stop 2024-04-26T23:05:22.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.217+0000 7f4ff7e00640 1 -- 172.21.15.40:0/1637730952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff80b6220 msgr2=0x7f4ff814f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:22.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.217+0000 7f4ff7e00640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff80b6220 0x7f4ff814f030 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800c460 tx=0x7f4fe800c920 comp rx=0 tx=0).stop 2024-04-26T23:05:22.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.218+0000 7f4ff7e00640 1 -- 172.21.15.40:0/1637730952 shutdown_connections 2024-04-26T23:05:22.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.218+0000 7f4ff7e00640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4fd406c8f0 0x7f4fd406edb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:22.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.218+0000 7f4ff7e00640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ff80d32e0 0x7f4ff815f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:22.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.218+0000 7f4ff7e00640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ff80b8d20 0x7f4ff81f4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:22.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.218+0000 7f4ff7e00640 1 --2- 172.21.15.40:0/1637730952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff80b6220 0x7f4ff814f030 secure :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800c460 tx=0x7f4fe800c920 comp rx=0 tx=0).stop 2024-04-26T23:05:22.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.218+0000 7f4ff7e00640 1 -- 172.21.15.40:0/1637730952 >> 172.21.15.40:0/1637730952 conn(0x7f4ff80d4d90 msgr2=0x7f4ff806e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:22.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.218+0000 7f4ff7e00640 1 -- 172.21.15.40:0/1637730952 shutdown_connections 2024-04-26T23:05:22.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.218+0000 7f4ff7e00640 1 -- 172.21.15.40:0/1637730952 wait complete. 2024-04-26T23:05:22.238 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_abort_mds0' 2024-04-26T23:05:22.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.445+0000 7f4ca5000640 1 Processor -- start 2024-04-26T23:05:22.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.445+0000 7f4ca5000640 1 -- start start 2024-04-26T23:05:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.447+0000 7f4ca5000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca0145970 0x7f4ca01612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.447+0000 7f4ca5000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ca00b59f0 con 0x7f4ca0146060 2024-04-26T23:05:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.447+0000 7f4ca5000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ca01617f0 con 0x7f4ca00d06b0 2024-04-26T23:05:22.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.447+0000 7f4ca5000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca00ceed0 con 0x7f4ca0145970 2024-04-26T23:05:22.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.447+0000 7f4c9ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4ca0146060 0x7f4ca00b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50610/0 (socket says 172.21.15.40:50610) 2024-04-26T23:05:22.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.447+0000 7f4c9ea00640 1 -- 172.21.15.40:0/984737212 learned_addr learned my addr 172.21.15.40:0/984737212 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:22.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.448+0000 7f4c9f400640 1 --2- 172.21.15.40:0/984737212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca0145970 0x7f4ca01612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:22.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.448+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 88236336 0 0) 0x7f4ca00b59f0 con 0x7f4ca0146060 2024-04-26T23:05:22.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.448+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c84003520 con 0x7f4ca0146060 2024-04-26T23:05:22.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.448+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2370309767 0 0) 0x7f4ca01617f0 con 0x7f4ca00d06b0 2024-04-26T23:05:22.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.448+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ca00b59f0 con 0x7f4ca00d06b0 2024-04-26T23:05:22.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.448+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3204273351 0 0) 0x7f4c84003520 con 0x7f4ca0146060 2024-04-26T23:05:22.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.448+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca0145970 msgr2=0x7f4ca01612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:22.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.448+0000 7f4c9e000640 1 --2- 172.21.15.40:0/984737212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca0145970 0x7f4ca01612b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:22.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.449+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 >> v1:172.21.15.103:6789/0 conn(0x7f4ca00d06b0 legacy=0x7f4ca00cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.449+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca00b4180 con 0x7f4ca0146060 2024-04-26T23:05:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.449+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f4c8c001670 con 0x7f4ca0146060 2024-04-26T23:05:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.449+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 >> v1:172.21.15.40:6789/0 conn(0x7f4ca0146060 legacy=0x7f4ca00b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.449+0000 7f4c9e000640 1 --2- 172.21.15.40:0/984737212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca0145970 0x7f4c84003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.449+0000 7f4c9e000640 1 --2- 172.21.15.40:0/984737212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c840040d0 0x7f4c84004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.449+0000 7f4c9e000640 1 --2- 172.21.15.40:0/984737212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c84004d40 0x7f4c840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.449+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8400ba70 con 0x7f4c84004d40 2024-04-26T23:05:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.449+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8400bc20 con 0x7f4c840040d0 2024-04-26T23:05:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.449+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4c8400be20 con 0x7f4ca0145970 2024-04-26T23:05:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.449+0000 7f4c9fe00640 1 --2- 172.21.15.40:0/984737212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c84004d40 0x7f4c840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.450+0000 7f4c9ea00640 1 --2- 172.21.15.40:0/984737212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c840040d0 0x7f4c84004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.450+0000 7f4c9f400640 1 --2- 172.21.15.40:0/984737212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca0145970 0x7f4c84003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.450+0000 7f4c9fe00640 1 -- 172.21.15.40:0/984737212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca0145970 msgr2=0x7f4c84003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:22.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.450+0000 7f4c9fe00640 1 --2- 172.21.15.40:0/984737212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca0145970 0x7f4c84003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:22.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.450+0000 7f4c9fe00640 1 -- 172.21.15.40:0/984737212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c840040d0 msgr2=0x7f4c84004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:22.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.450+0000 7f4c9fe00640 1 --2- 172.21.15.40:0/984737212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c840040d0 0x7f4c84004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:22.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.450+0000 7f4c9fe00640 1 -- 172.21.15.40:0/984737212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c84003520 con 0x7f4c84004d40 2024-04-26T23:05:22.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.451+0000 7f4c9fe00640 1 --2- 172.21.15.40:0/984737212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c84004d40 0x7f4c840051c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f4c98001fd0 tx=0x7f4c84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ffe998b086f1d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:22.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.451+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c98001bf0 con 0x7f4c84004d40 2024-04-26T23:05:22.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.451+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c98003b70 con 0x7f4c84004d40 2024-04-26T23:05:22.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.451+0000 7f4c9e000640 1 -- 172.21.15.40:0/984737212 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c980050d0 con 0x7f4c84004d40 2024-04-26T23:05:22.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.452+0000 7f4ca5000640 1 -- 172.21.15.40:0/984737212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c84004d40 msgr2=0x7f4c840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:22.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.452+0000 7f4ca5000640 1 --2- 172.21.15.40:0/984737212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c84004d40 0x7f4c840051c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f4c98001fd0 tx=0x7f4c84005830 comp rx=0 tx=0).stop 2024-04-26T23:05:22.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.452+0000 7f4c9fe00640 1 -- 172.21.15.40:0/984737212 reap_dead start 2024-04-26T23:05:22.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.452+0000 7f4ca5000640 1 -- 172.21.15.40:0/984737212 shutdown_connections 2024-04-26T23:05:22.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.452+0000 7f4ca5000640 1 -- 172.21.15.40:0/984737212 >> 172.21.15.40:0/984737212 conn(0x7f4ca0075eb0 msgr2=0x7f4ca0068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:22.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.452+0000 7f4ca5000640 1 -- 172.21.15.40:0/984737212 shutdown_connections 2024-04-26T23:05:22.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.452+0000 7f4ca5000640 1 -- 172.21.15.40:0/984737212 wait complete. 2024-04-26T23:05:22.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.454+0000 7f4ca5000640 1 Processor -- start 2024-04-26T23:05:22.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.454+0000 7f4ca5000640 1 -- start start 2024-04-26T23:05:22.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.455+0000 7f4ca5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c84004d40 0x7f4ca00acc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:22.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.455+0000 7f4ca5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca015c4b0 0x7f4ca013df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:22.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.455+0000 7f4ca5000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca00ae410 0x7f4ca00aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:22.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.455+0000 7f4ca5000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca00cf100 con 0x7f4ca015c4b0 2024-04-26T23:05:22.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.455+0000 7f4ca5000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca00b5c60 con 0x7f4c84004d40 2024-04-26T23:05:22.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.455+0000 7f4ca5000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca00cf3e0 con 0x7f4ca00ae410 2024-04-26T23:05:22.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.455+0000 7f4c9ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca015c4b0 0x7f4ca013df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:22.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.455+0000 7f4c9ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca015c4b0 0x7f4ca013df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44954/0 (socket says 172.21.15.40:44954) 2024-04-26T23:05:22.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.455+0000 7f4c9ea00640 1 -- 172.21.15.40:0/3853971909 learned_addr learned my addr 172.21.15.40:0/3853971909 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:22.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.456+0000 7f4c9fe00640 1 --2- 172.21.15.40:0/3853971909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca00ae410 0x7f4ca00aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:22.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.456+0000 7f4c9ea00640 1 -- 172.21.15.40:0/3853971909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca00ae410 msgr2=0x7f4ca00aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:22.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.456+0000 7f4c9ea00640 1 --2- 172.21.15.40:0/3853971909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca00ae410 0x7f4ca00aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:22.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.456+0000 7f4c9ea00640 1 -- 172.21.15.40:0/3853971909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c84004d40 msgr2=0x7f4ca00acc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:22.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.456+0000 7f4c9ea00640 1 --2- 172.21.15.40:0/3853971909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c84004d40 0x7f4ca00acc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:22.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.456+0000 7f4c9ea00640 1 -- 172.21.15.40:0/3853971909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca013b5d0 con 0x7f4ca015c4b0 2024-04-26T23:05:22.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.456+0000 7f4c9ea00640 1 --2- 172.21.15.40:0/3853971909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca015c4b0 0x7f4ca013df40 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00e7e0 tx=0x7f4c8c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:22.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.456+0000 7f4c97400640 1 -- 172.21.15.40:0/3853971909 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c00c8d0 con 0x7f4ca015c4b0 2024-04-26T23:05:22.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.456+0000 7f4c97400640 1 -- 172.21.15.40:0/3853971909 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c8c018070 con 0x7f4ca015c4b0 2024-04-26T23:05:22.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.456+0000 7f4c97400640 1 -- 172.21.15.40:0/3853971909 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c00ccb0 con 0x7f4ca015c4b0 2024-04-26T23:05:22.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.456+0000 7f4ca5000640 1 -- 172.21.15.40:0/3853971909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca013b8b0 con 0x7f4ca015c4b0 2024-04-26T23:05:22.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.456+0000 7f4ca5000640 1 -- 172.21.15.40:0/3853971909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca013be70 con 0x7f4ca015c4b0 2024-04-26T23:05:22.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.458+0000 7f4ca5000640 1 -- 172.21.15.40:0/3853971909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca00cdc10 con 0x7f4ca015c4b0 2024-04-26T23:05:22.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.459+0000 7f4c97400640 1 -- 172.21.15.40:0/3853971909 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4c8c01d460 con 0x7f4ca015c4b0 2024-04-26T23:05:22.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.460+0000 7f4c97400640 1 --2- 172.21.15.40:0/3853971909 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4c8006c8f0 0x7f4c8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:22.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.460+0000 7f4c97400640 1 -- 172.21.15.40:0/3853971909 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5507+0+0 (secure 0 0 0) 0x7f4c8c08ba40 con 0x7f4ca015c4b0 2024-04-26T23:05:22.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.461+0000 7f4c9f400640 1 --2- 172.21.15.40:0/3853971909 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4c8006c8f0 0x7f4c8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:22.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.462+0000 7f4c9f400640 1 --2- 172.21.15.40:0/3853971909 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4c8006c8f0 0x7f4c8006edb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009910 tx=0x7f4c90008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:22.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.465+0000 7f4c97400640 1 -- 172.21.15.40:0/3853971909 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4ca00cdc10 con 0x7f4ca015c4b0 2024-04-26T23:05:22.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:22.745+0000 7f4ca5000640 1 -- 172.21.15.40:0/3853971909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_abort_mds0"]} v 0) -- 0x7f4ca0069370 con 0x7f4ca015c4b0 2024-04-26T23:05:23.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.636+0000 7f4c97400640 1 -- 172.21.15.40:0/3853971909 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_abort_mds0"]}]=0 v238) ==== 143+0+0 (secure 0 0 0) 0x7f4c8c059f20 con 0x7f4ca015c4b0 2024-04-26T23:05:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.640+0000 7f4ca5000640 1 -- 172.21.15.40:0/3853971909 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4c8006c8f0 msgr2=0x7f4c8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.640+0000 7f4ca5000640 1 --2- 172.21.15.40:0/3853971909 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4c8006c8f0 0x7f4c8006edb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009910 tx=0x7f4c90008040 comp rx=0 tx=0).stop 2024-04-26T23:05:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.640+0000 7f4ca5000640 1 -- 172.21.15.40:0/3853971909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca015c4b0 msgr2=0x7f4ca013df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.640+0000 7f4ca5000640 1 --2- 172.21.15.40:0/3853971909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca015c4b0 0x7f4ca013df40 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00e7e0 tx=0x7f4c8c00eca0 comp rx=0 tx=0).stop 2024-04-26T23:05:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.640+0000 7f4ca5000640 1 -- 172.21.15.40:0/3853971909 shutdown_connections 2024-04-26T23:05:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.640+0000 7f4ca5000640 1 --2- 172.21.15.40:0/3853971909 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4c8006c8f0 0x7f4c8006edb0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.640+0000 7f4ca5000640 1 --2- 172.21.15.40:0/3853971909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca00ae410 0x7f4ca00aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.640+0000 7f4ca5000640 1 --2- 172.21.15.40:0/3853971909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca015c4b0 0x7f4ca013df40 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.640+0000 7f4ca5000640 1 --2- 172.21.15.40:0/3853971909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c84004d40 0x7f4ca00acc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:23.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.640+0000 7f4ca5000640 1 -- 172.21.15.40:0/3853971909 >> 172.21.15.40:0/3853971909 conn(0x7f4ca0075eb0 msgr2=0x7f4ca0068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:23.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.641+0000 7f4ca5000640 1 -- 172.21.15.40:0/3853971909 shutdown_connections 2024-04-26T23:05:23.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.641+0000 7f4ca5000640 1 -- 172.21.15.40:0/3853971909 wait complete. 2024-04-26T23:05:23.663 INFO:tasks.cephfs_test_runner:test_scrub_abort_mds0 (tasks.cephfs.test_multimds_misc.TestScrub2) ... ok 2024-04-26T23:05:23.663 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_backtrace (tasks.cephfs.test_multimds_misc.TestScrub2) 2024-04-26T23:05:23.663 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:05:23.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.878+0000 7f6080400640 1 Processor -- start 2024-04-26T23:05:23.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.879+0000 7f6080400640 1 -- start start 2024-04-26T23:05:23.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.880+0000 7f6080400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6078075880 0x7f607815b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:23.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.880+0000 7f6080400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607815b820 con 0x7f607815fac0 2024-04-26T23:05:23.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.880+0000 7f6080400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607806f5a0 con 0x7f60780cf170 2024-04-26T23:05:23.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.880+0000 7f6080400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f607806f350 con 0x7f6078075880 2024-04-26T23:05:23.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.881+0000 7f607d000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f607815fac0 0x7f60780b01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50624/0 (socket says 172.21.15.40:50624) 2024-04-26T23:05:23.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.881+0000 7f607d000640 1 -- 172.21.15.40:0/326274735 learned_addr learned my addr 172.21.15.40:0/326274735 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:23.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.881+0000 7f607da00640 1 --2- 172.21.15.40:0/326274735 >> v2:172.21.15.103:3300/0 conn(0x7f6078075880 0x7f607815b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.881+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1741042655 0 0) 0x7f607815b820 con 0x7f607815fac0 2024-04-26T23:05:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.881+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6060003520 con 0x7f607815fac0 2024-04-26T23:05:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 662986884 0 0) 0x7f6060003520 con 0x7f607815fac0 2024-04-26T23:05:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 >> v2:172.21.15.103:3300/0 conn(0x7f6078075880 msgr2=0x7f607815b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f6077e00640 1 --2- 172.21.15.40:0/326274735 >> v2:172.21.15.103:3300/0 conn(0x7f6078075880 0x7f607815b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 >> v1:172.21.15.103:6789/0 conn(0x7f60780cf170 legacy=0x7f60780cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607806f880 con 0x7f607815fac0 2024-04-26T23:05:23.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f6068001670 con 0x7f607815fac0 2024-04-26T23:05:23.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 >> v1:172.21.15.40:6789/0 conn(0x7f607815fac0 legacy=0x7f60780b01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:23.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f6077e00640 1 --2- 172.21.15.40:0/326274735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6060003a00 0x7f6060003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:23.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f6077e00640 1 --2- 172.21.15.40:0/326274735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6060004800 0x7f6060004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:23.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f6077e00640 1 --2- 172.21.15.40:0/326274735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6060005210 0x7f606000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:23.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f607806f350 con 0x7f6060004800 2024-04-26T23:05:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f606000e370 con 0x7f6060005210 2024-04-26T23:05:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f606000e570 con 0x7f6060003a00 2024-04-26T23:05:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.882+0000 7f607d000640 1 --2- 172.21.15.40:0/326274735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6060004800 0x7f6060004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.883+0000 7f607d000640 1 -- 172.21.15.40:0/326274735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6060003a00 msgr2=0x7f6060003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.883+0000 7f607da00640 1 --2- 172.21.15.40:0/326274735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6060003a00 0x7f6060003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.883+0000 7f607d000640 1 --2- 172.21.15.40:0/326274735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6060003a00 0x7f6060003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.883+0000 7f607d000640 1 -- 172.21.15.40:0/326274735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6060005210 msgr2=0x7f606000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.883+0000 7f607d000640 1 --2- 172.21.15.40:0/326274735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6060005210 0x7f606000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.883+0000 7f607d000640 1 -- 172.21.15.40:0/326274735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6060003520 con 0x7f6060004800 2024-04-26T23:05:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.883+0000 7f607e400640 1 -- 172.21.15.40:0/326274735 reap_dead start 2024-04-26T23:05:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.884+0000 7f607d000640 1 --2- 172.21.15.40:0/326274735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6060004800 0x7f6060004c80 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6068001050 tx=0x7f6060005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcaba84b60c0170b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.884+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068007c60 con 0x7f6060004800 2024-04-26T23:05:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.884+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6068005af0 con 0x7f6060004800 2024-04-26T23:05:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.884+0000 7f6077e00640 1 -- 172.21.15.40:0/326274735 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068007070 con 0x7f6060004800 2024-04-26T23:05:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.885+0000 7f6080400640 1 -- 172.21.15.40:0/326274735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6060004800 msgr2=0x7f6060004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.885+0000 7f6080400640 1 --2- 172.21.15.40:0/326274735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6060004800 0x7f6060004c80 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6068001050 tx=0x7f6060005830 comp rx=0 tx=0).stop 2024-04-26T23:05:23.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.885+0000 7f6080400640 1 -- 172.21.15.40:0/326274735 shutdown_connections 2024-04-26T23:05:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.885+0000 7f6080400640 1 --2- 172.21.15.40:0/326274735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6060004800 0x7f6060004c80 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.885+0000 7f6080400640 1 -- 172.21.15.40:0/326274735 >> 172.21.15.40:0/326274735 conn(0x7f60780d0fd0 msgr2=0x7f60780d13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.885+0000 7f6080400640 1 -- 172.21.15.40:0/326274735 shutdown_connections 2024-04-26T23:05:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.885+0000 7f6080400640 1 -- 172.21.15.40:0/326274735 wait complete. 2024-04-26T23:05:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.887+0000 7f6080400640 1 Processor -- start 2024-04-26T23:05:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.887+0000 7f6080400640 1 -- start start 2024-04-26T23:05:23.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f6080400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6060004800 0x7f60780b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:23.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f6080400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607814d700 0x7f607814f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:23.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f6080400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f607814fc50 0x7f6078152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:23.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f6080400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f607815b460 con 0x7f607814fc50 2024-04-26T23:05:23.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f6080400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f607815b2e0 con 0x7f607814d700 2024-04-26T23:05:23.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f6080400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f607815b5e0 con 0x7f6060004800 2024-04-26T23:05:23.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f607e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f607814fc50 0x7f6078152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:23.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f607e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f607814fc50 0x7f6078152c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44970/0 (socket says 172.21.15.40:44970) 2024-04-26T23:05:23.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f607e400640 1 -- 172.21.15.40:0/1519818994 learned_addr learned my addr 172.21.15.40:0/1519818994 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:23.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f607d000640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607814d700 0x7f607814f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:23.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f607e400640 1 -- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6060004800 msgr2=0x7f60780b3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:23.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f607da00640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6060004800 0x7f60780b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:23.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f607e400640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6060004800 0x7f60780b3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:23.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f607e400640 1 -- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607814d700 msgr2=0x7f607814f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:23.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.888+0000 7f607e400640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607814d700 0x7f607814f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:23.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.889+0000 7f607e400640 1 -- 172.21.15.40:0/1519818994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60780b4190 con 0x7f607814fc50 2024-04-26T23:05:23.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.889+0000 7f607d000640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607814d700 0x7f607814f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:23.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.889+0000 7f607e400640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f607814fc50 0x7f6078152c10 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f6078070320 tx=0x7f607000c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:23.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.889+0000 7f6075600640 1 -- 172.21.15.40:0/1519818994 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070017070 con 0x7f607814fc50 2024-04-26T23:05:23.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.889+0000 7f6075600640 1 -- 172.21.15.40:0/1519818994 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6070007070 con 0x7f607814fc50 2024-04-26T23:05:23.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.889+0000 7f6075600640 1 -- 172.21.15.40:0/1519818994 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070007460 con 0x7f607814fc50 2024-04-26T23:05:23.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.889+0000 7f6080400640 1 -- 172.21.15.40:0/1519818994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60781500d0 con 0x7f607814fc50 2024-04-26T23:05:23.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.889+0000 7f6080400640 1 -- 172.21.15.40:0/1519818994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60780c9570 con 0x7f607814fc50 2024-04-26T23:05:23.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.891+0000 7f6080400640 1 -- 172.21.15.40:0/1519818994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60780694f0 con 0x7f607814fc50 2024-04-26T23:05:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.892+0000 7f6075600640 1 -- 172.21.15.40:0/1519818994 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f607001a070 con 0x7f607814fc50 2024-04-26T23:05:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.893+0000 7f6075600640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f605c06c920 0x7f605c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.893+0000 7f6075600640 1 -- 172.21.15.40:0/1519818994 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5507+0+0 (secure 0 0 0) 0x7f607008acf0 con 0x7f607814fc50 2024-04-26T23:05:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.894+0000 7f607da00640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f605c06c920 0x7f605c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.895+0000 7f607da00640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f605c06c920 0x7f605c06ede0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f607806b230 tx=0x7f606c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:23.897+0000 7f6075600640 1 -- 172.21.15.40:0/1519818994 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f60780b4190 con 0x7f607814fc50 2024-04-26T23:05:24.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.103+0000 7f6080400640 1 -- 172.21.15.40:0/1519818994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f60780d1ac0 con 0x7f607814fc50 2024-04-26T23:05:24.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.105+0000 7f6075600640 1 -- 172.21.15.40:0/1519818994 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+13515 (secure 0 0 0) 0x7f60700591d0 con 0x7f607814fc50 2024-04-26T23:05:24.106 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:24.106 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":57,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:05:21.613884+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6837/670995452":"2024-04-27T23:05:18.253804+0000","172.21.15.40:6834/2304583125":"2024-04-27T23:05:18.253836+0000","172.21.15.40:6832/2304583125":"2024-04-27T23:05:18.253836+0000","172.21.15.103:6836/670995452":"2024-04-27T23:05:18.253804+0000","172.21.15.40:6835/618018110":"2024-04-27T23:05:18.253850+0000","172.21.15.40:6833/618018110":"2024-04-27T23:05:18.253850+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:05:24.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.107+0000 7f6080400640 1 -- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f605c06c920 msgr2=0x7f605c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:24.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.107+0000 7f6080400640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f605c06c920 0x7f605c06ede0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f607806b230 tx=0x7f606c008040 comp rx=0 tx=0).stop 2024-04-26T23:05:24.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.108+0000 7f6080400640 1 -- 172.21.15.40:0/1519818994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f607814fc50 msgr2=0x7f6078152c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:24.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.108+0000 7f6080400640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f607814fc50 0x7f6078152c10 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f6078070320 tx=0x7f607000c980 comp rx=0 tx=0).stop 2024-04-26T23:05:24.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.108+0000 7f6080400640 1 -- 172.21.15.40:0/1519818994 shutdown_connections 2024-04-26T23:05:24.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.108+0000 7f6080400640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f605c06c920 0x7f605c06ede0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.108+0000 7f6080400640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f607814fc50 0x7f6078152c10 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.108+0000 7f6080400640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607814d700 0x7f607814f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.108+0000 7f6080400640 1 --2- 172.21.15.40:0/1519818994 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6060004800 0x7f60780b3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.108+0000 7f6080400640 1 -- 172.21.15.40:0/1519818994 >> 172.21.15.40:0/1519818994 conn(0x7f60780d0fd0 msgr2=0x7f60780d13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:24.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.108+0000 7f6080400640 1 -- 172.21.15.40:0/1519818994 shutdown_connections 2024-04-26T23:05:24.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.108+0000 7f6080400640 1 -- 172.21.15.40:0/1519818994 wait complete. 2024-04-26T23:05:24.127 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:05:24.128 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:05:24.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.347+0000 7fad5c400640 1 Processor -- start 2024-04-26T23:05:24.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.347+0000 7fad5c400640 1 -- start start 2024-04-26T23:05:24.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.348+0000 7fad5c400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad54071e60 0x7fad5406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:24.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.348+0000 7fad5c400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fad5406fe20 0x7fad54072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:24.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.348+0000 7fad5c400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad5415d7c0 con 0x7fad54072780 2024-04-26T23:05:24.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.348+0000 7fad5c400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fad5415daa0 con 0x7fad54071e60 2024-04-26T23:05:24.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.348+0000 7fad5c400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fad5415dd80 con 0x7fad5406fe20 2024-04-26T23:05:24.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.349+0000 7fad59a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad54071e60 0x7fad5406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:24.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.349+0000 7fad5a400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fad54072780 0x7fad540cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50628/0 (socket says 172.21.15.40:50628) 2024-04-26T23:05:24.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.349+0000 7fad5a400640 1 -- 172.21.15.40:0/3188546843 learned_addr learned my addr 172.21.15.40:0/3188546843 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:24.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.349+0000 7fad59000640 1 --2- 172.21.15.40:0/3188546843 >> v2:172.21.15.103:3300/0 conn(0x7fad5406fe20 0x7fad54072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.349+0000 7fad59a00640 1 -- 172.21.15.40:0/3188546843 >> v2:172.21.15.103:3300/0 conn(0x7fad5406fe20 msgr2=0x7fad54072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.349+0000 7fad59a00640 1 --2- 172.21.15.40:0/3188546843 >> v2:172.21.15.103:3300/0 conn(0x7fad5406fe20 0x7fad54072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.349+0000 7fad59a00640 1 -- 172.21.15.40:0/3188546843 >> v1:172.21.15.40:6789/0 conn(0x7fad54072780 legacy=0x7fad540cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.349+0000 7fad59a00640 1 -- 172.21.15.40:0/3188546843 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5415e060 con 0x7fad54071e60 2024-04-26T23:05:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.349+0000 7fad53e00640 1 -- 172.21.15.40:0/3188546843 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3611484715 0 0) 0x7fad5415d7c0 con 0x7fad54072780 2024-04-26T23:05:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.351+0000 7fad59a00640 1 --2- 172.21.15.40:0/3188546843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad54071e60 0x7fad5406f8e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fad540b3820 tx=0x7fad4402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=308a2c1c3c83c92e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.351+0000 7fad53e00640 1 -- 172.21.15.40:0/3188546843 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44037070 con 0x7fad54071e60 2024-04-26T23:05:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.351+0000 7fad53e00640 1 -- 172.21.15.40:0/3188546843 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad44032440 con 0x7fad54071e60 2024-04-26T23:05:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.351+0000 7fad53e00640 1 -- 172.21.15.40:0/3188546843 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44032810 con 0x7fad54071e60 2024-04-26T23:05:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.351+0000 7fad5c400640 1 -- 172.21.15.40:0/3188546843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad54071e60 msgr2=0x7fad5406f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.351+0000 7fad5c400640 1 --2- 172.21.15.40:0/3188546843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad54071e60 0x7fad5406f8e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fad540b3820 tx=0x7fad4402a1a0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.352+0000 7fad5c400640 1 -- 172.21.15.40:0/3188546843 shutdown_connections 2024-04-26T23:05:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.352+0000 7fad5c400640 1 --2- 172.21.15.40:0/3188546843 >> v2:172.21.15.103:3300/0 conn(0x7fad5406fe20 0x7fad54072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.352+0000 7fad5c400640 1 --2- 172.21.15.40:0/3188546843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad54071e60 0x7fad5406f8e0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.352+0000 7fad5c400640 1 -- 172.21.15.40:0/3188546843 >> 172.21.15.40:0/3188546843 conn(0x7fad540d0be0 msgr2=0x7fad540d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.352+0000 7fad5c400640 1 -- 172.21.15.40:0/3188546843 shutdown_connections 2024-04-26T23:05:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.352+0000 7fad5c400640 1 -- 172.21.15.40:0/3188546843 wait complete. 2024-04-26T23:05:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.353+0000 7fad5c400640 1 Processor -- start 2024-04-26T23:05:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.354+0000 7fad5c400640 1 -- start start 2024-04-26T23:05:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.355+0000 7fad5c400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad5406fe20 0x7fad541f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.355+0000 7fad5c400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad54071e60 0x7fad541f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.355+0000 7fad5c400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad54072780 0x7fad540b1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.355+0000 7fad5c400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fad5415d3f0 con 0x7fad54072780 2024-04-26T23:05:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.355+0000 7fad5c400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fad54072bd0 con 0x7fad5406fe20 2024-04-26T23:05:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.355+0000 7fad5c400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fad5415d570 con 0x7fad54071e60 2024-04-26T23:05:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.355+0000 7fad5a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad54072780 0x7fad540b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.355+0000 7fad5a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad54072780 0x7fad540b1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44998/0 (socket says 172.21.15.40:44998) 2024-04-26T23:05:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.355+0000 7fad5a400640 1 -- 172.21.15.40:0/2898531217 learned_addr learned my addr 172.21.15.40:0/2898531217 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.355+0000 7fad59000640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad54071e60 0x7fad541f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.355+0000 7fad59a00640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad5406fe20 0x7fad541f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.356+0000 7fad5a400640 1 -- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad54071e60 msgr2=0x7fad541f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.356+0000 7fad5a400640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad54071e60 0x7fad541f1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.356+0000 7fad5a400640 1 -- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad5406fe20 msgr2=0x7fad541f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.356+0000 7fad5a400640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad5406fe20 0x7fad541f6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.356+0000 7fad5a400640 1 -- 172.21.15.40:0/2898531217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad541f6aa0 con 0x7fad54072780 2024-04-26T23:05:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.356+0000 7fad59a00640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad5406fe20 0x7fad541f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.356+0000 7fad5a400640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad54072780 0x7fad540b1860 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00c700 tx=0x7fad4c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.356+0000 7fad51600640 1 -- 172.21.15.40:0/2898531217 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4c018070 con 0x7fad54072780 2024-04-26T23:05:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.356+0000 7fad51600640 1 -- 172.21.15.40:0/2898531217 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad4c00ce10 con 0x7fad54072780 2024-04-26T23:05:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.356+0000 7fad51600640 1 -- 172.21.15.40:0/2898531217 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4c00a860 con 0x7fad54072780 2024-04-26T23:05:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.356+0000 7fad5c400640 1 -- 172.21.15.40:0/2898531217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad540c9080 con 0x7fad54072780 2024-04-26T23:05:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.356+0000 7fad5c400640 1 -- 172.21.15.40:0/2898531217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad540c9640 con 0x7fad54072780 2024-04-26T23:05:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.358+0000 7fad5c400640 1 -- 172.21.15.40:0/2898531217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad540694f0 con 0x7fad54072780 2024-04-26T23:05:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.359+0000 7fad51600640 1 -- 172.21.15.40:0/2898531217 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fad4c01b070 con 0x7fad54072780 2024-04-26T23:05:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.360+0000 7fad51600640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fad3806c8f0 0x7fad3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.360+0000 7fad51600640 1 -- 172.21.15.40:0/2898531217 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5507+0+0 (secure 0 0 0) 0x7fad4c08aff0 con 0x7fad54072780 2024-04-26T23:05:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.361+0000 7fad59a00640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fad3806c8f0 0x7fad3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.362+0000 7fad59a00640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fad3806c8f0 0x7fad3806edb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fad44034000 tx=0x7fad4402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.364+0000 7fad51600640 1 -- 172.21.15.40:0/2898531217 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fad541f6aa0 con 0x7fad54072780 2024-04-26T23:05:24.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.568+0000 7fad5c400640 1 -- 172.21.15.40:0/2898531217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fad5414eb40 con 0x7fad54072780 2024-04-26T23:05:24.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.569+0000 7fad51600640 1 -- 172.21.15.40:0/2898531217 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7fad4c0594f0 con 0x7fad54072780 2024-04-26T23:05:24.569 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:05:24.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.572+0000 7fad5c400640 1 -- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fad3806c8f0 msgr2=0x7fad3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:24.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.572+0000 7fad5c400640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fad3806c8f0 0x7fad3806edb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fad44034000 tx=0x7fad4402e000 comp rx=0 tx=0).stop 2024-04-26T23:05:24.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.572+0000 7fad5c400640 1 -- 172.21.15.40:0/2898531217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad54072780 msgr2=0x7fad540b1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.572+0000 7fad5c400640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad54072780 0x7fad540b1860 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00c700 tx=0x7fad4c00cbc0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.572+0000 7fad5c400640 1 -- 172.21.15.40:0/2898531217 shutdown_connections 2024-04-26T23:05:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.572+0000 7fad5c400640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fad3806c8f0 0x7fad3806edb0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.572+0000 7fad5c400640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad54072780 0x7fad540b1860 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.572+0000 7fad5c400640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad54071e60 0x7fad541f1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.572+0000 7fad5c400640 1 --2- 172.21.15.40:0/2898531217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad5406fe20 0x7fad541f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.572+0000 7fad5c400640 1 -- 172.21.15.40:0/2898531217 >> 172.21.15.40:0/2898531217 conn(0x7fad540d0be0 msgr2=0x7fad540cb330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.572+0000 7fad5c400640 1 -- 172.21.15.40:0/2898531217 shutdown_connections 2024-04-26T23:05:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.572+0000 7fad5c400640 1 -- 172.21.15.40:0/2898531217 wait complete. 2024-04-26T23:05:24.591 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_backtrace' 2024-04-26T23:05:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.799+0000 7f0b22600640 1 Processor -- start 2024-04-26T23:05:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.800+0000 7f0b22600640 1 -- start start 2024-04-26T23:05:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.801+0000 7f0b22600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0b1c0cc480 0x7f0b1c072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.801+0000 7f0b22600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b1c15d7c0 con 0x7f0b1c072780 2024-04-26T23:05:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.801+0000 7f0b22600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b1c15daa0 con 0x7f0b1c071e60 2024-04-26T23:05:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.801+0000 7f0b22600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f0b1c15dd80 con 0x7f0b1c0cc480 2024-04-26T23:05:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.801+0000 7f0b1aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0b1c0cc480 0x7f0b1c072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.802+0000 7f0b1be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0b1c072780 0x7f0b1c166ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50644/0 (socket says 172.21.15.40:50644) 2024-04-26T23:05:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.802+0000 7f0b1be00640 1 -- 172.21.15.40:0/2476031828 learned_addr learned my addr 172.21.15.40:0/2476031828 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.802+0000 7f0b1aa00640 1 -- 172.21.15.40:0/2476031828 >> v1:172.21.15.103:6789/0 conn(0x7f0b1c071e60 legacy=0x7f0b1c06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.802+0000 7f0b1aa00640 1 -- 172.21.15.40:0/2476031828 >> v1:172.21.15.40:6789/0 conn(0x7f0b1c072780 legacy=0x7f0b1c166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.802+0000 7f0b1aa00640 1 -- 172.21.15.40:0/2476031828 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1c15e060 con 0x7f0b1c0cc480 2024-04-26T23:05:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.802+0000 7f0b1a000640 1 -- 172.21.15.40:0/2476031828 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2743242833 0 0) 0x7f0b1c15d7c0 con 0x7f0b1c072780 2024-04-26T23:05:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.803+0000 7f0b1aa00640 1 --2- 172.21.15.40:0/2476031828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c0cc480 0x7f0b1c072240 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0b040097a0 tx=0x7f0b0402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9d51ac1c486389 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.804+0000 7f0b1a000640 1 -- 172.21.15.40:0/2476031828 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b04037070 con 0x7f0b1c0cc480 2024-04-26T23:05:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.804+0000 7f0b1a000640 1 -- 172.21.15.40:0/2476031828 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b04032440 con 0x7f0b1c0cc480 2024-04-26T23:05:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.804+0000 7f0b1a000640 1 -- 172.21.15.40:0/2476031828 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b04032810 con 0x7f0b1c0cc480 2024-04-26T23:05:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.804+0000 7f0b22600640 1 -- 172.21.15.40:0/2476031828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c0cc480 msgr2=0x7f0b1c072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.804+0000 7f0b22600640 1 --2- 172.21.15.40:0/2476031828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c0cc480 0x7f0b1c072240 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0b040097a0 tx=0x7f0b0402a230 comp rx=0 tx=0).stop 2024-04-26T23:05:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.804+0000 7f0b22600640 1 -- 172.21.15.40:0/2476031828 shutdown_connections 2024-04-26T23:05:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.804+0000 7f0b22600640 1 --2- 172.21.15.40:0/2476031828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c0cc480 0x7f0b1c072240 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.804+0000 7f0b22600640 1 -- 172.21.15.40:0/2476031828 >> 172.21.15.40:0/2476031828 conn(0x7f0b1c0d0be0 msgr2=0x7f0b1c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.804+0000 7f0b22600640 1 -- 172.21.15.40:0/2476031828 shutdown_connections 2024-04-26T23:05:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.804+0000 7f0b22600640 1 -- 172.21.15.40:0/2476031828 wait complete. 2024-04-26T23:05:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.806+0000 7f0b22600640 1 Processor -- start 2024-04-26T23:05:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.806+0000 7f0b22600640 1 -- start start 2024-04-26T23:05:24.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.807+0000 7f0b22600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b1c071e60 0x7f0b1c0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:24.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.807+0000 7f0b22600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c072780 0x7f0b1c1f8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:24.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.807+0000 7f0b22600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b1c0cc480 0x7f0b1c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:24.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.807+0000 7f0b22600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c15d3f0 con 0x7f0b1c072780 2024-04-26T23:05:24.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.807+0000 7f0b22600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c072bd0 con 0x7f0b1c0cc480 2024-04-26T23:05:24.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.807+0000 7f0b22600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0b1c15d570 con 0x7f0b1c071e60 2024-04-26T23:05:24.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.808+0000 7f0b1aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c072780 0x7f0b1c1f8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:24.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.808+0000 7f0b1aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c072780 0x7f0b1c1f8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45022/0 (socket says 172.21.15.40:45022) 2024-04-26T23:05:24.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.808+0000 7f0b1aa00640 1 -- 172.21.15.40:0/2397138327 learned_addr learned my addr 172.21.15.40:0/2397138327 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:24.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.808+0000 7f0b1be00640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b1c0cc480 0x7f0b1c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:24.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.808+0000 7f0b1b400640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b1c071e60 0x7f0b1c0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.808+0000 7f0b1aa00640 1 -- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b1c071e60 msgr2=0x7f0b1c0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.808+0000 7f0b1aa00640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b1c071e60 0x7f0b1c0b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.808+0000 7f0b1aa00640 1 -- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b1c0cc480 msgr2=0x7f0b1c0b60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.808+0000 7f0b1aa00640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b1c0cc480 0x7f0b1c0b60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.808+0000 7f0b1aa00640 1 -- 172.21.15.40:0/2397138327 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1c0b3c60 con 0x7f0b1c072780 2024-04-26T23:05:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.809+0000 7f0b1aa00640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c072780 0x7f0b1c1f8790 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0b04035040 tx=0x7f0b040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.809+0000 7f0b13400640 1 -- 172.21.15.40:0/2397138327 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b04040020 con 0x7f0b1c072780 2024-04-26T23:05:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.809+0000 7f0b13400640 1 -- 172.21.15.40:0/2397138327 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b04038040 con 0x7f0b1c072780 2024-04-26T23:05:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.809+0000 7f0b13400640 1 -- 172.21.15.40:0/2397138327 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b04037070 con 0x7f0b1c072780 2024-04-26T23:05:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.809+0000 7f0b22600640 1 -- 172.21.15.40:0/2397138327 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b1c0b3f40 con 0x7f0b1c072780 2024-04-26T23:05:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.809+0000 7f0b22600640 1 -- 172.21.15.40:0/2397138327 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b1c0b4500 con 0x7f0b1c072780 2024-04-26T23:05:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.810+0000 7f0b22600640 1 -- 172.21.15.40:0/2397138327 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b1c0d07c0 con 0x7f0b1c072780 2024-04-26T23:05:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.811+0000 7f0b13400640 1 -- 172.21.15.40:0/2397138327 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0b04043020 con 0x7f0b1c072780 2024-04-26T23:05:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.812+0000 7f0b13400640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0af806c8f0 0x7f0af806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.812+0000 7f0b13400640 1 -- 172.21.15.40:0/2397138327 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5507+0+0 (secure 0 0 0) 0x7f0b040ac220 con 0x7f0b1c072780 2024-04-26T23:05:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.813+0000 7f0b1b400640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0af806c8f0 0x7f0af806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:24.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.814+0000 7f0b1b400640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0af806c8f0 0x7f0af806edb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0012e0 tx=0x7f0b0c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:24.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:24.817+0000 7f0b13400640 1 -- 172.21.15.40:0/2397138327 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0b1c0d07c0 con 0x7f0b1c072780 2024-04-26T23:05:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.096+0000 7f0b22600640 1 -- 172.21.15.40:0/2397138327 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_backtrace"]} v 0) -- 0x7f0b1c0b24c0 con 0x7f0b1c072780 2024-04-26T23:05:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.651+0000 7f0b13400640 1 -- 172.21.15.40:0/2397138327 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_backtrace"]}]=0 v240) ==== 145+0+0 (secure 0 0 0) 0x7f0b1c0b24c0 con 0x7f0b1c072780 2024-04-26T23:05:25.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.654+0000 7f0b22600640 1 -- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0af806c8f0 msgr2=0x7f0af806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:25.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.654+0000 7f0b22600640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0af806c8f0 0x7f0af806edb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0012e0 tx=0x7f0b0c009040 comp rx=0 tx=0).stop 2024-04-26T23:05:25.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.654+0000 7f0b22600640 1 -- 172.21.15.40:0/2397138327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c072780 msgr2=0x7f0b1c1f8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:25.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.654+0000 7f0b22600640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c072780 0x7f0b1c1f8790 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0b04035040 tx=0x7f0b040046c0 comp rx=0 tx=0).stop 2024-04-26T23:05:25.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.654+0000 7f0b22600640 1 -- 172.21.15.40:0/2397138327 shutdown_connections 2024-04-26T23:05:25.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.654+0000 7f0b22600640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0af806c8f0 0x7f0af806edb0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:25.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.654+0000 7f0b22600640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b1c0cc480 0x7f0b1c0b60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:25.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.654+0000 7f0b22600640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c072780 0x7f0b1c1f8790 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:25.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.654+0000 7f0b22600640 1 --2- 172.21.15.40:0/2397138327 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b1c071e60 0x7f0b1c0b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:25.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.654+0000 7f0b22600640 1 -- 172.21.15.40:0/2397138327 >> 172.21.15.40:0/2397138327 conn(0x7f0b1c0d0be0 msgr2=0x7f0b1c0afaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:25.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.655+0000 7f0b22600640 1 -- 172.21.15.40:0/2397138327 shutdown_connections 2024-04-26T23:05:25.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.655+0000 7f0b22600640 1 -- 172.21.15.40:0/2397138327 wait complete. 2024-04-26T23:05:25.676 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-26T23:05:25.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.898+0000 7f850be00640 1 Processor -- start 2024-04-26T23:05:25.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.898+0000 7f850be00640 1 -- start start 2024-04-26T23:05:25.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.900+0000 7f850be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f850c160a50 0x7f850c073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:25.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.900+0000 7f850be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f850c06dd40 0x7f850c072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:25.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.900+0000 7f850be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f850c161880 con 0x7f850c0cfaf0 2024-04-26T23:05:25.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.900+0000 7f850be00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f850c071ce0 con 0x7f850c160a50 2024-04-26T23:05:25.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.900+0000 7f850be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f850c0722a0 con 0x7f850c06dd40 2024-04-26T23:05:25.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.901+0000 7f850b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f850c06dd40 0x7f850c072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:25.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.901+0000 7f850b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f850c06dd40 0x7f850c072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:37846/0 (socket says 172.21.15.40:37846) 2024-04-26T23:05:25.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.901+0000 7f850b400640 1 -- 172.21.15.40:0/1962764309 learned_addr learned my addr 172.21.15.40:0/1962764309 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:25.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.901+0000 7f850aa00640 1 --2- 172.21.15.40:0/1962764309 >> v2:172.21.15.103:3300/0 conn(0x7f850c160a50 0x7f850c073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:25.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.902+0000 7f850b400640 1 -- 172.21.15.40:0/1962764309 >> v2:172.21.15.103:3300/0 conn(0x7f850c160a50 msgr2=0x7f850c073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:25.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.902+0000 7f850b400640 1 --2- 172.21.15.40:0/1962764309 >> v2:172.21.15.103:3300/0 conn(0x7f850c160a50 0x7f850c073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:25.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.902+0000 7f850b400640 1 -- 172.21.15.40:0/1962764309 >> v1:172.21.15.103:6789/0 conn(0x7f850c0cfaf0 legacy=0x7f850c160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:25.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.902+0000 7f850a000640 1 -- 172.21.15.40:0/1962764309 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1387302856 0 0) 0x7f850c161880 con 0x7f850c0cfaf0 2024-04-26T23:05:25.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.902+0000 7f850b400640 1 -- 172.21.15.40:0/1962764309 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850c0cff80 con 0x7f850c06dd40 2024-04-26T23:05:25.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.903+0000 7f850b400640 1 --2- 172.21.15.40:0/1962764309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f850c06dd40 0x7f850c072730 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8504009e50 tx=0x7f850402a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a07102a33506dde1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:25.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.904+0000 7f850a000640 1 -- 172.21.15.40:0/1962764309 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504036070 con 0x7f850c06dd40 2024-04-26T23:05:25.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.904+0000 7f850a000640 1 -- 172.21.15.40:0/1962764309 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8504031600 con 0x7f850c06dd40 2024-04-26T23:05:25.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.904+0000 7f850a000640 1 -- 172.21.15.40:0/1962764309 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85040319d0 con 0x7f850c06dd40 2024-04-26T23:05:25.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.905+0000 7f850be00640 1 -- 172.21.15.40:0/1962764309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f850c06dd40 msgr2=0x7f850c072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:25.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.905+0000 7f850be00640 1 --2- 172.21.15.40:0/1962764309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f850c06dd40 0x7f850c072730 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8504009e50 tx=0x7f850402a710 comp rx=0 tx=0).stop 2024-04-26T23:05:25.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.905+0000 7f850be00640 1 -- 172.21.15.40:0/1962764309 shutdown_connections 2024-04-26T23:05:25.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.905+0000 7f850be00640 1 --2- 172.21.15.40:0/1962764309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f850c06dd40 0x7f850c072730 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:25.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.905+0000 7f850be00640 1 --2- 172.21.15.40:0/1962764309 >> v2:172.21.15.103:3300/0 conn(0x7f850c160a50 0x7f850c073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:25.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.905+0000 7f850be00640 1 -- 172.21.15.40:0/1962764309 >> 172.21.15.40:0/1962764309 conn(0x7f850c0b2720 msgr2=0x7f850c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:25.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.905+0000 7f850be00640 1 -- 172.21.15.40:0/1962764309 shutdown_connections 2024-04-26T23:05:25.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.905+0000 7f850be00640 1 -- 172.21.15.40:0/1962764309 wait complete. 2024-04-26T23:05:25.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.907+0000 7f850be00640 1 Processor -- start 2024-04-26T23:05:25.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.907+0000 7f850be00640 1 -- start start 2024-04-26T23:05:25.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.908+0000 7f850be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f850c06dd40 0x7f850c1f4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:25.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.908+0000 7f850be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f850c160a50 0x7f850c1411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:25.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.908+0000 7f850be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f850c144850 0x7f850c13f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:25.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.908+0000 7f850be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f850c071f30 con 0x7f850c144850 2024-04-26T23:05:25.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.908+0000 7f850be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f850c073f90 con 0x7f850c06dd40 2024-04-26T23:05:25.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.908+0000 7f850be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f850c06e1f0 con 0x7f850c160a50 2024-04-26T23:05:25.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.908+0000 7f850b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f850c144850 0x7f850c13f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:25.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.908+0000 7f850b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f850c144850 0x7f850c13f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45578/0 (socket says 172.21.15.40:45578) 2024-04-26T23:05:25.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.908+0000 7f850b400640 1 -- 172.21.15.40:0/1778204063 learned_addr learned my addr 172.21.15.40:0/1778204063 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:25.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.909+0000 7f8503e00640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f850c160a50 0x7f850c1411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:25.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.909+0000 7f850aa00640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f850c06dd40 0x7f850c1f4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:25.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.909+0000 7f850b400640 1 -- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f850c160a50 msgr2=0x7f850c1411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:25.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.909+0000 7f850b400640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f850c160a50 0x7f850c1411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:25.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.909+0000 7f850b400640 1 -- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f850c06dd40 msgr2=0x7f850c1f4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:25.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.909+0000 7f850b400640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f850c06dd40 0x7f850c1f4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:25.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.909+0000 7f850b400640 1 -- 172.21.15.40:0/1778204063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850c145580 con 0x7f850c144850 2024-04-26T23:05:25.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.909+0000 7f850aa00640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f850c06dd40 0x7f850c1f4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:25.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.909+0000 7f850b400640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f850c144850 0x7f850c13f6d0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f8504009f70 tx=0x7f8504004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:25.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.910+0000 7f8502a00640 1 -- 172.21.15.40:0/1778204063 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504041020 con 0x7f850c144850 2024-04-26T23:05:25.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.910+0000 7f8502a00640 1 -- 172.21.15.40:0/1778204063 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8504008730 con 0x7f850c144850 2024-04-26T23:05:25.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.910+0000 7f8502a00640 1 -- 172.21.15.40:0/1778204063 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504036040 con 0x7f850c144850 2024-04-26T23:05:25.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.910+0000 7f850be00640 1 -- 172.21.15.40:0/1778204063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850c0ae9f0 con 0x7f850c144850 2024-04-26T23:05:25.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.910+0000 7f850be00640 1 -- 172.21.15.40:0/1778204063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f850c0aec90 con 0x7f850c144850 2024-04-26T23:05:25.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.911+0000 7f850be00640 1 -- 172.21.15.40:0/1778204063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f850c069370 con 0x7f850c144850 2024-04-26T23:05:25.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.912+0000 7f8502a00640 1 -- 172.21.15.40:0/1778204063 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8504044020 con 0x7f850c144850 2024-04-26T23:05:25.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.913+0000 7f8502a00640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f84e806c8f0 0x7f84e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:25.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.913+0000 7f8502a00640 1 -- 172.21.15.40:0/1778204063 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5507+0+0 (secure 0 0 0) 0x7f85040ac650 con 0x7f850c144850 2024-04-26T23:05:25.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.913+0000 7f850aa00640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f84e806c8f0 0x7f84e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:25.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.915+0000 7f850aa00640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f84e806c8f0 0x7f84e806edb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009a20 tx=0x7f84fc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:25.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:25.917+0000 7f8502a00640 1 -- 172.21.15.40:0/1778204063 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f850404b030 con 0x7f850c144850 2024-04-26T23:05:26.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.128+0000 7f850be00640 1 -- 172.21.15.40:0/1778204063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f850c1612b0 con 0x7f850c144850 2024-04-26T23:05:26.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.134+0000 7f8502a00640 1 -- 172.21.15.40:0/1778204063 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) ==== 122+0+0 (secure 0 0 0) 0x7f850407ab30 con 0x7f850c144850 2024-04-26T23:05:26.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.137+0000 7f850be00640 1 -- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f84e806c8f0 msgr2=0x7f84e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:26.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.137+0000 7f850be00640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f84e806c8f0 0x7f84e806edb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009a20 tx=0x7f84fc008040 comp rx=0 tx=0).stop 2024-04-26T23:05:26.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.138+0000 7f850be00640 1 -- 172.21.15.40:0/1778204063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f850c144850 msgr2=0x7f850c13f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:26.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.138+0000 7f850be00640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f850c144850 0x7f850c13f6d0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f8504009f70 tx=0x7f8504004a40 comp rx=0 tx=0).stop 2024-04-26T23:05:26.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.138+0000 7f850be00640 1 -- 172.21.15.40:0/1778204063 shutdown_connections 2024-04-26T23:05:26.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.138+0000 7f850be00640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f84e806c8f0 0x7f84e806edb0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:26.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.138+0000 7f850be00640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f850c144850 0x7f850c13f6d0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:26.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.138+0000 7f850be00640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f850c160a50 0x7f850c1411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:26.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.138+0000 7f850be00640 1 --2- 172.21.15.40:0/1778204063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f850c06dd40 0x7f850c1f4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:26.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.138+0000 7f850be00640 1 -- 172.21.15.40:0/1778204063 >> 172.21.15.40:0/1778204063 conn(0x7f850c0b2720 msgr2=0x7f850c0b0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:26.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.138+0000 7f850be00640 1 -- 172.21.15.40:0/1778204063 shutdown_connections 2024-04-26T23:05:26.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.138+0000 7f850be00640 1 -- 172.21.15.40:0/1778204063 wait complete. 2024-04-26T23:05:26.158 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-26T23:05:26.158 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:05:26.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:05:26.377 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-26T23:05:26.377 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:05:26.377 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:05:26.437 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:05:26.437 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:05:26.437 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:05:26.495 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:05:26.496 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:05:26.496 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:05:26.556 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:05:26.556 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:05:26.556 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:05:26.614 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:05:26.614 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-26T23:05:26.614 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:05:26.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.832+0000 7f2b86a00640 1 Processor -- start 2024-04-26T23:05:26.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.832+0000 7f2b86a00640 1 -- start start 2024-04-26T23:05:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.834+0000 7f2b86a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2b80071e60 0x7f2b8006f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.834+0000 7f2b86a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b800cc480 0x7f2b80166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.834+0000 7f2b86a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b8015d7c0 con 0x7f2b8006fe20 2024-04-26T23:05:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.834+0000 7f2b86a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f2b8015daa0 con 0x7f2b800cc480 2024-04-26T23:05:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.834+0000 7f2b86a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2b8015dd80 con 0x7f2b80071e60 2024-04-26T23:05:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.834+0000 7f2b84a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b800cc480 0x7f2b80166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.834+0000 7f2b84a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b800cc480 0x7f2b80166ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45588/0 (socket says 172.21.15.40:45588) 2024-04-26T23:05:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.834+0000 7f2b84a00640 1 -- 172.21.15.40:0/613525931 learned_addr learned my addr 172.21.15.40:0/613525931 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.835+0000 7f2b84a00640 1 -- 172.21.15.40:0/613525931 >> v2:172.21.15.103:3300/0 conn(0x7f2b80071e60 msgr2=0x7f2b8006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.835+0000 7f2b84a00640 1 --2- 172.21.15.40:0/613525931 >> v2:172.21.15.103:3300/0 conn(0x7f2b80071e60 0x7f2b8006f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.835+0000 7f2b84a00640 1 -- 172.21.15.40:0/613525931 >> v1:172.21.15.103:6789/0 conn(0x7f2b8006fe20 legacy=0x7f2b80072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.835+0000 7f2b84a00640 1 -- 172.21.15.40:0/613525931 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8015e060 con 0x7f2b800cc480 2024-04-26T23:05:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.836+0000 7f2b84a00640 1 --2- 172.21.15.40:0/613525931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b800cc480 0x7f2b80166ad0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f2b800b3700 tx=0x7f2b7802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fc8df897a78b89 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.836+0000 7f2b7ea00640 1 -- 172.21.15.40:0/613525931 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78036070 con 0x7f2b800cc480 2024-04-26T23:05:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.836+0000 7f2b7ea00640 1 -- 172.21.15.40:0/613525931 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b78031600 con 0x7f2b800cc480 2024-04-26T23:05:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.836+0000 7f2b7ea00640 1 -- 172.21.15.40:0/613525931 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b780319d0 con 0x7f2b800cc480 2024-04-26T23:05:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.837+0000 7f2b86a00640 1 -- 172.21.15.40:0/613525931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b800cc480 msgr2=0x7f2b80166ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.837+0000 7f2b86a00640 1 --2- 172.21.15.40:0/613525931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b800cc480 0x7f2b80166ad0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f2b800b3700 tx=0x7f2b7802a6f0 comp rx=0 tx=0).stop 2024-04-26T23:05:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.837+0000 7f2b86a00640 1 -- 172.21.15.40:0/613525931 shutdown_connections 2024-04-26T23:05:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.837+0000 7f2b86a00640 1 --2- 172.21.15.40:0/613525931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b800cc480 0x7f2b80166ad0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.837+0000 7f2b86a00640 1 --2- 172.21.15.40:0/613525931 >> v2:172.21.15.103:3300/0 conn(0x7f2b80071e60 0x7f2b8006f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.837+0000 7f2b86a00640 1 -- 172.21.15.40:0/613525931 >> 172.21.15.40:0/613525931 conn(0x7f2b800d0be0 msgr2=0x7f2b800d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.837+0000 7f2b86a00640 1 -- 172.21.15.40:0/613525931 shutdown_connections 2024-04-26T23:05:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.837+0000 7f2b86a00640 1 -- 172.21.15.40:0/613525931 wait complete. 2024-04-26T23:05:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.839+0000 7f2b86a00640 1 Processor -- start 2024-04-26T23:05:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.839+0000 7f2b86a00640 1 -- start start 2024-04-26T23:05:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.840+0000 7f2b86a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b8006fe20 0x7f2b800b1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.840+0000 7f2b86a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b80071e60 0x7f2b80152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.840+0000 7f2b86a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b800cc480 0x7f2b800b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.840+0000 7f2b86a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8015d570 con 0x7f2b80071e60 2024-04-26T23:05:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.840+0000 7f2b86a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8015d3f0 con 0x7f2b800cc480 2024-04-26T23:05:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.840+0000 7f2b86a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2b80072950 con 0x7f2b8006fe20 2024-04-26T23:05:26.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.840+0000 7f2b7f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b80071e60 0x7f2b80152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:26.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.840+0000 7f2b7f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b80071e60 0x7f2b80152990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45590/0 (socket says 172.21.15.40:45590) 2024-04-26T23:05:26.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.840+0000 7f2b7f400640 1 -- 172.21.15.40:0/3224332362 learned_addr learned my addr 172.21.15.40:0/3224332362 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:26.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b84a00640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b800cc480 0x7f2b800b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:26.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b7fe00640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b8006fe20 0x7f2b800b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:26.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b7f400640 1 -- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b8006fe20 msgr2=0x7f2b800b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:26.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b7f400640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b8006fe20 0x7f2b800b1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:26.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b7f400640 1 -- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b800cc480 msgr2=0x7f2b800b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:26.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b7f400640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b800cc480 0x7f2b800b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:26.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b7f400640 1 -- 172.21.15.40:0/3224332362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b800b1c50 con 0x7f2b80071e60 2024-04-26T23:05:26.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b84a00640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b800cc480 0x7f2b800b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:26.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b7fe00640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b8006fe20 0x7f2b800b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:26.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b7f400640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b80071e60 0x7f2b80152990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00c440 tx=0x7f2b6c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:26.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b77e00640 1 -- 172.21.15.40:0/3224332362 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c017070 con 0x7f2b80071e60 2024-04-26T23:05:26.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b77e00640 1 -- 172.21.15.40:0/3224332362 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b6c0035f0 con 0x7f2b80071e60 2024-04-26T23:05:26.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b86a00640 1 -- 172.21.15.40:0/3224332362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b801f8940 con 0x7f2b80071e60 2024-04-26T23:05:26.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.841+0000 7f2b77e00640 1 -- 172.21.15.40:0/3224332362 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c0039e0 con 0x7f2b80071e60 2024-04-26T23:05:26.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.842+0000 7f2b86a00640 1 -- 172.21.15.40:0/3224332362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b80153440 con 0x7f2b80071e60 2024-04-26T23:05:26.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.843+0000 7f2b86a00640 1 -- 172.21.15.40:0/3224332362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b801f8730 con 0x7f2b80071e60 2024-04-26T23:05:26.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.844+0000 7f2b77e00640 1 -- 172.21.15.40:0/3224332362 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2b6c01a070 con 0x7f2b80071e60 2024-04-26T23:05:26.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.845+0000 7f2b77e00640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2b6006c8f0 0x7f2b6006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:26.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.845+0000 7f2b77e00640 1 -- 172.21.15.40:0/3224332362 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5507+0+0 (secure 0 0 0) 0x7f2b6c08ae50 con 0x7f2b80071e60 2024-04-26T23:05:26.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.846+0000 7f2b7fe00640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2b6006c8f0 0x7f2b6006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:26.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.847+0000 7f2b7fe00640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2b6006c8f0 0x7f2b6006edb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2b70006fd0 tx=0x7f2b70008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:26.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:26.849+0000 7f2b77e00640 1 -- 172.21.15.40:0/3224332362 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2b6c059330 con 0x7f2b80071e60 2024-04-26T23:05:27.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.092+0000 7f2b86a00640 1 -- 172.21.15.40:0/3224332362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b8006e9d0 con 0x7f2b80071e60 2024-04-26T23:05:27.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.093+0000 7f2b77e00640 1 -- 172.21.15.40:0/3224332362 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+3869 (secure 0 0 0) 0x7f2b6c0010c0 con 0x7f2b80071e60 2024-04-26T23:05:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:27.094 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":48,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/189004193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":189004193},{"type":"v1","addr":"172.21.15.103:6835","nonce":189004193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/658206194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":658206194},{"type":"v1","addr":"172.21.15.103:6837","nonce":658206194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6684,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/4236709740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4236709740},{"type":"v1","addr":"172.21.15.40:6834","nonce":4236709740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6693,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2225226633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2225226633},{"type":"v1","addr":"172.21.15.40:6835","nonce":2225226633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[]} 2024-04-26T23:05:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 48 2024-04-26T23:05:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.096+0000 7f2b86a00640 1 -- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2b6006c8f0 msgr2=0x7f2b6006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.096+0000 7f2b86a00640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2b6006c8f0 0x7f2b6006edb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2b70006fd0 tx=0x7f2b70008040 comp rx=0 tx=0).stop 2024-04-26T23:05:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.096+0000 7f2b86a00640 1 -- 172.21.15.40:0/3224332362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b80071e60 msgr2=0x7f2b80152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.096+0000 7f2b86a00640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b80071e60 0x7f2b80152990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00c440 tx=0x7f2b6c00c900 comp rx=0 tx=0).stop 2024-04-26T23:05:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.096+0000 7f2b86a00640 1 -- 172.21.15.40:0/3224332362 shutdown_connections 2024-04-26T23:05:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.096+0000 7f2b86a00640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2b6006c8f0 0x7f2b6006edb0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.096+0000 7f2b86a00640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b800cc480 0x7f2b800b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.096+0000 7f2b86a00640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b80071e60 0x7f2b80152990 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.096+0000 7f2b86a00640 1 --2- 172.21.15.40:0/3224332362 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b8006fe20 0x7f2b800b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.096+0000 7f2b86a00640 1 -- 172.21.15.40:0/3224332362 >> 172.21.15.40:0/3224332362 conn(0x7f2b800d0be0 msgr2=0x7f2b8015f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.096+0000 7f2b86a00640 1 -- 172.21.15.40:0/3224332362 shutdown_connections 2024-04-26T23:05:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.096+0000 7f2b86a00640 1 -- 172.21.15.40:0/3224332362 wait complete. 2024-04-26T23:05:27.115 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-26T23:05:27.115 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-26T23:05:27.115 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-26T23:05:27.115 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:05:27.115 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-26T23:05:27.116 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-26T23:05:27.116 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-26T23:05:27.116 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:05:27.116 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-26T23:05:27.116 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-26T23:05:27.116 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-26T23:05:27.116 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:05:27.116 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-26T23:05:27.117 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-26T23:05:27.117 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-26T23:05:27.117 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:05:27.135 INFO:tasks.ceph.mds.c:Stopped 2024-04-26T23:05:27.135 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-26T23:05:27.137 INFO:tasks.ceph.mds.a:Stopped 2024-04-26T23:05:27.137 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-26T23:05:27.138 INFO:tasks.ceph.mds.d:Stopped 2024-04-26T23:05:27.139 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-26T23:05:27.139 INFO:tasks.ceph.mds.b:Stopped 2024-04-26T23:05:27.140 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-26T23:05:27.182 INFO:tasks.ceph.mds.c:Started 2024-04-26T23:05:27.182 DEBUG:teuthology.parallel:result is None 2024-04-26T23:05:27.186 INFO:tasks.ceph.mds.d:Started 2024-04-26T23:05:27.186 DEBUG:teuthology.parallel:result is None 2024-04-26T23:05:27.189 INFO:tasks.ceph.mds.a:Started 2024-04-26T23:05:27.189 DEBUG:teuthology.parallel:result is None 2024-04-26T23:05:27.192 INFO:tasks.ceph.mds.b:Started 2024-04-26T23:05:27.192 DEBUG:teuthology.parallel:result is None 2024-04-26T23:05:27.192 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-26T23:05:27.330 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-04-26T23:05:27.330 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:05:27.337 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-26T23:05:27.339 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-26T23:05:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.366+0000 7fc560e00640 1 Processor -- start 2024-04-26T23:05:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.366+0000 7fc560e00640 1 -- start start 2024-04-26T23:05:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.367+0000 7fc560e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc55c06e7b0 0x7fc55c169df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.367+0000 7fc560e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc55c15f5a0 con 0x7fc55c15f1c0 2024-04-26T23:05:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.367+0000 7fc560e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc55c0d3b50 con 0x7fc55c0b22e0 2024-04-26T23:05:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.367+0000 7fc560e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc55c0b7390 con 0x7fc55c06e7b0 2024-04-26T23:05:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.367+0000 7fc55b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc55c15f1c0 0x7fc55c162c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38076/0 (socket says 172.21.15.40:38076) 2024-04-26T23:05:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.367+0000 7fc55b400640 1 -- 172.21.15.40:0/2337852009 learned_addr learned my addr 172.21.15.40:0/2337852009 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.368+0000 7fc55be00640 1 --2- 172.21.15.40:0/2337852009 >> v2:172.21.15.40:3300/0 conn(0x7fc55c06e7b0 0x7fc55c169df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.368+0000 7fc55a000640 1 -- 172.21.15.40:0/2337852009 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3017466564 0 0) 0x7fc55c15f5a0 con 0x7fc55c15f1c0 2024-04-26T23:05:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.368+0000 7fc55a000640 1 -- 172.21.15.40:0/2337852009 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc540003520 con 0x7fc55c15f1c0 2024-04-26T23:05:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.368+0000 7fc55be00640 1 -- 172.21.15.40:0/2337852009 >> v1:172.21.15.103:6789/0 conn(0x7fc55c0b22e0 legacy=0x7fc55c163380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.368+0000 7fc55be00640 1 -- 172.21.15.40:0/2337852009 >> v1:172.21.15.40:6789/0 conn(0x7fc55c15f1c0 legacy=0x7fc55c162c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.368+0000 7fc55be00640 1 -- 172.21.15.40:0/2337852009 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc55c0b7670 con 0x7fc55c06e7b0 2024-04-26T23:05:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.369+0000 7fc55be00640 1 --2- 172.21.15.40:0/2337852009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c06e7b0 0x7fc55c169df0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fc554009d90 tx=0x7fc55402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6404d95f64a4214c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.369+0000 7fc55a000640 1 -- 172.21.15.40:0/2337852009 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc554007740 con 0x7fc55c06e7b0 2024-04-26T23:05:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.369+0000 7fc55a000640 1 -- 172.21.15.40:0/2337852009 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5540078e0 con 0x7fc55c06e7b0 2024-04-26T23:05:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.369+0000 7fc55a000640 1 -- 172.21.15.40:0/2337852009 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc554007c90 con 0x7fc55c06e7b0 2024-04-26T23:05:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.370+0000 7fc560e00640 1 -- 172.21.15.40:0/2337852009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c06e7b0 msgr2=0x7fc55c169df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.370+0000 7fc560e00640 1 --2- 172.21.15.40:0/2337852009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c06e7b0 0x7fc55c169df0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fc554009d90 tx=0x7fc55402a6f0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.370+0000 7fc560e00640 1 -- 172.21.15.40:0/2337852009 shutdown_connections 2024-04-26T23:05:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.370+0000 7fc560e00640 1 --2- 172.21.15.40:0/2337852009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c06e7b0 0x7fc55c169df0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.370+0000 7fc560e00640 1 -- 172.21.15.40:0/2337852009 >> 172.21.15.40:0/2337852009 conn(0x7fc55c09ba50 msgr2=0x7fc55c09be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:27.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.370+0000 7fc560e00640 1 -- 172.21.15.40:0/2337852009 shutdown_connections 2024-04-26T23:05:27.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.370+0000 7fc560e00640 1 -- 172.21.15.40:0/2337852009 wait complete. 2024-04-26T23:05:27.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.372+0000 7fc560e00640 1 Processor -- start 2024-04-26T23:05:27.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.372+0000 7fc560e00640 1 -- start start 2024-04-26T23:05:27.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.373+0000 7fc560e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc55c06e7b0 0x7fc55c0b0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:27.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.373+0000 7fc560e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c0b22e0 0x7fc55c1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:27.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.373+0000 7fc560e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc55c15f1c0 0x7fc55c14d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.373+0000 7fc560e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc55c0702c0 con 0x7fc55c0b22e0 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.373+0000 7fc560e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc55c070140 con 0x7fc55c15f1c0 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.373+0000 7fc560e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc55c16a330 con 0x7fc55c06e7b0 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.373+0000 7fc55aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c0b22e0 0x7fc55c1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.373+0000 7fc55b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc55c06e7b0 0x7fc55c0b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.373+0000 7fc55aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c0b22e0 0x7fc55c1579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45644/0 (socket says 172.21.15.40:45644) 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.373+0000 7fc55aa00640 1 -- 172.21.15.40:0/531293197 learned_addr learned my addr 172.21.15.40:0/531293197 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.373+0000 7fc55be00640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc55c15f1c0 0x7fc55c14d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.374+0000 7fc55aa00640 1 -- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc55c06e7b0 msgr2=0x7fc55c0b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.374+0000 7fc55aa00640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc55c06e7b0 0x7fc55c0b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.374+0000 7fc55aa00640 1 -- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc55c15f1c0 msgr2=0x7fc55c14d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.374+0000 7fc55aa00640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc55c15f1c0 0x7fc55c14d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.374+0000 7fc55aa00640 1 -- 172.21.15.40:0/531293197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc55c156760 con 0x7fc55c0b22e0 2024-04-26T23:05:27.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.374+0000 7fc55b400640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc55c06e7b0 0x7fc55c0b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:05:27.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.374+0000 7fc55aa00640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c0b22e0 0x7fc55c1579a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc54800c400 tx=0x7fc54800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:27.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.374+0000 7fc553400640 1 -- 172.21.15.40:0/531293197 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5480052d0 con 0x7fc55c0b22e0 2024-04-26T23:05:27.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.374+0000 7fc553400640 1 -- 172.21.15.40:0/531293197 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc548005490 con 0x7fc55c0b22e0 2024-04-26T23:05:27.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.374+0000 7fc553400640 1 -- 172.21.15.40:0/531293197 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc548007260 con 0x7fc55c0b22e0 2024-04-26T23:05:27.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.374+0000 7fc560e00640 1 -- 172.21.15.40:0/531293197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc55c156a40 con 0x7fc55c0b22e0 2024-04-26T23:05:27.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.374+0000 7fc560e00640 1 -- 172.21.15.40:0/531293197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc55c157000 con 0x7fc55c0b22e0 2024-04-26T23:05:27.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.375+0000 7fc560e00640 1 -- 172.21.15.40:0/531293197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc55c069370 con 0x7fc55c0b22e0 2024-04-26T23:05:27.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.376+0000 7fc553400640 1 -- 172.21.15.40:0/531293197 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc548007400 con 0x7fc55c0b22e0 2024-04-26T23:05:27.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.377+0000 7fc553400640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc53c06c8f0 0x7fc53c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:27.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.377+0000 7fc553400640 1 -- 172.21.15.40:0/531293197 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5507+0+0 (secure 0 0 0) 0x7fc54808aff0 con 0x7fc55c0b22e0 2024-04-26T23:05:27.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.378+0000 7fc55b400640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc53c06c8f0 0x7fc53c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:27.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.379+0000 7fc55b400640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc53c06c8f0 0x7fc53c06edb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0011e0 tx=0x7fc54c007d30 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:27.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.382+0000 7fc553400640 1 -- 172.21.15.40:0/531293197 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc55c0d3b50 con 0x7fc55c0b22e0 2024-04-26T23:05:27.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.589+0000 7fc560e00640 1 -- 172.21.15.40:0/531293197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fc55c06cc40 con 0x7fc55c0b22e0 2024-04-26T23:05:27.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.677+0000 7fc553400640 1 -- 172.21.15.40:0/531293197 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v59) ==== 97+0+0 (secure 0 0 0) 0x7fc548058e10 con 0x7fc55c0b22e0 2024-04-26T23:05:27.678 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-04-26T23:05:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.680+0000 7fc560e00640 1 -- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc53c06c8f0 msgr2=0x7fc53c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.680+0000 7fc560e00640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc53c06c8f0 0x7fc53c06edb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0011e0 tx=0x7fc54c007d30 comp rx=0 tx=0).stop 2024-04-26T23:05:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.680+0000 7fc560e00640 1 -- 172.21.15.40:0/531293197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c0b22e0 msgr2=0x7fc55c1579a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.680+0000 7fc560e00640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c0b22e0 0x7fc55c1579a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc54800c400 tx=0x7fc54800c8c0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.680+0000 7fc560e00640 1 -- 172.21.15.40:0/531293197 shutdown_connections 2024-04-26T23:05:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.680+0000 7fc560e00640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc53c06c8f0 0x7fc53c06edb0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.680+0000 7fc560e00640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc55c15f1c0 0x7fc55c14d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.680+0000 7fc560e00640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c0b22e0 0x7fc55c1579a0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.680+0000 7fc560e00640 1 --2- 172.21.15.40:0/531293197 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc55c06e7b0 0x7fc55c0b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.680+0000 7fc560e00640 1 -- 172.21.15.40:0/531293197 >> 172.21.15.40:0/531293197 conn(0x7fc55c09ba50 msgr2=0x7fc55c09be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.680+0000 7fc560e00640 1 -- 172.21.15.40:0/531293197 shutdown_connections 2024-04-26T23:05:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.680+0000 7fc560e00640 1 -- 172.21.15.40:0/531293197 wait complete. 2024-04-26T23:05:27.702 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-26T23:05:27.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.903+0000 7f1c4da00640 1 Processor -- start 2024-04-26T23:05:27.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.903+0000 7f1c4da00640 1 -- start start 2024-04-26T23:05:27.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.905+0000 7f1c4da00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1c48073e10 0x7f1c481600d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:27.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.905+0000 7f1c4da00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c4815c160 0x7f1c480b4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:27.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.905+0000 7f1c4da00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c48160610 con 0x7f1c4815c5c0 2024-04-26T23:05:27.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.905+0000 7f1c4da00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1c48153040 con 0x7f1c48073e10 2024-04-26T23:05:27.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.905+0000 7f1c4da00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c4815fc40 con 0x7f1c4815c160 2024-04-26T23:05:27.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.905+0000 7f1c47e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1c48073e10 0x7f1c481600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:27.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.905+0000 7f1c47e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1c48073e10 0x7f1c481600d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45650/0 (socket says 172.21.15.40:45650) 2024-04-26T23:05:27.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.905+0000 7f1c47e00640 1 -- 172.21.15.40:0/3585027259 learned_addr learned my addr 172.21.15.40:0/3585027259 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:27.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.906+0000 7f1c3fe00640 1 --2- 172.21.15.40:0/3585027259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c4815c160 0x7f1c480b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:27.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.906+0000 7f1c47e00640 1 -- 172.21.15.40:0/3585027259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c4815c160 msgr2=0x7f1c480b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:27.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.906+0000 7f1c47e00640 1 --2- 172.21.15.40:0/3585027259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c4815c160 0x7f1c480b4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.906+0000 7f1c47e00640 1 -- 172.21.15.40:0/3585027259 >> v1:172.21.15.103:6789/0 conn(0x7f1c4815c5c0 legacy=0x7f1c4815d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:27.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.906+0000 7f1c47e00640 1 -- 172.21.15.40:0/3585027259 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4815cb90 con 0x7f1c48073e10 2024-04-26T23:05:27.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.907+0000 7f1c47e00640 1 --2- 172.21.15.40:0/3585027259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c48073e10 0x7f1c481600d0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f1c380098a0 tx=0x7f1c3802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46476fbedefcd03 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:27.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.907+0000 7f1c47400640 1 -- 172.21.15.40:0/3585027259 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c38037070 con 0x7f1c48073e10 2024-04-26T23:05:27.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.907+0000 7f1c47400640 1 -- 172.21.15.40:0/3585027259 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c38032440 con 0x7f1c48073e10 2024-04-26T23:05:27.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.907+0000 7f1c47400640 1 -- 172.21.15.40:0/3585027259 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c38032810 con 0x7f1c48073e10 2024-04-26T23:05:27.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.908+0000 7f1c4da00640 1 -- 172.21.15.40:0/3585027259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c48073e10 msgr2=0x7f1c481600d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:27.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.908+0000 7f1c4da00640 1 --2- 172.21.15.40:0/3585027259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c48073e10 0x7f1c481600d0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f1c380098a0 tx=0x7f1c3802a250 comp rx=0 tx=0).stop 2024-04-26T23:05:27.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.908+0000 7f1c4da00640 1 -- 172.21.15.40:0/3585027259 shutdown_connections 2024-04-26T23:05:27.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.908+0000 7f1c4da00640 1 --2- 172.21.15.40:0/3585027259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c4815c160 0x7f1c480b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.908+0000 7f1c4da00640 1 --2- 172.21.15.40:0/3585027259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c48073e10 0x7f1c481600d0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.908+0000 7f1c4da00640 1 -- 172.21.15.40:0/3585027259 >> 172.21.15.40:0/3585027259 conn(0x7f1c48075eb0 msgr2=0x7f1c48068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:27.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.908+0000 7f1c4da00640 1 -- 172.21.15.40:0/3585027259 shutdown_connections 2024-04-26T23:05:27.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.908+0000 7f1c4da00640 1 -- 172.21.15.40:0/3585027259 wait complete. 2024-04-26T23:05:27.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.910+0000 7f1c4da00640 1 Processor -- start 2024-04-26T23:05:27.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.910+0000 7f1c4da00640 1 -- start start 2024-04-26T23:05:27.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.911+0000 7f1c4da00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c48073e10 0x7f1c4814bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:27.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.911+0000 7f1c4da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c4815c160 0x7f1c481f7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:27.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.911+0000 7f1c4da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4815c5c0 0x7f1c48151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:27.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.911+0000 7f1c4da00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1c480733c0 con 0x7f1c4815c5c0 2024-04-26T23:05:27.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.911+0000 7f1c4da00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4815f8d0 con 0x7f1c4815c160 2024-04-26T23:05:27.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.911+0000 7f1c4da00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c48073540 con 0x7f1c48073e10 2024-04-26T23:05:27.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.911+0000 7f1c4d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4815c5c0 0x7f1c48151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:27.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.912+0000 7f1c4d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4815c5c0 0x7f1c48151270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45656/0 (socket says 172.21.15.40:45656) 2024-04-26T23:05:27.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.912+0000 7f1c4d000640 1 -- 172.21.15.40:0/198106542 learned_addr learned my addr 172.21.15.40:0/198106542 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:27.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.912+0000 7f1c3fe00640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c4815c160 0x7f1c481f7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:27.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.912+0000 7f1c47e00640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c48073e10 0x7f1c4814bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:27.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.912+0000 7f1c4d000640 1 -- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c48073e10 msgr2=0x7f1c4814bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:27.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.912+0000 7f1c4d000640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c48073e10 0x7f1c4814bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.912+0000 7f1c4d000640 1 -- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c4815c160 msgr2=0x7f1c481f7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:27.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.912+0000 7f1c4d000640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c4815c160 0x7f1c481f7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:27.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.912+0000 7f1c4d000640 1 -- 172.21.15.40:0/198106542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c481f5620 con 0x7f1c4815c5c0 2024-04-26T23:05:27.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.912+0000 7f1c4d000640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4815c5c0 0x7f1c48151270 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000b970 tx=0x7f1c4000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:27.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.913+0000 7f1c44c00640 1 -- 172.21.15.40:0/198106542 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c40019070 con 0x7f1c4815c5c0 2024-04-26T23:05:27.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.913+0000 7f1c44c00640 1 -- 172.21.15.40:0/198106542 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c4000cb40 con 0x7f1c4815c5c0 2024-04-26T23:05:27.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.913+0000 7f1c44c00640 1 -- 172.21.15.40:0/198106542 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c40014440 con 0x7f1c4815c5c0 2024-04-26T23:05:27.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.913+0000 7f1c4da00640 1 -- 172.21.15.40:0/198106542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c481f5900 con 0x7f1c4815c5c0 2024-04-26T23:05:27.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.913+0000 7f1c4da00640 1 -- 172.21.15.40:0/198106542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c481f5ec0 con 0x7f1c4815c5c0 2024-04-26T23:05:27.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.914+0000 7f1c4da00640 1 -- 172.21.15.40:0/198106542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c480cd9f0 con 0x7f1c4815c5c0 2024-04-26T23:05:27.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.915+0000 7f1c44c00640 1 -- 172.21.15.40:0/198106542 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1c4001c070 con 0x7f1c4815c5c0 2024-04-26T23:05:27.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.916+0000 7f1c44c00640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c2406c8f0 0x7f1c2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:27.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.916+0000 7f1c44c00640 1 -- 172.21.15.40:0/198106542 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5249+0+0 (secure 0 0 0) 0x7f1c4008b720 con 0x7f1c4815c5c0 2024-04-26T23:05:27.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.917+0000 7f1c47e00640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c2406c8f0 0x7f1c2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:27.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.918+0000 7f1c47e00640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c2406c8f0 0x7f1c2406edb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1c38034000 tx=0x7f1c3802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:27.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:27.920+0000 7f1c44c00640 1 -- 172.21.15.40:0/198106542 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1c480cd9f0 con 0x7f1c4815c5c0 2024-04-26T23:05:28.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.206+0000 7f1c4da00640 1 -- 172.21.15.40:0/198106542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f1c480cd710 con 0x7f1c4815c5c0 2024-04-26T23:05:28.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.207+0000 7f1c44c00640 1 -- 172.21.15.40:0/198106542 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5304 (secure 0 0 0) 0x7f1c480cd710 con 0x7f1c4815c5c0 2024-04-26T23:05:28.208 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:28.208 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:05:28.208 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-04-26T23:05:28.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.208 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-04-26T23:05:28.208 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmF/UQHBAACdb8iskp2MuBMV6S5tYFJg==", 2024-04-26T23:05:28.209 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.209 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:05:28.209 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:05:28.209 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:05:28.209 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.209 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.210 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-04-26T23:05:28.210 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmniCyMxAAT/rVbIKj9EnvwDiblKMCeg==", 2024-04-26T23:05:28.210 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.210 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:05:28.210 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:05:28.210 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:05:28.211 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.211 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.211 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-04-26T23:05:28.211 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmLGZhJxAAg/fsrRrNoF/Jx1PnBVjjeQ==", 2024-04-26T23:05:28.211 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.212 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:05:28.212 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:05:28.212 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:05:28.212 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.212 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.212 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-04-26T23:05:28.212 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxms3DnAxAAxJj/rybrv/0cLK8PmQ619A==", 2024-04-26T23:05:28.212 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.212 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:05:28.213 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:05:28.213 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:05:28.213 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.213 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.213 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-04-26T23:05:28.213 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDrMSxmbeONCxAATB67ujBDn2tVxpbMju7Mfg==", 2024-04-26T23:05:28.213 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.214 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:05:28.214 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:05:28.214 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.214 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.214 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-04-26T23:05:28.214 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDuMSxmqE/PJhAAsNNu5ar8tHWQauxaXOvagA==", 2024-04-26T23:05:28.214 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.214 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:05:28.214 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:05:28.214 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.215 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.215 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-04-26T23:05:28.215 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDyMSxmd2trBhAAbnl7EBldxgtOd1s0Ll0A9w==", 2024-04-26T23:05:28.215 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.215 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:05:28.215 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:05:28.215 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.216 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.216 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-04-26T23:05:28.216 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD1MSxmqdtKEhAA9oxBxOu/musCKyladrJgYA==", 2024-04-26T23:05:28.216 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.216 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:05:28.216 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:05:28.216 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.216 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.217 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-04-26T23:05:28.217 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD7MSxm+kl7IxAAlPAGIemdC8gcvp+koeULaQ==", 2024-04-26T23:05:28.217 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.217 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:05:28.217 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:05:28.217 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.217 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.217 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-04-26T23:05:28.217 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD+MSxmzKiaLRAA2UPS1wwJHgMxQf/zLMjMVQ==", 2024-04-26T23:05:28.217 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.218 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:05:28.218 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:05:28.218 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.218 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.218 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-04-26T23:05:28.218 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQACMixmluHrCxAAmbmb33f5MVSTWNuy1D0Zsg==", 2024-04-26T23:05:28.218 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.218 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:05:28.219 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:05:28.219 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.219 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.219 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-04-26T23:05:28.219 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAFMixm6GQjJRAArmdnX7ioekT8GSdsbeB1Ow==", 2024-04-26T23:05:28.219 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.219 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:05:28.219 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:05:28.219 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.219 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.220 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-04-26T23:05:28.220 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg==", 2024-04-26T23:05:28.220 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.220 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:05:28.220 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow r", 2024-04-26T23:05:28.220 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-26T23:05:28.220 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.220 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-04-26T23:05:28.220 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmVhn4IhAA94XsDk1x5Oax5smXPoFQKg==", 2024-04-26T23:05:28.221 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.221 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:05:28.221 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:05:28.221 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:05:28.221 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:05:28.221 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.221 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-04-26T23:05:28.221 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxm3qhbGRAAK62blgPVShIYskDsMcaffQ==", 2024-04-26T23:05:28.222 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.222 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:05:28.222 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:05:28.222 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:05:28.222 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:05:28.222 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.222 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-04-26T23:05:28.223 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmxof/KxAAUmXVV4Vdp2DZnRjQzQ0Inw==", 2024-04-26T23:05:28.223 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.223 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:05:28.223 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:05:28.223 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:05:28.223 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:05:28.223 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.224 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-04-26T23:05:28.224 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDlMSxmmVwcGRAAqgHymPPTPuS6XXC6LuOVPg==", 2024-04-26T23:05:28.224 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.224 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:05:28.224 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:05:28.224 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:05:28.224 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.224 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.225 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-04-26T23:05:28.225 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmVf+9EhAAcpk8NgFqgCLXeHA2z7oHmA==", 2024-04-26T23:05:28.225 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.225 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:05:28.225 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-04-26T23:05:28.225 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.225 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:05:28.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:28.225 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-04-26T23:05:28.226 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmjjpcCBAAjDQRUBn/TK9hF3T2WBObyw==", 2024-04-26T23:05:28.226 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:05:28.226 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:05:28.226 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-04-26T23:05:28.226 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:05:28.226 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.226 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:28.226 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-26T23:05:28.226 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:05:28.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.210+0000 7f1c4da00640 1 -- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c2406c8f0 msgr2=0x7f1c2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.210+0000 7f1c4da00640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c2406c8f0 0x7f1c2406edb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1c38034000 tx=0x7f1c3802e000 comp rx=0 tx=0).stop 2024-04-26T23:05:28.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.210+0000 7f1c4da00640 1 -- 172.21.15.40:0/198106542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4815c5c0 msgr2=0x7f1c48151270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.210+0000 7f1c4da00640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4815c5c0 0x7f1c48151270 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000b970 tx=0x7f1c4000be30 comp rx=0 tx=0).stop 2024-04-26T23:05:28.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.210+0000 7f1c4da00640 1 -- 172.21.15.40:0/198106542 shutdown_connections 2024-04-26T23:05:28.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.210+0000 7f1c4da00640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c2406c8f0 0x7f1c2406edb0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.210+0000 7f1c4da00640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c4815c5c0 0x7f1c48151270 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.210+0000 7f1c4da00640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c4815c160 0x7f1c481f7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.210+0000 7f1c4da00640 1 --2- 172.21.15.40:0/198106542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c48073e10 0x7f1c4814bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.210+0000 7f1c4da00640 1 -- 172.21.15.40:0/198106542 >> 172.21.15.40:0/198106542 conn(0x7f1c48075eb0 msgr2=0x7f1c480b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:28.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.210+0000 7f1c4da00640 1 -- 172.21.15.40:0/198106542 shutdown_connections 2024-04-26T23:05:28.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.210+0000 7f1c4da00640 1 -- 172.21.15.40:0/198106542 wait complete. 2024-04-26T23:05:28.231 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:05:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.441+0000 7f0deac00640 1 Processor -- start 2024-04-26T23:05:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.442+0000 7f0deac00640 1 -- start start 2024-04-26T23:05:28.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.443+0000 7f0deac00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0de40d3810 0x7f0de40ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:28.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.443+0000 7f0deac00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de40700b0 con 0x7f0de40cf570 2024-04-26T23:05:28.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.443+0000 7f0deac00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de4070390 con 0x7f0de415d150 2024-04-26T23:05:28.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.443+0000 7f0deac00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f0de415d530 con 0x7f0de40d3810 2024-04-26T23:05:28.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.444+0000 7f0dea200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0de40cf570 0x7f0de40d0b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38078/0 (socket says 172.21.15.40:38078) 2024-04-26T23:05:28.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.444+0000 7f0de8e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0de40d3810 0x7f0de40ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:28.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.444+0000 7f0dea200640 1 -- 172.21.15.40:0/3443763831 learned_addr learned my addr 172.21.15.40:0/3443763831 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:28.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.444+0000 7f0de8e00640 1 -- 172.21.15.40:0/3443763831 >> v1:172.21.15.103:6789/0 conn(0x7f0de415d150 legacy=0x7f0de4070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.444+0000 7f0de8e00640 1 -- 172.21.15.40:0/3443763831 >> v1:172.21.15.40:6789/0 conn(0x7f0de40cf570 legacy=0x7f0de40d0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.444+0000 7f0de8e00640 1 -- 172.21.15.40:0/3443763831 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de4070670 con 0x7f0de40d3810 2024-04-26T23:05:28.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.444+0000 7f0de3e00640 1 -- 172.21.15.40:0/3443763831 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 651402608 0 0) 0x7f0de40700b0 con 0x7f0de40cf570 2024-04-26T23:05:28.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.445+0000 7f0de8e00640 1 --2- 172.21.15.40:0/3443763831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de40d3810 0x7f0de40ce3e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009d90 tx=0x7f0dd402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436b04003ffc73c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:28.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.446+0000 7f0de3e00640 1 -- 172.21.15.40:0/3443763831 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4007740 con 0x7f0de40d3810 2024-04-26T23:05:28.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.446+0000 7f0de3e00640 1 -- 172.21.15.40:0/3443763831 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd40078e0 con 0x7f0de40d3810 2024-04-26T23:05:28.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.446+0000 7f0de3e00640 1 -- 172.21.15.40:0/3443763831 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4007cb0 con 0x7f0de40d3810 2024-04-26T23:05:28.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.446+0000 7f0deac00640 1 -- 172.21.15.40:0/3443763831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de40d3810 msgr2=0x7f0de40ce3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.446+0000 7f0deac00640 1 --2- 172.21.15.40:0/3443763831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de40d3810 0x7f0de40ce3e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009d90 tx=0x7f0dd402a6f0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.447+0000 7f0deac00640 1 -- 172.21.15.40:0/3443763831 shutdown_connections 2024-04-26T23:05:28.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.447+0000 7f0deac00640 1 --2- 172.21.15.40:0/3443763831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de40d3810 0x7f0de40ce3e0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.447+0000 7f0deac00640 1 -- 172.21.15.40:0/3443763831 >> 172.21.15.40:0/3443763831 conn(0x7f0de40d2c10 msgr2=0x7f0de40d3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:28.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.447+0000 7f0deac00640 1 -- 172.21.15.40:0/3443763831 shutdown_connections 2024-04-26T23:05:28.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.447+0000 7f0deac00640 1 -- 172.21.15.40:0/3443763831 wait complete. 2024-04-26T23:05:28.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.449+0000 7f0deac00640 1 Processor -- start 2024-04-26T23:05:28.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.449+0000 7f0deac00640 1 -- start start 2024-04-26T23:05:28.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.450+0000 7f0deac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0de40cf570 0x7f0de40c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:28.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.450+0000 7f0deac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de40d3810 0x7f0de40b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:28.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.450+0000 7f0deac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de415d150 0x7f0de40c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:28.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.450+0000 7f0deac00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0de40cfa00 con 0x7f0de40d3810 2024-04-26T23:05:28.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.450+0000 7f0deac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0de40709e0 con 0x7f0de40cf570 2024-04-26T23:05:28.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.450+0000 7f0deac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0de40cfb80 con 0x7f0de415d150 2024-04-26T23:05:28.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.450+0000 7f0de8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de40d3810 0x7f0de40b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:28.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.450+0000 7f0de8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de40d3810 0x7f0de40b36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45676/0 (socket says 172.21.15.40:45676) 2024-04-26T23:05:28.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.450+0000 7f0de8e00640 1 -- 172.21.15.40:0/1711801523 learned_addr learned my addr 172.21.15.40:0/1711801523 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:28.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.451+0000 7f0de9800640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0de40cf570 0x7f0de40c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:28.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.451+0000 7f0dea200640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de415d150 0x7f0de40c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.451+0000 7f0de8e00640 1 -- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de415d150 msgr2=0x7f0de40c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.451+0000 7f0de8e00640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de415d150 0x7f0de40c7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.451+0000 7f0de8e00640 1 -- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0de40cf570 msgr2=0x7f0de40c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.451+0000 7f0de8e00640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0de40cf570 0x7f0de40c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.451+0000 7f0de8e00640 1 -- 172.21.15.40:0/1711801523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de40b3e30 con 0x7f0de40d3810 2024-04-26T23:05:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.451+0000 7f0de8e00640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de40d3810 0x7f0de40b36c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009d90 tx=0x7f0dd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.452+0000 7f0de1600640 1 -- 172.21.15.40:0/1711801523 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4040070 con 0x7f0de40d3810 2024-04-26T23:05:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.452+0000 7f0de1600640 1 -- 172.21.15.40:0/1711801523 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd4031d00 con 0x7f0de40d3810 2024-04-26T23:05:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.452+0000 7f0de1600640 1 -- 172.21.15.40:0/1711801523 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4003b90 con 0x7f0de40d3810 2024-04-26T23:05:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.452+0000 7f0deac00640 1 -- 172.21.15.40:0/1711801523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de40b4110 con 0x7f0de40d3810 2024-04-26T23:05:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.452+0000 7f0deac00640 1 -- 172.21.15.40:0/1711801523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de40b46d0 con 0x7f0de40d3810 2024-04-26T23:05:28.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.453+0000 7f0deac00640 1 -- 172.21.15.40:0/1711801523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de4069a80 con 0x7f0de40d3810 2024-04-26T23:05:28.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.454+0000 7f0de1600640 1 -- 172.21.15.40:0/1711801523 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0dd4039040 con 0x7f0de40d3810 2024-04-26T23:05:28.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.455+0000 7f0de1600640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0dc806c8f0 0x7f0dc806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:28.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.456+0000 7f0de1600640 1 -- 172.21.15.40:0/1711801523 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5249+0+0 (secure 0 0 0) 0x7f0dd40acde0 con 0x7f0de40d3810 2024-04-26T23:05:28.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.456+0000 7f0de9800640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0dc806c8f0 0x7f0dc806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:28.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.457+0000 7f0de9800640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0dc806c8f0 0x7f0dc806edb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8000f60 tx=0x7f0dd8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.460+0000 7f0de1600640 1 -- 172.21.15.40:0/1711801523 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0dd407b330 con 0x7f0de40d3810 2024-04-26T23:05:28.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.664+0000 7f0deac00640 1 -- 172.21.15.40:0/1711801523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0de40b8b90 con 0x7f0de40d3810 2024-04-26T23:05:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.665+0000 7f0de1600640 1 -- 172.21.15.40:0/1711801523 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+13128 (secure 0 0 0) 0x7f0dd407ac30 con 0x7f0de40d3810 2024-04-26T23:05:28.666 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:28.666 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":61,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:05:27.691581+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:05:28.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.668+0000 7f0deac00640 1 -- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0dc806c8f0 msgr2=0x7f0dc806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.668+0000 7f0deac00640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0dc806c8f0 0x7f0dc806edb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8000f60 tx=0x7f0dd8009040 comp rx=0 tx=0).stop 2024-04-26T23:05:28.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.668+0000 7f0deac00640 1 -- 172.21.15.40:0/1711801523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de40d3810 msgr2=0x7f0de40b36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.668+0000 7f0deac00640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de40d3810 0x7f0de40b36c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009d90 tx=0x7f0dd4004a40 comp rx=0 tx=0).stop 2024-04-26T23:05:28.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.668+0000 7f0deac00640 1 -- 172.21.15.40:0/1711801523 shutdown_connections 2024-04-26T23:05:28.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.668+0000 7f0deac00640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0dc806c8f0 0x7f0dc806edb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.668+0000 7f0deac00640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de415d150 0x7f0de40c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.668+0000 7f0deac00640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de40d3810 0x7f0de40b36c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.668+0000 7f0deac00640 1 --2- 172.21.15.40:0/1711801523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0de40cf570 0x7f0de40c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.668+0000 7f0deac00640 1 -- 172.21.15.40:0/1711801523 >> 172.21.15.40:0/1711801523 conn(0x7f0de40d2c10 msgr2=0x7f0de40b0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:28.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.669+0000 7f0deac00640 1 -- 172.21.15.40:0/1711801523 shutdown_connections 2024-04-26T23:05:28.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.669+0000 7f0deac00640 1 -- 172.21.15.40:0/1711801523 wait complete. 2024-04-26T23:05:28.690 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:05:28.691 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:05:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.907+0000 7f4cb3000640 1 Processor -- start 2024-04-26T23:05:28.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.908+0000 7f4cb3000640 1 -- start start 2024-04-26T23:05:28.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.909+0000 7f4cb3000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4cac0cfaf0 0x7f4cac073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:28.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.909+0000 7f4cb3000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cac161880 con 0x7f4cac160a50 2024-04-26T23:05:28.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.909+0000 7f4cb3000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cac0cff80 con 0x7f4cac06dd40 2024-04-26T23:05:28.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.909+0000 7f4cb3000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4cac071ce0 con 0x7f4cac0cfaf0 2024-04-26T23:05:28.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.910+0000 7f4cb1c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4cac160a50 0x7f4cac160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38082/0 (socket says 172.21.15.40:38082) 2024-04-26T23:05:28.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.910+0000 7f4cb1c00640 1 -- 172.21.15.40:0/372244276 learned_addr learned my addr 172.21.15.40:0/372244276 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:28.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.910+0000 7f4cb1200640 1 --2- 172.21.15.40:0/372244276 >> v2:172.21.15.103:3300/0 conn(0x7f4cac0cfaf0 0x7f4cac073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:28.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.910+0000 7f4cabe00640 1 -- 172.21.15.40:0/372244276 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3581870125 0 0) 0x7f4cac161880 con 0x7f4cac160a50 2024-04-26T23:05:28.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.910+0000 7f4cabe00640 1 -- 172.21.15.40:0/372244276 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c90003520 con 0x7f4cac160a50 2024-04-26T23:05:28.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.911+0000 7f4cb1200640 1 -- 172.21.15.40:0/372244276 >> v1:172.21.15.103:6789/0 conn(0x7f4cac06dd40 legacy=0x7f4cac072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.911+0000 7f4cb1200640 1 -- 172.21.15.40:0/372244276 >> v1:172.21.15.40:6789/0 conn(0x7f4cac160a50 legacy=0x7f4cac160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.911+0000 7f4cabe00640 1 -- 172.21.15.40:0/372244276 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1646893975 0 0) 0x7f4c90003520 con 0x7f4cac160a50 2024-04-26T23:05:28.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.911+0000 7f4cb1200640 1 -- 172.21.15.40:0/372244276 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cac0d0260 con 0x7f4cac0cfaf0 2024-04-26T23:05:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.912+0000 7f4cb1200640 1 --2- 172.21.15.40:0/372244276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cac0cfaf0 0x7f4cac073a50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4c980097a0 tx=0x7f4c9802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1a489276bc14d44 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:28.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.913+0000 7f4cabe00640 1 -- 172.21.15.40:0/372244276 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c98037070 con 0x7f4cac0cfaf0 2024-04-26T23:05:28.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.913+0000 7f4cabe00640 1 -- 172.21.15.40:0/372244276 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c9802ac50 con 0x7f4cac0cfaf0 2024-04-26T23:05:28.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.913+0000 7f4cabe00640 1 -- 172.21.15.40:0/372244276 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c0019f0 con 0x7f4cac0cfaf0 2024-04-26T23:05:28.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.913+0000 7f4cb3000640 1 -- 172.21.15.40:0/372244276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cac0cfaf0 msgr2=0x7f4cac073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.913+0000 7f4cb3000640 1 --2- 172.21.15.40:0/372244276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cac0cfaf0 0x7f4cac073a50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4c980097a0 tx=0x7f4c9802a1c0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.913+0000 7f4cb3000640 1 -- 172.21.15.40:0/372244276 shutdown_connections 2024-04-26T23:05:28.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.913+0000 7f4cb3000640 1 --2- 172.21.15.40:0/372244276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cac0cfaf0 0x7f4cac073a50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.913+0000 7f4cb3000640 1 -- 172.21.15.40:0/372244276 >> 172.21.15.40:0/372244276 conn(0x7f4cac0b2720 msgr2=0x7f4cac0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:28.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.913+0000 7f4cb3000640 1 -- 172.21.15.40:0/372244276 shutdown_connections 2024-04-26T23:05:28.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.913+0000 7f4cb3000640 1 -- 172.21.15.40:0/372244276 wait complete. 2024-04-26T23:05:28.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.915+0000 7f4cb3000640 1 Processor -- start 2024-04-26T23:05:28.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.915+0000 7f4cb3000640 1 -- start start 2024-04-26T23:05:28.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.916+0000 7f4cb3000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cac06dd40 0x7f4cac1f4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:28.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.916+0000 7f4cb3000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cac160a50 0x7f4cac15a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:28.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.916+0000 7f4cb3000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cac15bcf0 0x7f4cac15c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:28.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.916+0000 7f4cb3000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4cac073f90 con 0x7f4cac06dd40 2024-04-26T23:05:28.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.916+0000 7f4cb3000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4cac0722a0 con 0x7f4cac15bcf0 2024-04-26T23:05:28.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.916+0000 7f4cb3000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4cac06e1f0 con 0x7f4cac160a50 2024-04-26T23:05:28.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.916+0000 7f4cb1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cac06dd40 0x7f4cac1f4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:28.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.917+0000 7f4cb1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cac06dd40 0x7f4cac1f4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45684/0 (socket says 172.21.15.40:45684) 2024-04-26T23:05:28.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.917+0000 7f4cb1c00640 1 -- 172.21.15.40:0/73542325 learned_addr learned my addr 172.21.15.40:0/73542325 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:28.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.917+0000 7f4cb1200640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cac160a50 0x7f4cac15a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:28.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.917+0000 7f4cb2600640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cac15bcf0 0x7f4cac15c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:28.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.917+0000 7f4cb1c00640 1 -- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cac160a50 msgr2=0x7f4cac15a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.917+0000 7f4cb1c00640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cac160a50 0x7f4cac15a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.917+0000 7f4cb1c00640 1 -- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cac15bcf0 msgr2=0x7f4cac15c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:28.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.917+0000 7f4cb1c00640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cac15bcf0 0x7f4cac15c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:28.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.917+0000 7f4cb1c00640 1 -- 172.21.15.40:0/73542325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cac13fb50 con 0x7f4cac06dd40 2024-04-26T23:05:28.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.917+0000 7f4cb2600640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cac15bcf0 0x7f4cac15c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:28.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.917+0000 7f4cb1c00640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cac06dd40 0x7f4cac1f4fc0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00ccc0 tx=0x7f4c9c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:28.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.917+0000 7f4ca9600640 1 -- 172.21.15.40:0/73542325 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c017070 con 0x7f4cac06dd40 2024-04-26T23:05:28.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.918+0000 7f4ca9600640 1 -- 172.21.15.40:0/73542325 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c9c012400 con 0x7f4cac06dd40 2024-04-26T23:05:28.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.918+0000 7f4ca9600640 1 -- 172.21.15.40:0/73542325 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c0127f0 con 0x7f4cac06dd40 2024-04-26T23:05:28.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.918+0000 7f4cb3000640 1 -- 172.21.15.40:0/73542325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cac13fe30 con 0x7f4cac06dd40 2024-04-26T23:05:28.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.918+0000 7f4cb3000640 1 -- 172.21.15.40:0/73542325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cac0b34e0 con 0x7f4cac06dd40 2024-04-26T23:05:28.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.919+0000 7f4cb3000640 1 -- 172.21.15.40:0/73542325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cac069370 con 0x7f4cac06dd40 2024-04-26T23:05:28.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.920+0000 7f4ca9600640 1 -- 172.21.15.40:0/73542325 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4c9c01a070 con 0x7f4cac06dd40 2024-04-26T23:05:28.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.921+0000 7f4ca9600640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4c8c06c8f0 0x7f4c8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:28.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.921+0000 7f4ca9600640 1 -- 172.21.15.40:0/73542325 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5249+0+0 (secure 0 0 0) 0x7f4c9c08b960 con 0x7f4cac06dd40 2024-04-26T23:05:28.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.922+0000 7f4cb1200640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4c8c06c8f0 0x7f4c8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:28.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.923+0000 7f4cb1200640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4c8c06c8f0 0x7f4c8c06edb0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4c98034000 tx=0x7f4c9802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:28.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:28.926+0000 7f4ca9600640 1 -- 172.21.15.40:0/73542325 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4c9c059eb0 con 0x7f4cac06dd40 2024-04-26T23:05:29.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.133+0000 7f4cb3000640 1 -- 172.21.15.40:0/73542325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4cac069b80 con 0x7f4cac06dd40 2024-04-26T23:05:29.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.133+0000 7f4ca9600640 1 -- 172.21.15.40:0/73542325 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) ==== 93+0+10 (secure 0 0 0) 0x7f4c9c0597b0 con 0x7f4cac06dd40 2024-04-26T23:05:29.134 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:05:29.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.136+0000 7f4cb3000640 1 -- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4c8c06c8f0 msgr2=0x7f4c8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.136+0000 7f4cb3000640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4c8c06c8f0 0x7f4c8c06edb0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4c98034000 tx=0x7f4c9802e000 comp rx=0 tx=0).stop 2024-04-26T23:05:29.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.136+0000 7f4cb3000640 1 -- 172.21.15.40:0/73542325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cac06dd40 msgr2=0x7f4cac1f4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.136+0000 7f4cb3000640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cac06dd40 0x7f4cac1f4fc0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00ccc0 tx=0x7f4c9c00a430 comp rx=0 tx=0).stop 2024-04-26T23:05:29.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.137+0000 7f4cb3000640 1 -- 172.21.15.40:0/73542325 shutdown_connections 2024-04-26T23:05:29.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.137+0000 7f4cb3000640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4c8c06c8f0 0x7f4c8c06edb0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.137+0000 7f4cb3000640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cac15bcf0 0x7f4cac15c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.137+0000 7f4cb3000640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4cac160a50 0x7f4cac15a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.137+0000 7f4cb3000640 1 --2- 172.21.15.40:0/73542325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cac06dd40 0x7f4cac1f4fc0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.137+0000 7f4cb3000640 1 -- 172.21.15.40:0/73542325 >> 172.21.15.40:0/73542325 conn(0x7f4cac0b2720 msgr2=0x7f4cac140ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:29.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.137+0000 7f4cb3000640 1 -- 172.21.15.40:0/73542325 shutdown_connections 2024-04-26T23:05:29.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.137+0000 7f4cb3000640 1 -- 172.21.15.40:0/73542325 wait complete. 2024-04-26T23:05:29.157 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T23:05:29.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.368+0000 7efd7de00640 1 Processor -- start 2024-04-26T23:05:29.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.368+0000 7efd7de00640 1 -- start start 2024-04-26T23:05:29.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.369+0000 7efd7de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd780d06b0 0x7efd780d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.369+0000 7efd7de00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd780b59f0 con 0x7efd78146060 2024-04-26T23:05:29.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.369+0000 7efd7de00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd780b4180 con 0x7efd78145970 2024-04-26T23:05:29.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.369+0000 7efd7de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efd780d1060 con 0x7efd780d06b0 2024-04-26T23:05:29.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.370+0000 7efd77e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efd78146060 0x7efd780b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38094/0 (socket says 172.21.15.40:38094) 2024-04-26T23:05:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.370+0000 7efd77e00640 1 -- 172.21.15.40:0/2902768979 learned_addr learned my addr 172.21.15.40:0/2902768979 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.370+0000 7efd7d400640 1 --2- 172.21.15.40:0/2902768979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd780d06b0 0x7efd780d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.370+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 219258681 0 0) 0x7efd780b59f0 con 0x7efd78146060 2024-04-26T23:05:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.370+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd60003520 con 0x7efd78146060 2024-04-26T23:05:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3980361727 0 0) 0x7efd60003520 con 0x7efd78146060 2024-04-26T23:05:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd780d06b0 msgr2=0x7efd780d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77400640 1 --2- 172.21.15.40:0/2902768979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd780d06b0 0x7efd780d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 >> v1:172.21.15.103:6789/0 conn(0x7efd78145970 legacy=0x7efd781612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd780b5f40 con 0x7efd78146060 2024-04-26T23:05:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7efd64001670 con 0x7efd78146060 2024-04-26T23:05:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 >> v1:172.21.15.40:6789/0 conn(0x7efd78146060 legacy=0x7efd780b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77400640 1 --2- 172.21.15.40:0/2902768979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd780d06b0 0x7efd600039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77400640 1 --2- 172.21.15.40:0/2902768979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd60003fb0 0x7efd60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77400640 1 --2- 172.21.15.40:0/2902768979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd60004d40 0x7efd600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efd780d1060 con 0x7efd60003fb0 2024-04-26T23:05:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efd6000bb00 con 0x7efd60004d40 2024-04-26T23:05:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efd6000bd00 con 0x7efd780d06b0 2024-04-26T23:05:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.371+0000 7efd77e00640 1 --2- 172.21.15.40:0/2902768979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd60003fb0 0x7efd60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.372+0000 7efd77e00640 1 -- 172.21.15.40:0/2902768979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd780d06b0 msgr2=0x7efd600039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.372+0000 7efd77e00640 1 --2- 172.21.15.40:0/2902768979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd780d06b0 0x7efd600039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.372+0000 7efd77e00640 1 -- 172.21.15.40:0/2902768979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd60004d40 msgr2=0x7efd600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.372+0000 7efd77e00640 1 --2- 172.21.15.40:0/2902768979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd60004d40 0x7efd600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.372+0000 7efd77e00640 1 -- 172.21.15.40:0/2902768979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd60003520 con 0x7efd60003fb0 2024-04-26T23:05:29.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.373+0000 7efd77e00640 1 --2- 172.21.15.40:0/2902768979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd60003fb0 0x7efd60004800 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7efd6400c190 tx=0x7efd60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4fec0d8ee5634b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:29.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.373+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd64007c60 con 0x7efd60003fb0 2024-04-26T23:05:29.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.373+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd64005ad0 con 0x7efd60003fb0 2024-04-26T23:05:29.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.373+0000 7efd77400640 1 -- 172.21.15.40:0/2902768979 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd64007070 con 0x7efd60003fb0 2024-04-26T23:05:29.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.373+0000 7efd7de00640 1 -- 172.21.15.40:0/2902768979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd60003fb0 msgr2=0x7efd60004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.373+0000 7efd7de00640 1 --2- 172.21.15.40:0/2902768979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd60003fb0 0x7efd60004800 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7efd6400c190 tx=0x7efd60005830 comp rx=0 tx=0).stop 2024-04-26T23:05:29.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.373+0000 7efd7d400640 1 -- 172.21.15.40:0/2902768979 reap_dead start 2024-04-26T23:05:29.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.374+0000 7efd7de00640 1 -- 172.21.15.40:0/2902768979 shutdown_connections 2024-04-26T23:05:29.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.374+0000 7efd7de00640 1 -- 172.21.15.40:0/2902768979 >> 172.21.15.40:0/2902768979 conn(0x7efd78075eb0 msgr2=0x7efd78068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:29.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.374+0000 7efd7de00640 1 -- 172.21.15.40:0/2902768979 shutdown_connections 2024-04-26T23:05:29.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.374+0000 7efd7de00640 1 -- 172.21.15.40:0/2902768979 wait complete. 2024-04-26T23:05:29.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.375+0000 7efd7de00640 1 Processor -- start 2024-04-26T23:05:29.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.376+0000 7efd7de00640 1 -- start start 2024-04-26T23:05:29.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd7de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd60003fb0 0x7efd780b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd7de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd781355e0 0x7efd7813b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd7de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd781f1630 0x7efd781f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd7de00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efd780cf100 con 0x7efd781355e0 2024-04-26T23:05:29.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd7de00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efd780cf3e0 con 0x7efd781f1630 2024-04-26T23:05:29.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd7de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efd78145d50 con 0x7efd60003fb0 2024-04-26T23:05:29.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd77e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd781355e0 0x7efd7813b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd77e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd781355e0 0x7efd7813b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45702/0 (socket says 172.21.15.40:45702) 2024-04-26T23:05:29.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd77e00640 1 -- 172.21.15.40:0/2268962794 learned_addr learned my addr 172.21.15.40:0/2268962794 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd7ca00640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd60003fb0 0x7efd780b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd7d400640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd781f1630 0x7efd781f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd77e00640 1 -- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd60003fb0 msgr2=0x7efd780b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd77e00640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd60003fb0 0x7efd780b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd77e00640 1 -- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd781f1630 msgr2=0x7efd781f1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd77e00640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd781f1630 0x7efd781f1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd77e00640 1 -- 172.21.15.40:0/2268962794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd78136310 con 0x7efd781355e0 2024-04-26T23:05:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.377+0000 7efd7d400640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd781f1630 0x7efd781f1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.378+0000 7efd77e00640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd781355e0 0x7efd7813b170 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7efd6400c1f0 tx=0x7efd64001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.378+0000 7efd74c00640 1 -- 172.21.15.40:0/2268962794 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd64040020 con 0x7efd781355e0 2024-04-26T23:05:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.378+0000 7efd74c00640 1 -- 172.21.15.40:0/2268962794 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd640013b0 con 0x7efd781355e0 2024-04-26T23:05:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.378+0000 7efd7de00640 1 -- 172.21.15.40:0/2268962794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd78135a70 con 0x7efd781355e0 2024-04-26T23:05:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.378+0000 7efd74c00640 1 -- 172.21.15.40:0/2268962794 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd64037040 con 0x7efd781355e0 2024-04-26T23:05:29.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.378+0000 7efd7de00640 1 -- 172.21.15.40:0/2268962794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd78136030 con 0x7efd781355e0 2024-04-26T23:05:29.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.380+0000 7efd7de00640 1 -- 172.21.15.40:0/2268962794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd781f1270 con 0x7efd781355e0 2024-04-26T23:05:29.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.380+0000 7efd74c00640 1 -- 172.21.15.40:0/2268962794 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efd64043020 con 0x7efd781355e0 2024-04-26T23:05:29.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.381+0000 7efd74c00640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efd5806c8f0 0x7efd5806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.381+0000 7efd74c00640 1 -- 172.21.15.40:0/2268962794 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5249+0+0 (secure 0 0 0) 0x7efd640ab7f0 con 0x7efd781355e0 2024-04-26T23:05:29.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.382+0000 7efd7ca00640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efd5806c8f0 0x7efd5806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.383+0000 7efd7ca00640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efd5806c8f0 0x7efd5806edb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efd6c007fd0 tx=0x7efd6c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:29.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.386+0000 7efd74c00640 1 -- 172.21.15.40:0/2268962794 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7efd78136310 con 0x7efd781355e0 2024-04-26T23:05:29.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.591+0000 7efd7de00640 1 -- 172.21.15.40:0/2268962794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7efd781f0f00 con 0x7efd781355e0 2024-04-26T23:05:29.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.592+0000 7efd74c00640 1 -- 172.21.15.40:0/2268962794 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v61) ==== 84+0+65 (secure 0 0 0) 0x7efd64005aa0 con 0x7efd781355e0 2024-04-26T23:05:29.592 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:29.593 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-04-26T23:05:29.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:05:29.593 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-04-26T23:05:29.593 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-04-26T23:05:29.593 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:05:29.593 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-04-26T23:05:29.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.595+0000 7efd7de00640 1 -- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efd5806c8f0 msgr2=0x7efd5806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.595+0000 7efd7de00640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efd5806c8f0 0x7efd5806edb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efd6c007fd0 tx=0x7efd6c009040 comp rx=0 tx=0).stop 2024-04-26T23:05:29.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.595+0000 7efd7de00640 1 -- 172.21.15.40:0/2268962794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd781355e0 msgr2=0x7efd7813b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.595+0000 7efd7de00640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd781355e0 0x7efd7813b170 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7efd6400c1f0 tx=0x7efd64001970 comp rx=0 tx=0).stop 2024-04-26T23:05:29.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.595+0000 7efd7de00640 1 -- 172.21.15.40:0/2268962794 shutdown_connections 2024-04-26T23:05:29.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.595+0000 7efd7de00640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efd5806c8f0 0x7efd5806edb0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.595+0000 7efd7de00640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd781f1630 0x7efd781f1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.595+0000 7efd7de00640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd781355e0 0x7efd7813b170 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.595+0000 7efd7de00640 1 --2- 172.21.15.40:0/2268962794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efd60003fb0 0x7efd780b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.595+0000 7efd7de00640 1 -- 172.21.15.40:0/2268962794 >> 172.21.15.40:0/2268962794 conn(0x7efd78075eb0 msgr2=0x7efd78141550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:29.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.595+0000 7efd7de00640 1 -- 172.21.15.40:0/2268962794 shutdown_connections 2024-04-26T23:05:29.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.595+0000 7efd7de00640 1 -- 172.21.15.40:0/2268962794 wait complete. 2024-04-26T23:05:29.616 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T23:05:29.616 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T23:05:29.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.832+0000 7fe311400640 1 Processor -- start 2024-04-26T23:05:29.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.832+0000 7fe311400640 1 -- start start 2024-04-26T23:05:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.834+0000 7fe311400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe30c15f3e0 0x7fe30c0d2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.834+0000 7fe311400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe30c160930 con 0x7fe30c0d31e0 2024-04-26T23:05:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.834+0000 7fe311400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe30c15fd20 con 0x7fe30c0d3da0 2024-04-26T23:05:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.834+0000 7fe311400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe30c160000 con 0x7fe30c15f3e0 2024-04-26T23:05:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.835+0000 7fe309e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe30c0d31e0 0x7fe30c0d3690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38104/0 (socket says 172.21.15.40:38104) 2024-04-26T23:05:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.835+0000 7fe309e00640 1 -- 172.21.15.40:0/95337762 learned_addr learned my addr 172.21.15.40:0/95337762 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:29.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.835+0000 7fe30a800640 1 --2- 172.21.15.40:0/95337762 >> v2:172.21.15.103:3300/0 conn(0x7fe30c15f3e0 0x7fe30c0d2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.835+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2429659838 0 0) 0x7fe30c160930 con 0x7fe30c0d31e0 2024-04-26T23:05:29.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.835+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2f0003520 con 0x7fe30c0d31e0 2024-04-26T23:05:29.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.835+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1529648595 0 0) 0x7fe2f0003520 con 0x7fe30c0d31e0 2024-04-26T23:05:29.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 >> v2:172.21.15.103:3300/0 conn(0x7fe30c15f3e0 msgr2=0x7fe30c0d2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:29.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe309400640 1 --2- 172.21.15.40:0/95337762 >> v2:172.21.15.103:3300/0 conn(0x7fe30c15f3e0 0x7fe30c0d2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 >> v1:172.21.15.103:6789/0 conn(0x7fe30c0d3da0 legacy=0x7fe30c1620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30c1602e0 con 0x7fe30c0d31e0 2024-04-26T23:05:29.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fe2f8001690 con 0x7fe30c0d31e0 2024-04-26T23:05:29.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 >> v1:172.21.15.40:6789/0 conn(0x7fe30c0d31e0 legacy=0x7fe30c0d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe309400640 1 --2- 172.21.15.40:0/95337762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f0003a00 0x7fe2f0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe309400640 1 --2- 172.21.15.40:0/95337762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f0004800 0x7fe2f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe309400640 1 --2- 172.21.15.40:0/95337762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2f0005210 0x7fe2f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c160000 con 0x7fe2f0005210 2024-04-26T23:05:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f000c160 con 0x7fe2f0003a00 2024-04-26T23:05:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f000c360 con 0x7fe2f0004800 2024-04-26T23:05:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe30b200640 1 --2- 172.21.15.40:0/95337762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2f0005210 0x7fe2f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe30a800640 1 --2- 172.21.15.40:0/95337762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f0003a00 0x7fe2f0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.836+0000 7fe309e00640 1 --2- 172.21.15.40:0/95337762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f0004800 0x7fe2f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.837+0000 7fe30b200640 1 -- 172.21.15.40:0/95337762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f0004800 msgr2=0x7fe2f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.837+0000 7fe30b200640 1 --2- 172.21.15.40:0/95337762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f0004800 0x7fe2f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.837+0000 7fe30b200640 1 -- 172.21.15.40:0/95337762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f0003a00 msgr2=0x7fe2f0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:29.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.837+0000 7fe30b200640 1 --2- 172.21.15.40:0/95337762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f0003a00 0x7fe2f0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.837+0000 7fe30b200640 1 -- 172.21.15.40:0/95337762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f0003520 con 0x7fe2f0005210 2024-04-26T23:05:29.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.837+0000 7fe309e00640 1 --2- 172.21.15.40:0/95337762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f0004800 0x7fe2f0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.837+0000 7fe30b200640 1 -- 172.21.15.40:0/95337762 reap_dead start 2024-04-26T23:05:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.838+0000 7fe30b200640 1 --2- 172.21.15.40:0/95337762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2f0005210 0x7fe2f000b9c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fe30c164000 tx=0x7fe2f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d1296fa4aa2e7eb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.838+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304001a70 con 0x7fe2f0005210 2024-04-26T23:05:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.838+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe30402ad80 con 0x7fe2f0005210 2024-04-26T23:05:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.838+0000 7fe309400640 1 -- 172.21.15.40:0/95337762 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304003bb0 con 0x7fe2f0005210 2024-04-26T23:05:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.839+0000 7fe311400640 1 -- 172.21.15.40:0/95337762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2f0005210 msgr2=0x7fe2f000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.839+0000 7fe311400640 1 --2- 172.21.15.40:0/95337762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2f0005210 0x7fe2f000b9c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fe30c164000 tx=0x7fe2f0005830 comp rx=0 tx=0).stop 2024-04-26T23:05:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.839+0000 7fe311400640 1 -- 172.21.15.40:0/95337762 shutdown_connections 2024-04-26T23:05:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.839+0000 7fe311400640 1 --2- 172.21.15.40:0/95337762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2f0005210 0x7fe2f000b9c0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.839+0000 7fe311400640 1 -- 172.21.15.40:0/95337762 >> 172.21.15.40:0/95337762 conn(0x7fe30c09e3c0 msgr2=0x7fe30c0d50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.839+0000 7fe311400640 1 -- 172.21.15.40:0/95337762 shutdown_connections 2024-04-26T23:05:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.839+0000 7fe311400640 1 -- 172.21.15.40:0/95337762 wait complete. 2024-04-26T23:05:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.841+0000 7fe311400640 1 Processor -- start 2024-04-26T23:05:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.841+0000 7fe311400640 1 -- start start 2024-04-26T23:05:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.843+0000 7fe311400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f0005210 0x7fe30c1597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.843+0000 7fe311400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe30c159f80 0x7fe30c0a2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.843+0000 7fe311400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe30c0a2bb0 0x7fe30c0b9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.843+0000 7fe311400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c15f7c0 con 0x7fe30c0a2bb0 2024-04-26T23:05:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.843+0000 7fe311400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c160d40 con 0x7fe30c159f80 2024-04-26T23:05:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.843+0000 7fe311400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe30c15f940 con 0x7fe2f0005210 2024-04-26T23:05:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.843+0000 7fe30b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe30c0a2bb0 0x7fe30c0b9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.843+0000 7fe30b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe30c0a2bb0 0x7fe30c0b9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45706/0 (socket says 172.21.15.40:45706) 2024-04-26T23:05:29.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.843+0000 7fe30b200640 1 -- 172.21.15.40:0/3950793246 learned_addr learned my addr 172.21.15.40:0/3950793246 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:29.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.844+0000 7fe309e00640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe30c159f80 0x7fe30c0a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.844+0000 7fe30a800640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f0005210 0x7fe30c1597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.844+0000 7fe30b200640 1 -- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f0005210 msgr2=0x7fe30c1597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.844+0000 7fe30b200640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f0005210 0x7fe30c1597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.844+0000 7fe30b200640 1 -- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe30c159f80 msgr2=0x7fe30c0a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:29.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.844+0000 7fe30b200640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe30c159f80 0x7fe30c0a2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:29.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.844+0000 7fe30b200640 1 -- 172.21.15.40:0/3950793246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30c15c800 con 0x7fe30c0a2bb0 2024-04-26T23:05:29.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.844+0000 7fe30b200640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe30c0a2bb0 0x7fe30c0b9d00 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fe304005300 tx=0x7fe3040014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:29.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.844+0000 7fe302a00640 1 -- 172.21.15.40:0/3950793246 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30402ab80 con 0x7fe30c0a2bb0 2024-04-26T23:05:29.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.844+0000 7fe302a00640 1 -- 172.21.15.40:0/3950793246 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3040055f0 con 0x7fe30c0a2bb0 2024-04-26T23:05:29.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.844+0000 7fe302a00640 1 -- 172.21.15.40:0/3950793246 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304032440 con 0x7fe30c0a2bb0 2024-04-26T23:05:29.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.845+0000 7fe311400640 1 -- 172.21.15.40:0/3950793246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe30c0c8d60 con 0x7fe30c0a2bb0 2024-04-26T23:05:29.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.845+0000 7fe311400640 1 -- 172.21.15.40:0/3950793246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe30c0c9320 con 0x7fe30c0a2bb0 2024-04-26T23:05:29.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.846+0000 7fe311400640 1 -- 172.21.15.40:0/3950793246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe30c069600 con 0x7fe30c0a2bb0 2024-04-26T23:05:29.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.847+0000 7fe302a00640 1 -- 172.21.15.40:0/3950793246 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe30403c030 con 0x7fe30c0a2bb0 2024-04-26T23:05:29.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.848+0000 7fe302a00640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe2ec06c8f0 0x7fe2ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:29.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.848+0000 7fe302a00640 1 -- 172.21.15.40:0/3950793246 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5249+0+0 (secure 0 0 0) 0x7fe3040aa8b0 con 0x7fe30c0a2bb0 2024-04-26T23:05:29.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.849+0000 7fe30a800640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe2ec06c8f0 0x7fe2ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:29.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.850+0000 7fe30a800640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe2ec06c8f0 0x7fe2ec06edb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009990 tx=0x7fe2fc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:29.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:29.852+0000 7fe302a00640 1 -- 172.21.15.40:0/3950793246 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe3040787b0 con 0x7fe30c0a2bb0 2024-04-26T23:05:30.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.067+0000 7fe311400640 1 -- 172.21.15.40:0/3950793246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fe30c0d2680 con 0x7fe30c0a2bb0 2024-04-26T23:05:30.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.711+0000 7fe302a00640 1 -- 172.21.15.40:0/3950793246 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v62) ==== 138+0+0 (secure 0 0 0) 0x7fe30407cab0 con 0x7fe30c0a2bb0 2024-04-26T23:05:30.711 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-04-26T23:05:30.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.714+0000 7fe311400640 1 -- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe2ec06c8f0 msgr2=0x7fe2ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:30.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.714+0000 7fe311400640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe2ec06c8f0 0x7fe2ec06edb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009990 tx=0x7fe2fc008040 comp rx=0 tx=0).stop 2024-04-26T23:05:30.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.714+0000 7fe311400640 1 -- 172.21.15.40:0/3950793246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe30c0a2bb0 msgr2=0x7fe30c0b9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:30.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.714+0000 7fe311400640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe30c0a2bb0 0x7fe30c0b9d00 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fe304005300 tx=0x7fe3040014a0 comp rx=0 tx=0).stop 2024-04-26T23:05:30.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.715+0000 7fe311400640 1 -- 172.21.15.40:0/3950793246 shutdown_connections 2024-04-26T23:05:30.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.715+0000 7fe311400640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe2ec06c8f0 0x7fe2ec06edb0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:30.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.715+0000 7fe311400640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe30c0a2bb0 0x7fe30c0b9d00 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:30.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.715+0000 7fe311400640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe30c159f80 0x7fe30c0a2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.715+0000 7fe311400640 1 --2- 172.21.15.40:0/3950793246 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2f0005210 0x7fe30c1597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.715+0000 7fe311400640 1 -- 172.21.15.40:0/3950793246 >> 172.21.15.40:0/3950793246 conn(0x7fe30c09e3c0 msgr2=0x7fe30c1614d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.715+0000 7fe311400640 1 -- 172.21.15.40:0/3950793246 shutdown_connections 2024-04-26T23:05:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.715+0000 7fe311400640 1 -- 172.21.15.40:0/3950793246 wait complete. 2024-04-26T23:05:30.742 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T23:05:30.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.953+0000 7faf20a00640 1 Processor -- start 2024-04-26T23:05:30.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.953+0000 7faf20a00640 1 -- start start 2024-04-26T23:05:30.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.955+0000 7faf20a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faf1c145970 0x7faf1c1612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:30.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.955+0000 7faf20a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faf1c0d06b0 0x7faf1c0d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:30.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.955+0000 7faf20a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf1c0b59f0 con 0x7faf1c146060 2024-04-26T23:05:30.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.955+0000 7faf20a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7faf1c0ceed0 con 0x7faf1c0d06b0 2024-04-26T23:05:30.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.955+0000 7faf20a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7faf1c0d1020 con 0x7faf1c145970 2024-04-26T23:05:30.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.955+0000 7faf1aa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7faf1c146060 0x7faf1c0b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38114/0 (socket says 172.21.15.40:38114) 2024-04-26T23:05:30.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.955+0000 7faf1aa00640 1 -- 172.21.15.40:0/48268880 learned_addr learned my addr 172.21.15.40:0/48268880 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:30.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.955+0000 7faf1be00640 1 --2- 172.21.15.40:0/48268880 >> v2:172.21.15.40:3300/0 conn(0x7faf1c0d06b0 0x7faf1c0d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:30.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.956+0000 7faf1b400640 1 --2- 172.21.15.40:0/48268880 >> v2:172.21.15.103:3300/0 conn(0x7faf1c145970 0x7faf1c1612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:30.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.956+0000 7faf1be00640 1 -- 172.21.15.40:0/48268880 >> v2:172.21.15.103:3300/0 conn(0x7faf1c145970 msgr2=0x7faf1c1612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:30.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.956+0000 7faf1be00640 1 --2- 172.21.15.40:0/48268880 >> v2:172.21.15.103:3300/0 conn(0x7faf1c145970 0x7faf1c1612b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:30.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.956+0000 7faf1be00640 1 -- 172.21.15.40:0/48268880 >> v1:172.21.15.40:6789/0 conn(0x7faf1c146060 legacy=0x7faf1c0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:30.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.956+0000 7faf1be00640 1 -- 172.21.15.40:0/48268880 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf1c1617f0 con 0x7faf1c0d06b0 2024-04-26T23:05:30.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.956+0000 7faf1a000640 1 -- 172.21.15.40:0/48268880 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3336064023 0 0) 0x7faf1c0b59f0 con 0x7faf1c146060 2024-04-26T23:05:30.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.957+0000 7faf1be00640 1 --2- 172.21.15.40:0/48268880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c0d06b0 0x7faf1c0d0ae0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0d17b0 tx=0x7faf1402a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b304eaad44b753 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:30.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.957+0000 7faf1a000640 1 -- 172.21.15.40:0/48268880 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf14007740 con 0x7faf1c0d06b0 2024-04-26T23:05:30.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.958+0000 7faf1a000640 1 -- 172.21.15.40:0/48268880 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf140078e0 con 0x7faf1c0d06b0 2024-04-26T23:05:30.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.958+0000 7faf1a000640 1 -- 172.21.15.40:0/48268880 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf14007cb0 con 0x7faf1c0d06b0 2024-04-26T23:05:30.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.958+0000 7faf20a00640 1 -- 172.21.15.40:0/48268880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c0d06b0 msgr2=0x7faf1c0d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:30.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.958+0000 7faf20a00640 1 --2- 172.21.15.40:0/48268880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c0d06b0 0x7faf1c0d0ae0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0d17b0 tx=0x7faf1402a6b0 comp rx=0 tx=0).stop 2024-04-26T23:05:30.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.958+0000 7faf20a00640 1 -- 172.21.15.40:0/48268880 shutdown_connections 2024-04-26T23:05:30.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.958+0000 7faf20a00640 1 --2- 172.21.15.40:0/48268880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c0d06b0 0x7faf1c0d0ae0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:30.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.958+0000 7faf20a00640 1 --2- 172.21.15.40:0/48268880 >> v2:172.21.15.103:3300/0 conn(0x7faf1c145970 0x7faf1c1612b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:30.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.958+0000 7faf20a00640 1 -- 172.21.15.40:0/48268880 >> 172.21.15.40:0/48268880 conn(0x7faf1c075eb0 msgr2=0x7faf1c068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:30.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.958+0000 7faf20a00640 1 -- 172.21.15.40:0/48268880 shutdown_connections 2024-04-26T23:05:30.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.958+0000 7faf20a00640 1 -- 172.21.15.40:0/48268880 wait complete. 2024-04-26T23:05:30.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.960+0000 7faf20a00640 1 Processor -- start 2024-04-26T23:05:30.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.960+0000 7faf20a00640 1 -- start start 2024-04-26T23:05:30.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.961+0000 7faf20a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf1c0d06b0 0x7faf1c15fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:30.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.961+0000 7faf20a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf1c145970 0x7faf1c1524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:30.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.961+0000 7faf20a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c146060 0x7faf1c140160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:30.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.961+0000 7faf20a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faf1c0b5c60 con 0x7faf1c146060 2024-04-26T23:05:30.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.961+0000 7faf20a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faf1c0b5f40 con 0x7faf1c0d06b0 2024-04-26T23:05:30.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.961+0000 7faf20a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faf1c0cf100 con 0x7faf1c145970 2024-04-26T23:05:30.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.961+0000 7faf1be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c146060 0x7faf1c140160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:30.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.961+0000 7faf1be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c146060 0x7faf1c140160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45720/0 (socket says 172.21.15.40:45720) 2024-04-26T23:05:30.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.961+0000 7faf1be00640 1 -- 172.21.15.40:0/3965042484 learned_addr learned my addr 172.21.15.40:0/3965042484 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:30.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.962+0000 7faf1b400640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf1c0d06b0 0x7faf1c15fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:30.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.962+0000 7faf1aa00640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf1c145970 0x7faf1c1524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:30.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.962+0000 7faf1be00640 1 -- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf1c145970 msgr2=0x7faf1c1524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:30.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.962+0000 7faf1be00640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf1c145970 0x7faf1c1524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:30.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.962+0000 7faf1be00640 1 -- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf1c0d06b0 msgr2=0x7faf1c15fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:30.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.962+0000 7faf1be00640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf1c0d06b0 0x7faf1c15fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:30.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.962+0000 7faf1be00640 1 -- 172.21.15.40:0/3965042484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf1c161240 con 0x7faf1c146060 2024-04-26T23:05:30.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.962+0000 7faf1be00640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c146060 0x7faf1c140160 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7faf14009eb0 tx=0x7faf14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:30.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.963+0000 7faf13400640 1 -- 172.21.15.40:0/3965042484 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf14040070 con 0x7faf1c146060 2024-04-26T23:05:30.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.963+0000 7faf13400640 1 -- 172.21.15.40:0/3965042484 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf14031c40 con 0x7faf1c146060 2024-04-26T23:05:30.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.963+0000 7faf13400640 1 -- 172.21.15.40:0/3965042484 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf14009450 con 0x7faf1c146060 2024-04-26T23:05:30.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.963+0000 7faf20a00640 1 -- 172.21.15.40:0/3965042484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf1c1601a0 con 0x7faf1c146060 2024-04-26T23:05:30.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.963+0000 7faf20a00640 1 -- 172.21.15.40:0/3965042484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf1c160760 con 0x7faf1c146060 2024-04-26T23:05:30.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.964+0000 7faf20a00640 1 -- 172.21.15.40:0/3965042484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf1c069370 con 0x7faf1c146060 2024-04-26T23:05:30.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.965+0000 7faf13400640 1 -- 172.21.15.40:0/3965042484 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faf14039040 con 0x7faf1c146060 2024-04-26T23:05:30.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.966+0000 7faf13400640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faefc06c8f0 0x7faefc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:30.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.966+0000 7faf13400640 1 -- 172.21.15.40:0/3965042484 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5619+0+0 (secure 0 0 0) 0x7faf140aca60 con 0x7faf1c146060 2024-04-26T23:05:30.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.967+0000 7faf1b400640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faefc06c8f0 0x7faefc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:30.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.968+0000 7faf1b400640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faefc06c8f0 0x7faefc06edb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7faf0c006fd0 tx=0x7faf0c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:30.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:30.972+0000 7faf13400640 1 -- 172.21.15.40:0/3965042484 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7faf1407aed0 con 0x7faf1c146060 2024-04-26T23:05:31.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.177+0000 7faf20a00640 1 -- 172.21.15.40:0/3965042484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7faf1c0b5730 con 0x7faf1c146060 2024-04-26T23:05:31.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.727+0000 7faf13400640 1 -- 172.21.15.40:0/3965042484 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v63) ==== 170+0+0 (secure 0 0 0) 0x7faf1407a7d0 con 0x7faf1c146060 2024-04-26T23:05:31.728 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-04-26T23:05:31.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.731+0000 7faf20a00640 1 -- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faefc06c8f0 msgr2=0x7faefc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:31.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.731+0000 7faf20a00640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faefc06c8f0 0x7faefc06edb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7faf0c006fd0 tx=0x7faf0c008040 comp rx=0 tx=0).stop 2024-04-26T23:05:31.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.731+0000 7faf20a00640 1 -- 172.21.15.40:0/3965042484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c146060 msgr2=0x7faf1c140160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:31.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.731+0000 7faf20a00640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c146060 0x7faf1c140160 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7faf14009eb0 tx=0x7faf14004a40 comp rx=0 tx=0).stop 2024-04-26T23:05:31.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.732+0000 7faf20a00640 1 -- 172.21.15.40:0/3965042484 shutdown_connections 2024-04-26T23:05:31.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.732+0000 7faf20a00640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faefc06c8f0 0x7faefc06edb0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:31.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.732+0000 7faf20a00640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c146060 0x7faf1c140160 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:31.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.732+0000 7faf20a00640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf1c145970 0x7faf1c1524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:31.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.732+0000 7faf20a00640 1 --2- 172.21.15.40:0/3965042484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf1c0d06b0 0x7faf1c15fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:31.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.732+0000 7faf20a00640 1 -- 172.21.15.40:0/3965042484 >> 172.21.15.40:0/3965042484 conn(0x7faf1c075eb0 msgr2=0x7faf1c0b1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:31.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.732+0000 7faf20a00640 1 -- 172.21.15.40:0/3965042484 shutdown_connections 2024-04-26T23:05:31.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.732+0000 7faf20a00640 1 -- 172.21.15.40:0/3965042484 wait complete. 2024-04-26T23:05:31.764 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T23:05:31.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.950+0000 7f9b55a00640 1 Processor -- start 2024-04-26T23:05:31.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.950+0000 7f9b55a00640 1 -- start start 2024-04-26T23:05:31.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.952+0000 7f9b55a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9b500cfaf0 0x7f9b50073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:31.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.952+0000 7f9b55a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b50161880 con 0x7f9b50160a50 2024-04-26T23:05:31.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.952+0000 7f9b55a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b500cff80 con 0x7f9b5006dd40 2024-04-26T23:05:31.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.952+0000 7f9b55a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9b50071ce0 con 0x7f9b500cfaf0 2024-04-26T23:05:31.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.952+0000 7f9b4fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9b50160a50 0x7f9b50160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38120/0 (socket says 172.21.15.40:38120) 2024-04-26T23:05:31.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.952+0000 7f9b4fe00640 1 -- 172.21.15.40:0/3426783312 learned_addr learned my addr 172.21.15.40:0/3426783312 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:31.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.953+0000 7f9b4f400640 1 --2- 172.21.15.40:0/3426783312 >> v2:172.21.15.103:3300/0 conn(0x7f9b500cfaf0 0x7f9b50073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:31.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.953+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2059864476 0 0) 0x7f9b50161880 con 0x7f9b50160a50 2024-04-26T23:05:31.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.953+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b34003520 con 0x7f9b50160a50 2024-04-26T23:05:31.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.953+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 660163070 0 0) 0x7f9b34003520 con 0x7f9b50160a50 2024-04-26T23:05:31.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.953+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 >> v2:172.21.15.103:3300/0 conn(0x7f9b500cfaf0 msgr2=0x7f9b50073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:31.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.953+0000 7f9b4ea00640 1 --2- 172.21.15.40:0/3426783312 >> v2:172.21.15.103:3300/0 conn(0x7f9b500cfaf0 0x7f9b50073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:31.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.953+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 >> v1:172.21.15.103:6789/0 conn(0x7f9b5006dd40 legacy=0x7f9b50072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:31.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b500d0260 con 0x7f9b50160a50 2024-04-26T23:05:31.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f9b400019f0 con 0x7f9b50160a50 2024-04-26T23:05:31.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 >> v1:172.21.15.40:6789/0 conn(0x7f9b50160a50 legacy=0x7f9b50160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:31.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4ea00640 1 --2- 172.21.15.40:0/3426783312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b340039b0 0x7f9b34003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:31.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4ea00640 1 --2- 172.21.15.40:0/3426783312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b34004800 0x7f9b34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4ea00640 1 --2- 172.21.15.40:0/3426783312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b340051c0 0x7f9b3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9b50071ce0 con 0x7f9b340039b0 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3400c010 con 0x7f9b340051c0 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9b3400c210 con 0x7f9b34004800 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4fe00640 1 --2- 172.21.15.40:0/3426783312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b340039b0 0x7f9b34003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4f400640 1 --2- 172.21.15.40:0/3426783312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b34004800 0x7f9b34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b55000640 1 --2- 172.21.15.40:0/3426783312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b340051c0 0x7f9b3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4fe00640 1 -- 172.21.15.40:0/3426783312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b34004800 msgr2=0x7f9b34004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4fe00640 1 --2- 172.21.15.40:0/3426783312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b34004800 0x7f9b34004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4fe00640 1 -- 172.21.15.40:0/3426783312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b340051c0 msgr2=0x7f9b3400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.954+0000 7f9b4fe00640 1 --2- 172.21.15.40:0/3426783312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b340051c0 0x7f9b3400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.955+0000 7f9b4fe00640 1 -- 172.21.15.40:0/3426783312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b500d0260 con 0x7f9b340039b0 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.955+0000 7f9b55000640 1 --2- 172.21.15.40:0/3426783312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b340051c0 0x7f9b3400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:31.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.955+0000 7f9b55000640 1 -- 172.21.15.40:0/3426783312 reap_dead start 2024-04-26T23:05:31.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.956+0000 7f9b4fe00640 1 --2- 172.21.15.40:0/3426783312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b340039b0 0x7f9b34003dd0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000a940 tx=0x7f9b34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df933ce4f8c57a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:31.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.956+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4000a540 con 0x7f9b340039b0 2024-04-26T23:05:31.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.956+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b4002f070 con 0x7f9b340039b0 2024-04-26T23:05:31.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.956+0000 7f9b4ea00640 1 -- 172.21.15.40:0/3426783312 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4002bdb0 con 0x7f9b340039b0 2024-04-26T23:05:31.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.956+0000 7f9b55a00640 1 -- 172.21.15.40:0/3426783312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b340039b0 msgr2=0x7f9b34003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:31.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.956+0000 7f9b55a00640 1 --2- 172.21.15.40:0/3426783312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b340039b0 0x7f9b34003dd0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000a940 tx=0x7f9b34005830 comp rx=0 tx=0).stop 2024-04-26T23:05:31.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.957+0000 7f9b55a00640 1 -- 172.21.15.40:0/3426783312 shutdown_connections 2024-04-26T23:05:31.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.957+0000 7f9b55a00640 1 --2- 172.21.15.40:0/3426783312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b340039b0 0x7f9b34003dd0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:31.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.957+0000 7f9b55a00640 1 -- 172.21.15.40:0/3426783312 >> 172.21.15.40:0/3426783312 conn(0x7f9b500b2720 msgr2=0x7f9b500b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:31.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.957+0000 7f9b55a00640 1 -- 172.21.15.40:0/3426783312 shutdown_connections 2024-04-26T23:05:31.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.957+0000 7f9b55a00640 1 -- 172.21.15.40:0/3426783312 wait complete. 2024-04-26T23:05:31.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.959+0000 7f9b55a00640 1 Processor -- start 2024-04-26T23:05:31.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.959+0000 7f9b55a00640 1 -- start start 2024-04-26T23:05:31.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.960+0000 7f9b55a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b340039b0 0x7f9b5015dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:31.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.960+0000 7f9b55a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b5015ff00 0x7f9b5015e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:31.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.960+0000 7f9b55a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b50140a00 0x7f9b501efb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:31.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.960+0000 7f9b55a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5006e1f0 con 0x7f9b50140a00 2024-04-26T23:05:31.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.960+0000 7f9b55a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b500722a0 con 0x7f9b5015ff00 2024-04-26T23:05:31.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.960+0000 7f9b55a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9b501468f0 con 0x7f9b340039b0 2024-04-26T23:05:31.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.960+0000 7f9b55000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b50140a00 0x7f9b501efb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:31.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.960+0000 7f9b55000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b50140a00 0x7f9b501efb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45730/0 (socket says 172.21.15.40:45730) 2024-04-26T23:05:31.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.960+0000 7f9b55000640 1 -- 172.21.15.40:0/2905233970 learned_addr learned my addr 172.21.15.40:0/2905233970 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:31.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.960+0000 7f9b4f400640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b5015ff00 0x7f9b5015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:31.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.960+0000 7f9b4fe00640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b340039b0 0x7f9b5015dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:31.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.960+0000 7f9b55000640 1 -- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b340039b0 msgr2=0x7f9b5015dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:31.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.960+0000 7f9b55000640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b340039b0 0x7f9b5015dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:31.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.961+0000 7f9b55000640 1 -- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b5015ff00 msgr2=0x7f9b5015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:31.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.961+0000 7f9b55000640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b5015ff00 0x7f9b5015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:31.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.961+0000 7f9b55000640 1 -- 172.21.15.40:0/2905233970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b5015ebc0 con 0x7f9b50140a00 2024-04-26T23:05:31.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.961+0000 7f9b4f400640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b5015ff00 0x7f9b5015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:31.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.961+0000 7f9b55000640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b50140a00 0x7f9b501efb40 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9b500cedd0 tx=0x7f9b4800d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:31.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.961+0000 7f9b47e00640 1 -- 172.21.15.40:0/2905233970 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b48018070 con 0x7f9b50140a00 2024-04-26T23:05:31.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.961+0000 7f9b47e00640 1 -- 172.21.15.40:0/2905233970 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b480035f0 con 0x7f9b50140a00 2024-04-26T23:05:31.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.961+0000 7f9b47e00640 1 -- 172.21.15.40:0/2905233970 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b480039e0 con 0x7f9b50140a00 2024-04-26T23:05:31.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.961+0000 7f9b55a00640 1 -- 172.21.15.40:0/2905233970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b5015ee60 con 0x7f9b50140a00 2024-04-26T23:05:31.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.961+0000 7f9b55a00640 1 -- 172.21.15.40:0/2905233970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b5015f420 con 0x7f9b50140a00 2024-04-26T23:05:31.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.962+0000 7f9b55a00640 1 -- 172.21.15.40:0/2905233970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b50069370 con 0x7f9b50140a00 2024-04-26T23:05:31.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.964+0000 7f9b47e00640 1 -- 172.21.15.40:0/2905233970 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9b4801b070 con 0x7f9b50140a00 2024-04-26T23:05:31.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.964+0000 7f9b47e00640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9b3006c8f0 0x7f9b3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:31.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.965+0000 7f9b47e00640 1 -- 172.21.15.40:0/2905233970 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6001+0+0 (secure 0 0 0) 0x7f9b4808b340 con 0x7f9b50140a00 2024-04-26T23:05:31.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.965+0000 7f9b4fe00640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9b3006c8f0 0x7f9b3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:31.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.967+0000 7f9b4fe00640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9b3006c8f0 0x7f9b3006edb0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9b40033000 tx=0x7f9b40038040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:31.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:31.970+0000 7f9b47e00640 1 -- 172.21.15.40:0/2905233970 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9b480595a0 con 0x7f9b50140a00 2024-04-26T23:05:32.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.210+0000 7f9b55a00640 1 -- 172.21.15.40:0/2905233970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f9b50161240 con 0x7f9b50140a00 2024-04-26T23:05:32.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.751+0000 7f9b47e00640 1 -- 172.21.15.40:0/2905233970 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v51) ==== 173+0+0 (secure 0 0 0) 0x7f9b48058ea0 con 0x7f9b50140a00 2024-04-26T23:05:32.752 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 9 and data pool 10 2024-04-26T23:05:32.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.760+0000 7f9b55a00640 1 -- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9b3006c8f0 msgr2=0x7f9b3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:32.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.760+0000 7f9b55a00640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9b3006c8f0 0x7f9b3006edb0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9b40033000 tx=0x7f9b40038040 comp rx=0 tx=0).stop 2024-04-26T23:05:32.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.760+0000 7f9b55a00640 1 -- 172.21.15.40:0/2905233970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b50140a00 msgr2=0x7f9b501efb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:32.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.760+0000 7f9b55a00640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b50140a00 0x7f9b501efb40 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9b500cedd0 tx=0x7f9b4800d890 comp rx=0 tx=0).stop 2024-04-26T23:05:32.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.761+0000 7f9b55a00640 1 -- 172.21.15.40:0/2905233970 shutdown_connections 2024-04-26T23:05:32.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.761+0000 7f9b55a00640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9b3006c8f0 0x7f9b3006edb0 secure :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9b40033000 tx=0x7f9b40038040 comp rx=0 tx=0).stop 2024-04-26T23:05:32.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.761+0000 7f9b55a00640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b50140a00 0x7f9b501efb40 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:32.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.761+0000 7f9b55a00640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b5015ff00 0x7f9b5015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:32.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.761+0000 7f9b55a00640 1 --2- 172.21.15.40:0/2905233970 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9b340039b0 0x7f9b5015dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:32.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.761+0000 7f9b55a00640 1 -- 172.21.15.40:0/2905233970 >> 172.21.15.40:0/2905233970 conn(0x7f9b500b2720 msgr2=0x7f9b500b2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:32.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.765+0000 7f9b55a00640 1 -- 172.21.15.40:0/2905233970 shutdown_connections 2024-04-26T23:05:32.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.765+0000 7f9b55a00640 1 -- 172.21.15.40:0/2905233970 wait complete. 2024-04-26T23:05:32.792 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:05:32.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.961+0000 7fc346800640 1 Processor -- start 2024-04-26T23:05:32.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.961+0000 7fc346800640 1 -- start start 2024-04-26T23:05:32.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.962+0000 7fc346800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc340071e60 0x7fc34006f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:32.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.962+0000 7fc346800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc34006fe20 0x7fc340072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:32.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.962+0000 7fc346800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc340072780 0x7fc3400cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:32.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.962+0000 7fc346800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc34015d7c0 con 0x7fc340071e60 2024-04-26T23:05:32.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.962+0000 7fc346800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc34015daa0 con 0x7fc34006fe20 2024-04-26T23:05:32.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.962+0000 7fc346800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc34015dd80 con 0x7fc340072780 2024-04-26T23:05:32.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.962+0000 7fc33f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc340071e60 0x7fc34006f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:32.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.962+0000 7fc33f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc340071e60 0x7fc34006f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45738/0 (socket says 172.21.15.40:45738) 2024-04-26T23:05:32.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.962+0000 7fc33f400640 1 -- 172.21.15.40:0/3139997609 learned_addr learned my addr 172.21.15.40:0/3139997609 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:32.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.963+0000 7fc33f400640 1 -- 172.21.15.40:0/3139997609 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc340072780 msgr2=0x7fc3400cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:32.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.963+0000 7fc33f400640 1 --2- 172.21.15.40:0/3139997609 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc340072780 0x7fc3400cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:32.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.963+0000 7fc33f400640 1 -- 172.21.15.40:0/3139997609 >> v2:172.21.15.103:3300/0 conn(0x7fc34006fe20 msgr2=0x7fc340072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:32.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.963+0000 7fc33f400640 1 --2- 172.21.15.40:0/3139997609 >> v2:172.21.15.103:3300/0 conn(0x7fc34006fe20 0x7fc340072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:32.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.963+0000 7fc33f400640 1 -- 172.21.15.40:0/3139997609 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc34015e060 con 0x7fc340071e60 2024-04-26T23:05:32.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.964+0000 7fc33f400640 1 --2- 172.21.15.40:0/3139997609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc340071e60 0x7fc34006f8e0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fc3400b3820 tx=0x7fc33002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d142c158a63f9438 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:32.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.964+0000 7fc33e000640 1 -- 172.21.15.40:0/3139997609 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330036070 con 0x7fc340071e60 2024-04-26T23:05:32.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.964+0000 7fc33e000640 1 -- 172.21.15.40:0/3139997609 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc330031600 con 0x7fc340071e60 2024-04-26T23:05:32.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.964+0000 7fc33e000640 1 -- 172.21.15.40:0/3139997609 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3300319d0 con 0x7fc340071e60 2024-04-26T23:05:32.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.965+0000 7fc346800640 1 -- 172.21.15.40:0/3139997609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc340071e60 msgr2=0x7fc34006f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:32.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.965+0000 7fc346800640 1 --2- 172.21.15.40:0/3139997609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc340071e60 0x7fc34006f8e0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fc3400b3820 tx=0x7fc33002a6a0 comp rx=0 tx=0).stop 2024-04-26T23:05:32.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.965+0000 7fc346800640 1 -- 172.21.15.40:0/3139997609 shutdown_connections 2024-04-26T23:05:32.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.965+0000 7fc346800640 1 --2- 172.21.15.40:0/3139997609 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc340072780 0x7fc3400cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:32.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.965+0000 7fc346800640 1 --2- 172.21.15.40:0/3139997609 >> v2:172.21.15.103:3300/0 conn(0x7fc34006fe20 0x7fc340072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:32.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.965+0000 7fc346800640 1 --2- 172.21.15.40:0/3139997609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc340071e60 0x7fc34006f8e0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:32.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.965+0000 7fc346800640 1 -- 172.21.15.40:0/3139997609 >> 172.21.15.40:0/3139997609 conn(0x7fc3400d0be0 msgr2=0x7fc3400d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:32.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.965+0000 7fc346800640 1 -- 172.21.15.40:0/3139997609 shutdown_connections 2024-04-26T23:05:32.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.965+0000 7fc346800640 1 -- 172.21.15.40:0/3139997609 wait complete. 2024-04-26T23:05:32.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.966+0000 7fc346800640 1 Processor -- start 2024-04-26T23:05:32.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.967+0000 7fc346800640 1 -- start start 2024-04-26T23:05:32.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.967+0000 7fc346800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34006fe20 0x7fc34014b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.967+0000 7fc346800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc340071e60 0x7fc3401f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.967+0000 7fc346800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc340072780 0x7fc3400b5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.967+0000 7fc346800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc34015d3f0 con 0x7fc340072780 2024-04-26T23:05:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.967+0000 7fc346800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc340072c10 con 0x7fc34006fe20 2024-04-26T23:05:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.967+0000 7fc346800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc34015d570 con 0x7fc340071e60 2024-04-26T23:05:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.968+0000 7fc33fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc340072780 0x7fc3400b5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.968+0000 7fc33fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc340072780 0x7fc3400b5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45754/0 (socket says 172.21.15.40:45754) 2024-04-26T23:05:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.968+0000 7fc33fe00640 1 -- 172.21.15.40:0/63232636 learned_addr learned my addr 172.21.15.40:0/63232636 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.968+0000 7fc33f400640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34006fe20 0x7fc34014b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.968+0000 7fc33ea00640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc340071e60 0x7fc3401f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.968+0000 7fc33fe00640 1 -- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc340071e60 msgr2=0x7fc3401f5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.968+0000 7fc33fe00640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc340071e60 0x7fc3401f5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:32.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.968+0000 7fc33fe00640 1 -- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34006fe20 msgr2=0x7fc34014b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:32.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.968+0000 7fc33fe00640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34006fe20 0x7fc34014b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:32.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.968+0000 7fc33fe00640 1 -- 172.21.15.40:0/63232636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3401f6340 con 0x7fc340072780 2024-04-26T23:05:32.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.969+0000 7fc33fe00640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc340072780 0x7fc3400b5890 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fc33800b410 tx=0x7fc33800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:32.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.969+0000 7fc337400640 1 -- 172.21.15.40:0/63232636 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338015020 con 0x7fc340072780 2024-04-26T23:05:32.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.969+0000 7fc337400640 1 -- 172.21.15.40:0/63232636 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc338011850 con 0x7fc340072780 2024-04-26T23:05:32.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.969+0000 7fc337400640 1 -- 172.21.15.40:0/63232636 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338011c40 con 0x7fc340072780 2024-04-26T23:05:32.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.969+0000 7fc346800640 1 -- 172.21.15.40:0/63232636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc34015b860 con 0x7fc340072780 2024-04-26T23:05:32.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.969+0000 7fc346800640 1 -- 172.21.15.40:0/63232636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc34015be20 con 0x7fc340072780 2024-04-26T23:05:32.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.970+0000 7fc346800640 1 -- 172.21.15.40:0/63232636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3400694f0 con 0x7fc340072780 2024-04-26T23:05:32.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.971+0000 7fc337400640 1 -- 172.21.15.40:0/63232636 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc338018020 con 0x7fc340072780 2024-04-26T23:05:32.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.972+0000 7fc337400640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc31c06c920 0x7fc31c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:32.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.973+0000 7fc337400640 1 -- 172.21.15.40:0/63232636 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7fc33808bb90 con 0x7fc340072780 2024-04-26T23:05:32.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.973+0000 7fc33f400640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc31c06c920 0x7fc31c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:32.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.974+0000 7fc33f400640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc31c06c920 0x7fc31c06ede0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc330033000 tx=0x7fc330009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:32.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:32.978+0000 7fc337400640 1 -- 172.21.15.40:0/63232636 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc33801f020 con 0x7fc340072780 2024-04-26T23:05:33.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.172+0000 7fc346800640 1 -- 172.21.15.40:0/63232636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc340160320 con 0x7fc340072780 2024-04-26T23:05:33.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.174+0000 7fc337400640 1 -- 172.21.15.40:0/63232636 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) ==== 74+0+16861 (secure 0 0 0) 0x7fc3380172c0 con 0x7fc340072780 2024-04-26T23:05:33.174 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:33.175 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":64,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:05:32.212395+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:05:30.081199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-26T23:05:31.192158+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:05:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.176+0000 7fc346800640 1 -- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc31c06c920 msgr2=0x7fc31c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.176+0000 7fc346800640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc31c06c920 0x7fc31c06ede0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc330033000 tx=0x7fc330009000 comp rx=0 tx=0).stop 2024-04-26T23:05:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.176+0000 7fc346800640 1 -- 172.21.15.40:0/63232636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc340072780 msgr2=0x7fc3400b5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.176+0000 7fc346800640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc340072780 0x7fc3400b5890 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fc33800b410 tx=0x7fc33800b8d0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.176+0000 7fc346800640 1 -- 172.21.15.40:0/63232636 shutdown_connections 2024-04-26T23:05:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.176+0000 7fc346800640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc31c06c920 0x7fc31c06ede0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.176+0000 7fc346800640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc340072780 0x7fc3400b5890 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.176+0000 7fc346800640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc340071e60 0x7fc3401f5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.176+0000 7fc346800640 1 --2- 172.21.15.40:0/63232636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34006fe20 0x7fc34014b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.176+0000 7fc346800640 1 -- 172.21.15.40:0/63232636 >> 172.21.15.40:0/63232636 conn(0x7fc3400d0be0 msgr2=0x7fc3400b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.177+0000 7fc346800640 1 -- 172.21.15.40:0/63232636 shutdown_connections 2024-04-26T23:05:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.177+0000 7fc346800640 1 -- 172.21.15.40:0/63232636 wait complete. 2024-04-26T23:05:33.195 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:05:33.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.397+0000 7fbf42a00640 1 Processor -- start 2024-04-26T23:05:33.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.397+0000 7fbf42a00640 1 -- start start 2024-04-26T23:05:33.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.399+0000 7fbf42a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf3c0b6220 0x7fbf3c0d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.399+0000 7fbf42a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbf3c0d39f0 0x7fbf3c0b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.399+0000 7fbf42a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf3c162c40 con 0x7fbf3c0d08d0 2024-04-26T23:05:33.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.399+0000 7fbf42a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbf3c0d2d80 con 0x7fbf3c0d39f0 2024-04-26T23:05:33.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.399+0000 7fbf42a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbf3c0b7c80 con 0x7fbf3c0b6220 2024-04-26T23:05:33.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.399+0000 7fbf40c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbf3c0d08d0 0x7fbf3c0d32e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38122/0 (socket says 172.21.15.40:38122) 2024-04-26T23:05:33.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.399+0000 7fbf40c00640 1 -- 172.21.15.40:0/4153206495 learned_addr learned my addr 172.21.15.40:0/4153206495 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:33.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1615216169 0 0) 0x7fbf3c162c40 con 0x7fbf3c0d08d0 2024-04-26T23:05:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf42000640 1 --2- 172.21.15.40:0/4153206495 >> v2:172.21.15.103:3300/0 conn(0x7fbf3c0d39f0 0x7fbf3c0b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf24003520 con 0x7fbf3c0d08d0 2024-04-26T23:05:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf41600640 1 --2- 172.21.15.40:0/4153206495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf3c0b6220 0x7fbf3c0d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3966402922 0 0) 0x7fbf24003520 con 0x7fbf3c0d08d0 2024-04-26T23:05:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf3c0b6220 msgr2=0x7fbf3c0d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 --2- 172.21.15.40:0/4153206495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf3c0b6220 0x7fbf3c0d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 >> v2:172.21.15.103:3300/0 conn(0x7fbf3c0d39f0 msgr2=0x7fbf3c0b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 --2- 172.21.15.40:0/4153206495 >> v2:172.21.15.103:3300/0 conn(0x7fbf3c0d39f0 0x7fbf3c0b8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf3c0b7f60 con 0x7fbf3c0d08d0 2024-04-26T23:05:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fbf2c0019f0 con 0x7fbf3c0d08d0 2024-04-26T23:05:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf42000640 1 --2- 172.21.15.40:0/4153206495 >> v2:172.21.15.103:3300/0 conn(0x7fbf3c0d39f0 0x7fbf3c0b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:05:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 >> v1:172.21.15.40:6789/0 conn(0x7fbf3c0d08d0 legacy=0x7fbf3c0d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 --2- 172.21.15.40:0/4153206495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf240039b0 0x7fbf24003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 --2- 172.21.15.40:0/4153206495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf24004800 0x7fbf24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 --2- 172.21.15.40:0/4153206495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf3c0b6220 0x7fbf240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3c0d2d80 con 0x7fbf240039b0 2024-04-26T23:05:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3c0b7c80 con 0x7fbf24004800 2024-04-26T23:05:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.400+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbf2400bac0 con 0x7fbf3c0b6220 2024-04-26T23:05:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.401+0000 7fbf41600640 1 --2- 172.21.15.40:0/4153206495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf240039b0 0x7fbf24003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.401+0000 7fbf42000640 1 --2- 172.21.15.40:0/4153206495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf3c0b6220 0x7fbf240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.401+0000 7fbf40c00640 1 --2- 172.21.15.40:0/4153206495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf24004800 0x7fbf24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.401+0000 7fbf41600640 1 -- 172.21.15.40:0/4153206495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf3c0b6220 msgr2=0x7fbf240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.401+0000 7fbf41600640 1 --2- 172.21.15.40:0/4153206495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf3c0b6220 0x7fbf240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.401+0000 7fbf41600640 1 -- 172.21.15.40:0/4153206495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf24004800 msgr2=0x7fbf24004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:33.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.401+0000 7fbf41600640 1 --2- 172.21.15.40:0/4153206495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf24004800 0x7fbf24004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.401+0000 7fbf41600640 1 -- 172.21.15.40:0/4153206495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf24003520 con 0x7fbf240039b0 2024-04-26T23:05:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.403+0000 7fbf41600640 1 --2- 172.21.15.40:0/4153206495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf240039b0 0x7fbf24003dd0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000ba00 tx=0x7fbf24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8785eb56b425a10f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.403+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf30030070 con 0x7fbf240039b0 2024-04-26T23:05:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.403+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf3002ad70 con 0x7fbf240039b0 2024-04-26T23:05:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.403+0000 7fbf3be00640 1 -- 172.21.15.40:0/4153206495 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3000b600 con 0x7fbf240039b0 2024-04-26T23:05:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.403+0000 7fbf42a00640 1 -- 172.21.15.40:0/4153206495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf240039b0 msgr2=0x7fbf24003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.403+0000 7fbf42a00640 1 --2- 172.21.15.40:0/4153206495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf240039b0 0x7fbf24003dd0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000ba00 tx=0x7fbf24005830 comp rx=0 tx=0).stop 2024-04-26T23:05:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.404+0000 7fbf42000640 1 -- 172.21.15.40:0/4153206495 reap_dead start 2024-04-26T23:05:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.404+0000 7fbf42a00640 1 -- 172.21.15.40:0/4153206495 shutdown_connections 2024-04-26T23:05:33.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.404+0000 7fbf42a00640 1 -- 172.21.15.40:0/4153206495 >> 172.21.15.40:0/4153206495 conn(0x7fbf3c0d4d90 msgr2=0x7fbf3c0d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:33.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.404+0000 7fbf42a00640 1 -- 172.21.15.40:0/4153206495 shutdown_connections 2024-04-26T23:05:33.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.404+0000 7fbf42a00640 1 -- 172.21.15.40:0/4153206495 wait complete. 2024-04-26T23:05:33.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.405+0000 7fbf42a00640 1 Processor -- start 2024-04-26T23:05:33.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.406+0000 7fbf42a00640 1 -- start start 2024-04-26T23:05:33.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.407+0000 7fbf42a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf240039b0 0x7fbf3c1f5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.407+0000 7fbf42a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf3c1581f0 0x7fbf3c0cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.407+0000 7fbf42a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf3c0ce320 0x7fbf3c1f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.407+0000 7fbf42a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3c0d0390 con 0x7fbf3c1581f0 2024-04-26T23:05:33.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.407+0000 7fbf42a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3c0b8520 con 0x7fbf240039b0 2024-04-26T23:05:33.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.407+0000 7fbf42a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbf3c0d0510 con 0x7fbf3c0ce320 2024-04-26T23:05:33.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.407+0000 7fbf40c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf3c1581f0 0x7fbf3c0cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.407+0000 7fbf40c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf3c1581f0 0x7fbf3c0cdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45768/0 (socket says 172.21.15.40:45768) 2024-04-26T23:05:33.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.407+0000 7fbf40c00640 1 -- 172.21.15.40:0/30188345 learned_addr learned my addr 172.21.15.40:0/30188345 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:33.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.407+0000 7fbf41600640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf240039b0 0x7fbf3c1f5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.408+0000 7fbf42000640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf3c0ce320 0x7fbf3c1f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.408+0000 7fbf40c00640 1 -- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf3c0ce320 msgr2=0x7fbf3c1f4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:33.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.408+0000 7fbf40c00640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf3c0ce320 0x7fbf3c1f4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.408+0000 7fbf40c00640 1 -- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf240039b0 msgr2=0x7fbf3c1f5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:33.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.408+0000 7fbf40c00640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf240039b0 0x7fbf3c1f5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.408+0000 7fbf40c00640 1 -- 172.21.15.40:0/30188345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf3c1f5b40 con 0x7fbf3c1581f0 2024-04-26T23:05:33.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.408+0000 7fbf40c00640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf3c1581f0 0x7fbf3c0cdde0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0013d0 tx=0x7fbf2c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:33.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.408+0000 7fbf39600640 1 -- 172.21.15.40:0/30188345 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c00edf0 con 0x7fbf3c1581f0 2024-04-26T23:05:33.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.408+0000 7fbf39600640 1 -- 172.21.15.40:0/30188345 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf2c018070 con 0x7fbf3c1581f0 2024-04-26T23:05:33.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.408+0000 7fbf42a00640 1 -- 172.21.15.40:0/30188345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf3c0b1980 con 0x7fbf3c1581f0 2024-04-26T23:05:33.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.408+0000 7fbf39600640 1 -- 172.21.15.40:0/30188345 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c013660 con 0x7fbf3c1581f0 2024-04-26T23:05:33.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.408+0000 7fbf42a00640 1 -- 172.21.15.40:0/30188345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf3c0b1f40 con 0x7fbf3c1581f0 2024-04-26T23:05:33.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.410+0000 7fbf42a00640 1 -- 172.21.15.40:0/30188345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf3c069370 con 0x7fbf3c1581f0 2024-04-26T23:05:33.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.410+0000 7fbf39600640 1 -- 172.21.15.40:0/30188345 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbf2c013800 con 0x7fbf3c1581f0 2024-04-26T23:05:33.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.412+0000 7fbf39600640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbf2006c920 0x7fbf2006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.412+0000 7fbf39600640 1 -- 172.21.15.40:0/30188345 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6101+0+0 (secure 0 0 0) 0x7fbf2c08c400 con 0x7fbf3c1581f0 2024-04-26T23:05:33.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.413+0000 7fbf41600640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbf2006c920 0x7fbf2006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.414+0000 7fbf41600640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbf2006c920 0x7fbf2006ede0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbf240057e0 tx=0x7fbf30030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:33.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.416+0000 7fbf39600640 1 -- 172.21.15.40:0/30188345 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fbf2c05a600 con 0x7fbf3c1581f0 2024-04-26T23:05:33.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.618+0000 7fbf42a00640 1 -- 172.21.15.40:0/30188345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf3c0d30c0 con 0x7fbf3c1581f0 2024-04-26T23:05:33.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.619+0000 7fbf39600640 1 -- 172.21.15.40:0/30188345 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) ==== 74+0+16861 (secure 0 0 0) 0x7fbf2c059f00 con 0x7fbf3c1581f0 2024-04-26T23:05:33.620 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:33.620 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":64,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:05:32.212395+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:05:30.081199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-26T23:05:31.192158+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:05:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.622+0000 7fbf42a00640 1 -- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbf2006c920 msgr2=0x7fbf2006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.622+0000 7fbf42a00640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbf2006c920 0x7fbf2006ede0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbf240057e0 tx=0x7fbf30030000 comp rx=0 tx=0).stop 2024-04-26T23:05:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.622+0000 7fbf42a00640 1 -- 172.21.15.40:0/30188345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf3c1581f0 msgr2=0x7fbf3c0cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.622+0000 7fbf42a00640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf3c1581f0 0x7fbf3c0cdde0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0013d0 tx=0x7fbf2c00e530 comp rx=0 tx=0).stop 2024-04-26T23:05:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.622+0000 7fbf42a00640 1 -- 172.21.15.40:0/30188345 shutdown_connections 2024-04-26T23:05:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.622+0000 7fbf42a00640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbf2006c920 0x7fbf2006ede0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.622+0000 7fbf42a00640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf3c0ce320 0x7fbf3c1f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.622+0000 7fbf42a00640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf3c1581f0 0x7fbf3c0cdde0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.622+0000 7fbf42a00640 1 --2- 172.21.15.40:0/30188345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf240039b0 0x7fbf3c1f5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.622+0000 7fbf42a00640 1 -- 172.21.15.40:0/30188345 >> 172.21.15.40:0/30188345 conn(0x7fbf3c0d4d90 msgr2=0x7fbf3c0cf860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.623+0000 7fbf42a00640 1 -- 172.21.15.40:0/30188345 shutdown_connections 2024-04-26T23:05:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.623+0000 7fbf42a00640 1 -- 172.21.15.40:0/30188345 wait complete. 2024-04-26T23:05:33.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T23:05:33.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.859+0000 7f5f2be00640 1 Processor -- start 2024-04-26T23:05:33.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.860+0000 7f5f2be00640 1 -- start start 2024-04-26T23:05:33.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.861+0000 7f5f2be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5f2c0b5120 0x7f5f2c0b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.861+0000 7f5f2be00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f2c0d3dc0 con 0x7f5f2c0b5500 2024-04-26T23:05:33.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.861+0000 7f5f2be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f2c0b75b0 con 0x7f5f2c0b59b0 2024-04-26T23:05:33.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.861+0000 7f5f2be00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5f2c0d03b0 con 0x7f5f2c0b5120 2024-04-26T23:05:33.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.861+0000 7f5f2a000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5f2c0b5500 0x7f5f2c0b6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38130/0 (socket says 172.21.15.40:38130) 2024-04-26T23:05:33.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.861+0000 7f5f2a000640 1 -- 172.21.15.40:0/2522580107 learned_addr learned my addr 172.21.15.40:0/2522580107 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:33.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.862+0000 7f5f2aa00640 1 --2- 172.21.15.40:0/2522580107 >> v2:172.21.15.103:3300/0 conn(0x7f5f2c0b5120 0x7f5f2c0b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.862+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4062144076 0 0) 0x7f5f2c0d3dc0 con 0x7f5f2c0b5500 2024-04-26T23:05:33.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.862+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f10003520 con 0x7f5f2c0b5500 2024-04-26T23:05:33.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.862+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3345901621 0 0) 0x7f5f10003520 con 0x7f5f2c0b5500 2024-04-26T23:05:33.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.862+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 >> v2:172.21.15.103:3300/0 conn(0x7f5f2c0b5120 msgr2=0x7f5f2c0b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:33.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.862+0000 7f5f29600640 1 --2- 172.21.15.40:0/2522580107 >> v2:172.21.15.103:3300/0 conn(0x7f5f2c0b5120 0x7f5f2c0b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.862+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 >> v1:172.21.15.103:6789/0 conn(0x7f5f2c0b59b0 legacy=0x7f5f2c0b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:33.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.862+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f2c0d0600 con 0x7f5f2c0b5500 2024-04-26T23:05:33.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.862+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f5f1c0019f0 con 0x7f5f2c0b5500 2024-04-26T23:05:33.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.862+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 >> v1:172.21.15.40:6789/0 conn(0x7f5f2c0b5500 legacy=0x7f5f2c0b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:33.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.862+0000 7f5f29600640 1 --2- 172.21.15.40:0/2522580107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f10003a00 0x7f5f10003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.863+0000 7f5f29600640 1 --2- 172.21.15.40:0/2522580107 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f10004800 0x7f5f10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.863+0000 7f5f29600640 1 --2- 172.21.15.40:0/2522580107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f100052e0 0x7f5f1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.863+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2c0d03b0 con 0x7f5f100052e0 2024-04-26T23:05:33.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.863+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5f1000c1e0 con 0x7f5f10003a00 2024-04-26T23:05:33.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.863+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5f1000c3e0 con 0x7f5f10004800 2024-04-26T23:05:33.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.863+0000 7f5f2b400640 1 --2- 172.21.15.40:0/2522580107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f100052e0 0x7f5f1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.863+0000 7f5f2aa00640 1 --2- 172.21.15.40:0/2522580107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f10003a00 0x7f5f10003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.863+0000 7f5f2a000640 1 --2- 172.21.15.40:0/2522580107 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f10004800 0x7f5f10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.864+0000 7f5f2b400640 1 -- 172.21.15.40:0/2522580107 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f10004800 msgr2=0x7f5f10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:33.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.864+0000 7f5f2b400640 1 --2- 172.21.15.40:0/2522580107 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f10004800 0x7f5f10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.864+0000 7f5f2b400640 1 -- 172.21.15.40:0/2522580107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f10003a00 msgr2=0x7f5f10003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:33.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.864+0000 7f5f2b400640 1 --2- 172.21.15.40:0/2522580107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f10003a00 0x7f5f10003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.864+0000 7f5f2b400640 1 -- 172.21.15.40:0/2522580107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f10003520 con 0x7f5f100052e0 2024-04-26T23:05:33.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.864+0000 7f5f2aa00640 1 --2- 172.21.15.40:0/2522580107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f10003a00 0x7f5f10003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:05:33.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.864+0000 7f5f2b400640 1 -- 172.21.15.40:0/2522580107 reap_dead start 2024-04-26T23:05:33.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.865+0000 7f5f2b400640 1 --2- 172.21.15.40:0/2522580107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f100052e0 0x7f5f1000b9c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f5f24001750 tx=0x7f5f10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6ce76347e33b75 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:33.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.865+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f24001370 con 0x7f5f100052e0 2024-04-26T23:05:33.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.865+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f240052f0 con 0x7f5f100052e0 2024-04-26T23:05:33.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.865+0000 7f5f29600640 1 -- 172.21.15.40:0/2522580107 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2402abe0 con 0x7f5f100052e0 2024-04-26T23:05:33.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.865+0000 7f5f2be00640 1 -- 172.21.15.40:0/2522580107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f100052e0 msgr2=0x7f5f1000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:33.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.865+0000 7f5f2be00640 1 --2- 172.21.15.40:0/2522580107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f100052e0 0x7f5f1000b9c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f5f24001750 tx=0x7f5f10005830 comp rx=0 tx=0).stop 2024-04-26T23:05:33.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.865+0000 7f5f2be00640 1 -- 172.21.15.40:0/2522580107 shutdown_connections 2024-04-26T23:05:33.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.865+0000 7f5f2be00640 1 --2- 172.21.15.40:0/2522580107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f100052e0 0x7f5f1000b9c0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.865+0000 7f5f2be00640 1 -- 172.21.15.40:0/2522580107 >> 172.21.15.40:0/2522580107 conn(0x7f5f2c0d4db0 msgr2=0x7f5f2c0d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:33.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.866+0000 7f5f2be00640 1 -- 172.21.15.40:0/2522580107 shutdown_connections 2024-04-26T23:05:33.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.866+0000 7f5f2be00640 1 -- 172.21.15.40:0/2522580107 wait complete. 2024-04-26T23:05:33.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.867+0000 7f5f2be00640 1 Processor -- start 2024-04-26T23:05:33.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.867+0000 7f5f2be00640 1 -- start start 2024-04-26T23:05:33.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.868+0000 7f5f2be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f100052e0 0x7f5f2c0b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.868+0000 7f5f2be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f2c1f5e80 0x7f5f2c1f4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.868+0000 7f5f2be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f2c1f52b0 0x7f5f2c1f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.868+0000 7f5f2be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2c0b5ee0 con 0x7f5f100052e0 2024-04-26T23:05:33.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.868+0000 7f5f2be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2c0d0bc0 con 0x7f5f2c1f52b0 2024-04-26T23:05:33.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.868+0000 7f5f2be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5f2c0b6060 con 0x7f5f2c1f5e80 2024-04-26T23:05:33.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.869+0000 7f5f2aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f100052e0 0x7f5f2c0b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.869+0000 7f5f2aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f100052e0 0x7f5f2c0b7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45778/0 (socket says 172.21.15.40:45778) 2024-04-26T23:05:33.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.869+0000 7f5f2aa00640 1 -- 172.21.15.40:0/1644490744 learned_addr learned my addr 172.21.15.40:0/1644490744 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:33.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.869+0000 7f5f2b400640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f2c1f52b0 0x7f5f2c1f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.869+0000 7f5f2a000640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f2c1f5e80 0x7f5f2c1f4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.869+0000 7f5f2aa00640 1 -- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f2c1f5e80 msgr2=0x7f5f2c1f4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:33.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.869+0000 7f5f2aa00640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f2c1f5e80 0x7f5f2c1f4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.869+0000 7f5f2aa00640 1 -- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f2c1f52b0 msgr2=0x7f5f2c1f1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:33.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.869+0000 7f5f2aa00640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f2c1f52b0 0x7f5f2c1f1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:33.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.869+0000 7f5f2aa00640 1 -- 172.21.15.40:0/1644490744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f2c0b9fd0 con 0x7f5f100052e0 2024-04-26T23:05:33.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.869+0000 7f5f2aa00640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f100052e0 0x7f5f2c0b7e30 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5f14011540 tx=0x7f5f14011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:33.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.870+0000 7f5f22a00640 1 -- 172.21.15.40:0/1644490744 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f14015870 con 0x7f5f100052e0 2024-04-26T23:05:33.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.870+0000 7f5f22a00640 1 -- 172.21.15.40:0/1644490744 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f14015a10 con 0x7f5f100052e0 2024-04-26T23:05:33.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.870+0000 7f5f22a00640 1 -- 172.21.15.40:0/1644490744 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f14015dc0 con 0x7f5f100052e0 2024-04-26T23:05:33.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.870+0000 7f5f2be00640 1 -- 172.21.15.40:0/1644490744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f2c1f5c30 con 0x7f5f100052e0 2024-04-26T23:05:33.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.870+0000 7f5f2be00640 1 -- 172.21.15.40:0/1644490744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f2c0ccee0 con 0x7f5f100052e0 2024-04-26T23:05:33.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.871+0000 7f5f2be00640 1 -- 172.21.15.40:0/1644490744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f2c069370 con 0x7f5f100052e0 2024-04-26T23:05:33.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.872+0000 7f5f22a00640 1 -- 172.21.15.40:0/1644490744 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5f1400f460 con 0x7f5f100052e0 2024-04-26T23:05:33.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.873+0000 7f5f22a00640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5f0806c920 0x7f5f0806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:33.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.873+0000 7f5f22a00640 1 -- 172.21.15.40:0/1644490744 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f5f1408c5d0 con 0x7f5f100052e0 2024-04-26T23:05:33.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.874+0000 7f5f2a000640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5f0806c920 0x7f5f0806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:33.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.875+0000 7f5f2a000640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5f0806c920 0x7f5f0806ede0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c001460 tx=0x7f5f1c0012d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:33.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:33.878+0000 7f5f22a00640 1 -- 172.21.15.40:0/1644490744 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5f14019030 con 0x7f5f100052e0 2024-04-26T23:05:34.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.120+0000 7f5f2be00640 1 -- 172.21.15.40:0/1644490744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5f2c0d1390 con 0x7f5f100052e0 2024-04-26T23:05:34.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.756+0000 7f5f22a00640 1 -- 172.21.15.40:0/1644490744 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v54) ==== 118+0+0 (secure 0 0 0) 0x7f5f1405a7d0 con 0x7f5f100052e0 2024-04-26T23:05:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.760+0000 7f5f2be00640 1 -- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5f0806c920 msgr2=0x7f5f0806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.760+0000 7f5f2be00640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5f0806c920 0x7f5f0806ede0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c001460 tx=0x7f5f1c0012d0 comp rx=0 tx=0).stop 2024-04-26T23:05:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.760+0000 7f5f2be00640 1 -- 172.21.15.40:0/1644490744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f100052e0 msgr2=0x7f5f2c0b7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.760+0000 7f5f2be00640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f100052e0 0x7f5f2c0b7e30 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5f14011540 tx=0x7f5f14011a00 comp rx=0 tx=0).stop 2024-04-26T23:05:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.761+0000 7f5f2be00640 1 -- 172.21.15.40:0/1644490744 shutdown_connections 2024-04-26T23:05:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.761+0000 7f5f2be00640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5f0806c920 0x7f5f0806ede0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.761+0000 7f5f2be00640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f2c1f52b0 0x7f5f2c1f1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.761+0000 7f5f2be00640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f2c1f5e80 0x7f5f2c1f4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:34.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.761+0000 7f5f2be00640 1 --2- 172.21.15.40:0/1644490744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f100052e0 0x7f5f2c0b7e30 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:34.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.761+0000 7f5f2be00640 1 -- 172.21.15.40:0/1644490744 >> 172.21.15.40:0/1644490744 conn(0x7f5f2c0d4db0 msgr2=0x7f5f2c0cf880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:34.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.761+0000 7f5f2be00640 1 -- 172.21.15.40:0/1644490744 shutdown_connections 2024-04-26T23:05:34.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:34.761+0000 7f5f2be00640 1 -- 172.21.15.40:0/1644490744 wait complete. 2024-04-26T23:05:34.785 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-26T23:05:34.785 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-26T23:05:35.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.007+0000 7f99c6000640 1 Processor -- start 2024-04-26T23:05:35.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.008+0000 7f99c6000640 1 -- start start 2024-04-26T23:05:35.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.009+0000 7f99c6000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00cc480 0x7f99c0072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:35.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.009+0000 7f99c6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99c015d7c0 con 0x7f99c0071e60 2024-04-26T23:05:35.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.009+0000 7f99c6000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99c015daa0 con 0x7f99c0072780 2024-04-26T23:05:35.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.009+0000 7f99c6000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f99c015dd80 con 0x7f99c00cc480 2024-04-26T23:05:35.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.010+0000 7f99bf400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f99c0071e60 0x7f99c006f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41144/0 (socket says 172.21.15.40:41144) 2024-04-26T23:05:35.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.010+0000 7f99bf400640 1 -- 172.21.15.40:0/224637343 learned_addr learned my addr 172.21.15.40:0/224637343 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:35.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.010+0000 7f99bea00640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00cc480 0x7f99c0072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:35.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.010+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1582522312 0 0) 0x7f99c015d7c0 con 0x7f99c0071e60 2024-04-26T23:05:35.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.010+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99a4003520 con 0x7f99c0071e60 2024-04-26T23:05:35.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1059326998 0 0) 0x7f99a4003520 con 0x7f99c0071e60 2024-04-26T23:05:35.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00cc480 msgr2=0x7f99c0072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:35.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99be000640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00cc480 0x7f99c0072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:35.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 >> v1:172.21.15.103:6789/0 conn(0x7f99c0072780 legacy=0x7f99c0166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:35.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c015e060 con 0x7f99c0071e60 2024-04-26T23:05:35.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f99b00019f0 con 0x7f99c0071e60 2024-04-26T23:05:35.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 >> v1:172.21.15.40:6789/0 conn(0x7f99c0071e60 legacy=0x7f99c006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:35.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99be000640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99a40039e0 0x7f99a4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:35.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99be000640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a4004390 0x7f99a4004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:35.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99be000640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00cc480 0x7f99a40050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:35.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f99c015dd80 con 0x7f99a40039e0 2024-04-26T23:05:35.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f99a400b9c0 con 0x7f99a4004390 2024-04-26T23:05:35.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f99a400bbc0 con 0x7f99c00cc480 2024-04-26T23:05:35.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.011+0000 7f99bf400640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99a40039e0 0x7f99a4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:35.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.012+0000 7f99bea00640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a4004390 0x7f99a4004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:35.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.012+0000 7f99bf400640 1 -- 172.21.15.40:0/224637343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00cc480 msgr2=0x7f99a40050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:35.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.012+0000 7f99bfe00640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00cc480 0x7f99a40050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:35.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.012+0000 7f99bf400640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00cc480 0x7f99a40050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:35.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.012+0000 7f99bf400640 1 -- 172.21.15.40:0/224637343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a4004390 msgr2=0x7f99a4004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:35.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.012+0000 7f99bf400640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a4004390 0x7f99a4004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:35.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.012+0000 7f99bf400640 1 -- 172.21.15.40:0/224637343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a4003520 con 0x7f99a40039e0 2024-04-26T23:05:35.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.012+0000 7f99bea00640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a4004390 0x7f99a4004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:35.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.012+0000 7f99bfe00640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00cc480 0x7f99a40050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:35.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.013+0000 7f99bf400640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99a40039e0 0x7f99a4003dc0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f99b000a7c0 tx=0x7f99a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3f026ab274da2d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:35.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.013+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b000a410 con 0x7f99a40039e0 2024-04-26T23:05:35.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.013+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b002f070 con 0x7f99a40039e0 2024-04-26T23:05:35.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.013+0000 7f99be000640 1 -- 172.21.15.40:0/224637343 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b002bc30 con 0x7f99a40039e0 2024-04-26T23:05:35.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.014+0000 7f99c6000640 1 -- 172.21.15.40:0/224637343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99a40039e0 msgr2=0x7f99a4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:35.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.014+0000 7f99c6000640 1 --2- 172.21.15.40:0/224637343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99a40039e0 0x7f99a4003dc0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f99b000a7c0 tx=0x7f99a4005830 comp rx=0 tx=0).stop 2024-04-26T23:05:35.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.014+0000 7f99bfe00640 1 -- 172.21.15.40:0/224637343 reap_dead start 2024-04-26T23:05:35.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.014+0000 7f99c6000640 1 -- 172.21.15.40:0/224637343 shutdown_connections 2024-04-26T23:05:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.014+0000 7f99c6000640 1 -- 172.21.15.40:0/224637343 >> 172.21.15.40:0/224637343 conn(0x7f99c00d0be0 msgr2=0x7f99c00d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.014+0000 7f99c6000640 1 -- 172.21.15.40:0/224637343 shutdown_connections 2024-04-26T23:05:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.014+0000 7f99c6000640 1 -- 172.21.15.40:0/224637343 wait complete. 2024-04-26T23:05:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.016+0000 7f99c6000640 1 Processor -- start 2024-04-26T23:05:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.016+0000 7f99c6000640 1 -- start start 2024-04-26T23:05:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.017+0000 7f99c6000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a40039e0 0x7f99c014ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.017+0000 7f99c6000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00b3850 0x7f99c00b5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.017+0000 7f99c6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99c00b3cd0 0x7f99c00b58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.017+0000 7f99c6000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f99c015d3f0 con 0x7f99c00b3cd0 2024-04-26T23:05:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.017+0000 7f99c6000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f99c0072c10 con 0x7f99a40039e0 2024-04-26T23:05:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.017+0000 7f99c6000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f99c015d570 con 0x7f99c00b3850 2024-04-26T23:05:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.017+0000 7f99bfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99c00b3cd0 0x7f99c00b58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:35.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.018+0000 7f99bfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99c00b3cd0 0x7f99c00b58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46198/0 (socket says 172.21.15.40:46198) 2024-04-26T23:05:35.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.018+0000 7f99bfe00640 1 -- 172.21.15.40:0/3983184354 learned_addr learned my addr 172.21.15.40:0/3983184354 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:35.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.018+0000 7f99bf400640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a40039e0 0x7f99c014ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:35.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.018+0000 7f99bea00640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00b3850 0x7f99c00b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:35.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.018+0000 7f99bfe00640 1 -- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00b3850 msgr2=0x7f99c00b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:35.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.018+0000 7f99bfe00640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00b3850 0x7f99c00b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:35.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.018+0000 7f99bfe00640 1 -- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a40039e0 msgr2=0x7f99c014ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:35.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.018+0000 7f99bfe00640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a40039e0 0x7f99c014ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:35.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.018+0000 7f99bfe00640 1 -- 172.21.15.40:0/3983184354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c00b4860 con 0x7f99c00b3cd0 2024-04-26T23:05:35.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.018+0000 7f99bf400640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a40039e0 0x7f99c014ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:05:35.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.018+0000 7f99bfe00640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99c00b3cd0 0x7f99c00b58c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f99c01609e0 tx=0x7f99b800d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:35.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.019+0000 7f99b7400640 1 -- 172.21.15.40:0/3983184354 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8018070 con 0x7f99c00b3cd0 2024-04-26T23:05:35.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.019+0000 7f99b7400640 1 -- 172.21.15.40:0/3983184354 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b80035f0 con 0x7f99c00b3cd0 2024-04-26T23:05:35.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.019+0000 7f99c6000640 1 -- 172.21.15.40:0/3983184354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99c00b42a0 con 0x7f99c00b3cd0 2024-04-26T23:05:35.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.019+0000 7f99b7400640 1 -- 172.21.15.40:0/3983184354 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b80039e0 con 0x7f99c00b3cd0 2024-04-26T23:05:35.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.019+0000 7f99c6000640 1 -- 172.21.15.40:0/3983184354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99c0158f70 con 0x7f99c00b3cd0 2024-04-26T23:05:35.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.020+0000 7f99c6000640 1 -- 172.21.15.40:0/3983184354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99c00694f0 con 0x7f99c00b3cd0 2024-04-26T23:05:35.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.021+0000 7f99b7400640 1 -- 172.21.15.40:0/3983184354 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f99b801b070 con 0x7f99c00b3cd0 2024-04-26T23:05:35.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.022+0000 7f99b7400640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f999c06c920 0x7f999c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:35.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.022+0000 7f99b7400640 1 -- 172.21.15.40:0/3983184354 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f99b808b1a0 con 0x7f99c00b3cd0 2024-04-26T23:05:35.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.023+0000 7f99bf400640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f999c06c920 0x7f999c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:35.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.024+0000 7f99bf400640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f999c06c920 0x7f999c06ede0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f99b000a7c0 tx=0x7f99b0007000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:35.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.028+0000 7f99b7400640 1 -- 172.21.15.40:0/3983184354 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f99b8059440 con 0x7f99c00b3cd0 2024-04-26T23:05:35.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.279+0000 7f99c6000640 1 -- 172.21.15.40:0/3983184354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f99c00b2060 con 0x7f99c00b3cd0 2024-04-26T23:05:35.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.759+0000 7f99b7400640 1 -- 172.21.15.40:0/3983184354 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v55) ==== 122+0+0 (secure 0 0 0) 0x7f99b8058d40 con 0x7f99c00b3cd0 2024-04-26T23:05:35.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.764+0000 7f99c6000640 1 -- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f999c06c920 msgr2=0x7f999c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:35.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.764+0000 7f99c6000640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f999c06c920 0x7f999c06ede0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f99b000a7c0 tx=0x7f99b0007000 comp rx=0 tx=0).stop 2024-04-26T23:05:35.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.764+0000 7f99c6000640 1 -- 172.21.15.40:0/3983184354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99c00b3cd0 msgr2=0x7f99c00b58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:35.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.764+0000 7f99c6000640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99c00b3cd0 0x7f99c00b58c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f99c01609e0 tx=0x7f99b800d890 comp rx=0 tx=0).stop 2024-04-26T23:05:35.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.764+0000 7f99c6000640 1 -- 172.21.15.40:0/3983184354 shutdown_connections 2024-04-26T23:05:35.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.764+0000 7f99c6000640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f999c06c920 0x7f999c06ede0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:35.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.764+0000 7f99c6000640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99c00b3cd0 0x7f99c00b58c0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:35.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.764+0000 7f99c6000640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99c00b3850 0x7f99c00b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:35.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.764+0000 7f99c6000640 1 --2- 172.21.15.40:0/3983184354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a40039e0 0x7f99c014ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:35.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.764+0000 7f99c6000640 1 -- 172.21.15.40:0/3983184354 >> 172.21.15.40:0/3983184354 conn(0x7f99c00d0be0 msgr2=0x7f99c015ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:35.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.764+0000 7f99c6000640 1 -- 172.21.15.40:0/3983184354 shutdown_connections 2024-04-26T23:05:35.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:35.764+0000 7f99c6000640 1 -- 172.21.15.40:0/3983184354 wait complete. 2024-04-26T23:05:35.789 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-26T23:05:35.790 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:05:36.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.001+0000 7f8339000640 1 Processor -- start 2024-04-26T23:05:36.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.001+0000 7f8339000640 1 -- start start 2024-04-26T23:05:36.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.003+0000 7f8339000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f83340af7a0 0x7f83340afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.003+0000 7f8339000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8334071fe0 0x7f8334073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.003+0000 7f8339000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83340d0820 con 0x7f833415ce60 2024-04-26T23:05:36.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.003+0000 7f8339000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f83340cc450 con 0x7f83340af7a0 2024-04-26T23:05:36.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.003+0000 7f8339000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f83341614c0 con 0x7f8334071fe0 2024-04-26T23:05:36.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.003+0000 7f8333400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f833415ce60 0x7f83340ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41150/0 (socket says 172.21.15.40:41150) 2024-04-26T23:05:36.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.003+0000 7f8333400640 1 -- 172.21.15.40:0/286494179 learned_addr learned my addr 172.21.15.40:0/286494179 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:36.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.003+0000 7f832be00640 1 --2- 172.21.15.40:0/286494179 >> v2:172.21.15.40:3300/0 conn(0x7f83340af7a0 0x7f83340afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.004+0000 7f8333e00640 1 --2- 172.21.15.40:0/286494179 >> v2:172.21.15.103:3300/0 conn(0x7f8334071fe0 0x7f8334073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.004+0000 7f8332a00640 1 -- 172.21.15.40:0/286494179 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3373750091 0 0) 0x7f83340d0820 con 0x7f833415ce60 2024-04-26T23:05:36.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.004+0000 7f8332a00640 1 -- 172.21.15.40:0/286494179 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8314003520 con 0x7f833415ce60 2024-04-26T23:05:36.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.004+0000 7f832be00640 1 -- 172.21.15.40:0/286494179 >> v2:172.21.15.103:3300/0 conn(0x7f8334071fe0 msgr2=0x7f8334073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:36.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.004+0000 7f832be00640 1 --2- 172.21.15.40:0/286494179 >> v2:172.21.15.103:3300/0 conn(0x7f8334071fe0 0x7f8334073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.004+0000 7f832be00640 1 -- 172.21.15.40:0/286494179 >> v1:172.21.15.40:6789/0 conn(0x7f833415ce60 legacy=0x7f83340ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.004+0000 7f832be00640 1 -- 172.21.15.40:0/286494179 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8334161760 con 0x7f83340af7a0 2024-04-26T23:05:36.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.005+0000 7f832be00640 1 --2- 172.21.15.40:0/286494179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83340af7a0 0x7f83340afc60 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f83340d1af0 tx=0x7f831c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0186134a0e328c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:36.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.005+0000 7f8332a00640 1 -- 172.21.15.40:0/286494179 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c037070 con 0x7f83340af7a0 2024-04-26T23:05:36.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.005+0000 7f8332a00640 1 -- 172.21.15.40:0/286494179 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f831c032440 con 0x7f83340af7a0 2024-04-26T23:05:36.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.005+0000 7f8332a00640 1 -- 172.21.15.40:0/286494179 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c02ae60 con 0x7f83340af7a0 2024-04-26T23:05:36.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.006+0000 7f8339000640 1 -- 172.21.15.40:0/286494179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83340af7a0 msgr2=0x7f83340afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.006+0000 7f8339000640 1 --2- 172.21.15.40:0/286494179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83340af7a0 0x7f83340afc60 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f83340d1af0 tx=0x7f831c02a110 comp rx=0 tx=0).stop 2024-04-26T23:05:36.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.006+0000 7f8339000640 1 -- 172.21.15.40:0/286494179 shutdown_connections 2024-04-26T23:05:36.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.006+0000 7f8339000640 1 --2- 172.21.15.40:0/286494179 >> v2:172.21.15.103:3300/0 conn(0x7f8334071fe0 0x7f8334073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.006+0000 7f8339000640 1 --2- 172.21.15.40:0/286494179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83340af7a0 0x7f83340afc60 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.006+0000 7f8339000640 1 -- 172.21.15.40:0/286494179 >> 172.21.15.40:0/286494179 conn(0x7f83340d0a10 msgr2=0x7f83340d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:36.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.006+0000 7f8339000640 1 -- 172.21.15.40:0/286494179 shutdown_connections 2024-04-26T23:05:36.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.006+0000 7f8339000640 1 -- 172.21.15.40:0/286494179 wait complete. 2024-04-26T23:05:36.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.008+0000 7f8339000640 1 Processor -- start 2024-04-26T23:05:36.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.008+0000 7f8339000640 1 -- start start 2024-04-26T23:05:36.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.009+0000 7f8339000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8334071fe0 0x7f83340c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.009+0000 7f8339000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83340af7a0 0x7f83340c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.009+0000 7f8339000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f833415ce60 0x7f83341f4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.009+0000 7f8339000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f83340cc1f0 con 0x7f83340af7a0 2024-04-26T23:05:36.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.009+0000 7f8339000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f83340cc070 con 0x7f833415ce60 2024-04-26T23:05:36.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.009+0000 7f8339000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8334073ca0 con 0x7f8334071fe0 2024-04-26T23:05:36.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.009+0000 7f832be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83340af7a0 0x7f83340c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.009+0000 7f832be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83340af7a0 0x7f83340c62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46222/0 (socket says 172.21.15.40:46222) 2024-04-26T23:05:36.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.009+0000 7f832be00640 1 -- 172.21.15.40:0/1829273846 learned_addr learned my addr 172.21.15.40:0/1829273846 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:36.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.009+0000 7f832be00640 1 -- 172.21.15.40:0/1829273846 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8334071fe0 msgr2=0x7f83340c7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.009+0000 7f832be00640 1 --2- 172.21.15.40:0/1829273846 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8334071fe0 0x7f83340c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.010+0000 7f8333e00640 1 --2- 172.21.15.40:0/1829273846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f833415ce60 0x7f83341f4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.010+0000 7f832be00640 1 -- 172.21.15.40:0/1829273846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f833415ce60 msgr2=0x7f83341f4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.010+0000 7f832be00640 1 --2- 172.21.15.40:0/1829273846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f833415ce60 0x7f83341f4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.010+0000 7f832be00640 1 -- 172.21.15.40:0/1829273846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83341f5340 con 0x7f83340af7a0 2024-04-26T23:05:36.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.010+0000 7f832be00640 1 --2- 172.21.15.40:0/1829273846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83340af7a0 0x7f83340c62e0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f831c009890 tx=0x7f831c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:36.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.010+0000 7f832b400640 1 -- 172.21.15.40:0/1829273846 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c037040 con 0x7f83340af7a0 2024-04-26T23:05:36.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.010+0000 7f832b400640 1 -- 172.21.15.40:0/1829273846 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f831c042070 con 0x7f83340af7a0 2024-04-26T23:05:36.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.010+0000 7f832b400640 1 -- 172.21.15.40:0/1829273846 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c030920 con 0x7f83340af7a0 2024-04-26T23:05:36.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.010+0000 7f8339000640 1 -- 172.21.15.40:0/1829273846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83341f5620 con 0x7f83340af7a0 2024-04-26T23:05:36.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.010+0000 7f8339000640 1 -- 172.21.15.40:0/1829273846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83340c7880 con 0x7f83340af7a0 2024-04-26T23:05:36.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.011+0000 7f8339000640 1 -- 172.21.15.40:0/1829273846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8334069370 con 0x7f83340af7a0 2024-04-26T23:05:36.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.012+0000 7f832b400640 1 -- 172.21.15.40:0/1829273846 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f831c030ac0 con 0x7f83340af7a0 2024-04-26T23:05:36.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.013+0000 7f832b400640 1 --2- 172.21.15.40:0/1829273846 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f831006c8f0 0x7f831006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.013+0000 7f832b400640 1 -- 172.21.15.40:0/1829273846 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f831c0ac690 con 0x7f83340af7a0 2024-04-26T23:05:36.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.014+0000 7f8333400640 1 --2- 172.21.15.40:0/1829273846 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f831006c8f0 0x7f831006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.015+0000 7f8333400640 1 --2- 172.21.15.40:0/1829273846 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f831006c8f0 0x7f831006edb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8324000f60 tx=0x7f83240078f0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:36.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.018+0000 7f832b400640 1 -- 172.21.15.40:0/1829273846 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f831c07a2d0 con 0x7f83340af7a0 2024-04-26T23:05:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.252+0000 7f8339000640 1 -- 172.21.15.40:0/1829273846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83340d01f0 con 0x7f83340af7a0 2024-04-26T23:05:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.253+0000 7f832b400640 1 -- 172.21.15.40:0/1829273846 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5185 (secure 0 0 0) 0x7f831c07e5d0 con 0x7f83340af7a0 2024-04-26T23:05:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:36.254 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":55,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/4194140680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4194140680},{"type":"v1","addr":"172.21.15.103:6836","nonce":4194140680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":5324,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2595375825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2595375825},{"type":"v1","addr":"172.21.15.103:6837","nonce":2595375825}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":5614,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/2117861921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2117861921},{"type":"v1","addr":"172.21.15.40:6834","nonce":2117861921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:05:32.212319+0000","modified":"2024-04-26T23:05:35.281037+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/931483025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":931483025},{"type":"v1","addr":"172.21.15.40:6835","nonce":931483025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6774,"qdb_cluster":[6774]},"id":4}]} 2024-04-26T23:05:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 55 2024-04-26T23:05:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.256+0000 7f8339000640 1 -- 172.21.15.40:0/1829273846 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f831006c8f0 msgr2=0x7f831006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.256+0000 7f8339000640 1 --2- 172.21.15.40:0/1829273846 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f831006c8f0 0x7f831006edb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8324000f60 tx=0x7f83240078f0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.256+0000 7f8339000640 1 -- 172.21.15.40:0/1829273846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83340af7a0 msgr2=0x7f83340c62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.256+0000 7f8339000640 1 --2- 172.21.15.40:0/1829273846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83340af7a0 0x7f83340c62e0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f831c009890 tx=0x7f831c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.257+0000 7f8339000640 1 -- 172.21.15.40:0/1829273846 shutdown_connections 2024-04-26T23:05:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.257+0000 7f8339000640 1 --2- 172.21.15.40:0/1829273846 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f831006c8f0 0x7f831006edb0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.257+0000 7f8339000640 1 --2- 172.21.15.40:0/1829273846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f833415ce60 0x7f83341f4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.257+0000 7f8339000640 1 --2- 172.21.15.40:0/1829273846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83340af7a0 0x7f83340c62e0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.257+0000 7f8339000640 1 --2- 172.21.15.40:0/1829273846 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8334071fe0 0x7f83340c7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.257+0000 7f8339000640 1 -- 172.21.15.40:0/1829273846 >> 172.21.15.40:0/1829273846 conn(0x7f83340d0a10 msgr2=0x7f83340b1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.257+0000 7f8339000640 1 -- 172.21.15.40:0/1829273846 shutdown_connections 2024-04-26T23:05:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.257+0000 7f8339000640 1 -- 172.21.15.40:0/1829273846 wait complete. 2024-04-26T23:05:36.275 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:05:36.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.482+0000 7fa423000640 1 Processor -- start 2024-04-26T23:05:36.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.482+0000 7fa423000640 1 -- start start 2024-04-26T23:05:36.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.483+0000 7fa423000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa41c0b6220 0x7fa41c0d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.484+0000 7fa423000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41c0d39f0 0x7fa41c0b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.484+0000 7fa423000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41c162c40 con 0x7fa41c0d08d0 2024-04-26T23:05:36.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.484+0000 7fa423000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa41c0d2d80 con 0x7fa41c0b6220 2024-04-26T23:05:36.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.484+0000 7fa423000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa41c0b7c80 con 0x7fa41c0d39f0 2024-04-26T23:05:36.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.484+0000 7fa421200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa41c0d08d0 0x7fa41c0d32e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41158/0 (socket says 172.21.15.40:41158) 2024-04-26T23:05:36.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.484+0000 7fa421200640 1 -- 172.21.15.40:0/2467475996 learned_addr learned my addr 172.21.15.40:0/2467475996 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:36.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.484+0000 7fa421c00640 1 --2- 172.21.15.40:0/2467475996 >> v2:172.21.15.40:3300/0 conn(0x7fa41c0b6220 0x7fa41c0d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.484+0000 7fa422600640 1 --2- 172.21.15.40:0/2467475996 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41c0d39f0 0x7fa41c0b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.485+0000 7fa41be00640 1 -- 172.21.15.40:0/2467475996 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2862893538 0 0) 0x7fa41c162c40 con 0x7fa41c0d08d0 2024-04-26T23:05:36.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.485+0000 7fa421c00640 1 -- 172.21.15.40:0/2467475996 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41c0d39f0 msgr2=0x7fa41c0b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:36.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.485+0000 7fa421c00640 1 --2- 172.21.15.40:0/2467475996 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41c0d39f0 0x7fa41c0b8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.485+0000 7fa421c00640 1 -- 172.21.15.40:0/2467475996 >> v1:172.21.15.40:6789/0 conn(0x7fa41c0d08d0 legacy=0x7fa41c0d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.485+0000 7fa421c00640 1 -- 172.21.15.40:0/2467475996 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41c0b7f60 con 0x7fa41c0b6220 2024-04-26T23:05:36.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.486+0000 7fa421c00640 1 --2- 172.21.15.40:0/2467475996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa41c0b6220 0x7fa41c0d0390 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fa40c009d90 tx=0x7fa40c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2cc80acb4f82b8b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:36.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.486+0000 7fa41be00640 1 -- 172.21.15.40:0/2467475996 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c007740 con 0x7fa41c0b6220 2024-04-26T23:05:36.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.486+0000 7fa41be00640 1 -- 172.21.15.40:0/2467475996 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa40c0078e0 con 0x7fa41c0b6220 2024-04-26T23:05:36.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.486+0000 7fa41be00640 1 -- 172.21.15.40:0/2467475996 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c007cb0 con 0x7fa41c0b6220 2024-04-26T23:05:36.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.487+0000 7fa423000640 1 -- 172.21.15.40:0/2467475996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa41c0b6220 msgr2=0x7fa41c0d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.487+0000 7fa423000640 1 --2- 172.21.15.40:0/2467475996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa41c0b6220 0x7fa41c0d0390 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fa40c009d90 tx=0x7fa40c02a660 comp rx=0 tx=0).stop 2024-04-26T23:05:36.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.487+0000 7fa423000640 1 -- 172.21.15.40:0/2467475996 shutdown_connections 2024-04-26T23:05:36.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.487+0000 7fa423000640 1 --2- 172.21.15.40:0/2467475996 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41c0d39f0 0x7fa41c0b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.487+0000 7fa423000640 1 --2- 172.21.15.40:0/2467475996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa41c0b6220 0x7fa41c0d0390 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.487+0000 7fa423000640 1 -- 172.21.15.40:0/2467475996 >> 172.21.15.40:0/2467475996 conn(0x7fa41c0d4d90 msgr2=0x7fa41c0d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:36.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.487+0000 7fa423000640 1 -- 172.21.15.40:0/2467475996 shutdown_connections 2024-04-26T23:05:36.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.487+0000 7fa423000640 1 -- 172.21.15.40:0/2467475996 wait complete. 2024-04-26T23:05:36.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.488+0000 7fa423000640 1 Processor -- start 2024-04-26T23:05:36.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.489+0000 7fa423000640 1 -- start start 2024-04-26T23:05:36.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.490+0000 7fa423000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41c0b6220 0x7fa41c0a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.490+0000 7fa423000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa41c0d08d0 0x7fa41c0a2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.490+0000 7fa423000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41c0d39f0 0x7fa41c0a2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.490+0000 7fa423000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa41c0b9260 con 0x7fa41c0d08d0 2024-04-26T23:05:36.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.490+0000 7fa423000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa41c0b8520 con 0x7fa41c0d39f0 2024-04-26T23:05:36.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.490+0000 7fa423000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa41c0b93e0 con 0x7fa41c0b6220 2024-04-26T23:05:36.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.490+0000 7fa421200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa41c0d08d0 0x7fa41c0a2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.490+0000 7fa421200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa41c0d08d0 0x7fa41c0a2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46236/0 (socket says 172.21.15.40:46236) 2024-04-26T23:05:36.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.490+0000 7fa421200640 1 -- 172.21.15.40:0/949377477 learned_addr learned my addr 172.21.15.40:0/949377477 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:36.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.490+0000 7fa421c00640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41c0b6220 0x7fa41c0a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.490+0000 7fa422600640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41c0d39f0 0x7fa41c0a2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.491+0000 7fa421200640 1 -- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41c0b6220 msgr2=0x7fa41c0a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.491+0000 7fa421200640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41c0b6220 0x7fa41c0a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.491+0000 7fa421200640 1 -- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41c0d39f0 msgr2=0x7fa41c0a2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.491+0000 7fa421200640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41c0d39f0 0x7fa41c0a2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.491+0000 7fa421200640 1 -- 172.21.15.40:0/949377477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41c0b29d0 con 0x7fa41c0d08d0 2024-04-26T23:05:36.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.491+0000 7fa421200640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa41c0d08d0 0x7fa41c0a2010 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fa40800c4d0 tx=0x7fa40800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:36.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.491+0000 7fa419600640 1 -- 172.21.15.40:0/949377477 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408018070 con 0x7fa41c0d08d0 2024-04-26T23:05:36.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.491+0000 7fa419600640 1 -- 172.21.15.40:0/949377477 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa408007500 con 0x7fa41c0d08d0 2024-04-26T23:05:36.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.491+0000 7fa419600640 1 -- 172.21.15.40:0/949377477 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408004a20 con 0x7fa41c0d08d0 2024-04-26T23:05:36.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.491+0000 7fa423000640 1 -- 172.21.15.40:0/949377477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa41c0b2cb0 con 0x7fa41c0d08d0 2024-04-26T23:05:36.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.491+0000 7fa423000640 1 -- 172.21.15.40:0/949377477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa41c0b3270 con 0x7fa41c0d08d0 2024-04-26T23:05:36.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.493+0000 7fa423000640 1 -- 172.21.15.40:0/949377477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa41c069370 con 0x7fa41c0d08d0 2024-04-26T23:05:36.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.494+0000 7fa419600640 1 -- 172.21.15.40:0/949377477 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa408004bc0 con 0x7fa41c0d08d0 2024-04-26T23:05:36.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.494+0000 7fa419600640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa3fc06c8f0 0x7fa3fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.495+0000 7fa419600640 1 -- 172.21.15.40:0/949377477 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fa40808af40 con 0x7fa41c0d08d0 2024-04-26T23:05:36.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.495+0000 7fa421c00640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa3fc06c8f0 0x7fa3fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.496+0000 7fa421c00640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa3fc06c8f0 0x7fa3fc06edb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa40c034000 tx=0x7fa40c02f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:36.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.499+0000 7fa419600640 1 -- 172.21.15.40:0/949377477 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa408059270 con 0x7fa41c0d08d0 2024-04-26T23:05:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.702+0000 7fa423000640 1 -- 172.21.15.40:0/949377477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa41c0d30c0 con 0x7fa41c0d08d0 2024-04-26T23:05:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.705+0000 7fa419600640 1 -- 172.21.15.40:0/949377477 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) ==== 74+0+16848 (secure 0 0 0) 0x7fa408058b70 con 0x7fa41c0d08d0 2024-04-26T23:05:36.705 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:36.705 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":65,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:05:33.743646+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:05:30.081199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-26T23:05:31.192158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:05:36.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.707+0000 7fa423000640 1 -- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa3fc06c8f0 msgr2=0x7fa3fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.707+0000 7fa423000640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa3fc06c8f0 0x7fa3fc06edb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa40c034000 tx=0x7fa40c02f000 comp rx=0 tx=0).stop 2024-04-26T23:05:36.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.707+0000 7fa423000640 1 -- 172.21.15.40:0/949377477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa41c0d08d0 msgr2=0x7fa41c0a2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.707+0000 7fa423000640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa41c0d08d0 0x7fa41c0a2010 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fa40800c4d0 tx=0x7fa40800c990 comp rx=0 tx=0).stop 2024-04-26T23:05:36.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.708+0000 7fa423000640 1 -- 172.21.15.40:0/949377477 shutdown_connections 2024-04-26T23:05:36.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.708+0000 7fa423000640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa3fc06c8f0 0x7fa3fc06edb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.708+0000 7fa423000640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa41c0d39f0 0x7fa41c0a2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.708+0000 7fa423000640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa41c0d08d0 0x7fa41c0a2010 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.708+0000 7fa423000640 1 --2- 172.21.15.40:0/949377477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa41c0b6220 0x7fa41c0a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.708+0000 7fa423000640 1 -- 172.21.15.40:0/949377477 >> 172.21.15.40:0/949377477 conn(0x7fa41c0d4d90 msgr2=0x7fa41c0d32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:36.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.708+0000 7fa423000640 1 -- 172.21.15.40:0/949377477 shutdown_connections 2024-04-26T23:05:36.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.708+0000 7fa423000640 1 -- 172.21.15.40:0/949377477 wait complete. 2024-04-26T23:05:36.727 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T23:05:36.727 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:05:36.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.931+0000 7fe844a00640 1 Processor -- start 2024-04-26T23:05:36.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.931+0000 7fe844a00640 1 -- start start 2024-04-26T23:05:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.933+0000 7fe844a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe8400d32e0 0x7fe8400d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.933+0000 7fe844a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8400b8d20 0x7fe8400b91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.933+0000 7fe844a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe840162c40 con 0x7fe8400b6220 2024-04-26T23:05:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.933+0000 7fe844a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe8400d3c80 con 0x7fe8400d32e0 2024-04-26T23:05:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.933+0000 7fe844a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe8400d2d80 con 0x7fe8400b8d20 2024-04-26T23:05:36.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.934+0000 7fe83de00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe8400d32e0 0x7fe8400d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.934+0000 7fe83f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8400b8d20 0x7fe8400b91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.934+0000 7fe83f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8400b8d20 0x7fe8400b91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:34898/0 (socket says 172.21.15.40:34898) 2024-04-26T23:05:36.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.934+0000 7fe83f200640 1 -- 172.21.15.40:0/413830393 learned_addr learned my addr 172.21.15.40:0/413830393 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:36.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.935+0000 7fe83f200640 1 -- 172.21.15.40:0/413830393 >> v2:172.21.15.103:3300/0 conn(0x7fe8400d32e0 msgr2=0x7fe8400d3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:36.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.935+0000 7fe83f200640 1 --2- 172.21.15.40:0/413830393 >> v2:172.21.15.103:3300/0 conn(0x7fe8400d32e0 0x7fe8400d3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.935+0000 7fe83f200640 1 -- 172.21.15.40:0/413830393 >> v1:172.21.15.103:6789/0 conn(0x7fe8400b6220 legacy=0x7fe8400d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.935+0000 7fe83f200640 1 -- 172.21.15.40:0/413830393 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8400b7c80 con 0x7fe8400b8d20 2024-04-26T23:05:36.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.936+0000 7fe83f200640 1 --2- 172.21.15.40:0/413830393 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8400b8d20 0x7fe8400b91b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe838009e50 tx=0x7fe83802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a8c9ab8400d91e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:36.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.936+0000 7fe83d400640 1 -- 172.21.15.40:0/413830393 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838036070 con 0x7fe8400b8d20 2024-04-26T23:05:36.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.937+0000 7fe83d400640 1 -- 172.21.15.40:0/413830393 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe838031600 con 0x7fe8400b8d20 2024-04-26T23:05:36.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.937+0000 7fe83d400640 1 -- 172.21.15.40:0/413830393 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8380319d0 con 0x7fe8400b8d20 2024-04-26T23:05:36.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.937+0000 7fe844a00640 1 -- 172.21.15.40:0/413830393 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8400b8d20 msgr2=0x7fe8400b91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.937+0000 7fe844a00640 1 --2- 172.21.15.40:0/413830393 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8400b8d20 0x7fe8400b91b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe838009e50 tx=0x7fe83802a710 comp rx=0 tx=0).stop 2024-04-26T23:05:36.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.937+0000 7fe844a00640 1 -- 172.21.15.40:0/413830393 shutdown_connections 2024-04-26T23:05:36.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.937+0000 7fe844a00640 1 --2- 172.21.15.40:0/413830393 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8400b8d20 0x7fe8400b91b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.937+0000 7fe844a00640 1 --2- 172.21.15.40:0/413830393 >> v2:172.21.15.103:3300/0 conn(0x7fe8400d32e0 0x7fe8400d3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.937+0000 7fe844a00640 1 -- 172.21.15.40:0/413830393 >> 172.21.15.40:0/413830393 conn(0x7fe8400d4d90 msgr2=0x7fe8400d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:36.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.938+0000 7fe844a00640 1 -- 172.21.15.40:0/413830393 shutdown_connections 2024-04-26T23:05:36.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.938+0000 7fe844a00640 1 -- 172.21.15.40:0/413830393 wait complete. 2024-04-26T23:05:36.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.940+0000 7fe844a00640 1 Processor -- start 2024-04-26T23:05:36.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.940+0000 7fe844a00640 1 -- start start 2024-04-26T23:05:36.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.941+0000 7fe844a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8400b6220 0x7fe840157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.941+0000 7fe844a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8400b8d20 0x7fe8400cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.941+0000 7fe844a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8400d32e0 0x7fe8400ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.941+0000 7fe844a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe8400b8520 con 0x7fe8400b8d20 2024-04-26T23:05:36.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.941+0000 7fe844a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe8400b8240 con 0x7fe8400d32e0 2024-04-26T23:05:36.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.941+0000 7fe844a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe8400d0aa0 con 0x7fe8400b6220 2024-04-26T23:05:36.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.942+0000 7fe83de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8400b8d20 0x7fe8400cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.942+0000 7fe83de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8400b8d20 0x7fe8400cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46246/0 (socket says 172.21.15.40:46246) 2024-04-26T23:05:36.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.942+0000 7fe83de00640 1 -- 172.21.15.40:0/1414838709 learned_addr learned my addr 172.21.15.40:0/1414838709 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:36.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.942+0000 7fe83e800640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8400b6220 0x7fe840157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.942+0000 7fe83f200640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8400d32e0 0x7fe8400ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.942+0000 7fe83de00640 1 -- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8400b6220 msgr2=0x7fe840157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.942+0000 7fe83de00640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8400b6220 0x7fe840157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.942+0000 7fe83de00640 1 -- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8400d32e0 msgr2=0x7fe8400ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:36.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.942+0000 7fe83de00640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8400d32e0 0x7fe8400ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:36.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.942+0000 7fe83de00640 1 -- 172.21.15.40:0/1414838709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84015ef70 con 0x7fe8400b8d20 2024-04-26T23:05:36.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.943+0000 7fe83de00640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8400b8d20 0x7fe8400cdfa0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00e870 tx=0x7fe82c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:36.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.943+0000 7fe836a00640 1 -- 172.21.15.40:0/1414838709 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c017070 con 0x7fe8400b8d20 2024-04-26T23:05:36.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.943+0000 7fe836a00640 1 -- 172.21.15.40:0/1414838709 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe82c00c940 con 0x7fe8400b8d20 2024-04-26T23:05:36.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.943+0000 7fe836a00640 1 -- 172.21.15.40:0/1414838709 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c00cd30 con 0x7fe8400b8d20 2024-04-26T23:05:36.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.943+0000 7fe844a00640 1 -- 172.21.15.40:0/1414838709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84015ec90 con 0x7fe8400b8d20 2024-04-26T23:05:36.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.943+0000 7fe844a00640 1 -- 172.21.15.40:0/1414838709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe840159150 con 0x7fe8400b8d20 2024-04-26T23:05:36.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.945+0000 7fe844a00640 1 -- 172.21.15.40:0/1414838709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe840069370 con 0x7fe8400b8d20 2024-04-26T23:05:36.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.946+0000 7fe836a00640 1 -- 172.21.15.40:0/1414838709 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe82c01a070 con 0x7fe8400b8d20 2024-04-26T23:05:36.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.947+0000 7fe836a00640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe82006c8f0 0x7fe82006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:36.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.947+0000 7fe836a00640 1 -- 172.21.15.40:0/1414838709 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fe82c08bbf0 con 0x7fe8400b8d20 2024-04-26T23:05:36.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.948+0000 7fe83e800640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe82006c8f0 0x7fe82006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:36.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.949+0000 7fe83e800640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe82006c8f0 0x7fe82006edb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe8401f8810 tx=0x7fe830009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:36.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:36.952+0000 7fe836a00640 1 -- 172.21.15.40:0/1414838709 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe82c059df0 con 0x7fe8400b8d20 2024-04-26T23:05:37.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.160+0000 7fe844a00640 1 -- 172.21.15.40:0/1414838709 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe8401586c0 con 0x7fe82006c8f0 2024-04-26T23:05:37.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.181+0000 7fe836a00640 1 -- 172.21.15.40:0/1414838709 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251949 (secure 0 0 0) 0x7fe8300050d0 con 0x7fe82006c8f0 2024-04-26T23:05:37.184 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:37.186 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:05:37.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.186+0000 7fe844a00640 1 -- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe82006c8f0 msgr2=0x7fe82006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:37.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.186+0000 7fe844a00640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe82006c8f0 0x7fe82006edb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe8401f8810 tx=0x7fe830009040 comp rx=0 tx=0).stop 2024-04-26T23:05:37.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.186+0000 7fe844a00640 1 -- 172.21.15.40:0/1414838709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8400b8d20 msgr2=0x7fe8400cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:37.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.186+0000 7fe844a00640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8400b8d20 0x7fe8400cdfa0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00e870 tx=0x7fe82c00ed30 comp rx=0 tx=0).stop 2024-04-26T23:05:37.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.186+0000 7fe844a00640 1 -- 172.21.15.40:0/1414838709 shutdown_connections 2024-04-26T23:05:37.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.186+0000 7fe844a00640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe82006c8f0 0x7fe82006edb0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.186+0000 7fe844a00640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8400d32e0 0x7fe8400ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.186+0000 7fe844a00640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8400b8d20 0x7fe8400cdfa0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.186+0000 7fe844a00640 1 --2- 172.21.15.40:0/1414838709 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe8400b6220 0x7fe840157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.186+0000 7fe844a00640 1 -- 172.21.15.40:0/1414838709 >> 172.21.15.40:0/1414838709 conn(0x7fe8400d4d90 msgr2=0x7fe840162390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:37.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.186+0000 7fe844a00640 1 -- 172.21.15.40:0/1414838709 shutdown_connections 2024-04-26T23:05:37.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.186+0000 7fe844a00640 1 -- 172.21.15.40:0/1414838709 wait complete. 2024-04-26T23:05:37.207 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-04-26T23:05:35.242785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240012,"kb_used_data":7188,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734708,"statfs":{"total":773094113280,"available":772848340992,"internally_reserved":0,"allocated":7360512,"data_stored":3986429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.528482"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766926+0000","last_change":"2024-04-26T23:05:32.764307+0000","last_active":"2024-04-26T23:05:32.766926+0000","last_peered":"2024-04-26T23:05:32.766926+0000","last_clean":"2024-04-26T23:05:32.766926+0000","last_became_active":"2024-04-26T23:05:32.763424+0000","last_became_peered":"2024-04-26T23:05:32.763424+0000","last_unstale":"2024-04-26T23:05:32.766926+0000","last_undegraded":"2024-04-26T23:05:32.766926+0000","last_fullsized":"2024-04-26T23:05:32.766926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:23.483542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766379+0000","last_change":"2024-04-26T23:05:32.762530+0000","last_active":"2024-04-26T23:05:32.766379+0000","last_peered":"2024-04-26T23:05:32.766379+0000","last_clean":"2024-04-26T23:05:32.766379+0000","last_became_active":"2024-04-26T23:05:32.762234+0000","last_became_peered":"2024-04-26T23:05:32.762234+0000","last_unstale":"2024-04-26T23:05:32.766379+0000","last_undegraded":"2024-04-26T23:05:32.766379+0000","last_fullsized":"2024-04-26T23:05:32.766379+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:48:10.275021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.780372+0000","last_change":"2024-04-26T23:05:32.778787+0000","last_active":"2024-04-26T23:05:32.780372+0000","last_peered":"2024-04-26T23:05:32.780372+0000","last_clean":"2024-04-26T23:05:32.780372+0000","last_became_active":"2024-04-26T23:05:32.778486+0000","last_became_peered":"2024-04-26T23:05:32.778486+0000","last_unstale":"2024-04-26T23:05:32.780372+0000","last_undegraded":"2024-04-26T23:05:32.780372+0000","last_fullsized":"2024-04-26T23:05:32.780372+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:29.278920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.764758+0000","last_change":"2024-04-26T23:05:32.753307+0000","last_active":"2024-04-26T23:05:32.764758+0000","last_peered":"2024-04-26T23:05:32.764758+0000","last_clean":"2024-04-26T23:05:32.764758+0000","last_became_active":"2024-04-26T23:05:32.753017+0000","last_became_peered":"2024-04-26T23:05:32.753017+0000","last_unstale":"2024-04-26T23:05:32.764758+0000","last_undegraded":"2024-04-26T23:05:32.764758+0000","last_fullsized":"2024-04-26T23:05:32.764758+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:21:24.525954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759953+0000","last_change":"2024-04-26T23:05:32.746080+0000","last_active":"2024-04-26T23:05:32.759953+0000","last_peered":"2024-04-26T23:05:32.759953+0000","last_clean":"2024-04-26T23:05:32.759953+0000","last_became_active":"2024-04-26T23:05:32.745767+0000","last_became_peered":"2024-04-26T23:05:32.745767+0000","last_unstale":"2024-04-26T23:05:32.759953+0000","last_undegraded":"2024-04-26T23:05:32.759953+0000","last_fullsized":"2024-04-26T23:05:32.759953+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:28:05.145189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.776305+0000","last_change":"2024-04-26T23:05:32.767886+0000","last_active":"2024-04-26T23:05:32.776305+0000","last_peered":"2024-04-26T23:05:32.776305+0000","last_clean":"2024-04-26T23:05:32.776305+0000","last_became_active":"2024-04-26T23:05:32.766263+0000","last_became_peered":"2024-04-26T23:05:32.766263+0000","last_unstale":"2024-04-26T23:05:32.776305+0000","last_undegraded":"2024-04-26T23:05:32.776305+0000","last_fullsized":"2024-04-26T23:05:32.776305+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:58:01.663542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771035+0000","last_change":"2024-04-26T23:05:32.764840+0000","last_active":"2024-04-26T23:05:32.771035+0000","last_peered":"2024-04-26T23:05:32.771035+0000","last_clean":"2024-04-26T23:05:32.771035+0000","last_became_active":"2024-04-26T23:05:32.764474+0000","last_became_peered":"2024-04-26T23:05:32.764474+0000","last_unstale":"2024-04-26T23:05:32.771035+0000","last_undegraded":"2024-04-26T23:05:32.771035+0000","last_fullsized":"2024-04-26T23:05:32.771035+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:52:16.509999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.769295+0000","last_change":"2024-04-26T23:05:32.765699+0000","last_active":"2024-04-26T23:05:32.769295+0000","last_peered":"2024-04-26T23:05:32.769295+0000","last_clean":"2024-04-26T23:05:32.769295+0000","last_became_active":"2024-04-26T23:05:32.765385+0000","last_became_peered":"2024-04-26T23:05:32.765385+0000","last_unstale":"2024-04-26T23:05:32.769295+0000","last_undegraded":"2024-04-26T23:05:32.769295+0000","last_fullsized":"2024-04-26T23:05:32.769295+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:42:49.983638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.770590+0000","last_change":"2024-04-26T23:05:32.767125+0000","last_active":"2024-04-26T23:05:32.770590+0000","last_peered":"2024-04-26T23:05:32.770590+0000","last_clean":"2024-04-26T23:05:32.770590+0000","last_became_active":"2024-04-26T23:05:32.766787+0000","last_became_peered":"2024-04-26T23:05:32.766787+0000","last_unstale":"2024-04-26T23:05:32.770590+0000","last_undegraded":"2024-04-26T23:05:32.770590+0000","last_fullsized":"2024-04-26T23:05:32.770590+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:44:44.767658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771698+0000","last_change":"2024-04-26T23:05:32.765251+0000","last_active":"2024-04-26T23:05:32.771698+0000","last_peered":"2024-04-26T23:05:32.771698+0000","last_clean":"2024-04-26T23:05:32.771698+0000","last_became_active":"2024-04-26T23:05:32.764938+0000","last_became_peered":"2024-04-26T23:05:32.764938+0000","last_unstale":"2024-04-26T23:05:32.771698+0000","last_undegraded":"2024-04-26T23:05:32.771698+0000","last_fullsized":"2024-04-26T23:05:32.771698+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:35.989406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.764041+0000","last_change":"2024-04-26T23:05:32.760836+0000","last_active":"2024-04-26T23:05:32.764041+0000","last_peered":"2024-04-26T23:05:32.764041+0000","last_clean":"2024-04-26T23:05:32.764041+0000","last_became_active":"2024-04-26T23:05:32.760557+0000","last_became_peered":"2024-04-26T23:05:32.760557+0000","last_unstale":"2024-04-26T23:05:32.764041+0000","last_undegraded":"2024-04-26T23:05:32.764041+0000","last_fullsized":"2024-04-26T23:05:32.764041+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:48:12.566024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.769853+0000","last_change":"2024-04-26T23:05:32.764677+0000","last_active":"2024-04-26T23:05:32.769853+0000","last_peered":"2024-04-26T23:05:32.769853+0000","last_clean":"2024-04-26T23:05:32.769853+0000","last_became_active":"2024-04-26T23:05:32.761915+0000","last_became_peered":"2024-04-26T23:05:32.761915+0000","last_unstale":"2024-04-26T23:05:32.769853+0000","last_undegraded":"2024-04-26T23:05:32.769853+0000","last_fullsized":"2024-04-26T23:05:32.769853+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:35.989406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.770547+0000","last_change":"2024-04-26T23:05:32.767323+0000","last_active":"2024-04-26T23:05:32.770547+0000","last_peered":"2024-04-26T23:05:32.770547+0000","last_clean":"2024-04-26T23:05:32.770547+0000","last_became_active":"2024-04-26T23:05:32.766419+0000","last_became_peered":"2024-04-26T23:05:32.766419+0000","last_unstale":"2024-04-26T23:05:32.770547+0000","last_undegraded":"2024-04-26T23:05:32.770547+0000","last_fullsized":"2024-04-26T23:05:32.770547+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:24:54.832654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.779405+0000","last_change":"2024-04-26T23:05:32.775842+0000","last_active":"2024-04-26T23:05:32.779405+0000","last_peered":"2024-04-26T23:05:32.779405+0000","last_clean":"2024-04-26T23:05:32.779405+0000","last_became_active":"2024-04-26T23:05:32.775555+0000","last_became_peered":"2024-04-26T23:05:32.775555+0000","last_unstale":"2024-04-26T23:05:32.779405+0000","last_undegraded":"2024-04-26T23:05:32.779405+0000","last_fullsized":"2024-04-26T23:05:32.779405+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:34.076640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.778559+0000","last_change":"2024-04-26T23:05:32.775545+0000","last_active":"2024-04-26T23:05:32.778559+0000","last_peered":"2024-04-26T23:05:32.778559+0000","last_clean":"2024-04-26T23:05:32.778559+0000","last_became_active":"2024-04-26T23:05:32.775205+0000","last_became_peered":"2024-04-26T23:05:32.775205+0000","last_unstale":"2024-04-26T23:05:32.778559+0000","last_undegraded":"2024-04-26T23:05:32.778559+0000","last_fullsized":"2024-04-26T23:05:32.778559+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:53:00.646163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759109+0000","last_change":"2024-04-26T23:05:32.754414+0000","last_active":"2024-04-26T23:05:32.759109+0000","last_peered":"2024-04-26T23:05:32.759109+0000","last_clean":"2024-04-26T23:05:32.759109+0000","last_became_active":"2024-04-26T23:05:32.754060+0000","last_became_peered":"2024-04-26T23:05:32.754060+0000","last_unstale":"2024-04-26T23:05:32.759109+0000","last_undegraded":"2024-04-26T23:05:32.759109+0000","last_fullsized":"2024-04-26T23:05:32.759109+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:06:40.143755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.763609+0000","last_change":"2024-04-26T23:05:32.758763+0000","last_active":"2024-04-26T23:05:32.763609+0000","last_peered":"2024-04-26T23:05:32.763609+0000","last_clean":"2024-04-26T23:05:32.763609+0000","last_became_active":"2024-04-26T23:05:32.750829+0000","last_became_peered":"2024-04-26T23:05:32.750829+0000","last_unstale":"2024-04-26T23:05:32.763609+0000","last_undegraded":"2024-04-26T23:05:32.763609+0000","last_fullsized":"2024-04-26T23:05:32.763609+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:31.289442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.763145+0000","last_change":"2024-04-26T23:05:32.756213+0000","last_active":"2024-04-26T23:05:32.763145+0000","last_peered":"2024-04-26T23:05:32.763145+0000","last_clean":"2024-04-26T23:05:32.763145+0000","last_became_active":"2024-04-26T23:05:32.755856+0000","last_became_peered":"2024-04-26T23:05:32.755856+0000","last_unstale":"2024-04-26T23:05:32.763145+0000","last_undegraded":"2024-04-26T23:05:32.763145+0000","last_fullsized":"2024-04-26T23:05:32.763145+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:35:40.312530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759513+0000","last_change":"2024-04-26T23:05:32.753469+0000","last_active":"2024-04-26T23:05:32.759513+0000","last_peered":"2024-04-26T23:05:32.759513+0000","last_clean":"2024-04-26T23:05:32.759513+0000","last_became_active":"2024-04-26T23:05:32.752705+0000","last_became_peered":"2024-04-26T23:05:32.752705+0000","last_unstale":"2024-04-26T23:05:32.759513+0000","last_undegraded":"2024-04-26T23:05:32.759513+0000","last_fullsized":"2024-04-26T23:05:32.759513+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:47:04.134977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.753171+0000","last_change":"2024-04-26T23:05:32.747912+0000","last_active":"2024-04-26T23:05:32.753171+0000","last_peered":"2024-04-26T23:05:32.753171+0000","last_clean":"2024-04-26T23:05:32.753171+0000","last_became_active":"2024-04-26T23:05:32.747455+0000","last_became_peered":"2024-04-26T23:05:32.747455+0000","last_unstale":"2024-04-26T23:05:32.753171+0000","last_undegraded":"2024-04-26T23:05:32.753171+0000","last_fullsized":"2024-04-26T23:05:32.753171+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:26:54.412751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.767713+0000","last_change":"2024-04-26T23:05:32.763103+0000","last_active":"2024-04-26T23:05:32.767713+0000","last_peered":"2024-04-26T23:05:32.767713+0000","last_clean":"2024-04-26T23:05:32.767713+0000","last_became_active":"2024-04-26T23:05:32.762730+0000","last_became_peered":"2024-04-26T23:05:32.762730+0000","last_unstale":"2024-04-26T23:05:32.767713+0000","last_undegraded":"2024-04-26T23:05:32.767713+0000","last_fullsized":"2024-04-26T23:05:32.767713+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:48:10.275021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.767065+0000","last_change":"2024-04-26T23:05:32.761526+0000","last_active":"2024-04-26T23:05:32.767065+0000","last_peered":"2024-04-26T23:05:32.767065+0000","last_clean":"2024-04-26T23:05:32.767065+0000","last_became_active":"2024-04-26T23:05:32.761187+0000","last_became_peered":"2024-04-26T23:05:32.761187+0000","last_unstale":"2024-04-26T23:05:32.767065+0000","last_undegraded":"2024-04-26T23:05:32.767065+0000","last_fullsized":"2024-04-26T23:05:32.767065+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:58.027734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.770092+0000","last_change":"2024-04-26T23:05:32.765535+0000","last_active":"2024-04-26T23:05:32.770092+0000","last_peered":"2024-04-26T23:05:32.770092+0000","last_clean":"2024-04-26T23:05:32.770092+0000","last_became_active":"2024-04-26T23:05:32.765217+0000","last_became_peered":"2024-04-26T23:05:32.765217+0000","last_unstale":"2024-04-26T23:05:32.770092+0000","last_undegraded":"2024-04-26T23:05:32.770092+0000","last_fullsized":"2024-04-26T23:05:32.770092+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:46.183586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.780280+0000","last_change":"2024-04-26T23:05:32.777775+0000","last_active":"2024-04-26T23:05:32.780280+0000","last_peered":"2024-04-26T23:05:32.780280+0000","last_clean":"2024-04-26T23:05:32.780280+0000","last_became_active":"2024-04-26T23:05:32.776672+0000","last_became_peered":"2024-04-26T23:05:32.776672+0000","last_unstale":"2024-04-26T23:05:32.780280+0000","last_undegraded":"2024-04-26T23:05:32.780280+0000","last_fullsized":"2024-04-26T23:05:32.780280+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:09.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.769650+0000","last_change":"2024-04-26T23:05:32.764515+0000","last_active":"2024-04-26T23:05:32.769650+0000","last_peered":"2024-04-26T23:05:32.769650+0000","last_clean":"2024-04-26T23:05:32.769650+0000","last_became_active":"2024-04-26T23:05:32.763920+0000","last_became_peered":"2024-04-26T23:05:32.763920+0000","last_unstale":"2024-04-26T23:05:32.769650+0000","last_undegraded":"2024-04-26T23:05:32.769650+0000","last_fullsized":"2024-04-26T23:05:32.769650+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:00.038837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766657+0000","last_change":"2024-04-26T23:05:32.763303+0000","last_active":"2024-04-26T23:05:32.766657+0000","last_peered":"2024-04-26T23:05:32.766657+0000","last_clean":"2024-04-26T23:05:32.766657+0000","last_became_active":"2024-04-26T23:05:32.763020+0000","last_became_peered":"2024-04-26T23:05:32.763020+0000","last_unstale":"2024-04-26T23:05:32.766657+0000","last_undegraded":"2024-04-26T23:05:32.766657+0000","last_fullsized":"2024-04-26T23:05:32.766657+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:23.483542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.762623+0000","last_change":"2024-04-26T23:05:32.752684+0000","last_active":"2024-04-26T23:05:32.762623+0000","last_peered":"2024-04-26T23:05:32.762623+0000","last_clean":"2024-04-26T23:05:32.762623+0000","last_became_active":"2024-04-26T23:05:32.749863+0000","last_became_peered":"2024-04-26T23:05:32.749863+0000","last_unstale":"2024-04-26T23:05:32.762623+0000","last_undegraded":"2024-04-26T23:05:32.762623+0000","last_fullsized":"2024-04-26T23:05:32.762623+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:14.712885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.761459+0000","last_change":"2024-04-26T23:05:32.758274+0000","last_active":"2024-04-26T23:05:32.761459+0000","last_peered":"2024-04-26T23:05:32.761459+0000","last_clean":"2024-04-26T23:05:32.761459+0000","last_became_active":"2024-04-26T23:05:32.757953+0000","last_became_peered":"2024-04-26T23:05:32.757953+0000","last_unstale":"2024-04-26T23:05:32.761459+0000","last_undegraded":"2024-04-26T23:05:32.761459+0000","last_fullsized":"2024-04-26T23:05:32.761459+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:58.444855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771490+0000","last_change":"2024-04-26T23:05:32.767346+0000","last_active":"2024-04-26T23:05:32.771490+0000","last_peered":"2024-04-26T23:05:32.771490+0000","last_clean":"2024-04-26T23:05:32.771490+0000","last_became_active":"2024-04-26T23:05:32.766954+0000","last_became_peered":"2024-04-26T23:05:32.766954+0000","last_unstale":"2024-04-26T23:05:32.771490+0000","last_undegraded":"2024-04-26T23:05:32.771490+0000","last_fullsized":"2024-04-26T23:05:32.771490+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:42:49.983638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.769513+0000","last_change":"2024-04-26T23:05:32.766488+0000","last_active":"2024-04-26T23:05:32.769513+0000","last_peered":"2024-04-26T23:05:32.769513+0000","last_clean":"2024-04-26T23:05:32.769513+0000","last_became_active":"2024-04-26T23:05:32.764925+0000","last_became_peered":"2024-04-26T23:05:32.764925+0000","last_unstale":"2024-04-26T23:05:32.769513+0000","last_undegraded":"2024-04-26T23:05:32.769513+0000","last_fullsized":"2024-04-26T23:05:32.769513+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:44:44.767658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.756019+0000","last_change":"2024-04-26T23:05:32.751512+0000","last_active":"2024-04-26T23:05:32.756019+0000","last_peered":"2024-04-26T23:05:32.756019+0000","last_clean":"2024-04-26T23:05:32.756019+0000","last_became_active":"2024-04-26T23:05:32.750122+0000","last_became_peered":"2024-04-26T23:05:32.750122+0000","last_unstale":"2024-04-26T23:05:32.756019+0000","last_undegraded":"2024-04-26T23:05:32.756019+0000","last_fullsized":"2024-04-26T23:05:32.756019+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:35.358920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771737+0000","last_change":"2024-04-26T23:05:32.756778+0000","last_active":"2024-04-26T23:05:32.771737+0000","last_peered":"2024-04-26T23:05:32.771737+0000","last_clean":"2024-04-26T23:05:32.771737+0000","last_became_active":"2024-04-26T23:05:32.756219+0000","last_became_peered":"2024-04-26T23:05:32.756219+0000","last_unstale":"2024-04-26T23:05:32.771737+0000","last_undegraded":"2024-04-26T23:05:32.771737+0000","last_fullsized":"2024-04-26T23:05:32.771737+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:51.859455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.765990+0000","last_change":"2024-04-26T23:05:32.761026+0000","last_active":"2024-04-26T23:05:32.765990+0000","last_peered":"2024-04-26T23:05:32.765990+0000","last_clean":"2024-04-26T23:05:32.765990+0000","last_became_active":"2024-04-26T23:05:32.760738+0000","last_became_peered":"2024-04-26T23:05:32.760738+0000","last_unstale":"2024-04-26T23:05:32.765990+0000","last_undegraded":"2024-04-26T23:05:32.765990+0000","last_fullsized":"2024-04-26T23:05:32.765990+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:34:57.363296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"64'4","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.780727+0000","last_change":"2024-04-26T23:05:31.770725+0000","last_active":"2024-04-26T23:05:32.780727+0000","last_peered":"2024-04-26T23:05:32.780727+0000","last_clean":"2024-04-26T23:05:32.780727+0000","last_became_active":"2024-04-26T23:05:31.770414+0000","last_became_peered":"2024-04-26T23:05:31.770414+0000","last_unstale":"2024-04-26T23:05:32.780727+0000","last_undegraded":"2024-04-26T23:05:32.780727+0000","last_fullsized":"2024-04-26T23:05:32.780727+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:58:13.589814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.762649+0000","last_change":"2024-04-26T23:05:32.754181+0000","last_active":"2024-04-26T23:05:32.762649+0000","last_peered":"2024-04-26T23:05:32.762649+0000","last_clean":"2024-04-26T23:05:32.762649+0000","last_became_active":"2024-04-26T23:05:32.753872+0000","last_became_peered":"2024-04-26T23:05:32.753872+0000","last_unstale":"2024-04-26T23:05:32.762649+0000","last_undegraded":"2024-04-26T23:05:32.762649+0000","last_fullsized":"2024-04-26T23:05:32.762649+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:47:04.134977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"64'1","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.779837+0000","last_change":"2024-04-26T23:05:31.770026+0000","last_active":"2024-04-26T23:05:32.779837+0000","last_peered":"2024-04-26T23:05:32.779837+0000","last_clean":"2024-04-26T23:05:32.779837+0000","last_became_active":"2024-04-26T23:05:31.769545+0000","last_became_peered":"2024-04-26T23:05:31.769545+0000","last_unstale":"2024-04-26T23:05:32.779837+0000","last_undegraded":"2024-04-26T23:05:32.779837+0000","last_fullsized":"2024-04-26T23:05:32.779837+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:31.908410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.776323+0000","last_change":"2024-04-26T23:05:32.761309+0000","last_active":"2024-04-26T23:05:32.776323+0000","last_peered":"2024-04-26T23:05:32.776323+0000","last_clean":"2024-04-26T23:05:32.776323+0000","last_became_active":"2024-04-26T23:05:32.760980+0000","last_became_peered":"2024-04-26T23:05:32.760980+0000","last_unstale":"2024-04-26T23:05:32.776323+0000","last_undegraded":"2024-04-26T23:05:32.776323+0000","last_fullsized":"2024-04-26T23:05:32.776323+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:43:46.200740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"64'2","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.777711+0000","last_change":"2024-04-26T23:05:31.767871+0000","last_active":"2024-04-26T23:05:32.777711+0000","last_peered":"2024-04-26T23:05:32.777711+0000","last_clean":"2024-04-26T23:05:32.777711+0000","last_became_active":"2024-04-26T23:05:31.767184+0000","last_became_peered":"2024-04-26T23:05:31.767184+0000","last_unstale":"2024-04-26T23:05:32.777711+0000","last_undegraded":"2024-04-26T23:05:32.777711+0000","last_fullsized":"2024-04-26T23:05:32.777711+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:43:50.062402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.773482+0000","last_change":"2024-04-26T23:05:32.769325+0000","last_active":"2024-04-26T23:05:32.773482+0000","last_peered":"2024-04-26T23:05:32.773482+0000","last_clean":"2024-04-26T23:05:32.773482+0000","last_became_active":"2024-04-26T23:05:32.769062+0000","last_became_peered":"2024-04-26T23:05:32.769062+0000","last_unstale":"2024-04-26T23:05:32.773482+0000","last_undegraded":"2024-04-26T23:05:32.773482+0000","last_fullsized":"2024-04-26T23:05:32.773482+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:24:54.832654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"64'2","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.778964+0000","last_change":"2024-04-26T23:05:31.754696+0000","last_active":"2024-04-26T23:05:32.778964+0000","last_peered":"2024-04-26T23:05:32.778964+0000","last_clean":"2024-04-26T23:05:32.778964+0000","last_became_active":"2024-04-26T23:05:31.754353+0000","last_became_peered":"2024-04-26T23:05:31.754353+0000","last_unstale":"2024-04-26T23:05:32.778964+0000","last_undegraded":"2024-04-26T23:05:32.778964+0000","last_fullsized":"2024-04-26T23:05:32.778964+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:28.262197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.755471+0000","last_change":"2024-04-26T23:05:32.749775+0000","last_active":"2024-04-26T23:05:32.755471+0000","last_peered":"2024-04-26T23:05:32.755471+0000","last_clean":"2024-04-26T23:05:32.755471+0000","last_became_active":"2024-04-26T23:05:32.749407+0000","last_became_peered":"2024-04-26T23:05:32.749407+0000","last_unstale":"2024-04-26T23:05:32.755471+0000","last_undegraded":"2024-04-26T23:05:32.755471+0000","last_fullsized":"2024-04-26T23:05:32.755471+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:47.499283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"64'1","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.776633+0000","last_change":"2024-04-26T23:05:31.753361+0000","last_active":"2024-04-26T23:05:32.776633+0000","last_peered":"2024-04-26T23:05:32.776633+0000","last_clean":"2024-04-26T23:05:32.776633+0000","last_became_active":"2024-04-26T23:05:31.752976+0000","last_became_peered":"2024-04-26T23:05:31.752976+0000","last_unstale":"2024-04-26T23:05:32.776633+0000","last_undegraded":"2024-04-26T23:05:32.776633+0000","last_fullsized":"2024-04-26T23:05:32.776633+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:23:43.447217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759892+0000","last_change":"2024-04-26T23:05:32.750891+0000","last_active":"2024-04-26T23:05:32.759892+0000","last_peered":"2024-04-26T23:05:32.759892+0000","last_clean":"2024-04-26T23:05:32.759892+0000","last_became_active":"2024-04-26T23:05:32.750481+0000","last_became_peered":"2024-04-26T23:05:32.750481+0000","last_unstale":"2024-04-26T23:05:32.759892+0000","last_undegraded":"2024-04-26T23:05:32.759892+0000","last_fullsized":"2024-04-26T23:05:32.759892+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:21:38.429391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"64'1","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771888+0000","last_change":"2024-04-26T23:05:31.746607+0000","last_active":"2024-04-26T23:05:32.771888+0000","last_peered":"2024-04-26T23:05:32.771888+0000","last_clean":"2024-04-26T23:05:32.771888+0000","last_became_active":"2024-04-26T23:05:31.746253+0000","last_became_peered":"2024-04-26T23:05:31.746253+0000","last_unstale":"2024-04-26T23:05:32.771888+0000","last_undegraded":"2024-04-26T23:05:32.771888+0000","last_fullsized":"2024-04-26T23:05:32.771888+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:34.392268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759585+0000","last_change":"2024-04-26T23:05:32.753390+0000","last_active":"2024-04-26T23:05:32.759585+0000","last_peered":"2024-04-26T23:05:32.759585+0000","last_clean":"2024-04-26T23:05:32.759585+0000","last_became_active":"2024-04-26T23:05:32.752333+0000","last_became_peered":"2024-04-26T23:05:32.752333+0000","last_unstale":"2024-04-26T23:05:32.759585+0000","last_undegraded":"2024-04-26T23:05:32.759585+0000","last_fullsized":"2024-04-26T23:05:32.759585+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:21:24.525954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.725901+0000","last_change":"2024-04-26T23:05:31.760608+0000","last_active":"2024-04-26T23:05:32.725901+0000","last_peered":"2024-04-26T23:05:32.725901+0000","last_clean":"2024-04-26T23:05:32.725901+0000","last_became_active":"2024-04-26T23:05:31.759722+0000","last_became_peered":"2024-04-26T23:05:31.759722+0000","last_unstale":"2024-04-26T23:05:32.725901+0000","last_undegraded":"2024-04-26T23:05:32.725901+0000","last_fullsized":"2024-04-26T23:05:32.725901+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:13:10.834587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.753118+0000","last_change":"2024-04-26T23:05:32.747738+0000","last_active":"2024-04-26T23:05:32.753118+0000","last_peered":"2024-04-26T23:05:32.753118+0000","last_clean":"2024-04-26T23:05:32.753118+0000","last_became_active":"2024-04-26T23:05:32.747344+0000","last_became_peered":"2024-04-26T23:05:32.747344+0000","last_unstale":"2024-04-26T23:05:32.753118+0000","last_undegraded":"2024-04-26T23:05:32.753118+0000","last_fullsized":"2024-04-26T23:05:32.753118+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:19.465420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.731134+0000","last_change":"2024-04-26T23:05:31.768493+0000","last_active":"2024-04-26T23:05:32.731134+0000","last_peered":"2024-04-26T23:05:32.731134+0000","last_clean":"2024-04-26T23:05:32.731134+0000","last_became_active":"2024-04-26T23:05:31.767906+0000","last_became_peered":"2024-04-26T23:05:31.767906+0000","last_unstale":"2024-04-26T23:05:32.731134+0000","last_undegraded":"2024-04-26T23:05:32.731134+0000","last_fullsized":"2024-04-26T23:05:32.731134+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:20:04.698291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766361+0000","last_change":"2024-04-26T23:05:32.764563+0000","last_active":"2024-04-26T23:05:32.766361+0000","last_peered":"2024-04-26T23:05:32.766361+0000","last_clean":"2024-04-26T23:05:32.766361+0000","last_became_active":"2024-04-26T23:05:32.762251+0000","last_became_peered":"2024-04-26T23:05:32.762251+0000","last_unstale":"2024-04-26T23:05:32.766361+0000","last_undegraded":"2024-04-26T23:05:32.766361+0000","last_fullsized":"2024-04-26T23:05:32.766361+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:00.038837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.756401+0000","last_change":"2024-04-26T23:05:31.772660+0000","last_active":"2024-04-26T23:05:32.756401+0000","last_peered":"2024-04-26T23:05:32.756401+0000","last_clean":"2024-04-26T23:05:32.756401+0000","last_became_active":"2024-04-26T23:05:31.772338+0000","last_became_peered":"2024-04-26T23:05:31.772338+0000","last_unstale":"2024-04-26T23:05:32.756401+0000","last_undegraded":"2024-04-26T23:05:32.756401+0000","last_fullsized":"2024-04-26T23:05:32.756401+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:22.466819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766066+0000","last_change":"2024-04-26T23:05:32.757463+0000","last_active":"2024-04-26T23:05:32.766066+0000","last_peered":"2024-04-26T23:05:32.766066+0000","last_clean":"2024-04-26T23:05:32.766066+0000","last_became_active":"2024-04-26T23:05:32.757171+0000","last_became_peered":"2024-04-26T23:05:32.757171+0000","last_unstale":"2024-04-26T23:05:32.766066+0000","last_undegraded":"2024-04-26T23:05:32.766066+0000","last_fullsized":"2024-04-26T23:05:32.766066+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:44.040675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.735496+0000","last_change":"2024-04-26T23:05:31.757301+0000","last_active":"2024-04-26T23:05:32.735496+0000","last_peered":"2024-04-26T23:05:32.735496+0000","last_clean":"2024-04-26T23:05:32.735496+0000","last_became_active":"2024-04-26T23:05:31.755678+0000","last_became_peered":"2024-04-26T23:05:31.755678+0000","last_unstale":"2024-04-26T23:05:32.735496+0000","last_undegraded":"2024-04-26T23:05:32.735496+0000","last_fullsized":"2024-04-26T23:05:32.735496+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:20:04.698291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.762054+0000","last_change":"2024-04-26T23:05:32.755223+0000","last_active":"2024-04-26T23:05:32.762054+0000","last_peered":"2024-04-26T23:05:32.762054+0000","last_clean":"2024-04-26T23:05:32.762054+0000","last_became_active":"2024-04-26T23:05:32.754915+0000","last_became_peered":"2024-04-26T23:05:32.754915+0000","last_unstale":"2024-04-26T23:05:32.762054+0000","last_undegraded":"2024-04-26T23:05:32.762054+0000","last_fullsized":"2024-04-26T23:05:32.762054+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:01:32.493056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"64'3","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.781767+0000","last_change":"2024-04-26T23:05:31.772480+0000","last_active":"2024-04-26T23:05:32.781767+0000","last_peered":"2024-04-26T23:05:32.781767+0000","last_clean":"2024-04-26T23:05:32.781767+0000","last_became_active":"2024-04-26T23:05:31.772165+0000","last_became_peered":"2024-04-26T23:05:31.772165+0000","last_unstale":"2024-04-26T23:05:32.781767+0000","last_undegraded":"2024-04-26T23:05:32.781767+0000","last_fullsized":"2024-04-26T23:05:32.781767+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:30.272719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.761329+0000","last_change":"2024-04-26T23:05:32.755588+0000","last_active":"2024-04-26T23:05:32.761329+0000","last_peered":"2024-04-26T23:05:32.761329+0000","last_clean":"2024-04-26T23:05:32.761329+0000","last_became_active":"2024-04-26T23:05:32.755282+0000","last_became_peered":"2024-04-26T23:05:32.755282+0000","last_unstale":"2024-04-26T23:05:32.761329+0000","last_undegraded":"2024-04-26T23:05:32.761329+0000","last_fullsized":"2024-04-26T23:05:32.761329+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:01:32.493056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"64'6","reported_seq":18,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.782019+0000","last_change":"2024-04-26T23:05:31.760989+0000","last_active":"2024-04-26T23:05:32.782019+0000","last_peered":"2024-04-26T23:05:32.782019+0000","last_clean":"2024-04-26T23:05:32.782019+0000","last_became_active":"2024-04-26T23:05:31.760456+0000","last_became_peered":"2024-04-26T23:05:31.760456+0000","last_unstale":"2024-04-26T23:05:32.782019+0000","last_undegraded":"2024-04-26T23:05:32.782019+0000","last_fullsized":"2024-04-26T23:05:32.782019+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:46.525277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771143+0000","last_change":"2024-04-26T23:05:32.765693+0000","last_active":"2024-04-26T23:05:32.771143+0000","last_peered":"2024-04-26T23:05:32.771143+0000","last_clean":"2024-04-26T23:05:32.771143+0000","last_became_active":"2024-04-26T23:05:32.763803+0000","last_became_peered":"2024-04-26T23:05:32.763803+0000","last_unstale":"2024-04-26T23:05:32.771143+0000","last_undegraded":"2024-04-26T23:05:32.771143+0000","last_fullsized":"2024-04-26T23:05:32.771143+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:22.872508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"64'2","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.779891+0000","last_change":"2024-04-26T23:05:31.770239+0000","last_active":"2024-04-26T23:05:32.779891+0000","last_peered":"2024-04-26T23:05:32.779891+0000","last_clean":"2024-04-26T23:05:32.779891+0000","last_became_active":"2024-04-26T23:05:31.769941+0000","last_became_peered":"2024-04-26T23:05:31.769941+0000","last_unstale":"2024-04-26T23:05:32.779891+0000","last_undegraded":"2024-04-26T23:05:32.779891+0000","last_fullsized":"2024-04-26T23:05:32.779891+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:30.602553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.776599+0000","last_change":"2024-04-26T23:05:32.772869+0000","last_active":"2024-04-26T23:05:32.776599+0000","last_peered":"2024-04-26T23:05:32.776599+0000","last_clean":"2024-04-26T23:05:32.776599+0000","last_became_active":"2024-04-26T23:05:32.772567+0000","last_became_peered":"2024-04-26T23:05:32.772567+0000","last_unstale":"2024-04-26T23:05:32.776599+0000","last_undegraded":"2024-04-26T23:05:32.776599+0000","last_fullsized":"2024-04-26T23:05:32.776599+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:26.182654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.749657+0000","last_change":"2024-04-26T23:05:31.770279+0000","last_active":"2024-04-26T23:05:32.749657+0000","last_peered":"2024-04-26T23:05:32.749657+0000","last_clean":"2024-04-26T23:05:32.749657+0000","last_became_active":"2024-04-26T23:05:31.769923+0000","last_became_peered":"2024-04-26T23:05:31.769923+0000","last_unstale":"2024-04-26T23:05:32.749657+0000","last_undegraded":"2024-04-26T23:05:32.749657+0000","last_fullsized":"2024-04-26T23:05:32.749657+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:30.602553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.763602+0000","last_change":"2024-04-26T23:05:32.755633+0000","last_active":"2024-04-26T23:05:32.763602+0000","last_peered":"2024-04-26T23:05:32.763602+0000","last_clean":"2024-04-26T23:05:32.763602+0000","last_became_active":"2024-04-26T23:05:32.753211+0000","last_became_peered":"2024-04-26T23:05:32.753211+0000","last_unstale":"2024-04-26T23:05:32.763602+0000","last_undegraded":"2024-04-26T23:05:32.763602+0000","last_fullsized":"2024-04-26T23:05:32.763602+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:35:40.312530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.756535+0000","last_change":"2024-04-26T23:05:31.768537+0000","last_active":"2024-04-26T23:05:32.756535+0000","last_peered":"2024-04-26T23:05:32.756535+0000","last_clean":"2024-04-26T23:05:32.756535+0000","last_became_active":"2024-04-26T23:05:31.768175+0000","last_became_peered":"2024-04-26T23:05:31.768175+0000","last_unstale":"2024-04-26T23:05:32.756535+0000","last_undegraded":"2024-04-26T23:05:32.756535+0000","last_fullsized":"2024-04-26T23:05:32.756535+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:01:59.022114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.770370+0000","last_change":"2024-04-26T23:05:32.766588+0000","last_active":"2024-04-26T23:05:32.770370+0000","last_peered":"2024-04-26T23:05:32.770370+0000","last_clean":"2024-04-26T23:05:32.770370+0000","last_became_active":"2024-04-26T23:05:32.762965+0000","last_became_peered":"2024-04-26T23:05:32.762965+0000","last_unstale":"2024-04-26T23:05:32.770370+0000","last_undegraded":"2024-04-26T23:05:32.770370+0000","last_fullsized":"2024-04-26T23:05:32.770370+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:35.989406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"64'8","reported_seq":20,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.780778+0000","last_change":"2024-04-26T23:05:31.765009+0000","last_active":"2024-04-26T23:05:32.780778+0000","last_peered":"2024-04-26T23:05:32.780778+0000","last_clean":"2024-04-26T23:05:32.780778+0000","last_became_active":"2024-04-26T23:05:31.764477+0000","last_became_peered":"2024-04-26T23:05:31.764477+0000","last_unstale":"2024-04-26T23:05:32.780778+0000","last_undegraded":"2024-04-26T23:05:32.780778+0000","last_fullsized":"2024-04-26T23:05:32.780778+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:28:04.128466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.777622+0000","last_change":"2024-04-26T23:05:32.774045+0000","last_active":"2024-04-26T23:05:32.777622+0000","last_peered":"2024-04-26T23:05:32.777622+0000","last_clean":"2024-04-26T23:05:32.777622+0000","last_became_active":"2024-04-26T23:05:32.772012+0000","last_became_peered":"2024-04-26T23:05:32.772012+0000","last_unstale":"2024-04-26T23:05:32.777622+0000","last_undegraded":"2024-04-26T23:05:32.777622+0000","last_fullsized":"2024-04-26T23:05:32.777622+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:26.182654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"64'1","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.777986+0000","last_change":"2024-04-26T23:05:31.769936+0000","last_active":"2024-04-26T23:05:32.777986+0000","last_peered":"2024-04-26T23:05:32.777986+0000","last_clean":"2024-04-26T23:05:32.777986+0000","last_became_active":"2024-04-26T23:05:31.767600+0000","last_became_peered":"2024-04-26T23:05:31.767600+0000","last_unstale":"2024-04-26T23:05:32.777986+0000","last_undegraded":"2024-04-26T23:05:32.777986+0000","last_fullsized":"2024-04-26T23:05:32.777986+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:34:08.035750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.758574+0000","last_change":"2024-04-26T23:05:32.751224+0000","last_active":"2024-04-26T23:05:32.758574+0000","last_peered":"2024-04-26T23:05:32.758574+0000","last_clean":"2024-04-26T23:05:32.758574+0000","last_became_active":"2024-04-26T23:05:32.750525+0000","last_became_peered":"2024-04-26T23:05:32.750525+0000","last_unstale":"2024-04-26T23:05:32.758574+0000","last_undegraded":"2024-04-26T23:05:32.758574+0000","last_fullsized":"2024-04-26T23:05:32.758574+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:35.358920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"64'2","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.778479+0000","last_change":"2024-04-26T23:05:31.769437+0000","last_active":"2024-04-26T23:05:32.778479+0000","last_peered":"2024-04-26T23:05:32.778479+0000","last_clean":"2024-04-26T23:05:32.778479+0000","last_became_active":"2024-04-26T23:05:31.768705+0000","last_became_peered":"2024-04-26T23:05:31.768705+0000","last_unstale":"2024-04-26T23:05:32.778479+0000","last_undegraded":"2024-04-26T23:05:32.778479+0000","last_fullsized":"2024-04-26T23:05:32.778479+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:48:09.258298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.764533+0000","last_change":"2024-04-26T23:05:32.761715+0000","last_active":"2024-04-26T23:05:32.764533+0000","last_peered":"2024-04-26T23:05:32.764533+0000","last_clean":"2024-04-26T23:05:32.764533+0000","last_became_active":"2024-04-26T23:05:32.758821+0000","last_became_peered":"2024-04-26T23:05:32.758821+0000","last_unstale":"2024-04-26T23:05:32.764533+0000","last_undegraded":"2024-04-26T23:05:32.764533+0000","last_fullsized":"2024-04-26T23:05:32.764533+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:23:41.516794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"64'1","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.781798+0000","last_change":"2024-04-26T23:05:31.769171+0000","last_active":"2024-04-26T23:05:32.781798+0000","last_peered":"2024-04-26T23:05:32.781798+0000","last_clean":"2024-04-26T23:05:32.781798+0000","last_became_active":"2024-04-26T23:05:31.768464+0000","last_became_peered":"2024-04-26T23:05:31.768464+0000","last_unstale":"2024-04-26T23:05:32.781798+0000","last_undegraded":"2024-04-26T23:05:32.781798+0000","last_fullsized":"2024-04-26T23:05:32.781798+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:57.428132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.762470+0000","last_change":"2024-04-26T23:05:32.757835+0000","last_active":"2024-04-26T23:05:32.762470+0000","last_peered":"2024-04-26T23:05:32.762470+0000","last_clean":"2024-04-26T23:05:32.762470+0000","last_became_active":"2024-04-26T23:05:32.756941+0000","last_became_peered":"2024-04-26T23:05:32.756941+0000","last_unstale":"2024-04-26T23:05:32.762470+0000","last_undegraded":"2024-04-26T23:05:32.762470+0000","last_fullsized":"2024-04-26T23:05:32.762470+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:48:12.566024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.726739+0000","last_change":"2024-04-26T23:05:31.761468+0000","last_active":"2024-04-26T23:05:32.726739+0000","last_peered":"2024-04-26T23:05:32.726739+0000","last_clean":"2024-04-26T23:05:32.726739+0000","last_became_active":"2024-04-26T23:05:31.761152+0000","last_became_peered":"2024-04-26T23:05:31.761152+0000","last_unstale":"2024-04-26T23:05:32.726739+0000","last_undegraded":"2024-04-26T23:05:32.726739+0000","last_fullsized":"2024-04-26T23:05:32.726739+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:37:26.663900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771692+0000","last_change":"2024-04-26T23:05:32.756293+0000","last_active":"2024-04-26T23:05:32.771692+0000","last_peered":"2024-04-26T23:05:32.771692+0000","last_clean":"2024-04-26T23:05:32.771692+0000","last_became_active":"2024-04-26T23:05:32.755936+0000","last_became_peered":"2024-04-26T23:05:32.755936+0000","last_unstale":"2024-04-26T23:05:32.771692+0000","last_undegraded":"2024-04-26T23:05:32.771692+0000","last_fullsized":"2024-04-26T23:05:32.771692+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:05.347403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"64'2","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.778918+0000","last_change":"2024-04-26T23:05:31.763776+0000","last_active":"2024-04-26T23:05:32.778918+0000","last_peered":"2024-04-26T23:05:32.778918+0000","last_clean":"2024-04-26T23:05:32.778918+0000","last_became_active":"2024-04-26T23:05:31.763362+0000","last_became_peered":"2024-04-26T23:05:31.763362+0000","last_unstale":"2024-04-26T23:05:32.778918+0000","last_undegraded":"2024-04-26T23:05:32.778918+0000","last_fullsized":"2024-04-26T23:05:32.778918+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:32:41.819376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766276+0000","last_change":"2024-04-26T23:05:32.761748+0000","last_active":"2024-04-26T23:05:32.766276+0000","last_peered":"2024-04-26T23:05:32.766276+0000","last_clean":"2024-04-26T23:05:32.766276+0000","last_became_active":"2024-04-26T23:05:32.758119+0000","last_became_peered":"2024-04-26T23:05:32.758119+0000","last_unstale":"2024-04-26T23:05:32.766276+0000","last_undegraded":"2024-04-26T23:05:32.766276+0000","last_fullsized":"2024-04-26T23:05:32.766276+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:31.289442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.749828+0000","last_change":"2024-04-26T23:05:31.769307+0000","last_active":"2024-04-26T23:05:32.749828+0000","last_peered":"2024-04-26T23:05:32.749828+0000","last_clean":"2024-04-26T23:05:32.749828+0000","last_became_active":"2024-04-26T23:05:31.769034+0000","last_became_peered":"2024-04-26T23:05:31.769034+0000","last_unstale":"2024-04-26T23:05:32.749828+0000","last_undegraded":"2024-04-26T23:05:32.749828+0000","last_fullsized":"2024-04-26T23:05:32.749828+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:21:37.412668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759564+0000","last_change":"2024-04-26T23:05:32.752754+0000","last_active":"2024-04-26T23:05:32.759564+0000","last_peered":"2024-04-26T23:05:32.759564+0000","last_clean":"2024-04-26T23:05:32.759564+0000","last_became_active":"2024-04-26T23:05:32.751685+0000","last_became_peered":"2024-04-26T23:05:32.751685+0000","last_unstale":"2024-04-26T23:05:32.759564+0000","last_undegraded":"2024-04-26T23:05:32.759564+0000","last_fullsized":"2024-04-26T23:05:32.759564+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:14.712885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.745528+0000","last_change":"2024-04-26T23:05:31.769617+0000","last_active":"2024-04-26T23:05:32.745528+0000","last_peered":"2024-04-26T23:05:32.745528+0000","last_clean":"2024-04-26T23:05:32.745528+0000","last_became_active":"2024-04-26T23:05:31.769190+0000","last_became_peered":"2024-04-26T23:05:31.769190+0000","last_unstale":"2024-04-26T23:05:32.745528+0000","last_undegraded":"2024-04-26T23:05:32.745528+0000","last_fullsized":"2024-04-26T23:05:32.745528+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:55:07.062224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771943+0000","last_change":"2024-04-26T23:05:32.757470+0000","last_active":"2024-04-26T23:05:32.771943+0000","last_peered":"2024-04-26T23:05:32.771943+0000","last_clean":"2024-04-26T23:05:32.771943+0000","last_became_active":"2024-04-26T23:05:32.757141+0000","last_became_peered":"2024-04-26T23:05:32.757141+0000","last_unstale":"2024-04-26T23:05:32.771943+0000","last_undegraded":"2024-04-26T23:05:32.771943+0000","last_fullsized":"2024-04-26T23:05:32.771943+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:18:44.249170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"64'2","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.780252+0000","last_change":"2024-04-26T23:05:31.771257+0000","last_active":"2024-04-26T23:05:32.780252+0000","last_peered":"2024-04-26T23:05:32.780252+0000","last_clean":"2024-04-26T23:05:32.780252+0000","last_became_active":"2024-04-26T23:05:31.770330+0000","last_became_peered":"2024-04-26T23:05:31.770330+0000","last_unstale":"2024-04-26T23:05:32.780252+0000","last_undegraded":"2024-04-26T23:05:32.780252+0000","last_fullsized":"2024-04-26T23:05:32.780252+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:46.525277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759360+0000","last_change":"2024-04-26T23:05:32.756115+0000","last_active":"2024-04-26T23:05:32.759360+0000","last_peered":"2024-04-26T23:05:32.759360+0000","last_clean":"2024-04-26T23:05:32.759360+0000","last_became_active":"2024-04-26T23:05:32.755800+0000","last_became_peered":"2024-04-26T23:05:32.755800+0000","last_unstale":"2024-04-26T23:05:32.759360+0000","last_undegraded":"2024-04-26T23:05:32.759360+0000","last_fullsized":"2024-04-26T23:05:32.759360+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:54:14.272684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.750555+0000","last_change":"2024-04-26T23:05:31.767074+0000","last_active":"2024-04-26T23:05:32.750555+0000","last_peered":"2024-04-26T23:05:32.750555+0000","last_clean":"2024-04-26T23:05:32.750555+0000","last_became_active":"2024-04-26T23:05:31.765961+0000","last_became_peered":"2024-04-26T23:05:31.765961+0000","last_unstale":"2024-04-26T23:05:32.750555+0000","last_undegraded":"2024-04-26T23:05:32.750555+0000","last_fullsized":"2024-04-26T23:05:32.750555+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:57.428132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.776777+0000","last_change":"2024-04-26T23:05:32.774635+0000","last_active":"2024-04-26T23:05:32.776777+0000","last_peered":"2024-04-26T23:05:32.776777+0000","last_clean":"2024-04-26T23:05:32.776777+0000","last_became_active":"2024-04-26T23:05:32.772832+0000","last_became_peered":"2024-04-26T23:05:32.772832+0000","last_unstale":"2024-04-26T23:05:32.776777+0000","last_undegraded":"2024-04-26T23:05:32.776777+0000","last_fullsized":"2024-04-26T23:05:32.776777+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:53:00.646163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.749182+0000","last_change":"2024-04-26T23:05:31.766535+0000","last_active":"2024-04-26T23:05:32.749182+0000","last_peered":"2024-04-26T23:05:32.749182+0000","last_clean":"2024-04-26T23:05:32.749182+0000","last_became_active":"2024-04-26T23:05:31.766192+0000","last_became_peered":"2024-04-26T23:05:31.766192+0000","last_unstale":"2024-04-26T23:05:32.749182+0000","last_undegraded":"2024-04-26T23:05:32.749182+0000","last_fullsized":"2024-04-26T23:05:32.749182+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:48:34.632624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.765599+0000","last_change":"2024-04-26T23:05:32.759298+0000","last_active":"2024-04-26T23:05:32.765599+0000","last_peered":"2024-04-26T23:05:32.765599+0000","last_clean":"2024-04-26T23:05:32.765599+0000","last_became_active":"2024-04-26T23:05:32.758971+0000","last_became_peered":"2024-04-26T23:05:32.758971+0000","last_unstale":"2024-04-26T23:05:32.765599+0000","last_undegraded":"2024-04-26T23:05:32.765599+0000","last_fullsized":"2024-04-26T23:05:32.765599+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:23:41.516794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.743548+0000","last_change":"2024-04-26T23:05:31.756036+0000","last_active":"2024-04-26T23:05:32.743548+0000","last_peered":"2024-04-26T23:05:32.743548+0000","last_clean":"2024-04-26T23:05:32.743548+0000","last_became_active":"2024-04-26T23:05:31.755674+0000","last_became_peered":"2024-04-26T23:05:31.755674+0000","last_unstale":"2024-04-26T23:05:32.743548+0000","last_undegraded":"2024-04-26T23:05:32.743548+0000","last_fullsized":"2024-04-26T23:05:32.743548+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:34:08.035750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.767384+0000","last_change":"2024-04-26T23:05:32.764451+0000","last_active":"2024-04-26T23:05:32.767384+0000","last_peered":"2024-04-26T23:05:32.767384+0000","last_clean":"2024-04-26T23:05:32.767384+0000","last_became_active":"2024-04-26T23:05:32.763008+0000","last_became_peered":"2024-04-26T23:05:32.763008+0000","last_unstale":"2024-04-26T23:05:32.767384+0000","last_undegraded":"2024-04-26T23:05:32.767384+0000","last_fullsized":"2024-04-26T23:05:32.767384+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:58:14.606536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.748146+0000","last_change":"2024-04-26T23:05:31.768363+0000","last_active":"2024-04-26T23:05:32.748146+0000","last_peered":"2024-04-26T23:05:32.748146+0000","last_clean":"2024-04-26T23:05:32.748146+0000","last_became_active":"2024-04-26T23:05:31.768061+0000","last_became_peered":"2024-04-26T23:05:31.768061+0000","last_unstale":"2024-04-26T23:05:32.748146+0000","last_undegraded":"2024-04-26T23:05:32.748146+0000","last_fullsized":"2024-04-26T23:05:32.748146+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:21:37.412668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771542+0000","last_change":"2024-04-26T23:05:32.768067+0000","last_active":"2024-04-26T23:05:32.771542+0000","last_peered":"2024-04-26T23:05:32.771542+0000","last_clean":"2024-04-26T23:05:32.771542+0000","last_became_active":"2024-04-26T23:05:32.767731+0000","last_became_peered":"2024-04-26T23:05:32.767731+0000","last_unstale":"2024-04-26T23:05:32.771542+0000","last_undegraded":"2024-04-26T23:05:32.771542+0000","last_fullsized":"2024-04-26T23:05:32.771542+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:34:57.363296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.726001+0000","last_change":"2024-04-26T23:05:31.749289+0000","last_active":"2024-04-26T23:05:32.726001+0000","last_peered":"2024-04-26T23:05:32.726001+0000","last_clean":"2024-04-26T23:05:32.726001+0000","last_became_active":"2024-04-26T23:05:31.748930+0000","last_became_peered":"2024-04-26T23:05:31.748930+0000","last_unstale":"2024-04-26T23:05:32.726001+0000","last_undegraded":"2024-04-26T23:05:32.726001+0000","last_fullsized":"2024-04-26T23:05:32.726001+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:43.540669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759808+0000","last_change":"2024-04-26T23:05:32.750715+0000","last_active":"2024-04-26T23:05:32.759808+0000","last_peered":"2024-04-26T23:05:32.759808+0000","last_clean":"2024-04-26T23:05:32.759808+0000","last_became_active":"2024-04-26T23:05:32.750391+0000","last_became_peered":"2024-04-26T23:05:32.750391+0000","last_unstale":"2024-04-26T23:05:32.759808+0000","last_undegraded":"2024-04-26T23:05:32.759808+0000","last_fullsized":"2024-04-26T23:05:32.759808+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:31.619276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.726431+0000","last_change":"2024-04-26T23:05:31.755397+0000","last_active":"2024-04-26T23:05:32.726431+0000","last_peered":"2024-04-26T23:05:32.726431+0000","last_clean":"2024-04-26T23:05:32.726431+0000","last_became_active":"2024-04-26T23:05:31.755053+0000","last_became_peered":"2024-04-26T23:05:31.755053+0000","last_unstale":"2024-04-26T23:05:32.726431+0000","last_undegraded":"2024-04-26T23:05:32.726431+0000","last_fullsized":"2024-04-26T23:05:32.726431+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:11.732013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759242+0000","last_change":"2024-04-26T23:05:32.754623+0000","last_active":"2024-04-26T23:05:32.759242+0000","last_peered":"2024-04-26T23:05:32.759242+0000","last_clean":"2024-04-26T23:05:32.759242+0000","last_became_active":"2024-04-26T23:05:32.754292+0000","last_became_peered":"2024-04-26T23:05:32.754292+0000","last_unstale":"2024-04-26T23:05:32.759242+0000","last_undegraded":"2024-04-26T23:05:32.759242+0000","last_fullsized":"2024-04-26T23:05:32.759242+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:40.736465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.738119+0000","last_change":"2024-04-26T23:05:31.762384+0000","last_active":"2024-04-26T23:05:32.738119+0000","last_peered":"2024-04-26T23:05:32.738119+0000","last_clean":"2024-04-26T23:05:32.738119+0000","last_became_active":"2024-04-26T23:05:31.762087+0000","last_became_peered":"2024-04-26T23:05:31.762087+0000","last_unstale":"2024-04-26T23:05:32.738119+0000","last_undegraded":"2024-04-26T23:05:32.738119+0000","last_fullsized":"2024-04-26T23:05:32.738119+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:34:56.346573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766318+0000","last_change":"2024-04-26T23:05:32.759868+0000","last_active":"2024-04-26T23:05:32.766318+0000","last_peered":"2024-04-26T23:05:32.766318+0000","last_clean":"2024-04-26T23:05:32.766318+0000","last_became_active":"2024-04-26T23:05:32.759566+0000","last_became_peered":"2024-04-26T23:05:32.759566+0000","last_unstale":"2024-04-26T23:05:32.766318+0000","last_undegraded":"2024-04-26T23:05:32.766318+0000","last_fullsized":"2024-04-26T23:05:32.766318+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:22.872508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.725362+0000","last_change":"2024-04-26T23:05:31.749603+0000","last_active":"2024-04-26T23:05:32.725362+0000","last_peered":"2024-04-26T23:05:32.725362+0000","last_clean":"2024-04-26T23:05:32.725362+0000","last_became_active":"2024-04-26T23:05:31.749283+0000","last_became_peered":"2024-04-26T23:05:31.749283+0000","last_unstale":"2024-04-26T23:05:32.725362+0000","last_undegraded":"2024-04-26T23:05:32.725362+0000","last_fullsized":"2024-04-26T23:05:32.725362+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:58:00.646819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":254,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.750023+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:05:32.750023+0000","last_peered":"2024-04-26T23:05:32.750023+0000","last_clean":"2024-04-26T23:05:32.750023+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:05:32.750023+0000","last_undegraded":"2024-04-26T23:05:32.750023+0000","last_fullsized":"2024-04-26T23:05:32.750023+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":936396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":774144,"data_stored":353195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":778240,"data_stored":353217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":354807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":774144,"data_stored":353195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31928,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":761856,"data_stored":346028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":774144,"data_stored":353195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":936396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:05:37.212 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:05:37.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.455+0000 7f3b6d800640 1 Processor -- start 2024-04-26T23:05:37.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.456+0000 7f3b6d800640 1 -- start start 2024-04-26T23:05:37.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.457+0000 7f3b6d800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3b68073b70 0x7f3b680aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:37.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.457+0000 7f3b6d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b680cbb50 0x7f3b680af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:37.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.457+0000 7f3b6d800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b68072c00 con 0x7f3b6815f380 2024-04-26T23:05:37.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.457+0000 7f3b6d800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3b680729b0 con 0x7f3b68073b70 2024-04-26T23:05:37.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.457+0000 7f3b6d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3b68072ee0 con 0x7f3b680cbb50 2024-04-26T23:05:37.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.458+0000 7f3b66000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3b68073b70 0x7f3b680aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:37.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.458+0000 7f3b67400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b680cbb50 0x7f3b680af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:37.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.458+0000 7f3b66000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3b68073b70 0x7f3b680aecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:58976/0 (socket says 172.21.15.40:58976) 2024-04-26T23:05:37.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.458+0000 7f3b66000640 1 -- 172.21.15.40:0/2887413651 learned_addr learned my addr 172.21.15.40:0/2887413651 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:37.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.458+0000 7f3b66000640 1 -- 172.21.15.40:0/2887413651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b680cbb50 msgr2=0x7f3b680af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:37.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.458+0000 7f3b66000640 1 --2- 172.21.15.40:0/2887413651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b680cbb50 0x7f3b680af1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.458+0000 7f3b66000640 1 -- 172.21.15.40:0/2887413651 >> v1:172.21.15.103:6789/0 conn(0x7f3b6815f380 legacy=0x7f3b680cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:37.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.458+0000 7f3b66000640 1 -- 172.21.15.40:0/2887413651 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b680731c0 con 0x7f3b68073b70 2024-04-26T23:05:37.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.460+0000 7f3b66000640 1 --2- 172.21.15.40:0/2887413651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b68073b70 0x7f3b680aecb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3b50009870 tx=0x7f3b5002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=510d100d9cfee623 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:37.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.460+0000 7f3b65600640 1 -- 172.21.15.40:0/2887413651 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b50037070 con 0x7f3b68073b70 2024-04-26T23:05:37.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.460+0000 7f3b65600640 1 -- 172.21.15.40:0/2887413651 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b50032440 con 0x7f3b68073b70 2024-04-26T23:05:37.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.460+0000 7f3b65600640 1 -- 172.21.15.40:0/2887413651 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b50032810 con 0x7f3b68073b70 2024-04-26T23:05:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.461+0000 7f3b6d800640 1 -- 172.21.15.40:0/2887413651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b68073b70 msgr2=0x7f3b680aecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.461+0000 7f3b6d800640 1 --2- 172.21.15.40:0/2887413651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b68073b70 0x7f3b680aecb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3b50009870 tx=0x7f3b5002a270 comp rx=0 tx=0).stop 2024-04-26T23:05:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.461+0000 7f3b6d800640 1 -- 172.21.15.40:0/2887413651 shutdown_connections 2024-04-26T23:05:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.461+0000 7f3b6d800640 1 --2- 172.21.15.40:0/2887413651 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b680cbb50 0x7f3b680af1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.461+0000 7f3b6d800640 1 --2- 172.21.15.40:0/2887413651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b68073b70 0x7f3b680aecb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.461+0000 7f3b6d800640 1 -- 172.21.15.40:0/2887413651 >> 172.21.15.40:0/2887413651 conn(0x7f3b680d0be0 msgr2=0x7f3b680d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.461+0000 7f3b6d800640 1 -- 172.21.15.40:0/2887413651 shutdown_connections 2024-04-26T23:05:37.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.461+0000 7f3b6d800640 1 -- 172.21.15.40:0/2887413651 wait complete. 2024-04-26T23:05:37.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.463+0000 7f3b6d800640 1 Processor -- start 2024-04-26T23:05:37.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.463+0000 7f3b6d800640 1 -- start start 2024-04-26T23:05:37.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.464+0000 7f3b6d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b680cbb50 0x7f3b6814ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:37.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.464+0000 7f3b6d800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b6815f380 0x7f3b680ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:37.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.464+0000 7f3b6d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b680c4710 0x7f3b68151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:37.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.464+0000 7f3b6d800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3b68073620 con 0x7f3b680c4710 2024-04-26T23:05:37.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.464+0000 7f3b6d800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3b680d08b0 con 0x7f3b6815f380 2024-04-26T23:05:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.464+0000 7f3b6d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3b680737a0 con 0x7f3b680cbb50 2024-04-26T23:05:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.464+0000 7f3b67400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b680c4710 0x7f3b68151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.464+0000 7f3b67400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b680c4710 0x7f3b68151460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46248/0 (socket says 172.21.15.40:46248) 2024-04-26T23:05:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.464+0000 7f3b67400640 1 -- 172.21.15.40:0/3785012268 learned_addr learned my addr 172.21.15.40:0/3785012268 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b66000640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b6815f380 0x7f3b680ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b66a00640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b680cbb50 0x7f3b6814ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b67400640 1 -- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b680cbb50 msgr2=0x7f3b6814ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b67400640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b680cbb50 0x7f3b6814ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b67400640 1 -- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b6815f380 msgr2=0x7f3b680ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b67400640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b6815f380 0x7f3b680ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b67400640 1 -- 172.21.15.40:0/3785012268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b6814f150 con 0x7f3b680c4710 2024-04-26T23:05:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b66a00640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b680cbb50 0x7f3b6814ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b66000640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b6815f380 0x7f3b680ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b67400640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b680c4710 0x7f3b68151460 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f3b6000efc0 tx=0x7f3b6000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b5ea00640 1 -- 172.21.15.40:0/3785012268 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b60018020 con 0x7f3b680c4710 2024-04-26T23:05:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b5ea00640 1 -- 172.21.15.40:0/3785012268 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b60014480 con 0x7f3b680c4710 2024-04-26T23:05:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b6d800640 1 -- 172.21.15.40:0/3785012268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b680c9610 con 0x7f3b680c4710 2024-04-26T23:05:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.465+0000 7f3b5ea00640 1 -- 172.21.15.40:0/3785012268 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b60014870 con 0x7f3b680c4710 2024-04-26T23:05:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.466+0000 7f3b6d800640 1 -- 172.21.15.40:0/3785012268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b680c7840 con 0x7f3b680c4710 2024-04-26T23:05:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.467+0000 7f3b6d800640 1 -- 172.21.15.40:0/3785012268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b680c7a80 con 0x7f3b680c4710 2024-04-26T23:05:37.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.468+0000 7f3b5ea00640 1 -- 172.21.15.40:0/3785012268 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3b60005d10 con 0x7f3b680c4710 2024-04-26T23:05:37.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.469+0000 7f3b5ea00640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3b4406c8f0 0x7f3b4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:37.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.469+0000 7f3b5ea00640 1 -- 172.21.15.40:0/3785012268 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f3b6008ca90 con 0x7f3b680c4710 2024-04-26T23:05:37.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.469+0000 7f3b66a00640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3b4406c8f0 0x7f3b4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:37.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.470+0000 7f3b66a00640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3b4406c8f0 0x7f3b4406edb0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3b580015e0 tx=0x7f3b58009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:37.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.473+0000 7f3b5ea00640 1 -- 172.21.15.40:0/3785012268 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3b680c7a80 con 0x7f3b680c4710 2024-04-26T23:05:37.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.671+0000 7f3b6d800640 1 -- 172.21.15.40:0/3785012268 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3b680694f0 con 0x7f3b4406c8f0 2024-04-26T23:05:37.675 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:05:37.675+0000 7fa16ce00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T23:05:37.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.691+0000 7f3b5ea00640 1 -- 172.21.15.40:0/3785012268 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251949 (secure 0 0 0) 0x7f3b680694f0 con 0x7f3b4406c8f0 2024-04-26T23:05:37.694 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:37.696 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:05:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.698+0000 7f3b6d800640 1 -- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3b4406c8f0 msgr2=0x7f3b4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.698+0000 7f3b6d800640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3b4406c8f0 0x7f3b4406edb0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3b580015e0 tx=0x7f3b58009040 comp rx=0 tx=0).stop 2024-04-26T23:05:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.698+0000 7f3b6d800640 1 -- 172.21.15.40:0/3785012268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b680c4710 msgr2=0x7f3b68151460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.698+0000 7f3b6d800640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b680c4710 0x7f3b68151460 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f3b6000efc0 tx=0x7f3b6000c490 comp rx=0 tx=0).stop 2024-04-26T23:05:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.699+0000 7f3b6d800640 1 -- 172.21.15.40:0/3785012268 shutdown_connections 2024-04-26T23:05:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.699+0000 7f3b6d800640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3b4406c8f0 0x7f3b4406edb0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.699+0000 7f3b6d800640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b680c4710 0x7f3b68151460 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.699+0000 7f3b6d800640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b6815f380 0x7f3b680ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.699+0000 7f3b6d800640 1 --2- 172.21.15.40:0/3785012268 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b680cbb50 0x7f3b6814ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.699+0000 7f3b6d800640 1 -- 172.21.15.40:0/3785012268 >> 172.21.15.40:0/3785012268 conn(0x7f3b680d0be0 msgr2=0x7f3b680aecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.699+0000 7f3b6d800640 1 -- 172.21.15.40:0/3785012268 shutdown_connections 2024-04-26T23:05:37.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.699+0000 7f3b6d800640 1 -- 172.21.15.40:0/3785012268 wait complete. 2024-04-26T23:05:37.721 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-26T23:05:37.243403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240012,"kb_used_data":7188,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734708,"statfs":{"total":773094113280,"available":772848340992,"internally_reserved":0,"allocated":7360512,"data_stored":3986429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001892"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766926+0000","last_change":"2024-04-26T23:05:32.764307+0000","last_active":"2024-04-26T23:05:32.766926+0000","last_peered":"2024-04-26T23:05:32.766926+0000","last_clean":"2024-04-26T23:05:32.766926+0000","last_became_active":"2024-04-26T23:05:32.763424+0000","last_became_peered":"2024-04-26T23:05:32.763424+0000","last_unstale":"2024-04-26T23:05:32.766926+0000","last_undegraded":"2024-04-26T23:05:32.766926+0000","last_fullsized":"2024-04-26T23:05:32.766926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:23.483542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766379+0000","last_change":"2024-04-26T23:05:32.762530+0000","last_active":"2024-04-26T23:05:32.766379+0000","last_peered":"2024-04-26T23:05:32.766379+0000","last_clean":"2024-04-26T23:05:32.766379+0000","last_became_active":"2024-04-26T23:05:32.762234+0000","last_became_peered":"2024-04-26T23:05:32.762234+0000","last_unstale":"2024-04-26T23:05:32.766379+0000","last_undegraded":"2024-04-26T23:05:32.766379+0000","last_fullsized":"2024-04-26T23:05:32.766379+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:48:10.275021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.780372+0000","last_change":"2024-04-26T23:05:32.778787+0000","last_active":"2024-04-26T23:05:32.780372+0000","last_peered":"2024-04-26T23:05:32.780372+0000","last_clean":"2024-04-26T23:05:32.780372+0000","last_became_active":"2024-04-26T23:05:32.778486+0000","last_became_peered":"2024-04-26T23:05:32.778486+0000","last_unstale":"2024-04-26T23:05:32.780372+0000","last_undegraded":"2024-04-26T23:05:32.780372+0000","last_fullsized":"2024-04-26T23:05:32.780372+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:29.278920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.764758+0000","last_change":"2024-04-26T23:05:32.753307+0000","last_active":"2024-04-26T23:05:32.764758+0000","last_peered":"2024-04-26T23:05:32.764758+0000","last_clean":"2024-04-26T23:05:32.764758+0000","last_became_active":"2024-04-26T23:05:32.753017+0000","last_became_peered":"2024-04-26T23:05:32.753017+0000","last_unstale":"2024-04-26T23:05:32.764758+0000","last_undegraded":"2024-04-26T23:05:32.764758+0000","last_fullsized":"2024-04-26T23:05:32.764758+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:21:24.525954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759953+0000","last_change":"2024-04-26T23:05:32.746080+0000","last_active":"2024-04-26T23:05:32.759953+0000","last_peered":"2024-04-26T23:05:32.759953+0000","last_clean":"2024-04-26T23:05:32.759953+0000","last_became_active":"2024-04-26T23:05:32.745767+0000","last_became_peered":"2024-04-26T23:05:32.745767+0000","last_unstale":"2024-04-26T23:05:32.759953+0000","last_undegraded":"2024-04-26T23:05:32.759953+0000","last_fullsized":"2024-04-26T23:05:32.759953+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:28:05.145189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.776305+0000","last_change":"2024-04-26T23:05:32.767886+0000","last_active":"2024-04-26T23:05:32.776305+0000","last_peered":"2024-04-26T23:05:32.776305+0000","last_clean":"2024-04-26T23:05:32.776305+0000","last_became_active":"2024-04-26T23:05:32.766263+0000","last_became_peered":"2024-04-26T23:05:32.766263+0000","last_unstale":"2024-04-26T23:05:32.776305+0000","last_undegraded":"2024-04-26T23:05:32.776305+0000","last_fullsized":"2024-04-26T23:05:32.776305+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:58:01.663542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771035+0000","last_change":"2024-04-26T23:05:32.764840+0000","last_active":"2024-04-26T23:05:32.771035+0000","last_peered":"2024-04-26T23:05:32.771035+0000","last_clean":"2024-04-26T23:05:32.771035+0000","last_became_active":"2024-04-26T23:05:32.764474+0000","last_became_peered":"2024-04-26T23:05:32.764474+0000","last_unstale":"2024-04-26T23:05:32.771035+0000","last_undegraded":"2024-04-26T23:05:32.771035+0000","last_fullsized":"2024-04-26T23:05:32.771035+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:52:16.509999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.769295+0000","last_change":"2024-04-26T23:05:32.765699+0000","last_active":"2024-04-26T23:05:32.769295+0000","last_peered":"2024-04-26T23:05:32.769295+0000","last_clean":"2024-04-26T23:05:32.769295+0000","last_became_active":"2024-04-26T23:05:32.765385+0000","last_became_peered":"2024-04-26T23:05:32.765385+0000","last_unstale":"2024-04-26T23:05:32.769295+0000","last_undegraded":"2024-04-26T23:05:32.769295+0000","last_fullsized":"2024-04-26T23:05:32.769295+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:42:49.983638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.770590+0000","last_change":"2024-04-26T23:05:32.767125+0000","last_active":"2024-04-26T23:05:32.770590+0000","last_peered":"2024-04-26T23:05:32.770590+0000","last_clean":"2024-04-26T23:05:32.770590+0000","last_became_active":"2024-04-26T23:05:32.766787+0000","last_became_peered":"2024-04-26T23:05:32.766787+0000","last_unstale":"2024-04-26T23:05:32.770590+0000","last_undegraded":"2024-04-26T23:05:32.770590+0000","last_fullsized":"2024-04-26T23:05:32.770590+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:44:44.767658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771698+0000","last_change":"2024-04-26T23:05:32.765251+0000","last_active":"2024-04-26T23:05:32.771698+0000","last_peered":"2024-04-26T23:05:32.771698+0000","last_clean":"2024-04-26T23:05:32.771698+0000","last_became_active":"2024-04-26T23:05:32.764938+0000","last_became_peered":"2024-04-26T23:05:32.764938+0000","last_unstale":"2024-04-26T23:05:32.771698+0000","last_undegraded":"2024-04-26T23:05:32.771698+0000","last_fullsized":"2024-04-26T23:05:32.771698+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:35.989406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.764041+0000","last_change":"2024-04-26T23:05:32.760836+0000","last_active":"2024-04-26T23:05:32.764041+0000","last_peered":"2024-04-26T23:05:32.764041+0000","last_clean":"2024-04-26T23:05:32.764041+0000","last_became_active":"2024-04-26T23:05:32.760557+0000","last_became_peered":"2024-04-26T23:05:32.760557+0000","last_unstale":"2024-04-26T23:05:32.764041+0000","last_undegraded":"2024-04-26T23:05:32.764041+0000","last_fullsized":"2024-04-26T23:05:32.764041+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:48:12.566024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.769853+0000","last_change":"2024-04-26T23:05:32.764677+0000","last_active":"2024-04-26T23:05:32.769853+0000","last_peered":"2024-04-26T23:05:32.769853+0000","last_clean":"2024-04-26T23:05:32.769853+0000","last_became_active":"2024-04-26T23:05:32.761915+0000","last_became_peered":"2024-04-26T23:05:32.761915+0000","last_unstale":"2024-04-26T23:05:32.769853+0000","last_undegraded":"2024-04-26T23:05:32.769853+0000","last_fullsized":"2024-04-26T23:05:32.769853+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:35.989406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.770547+0000","last_change":"2024-04-26T23:05:32.767323+0000","last_active":"2024-04-26T23:05:32.770547+0000","last_peered":"2024-04-26T23:05:32.770547+0000","last_clean":"2024-04-26T23:05:32.770547+0000","last_became_active":"2024-04-26T23:05:32.766419+0000","last_became_peered":"2024-04-26T23:05:32.766419+0000","last_unstale":"2024-04-26T23:05:32.770547+0000","last_undegraded":"2024-04-26T23:05:32.770547+0000","last_fullsized":"2024-04-26T23:05:32.770547+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:24:54.832654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.779405+0000","last_change":"2024-04-26T23:05:32.775842+0000","last_active":"2024-04-26T23:05:32.779405+0000","last_peered":"2024-04-26T23:05:32.779405+0000","last_clean":"2024-04-26T23:05:32.779405+0000","last_became_active":"2024-04-26T23:05:32.775555+0000","last_became_peered":"2024-04-26T23:05:32.775555+0000","last_unstale":"2024-04-26T23:05:32.779405+0000","last_undegraded":"2024-04-26T23:05:32.779405+0000","last_fullsized":"2024-04-26T23:05:32.779405+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:34.076640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.778559+0000","last_change":"2024-04-26T23:05:32.775545+0000","last_active":"2024-04-26T23:05:32.778559+0000","last_peered":"2024-04-26T23:05:32.778559+0000","last_clean":"2024-04-26T23:05:32.778559+0000","last_became_active":"2024-04-26T23:05:32.775205+0000","last_became_peered":"2024-04-26T23:05:32.775205+0000","last_unstale":"2024-04-26T23:05:32.778559+0000","last_undegraded":"2024-04-26T23:05:32.778559+0000","last_fullsized":"2024-04-26T23:05:32.778559+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:53:00.646163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759109+0000","last_change":"2024-04-26T23:05:32.754414+0000","last_active":"2024-04-26T23:05:32.759109+0000","last_peered":"2024-04-26T23:05:32.759109+0000","last_clean":"2024-04-26T23:05:32.759109+0000","last_became_active":"2024-04-26T23:05:32.754060+0000","last_became_peered":"2024-04-26T23:05:32.754060+0000","last_unstale":"2024-04-26T23:05:32.759109+0000","last_undegraded":"2024-04-26T23:05:32.759109+0000","last_fullsized":"2024-04-26T23:05:32.759109+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:06:40.143755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.763609+0000","last_change":"2024-04-26T23:05:32.758763+0000","last_active":"2024-04-26T23:05:32.763609+0000","last_peered":"2024-04-26T23:05:32.763609+0000","last_clean":"2024-04-26T23:05:32.763609+0000","last_became_active":"2024-04-26T23:05:32.750829+0000","last_became_peered":"2024-04-26T23:05:32.750829+0000","last_unstale":"2024-04-26T23:05:32.763609+0000","last_undegraded":"2024-04-26T23:05:32.763609+0000","last_fullsized":"2024-04-26T23:05:32.763609+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:31.289442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.763145+0000","last_change":"2024-04-26T23:05:32.756213+0000","last_active":"2024-04-26T23:05:32.763145+0000","last_peered":"2024-04-26T23:05:32.763145+0000","last_clean":"2024-04-26T23:05:32.763145+0000","last_became_active":"2024-04-26T23:05:32.755856+0000","last_became_peered":"2024-04-26T23:05:32.755856+0000","last_unstale":"2024-04-26T23:05:32.763145+0000","last_undegraded":"2024-04-26T23:05:32.763145+0000","last_fullsized":"2024-04-26T23:05:32.763145+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:35:40.312530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759513+0000","last_change":"2024-04-26T23:05:32.753469+0000","last_active":"2024-04-26T23:05:32.759513+0000","last_peered":"2024-04-26T23:05:32.759513+0000","last_clean":"2024-04-26T23:05:32.759513+0000","last_became_active":"2024-04-26T23:05:32.752705+0000","last_became_peered":"2024-04-26T23:05:32.752705+0000","last_unstale":"2024-04-26T23:05:32.759513+0000","last_undegraded":"2024-04-26T23:05:32.759513+0000","last_fullsized":"2024-04-26T23:05:32.759513+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:47:04.134977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.753171+0000","last_change":"2024-04-26T23:05:32.747912+0000","last_active":"2024-04-26T23:05:32.753171+0000","last_peered":"2024-04-26T23:05:32.753171+0000","last_clean":"2024-04-26T23:05:32.753171+0000","last_became_active":"2024-04-26T23:05:32.747455+0000","last_became_peered":"2024-04-26T23:05:32.747455+0000","last_unstale":"2024-04-26T23:05:32.753171+0000","last_undegraded":"2024-04-26T23:05:32.753171+0000","last_fullsized":"2024-04-26T23:05:32.753171+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:26:54.412751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.767713+0000","last_change":"2024-04-26T23:05:32.763103+0000","last_active":"2024-04-26T23:05:32.767713+0000","last_peered":"2024-04-26T23:05:32.767713+0000","last_clean":"2024-04-26T23:05:32.767713+0000","last_became_active":"2024-04-26T23:05:32.762730+0000","last_became_peered":"2024-04-26T23:05:32.762730+0000","last_unstale":"2024-04-26T23:05:32.767713+0000","last_undegraded":"2024-04-26T23:05:32.767713+0000","last_fullsized":"2024-04-26T23:05:32.767713+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:48:10.275021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.767065+0000","last_change":"2024-04-26T23:05:32.761526+0000","last_active":"2024-04-26T23:05:32.767065+0000","last_peered":"2024-04-26T23:05:32.767065+0000","last_clean":"2024-04-26T23:05:32.767065+0000","last_became_active":"2024-04-26T23:05:32.761187+0000","last_became_peered":"2024-04-26T23:05:32.761187+0000","last_unstale":"2024-04-26T23:05:32.767065+0000","last_undegraded":"2024-04-26T23:05:32.767065+0000","last_fullsized":"2024-04-26T23:05:32.767065+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:58.027734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.770092+0000","last_change":"2024-04-26T23:05:32.765535+0000","last_active":"2024-04-26T23:05:32.770092+0000","last_peered":"2024-04-26T23:05:32.770092+0000","last_clean":"2024-04-26T23:05:32.770092+0000","last_became_active":"2024-04-26T23:05:32.765217+0000","last_became_peered":"2024-04-26T23:05:32.765217+0000","last_unstale":"2024-04-26T23:05:32.770092+0000","last_undegraded":"2024-04-26T23:05:32.770092+0000","last_fullsized":"2024-04-26T23:05:32.770092+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:17:46.183586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.780280+0000","last_change":"2024-04-26T23:05:32.777775+0000","last_active":"2024-04-26T23:05:32.780280+0000","last_peered":"2024-04-26T23:05:32.780280+0000","last_clean":"2024-04-26T23:05:32.780280+0000","last_became_active":"2024-04-26T23:05:32.776672+0000","last_became_peered":"2024-04-26T23:05:32.776672+0000","last_unstale":"2024-04-26T23:05:32.780280+0000","last_undegraded":"2024-04-26T23:05:32.780280+0000","last_fullsized":"2024-04-26T23:05:32.780280+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:09.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.769650+0000","last_change":"2024-04-26T23:05:32.764515+0000","last_active":"2024-04-26T23:05:32.769650+0000","last_peered":"2024-04-26T23:05:32.769650+0000","last_clean":"2024-04-26T23:05:32.769650+0000","last_became_active":"2024-04-26T23:05:32.763920+0000","last_became_peered":"2024-04-26T23:05:32.763920+0000","last_unstale":"2024-04-26T23:05:32.769650+0000","last_undegraded":"2024-04-26T23:05:32.769650+0000","last_fullsized":"2024-04-26T23:05:32.769650+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:00.038837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766657+0000","last_change":"2024-04-26T23:05:32.763303+0000","last_active":"2024-04-26T23:05:32.766657+0000","last_peered":"2024-04-26T23:05:32.766657+0000","last_clean":"2024-04-26T23:05:32.766657+0000","last_became_active":"2024-04-26T23:05:32.763020+0000","last_became_peered":"2024-04-26T23:05:32.763020+0000","last_unstale":"2024-04-26T23:05:32.766657+0000","last_undegraded":"2024-04-26T23:05:32.766657+0000","last_fullsized":"2024-04-26T23:05:32.766657+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:23.483542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.762623+0000","last_change":"2024-04-26T23:05:32.752684+0000","last_active":"2024-04-26T23:05:32.762623+0000","last_peered":"2024-04-26T23:05:32.762623+0000","last_clean":"2024-04-26T23:05:32.762623+0000","last_became_active":"2024-04-26T23:05:32.749863+0000","last_became_peered":"2024-04-26T23:05:32.749863+0000","last_unstale":"2024-04-26T23:05:32.762623+0000","last_undegraded":"2024-04-26T23:05:32.762623+0000","last_fullsized":"2024-04-26T23:05:32.762623+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:14.712885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.761459+0000","last_change":"2024-04-26T23:05:32.758274+0000","last_active":"2024-04-26T23:05:32.761459+0000","last_peered":"2024-04-26T23:05:32.761459+0000","last_clean":"2024-04-26T23:05:32.761459+0000","last_became_active":"2024-04-26T23:05:32.757953+0000","last_became_peered":"2024-04-26T23:05:32.757953+0000","last_unstale":"2024-04-26T23:05:32.761459+0000","last_undegraded":"2024-04-26T23:05:32.761459+0000","last_fullsized":"2024-04-26T23:05:32.761459+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:58.444855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771490+0000","last_change":"2024-04-26T23:05:32.767346+0000","last_active":"2024-04-26T23:05:32.771490+0000","last_peered":"2024-04-26T23:05:32.771490+0000","last_clean":"2024-04-26T23:05:32.771490+0000","last_became_active":"2024-04-26T23:05:32.766954+0000","last_became_peered":"2024-04-26T23:05:32.766954+0000","last_unstale":"2024-04-26T23:05:32.771490+0000","last_undegraded":"2024-04-26T23:05:32.771490+0000","last_fullsized":"2024-04-26T23:05:32.771490+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:42:49.983638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.769513+0000","last_change":"2024-04-26T23:05:32.766488+0000","last_active":"2024-04-26T23:05:32.769513+0000","last_peered":"2024-04-26T23:05:32.769513+0000","last_clean":"2024-04-26T23:05:32.769513+0000","last_became_active":"2024-04-26T23:05:32.764925+0000","last_became_peered":"2024-04-26T23:05:32.764925+0000","last_unstale":"2024-04-26T23:05:32.769513+0000","last_undegraded":"2024-04-26T23:05:32.769513+0000","last_fullsized":"2024-04-26T23:05:32.769513+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:44:44.767658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.756019+0000","last_change":"2024-04-26T23:05:32.751512+0000","last_active":"2024-04-26T23:05:32.756019+0000","last_peered":"2024-04-26T23:05:32.756019+0000","last_clean":"2024-04-26T23:05:32.756019+0000","last_became_active":"2024-04-26T23:05:32.750122+0000","last_became_peered":"2024-04-26T23:05:32.750122+0000","last_unstale":"2024-04-26T23:05:32.756019+0000","last_undegraded":"2024-04-26T23:05:32.756019+0000","last_fullsized":"2024-04-26T23:05:32.756019+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:35.358920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771737+0000","last_change":"2024-04-26T23:05:32.756778+0000","last_active":"2024-04-26T23:05:32.771737+0000","last_peered":"2024-04-26T23:05:32.771737+0000","last_clean":"2024-04-26T23:05:32.771737+0000","last_became_active":"2024-04-26T23:05:32.756219+0000","last_became_peered":"2024-04-26T23:05:32.756219+0000","last_unstale":"2024-04-26T23:05:32.771737+0000","last_undegraded":"2024-04-26T23:05:32.771737+0000","last_fullsized":"2024-04-26T23:05:32.771737+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:51.859455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.765990+0000","last_change":"2024-04-26T23:05:32.761026+0000","last_active":"2024-04-26T23:05:32.765990+0000","last_peered":"2024-04-26T23:05:32.765990+0000","last_clean":"2024-04-26T23:05:32.765990+0000","last_became_active":"2024-04-26T23:05:32.760738+0000","last_became_peered":"2024-04-26T23:05:32.760738+0000","last_unstale":"2024-04-26T23:05:32.765990+0000","last_undegraded":"2024-04-26T23:05:32.765990+0000","last_fullsized":"2024-04-26T23:05:32.765990+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:34:57.363296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"64'4","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.780727+0000","last_change":"2024-04-26T23:05:31.770725+0000","last_active":"2024-04-26T23:05:32.780727+0000","last_peered":"2024-04-26T23:05:32.780727+0000","last_clean":"2024-04-26T23:05:32.780727+0000","last_became_active":"2024-04-26T23:05:31.770414+0000","last_became_peered":"2024-04-26T23:05:31.770414+0000","last_unstale":"2024-04-26T23:05:32.780727+0000","last_undegraded":"2024-04-26T23:05:32.780727+0000","last_fullsized":"2024-04-26T23:05:32.780727+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:58:13.589814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.762649+0000","last_change":"2024-04-26T23:05:32.754181+0000","last_active":"2024-04-26T23:05:32.762649+0000","last_peered":"2024-04-26T23:05:32.762649+0000","last_clean":"2024-04-26T23:05:32.762649+0000","last_became_active":"2024-04-26T23:05:32.753872+0000","last_became_peered":"2024-04-26T23:05:32.753872+0000","last_unstale":"2024-04-26T23:05:32.762649+0000","last_undegraded":"2024-04-26T23:05:32.762649+0000","last_fullsized":"2024-04-26T23:05:32.762649+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:47:04.134977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"64'1","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.779837+0000","last_change":"2024-04-26T23:05:31.770026+0000","last_active":"2024-04-26T23:05:32.779837+0000","last_peered":"2024-04-26T23:05:32.779837+0000","last_clean":"2024-04-26T23:05:32.779837+0000","last_became_active":"2024-04-26T23:05:31.769545+0000","last_became_peered":"2024-04-26T23:05:31.769545+0000","last_unstale":"2024-04-26T23:05:32.779837+0000","last_undegraded":"2024-04-26T23:05:32.779837+0000","last_fullsized":"2024-04-26T23:05:32.779837+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:31.908410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.776323+0000","last_change":"2024-04-26T23:05:32.761309+0000","last_active":"2024-04-26T23:05:32.776323+0000","last_peered":"2024-04-26T23:05:32.776323+0000","last_clean":"2024-04-26T23:05:32.776323+0000","last_became_active":"2024-04-26T23:05:32.760980+0000","last_became_peered":"2024-04-26T23:05:32.760980+0000","last_unstale":"2024-04-26T23:05:32.776323+0000","last_undegraded":"2024-04-26T23:05:32.776323+0000","last_fullsized":"2024-04-26T23:05:32.776323+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:43:46.200740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"64'2","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.777711+0000","last_change":"2024-04-26T23:05:31.767871+0000","last_active":"2024-04-26T23:05:32.777711+0000","last_peered":"2024-04-26T23:05:32.777711+0000","last_clean":"2024-04-26T23:05:32.777711+0000","last_became_active":"2024-04-26T23:05:31.767184+0000","last_became_peered":"2024-04-26T23:05:31.767184+0000","last_unstale":"2024-04-26T23:05:32.777711+0000","last_undegraded":"2024-04-26T23:05:32.777711+0000","last_fullsized":"2024-04-26T23:05:32.777711+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:43:50.062402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.773482+0000","last_change":"2024-04-26T23:05:32.769325+0000","last_active":"2024-04-26T23:05:32.773482+0000","last_peered":"2024-04-26T23:05:32.773482+0000","last_clean":"2024-04-26T23:05:32.773482+0000","last_became_active":"2024-04-26T23:05:32.769062+0000","last_became_peered":"2024-04-26T23:05:32.769062+0000","last_unstale":"2024-04-26T23:05:32.773482+0000","last_undegraded":"2024-04-26T23:05:32.773482+0000","last_fullsized":"2024-04-26T23:05:32.773482+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:24:54.832654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"64'2","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.778964+0000","last_change":"2024-04-26T23:05:31.754696+0000","last_active":"2024-04-26T23:05:32.778964+0000","last_peered":"2024-04-26T23:05:32.778964+0000","last_clean":"2024-04-26T23:05:32.778964+0000","last_became_active":"2024-04-26T23:05:31.754353+0000","last_became_peered":"2024-04-26T23:05:31.754353+0000","last_unstale":"2024-04-26T23:05:32.778964+0000","last_undegraded":"2024-04-26T23:05:32.778964+0000","last_fullsized":"2024-04-26T23:05:32.778964+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:28.262197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.755471+0000","last_change":"2024-04-26T23:05:32.749775+0000","last_active":"2024-04-26T23:05:32.755471+0000","last_peered":"2024-04-26T23:05:32.755471+0000","last_clean":"2024-04-26T23:05:32.755471+0000","last_became_active":"2024-04-26T23:05:32.749407+0000","last_became_peered":"2024-04-26T23:05:32.749407+0000","last_unstale":"2024-04-26T23:05:32.755471+0000","last_undegraded":"2024-04-26T23:05:32.755471+0000","last_fullsized":"2024-04-26T23:05:32.755471+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:47.499283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"64'1","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.776633+0000","last_change":"2024-04-26T23:05:31.753361+0000","last_active":"2024-04-26T23:05:32.776633+0000","last_peered":"2024-04-26T23:05:32.776633+0000","last_clean":"2024-04-26T23:05:32.776633+0000","last_became_active":"2024-04-26T23:05:31.752976+0000","last_became_peered":"2024-04-26T23:05:31.752976+0000","last_unstale":"2024-04-26T23:05:32.776633+0000","last_undegraded":"2024-04-26T23:05:32.776633+0000","last_fullsized":"2024-04-26T23:05:32.776633+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:23:43.447217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759892+0000","last_change":"2024-04-26T23:05:32.750891+0000","last_active":"2024-04-26T23:05:32.759892+0000","last_peered":"2024-04-26T23:05:32.759892+0000","last_clean":"2024-04-26T23:05:32.759892+0000","last_became_active":"2024-04-26T23:05:32.750481+0000","last_became_peered":"2024-04-26T23:05:32.750481+0000","last_unstale":"2024-04-26T23:05:32.759892+0000","last_undegraded":"2024-04-26T23:05:32.759892+0000","last_fullsized":"2024-04-26T23:05:32.759892+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:21:38.429391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"64'1","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771888+0000","last_change":"2024-04-26T23:05:31.746607+0000","last_active":"2024-04-26T23:05:32.771888+0000","last_peered":"2024-04-26T23:05:32.771888+0000","last_clean":"2024-04-26T23:05:32.771888+0000","last_became_active":"2024-04-26T23:05:31.746253+0000","last_became_peered":"2024-04-26T23:05:31.746253+0000","last_unstale":"2024-04-26T23:05:32.771888+0000","last_undegraded":"2024-04-26T23:05:32.771888+0000","last_fullsized":"2024-04-26T23:05:32.771888+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:34.392268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759585+0000","last_change":"2024-04-26T23:05:32.753390+0000","last_active":"2024-04-26T23:05:32.759585+0000","last_peered":"2024-04-26T23:05:32.759585+0000","last_clean":"2024-04-26T23:05:32.759585+0000","last_became_active":"2024-04-26T23:05:32.752333+0000","last_became_peered":"2024-04-26T23:05:32.752333+0000","last_unstale":"2024-04-26T23:05:32.759585+0000","last_undegraded":"2024-04-26T23:05:32.759585+0000","last_fullsized":"2024-04-26T23:05:32.759585+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:21:24.525954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.725901+0000","last_change":"2024-04-26T23:05:31.760608+0000","last_active":"2024-04-26T23:05:32.725901+0000","last_peered":"2024-04-26T23:05:32.725901+0000","last_clean":"2024-04-26T23:05:32.725901+0000","last_became_active":"2024-04-26T23:05:31.759722+0000","last_became_peered":"2024-04-26T23:05:31.759722+0000","last_unstale":"2024-04-26T23:05:32.725901+0000","last_undegraded":"2024-04-26T23:05:32.725901+0000","last_fullsized":"2024-04-26T23:05:32.725901+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:13:10.834587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.753118+0000","last_change":"2024-04-26T23:05:32.747738+0000","last_active":"2024-04-26T23:05:32.753118+0000","last_peered":"2024-04-26T23:05:32.753118+0000","last_clean":"2024-04-26T23:05:32.753118+0000","last_became_active":"2024-04-26T23:05:32.747344+0000","last_became_peered":"2024-04-26T23:05:32.747344+0000","last_unstale":"2024-04-26T23:05:32.753118+0000","last_undegraded":"2024-04-26T23:05:32.753118+0000","last_fullsized":"2024-04-26T23:05:32.753118+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:19.465420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.731134+0000","last_change":"2024-04-26T23:05:31.768493+0000","last_active":"2024-04-26T23:05:32.731134+0000","last_peered":"2024-04-26T23:05:32.731134+0000","last_clean":"2024-04-26T23:05:32.731134+0000","last_became_active":"2024-04-26T23:05:31.767906+0000","last_became_peered":"2024-04-26T23:05:31.767906+0000","last_unstale":"2024-04-26T23:05:32.731134+0000","last_undegraded":"2024-04-26T23:05:32.731134+0000","last_fullsized":"2024-04-26T23:05:32.731134+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:20:04.698291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766361+0000","last_change":"2024-04-26T23:05:32.764563+0000","last_active":"2024-04-26T23:05:32.766361+0000","last_peered":"2024-04-26T23:05:32.766361+0000","last_clean":"2024-04-26T23:05:32.766361+0000","last_became_active":"2024-04-26T23:05:32.762251+0000","last_became_peered":"2024-04-26T23:05:32.762251+0000","last_unstale":"2024-04-26T23:05:32.766361+0000","last_undegraded":"2024-04-26T23:05:32.766361+0000","last_fullsized":"2024-04-26T23:05:32.766361+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:00.038837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.756401+0000","last_change":"2024-04-26T23:05:31.772660+0000","last_active":"2024-04-26T23:05:32.756401+0000","last_peered":"2024-04-26T23:05:32.756401+0000","last_clean":"2024-04-26T23:05:32.756401+0000","last_became_active":"2024-04-26T23:05:31.772338+0000","last_became_peered":"2024-04-26T23:05:31.772338+0000","last_unstale":"2024-04-26T23:05:32.756401+0000","last_undegraded":"2024-04-26T23:05:32.756401+0000","last_fullsized":"2024-04-26T23:05:32.756401+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:22.466819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766066+0000","last_change":"2024-04-26T23:05:32.757463+0000","last_active":"2024-04-26T23:05:32.766066+0000","last_peered":"2024-04-26T23:05:32.766066+0000","last_clean":"2024-04-26T23:05:32.766066+0000","last_became_active":"2024-04-26T23:05:32.757171+0000","last_became_peered":"2024-04-26T23:05:32.757171+0000","last_unstale":"2024-04-26T23:05:32.766066+0000","last_undegraded":"2024-04-26T23:05:32.766066+0000","last_fullsized":"2024-04-26T23:05:32.766066+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:44.040675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.735496+0000","last_change":"2024-04-26T23:05:31.757301+0000","last_active":"2024-04-26T23:05:32.735496+0000","last_peered":"2024-04-26T23:05:32.735496+0000","last_clean":"2024-04-26T23:05:32.735496+0000","last_became_active":"2024-04-26T23:05:31.755678+0000","last_became_peered":"2024-04-26T23:05:31.755678+0000","last_unstale":"2024-04-26T23:05:32.735496+0000","last_undegraded":"2024-04-26T23:05:32.735496+0000","last_fullsized":"2024-04-26T23:05:32.735496+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:20:04.698291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.762054+0000","last_change":"2024-04-26T23:05:32.755223+0000","last_active":"2024-04-26T23:05:32.762054+0000","last_peered":"2024-04-26T23:05:32.762054+0000","last_clean":"2024-04-26T23:05:32.762054+0000","last_became_active":"2024-04-26T23:05:32.754915+0000","last_became_peered":"2024-04-26T23:05:32.754915+0000","last_unstale":"2024-04-26T23:05:32.762054+0000","last_undegraded":"2024-04-26T23:05:32.762054+0000","last_fullsized":"2024-04-26T23:05:32.762054+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:01:32.493056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"64'3","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.781767+0000","last_change":"2024-04-26T23:05:31.772480+0000","last_active":"2024-04-26T23:05:32.781767+0000","last_peered":"2024-04-26T23:05:32.781767+0000","last_clean":"2024-04-26T23:05:32.781767+0000","last_became_active":"2024-04-26T23:05:31.772165+0000","last_became_peered":"2024-04-26T23:05:31.772165+0000","last_unstale":"2024-04-26T23:05:32.781767+0000","last_undegraded":"2024-04-26T23:05:32.781767+0000","last_fullsized":"2024-04-26T23:05:32.781767+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:30.272719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.761329+0000","last_change":"2024-04-26T23:05:32.755588+0000","last_active":"2024-04-26T23:05:32.761329+0000","last_peered":"2024-04-26T23:05:32.761329+0000","last_clean":"2024-04-26T23:05:32.761329+0000","last_became_active":"2024-04-26T23:05:32.755282+0000","last_became_peered":"2024-04-26T23:05:32.755282+0000","last_unstale":"2024-04-26T23:05:32.761329+0000","last_undegraded":"2024-04-26T23:05:32.761329+0000","last_fullsized":"2024-04-26T23:05:32.761329+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:01:32.493056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"64'6","reported_seq":18,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.782019+0000","last_change":"2024-04-26T23:05:31.760989+0000","last_active":"2024-04-26T23:05:32.782019+0000","last_peered":"2024-04-26T23:05:32.782019+0000","last_clean":"2024-04-26T23:05:32.782019+0000","last_became_active":"2024-04-26T23:05:31.760456+0000","last_became_peered":"2024-04-26T23:05:31.760456+0000","last_unstale":"2024-04-26T23:05:32.782019+0000","last_undegraded":"2024-04-26T23:05:32.782019+0000","last_fullsized":"2024-04-26T23:05:32.782019+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:46.525277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771143+0000","last_change":"2024-04-26T23:05:32.765693+0000","last_active":"2024-04-26T23:05:32.771143+0000","last_peered":"2024-04-26T23:05:32.771143+0000","last_clean":"2024-04-26T23:05:32.771143+0000","last_became_active":"2024-04-26T23:05:32.763803+0000","last_became_peered":"2024-04-26T23:05:32.763803+0000","last_unstale":"2024-04-26T23:05:32.771143+0000","last_undegraded":"2024-04-26T23:05:32.771143+0000","last_fullsized":"2024-04-26T23:05:32.771143+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:22.872508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"64'2","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.779891+0000","last_change":"2024-04-26T23:05:31.770239+0000","last_active":"2024-04-26T23:05:32.779891+0000","last_peered":"2024-04-26T23:05:32.779891+0000","last_clean":"2024-04-26T23:05:32.779891+0000","last_became_active":"2024-04-26T23:05:31.769941+0000","last_became_peered":"2024-04-26T23:05:31.769941+0000","last_unstale":"2024-04-26T23:05:32.779891+0000","last_undegraded":"2024-04-26T23:05:32.779891+0000","last_fullsized":"2024-04-26T23:05:32.779891+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:30.602553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.776599+0000","last_change":"2024-04-26T23:05:32.772869+0000","last_active":"2024-04-26T23:05:32.776599+0000","last_peered":"2024-04-26T23:05:32.776599+0000","last_clean":"2024-04-26T23:05:32.776599+0000","last_became_active":"2024-04-26T23:05:32.772567+0000","last_became_peered":"2024-04-26T23:05:32.772567+0000","last_unstale":"2024-04-26T23:05:32.776599+0000","last_undegraded":"2024-04-26T23:05:32.776599+0000","last_fullsized":"2024-04-26T23:05:32.776599+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:26.182654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.749657+0000","last_change":"2024-04-26T23:05:31.770279+0000","last_active":"2024-04-26T23:05:32.749657+0000","last_peered":"2024-04-26T23:05:32.749657+0000","last_clean":"2024-04-26T23:05:32.749657+0000","last_became_active":"2024-04-26T23:05:31.769923+0000","last_became_peered":"2024-04-26T23:05:31.769923+0000","last_unstale":"2024-04-26T23:05:32.749657+0000","last_undegraded":"2024-04-26T23:05:32.749657+0000","last_fullsized":"2024-04-26T23:05:32.749657+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:30.602553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.763602+0000","last_change":"2024-04-26T23:05:32.755633+0000","last_active":"2024-04-26T23:05:32.763602+0000","last_peered":"2024-04-26T23:05:32.763602+0000","last_clean":"2024-04-26T23:05:32.763602+0000","last_became_active":"2024-04-26T23:05:32.753211+0000","last_became_peered":"2024-04-26T23:05:32.753211+0000","last_unstale":"2024-04-26T23:05:32.763602+0000","last_undegraded":"2024-04-26T23:05:32.763602+0000","last_fullsized":"2024-04-26T23:05:32.763602+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:35:40.312530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.756535+0000","last_change":"2024-04-26T23:05:31.768537+0000","last_active":"2024-04-26T23:05:32.756535+0000","last_peered":"2024-04-26T23:05:32.756535+0000","last_clean":"2024-04-26T23:05:32.756535+0000","last_became_active":"2024-04-26T23:05:31.768175+0000","last_became_peered":"2024-04-26T23:05:31.768175+0000","last_unstale":"2024-04-26T23:05:32.756535+0000","last_undegraded":"2024-04-26T23:05:32.756535+0000","last_fullsized":"2024-04-26T23:05:32.756535+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:01:59.022114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.770370+0000","last_change":"2024-04-26T23:05:32.766588+0000","last_active":"2024-04-26T23:05:32.770370+0000","last_peered":"2024-04-26T23:05:32.770370+0000","last_clean":"2024-04-26T23:05:32.770370+0000","last_became_active":"2024-04-26T23:05:32.762965+0000","last_became_peered":"2024-04-26T23:05:32.762965+0000","last_unstale":"2024-04-26T23:05:32.770370+0000","last_undegraded":"2024-04-26T23:05:32.770370+0000","last_fullsized":"2024-04-26T23:05:32.770370+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:35.989406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"64'8","reported_seq":20,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.780778+0000","last_change":"2024-04-26T23:05:31.765009+0000","last_active":"2024-04-26T23:05:32.780778+0000","last_peered":"2024-04-26T23:05:32.780778+0000","last_clean":"2024-04-26T23:05:32.780778+0000","last_became_active":"2024-04-26T23:05:31.764477+0000","last_became_peered":"2024-04-26T23:05:31.764477+0000","last_unstale":"2024-04-26T23:05:32.780778+0000","last_undegraded":"2024-04-26T23:05:32.780778+0000","last_fullsized":"2024-04-26T23:05:32.780778+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:28:04.128466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.777622+0000","last_change":"2024-04-26T23:05:32.774045+0000","last_active":"2024-04-26T23:05:32.777622+0000","last_peered":"2024-04-26T23:05:32.777622+0000","last_clean":"2024-04-26T23:05:32.777622+0000","last_became_active":"2024-04-26T23:05:32.772012+0000","last_became_peered":"2024-04-26T23:05:32.772012+0000","last_unstale":"2024-04-26T23:05:32.777622+0000","last_undegraded":"2024-04-26T23:05:32.777622+0000","last_fullsized":"2024-04-26T23:05:32.777622+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:26.182654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"64'1","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.777986+0000","last_change":"2024-04-26T23:05:31.769936+0000","last_active":"2024-04-26T23:05:32.777986+0000","last_peered":"2024-04-26T23:05:32.777986+0000","last_clean":"2024-04-26T23:05:32.777986+0000","last_became_active":"2024-04-26T23:05:31.767600+0000","last_became_peered":"2024-04-26T23:05:31.767600+0000","last_unstale":"2024-04-26T23:05:32.777986+0000","last_undegraded":"2024-04-26T23:05:32.777986+0000","last_fullsized":"2024-04-26T23:05:32.777986+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:34:08.035750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.758574+0000","last_change":"2024-04-26T23:05:32.751224+0000","last_active":"2024-04-26T23:05:32.758574+0000","last_peered":"2024-04-26T23:05:32.758574+0000","last_clean":"2024-04-26T23:05:32.758574+0000","last_became_active":"2024-04-26T23:05:32.750525+0000","last_became_peered":"2024-04-26T23:05:32.750525+0000","last_unstale":"2024-04-26T23:05:32.758574+0000","last_undegraded":"2024-04-26T23:05:32.758574+0000","last_fullsized":"2024-04-26T23:05:32.758574+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:35.358920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"64'2","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.778479+0000","last_change":"2024-04-26T23:05:31.769437+0000","last_active":"2024-04-26T23:05:32.778479+0000","last_peered":"2024-04-26T23:05:32.778479+0000","last_clean":"2024-04-26T23:05:32.778479+0000","last_became_active":"2024-04-26T23:05:31.768705+0000","last_became_peered":"2024-04-26T23:05:31.768705+0000","last_unstale":"2024-04-26T23:05:32.778479+0000","last_undegraded":"2024-04-26T23:05:32.778479+0000","last_fullsized":"2024-04-26T23:05:32.778479+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:48:09.258298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.764533+0000","last_change":"2024-04-26T23:05:32.761715+0000","last_active":"2024-04-26T23:05:32.764533+0000","last_peered":"2024-04-26T23:05:32.764533+0000","last_clean":"2024-04-26T23:05:32.764533+0000","last_became_active":"2024-04-26T23:05:32.758821+0000","last_became_peered":"2024-04-26T23:05:32.758821+0000","last_unstale":"2024-04-26T23:05:32.764533+0000","last_undegraded":"2024-04-26T23:05:32.764533+0000","last_fullsized":"2024-04-26T23:05:32.764533+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:23:41.516794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"64'1","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.781798+0000","last_change":"2024-04-26T23:05:31.769171+0000","last_active":"2024-04-26T23:05:32.781798+0000","last_peered":"2024-04-26T23:05:32.781798+0000","last_clean":"2024-04-26T23:05:32.781798+0000","last_became_active":"2024-04-26T23:05:31.768464+0000","last_became_peered":"2024-04-26T23:05:31.768464+0000","last_unstale":"2024-04-26T23:05:32.781798+0000","last_undegraded":"2024-04-26T23:05:32.781798+0000","last_fullsized":"2024-04-26T23:05:32.781798+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:57.428132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.762470+0000","last_change":"2024-04-26T23:05:32.757835+0000","last_active":"2024-04-26T23:05:32.762470+0000","last_peered":"2024-04-26T23:05:32.762470+0000","last_clean":"2024-04-26T23:05:32.762470+0000","last_became_active":"2024-04-26T23:05:32.756941+0000","last_became_peered":"2024-04-26T23:05:32.756941+0000","last_unstale":"2024-04-26T23:05:32.762470+0000","last_undegraded":"2024-04-26T23:05:32.762470+0000","last_fullsized":"2024-04-26T23:05:32.762470+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:48:12.566024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.726739+0000","last_change":"2024-04-26T23:05:31.761468+0000","last_active":"2024-04-26T23:05:32.726739+0000","last_peered":"2024-04-26T23:05:32.726739+0000","last_clean":"2024-04-26T23:05:32.726739+0000","last_became_active":"2024-04-26T23:05:31.761152+0000","last_became_peered":"2024-04-26T23:05:31.761152+0000","last_unstale":"2024-04-26T23:05:32.726739+0000","last_undegraded":"2024-04-26T23:05:32.726739+0000","last_fullsized":"2024-04-26T23:05:32.726739+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:37:26.663900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771692+0000","last_change":"2024-04-26T23:05:32.756293+0000","last_active":"2024-04-26T23:05:32.771692+0000","last_peered":"2024-04-26T23:05:32.771692+0000","last_clean":"2024-04-26T23:05:32.771692+0000","last_became_active":"2024-04-26T23:05:32.755936+0000","last_became_peered":"2024-04-26T23:05:32.755936+0000","last_unstale":"2024-04-26T23:05:32.771692+0000","last_undegraded":"2024-04-26T23:05:32.771692+0000","last_fullsized":"2024-04-26T23:05:32.771692+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:05.347403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"64'2","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.778918+0000","last_change":"2024-04-26T23:05:31.763776+0000","last_active":"2024-04-26T23:05:32.778918+0000","last_peered":"2024-04-26T23:05:32.778918+0000","last_clean":"2024-04-26T23:05:32.778918+0000","last_became_active":"2024-04-26T23:05:31.763362+0000","last_became_peered":"2024-04-26T23:05:31.763362+0000","last_unstale":"2024-04-26T23:05:32.778918+0000","last_undegraded":"2024-04-26T23:05:32.778918+0000","last_fullsized":"2024-04-26T23:05:32.778918+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:32:41.819376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766276+0000","last_change":"2024-04-26T23:05:32.761748+0000","last_active":"2024-04-26T23:05:32.766276+0000","last_peered":"2024-04-26T23:05:32.766276+0000","last_clean":"2024-04-26T23:05:32.766276+0000","last_became_active":"2024-04-26T23:05:32.758119+0000","last_became_peered":"2024-04-26T23:05:32.758119+0000","last_unstale":"2024-04-26T23:05:32.766276+0000","last_undegraded":"2024-04-26T23:05:32.766276+0000","last_fullsized":"2024-04-26T23:05:32.766276+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:31.289442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.749828+0000","last_change":"2024-04-26T23:05:31.769307+0000","last_active":"2024-04-26T23:05:32.749828+0000","last_peered":"2024-04-26T23:05:32.749828+0000","last_clean":"2024-04-26T23:05:32.749828+0000","last_became_active":"2024-04-26T23:05:31.769034+0000","last_became_peered":"2024-04-26T23:05:31.769034+0000","last_unstale":"2024-04-26T23:05:32.749828+0000","last_undegraded":"2024-04-26T23:05:32.749828+0000","last_fullsized":"2024-04-26T23:05:32.749828+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:21:37.412668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759564+0000","last_change":"2024-04-26T23:05:32.752754+0000","last_active":"2024-04-26T23:05:32.759564+0000","last_peered":"2024-04-26T23:05:32.759564+0000","last_clean":"2024-04-26T23:05:32.759564+0000","last_became_active":"2024-04-26T23:05:32.751685+0000","last_became_peered":"2024-04-26T23:05:32.751685+0000","last_unstale":"2024-04-26T23:05:32.759564+0000","last_undegraded":"2024-04-26T23:05:32.759564+0000","last_fullsized":"2024-04-26T23:05:32.759564+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:14.712885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.745528+0000","last_change":"2024-04-26T23:05:31.769617+0000","last_active":"2024-04-26T23:05:32.745528+0000","last_peered":"2024-04-26T23:05:32.745528+0000","last_clean":"2024-04-26T23:05:32.745528+0000","last_became_active":"2024-04-26T23:05:31.769190+0000","last_became_peered":"2024-04-26T23:05:31.769190+0000","last_unstale":"2024-04-26T23:05:32.745528+0000","last_undegraded":"2024-04-26T23:05:32.745528+0000","last_fullsized":"2024-04-26T23:05:32.745528+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:55:07.062224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771943+0000","last_change":"2024-04-26T23:05:32.757470+0000","last_active":"2024-04-26T23:05:32.771943+0000","last_peered":"2024-04-26T23:05:32.771943+0000","last_clean":"2024-04-26T23:05:32.771943+0000","last_became_active":"2024-04-26T23:05:32.757141+0000","last_became_peered":"2024-04-26T23:05:32.757141+0000","last_unstale":"2024-04-26T23:05:32.771943+0000","last_undegraded":"2024-04-26T23:05:32.771943+0000","last_fullsized":"2024-04-26T23:05:32.771943+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:18:44.249170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"64'2","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.780252+0000","last_change":"2024-04-26T23:05:31.771257+0000","last_active":"2024-04-26T23:05:32.780252+0000","last_peered":"2024-04-26T23:05:32.780252+0000","last_clean":"2024-04-26T23:05:32.780252+0000","last_became_active":"2024-04-26T23:05:31.770330+0000","last_became_peered":"2024-04-26T23:05:31.770330+0000","last_unstale":"2024-04-26T23:05:32.780252+0000","last_undegraded":"2024-04-26T23:05:32.780252+0000","last_fullsized":"2024-04-26T23:05:32.780252+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:46.525277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759360+0000","last_change":"2024-04-26T23:05:32.756115+0000","last_active":"2024-04-26T23:05:32.759360+0000","last_peered":"2024-04-26T23:05:32.759360+0000","last_clean":"2024-04-26T23:05:32.759360+0000","last_became_active":"2024-04-26T23:05:32.755800+0000","last_became_peered":"2024-04-26T23:05:32.755800+0000","last_unstale":"2024-04-26T23:05:32.759360+0000","last_undegraded":"2024-04-26T23:05:32.759360+0000","last_fullsized":"2024-04-26T23:05:32.759360+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:54:14.272684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.750555+0000","last_change":"2024-04-26T23:05:31.767074+0000","last_active":"2024-04-26T23:05:32.750555+0000","last_peered":"2024-04-26T23:05:32.750555+0000","last_clean":"2024-04-26T23:05:32.750555+0000","last_became_active":"2024-04-26T23:05:31.765961+0000","last_became_peered":"2024-04-26T23:05:31.765961+0000","last_unstale":"2024-04-26T23:05:32.750555+0000","last_undegraded":"2024-04-26T23:05:32.750555+0000","last_fullsized":"2024-04-26T23:05:32.750555+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:57.428132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.776777+0000","last_change":"2024-04-26T23:05:32.774635+0000","last_active":"2024-04-26T23:05:32.776777+0000","last_peered":"2024-04-26T23:05:32.776777+0000","last_clean":"2024-04-26T23:05:32.776777+0000","last_became_active":"2024-04-26T23:05:32.772832+0000","last_became_peered":"2024-04-26T23:05:32.772832+0000","last_unstale":"2024-04-26T23:05:32.776777+0000","last_undegraded":"2024-04-26T23:05:32.776777+0000","last_fullsized":"2024-04-26T23:05:32.776777+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:53:00.646163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.749182+0000","last_change":"2024-04-26T23:05:31.766535+0000","last_active":"2024-04-26T23:05:32.749182+0000","last_peered":"2024-04-26T23:05:32.749182+0000","last_clean":"2024-04-26T23:05:32.749182+0000","last_became_active":"2024-04-26T23:05:31.766192+0000","last_became_peered":"2024-04-26T23:05:31.766192+0000","last_unstale":"2024-04-26T23:05:32.749182+0000","last_undegraded":"2024-04-26T23:05:32.749182+0000","last_fullsized":"2024-04-26T23:05:32.749182+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:48:34.632624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.765599+0000","last_change":"2024-04-26T23:05:32.759298+0000","last_active":"2024-04-26T23:05:32.765599+0000","last_peered":"2024-04-26T23:05:32.765599+0000","last_clean":"2024-04-26T23:05:32.765599+0000","last_became_active":"2024-04-26T23:05:32.758971+0000","last_became_peered":"2024-04-26T23:05:32.758971+0000","last_unstale":"2024-04-26T23:05:32.765599+0000","last_undegraded":"2024-04-26T23:05:32.765599+0000","last_fullsized":"2024-04-26T23:05:32.765599+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:23:41.516794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.743548+0000","last_change":"2024-04-26T23:05:31.756036+0000","last_active":"2024-04-26T23:05:32.743548+0000","last_peered":"2024-04-26T23:05:32.743548+0000","last_clean":"2024-04-26T23:05:32.743548+0000","last_became_active":"2024-04-26T23:05:31.755674+0000","last_became_peered":"2024-04-26T23:05:31.755674+0000","last_unstale":"2024-04-26T23:05:32.743548+0000","last_undegraded":"2024-04-26T23:05:32.743548+0000","last_fullsized":"2024-04-26T23:05:32.743548+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:34:08.035750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.767384+0000","last_change":"2024-04-26T23:05:32.764451+0000","last_active":"2024-04-26T23:05:32.767384+0000","last_peered":"2024-04-26T23:05:32.767384+0000","last_clean":"2024-04-26T23:05:32.767384+0000","last_became_active":"2024-04-26T23:05:32.763008+0000","last_became_peered":"2024-04-26T23:05:32.763008+0000","last_unstale":"2024-04-26T23:05:32.767384+0000","last_undegraded":"2024-04-26T23:05:32.767384+0000","last_fullsized":"2024-04-26T23:05:32.767384+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:58:14.606536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.748146+0000","last_change":"2024-04-26T23:05:31.768363+0000","last_active":"2024-04-26T23:05:32.748146+0000","last_peered":"2024-04-26T23:05:32.748146+0000","last_clean":"2024-04-26T23:05:32.748146+0000","last_became_active":"2024-04-26T23:05:31.768061+0000","last_became_peered":"2024-04-26T23:05:31.768061+0000","last_unstale":"2024-04-26T23:05:32.748146+0000","last_undegraded":"2024-04-26T23:05:32.748146+0000","last_fullsized":"2024-04-26T23:05:32.748146+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:21:37.412668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.771542+0000","last_change":"2024-04-26T23:05:32.768067+0000","last_active":"2024-04-26T23:05:32.771542+0000","last_peered":"2024-04-26T23:05:32.771542+0000","last_clean":"2024-04-26T23:05:32.771542+0000","last_became_active":"2024-04-26T23:05:32.767731+0000","last_became_peered":"2024-04-26T23:05:32.767731+0000","last_unstale":"2024-04-26T23:05:32.771542+0000","last_undegraded":"2024-04-26T23:05:32.771542+0000","last_fullsized":"2024-04-26T23:05:32.771542+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:34:57.363296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.726001+0000","last_change":"2024-04-26T23:05:31.749289+0000","last_active":"2024-04-26T23:05:32.726001+0000","last_peered":"2024-04-26T23:05:32.726001+0000","last_clean":"2024-04-26T23:05:32.726001+0000","last_became_active":"2024-04-26T23:05:31.748930+0000","last_became_peered":"2024-04-26T23:05:31.748930+0000","last_unstale":"2024-04-26T23:05:32.726001+0000","last_undegraded":"2024-04-26T23:05:32.726001+0000","last_fullsized":"2024-04-26T23:05:32.726001+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:43.540669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759808+0000","last_change":"2024-04-26T23:05:32.750715+0000","last_active":"2024-04-26T23:05:32.759808+0000","last_peered":"2024-04-26T23:05:32.759808+0000","last_clean":"2024-04-26T23:05:32.759808+0000","last_became_active":"2024-04-26T23:05:32.750391+0000","last_became_peered":"2024-04-26T23:05:32.750391+0000","last_unstale":"2024-04-26T23:05:32.759808+0000","last_undegraded":"2024-04-26T23:05:32.759808+0000","last_fullsized":"2024-04-26T23:05:32.759808+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:31.619276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.726431+0000","last_change":"2024-04-26T23:05:31.755397+0000","last_active":"2024-04-26T23:05:32.726431+0000","last_peered":"2024-04-26T23:05:32.726431+0000","last_clean":"2024-04-26T23:05:32.726431+0000","last_became_active":"2024-04-26T23:05:31.755053+0000","last_became_peered":"2024-04-26T23:05:31.755053+0000","last_unstale":"2024-04-26T23:05:32.726431+0000","last_undegraded":"2024-04-26T23:05:32.726431+0000","last_fullsized":"2024-04-26T23:05:32.726431+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:11.732013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.759242+0000","last_change":"2024-04-26T23:05:32.754623+0000","last_active":"2024-04-26T23:05:32.759242+0000","last_peered":"2024-04-26T23:05:32.759242+0000","last_clean":"2024-04-26T23:05:32.759242+0000","last_became_active":"2024-04-26T23:05:32.754292+0000","last_became_peered":"2024-04-26T23:05:32.754292+0000","last_unstale":"2024-04-26T23:05:32.759242+0000","last_undegraded":"2024-04-26T23:05:32.759242+0000","last_fullsized":"2024-04-26T23:05:32.759242+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:40.736465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.738119+0000","last_change":"2024-04-26T23:05:31.762384+0000","last_active":"2024-04-26T23:05:32.738119+0000","last_peered":"2024-04-26T23:05:32.738119+0000","last_clean":"2024-04-26T23:05:32.738119+0000","last_became_active":"2024-04-26T23:05:31.762087+0000","last_became_peered":"2024-04-26T23:05:31.762087+0000","last_unstale":"2024-04-26T23:05:32.738119+0000","last_undegraded":"2024-04-26T23:05:32.738119+0000","last_fullsized":"2024-04-26T23:05:32.738119+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:34:56.346573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.766318+0000","last_change":"2024-04-26T23:05:32.759868+0000","last_active":"2024-04-26T23:05:32.766318+0000","last_peered":"2024-04-26T23:05:32.766318+0000","last_clean":"2024-04-26T23:05:32.766318+0000","last_became_active":"2024-04-26T23:05:32.759566+0000","last_became_peered":"2024-04-26T23:05:32.759566+0000","last_unstale":"2024-04-26T23:05:32.766318+0000","last_undegraded":"2024-04-26T23:05:32.766318+0000","last_fullsized":"2024-04-26T23:05:32.766318+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:31.712707+0000","last_clean_scrub_stamp":"2024-04-26T23:05:31.712707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:22.872508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.725362+0000","last_change":"2024-04-26T23:05:31.749603+0000","last_active":"2024-04-26T23:05:32.725362+0000","last_peered":"2024-04-26T23:05:32.725362+0000","last_clean":"2024-04-26T23:05:32.725362+0000","last_became_active":"2024-04-26T23:05:31.749283+0000","last_became_peered":"2024-04-26T23:05:31.749283+0000","last_unstale":"2024-04-26T23:05:32.725362+0000","last_undegraded":"2024-04-26T23:05:32.725362+0000","last_fullsized":"2024-04-26T23:05:32.725362+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:05:30.695984+0000","last_clean_scrub_stamp":"2024-04-26T23:05:30.695984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:58:00.646819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":254,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-26T23:05:32.750023+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:05:32.750023+0000","last_peered":"2024-04-26T23:05:32.750023+0000","last_clean":"2024-04-26T23:05:32.750023+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:05:32.750023+0000","last_undegraded":"2024-04-26T23:05:32.750023+0000","last_fullsized":"2024-04-26T23:05:32.750023+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":936396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":774144,"data_stored":353195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":778240,"data_stored":353217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":354807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":774144,"data_stored":353195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31928,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":761856,"data_stored":346028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":774144,"data_stored":353195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":936396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:05:37.725 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T23:05:37.725 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-26T23:05:37.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.980+0000 7fcd9be00640 1 Processor -- start 2024-04-26T23:05:37.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.980+0000 7fcd9be00640 1 -- start start 2024-04-26T23:05:37.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.982+0000 7fcd9be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcd940c7a90 0x7fcd940cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:37.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.982+0000 7fcd9be00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd940b41d0 con 0x7fcd94160630 2024-04-26T23:05:37.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.982+0000 7fcd9be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd940b44b0 con 0x7fcd94160a10 2024-04-26T23:05:37.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.982+0000 7fcd9be00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fcd940d0570 con 0x7fcd940c7a90 2024-04-26T23:05:37.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.982+0000 7fcd99400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcd94160630 0x7fcd940d0ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41172/0 (socket says 172.21.15.40:41172) 2024-04-26T23:05:37.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.982+0000 7fcd99400640 1 -- 172.21.15.40:0/970099210 learned_addr learned my addr 172.21.15.40:0/970099210 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:37.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.982+0000 7fcd99e00640 1 --2- 172.21.15.40:0/970099210 >> v2:172.21.15.40:3300/0 conn(0x7fcd940c7a90 0x7fcd940cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:37.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.983+0000 7fcd93e00640 1 -- 172.21.15.40:0/970099210 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2695411925 0 0) 0x7fcd940b41d0 con 0x7fcd94160630 2024-04-26T23:05:37.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.983+0000 7fcd93e00640 1 -- 172.21.15.40:0/970099210 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd7c003520 con 0x7fcd94160630 2024-04-26T23:05:37.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.983+0000 7fcd99e00640 1 -- 172.21.15.40:0/970099210 >> v1:172.21.15.103:6789/0 conn(0x7fcd94160a10 legacy=0x7fcd940cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:37.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.983+0000 7fcd99e00640 1 -- 172.21.15.40:0/970099210 >> v1:172.21.15.40:6789/0 conn(0x7fcd94160630 legacy=0x7fcd940d0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:37.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.983+0000 7fcd99e00640 1 -- 172.21.15.40:0/970099210 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd940d0200 con 0x7fcd940c7a90 2024-04-26T23:05:37.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.984+0000 7fcd99e00640 1 --2- 172.21.15.40:0/970099210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940c7a90 0x7fcd940cfb00 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c009d90 tx=0x7fcd8c02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621e9563834ac627 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:37.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.984+0000 7fcd93e00640 1 -- 172.21.15.40:0/970099210 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c007740 con 0x7fcd940c7a90 2024-04-26T23:05:37.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.984+0000 7fcd93e00640 1 -- 172.21.15.40:0/970099210 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd8c0078e0 con 0x7fcd940c7a90 2024-04-26T23:05:37.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.984+0000 7fcd93e00640 1 -- 172.21.15.40:0/970099210 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c007c90 con 0x7fcd940c7a90 2024-04-26T23:05:37.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.985+0000 7fcd9be00640 1 -- 172.21.15.40:0/970099210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940c7a90 msgr2=0x7fcd940cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:37.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.985+0000 7fcd9be00640 1 --2- 172.21.15.40:0/970099210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940c7a90 0x7fcd940cfb00 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c009d90 tx=0x7fcd8c02a730 comp rx=0 tx=0).stop 2024-04-26T23:05:37.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.985+0000 7fcd9be00640 1 -- 172.21.15.40:0/970099210 shutdown_connections 2024-04-26T23:05:37.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.985+0000 7fcd9be00640 1 --2- 172.21.15.40:0/970099210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940c7a90 0x7fcd940cfb00 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.985+0000 7fcd9be00640 1 -- 172.21.15.40:0/970099210 >> 172.21.15.40:0/970099210 conn(0x7fcd940d2dc0 msgr2=0x7fcd940d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:37.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.985+0000 7fcd9be00640 1 -- 172.21.15.40:0/970099210 shutdown_connections 2024-04-26T23:05:37.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.985+0000 7fcd9be00640 1 -- 172.21.15.40:0/970099210 wait complete. 2024-04-26T23:05:37.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.987+0000 7fcd9be00640 1 Processor -- start 2024-04-26T23:05:37.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.987+0000 7fcd9be00640 1 -- start start 2024-04-26T23:05:37.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.988+0000 7fcd9be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940c7a90 0x7fcd9415e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:37.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.988+0000 7fcd9be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd94160630 0x7fcd9414fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:37.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.988+0000 7fcd9be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd94160a10 0x7fcd9414e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:37.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.988+0000 7fcd9be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcd94167f60 con 0x7fcd940c7a90 2024-04-26T23:05:37.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.988+0000 7fcd9be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcd940d3b10 con 0x7fcd94160630 2024-04-26T23:05:37.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.988+0000 7fcd9be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcd941680e0 con 0x7fcd94160a10 2024-04-26T23:05:37.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.988+0000 7fcd99400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940c7a90 0x7fcd9415e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:37.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.988+0000 7fcd99400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940c7a90 0x7fcd9415e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46280/0 (socket says 172.21.15.40:46280) 2024-04-26T23:05:37.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.988+0000 7fcd99400640 1 -- 172.21.15.40:0/2720836433 learned_addr learned my addr 172.21.15.40:0/2720836433 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.988+0000 7fcd98a00640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd94160630 0x7fcd9414fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.988+0000 7fcd99400640 1 -- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd94160a10 msgr2=0x7fcd9414e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.988+0000 7fcd99400640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd94160a10 0x7fcd9414e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.989+0000 7fcd99400640 1 -- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd94160630 msgr2=0x7fcd9414fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.989+0000 7fcd99400640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd94160630 0x7fcd9414fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.989+0000 7fcd99400640 1 -- 172.21.15.40:0/2720836433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd9414f340 con 0x7fcd940c7a90 2024-04-26T23:05:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.989+0000 7fcd98a00640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd94160630 0x7fcd9414fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.989+0000 7fcd99400640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940c7a90 0x7fcd9415e6b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800c430 tx=0x7fcd8800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.989+0000 7fcd91600640 1 -- 172.21.15.40:0/2720836433 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd88017070 con 0x7fcd940c7a90 2024-04-26T23:05:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.989+0000 7fcd91600640 1 -- 172.21.15.40:0/2720836433 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd88005ce0 con 0x7fcd940c7a90 2024-04-26T23:05:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.989+0000 7fcd91600640 1 -- 172.21.15.40:0/2720836433 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd880072f0 con 0x7fcd940c7a90 2024-04-26T23:05:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.989+0000 7fcd9be00640 1 -- 172.21.15.40:0/2720836433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd9414ee30 con 0x7fcd940c7a90 2024-04-26T23:05:37.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.989+0000 7fcd9be00640 1 -- 172.21.15.40:0/2720836433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd94152430 con 0x7fcd940c7a90 2024-04-26T23:05:37.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.991+0000 7fcd9be00640 1 -- 172.21.15.40:0/2720836433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd940694f0 con 0x7fcd940c7a90 2024-04-26T23:05:37.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.992+0000 7fcd91600640 1 -- 172.21.15.40:0/2720836433 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcd8801a070 con 0x7fcd940c7a90 2024-04-26T23:05:37.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.993+0000 7fcd91600640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd7806c920 0x7fcd7806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:37.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.993+0000 7fcd91600640 1 -- 172.21.15.40:0/2720836433 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fcd8808b1f0 con 0x7fcd940c7a90 2024-04-26T23:05:37.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.994+0000 7fcd98a00640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd7806c920 0x7fcd7806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:37.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.995+0000 7fcd98a00640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd7806c920 0x7fcd7806ede0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcd940c7750 tx=0x7fcd84008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:37.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:37.997+0000 7fcd91600640 1 -- 172.21.15.40:0/2720836433 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fcd9414ee30 con 0x7fcd940c7a90 2024-04-26T23:05:38.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.277+0000 7fcd9be00640 1 -- 172.21.15.40:0/2720836433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fcd94159880 con 0x7fcd940c7a90 2024-04-26T23:05:38.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.283+0000 7fcd91600640 1 -- 172.21.15.40:0/2720836433 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7fcd9414f340 con 0x7fcd940c7a90 2024-04-26T23:05:38.284 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-04-26T23:05:38.284 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg== 2024-04-26T23:05:38.284 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-04-26T23:05:38.284 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-04-26T23:05:38.284 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-26T23:05:38.284 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-04-26T23:05:38.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.287+0000 7fcd9be00640 1 -- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd7806c920 msgr2=0x7fcd7806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.287+0000 7fcd9be00640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd7806c920 0x7fcd7806ede0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcd940c7750 tx=0x7fcd84008040 comp rx=0 tx=0).stop 2024-04-26T23:05:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.287+0000 7fcd9be00640 1 -- 172.21.15.40:0/2720836433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940c7a90 msgr2=0x7fcd9415e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.287+0000 7fcd9be00640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940c7a90 0x7fcd9415e6b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800c430 tx=0x7fcd8800c8f0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.287+0000 7fcd9be00640 1 -- 172.21.15.40:0/2720836433 shutdown_connections 2024-04-26T23:05:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.287+0000 7fcd9be00640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd7806c920 0x7fcd7806ede0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.287+0000 7fcd9be00640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd94160a10 0x7fcd9414e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.287+0000 7fcd9be00640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd94160630 0x7fcd9414fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.287+0000 7fcd9be00640 1 --2- 172.21.15.40:0/2720836433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd940c7a90 0x7fcd9415e6b0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.287+0000 7fcd9be00640 1 -- 172.21.15.40:0/2720836433 >> 172.21.15.40:0/2720836433 conn(0x7fcd940d2dc0 msgr2=0x7fcd9415f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.287+0000 7fcd9be00640 1 -- 172.21.15.40:0/2720836433 shutdown_connections 2024-04-26T23:05:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.287+0000 7fcd9be00640 1 -- 172.21.15.40:0/2720836433 wait complete. 2024-04-26T23:05:38.306 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:05:38.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.512+0000 7faed7e00640 1 Processor -- start 2024-04-26T23:05:38.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.513+0000 7faed7e00640 1 -- start start 2024-04-26T23:05:38.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.514+0000 7faed7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed80b6220 0x7faed80d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:38.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.514+0000 7faed7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faed80d08d0 0x7faed80d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:38.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.514+0000 7faed7e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faed8162c40 con 0x7faed80d3820 2024-04-26T23:05:38.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.514+0000 7faed7e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7faed80d2d80 con 0x7faed80d08d0 2024-04-26T23:05:38.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.514+0000 7faed7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faed80b7c80 con 0x7faed80b6220 2024-04-26T23:05:38.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.514+0000 7faecfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faed80d08d0 0x7faed80d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:38.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.515+0000 7faecfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faed80d08d0 0x7faed80d32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46286/0 (socket says 172.21.15.40:46286) 2024-04-26T23:05:38.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.515+0000 7faecfe00640 1 -- 172.21.15.40:0/3348252718 learned_addr learned my addr 172.21.15.40:0/3348252718 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.515+0000 7faed6a00640 1 --2- 172.21.15.40:0/3348252718 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed80b6220 0x7faed80d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.515+0000 7faecfe00640 1 -- 172.21.15.40:0/3348252718 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed80b6220 msgr2=0x7faed80d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.515+0000 7faecfe00640 1 --2- 172.21.15.40:0/3348252718 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed80b6220 0x7faed80d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.515+0000 7faecfe00640 1 -- 172.21.15.40:0/3348252718 >> v1:172.21.15.40:6789/0 conn(0x7faed80d3820 legacy=0x7faed80b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.515+0000 7faecfe00640 1 -- 172.21.15.40:0/3348252718 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed80b7f60 con 0x7faed80d08d0 2024-04-26T23:05:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.515+0000 7faed6a00640 1 --2- 172.21.15.40:0/3348252718 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed80b6220 0x7faed80d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.516+0000 7faecfe00640 1 --2- 172.21.15.40:0/3348252718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed80d08d0 0x7faed80d32e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7faec0009870 tx=0x7faec002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc4fd973cb5e05e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:38.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.516+0000 7faed6000640 1 -- 172.21.15.40:0/3348252718 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec0037070 con 0x7faed80d08d0 2024-04-26T23:05:38.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.516+0000 7faed6000640 1 -- 172.21.15.40:0/3348252718 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec0032440 con 0x7faed80d08d0 2024-04-26T23:05:38.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.516+0000 7faed6000640 1 -- 172.21.15.40:0/3348252718 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec0032810 con 0x7faed80d08d0 2024-04-26T23:05:38.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.517+0000 7faed7e00640 1 -- 172.21.15.40:0/3348252718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed80d08d0 msgr2=0x7faed80d32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:38.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.517+0000 7faed7e00640 1 --2- 172.21.15.40:0/3348252718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed80d08d0 0x7faed80d32e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7faec0009870 tx=0x7faec002a250 comp rx=0 tx=0).stop 2024-04-26T23:05:38.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.517+0000 7faed7e00640 1 -- 172.21.15.40:0/3348252718 shutdown_connections 2024-04-26T23:05:38.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.517+0000 7faed7e00640 1 --2- 172.21.15.40:0/3348252718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed80d08d0 0x7faed80d32e0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.517+0000 7faed7e00640 1 --2- 172.21.15.40:0/3348252718 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed80b6220 0x7faed80d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.517+0000 7faed7e00640 1 -- 172.21.15.40:0/3348252718 >> 172.21.15.40:0/3348252718 conn(0x7faed80d4d90 msgr2=0x7faed80d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:38.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.517+0000 7faed7e00640 1 -- 172.21.15.40:0/3348252718 shutdown_connections 2024-04-26T23:05:38.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.517+0000 7faed7e00640 1 -- 172.21.15.40:0/3348252718 wait complete. 2024-04-26T23:05:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.519+0000 7faed7e00640 1 Processor -- start 2024-04-26T23:05:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.519+0000 7faed7e00640 1 -- start start 2024-04-26T23:05:38.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.520+0000 7faed7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed80b6220 0x7faed80cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:38.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.520+0000 7faed7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed80d08d0 0x7faed815c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:38.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.520+0000 7faed7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed80d3820 0x7faed80cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:38.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.520+0000 7faed7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faed80b9d30 con 0x7faed80d08d0 2024-04-26T23:05:38.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.520+0000 7faed7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faed80b8520 con 0x7faed80b6220 2024-04-26T23:05:38.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.520+0000 7faed7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faed80b9eb0 con 0x7faed80d3820 2024-04-26T23:05:38.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.520+0000 7faecfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed80d08d0 0x7faed815c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:38.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.521+0000 7faecfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed80d08d0 0x7faed815c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46292/0 (socket says 172.21.15.40:46292) 2024-04-26T23:05:38.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.521+0000 7faecfe00640 1 -- 172.21.15.40:0/1519391644 learned_addr learned my addr 172.21.15.40:0/1519391644 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:38.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.521+0000 7faed7400640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed80d3820 0x7faed80cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:38.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.521+0000 7faed6a00640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed80b6220 0x7faed80cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:38.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.521+0000 7faecfe00640 1 -- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed80d3820 msgr2=0x7faed80cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.521+0000 7faecfe00640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed80d3820 0x7faed80cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.521+0000 7faecfe00640 1 -- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed80b6220 msgr2=0x7faed80cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.521+0000 7faecfe00640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed80b6220 0x7faed80cad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.521+0000 7faecfe00640 1 -- 172.21.15.40:0/1519391644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed815f100 con 0x7faed80d08d0 2024-04-26T23:05:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.521+0000 7faed7400640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed80d3820 0x7faed80cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.521+0000 7faecfe00640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed80d08d0 0x7faed815c210 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7faec002a290 tx=0x7faec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.522+0000 7faecea00640 1 -- 172.21.15.40:0/1519391644 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec0037040 con 0x7faed80d08d0 2024-04-26T23:05:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.522+0000 7faecea00640 1 -- 172.21.15.40:0/1519391644 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec0042070 con 0x7faed80d08d0 2024-04-26T23:05:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.522+0000 7faecea00640 1 -- 172.21.15.40:0/1519391644 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec0030bd0 con 0x7faed80d08d0 2024-04-26T23:05:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.522+0000 7faed7e00640 1 -- 172.21.15.40:0/1519391644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed815e340 con 0x7faed80d08d0 2024-04-26T23:05:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.522+0000 7faed7e00640 1 -- 172.21.15.40:0/1519391644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed815e900 con 0x7faed80d08d0 2024-04-26T23:05:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.523+0000 7faed7e00640 1 -- 172.21.15.40:0/1519391644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed8069370 con 0x7faed80d08d0 2024-04-26T23:05:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.524+0000 7faecea00640 1 -- 172.21.15.40:0/1519391644 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faec0047460 con 0x7faed80d08d0 2024-04-26T23:05:38.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.525+0000 7faecea00640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeb406c8f0 0x7faeb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:38.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.525+0000 7faecea00640 1 -- 172.21.15.40:0/1519391644 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7faec00ac960 con 0x7faed80d08d0 2024-04-26T23:05:38.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.526+0000 7faed6a00640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeb406c8f0 0x7faeb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:38.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.527+0000 7faed6a00640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeb406c8f0 0x7faeb406edb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7faec800b840 tx=0x7faec8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.529+0000 7faecea00640 1 -- 172.21.15.40:0/1519391644 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7faec007abf0 con 0x7faed80d08d0 2024-04-26T23:05:38.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.778+0000 7faed7e00640 1 -- 172.21.15.40:0/1519391644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faed806e990 con 0x7faed80d08d0 2024-04-26T23:05:38.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.784+0000 7faecea00640 1 -- 172.21.15.40:0/1519391644 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5185 (secure 0 0 0) 0x7faec007a4f0 con 0x7faed80d08d0 2024-04-26T23:05:38.785 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:38.785 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":55,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/4194140680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4194140680},{"type":"v1","addr":"172.21.15.103:6836","nonce":4194140680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":5324,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2595375825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2595375825},{"type":"v1","addr":"172.21.15.103:6837","nonce":2595375825}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":5614,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/2117861921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2117861921},{"type":"v1","addr":"172.21.15.40:6834","nonce":2117861921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:05:32.212319+0000","modified":"2024-04-26T23:05:35.281037+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/931483025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":931483025},{"type":"v1","addr":"172.21.15.40:6835","nonce":931483025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6774,"qdb_cluster":[6774]},"id":4}]} 2024-04-26T23:05:38.785 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 55 2024-04-26T23:05:38.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.787+0000 7faed7e00640 1 -- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeb406c8f0 msgr2=0x7faeb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:38.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.787+0000 7faed7e00640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeb406c8f0 0x7faeb406edb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7faec800b840 tx=0x7faec8008040 comp rx=0 tx=0).stop 2024-04-26T23:05:38.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.787+0000 7faed7e00640 1 -- 172.21.15.40:0/1519391644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed80d08d0 msgr2=0x7faed815c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:38.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.787+0000 7faed7e00640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed80d08d0 0x7faed815c210 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7faec002a290 tx=0x7faec00046c0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.788+0000 7faed7e00640 1 -- 172.21.15.40:0/1519391644 shutdown_connections 2024-04-26T23:05:38.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.788+0000 7faed7e00640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeb406c8f0 0x7faeb406edb0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.788+0000 7faed7e00640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed80d3820 0x7faed80cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.788+0000 7faed7e00640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed80d08d0 0x7faed815c210 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.788+0000 7faed7e00640 1 --2- 172.21.15.40:0/1519391644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed80b6220 0x7faed80cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:38.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.788+0000 7faed7e00640 1 -- 172.21.15.40:0/1519391644 >> 172.21.15.40:0/1519391644 conn(0x7faed80d4d90 msgr2=0x7faed806d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:38.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.788+0000 7faed7e00640 1 -- 172.21.15.40:0/1519391644 shutdown_connections 2024-04-26T23:05:38.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:38.788+0000 7faed7e00640 1 -- 172.21.15.40:0/1519391644 wait complete. 2024-04-26T23:05:39.209 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:05:32.212319+0000', 'modified': '2024-04-26T23:05:35.281037+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6774}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6774': {'gid': 6774, 'name': 'c', 'rank': 0, 'incarnation': 51, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/931483025', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 931483025}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 931483025}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6774, 'qdb_cluster': [6774]} 2024-04-26T23:05:39.210 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-26T23:05:39.210 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T23:05:39.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.433+0000 7fc403e00640 1 Processor -- start 2024-04-26T23:05:39.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.434+0000 7fc403e00640 1 -- start start 2024-04-26T23:05:39.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.435+0000 7fc403e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc40415ce60 0x7fc4040afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.435+0000 7fc403e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc404071fe0 0x7fc404073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.435+0000 7fc403e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4040d0820 con 0x7fc4040af7a0 2024-04-26T23:05:39.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.435+0000 7fc403e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc4040cc450 con 0x7fc40415ce60 2024-04-26T23:05:39.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.435+0000 7fc403e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc4041614c0 con 0x7fc404071fe0 2024-04-26T23:05:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.436+0000 7fc402a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc40415ce60 0x7fc4040afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.436+0000 7fc402000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc4040af7a0 0x7fc4040ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41194/0 (socket says 172.21.15.40:41194) 2024-04-26T23:05:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.436+0000 7fc402000640 1 -- 172.21.15.40:0/1681059109 learned_addr learned my addr 172.21.15.40:0/1681059109 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.436+0000 7fc403400640 1 --2- 172.21.15.40:0/1681059109 >> v2:172.21.15.103:3300/0 conn(0x7fc404071fe0 0x7fc404073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.436+0000 7fc402a00640 1 -- 172.21.15.40:0/1681059109 >> v2:172.21.15.103:3300/0 conn(0x7fc404071fe0 msgr2=0x7fc404073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.436+0000 7fc402a00640 1 --2- 172.21.15.40:0/1681059109 >> v2:172.21.15.103:3300/0 conn(0x7fc404071fe0 0x7fc404073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.436+0000 7fc402a00640 1 -- 172.21.15.40:0/1681059109 >> v1:172.21.15.40:6789/0 conn(0x7fc4040af7a0 legacy=0x7fc4040ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.436+0000 7fc402a00640 1 -- 172.21.15.40:0/1681059109 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc404161760 con 0x7fc40415ce60 2024-04-26T23:05:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.436+0000 7fc401600640 1 -- 172.21.15.40:0/1681059109 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1027108053 0 0) 0x7fc4040d0820 con 0x7fc4040af7a0 2024-04-26T23:05:39.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.438+0000 7fc402a00640 1 --2- 172.21.15.40:0/1681059109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc40415ce60 0x7fc4040afc60 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc4040d1af0 tx=0x7fc3f402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc71bfbeae02f8b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:39.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.438+0000 7fc401600640 1 -- 172.21.15.40:0/1681059109 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4037070 con 0x7fc40415ce60 2024-04-26T23:05:39.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.438+0000 7fc401600640 1 -- 172.21.15.40:0/1681059109 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3f4032440 con 0x7fc40415ce60 2024-04-26T23:05:39.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.438+0000 7fc401600640 1 -- 172.21.15.40:0/1681059109 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f402ae60 con 0x7fc40415ce60 2024-04-26T23:05:39.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.438+0000 7fc403e00640 1 -- 172.21.15.40:0/1681059109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc40415ce60 msgr2=0x7fc4040afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.438+0000 7fc403e00640 1 --2- 172.21.15.40:0/1681059109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc40415ce60 0x7fc4040afc60 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc4040d1af0 tx=0x7fc3f402a110 comp rx=0 tx=0).stop 2024-04-26T23:05:39.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.439+0000 7fc403e00640 1 -- 172.21.15.40:0/1681059109 shutdown_connections 2024-04-26T23:05:39.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.439+0000 7fc403e00640 1 --2- 172.21.15.40:0/1681059109 >> v2:172.21.15.103:3300/0 conn(0x7fc404071fe0 0x7fc404073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.439+0000 7fc403e00640 1 --2- 172.21.15.40:0/1681059109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc40415ce60 0x7fc4040afc60 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.439+0000 7fc403e00640 1 -- 172.21.15.40:0/1681059109 >> 172.21.15.40:0/1681059109 conn(0x7fc4040d0a10 msgr2=0x7fc4040d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:39.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.439+0000 7fc403e00640 1 -- 172.21.15.40:0/1681059109 shutdown_connections 2024-04-26T23:05:39.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.439+0000 7fc403e00640 1 -- 172.21.15.40:0/1681059109 wait complete. 2024-04-26T23:05:39.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.440+0000 7fc403e00640 1 Processor -- start 2024-04-26T23:05:39.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.441+0000 7fc403e00640 1 -- start start 2024-04-26T23:05:39.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.441+0000 7fc403e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc404071fe0 0x7fc404151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc403e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4040af7a0 0x7fc4040c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc403e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc40415ce60 0x7fc4040b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc403e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc4040cc1f0 con 0x7fc4040af7a0 2024-04-26T23:05:39.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc403e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc4040cc070 con 0x7fc404071fe0 2024-04-26T23:05:39.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc403e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc404073ca0 con 0x7fc40415ce60 2024-04-26T23:05:39.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc402000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4040af7a0 0x7fc4040c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc402000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4040af7a0 0x7fc4040c9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46316/0 (socket says 172.21.15.40:46316) 2024-04-26T23:05:39.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc402000640 1 -- 172.21.15.40:0/801402117 learned_addr learned my addr 172.21.15.40:0/801402117 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:39.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc402a00640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc404071fe0 0x7fc404151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc403400640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc40415ce60 0x7fc4040b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc402000640 1 -- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc40415ce60 msgr2=0x7fc4040b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc402000640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc40415ce60 0x7fc4040b05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc402000640 1 -- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc404071fe0 msgr2=0x7fc404151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc402000640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc404071fe0 0x7fc404151280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc402000640 1 -- 172.21.15.40:0/801402117 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4040c9d60 con 0x7fc4040af7a0 2024-04-26T23:05:39.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.442+0000 7fc402a00640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc404071fe0 0x7fc404151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:39.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.443+0000 7fc402000640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4040af7a0 0x7fc4040c9460 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00c890 tx=0x7fc3ec00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:39.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.443+0000 7fc3faa00640 1 -- 172.21.15.40:0/801402117 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec018070 con 0x7fc4040af7a0 2024-04-26T23:05:39.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.443+0000 7fc3faa00640 1 -- 172.21.15.40:0/801402117 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3ec00a640 con 0x7fc4040af7a0 2024-04-26T23:05:39.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.443+0000 7fc3faa00640 1 -- 172.21.15.40:0/801402117 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec00aa30 con 0x7fc4040af7a0 2024-04-26T23:05:39.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.443+0000 7fc403e00640 1 -- 172.21.15.40:0/801402117 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4040ca040 con 0x7fc4040af7a0 2024-04-26T23:05:39.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.443+0000 7fc403e00640 1 -- 172.21.15.40:0/801402117 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4040ca600 con 0x7fc4040af7a0 2024-04-26T23:05:39.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.445+0000 7fc3faa00640 1 -- 172.21.15.40:0/801402117 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc3ec00abd0 con 0x7fc4040af7a0 2024-04-26T23:05:39.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.446+0000 7fc3faa00640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc3e006c8f0 0x7fc3e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.446+0000 7fc3faa00640 1 -- 172.21.15.40:0/801402117 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fc3ec08b1f0 con 0x7fc4040af7a0 2024-04-26T23:05:39.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.447+0000 7fc402a00640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc3e006c8f0 0x7fc3e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.447+0000 7fc40c62b740 1 Processor -- start 2024-04-26T23:05:39.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.447+0000 7fc40c62b740 1 -- start start 2024-04-26T23:05:39.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.448+0000 7fc40c62b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587737b0 0x564b58775910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.448+0000 7fc40c62b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587afe80 0x564b58773b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.448+0000 7fc40c62b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b5879e270 0x564b5879e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.448+0000 7fc40c62b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564b586d7600 con 0x564b587737b0 2024-04-26T23:05:39.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.448+0000 7fc40c62b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564b58674100 con 0x564b587afe80 2024-04-26T23:05:39.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.448+0000 7fc40c62b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564b586d35c0 con 0x564b5879e270 2024-04-26T23:05:39.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.448+0000 7fc402a00640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc3e006c8f0 0x7fc3e006edb0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4034000 tx=0x7fc3f402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:39.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.449+0000 7fc402a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587737b0 0x564b58775910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.449+0000 7fc402a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587737b0 0x564b58775910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46324/0 (socket says 172.21.15.40:46324) 2024-04-26T23:05:39.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.449+0000 7fc402a00640 1 -- 172.21.15.40:0/790653402 learned_addr learned my addr 172.21.15.40:0/790653402 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:39.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.449+0000 7fc402000640 1 --2- 172.21.15.40:0/790653402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587afe80 0x564b58773b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.449+0000 7fc403400640 1 --2- 172.21.15.40:0/790653402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b5879e270 0x564b5879e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.449+0000 7fc402a00640 1 -- 172.21.15.40:0/790653402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b5879e270 msgr2=0x564b5879e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:39.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.449+0000 7fc402a00640 1 --2- 172.21.15.40:0/790653402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b5879e270 0x564b5879e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.449+0000 7fc402a00640 1 -- 172.21.15.40:0/790653402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587afe80 msgr2=0x564b58773b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:39.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.449+0000 7fc402a00640 1 --2- 172.21.15.40:0/790653402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587afe80 0x564b58773b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.449+0000 7fc402a00640 1 -- 172.21.15.40:0/790653402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b583e92d0 con 0x564b587737b0 2024-04-26T23:05:39.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.449+0000 7fc402000640 1 --2- 172.21.15.40:0/790653402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587afe80 0x564b58773b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:39.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.449+0000 7fc402a00640 1 --2- 172.21.15.40:0/790653402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587737b0 0x564b58775910 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc3f402a150 tx=0x7fc3f402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1fed7b19c0d95b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:39.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.450+0000 7fc3f3400640 1 -- 172.21.15.40:0/790653402 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f404e070 con 0x564b587737b0 2024-04-26T23:05:39.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.450+0000 7fc3f3400640 1 -- 172.21.15.40:0/790653402 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3f4032d90 con 0x564b587737b0 2024-04-26T23:05:39.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.450+0000 7fc3f3400640 1 -- 172.21.15.40:0/790653402 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4037040 con 0x564b587737b0 2024-04-26T23:05:39.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.450+0000 7fc40c62b740 1 -- 172.21.15.40:0/790653402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587737b0 msgr2=0x564b58775910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.450+0000 7fc40c62b740 1 --2- 172.21.15.40:0/790653402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587737b0 0x564b58775910 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc3f402a150 tx=0x7fc3f402c000 comp rx=0 tx=0).stop 2024-04-26T23:05:39.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.450+0000 7fc40c62b740 1 -- 172.21.15.40:0/790653402 shutdown_connections 2024-04-26T23:05:39.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.450+0000 7fc40c62b740 1 --2- 172.21.15.40:0/790653402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b5879e270 0x564b5879e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.450+0000 7fc40c62b740 1 --2- 172.21.15.40:0/790653402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587afe80 0x564b58773b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.450+0000 7fc40c62b740 1 --2- 172.21.15.40:0/790653402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587737b0 0x564b58775910 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.450+0000 7fc40c62b740 1 -- 172.21.15.40:0/790653402 >> 172.21.15.40:0/790653402 conn(0x564b58707d10 msgr2=0x564b587b9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:39.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.450+0000 7fc40c62b740 1 -- 172.21.15.40:0/790653402 shutdown_connections 2024-04-26T23:05:39.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.451+0000 7fc40c62b740 1 -- 172.21.15.40:0/790653402 wait complete. 2024-04-26T23:05:39.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.451+0000 7fc40c62b740 1 -- start start 2024-04-26T23:05:39.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.451+0000 7fc40c62b740 1 Processor -- start 2024-04-26T23:05:39.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.452+0000 7fc40c62b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 0x564b58790b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.452+0000 7fc40c62b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 0x564b5879f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.452+0000 7fc40c62b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b587afe80 0x564b5879fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.452+0000 7fc40c62b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564b5865ef70 con 0x564b5879e270 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.452+0000 7fc40c62b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564b5879bdc0 con 0x564b587737b0 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.452+0000 7fc40c62b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564b58791070 con 0x564b587afe80 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.452+0000 7fc40c62b740 10 client.0 _ll_register_callbacks cb 0x564b586cd480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.453+0000 7fc402000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 0x564b5879f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.453+0000 7fc402000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 0x564b5879f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46336/0 (socket says 172.21.15.40:46336) 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.453+0000 7fc402000640 1 -- 172.21.15.40:0/14508208 learned_addr learned my addr 172.21.15.40:0/14508208 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.453+0000 7fc402a00640 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 0x564b58790b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.453+0000 7fc403400640 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b587afe80 0x564b5879fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.453+0000 7fc402000640 1 -- 172.21.15.40:0/14508208 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b587afe80 msgr2=0x564b5879fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.453+0000 7fc402000640 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b587afe80 0x564b5879fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.453+0000 7fc402000640 1 -- 172.21.15.40:0/14508208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 msgr2=0x564b58790b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.453+0000 7fc402000640 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 0x564b58790b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.453+0000 7fc402000640 1 -- 172.21.15.40:0/14508208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b584b36b0 con 0x564b5879e270 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.453+0000 7fc403400640 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b587afe80 0x564b5879fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:39.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.453+0000 7fc402000640 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 0x564b5879f4e0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec089620 tx=0x7fc3ec059520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f3e9280ad19520 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.453+0000 7fc3ebe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc3ebe00640 1 -- 172.21.15.40:0/14508208 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec0713d0 con 0x564b5879e270 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc3ebe00640 1 -- 172.21.15.40:0/14508208 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fc3ec058270 con 0x564b5879e270 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc40c62b740 1 -- 172.21.15.40:0/14508208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3f4032d90 con 0x564b5879e270 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc3ebe00640 1 -- 172.21.15.40:0/14508208 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fc3ec089050 con 0x564b5879e270 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc40c62b740 10 client.6921 fetch_fsmap learned FSMap version 55 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc40c62b740 1 -- 172.21.15.40:0/14508208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) -- 0x564b587d0f00 con 0x564b5879e270 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc3ebe00640 1 -- 172.21.15.40:0/14508208 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3f4032d90 con 0x564b5879e270 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc3ebe00640 1 -- 172.21.15.40:0/14508208 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 55) ==== 3494+0+0 (secure 0 0 0) 0x7fc3ec05f3d0 con 0x564b5879e270 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc40c62b740 10 client.6921 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc40c62b740 10 client.6921 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc40c62b740 20 client.6921 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc40c62b740 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x564b587ba3a0 0x564b587ba780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc40c62b740 4 client.6921 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:05:39.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.454+0000 7fc40c62b740 1 -- 172.21.15.40:0/14508208 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b586e5210 con 0x564b587ba3a0 2024-04-26T23:05:39.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.455+0000 7fc402a00640 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x564b587ba3a0 0x564b587ba780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.455+0000 7fc402a00640 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x564b587ba3a0 0x564b587ba780 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c156f65d634266a server_cookie=bfd12d54fcac6b84 in_seq=0 out_seq=0 2024-04-26T23:05:39.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.455+0000 7fc3ebe00640 10 client.6921 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:05:39.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.456+0000 7fc3ebe00640 1 -- 172.21.15.40:0/14508208 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564b586e5210 con 0x564b587ba3a0 2024-04-26T23:05:39.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.456+0000 7fc3ebe00640 10 client.6921 handle_command_reply: tid=0 2024-04-26T23:05:39.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.456+0000 7fc40c62b740 1 client.6921 shutdown 2024-04-26T23:05:39.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.456+0000 7fc40c62b740 1 -- 172.21.15.40:0/14508208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 msgr2=0x564b5879f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.456+0000 7fc40c62b740 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 0x564b5879f4e0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec089620 tx=0x7fc3ec059520 comp rx=0 tx=0).stop 2024-04-26T23:05:39.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.457+0000 7fc40c62b740 1 -- 172.21.15.40:0/14508208 shutdown_connections 2024-04-26T23:05:39.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.457+0000 7fc40c62b740 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x564b587ba3a0 0x564b587ba780 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.457+0000 7fc40c62b740 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b587afe80 0x564b5879fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.457+0000 7fc40c62b740 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 0x564b5879f4e0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.457+0000 7fc40c62b740 1 --2- 172.21.15.40:0/14508208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 0x564b58790b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.457+0000 7fc3ebe00640 0 client.6921 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:05:39.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.457+0000 7fc40c62b740 1 -- 172.21.15.40:0/14508208 >> 172.21.15.40:0/14508208 conn(0x564b58707d10 msgr2=0x564b587b9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:39.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.457+0000 7fc40c62b740 1 -- 172.21.15.40:0/14508208 shutdown_connections 2024-04-26T23:05:39.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.457+0000 7fc40c62b740 1 -- 172.21.15.40:0/14508208 wait complete. 2024-04-26T23:05:39.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.457+0000 7fc40c62b740 20 client.6921 trim_cache size 0 max 16384 2024-04-26T23:05:39.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.477+0000 7fc40c62b740 1 Processor -- start 2024-04-26T23:05:39.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.477+0000 7fc40c62b740 1 -- start start 2024-04-26T23:05:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.477+0000 7fc40c62b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 0x564b587bc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.477+0000 7fc40c62b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b5879e270 0x564b587aec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.478+0000 7fc40c62b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587afe80 0x564b587af140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.478+0000 7fc40c62b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc4040c7e20 con 0x564b587afe80 2024-04-26T23:05:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.478+0000 7fc40c62b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc404159dc0 con 0x564b587737b0 2024-04-26T23:05:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.478+0000 7fc40c62b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc4040c77d0 con 0x564b5879e270 2024-04-26T23:05:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.478+0000 7fc403400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587afe80 0x564b587af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.478+0000 7fc403400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587afe80 0x564b587af140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46350/0 (socket says 172.21.15.40:46350) 2024-04-26T23:05:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.478+0000 7fc403400640 1 -- 172.21.15.40:0/217016113 learned_addr learned my addr 172.21.15.40:0/217016113 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.478+0000 7fc403400640 1 -- 172.21.15.40:0/217016113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b5879e270 msgr2=0x564b587aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.478+0000 7fc402a00640 1 --2- 172.21.15.40:0/217016113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 0x564b587bc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.478+0000 7fc403400640 1 --2- 172.21.15.40:0/217016113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b5879e270 0x564b587aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.478+0000 7fc403400640 1 -- 172.21.15.40:0/217016113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 msgr2=0x564b587bc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.478+0000 7fc403400640 1 --2- 172.21.15.40:0/217016113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 0x564b587bc980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.478+0000 7fc403400640 1 -- 172.21.15.40:0/217016113 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b5868c570 con 0x564b587afe80 2024-04-26T23:05:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.479+0000 7fc403400640 1 --2- 172.21.15.40:0/217016113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587afe80 0x564b587af140 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc006fd0 tx=0x7fc3fc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9428850b8fce3eb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.479+0000 7fc3f2a00640 1 -- 172.21.15.40:0/217016113 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc010a40 con 0x564b587afe80 2024-04-26T23:05:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.479+0000 7fc3f2a00640 1 -- 172.21.15.40:0/217016113 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3fc010c20 con 0x564b587afe80 2024-04-26T23:05:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.479+0000 7fc3f2a00640 1 -- 172.21.15.40:0/217016113 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc004020 con 0x564b587afe80 2024-04-26T23:05:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.479+0000 7fc40c62b740 1 -- 172.21.15.40:0/217016113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587afe80 msgr2=0x564b587af140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.479+0000 7fc40c62b740 1 --2- 172.21.15.40:0/217016113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587afe80 0x564b587af140 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc006fd0 tx=0x7fc3fc00ed00 comp rx=0 tx=0).stop 2024-04-26T23:05:39.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.479+0000 7fc40c62b740 1 -- 172.21.15.40:0/217016113 shutdown_connections 2024-04-26T23:05:39.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.479+0000 7fc40c62b740 1 --2- 172.21.15.40:0/217016113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b587afe80 0x564b587af140 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.479+0000 7fc40c62b740 1 --2- 172.21.15.40:0/217016113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b5879e270 0x564b587aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.480+0000 7fc40c62b740 1 --2- 172.21.15.40:0/217016113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 0x564b587bc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.480+0000 7fc40c62b740 1 -- 172.21.15.40:0/217016113 >> 172.21.15.40:0/217016113 conn(0x564b58707d10 msgr2=0x564b58774690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:39.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.480+0000 7fc40c62b740 1 -- 172.21.15.40:0/217016113 shutdown_connections 2024-04-26T23:05:39.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.480+0000 7fc40c62b740 1 -- 172.21.15.40:0/217016113 wait complete. 2024-04-26T23:05:39.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.480+0000 7fc40c62b740 1 -- start start 2024-04-26T23:05:39.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.480+0000 7fc40c62b740 1 Processor -- start 2024-04-26T23:05:39.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.481+0000 7fc40c62b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 0x564b58835060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.481+0000 7fc40c62b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 0x564b587cb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.481+0000 7fc40c62b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b587afe80 0x564b587cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.481+0000 7fc40c62b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564b587b24c0 con 0x564b5879e270 2024-04-26T23:05:39.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.481+0000 7fc40c62b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564b587b2070 con 0x564b587737b0 2024-04-26T23:05:39.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.481+0000 7fc40c62b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564b587cd560 con 0x564b587afe80 2024-04-26T23:05:39.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.482+0000 7fc40c62b740 10 client.0 _ll_register_callbacks cb 0x7fc3ec0550c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:39.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.482+0000 7fc402000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 0x564b587cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.482+0000 7fc402000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 0x564b587cb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46362/0 (socket says 172.21.15.40:46362) 2024-04-26T23:05:39.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.482+0000 7fc402000640 1 -- 172.21.15.40:0/3020230092 learned_addr learned my addr 172.21.15.40:0/3020230092 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:39.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.482+0000 7fc402a00640 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 0x564b58835060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.482+0000 7fc403400640 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b587afe80 0x564b587cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.482+0000 7fc402000640 1 -- 172.21.15.40:0/3020230092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b587afe80 msgr2=0x564b587cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.482+0000 7fc402000640 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b587afe80 0x564b587cbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.482+0000 7fc402000640 1 -- 172.21.15.40:0/3020230092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 msgr2=0x564b58835060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.482+0000 7fc402000640 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 0x564b58835060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.482+0000 7fc402000640 1 -- 172.21.15.40:0/3020230092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b5878f9e0 con 0x564b5879e270 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.482+0000 7fc402a00640 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 0x564b58835060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc402000640 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 0x564b587cb820 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec058fe0 tx=0x7fc3ec004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b77d4ac427923c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc3ebe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc3ebe00640 1 -- 172.21.15.40:0/3020230092 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec058780 con 0x564b5879e270 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc3ebe00640 1 -- 172.21.15.40:0/3020230092 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fc3ec078370 con 0x564b5879e270 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc40c62b740 1 -- 172.21.15.40:0/3020230092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3fc010c20 con 0x564b5879e270 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc3ebe00640 1 -- 172.21.15.40:0/3020230092 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fc3ec07e050 con 0x564b5879e270 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc40c62b740 10 client.6927 fetch_fsmap learned FSMap version 55 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc40c62b740 1 -- 172.21.15.40:0/3020230092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) -- 0x564b58772f30 con 0x564b5879e270 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc3ebe00640 1 -- 172.21.15.40:0/3020230092 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3fc010c20 con 0x564b5879e270 2024-04-26T23:05:39.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc3ebe00640 1 -- 172.21.15.40:0/3020230092 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 55) ==== 3494+0+0 (secure 0 0 0) 0x7fc3ec081080 con 0x564b5879e270 2024-04-26T23:05:39.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc40c62b740 10 client.6927 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T23:05:39.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc40c62b740 10 client.6927 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:05:39.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc40c62b740 20 client.6927 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:39.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc40c62b740 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x564b58772a50 0x564b5883aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc40c62b740 4 client.6927 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:05:39.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.483+0000 7fc40c62b740 1 -- 172.21.15.40:0/3020230092 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "status"}) -- 0x564b586df080 con 0x564b58772a50 2024-04-26T23:05:39.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.484+0000 7fc402a00640 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x564b58772a50 0x564b5883aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.484+0000 7fc402a00640 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x564b58772a50 0x564b5883aff0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50beda4100de43c server_cookie=3e565e883757f033 in_seq=0 out_seq=0 2024-04-26T23:05:39.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.484+0000 7fc3ebe00640 10 client.6927 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:05:39.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.484+0000 7fc3ebe00640 1 -- 172.21.15.40:0/3020230092 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x564b586df080 con 0x564b58772a50 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.484+0000 7fc3ebe00640 10 client.6927 handle_command_reply: tid=0 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.484+0000 7fc40c62b740 1 client.6927 shutdown 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.485+0000 7fc40c62b740 1 -- 172.21.15.40:0/3020230092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 msgr2=0x564b587cb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.485+0000 7fc40c62b740 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 0x564b587cb820 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec058fe0 tx=0x7fc3ec004f90 comp rx=0 tx=0).stop 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.485+0000 7fc40c62b740 1 -- 172.21.15.40:0/3020230092 shutdown_connections 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.485+0000 7fc40c62b740 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x564b58772a50 0x564b5883aff0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.485+0000 7fc40c62b740 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564b587afe80 0x564b587cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.485+0000 7fc40c62b740 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b5879e270 0x564b587cb820 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.485+0000 7fc40c62b740 1 --2- 172.21.15.40:0/3020230092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b587737b0 0x564b58835060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.485+0000 7fc3ebe00640 0 client.6927 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.485+0000 7fc40c62b740 1 -- 172.21.15.40:0/3020230092 >> 172.21.15.40:0/3020230092 conn(0x564b58707d10 msgr2=0x564b587bc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.485+0000 7fc40c62b740 1 -- 172.21.15.40:0/3020230092 shutdown_connections 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.485+0000 7fc40c62b740 1 -- 172.21.15.40:0/3020230092 wait complete. 2024-04-26T23:05:39.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.485+0000 7fc40c62b740 20 client.6927 trim_cache size 0 max 16384 2024-04-26T23:05:39.489 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:05:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:05:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:05:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6774, 2024-04-26T23:05:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:05:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:05:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:05:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 6.7387760419999996, 2024-04-26T23:05:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 55, 2024-04-26T23:05:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 65, 2024-04-26T23:05:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 65, 2024-04-26T23:05:39.489 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 12.150305465000001 2024-04-26T23:05:39.490 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:05:39.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.486+0000 7fc403e00640 1 -- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc3e006c8f0 msgr2=0x7fc3e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.486+0000 7fc403e00640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc3e006c8f0 0x7fc3e006edb0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4034000 tx=0x7fc3f402e000 comp rx=0 tx=0).stop 2024-04-26T23:05:39.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.486+0000 7fc403e00640 1 -- 172.21.15.40:0/801402117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4040af7a0 msgr2=0x7fc4040c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.486+0000 7fc403e00640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4040af7a0 0x7fc4040c9460 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00c890 tx=0x7fc3ec00cd50 comp rx=0 tx=0).stop 2024-04-26T23:05:39.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.486+0000 7fc403e00640 1 -- 172.21.15.40:0/801402117 shutdown_connections 2024-04-26T23:05:39.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.486+0000 7fc403e00640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc3e006c8f0 0x7fc3e006edb0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.486+0000 7fc403e00640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc40415ce60 0x7fc4040b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.486+0000 7fc403e00640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4040af7a0 0x7fc4040c9460 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.486+0000 7fc403e00640 1 --2- 172.21.15.40:0/801402117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc404071fe0 0x7fc404151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.486+0000 7fc403e00640 1 -- 172.21.15.40:0/801402117 >> 172.21.15.40:0/801402117 conn(0x7fc4040d0a10 msgr2=0x7fc404160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:39.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.486+0000 7fc403e00640 1 -- 172.21.15.40:0/801402117 shutdown_connections 2024-04-26T23:05:39.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.486+0000 7fc403e00640 1 -- 172.21.15.40:0/801402117 wait complete. 2024-04-26T23:05:39.503 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-26T23:05:39.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.692+0000 7f6785600640 1 Processor -- start 2024-04-26T23:05:39.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.692+0000 7f6785600640 1 -- start start 2024-04-26T23:05:39.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.694+0000 7f6785600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6780073e10 0x7f67801600d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.694+0000 7f6785600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f67800b4f50 0x7f678015d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.694+0000 7f6785600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6780160610 con 0x7f678015c160 2024-04-26T23:05:39.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.694+0000 7f6785600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6780153040 con 0x7f67800b4f50 2024-04-26T23:05:39.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.694+0000 7f6785600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f678015fc40 con 0x7f6780073e10 2024-04-26T23:05:39.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.694+0000 7f6777e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f678015c160 0x7f67800b4840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41210/0 (socket says 172.21.15.40:41210) 2024-04-26T23:05:39.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.695+0000 7f6777e00640 1 -- 172.21.15.40:0/3066069247 learned_addr learned my addr 172.21.15.40:0/3066069247 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:39.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.695+0000 7f677fe00640 1 --2- 172.21.15.40:0/3066069247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6780073e10 0x7f67801600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.695+0000 7f6784c00640 1 --2- 172.21.15.40:0/3066069247 >> v2:172.21.15.103:3300/0 conn(0x7f67800b4f50 0x7f678015d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.695+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4062891817 0 0) 0x7f6780160610 con 0x7f678015c160 2024-04-26T23:05:39.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.695+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6764005730 con 0x7f678015c160 2024-04-26T23:05:39.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.695+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1264563357 0 0) 0x7f6764005730 con 0x7f678015c160 2024-04-26T23:05:39.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.695+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6780073e10 msgr2=0x7f67801600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:39.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.695+0000 7f677f400640 1 --2- 172.21.15.40:0/3066069247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6780073e10 0x7f67801600d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.695+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 >> v2:172.21.15.103:3300/0 conn(0x7f67800b4f50 msgr2=0x7f678015d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:39.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.695+0000 7f677f400640 1 --2- 172.21.15.40:0/3066069247 >> v2:172.21.15.103:3300/0 conn(0x7f67800b4f50 0x7f678015d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f678015cb90 con 0x7f678015c160 2024-04-26T23:05:39.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f6768001690 con 0x7f678015c160 2024-04-26T23:05:39.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 >> v1:172.21.15.40:6789/0 conn(0x7f678015c160 legacy=0x7f67800b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f677f400640 1 --2- 172.21.15.40:0/3066069247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6780073e10 0x7f6764005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f677f400640 1 --2- 172.21.15.40:0/3066069247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6764006140 0x7f6764006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f677f400640 1 --2- 172.21.15.40:0/3066069247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6764006f50 0x7f67640073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6780153040 con 0x7f6764006f50 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f678015fc40 con 0x7f6764006140 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f676400dbd0 con 0x7f6780073e10 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f6784c00640 1 --2- 172.21.15.40:0/3066069247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6764006f50 0x7f67640073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f6777e00640 1 --2- 172.21.15.40:0/3066069247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6764006140 0x7f6764006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f677fe00640 1 --2- 172.21.15.40:0/3066069247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6780073e10 0x7f6764005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f6784c00640 1 -- 172.21.15.40:0/3066069247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6780073e10 msgr2=0x7f6764005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f6784c00640 1 --2- 172.21.15.40:0/3066069247 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6780073e10 0x7f6764005bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f6784c00640 1 -- 172.21.15.40:0/3066069247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6764006140 msgr2=0x7f6764006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f6784c00640 1 --2- 172.21.15.40:0/3066069247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6764006140 0x7f6764006a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.696+0000 7f6784c00640 1 -- 172.21.15.40:0/3066069247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6764005730 con 0x7f6764006f50 2024-04-26T23:05:39.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.698+0000 7f6784c00640 1 --2- 172.21.15.40:0/3066069247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6764006f50 0x7f67640073d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f6778009f90 tx=0x7f6764007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882f3ed08acf3101 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:39.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.698+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6778009c10 con 0x7f6764006f50 2024-04-26T23:05:39.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.698+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6778036070 con 0x7f6764006f50 2024-04-26T23:05:39.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.698+0000 7f677f400640 1 -- 172.21.15.40:0/3066069247 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6778031670 con 0x7f6764006f50 2024-04-26T23:05:39.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.698+0000 7f6785600640 1 -- 172.21.15.40:0/3066069247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6764006f50 msgr2=0x7f67640073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.698+0000 7f6785600640 1 --2- 172.21.15.40:0/3066069247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6764006f50 0x7f67640073d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f6778009f90 tx=0x7f6764007a40 comp rx=0 tx=0).stop 2024-04-26T23:05:39.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.698+0000 7f6784c00640 1 -- 172.21.15.40:0/3066069247 reap_dead start 2024-04-26T23:05:39.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.699+0000 7f6785600640 1 -- 172.21.15.40:0/3066069247 shutdown_connections 2024-04-26T23:05:39.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.699+0000 7f6785600640 1 -- 172.21.15.40:0/3066069247 >> 172.21.15.40:0/3066069247 conn(0x7f6780075eb0 msgr2=0x7f6780068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:39.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.699+0000 7f6785600640 1 -- 172.21.15.40:0/3066069247 shutdown_connections 2024-04-26T23:05:39.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.699+0000 7f6785600640 1 -- 172.21.15.40:0/3066069247 wait complete. 2024-04-26T23:05:39.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.700+0000 7f6785600640 1 Processor -- start 2024-04-26T23:05:39.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.700+0000 7f6785600640 1 -- start start 2024-04-26T23:05:39.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.701+0000 7f6785600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6764006f50 0x7f67800d0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.701+0000 7f6785600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f678014b400 0x7f67800cf350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.701+0000 7f6785600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67801f8650 0x7f67801f8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.701+0000 7f6785600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f67800733c0 con 0x7f6764006f50 2024-04-26T23:05:39.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.701+0000 7f6785600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f678015f8d0 con 0x7f678014b400 2024-04-26T23:05:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.701+0000 7f6785600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6780073540 con 0x7f67801f8650 2024-04-26T23:05:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.702+0000 7f677fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6764006f50 0x7f67800d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.702+0000 7f677fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6764006f50 0x7f67800d0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46374/0 (socket says 172.21.15.40:46374) 2024-04-26T23:05:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.702+0000 7f677fe00640 1 -- 172.21.15.40:0/4262544477 learned_addr learned my addr 172.21.15.40:0/4262544477 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.702+0000 7f677fe00640 1 -- 172.21.15.40:0/4262544477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67801f8650 msgr2=0x7f67801f8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.702+0000 7f677fe00640 1 --2- 172.21.15.40:0/4262544477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67801f8650 0x7f67801f8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.702+0000 7f6777e00640 1 --2- 172.21.15.40:0/4262544477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f678014b400 0x7f67800cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.702+0000 7f677fe00640 1 -- 172.21.15.40:0/4262544477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f678014b400 msgr2=0x7f67800cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.702+0000 7f677fe00640 1 --2- 172.21.15.40:0/4262544477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f678014b400 0x7f67800cf350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.702+0000 7f677fe00640 1 -- 172.21.15.40:0/4262544477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67800cf890 con 0x7f6764006f50 2024-04-26T23:05:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.702+0000 7f677fe00640 1 --2- 172.21.15.40:0/4262544477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6764006f50 0x7f67800d0990 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f67700107e0 tx=0x7f6770010ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:39.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.703+0000 7f677cc00640 1 -- 172.21.15.40:0/4262544477 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677000f040 con 0x7f6764006f50 2024-04-26T23:05:39.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.703+0000 7f677cc00640 1 -- 172.21.15.40:0/4262544477 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f677000eb60 con 0x7f6764006f50 2024-04-26T23:05:39.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.703+0000 7f677cc00640 1 -- 172.21.15.40:0/4262544477 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677001c400 con 0x7f6764006f50 2024-04-26T23:05:39.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.703+0000 7f6785600640 1 -- 172.21.15.40:0/4262544477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6780158d30 con 0x7f6764006f50 2024-04-26T23:05:39.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.703+0000 7f6785600640 1 -- 172.21.15.40:0/4262544477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f678014aeb0 con 0x7f6764006f50 2024-04-26T23:05:39.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.704+0000 7f6785600640 1 -- 172.21.15.40:0/4262544477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67800d10e0 con 0x7f6764006f50 2024-04-26T23:05:39.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.705+0000 7f677cc00640 1 -- 172.21.15.40:0/4262544477 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f677001c5a0 con 0x7f6764006f50 2024-04-26T23:05:39.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.706+0000 7f677cc00640 1 --2- 172.21.15.40:0/4262544477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f675c06c8f0 0x7f675c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:39.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.706+0000 7f677cc00640 1 -- 172.21.15.40:0/4262544477 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f677008c2e0 con 0x7f6764006f50 2024-04-26T23:05:39.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.707+0000 7f6777e00640 1 --2- 172.21.15.40:0/4262544477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f675c06c8f0 0x7f675c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.708+0000 7f6777e00640 1 --2- 172.21.15.40:0/4262544477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f675c06c8f0 0x7f675c06edb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f67800b3ab0 tx=0x7f6768009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.710+0000 7f677cc00640 1 -- 172.21.15.40:0/4262544477 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f67800d10e0 con 0x7f6764006f50 2024-04-26T23:05:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.959+0000 7f6785600640 1 -- 172.21.15.40:0/4262544477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6780069370 con 0x7f6764006f50 2024-04-26T23:05:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.960+0000 7f677cc00640 1 -- 172.21.15.40:0/4262544477 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v55) ==== 53+0+73 (secure 0 0 0) 0x7f6770018030 con 0x7f6764006f50 2024-04-26T23:05:39.960 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-26T23:05:39.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.963+0000 7f6785600640 1 -- 172.21.15.40:0/4262544477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f675c06c8f0 msgr2=0x7f675c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.963+0000 7f6785600640 1 --2- 172.21.15.40:0/4262544477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f675c06c8f0 0x7f675c06edb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f67800b3ab0 tx=0x7f6768009040 comp rx=0 tx=0).stop 2024-04-26T23:05:39.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.963+0000 7f6785600640 1 -- 172.21.15.40:0/4262544477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6764006f50 msgr2=0x7f67800d0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:39.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.963+0000 7f6785600640 1 --2- 172.21.15.40:0/4262544477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6764006f50 0x7f67800d0990 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f67700107e0 tx=0x7f6770010ca0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.963+0000 7f6785600640 1 -- 172.21.15.40:0/4262544477 shutdown_connections 2024-04-26T23:05:39.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.963+0000 7f6785600640 1 --2- 172.21.15.40:0/4262544477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f675c06c8f0 0x7f675c06edb0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.963+0000 7f6785600640 1 --2- 172.21.15.40:0/4262544477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f67801f8650 0x7f67801f8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.963+0000 7f6785600640 1 --2- 172.21.15.40:0/4262544477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f678014b400 0x7f67800cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.963+0000 7f6785600640 1 --2- 172.21.15.40:0/4262544477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6764006f50 0x7f67800d0990 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:39.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.963+0000 7f6785600640 1 -- 172.21.15.40:0/4262544477 >> 172.21.15.40:0/4262544477 conn(0x7f6780075eb0 msgr2=0x7f678006de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:39.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.963+0000 7f6785600640 1 -- 172.21.15.40:0/4262544477 shutdown_connections 2024-04-26T23:05:39.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:39.963+0000 7f6785600640 1 -- 172.21.15.40:0/4262544477 wait complete. 2024-04-26T23:05:39.983 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-26T23:05:39.983 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:05:39.983 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-04-26T23:05:39.983 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T23:05:39.983 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T23:05:39.983 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-26T23:05:39.983 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-26T23:05:39.983 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:05:39.984 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:05:39.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:39.984 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-04-26T23:05:40.040 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:05:40.040 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:05:40.040 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:05:40.040 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:05:40.040 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-26T23:05:40.040 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:05:40.040 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:05:40.040 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:05:40.040 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-04-26T23:05:40.040 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:05:40.040 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42374sec preferred_lft 42374sec 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 8e:8e:79:d7:cf:5d brd ff:ff:ff:ff:ff:ff 2024-04-26T23:05:40.041 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-26T23:05:40.042 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:05:40.042 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::4ce7:9bff:fe51:b1/64 scope link 2024-04-26T23:05:40.042 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:05:40.042 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:05:40.042 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether be:f1:28:e1:99:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T23:05:40.042 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::bcf1:28ff:fee1:9921/64 scope link 2024-04-26T23:05:40.042 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:05:40.042 INFO:teuthology.orchestra.run.smithi040.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:05:40.042 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 8e:8e:79:d7:cf:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-26T23:05:40.042 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::8c8e:79ff:fed7:cf5d/64 scope link 2024-04-26T23:05:40.042 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:05:40.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:40.043 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:05:40.130 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-04-26T23:05:40.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:40.134 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-04-26T23:05:40.190 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-04-26T23:05:40.190 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:05:40.190 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T23:05:40.190 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T23:05:40.190 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:05:40.191 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:05:40.191 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:05:40.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T23:05:40.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T23:05:40.192 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T23:05:40.192 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:05:40.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:05:40.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:05:40.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:40.419 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:05:40.473 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T23:05:40.473 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-26T23:05:40.474 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:05:40.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:40.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-04-26T23:05:40.556 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:05:40.556 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T23:05:40.612 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:05:40.613 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:05:40.613 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:05:40.668 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-26T23:05:40.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:40.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:05:40.811 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:05:40.811 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:05:40.811 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:05:40.832 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:05:40.833 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:05:40.833 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:05:40.833 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:05:40.833 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:05:40.863 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:05:40.863 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:05:40.863 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:05:40.863 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:05:40.863 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:05:40.863 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-26T23:05:40.864 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-26T23:05:40.864 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=815936ab-d4ac-471e-b069-1a0dc70078ee" will pass to kernel 2024-04-26T23:05:40.867 INFO:teuthology.orchestra.run.smithi040.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T23:05:40.867 INFO:teuthology.orchestra.run.smithi040.stdout: You just mounted a file system that supports labels which does not 2024-04-26T23:05:40.867 INFO:teuthology.orchestra.run.smithi040.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T23:05:40.867 INFO:teuthology.orchestra.run.smithi040.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T23:05:40.867 INFO:teuthology.orchestra.run.smithi040.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T23:05:40.872 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T23:05:40.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:40.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:05:40.961 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:05:41.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:05:41.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:05:41.101 INFO:teuthology.orchestra.run.smithi040.stdout:client5372 2024-04-26T23:05:41.102 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:05:41.102 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:05:41.153 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-26T23:05:41.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.371+0000 7fdddd800640 1 Processor -- start 2024-04-26T23:05:41.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.371+0000 7fdddd800640 1 -- start start 2024-04-26T23:05:41.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.373+0000 7fdddd800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b5120 0x7fddd80b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:41.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.373+0000 7fdddd800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fddd80b5500 0x7fddd80b6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:41.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.373+0000 7fdddd800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddd80d3dc0 con 0x7fddd80b59b0 2024-04-26T23:05:41.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.373+0000 7fdddd800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fddd80d03b0 con 0x7fddd80b5500 2024-04-26T23:05:41.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.373+0000 7fdddd800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fddd80b75b0 con 0x7fddd80b5120 2024-04-26T23:05:41.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.373+0000 7fddcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fddd80b5500 0x7fddd80b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:41.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.373+0000 7fddcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fddd80b5500 0x7fddd80b6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46388/0 (socket says 172.21.15.40:46388) 2024-04-26T23:05:41.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.373+0000 7fddcfe00640 1 -- 172.21.15.40:0/811198125 learned_addr learned my addr 172.21.15.40:0/811198125 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:41.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.374+0000 7fddd7e00640 1 --2- 172.21.15.40:0/811198125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b5120 0x7fddd80b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:41.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.374+0000 7fddcfe00640 1 -- 172.21.15.40:0/811198125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b5120 msgr2=0x7fddd80b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:41.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.374+0000 7fddcfe00640 1 --2- 172.21.15.40:0/811198125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b5120 0x7fddd80b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:41.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.374+0000 7fddcfe00640 1 -- 172.21.15.40:0/811198125 >> v1:172.21.15.103:6789/0 conn(0x7fddd80b59b0 legacy=0x7fddd80b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:41.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.374+0000 7fddcfe00640 1 -- 172.21.15.40:0/811198125 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd80d0600 con 0x7fddd80b5500 2024-04-26T23:05:41.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.376+0000 7fddcfe00640 1 --2- 172.21.15.40:0/811198125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddd80b5500 0x7fddd80b6830 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fddc0009870 tx=0x7fddc002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ec991810bdf25e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:41.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.376+0000 7fddd7400640 1 -- 172.21.15.40:0/811198125 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc0037070 con 0x7fddd80b5500 2024-04-26T23:05:41.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.376+0000 7fddd7400640 1 -- 172.21.15.40:0/811198125 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddc002ae10 con 0x7fddd80b5500 2024-04-26T23:05:41.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.376+0000 7fddd7400640 1 -- 172.21.15.40:0/811198125 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc0032660 con 0x7fddd80b5500 2024-04-26T23:05:41.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.377+0000 7fdddd800640 1 -- 172.21.15.40:0/811198125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddd80b5500 msgr2=0x7fddd80b6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:41.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.377+0000 7fdddd800640 1 --2- 172.21.15.40:0/811198125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddd80b5500 0x7fddd80b6830 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fddc0009870 tx=0x7fddc002a100 comp rx=0 tx=0).stop 2024-04-26T23:05:41.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.377+0000 7fdddd800640 1 -- 172.21.15.40:0/811198125 shutdown_connections 2024-04-26T23:05:41.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.377+0000 7fdddd800640 1 --2- 172.21.15.40:0/811198125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddd80b5500 0x7fddd80b6830 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:41.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.377+0000 7fdddd800640 1 --2- 172.21.15.40:0/811198125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b5120 0x7fddd80b5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.377+0000 7fdddd800640 1 -- 172.21.15.40:0/811198125 >> 172.21.15.40:0/811198125 conn(0x7fddd80d4db0 msgr2=0x7fddd80d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.377+0000 7fdddd800640 1 -- 172.21.15.40:0/811198125 shutdown_connections 2024-04-26T23:05:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.377+0000 7fdddd800640 1 -- 172.21.15.40:0/811198125 wait complete. 2024-04-26T23:05:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.379+0000 7fdddd800640 1 Processor -- start 2024-04-26T23:05:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.379+0000 7fdddd800640 1 -- start start 2024-04-26T23:05:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.380+0000 7fdddd800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd80b5120 0x7fddd80b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.380+0000 7fdddd800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddd80b5500 0x7fddd8153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.380+0000 7fdddd800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b59b0 0x7fddd806cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.380+0000 7fdddd800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fddd80b7c00 con 0x7fddd80b5500 2024-04-26T23:05:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.380+0000 7fdddd800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fddd80d0bc0 con 0x7fddd80b5120 2024-04-26T23:05:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.380+0000 7fdddd800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fddd80b7d80 con 0x7fddd80b59b0 2024-04-26T23:05:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.380+0000 7fddcfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddd80b5500 0x7fddd8153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.380+0000 7fddcfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddd80b5500 0x7fddd8153210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46402/0 (socket says 172.21.15.40:46402) 2024-04-26T23:05:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.380+0000 7fddcfe00640 1 -- 172.21.15.40:0/2299954785 learned_addr learned my addr 172.21.15.40:0/2299954785 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.381+0000 7fddcfe00640 1 -- 172.21.15.40:0/2299954785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b59b0 msgr2=0x7fddd806cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.381+0000 7fddcfe00640 1 --2- 172.21.15.40:0/2299954785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b59b0 0x7fddd806cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.381+0000 7fddcfe00640 1 -- 172.21.15.40:0/2299954785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd80b5120 msgr2=0x7fddd80b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.381+0000 7fddd7e00640 1 --2- 172.21.15.40:0/2299954785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd80b5120 0x7fddd80b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.381+0000 7fddcfe00640 1 --2- 172.21.15.40:0/2299954785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd80b5120 0x7fddd80b4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.381+0000 7fddcfe00640 1 -- 172.21.15.40:0/2299954785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd80b48e0 con 0x7fddd80b5500 2024-04-26T23:05:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.381+0000 7fddcfe00640 1 --2- 172.21.15.40:0/2299954785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddd80b5500 0x7fddd8153210 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fddc0000c00 tx=0x7fddc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.381+0000 7fddd4c00640 1 -- 172.21.15.40:0/2299954785 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc0042070 con 0x7fddd80b5500 2024-04-26T23:05:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.381+0000 7fddd4c00640 1 -- 172.21.15.40:0/2299954785 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddc003e670 con 0x7fddd80b5500 2024-04-26T23:05:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.381+0000 7fddd4c00640 1 -- 172.21.15.40:0/2299954785 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc0037040 con 0x7fddd80b5500 2024-04-26T23:05:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.381+0000 7fdddd800640 1 -- 172.21.15.40:0/2299954785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd80b4600 con 0x7fddd80b5500 2024-04-26T23:05:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.382+0000 7fdddd800640 1 -- 172.21.15.40:0/2299954785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd815d890 con 0x7fddd80b5500 2024-04-26T23:05:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.383+0000 7fdddd800640 1 -- 172.21.15.40:0/2299954785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd8069370 con 0x7fddd80b5500 2024-04-26T23:05:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.384+0000 7fddd4c00640 1 -- 172.21.15.40:0/2299954785 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fddc0045070 con 0x7fddd80b5500 2024-04-26T23:05:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.385+0000 7fddd4c00640 1 --2- 172.21.15.40:0/2299954785 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fddb406c8f0 0x7fddb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.385+0000 7fddd4c00640 1 -- 172.21.15.40:0/2299954785 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7fddc00ac520 con 0x7fddd80b5500 2024-04-26T23:05:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.385+0000 7fddd7e00640 1 --2- 172.21.15.40:0/2299954785 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fddb406c8f0 0x7fddb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.386+0000 7fddd7e00640 1 --2- 172.21.15.40:0/2299954785 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fddb406c8f0 0x7fddb406edb0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fddc8006fd0 tx=0x7fddc8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:41.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.390+0000 7fddd4c00640 1 -- 172.21.15.40:0/2299954785 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fddc007a8d0 con 0x7fddd80b5500 2024-04-26T23:05:41.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.639+0000 7fdddd800640 1 -- 172.21.15.40:0/2299954785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fddd80d3ba0 con 0x7fddd80b5500 2024-04-26T23:05:41.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.805+0000 7fddd4c00640 1 -- 172.21.15.40:0/2299954785 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v56) ==== 105+0+0 (secure 0 0 0) 0x7fddc007a1d0 con 0x7fddd80b5500 2024-04-26T23:05:41.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.810+0000 7fdddd800640 1 -- 172.21.15.40:0/2299954785 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fddb406c8f0 msgr2=0x7fddb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:41.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.810+0000 7fdddd800640 1 --2- 172.21.15.40:0/2299954785 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fddb406c8f0 0x7fddb406edb0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fddc8006fd0 tx=0x7fddc8008040 comp rx=0 tx=0).stop 2024-04-26T23:05:41.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.810+0000 7fdddd800640 1 -- 172.21.15.40:0/2299954785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddd80b5500 msgr2=0x7fddd8153210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:41.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.810+0000 7fdddd800640 1 --2- 172.21.15.40:0/2299954785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddd80b5500 0x7fddd8153210 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fddc0000c00 tx=0x7fddc00046c0 comp rx=0 tx=0).stop 2024-04-26T23:05:41.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.810+0000 7fdddd800640 1 -- 172.21.15.40:0/2299954785 shutdown_connections 2024-04-26T23:05:41.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.810+0000 7fdddd800640 1 --2- 172.21.15.40:0/2299954785 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fddb406c8f0 0x7fddb406edb0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:41.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.810+0000 7fdddd800640 1 --2- 172.21.15.40:0/2299954785 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b59b0 0x7fddd806cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:41.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.810+0000 7fdddd800640 1 --2- 172.21.15.40:0/2299954785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddd80b5500 0x7fddd8153210 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:41.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.810+0000 7fdddd800640 1 --2- 172.21.15.40:0/2299954785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd80b5120 0x7fddd80b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:41.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.810+0000 7fdddd800640 1 -- 172.21.15.40:0/2299954785 >> 172.21.15.40:0/2299954785 conn(0x7fddd80d4db0 msgr2=0x7fddd806e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:41.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.811+0000 7fdddd800640 1 -- 172.21.15.40:0/2299954785 shutdown_connections 2024-04-26T23:05:41.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:41.811+0000 7fdddd800640 1 -- 172.21.15.40:0/2299954785 wait complete. 2024-04-26T23:05:41.842 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:05:42.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.022+0000 7f5b93e00640 1 Processor -- start 2024-04-26T23:05:42.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.022+0000 7f5b93e00640 1 -- start start 2024-04-26T23:05:42.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.024+0000 7f5b93e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b940b6220 0x7f5b940d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:42.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.024+0000 7f5b93e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5b940d39f0 0x7f5b940b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:42.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.024+0000 7f5b93e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b94162c40 con 0x7f5b940d08d0 2024-04-26T23:05:42.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.024+0000 7f5b93e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5b940d2d80 con 0x7f5b940d39f0 2024-04-26T23:05:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.024+0000 7f5b93e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5b940b7c80 con 0x7f5b940b6220 2024-04-26T23:05:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.024+0000 7f5b8be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5b940d08d0 0x7f5b940d32e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41212/0 (socket says 172.21.15.40:41212) 2024-04-26T23:05:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.024+0000 7f5b8be00640 1 -- 172.21.15.40:0/760202474 learned_addr learned my addr 172.21.15.40:0/760202474 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.025+0000 7f5b93400640 1 --2- 172.21.15.40:0/760202474 >> v2:172.21.15.103:3300/0 conn(0x7f5b940d39f0 0x7f5b940b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.025+0000 7f5b92a00640 1 --2- 172.21.15.40:0/760202474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b940b6220 0x7f5b940d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.025+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3231075042 0 0) 0x7f5b94162c40 con 0x7f5b940d08d0 2024-04-26T23:05:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.025+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b78003520 con 0x7f5b940d08d0 2024-04-26T23:05:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.025+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3476224340 0 0) 0x7f5b78003520 con 0x7f5b940d08d0 2024-04-26T23:05:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.025+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b940b6220 msgr2=0x7f5b940d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.025+0000 7f5b92000640 1 --2- 172.21.15.40:0/760202474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b940b6220 0x7f5b940d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.025+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 >> v2:172.21.15.103:3300/0 conn(0x7f5b940d39f0 msgr2=0x7f5b940b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.025+0000 7f5b92000640 1 --2- 172.21.15.40:0/760202474 >> v2:172.21.15.103:3300/0 conn(0x7f5b940d39f0 0x7f5b940b8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.025+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b940b7f60 con 0x7f5b940d08d0 2024-04-26T23:05:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.026+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f5b7c001670 con 0x7f5b940d08d0 2024-04-26T23:05:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.026+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 >> v1:172.21.15.40:6789/0 conn(0x7f5b940d08d0 legacy=0x7f5b940d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.026+0000 7f5b92000640 1 --2- 172.21.15.40:0/760202474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b780039b0 0x7f5b78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.026+0000 7f5b92000640 1 --2- 172.21.15.40:0/760202474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b940b6220 0x7f5b78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.026+0000 7f5b92000640 1 --2- 172.21.15.40:0/760202474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b78004d40 0x7f5b780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.026+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b940d2d80 con 0x7f5b780039b0 2024-04-26T23:05:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.026+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b940b7c80 con 0x7f5b78004d40 2024-04-26T23:05:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.026+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5b7800b9c0 con 0x7f5b940b6220 2024-04-26T23:05:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.026+0000 7f5b92a00640 1 --2- 172.21.15.40:0/760202474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b780039b0 0x7f5b78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.026+0000 7f5b8be00640 1 --2- 172.21.15.40:0/760202474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b940b6220 0x7f5b78004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.026+0000 7f5b93400640 1 --2- 172.21.15.40:0/760202474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b78004d40 0x7f5b780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.027+0000 7f5b92a00640 1 -- 172.21.15.40:0/760202474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b940b6220 msgr2=0x7f5b78004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.027+0000 7f5b92a00640 1 --2- 172.21.15.40:0/760202474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b940b6220 0x7f5b78004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.027+0000 7f5b92a00640 1 -- 172.21.15.40:0/760202474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b78004d40 msgr2=0x7f5b780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.027+0000 7f5b92a00640 1 --2- 172.21.15.40:0/760202474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b78004d40 0x7f5b780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:42.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.027+0000 7f5b92a00640 1 -- 172.21.15.40:0/760202474 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b78003520 con 0x7f5b780039b0 2024-04-26T23:05:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.028+0000 7f5b92a00640 1 --2- 172.21.15.40:0/760202474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b780039b0 0x7f5b78003dd0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f5b84009ac0 tx=0x7f5b78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=169a1730e2692000 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.028+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84036070 con 0x7f5b780039b0 2024-04-26T23:05:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.028+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b84031400 con 0x7f5b780039b0 2024-04-26T23:05:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.028+0000 7f5b92000640 1 -- 172.21.15.40:0/760202474 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84009690 con 0x7f5b780039b0 2024-04-26T23:05:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.029+0000 7f5b93e00640 1 -- 172.21.15.40:0/760202474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b780039b0 msgr2=0x7f5b78003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.029+0000 7f5b93e00640 1 --2- 172.21.15.40:0/760202474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b780039b0 0x7f5b78003dd0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f5b84009ac0 tx=0x7f5b78005830 comp rx=0 tx=0).stop 2024-04-26T23:05:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.029+0000 7f5b93400640 1 -- 172.21.15.40:0/760202474 reap_dead start 2024-04-26T23:05:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.029+0000 7f5b93e00640 1 -- 172.21.15.40:0/760202474 shutdown_connections 2024-04-26T23:05:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.029+0000 7f5b93e00640 1 -- 172.21.15.40:0/760202474 >> 172.21.15.40:0/760202474 conn(0x7f5b940d4d90 msgr2=0x7f5b940d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.029+0000 7f5b93e00640 1 -- 172.21.15.40:0/760202474 shutdown_connections 2024-04-26T23:05:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.029+0000 7f5b93e00640 1 -- 172.21.15.40:0/760202474 wait complete. 2024-04-26T23:05:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.031+0000 7f5b93e00640 1 Processor -- start 2024-04-26T23:05:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.031+0000 7f5b93e00640 1 -- start start 2024-04-26T23:05:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.032+0000 7f5b93e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b780039b0 0x7f5b940a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.032+0000 7f5b93e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b940cd430 0x7f5b941f4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.032+0000 7f5b93e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b94159670 0x7f5b94159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.032+0000 7f5b93e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b940d0390 con 0x7f5b94159670 2024-04-26T23:05:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.032+0000 7f5b93e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b940b8520 con 0x7f5b780039b0 2024-04-26T23:05:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.032+0000 7f5b93e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5b940d0510 con 0x7f5b940cd430 2024-04-26T23:05:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.032+0000 7f5b93400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b94159670 0x7f5b94159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.032+0000 7f5b93400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b94159670 0x7f5b94159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46416/0 (socket says 172.21.15.40:46416) 2024-04-26T23:05:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.032+0000 7f5b93400640 1 -- 172.21.15.40:0/3793743417 learned_addr learned my addr 172.21.15.40:0/3793743417 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.032+0000 7f5b92a00640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b780039b0 0x7f5b940a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.032+0000 7f5b8be00640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b940cd430 0x7f5b941f4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.033+0000 7f5b93400640 1 -- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b940cd430 msgr2=0x7f5b941f4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.033+0000 7f5b93400640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b940cd430 0x7f5b941f4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.033+0000 7f5b93400640 1 -- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b780039b0 msgr2=0x7f5b940a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.033+0000 7f5b93400640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b780039b0 0x7f5b940a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.033+0000 7f5b93400640 1 -- 172.21.15.40:0/3793743417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b941f52b0 con 0x7f5b94159670 2024-04-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.033+0000 7f5b93400640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b94159670 0x7f5b94159af0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f5b9408d530 tx=0x7f5b8c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.033+0000 7f5b8aa00640 1 -- 172.21.15.40:0/3793743417 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c011e10 con 0x7f5b94159670 2024-04-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.033+0000 7f5b8aa00640 1 -- 172.21.15.40:0/3793743417 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b8c01b070 con 0x7f5b94159670 2024-04-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.033+0000 7f5b8aa00640 1 -- 172.21.15.40:0/3793743417 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c003990 con 0x7f5b94159670 2024-04-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.033+0000 7f5b93e00640 1 -- 172.21.15.40:0/3793743417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b941f5590 con 0x7f5b94159670 2024-04-26T23:05:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.033+0000 7f5b93e00640 1 -- 172.21.15.40:0/3793743417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b94158e70 con 0x7f5b94159670 2024-04-26T23:05:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.035+0000 7f5b93e00640 1 -- 172.21.15.40:0/3793743417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b94069370 con 0x7f5b94159670 2024-04-26T23:05:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.036+0000 7f5b8aa00640 1 -- 172.21.15.40:0/3793743417 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5b8c005d10 con 0x7f5b94159670 2024-04-26T23:05:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.037+0000 7f5b8aa00640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5b7006c8f0 0x7f5b7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.037+0000 7f5b8aa00640 1 -- 172.21.15.40:0/3793743417 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f5b8c08c320 con 0x7f5b94159670 2024-04-26T23:05:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.038+0000 7f5b92a00640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5b7006c8f0 0x7f5b7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:42.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.039+0000 7f5b92a00640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5b7006c8f0 0x7f5b7006edb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5b84033000 tx=0x7f5b84008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.044+0000 7f5b8aa00640 1 -- 172.21.15.40:0/3793743417 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5b8c025020 con 0x7f5b94159670 2024-04-26T23:05:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.297+0000 7f5b93e00640 1 -- 172.21.15.40:0/3793743417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b9406d610 con 0x7f5b94159670 2024-04-26T23:05:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.298+0000 7f5b8aa00640 1 -- 172.21.15.40:0/3793743417 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5201 (secure 0 0 0) 0x7f5b8c01d310 con 0x7f5b94159670 2024-04-26T23:05:42.299 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:42.299 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":57,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/4194140680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4194140680},{"type":"v1","addr":"172.21.15.103:6836","nonce":4194140680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":5324,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2595375825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2595375825},{"type":"v1","addr":"172.21.15.103:6837","nonce":2595375825}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:05:32.212319+0000","modified":"2024-04-26T23:05:41.811225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1],"up":{"mds_0":6774,"mds_1":5614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5614":{"gid":5614,"name":"a","rank":1,"incarnation":57,"state":"up:creating","state_seq":1,"addr":"172.21.15.40:6834/2117861921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2117861921},{"type":"v1","addr":"172.21.15.40:6834","nonce":2117861921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/931483025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":931483025},{"type":"v1","addr":"172.21.15.40:6835","nonce":931483025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6774,"qdb_cluster":[6774]},"id":4}]} 2024-04-26T23:05:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 57 2024-04-26T23:05:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.301+0000 7f5b93e00640 1 -- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5b7006c8f0 msgr2=0x7f5b7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.301+0000 7f5b93e00640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5b7006c8f0 0x7f5b7006edb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5b84033000 tx=0x7f5b84008030 comp rx=0 tx=0).stop 2024-04-26T23:05:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.301+0000 7f5b93e00640 1 -- 172.21.15.40:0/3793743417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b94159670 msgr2=0x7f5b94159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.301+0000 7f5b93e00640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b94159670 0x7f5b94159af0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f5b9408d530 tx=0x7f5b8c0115a0 comp rx=0 tx=0).stop 2024-04-26T23:05:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.301+0000 7f5b93e00640 1 -- 172.21.15.40:0/3793743417 shutdown_connections 2024-04-26T23:05:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.301+0000 7f5b93e00640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5b7006c8f0 0x7f5b7006edb0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.301+0000 7f5b93e00640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b94159670 0x7f5b94159af0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.301+0000 7f5b93e00640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b940cd430 0x7f5b941f4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.301+0000 7f5b93e00640 1 --2- 172.21.15.40:0/3793743417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b780039b0 0x7f5b940a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.301+0000 7f5b93e00640 1 -- 172.21.15.40:0/3793743417 >> 172.21.15.40:0/3793743417 conn(0x7f5b940d4d90 msgr2=0x7f5b940ba600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.302+0000 7f5b93e00640 1 -- 172.21.15.40:0/3793743417 shutdown_connections 2024-04-26T23:05:42.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:42.302+0000 7f5b93e00640 1 -- 172.21.15.40:0/3793743417 wait complete. 2024-04-26T23:05:42.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:05:32.212319+0000', 'modified': '2024-04-26T23:05:41.811225+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 6774, 'mds_1': 5614}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5614': {'gid': 5614, 'name': 'a', 'rank': 1, 'incarnation': 57, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.40:6834/2117861921', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 2117861921}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 2117861921}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_6774': {'gid': 6774, 'name': 'c', 'rank': 0, 'incarnation': 51, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/931483025', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 931483025}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 931483025}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6774, 'qdb_cluster': [6774]} 2024-04-26T23:05:42.722 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5614:up:creating 2024-04-26T23:05:43.723 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:05:43.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.931+0000 7ff487e00640 1 Processor -- start 2024-04-26T23:05:43.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.931+0000 7ff487e00640 1 -- start start 2024-04-26T23:05:43.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.933+0000 7ff487e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4880c6f10 0x7ff4880c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:43.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.933+0000 7ff487e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff4880c6010 0x7ff488158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:43.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.933+0000 7ff487e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4880cce10 con 0x7ff4880cc980 2024-04-26T23:05:43.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.933+0000 7ff487e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff4880c72f0 con 0x7ff4880c6010 2024-04-26T23:05:43.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.933+0000 7ff487e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff4880cb860 con 0x7ff4880c6f10 2024-04-26T23:05:43.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.934+0000 7ff486a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4880c6f10 0x7ff4880c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:43.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.934+0000 7ff487400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff4880c6010 0x7ff488158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:43.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.934+0000 7ff487400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff4880c6010 0x7ff488158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:59134/0 (socket says 172.21.15.40:59134) 2024-04-26T23:05:43.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.934+0000 7ff487400640 1 -- 172.21.15.40:0/1463338402 learned_addr learned my addr 172.21.15.40:0/1463338402 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:43.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.934+0000 7ff486a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4880c6f10 0x7ff4880c5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:35114/0 (socket says 172.21.15.40:35114) 2024-04-26T23:05:43.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.935+0000 7ff487400640 1 -- 172.21.15.40:0/1463338402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4880c6f10 msgr2=0x7ff4880c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:43.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.935+0000 7ff487400640 1 --2- 172.21.15.40:0/1463338402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4880c6f10 0x7ff4880c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:43.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.935+0000 7ff487400640 1 -- 172.21.15.40:0/1463338402 >> v1:172.21.15.103:6789/0 conn(0x7ff4880cc980 legacy=0x7ff4880cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:43.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.935+0000 7ff487400640 1 -- 172.21.15.40:0/1463338402 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4880cd0f0 con 0x7ff4880c6010 2024-04-26T23:05:43.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.935+0000 7ff486a00640 1 --2- 172.21.15.40:0/1463338402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4880c6f10 0x7ff4880c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:05:43.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.936+0000 7ff487400640 1 --2- 172.21.15.40:0/1463338402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4880c6010 0x7ff488158cd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff480009870 tx=0x7ff48002a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6c4db4f11239a80 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:43.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.936+0000 7ff486000640 1 -- 172.21.15.40:0/1463338402 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480037070 con 0x7ff4880c6010 2024-04-26T23:05:43.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.936+0000 7ff486000640 1 -- 172.21.15.40:0/1463338402 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff480032440 con 0x7ff4880c6010 2024-04-26T23:05:43.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.936+0000 7ff486000640 1 -- 172.21.15.40:0/1463338402 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480032810 con 0x7ff4880c6010 2024-04-26T23:05:43.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.937+0000 7ff487e00640 1 -- 172.21.15.40:0/1463338402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4880c6010 msgr2=0x7ff488158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:43.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.937+0000 7ff487e00640 1 --2- 172.21.15.40:0/1463338402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4880c6010 0x7ff488158cd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff480009870 tx=0x7ff48002a1e0 comp rx=0 tx=0).stop 2024-04-26T23:05:43.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.937+0000 7ff487e00640 1 -- 172.21.15.40:0/1463338402 shutdown_connections 2024-04-26T23:05:43.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.937+0000 7ff487e00640 1 --2- 172.21.15.40:0/1463338402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4880c6010 0x7ff488158cd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:43.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.937+0000 7ff487e00640 1 --2- 172.21.15.40:0/1463338402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4880c6f10 0x7ff4880c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:43.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.937+0000 7ff487e00640 1 -- 172.21.15.40:0/1463338402 >> 172.21.15.40:0/1463338402 conn(0x7ff4880d0a10 msgr2=0x7ff4880d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:43.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.937+0000 7ff487e00640 1 -- 172.21.15.40:0/1463338402 shutdown_connections 2024-04-26T23:05:43.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.937+0000 7ff487e00640 1 -- 172.21.15.40:0/1463338402 wait complete. 2024-04-26T23:05:43.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.939+0000 7ff487e00640 1 Processor -- start 2024-04-26T23:05:43.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.939+0000 7ff487e00640 1 -- start start 2024-04-26T23:05:43.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.940+0000 7ff487e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4880c6010 0x7ff488150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:43.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.940+0000 7ff487e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4880c6f10 0x7ff4880c8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:43.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.940+0000 7ff487e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4881f5140 0x7ff4880b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:43.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.940+0000 7ff487e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff4880cc400 con 0x7ff4881f5140 2024-04-26T23:05:43.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.940+0000 7ff487e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff4880cb1a0 con 0x7ff4880c6010 2024-04-26T23:05:43.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.940+0000 7ff487e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff4880cc580 con 0x7ff4880c6f10 2024-04-26T23:05:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.940+0000 7ff487400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4881f5140 0x7ff4880b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.940+0000 7ff487400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4881f5140 0x7ff4880b43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46432/0 (socket says 172.21.15.40:46432) 2024-04-26T23:05:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.940+0000 7ff487400640 1 -- 172.21.15.40:0/3080216983 learned_addr learned my addr 172.21.15.40:0/3080216983 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.941+0000 7ff486a00640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4880c6010 0x7ff488150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.941+0000 7ff47e000640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4880c6f10 0x7ff4880c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.941+0000 7ff487400640 1 -- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4880c6f10 msgr2=0x7ff4880c8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.941+0000 7ff487400640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4880c6f10 0x7ff4880c8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.941+0000 7ff487400640 1 -- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4880c6010 msgr2=0x7ff488150630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.941+0000 7ff487400640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4880c6010 0x7ff488150630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.941+0000 7ff487400640 1 -- 172.21.15.40:0/3080216983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4881f5b90 con 0x7ff4881f5140 2024-04-26T23:05:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.941+0000 7ff486a00640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4880c6010 0x7ff488150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.941+0000 7ff487400640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4881f5140 0x7ff4880b43e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff480009870 tx=0x7ff4800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:43.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.942+0000 7ff47f400640 1 -- 172.21.15.40:0/3080216983 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480041020 con 0x7ff4881f5140 2024-04-26T23:05:43.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.942+0000 7ff47f400640 1 -- 172.21.15.40:0/3080216983 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff48003d640 con 0x7ff4881f5140 2024-04-26T23:05:43.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.942+0000 7ff487e00640 1 -- 172.21.15.40:0/3080216983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4881f5e70 con 0x7ff4881f5140 2024-04-26T23:05:43.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.942+0000 7ff47f400640 1 -- 172.21.15.40:0/3080216983 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480037040 con 0x7ff4881f5140 2024-04-26T23:05:43.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.942+0000 7ff487e00640 1 -- 172.21.15.40:0/3080216983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4880c7f60 con 0x7ff4881f5140 2024-04-26T23:05:43.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.943+0000 7ff487e00640 1 -- 172.21.15.40:0/3080216983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff488069370 con 0x7ff4881f5140 2024-04-26T23:05:43.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.944+0000 7ff47f400640 1 -- 172.21.15.40:0/3080216983 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff480044020 con 0x7ff4881f5140 2024-04-26T23:05:43.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.945+0000 7ff47f400640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff46406c940 0x7ff46406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:43.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.945+0000 7ff47f400640 1 -- 172.21.15.40:0/3080216983 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7ff4800ac470 con 0x7ff4881f5140 2024-04-26T23:05:43.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.946+0000 7ff486a00640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff46406c940 0x7ff46406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.947+0000 7ff486a00640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff46406c940 0x7ff46406ee00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff478009bd0 tx=0x7ff47800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:43.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:43.949+0000 7ff47f400640 1 -- 172.21.15.40:0/3080216983 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff48004b030 con 0x7ff4881f5140 2024-04-26T23:05:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.192+0000 7ff487e00640 1 -- 172.21.15.40:0/3080216983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff488070020 con 0x7ff4881f5140 2024-04-26T23:05:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.194+0000 7ff47f400640 1 -- 172.21.15.40:0/3080216983 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5223 (secure 0 0 0) 0x7ff48007a700 con 0x7ff4881f5140 2024-04-26T23:05:44.194 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:44.194 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":60,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/4194140680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4194140680},{"type":"v1","addr":"172.21.15.103:6836","nonce":4194140680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:05:32.212319+0000","modified":"2024-04-26T23:05:43.829475+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6774,"mds_1":5614,"mds_2":5324},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5324":{"gid":5324,"name":"b","rank":2,"incarnation":59,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/2595375825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2595375825},{"type":"v1","addr":"172.21.15.103:6837","nonce":2595375825}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5614":{"gid":5614,"name":"a","rank":1,"incarnation":57,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/2117861921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2117861921},{"type":"v1","addr":"172.21.15.40:6834","nonce":2117861921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/931483025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":931483025},{"type":"v1","addr":"172.21.15.40:6835","nonce":931483025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6774,"qdb_cluster":[5324,5614,6774]},"id":4}]} 2024-04-26T23:05:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 60 2024-04-26T23:05:44.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.196+0000 7ff487e00640 1 -- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff46406c940 msgr2=0x7ff46406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:44.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.196+0000 7ff487e00640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff46406c940 0x7ff46406ee00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff478009bd0 tx=0x7ff47800b040 comp rx=0 tx=0).stop 2024-04-26T23:05:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.196+0000 7ff487e00640 1 -- 172.21.15.40:0/3080216983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4881f5140 msgr2=0x7ff4880b43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.196+0000 7ff487e00640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4881f5140 0x7ff4880b43e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff480009870 tx=0x7ff4800046c0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.196+0000 7ff487e00640 1 -- 172.21.15.40:0/3080216983 shutdown_connections 2024-04-26T23:05:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.196+0000 7ff487e00640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff46406c940 0x7ff46406ee00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.196+0000 7ff487e00640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4881f5140 0x7ff4880b43e0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.196+0000 7ff487e00640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4880c6f10 0x7ff4880c8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.196+0000 7ff487e00640 1 --2- 172.21.15.40:0/3080216983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4880c6010 0x7ff488150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.196+0000 7ff487e00640 1 -- 172.21.15.40:0/3080216983 >> 172.21.15.40:0/3080216983 conn(0x7ff4880d0a10 msgr2=0x7ff48815d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.196+0000 7ff487e00640 1 -- 172.21.15.40:0/3080216983 shutdown_connections 2024-04-26T23:05:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.196+0000 7ff487e00640 1 -- 172.21.15.40:0/3080216983 wait complete. 2024-04-26T23:05:44.618 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:05:32.212319+0000', 'modified': '2024-04-26T23:05:43.829475+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 6774, 'mds_1': 5614, 'mds_2': 5324}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5324': {'gid': 5324, 'name': 'b', 'rank': 2, 'incarnation': 59, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.103:6837/2595375825', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6835', 'nonce': 2595375825}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 2595375825}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_5614': {'gid': 5614, 'name': 'a', 'rank': 1, 'incarnation': 57, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.40:6834/2117861921', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 2117861921}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 2117861921}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_6774': {'gid': 6774, 'name': 'c', 'rank': 0, 'incarnation': 51, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/931483025', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 931483025}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 931483025}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6774, 'qdb_cluster': [5324, 5614, 6774]} 2024-04-26T23:05:44.618 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T23:05:44.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T23:05:44.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.818+0000 7f231c600640 1 Processor -- start 2024-04-26T23:05:44.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.819+0000 7f231c600640 1 -- start start 2024-04-26T23:05:44.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.820+0000 7f231c600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f23140711d0 0x7f23140703e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.820+0000 7f231c600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23140cf2b0 con 0x7f23140ceed0 2024-04-26T23:05:44.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.820+0000 7f231c600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23140ce4c0 con 0x7f23140d3980 2024-04-26T23:05:44.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.820+0000 7f231c600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2314070920 con 0x7f23140711d0 2024-04-26T23:05:44.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.820+0000 7f2319c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f23140ceed0 0x7f2314070ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41220/0 (socket says 172.21.15.40:41220) 2024-04-26T23:05:44.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.820+0000 7f2319c00640 1 -- 172.21.15.40:0/569733810 learned_addr learned my addr 172.21.15.40:0/569733810 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.820+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2541351792 0 0) 0x7f23140cf2b0 con 0x7f23140ceed0 2024-04-26T23:05:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2319200640 1 --2- 172.21.15.40:0/569733810 >> v2:172.21.15.103:3300/0 conn(0x7f23140711d0 0x7f23140703e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22fc003520 con 0x7f23140ceed0 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4006172353 0 0) 0x7f22fc003520 con 0x7f23140ceed0 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 >> v2:172.21.15.103:3300/0 conn(0x7f23140711d0 msgr2=0x7f23140703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 --2- 172.21.15.40:0/569733810 >> v2:172.21.15.103:3300/0 conn(0x7f23140711d0 0x7f23140703e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 >> v1:172.21.15.103:6789/0 conn(0x7f23140d3980 legacy=0x7f231415ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23140d12d0 con 0x7f23140ceed0 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f2308001670 con 0x7f23140ceed0 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 >> v1:172.21.15.40:6789/0 conn(0x7f23140ceed0 legacy=0x7f2314070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 --2- 172.21.15.40:0/569733810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22fc003a00 0x7f22fc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 --2- 172.21.15.40:0/569733810 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22fc004800 0x7f22fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 --2- 172.21.15.40:0/569733810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22fc0051c0 0x7f22fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2314070920 con 0x7f22fc0051c0 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc00c0c0 con 0x7f22fc003a00 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f22fc00c2c0 con 0x7f22fc004800 2024-04-26T23:05:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.821+0000 7f231a600640 1 --2- 172.21.15.40:0/569733810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22fc0051c0 0x7f22fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.822+0000 7f2319c00640 1 --2- 172.21.15.40:0/569733810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22fc003a00 0x7f22fc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.822+0000 7f2319200640 1 --2- 172.21.15.40:0/569733810 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22fc004800 0x7f22fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.822+0000 7f231a600640 1 -- 172.21.15.40:0/569733810 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22fc004800 msgr2=0x7f22fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:44.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.822+0000 7f231a600640 1 --2- 172.21.15.40:0/569733810 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22fc004800 0x7f22fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.822+0000 7f231a600640 1 -- 172.21.15.40:0/569733810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22fc003a00 msgr2=0x7f22fc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:44.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.822+0000 7f231a600640 1 --2- 172.21.15.40:0/569733810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22fc003a00 0x7f22fc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.822+0000 7f231a600640 1 -- 172.21.15.40:0/569733810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22fc003520 con 0x7f22fc0051c0 2024-04-26T23:05:44.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.823+0000 7f231a600640 1 -- 172.21.15.40:0/569733810 reap_dead start 2024-04-26T23:05:44.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.823+0000 7f231a600640 1 --2- 172.21.15.40:0/569733810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22fc0051c0 0x7f22fc00b9c0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f230c001e50 tx=0x7f22fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e992e753b0ffcb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:44.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.823+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c001a70 con 0x7f22fc0051c0 2024-04-26T23:05:44.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.823+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f230c003970 con 0x7f22fc0051c0 2024-04-26T23:05:44.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.823+0000 7f2313e00640 1 -- 172.21.15.40:0/569733810 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c003d60 con 0x7f22fc0051c0 2024-04-26T23:05:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.824+0000 7f231c600640 1 -- 172.21.15.40:0/569733810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22fc0051c0 msgr2=0x7f22fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.824+0000 7f231c600640 1 --2- 172.21.15.40:0/569733810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22fc0051c0 0x7f22fc00b9c0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f230c001e50 tx=0x7f22fc005830 comp rx=0 tx=0).stop 2024-04-26T23:05:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.824+0000 7f231c600640 1 -- 172.21.15.40:0/569733810 shutdown_connections 2024-04-26T23:05:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.824+0000 7f231c600640 1 --2- 172.21.15.40:0/569733810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22fc0051c0 0x7f22fc00b9c0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.824+0000 7f231c600640 1 -- 172.21.15.40:0/569733810 >> 172.21.15.40:0/569733810 conn(0x7f23140d2e60 msgr2=0x7f23140d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.824+0000 7f231c600640 1 -- 172.21.15.40:0/569733810 shutdown_connections 2024-04-26T23:05:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.824+0000 7f231c600640 1 -- 172.21.15.40:0/569733810 wait complete. 2024-04-26T23:05:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.826+0000 7f231c600640 1 Processor -- start 2024-04-26T23:05:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.826+0000 7f231c600640 1 -- start start 2024-04-26T23:05:44.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.827+0000 7f231c600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22fc0051c0 0x7f23141e2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.827+0000 7f231c600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23141d8080 0x7f23140c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.827+0000 7f231c600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23140c6e00 0x7f23141da310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.827+0000 7f231c600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f23140d1890 con 0x7f23141d8080 2024-04-26T23:05:44.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.827+0000 7f231c600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f23140d1b70 con 0x7f23140c6e00 2024-04-26T23:05:44.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.827+0000 7f231c600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f23140d1e50 con 0x7f22fc0051c0 2024-04-26T23:05:44.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.827+0000 7f2319200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23141d8080 0x7f23140c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.827+0000 7f2319200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23141d8080 0x7f23140c68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46452/0 (socket says 172.21.15.40:46452) 2024-04-26T23:05:44.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.827+0000 7f2319200640 1 -- 172.21.15.40:0/2679167656 learned_addr learned my addr 172.21.15.40:0/2679167656 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:44.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.828+0000 7f231a600640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23140c6e00 0x7f23141da310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.828+0000 7f2319c00640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22fc0051c0 0x7f23141e2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.828+0000 7f2319200640 1 -- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22fc0051c0 msgr2=0x7f23141e2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:44.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.828+0000 7f2319200640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22fc0051c0 0x7f23141e2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.828+0000 7f2319200640 1 -- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23140c6e00 msgr2=0x7f23141da310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:44.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.828+0000 7f2319200640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23140c6e00 0x7f23141da310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.828+0000 7f2319200640 1 -- 172.21.15.40:0/2679167656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23141e81d0 con 0x7f23141d8080 2024-04-26T23:05:44.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.828+0000 7f2319200640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23141d8080 0x7f23140c68c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f23000115a0 tx=0x7f2300011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:44.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.828+0000 7f2311600640 1 -- 172.21.15.40:0/2679167656 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300015930 con 0x7f23141d8080 2024-04-26T23:05:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.828+0000 7f2311600640 1 -- 172.21.15.40:0/2679167656 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2300015ad0 con 0x7f23141d8080 2024-04-26T23:05:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.828+0000 7f2311600640 1 -- 172.21.15.40:0/2679167656 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230001d400 con 0x7f23141d8080 2024-04-26T23:05:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.828+0000 7f231c600640 1 -- 172.21.15.40:0/2679167656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23141e84b0 con 0x7f23141d8080 2024-04-26T23:05:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.829+0000 7f231c600640 1 -- 172.21.15.40:0/2679167656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23141e8a70 con 0x7f23141d8080 2024-04-26T23:05:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.831+0000 7f2311600640 1 -- 172.21.15.40:0/2679167656 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f230001d5a0 con 0x7f23141d8080 2024-04-26T23:05:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.831+0000 7f2311600640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f22f406c920 0x7f22f406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.832+0000 7f2311600640 1 -- 172.21.15.40:0/2679167656 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f230008c570 con 0x7f23141d8080 2024-04-26T23:05:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.832+0000 7f2319c00640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f22f406c920 0x7f22f406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.832+0000 7f231f122740 1 Processor -- start 2024-04-26T23:05:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.832+0000 7f231f122740 1 -- start start 2024-04-26T23:05:44.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.833+0000 7f231f122740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 0x556354fb3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.833+0000 7f231f122740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fedcb0 0x556354fb19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.833+0000 7f231f122740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 0x556354fdc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.833+0000 7f231f122740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556354f15600 con 0x556354fdc0a0 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.833+0000 7f231f122740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556354eb2100 con 0x556354fb15e0 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.833+0000 7f231f122740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556354f115c0 con 0x556354fedcb0 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.834+0000 7f2319c00640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f22f406c920 0x7f22f406ede0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f23140c7660 tx=0x7f2308000f10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.834+0000 7f231a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 0x556354fdc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.834+0000 7f231a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 0x556354fdc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46454/0 (socket says 172.21.15.40:46454) 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.834+0000 7f231a600640 1 -- 172.21.15.40:0/2700729449 learned_addr learned my addr 172.21.15.40:0/2700729449 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.834+0000 7f2319200640 1 --2- 172.21.15.40:0/2700729449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fedcb0 0x556354fb19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.834+0000 7f2319c00640 1 --2- 172.21.15.40:0/2700729449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 0x556354fb3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.834+0000 7f231a600640 1 -- 172.21.15.40:0/2700729449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fedcb0 msgr2=0x556354fb19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.834+0000 7f231a600640 1 --2- 172.21.15.40:0/2700729449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fedcb0 0x556354fb19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.834+0000 7f231a600640 1 -- 172.21.15.40:0/2700729449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 msgr2=0x556354fb3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.834+0000 7f231a600640 1 --2- 172.21.15.40:0/2700729449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 0x556354fb3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.834+0000 7f231a600640 1 -- 172.21.15.40:0/2700729449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556354c272d0 con 0x556354fdc0a0 2024-04-26T23:05:44.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.834+0000 7f231a600640 1 --2- 172.21.15.40:0/2700729449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 0x556354fdc520 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f230c001f40 tx=0x7f230c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecf58eee27566b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:44.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.835+0000 7f2306000640 1 -- 172.21.15.40:0/2700729449 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c006b50 con 0x556354fdc0a0 2024-04-26T23:05:44.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.835+0000 7f2306000640 1 -- 172.21.15.40:0/2700729449 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f230c006cf0 con 0x556354fdc0a0 2024-04-26T23:05:44.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.835+0000 7f2306000640 1 -- 172.21.15.40:0/2700729449 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c003b70 con 0x556354fdc0a0 2024-04-26T23:05:44.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.835+0000 7f231f122740 1 -- 172.21.15.40:0/2700729449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 msgr2=0x556354fdc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:44.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.835+0000 7f231f122740 1 --2- 172.21.15.40:0/2700729449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 0x556354fdc520 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f230c001f40 tx=0x7f230c02c000 comp rx=0 tx=0).stop 2024-04-26T23:05:44.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.835+0000 7f231f122740 1 -- 172.21.15.40:0/2700729449 shutdown_connections 2024-04-26T23:05:44.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.835+0000 7f231f122740 1 --2- 172.21.15.40:0/2700729449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 0x556354fdc520 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.835+0000 7f231f122740 1 --2- 172.21.15.40:0/2700729449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fedcb0 0x556354fb19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.835+0000 7f231f122740 1 --2- 172.21.15.40:0/2700729449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 0x556354fb3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.835+0000 7f231f122740 1 -- 172.21.15.40:0/2700729449 >> 172.21.15.40:0/2700729449 conn(0x556354f45d10 msgr2=0x556354ff70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:44.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.835+0000 7f231f122740 1 -- 172.21.15.40:0/2700729449 shutdown_connections 2024-04-26T23:05:44.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.835+0000 7f231f122740 1 -- 172.21.15.40:0/2700729449 wait complete. 2024-04-26T23:05:44.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.836+0000 7f231f122740 1 -- start start 2024-04-26T23:05:44.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.836+0000 7f231f122740 1 Processor -- start 2024-04-26T23:05:44.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.837+0000 7f231f122740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 0x556354fd6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.837+0000 7f231f122740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fdc0a0 0x556354fdd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.837+0000 7f231f122740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 0x556354fddcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.837+0000 7f231f122740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556354e9cf70 con 0x556354fedcb0 2024-04-26T23:05:44.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.837+0000 7f231f122740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556354fd9bf0 con 0x556354fb15e0 2024-04-26T23:05:44.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.837+0000 7f231f122740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556354ff2bd0 con 0x556354fdc0a0 2024-04-26T23:05:44.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.837+0000 7f231f122740 10 client.0 _ll_register_callbacks cb 0x556354f0b480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:44.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.837+0000 7f231a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 0x556354fddcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.838+0000 7f231a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 0x556354fddcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46456/0 (socket says 172.21.15.40:46456) 2024-04-26T23:05:44.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.838+0000 7f231a600640 1 -- 172.21.15.40:0/920018462 learned_addr learned my addr 172.21.15.40:0/920018462 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:44.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.838+0000 7f231a600640 1 -- 172.21.15.40:0/920018462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fdc0a0 msgr2=0x556354fdd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:44.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.838+0000 7f231a600640 1 --2- 172.21.15.40:0/920018462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fdc0a0 0x556354fdd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.838+0000 7f231a600640 1 -- 172.21.15.40:0/920018462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 msgr2=0x556354fd6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:44.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.838+0000 7f231a600640 1 --2- 172.21.15.40:0/920018462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 0x556354fd6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.838+0000 7f231a600640 1 -- 172.21.15.40:0/920018462 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556354cf16b0 con 0x556354fedcb0 2024-04-26T23:05:44.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.838+0000 7f231a600640 1 --2- 172.21.15.40:0/920018462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 0x556354fddcc0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f230c008000 tx=0x7f230c001510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16bdb549269b5cba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.838+0000 7f22faa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.838+0000 7f22faa00640 1 -- 172.21.15.40:0/920018462 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c006b50 con 0x556354fedcb0 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.838+0000 7f22faa00640 1 -- 172.21.15.40:0/920018462 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f230c0052e0 con 0x556354fedcb0 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.838+0000 7f231f122740 1 -- 172.21.15.40:0/920018462 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f230c006cf0 con 0x556354fedcb0 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.839+0000 7f22faa00640 1 -- 172.21.15.40:0/920018462 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f230c04b050 con 0x556354fedcb0 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.839+0000 7f231f122740 10 client.6960 fetch_fsmap learned FSMap version 60 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.839+0000 7f231f122740 1 -- 172.21.15.40:0/920018462 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) -- 0x556354ff2910 con 0x556354fedcb0 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.839+0000 7f22faa00640 1 -- 172.21.15.40:0/920018462 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f230c006cf0 con 0x556354fedcb0 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.839+0000 7f22faa00640 1 -- 172.21.15.40:0/920018462 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 60) ==== 3534+0+0 (secure 0 0 0) 0x7f230c04e020 con 0x556354fedcb0 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.839+0000 7f231f122740 10 client.6960 fetch_fsmap finished waiting for FSMap version 60 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.839+0000 7f231f122740 10 client.6960 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.839+0000 7f231f122740 20 client.6960 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.839+0000 7f231f122740 1 --2- 172.21.15.40:0/920018462 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x556354ff1890 0x556354ff1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.839+0000 7f231f122740 4 client.6960 mds_command: new command op to 5324 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:05:44.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.839+0000 7f231f122740 1 -- 172.21.15.40:0/920018462 --> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556354f23210 con 0x556354ff1890 2024-04-26T23:05:44.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.840+0000 7f2319c00640 1 --2- 172.21.15.40:0/920018462 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x556354ff1890 0x556354ff1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.841+0000 7f2319c00640 1 --2- 172.21.15.40:0/920018462 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x556354ff1890 0x556354ff1c70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=73c859bd7dd7a849 server_cookie=6283d2ef6f0cf361 in_seq=0 out_seq=0 2024-04-26T23:05:44.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.841+0000 7f22faa00640 10 client.6960 ms_handle_connect on v2:172.21.15.103:6835/2595375825 2024-04-26T23:05:44.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.844+0000 7f22faa00640 1 -- 172.21.15.40:0/920018462 <== mds.2 v2:172.21.15.103:6835/2595375825 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556354f23210 con 0x556354ff1890 2024-04-26T23:05:44.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.844+0000 7f22faa00640 10 client.6960 handle_command_reply: tid=0 2024-04-26T23:05:44.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.844+0000 7f231f122740 1 client.6960 shutdown 2024-04-26T23:05:44.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.844+0000 7f231f122740 1 -- 172.21.15.40:0/920018462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 msgr2=0x556354fddcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:44.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.844+0000 7f231f122740 1 --2- 172.21.15.40:0/920018462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 0x556354fddcc0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f230c008000 tx=0x7f230c001510 comp rx=0 tx=0).stop 2024-04-26T23:05:44.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.844+0000 7f231f122740 1 -- 172.21.15.40:0/920018462 shutdown_connections 2024-04-26T23:05:44.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.844+0000 7f231f122740 1 --2- 172.21.15.40:0/920018462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 0x556354fddcc0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.844+0000 7f231f122740 1 --2- 172.21.15.40:0/920018462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fdc0a0 0x556354fdd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.844+0000 7f231f122740 1 --2- 172.21.15.40:0/920018462 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x556354ff1890 0x556354ff1c70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.844+0000 7f231f122740 1 --2- 172.21.15.40:0/920018462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 0x556354fd6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.844+0000 7f231f122740 1 -- 172.21.15.40:0/920018462 >> 172.21.15.40:0/920018462 conn(0x556354f45d10 msgr2=0x556354ffa590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:44.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.844+0000 7f22faa00640 0 client.6960 ms_handle_reset on v2:172.21.15.103:6835/2595375825 2024-04-26T23:05:44.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.845+0000 7f231f122740 1 -- 172.21.15.40:0/920018462 shutdown_connections 2024-04-26T23:05:44.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.845+0000 7f231f122740 1 -- 172.21.15.40:0/920018462 wait complete. 2024-04-26T23:05:44.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.845+0000 7f231f122740 20 client.6960 trim_cache size 0 max 16384 2024-04-26T23:05:44.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.864+0000 7f231f122740 1 Processor -- start 2024-04-26T23:05:44.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.864+0000 7f231f122740 1 -- start start 2024-04-26T23:05:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.865+0000 7f231f122740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 0x556354fee090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.865+0000 7f231f122740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fdc0a0 0x556354faf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.865+0000 7f231f122740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 0x556354fb06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.865+0000 7f231f122740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2314201870 con 0x556354fedcb0 2024-04-26T23:05:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.865+0000 7f231f122740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2314201f40 con 0x556354fb15e0 2024-04-26T23:05:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.865+0000 7f231f122740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2314200aa0 con 0x556354fdc0a0 2024-04-26T23:05:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.865+0000 7f231a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 0x556354fb06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.865+0000 7f231a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 0x556354fb06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36622/0 (socket says 172.21.15.40:36622) 2024-04-26T23:05:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.865+0000 7f231a600640 1 -- 172.21.15.40:0/1919159522 learned_addr learned my addr 172.21.15.40:0/1919159522 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.866+0000 7f231a600640 1 -- 172.21.15.40:0/1919159522 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fdc0a0 msgr2=0x556354faf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.866+0000 7f231a600640 1 --2- 172.21.15.40:0/1919159522 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fdc0a0 0x556354faf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.866+0000 7f231a600640 1 -- 172.21.15.40:0/1919159522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 msgr2=0x556354fee090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:44.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.866+0000 7f231a600640 1 --2- 172.21.15.40:0/1919159522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 0x556354fee090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.866+0000 7f231a600640 1 -- 172.21.15.40:0/1919159522 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556354fad890 con 0x556354fedcb0 2024-04-26T23:05:44.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.866+0000 7f231a600640 1 --2- 172.21.15.40:0/1919159522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 0x556354fb06b0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f230c003f00 tx=0x7f230c03e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4d5d382f12528c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:44.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.866+0000 7f2305600640 1 -- 172.21.15.40:0/1919159522 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c005790 con 0x556354fedcb0 2024-04-26T23:05:44.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.866+0000 7f2305600640 1 -- 172.21.15.40:0/1919159522 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f230c043070 con 0x556354fedcb0 2024-04-26T23:05:44.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.866+0000 7f2305600640 1 -- 172.21.15.40:0/1919159522 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c006b50 con 0x556354fedcb0 2024-04-26T23:05:44.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.867+0000 7f231f122740 1 -- 172.21.15.40:0/1919159522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 msgr2=0x556354fb06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:44.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.867+0000 7f231f122740 1 --2- 172.21.15.40:0/1919159522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 0x556354fb06b0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f230c003f00 tx=0x7f230c03e000 comp rx=0 tx=0).stop 2024-04-26T23:05:44.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.867+0000 7f231f122740 1 -- 172.21.15.40:0/1919159522 shutdown_connections 2024-04-26T23:05:44.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.867+0000 7f231f122740 1 --2- 172.21.15.40:0/1919159522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fedcb0 0x556354fb06b0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.867+0000 7f231f122740 1 --2- 172.21.15.40:0/1919159522 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fdc0a0 0x556354faf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.867+0000 7f231f122740 1 --2- 172.21.15.40:0/1919159522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 0x556354fee090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.867+0000 7f231f122740 1 -- 172.21.15.40:0/1919159522 >> 172.21.15.40:0/1919159522 conn(0x556354f45d10 msgr2=0x556354fdd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:44.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.867+0000 7f231f122740 1 -- 172.21.15.40:0/1919159522 shutdown_connections 2024-04-26T23:05:44.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.867+0000 7f231f122740 1 -- 172.21.15.40:0/1919159522 wait complete. 2024-04-26T23:05:44.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.868+0000 7f231f122740 1 -- start start 2024-04-26T23:05:44.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.868+0000 7f231f122740 1 Processor -- start 2024-04-26T23:05:44.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.869+0000 7f231f122740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 0x556355010320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.869+0000 7f231f122740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 0x556354fdf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.869+0000 7f231f122740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fedcb0 0x556354ff1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.869+0000 7f231f122740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556354ff0640 con 0x556354fdc0a0 2024-04-26T23:05:44.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.869+0000 7f231f122740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556354e9cf70 con 0x556354fb15e0 2024-04-26T23:05:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.869+0000 7f231f122740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556354fdc760 con 0x556354fedcb0 2024-04-26T23:05:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.869+0000 7f231f122740 10 client.0 _ll_register_callbacks cb 0x7f230c036070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.869+0000 7f2319200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 0x556354fdf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.869+0000 7f2319200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 0x556354fdf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36634/0 (socket says 172.21.15.40:36634) 2024-04-26T23:05:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.869+0000 7f2319200640 1 -- 172.21.15.40:0/877081163 learned_addr learned my addr 172.21.15.40:0/877081163 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.869+0000 7f2319200640 1 -- 172.21.15.40:0/877081163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fedcb0 msgr2=0x556354ff1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.869+0000 7f2319200640 1 --2- 172.21.15.40:0/877081163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fedcb0 0x556354ff1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.870+0000 7f2319200640 1 -- 172.21.15.40:0/877081163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 msgr2=0x556355010320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.870+0000 7f2319200640 1 --2- 172.21.15.40:0/877081163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 0x556355010320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.870+0000 7f2319200640 1 -- 172.21.15.40:0/877081163 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556354fb1200 con 0x556354fdc0a0 2024-04-26T23:05:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.870+0000 7f2319200640 1 --2- 172.21.15.40:0/877081163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 0x556354fdf0b0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f2300062720 tx=0x7f230005e4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554490fdbc05417c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.870+0000 7f22faa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.870+0000 7f22faa00640 1 -- 172.21.15.40:0/877081163 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300022050 con 0x556354fdc0a0 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.870+0000 7f22faa00640 1 -- 172.21.15.40:0/877081163 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f230008c2f0 con 0x556354fdc0a0 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.870+0000 7f231f122740 1 -- 172.21.15.40:0/877081163 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f230c043070 con 0x556354fdc0a0 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.870+0000 7f22faa00640 1 -- 172.21.15.40:0/877081163 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f23000584e0 con 0x556354fdc0a0 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.870+0000 7f231f122740 10 client.6966 fetch_fsmap learned FSMap version 60 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.870+0000 7f231f122740 1 -- 172.21.15.40:0/877081163 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) -- 0x556354ff7b60 con 0x556354fdc0a0 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.870+0000 7f22faa00640 1 -- 172.21.15.40:0/877081163 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f230c043070 con 0x556354fdc0a0 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.870+0000 7f22faa00640 1 -- 172.21.15.40:0/877081163 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 60) ==== 3534+0+0 (secure 0 0 0) 0x7f23000627c0 con 0x556354fdc0a0 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.871+0000 7f231f122740 10 client.6966 fetch_fsmap finished waiting for FSMap version 60 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.871+0000 7f231f122740 10 client.6966 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.871+0000 7f231f122740 20 client.6966 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.871+0000 7f231f122740 1 --2- 172.21.15.40:0/877081163 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x556354fd3f70 0x556354fd4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.871+0000 7f231f122740 4 client.6966 mds_command: new command op to 5324 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:05:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.871+0000 7f231f122740 1 -- 172.21.15.40:0/877081163 --> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] -- command(tid 0: {"prefix": "status"}) -- 0x556354f1d080 con 0x556354fd3f70 2024-04-26T23:05:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.871+0000 7f2319c00640 1 --2- 172.21.15.40:0/877081163 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x556354fd3f70 0x556354fd4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:44.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.872+0000 7f2319c00640 1 --2- 172.21.15.40:0/877081163 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x556354fd3f70 0x556354fd4350 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6f553bdb4c465a65 server_cookie=788acc675eb9d04f in_seq=0 out_seq=0 2024-04-26T23:05:44.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.872+0000 7f22faa00640 10 client.6966 ms_handle_connect on v2:172.21.15.103:6835/2595375825 2024-04-26T23:05:44.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f22faa00640 1 -- 172.21.15.40:0/877081163 <== mds.2 v2:172.21.15.103:6835/2595375825 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x556354f1d080 con 0x556354fd3f70 2024-04-26T23:05:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f22faa00640 10 client.6966 handle_command_reply: tid=0 2024-04-26T23:05:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f231f122740 1 client.6966 shutdown 2024-04-26T23:05:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f231f122740 1 -- 172.21.15.40:0/877081163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 msgr2=0x556354fdf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f231f122740 1 --2- 172.21.15.40:0/877081163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 0x556354fdf0b0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f2300062720 tx=0x7f230005e4e0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f231f122740 1 -- 172.21.15.40:0/877081163 shutdown_connections 2024-04-26T23:05:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f231f122740 1 --2- 172.21.15.40:0/877081163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556354fedcb0 0x556354ff1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f231f122740 1 --2- 172.21.15.40:0/877081163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556354fdc0a0 0x556354fdf0b0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f231f122740 1 --2- 172.21.15.40:0/877081163 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x556354fd3f70 0x556354fd4350 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f231f122740 1 --2- 172.21.15.40:0/877081163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556354fb15e0 0x556355010320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f22faa00640 0 client.6966 ms_handle_reset on v2:172.21.15.103:6835/2595375825 2024-04-26T23:05:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f231f122740 1 -- 172.21.15.40:0/877081163 >> 172.21.15.40:0/877081163 conn(0x556354f45d10 msgr2=0x556354fdd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f231f122740 1 -- 172.21.15.40:0/877081163 shutdown_connections 2024-04-26T23:05:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f231f122740 1 -- 172.21.15.40:0/877081163 wait complete. 2024-04-26T23:05:44.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.873+0000 7f231f122740 20 client.6966 trim_cache size 0 max 16384 2024-04-26T23:05:44.878 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:05:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:05:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-04-26T23:05:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5324, 2024-04-26T23:05:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:05:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:05:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:05:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 2.0420243650000001, 2024-04-26T23:05:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 60, 2024-04-26T23:05:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 65, 2024-04-26T23:05:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 65, 2024-04-26T23:05:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 17.529398148999999 2024-04-26T23:05:44.879 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:05:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.874+0000 7f231c600640 1 -- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f22f406c920 msgr2=0x7f22f406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.874+0000 7f231c600640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f22f406c920 0x7f22f406ede0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f23140c7660 tx=0x7f2308000f10 comp rx=0 tx=0).stop 2024-04-26T23:05:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.874+0000 7f231c600640 1 -- 172.21.15.40:0/2679167656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23141d8080 msgr2=0x7f23140c68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.874+0000 7f231c600640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23141d8080 0x7f23140c68c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f23000115a0 tx=0x7f2300011a60 comp rx=0 tx=0).stop 2024-04-26T23:05:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.874+0000 7f231c600640 1 -- 172.21.15.40:0/2679167656 shutdown_connections 2024-04-26T23:05:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.874+0000 7f231c600640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f22f406c920 0x7f22f406ede0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.874+0000 7f231c600640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23140c6e00 0x7f23141da310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.874+0000 7f231c600640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23141d8080 0x7f23140c68c0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.874+0000 7f231c600640 1 --2- 172.21.15.40:0/2679167656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22fc0051c0 0x7f23141e2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.874+0000 7f231c600640 1 -- 172.21.15.40:0/2679167656 >> 172.21.15.40:0/2679167656 conn(0x7f23140d2e60 msgr2=0x7f23141ead60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.874+0000 7f231c600640 1 -- 172.21.15.40:0/2679167656 shutdown_connections 2024-04-26T23:05:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:44.874+0000 7f231c600640 1 -- 172.21.15.40:0/2679167656 wait complete. 2024-04-26T23:05:44.890 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T23:05:45.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.066+0000 7f7bb5c00640 1 Processor -- start 2024-04-26T23:05:45.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.067+0000 7f7bb5c00640 1 -- start start 2024-04-26T23:05:45.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.068+0000 7f7bb5c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bb00cf810 0x7f7bb016e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.068+0000 7f7bb5c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bb0163810 con 0x7f7bb0163430 2024-04-26T23:05:45.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.068+0000 7f7bb5c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bb00cb100 con 0x7f7bb00c72d0 2024-04-26T23:05:45.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.068+0000 7f7bb5c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7bb00cb400 con 0x7f7bb00cf810 2024-04-26T23:05:45.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.068+0000 7f7bafe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7bb0163430 0x7f7bb0166f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60638/0 (socket says 172.21.15.40:60638) 2024-04-26T23:05:45.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.068+0000 7f7bafe00640 1 -- 172.21.15.40:0/2236826531 learned_addr learned my addr 172.21.15.40:0/2236826531 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7bb5200640 1 --2- 172.21.15.40:0/2236826531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bb00cf810 0x7f7bb016e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 811702459 0 0) 0x7f7bb0163810 con 0x7f7bb0163430 2024-04-26T23:05:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b94003520 con 0x7f7bb0163430 2024-04-26T23:05:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1851654709 0 0) 0x7f7b94003520 con 0x7f7bb0163430 2024-04-26T23:05:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bb00cf810 msgr2=0x7f7bb016e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 --2- 172.21.15.40:0/2236826531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bb00cf810 0x7f7bb016e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 >> v1:172.21.15.103:6789/0 conn(0x7f7bb00c72d0 legacy=0x7f7bb0167670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb00ccc60 con 0x7f7bb0163430 2024-04-26T23:05:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f7ba0001690 con 0x7f7bb0163430 2024-04-26T23:05:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 >> v1:172.21.15.40:6789/0 conn(0x7f7bb0163430 legacy=0x7f7bb0166f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 --2- 172.21.15.40:0/2236826531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b940039e0 0x7f7b94003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 --2- 172.21.15.40:0/2236826531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b94004800 0x7f7b94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 --2- 172.21.15.40:0/2236826531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bb00cf810 0x7f7b940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb00cb400 con 0x7f7b94004800 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9400bad0 con 0x7f7b940039e0 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.069+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7b9400bcd0 con 0x7f7bb00cf810 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.070+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/2236826531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b94004800 0x7f7b94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.070+0000 7f7bafe00640 1 --2- 172.21.15.40:0/2236826531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b940039e0 0x7f7b94003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.070+0000 7f7ba7e00640 1 -- 172.21.15.40:0/2236826531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bb00cf810 msgr2=0x7f7b940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.070+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/2236826531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bb00cf810 0x7f7b940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.070+0000 7f7ba7e00640 1 -- 172.21.15.40:0/2236826531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b940039e0 msgr2=0x7f7b94003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.070+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/2236826531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b940039e0 0x7f7b94003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.070+0000 7f7ba7e00640 1 -- 172.21.15.40:0/2236826531 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b94003520 con 0x7f7b94004800 2024-04-26T23:05:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.070+0000 7f7bafe00640 1 --2- 172.21.15.40:0/2236826531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b940039e0 0x7f7b94003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.071+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/2236826531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b94004800 0x7f7b94004c80 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f7bb006d530 tx=0x7f7b94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3423e289730d72 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.071+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98001370 con 0x7f7b94004800 2024-04-26T23:05:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.071+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b980052d0 con 0x7f7b94004800 2024-04-26T23:05:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.071+0000 7f7baf400640 1 -- 172.21.15.40:0/2236826531 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9802aab0 con 0x7f7b94004800 2024-04-26T23:05:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.072+0000 7f7bb5c00640 1 -- 172.21.15.40:0/2236826531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b94004800 msgr2=0x7f7b94004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.072+0000 7f7bb5c00640 1 --2- 172.21.15.40:0/2236826531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b94004800 0x7f7b94004c80 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f7bb006d530 tx=0x7f7b94005830 comp rx=0 tx=0).stop 2024-04-26T23:05:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.072+0000 7f7bb5200640 1 -- 172.21.15.40:0/2236826531 reap_dead start 2024-04-26T23:05:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.072+0000 7f7bb5c00640 1 -- 172.21.15.40:0/2236826531 shutdown_connections 2024-04-26T23:05:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.072+0000 7f7bb5c00640 1 -- 172.21.15.40:0/2236826531 >> 172.21.15.40:0/2236826531 conn(0x7f7bb00d0a10 msgr2=0x7f7bb00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.072+0000 7f7bb5c00640 1 -- 172.21.15.40:0/2236826531 shutdown_connections 2024-04-26T23:05:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.072+0000 7f7bb5c00640 1 -- 172.21.15.40:0/2236826531 wait complete. 2024-04-26T23:05:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.074+0000 7f7bb5c00640 1 Processor -- start 2024-04-26T23:05:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.074+0000 7f7bb5c00640 1 -- start start 2024-04-26T23:05:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bb5c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b94004800 0x7f7bb00720d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bb5c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bb015c410 0x7f7bb015c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bb5c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bb015a9c0 0x7f7bb015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bb5c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb0167d80 con 0x7f7b94004800 2024-04-26T23:05:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bb5c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb00afe80 con 0x7f7bb015a9c0 2024-04-26T23:05:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bb5c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7bb00cd1e0 con 0x7f7bb015c410 2024-04-26T23:05:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bafe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b94004800 0x7f7bb00720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bafe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b94004800 0x7f7bb00720d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36646/0 (socket says 172.21.15.40:36646) 2024-04-26T23:05:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bafe00640 1 -- 172.21.15.40:0/2398785517 learned_addr learned my addr 172.21.15.40:0/2398785517 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bafe00640 1 -- 172.21.15.40:0/2398785517 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bb015c410 msgr2=0x7f7bb015c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bafe00640 1 --2- 172.21.15.40:0/2398785517 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bb015c410 0x7f7bb015c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bafe00640 1 -- 172.21.15.40:0/2398785517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bb015a9c0 msgr2=0x7f7bb015ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bafe00640 1 --2- 172.21.15.40:0/2398785517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bb015a9c0 0x7f7bb015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.075+0000 7f7bafe00640 1 -- 172.21.15.40:0/2398785517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb01f9f10 con 0x7f7b94004800 2024-04-26T23:05:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.076+0000 7f7bafe00640 1 --2- 172.21.15.40:0/2398785517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b94004800 0x7f7bb00720d0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000d810 tx=0x7f7ba000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.076+0000 7f7bacc00640 1 -- 172.21.15.40:0/2398785517 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0019070 con 0x7f7b94004800 2024-04-26T23:05:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.076+0000 7f7bacc00640 1 -- 172.21.15.40:0/2398785517 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ba00074e0 con 0x7f7b94004800 2024-04-26T23:05:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.076+0000 7f7bacc00640 1 -- 172.21.15.40:0/2398785517 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba00049e0 con 0x7f7b94004800 2024-04-26T23:05:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.076+0000 7f7bb5c00640 1 -- 172.21.15.40:0/2398785517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bb015cfd0 con 0x7f7b94004800 2024-04-26T23:05:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.076+0000 7f7bb5c00640 1 -- 172.21.15.40:0/2398785517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bb01fca20 con 0x7f7b94004800 2024-04-26T23:05:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.078+0000 7f7bacc00640 1 -- 172.21.15.40:0/2398785517 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7ba0004b80 con 0x7f7b94004800 2024-04-26T23:05:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.080+0000 7f7bacc00640 1 --2- 172.21.15.40:0/2398785517 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7b8c06c920 0x7f7b8c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.080+0000 7f7bacc00640 1 -- 172.21.15.40:0/2398785517 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f7ba008b7d0 con 0x7f7b94004800 2024-04-26T23:05:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.081+0000 7f7bba1dd740 1 Processor -- start 2024-04-26T23:05:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.081+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/2398785517 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7b8c06c920 0x7f7b8c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.081+0000 7f7bba1dd740 1 -- start start 2024-04-26T23:05:45.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.082+0000 7f7bba1dd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f51e7b0 0x55de4f520910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.082+0000 7f7bba1dd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f55ae80 0x55de4f51eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.082+0000 7f7bba1dd740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f549270 0x55de4f5496f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.082+0000 7f7bba1dd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55de4f482600 con 0x55de4f55ae80 2024-04-26T23:05:45.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.082+0000 7f7bba1dd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55de4f41f100 con 0x55de4f51e7b0 2024-04-26T23:05:45.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.082+0000 7f7bba1dd740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55de4f47e5c0 con 0x55de4f549270 2024-04-26T23:05:45.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.082+0000 7f7ba7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f55ae80 0x55de4f51eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.082+0000 7f7ba7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f55ae80 0x55de4f51eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36660/0 (socket says 172.21.15.40:36660) 2024-04-26T23:05:45.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.082+0000 7f7ba7e00640 1 -- 172.21.15.40:0/462626965 learned_addr learned my addr 172.21.15.40:0/462626965 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:45.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.082+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/2398785517 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7b8c06c920 0x7f7b8c06ede0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7b98032000 tx=0x7f7b9802c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.083+0000 7f7bafe00640 1 --2- 172.21.15.40:0/462626965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f51e7b0 0x55de4f520910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.083+0000 7f7ba7e00640 1 -- 172.21.15.40:0/462626965 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f549270 msgr2=0x55de4f5496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.083+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/462626965 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f549270 0x55de4f5496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.083+0000 7f7ba7e00640 1 -- 172.21.15.40:0/462626965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f51e7b0 msgr2=0x55de4f520910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.083+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/462626965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f51e7b0 0x55de4f520910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.083+0000 7f7ba7e00640 1 -- 172.21.15.40:0/462626965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de4f1942d0 con 0x55de4f55ae80 2024-04-26T23:05:45.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.083+0000 7f7bafe00640 1 --2- 172.21.15.40:0/462626965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f51e7b0 0x55de4f520910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:05:45.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.083+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/462626965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f55ae80 0x55de4f51eb90 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0071ae0 tx=0x7f7b9802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3c0c2ead98f8d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.083+0000 7f7ba4c00640 1 -- 172.21.15.40:0/462626965 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9804b070 con 0x55de4f55ae80 2024-04-26T23:05:45.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.083+0000 7f7ba4c00640 1 -- 172.21.15.40:0/462626965 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b98007110 con 0x55de4f55ae80 2024-04-26T23:05:45.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.083+0000 7f7ba4c00640 1 -- 172.21.15.40:0/462626965 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b980074c0 con 0x55de4f55ae80 2024-04-26T23:05:45.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.084+0000 7f7bba1dd740 1 -- 172.21.15.40:0/462626965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f55ae80 msgr2=0x55de4f51eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.084+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/462626965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f55ae80 0x55de4f51eb90 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0071ae0 tx=0x7f7b9802e000 comp rx=0 tx=0).stop 2024-04-26T23:05:45.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.084+0000 7f7bba1dd740 1 -- 172.21.15.40:0/462626965 shutdown_connections 2024-04-26T23:05:45.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.084+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/462626965 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f549270 0x55de4f5496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.084+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/462626965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f55ae80 0x55de4f51eb90 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.084+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/462626965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f51e7b0 0x55de4f520910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.084+0000 7f7bba1dd740 1 -- 172.21.15.40:0/462626965 >> 172.21.15.40:0/462626965 conn(0x55de4f4b2d10 msgr2=0x55de4f564270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.084+0000 7f7bba1dd740 1 -- 172.21.15.40:0/462626965 shutdown_connections 2024-04-26T23:05:45.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.084+0000 7f7bba1dd740 1 -- 172.21.15.40:0/462626965 wait complete. 2024-04-26T23:05:45.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.085+0000 7f7bba1dd740 1 -- start start 2024-04-26T23:05:45.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.085+0000 7f7bba1dd740 1 Processor -- start 2024-04-26T23:05:45.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.086+0000 7f7bba1dd740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f51e7b0 0x55de4f55d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.086+0000 7f7bba1dd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 0x55de4f5430b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.086+0000 7f7bba1dd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f55ae80 0x55de4f5435f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.086+0000 7f7bba1dd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55de4f409f70 con 0x55de4f549270 2024-04-26T23:05:45.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.086+0000 7f7bba1dd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55de4f546dc0 con 0x55de4f55ae80 2024-04-26T23:05:45.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.086+0000 7f7bba1dd740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55de4f54af50 con 0x55de4f51e7b0 2024-04-26T23:05:45.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.086+0000 7f7bba1dd740 10 client.0 _ll_register_callbacks cb 0x55de4f478480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:45.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.086+0000 7f7ba7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 0x55de4f5430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.086+0000 7f7ba7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 0x55de4f5430b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36672/0 (socket says 172.21.15.40:36672) 2024-04-26T23:05:45.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.086+0000 7f7ba7e00640 1 -- 172.21.15.40:0/1278817841 learned_addr learned my addr 172.21.15.40:0/1278817841 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:45.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.087+0000 7f7ba7e00640 1 -- 172.21.15.40:0/1278817841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f51e7b0 msgr2=0x55de4f55d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.087+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/1278817841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f51e7b0 0x55de4f55d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.087+0000 7f7ba7e00640 1 -- 172.21.15.40:0/1278817841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f55ae80 msgr2=0x55de4f5435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.087+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/1278817841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f55ae80 0x55de4f5435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.087+0000 7f7ba7e00640 1 -- 172.21.15.40:0/1278817841 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de4f25e6b0 con 0x55de4f549270 2024-04-26T23:05:45.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.087+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/1278817841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 0x55de4f5430b0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f7b98049040 tx=0x7f7b9802c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72db611bf61d2427 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.087+0000 7f7b9d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:45.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.087+0000 7f7b9d600640 1 -- 172.21.15.40:0/1278817841 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98055060 con 0x55de4f549270 2024-04-26T23:05:45.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.087+0000 7f7b9d600640 1 -- 172.21.15.40:0/1278817841 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f7b98006be0 con 0x55de4f549270 2024-04-26T23:05:45.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.087+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1278817841 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b98007110 con 0x55de4f549270 2024-04-26T23:05:45.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.087+0000 7f7b9d600640 1 -- 172.21.15.40:0/1278817841 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f7b9805b050 con 0x55de4f549270 2024-04-26T23:05:45.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.087+0000 7f7bba1dd740 10 client.6978 fetch_fsmap learned FSMap version 60 2024-04-26T23:05:45.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.087+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1278817841 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) -- 0x55de4f54ac90 con 0x55de4f549270 2024-04-26T23:05:45.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.088+0000 7f7b9d600640 1 -- 172.21.15.40:0/1278817841 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b98007110 con 0x55de4f549270 2024-04-26T23:05:45.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.088+0000 7f7b9d600640 1 -- 172.21.15.40:0/1278817841 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 60) ==== 3534+0+0 (secure 0 0 0) 0x7f7b9805e020 con 0x55de4f549270 2024-04-26T23:05:45.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.088+0000 7f7bba1dd740 10 client.6978 fetch_fsmap finished waiting for FSMap version 60 2024-04-26T23:05:45.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.088+0000 7f7bba1dd740 10 client.6978 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:05:45.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.088+0000 7f7bba1dd740 20 client.6978 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:45.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.088+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/1278817841 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55de4f549c10 0x55de4f549ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.088+0000 7f7bba1dd740 4 client.6978 mds_command: new command op to 5614 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:05:45.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.088+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1278817841 --> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de4f490210 con 0x55de4f549c10 2024-04-26T23:05:45.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.088+0000 7f7bafe00640 1 --2- 172.21.15.40:0/1278817841 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55de4f549c10 0x55de4f549ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.089+0000 7f7bafe00640 1 --2- 172.21.15.40:0/1278817841 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55de4f549c10 0x55de4f549ff0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=feb9c6acb261cc6e server_cookie=3ace45138cf9b1c1 in_seq=0 out_seq=0 2024-04-26T23:05:45.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.089+0000 7f7b9d600640 10 client.6978 ms_handle_connect on v2:172.21.15.40:6832/2117861921 2024-04-26T23:05:45.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7b9d600640 1 -- 172.21.15.40:0/1278817841 <== mds.1 v2:172.21.15.40:6832/2117861921 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55de4f490210 con 0x55de4f549c10 2024-04-26T23:05:45.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7b9d600640 10 client.6978 handle_command_reply: tid=0 2024-04-26T23:05:45.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7bba1dd740 1 client.6978 shutdown 2024-04-26T23:05:45.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1278817841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 msgr2=0x55de4f5430b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/1278817841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 0x55de4f5430b0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f7b98049040 tx=0x7f7b9802c070 comp rx=0 tx=0).stop 2024-04-26T23:05:45.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1278817841 shutdown_connections 2024-04-26T23:05:45.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/1278817841 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55de4f549c10 0x55de4f549ff0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/1278817841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f55ae80 0x55de4f5435f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/1278817841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 0x55de4f5430b0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/1278817841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f51e7b0 0x55de4f55d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1278817841 >> 172.21.15.40:0/1278817841 conn(0x55de4f4b2d10 msgr2=0x55de4f567760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7b9d600640 0 client.6978 ms_handle_reset on v2:172.21.15.40:6832/2117861921 2024-04-26T23:05:45.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1278817841 shutdown_connections 2024-04-26T23:05:45.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1278817841 wait complete. 2024-04-26T23:05:45.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.090+0000 7f7bba1dd740 20 client.6978 trim_cache size 0 max 16384 2024-04-26T23:05:45.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.109+0000 7f7bba1dd740 1 Processor -- start 2024-04-26T23:05:45.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.109+0000 7f7bba1dd740 1 -- start start 2024-04-26T23:05:45.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.110+0000 7f7bba1dd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f51e7b0 0x55de4f54b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.110+0000 7f7bba1dd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f549270 0x55de4f54b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.110+0000 7f7bba1dd740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f55ae80 0x55de4f567760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.110+0000 7f7bba1dd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb0205970 con 0x55de4f51e7b0 2024-04-26T23:05:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.110+0000 7f7bba1dd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb0206040 con 0x55de4f549270 2024-04-26T23:05:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.110+0000 7f7bba1dd740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7bb0204a40 con 0x55de4f55ae80 2024-04-26T23:05:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.110+0000 7f7bafe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f51e7b0 0x55de4f54b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.110+0000 7f7bafe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f51e7b0 0x55de4f54b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36674/0 (socket says 172.21.15.40:36674) 2024-04-26T23:05:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.110+0000 7f7bafe00640 1 -- 172.21.15.40:0/2671318999 learned_addr learned my addr 172.21.15.40:0/2671318999 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.111+0000 7f7bafe00640 1 -- 172.21.15.40:0/2671318999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f55ae80 msgr2=0x55de4f567760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.111+0000 7f7bafe00640 1 --2- 172.21.15.40:0/2671318999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f55ae80 0x55de4f567760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.111+0000 7f7bafe00640 1 -- 172.21.15.40:0/2671318999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f549270 msgr2=0x55de4f54b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.111+0000 7f7bafe00640 1 --2- 172.21.15.40:0/2671318999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f549270 0x55de4f54b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.111+0000 7f7bafe00640 1 -- 172.21.15.40:0/2671318999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de4f55bd30 con 0x55de4f51e7b0 2024-04-26T23:05:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.111+0000 7f7bafe00640 1 --2- 172.21.15.40:0/2671318999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f51e7b0 0x55de4f54b0e0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0059260 tx=0x7f7ba0000fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4c83fe41abcce2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.111+0000 7f7b9fe00640 1 -- 172.21.15.40:0/2671318999 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0088070 con 0x55de4f51e7b0 2024-04-26T23:05:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.111+0000 7f7b9fe00640 1 -- 172.21.15.40:0/2671318999 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ba0071030 con 0x55de4f51e7b0 2024-04-26T23:05:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.111+0000 7f7b9fe00640 1 -- 172.21.15.40:0/2671318999 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba00575d0 con 0x55de4f51e7b0 2024-04-26T23:05:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.112+0000 7f7bba1dd740 1 -- 172.21.15.40:0/2671318999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f51e7b0 msgr2=0x55de4f54b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.112+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/2671318999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f51e7b0 0x55de4f54b0e0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0059260 tx=0x7f7ba0000fd0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.112+0000 7f7bba1dd740 1 -- 172.21.15.40:0/2671318999 shutdown_connections 2024-04-26T23:05:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.112+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/2671318999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f55ae80 0x55de4f567760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.112+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/2671318999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f549270 0x55de4f54b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.112+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/2671318999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f51e7b0 0x55de4f54b0e0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.112+0000 7f7bba1dd740 1 -- 172.21.15.40:0/2671318999 >> 172.21.15.40:0/2671318999 conn(0x55de4f4b2d10 msgr2=0x55de4f514df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.112+0000 7f7bba1dd740 1 -- 172.21.15.40:0/2671318999 shutdown_connections 2024-04-26T23:05:45.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.112+0000 7f7bba1dd740 1 -- 172.21.15.40:0/2671318999 wait complete. 2024-04-26T23:05:45.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.113+0000 7f7bba1dd740 1 -- start start 2024-04-26T23:05:45.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.113+0000 7f7bba1dd740 1 Processor -- start 2024-04-26T23:05:45.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7bba1dd740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f51e7b0 0x55de4f5dcec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7bba1dd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 0x55de4f567760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7bba1dd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f55ae80 0x55de4f567ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7bba1dd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55de4f55e0a0 con 0x55de4f549270 2024-04-26T23:05:45.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7bba1dd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55de4f409f70 con 0x55de4f55ae80 2024-04-26T23:05:45.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7bba1dd740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55de4f5773b0 con 0x55de4f51e7b0 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7bba1dd740 10 client.0 _ll_register_callbacks cb 0x7f7b98049070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7ba7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 0x55de4f567760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7ba7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 0x55de4f567760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36678/0 (socket says 172.21.15.40:36678) 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7ba7e00640 1 -- 172.21.15.40:0/1317374070 learned_addr learned my addr 172.21.15.40:0/1317374070 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7ba7e00640 1 -- 172.21.15.40:0/1317374070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f51e7b0 msgr2=0x55de4f5dcec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/1317374070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f51e7b0 0x55de4f5dcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7ba7e00640 1 -- 172.21.15.40:0/1317374070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f55ae80 msgr2=0x55de4f567ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/1317374070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f55ae80 0x55de4f567ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.114+0000 7f7ba7e00640 1 -- 172.21.15.40:0/1317374070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de4f559d90 con 0x55de4f549270 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.115+0000 7f7ba7e00640 1 --2- 172.21.15.40:0/1317374070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 0x55de4f567760 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f7b9802a620 tx=0x7f7b980010c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e5d1657696ca3d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.115+0000 7f7b9d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.115+0000 7f7b9d600640 1 -- 172.21.15.40:0/1317374070 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9804b070 con 0x55de4f549270 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.115+0000 7f7b9d600640 1 -- 172.21.15.40:0/1317374070 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f7b98003cb0 con 0x55de4f549270 2024-04-26T23:05:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.115+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1317374070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ba0071030 con 0x55de4f549270 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.115+0000 7f7b9d600640 1 -- 172.21.15.40:0/1317374070 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f7b98066050 con 0x55de4f549270 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.115+0000 7f7bba1dd740 10 client.6984 fetch_fsmap learned FSMap version 60 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.115+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1317374070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) -- 0x55de4f55a030 con 0x55de4f549270 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.115+0000 7f7b9d600640 1 -- 172.21.15.40:0/1317374070 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ba0071030 con 0x55de4f549270 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.116+0000 7f7b9d600640 1 -- 172.21.15.40:0/1317374070 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 60) ==== 3534+0+0 (secure 0 0 0) 0x7f7b98069020 con 0x55de4f549270 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.116+0000 7f7bba1dd740 10 client.6984 fetch_fsmap finished waiting for FSMap version 60 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.116+0000 7f7bba1dd740 10 client.6984 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.116+0000 7f7bba1dd740 20 client.6984 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.116+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/1317374070 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55de4f573660 0x55de4f586560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.116+0000 7f7bba1dd740 4 client.6984 mds_command: new command op to 5614 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.116+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1317374070 --> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] -- command(tid 0: {"prefix": "status"}) -- 0x55de4f55c550 con 0x55de4f573660 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.116+0000 7f7bafe00640 1 --2- 172.21.15.40:0/1317374070 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55de4f573660 0x55de4f586560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.116+0000 7f7bafe00640 1 --2- 172.21.15.40:0/1317374070 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55de4f573660 0x55de4f586560 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=85920a7157912c37 server_cookie=ed58ea789bb335ed in_seq=0 out_seq=0 2024-04-26T23:05:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.116+0000 7f7b9d600640 10 client.6984 ms_handle_connect on v2:172.21.15.40:6832/2117861921 2024-04-26T23:05:45.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7b9d600640 1 -- 172.21.15.40:0/1317374070 <== mds.1 v2:172.21.15.40:6832/2117861921 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55de4f55c550 con 0x55de4f573660 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7b9d600640 10 client.6984 handle_command_reply: tid=0 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7bba1dd740 1 client.6984 shutdown 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1317374070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 msgr2=0x55de4f567760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/1317374070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 0x55de4f567760 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f7b9802a620 tx=0x7f7b980010c0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1317374070 shutdown_connections 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/1317374070 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55de4f573660 0x55de4f586560 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/1317374070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de4f55ae80 0x55de4f567ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/1317374070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de4f549270 0x55de4f567760 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7bba1dd740 1 --2- 172.21.15.40:0/1317374070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55de4f51e7b0 0x55de4f5dcec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7b9d600640 0 client.6984 ms_handle_reset on v2:172.21.15.40:6832/2117861921 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1317374070 >> 172.21.15.40:0/1317374070 conn(0x55de4f4b2d10 msgr2=0x55de4f54b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1317374070 shutdown_connections 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7bba1dd740 1 -- 172.21.15.40:0/1317374070 wait complete. 2024-04-26T23:05:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.117+0000 7f7bba1dd740 20 client.6984 trim_cache size 0 max 16384 2024-04-26T23:05:45.121 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:05:45.121 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:05:45.121 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-04-26T23:05:45.121 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5614, 2024-04-26T23:05:45.121 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:05:45.121 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:05:45.121 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:05:45.121 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.2992556400000002, 2024-04-26T23:05:45.121 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 60, 2024-04-26T23:05:45.121 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 65, 2024-04-26T23:05:45.121 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 65, 2024-04-26T23:05:45.121 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 17.782797405 2024-04-26T23:05:45.122 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:05:45.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.118+0000 7f7bb5c00640 1 -- 172.21.15.40:0/2398785517 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7b8c06c920 msgr2=0x7f7b8c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.118+0000 7f7bb5c00640 1 --2- 172.21.15.40:0/2398785517 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7b8c06c920 0x7f7b8c06ede0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7b98032000 tx=0x7f7b9802c000 comp rx=0 tx=0).stop 2024-04-26T23:05:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.118+0000 7f7bb5c00640 1 -- 172.21.15.40:0/2398785517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b94004800 msgr2=0x7f7bb00720d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.118+0000 7f7bb5c00640 1 --2- 172.21.15.40:0/2398785517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b94004800 0x7f7bb00720d0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000d810 tx=0x7f7ba000dcd0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.118+0000 7f7bb5c00640 1 -- 172.21.15.40:0/2398785517 shutdown_connections 2024-04-26T23:05:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.118+0000 7f7bb5c00640 1 --2- 172.21.15.40:0/2398785517 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7b8c06c920 0x7f7b8c06ede0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.118+0000 7f7bb5c00640 1 --2- 172.21.15.40:0/2398785517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bb015a9c0 0x7f7bb015ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.118+0000 7f7bb5c00640 1 --2- 172.21.15.40:0/2398785517 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bb015c410 0x7f7bb015c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.118+0000 7f7bb5c00640 1 --2- 172.21.15.40:0/2398785517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b94004800 0x7f7bb00720d0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.118+0000 7f7bb5c00640 1 -- 172.21.15.40:0/2398785517 >> 172.21.15.40:0/2398785517 conn(0x7f7bb00d0a10 msgr2=0x7f7bb006e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.118+0000 7f7bb5c00640 1 -- 172.21.15.40:0/2398785517 shutdown_connections 2024-04-26T23:05:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.118+0000 7f7bb5c00640 1 -- 172.21.15.40:0/2398785517 wait complete. 2024-04-26T23:05:45.135 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T23:05:45.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.332+0000 7f4d2a800640 1 Processor -- start 2024-04-26T23:05:45.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.333+0000 7f4d2a800640 1 -- start start 2024-04-26T23:05:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.334+0000 7f4d2a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4d24071e60 0x7f4d2406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.334+0000 7f4d2a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4d2406fe20 0x7f4d24072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.334+0000 7f4d2a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d24072780 0x7f4d240cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.334+0000 7f4d2a800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f4d2415d7c0 con 0x7f4d24071e60 2024-04-26T23:05:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.334+0000 7f4d2a800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4d2415daa0 con 0x7f4d2406fe20 2024-04-26T23:05:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.334+0000 7f4d2a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4d2415dd80 con 0x7f4d24072780 2024-04-26T23:05:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.335+0000 7f4d23400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4d24071e60 0x7f4d2406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.335+0000 7f4d23400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4d24071e60 0x7f4d2406f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36694/0 (socket says 172.21.15.40:36694) 2024-04-26T23:05:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.335+0000 7f4d23400640 1 -- 172.21.15.40:0/3086777653 learned_addr learned my addr 172.21.15.40:0/3086777653 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.335+0000 7f4d1be00640 1 --2- 172.21.15.40:0/3086777653 >> v2:172.21.15.103:3300/0 conn(0x7f4d2406fe20 0x7f4d24072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.335+0000 7f4d23e00640 1 --2- 172.21.15.40:0/3086777653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d24072780 0x7f4d240cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.335+0000 7f4d23400640 1 -- 172.21.15.40:0/3086777653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d24072780 msgr2=0x7f4d240cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.335+0000 7f4d23400640 1 --2- 172.21.15.40:0/3086777653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d24072780 0x7f4d240cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.335+0000 7f4d23400640 1 -- 172.21.15.40:0/3086777653 >> v2:172.21.15.103:3300/0 conn(0x7f4d2406fe20 msgr2=0x7f4d24072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.335+0000 7f4d23400640 1 --2- 172.21.15.40:0/3086777653 >> v2:172.21.15.103:3300/0 conn(0x7f4d2406fe20 0x7f4d24072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.335+0000 7f4d23400640 1 -- 172.21.15.40:0/3086777653 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2415e060 con 0x7f4d24071e60 2024-04-26T23:05:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.336+0000 7f4d23400640 1 --2- 172.21.15.40:0/3086777653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d24071e60 0x7f4d2406f8e0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f4d240b3820 tx=0x7f4d1402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913fc9702086eae8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.337+0000 7f4d22a00640 1 -- 172.21.15.40:0/3086777653 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d14037070 con 0x7f4d24071e60 2024-04-26T23:05:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.337+0000 7f4d22a00640 1 -- 172.21.15.40:0/3086777653 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d14032440 con 0x7f4d24071e60 2024-04-26T23:05:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.337+0000 7f4d22a00640 1 -- 172.21.15.40:0/3086777653 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d14032810 con 0x7f4d24071e60 2024-04-26T23:05:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.338+0000 7f4d2a800640 1 -- 172.21.15.40:0/3086777653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d24071e60 msgr2=0x7f4d2406f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.338+0000 7f4d2a800640 1 --2- 172.21.15.40:0/3086777653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d24071e60 0x7f4d2406f8e0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f4d240b3820 tx=0x7f4d1402a1e0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.338+0000 7f4d2a800640 1 -- 172.21.15.40:0/3086777653 shutdown_connections 2024-04-26T23:05:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.338+0000 7f4d2a800640 1 --2- 172.21.15.40:0/3086777653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d24072780 0x7f4d240cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.338+0000 7f4d2a800640 1 --2- 172.21.15.40:0/3086777653 >> v2:172.21.15.103:3300/0 conn(0x7f4d2406fe20 0x7f4d24072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.338+0000 7f4d2a800640 1 --2- 172.21.15.40:0/3086777653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d24071e60 0x7f4d2406f8e0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.338+0000 7f4d2a800640 1 -- 172.21.15.40:0/3086777653 >> 172.21.15.40:0/3086777653 conn(0x7f4d240d0be0 msgr2=0x7f4d240d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.338+0000 7f4d2a800640 1 -- 172.21.15.40:0/3086777653 shutdown_connections 2024-04-26T23:05:45.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.338+0000 7f4d2a800640 1 -- 172.21.15.40:0/3086777653 wait complete. 2024-04-26T23:05:45.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.339+0000 7f4d2a800640 1 Processor -- start 2024-04-26T23:05:45.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.340+0000 7f4d2a800640 1 -- start start 2024-04-26T23:05:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.340+0000 7f4d2a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d2406fe20 0x7f4d2414ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.340+0000 7f4d2a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24071e60 0x7f4d240c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.340+0000 7f4d2a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d24072780 0x7f4d240ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.340+0000 7f4d2a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4d24072b90 con 0x7f4d2406fe20 2024-04-26T23:05:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.340+0000 7f4d2a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4d24070200 con 0x7f4d24071e60 2024-04-26T23:05:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.340+0000 7f4d2a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4d2415d3f0 con 0x7f4d24072780 2024-04-26T23:05:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.341+0000 7f4d23400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d2406fe20 0x7f4d2414ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.341+0000 7f4d23400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d2406fe20 0x7f4d2414ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36710/0 (socket says 172.21.15.40:36710) 2024-04-26T23:05:45.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.341+0000 7f4d23400640 1 -- 172.21.15.40:0/646870724 learned_addr learned my addr 172.21.15.40:0/646870724 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:45.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.341+0000 7f4d23e00640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d24072780 0x7f4d240ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.341+0000 7f4d1be00640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24071e60 0x7f4d240c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.341+0000 7f4d23400640 1 -- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d24072780 msgr2=0x7f4d240ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.341+0000 7f4d23400640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d24072780 0x7f4d240ca2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.341+0000 7f4d23400640 1 -- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24071e60 msgr2=0x7f4d240c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.341+0000 7f4d23400640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24071e60 0x7f4d240c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.341+0000 7f4d23400640 1 -- 172.21.15.40:0/646870724 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d240b54b0 con 0x7f4d2406fe20 2024-04-26T23:05:45.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.342+0000 7f4d23400640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d2406fe20 0x7f4d2414ce80 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f4d1402a220 tx=0x7f4d140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.342+0000 7f4d1b400640 1 -- 172.21.15.40:0/646870724 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d14041070 con 0x7f4d2406fe20 2024-04-26T23:05:45.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.342+0000 7f4d1b400640 1 -- 172.21.15.40:0/646870724 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d1403c870 con 0x7f4d2406fe20 2024-04-26T23:05:45.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.342+0000 7f4d2a800640 1 -- 172.21.15.40:0/646870724 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d240b5790 con 0x7f4d2406fe20 2024-04-26T23:05:45.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.342+0000 7f4d1b400640 1 -- 172.21.15.40:0/646870724 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d14037040 con 0x7f4d2406fe20 2024-04-26T23:05:45.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.342+0000 7f4d2a800640 1 -- 172.21.15.40:0/646870724 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d240c8d80 con 0x7f4d2406fe20 2024-04-26T23:05:45.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.344+0000 7f4d1b400640 1 -- 172.21.15.40:0/646870724 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4d14044070 con 0x7f4d2406fe20 2024-04-26T23:05:45.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.345+0000 7f4d1b400640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4d0006c8f0 0x7f4d0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.345+0000 7f4d1b400640 1 -- 172.21.15.40:0/646870724 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f4d140acc30 con 0x7f4d2406fe20 2024-04-26T23:05:45.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.346+0000 7f4d2d79a740 1 Processor -- start 2024-04-26T23:05:45.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.346+0000 7f4d1be00640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4d0006c8f0 0x7f4d0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.346+0000 7f4d2d79a740 1 -- start start 2024-04-26T23:05:45.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d2d79a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25d655e0 0x55ab25d67740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d2d79a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25da1cb0 0x55ab25d659c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d2d79a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25d900a0 0x55ab25d90520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d2d79a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ab25cc9600 con 0x55ab25da1cb0 2024-04-26T23:05:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d2d79a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ab25c66100 con 0x55ab25d655e0 2024-04-26T23:05:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d2d79a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55ab25cc55c0 con 0x55ab25d900a0 2024-04-26T23:05:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d1be00640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4d0006c8f0 0x7f4d0006edb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4d240d07a0 tx=0x7f4d0c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d1be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25da1cb0 0x55ab25d659c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d1be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25da1cb0 0x55ab25d659c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36718/0 (socket says 172.21.15.40:36718) 2024-04-26T23:05:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d1be00640 1 -- 172.21.15.40:0/1785829354 learned_addr learned my addr 172.21.15.40:0/1785829354 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d1be00640 1 -- 172.21.15.40:0/1785829354 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25d900a0 msgr2=0x55ab25d90520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d1be00640 1 --2- 172.21.15.40:0/1785829354 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25d900a0 0x55ab25d90520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d1be00640 1 -- 172.21.15.40:0/1785829354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25d655e0 msgr2=0x55ab25d67740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d1be00640 1 --2- 172.21.15.40:0/1785829354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25d655e0 0x55ab25d67740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.347+0000 7f4d1be00640 1 -- 172.21.15.40:0/1785829354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab259db2d0 con 0x55ab25da1cb0 2024-04-26T23:05:45.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.348+0000 7f4d1be00640 1 --2- 172.21.15.40:0/1785829354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25da1cb0 0x55ab25d659c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00fa60 tx=0x7f4d0c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f51893f46cf99d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.348+0000 7f4d13e00640 1 -- 172.21.15.40:0/1785829354 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c01b050 con 0x55ab25da1cb0 2024-04-26T23:05:45.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.348+0000 7f4d13e00640 1 -- 172.21.15.40:0/1785829354 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d0c025070 con 0x55ab25da1cb0 2024-04-26T23:05:45.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.348+0000 7f4d13e00640 1 -- 172.21.15.40:0/1785829354 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c014e00 con 0x55ab25da1cb0 2024-04-26T23:05:45.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.349+0000 7f4d2d79a740 1 -- 172.21.15.40:0/1785829354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25da1cb0 msgr2=0x55ab25d659c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.349+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/1785829354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25da1cb0 0x55ab25d659c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00fa60 tx=0x7f4d0c012000 comp rx=0 tx=0).stop 2024-04-26T23:05:45.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.349+0000 7f4d2d79a740 1 -- 172.21.15.40:0/1785829354 shutdown_connections 2024-04-26T23:05:45.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.349+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/1785829354 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25d900a0 0x55ab25d90520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.349+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/1785829354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25da1cb0 0x55ab25d659c0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.349+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/1785829354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25d655e0 0x55ab25d67740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.349+0000 7f4d2d79a740 1 -- 172.21.15.40:0/1785829354 >> 172.21.15.40:0/1785829354 conn(0x55ab25cf9d10 msgr2=0x55ab25dab0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.349+0000 7f4d2d79a740 1 -- 172.21.15.40:0/1785829354 shutdown_connections 2024-04-26T23:05:45.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.349+0000 7f4d2d79a740 1 -- 172.21.15.40:0/1785829354 wait complete. 2024-04-26T23:05:45.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.350+0000 7f4d2d79a740 1 -- start start 2024-04-26T23:05:45.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.350+0000 7f4d2d79a740 1 Processor -- start 2024-04-26T23:05:45.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d2d79a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d655e0 0x55ab25da3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d2d79a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25d900a0 0x55ab25d79380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d2d79a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25da1cb0 0x55ab25d8c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d2d79a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ab25c50f70 con 0x55ab25d655e0 2024-04-26T23:05:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d2d79a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ab25d8dbf0 con 0x55ab25da1cb0 2024-04-26T23:05:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d2d79a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55ab25d91bd0 con 0x55ab25d900a0 2024-04-26T23:05:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d2d79a740 10 client.0 _ll_register_callbacks cb 0x55ab25cbf480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d23400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d655e0 0x55ab25da3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d23400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d655e0 0x55ab25da3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36724/0 (socket says 172.21.15.40:36724) 2024-04-26T23:05:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d23400640 1 -- 172.21.15.40:0/4039172186 learned_addr learned my addr 172.21.15.40:0/4039172186 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d23400640 1 -- 172.21.15.40:0/4039172186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25d900a0 msgr2=0x55ab25d79380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d23400640 1 --2- 172.21.15.40:0/4039172186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25d900a0 0x55ab25d79380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d23400640 1 -- 172.21.15.40:0/4039172186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25da1cb0 msgr2=0x55ab25d8c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d23400640 1 --2- 172.21.15.40:0/4039172186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25da1cb0 0x55ab25d8c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.351+0000 7f4d23400640 1 -- 172.21.15.40:0/4039172186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab25aa56b0 con 0x55ab25d655e0 2024-04-26T23:05:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.352+0000 7f4d23400640 1 --2- 172.21.15.40:0/4039172186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d655e0 0x55ab25da3de0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4d1407a550 tx=0x7f4d1407b010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b43a5f9ff96eac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.352+0000 7f4d10c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.352+0000 7f4d10c00640 1 -- 172.21.15.40:0/4039172186 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d14083430 con 0x55ab25d655e0 2024-04-26T23:05:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.352+0000 7f4d10c00640 1 -- 172.21.15.40:0/4039172186 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f4d14081070 con 0x55ab25d655e0 2024-04-26T23:05:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.352+0000 7f4d2d79a740 1 -- 172.21.15.40:0/4039172186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d0c025070 con 0x55ab25d655e0 2024-04-26T23:05:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.352+0000 7f4d10c00640 1 -- 172.21.15.40:0/4039172186 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f4d14082fa0 con 0x55ab25d655e0 2024-04-26T23:05:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.352+0000 7f4d2d79a740 10 client.6996 fetch_fsmap learned FSMap version 60 2024-04-26T23:05:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.352+0000 7f4d2d79a740 1 -- 172.21.15.40:0/4039172186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) -- 0x55ab25d91910 con 0x55ab25d655e0 2024-04-26T23:05:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.352+0000 7f4d10c00640 1 -- 172.21.15.40:0/4039172186 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d0c025070 con 0x55ab25d655e0 2024-04-26T23:05:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.352+0000 7f4d10c00640 1 -- 172.21.15.40:0/4039172186 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 60) ==== 3534+0+0 (secure 0 0 0) 0x7f4d1407bb80 con 0x55ab25d655e0 2024-04-26T23:05:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.353+0000 7f4d2d79a740 10 client.6996 fetch_fsmap finished waiting for FSMap version 60 2024-04-26T23:05:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.353+0000 7f4d2d79a740 10 client.6996 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:05:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.353+0000 7f4d2d79a740 20 client.6996 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:45.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.353+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/4039172186 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55ab25d7bc50 0x55ab25d90af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.353+0000 7f4d2d79a740 4 client.6996 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:05:45.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.353+0000 7f4d2d79a740 1 -- 172.21.15.40:0/4039172186 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab25cd7210 con 0x55ab25d7bc50 2024-04-26T23:05:45.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.353+0000 7f4d1be00640 1 --2- 172.21.15.40:0/4039172186 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55ab25d7bc50 0x55ab25d90af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.353+0000 7f4d1be00640 1 --2- 172.21.15.40:0/4039172186 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55ab25d7bc50 0x55ab25d90af0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ebcb7ee94adf046 server_cookie=f21f3f167a548a7f in_seq=0 out_seq=0 2024-04-26T23:05:45.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.353+0000 7f4d10c00640 10 client.6996 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:05:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.354+0000 7f4d10c00640 1 -- 172.21.15.40:0/4039172186 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ab25cd7210 con 0x55ab25d7bc50 2024-04-26T23:05:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.354+0000 7f4d10c00640 10 client.6996 handle_command_reply: tid=0 2024-04-26T23:05:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.354+0000 7f4d2d79a740 1 client.6996 shutdown 2024-04-26T23:05:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.354+0000 7f4d2d79a740 1 -- 172.21.15.40:0/4039172186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d655e0 msgr2=0x55ab25da3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.354+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/4039172186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d655e0 0x55ab25da3de0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4d1407a550 tx=0x7f4d1407b010 comp rx=0 tx=0).stop 2024-04-26T23:05:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.354+0000 7f4d2d79a740 1 -- 172.21.15.40:0/4039172186 shutdown_connections 2024-04-26T23:05:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.354+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/4039172186 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55ab25d7bc50 0x55ab25d90af0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.354+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/4039172186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25da1cb0 0x55ab25d8c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.354+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/4039172186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25d900a0 0x55ab25d79380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.354+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/4039172186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d655e0 0x55ab25da3de0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.354+0000 7f4d10c00640 0 client.6996 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:05:45.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.355+0000 7f4d2d79a740 1 -- 172.21.15.40:0/4039172186 >> 172.21.15.40:0/4039172186 conn(0x55ab25cf9d10 msgr2=0x55ab25d8dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.355+0000 7f4d2d79a740 1 -- 172.21.15.40:0/4039172186 shutdown_connections 2024-04-26T23:05:45.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.355+0000 7f4d2d79a740 1 -- 172.21.15.40:0/4039172186 wait complete. 2024-04-26T23:05:45.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.355+0000 7f4d2d79a740 20 client.6996 trim_cache size 0 max 16384 2024-04-26T23:05:45.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.375+0000 7f4d2d79a740 1 Processor -- start 2024-04-26T23:05:45.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.375+0000 7f4d2d79a740 1 -- start start 2024-04-26T23:05:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.375+0000 7f4d2d79a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25d655e0 0x55ab25da22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.375+0000 7f4d2d79a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 0x55ab25dae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.375+0000 7f4d2d79a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25da1cb0 0x55ab25dae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.376+0000 7f4d2d79a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4d24200f60 con 0x55ab25d900a0 2024-04-26T23:05:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.376+0000 7f4d2d79a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4d242015f0 con 0x55ab25d655e0 2024-04-26T23:05:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.376+0000 7f4d2d79a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4d241f6bf0 con 0x55ab25da1cb0 2024-04-26T23:05:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.376+0000 7f4d1be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 0x55ab25dae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.376+0000 7f4d1be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 0x55ab25dae1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36734/0 (socket says 172.21.15.40:36734) 2024-04-26T23:05:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.376+0000 7f4d1be00640 1 -- 172.21.15.40:0/4206714531 learned_addr learned my addr 172.21.15.40:0/4206714531 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:45.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.376+0000 7f4d1be00640 1 -- 172.21.15.40:0/4206714531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25da1cb0 msgr2=0x55ab25dae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.376+0000 7f4d1be00640 1 --2- 172.21.15.40:0/4206714531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25da1cb0 0x55ab25dae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.376+0000 7f4d23400640 1 --2- 172.21.15.40:0/4206714531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25d655e0 0x55ab25da22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.376+0000 7f4d1be00640 1 -- 172.21.15.40:0/4206714531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25d655e0 msgr2=0x55ab25da22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.376+0000 7f4d1be00640 1 --2- 172.21.15.40:0/4206714531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25d655e0 0x55ab25da22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.376+0000 7f4d1be00640 1 -- 172.21.15.40:0/4206714531 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab25d7b4d0 con 0x55ab25d900a0 2024-04-26T23:05:45.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.377+0000 7f4d1be00640 1 --2- 172.21.15.40:0/4206714531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 0x55ab25dae1e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c021000 tx=0x7f4d0c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf824a0507a7f4d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.377+0000 7f4d13400640 1 -- 172.21.15.40:0/4206714531 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c01b040 con 0x55ab25d900a0 2024-04-26T23:05:45.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.377+0000 7f4d13400640 1 -- 172.21.15.40:0/4206714531 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d0c03a070 con 0x55ab25d900a0 2024-04-26T23:05:45.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.377+0000 7f4d13400640 1 -- 172.21.15.40:0/4206714531 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c0347f0 con 0x55ab25d900a0 2024-04-26T23:05:45.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.378+0000 7f4d2d79a740 1 -- 172.21.15.40:0/4206714531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 msgr2=0x55ab25dae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.378+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/4206714531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 0x55ab25dae1e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c021000 tx=0x7f4d0c014000 comp rx=0 tx=0).stop 2024-04-26T23:05:45.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.378+0000 7f4d2d79a740 1 -- 172.21.15.40:0/4206714531 shutdown_connections 2024-04-26T23:05:45.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.378+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/4206714531 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25da1cb0 0x55ab25dae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.378+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/4206714531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 0x55ab25dae1e0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.378+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/4206714531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25d655e0 0x55ab25da22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.378+0000 7f4d2d79a740 1 -- 172.21.15.40:0/4206714531 >> 172.21.15.40:0/4206714531 conn(0x55ab25cf9d10 msgr2=0x55ab25d8c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.378+0000 7f4d2d79a740 1 -- 172.21.15.40:0/4206714531 shutdown_connections 2024-04-26T23:05:45.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.378+0000 7f4d2d79a740 1 -- 172.21.15.40:0/4206714531 wait complete. 2024-04-26T23:05:45.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.378+0000 7f4d2d79a740 1 -- start start 2024-04-26T23:05:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.379+0000 7f4d2d79a740 1 Processor -- start 2024-04-26T23:05:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.379+0000 7f4d2d79a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25d655e0 0x55ab25d90890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d2d79a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 0x55ab25dba130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d2d79a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25da1cb0 0x55ab25d64360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d2d79a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ab25d7bad0 con 0x55ab25d900a0 2024-04-26T23:05:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d2d79a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ab25d8ce70 con 0x55ab25da1cb0 2024-04-26T23:05:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d2d79a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55ab25d7c120 con 0x55ab25d655e0 2024-04-26T23:05:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d2d79a740 10 client.0 _ll_register_callbacks cb 0x7f4d140091c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:05:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d1be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 0x55ab25dba130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d1be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 0x55ab25dba130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36744/0 (socket says 172.21.15.40:36744) 2024-04-26T23:05:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d1be00640 1 -- 172.21.15.40:0/2148259326 learned_addr learned my addr 172.21.15.40:0/2148259326 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:45.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d1be00640 1 -- 172.21.15.40:0/2148259326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25d655e0 msgr2=0x55ab25d90890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d1be00640 1 --2- 172.21.15.40:0/2148259326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25d655e0 0x55ab25d90890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d1be00640 1 -- 172.21.15.40:0/2148259326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25da1cb0 msgr2=0x55ab25d64360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d1be00640 1 --2- 172.21.15.40:0/2148259326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25da1cb0 0x55ab25d64360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d1be00640 1 -- 172.21.15.40:0/2148259326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab25d61660 con 0x55ab25d900a0 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d1be00640 1 --2- 172.21.15.40:0/2148259326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 0x55ab25dba130 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c037040 tx=0x7f4d0c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40658da49514b1b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.380+0000 7f4d10c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.381+0000 7f4d10c00640 1 -- 172.21.15.40:0/2148259326 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c01b040 con 0x55ab25d900a0 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.381+0000 7f4d10c00640 1 -- 172.21.15.40:0/2148259326 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f4d0c005da0 con 0x55ab25d900a0 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.381+0000 7f4d2d79a740 1 -- 172.21.15.40:0/2148259326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d0c03a070 con 0x55ab25d900a0 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.381+0000 7f4d10c00640 1 -- 172.21.15.40:0/2148259326 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f4d0c04b050 con 0x55ab25d900a0 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.381+0000 7f4d2d79a740 10 client.7002 fetch_fsmap learned FSMap version 60 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.381+0000 7f4d2d79a740 1 -- 172.21.15.40:0/2148259326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) -- 0x55ab25da0f70 con 0x55ab25d900a0 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.381+0000 7f4d10c00640 1 -- 172.21.15.40:0/2148259326 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d0c03a070 con 0x55ab25d900a0 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.381+0000 7f4d10c00640 1 -- 172.21.15.40:0/2148259326 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 60) ==== 3534+0+0 (secure 0 0 0) 0x7f4d0c04e020 con 0x55ab25d900a0 2024-04-26T23:05:45.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.381+0000 7f4d2d79a740 10 client.7002 fetch_fsmap finished waiting for FSMap version 60 2024-04-26T23:05:45.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.381+0000 7f4d2d79a740 10 client.7002 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:05:45.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.381+0000 7f4d2d79a740 20 client.7002 populate_metadata read hostname 'smithi040' 2024-04-26T23:05:45.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.382+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/2148259326 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55ab25d78330 0x55ab25d78710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.382+0000 7f4d2d79a740 4 client.7002 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:05:45.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.382+0000 7f4d2d79a740 1 -- 172.21.15.40:0/2148259326 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "status"}) -- 0x55ab25cd1080 con 0x55ab25d78330 2024-04-26T23:05:45.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.382+0000 7f4d23400640 1 --2- 172.21.15.40:0/2148259326 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55ab25d78330 0x55ab25d78710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.382+0000 7f4d23400640 1 --2- 172.21.15.40:0/2148259326 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55ab25d78330 0x55ab25d78710 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55e671db40e19aa9 server_cookie=8eda2dae7073de04 in_seq=0 out_seq=0 2024-04-26T23:05:45.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.382+0000 7f4d10c00640 10 client.7002 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:05:45.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d10c00640 1 -- 172.21.15.40:0/2148259326 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55ab25cd1080 con 0x55ab25d78330 2024-04-26T23:05:45.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d10c00640 10 client.7002 handle_command_reply: tid=0 2024-04-26T23:05:45.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d2d79a740 1 client.7002 shutdown 2024-04-26T23:05:45.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d2d79a740 1 -- 172.21.15.40:0/2148259326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 msgr2=0x55ab25dba130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/2148259326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 0x55ab25dba130 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c037040 tx=0x7f4d0c004730 comp rx=0 tx=0).stop 2024-04-26T23:05:45.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d2d79a740 1 -- 172.21.15.40:0/2148259326 shutdown_connections 2024-04-26T23:05:45.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/2148259326 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55ab25d78330 0x55ab25d78710 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/2148259326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ab25da1cb0 0x55ab25d64360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/2148259326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab25d900a0 0x55ab25dba130 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d2d79a740 1 --2- 172.21.15.40:0/2148259326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ab25d655e0 0x55ab25d90890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d10c00640 0 client.7002 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:05:45.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d2d79a740 1 -- 172.21.15.40:0/2148259326 >> 172.21.15.40:0/2148259326 conn(0x55ab25cf9d10 msgr2=0x55ab25d90f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d2d79a740 1 -- 172.21.15.40:0/2148259326 shutdown_connections 2024-04-26T23:05:45.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d2d79a740 1 -- 172.21.15.40:0/2148259326 wait complete. 2024-04-26T23:05:45.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.383+0000 7f4d2d79a740 20 client.7002 trim_cache size 0 max 16384 2024-04-26T23:05:45.388 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:05:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:05:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:05:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6774, 2024-04-26T23:05:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:05:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:05:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:05:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 12.636979907000001, 2024-04-26T23:05:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 60, 2024-04-26T23:05:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 65, 2024-04-26T23:05:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 65, 2024-04-26T23:05:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 18.048509023000001 2024-04-26T23:05:45.389 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:05:45.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.384+0000 7f4d2a800640 1 -- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4d0006c8f0 msgr2=0x7f4d0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.384+0000 7f4d2a800640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4d0006c8f0 0x7f4d0006edb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4d240d07a0 tx=0x7f4d0c008040 comp rx=0 tx=0).stop 2024-04-26T23:05:45.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.384+0000 7f4d2a800640 1 -- 172.21.15.40:0/646870724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d2406fe20 msgr2=0x7f4d2414ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.384+0000 7f4d2a800640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d2406fe20 0x7f4d2414ce80 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f4d1402a220 tx=0x7f4d140046c0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.384+0000 7f4d2a800640 1 -- 172.21.15.40:0/646870724 shutdown_connections 2024-04-26T23:05:45.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.384+0000 7f4d2a800640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4d0006c8f0 0x7f4d0006edb0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.384+0000 7f4d2a800640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4d24072780 0x7f4d240ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.384+0000 7f4d2a800640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24071e60 0x7f4d240c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.384+0000 7f4d2a800640 1 --2- 172.21.15.40:0/646870724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d2406fe20 0x7f4d2414ce80 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.384+0000 7f4d2a800640 1 -- 172.21.15.40:0/646870724 >> 172.21.15.40:0/646870724 conn(0x7f4d240d0be0 msgr2=0x7f4d240d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.384+0000 7f4d2a800640 1 -- 172.21.15.40:0/646870724 shutdown_connections 2024-04-26T23:05:45.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.384+0000 7f4d2a800640 1 -- 172.21.15.40:0/646870724 wait complete. 2024-04-26T23:05:45.400 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:05:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.599+0000 7f3397200640 1 Processor -- start 2024-04-26T23:05:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.600+0000 7f3397200640 1 -- start start 2024-04-26T23:05:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.601+0000 7f3397200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33900b6220 0x7f33900d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.601+0000 7f3397200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33900d08d0 0x7f33900d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.601+0000 7f3397200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3390162c40 con 0x7f33900d3820 2024-04-26T23:05:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.601+0000 7f3397200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f33900d2d80 con 0x7f33900b6220 2024-04-26T23:05:45.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.601+0000 7f3397200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33900b7c80 con 0x7f33900d08d0 2024-04-26T23:05:45.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.601+0000 7f3396800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f33900d3820 0x7f33900b6810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60640/0 (socket says 172.21.15.40:60640) 2024-04-26T23:05:45.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.602+0000 7f3396800640 1 -- 172.21.15.40:0/1841383579 learned_addr learned my addr 172.21.15.40:0/1841383579 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:05:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.602+0000 7f3395e00640 1 --2- 172.21.15.40:0/1841383579 >> v2:172.21.15.40:3300/0 conn(0x7f33900b6220 0x7f33900d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.602+0000 7f3395400640 1 --2- 172.21.15.40:0/1841383579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33900d08d0 0x7f33900d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.602+0000 7f3394a00640 1 -- 172.21.15.40:0/1841383579 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 377519770 0 0) 0x7f3390162c40 con 0x7f33900d3820 2024-04-26T23:05:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.602+0000 7f3394a00640 1 -- 172.21.15.40:0/1841383579 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3378003520 con 0x7f33900d3820 2024-04-26T23:05:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.602+0000 7f3395e00640 1 -- 172.21.15.40:0/1841383579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33900d08d0 msgr2=0x7f33900d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:05:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.602+0000 7f3395e00640 1 --2- 172.21.15.40:0/1841383579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33900d08d0 0x7f33900d32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.602+0000 7f3394a00640 1 -- 172.21.15.40:0/1841383579 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3704555802 0 0) 0x7f3378003520 con 0x7f33900d3820 2024-04-26T23:05:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.602+0000 7f3395e00640 1 -- 172.21.15.40:0/1841383579 >> v1:172.21.15.40:6789/0 conn(0x7f33900d3820 legacy=0x7f33900b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.602+0000 7f3395e00640 1 -- 172.21.15.40:0/1841383579 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33900b7f60 con 0x7f33900b6220 2024-04-26T23:05:45.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.604+0000 7f3395e00640 1 --2- 172.21.15.40:0/1841383579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33900b6220 0x7f33900d0390 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f33840097a0 tx=0x7f338402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ebc24aba93454a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.604+0000 7f3394a00640 1 -- 172.21.15.40:0/1841383579 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3384037070 con 0x7f33900b6220 2024-04-26T23:05:45.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.604+0000 7f3394a00640 1 -- 172.21.15.40:0/1841383579 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f338402aba0 con 0x7f33900b6220 2024-04-26T23:05:45.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.604+0000 7f3394a00640 1 -- 172.21.15.40:0/1841383579 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338c0019f0 con 0x7f33900b6220 2024-04-26T23:05:45.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.605+0000 7f3397200640 1 -- 172.21.15.40:0/1841383579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33900b6220 msgr2=0x7f33900d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.605+0000 7f3397200640 1 --2- 172.21.15.40:0/1841383579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33900b6220 0x7f33900d0390 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f33840097a0 tx=0x7f338402a110 comp rx=0 tx=0).stop 2024-04-26T23:05:45.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.605+0000 7f3397200640 1 -- 172.21.15.40:0/1841383579 shutdown_connections 2024-04-26T23:05:45.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.605+0000 7f3397200640 1 --2- 172.21.15.40:0/1841383579 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33900d08d0 0x7f33900d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.605+0000 7f3397200640 1 --2- 172.21.15.40:0/1841383579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33900b6220 0x7f33900d0390 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.605+0000 7f3397200640 1 -- 172.21.15.40:0/1841383579 >> 172.21.15.40:0/1841383579 conn(0x7f33900d4d90 msgr2=0x7f33900d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.605+0000 7f3397200640 1 -- 172.21.15.40:0/1841383579 shutdown_connections 2024-04-26T23:05:45.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.605+0000 7f3397200640 1 -- 172.21.15.40:0/1841383579 wait complete. 2024-04-26T23:05:45.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.607+0000 7f3397200640 1 Processor -- start 2024-04-26T23:05:45.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.607+0000 7f3397200640 1 -- start start 2024-04-26T23:05:45.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.608+0000 7f3397200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33900b6220 0x7f33900b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.608+0000 7f3397200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33900d08d0 0x7f339009f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.608+0000 7f3397200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33900d3820 0x7f33900a1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.608+0000 7f3397200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f33900b9d30 con 0x7f33900b6220 2024-04-26T23:05:45.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.608+0000 7f3397200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33900b8520 con 0x7f33900d08d0 2024-04-26T23:05:45.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.608+0000 7f3397200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33900b9eb0 con 0x7f33900d3820 2024-04-26T23:05:45.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.608+0000 7f3395e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33900b6220 0x7f33900b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.608+0000 7f3395e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33900b6220 0x7f33900b4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36756/0 (socket says 172.21.15.40:36756) 2024-04-26T23:05:45.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.608+0000 7f3395e00640 1 -- 172.21.15.40:0/436711059 learned_addr learned my addr 172.21.15.40:0/436711059 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:05:45.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.609+0000 7f3396800640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33900d3820 0x7f33900a1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.609+0000 7f3395400640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33900d08d0 0x7f339009f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.609+0000 7f3395e00640 1 -- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33900d3820 msgr2=0x7f33900a1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.609+0000 7f3395e00640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33900d3820 0x7f33900a1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.609+0000 7f3395e00640 1 -- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33900d08d0 msgr2=0x7f339009f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.609+0000 7f3395e00640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33900d08d0 0x7f339009f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.609+0000 7f3395e00640 1 -- 172.21.15.40:0/436711059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339009fef0 con 0x7f33900b6220 2024-04-26T23:05:45.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.609+0000 7f3395e00640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33900b6220 0x7f33900b4b20 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f33840098c0 tx=0x7f33840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.609+0000 7f338a000640 1 -- 172.21.15.40:0/436711059 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3384037050 con 0x7f33900b6220 2024-04-26T23:05:45.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.610+0000 7f338a000640 1 -- 172.21.15.40:0/436711059 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3384038040 con 0x7f33900b6220 2024-04-26T23:05:45.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.610+0000 7f338a000640 1 -- 172.21.15.40:0/436711059 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3384004490 con 0x7f33900b6220 2024-04-26T23:05:45.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.610+0000 7f3397200640 1 -- 172.21.15.40:0/436711059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33900a01d0 con 0x7f33900b6220 2024-04-26T23:05:45.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.610+0000 7f3397200640 1 -- 172.21.15.40:0/436711059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33900a0790 con 0x7f33900b6220 2024-04-26T23:05:45.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.611+0000 7f3397200640 1 -- 172.21.15.40:0/436711059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3390069370 con 0x7f33900b6220 2024-04-26T23:05:45.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.612+0000 7f338a000640 1 -- 172.21.15.40:0/436711059 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3384003610 con 0x7f33900b6220 2024-04-26T23:05:45.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.613+0000 7f338a000640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f337406c8f0 0x7f337406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:05:45.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.613+0000 7f338a000640 1 -- 172.21.15.40:0/436711059 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6101+0+0 (secure 0 0 0) 0x7f33840ac4f0 con 0x7f33900b6220 2024-04-26T23:05:45.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.613+0000 7f3395400640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f337406c8f0 0x7f337406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:05:45.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.615+0000 7f3395400640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f337406c8f0 0x7f337406edb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f33901618a0 tx=0x7f3380008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:05:45.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.618+0000 7f338a000640 1 -- 172.21.15.40:0/436711059 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f338407a6f0 con 0x7f33900b6220 2024-04-26T23:05:45.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.854+0000 7f3397200640 1 -- 172.21.15.40:0/436711059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33900d30c0 con 0x7f33900b6220 2024-04-26T23:05:45.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.855+0000 7f338a000640 1 -- 172.21.15.40:0/436711059 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5223 (secure 0 0 0) 0x7f3384079ff0 con 0x7f33900b6220 2024-04-26T23:05:45.856 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:05:45.856 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":60,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/4194140680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4194140680},{"type":"v1","addr":"172.21.15.103:6836","nonce":4194140680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:05:32.212319+0000","modified":"2024-04-26T23:05:43.829475+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6774,"mds_1":5614,"mds_2":5324},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5324":{"gid":5324,"name":"b","rank":2,"incarnation":59,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/2595375825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2595375825},{"type":"v1","addr":"172.21.15.103:6837","nonce":2595375825}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5614":{"gid":5614,"name":"a","rank":1,"incarnation":57,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/2117861921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2117861921},{"type":"v1","addr":"172.21.15.40:6834","nonce":2117861921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/931483025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":931483025},{"type":"v1","addr":"172.21.15.40:6835","nonce":931483025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6774,"qdb_cluster":[5324,5614,6774]},"id":4}]} 2024-04-26T23:05:45.856 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 60 2024-04-26T23:05:45.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.858+0000 7f3397200640 1 -- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f337406c8f0 msgr2=0x7f337406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.858+0000 7f3397200640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f337406c8f0 0x7f337406edb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f33901618a0 tx=0x7f3380008040 comp rx=0 tx=0).stop 2024-04-26T23:05:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.858+0000 7f3397200640 1 -- 172.21.15.40:0/436711059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33900b6220 msgr2=0x7f33900b4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:05:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.858+0000 7f3397200640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33900b6220 0x7f33900b4b20 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f33840098c0 tx=0x7f33840046c0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.858+0000 7f3397200640 1 -- 172.21.15.40:0/436711059 shutdown_connections 2024-04-26T23:05:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.858+0000 7f3397200640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f337406c8f0 0x7f337406edb0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.858+0000 7f3397200640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33900d3820 0x7f33900a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.858+0000 7f3397200640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33900d08d0 0x7f339009f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.858+0000 7f3397200640 1 --2- 172.21.15.40:0/436711059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33900b6220 0x7f33900b4b20 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:05:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.858+0000 7f3397200640 1 -- 172.21.15.40:0/436711059 >> 172.21.15.40:0/436711059 conn(0x7f33900d4d90 msgr2=0x7f33900cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:05:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.858+0000 7f3397200640 1 -- 172.21.15.40:0/436711059 shutdown_connections 2024-04-26T23:05:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:05:45.858+0000 7f3397200640 1 -- 172.21.15.40:0/436711059 wait complete. 2024-04-26T23:05:45.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:45.879 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p d1/d2/d3/d4/d5/d6/d7/d8) 2024-04-26T23:05:45.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:45.953 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec sync d1/d2/d3/d4/d5/d6/d7/d8) 2024-04-26T23:05:46.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:46.020 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 d1/d2) 2024-04-26T23:05:46.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:46.076 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 d1/d2/d3/d4) 2024-04-26T23:05:46.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:05:46.126 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 2 d1/d2/d3/d4/d5/d6) 2024-04-26T23:05:46.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:05:46.455 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"5","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:05:46.456 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:05:46.456 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:05:47.677 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:05:47.676+0000 7fa16ce00640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T23:05:48.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:05:48.234 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:05:48.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:05:48.700 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"7","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:05:48.700 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:05:48.701 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:05:50.702 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:05:50.948 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"7","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:05:50.949 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:05:50.949 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:05:52.950 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:05:53.191 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"13","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/d1/d2","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-04-26T23:05:53.192 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000001", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/d1/d2/d3/d4", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-04-26T23:05:53.192 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:05:53.192 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:05:53.445 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:05:53.445 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:05:53.446 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:05:55.447 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:05:55.688 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:05:55.689 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:05:55.689 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:05:57.690 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:05:57.926 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:05:57.927 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:05:57.927 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:05:59.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:06:00.168 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1,"2":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":2,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/d1/d2/d3/d4/d5/d6","dirfrag":"0x10000000005","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-04-26T23:06:00.168 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1, "2": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "2", "dirfrag": "0x10000000005", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/d1/d2/d3/d4/d5/d6", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "6", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 2, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false } ] 2024-04-26T23:06:00.169 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:06:00.169 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-04-26T23:06:00.401 INFO:teuthology.orchestra.run.smithi103.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"0","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dnwaiter":0,"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":2,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/d1/d2/d3/d4/d5/d6","dirfrag":"0x10000000005","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:06:00.402 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dnwaiter": 0, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "2", "dirfrag": "0x10000000005", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4/d5/d6", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 2, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:06:00.402 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:06:00.402 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 flush journal 2024-04-26T23:06:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.621+0000 7fa1cb200640 1 Processor -- start 2024-04-26T23:06:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.622+0000 7fa1cb200640 1 -- start start 2024-04-26T23:06:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.623+0000 7fa1cb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1cc0b6470 0x7fa1cc0cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.623+0000 7fa1cb200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa1cc15f680 0x7fa1cc0cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.623+0000 7fa1cb200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1cc0d5d30 con 0x7fa1cc0d1770 2024-04-26T23:06:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.623+0000 7fa1cb200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa1cc0d3760 con 0x7fa1cc15f680 2024-04-26T23:06:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.623+0000 7fa1cb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa1cc0d5a70 con 0x7fa1cc0b6470 2024-04-26T23:06:00.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.624+0000 7fa1ca800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa1cc15f680 0x7fa1cc0cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.624+0000 7fa1c9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1cc0b6470 0x7fa1cc0cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.624+0000 7fa1ca800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa1cc15f680 0x7fa1cc0cff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:55510/0 (socket says 172.21.15.40:55510) 2024-04-26T23:06:00.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.624+0000 7fa1ca800640 1 -- 172.21.15.40:0/1157408738 learned_addr learned my addr 172.21.15.40:0/1157408738 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:00.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.625+0000 7fa1c9400640 1 -- 172.21.15.40:0/1157408738 >> v2:172.21.15.103:3300/0 conn(0x7fa1cc15f680 msgr2=0x7fa1cc0cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.625+0000 7fa1c9400640 1 --2- 172.21.15.40:0/1157408738 >> v2:172.21.15.103:3300/0 conn(0x7fa1cc15f680 0x7fa1cc0cff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.625+0000 7fa1c9400640 1 -- 172.21.15.40:0/1157408738 >> v1:172.21.15.103:6789/0 conn(0x7fa1cc0d1770 legacy=0x7fa1cc15ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.625+0000 7fa1c9400640 1 -- 172.21.15.40:0/1157408738 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1cc0b5b00 con 0x7fa1cc0b6470 2024-04-26T23:06:00.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.627+0000 7fa1c9400640 1 --2- 172.21.15.40:0/1157408738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1cc0b6470 0x7fa1cc0cf9e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa1b40097a0 tx=0x7fa1b402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcb041463c5cc09e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.627+0000 7fa1c8a00640 1 -- 172.21.15.40:0/1157408738 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4037070 con 0x7fa1cc0b6470 2024-04-26T23:06:00.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.628+0000 7fa1c8a00640 1 -- 172.21.15.40:0/1157408738 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b4032440 con 0x7fa1cc0b6470 2024-04-26T23:06:00.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.628+0000 7fa1c8a00640 1 -- 172.21.15.40:0/1157408738 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4032810 con 0x7fa1cc0b6470 2024-04-26T23:06:00.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.628+0000 7fa1cb200640 1 -- 172.21.15.40:0/1157408738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1cc0b6470 msgr2=0x7fa1cc0cf9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.628+0000 7fa1cb200640 1 --2- 172.21.15.40:0/1157408738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1cc0b6470 0x7fa1cc0cf9e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa1b40097a0 tx=0x7fa1b402a180 comp rx=0 tx=0).stop 2024-04-26T23:06:00.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.628+0000 7fa1cb200640 1 -- 172.21.15.40:0/1157408738 shutdown_connections 2024-04-26T23:06:00.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.628+0000 7fa1cb200640 1 --2- 172.21.15.40:0/1157408738 >> v2:172.21.15.103:3300/0 conn(0x7fa1cc15f680 0x7fa1cc0cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.628+0000 7fa1cb200640 1 --2- 172.21.15.40:0/1157408738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1cc0b6470 0x7fa1cc0cf9e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.628+0000 7fa1cb200640 1 -- 172.21.15.40:0/1157408738 >> 172.21.15.40:0/1157408738 conn(0x7fa1cc0d4dd0 msgr2=0x7fa1cc0d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:00.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.628+0000 7fa1cb200640 1 -- 172.21.15.40:0/1157408738 shutdown_connections 2024-04-26T23:06:00.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.628+0000 7fa1cb200640 1 -- 172.21.15.40:0/1157408738 wait complete. 2024-04-26T23:06:00.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.630+0000 7fa1cb200640 1 Processor -- start 2024-04-26T23:06:00.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.630+0000 7fa1cb200640 1 -- start start 2024-04-26T23:06:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.631+0000 7fa1cb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1cc0b6470 0x7fa1cc0cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.631+0000 7fa1cb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1cc0d1770 0x7fa1cc0b8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.631+0000 7fa1cb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1cc15f680 0x7fa1cc0b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.631+0000 7fa1cb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa1cc0d3d70 con 0x7fa1cc15f680 2024-04-26T23:06:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.631+0000 7fa1cb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1cc0d3bf0 con 0x7fa1cc0d1770 2024-04-26T23:06:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.631+0000 7fa1cb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa1cc0d1b50 con 0x7fa1cc0b6470 2024-04-26T23:06:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.631+0000 7fa1ca800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1cc15f680 0x7fa1cc0b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.632+0000 7fa1ca800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1cc15f680 0x7fa1cc0b8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33424/0 (socket says 172.21.15.40:33424) 2024-04-26T23:06:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.632+0000 7fa1ca800640 1 -- 172.21.15.40:0/1834390992 learned_addr learned my addr 172.21.15.40:0/1834390992 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.632+0000 7fa1c9e00640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1cc0b6470 0x7fa1cc0cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.632+0000 7fa1c9400640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1cc0d1770 0x7fa1cc0b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.632+0000 7fa1ca800640 1 -- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1cc0b6470 msgr2=0x7fa1cc0cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.632+0000 7fa1ca800640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1cc0b6470 0x7fa1cc0cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.632+0000 7fa1ca800640 1 -- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1cc0d1770 msgr2=0x7fa1cc0b8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.632+0000 7fa1ca800640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1cc0d1770 0x7fa1cc0b8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.632+0000 7fa1ca800640 1 -- 172.21.15.40:0/1834390992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1cc0b7c80 con 0x7fa1cc15f680 2024-04-26T23:06:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.632+0000 7fa1ca800640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1cc15f680 0x7fa1cc0b8970 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400e840 tx=0x7fa1c400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.633+0000 7fa1c2000640 1 -- 172.21.15.40:0/1834390992 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4017070 con 0x7fa1cc15f680 2024-04-26T23:06:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.633+0000 7fa1c2000640 1 -- 172.21.15.40:0/1834390992 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1c400c730 con 0x7fa1cc15f680 2024-04-26T23:06:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.633+0000 7fa1c2000640 1 -- 172.21.15.40:0/1834390992 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c400cb00 con 0x7fa1cc15f680 2024-04-26T23:06:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.633+0000 7fa1cb200640 1 -- 172.21.15.40:0/1834390992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1cc0b7f60 con 0x7fa1cc15f680 2024-04-26T23:06:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.633+0000 7fa1cb200640 1 -- 172.21.15.40:0/1834390992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1cc0cf210 con 0x7fa1cc15f680 2024-04-26T23:06:00.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.636+0000 7fa1c2000640 1 -- 172.21.15.40:0/1834390992 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa1c401a070 con 0x7fa1cc15f680 2024-04-26T23:06:00.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.637+0000 7fa1c2000640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa1a806c8f0 0x7fa1a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.637+0000 7fa1c2000640 1 -- 172.21.15.40:0/1834390992 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7fa1c408bbe0 con 0x7fa1cc15f680 2024-04-26T23:06:00.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.637+0000 7fa1d387d740 1 Processor -- start 2024-04-26T23:06:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.637+0000 7fa1d387d740 1 -- start start 2024-04-26T23:06:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.638+0000 7fa1c9e00640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa1a806c8f0 0x7fa1a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.638+0000 7fa1d387d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103b710 0x559b0103baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.638+0000 7fa1d387d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b01058fb0 0x559b01059500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.638+0000 7fa1d387d740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b010113f0 0x559b01057670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.638+0000 7fa1d387d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559b00f21030 con 0x559b01058fb0 2024-04-26T23:06:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.638+0000 7fa1d387d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559b00f3c7f0 con 0x559b0103b710 2024-04-26T23:06:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.638+0000 7fa1d387d740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x559b00ef77e0 con 0x559b010113f0 2024-04-26T23:06:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.639+0000 7fa1c9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b01058fb0 0x559b01059500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.639+0000 7fa1c9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b01058fb0 0x559b01059500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33440/0 (socket says 172.21.15.40:33440) 2024-04-26T23:06:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.639+0000 7fa1c9400640 1 -- 172.21.15.40:0/537232269 learned_addr learned my addr 172.21.15.40:0/537232269 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.639+0000 7fa1c9e00640 1 --2- 172.21.15.40:0/537232269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103b710 0x559b0103baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.639+0000 7fa1ca800640 1 --2- 172.21.15.40:0/537232269 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b010113f0 0x559b01057670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.639+0000 7fa1c9e00640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa1a806c8f0 0x7fa1a806edb0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0012e0 tx=0x7fa1bc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.639+0000 7fa1c9400640 1 -- 172.21.15.40:0/537232269 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b010113f0 msgr2=0x559b01057670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.639+0000 7fa1c9400640 1 --2- 172.21.15.40:0/537232269 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b010113f0 0x559b01057670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.639+0000 7fa1c9400640 1 -- 172.21.15.40:0/537232269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103b710 msgr2=0x559b0103baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.639+0000 7fa1c9400640 1 --2- 172.21.15.40:0/537232269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103b710 0x559b0103baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.639+0000 7fa1c9400640 1 -- 172.21.15.40:0/537232269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b00f03050 con 0x559b01058fb0 2024-04-26T23:06:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.640+0000 7fa1c9400640 1 --2- 172.21.15.40:0/537232269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b01058fb0 0x559b01059500 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fa1b40098c0 tx=0x7fa1b402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e8f4591e6086e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.640+0000 7fa1ba600640 1 -- 172.21.15.40:0/537232269 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4037040 con 0x559b01058fb0 2024-04-26T23:06:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.640+0000 7fa1ba600640 1 -- 172.21.15.40:0/537232269 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b4032780 con 0x559b01058fb0 2024-04-26T23:06:00.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.640+0000 7fa1ba600640 1 -- 172.21.15.40:0/537232269 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4032b50 con 0x559b01058fb0 2024-04-26T23:06:00.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.640+0000 7fa1d387d740 1 -- 172.21.15.40:0/537232269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b01058fb0 msgr2=0x559b01059500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.640+0000 7fa1d387d740 1 --2- 172.21.15.40:0/537232269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b01058fb0 0x559b01059500 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fa1b40098c0 tx=0x7fa1b402c000 comp rx=0 tx=0).stop 2024-04-26T23:06:00.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.641+0000 7fa1d387d740 1 -- 172.21.15.40:0/537232269 shutdown_connections 2024-04-26T23:06:00.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.641+0000 7fa1d387d740 1 --2- 172.21.15.40:0/537232269 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b010113f0 0x559b01057670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.641+0000 7fa1d387d740 1 --2- 172.21.15.40:0/537232269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b01058fb0 0x559b01059500 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.641+0000 7fa1d387d740 1 --2- 172.21.15.40:0/537232269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103b710 0x559b0103baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.641+0000 7fa1d387d740 1 -- 172.21.15.40:0/537232269 >> 172.21.15.40:0/537232269 conn(0x559b0100dd20 msgr2=0x559b01056520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:00.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.641+0000 7fa1d387d740 1 -- 172.21.15.40:0/537232269 shutdown_connections 2024-04-26T23:06:00.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.641+0000 7fa1d387d740 1 -- 172.21.15.40:0/537232269 wait complete. 2024-04-26T23:06:00.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.641+0000 7fa1d387d740 1 -- start start 2024-04-26T23:06:00.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.641+0000 7fa1d387d740 1 Processor -- start 2024-04-26T23:06:00.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.642+0000 7fa1d387d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 0x559b01027f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.642+0000 7fa1d387d740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b01058fb0 0x559b0103d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.642+0000 7fa1d387d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103df20 0x559b010288c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.642+0000 7fa1d387d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559b00f79200 con 0x559b0103b710 2024-04-26T23:06:00.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.642+0000 7fa1d387d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559b00f73ee0 con 0x559b0103df20 2024-04-26T23:06:00.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.642+0000 7fa1d387d740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x559b00f80e70 con 0x559b01058fb0 2024-04-26T23:06:00.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1d387d740 10 client.0 _ll_register_callbacks cb 0x559b00f6a270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:00.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1c9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 0x559b01027f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1c9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 0x559b01027f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33442/0 (socket says 172.21.15.40:33442) 2024-04-26T23:06:00.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1c9e00640 1 -- 172.21.15.40:0/381568825 learned_addr learned my addr 172.21.15.40:0/381568825 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:00.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1c9400640 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b01058fb0 0x559b0103d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1ca800640 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103df20 0x559b010288c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1c9e00640 1 -- 172.21.15.40:0/381568825 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b01058fb0 msgr2=0x559b0103d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1c9e00640 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b01058fb0 0x559b0103d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1c9e00640 1 -- 172.21.15.40:0/381568825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103df20 msgr2=0x559b010288c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1c9e00640 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103df20 0x559b010288c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1c9e00640 1 -- 172.21.15.40:0/381568825 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b00d507d0 con 0x559b0103b710 2024-04-26T23:06:00.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1c9400640 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b01058fb0 0x559b0103d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:00.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1c9e00640 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 0x559b01027f90 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc014710 tx=0x7fa1bc001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be58849f5176c50b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.643+0000 7fa1aea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.644+0000 7fa1aea00640 1 -- 172.21.15.40:0/381568825 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc027060 con 0x559b0103b710 2024-04-26T23:06:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.644+0000 7fa1aea00640 1 -- 172.21.15.40:0/381568825 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7fa1bc022d30 con 0x559b0103b710 2024-04-26T23:06:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.644+0000 7fa1d387d740 1 -- 172.21.15.40:0/381568825 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1b4032780 con 0x559b0103b710 2024-04-26T23:06:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.644+0000 7fa1aea00640 1 -- 172.21.15.40:0/381568825 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fa1bc01a070 con 0x559b0103b710 2024-04-26T23:06:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.644+0000 7fa1d387d740 10 client.7020 fetch_fsmap learned FSMap version 62 2024-04-26T23:06:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.644+0000 7fa1d387d740 1 -- 172.21.15.40:0/381568825 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) -- 0x559b00c863f0 con 0x559b0103b710 2024-04-26T23:06:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.644+0000 7fa1aea00640 1 -- 172.21.15.40:0/381568825 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b4032780 con 0x559b0103b710 2024-04-26T23:06:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.644+0000 7fa1aea00640 1 -- 172.21.15.40:0/381568825 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 62) ==== 3546+0+0 (secure 0 0 0) 0x7fa1bc003b80 con 0x559b0103b710 2024-04-26T23:06:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.645+0000 7fa1d387d740 10 client.7020 fetch_fsmap finished waiting for FSMap version 62 2024-04-26T23:06:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.645+0000 7fa1d387d740 10 client.7020 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-26T23:06:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.645+0000 7fa1d387d740 20 client.7020 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.645+0000 7fa1d387d740 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x559b01028e00 0x559b0100a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.645+0000 7fa1d387d740 4 client.7020 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:06:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.645+0000 7fa1d387d740 1 -- 172.21.15.40:0/381568825 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b00f827e0 con 0x559b01028e00 2024-04-26T23:06:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.645+0000 7fa1c9400640 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x559b01028e00 0x559b0100a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.645+0000 7fa1c9400640 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x559b01028e00 0x559b0100a960 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6aec870ff1b305d server_cookie=43d2f2fa8d51b814 in_seq=0 out_seq=0 2024-04-26T23:06:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.645+0000 7fa1aea00640 10 client.7020 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.646+0000 7fa1aea00640 1 -- 172.21.15.40:0/381568825 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559b00f827e0 con 0x559b01028e00 2024-04-26T23:06:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.646+0000 7fa1aea00640 10 client.7020 handle_command_reply: tid=0 2024-04-26T23:06:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.646+0000 7fa1d387d740 1 client.7020 shutdown 2024-04-26T23:06:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.646+0000 7fa1d387d740 1 -- 172.21.15.40:0/381568825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 msgr2=0x559b01027f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.646+0000 7fa1d387d740 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 0x559b01027f90 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc014710 tx=0x7fa1bc001250 comp rx=0 tx=0).stop 2024-04-26T23:06:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.646+0000 7fa1d387d740 1 -- 172.21.15.40:0/381568825 shutdown_connections 2024-04-26T23:06:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.646+0000 7fa1d387d740 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x559b01028e00 0x559b0100a960 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.646+0000 7fa1d387d740 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103df20 0x559b010288c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.646+0000 7fa1d387d740 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b01058fb0 0x559b0103d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.646+0000 7fa1d387d740 1 --2- 172.21.15.40:0/381568825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 0x559b01027f90 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.646+0000 7fa1aea00640 0 client.7020 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.646+0000 7fa1d387d740 1 -- 172.21.15.40:0/381568825 >> 172.21.15.40:0/381568825 conn(0x559b010113f0 msgr2=0x559b0103a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.647+0000 7fa1d387d740 1 -- 172.21.15.40:0/381568825 shutdown_connections 2024-04-26T23:06:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.647+0000 7fa1d387d740 1 -- 172.21.15.40:0/381568825 wait complete. 2024-04-26T23:06:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.647+0000 7fa1d387d740 20 client.7020 trim_cache size 0 max 16384 2024-04-26T23:06:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.663+0000 7fa1d387d740 1 Processor -- start 2024-04-26T23:06:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.663+0000 7fa1d387d740 1 -- start start 2024-04-26T23:06:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.664+0000 7fa1d387d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103b710 0x559b010073b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.664+0000 7fa1d387d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103df20 0x559b0100f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.664+0000 7fa1d387d740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b01058fb0 0x559b0100ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.664+0000 7fa1d387d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa1cc201e90 con 0x559b0103df20 2024-04-26T23:06:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.664+0000 7fa1d387d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1cc201060 con 0x559b0103b710 2024-04-26T23:06:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.664+0000 7fa1d387d740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa1cc201760 con 0x559b01058fb0 2024-04-26T23:06:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.664+0000 7fa1c9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103df20 0x559b0100f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.664+0000 7fa1c9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103df20 0x559b0100f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33452/0 (socket says 172.21.15.40:33452) 2024-04-26T23:06:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.664+0000 7fa1c9400640 1 -- 172.21.15.40:0/377267005 learned_addr learned my addr 172.21.15.40:0/377267005 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.665+0000 7fa1ca800640 1 --2- 172.21.15.40:0/377267005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b01058fb0 0x559b0100ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.665+0000 7fa1c9e00640 1 --2- 172.21.15.40:0/377267005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103b710 0x559b010073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.665+0000 7fa1c9400640 1 -- 172.21.15.40:0/377267005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b01058fb0 msgr2=0x559b0100ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.665+0000 7fa1c9400640 1 --2- 172.21.15.40:0/377267005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b01058fb0 0x559b0100ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.665+0000 7fa1c9400640 1 -- 172.21.15.40:0/377267005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103b710 msgr2=0x559b010073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.665+0000 7fa1c9400640 1 --2- 172.21.15.40:0/377267005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103b710 0x559b010073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.665+0000 7fa1c9400640 1 -- 172.21.15.40:0/377267005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b00c460f0 con 0x559b0103df20 2024-04-26T23:06:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.665+0000 7fa1c9400640 1 --2- 172.21.15.40:0/377267005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103df20 0x559b0100f9f0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4000c00 tx=0x7fa1b4008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9906d327ba08ea13 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.665+0000 7fa1b9c00640 1 -- 172.21.15.40:0/377267005 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4051070 con 0x559b0103df20 2024-04-26T23:06:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.665+0000 7fa1b9c00640 1 -- 172.21.15.40:0/377267005 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b40326e0 con 0x559b0103df20 2024-04-26T23:06:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.665+0000 7fa1b9c00640 1 -- 172.21.15.40:0/377267005 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4037040 con 0x559b0103df20 2024-04-26T23:06:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.666+0000 7fa1d387d740 1 -- 172.21.15.40:0/377267005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103df20 msgr2=0x559b0100f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.666+0000 7fa1d387d740 1 --2- 172.21.15.40:0/377267005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103df20 0x559b0100f9f0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4000c00 tx=0x7fa1b4008030 comp rx=0 tx=0).stop 2024-04-26T23:06:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.666+0000 7fa1d387d740 1 -- 172.21.15.40:0/377267005 shutdown_connections 2024-04-26T23:06:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.666+0000 7fa1d387d740 1 --2- 172.21.15.40:0/377267005 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b01058fb0 0x559b0100ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.666+0000 7fa1d387d740 1 --2- 172.21.15.40:0/377267005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103df20 0x559b0100f9f0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.666+0000 7fa1d387d740 1 --2- 172.21.15.40:0/377267005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0103b710 0x559b010073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.666+0000 7fa1d387d740 1 -- 172.21.15.40:0/377267005 >> 172.21.15.40:0/377267005 conn(0x559b010113f0 msgr2=0x559b01010970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.666+0000 7fa1d387d740 1 -- 172.21.15.40:0/377267005 shutdown_connections 2024-04-26T23:06:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.666+0000 7fa1d387d740 1 -- 172.21.15.40:0/377267005 wait complete. 2024-04-26T23:06:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.667+0000 7fa1d387d740 1 -- start start 2024-04-26T23:06:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.667+0000 7fa1d387d740 1 Processor -- start 2024-04-26T23:06:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.668+0000 7fa1d387d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 0x559b0103c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.668+0000 7fa1d387d740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b0103df20 0x559b010cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.668+0000 7fa1d387d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b01058fb0 0x559b010d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.668+0000 7fa1d387d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559b00f81fd0 con 0x559b0103b710 2024-04-26T23:06:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.668+0000 7fa1d387d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1cc0ce5f0 con 0x559b01058fb0 2024-04-26T23:06:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.668+0000 7fa1d387d740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x559b01027f90 con 0x559b0103df20 2024-04-26T23:06:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.668+0000 7fa1d387d740 10 client.0 _ll_register_callbacks cb 0x7fa1bc00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:00.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.668+0000 7fa1c9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 0x559b0103c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.668+0000 7fa1c9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 0x559b0103c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33456/0 (socket says 172.21.15.40:33456) 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.668+0000 7fa1c9e00640 1 -- 172.21.15.40:0/763150343 learned_addr learned my addr 172.21.15.40:0/763150343 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.669+0000 7fa1c9400640 1 --2- 172.21.15.40:0/763150343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b0103df20 0x559b010cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.669+0000 7fa1ca800640 1 --2- 172.21.15.40:0/763150343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b01058fb0 0x559b010d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.669+0000 7fa1c9e00640 1 -- 172.21.15.40:0/763150343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b0103df20 msgr2=0x559b010cfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.669+0000 7fa1c9e00640 1 --2- 172.21.15.40:0/763150343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b0103df20 0x559b010cfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.669+0000 7fa1c9e00640 1 -- 172.21.15.40:0/763150343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b01058fb0 msgr2=0x559b010d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.669+0000 7fa1c9e00640 1 --2- 172.21.15.40:0/763150343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b01058fb0 0x559b010d0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.669+0000 7fa1c9e00640 1 -- 172.21.15.40:0/763150343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b00f3ca30 con 0x559b0103b710 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.669+0000 7fa1c9e00640 1 --2- 172.21.15.40:0/763150343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 0x559b0103c7d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0052b0 tx=0x7fa1bc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9d6936dc5b652e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.669+0000 7fa1aea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.669+0000 7fa1aea00640 1 -- 172.21.15.40:0/763150343 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc005340 con 0x559b0103b710 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.669+0000 7fa1aea00640 1 -- 172.21.15.40:0/763150343 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7fa1bc032040 con 0x559b0103b710 2024-04-26T23:06:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.669+0000 7fa1d387d740 1 -- 172.21.15.40:0/763150343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1b40326e0 con 0x559b0103b710 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.670+0000 7fa1aea00640 1 -- 172.21.15.40:0/763150343 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fa1bc0054e0 con 0x559b0103b710 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.670+0000 7fa1d387d740 10 client.7026 fetch_fsmap learned FSMap version 62 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.670+0000 7fa1d387d740 1 -- 172.21.15.40:0/763150343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) -- 0x559b01058360 con 0x559b0103b710 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.670+0000 7fa1aea00640 1 -- 172.21.15.40:0/763150343 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b40326e0 con 0x559b0103b710 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.670+0000 7fa1aea00640 1 -- 172.21.15.40:0/763150343 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 62) ==== 3546+0+0 (secure 0 0 0) 0x7fa1bc0135a0 con 0x559b0103b710 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.670+0000 7fa1d387d740 10 client.7026 fetch_fsmap finished waiting for FSMap version 62 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.670+0000 7fa1d387d740 10 client.7026 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.670+0000 7fa1d387d740 20 client.7026 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.670+0000 7fa1d387d740 1 --2- 172.21.15.40:0/763150343 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x559b010102c0 0x559b010ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.670+0000 7fa1d387d740 4 client.7026 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.670+0000 7fa1d387d740 1 -- 172.21.15.40:0/763150343 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559b00f7c650 con 0x559b010102c0 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.670+0000 7fa1c9400640 1 --2- 172.21.15.40:0/763150343 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x559b010102c0 0x559b010ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.671+0000 7fa1c9400640 1 --2- 172.21.15.40:0/763150343 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x559b010102c0 0x559b010ccd90 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4636ea1265db779 server_cookie=4ed0a7718e00d2c0 in_seq=0 out_seq=0 2024-04-26T23:06:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.671+0000 7fa1aea00640 10 client.7026 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:00.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.697+0000 7fa1aea00640 1 -- 172.21.15.40:0/763150343 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559b00f7c650 con 0x559b010102c0 2024-04-26T23:06:00.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.697+0000 7fa1aea00640 10 client.7026 handle_command_reply: tid=0 2024-04-26T23:06:00.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.697+0000 7fa1d387d740 1 client.7026 shutdown 2024-04-26T23:06:00.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.697+0000 7fa1d387d740 1 -- 172.21.15.40:0/763150343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 msgr2=0x559b0103c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.697+0000 7fa1d387d740 1 --2- 172.21.15.40:0/763150343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 0x559b0103c7d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0052b0 tx=0x7fa1bc015000 comp rx=0 tx=0).stop 2024-04-26T23:06:00.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.697+0000 7fa1d387d740 1 -- 172.21.15.40:0/763150343 shutdown_connections 2024-04-26T23:06:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.697+0000 7fa1d387d740 1 --2- 172.21.15.40:0/763150343 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x559b010102c0 0x559b010ccd90 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.697+0000 7fa1d387d740 1 --2- 172.21.15.40:0/763150343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b01058fb0 0x559b010d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.697+0000 7fa1d387d740 1 --2- 172.21.15.40:0/763150343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x559b0103df20 0x559b010cfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.697+0000 7fa1d387d740 1 --2- 172.21.15.40:0/763150343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559b0103b710 0x559b0103c7d0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.697+0000 7fa1aea00640 0 client.7026 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.697+0000 7fa1d387d740 1 -- 172.21.15.40:0/763150343 >> 172.21.15.40:0/763150343 conn(0x559b010113f0 msgr2=0x559b01010970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.698+0000 7fa1d387d740 1 -- 172.21.15.40:0/763150343 shutdown_connections 2024-04-26T23:06:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.698+0000 7fa1d387d740 1 -- 172.21.15.40:0/763150343 wait complete. 2024-04-26T23:06:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.698+0000 7fa1d387d740 20 client.7026 trim_cache size 0 max 16384 2024-04-26T23:06:00.700 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:06:00.700 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:06:00.700 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:06:00.700 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:06:00.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.699+0000 7fa1cb200640 1 -- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa1a806c8f0 msgr2=0x7fa1a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.699+0000 7fa1cb200640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa1a806c8f0 0x7fa1a806edb0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0012e0 tx=0x7fa1bc009040 comp rx=0 tx=0).stop 2024-04-26T23:06:00.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.699+0000 7fa1cb200640 1 -- 172.21.15.40:0/1834390992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1cc15f680 msgr2=0x7fa1cc0b8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.699+0000 7fa1cb200640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1cc15f680 0x7fa1cc0b8970 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400e840 tx=0x7fa1c400ed00 comp rx=0 tx=0).stop 2024-04-26T23:06:00.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.699+0000 7fa1cb200640 1 -- 172.21.15.40:0/1834390992 shutdown_connections 2024-04-26T23:06:00.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.699+0000 7fa1cb200640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa1a806c8f0 0x7fa1a806edb0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.699+0000 7fa1cb200640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1cc15f680 0x7fa1cc0b8970 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.699+0000 7fa1cb200640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1cc0d1770 0x7fa1cc0b8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.699+0000 7fa1cb200640 1 --2- 172.21.15.40:0/1834390992 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1cc0b6470 0x7fa1cc0cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.699+0000 7fa1cb200640 1 -- 172.21.15.40:0/1834390992 >> 172.21.15.40:0/1834390992 conn(0x7fa1cc0d4dd0 msgr2=0x7fa1cc0b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:00.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.699+0000 7fa1cb200640 1 -- 172.21.15.40:0/1834390992 shutdown_connections 2024-04-26T23:06:00.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.699+0000 7fa1cb200640 1 -- 172.21.15.40:0/1834390992 wait complete. 2024-04-26T23:06:00.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:1 flush journal 2024-04-26T23:06:00.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.924+0000 7f2443200640 1 Processor -- start 2024-04-26T23:06:00.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.924+0000 7f2443200640 1 -- start start 2024-04-26T23:06:00.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.926+0000 7f2443200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f24440d1770 0x7f244415ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.926+0000 7f2443200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440b6470 0x7f244415f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.926+0000 7f2443200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24440d5d30 con 0x7f24440d1b50 2024-04-26T23:06:00.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.926+0000 7f2443200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f24440d3760 con 0x7f24440d1770 2024-04-26T23:06:00.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.926+0000 7f2443200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f24440d09f0 con 0x7f24440b6470 2024-04-26T23:06:00.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.926+0000 7f2442800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f24440d1b50 0x7f24440b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:46314/0 (socket says 172.21.15.40:46314) 2024-04-26T23:06:00.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.926+0000 7f2442800640 1 -- 172.21.15.40:0/974731611 learned_addr learned my addr 172.21.15.40:0/974731611 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:00.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.927+0000 7f2441e00640 1 --2- 172.21.15.40:0/974731611 >> v2:172.21.15.103:3300/0 conn(0x7f24440d1770 0x7f244415ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.927+0000 7f243be00640 1 --2- 172.21.15.40:0/974731611 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440b6470 0x7f244415f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.927+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2633143349 0 0) 0x7f24440d5d30 con 0x7f24440d1b50 2024-04-26T23:06:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.927+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2428003520 con 0x7f24440d1b50 2024-04-26T23:06:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.927+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1442162572 0 0) 0x7f2428003520 con 0x7f24440d1b50 2024-04-26T23:06:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.927+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440b6470 msgr2=0x7f244415f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.927+0000 7f2441400640 1 --2- 172.21.15.40:0/974731611 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440b6470 0x7f244415f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.927+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 >> v2:172.21.15.103:3300/0 conn(0x7f24440d1770 msgr2=0x7f244415ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.927+0000 7f2441400640 1 --2- 172.21.15.40:0/974731611 >> v2:172.21.15.103:3300/0 conn(0x7f24440d1770 0x7f244415ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.927+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24440d5a70 con 0x7f24440d1b50 2024-04-26T23:06:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.927+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f243c0019f0 con 0x7f24440d1b50 2024-04-26T23:06:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.927+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 >> v1:172.21.15.40:6789/0 conn(0x7f24440d1b50 legacy=0x7f24440b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.927+0000 7f2441400640 1 --2- 172.21.15.40:0/974731611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24280039b0 0x7f2428003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.928+0000 7f2441400640 1 --2- 172.21.15.40:0/974731611 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440b6470 0x7f2428004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.928+0000 7f2441400640 1 --2- 172.21.15.40:0/974731611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2428004d40 0x7f24280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.928+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f24440d3760 con 0x7f2428004d40 2024-04-26T23:06:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.928+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24440d09f0 con 0x7f24280039b0 2024-04-26T23:06:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.928+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f242800b9c0 con 0x7f24440b6470 2024-04-26T23:06:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.928+0000 7f2442800640 1 --2- 172.21.15.40:0/974731611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2428004d40 0x7f24280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.928+0000 7f2441e00640 1 --2- 172.21.15.40:0/974731611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24280039b0 0x7f2428003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.928+0000 7f2442800640 1 -- 172.21.15.40:0/974731611 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440b6470 msgr2=0x7f2428004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.928+0000 7f2442800640 1 --2- 172.21.15.40:0/974731611 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440b6470 0x7f2428004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.928+0000 7f2442800640 1 -- 172.21.15.40:0/974731611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24280039b0 msgr2=0x7f2428003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.928+0000 7f2442800640 1 --2- 172.21.15.40:0/974731611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24280039b0 0x7f2428003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.928+0000 7f2442800640 1 -- 172.21.15.40:0/974731611 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2428003520 con 0x7f2428004d40 2024-04-26T23:06:00.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.929+0000 7f2442800640 1 --2- 172.21.15.40:0/974731611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2428004d40 0x7f24280051c0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f243c00a7a0 tx=0x7f2428005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac9b657b69f83a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.930+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c00a3f0 con 0x7f2428004d40 2024-04-26T23:06:00.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.930+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f243c02f070 con 0x7f2428004d40 2024-04-26T23:06:00.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.930+0000 7f2441400640 1 -- 172.21.15.40:0/974731611 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c02bc20 con 0x7f2428004d40 2024-04-26T23:06:00.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.930+0000 7f2443200640 1 -- 172.21.15.40:0/974731611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2428004d40 msgr2=0x7f24280051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.930+0000 7f2443200640 1 --2- 172.21.15.40:0/974731611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2428004d40 0x7f24280051c0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f243c00a7a0 tx=0x7f2428005830 comp rx=0 tx=0).stop 2024-04-26T23:06:00.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.930+0000 7f2442800640 1 -- 172.21.15.40:0/974731611 reap_dead start 2024-04-26T23:06:00.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.930+0000 7f2443200640 1 -- 172.21.15.40:0/974731611 shutdown_connections 2024-04-26T23:06:00.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.930+0000 7f2443200640 1 -- 172.21.15.40:0/974731611 >> 172.21.15.40:0/974731611 conn(0x7f24440d4dd0 msgr2=0x7f24440d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:00.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.931+0000 7f2443200640 1 -- 172.21.15.40:0/974731611 shutdown_connections 2024-04-26T23:06:00.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.931+0000 7f2443200640 1 -- 172.21.15.40:0/974731611 wait complete. 2024-04-26T23:06:00.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.932+0000 7f2443200640 1 Processor -- start 2024-04-26T23:06:00.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.933+0000 7f2443200640 1 -- start start 2024-04-26T23:06:00.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.933+0000 7f2443200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2428004d40 0x7f244415aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.933+0000 7f2443200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440b90e0 0x7f2444161660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.933+0000 7f2443200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444161ba0 0x7f244406d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.933+0000 7f2443200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f24440d3bf0 con 0x7f2444161ba0 2024-04-26T23:06:00.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.933+0000 7f2443200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24440d2670 con 0x7f2428004d40 2024-04-26T23:06:00.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.933+0000 7f2443200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f24440d3d70 con 0x7f24440b90e0 2024-04-26T23:06:00.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.934+0000 7f2442800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444161ba0 0x7f244406d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.934+0000 7f2442800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444161ba0 0x7f244406d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33472/0 (socket says 172.21.15.40:33472) 2024-04-26T23:06:00.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.934+0000 7f2442800640 1 -- 172.21.15.40:0/2540238862 learned_addr learned my addr 172.21.15.40:0/2540238862 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:00.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.934+0000 7f243be00640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440b90e0 0x7f2444161660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.934+0000 7f2441e00640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2428004d40 0x7f244415aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.934+0000 7f2442800640 1 -- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440b90e0 msgr2=0x7f2444161660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.934+0000 7f2442800640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440b90e0 0x7f2444161660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.934+0000 7f2442800640 1 -- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2428004d40 msgr2=0x7f244415aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.934+0000 7f2442800640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2428004d40 0x7f244415aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.934+0000 7f2442800640 1 -- 172.21.15.40:0/2540238862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2444162120 con 0x7f2444161ba0 2024-04-26T23:06:00.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.935+0000 7f2442800640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444161ba0 0x7f244406d6e0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f243c00a7a0 tx=0x7f243c001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.935+0000 7f243a000640 1 -- 172.21.15.40:0/2540238862 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c045070 con 0x7f2444161ba0 2024-04-26T23:06:00.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.935+0000 7f243a000640 1 -- 172.21.15.40:0/2540238862 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f243c034470 con 0x7f2444161ba0 2024-04-26T23:06:00.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.935+0000 7f243a000640 1 -- 172.21.15.40:0/2540238862 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c00a3f0 con 0x7f2444161ba0 2024-04-26T23:06:00.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.935+0000 7f2443200640 1 -- 172.21.15.40:0/2540238862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2444162400 con 0x7f2444161ba0 2024-04-26T23:06:00.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.935+0000 7f2443200640 1 -- 172.21.15.40:0/2540238862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24441629c0 con 0x7f2444161ba0 2024-04-26T23:06:00.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.938+0000 7f243a000640 1 -- 172.21.15.40:0/2540238862 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f243c048070 con 0x7f2444161ba0 2024-04-26T23:06:00.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.939+0000 7f243a000640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f242006c940 0x7f242006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.939+0000 7f243a000640 1 -- 172.21.15.40:0/2540238862 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f243c0abaf0 con 0x7f2444161ba0 2024-04-26T23:06:00.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.940+0000 7f2441e00640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f242006c940 0x7f242006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.940+0000 7f244b793740 1 Processor -- start 2024-04-26T23:06:00.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.940+0000 7f244b793740 1 -- start start 2024-04-26T23:06:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.941+0000 7f244b793740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645679c4710 0x5645679c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.941+0000 7f244b793740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679e1fb0 0x5645679e2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.941+0000 7f244b793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56456799a3f0 0x5645679e0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.941+0000 7f244b793740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5645678aa030 con 0x56456799a3f0 2024-04-26T23:06:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.941+0000 7f244b793740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5645678c57f0 con 0x5645679e1fb0 2024-04-26T23:06:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.941+0000 7f244b793740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5645678807e0 con 0x5645679c4710 2024-04-26T23:06:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.941+0000 7f2441e00640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f242006c940 0x7f242006ee00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2434009a90 tx=0x7f2434008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.941+0000 7f2442800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56456799a3f0 0x5645679e0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.942+0000 7f2442800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56456799a3f0 0x5645679e0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33478/0 (socket says 172.21.15.40:33478) 2024-04-26T23:06:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.942+0000 7f2442800640 1 -- 172.21.15.40:0/2694439655 learned_addr learned my addr 172.21.15.40:0/2694439655 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.942+0000 7f2441e00640 1 --2- 172.21.15.40:0/2694439655 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645679c4710 0x5645679c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.942+0000 7f243be00640 1 --2- 172.21.15.40:0/2694439655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679e1fb0 0x5645679e2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.942+0000 7f2442800640 1 -- 172.21.15.40:0/2694439655 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645679c4710 msgr2=0x5645679c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.942+0000 7f2442800640 1 --2- 172.21.15.40:0/2694439655 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645679c4710 0x5645679c4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.942+0000 7f2442800640 1 -- 172.21.15.40:0/2694439655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679e1fb0 msgr2=0x5645679e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.942+0000 7f2442800640 1 --2- 172.21.15.40:0/2694439655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679e1fb0 0x5645679e2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.942+0000 7f2442800640 1 -- 172.21.15.40:0/2694439655 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56456788c050 con 0x56456799a3f0 2024-04-26T23:06:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.942+0000 7f2441e00640 1 --2- 172.21.15.40:0/2694439655 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645679c4710 0x5645679c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:06:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.942+0000 7f2442800640 1 --2- 172.21.15.40:0/2694439655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56456799a3f0 0x5645679e0670 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f243c000f80 tx=0x7f243c075a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d967e69103084868 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.943+0000 7f2432600640 1 -- 172.21.15.40:0/2694439655 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c07da40 con 0x56456799a3f0 2024-04-26T23:06:00.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.943+0000 7f2432600640 1 -- 172.21.15.40:0/2694439655 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f243c02f040 con 0x56456799a3f0 2024-04-26T23:06:00.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.943+0000 7f2432600640 1 -- 172.21.15.40:0/2694439655 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c082270 con 0x56456799a3f0 2024-04-26T23:06:00.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.943+0000 7f244b793740 1 -- 172.21.15.40:0/2694439655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56456799a3f0 msgr2=0x5645679e0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.943+0000 7f244b793740 1 --2- 172.21.15.40:0/2694439655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56456799a3f0 0x5645679e0670 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f243c000f80 tx=0x7f243c075a60 comp rx=0 tx=0).stop 2024-04-26T23:06:00.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.943+0000 7f244b793740 1 -- 172.21.15.40:0/2694439655 shutdown_connections 2024-04-26T23:06:00.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.943+0000 7f244b793740 1 --2- 172.21.15.40:0/2694439655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56456799a3f0 0x5645679e0670 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.943+0000 7f244b793740 1 --2- 172.21.15.40:0/2694439655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679e1fb0 0x5645679e2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.943+0000 7f244b793740 1 --2- 172.21.15.40:0/2694439655 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645679c4710 0x5645679c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.943+0000 7f244b793740 1 -- 172.21.15.40:0/2694439655 >> 172.21.15.40:0/2694439655 conn(0x564567996d20 msgr2=0x5645679df520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:00.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.944+0000 7f244b793740 1 -- 172.21.15.40:0/2694439655 shutdown_connections 2024-04-26T23:06:00.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.944+0000 7f244b793740 1 -- 172.21.15.40:0/2694439655 wait complete. 2024-04-26T23:06:00.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.944+0000 7f244b793740 1 -- start start 2024-04-26T23:06:00.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.945+0000 7f244b793740 1 Processor -- start 2024-04-26T23:06:00.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f244b793740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679c4710 0x5645679be8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f244b793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679e1fb0 0x5645679bee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f244b793740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564567993ff0 0x5645679ef270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f244b793740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564567902200 con 0x5645679e1fb0 2024-04-26T23:06:00.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f244b793740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5645678fcee0 con 0x5645679c4710 2024-04-26T23:06:00.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f244b793740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564567909e70 con 0x564567993ff0 2024-04-26T23:06:00.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f244b793740 10 client.0 _ll_register_callbacks cb 0x5645678f3270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:00.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f243be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679e1fb0 0x5645679bee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f243be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679e1fb0 0x5645679bee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33488/0 (socket says 172.21.15.40:33488) 2024-04-26T23:06:00.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f243be00640 1 -- 172.21.15.40:0/825778771 learned_addr learned my addr 172.21.15.40:0/825778771 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:00.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f2442800640 1 --2- 172.21.15.40:0/825778771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564567993ff0 0x5645679ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f2441e00640 1 --2- 172.21.15.40:0/825778771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679c4710 0x5645679be8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f243be00640 1 -- 172.21.15.40:0/825778771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564567993ff0 msgr2=0x5645679ef270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f243be00640 1 --2- 172.21.15.40:0/825778771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564567993ff0 0x5645679ef270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f243be00640 1 -- 172.21.15.40:0/825778771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679c4710 msgr2=0x5645679be8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f243be00640 1 --2- 172.21.15.40:0/825778771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679c4710 0x5645679be8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.946+0000 7f243be00640 1 -- 172.21.15.40:0/825778771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645676d97d0 con 0x5645679e1fb0 2024-04-26T23:06:00.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.947+0000 7f243be00640 1 --2- 172.21.15.40:0/825778771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679e1fb0 0x5645679bee30 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f242c009c60 tx=0x7f242c0136b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902465344a517e13 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.947+0000 7f2426a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:00.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.947+0000 7f2426a00640 1 -- 172.21.15.40:0/825778771 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c01a070 con 0x5645679e1fb0 2024-04-26T23:06:00.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.947+0000 7f2426a00640 1 -- 172.21.15.40:0/825778771 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f242c004230 con 0x5645679e1fb0 2024-04-26T23:06:00.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.947+0000 7f244b793740 1 -- 172.21.15.40:0/825778771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f243c02f040 con 0x5645679e1fb0 2024-04-26T23:06:00.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.947+0000 7f2426a00640 1 -- 172.21.15.40:0/825778771 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f242c025050 con 0x5645679e1fb0 2024-04-26T23:06:00.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.947+0000 7f244b793740 10 client.7038 fetch_fsmap learned FSMap version 62 2024-04-26T23:06:00.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.947+0000 7f244b793740 1 -- 172.21.15.40:0/825778771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) -- 0x56456760f3f0 con 0x5645679e1fb0 2024-04-26T23:06:00.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.948+0000 7f2426a00640 1 -- 172.21.15.40:0/825778771 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f243c02f040 con 0x5645679e1fb0 2024-04-26T23:06:00.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.948+0000 7f2426a00640 1 -- 172.21.15.40:0/825778771 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 62) ==== 3546+0+0 (secure 0 0 0) 0x7f242c028020 con 0x5645679e1fb0 2024-04-26T23:06:00.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.948+0000 7f244b793740 10 client.7038 fetch_fsmap finished waiting for FSMap version 62 2024-04-26T23:06:00.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.948+0000 7f244b793740 10 client.7038 resolve_mds: resolved 4:1 to role '4:1' aka daemon mds.a 2024-04-26T23:06:00.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.948+0000 7f244b793740 20 client.7038 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:00.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.948+0000 7f244b793740 1 --2- 172.21.15.40:0/825778771 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x5645679bd140 0x5645679ef7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.948+0000 7f244b793740 4 client.7038 mds_command: new command op to 5614 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:06:00.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.948+0000 7f244b793740 1 -- 172.21.15.40:0/825778771 --> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56456790b7e0 con 0x5645679bd140 2024-04-26T23:06:00.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.948+0000 7f2441e00640 1 --2- 172.21.15.40:0/825778771 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x5645679bd140 0x5645679ef7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.949+0000 7f2441e00640 1 --2- 172.21.15.40:0/825778771 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x5645679bd140 0x5645679ef7b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73ef88ab962f033d server_cookie=8154b24ab9b9f93d in_seq=0 out_seq=0 2024-04-26T23:06:00.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.949+0000 7f2426a00640 10 client.7038 ms_handle_connect on v2:172.21.15.40:6832/2117861921 2024-04-26T23:06:00.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f2426a00640 1 -- 172.21.15.40:0/825778771 <== mds.1 v2:172.21.15.40:6832/2117861921 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56456790b7e0 con 0x5645679bd140 2024-04-26T23:06:00.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f2426a00640 10 client.7038 handle_command_reply: tid=0 2024-04-26T23:06:00.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f244b793740 1 client.7038 shutdown 2024-04-26T23:06:00.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f244b793740 1 -- 172.21.15.40:0/825778771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679e1fb0 msgr2=0x5645679bee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f244b793740 1 --2- 172.21.15.40:0/825778771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679e1fb0 0x5645679bee30 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f242c009c60 tx=0x7f242c0136b0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f244b793740 1 -- 172.21.15.40:0/825778771 shutdown_connections 2024-04-26T23:06:00.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f244b793740 1 --2- 172.21.15.40:0/825778771 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x5645679bd140 0x5645679ef7b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f244b793740 1 --2- 172.21.15.40:0/825778771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564567993ff0 0x5645679ef270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f244b793740 1 --2- 172.21.15.40:0/825778771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679e1fb0 0x5645679bee30 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f244b793740 1 --2- 172.21.15.40:0/825778771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679c4710 0x5645679be8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f2426a00640 0 client.7038 ms_handle_reset on v2:172.21.15.40:6832/2117861921 2024-04-26T23:06:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f244b793740 1 -- 172.21.15.40:0/825778771 >> 172.21.15.40:0/825778771 conn(0x56456799a3f0 msgr2=0x5645679982c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f244b793740 1 -- 172.21.15.40:0/825778771 shutdown_connections 2024-04-26T23:06:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.950+0000 7f244b793740 1 -- 172.21.15.40:0/825778771 wait complete. 2024-04-26T23:06:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.951+0000 7f244b793740 20 client.7038 trim_cache size 0 max 16384 2024-04-26T23:06:00.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.967+0000 7f244b793740 1 Processor -- start 2024-04-26T23:06:00.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.967+0000 7f244b793740 1 -- start start 2024-04-26T23:06:00.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.968+0000 7f244b793740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564567993ff0 0x5645679bee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.968+0000 7f244b793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 0x56456799b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.968+0000 7f244b793740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679e1fb0 0x564567990340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.968+0000 7f244b793740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f24442023e0 con 0x5645679c4710 2024-04-26T23:06:00.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.968+0000 7f244b793740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24442015b0 con 0x5645679e1fb0 2024-04-26T23:06:00.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.968+0000 7f244b793740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2444201cb0 con 0x564567993ff0 2024-04-26T23:06:00.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.968+0000 7f243be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 0x56456799b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.968+0000 7f243be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 0x56456799b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33500/0 (socket says 172.21.15.40:33500) 2024-04-26T23:06:00.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.969+0000 7f243be00640 1 -- 172.21.15.40:0/1743891835 learned_addr learned my addr 172.21.15.40:0/1743891835 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:00.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.969+0000 7f243be00640 1 -- 172.21.15.40:0/1743891835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564567993ff0 msgr2=0x5645679bee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.969+0000 7f243be00640 1 --2- 172.21.15.40:0/1743891835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564567993ff0 0x5645679bee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.969+0000 7f243be00640 1 -- 172.21.15.40:0/1743891835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679e1fb0 msgr2=0x564567990340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.969+0000 7f243be00640 1 --2- 172.21.15.40:0/1743891835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679e1fb0 0x564567990340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.969+0000 7f243be00640 1 -- 172.21.15.40:0/1743891835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645675cf0f0 con 0x5645679c4710 2024-04-26T23:06:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.969+0000 7f243be00640 1 --2- 172.21.15.40:0/1743891835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 0x56456799b730 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f242c013cb0 tx=0x7f242c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398071f6d97314ae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.970+0000 7f2431c00640 1 -- 172.21.15.40:0/1743891835 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c01a040 con 0x5645679c4710 2024-04-26T23:06:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.970+0000 7f2431c00640 1 -- 172.21.15.40:0/1743891835 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f242c02e070 con 0x5645679c4710 2024-04-26T23:06:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.970+0000 7f2431c00640 1 -- 172.21.15.40:0/1743891835 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c006280 con 0x5645679c4710 2024-04-26T23:06:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.970+0000 7f244b793740 1 -- 172.21.15.40:0/1743891835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 msgr2=0x56456799b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.970+0000 7f244b793740 1 --2- 172.21.15.40:0/1743891835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 0x56456799b730 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f242c013cb0 tx=0x7f242c011000 comp rx=0 tx=0).stop 2024-04-26T23:06:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.971+0000 7f244b793740 1 -- 172.21.15.40:0/1743891835 shutdown_connections 2024-04-26T23:06:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.971+0000 7f244b793740 1 --2- 172.21.15.40:0/1743891835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645679e1fb0 0x564567990340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.971+0000 7f244b793740 1 --2- 172.21.15.40:0/1743891835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 0x56456799b730 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.971+0000 7f244b793740 1 --2- 172.21.15.40:0/1743891835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564567993ff0 0x5645679bee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.971+0000 7f244b793740 1 -- 172.21.15.40:0/1743891835 >> 172.21.15.40:0/1743891835 conn(0x56456799a3f0 msgr2=0x5645679c7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:00.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.971+0000 7f244b793740 1 -- 172.21.15.40:0/1743891835 shutdown_connections 2024-04-26T23:06:00.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.971+0000 7f244b793740 1 -- 172.21.15.40:0/1743891835 wait complete. 2024-04-26T23:06:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.971+0000 7f244b793740 1 -- start start 2024-04-26T23:06:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.972+0000 7f244b793740 1 Processor -- start 2024-04-26T23:06:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f244b793740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564567993ff0 0x564567a58260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f244b793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 0x564567a5bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f244b793740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645679e1fb0 0x564567a590f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f244b793740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56456790afd0 con 0x5645679c4710 2024-04-26T23:06:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f244b793740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24440baa30 con 0x564567993ff0 2024-04-26T23:06:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f244b793740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564567a5c870 con 0x5645679e1fb0 2024-04-26T23:06:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f244b793740 10 client.0 _ll_register_callbacks cb 0x7f242c009580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f243be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 0x564567a5bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f243be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 0x564567a5bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33514/0 (socket says 172.21.15.40:33514) 2024-04-26T23:06:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f243be00640 1 -- 172.21.15.40:0/745718169 learned_addr learned my addr 172.21.15.40:0/745718169 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f2441e00640 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564567993ff0 0x564567a58260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f243be00640 1 -- 172.21.15.40:0/745718169 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645679e1fb0 msgr2=0x564567a590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f2442800640 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645679e1fb0 0x564567a590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f243be00640 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645679e1fb0 0x564567a590f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f243be00640 1 -- 172.21.15.40:0/745718169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564567993ff0 msgr2=0x564567a58260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:00.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f243be00640 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564567993ff0 0x564567a58260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f243be00640 1 -- 172.21.15.40:0/745718169 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645678c5a30 con 0x5645679c4710 2024-04-26T23:06:00.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.973+0000 7f2441e00640 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564567993ff0 0x564567a58260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:00.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.974+0000 7f243be00640 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 0x564567a5bdf0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f242c013f10 tx=0x7f242c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65cd66f90912dfa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:00.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.974+0000 7f2426a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:00.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.974+0000 7f2426a00640 1 -- 172.21.15.40:0/745718169 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c036070 con 0x5645679c4710 2024-04-26T23:06:00.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.974+0000 7f2426a00640 1 -- 172.21.15.40:0/745718169 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f242c0169b0 con 0x5645679c4710 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.974+0000 7f244b793740 1 -- 172.21.15.40:0/745718169 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f242c02e070 con 0x5645679c4710 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.974+0000 7f2426a00640 1 -- 172.21.15.40:0/745718169 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f242c043050 con 0x5645679c4710 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.974+0000 7f244b793740 10 client.7044 fetch_fsmap learned FSMap version 62 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.974+0000 7f244b793740 1 -- 172.21.15.40:0/745718169 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) -- 0x564567a5c330 con 0x5645679c4710 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.975+0000 7f2426a00640 1 -- 172.21.15.40:0/745718169 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f242c02e070 con 0x5645679c4710 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.975+0000 7f2426a00640 1 -- 172.21.15.40:0/745718169 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 62) ==== 3546+0+0 (secure 0 0 0) 0x7f242c046020 con 0x5645679c4710 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.975+0000 7f244b793740 10 client.7044 fetch_fsmap finished waiting for FSMap version 62 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.975+0000 7f244b793740 10 client.7044 resolve_mds: resolved 4:1 to role '4:1' aka daemon mds.a 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.975+0000 7f244b793740 20 client.7044 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.975+0000 7f244b793740 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x564567997eb0 0x56456799a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.975+0000 7f244b793740 4 client.7044 mds_command: new command op to 5614 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.975+0000 7f244b793740 1 -- 172.21.15.40:0/745718169 --> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564567905650 con 0x564567997eb0 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.975+0000 7f2441e00640 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x564567997eb0 0x56456799a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.976+0000 7f2441e00640 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x564567997eb0 0x56456799a810 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=81dbc700805f1864 server_cookie=e012a78e5125d106 in_seq=0 out_seq=0 2024-04-26T23:06:00.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.976+0000 7f2426a00640 10 client.7044 ms_handle_connect on v2:172.21.15.40:6832/2117861921 2024-04-26T23:06:00.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f2426a00640 1 -- 172.21.15.40:0/745718169 <== mds.1 v2:172.21.15.40:6832/2117861921 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564567905650 con 0x564567997eb0 2024-04-26T23:06:00.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f2426a00640 10 client.7044 handle_command_reply: tid=0 2024-04-26T23:06:00.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f244b793740 1 client.7044 shutdown 2024-04-26T23:06:00.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f244b793740 1 -- 172.21.15.40:0/745718169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 msgr2=0x564567a5bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f244b793740 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 0x564567a5bdf0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f242c013f10 tx=0x7f242c009190 comp rx=0 tx=0).stop 2024-04-26T23:06:00.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f244b793740 1 -- 172.21.15.40:0/745718169 shutdown_connections 2024-04-26T23:06:00.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f244b793740 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x564567997eb0 0x56456799a810 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f244b793740 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645679e1fb0 0x564567a590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f244b793740 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645679c4710 0x564567a5bdf0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f244b793740 1 --2- 172.21.15.40:0/745718169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564567993ff0 0x564567a58260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f2426a00640 0 client.7044 ms_handle_reset on v2:172.21.15.40:6832/2117861921 2024-04-26T23:06:00.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f244b793740 1 -- 172.21.15.40:0/745718169 >> 172.21.15.40:0/745718169 conn(0x56456799a3f0 msgr2=0x56456799b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:00.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f244b793740 1 -- 172.21.15.40:0/745718169 shutdown_connections 2024-04-26T23:06:00.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f244b793740 1 -- 172.21.15.40:0/745718169 wait complete. 2024-04-26T23:06:00.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.993+0000 7f244b793740 20 client.7044 trim_cache size 0 max 16384 2024-04-26T23:06:00.995 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:06:00.995 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:06:00.995 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:06:00.995 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:06:00.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.994+0000 7f2443200640 1 -- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f242006c940 msgr2=0x7f242006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.994+0000 7f2443200640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f242006c940 0x7f242006ee00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2434009a90 tx=0x7f2434008040 comp rx=0 tx=0).stop 2024-04-26T23:06:00.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.994+0000 7f2443200640 1 -- 172.21.15.40:0/2540238862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444161ba0 msgr2=0x7f244406d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:00.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.994+0000 7f2443200640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444161ba0 0x7f244406d6e0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f243c00a7a0 tx=0x7f243c001cf0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.994+0000 7f2443200640 1 -- 172.21.15.40:0/2540238862 shutdown_connections 2024-04-26T23:06:00.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.994+0000 7f2443200640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f242006c940 0x7f242006ee00 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.994+0000 7f2443200640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444161ba0 0x7f244406d6e0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.994+0000 7f2443200640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24440b90e0 0x7f2444161660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.994+0000 7f2443200640 1 --2- 172.21.15.40:0/2540238862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2428004d40 0x7f244415aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:00.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.995+0000 7f2443200640 1 -- 172.21.15.40:0/2540238862 >> 172.21.15.40:0/2540238862 conn(0x7f24440d4dd0 msgr2=0x7f244415da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:00.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.995+0000 7f2443200640 1 -- 172.21.15.40:0/2540238862 shutdown_connections 2024-04-26T23:06:00.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:00.995+0000 7f2443200640 1 -- 172.21.15.40:0/2540238862 wait complete. 2024-04-26T23:06:01.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:2 flush journal 2024-04-26T23:06:01.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.211+0000 7f87c8a00640 1 Processor -- start 2024-04-26T23:06:01.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.211+0000 7f87c8a00640 1 -- start start 2024-04-26T23:06:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.212+0000 7f87c8a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87c406f790 0x7f87c406fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.213+0000 7f87c8a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87c4156840 con 0x7f87c4156460 2024-04-26T23:06:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.213+0000 7f87c8a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87c40cbf20 con 0x7f87c4072c50 2024-04-26T23:06:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.213+0000 7f87c8a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f87c40cc200 con 0x7f87c406f790 2024-04-26T23:06:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.213+0000 7f87c1e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f87c4156460 0x7f87c40cb4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:46320/0 (socket says 172.21.15.40:46320) 2024-04-26T23:06:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.213+0000 7f87c1e00640 1 -- 172.21.15.40:0/2927441221 learned_addr learned my addr 172.21.15.40:0/2927441221 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.213+0000 7f87c1400640 1 --2- 172.21.15.40:0/2927441221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87c406f790 0x7f87c406fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.213+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2494805375 0 0) 0x7f87c4156840 con 0x7f87c4156460 2024-04-26T23:06:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.213+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87a8005730 con 0x7f87c4156460 2024-04-26T23:06:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 876162091 0 0) 0x7f87a8005730 con 0x7f87c4156460 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87c406f790 msgr2=0x7f87c406fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c0a00640 1 --2- 172.21.15.40:0/2927441221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87c406f790 0x7f87c406fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 >> v1:172.21.15.103:6789/0 conn(0x7f87c4072c50 legacy=0x7f87c40730c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c40cc4e0 con 0x7f87c4156460 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f87b4001670 con 0x7f87c4156460 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 >> v1:172.21.15.40:6789/0 conn(0x7f87c4156460 legacy=0x7f87c40cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c0a00640 1 --2- 172.21.15.40:0/2927441221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87c406f790 0x7f87a8005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c0a00640 1 --2- 172.21.15.40:0/2927441221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a8006270 0x7f87a8006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c0a00640 1 --2- 172.21.15.40:0/2927441221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87a8006f50 0x7f87a80073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f87c40cc200 con 0x7f87a8006f50 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87a800de10 con 0x7f87a8006270 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f87a800e010 con 0x7f87c406f790 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.214+0000 7f87c2800640 1 --2- 172.21.15.40:0/2927441221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87a8006f50 0x7f87a80073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.215+0000 7f87c1e00640 1 --2- 172.21.15.40:0/2927441221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87c406f790 0x7f87a8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.215+0000 7f87c1400640 1 --2- 172.21.15.40:0/2927441221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a8006270 0x7f87a8006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.215+0000 7f87c2800640 1 -- 172.21.15.40:0/2927441221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87c406f790 msgr2=0x7f87a8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:01.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.215+0000 7f87c2800640 1 --2- 172.21.15.40:0/2927441221 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87c406f790 0x7f87a8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.215+0000 7f87c2800640 1 -- 172.21.15.40:0/2927441221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a8006270 msgr2=0x7f87a8006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:01.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.215+0000 7f87c2800640 1 --2- 172.21.15.40:0/2927441221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a8006270 0x7f87a8006a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.215+0000 7f87c2800640 1 -- 172.21.15.40:0/2927441221 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a8005730 con 0x7f87a8006f50 2024-04-26T23:06:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.216+0000 7f87c2800640 1 --2- 172.21.15.40:0/2927441221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87a8006f50 0x7f87a80073d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f87bc001e50 tx=0x7f87a8007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980406ba809dc8e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.216+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc001a70 con 0x7f87a8006f50 2024-04-26T23:06:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.216+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87bc003970 con 0x7f87a8006f50 2024-04-26T23:06:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.216+0000 7f87c0a00640 1 -- 172.21.15.40:0/2927441221 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc003d60 con 0x7f87a8006f50 2024-04-26T23:06:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.217+0000 7f87c8a00640 1 -- 172.21.15.40:0/2927441221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87a8006f50 msgr2=0x7f87a80073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.217+0000 7f87c8a00640 1 --2- 172.21.15.40:0/2927441221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87a8006f50 0x7f87a80073d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f87bc001e50 tx=0x7f87a8007a40 comp rx=0 tx=0).stop 2024-04-26T23:06:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.217+0000 7f87c2800640 1 -- 172.21.15.40:0/2927441221 reap_dead start 2024-04-26T23:06:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.217+0000 7f87c8a00640 1 -- 172.21.15.40:0/2927441221 shutdown_connections 2024-04-26T23:06:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.217+0000 7f87c8a00640 1 -- 172.21.15.40:0/2927441221 >> 172.21.15.40:0/2927441221 conn(0x7f87c40d0be0 msgr2=0x7f87c40d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:01.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.217+0000 7f87c8a00640 1 -- 172.21.15.40:0/2927441221 shutdown_connections 2024-04-26T23:06:01.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.217+0000 7f87c8a00640 1 -- 172.21.15.40:0/2927441221 wait complete. 2024-04-26T23:06:01.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.219+0000 7f87c8a00640 1 Processor -- start 2024-04-26T23:06:01.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.219+0000 7f87c8a00640 1 -- start start 2024-04-26T23:06:01.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.220+0000 7f87c8a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87a8006f50 0x7f87c4154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.220+0000 7f87c8a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87c40c4270 0x7f87c40b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.220+0000 7f87c8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c415e5f0 0x7f87c40b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.220+0000 7f87c8a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f87c40cbbe0 con 0x7f87c415e5f0 2024-04-26T23:06:01.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.220+0000 7f87c8a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87c4072700 con 0x7f87c40c4270 2024-04-26T23:06:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.220+0000 7f87c8a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f87c406e260 con 0x7f87a8006f50 2024-04-26T23:06:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.220+0000 7f87c2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c415e5f0 0x7f87c40b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.220+0000 7f87c2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c415e5f0 0x7f87c40b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33520/0 (socket says 172.21.15.40:33520) 2024-04-26T23:06:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.220+0000 7f87c2800640 1 -- 172.21.15.40:0/1997759843 learned_addr learned my addr 172.21.15.40:0/1997759843 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.221+0000 7f87c1e00640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87a8006f50 0x7f87c4154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.221+0000 7f87c1400640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87c40c4270 0x7f87c40b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.221+0000 7f87c2800640 1 -- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87a8006f50 msgr2=0x7f87c4154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.221+0000 7f87c2800640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87a8006f50 0x7f87c4154db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.221+0000 7f87c2800640 1 -- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87c40c4270 msgr2=0x7f87c40b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.221+0000 7f87c2800640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87c40c4270 0x7f87c40b3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.221+0000 7f87c2800640 1 -- 172.21.15.40:0/1997759843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c40b05d0 con 0x7f87c415e5f0 2024-04-26T23:06:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.221+0000 7f87c2800640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c415e5f0 0x7f87c40b3da0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f87bc005040 tx=0x7f87bc0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:01.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.222+0000 7f87ba000640 1 -- 172.21.15.40:0/1997759843 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc037040 con 0x7f87c415e5f0 2024-04-26T23:06:01.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.222+0000 7f87ba000640 1 -- 172.21.15.40:0/1997759843 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87bc003e50 con 0x7f87c415e5f0 2024-04-26T23:06:01.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.222+0000 7f87ba000640 1 -- 172.21.15.40:0/1997759843 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc007db0 con 0x7f87c415e5f0 2024-04-26T23:06:01.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.222+0000 7f87c8a00640 1 -- 172.21.15.40:0/1997759843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87c40b0890 con 0x7f87c415e5f0 2024-04-26T23:06:01.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.222+0000 7f87c8a00640 1 -- 172.21.15.40:0/1997759843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87c40c7d50 con 0x7f87c415e5f0 2024-04-26T23:06:01.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.224+0000 7f87ba000640 1 -- 172.21.15.40:0/1997759843 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f87bc042020 con 0x7f87c415e5f0 2024-04-26T23:06:01.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.225+0000 7f87ba000640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f87a406c920 0x7f87a406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.225+0000 7f87ba000640 1 -- 172.21.15.40:0/1997759843 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f87bc0ab3e0 con 0x7f87c415e5f0 2024-04-26T23:06:01.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.226+0000 7f87cb9d8740 1 Processor -- start 2024-04-26T23:06:01.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.226+0000 7f87c1e00640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f87a406c920 0x7f87a406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.226+0000 7f87cb9d8740 1 -- start start 2024-04-26T23:06:01.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.227+0000 7f87cb9d8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a09af0 0x55f4c29e2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.227+0000 7f87cb9d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a09ed0 0x55f4c2a0c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.227+0000 7f87cb9d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 0x55f4c29e00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.227+0000 7f87cb9d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f4c28f2030 con 0x55f4c2a0c890 2024-04-26T23:06:01.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.227+0000 7f87cb9d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f4c290d7f0 con 0x55f4c2a09ed0 2024-04-26T23:06:01.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.227+0000 7f87cb9d8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55f4c28c87e0 con 0x55f4c2a09af0 2024-04-26T23:06:01.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.227+0000 7f87c1e00640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f87a406c920 0x7f87a406ede0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f87b40058b0 tx=0x7f87b4000f50 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:01.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.227+0000 7f87c2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 0x55f4c29e00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.227+0000 7f87c2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 0x55f4c29e00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33524/0 (socket says 172.21.15.40:33524) 2024-04-26T23:06:01.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.227+0000 7f87c2800640 1 -- 172.21.15.40:0/3875090908 learned_addr learned my addr 172.21.15.40:0/3875090908 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:01.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.228+0000 7f87c1e00640 1 --2- 172.21.15.40:0/3875090908 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a09af0 0x55f4c29e2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.228+0000 7f87c1400640 1 --2- 172.21.15.40:0/3875090908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a09ed0 0x55f4c2a0c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.228+0000 7f87c2800640 1 -- 172.21.15.40:0/3875090908 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a09af0 msgr2=0x55f4c29e2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:01.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.228+0000 7f87c2800640 1 --2- 172.21.15.40:0/3875090908 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a09af0 0x55f4c29e2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.228+0000 7f87c2800640 1 -- 172.21.15.40:0/3875090908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a09ed0 msgr2=0x55f4c2a0c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:01.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.228+0000 7f87c2800640 1 --2- 172.21.15.40:0/3875090908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a09ed0 0x55f4c2a0c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.228+0000 7f87c2800640 1 -- 172.21.15.40:0/3875090908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4c28d4050 con 0x55f4c2a0c890 2024-04-26T23:06:01.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.228+0000 7f87c2800640 1 --2- 172.21.15.40:0/3875090908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 0x55f4c29e00f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0a9660 tx=0x7f87bc078fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be9be47cdf04782 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:01.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.228+0000 7f87ae600640 1 -- 172.21.15.40:0/3875090908 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc0771f0 con 0x55f4c2a0c890 2024-04-26T23:06:01.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.228+0000 7f87ae600640 1 -- 172.21.15.40:0/3875090908 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87bc001a70 con 0x55f4c2a0c890 2024-04-26T23:06:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.228+0000 7f87ae600640 1 -- 172.21.15.40:0/3875090908 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc075560 con 0x55f4c2a0c890 2024-04-26T23:06:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.229+0000 7f87cb9d8740 1 -- 172.21.15.40:0/3875090908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 msgr2=0x55f4c29e00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.229+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/3875090908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 0x55f4c29e00f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0a9660 tx=0x7f87bc078fb0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.229+0000 7f87cb9d8740 1 -- 172.21.15.40:0/3875090908 shutdown_connections 2024-04-26T23:06:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.229+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/3875090908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 0x55f4c29e00f0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.229+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/3875090908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a09ed0 0x55f4c2a0c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.229+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/3875090908 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a09af0 0x55f4c29e2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.229+0000 7f87cb9d8740 1 -- 172.21.15.40:0/3875090908 >> 172.21.15.40:0/3875090908 conn(0x55f4c29deb50 msgr2=0x55f4c2a27350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.229+0000 7f87cb9d8740 1 -- 172.21.15.40:0/3875090908 shutdown_connections 2024-04-26T23:06:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.229+0000 7f87cb9d8740 1 -- 172.21.15.40:0/3875090908 wait complete. 2024-04-26T23:06:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.230+0000 7f87cb9d8740 1 -- start start 2024-04-26T23:06:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.230+0000 7f87cb9d8740 1 Processor -- start 2024-04-26T23:06:01.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.231+0000 7f87cb9d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 0x55f4c2a08fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.231+0000 7f87cb9d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0c890 0x55f4c2a0d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.231+0000 7f87cb9d8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a0dc40 0x55f4c29db7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.231+0000 7f87cb9d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f4c294a200 con 0x55f4c2a09ed0 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.231+0000 7f87cb9d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f4c2944ee0 con 0x55f4c2a0c890 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.231+0000 7f87cb9d8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55f4c2951e70 con 0x55f4c2a0dc40 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.231+0000 7f87cb9d8740 10 client.0 _ll_register_callbacks cb 0x55f4c293b270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.232+0000 7f87c1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 0x55f4c2a08fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.232+0000 7f87c1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 0x55f4c2a08fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33532/0 (socket says 172.21.15.40:33532) 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.232+0000 7f87c1e00640 1 -- 172.21.15.40:0/2245438510 learned_addr learned my addr 172.21.15.40:0/2245438510 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.232+0000 7f87c1400640 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0c890 0x55f4c2a0d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.232+0000 7f87c2800640 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a0dc40 0x55f4c29db7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.232+0000 7f87c1e00640 1 -- 172.21.15.40:0/2245438510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a0dc40 msgr2=0x55f4c29db7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.232+0000 7f87c1e00640 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a0dc40 0x55f4c29db7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.232+0000 7f87c1e00640 1 -- 172.21.15.40:0/2245438510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0c890 msgr2=0x55f4c2a0d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.232+0000 7f87c1e00640 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0c890 0x55f4c2a0d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.232+0000 7f87c1e00640 1 -- 172.21.15.40:0/2245438510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4c27217d0 con 0x55f4c2a09ed0 2024-04-26T23:06:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.232+0000 7f87c1400640 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0c890 0x55f4c2a0d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.232+0000 7f87c1e00640 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 0x55f4c2a08fa0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f87b400ff80 tx=0x7f87b4009070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a761960b57aed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.232+0000 7f87a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.233+0000 7f87a2a00640 1 -- 172.21.15.40:0/2245438510 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4028060 con 0x55f4c2a09ed0 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.233+0000 7f87a2a00640 1 -- 172.21.15.40:0/2245438510 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f87b4023c90 con 0x55f4c2a09ed0 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.233+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2245438510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87bc001a70 con 0x55f4c2a09ed0 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.233+0000 7f87a2a00640 1 -- 172.21.15.40:0/2245438510 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f87b4010040 con 0x55f4c2a09ed0 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.233+0000 7f87cb9d8740 10 client.7056 fetch_fsmap learned FSMap version 62 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.233+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2245438510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) -- 0x55f4c26573f0 con 0x55f4c2a09ed0 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.233+0000 7f87a2a00640 1 -- 172.21.15.40:0/2245438510 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87bc001a70 con 0x55f4c2a09ed0 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.234+0000 7f87a2a00640 1 -- 172.21.15.40:0/2245438510 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 62) ==== 3546+0+0 (secure 0 0 0) 0x7f87b4013070 con 0x55f4c2a09ed0 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.234+0000 7f87cb9d8740 10 client.7056 fetch_fsmap finished waiting for FSMap version 62 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.234+0000 7f87cb9d8740 10 client.7056 resolve_mds: resolved 4:2 to role '4:2' aka daemon mds.b 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.234+0000 7f87cb9d8740 20 client.7056 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.234+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x55f4c2a267b0 0x55f4c2a26c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.234+0000 7f87cb9d8740 4 client.7056 mds_command: new command op to 5324 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:06:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.234+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2245438510 --> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4c29537e0 con 0x55f4c2a267b0 2024-04-26T23:06:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.234+0000 7f87c1400640 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x55f4c2a267b0 0x55f4c2a26c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.235+0000 7f87c1400640 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x55f4c2a267b0 0x55f4c2a26c40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fe6d513efc309053 server_cookie=4fe1a0a90c704463 in_seq=0 out_seq=0 2024-04-26T23:06:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.235+0000 7f87a2a00640 10 client.7056 ms_handle_connect on v2:172.21.15.103:6835/2595375825 2024-04-26T23:06:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.237+0000 7f87a2a00640 1 -- 172.21.15.40:0/2245438510 <== mds.2 v2:172.21.15.103:6835/2595375825 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f4c29537e0 con 0x55f4c2a267b0 2024-04-26T23:06:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.237+0000 7f87a2a00640 10 client.7056 handle_command_reply: tid=0 2024-04-26T23:06:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.237+0000 7f87cb9d8740 1 client.7056 shutdown 2024-04-26T23:06:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.237+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2245438510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 msgr2=0x55f4c2a08fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.237+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 0x55f4c2a08fa0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f87b400ff80 tx=0x7f87b4009070 comp rx=0 tx=0).stop 2024-04-26T23:06:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.237+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2245438510 shutdown_connections 2024-04-26T23:06:01.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.237+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a0dc40 0x55f4c29db7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.237+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x55f4c2a267b0 0x55f4c2a26c40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.237+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0c890 0x55f4c2a0d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.237+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/2245438510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 0x55f4c2a08fa0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.237+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2245438510 >> 172.21.15.40:0/2245438510 conn(0x55f4c2a09af0 msgr2=0x55f4c2a29db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:01.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.237+0000 7f87a2a00640 0 client.7056 ms_handle_reset on v2:172.21.15.103:6835/2595375825 2024-04-26T23:06:01.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.238+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2245438510 shutdown_connections 2024-04-26T23:06:01.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.238+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2245438510 wait complete. 2024-04-26T23:06:01.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.238+0000 7f87cb9d8740 20 client.7056 trim_cache size 0 max 16384 2024-04-26T23:06:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.254+0000 7f87cb9d8740 1 Processor -- start 2024-04-26T23:06:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.254+0000 7f87cb9d8740 1 -- start start 2024-04-26T23:06:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.255+0000 7f87cb9d8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a09ed0 0x55f4c2a08f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.255+0000 7f87cb9d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 0x55f4c2a06bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.255+0000 7f87cb9d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0dc40 0x55f4c2a2a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.255+0000 7f87cb9d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41fe310 con 0x55f4c2a0c890 2024-04-26T23:06:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.255+0000 7f87cb9d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41fd4e0 con 0x55f4c2a0dc40 2024-04-26T23:06:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.255+0000 7f87cb9d8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f87c41fdbe0 con 0x55f4c2a09ed0 2024-04-26T23:06:01.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.256+0000 7f87c1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 0x55f4c2a06bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.256+0000 7f87c1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 0x55f4c2a06bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33534/0 (socket says 172.21.15.40:33534) 2024-04-26T23:06:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.256+0000 7f87c1400640 1 -- 172.21.15.40:0/3330466628 learned_addr learned my addr 172.21.15.40:0/3330466628 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.256+0000 7f87c2800640 1 --2- 172.21.15.40:0/3330466628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0dc40 0x55f4c2a2a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.256+0000 7f87c1e00640 1 --2- 172.21.15.40:0/3330466628 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a09ed0 0x55f4c2a08f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.256+0000 7f87c1400640 1 -- 172.21.15.40:0/3330466628 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a09ed0 msgr2=0x55f4c2a08f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.256+0000 7f87c1400640 1 --2- 172.21.15.40:0/3330466628 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a09ed0 0x55f4c2a08f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.256+0000 7f87c1400640 1 -- 172.21.15.40:0/3330466628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0dc40 msgr2=0x55f4c2a2a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.256+0000 7f87c1400640 1 --2- 172.21.15.40:0/3330466628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0dc40 0x55f4c2a2a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.256+0000 7f87c1400640 1 -- 172.21.15.40:0/3330466628 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4c26170f0 con 0x55f4c2a0c890 2024-04-26T23:06:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.257+0000 7f87c1400640 1 --2- 172.21.15.40:0/3330466628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 0x55f4c2a06bf0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f87b000d9a0 tx=0x7f87b000df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ec5c45318d087e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:01.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.257+0000 7f87adc00640 1 -- 172.21.15.40:0/3330466628 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b0017ab0 con 0x55f4c2a0c890 2024-04-26T23:06:01.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.257+0000 7f87adc00640 1 -- 172.21.15.40:0/3330466628 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b0020030 con 0x55f4c2a0c890 2024-04-26T23:06:01.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.257+0000 7f87adc00640 1 -- 172.21.15.40:0/3330466628 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b0004030 con 0x55f4c2a0c890 2024-04-26T23:06:01.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.258+0000 7f87cb9d8740 1 -- 172.21.15.40:0/3330466628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 msgr2=0x55f4c2a06bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:01.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.258+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/3330466628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 0x55f4c2a06bf0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f87b000d9a0 tx=0x7f87b000df00 comp rx=0 tx=0).stop 2024-04-26T23:06:01.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.258+0000 7f87cb9d8740 1 -- 172.21.15.40:0/3330466628 shutdown_connections 2024-04-26T23:06:01.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.258+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/3330466628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0dc40 0x55f4c2a2a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.258+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/3330466628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a0c890 0x55f4c2a06bf0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.258+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/3330466628 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a09ed0 0x55f4c2a08f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.258+0000 7f87cb9d8740 1 -- 172.21.15.40:0/3330466628 >> 172.21.15.40:0/3330466628 conn(0x55f4c2a09af0 msgr2=0x55f4c2a0cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.258+0000 7f87cb9d8740 1 -- 172.21.15.40:0/3330466628 shutdown_connections 2024-04-26T23:06:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.258+0000 7f87cb9d8740 1 -- 172.21.15.40:0/3330466628 wait complete. 2024-04-26T23:06:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.259+0000 7f87cb9d8740 1 -- start start 2024-04-26T23:06:01.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.259+0000 7f87cb9d8740 1 Processor -- start 2024-04-26T23:06:01.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.260+0000 7f87cb9d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 0x55f4c29e1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.260+0000 7f87cb9d8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a0c890 0x55f4c2aa5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.260+0000 7f87cb9d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0dc40 0x55f4c2a2a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.260+0000 7f87cb9d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f4c2952fd0 con 0x55f4c2a09ed0 2024-04-26T23:06:01.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.260+0000 7f87cb9d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41f5090 con 0x55f4c2a0dc40 2024-04-26T23:06:01.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.260+0000 7f87cb9d8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55f4c2a20f40 con 0x55f4c2a0c890 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.260+0000 7f87cb9d8740 10 client.0 _ll_register_callbacks cb 0x7f87b4005f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.260+0000 7f87c1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 0x55f4c29e1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.260+0000 7f87c1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 0x55f4c29e1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33540/0 (socket says 172.21.15.40:33540) 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.260+0000 7f87c1e00640 1 -- 172.21.15.40:0/2791247325 learned_addr learned my addr 172.21.15.40:0/2791247325 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.261+0000 7f87c2800640 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0dc40 0x55f4c2a2a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.261+0000 7f87c1e00640 1 -- 172.21.15.40:0/2791247325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a0c890 msgr2=0x55f4c2aa5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.261+0000 7f87c1400640 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a0c890 0x55f4c2aa5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.261+0000 7f87c1e00640 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a0c890 0x55f4c2aa5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.261+0000 7f87c1e00640 1 -- 172.21.15.40:0/2791247325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0dc40 msgr2=0x55f4c2a2a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.261+0000 7f87c1e00640 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0dc40 0x55f4c2a2a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.261+0000 7f87c1e00640 1 -- 172.21.15.40:0/2791247325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4c290da30 con 0x55f4c2a09ed0 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.261+0000 7f87c2800640 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0dc40 0x55f4c2a2a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.261+0000 7f87c1e00640 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 0x55f4c29e1bb0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f87b4014790 tx=0x7f87c415c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305c2fcca71c2b84 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:01.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.261+0000 7f87a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.261+0000 7f87a2a00640 1 -- 172.21.15.40:0/2791247325 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b400fb00 con 0x55f4c2a09ed0 2024-04-26T23:06:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.261+0000 7f87a2a00640 1 -- 172.21.15.40:0/2791247325 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f87b401f070 con 0x55f4c2a09ed0 2024-04-26T23:06:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.261+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2791247325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87b0020030 con 0x55f4c2a09ed0 2024-04-26T23:06:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.262+0000 7f87a2a00640 1 -- 172.21.15.40:0/2791247325 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f87b4014b80 con 0x55f4c2a09ed0 2024-04-26T23:06:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.262+0000 7f87cb9d8740 10 client.7062 fetch_fsmap learned FSMap version 62 2024-04-26T23:06:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.262+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2791247325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) -- 0x55f4c29e1410 con 0x55f4c2a09ed0 2024-04-26T23:06:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.262+0000 7f87a2a00640 1 -- 172.21.15.40:0/2791247325 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b0020030 con 0x55f4c2a09ed0 2024-04-26T23:06:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.262+0000 7f87a2a00640 1 -- 172.21.15.40:0/2791247325 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 62) ==== 3546+0+0 (secure 0 0 0) 0x7f87b4040080 con 0x55f4c2a09ed0 2024-04-26T23:06:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.262+0000 7f87cb9d8740 10 client.7062 fetch_fsmap finished waiting for FSMap version 62 2024-04-26T23:06:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.262+0000 7f87cb9d8740 10 client.7062 resolve_mds: resolved 4:2 to role '4:2' aka daemon mds.b 2024-04-26T23:06:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.262+0000 7f87cb9d8740 20 client.7062 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.262+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x55f4c2aa8190 0x55f4c2aa85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.262+0000 7f87cb9d8740 4 client.7062 mds_command: new command op to 5324 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:06:01.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.262+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2791247325 --> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f4c294d650 con 0x55f4c2aa8190 2024-04-26T23:06:01.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.263+0000 7f87c1400640 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x55f4c2aa8190 0x55f4c2aa85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:01.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.264+0000 7f87c1400640 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x55f4c2aa8190 0x55f4c2aa85d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fc622c7ab127a193 server_cookie=84ccede1c0f97b72 in_seq=0 out_seq=0 2024-04-26T23:06:01.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.264+0000 7f87a2a00640 10 client.7062 ms_handle_connect on v2:172.21.15.103:6835/2595375825 2024-04-26T23:06:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.282+0000 7f87a2a00640 1 -- 172.21.15.40:0/2791247325 <== mds.2 v2:172.21.15.103:6835/2595375825 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f4c294d650 con 0x55f4c2aa8190 2024-04-26T23:06:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.282+0000 7f87a2a00640 10 client.7062 handle_command_reply: tid=0 2024-04-26T23:06:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.282+0000 7f87cb9d8740 1 client.7062 shutdown 2024-04-26T23:06:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.282+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2791247325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 msgr2=0x55f4c29e1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.282+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 0x55f4c29e1bb0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f87b4014790 tx=0x7f87c415c390 comp rx=0 tx=0).stop 2024-04-26T23:06:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.282+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2791247325 shutdown_connections 2024-04-26T23:06:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.282+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x55f4c2aa8190 0x55f4c2aa85d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.283+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f4c2a0dc40 0x55f4c2a2a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.283+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55f4c2a0c890 0x55f4c2aa5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.283+0000 7f87cb9d8740 1 --2- 172.21.15.40:0/2791247325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f4c2a09ed0 0x55f4c29e1bb0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.283+0000 7f87a2a00640 0 client.7062 ms_handle_reset on v2:172.21.15.103:6835/2595375825 2024-04-26T23:06:01.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.283+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2791247325 >> 172.21.15.40:0/2791247325 conn(0x55f4c2a09af0 msgr2=0x55f4c29e0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:01.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.283+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2791247325 shutdown_connections 2024-04-26T23:06:01.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.283+0000 7f87cb9d8740 1 -- 172.21.15.40:0/2791247325 wait complete. 2024-04-26T23:06:01.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.283+0000 7f87cb9d8740 20 client.7062 trim_cache size 0 max 16384 2024-04-26T23:06:01.284 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:06:01.284 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:06:01.284 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:06:01.284 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:06:01.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.284+0000 7f87c8a00640 1 -- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f87a406c920 msgr2=0x7f87a406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:01.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.284+0000 7f87c8a00640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f87a406c920 0x7f87a406ede0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f87b40058b0 tx=0x7f87b4000f50 comp rx=0 tx=0).stop 2024-04-26T23:06:01.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.284+0000 7f87c8a00640 1 -- 172.21.15.40:0/1997759843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c415e5f0 msgr2=0x7f87c40b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:01.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.284+0000 7f87c8a00640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c415e5f0 0x7f87c40b3da0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f87bc005040 tx=0x7f87bc0014a0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.284+0000 7f87c8a00640 1 -- 172.21.15.40:0/1997759843 shutdown_connections 2024-04-26T23:06:01.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.284+0000 7f87c8a00640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f87a406c920 0x7f87a406ede0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.284+0000 7f87c8a00640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c415e5f0 0x7f87c40b3da0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.284+0000 7f87c8a00640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87c40c4270 0x7f87c40b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.284+0000 7f87c8a00640 1 --2- 172.21.15.40:0/1997759843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87a8006f50 0x7f87c4154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:01.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.284+0000 7f87c8a00640 1 -- 172.21.15.40:0/1997759843 >> 172.21.15.40:0/1997759843 conn(0x7f87c40d0be0 msgr2=0x7f87c406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:01.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.284+0000 7f87c8a00640 1 -- 172.21.15.40:0/1997759843 shutdown_connections 2024-04-26T23:06:01.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01.284+0000 7f87c8a00640 1 -- 172.21.15.40:0/1997759843 wait complete. 2024-04-26T23:06:01.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:06:01.303 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec find d1/d2/d3/) 2024-04-26T23:06:01.378 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/ 2024-04-26T23:06:01.379 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4 2024-04-26T23:06:01.379 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5 2024-04-26T23:06:01.379 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6 2024-04-26T23:06:01.379 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6/d7 2024-04-26T23:06:01.379 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6/d7/d8 2024-04-26T23:06:01.380 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:06:01.380 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:06:01.380 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:06:01.380 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:06:01.381 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/").st_ino) 2024-04-26T23:06:01.381 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:06:01.526 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627778 2024-04-26T23:06:01.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:06:01.540 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:06:01.540 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:06:01.540 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:06:01.541 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:06:01.541 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4").st_ino) 2024-04-26T23:06:01.541 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:06:01.738 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627779 2024-04-26T23:06:01.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:06:01.752 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:06:01.752 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:06:01.752 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:06:01.752 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:06:01.753 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5").st_ino) 2024-04-26T23:06:01.753 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:06:01.947 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627780 2024-04-26T23:06:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:06:01.962 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:06:01.962 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:06:01.962 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:06:01.962 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:06:01.962 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6").st_ino) 2024-04-26T23:06:01.962 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:06:02.155 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627781 2024-04-26T23:06:02.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:06:02.169 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:06:02.169 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:06:02.169 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:06:02.169 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:06:02.169 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6/d7").st_ino) 2024-04-26T23:06:02.169 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:06:02.365 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627782 2024-04-26T23:06:02.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:06:02.379 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:06:02.380 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:06:02.380 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:06:02.380 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:06:02.380 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6/d7/d8").st_ino) 2024-04-26T23:06:02.380 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:06:02.571 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627783 2024-04-26T23:06:02.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:06:02.585 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000002.00000000 parent 2024-04-26T23:06:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.746+0000 7f8136d1ef40 1 Processor -- start 2024-04-26T23:06:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.747+0000 7f8136d1ef40 1 -- start start 2024-04-26T23:06:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.748+0000 7f8136d1ef40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556e82670e90 0x556e8263d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.748+0000 7f8136d1ef40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e82672070 0x556e82711c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.748+0000 7f8136d1ef40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e826703e0 0x556e8266d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.748+0000 7f8136d1ef40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556e82717030 con 0x556e82672070 2024-04-26T23:06:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.748+0000 7f8136d1ef40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556e825e6d20 con 0x556e826703e0 2024-04-26T23:06:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.748+0000 7f8136d1ef40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556e82717310 con 0x556e82670e90 2024-04-26T23:06:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.748+0000 7f8132a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e82672070 0x556e82711c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.748+0000 7f8132a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e82672070 0x556e82711c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33552/0 (socket says 172.21.15.40:33552) 2024-04-26T23:06:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.748+0000 7f8132a00640 1 -- 172.21.15.40:0/1074096236 learned_addr learned my addr 172.21.15.40:0/1074096236 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.748+0000 7f8133e00640 1 --2- 172.21.15.40:0/1074096236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e826703e0 0x556e8266d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.748+0000 7f8133400640 1 --2- 172.21.15.40:0/1074096236 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556e82670e90 0x556e8263d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.749+0000 7f8132a00640 1 -- 172.21.15.40:0/1074096236 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556e82670e90 msgr2=0x556e8263d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.749+0000 7f8132a00640 1 --2- 172.21.15.40:0/1074096236 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556e82670e90 0x556e8263d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.749+0000 7f8132a00640 1 -- 172.21.15.40:0/1074096236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e826703e0 msgr2=0x556e8266d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.749+0000 7f8132a00640 1 --2- 172.21.15.40:0/1074096236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e826703e0 0x556e8266d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.749+0000 7f8132a00640 1 -- 172.21.15.40:0/1074096236 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e82710970 con 0x556e82672070 2024-04-26T23:06:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.749+0000 7f8133400640 1 --2- 172.21.15.40:0/1074096236 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556e82670e90 0x556e8263d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.749+0000 7f8132a00640 1 --2- 172.21.15.40:0/1074096236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e82672070 0x556e82711c10 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f812000a7f0 tx=0x7f812000acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54e4b8ff9505562 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:02.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.749+0000 7f8131600640 1 -- 172.21.15.40:0/1074096236 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8120016020 con 0x556e82672070 2024-04-26T23:06:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.749+0000 7f8131600640 1 -- 172.21.15.40:0/1074096236 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f812000d9c0 con 0x556e82672070 2024-04-26T23:06:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.749+0000 7f8131600640 1 -- 172.21.15.40:0/1074096236 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812000dd90 con 0x556e82672070 2024-04-26T23:06:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.750+0000 7f8136d1ef40 1 -- 172.21.15.40:0/1074096236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e82672070 msgr2=0x556e82711c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.750+0000 7f8136d1ef40 1 --2- 172.21.15.40:0/1074096236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e82672070 0x556e82711c10 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f812000a7f0 tx=0x7f812000acb0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.750+0000 7f8136d1ef40 1 -- 172.21.15.40:0/1074096236 shutdown_connections 2024-04-26T23:06:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.750+0000 7f8136d1ef40 1 --2- 172.21.15.40:0/1074096236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e826703e0 0x556e8266d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.750+0000 7f8136d1ef40 1 --2- 172.21.15.40:0/1074096236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e82672070 0x556e82711c10 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.750+0000 7f8136d1ef40 1 --2- 172.21.15.40:0/1074096236 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556e82670e90 0x556e8263d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.750+0000 7f8136d1ef40 1 -- 172.21.15.40:0/1074096236 >> 172.21.15.40:0/1074096236 conn(0x556e8268cdf0 msgr2=0x556e8266c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.750+0000 7f8136d1ef40 1 -- 172.21.15.40:0/1074096236 shutdown_connections 2024-04-26T23:06:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.750+0000 7f8136d1ef40 1 -- 172.21.15.40:0/1074096236 wait complete. 2024-04-26T23:06:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.751+0000 7f8136d1ef40 1 Processor -- start 2024-04-26T23:06:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.751+0000 7f8136d1ef40 1 -- start start 2024-04-26T23:06:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.752+0000 7f8136d1ef40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556e826703e0 0x556e82708590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.752+0000 7f8136d1ef40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e82670e90 0x556e827020c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.752+0000 7f8136d1ef40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e82672070 0x556e8270a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.752+0000 7f8136d1ef40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556e8268c880 con 0x556e82672070 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.752+0000 7f8136d1ef40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556e8266e0b0 con 0x556e82670e90 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.752+0000 7f8136d1ef40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x556e8266ddd0 con 0x556e826703e0 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.753+0000 7f8133e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e82672070 0x556e8270a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.753+0000 7f8133e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e82672070 0x556e8270a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33564/0 (socket says 172.21.15.40:33564) 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.753+0000 7f8133e00640 1 -- 172.21.15.40:0/4161008379 learned_addr learned my addr 172.21.15.40:0/4161008379 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.753+0000 7f8132a00640 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e82670e90 0x556e827020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.753+0000 7f8133400640 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556e826703e0 0x556e82708590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.753+0000 7f8133e00640 1 -- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556e826703e0 msgr2=0x556e82708590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.753+0000 7f8133e00640 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x556e826703e0 0x556e82708590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.753+0000 7f8133e00640 1 -- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e82670e90 msgr2=0x556e827020c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.753+0000 7f8133e00640 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e82670e90 0x556e827020c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.753+0000 7f8133e00640 1 -- 172.21.15.40:0/4161008379 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e82707e30 con 0x556e82672070 2024-04-26T23:06:02.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.754+0000 7f8133e00640 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e82672070 0x556e8270a9f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f812c00de20 tx=0x7f812c00b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:02.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.754+0000 7f812be00640 1 -- 172.21.15.40:0/4161008379 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812c014020 con 0x556e82672070 2024-04-26T23:06:02.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.754+0000 7f812be00640 1 -- 172.21.15.40:0/4161008379 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f812c010630 con 0x556e82672070 2024-04-26T23:06:02.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.754+0000 7f8136d1ef40 1 -- 172.21.15.40:0/4161008379 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556e82709f10 con 0x556e82672070 2024-04-26T23:06:02.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.754+0000 7f812be00640 1 -- 172.21.15.40:0/4161008379 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812c010a20 con 0x556e82672070 2024-04-26T23:06:02.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.754+0000 7f8136d1ef40 1 -- 172.21.15.40:0/4161008379 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e8270a4d0 con 0x556e82672070 2024-04-26T23:06:02.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.757+0000 7f812be00640 1 -- 172.21.15.40:0/4161008379 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f812c017020 con 0x556e82672070 2024-04-26T23:06:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.757+0000 7f812be00640 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f811406b2d0 0x7f811406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.758+0000 7f812be00640 1 -- 172.21.15.40:0/4161008379 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f812c08aee0 con 0x556e82672070 2024-04-26T23:06:02.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.758+0000 7f8136d1ef40 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x556e8270f7a0 0x556e8270fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.758+0000 7f8136d1ef40 1 -- 172.21.15.40:0/4161008379 --> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] -- osd_op(unknown.0.0:1 9.12 9:4bec3886:::10000000002.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67) -- 0x556e8270d580 con 0x556e8270f7a0 2024-04-26T23:06:02.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.758+0000 7f8133400640 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f811406b2d0 0x7f811406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.759+0000 7f8132a00640 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x556e8270f7a0 0x556e8270fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.760+0000 7f8133400640 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f811406b2d0 0x7f811406d790 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8124056000 tx=0x7f8124050000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:02.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.760+0000 7f8132a00640 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x556e8270f7a0 0x556e8270fb80 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.763+0000 7f8132a00640 1 -- 172.21.15.40:0/4161008379 <== osd.5 v2:172.21.15.103:6800/157083760 1 ==== osd_op_reply(1 10000000002.00000000 [rmxattr] v67'8 uv8 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f812000dcf0 con 0x556e8270f7a0 2024-04-26T23:06:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.763+0000 7f8136d1ef40 1 -- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x556e8270f7a0 msgr2=0x556e8270fb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.763+0000 7f8136d1ef40 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x556e8270f7a0 0x556e8270fb80 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.763+0000 7f8136d1ef40 1 -- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f811406b2d0 msgr2=0x7f811406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.763+0000 7f8136d1ef40 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f811406b2d0 0x7f811406d790 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8124056000 tx=0x7f8124050000 comp rx=0 tx=0).stop 2024-04-26T23:06:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.763+0000 7f8136d1ef40 1 -- 172.21.15.40:0/4161008379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e82672070 msgr2=0x556e8270a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.763+0000 7f8136d1ef40 1 --2- 172.21.15.40:0/4161008379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e82672070 0x556e8270a9f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f812c00de20 tx=0x7f812c00b600 comp rx=0 tx=0).stop 2024-04-26T23:06:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.763+0000 7f8133e00640 1 -- 172.21.15.40:0/4161008379 reap_dead start 2024-04-26T23:06:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.763+0000 7f8136d1ef40 1 -- 172.21.15.40:0/4161008379 shutdown_connections 2024-04-26T23:06:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.763+0000 7f8136d1ef40 1 -- 172.21.15.40:0/4161008379 >> 172.21.15.40:0/4161008379 conn(0x556e8268cdf0 msgr2=0x556e8268b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.763+0000 7f8136d1ef40 1 -- 172.21.15.40:0/4161008379 shutdown_connections 2024-04-26T23:06:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.764+0000 7f8136d1ef40 1 -- 172.21.15.40:0/4161008379 wait complete. 2024-04-26T23:06:02.771 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000003.00000000 parent 2024-04-26T23:06:02.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.883+0000 7ff9e3b1df40 1 Processor -- start 2024-04-26T23:06:02.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.883+0000 7ff9e3b1df40 1 -- start start 2024-04-26T23:06:02.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9e3b1df40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dc2b749c00 0x55dc2b81a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9e3b1df40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dc2b77a0a0 0x55dc2b778b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9e3b1df40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55dc2b77a480 0x55dc2b779040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9e3b1df40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dc2b8271e0 con 0x55dc2b77a0a0 2024-04-26T23:06:02.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9e3b1df40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dc2b6f1d20 con 0x55dc2b749c00 2024-04-26T23:06:02.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9e3b1df40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55dc2b793a10 con 0x55dc2b77a480 2024-04-26T23:06:02.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9d7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dc2b77a0a0 0x55dc2b778b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9d7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dc2b77a0a0 0x55dc2b778b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33576/0 (socket says 172.21.15.40:33576) 2024-04-26T23:06:02.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9d7e00640 1 -- 172.21.15.40:0/3994805724 learned_addr learned my addr 172.21.15.40:0/3994805724 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:02.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9e0c00640 1 --2- 172.21.15.40:0/3994805724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55dc2b77a480 0x55dc2b779040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9d7e00640 1 -- 172.21.15.40:0/3994805724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55dc2b77a480 msgr2=0x55dc2b779040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:02.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9e0200640 1 --2- 172.21.15.40:0/3994805724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dc2b749c00 0x55dc2b81a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9d7e00640 1 --2- 172.21.15.40:0/3994805724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55dc2b77a480 0x55dc2b779040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9d7e00640 1 -- 172.21.15.40:0/3994805724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dc2b749c00 msgr2=0x55dc2b81a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:02.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9d7e00640 1 --2- 172.21.15.40:0/3994805724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dc2b749c00 0x55dc2b81a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9d7e00640 1 -- 172.21.15.40:0/3994805724 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc2b74c430 con 0x55dc2b77a0a0 2024-04-26T23:06:02.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.884+0000 7ff9e0c00640 1 --2- 172.21.15.40:0/3994805724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55dc2b77a480 0x55dc2b779040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:02.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.885+0000 7ff9d7e00640 1 --2- 172.21.15.40:0/3994805724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dc2b77a0a0 0x55dc2b778b00 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8001d30 tx=0x7ff9c800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b62624ca2db18e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:02.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.885+0000 7ff9dee00640 1 -- 172.21.15.40:0/3994805724 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c800bd80 con 0x55dc2b77a0a0 2024-04-26T23:06:02.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.885+0000 7ff9dee00640 1 -- 172.21.15.40:0/3994805724 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c800f400 con 0x55dc2b77a0a0 2024-04-26T23:06:02.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.885+0000 7ff9dee00640 1 -- 172.21.15.40:0/3994805724 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c800f830 con 0x55dc2b77a0a0 2024-04-26T23:06:02.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.886+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/3994805724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dc2b77a0a0 msgr2=0x55dc2b778b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:02.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.886+0000 7ff9e3b1df40 1 --2- 172.21.15.40:0/3994805724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dc2b77a0a0 0x55dc2b778b00 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8001d30 tx=0x7ff9c800b5b0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.886+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/3994805724 shutdown_connections 2024-04-26T23:06:02.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.886+0000 7ff9e3b1df40 1 --2- 172.21.15.40:0/3994805724 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55dc2b77a480 0x55dc2b779040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.886+0000 7ff9e3b1df40 1 --2- 172.21.15.40:0/3994805724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dc2b77a0a0 0x55dc2b778b00 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.886+0000 7ff9e3b1df40 1 --2- 172.21.15.40:0/3994805724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dc2b749c00 0x55dc2b81a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.886+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/3994805724 >> 172.21.15.40:0/3994805724 conn(0x55dc2b777c00 msgr2=0x55dc2b8273c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:02.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.886+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/3994805724 shutdown_connections 2024-04-26T23:06:02.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.886+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/3994805724 wait complete. 2024-04-26T23:06:02.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.887+0000 7ff9e3b1df40 1 Processor -- start 2024-04-26T23:06:02.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.887+0000 7ff9e3b1df40 1 -- start start 2024-04-26T23:06:02.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.888+0000 7ff9e3b1df40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dc2b749c00 0x55dc2b78b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.888+0000 7ff9e3b1df40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dc2b77a0a0 0x55dc2b78c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.888+0000 7ff9e3b1df40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55dc2b77a480 0x55dc2b80c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.888+0000 7ff9e3b1df40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dc2b74b9e0 con 0x55dc2b749c00 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.888+0000 7ff9e3b1df40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dc2b749210 con 0x55dc2b77a0a0 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.888+0000 7ff9e3b1df40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55dc2b777240 con 0x55dc2b77a480 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.888+0000 7ff9e0200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dc2b749c00 0x55dc2b78b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.888+0000 7ff9e0200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dc2b749c00 0x55dc2b78b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33590/0 (socket says 172.21.15.40:33590) 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.888+0000 7ff9e0200640 1 -- 172.21.15.40:0/2410161951 learned_addr learned my addr 172.21.15.40:0/2410161951 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.888+0000 7ff9d7e00640 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dc2b77a0a0 0x55dc2b78c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.888+0000 7ff9e0c00640 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55dc2b77a480 0x55dc2b80c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.889+0000 7ff9e0200640 1 -- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55dc2b77a480 msgr2=0x55dc2b80c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.889+0000 7ff9e0200640 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55dc2b77a480 0x55dc2b80c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.889+0000 7ff9e0200640 1 -- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dc2b77a0a0 msgr2=0x55dc2b78c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.889+0000 7ff9e0200640 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dc2b77a0a0 0x55dc2b78c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.889+0000 7ff9e0200640 1 -- 172.21.15.40:0/2410161951 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc2b78bbd0 con 0x55dc2b749c00 2024-04-26T23:06:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.889+0000 7ff9e0200640 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dc2b749c00 0x55dc2b78b690 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000beb0 tx=0x7ff9d00073f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.889+0000 7ff9dda00640 1 -- 172.21.15.40:0/2410161951 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0019070 con 0x55dc2b749c00 2024-04-26T23:06:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.889+0000 7ff9dda00640 1 -- 172.21.15.40:0/2410161951 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9d0007bc0 con 0x55dc2b749c00 2024-04-26T23:06:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.889+0000 7ff9dda00640 1 -- 172.21.15.40:0/2410161951 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0014410 con 0x55dc2b749c00 2024-04-26T23:06:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.889+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/2410161951 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55dc2b824360 con 0x55dc2b749c00 2024-04-26T23:06:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.890+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/2410161951 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc2b824920 con 0x55dc2b749c00 2024-04-26T23:06:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.892+0000 7ff9dda00640 1 -- 172.21.15.40:0/2410161951 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff9d001c070 con 0x55dc2b749c00 2024-04-26T23:06:02.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.893+0000 7ff9dda00640 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff9bc06b2d0 0x7ff9bc06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.893+0000 7ff9dda00640 1 -- 172.21.15.40:0/2410161951 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7ff9d0089b30 con 0x55dc2b749c00 2024-04-26T23:06:02.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.893+0000 7ff9e3b1df40 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x55dc2b78cd60 0x55dc2b825120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:02.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.893+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/2410161951 --> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] -- osd_op(unknown.0.0:1 9.12 9:4def72d4:::10000000003.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67) -- 0x55dc2b8bc290 con 0x55dc2b78cd60 2024-04-26T23:06:02.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.893+0000 7ff9d7e00640 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff9bc06b2d0 0x7ff9bc06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.894+0000 7ff9e0c00640 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x55dc2b78cd60 0x55dc2b825120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:02.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.895+0000 7ff9d7e00640 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff9bc06b2d0 0x7ff9bc06d790 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8001d30 tx=0x7ff9c8009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:02.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.895+0000 7ff9e0c00640 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x55dc2b78cd60 0x55dc2b825120 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:02.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.898+0000 7ff9e0c00640 1 -- 172.21.15.40:0/2410161951 <== osd.5 v2:172.21.15.103:6800/157083760 1 ==== osd_op_reply(1 10000000003.00000000 [rmxattr] v67'9 uv9 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff9d80518f0 con 0x55dc2b78cd60 2024-04-26T23:06:02.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.898+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x55dc2b78cd60 msgr2=0x55dc2b825120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.898+0000 7ff9e3b1df40 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x55dc2b78cd60 0x55dc2b825120 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.898+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff9bc06b2d0 msgr2=0x7ff9bc06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.898+0000 7ff9e3b1df40 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff9bc06b2d0 0x7ff9bc06d790 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8001d30 tx=0x7ff9c8009000 comp rx=0 tx=0).stop 2024-04-26T23:06:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.898+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/2410161951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dc2b749c00 msgr2=0x55dc2b78b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.898+0000 7ff9e3b1df40 1 --2- 172.21.15.40:0/2410161951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dc2b749c00 0x55dc2b78b690 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000beb0 tx=0x7ff9d00073f0 comp rx=0 tx=0).stop 2024-04-26T23:06:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.898+0000 7ff9e0c00640 1 -- 172.21.15.40:0/2410161951 reap_dead start 2024-04-26T23:06:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.898+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/2410161951 shutdown_connections 2024-04-26T23:06:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.898+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/2410161951 >> 172.21.15.40:0/2410161951 conn(0x55dc2b777c00 msgr2=0x55dc2b8273c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.898+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/2410161951 shutdown_connections 2024-04-26T23:06:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:02.898+0000 7ff9e3b1df40 1 -- 172.21.15.40:0/2410161951 wait complete. 2024-04-26T23:06:02.904 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000004.00000000 parent 2024-04-26T23:06:03.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.010+0000 7f1920b78f40 1 Processor -- start 2024-04-26T23:06:03.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.010+0000 7f1920b78f40 1 -- start start 2024-04-26T23:06:03.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.011+0000 7f1920b78f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558344f0be90 0x558344fad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.011+0000 7f1920b78f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558344f0d070 0x558344f08700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.011+0000 7f1920b78f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558344fabc90 0x558344f08c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.011+0000 7f1920b78f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558344fb2310 con 0x558344fabc90 2024-04-26T23:06:03.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.011+0000 7f1920b78f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558344e81d20 con 0x558344f0be90 2024-04-26T23:06:03.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.011+0000 7f1920b78f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558344fb25f0 con 0x558344f0d070 2024-04-26T23:06:03.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.011+0000 7f191e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558344fabc90 0x558344f08c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.012+0000 7f191e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558344fabc90 0x558344f08c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33610/0 (socket says 172.21.15.40:33610) 2024-04-26T23:06:03.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.012+0000 7f191e800640 1 -- 172.21.15.40:0/4046995116 learned_addr learned my addr 172.21.15.40:0/4046995116 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.012+0000 7f191d400640 1 --2- 172.21.15.40:0/4046995116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558344f0d070 0x558344f08700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.012+0000 7f191e800640 1 -- 172.21.15.40:0/4046995116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558344f0d070 msgr2=0x558344f08700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.012+0000 7f191e800640 1 --2- 172.21.15.40:0/4046995116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558344f0d070 0x558344f08700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.012+0000 7f191e800640 1 -- 172.21.15.40:0/4046995116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558344f0be90 msgr2=0x558344fad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.012+0000 7f191e800640 1 --2- 172.21.15.40:0/4046995116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558344f0be90 0x558344fad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.012+0000 7f191e800640 1 -- 172.21.15.40:0/4046995116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558344fb32f0 con 0x558344fabc90 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.012+0000 7f191e800640 1 --2- 172.21.15.40:0/4046995116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558344fabc90 0x558344f08c40 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f191800c480 tx=0x7f191800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc7717e0abb9ecd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.013+0000 7f1917e00640 1 -- 172.21.15.40:0/4046995116 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19180116b0 con 0x558344fabc90 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.013+0000 7f1917e00640 1 -- 172.21.15.40:0/4046995116 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1918017070 con 0x558344fabc90 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.013+0000 7f1917e00640 1 -- 172.21.15.40:0/4046995116 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1918011ab0 con 0x558344fabc90 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.013+0000 7f1920b78f40 1 -- 172.21.15.40:0/4046995116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558344fabc90 msgr2=0x558344f08c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.013+0000 7f1920b78f40 1 --2- 172.21.15.40:0/4046995116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558344fabc90 0x558344f08c40 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f191800c480 tx=0x7f191800c940 comp rx=0 tx=0).stop 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.013+0000 7f1920b78f40 1 -- 172.21.15.40:0/4046995116 shutdown_connections 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.013+0000 7f1920b78f40 1 --2- 172.21.15.40:0/4046995116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558344fabc90 0x558344f08c40 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.013+0000 7f1920b78f40 1 --2- 172.21.15.40:0/4046995116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558344f0d070 0x558344f08700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.013+0000 7f1920b78f40 1 --2- 172.21.15.40:0/4046995116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558344f0be90 0x558344fad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.013+0000 7f1920b78f40 1 -- 172.21.15.40:0/4046995116 >> 172.21.15.40:0/4046995116 conn(0x558344f27df0 msgr2=0x558344f0c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.013+0000 7f1920b78f40 1 -- 172.21.15.40:0/4046995116 shutdown_connections 2024-04-26T23:06:03.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.013+0000 7f1920b78f40 1 -- 172.21.15.40:0/4046995116 wait complete. 2024-04-26T23:06:03.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.014+0000 7f1920b78f40 1 Processor -- start 2024-04-26T23:06:03.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.014+0000 7f1920b78f40 1 -- start start 2024-04-26T23:06:03.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.015+0000 7f1920b78f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558344f0be90 0x558344f9e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.015+0000 7f1920b78f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558344f0d070 0x558344f0b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.015+0000 7f1920b78f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558344fabc90 0x558344f0b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.015+0000 7f1920b78f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558344fb38b0 con 0x558344f0d070 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.015+0000 7f1920b78f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558344fb35d0 con 0x558344fabc90 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.015+0000 7f1920b78f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558344fab400 con 0x558344f0be90 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.015+0000 7f191d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558344f0d070 0x558344f0b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.016+0000 7f191d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558344f0d070 0x558344f0b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33626/0 (socket says 172.21.15.40:33626) 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.016+0000 7f191d400640 1 -- 172.21.15.40:0/505189788 learned_addr learned my addr 172.21.15.40:0/505189788 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.016+0000 7f191e800640 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558344fabc90 0x558344f0b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.016+0000 7f191de00640 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558344f0be90 0x558344f9e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.016+0000 7f191d400640 1 -- 172.21.15.40:0/505189788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558344f0be90 msgr2=0x558344f9e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.016+0000 7f191d400640 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558344f0be90 0x558344f9e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.016+0000 7f191d400640 1 -- 172.21.15.40:0/505189788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558344fabc90 msgr2=0x558344f0b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.016+0000 7f191d400640 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558344fabc90 0x558344f0b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.016+0000 7f191d400640 1 -- 172.21.15.40:0/505189788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558344f0ada0 con 0x558344f0d070 2024-04-26T23:06:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.016+0000 7f191e800640 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558344fabc90 0x558344f0b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.016+0000 7f191de00640 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558344f0be90 0x558344f9e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.016+0000 7f191d400640 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558344f0d070 0x558344f0b030 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f1910000c00 tx=0x7f1910002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.017+0000 7f1916a00640 1 -- 172.21.15.40:0/505189788 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910063070 con 0x558344f0d070 2024-04-26T23:06:03.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.017+0000 7f1916a00640 1 -- 172.21.15.40:0/505189788 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1910054b70 con 0x558344f0d070 2024-04-26T23:06:03.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.017+0000 7f1920b78f40 1 -- 172.21.15.40:0/505189788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558344fb60f0 con 0x558344f0d070 2024-04-26T23:06:03.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.017+0000 7f1916a00640 1 -- 172.21.15.40:0/505189788 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910068460 con 0x558344f0d070 2024-04-26T23:06:03.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.017+0000 7f1920b78f40 1 -- 172.21.15.40:0/505189788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558344fa1df0 con 0x558344f0d070 2024-04-26T23:06:03.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.019+0000 7f1916a00640 1 -- 172.21.15.40:0/505189788 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f191005c040 con 0x558344f0d070 2024-04-26T23:06:03.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.020+0000 7f1916a00640 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f18fc06b320 0x7f18fc06d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.020+0000 7f1916a00640 1 -- 172.21.15.40:0/505189788 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f19100cefa0 con 0x558344f0d070 2024-04-26T23:06:03.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.020+0000 7f1920b78f40 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x558344f0c270 0x55834504c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.020+0000 7f1920b78f40 1 -- 172.21.15.40:0/505189788 --> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] -- osd_op(unknown.0.0:1 9.5 9:a367a3e8:::10000000004.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67) -- 0x558344fa36a0 con 0x558344f0c270 2024-04-26T23:06:03.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.021+0000 7f191de00640 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f18fc06b320 0x7f18fc06d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.021+0000 7f191e800640 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x558344f0c270 0x55834504c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.021+0000 7f191e800640 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x558344f0c270 0x55834504c930 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.022+0000 7f191de00640 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f18fc06b320 0x7f18fc06d7e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x558344f240d0 tx=0x7f1908005570 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.024+0000 7f191e800640 1 -- 172.21.15.40:0/505189788 <== osd.0 v2:172.21.15.40:6800/2897145808 1 ==== osd_op_reply(1 10000000004.00000000 [rmxattr] v67'6 uv6 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f19180118f0 con 0x558344f0c270 2024-04-26T23:06:03.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.024+0000 7f1920b78f40 1 -- 172.21.15.40:0/505189788 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x558344f0c270 msgr2=0x55834504c930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.024+0000 7f1920b78f40 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x558344f0c270 0x55834504c930 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.024+0000 7f1920b78f40 1 -- 172.21.15.40:0/505189788 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f18fc06b320 msgr2=0x7f18fc06d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.024+0000 7f1920b78f40 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f18fc06b320 0x7f18fc06d7e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x558344f240d0 tx=0x7f1908005570 comp rx=0 tx=0).stop 2024-04-26T23:06:03.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.024+0000 7f1920b78f40 1 -- 172.21.15.40:0/505189788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558344f0d070 msgr2=0x558344f0b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.024+0000 7f1920b78f40 1 --2- 172.21.15.40:0/505189788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558344f0d070 0x558344f0b030 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f1910000c00 tx=0x7f1910002fb0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.025+0000 7f191e800640 1 -- 172.21.15.40:0/505189788 reap_dead start 2024-04-26T23:06:03.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.025+0000 7f1920b78f40 1 -- 172.21.15.40:0/505189788 shutdown_connections 2024-04-26T23:06:03.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.025+0000 7f1920b78f40 1 -- 172.21.15.40:0/505189788 >> 172.21.15.40:0/505189788 conn(0x558344f27df0 msgr2=0x558344fa84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.025+0000 7f1920b78f40 1 -- 172.21.15.40:0/505189788 shutdown_connections 2024-04-26T23:06:03.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.025+0000 7f1920b78f40 1 -- 172.21.15.40:0/505189788 wait complete. 2024-04-26T23:06:03.030 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000005.00000000 parent 2024-04-26T23:06:03.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.135+0000 7f4bf371df40 1 Processor -- start 2024-04-26T23:06:03.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.135+0000 7f4bf371df40 1 -- start start 2024-04-26T23:06:03.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.136+0000 7f4bf371df40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d05f986fe0 0x55d05f9900e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.136+0000 7f4bf371df40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d05f9873c0 0x55d05f8be6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.136+0000 7f4bf371df40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d05f997a40 0x55d05f8bebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.136+0000 7f4bf371df40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d05f996030 con 0x55d05f997a40 2024-04-26T23:06:03.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.136+0000 7f4bf371df40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d05f865d20 con 0x55d05f986fe0 2024-04-26T23:06:03.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.136+0000 7f4bf371df40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d05f996310 con 0x55d05f9873c0 2024-04-26T23:06:03.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.137+0000 7f4bf0800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d05f997a40 0x55d05f8bebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.137+0000 7f4bf0800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d05f997a40 0x55d05f8bebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33644/0 (socket says 172.21.15.40:33644) 2024-04-26T23:06:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.137+0000 7f4bf0800640 1 -- 172.21.15.40:0/378061213 learned_addr learned my addr 172.21.15.40:0/378061213 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.137+0000 7f4bef400640 1 --2- 172.21.15.40:0/378061213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d05f9873c0 0x55d05f8be6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.137+0000 7f4befe00640 1 --2- 172.21.15.40:0/378061213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d05f986fe0 0x55d05f9900e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.137+0000 7f4bf0800640 1 -- 172.21.15.40:0/378061213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d05f9873c0 msgr2=0x55d05f8be6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.137+0000 7f4bf0800640 1 --2- 172.21.15.40:0/378061213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d05f9873c0 0x55d05f8be6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.137+0000 7f4bf0800640 1 -- 172.21.15.40:0/378061213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d05f986fe0 msgr2=0x55d05f9900e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.137+0000 7f4bf0800640 1 --2- 172.21.15.40:0/378061213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d05f986fe0 0x55d05f9900e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.137+0000 7f4bf0800640 1 -- 172.21.15.40:0/378061213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d05f994050 con 0x55d05f997a40 2024-04-26T23:06:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.137+0000 7f4befe00640 1 --2- 172.21.15.40:0/378061213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d05f986fe0 0x55d05f9900e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.137+0000 7f4bf0800640 1 --2- 172.21.15.40:0/378061213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d05f997a40 0x55d05f8bebe0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f4be8003480 tx=0x7f4be8006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ac159134ca3360 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.138+0000 7f4bee000640 1 -- 172.21.15.40:0/378061213 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be80019b0 con 0x55d05f997a40 2024-04-26T23:06:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.138+0000 7f4bee000640 1 -- 172.21.15.40:0/378061213 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be8006070 con 0x55d05f997a40 2024-04-26T23:06:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.138+0000 7f4bee000640 1 -- 172.21.15.40:0/378061213 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be804d950 con 0x55d05f997a40 2024-04-26T23:06:03.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.138+0000 7f4bf371df40 1 -- 172.21.15.40:0/378061213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d05f997a40 msgr2=0x55d05f8bebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.138+0000 7f4bf371df40 1 --2- 172.21.15.40:0/378061213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d05f997a40 0x55d05f8bebe0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f4be8003480 tx=0x7f4be8006000 comp rx=0 tx=0).stop 2024-04-26T23:06:03.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.138+0000 7f4bf371df40 1 -- 172.21.15.40:0/378061213 shutdown_connections 2024-04-26T23:06:03.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.138+0000 7f4bf371df40 1 --2- 172.21.15.40:0/378061213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d05f997a40 0x55d05f8bebe0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.138+0000 7f4bf371df40 1 --2- 172.21.15.40:0/378061213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d05f9873c0 0x55d05f8be6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.138+0000 7f4bf371df40 1 --2- 172.21.15.40:0/378061213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d05f986fe0 0x55d05f9900e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.138+0000 7f4bf371df40 1 -- 172.21.15.40:0/378061213 >> 172.21.15.40:0/378061213 conn(0x55d05f90bdf0 msgr2=0x55d05f90c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.139+0000 7f4bf371df40 1 -- 172.21.15.40:0/378061213 shutdown_connections 2024-04-26T23:06:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.139+0000 7f4bf371df40 1 -- 172.21.15.40:0/378061213 wait complete. 2024-04-26T23:06:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.139+0000 7f4bf371df40 1 Processor -- start 2024-04-26T23:06:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.139+0000 7f4bf371df40 1 -- start start 2024-04-26T23:06:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.140+0000 7f4bf371df40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d05f986fe0 0x55d05f976ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.140+0000 7f4bf371df40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d05f9873c0 0x55d05f98e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.140+0000 7f4bf371df40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d05f997a40 0x55d05f98edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.140+0000 7f4bf371df40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d05f998990 con 0x55d05f9873c0 2024-04-26T23:06:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.140+0000 7f4bf371df40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d05f998be0 con 0x55d05f986fe0 2024-04-26T23:06:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.140+0000 7f4bf371df40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d05f993d70 con 0x55d05f997a40 2024-04-26T23:06:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.141+0000 7f4bef400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d05f9873c0 0x55d05f98e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.141+0000 7f4bef400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d05f9873c0 0x55d05f98e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33646/0 (socket says 172.21.15.40:33646) 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.141+0000 7f4bef400640 1 -- 172.21.15.40:0/1890394371 learned_addr learned my addr 172.21.15.40:0/1890394371 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.141+0000 7f4befe00640 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d05f986fe0 0x55d05f976ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.141+0000 7f4bf0800640 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d05f997a40 0x55d05f98edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.141+0000 7f4bef400640 1 -- 172.21.15.40:0/1890394371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d05f997a40 msgr2=0x55d05f98edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.141+0000 7f4bef400640 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d05f997a40 0x55d05f98edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.141+0000 7f4bef400640 1 -- 172.21.15.40:0/1890394371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d05f986fe0 msgr2=0x55d05f976ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.141+0000 7f4bef400640 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d05f986fe0 0x55d05f976ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.141+0000 7f4bef400640 1 -- 172.21.15.40:0/1890394371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d05f990ca0 con 0x55d05f9873c0 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.141+0000 7f4bf0800640 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d05f997a40 0x55d05f98edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.142+0000 7f4bef400640 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d05f9873c0 0x55d05f98e880 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc009e60 tx=0x7f4bdc009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.142+0000 7f4becc00640 1 -- 172.21.15.40:0/1890394371 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc0153f0 con 0x55d05f9873c0 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.142+0000 7f4becc00640 1 -- 172.21.15.40:0/1890394371 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bdc015590 con 0x55d05f9873c0 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.142+0000 7f4becc00640 1 -- 172.21.15.40:0/1890394371 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc015940 con 0x55d05f9873c0 2024-04-26T23:06:03.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.142+0000 7f4bf371df40 1 -- 172.21.15.40:0/1890394371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d05f979920 con 0x55d05f9873c0 2024-04-26T23:06:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.142+0000 7f4bf371df40 1 -- 172.21.15.40:0/1890394371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d05f979ee0 con 0x55d05f9873c0 2024-04-26T23:06:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.144+0000 7f4becc00640 1 -- 172.21.15.40:0/1890394371 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4bdc01e440 con 0x55d05f9873c0 2024-04-26T23:06:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.145+0000 7f4becc00640 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bd006b2d0 0x7f4bd006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.145+0000 7f4becc00640 1 -- 172.21.15.40:0/1890394371 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f4bdc08a2b0 con 0x55d05f9873c0 2024-04-26T23:06:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.145+0000 7f4bf371df40 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55d05f990f30 0x55d05f991340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.146+0000 7f4bf371df40 1 -- 172.21.15.40:0/1890394371 --> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] -- osd_op(unknown.0.0:1 9.9 9:9258b67d:::10000000005.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67) -- 0x55d05f991880 con 0x55d05f990f30 2024-04-26T23:06:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.146+0000 7f4befe00640 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bd006b2d0 0x7f4bd006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.146+0000 7f4bf0800640 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55d05f990f30 0x55d05f991340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.147+0000 7f4bf0800640 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55d05f990f30 0x55d05f991340 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.147+0000 7f4befe00640 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bd006b2d0 0x7f4bd006d790 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4be00096a0 tx=0x7f4be000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.148+0000 7f4bf0800640 1 -- 172.21.15.40:0/1890394371 <== osd.0 v2:172.21.15.40:6800/2897145808 1 ==== osd_op_reply(1 10000000005.00000000 [rmxattr] v67'6 uv6 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4be8006220 con 0x55d05f990f30 2024-04-26T23:06:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.148+0000 7f4bf371df40 1 -- 172.21.15.40:0/1890394371 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55d05f990f30 msgr2=0x55d05f991340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.148+0000 7f4bf371df40 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55d05f990f30 0x55d05f991340 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.148+0000 7f4bf371df40 1 -- 172.21.15.40:0/1890394371 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bd006b2d0 msgr2=0x7f4bd006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.148+0000 7f4bf371df40 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bd006b2d0 0x7f4bd006d790 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4be00096a0 tx=0x7f4be000b040 comp rx=0 tx=0).stop 2024-04-26T23:06:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.148+0000 7f4bf371df40 1 -- 172.21.15.40:0/1890394371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d05f9873c0 msgr2=0x55d05f98e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.148+0000 7f4bf371df40 1 --2- 172.21.15.40:0/1890394371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d05f9873c0 0x55d05f98e880 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc009e60 tx=0x7f4bdc009f50 comp rx=0 tx=0).stop 2024-04-26T23:06:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.148+0000 7f4bf0800640 1 -- 172.21.15.40:0/1890394371 reap_dead start 2024-04-26T23:06:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.148+0000 7f4bf371df40 1 -- 172.21.15.40:0/1890394371 shutdown_connections 2024-04-26T23:06:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.148+0000 7f4bf371df40 1 -- 172.21.15.40:0/1890394371 >> 172.21.15.40:0/1890394371 conn(0x55d05f90bdf0 msgr2=0x55d05f90c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.149+0000 7f4bf371df40 1 -- 172.21.15.40:0/1890394371 shutdown_connections 2024-04-26T23:06:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.149+0000 7f4bf371df40 1 -- 172.21.15.40:0/1890394371 wait complete. 2024-04-26T23:06:03.153 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000006.00000000 parent 2024-04-26T23:06:03.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.239+0000 7fc84a41bf40 1 Processor -- start 2024-04-26T23:06:03.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.240+0000 7fc84a41bf40 1 -- start start 2024-04-26T23:06:03.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.240+0000 7fc84a41bf40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5650b2c25e90 0x5650b2c40810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.240+0000 7fc84a41bf40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5650b2c27070 0x5650b2ccd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.240+0000 7fc84a41bf40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650b2ccd600 0x5650b2cca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.240+0000 7fc84a41bf40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5650b2b9bd20 con 0x5650b2ccd600 2024-04-26T23:06:03.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.240+0000 7fc84a41bf40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5650b2b9fd60 con 0x5650b2c25e90 2024-04-26T23:06:03.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.240+0000 7fc84a41bf40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5650b2ccc5f0 con 0x5650b2c27070 2024-04-26T23:06:03.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.241+0000 7fc848000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650b2ccd600 0x5650b2cca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.241+0000 7fc848000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650b2ccd600 0x5650b2cca5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33654/0 (socket says 172.21.15.40:33654) 2024-04-26T23:06:03.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.241+0000 7fc848000640 1 -- 172.21.15.40:0/3702579822 learned_addr learned my addr 172.21.15.40:0/3702579822 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.241+0000 7fc847600640 1 --2- 172.21.15.40:0/3702579822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5650b2c25e90 0x5650b2c40810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.241+0000 7fc848000640 1 -- 172.21.15.40:0/3702579822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5650b2c27070 msgr2=0x5650b2ccd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.241+0000 7fc848000640 1 --2- 172.21.15.40:0/3702579822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5650b2c27070 0x5650b2ccd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.241+0000 7fc848000640 1 -- 172.21.15.40:0/3702579822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5650b2c25e90 msgr2=0x5650b2c40810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.241+0000 7fc848000640 1 --2- 172.21.15.40:0/3702579822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5650b2c25e90 0x5650b2c40810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.241+0000 7fc848000640 1 -- 172.21.15.40:0/3702579822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650b2cc58b0 con 0x5650b2ccd600 2024-04-26T23:06:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.242+0000 7fc848000640 1 --2- 172.21.15.40:0/3702579822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650b2ccd600 0x5650b2cca5c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fc840002660 tx=0x7fc840002b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=278d887d5bc21700 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.242+0000 7fc845800640 1 -- 172.21.15.40:0/3702579822 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840016070 con 0x5650b2ccd600 2024-04-26T23:06:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.242+0000 7fc845800640 1 -- 172.21.15.40:0/3702579822 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc84000f730 con 0x5650b2ccd600 2024-04-26T23:06:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.242+0000 7fc845800640 1 -- 172.21.15.40:0/3702579822 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84000fb20 con 0x5650b2ccd600 2024-04-26T23:06:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.242+0000 7fc84a41bf40 1 -- 172.21.15.40:0/3702579822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650b2ccd600 msgr2=0x5650b2cca5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.242+0000 7fc84a41bf40 1 --2- 172.21.15.40:0/3702579822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650b2ccd600 0x5650b2cca5c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fc840002660 tx=0x7fc840002b20 comp rx=0 tx=0).stop 2024-04-26T23:06:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.242+0000 7fc84a41bf40 1 -- 172.21.15.40:0/3702579822 shutdown_connections 2024-04-26T23:06:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.242+0000 7fc84a41bf40 1 --2- 172.21.15.40:0/3702579822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650b2ccd600 0x5650b2cca5c0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.242+0000 7fc84a41bf40 1 --2- 172.21.15.40:0/3702579822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5650b2c27070 0x5650b2ccd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.242+0000 7fc84a41bf40 1 --2- 172.21.15.40:0/3702579822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5650b2c25e90 0x5650b2c40810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.242+0000 7fc84a41bf40 1 -- 172.21.15.40:0/3702579822 >> 172.21.15.40:0/3702579822 conn(0x5650b2c41df0 msgr2=0x5650b2c26270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.243+0000 7fc84a41bf40 1 -- 172.21.15.40:0/3702579822 shutdown_connections 2024-04-26T23:06:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.243+0000 7fc84a41bf40 1 -- 172.21.15.40:0/3702579822 wait complete. 2024-04-26T23:06:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.243+0000 7fc84a41bf40 1 Processor -- start 2024-04-26T23:06:03.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.244+0000 7fc84a41bf40 1 -- start start 2024-04-26T23:06:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.244+0000 7fc84a41bf40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650b2c25e90 0x5650b2cca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.244+0000 7fc84a41bf40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5650b2c27070 0x5650b2cc9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.245+0000 7fc84a41bf40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5650b2ccd600 0x5650b2d566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.245+0000 7fc84a41bf40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5650b2c421d0 con 0x5650b2c25e90 2024-04-26T23:06:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.245+0000 7fc84a41bf40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5650b2cc5bb0 con 0x5650b2c27070 2024-04-26T23:06:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.245+0000 7fc84a41bf40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5650b2bf49a0 con 0x5650b2ccd600 2024-04-26T23:06:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.245+0000 7fc847600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650b2c25e90 0x5650b2cca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.245+0000 7fc847600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650b2c25e90 0x5650b2cca770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33662/0 (socket says 172.21.15.40:33662) 2024-04-26T23:06:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.245+0000 7fc847600640 1 -- 172.21.15.40:0/1002186692 learned_addr learned my addr 172.21.15.40:0/1002186692 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.245+0000 7fc847600640 1 -- 172.21.15.40:0/1002186692 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5650b2ccd600 msgr2=0x5650b2d566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.245+0000 7fc847600640 1 --2- 172.21.15.40:0/1002186692 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5650b2ccd600 0x5650b2d566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.245+0000 7fc847600640 1 -- 172.21.15.40:0/1002186692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5650b2c27070 msgr2=0x5650b2cc9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.245+0000 7fc847600640 1 --2- 172.21.15.40:0/1002186692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5650b2c27070 0x5650b2cc9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.245+0000 7fc847600640 1 -- 172.21.15.40:0/1002186692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650b2c257a0 con 0x5650b2c25e90 2024-04-26T23:06:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.246+0000 7fc847600640 1 --2- 172.21.15.40:0/1002186692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650b2c25e90 0x5650b2cca770 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fc83800df80 tx=0x7fc83800b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.246+0000 7fc83fe00640 1 -- 172.21.15.40:0/1002186692 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc838014020 con 0x5650b2c25e90 2024-04-26T23:06:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.246+0000 7fc84a41bf40 1 -- 172.21.15.40:0/1002186692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5650b2c25a80 con 0x5650b2c25e90 2024-04-26T23:06:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.246+0000 7fc83fe00640 1 -- 172.21.15.40:0/1002186692 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc838010620 con 0x5650b2c25e90 2024-04-26T23:06:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.246+0000 7fc83fe00640 1 -- 172.21.15.40:0/1002186692 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc838008030 con 0x5650b2c25e90 2024-04-26T23:06:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.246+0000 7fc84a41bf40 1 -- 172.21.15.40:0/1002186692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650b2cc50c0 con 0x5650b2c25e90 2024-04-26T23:06:03.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.248+0000 7fc83fe00640 1 -- 172.21.15.40:0/1002186692 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc838017020 con 0x5650b2c25e90 2024-04-26T23:06:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.249+0000 7fc83fe00640 1 --2- 172.21.15.40:0/1002186692 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc82406b300 0x7fc82406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.249+0000 7fc83fe00640 1 -- 172.21.15.40:0/1002186692 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7fc83808abc0 con 0x5650b2c25e90 2024-04-26T23:06:03.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.250+0000 7fc84a41bf40 1 --2- 172.21.15.40:0/1002186692 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5650b2cc7ee0 0x5650b2d65be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.250+0000 7fc84a41bf40 1 -- 172.21.15.40:0/1002186692 --> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] -- osd_op(unknown.0.0:1 9.5 9:a5feb6f3:::10000000006.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67) -- 0x5650b2cc9560 con 0x5650b2cc7ee0 2024-04-26T23:06:03.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.250+0000 7fc846c00640 1 --2- 172.21.15.40:0/1002186692 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc82406b300 0x7fc82406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.250+0000 7fc848000640 1 --2- 172.21.15.40:0/1002186692 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5650b2cc7ee0 0x5650b2d65be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.251+0000 7fc848000640 1 --2- 172.21.15.40:0/1002186692 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5650b2cc7ee0 0x5650b2d65be0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.251+0000 7fc846c00640 1 --2- 172.21.15.40:0/1002186692 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc82406b300 0x7fc82406d7c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc830057000 tx=0x7fc83004f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.254+0000 7fc848000640 1 -- 172.21.15.40:0/1002186692 <== osd.0 v2:172.21.15.40:6800/2897145808 1 ==== osd_op_reply(1 10000000006.00000000 [rmxattr] v67'7 uv7 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc84000f070 con 0x5650b2cc7ee0 2024-04-26T23:06:03.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.254+0000 7fc84a41bf40 1 -- 172.21.15.40:0/1002186692 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5650b2cc7ee0 msgr2=0x5650b2d65be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.254+0000 7fc84a41bf40 1 --2- 172.21.15.40:0/1002186692 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x5650b2cc7ee0 0x5650b2d65be0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.254+0000 7fc84a41bf40 1 -- 172.21.15.40:0/1002186692 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc82406b300 msgr2=0x7fc82406d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.254+0000 7fc84a41bf40 1 --2- 172.21.15.40:0/1002186692 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc82406b300 0x7fc82406d7c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc830057000 tx=0x7fc83004f000 comp rx=0 tx=0).stop 2024-04-26T23:06:03.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.254+0000 7fc84a41bf40 1 -- 172.21.15.40:0/1002186692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650b2c25e90 msgr2=0x5650b2cca770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.254+0000 7fc84a41bf40 1 --2- 172.21.15.40:0/1002186692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5650b2c25e90 0x5650b2cca770 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fc83800df80 tx=0x7fc83800b6a0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.255+0000 7fc848000640 1 -- 172.21.15.40:0/1002186692 reap_dead start 2024-04-26T23:06:03.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.255+0000 7fc84a41bf40 1 -- 172.21.15.40:0/1002186692 shutdown_connections 2024-04-26T23:06:03.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.255+0000 7fc84a41bf40 1 -- 172.21.15.40:0/1002186692 >> 172.21.15.40:0/1002186692 conn(0x5650b2c41df0 msgr2=0x5650b2c26270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.255+0000 7fc84a41bf40 1 -- 172.21.15.40:0/1002186692 shutdown_connections 2024-04-26T23:06:03.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.255+0000 7fc84a41bf40 1 -- 172.21.15.40:0/1002186692 wait complete. 2024-04-26T23:06:03.260 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000007.00000000 parent 2024-04-26T23:06:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.346+0000 7fe84b35df40 1 Processor -- start 2024-04-26T23:06:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.346+0000 7fe84b35df40 1 -- start start 2024-04-26T23:06:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe84b35df40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a60ed68e90 0x55a60ed66440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe84b35df40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a60ed6a070 0x55a60ed67dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe84b35df40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a60ed83590 0x55a60ee11000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe84b35df40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a60ed81ef0 con 0x55a60ed68e90 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe84b35df40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a60ecded20 con 0x55a60ed83590 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe84b35df40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a60ee0f030 con 0x55a60ed6a070 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe848600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a60ed68e90 0x55a60ed66440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe848600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a60ed68e90 0x55a60ed66440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33694/0 (socket says 172.21.15.40:33694) 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe848600640 1 -- 172.21.15.40:0/1737658954 learned_addr learned my addr 172.21.15.40:0/1737658954 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe848600640 1 -- 172.21.15.40:0/1737658954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a60ed6a070 msgr2=0x55a60ed67dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe848600640 1 --2- 172.21.15.40:0/1737658954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a60ed6a070 0x55a60ed67dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe848600640 1 -- 172.21.15.40:0/1737658954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a60ed83590 msgr2=0x55a60ee11000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe848600640 1 --2- 172.21.15.40:0/1737658954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a60ed83590 0x55a60ee11000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.347+0000 7fe848600640 1 -- 172.21.15.40:0/1737658954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a60ee084d0 con 0x55a60ed68e90 2024-04-26T23:06:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.348+0000 7fe848600640 1 --2- 172.21.15.40:0/1737658954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a60ed68e90 0x55a60ed66440 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fe83800a520 tx=0x7fe83804e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64e9330480ad3a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.348+0000 7fe847200640 1 -- 172.21.15.40:0/1737658954 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838065070 con 0x55a60ed68e90 2024-04-26T23:06:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.348+0000 7fe847200640 1 -- 172.21.15.40:0/1737658954 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8380549f0 con 0x55a60ed68e90 2024-04-26T23:06:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.348+0000 7fe847200640 1 -- 172.21.15.40:0/1737658954 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838059040 con 0x55a60ed68e90 2024-04-26T23:06:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.349+0000 7fe84b35df40 1 -- 172.21.15.40:0/1737658954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a60ed68e90 msgr2=0x55a60ed66440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.349+0000 7fe84b35df40 1 --2- 172.21.15.40:0/1737658954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a60ed68e90 0x55a60ed66440 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fe83800a520 tx=0x7fe83804e000 comp rx=0 tx=0).stop 2024-04-26T23:06:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.349+0000 7fe84b35df40 1 -- 172.21.15.40:0/1737658954 shutdown_connections 2024-04-26T23:06:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.349+0000 7fe84b35df40 1 --2- 172.21.15.40:0/1737658954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a60ed83590 0x55a60ee11000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.349+0000 7fe84b35df40 1 --2- 172.21.15.40:0/1737658954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a60ed6a070 0x55a60ed67dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.349+0000 7fe84b35df40 1 --2- 172.21.15.40:0/1737658954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a60ed68e90 0x55a60ed66440 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.349+0000 7fe84b35df40 1 -- 172.21.15.40:0/1737658954 >> 172.21.15.40:0/1737658954 conn(0x55a60ed84df0 msgr2=0x55a60ee12b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.349+0000 7fe84b35df40 1 -- 172.21.15.40:0/1737658954 shutdown_connections 2024-04-26T23:06:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.349+0000 7fe84b35df40 1 -- 172.21.15.40:0/1737658954 wait complete. 2024-04-26T23:06:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.350+0000 7fe84b35df40 1 Processor -- start 2024-04-26T23:06:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.350+0000 7fe84b35df40 1 -- start start 2024-04-26T23:06:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe84b35df40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a60ed68e90 0x55a60ee12790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe84b35df40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a60ed6a070 0x55a60ee0bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe84b35df40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a60ed83590 0x55a60ee10fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe84b35df40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a60ee088e0 con 0x55a60ed83590 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe84b35df40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a60ed69c60 con 0x55a60ed68e90 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe84b35df40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a60ee08e00 con 0x55a60ed6a070 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe849000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a60ed83590 0x55a60ee10fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe849000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a60ed83590 0x55a60ee10fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33706/0 (socket says 172.21.15.40:33706) 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe849000640 1 -- 172.21.15.40:0/3297328975 learned_addr learned my addr 172.21.15.40:0/3297328975 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe848600640 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a60ed68e90 0x55a60ee12790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe83fe00640 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a60ed6a070 0x55a60ee0bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe849000640 1 -- 172.21.15.40:0/3297328975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a60ed6a070 msgr2=0x55a60ee0bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe849000640 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a60ed6a070 0x55a60ee0bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe849000640 1 -- 172.21.15.40:0/3297328975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a60ed68e90 msgr2=0x55a60ee12790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe849000640 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a60ed68e90 0x55a60ee12790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.351+0000 7fe849000640 1 -- 172.21.15.40:0/3297328975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a60ee09fd0 con 0x55a60ed83590 2024-04-26T23:06:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.352+0000 7fe849000640 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a60ed83590 0x55a60ee10fc0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe840005f90 tx=0x7fe84000a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.352+0000 7fe845e00640 1 -- 172.21.15.40:0/3297328975 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe840013430 con 0x55a60ed83590 2024-04-26T23:06:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.352+0000 7fe845e00640 1 -- 172.21.15.40:0/3297328975 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe840018070 con 0x55a60ed83590 2024-04-26T23:06:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.352+0000 7fe845e00640 1 -- 172.21.15.40:0/3297328975 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe840013810 con 0x55a60ed83590 2024-04-26T23:06:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.352+0000 7fe84b35df40 1 -- 172.21.15.40:0/3297328975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a60ee0a2b0 con 0x55a60ed83590 2024-04-26T23:06:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.352+0000 7fe84b35df40 1 -- 172.21.15.40:0/3297328975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a60ee0a870 con 0x55a60ed83590 2024-04-26T23:06:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.354+0000 7fe845e00640 1 -- 172.21.15.40:0/3297328975 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe8400139b0 con 0x55a60ed83590 2024-04-26T23:06:03.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.355+0000 7fe845e00640 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe82406b2d0 0x7fe82406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.355+0000 7fe845e00640 1 -- 172.21.15.40:0/3297328975 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7fe84008abb0 con 0x55a60ed83590 2024-04-26T23:06:03.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.356+0000 7fe84b35df40 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55a60ee0c240 0x55a60eea5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.356+0000 7fe84b35df40 1 -- 172.21.15.40:0/3297328975 --> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] -- osd_op(unknown.0.0:1 9.1d 9:bbf5850a:::10000000007.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67) -- 0x55a60eea5dc0 con 0x55a60ee0c240 2024-04-26T23:06:03.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.356+0000 7fe848600640 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe82406b2d0 0x7fe82406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.356+0000 7fe83fe00640 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55a60ee0c240 0x55a60eea5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.357+0000 7fe83fe00640 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55a60ee0c240 0x55a60eea5880 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.357+0000 7fe848600640 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe82406b2d0 0x7fe82406d790 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe838002000 tx=0x7fe838051000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.360+0000 7fe83fe00640 1 -- 172.21.15.40:0/3297328975 <== osd.0 v2:172.21.15.40:6800/2897145808 1 ==== osd_op_reply(1 10000000007.00000000 [rmxattr] v67'7 uv7 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe83000d490 con 0x55a60ee0c240 2024-04-26T23:06:03.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.360+0000 7fe84b35df40 1 -- 172.21.15.40:0/3297328975 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55a60ee0c240 msgr2=0x55a60eea5880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.360+0000 7fe84b35df40 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55a60ee0c240 0x55a60eea5880 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.360+0000 7fe84b35df40 1 -- 172.21.15.40:0/3297328975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe82406b2d0 msgr2=0x7fe82406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.360+0000 7fe84b35df40 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe82406b2d0 0x7fe82406d790 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe838002000 tx=0x7fe838051000 comp rx=0 tx=0).stop 2024-04-26T23:06:03.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.360+0000 7fe84b35df40 1 -- 172.21.15.40:0/3297328975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a60ed83590 msgr2=0x55a60ee10fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.360+0000 7fe84b35df40 1 --2- 172.21.15.40:0/3297328975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a60ed83590 0x55a60ee10fc0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe840005f90 tx=0x7fe84000a570 comp rx=0 tx=0).stop 2024-04-26T23:06:03.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.360+0000 7fe849000640 1 -- 172.21.15.40:0/3297328975 reap_dead start 2024-04-26T23:06:03.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.360+0000 7fe84b35df40 1 -- 172.21.15.40:0/3297328975 shutdown_connections 2024-04-26T23:06:03.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.360+0000 7fe84b35df40 1 -- 172.21.15.40:0/3297328975 >> 172.21.15.40:0/3297328975 conn(0x55a60ed84df0 msgr2=0x55a60ed66270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.360+0000 7fe84b35df40 1 -- 172.21.15.40:0/3297328975 shutdown_connections 2024-04-26T23:06:03.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.360+0000 7fe84b35df40 1 -- 172.21.15.40:0/3297328975 wait complete. 2024-04-26T23:06:03.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:06:03.365 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub start /d1/d2/d3 recursive,force 2024-04-26T23:06:03.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.553+0000 7f2f3cc00640 1 Processor -- start 2024-04-26T23:06:03.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.553+0000 7f2f3cc00640 1 -- start start 2024-04-26T23:06:03.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.555+0000 7f2f3cc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2f380b6b40 0x7f2f380b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.555+0000 7f2f3cc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f38163f40 0x7f2f380b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.555+0000 7f2f3cc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f380b7880 con 0x7f2f381633a0 2024-04-26T23:06:03.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.555+0000 7f2f3cc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2f380d0bb0 con 0x7f2f380b6b40 2024-04-26T23:06:03.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.555+0000 7f2f3cc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2f380d05a0 con 0x7f2f38163f40 2024-04-26T23:06:03.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.556+0000 7f2f37400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2f380b6b40 0x7f2f380b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.556+0000 7f2f37400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2f380b6b40 0x7f2f380b6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:55836/0 (socket says 172.21.15.40:55836) 2024-04-26T23:06:03.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.556+0000 7f2f37400640 1 -- 172.21.15.40:0/1493939811 learned_addr learned my addr 172.21.15.40:0/1493939811 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.556+0000 7f2f37e00640 1 --2- 172.21.15.40:0/1493939811 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f38163f40 0x7f2f380b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.556+0000 7f2f37e00640 1 -- 172.21.15.40:0/1493939811 >> v2:172.21.15.103:3300/0 conn(0x7f2f380b6b40 msgr2=0x7f2f380b6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.556+0000 7f2f37e00640 1 --2- 172.21.15.40:0/1493939811 >> v2:172.21.15.103:3300/0 conn(0x7f2f380b6b40 0x7f2f380b6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.556+0000 7f2f37e00640 1 -- 172.21.15.40:0/1493939811 >> v1:172.21.15.103:6789/0 conn(0x7f2f381633a0 legacy=0x7f2f38163830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.556+0000 7f2f37e00640 1 -- 172.21.15.40:0/1493939811 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f380b75a0 con 0x7f2f38163f40 2024-04-26T23:06:03.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.557+0000 7f2f37400640 1 --2- 172.21.15.40:0/1493939811 >> v2:172.21.15.103:3300/0 conn(0x7f2f380b6b40 0x7f2f380b6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:06:03.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.558+0000 7f2f37e00640 1 --2- 172.21.15.40:0/1493939811 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f38163f40 0x7f2f380b47b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2f300097a0 tx=0x7f2f3002a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4325b862b642ff72 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.558+0000 7f2f36000640 1 -- 172.21.15.40:0/1493939811 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30037070 con 0x7f2f38163f40 2024-04-26T23:06:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.558+0000 7f2f36000640 1 -- 172.21.15.40:0/1493939811 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f3002ae40 con 0x7f2f38163f40 2024-04-26T23:06:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.558+0000 7f2f36000640 1 -- 172.21.15.40:0/1493939811 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30032660 con 0x7f2f38163f40 2024-04-26T23:06:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.559+0000 7f2f3cc00640 1 -- 172.21.15.40:0/1493939811 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f38163f40 msgr2=0x7f2f380b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.559+0000 7f2f3cc00640 1 --2- 172.21.15.40:0/1493939811 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f38163f40 0x7f2f380b47b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2f300097a0 tx=0x7f2f3002a0f0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.559+0000 7f2f3cc00640 1 -- 172.21.15.40:0/1493939811 shutdown_connections 2024-04-26T23:06:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.559+0000 7f2f3cc00640 1 --2- 172.21.15.40:0/1493939811 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f38163f40 0x7f2f380b47b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.559+0000 7f2f3cc00640 1 --2- 172.21.15.40:0/1493939811 >> v2:172.21.15.103:3300/0 conn(0x7f2f380b6b40 0x7f2f380b6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.559+0000 7f2f3cc00640 1 -- 172.21.15.40:0/1493939811 >> 172.21.15.40:0/1493939811 conn(0x7f2f380d4e50 msgr2=0x7f2f380d5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.559+0000 7f2f3cc00640 1 -- 172.21.15.40:0/1493939811 shutdown_connections 2024-04-26T23:06:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.559+0000 7f2f3cc00640 1 -- 172.21.15.40:0/1493939811 wait complete. 2024-04-26T23:06:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.561+0000 7f2f3cc00640 1 Processor -- start 2024-04-26T23:06:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.561+0000 7f2f3cc00640 1 -- start start 2024-04-26T23:06:03.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.562+0000 7f2f3cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f380b6b40 0x7f2f38150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.562+0000 7f2f3cc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f381633a0 0x7f2f380b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.562+0000 7f2f3cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f38163f40 0x7f2f380b5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.562+0000 7f2f3cc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f380b6300 con 0x7f2f380b6b40 2024-04-26T23:06:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.562+0000 7f2f3cc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2f380b6180 con 0x7f2f38163f40 2024-04-26T23:06:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.562+0000 7f2f3cc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2f380d3aa0 con 0x7f2f381633a0 2024-04-26T23:06:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.562+0000 7f2f37400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f380b6b40 0x7f2f38150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.562+0000 7f2f37400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f380b6b40 0x7f2f38150380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33716/0 (socket says 172.21.15.40:33716) 2024-04-26T23:06:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.562+0000 7f2f37400640 1 -- 172.21.15.40:0/3023169201 learned_addr learned my addr 172.21.15.40:0/3023169201 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.562+0000 7f2f37e00640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f38163f40 0x7f2f380b5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.562+0000 7f2f37400640 1 -- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f381633a0 msgr2=0x7f2f380b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.562+0000 7f2f36a00640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f381633a0 0x7f2f380b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.563+0000 7f2f37400640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f381633a0 0x7f2f380b5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.563+0000 7f2f37400640 1 -- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f38163f40 msgr2=0x7f2f380b5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.563+0000 7f2f37400640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f38163f40 0x7f2f380b5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.563+0000 7f2f37400640 1 -- 172.21.15.40:0/3023169201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f381f55c0 con 0x7f2f380b6b40 2024-04-26T23:06:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.563+0000 7f2f37e00640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f38163f40 0x7f2f380b5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.563+0000 7f2f36a00640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f381633a0 0x7f2f380b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.563+0000 7f2f37400640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f380b6b40 0x7f2f38150380 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800ea00 tx=0x7f2f2800eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.563+0000 7f2f2f400640 1 -- 172.21.15.40:0/3023169201 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f28017070 con 0x7f2f380b6b40 2024-04-26T23:06:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.563+0000 7f2f2f400640 1 -- 172.21.15.40:0/3023169201 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f2800ca60 con 0x7f2f380b6b40 2024-04-26T23:06:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.563+0000 7f2f3cc00640 1 -- 172.21.15.40:0/3023169201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f381f58a0 con 0x7f2f380b6b40 2024-04-26T23:06:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.563+0000 7f2f2f400640 1 -- 172.21.15.40:0/3023169201 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2800ce10 con 0x7f2f380b6b40 2024-04-26T23:06:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.563+0000 7f2f3cc00640 1 -- 172.21.15.40:0/3023169201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f381f52d0 con 0x7f2f380b6b40 2024-04-26T23:06:03.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.566+0000 7f2f2f400640 1 -- 172.21.15.40:0/3023169201 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2f2801a070 con 0x7f2f380b6b40 2024-04-26T23:06:03.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.567+0000 7f2f40faf740 1 Processor -- start 2024-04-26T23:06:03.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.567+0000 7f2f40faf740 1 -- start start 2024-04-26T23:06:03.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.567+0000 7f2f2f400640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f1406c8f0 0x7f2f1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.568+0000 7f2f2f400640 1 -- 172.21.15.40:0/3023169201 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f2f2808c070 con 0x7f2f380b6b40 2024-04-26T23:06:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.568+0000 7f2f37e00640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f1406c8f0 0x7f2f1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.568+0000 7f2f40faf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 0x563786e269a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.568+0000 7f2f40faf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e57130 0x563786e564b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.568+0000 7f2f40faf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e569f0 0x563786e2b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.568+0000 7f2f40faf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563786d90b80 con 0x563786e57130 2024-04-26T23:06:03.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.568+0000 7f2f40faf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563786d2d420 con 0x563786e569f0 2024-04-26T23:06:03.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.568+0000 7f2f40faf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x563786d8cb40 con 0x563786e74010 2024-04-26T23:06:03.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.569+0000 7f2f36a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e57130 0x563786e564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.569+0000 7f2f36a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e57130 0x563786e564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33724/0 (socket says 172.21.15.40:33724) 2024-04-26T23:06:03.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.569+0000 7f2f36a00640 1 -- 172.21.15.40:0/2583527387 learned_addr learned my addr 172.21.15.40:0/2583527387 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.569+0000 7f2f37400640 1 --2- 172.21.15.40:0/2583527387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 0x563786e269a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.569+0000 7f2f36a00640 1 -- 172.21.15.40:0/2583527387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 msgr2=0x563786e269a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.569+0000 7f2f36a00640 1 --2- 172.21.15.40:0/2583527387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 0x563786e269a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.569+0000 7f2f36a00640 1 -- 172.21.15.40:0/2583527387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e569f0 msgr2=0x563786e2b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.569+0000 7f2f36a00640 1 --2- 172.21.15.40:0/2583527387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e569f0 0x563786e2b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.569+0000 7f2f36a00640 1 -- 172.21.15.40:0/2583527387 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563786b6aa50 con 0x563786e57130 2024-04-26T23:06:03.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.569+0000 7f2f36a00640 1 --2- 172.21.15.40:0/2583527387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e57130 0x563786e564b0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f2f20001e70 tx=0x7f2f2000d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632fbaee187580c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.569+0000 7f2f37e00640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f1406c8f0 0x7f2f1406edb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2f30034000 tx=0x7f2f3002c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.570+0000 7f2f27400640 1 -- 172.21.15.40:0/2583527387 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f20018070 con 0x563786e57130 2024-04-26T23:06:03.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.570+0000 7f2f27400640 1 -- 172.21.15.40:0/2583527387 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f20002080 con 0x563786e57130 2024-04-26T23:06:03.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.570+0000 7f2f27400640 1 -- 172.21.15.40:0/2583527387 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f20003bb0 con 0x563786e57130 2024-04-26T23:06:03.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.570+0000 7f2f40faf740 1 -- 172.21.15.40:0/2583527387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e57130 msgr2=0x563786e564b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.570+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2583527387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e57130 0x563786e564b0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f2f20001e70 tx=0x7f2f2000d520 comp rx=0 tx=0).stop 2024-04-26T23:06:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.570+0000 7f2f40faf740 1 -- 172.21.15.40:0/2583527387 shutdown_connections 2024-04-26T23:06:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.570+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2583527387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e569f0 0x563786e2b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.570+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2583527387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e57130 0x563786e564b0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.570+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2583527387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 0x563786e269a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.570+0000 7f2f40faf740 1 -- 172.21.15.40:0/2583527387 >> 172.21.15.40:0/2583527387 conn(0x563786e28c10 msgr2=0x563786e714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.570+0000 7f2f40faf740 1 -- 172.21.15.40:0/2583527387 shutdown_connections 2024-04-26T23:06:03.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.570+0000 7f2f40faf740 1 -- 172.21.15.40:0/2583527387 wait complete. 2024-04-26T23:06:03.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.571+0000 7f2f40faf740 1 -- start start 2024-04-26T23:06:03.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.571+0000 7f2f40faf740 1 Processor -- start 2024-04-26T23:06:03.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.572+0000 7f2f40faf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e57130 0x563786e68170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.572+0000 7f2f40faf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 0x563786e59c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.572+0000 7f2f40faf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e5a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.572+0000 7f2f40faf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563786e2adc0 con 0x563786e58d90 2024-04-26T23:06:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.572+0000 7f2f40faf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563786d18370 con 0x563786e57130 2024-04-26T23:06:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.572+0000 7f2f40faf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x563786e2a1b0 con 0x563786e74010 2024-04-26T23:06:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.572+0000 7f2f40faf740 10 client.0 _ll_register_callbacks cb 0x563786d87890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.572+0000 7f2f37e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e5a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.573+0000 7f2f37e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e5a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33728/0 (socket says 172.21.15.40:33728) 2024-04-26T23:06:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.573+0000 7f2f37e00640 1 -- 172.21.15.40:0/2882285461 learned_addr learned my addr 172.21.15.40:0/2882285461 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.573+0000 7f2f36a00640 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 0x563786e59c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.573+0000 7f2f37400640 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e57130 0x563786e68170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.573+0000 7f2f37e00640 1 -- 172.21.15.40:0/2882285461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 msgr2=0x563786e59c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.573+0000 7f2f37e00640 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 0x563786e59c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.573+0000 7f2f37e00640 1 -- 172.21.15.40:0/2882285461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e57130 msgr2=0x563786e68170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.573+0000 7f2f37e00640 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e57130 0x563786e68170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.573+0000 7f2f37e00640 1 -- 172.21.15.40:0/2882285461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563786a623e0 con 0x563786e58d90 2024-04-26T23:06:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.573+0000 7f2f36a00640 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 0x563786e59c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.573+0000 7f2f37400640 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e57130 0x563786e68170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.573+0000 7f2f37e00640 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e5a140 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f2f30004230 tx=0x7f2f30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15647ad419e654e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.573+0000 7f2f1be00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f1be00640 1 -- 172.21.15.40:0/2882285461 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3004d060 con 0x563786e58d90 2024-04-26T23:06:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f40faf740 1 -- 172.21.15.40:0/2882285461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f20002080 con 0x563786e58d90 2024-04-26T23:06:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f1be00640 1 -- 172.21.15.40:0/2882285461 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f2f30003610 con 0x563786e58d90 2024-04-26T23:06:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f1be00640 1 -- 172.21.15.40:0/2882285461 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f2f30053050 con 0x563786e58d90 2024-04-26T23:06:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f40faf740 10 client.7128 fetch_fsmap learned FSMap version 63 2024-04-26T23:06:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f40faf740 1 -- 172.21.15.40:0/2882285461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) -- 0x563786bcb1e0 con 0x563786e58d90 2024-04-26T23:06:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f1be00640 1 -- 172.21.15.40:0/2882285461 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f20002080 con 0x563786e58d90 2024-04-26T23:06:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f1be00640 1 -- 172.21.15.40:0/2882285461 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 63) ==== 3550+0+0 (secure 0 0 0) 0x7f2f30056020 con 0x563786e58d90 2024-04-26T23:06:03.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f40faf740 10 client.7128 fetch_fsmap finished waiting for FSMap version 63 2024-04-26T23:06:03.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f40faf740 10 client.7128 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-26T23:06:03.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f40faf740 20 client.7128 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:03.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x563786e54470 0x563786e54880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f40faf740 4 client.7128 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:06:03.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.574+0000 7f2f40faf740 1 -- 172.21.15.40:0/2882285461 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563786d9e790 con 0x563786e54470 2024-04-26T23:06:03.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.575+0000 7f2f37400640 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x563786e54470 0x563786e54880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.575+0000 7f2f37400640 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x563786e54470 0x563786e54880 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22694298cf81f02d server_cookie=d2c18ef4781dd8d9 in_seq=0 out_seq=0 2024-04-26T23:06:03.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.575+0000 7f2f1be00640 10 client.7128 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:03.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.576+0000 7f2f1be00640 1 -- 172.21.15.40:0/2882285461 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563786d9e790 con 0x563786e54470 2024-04-26T23:06:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.576+0000 7f2f1be00640 10 client.7128 handle_command_reply: tid=0 2024-04-26T23:06:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.576+0000 7f2f40faf740 1 client.7128 shutdown 2024-04-26T23:06:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.576+0000 7f2f40faf740 1 -- 172.21.15.40:0/2882285461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 msgr2=0x563786e5a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.576+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e5a140 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f2f30004230 tx=0x7f2f30004730 comp rx=0 tx=0).stop 2024-04-26T23:06:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.576+0000 7f2f40faf740 1 -- 172.21.15.40:0/2882285461 shutdown_connections 2024-04-26T23:06:03.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.576+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x563786e54470 0x563786e54880 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.576+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e5a140 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.576+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 0x563786e59c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.576+0000 7f2f1be00640 0 client.7128 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:03.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.576+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2882285461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e57130 0x563786e68170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.576+0000 7f2f40faf740 1 -- 172.21.15.40:0/2882285461 >> 172.21.15.40:0/2882285461 conn(0x563786e569f0 msgr2=0x563786e714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.577+0000 7f2f40faf740 1 -- 172.21.15.40:0/2882285461 shutdown_connections 2024-04-26T23:06:03.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.577+0000 7f2f40faf740 1 -- 172.21.15.40:0/2882285461 wait complete. 2024-04-26T23:06:03.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.577+0000 7f2f40faf740 20 client.7128 trim_cache size 0 max 16384 2024-04-26T23:06:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.596+0000 7f2f40faf740 1 Processor -- start 2024-04-26T23:06:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.596+0000 7f2f40faf740 1 -- start start 2024-04-26T23:06:03.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.597+0000 7f2f40faf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e57130 0x563786e68f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.597+0000 7f2f40faf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.597+0000 7f2f40faf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 0x563786e71a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.597+0000 7f2f40faf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f38201f70 con 0x563786e58d90 2024-04-26T23:06:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.597+0000 7f2f40faf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2f38201110 con 0x563786e57130 2024-04-26T23:06:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.597+0000 7f2f40faf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2f38201840 con 0x563786e74010 2024-04-26T23:06:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.597+0000 7f2f36a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.597+0000 7f2f36a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e714e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33740/0 (socket says 172.21.15.40:33740) 2024-04-26T23:06:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.597+0000 7f2f36a00640 1 -- 172.21.15.40:0/2373627286 learned_addr learned my addr 172.21.15.40:0/2373627286 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.598+0000 7f2f37400640 1 --2- 172.21.15.40:0/2373627286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e57130 0x563786e68f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.598+0000 7f2f36a00640 1 -- 172.21.15.40:0/2373627286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 msgr2=0x563786e71a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.598+0000 7f2f36a00640 1 --2- 172.21.15.40:0/2373627286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 0x563786e71a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.598+0000 7f2f36a00640 1 -- 172.21.15.40:0/2373627286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e57130 msgr2=0x563786e68f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.598+0000 7f2f36a00640 1 --2- 172.21.15.40:0/2373627286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e57130 0x563786e68f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.598+0000 7f2f36a00640 1 -- 172.21.15.40:0/2373627286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563786a3b4a0 con 0x563786e58d90 2024-04-26T23:06:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.598+0000 7f2f37400640 1 --2- 172.21.15.40:0/2373627286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e57130 0x563786e68f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.598+0000 7f2f36a00640 1 --2- 172.21.15.40:0/2373627286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e714e0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f2f20012000 tx=0x7f2f2000b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d59988d151f44d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.598+0000 7f2f26a00640 1 -- 172.21.15.40:0/2373627286 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f20024070 con 0x563786e58d90 2024-04-26T23:06:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.598+0000 7f2f26a00640 1 -- 172.21.15.40:0/2373627286 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f20001910 con 0x563786e58d90 2024-04-26T23:06:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.598+0000 7f2f26a00640 1 -- 172.21.15.40:0/2373627286 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f20018070 con 0x563786e58d90 2024-04-26T23:06:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.599+0000 7f2f40faf740 1 -- 172.21.15.40:0/2373627286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 msgr2=0x563786e714e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.599+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2373627286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e714e0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f2f20012000 tx=0x7f2f2000b000 comp rx=0 tx=0).stop 2024-04-26T23:06:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.599+0000 7f2f40faf740 1 -- 172.21.15.40:0/2373627286 shutdown_connections 2024-04-26T23:06:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.599+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2373627286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e74010 0x563786e71a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.599+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2373627286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e714e0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.599+0000 7f2f40faf740 1 --2- 172.21.15.40:0/2373627286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e57130 0x563786e68f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.599+0000 7f2f40faf740 1 -- 172.21.15.40:0/2373627286 >> 172.21.15.40:0/2373627286 conn(0x563786e569f0 msgr2=0x563786e55320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.599+0000 7f2f40faf740 1 -- 172.21.15.40:0/2373627286 shutdown_connections 2024-04-26T23:06:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.599+0000 7f2f40faf740 1 -- 172.21.15.40:0/2373627286 wait complete. 2024-04-26T23:06:03.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.600+0000 7f2f40faf740 1 -- start start 2024-04-26T23:06:03.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.600+0000 7f2f40faf740 1 Processor -- start 2024-04-26T23:06:03.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f40faf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e57130 0x563786e75290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f40faf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f40faf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e74010 0x563786e55810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f40faf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f380ccce0 con 0x563786e58d90 2024-04-26T23:06:03.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f40faf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563786e560f0 con 0x563786e74010 2024-04-26T23:06:03.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f40faf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x563786e59660 con 0x563786e57130 2024-04-26T23:06:03.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f40faf740 10 client.0 _ll_register_callbacks cb 0x7f2f30035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:03.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f36a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f36a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33744/0 (socket says 172.21.15.40:33744) 2024-04-26T23:06:03.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f36a00640 1 -- 172.21.15.40:0/1926923639 learned_addr learned my addr 172.21.15.40:0/1926923639 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f36a00640 1 -- 172.21.15.40:0/1926923639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e57130 msgr2=0x563786e75290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f36a00640 1 --2- 172.21.15.40:0/1926923639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e57130 0x563786e75290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f36a00640 1 -- 172.21.15.40:0/1926923639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e74010 msgr2=0x563786e55810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f36a00640 1 --2- 172.21.15.40:0/1926923639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e74010 0x563786e55810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.602+0000 7f2f36a00640 1 -- 172.21.15.40:0/1926923639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563786e59190 con 0x563786e58d90 2024-04-26T23:06:03.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.603+0000 7f2f36a00640 1 --2- 172.21.15.40:0/1926923639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e552d0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009010 tx=0x7f2f20000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f29e4a73253e9f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.603+0000 7f2f1be00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:03.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.603+0000 7f2f1be00640 1 -- 172.21.15.40:0/1926923639 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f20018050 con 0x563786e58d90 2024-04-26T23:06:03.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.603+0000 7f2f1be00640 1 -- 172.21.15.40:0/1926923639 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f2f20004d30 con 0x563786e58d90 2024-04-26T23:06:03.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.603+0000 7f2f40faf740 1 -- 172.21.15.40:0/1926923639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f20001910 con 0x563786e58d90 2024-04-26T23:06:03.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.603+0000 7f2f1be00640 1 -- 172.21.15.40:0/1926923639 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f2f20035050 con 0x563786e58d90 2024-04-26T23:06:03.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.603+0000 7f2f40faf740 10 client.7134 fetch_fsmap learned FSMap version 63 2024-04-26T23:06:03.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.603+0000 7f2f40faf740 1 -- 172.21.15.40:0/1926923639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) -- 0x563786e593a0 con 0x563786e58d90 2024-04-26T23:06:03.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.604+0000 7f2f1be00640 1 -- 172.21.15.40:0/1926923639 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f20001910 con 0x563786e58d90 2024-04-26T23:06:03.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.604+0000 7f2f1be00640 1 -- 172.21.15.40:0/1926923639 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 63) ==== 3550+0+0 (secure 0 0 0) 0x7f2f20038020 con 0x563786e58d90 2024-04-26T23:06:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.604+0000 7f2f40faf740 10 client.7134 fetch_fsmap finished waiting for FSMap version 63 2024-04-26T23:06:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.604+0000 7f2f40faf740 10 client.7134 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-26T23:06:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.604+0000 7f2f40faf740 20 client.7134 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.604+0000 7f2f40faf740 1 --2- 172.21.15.40:0/1926923639 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x563786e73a80 0x563786e4e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.604+0000 7f2f40faf740 4 client.7134 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/d1/d2/d3", "scrubops": ["recursive", "force"]}] 2024-04-26T23:06:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.604+0000 7f2f40faf740 1 -- 172.21.15.40:0/1926923639 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "scrub start", "path": "/d1/d2/d3", "scrubops": ["recursive", "force"]}) -- 0x563786d98600 con 0x563786e73a80 2024-04-26T23:06:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.604+0000 7f2f37400640 1 --2- 172.21.15.40:0/1926923639 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x563786e73a80 0x563786e4e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.604+0000 7f2f1be00640 10 client.7134 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.604+0000 7f2f37400640 1 --2- 172.21.15.40:0/1926923639 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x563786e73a80 0x563786e4e2e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29b369045c9a56e0 server_cookie=b78925623bc8ff6d in_seq=0 out_seq=0 2024-04-26T23:06:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.605+0000 7f2f1be00640 1 -- 172.21.15.40:0/1926923639 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563786d98600 con 0x563786e73a80 2024-04-26T23:06:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.605+0000 7f2f1be00640 10 client.7134 handle_command_reply: tid=0 2024-04-26T23:06:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.605+0000 7f2f40faf740 1 client.7134 shutdown 2024-04-26T23:06:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.606+0000 7f2f40faf740 1 -- 172.21.15.40:0/1926923639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 msgr2=0x563786e552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.606+0000 7f2f40faf740 1 --2- 172.21.15.40:0/1926923639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e552d0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009010 tx=0x7f2f20000f40 comp rx=0 tx=0).stop 2024-04-26T23:06:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.606+0000 7f2f40faf740 1 -- 172.21.15.40:0/1926923639 shutdown_connections 2024-04-26T23:06:03.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.606+0000 7f2f40faf740 1 --2- 172.21.15.40:0/1926923639 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x563786e73a80 0x563786e4e2e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.606+0000 7f2f40faf740 1 --2- 172.21.15.40:0/1926923639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563786e74010 0x563786e55810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.606+0000 7f2f40faf740 1 --2- 172.21.15.40:0/1926923639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563786e58d90 0x563786e552d0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.606+0000 7f2f1be00640 0 client.7134 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:03.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.606+0000 7f2f40faf740 1 --2- 172.21.15.40:0/1926923639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563786e57130 0x563786e75290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.606+0000 7f2f40faf740 1 -- 172.21.15.40:0/1926923639 >> 172.21.15.40:0/1926923639 conn(0x563786e569f0 msgr2=0x563786e732a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.606+0000 7f2f40faf740 1 -- 172.21.15.40:0/1926923639 shutdown_connections 2024-04-26T23:06:03.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.606+0000 7f2f40faf740 1 -- 172.21.15.40:0/1926923639 wait complete. 2024-04-26T23:06:03.616 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-04-26T23:06:03.606+0000 7fa834400640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000004 [...2,head] /d1/d2/d3/d4/d5/ auth{0=1,2=1} v15 ap=1 f(v0 m2024-04-26T23:05:45.944577+0000 1=0+1) n(v2 rc2024-04-26T23:05:46.223573+0000 4=0+4) (inest lock) (ifile excl) caps={5372=pAsLsXsFsx/pAsLsXsFsxcral@4},l=5372 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 authpin=1 scrubqueue=0 0x56322dc9a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000004:[<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:05:45.944577+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:05:46.223573+0000 4=0+4)","memory_value.dirstat":"f(v0 m2024-04-26T23:05:45.944577+0000 1=0+1)","memory_value.rstat":"n(v2 rc2024-04-26T23:05:46.223573+0000 4=0+4)","error_str":""},"return_code":-61} 2024-04-26T23:06:03.617 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:06:03.606+0000 7fa167e00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000002 [...2,head] /d1/d2/d3/ auth{1=1,2=1} v19 ap=1 f(v0 m2024-04-26T23:05:45.942577+0000 1=0+1) n(v3 rc2024-04-26T23:05:46.223573+0000 6=0+6) (inest mix dirty) (ifile excl) caps={5372=pAsLsXsFsx/pAsLsXsFsxcral@4},l=5372 | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 0x5617f06dc680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000002:[<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:05:45.942577+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:05:46.223573+0000 6=0+6)","memory_value.dirstat":"f(v0 m2024-04-26T23:05:45.942577+0000 1=0+1)","memory_value.rstat":"n(v3 rc2024-04-26T23:05:46.223573+0000 6=0+6)","error_str":""},"return_code":-61} 2024-04-26T23:06:03.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.606+0000 7f2f40faf740 20 client.7134 trim_cache size 0 max 16384 2024-04-26T23:06:03.617 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:06:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-04-26T23:06:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "448789f1-f121-48b8-90f3-93a2f4a0d6a1", 2024-04-26T23:06:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-04-26T23:06:03.618 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:06:03.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.607+0000 7f2f3cc00640 1 -- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f1406c8f0 msgr2=0x7f2f1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.607+0000 7f2f3cc00640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f1406c8f0 0x7f2f1406edb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2f30034000 tx=0x7f2f3002c000 comp rx=0 tx=0).stop 2024-04-26T23:06:03.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.607+0000 7f2f3cc00640 1 -- 172.21.15.40:0/3023169201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f380b6b40 msgr2=0x7f2f38150380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.607+0000 7f2f3cc00640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f380b6b40 0x7f2f38150380 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800ea00 tx=0x7f2f2800eec0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.607+0000 7f2f3cc00640 1 -- 172.21.15.40:0/3023169201 shutdown_connections 2024-04-26T23:06:03.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.607+0000 7f2f3cc00640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f1406c8f0 0x7f2f1406edb0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.607+0000 7f2f3cc00640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f38163f40 0x7f2f380b5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.607+0000 7f2f3cc00640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f381633a0 0x7f2f380b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.607+0000 7f2f3cc00640 1 --2- 172.21.15.40:0/3023169201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f380b6b40 0x7f2f38150380 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.607+0000 7f2f3cc00640 1 -- 172.21.15.40:0/3023169201 >> 172.21.15.40:0/3023169201 conn(0x7f2f380d4e50 msgr2=0x7f2f380b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.608+0000 7f2f3cc00640 1 -- 172.21.15.40:0/3023169201 shutdown_connections 2024-04-26T23:06:03.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.608+0000 7f2f3cc00640 1 -- 172.21.15.40:0/3023169201 wait complete. 2024-04-26T23:06:03.622 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:06:03.609+0000 7fa170000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000003 [...2,head] /d1/d2/d3/d4/ auth{1=1,2=1} v19 ap=2 f(v0 m2024-04-26T23:05:45.943577+0000 1=0+1) n(v4 rc2024-04-26T23:05:46.223573+0000 5=0+5) (inest sync->mix g=1,2) caps={5372=pAsLsXsFs/pAsLsXsFsxcral@8} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 export_pin=1 0x5617f06dcc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000003:[<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:05:45.943577+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:05:46.223573+0000 5=0+5)","memory_value.dirstat":"f(v0 m2024-04-26T23:05:45.943577+0000 1=0+1)","memory_value.rstat":"n(v4 rc2024-04-26T23:05:46.223573+0000 5=0+5)","error_str":""},"return_code":-61} 2024-04-26T23:06:03.622 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-04-26T23:06:03.609+0000 7fa83c600640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000005 [...2,head] /d1/d2/d3/d4/d5/d6/ auth{0=1,2=1} v11 ap=2 f(v0 m2024-04-26T23:05:45.946577+0000 1=0+1) n(v1 rc2024-04-26T23:05:46.223573+0000 3=0+3) (inest sync->mix g=0,2 scatter_wanted) caps={5372=pAsLsXsFs/pAsLsXsFsxcral@7} | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 export_pin=2 0x56322dc9ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000005:[<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:05:45.946577+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:05:45.948577+0000 3=0+3)","memory_value.dirstat":"f(v0 m2024-04-26T23:05:45.946577+0000 1=0+1)","memory_value.rstat":"n(v1 rc2024-04-26T23:05:46.223573+0000 3=0+3)","error_str":""},"return_code":-61} 2024-04-26T23:06:03.623 INFO:tasks.ceph.mds.b.smithi103.stderr:2024-04-26T23:06:03.608+0000 7f043fa00640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000006 [...2,head] /d1/d2/d3/d4/d5/d6/d7/ auth v4 ap=1 f(v0 m2024-04-26T23:05:45.948577+0000 1=0+1) n(v0 rc2024-04-26T23:05:45.948577+0000 2=0+2) (inest lock) (ifile excl) caps={5372=pAsLsXsFsx/pAsLsXsFsxcral@4},l=5372 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x561447704000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000006:[<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:05:45.948577+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:05:45.948577+0000 2=0+2)","memory_value.dirstat":"f(v0 m2024-04-26T23:05:45.948577+0000 1=0+1)","memory_value.rstat":"n(v0 rc2024-04-26T23:05:45.948577+0000 2=0+2)","error_str":""},"return_code":-61} 2024-04-26T23:06:03.623 INFO:tasks.ceph.mds.b.smithi103.stderr:2024-04-26T23:06:03.608+0000 7f043fa00640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000007 [...2,head] /d1/d2/d3/d4/d5/d6/d7/d8/ auth v2 ap=1 f() n(v0 1=0+1) (inest lock) (ifile excl) caps={5372=pAsLsXsFsx/AsLsXsFsx@2},l=5372 | importingcaps=0 dirfrag=0 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x561447704580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000007:[<0x10000000006/d8 v2>,<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 1=0+1)","error_str":""},"return_code":-61} 2024-04-26T23:06:03.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:06:03.625 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-04-26T23:06:03.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.816+0000 7f5707000640 1 Processor -- start 2024-04-26T23:06:03.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.817+0000 7f5707000640 1 -- start start 2024-04-26T23:06:03.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.818+0000 7f5707000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f57000d1770 0x7f570015ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.818+0000 7f5707000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f57000b6470 0x7f570015f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.818+0000 7f5707000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57000d5d30 con 0x7f57000d1b50 2024-04-26T23:06:03.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.818+0000 7f5707000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f57000d3760 con 0x7f57000d1770 2024-04-26T23:06:03.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.818+0000 7f5707000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f57000d09f0 con 0x7f57000b6470 2024-04-26T23:06:03.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.818+0000 7f5705c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f57000d1770 0x7f570015ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.819+0000 7f5705c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f57000d1770 0x7f570015ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33760/0 (socket says 172.21.15.40:33760) 2024-04-26T23:06:03.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.819+0000 7f5705c00640 1 -- 172.21.15.40:0/2475922221 learned_addr learned my addr 172.21.15.40:0/2475922221 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.819+0000 7f5705c00640 1 -- 172.21.15.40:0/2475922221 >> v2:172.21.15.103:3300/0 conn(0x7f57000b6470 msgr2=0x7f570015f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.819+0000 7f5705200640 1 --2- 172.21.15.40:0/2475922221 >> v2:172.21.15.103:3300/0 conn(0x7f57000b6470 0x7f570015f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.819+0000 7f5705c00640 1 --2- 172.21.15.40:0/2475922221 >> v2:172.21.15.103:3300/0 conn(0x7f57000b6470 0x7f570015f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.819+0000 7f5705c00640 1 -- 172.21.15.40:0/2475922221 >> v1:172.21.15.103:6789/0 conn(0x7f57000d1b50 legacy=0x7f57000b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.819+0000 7f5705c00640 1 -- 172.21.15.40:0/2475922221 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57000d5a70 con 0x7f57000d1770 2024-04-26T23:06:03.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.820+0000 7f5705c00640 1 --2- 172.21.15.40:0/2475922221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57000d1770 0x7f570015ef70 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f570008e290 tx=0x7f56f402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c32acbf3511bf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.820+0000 7f56ffe00640 1 -- 172.21.15.40:0/2475922221 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4037070 con 0x7f57000d1770 2024-04-26T23:06:03.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.820+0000 7f56ffe00640 1 -- 172.21.15.40:0/2475922221 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f4032440 con 0x7f57000d1770 2024-04-26T23:06:03.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.820+0000 7f56ffe00640 1 -- 172.21.15.40:0/2475922221 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4032810 con 0x7f57000d1770 2024-04-26T23:06:03.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.821+0000 7f5707000640 1 -- 172.21.15.40:0/2475922221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57000d1770 msgr2=0x7f570015ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.821+0000 7f5707000640 1 --2- 172.21.15.40:0/2475922221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57000d1770 0x7f570015ef70 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f570008e290 tx=0x7f56f402a1a0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.821+0000 7f5707000640 1 -- 172.21.15.40:0/2475922221 shutdown_connections 2024-04-26T23:06:03.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.821+0000 7f5707000640 1 --2- 172.21.15.40:0/2475922221 >> v2:172.21.15.103:3300/0 conn(0x7f57000b6470 0x7f570015f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.821+0000 7f5707000640 1 --2- 172.21.15.40:0/2475922221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57000d1770 0x7f570015ef70 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.821+0000 7f5707000640 1 -- 172.21.15.40:0/2475922221 >> 172.21.15.40:0/2475922221 conn(0x7f57000d4dd0 msgr2=0x7f57000d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.821+0000 7f5707000640 1 -- 172.21.15.40:0/2475922221 shutdown_connections 2024-04-26T23:06:03.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.821+0000 7f5707000640 1 -- 172.21.15.40:0/2475922221 wait complete. 2024-04-26T23:06:03.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.823+0000 7f5707000640 1 Processor -- start 2024-04-26T23:06:03.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.823+0000 7f5707000640 1 -- start start 2024-04-26T23:06:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.824+0000 7f5707000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57000b6470 0x7f570015b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.824+0000 7f5707000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57000d1770 0x7f57001f66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.824+0000 7f5707000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57000d1b50 0x7f5700158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.824+0000 7f5707000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f57000d3bf0 con 0x7f57000d1b50 2024-04-26T23:06:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.824+0000 7f5707000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57000d2670 con 0x7f57000d1770 2024-04-26T23:06:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.824+0000 7f5707000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f57000d3d70 con 0x7f57000b6470 2024-04-26T23:06:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.824+0000 7f5706600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57000d1b50 0x7f5700158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.824+0000 7f5706600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57000d1b50 0x7f5700158430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33770/0 (socket says 172.21.15.40:33770) 2024-04-26T23:06:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.824+0000 7f5706600640 1 -- 172.21.15.40:0/513914338 learned_addr learned my addr 172.21.15.40:0/513914338 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.825+0000 7f5705c00640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57000b6470 0x7f570015b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.825+0000 7f5705200640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57000d1770 0x7f57001f66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.825+0000 7f5706600640 1 -- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57000b6470 msgr2=0x7f570015b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.825+0000 7f5706600640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57000b6470 0x7f570015b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.825+0000 7f5706600640 1 -- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57000d1770 msgr2=0x7f57001f66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.825+0000 7f5706600640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57000d1770 0x7f57001f66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.825+0000 7f5706600640 1 -- 172.21.15.40:0/513914338 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57001f7ec0 con 0x7f57000d1b50 2024-04-26T23:06:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.825+0000 7f5705200640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57000d1770 0x7f57001f66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.825+0000 7f5705c00640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57000b6470 0x7f570015b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.825+0000 7f5706600640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57000d1b50 0x7f5700158430 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f56f800c780 tx=0x7f56f800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.825+0000 7f56fd600640 1 -- 172.21.15.40:0/513914338 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f8017070 con 0x7f57000d1b50 2024-04-26T23:06:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.825+0000 7f56fd600640 1 -- 172.21.15.40:0/513914338 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f8003da0 con 0x7f57000d1b50 2024-04-26T23:06:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.825+0000 7f56fd600640 1 -- 172.21.15.40:0/513914338 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f80052b0 con 0x7f57000d1b50 2024-04-26T23:06:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.826+0000 7f5707000640 1 -- 172.21.15.40:0/513914338 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5700158970 con 0x7f57000d1b50 2024-04-26T23:06:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.826+0000 7f5707000640 1 -- 172.21.15.40:0/513914338 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57001f7100 con 0x7f57000d1b50 2024-04-26T23:06:03.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.828+0000 7f56fd600640 1 -- 172.21.15.40:0/513914338 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f56f8005450 con 0x7f57000d1b50 2024-04-26T23:06:03.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.829+0000 7f56fd600640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f56e006c920 0x7f56e006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.829+0000 7f56fd600640 1 -- 172.21.15.40:0/513914338 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f56f808b290 con 0x7f57000d1b50 2024-04-26T23:06:03.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.830+0000 7f570b596740 1 Processor -- start 2024-04-26T23:06:03.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.830+0000 7f570b596740 1 -- start start 2024-04-26T23:06:03.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.830+0000 7f5705c00640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f56e006c920 0x7f56e006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.831+0000 7f570b596740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.831+0000 7f570b596740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d960cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.831+0000 7f570b596740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d960833f0 0x561d960c9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.831+0000 7f570b596740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561d95f93030 con 0x561d960ad710 2024-04-26T23:06:03.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.831+0000 7f570b596740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561d95fae7f0 con 0x561d960cafb0 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.831+0000 7f570b596740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561d95f697e0 con 0x561d960833f0 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.831+0000 7f5705c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.831+0000 7f5705c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960adaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33786/0 (socket says 172.21.15.40:33786) 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.831+0000 7f5705c00640 1 -- 172.21.15.40:0/633617899 learned_addr learned my addr 172.21.15.40:0/633617899 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.831+0000 7f5706600640 1 --2- 172.21.15.40:0/633617899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d960833f0 0x561d960c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.831+0000 7f5705200640 1 --2- 172.21.15.40:0/633617899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d960cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.831+0000 7f5705c00640 1 -- 172.21.15.40:0/633617899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d960833f0 msgr2=0x561d960c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.831+0000 7f5705c00640 1 --2- 172.21.15.40:0/633617899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d960833f0 0x561d960c9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.832+0000 7f5705c00640 1 -- 172.21.15.40:0/633617899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 msgr2=0x561d960cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.832+0000 7f5705c00640 1 --2- 172.21.15.40:0/633617899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d960cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.832+0000 7f5705c00640 1 -- 172.21.15.40:0/633617899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d95f75050 con 0x561d960ad710 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.832+0000 7f5705c00640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f56e006c920 0x7f56e006ede0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f56f4034000 tx=0x7f56f402c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.832+0000 7f5705c00640 1 --2- 172.21.15.40:0/633617899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960adaf0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f56f40090b0 tx=0x7f56f403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d88031662e73103 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.832+0000 7f56f2000640 1 -- 172.21.15.40:0/633617899 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4043070 con 0x561d960ad710 2024-04-26T23:06:03.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.832+0000 7f56f2000640 1 -- 172.21.15.40:0/633617899 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f404c070 con 0x561d960ad710 2024-04-26T23:06:03.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.832+0000 7f56f2000640 1 -- 172.21.15.40:0/633617899 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4037040 con 0x561d960ad710 2024-04-26T23:06:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.833+0000 7f570b596740 1 -- 172.21.15.40:0/633617899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 msgr2=0x561d960adaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.833+0000 7f570b596740 1 --2- 172.21.15.40:0/633617899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960adaf0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f56f40090b0 tx=0x7f56f403c000 comp rx=0 tx=0).stop 2024-04-26T23:06:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.833+0000 7f570b596740 1 -- 172.21.15.40:0/633617899 shutdown_connections 2024-04-26T23:06:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.833+0000 7f570b596740 1 --2- 172.21.15.40:0/633617899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d960833f0 0x561d960c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.833+0000 7f570b596740 1 --2- 172.21.15.40:0/633617899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d960cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.833+0000 7f570b596740 1 --2- 172.21.15.40:0/633617899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960adaf0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.833+0000 7f570b596740 1 -- 172.21.15.40:0/633617899 >> 172.21.15.40:0/633617899 conn(0x561d9607fd20 msgr2=0x561d960c8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.833+0000 7f570b596740 1 -- 172.21.15.40:0/633617899 shutdown_connections 2024-04-26T23:06:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.833+0000 7f570b596740 1 -- 172.21.15.40:0/633617899 wait complete. 2024-04-26T23:06:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.834+0000 7f570b596740 1 -- start start 2024-04-26T23:06:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.834+0000 7f570b596740 1 Processor -- start 2024-04-26T23:06:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.835+0000 7f570b596740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.835+0000 7f570b596740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d9609af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.835+0000 7f570b596740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 0x561d960c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.835+0000 7f570b596740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561d95feb200 con 0x561d960ad710 2024-04-26T23:06:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.835+0000 7f570b596740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561d95fe5ee0 con 0x561d960cafb0 2024-04-26T23:06:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.835+0000 7f570b596740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561d95ff2e70 con 0x561d9609ec00 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.835+0000 7f570b596740 10 client.0 _ll_register_callbacks cb 0x561d95fdc270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.835+0000 7f5705c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.835+0000 7f5705c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960c63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33792/0 (socket says 172.21.15.40:33792) 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.835+0000 7f5705c00640 1 -- 172.21.15.40:0/309152852 learned_addr learned my addr 172.21.15.40:0/309152852 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.836+0000 7f5705c00640 1 -- 172.21.15.40:0/309152852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 msgr2=0x561d960c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.836+0000 7f5705c00640 1 --2- 172.21.15.40:0/309152852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 0x561d960c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.836+0000 7f5705c00640 1 -- 172.21.15.40:0/309152852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 msgr2=0x561d9609af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.836+0000 7f5705c00640 1 --2- 172.21.15.40:0/309152852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d9609af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.836+0000 7f5705c00640 1 -- 172.21.15.40:0/309152852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d95dc27d0 con 0x561d960ad710 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.836+0000 7f5705c00640 1 --2- 172.21.15.40:0/309152852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960c63a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f56f4000c00 tx=0x7f56f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a3f50527c6ba4b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.836+0000 7f56e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.836+0000 7f56e6a00640 1 -- 172.21.15.40:0/309152852 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4037070 con 0x561d960ad710 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.836+0000 7f56e6a00640 1 -- 172.21.15.40:0/309152852 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f56f4003a20 con 0x561d960ad710 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.836+0000 7f570b596740 1 -- 172.21.15.40:0/309152852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56f404c070 con 0x561d960ad710 2024-04-26T23:06:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.836+0000 7f56e6a00640 1 -- 172.21.15.40:0/309152852 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f56f405d050 con 0x561d960ad710 2024-04-26T23:06:03.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.836+0000 7f570b596740 10 client.7146 fetch_fsmap learned FSMap version 63 2024-04-26T23:06:03.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.836+0000 7f570b596740 1 -- 172.21.15.40:0/309152852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) -- 0x561d95cf83f0 con 0x561d960ad710 2024-04-26T23:06:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.837+0000 7f56e6a00640 1 -- 172.21.15.40:0/309152852 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f404c070 con 0x561d960ad710 2024-04-26T23:06:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.837+0000 7f56e6a00640 1 -- 172.21.15.40:0/309152852 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 63) ==== 3550+0+0 (secure 0 0 0) 0x7f56f4060020 con 0x561d960ad710 2024-04-26T23:06:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.837+0000 7f570b596740 10 client.7146 fetch_fsmap finished waiting for FSMap version 63 2024-04-26T23:06:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.837+0000 7f570b596740 10 client.7146 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-26T23:06:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.837+0000 7f570b596740 20 client.7146 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.837+0000 7f570b596740 1 --2- 172.21.15.40:0/309152852 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x561d960bfaa0 0x561d960bff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.837+0000 7f570b596740 4 client.7146 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:06:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.837+0000 7f570b596740 1 -- 172.21.15.40:0/309152852 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d95ff47e0 con 0x561d960bfaa0 2024-04-26T23:06:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.837+0000 7f5705200640 1 --2- 172.21.15.40:0/309152852 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x561d960bfaa0 0x561d960bff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.838+0000 7f5705200640 1 --2- 172.21.15.40:0/309152852 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x561d960bfaa0 0x561d960bff30 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c78b2de066b88191 server_cookie=af554bd5d4792ee6 in_seq=0 out_seq=0 2024-04-26T23:06:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.838+0000 7f56e6a00640 10 client.7146 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.839+0000 7f56e6a00640 1 -- 172.21.15.40:0/309152852 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561d95ff47e0 con 0x561d960bfaa0 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.839+0000 7f56e6a00640 10 client.7146 handle_command_reply: tid=0 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.839+0000 7f570b596740 1 client.7146 shutdown 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.839+0000 7f570b596740 1 -- 172.21.15.40:0/309152852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 msgr2=0x561d960c63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.839+0000 7f570b596740 1 --2- 172.21.15.40:0/309152852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960c63a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f56f4000c00 tx=0x7f56f4004730 comp rx=0 tx=0).stop 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.839+0000 7f570b596740 1 -- 172.21.15.40:0/309152852 shutdown_connections 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.839+0000 7f570b596740 1 --2- 172.21.15.40:0/309152852 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x561d960bfaa0 0x561d960bff30 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.839+0000 7f570b596740 1 --2- 172.21.15.40:0/309152852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 0x561d960c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.839+0000 7f570b596740 1 --2- 172.21.15.40:0/309152852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d9609af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.839+0000 7f570b596740 1 --2- 172.21.15.40:0/309152852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960c63a0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.839+0000 7f570b596740 1 -- 172.21.15.40:0/309152852 >> 172.21.15.40:0/309152852 conn(0x561d960833f0 msgr2=0x561d96082690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.839+0000 7f56e6a00640 0 client.7146 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.840+0000 7f570b596740 1 -- 172.21.15.40:0/309152852 shutdown_connections 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.840+0000 7f570b596740 1 -- 172.21.15.40:0/309152852 wait complete. 2024-04-26T23:06:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.840+0000 7f570b596740 20 client.7146 trim_cache size 0 max 16384 2024-04-26T23:06:03.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.876+0000 7f570b596740 1 Processor -- start 2024-04-26T23:06:03.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.877+0000 7f570b596740 1 -- start start 2024-04-26T23:06:03.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.878+0000 7f570b596740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 0x561d961445d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.878+0000 7f570b596740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960da920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.878+0000 7f570b596740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d960dae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.878+0000 7f570b596740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5700201ed0 con 0x561d960ad710 2024-04-26T23:06:03.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.878+0000 7f570b596740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5700201070 con 0x561d960cafb0 2024-04-26T23:06:03.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.878+0000 7f570b596740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f57002017a0 con 0x561d9609ec00 2024-04-26T23:06:03.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.878+0000 7f5705200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960da920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.878+0000 7f5705200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960da920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33808/0 (socket says 172.21.15.40:33808) 2024-04-26T23:06:03.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.878+0000 7f5705200640 1 -- 172.21.15.40:0/1165064690 learned_addr learned my addr 172.21.15.40:0/1165064690 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.879+0000 7f5706600640 1 --2- 172.21.15.40:0/1165064690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d960dae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.879+0000 7f5705c00640 1 --2- 172.21.15.40:0/1165064690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 0x561d961445d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.879+0000 7f5705200640 1 -- 172.21.15.40:0/1165064690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 msgr2=0x561d961445d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.879+0000 7f5705200640 1 --2- 172.21.15.40:0/1165064690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 0x561d961445d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.879+0000 7f5705200640 1 -- 172.21.15.40:0/1165064690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 msgr2=0x561d960dae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.879+0000 7f5705200640 1 --2- 172.21.15.40:0/1165064690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d960dae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.879+0000 7f5705200640 1 -- 172.21.15.40:0/1165064690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d95cb80f0 con 0x561d960ad710 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.879+0000 7f5705c00640 1 --2- 172.21.15.40:0/1165064690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 0x561d961445d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.879+0000 7f5705200640 1 --2- 172.21.15.40:0/1165064690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960da920 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f57000b6890 tx=0x7f56ec00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f76ac9bb2a3d626 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.879+0000 7f56f1600640 1 -- 172.21.15.40:0/1165064690 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec013ac0 con 0x561d960ad710 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.880+0000 7f56f1600640 1 -- 172.21.15.40:0/1165064690 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ec013c80 con 0x561d960ad710 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.880+0000 7f56f1600640 1 -- 172.21.15.40:0/1165064690 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec0215c0 con 0x561d960ad710 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.880+0000 7f570b596740 1 -- 172.21.15.40:0/1165064690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 msgr2=0x561d960da920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.880+0000 7f570b596740 1 --2- 172.21.15.40:0/1165064690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960da920 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f57000b6890 tx=0x7f56ec00be80 comp rx=0 tx=0).stop 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.880+0000 7f570b596740 1 -- 172.21.15.40:0/1165064690 shutdown_connections 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.880+0000 7f570b596740 1 --2- 172.21.15.40:0/1165064690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d960dae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.880+0000 7f570b596740 1 --2- 172.21.15.40:0/1165064690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960da920 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.880+0000 7f570b596740 1 --2- 172.21.15.40:0/1165064690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 0x561d961445d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.880+0000 7f570b596740 1 -- 172.21.15.40:0/1165064690 >> 172.21.15.40:0/1165064690 conn(0x561d960833f0 msgr2=0x561d960c1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.880+0000 7f570b596740 1 -- 172.21.15.40:0/1165064690 shutdown_connections 2024-04-26T23:06:03.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.880+0000 7f570b596740 1 -- 172.21.15.40:0/1165064690 wait complete. 2024-04-26T23:06:03.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.881+0000 7f570b596740 1 -- start start 2024-04-26T23:06:03.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.881+0000 7f570b596740 1 Processor -- start 2024-04-26T23:06:03.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.882+0000 7f570b596740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 0x561d96142d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.882+0000 7f570b596740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960cba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.882+0000 7f570b596740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d960cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.882+0000 7f570b596740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561d95ff3fd0 con 0x561d960ad710 2024-04-26T23:06:03.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.882+0000 7f570b596740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57000b88f0 con 0x561d960cafb0 2024-04-26T23:06:03.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.882+0000 7f570b596740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561d960ac310 con 0x561d9609ec00 2024-04-26T23:06:03.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.882+0000 7f570b596740 10 client.0 _ll_register_callbacks cb 0x7f56f40096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:03.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f5705200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f5705200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960cba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33810/0 (socket says 172.21.15.40:33810) 2024-04-26T23:06:03.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f5705200640 1 -- 172.21.15.40:0/4196818413 learned_addr learned my addr 172.21.15.40:0/4196818413 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:03.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f5705c00640 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 0x561d96142d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f5706600640 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d960cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f5705200640 1 -- 172.21.15.40:0/4196818413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 msgr2=0x561d96142d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f5705200640 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 0x561d96142d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f5705200640 1 -- 172.21.15.40:0/4196818413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 msgr2=0x561d960cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f5705200640 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d960cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f5705200640 1 -- 172.21.15.40:0/4196818413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d95faea30 con 0x561d960ad710 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f5706600640 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d960cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f5705c00640 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 0x561d96142d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f5705200640 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960cba00 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00bfd0 tx=0x7f56ec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d9e6c03411aaf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.883+0000 7f56e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f56e6a00640 1 -- 172.21.15.40:0/4196818413 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec02c640 con 0x561d960ad710 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f56e6a00640 1 -- 172.21.15.40:0/4196818413 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f56ec022420 con 0x561d960ad710 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f570b596740 1 -- 172.21.15.40:0/4196818413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56ec013c80 con 0x561d960ad710 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f56e6a00640 1 -- 172.21.15.40:0/4196818413 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f56ec02c7e0 con 0x561d960ad710 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f570b596740 10 client.7152 fetch_fsmap learned FSMap version 63 2024-04-26T23:06:03.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f570b596740 1 -- 172.21.15.40:0/4196818413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) -- 0x561d960eae50 con 0x561d960ad710 2024-04-26T23:06:03.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f56e6a00640 1 -- 172.21.15.40:0/4196818413 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ec013c80 con 0x561d960ad710 2024-04-26T23:06:03.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f56e6a00640 1 -- 172.21.15.40:0/4196818413 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 63) ==== 3550+0+0 (secure 0 0 0) 0x7f56ec02a070 con 0x561d960ad710 2024-04-26T23:06:03.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f570b596740 10 client.7152 fetch_fsmap finished waiting for FSMap version 63 2024-04-26T23:06:03.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f570b596740 10 client.7152 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-26T23:06:03.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f570b596740 20 client.7152 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:03.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f570b596740 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x561d960c8e00 0x561d960b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:03.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f570b596740 4 client.7152 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:06:03.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.884+0000 7f570b596740 1 -- 172.21.15.40:0/4196818413 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561d95fee650 con 0x561d960c8e00 2024-04-26T23:06:03.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.885+0000 7f5705c00640 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x561d960c8e00 0x561d960b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:03.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.885+0000 7f5705c00640 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x561d960c8e00 0x561d960b03e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7e2e90864da137b server_cookie=2b95556383dd4944 in_seq=0 out_seq=0 2024-04-26T23:06:03.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.885+0000 7f56e6a00640 10 client.7152 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:03.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.885+0000 7f56e6a00640 1 -- 172.21.15.40:0/4196818413 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x561d95fee650 con 0x561d960c8e00 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.885+0000 7f56e6a00640 10 client.7152 handle_command_reply: tid=0 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.885+0000 7f570b596740 1 client.7152 shutdown 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.886+0000 7f570b596740 1 -- 172.21.15.40:0/4196818413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 msgr2=0x561d960cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.886+0000 7f570b596740 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960cba00 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00bfd0 tx=0x7f56ec008040 comp rx=0 tx=0).stop 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.886+0000 7f570b596740 1 -- 172.21.15.40:0/4196818413 shutdown_connections 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.886+0000 7f570b596740 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x561d960c8e00 0x561d960b03e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.886+0000 7f570b596740 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d960cafb0 0x561d960cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.886+0000 7f570b596740 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d960ad710 0x561d960cba00 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.886+0000 7f570b596740 1 --2- 172.21.15.40:0/4196818413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561d9609ec00 0x561d96142d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.886+0000 7f56e6a00640 0 client.7152 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.886+0000 7f570b596740 1 -- 172.21.15.40:0/4196818413 >> 172.21.15.40:0/4196818413 conn(0x561d960833f0 msgr2=0x561d960da920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.886+0000 7f570b596740 1 -- 172.21.15.40:0/4196818413 shutdown_connections 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.886+0000 7f570b596740 1 -- 172.21.15.40:0/4196818413 wait complete. 2024-04-26T23:06:03.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.886+0000 7f570b596740 20 client.7152 trim_cache size 0 max 16384 2024-04-26T23:06:03.890 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:06:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-26T23:06:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:06:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: "448789f1-f121-48b8-90f3-93a2f4a0d6a1": { 2024-04-26T23:06:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:06:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "448789f1-f121-48b8-90f3-93a2f4a0d6a1", 2024-04-26T23:06:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:06:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:03.890 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:06:03.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.887+0000 7f5707000640 1 -- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f56e006c920 msgr2=0x7f56e006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.887+0000 7f5707000640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f56e006c920 0x7f56e006ede0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f56f4034000 tx=0x7f56f402c000 comp rx=0 tx=0).stop 2024-04-26T23:06:03.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.887+0000 7f5707000640 1 -- 172.21.15.40:0/513914338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57000d1b50 msgr2=0x7f5700158430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:03.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.887+0000 7f5707000640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57000d1b50 0x7f5700158430 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f56f800c780 tx=0x7f56f800cc40 comp rx=0 tx=0).stop 2024-04-26T23:06:03.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.887+0000 7f5707000640 1 -- 172.21.15.40:0/513914338 shutdown_connections 2024-04-26T23:06:03.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.887+0000 7f5707000640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f56e006c920 0x7f56e006ede0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.887+0000 7f5707000640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57000d1b50 0x7f5700158430 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.887+0000 7f5707000640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57000d1770 0x7f57001f66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.887+0000 7f5707000640 1 --2- 172.21.15.40:0/513914338 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57000b6470 0x7f570015b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:03.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.887+0000 7f5707000640 1 -- 172.21.15.40:0/513914338 >> 172.21.15.40:0/513914338 conn(0x7f57000d4dd0 msgr2=0x7f57000cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:03.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.887+0000 7f5707000640 1 -- 172.21.15.40:0/513914338 shutdown_connections 2024-04-26T23:06:03.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:03.887+0000 7f5707000640 1 -- 172.21.15.40:0/513914338 wait complete. 2024-04-26T23:06:03.902 INFO:tasks.cephfs.filesystem:scrub status for tag:448789f1-f121-48b8-90f3-93a2f4a0d6a1 - {'path': '/d1/d2/d3', 'tag': '448789f1-f121-48b8-90f3-93a2f4a0d6a1', 'options': 'recursive,force'} 2024-04-26T23:06:08.989 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:06:08.988+0000 7f36ffc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-26T23:06:15.024 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:06:15.024+0000 7f36ffc00640 -1 log_channel(cluster) log [ERR] : Health check update: 3 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-26T23:06:18.362 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:06:18.374 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:06:33.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:06:33.903 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-04-26T23:06:34.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.132+0000 7f0e2b000640 1 Processor -- start 2024-04-26T23:06:34.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.133+0000 7f0e2b000640 1 -- start start 2024-04-26T23:06:34.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.134+0000 7f0e2b000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0e240cfaf0 0x7f0e24073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.134+0000 7f0e2b000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e2406dd40 0x7f0e24072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.134+0000 7f0e2b000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e24161880 con 0x7f0e24160a50 2024-04-26T23:06:34.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.134+0000 7f0e2b000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0e24071ce0 con 0x7f0e240cfaf0 2024-04-26T23:06:34.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.134+0000 7f0e2b000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0e240722a0 con 0x7f0e2406dd40 2024-04-26T23:06:34.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.135+0000 7f0e29200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0e240cfaf0 0x7f0e24073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.135+0000 7f0e29200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0e240cfaf0 0x7f0e24073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:34056/0 (socket says 172.21.15.40:34056) 2024-04-26T23:06:34.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.135+0000 7f0e29200640 1 -- 172.21.15.40:0/3732529320 learned_addr learned my addr 172.21.15.40:0/3732529320 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.135+0000 7f0e29c00640 1 --1- 172.21.15.40:0/3732529320 >> v1:172.21.15.103:6789/0 conn(0x7f0e24160a50 0x7f0e24160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:36744/0 (socket says 172.21.15.40:36744) 2024-04-26T23:06:34.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.135+0000 7f0e2a600640 1 --2- 172.21.15.40:0/3732529320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e2406dd40 0x7f0e24072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.136+0000 7f0e2a600640 1 -- 172.21.15.40:0/3732529320 >> v2:172.21.15.103:3300/0 conn(0x7f0e240cfaf0 msgr2=0x7f0e24073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.136+0000 7f0e2a600640 1 --2- 172.21.15.40:0/3732529320 >> v2:172.21.15.103:3300/0 conn(0x7f0e240cfaf0 0x7f0e24073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.136+0000 7f0e2a600640 1 -- 172.21.15.40:0/3732529320 >> v1:172.21.15.103:6789/0 conn(0x7f0e24160a50 legacy=0x7f0e24160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.136+0000 7f0e2a600640 1 -- 172.21.15.40:0/3732529320 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e240cff80 con 0x7f0e2406dd40 2024-04-26T23:06:34.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.136+0000 7f0e29200640 1 --2- 172.21.15.40:0/3732529320 >> v2:172.21.15.103:3300/0 conn(0x7f0e240cfaf0 0x7f0e24073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:06:34.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.137+0000 7f0e2a600640 1 --2- 172.21.15.40:0/3732529320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e2406dd40 0x7f0e24072730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009d90 tx=0x7f0e1c02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e2cc8898391ab08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.138+0000 7f0e23e00640 1 -- 172.21.15.40:0/3732529320 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c007740 con 0x7f0e2406dd40 2024-04-26T23:06:34.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.138+0000 7f0e23e00640 1 -- 172.21.15.40:0/3732529320 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e1c0078e0 con 0x7f0e2406dd40 2024-04-26T23:06:34.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.138+0000 7f0e23e00640 1 -- 172.21.15.40:0/3732529320 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c007cb0 con 0x7f0e2406dd40 2024-04-26T23:06:34.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.139+0000 7f0e2b000640 1 -- 172.21.15.40:0/3732529320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e2406dd40 msgr2=0x7f0e24072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.139+0000 7f0e2b000640 1 --2- 172.21.15.40:0/3732529320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e2406dd40 0x7f0e24072730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009d90 tx=0x7f0e1c02a660 comp rx=0 tx=0).stop 2024-04-26T23:06:34.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.139+0000 7f0e2b000640 1 -- 172.21.15.40:0/3732529320 shutdown_connections 2024-04-26T23:06:34.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.139+0000 7f0e2b000640 1 --2- 172.21.15.40:0/3732529320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e2406dd40 0x7f0e24072730 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.139+0000 7f0e2b000640 1 --2- 172.21.15.40:0/3732529320 >> v2:172.21.15.103:3300/0 conn(0x7f0e240cfaf0 0x7f0e24073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.139+0000 7f0e2b000640 1 -- 172.21.15.40:0/3732529320 >> 172.21.15.40:0/3732529320 conn(0x7f0e240b2720 msgr2=0x7f0e240b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.139+0000 7f0e2b000640 1 -- 172.21.15.40:0/3732529320 shutdown_connections 2024-04-26T23:06:34.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.139+0000 7f0e2b000640 1 -- 172.21.15.40:0/3732529320 wait complete. 2024-04-26T23:06:34.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.141+0000 7f0e2b000640 1 Processor -- start 2024-04-26T23:06:34.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.141+0000 7f0e2b000640 1 -- start start 2024-04-26T23:06:34.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.142+0000 7f0e2b000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2406dd40 0x7f0e241f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.142+0000 7f0e2b000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e24160a50 0x7f0e240b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.142+0000 7f0e2b000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e240b2040 0x7f0e24143e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.142+0000 7f0e2b000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0e24071f30 con 0x7f0e2406dd40 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.142+0000 7f0e2b000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0e24073f90 con 0x7f0e24160a50 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.142+0000 7f0e2b000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0e2406e1f0 con 0x7f0e240b2040 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.142+0000 7f0e29c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2406dd40 0x7f0e241f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.142+0000 7f0e29c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2406dd40 0x7f0e241f5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56826/0 (socket says 172.21.15.40:56826) 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.142+0000 7f0e29c00640 1 -- 172.21.15.40:0/898908731 learned_addr learned my addr 172.21.15.40:0/898908731 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.143+0000 7f0e2a600640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e240b2040 0x7f0e24143e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.143+0000 7f0e29200640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e24160a50 0x7f0e240b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.143+0000 7f0e29c00640 1 -- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e240b2040 msgr2=0x7f0e24143e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.143+0000 7f0e29c00640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e240b2040 0x7f0e24143e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.143+0000 7f0e29c00640 1 -- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e24160a50 msgr2=0x7f0e240b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.143+0000 7f0e29c00640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e24160a50 0x7f0e240b1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.143+0000 7f0e29c00640 1 -- 172.21.15.40:0/898908731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e24144b90 con 0x7f0e2406dd40 2024-04-26T23:06:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.143+0000 7f0e29200640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e24160a50 0x7f0e240b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.143+0000 7f0e29c00640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2406dd40 0x7f0e241f5ed0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800c430 tx=0x7f0e1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.143+0000 7f0e21600640 1 -- 172.21.15.40:0/898908731 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e180074e0 con 0x7f0e2406dd40 2024-04-26T23:06:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.143+0000 7f0e21600640 1 -- 172.21.15.40:0/898908731 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e18007680 con 0x7f0e2406dd40 2024-04-26T23:06:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.143+0000 7f0e21600640 1 -- 172.21.15.40:0/898908731 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e18004a00 con 0x7f0e2406dd40 2024-04-26T23:06:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.143+0000 7f0e2b000640 1 -- 172.21.15.40:0/898908731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e24144e70 con 0x7f0e2406dd40 2024-04-26T23:06:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.144+0000 7f0e2b000640 1 -- 172.21.15.40:0/898908731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e241f6410 con 0x7f0e2406dd40 2024-04-26T23:06:34.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.146+0000 7f0e21600640 1 -- 172.21.15.40:0/898908731 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0e1801a020 con 0x7f0e2406dd40 2024-04-26T23:06:34.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.148+0000 7f0e21600640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0e0406c920 0x7f0e0406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.148+0000 7f0e21600640 1 -- 172.21.15.40:0/898908731 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f0e1808b650 con 0x7f0e2406dd40 2024-04-26T23:06:34.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.148+0000 7f0e29200640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0e0406c920 0x7f0e0406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.148+0000 7f0e2f582740 1 Processor -- start 2024-04-26T23:06:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.149+0000 7f0e2f582740 1 -- start start 2024-04-26T23:06:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.150+0000 7f0e2f582740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc2ecc0 0x55893bc07320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.150+0000 7f0e2f582740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc2f0a0 0x55893bc31520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.150+0000 7f0e2f582740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 0x55893bc052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.150+0000 7f0e2f582740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55893bb17030 con 0x55893bc31a60 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.150+0000 7f0e2f582740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55893bb327f0 con 0x55893bc2f0a0 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.150+0000 7f0e2f582740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55893baed7e0 con 0x55893bc2ecc0 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.150+0000 7f0e29200640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0e0406c920 0x7f0e0406ede0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009630 tx=0x7f0e10008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.150+0000 7f0e2a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 0x55893bc052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.150+0000 7f0e2a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 0x55893bc052c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56830/0 (socket says 172.21.15.40:56830) 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.150+0000 7f0e2a600640 1 -- 172.21.15.40:0/3469396781 learned_addr learned my addr 172.21.15.40:0/3469396781 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.150+0000 7f0e29200640 1 --2- 172.21.15.40:0/3469396781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc2f0a0 0x55893bc31520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.150+0000 7f0e29c00640 1 --2- 172.21.15.40:0/3469396781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc2ecc0 0x55893bc07320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.151+0000 7f0e2a600640 1 -- 172.21.15.40:0/3469396781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc2ecc0 msgr2=0x55893bc07320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.151+0000 7f0e2a600640 1 --2- 172.21.15.40:0/3469396781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc2ecc0 0x55893bc07320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.151+0000 7f0e2a600640 1 -- 172.21.15.40:0/3469396781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc2f0a0 msgr2=0x55893bc31520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.151+0000 7f0e2a600640 1 --2- 172.21.15.40:0/3469396781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc2f0a0 0x55893bc31520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.151+0000 7f0e2a600640 1 -- 172.21.15.40:0/3469396781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55893baf9050 con 0x55893bc31a60 2024-04-26T23:06:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.151+0000 7f0e29c00640 1 --2- 172.21.15.40:0/3469396781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc2ecc0 0x55893bc07320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:34.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.151+0000 7f0e29200640 1 --2- 172.21.15.40:0/3469396781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc2f0a0 0x55893bc31520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:34.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.151+0000 7f0e2a600640 1 --2- 172.21.15.40:0/3469396781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 0x55893bc052c0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009e80 tx=0x7f0e1c02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c26738e90415bee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.151+0000 7f0e16000640 1 -- 172.21.15.40:0/3469396781 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c007880 con 0x55893bc31a60 2024-04-26T23:06:34.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.151+0000 7f0e16000640 1 -- 172.21.15.40:0/3469396781 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e1c007a20 con 0x55893bc31a60 2024-04-26T23:06:34.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.151+0000 7f0e16000640 1 -- 172.21.15.40:0/3469396781 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c007dd0 con 0x55893bc31a60 2024-04-26T23:06:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.152+0000 7f0e2f582740 1 -- 172.21.15.40:0/3469396781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 msgr2=0x55893bc052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.152+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3469396781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 0x55893bc052c0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009e80 tx=0x7f0e1c02f000 comp rx=0 tx=0).stop 2024-04-26T23:06:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.152+0000 7f0e2f582740 1 -- 172.21.15.40:0/3469396781 shutdown_connections 2024-04-26T23:06:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.152+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3469396781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 0x55893bc052c0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.152+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3469396781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc2f0a0 0x55893bc31520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.152+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3469396781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc2ecc0 0x55893bc07320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.152+0000 7f0e2f582740 1 -- 172.21.15.40:0/3469396781 >> 172.21.15.40:0/3469396781 conn(0x55893bc03d20 msgr2=0x55893bc4c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.152+0000 7f0e2f582740 1 -- 172.21.15.40:0/3469396781 shutdown_connections 2024-04-26T23:06:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.152+0000 7f0e2f582740 1 -- 172.21.15.40:0/3469396781 wait complete. 2024-04-26T23:06:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.153+0000 7f0e2f582740 1 -- start start 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.153+0000 7f0e2f582740 1 Processor -- start 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.154+0000 7f0e2f582740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc2f0a0 0x55893bc2d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.154+0000 7f0e2f582740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc31a60 0x55893bc2d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.154+0000 7f0e2f582740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc64090 0x55893bc644f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.154+0000 7f0e2f582740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55893bb6f200 con 0x55893bc2f0a0 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.154+0000 7f0e2f582740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55893bb69ee0 con 0x55893bc64090 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.154+0000 7f0e2f582740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55893bb76e70 con 0x55893bc31a60 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.154+0000 7f0e2f582740 10 client.0 _ll_register_callbacks cb 0x55893bb60270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.154+0000 7f0e29c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc2f0a0 0x55893bc2d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.155+0000 7f0e29c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc2f0a0 0x55893bc2d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56834/0 (socket says 172.21.15.40:56834) 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.155+0000 7f0e29c00640 1 -- 172.21.15.40:0/3092872475 learned_addr learned my addr 172.21.15.40:0/3092872475 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.155+0000 7f0e2a600640 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc64090 0x55893bc644f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.155+0000 7f0e29200640 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc31a60 0x55893bc2d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.155+0000 7f0e29c00640 1 -- 172.21.15.40:0/3092872475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc31a60 msgr2=0x55893bc2d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.155+0000 7f0e29c00640 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc31a60 0x55893bc2d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.155+0000 7f0e29c00640 1 -- 172.21.15.40:0/3092872475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc64090 msgr2=0x55893bc644f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.155+0000 7f0e29c00640 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc64090 0x55893bc644f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.155+0000 7f0e29c00640 1 -- 172.21.15.40:0/3092872475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55893b9467d0 con 0x55893bc2f0a0 2024-04-26T23:06:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.155+0000 7f0e29200640 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc31a60 0x55893bc2d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.155+0000 7f0e29c00640 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc2f0a0 0x55893bc2d390 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800c400 tx=0x7f0e180590e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5003d58517879e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.155+0000 7f0e0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.156+0000 7f0e0aa00640 1 -- 172.21.15.40:0/3092872475 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e18059340 con 0x55893bc2f0a0 2024-04-26T23:06:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.156+0000 7f0e0aa00640 1 -- 172.21.15.40:0/3092872475 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f0e180594e0 con 0x55893bc2f0a0 2024-04-26T23:06:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.156+0000 7f0e2f582740 1 -- 172.21.15.40:0/3092872475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e1c007a20 con 0x55893bc2f0a0 2024-04-26T23:06:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.156+0000 7f0e0aa00640 1 -- 172.21.15.40:0/3092872475 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f0e18061830 con 0x55893bc2f0a0 2024-04-26T23:06:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.156+0000 7f0e2f582740 10 client.7161 fetch_fsmap learned FSMap version 66 2024-04-26T23:06:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.156+0000 7f0e2f582740 1 -- 172.21.15.40:0/3092872475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) -- 0x55893b87c3f0 con 0x55893bc2f0a0 2024-04-26T23:06:34.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.156+0000 7f0e0aa00640 1 -- 172.21.15.40:0/3092872475 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e1c007a20 con 0x55893bc2f0a0 2024-04-26T23:06:34.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.156+0000 7f0e0aa00640 1 -- 172.21.15.40:0/3092872475 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 66) ==== 3550+0+0 (secure 0 0 0) 0x7f0e18004ba0 con 0x55893bc2f0a0 2024-04-26T23:06:34.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.156+0000 7f0e2f582740 10 client.7161 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T23:06:34.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.156+0000 7f0e2f582740 10 client.7161 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-26T23:06:34.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.156+0000 7f0e2f582740 20 client.7161 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:34.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.157+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55893bc1ebf0 0x55893bc18530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.157+0000 7f0e2f582740 4 client.7161 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.157+0000 7f0e2f582740 1 -- 172.21.15.40:0/3092872475 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55893bb787e0 con 0x55893bc1ebf0 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.157+0000 7f0e29200640 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55893bc1ebf0 0x55893bc18530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.157+0000 7f0e29200640 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55893bc1ebf0 0x55893bc18530 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f782f36e0e6f77d1 server_cookie=22baf295b42d46a3 in_seq=0 out_seq=0 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.157+0000 7f0e0aa00640 10 client.7161 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.158+0000 7f0e0aa00640 1 -- 172.21.15.40:0/3092872475 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55893bb787e0 con 0x55893bc1ebf0 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.158+0000 7f0e0aa00640 10 client.7161 handle_command_reply: tid=0 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.158+0000 7f0e2f582740 1 client.7161 shutdown 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.158+0000 7f0e2f582740 1 -- 172.21.15.40:0/3092872475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc2f0a0 msgr2=0x55893bc2d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.158+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc2f0a0 0x55893bc2d390 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800c400 tx=0x7f0e180590e0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.159+0000 7f0e2f582740 1 -- 172.21.15.40:0/3092872475 shutdown_connections 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.159+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55893bc1ebf0 0x55893bc18530 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.159+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc64090 0x55893bc644f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.159+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc31a60 0x55893bc2d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.159+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3092872475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc2f0a0 0x55893bc2d390 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.159+0000 7f0e0aa00640 0 client.7161 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.159+0000 7f0e2f582740 1 -- 172.21.15.40:0/3092872475 >> 172.21.15.40:0/3092872475 conn(0x55893bc2ecc0 msgr2=0x55893bc052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.159+0000 7f0e2f582740 1 -- 172.21.15.40:0/3092872475 shutdown_connections 2024-04-26T23:06:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.159+0000 7f0e2f582740 1 -- 172.21.15.40:0/3092872475 wait complete. 2024-04-26T23:06:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.159+0000 7f0e2f582740 20 client.7161 trim_cache size 0 max 16384 2024-04-26T23:06:34.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.178+0000 7f0e2f582740 1 Processor -- start 2024-04-26T23:06:34.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.178+0000 7f0e2f582740 1 -- start start 2024-04-26T23:06:34.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.179+0000 7f0e2f582740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc2f0a0 0x55893bc17b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.179+0000 7f0e2f582740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc31a60 0x55893bc18db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.179+0000 7f0e2f582740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc64090 0x55893bc054b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.179+0000 7f0e2f582740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0e241fdc30 con 0x55893bc64090 2024-04-26T23:06:34.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.179+0000 7f0e2f582740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0e240b5640 con 0x55893bc31a60 2024-04-26T23:06:34.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.179+0000 7f0e2f582740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0e2413a3a0 con 0x55893bc2f0a0 2024-04-26T23:06:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.180+0000 7f0e2a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc64090 0x55893bc054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.180+0000 7f0e2a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc64090 0x55893bc054b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56846/0 (socket says 172.21.15.40:56846) 2024-04-26T23:06:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.180+0000 7f0e2a600640 1 -- 172.21.15.40:0/3680161818 learned_addr learned my addr 172.21.15.40:0/3680161818 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.180+0000 7f0e29c00640 1 --2- 172.21.15.40:0/3680161818 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc2f0a0 0x55893bc17b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.180+0000 7f0e29200640 1 --2- 172.21.15.40:0/3680161818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc31a60 0x55893bc18db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.180+0000 7f0e2a600640 1 -- 172.21.15.40:0/3680161818 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc2f0a0 msgr2=0x55893bc17b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.180+0000 7f0e2a600640 1 --2- 172.21.15.40:0/3680161818 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc2f0a0 0x55893bc17b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.180+0000 7f0e2a600640 1 -- 172.21.15.40:0/3680161818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc31a60 msgr2=0x55893bc18db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.180+0000 7f0e2a600640 1 --2- 172.21.15.40:0/3680161818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc31a60 0x55893bc18db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.180+0000 7f0e2a600640 1 -- 172.21.15.40:0/3680161818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55893b83c0f0 con 0x55893bc64090 2024-04-26T23:06:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.180+0000 7f0e29c00640 1 --2- 172.21.15.40:0/3680161818 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc2f0a0 0x55893bc17b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.180+0000 7f0e2a600640 1 --2- 172.21.15.40:0/3680161818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc64090 0x55893bc054b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c003bd0 tx=0x7f0e1c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770c3ff12e44d2ab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.181+0000 7f0e15600640 1 -- 172.21.15.40:0/3680161818 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c009a30 con 0x55893bc64090 2024-04-26T23:06:34.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.181+0000 7f0e15600640 1 -- 172.21.15.40:0/3680161818 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e1c003db0 con 0x55893bc64090 2024-04-26T23:06:34.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.181+0000 7f0e15600640 1 -- 172.21.15.40:0/3680161818 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c0078a0 con 0x55893bc64090 2024-04-26T23:06:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.181+0000 7f0e2f582740 1 -- 172.21.15.40:0/3680161818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc64090 msgr2=0x55893bc054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.181+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3680161818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc64090 0x55893bc054b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c003bd0 tx=0x7f0e1c02e000 comp rx=0 tx=0).stop 2024-04-26T23:06:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.181+0000 7f0e2f582740 1 -- 172.21.15.40:0/3680161818 shutdown_connections 2024-04-26T23:06:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.181+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3680161818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc64090 0x55893bc054b0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.181+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3680161818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc31a60 0x55893bc18db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.181+0000 7f0e2f582740 1 --2- 172.21.15.40:0/3680161818 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc2f0a0 0x55893bc17b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.181+0000 7f0e2f582740 1 -- 172.21.15.40:0/3680161818 >> 172.21.15.40:0/3680161818 conn(0x55893bc2ecc0 msgr2=0x55893bc34df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.182+0000 7f0e2f582740 1 -- 172.21.15.40:0/3680161818 shutdown_connections 2024-04-26T23:06:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.182+0000 7f0e2f582740 1 -- 172.21.15.40:0/3680161818 wait complete. 2024-04-26T23:06:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.182+0000 7f0e2f582740 1 -- start start 2024-04-26T23:06:34.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.182+0000 7f0e2f582740 1 Processor -- start 2024-04-26T23:06:34.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.183+0000 7f0e2f582740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc2f0a0 0x55893bc17bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.183+0000 7f0e2f582740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 0x55893bcc4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.183+0000 7f0e2f582740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc64090 0x55893bc06650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.183+0000 7f0e2f582740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55893bb77fd0 con 0x55893bc31a60 2024-04-26T23:06:34.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.183+0000 7f0e2f582740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0e24139d50 con 0x55893bc2f0a0 2024-04-26T23:06:34.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.183+0000 7f0e2f582740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55893bc18210 con 0x55893bc64090 2024-04-26T23:06:34.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.183+0000 7f0e2f582740 10 client.0 _ll_register_callbacks cb 0x7f0e18058eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:34.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.184+0000 7f0e29200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 0x55893bcc4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.184+0000 7f0e29200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 0x55893bcc4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56852/0 (socket says 172.21.15.40:56852) 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.184+0000 7f0e29200640 1 -- 172.21.15.40:0/30274660 learned_addr learned my addr 172.21.15.40:0/30274660 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.184+0000 7f0e29200640 1 -- 172.21.15.40:0/30274660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc64090 msgr2=0x55893bc06650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.184+0000 7f0e29200640 1 --2- 172.21.15.40:0/30274660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc64090 0x55893bc06650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.184+0000 7f0e29200640 1 -- 172.21.15.40:0/30274660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc2f0a0 msgr2=0x55893bc17bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.184+0000 7f0e29200640 1 --2- 172.21.15.40:0/30274660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc2f0a0 0x55893bc17bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.184+0000 7f0e29200640 1 -- 172.21.15.40:0/30274660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55893bb32a30 con 0x55893bc31a60 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.184+0000 7f0e29200640 1 --2- 172.21.15.40:0/30274660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 0x55893bcc4ed0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f0e2415ca90 tx=0x7f0e100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8573abbcc5ebaf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.184+0000 7f0e0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e0aa00640 1 -- 172.21.15.40:0/30274660 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1000f040 con 0x55893bc31a60 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e0aa00640 1 -- 172.21.15.40:0/30274660 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f0e10025050 con 0x55893bc31a60 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e2f582740 1 -- 172.21.15.40:0/30274660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e1c003db0 con 0x55893bc31a60 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e0aa00640 1 -- 172.21.15.40:0/30274660 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f0e10033050 con 0x55893bc31a60 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e2f582740 10 client.7167 fetch_fsmap learned FSMap version 66 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e2f582740 1 -- 172.21.15.40:0/30274660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) -- 0x55893bccb5f0 con 0x55893bc31a60 2024-04-26T23:06:34.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e0aa00640 1 -- 172.21.15.40:0/30274660 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e1c003db0 con 0x55893bc31a60 2024-04-26T23:06:34.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e0aa00640 1 -- 172.21.15.40:0/30274660 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 66) ==== 3550+0+0 (secure 0 0 0) 0x7f0e10036020 con 0x55893bc31a60 2024-04-26T23:06:34.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e2f582740 10 client.7167 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T23:06:34.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e2f582740 10 client.7167 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-26T23:06:34.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e2f582740 20 client.7167 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:34.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e2f582740 1 --2- 172.21.15.40:0/30274660 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55893bccbb10 0x55893bc06b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e2f582740 4 client.7167 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:06:34.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.185+0000 7f0e2f582740 1 -- 172.21.15.40:0/30274660 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55893bb72650 con 0x55893bccbb10 2024-04-26T23:06:34.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.186+0000 7f0e29c00640 1 --2- 172.21.15.40:0/30274660 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55893bccbb10 0x55893bc06b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.186+0000 7f0e29c00640 1 --2- 172.21.15.40:0/30274660 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55893bccbb10 0x55893bc06b90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fdec628b5cca532 server_cookie=3f484baec9a131cc in_seq=0 out_seq=0 2024-04-26T23:06:34.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.186+0000 7f0e0aa00640 10 client.7167 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:34.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.186+0000 7f0e0aa00640 1 -- 172.21.15.40:0/30274660 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55893bb72650 con 0x55893bccbb10 2024-04-26T23:06:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.186+0000 7f0e0aa00640 10 client.7167 handle_command_reply: tid=0 2024-04-26T23:06:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.186+0000 7f0e2f582740 1 client.7167 shutdown 2024-04-26T23:06:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.187+0000 7f0e2f582740 1 -- 172.21.15.40:0/30274660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 msgr2=0x55893bcc4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.187+0000 7f0e2f582740 1 --2- 172.21.15.40:0/30274660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 0x55893bcc4ed0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f0e2415ca90 tx=0x7f0e100046c0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.187+0000 7f0e2f582740 1 -- 172.21.15.40:0/30274660 shutdown_connections 2024-04-26T23:06:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.187+0000 7f0e2f582740 1 --2- 172.21.15.40:0/30274660 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x55893bccbb10 0x55893bc06b90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.187+0000 7f0e2f582740 1 --2- 172.21.15.40:0/30274660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55893bc64090 0x55893bc06650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.187+0000 7f0e2f582740 1 --2- 172.21.15.40:0/30274660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55893bc31a60 0x55893bcc4ed0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.187+0000 7f0e2f582740 1 --2- 172.21.15.40:0/30274660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55893bc2f0a0 0x55893bc17bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.187+0000 7f0e0aa00640 0 client.7167 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.187+0000 7f0e2f582740 1 -- 172.21.15.40:0/30274660 >> 172.21.15.40:0/30274660 conn(0x55893bc2ecc0 msgr2=0x55893bc43f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.187+0000 7f0e2f582740 1 -- 172.21.15.40:0/30274660 shutdown_connections 2024-04-26T23:06:34.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.187+0000 7f0e2f582740 1 -- 172.21.15.40:0/30274660 wait complete. 2024-04-26T23:06:34.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.187+0000 7f0e2f582740 20 client.7167 trim_cache size 0 max 16384 2024-04-26T23:06:34.191 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:06:34.191 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-04-26T23:06:34.191 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-04-26T23:06:34.191 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:06:34.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.188+0000 7f0e2b000640 1 -- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0e0406c920 msgr2=0x7f0e0406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.188+0000 7f0e2b000640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0e0406c920 0x7f0e0406ede0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009630 tx=0x7f0e10008040 comp rx=0 tx=0).stop 2024-04-26T23:06:34.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.188+0000 7f0e2b000640 1 -- 172.21.15.40:0/898908731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2406dd40 msgr2=0x7f0e241f5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.188+0000 7f0e2b000640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2406dd40 0x7f0e241f5ed0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800c430 tx=0x7f0e1800c8f0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.188+0000 7f0e2b000640 1 -- 172.21.15.40:0/898908731 shutdown_connections 2024-04-26T23:06:34.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.188+0000 7f0e2b000640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0e0406c920 0x7f0e0406ede0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.188+0000 7f0e2b000640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e240b2040 0x7f0e24143e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.188+0000 7f0e2b000640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e24160a50 0x7f0e240b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.188+0000 7f0e2b000640 1 --2- 172.21.15.40:0/898908731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2406dd40 0x7f0e241f5ed0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.188+0000 7f0e2b000640 1 -- 172.21.15.40:0/898908731 >> 172.21.15.40:0/898908731 conn(0x7f0e240b2720 msgr2=0x7f0e2415e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.188+0000 7f0e2b000640 1 -- 172.21.15.40:0/898908731 shutdown_connections 2024-04-26T23:06:34.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.188+0000 7f0e2b000640 1 -- 172.21.15.40:0/898908731 wait complete. 2024-04-26T23:06:34.206 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:06:34.206 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 damage ls 2024-04-26T23:06:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.422+0000 7f7469400640 1 Processor -- start 2024-04-26T23:06:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.422+0000 7f7469400640 1 -- start start 2024-04-26T23:06:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.423+0000 7f7469400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7464071e60 0x7f746406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.423+0000 7f7469400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f746406fe20 0x7f7464072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.423+0000 7f7469400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746415d7c0 con 0x7f7464072780 2024-04-26T23:06:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.423+0000 7f7469400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f746415daa0 con 0x7f7464071e60 2024-04-26T23:06:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.423+0000 7f7469400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f746415dd80 con 0x7f746406fe20 2024-04-26T23:06:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.424+0000 7f7463200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7464072780 0x7f74640cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59278/0 (socket says 172.21.15.40:59278) 2024-04-26T23:06:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.424+0000 7f7462800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7464071e60 0x7f746406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.424+0000 7f7463200640 1 -- 172.21.15.40:0/167321373 learned_addr learned my addr 172.21.15.40:0/167321373 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.424+0000 7f7461e00640 1 --2- 172.21.15.40:0/167321373 >> v2:172.21.15.103:3300/0 conn(0x7f746406fe20 0x7f7464072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.424+0000 7f7462800640 1 -- 172.21.15.40:0/167321373 >> v2:172.21.15.103:3300/0 conn(0x7f746406fe20 msgr2=0x7f7464072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.424+0000 7f7462800640 1 --2- 172.21.15.40:0/167321373 >> v2:172.21.15.103:3300/0 conn(0x7f746406fe20 0x7f7464072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.424+0000 7f7462800640 1 -- 172.21.15.40:0/167321373 >> v1:172.21.15.40:6789/0 conn(0x7f7464072780 legacy=0x7f74640cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.424+0000 7f7462800640 1 -- 172.21.15.40:0/167321373 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746415e060 con 0x7f7464071e60 2024-04-26T23:06:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.426+0000 7f7462800640 1 --2- 172.21.15.40:0/167321373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7464071e60 0x7f746406f8e0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f74640b3820 tx=0x7f745402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3901134f0731bf9b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.426+0000 7f7461400640 1 -- 172.21.15.40:0/167321373 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7454007740 con 0x7f7464071e60 2024-04-26T23:06:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.426+0000 7f7461400640 1 -- 172.21.15.40:0/167321373 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74540078e0 con 0x7f7464071e60 2024-04-26T23:06:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.426+0000 7f7461400640 1 -- 172.21.15.40:0/167321373 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7454007cb0 con 0x7f7464071e60 2024-04-26T23:06:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.427+0000 7f7469400640 1 -- 172.21.15.40:0/167321373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7464071e60 msgr2=0x7f746406f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.427+0000 7f7469400640 1 --2- 172.21.15.40:0/167321373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7464071e60 0x7f746406f8e0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f74640b3820 tx=0x7f745402a680 comp rx=0 tx=0).stop 2024-04-26T23:06:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.427+0000 7f7469400640 1 -- 172.21.15.40:0/167321373 shutdown_connections 2024-04-26T23:06:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.427+0000 7f7469400640 1 --2- 172.21.15.40:0/167321373 >> v2:172.21.15.103:3300/0 conn(0x7f746406fe20 0x7f7464072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.427+0000 7f7469400640 1 --2- 172.21.15.40:0/167321373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7464071e60 0x7f746406f8e0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.427+0000 7f7469400640 1 -- 172.21.15.40:0/167321373 >> 172.21.15.40:0/167321373 conn(0x7f74640d0be0 msgr2=0x7f74640d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.427+0000 7f7469400640 1 -- 172.21.15.40:0/167321373 shutdown_connections 2024-04-26T23:06:34.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.427+0000 7f7469400640 1 -- 172.21.15.40:0/167321373 wait complete. 2024-04-26T23:06:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.429+0000 7f7469400640 1 Processor -- start 2024-04-26T23:06:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.429+0000 7f7469400640 1 -- start start 2024-04-26T23:06:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.430+0000 7f7469400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f746406fe20 0x7f7464150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.430+0000 7f7469400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7464071e60 0x7f74640c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.430+0000 7f7469400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7464072780 0x7f74640c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.430+0000 7f7469400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f746415d3f0 con 0x7f7464071e60 2024-04-26T23:06:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.430+0000 7f7469400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7464072bd0 con 0x7f746406fe20 2024-04-26T23:06:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.430+0000 7f7469400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f746415d570 con 0x7f7464072780 2024-04-26T23:06:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.430+0000 7f7461e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7464071e60 0x7f74640c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.430+0000 7f7461e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7464071e60 0x7f74640c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56876/0 (socket says 172.21.15.40:56876) 2024-04-26T23:06:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.430+0000 7f7461e00640 1 -- 172.21.15.40:0/2481057520 learned_addr learned my addr 172.21.15.40:0/2481057520 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.430+0000 7f7462800640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f746406fe20 0x7f7464150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.431+0000 7f7463200640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7464072780 0x7f74640c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.431+0000 7f7461e00640 1 -- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7464072780 msgr2=0x7f74640c9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.431+0000 7f7461e00640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7464072780 0x7f74640c9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.431+0000 7f7461e00640 1 -- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f746406fe20 msgr2=0x7f7464150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.431+0000 7f7461e00640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f746406fe20 0x7f7464150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.431+0000 7f7461e00640 1 -- 172.21.15.40:0/2481057520 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7464151490 con 0x7f7464071e60 2024-04-26T23:06:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.431+0000 7f7461e00640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7464071e60 0x7f74640c8d00 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f744c00d800 tx=0x7f744c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.431+0000 7f745aa00640 1 -- 172.21.15.40:0/2481057520 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c00b6c0 con 0x7f7464071e60 2024-04-26T23:06:34.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.431+0000 7f745aa00640 1 -- 172.21.15.40:0/2481057520 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f744c019070 con 0x7f7464071e60 2024-04-26T23:06:34.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.431+0000 7f745aa00640 1 -- 172.21.15.40:0/2481057520 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c00ba80 con 0x7f7464071e60 2024-04-26T23:06:34.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.431+0000 7f7469400640 1 -- 172.21.15.40:0/2481057520 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74640d0550 con 0x7f7464071e60 2024-04-26T23:06:34.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.432+0000 7f7469400640 1 -- 172.21.15.40:0/2481057520 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74640af760 con 0x7f7464071e60 2024-04-26T23:06:34.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.434+0000 7f745aa00640 1 -- 172.21.15.40:0/2481057520 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f744c00bc20 con 0x7f7464071e60 2024-04-26T23:06:34.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.435+0000 7f745aa00640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f744006c8f0 0x7f744006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.436+0000 7f745aa00640 1 -- 172.21.15.40:0/2481057520 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f744c08c200 con 0x7f7464071e60 2024-04-26T23:06:34.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.436+0000 7f746c1f8740 1 Processor -- start 2024-04-26T23:06:34.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.436+0000 7f7462800640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f744006c8f0 0x7f744006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.436+0000 7f746c1f8740 1 -- start start 2024-04-26T23:06:34.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.437+0000 7f746c1f8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c00b00 0x5611a7c1c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.437+0000 7f746c1f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c1eb00 0x5611a7c00ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.437+0000 7f746c1f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 0x5611a7c120c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.437+0000 7f746c1f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5611a7b39c70 con 0x5611a7c11c80 2024-04-26T23:06:34.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.437+0000 7f746c1f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5611a7af2980 con 0x5611a7c1eb00 2024-04-26T23:06:34.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.437+0000 7f746c1f8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5611a7b35c30 con 0x5611a7c00b00 2024-04-26T23:06:34.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.437+0000 7f7462800640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f744006c8f0 0x7f744006edb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7454034000 tx=0x7f745402f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.437+0000 7f7463200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 0x5611a7c120c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.437+0000 7f7463200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 0x5611a7c120c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56890/0 (socket says 172.21.15.40:56890) 2024-04-26T23:06:34.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.437+0000 7f7463200640 1 -- 172.21.15.40:0/2708236438 learned_addr learned my addr 172.21.15.40:0/2708236438 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.438+0000 7f7461e00640 1 --2- 172.21.15.40:0/2708236438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c1eb00 0x5611a7c00ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.438+0000 7f7462800640 1 --2- 172.21.15.40:0/2708236438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c00b00 0x5611a7c1c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.438+0000 7f7463200640 1 -- 172.21.15.40:0/2708236438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c00b00 msgr2=0x5611a7c1c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.438+0000 7f7463200640 1 --2- 172.21.15.40:0/2708236438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c00b00 0x5611a7c1c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.438+0000 7f7463200640 1 -- 172.21.15.40:0/2708236438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c1eb00 msgr2=0x5611a7c00ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.438+0000 7f7463200640 1 --2- 172.21.15.40:0/2708236438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c1eb00 0x5611a7c00ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.438+0000 7f7463200640 1 -- 172.21.15.40:0/2708236438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611a7aca7b0 con 0x5611a7c11c80 2024-04-26T23:06:34.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.438+0000 7f7463200640 1 --2- 172.21.15.40:0/2708236438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 0x5611a7c120c0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f745c001520 tx=0x7f745c0019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53cb389336b331b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.438+0000 7f7453000640 1 -- 172.21.15.40:0/2708236438 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c019070 con 0x5611a7c11c80 2024-04-26T23:06:34.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.439+0000 7f7453000640 1 -- 172.21.15.40:0/2708236438 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f745c005ac0 con 0x5611a7c11c80 2024-04-26T23:06:34.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.439+0000 7f7453000640 1 -- 172.21.15.40:0/2708236438 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c0070d0 con 0x5611a7c11c80 2024-04-26T23:06:34.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.439+0000 7f746c1f8740 1 -- 172.21.15.40:0/2708236438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 msgr2=0x5611a7c120c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.439+0000 7f746c1f8740 1 --2- 172.21.15.40:0/2708236438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 0x5611a7c120c0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f745c001520 tx=0x7f745c0019e0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.439+0000 7f746c1f8740 1 -- 172.21.15.40:0/2708236438 shutdown_connections 2024-04-26T23:06:34.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.439+0000 7f746c1f8740 1 --2- 172.21.15.40:0/2708236438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 0x5611a7c120c0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.439+0000 7f746c1f8740 1 --2- 172.21.15.40:0/2708236438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c1eb00 0x5611a7c00ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.439+0000 7f746c1f8740 1 --2- 172.21.15.40:0/2708236438 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c00b00 0x5611a7c1c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.439+0000 7f746c1f8740 1 -- 172.21.15.40:0/2708236438 >> 172.21.15.40:0/2708236438 conn(0x5611a7bd31c0 msgr2=0x5611a7c1bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.439+0000 7f746c1f8740 1 -- 172.21.15.40:0/2708236438 shutdown_connections 2024-04-26T23:06:34.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.439+0000 7f746c1f8740 1 -- 172.21.15.40:0/2708236438 wait complete. 2024-04-26T23:06:34.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.440+0000 7f746c1f8740 1 -- start start 2024-04-26T23:06:34.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.440+0000 7f746c1f8740 1 Processor -- start 2024-04-26T23:06:34.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.441+0000 7f746c1f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c11c80 0x5611a7c294f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.441+0000 7f746c1f8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c1eb00 0x5611a7bfc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.441+0000 7f746c1f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 0x5611a7bfe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.441+0000 7f746c1f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5611a7b38f80 con 0x5611a7bfd460 2024-04-26T23:06:34.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.441+0000 7f746c1f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5611a7ae6260 con 0x5611a7c11c80 2024-04-26T23:06:34.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.441+0000 7f746c1f8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5611a7b3e2a0 con 0x5611a7c1eb00 2024-04-26T23:06:34.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.441+0000 7f746c1f8740 10 client.0 _ll_register_callbacks cb 0x5611a7b30960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:34.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.442+0000 7f7463200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 0x5611a7bfe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.442+0000 7f7463200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 0x5611a7bfe7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56896/0 (socket says 172.21.15.40:56896) 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.442+0000 7f7463200640 1 -- 172.21.15.40:0/7861054 learned_addr learned my addr 172.21.15.40:0/7861054 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.442+0000 7f7462800640 1 --2- 172.21.15.40:0/7861054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c11c80 0x5611a7c294f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.442+0000 7f7461e00640 1 --2- 172.21.15.40:0/7861054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c1eb00 0x5611a7bfc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.442+0000 7f7463200640 1 -- 172.21.15.40:0/7861054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c1eb00 msgr2=0x5611a7bfc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.442+0000 7f7463200640 1 --2- 172.21.15.40:0/7861054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c1eb00 0x5611a7bfc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.442+0000 7f7463200640 1 -- 172.21.15.40:0/7861054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c11c80 msgr2=0x5611a7c294f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.442+0000 7f7463200640 1 --2- 172.21.15.40:0/7861054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c11c80 0x5611a7c294f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.442+0000 7f7463200640 1 -- 172.21.15.40:0/7861054 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611a7ac81c0 con 0x5611a7bfd460 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.442+0000 7f7463200640 1 --2- 172.21.15.40:0/7861054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 0x5611a7bfe7f0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f745c005880 tx=0x7f745c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd74ec0638d6cc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.442+0000 7f7447400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.443+0000 7f7447400640 1 -- 172.21.15.40:0/7861054 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c022060 con 0x5611a7bfd460 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.443+0000 7f7447400640 1 -- 172.21.15.40:0/7861054 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f745c007760 con 0x5611a7bfd460 2024-04-26T23:06:34.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.443+0000 7f746c1f8740 1 -- 172.21.15.40:0/7861054 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f745c005ac0 con 0x5611a7bfd460 2024-04-26T23:06:34.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.443+0000 7f7447400640 1 -- 172.21.15.40:0/7861054 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f745c028050 con 0x5611a7bfd460 2024-04-26T23:06:34.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.443+0000 7f746c1f8740 10 client.7182 fetch_fsmap learned FSMap version 66 2024-04-26T23:06:34.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.443+0000 7f746c1f8740 1 -- 172.21.15.40:0/7861054 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) -- 0x5611a784b3f0 con 0x5611a7bfd460 2024-04-26T23:06:34.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.443+0000 7f7447400640 1 -- 172.21.15.40:0/7861054 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f745c005ac0 con 0x5611a7bfd460 2024-04-26T23:06:34.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.444+0000 7f7447400640 1 -- 172.21.15.40:0/7861054 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 66) ==== 3550+0+0 (secure 0 0 0) 0x7f745c02b020 con 0x5611a7bfd460 2024-04-26T23:06:34.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.444+0000 7f746c1f8740 10 client.7182 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T23:06:34.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.444+0000 7f746c1f8740 10 client.7182 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-26T23:06:34.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.444+0000 7f746c1f8740 20 client.7182 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:34.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.444+0000 7f746c1f8740 1 --2- 172.21.15.40:0/7861054 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x5611a7bfed30 0x5611a7c043e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.444+0000 7f746c1f8740 4 client.7182 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:06:34.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.444+0000 7f746c1f8740 1 -- 172.21.15.40:0/7861054 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611a7b47880 con 0x5611a7bfed30 2024-04-26T23:06:34.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.444+0000 7f7462800640 1 --2- 172.21.15.40:0/7861054 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x5611a7bfed30 0x5611a7c043e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.445+0000 7f7462800640 1 --2- 172.21.15.40:0/7861054 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x5611a7bfed30 0x5611a7c043e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4425b9cd80fd2a67 server_cookie=1c36fc484b01fc3 in_seq=0 out_seq=0 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.445+0000 7f7447400640 10 client.7182 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.445+0000 7f7447400640 1 -- 172.21.15.40:0/7861054 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5611a7b47880 con 0x5611a7bfed30 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.445+0000 7f7447400640 10 client.7182 handle_command_reply: tid=0 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.445+0000 7f746c1f8740 1 client.7182 shutdown 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.446+0000 7f746c1f8740 1 -- 172.21.15.40:0/7861054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 msgr2=0x5611a7bfe7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.446+0000 7f746c1f8740 1 --2- 172.21.15.40:0/7861054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 0x5611a7bfe7f0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f745c005880 tx=0x7f745c000ed0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.446+0000 7f746c1f8740 1 -- 172.21.15.40:0/7861054 shutdown_connections 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.446+0000 7f746c1f8740 1 --2- 172.21.15.40:0/7861054 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x5611a7bfed30 0x5611a7c043e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.446+0000 7f746c1f8740 1 --2- 172.21.15.40:0/7861054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 0x5611a7bfe7f0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.446+0000 7f746c1f8740 1 --2- 172.21.15.40:0/7861054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c1eb00 0x5611a7bfc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.446+0000 7f746c1f8740 1 --2- 172.21.15.40:0/7861054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c11c80 0x5611a7c294f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.446+0000 7f7447400640 0 client.7182 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:34.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.446+0000 7f746c1f8740 1 -- 172.21.15.40:0/7861054 >> 172.21.15.40:0/7861054 conn(0x5611a7c00b00 msgr2=0x5611a7bd0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.446+0000 7f746c1f8740 1 -- 172.21.15.40:0/7861054 shutdown_connections 2024-04-26T23:06:34.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.446+0000 7f746c1f8740 1 -- 172.21.15.40:0/7861054 wait complete. 2024-04-26T23:06:34.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.446+0000 7f746c1f8740 20 client.7182 trim_cache size 0 max 16384 2024-04-26T23:06:34.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.461+0000 7f746c1f8740 1 Processor -- start 2024-04-26T23:06:34.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.461+0000 7f746c1f8740 1 -- start start 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.462+0000 7f746c1f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7bfd460 0x5611a7c19c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.462+0000 7f746c1f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 0x5611a7c1a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.462+0000 7f746c1f8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c1eb00 0x5611a7bd6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.462+0000 7f746c1f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f74640acf70 con 0x5611a7c11c80 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.462+0000 7f746c1f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f74640ac110 con 0x5611a7bfd460 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.462+0000 7f746c1f8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f74640ac840 con 0x5611a7c1eb00 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.462+0000 7f7461e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 0x5611a7c1a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.462+0000 7f7461e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 0x5611a7c1a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56910/0 (socket says 172.21.15.40:56910) 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.462+0000 7f7461e00640 1 -- 172.21.15.40:0/944924286 learned_addr learned my addr 172.21.15.40:0/944924286 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.463+0000 7f7462800640 1 --2- 172.21.15.40:0/944924286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7bfd460 0x5611a7c19c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.463+0000 7f7461e00640 1 -- 172.21.15.40:0/944924286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c1eb00 msgr2=0x5611a7bd6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.463+0000 7f7461e00640 1 --2- 172.21.15.40:0/944924286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c1eb00 0x5611a7bd6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.463+0000 7f7461e00640 1 -- 172.21.15.40:0/944924286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7bfd460 msgr2=0x5611a7c19c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.463+0000 7f7461e00640 1 --2- 172.21.15.40:0/944924286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7bfd460 0x5611a7c19c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.463+0000 7f7461e00640 1 -- 172.21.15.40:0/944924286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611a79157d0 con 0x5611a7c11c80 2024-04-26T23:06:34.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.463+0000 7f7461e00640 1 --2- 172.21.15.40:0/944924286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 0x5611a7c1a1b0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f744c00b4b0 tx=0x7f744c059680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c109fc1fe8ec21b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.463+0000 7f7452600640 1 -- 172.21.15.40:0/944924286 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c088070 con 0x5611a7c11c80 2024-04-26T23:06:34.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.463+0000 7f7452600640 1 -- 172.21.15.40:0/944924286 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f744c059d20 con 0x5611a7c11c80 2024-04-26T23:06:34.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.463+0000 7f7452600640 1 -- 172.21.15.40:0/944924286 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c072070 con 0x5611a7c11c80 2024-04-26T23:06:34.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.464+0000 7f746c1f8740 1 -- 172.21.15.40:0/944924286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 msgr2=0x5611a7c1a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.464+0000 7f746c1f8740 1 --2- 172.21.15.40:0/944924286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 0x5611a7c1a1b0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f744c00b4b0 tx=0x7f744c059680 comp rx=0 tx=0).stop 2024-04-26T23:06:34.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.464+0000 7f746c1f8740 1 -- 172.21.15.40:0/944924286 shutdown_connections 2024-04-26T23:06:34.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.464+0000 7f746c1f8740 1 --2- 172.21.15.40:0/944924286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c1eb00 0x5611a7bd6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.464+0000 7f746c1f8740 1 --2- 172.21.15.40:0/944924286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7c11c80 0x5611a7c1a1b0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.464+0000 7f746c1f8740 1 --2- 172.21.15.40:0/944924286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7bfd460 0x5611a7c19c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.464+0000 7f746c1f8740 1 -- 172.21.15.40:0/944924286 >> 172.21.15.40:0/944924286 conn(0x5611a7c00b00 msgr2=0x5611a7bfc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.464+0000 7f746c1f8740 1 -- 172.21.15.40:0/944924286 shutdown_connections 2024-04-26T23:06:34.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.464+0000 7f746c1f8740 1 -- 172.21.15.40:0/944924286 wait complete. 2024-04-26T23:06:34.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.465+0000 7f746c1f8740 1 -- start start 2024-04-26T23:06:34.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.465+0000 7f746c1f8740 1 Processor -- start 2024-04-26T23:06:34.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.466+0000 7f746c1f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 0x5611a7ca02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.466+0000 7f746c1f8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c11c80 0x5611a7c19b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.466+0000 7f746c1f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c1eb00 0x5611a7c1a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.466+0000 7f746c1f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f74640cfef0 con 0x5611a7bfd460 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.466+0000 7f746c1f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5611a7c1c680 con 0x5611a7c1eb00 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.466+0000 7f746c1f8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5611a7c45880 con 0x5611a7c11c80 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.466+0000 7f746c1f8740 10 client.0 _ll_register_callbacks cb 0x7f745c017070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.466+0000 7f7462800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 0x5611a7ca02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.466+0000 7f7462800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 0x5611a7ca02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56920/0 (socket says 172.21.15.40:56920) 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.466+0000 7f7462800640 1 -- 172.21.15.40:0/2336922021 learned_addr learned my addr 172.21.15.40:0/2336922021 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.467+0000 7f7461e00640 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c11c80 0x5611a7c19b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.467+0000 7f7463200640 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c1eb00 0x5611a7c1a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.467+0000 7f7462800640 1 -- 172.21.15.40:0/2336922021 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c11c80 msgr2=0x5611a7c19b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.467+0000 7f7462800640 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c11c80 0x5611a7c19b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.467+0000 7f7462800640 1 -- 172.21.15.40:0/2336922021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c1eb00 msgr2=0x5611a7c1a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.467+0000 7f7462800640 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c1eb00 0x5611a7c1a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.467+0000 7f7462800640 1 -- 172.21.15.40:0/2336922021 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611a7ae02a0 con 0x5611a7bfd460 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.467+0000 7f7463200640 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c1eb00 0x5611a7c1a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.467+0000 7f7462800640 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 0x5611a7ca02e0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f745402ac70 tx=0x7f7454004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97cc3fb765d24a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.467+0000 7f7447400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.467+0000 7f7447400640 1 -- 172.21.15.40:0/2336922021 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7454053800 con 0x5611a7bfd460 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.467+0000 7f7447400640 1 -- 172.21.15.40:0/2336922021 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f745403d430 con 0x5611a7bfd460 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.467+0000 7f746c1f8740 1 -- 172.21.15.40:0/2336922021 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f744c059d20 con 0x5611a7bfd460 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.468+0000 7f7447400640 1 -- 172.21.15.40:0/2336922021 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f745405c050 con 0x5611a7bfd460 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.468+0000 7f746c1f8740 10 client.7188 fetch_fsmap learned FSMap version 66 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.468+0000 7f746c1f8740 1 -- 172.21.15.40:0/2336922021 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) -- 0x5611a7c45d10 con 0x5611a7bfd460 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.468+0000 7f7447400640 1 -- 172.21.15.40:0/2336922021 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f744c059d20 con 0x5611a7bfd460 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.468+0000 7f7447400640 1 -- 172.21.15.40:0/2336922021 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 66) ==== 3550+0+0 (secure 0 0 0) 0x7f745403d630 con 0x5611a7bfd460 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.468+0000 7f746c1f8740 10 client.7188 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.468+0000 7f746c1f8740 10 client.7188 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-26T23:06:34.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.468+0000 7f746c1f8740 20 client.7188 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:34.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.468+0000 7f746c1f8740 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x5611a7c291d0 0x5611a7c1b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.468+0000 7f746c1f8740 4 client.7188 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-26T23:06:34.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.468+0000 7f746c1f8740 1 -- 172.21.15.40:0/2336922021 --> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5611a7b416f0 con 0x5611a7c291d0 2024-04-26T23:06:34.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.468+0000 7f7461e00640 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x5611a7c291d0 0x5611a7c1b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.469+0000 7f7461e00640 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x5611a7c291d0 0x5611a7c1b760 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=529561bcd03352f1 server_cookie=d4ea35956522689a in_seq=0 out_seq=0 2024-04-26T23:06:34.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.469+0000 7f7447400640 10 client.7188 ms_handle_connect on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:34.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.469+0000 7f7447400640 1 -- 172.21.15.40:0/2336922021 <== mds.0 v2:172.21.15.40:6833/931483025 1 ==== command_reply(tid 0: 0 ) ==== 8+0+270 (crc 0 0 0) 0x5611a7b416f0 con 0x5611a7c291d0 2024-04-26T23:06:34.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.469+0000 7f7447400640 10 client.7188 handle_command_reply: tid=0 2024-04-26T23:06:34.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.469+0000 7f746c1f8740 1 client.7188 shutdown 2024-04-26T23:06:34.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.469+0000 7f746c1f8740 1 -- 172.21.15.40:0/2336922021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 msgr2=0x5611a7ca02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.469+0000 7f746c1f8740 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 0x5611a7ca02e0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f745402ac70 tx=0x7f7454004ab0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f746c1f8740 1 -- 172.21.15.40:0/2336922021 shutdown_connections 2024-04-26T23:06:34.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f746c1f8740 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.40:6833/931483025,v1:172.21.15.40:6835/931483025] conn(0x5611a7c291d0 0x5611a7c1b760 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f746c1f8740 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611a7c1eb00 0x5611a7c1a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f746c1f8740 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5611a7c11c80 0x5611a7c19b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f746c1f8740 1 --2- 172.21.15.40:0/2336922021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611a7bfd460 0x5611a7ca02e0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f7447400640 0 client.7188 ms_handle_reset on v2:172.21.15.40:6833/931483025 2024-04-26T23:06:34.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f746c1f8740 1 -- 172.21.15.40:0/2336922021 >> 172.21.15.40:0/2336922021 conn(0x5611a7c00b00 msgr2=0x5611a7c27f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f746c1f8740 1 -- 172.21.15.40:0/2336922021 shutdown_connections 2024-04-26T23:06:34.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f746c1f8740 1 -- 172.21.15.40:0/2336922021 wait complete. 2024-04-26T23:06:34.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f746c1f8740 20 client.7188 trim_cache size 0 max 16384 2024-04-26T23:06:34.473 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-04-26T23:06:34.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:34.473 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-26T23:06:34.474 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 264882856, 2024-04-26T23:06:34.474 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511627779, 2024-04-26T23:06:34.474 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3/d4" 2024-04-26T23:06:34.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:34.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:34.474 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-26T23:06:34.474 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 983120154, 2024-04-26T23:06:34.474 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511627778, 2024-04-26T23:06:34.474 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3" 2024-04-26T23:06:34.474 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:34.474 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-04-26T23:06:34.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f7469400640 1 -- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f744006c8f0 msgr2=0x7f744006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f7469400640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f744006c8f0 0x7f744006edb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7454034000 tx=0x7f745402f000 comp rx=0 tx=0).stop 2024-04-26T23:06:34.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f7469400640 1 -- 172.21.15.40:0/2481057520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7464071e60 msgr2=0x7f74640c8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.470+0000 7f7469400640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7464071e60 0x7f74640c8d00 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f744c00d800 tx=0x7f744c00dcc0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.471+0000 7f7469400640 1 -- 172.21.15.40:0/2481057520 shutdown_connections 2024-04-26T23:06:34.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.471+0000 7f7469400640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f744006c8f0 0x7f744006edb0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.471+0000 7f7469400640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7464072780 0x7f74640c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.471+0000 7f7469400640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7464071e60 0x7f74640c8d00 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.471+0000 7f7469400640 1 --2- 172.21.15.40:0/2481057520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f746406fe20 0x7f7464150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.471+0000 7f7469400640 1 -- 172.21.15.40:0/2481057520 >> 172.21.15.40:0/2481057520 conn(0x7f74640d0be0 msgr2=0x7f7464072d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.471+0000 7f7469400640 1 -- 172.21.15.40:0/2481057520 shutdown_connections 2024-04-26T23:06:34.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.471+0000 7f7469400640 1 -- 172.21.15.40:0/2481057520 wait complete. 2024-04-26T23:06:34.489 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:1 damage ls 2024-04-26T23:06:34.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.691+0000 7f3889000640 1 Processor -- start 2024-04-26T23:06:34.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.691+0000 7f3889000640 1 -- start start 2024-04-26T23:06:34.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.692+0000 7f3889000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f388415f1a0 0x7f388406d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.692+0000 7f3889000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38840d18d0 0x7f388406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.692+0000 7f3889000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38840d39f0 con 0x7f388406e210 2024-04-26T23:06:34.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.692+0000 7f3889000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f388406db30 con 0x7f388415f1a0 2024-04-26T23:06:34.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.692+0000 7f3889000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f38840cd870 con 0x7f38840d18d0 2024-04-26T23:06:34.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.692+0000 7f3883e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f388406e210 0x7f38840cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59282/0 (socket says 172.21.15.40:59282) 2024-04-26T23:06:34.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.692+0000 7f3883e00640 1 -- 172.21.15.40:0/25291641 learned_addr learned my addr 172.21.15.40:0/25291641 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:34.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3882a00640 1 --2- 172.21.15.40:0/25291641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38840d18d0 0x7f388406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3883400640 1 --2- 172.21.15.40:0/25291641 >> v2:172.21.15.103:3300/0 conn(0x7f388415f1a0 0x7f388406d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2378897658 0 0) 0x7f38840d39f0 con 0x7f388406e210 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3868003520 con 0x7f388406e210 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3392408968 0 0) 0x7f3868003520 con 0x7f388406e210 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38840d18d0 msgr2=0x7f388406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3882000640 1 --2- 172.21.15.40:0/25291641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38840d18d0 0x7f388406dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 >> v2:172.21.15.103:3300/0 conn(0x7f388415f1a0 msgr2=0x7f388406d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3882000640 1 --2- 172.21.15.40:0/25291641 >> v2:172.21.15.103:3300/0 conn(0x7f388415f1a0 0x7f388406d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38840d1e20 con 0x7f388406e210 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f387c001670 con 0x7f388406e210 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 >> v1:172.21.15.40:6789/0 conn(0x7f388406e210 legacy=0x7f38840cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3882000640 1 --2- 172.21.15.40:0/25291641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38680039b0 0x7f3868003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.693+0000 7f3882000640 1 --2- 172.21.15.40:0/25291641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38840d18d0 0x7f3868004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.694+0000 7f3882000640 1 --2- 172.21.15.40:0/25291641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3868004d40 0x7f38680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.694+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f388406db30 con 0x7f3868004d40 2024-04-26T23:06:34.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.694+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f38840cd870 con 0x7f38680039b0 2024-04-26T23:06:34.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.694+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f386800b9c0 con 0x7f38840d18d0 2024-04-26T23:06:34.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.694+0000 7f3883400640 1 --2- 172.21.15.40:0/25291641 >> v2:172.21.15.103:3300/0 conn(0x7f388415f1a0 0x7f388406d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:06:34.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.694+0000 7f3883e00640 1 --2- 172.21.15.40:0/25291641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3868004d40 0x7f38680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.694+0000 7f3883e00640 1 -- 172.21.15.40:0/25291641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38840d18d0 msgr2=0x7f3868004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.694+0000 7f3883e00640 1 --2- 172.21.15.40:0/25291641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38840d18d0 0x7f3868004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.694+0000 7f3883e00640 1 -- 172.21.15.40:0/25291641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38680039b0 msgr2=0x7f3868003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.694+0000 7f3883e00640 1 --2- 172.21.15.40:0/25291641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38680039b0 0x7f3868003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.694+0000 7f3883e00640 1 -- 172.21.15.40:0/25291641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3868003520 con 0x7f3868004d40 2024-04-26T23:06:34.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.695+0000 7f3883e00640 1 --2- 172.21.15.40:0/25291641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3868004d40 0x7f38680051c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f387c00c2c0 tx=0x7f3868005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b50bd9d369cd4a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.696+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c007d50 con 0x7f3868004d40 2024-04-26T23:06:34.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.696+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f387c02e070 con 0x7f3868004d40 2024-04-26T23:06:34.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.696+0000 7f3882000640 1 -- 172.21.15.40:0/25291641 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c005d30 con 0x7f3868004d40 2024-04-26T23:06:34.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.696+0000 7f3889000640 1 -- 172.21.15.40:0/25291641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3868004d40 msgr2=0x7f38680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.696+0000 7f3889000640 1 --2- 172.21.15.40:0/25291641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3868004d40 0x7f38680051c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f387c00c2c0 tx=0x7f3868005830 comp rx=0 tx=0).stop 2024-04-26T23:06:34.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.696+0000 7f3883e00640 1 -- 172.21.15.40:0/25291641 reap_dead start 2024-04-26T23:06:34.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.696+0000 7f3889000640 1 -- 172.21.15.40:0/25291641 shutdown_connections 2024-04-26T23:06:34.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.696+0000 7f3889000640 1 -- 172.21.15.40:0/25291641 >> 172.21.15.40:0/25291641 conn(0x7f38840d2bd0 msgr2=0x7f38840d2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.696+0000 7f3889000640 1 -- 172.21.15.40:0/25291641 shutdown_connections 2024-04-26T23:06:34.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.696+0000 7f3889000640 1 -- 172.21.15.40:0/25291641 wait complete. 2024-04-26T23:06:34.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.698+0000 7f3889000640 1 Processor -- start 2024-04-26T23:06:34.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.698+0000 7f3889000640 1 -- start start 2024-04-26T23:06:34.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.699+0000 7f3889000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3868004d40 0x7f38840ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.699+0000 7f3889000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38840cfbf0 0x7f3884150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.699+0000 7f3889000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38840cbe70 0x7f38840cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.699+0000 7f3889000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f38840cd500 con 0x7f3868004d40 2024-04-26T23:06:34.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.699+0000 7f3889000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f38840cd220 con 0x7f38840cfbf0 2024-04-26T23:06:34.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.699+0000 7f3889000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f388415f580 con 0x7f38840cbe70 2024-04-26T23:06:34.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.700+0000 7f3883400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3868004d40 0x7f38840ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.700+0000 7f3883400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3868004d40 0x7f38840ccb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56942/0 (socket says 172.21.15.40:56942) 2024-04-26T23:06:34.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.700+0000 7f3883400640 1 -- 172.21.15.40:0/76097102 learned_addr learned my addr 172.21.15.40:0/76097102 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.700+0000 7f3882a00640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38840cfbf0 0x7f3884150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.700+0000 7f3883e00640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38840cbe70 0x7f38840cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.700+0000 7f3883400640 1 -- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38840cbe70 msgr2=0x7f38840cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.700+0000 7f3883400640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38840cbe70 0x7f38840cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.700+0000 7f3883400640 1 -- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38840cfbf0 msgr2=0x7f3884150fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.700+0000 7f3883400640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38840cfbf0 0x7f3884150fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.700+0000 7f3883400640 1 -- 172.21.15.40:0/76097102 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38840d0390 con 0x7f3868004d40 2024-04-26T23:06:34.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.700+0000 7f3883e00640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38840cbe70 0x7f38840cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:34.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.700+0000 7f3883400640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3868004d40 0x7f38840ccb30 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f3874007f90 tx=0x7f387400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.701+0000 7f387b400640 1 -- 172.21.15.40:0/76097102 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387401a070 con 0x7f3868004d40 2024-04-26T23:06:34.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.701+0000 7f387b400640 1 -- 172.21.15.40:0/76097102 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3874003bb0 con 0x7f3868004d40 2024-04-26T23:06:34.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.701+0000 7f387b400640 1 -- 172.21.15.40:0/76097102 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874004030 con 0x7f3868004d40 2024-04-26T23:06:34.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.701+0000 7f3889000640 1 -- 172.21.15.40:0/76097102 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38840d00b0 con 0x7f3868004d40 2024-04-26T23:06:34.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.701+0000 7f3889000640 1 -- 172.21.15.40:0/76097102 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38841585f0 con 0x7f3868004d40 2024-04-26T23:06:34.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.704+0000 7f387b400640 1 -- 172.21.15.40:0/76097102 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f387401d070 con 0x7f3868004d40 2024-04-26T23:06:34.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.705+0000 7f387b400640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f386006c920 0x7f386006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.705+0000 7f387b400640 1 -- 172.21.15.40:0/76097102 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f387408c130 con 0x7f3868004d40 2024-04-26T23:06:34.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.706+0000 7f388d36a740 1 Processor -- start 2024-04-26T23:06:34.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.706+0000 7f3882a00640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f386006c920 0x7f386006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.706+0000 7f388d36a740 1 -- start start 2024-04-26T23:06:34.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f388d36a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d24a0 0x55e4148d4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f388d36a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 0x55e4148d51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f388d36a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148a5100 0x55e4148a8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f388d36a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e41480dc70 con 0x55e4148d2880 2024-04-26T23:06:34.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f388d36a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e4147c6980 con 0x55e4148d24a0 2024-04-26T23:06:34.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f388d36a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e414809c30 con 0x55e4148a5100 2024-04-26T23:06:34.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f3882a00640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f386006c920 0x7f386006ede0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f386c004770 tx=0x7f386c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f3882a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 0x55e4148d51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f3882a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 0x55e4148d51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56956/0 (socket says 172.21.15.40:56956) 2024-04-26T23:06:34.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f3882a00640 1 -- 172.21.15.40:0/1439138396 learned_addr learned my addr 172.21.15.40:0/1439138396 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f3883400640 1 --2- 172.21.15.40:0/1439138396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d24a0 0x55e4148d4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f3883e00640 1 --2- 172.21.15.40:0/1439138396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148a5100 0x55e4148a8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f3882a00640 1 -- 172.21.15.40:0/1439138396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148a5100 msgr2=0x55e4148a8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f3882a00640 1 --2- 172.21.15.40:0/1439138396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148a5100 0x55e4148a8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f3882a00640 1 -- 172.21.15.40:0/1439138396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d24a0 msgr2=0x55e4148d4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f3882a00640 1 --2- 172.21.15.40:0/1439138396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d24a0 0x55e4148d4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.707+0000 7f3882a00640 1 -- 172.21.15.40:0/1439138396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e41479e7b0 con 0x55e4148d2880 2024-04-26T23:06:34.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.708+0000 7f3882a00640 1 --2- 172.21.15.40:0/1439138396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 0x55e4148d51f0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f386c010950 tx=0x7f386c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbf5612585c10a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.708+0000 7f3873e00640 1 -- 172.21.15.40:0/1439138396 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c01c050 con 0x55e4148d2880 2024-04-26T23:06:34.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.708+0000 7f3873e00640 1 -- 172.21.15.40:0/1439138396 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f386c026070 con 0x55e4148d2880 2024-04-26T23:06:34.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.708+0000 7f3873e00640 1 -- 172.21.15.40:0/1439138396 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c00f040 con 0x55e4148d2880 2024-04-26T23:06:34.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.709+0000 7f388d36a740 1 -- 172.21.15.40:0/1439138396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 msgr2=0x55e4148d51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.709+0000 7f388d36a740 1 --2- 172.21.15.40:0/1439138396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 0x55e4148d51f0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f386c010950 tx=0x7f386c013000 comp rx=0 tx=0).stop 2024-04-26T23:06:34.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.709+0000 7f388d36a740 1 -- 172.21.15.40:0/1439138396 shutdown_connections 2024-04-26T23:06:34.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.709+0000 7f388d36a740 1 --2- 172.21.15.40:0/1439138396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148a5100 0x55e4148a8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.709+0000 7f388d36a740 1 --2- 172.21.15.40:0/1439138396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 0x55e4148d51f0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.709+0000 7f388d36a740 1 --2- 172.21.15.40:0/1439138396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d24a0 0x55e4148d4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.709+0000 7f388d36a740 1 -- 172.21.15.40:0/1439138396 >> 172.21.15.40:0/1439138396 conn(0x55e4148a7370 msgr2=0x55e4148efcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.709+0000 7f388d36a740 1 -- 172.21.15.40:0/1439138396 shutdown_connections 2024-04-26T23:06:34.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.709+0000 7f388d36a740 1 -- 172.21.15.40:0/1439138396 wait complete. 2024-04-26T23:06:34.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.710+0000 7f388d36a740 1 -- start start 2024-04-26T23:06:34.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.710+0000 7f388d36a740 1 Processor -- start 2024-04-26T23:06:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.711+0000 7f388d36a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d24a0 0x55e4148ea370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.711+0000 7f388d36a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d2880 0x55e4148d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.711+0000 7f388d36a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 0x55e4148cb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.711+0000 7f388d36a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e41480cf80 con 0x55e4148cb1f0 2024-04-26T23:06:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.711+0000 7f388d36a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e4147ba260 con 0x55e4148d2880 2024-04-26T23:06:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.711+0000 7f388d36a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e4148122a0 con 0x55e4148d24a0 2024-04-26T23:06:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.711+0000 7f388d36a740 10 client.0 _ll_register_callbacks cb 0x55e414804960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.711+0000 7f3883e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 0x55e4148cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.711+0000 7f3883e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 0x55e4148cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56970/0 (socket says 172.21.15.40:56970) 2024-04-26T23:06:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.711+0000 7f3883e00640 1 -- 172.21.15.40:0/913243038 learned_addr learned my addr 172.21.15.40:0/913243038 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.712+0000 7f3883400640 1 --2- 172.21.15.40:0/913243038 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d24a0 0x55e4148ea370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.712+0000 7f3882a00640 1 --2- 172.21.15.40:0/913243038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d2880 0x55e4148d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.712+0000 7f3883e00640 1 -- 172.21.15.40:0/913243038 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d24a0 msgr2=0x55e4148ea370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.712+0000 7f3883e00640 1 --2- 172.21.15.40:0/913243038 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d24a0 0x55e4148ea370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.712+0000 7f3883e00640 1 -- 172.21.15.40:0/913243038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d2880 msgr2=0x55e4148d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.712+0000 7f3883e00640 1 --2- 172.21.15.40:0/913243038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d2880 0x55e4148d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.712+0000 7f3883e00640 1 -- 172.21.15.40:0/913243038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e41479c1c0 con 0x55e4148cb1f0 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.712+0000 7f3883e00640 1 --2- 172.21.15.40:0/913243038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 0x55e4148cb670 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f387c02ae80 tx=0x7f387c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b177937afd0b55c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.712+0000 7f3870c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.712+0000 7f3870c00640 1 -- 172.21.15.40:0/913243038 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c001c80 con 0x55e4148cb1f0 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.713+0000 7f3870c00640 1 -- 172.21.15.40:0/913243038 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f387c038040 con 0x55e4148cb1f0 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.713+0000 7f388d36a740 1 -- 172.21.15.40:0/913243038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f386c026070 con 0x55e4148cb1f0 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.713+0000 7f3870c00640 1 -- 172.21.15.40:0/913243038 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f387c004d20 con 0x55e4148cb1f0 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.713+0000 7f388d36a740 10 client.7200 fetch_fsmap learned FSMap version 66 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.713+0000 7f388d36a740 1 -- 172.21.15.40:0/913243038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) -- 0x55e41451f3f0 con 0x55e4148cb1f0 2024-04-26T23:06:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.713+0000 7f3870c00640 1 -- 172.21.15.40:0/913243038 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f386c026070 con 0x55e4148cb1f0 2024-04-26T23:06:34.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.713+0000 7f3870c00640 1 -- 172.21.15.40:0/913243038 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 66) ==== 3550+0+0 (secure 0 0 0) 0x7f387c04e080 con 0x55e4148cb1f0 2024-04-26T23:06:34.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.713+0000 7f388d36a740 10 client.7200 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T23:06:34.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.713+0000 7f388d36a740 10 client.7200 resolve_mds: resolved 4:1 to role '4:1' aka daemon mds.a 2024-04-26T23:06:34.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.713+0000 7f388d36a740 20 client.7200 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:34.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.713+0000 7f388d36a740 1 --2- 172.21.15.40:0/913243038 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55e4148c97c0 0x55e4148f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.713+0000 7f388d36a740 4 client.7200 mds_command: new command op to 5614 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:06:34.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.713+0000 7f388d36a740 1 -- 172.21.15.40:0/913243038 --> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e41481b880 con 0x55e4148c97c0 2024-04-26T23:06:34.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.714+0000 7f3883400640 1 --2- 172.21.15.40:0/913243038 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55e4148c97c0 0x55e4148f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.714+0000 7f3883400640 1 --2- 172.21.15.40:0/913243038 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55e4148c97c0 0x55e4148f32d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=684fdeff8634e799 server_cookie=7c272cf46f55e3c5 in_seq=0 out_seq=0 2024-04-26T23:06:34.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.714+0000 7f3870c00640 10 client.7200 ms_handle_connect on v2:172.21.15.40:6832/2117861921 2024-04-26T23:06:34.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.715+0000 7f3870c00640 1 -- 172.21.15.40:0/913243038 <== mds.1 v2:172.21.15.40:6832/2117861921 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e41481b880 con 0x55e4148c97c0 2024-04-26T23:06:34.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.715+0000 7f3870c00640 10 client.7200 handle_command_reply: tid=0 2024-04-26T23:06:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.715+0000 7f388d36a740 1 client.7200 shutdown 2024-04-26T23:06:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.715+0000 7f388d36a740 1 -- 172.21.15.40:0/913243038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 msgr2=0x55e4148cb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.715+0000 7f388d36a740 1 --2- 172.21.15.40:0/913243038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 0x55e4148cb670 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f387c02ae80 tx=0x7f387c001a40 comp rx=0 tx=0).stop 2024-04-26T23:06:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.715+0000 7f388d36a740 1 -- 172.21.15.40:0/913243038 shutdown_connections 2024-04-26T23:06:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.715+0000 7f388d36a740 1 --2- 172.21.15.40:0/913243038 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55e4148c97c0 0x55e4148f32d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.715+0000 7f388d36a740 1 --2- 172.21.15.40:0/913243038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 0x55e4148cb670 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.715+0000 7f388d36a740 1 --2- 172.21.15.40:0/913243038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d2880 0x55e4148d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.715+0000 7f388d36a740 1 --2- 172.21.15.40:0/913243038 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d24a0 0x55e4148ea370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.715+0000 7f388d36a740 1 -- 172.21.15.40:0/913243038 >> 172.21.15.40:0/913243038 conn(0x55e4148a5100 msgr2=0x55e4148d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.715+0000 7f3870c00640 0 client.7200 ms_handle_reset on v2:172.21.15.40:6832/2117861921 2024-04-26T23:06:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.716+0000 7f388d36a740 1 -- 172.21.15.40:0/913243038 shutdown_connections 2024-04-26T23:06:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.716+0000 7f388d36a740 1 -- 172.21.15.40:0/913243038 wait complete. 2024-04-26T23:06:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.716+0000 7f388d36a740 20 client.7200 trim_cache size 0 max 16384 2024-04-26T23:06:34.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.731+0000 7f388d36a740 1 Processor -- start 2024-04-26T23:06:34.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.731+0000 7f388d36a740 1 -- start start 2024-04-26T23:06:34.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.731+0000 7f388d36a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148cb1f0 0x55e4148a58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f388d36a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d24a0 0x55e4148d8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f388d36a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 0x55e4148ec1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f388d36a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3884202670 con 0x55e4148d2880 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f388d36a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3884201810 con 0x55e4148cb1f0 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f388d36a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3884201f40 con 0x55e4148d24a0 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f3883e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 0x55e4148ec1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f3883e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 0x55e4148ec1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56974/0 (socket says 172.21.15.40:56974) 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f3883e00640 1 -- 172.21.15.40:0/1106346191 learned_addr learned my addr 172.21.15.40:0/1106346191 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f3883400640 1 --2- 172.21.15.40:0/1106346191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148cb1f0 0x55e4148a58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f3882a00640 1 --2- 172.21.15.40:0/1106346191 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d24a0 0x55e4148d8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f3883e00640 1 -- 172.21.15.40:0/1106346191 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d24a0 msgr2=0x55e4148d8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f3883e00640 1 --2- 172.21.15.40:0/1106346191 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d24a0 0x55e4148d8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f3883e00640 1 -- 172.21.15.40:0/1106346191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148cb1f0 msgr2=0x55e4148a58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f3883e00640 1 --2- 172.21.15.40:0/1106346191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148cb1f0 0x55e4148a58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.732+0000 7f3883e00640 1 -- 172.21.15.40:0/1106346191 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4145e97d0 con 0x55e4148d2880 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.733+0000 7f3883e00640 1 --2- 172.21.15.40:0/1106346191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 0x55e4148ec1c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f387c007980 tx=0x7f387c03e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b81eb7e8f107dfe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.733+0000 7f3873400640 1 -- 172.21.15.40:0/1106346191 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c03ec20 con 0x55e4148d2880 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.733+0000 7f3873400640 1 -- 172.21.15.40:0/1106346191 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f387c03edc0 con 0x55e4148d2880 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.733+0000 7f3873400640 1 -- 172.21.15.40:0/1106346191 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c03e670 con 0x55e4148d2880 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.734+0000 7f388d36a740 1 -- 172.21.15.40:0/1106346191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 msgr2=0x55e4148ec1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.734+0000 7f388d36a740 1 --2- 172.21.15.40:0/1106346191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 0x55e4148ec1c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f387c007980 tx=0x7f387c03e000 comp rx=0 tx=0).stop 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.734+0000 7f388d36a740 1 -- 172.21.15.40:0/1106346191 shutdown_connections 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.734+0000 7f388d36a740 1 --2- 172.21.15.40:0/1106346191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148d2880 0x55e4148ec1c0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.734+0000 7f388d36a740 1 --2- 172.21.15.40:0/1106346191 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d24a0 0x55e4148d8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.734+0000 7f388d36a740 1 --2- 172.21.15.40:0/1106346191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148cb1f0 0x55e4148a58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.734+0000 7f388d36a740 1 -- 172.21.15.40:0/1106346191 >> 172.21.15.40:0/1106346191 conn(0x55e4148a5100 msgr2=0x55e4148d1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.734+0000 7f388d36a740 1 -- 172.21.15.40:0/1106346191 shutdown_connections 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.734+0000 7f388d36a740 1 -- 172.21.15.40:0/1106346191 wait complete. 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.735+0000 7f388d36a740 1 -- start start 2024-04-26T23:06:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.735+0000 7f388d36a740 1 Processor -- start 2024-04-26T23:06:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.736+0000 7f388d36a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 0x55e4148a4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.736+0000 7f388d36a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d24a0 0x55e414912230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.736+0000 7f388d36a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d2880 0x55e414912770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.736+0000 7f388d36a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e4148ef8e0 con 0x55e4148cb1f0 2024-04-26T23:06:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.736+0000 7f388d36a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f38842007d0 con 0x55e4148d24a0 2024-04-26T23:06:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.736+0000 7f388d36a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e4148d2130 con 0x55e4148d2880 2024-04-26T23:06:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.736+0000 7f388d36a740 10 client.0 _ll_register_callbacks cb 0x7f3868004470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.736+0000 7f3883400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 0x55e4148a4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.736+0000 7f3883400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 0x55e4148a4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56978/0 (socket says 172.21.15.40:56978) 2024-04-26T23:06:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.736+0000 7f3883400640 1 -- 172.21.15.40:0/1285466814 learned_addr learned my addr 172.21.15.40:0/1285466814 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f3882a00640 1 --2- 172.21.15.40:0/1285466814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d24a0 0x55e414912230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f3883400640 1 -- 172.21.15.40:0/1285466814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d2880 msgr2=0x55e414912770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f3883e00640 1 --2- 172.21.15.40:0/1285466814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d2880 0x55e414912770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f3883400640 1 --2- 172.21.15.40:0/1285466814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d2880 0x55e414912770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f3883400640 1 -- 172.21.15.40:0/1285466814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d24a0 msgr2=0x55e414912230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f3883400640 1 --2- 172.21.15.40:0/1285466814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d24a0 0x55e414912230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f3883400640 1 -- 172.21.15.40:0/1285466814 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4147b42a0 con 0x55e4148cb1f0 2024-04-26T23:06:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f3883400640 1 --2- 172.21.15.40:0/1285466814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 0x55e4148a4320 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f3874059b00 tx=0x7f387408a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4a06c2eedd1238 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f3870c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f3870c00640 1 -- 172.21.15.40:0/1285466814 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874059e20 con 0x55e4148cb1f0 2024-04-26T23:06:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f3870c00640 1 -- 172.21.15.40:0/1285466814 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f3874059fc0 con 0x55e4148cb1f0 2024-04-26T23:06:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f388d36a740 1 -- 172.21.15.40:0/1285466814 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f387c03edc0 con 0x55e4148cb1f0 2024-04-26T23:06:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f3870c00640 1 -- 172.21.15.40:0/1285466814 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f3874062310 con 0x55e4148cb1f0 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f388d36a740 10 client.7206 fetch_fsmap learned FSMap version 66 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.737+0000 7f388d36a740 1 -- 172.21.15.40:0/1285466814 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) -- 0x55e414901e40 con 0x55e4148cb1f0 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.738+0000 7f3870c00640 1 -- 172.21.15.40:0/1285466814 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f387c03edc0 con 0x55e4148cb1f0 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.738+0000 7f3870c00640 1 -- 172.21.15.40:0/1285466814 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 66) ==== 3550+0+0 (secure 0 0 0) 0x7f3874059010 con 0x55e4148cb1f0 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.738+0000 7f388d36a740 10 client.7206 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.738+0000 7f388d36a740 10 client.7206 resolve_mds: resolved 4:1 to role '4:1' aka daemon mds.a 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.738+0000 7f388d36a740 20 client.7206 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.738+0000 7f388d36a740 1 --2- 172.21.15.40:0/1285466814 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55e4148e62e0 0x55e414908c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.738+0000 7f388d36a740 4 client.7206 mds_command: new command op to 5614 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.738+0000 7f388d36a740 1 -- 172.21.15.40:0/1285466814 --> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e4148156f0 con 0x55e4148e62e0 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.738+0000 7f3882a00640 1 --2- 172.21.15.40:0/1285466814 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55e4148e62e0 0x55e414908c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f3882a00640 1 --2- 172.21.15.40:0/1285466814 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55e4148e62e0 0x55e414908c30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7734a0c0637abbaa server_cookie=71cb9abf150c352e in_seq=0 out_seq=0 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f3870c00640 10 client.7206 ms_handle_connect on v2:172.21.15.40:6832/2117861921 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f3870c00640 1 -- 172.21.15.40:0/1285466814 <== mds.1 v2:172.21.15.40:6832/2117861921 1 ==== command_reply(tid 0: 0 ) ==== 8+0+282 (crc 0 0 0) 0x55e4148156f0 con 0x55e4148e62e0 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f3870c00640 10 client.7206 handle_command_reply: tid=0 2024-04-26T23:06:34.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f388d36a740 1 client.7206 shutdown 2024-04-26T23:06:34.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f388d36a740 1 -- 172.21.15.40:0/1285466814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 msgr2=0x55e4148a4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f388d36a740 1 --2- 172.21.15.40:0/1285466814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 0x55e4148a4320 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f3874059b00 tx=0x7f387408a290 comp rx=0 tx=0).stop 2024-04-26T23:06:34.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f388d36a740 1 -- 172.21.15.40:0/1285466814 shutdown_connections 2024-04-26T23:06:34.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f388d36a740 1 --2- 172.21.15.40:0/1285466814 >> [v2:172.21.15.40:6832/2117861921,v1:172.21.15.40:6834/2117861921] conn(0x55e4148e62e0 0x55e414908c30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f388d36a740 1 --2- 172.21.15.40:0/1285466814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e4148d2880 0x55e414912770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f388d36a740 1 --2- 172.21.15.40:0/1285466814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e4148d24a0 0x55e414912230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f3870c00640 0 client.7206 ms_handle_reset on v2:172.21.15.40:6832/2117861921 2024-04-26T23:06:34.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f388d36a740 1 --2- 172.21.15.40:0/1285466814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e4148cb1f0 0x55e4148a4320 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.739+0000 7f388d36a740 1 -- 172.21.15.40:0/1285466814 >> 172.21.15.40:0/1285466814 conn(0x55e4148a5100 msgr2=0x55e4148a08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.740+0000 7f388d36a740 1 -- 172.21.15.40:0/1285466814 shutdown_connections 2024-04-26T23:06:34.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.740+0000 7f388d36a740 1 -- 172.21.15.40:0/1285466814 wait complete. 2024-04-26T23:06:34.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.740+0000 7f388d36a740 20 client.7206 trim_cache size 0 max 16384 2024-04-26T23:06:34.745 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-04-26T23:06:34.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:34.745 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-26T23:06:34.745 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 616926524, 2024-04-26T23:06:34.745 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511627781, 2024-04-26T23:06:34.745 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3/d4/d5/d6" 2024-04-26T23:06:34.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:34.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:34.745 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-26T23:06:34.745 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 733518437, 2024-04-26T23:06:34.745 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511627780, 2024-04-26T23:06:34.746 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3/d4/d5" 2024-04-26T23:06:34.746 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:34.746 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-04-26T23:06:34.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.740+0000 7f3889000640 1 -- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f386006c920 msgr2=0x7f386006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.740+0000 7f3889000640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f386006c920 0x7f386006ede0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f386c004770 tx=0x7f386c008040 comp rx=0 tx=0).stop 2024-04-26T23:06:34.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.740+0000 7f3889000640 1 -- 172.21.15.40:0/76097102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3868004d40 msgr2=0x7f38840ccb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.740+0000 7f3889000640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3868004d40 0x7f38840ccb30 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f3874007f90 tx=0x7f387400d560 comp rx=0 tx=0).stop 2024-04-26T23:06:34.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.740+0000 7f3889000640 1 -- 172.21.15.40:0/76097102 shutdown_connections 2024-04-26T23:06:34.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.740+0000 7f3889000640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f386006c920 0x7f386006ede0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.740+0000 7f3889000640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38840cbe70 0x7f38840cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.740+0000 7f3889000640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38840cfbf0 0x7f3884150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.740+0000 7f3889000640 1 --2- 172.21.15.40:0/76097102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3868004d40 0x7f38840ccb30 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.741+0000 7f3889000640 1 -- 172.21.15.40:0/76097102 >> 172.21.15.40:0/76097102 conn(0x7f38840d2bd0 msgr2=0x7f3884070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.741+0000 7f3889000640 1 -- 172.21.15.40:0/76097102 shutdown_connections 2024-04-26T23:06:34.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.741+0000 7f3889000640 1 -- 172.21.15.40:0/76097102 wait complete. 2024-04-26T23:06:34.758 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:2 damage ls 2024-04-26T23:06:34.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.949+0000 7f959ac00640 1 Processor -- start 2024-04-26T23:06:34.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.949+0000 7f959ac00640 1 -- start start 2024-04-26T23:06:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.951+0000 7f959ac00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9594160a50 0x7f9594073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.951+0000 7f959ac00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9594161880 con 0x7f959406dd40 2024-04-26T23:06:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.951+0000 7f959ac00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95940cff80 con 0x7f95940cfaf0 2024-04-26T23:06:34.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.951+0000 7f959ac00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9594071ce0 con 0x7f9594160a50 2024-04-26T23:06:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.951+0000 7f959a200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f959406dd40 0x7f9594072730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60314/0 (socket says 172.21.15.40:60314) 2024-04-26T23:06:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.951+0000 7f959a200640 1 -- 172.21.15.40:0/1472845397 learned_addr learned my addr 172.21.15.40:0/1472845397 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.952+0000 7f9599800640 1 --2- 172.21.15.40:0/1472845397 >> v2:172.21.15.103:3300/0 conn(0x7f9594160a50 0x7f9594073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.952+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3044837025 0 0) 0x7f9594161880 con 0x7f959406dd40 2024-04-26T23:06:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.952+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f957c003520 con 0x7f959406dd40 2024-04-26T23:06:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.952+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 457513847 0 0) 0x7f957c003520 con 0x7f959406dd40 2024-04-26T23:06:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.952+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 >> v2:172.21.15.103:3300/0 conn(0x7f9594160a50 msgr2=0x7f9594073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.952+0000 7f9593e00640 1 --2- 172.21.15.40:0/1472845397 >> v2:172.21.15.103:3300/0 conn(0x7f9594160a50 0x7f9594073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.952+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 >> v1:172.21.15.103:6789/0 conn(0x7f95940cfaf0 legacy=0x7f9594160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.952+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95940d0260 con 0x7f959406dd40 2024-04-26T23:06:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.952+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f958c0019f0 con 0x7f959406dd40 2024-04-26T23:06:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.952+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 >> v1:172.21.15.40:6789/0 conn(0x7f959406dd40 legacy=0x7f9594072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.952+0000 7f9593e00640 1 --2- 172.21.15.40:0/1472845397 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f957c0039b0 0x7f957c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.952+0000 7f9593e00640 1 --2- 172.21.15.40:0/1472845397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f957c004800 0x7f957c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.953+0000 7f9593e00640 1 --2- 172.21.15.40:0/1472845397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f957c0051c0 0x7f957c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.953+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9594071ce0 con 0x7f957c0051c0 2024-04-26T23:06:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.953+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f957c00c060 con 0x7f957c004800 2024-04-26T23:06:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.953+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f957c00c260 con 0x7f957c0039b0 2024-04-26T23:06:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.953+0000 7f959a200640 1 --2- 172.21.15.40:0/1472845397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f957c0051c0 0x7f957c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.953+0000 7f9599800640 1 --2- 172.21.15.40:0/1472845397 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f957c0039b0 0x7f957c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.953+0000 7f9598e00640 1 --2- 172.21.15.40:0/1472845397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f957c004800 0x7f957c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.953+0000 7f959a200640 1 -- 172.21.15.40:0/1472845397 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f957c0039b0 msgr2=0x7f957c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.953+0000 7f959a200640 1 --2- 172.21.15.40:0/1472845397 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f957c0039b0 0x7f957c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.953+0000 7f959a200640 1 -- 172.21.15.40:0/1472845397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f957c004800 msgr2=0x7f957c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.953+0000 7f959a200640 1 --2- 172.21.15.40:0/1472845397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f957c004800 0x7f957c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.953+0000 7f959a200640 1 -- 172.21.15.40:0/1472845397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957c003520 con 0x7f957c0051c0 2024-04-26T23:06:34.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.954+0000 7f959a200640 1 -- 172.21.15.40:0/1472845397 reap_dead start 2024-04-26T23:06:34.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.954+0000 7f959a200640 1 --2- 172.21.15.40:0/1472845397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f957c0051c0 0x7f957c00b9c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f958c00a850 tx=0x7f957c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50791bf93c7d653 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.955+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c00a4a0 con 0x7f957c0051c0 2024-04-26T23:06:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.955+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f958c02f070 con 0x7f957c0051c0 2024-04-26T23:06:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.955+0000 7f9593e00640 1 -- 172.21.15.40:0/1472845397 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c02bce0 con 0x7f957c0051c0 2024-04-26T23:06:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.955+0000 7f959ac00640 1 -- 172.21.15.40:0/1472845397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f957c0051c0 msgr2=0x7f957c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.955+0000 7f959ac00640 1 --2- 172.21.15.40:0/1472845397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f957c0051c0 0x7f957c00b9c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f958c00a850 tx=0x7f957c005830 comp rx=0 tx=0).stop 2024-04-26T23:06:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.955+0000 7f959ac00640 1 -- 172.21.15.40:0/1472845397 shutdown_connections 2024-04-26T23:06:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.955+0000 7f959ac00640 1 --2- 172.21.15.40:0/1472845397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f957c0051c0 0x7f957c00b9c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.955+0000 7f959ac00640 1 -- 172.21.15.40:0/1472845397 >> 172.21.15.40:0/1472845397 conn(0x7f95940b2720 msgr2=0x7f95940b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.956+0000 7f959ac00640 1 -- 172.21.15.40:0/1472845397 shutdown_connections 2024-04-26T23:06:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.956+0000 7f959ac00640 1 -- 172.21.15.40:0/1472845397 wait complete. 2024-04-26T23:06:34.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.957+0000 7f959ac00640 1 Processor -- start 2024-04-26T23:06:34.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.958+0000 7f959ac00640 1 -- start start 2024-04-26T23:06:34.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.958+0000 7f959ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f957c0051c0 0x7f9594134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f959ac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9594144160 0x7f95940b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f959ac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95941445e0 0x7f95940b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f959ac00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f959406e1f0 con 0x7f9594144160 2024-04-26T23:06:34.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f959ac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f95940722a0 con 0x7f95941445e0 2024-04-26T23:06:34.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f959ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f95941468f0 con 0x7f957c0051c0 2024-04-26T23:06:34.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f9598e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9594144160 0x7f95940b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f9598e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9594144160 0x7f95940b1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42854/0 (socket says 172.21.15.40:42854) 2024-04-26T23:06:34.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f9598e00640 1 -- 172.21.15.40:0/3171491179 learned_addr learned my addr 172.21.15.40:0/3171491179 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f9599800640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f957c0051c0 0x7f9594134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f9598e00640 1 -- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f957c0051c0 msgr2=0x7f9594134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f9598e00640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f957c0051c0 0x7f9594134560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f9598e00640 1 -- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95941445e0 msgr2=0x7f95940b1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f959a200640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95941445e0 0x7f95940b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f9598e00640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95941445e0 0x7f95940b1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.959+0000 7f9598e00640 1 -- 172.21.15.40:0/3171491179 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9594134aa0 con 0x7f9594144160 2024-04-26T23:06:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.960+0000 7f9598e00640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9594144160 0x7f95940b1520 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f958000e4b0 tx=0x7f958000e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.960+0000 7f9591600640 1 -- 172.21.15.40:0/3171491179 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95800052e0 con 0x7f9594144160 2024-04-26T23:06:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.960+0000 7f9591600640 1 -- 172.21.15.40:0/3171491179 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9580018070 con 0x7f9594144160 2024-04-26T23:06:34.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.960+0000 7f959ac00640 1 -- 172.21.15.40:0/3171491179 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9594137e60 con 0x7f9594144160 2024-04-26T23:06:34.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.960+0000 7f9591600640 1 -- 172.21.15.40:0/3171491179 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580003830 con 0x7f9594144160 2024-04-26T23:06:34.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.960+0000 7f959ac00640 1 -- 172.21.15.40:0/3171491179 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95941f9090 con 0x7f9594144160 2024-04-26T23:06:34.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.963+0000 7f9591600640 1 -- 172.21.15.40:0/3171491179 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f95800039d0 con 0x7f9594144160 2024-04-26T23:06:34.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.963+0000 7f9591600640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f957406c920 0x7f957406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.964+0000 7f9591600640 1 -- 172.21.15.40:0/3171491179 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f958008ba50 con 0x7f9594144160 2024-04-26T23:06:34.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.964+0000 7f959f010740 1 Processor -- start 2024-04-26T23:06:34.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.964+0000 7f9599800640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f957406c920 0x7f957406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.964+0000 7f959f010740 1 -- start start 2024-04-26T23:06:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.965+0000 7f959f010740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 0x55962630fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.965+0000 7f959f010740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962630d880 0x5596263101f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.965+0000 7f959f010740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596262e0100 0x5596262e3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.965+0000 7f959f010740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559626248c70 con 0x5596262e0100 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.965+0000 7f959f010740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559626201980 con 0x55962630d880 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.965+0000 7f959f010740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x559626244c30 con 0x55962630d4a0 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.965+0000 7f9599800640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f957406c920 0x7f957406ede0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9588009990 tx=0x7f9588008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.965+0000 7f959a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596262e0100 0x5596262e3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.966+0000 7f959a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596262e0100 0x5596262e3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42870/0 (socket says 172.21.15.40:42870) 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.966+0000 7f959a200640 1 -- 172.21.15.40:0/3059154831 learned_addr learned my addr 172.21.15.40:0/3059154831 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.966+0000 7f9598e00640 1 --2- 172.21.15.40:0/3059154831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962630d880 0x5596263101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.966+0000 7f9599800640 1 --2- 172.21.15.40:0/3059154831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 0x55962630fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.966+0000 7f959a200640 1 -- 172.21.15.40:0/3059154831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 msgr2=0x55962630fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.966+0000 7f959a200640 1 --2- 172.21.15.40:0/3059154831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 0x55962630fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.966+0000 7f959a200640 1 -- 172.21.15.40:0/3059154831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962630d880 msgr2=0x5596263101f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.966+0000 7f959a200640 1 --2- 172.21.15.40:0/3059154831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962630d880 0x5596263101f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.966+0000 7f959a200640 1 -- 172.21.15.40:0/3059154831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596261d97b0 con 0x5596262e0100 2024-04-26T23:06:34.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.966+0000 7f959a200640 1 --2- 172.21.15.40:0/3059154831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596262e0100 0x5596262e3910 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f958c000f80 tx=0x7f958c009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28c61cf92a4cc87 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.967+0000 7f9586000640 1 -- 172.21.15.40:0/3059154831 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c044070 con 0x5596262e0100 2024-04-26T23:06:34.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.967+0000 7f9586000640 1 -- 172.21.15.40:0/3059154831 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f958c02f040 con 0x5596262e0100 2024-04-26T23:06:34.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.967+0000 7f9586000640 1 -- 172.21.15.40:0/3059154831 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c02bac0 con 0x5596262e0100 2024-04-26T23:06:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.967+0000 7f959f010740 1 -- 172.21.15.40:0/3059154831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596262e0100 msgr2=0x5596262e3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.967+0000 7f959f010740 1 --2- 172.21.15.40:0/3059154831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596262e0100 0x5596262e3910 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f958c000f80 tx=0x7f958c009030 comp rx=0 tx=0).stop 2024-04-26T23:06:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.967+0000 7f959f010740 1 -- 172.21.15.40:0/3059154831 shutdown_connections 2024-04-26T23:06:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.967+0000 7f959f010740 1 --2- 172.21.15.40:0/3059154831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596262e0100 0x5596262e3910 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.967+0000 7f959f010740 1 --2- 172.21.15.40:0/3059154831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962630d880 0x5596263101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.967+0000 7f959f010740 1 --2- 172.21.15.40:0/3059154831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 0x55962630fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.967+0000 7f959f010740 1 -- 172.21.15.40:0/3059154831 >> 172.21.15.40:0/3059154831 conn(0x5596262e2370 msgr2=0x55962632acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.968+0000 7f959f010740 1 -- 172.21.15.40:0/3059154831 shutdown_connections 2024-04-26T23:06:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.968+0000 7f959f010740 1 -- 172.21.15.40:0/3059154831 wait complete. 2024-04-26T23:06:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.968+0000 7f959f010740 1 -- start start 2024-04-26T23:06:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.969+0000 7f959f010740 1 Processor -- start 2024-04-26T23:06:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f959f010740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962630d4a0 0x5596262f84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f959f010740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x559626321940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f959f010740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962632d840 0x559626321e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f959f010740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559626247f80 con 0x55962630d880 2024-04-26T23:06:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f959f010740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5596261f5260 con 0x55962630d4a0 2024-04-26T23:06:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f959f010740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55962624d2a0 con 0x55962632d840 2024-04-26T23:06:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f959f010740 10 client.0 _ll_register_callbacks cb 0x55962623f960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f9598e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x559626321940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f9598e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x559626321940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42886/0 (socket says 172.21.15.40:42886) 2024-04-26T23:06:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f9598e00640 1 -- 172.21.15.40:0/978031604 learned_addr learned my addr 172.21.15.40:0/978031604 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f9599800640 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962630d4a0 0x5596262f84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f959a200640 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962632d840 0x559626321e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f9598e00640 1 -- 172.21.15.40:0/978031604 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962632d840 msgr2=0x559626321e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f9598e00640 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962632d840 0x559626321e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f9598e00640 1 -- 172.21.15.40:0/978031604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962630d4a0 msgr2=0x5596262f84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f9598e00640 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962630d4a0 0x5596262f84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f9598e00640 1 -- 172.21.15.40:0/978031604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596261d71c0 con 0x55962630d880 2024-04-26T23:06:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.970+0000 7f9599800640 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962630d4a0 0x5596262f84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.971+0000 7f9598e00640 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x559626321940 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f958000ee70 tx=0x7f9580059130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca433b20ded1132 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.971+0000 7f957aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.971+0000 7f957aa00640 1 -- 172.21.15.40:0/978031604 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95800620c0 con 0x55962630d880 2024-04-26T23:06:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.971+0000 7f957aa00640 1 -- 172.21.15.40:0/978031604 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f9580055b50 con 0x55962630d880 2024-04-26T23:06:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.971+0000 7f959f010740 1 -- 172.21.15.40:0/978031604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f958c02f040 con 0x55962630d880 2024-04-26T23:06:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.971+0000 7f957aa00640 1 -- 172.21.15.40:0/978031604 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f9580078050 con 0x55962630d880 2024-04-26T23:06:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.971+0000 7f959f010740 10 client.7218 fetch_fsmap learned FSMap version 66 2024-04-26T23:06:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.971+0000 7f959f010740 1 -- 172.21.15.40:0/978031604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) -- 0x559625f5a3f0 con 0x55962630d880 2024-04-26T23:06:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.971+0000 7f957aa00640 1 -- 172.21.15.40:0/978031604 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f958c02f040 con 0x55962630d880 2024-04-26T23:06:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.972+0000 7f957aa00640 1 -- 172.21.15.40:0/978031604 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 66) ==== 3550+0+0 (secure 0 0 0) 0x7f9580003cc0 con 0x55962630d880 2024-04-26T23:06:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.972+0000 7f959f010740 10 client.7218 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T23:06:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.972+0000 7f959f010740 10 client.7218 resolve_mds: resolved 4:2 to role '4:2' aka daemon mds.b 2024-04-26T23:06:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.972+0000 7f959f010740 20 client.7218 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.972+0000 7f959f010740 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x559626323a70 0x559626323e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.972+0000 7f959f010740 4 client.7218 mds_command: new command op to 5324 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:06:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.972+0000 7f959f010740 1 -- 172.21.15.40:0/978031604 --> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559626256880 con 0x559626323a70 2024-04-26T23:06:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.972+0000 7f9599800640 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x559626323a70 0x559626323e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.973+0000 7f9599800640 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x559626323a70 0x559626323e90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f8d456dc15d780f7 server_cookie=d496c5d39ad3d430 in_seq=0 out_seq=0 2024-04-26T23:06:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.973+0000 7f957aa00640 10 client.7218 ms_handle_connect on v2:172.21.15.103:6835/2595375825 2024-04-26T23:06:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f957aa00640 1 -- 172.21.15.40:0/978031604 <== mds.2 v2:172.21.15.103:6835/2595375825 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559626256880 con 0x559626323a70 2024-04-26T23:06:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f957aa00640 10 client.7218 handle_command_reply: tid=0 2024-04-26T23:06:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f959f010740 1 client.7218 shutdown 2024-04-26T23:06:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f959f010740 1 -- 172.21.15.40:0/978031604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 msgr2=0x559626321940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f959f010740 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x559626321940 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f958000ee70 tx=0x7f9580059130 comp rx=0 tx=0).stop 2024-04-26T23:06:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f959f010740 1 -- 172.21.15.40:0/978031604 shutdown_connections 2024-04-26T23:06:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f959f010740 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962632d840 0x559626321e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f959f010740 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x559626321940 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f959f010740 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x559626323a70 0x559626323e90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f959f010740 1 --2- 172.21.15.40:0/978031604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962630d4a0 0x5596262f84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f957aa00640 0 client.7218 ms_handle_reset on v2:172.21.15.103:6835/2595375825 2024-04-26T23:06:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f959f010740 1 -- 172.21.15.40:0/978031604 >> 172.21.15.40:0/978031604 conn(0x5596262e0100 msgr2=0x5596263286c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f959f010740 1 -- 172.21.15.40:0/978031604 shutdown_connections 2024-04-26T23:06:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f959f010740 1 -- 172.21.15.40:0/978031604 wait complete. 2024-04-26T23:06:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.976+0000 7f959f010740 20 client.7218 trim_cache size 0 max 16384 2024-04-26T23:06:34.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.991+0000 7f959f010740 1 Processor -- start 2024-04-26T23:06:34.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.991+0000 7f959f010740 1 -- start start 2024-04-26T23:06:34.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.992+0000 7f959f010740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 0x559626321b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.992+0000 7f959f010740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x55962630c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.992+0000 7f959f010740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962632d840 0x55962630c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.992+0000 7f959f010740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f95942025c0 con 0x55962630d880 2024-04-26T23:06:34.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.992+0000 7f959f010740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9594201760 con 0x55962632d840 2024-04-26T23:06:34.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.992+0000 7f959f010740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9594201e90 con 0x55962630d4a0 2024-04-26T23:06:34.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.993+0000 7f9598e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x55962630c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.993+0000 7f9598e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x55962630c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42902/0 (socket says 172.21.15.40:42902) 2024-04-26T23:06:34.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.993+0000 7f9598e00640 1 -- 172.21.15.40:0/2127145233 learned_addr learned my addr 172.21.15.40:0/2127145233 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.993+0000 7f9598e00640 1 -- 172.21.15.40:0/2127145233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 msgr2=0x559626321b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.993+0000 7f9598e00640 1 --2- 172.21.15.40:0/2127145233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 0x559626321b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.993+0000 7f9598e00640 1 -- 172.21.15.40:0/2127145233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962632d840 msgr2=0x55962630c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.993+0000 7f9598e00640 1 --2- 172.21.15.40:0/2127145233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962632d840 0x55962630c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.993+0000 7f9598e00640 1 -- 172.21.15.40:0/2127145233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596260247d0 con 0x55962630d880 2024-04-26T23:06:34.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.993+0000 7f9598e00640 1 --2- 172.21.15.40:0/2127145233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x55962630c1b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f95800591e0 tx=0x7f958005c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d91f6dc3dbc78a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:34.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.993+0000 7f9585600640 1 -- 172.21.15.40:0/2127145233 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95800620c0 con 0x55962630d880 2024-04-26T23:06:34.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.993+0000 7f9585600640 1 -- 172.21.15.40:0/2127145233 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9580088070 con 0x55962630d880 2024-04-26T23:06:34.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.993+0000 7f9585600640 1 -- 172.21.15.40:0/2127145233 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580058e40 con 0x55962630d880 2024-04-26T23:06:34.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.994+0000 7f959f010740 1 -- 172.21.15.40:0/2127145233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 msgr2=0x55962630c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:34.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.994+0000 7f959f010740 1 --2- 172.21.15.40:0/2127145233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x55962630c1b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f95800591e0 tx=0x7f958005c070 comp rx=0 tx=0).stop 2024-04-26T23:06:34.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.994+0000 7f959f010740 1 -- 172.21.15.40:0/2127145233 shutdown_connections 2024-04-26T23:06:34.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.994+0000 7f959f010740 1 --2- 172.21.15.40:0/2127145233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962632d840 0x55962630c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.994+0000 7f959f010740 1 --2- 172.21.15.40:0/2127145233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x55962630c1b0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.994+0000 7f959f010740 1 --2- 172.21.15.40:0/2127145233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 0x559626321b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.994+0000 7f959f010740 1 -- 172.21.15.40:0/2127145233 >> 172.21.15.40:0/2127145233 conn(0x5596262e0100 msgr2=0x5596262e5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:34.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.994+0000 7f959f010740 1 -- 172.21.15.40:0/2127145233 shutdown_connections 2024-04-26T23:06:34.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.994+0000 7f959f010740 1 -- 172.21.15.40:0/2127145233 wait complete. 2024-04-26T23:06:34.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.995+0000 7f959f010740 1 -- start start 2024-04-26T23:06:34.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.995+0000 7f959f010740 1 Processor -- start 2024-04-26T23:06:34.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.996+0000 7f959f010740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 0x559626306de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.996+0000 7f959f010740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x55962630c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.996+0000 7f959f010740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962632d840 0x55962638cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.996+0000 7f959f010740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559626324a70 con 0x55962630d880 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.996+0000 7f959f010740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f959415ff80 con 0x55962632d840 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.996+0000 7f959f010740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x559626322d40 con 0x55962630d4a0 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.996+0000 7f959f010740 10 client.0 _ll_register_callbacks cb 0x7f9580055780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.996+0000 7f9598e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x55962630c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.996+0000 7f9598e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x55962630c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42906/0 (socket says 172.21.15.40:42906) 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.996+0000 7f9598e00640 1 -- 172.21.15.40:0/1198552537 learned_addr learned my addr 172.21.15.40:0/1198552537 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f9599800640 1 --2- 172.21.15.40:0/1198552537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 0x559626306de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f9598e00640 1 -- 172.21.15.40:0/1198552537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 msgr2=0x559626306de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f9598e00640 1 --2- 172.21.15.40:0/1198552537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 0x559626306de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f9598e00640 1 -- 172.21.15.40:0/1198552537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962632d840 msgr2=0x55962638cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:34.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f9598e00640 1 --2- 172.21.15.40:0/1198552537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962632d840 0x55962638cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f9598e00640 1 -- 172.21.15.40:0/1198552537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596261ef2a0 con 0x55962630d880 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f9598e00640 1 --2- 172.21.15.40:0/1198552537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x55962630c5b0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f9580057790 tx=0x7f9580089bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150d3b93db504c07 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f957aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f957aa00640 1 -- 172.21.15.40:0/1198552537 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95800592b0 con 0x55962630d880 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f957aa00640 1 -- 172.21.15.40:0/1198552537 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f9580078e00 con 0x55962630d880 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f959f010740 1 -- 172.21.15.40:0/1198552537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9580088070 con 0x55962630d880 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f957aa00640 1 -- 172.21.15.40:0/1198552537 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f9580058930 con 0x55962630d880 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f959f010740 10 client.7224 fetch_fsmap learned FSMap version 66 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.997+0000 7f959f010740 1 -- 172.21.15.40:0/1198552537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) -- 0x5596263a4770 con 0x55962630d880 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.998+0000 7f957aa00640 1 -- 172.21.15.40:0/1198552537 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9580088070 con 0x55962630d880 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.998+0000 7f957aa00640 1 -- 172.21.15.40:0/1198552537 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 66) ==== 3550+0+0 (secure 0 0 0) 0x7f9580058ad0 con 0x55962630d880 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.998+0000 7f959f010740 10 client.7224 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.998+0000 7f959f010740 10 client.7224 resolve_mds: resolved 4:2 to role '4:2' aka daemon mds.b 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.998+0000 7f959f010740 20 client.7224 populate_metadata read hostname 'smithi040' 2024-04-26T23:06:35.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.998+0000 7f959f010740 1 --2- 172.21.15.40:0/1198552537 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x5596263231d0 0x55962638d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:35.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.998+0000 7f959f010740 4 client.7224 mds_command: new command op to 5324 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-26T23:06:35.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.998+0000 7f959f010740 1 -- 172.21.15.40:0/1198552537 --> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5596263227e0 con 0x5596263231d0 2024-04-26T23:06:35.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:34.999+0000 7f9599800640 1 --2- 172.21.15.40:0/1198552537 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x5596263231d0 0x55962638d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:35.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f9599800640 1 --2- 172.21.15.40:0/1198552537 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x5596263231d0 0x55962638d010 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8276a54d6ce4809f server_cookie=7f20cd8608a90707 in_seq=0 out_seq=0 2024-04-26T23:06:35.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f957aa00640 10 client.7224 ms_handle_connect on v2:172.21.15.103:6835/2595375825 2024-04-26T23:06:35.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f957aa00640 1 -- 172.21.15.40:0/1198552537 <== mds.2 v2:172.21.15.103:6835/2595375825 1 ==== command_reply(tid 0: 0 ) ==== 8+0+296 (crc 0 0 0) 0x5596263227e0 con 0x5596263231d0 2024-04-26T23:06:35.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f957aa00640 10 client.7224 handle_command_reply: tid=0 2024-04-26T23:06:35.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f959f010740 1 client.7224 shutdown 2024-04-26T23:06:35.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f959f010740 1 -- 172.21.15.40:0/1198552537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 msgr2=0x55962630c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:35.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f959f010740 1 --2- 172.21.15.40:0/1198552537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x55962630c5b0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f9580057790 tx=0x7f9580089bb0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f959f010740 1 -- 172.21.15.40:0/1198552537 shutdown_connections 2024-04-26T23:06:35.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f959f010740 1 --2- 172.21.15.40:0/1198552537 >> [v2:172.21.15.103:6835/2595375825,v1:172.21.15.103:6837/2595375825] conn(0x5596263231d0 0x55962638d010 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f959f010740 1 --2- 172.21.15.40:0/1198552537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55962632d840 0x55962638cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f959f010740 1 --2- 172.21.15.40:0/1198552537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55962630d880 0x55962630c5b0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f959f010740 1 --2- 172.21.15.40:0/1198552537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55962630d4a0 0x559626306de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f957aa00640 0 client.7224 ms_handle_reset on v2:172.21.15.103:6835/2595375825 2024-04-26T23:06:35.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.000+0000 7f959f010740 1 -- 172.21.15.40:0/1198552537 >> 172.21.15.40:0/1198552537 conn(0x5596262e0100 msgr2=0x5596262e5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:35.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.001+0000 7f959f010740 1 -- 172.21.15.40:0/1198552537 shutdown_connections 2024-04-26T23:06:35.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.001+0000 7f959f010740 1 -- 172.21.15.40:0/1198552537 wait complete. 2024-04-26T23:06:35.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.001+0000 7f959f010740 20 client.7224 trim_cache size 0 max 16384 2024-04-26T23:06:35.004 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-04-26T23:06:35.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:35.004 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-26T23:06:35.004 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1466700392, 2024-04-26T23:06:35.004 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511627782, 2024-04-26T23:06:35.004 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3/d4/d5/d6/d7" 2024-04-26T23:06:35.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:35.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:35.004 INFO:teuthology.orchestra.run.smithi040.stdout: "damage_type": "backtrace", 2024-04-26T23:06:35.004 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 1592627377, 2024-04-26T23:06:35.004 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1099511627783, 2024-04-26T23:06:35.004 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3/d4/d5/d6/d7/d8" 2024-04-26T23:06:35.005 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:35.005 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-04-26T23:06:35.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.001+0000 7f959ac00640 1 -- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f957406c920 msgr2=0x7f957406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:35.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.001+0000 7f959ac00640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f957406c920 0x7f957406ede0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9588009990 tx=0x7f9588008040 comp rx=0 tx=0).stop 2024-04-26T23:06:35.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.001+0000 7f959ac00640 1 -- 172.21.15.40:0/3171491179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9594144160 msgr2=0x7f95940b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:35.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.001+0000 7f959ac00640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9594144160 0x7f95940b1520 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f958000e4b0 tx=0x7f958000e970 comp rx=0 tx=0).stop 2024-04-26T23:06:35.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.001+0000 7f959ac00640 1 -- 172.21.15.40:0/3171491179 shutdown_connections 2024-04-26T23:06:35.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.001+0000 7f959ac00640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f957406c920 0x7f957406ede0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.001+0000 7f959ac00640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95941445e0 0x7f95940b1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.001+0000 7f959ac00640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9594144160 0x7f95940b1520 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.001+0000 7f959ac00640 1 --2- 172.21.15.40:0/3171491179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f957c0051c0 0x7f9594134560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.001+0000 7f959ac00640 1 -- 172.21.15.40:0/3171491179 >> 172.21.15.40:0/3171491179 conn(0x7f95940b2720 msgr2=0x7f95940af150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:35.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.002+0000 7f959ac00640 1 -- 172.21.15.40:0/3171491179 shutdown_connections 2024-04-26T23:06:35.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.002+0000 7f959ac00640 1 -- 172.21.15.40:0/3171491179 wait complete. 2024-04-26T23:06:35.019 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-26T23:06:35.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:06:35.126 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:06:35.266 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-26T23:06:35.266 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:06:35.266 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:06:35.266 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:06:35.325 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:06:35.325 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:06:35.381 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:06:35.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.597+0000 7f8667200640 1 Processor -- start 2024-04-26T23:06:35.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.597+0000 7f8667200640 1 -- start start 2024-04-26T23:06:35.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.599+0000 7f8667200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8668071fe0 0x7f86680afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:35.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.599+0000 7f8667200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86680d0820 con 0x7f86680af7a0 2024-04-26T23:06:35.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.599+0000 7f8667200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86681614c0 con 0x7f866815ce60 2024-04-26T23:06:35.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.599+0000 7f8667200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f86680cc450 con 0x7f8668071fe0 2024-04-26T23:06:35.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.599+0000 7f865d400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f86680af7a0 0x7f8668073760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60324/0 (socket says 172.21.15.40:60324) 2024-04-26T23:06:35.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.599+0000 7f865d400640 1 -- 172.21.15.40:0/410740844 learned_addr learned my addr 172.21.15.40:0/410740844 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:35.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8666800640 1 --2- 172.21.15.40:0/410740844 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8668071fe0 0x7f86680afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:35.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1908419275 0 0) 0x7f86680d0820 con 0x7f86680af7a0 2024-04-26T23:06:35.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f864c003520 con 0x7f86680af7a0 2024-04-26T23:06:35.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 674865712 0 0) 0x7f864c003520 con 0x7f86680af7a0 2024-04-26T23:06:35.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8668071fe0 msgr2=0x7f86680afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:35.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 --2- 172.21.15.40:0/410740844 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8668071fe0 0x7f86680afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 >> v1:172.21.15.103:6789/0 conn(0x7f866815ce60 legacy=0x7f86680ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:35.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8668161760 con 0x7f86680af7a0 2024-04-26T23:06:35.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f8650001670 con 0x7f86680af7a0 2024-04-26T23:06:35.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 >> v1:172.21.15.40:6789/0 conn(0x7f86680af7a0 legacy=0x7f8668073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:35.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 --2- 172.21.15.40:0/410740844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f864c0039e0 0x7f864c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:35.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 --2- 172.21.15.40:0/410740844 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8668071fe0 0x7f864c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:35.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 --2- 172.21.15.40:0/410740844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f864c004d40 0x7f864c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:35.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f86680cc450 con 0x7f864c0039e0 2024-04-26T23:06:35.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f864c00bb20 con 0x7f864c004d40 2024-04-26T23:06:35.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f864c00bd20 con 0x7f8668071fe0 2024-04-26T23:06:35.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.600+0000 7f8665e00640 1 --2- 172.21.15.40:0/410740844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f864c0039e0 0x7f864c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:35.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.601+0000 7f8665e00640 1 -- 172.21.15.40:0/410740844 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8668071fe0 msgr2=0x7f864c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:35.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.601+0000 7f8665e00640 1 --2- 172.21.15.40:0/410740844 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8668071fe0 0x7f864c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.601+0000 7f8665e00640 1 -- 172.21.15.40:0/410740844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f864c004d40 msgr2=0x7f864c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:35.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.601+0000 7f8666800640 1 --2- 172.21.15.40:0/410740844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f864c004d40 0x7f864c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:35.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.601+0000 7f8665e00640 1 --2- 172.21.15.40:0/410740844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f864c004d40 0x7f864c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.601+0000 7f8665e00640 1 -- 172.21.15.40:0/410740844 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c003520 con 0x7f864c0039e0 2024-04-26T23:06:35.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.601+0000 7f8666800640 1 --2- 172.21.15.40:0/410740844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f864c004d40 0x7f864c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:35.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.602+0000 7f8665e00640 1 --2- 172.21.15.40:0/410740844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f864c0039e0 0x7f864c003e00 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f8658001e50 tx=0x7f864c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbefadbc7933828c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:35.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.602+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658001a70 con 0x7f864c0039e0 2024-04-26T23:06:35.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.602+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f865802ada0 con 0x7f864c0039e0 2024-04-26T23:06:35.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.602+0000 7f8665400640 1 -- 172.21.15.40:0/410740844 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658003bb0 con 0x7f864c0039e0 2024-04-26T23:06:35.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.603+0000 7f8667200640 1 -- 172.21.15.40:0/410740844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f864c0039e0 msgr2=0x7f864c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:35.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.603+0000 7f8667200640 1 --2- 172.21.15.40:0/410740844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f864c0039e0 0x7f864c003e00 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f8658001e50 tx=0x7f864c005830 comp rx=0 tx=0).stop 2024-04-26T23:06:35.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.603+0000 7f8666800640 1 -- 172.21.15.40:0/410740844 reap_dead start 2024-04-26T23:06:35.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.603+0000 7f8667200640 1 -- 172.21.15.40:0/410740844 shutdown_connections 2024-04-26T23:06:35.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.603+0000 7f8667200640 1 -- 172.21.15.40:0/410740844 >> 172.21.15.40:0/410740844 conn(0x7f86680d0a10 msgr2=0x7f86680d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:35.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.603+0000 7f8667200640 1 -- 172.21.15.40:0/410740844 shutdown_connections 2024-04-26T23:06:35.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.603+0000 7f8667200640 1 -- 172.21.15.40:0/410740844 wait complete. 2024-04-26T23:06:35.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.605+0000 7f8667200640 1 Processor -- start 2024-04-26T23:06:35.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.605+0000 7f8667200640 1 -- start start 2024-04-26T23:06:35.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.606+0000 7f8667200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f864c0039e0 0x7f866814deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:35.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.606+0000 7f8667200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f866814d690 0x7f86681f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:35.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.606+0000 7f8667200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86681f65f0 0x7f86681f6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:35.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.606+0000 7f8667200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f86680cc1f0 con 0x7f86681f65f0 2024-04-26T23:06:35.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.606+0000 7f8667200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f86680cc070 con 0x7f864c0039e0 2024-04-26T23:06:35.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.606+0000 7f8667200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8668073e70 con 0x7f866814d690 2024-04-26T23:06:35.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.606+0000 7f8666800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86681f65f0 0x7f86681f6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:35.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.606+0000 7f8666800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86681f65f0 0x7f86681f6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42914/0 (socket says 172.21.15.40:42914) 2024-04-26T23:06:35.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.606+0000 7f8666800640 1 -- 172.21.15.40:0/2080550837 learned_addr learned my addr 172.21.15.40:0/2080550837 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:35.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.607+0000 7f865d400640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f866814d690 0x7f86681f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:35.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.607+0000 7f8665e00640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f864c0039e0 0x7f866814deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:35.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.607+0000 7f8666800640 1 -- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f866814d690 msgr2=0x7f86681f60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:35.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.607+0000 7f8666800640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f866814d690 0x7f86681f60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.607+0000 7f8666800640 1 -- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f864c0039e0 msgr2=0x7f866814deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:35.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.607+0000 7f8666800640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f864c0039e0 0x7f866814deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.607+0000 7f8666800640 1 -- 172.21.15.40:0/2080550837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86681512b0 con 0x7f86681f65f0 2024-04-26T23:06:35.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.607+0000 7f865d400640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f866814d690 0x7f86681f60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:35.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.607+0000 7f8666800640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86681f65f0 0x7f86681f6a70 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f8668051040 tx=0x7f8660010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:35.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.607+0000 7f865ea00640 1 -- 172.21.15.40:0/2080550837 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866000f040 con 0x7f86681f65f0 2024-04-26T23:06:35.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.607+0000 7f865ea00640 1 -- 172.21.15.40:0/2080550837 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f866000e8f0 con 0x7f86681f65f0 2024-04-26T23:06:35.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.607+0000 7f8667200640 1 -- 172.21.15.40:0/2080550837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8668151590 con 0x7f86681f65f0 2024-04-26T23:06:35.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.607+0000 7f865ea00640 1 -- 172.21.15.40:0/2080550837 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866000ece0 con 0x7f86681f65f0 2024-04-26T23:06:35.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.608+0000 7f8667200640 1 -- 172.21.15.40:0/2080550837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f866814db10 con 0x7f86681f65f0 2024-04-26T23:06:35.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.609+0000 7f8667200640 1 -- 172.21.15.40:0/2080550837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8668069370 con 0x7f86681f65f0 2024-04-26T23:06:35.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.610+0000 7f865ea00640 1 -- 172.21.15.40:0/2080550837 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f866001c400 con 0x7f86681f65f0 2024-04-26T23:06:35.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.611+0000 7f865ea00640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f864406c920 0x7f864406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:35.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.611+0000 7f865ea00640 1 -- 172.21.15.40:0/2080550837 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f866008bee0 con 0x7f86681f65f0 2024-04-26T23:06:35.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.612+0000 7f8665e00640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f864406c920 0x7f864406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:35.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.613+0000 7f8665e00640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f864406c920 0x7f864406ede0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8658032000 tx=0x7f865802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:35.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.616+0000 7f865ea00640 1 -- 172.21.15.40:0/2080550837 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8660018030 con 0x7f86681f65f0 2024-04-26T23:06:35.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.823+0000 7f8667200640 1 -- 172.21.15.40:0/2080550837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8668151c20 con 0x7f86681f65f0 2024-04-26T23:06:35.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.825+0000 7f865ea00640 1 -- 172.21.15.40:0/2080550837 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+17040 (secure 0 0 0) 0x7f866005a100 con 0x7f86681f65f0 2024-04-26T23:06:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:35.826 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":67,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:05:48.861635+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:05:30.081199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-26T23:05:31.192158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.9","mappings":[{"from":7,"to":2}]},{"pgid":"10.10","mappings":[{"from":4,"to":3}]},{"pgid":"10.22","mappings":[{"from":7,"to":3}]},{"pgid":"10.35","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:06:35.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.828+0000 7f8667200640 1 -- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f864406c920 msgr2=0x7f864406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:35.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.828+0000 7f8667200640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f864406c920 0x7f864406ede0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8658032000 tx=0x7f865802e000 comp rx=0 tx=0).stop 2024-04-26T23:06:35.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.828+0000 7f8667200640 1 -- 172.21.15.40:0/2080550837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86681f65f0 msgr2=0x7f86681f6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:35.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.828+0000 7f8667200640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86681f65f0 0x7f86681f6a70 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f8668051040 tx=0x7f8660010c20 comp rx=0 tx=0).stop 2024-04-26T23:06:35.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.828+0000 7f8667200640 1 -- 172.21.15.40:0/2080550837 shutdown_connections 2024-04-26T23:06:35.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.828+0000 7f8667200640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f864406c920 0x7f864406ede0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.828+0000 7f8667200640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86681f65f0 0x7f86681f6a70 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.828+0000 7f8667200640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f866814d690 0x7f86681f60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.828+0000 7f8667200640 1 --2- 172.21.15.40:0/2080550837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f864c0039e0 0x7f866814deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:35.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.828+0000 7f8667200640 1 -- 172.21.15.40:0/2080550837 >> 172.21.15.40:0/2080550837 conn(0x7f86680d0a10 msgr2=0x7f86680b1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:35.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.829+0000 7f8667200640 1 -- 172.21.15.40:0/2080550837 shutdown_connections 2024-04-26T23:06:35.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:35.829+0000 7f8667200640 1 -- 172.21.15.40:0/2080550837 wait complete. 2024-04-26T23:06:35.848 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:05:30.081199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:05:31.192158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:06:35.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:06:36.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.058+0000 7fe821200640 1 Processor -- start 2024-04-26T23:06:36.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.058+0000 7fe821200640 1 -- start start 2024-04-26T23:06:36.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.060+0000 7fe821200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe81c1629b0 0x7fe81c0b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.060+0000 7fe821200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe81c0b6f40 0x7fe81c0b7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.060+0000 7fe821200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe81c0d3510 con 0x7fe81c1625d0 2024-04-26T23:06:36.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.060+0000 7fe821200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe81c0d3e40 con 0x7fe81c1629b0 2024-04-26T23:06:36.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.060+0000 7fe821200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe81c0d37f0 con 0x7fe81c0b6f40 2024-04-26T23:06:36.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.060+0000 7fe81b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe81c1625d0 0x7fe81c0b6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60334/0 (socket says 172.21.15.40:60334) 2024-04-26T23:06:36.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.060+0000 7fe81b400640 1 -- 172.21.15.40:0/3381139319 learned_addr learned my addr 172.21.15.40:0/3381139319 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.061+0000 7fe81aa00640 1 --2- 172.21.15.40:0/3381139319 >> v2:172.21.15.103:3300/0 conn(0x7fe81c1629b0 0x7fe81c0b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.061+0000 7fe81be00640 1 --2- 172.21.15.40:0/3381139319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe81c0b6f40 0x7fe81c0b7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.061+0000 7fe81a000640 1 -- 172.21.15.40:0/3381139319 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2961963878 0 0) 0x7fe81c0d3510 con 0x7fe81c1625d0 2024-04-26T23:06:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.061+0000 7fe81a000640 1 -- 172.21.15.40:0/3381139319 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe800003520 con 0x7fe81c1625d0 2024-04-26T23:06:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.062+0000 7fe81be00640 1 -- 172.21.15.40:0/3381139319 >> v2:172.21.15.103:3300/0 conn(0x7fe81c1629b0 msgr2=0x7fe81c0b5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.062+0000 7fe81be00640 1 --2- 172.21.15.40:0/3381139319 >> v2:172.21.15.103:3300/0 conn(0x7fe81c1629b0 0x7fe81c0b5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.062+0000 7fe81be00640 1 -- 172.21.15.40:0/3381139319 >> v1:172.21.15.40:6789/0 conn(0x7fe81c1625d0 legacy=0x7fe81c0b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.062+0000 7fe81be00640 1 -- 172.21.15.40:0/3381139319 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81c0d3ad0 con 0x7fe81c0b6f40 2024-04-26T23:06:36.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.063+0000 7fe81be00640 1 --2- 172.21.15.40:0/3381139319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe81c0b6f40 0x7fe81c0b7fe0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe814009d90 tx=0x7fe81402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=648273a24ee3022b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.064+0000 7fe81a000640 1 -- 172.21.15.40:0/3381139319 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814007740 con 0x7fe81c0b6f40 2024-04-26T23:06:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.064+0000 7fe81a000640 1 -- 172.21.15.40:0/3381139319 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8140078e0 con 0x7fe81c0b6f40 2024-04-26T23:06:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.064+0000 7fe81a000640 1 -- 172.21.15.40:0/3381139319 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814007c90 con 0x7fe81c0b6f40 2024-04-26T23:06:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.064+0000 7fe821200640 1 -- 172.21.15.40:0/3381139319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe81c0b6f40 msgr2=0x7fe81c0b7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.064+0000 7fe821200640 1 --2- 172.21.15.40:0/3381139319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe81c0b6f40 0x7fe81c0b7fe0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe814009d90 tx=0x7fe81402a660 comp rx=0 tx=0).stop 2024-04-26T23:06:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.064+0000 7fe821200640 1 -- 172.21.15.40:0/3381139319 shutdown_connections 2024-04-26T23:06:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.064+0000 7fe821200640 1 --2- 172.21.15.40:0/3381139319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe81c0b6f40 0x7fe81c0b7fe0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.064+0000 7fe821200640 1 --2- 172.21.15.40:0/3381139319 >> v2:172.21.15.103:3300/0 conn(0x7fe81c1629b0 0x7fe81c0b5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.065+0000 7fe821200640 1 -- 172.21.15.40:0/3381139319 >> 172.21.15.40:0/3381139319 conn(0x7fe81c0d4db0 msgr2=0x7fe81c0d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.065+0000 7fe821200640 1 -- 172.21.15.40:0/3381139319 shutdown_connections 2024-04-26T23:06:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.065+0000 7fe821200640 1 -- 172.21.15.40:0/3381139319 wait complete. 2024-04-26T23:06:36.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.067+0000 7fe821200640 1 Processor -- start 2024-04-26T23:06:36.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.067+0000 7fe821200640 1 -- start start 2024-04-26T23:06:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.068+0000 7fe821200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe81c0b6f40 0x7fe81c1f4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.068+0000 7fe821200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe81c1625d0 0x7fe81c1507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.068+0000 7fe821200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe81c1629b0 0x7fe81c160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.068+0000 7fe821200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c0b7510 con 0x7fe81c0b6f40 2024-04-26T23:06:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.068+0000 7fe821200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c0b7390 con 0x7fe81c1625d0 2024-04-26T23:06:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.068+0000 7fe821200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe81c0b7690 con 0x7fe81c1629b0 2024-04-26T23:06:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.068+0000 7fe81b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe81c0b6f40 0x7fe81c1f4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.068+0000 7fe81b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe81c0b6f40 0x7fe81c1f4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42922/0 (socket says 172.21.15.40:42922) 2024-04-26T23:06:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.068+0000 7fe81b400640 1 -- 172.21.15.40:0/542184355 learned_addr learned my addr 172.21.15.40:0/542184355 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.069+0000 7fe81aa00640 1 --2- 172.21.15.40:0/542184355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe81c1625d0 0x7fe81c1507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.069+0000 7fe81b400640 1 -- 172.21.15.40:0/542184355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe81c1629b0 msgr2=0x7fe81c160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.069+0000 7fe81b400640 1 --2- 172.21.15.40:0/542184355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe81c1629b0 0x7fe81c160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.069+0000 7fe81b400640 1 -- 172.21.15.40:0/542184355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe81c1625d0 msgr2=0x7fe81c1507c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.069+0000 7fe81b400640 1 --2- 172.21.15.40:0/542184355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe81c1625d0 0x7fe81c1507c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.069+0000 7fe81b400640 1 -- 172.21.15.40:0/542184355 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81c150d00 con 0x7fe81c0b6f40 2024-04-26T23:06:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.069+0000 7fe81b400640 1 --2- 172.21.15.40:0/542184355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe81c0b6f40 0x7fe81c1f4360 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00c430 tx=0x7fe80c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.069+0000 7fe813400640 1 -- 172.21.15.40:0/542184355 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c017070 con 0x7fe81c0b6f40 2024-04-26T23:06:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.069+0000 7fe813400640 1 -- 172.21.15.40:0/542184355 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe80c005ce0 con 0x7fe81c0b6f40 2024-04-26T23:06:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.069+0000 7fe821200640 1 -- 172.21.15.40:0/542184355 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81c1f8180 con 0x7fe81c0b6f40 2024-04-26T23:06:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.069+0000 7fe813400640 1 -- 172.21.15.40:0/542184355 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c0072f0 con 0x7fe81c0b6f40 2024-04-26T23:06:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.069+0000 7fe821200640 1 -- 172.21.15.40:0/542184355 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe81c1f8740 con 0x7fe81c0b6f40 2024-04-26T23:06:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.071+0000 7fe821200640 1 -- 172.21.15.40:0/542184355 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe81c1f8e00 con 0x7fe81c0b6f40 2024-04-26T23:06:36.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.072+0000 7fe813400640 1 -- 172.21.15.40:0/542184355 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe80c01a070 con 0x7fe81c0b6f40 2024-04-26T23:06:36.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.073+0000 7fe813400640 1 --2- 172.21.15.40:0/542184355 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe7fc06c8f0 0x7fe7fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.073+0000 7fe813400640 1 -- 172.21.15.40:0/542184355 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7fe80c08b2d0 con 0x7fe81c0b6f40 2024-04-26T23:06:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.074+0000 7fe81aa00640 1 --2- 172.21.15.40:0/542184355 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe7fc06c8f0 0x7fe7fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.075+0000 7fe81aa00640 1 --2- 172.21.15.40:0/542184355 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe7fc06c8f0 0x7fe7fc06edb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0a2cc0 tx=0x7fe808008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:36.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.078+0000 7fe813400640 1 -- 172.21.15.40:0/542184355 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe81c1f8e00 con 0x7fe81c0b6f40 2024-04-26T23:06:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.283+0000 7fe821200640 1 -- 172.21.15.40:0/542184355 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe81c069370 con 0x7fe81c0b6f40 2024-04-26T23:06:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.284+0000 7fe813400640 1 -- 172.21.15.40:0/542184355 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) ==== 93+0+10 (secure 0 0 0) 0x7fe81c150d00 con 0x7fe81c0b6f40 2024-04-26T23:06:36.285 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:06:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.286+0000 7fe821200640 1 -- 172.21.15.40:0/542184355 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe7fc06c8f0 msgr2=0x7fe7fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.286+0000 7fe821200640 1 --2- 172.21.15.40:0/542184355 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe7fc06c8f0 0x7fe7fc06edb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0a2cc0 tx=0x7fe808008040 comp rx=0 tx=0).stop 2024-04-26T23:06:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.286+0000 7fe821200640 1 -- 172.21.15.40:0/542184355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe81c0b6f40 msgr2=0x7fe81c1f4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.286+0000 7fe821200640 1 --2- 172.21.15.40:0/542184355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe81c0b6f40 0x7fe81c1f4360 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00c430 tx=0x7fe80c00c8f0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.286+0000 7fe821200640 1 -- 172.21.15.40:0/542184355 shutdown_connections 2024-04-26T23:06:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.286+0000 7fe821200640 1 --2- 172.21.15.40:0/542184355 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe7fc06c8f0 0x7fe7fc06edb0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.286+0000 7fe821200640 1 --2- 172.21.15.40:0/542184355 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe81c1629b0 0x7fe81c160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.286+0000 7fe821200640 1 --2- 172.21.15.40:0/542184355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe81c1625d0 0x7fe81c1507c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.286+0000 7fe821200640 1 --2- 172.21.15.40:0/542184355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe81c0b6f40 0x7fe81c1f4360 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.286+0000 7fe821200640 1 -- 172.21.15.40:0/542184355 >> 172.21.15.40:0/542184355 conn(0x7fe81c0d4db0 msgr2=0x7fe81c0d1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:36.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.287+0000 7fe821200640 1 -- 172.21.15.40:0/542184355 shutdown_connections 2024-04-26T23:06:36.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.287+0000 7fe821200640 1 -- 172.21.15.40:0/542184355 wait complete. 2024-04-26T23:06:36.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:06:36.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.512+0000 7f33ee800640 1 Processor -- start 2024-04-26T23:06:36.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.512+0000 7f33ee800640 1 -- start start 2024-04-26T23:06:36.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.514+0000 7f33ee800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33e8073600 0x7f33e8072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.514+0000 7f33ee800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33e80739e0 0x7f33e815f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.514+0000 7f33ee800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f33e815fcb0 0x7f33e80ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.514+0000 7f33ee800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f33e8073220 con 0x7f33e80739e0 2024-04-26T23:06:36.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.514+0000 7f33ee800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f33e8072850 con 0x7f33e815fcb0 2024-04-26T23:06:36.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.514+0000 7f33ee800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33e81604e0 con 0x7f33e8073600 2024-04-26T23:06:36.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.514+0000 7f33eca00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33e80739e0 0x7f33e815f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.515+0000 7f33eca00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33e80739e0 0x7f33e815f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42928/0 (socket says 172.21.15.40:42928) 2024-04-26T23:06:36.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.515+0000 7f33eca00640 1 -- 172.21.15.40:0/688212301 learned_addr learned my addr 172.21.15.40:0/688212301 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:36.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.515+0000 7f33ed400640 1 --2- 172.21.15.40:0/688212301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33e8073600 0x7f33e8072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.515+0000 7f33ede00640 1 --2- 172.21.15.40:0/688212301 >> v2:172.21.15.103:3300/0 conn(0x7f33e815fcb0 0x7f33e80ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.515+0000 7f33eca00640 1 -- 172.21.15.40:0/688212301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33e8073600 msgr2=0x7f33e8072ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:36.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.515+0000 7f33eca00640 1 --2- 172.21.15.40:0/688212301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33e8073600 0x7f33e8072ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.515+0000 7f33eca00640 1 -- 172.21.15.40:0/688212301 >> v2:172.21.15.103:3300/0 conn(0x7f33e815fcb0 msgr2=0x7f33e80ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:36.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.515+0000 7f33eca00640 1 --2- 172.21.15.40:0/688212301 >> v2:172.21.15.103:3300/0 conn(0x7f33e815fcb0 0x7f33e80ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.515+0000 7f33eca00640 1 -- 172.21.15.40:0/688212301 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e8072200 con 0x7f33e80739e0 2024-04-26T23:06:36.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.516+0000 7f33eca00640 1 --2- 172.21.15.40:0/688212301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e80739e0 0x7f33e815f770 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f33e80b0640 tx=0x7f33d402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb7e0dee539473aa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:36.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.516+0000 7f33e7e00640 1 -- 172.21.15.40:0/688212301 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d4037070 con 0x7f33e80739e0 2024-04-26T23:06:36.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.517+0000 7f33e7e00640 1 -- 172.21.15.40:0/688212301 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33d4032440 con 0x7f33e80739e0 2024-04-26T23:06:36.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.517+0000 7f33e7e00640 1 -- 172.21.15.40:0/688212301 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d4032810 con 0x7f33e80739e0 2024-04-26T23:06:36.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.517+0000 7f33ee800640 1 -- 172.21.15.40:0/688212301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e80739e0 msgr2=0x7f33e815f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.517+0000 7f33ee800640 1 --2- 172.21.15.40:0/688212301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e80739e0 0x7f33e815f770 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f33e80b0640 tx=0x7f33d402a1e0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.517+0000 7f33ee800640 1 -- 172.21.15.40:0/688212301 shutdown_connections 2024-04-26T23:06:36.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.517+0000 7f33ee800640 1 --2- 172.21.15.40:0/688212301 >> v2:172.21.15.103:3300/0 conn(0x7f33e815fcb0 0x7f33e80ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.517+0000 7f33ee800640 1 --2- 172.21.15.40:0/688212301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e80739e0 0x7f33e815f770 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.517+0000 7f33ee800640 1 --2- 172.21.15.40:0/688212301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33e8073600 0x7f33e8072ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.517+0000 7f33ee800640 1 -- 172.21.15.40:0/688212301 >> 172.21.15.40:0/688212301 conn(0x7f33e80b2720 msgr2=0x7f33e80b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:36.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.518+0000 7f33ee800640 1 -- 172.21.15.40:0/688212301 shutdown_connections 2024-04-26T23:06:36.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.518+0000 7f33ee800640 1 -- 172.21.15.40:0/688212301 wait complete. 2024-04-26T23:06:36.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.519+0000 7f33ee800640 1 Processor -- start 2024-04-26T23:06:36.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.520+0000 7f33ee800640 1 -- start start 2024-04-26T23:06:36.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ee800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33e8073600 0x7f33e8146d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ee800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e80739e0 0x7f33e81463c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ee800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815fcb0 0x7f33e80cf320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ee800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f33e8071e20 con 0x7f33e815fcb0 2024-04-26T23:06:36.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ee800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33e8073e60 con 0x7f33e80739e0 2024-04-26T23:06:36.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ee800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33e8071fa0 con 0x7f33e8073600 2024-04-26T23:06:36.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ede00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815fcb0 0x7f33e80cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ede00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815fcb0 0x7f33e80cf320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42942/0 (socket says 172.21.15.40:42942) 2024-04-26T23:06:36.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ede00640 1 -- 172.21.15.40:0/4036489650 learned_addr learned my addr 172.21.15.40:0/4036489650 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:36.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ed400640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33e8073600 0x7f33e8146d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33eca00640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e80739e0 0x7f33e81463c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ede00640 1 -- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33e8073600 msgr2=0x7f33e8146d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ede00640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33e8073600 0x7f33e8146d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ede00640 1 -- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e80739e0 msgr2=0x7f33e81463c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ede00640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e80739e0 0x7f33e81463c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.521+0000 7f33ede00640 1 -- 172.21.15.40:0/4036489650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e81607c0 con 0x7f33e815fcb0 2024-04-26T23:06:36.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.522+0000 7f33eca00640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e80739e0 0x7f33e81463c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:36.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.522+0000 7f33ede00640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815fcb0 0x7f33e80cf320 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f33e0007e60 tx=0x7f33e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:36.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.522+0000 7f33e5600640 1 -- 172.21.15.40:0/4036489650 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e0014420 con 0x7f33e815fcb0 2024-04-26T23:06:36.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.522+0000 7f33e5600640 1 -- 172.21.15.40:0/4036489650 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33e0019070 con 0x7f33e815fcb0 2024-04-26T23:06:36.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.522+0000 7f33e5600640 1 -- 172.21.15.40:0/4036489650 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e0014820 con 0x7f33e815fcb0 2024-04-26T23:06:36.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.522+0000 7f33ee800640 1 -- 172.21.15.40:0/4036489650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33e8160aa0 con 0x7f33e815fcb0 2024-04-26T23:06:36.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.522+0000 7f33ee800640 1 -- 172.21.15.40:0/4036489650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33e8161060 con 0x7f33e815fcb0 2024-04-26T23:06:36.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.523+0000 7f33ee800640 1 -- 172.21.15.40:0/4036489650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33e8069370 con 0x7f33e815fcb0 2024-04-26T23:06:36.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.525+0000 7f33e5600640 1 -- 172.21.15.40:0/4036489650 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f33e00149c0 con 0x7f33e815fcb0 2024-04-26T23:06:36.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.525+0000 7f33e5600640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f33c806c8f0 0x7f33c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.526+0000 7f33e5600640 1 -- 172.21.15.40:0/4036489650 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f33e008c650 con 0x7f33e815fcb0 2024-04-26T23:06:36.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.526+0000 7f33ed400640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f33c806c8f0 0x7f33c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.527+0000 7f33ed400640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f33c806c8f0 0x7f33c806edb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f33dc006fd0 tx=0x7f33dc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:36.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.530+0000 7f33e5600640 1 -- 172.21.15.40:0/4036489650 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f33e005a7e0 con 0x7f33e815fcb0 2024-04-26T23:06:36.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.732+0000 7f33ee800640 1 -- 172.21.15.40:0/4036489650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f33e80b3480 con 0x7f33e815fcb0 2024-04-26T23:06:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.733+0000 7f33e5600640 1 -- 172.21.15.40:0/4036489650 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) ==== 104+0+11 (secure 0 0 0) 0x7f33e005a0e0 con 0x7f33e815fcb0 2024-04-26T23:06:36.733 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:06:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.736+0000 7f33ee800640 1 -- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f33c806c8f0 msgr2=0x7f33c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.736+0000 7f33ee800640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f33c806c8f0 0x7f33c806edb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f33dc006fd0 tx=0x7f33dc008040 comp rx=0 tx=0).stop 2024-04-26T23:06:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.736+0000 7f33ee800640 1 -- 172.21.15.40:0/4036489650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815fcb0 msgr2=0x7f33e80cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.736+0000 7f33ee800640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815fcb0 0x7f33e80cf320 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f33e0007e60 tx=0x7f33e000e4b0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.736+0000 7f33ee800640 1 -- 172.21.15.40:0/4036489650 shutdown_connections 2024-04-26T23:06:36.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.736+0000 7f33ee800640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f33c806c8f0 0x7f33c806edb0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.736+0000 7f33ee800640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815fcb0 0x7f33e80cf320 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.736+0000 7f33ee800640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e80739e0 0x7f33e81463c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.736+0000 7f33ee800640 1 --2- 172.21.15.40:0/4036489650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33e8073600 0x7f33e8146d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.736+0000 7f33ee800640 1 -- 172.21.15.40:0/4036489650 >> 172.21.15.40:0/4036489650 conn(0x7f33e80b2720 msgr2=0x7f33e806f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:36.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.736+0000 7f33ee800640 1 -- 172.21.15.40:0/4036489650 shutdown_connections 2024-04-26T23:06:36.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.736+0000 7f33ee800640 1 -- 172.21.15.40:0/4036489650 wait complete. 2024-04-26T23:06:36.755 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:06:36.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.970+0000 7f8e1e200640 1 Processor -- start 2024-04-26T23:06:36.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.970+0000 7f8e1e200640 1 -- start start 2024-04-26T23:06:36.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.971+0000 7f8e1e200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8e1815f1a0 0x7f8e180b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.971+0000 7f8e1e200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e180d39d0 0x7f8e1815f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.971+0000 7f8e1e200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e180b8bd0 con 0x7f8e180d2ae0 2024-04-26T23:06:36.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.971+0000 7f8e1e200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f8e180bac80 con 0x7f8e180d39d0 2024-04-26T23:06:36.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.971+0000 7f8e1e200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8e180b8eb0 con 0x7f8e1815f1a0 2024-04-26T23:06:36.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.972+0000 7f8e1d800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e180d39d0 0x7f8e1815f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.972+0000 7f8e1d800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e180d39d0 0x7f8e1815f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42946/0 (socket says 172.21.15.40:42946) 2024-04-26T23:06:36.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.972+0000 7f8e1d800640 1 -- 172.21.15.40:0/4121113009 learned_addr learned my addr 172.21.15.40:0/4121113009 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:36.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.972+0000 7f8e1ce00640 1 --2- 172.21.15.40:0/4121113009 >> v2:172.21.15.103:3300/0 conn(0x7f8e1815f1a0 0x7f8e180b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.972+0000 7f8e1d800640 1 -- 172.21.15.40:0/4121113009 >> v2:172.21.15.103:3300/0 conn(0x7f8e1815f1a0 msgr2=0x7f8e180b85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:36.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.972+0000 7f8e1d800640 1 --2- 172.21.15.40:0/4121113009 >> v2:172.21.15.103:3300/0 conn(0x7f8e1815f1a0 0x7f8e180b85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.972+0000 7f8e1d800640 1 -- 172.21.15.40:0/4121113009 >> v1:172.21.15.103:6789/0 conn(0x7f8e180d2ae0 legacy=0x7f8e180d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:36.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.972+0000 7f8e1d800640 1 -- 172.21.15.40:0/4121113009 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e180b9190 con 0x7f8e180d39d0 2024-04-26T23:06:36.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.974+0000 7f8e1d800640 1 --2- 172.21.15.40:0/4121113009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e180d39d0 0x7f8e1815f580 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f8e1808d260 tx=0x7f8e1002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3bcc2630c0b9f2c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:36.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.974+0000 7f8e17400640 1 -- 172.21.15.40:0/4121113009 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e10037070 con 0x7f8e180d39d0 2024-04-26T23:06:36.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.974+0000 7f8e17400640 1 -- 172.21.15.40:0/4121113009 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e10032440 con 0x7f8e180d39d0 2024-04-26T23:06:36.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.974+0000 7f8e17400640 1 -- 172.21.15.40:0/4121113009 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e10032810 con 0x7f8e180d39d0 2024-04-26T23:06:36.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.974+0000 7f8e1e200640 1 -- 172.21.15.40:0/4121113009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e180d39d0 msgr2=0x7f8e1815f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.974+0000 7f8e1e200640 1 --2- 172.21.15.40:0/4121113009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e180d39d0 0x7f8e1815f580 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f8e1808d260 tx=0x7f8e1002a230 comp rx=0 tx=0).stop 2024-04-26T23:06:36.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.975+0000 7f8e1e200640 1 -- 172.21.15.40:0/4121113009 shutdown_connections 2024-04-26T23:06:36.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.975+0000 7f8e1e200640 1 --2- 172.21.15.40:0/4121113009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e180d39d0 0x7f8e1815f580 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.975+0000 7f8e1e200640 1 --2- 172.21.15.40:0/4121113009 >> v2:172.21.15.103:3300/0 conn(0x7f8e1815f1a0 0x7f8e180b85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.975+0000 7f8e1e200640 1 -- 172.21.15.40:0/4121113009 >> 172.21.15.40:0/4121113009 conn(0x7f8e180d4d90 msgr2=0x7f8e180d5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:36.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.975+0000 7f8e1e200640 1 -- 172.21.15.40:0/4121113009 shutdown_connections 2024-04-26T23:06:36.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.975+0000 7f8e1e200640 1 -- 172.21.15.40:0/4121113009 wait complete. 2024-04-26T23:06:36.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.977+0000 7f8e1e200640 1 Processor -- start 2024-04-26T23:06:36.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.977+0000 7f8e1e200640 1 -- start start 2024-04-26T23:06:36.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.978+0000 7f8e1e200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e180d2ae0 0x7f8e180ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.978+0000 7f8e1e200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e180d39d0 0x7f8e18161b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.978+0000 7f8e1e200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e1815f1a0 0x7f8e180d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.978+0000 7f8e1e200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8e180d0bb0 con 0x7f8e180d39d0 2024-04-26T23:06:36.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.978+0000 7f8e1e200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e180b9750 con 0x7f8e180d2ae0 2024-04-26T23:06:36.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.978+0000 7f8e1e200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8e180b9c70 con 0x7f8e1815f1a0 2024-04-26T23:06:36.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.978+0000 7f8e17e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e180d39d0 0x7f8e18161b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.978+0000 7f8e17e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e180d39d0 0x7f8e18161b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42950/0 (socket says 172.21.15.40:42950) 2024-04-26T23:06:36.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.978+0000 7f8e17e00640 1 -- 172.21.15.40:0/1456746861 learned_addr learned my addr 172.21.15.40:0/1456746861 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:36.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.978+0000 7f8e17e00640 1 -- 172.21.15.40:0/1456746861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e1815f1a0 msgr2=0x7f8e180d16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.978+0000 7f8e17e00640 1 --2- 172.21.15.40:0/1456746861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e1815f1a0 0x7f8e180d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.979+0000 7f8e17e00640 1 -- 172.21.15.40:0/1456746861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e180d2ae0 msgr2=0x7f8e180ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:36.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.979+0000 7f8e17e00640 1 --2- 172.21.15.40:0/1456746861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e180d2ae0 0x7f8e180ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:36.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.979+0000 7f8e17e00640 1 -- 172.21.15.40:0/1456746861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e181f5d40 con 0x7f8e180d39d0 2024-04-26T23:06:36.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.979+0000 7f8e17e00640 1 --2- 172.21.15.40:0/1456746861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e180d39d0 0x7f8e18161b40 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000c440 tx=0x7f8e0000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:36.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.979+0000 7f8e14c00640 1 -- 172.21.15.40:0/1456746861 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00017070 con 0x7f8e180d39d0 2024-04-26T23:06:36.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.979+0000 7f8e14c00640 1 -- 172.21.15.40:0/1456746861 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e000035f0 con 0x7f8e180d39d0 2024-04-26T23:06:36.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.979+0000 7f8e14c00640 1 -- 172.21.15.40:0/1456746861 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e000039e0 con 0x7f8e180d39d0 2024-04-26T23:06:36.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.979+0000 7f8e1e200640 1 -- 172.21.15.40:0/1456746861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e181f5a60 con 0x7f8e180d39d0 2024-04-26T23:06:36.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.979+0000 7f8e1e200640 1 -- 172.21.15.40:0/1456746861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e180c9c70 con 0x7f8e180d39d0 2024-04-26T23:06:36.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.980+0000 7f8e1e200640 1 -- 172.21.15.40:0/1456746861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e18069370 con 0x7f8e180d39d0 2024-04-26T23:06:36.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.982+0000 7f8e14c00640 1 -- 172.21.15.40:0/1456746861 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8e00003b80 con 0x7f8e180d39d0 2024-04-26T23:06:36.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.982+0000 7f8e14c00640 1 --2- 172.21.15.40:0/1456746861 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8df406c920 0x7f8df406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:36.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.983+0000 7f8e14c00640 1 -- 172.21.15.40:0/1456746861 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f8e0008afd0 con 0x7f8e180d39d0 2024-04-26T23:06:36.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.983+0000 7f8e1ce00640 1 --2- 172.21.15.40:0/1456746861 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8df406c920 0x7f8df406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:36.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.984+0000 7f8e1ce00640 1 --2- 172.21.15.40:0/1456746861 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8df406c920 0x7f8df406ede0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8e080070b0 tx=0x7f8e08007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:36.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:36.987+0000 7f8e14c00640 1 -- 172.21.15.40:0/1456746861 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8e00059160 con 0x7f8e180d39d0 2024-04-26T23:06:37.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.196+0000 7f8e1e200640 1 -- 172.21.15.40:0/1456746861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8e1806cb60 con 0x7f8e180d39d0 2024-04-26T23:06:37.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.197+0000 7f8e14c00640 1 -- 172.21.15.40:0/1456746861 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) ==== 100+0+11 (secure 0 0 0) 0x7f8e000010c0 con 0x7f8e180d39d0 2024-04-26T23:06:37.197 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:06:37.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.199+0000 7f8e1e200640 1 -- 172.21.15.40:0/1456746861 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8df406c920 msgr2=0x7f8df406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:37.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.199+0000 7f8e1e200640 1 --2- 172.21.15.40:0/1456746861 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8df406c920 0x7f8df406ede0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8e080070b0 tx=0x7f8e08007040 comp rx=0 tx=0).stop 2024-04-26T23:06:37.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.199+0000 7f8e1e200640 1 -- 172.21.15.40:0/1456746861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e180d39d0 msgr2=0x7f8e18161b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:37.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.199+0000 7f8e1e200640 1 --2- 172.21.15.40:0/1456746861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e180d39d0 0x7f8e18161b40 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000c440 tx=0x7f8e0000c900 comp rx=0 tx=0).stop 2024-04-26T23:06:37.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.199+0000 7f8e1e200640 1 -- 172.21.15.40:0/1456746861 shutdown_connections 2024-04-26T23:06:37.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.199+0000 7f8e1e200640 1 --2- 172.21.15.40:0/1456746861 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8df406c920 0x7f8df406ede0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:37.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.199+0000 7f8e1e200640 1 --2- 172.21.15.40:0/1456746861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e1815f1a0 0x7f8e180d16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:37.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.199+0000 7f8e1e200640 1 --2- 172.21.15.40:0/1456746861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e180d39d0 0x7f8e18161b40 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:37.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.199+0000 7f8e1e200640 1 --2- 172.21.15.40:0/1456746861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e180d2ae0 0x7f8e180ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:37.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.199+0000 7f8e1e200640 1 -- 172.21.15.40:0/1456746861 >> 172.21.15.40:0/1456746861 conn(0x7f8e180d4d90 msgr2=0x7f8e1806d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:37.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.200+0000 7f8e1e200640 1 -- 172.21.15.40:0/1456746861 shutdown_connections 2024-04-26T23:06:37.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.200+0000 7f8e1e200640 1 -- 172.21.15.40:0/1456746861 wait complete. 2024-04-26T23:06:37.219 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:06:37.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.432+0000 7fe437200640 1 Processor -- start 2024-04-26T23:06:37.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.432+0000 7fe437200640 1 -- start start 2024-04-26T23:06:37.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.433+0000 7fe437200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe4380b6220 0x7fe4380d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:37.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.433+0000 7fe437200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe438162c40 con 0x7fe4380d08d0 2024-04-26T23:06:37.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.433+0000 7fe437200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4380d2d80 con 0x7fe4380d39f0 2024-04-26T23:06:37.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.433+0000 7fe437200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe4380b7c80 con 0x7fe4380b6220 2024-04-26T23:06:37.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.434+0000 7fe435400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe4380d08d0 0x7fe4380d32e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60338/0 (socket says 172.21.15.40:60338) 2024-04-26T23:06:37.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.434+0000 7fe435e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe4380b6220 0x7fe4380d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:37.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.434+0000 7fe435400640 1 -- 172.21.15.40:0/894380519 learned_addr learned my addr 172.21.15.40:0/894380519 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:37.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.435+0000 7fe434a00640 1 -- 172.21.15.40:0/894380519 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3704841637 0 0) 0x7fe438162c40 con 0x7fe4380d08d0 2024-04-26T23:06:37.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.435+0000 7fe434a00640 1 -- 172.21.15.40:0/894380519 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe418003520 con 0x7fe4380d08d0 2024-04-26T23:06:37.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.435+0000 7fe435e00640 1 -- 172.21.15.40:0/894380519 >> v1:172.21.15.103:6789/0 conn(0x7fe4380d39f0 legacy=0x7fe4380b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:37.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.435+0000 7fe435e00640 1 -- 172.21.15.40:0/894380519 >> v1:172.21.15.40:6789/0 conn(0x7fe4380d08d0 legacy=0x7fe4380d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:37.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.435+0000 7fe435e00640 1 -- 172.21.15.40:0/894380519 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4380b7f60 con 0x7fe4380b6220 2024-04-26T23:06:37.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.436+0000 7fe435e00640 1 --2- 172.21.15.40:0/894380519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4380b6220 0x7fe4380d0390 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fe4280097a0 tx=0x7fe42802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d172611692c1fb55 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:37.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.436+0000 7fe434a00640 1 -- 172.21.15.40:0/894380519 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe428037070 con 0x7fe4380b6220 2024-04-26T23:06:37.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.436+0000 7fe434a00640 1 -- 172.21.15.40:0/894380519 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe428032440 con 0x7fe4380b6220 2024-04-26T23:06:37.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.436+0000 7fe434a00640 1 -- 172.21.15.40:0/894380519 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4280327f0 con 0x7fe4380b6220 2024-04-26T23:06:37.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.437+0000 7fe437200640 1 -- 172.21.15.40:0/894380519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4380b6220 msgr2=0x7fe4380d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:37.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.437+0000 7fe437200640 1 --2- 172.21.15.40:0/894380519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4380b6220 0x7fe4380d0390 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fe4280097a0 tx=0x7fe42802a230 comp rx=0 tx=0).stop 2024-04-26T23:06:37.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.437+0000 7fe437200640 1 -- 172.21.15.40:0/894380519 shutdown_connections 2024-04-26T23:06:37.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.437+0000 7fe437200640 1 --2- 172.21.15.40:0/894380519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4380b6220 0x7fe4380d0390 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:37.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.437+0000 7fe437200640 1 -- 172.21.15.40:0/894380519 >> 172.21.15.40:0/894380519 conn(0x7fe4380d4d90 msgr2=0x7fe4380d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:37.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.437+0000 7fe437200640 1 -- 172.21.15.40:0/894380519 shutdown_connections 2024-04-26T23:06:37.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.437+0000 7fe437200640 1 -- 172.21.15.40:0/894380519 wait complete. 2024-04-26T23:06:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.439+0000 7fe437200640 1 Processor -- start 2024-04-26T23:06:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.439+0000 7fe437200640 1 -- start start 2024-04-26T23:06:37.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.440+0000 7fe437200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4380b6220 0x7fe43815bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:37.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.440+0000 7fe437200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4380d08d0 0x7fe43815f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:37.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.440+0000 7fe437200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4380d39f0 0x7fe4380b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:37.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.440+0000 7fe437200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe4380b9430 con 0x7fe4380d39f0 2024-04-26T23:06:37.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.440+0000 7fe437200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4380b8520 con 0x7fe4380d08d0 2024-04-26T23:06:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.440+0000 7fe437200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe4380b95b0 con 0x7fe4380b6220 2024-04-26T23:06:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.441+0000 7fe436800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4380d39f0 0x7fe4380b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.441+0000 7fe436800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4380d39f0 0x7fe4380b6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42964/0 (socket says 172.21.15.40:42964) 2024-04-26T23:06:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.441+0000 7fe436800640 1 -- 172.21.15.40:0/1386161037 learned_addr learned my addr 172.21.15.40:0/1386161037 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.441+0000 7fe435400640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4380d08d0 0x7fe43815f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.441+0000 7fe435e00640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4380b6220 0x7fe43815bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.441+0000 7fe436800640 1 -- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4380b6220 msgr2=0x7fe43815bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.441+0000 7fe436800640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4380b6220 0x7fe43815bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.441+0000 7fe436800640 1 -- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4380d08d0 msgr2=0x7fe43815f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.441+0000 7fe436800640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4380d08d0 0x7fe43815f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.441+0000 7fe436800640 1 -- 172.21.15.40:0/1386161037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43809ee00 con 0x7fe4380d39f0 2024-04-26T23:06:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.441+0000 7fe435e00640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4380b6220 0x7fe43815bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.441+0000 7fe436800640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4380d39f0 0x7fe4380b6810 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fe43000c390 tx=0x7fe43000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:37.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.442+0000 7fe42e000640 1 -- 172.21.15.40:0/1386161037 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe430017070 con 0x7fe4380d39f0 2024-04-26T23:06:37.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.442+0000 7fe42e000640 1 -- 172.21.15.40:0/1386161037 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4300052d0 con 0x7fe4380d39f0 2024-04-26T23:06:37.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.442+0000 7fe42e000640 1 -- 172.21.15.40:0/1386161037 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe430003810 con 0x7fe4380d39f0 2024-04-26T23:06:37.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.442+0000 7fe437200640 1 -- 172.21.15.40:0/1386161037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe43809f0e0 con 0x7fe4380d39f0 2024-04-26T23:06:37.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.442+0000 7fe437200640 1 -- 172.21.15.40:0/1386161037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe43809f6a0 con 0x7fe4380d39f0 2024-04-26T23:06:37.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.443+0000 7fe437200640 1 -- 172.21.15.40:0/1386161037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe438069370 con 0x7fe4380d39f0 2024-04-26T23:06:37.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.444+0000 7fe42e000640 1 -- 172.21.15.40:0/1386161037 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe43001a070 con 0x7fe4380d39f0 2024-04-26T23:06:37.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.445+0000 7fe42e000640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe41406c8f0 0x7fe41406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:37.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.445+0000 7fe42e000640 1 -- 172.21.15.40:0/1386161037 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7fe43008b060 con 0x7fe4380d39f0 2024-04-26T23:06:37.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.446+0000 7fe435e00640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe41406c8f0 0x7fe41406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:37.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.447+0000 7fe435e00640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe41406c8f0 0x7fe41406edb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe428034000 tx=0x7fe42802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:37.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.449+0000 7fe42e000640 1 -- 172.21.15.40:0/1386161037 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe430059280 con 0x7fe4380d39f0 2024-04-26T23:06:37.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.669+0000 7fe437200640 1 -- 172.21.15.40:0/1386161037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe4381628e0 con 0x7fe4380d39f0 2024-04-26T23:06:37.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.670+0000 7fe42e000640 1 -- 172.21.15.40:0/1386161037 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v67) ==== 80+0+0 (secure 0 0 0) 0x7fe430058b80 con 0x7fe4380d39f0 2024-04-26T23:06:37.670 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-04-26T23:06:37.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.672+0000 7fe437200640 1 -- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe41406c8f0 msgr2=0x7fe41406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:37.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.672+0000 7fe437200640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe41406c8f0 0x7fe41406edb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe428034000 tx=0x7fe42802e000 comp rx=0 tx=0).stop 2024-04-26T23:06:37.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.672+0000 7fe437200640 1 -- 172.21.15.40:0/1386161037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4380d39f0 msgr2=0x7fe4380b6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:37.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.672+0000 7fe437200640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4380d39f0 0x7fe4380b6810 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fe43000c390 tx=0x7fe43000c850 comp rx=0 tx=0).stop 2024-04-26T23:06:37.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.673+0000 7fe437200640 1 -- 172.21.15.40:0/1386161037 shutdown_connections 2024-04-26T23:06:37.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.673+0000 7fe437200640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe41406c8f0 0x7fe41406edb0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:37.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.673+0000 7fe437200640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4380d39f0 0x7fe4380b6810 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:37.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.673+0000 7fe437200640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4380d08d0 0x7fe43815f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.673+0000 7fe437200640 1 --2- 172.21.15.40:0/1386161037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4380b6220 0x7fe43815bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.673+0000 7fe437200640 1 -- 172.21.15.40:0/1386161037 >> 172.21.15.40:0/1386161037 conn(0x7fe4380d4d90 msgr2=0x7fe43806d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.673+0000 7fe437200640 1 -- 172.21.15.40:0/1386161037 shutdown_connections 2024-04-26T23:06:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:37.673+0000 7fe437200640 1 -- 172.21.15.40:0/1386161037 wait complete. 2024-04-26T23:06:37.691 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T23:06:37.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:06:37.691 DEBUG:teuthology.orchestra.run.smithi040:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-26T23:06:37.791 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:06:37.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:06:37.791 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T23:06:37.842 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:06:37.842 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:06:37.842 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:06:37.901 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:06:37.901 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:06:37.901 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:06:37.901 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:06:37.958 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:06:37.958 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:06:37.958 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:06:37.958 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:06:38.016 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:06:38.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:06:38.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.243+0000 7f9fa0800640 1 Processor -- start 2024-04-26T23:06:38.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.243+0000 7f9fa0800640 1 -- start start 2024-04-26T23:06:38.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.244+0000 7f9fa0800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f9815f5a0 0x7f9f980d5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:38.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.245+0000 7f9fa0800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9f98163610 0x7f9f980cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:38.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.245+0000 7f9fa0800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f980d1a70 con 0x7f9f980d2830 2024-04-26T23:06:38.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.245+0000 7f9fa0800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f9f98163d30 con 0x7f9f9815f5a0 2024-04-26T23:06:38.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.245+0000 7f9fa0800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9f980d1d50 con 0x7f9f98163610 2024-04-26T23:06:38.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.245+0000 7f9f9de00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f9815f5a0 0x7f9f980d5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:38.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.245+0000 7f9f9de00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f9815f5a0 0x7f9f980d5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42966/0 (socket says 172.21.15.40:42966) 2024-04-26T23:06:38.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.245+0000 7f9f9de00640 1 -- 172.21.15.40:0/3385844264 learned_addr learned my addr 172.21.15.40:0/3385844264 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:38.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.245+0000 7f9f9e800640 1 --2- 172.21.15.40:0/3385844264 >> v2:172.21.15.103:3300/0 conn(0x7f9f98163610 0x7f9f980cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:38.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.245+0000 7f9f9de00640 1 -- 172.21.15.40:0/3385844264 >> v2:172.21.15.103:3300/0 conn(0x7f9f98163610 msgr2=0x7f9f980cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:38.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.245+0000 7f9f9de00640 1 --2- 172.21.15.40:0/3385844264 >> v2:172.21.15.103:3300/0 conn(0x7f9f98163610 0x7f9f980cfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.245+0000 7f9f9de00640 1 -- 172.21.15.40:0/3385844264 >> v1:172.21.15.103:6789/0 conn(0x7f9f980d2830 legacy=0x7f9f98162f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:38.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.246+0000 7f9f9de00640 1 -- 172.21.15.40:0/3385844264 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f980d2030 con 0x7f9f9815f5a0 2024-04-26T23:06:38.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.246+0000 7f9f9e800640 1 --2- 172.21.15.40:0/3385844264 >> v2:172.21.15.103:3300/0 conn(0x7f9f98163610 0x7f9f980cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.247+0000 7f9f9de00640 1 --2- 172.21.15.40:0/3385844264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f9815f5a0 0x7f9f980d5cb0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f9f9808e3e0 tx=0x7f9f8c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c807398d84b10e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.247+0000 7f9f9ca00640 1 -- 172.21.15.40:0/3385844264 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c036070 con 0x7f9f9815f5a0 2024-04-26T23:06:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.247+0000 7f9f9ca00640 1 -- 172.21.15.40:0/3385844264 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f8c031600 con 0x7f9f9815f5a0 2024-04-26T23:06:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.247+0000 7f9f9ca00640 1 -- 172.21.15.40:0/3385844264 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c0319d0 con 0x7f9f9815f5a0 2024-04-26T23:06:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.248+0000 7f9fa0800640 1 -- 172.21.15.40:0/3385844264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f9815f5a0 msgr2=0x7f9f980d5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.248+0000 7f9fa0800640 1 --2- 172.21.15.40:0/3385844264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f9815f5a0 0x7f9f980d5cb0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f9f9808e3e0 tx=0x7f9f8c02a660 comp rx=0 tx=0).stop 2024-04-26T23:06:38.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.248+0000 7f9fa0800640 1 -- 172.21.15.40:0/3385844264 shutdown_connections 2024-04-26T23:06:38.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.248+0000 7f9fa0800640 1 --2- 172.21.15.40:0/3385844264 >> v2:172.21.15.103:3300/0 conn(0x7f9f98163610 0x7f9f980cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.248+0000 7f9fa0800640 1 --2- 172.21.15.40:0/3385844264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f9815f5a0 0x7f9f980d5cb0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.248+0000 7f9fa0800640 1 -- 172.21.15.40:0/3385844264 >> 172.21.15.40:0/3385844264 conn(0x7f9f980d5190 msgr2=0x7f9f980d55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:38.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.248+0000 7f9fa0800640 1 -- 172.21.15.40:0/3385844264 shutdown_connections 2024-04-26T23:06:38.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.248+0000 7f9fa0800640 1 -- 172.21.15.40:0/3385844264 wait complete. 2024-04-26T23:06:38.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.250+0000 7f9fa0800640 1 Processor -- start 2024-04-26T23:06:38.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.250+0000 7f9fa0800640 1 -- start start 2024-04-26T23:06:38.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.251+0000 7f9fa0800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f980d2830 0x7f9f9814d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.251+0000 7f9fa0800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f98163610 0x7f9f980cb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.251+0000 7f9fa0800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f980cbcc0 0x7f9f980b93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.251+0000 7f9fa0800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9f980d3c20 con 0x7f9f980cbcc0 2024-04-26T23:06:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.251+0000 7f9fa0800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9f980d4190 con 0x7f9f98163610 2024-04-26T23:06:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.251+0000 7f9fa0800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9f980d3da0 con 0x7f9f980d2830 2024-04-26T23:06:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.252+0000 7f9f9e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f980cbcc0 0x7f9f980b93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.252+0000 7f9f9e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f980cbcc0 0x7f9f980b93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42976/0 (socket says 172.21.15.40:42976) 2024-04-26T23:06:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.252+0000 7f9f9e800640 1 -- 172.21.15.40:0/3629819880 learned_addr learned my addr 172.21.15.40:0/3629819880 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.252+0000 7f9f9d400640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f98163610 0x7f9f980cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.252+0000 7f9f9de00640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f980d2830 0x7f9f9814d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.252+0000 7f9f9e800640 1 -- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f980d2830 msgr2=0x7f9f9814d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.252+0000 7f9f9e800640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f980d2830 0x7f9f9814d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.252+0000 7f9f9e800640 1 -- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f98163610 msgr2=0x7f9f980cb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.252+0000 7f9f9e800640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f98163610 0x7f9f980cb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.252+0000 7f9f9e800640 1 -- 172.21.15.40:0/3629819880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f981e9b30 con 0x7f9f980cbcc0 2024-04-26T23:06:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.252+0000 7f9f9e800640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f980cbcc0 0x7f9f980b93a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400c5d0 tx=0x7f9f9400ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.253+0000 7f9f92000640 1 -- 172.21.15.40:0/3629819880 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f94015070 con 0x7f9f980cbcc0 2024-04-26T23:06:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.253+0000 7f9f92000640 1 -- 172.21.15.40:0/3629819880 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f94004240 con 0x7f9f980cbcc0 2024-04-26T23:06:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.253+0000 7f9f92000640 1 -- 172.21.15.40:0/3629819880 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f94003610 con 0x7f9f980cbcc0 2024-04-26T23:06:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.253+0000 7f9fa0800640 1 -- 172.21.15.40:0/3629819880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f981e9e10 con 0x7f9f980cbcc0 2024-04-26T23:06:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.253+0000 7f9fa0800640 1 -- 172.21.15.40:0/3629819880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f9814da80 con 0x7f9f980cbcc0 2024-04-26T23:06:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.254+0000 7f9fa0800640 1 -- 172.21.15.40:0/3629819880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f980694f0 con 0x7f9f980cbcc0 2024-04-26T23:06:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.255+0000 7f9f92000640 1 -- 172.21.15.40:0/3629819880 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9f940037b0 con 0x7f9f980cbcc0 2024-04-26T23:06:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.256+0000 7f9f92000640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9f7c06c920 0x7f9f7c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.256+0000 7f9f92000640 1 -- 172.21.15.40:0/3629819880 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f9f9408bb20 con 0x7f9f980cbcc0 2024-04-26T23:06:38.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.257+0000 7f9f9de00640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9f7c06c920 0x7f9f7c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:38.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.258+0000 7f9f9de00640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9f7c06c920 0x7f9f7c06ede0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c033000 tx=0x7f9f8c009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:38.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.261+0000 7f9f92000640 1 -- 172.21.15.40:0/3629819880 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9f94059d40 con 0x7f9f980cbcc0 2024-04-26T23:06:38.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.505+0000 7f9fa0800640 1 -- 172.21.15.40:0/3629819880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f98068d30 con 0x7f9f980cbcc0 2024-04-26T23:06:38.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.506+0000 7f9f92000640 1 -- 172.21.15.40:0/3629819880 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+5232 (secure 0 0 0) 0x7f9f94059640 con 0x7f9f980cbcc0 2024-04-26T23:06:38.506 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:38.506 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":66,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/4194140680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4194140680},{"type":"v1","addr":"172.21.15.103:6836","nonce":4194140680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:05:32.212319+0000","modified":"2024-04-26T23:06:10.856590+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6774,"mds_1":5614,"mds_2":5324},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5324":{"gid":5324,"name":"b","rank":2,"incarnation":59,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2595375825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2595375825},{"type":"v1","addr":"172.21.15.103:6837","nonce":2595375825}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5614":{"gid":5614,"name":"a","rank":1,"incarnation":57,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6834/2117861921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2117861921},{"type":"v1","addr":"172.21.15.40:6834","nonce":2117861921}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/931483025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":931483025},{"type":"v1","addr":"172.21.15.40:6835","nonce":931483025}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6774,"qdb_cluster":[5324,5614,6774]},"id":4}]} 2024-04-26T23:06:38.507 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 66 2024-04-26T23:06:38.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.508+0000 7f9fa0800640 1 -- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9f7c06c920 msgr2=0x7f9f7c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:38.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.508+0000 7f9fa0800640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9f7c06c920 0x7f9f7c06ede0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c033000 tx=0x7f9f8c009000 comp rx=0 tx=0).stop 2024-04-26T23:06:38.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.508+0000 7f9fa0800640 1 -- 172.21.15.40:0/3629819880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f980cbcc0 msgr2=0x7f9f980b93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.508+0000 7f9fa0800640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f980cbcc0 0x7f9f980b93a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400c5d0 tx=0x7f9f9400ca90 comp rx=0 tx=0).stop 2024-04-26T23:06:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.508+0000 7f9fa0800640 1 -- 172.21.15.40:0/3629819880 shutdown_connections 2024-04-26T23:06:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.508+0000 7f9fa0800640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9f7c06c920 0x7f9f7c06ede0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.508+0000 7f9fa0800640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f980cbcc0 0x7f9f980b93a0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.508+0000 7f9fa0800640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f98163610 0x7f9f980cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.509+0000 7f9fa0800640 1 --2- 172.21.15.40:0/3629819880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f980d2830 0x7f9f9814d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.509+0000 7f9fa0800640 1 -- 172.21.15.40:0/3629819880 >> 172.21.15.40:0/3629819880 conn(0x7f9f980d5190 msgr2=0x7f9f980b5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:38.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.509+0000 7f9fa0800640 1 -- 172.21.15.40:0/3629819880 shutdown_connections 2024-04-26T23:06:38.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.509+0000 7f9fa0800640 1 -- 172.21.15.40:0/3629819880 wait complete. 2024-04-26T23:06:38.530 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:06:38.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.741+0000 7f0201800640 1 Processor -- start 2024-04-26T23:06:38.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.741+0000 7f0201800640 1 -- start start 2024-04-26T23:06:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.743+0000 7f0201800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01fc163430 0x7f01fc166f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.743+0000 7f0201800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f01fc1674a0 0x7f01fc167920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.743+0000 7f0201800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f01fc0c62c0 0x7f01fc0c6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.743+0000 7f0201800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f01fc0cb400 con 0x7f01fc1674a0 2024-04-26T23:06:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.743+0000 7f0201800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f01fc167e60 con 0x7f01fc0c62c0 2024-04-26T23:06:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.743+0000 7f0201800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f01fc163810 con 0x7f01fc163430 2024-04-26T23:06:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.743+0000 7f01fb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f01fc1674a0 0x7f01fc167920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.744+0000 7f01fb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f01fc1674a0 0x7f01fc167920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42990/0 (socket says 172.21.15.40:42990) 2024-04-26T23:06:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.744+0000 7f01fb400640 1 -- 172.21.15.40:0/2501033612 learned_addr learned my addr 172.21.15.40:0/2501033612 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.744+0000 7f0200e00640 1 --2- 172.21.15.40:0/2501033612 >> v2:172.21.15.103:3300/0 conn(0x7f01fc0c62c0 0x7f01fc0c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.744+0000 7f01fbe00640 1 --2- 172.21.15.40:0/2501033612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01fc163430 0x7f01fc166f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.744+0000 7f01fb400640 1 -- 172.21.15.40:0/2501033612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01fc163430 msgr2=0x7f01fc166f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.744+0000 7f01fb400640 1 --2- 172.21.15.40:0/2501033612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01fc163430 0x7f01fc166f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.744+0000 7f01fb400640 1 -- 172.21.15.40:0/2501033612 >> v2:172.21.15.103:3300/0 conn(0x7f01fc0c62c0 msgr2=0x7f01fc0c6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.744+0000 7f01fb400640 1 --2- 172.21.15.40:0/2501033612 >> v2:172.21.15.103:3300/0 conn(0x7f01fc0c62c0 0x7f01fc0c6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.744+0000 7f01fb400640 1 -- 172.21.15.40:0/2501033612 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01fc0cb100 con 0x7f01fc1674a0 2024-04-26T23:06:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.745+0000 7f01fb400640 1 --2- 172.21.15.40:0/2501033612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc1674a0 0x7f01fc167920 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f01fc071ac0 tx=0x7f01e802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef48450b264243b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.745+0000 7f01faa00640 1 -- 172.21.15.40:0/2501033612 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8037070 con 0x7f01fc1674a0 2024-04-26T23:06:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.746+0000 7f01faa00640 1 -- 172.21.15.40:0/2501033612 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01e8032440 con 0x7f01fc1674a0 2024-04-26T23:06:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.746+0000 7f01faa00640 1 -- 172.21.15.40:0/2501033612 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8032810 con 0x7f01fc1674a0 2024-04-26T23:06:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.746+0000 7f0201800640 1 -- 172.21.15.40:0/2501033612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc1674a0 msgr2=0x7f01fc167920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.746+0000 7f0201800640 1 --2- 172.21.15.40:0/2501033612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc1674a0 0x7f01fc167920 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f01fc071ac0 tx=0x7f01e802a1e0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.746+0000 7f0201800640 1 -- 172.21.15.40:0/2501033612 shutdown_connections 2024-04-26T23:06:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.746+0000 7f0201800640 1 --2- 172.21.15.40:0/2501033612 >> v2:172.21.15.103:3300/0 conn(0x7f01fc0c62c0 0x7f01fc0c6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.746+0000 7f0201800640 1 --2- 172.21.15.40:0/2501033612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc1674a0 0x7f01fc167920 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.746+0000 7f0201800640 1 --2- 172.21.15.40:0/2501033612 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01fc163430 0x7f01fc166f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.746+0000 7f0201800640 1 -- 172.21.15.40:0/2501033612 >> 172.21.15.40:0/2501033612 conn(0x7f01fc0d0a10 msgr2=0x7f01fc0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.746+0000 7f0201800640 1 -- 172.21.15.40:0/2501033612 shutdown_connections 2024-04-26T23:06:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.746+0000 7f0201800640 1 -- 172.21.15.40:0/2501033612 wait complete. 2024-04-26T23:06:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.748+0000 7f0201800640 1 Processor -- start 2024-04-26T23:06:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.748+0000 7f0201800640 1 -- start start 2024-04-26T23:06:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.749+0000 7f0201800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc0c62c0 0x7f01fc14c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f0201800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc163430 0x7f01fc1ecda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f0201800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01fc1674a0 0x7f01fc0c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f0201800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc0afe80 con 0x7f01fc163430 2024-04-26T23:06:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f0201800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc0ccc60 con 0x7f01fc0c62c0 2024-04-26T23:06:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f0201800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f01fc0cd1e0 con 0x7f01fc1674a0 2024-04-26T23:06:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f01fb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc163430 0x7f01fc1ecda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f01fb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc163430 0x7f01fc1ecda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43004/0 (socket says 172.21.15.40:43004) 2024-04-26T23:06:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f01fb400640 1 -- 172.21.15.40:0/1368791505 learned_addr learned my addr 172.21.15.40:0/1368791505 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f01fbe00640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc0c62c0 0x7f01fc14c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f0200e00640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01fc1674a0 0x7f01fc0c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f01fb400640 1 -- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01fc1674a0 msgr2=0x7f01fc0c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f01fb400640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01fc1674a0 0x7f01fc0c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f01fb400640 1 -- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc0c62c0 msgr2=0x7f01fc14c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f01fb400640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc0c62c0 0x7f01fc14c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.750+0000 7f01fb400640 1 -- 172.21.15.40:0/1368791505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01fc1ed840 con 0x7f01fc163430 2024-04-26T23:06:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.751+0000 7f01fb400640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc163430 0x7f01fc1ecda0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f01e802a220 tx=0x7f01e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.751+0000 7f01f3e00640 1 -- 172.21.15.40:0/1368791505 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8041020 con 0x7f01fc163430 2024-04-26T23:06:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.751+0000 7f01f3e00640 1 -- 172.21.15.40:0/1368791505 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01e80306c0 con 0x7f01fc163430 2024-04-26T23:06:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.751+0000 7f01f3e00640 1 -- 172.21.15.40:0/1368791505 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8037040 con 0x7f01fc163430 2024-04-26T23:06:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.751+0000 7f0201800640 1 -- 172.21.15.40:0/1368791505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01fc1edb20 con 0x7f01fc163430 2024-04-26T23:06:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.751+0000 7f0201800640 1 -- 172.21.15.40:0/1368791505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01fc163c30 con 0x7f01fc163430 2024-04-26T23:06:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.753+0000 7f0201800640 1 -- 172.21.15.40:0/1368791505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01fc069370 con 0x7f01fc163430 2024-04-26T23:06:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.754+0000 7f01f3e00640 1 -- 172.21.15.40:0/1368791505 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f01e8044020 con 0x7f01fc163430 2024-04-26T23:06:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.755+0000 7f01f3e00640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f01dc06c940 0x7f01dc06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.755+0000 7f01f3e00640 1 -- 172.21.15.40:0/1368791505 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f01e80ac540 con 0x7f01fc163430 2024-04-26T23:06:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.755+0000 7f01fbe00640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f01dc06c940 0x7f01dc06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:38.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.757+0000 7f01fbe00640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f01dc06c940 0x7f01dc06ee00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f01ec006fd0 tx=0x7f01ec008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:38.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.759+0000 7f01f3e00640 1 -- 172.21.15.40:0/1368791505 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f01e804b030 con 0x7f01fc163430 2024-04-26T23:06:38.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.971+0000 7f0201800640 1 -- 172.21.15.40:0/1368791505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01fc0ce5e0 con 0x7f01fc163430 2024-04-26T23:06:38.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.973+0000 7f01f3e00640 1 -- 172.21.15.40:0/1368791505 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+17040 (secure 0 0 0) 0x7f01e807a7e0 con 0x7f01fc163430 2024-04-26T23:06:38.974 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:38.974 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":67,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:05:48.861635+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:05:30.081199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-26T23:05:31.192158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.9","mappings":[{"from":7,"to":2}]},{"pgid":"10.10","mappings":[{"from":4,"to":3}]},{"pgid":"10.22","mappings":[{"from":7,"to":3}]},{"pgid":"10.35","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:06:38.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.976+0000 7f0201800640 1 -- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f01dc06c940 msgr2=0x7f01dc06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:38.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.976+0000 7f0201800640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f01dc06c940 0x7f01dc06ee00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f01ec006fd0 tx=0x7f01ec008040 comp rx=0 tx=0).stop 2024-04-26T23:06:38.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.976+0000 7f0201800640 1 -- 172.21.15.40:0/1368791505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc163430 msgr2=0x7f01fc1ecda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:38.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.976+0000 7f0201800640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc163430 0x7f01fc1ecda0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f01e802a220 tx=0x7f01e80046c0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.976+0000 7f0201800640 1 -- 172.21.15.40:0/1368791505 shutdown_connections 2024-04-26T23:06:38.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.976+0000 7f0201800640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f01dc06c940 0x7f01dc06ee00 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.976+0000 7f0201800640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01fc1674a0 0x7f01fc0c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.976+0000 7f0201800640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc163430 0x7f01fc1ecda0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.976+0000 7f0201800640 1 --2- 172.21.15.40:0/1368791505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc0c62c0 0x7f01fc14c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:38.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.976+0000 7f0201800640 1 -- 172.21.15.40:0/1368791505 >> 172.21.15.40:0/1368791505 conn(0x7f01fc0d0a10 msgr2=0x7f01fc0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:38.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.977+0000 7f0201800640 1 -- 172.21.15.40:0/1368791505 shutdown_connections 2024-04-26T23:06:38.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:38.977+0000 7f0201800640 1 -- 172.21.15.40:0/1368791505 wait complete. 2024-04-26T23:06:38.997 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:05:30.081199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:05:31.192158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:06:38.998 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:06:39.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.212+0000 7fe380c00640 1 Processor -- start 2024-04-26T23:06:39.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.212+0000 7fe380c00640 1 -- start start 2024-04-26T23:06:39.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.213+0000 7fe380c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe37c0b5500 0x7fe37c0b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:39.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.213+0000 7fe380c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe37c0d3dc0 con 0x7fe37c0b5120 2024-04-26T23:06:39.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.213+0000 7fe380c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe37c0b75b0 con 0x7fe37c0b59b0 2024-04-26T23:06:39.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.213+0000 7fe380c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe37c0d03b0 con 0x7fe37c0b5500 2024-04-26T23:06:39.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.214+0000 7fe373e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe37c0b5500 0x7fe37c0b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:39.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.214+0000 7fe37b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe37c0b5120 0x7fe37c0b6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60344/0 (socket says 172.21.15.40:60344) 2024-04-26T23:06:39.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.214+0000 7fe37b400640 1 -- 172.21.15.40:0/4155975557 learned_addr learned my addr 172.21.15.40:0/4155975557 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:39.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.214+0000 7fe373e00640 1 -- 172.21.15.40:0/4155975557 >> v1:172.21.15.103:6789/0 conn(0x7fe37c0b59b0 legacy=0x7fe37c0b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:39.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.214+0000 7fe373e00640 1 -- 172.21.15.40:0/4155975557 >> v1:172.21.15.40:6789/0 conn(0x7fe37c0b5120 legacy=0x7fe37c0b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:39.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.214+0000 7fe373e00640 1 -- 172.21.15.40:0/4155975557 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37c0d0600 con 0x7fe37c0b5500 2024-04-26T23:06:39.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.214+0000 7fe37aa00640 1 -- 172.21.15.40:0/4155975557 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3672898324 0 0) 0x7fe37c0d3dc0 con 0x7fe37c0b5120 2024-04-26T23:06:39.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.216+0000 7fe373e00640 1 --2- 172.21.15.40:0/4155975557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe37c0b5500 0x7fe37c0b5ee0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fe3640097d0 tx=0x7fe36402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a308d96427cfb754 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:39.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.216+0000 7fe37aa00640 1 -- 172.21.15.40:0/4155975557 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364037070 con 0x7fe37c0b5500 2024-04-26T23:06:39.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.216+0000 7fe37aa00640 1 -- 172.21.15.40:0/4155975557 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36402ae10 con 0x7fe37c0b5500 2024-04-26T23:06:39.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.216+0000 7fe37aa00640 1 -- 172.21.15.40:0/4155975557 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364032660 con 0x7fe37c0b5500 2024-04-26T23:06:39.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.217+0000 7fe380c00640 1 -- 172.21.15.40:0/4155975557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe37c0b5500 msgr2=0x7fe37c0b5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:39.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.217+0000 7fe380c00640 1 --2- 172.21.15.40:0/4155975557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe37c0b5500 0x7fe37c0b5ee0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fe3640097d0 tx=0x7fe36402a100 comp rx=0 tx=0).stop 2024-04-26T23:06:39.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.217+0000 7fe380c00640 1 -- 172.21.15.40:0/4155975557 shutdown_connections 2024-04-26T23:06:39.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.217+0000 7fe380c00640 1 --2- 172.21.15.40:0/4155975557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe37c0b5500 0x7fe37c0b5ee0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.217+0000 7fe380c00640 1 -- 172.21.15.40:0/4155975557 >> 172.21.15.40:0/4155975557 conn(0x7fe37c0d4db0 msgr2=0x7fe37c0d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:39.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.217+0000 7fe380c00640 1 -- 172.21.15.40:0/4155975557 shutdown_connections 2024-04-26T23:06:39.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.217+0000 7fe380c00640 1 -- 172.21.15.40:0/4155975557 wait complete. 2024-04-26T23:06:39.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.219+0000 7fe380c00640 1 Processor -- start 2024-04-26T23:06:39.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.219+0000 7fe380c00640 1 -- start start 2024-04-26T23:06:39.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe380c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe37c0b5120 0x7fe37c0ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:39.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe380c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe37c0b5500 0x7fe37c1f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:39.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe380c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe37c0b59b0 0x7fe37c1f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe380c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c0b6f40 con 0x7fe37c0b5500 2024-04-26T23:06:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe380c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c0d0bc0 con 0x7fe37c0b59b0 2024-04-26T23:06:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe380c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe37c0b70c0 con 0x7fe37c0b5120 2024-04-26T23:06:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe373e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe37c0b5500 0x7fe37c1f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe373e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe37c0b5500 0x7fe37c1f3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43024/0 (socket says 172.21.15.40:43024) 2024-04-26T23:06:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe373e00640 1 -- 172.21.15.40:0/3204969932 learned_addr learned my addr 172.21.15.40:0/3204969932 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe37b400640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe37c0b5120 0x7fe37c0ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe37be00640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe37c0b59b0 0x7fe37c1f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe373e00640 1 -- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe37c0b5120 msgr2=0x7fe37c0ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe373e00640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe37c0b5120 0x7fe37c0ca8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe373e00640 1 -- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe37c0b59b0 msgr2=0x7fe37c1f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe373e00640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe37c0b59b0 0x7fe37c1f4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe373e00640 1 -- 172.21.15.40:0/3204969932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37c0c93b0 con 0x7fe37c0b5500 2024-04-26T23:06:39.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.220+0000 7fe37be00640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe37c0b59b0 0x7fe37c1f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:39.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.221+0000 7fe373e00640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe37c0b5500 0x7fe37c1f3f30 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fe364009720 tx=0x7fe3640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:39.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.221+0000 7fe373400640 1 -- 172.21.15.40:0/3204969932 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364037040 con 0x7fe37c0b5500 2024-04-26T23:06:39.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.221+0000 7fe373400640 1 -- 172.21.15.40:0/3204969932 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe364004020 con 0x7fe37c0b5500 2024-04-26T23:06:39.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.221+0000 7fe380c00640 1 -- 172.21.15.40:0/3204969932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe37c0c9690 con 0x7fe37c0b5500 2024-04-26T23:06:39.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.221+0000 7fe373400640 1 -- 172.21.15.40:0/3204969932 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364004410 con 0x7fe37c0b5500 2024-04-26T23:06:39.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.221+0000 7fe380c00640 1 -- 172.21.15.40:0/3204969932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe37c0c9c50 con 0x7fe37c0b5500 2024-04-26T23:06:39.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.223+0000 7fe380c00640 1 -- 172.21.15.40:0/3204969932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe37c069370 con 0x7fe37c0b5500 2024-04-26T23:06:39.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.224+0000 7fe373400640 1 -- 172.21.15.40:0/3204969932 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe36403a040 con 0x7fe37c0b5500 2024-04-26T23:06:39.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.224+0000 7fe373400640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe35806c8f0 0x7fe35806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:39.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.225+0000 7fe373400640 1 -- 172.21.15.40:0/3204969932 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7fe3640ac490 con 0x7fe37c0b5500 2024-04-26T23:06:39.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.225+0000 7fe37b400640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe35806c8f0 0x7fe35806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:39.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.227+0000 7fe37b400640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe35806c8f0 0x7fe35806edb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe36c007fd0 tx=0x7fe36c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:39.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.229+0000 7fe373400640 1 -- 172.21.15.40:0/3204969932 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe36404b030 con 0x7fe37c0b5500 2024-04-26T23:06:39.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.432+0000 7fe380c00640 1 -- 172.21.15.40:0/3204969932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe37c074360 con 0x7fe37c0b5500 2024-04-26T23:06:39.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.433+0000 7fe373400640 1 -- 172.21.15.40:0/3204969932 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) ==== 93+0+10 (secure 0 0 0) 0x7fe364042030 con 0x7fe37c0b5500 2024-04-26T23:06:39.433 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:06:39.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.436+0000 7fe380c00640 1 -- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe35806c8f0 msgr2=0x7fe35806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:39.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.436+0000 7fe380c00640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe35806c8f0 0x7fe35806edb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe36c007fd0 tx=0x7fe36c009040 comp rx=0 tx=0).stop 2024-04-26T23:06:39.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.436+0000 7fe380c00640 1 -- 172.21.15.40:0/3204969932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe37c0b5500 msgr2=0x7fe37c1f3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:39.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.436+0000 7fe380c00640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe37c0b5500 0x7fe37c1f3f30 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fe364009720 tx=0x7fe3640046c0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.436+0000 7fe380c00640 1 -- 172.21.15.40:0/3204969932 shutdown_connections 2024-04-26T23:06:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.436+0000 7fe380c00640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe35806c8f0 0x7fe35806edb0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.436+0000 7fe380c00640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe37c0b59b0 0x7fe37c1f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.436+0000 7fe380c00640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe37c0b5500 0x7fe37c1f3f30 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.436+0000 7fe380c00640 1 --2- 172.21.15.40:0/3204969932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe37c0b5120 0x7fe37c0ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.436+0000 7fe380c00640 1 -- 172.21.15.40:0/3204969932 >> 172.21.15.40:0/3204969932 conn(0x7fe37c0d4db0 msgr2=0x7fe37c0cf880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.436+0000 7fe380c00640 1 -- 172.21.15.40:0/3204969932 shutdown_connections 2024-04-26T23:06:39.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.436+0000 7fe380c00640 1 -- 172.21.15.40:0/3204969932 wait complete. 2024-04-26T23:06:39.455 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:06:39.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.679+0000 7ff573400640 1 Processor -- start 2024-04-26T23:06:39.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.679+0000 7ff573400640 1 -- start start 2024-04-26T23:06:39.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.681+0000 7ff573400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff56c073b70 0x7ff56c0aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:39.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.681+0000 7ff573400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff56c0cbb50 0x7ff56c0af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:39.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.681+0000 7ff573400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff56c072c00 con 0x7ff56c15f380 2024-04-26T23:06:39.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.681+0000 7ff573400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff56c0729b0 con 0x7ff56c0cbb50 2024-04-26T23:06:39.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.681+0000 7ff573400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff56c072ee0 con 0x7ff56c073b70 2024-04-26T23:06:39.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.681+0000 7ff570a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff56c15f380 0x7ff56c0cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60354/0 (socket says 172.21.15.40:60354) 2024-04-26T23:06:39.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.681+0000 7ff570a00640 1 -- 172.21.15.40:0/3620902959 learned_addr learned my addr 172.21.15.40:0/3620902959 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:39.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.681+0000 7ff571400640 1 --2- 172.21.15.40:0/3620902959 >> v2:172.21.15.40:3300/0 conn(0x7ff56c0cbb50 0x7ff56c0af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:39.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.682+0000 7ff56be00640 1 --2- 172.21.15.40:0/3620902959 >> v2:172.21.15.103:3300/0 conn(0x7ff56c073b70 0x7ff56c0aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:39.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.682+0000 7ff56b400640 1 -- 172.21.15.40:0/3620902959 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1767728607 0 0) 0x7ff56c072c00 con 0x7ff56c15f380 2024-04-26T23:06:39.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.682+0000 7ff56b400640 1 -- 172.21.15.40:0/3620902959 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff554003520 con 0x7ff56c15f380 2024-04-26T23:06:39.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.682+0000 7ff571400640 1 -- 172.21.15.40:0/3620902959 >> v2:172.21.15.103:3300/0 conn(0x7ff56c073b70 msgr2=0x7ff56c0aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:39.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.682+0000 7ff571400640 1 --2- 172.21.15.40:0/3620902959 >> v2:172.21.15.103:3300/0 conn(0x7ff56c073b70 0x7ff56c0aecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.682+0000 7ff571400640 1 -- 172.21.15.40:0/3620902959 >> v1:172.21.15.40:6789/0 conn(0x7ff56c15f380 legacy=0x7ff56c0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:39.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.682+0000 7ff571400640 1 -- 172.21.15.40:0/3620902959 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56c0731c0 con 0x7ff56c0cbb50 2024-04-26T23:06:39.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.684+0000 7ff571400640 1 --2- 172.21.15.40:0/3620902959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56c0cbb50 0x7ff56c0af1f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0ce700 tx=0x7ff56402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae510657d9ad9d6f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:39.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.684+0000 7ff56b400640 1 -- 172.21.15.40:0/3620902959 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff564007740 con 0x7ff56c0cbb50 2024-04-26T23:06:39.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.684+0000 7ff56b400640 1 -- 172.21.15.40:0/3620902959 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5640078e0 con 0x7ff56c0cbb50 2024-04-26T23:06:39.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.684+0000 7ff56b400640 1 -- 172.21.15.40:0/3620902959 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff564007cb0 con 0x7ff56c0cbb50 2024-04-26T23:06:39.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.685+0000 7ff573400640 1 -- 172.21.15.40:0/3620902959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56c0cbb50 msgr2=0x7ff56c0af1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:39.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.685+0000 7ff573400640 1 --2- 172.21.15.40:0/3620902959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56c0cbb50 0x7ff56c0af1f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0ce700 tx=0x7ff56402a680 comp rx=0 tx=0).stop 2024-04-26T23:06:39.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.685+0000 7ff573400640 1 -- 172.21.15.40:0/3620902959 shutdown_connections 2024-04-26T23:06:39.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.685+0000 7ff573400640 1 --2- 172.21.15.40:0/3620902959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56c0cbb50 0x7ff56c0af1f0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.685+0000 7ff573400640 1 --2- 172.21.15.40:0/3620902959 >> v2:172.21.15.103:3300/0 conn(0x7ff56c073b70 0x7ff56c0aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.685+0000 7ff573400640 1 -- 172.21.15.40:0/3620902959 >> 172.21.15.40:0/3620902959 conn(0x7ff56c0d0be0 msgr2=0x7ff56c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:39.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.685+0000 7ff573400640 1 -- 172.21.15.40:0/3620902959 shutdown_connections 2024-04-26T23:06:39.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.685+0000 7ff573400640 1 -- 172.21.15.40:0/3620902959 wait complete. 2024-04-26T23:06:39.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.687+0000 7ff573400640 1 Processor -- start 2024-04-26T23:06:39.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.687+0000 7ff573400640 1 -- start start 2024-04-26T23:06:39.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.688+0000 7ff573400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56c0cbb50 0x7ff56c15b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:39.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.688+0000 7ff573400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c15f380 0x7ff56c0cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:39.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.688+0000 7ff573400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56c0afa20 0x7ff56c0cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:39.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.688+0000 7ff573400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c073620 con 0x7ff56c0cbb50 2024-04-26T23:06:39.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.688+0000 7ff573400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c0d08b0 con 0x7ff56c0afa20 2024-04-26T23:06:39.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.688+0000 7ff573400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff56c0737a0 con 0x7ff56c15f380 2024-04-26T23:06:39.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.688+0000 7ff570a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56c0cbb50 0x7ff56c15b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:39.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.688+0000 7ff570a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56c0cbb50 0x7ff56c15b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43036/0 (socket says 172.21.15.40:43036) 2024-04-26T23:06:39.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.689+0000 7ff570a00640 1 -- 172.21.15.40:0/3137330896 learned_addr learned my addr 172.21.15.40:0/3137330896 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:39.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.689+0000 7ff56be00640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c15f380 0x7ff56c0cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:39.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.689+0000 7ff570a00640 1 -- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c15f380 msgr2=0x7ff56c0cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:39.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.689+0000 7ff571400640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56c0afa20 0x7ff56c0cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:39.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.689+0000 7ff570a00640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c15f380 0x7ff56c0cd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.689+0000 7ff570a00640 1 -- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56c0afa20 msgr2=0x7ff56c0cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:39.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.689+0000 7ff570a00640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56c0afa20 0x7ff56c0cdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.689+0000 7ff570a00640 1 -- 172.21.15.40:0/3137330896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56c1e5410 con 0x7ff56c0cbb50 2024-04-26T23:06:39.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.689+0000 7ff56be00640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c15f380 0x7ff56c0cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:39.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.689+0000 7ff570a00640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56c0cbb50 0x7ff56c15b4b0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7ff56000c390 tx=0x7ff56000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:39.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.690+0000 7ff568c00640 1 -- 172.21.15.40:0/3137330896 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560017070 con 0x7ff56c0cbb50 2024-04-26T23:06:39.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.690+0000 7ff568c00640 1 -- 172.21.15.40:0/3137330896 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff560005ce0 con 0x7ff56c0cbb50 2024-04-26T23:06:39.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.690+0000 7ff568c00640 1 -- 172.21.15.40:0/3137330896 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5600072f0 con 0x7ff56c0cbb50 2024-04-26T23:06:39.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.690+0000 7ff573400640 1 -- 172.21.15.40:0/3137330896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff56c0c4cd0 con 0x7ff56c0cbb50 2024-04-26T23:06:39.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.690+0000 7ff573400640 1 -- 172.21.15.40:0/3137330896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff56c0c5290 con 0x7ff56c0cbb50 2024-04-26T23:06:39.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.691+0000 7ff573400640 1 -- 172.21.15.40:0/3137330896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff56c0694f0 con 0x7ff56c0cbb50 2024-04-26T23:06:39.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.692+0000 7ff568c00640 1 -- 172.21.15.40:0/3137330896 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff56001a070 con 0x7ff56c0cbb50 2024-04-26T23:06:39.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.693+0000 7ff568c00640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff54c06c920 0x7ff54c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:39.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.693+0000 7ff568c00640 1 -- 172.21.15.40:0/3137330896 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7ff56008b100 con 0x7ff56c0cbb50 2024-04-26T23:06:39.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.694+0000 7ff56be00640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff54c06c920 0x7ff54c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:39.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.695+0000 7ff56be00640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff54c06c920 0x7ff54c06ede0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0b6a90 tx=0x7ff558008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:39.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.699+0000 7ff568c00640 1 -- 172.21.15.40:0/3137330896 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff560059320 con 0x7ff56c0cbb50 2024-04-26T23:06:39.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.911+0000 7ff573400640 1 -- 172.21.15.40:0/3137330896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff56c1e55e0 con 0x7ff56c0cbb50 2024-04-26T23:06:39.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.911+0000 7ff568c00640 1 -- 172.21.15.40:0/3137330896 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) ==== 104+0+11 (secure 0 0 0) 0x7ff560058c20 con 0x7ff56c0cbb50 2024-04-26T23:06:39.912 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:06:39.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.914+0000 7ff573400640 1 -- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff54c06c920 msgr2=0x7ff54c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:39.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.914+0000 7ff573400640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff54c06c920 0x7ff54c06ede0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0b6a90 tx=0x7ff558008040 comp rx=0 tx=0).stop 2024-04-26T23:06:39.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.914+0000 7ff573400640 1 -- 172.21.15.40:0/3137330896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56c0cbb50 msgr2=0x7ff56c15b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:39.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.914+0000 7ff573400640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56c0cbb50 0x7ff56c15b4b0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7ff56000c390 tx=0x7ff56000c850 comp rx=0 tx=0).stop 2024-04-26T23:06:39.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.914+0000 7ff573400640 1 -- 172.21.15.40:0/3137330896 shutdown_connections 2024-04-26T23:06:39.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.914+0000 7ff573400640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff54c06c920 0x7ff54c06ede0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.914+0000 7ff573400640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56c0afa20 0x7ff56c0cdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.914+0000 7ff573400640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c15f380 0x7ff56c0cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.914+0000 7ff573400640 1 --2- 172.21.15.40:0/3137330896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56c0cbb50 0x7ff56c15b4b0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:39.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.914+0000 7ff573400640 1 -- 172.21.15.40:0/3137330896 >> 172.21.15.40:0/3137330896 conn(0x7ff56c0d0be0 msgr2=0x7ff56c15e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:39.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.914+0000 7ff573400640 1 -- 172.21.15.40:0/3137330896 shutdown_connections 2024-04-26T23:06:39.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:39.915+0000 7ff573400640 1 -- 172.21.15.40:0/3137330896 wait complete. 2024-04-26T23:06:39.935 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:06:40.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.153+0000 7f8fb3200640 1 Processor -- start 2024-04-26T23:06:40.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.153+0000 7f8fb3200640 1 -- start start 2024-04-26T23:06:40.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.154+0000 7f8fb3200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8fb415f1a0 0x7f8fb40b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:40.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.154+0000 7f8fb3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8fb40d2ae0 0x7f8fb415f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:40.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.154+0000 7f8fb3200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb40bac80 con 0x7f8fb40d32c0 2024-04-26T23:06:40.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.154+0000 7f8fb3200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f8fb40b8bd0 con 0x7f8fb415f1a0 2024-04-26T23:06:40.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.154+0000 7f8fb3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8fb40b8eb0 con 0x7f8fb40d2ae0 2024-04-26T23:06:40.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.155+0000 7f8fb2800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8fb40d32c0 0x7f8fb40cf910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60366/0 (socket says 172.21.15.40:60366) 2024-04-26T23:06:40.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.155+0000 7f8fb2800640 1 -- 172.21.15.40:0/3972226489 learned_addr learned my addr 172.21.15.40:0/3972226489 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:40.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.155+0000 7f8fb1e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8fb415f1a0 0x7f8fb40b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:40.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.155+0000 7f8fb1400640 1 --2- 172.21.15.40:0/3972226489 >> v2:172.21.15.103:3300/0 conn(0x7f8fb40d2ae0 0x7f8fb415f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:40.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.155+0000 7f8fb0a00640 1 -- 172.21.15.40:0/3972226489 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2841705760 0 0) 0x7f8fb40bac80 con 0x7f8fb40d32c0 2024-04-26T23:06:40.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.155+0000 7f8fb1e00640 1 -- 172.21.15.40:0/3972226489 >> v2:172.21.15.103:3300/0 conn(0x7f8fb40d2ae0 msgr2=0x7f8fb415f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:40.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.155+0000 7f8fb1e00640 1 --2- 172.21.15.40:0/3972226489 >> v2:172.21.15.103:3300/0 conn(0x7f8fb40d2ae0 0x7f8fb415f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.155+0000 7f8fb1e00640 1 -- 172.21.15.40:0/3972226489 >> v1:172.21.15.40:6789/0 conn(0x7f8fb40d32c0 legacy=0x7f8fb40cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:40.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.155+0000 7f8fb1e00640 1 -- 172.21.15.40:0/3972226489 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb40b9190 con 0x7f8fb415f1a0 2024-04-26T23:06:40.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.157+0000 7f8fb1e00640 1 --2- 172.21.15.40:0/3972226489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fb415f1a0 0x7f8fb40b85f0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40d0eb0 tx=0x7f8fa402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d915dbb66fc3556b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:40.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.157+0000 7f8fb0a00640 1 -- 172.21.15.40:0/3972226489 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4037070 con 0x7f8fb415f1a0 2024-04-26T23:06:40.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.157+0000 7f8fb0a00640 1 -- 172.21.15.40:0/3972226489 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fa4032440 con 0x7f8fb415f1a0 2024-04-26T23:06:40.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.157+0000 7f8fb0a00640 1 -- 172.21.15.40:0/3972226489 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4032810 con 0x7f8fb415f1a0 2024-04-26T23:06:40.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.158+0000 7f8fb3200640 1 -- 172.21.15.40:0/3972226489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fb415f1a0 msgr2=0x7f8fb40b85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:40.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.158+0000 7f8fb3200640 1 --2- 172.21.15.40:0/3972226489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fb415f1a0 0x7f8fb40b85f0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40d0eb0 tx=0x7f8fa402a1a0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.158+0000 7f8fb3200640 1 -- 172.21.15.40:0/3972226489 shutdown_connections 2024-04-26T23:06:40.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.158+0000 7f8fb3200640 1 --2- 172.21.15.40:0/3972226489 >> v2:172.21.15.103:3300/0 conn(0x7f8fb40d2ae0 0x7f8fb415f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.158+0000 7f8fb3200640 1 --2- 172.21.15.40:0/3972226489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fb415f1a0 0x7f8fb40b85f0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.158+0000 7f8fb3200640 1 -- 172.21.15.40:0/3972226489 >> 172.21.15.40:0/3972226489 conn(0x7f8fb40d4d90 msgr2=0x7f8fb40d5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:40.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.158+0000 7f8fb3200640 1 -- 172.21.15.40:0/3972226489 shutdown_connections 2024-04-26T23:06:40.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.158+0000 7f8fb3200640 1 -- 172.21.15.40:0/3972226489 wait complete. 2024-04-26T23:06:40.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.160+0000 7f8fb3200640 1 Processor -- start 2024-04-26T23:06:40.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.160+0000 7f8fb3200640 1 -- start start 2024-04-26T23:06:40.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.162+0000 7f8fb3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fb40d2ae0 0x7f8fb41fcd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:40.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.162+0000 7f8fb3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fb40d32c0 0x7f8fb415a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:40.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.162+0000 7f8fb3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fb415f1a0 0x7f8fb415f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:40.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.162+0000 7f8fb3200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb40d0bb0 con 0x7f8fb415f1a0 2024-04-26T23:06:40.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.162+0000 7f8fb3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb40b9750 con 0x7f8fb40d2ae0 2024-04-26T23:06:40.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.162+0000 7f8fb3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8fb40d3750 con 0x7f8fb40d32c0 2024-04-26T23:06:40.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.162+0000 7f8fb2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fb415f1a0 0x7f8fb415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:40.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.162+0000 7f8fb2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fb415f1a0 0x7f8fb415f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43050/0 (socket says 172.21.15.40:43050) 2024-04-26T23:06:40.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.162+0000 7f8fb2800640 1 -- 172.21.15.40:0/2163907685 learned_addr learned my addr 172.21.15.40:0/2163907685 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:40.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.162+0000 7f8fb1e00640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fb40d2ae0 0x7f8fb41fcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:40.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.162+0000 7f8fb1400640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fb40d32c0 0x7f8fb415a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:40.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.163+0000 7f8fb2800640 1 -- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fb40d32c0 msgr2=0x7f8fb415a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:40.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.163+0000 7f8fb2800640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fb40d32c0 0x7f8fb415a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.163+0000 7f8fb2800640 1 -- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fb40d2ae0 msgr2=0x7f8fb41fcd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:40.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.163+0000 7f8fb2800640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fb40d2ae0 0x7f8fb41fcd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.163+0000 7f8fb2800640 1 -- 172.21.15.40:0/2163907685 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb40cd080 con 0x7f8fb415f1a0 2024-04-26T23:06:40.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.163+0000 7f8fb1400640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fb40d32c0 0x7f8fb415a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:06:40.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.163+0000 7f8fb2800640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fb415f1a0 0x7f8fb415f920 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00c7a0 tx=0x7f8fac00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:40.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.163+0000 7f8faa000640 1 -- 172.21.15.40:0/2163907685 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac018070 con 0x7f8fb415f1a0 2024-04-26T23:06:40.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.163+0000 7f8faa000640 1 -- 172.21.15.40:0/2163907685 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fac00a640 con 0x7f8fb415f1a0 2024-04-26T23:06:40.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.163+0000 7f8faa000640 1 -- 172.21.15.40:0/2163907685 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac00aa70 con 0x7f8fb415f1a0 2024-04-26T23:06:40.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.163+0000 7f8fb3200640 1 -- 172.21.15.40:0/2163907685 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fb40cd360 con 0x7f8fb415f1a0 2024-04-26T23:06:40.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.164+0000 7f8fb3200640 1 -- 172.21.15.40:0/2163907685 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fb40cd920 con 0x7f8fb415f1a0 2024-04-26T23:06:40.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.165+0000 7f8fb3200640 1 -- 172.21.15.40:0/2163907685 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fb4069370 con 0x7f8fb415f1a0 2024-04-26T23:06:40.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.166+0000 7f8faa000640 1 -- 172.21.15.40:0/2163907685 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8fac00ac10 con 0x7f8fb415f1a0 2024-04-26T23:06:40.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.167+0000 7f8faa000640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f9006c8f0 0x7f8f9006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:40.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.167+0000 7f8faa000640 1 -- 172.21.15.40:0/2163907685 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f8fac08b3b0 con 0x7f8fb415f1a0 2024-04-26T23:06:40.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.168+0000 7f8fb1e00640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f9006c8f0 0x7f8f9006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:40.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.169+0000 7f8fb1e00640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f9006c8f0 0x7f8f9006edb0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4034000 tx=0x7f8fa402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:40.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.172+0000 7f8faa000640 1 -- 172.21.15.40:0/2163907685 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8fac0595d0 con 0x7f8fb415f1a0 2024-04-26T23:06:40.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.378+0000 7f8fb3200640 1 -- 172.21.15.40:0/2163907685 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8fb40b7370 con 0x7f8fb415f1a0 2024-04-26T23:06:40.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.379+0000 7f8faa000640 1 -- 172.21.15.40:0/2163907685 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) ==== 100+0+11 (secure 0 0 0) 0x7f8fac058ed0 con 0x7f8fb415f1a0 2024-04-26T23:06:40.380 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:06:40.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.382+0000 7f8fb3200640 1 -- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f9006c8f0 msgr2=0x7f8f9006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:40.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.382+0000 7f8fb3200640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f9006c8f0 0x7f8f9006edb0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4034000 tx=0x7f8fa402e000 comp rx=0 tx=0).stop 2024-04-26T23:06:40.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.382+0000 7f8fb3200640 1 -- 172.21.15.40:0/2163907685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fb415f1a0 msgr2=0x7f8fb415f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:40.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.382+0000 7f8fb3200640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fb415f1a0 0x7f8fb415f920 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00c7a0 tx=0x7f8fac00cc60 comp rx=0 tx=0).stop 2024-04-26T23:06:40.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.382+0000 7f8fb3200640 1 -- 172.21.15.40:0/2163907685 shutdown_connections 2024-04-26T23:06:40.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.382+0000 7f8fb3200640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f9006c8f0 0x7f8f9006edb0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.382+0000 7f8fb3200640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fb415f1a0 0x7f8fb415f920 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.382+0000 7f8fb3200640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8fb40d32c0 0x7f8fb415a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.382+0000 7f8fb3200640 1 --2- 172.21.15.40:0/2163907685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fb40d2ae0 0x7f8fb41fcd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.382+0000 7f8fb3200640 1 -- 172.21.15.40:0/2163907685 >> 172.21.15.40:0/2163907685 conn(0x7f8fb40d4d90 msgr2=0x7f8fb40d20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:40.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.382+0000 7f8fb3200640 1 -- 172.21.15.40:0/2163907685 shutdown_connections 2024-04-26T23:06:40.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.382+0000 7f8fb3200640 1 -- 172.21.15.40:0/2163907685 wait complete. 2024-04-26T23:06:40.401 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:06:40.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.612+0000 7f2c5be00640 1 Processor -- start 2024-04-26T23:06:40.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.613+0000 7f2c5be00640 1 -- start start 2024-04-26T23:06:40.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.614+0000 7f2c5be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2c5c0b4770 0x7f2c5c0d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:40.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.614+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c5c0b63e0 0x7f2c5c161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:40.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.614+0000 7f2c5be00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c5c0b4110 con 0x7f2c5c161850 2024-04-26T23:06:40.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.614+0000 7f2c5be00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f2c5c0d3850 con 0x7f2c5c0b4770 2024-04-26T23:06:40.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.614+0000 7f2c5be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2c5c0d3b30 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.615+0000 7f2c5aa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2c5c161850 0x7f2c5c0d1440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60368/0 (socket says 172.21.15.40:60368) 2024-04-26T23:06:40.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.615+0000 7f2c5aa00640 1 -- 172.21.15.40:0/1251469515 learned_addr learned my addr 172.21.15.40:0/1251469515 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:40.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.615+0000 7f2c5a000640 1 --2- 172.21.15.40:0/1251469515 >> v2:172.21.15.40:3300/0 conn(0x7f2c5c0b4770 0x7f2c5c0d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:40.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.615+0000 7f2c5b400640 1 --2- 172.21.15.40:0/1251469515 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c5c0b63e0 0x7f2c5c161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:40.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.615+0000 7f2c59600640 1 -- 172.21.15.40:0/1251469515 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 375884372 0 0) 0x7f2c5c0b4110 con 0x7f2c5c161850 2024-04-26T23:06:40.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.615+0000 7f2c59600640 1 -- 172.21.15.40:0/1251469515 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c40003520 con 0x7f2c5c161850 2024-04-26T23:06:40.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.615+0000 7f2c5a000640 1 -- 172.21.15.40:0/1251469515 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c5c0b63e0 msgr2=0x7f2c5c161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:40.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.615+0000 7f2c5a000640 1 --2- 172.21.15.40:0/1251469515 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c5c0b63e0 0x7f2c5c161c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.616+0000 7f2c5a000640 1 -- 172.21.15.40:0/1251469515 >> v1:172.21.15.40:6789/0 conn(0x7f2c5c161850 legacy=0x7f2c5c0d1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:40.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.616+0000 7f2c5a000640 1 -- 172.21.15.40:0/1251469515 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c5c0b3d70 con 0x7f2c5c0b4770 2024-04-26T23:06:40.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.617+0000 7f2c5a000640 1 --2- 172.21.15.40:0/1251469515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c5c0b4770 0x7f2c5c0d1b50 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009d90 tx=0x7f2c4c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713f2f681187a878 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:40.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.617+0000 7f2c59600640 1 -- 172.21.15.40:0/1251469515 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c031600 con 0x7f2c5c0b4770 2024-04-26T23:06:40.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.617+0000 7f2c59600640 1 -- 172.21.15.40:0/1251469515 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c4c0317a0 con 0x7f2c5c0b4770 2024-04-26T23:06:40.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.617+0000 7f2c59600640 1 -- 172.21.15.40:0/1251469515 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c031b70 con 0x7f2c5c0b4770 2024-04-26T23:06:40.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.617+0000 7f2c5be00640 1 -- 172.21.15.40:0/1251469515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c5c0b4770 msgr2=0x7f2c5c0d1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:40.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.617+0000 7f2c5be00640 1 --2- 172.21.15.40:0/1251469515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c5c0b4770 0x7f2c5c0d1b50 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009d90 tx=0x7f2c4c02a5d0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.618+0000 7f2c5be00640 1 -- 172.21.15.40:0/1251469515 shutdown_connections 2024-04-26T23:06:40.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.618+0000 7f2c5be00640 1 --2- 172.21.15.40:0/1251469515 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c5c0b63e0 0x7f2c5c161c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.618+0000 7f2c5be00640 1 --2- 172.21.15.40:0/1251469515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c5c0b4770 0x7f2c5c0d1b50 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.618+0000 7f2c5be00640 1 -- 172.21.15.40:0/1251469515 >> 172.21.15.40:0/1251469515 conn(0x7f2c5c0d4e10 msgr2=0x7f2c5c0d5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:40.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.618+0000 7f2c5be00640 1 -- 172.21.15.40:0/1251469515 shutdown_connections 2024-04-26T23:06:40.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.618+0000 7f2c5be00640 1 -- 172.21.15.40:0/1251469515 wait complete. 2024-04-26T23:06:40.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.619+0000 7f2c5be00640 1 Processor -- start 2024-04-26T23:06:40.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.620+0000 7f2c5be00640 1 -- start start 2024-04-26T23:06:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c5c0b4770 0x7f2c5c153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c5c0b63e0 0x7f2c5c0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c5c161850 0x7f2c5c0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2c5c0d2210 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2c5c0d2090 con 0x7f2c5c0b4770 2024-04-26T23:06:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2c5c162170 con 0x7f2c5c161850 2024-04-26T23:06:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c5c0b63e0 0x7f2c5c0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c5c0b63e0 0x7f2c5c0ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43074/0 (socket says 172.21.15.40:43074) 2024-04-26T23:06:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5a000640 1 -- 172.21.15.40:0/1261917926 learned_addr learned my addr 172.21.15.40:0/1261917926 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:40.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5b400640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c5c161850 0x7f2c5c0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:40.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5aa00640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c5c0b4770 0x7f2c5c153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:40.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5a000640 1 -- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c5c161850 msgr2=0x7f2c5c0cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:40.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5a000640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c5c161850 0x7f2c5c0cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5a000640 1 -- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c5c0b4770 msgr2=0x7f2c5c153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:40.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5a000640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c5c0b4770 0x7f2c5c153cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.621+0000 7f2c5a000640 1 -- 172.21.15.40:0/1261917926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c5c152750 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.622+0000 7f2c5a000640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c5c0b63e0 0x7f2c5c0ce590 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009d90 tx=0x7f2c4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:40.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.622+0000 7f2c52a00640 1 -- 172.21.15.40:0/1261917926 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c040070 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.622+0000 7f2c52a00640 1 -- 172.21.15.40:0/1261917926 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c4c03c830 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.622+0000 7f2c52a00640 1 -- 172.21.15.40:0/1261917926 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c03cc20 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.622+0000 7f2c5be00640 1 -- 172.21.15.40:0/1261917926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c5c152a30 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.622+0000 7f2c5be00640 1 -- 172.21.15.40:0/1261917926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c5c152ff0 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.623+0000 7f2c5be00640 1 -- 172.21.15.40:0/1261917926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c5c069a80 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.624+0000 7f2c52a00640 1 -- 172.21.15.40:0/1261917926 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2c4c039040 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.625+0000 7f2c52a00640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2c3806c8f0 0x7f2c3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:40.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.625+0000 7f2c52a00640 1 -- 172.21.15.40:0/1261917926 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f2c4c0ad340 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.626+0000 7f2c5aa00640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2c3806c8f0 0x7f2c3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:40.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.627+0000 7f2c5aa00640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2c3806c8f0 0x7f2c3806edb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2c44000f60 tx=0x7f2c44009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:40.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.630+0000 7f2c52a00640 1 -- 172.21.15.40:0/1261917926 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2c4c07b680 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.871+0000 7f2c5be00640 1 -- 172.21.15.40:0/1261917926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c5c0b59f0 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.872+0000 7f2c52a00640 1 -- 172.21.15.40:0/1261917926 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+5232 (secure 0 0 0) 0x7f2c4c03ae50 con 0x7f2c5c0b63e0 2024-04-26T23:06:40.873 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:40.873 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":66,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/4194140680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4194140680},{"type":"v1","addr":"172.21.15.103:6836","nonce":4194140680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:05:32.212319+0000","modified":"2024-04-26T23:06:10.856590+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6774,"mds_1":5614,"mds_2":5324},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5324":{"gid":5324,"name":"b","rank":2,"incarnation":59,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2595375825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2595375825},{"type":"v1","addr":"172.21.15.103:6837","nonce":2595375825}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5614":{"gid":5614,"name":"a","rank":1,"incarnation":57,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6834/2117861921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2117861921},{"type":"v1","addr":"172.21.15.40:6834","nonce":2117861921}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/931483025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":931483025},{"type":"v1","addr":"172.21.15.40:6835","nonce":931483025}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6774,"qdb_cluster":[5324,5614,6774]},"id":4}]} 2024-04-26T23:06:40.873 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 66 2024-04-26T23:06:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.875+0000 7f2c5be00640 1 -- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2c3806c8f0 msgr2=0x7f2c3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.875+0000 7f2c5be00640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2c3806c8f0 0x7f2c3806edb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2c44000f60 tx=0x7f2c44009040 comp rx=0 tx=0).stop 2024-04-26T23:06:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.875+0000 7f2c5be00640 1 -- 172.21.15.40:0/1261917926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c5c0b63e0 msgr2=0x7f2c5c0ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.875+0000 7f2c5be00640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c5c0b63e0 0x7f2c5c0ce590 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009d90 tx=0x7f2c4c004a40 comp rx=0 tx=0).stop 2024-04-26T23:06:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.875+0000 7f2c5be00640 1 -- 172.21.15.40:0/1261917926 shutdown_connections 2024-04-26T23:06:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.875+0000 7f2c5be00640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2c3806c8f0 0x7f2c3806edb0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.875+0000 7f2c5be00640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c5c161850 0x7f2c5c0cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.875+0000 7f2c5be00640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c5c0b63e0 0x7f2c5c0ce590 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.875+0000 7f2c5be00640 1 --2- 172.21.15.40:0/1261917926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c5c0b4770 0x7f2c5c153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:40.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.875+0000 7f2c5be00640 1 -- 172.21.15.40:0/1261917926 >> 172.21.15.40:0/1261917926 conn(0x7f2c5c0d4e10 msgr2=0x7f2c5c0d2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:40.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.875+0000 7f2c5be00640 1 -- 172.21.15.40:0/1261917926 shutdown_connections 2024-04-26T23:06:40.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:40.875+0000 7f2c5be00640 1 -- 172.21.15.40:0/1261917926 wait complete. 2024-04-26T23:06:40.893 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:06:41.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.095+0000 7f8895600640 1 Processor -- start 2024-04-26T23:06:41.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.095+0000 7f8895600640 1 -- start start 2024-04-26T23:06:41.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.097+0000 7f8895600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8890156460 0x7f88900cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.097+0000 7f8895600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f88900cba10 0x7f889006f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.097+0000 7f8895600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8890156840 con 0x7f889006fcd0 2024-04-26T23:06:41.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.097+0000 7f8895600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f88900cbf20 con 0x7f88900cba10 2024-04-26T23:06:41.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.097+0000 7f8895600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f88900cc200 con 0x7f8890156460 2024-04-26T23:06:41.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.097+0000 7f888f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f889006fcd0 0x7f889015f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60370/0 (socket says 172.21.15.40:60370) 2024-04-26T23:06:41.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.097+0000 7f888f400640 1 -- 172.21.15.40:0/125996524 learned_addr learned my addr 172.21.15.40:0/125996524 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:41.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.097+0000 7f8886000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f88900cba10 0x7f889006f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:41.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.098+0000 7f888ea00640 1 --2- 172.21.15.40:0/125996524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8890156460 0x7f88900cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.098+0000 7f8886000640 1 -- 172.21.15.40:0/125996524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8890156460 msgr2=0x7f88900cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.098+0000 7f8886000640 1 --2- 172.21.15.40:0/125996524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8890156460 0x7f88900cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.098+0000 7f8886000640 1 -- 172.21.15.40:0/125996524 >> v1:172.21.15.40:6789/0 conn(0x7f889006fcd0 legacy=0x7f889015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.098+0000 7f8886000640 1 -- 172.21.15.40:0/125996524 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88900cc4e0 con 0x7f88900cba10 2024-04-26T23:06:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.098+0000 7f888ea00640 1 --2- 172.21.15.40:0/125996524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8890156460 0x7f88900cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:06:41.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.098+0000 7f888e000640 1 -- 172.21.15.40:0/125996524 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1761626915 0 0) 0x7f8890156840 con 0x7f889006fcd0 2024-04-26T23:06:41.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.099+0000 7f8886000640 1 --2- 172.21.15.40:0/125996524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88900cba10 0x7f889006f790 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f88780097a0 tx=0x7f887802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85fba2617ed729a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:41.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.100+0000 7f888e000640 1 -- 172.21.15.40:0/125996524 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878037070 con 0x7f88900cba10 2024-04-26T23:06:41.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.100+0000 7f888e000640 1 -- 172.21.15.40:0/125996524 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8878032440 con 0x7f88900cba10 2024-04-26T23:06:41.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.100+0000 7f888e000640 1 -- 172.21.15.40:0/125996524 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878032810 con 0x7f88900cba10 2024-04-26T23:06:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.100+0000 7f8895600640 1 -- 172.21.15.40:0/125996524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88900cba10 msgr2=0x7f889006f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.100+0000 7f8895600640 1 --2- 172.21.15.40:0/125996524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88900cba10 0x7f889006f790 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f88780097a0 tx=0x7f887802a180 comp rx=0 tx=0).stop 2024-04-26T23:06:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.100+0000 7f8895600640 1 -- 172.21.15.40:0/125996524 shutdown_connections 2024-04-26T23:06:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.100+0000 7f8895600640 1 --2- 172.21.15.40:0/125996524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88900cba10 0x7f889006f790 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.100+0000 7f8895600640 1 --2- 172.21.15.40:0/125996524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8890156460 0x7f88900cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.100+0000 7f8895600640 1 -- 172.21.15.40:0/125996524 >> 172.21.15.40:0/125996524 conn(0x7f88900d0be0 msgr2=0x7f88900d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.100+0000 7f8895600640 1 -- 172.21.15.40:0/125996524 shutdown_connections 2024-04-26T23:06:41.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.100+0000 7f8895600640 1 -- 172.21.15.40:0/125996524 wait complete. 2024-04-26T23:06:41.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.102+0000 7f8895600640 1 Processor -- start 2024-04-26T23:06:41.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.102+0000 7f8895600640 1 -- start start 2024-04-26T23:06:41.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.103+0000 7f8895600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f889006fcd0 0x7f8890151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.103+0000 7f8895600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88900cba10 0x7f88900c6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.103+0000 7f8895600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8890156460 0x7f88901e88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.103+0000 7f8895600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f889006e260 con 0x7f88900cba10 2024-04-26T23:06:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.103+0000 7f8895600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8890072700 con 0x7f889006fcd0 2024-04-26T23:06:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.103+0000 7f8895600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f889006e3e0 con 0x7f8890156460 2024-04-26T23:06:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.104+0000 7f8886000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88900cba10 0x7f88900c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.104+0000 7f8886000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88900cba10 0x7f88900c6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43092/0 (socket says 172.21.15.40:43092) 2024-04-26T23:06:41.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.104+0000 7f8886000640 1 -- 172.21.15.40:0/718220822 learned_addr learned my addr 172.21.15.40:0/718220822 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:41.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.104+0000 7f888f400640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8890156460 0x7f88901e88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:41.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.104+0000 7f888ea00640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f889006fcd0 0x7f8890151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:41.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.104+0000 7f8886000640 1 -- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8890156460 msgr2=0x7f88901e88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:41.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.104+0000 7f8886000640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8890156460 0x7f88901e88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.104+0000 7f8886000640 1 -- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f889006fcd0 msgr2=0x7f8890151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:41.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.104+0000 7f8886000640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f889006fcd0 0x7f8890151d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.104+0000 7f8886000640 1 -- 172.21.15.40:0/718220822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88900c70c0 con 0x7f88900cba10 2024-04-26T23:06:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.104+0000 7f888ea00640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f889006fcd0 0x7f8890151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.104+0000 7f8886000640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88900cba10 0x7f88900c6700 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f8878035040 tx=0x7f88780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.105+0000 7f8887400640 1 -- 172.21.15.40:0/718220822 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878037040 con 0x7f88900cba10 2024-04-26T23:06:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.105+0000 7f8887400640 1 -- 172.21.15.40:0/718220822 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8878030770 con 0x7f88900cba10 2024-04-26T23:06:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.105+0000 7f8887400640 1 -- 172.21.15.40:0/718220822 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878030b40 con 0x7f88900cba10 2024-04-26T23:06:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.105+0000 7f8895600640 1 -- 172.21.15.40:0/718220822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88900c73a0 con 0x7f88900cba10 2024-04-26T23:06:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.105+0000 7f8895600640 1 -- 172.21.15.40:0/718220822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88900b3860 con 0x7f88900cba10 2024-04-26T23:06:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.106+0000 7f8895600640 1 -- 172.21.15.40:0/718220822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88900694f0 con 0x7f88900cba10 2024-04-26T23:06:41.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.107+0000 7f8887400640 1 -- 172.21.15.40:0/718220822 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8878044020 con 0x7f88900cba10 2024-04-26T23:06:41.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.109+0000 7f8887400640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f886c06c8f0 0x7f886c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.109+0000 7f8887400640 1 -- 172.21.15.40:0/718220822 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f88780ac2c0 con 0x7f88900cba10 2024-04-26T23:06:41.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.110+0000 7f888ea00640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f886c06c8f0 0x7f886c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:41.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.111+0000 7f888ea00640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f886c06c8f0 0x7f886c06edb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f8880006fd0 tx=0x7f8880008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:41.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.115+0000 7f8887400640 1 -- 172.21.15.40:0/718220822 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8878032e20 con 0x7f88900cba10 2024-04-26T23:06:41.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.329+0000 7f8895600640 1 -- 172.21.15.40:0/718220822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88900cead0 con 0x7f88900cba10 2024-04-26T23:06:41.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.332+0000 7f8887400640 1 -- 172.21.15.40:0/718220822 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+17040 (secure 0 0 0) 0x7f887807a4f0 con 0x7f88900cba10 2024-04-26T23:06:41.333 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:41.333 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":67,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:05:48.861635+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:05:30.081199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-26T23:05:31.192158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.9","mappings":[{"from":7,"to":2}]},{"pgid":"10.10","mappings":[{"from":4,"to":3}]},{"pgid":"10.22","mappings":[{"from":7,"to":3}]},{"pgid":"10.35","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:06:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.335+0000 7f8895600640 1 -- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f886c06c8f0 msgr2=0x7f886c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.335+0000 7f8895600640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f886c06c8f0 0x7f886c06edb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f8880006fd0 tx=0x7f8880008040 comp rx=0 tx=0).stop 2024-04-26T23:06:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.335+0000 7f8895600640 1 -- 172.21.15.40:0/718220822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88900cba10 msgr2=0x7f88900c6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.335+0000 7f8895600640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88900cba10 0x7f88900c6700 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f8878035040 tx=0x7f88780046c0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.335+0000 7f8895600640 1 -- 172.21.15.40:0/718220822 shutdown_connections 2024-04-26T23:06:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.335+0000 7f8895600640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f886c06c8f0 0x7f886c06edb0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.335+0000 7f8895600640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8890156460 0x7f88901e88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.335+0000 7f8895600640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88900cba10 0x7f88900c6700 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.335+0000 7f8895600640 1 --2- 172.21.15.40:0/718220822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f889006fcd0 0x7f8890151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.335+0000 7f8895600640 1 -- 172.21.15.40:0/718220822 >> 172.21.15.40:0/718220822 conn(0x7f88900d0be0 msgr2=0x7f88900b0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.336+0000 7f8895600640 1 -- 172.21.15.40:0/718220822 shutdown_connections 2024-04-26T23:06:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.336+0000 7f8895600640 1 -- 172.21.15.40:0/718220822 wait complete. 2024-04-26T23:06:41.355 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T23:06:41.355 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:06:41.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.565+0000 7fed2ae00640 1 Processor -- start 2024-04-26T23:06:41.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.565+0000 7fed2ae00640 1 -- start start 2024-04-26T23:06:41.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.567+0000 7fed2ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed240af7a0 0x7fed240afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.567+0000 7fed2ae00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fed24071fe0 0x7fed24073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.567+0000 7fed2ae00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed240d0820 con 0x7fed2415ce60 2024-04-26T23:06:41.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.567+0000 7fed2ae00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fed240cc450 con 0x7fed24071fe0 2024-04-26T23:06:41.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.567+0000 7fed2ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fed241614c0 con 0x7fed240af7a0 2024-04-26T23:06:41.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.567+0000 7fed29a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fed2415ce60 0x7fed240ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60378/0 (socket says 172.21.15.40:60378) 2024-04-26T23:06:41.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.567+0000 7fed29a00640 1 -- 172.21.15.40:0/2326155597 learned_addr learned my addr 172.21.15.40:0/2326155597 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:41.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.568+0000 7fed29000640 1 --2- 172.21.15.40:0/2326155597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed240af7a0 0x7fed240afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.568+0000 7fed2a400640 1 --2- 172.21.15.40:0/2326155597 >> v2:172.21.15.103:3300/0 conn(0x7fed24071fe0 0x7fed24073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.568+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3290068702 0 0) 0x7fed240d0820 con 0x7fed2415ce60 2024-04-26T23:06:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.568+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed0c003520 con 0x7fed2415ce60 2024-04-26T23:06:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.568+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 644601188 0 0) 0x7fed0c003520 con 0x7fed2415ce60 2024-04-26T23:06:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.568+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed240af7a0 msgr2=0x7fed240afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.568+0000 7fed23e00640 1 --2- 172.21.15.40:0/2326155597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed240af7a0 0x7fed240afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.568+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 >> v2:172.21.15.103:3300/0 conn(0x7fed24071fe0 msgr2=0x7fed24073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.568+0000 7fed23e00640 1 --2- 172.21.15.40:0/2326155597 >> v2:172.21.15.103:3300/0 conn(0x7fed24071fe0 0x7fed24073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.568+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed24161760 con 0x7fed2415ce60 2024-04-26T23:06:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.568+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fed18001670 con 0x7fed2415ce60 2024-04-26T23:06:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.568+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 >> v1:172.21.15.40:6789/0 conn(0x7fed2415ce60 legacy=0x7fed240ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.568+0000 7fed23e00640 1 --2- 172.21.15.40:0/2326155597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed0c0039b0 0x7fed0c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.569+0000 7fed23e00640 1 --2- 172.21.15.40:0/2326155597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed0c004800 0x7fed0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.569+0000 7fed23e00640 1 --2- 172.21.15.40:0/2326155597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed240af7a0 0x7fed0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.569+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fed240cc450 con 0x7fed0c004800 2024-04-26T23:06:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.569+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fed241614c0 con 0x7fed0c0039b0 2024-04-26T23:06:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.569+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fed0c00b9c0 con 0x7fed240af7a0 2024-04-26T23:06:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.569+0000 7fed29000640 1 --2- 172.21.15.40:0/2326155597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed0c004800 0x7fed0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.569+0000 7fed29000640 1 -- 172.21.15.40:0/2326155597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed240af7a0 msgr2=0x7fed0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.569+0000 7fed29000640 1 --2- 172.21.15.40:0/2326155597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed240af7a0 0x7fed0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.569+0000 7fed29000640 1 -- 172.21.15.40:0/2326155597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed0c0039b0 msgr2=0x7fed0c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.569+0000 7fed29000640 1 --2- 172.21.15.40:0/2326155597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed0c0039b0 0x7fed0c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.569+0000 7fed29000640 1 -- 172.21.15.40:0/2326155597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0c003520 con 0x7fed0c004800 2024-04-26T23:06:41.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.570+0000 7fed29000640 1 --2- 172.21.15.40:0/2326155597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed0c004800 0x7fed0c004c80 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fed10009af0 tx=0x7fed0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e54401f612ad306a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:41.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.571+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10036070 con 0x7fed0c004800 2024-04-26T23:06:41.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.571+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed10031400 con 0x7fed0c004800 2024-04-26T23:06:41.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.571+0000 7fed23e00640 1 -- 172.21.15.40:0/2326155597 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10009690 con 0x7fed0c004800 2024-04-26T23:06:41.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.571+0000 7fed2ae00640 1 -- 172.21.15.40:0/2326155597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed0c004800 msgr2=0x7fed0c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:41.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.571+0000 7fed2ae00640 1 --2- 172.21.15.40:0/2326155597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed0c004800 0x7fed0c004c80 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fed10009af0 tx=0x7fed0c005830 comp rx=0 tx=0).stop 2024-04-26T23:06:41.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.571+0000 7fed2a400640 1 -- 172.21.15.40:0/2326155597 reap_dead start 2024-04-26T23:06:41.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.571+0000 7fed2ae00640 1 -- 172.21.15.40:0/2326155597 shutdown_connections 2024-04-26T23:06:41.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.571+0000 7fed2ae00640 1 -- 172.21.15.40:0/2326155597 >> 172.21.15.40:0/2326155597 conn(0x7fed240d0a10 msgr2=0x7fed240d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:41.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.572+0000 7fed2ae00640 1 -- 172.21.15.40:0/2326155597 shutdown_connections 2024-04-26T23:06:41.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.572+0000 7fed2ae00640 1 -- 172.21.15.40:0/2326155597 wait complete. 2024-04-26T23:06:41.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.573+0000 7fed2ae00640 1 Processor -- start 2024-04-26T23:06:41.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.573+0000 7fed2ae00640 1 -- start start 2024-04-26T23:06:41.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.574+0000 7fed2ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed0c004800 0x7fed2414b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.574+0000 7fed2ae00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed240c7ae0 0x7fed2415f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.574+0000 7fed2ae00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed2415e710 0x7fed2415ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.574+0000 7fed2ae00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fed240cc1f0 con 0x7fed2415e710 2024-04-26T23:06:41.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.574+0000 7fed2ae00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fed240cc070 con 0x7fed240c7ae0 2024-04-26T23:06:41.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.574+0000 7fed2ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fed24073ca0 con 0x7fed0c004800 2024-04-26T23:06:41.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.575+0000 7fed2a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed2415e710 0x7fed2415ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:41.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.575+0000 7fed2a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed2415e710 0x7fed2415ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43102/0 (socket says 172.21.15.40:43102) 2024-04-26T23:06:41.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.575+0000 7fed2a400640 1 -- 172.21.15.40:0/3942252180 learned_addr learned my addr 172.21.15.40:0/3942252180 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.575+0000 7fed29a00640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed0c004800 0x7fed2414b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.575+0000 7fed29000640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed240c7ae0 0x7fed2415f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.575+0000 7fed2a400640 1 -- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed0c004800 msgr2=0x7fed2414b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.575+0000 7fed2a400640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed0c004800 0x7fed2414b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.575+0000 7fed2a400640 1 -- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed240c7ae0 msgr2=0x7fed2415f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.575+0000 7fed2a400640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed240c7ae0 0x7fed2415f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.575+0000 7fed2a400640 1 -- 172.21.15.40:0/3942252180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed240c7fe0 con 0x7fed2415e710 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.575+0000 7fed29000640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed240c7ae0 0x7fed2415f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.576+0000 7fed2a400640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed2415e710 0x7fed2415ebc0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fed24069040 tx=0x7fed1c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.576+0000 7fed21600640 1 -- 172.21.15.40:0/3942252180 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c00ba80 con 0x7fed2415e710 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.576+0000 7fed21600640 1 -- 172.21.15.40:0/3942252180 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed1c01a070 con 0x7fed2415e710 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.576+0000 7fed2ae00640 1 -- 172.21.15.40:0/3942252180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed240c82c0 con 0x7fed2415e710 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.576+0000 7fed21600640 1 -- 172.21.15.40:0/3942252180 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c00be40 con 0x7fed2415e710 2024-04-26T23:06:41.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.576+0000 7fed2ae00640 1 -- 172.21.15.40:0/3942252180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed240c8880 con 0x7fed2415e710 2024-04-26T23:06:41.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.577+0000 7fed2ae00640 1 -- 172.21.15.40:0/3942252180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed24069370 con 0x7fed2415e710 2024-04-26T23:06:41.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.578+0000 7fed21600640 1 -- 172.21.15.40:0/3942252180 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fed1c005d10 con 0x7fed2415e710 2024-04-26T23:06:41.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.579+0000 7fed21600640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fed0406c920 0x7fed0406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:41.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.579+0000 7fed21600640 1 -- 172.21.15.40:0/3942252180 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7fed1c08bf10 con 0x7fed2415e710 2024-04-26T23:06:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.580+0000 7fed29a00640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fed0406c920 0x7fed0406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:41.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.581+0000 7fed29a00640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fed0406c920 0x7fed0406ede0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fed18001a80 tx=0x7fed18001c80 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:41.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.583+0000 7fed21600640 1 -- 172.21.15.40:0/3942252180 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fed1c024020 con 0x7fed2415e710 2024-04-26T23:06:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.822+0000 7fed2ae00640 1 -- 172.21.15.40:0/3942252180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed240d01f0 con 0x7fed2415e710 2024-04-26T23:06:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.823+0000 7fed21600640 1 -- 172.21.15.40:0/3942252180 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+5232 (secure 0 0 0) 0x7fed1c05a120 con 0x7fed2415e710 2024-04-26T23:06:41.823 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:41.824 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":66,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/4194140680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4194140680},{"type":"v1","addr":"172.21.15.103:6836","nonce":4194140680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:05:32.212319+0000","modified":"2024-04-26T23:06:10.856590+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6774,"mds_1":5614,"mds_2":5324},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5324":{"gid":5324,"name":"b","rank":2,"incarnation":59,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2595375825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2595375825},{"type":"v1","addr":"172.21.15.103:6837","nonce":2595375825}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5614":{"gid":5614,"name":"a","rank":1,"incarnation":57,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6834/2117861921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2117861921},{"type":"v1","addr":"172.21.15.40:6834","nonce":2117861921}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/931483025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":931483025},{"type":"v1","addr":"172.21.15.40:6835","nonce":931483025}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6774,"qdb_cluster":[5324,5614,6774]},"id":4}]} 2024-04-26T23:06:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 66 2024-04-26T23:06:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.825+0000 7fed2ae00640 1 -- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fed0406c920 msgr2=0x7fed0406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.825+0000 7fed2ae00640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fed0406c920 0x7fed0406ede0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fed18001a80 tx=0x7fed18001c80 comp rx=0 tx=0).stop 2024-04-26T23:06:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.825+0000 7fed2ae00640 1 -- 172.21.15.40:0/3942252180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed2415e710 msgr2=0x7fed2415ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.825+0000 7fed2ae00640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed2415e710 0x7fed2415ebc0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fed24069040 tx=0x7fed1c00b480 comp rx=0 tx=0).stop 2024-04-26T23:06:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.826+0000 7fed2ae00640 1 -- 172.21.15.40:0/3942252180 shutdown_connections 2024-04-26T23:06:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.826+0000 7fed2ae00640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fed0406c920 0x7fed0406ede0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.826+0000 7fed2ae00640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed2415e710 0x7fed2415ebc0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.826+0000 7fed2ae00640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed240c7ae0 0x7fed2415f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.826+0000 7fed2ae00640 1 --2- 172.21.15.40:0/3942252180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed0c004800 0x7fed2414b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.826+0000 7fed2ae00640 1 -- 172.21.15.40:0/3942252180 >> 172.21.15.40:0/3942252180 conn(0x7fed240d0a10 msgr2=0x7fed240b12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.826+0000 7fed2ae00640 1 -- 172.21.15.40:0/3942252180 shutdown_connections 2024-04-26T23:06:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:41.826+0000 7fed2ae00640 1 -- 172.21.15.40:0/3942252180 wait complete. 2024-04-26T23:06:41.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:06:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.035+0000 7f5676400640 1 Processor -- start 2024-04-26T23:06:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.035+0000 7f5676400640 1 -- start start 2024-04-26T23:06:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.037+0000 7f5676400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f56700af7a0 0x7f56700afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.037+0000 7f5676400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5670071fe0 0x7f5670073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.037+0000 7f5676400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56700d0820 con 0x7f567015ce60 2024-04-26T23:06:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.037+0000 7f5676400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f56700cc450 con 0x7f56700af7a0 2024-04-26T23:06:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.037+0000 7f5676400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f56701614c0 con 0x7f5670071fe0 2024-04-26T23:06:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.037+0000 7f5675000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f567015ce60 0x7f56700ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60386/0 (socket says 172.21.15.40:60386) 2024-04-26T23:06:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.037+0000 7f5675000640 1 -- 172.21.15.40:0/968660764 learned_addr learned my addr 172.21.15.40:0/968660764 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3091232235 0 0) 0x7f56700d0820 con 0x7f567015ce60 2024-04-26T23:06:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f5667e00640 1 --2- 172.21.15.40:0/968660764 >> v2:172.21.15.103:3300/0 conn(0x7f56700af7a0 0x7f56700afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5654003520 con 0x7f567015ce60 2024-04-26T23:06:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f5675a00640 1 --2- 172.21.15.40:0/968660764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5670071fe0 0x7f5670073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3316926517 0 0) 0x7f5654003520 con 0x7f567015ce60 2024-04-26T23:06:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5670071fe0 msgr2=0x7f5670073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 --2- 172.21.15.40:0/968660764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5670071fe0 0x7f5670073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 >> v2:172.21.15.103:3300/0 conn(0x7f56700af7a0 msgr2=0x7f56700afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 --2- 172.21.15.40:0/968660764 >> v2:172.21.15.103:3300/0 conn(0x7f56700af7a0 0x7f56700afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5670161760 con 0x7f567015ce60 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f5660001670 con 0x7f567015ce60 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f5675a00640 1 --2- 172.21.15.40:0/968660764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5670071fe0 0x7f5670073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 >> v1:172.21.15.40:6789/0 conn(0x7f567015ce60 legacy=0x7f56700ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 --2- 172.21.15.40:0/968660764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56540039b0 0x7f5654003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 --2- 172.21.15.40:0/968660764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5670071fe0 0x7f5654004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 --2- 172.21.15.40:0/968660764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5654004d40 0x7f56540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f56700cc450 con 0x7f56540039b0 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f56701614c0 con 0x7f5654004d40 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.038+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f565400baa0 con 0x7f5670071fe0 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.039+0000 7f5675000640 1 --2- 172.21.15.40:0/968660764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56540039b0 0x7f5654003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.039+0000 7f5667e00640 1 --2- 172.21.15.40:0/968660764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5670071fe0 0x7f5654004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.039+0000 7f5675a00640 1 --2- 172.21.15.40:0/968660764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5654004d40 0x7f56540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.039+0000 7f5675000640 1 -- 172.21.15.40:0/968660764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5670071fe0 msgr2=0x7f5654004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.039+0000 7f5675000640 1 --2- 172.21.15.40:0/968660764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5670071fe0 0x7f5654004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.039+0000 7f5675000640 1 -- 172.21.15.40:0/968660764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5654004d40 msgr2=0x7f56540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.039+0000 7f5675000640 1 --2- 172.21.15.40:0/968660764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5654004d40 0x7f56540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.039+0000 7f5675000640 1 -- 172.21.15.40:0/968660764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5654003520 con 0x7f56540039b0 2024-04-26T23:06:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.040+0000 7f5675000640 1 --2- 172.21.15.40:0/968660764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56540039b0 0x7f5654003dd0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f566000c160 tx=0x7f5654005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f876dd36a5b68e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.041+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5660007c80 con 0x7f56540039b0 2024-04-26T23:06:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.041+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5660005ad0 con 0x7f56540039b0 2024-04-26T23:06:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.041+0000 7f566fe00640 1 -- 172.21.15.40:0/968660764 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5660007070 con 0x7f56540039b0 2024-04-26T23:06:42.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.041+0000 7f5676400640 1 -- 172.21.15.40:0/968660764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56540039b0 msgr2=0x7f5654003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.041+0000 7f5676400640 1 --2- 172.21.15.40:0/968660764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56540039b0 0x7f5654003dd0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f566000c160 tx=0x7f5654005830 comp rx=0 tx=0).stop 2024-04-26T23:06:42.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.041+0000 7f5675a00640 1 -- 172.21.15.40:0/968660764 reap_dead start 2024-04-26T23:06:42.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.041+0000 7f5676400640 1 -- 172.21.15.40:0/968660764 shutdown_connections 2024-04-26T23:06:42.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.041+0000 7f5676400640 1 -- 172.21.15.40:0/968660764 >> 172.21.15.40:0/968660764 conn(0x7f56700d0a10 msgr2=0x7f56700d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:42.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.041+0000 7f5676400640 1 -- 172.21.15.40:0/968660764 shutdown_connections 2024-04-26T23:06:42.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.041+0000 7f5676400640 1 -- 172.21.15.40:0/968660764 wait complete. 2024-04-26T23:06:42.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.043+0000 7f5676400640 1 Processor -- start 2024-04-26T23:06:42.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.043+0000 7f5676400640 1 -- start start 2024-04-26T23:06:42.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.044+0000 7f5676400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56540039b0 0x7f56700c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.044+0000 7f5676400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56700b1a70 0x7f56700b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.044+0000 7f5676400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f567015bc50 0x7f567015c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.044+0000 7f5676400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f56700cc1f0 con 0x7f56700b1a70 2024-04-26T23:06:42.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.044+0000 7f5676400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f56700cc070 con 0x7f567015bc50 2024-04-26T23:06:42.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.044+0000 7f5676400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5670073760 con 0x7f56540039b0 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.045+0000 7f5667e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56700b1a70 0x7f56700b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.045+0000 7f5667e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56700b1a70 0x7f56700b1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43118/0 (socket says 172.21.15.40:43118) 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.045+0000 7f5667e00640 1 -- 172.21.15.40:0/1984922840 learned_addr learned my addr 172.21.15.40:0/1984922840 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.045+0000 7f5675a00640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f567015bc50 0x7f567015c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.045+0000 7f5675000640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56540039b0 0x7f56700c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.045+0000 7f5667e00640 1 -- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56540039b0 msgr2=0x7f56700c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.045+0000 7f5667e00640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56540039b0 0x7f56700c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.045+0000 7f5667e00640 1 -- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f567015bc50 msgr2=0x7f567015c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.045+0000 7f5667e00640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f567015bc50 0x7f567015c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.045+0000 7f5667e00640 1 -- 172.21.15.40:0/1984922840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5670157810 con 0x7f56700b1a70 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.045+0000 7f5675000640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56540039b0 0x7f56700c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.045+0000 7f5667e00640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56700b1a70 0x7f56700b1ed0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f5658011730 tx=0x7f5658011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.046+0000 7f566d600640 1 -- 172.21.15.40:0/1984922840 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565800f960 con 0x7f56700b1a70 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.046+0000 7f566d600640 1 -- 172.21.15.40:0/1984922840 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f565800fb00 con 0x7f56700b1a70 2024-04-26T23:06:42.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.046+0000 7f566d600640 1 -- 172.21.15.40:0/1984922840 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5658015460 con 0x7f56700b1a70 2024-04-26T23:06:42.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.046+0000 7f5676400640 1 -- 172.21.15.40:0/1984922840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5670157af0 con 0x7f56700b1a70 2024-04-26T23:06:42.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.046+0000 7f5676400640 1 -- 172.21.15.40:0/1984922840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56701580b0 con 0x7f56700b1a70 2024-04-26T23:06:42.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.047+0000 7f5676400640 1 -- 172.21.15.40:0/1984922840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5670069370 con 0x7f56700b1a70 2024-04-26T23:06:42.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.048+0000 7f566d600640 1 -- 172.21.15.40:0/1984922840 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5658015600 con 0x7f56700b1a70 2024-04-26T23:06:42.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.049+0000 7f566d600640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f564c06c8f0 0x7f564c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.049+0000 7f566d600640 1 -- 172.21.15.40:0/1984922840 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f565808c720 con 0x7f56700b1a70 2024-04-26T23:06:42.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.049+0000 7f5675000640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f564c06c8f0 0x7f564c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.051+0000 7f5675000640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f564c06c8f0 0x7f564c06edb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5660032000 tx=0x7f5660007000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:42.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.053+0000 7f566d600640 1 -- 172.21.15.40:0/1984922840 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5658019030 con 0x7f56700b1a70 2024-04-26T23:06:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.297+0000 7f5676400640 1 -- 172.21.15.40:0/1984922840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56700cc9d0 con 0x7f56700b1a70 2024-04-26T23:06:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.299+0000 7f566d600640 1 -- 172.21.15.40:0/1984922840 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+5232 (secure 0 0 0) 0x7f565805a820 con 0x7f56700b1a70 2024-04-26T23:06:42.299 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:42.300 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":66,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/4194140680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4194140680},{"type":"v1","addr":"172.21.15.103:6836","nonce":4194140680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:05:32.212319+0000","modified":"2024-04-26T23:06:10.856590+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6774,"mds_1":5614,"mds_2":5324},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5324":{"gid":5324,"name":"b","rank":2,"incarnation":59,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2595375825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2595375825},{"type":"v1","addr":"172.21.15.103:6837","nonce":2595375825}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5614":{"gid":5614,"name":"a","rank":1,"incarnation":57,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6834/2117861921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2117861921},{"type":"v1","addr":"172.21.15.40:6834","nonce":2117861921}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/931483025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":931483025},{"type":"v1","addr":"172.21.15.40:6835","nonce":931483025}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6774,"qdb_cluster":[5324,5614,6774]},"id":4}]} 2024-04-26T23:06:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 66 2024-04-26T23:06:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.301+0000 7f5676400640 1 -- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f564c06c8f0 msgr2=0x7f564c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.301+0000 7f5676400640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f564c06c8f0 0x7f564c06edb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5660032000 tx=0x7f5660007000 comp rx=0 tx=0).stop 2024-04-26T23:06:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.301+0000 7f5676400640 1 -- 172.21.15.40:0/1984922840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56700b1a70 msgr2=0x7f56700b1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.301+0000 7f5676400640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56700b1a70 0x7f56700b1ed0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f5658011730 tx=0x7f5658011bf0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.301+0000 7f5676400640 1 -- 172.21.15.40:0/1984922840 shutdown_connections 2024-04-26T23:06:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.301+0000 7f5676400640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f564c06c8f0 0x7f564c06edb0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.301+0000 7f5676400640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f567015bc50 0x7f567015c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.301+0000 7f5676400640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56700b1a70 0x7f56700b1ed0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.301+0000 7f5676400640 1 --2- 172.21.15.40:0/1984922840 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56540039b0 0x7f56700c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.301+0000 7f5676400640 1 -- 172.21.15.40:0/1984922840 >> 172.21.15.40:0/1984922840 conn(0x7f56700d0a10 msgr2=0x7f56700cd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.302+0000 7f5676400640 1 -- 172.21.15.40:0/1984922840 shutdown_connections 2024-04-26T23:06:42.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.302+0000 7f5676400640 1 -- 172.21.15.40:0/1984922840 wait complete. 2024-04-26T23:06:42.321 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:06:42.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.535+0000 7f3313000640 1 Processor -- start 2024-04-26T23:06:42.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.535+0000 7f3313000640 1 -- start start 2024-04-26T23:06:42.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.536+0000 7f3313000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f330c15ce60 0x7f330c0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.536+0000 7f3313000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f330c0af7a0 0x7f330c0ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.536+0000 7f3313000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f330c0d0820 con 0x7f330c071fe0 2024-04-26T23:06:42.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.536+0000 7f3313000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f330c0cc450 con 0x7f330c15ce60 2024-04-26T23:06:42.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.536+0000 7f3313000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f330c1614c0 con 0x7f330c0af7a0 2024-04-26T23:06:42.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.537+0000 7f3311c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f330c15ce60 0x7f330c0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.537+0000 7f3311c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f330c15ce60 0x7f330c0afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43128/0 (socket says 172.21.15.40:43128) 2024-04-26T23:06:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.537+0000 7f3311c00640 1 -- 172.21.15.40:0/2507804951 learned_addr learned my addr 172.21.15.40:0/2507804951 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.537+0000 7f3311200640 1 --2- 172.21.15.40:0/2507804951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f330c0af7a0 0x7f330c0ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.538+0000 7f3311c00640 1 -- 172.21.15.40:0/2507804951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f330c0af7a0 msgr2=0x7f330c0ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.538+0000 7f3311c00640 1 --2- 172.21.15.40:0/2507804951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f330c0af7a0 0x7f330c0ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.538+0000 7f3311c00640 1 -- 172.21.15.40:0/2507804951 >> v1:172.21.15.103:6789/0 conn(0x7f330c071fe0 legacy=0x7f330c073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.538+0000 7f3311c00640 1 -- 172.21.15.40:0/2507804951 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330c161760 con 0x7f330c15ce60 2024-04-26T23:06:42.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.539+0000 7f3311c00640 1 --2- 172.21.15.40:0/2507804951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f330c15ce60 0x7f330c0afc60 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f3300009e50 tx=0x7f330002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b44209b9e614ca8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:42.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.540+0000 7f330be00640 1 -- 172.21.15.40:0/2507804951 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300036070 con 0x7f330c15ce60 2024-04-26T23:06:42.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.540+0000 7f330be00640 1 -- 172.21.15.40:0/2507804951 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3300031600 con 0x7f330c15ce60 2024-04-26T23:06:42.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.540+0000 7f330be00640 1 -- 172.21.15.40:0/2507804951 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33000319d0 con 0x7f330c15ce60 2024-04-26T23:06:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.540+0000 7f3313000640 1 -- 172.21.15.40:0/2507804951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f330c15ce60 msgr2=0x7f330c0afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.540+0000 7f3313000640 1 --2- 172.21.15.40:0/2507804951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f330c15ce60 0x7f330c0afc60 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f3300009e50 tx=0x7f330002a680 comp rx=0 tx=0).stop 2024-04-26T23:06:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.540+0000 7f3313000640 1 -- 172.21.15.40:0/2507804951 shutdown_connections 2024-04-26T23:06:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.540+0000 7f3313000640 1 --2- 172.21.15.40:0/2507804951 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f330c0af7a0 0x7f330c0ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.540+0000 7f3313000640 1 --2- 172.21.15.40:0/2507804951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f330c15ce60 0x7f330c0afc60 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.540+0000 7f3313000640 1 -- 172.21.15.40:0/2507804951 >> 172.21.15.40:0/2507804951 conn(0x7f330c0d0a10 msgr2=0x7f330c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.541+0000 7f3313000640 1 -- 172.21.15.40:0/2507804951 shutdown_connections 2024-04-26T23:06:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.541+0000 7f3313000640 1 -- 172.21.15.40:0/2507804951 wait complete. 2024-04-26T23:06:42.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.542+0000 7f3313000640 1 Processor -- start 2024-04-26T23:06:42.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.542+0000 7f3313000640 1 -- start start 2024-04-26T23:06:42.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.543+0000 7f3313000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f330c071fe0 0x7f330c0ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3313000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f330c0af7a0 0x7f330c153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3313000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f330c15ce60 0x7f330c0b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3313000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f330c073e70 con 0x7f330c0af7a0 2024-04-26T23:06:42.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3313000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f330c0cf530 con 0x7f330c15ce60 2024-04-26T23:06:42.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3313000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f330c073ff0 con 0x7f330c071fe0 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3311200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f330c0af7a0 0x7f330c153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3311200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f330c0af7a0 0x7f330c153a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43136/0 (socket says 172.21.15.40:43136) 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3311200640 1 -- 172.21.15.40:0/1007218326 learned_addr learned my addr 172.21.15.40:0/1007218326 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3311c00640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f330c071fe0 0x7f330c0ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3312600640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f330c15ce60 0x7f330c0b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3311200640 1 -- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f330c071fe0 msgr2=0x7f330c0ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3311200640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f330c071fe0 0x7f330c0ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3311200640 1 -- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f330c15ce60 msgr2=0x7f330c0b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3311200640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f330c15ce60 0x7f330c0b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3311200640 1 -- 172.21.15.40:0/1007218326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330c157af0 con 0x7f330c0af7a0 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3311c00640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f330c071fe0 0x7f330c0ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.544+0000 7f3312600640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f330c15ce60 0x7f330c0b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.545+0000 7f3311200640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f330c0af7a0 0x7f330c153a20 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f32f800e9e0 tx=0x7f32f800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.545+0000 7f3309600640 1 -- 172.21.15.40:0/1007218326 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f8017070 con 0x7f330c0af7a0 2024-04-26T23:06:42.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.545+0000 7f3309600640 1 -- 172.21.15.40:0/1007218326 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32f800c970 con 0x7f330c0af7a0 2024-04-26T23:06:42.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.545+0000 7f3313000640 1 -- 172.21.15.40:0/1007218326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f330c157dd0 con 0x7f330c0af7a0 2024-04-26T23:06:42.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.545+0000 7f3309600640 1 -- 172.21.15.40:0/1007218326 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f800cd60 con 0x7f330c0af7a0 2024-04-26T23:06:42.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.545+0000 7f3313000640 1 -- 172.21.15.40:0/1007218326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f330c158390 con 0x7f330c0af7a0 2024-04-26T23:06:42.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.547+0000 7f3313000640 1 -- 172.21.15.40:0/1007218326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f330c069370 con 0x7f330c0af7a0 2024-04-26T23:06:42.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.548+0000 7f3309600640 1 -- 172.21.15.40:0/1007218326 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f32f801a070 con 0x7f330c0af7a0 2024-04-26T23:06:42.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.549+0000 7f3309600640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f32ec06c8f0 0x7f32ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.549+0000 7f3309600640 1 -- 172.21.15.40:0/1007218326 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f32f808bda0 con 0x7f330c0af7a0 2024-04-26T23:06:42.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.550+0000 7f3311c00640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f32ec06c8f0 0x7f32ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.551+0000 7f3311c00640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f32ec06c8f0 0x7f32ec06edb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3300033000 tx=0x7f3300009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.553+0000 7f3309600640 1 -- 172.21.15.40:0/1007218326 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f32f8059f30 con 0x7f330c0af7a0 2024-04-26T23:06:42.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.761+0000 7f3313000640 1 -- 172.21.15.40:0/1007218326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f330c0d01f0 con 0x7f330c0af7a0 2024-04-26T23:06:42.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.763+0000 7f3309600640 1 -- 172.21.15.40:0/1007218326 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+17040 (secure 0 0 0) 0x7f32f8059830 con 0x7f330c0af7a0 2024-04-26T23:06:42.764 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:42.764 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":67,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:05:48.861635+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:05:30.081199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-26T23:05:31.192158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.9","mappings":[{"from":7,"to":2}]},{"pgid":"10.10","mappings":[{"from":4,"to":3}]},{"pgid":"10.22","mappings":[{"from":7,"to":3}]},{"pgid":"10.35","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:06:42.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.766+0000 7f3313000640 1 -- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f32ec06c8f0 msgr2=0x7f32ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.766+0000 7f3313000640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f32ec06c8f0 0x7f32ec06edb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3300033000 tx=0x7f3300009000 comp rx=0 tx=0).stop 2024-04-26T23:06:42.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.766+0000 7f3313000640 1 -- 172.21.15.40:0/1007218326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f330c0af7a0 msgr2=0x7f330c153a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.766+0000 7f3313000640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f330c0af7a0 0x7f330c153a20 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f32f800e9e0 tx=0x7f32f800eea0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.766+0000 7f3313000640 1 -- 172.21.15.40:0/1007218326 shutdown_connections 2024-04-26T23:06:42.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.766+0000 7f3313000640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f32ec06c8f0 0x7f32ec06edb0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.766+0000 7f3313000640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f330c15ce60 0x7f330c0b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.766+0000 7f3313000640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f330c0af7a0 0x7f330c153a20 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.766+0000 7f3313000640 1 --2- 172.21.15.40:0/1007218326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f330c071fe0 0x7f330c0ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.767+0000 7f3313000640 1 -- 172.21.15.40:0/1007218326 >> 172.21.15.40:0/1007218326 conn(0x7f330c0d0a10 msgr2=0x7f330c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:42.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.767+0000 7f3313000640 1 -- 172.21.15.40:0/1007218326 shutdown_connections 2024-04-26T23:06:42.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.767+0000 7f3313000640 1 -- 172.21.15.40:0/1007218326 wait complete. 2024-04-26T23:06:42.786 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-26T23:06:42.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.986+0000 7f061fe00640 1 Processor -- start 2024-04-26T23:06:42.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.986+0000 7f061fe00640 1 -- start start 2024-04-26T23:06:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.988+0000 7f061fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f06200ce5e0 0x7f06200afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.988+0000 7f061fe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06200af7b0 con 0x7f06200cc590 2024-04-26T23:06:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.988+0000 7f061fe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06200afa90 con 0x7f062015f2d0 2024-04-26T23:06:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.988+0000 7f061fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f06200cbd90 con 0x7f06200ce5e0 2024-04-26T23:06:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.988+0000 7f061f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f06200cc590 0x7f062006f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60388/0 (socket says 172.21.15.40:60388) 2024-04-26T23:06:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.988+0000 7f061f400640 1 -- 172.21.15.40:0/245422334 learned_addr learned my addr 172.21.15.40:0/245422334 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 489851061 0 0) 0x7f06200af7b0 con 0x7f06200cc590 2024-04-26T23:06:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0604005730 con 0x7f06200cc590 2024-04-26T23:06:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061e000640 1 --2- 172.21.15.40:0/245422334 >> v2:172.21.15.103:3300/0 conn(0x7f06200ce5e0 0x7f06200afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 896735370 0 0) 0x7f0604005730 con 0x7f06200cc590 2024-04-26T23:06:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 >> v2:172.21.15.103:3300/0 conn(0x7f06200ce5e0 msgr2=0x7f06200afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 --2- 172.21.15.40:0/245422334 >> v2:172.21.15.103:3300/0 conn(0x7f06200ce5e0 0x7f06200afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 >> v1:172.21.15.103:6789/0 conn(0x7f062015f2d0 legacy=0x7f06200c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06200cc070 con 0x7f06200cc590 2024-04-26T23:06:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f06180019f0 con 0x7f06200cc590 2024-04-26T23:06:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 >> v1:172.21.15.40:6789/0 conn(0x7f06200cc590 legacy=0x7f062006f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 --2- 172.21.15.40:0/245422334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0604005bc0 0x7f0604005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 --2- 172.21.15.40:0/245422334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0604006a10 0x7f0604006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 --2- 172.21.15.40:0/245422334 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06040073d0 0x7f060400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f06200cbd90 con 0x7f0604005bc0 2024-04-26T23:06:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f060400e320 con 0x7f0604006a10 2024-04-26T23:06:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f060400e520 con 0x7f06040073d0 2024-04-26T23:06:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.989+0000 7f061ea00640 1 --2- 172.21.15.40:0/245422334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0604005bc0 0x7f0604005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.990+0000 7f061f400640 1 --2- 172.21.15.40:0/245422334 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06040073d0 0x7f060400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.990+0000 7f061ea00640 1 -- 172.21.15.40:0/245422334 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06040073d0 msgr2=0x7f060400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.990+0000 7f061e000640 1 --2- 172.21.15.40:0/245422334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0604006a10 0x7f0604006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.990+0000 7f061ea00640 1 --2- 172.21.15.40:0/245422334 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06040073d0 0x7f060400dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.990+0000 7f061ea00640 1 -- 172.21.15.40:0/245422334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0604006a10 msgr2=0x7f0604006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.990+0000 7f061ea00640 1 --2- 172.21.15.40:0/245422334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0604006a10 0x7f0604006e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.990+0000 7f061ea00640 1 -- 172.21.15.40:0/245422334 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0604005730 con 0x7f0604005bc0 2024-04-26T23:06:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.990+0000 7f061f400640 1 -- 172.21.15.40:0/245422334 reap_dead start 2024-04-26T23:06:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.991+0000 7f061ea00640 1 --2- 172.21.15.40:0/245422334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0604005bc0 0x7f0604005fe0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f0610001750 tx=0x7f0604007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc2e0c167585eaa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.991+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0610001370 con 0x7f0604005bc0 2024-04-26T23:06:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.991+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06100052d0 con 0x7f0604005bc0 2024-04-26T23:06:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.991+0000 7f061d600640 1 -- 172.21.15.40:0/245422334 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061002abe0 con 0x7f0604005bc0 2024-04-26T23:06:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.992+0000 7f061fe00640 1 -- 172.21.15.40:0/245422334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0604005bc0 msgr2=0x7f0604005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.992+0000 7f061fe00640 1 --2- 172.21.15.40:0/245422334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0604005bc0 0x7f0604005fe0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f0610001750 tx=0x7f0604007a40 comp rx=0 tx=0).stop 2024-04-26T23:06:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.992+0000 7f061fe00640 1 -- 172.21.15.40:0/245422334 shutdown_connections 2024-04-26T23:06:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.992+0000 7f061fe00640 1 --2- 172.21.15.40:0/245422334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0604005bc0 0x7f0604005fe0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.992+0000 7f061fe00640 1 -- 172.21.15.40:0/245422334 >> 172.21.15.40:0/245422334 conn(0x7f06200d0a10 msgr2=0x7f06200d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.992+0000 7f061fe00640 1 -- 172.21.15.40:0/245422334 shutdown_connections 2024-04-26T23:06:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.992+0000 7f061fe00640 1 -- 172.21.15.40:0/245422334 wait complete. 2024-04-26T23:06:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.994+0000 7f061fe00640 1 Processor -- start 2024-04-26T23:06:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.994+0000 7f061fe00640 1 -- start start 2024-04-26T23:06:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.995+0000 7f061fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0604005bc0 0x7f06201500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.995+0000 7f061fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06200c7ae0 0x7f06200c8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.995+0000 7f061fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06200c7f60 0x7f06200c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.995+0000 7f061fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f06200afdf0 con 0x7f0604005bc0 2024-04-26T23:06:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.995+0000 7f061fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f062006e270 con 0x7f06200c7ae0 2024-04-26T23:06:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.995+0000 7f061fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f06200aff70 con 0x7f06200c7f60 2024-04-26T23:06:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.996+0000 7f061ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0604005bc0 0x7f06201500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.996+0000 7f061ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0604005bc0 0x7f06201500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43152/0 (socket says 172.21.15.40:43152) 2024-04-26T23:06:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.996+0000 7f061ea00640 1 -- 172.21.15.40:0/2100025225 learned_addr learned my addr 172.21.15.40:0/2100025225 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.996+0000 7f061e000640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06200c7ae0 0x7f06200c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.996+0000 7f061ea00640 1 -- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06200c7f60 msgr2=0x7f06200c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.996+0000 7f061f400640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06200c7f60 0x7f06200c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.996+0000 7f061ea00640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06200c7f60 0x7f06200c8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.996+0000 7f061ea00640 1 -- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06200c7ae0 msgr2=0x7f06200c8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.996+0000 7f061ea00640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06200c7ae0 0x7f06200c8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.996+0000 7f061ea00640 1 -- 172.21.15.40:0/2100025225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06201f5480 con 0x7f0604005bc0 2024-04-26T23:06:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.997+0000 7f061ea00640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0604005bc0 0x7f06201500e0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f0610001870 tx=0x7f0610001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.997+0000 7f0617400640 1 -- 172.21.15.40:0/2100025225 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0610037040 con 0x7f0604005bc0 2024-04-26T23:06:43.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.997+0000 7f0617400640 1 -- 172.21.15.40:0/2100025225 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0610040070 con 0x7f0604005bc0 2024-04-26T23:06:43.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.997+0000 7f0617400640 1 -- 172.21.15.40:0/2100025225 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0610001370 con 0x7f0604005bc0 2024-04-26T23:06:43.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.997+0000 7f061fe00640 1 -- 172.21.15.40:0/2100025225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06201f5760 con 0x7f0604005bc0 2024-04-26T23:06:43.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.997+0000 7f061fe00640 1 -- 172.21.15.40:0/2100025225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06201f5d20 con 0x7f0604005bc0 2024-04-26T23:06:43.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.998+0000 7f061fe00640 1 -- 172.21.15.40:0/2100025225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0620069370 con 0x7f0604005bc0 2024-04-26T23:06:43.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:42.999+0000 7f0617400640 1 -- 172.21.15.40:0/2100025225 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f06100052d0 con 0x7f0604005bc0 2024-04-26T23:06:43.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.000+0000 7f0617400640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f060006c8f0 0x7f060006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:43.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.000+0000 7f0617400640 1 -- 172.21.15.40:0/2100025225 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6217+0+0 (secure 0 0 0) 0x7f06100abad0 con 0x7f0604005bc0 2024-04-26T23:06:43.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.001+0000 7f061e000640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f060006c8f0 0x7f060006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:43.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.002+0000 7f061e000640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f060006c8f0 0x7f060006edb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f06200b1850 tx=0x7f060c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:43.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.005+0000 7f0617400640 1 -- 172.21.15.40:0/2100025225 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0610079c60 con 0x7f0604005bc0 2024-04-26T23:06:43.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.241+0000 7f061fe00640 1 -- 172.21.15.40:0/2100025225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f06200d0370 con 0x7f0604005bc0 2024-04-26T23:06:43.242 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:06:43.242+0000 7f36fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T23:06:43.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.246+0000 7f0617400640 1 -- 172.21.15.40:0/2100025225 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) ==== 161+0+0 (secure 0 0 0) 0x7f0610079560 con 0x7f0604005bc0 2024-04-26T23:06:43.246 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T23:06:43.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.249+0000 7f061fe00640 1 -- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f060006c8f0 msgr2=0x7f060006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:43.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.249+0000 7f061fe00640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f060006c8f0 0x7f060006edb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f06200b1850 tx=0x7f060c008040 comp rx=0 tx=0).stop 2024-04-26T23:06:43.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.249+0000 7f061fe00640 1 -- 172.21.15.40:0/2100025225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0604005bc0 msgr2=0x7f06201500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:43.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.249+0000 7f061fe00640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0604005bc0 0x7f06201500e0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f0610001870 tx=0x7f0610001120 comp rx=0 tx=0).stop 2024-04-26T23:06:43.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.250+0000 7f061fe00640 1 -- 172.21.15.40:0/2100025225 shutdown_connections 2024-04-26T23:06:43.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.250+0000 7f061fe00640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f060006c8f0 0x7f060006edb0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:43.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.250+0000 7f061fe00640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06200c7f60 0x7f06200c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:43.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.250+0000 7f061fe00640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06200c7ae0 0x7f06200c8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:43.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.250+0000 7f061fe00640 1 --2- 172.21.15.40:0/2100025225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0604005bc0 0x7f06201500e0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:43.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.250+0000 7f061fe00640 1 -- 172.21.15.40:0/2100025225 >> 172.21.15.40:0/2100025225 conn(0x7f06200d0a10 msgr2=0x7f0620153230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:43.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.251+0000 7f061fe00640 1 -- 172.21.15.40:0/2100025225 shutdown_connections 2024-04-26T23:06:43.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.251+0000 7f061fe00640 1 -- 172.21.15.40:0/2100025225 wait complete. 2024-04-26T23:06:43.263 INFO:tasks.ceph.mds.c.smithi040.stderr: -8483> 2024-04-26T23:05:47.676+0000 7fa16ce00640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T23:06:43.268 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T23:06:43.273 INFO:tasks.ceph.mds.c.smithi040.stderr: -3756> 2024-04-26T23:06:03.606+0000 7fa167e00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000002 [...2,head] /d1/d2/d3/ auth{1=1,2=1} v19 ap=1 f(v0 m2024-04-26T23:05:45.942577+0000 1=0+1) n(v3 rc2024-04-26T23:05:46.223573+0000 6=0+6) (inest mix dirty) (ifile excl) caps={5372=pAsLsXsFsx/pAsLsXsFsxcral@4},l=5372 | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 0x5617f06dc680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000002:[<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:05:45.942577+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:05:46.223573+0000 6=0+6)","memory_value.dirstat":"f(v0 m2024-04-26T23:05:45.942577+0000 1=0+1)","memory_value.rstat":"n(v3 rc2024-04-26T23:05:46.223573+0000 6=0+6)","error_str":""},"return_code":-61} 2024-04-26T23:06:43.273 INFO:tasks.ceph.mds.c.smithi040.stderr: -3647> 2024-04-26T23:06:03.609+0000 7fa170000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000003 [...2,head] /d1/d2/d3/d4/ auth{1=1,2=1} v19 ap=2 f(v0 m2024-04-26T23:05:45.943577+0000 1=0+1) n(v4 rc2024-04-26T23:05:46.223573+0000 5=0+5) (inest sync->mix g=1,2) caps={5372=pAsLsXsFs/pAsLsXsFsxcral@8} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 export_pin=1 0x5617f06dcc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000003:[<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:05:45.943577+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:05:46.223573+0000 5=0+5)","memory_value.dirstat":"f(v0 m2024-04-26T23:05:45.943577+0000 1=0+1)","memory_value.rstat":"n(v4 rc2024-04-26T23:05:46.223573+0000 5=0+5)","error_str":""},"return_code":-61} 2024-04-26T23:06:43.290 INFO:tasks.ceph.mds.a.smithi040.stderr: -1743> 2024-04-26T23:06:03.606+0000 7fa834400640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000004 [...2,head] /d1/d2/d3/d4/d5/ auth{0=1,2=1} v15 ap=1 f(v0 m2024-04-26T23:05:45.944577+0000 1=0+1) n(v2 rc2024-04-26T23:05:46.223573+0000 4=0+4) (inest lock) (ifile excl) caps={5372=pAsLsXsFsx/pAsLsXsFsxcral@4},l=5372 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 authpin=1 scrubqueue=0 0x56322dc9a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000004:[<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:05:45.944577+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:05:46.223573+0000 4=0+4)","memory_value.dirstat":"f(v0 m2024-04-26T23:05:45.944577+0000 1=0+1)","memory_value.rstat":"n(v2 rc2024-04-26T23:05:46.223573+0000 4=0+4)","error_str":""},"return_code":-61} 2024-04-26T23:06:43.290 INFO:tasks.ceph.mds.a.smithi040.stderr: -1603> 2024-04-26T23:06:03.609+0000 7fa83c600640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000005 [...2,head] /d1/d2/d3/d4/d5/d6/ auth{0=1,2=1} v11 ap=2 f(v0 m2024-04-26T23:05:45.946577+0000 1=0+1) n(v1 rc2024-04-26T23:05:46.223573+0000 3=0+3) (inest sync->mix g=0,2 scatter_wanted) caps={5372=pAsLsXsFs/pAsLsXsFsxcral@7} | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 export_pin=2 0x56322dc9ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000005:[<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:05:45.946577+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:05:45.948577+0000 3=0+3)","memory_value.dirstat":"f(v0 m2024-04-26T23:05:45.946577+0000 1=0+1)","memory_value.rstat":"n(v1 rc2024-04-26T23:05:46.223573+0000 3=0+3)","error_str":""},"return_code":-61} 2024-04-26T23:06:43.293 INFO:tasks.ceph.mds.b.smithi103.stderr: -1641> 2024-04-26T23:06:03.608+0000 7f043fa00640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000006 [...2,head] /d1/d2/d3/d4/d5/d6/d7/ auth v4 ap=1 f(v0 m2024-04-26T23:05:45.948577+0000 1=0+1) n(v0 rc2024-04-26T23:05:45.948577+0000 2=0+2) (inest lock) (ifile excl) caps={5372=pAsLsXsFsx/pAsLsXsFsxcral@4},l=5372 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x561447704000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000006:[<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:05:45.948577+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:05:45.948577+0000 2=0+2)","memory_value.dirstat":"f(v0 m2024-04-26T23:05:45.948577+0000 1=0+1)","memory_value.rstat":"n(v0 rc2024-04-26T23:05:45.948577+0000 2=0+2)","error_str":""},"return_code":-61} 2024-04-26T23:06:43.293 INFO:tasks.ceph.mds.b.smithi103.stderr: -1624> 2024-04-26T23:06:03.608+0000 7f043fa00640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000007 [...2,head] /d1/d2/d3/d4/d5/d6/d7/d8/ auth v2 ap=1 f() n(v0 1=0+1) (inest lock) (ifile excl) caps={5372=pAsLsXsFsx/AsLsXsFsx@2},l=5372 | importingcaps=0 dirfrag=0 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x561447704580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000007:[<0x10000000006/d8 v2>,<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 1=0+1)","error_str":""},"return_code":-61} 2024-04-26T23:06:43.321 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:06:43.340 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-04-26T23:06:43.372 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-26T23:06:43.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.450+0000 7fe1fd600640 1 Processor -- start 2024-04-26T23:06:43.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.450+0000 7fe1fd600640 1 -- start start 2024-04-26T23:06:43.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.451+0000 7fe1fd600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1f80cf600 0x7fe1f80cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:43.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.451+0000 7fe1fd600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe1f8071d50 0x7fe1f806ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:43.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.451+0000 7fe1fd600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1f80b2470 con 0x7fe1f80704e0 2024-04-26T23:06:43.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.451+0000 7fe1fd600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe1f8071670 con 0x7fe1f8071d50 2024-04-26T23:06:43.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.451+0000 7fe1fd600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe1f80d3b90 con 0x7fe1f80cf600 2024-04-26T23:06:43.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.452+0000 7fe1f7400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe1f8071d50 0x7fe1f806ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:43.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.452+0000 7fe1f7400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe1f8071d50 0x7fe1f806ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48458/0 (socket says 172.21.15.40:48458) 2024-04-26T23:06:43.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.452+0000 7fe1f7400640 1 -- 172.21.15.40:0/1505939279 learned_addr learned my addr 172.21.15.40:0/1505939279 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:43.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.452+0000 7fe1f7e00640 1 --2- 172.21.15.40:0/1505939279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1f80cf600 0x7fe1f80cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:43.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.453+0000 7fe1f7400640 1 -- 172.21.15.40:0/1505939279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1f80cf600 msgr2=0x7fe1f80cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:43.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.453+0000 7fe1f7400640 1 --2- 172.21.15.40:0/1505939279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1f80cf600 0x7fe1f80cf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:43.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.453+0000 7fe1f7400640 1 -- 172.21.15.40:0/1505939279 >> v1:172.21.15.103:6789/0 conn(0x7fe1f80704e0 legacy=0x7fe1f80ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:43.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.453+0000 7fe1f7400640 1 -- 172.21.15.40:0/1505939279 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f80cd620 con 0x7fe1f8071d50 2024-04-26T23:06:43.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.453+0000 7fe1f7e00640 1 --2- 172.21.15.40:0/1505939279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1f80cf600 0x7fe1f80cf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:06:43.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.454+0000 7fe1f7400640 1 --2- 172.21.15.40:0/1505939279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1f8071d50 0x7fe1f806ffa0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009e50 tx=0x7fe1e802a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f864bc3659721c2e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:43.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.454+0000 7fe1f6a00640 1 -- 172.21.15.40:0/1505939279 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8036070 con 0x7fe1f8071d50 2024-04-26T23:06:43.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.455+0000 7fe1f6a00640 1 -- 172.21.15.40:0/1505939279 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1e8031600 con 0x7fe1f8071d50 2024-04-26T23:06:43.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.455+0000 7fe1f6a00640 1 -- 172.21.15.40:0/1505939279 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e80319d0 con 0x7fe1f8071d50 2024-04-26T23:06:43.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.455+0000 7fe1fd600640 1 -- 172.21.15.40:0/1505939279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1f8071d50 msgr2=0x7fe1f806ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:43.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.455+0000 7fe1fd600640 1 --2- 172.21.15.40:0/1505939279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1f8071d50 0x7fe1f806ffa0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009e50 tx=0x7fe1e802a710 comp rx=0 tx=0).stop 2024-04-26T23:06:43.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.455+0000 7fe1fd600640 1 -- 172.21.15.40:0/1505939279 shutdown_connections 2024-04-26T23:06:43.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.455+0000 7fe1fd600640 1 --2- 172.21.15.40:0/1505939279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1f8071d50 0x7fe1f806ffa0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:43.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.455+0000 7fe1fd600640 1 --2- 172.21.15.40:0/1505939279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1f80cf600 0x7fe1f80cf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:43.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.455+0000 7fe1fd600640 1 -- 172.21.15.40:0/1505939279 >> 172.21.15.40:0/1505939279 conn(0x7fe1f80d2bf0 msgr2=0x7fe1f80d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:43.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.455+0000 7fe1fd600640 1 -- 172.21.15.40:0/1505939279 shutdown_connections 2024-04-26T23:06:43.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.455+0000 7fe1fd600640 1 -- 172.21.15.40:0/1505939279 wait complete. 2024-04-26T23:06:43.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.457+0000 7fe1fd600640 1 Processor -- start 2024-04-26T23:06:43.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.457+0000 7fe1fd600640 1 -- start start 2024-04-26T23:06:43.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.458+0000 7fe1fd600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1f80704e0 0x7fe1f809da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:43.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.458+0000 7fe1fd600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1f8071d50 0x7fe1f80b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:43.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.458+0000 7fe1fd600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1f80cf600 0x7fe1f80c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:43.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.458+0000 7fe1fd600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f80cd8a0 con 0x7fe1f80704e0 2024-04-26T23:06:43.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.458+0000 7fe1fd600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f80d38b0 con 0x7fe1f8071d50 2024-04-26T23:06:43.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.458+0000 7fe1fd600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe1f80cdb80 con 0x7fe1f80cf600 2024-04-26T23:06:43.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.458+0000 7fe1f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1f80704e0 0x7fe1f809da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:43.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.458+0000 7fe1f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1f80704e0 0x7fe1f809da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43210/0 (socket says 172.21.15.40:43210) 2024-04-26T23:06:43.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.458+0000 7fe1f7e00640 1 -- 172.21.15.40:0/3395856928 learned_addr learned my addr 172.21.15.40:0/3395856928 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:43.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.459+0000 7fe1f7400640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1f8071d50 0x7fe1f80b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:43.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.459+0000 7fe1f7e00640 1 -- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1f80cf600 msgr2=0x7fe1f80c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:43.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.459+0000 7fe1fcc00640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1f80cf600 0x7fe1f80c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:43.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.459+0000 7fe1f7e00640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1f80cf600 0x7fe1f80c7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:43.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.459+0000 7fe1f7e00640 1 -- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1f8071d50 msgr2=0x7fe1f80b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:43.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.459+0000 7fe1f7e00640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1f8071d50 0x7fe1f80b12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:43.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.459+0000 7fe1f7e00640 1 -- 172.21.15.40:0/3395856928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f80c8330 con 0x7fe1f80704e0 2024-04-26T23:06:43.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.459+0000 7fe1f7400640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1f8071d50 0x7fe1f80b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:43.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.459+0000 7fe1f7e00640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1f80704e0 0x7fe1f809da60 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000e810 tx=0x7fe1e000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:43.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.459+0000 7fe1efe00640 1 -- 172.21.15.40:0/3395856928 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e000f040 con 0x7fe1f80704e0 2024-04-26T23:06:43.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.460+0000 7fe1efe00640 1 -- 172.21.15.40:0/3395856928 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1e0004030 con 0x7fe1f80704e0 2024-04-26T23:06:43.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.460+0000 7fe1efe00640 1 -- 172.21.15.40:0/3395856928 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0004400 con 0x7fe1f80704e0 2024-04-26T23:06:43.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.460+0000 7fe1fd600640 1 -- 172.21.15.40:0/3395856928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1f80c9d60 con 0x7fe1f80704e0 2024-04-26T23:06:43.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.460+0000 7fe1fd600640 1 -- 172.21.15.40:0/3395856928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1f80ca320 con 0x7fe1f80704e0 2024-04-26T23:06:43.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.461+0000 7fe1fd600640 1 -- 172.21.15.40:0/3395856928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1f8069a80 con 0x7fe1f80704e0 2024-04-26T23:06:43.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.462+0000 7fe1efe00640 1 -- 172.21.15.40:0/3395856928 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe1e000c6d0 con 0x7fe1f80704e0 2024-04-26T23:06:43.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.463+0000 7fe1efe00640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1d406c8f0 0x7fe1d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:43.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.463+0000 7fe1efe00640 1 -- 172.21.15.40:0/3395856928 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6475+0+0 (secure 0 0 0) 0x7fe1e008bc70 con 0x7fe1f80704e0 2024-04-26T23:06:43.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.464+0000 7fe1f7400640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1d406c8f0 0x7fe1d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:43.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.465+0000 7fe1f7400640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1d406c8f0 0x7fe1d406edb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8033000 tx=0x7fe1e8009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:43.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.467+0000 7fe1efe00640 1 -- 172.21.15.40:0/3395856928 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe1e0017020 con 0x7fe1f80704e0 2024-04-26T23:06:43.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:43.713+0000 7fe1fd600640 1 -- 172.21.15.40:0/3395856928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fe1f806ccb0 con 0x7fe1f80704e0 2024-04-26T23:06:44.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.249+0000 7fe1efe00640 1 -- 172.21.15.40:0/3395856928 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v68) ==== 104+0+0 (secure 0 0 0) 0x7fe1e0059d90 con 0x7fe1f80704e0 2024-04-26T23:06:44.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.253+0000 7fe1fd600640 1 -- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1d406c8f0 msgr2=0x7fe1d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:44.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.253+0000 7fe1fd600640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1d406c8f0 0x7fe1d406edb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8033000 tx=0x7fe1e8009000 comp rx=0 tx=0).stop 2024-04-26T23:06:44.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.253+0000 7fe1fd600640 1 -- 172.21.15.40:0/3395856928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1f80704e0 msgr2=0x7fe1f809da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:44.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.253+0000 7fe1fd600640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1f80704e0 0x7fe1f809da60 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000e810 tx=0x7fe1e000ecd0 comp rx=0 tx=0).stop 2024-04-26T23:06:44.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.253+0000 7fe1fd600640 1 -- 172.21.15.40:0/3395856928 shutdown_connections 2024-04-26T23:06:44.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.253+0000 7fe1fd600640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe1d406c8f0 0x7fe1d406edb0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.253+0000 7fe1fd600640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1f80cf600 0x7fe1f80c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.253+0000 7fe1fd600640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1f8071d50 0x7fe1f80b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.253+0000 7fe1fd600640 1 --2- 172.21.15.40:0/3395856928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1f80704e0 0x7fe1f809da60 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.253+0000 7fe1fd600640 1 -- 172.21.15.40:0/3395856928 >> 172.21.15.40:0/3395856928 conn(0x7fe1f80d2bf0 msgr2=0x7fe1f80d09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.253+0000 7fe1fd600640 1 -- 172.21.15.40:0/3395856928 shutdown_connections 2024-04-26T23:06:44.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.253+0000 7fe1fd600640 1 -- 172.21.15.40:0/3395856928 wait complete. 2024-04-26T23:06:44.277 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T23:06:44.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.542+0000 7fbfe9600640 1 Processor -- start 2024-04-26T23:06:44.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.542+0000 7fbfe9600640 1 -- start start 2024-04-26T23:06:44.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.543+0000 7fbfe9600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbfe4146790 0x7fbfe415dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:44.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.544+0000 7fbfe9600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbfe40b5770 0x7fbfe40b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:44.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.544+0000 7fbfe9600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfe415e260 con 0x7fbfe406de30 2024-04-26T23:06:44.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.544+0000 7fbfe9600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fbfe40d04e0 con 0x7fbfe4146790 2024-04-26T23:06:44.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.544+0000 7fbfe9600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbfe40d07c0 con 0x7fbfe40b5770 2024-04-26T23:06:44.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.544+0000 7fbfe3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbfe4146790 0x7fbfe415dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:44.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.544+0000 7fbfe3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbfe4146790 0x7fbfe415dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43212/0 (socket says 172.21.15.40:43212) 2024-04-26T23:06:44.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.544+0000 7fbfe3e00640 1 -- 172.21.15.40:0/2193274083 learned_addr learned my addr 172.21.15.40:0/2193274083 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:44.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.544+0000 7fbfe3400640 1 --2- 172.21.15.40:0/2193274083 >> v2:172.21.15.103:3300/0 conn(0x7fbfe40b5770 0x7fbfe40b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:44.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.544+0000 7fbfe3e00640 1 -- 172.21.15.40:0/2193274083 >> v2:172.21.15.103:3300/0 conn(0x7fbfe40b5770 msgr2=0x7fbfe40b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:44.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.544+0000 7fbfe3e00640 1 --2- 172.21.15.40:0/2193274083 >> v2:172.21.15.103:3300/0 conn(0x7fbfe40b5770 0x7fbfe40b5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:44.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.544+0000 7fbfe3e00640 1 -- 172.21.15.40:0/2193274083 >> v1:172.21.15.40:6789/0 conn(0x7fbfe406de30 legacy=0x7fbfe40cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:44.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.545+0000 7fbfe3e00640 1 -- 172.21.15.40:0/2193274083 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe40b4c90 con 0x7fbfe4146790 2024-04-26T23:06:44.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.546+0000 7fbfe3e00640 1 --2- 172.21.15.40:0/2193274083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfe4146790 0x7fbfe415dd20 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4160820 tx=0x7fbfd402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d00cd85cb6860bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:44.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.546+0000 7fbfe2a00640 1 -- 172.21.15.40:0/2193274083 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4036070 con 0x7fbfe4146790 2024-04-26T23:06:44.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.546+0000 7fbfe2a00640 1 -- 172.21.15.40:0/2193274083 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfd4031600 con 0x7fbfe4146790 2024-04-26T23:06:44.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.546+0000 7fbfe2a00640 1 -- 172.21.15.40:0/2193274083 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd40319d0 con 0x7fbfe4146790 2024-04-26T23:06:44.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.547+0000 7fbfe9600640 1 -- 172.21.15.40:0/2193274083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfe4146790 msgr2=0x7fbfe415dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:44.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.547+0000 7fbfe9600640 1 --2- 172.21.15.40:0/2193274083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfe4146790 0x7fbfe415dd20 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4160820 tx=0x7fbfd402a6a0 comp rx=0 tx=0).stop 2024-04-26T23:06:44.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.547+0000 7fbfe9600640 1 -- 172.21.15.40:0/2193274083 shutdown_connections 2024-04-26T23:06:44.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.547+0000 7fbfe9600640 1 --2- 172.21.15.40:0/2193274083 >> v2:172.21.15.103:3300/0 conn(0x7fbfe40b5770 0x7fbfe40b5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:44.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.547+0000 7fbfe9600640 1 --2- 172.21.15.40:0/2193274083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfe4146790 0x7fbfe415dd20 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:44.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.547+0000 7fbfe9600640 1 -- 172.21.15.40:0/2193274083 >> 172.21.15.40:0/2193274083 conn(0x7fbfe4075f10 msgr2=0x7fbfe4068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:44.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.547+0000 7fbfe9600640 1 -- 172.21.15.40:0/2193274083 shutdown_connections 2024-04-26T23:06:44.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.547+0000 7fbfe9600640 1 -- 172.21.15.40:0/2193274083 wait complete. 2024-04-26T23:06:44.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.549+0000 7fbfe9600640 1 Processor -- start 2024-04-26T23:06:44.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.549+0000 7fbfe9600640 1 -- start start 2024-04-26T23:06:44.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.550+0000 7fbfe9600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfe406de30 0x7fbfe40cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:44.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.550+0000 7fbfe9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfe40b5770 0x7fbfe415c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:44.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.550+0000 7fbfe9600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbfe4146790 0x7fbfe4141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:44.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.550+0000 7fbfe9600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe40d00b0 con 0x7fbfe40b5770 2024-04-26T23:06:44.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.550+0000 7fbfe9600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe40b5250 con 0x7fbfe406de30 2024-04-26T23:06:44.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.550+0000 7fbfe9600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe40d0230 con 0x7fbfe4146790 2024-04-26T23:06:44.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.550+0000 7fbfe3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfe40b5770 0x7fbfe415c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:44.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.550+0000 7fbfe3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfe40b5770 0x7fbfe415c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43220/0 (socket says 172.21.15.40:43220) 2024-04-26T23:06:44.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.550+0000 7fbfe3400640 1 -- 172.21.15.40:0/3368879018 learned_addr learned my addr 172.21.15.40:0/3368879018 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:44.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfe8c00640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbfe4146790 0x7fbfe4141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfe3e00640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfe406de30 0x7fbfe40cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfe3400640 1 -- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbfe4146790 msgr2=0x7fbfe4141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfe3400640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbfe4146790 0x7fbfe4141360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfe3400640 1 -- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfe406de30 msgr2=0x7fbfe40cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfe3400640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfe406de30 0x7fbfe40cc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfe3400640 1 -- 172.21.15.40:0/3368879018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe415f3f0 con 0x7fbfe40b5770 2024-04-26T23:06:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfe8c00640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbfe4146790 0x7fbfe4141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfe3e00640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfe406de30 0x7fbfe40cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfe3400640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfe40b5770 0x7fbfe415c620 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc00c7f0 tx=0x7fbfcc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfdbe00640 1 -- 172.21.15.40:0/3368879018 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc015070 con 0x7fbfe40b5770 2024-04-26T23:06:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfdbe00640 1 -- 172.21.15.40:0/3368879018 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfcc004220 con 0x7fbfe40b5770 2024-04-26T23:06:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfe9600640 1 -- 172.21.15.40:0/3368879018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe415e630 con 0x7fbfe40b5770 2024-04-26T23:06:44.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.551+0000 7fbfdbe00640 1 -- 172.21.15.40:0/3368879018 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc003610 con 0x7fbfe40b5770 2024-04-26T23:06:44.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.552+0000 7fbfe9600640 1 -- 172.21.15.40:0/3368879018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe415ebf0 con 0x7fbfe40b5770 2024-04-26T23:06:44.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.553+0000 7fbfe9600640 1 -- 172.21.15.40:0/3368879018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfe4069370 con 0x7fbfe40b5770 2024-04-26T23:06:44.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.554+0000 7fbfdbe00640 1 -- 172.21.15.40:0/3368879018 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbfcc0039c0 con 0x7fbfe40b5770 2024-04-26T23:06:44.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.555+0000 7fbfdbe00640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfc006c8f0 0x7fbfc006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:44.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.555+0000 7fbfdbe00640 1 -- 172.21.15.40:0/3368879018 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6475+0+0 (secure 0 0 0) 0x7fbfcc08c140 con 0x7fbfe40b5770 2024-04-26T23:06:44.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.555+0000 7fbfe3e00640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfc006c8f0 0x7fbfc006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:44.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.557+0000 7fbfe3e00640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfc006c8f0 0x7fbfc006edb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4033000 tx=0x7fbfd4009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:44.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.559+0000 7fbfdbe00640 1 -- 172.21.15.40:0/3368879018 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fbfcc05a260 con 0x7fbfe40b5770 2024-04-26T23:06:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:44.769+0000 7fbfe9600640 1 -- 172.21.15.40:0/3368879018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbfe40ce5e0 con 0x7fbfe40b5770 2024-04-26T23:06:45.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.262+0000 7fbfdbe00640 1 -- 172.21.15.40:0/3368879018 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v69) ==== 181+0+0 (secure 0 0 0) 0x7fbfcc059b60 con 0x7fbfe40b5770 2024-04-26T23:06:45.263 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-04-26T23:06:45.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.265+0000 7fbfe9600640 1 -- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfc006c8f0 msgr2=0x7fbfc006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:45.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.265+0000 7fbfe9600640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfc006c8f0 0x7fbfc006edb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4033000 tx=0x7fbfd4009000 comp rx=0 tx=0).stop 2024-04-26T23:06:45.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.265+0000 7fbfe9600640 1 -- 172.21.15.40:0/3368879018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfe40b5770 msgr2=0x7fbfe415c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:45.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.265+0000 7fbfe9600640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfe40b5770 0x7fbfe415c620 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc00c7f0 tx=0x7fbfcc00ccb0 comp rx=0 tx=0).stop 2024-04-26T23:06:45.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.266+0000 7fbfe9600640 1 -- 172.21.15.40:0/3368879018 shutdown_connections 2024-04-26T23:06:45.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.266+0000 7fbfe9600640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfc006c8f0 0x7fbfc006edb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:45.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.266+0000 7fbfe9600640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbfe4146790 0x7fbfe4141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:45.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.266+0000 7fbfe9600640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfe40b5770 0x7fbfe415c620 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:45.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.266+0000 7fbfe9600640 1 --2- 172.21.15.40:0/3368879018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfe406de30 0x7fbfe40cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:45.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.266+0000 7fbfe9600640 1 -- 172.21.15.40:0/3368879018 >> 172.21.15.40:0/3368879018 conn(0x7fbfe4075f10 msgr2=0x7fbfe4068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:45.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.266+0000 7fbfe9600640 1 -- 172.21.15.40:0/3368879018 shutdown_connections 2024-04-26T23:06:45.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.266+0000 7fbfe9600640 1 -- 172.21.15.40:0/3368879018 wait complete. 2024-04-26T23:06:45.290 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T23:06:45.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.499+0000 7f7969600640 1 Processor -- start 2024-04-26T23:06:45.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.499+0000 7f7969600640 1 -- start start 2024-04-26T23:06:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.500+0000 7f7969600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7964145970 0x7f79641612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.500+0000 7f7969600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79640d06b0 0x7f79640d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.500+0000 7f7969600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79640b59f0 con 0x7f7964146060 2024-04-26T23:06:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.500+0000 7f7969600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f79640ceed0 con 0x7f7964145970 2024-04-26T23:06:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.500+0000 7f7969600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f79640d1040 con 0x7f79640d06b0 2024-04-26T23:06:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.501+0000 7f7963e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7964145970 0x7f79641612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.501+0000 7f7963400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7964146060 0x7f79640b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35388/0 (socket says 172.21.15.40:35388) 2024-04-26T23:06:45.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.501+0000 7f7963400640 1 -- 172.21.15.40:0/4207288043 learned_addr learned my addr 172.21.15.40:0/4207288043 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.501+0000 7f7968c00640 1 --2- 172.21.15.40:0/4207288043 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79640d06b0 0x7f79640d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.501+0000 7f7963e00640 1 -- 172.21.15.40:0/4207288043 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79640d06b0 msgr2=0x7f79640d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.501+0000 7f7963e00640 1 --2- 172.21.15.40:0/4207288043 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79640d06b0 0x7f79640d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.501+0000 7f7963e00640 1 -- 172.21.15.40:0/4207288043 >> v1:172.21.15.40:6789/0 conn(0x7f7964146060 legacy=0x7f79640b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.501+0000 7f7963e00640 1 -- 172.21.15.40:0/4207288043 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79641617f0 con 0x7f7964145970 2024-04-26T23:06:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.501+0000 7f7962a00640 1 -- 172.21.15.40:0/4207288043 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1046947233 0 0) 0x7f79640b59f0 con 0x7f7964146060 2024-04-26T23:06:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.503+0000 7f7963e00640 1 --2- 172.21.15.40:0/4207288043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7964145970 0x7f79641612b0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f79540097a0 tx=0x7f795402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44610147e7b341e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.503+0000 7f7962a00640 1 -- 172.21.15.40:0/4207288043 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954037070 con 0x7f7964145970 2024-04-26T23:06:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.503+0000 7f7962a00640 1 -- 172.21.15.40:0/4207288043 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7954032440 con 0x7f7964145970 2024-04-26T23:06:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.503+0000 7f7962a00640 1 -- 172.21.15.40:0/4207288043 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954032810 con 0x7f7964145970 2024-04-26T23:06:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.503+0000 7f7969600640 1 -- 172.21.15.40:0/4207288043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7964145970 msgr2=0x7f79641612b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.503+0000 7f7969600640 1 --2- 172.21.15.40:0/4207288043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7964145970 0x7f79641612b0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f79540097a0 tx=0x7f795402a180 comp rx=0 tx=0).stop 2024-04-26T23:06:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.504+0000 7f7969600640 1 -- 172.21.15.40:0/4207288043 shutdown_connections 2024-04-26T23:06:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.504+0000 7f7969600640 1 --2- 172.21.15.40:0/4207288043 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79640d06b0 0x7f79640d0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.504+0000 7f7969600640 1 --2- 172.21.15.40:0/4207288043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7964145970 0x7f79641612b0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.504+0000 7f7969600640 1 -- 172.21.15.40:0/4207288043 >> 172.21.15.40:0/4207288043 conn(0x7f7964075eb0 msgr2=0x7f7964068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.504+0000 7f7969600640 1 -- 172.21.15.40:0/4207288043 shutdown_connections 2024-04-26T23:06:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.504+0000 7f7969600640 1 -- 172.21.15.40:0/4207288043 wait complete. 2024-04-26T23:06:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.505+0000 7f7969600640 1 Processor -- start 2024-04-26T23:06:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.506+0000 7f7969600640 1 -- start start 2024-04-26T23:06:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.506+0000 7f7969600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79640d06b0 0x7f796415a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.506+0000 7f7969600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7964145970 0x7f79641f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.506+0000 7f7969600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7964146060 0x7f7964140010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.506+0000 7f7969600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f79640b5c60 con 0x7f7964145970 2024-04-26T23:06:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.506+0000 7f7969600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79640b5f40 con 0x7f7964146060 2024-04-26T23:06:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.506+0000 7f7969600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f79640cf100 con 0x7f79640d06b0 2024-04-26T23:06:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.507+0000 7f7963400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7964145970 0x7f79641f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.507+0000 7f7963400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7964145970 0x7f79641f46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58624/0 (socket says 172.21.15.40:58624) 2024-04-26T23:06:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.507+0000 7f7963400640 1 -- 172.21.15.40:0/2293641640 learned_addr learned my addr 172.21.15.40:0/2293641640 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.507+0000 7f7963400640 1 -- 172.21.15.40:0/2293641640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79640d06b0 msgr2=0x7f796415a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.507+0000 7f7963400640 1 --2- 172.21.15.40:0/2293641640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79640d06b0 0x7f796415a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.507+0000 7f7963400640 1 -- 172.21.15.40:0/2293641640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7964146060 msgr2=0x7f7964140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.507+0000 7f7968c00640 1 --2- 172.21.15.40:0/2293641640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7964146060 0x7f7964140010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.507+0000 7f7963400640 1 --2- 172.21.15.40:0/2293641640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7964146060 0x7f7964140010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.507+0000 7f7963400640 1 -- 172.21.15.40:0/2293641640 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79641591a0 con 0x7f7964145970 2024-04-26T23:06:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.507+0000 7f7963400640 1 --2- 172.21.15.40:0/2293641640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7964145970 0x7f79641f46b0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f794c00c430 tx=0x7f794c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.508+0000 7f795be00640 1 -- 172.21.15.40:0/2293641640 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c0074e0 con 0x7f7964145970 2024-04-26T23:06:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.508+0000 7f795be00640 1 -- 172.21.15.40:0/2293641640 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f794c0047e0 con 0x7f7964145970 2024-04-26T23:06:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.508+0000 7f7969600640 1 -- 172.21.15.40:0/2293641640 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7964159460 con 0x7f7964145970 2024-04-26T23:06:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.508+0000 7f795be00640 1 -- 172.21.15.40:0/2293641640 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c004bd0 con 0x7f7964145970 2024-04-26T23:06:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.508+0000 7f7969600640 1 -- 172.21.15.40:0/2293641640 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7964159a20 con 0x7f7964145970 2024-04-26T23:06:45.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.509+0000 7f7969600640 1 -- 172.21.15.40:0/2293641640 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7964069370 con 0x7f7964145970 2024-04-26T23:06:45.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.510+0000 7f795be00640 1 -- 172.21.15.40:0/2293641640 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f794c011070 con 0x7f7964145970 2024-04-26T23:06:45.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.511+0000 7f795be00640 1 --2- 172.21.15.40:0/2293641640 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f794006c8f0 0x7f794006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:45.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.511+0000 7f795be00640 1 -- 172.21.15.40:0/2293641640 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6008+0+0 (secure 0 0 0) 0x7f794c08b8f0 con 0x7f7964145970 2024-04-26T23:06:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.512+0000 7f7963e00640 1 --2- 172.21.15.40:0/2293641640 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f794006c8f0 0x7f794006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:45.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.513+0000 7f7963e00640 1 --2- 172.21.15.40:0/2293641640 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f794006c8f0 0x7f794006edb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7954034000 tx=0x7f795402c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.516+0000 7f795be00640 1 -- 172.21.15.40:0/2293641640 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f794c00e280 con 0x7f7964145970 2024-04-26T23:06:45.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:45.731+0000 7f7969600640 1 -- 172.21.15.40:0/2293641640 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f79640d1400 con 0x7f7964145970 2024-04-26T23:06:46.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.272+0000 7f795be00640 1 -- 172.21.15.40:0/2293641640 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v70) ==== 169+0+0 (secure 0 0 0) 0x7f794c059be0 con 0x7f7964145970 2024-04-26T23:06:46.273 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-04-26T23:06:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.276+0000 7f7969600640 1 -- 172.21.15.40:0/2293641640 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f794006c8f0 msgr2=0x7f794006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.276+0000 7f7969600640 1 --2- 172.21.15.40:0/2293641640 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f794006c8f0 0x7f794006edb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7954034000 tx=0x7f795402c000 comp rx=0 tx=0).stop 2024-04-26T23:06:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.276+0000 7f7969600640 1 -- 172.21.15.40:0/2293641640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7964145970 msgr2=0x7f79641f46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.276+0000 7f7969600640 1 --2- 172.21.15.40:0/2293641640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7964145970 0x7f79641f46b0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f794c00c430 tx=0x7f794c00c8f0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.276+0000 7f7969600640 1 -- 172.21.15.40:0/2293641640 shutdown_connections 2024-04-26T23:06:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.276+0000 7f7969600640 1 --2- 172.21.15.40:0/2293641640 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f794006c8f0 0x7f794006edb0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.276+0000 7f7969600640 1 --2- 172.21.15.40:0/2293641640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7964146060 0x7f7964140010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.276+0000 7f7969600640 1 --2- 172.21.15.40:0/2293641640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7964145970 0x7f79641f46b0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.276+0000 7f7969600640 1 --2- 172.21.15.40:0/2293641640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79640d06b0 0x7f796415a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.276+0000 7f7969600640 1 -- 172.21.15.40:0/2293641640 >> 172.21.15.40:0/2293641640 conn(0x7f7964075eb0 msgr2=0x7f79640b4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.277+0000 7f7969600640 1 -- 172.21.15.40:0/2293641640 shutdown_connections 2024-04-26T23:06:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.277+0000 7f7969600640 1 -- 172.21.15.40:0/2293641640 wait complete. 2024-04-26T23:06:46.303 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-26T23:06:46.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.512+0000 7f4c16200640 1 Processor -- start 2024-04-26T23:06:46.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.512+0000 7f4c16200640 1 -- start start 2024-04-26T23:06:46.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.514+0000 7f4c16200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4c100af7a0 0x7f4c100afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:46.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.514+0000 7f4c16200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c100d0820 con 0x7f4c1015ce60 2024-04-26T23:06:46.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.514+0000 7f4c16200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c101614c0 con 0x7f4c10071fe0 2024-04-26T23:06:46.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.514+0000 7f4c16200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f4c100cc450 con 0x7f4c100af7a0 2024-04-26T23:06:46.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.514+0000 7f4c0fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4c100af7a0 0x7f4c100afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:46.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.515+0000 7f4c14e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4c1015ce60 0x7f4c100ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35396/0 (socket says 172.21.15.40:35396) 2024-04-26T23:06:46.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.515+0000 7f4c14e00640 1 -- 172.21.15.40:0/3061974436 learned_addr learned my addr 172.21.15.40:0/3061974436 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:46.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.515+0000 7f4c0fe00640 1 -- 172.21.15.40:0/3061974436 >> v1:172.21.15.103:6789/0 conn(0x7f4c10071fe0 legacy=0x7f4c10073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:46.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.515+0000 7f4c0fe00640 1 -- 172.21.15.40:0/3061974436 >> v1:172.21.15.40:6789/0 conn(0x7f4c1015ce60 legacy=0x7f4c100ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:46.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.515+0000 7f4c0fe00640 1 -- 172.21.15.40:0/3061974436 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c10161760 con 0x7f4c100af7a0 2024-04-26T23:06:46.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.515+0000 7f4c0f400640 1 -- 172.21.15.40:0/3061974436 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2963690159 0 0) 0x7f4c100d0820 con 0x7f4c1015ce60 2024-04-26T23:06:46.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.517+0000 7f4c0fe00640 1 --2- 172.21.15.40:0/3061974436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c100af7a0 0x7f4c100afc60 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0097a0 tx=0x7f4bfc02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa0fc7c05bd1396 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:46.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.517+0000 7f4c0f400640 1 -- 172.21.15.40:0/3061974436 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc037070 con 0x7f4c100af7a0 2024-04-26T23:06:46.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.517+0000 7f4c0f400640 1 -- 172.21.15.40:0/3061974436 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bfc02ae50 con 0x7f4c100af7a0 2024-04-26T23:06:46.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.517+0000 7f4c0f400640 1 -- 172.21.15.40:0/3061974436 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc032660 con 0x7f4c100af7a0 2024-04-26T23:06:46.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.517+0000 7f4c16200640 1 -- 172.21.15.40:0/3061974436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c100af7a0 msgr2=0x7f4c100afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:46.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.517+0000 7f4c16200640 1 --2- 172.21.15.40:0/3061974436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c100af7a0 0x7f4c100afc60 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0097a0 tx=0x7f4bfc02a140 comp rx=0 tx=0).stop 2024-04-26T23:06:46.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.518+0000 7f4c16200640 1 -- 172.21.15.40:0/3061974436 shutdown_connections 2024-04-26T23:06:46.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.518+0000 7f4c16200640 1 --2- 172.21.15.40:0/3061974436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c100af7a0 0x7f4c100afc60 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.518+0000 7f4c16200640 1 -- 172.21.15.40:0/3061974436 >> 172.21.15.40:0/3061974436 conn(0x7f4c100d0a10 msgr2=0x7f4c100d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:46.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.518+0000 7f4c16200640 1 -- 172.21.15.40:0/3061974436 shutdown_connections 2024-04-26T23:06:46.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.518+0000 7f4c16200640 1 -- 172.21.15.40:0/3061974436 wait complete. 2024-04-26T23:06:46.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.519+0000 7f4c16200640 1 Processor -- start 2024-04-26T23:06:46.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.520+0000 7f4c16200640 1 -- start start 2024-04-26T23:06:46.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.520+0000 7f4c16200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10071fe0 0x7f4c100b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:46.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.520+0000 7f4c16200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c100af7a0 0x7f4c1014db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c16200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c1015ce60 0x7f4c1015b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c16200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4c100cc1f0 con 0x7f4c10071fe0 2024-04-26T23:06:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c16200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c100cc070 con 0x7f4c100af7a0 2024-04-26T23:06:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c16200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4c10073e70 con 0x7f4c1015ce60 2024-04-26T23:06:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c14e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10071fe0 0x7f4c100b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c14e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10071fe0 0x7f4c100b2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58636/0 (socket says 172.21.15.40:58636) 2024-04-26T23:06:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c14e00640 1 -- 172.21.15.40:0/3625512114 learned_addr learned my addr 172.21.15.40:0/3625512114 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c0fe00640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c100af7a0 0x7f4c1014db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c15800640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c1015ce60 0x7f4c1015b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c14e00640 1 -- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c1015ce60 msgr2=0x7f4c1015b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c14e00640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c1015ce60 0x7f4c1015b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c14e00640 1 -- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c100af7a0 msgr2=0x7f4c1014db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c14e00640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c100af7a0 0x7f4c1014db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c14e00640 1 -- 172.21.15.40:0/3625512114 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1006f1f0 con 0x7f4c10071fe0 2024-04-26T23:06:46.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.521+0000 7f4c15800640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c1015ce60 0x7f4c1015b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:46.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.522+0000 7f4c14e00640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10071fe0 0x7f4c100b2040 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400c460 tx=0x7f4c0400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:46.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.522+0000 7f4c0cc00640 1 -- 172.21.15.40:0/3625512114 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c04018070 con 0x7f4c10071fe0 2024-04-26T23:06:46.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.522+0000 7f4c0cc00640 1 -- 172.21.15.40:0/3625512114 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c04007500 con 0x7f4c10071fe0 2024-04-26T23:06:46.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.522+0000 7f4c0cc00640 1 -- 172.21.15.40:0/3625512114 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c04004a20 con 0x7f4c10071fe0 2024-04-26T23:06:46.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.522+0000 7f4c16200640 1 -- 172.21.15.40:0/3625512114 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c1015c010 con 0x7f4c10071fe0 2024-04-26T23:06:46.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.522+0000 7f4c16200640 1 -- 172.21.15.40:0/3625512114 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c100c60d0 con 0x7f4c10071fe0 2024-04-26T23:06:46.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.523+0000 7f4c16200640 1 -- 172.21.15.40:0/3625512114 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c10069370 con 0x7f4c10071fe0 2024-04-26T23:06:46.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.524+0000 7f4c0cc00640 1 -- 172.21.15.40:0/3625512114 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4c04004bc0 con 0x7f4c10071fe0 2024-04-26T23:06:46.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.526+0000 7f4c0cc00640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bf006c8f0 0x7f4bf006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:46.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.526+0000 7f4c0cc00640 1 -- 172.21.15.40:0/3625512114 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5507+0+0 (secure 0 0 0) 0x7f4c0408ae10 con 0x7f4c10071fe0 2024-04-26T23:06:46.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.526+0000 7f4c0fe00640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bf006c8f0 0x7f4bf006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:46.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.528+0000 7f4c0fe00640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bf006c8f0 0x7f4bf006edb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc034000 tx=0x7f4bfc02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:46.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.530+0000 7f4c0cc00640 1 -- 172.21.15.40:0/3625512114 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4c0408e030 con 0x7f4c10071fe0 2024-04-26T23:06:46.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.736+0000 7f4c16200640 1 -- 172.21.15.40:0/3625512114 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f4c100d01f0 con 0x7f4c10071fe0 2024-04-26T23:06:46.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.741+0000 7f4c0cc00640 1 -- 172.21.15.40:0/3625512114 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) ==== 104+0+0 (secure 0 0 0) 0x7f4c1006f1f0 con 0x7f4c10071fe0 2024-04-26T23:06:46.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.743+0000 7f4c16200640 1 -- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bf006c8f0 msgr2=0x7f4bf006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:46.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.743+0000 7f4c16200640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bf006c8f0 0x7f4bf006edb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc034000 tx=0x7f4bfc02e000 comp rx=0 tx=0).stop 2024-04-26T23:06:46.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.744+0000 7f4c16200640 1 -- 172.21.15.40:0/3625512114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10071fe0 msgr2=0x7f4c100b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:46.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.744+0000 7f4c16200640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10071fe0 0x7f4c100b2040 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400c460 tx=0x7f4c0400c920 comp rx=0 tx=0).stop 2024-04-26T23:06:46.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.744+0000 7f4c16200640 1 -- 172.21.15.40:0/3625512114 shutdown_connections 2024-04-26T23:06:46.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.744+0000 7f4c16200640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4bf006c8f0 0x7f4bf006edb0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.744+0000 7f4c16200640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c1015ce60 0x7f4c1015b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.744+0000 7f4c16200640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c100af7a0 0x7f4c1014db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.744+0000 7f4c16200640 1 --2- 172.21.15.40:0/3625512114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c10071fe0 0x7f4c100b2040 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.744+0000 7f4c16200640 1 -- 172.21.15.40:0/3625512114 >> 172.21.15.40:0/3625512114 conn(0x7f4c100d0a10 msgr2=0x7f4c100b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:46.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.744+0000 7f4c16200640 1 -- 172.21.15.40:0/3625512114 shutdown_connections 2024-04-26T23:06:46.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.744+0000 7f4c16200640 1 -- 172.21.15.40:0/3625512114 wait complete. 2024-04-26T23:06:46.765 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_backtrace' 2024-04-26T23:06:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.980+0000 7f4a27200640 1 Processor -- start 2024-04-26T23:06:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.980+0000 7f4a27200640 1 -- start start 2024-04-26T23:06:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.982+0000 7f4a27200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4a2815f1e0 0x7f4a280713a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.982+0000 7f4a27200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a2806c680 0x7f4a280affe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.982+0000 7f4a27200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a2806f950 con 0x7f4a2806c240 2024-04-26T23:06:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.982+0000 7f4a27200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f4a280b15a0 con 0x7f4a2806c680 2024-04-26T23:06:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.982+0000 7f4a27200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4a2806fc30 con 0x7f4a2815f1e0 2024-04-26T23:06:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.982+0000 7f4a26800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a2806c680 0x7f4a280affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.982+0000 7f4a26800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a2806c680 0x7f4a280affe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58650/0 (socket says 172.21.15.40:58650) 2024-04-26T23:06:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.982+0000 7f4a26800640 1 -- 172.21.15.40:0/3665817924 learned_addr learned my addr 172.21.15.40:0/3665817924 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.982+0000 7f4a25e00640 1 --2- 172.21.15.40:0/3665817924 >> v2:172.21.15.103:3300/0 conn(0x7f4a2815f1e0 0x7f4a280713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:46.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.983+0000 7f4a26800640 1 -- 172.21.15.40:0/3665817924 >> v2:172.21.15.103:3300/0 conn(0x7f4a2815f1e0 msgr2=0x7f4a280713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:46.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.983+0000 7f4a26800640 1 --2- 172.21.15.40:0/3665817924 >> v2:172.21.15.103:3300/0 conn(0x7f4a2815f1e0 0x7f4a280713a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.983+0000 7f4a26800640 1 -- 172.21.15.40:0/3665817924 >> v1:172.21.15.103:6789/0 conn(0x7f4a2806c240 legacy=0x7f4a280ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:46.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.983+0000 7f4a26800640 1 -- 172.21.15.40:0/3665817924 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2806f040 con 0x7f4a2806c680 2024-04-26T23:06:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.984+0000 7f4a26800640 1 --2- 172.21.15.40:0/3665817924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a2806c680 0x7f4a280affe0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f4a28051040 tx=0x7f4a2002a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58cc5a3b3d3fe962 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.984+0000 7f4a24a00640 1 -- 172.21.15.40:0/3665817924 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a20036070 con 0x7f4a2806c680 2024-04-26T23:06:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.984+0000 7f4a24a00640 1 -- 172.21.15.40:0/3665817924 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a20031600 con 0x7f4a2806c680 2024-04-26T23:06:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.984+0000 7f4a24a00640 1 -- 172.21.15.40:0/3665817924 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a200319d0 con 0x7f4a2806c680 2024-04-26T23:06:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.985+0000 7f4a27200640 1 -- 172.21.15.40:0/3665817924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a2806c680 msgr2=0x7f4a280affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.985+0000 7f4a27200640 1 --2- 172.21.15.40:0/3665817924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a2806c680 0x7f4a280affe0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f4a28051040 tx=0x7f4a2002a780 comp rx=0 tx=0).stop 2024-04-26T23:06:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.985+0000 7f4a27200640 1 -- 172.21.15.40:0/3665817924 shutdown_connections 2024-04-26T23:06:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.985+0000 7f4a27200640 1 --2- 172.21.15.40:0/3665817924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a2806c680 0x7f4a280affe0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.985+0000 7f4a27200640 1 --2- 172.21.15.40:0/3665817924 >> v2:172.21.15.103:3300/0 conn(0x7f4a2815f1e0 0x7f4a280713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.985+0000 7f4a27200640 1 -- 172.21.15.40:0/3665817924 >> 172.21.15.40:0/3665817924 conn(0x7f4a280d1b50 msgr2=0x7f4a280d1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:46.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.985+0000 7f4a27200640 1 -- 172.21.15.40:0/3665817924 shutdown_connections 2024-04-26T23:06:46.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.985+0000 7f4a27200640 1 -- 172.21.15.40:0/3665817924 wait complete. 2024-04-26T23:06:46.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.987+0000 7f4a27200640 1 Processor -- start 2024-04-26T23:06:46.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.987+0000 7f4a27200640 1 -- start start 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.988+0000 7f4a27200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a2806c240 0x7f4a2814e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.988+0000 7f4a27200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a2806c680 0x7f4a2815f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.988+0000 7f4a27200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a2815f1e0 0x7f4a2815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.988+0000 7f4a27200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2806cad0 con 0x7f4a2806c240 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.988+0000 7f4a27200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4a280d29e0 con 0x7f4a2806c680 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.988+0000 7f4a27200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4a280d2670 con 0x7f4a2815f1e0 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.988+0000 7f4a25e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a2806c240 0x7f4a2814e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.988+0000 7f4a25e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a2806c240 0x7f4a2814e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58664/0 (socket says 172.21.15.40:58664) 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.988+0000 7f4a25e00640 1 -- 172.21.15.40:0/1065415269 learned_addr learned my addr 172.21.15.40:0/1065415269 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.989+0000 7f4a25e00640 1 -- 172.21.15.40:0/1065415269 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a2815f1e0 msgr2=0x7f4a2815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.989+0000 7f4a25e00640 1 --2- 172.21.15.40:0/1065415269 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a2815f1e0 0x7f4a2815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.989+0000 7f4a25e00640 1 -- 172.21.15.40:0/1065415269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a2806c680 msgr2=0x7f4a2815f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.989+0000 7f4a25400640 1 --2- 172.21.15.40:0/1065415269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a2806c680 0x7f4a2815f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.989+0000 7f4a25e00640 1 --2- 172.21.15.40:0/1065415269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a2806c680 0x7f4a2815f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.989+0000 7f4a25e00640 1 -- 172.21.15.40:0/1065415269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a28160020 con 0x7f4a2806c240 2024-04-26T23:06:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.989+0000 7f4a25e00640 1 --2- 172.21.15.40:0/1065415269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a2806c240 0x7f4a2814e500 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800d800 tx=0x7f4a1800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.989+0000 7f4a1e000640 1 -- 172.21.15.40:0/1065415269 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1800b6c0 con 0x7f4a2806c240 2024-04-26T23:06:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.989+0000 7f4a1e000640 1 -- 172.21.15.40:0/1065415269 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a18019070 con 0x7f4a2806c240 2024-04-26T23:06:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.989+0000 7f4a27200640 1 -- 172.21.15.40:0/1065415269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a28160300 con 0x7f4a2806c240 2024-04-26T23:06:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.989+0000 7f4a1e000640 1 -- 172.21.15.40:0/1065415269 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1800ba80 con 0x7f4a2806c240 2024-04-26T23:06:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.990+0000 7f4a27200640 1 -- 172.21.15.40:0/1065415269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a281486b0 con 0x7f4a2806c240 2024-04-26T23:06:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.991+0000 7f4a27200640 1 -- 172.21.15.40:0/1065415269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a28069a80 con 0x7f4a2806c240 2024-04-26T23:06:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.992+0000 7f4a1e000640 1 -- 172.21.15.40:0/1065415269 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4a1800bc20 con 0x7f4a2806c240 2024-04-26T23:06:46.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.993+0000 7f4a1e000640 1 --2- 172.21.15.40:0/1065415269 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a0806c8f0 0x7f4a0806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:46.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.993+0000 7f4a1e000640 1 -- 172.21.15.40:0/1065415269 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5507+0+0 (secure 0 0 0) 0x7f4a1808bf40 con 0x7f4a2806c240 2024-04-26T23:06:46.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.993+0000 7f4a25400640 1 --2- 172.21.15.40:0/1065415269 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a0806c8f0 0x7f4a0806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:46.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.995+0000 7f4a25400640 1 --2- 172.21.15.40:0/1065415269 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a0806c8f0 0x7f4a0806edb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4a2815f860 tx=0x7f4a14008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:46.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:46.997+0000 7f4a1e000640 1 -- 172.21.15.40:0/1065415269 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4a1805a420 con 0x7f4a2806c240 2024-04-26T23:06:47.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:47.292+0000 7f4a27200640 1 -- 172.21.15.40:0/1065415269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_backtrace"]} v 0) -- 0x7f4a280cdc40 con 0x7f4a2806c240 2024-04-26T23:06:48.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.278+0000 7f4a1e000640 1 -- 172.21.15.40:0/1065415269 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_backtrace"]}]=0 v303) ==== 142+0+0 (secure 0 0 0) 0x7f4a18059d20 con 0x7f4a2806c240 2024-04-26T23:06:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.282+0000 7f4a27200640 1 -- 172.21.15.40:0/1065415269 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a0806c8f0 msgr2=0x7f4a0806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.282+0000 7f4a27200640 1 --2- 172.21.15.40:0/1065415269 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a0806c8f0 0x7f4a0806edb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4a2815f860 tx=0x7f4a14008040 comp rx=0 tx=0).stop 2024-04-26T23:06:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.282+0000 7f4a27200640 1 -- 172.21.15.40:0/1065415269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a2806c240 msgr2=0x7f4a2814e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.282+0000 7f4a27200640 1 --2- 172.21.15.40:0/1065415269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a2806c240 0x7f4a2814e500 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800d800 tx=0x7f4a1800dcc0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.282+0000 7f4a27200640 1 -- 172.21.15.40:0/1065415269 shutdown_connections 2024-04-26T23:06:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.282+0000 7f4a27200640 1 --2- 172.21.15.40:0/1065415269 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a0806c8f0 0x7f4a0806edb0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.282+0000 7f4a27200640 1 --2- 172.21.15.40:0/1065415269 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a2815f1e0 0x7f4a2815aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.282+0000 7f4a27200640 1 --2- 172.21.15.40:0/1065415269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a2806c680 0x7f4a2815f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.282+0000 7f4a27200640 1 --2- 172.21.15.40:0/1065415269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a2806c240 0x7f4a2814e500 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.282+0000 7f4a27200640 1 -- 172.21.15.40:0/1065415269 >> 172.21.15.40:0/1065415269 conn(0x7f4a280d1b50 msgr2=0x7f4a2815a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.283+0000 7f4a27200640 1 -- 172.21.15.40:0/1065415269 shutdown_connections 2024-04-26T23:06:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.283+0000 7f4a27200640 1 -- 172.21.15.40:0/1065415269 wait complete. 2024-04-26T23:06:48.306 INFO:tasks.cephfs_test_runner:test_scrub_backtrace (tasks.cephfs.test_multimds_misc.TestScrub2) ... ok 2024-04-26T23:06:48.306 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_non_mds0 (tasks.cephfs.test_multimds_misc.TestScrub2) 2024-04-26T23:06:48.306 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:06:48.486 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:06:48.495 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:06:48.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.518+0000 7f33af600640 1 Processor -- start 2024-04-26T23:06:48.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.518+0000 7f33af600640 1 -- start start 2024-04-26T23:06:48.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.519+0000 7f33af600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33a815f540 0x7f33a80d3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.519+0000 7f33af600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f33a80b3ff0 0x7f33a80d39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.519+0000 7f33af600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33a80d1770 0x7f33a80d1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.519+0000 7f33af600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f33a815f920 con 0x7f33a80d1770 2024-04-26T23:06:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.519+0000 7f33af600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f33a80d0f30 con 0x7f33a80b3ff0 2024-04-26T23:06:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.519+0000 7f33af600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33a80d2d70 con 0x7f33a815f540 2024-04-26T23:06:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.519+0000 7f33ad600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33a80d1770 0x7f33a80d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.520+0000 7f33ad600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33a80d1770 0x7f33a80d1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58672/0 (socket says 172.21.15.40:58672) 2024-04-26T23:06:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.520+0000 7f33ad600640 1 -- 172.21.15.40:0/1096810700 learned_addr learned my addr 172.21.15.40:0/1096810700 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.520+0000 7f33acc00640 1 --2- 172.21.15.40:0/1096810700 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33a815f540 0x7f33a80d3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.520+0000 7f339fe00640 1 --2- 172.21.15.40:0/1096810700 >> v2:172.21.15.103:3300/0 conn(0x7f33a80b3ff0 0x7f33a80d39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.520+0000 7f33ad600640 1 -- 172.21.15.40:0/1096810700 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33a815f540 msgr2=0x7f33a80d3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:48.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.520+0000 7f33ad600640 1 --2- 172.21.15.40:0/1096810700 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33a815f540 0x7f33a80d3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.520+0000 7f33ad600640 1 -- 172.21.15.40:0/1096810700 >> v2:172.21.15.103:3300/0 conn(0x7f33a80b3ff0 msgr2=0x7f33a80d39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:48.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.520+0000 7f33ad600640 1 --2- 172.21.15.40:0/1096810700 >> v2:172.21.15.103:3300/0 conn(0x7f33a80b3ff0 0x7f33a80d39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.520+0000 7f33ad600640 1 -- 172.21.15.40:0/1096810700 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a80d3050 con 0x7f33a80d1770 2024-04-26T23:06:48.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.521+0000 7f33ad600640 1 --2- 172.21.15.40:0/1096810700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33a80d1770 0x7f33a80d1c00 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f33a8073c50 tx=0x7f33a002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167c37159018456e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:48.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.522+0000 7f33a7e00640 1 -- 172.21.15.40:0/1096810700 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a0036070 con 0x7f33a80d1770 2024-04-26T23:06:48.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.522+0000 7f33a7e00640 1 -- 172.21.15.40:0/1096810700 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a0031600 con 0x7f33a80d1770 2024-04-26T23:06:48.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.522+0000 7f33a7e00640 1 -- 172.21.15.40:0/1096810700 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a00319d0 con 0x7f33a80d1770 2024-04-26T23:06:48.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.522+0000 7f33af600640 1 -- 172.21.15.40:0/1096810700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33a80d1770 msgr2=0x7f33a80d1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:48.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.522+0000 7f33af600640 1 --2- 172.21.15.40:0/1096810700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33a80d1770 0x7f33a80d1c00 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f33a8073c50 tx=0x7f33a002a6a0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.522+0000 7f33af600640 1 -- 172.21.15.40:0/1096810700 shutdown_connections 2024-04-26T23:06:48.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.522+0000 7f33af600640 1 --2- 172.21.15.40:0/1096810700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33a80d1770 0x7f33a80d1c00 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.522+0000 7f33af600640 1 --2- 172.21.15.40:0/1096810700 >> v2:172.21.15.103:3300/0 conn(0x7f33a80b3ff0 0x7f33a80d39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.522+0000 7f33af600640 1 --2- 172.21.15.40:0/1096810700 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33a815f540 0x7f33a80d3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.522+0000 7f33af600640 1 -- 172.21.15.40:0/1096810700 >> 172.21.15.40:0/1096810700 conn(0x7f33a80d4f90 msgr2=0x7f33a80d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:48.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.522+0000 7f33af600640 1 -- 172.21.15.40:0/1096810700 shutdown_connections 2024-04-26T23:06:48.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.522+0000 7f33af600640 1 -- 172.21.15.40:0/1096810700 wait complete. 2024-04-26T23:06:48.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.524+0000 7f33af600640 1 Processor -- start 2024-04-26T23:06:48.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.524+0000 7f33af600640 1 -- start start 2024-04-26T23:06:48.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.525+0000 7f33af600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33a80b3ff0 0x7f33a81534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.525+0000 7f33af600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33a80d1770 0x7f33a80b7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.525+0000 7f33af600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33a815f540 0x7f33a8153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.525+0000 7f33af600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f33a8068e50 con 0x7f33a80d1770 2024-04-26T23:06:48.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.525+0000 7f33af600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33a80d0bc0 con 0x7f33a815f540 2024-04-26T23:06:48.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.525+0000 7f33af600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33a80d2140 con 0x7f33a80b3ff0 2024-04-26T23:06:48.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.525+0000 7f339fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33a80d1770 0x7f33a80b7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.526+0000 7f339fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33a80d1770 0x7f33a80b7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58680/0 (socket says 172.21.15.40:58680) 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.526+0000 7f339fe00640 1 -- 172.21.15.40:0/3826784560 learned_addr learned my addr 172.21.15.40:0/3826784560 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.526+0000 7f33ad600640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33a815f540 0x7f33a8153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.526+0000 7f33acc00640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33a80b3ff0 0x7f33a81534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.526+0000 7f339fe00640 1 -- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33a80b3ff0 msgr2=0x7f33a81534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.526+0000 7f339fe00640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33a80b3ff0 0x7f33a81534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.526+0000 7f339fe00640 1 -- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33a815f540 msgr2=0x7f33a8153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.526+0000 7f339fe00640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33a815f540 0x7f33a8153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.526+0000 7f339fe00640 1 -- 172.21.15.40:0/3826784560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a80b80f0 con 0x7f33a80d1770 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.526+0000 7f33acc00640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33a80b3ff0 0x7f33a81534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.526+0000 7f339fe00640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33a80d1770 0x7f33a80b7bb0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f339000e890 tx=0x7f339000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.527+0000 7f33a5600640 1 -- 172.21.15.40:0/3826784560 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3390017070 con 0x7f33a80d1770 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.527+0000 7f33a5600640 1 -- 172.21.15.40:0/3826784560 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f339000c940 con 0x7f33a80d1770 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.527+0000 7f33a5600640 1 -- 172.21.15.40:0/3826784560 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339000cd10 con 0x7f33a80d1770 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.527+0000 7f33af600640 1 -- 172.21.15.40:0/3826784560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a80c9df0 con 0x7f33a80d1770 2024-04-26T23:06:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.527+0000 7f33af600640 1 -- 172.21.15.40:0/3826784560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a80ca3b0 con 0x7f33a80d1770 2024-04-26T23:06:48.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.528+0000 7f33af600640 1 -- 172.21.15.40:0/3826784560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a80694f0 con 0x7f33a80d1770 2024-04-26T23:06:48.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.529+0000 7f33a5600640 1 -- 172.21.15.40:0/3826784560 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f339001a070 con 0x7f33a80d1770 2024-04-26T23:06:48.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.530+0000 7f33a5600640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f338406c8f0 0x7f338406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.530+0000 7f33a5600640 1 -- 172.21.15.40:0/3826784560 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5507+0+0 (secure 0 0 0) 0x7f339008bd20 con 0x7f33a80d1770 2024-04-26T23:06:48.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.530+0000 7f33acc00640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f338406c8f0 0x7f338406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.531+0000 7f33acc00640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f338406c8f0 0x7f338406edb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3398009af0 tx=0x7f3398008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.534+0000 7f33a5600640 1 -- 172.21.15.40:0/3826784560 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3390059bc0 con 0x7f33a80d1770 2024-04-26T23:06:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.756+0000 7f33af600640 1 -- 172.21.15.40:0/3826784560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f33a80b55d0 con 0x7f33a80d1770 2024-04-26T23:06:48.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.757+0000 7f33a5600640 1 -- 172.21.15.40:0/3826784560 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+13518 (secure 0 0 0) 0x7f339005dec0 con 0x7f33a80d1770 2024-04-26T23:06:48.758 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:48.758 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":70,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:06:46.263328+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6837/2595375825":"2024-04-27T23:06:43.243211+0000","172.21.15.40:6835/931483025":"2024-04-27T23:06:43.243232+0000","172.21.15.103:6835/2595375825":"2024-04-27T23:06:43.243211+0000","172.21.15.40:6833/931483025":"2024-04-27T23:06:43.243232+0000","172.21.15.40:6834/2117861921":"2024-04-27T23:06:43.243226+0000","172.21.15.40:6832/2117861921":"2024-04-27T23:06:43.243226+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:06:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.760+0000 7f33af600640 1 -- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f338406c8f0 msgr2=0x7f338406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.760+0000 7f33af600640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f338406c8f0 0x7f338406edb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3398009af0 tx=0x7f3398008040 comp rx=0 tx=0).stop 2024-04-26T23:06:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.760+0000 7f33af600640 1 -- 172.21.15.40:0/3826784560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33a80d1770 msgr2=0x7f33a80b7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.760+0000 7f33af600640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33a80d1770 0x7f33a80b7bb0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f339000e890 tx=0x7f339000ed50 comp rx=0 tx=0).stop 2024-04-26T23:06:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.761+0000 7f33af600640 1 -- 172.21.15.40:0/3826784560 shutdown_connections 2024-04-26T23:06:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.761+0000 7f33af600640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f338406c8f0 0x7f338406edb0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.761+0000 7f33af600640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33a815f540 0x7f33a8153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.761+0000 7f33af600640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33a80d1770 0x7f33a80b7bb0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.761+0000 7f33af600640 1 --2- 172.21.15.40:0/3826784560 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f33a80b3ff0 0x7f33a81534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.761+0000 7f33af600640 1 -- 172.21.15.40:0/3826784560 >> 172.21.15.40:0/3826784560 conn(0x7f33a80d4f90 msgr2=0x7f33a80d56b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:48.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.761+0000 7f33af600640 1 -- 172.21.15.40:0/3826784560 shutdown_connections 2024-04-26T23:06:48.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.761+0000 7f33af600640 1 -- 172.21.15.40:0/3826784560 wait complete. 2024-04-26T23:06:48.779 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:06:48.780 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:06:48.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.986+0000 7f16b3e00640 1 Processor -- start 2024-04-26T23:06:48.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.986+0000 7f16b3e00640 1 -- start start 2024-04-26T23:06:48.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.988+0000 7f16b3e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f16ac1678f0 0x7f16ac0cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.988+0000 7f16b3e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16ac163a50 con 0x7f16ac0c41d0 2024-04-26T23:06:48.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.988+0000 7f16b3e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16ac0ceb90 con 0x7f16ac073f30 2024-04-26T23:06:48.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.988+0000 7f16b3e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f16ac0cf7a0 con 0x7f16ac1678f0 2024-04-26T23:06:48.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.988+0000 7f16b1400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f16ac0c41d0 0x7f16ac0cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35408/0 (socket says 172.21.15.40:35408) 2024-04-26T23:06:48.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.988+0000 7f16b1400640 1 -- 172.21.15.40:0/2994520684 learned_addr learned my addr 172.21.15.40:0/2994520684 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:48.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16b1e00640 1 --2- 172.21.15.40:0/2994520684 >> v2:172.21.15.103:3300/0 conn(0x7f16ac1678f0 0x7f16ac0cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3494044866 0 0) 0x7f16ac163a50 con 0x7f16ac0c41d0 2024-04-26T23:06:48.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1694003520 con 0x7f16ac0c41d0 2024-04-26T23:06:48.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1510890988 0 0) 0x7f1694003520 con 0x7f16ac0c41d0 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 >> v2:172.21.15.103:3300/0 conn(0x7f16ac1678f0 msgr2=0x7f16ac0cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 --2- 172.21.15.40:0/2994520684 >> v2:172.21.15.103:3300/0 conn(0x7f16ac1678f0 0x7f16ac0cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 >> v1:172.21.15.103:6789/0 conn(0x7f16ac073f30 legacy=0x7f16ac1671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16ac0cee70 con 0x7f16ac0c41d0 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f16a0001670 con 0x7f16ac0c41d0 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 >> v1:172.21.15.40:6789/0 conn(0x7f16ac0c41d0 legacy=0x7f16ac0cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 --2- 172.21.15.40:0/2994520684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16940039b0 0x7f1694003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 --2- 172.21.15.40:0/2994520684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1694004800 0x7f1694004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 --2- 172.21.15.40:0/2994520684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16940051c0 0x7f169400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f16ac0cf7a0 con 0x7f16940039b0 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f169400c060 con 0x7f1694004800 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.989+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f169400c260 con 0x7f16940051c0 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.990+0000 7f16b1400640 1 --2- 172.21.15.40:0/2994520684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16940039b0 0x7f1694003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.990+0000 7f16b1e00640 1 --2- 172.21.15.40:0/2994520684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16940051c0 0x7f169400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.990+0000 7f16b1400640 1 -- 172.21.15.40:0/2994520684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16940051c0 msgr2=0x7f169400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:48.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.990+0000 7f16b0a00640 1 --2- 172.21.15.40:0/2994520684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1694004800 0x7f1694004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.990+0000 7f16b1400640 1 --2- 172.21.15.40:0/2994520684 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16940051c0 0x7f169400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.990+0000 7f16b1400640 1 -- 172.21.15.40:0/2994520684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1694004800 msgr2=0x7f1694004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:48.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.990+0000 7f16b1400640 1 --2- 172.21.15.40:0/2994520684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1694004800 0x7f1694004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.990+0000 7f16b1e00640 1 -- 172.21.15.40:0/2994520684 reap_dead start 2024-04-26T23:06:48.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.990+0000 7f16b1400640 1 -- 172.21.15.40:0/2994520684 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1694003520 con 0x7f16940039b0 2024-04-26T23:06:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.991+0000 7f16b1400640 1 --2- 172.21.15.40:0/2994520684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16940039b0 0x7f1694003dd0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f16a000c240 tx=0x7f1694005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32a81a24aa30991 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.991+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0007cf0 con 0x7f16940039b0 2024-04-26T23:06:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.992+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16a0005af0 con 0x7f16940039b0 2024-04-26T23:06:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.992+0000 7f16abe00640 1 -- 172.21.15.40:0/2994520684 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0007070 con 0x7f16940039b0 2024-04-26T23:06:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.992+0000 7f16b3e00640 1 -- 172.21.15.40:0/2994520684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16940039b0 msgr2=0x7f1694003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.992+0000 7f16b3e00640 1 --2- 172.21.15.40:0/2994520684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16940039b0 0x7f1694003dd0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f16a000c240 tx=0x7f1694005830 comp rx=0 tx=0).stop 2024-04-26T23:06:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.992+0000 7f16b3e00640 1 -- 172.21.15.40:0/2994520684 shutdown_connections 2024-04-26T23:06:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.992+0000 7f16b3e00640 1 --2- 172.21.15.40:0/2994520684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16940039b0 0x7f1694003dd0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.992+0000 7f16b3e00640 1 -- 172.21.15.40:0/2994520684 >> 172.21.15.40:0/2994520684 conn(0x7f16ac0d0be0 msgr2=0x7f16ac0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.992+0000 7f16b3e00640 1 -- 172.21.15.40:0/2994520684 shutdown_connections 2024-04-26T23:06:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.992+0000 7f16b3e00640 1 -- 172.21.15.40:0/2994520684 wait complete. 2024-04-26T23:06:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.994+0000 7f16b3e00640 1 Processor -- start 2024-04-26T23:06:48.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.994+0000 7f16b3e00640 1 -- start start 2024-04-26T23:06:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.995+0000 7f16b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16940039b0 0x7f16ac15ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.995+0000 7f16b3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16ac1f9db0 0x7f16ac0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.995+0000 7f16b3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16ac1ec9e0 0x7f16ac1f90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.995+0000 7f16b3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f16ac0ce7b0 con 0x7f16940039b0 2024-04-26T23:06:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.995+0000 7f16b3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f16ac0cf430 con 0x7f16ac1ec9e0 2024-04-26T23:06:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.995+0000 7f16b3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f16ac0ce930 con 0x7f16ac1f9db0 2024-04-26T23:06:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.996+0000 7f16b1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16940039b0 0x7f16ac15ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.996+0000 7f16b1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16940039b0 0x7f16ac15ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58694/0 (socket says 172.21.15.40:58694) 2024-04-26T23:06:48.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.996+0000 7f16b1400640 1 -- 172.21.15.40:0/1266967374 learned_addr learned my addr 172.21.15.40:0/1266967374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.996+0000 7f16b1e00640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16ac1ec9e0 0x7f16ac1f90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.996+0000 7f16b0a00640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16ac1f9db0 0x7f16ac0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.996+0000 7f16b1400640 1 -- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16ac1f9db0 msgr2=0x7f16ac0b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:48.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.996+0000 7f16b1400640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16ac1f9db0 0x7f16ac0b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.996+0000 7f16b1400640 1 -- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16ac1ec9e0 msgr2=0x7f16ac1f90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.996+0000 7f16b1400640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16ac1ec9e0 0x7f16ac1f90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.996+0000 7f16b1400640 1 -- 172.21.15.40:0/1266967374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16ac0b5210 con 0x7f16940039b0 2024-04-26T23:06:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.997+0000 7f16b1400640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16940039b0 0x7f16ac15ccf0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f16a000c240 tx=0x7f16ac0cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.997+0000 7f16a9600640 1 -- 172.21.15.40:0/1266967374 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0037040 con 0x7f16940039b0 2024-04-26T23:06:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.997+0000 7f16a9600640 1 -- 172.21.15.40:0/1266967374 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16a0041070 con 0x7f16940039b0 2024-04-26T23:06:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.997+0000 7f16b3e00640 1 -- 172.21.15.40:0/1266967374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16ac0b4a80 con 0x7f16940039b0 2024-04-26T23:06:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.997+0000 7f16a9600640 1 -- 172.21.15.40:0/1266967374 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0001670 con 0x7f16940039b0 2024-04-26T23:06:48.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.997+0000 7f16b3e00640 1 -- 172.21.15.40:0/1266967374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16ac0b57b0 con 0x7f16940039b0 2024-04-26T23:06:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.998+0000 7f16b3e00640 1 -- 172.21.15.40:0/1266967374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16ac0694f0 con 0x7f16940039b0 2024-04-26T23:06:48.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:48.999+0000 7f16a9600640 1 -- 172.21.15.40:0/1266967374 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f16a00049e0 con 0x7f16940039b0 2024-04-26T23:06:49.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.001+0000 7f16a9600640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f169006c920 0x7f169006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:49.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.001+0000 7f16a9600640 1 -- 172.21.15.40:0/1266967374 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5507+0+0 (secure 0 0 0) 0x7f16a00ab7a0 con 0x7f16940039b0 2024-04-26T23:06:49.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.002+0000 7f16b0a00640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f169006c920 0x7f169006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:49.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.003+0000 7f16b0a00640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f169006c920 0x7f169006ede0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0c6f90 tx=0x7f169c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:49.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.005+0000 7f16a9600640 1 -- 172.21.15.40:0/1266967374 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f16a0079c80 con 0x7f16940039b0 2024-04-26T23:06:49.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.210+0000 7f16b3e00640 1 -- 172.21.15.40:0/1266967374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f16ac0d0430 con 0x7f16940039b0 2024-04-26T23:06:49.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.211+0000 7f16a9600640 1 -- 172.21.15.40:0/1266967374 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) ==== 93+0+10 (secure 0 0 0) 0x7f16a0079580 con 0x7f16940039b0 2024-04-26T23:06:49.211 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:06:49.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.214+0000 7f16b3e00640 1 -- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f169006c920 msgr2=0x7f169006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:49.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.214+0000 7f16b3e00640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f169006c920 0x7f169006ede0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0c6f90 tx=0x7f169c009040 comp rx=0 tx=0).stop 2024-04-26T23:06:49.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.214+0000 7f16b3e00640 1 -- 172.21.15.40:0/1266967374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16940039b0 msgr2=0x7f16ac15ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:49.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.214+0000 7f16b3e00640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16940039b0 0x7f16ac15ccf0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f16a000c240 tx=0x7f16ac0cf040 comp rx=0 tx=0).stop 2024-04-26T23:06:49.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.214+0000 7f16b3e00640 1 -- 172.21.15.40:0/1266967374 shutdown_connections 2024-04-26T23:06:49.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.214+0000 7f16b3e00640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f169006c920 0x7f169006ede0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:49.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.214+0000 7f16b3e00640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16ac1ec9e0 0x7f16ac1f90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:49.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.214+0000 7f16b3e00640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16ac1f9db0 0x7f16ac0b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:49.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.214+0000 7f16b3e00640 1 --2- 172.21.15.40:0/1266967374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16940039b0 0x7f16ac15ccf0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:49.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.214+0000 7f16b3e00640 1 -- 172.21.15.40:0/1266967374 >> 172.21.15.40:0/1266967374 conn(0x7f16ac0d0be0 msgr2=0x7f16ac06e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:49.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.214+0000 7f16b3e00640 1 -- 172.21.15.40:0/1266967374 shutdown_connections 2024-04-26T23:06:49.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.214+0000 7f16b3e00640 1 -- 172.21.15.40:0/1266967374 wait complete. 2024-04-26T23:06:49.231 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_non_mds0' 2024-04-26T23:06:49.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.429+0000 7fb336e00640 1 Processor -- start 2024-04-26T23:06:49.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.429+0000 7fb336e00640 1 -- start start 2024-04-26T23:06:49.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.431+0000 7fb336e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb330070750 0x7fb3300b56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.431+0000 7fb336e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3300af6e0 0x7fb3300afb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.431+0000 7fb336e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3300b5e30 con 0x7fb330072fb0 2024-04-26T23:06:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.431+0000 7fb336e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb3300b5c00 con 0x7fb3300af6e0 2024-04-26T23:06:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.431+0000 7fb336e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb33006f2b0 con 0x7fb330070750 2024-04-26T23:06:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.431+0000 7fb334e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3300af6e0 0x7fb3300afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.432+0000 7fb334e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3300af6e0 0x7fb3300afb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58710/0 (socket says 172.21.15.40:58710) 2024-04-26T23:06:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.432+0000 7fb334e00640 1 -- 172.21.15.40:0/1916817618 learned_addr learned my addr 172.21.15.40:0/1916817618 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.432+0000 7fb327e00640 1 --2- 172.21.15.40:0/1916817618 >> v2:172.21.15.103:3300/0 conn(0x7fb330070750 0x7fb3300b56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.432+0000 7fb334e00640 1 -- 172.21.15.40:0/1916817618 >> v2:172.21.15.103:3300/0 conn(0x7fb330070750 msgr2=0x7fb3300b56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.432+0000 7fb334e00640 1 --2- 172.21.15.40:0/1916817618 >> v2:172.21.15.103:3300/0 conn(0x7fb330070750 0x7fb3300b56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.432+0000 7fb334e00640 1 -- 172.21.15.40:0/1916817618 >> v1:172.21.15.103:6789/0 conn(0x7fb330072fb0 legacy=0x7fb330073390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.432+0000 7fb334e00640 1 -- 172.21.15.40:0/1916817618 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3300b6110 con 0x7fb3300af6e0 2024-04-26T23:06:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.433+0000 7fb334e00640 1 --2- 172.21.15.40:0/1916817618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3300af6e0 0x7fb3300afb70 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fb3300a00e0 tx=0x7fb32802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8915bae6beee5249 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.433+0000 7fb32f400640 1 -- 172.21.15.40:0/1916817618 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328036070 con 0x7fb3300af6e0 2024-04-26T23:06:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.434+0000 7fb32f400640 1 -- 172.21.15.40:0/1916817618 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb328031600 con 0x7fb3300af6e0 2024-04-26T23:06:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.434+0000 7fb32f400640 1 -- 172.21.15.40:0/1916817618 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3280319d0 con 0x7fb3300af6e0 2024-04-26T23:06:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.434+0000 7fb336e00640 1 -- 172.21.15.40:0/1916817618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3300af6e0 msgr2=0x7fb3300afb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.434+0000 7fb336e00640 1 --2- 172.21.15.40:0/1916817618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3300af6e0 0x7fb3300afb70 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fb3300a00e0 tx=0x7fb32802a6f0 comp rx=0 tx=0).stop 2024-04-26T23:06:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.434+0000 7fb336e00640 1 -- 172.21.15.40:0/1916817618 shutdown_connections 2024-04-26T23:06:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.434+0000 7fb336e00640 1 --2- 172.21.15.40:0/1916817618 >> v2:172.21.15.103:3300/0 conn(0x7fb330070750 0x7fb3300b56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.434+0000 7fb336e00640 1 --2- 172.21.15.40:0/1916817618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3300af6e0 0x7fb3300afb70 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.434+0000 7fb336e00640 1 -- 172.21.15.40:0/1916817618 >> 172.21.15.40:0/1916817618 conn(0x7fb3300b39b0 msgr2=0x7fb3300b3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.434+0000 7fb336e00640 1 -- 172.21.15.40:0/1916817618 shutdown_connections 2024-04-26T23:06:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.434+0000 7fb336e00640 1 -- 172.21.15.40:0/1916817618 wait complete. 2024-04-26T23:06:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.436+0000 7fb336e00640 1 Processor -- start 2024-04-26T23:06:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.436+0000 7fb336e00640 1 -- start start 2024-04-26T23:06:49.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.437+0000 7fb336e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb330070750 0x7fb330158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:49.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.437+0000 7fb336e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb330072fb0 0x7fb33015a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:49.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.437+0000 7fb336e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3300af6e0 0x7fb33015a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:49.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.437+0000 7fb336e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb33015c2e0 con 0x7fb330072fb0 2024-04-26T23:06:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.437+0000 7fb336e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb33015c160 con 0x7fb330070750 2024-04-26T23:06:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.437+0000 7fb336e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb3300b00b0 con 0x7fb3300af6e0 2024-04-26T23:06:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.438+0000 7fb327e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb330072fb0 0x7fb33015a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.438+0000 7fb327e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb330072fb0 0x7fb33015a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58722/0 (socket says 172.21.15.40:58722) 2024-04-26T23:06:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.438+0000 7fb327e00640 1 -- 172.21.15.40:0/1082173479 learned_addr learned my addr 172.21.15.40:0/1082173479 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.438+0000 7fb32fe00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb330070750 0x7fb330158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.438+0000 7fb334e00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3300af6e0 0x7fb33015a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.438+0000 7fb327e00640 1 -- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3300af6e0 msgr2=0x7fb33015a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.438+0000 7fb327e00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3300af6e0 0x7fb33015a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.438+0000 7fb327e00640 1 -- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb330070750 msgr2=0x7fb330158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.438+0000 7fb327e00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb330070750 0x7fb330158220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.438+0000 7fb327e00640 1 -- 172.21.15.40:0/1082173479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3300c7c10 con 0x7fb330072fb0 2024-04-26T23:06:49.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.439+0000 7fb327e00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb330072fb0 0x7fb33015a400 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fb31800d800 tx=0x7fb31800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:49.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.439+0000 7fb32cc00640 1 -- 172.21.15.40:0/1082173479 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31800b6c0 con 0x7fb330072fb0 2024-04-26T23:06:49.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.439+0000 7fb32cc00640 1 -- 172.21.15.40:0/1082173479 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb318019070 con 0x7fb330072fb0 2024-04-26T23:06:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.439+0000 7fb32cc00640 1 -- 172.21.15.40:0/1082173479 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31800ba80 con 0x7fb330072fb0 2024-04-26T23:06:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.439+0000 7fb336e00640 1 -- 172.21.15.40:0/1082173479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3300c7ef0 con 0x7fb330072fb0 2024-04-26T23:06:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.439+0000 7fb336e00640 1 -- 172.21.15.40:0/1082173479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3300c8790 con 0x7fb330072fb0 2024-04-26T23:06:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.440+0000 7fb336e00640 1 -- 172.21.15.40:0/1082173479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3300c80c0 con 0x7fb330072fb0 2024-04-26T23:06:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.441+0000 7fb32cc00640 1 -- 172.21.15.40:0/1082173479 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb31800bc20 con 0x7fb330072fb0 2024-04-26T23:06:49.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.442+0000 7fb32cc00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb30c06c8f0 0x7fb30c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:49.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.442+0000 7fb32cc00640 1 -- 172.21.15.40:0/1082173479 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5507+0+0 (secure 0 0 0) 0x7fb31808bf90 con 0x7fb330072fb0 2024-04-26T23:06:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.443+0000 7fb32fe00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb30c06c8f0 0x7fb30c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:49.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.444+0000 7fb32fe00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb30c06c8f0 0x7fb30c06edb0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb320004ea0 tx=0x7fb320009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:49.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.447+0000 7fb32cc00640 1 -- 172.21.15.40:0/1082173479 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fb31801c360 con 0x7fb330072fb0 2024-04-26T23:06:49.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:49.714+0000 7fb336e00640 1 -- 172.21.15.40:0/1082173479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_non_mds0"]} v 0) -- 0x7fb3300694f0 con 0x7fb330072fb0 2024-04-26T23:06:50.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.291+0000 7fb32cc00640 1 -- 172.21.15.40:0/1082173479 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_non_mds0"]}]=0 v305) ==== 144+0+0 (secure 0 0 0) 0x7fb31805a470 con 0x7fb330072fb0 2024-04-26T23:06:50.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.294+0000 7fb336e00640 1 -- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb30c06c8f0 msgr2=0x7fb30c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:50.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.294+0000 7fb336e00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb30c06c8f0 0x7fb30c06edb0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb320004ea0 tx=0x7fb320009040 comp rx=0 tx=0).stop 2024-04-26T23:06:50.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.294+0000 7fb336e00640 1 -- 172.21.15.40:0/1082173479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb330072fb0 msgr2=0x7fb33015a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:50.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.294+0000 7fb336e00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb330072fb0 0x7fb33015a400 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fb31800d800 tx=0x7fb31800dcc0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.294+0000 7fb336e00640 1 -- 172.21.15.40:0/1082173479 shutdown_connections 2024-04-26T23:06:50.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.294+0000 7fb336e00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb30c06c8f0 0x7fb30c06edb0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.294+0000 7fb336e00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3300af6e0 0x7fb33015a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.294+0000 7fb336e00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb330072fb0 0x7fb33015a400 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.294+0000 7fb336e00640 1 --2- 172.21.15.40:0/1082173479 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb330070750 0x7fb330158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.294+0000 7fb336e00640 1 -- 172.21.15.40:0/1082173479 >> 172.21.15.40:0/1082173479 conn(0x7fb3300b39b0 msgr2=0x7fb3300b0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:50.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.295+0000 7fb336e00640 1 -- 172.21.15.40:0/1082173479 shutdown_connections 2024-04-26T23:06:50.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.295+0000 7fb336e00640 1 -- 172.21.15.40:0/1082173479 wait complete. 2024-04-26T23:06:50.316 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-26T23:06:50.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.526+0000 7f1267200640 1 Processor -- start 2024-04-26T23:06:50.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.527+0000 7f1267200640 1 -- start start 2024-04-26T23:06:50.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.528+0000 7f1267200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1268160a50 0x7f1268073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:50.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.528+0000 7f1267200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12680cfaf0 0x7f1268160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:50.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.528+0000 7f1267200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1268161880 con 0x7f126806dd40 2024-04-26T23:06:50.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.529+0000 7f1267200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1268071ce0 con 0x7f1268160a50 2024-04-26T23:06:50.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.529+0000 7f1267200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f12680722a0 con 0x7f12680cfaf0 2024-04-26T23:06:50.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.529+0000 7f1265e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1268160a50 0x7f1268073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:50.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.529+0000 7f1265e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1268160a50 0x7f1268073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58726/0 (socket says 172.21.15.40:58726) 2024-04-26T23:06:50.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.529+0000 7f1265e00640 1 -- 172.21.15.40:0/3909337847 learned_addr learned my addr 172.21.15.40:0/3909337847 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:50.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.529+0000 7f1265e00640 1 -- 172.21.15.40:0/3909337847 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12680cfaf0 msgr2=0x7f1268160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:50.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.529+0000 7f1265e00640 1 --2- 172.21.15.40:0/3909337847 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12680cfaf0 0x7f1268160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.529+0000 7f1265e00640 1 -- 172.21.15.40:0/3909337847 >> v1:172.21.15.103:6789/0 conn(0x7f126806dd40 legacy=0x7f1268072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:50.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.529+0000 7f1265e00640 1 -- 172.21.15.40:0/3909337847 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1268161370 con 0x7f1268160a50 2024-04-26T23:06:50.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.530+0000 7f1265e00640 1 --2- 172.21.15.40:0/3909337847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1268160a50 0x7f1268073a50 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f1258009870 tx=0x7f125802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846c8a34521ae268 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:50.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.531+0000 7f1264a00640 1 -- 172.21.15.40:0/3909337847 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258037070 con 0x7f1268160a50 2024-04-26T23:06:50.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.531+0000 7f1264a00640 1 -- 172.21.15.40:0/3909337847 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1258032440 con 0x7f1268160a50 2024-04-26T23:06:50.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.531+0000 7f1264a00640 1 -- 172.21.15.40:0/3909337847 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258032810 con 0x7f1268160a50 2024-04-26T23:06:50.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.531+0000 7f1267200640 1 -- 172.21.15.40:0/3909337847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1268160a50 msgr2=0x7f1268073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:50.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.531+0000 7f1267200640 1 --2- 172.21.15.40:0/3909337847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1268160a50 0x7f1268073a50 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f1258009870 tx=0x7f125802a210 comp rx=0 tx=0).stop 2024-04-26T23:06:50.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.531+0000 7f1267200640 1 -- 172.21.15.40:0/3909337847 shutdown_connections 2024-04-26T23:06:50.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.531+0000 7f1267200640 1 --2- 172.21.15.40:0/3909337847 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12680cfaf0 0x7f1268160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.531+0000 7f1267200640 1 --2- 172.21.15.40:0/3909337847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1268160a50 0x7f1268073a50 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.531+0000 7f1267200640 1 -- 172.21.15.40:0/3909337847 >> 172.21.15.40:0/3909337847 conn(0x7f12680b2720 msgr2=0x7f12680b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:50.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.531+0000 7f1267200640 1 -- 172.21.15.40:0/3909337847 shutdown_connections 2024-04-26T23:06:50.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.531+0000 7f1267200640 1 -- 172.21.15.40:0/3909337847 wait complete. 2024-04-26T23:06:50.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.533+0000 7f1267200640 1 Processor -- start 2024-04-26T23:06:50.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.533+0000 7f1267200640 1 -- start start 2024-04-26T23:06:50.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.534+0000 7f1267200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f126806dd40 0x7f1268139970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:50.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.534+0000 7f1267200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1268160a50 0x7f126815c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:50.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.534+0000 7f1267200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1268155870 0x7f1268155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:50.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.534+0000 7f1267200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f12680d0260 con 0x7f1268155870 2024-04-26T23:06:50.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.534+0000 7f1267200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1268073f90 con 0x7f1268160a50 2024-04-26T23:06:50.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.534+0000 7f1267200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1268071f30 con 0x7f126806dd40 2024-04-26T23:06:50.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.534+0000 7f1266800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1268155870 0x7f1268155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:50.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.535+0000 7f1266800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1268155870 0x7f1268155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58740/0 (socket says 172.21.15.40:58740) 2024-04-26T23:06:50.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.535+0000 7f1266800640 1 -- 172.21.15.40:0/825511278 learned_addr learned my addr 172.21.15.40:0/825511278 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:50.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.535+0000 7f1265e00640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f126806dd40 0x7f1268139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:50.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.535+0000 7f1266800640 1 -- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f126806dd40 msgr2=0x7f1268139970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:50.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.535+0000 7f1265400640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1268160a50 0x7f126815c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:50.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.535+0000 7f1266800640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f126806dd40 0x7f1268139970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.535+0000 7f1266800640 1 -- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1268160a50 msgr2=0x7f126815c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:50.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.535+0000 7f1266800640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1268160a50 0x7f126815c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.535+0000 7f1266800640 1 -- 172.21.15.40:0/825511278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1268156230 con 0x7f1268155870 2024-04-26T23:06:50.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.535+0000 7f1266800640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1268155870 0x7f1268155cf0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f126000c780 tx=0x7f126000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:50.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.536+0000 7f125e000640 1 -- 172.21.15.40:0/825511278 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1260017070 con 0x7f1268155870 2024-04-26T23:06:50.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.536+0000 7f125e000640 1 -- 172.21.15.40:0/825511278 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1260003da0 con 0x7f1268155870 2024-04-26T23:06:50.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.536+0000 7f125e000640 1 -- 172.21.15.40:0/825511278 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12600052b0 con 0x7f1268155870 2024-04-26T23:06:50.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.536+0000 7f1267200640 1 -- 172.21.15.40:0/825511278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1268137970 con 0x7f1268155870 2024-04-26T23:06:50.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.536+0000 7f1267200640 1 -- 172.21.15.40:0/825511278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1268137f10 con 0x7f1268155870 2024-04-26T23:06:50.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.537+0000 7f1267200640 1 -- 172.21.15.40:0/825511278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1268069370 con 0x7f1268155870 2024-04-26T23:06:50.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.538+0000 7f125e000640 1 -- 172.21.15.40:0/825511278 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f126001a070 con 0x7f1268155870 2024-04-26T23:06:50.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.539+0000 7f125e000640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f124806c8f0 0x7f124806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:50.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.539+0000 7f125e000640 1 -- 172.21.15.40:0/825511278 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5507+0+0 (secure 0 0 0) 0x7f126008b0d0 con 0x7f1268155870 2024-04-26T23:06:50.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.540+0000 7f1265e00640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f124806c8f0 0x7f124806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:50.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.541+0000 7f1265e00640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f124806c8f0 0x7f124806edb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1258034000 tx=0x7f125802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:50.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.544+0000 7f125e000640 1 -- 172.21.15.40:0/825511278 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f12600595b0 con 0x7f1268155870 2024-04-26T23:06:50.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.746+0000 7f1267200640 1 -- 172.21.15.40:0/825511278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f126806fa60 con 0x7f1268155870 2024-04-26T23:06:50.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.753+0000 7f125e000640 1 -- 172.21.15.40:0/825511278 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f1260058eb0 con 0x7f1268155870 2024-04-26T23:06:50.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.756+0000 7f1267200640 1 -- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f124806c8f0 msgr2=0x7f124806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:50.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.756+0000 7f1267200640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f124806c8f0 0x7f124806edb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1258034000 tx=0x7f125802e000 comp rx=0 tx=0).stop 2024-04-26T23:06:50.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.756+0000 7f1267200640 1 -- 172.21.15.40:0/825511278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1268155870 msgr2=0x7f1268155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:50.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.756+0000 7f1267200640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1268155870 0x7f1268155cf0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f126000c780 tx=0x7f126000cc40 comp rx=0 tx=0).stop 2024-04-26T23:06:50.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.756+0000 7f1267200640 1 -- 172.21.15.40:0/825511278 shutdown_connections 2024-04-26T23:06:50.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.756+0000 7f1267200640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f124806c8f0 0x7f124806edb0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.756+0000 7f1267200640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1268155870 0x7f1268155cf0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.756+0000 7f1267200640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1268160a50 0x7f126815c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.756+0000 7f1267200640 1 --2- 172.21.15.40:0/825511278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f126806dd40 0x7f1268139970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:50.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.756+0000 7f1267200640 1 -- 172.21.15.40:0/825511278 >> 172.21.15.40:0/825511278 conn(0x7f12680b2720 msgr2=0x7f126806f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:50.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.756+0000 7f1267200640 1 -- 172.21.15.40:0/825511278 shutdown_connections 2024-04-26T23:06:50.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:50.756+0000 7f1267200640 1 -- 172.21.15.40:0/825511278 wait complete. 2024-04-26T23:06:50.776 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-26T23:06:50.776 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:06:50.895 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:06:51.004 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-26T23:06:51.004 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:06:51.004 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:06:51.064 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:06:51.065 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:06:51.065 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:06:51.121 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:06:51.122 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:06:51.122 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:06:51.182 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:06:51.182 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:06:51.182 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:06:51.238 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:06:51.238 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-26T23:06:51.238 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:06:51.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.461+0000 7f769b400640 1 Processor -- start 2024-04-26T23:06:51.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.461+0000 7f769b400640 1 -- start start 2024-04-26T23:06:51.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.463+0000 7f769b400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7694071e60 0x7f769406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:51.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.463+0000 7f769b400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f769415d7c0 con 0x7f769406fe20 2024-04-26T23:06:51.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.463+0000 7f769b400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f769415daa0 con 0x7f76940cc480 2024-04-26T23:06:51.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.463+0000 7f769b400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f769415dd80 con 0x7f7694071e60 2024-04-26T23:06:51.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.463+0000 7f7693e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f769406fe20 0x7f7694072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35418/0 (socket says 172.21.15.40:35418) 2024-04-26T23:06:51.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.463+0000 7f7693e00640 1 -- 172.21.15.40:0/3467014402 learned_addr learned my addr 172.21.15.40:0/3467014402 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:51.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.464+0000 7f7698a00640 1 --2- 172.21.15.40:0/3467014402 >> v2:172.21.15.103:3300/0 conn(0x7f7694071e60 0x7f769406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:51.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.464+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2965530067 0 0) 0x7f769415d7c0 con 0x7f769406fe20 2024-04-26T23:06:51.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.464+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f767c003520 con 0x7f769406fe20 2024-04-26T23:06:51.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.464+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3575146648 0 0) 0x7f767c003520 con 0x7f769406fe20 2024-04-26T23:06:51.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.464+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 >> v2:172.21.15.103:3300/0 conn(0x7f7694071e60 msgr2=0x7f769406f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:51.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.464+0000 7f7693400640 1 --2- 172.21.15.40:0/3467014402 >> v2:172.21.15.103:3300/0 conn(0x7f7694071e60 0x7f769406f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:51.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.464+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 >> v1:172.21.15.103:6789/0 conn(0x7f76940cc480 legacy=0x7f7694166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:51.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.464+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f769415e060 con 0x7f769406fe20 2024-04-26T23:06:51.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.464+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f7680001670 con 0x7f769406fe20 2024-04-26T23:06:51.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.464+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 >> v1:172.21.15.40:6789/0 conn(0x7f769406fe20 legacy=0x7f7694072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:51.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.464+0000 7f7693400640 1 --2- 172.21.15.40:0/3467014402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f767c0039b0 0x7f767c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:51.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7693400640 1 --2- 172.21.15.40:0/3467014402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c004800 0x7f767c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:51.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7693400640 1 --2- 172.21.15.40:0/3467014402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f767c0051c0 0x7f767c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:51.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f769415dd80 con 0x7f767c004800 2024-04-26T23:06:51.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f767c00c060 con 0x7f767c0039b0 2024-04-26T23:06:51.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f767c00c260 con 0x7f767c0051c0 2024-04-26T23:06:51.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7693e00640 1 --2- 172.21.15.40:0/3467014402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c004800 0x7f767c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:51.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7698a00640 1 --2- 172.21.15.40:0/3467014402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f767c0039b0 0x7f767c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:51.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7699400640 1 --2- 172.21.15.40:0/3467014402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f767c0051c0 0x7f767c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:51.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7693e00640 1 -- 172.21.15.40:0/3467014402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f767c0051c0 msgr2=0x7f767c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:51.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7693e00640 1 --2- 172.21.15.40:0/3467014402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f767c0051c0 0x7f767c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:51.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7693e00640 1 -- 172.21.15.40:0/3467014402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f767c0039b0 msgr2=0x7f767c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:51.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7693e00640 1 --2- 172.21.15.40:0/3467014402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f767c0039b0 0x7f767c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:51.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7693e00640 1 -- 172.21.15.40:0/3467014402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f767c003520 con 0x7f767c004800 2024-04-26T23:06:51.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7699400640 1 --2- 172.21.15.40:0/3467014402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f767c0051c0 0x7f767c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:51.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.465+0000 7f7699400640 1 -- 172.21.15.40:0/3467014402 reap_dead start 2024-04-26T23:06:51.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.466+0000 7f7693e00640 1 --2- 172.21.15.40:0/3467014402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c004800 0x7f767c004c80 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f768000c240 tx=0x7f767c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700cb20062c370d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:51.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.467+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680007cf0 con 0x7f767c004800 2024-04-26T23:06:51.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.467+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76800058d0 con 0x7f767c004800 2024-04-26T23:06:51.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.467+0000 7f7693400640 1 -- 172.21.15.40:0/3467014402 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680005d20 con 0x7f767c004800 2024-04-26T23:06:51.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.467+0000 7f769b400640 1 -- 172.21.15.40:0/3467014402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c004800 msgr2=0x7f767c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.467+0000 7f769b400640 1 --2- 172.21.15.40:0/3467014402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c004800 0x7f767c004c80 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f768000c240 tx=0x7f767c005830 comp rx=0 tx=0).stop 2024-04-26T23:06:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.467+0000 7f769b400640 1 -- 172.21.15.40:0/3467014402 shutdown_connections 2024-04-26T23:06:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.467+0000 7f769b400640 1 --2- 172.21.15.40:0/3467014402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c004800 0x7f767c004c80 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.467+0000 7f769b400640 1 -- 172.21.15.40:0/3467014402 >> 172.21.15.40:0/3467014402 conn(0x7f76940d0be0 msgr2=0x7f76940d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.467+0000 7f769b400640 1 -- 172.21.15.40:0/3467014402 shutdown_connections 2024-04-26T23:06:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.468+0000 7f769b400640 1 -- 172.21.15.40:0/3467014402 wait complete. 2024-04-26T23:06:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.469+0000 7f769b400640 1 Processor -- start 2024-04-26T23:06:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.469+0000 7f769b400640 1 -- start start 2024-04-26T23:06:51.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.470+0000 7f769b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c004800 0x7f76940b5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:51.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.470+0000 7f769b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76940b1630 0x7f769414f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:51.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.470+0000 7f769b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76940aebe0 0x7f76940af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:51.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.470+0000 7f769b400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f769415d570 con 0x7f767c004800 2024-04-26T23:06:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.470+0000 7f769b400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f769415d3f0 con 0x7f76940aebe0 2024-04-26T23:06:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.470+0000 7f769b400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f769406f8e0 con 0x7f76940b1630 2024-04-26T23:06:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.471+0000 7f7698a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c004800 0x7f76940b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.471+0000 7f7698a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c004800 0x7f76940b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58754/0 (socket says 172.21.15.40:58754) 2024-04-26T23:06:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.471+0000 7f7698a00640 1 -- 172.21.15.40:0/24082925 learned_addr learned my addr 172.21.15.40:0/24082925 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.471+0000 7f7699400640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76940aebe0 0x7f76940af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.471+0000 7f7698a00640 1 -- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76940b1630 msgr2=0x7f769414f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.471+0000 7f7693e00640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76940b1630 0x7f769414f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.471+0000 7f7698a00640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76940b1630 0x7f769414f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.471+0000 7f7698a00640 1 -- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76940aebe0 msgr2=0x7f76940af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.471+0000 7f7698a00640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76940aebe0 0x7f76940af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.471+0000 7f7698a00640 1 -- 172.21.15.40:0/24082925 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76940b5ac0 con 0x7f767c004800 2024-04-26T23:06:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.471+0000 7f7698a00640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c004800 0x7f76940b5270 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f76880116d0 tx=0x7f7688011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:51.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.472+0000 7f7690c00640 1 -- 172.21.15.40:0/24082925 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768800f9c0 con 0x7f767c004800 2024-04-26T23:06:51.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.472+0000 7f7690c00640 1 -- 172.21.15.40:0/24082925 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f768800fb60 con 0x7f767c004800 2024-04-26T23:06:51.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.472+0000 7f7690c00640 1 -- 172.21.15.40:0/24082925 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7688015460 con 0x7f767c004800 2024-04-26T23:06:51.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.472+0000 7f769b400640 1 -- 172.21.15.40:0/24082925 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76940b5da0 con 0x7f767c004800 2024-04-26T23:06:51.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.472+0000 7f769b400640 1 -- 172.21.15.40:0/24082925 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f769414fd70 con 0x7f767c004800 2024-04-26T23:06:51.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.473+0000 7f769b400640 1 -- 172.21.15.40:0/24082925 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76940694f0 con 0x7f767c004800 2024-04-26T23:06:51.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.474+0000 7f7690c00640 1 -- 172.21.15.40:0/24082925 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7688015600 con 0x7f767c004800 2024-04-26T23:06:51.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.476+0000 7f7690c00640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f767406c8f0 0x7f767406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:51.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.476+0000 7f7690c00640 1 -- 172.21.15.40:0/24082925 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5507+0+0 (secure 0 0 0) 0x7f768808c570 con 0x7f767c004800 2024-04-26T23:06:51.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.476+0000 7f7693e00640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f767406c8f0 0x7f767406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:51.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.478+0000 7f7693e00640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f767406c8f0 0x7f767406edb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7680033000 tx=0x7f7680009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:51.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.481+0000 7f7690c00640 1 -- 172.21.15.40:0/24082925 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7688019030 con 0x7f767c004800 2024-04-26T23:06:51.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.724+0000 7f769b400640 1 -- 172.21.15.40:0/24082925 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76940735d0 con 0x7f767c004800 2024-04-26T23:06:51.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.724+0000 7f7690c00640 1 -- 172.21.15.40:0/24082925 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+3872 (secure 0 0 0) 0x7f768805a9c0 con 0x7f767c004800 2024-04-26T23:06:51.725 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:51.725 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":68,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/4194140680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4194140680},{"type":"v1","addr":"172.21.15.103:6836","nonce":4194140680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":5908,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/705721464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":705721464},{"type":"v1","addr":"172.21.15.103:6837","nonce":705721464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":7344,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2790227845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2790227845},{"type":"v1","addr":"172.21.15.40:6833","nonce":2790227845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":7353,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2787505959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2787505959},{"type":"v1","addr":"172.21.15.40:6835","nonce":2787505959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[]} 2024-04-26T23:06:51.725 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 68 2024-04-26T23:06:51.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.727+0000 7f769b400640 1 -- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f767406c8f0 msgr2=0x7f767406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.727+0000 7f769b400640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f767406c8f0 0x7f767406edb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7680033000 tx=0x7f7680009030 comp rx=0 tx=0).stop 2024-04-26T23:06:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.727+0000 7f769b400640 1 -- 172.21.15.40:0/24082925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c004800 msgr2=0x7f76940b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.727+0000 7f769b400640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c004800 0x7f76940b5270 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f76880116d0 tx=0x7f7688011b90 comp rx=0 tx=0).stop 2024-04-26T23:06:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.727+0000 7f769b400640 1 -- 172.21.15.40:0/24082925 shutdown_connections 2024-04-26T23:06:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.727+0000 7f769b400640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f767406c8f0 0x7f767406edb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.727+0000 7f769b400640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76940aebe0 0x7f76940af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.727+0000 7f769b400640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76940b1630 0x7f769414f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.727+0000 7f769b400640 1 --2- 172.21.15.40:0/24082925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c004800 0x7f76940b5270 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.727+0000 7f769b400640 1 -- 172.21.15.40:0/24082925 >> 172.21.15.40:0/24082925 conn(0x7f76940d0be0 msgr2=0x7f769415e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.727+0000 7f769b400640 1 -- 172.21.15.40:0/24082925 shutdown_connections 2024-04-26T23:06:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:51.727+0000 7f769b400640 1 -- 172.21.15.40:0/24082925 wait complete. 2024-04-26T23:06:51.747 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-26T23:06:51.747 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-26T23:06:51.747 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-26T23:06:51.747 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:06:51.747 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-26T23:06:51.747 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-26T23:06:51.748 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-26T23:06:51.748 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:06:51.748 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-26T23:06:51.748 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-26T23:06:51.748 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-26T23:06:51.748 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:06:51.748 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-26T23:06:51.749 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-26T23:06:51.749 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-26T23:06:51.749 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:06:51.764 INFO:tasks.ceph.mds.a:Stopped 2024-04-26T23:06:51.764 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-26T23:06:51.767 INFO:tasks.ceph.mds.c:Stopped 2024-04-26T23:06:51.767 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-26T23:06:51.773 INFO:tasks.ceph.mds.b:Stopped 2024-04-26T23:06:51.773 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-26T23:06:51.774 INFO:tasks.ceph.mds.d:Stopped 2024-04-26T23:06:51.774 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-26T23:06:51.813 INFO:tasks.ceph.mds.a:Started 2024-04-26T23:06:51.813 DEBUG:teuthology.parallel:result is None 2024-04-26T23:06:51.819 INFO:tasks.ceph.mds.c:Started 2024-04-26T23:06:51.819 DEBUG:teuthology.parallel:result is None 2024-04-26T23:06:51.821 INFO:tasks.ceph.mds.b:Started 2024-04-26T23:06:51.821 DEBUG:teuthology.parallel:result is None 2024-04-26T23:06:51.827 INFO:tasks.ceph.mds.d:Started 2024-04-26T23:06:51.827 DEBUG:teuthology.parallel:result is None 2024-04-26T23:06:51.827 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-26T23:06:51.969 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-04-26T23:06:51.971 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:06:51.973 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-26T23:06:51.975 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-26T23:06:52.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.012+0000 7fdba2a00640 1 Processor -- start 2024-04-26T23:06:52.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.012+0000 7fdba2a00640 1 -- start start 2024-04-26T23:06:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.013+0000 7fdba2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb9c0cc480 0x7fdb9c072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.013+0000 7fdba2a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdb9c072780 0x7fdb9c166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.013+0000 7fdba2a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb9c15d7c0 con 0x7fdb9c071e60 2024-04-26T23:06:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.013+0000 7fdba2a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdb9c15daa0 con 0x7fdb9c072780 2024-04-26T23:06:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.013+0000 7fdba2a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdb9c15dd80 con 0x7fdb9c0cc480 2024-04-26T23:06:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.014+0000 7fdb9b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb9c0cc480 0x7fdb9c072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.014+0000 7fdba0a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdb9c072780 0x7fdb9c166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.014+0000 7fdb9b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb9c0cc480 0x7fdb9c072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:43726/0 (socket says 172.21.15.40:43726) 2024-04-26T23:06:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.014+0000 7fdb9b400640 1 -- 172.21.15.40:0/3761486694 learned_addr learned my addr 172.21.15.40:0/3761486694 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.014+0000 7fdb9b400640 1 -- 172.21.15.40:0/3761486694 >> v2:172.21.15.103:3300/0 conn(0x7fdb9c072780 msgr2=0x7fdb9c166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.014+0000 7fdb9b400640 1 --2- 172.21.15.40:0/3761486694 >> v2:172.21.15.103:3300/0 conn(0x7fdb9c072780 0x7fdb9c166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.014+0000 7fdb9b400640 1 -- 172.21.15.40:0/3761486694 >> v1:172.21.15.103:6789/0 conn(0x7fdb9c071e60 legacy=0x7fdb9c06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.014+0000 7fdb9b400640 1 -- 172.21.15.40:0/3761486694 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9c15e060 con 0x7fdb9c0cc480 2024-04-26T23:06:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.014+0000 7fdb9aa00640 1 -- 172.21.15.40:0/3761486694 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3325403552 0 0) 0x7fdb9c15d7c0 con 0x7fdb9c071e60 2024-04-26T23:06:52.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.015+0000 7fdb9b400640 1 --2- 172.21.15.40:0/3761486694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb9c0cc480 0x7fdb9c072240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009870 tx=0x7fdb8802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5836fbbe22b8a1a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.016+0000 7fdb9aa00640 1 -- 172.21.15.40:0/3761486694 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88037070 con 0x7fdb9c0cc480 2024-04-26T23:06:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.016+0000 7fdb9aa00640 1 -- 172.21.15.40:0/3761486694 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb88032440 con 0x7fdb9c0cc480 2024-04-26T23:06:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.016+0000 7fdb9aa00640 1 -- 172.21.15.40:0/3761486694 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88032810 con 0x7fdb9c0cc480 2024-04-26T23:06:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.016+0000 7fdba2a00640 1 -- 172.21.15.40:0/3761486694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb9c0cc480 msgr2=0x7fdb9c072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.016+0000 7fdba2a00640 1 --2- 172.21.15.40:0/3761486694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb9c0cc480 0x7fdb9c072240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009870 tx=0x7fdb8802a270 comp rx=0 tx=0).stop 2024-04-26T23:06:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.016+0000 7fdba2a00640 1 -- 172.21.15.40:0/3761486694 shutdown_connections 2024-04-26T23:06:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.016+0000 7fdba2a00640 1 --2- 172.21.15.40:0/3761486694 >> v2:172.21.15.103:3300/0 conn(0x7fdb9c072780 0x7fdb9c166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.016+0000 7fdba2a00640 1 --2- 172.21.15.40:0/3761486694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb9c0cc480 0x7fdb9c072240 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.016+0000 7fdba2a00640 1 -- 172.21.15.40:0/3761486694 >> 172.21.15.40:0/3761486694 conn(0x7fdb9c0d0be0 msgr2=0x7fdb9c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.016+0000 7fdba2a00640 1 -- 172.21.15.40:0/3761486694 shutdown_connections 2024-04-26T23:06:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.017+0000 7fdba2a00640 1 -- 172.21.15.40:0/3761486694 wait complete. 2024-04-26T23:06:52.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.018+0000 7fdba2a00640 1 Processor -- start 2024-04-26T23:06:52.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.018+0000 7fdba2a00640 1 -- start start 2024-04-26T23:06:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.019+0000 7fdba2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c071e60 0x7fdb9c15a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.019+0000 7fdba2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb9c072780 0x7fdb9c1f8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.019+0000 7fdba2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb9c0cc480 0x7fdb9c0b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.019+0000 7fdba2a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9c15d3f0 con 0x7fdb9c071e60 2024-04-26T23:06:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.019+0000 7fdba2a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9c072c10 con 0x7fdb9c072780 2024-04-26T23:06:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.019+0000 7fdba2a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdb9c15d570 con 0x7fdb9c0cc480 2024-04-26T23:06:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.020+0000 7fdb9be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c071e60 0x7fdb9c15a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.020+0000 7fdb9be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c071e60 0x7fdb9c15a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58800/0 (socket says 172.21.15.40:58800) 2024-04-26T23:06:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.020+0000 7fdb9be00640 1 -- 172.21.15.40:0/2807529466 learned_addr learned my addr 172.21.15.40:0/2807529466 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.020+0000 7fdba0a00640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb9c0cc480 0x7fdb9c0b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.020+0000 7fdb9b400640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb9c072780 0x7fdb9c1f8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.020+0000 7fdb9be00640 1 -- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb9c0cc480 msgr2=0x7fdb9c0b5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.020+0000 7fdb9be00640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb9c0cc480 0x7fdb9c0b5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.020+0000 7fdb9be00640 1 -- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb9c072780 msgr2=0x7fdb9c1f8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.020+0000 7fdb9be00640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb9c072780 0x7fdb9c1f8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.020+0000 7fdb9be00640 1 -- 172.21.15.40:0/2807529466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9c1f9280 con 0x7fdb9c071e60 2024-04-26T23:06:52.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.020+0000 7fdb9be00640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c071e60 0x7fdb9c15a210 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c00c460 tx=0x7fdb8c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:52.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.021+0000 7fdb93e00640 1 -- 172.21.15.40:0/2807529466 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c005480 con 0x7fdb9c071e60 2024-04-26T23:06:52.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.021+0000 7fdb93e00640 1 -- 172.21.15.40:0/2807529466 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb8c0035f0 con 0x7fdb9c071e60 2024-04-26T23:06:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.021+0000 7fdba2a00640 1 -- 172.21.15.40:0/2807529466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb9c0aec40 con 0x7fdb9c071e60 2024-04-26T23:06:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.021+0000 7fdb93e00640 1 -- 172.21.15.40:0/2807529466 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c003a00 con 0x7fdb9c071e60 2024-04-26T23:06:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.021+0000 7fdba2a00640 1 -- 172.21.15.40:0/2807529466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb9c0af200 con 0x7fdb9c071e60 2024-04-26T23:06:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.022+0000 7fdba2a00640 1 -- 172.21.15.40:0/2807529466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb9c0af810 con 0x7fdb9c071e60 2024-04-26T23:06:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.023+0000 7fdb93e00640 1 -- 172.21.15.40:0/2807529466 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb8c003ba0 con 0x7fdb9c071e60 2024-04-26T23:06:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.024+0000 7fdb93e00640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb7c06cac0 0x7fdb7c06ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.024+0000 7fdb93e00640 1 -- 172.21.15.40:0/2807529466 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5507+0+0 (secure 0 0 0) 0x7fdb8c08b5d0 con 0x7fdb9c071e60 2024-04-26T23:06:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.025+0000 7fdb9b400640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb7c06cac0 0x7fdb7c06ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.025+0000 7fdb9b400640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb7c06cac0 0x7fdb7c06ef80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdb88034000 tx=0x7fdb8802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.029+0000 7fdb93e00640 1 -- 172.21.15.40:0/2807529466 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fdb8c0593d0 con 0x7fdb9c071e60 2024-04-26T23:06:52.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:52.250+0000 7fdba2a00640 1 -- 172.21.15.40:0/2807529466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fdb9c0694f0 con 0x7fdb9c071e60 2024-04-26T23:06:53.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.336+0000 7fdb93e00640 1 -- 172.21.15.40:0/2807529466 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v75) ==== 97+0+0 (secure 0 0 0) 0x7fdb8c05d6d0 con 0x7fdb9c071e60 2024-04-26T23:06:53.337 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-04-26T23:06:53.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.340+0000 7fdba2a00640 1 -- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb7c06cac0 msgr2=0x7fdb7c06ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:53.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.340+0000 7fdba2a00640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb7c06cac0 0x7fdb7c06ef80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdb88034000 tx=0x7fdb8802e000 comp rx=0 tx=0).stop 2024-04-26T23:06:53.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.340+0000 7fdba2a00640 1 -- 172.21.15.40:0/2807529466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c071e60 msgr2=0x7fdb9c15a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:53.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.340+0000 7fdba2a00640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c071e60 0x7fdb9c15a210 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c00c460 tx=0x7fdb8c00c920 comp rx=0 tx=0).stop 2024-04-26T23:06:53.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.340+0000 7fdba2a00640 1 -- 172.21.15.40:0/2807529466 shutdown_connections 2024-04-26T23:06:53.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.340+0000 7fdba2a00640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb7c06cac0 0x7fdb7c06ef80 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.340+0000 7fdba2a00640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb9c0cc480 0x7fdb9c0b5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.340+0000 7fdba2a00640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb9c072780 0x7fdb9c1f8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.341+0000 7fdba2a00640 1 --2- 172.21.15.40:0/2807529466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c071e60 0x7fdb9c15a210 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.341+0000 7fdba2a00640 1 -- 172.21.15.40:0/2807529466 >> 172.21.15.40:0/2807529466 conn(0x7fdb9c0d0be0 msgr2=0x7fdb9c15f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:53.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.341+0000 7fdba2a00640 1 -- 172.21.15.40:0/2807529466 shutdown_connections 2024-04-26T23:06:53.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.341+0000 7fdba2a00640 1 -- 172.21.15.40:0/2807529466 wait complete. 2024-04-26T23:06:53.363 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-26T23:06:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.581+0000 7f9e39400640 1 Processor -- start 2024-04-26T23:06:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.581+0000 7f9e39400640 1 -- start start 2024-04-26T23:06:53.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.583+0000 7f9e39400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9e34145970 0x7f9e341612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:53.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.583+0000 7f9e39400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e34146060 0x7f9e340b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:53.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.583+0000 7f9e39400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e340b3fb0 0x7f9e340d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:53.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.583+0000 7f9e39400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f9e340ceed0 con 0x7f9e340b3fb0 2024-04-26T23:06:53.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.583+0000 7f9e39400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9e340b59f0 con 0x7f9e34145970 2024-04-26T23:06:53.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.583+0000 7f9e39400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e341617f0 con 0x7f9e34146060 2024-04-26T23:06:53.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.584+0000 7f9e38a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e340b3fb0 0x7f9e340d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:53.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.584+0000 7f9e38a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e340b3fb0 0x7f9e340d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58810/0 (socket says 172.21.15.40:58810) 2024-04-26T23:06:53.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.584+0000 7f9e38a00640 1 -- 172.21.15.40:0/2528148399 learned_addr learned my addr 172.21.15.40:0/2528148399 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:53.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.584+0000 7f9e33e00640 1 --2- 172.21.15.40:0/2528148399 >> v2:172.21.15.103:3300/0 conn(0x7f9e34145970 0x7f9e341612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:53.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.584+0000 7f9e33400640 1 --2- 172.21.15.40:0/2528148399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e34146060 0x7f9e340b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:53.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.584+0000 7f9e38a00640 1 -- 172.21.15.40:0/2528148399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e34146060 msgr2=0x7f9e340b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:53.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.584+0000 7f9e38a00640 1 --2- 172.21.15.40:0/2528148399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e34146060 0x7f9e340b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.584+0000 7f9e38a00640 1 -- 172.21.15.40:0/2528148399 >> v2:172.21.15.103:3300/0 conn(0x7f9e34145970 msgr2=0x7f9e341612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:53.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.584+0000 7f9e38a00640 1 --2- 172.21.15.40:0/2528148399 >> v2:172.21.15.103:3300/0 conn(0x7f9e34145970 0x7f9e341612b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.584+0000 7f9e38a00640 1 -- 172.21.15.40:0/2528148399 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e340b5f40 con 0x7f9e340b3fb0 2024-04-26T23:06:53.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.584+0000 7f9e33400640 1 --2- 172.21.15.40:0/2528148399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e34146060 0x7f9e340b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:53.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.585+0000 7f9e38a00640 1 --2- 172.21.15.40:0/2528148399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e340b3fb0 0x7f9e340d06b0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c009e50 tx=0x7f9e2c02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82634b3e0036cb36 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:53.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.586+0000 7f9e32a00640 1 -- 172.21.15.40:0/2528148399 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c036070 con 0x7f9e340b3fb0 2024-04-26T23:06:53.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.586+0000 7f9e32a00640 1 -- 172.21.15.40:0/2528148399 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e2c031600 con 0x7f9e340b3fb0 2024-04-26T23:06:53.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.586+0000 7f9e32a00640 1 -- 172.21.15.40:0/2528148399 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c0319d0 con 0x7f9e340b3fb0 2024-04-26T23:06:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.586+0000 7f9e39400640 1 -- 172.21.15.40:0/2528148399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e340b3fb0 msgr2=0x7f9e340d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.586+0000 7f9e39400640 1 --2- 172.21.15.40:0/2528148399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e340b3fb0 0x7f9e340d06b0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c009e50 tx=0x7f9e2c02a6c0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.587+0000 7f9e39400640 1 -- 172.21.15.40:0/2528148399 shutdown_connections 2024-04-26T23:06:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.587+0000 7f9e39400640 1 --2- 172.21.15.40:0/2528148399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e340b3fb0 0x7f9e340d06b0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.587+0000 7f9e39400640 1 --2- 172.21.15.40:0/2528148399 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e34146060 0x7f9e340b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.587+0000 7f9e39400640 1 --2- 172.21.15.40:0/2528148399 >> v2:172.21.15.103:3300/0 conn(0x7f9e34145970 0x7f9e341612b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.587+0000 7f9e39400640 1 -- 172.21.15.40:0/2528148399 >> 172.21.15.40:0/2528148399 conn(0x7f9e34075eb0 msgr2=0x7f9e34068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.587+0000 7f9e39400640 1 -- 172.21.15.40:0/2528148399 shutdown_connections 2024-04-26T23:06:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.587+0000 7f9e39400640 1 -- 172.21.15.40:0/2528148399 wait complete. 2024-04-26T23:06:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.589+0000 7f9e39400640 1 Processor -- start 2024-04-26T23:06:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.589+0000 7f9e39400640 1 -- start start 2024-04-26T23:06:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e39400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e34145970 0x7f9e3413cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e39400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e34146060 0x7f9e340cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e39400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e340ccb30 0x7f9e34153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e39400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9e340cf3e0 con 0x7f9e34145970 2024-04-26T23:06:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e39400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e340cf100 con 0x7f9e340ccb30 2024-04-26T23:06:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e39400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e34145d50 con 0x7f9e34146060 2024-04-26T23:06:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e33e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e34145970 0x7f9e3413cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e33e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e34145970 0x7f9e3413cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58814/0 (socket says 172.21.15.40:58814) 2024-04-26T23:06:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e33e00640 1 -- 172.21.15.40:0/1232350375 learned_addr learned my addr 172.21.15.40:0/1232350375 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e33e00640 1 -- 172.21.15.40:0/1232350375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e34146060 msgr2=0x7f9e340cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e33e00640 1 --2- 172.21.15.40:0/1232350375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e34146060 0x7f9e340cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e33e00640 1 -- 172.21.15.40:0/1232350375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e340ccb30 msgr2=0x7f9e34153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e33e00640 1 --2- 172.21.15.40:0/1232350375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e340ccb30 0x7f9e34153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.590+0000 7f9e33e00640 1 -- 172.21.15.40:0/1232350375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e34073510 con 0x7f9e34145970 2024-04-26T23:06:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.591+0000 7f9e33e00640 1 --2- 172.21.15.40:0/1232350375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e34145970 0x7f9e3413cbe0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400d800 tx=0x7f9e2400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.591+0000 7f9e2be00640 1 -- 172.21.15.40:0/1232350375 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2400b6c0 con 0x7f9e34145970 2024-04-26T23:06:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.591+0000 7f9e2be00640 1 -- 172.21.15.40:0/1232350375 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e24019070 con 0x7f9e34145970 2024-04-26T23:06:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.591+0000 7f9e39400640 1 -- 172.21.15.40:0/1232350375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e340cd050 con 0x7f9e34145970 2024-04-26T23:06:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.591+0000 7f9e2be00640 1 -- 172.21.15.40:0/1232350375 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2400ba80 con 0x7f9e34145970 2024-04-26T23:06:53.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.591+0000 7f9e39400640 1 -- 172.21.15.40:0/1232350375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e34154500 con 0x7f9e34145970 2024-04-26T23:06:53.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.593+0000 7f9e39400640 1 -- 172.21.15.40:0/1232350375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e340ce1c0 con 0x7f9e34145970 2024-04-26T23:06:53.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.593+0000 7f9e2be00640 1 -- 172.21.15.40:0/1232350375 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9e2400bc20 con 0x7f9e34145970 2024-04-26T23:06:53.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.594+0000 7f9e2be00640 1 --2- 172.21.15.40:0/1232350375 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9e1406c8f0 0x7f9e1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:53.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.595+0000 7f9e2be00640 1 -- 172.21.15.40:0/1232350375 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5249+0+0 (secure 0 0 0) 0x7f9e2408bfb0 con 0x7f9e34145970 2024-04-26T23:06:53.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.595+0000 7f9e33400640 1 --2- 172.21.15.40:0/1232350375 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9e1406c8f0 0x7f9e1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:53.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.597+0000 7f9e33400640 1 --2- 172.21.15.40:0/1232350375 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9e1406c8f0 0x7f9e1406edb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9e20006b10 tx=0x7f9e20008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:53.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.599+0000 7f9e2be00640 1 -- 172.21.15.40:0/1232350375 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9e340ce1c0 con 0x7f9e34145970 2024-04-26T23:06:53.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.873+0000 7f9e39400640 1 -- 172.21.15.40:0/1232350375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f9e34069370 con 0x7f9e34145970 2024-04-26T23:06:53.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.874+0000 7f9e2be00640 1 -- 172.21.15.40:0/1232350375 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) ==== 80+0+5304 (secure 0 0 0) 0x7f9e2401c360 con 0x7f9e34145970 2024-04-26T23:06:53.874 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmF/UQHBAACdb8iskp2MuBMV6S5tYFJg==", 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.875 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmniCyMxAAT/rVbIKj9EnvwDiblKMCeg==", 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmLGZhJxAAg/fsrRrNoF/Jx1PnBVjjeQ==", 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:06:53.876 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:06:53.877 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:06:53.877 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.877 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.877 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.877 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-04-26T23:06:53.877 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxms3DnAxAAxJj/rybrv/0cLK8PmQ619A==", 2024-04-26T23:06:53.877 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.877 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:06:53.877 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:06:53.877 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:06:53.877 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.877 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDrMSxmbeONCxAATB67ujBDn2tVxpbMju7Mfg==", 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDuMSxmqE/PJhAAsNNu5ar8tHWQauxaXOvagA==", 2024-04-26T23:06:53.878 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.879 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:06:53.879 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:06:53.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.879 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.879 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.879 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-04-26T23:06:53.879 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDyMSxmd2trBhAAbnl7EBldxgtOd1s0Ll0A9w==", 2024-04-26T23:06:53.879 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.879 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD1MSxmqdtKEhAA9oxBxOu/musCKyladrJgYA==", 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.880 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD7MSxm+kl7IxAAlPAGIemdC8gcvp+koeULaQ==", 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD+MSxmzKiaLRAA2UPS1wwJHgMxQf/zLMjMVQ==", 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQACMixmluHrCxAAmbmb33f5MVSTWNuy1D0Zsg==", 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAFMixm6GQjJRAArmdnX7ioekT8GSdsbeB1Ow==", 2024-04-26T23:06:53.882 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg==", 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow r", 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.883 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmVhn4IhAA94XsDk1x5Oax5smXPoFQKg==", 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxm3qhbGRAAK62blgPVShIYskDsMcaffQ==", 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:06:53.884 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmxof/KxAAUmXVV4Vdp2DZnRjQzQ0Inw==", 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.885 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDlMSxmmVwcGRAAqgHymPPTPuS6XXC6LuOVPg==", 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmVf+9EhAAcpk8NgFqgCLXeHA2z7oHmA==", 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.886 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:06:53.887 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-04-26T23:06:53.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.887 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:06:53.887 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:53.887 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-04-26T23:06:53.887 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmjjpcCBAAjDQRUBn/TK9hF3T2WBObyw==", 2024-04-26T23:06:53.887 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:06:53.887 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:06:53.887 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-04-26T23:06:53.887 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:06:53.887 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.887 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:53.888 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-26T23:06:53.888 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:06:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.877+0000 7f9e39400640 1 -- 172.21.15.40:0/1232350375 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9e1406c8f0 msgr2=0x7f9e1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.877+0000 7f9e39400640 1 --2- 172.21.15.40:0/1232350375 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9e1406c8f0 0x7f9e1406edb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9e20006b10 tx=0x7f9e20008040 comp rx=0 tx=0).stop 2024-04-26T23:06:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.877+0000 7f9e39400640 1 -- 172.21.15.40:0/1232350375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e34145970 msgr2=0x7f9e3413cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.877+0000 7f9e39400640 1 --2- 172.21.15.40:0/1232350375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e34145970 0x7f9e3413cbe0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400d800 tx=0x7f9e2400dcc0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.877+0000 7f9e39400640 1 -- 172.21.15.40:0/1232350375 shutdown_connections 2024-04-26T23:06:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.877+0000 7f9e39400640 1 --2- 172.21.15.40:0/1232350375 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9e1406c8f0 0x7f9e1406edb0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.877+0000 7f9e39400640 1 --2- 172.21.15.40:0/1232350375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e340ccb30 0x7f9e34153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.877+0000 7f9e39400640 1 --2- 172.21.15.40:0/1232350375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e34146060 0x7f9e340cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.877+0000 7f9e39400640 1 --2- 172.21.15.40:0/1232350375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e34145970 0x7f9e3413cbe0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.877+0000 7f9e39400640 1 -- 172.21.15.40:0/1232350375 >> 172.21.15.40:0/1232350375 conn(0x7f9e34075eb0 msgr2=0x7f9e34142e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.877+0000 7f9e39400640 1 -- 172.21.15.40:0/1232350375 shutdown_connections 2024-04-26T23:06:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:53.877+0000 7f9e39400640 1 -- 172.21.15.40:0/1232350375 wait complete. 2024-04-26T23:06:53.898 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:06:54.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.104+0000 7fd431800640 1 Processor -- start 2024-04-26T23:06:54.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.104+0000 7fd431800640 1 -- start start 2024-04-26T23:06:54.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.106+0000 7fd431800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd42c073b70 0x7fd42c0aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:54.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.106+0000 7fd431800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42c072c00 con 0x7fd42c0cbb50 2024-04-26T23:06:54.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.106+0000 7fd431800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42c072ee0 con 0x7fd42c15f380 2024-04-26T23:06:54.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.106+0000 7fd431800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fd42c0729b0 con 0x7fd42c073b70 2024-04-26T23:06:54.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.106+0000 7fd42b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd42c0cbb50 0x7fd42c0af1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35452/0 (socket says 172.21.15.40:35452) 2024-04-26T23:06:54.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.106+0000 7fd42b400640 1 -- 172.21.15.40:0/658056854 learned_addr learned my addr 172.21.15.40:0/658056854 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:54.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.106+0000 7fd42a000640 1 --2- 172.21.15.40:0/658056854 >> v2:172.21.15.40:3300/0 conn(0x7fd42c073b70 0x7fd42c0aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:54.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.106+0000 7fd429600640 1 -- 172.21.15.40:0/658056854 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 933835753 0 0) 0x7fd42c072c00 con 0x7fd42c0cbb50 2024-04-26T23:06:54.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.107+0000 7fd429600640 1 -- 172.21.15.40:0/658056854 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd410003520 con 0x7fd42c0cbb50 2024-04-26T23:06:54.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.107+0000 7fd42a000640 1 -- 172.21.15.40:0/658056854 >> v1:172.21.15.103:6789/0 conn(0x7fd42c15f380 legacy=0x7fd42c0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:54.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.107+0000 7fd429600640 1 -- 172.21.15.40:0/658056854 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3865156193 0 0) 0x7fd410003520 con 0x7fd42c0cbb50 2024-04-26T23:06:54.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.107+0000 7fd42a000640 1 -- 172.21.15.40:0/658056854 >> v1:172.21.15.40:6789/0 conn(0x7fd42c0cbb50 legacy=0x7fd42c0af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:54.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.107+0000 7fd42a000640 1 -- 172.21.15.40:0/658056854 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42c0731c0 con 0x7fd42c073b70 2024-04-26T23:06:54.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.108+0000 7fd42a000640 1 --2- 172.21.15.40:0/658056854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd42c073b70 0x7fd42c0aecb0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd4180097a0 tx=0x7fd41802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0f87099898ef6b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:54.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.108+0000 7fd429600640 1 -- 172.21.15.40:0/658056854 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd418037070 con 0x7fd42c073b70 2024-04-26T23:06:54.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.108+0000 7fd429600640 1 -- 172.21.15.40:0/658056854 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd41802ac50 con 0x7fd42c073b70 2024-04-26T23:06:54.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.108+0000 7fd429600640 1 -- 172.21.15.40:0/658056854 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4240019f0 con 0x7fd42c073b70 2024-04-26T23:06:54.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.109+0000 7fd431800640 1 -- 172.21.15.40:0/658056854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd42c073b70 msgr2=0x7fd42c0aecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:54.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.109+0000 7fd431800640 1 --2- 172.21.15.40:0/658056854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd42c073b70 0x7fd42c0aecb0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd4180097a0 tx=0x7fd41802a1c0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.109+0000 7fd431800640 1 -- 172.21.15.40:0/658056854 shutdown_connections 2024-04-26T23:06:54.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.109+0000 7fd431800640 1 --2- 172.21.15.40:0/658056854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd42c073b70 0x7fd42c0aecb0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.109+0000 7fd431800640 1 -- 172.21.15.40:0/658056854 >> 172.21.15.40:0/658056854 conn(0x7fd42c0d0be0 msgr2=0x7fd42c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:54.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.109+0000 7fd431800640 1 -- 172.21.15.40:0/658056854 shutdown_connections 2024-04-26T23:06:54.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.109+0000 7fd431800640 1 -- 172.21.15.40:0/658056854 wait complete. 2024-04-26T23:06:54.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.111+0000 7fd431800640 1 Processor -- start 2024-04-26T23:06:54.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.111+0000 7fd431800640 1 -- start start 2024-04-26T23:06:54.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.112+0000 7fd431800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd42c0cbb50 0x7fd42c0c7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:54.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.112+0000 7fd431800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd42c15f380 0x7fd42c0c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:54.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.112+0000 7fd431800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd42c158540 0x7fd42c152c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:54.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.112+0000 7fd431800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd42c073620 con 0x7fd42c0cbb50 2024-04-26T23:06:54.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.112+0000 7fd431800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd42c0d08f0 con 0x7fd42c15f380 2024-04-26T23:06:54.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.112+0000 7fd431800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd42c0737a0 con 0x7fd42c158540 2024-04-26T23:06:54.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.112+0000 7fd42aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd42c0cbb50 0x7fd42c0c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:54.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.112+0000 7fd42aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd42c0cbb50 0x7fd42c0c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58830/0 (socket says 172.21.15.40:58830) 2024-04-26T23:06:54.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.112+0000 7fd42aa00640 1 -- 172.21.15.40:0/1359929931 learned_addr learned my addr 172.21.15.40:0/1359929931 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.113+0000 7fd42a000640 1 --2- 172.21.15.40:0/1359929931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd42c15f380 0x7fd42c0c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.113+0000 7fd42aa00640 1 -- 172.21.15.40:0/1359929931 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd42c158540 msgr2=0x7fd42c152c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.113+0000 7fd42aa00640 1 --2- 172.21.15.40:0/1359929931 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd42c158540 0x7fd42c152c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.113+0000 7fd42aa00640 1 -- 172.21.15.40:0/1359929931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd42c15f380 msgr2=0x7fd42c0c5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.113+0000 7fd42aa00640 1 --2- 172.21.15.40:0/1359929931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd42c15f380 0x7fd42c0c5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.113+0000 7fd42aa00640 1 -- 172.21.15.40:0/1359929931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42c0c9e10 con 0x7fd42c0cbb50 2024-04-26T23:06:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.113+0000 7fd42aa00640 1 --2- 172.21.15.40:0/1359929931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd42c0cbb50 0x7fd42c0c7060 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00c450 tx=0x7fd41c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.113+0000 7fd422a00640 1 -- 172.21.15.40:0/1359929931 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c0052d0 con 0x7fd42c0cbb50 2024-04-26T23:06:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.113+0000 7fd422a00640 1 -- 172.21.15.40:0/1359929931 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd41c005470 con 0x7fd42c0cbb50 2024-04-26T23:06:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.113+0000 7fd422a00640 1 -- 172.21.15.40:0/1359929931 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c012870 con 0x7fd42c0cbb50 2024-04-26T23:06:54.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.113+0000 7fd431800640 1 -- 172.21.15.40:0/1359929931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd42c0ca0f0 con 0x7fd42c0cbb50 2024-04-26T23:06:54.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.113+0000 7fd431800640 1 -- 172.21.15.40:0/1359929931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd42c0ca6b0 con 0x7fd42c0cbb50 2024-04-26T23:06:54.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.114+0000 7fd431800640 1 -- 172.21.15.40:0/1359929931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd42c0694f0 con 0x7fd42c0cbb50 2024-04-26T23:06:54.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.115+0000 7fd422a00640 1 -- 172.21.15.40:0/1359929931 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd41c012a10 con 0x7fd42c0cbb50 2024-04-26T23:06:54.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.116+0000 7fd422a00640 1 --2- 172.21.15.40:0/1359929931 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd40c06c8f0 0x7fd40c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:54.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.116+0000 7fd422a00640 1 -- 172.21.15.40:0/1359929931 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5249+0+0 (secure 0 0 0) 0x7fd41c08b660 con 0x7fd42c0cbb50 2024-04-26T23:06:54.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.117+0000 7fd42a000640 1 --2- 172.21.15.40:0/1359929931 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd40c06c8f0 0x7fd40c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:54.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.118+0000 7fd42a000640 1 --2- 172.21.15.40:0/1359929931 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd40c06c8f0 0x7fd40c06edb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd418034000 tx=0x7fd41802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.121+0000 7fd422a00640 1 -- 172.21.15.40:0/1359929931 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd41c059bb0 con 0x7fd42c0cbb50 2024-04-26T23:06:54.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.329+0000 7fd431800640 1 -- 172.21.15.40:0/1359929931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd42c0cafa0 con 0x7fd42c0cbb50 2024-04-26T23:06:54.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.332+0000 7fd422a00640 1 -- 172.21.15.40:0/1359929931 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+13129 (secure 0 0 0) 0x7fd41c0594b0 con 0x7fd42c0cbb50 2024-04-26T23:06:54.332 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:54.332 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":75,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:06:53.333584+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:06:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.334+0000 7fd431800640 1 -- 172.21.15.40:0/1359929931 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd40c06c8f0 msgr2=0x7fd40c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.334+0000 7fd431800640 1 --2- 172.21.15.40:0/1359929931 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd40c06c8f0 0x7fd40c06edb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd418034000 tx=0x7fd41802e000 comp rx=0 tx=0).stop 2024-04-26T23:06:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.335+0000 7fd431800640 1 -- 172.21.15.40:0/1359929931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd42c0cbb50 msgr2=0x7fd42c0c7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.335+0000 7fd431800640 1 --2- 172.21.15.40:0/1359929931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd42c0cbb50 0x7fd42c0c7060 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00c450 tx=0x7fd41c00c910 comp rx=0 tx=0).stop 2024-04-26T23:06:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.335+0000 7fd431800640 1 -- 172.21.15.40:0/1359929931 shutdown_connections 2024-04-26T23:06:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.335+0000 7fd431800640 1 --2- 172.21.15.40:0/1359929931 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd40c06c8f0 0x7fd40c06edb0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.335+0000 7fd431800640 1 --2- 172.21.15.40:0/1359929931 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd42c158540 0x7fd42c152c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.335+0000 7fd431800640 1 --2- 172.21.15.40:0/1359929931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd42c15f380 0x7fd42c0c5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.335+0000 7fd431800640 1 --2- 172.21.15.40:0/1359929931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd42c0cbb50 0x7fd42c0c7060 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.335+0000 7fd431800640 1 -- 172.21.15.40:0/1359929931 >> 172.21.15.40:0/1359929931 conn(0x7fd42c0d0be0 msgr2=0x7fd42c0cb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:54.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.335+0000 7fd431800640 1 -- 172.21.15.40:0/1359929931 shutdown_connections 2024-04-26T23:06:54.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.335+0000 7fd431800640 1 -- 172.21.15.40:0/1359929931 wait complete. 2024-04-26T23:06:54.356 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:06:54.357 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:06:54.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.570+0000 7f747b200640 1 Processor -- start 2024-04-26T23:06:54.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.570+0000 7f747b200640 1 -- start start 2024-04-26T23:06:54.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.572+0000 7f747b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f747c075b40 0x7f747c0cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.572+0000 7f747b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f747c0b1400 0x7f747c0b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.572+0000 7f747b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f747c0ce2e0 con 0x7f747c0d0470 2024-04-26T23:06:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.572+0000 7f747b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f747c0b0c90 con 0x7f747c075b40 2024-04-26T23:06:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.572+0000 7f747b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f747c0ce5c0 con 0x7f747c0b1400 2024-04-26T23:06:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.573+0000 7f747a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f747c0b1400 0x7f747c0b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.573+0000 7f7479400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f747c075b40 0x7f747c0cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.573+0000 7f747a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f747c0b1400 0x7f747c0b18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:43770/0 (socket says 172.21.15.40:43770) 2024-04-26T23:06:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.573+0000 7f7479e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f747c0d0470 0x7f747c0d0850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:32870/0 (socket says 172.21.15.40:32870) 2024-04-26T23:06:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.573+0000 7f747a800640 1 -- 172.21.15.40:0/3453572004 learned_addr learned my addr 172.21.15.40:0/3453572004 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.574+0000 7f747a800640 1 -- 172.21.15.40:0/3453572004 >> v2:172.21.15.103:3300/0 conn(0x7f747c075b40 msgr2=0x7f747c0cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:54.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.574+0000 7f747a800640 1 --2- 172.21.15.40:0/3453572004 >> v2:172.21.15.103:3300/0 conn(0x7f747c075b40 0x7f747c0cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.574+0000 7f747a800640 1 -- 172.21.15.40:0/3453572004 >> v1:172.21.15.103:6789/0 conn(0x7f747c0d0470 legacy=0x7f747c0d0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:54.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.574+0000 7f747a800640 1 -- 172.21.15.40:0/3453572004 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747c0ce8a0 con 0x7f747c0b1400 2024-04-26T23:06:54.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.574+0000 7f7479400640 1 --2- 172.21.15.40:0/3453572004 >> v2:172.21.15.103:3300/0 conn(0x7f747c075b40 0x7f747c0cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:06:54.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.575+0000 7f747a800640 1 --2- 172.21.15.40:0/3453572004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f747c0b1400 0x7f747c0b18b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7474009e50 tx=0x7f747402a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af1364ef88447c81 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:54.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.575+0000 7f7478a00640 1 -- 172.21.15.40:0/3453572004 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7474036070 con 0x7f747c0b1400 2024-04-26T23:06:54.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.575+0000 7f7478a00640 1 -- 172.21.15.40:0/3453572004 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7474031600 con 0x7f747c0b1400 2024-04-26T23:06:54.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.575+0000 7f7478a00640 1 -- 172.21.15.40:0/3453572004 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74740319d0 con 0x7f747c0b1400 2024-04-26T23:06:54.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.576+0000 7f747b200640 1 -- 172.21.15.40:0/3453572004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f747c0b1400 msgr2=0x7f747c0b18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:54.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.576+0000 7f747b200640 1 --2- 172.21.15.40:0/3453572004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f747c0b1400 0x7f747c0b18b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7474009e50 tx=0x7f747402a5f0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.576+0000 7f747b200640 1 -- 172.21.15.40:0/3453572004 shutdown_connections 2024-04-26T23:06:54.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.576+0000 7f747b200640 1 --2- 172.21.15.40:0/3453572004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f747c0b1400 0x7f747c0b18b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.576+0000 7f747b200640 1 --2- 172.21.15.40:0/3453572004 >> v2:172.21.15.103:3300/0 conn(0x7f747c075b40 0x7f747c0cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.576+0000 7f747b200640 1 -- 172.21.15.40:0/3453572004 >> 172.21.15.40:0/3453572004 conn(0x7f747c0b2ee0 msgr2=0x7f747c0b32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:54.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.576+0000 7f747b200640 1 -- 172.21.15.40:0/3453572004 shutdown_connections 2024-04-26T23:06:54.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.576+0000 7f747b200640 1 -- 172.21.15.40:0/3453572004 wait complete. 2024-04-26T23:06:54.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.578+0000 7f747b200640 1 Processor -- start 2024-04-26T23:06:54.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.578+0000 7f747b200640 1 -- start start 2024-04-26T23:06:54.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.579+0000 7f747b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f747c075b40 0x7f747c0b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:54.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.579+0000 7f747b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f747c0b1400 0x7f747c1f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:54.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.579+0000 7f747b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f747c0d0470 0x7f747c153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:54.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.579+0000 7f747b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f747c0ae1c0 con 0x7f747c0b1400 2024-04-26T23:06:54.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.579+0000 7f747b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f747c0b2c10 con 0x7f747c0d0470 2024-04-26T23:06:54.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.579+0000 7f747b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f747c0ae340 con 0x7f747c075b40 2024-04-26T23:06:54.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.580+0000 7f7479400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f747c0b1400 0x7f747c1f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:54.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.580+0000 7f7479400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f747c0b1400 0x7f747c1f42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58840/0 (socket says 172.21.15.40:58840) 2024-04-26T23:06:54.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.580+0000 7f7479400640 1 -- 172.21.15.40:0/1586781504 learned_addr learned my addr 172.21.15.40:0/1586781504 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:54.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.580+0000 7f7479e00640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f747c075b40 0x7f747c0b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:54.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.580+0000 7f747a800640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f747c0d0470 0x7f747c153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:54.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.580+0000 7f7479400640 1 -- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f747c075b40 msgr2=0x7f747c0b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:54.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.580+0000 7f7479400640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f747c075b40 0x7f747c0b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.580+0000 7f7479400640 1 -- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f747c0d0470 msgr2=0x7f747c153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:54.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.580+0000 7f7479400640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f747c0d0470 0x7f747c153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.580+0000 7f7479400640 1 -- 172.21.15.40:0/1586781504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747c0c7c00 con 0x7f747c0b1400 2024-04-26T23:06:54.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.581+0000 7f7479400640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f747c0b1400 0x7f747c1f42f0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f746400e960 tx=0x7f746400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:54.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.581+0000 7f7472000640 1 -- 172.21.15.40:0/1586781504 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7464017070 con 0x7f747c0b1400 2024-04-26T23:06:54.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.581+0000 7f7472000640 1 -- 172.21.15.40:0/1586781504 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f746400ca00 con 0x7f747c0b1400 2024-04-26T23:06:54.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.581+0000 7f747b200640 1 -- 172.21.15.40:0/1586781504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f747c0b5660 con 0x7f747c0b1400 2024-04-26T23:06:54.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.581+0000 7f7472000640 1 -- 172.21.15.40:0/1586781504 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746400cdd0 con 0x7f747c0b1400 2024-04-26T23:06:54.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.581+0000 7f747b200640 1 -- 172.21.15.40:0/1586781504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f747c0b5c20 con 0x7f747c0b1400 2024-04-26T23:06:54.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.582+0000 7f747b200640 1 -- 172.21.15.40:0/1586781504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f747c069a80 con 0x7f747c0b1400 2024-04-26T23:06:54.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.583+0000 7f7472000640 1 -- 172.21.15.40:0/1586781504 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f746401a070 con 0x7f747c0b1400 2024-04-26T23:06:54.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.584+0000 7f7472000640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f745806c8f0 0x7f745806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:54.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.584+0000 7f7472000640 1 -- 172.21.15.40:0/1586781504 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5249+0+0 (secure 0 0 0) 0x7f746408b830 con 0x7f747c0b1400 2024-04-26T23:06:54.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.585+0000 7f7479e00640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f745806c8f0 0x7f745806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:54.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.586+0000 7f7479e00640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f745806c8f0 0x7f745806edb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f746c007fd0 tx=0x7f746c007e80 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:54.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.589+0000 7f7472000640 1 -- 172.21.15.40:0/1586781504 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7464059730 con 0x7f747c0b1400 2024-04-26T23:06:54.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.791+0000 7f747b200640 1 -- 172.21.15.40:0/1586781504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f747c0afae0 con 0x7f747c0b1400 2024-04-26T23:06:54.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.792+0000 7f7472000640 1 -- 172.21.15.40:0/1586781504 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) ==== 93+0+10 (secure 0 0 0) 0x7f746405da30 con 0x7f747c0b1400 2024-04-26T23:06:54.792 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:06:54.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.795+0000 7f747b200640 1 -- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f745806c8f0 msgr2=0x7f745806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:54.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.795+0000 7f747b200640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f745806c8f0 0x7f745806edb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f746c007fd0 tx=0x7f746c007e80 comp rx=0 tx=0).stop 2024-04-26T23:06:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.795+0000 7f747b200640 1 -- 172.21.15.40:0/1586781504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f747c0b1400 msgr2=0x7f747c1f42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.795+0000 7f747b200640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f747c0b1400 0x7f747c1f42f0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f746400e960 tx=0x7f746400ee20 comp rx=0 tx=0).stop 2024-04-26T23:06:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.795+0000 7f747b200640 1 -- 172.21.15.40:0/1586781504 shutdown_connections 2024-04-26T23:06:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.795+0000 7f747b200640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f745806c8f0 0x7f745806edb0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.795+0000 7f747b200640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f747c0d0470 0x7f747c153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.795+0000 7f747b200640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f747c0b1400 0x7f747c1f42f0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.795+0000 7f747b200640 1 --2- 172.21.15.40:0/1586781504 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f747c075b40 0x7f747c0b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.795+0000 7f747b200640 1 -- 172.21.15.40:0/1586781504 >> 172.21.15.40:0/1586781504 conn(0x7f747c0b2ee0 msgr2=0x7f747c15c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:54.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.795+0000 7f747b200640 1 -- 172.21.15.40:0/1586781504 shutdown_connections 2024-04-26T23:06:54.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:54.795+0000 7f747b200640 1 -- 172.21.15.40:0/1586781504 wait complete. 2024-04-26T23:06:54.815 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T23:06:55.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.033+0000 7f6508800640 1 Processor -- start 2024-04-26T23:06:55.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.034+0000 7f6508800640 1 -- start start 2024-04-26T23:06:55.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.035+0000 7f6508800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f650015d1a0 0x7f6500071080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:55.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.035+0000 7f6508800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6500160ce0 0x7f6500161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:55.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.035+0000 7f6508800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65000702c0 con 0x7f65000d3b20 2024-04-26T23:06:55.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.035+0000 7f6508800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f6500161670 con 0x7f650015d1a0 2024-04-26T23:06:55.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.035+0000 7f6508800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f65000cec30 con 0x7f6500160ce0 2024-04-26T23:06:55.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.036+0000 7f6505e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f650015d1a0 0x7f6500071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:55.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.036+0000 7f6505400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f65000d3b20 0x7f65000ce520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59426/0 (socket says 172.21.15.40:59426) 2024-04-26T23:06:55.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.036+0000 7f6505400640 1 -- 172.21.15.40:0/939334138 learned_addr learned my addr 172.21.15.40:0/939334138 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:55.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.036+0000 7f6506800640 1 --2- 172.21.15.40:0/939334138 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6500160ce0 0x7f6500161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:55.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.036+0000 7f6504a00640 1 -- 172.21.15.40:0/939334138 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1188425766 0 0) 0x7f65000702c0 con 0x7f65000d3b20 2024-04-26T23:06:55.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.036+0000 7f6505e00640 1 -- 172.21.15.40:0/939334138 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6500160ce0 msgr2=0x7f6500161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:55.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.036+0000 7f6505e00640 1 --2- 172.21.15.40:0/939334138 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6500160ce0 0x7f6500161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.036+0000 7f6505e00640 1 -- 172.21.15.40:0/939334138 >> v1:172.21.15.40:6789/0 conn(0x7f65000d3b20 legacy=0x7f65000ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:55.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.036+0000 7f6505e00640 1 -- 172.21.15.40:0/939334138 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65000705a0 con 0x7f650015d1a0 2024-04-26T23:06:55.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.037+0000 7f6505e00640 1 --2- 172.21.15.40:0/939334138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f650015d1a0 0x7f6500071080 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f64f40097a0 tx=0x7f64f402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1a8989ea0dbd58 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:55.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.038+0000 7f6504a00640 1 -- 172.21.15.40:0/939334138 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f4037070 con 0x7f650015d1a0 2024-04-26T23:06:55.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.038+0000 7f6504a00640 1 -- 172.21.15.40:0/939334138 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64f4032440 con 0x7f650015d1a0 2024-04-26T23:06:55.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.038+0000 7f6504a00640 1 -- 172.21.15.40:0/939334138 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f4032810 con 0x7f650015d1a0 2024-04-26T23:06:55.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.038+0000 7f6508800640 1 -- 172.21.15.40:0/939334138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f650015d1a0 msgr2=0x7f6500071080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:55.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.038+0000 7f6508800640 1 --2- 172.21.15.40:0/939334138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f650015d1a0 0x7f6500071080 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f64f40097a0 tx=0x7f64f402a180 comp rx=0 tx=0).stop 2024-04-26T23:06:55.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.038+0000 7f6508800640 1 -- 172.21.15.40:0/939334138 shutdown_connections 2024-04-26T23:06:55.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.038+0000 7f6508800640 1 --2- 172.21.15.40:0/939334138 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6500160ce0 0x7f6500161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.038+0000 7f6508800640 1 --2- 172.21.15.40:0/939334138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f650015d1a0 0x7f6500071080 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.038+0000 7f6508800640 1 -- 172.21.15.40:0/939334138 >> 172.21.15.40:0/939334138 conn(0x7f65000d2e00 msgr2=0x7f65000d31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:55.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.039+0000 7f6508800640 1 -- 172.21.15.40:0/939334138 shutdown_connections 2024-04-26T23:06:55.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.039+0000 7f6508800640 1 -- 172.21.15.40:0/939334138 wait complete. 2024-04-26T23:06:55.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.040+0000 7f6508800640 1 Processor -- start 2024-04-26T23:06:55.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.040+0000 7f6508800640 1 -- start start 2024-04-26T23:06:55.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.041+0000 7f6508800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65000d3b20 0x7f650014d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:55.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.041+0000 7f6508800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f650015d1a0 0x7f650009ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:55.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.041+0000 7f6508800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6500160ce0 0x7f650014dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:55.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.041+0000 7f6508800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f650015d580 con 0x7f6500160ce0 2024-04-26T23:06:55.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.041+0000 7f6508800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6500070b60 con 0x7f650015d1a0 2024-04-26T23:06:55.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.041+0000 7f6508800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f65000d0d00 con 0x7f65000d3b20 2024-04-26T23:06:55.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.042+0000 7f6506800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6500160ce0 0x7f650014dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.042+0000 7f6506800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6500160ce0 0x7f650014dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54204/0 (socket says 172.21.15.40:54204) 2024-04-26T23:06:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.042+0000 7f6506800640 1 -- 172.21.15.40:0/411964473 learned_addr learned my addr 172.21.15.40:0/411964473 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.042+0000 7f6505e00640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65000d3b20 0x7f650014d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.042+0000 7f6505400640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f650015d1a0 0x7f650009ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.042+0000 7f6506800640 1 -- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65000d3b20 msgr2=0x7f650014d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.042+0000 7f6506800640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65000d3b20 0x7f650014d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.042+0000 7f6506800640 1 -- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f650015d1a0 msgr2=0x7f650009ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.042+0000 7f6506800640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f650015d1a0 0x7f650009ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.042+0000 7f6506800640 1 -- 172.21.15.40:0/411964473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650009f9b0 con 0x7f6500160ce0 2024-04-26T23:06:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.043+0000 7f6506800640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6500160ce0 0x7f650014dfe0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f64fc007e90 tx=0x7f64fc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:55.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.043+0000 7f64fa000640 1 -- 172.21.15.40:0/411964473 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc014420 con 0x7f6500160ce0 2024-04-26T23:06:55.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.043+0000 7f64fa000640 1 -- 172.21.15.40:0/411964473 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64fc019070 con 0x7f6500160ce0 2024-04-26T23:06:55.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.043+0000 7f64fa000640 1 -- 172.21.15.40:0/411964473 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc014800 con 0x7f6500160ce0 2024-04-26T23:06:55.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.043+0000 7f6508800640 1 -- 172.21.15.40:0/411964473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f650009fc90 con 0x7f6500160ce0 2024-04-26T23:06:55.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.043+0000 7f6508800640 1 -- 172.21.15.40:0/411964473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65000a0250 con 0x7f6500160ce0 2024-04-26T23:06:55.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.044+0000 7f6508800640 1 -- 172.21.15.40:0/411964473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65000694f0 con 0x7f6500160ce0 2024-04-26T23:06:55.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.045+0000 7f64fa000640 1 -- 172.21.15.40:0/411964473 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f64fc0149a0 con 0x7f6500160ce0 2024-04-26T23:06:55.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.047+0000 7f64fa000640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f64e406c8f0 0x7f64e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:55.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.047+0000 7f64fa000640 1 -- 172.21.15.40:0/411964473 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5249+0+0 (secure 0 0 0) 0x7f64fc08c270 con 0x7f6500160ce0 2024-04-26T23:06:55.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.047+0000 7f6505e00640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f64e406c8f0 0x7f64e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:55.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.049+0000 7f6505e00640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f64e406c8f0 0x7f64e406edb0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f64f4034000 tx=0x7f64f402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:55.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.052+0000 7f64fa000640 1 -- 172.21.15.40:0/411964473 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f64fc05a7c0 con 0x7f6500160ce0 2024-04-26T23:06:55.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.256+0000 7f6508800640 1 -- 172.21.15.40:0/411964473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f650014da30 con 0x7f6500160ce0 2024-04-26T23:06:55.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.257+0000 7f64fa000640 1 -- 172.21.15.40:0/411964473 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v75) ==== 84+0+65 (secure 0 0 0) 0x7f64fc05a0c0 con 0x7f6500160ce0 2024-04-26T23:06:55.257 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:55.257 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-04-26T23:06:55.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:06:55.258 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-04-26T23:06:55.258 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-04-26T23:06:55.258 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:06:55.258 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-04-26T23:06:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.259+0000 7f6508800640 1 -- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f64e406c8f0 msgr2=0x7f64e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.259+0000 7f6508800640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f64e406c8f0 0x7f64e406edb0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f64f4034000 tx=0x7f64f402e000 comp rx=0 tx=0).stop 2024-04-26T23:06:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.259+0000 7f6508800640 1 -- 172.21.15.40:0/411964473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6500160ce0 msgr2=0x7f650014dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.259+0000 7f6508800640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6500160ce0 0x7f650014dfe0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f64fc007e90 tx=0x7f64fc00e4b0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.259+0000 7f6508800640 1 -- 172.21.15.40:0/411964473 shutdown_connections 2024-04-26T23:06:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.259+0000 7f6508800640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f64e406c8f0 0x7f64e406edb0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.259+0000 7f6508800640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6500160ce0 0x7f650014dfe0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.259+0000 7f6508800640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f650015d1a0 0x7f650009ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.259+0000 7f6508800640 1 --2- 172.21.15.40:0/411964473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65000d3b20 0x7f650014d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.260+0000 7f6508800640 1 -- 172.21.15.40:0/411964473 >> 172.21.15.40:0/411964473 conn(0x7f65000d2e00 msgr2=0x7f65000ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:55.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.260+0000 7f6508800640 1 -- 172.21.15.40:0/411964473 shutdown_connections 2024-04-26T23:06:55.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.260+0000 7f6508800640 1 -- 172.21.15.40:0/411964473 wait complete. 2024-04-26T23:06:55.279 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T23:06:55.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T23:06:55.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.497+0000 7f4922c00640 1 Processor -- start 2024-04-26T23:06:55.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.497+0000 7f4922c00640 1 -- start start 2024-04-26T23:06:55.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.499+0000 7f4922c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f491c0d11a0 0x7f491c06e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:55.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.499+0000 7f4922c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f491c06eaa0 0x7f491c0b5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:55.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.499+0000 7f4922c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f491c15fca0 con 0x7f491c155f20 2024-04-26T23:06:55.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.499+0000 7f4922c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f491c0d1610 con 0x7f491c0d11a0 2024-04-26T23:06:55.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.499+0000 7f4922c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f491c15ff80 con 0x7f491c06eaa0 2024-04-26T23:06:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.500+0000 7f4922200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f491c06eaa0 0x7f491c0b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.500+0000 7f4922200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f491c06eaa0 0x7f491c0b5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:48854/0 (socket says 172.21.15.40:48854) 2024-04-26T23:06:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.500+0000 7f4920e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f491c0d11a0 0x7f491c06e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.500+0000 7f4922200640 1 -- 172.21.15.40:0/433945783 learned_addr learned my addr 172.21.15.40:0/433945783 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:55.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.501+0000 7f4920e00640 1 -- 172.21.15.40:0/433945783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f491c06eaa0 msgr2=0x7f491c0b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:55.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.501+0000 7f4920e00640 1 --2- 172.21.15.40:0/433945783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f491c06eaa0 0x7f491c0b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.501+0000 7f4920e00640 1 -- 172.21.15.40:0/433945783 >> v1:172.21.15.103:6789/0 conn(0x7f491c155f20 legacy=0x7f491c15f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:55.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.501+0000 7f4920e00640 1 -- 172.21.15.40:0/433945783 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f491c160260 con 0x7f491c0d11a0 2024-04-26T23:06:55.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.501+0000 7f4922200640 1 --2- 172.21.15.40:0/433945783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f491c06eaa0 0x7f491c0b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:06:55.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.502+0000 7f4920e00640 1 --2- 172.21.15.40:0/433945783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f491c0d11a0 0x7f491c06e560 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f49080097a0 tx=0x7f490802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc0b3bb35acf8e96 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.503+0000 7f491be00640 1 -- 172.21.15.40:0/433945783 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908037070 con 0x7f491c0d11a0 2024-04-26T23:06:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.503+0000 7f491be00640 1 -- 172.21.15.40:0/433945783 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4908032440 con 0x7f491c0d11a0 2024-04-26T23:06:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.503+0000 7f491be00640 1 -- 172.21.15.40:0/433945783 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908032810 con 0x7f491c0d11a0 2024-04-26T23:06:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.503+0000 7f4922c00640 1 -- 172.21.15.40:0/433945783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f491c0d11a0 msgr2=0x7f491c06e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.503+0000 7f4922c00640 1 --2- 172.21.15.40:0/433945783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f491c0d11a0 0x7f491c06e560 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f49080097a0 tx=0x7f490802a180 comp rx=0 tx=0).stop 2024-04-26T23:06:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.503+0000 7f4922c00640 1 -- 172.21.15.40:0/433945783 shutdown_connections 2024-04-26T23:06:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.503+0000 7f4922c00640 1 --2- 172.21.15.40:0/433945783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f491c06eaa0 0x7f491c0b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.503+0000 7f4922c00640 1 --2- 172.21.15.40:0/433945783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f491c0d11a0 0x7f491c06e560 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.503+0000 7f4922c00640 1 -- 172.21.15.40:0/433945783 >> 172.21.15.40:0/433945783 conn(0x7f491c075eb0 msgr2=0x7f491c068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.503+0000 7f4922c00640 1 -- 172.21.15.40:0/433945783 shutdown_connections 2024-04-26T23:06:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.504+0000 7f4922c00640 1 -- 172.21.15.40:0/433945783 wait complete. 2024-04-26T23:06:55.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.505+0000 7f4922c00640 1 Processor -- start 2024-04-26T23:06:55.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.505+0000 7f4922c00640 1 -- start start 2024-04-26T23:06:55.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.506+0000 7f4922c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f491c06eaa0 0x7f491c14ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:55.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.506+0000 7f4922c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f491c0d11a0 0x7f491c1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:55.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.506+0000 7f4922c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f491c155f20 0x7f491c1f8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:55.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.506+0000 7f4922c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f491c0b5a40 con 0x7f491c155f20 2024-04-26T23:06:55.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.506+0000 7f4922c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f491c156300 con 0x7f491c0d11a0 2024-04-26T23:06:55.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.506+0000 7f4922c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f491c0b5bc0 con 0x7f491c06eaa0 2024-04-26T23:06:55.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.507+0000 7f4922200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f491c155f20 0x7f491c1f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:55.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.507+0000 7f4922200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f491c155f20 0x7f491c1f8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54220/0 (socket says 172.21.15.40:54220) 2024-04-26T23:06:55.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.507+0000 7f4922200640 1 -- 172.21.15.40:0/908999014 learned_addr learned my addr 172.21.15.40:0/908999014 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:55.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.507+0000 7f4921800640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f491c06eaa0 0x7f491c14ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:55.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.507+0000 7f4920e00640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f491c0d11a0 0x7f491c1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.507+0000 7f4922200640 1 -- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f491c06eaa0 msgr2=0x7f491c14ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.507+0000 7f4922200640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f491c06eaa0 0x7f491c14ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.507+0000 7f4922200640 1 -- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f491c0d11a0 msgr2=0x7f491c1584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.507+0000 7f4922200640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f491c0d11a0 0x7f491c1584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.507+0000 7f4922200640 1 -- 172.21.15.40:0/908999014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f491c1e8d70 con 0x7f491c155f20 2024-04-26T23:06:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.508+0000 7f4922200640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f491c155f20 0x7f491c1f8910 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f491400efc0 tx=0x7f491400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.508+0000 7f4919600640 1 -- 172.21.15.40:0/908999014 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491400cbe0 con 0x7f491c155f20 2024-04-26T23:06:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.508+0000 7f4919600640 1 -- 172.21.15.40:0/908999014 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f491400cd80 con 0x7f491c155f20 2024-04-26T23:06:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.508+0000 7f4919600640 1 -- 172.21.15.40:0/908999014 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4914003bb0 con 0x7f491c155f20 2024-04-26T23:06:55.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.508+0000 7f4922c00640 1 -- 172.21.15.40:0/908999014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f491c157800 con 0x7f491c155f20 2024-04-26T23:06:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.508+0000 7f4922c00640 1 -- 172.21.15.40:0/908999014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f491c1e9030 con 0x7f491c155f20 2024-04-26T23:06:55.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.510+0000 7f4922c00640 1 -- 172.21.15.40:0/908999014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f491c069370 con 0x7f491c155f20 2024-04-26T23:06:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.510+0000 7f4919600640 1 -- 172.21.15.40:0/908999014 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f491401b030 con 0x7f491c155f20 2024-04-26T23:06:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.511+0000 7f4919600640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48fc06c8f0 0x7f48fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.512+0000 7f4919600640 1 -- 172.21.15.40:0/908999014 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5249+0+0 (secure 0 0 0) 0x7f491408bce0 con 0x7f491c155f20 2024-04-26T23:06:55.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.512+0000 7f4921800640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48fc06c8f0 0x7f48fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:55.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.513+0000 7f4921800640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48fc06c8f0 0x7f48fc06edb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f49100016e0 tx=0x7f4910009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.516+0000 7f4919600640 1 -- 172.21.15.40:0/908999014 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4914022020 con 0x7f491c155f20 2024-04-26T23:06:55.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:55.724+0000 7f4922c00640 1 -- 172.21.15.40:0/908999014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f491c0b39a0 con 0x7f491c155f20 2024-04-26T23:06:56.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.366+0000 7f4919600640 1 -- 172.21.15.40:0/908999014 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v76) ==== 138+0+0 (secure 0 0 0) 0x7f4914010040 con 0x7f491c155f20 2024-04-26T23:06:56.367 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-04-26T23:06:56.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.370+0000 7f4922c00640 1 -- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48fc06c8f0 msgr2=0x7f48fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:56.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.370+0000 7f4922c00640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48fc06c8f0 0x7f48fc06edb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f49100016e0 tx=0x7f4910009040 comp rx=0 tx=0).stop 2024-04-26T23:06:56.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.370+0000 7f4922c00640 1 -- 172.21.15.40:0/908999014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f491c155f20 msgr2=0x7f491c1f8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:56.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.370+0000 7f4922c00640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f491c155f20 0x7f491c1f8910 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f491400efc0 tx=0x7f491400c490 comp rx=0 tx=0).stop 2024-04-26T23:06:56.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.370+0000 7f4922c00640 1 -- 172.21.15.40:0/908999014 shutdown_connections 2024-04-26T23:06:56.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.370+0000 7f4922c00640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48fc06c8f0 0x7f48fc06edb0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:56.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.370+0000 7f4922c00640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f491c155f20 0x7f491c1f8910 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:56.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.370+0000 7f4922c00640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f491c0d11a0 0x7f491c1584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:56.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.370+0000 7f4922c00640 1 --2- 172.21.15.40:0/908999014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f491c06eaa0 0x7f491c14ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:56.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.370+0000 7f4922c00640 1 -- 172.21.15.40:0/908999014 >> 172.21.15.40:0/908999014 conn(0x7f491c075eb0 msgr2=0x7f491c0d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:56.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.371+0000 7f4922c00640 1 -- 172.21.15.40:0/908999014 shutdown_connections 2024-04-26T23:06:56.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.371+0000 7f4922c00640 1 -- 172.21.15.40:0/908999014 wait complete. 2024-04-26T23:06:56.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T23:06:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.601+0000 7f69ebe00640 1 Processor -- start 2024-04-26T23:06:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.601+0000 7f69ebe00640 1 -- start start 2024-04-26T23:06:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.603+0000 7f69ebe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69ec145970 0x7f69ec1612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.603+0000 7f69ebe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f69ec146060 0x7f69ec0b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.603+0000 7f69ebe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f69ec0b3fb0 0x7f69ec0d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.603+0000 7f69ebe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f69ec0ceed0 con 0x7f69ec0b3fb0 2024-04-26T23:06:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.603+0000 7f69ebe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f69ec0b59f0 con 0x7f69ec146060 2024-04-26T23:06:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.603+0000 7f69ebe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f69ec1617f0 con 0x7f69ec145970 2024-04-26T23:06:56.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.603+0000 7f69eb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f69ec0b3fb0 0x7f69ec0d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:56.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.603+0000 7f69eb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f69ec0b3fb0 0x7f69ec0d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54228/0 (socket says 172.21.15.40:54228) 2024-04-26T23:06:56.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.603+0000 7f69eb400640 1 -- 172.21.15.40:0/408848436 learned_addr learned my addr 172.21.15.40:0/408848436 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:56.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.604+0000 7f69eaa00640 1 --2- 172.21.15.40:0/408848436 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69ec145970 0x7f69ec1612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:56.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.604+0000 7f69ea000640 1 --2- 172.21.15.40:0/408848436 >> v2:172.21.15.103:3300/0 conn(0x7f69ec146060 0x7f69ec0b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:56.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.604+0000 7f69eb400640 1 -- 172.21.15.40:0/408848436 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69ec145970 msgr2=0x7f69ec1612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:56.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.604+0000 7f69eb400640 1 --2- 172.21.15.40:0/408848436 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69ec145970 0x7f69ec1612b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:56.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.604+0000 7f69eb400640 1 -- 172.21.15.40:0/408848436 >> v2:172.21.15.103:3300/0 conn(0x7f69ec146060 msgr2=0x7f69ec0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:56.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.604+0000 7f69eb400640 1 --2- 172.21.15.40:0/408848436 >> v2:172.21.15.103:3300/0 conn(0x7f69ec146060 0x7f69ec0b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:56.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.604+0000 7f69eb400640 1 -- 172.21.15.40:0/408848436 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ec0b5f40 con 0x7f69ec0b3fb0 2024-04-26T23:06:56.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.604+0000 7f69eaa00640 1 --2- 172.21.15.40:0/408848436 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69ec145970 0x7f69ec1612b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:56.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.604+0000 7f69ea000640 1 --2- 172.21.15.40:0/408848436 >> v2:172.21.15.103:3300/0 conn(0x7f69ec146060 0x7f69ec0b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:56.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.605+0000 7f69eb400640 1 --2- 172.21.15.40:0/408848436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69ec0b3fb0 0x7f69ec0d06b0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0741c0 tx=0x7f69e402a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea705023bd7d175f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.605+0000 7f69e9600640 1 -- 172.21.15.40:0/408848436 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4036070 con 0x7f69ec0b3fb0 2024-04-26T23:06:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.605+0000 7f69e9600640 1 -- 172.21.15.40:0/408848436 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69e4031600 con 0x7f69ec0b3fb0 2024-04-26T23:06:56.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.605+0000 7f69e9600640 1 -- 172.21.15.40:0/408848436 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e40319d0 con 0x7f69ec0b3fb0 2024-04-26T23:06:56.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.606+0000 7f69ebe00640 1 -- 172.21.15.40:0/408848436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69ec0b3fb0 msgr2=0x7f69ec0d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:56.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.606+0000 7f69ebe00640 1 --2- 172.21.15.40:0/408848436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69ec0b3fb0 0x7f69ec0d06b0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0741c0 tx=0x7f69e402a6d0 comp rx=0 tx=0).stop 2024-04-26T23:06:56.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.606+0000 7f69ebe00640 1 -- 172.21.15.40:0/408848436 shutdown_connections 2024-04-26T23:06:56.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.606+0000 7f69ebe00640 1 --2- 172.21.15.40:0/408848436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69ec0b3fb0 0x7f69ec0d06b0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:56.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.606+0000 7f69ebe00640 1 --2- 172.21.15.40:0/408848436 >> v2:172.21.15.103:3300/0 conn(0x7f69ec146060 0x7f69ec0b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:56.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.606+0000 7f69ebe00640 1 --2- 172.21.15.40:0/408848436 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69ec145970 0x7f69ec1612b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:56.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.606+0000 7f69ebe00640 1 -- 172.21.15.40:0/408848436 >> 172.21.15.40:0/408848436 conn(0x7f69ec075eb0 msgr2=0x7f69ec068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:56.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.606+0000 7f69ebe00640 1 -- 172.21.15.40:0/408848436 shutdown_connections 2024-04-26T23:06:56.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.606+0000 7f69ebe00640 1 -- 172.21.15.40:0/408848436 wait complete. 2024-04-26T23:06:56.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.608+0000 7f69ebe00640 1 Processor -- start 2024-04-26T23:06:56.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.608+0000 7f69ebe00640 1 -- start start 2024-04-26T23:06:56.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.609+0000 7f69ebe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69ec145970 0x7f69ec12fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:56.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.609+0000 7f69ebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69ec146060 0x7f69ec0cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:56.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.609+0000 7f69ebe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69ec15c5c0 0x7f69ec131240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:56.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.609+0000 7f69ebe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec0cf3e0 con 0x7f69ec146060 2024-04-26T23:06:56.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.609+0000 7f69ebe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec0cf100 con 0x7f69ec15c5c0 2024-04-26T23:06:56.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.609+0000 7f69ebe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f69ec145d50 con 0x7f69ec145970 2024-04-26T23:06:56.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.609+0000 7f69ea000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69ec146060 0x7f69ec0cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:56.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.609+0000 7f69ea000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69ec146060 0x7f69ec0cd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54238/0 (socket says 172.21.15.40:54238) 2024-04-26T23:06:56.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.609+0000 7f69ea000640 1 -- 172.21.15.40:0/1433281412 learned_addr learned my addr 172.21.15.40:0/1433281412 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:56.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.610+0000 7f69eb400640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69ec15c5c0 0x7f69ec131240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:56.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.610+0000 7f69eaa00640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69ec145970 0x7f69ec12fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:56.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.610+0000 7f69ea000640 1 -- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69ec145970 msgr2=0x7f69ec12fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.610+0000 7f69ea000640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69ec145970 0x7f69ec12fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.610+0000 7f69ea000640 1 -- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69ec15c5c0 msgr2=0x7f69ec131240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.610+0000 7f69ea000640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69ec15c5c0 0x7f69ec131240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.610+0000 7f69ea000640 1 -- 172.21.15.40:0/1433281412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ec0cd5a0 con 0x7f69ec146060 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.610+0000 7f69ea000640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69ec146060 0x7f69ec0cd060 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f69d400c820 tx=0x7f69d400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.610+0000 7f69e2a00640 1 -- 172.21.15.40:0/1433281412 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d4015070 con 0x7f69ec146060 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.610+0000 7f69e2a00640 1 -- 172.21.15.40:0/1433281412 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69d4007a30 con 0x7f69ec146060 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.610+0000 7f69e2a00640 1 -- 172.21.15.40:0/1433281412 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d4007de0 con 0x7f69ec146060 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.610+0000 7f69ebe00640 1 -- 172.21.15.40:0/1433281412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69ec1301a0 con 0x7f69ec146060 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.611+0000 7f69ebe00640 1 -- 172.21.15.40:0/1433281412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69ec130760 con 0x7f69ec146060 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.612+0000 7f69ebe00640 1 -- 172.21.15.40:0/1433281412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69ec0734e0 con 0x7f69ec146060 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.613+0000 7f69e2a00640 1 -- 172.21.15.40:0/1433281412 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f69d4018070 con 0x7f69ec146060 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.615+0000 7f69e2a00640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f69c806c920 0x7f69c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.615+0000 7f69e2a00640 1 -- 172.21.15.40:0/1433281412 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5619+0+0 (secure 0 0 0) 0x7f69d408b9e0 con 0x7f69ec146060 2024-04-26T23:06:56.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.615+0000 7f69eaa00640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f69c806c920 0x7f69c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:56.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.616+0000 7f69eaa00640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f69c806c920 0x7f69c806ede0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f69dc008850 tx=0x7f69dc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.620+0000 7f69e2a00640 1 -- 172.21.15.40:0/1433281412 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f69ec0734e0 con 0x7f69ec146060 2024-04-26T23:06:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:56.820+0000 7f69ebe00640 1 -- 172.21.15.40:0/1433281412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f69ec073a40 con 0x7f69ec146060 2024-04-26T23:06:57.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.373+0000 7f69e2a00640 1 -- 172.21.15.40:0/1433281412 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v77) ==== 170+0+0 (secure 0 0 0) 0x7f69ec0cd5a0 con 0x7f69ec146060 2024-04-26T23:06:57.374 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-04-26T23:06:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.377+0000 7f69ebe00640 1 -- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f69c806c920 msgr2=0x7f69c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.377+0000 7f69ebe00640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f69c806c920 0x7f69c806ede0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f69dc008850 tx=0x7f69dc008040 comp rx=0 tx=0).stop 2024-04-26T23:06:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.377+0000 7f69ebe00640 1 -- 172.21.15.40:0/1433281412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69ec146060 msgr2=0x7f69ec0cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.377+0000 7f69ebe00640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69ec146060 0x7f69ec0cd060 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f69d400c820 tx=0x7f69d400cce0 comp rx=0 tx=0).stop 2024-04-26T23:06:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.377+0000 7f69ebe00640 1 -- 172.21.15.40:0/1433281412 shutdown_connections 2024-04-26T23:06:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.377+0000 7f69ebe00640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f69c806c920 0x7f69c806ede0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.377+0000 7f69ebe00640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69ec15c5c0 0x7f69ec131240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.377+0000 7f69ebe00640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69ec146060 0x7f69ec0cd060 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.377+0000 7f69ebe00640 1 --2- 172.21.15.40:0/1433281412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69ec145970 0x7f69ec12fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.377+0000 7f69ebe00640 1 -- 172.21.15.40:0/1433281412 >> 172.21.15.40:0/1433281412 conn(0x7f69ec075eb0 msgr2=0x7f69ec071d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.377+0000 7f69ebe00640 1 -- 172.21.15.40:0/1433281412 shutdown_connections 2024-04-26T23:06:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.378+0000 7f69ebe00640 1 -- 172.21.15.40:0/1433281412 wait complete. 2024-04-26T23:06:57.411 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T23:06:57.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.595+0000 7fb4d7000640 1 Processor -- start 2024-04-26T23:06:57.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.595+0000 7fb4d7000640 1 -- start start 2024-04-26T23:06:57.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.597+0000 7fb4d7000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d015bc70 0x7fb4d00cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:57.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.597+0000 7fb4d7000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb4d0071ef0 0x7fb4d0072990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:57.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.597+0000 7fb4d7000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb4d0072440 0x7fb4d0072ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:57.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.597+0000 7fb4d7000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb4d006e3a0 con 0x7fb4d0071ef0 2024-04-26T23:06:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.597+0000 7fb4d7000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb4d00b4840 con 0x7fb4d0072440 2024-04-26T23:06:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.597+0000 7fb4d7000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb4d006dde0 con 0x7fb4d015bc70 2024-04-26T23:06:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.597+0000 7fb4d5200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb4d0071ef0 0x7fb4d0072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.597+0000 7fb4d5200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb4d0071ef0 0x7fb4d0072990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54246/0 (socket says 172.21.15.40:54246) 2024-04-26T23:06:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.597+0000 7fb4d5200640 1 -- 172.21.15.40:0/2019273587 learned_addr learned my addr 172.21.15.40:0/2019273587 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.598+0000 7fb4d6600640 1 --2- 172.21.15.40:0/2019273587 >> v2:172.21.15.103:3300/0 conn(0x7fb4d0072440 0x7fb4d0072ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.598+0000 7fb4d5c00640 1 --2- 172.21.15.40:0/2019273587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d015bc70 0x7fb4d00cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.598+0000 7fb4d5200640 1 -- 172.21.15.40:0/2019273587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d015bc70 msgr2=0x7fb4d00cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.598+0000 7fb4d5200640 1 --2- 172.21.15.40:0/2019273587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d015bc70 0x7fb4d00cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.598+0000 7fb4d5200640 1 -- 172.21.15.40:0/2019273587 >> v2:172.21.15.103:3300/0 conn(0x7fb4d0072440 msgr2=0x7fb4d0072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.598+0000 7fb4d5200640 1 --2- 172.21.15.40:0/2019273587 >> v2:172.21.15.103:3300/0 conn(0x7fb4d0072440 0x7fb4d0072ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.598+0000 7fb4d5200640 1 -- 172.21.15.40:0/2019273587 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d006e0c0 con 0x7fb4d0071ef0 2024-04-26T23:06:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.598+0000 7fb4d5c00640 1 --2- 172.21.15.40:0/2019273587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d015bc70 0x7fb4d00cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.598+0000 7fb4d6600640 1 --2- 172.21.15.40:0/2019273587 >> v2:172.21.15.103:3300/0 conn(0x7fb4d0072440 0x7fb4d0072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.599+0000 7fb4d5200640 1 --2- 172.21.15.40:0/2019273587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0071ef0 0x7fb4d0072990 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fb4d006daf0 tx=0x7fb4bc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d96b8d0ce261812 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:57.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.599+0000 7fb4cfe00640 1 -- 172.21.15.40:0/2019273587 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc037070 con 0x7fb4d0071ef0 2024-04-26T23:06:57.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.599+0000 7fb4cfe00640 1 -- 172.21.15.40:0/2019273587 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4bc032440 con 0x7fb4d0071ef0 2024-04-26T23:06:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.599+0000 7fb4cfe00640 1 -- 172.21.15.40:0/2019273587 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc032810 con 0x7fb4d0071ef0 2024-04-26T23:06:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.600+0000 7fb4d7000640 1 -- 172.21.15.40:0/2019273587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0071ef0 msgr2=0x7fb4d0072990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.600+0000 7fb4d7000640 1 --2- 172.21.15.40:0/2019273587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0071ef0 0x7fb4d0072990 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fb4d006daf0 tx=0x7fb4bc02a1e0 comp rx=0 tx=0).stop 2024-04-26T23:06:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.600+0000 7fb4d7000640 1 -- 172.21.15.40:0/2019273587 shutdown_connections 2024-04-26T23:06:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.600+0000 7fb4d7000640 1 --2- 172.21.15.40:0/2019273587 >> v2:172.21.15.103:3300/0 conn(0x7fb4d0072440 0x7fb4d0072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.600+0000 7fb4d7000640 1 --2- 172.21.15.40:0/2019273587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0071ef0 0x7fb4d0072990 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.600+0000 7fb4d7000640 1 --2- 172.21.15.40:0/2019273587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d015bc70 0x7fb4d00cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.600+0000 7fb4d7000640 1 -- 172.21.15.40:0/2019273587 >> 172.21.15.40:0/2019273587 conn(0x7fb4d00b2720 msgr2=0x7fb4d00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.600+0000 7fb4d7000640 1 -- 172.21.15.40:0/2019273587 shutdown_connections 2024-04-26T23:06:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.600+0000 7fb4d7000640 1 -- 172.21.15.40:0/2019273587 wait complete. 2024-04-26T23:06:57.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.602+0000 7fb4d7000640 1 Processor -- start 2024-04-26T23:06:57.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.602+0000 7fb4d7000640 1 -- start start 2024-04-26T23:06:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.603+0000 7fb4d7000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d0071ef0 0x7fb4d01f7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.603+0000 7fb4d7000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0072440 0x7fb4d0158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.603+0000 7fb4d7000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d015bc70 0x7fb4d01590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.603+0000 7fb4d7000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d015c050 con 0x7fb4d0072440 2024-04-26T23:06:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.603+0000 7fb4d7000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d006fad0 con 0x7fb4d0071ef0 2024-04-26T23:06:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.603+0000 7fb4d7000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb4d015c1d0 con 0x7fb4d015bc70 2024-04-26T23:06:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.603+0000 7fb4d5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0072440 0x7fb4d0158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.603+0000 7fb4d5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0072440 0x7fb4d0158bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54252/0 (socket says 172.21.15.40:54252) 2024-04-26T23:06:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.603+0000 7fb4d5200640 1 -- 172.21.15.40:0/3707804358 learned_addr learned my addr 172.21.15.40:0/3707804358 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.604+0000 7fb4d5c00640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d0071ef0 0x7fb4d01f7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.604+0000 7fb4d6600640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d015bc70 0x7fb4d01590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.604+0000 7fb4d5200640 1 -- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d015bc70 msgr2=0x7fb4d01590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.604+0000 7fb4d5200640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d015bc70 0x7fb4d01590f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.604+0000 7fb4d5200640 1 -- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d0071ef0 msgr2=0x7fb4d01f7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.604+0000 7fb4d5200640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d0071ef0 0x7fb4d01f7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.604+0000 7fb4d5200640 1 -- 172.21.15.40:0/3707804358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d0153040 con 0x7fb4d0072440 2024-04-26T23:06:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.604+0000 7fb4d5c00640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d0071ef0 0x7fb4d01f7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.604+0000 7fb4d5200640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0072440 0x7fb4d0158bb0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc02a220 tx=0x7fb4bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.604+0000 7fb4cd600640 1 -- 172.21.15.40:0/3707804358 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc037040 con 0x7fb4d0072440 2024-04-26T23:06:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.604+0000 7fb4cd600640 1 -- 172.21.15.40:0/3707804358 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4bc042070 con 0x7fb4d0072440 2024-04-26T23:06:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.604+0000 7fb4cd600640 1 -- 172.21.15.40:0/3707804358 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc030bb0 con 0x7fb4d0072440 2024-04-26T23:06:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.605+0000 7fb4d7000640 1 -- 172.21.15.40:0/3707804358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4d0159750 con 0x7fb4d0072440 2024-04-26T23:06:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.605+0000 7fb4d7000640 1 -- 172.21.15.40:0/3707804358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4d0159d10 con 0x7fb4d0072440 2024-04-26T23:06:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.606+0000 7fb4d7000640 1 -- 172.21.15.40:0/3707804358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4d0069370 con 0x7fb4d0072440 2024-04-26T23:06:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.607+0000 7fb4cd600640 1 -- 172.21.15.40:0/3707804358 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb4bc047460 con 0x7fb4d0072440 2024-04-26T23:06:57.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.608+0000 7fb4cd600640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4b006c920 0x7fb4b006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:57.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.608+0000 7fb4cd600640 1 -- 172.21.15.40:0/3707804358 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6001+0+0 (secure 0 0 0) 0x7fb4bc0ac990 con 0x7fb4d0072440 2024-04-26T23:06:57.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.609+0000 7fb4d5c00640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4b006c920 0x7fb4b006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.610+0000 7fb4d5c00640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4b006c920 0x7fb4b006ede0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4006fd0 tx=0x7fb4c4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:57.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.613+0000 7fb4cd600640 1 -- 172.21.15.40:0/3707804358 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fb4bc045360 con 0x7fb4d0072440 2024-04-26T23:06:57.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:57.852+0000 7fb4d7000640 1 -- 172.21.15.40:0/3707804358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb4d00cd1f0 con 0x7fb4d0072440 2024-04-26T23:06:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.403+0000 7fb4cd600640 1 -- 172.21.15.40:0/3707804358 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v70) ==== 174+0+0 (secure 0 0 0) 0x7fb4bc07ab60 con 0x7fb4d0072440 2024-04-26T23:06:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 11 and data pool 12 2024-04-26T23:06:58.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.409+0000 7fb4d7000640 1 -- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4b006c920 msgr2=0x7fb4b006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:58.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.409+0000 7fb4d7000640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4b006c920 0x7fb4b006ede0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4006fd0 tx=0x7fb4c4008040 comp rx=0 tx=0).stop 2024-04-26T23:06:58.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.409+0000 7fb4d7000640 1 -- 172.21.15.40:0/3707804358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0072440 msgr2=0x7fb4d0158bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:58.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.409+0000 7fb4d7000640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0072440 0x7fb4d0158bb0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc02a220 tx=0x7fb4bc0046c0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.413+0000 7fb4d7000640 1 -- 172.21.15.40:0/3707804358 shutdown_connections 2024-04-26T23:06:58.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.413+0000 7fb4d7000640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4b006c920 0x7fb4b006ede0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.413+0000 7fb4d7000640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d015bc70 0x7fb4d01590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.413+0000 7fb4d7000640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0072440 0x7fb4d0158bb0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.413+0000 7fb4d7000640 1 --2- 172.21.15.40:0/3707804358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d0071ef0 0x7fb4d01f7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.413+0000 7fb4d7000640 1 -- 172.21.15.40:0/3707804358 >> 172.21.15.40:0/3707804358 conn(0x7fb4d00b2720 msgr2=0x7fb4d00b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:58.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.413+0000 7fb4d7000640 1 -- 172.21.15.40:0/3707804358 shutdown_connections 2024-04-26T23:06:58.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.414+0000 7fb4d7000640 1 -- 172.21.15.40:0/3707804358 wait complete. 2024-04-26T23:06:58.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:06:58.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.629+0000 7f748b200640 1 Processor -- start 2024-04-26T23:06:58.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.630+0000 7f748b200640 1 -- start start 2024-04-26T23:06:58.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.631+0000 7f748b200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7484071e60 0x7f748406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:58.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.631+0000 7f748b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f74840cc480 0x7f7484166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:58.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.631+0000 7f748b200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f748415d7c0 con 0x7f748406fe20 2024-04-26T23:06:58.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.631+0000 7f748b200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f748415daa0 con 0x7f7484071e60 2024-04-26T23:06:58.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.631+0000 7f748b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f748415dd80 con 0x7f74840cc480 2024-04-26T23:06:58.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.631+0000 7f7483e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7484071e60 0x7f748406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:58.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.631+0000 7f7483400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f748406fe20 0x7f7484072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59434/0 (socket says 172.21.15.40:59434) 2024-04-26T23:06:58.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.631+0000 7f7483400640 1 -- 172.21.15.40:0/2412094131 learned_addr learned my addr 172.21.15.40:0/2412094131 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:58.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.632+0000 7f7489200640 1 --2- 172.21.15.40:0/2412094131 >> v2:172.21.15.103:3300/0 conn(0x7f74840cc480 0x7f7484166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:58.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.632+0000 7f7483e00640 1 -- 172.21.15.40:0/2412094131 >> v2:172.21.15.103:3300/0 conn(0x7f74840cc480 msgr2=0x7f7484166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:58.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.632+0000 7f7483e00640 1 --2- 172.21.15.40:0/2412094131 >> v2:172.21.15.103:3300/0 conn(0x7f74840cc480 0x7f7484166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.632+0000 7f7483e00640 1 -- 172.21.15.40:0/2412094131 >> v1:172.21.15.40:6789/0 conn(0x7f748406fe20 legacy=0x7f7484072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:58.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.632+0000 7f7483e00640 1 -- 172.21.15.40:0/2412094131 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f748415e060 con 0x7f7484071e60 2024-04-26T23:06:58.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.632+0000 7f7482a00640 1 -- 172.21.15.40:0/2412094131 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1516817312 0 0) 0x7f748415d7c0 con 0x7f748406fe20 2024-04-26T23:06:58.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.633+0000 7f7483e00640 1 --2- 172.21.15.40:0/2412094131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7484071e60 0x7f748406f8e0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f74840b06b0 tx=0x7f747402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e39fc3168d1a20f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:58.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.633+0000 7f7482a00640 1 -- 172.21.15.40:0/2412094131 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7474037070 con 0x7f7484071e60 2024-04-26T23:06:58.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.633+0000 7f7482a00640 1 -- 172.21.15.40:0/2412094131 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7474032440 con 0x7f7484071e60 2024-04-26T23:06:58.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.633+0000 7f7482a00640 1 -- 172.21.15.40:0/2412094131 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7474032810 con 0x7f7484071e60 2024-04-26T23:06:58.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.634+0000 7f748b200640 1 -- 172.21.15.40:0/2412094131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7484071e60 msgr2=0x7f748406f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:58.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.634+0000 7f748b200640 1 --2- 172.21.15.40:0/2412094131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7484071e60 0x7f748406f8e0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f74840b06b0 tx=0x7f747402a1a0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.634+0000 7f748b200640 1 -- 172.21.15.40:0/2412094131 shutdown_connections 2024-04-26T23:06:58.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.634+0000 7f748b200640 1 --2- 172.21.15.40:0/2412094131 >> v2:172.21.15.103:3300/0 conn(0x7f74840cc480 0x7f7484166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.634+0000 7f748b200640 1 --2- 172.21.15.40:0/2412094131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7484071e60 0x7f748406f8e0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.634+0000 7f748b200640 1 -- 172.21.15.40:0/2412094131 >> 172.21.15.40:0/2412094131 conn(0x7f74840d0be0 msgr2=0x7f74840d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:58.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.634+0000 7f748b200640 1 -- 172.21.15.40:0/2412094131 shutdown_connections 2024-04-26T23:06:58.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.634+0000 7f748b200640 1 -- 172.21.15.40:0/2412094131 wait complete. 2024-04-26T23:06:58.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.636+0000 7f748b200640 1 Processor -- start 2024-04-26T23:06:58.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.636+0000 7f748b200640 1 -- start start 2024-04-26T23:06:58.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.637+0000 7f748b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f748406fe20 0x7f74840b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:58.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.637+0000 7f748b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7484071e60 0x7f74840b5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:58.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.637+0000 7f748b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74840cc480 0x7f7484159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:58.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.637+0000 7f748b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f748415d570 con 0x7f748406fe20 2024-04-26T23:06:58.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.638+0000 7f748b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f748415d3f0 con 0x7f7484071e60 2024-04-26T23:06:58.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.638+0000 7f748b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7484072950 con 0x7f74840cc480 2024-04-26T23:06:58.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.638+0000 7f7483e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f748406fe20 0x7f74840b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:58.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.638+0000 7f7483e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f748406fe20 0x7f74840b4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54270/0 (socket says 172.21.15.40:54270) 2024-04-26T23:06:58.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.638+0000 7f7483e00640 1 -- 172.21.15.40:0/3762794478 learned_addr learned my addr 172.21.15.40:0/3762794478 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:58.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.638+0000 7f7483e00640 1 -- 172.21.15.40:0/3762794478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74840cc480 msgr2=0x7f7484159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:58.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.638+0000 7f7483e00640 1 --2- 172.21.15.40:0/3762794478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74840cc480 0x7f7484159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.638+0000 7f7483e00640 1 -- 172.21.15.40:0/3762794478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7484071e60 msgr2=0x7f74840b5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:58.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.638+0000 7f7483e00640 1 --2- 172.21.15.40:0/3762794478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7484071e60 0x7f74840b5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.638+0000 7f7483e00640 1 -- 172.21.15.40:0/3762794478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74841f5fd0 con 0x7f748406fe20 2024-04-26T23:06:58.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.639+0000 7f7483e00640 1 --2- 172.21.15.40:0/3762794478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f748406fe20 0x7f74840b4220 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f7474009890 tx=0x7f74740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:58.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.639+0000 7f747be00640 1 -- 172.21.15.40:0/3762794478 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7474037040 con 0x7f748406fe20 2024-04-26T23:06:58.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.639+0000 7f747be00640 1 -- 172.21.15.40:0/3762794478 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7474042070 con 0x7f748406fe20 2024-04-26T23:06:58.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.639+0000 7f747be00640 1 -- 172.21.15.40:0/3762794478 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7474030bd0 con 0x7f748406fe20 2024-04-26T23:06:58.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.639+0000 7f748b200640 1 -- 172.21.15.40:0/3762794478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7484156ad0 con 0x7f748406fe20 2024-04-26T23:06:58.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.639+0000 7f748b200640 1 -- 172.21.15.40:0/3762794478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7484157090 con 0x7f748406fe20 2024-04-26T23:06:58.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.640+0000 7f748b200640 1 -- 172.21.15.40:0/3762794478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74840694f0 con 0x7f748406fe20 2024-04-26T23:06:58.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.641+0000 7f747be00640 1 -- 172.21.15.40:0/3762794478 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7474047460 con 0x7f748406fe20 2024-04-26T23:06:58.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.642+0000 7f747be00640 1 --2- 172.21.15.40:0/3762794478 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f746006c8f0 0x7f746006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:58.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.642+0000 7f747be00640 1 -- 172.21.15.40:0/3762794478 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7f74740ac8d0 con 0x7f748406fe20 2024-04-26T23:06:58.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.643+0000 7f7483400640 1 --2- 172.21.15.40:0/3762794478 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f746006c8f0 0x7f746006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:58.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.644+0000 7f7483400640 1 --2- 172.21.15.40:0/3762794478 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f746006c8f0 0x7f746006edb0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f746c001560 tx=0x7f746c007ca0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:58.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.647+0000 7f747be00640 1 -- 172.21.15.40:0/3762794478 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7474032e20 con 0x7f748406fe20 2024-04-26T23:06:58.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.845+0000 7f748b200640 1 -- 172.21.15.40:0/3762794478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74840b0d60 con 0x7f748406fe20 2024-04-26T23:06:58.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.848+0000 7f747be00640 1 -- 172.21.15.40:0/3762794478 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16922 (secure 0 0 0) 0x7f747407ab60 con 0x7f748406fe20 2024-04-26T23:06:58.849 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:58.849 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":78,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:06:57.853810+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:06:55.738277+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-26T23:06:56.838617+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:06:58.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.851+0000 7f748b200640 1 -- 172.21.15.40:0/3762794478 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f746006c8f0 msgr2=0x7f746006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:58.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.851+0000 7f748b200640 1 --2- 172.21.15.40:0/3762794478 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f746006c8f0 0x7f746006edb0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f746c001560 tx=0x7f746c007ca0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.851+0000 7f748b200640 1 -- 172.21.15.40:0/3762794478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f748406fe20 msgr2=0x7f74840b4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:58.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.851+0000 7f748b200640 1 --2- 172.21.15.40:0/3762794478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f748406fe20 0x7f74840b4220 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f7474009890 tx=0x7f74740046c0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.851+0000 7f748b200640 1 -- 172.21.15.40:0/3762794478 shutdown_connections 2024-04-26T23:06:58.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.851+0000 7f748b200640 1 --2- 172.21.15.40:0/3762794478 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f746006c8f0 0x7f746006edb0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.851+0000 7f748b200640 1 --2- 172.21.15.40:0/3762794478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74840cc480 0x7f7484159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.851+0000 7f748b200640 1 --2- 172.21.15.40:0/3762794478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7484071e60 0x7f74840b5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.851+0000 7f748b200640 1 --2- 172.21.15.40:0/3762794478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f748406fe20 0x7f74840b4220 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:58.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.851+0000 7f748b200640 1 -- 172.21.15.40:0/3762794478 >> 172.21.15.40:0/3762794478 conn(0x7f74840d0be0 msgr2=0x7f74840cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:58.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.851+0000 7f748b200640 1 -- 172.21.15.40:0/3762794478 shutdown_connections 2024-04-26T23:06:58.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:58.851+0000 7f748b200640 1 -- 172.21.15.40:0/3762794478 wait complete. 2024-04-26T23:06:58.869 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:06:59.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.075+0000 7fba5b200640 1 Processor -- start 2024-04-26T23:06:59.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.076+0000 7fba5b200640 1 -- start start 2024-04-26T23:06:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.078+0000 7fba5b200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fba540b0380 0x7fba540ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.078+0000 7fba5b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fba540b10b0 0x7fba5406c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.078+0000 7fba5b200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba540cfd80 con 0x7fba5406c780 2024-04-26T23:06:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.078+0000 7fba5b200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fba5406cbd0 con 0x7fba540b0380 2024-04-26T23:06:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.078+0000 7fba5b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fba540af8e0 con 0x7fba540b10b0 2024-04-26T23:06:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.078+0000 7fba53e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fba540b0380 0x7fba540ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.078+0000 7fba59200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fba5406c780 0x7fba54072f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59450/0 (socket says 172.21.15.40:59450) 2024-04-26T23:06:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.078+0000 7fba59200640 1 -- 172.21.15.40:0/3623785144 learned_addr learned my addr 172.21.15.40:0/3623785144 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:06:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.078+0000 7fba53e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fba540b0380 0x7fba540ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54286/0 (socket says 172.21.15.40:54286) 2024-04-26T23:06:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.078+0000 7fba4be00640 1 --2- 172.21.15.40:0/3623785144 >> v2:172.21.15.103:3300/0 conn(0x7fba540b10b0 0x7fba5406c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.079+0000 7fba53e00640 1 -- 172.21.15.40:0/3623785144 >> v2:172.21.15.103:3300/0 conn(0x7fba540b10b0 msgr2=0x7fba5406c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.079+0000 7fba53e00640 1 --2- 172.21.15.40:0/3623785144 >> v2:172.21.15.103:3300/0 conn(0x7fba540b10b0 0x7fba5406c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.079+0000 7fba53e00640 1 -- 172.21.15.40:0/3623785144 >> v1:172.21.15.40:6789/0 conn(0x7fba5406c780 legacy=0x7fba54072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.079+0000 7fba53e00640 1 -- 172.21.15.40:0/3623785144 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba540d0020 con 0x7fba540b0380 2024-04-26T23:06:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.079+0000 7fba53400640 1 -- 172.21.15.40:0/3623785144 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2972239348 0 0) 0x7fba540cfd80 con 0x7fba5406c780 2024-04-26T23:06:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.080+0000 7fba53e00640 1 --2- 172.21.15.40:0/3623785144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba540b0380 0x7fba540ccd50 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fba540b48c0 tx=0x7fba4402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa08904d8889b31 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.080+0000 7fba53400640 1 -- 172.21.15.40:0/3623785144 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba44037070 con 0x7fba540b0380 2024-04-26T23:06:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.080+0000 7fba53400640 1 -- 172.21.15.40:0/3623785144 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba44032440 con 0x7fba540b0380 2024-04-26T23:06:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.080+0000 7fba53400640 1 -- 172.21.15.40:0/3623785144 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4402ae60 con 0x7fba540b0380 2024-04-26T23:06:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.081+0000 7fba5b200640 1 -- 172.21.15.40:0/3623785144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba540b0380 msgr2=0x7fba540ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.081+0000 7fba5b200640 1 --2- 172.21.15.40:0/3623785144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba540b0380 0x7fba540ccd50 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fba540b48c0 tx=0x7fba4402a110 comp rx=0 tx=0).stop 2024-04-26T23:06:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.081+0000 7fba5b200640 1 -- 172.21.15.40:0/3623785144 shutdown_connections 2024-04-26T23:06:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.081+0000 7fba5b200640 1 --2- 172.21.15.40:0/3623785144 >> v2:172.21.15.103:3300/0 conn(0x7fba540b10b0 0x7fba5406c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.081+0000 7fba5b200640 1 --2- 172.21.15.40:0/3623785144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba540b0380 0x7fba540ccd50 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.081+0000 7fba5b200640 1 -- 172.21.15.40:0/3623785144 >> 172.21.15.40:0/3623785144 conn(0x7fba540d1cf0 msgr2=0x7fba540d20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:59.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.081+0000 7fba5b200640 1 -- 172.21.15.40:0/3623785144 shutdown_connections 2024-04-26T23:06:59.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.081+0000 7fba5b200640 1 -- 172.21.15.40:0/3623785144 wait complete. 2024-04-26T23:06:59.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.082+0000 7fba5b200640 1 Processor -- start 2024-04-26T23:06:59.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.083+0000 7fba5b200640 1 -- start start 2024-04-26T23:06:59.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.083+0000 7fba5b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba5406c780 0x7fba540cb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:59.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.083+0000 7fba5b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba540b0380 0x7fba540c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:59.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.083+0000 7fba5b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba540b10b0 0x7fba54160a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:59.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.083+0000 7fba5b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fba540d1980 con 0x7fba540b0380 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.083+0000 7fba5b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fba540d05e0 con 0x7fba5406c780 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.083+0000 7fba5b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fba540b0760 con 0x7fba540b10b0 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.084+0000 7fba4be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba540b0380 0x7fba540c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.084+0000 7fba4be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba540b0380 0x7fba540c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54288/0 (socket says 172.21.15.40:54288) 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.084+0000 7fba4be00640 1 -- 172.21.15.40:0/2165646267 learned_addr learned my addr 172.21.15.40:0/2165646267 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.084+0000 7fba59200640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba540b10b0 0x7fba54160a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.084+0000 7fba4be00640 1 -- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba540b10b0 msgr2=0x7fba54160a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.084+0000 7fba4be00640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba540b10b0 0x7fba54160a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.084+0000 7fba53e00640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba5406c780 0x7fba540cb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.084+0000 7fba4be00640 1 -- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba5406c780 msgr2=0x7fba540cb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.084+0000 7fba4be00640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba5406c780 0x7fba540cb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.084+0000 7fba4be00640 1 -- 172.21.15.40:0/2165646267 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba540c9810 con 0x7fba540b0380 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.084+0000 7fba53e00640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba5406c780 0x7fba540cb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:59.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.085+0000 7fba4be00640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba540b0380 0x7fba540c8dc0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00d800 tx=0x7fba3c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:59.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.085+0000 7fba50c00640 1 -- 172.21.15.40:0/2165646267 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c00b6c0 con 0x7fba540b0380 2024-04-26T23:06:59.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.085+0000 7fba50c00640 1 -- 172.21.15.40:0/2165646267 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba3c019070 con 0x7fba540b0380 2024-04-26T23:06:59.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.085+0000 7fba50c00640 1 -- 172.21.15.40:0/2165646267 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c00ba80 con 0x7fba540b0380 2024-04-26T23:06:59.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.085+0000 7fba5b200640 1 -- 172.21.15.40:0/2165646267 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba540c9af0 con 0x7fba540b0380 2024-04-26T23:06:59.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.085+0000 7fba5b200640 1 -- 172.21.15.40:0/2165646267 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba540ca0b0 con 0x7fba540b0380 2024-04-26T23:06:59.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.086+0000 7fba5b200640 1 -- 172.21.15.40:0/2165646267 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba540694f0 con 0x7fba540b0380 2024-04-26T23:06:59.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.087+0000 7fba50c00640 1 -- 172.21.15.40:0/2165646267 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fba3c00bc20 con 0x7fba540b0380 2024-04-26T23:06:59.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.088+0000 7fba50c00640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fba3006c8f0 0x7fba3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:59.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.088+0000 7fba50c00640 1 -- 172.21.15.40:0/2165646267 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6101+0+0 (secure 0 0 0) 0x7fba3c08c270 con 0x7fba540b0380 2024-04-26T23:06:59.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.089+0000 7fba53e00640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fba3006c8f0 0x7fba3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:59.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.090+0000 7fba53e00640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fba3006c8f0 0x7fba3006edb0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fba44034000 tx=0x7fba4402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:59.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.093+0000 7fba50c00640 1 -- 172.21.15.40:0/2165646267 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fba3c01c360 con 0x7fba540b0380 2024-04-26T23:06:59.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.301+0000 7fba5b200640 1 -- 172.21.15.40:0/2165646267 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fba540d0f10 con 0x7fba540b0380 2024-04-26T23:06:59.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.304+0000 7fba50c00640 1 -- 172.21.15.40:0/2165646267 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16922 (secure 0 0 0) 0x7fba3c05a470 con 0x7fba540b0380 2024-04-26T23:06:59.305 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:06:59.305 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":78,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:06:57.853810+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:06:55.738277+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-26T23:06:56.838617+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:06:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.307+0000 7fba5b200640 1 -- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fba3006c8f0 msgr2=0x7fba3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.307+0000 7fba5b200640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fba3006c8f0 0x7fba3006edb0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fba44034000 tx=0x7fba4402e000 comp rx=0 tx=0).stop 2024-04-26T23:06:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.307+0000 7fba5b200640 1 -- 172.21.15.40:0/2165646267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba540b0380 msgr2=0x7fba540c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.307+0000 7fba5b200640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba540b0380 0x7fba540c8dc0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00d800 tx=0x7fba3c00dcc0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.307+0000 7fba5b200640 1 -- 172.21.15.40:0/2165646267 shutdown_connections 2024-04-26T23:06:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.307+0000 7fba5b200640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fba3006c8f0 0x7fba3006edb0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.307+0000 7fba5b200640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba540b10b0 0x7fba54160a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.307+0000 7fba5b200640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba540b0380 0x7fba540c8dc0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.307+0000 7fba5b200640 1 --2- 172.21.15.40:0/2165646267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba5406c780 0x7fba540cb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.307+0000 7fba5b200640 1 -- 172.21.15.40:0/2165646267 >> 172.21.15.40:0/2165646267 conn(0x7fba540d1cf0 msgr2=0x7fba54161f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:59.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.307+0000 7fba5b200640 1 -- 172.21.15.40:0/2165646267 shutdown_connections 2024-04-26T23:06:59.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.307+0000 7fba5b200640 1 -- 172.21.15.40:0/2165646267 wait complete. 2024-04-26T23:06:59.326 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T23:06:59.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.540+0000 7fa915a00640 1 Processor -- start 2024-04-26T23:06:59.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.540+0000 7fa915a00640 1 -- start start 2024-04-26T23:06:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.542+0000 7fa915a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa910072ce0 0x7fa91015f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.542+0000 7fa915a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa91015fcb0 0x7fa9100ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.542+0000 7fa915a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa910073190 con 0x7fa910073600 2024-04-26T23:06:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.542+0000 7fa915a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa910072850 con 0x7fa910072ce0 2024-04-26T23:06:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.542+0000 7fa915a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa9101604e0 con 0x7fa91015fcb0 2024-04-26T23:06:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.543+0000 7fa907e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa910072ce0 0x7fa91015f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.543+0000 7fa907e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa910072ce0 0x7fa91015f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54298/0 (socket says 172.21.15.40:54298) 2024-04-26T23:06:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.543+0000 7fa907e00640 1 -- 172.21.15.40:0/1197722255 learned_addr learned my addr 172.21.15.40:0/1197722255 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.543+0000 7fa915000640 1 --2- 172.21.15.40:0/1197722255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa91015fcb0 0x7fa9100ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.543+0000 7fa907e00640 1 -- 172.21.15.40:0/1197722255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa91015fcb0 msgr2=0x7fa9100ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.543+0000 7fa907e00640 1 --2- 172.21.15.40:0/1197722255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa91015fcb0 0x7fa9100ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.543+0000 7fa907e00640 1 -- 172.21.15.40:0/1197722255 >> v1:172.21.15.103:6789/0 conn(0x7fa910073600 legacy=0x7fa9100739e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:06:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.543+0000 7fa907e00640 1 -- 172.21.15.40:0/1197722255 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa910072200 con 0x7fa910072ce0 2024-04-26T23:06:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.543+0000 7fa915000640 1 --2- 172.21.15.40:0/1197722255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa91015fcb0 0x7fa9100ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:06:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.544+0000 7fa907e00640 1 --2- 172.21.15.40:0/1197722255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa910072ce0 0x7fa91015f770 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8009870 tx=0x7fa8f802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0e0c4a025acf63 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.544+0000 7fa90f400640 1 -- 172.21.15.40:0/1197722255 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8037070 con 0x7fa910072ce0 2024-04-26T23:06:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.544+0000 7fa90f400640 1 -- 172.21.15.40:0/1197722255 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8f8032440 con 0x7fa910072ce0 2024-04-26T23:06:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.544+0000 7fa90f400640 1 -- 172.21.15.40:0/1197722255 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8032810 con 0x7fa910072ce0 2024-04-26T23:06:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.545+0000 7fa915a00640 1 -- 172.21.15.40:0/1197722255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa910072ce0 msgr2=0x7fa91015f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.545+0000 7fa915a00640 1 --2- 172.21.15.40:0/1197722255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa910072ce0 0x7fa91015f770 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8009870 tx=0x7fa8f802a210 comp rx=0 tx=0).stop 2024-04-26T23:06:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.545+0000 7fa915a00640 1 -- 172.21.15.40:0/1197722255 shutdown_connections 2024-04-26T23:06:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.545+0000 7fa915a00640 1 --2- 172.21.15.40:0/1197722255 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa91015fcb0 0x7fa9100ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.545+0000 7fa915a00640 1 --2- 172.21.15.40:0/1197722255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa910072ce0 0x7fa91015f770 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.545+0000 7fa915a00640 1 -- 172.21.15.40:0/1197722255 >> 172.21.15.40:0/1197722255 conn(0x7fa9100b2720 msgr2=0x7fa9100b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:06:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.545+0000 7fa915a00640 1 -- 172.21.15.40:0/1197722255 shutdown_connections 2024-04-26T23:06:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.545+0000 7fa915a00640 1 -- 172.21.15.40:0/1197722255 wait complete. 2024-04-26T23:06:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.547+0000 7fa915a00640 1 Processor -- start 2024-04-26T23:06:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.547+0000 7fa915a00640 1 -- start start 2024-04-26T23:06:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.548+0000 7fa915a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa910072ce0 0x7fa9101f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.548+0000 7fa915a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa910073600 0x7fa910156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.548+0000 7fa915a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa91015fcb0 0x7fa910156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.548+0000 7fa915a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa9101602a0 con 0x7fa910073600 2024-04-26T23:06:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.548+0000 7fa915a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa910160120 con 0x7fa91015fcb0 2024-04-26T23:06:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.548+0000 7fa915a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa910071e20 con 0x7fa910072ce0 2024-04-26T23:06:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.548+0000 7fa907e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa910073600 0x7fa910156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.548+0000 7fa907e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa910073600 0x7fa910156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54308/0 (socket says 172.21.15.40:54308) 2024-04-26T23:06:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.548+0000 7fa907e00640 1 -- 172.21.15.40:0/1361142625 learned_addr learned my addr 172.21.15.40:0/1361142625 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:06:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.549+0000 7fa907e00640 1 -- 172.21.15.40:0/1361142625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa910072ce0 msgr2=0x7fa9101f50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.549+0000 7fa907e00640 1 --2- 172.21.15.40:0/1361142625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa910072ce0 0x7fa9101f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.549+0000 7fa907e00640 1 -- 172.21.15.40:0/1361142625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa91015fcb0 msgr2=0x7fa910156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:06:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.549+0000 7fa907e00640 1 --2- 172.21.15.40:0/1361142625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa91015fcb0 0x7fa910156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:06:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.549+0000 7fa907e00640 1 -- 172.21.15.40:0/1361142625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9100cfb70 con 0x7fa910073600 2024-04-26T23:06:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.549+0000 7fa907e00640 1 --2- 172.21.15.40:0/1361142625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa910073600 0x7fa910156a50 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8009870 tx=0x7fa8f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.549+0000 7fa90cc00640 1 -- 172.21.15.40:0/1361142625 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8041020 con 0x7fa910073600 2024-04-26T23:06:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.549+0000 7fa90cc00640 1 -- 172.21.15.40:0/1361142625 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8f8030970 con 0x7fa910073600 2024-04-26T23:06:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.549+0000 7fa915a00640 1 -- 172.21.15.40:0/1361142625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa910160890 con 0x7fa910073600 2024-04-26T23:06:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.549+0000 7fa90cc00640 1 -- 172.21.15.40:0/1361142625 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8037040 con 0x7fa910073600 2024-04-26T23:06:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.550+0000 7fa915a00640 1 -- 172.21.15.40:0/1361142625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa910160e50 con 0x7fa910073600 2024-04-26T23:06:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.551+0000 7fa915a00640 1 -- 172.21.15.40:0/1361142625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa910069370 con 0x7fa910073600 2024-04-26T23:06:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.551+0000 7fa90cc00640 1 -- 172.21.15.40:0/1361142625 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa8f8044020 con 0x7fa910073600 2024-04-26T23:06:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.553+0000 7fa90cc00640 1 --2- 172.21.15.40:0/1361142625 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa8ec06c8f0 0x7fa8ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:06:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.553+0000 7fa90cc00640 1 -- 172.21.15.40:0/1361142625 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fa8f80ac6d0 con 0x7fa910073600 2024-04-26T23:06:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.554+0000 7fa90fe00640 1 --2- 172.21.15.40:0/1361142625 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa8ec06c8f0 0x7fa8ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:06:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.555+0000 7fa90fe00640 1 --2- 172.21.15.40:0/1361142625 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa8ec06c8f0 0x7fa8ec06edb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa900003cc0 tx=0x7fa9000079b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:06:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.558+0000 7fa90cc00640 1 -- 172.21.15.40:0/1361142625 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa8f804b030 con 0x7fa910073600 2024-04-26T23:06:59.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:06:59.823+0000 7fa915a00640 1 -- 172.21.15.40:0/1361142625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa910163a50 con 0x7fa910073600 2024-04-26T23:07:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.448+0000 7fa90cc00640 1 -- 172.21.15.40:0/1361142625 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v73) ==== 118+0+0 (secure 0 0 0) 0x7fa8f807a970 con 0x7fa910073600 2024-04-26T23:07:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.451+0000 7fa915a00640 1 -- 172.21.15.40:0/1361142625 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa8ec06c8f0 msgr2=0x7fa8ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.451+0000 7fa915a00640 1 --2- 172.21.15.40:0/1361142625 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa8ec06c8f0 0x7fa8ec06edb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa900003cc0 tx=0x7fa9000079b0 comp rx=0 tx=0).stop 2024-04-26T23:07:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.451+0000 7fa915a00640 1 -- 172.21.15.40:0/1361142625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa910073600 msgr2=0x7fa910156a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.451+0000 7fa915a00640 1 --2- 172.21.15.40:0/1361142625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa910073600 0x7fa910156a50 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8009870 tx=0x7fa8f80046c0 comp rx=0 tx=0).stop 2024-04-26T23:07:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.452+0000 7fa915a00640 1 -- 172.21.15.40:0/1361142625 shutdown_connections 2024-04-26T23:07:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.452+0000 7fa915a00640 1 --2- 172.21.15.40:0/1361142625 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa8ec06c8f0 0x7fa8ec06edb0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.452+0000 7fa915a00640 1 --2- 172.21.15.40:0/1361142625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa91015fcb0 0x7fa910156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.452+0000 7fa915a00640 1 --2- 172.21.15.40:0/1361142625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa910073600 0x7fa910156a50 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.452+0000 7fa915a00640 1 --2- 172.21.15.40:0/1361142625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa910072ce0 0x7fa9101f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.452+0000 7fa915a00640 1 -- 172.21.15.40:0/1361142625 >> 172.21.15.40:0/1361142625 conn(0x7fa9100b2720 msgr2=0x7fa9100b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.452+0000 7fa915a00640 1 -- 172.21.15.40:0/1361142625 shutdown_connections 2024-04-26T23:07:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.452+0000 7fa915a00640 1 -- 172.21.15.40:0/1361142625 wait complete. 2024-04-26T23:07:00.476 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-26T23:07:00.477 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-26T23:07:00.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.696+0000 7fe83be00640 1 Processor -- start 2024-04-26T23:07:00.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.696+0000 7fe83be00640 1 -- start start 2024-04-26T23:07:00.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.697+0000 7fe83be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe83c160a50 0x7fe83c073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:00.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.697+0000 7fe83be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe83c0cfaf0 0x7fe83c160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:00.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.698+0000 7fe83be00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe83c161880 con 0x7fe83c06dd40 2024-04-26T23:07:00.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.698+0000 7fe83be00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe83c071ce0 con 0x7fe83c0cfaf0 2024-04-26T23:07:00.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.698+0000 7fe83be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe83c0722a0 con 0x7fe83c160a50 2024-04-26T23:07:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.698+0000 7fe832000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe83c0cfaf0 0x7fe83c160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.698+0000 7fe83b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe83c06dd40 0x7fe83c072730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59456/0 (socket says 172.21.15.40:59456) 2024-04-26T23:07:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.698+0000 7fe83b400640 1 -- 172.21.15.40:0/1174833004 learned_addr learned my addr 172.21.15.40:0/1174833004 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.698+0000 7fe83aa00640 1 --2- 172.21.15.40:0/1174833004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe83c160a50 0x7fe83c073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.699+0000 7fe832000640 1 -- 172.21.15.40:0/1174833004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe83c160a50 msgr2=0x7fe83c073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.699+0000 7fe832000640 1 --2- 172.21.15.40:0/1174833004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe83c160a50 0x7fe83c073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.699+0000 7fe832000640 1 -- 172.21.15.40:0/1174833004 >> v1:172.21.15.40:6789/0 conn(0x7fe83c06dd40 legacy=0x7fe83c072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.699+0000 7fe832000640 1 -- 172.21.15.40:0/1174833004 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83c161370 con 0x7fe83c0cfaf0 2024-04-26T23:07:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.699+0000 7fe83a000640 1 -- 172.21.15.40:0/1174833004 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1679792415 0 0) 0x7fe83c161880 con 0x7fe83c06dd40 2024-04-26T23:07:00.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.700+0000 7fe832000640 1 --2- 172.21.15.40:0/1174833004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c0cfaf0 0x7fe83c160e30 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fe8240097a0 tx=0x7fe82402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba8439a513bae534 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:00.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.700+0000 7fe83a000640 1 -- 172.21.15.40:0/1174833004 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe824037070 con 0x7fe83c0cfaf0 2024-04-26T23:07:00.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.700+0000 7fe83a000640 1 -- 172.21.15.40:0/1174833004 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe824032440 con 0x7fe83c0cfaf0 2024-04-26T23:07:00.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.700+0000 7fe83a000640 1 -- 172.21.15.40:0/1174833004 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe824032810 con 0x7fe83c0cfaf0 2024-04-26T23:07:00.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.701+0000 7fe83be00640 1 -- 172.21.15.40:0/1174833004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c0cfaf0 msgr2=0x7fe83c160e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:00.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.701+0000 7fe83be00640 1 --2- 172.21.15.40:0/1174833004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c0cfaf0 0x7fe83c160e30 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fe8240097a0 tx=0x7fe82402a180 comp rx=0 tx=0).stop 2024-04-26T23:07:00.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.701+0000 7fe83be00640 1 -- 172.21.15.40:0/1174833004 shutdown_connections 2024-04-26T23:07:00.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.701+0000 7fe83be00640 1 --2- 172.21.15.40:0/1174833004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c0cfaf0 0x7fe83c160e30 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:00.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.701+0000 7fe83be00640 1 --2- 172.21.15.40:0/1174833004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe83c160a50 0x7fe83c073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:00.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.701+0000 7fe83be00640 1 -- 172.21.15.40:0/1174833004 >> 172.21.15.40:0/1174833004 conn(0x7fe83c0b2720 msgr2=0x7fe83c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:00.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.701+0000 7fe83be00640 1 -- 172.21.15.40:0/1174833004 shutdown_connections 2024-04-26T23:07:00.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.701+0000 7fe83be00640 1 -- 172.21.15.40:0/1174833004 wait complete. 2024-04-26T23:07:00.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.703+0000 7fe83be00640 1 Processor -- start 2024-04-26T23:07:00.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.703+0000 7fe83be00640 1 -- start start 2024-04-26T23:07:00.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.704+0000 7fe83be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c06dd40 0x7fe83c13e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:00.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.704+0000 7fe83be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe83c160a50 0x7fe83c1f5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:00.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.704+0000 7fe83be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe83c15fa40 0x7fe83c0b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:00.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.704+0000 7fe83be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe83c0d0260 con 0x7fe83c06dd40 2024-04-26T23:07:00.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.704+0000 7fe83be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe83c073f90 con 0x7fe83c160a50 2024-04-26T23:07:00.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.704+0000 7fe83be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe83c071f30 con 0x7fe83c15fa40 2024-04-26T23:07:00.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.704+0000 7fe83aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c06dd40 0x7fe83c13e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:00.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.704+0000 7fe83aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c06dd40 0x7fe83c13e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54324/0 (socket says 172.21.15.40:54324) 2024-04-26T23:07:00.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.704+0000 7fe83aa00640 1 -- 172.21.15.40:0/4250185461 learned_addr learned my addr 172.21.15.40:0/4250185461 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:00.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.705+0000 7fe832000640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe83c160a50 0x7fe83c1f5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:00.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.705+0000 7fe83b400640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe83c15fa40 0x7fe83c0b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:00.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.705+0000 7fe83aa00640 1 -- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe83c15fa40 msgr2=0x7fe83c0b4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:00.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.705+0000 7fe83aa00640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe83c15fa40 0x7fe83c0b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:00.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.705+0000 7fe83aa00640 1 -- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe83c160a50 msgr2=0x7fe83c1f5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:00.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.705+0000 7fe83aa00640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe83c160a50 0x7fe83c1f5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:00.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.705+0000 7fe83aa00640 1 -- 172.21.15.40:0/4250185461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83c160450 con 0x7fe83c06dd40 2024-04-26T23:07:00.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.705+0000 7fe832000640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe83c160a50 0x7fe83c1f5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:00.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.705+0000 7fe83aa00640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c06dd40 0x7fe83c13e4e0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fe82c007e40 tx=0x7fe82c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:00.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.705+0000 7fe833e00640 1 -- 172.21.15.40:0/4250185461 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c014620 con 0x7fe83c06dd40 2024-04-26T23:07:00.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.706+0000 7fe833e00640 1 -- 172.21.15.40:0/4250185461 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe82c019070 con 0x7fe83c06dd40 2024-04-26T23:07:00.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.706+0000 7fe83be00640 1 -- 172.21.15.40:0/4250185461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe83c0ae9f0 con 0x7fe83c06dd40 2024-04-26T23:07:00.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.706+0000 7fe833e00640 1 -- 172.21.15.40:0/4250185461 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c014a00 con 0x7fe83c06dd40 2024-04-26T23:07:00.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.706+0000 7fe83be00640 1 -- 172.21.15.40:0/4250185461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe83c0aef70 con 0x7fe83c06dd40 2024-04-26T23:07:00.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.707+0000 7fe83be00640 1 -- 172.21.15.40:0/4250185461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe83c069370 con 0x7fe83c06dd40 2024-04-26T23:07:00.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.708+0000 7fe833e00640 1 -- 172.21.15.40:0/4250185461 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe82c014ba0 con 0x7fe83c06dd40 2024-04-26T23:07:00.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.709+0000 7fe833e00640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe81806c920 0x7fe81806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:00.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.709+0000 7fe833e00640 1 -- 172.21.15.40:0/4250185461 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fe82c08c570 con 0x7fe83c06dd40 2024-04-26T23:07:00.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.710+0000 7fe832000640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe81806c920 0x7fe81806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:00.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.711+0000 7fe832000640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe81806c920 0x7fe81806ede0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe824034000 tx=0x7fe82402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:00.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.714+0000 7fe833e00640 1 -- 172.21.15.40:0/4250185461 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe82c05a800 con 0x7fe83c06dd40 2024-04-26T23:07:00.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:00.962+0000 7fe83be00640 1 -- 172.21.15.40:0/4250185461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fe83c15df10 con 0x7fe83c06dd40 2024-04-26T23:07:01.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.458+0000 7fe833e00640 1 -- 172.21.15.40:0/4250185461 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v74) ==== 122+0+0 (secure 0 0 0) 0x7fe82c05a100 con 0x7fe83c06dd40 2024-04-26T23:07:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.462+0000 7fe83be00640 1 -- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe81806c920 msgr2=0x7fe81806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.462+0000 7fe83be00640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe81806c920 0x7fe81806ede0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe824034000 tx=0x7fe82402e000 comp rx=0 tx=0).stop 2024-04-26T23:07:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.462+0000 7fe83be00640 1 -- 172.21.15.40:0/4250185461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c06dd40 msgr2=0x7fe83c13e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.462+0000 7fe83be00640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c06dd40 0x7fe83c13e4e0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fe82c007e40 tx=0x7fe82c00e4b0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.462+0000 7fe83be00640 1 -- 172.21.15.40:0/4250185461 shutdown_connections 2024-04-26T23:07:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.462+0000 7fe83be00640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe81806c920 0x7fe81806ede0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.462+0000 7fe83be00640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe83c15fa40 0x7fe83c0b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.462+0000 7fe83be00640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe83c160a50 0x7fe83c1f5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.462+0000 7fe83be00640 1 --2- 172.21.15.40:0/4250185461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe83c06dd40 0x7fe83c13e4e0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.462+0000 7fe83be00640 1 -- 172.21.15.40:0/4250185461 >> 172.21.15.40:0/4250185461 conn(0x7fe83c0b2720 msgr2=0x7fe83c15d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.462+0000 7fe83be00640 1 -- 172.21.15.40:0/4250185461 shutdown_connections 2024-04-26T23:07:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.462+0000 7fe83be00640 1 -- 172.21.15.40:0/4250185461 wait complete. 2024-04-26T23:07:01.484 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-26T23:07:01.484 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:01.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.698+0000 7f515ac00640 1 Processor -- start 2024-04-26T23:07:01.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.699+0000 7f515ac00640 1 -- start start 2024-04-26T23:07:01.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.700+0000 7f515ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51540cc480 0x7f5154072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:01.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.700+0000 7f515ac00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5154072780 0x7f5154166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:01.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.700+0000 7f515ac00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f515415d7c0 con 0x7f5154071e60 2024-04-26T23:07:01.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.700+0000 7f515ac00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f515415daa0 con 0x7f5154072780 2024-04-26T23:07:01.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.700+0000 7f515ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f515415dd80 con 0x7f51540cc480 2024-04-26T23:07:01.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.700+0000 7f5153e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5154071e60 0x7f515406f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59462/0 (socket says 172.21.15.40:59462) 2024-04-26T23:07:01.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.700+0000 7f5153e00640 1 -- 172.21.15.40:0/3836173020 learned_addr learned my addr 172.21.15.40:0/3836173020 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:01.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2821664108 0 0) 0x7f515415d7c0 con 0x7f5154071e60 2024-04-26T23:07:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5158c00640 1 --2- 172.21.15.40:0/3836173020 >> v2:172.21.15.103:3300/0 conn(0x7f5154072780 0x7f5154166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f514b400640 1 --2- 172.21.15.40:0/3836173020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51540cc480 0x7f5154072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5138003520 con 0x7f5154071e60 2024-04-26T23:07:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3601401880 0 0) 0x7f5138003520 con 0x7f5154071e60 2024-04-26T23:07:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51540cc480 msgr2=0x7f5154072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 --2- 172.21.15.40:0/3836173020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51540cc480 0x7f5154072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 >> v2:172.21.15.103:3300/0 conn(0x7f5154072780 msgr2=0x7f5154166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 --2- 172.21.15.40:0/3836173020 >> v2:172.21.15.103:3300/0 conn(0x7f5154072780 0x7f5154166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515415e060 con 0x7f5154071e60 2024-04-26T23:07:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f51440019f0 con 0x7f5154071e60 2024-04-26T23:07:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 >> v1:172.21.15.40:6789/0 conn(0x7f5154071e60 legacy=0x7f515406f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 --2- 172.21.15.40:0/3836173020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51380039b0 0x7f5138003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 --2- 172.21.15.40:0/3836173020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51540cc480 0x7f5138004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 --2- 172.21.15.40:0/3836173020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5138004d40 0x7f51380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f515415daa0 con 0x7f51380039b0 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f515415dd80 con 0x7f5138004d40 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f513800baf0 con 0x7f51540cc480 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.701+0000 7f5153e00640 1 --2- 172.21.15.40:0/3836173020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51380039b0 0x7f5138003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.702+0000 7f514b400640 1 --2- 172.21.15.40:0/3836173020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51540cc480 0x7f5138004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.702+0000 7f5153e00640 1 -- 172.21.15.40:0/3836173020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51540cc480 msgr2=0x7f5138004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.702+0000 7f5153e00640 1 --2- 172.21.15.40:0/3836173020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51540cc480 0x7f5138004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.702+0000 7f5153e00640 1 -- 172.21.15.40:0/3836173020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5138004d40 msgr2=0x7f51380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.702+0000 7f5153e00640 1 --2- 172.21.15.40:0/3836173020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5138004d40 0x7f51380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.702+0000 7f5153e00640 1 -- 172.21.15.40:0/3836173020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5138003520 con 0x7f51380039b0 2024-04-26T23:07:01.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.703+0000 7f5153e00640 1 --2- 172.21.15.40:0/3836173020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51380039b0 0x7f5138003dd0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f514400a7b0 tx=0x7f5138005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17e1b3f9668c63b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:01.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.703+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514400a3d0 con 0x7f51380039b0 2024-04-26T23:07:01.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.703+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f514402f070 con 0x7f51380039b0 2024-04-26T23:07:01.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.703+0000 7f5153400640 1 -- 172.21.15.40:0/3836173020 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514402bc20 con 0x7f51380039b0 2024-04-26T23:07:01.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.704+0000 7f515ac00640 1 -- 172.21.15.40:0/3836173020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51380039b0 msgr2=0x7f5138003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:01.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.704+0000 7f515ac00640 1 --2- 172.21.15.40:0/3836173020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51380039b0 0x7f5138003dd0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f514400a7b0 tx=0x7f5138005830 comp rx=0 tx=0).stop 2024-04-26T23:07:01.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.704+0000 7f5158c00640 1 -- 172.21.15.40:0/3836173020 reap_dead start 2024-04-26T23:07:01.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.704+0000 7f515ac00640 1 -- 172.21.15.40:0/3836173020 shutdown_connections 2024-04-26T23:07:01.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.704+0000 7f515ac00640 1 -- 172.21.15.40:0/3836173020 >> 172.21.15.40:0/3836173020 conn(0x7f51540d0be0 msgr2=0x7f51540d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:01.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.704+0000 7f515ac00640 1 -- 172.21.15.40:0/3836173020 shutdown_connections 2024-04-26T23:07:01.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.704+0000 7f515ac00640 1 -- 172.21.15.40:0/3836173020 wait complete. 2024-04-26T23:07:01.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.706+0000 7f515ac00640 1 Processor -- start 2024-04-26T23:07:01.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.706+0000 7f515ac00640 1 -- start start 2024-04-26T23:07:01.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f515ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51380039b0 0x7f5154148af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:01.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f515ac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51540c9840 0x7f51540c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:01.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f515ac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51540c6280 0x7f51540c6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:01.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f515ac00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f515415d3f0 con 0x7f51540c6280 2024-04-26T23:07:01.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f515ac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5154072c10 con 0x7f51540c9840 2024-04-26T23:07:01.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f515ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f515415d570 con 0x7f51380039b0 2024-04-26T23:07:01.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f5158c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51540c6280 0x7f51540c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:01.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f5158c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51540c6280 0x7f51540c6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54346/0 (socket says 172.21.15.40:54346) 2024-04-26T23:07:01.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f5158c00640 1 -- 172.21.15.40:0/796990286 learned_addr learned my addr 172.21.15.40:0/796990286 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f5153e00640 1 --2- 172.21.15.40:0/796990286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51380039b0 0x7f5154148af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f5158c00640 1 -- 172.21.15.40:0/796990286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51380039b0 msgr2=0x7f5154148af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f5158c00640 1 --2- 172.21.15.40:0/796990286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51380039b0 0x7f5154148af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f5158c00640 1 -- 172.21.15.40:0/796990286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51540c9840 msgr2=0x7f51540c9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f5158c00640 1 --2- 172.21.15.40:0/796990286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51540c9840 0x7f51540c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.707+0000 7f5158c00640 1 -- 172.21.15.40:0/796990286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51540ca300 con 0x7f51540c6280 2024-04-26T23:07:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.708+0000 7f5158c00640 1 --2- 172.21.15.40:0/796990286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51540c6280 0x7f51540c6700 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f51541609e0 tx=0x7f514c00d820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.708+0000 7f5150c00640 1 -- 172.21.15.40:0/796990286 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c004020 con 0x7f51540c6280 2024-04-26T23:07:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.708+0000 7f5150c00640 1 -- 172.21.15.40:0/796990286 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f514c0041e0 con 0x7f51540c6280 2024-04-26T23:07:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.708+0000 7f515ac00640 1 -- 172.21.15.40:0/796990286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51540ca5e0 con 0x7f51540c6280 2024-04-26T23:07:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.708+0000 7f5150c00640 1 -- 172.21.15.40:0/796990286 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c0043d0 con 0x7f51540c6280 2024-04-26T23:07:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.708+0000 7f515ac00640 1 -- 172.21.15.40:0/796990286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5154146c70 con 0x7f51540c6280 2024-04-26T23:07:01.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.710+0000 7f515ac00640 1 -- 172.21.15.40:0/796990286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51540694f0 con 0x7f51540c6280 2024-04-26T23:07:01.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.710+0000 7f5150c00640 1 -- 172.21.15.40:0/796990286 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f514c01a020 con 0x7f51540c6280 2024-04-26T23:07:01.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.711+0000 7f5150c00640 1 --2- 172.21.15.40:0/796990286 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f513006c920 0x7f513006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:01.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.712+0000 7f5150c00640 1 -- 172.21.15.40:0/796990286 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f514c08b3e0 con 0x7f51540c6280 2024-04-26T23:07:01.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.712+0000 7f5153e00640 1 --2- 172.21.15.40:0/796990286 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f513006c920 0x7f513006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:01.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.713+0000 7f5153e00640 1 --2- 172.21.15.40:0/796990286 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f513006c920 0x7f513006ede0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5144033000 tx=0x7f5144007000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:01.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.716+0000 7f5150c00640 1 -- 172.21.15.40:0/796990286 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f514c0595e0 con 0x7f51540c6280 2024-04-26T23:07:01.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.963+0000 7f515ac00640 1 -- 172.21.15.40:0/796990286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51540cb010 con 0x7f51540c6280 2024-04-26T23:07:01.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.964+0000 7f5150c00640 1 -- 172.21.15.40:0/796990286 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5183 (secure 0 0 0) 0x7f514c058ee0 con 0x7f51540c6280 2024-04-26T23:07:01.965 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:01.965 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":74,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/755627718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":755627718},{"type":"v1","addr":"172.21.15.103:6837","nonce":755627718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":5965,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2914603370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2914603370},{"type":"v1","addr":"172.21.15.103:6836","nonce":2914603370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":7443,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/216893598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":216893598},{"type":"v1","addr":"172.21.15.40:6834","nonce":216893598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:06:57.853769+0000","modified":"2024-04-26T23:07:00.963881+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7446":{"gid":7446,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4227275373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":4227275373},{"type":"v1","addr":"172.21.15.40:6835","nonce":4227275373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7446,"qdb_cluster":[7446]},"id":5}]} 2024-04-26T23:07:01.965 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 74 2024-04-26T23:07:01.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.967+0000 7f515ac00640 1 -- 172.21.15.40:0/796990286 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f513006c920 msgr2=0x7f513006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:01.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.967+0000 7f515ac00640 1 --2- 172.21.15.40:0/796990286 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f513006c920 0x7f513006ede0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5144033000 tx=0x7f5144007000 comp rx=0 tx=0).stop 2024-04-26T23:07:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.967+0000 7f515ac00640 1 -- 172.21.15.40:0/796990286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51540c6280 msgr2=0x7f51540c6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.967+0000 7f515ac00640 1 --2- 172.21.15.40:0/796990286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51540c6280 0x7f51540c6700 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f51541609e0 tx=0x7f514c00d820 comp rx=0 tx=0).stop 2024-04-26T23:07:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.967+0000 7f515ac00640 1 -- 172.21.15.40:0/796990286 shutdown_connections 2024-04-26T23:07:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.967+0000 7f515ac00640 1 --2- 172.21.15.40:0/796990286 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f513006c920 0x7f513006ede0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.967+0000 7f515ac00640 1 --2- 172.21.15.40:0/796990286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51540c6280 0x7f51540c6700 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.967+0000 7f515ac00640 1 --2- 172.21.15.40:0/796990286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51540c9840 0x7f51540c9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.967+0000 7f515ac00640 1 --2- 172.21.15.40:0/796990286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f51380039b0 0x7f5154148af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.967+0000 7f515ac00640 1 -- 172.21.15.40:0/796990286 >> 172.21.15.40:0/796990286 conn(0x7f51540d0be0 msgr2=0x7f51540b5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.967+0000 7f515ac00640 1 -- 172.21.15.40:0/796990286 shutdown_connections 2024-04-26T23:07:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:01.967+0000 7f515ac00640 1 -- 172.21.15.40:0/796990286 wait complete. 2024-04-26T23:07:01.987 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:07:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.198+0000 7f86cdc00640 1 Processor -- start 2024-04-26T23:07:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.199+0000 7f86cdc00640 1 -- start start 2024-04-26T23:07:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.200+0000 7f86cdc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86c8162880 0x7f86c80d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.200+0000 7f86cdc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f86c80d3bc0 0x7f86c80b8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.200+0000 7f86cdc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86c80b7e50 con 0x7f86c80b5f20 2024-04-26T23:07:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.200+0000 7f86cdc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f86c80d0580 con 0x7f86c80d3bc0 2024-04-26T23:07:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.200+0000 7f86cdc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f86c80d1800 con 0x7f86c8162880 2024-04-26T23:07:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.201+0000 7f86c7400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f86c80d3bc0 0x7f86c80b8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.201+0000 7f86c7400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f86c80d3bc0 0x7f86c80b8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54350/0 (socket says 172.21.15.40:54350) 2024-04-26T23:07:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.201+0000 7f86c7400640 1 -- 172.21.15.40:0/2727140884 learned_addr learned my addr 172.21.15.40:0/2727140884 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.201+0000 7f86c6a00640 1 --2- 172.21.15.40:0/2727140884 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86c8162880 0x7f86c80d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.201+0000 7f86c7400640 1 -- 172.21.15.40:0/2727140884 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86c8162880 msgr2=0x7f86c80d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.201+0000 7f86c7400640 1 --2- 172.21.15.40:0/2727140884 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86c8162880 0x7f86c80d2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.201+0000 7f86c7400640 1 -- 172.21.15.40:0/2727140884 >> v1:172.21.15.103:6789/0 conn(0x7f86c80b5f20 legacy=0x7f86c80d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.201+0000 7f86c7400640 1 -- 172.21.15.40:0/2727140884 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86c80d07d0 con 0x7f86c80d3bc0 2024-04-26T23:07:02.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.202+0000 7f86c7400640 1 --2- 172.21.15.40:0/2727140884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86c80d3bc0 0x7f86c80b8300 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e50 tx=0x7f86c002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6042437d3bb2241d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:02.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.203+0000 7f86c5600640 1 -- 172.21.15.40:0/2727140884 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c0036070 con 0x7f86c80d3bc0 2024-04-26T23:07:02.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.203+0000 7f86c5600640 1 -- 172.21.15.40:0/2727140884 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86c0031600 con 0x7f86c80d3bc0 2024-04-26T23:07:02.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.203+0000 7f86c5600640 1 -- 172.21.15.40:0/2727140884 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c00319d0 con 0x7f86c80d3bc0 2024-04-26T23:07:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.203+0000 7f86cdc00640 1 -- 172.21.15.40:0/2727140884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86c80d3bc0 msgr2=0x7f86c80b8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.203+0000 7f86cdc00640 1 --2- 172.21.15.40:0/2727140884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86c80d3bc0 0x7f86c80b8300 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e50 tx=0x7f86c002a710 comp rx=0 tx=0).stop 2024-04-26T23:07:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.204+0000 7f86cdc00640 1 -- 172.21.15.40:0/2727140884 shutdown_connections 2024-04-26T23:07:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.204+0000 7f86cdc00640 1 --2- 172.21.15.40:0/2727140884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86c80d3bc0 0x7f86c80b8300 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.204+0000 7f86cdc00640 1 --2- 172.21.15.40:0/2727140884 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86c8162880 0x7f86c80d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.204+0000 7f86cdc00640 1 -- 172.21.15.40:0/2727140884 >> 172.21.15.40:0/2727140884 conn(0x7f86c80d4f60 msgr2=0x7f86c80d5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.204+0000 7f86cdc00640 1 -- 172.21.15.40:0/2727140884 shutdown_connections 2024-04-26T23:07:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.204+0000 7f86cdc00640 1 -- 172.21.15.40:0/2727140884 wait complete. 2024-04-26T23:07:02.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.206+0000 7f86cdc00640 1 Processor -- start 2024-04-26T23:07:02.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.206+0000 7f86cdc00640 1 -- start start 2024-04-26T23:07:02.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.207+0000 7f86cdc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c80b5f20 0x7f86c8159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:02.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.207+0000 7f86cdc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86c80d3bc0 0x7f86c80cd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:02.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.207+0000 7f86cdc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86c8162880 0x7f86c806dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.207+0000 7f86cdc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f86c8162c60 con 0x7f86c8162880 2024-04-26T23:07:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.207+0000 7f86cdc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f86c80d0d90 con 0x7f86c80b5f20 2024-04-26T23:07:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.207+0000 7f86cdc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f86c8162de0 con 0x7f86c80d3bc0 2024-04-26T23:07:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.207+0000 7f86c7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86c8162880 0x7f86c806dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.207+0000 7f86c7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86c8162880 0x7f86c806dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54366/0 (socket says 172.21.15.40:54366) 2024-04-26T23:07:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.207+0000 7f86c7400640 1 -- 172.21.15.40:0/2022975621 learned_addr learned my addr 172.21.15.40:0/2022975621 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.207+0000 7f86c6a00640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c80b5f20 0x7f86c8159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.208+0000 7f86c6000640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86c80d3bc0 0x7f86c80cd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.208+0000 7f86c7400640 1 -- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86c80d3bc0 msgr2=0x7f86c80cd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.208+0000 7f86c7400640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86c80d3bc0 0x7f86c80cd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.208+0000 7f86c7400640 1 -- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c80b5f20 msgr2=0x7f86c8159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.208+0000 7f86c7400640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c80b5f20 0x7f86c8159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.208+0000 7f86c7400640 1 -- 172.21.15.40:0/2022975621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86c80cea90 con 0x7f86c8162880 2024-04-26T23:07:02.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.208+0000 7f86c6a00640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c80b5f20 0x7f86c8159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:02.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.208+0000 7f86c6000640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86c80d3bc0 0x7f86c80cd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:02.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.208+0000 7f86c7400640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86c8162880 0x7f86c806dd60 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e50 tx=0x7f86c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:02.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.208+0000 7f86bea00640 1 -- 172.21.15.40:0/2022975621 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c0041020 con 0x7f86c8162880 2024-04-26T23:07:02.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.208+0000 7f86bea00640 1 -- 172.21.15.40:0/2022975621 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86c0008710 con 0x7f86c8162880 2024-04-26T23:07:02.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.208+0000 7f86bea00640 1 -- 172.21.15.40:0/2022975621 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c0036040 con 0x7f86c8162880 2024-04-26T23:07:02.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.208+0000 7f86cdc00640 1 -- 172.21.15.40:0/2022975621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86c80ce1f0 con 0x7f86c8162880 2024-04-26T23:07:02.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.209+0000 7f86cdc00640 1 -- 172.21.15.40:0/2022975621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86c80ce7b0 con 0x7f86c8162880 2024-04-26T23:07:02.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.210+0000 7f86cdc00640 1 -- 172.21.15.40:0/2022975621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86c80694f0 con 0x7f86c8162880 2024-04-26T23:07:02.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.211+0000 7f86bea00640 1 -- 172.21.15.40:0/2022975621 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f86c0044020 con 0x7f86c8162880 2024-04-26T23:07:02.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.212+0000 7f86bea00640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f86a406c8f0 0x7f86a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:02.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.212+0000 7f86bea00640 1 -- 172.21.15.40:0/2022975621 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f86c00ac9a0 con 0x7f86c8162880 2024-04-26T23:07:02.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.213+0000 7f86c6a00640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f86a406c8f0 0x7f86a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:02.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.214+0000 7f86c6a00640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f86a406c8f0 0x7f86a406edb0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f86b8006fd0 tx=0x7f86b8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:02.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.216+0000 7f86bea00640 1 -- 172.21.15.40:0/2022975621 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f86c004b070 con 0x7f86c8162880 2024-04-26T23:07:02.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.420+0000 7f86cdc00640 1 -- 172.21.15.40:0/2022975621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86c80d2ec0 con 0x7f86c8162880 2024-04-26T23:07:02.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.423+0000 7f86bea00640 1 -- 172.21.15.40:0/2022975621 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+16909 (secure 0 0 0) 0x7f86c007ac40 con 0x7f86c8162880 2024-04-26T23:07:02.424 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:02.424 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":79,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:06:59.420463+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:06:55.738277+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-26T23:06:56.838617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:07:02.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.426+0000 7f86cdc00640 1 -- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f86a406c8f0 msgr2=0x7f86a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:02.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.426+0000 7f86cdc00640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f86a406c8f0 0x7f86a406edb0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f86b8006fd0 tx=0x7f86b8008040 comp rx=0 tx=0).stop 2024-04-26T23:07:02.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.426+0000 7f86cdc00640 1 -- 172.21.15.40:0/2022975621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86c8162880 msgr2=0x7f86c806dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:02.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.426+0000 7f86cdc00640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86c8162880 0x7f86c806dd60 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e50 tx=0x7f86c0004a40 comp rx=0 tx=0).stop 2024-04-26T23:07:02.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.426+0000 7f86cdc00640 1 -- 172.21.15.40:0/2022975621 shutdown_connections 2024-04-26T23:07:02.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.426+0000 7f86cdc00640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f86a406c8f0 0x7f86a406edb0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.426+0000 7f86cdc00640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86c8162880 0x7f86c806dd60 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.426+0000 7f86cdc00640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86c80d3bc0 0x7f86c80cd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.426+0000 7f86cdc00640 1 --2- 172.21.15.40:0/2022975621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c80b5f20 0x7f86c8159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.426+0000 7f86cdc00640 1 -- 172.21.15.40:0/2022975621 >> 172.21.15.40:0/2022975621 conn(0x7f86c80d4f60 msgr2=0x7f86c80d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:02.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.426+0000 7f86cdc00640 1 -- 172.21.15.40:0/2022975621 shutdown_connections 2024-04-26T23:07:02.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.426+0000 7f86cdc00640 1 -- 172.21.15.40:0/2022975621 wait complete. 2024-04-26T23:07:02.446 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T23:07:02.447 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:07:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.659+0000 7f2fbfc00640 1 Processor -- start 2024-04-26T23:07:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.659+0000 7f2fbfc00640 1 -- start start 2024-04-26T23:07:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.660+0000 7f2fbfc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2fb80cc480 0x7f2fb8072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.660+0000 7f2fbfc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2fb8072780 0x7f2fb8166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.660+0000 7f2fbfc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb815d7c0 con 0x7f2fb8071e60 2024-04-26T23:07:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.660+0000 7f2fbfc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f2fb815daa0 con 0x7f2fb8072780 2024-04-26T23:07:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.660+0000 7f2fbfc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2fb815dd80 con 0x7f2fb80cc480 2024-04-26T23:07:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.661+0000 7f2fbdc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2fb8072780 0x7f2fb8166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.661+0000 7f2fbd200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2fb8071e60 0x7f2fb806f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59470/0 (socket says 172.21.15.40:59470) 2024-04-26T23:07:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.661+0000 7f2fbd200640 1 -- 172.21.15.40:0/501247508 learned_addr learned my addr 172.21.15.40:0/501247508 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.661+0000 7f2fbdc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2fb8072780 0x7f2fb8166ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54372/0 (socket says 172.21.15.40:54372) 2024-04-26T23:07:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.661+0000 7f2fbdc00640 1 -- 172.21.15.40:0/501247508 >> v2:172.21.15.103:3300/0 conn(0x7f2fb80cc480 msgr2=0x7f2fb8072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.661+0000 7f2fb7e00640 1 --2- 172.21.15.40:0/501247508 >> v2:172.21.15.103:3300/0 conn(0x7f2fb80cc480 0x7f2fb8072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.661+0000 7f2fbdc00640 1 --2- 172.21.15.40:0/501247508 >> v2:172.21.15.103:3300/0 conn(0x7f2fb80cc480 0x7f2fb8072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.661+0000 7f2fbdc00640 1 -- 172.21.15.40:0/501247508 >> v1:172.21.15.40:6789/0 conn(0x7f2fb8071e60 legacy=0x7f2fb806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.661+0000 7f2fbdc00640 1 -- 172.21.15.40:0/501247508 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb815e060 con 0x7f2fb8072780 2024-04-26T23:07:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.661+0000 7f2fb7e00640 1 --2- 172.21.15.40:0/501247508 >> v2:172.21.15.103:3300/0 conn(0x7f2fb80cc480 0x7f2fb8072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.661+0000 7f2fb7400640 1 -- 172.21.15.40:0/501247508 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3077117873 0 0) 0x7f2fb815d7c0 con 0x7f2fb8071e60 2024-04-26T23:07:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.662+0000 7f2fbdc00640 1 --2- 172.21.15.40:0/501247508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb8072780 0x7f2fb8166ad0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80b3820 tx=0x7f2fb002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5dcf9d9c5bde0d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.663+0000 7f2fb7400640 1 -- 172.21.15.40:0/501247508 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0007740 con 0x7f2fb8072780 2024-04-26T23:07:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.663+0000 7f2fb7400640 1 -- 172.21.15.40:0/501247508 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb00078e0 con 0x7f2fb8072780 2024-04-26T23:07:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.663+0000 7f2fb7400640 1 -- 172.21.15.40:0/501247508 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0007cb0 con 0x7f2fb8072780 2024-04-26T23:07:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.663+0000 7f2fbfc00640 1 -- 172.21.15.40:0/501247508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb8072780 msgr2=0x7f2fb8166ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.663+0000 7f2fbfc00640 1 --2- 172.21.15.40:0/501247508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb8072780 0x7f2fb8166ad0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80b3820 tx=0x7f2fb002a680 comp rx=0 tx=0).stop 2024-04-26T23:07:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.663+0000 7f2fbfc00640 1 -- 172.21.15.40:0/501247508 shutdown_connections 2024-04-26T23:07:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.663+0000 7f2fbfc00640 1 --2- 172.21.15.40:0/501247508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb8072780 0x7f2fb8166ad0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.663+0000 7f2fbfc00640 1 --2- 172.21.15.40:0/501247508 >> v2:172.21.15.103:3300/0 conn(0x7f2fb80cc480 0x7f2fb8072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.663+0000 7f2fbfc00640 1 -- 172.21.15.40:0/501247508 >> 172.21.15.40:0/501247508 conn(0x7f2fb80d0be0 msgr2=0x7f2fb80d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.663+0000 7f2fbfc00640 1 -- 172.21.15.40:0/501247508 shutdown_connections 2024-04-26T23:07:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.664+0000 7f2fbfc00640 1 -- 172.21.15.40:0/501247508 wait complete. 2024-04-26T23:07:02.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.665+0000 7f2fbfc00640 1 Processor -- start 2024-04-26T23:07:02.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.665+0000 7f2fbfc00640 1 -- start start 2024-04-26T23:07:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.666+0000 7f2fbfc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fb8071e60 0x7f2fb8154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.666+0000 7f2fbfc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb8072780 0x7f2fb81f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fbfc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb80cc480 0x7f2fb8153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fbfc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb815d3f0 con 0x7f2fb80cc480 2024-04-26T23:07:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fbfc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb8072bd0 con 0x7f2fb8072780 2024-04-26T23:07:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fbfc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb815d570 con 0x7f2fb8071e60 2024-04-26T23:07:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fbdc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb80cc480 0x7f2fb8153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fbdc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb80cc480 0x7f2fb8153c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54382/0 (socket says 172.21.15.40:54382) 2024-04-26T23:07:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fbdc00640 1 -- 172.21.15.40:0/1370422853 learned_addr learned my addr 172.21.15.40:0/1370422853 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:02.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fb7e00640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb8072780 0x7f2fb81f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fbd200640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fb8071e60 0x7f2fb8154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fbdc00640 1 -- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fb8071e60 msgr2=0x7f2fb8154820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fbdc00640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fb8071e60 0x7f2fb8154820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fbdc00640 1 -- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb8072780 msgr2=0x7f2fb81f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fbdc00640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb8072780 0x7f2fb81f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.667+0000 7f2fbdc00640 1 -- 172.21.15.40:0/1370422853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb81f3980 con 0x7f2fb80cc480 2024-04-26T23:07:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.668+0000 7f2fbdc00640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb80cc480 0x7f2fb8153c10 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009e50 tx=0x7f2fb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.668+0000 7f2fb4c00640 1 -- 172.21.15.40:0/1370422853 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0040070 con 0x7f2fb80cc480 2024-04-26T23:07:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.668+0000 7f2fb4c00640 1 -- 172.21.15.40:0/1370422853 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb0031cc0 con 0x7f2fb80cc480 2024-04-26T23:07:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.668+0000 7f2fbfc00640 1 -- 172.21.15.40:0/1370422853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb8154e20 con 0x7f2fb80cc480 2024-04-26T23:07:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.668+0000 7f2fb4c00640 1 -- 172.21.15.40:0/1370422853 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0045680 con 0x7f2fb80cc480 2024-04-26T23:07:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.668+0000 7f2fbfc00640 1 -- 172.21.15.40:0/1370422853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb8153410 con 0x7f2fb80cc480 2024-04-26T23:07:02.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.669+0000 7f2fbfc00640 1 -- 172.21.15.40:0/1370422853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb80694f0 con 0x7f2fb80cc480 2024-04-26T23:07:02.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.670+0000 7f2fb4c00640 1 -- 172.21.15.40:0/1370422853 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2fb0039040 con 0x7f2fb80cc480 2024-04-26T23:07:02.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.671+0000 7f2fb4c00640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f9806c8f0 0x7f2f9806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:02.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.671+0000 7f2fb4c00640 1 -- 172.21.15.40:0/1370422853 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f2fb00ad6e0 con 0x7f2fb80cc480 2024-04-26T23:07:02.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.672+0000 7f2fbd200640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f9806c8f0 0x7f2f9806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:02.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.673+0000 7f2fbd200640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f9806c8f0 0x7f2f9806edb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8000f60 tx=0x7f2fa8007b50 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:02.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.676+0000 7f2fb4c00640 1 -- 172.21.15.40:0/1370422853 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2fb007ba30 con 0x7f2fb80cc480 2024-04-26T23:07:02.774 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:07:02.774+0000 7fdbb2a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T23:07:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.879+0000 7f2fbfc00640 1 -- 172.21.15.40:0/1370422853 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2fb806ddf0 con 0x7f2f9806c8f0 2024-04-26T23:07:02.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.898+0000 7f2fb4c00640 1 -- 172.21.15.40:0/1370422853 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246753 (secure 0 0 0) 0x7f2fb806ddf0 con 0x7f2f9806c8f0 2024-04-26T23:07:02.900 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:07:02.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.904+0000 7f2fbfc00640 1 -- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f9806c8f0 msgr2=0x7f2f9806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:02.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.904+0000 7f2fbfc00640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f9806c8f0 0x7f2f9806edb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8000f60 tx=0x7f2fa8007b50 comp rx=0 tx=0).stop 2024-04-26T23:07:02.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.904+0000 7f2fbfc00640 1 -- 172.21.15.40:0/1370422853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb80cc480 msgr2=0x7f2fb8153c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:02.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.904+0000 7f2fbfc00640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb80cc480 0x7f2fb8153c10 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009e50 tx=0x7f2fb0004a40 comp rx=0 tx=0).stop 2024-04-26T23:07:02.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.905+0000 7f2fbfc00640 1 -- 172.21.15.40:0/1370422853 shutdown_connections 2024-04-26T23:07:02.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.905+0000 7f2fbfc00640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f9806c8f0 0x7f2f9806edb0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.905+0000 7f2fbfc00640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb80cc480 0x7f2fb8153c10 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.905+0000 7f2fbfc00640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb8072780 0x7f2fb81f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.905+0000 7f2fbfc00640 1 --2- 172.21.15.40:0/1370422853 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fb8071e60 0x7f2fb8154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:02.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.905+0000 7f2fbfc00640 1 -- 172.21.15.40:0/1370422853 >> 172.21.15.40:0/1370422853 conn(0x7f2fb80d0be0 msgr2=0x7f2fb80cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:02.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.905+0000 7f2fbfc00640 1 -- 172.21.15.40:0/1370422853 shutdown_connections 2024-04-26T23:07:02.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:02.905+0000 7f2fbfc00640 1 -- 172.21.15.40:0/1370422853 wait complete. 2024-04-26T23:07:02.926 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-04-26T23:07:01.277169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253796,"kb_used_data":8496,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754720924,"statfs":{"total":773094113280,"available":772834226176,"internally_reserved":0,"allocated":8699904,"data_stored":4639466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":251055666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.906908"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.403807+0000","last_change":"2024-04-26T23:06:57.403807+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.403807+0000","last_undegraded":"2024-04-26T23:06:57.403807+0000","last_fullsized":"2024-04-26T23:06:57.403807+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.397033+0000","last_change":"2024-04-26T23:06:57.397033+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.397033+0000","last_undegraded":"2024-04-26T23:06:57.397033+0000","last_fullsized":"2024-04-26T23:06:57.397033+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.388526+0000","last_change":"2024-04-26T23:06:57.388526+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.388526+0000","last_undegraded":"2024-04-26T23:06:57.388526+0000","last_fullsized":"2024-04-26T23:06:57.388526+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.399338+0000","last_change":"2024-04-26T23:06:57.399338+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.399338+0000","last_undegraded":"2024-04-26T23:06:57.399338+0000","last_fullsized":"2024-04-26T23:06:57.399338+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.398132+0000","last_change":"2024-04-26T23:06:57.398132+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.398132+0000","last_undegraded":"2024-04-26T23:06:57.398132+0000","last_fullsized":"2024-04-26T23:06:57.398132+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.398641+0000","last_change":"2024-04-26T23:06:57.398641+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.398641+0000","last_undegraded":"2024-04-26T23:06:57.398641+0000","last_fullsized":"2024-04-26T23:06:57.398641+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.384472+0000","last_change":"2024-04-26T23:06:57.384472+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.384472+0000","last_undegraded":"2024-04-26T23:06:57.384472+0000","last_fullsized":"2024-04-26T23:06:57.384472+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.394050+0000","last_change":"2024-04-26T23:06:57.394050+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.394050+0000","last_undegraded":"2024-04-26T23:06:57.394050+0000","last_fullsized":"2024-04-26T23:06:57.394050+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.390963+0000","last_change":"2024-04-26T23:06:57.390963+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.390963+0000","last_undegraded":"2024-04-26T23:06:57.390963+0000","last_fullsized":"2024-04-26T23:06:57.390963+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.398620+0000","last_change":"2024-04-26T23:06:57.398620+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.398620+0000","last_undegraded":"2024-04-26T23:06:57.398620+0000","last_fullsized":"2024-04-26T23:06:57.398620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.390303+0000","last_change":"2024-04-26T23:06:57.390303+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.390303+0000","last_undegraded":"2024-04-26T23:06:57.390303+0000","last_fullsized":"2024-04-26T23:06:57.390303+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.397825+0000","last_change":"2024-04-26T23:06:57.397825+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.397825+0000","last_undegraded":"2024-04-26T23:06:57.397825+0000","last_fullsized":"2024-04-26T23:06:57.397825+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.397666+0000","last_change":"2024-04-26T23:06:57.397666+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.397666+0000","last_undegraded":"2024-04-26T23:06:57.397666+0000","last_fullsized":"2024-04-26T23:06:57.397666+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.393597+0000","last_change":"2024-04-26T23:06:57.393597+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.393597+0000","last_undegraded":"2024-04-26T23:06:57.393597+0000","last_fullsized":"2024-04-26T23:06:57.393597+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.389999+0000","last_change":"2024-04-26T23:06:57.389999+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.389999+0000","last_undegraded":"2024-04-26T23:06:57.389999+0000","last_fullsized":"2024-04-26T23:06:57.389999+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.401388+0000","last_change":"2024-04-26T23:06:57.401388+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.401388+0000","last_undegraded":"2024-04-26T23:06:57.401388+0000","last_fullsized":"2024-04-26T23:06:57.401388+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.397223+0000","last_change":"2024-04-26T23:06:57.397223+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.397223+0000","last_undegraded":"2024-04-26T23:06:57.397223+0000","last_fullsized":"2024-04-26T23:06:57.397223+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.388862+0000","last_change":"2024-04-26T23:06:57.388862+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.388862+0000","last_undegraded":"2024-04-26T23:06:57.388862+0000","last_fullsized":"2024-04-26T23:06:57.388862+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.394608+0000","last_change":"2024-04-26T23:06:57.394608+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.394608+0000","last_undegraded":"2024-04-26T23:06:57.394608+0000","last_fullsized":"2024-04-26T23:06:57.394608+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.399063+0000","last_change":"2024-04-26T23:06:57.399063+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.399063+0000","last_undegraded":"2024-04-26T23:06:57.399063+0000","last_fullsized":"2024-04-26T23:06:57.399063+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.398398+0000","last_change":"2024-04-26T23:06:57.398398+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.398398+0000","last_undegraded":"2024-04-26T23:06:57.398398+0000","last_fullsized":"2024-04-26T23:06:57.398398+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.397328+0000","last_change":"2024-04-26T23:06:57.397328+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.397328+0000","last_undegraded":"2024-04-26T23:06:57.397328+0000","last_fullsized":"2024-04-26T23:06:57.397328+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.387378+0000","last_change":"2024-04-26T23:06:57.387378+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.387378+0000","last_undegraded":"2024-04-26T23:06:57.387378+0000","last_fullsized":"2024-04-26T23:06:57.387378+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.390170+0000","last_change":"2024-04-26T23:06:57.390170+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.390170+0000","last_undegraded":"2024-04-26T23:06:57.390170+0000","last_fullsized":"2024-04-26T23:06:57.390170+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.391840+0000","last_change":"2024-04-26T23:06:57.391840+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.391840+0000","last_undegraded":"2024-04-26T23:06:57.391840+0000","last_fullsized":"2024-04-26T23:06:57.391840+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.397759+0000","last_change":"2024-04-26T23:06:57.397759+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.397759+0000","last_undegraded":"2024-04-26T23:06:57.397759+0000","last_fullsized":"2024-04-26T23:06:57.397759+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.403073+0000","last_change":"2024-04-26T23:06:57.403073+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.403073+0000","last_undegraded":"2024-04-26T23:06:57.403073+0000","last_fullsized":"2024-04-26T23:06:57.403073+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.398598+0000","last_change":"2024-04-26T23:06:57.398598+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.398598+0000","last_undegraded":"2024-04-26T23:06:57.398598+0000","last_fullsized":"2024-04-26T23:06:57.398598+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.389479+0000","last_change":"2024-04-26T23:06:57.389479+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.389479+0000","last_undegraded":"2024-04-26T23:06:57.389479+0000","last_fullsized":"2024-04-26T23:06:57.389479+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.400250+0000","last_change":"2024-04-26T23:06:57.400250+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.400250+0000","last_undegraded":"2024-04-26T23:06:57.400250+0000","last_fullsized":"2024-04-26T23:06:57.400250+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.388163+0000","last_change":"2024-04-26T23:06:57.388163+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.388163+0000","last_undegraded":"2024-04-26T23:06:57.388163+0000","last_fullsized":"2024-04-26T23:06:57.388163+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.400853+0000","last_change":"2024-04-26T23:06:57.400853+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.400853+0000","last_undegraded":"2024-04-26T23:06:57.400853+0000","last_fullsized":"2024-04-26T23:06:57.400853+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.388086+0000","last_change":"2024-04-26T23:06:57.388086+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.388086+0000","last_undegraded":"2024-04-26T23:06:57.388086+0000","last_fullsized":"2024-04-26T23:06:57.388086+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.417465+0000","last_change":"2024-04-26T23:06:57.412094+0000","last_active":"2024-04-26T23:06:57.417465+0000","last_peered":"2024-04-26T23:06:57.417465+0000","last_clean":"2024-04-26T23:06:57.417465+0000","last_became_active":"2024-04-26T23:06:57.411531+0000","last_became_peered":"2024-04-26T23:06:57.411531+0000","last_unstale":"2024-04-26T23:06:57.417465+0000","last_undegraded":"2024-04-26T23:06:57.417465+0000","last_fullsized":"2024-04-26T23:06:57.417465+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:31:12.181793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.389417+0000","last_change":"2024-04-26T23:06:57.389417+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.389417+0000","last_undegraded":"2024-04-26T23:06:57.389417+0000","last_fullsized":"2024-04-26T23:06:57.389417+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.424711+0000","last_change":"2024-04-26T23:06:57.418786+0000","last_active":"2024-04-26T23:06:57.424711+0000","last_peered":"2024-04-26T23:06:57.424711+0000","last_clean":"2024-04-26T23:06:57.424711+0000","last_became_active":"2024-04-26T23:06:57.418446+0000","last_became_peered":"2024-04-26T23:06:57.418446+0000","last_unstale":"2024-04-26T23:06:57.424711+0000","last_undegraded":"2024-04-26T23:06:57.424711+0000","last_fullsized":"2024-04-26T23:06:57.424711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:29:51.103012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.388694+0000","last_change":"2024-04-26T23:06:57.388694+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.388694+0000","last_undegraded":"2024-04-26T23:06:57.388694+0000","last_fullsized":"2024-04-26T23:06:57.388694+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.417770+0000","last_change":"2024-04-26T23:06:57.413701+0000","last_active":"2024-04-26T23:06:57.417770+0000","last_peered":"2024-04-26T23:06:57.417770+0000","last_clean":"2024-04-26T23:06:57.417770+0000","last_became_active":"2024-04-26T23:06:57.413402+0000","last_became_peered":"2024-04-26T23:06:57.413402+0000","last_unstale":"2024-04-26T23:06:57.417770+0000","last_undegraded":"2024-04-26T23:06:57.417770+0000","last_fullsized":"2024-04-26T23:06:57.417770+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:08.307472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.394365+0000","last_change":"2024-04-26T23:06:57.394365+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.394365+0000","last_undegraded":"2024-04-26T23:06:57.394365+0000","last_fullsized":"2024-04-26T23:06:57.394365+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.417664+0000","last_change":"2024-04-26T23:06:57.412204+0000","last_active":"2024-04-26T23:06:57.417664+0000","last_peered":"2024-04-26T23:06:57.417664+0000","last_clean":"2024-04-26T23:06:57.417664+0000","last_became_active":"2024-04-26T23:06:57.410478+0000","last_became_peered":"2024-04-26T23:06:57.410478+0000","last_unstale":"2024-04-26T23:06:57.417664+0000","last_undegraded":"2024-04-26T23:06:57.417664+0000","last_fullsized":"2024-04-26T23:06:57.417664+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:52.428086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.391462+0000","last_change":"2024-04-26T23:06:57.391462+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.391462+0000","last_undegraded":"2024-04-26T23:06:57.391462+0000","last_fullsized":"2024-04-26T23:06:57.391462+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.411606+0000","last_change":"2024-04-26T23:06:57.408245+0000","last_active":"2024-04-26T23:06:57.411606+0000","last_peered":"2024-04-26T23:06:57.411606+0000","last_clean":"2024-04-26T23:06:57.411606+0000","last_became_active":"2024-04-26T23:06:57.407905+0000","last_became_peered":"2024-04-26T23:06:57.407905+0000","last_unstale":"2024-04-26T23:06:57.411606+0000","last_undegraded":"2024-04-26T23:06:57.411606+0000","last_fullsized":"2024-04-26T23:06:57.411606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:02:55.277548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.389507+0000","last_change":"2024-04-26T23:06:57.389507+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.389507+0000","last_undegraded":"2024-04-26T23:06:57.389507+0000","last_fullsized":"2024-04-26T23:06:57.389507+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.410079+0000","last_change":"2024-04-26T23:06:57.407272+0000","last_active":"2024-04-26T23:06:57.410079+0000","last_peered":"2024-04-26T23:06:57.410079+0000","last_clean":"2024-04-26T23:06:57.410079+0000","last_became_active":"2024-04-26T23:06:57.406963+0000","last_became_peered":"2024-04-26T23:06:57.406963+0000","last_unstale":"2024-04-26T23:06:57.410079+0000","last_undegraded":"2024-04-26T23:06:57.410079+0000","last_fullsized":"2024-04-26T23:06:57.410079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:42:11.099238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.390596+0000","last_change":"2024-04-26T23:06:57.390596+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.390596+0000","last_undegraded":"2024-04-26T23:06:57.390596+0000","last_fullsized":"2024-04-26T23:06:57.390596+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.408314+0000","last_change":"2024-04-26T23:06:57.405640+0000","last_active":"2024-04-26T23:06:57.408314+0000","last_peered":"2024-04-26T23:06:57.408314+0000","last_clean":"2024-04-26T23:06:57.408314+0000","last_became_active":"2024-04-26T23:06:57.405336+0000","last_became_peered":"2024-04-26T23:06:57.405336+0000","last_unstale":"2024-04-26T23:06:57.408314+0000","last_undegraded":"2024-04-26T23:06:57.408314+0000","last_fullsized":"2024-04-26T23:06:57.408314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:45:47.218851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.387087+0000","last_change":"2024-04-26T23:06:57.387087+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.387087+0000","last_undegraded":"2024-04-26T23:06:57.387087+0000","last_fullsized":"2024-04-26T23:06:57.387087+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.417533+0000","last_change":"2024-04-26T23:06:57.412429+0000","last_active":"2024-04-26T23:06:57.417533+0000","last_peered":"2024-04-26T23:06:57.417533+0000","last_clean":"2024-04-26T23:06:57.417533+0000","last_became_active":"2024-04-26T23:06:57.411673+0000","last_became_peered":"2024-04-26T23:06:57.411673+0000","last_unstale":"2024-04-26T23:06:57.417533+0000","last_undegraded":"2024-04-26T23:06:57.417533+0000","last_fullsized":"2024-04-26T23:06:57.417533+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:39:56.259069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.389570+0000","last_change":"2024-04-26T23:06:57.389570+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.389570+0000","last_undegraded":"2024-04-26T23:06:57.389570+0000","last_fullsized":"2024-04-26T23:06:57.389570+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.424737+0000","last_change":"2024-04-26T23:06:57.418763+0000","last_active":"2024-04-26T23:06:57.424737+0000","last_peered":"2024-04-26T23:06:57.424737+0000","last_clean":"2024-04-26T23:06:57.424737+0000","last_became_active":"2024-04-26T23:06:57.418441+0000","last_became_peered":"2024-04-26T23:06:57.418441+0000","last_unstale":"2024-04-26T23:06:57.424737+0000","last_undegraded":"2024-04-26T23:06:57.424737+0000","last_fullsized":"2024-04-26T23:06:57.424737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:51.250935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.394752+0000","last_change":"2024-04-26T23:06:57.394752+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.394752+0000","last_undegraded":"2024-04-26T23:06:57.394752+0000","last_fullsized":"2024-04-26T23:06:57.394752+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.417544+0000","last_change":"2024-04-26T23:06:57.413101+0000","last_active":"2024-04-26T23:06:57.417544+0000","last_peered":"2024-04-26T23:06:57.417544+0000","last_clean":"2024-04-26T23:06:57.417544+0000","last_became_active":"2024-04-26T23:06:57.412770+0000","last_became_peered":"2024-04-26T23:06:57.412770+0000","last_unstale":"2024-04-26T23:06:57.417544+0000","last_undegraded":"2024-04-26T23:06:57.417544+0000","last_fullsized":"2024-04-26T23:06:57.417544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:08.307472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.393070+0000","last_change":"2024-04-26T23:06:57.393070+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.393070+0000","last_undegraded":"2024-04-26T23:06:57.393070+0000","last_fullsized":"2024-04-26T23:06:57.393070+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.415761+0000","last_change":"2024-04-26T23:06:57.409856+0000","last_active":"2024-04-26T23:06:57.415761+0000","last_peered":"2024-04-26T23:06:57.415761+0000","last_clean":"2024-04-26T23:06:57.415761+0000","last_became_active":"2024-04-26T23:06:57.409578+0000","last_became_peered":"2024-04-26T23:06:57.409578+0000","last_unstale":"2024-04-26T23:06:57.415761+0000","last_undegraded":"2024-04-26T23:06:57.415761+0000","last_fullsized":"2024-04-26T23:06:57.415761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:35:33.692265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.388674+0000","last_change":"2024-04-26T23:06:57.388674+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.388674+0000","last_undegraded":"2024-04-26T23:06:57.388674+0000","last_fullsized":"2024-04-26T23:06:57.388674+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.403111+0000","last_change":"2024-04-26T23:06:57.399434+0000","last_active":"2024-04-26T23:06:57.403111+0000","last_peered":"2024-04-26T23:06:57.403111+0000","last_clean":"2024-04-26T23:06:57.403111+0000","last_became_active":"2024-04-26T23:06:57.399100+0000","last_became_peered":"2024-04-26T23:06:57.399100+0000","last_unstale":"2024-04-26T23:06:57.403111+0000","last_undegraded":"2024-04-26T23:06:57.403111+0000","last_fullsized":"2024-04-26T23:06:57.403111+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:14:50.064153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.397291+0000","last_change":"2024-04-26T23:06:57.397291+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.397291+0000","last_undegraded":"2024-04-26T23:06:57.397291+0000","last_fullsized":"2024-04-26T23:06:57.397291+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.417550+0000","last_change":"2024-04-26T23:06:57.412138+0000","last_active":"2024-04-26T23:06:57.417550+0000","last_peered":"2024-04-26T23:06:57.417550+0000","last_clean":"2024-04-26T23:06:57.417550+0000","last_became_active":"2024-04-26T23:06:57.411299+0000","last_became_peered":"2024-04-26T23:06:57.411299+0000","last_unstale":"2024-04-26T23:06:57.417550+0000","last_undegraded":"2024-04-26T23:06:57.417550+0000","last_fullsized":"2024-04-26T23:06:57.417550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:27:56.624319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.389617+0000","last_change":"2024-04-26T23:06:57.389617+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.389617+0000","last_undegraded":"2024-04-26T23:06:57.389617+0000","last_fullsized":"2024-04-26T23:06:57.389617+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.419661+0000","last_change":"2024-04-26T23:06:57.416632+0000","last_active":"2024-04-26T23:06:57.419661+0000","last_peered":"2024-04-26T23:06:57.419661+0000","last_clean":"2024-04-26T23:06:57.419661+0000","last_became_active":"2024-04-26T23:06:57.416299+0000","last_became_peered":"2024-04-26T23:06:57.416299+0000","last_unstale":"2024-04-26T23:06:57.419661+0000","last_undegraded":"2024-04-26T23:06:57.419661+0000","last_fullsized":"2024-04-26T23:06:57.419661+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:38.258327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.383408+0000","last_change":"2024-04-26T23:06:57.383408+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.383408+0000","last_undegraded":"2024-04-26T23:06:57.383408+0000","last_fullsized":"2024-04-26T23:06:57.383408+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.417596+0000","last_change":"2024-04-26T23:06:57.414417+0000","last_active":"2024-04-26T23:06:57.417596+0000","last_peered":"2024-04-26T23:06:57.417596+0000","last_clean":"2024-04-26T23:06:57.417596+0000","last_became_active":"2024-04-26T23:06:57.412565+0000","last_became_peered":"2024-04-26T23:06:57.412565+0000","last_unstale":"2024-04-26T23:06:57.417596+0000","last_undegraded":"2024-04-26T23:06:57.417596+0000","last_fullsized":"2024-04-26T23:06:57.417596+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:55.929235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.384945+0000","last_change":"2024-04-26T23:06:57.384945+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.384945+0000","last_undegraded":"2024-04-26T23:06:57.384945+0000","last_fullsized":"2024-04-26T23:06:57.384945+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.411021+0000","last_change":"2024-04-26T23:06:57.403621+0000","last_active":"2024-04-26T23:06:57.411021+0000","last_peered":"2024-04-26T23:06:57.411021+0000","last_clean":"2024-04-26T23:06:57.411021+0000","last_became_active":"2024-04-26T23:06:57.403306+0000","last_became_peered":"2024-04-26T23:06:57.403306+0000","last_unstale":"2024-04-26T23:06:57.411021+0000","last_undegraded":"2024-04-26T23:06:57.411021+0000","last_fullsized":"2024-04-26T23:06:57.411021+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:28:08.367994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.391268+0000","last_change":"2024-04-26T23:06:57.391268+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.391268+0000","last_undegraded":"2024-04-26T23:06:57.391268+0000","last_fullsized":"2024-04-26T23:06:57.391268+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.424723+0000","last_change":"2024-04-26T23:06:57.418000+0000","last_active":"2024-04-26T23:06:57.424723+0000","last_peered":"2024-04-26T23:06:57.424723+0000","last_clean":"2024-04-26T23:06:57.424723+0000","last_became_active":"2024-04-26T23:06:57.417711+0000","last_became_peered":"2024-04-26T23:06:57.417711+0000","last_unstale":"2024-04-26T23:06:57.424723+0000","last_undegraded":"2024-04-26T23:06:57.424723+0000","last_fullsized":"2024-04-26T23:06:57.424723+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:27.331965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.400380+0000","last_change":"2024-04-26T23:06:57.400380+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.400380+0000","last_undegraded":"2024-04-26T23:06:57.400380+0000","last_fullsized":"2024-04-26T23:06:57.400380+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.410127+0000","last_change":"2024-04-26T23:06:57.405040+0000","last_active":"2024-04-26T23:06:57.410127+0000","last_peered":"2024-04-26T23:06:57.410127+0000","last_clean":"2024-04-26T23:06:57.410127+0000","last_became_active":"2024-04-26T23:06:57.404745+0000","last_became_peered":"2024-04-26T23:06:57.404745+0000","last_unstale":"2024-04-26T23:06:57.410127+0000","last_undegraded":"2024-04-26T23:06:57.410127+0000","last_fullsized":"2024-04-26T23:06:57.410127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:23.878330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.400730+0000","last_change":"2024-04-26T23:06:57.400730+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.400730+0000","last_undegraded":"2024-04-26T23:06:57.400730+0000","last_fullsized":"2024-04-26T23:06:57.400730+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.417614+0000","last_change":"2024-04-26T23:06:57.412631+0000","last_active":"2024-04-26T23:06:57.417614+0000","last_peered":"2024-04-26T23:06:57.417614+0000","last_clean":"2024-04-26T23:06:57.417614+0000","last_became_active":"2024-04-26T23:06:57.412310+0000","last_became_peered":"2024-04-26T23:06:57.412310+0000","last_unstale":"2024-04-26T23:06:57.417614+0000","last_undegraded":"2024-04-26T23:06:57.417614+0000","last_fullsized":"2024-04-26T23:06:57.417614+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:23:03.069184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.394799+0000","last_change":"2024-04-26T23:06:57.394799+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.394799+0000","last_undegraded":"2024-04-26T23:06:57.394799+0000","last_fullsized":"2024-04-26T23:06:57.394799+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.412937+0000","last_change":"2024-04-26T23:06:57.408116+0000","last_active":"2024-04-26T23:06:57.412937+0000","last_peered":"2024-04-26T23:06:57.412937+0000","last_clean":"2024-04-26T23:06:57.412937+0000","last_became_active":"2024-04-26T23:06:57.407806+0000","last_became_peered":"2024-04-26T23:06:57.407806+0000","last_unstale":"2024-04-26T23:06:57.412937+0000","last_undegraded":"2024-04-26T23:06:57.412937+0000","last_fullsized":"2024-04-26T23:06:57.412937+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:21.777197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.390748+0000","last_change":"2024-04-26T23:06:57.390748+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.390748+0000","last_undegraded":"2024-04-26T23:06:57.390748+0000","last_fullsized":"2024-04-26T23:06:57.390748+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.414374+0000","last_change":"2024-04-26T23:06:57.410225+0000","last_active":"2024-04-26T23:06:57.414374+0000","last_peered":"2024-04-26T23:06:57.414374+0000","last_clean":"2024-04-26T23:06:57.414374+0000","last_became_active":"2024-04-26T23:06:57.409931+0000","last_became_peered":"2024-04-26T23:06:57.409931+0000","last_unstale":"2024-04-26T23:06:57.414374+0000","last_undegraded":"2024-04-26T23:06:57.414374+0000","last_fullsized":"2024-04-26T23:06:57.414374+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:23.619843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.391827+0000","last_change":"2024-04-26T23:06:57.391827+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.391827+0000","last_undegraded":"2024-04-26T23:06:57.391827+0000","last_fullsized":"2024-04-26T23:06:57.391827+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.414484+0000","last_change":"2024-04-26T23:06:57.411645+0000","last_active":"2024-04-26T23:06:57.414484+0000","last_peered":"2024-04-26T23:06:57.414484+0000","last_clean":"2024-04-26T23:06:57.414484+0000","last_became_active":"2024-04-26T23:06:57.411360+0000","last_became_peered":"2024-04-26T23:06:57.411360+0000","last_unstale":"2024-04-26T23:06:57.414484+0000","last_undegraded":"2024-04-26T23:06:57.414484+0000","last_fullsized":"2024-04-26T23:06:57.414484+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:23:42.814680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.383505+0000","last_change":"2024-04-26T23:06:57.383505+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.383505+0000","last_undegraded":"2024-04-26T23:06:57.383505+0000","last_fullsized":"2024-04-26T23:06:57.383505+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.409159+0000","last_change":"2024-04-26T23:06:57.404872+0000","last_active":"2024-04-26T23:06:57.409159+0000","last_peered":"2024-04-26T23:06:57.409159+0000","last_clean":"2024-04-26T23:06:57.409159+0000","last_became_active":"2024-04-26T23:06:57.404587+0000","last_became_peered":"2024-04-26T23:06:57.404587+0000","last_unstale":"2024-04-26T23:06:57.409159+0000","last_undegraded":"2024-04-26T23:06:57.409159+0000","last_fullsized":"2024-04-26T23:06:57.409159+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:46.739074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.384810+0000","last_change":"2024-04-26T23:06:57.384810+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.384810+0000","last_undegraded":"2024-04-26T23:06:57.384810+0000","last_fullsized":"2024-04-26T23:06:57.384810+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.408937+0000","last_change":"2024-04-26T23:06:57.404490+0000","last_active":"2024-04-26T23:06:57.408937+0000","last_peered":"2024-04-26T23:06:57.408937+0000","last_clean":"2024-04-26T23:06:57.408937+0000","last_became_active":"2024-04-26T23:06:57.404076+0000","last_became_peered":"2024-04-26T23:06:57.404076+0000","last_unstale":"2024-04-26T23:06:57.408937+0000","last_undegraded":"2024-04-26T23:06:57.408937+0000","last_fullsized":"2024-04-26T23:06:57.408937+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:17:48.160745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.383073+0000","last_change":"2024-04-26T23:06:57.383073+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.383073+0000","last_undegraded":"2024-04-26T23:06:57.383073+0000","last_fullsized":"2024-04-26T23:06:57.383073+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.408339+0000","last_change":"2024-04-26T23:06:57.403494+0000","last_active":"2024-04-26T23:06:57.408339+0000","last_peered":"2024-04-26T23:06:57.408339+0000","last_clean":"2024-04-26T23:06:57.408339+0000","last_became_active":"2024-04-26T23:06:57.403173+0000","last_became_peered":"2024-04-26T23:06:57.403173+0000","last_unstale":"2024-04-26T23:06:57.408339+0000","last_undegraded":"2024-04-26T23:06:57.408339+0000","last_fullsized":"2024-04-26T23:06:57.408339+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:29:51.103012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.387585+0000","last_change":"2024-04-26T23:06:57.387585+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.387585+0000","last_undegraded":"2024-04-26T23:06:57.387585+0000","last_fullsized":"2024-04-26T23:06:57.387585+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.417933+0000","last_change":"2024-04-26T23:06:57.416052+0000","last_active":"2024-04-26T23:06:57.417933+0000","last_peered":"2024-04-26T23:06:57.417933+0000","last_clean":"2024-04-26T23:06:57.417933+0000","last_became_active":"2024-04-26T23:06:57.415704+0000","last_became_peered":"2024-04-26T23:06:57.415704+0000","last_unstale":"2024-04-26T23:06:57.417933+0000","last_undegraded":"2024-04-26T23:06:57.417933+0000","last_fullsized":"2024-04-26T23:06:57.417933+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:11.112333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.388604+0000","last_change":"2024-04-26T23:06:57.388604+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.388604+0000","last_undegraded":"2024-04-26T23:06:57.388604+0000","last_fullsized":"2024-04-26T23:06:57.388604+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.417897+0000","last_change":"2024-04-26T23:06:57.415604+0000","last_active":"2024-04-26T23:06:57.417897+0000","last_peered":"2024-04-26T23:06:57.417897+0000","last_clean":"2024-04-26T23:06:57.417897+0000","last_became_active":"2024-04-26T23:06:57.415245+0000","last_became_peered":"2024-04-26T23:06:57.415245+0000","last_unstale":"2024-04-26T23:06:57.417897+0000","last_undegraded":"2024-04-26T23:06:57.417897+0000","last_fullsized":"2024-04-26T23:06:57.417897+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:23.635701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.384804+0000","last_change":"2024-04-26T23:06:57.384804+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.384804+0000","last_undegraded":"2024-04-26T23:06:57.384804+0000","last_fullsized":"2024-04-26T23:06:57.384804+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.417871+0000","last_change":"2024-04-26T23:06:57.410530+0000","last_active":"2024-04-26T23:06:57.417871+0000","last_peered":"2024-04-26T23:06:57.417871+0000","last_clean":"2024-04-26T23:06:57.417871+0000","last_became_active":"2024-04-26T23:06:57.410240+0000","last_became_peered":"2024-04-26T23:06:57.410240+0000","last_unstale":"2024-04-26T23:06:57.417871+0000","last_undegraded":"2024-04-26T23:06:57.417871+0000","last_fullsized":"2024-04-26T23:06:57.417871+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:18.348296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.385725+0000","last_change":"2024-04-26T23:06:57.385725+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.385725+0000","last_undegraded":"2024-04-26T23:06:57.385725+0000","last_fullsized":"2024-04-26T23:06:57.385725+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.417560+0000","last_change":"2024-04-26T23:06:57.413970+0000","last_active":"2024-04-26T23:06:57.417560+0000","last_peered":"2024-04-26T23:06:57.417560+0000","last_clean":"2024-04-26T23:06:57.417560+0000","last_became_active":"2024-04-26T23:06:57.412222+0000","last_became_peered":"2024-04-26T23:06:57.412222+0000","last_unstale":"2024-04-26T23:06:57.417560+0000","last_undegraded":"2024-04-26T23:06:57.417560+0000","last_fullsized":"2024-04-26T23:06:57.417560+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:56:23.084648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.386772+0000","last_change":"2024-04-26T23:06:57.386772+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.386772+0000","last_undegraded":"2024-04-26T23:06:57.386772+0000","last_fullsized":"2024-04-26T23:06:57.386772+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.417459+0000","last_change":"2024-04-26T23:06:57.411122+0000","last_active":"2024-04-26T23:06:57.417459+0000","last_peered":"2024-04-26T23:06:57.417459+0000","last_clean":"2024-04-26T23:06:57.417459+0000","last_became_active":"2024-04-26T23:06:57.410137+0000","last_became_peered":"2024-04-26T23:06:57.410137+0000","last_unstale":"2024-04-26T23:06:57.417459+0000","last_undegraded":"2024-04-26T23:06:57.417459+0000","last_fullsized":"2024-04-26T23:06:57.417459+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:24.211004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.388383+0000","last_change":"2024-04-26T23:06:57.388383+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.388383+0000","last_undegraded":"2024-04-26T23:06:57.388383+0000","last_fullsized":"2024-04-26T23:06:57.388383+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.409256+0000","last_change":"2024-04-26T23:06:57.406278+0000","last_active":"2024-04-26T23:06:57.409256+0000","last_peered":"2024-04-26T23:06:57.409256+0000","last_clean":"2024-04-26T23:06:57.409256+0000","last_became_active":"2024-04-26T23:06:57.405984+0000","last_became_peered":"2024-04-26T23:06:57.405984+0000","last_unstale":"2024-04-26T23:06:57.409256+0000","last_undegraded":"2024-04-26T23:06:57.409256+0000","last_fullsized":"2024-04-26T23:06:57.409256+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:56.039206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-26T23:06:57.402438+0000","last_change":"2024-04-26T23:06:57.402438+0000","last_active":"2024-04-26T23:06:57.367271+0000","last_peered":"2024-04-26T23:06:57.367271+0000","last_clean":"2024-04-26T23:06:57.367271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:06:57.402438+0000","last_undegraded":"2024-04-26T23:06:57.402438+0000","last_fullsized":"2024-04-26T23:06:57.402438+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.409498+0000","last_change":"2024-04-26T23:06:57.405613+0000","last_active":"2024-04-26T23:06:57.409498+0000","last_peered":"2024-04-26T23:06:57.409498+0000","last_clean":"2024-04-26T23:06:57.409498+0000","last_became_active":"2024-04-26T23:06:57.405175+0000","last_became_peered":"2024-04-26T23:06:57.405175+0000","last_unstale":"2024-04-26T23:06:57.409498+0000","last_undegraded":"2024-04-26T23:06:57.409498+0000","last_fullsized":"2024-04-26T23:06:57.409498+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:13.911421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":279,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-26T23:06:57.407643+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:06:57.407643+0000","last_peered":"2024-04-26T23:06:57.407643+0000","last_clean":"2024-04-26T23:06:57.407643+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:06:57.407643+0000","last_undegraded":"2024-04-26T23:06:57.407643+0000","last_fullsized":"2024-04-26T23:06:57.407643+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1536000,"data_stored":1024376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574923,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":942080,"data_stored":434008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574923,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":427341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":942080,"data_stored":434008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574923,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":942080,"data_stored":434008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":929792,"data_stored":427341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":434008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574924,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1536000,"data_stored":1024376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:07:02.930 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:07:03.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.183+0000 7fc305000640 1 Processor -- start 2024-04-26T23:07:03.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.183+0000 7fc305000640 1 -- start start 2024-04-26T23:07:03.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.185+0000 7fc305000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc30015e8c0 0x7fc3000ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.185+0000 7fc305000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000cccc0 0x7fc30015ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.185+0000 7fc305000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3000ced50 con 0x7fc30006e5d0 2024-04-26T23:07:03.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.185+0000 7fc305000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc3000ce5e0 con 0x7fc30015e8c0 2024-04-26T23:07:03.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.185+0000 7fc305000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc3000cf530 con 0x7fc3000cccc0 2024-04-26T23:07:03.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.185+0000 7fc2ffe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc30006e5d0 0x7fc3000c62c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59472/0 (socket says 172.21.15.40:59472) 2024-04-26T23:07:03.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.185+0000 7fc2ffe00640 1 -- 172.21.15.40:0/3170972885 learned_addr learned my addr 172.21.15.40:0/3170972885 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:03.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.185+0000 7fc2ff400640 1 --2- 172.21.15.40:0/3170972885 >> v2:172.21.15.103:3300/0 conn(0x7fc30015e8c0 0x7fc3000ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.185+0000 7fc2f7e00640 1 --2- 172.21.15.40:0/3170972885 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000cccc0 0x7fc30015ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1363519277 0 0) 0x7fc3000ced50 con 0x7fc30006e5d0 2024-04-26T23:07:03.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2e0003520 con 0x7fc30006e5d0 2024-04-26T23:07:03.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1674784645 0 0) 0x7fc2e0003520 con 0x7fc30006e5d0 2024-04-26T23:07:03.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000cccc0 msgr2=0x7fc30015ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:03.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 --2- 172.21.15.40:0/3170972885 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000cccc0 0x7fc30015ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 >> v2:172.21.15.103:3300/0 conn(0x7fc30015e8c0 msgr2=0x7fc3000ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:03.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 --2- 172.21.15.40:0/3170972885 >> v2:172.21.15.103:3300/0 conn(0x7fc30015e8c0 0x7fc3000ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3000c7490 con 0x7fc30006e5d0 2024-04-26T23:07:03.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fc2f80019f0 con 0x7fc30006e5d0 2024-04-26T23:07:03.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 >> v1:172.21.15.40:6789/0 conn(0x7fc30006e5d0 legacy=0x7fc3000c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:03.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 --2- 172.21.15.40:0/3170972885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2e00039b0 0x7fc2e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 --2- 172.21.15.40:0/3170972885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2e0004800 0x7fc2e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 --2- 172.21.15.40:0/3170972885 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000cccc0 0x7fc2e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc3000ce5e0 con 0x7fc2e00039b0 2024-04-26T23:07:03.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc3000cf530 con 0x7fc2e0004800 2024-04-26T23:07:03.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.186+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc2e000b9c0 con 0x7fc3000cccc0 2024-04-26T23:07:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.187+0000 7fc2ff400640 1 --2- 172.21.15.40:0/3170972885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2e00039b0 0x7fc2e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.187+0000 7fc2f7e00640 1 --2- 172.21.15.40:0/3170972885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2e0004800 0x7fc2e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.187+0000 7fc2ffe00640 1 --2- 172.21.15.40:0/3170972885 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000cccc0 0x7fc2e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.187+0000 7fc2ff400640 1 -- 172.21.15.40:0/3170972885 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000cccc0 msgr2=0x7fc2e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.187+0000 7fc2ff400640 1 --2- 172.21.15.40:0/3170972885 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000cccc0 0x7fc2e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.187+0000 7fc2ff400640 1 -- 172.21.15.40:0/3170972885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2e0004800 msgr2=0x7fc2e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.187+0000 7fc2ff400640 1 --2- 172.21.15.40:0/3170972885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2e0004800 0x7fc2e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.187+0000 7fc2ff400640 1 -- 172.21.15.40:0/3170972885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e0003520 con 0x7fc2e00039b0 2024-04-26T23:07:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.187+0000 7fc2ffe00640 1 --2- 172.21.15.40:0/3170972885 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000cccc0 0x7fc2e00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.188+0000 7fc2ff400640 1 --2- 172.21.15.40:0/3170972885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2e00039b0 0x7fc2e0003dd0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009af0 tx=0x7fc2e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eeaf2802a044933 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.188+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8036070 con 0x7fc2e00039b0 2024-04-26T23:07:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.188+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e8031400 con 0x7fc2e00039b0 2024-04-26T23:07:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.188+0000 7fc2fea00640 1 -- 172.21.15.40:0/3170972885 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8009690 con 0x7fc2e00039b0 2024-04-26T23:07:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.189+0000 7fc305000640 1 -- 172.21.15.40:0/3170972885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2e00039b0 msgr2=0x7fc2e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.189+0000 7fc305000640 1 --2- 172.21.15.40:0/3170972885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2e00039b0 0x7fc2e0003dd0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009af0 tx=0x7fc2e0005830 comp rx=0 tx=0).stop 2024-04-26T23:07:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.189+0000 7fc2ffe00640 1 -- 172.21.15.40:0/3170972885 reap_dead start 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.189+0000 7fc305000640 1 -- 172.21.15.40:0/3170972885 shutdown_connections 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.189+0000 7fc305000640 1 -- 172.21.15.40:0/3170972885 >> 172.21.15.40:0/3170972885 conn(0x7fc3000d0a10 msgr2=0x7fc3000d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.189+0000 7fc305000640 1 -- 172.21.15.40:0/3170972885 shutdown_connections 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.189+0000 7fc305000640 1 -- 172.21.15.40:0/3170972885 wait complete. 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.191+0000 7fc305000640 1 Processor -- start 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.191+0000 7fc305000640 1 -- start start 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.192+0000 7fc305000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2e00039b0 0x7fc30015b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.192+0000 7fc305000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc30015b6a0 0x7fc3000c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.192+0000 7fc305000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000c81d0 0x7fc3000c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc305000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc30015eca0 con 0x7fc30015b6a0 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc305000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc3000ce830 con 0x7fc2e00039b0 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc305000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc3000c69d0 con 0x7fc3000c81d0 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc2f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc30015b6a0 0x7fc3000c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc2f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc30015b6a0 0x7fc3000c7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54396/0 (socket says 172.21.15.40:54396) 2024-04-26T23:07:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc2f7e00640 1 -- 172.21.15.40:0/280204748 learned_addr learned my addr 172.21.15.40:0/280204748 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc2ff400640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2e00039b0 0x7fc30015b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc2ffe00640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000c81d0 0x7fc3000c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc2f7e00640 1 -- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000c81d0 msgr2=0x7fc3000c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc2f7e00640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000c81d0 0x7fc3000c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc2f7e00640 1 -- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2e00039b0 msgr2=0x7fc30015b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc2f7e00640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2e00039b0 0x7fc30015b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc2f7e00640 1 -- 172.21.15.40:0/280204748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3000b5580 con 0x7fc30015b6a0 2024-04-26T23:07:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.193+0000 7fc2ff400640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2e00039b0 0x7fc30015b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:07:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.194+0000 7fc2f7e00640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc30015b6a0 0x7fc3000c7c90 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0011600 tx=0x7fc2f0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.194+0000 7fc2f7400640 1 -- 172.21.15.40:0/280204748 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f000f920 con 0x7fc30015b6a0 2024-04-26T23:07:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.194+0000 7fc2f7400640 1 -- 172.21.15.40:0/280204748 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f000fac0 con 0x7fc30015b6a0 2024-04-26T23:07:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.194+0000 7fc2f7400640 1 -- 172.21.15.40:0/280204748 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f000fe70 con 0x7fc30015b6a0 2024-04-26T23:07:03.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.194+0000 7fc305000640 1 -- 172.21.15.40:0/280204748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3000b5860 con 0x7fc30015b6a0 2024-04-26T23:07:03.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.194+0000 7fc305000640 1 -- 172.21.15.40:0/280204748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3000b5e20 con 0x7fc30015b6a0 2024-04-26T23:07:03.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.195+0000 7fc305000640 1 -- 172.21.15.40:0/280204748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc300069370 con 0x7fc30015b6a0 2024-04-26T23:07:03.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.197+0000 7fc2f7400640 1 -- 172.21.15.40:0/280204748 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc2f0015460 con 0x7fc30015b6a0 2024-04-26T23:07:03.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.197+0000 7fc2f7400640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc2dc06c920 0x7fc2dc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.198+0000 7fc2f7400640 1 -- 172.21.15.40:0/280204748 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fc2f008c700 con 0x7fc30015b6a0 2024-04-26T23:07:03.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.198+0000 7fc2ff400640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc2dc06c920 0x7fc2dc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.199+0000 7fc2ff400640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc2dc06c920 0x7fc2dc06ede0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8033000 tx=0x7fc2e8008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:03.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.204+0000 7fc2f7400640 1 -- 172.21.15.40:0/280204748 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc2f0019030 con 0x7fc30015b6a0 2024-04-26T23:07:03.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.407+0000 7fc305000640 1 -- 172.21.15.40:0/280204748 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc30015af40 con 0x7fc2dc06c920 2024-04-26T23:07:03.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.426+0000 7fc2f7400640 1 -- 172.21.15.40:0/280204748 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252073 (secure 0 0 0) 0x7fc30015af40 con 0x7fc2dc06c920 2024-04-26T23:07:03.430 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:03.431 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:07:03.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.431+0000 7fc305000640 1 -- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc2dc06c920 msgr2=0x7fc2dc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:03.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.431+0000 7fc305000640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc2dc06c920 0x7fc2dc06ede0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8033000 tx=0x7fc2e8008030 comp rx=0 tx=0).stop 2024-04-26T23:07:03.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.431+0000 7fc305000640 1 -- 172.21.15.40:0/280204748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc30015b6a0 msgr2=0x7fc3000c7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:03.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.431+0000 7fc305000640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc30015b6a0 0x7fc3000c7c90 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0011600 tx=0x7fc2f0011ac0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.431+0000 7fc305000640 1 -- 172.21.15.40:0/280204748 shutdown_connections 2024-04-26T23:07:03.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.431+0000 7fc305000640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc2dc06c920 0x7fc2dc06ede0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.431+0000 7fc305000640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc3000c81d0 0x7fc3000c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.431+0000 7fc305000640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc30015b6a0 0x7fc3000c7c90 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.431+0000 7fc305000640 1 --2- 172.21.15.40:0/280204748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2e00039b0 0x7fc30015b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.431+0000 7fc305000640 1 -- 172.21.15.40:0/280204748 >> 172.21.15.40:0/280204748 conn(0x7fc3000d0a10 msgr2=0x7fc3000c58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:03.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.432+0000 7fc305000640 1 -- 172.21.15.40:0/280204748 shutdown_connections 2024-04-26T23:07:03.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.432+0000 7fc305000640 1 -- 172.21.15.40:0/280204748 wait complete. 2024-04-26T23:07:03.453 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-04-26T23:07:03.277937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254072,"kb_used_data":8776,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754720648,"statfs":{"total":773094113280,"available":772833943552,"internally_reserved":0,"allocated":8986624,"data_stored":4768492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":251055666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002126"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.174806+0000","last_change":"2024-04-26T23:06:58.458218+0000","last_active":"2024-04-26T23:07:00.174806+0000","last_peered":"2024-04-26T23:07:00.174806+0000","last_clean":"2024-04-26T23:07:00.174806+0000","last_became_active":"2024-04-26T23:06:58.457912+0000","last_became_peered":"2024-04-26T23:06:58.457912+0000","last_unstale":"2024-04-26T23:07:00.174806+0000","last_undegraded":"2024-04-26T23:07:00.174806+0000","last_fullsized":"2024-04-26T23:07:00.174806+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:56:16.271258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.421910+0000","last_change":"2024-04-26T23:06:58.441078+0000","last_active":"2024-04-26T23:07:00.421910+0000","last_peered":"2024-04-26T23:07:00.421910+0000","last_clean":"2024-04-26T23:07:00.421910+0000","last_became_active":"2024-04-26T23:06:58.439036+0000","last_became_peered":"2024-04-26T23:06:58.439036+0000","last_unstale":"2024-04-26T23:07:00.421910+0000","last_undegraded":"2024-04-26T23:07:00.421910+0000","last_fullsized":"2024-04-26T23:07:00.421910+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:12:26.626482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.439052+0000","last_change":"2024-04-26T23:06:58.430886+0000","last_active":"2024-04-26T23:06:59.439052+0000","last_peered":"2024-04-26T23:06:59.439052+0000","last_clean":"2024-04-26T23:06:59.439052+0000","last_became_active":"2024-04-26T23:06:58.430377+0000","last_became_peered":"2024-04-26T23:06:58.430377+0000","last_unstale":"2024-04-26T23:06:59.439052+0000","last_undegraded":"2024-04-26T23:06:59.439052+0000","last_fullsized":"2024-04-26T23:06:59.439052+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:29:52.117783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.510664+0000","last_change":"2024-04-26T23:06:58.455535+0000","last_active":"2024-04-26T23:07:00.510664+0000","last_peered":"2024-04-26T23:07:00.510664+0000","last_clean":"2024-04-26T23:07:00.510664+0000","last_became_active":"2024-04-26T23:06:58.454803+0000","last_became_peered":"2024-04-26T23:06:58.454803+0000","last_unstale":"2024-04-26T23:07:00.510664+0000","last_undegraded":"2024-04-26T23:07:00.510664+0000","last_fullsized":"2024-04-26T23:07:00.510664+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:28:57.851062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.043814+0000","last_change":"2024-04-26T23:06:58.450060+0000","last_active":"2024-04-26T23:07:00.043814+0000","last_peered":"2024-04-26T23:07:00.043814+0000","last_clean":"2024-04-26T23:07:00.043814+0000","last_became_active":"2024-04-26T23:06:58.449733+0000","last_became_peered":"2024-04-26T23:06:58.449733+0000","last_unstale":"2024-04-26T23:07:00.043814+0000","last_undegraded":"2024-04-26T23:07:00.043814+0000","last_fullsized":"2024-04-26T23:07:00.043814+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:59:27.318106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.509436+0000","last_change":"2024-04-26T23:06:58.436872+0000","last_active":"2024-04-26T23:07:00.509436+0000","last_peered":"2024-04-26T23:07:00.509436+0000","last_clean":"2024-04-26T23:07:00.509436+0000","last_became_active":"2024-04-26T23:06:58.436458+0000","last_became_peered":"2024-04-26T23:06:58.436458+0000","last_unstale":"2024-04-26T23:07:00.509436+0000","last_undegraded":"2024-04-26T23:07:00.509436+0000","last_fullsized":"2024-04-26T23:07:00.509436+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:05:31.728047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.443636+0000","last_change":"2024-04-26T23:06:58.441571+0000","last_active":"2024-04-26T23:06:59.443636+0000","last_peered":"2024-04-26T23:06:59.443636+0000","last_clean":"2024-04-26T23:06:59.443636+0000","last_became_active":"2024-04-26T23:06:58.441295+0000","last_became_peered":"2024-04-26T23:06:58.441295+0000","last_unstale":"2024-04-26T23:06:59.443636+0000","last_undegraded":"2024-04-26T23:06:59.443636+0000","last_fullsized":"2024-04-26T23:06:59.443636+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:29:52.117783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.043157+0000","last_change":"2024-04-26T23:06:58.439662+0000","last_active":"2024-04-26T23:07:00.043157+0000","last_peered":"2024-04-26T23:07:00.043157+0000","last_clean":"2024-04-26T23:07:00.043157+0000","last_became_active":"2024-04-26T23:06:58.439309+0000","last_became_peered":"2024-04-26T23:06:58.439309+0000","last_unstale":"2024-04-26T23:07:00.043157+0000","last_undegraded":"2024-04-26T23:07:00.043157+0000","last_fullsized":"2024-04-26T23:07:00.043157+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:31.001967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.442836+0000","last_change":"2024-04-26T23:06:58.434083+0000","last_active":"2024-04-26T23:06:59.442836+0000","last_peered":"2024-04-26T23:06:59.442836+0000","last_clean":"2024-04-26T23:06:59.442836+0000","last_became_active":"2024-04-26T23:06:58.433154+0000","last_became_peered":"2024-04-26T23:06:58.433154+0000","last_unstale":"2024-04-26T23:06:59.442836+0000","last_undegraded":"2024-04-26T23:06:59.442836+0000","last_fullsized":"2024-04-26T23:06:59.442836+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:28.346736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.510678+0000","last_change":"2024-04-26T23:06:58.452345+0000","last_active":"2024-04-26T23:07:00.510678+0000","last_peered":"2024-04-26T23:07:00.510678+0000","last_clean":"2024-04-26T23:07:00.510678+0000","last_became_active":"2024-04-26T23:06:58.450314+0000","last_became_peered":"2024-04-26T23:06:58.450314+0000","last_unstale":"2024-04-26T23:07:00.510678+0000","last_undegraded":"2024-04-26T23:07:00.510678+0000","last_fullsized":"2024-04-26T23:07:00.510678+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:25.404566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.421639+0000","last_change":"2024-04-26T23:06:58.441157+0000","last_active":"2024-04-26T23:07:00.421639+0000","last_peered":"2024-04-26T23:07:00.421639+0000","last_clean":"2024-04-26T23:07:00.421639+0000","last_became_active":"2024-04-26T23:06:58.440445+0000","last_became_peered":"2024-04-26T23:06:58.440445+0000","last_unstale":"2024-04-26T23:07:00.421639+0000","last_undegraded":"2024-04-26T23:07:00.421639+0000","last_fullsized":"2024-04-26T23:07:00.421639+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:01.940282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.510625+0000","last_change":"2024-04-26T23:06:58.455341+0000","last_active":"2024-04-26T23:07:00.510625+0000","last_peered":"2024-04-26T23:07:00.510625+0000","last_clean":"2024-04-26T23:07:00.510625+0000","last_became_active":"2024-04-26T23:06:58.454478+0000","last_became_peered":"2024-04-26T23:06:58.454478+0000","last_unstale":"2024-04-26T23:07:00.510625+0000","last_undegraded":"2024-04-26T23:07:00.510625+0000","last_fullsized":"2024-04-26T23:07:00.510625+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:44:23.450506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.043922+0000","last_change":"2024-04-26T23:06:58.437578+0000","last_active":"2024-04-26T23:07:00.043922+0000","last_peered":"2024-04-26T23:07:00.043922+0000","last_clean":"2024-04-26T23:07:00.043922+0000","last_became_active":"2024-04-26T23:06:58.437287+0000","last_became_peered":"2024-04-26T23:06:58.437287+0000","last_unstale":"2024-04-26T23:07:00.043922+0000","last_undegraded":"2024-04-26T23:07:00.043922+0000","last_fullsized":"2024-04-26T23:07:00.043922+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:01.063555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.509744+0000","last_change":"2024-04-26T23:06:58.433791+0000","last_active":"2024-04-26T23:07:00.509744+0000","last_peered":"2024-04-26T23:07:00.509744+0000","last_clean":"2024-04-26T23:07:00.509744+0000","last_became_active":"2024-04-26T23:06:58.433074+0000","last_became_peered":"2024-04-26T23:06:58.433074+0000","last_unstale":"2024-04-26T23:07:00.509744+0000","last_undegraded":"2024-04-26T23:07:00.509744+0000","last_fullsized":"2024-04-26T23:07:00.509744+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:50.447855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.420407+0000","last_change":"2024-04-26T23:06:58.426995+0000","last_active":"2024-04-26T23:07:00.420407+0000","last_peered":"2024-04-26T23:07:00.420407+0000","last_clean":"2024-04-26T23:07:00.420407+0000","last_became_active":"2024-04-26T23:06:58.426676+0000","last_became_peered":"2024-04-26T23:06:58.426676+0000","last_unstale":"2024-04-26T23:07:00.420407+0000","last_undegraded":"2024-04-26T23:07:00.420407+0000","last_fullsized":"2024-04-26T23:07:00.420407+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:07:08.079483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.174171+0000","last_change":"2024-04-26T23:06:58.450529+0000","last_active":"2024-04-26T23:07:00.174171+0000","last_peered":"2024-04-26T23:07:00.174171+0000","last_clean":"2024-04-26T23:07:00.174171+0000","last_became_active":"2024-04-26T23:06:58.450214+0000","last_became_peered":"2024-04-26T23:06:58.450214+0000","last_unstale":"2024-04-26T23:07:00.174171+0000","last_undegraded":"2024-04-26T23:07:00.174171+0000","last_fullsized":"2024-04-26T23:07:00.174171+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:23:43.829451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.044349+0000","last_change":"2024-04-26T23:06:58.447475+0000","last_active":"2024-04-26T23:07:00.044349+0000","last_peered":"2024-04-26T23:07:00.044349+0000","last_clean":"2024-04-26T23:07:00.044349+0000","last_became_active":"2024-04-26T23:06:58.445510+0000","last_became_peered":"2024-04-26T23:06:58.445510+0000","last_unstale":"2024-04-26T23:07:00.044349+0000","last_undegraded":"2024-04-26T23:07:00.044349+0000","last_fullsized":"2024-04-26T23:07:00.044349+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:38.403301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.439123+0000","last_change":"2024-04-26T23:06:58.455925+0000","last_active":"2024-04-26T23:06:59.439123+0000","last_peered":"2024-04-26T23:06:59.439123+0000","last_clean":"2024-04-26T23:06:59.439123+0000","last_became_active":"2024-04-26T23:06:58.455681+0000","last_became_peered":"2024-04-26T23:06:58.455681+0000","last_unstale":"2024-04-26T23:06:59.439123+0000","last_undegraded":"2024-04-26T23:06:59.439123+0000","last_fullsized":"2024-04-26T23:06:59.439123+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:45:48.233623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.509976+0000","last_change":"2024-04-26T23:06:58.435898+0000","last_active":"2024-04-26T23:07:00.509976+0000","last_peered":"2024-04-26T23:07:00.509976+0000","last_clean":"2024-04-26T23:07:00.509976+0000","last_became_active":"2024-04-26T23:06:58.435560+0000","last_became_peered":"2024-04-26T23:06:58.435560+0000","last_unstale":"2024-04-26T23:07:00.509976+0000","last_undegraded":"2024-04-26T23:07:00.509976+0000","last_fullsized":"2024-04-26T23:07:00.509976+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:52:56.690774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.043735+0000","last_change":"2024-04-26T23:06:58.429051+0000","last_active":"2024-04-26T23:07:00.043735+0000","last_peered":"2024-04-26T23:07:00.043735+0000","last_clean":"2024-04-26T23:07:00.043735+0000","last_became_active":"2024-04-26T23:06:58.428155+0000","last_became_peered":"2024-04-26T23:06:58.428155+0000","last_unstale":"2024-04-26T23:07:00.043735+0000","last_undegraded":"2024-04-26T23:07:00.043735+0000","last_fullsized":"2024-04-26T23:07:00.043735+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:45:11.855305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.442216+0000","last_change":"2024-04-26T23:06:58.445069+0000","last_active":"2024-04-26T23:06:59.442216+0000","last_peered":"2024-04-26T23:06:59.442216+0000","last_clean":"2024-04-26T23:06:59.442216+0000","last_became_active":"2024-04-26T23:06:58.444573+0000","last_became_peered":"2024-04-26T23:06:58.444573+0000","last_unstale":"2024-04-26T23:06:59.442216+0000","last_undegraded":"2024-04-26T23:06:59.442216+0000","last_fullsized":"2024-04-26T23:06:59.442216+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:52:09.322243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.044130+0000","last_change":"2024-04-26T23:06:58.435277+0000","last_active":"2024-04-26T23:07:00.044130+0000","last_peered":"2024-04-26T23:07:00.044130+0000","last_clean":"2024-04-26T23:07:00.044130+0000","last_became_active":"2024-04-26T23:06:58.434917+0000","last_became_peered":"2024-04-26T23:06:58.434917+0000","last_unstale":"2024-04-26T23:07:00.044130+0000","last_undegraded":"2024-04-26T23:07:00.044130+0000","last_fullsized":"2024-04-26T23:07:00.044130+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:52:10.211956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.442244+0000","last_change":"2024-04-26T23:06:58.445859+0000","last_active":"2024-04-26T23:06:59.442244+0000","last_peered":"2024-04-26T23:06:59.442244+0000","last_clean":"2024-04-26T23:06:59.442244+0000","last_became_active":"2024-04-26T23:06:58.443678+0000","last_became_peered":"2024-04-26T23:06:58.443678+0000","last_unstale":"2024-04-26T23:06:59.442244+0000","last_undegraded":"2024-04-26T23:06:59.442244+0000","last_fullsized":"2024-04-26T23:06:59.442244+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:24.650472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.420527+0000","last_change":"2024-04-26T23:06:58.429456+0000","last_active":"2024-04-26T23:07:00.420527+0000","last_peered":"2024-04-26T23:07:00.420527+0000","last_clean":"2024-04-26T23:07:00.420527+0000","last_became_active":"2024-04-26T23:06:58.429044+0000","last_became_peered":"2024-04-26T23:06:58.429044+0000","last_unstale":"2024-04-26T23:07:00.420527+0000","last_undegraded":"2024-04-26T23:07:00.420527+0000","last_fullsized":"2024-04-26T23:07:00.420527+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:23:40.169695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.421798+0000","last_change":"2024-04-26T23:06:58.429565+0000","last_active":"2024-04-26T23:07:00.421798+0000","last_peered":"2024-04-26T23:07:00.421798+0000","last_clean":"2024-04-26T23:07:00.421798+0000","last_became_active":"2024-04-26T23:06:58.429264+0000","last_became_peered":"2024-04-26T23:06:58.429264+0000","last_unstale":"2024-04-26T23:07:00.421798+0000","last_undegraded":"2024-04-26T23:07:00.421798+0000","last_fullsized":"2024-04-26T23:07:00.421798+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:38.432361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.510710+0000","last_change":"2024-04-26T23:06:58.451350+0000","last_active":"2024-04-26T23:07:00.510710+0000","last_peered":"2024-04-26T23:07:00.510710+0000","last_clean":"2024-04-26T23:07:00.510710+0000","last_became_active":"2024-04-26T23:06:58.450659+0000","last_became_peered":"2024-04-26T23:06:58.450659+0000","last_unstale":"2024-04-26T23:07:00.510710+0000","last_undegraded":"2024-04-26T23:07:00.510710+0000","last_fullsized":"2024-04-26T23:07:00.510710+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:32:54.042192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.174172+0000","last_change":"2024-04-26T23:06:58.448490+0000","last_active":"2024-04-26T23:07:00.174172+0000","last_peered":"2024-04-26T23:07:00.174172+0000","last_clean":"2024-04-26T23:07:00.174172+0000","last_became_active":"2024-04-26T23:06:58.448168+0000","last_became_peered":"2024-04-26T23:06:58.448168+0000","last_unstale":"2024-04-26T23:07:00.174172+0000","last_undegraded":"2024-04-26T23:07:00.174172+0000","last_fullsized":"2024-04-26T23:07:00.174172+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:14.612878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.043971+0000","last_change":"2024-04-26T23:06:58.444429+0000","last_active":"2024-04-26T23:07:00.043971+0000","last_peered":"2024-04-26T23:07:00.043971+0000","last_clean":"2024-04-26T23:07:00.043971+0000","last_became_active":"2024-04-26T23:06:58.444128+0000","last_became_peered":"2024-04-26T23:06:58.444128+0000","last_unstale":"2024-04-26T23:07:00.043971+0000","last_undegraded":"2024-04-26T23:07:00.043971+0000","last_fullsized":"2024-04-26T23:07:00.043971+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:14:37.505874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.420892+0000","last_change":"2024-04-26T23:06:58.444867+0000","last_active":"2024-04-26T23:07:00.420892+0000","last_peered":"2024-04-26T23:07:00.420892+0000","last_clean":"2024-04-26T23:07:00.420892+0000","last_became_active":"2024-04-26T23:06:58.444606+0000","last_became_peered":"2024-04-26T23:06:58.444606+0000","last_unstale":"2024-04-26T23:07:00.420892+0000","last_undegraded":"2024-04-26T23:07:00.420892+0000","last_fullsized":"2024-04-26T23:07:00.420892+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:06:16.968235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.174341+0000","last_change":"2024-04-26T23:06:58.448975+0000","last_active":"2024-04-26T23:07:00.174341+0000","last_peered":"2024-04-26T23:07:00.174341+0000","last_clean":"2024-04-26T23:07:00.174341+0000","last_became_active":"2024-04-26T23:06:58.448530+0000","last_became_peered":"2024-04-26T23:06:58.448530+0000","last_unstale":"2024-04-26T23:07:00.174341+0000","last_undegraded":"2024-04-26T23:07:00.174341+0000","last_fullsized":"2024-04-26T23:07:00.174341+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:19:50.059292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.438457+0000","last_change":"2024-04-26T23:06:58.452619+0000","last_active":"2024-04-26T23:06:59.438457+0000","last_peered":"2024-04-26T23:06:59.438457+0000","last_clean":"2024-04-26T23:06:59.438457+0000","last_became_active":"2024-04-26T23:06:58.451135+0000","last_became_peered":"2024-04-26T23:06:58.451135+0000","last_unstale":"2024-04-26T23:06:59.438457+0000","last_undegraded":"2024-04-26T23:06:59.438457+0000","last_fullsized":"2024-04-26T23:06:59.438457+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:28.346736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.174138+0000","last_change":"2024-04-26T23:06:58.453246+0000","last_active":"2024-04-26T23:07:00.174138+0000","last_peered":"2024-04-26T23:07:00.174138+0000","last_clean":"2024-04-26T23:07:00.174138+0000","last_became_active":"2024-04-26T23:06:58.452959+0000","last_became_peered":"2024-04-26T23:06:58.452959+0000","last_unstale":"2024-04-26T23:07:00.174138+0000","last_undegraded":"2024-04-26T23:07:00.174138+0000","last_fullsized":"2024-04-26T23:07:00.174138+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:24.634615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.438303+0000","last_change":"2024-04-26T23:06:58.436042+0000","last_active":"2024-04-26T23:06:59.438303+0000","last_peered":"2024-04-26T23:06:59.438303+0000","last_clean":"2024-04-26T23:06:59.438303+0000","last_became_active":"2024-04-26T23:06:58.435758+0000","last_became_peered":"2024-04-26T23:06:58.435758+0000","last_unstale":"2024-04-26T23:06:59.438303+0000","last_undegraded":"2024-04-26T23:06:59.438303+0000","last_fullsized":"2024-04-26T23:06:59.438303+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:39.273098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"78'4","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.044285+0000","last_change":"2024-04-26T23:06:57.412094+0000","last_active":"2024-04-26T23:07:00.044285+0000","last_peered":"2024-04-26T23:07:00.044285+0000","last_clean":"2024-04-26T23:07:00.044285+0000","last_became_active":"2024-04-26T23:06:57.411531+0000","last_became_peered":"2024-04-26T23:06:57.411531+0000","last_unstale":"2024-04-26T23:07:00.044285+0000","last_undegraded":"2024-04-26T23:07:00.044285+0000","last_fullsized":"2024-04-26T23:07:00.044285+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:44:14.623431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.420355+0000","last_change":"2024-04-26T23:06:58.444622+0000","last_active":"2024-04-26T23:07:00.420355+0000","last_peered":"2024-04-26T23:07:00.420355+0000","last_clean":"2024-04-26T23:07:00.420355+0000","last_became_active":"2024-04-26T23:06:58.444286+0000","last_became_peered":"2024-04-26T23:06:58.444286+0000","last_unstale":"2024-04-26T23:07:00.420355+0000","last_undegraded":"2024-04-26T23:07:00.420355+0000","last_fullsized":"2024-04-26T23:07:00.420355+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:16.696415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.174323+0000","last_change":"2024-04-26T23:06:57.418786+0000","last_active":"2024-04-26T23:07:00.174323+0000","last_peered":"2024-04-26T23:07:00.174323+0000","last_clean":"2024-04-26T23:07:00.174323+0000","last_became_active":"2024-04-26T23:06:57.418446+0000","last_became_peered":"2024-04-26T23:06:57.418446+0000","last_unstale":"2024-04-26T23:07:00.174323+0000","last_undegraded":"2024-04-26T23:07:00.174323+0000","last_fullsized":"2024-04-26T23:07:00.174323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:07:09.918915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.441943+0000","last_change":"2024-04-26T23:06:58.430301+0000","last_active":"2024-04-26T23:06:59.441943+0000","last_peered":"2024-04-26T23:06:59.441943+0000","last_clean":"2024-04-26T23:06:59.441943+0000","last_became_active":"2024-04-26T23:06:58.429558+0000","last_became_peered":"2024-04-26T23:06:58.429558+0000","last_unstale":"2024-04-26T23:06:59.441943+0000","last_undegraded":"2024-04-26T23:06:59.441943+0000","last_fullsized":"2024-04-26T23:06:59.441943+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:42:12.114009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.174554+0000","last_change":"2024-04-26T23:06:57.413701+0000","last_active":"2024-04-26T23:07:00.174554+0000","last_peered":"2024-04-26T23:07:00.174554+0000","last_clean":"2024-04-26T23:07:00.174554+0000","last_became_active":"2024-04-26T23:06:57.413402+0000","last_became_peered":"2024-04-26T23:06:57.413402+0000","last_unstale":"2024-04-26T23:07:00.174554+0000","last_undegraded":"2024-04-26T23:07:00.174554+0000","last_fullsized":"2024-04-26T23:07:00.174554+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:48:03.854225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.443427+0000","last_change":"2024-04-26T23:06:58.444833+0000","last_active":"2024-04-26T23:06:59.443427+0000","last_peered":"2024-04-26T23:06:59.443427+0000","last_clean":"2024-04-26T23:06:59.443427+0000","last_became_active":"2024-04-26T23:06:58.444232+0000","last_became_peered":"2024-04-26T23:06:58.444232+0000","last_unstale":"2024-04-26T23:06:59.443427+0000","last_undegraded":"2024-04-26T23:06:59.443427+0000","last_fullsized":"2024-04-26T23:06:59.443427+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:52.265707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.444092+0000","last_change":"2024-04-26T23:06:57.412204+0000","last_active":"2024-04-26T23:06:59.444092+0000","last_peered":"2024-04-26T23:06:59.444092+0000","last_clean":"2024-04-26T23:06:59.444092+0000","last_became_active":"2024-04-26T23:06:57.410478+0000","last_became_peered":"2024-04-26T23:06:57.410478+0000","last_unstale":"2024-04-26T23:06:59.444092+0000","last_undegraded":"2024-04-26T23:06:59.444092+0000","last_fullsized":"2024-04-26T23:06:59.444092+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:23.635701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.421373+0000","last_change":"2024-04-26T23:06:58.441449+0000","last_active":"2024-04-26T23:07:00.421373+0000","last_peered":"2024-04-26T23:07:00.421373+0000","last_clean":"2024-04-26T23:07:00.421373+0000","last_became_active":"2024-04-26T23:06:58.439748+0000","last_became_peered":"2024-04-26T23:06:58.439748+0000","last_unstale":"2024-04-26T23:07:00.421373+0000","last_undegraded":"2024-04-26T23:07:00.421373+0000","last_fullsized":"2024-04-26T23:07:00.421373+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:12:07.452896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.442476+0000","last_change":"2024-04-26T23:06:57.408245+0000","last_active":"2024-04-26T23:06:59.442476+0000","last_peered":"2024-04-26T23:06:59.442476+0000","last_clean":"2024-04-26T23:06:59.442476+0000","last_became_active":"2024-04-26T23:06:57.407905+0000","last_became_peered":"2024-04-26T23:06:57.407905+0000","last_unstale":"2024-04-26T23:06:59.442476+0000","last_undegraded":"2024-04-26T23:06:59.442476+0000","last_fullsized":"2024-04-26T23:06:59.442476+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:18.348296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.422178+0000","last_change":"2024-04-26T23:06:58.438998+0000","last_active":"2024-04-26T23:07:00.422178+0000","last_peered":"2024-04-26T23:07:00.422178+0000","last_clean":"2024-04-26T23:07:00.422178+0000","last_became_active":"2024-04-26T23:06:58.438704+0000","last_became_peered":"2024-04-26T23:06:58.438704+0000","last_unstale":"2024-04-26T23:07:00.422178+0000","last_undegraded":"2024-04-26T23:07:00.422178+0000","last_fullsized":"2024-04-26T23:07:00.422178+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:50.447855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.510047+0000","last_change":"2024-04-26T23:06:57.407272+0000","last_active":"2024-04-26T23:07:00.510047+0000","last_peered":"2024-04-26T23:07:00.510047+0000","last_clean":"2024-04-26T23:07:00.510047+0000","last_became_active":"2024-04-26T23:06:57.406963+0000","last_became_peered":"2024-04-26T23:06:57.406963+0000","last_unstale":"2024-04-26T23:07:00.510047+0000","last_undegraded":"2024-04-26T23:07:00.510047+0000","last_fullsized":"2024-04-26T23:07:00.510047+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:38.258327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.442672+0000","last_change":"2024-04-26T23:06:58.434550+0000","last_active":"2024-04-26T23:06:59.442672+0000","last_peered":"2024-04-26T23:06:59.442672+0000","last_clean":"2024-04-26T23:06:59.442672+0000","last_became_active":"2024-04-26T23:06:58.434059+0000","last_became_peered":"2024-04-26T23:06:58.434059+0000","last_unstale":"2024-04-26T23:06:59.442672+0000","last_undegraded":"2024-04-26T23:06:59.442672+0000","last_fullsized":"2024-04-26T23:06:59.442672+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:39.273098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.420641+0000","last_change":"2024-04-26T23:06:57.405640+0000","last_active":"2024-04-26T23:07:00.420641+0000","last_peered":"2024-04-26T23:07:00.420641+0000","last_clean":"2024-04-26T23:07:00.420641+0000","last_became_active":"2024-04-26T23:06:57.405336+0000","last_became_peered":"2024-04-26T23:06:57.405336+0000","last_unstale":"2024-04-26T23:07:00.420641+0000","last_undegraded":"2024-04-26T23:07:00.420641+0000","last_fullsized":"2024-04-26T23:07:00.420641+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:48:03.854225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.445189+0000","last_change":"2024-04-26T23:06:58.446032+0000","last_active":"2024-04-26T23:06:59.445189+0000","last_peered":"2024-04-26T23:06:59.445189+0000","last_clean":"2024-04-26T23:06:59.445189+0000","last_became_active":"2024-04-26T23:06:58.445735+0000","last_became_peered":"2024-04-26T23:06:58.445735+0000","last_unstale":"2024-04-26T23:06:59.445189+0000","last_undegraded":"2024-04-26T23:06:59.445189+0000","last_fullsized":"2024-04-26T23:06:59.445189+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:07:10.933687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.043198+0000","last_change":"2024-04-26T23:06:57.412429+0000","last_active":"2024-04-26T23:07:00.043198+0000","last_peered":"2024-04-26T23:07:00.043198+0000","last_clean":"2024-04-26T23:07:00.043198+0000","last_became_active":"2024-04-26T23:06:57.411673+0000","last_became_peered":"2024-04-26T23:06:57.411673+0000","last_unstale":"2024-04-26T23:07:00.043198+0000","last_undegraded":"2024-04-26T23:07:00.043198+0000","last_fullsized":"2024-04-26T23:07:00.043198+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:34.914814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.420338+0000","last_change":"2024-04-26T23:06:58.437019+0000","last_active":"2024-04-26T23:07:00.420338+0000","last_peered":"2024-04-26T23:07:00.420338+0000","last_clean":"2024-04-26T23:07:00.420338+0000","last_became_active":"2024-04-26T23:06:58.436697+0000","last_became_peered":"2024-04-26T23:06:58.436697+0000","last_unstale":"2024-04-26T23:07:00.420338+0000","last_undegraded":"2024-04-26T23:07:00.420338+0000","last_fullsized":"2024-04-26T23:07:00.420338+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:25.422977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.174751+0000","last_change":"2024-04-26T23:06:57.418763+0000","last_active":"2024-04-26T23:07:00.174751+0000","last_peered":"2024-04-26T23:07:00.174751+0000","last_clean":"2024-04-26T23:07:00.174751+0000","last_became_active":"2024-04-26T23:06:57.418441+0000","last_became_peered":"2024-04-26T23:06:57.418441+0000","last_unstale":"2024-04-26T23:07:00.174751+0000","last_undegraded":"2024-04-26T23:07:00.174751+0000","last_fullsized":"2024-04-26T23:07:00.174751+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:10:16.182193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.510362+0000","last_change":"2024-04-26T23:06:58.451379+0000","last_active":"2024-04-26T23:07:00.510362+0000","last_peered":"2024-04-26T23:07:00.510362+0000","last_clean":"2024-04-26T23:07:00.510362+0000","last_became_active":"2024-04-26T23:06:58.451103+0000","last_became_peered":"2024-04-26T23:06:58.451103+0000","last_unstale":"2024-04-26T23:07:00.510362+0000","last_undegraded":"2024-04-26T23:07:00.510362+0000","last_fullsized":"2024-04-26T23:07:00.510362+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:10:17.196964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.510976+0000","last_change":"2024-04-26T23:06:57.413101+0000","last_active":"2024-04-26T23:07:00.510976+0000","last_peered":"2024-04-26T23:07:00.510976+0000","last_clean":"2024-04-26T23:07:00.510976+0000","last_became_active":"2024-04-26T23:06:57.412770+0000","last_became_peered":"2024-04-26T23:06:57.412770+0000","last_unstale":"2024-04-26T23:07:00.510976+0000","last_undegraded":"2024-04-26T23:07:00.510976+0000","last_fullsized":"2024-04-26T23:07:00.510976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:11.112333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.509850+0000","last_change":"2024-04-26T23:06:58.453847+0000","last_active":"2024-04-26T23:07:00.509850+0000","last_peered":"2024-04-26T23:07:00.509850+0000","last_clean":"2024-04-26T23:07:00.509850+0000","last_became_active":"2024-04-26T23:06:58.453419+0000","last_became_peered":"2024-04-26T23:06:58.453419+0000","last_unstale":"2024-04-26T23:07:00.509850+0000","last_undegraded":"2024-04-26T23:07:00.509850+0000","last_fullsized":"2024-04-26T23:07:00.509850+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:38.432361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"78'3","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.044452+0000","last_change":"2024-04-26T23:06:57.409856+0000","last_active":"2024-04-26T23:07:00.044452+0000","last_peered":"2024-04-26T23:07:00.044452+0000","last_clean":"2024-04-26T23:07:00.044452+0000","last_became_active":"2024-04-26T23:06:57.409578+0000","last_became_peered":"2024-04-26T23:06:57.409578+0000","last_unstale":"2024-04-26T23:07:00.044452+0000","last_undegraded":"2024-04-26T23:07:00.044452+0000","last_fullsized":"2024-04-26T23:07:00.044452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:56:00.629199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.438791+0000","last_change":"2024-04-26T23:06:58.452742+0000","last_active":"2024-04-26T23:06:59.438791+0000","last_peered":"2024-04-26T23:06:59.438791+0000","last_clean":"2024-04-26T23:06:59.438791+0000","last_became_active":"2024-04-26T23:06:58.452392+0000","last_became_peered":"2024-04-26T23:06:58.452392+0000","last_unstale":"2024-04-26T23:06:59.438791+0000","last_undegraded":"2024-04-26T23:06:59.438791+0000","last_fullsized":"2024-04-26T23:06:59.438791+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:52.265707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"78'6","reported_seq":20,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.444190+0000","last_change":"2024-04-26T23:06:57.399434+0000","last_active":"2024-04-26T23:06:59.444190+0000","last_peered":"2024-04-26T23:06:59.444190+0000","last_clean":"2024-04-26T23:06:59.444190+0000","last_became_active":"2024-04-26T23:06:57.399100+0000","last_became_peered":"2024-04-26T23:06:57.399100+0000","last_unstale":"2024-04-26T23:06:59.444190+0000","last_undegraded":"2024-04-26T23:06:59.444190+0000","last_fullsized":"2024-04-26T23:06:59.444190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:27.331965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.511086+0000","last_change":"2024-04-26T23:06:58.443607+0000","last_active":"2024-04-26T23:07:00.511086+0000","last_peered":"2024-04-26T23:07:00.511086+0000","last_clean":"2024-04-26T23:07:00.511086+0000","last_became_active":"2024-04-26T23:06:58.443323+0000","last_became_peered":"2024-04-26T23:06:58.443323+0000","last_unstale":"2024-04-26T23:07:00.511086+0000","last_undegraded":"2024-04-26T23:07:00.511086+0000","last_fullsized":"2024-04-26T23:07:00.511086+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:52.930395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.510083+0000","last_change":"2024-04-26T23:06:57.412138+0000","last_active":"2024-04-26T23:07:00.510083+0000","last_peered":"2024-04-26T23:07:00.510083+0000","last_clean":"2024-04-26T23:07:00.510083+0000","last_became_active":"2024-04-26T23:06:57.411299+0000","last_became_peered":"2024-04-26T23:06:57.411299+0000","last_unstale":"2024-04-26T23:07:00.510083+0000","last_undegraded":"2024-04-26T23:07:00.510083+0000","last_fullsized":"2024-04-26T23:07:00.510083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:27.331965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.444451+0000","last_change":"2024-04-26T23:06:58.437318+0000","last_active":"2024-04-26T23:06:59.444451+0000","last_peered":"2024-04-26T23:06:59.444451+0000","last_clean":"2024-04-26T23:06:59.444451+0000","last_became_active":"2024-04-26T23:06:58.435497+0000","last_became_peered":"2024-04-26T23:06:58.435497+0000","last_unstale":"2024-04-26T23:06:59.444451+0000","last_undegraded":"2024-04-26T23:06:59.444451+0000","last_fullsized":"2024-04-26T23:06:59.444451+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:45:48.233623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.173922+0000","last_change":"2024-04-26T23:06:57.416632+0000","last_active":"2024-04-26T23:07:00.173922+0000","last_peered":"2024-04-26T23:07:00.173922+0000","last_clean":"2024-04-26T23:07:00.173922+0000","last_became_active":"2024-04-26T23:06:57.416299+0000","last_became_peered":"2024-04-26T23:06:57.416299+0000","last_unstale":"2024-04-26T23:07:00.173922+0000","last_undegraded":"2024-04-26T23:07:00.173922+0000","last_fullsized":"2024-04-26T23:07:00.173922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:28:08.367994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.445133+0000","last_change":"2024-04-26T23:06:58.443279+0000","last_active":"2024-04-26T23:06:59.445133+0000","last_peered":"2024-04-26T23:06:59.445133+0000","last_clean":"2024-04-26T23:06:59.445133+0000","last_became_active":"2024-04-26T23:06:58.442948+0000","last_became_peered":"2024-04-26T23:06:58.442948+0000","last_unstale":"2024-04-26T23:06:59.445133+0000","last_undegraded":"2024-04-26T23:06:59.445133+0000","last_fullsized":"2024-04-26T23:06:59.445133+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:57.135459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.043253+0000","last_change":"2024-04-26T23:06:57.414417+0000","last_active":"2024-04-26T23:07:00.043253+0000","last_peered":"2024-04-26T23:07:00.043253+0000","last_clean":"2024-04-26T23:07:00.043253+0000","last_became_active":"2024-04-26T23:06:57.412565+0000","last_became_peered":"2024-04-26T23:06:57.412565+0000","last_unstale":"2024-04-26T23:07:00.043253+0000","last_undegraded":"2024-04-26T23:07:00.043253+0000","last_fullsized":"2024-04-26T23:07:00.043253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:48.123335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.445343+0000","last_change":"2024-04-26T23:06:58.444459+0000","last_active":"2024-04-26T23:06:59.445343+0000","last_peered":"2024-04-26T23:06:59.445343+0000","last_clean":"2024-04-26T23:06:59.445343+0000","last_became_active":"2024-04-26T23:06:58.444064+0000","last_became_peered":"2024-04-26T23:06:58.444064+0000","last_unstale":"2024-04-26T23:06:59.445343+0000","last_undegraded":"2024-04-26T23:06:59.445343+0000","last_fullsized":"2024-04-26T23:06:59.445343+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:10:17.196964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"78'8","reported_seq":22,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.420654+0000","last_change":"2024-04-26T23:06:57.403621+0000","last_active":"2024-04-26T23:07:00.420654+0000","last_peered":"2024-04-26T23:07:00.420654+0000","last_clean":"2024-04-26T23:07:00.420654+0000","last_became_active":"2024-04-26T23:06:57.403306+0000","last_became_peered":"2024-04-26T23:06:57.403306+0000","last_unstale":"2024-04-26T23:07:00.420654+0000","last_undegraded":"2024-04-26T23:07:00.420654+0000","last_fullsized":"2024-04-26T23:07:00.420654+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:07:09.918915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.509528+0000","last_change":"2024-04-26T23:06:58.428990+0000","last_active":"2024-04-26T23:07:00.509528+0000","last_peered":"2024-04-26T23:07:00.509528+0000","last_clean":"2024-04-26T23:07:00.509528+0000","last_became_active":"2024-04-26T23:06:58.428699+0000","last_became_peered":"2024-04-26T23:06:58.428699+0000","last_unstale":"2024-04-26T23:07:00.509528+0000","last_undegraded":"2024-04-26T23:07:00.509528+0000","last_fullsized":"2024-04-26T23:07:00.509528+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:01.940282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.175058+0000","last_change":"2024-04-26T23:06:57.418000+0000","last_active":"2024-04-26T23:07:00.175058+0000","last_peered":"2024-04-26T23:07:00.175058+0000","last_clean":"2024-04-26T23:07:00.175058+0000","last_became_active":"2024-04-26T23:06:57.417711+0000","last_became_peered":"2024-04-26T23:06:57.417711+0000","last_unstale":"2024-04-26T23:07:00.175058+0000","last_undegraded":"2024-04-26T23:07:00.175058+0000","last_fullsized":"2024-04-26T23:07:00.175058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:44:22.435735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.043179+0000","last_change":"2024-04-26T23:06:58.447191+0000","last_active":"2024-04-26T23:07:00.043179+0000","last_peered":"2024-04-26T23:07:00.043179+0000","last_clean":"2024-04-26T23:07:00.043179+0000","last_became_active":"2024-04-26T23:06:58.446868+0000","last_became_peered":"2024-04-26T23:06:58.446868+0000","last_unstale":"2024-04-26T23:07:00.043179+0000","last_undegraded":"2024-04-26T23:07:00.043179+0000","last_fullsized":"2024-04-26T23:07:00.043179+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:17.514019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.442973+0000","last_change":"2024-04-26T23:06:57.405040+0000","last_active":"2024-04-26T23:06:59.442973+0000","last_peered":"2024-04-26T23:06:59.442973+0000","last_clean":"2024-04-26T23:06:59.442973+0000","last_became_active":"2024-04-26T23:06:57.404745+0000","last_became_peered":"2024-04-26T23:06:57.404745+0000","last_unstale":"2024-04-26T23:06:59.442973+0000","last_undegraded":"2024-04-26T23:06:59.442973+0000","last_fullsized":"2024-04-26T23:06:59.442973+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:14:50.064153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.043395+0000","last_change":"2024-04-26T23:06:58.429772+0000","last_active":"2024-04-26T23:07:00.043395+0000","last_peered":"2024-04-26T23:07:00.043395+0000","last_clean":"2024-04-26T23:07:00.043395+0000","last_became_active":"2024-04-26T23:06:58.429476+0000","last_became_peered":"2024-04-26T23:06:58.429476+0000","last_unstale":"2024-04-26T23:07:00.043395+0000","last_undegraded":"2024-04-26T23:07:00.043395+0000","last_fullsized":"2024-04-26T23:07:00.043395+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:09.903734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"78'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.044529+0000","last_change":"2024-04-26T23:06:57.412631+0000","last_active":"2024-04-26T23:07:00.044529+0000","last_peered":"2024-04-26T23:07:00.044529+0000","last_clean":"2024-04-26T23:07:00.044529+0000","last_became_active":"2024-04-26T23:06:57.412310+0000","last_became_peered":"2024-04-26T23:06:57.412310+0000","last_unstale":"2024-04-26T23:07:00.044529+0000","last_undegraded":"2024-04-26T23:07:00.044529+0000","last_fullsized":"2024-04-26T23:07:00.044529+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:46:09.407451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.509862+0000","last_change":"2024-04-26T23:06:58.446808+0000","last_active":"2024-04-26T23:07:00.509862+0000","last_peered":"2024-04-26T23:07:00.509862+0000","last_clean":"2024-04-26T23:07:00.509862+0000","last_became_active":"2024-04-26T23:06:58.446456+0000","last_became_peered":"2024-04-26T23:06:58.446456+0000","last_unstale":"2024-04-26T23:07:00.509862+0000","last_undegraded":"2024-04-26T23:07:00.509862+0000","last_fullsized":"2024-04-26T23:07:00.509862+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:12:26.626482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.438702+0000","last_change":"2024-04-26T23:06:57.408116+0000","last_active":"2024-04-26T23:06:59.438702+0000","last_peered":"2024-04-26T23:06:59.438702+0000","last_clean":"2024-04-26T23:06:59.438702+0000","last_became_active":"2024-04-26T23:06:57.407806+0000","last_became_peered":"2024-04-26T23:06:57.407806+0000","last_unstale":"2024-04-26T23:06:59.438702+0000","last_undegraded":"2024-04-26T23:06:59.438702+0000","last_fullsized":"2024-04-26T23:06:59.438702+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:42:11.099238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.421472+0000","last_change":"2024-04-26T23:06:58.448241+0000","last_active":"2024-04-26T23:07:00.421472+0000","last_peered":"2024-04-26T23:07:00.421472+0000","last_clean":"2024-04-26T23:07:00.421472+0000","last_became_active":"2024-04-26T23:06:58.447901+0000","last_became_peered":"2024-04-26T23:06:58.447901+0000","last_unstale":"2024-04-26T23:07:00.421472+0000","last_undegraded":"2024-04-26T23:07:00.421472+0000","last_fullsized":"2024-04-26T23:07:00.421472+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:05:31.728047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.509882+0000","last_change":"2024-04-26T23:06:57.410225+0000","last_active":"2024-04-26T23:07:00.509882+0000","last_peered":"2024-04-26T23:07:00.509882+0000","last_clean":"2024-04-26T23:07:00.509882+0000","last_became_active":"2024-04-26T23:06:57.409931+0000","last_became_peered":"2024-04-26T23:06:57.409931+0000","last_unstale":"2024-04-26T23:07:00.509882+0000","last_undegraded":"2024-04-26T23:07:00.509882+0000","last_fullsized":"2024-04-26T23:07:00.509882+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:24.408205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.421978+0000","last_change":"2024-04-26T23:06:58.440587+0000","last_active":"2024-04-26T23:07:00.421978+0000","last_peered":"2024-04-26T23:07:00.421978+0000","last_clean":"2024-04-26T23:07:00.421978+0000","last_became_active":"2024-04-26T23:06:58.440305+0000","last_became_peered":"2024-04-26T23:06:58.440305+0000","last_unstale":"2024-04-26T23:07:00.421978+0000","last_undegraded":"2024-04-26T23:07:00.421978+0000","last_fullsized":"2024-04-26T23:07:00.421978+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:52:56.690774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.509430+0000","last_change":"2024-04-26T23:06:57.411645+0000","last_active":"2024-04-26T23:07:00.509430+0000","last_peered":"2024-04-26T23:07:00.509430+0000","last_clean":"2024-04-26T23:07:00.509430+0000","last_became_active":"2024-04-26T23:06:57.411360+0000","last_became_peered":"2024-04-26T23:06:57.411360+0000","last_unstale":"2024-04-26T23:07:00.509430+0000","last_undegraded":"2024-04-26T23:07:00.509430+0000","last_fullsized":"2024-04-26T23:07:00.509430+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:19:49.044521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.445446+0000","last_change":"2024-04-26T23:06:58.445781+0000","last_active":"2024-04-26T23:06:59.445446+0000","last_peered":"2024-04-26T23:06:59.445446+0000","last_clean":"2024-04-26T23:06:59.445446+0000","last_became_active":"2024-04-26T23:06:58.444508+0000","last_became_peered":"2024-04-26T23:06:58.444508+0000","last_unstale":"2024-04-26T23:06:59.445446+0000","last_undegraded":"2024-04-26T23:06:59.445446+0000","last_fullsized":"2024-04-26T23:06:59.445446+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:44:23.450506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.509477+0000","last_change":"2024-04-26T23:06:57.404872+0000","last_active":"2024-04-26T23:07:00.509477+0000","last_peered":"2024-04-26T23:07:00.509477+0000","last_clean":"2024-04-26T23:07:00.509477+0000","last_became_active":"2024-04-26T23:06:57.404587+0000","last_became_peered":"2024-04-26T23:06:57.404587+0000","last_unstale":"2024-04-26T23:07:00.509477+0000","last_undegraded":"2024-04-26T23:07:00.509477+0000","last_fullsized":"2024-04-26T23:07:00.509477+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:13.598107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.442493+0000","last_change":"2024-04-26T23:06:58.451209+0000","last_active":"2024-04-26T23:06:59.442493+0000","last_peered":"2024-04-26T23:06:59.442493+0000","last_clean":"2024-04-26T23:06:59.442493+0000","last_became_active":"2024-04-26T23:06:58.450586+0000","last_became_peered":"2024-04-26T23:06:58.450586+0000","last_unstale":"2024-04-26T23:06:59.442493+0000","last_undegraded":"2024-04-26T23:06:59.442493+0000","last_fullsized":"2024-04-26T23:06:59.442493+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:12.127104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"78'2","reported_seq":16,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.510908+0000","last_change":"2024-04-26T23:06:57.404490+0000","last_active":"2024-04-26T23:07:00.510908+0000","last_peered":"2024-04-26T23:07:00.510908+0000","last_clean":"2024-04-26T23:07:00.510908+0000","last_became_active":"2024-04-26T23:06:57.404076+0000","last_became_peered":"2024-04-26T23:06:57.404076+0000","last_unstale":"2024-04-26T23:07:00.510908+0000","last_undegraded":"2024-04-26T23:07:00.510908+0000","last_fullsized":"2024-04-26T23:07:00.510908+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:51.250935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.443810+0000","last_change":"2024-04-26T23:06:58.438709+0000","last_active":"2024-04-26T23:06:59.443810+0000","last_peered":"2024-04-26T23:06:59.443810+0000","last_clean":"2024-04-26T23:06:59.443810+0000","last_became_active":"2024-04-26T23:06:58.434892+0000","last_became_peered":"2024-04-26T23:06:58.434892+0000","last_unstale":"2024-04-26T23:06:59.443810+0000","last_undegraded":"2024-04-26T23:06:59.443810+0000","last_fullsized":"2024-04-26T23:06:59.443810+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:28:09.382765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"79'1","reported_seq":15,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.422252+0000","last_change":"2024-04-26T23:06:57.403494+0000","last_active":"2024-04-26T23:07:00.422252+0000","last_peered":"2024-04-26T23:07:00.422252+0000","last_clean":"2024-04-26T23:07:00.422252+0000","last_became_active":"2024-04-26T23:06:57.403173+0000","last_became_peered":"2024-04-26T23:06:57.403173+0000","last_unstale":"2024-04-26T23:07:00.422252+0000","last_undegraded":"2024-04-26T23:07:00.422252+0000","last_fullsized":"2024-04-26T23:07:00.422252+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:56.120688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.442066+0000","last_change":"2024-04-26T23:06:58.435297+0000","last_active":"2024-04-26T23:06:59.442066+0000","last_peered":"2024-04-26T23:06:59.442066+0000","last_clean":"2024-04-26T23:06:59.442066+0000","last_became_active":"2024-04-26T23:06:58.434997+0000","last_became_peered":"2024-04-26T23:06:58.434997+0000","last_unstale":"2024-04-26T23:06:59.442066+0000","last_undegraded":"2024-04-26T23:06:59.442066+0000","last_fullsized":"2024-04-26T23:06:59.442066+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:27:57.639090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.174898+0000","last_change":"2024-04-26T23:06:57.416052+0000","last_active":"2024-04-26T23:07:00.174898+0000","last_peered":"2024-04-26T23:07:00.174898+0000","last_clean":"2024-04-26T23:07:00.174898+0000","last_became_active":"2024-04-26T23:06:57.415704+0000","last_became_peered":"2024-04-26T23:06:57.415704+0000","last_unstale":"2024-04-26T23:07:00.174898+0000","last_undegraded":"2024-04-26T23:07:00.174898+0000","last_fullsized":"2024-04-26T23:07:00.174898+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:56.120688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.420933+0000","last_change":"2024-04-26T23:06:58.447881+0000","last_active":"2024-04-26T23:07:00.420933+0000","last_peered":"2024-04-26T23:07:00.420933+0000","last_clean":"2024-04-26T23:07:00.420933+0000","last_became_active":"2024-04-26T23:06:58.447574+0000","last_became_peered":"2024-04-26T23:06:58.447574+0000","last_unstale":"2024-04-26T23:07:00.420933+0000","last_undegraded":"2024-04-26T23:07:00.420933+0000","last_fullsized":"2024-04-26T23:07:00.420933+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:20.034475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.174564+0000","last_change":"2024-04-26T23:06:57.415604+0000","last_active":"2024-04-26T23:07:00.174564+0000","last_peered":"2024-04-26T23:07:00.174564+0000","last_clean":"2024-04-26T23:07:00.174564+0000","last_became_active":"2024-04-26T23:06:57.415245+0000","last_became_peered":"2024-04-26T23:06:57.415245+0000","last_unstale":"2024-04-26T23:07:00.174564+0000","last_undegraded":"2024-04-26T23:07:00.174564+0000","last_fullsized":"2024-04-26T23:07:00.174564+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:45:47.218851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.442908+0000","last_change":"2024-04-26T23:06:58.442588+0000","last_active":"2024-04-26T23:06:59.442908+0000","last_peered":"2024-04-26T23:06:59.442908+0000","last_clean":"2024-04-26T23:06:59.442908+0000","last_became_active":"2024-04-26T23:06:58.442288+0000","last_became_peered":"2024-04-26T23:06:58.442288+0000","last_unstale":"2024-04-26T23:06:59.442908+0000","last_undegraded":"2024-04-26T23:06:59.442908+0000","last_fullsized":"2024-04-26T23:06:59.442908+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:52.265707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.443037+0000","last_change":"2024-04-26T23:06:57.410530+0000","last_active":"2024-04-26T23:06:59.443037+0000","last_peered":"2024-04-26T23:06:59.443037+0000","last_clean":"2024-04-26T23:06:59.443037+0000","last_became_active":"2024-04-26T23:06:57.410240+0000","last_became_peered":"2024-04-26T23:06:57.410240+0000","last_unstale":"2024-04-26T23:06:59.443037+0000","last_undegraded":"2024-04-26T23:06:59.443037+0000","last_fullsized":"2024-04-26T23:06:59.443037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:21.777197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.444577+0000","last_change":"2024-04-26T23:06:58.441866+0000","last_active":"2024-04-26T23:06:59.444577+0000","last_peered":"2024-04-26T23:06:59.444577+0000","last_clean":"2024-04-26T23:06:59.444577+0000","last_became_active":"2024-04-26T23:06:58.437656+0000","last_became_peered":"2024-04-26T23:06:58.437656+0000","last_unstale":"2024-04-26T23:06:59.444577+0000","last_undegraded":"2024-04-26T23:06:59.444577+0000","last_fullsized":"2024-04-26T23:06:59.444577+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:48:04.868996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.043674+0000","last_change":"2024-04-26T23:06:57.413970+0000","last_active":"2024-04-26T23:07:00.043674+0000","last_peered":"2024-04-26T23:07:00.043674+0000","last_clean":"2024-04-26T23:07:00.043674+0000","last_became_active":"2024-04-26T23:06:57.412222+0000","last_became_peered":"2024-04-26T23:06:57.412222+0000","last_unstale":"2024-04-26T23:07:00.043674+0000","last_undegraded":"2024-04-26T23:07:00.043674+0000","last_fullsized":"2024-04-26T23:07:00.043674+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:03:24.678630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.439031+0000","last_change":"2024-04-26T23:06:58.430732+0000","last_active":"2024-04-26T23:06:59.439031+0000","last_peered":"2024-04-26T23:06:59.439031+0000","last_clean":"2024-04-26T23:06:59.439031+0000","last_became_active":"2024-04-26T23:06:58.430415+0000","last_became_peered":"2024-04-26T23:06:58.430415+0000","last_unstale":"2024-04-26T23:06:59.439031+0000","last_undegraded":"2024-04-26T23:06:59.439031+0000","last_fullsized":"2024-04-26T23:06:59.439031+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:28:09.382765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.443922+0000","last_change":"2024-04-26T23:06:57.411122+0000","last_active":"2024-04-26T23:06:59.443922+0000","last_peered":"2024-04-26T23:06:59.443922+0000","last_clean":"2024-04-26T23:06:59.443922+0000","last_became_active":"2024-04-26T23:06:57.410137+0000","last_became_peered":"2024-04-26T23:06:57.410137+0000","last_unstale":"2024-04-26T23:06:59.443922+0000","last_undegraded":"2024-04-26T23:06:59.443922+0000","last_fullsized":"2024-04-26T23:06:59.443922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:56.039206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.439304+0000","last_change":"2024-04-26T23:06:58.449188+0000","last_active":"2024-04-26T23:06:59.439304+0000","last_peered":"2024-04-26T23:06:59.439304+0000","last_clean":"2024-04-26T23:06:59.439304+0000","last_became_active":"2024-04-26T23:06:58.448797+0000","last_became_peered":"2024-04-26T23:06:58.448797+0000","last_unstale":"2024-04-26T23:06:59.439304+0000","last_undegraded":"2024-04-26T23:06:59.439304+0000","last_fullsized":"2024-04-26T23:06:59.439304+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:48:04.868996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:06:59.438263+0000","last_change":"2024-04-26T23:06:57.406278+0000","last_active":"2024-04-26T23:06:59.438263+0000","last_peered":"2024-04-26T23:06:59.438263+0000","last_clean":"2024-04-26T23:06:59.438263+0000","last_became_active":"2024-04-26T23:06:57.405984+0000","last_became_peered":"2024-04-26T23:06:57.405984+0000","last_unstale":"2024-04-26T23:06:59.438263+0000","last_undegraded":"2024-04-26T23:06:59.438263+0000","last_fullsized":"2024-04-26T23:06:59.438263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:32:22.989603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.174971+0000","last_change":"2024-04-26T23:06:58.457450+0000","last_active":"2024-04-26T23:07:00.174971+0000","last_peered":"2024-04-26T23:07:00.174971+0000","last_clean":"2024-04-26T23:07:00.174971+0000","last_became_active":"2024-04-26T23:06:58.455842+0000","last_became_peered":"2024-04-26T23:06:58.455842+0000","last_unstale":"2024-04-26T23:07:00.174971+0000","last_undegraded":"2024-04-26T23:07:00.174971+0000","last_fullsized":"2024-04-26T23:07:00.174971+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:57.367271+0000","last_clean_scrub_stamp":"2024-04-26T23:06:57.367271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:25.422977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.510237+0000","last_change":"2024-04-26T23:06:57.405613+0000","last_active":"2024-04-26T23:07:00.510237+0000","last_peered":"2024-04-26T23:07:00.510237+0000","last_clean":"2024-04-26T23:07:00.510237+0000","last_became_active":"2024-04-26T23:06:57.405175+0000","last_became_peered":"2024-04-26T23:06:57.405175+0000","last_unstale":"2024-04-26T23:07:00.510237+0000","last_undegraded":"2024-04-26T23:07:00.510237+0000","last_fullsized":"2024-04-26T23:07:00.510237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:06:56.352500+0000","last_clean_scrub_stamp":"2024-04-26T23:06:56.352500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:56:15.256487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":283,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-26T23:07:00.174459+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:07:00.174459+0000","last_peered":"2024-04-26T23:07:00.174459+0000","last_clean":"2024-04-26T23:07:00.174459+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:07:00.174459+0000","last_undegraded":"2024-04-26T23:07:00.174459+0000","last_fullsized":"2024-04-26T23:07:00.174459+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574924,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1564672,"data_stored":1038257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574924,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":978944,"data_stored":449479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574924,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":970752,"data_stored":447901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574924,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":987136,"data_stored":449583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574924,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":970752,"data_stored":448449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574926,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":978944,"data_stored":448609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":974848,"data_stored":447979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574925,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1560576,"data_stored":1038235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:07:03.457 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T23:07:03.458 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-26T23:07:03.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.720+0000 7fce23e00640 1 Processor -- start 2024-04-26T23:07:03.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.720+0000 7fce23e00640 1 -- start start 2024-04-26T23:07:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.722+0000 7fce23e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fce240d06b0 0x7fce240d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.722+0000 7fce23e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce240b59f0 con 0x7fce24146060 2024-04-26T23:07:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.722+0000 7fce23e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce240b4180 con 0x7fce24145970 2024-04-26T23:07:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.722+0000 7fce23e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fce240ceed0 con 0x7fce240d06b0 2024-04-26T23:07:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.722+0000 7fce22000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fce24146060 0x7fce240b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59488/0 (socket says 172.21.15.40:59488) 2024-04-26T23:07:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.722+0000 7fce22000640 1 -- 172.21.15.40:0/3681210277 learned_addr learned my addr 172.21.15.40:0/3681210277 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:03.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.722+0000 7fce23400640 1 --2- 172.21.15.40:0/3681210277 >> v2:172.21.15.103:3300/0 conn(0x7fce240d06b0 0x7fce240d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4087042601 0 0) 0x7fce240b59f0 con 0x7fce24146060 2024-04-26T23:07:03.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce08003520 con 0x7fce24146060 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3599708069 0 0) 0x7fce08003520 con 0x7fce24146060 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 >> v2:172.21.15.103:3300/0 conn(0x7fce240d06b0 msgr2=0x7fce240d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 --2- 172.21.15.40:0/3681210277 >> v2:172.21.15.103:3300/0 conn(0x7fce240d06b0 0x7fce240d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 >> v1:172.21.15.103:6789/0 conn(0x7fce24145970 legacy=0x7fce241612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce240b5f40 con 0x7fce24146060 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fce10001670 con 0x7fce24146060 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 >> v1:172.21.15.40:6789/0 conn(0x7fce24146060 legacy=0x7fce240b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 --2- 172.21.15.40:0/3681210277 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce080039b0 0x7fce08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 --2- 172.21.15.40:0/3681210277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce08004800 0x7fce08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 --2- 172.21.15.40:0/3681210277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce080051c0 0x7fce0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fce0800c030 con 0x7fce08004800 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fce0800c1e0 con 0x7fce080051c0 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.723+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fce0800c3e0 con 0x7fce080039b0 2024-04-26T23:07:03.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.724+0000 7fce22000640 1 --2- 172.21.15.40:0/3681210277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce08004800 0x7fce08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.724+0000 7fce22a00640 1 --2- 172.21.15.40:0/3681210277 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce080039b0 0x7fce08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.724+0000 7fce23400640 1 --2- 172.21.15.40:0/3681210277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce080051c0 0x7fce0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.724+0000 7fce22000640 1 -- 172.21.15.40:0/3681210277 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce080039b0 msgr2=0x7fce08003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:03.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.724+0000 7fce22000640 1 --2- 172.21.15.40:0/3681210277 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce080039b0 0x7fce08003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.724+0000 7fce22000640 1 -- 172.21.15.40:0/3681210277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce080051c0 msgr2=0x7fce0800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:03.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.724+0000 7fce22000640 1 --2- 172.21.15.40:0/3681210277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce080051c0 0x7fce0800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.724+0000 7fce23400640 1 -- 172.21.15.40:0/3681210277 reap_dead start 2024-04-26T23:07:03.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.724+0000 7fce22000640 1 -- 172.21.15.40:0/3681210277 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce08003520 con 0x7fce08004800 2024-04-26T23:07:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.725+0000 7fce22000640 1 --2- 172.21.15.40:0/3681210277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce08004800 0x7fce08004c80 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fce1000c210 tx=0x7fce08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d02521792ae6259 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.726+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce10007d30 con 0x7fce08004800 2024-04-26T23:07:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.726+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce100058d0 con 0x7fce08004800 2024-04-26T23:07:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.726+0000 7fce21600640 1 -- 172.21.15.40:0/3681210277 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce10005d20 con 0x7fce08004800 2024-04-26T23:07:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.726+0000 7fce23e00640 1 -- 172.21.15.40:0/3681210277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce08004800 msgr2=0x7fce08004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.726+0000 7fce23e00640 1 --2- 172.21.15.40:0/3681210277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce08004800 0x7fce08004c80 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fce1000c210 tx=0x7fce08005830 comp rx=0 tx=0).stop 2024-04-26T23:07:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.726+0000 7fce23e00640 1 -- 172.21.15.40:0/3681210277 shutdown_connections 2024-04-26T23:07:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.726+0000 7fce23e00640 1 --2- 172.21.15.40:0/3681210277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce08004800 0x7fce08004c80 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.726+0000 7fce23e00640 1 -- 172.21.15.40:0/3681210277 >> 172.21.15.40:0/3681210277 conn(0x7fce24075eb0 msgr2=0x7fce24068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.726+0000 7fce23e00640 1 -- 172.21.15.40:0/3681210277 shutdown_connections 2024-04-26T23:07:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.726+0000 7fce23e00640 1 -- 172.21.15.40:0/3681210277 wait complete. 2024-04-26T23:07:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.728+0000 7fce23e00640 1 Processor -- start 2024-04-26T23:07:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.728+0000 7fce23e00640 1 -- start start 2024-04-26T23:07:03.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.729+0000 7fce23e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce08004800 0x7fce2415c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.729+0000 7fce23e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce240ae8c0 0x7fce24160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.729+0000 7fce23e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce24071d90 0x7fce24143220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.729+0000 7fce23e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fce240cf3e0 con 0x7fce08004800 2024-04-26T23:07:03.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.729+0000 7fce23e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fce240cf100 con 0x7fce240ae8c0 2024-04-26T23:07:03.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.729+0000 7fce23e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fce24145d50 con 0x7fce24071d90 2024-04-26T23:07:03.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.729+0000 7fce22a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce08004800 0x7fce2415c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.730+0000 7fce22a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce08004800 0x7fce2415c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54406/0 (socket says 172.21.15.40:54406) 2024-04-26T23:07:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.730+0000 7fce22a00640 1 -- 172.21.15.40:0/1904003205 learned_addr learned my addr 172.21.15.40:0/1904003205 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.730+0000 7fce22a00640 1 -- 172.21.15.40:0/1904003205 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce24071d90 msgr2=0x7fce24143220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.730+0000 7fce22a00640 1 --2- 172.21.15.40:0/1904003205 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce24071d90 0x7fce24143220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.730+0000 7fce22a00640 1 -- 172.21.15.40:0/1904003205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce240ae8c0 msgr2=0x7fce24160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.730+0000 7fce22a00640 1 --2- 172.21.15.40:0/1904003205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce240ae8c0 0x7fce24160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.730+0000 7fce22a00640 1 -- 172.21.15.40:0/1904003205 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce241613f0 con 0x7fce08004800 2024-04-26T23:07:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.730+0000 7fce22a00640 1 --2- 172.21.15.40:0/1904003205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce08004800 0x7fce2415c990 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fce1400e500 tx=0x7fce1400e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.731+0000 7fce1b400640 1 -- 172.21.15.40:0/1904003205 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce140052e0 con 0x7fce08004800 2024-04-26T23:07:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.731+0000 7fce1b400640 1 -- 172.21.15.40:0/1904003205 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce14018070 con 0x7fce08004800 2024-04-26T23:07:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.731+0000 7fce23e00640 1 -- 172.21.15.40:0/1904003205 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce240aee30 con 0x7fce08004800 2024-04-26T23:07:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.731+0000 7fce1b400640 1 -- 172.21.15.40:0/1904003205 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce14003810 con 0x7fce08004800 2024-04-26T23:07:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.731+0000 7fce23e00640 1 -- 172.21.15.40:0/1904003205 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce240af3f0 con 0x7fce08004800 2024-04-26T23:07:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.732+0000 7fce23e00640 1 -- 172.21.15.40:0/1904003205 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce24069370 con 0x7fce08004800 2024-04-26T23:07:03.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.733+0000 7fce1b400640 1 -- 172.21.15.40:0/1904003205 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fce140039b0 con 0x7fce08004800 2024-04-26T23:07:03.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.734+0000 7fce1b400640 1 --2- 172.21.15.40:0/1904003205 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fce0406c8f0 0x7fce0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:03.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.734+0000 7fce1b400640 1 -- 172.21.15.40:0/1904003205 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fce1408b9b0 con 0x7fce08004800 2024-04-26T23:07:03.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.735+0000 7fce22000640 1 --2- 172.21.15.40:0/1904003205 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fce0406c8f0 0x7fce0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:03.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.736+0000 7fce22000640 1 --2- 172.21.15.40:0/1904003205 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fce0406c8f0 0x7fce0406edb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fce10033000 tx=0x7fce10009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:03.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:03.739+0000 7fce1b400640 1 -- 172.21.15.40:0/1904003205 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fce241613f0 con 0x7fce08004800 2024-04-26T23:07:04.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.013+0000 7fce23e00640 1 -- 172.21.15.40:0/1904003205 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fce240cdaf0 con 0x7fce08004800 2024-04-26T23:07:04.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.019+0000 7fce1b400640 1 -- 172.21.15.40:0/1904003205 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) ==== 190+0+147 (secure 0 0 0) 0x7fce14059c40 con 0x7fce08004800 2024-04-26T23:07:04.019 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-04-26T23:07:04.019 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg== 2024-04-26T23:07:04.019 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-04-26T23:07:04.019 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-04-26T23:07:04.019 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-26T23:07:04.019 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-04-26T23:07:04.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.021+0000 7fce23e00640 1 -- 172.21.15.40:0/1904003205 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fce0406c8f0 msgr2=0x7fce0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:04.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.021+0000 7fce23e00640 1 --2- 172.21.15.40:0/1904003205 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fce0406c8f0 0x7fce0406edb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fce10033000 tx=0x7fce10009030 comp rx=0 tx=0).stop 2024-04-26T23:07:04.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.021+0000 7fce23e00640 1 -- 172.21.15.40:0/1904003205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce08004800 msgr2=0x7fce2415c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:04.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.021+0000 7fce23e00640 1 --2- 172.21.15.40:0/1904003205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce08004800 0x7fce2415c990 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fce1400e500 tx=0x7fce1400e9c0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.021+0000 7fce23e00640 1 -- 172.21.15.40:0/1904003205 shutdown_connections 2024-04-26T23:07:04.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.021+0000 7fce23e00640 1 --2- 172.21.15.40:0/1904003205 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fce0406c8f0 0x7fce0406edb0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.021+0000 7fce23e00640 1 --2- 172.21.15.40:0/1904003205 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fce24071d90 0x7fce24143220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.021+0000 7fce23e00640 1 --2- 172.21.15.40:0/1904003205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce240ae8c0 0x7fce24160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.021+0000 7fce23e00640 1 --2- 172.21.15.40:0/1904003205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce08004800 0x7fce2415c990 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.021+0000 7fce23e00640 1 -- 172.21.15.40:0/1904003205 >> 172.21.15.40:0/1904003205 conn(0x7fce24075eb0 msgr2=0x7fce2413fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:04.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.021+0000 7fce23e00640 1 -- 172.21.15.40:0/1904003205 shutdown_connections 2024-04-26T23:07:04.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.022+0000 7fce23e00640 1 -- 172.21.15.40:0/1904003205 wait complete. 2024-04-26T23:07:04.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:04.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.249+0000 7f0055200640 1 Processor -- start 2024-04-26T23:07:04.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.249+0000 7f0055200640 1 -- start start 2024-04-26T23:07:04.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.251+0000 7f0055200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f00500cc480 0x7f0050072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:04.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.251+0000 7f0055200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050072780 0x7f0050166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:04.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.251+0000 7f0055200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f005015d7c0 con 0x7f0050071e60 2024-04-26T23:07:04.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.251+0000 7f0055200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f005015daa0 con 0x7f00500cc480 2024-04-26T23:07:04.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.251+0000 7f0055200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f005015dd80 con 0x7f0050072780 2024-04-26T23:07:04.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.251+0000 7f004d400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f00500cc480 0x7f0050072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:04.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.252+0000 7f004d400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f00500cc480 0x7f0050072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:57010/0 (socket says 172.21.15.40:57010) 2024-04-26T23:07:04.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.252+0000 7f004d400640 1 -- 172.21.15.40:0/3287037777 learned_addr learned my addr 172.21.15.40:0/3287037777 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:04.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.252+0000 7f004e800640 1 --2- 172.21.15.40:0/3287037777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050072780 0x7f0050166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:04.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.252+0000 7f004d400640 1 -- 172.21.15.40:0/3287037777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050072780 msgr2=0x7f0050166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:04.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.252+0000 7f004d400640 1 --2- 172.21.15.40:0/3287037777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050072780 0x7f0050166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.252+0000 7f004d400640 1 -- 172.21.15.40:0/3287037777 >> v1:172.21.15.103:6789/0 conn(0x7f0050071e60 legacy=0x7f005006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:04.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.252+0000 7f004d400640 1 -- 172.21.15.40:0/3287037777 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005015e060 con 0x7f00500cc480 2024-04-26T23:07:04.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.252+0000 7f004ca00640 1 -- 172.21.15.40:0/3287037777 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1243541901 0 0) 0x7f005015d7c0 con 0x7f0050071e60 2024-04-26T23:07:04.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.254+0000 7f004d400640 1 --2- 172.21.15.40:0/3287037777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00500cc480 0x7f0050072240 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f003c009870 tx=0x7f003c02a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52db9ac153ad1c06 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:04.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.254+0000 7f004ca00640 1 -- 172.21.15.40:0/3287037777 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c037070 con 0x7f00500cc480 2024-04-26T23:07:04.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.254+0000 7f004ca00640 1 -- 172.21.15.40:0/3287037777 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f003c032440 con 0x7f00500cc480 2024-04-26T23:07:04.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.254+0000 7f004ca00640 1 -- 172.21.15.40:0/3287037777 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c032810 con 0x7f00500cc480 2024-04-26T23:07:04.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.255+0000 7f0055200640 1 -- 172.21.15.40:0/3287037777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00500cc480 msgr2=0x7f0050072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:04.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.255+0000 7f0055200640 1 --2- 172.21.15.40:0/3287037777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00500cc480 0x7f0050072240 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f003c009870 tx=0x7f003c02a270 comp rx=0 tx=0).stop 2024-04-26T23:07:04.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.255+0000 7f0055200640 1 -- 172.21.15.40:0/3287037777 shutdown_connections 2024-04-26T23:07:04.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.255+0000 7f0055200640 1 --2- 172.21.15.40:0/3287037777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050072780 0x7f0050166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.255+0000 7f0055200640 1 --2- 172.21.15.40:0/3287037777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00500cc480 0x7f0050072240 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.255+0000 7f0055200640 1 -- 172.21.15.40:0/3287037777 >> 172.21.15.40:0/3287037777 conn(0x7f00500d0be0 msgr2=0x7f00500d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:04.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.255+0000 7f0055200640 1 -- 172.21.15.40:0/3287037777 shutdown_connections 2024-04-26T23:07:04.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.255+0000 7f0055200640 1 -- 172.21.15.40:0/3287037777 wait complete. 2024-04-26T23:07:04.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.257+0000 7f0055200640 1 Processor -- start 2024-04-26T23:07:04.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.257+0000 7f0055200640 1 -- start start 2024-04-26T23:07:04.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.258+0000 7f0055200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071e60 0x7f00500c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:04.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.258+0000 7f0055200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050072780 0x7f00501f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:04.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.258+0000 7f0055200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00500cc480 0x7f00500b4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.258+0000 7f0055200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f005015d3f0 con 0x7f0050071e60 2024-04-26T23:07:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.258+0000 7f0055200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0050072c10 con 0x7f00500cc480 2024-04-26T23:07:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.258+0000 7f0055200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f005015d570 con 0x7f0050072780 2024-04-26T23:07:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.258+0000 7f004de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071e60 0x7f00500c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.258+0000 7f004de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071e60 0x7f00500c86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54412/0 (socket says 172.21.15.40:54412) 2024-04-26T23:07:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.258+0000 7f004de00640 1 -- 172.21.15.40:0/1052410688 learned_addr learned my addr 172.21.15.40:0/1052410688 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.258+0000 7f004d400640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050072780 0x7f00501f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.258+0000 7f004e800640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00500cc480 0x7f00500b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.259+0000 7f004de00640 1 -- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050072780 msgr2=0x7f00501f3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.259+0000 7f004de00640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050072780 0x7f00501f3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.259+0000 7f004de00640 1 -- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00500cc480 msgr2=0x7f00500b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.259+0000 7f004de00640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00500cc480 0x7f00500b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.259+0000 7f004de00640 1 -- 172.21.15.40:0/1052410688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00500b3880 con 0x7f0050071e60 2024-04-26T23:07:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.259+0000 7f004de00640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071e60 0x7f00500c86c0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f004000c460 tx=0x7f004000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.259+0000 7f0046000640 1 -- 172.21.15.40:0/1052410688 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0040005480 con 0x7f0050071e60 2024-04-26T23:07:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.259+0000 7f0046000640 1 -- 172.21.15.40:0/1052410688 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00400035f0 con 0x7f0050071e60 2024-04-26T23:07:04.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.259+0000 7f0046000640 1 -- 172.21.15.40:0/1052410688 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0040003a00 con 0x7f0050071e60 2024-04-26T23:07:04.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.259+0000 7f0055200640 1 -- 172.21.15.40:0/1052410688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00500b3bc0 con 0x7f0050071e60 2024-04-26T23:07:04.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.259+0000 7f0055200640 1 -- 172.21.15.40:0/1052410688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00500b4400 con 0x7f0050071e60 2024-04-26T23:07:04.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.261+0000 7f0055200640 1 -- 172.21.15.40:0/1052410688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00500694f0 con 0x7f0050071e60 2024-04-26T23:07:04.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.262+0000 7f0046000640 1 -- 172.21.15.40:0/1052410688 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0040003ba0 con 0x7f0050071e60 2024-04-26T23:07:04.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.263+0000 7f0046000640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f003006ca90 0x7f003006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:04.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.263+0000 7f0046000640 1 -- 172.21.15.40:0/1052410688 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f004008b9b0 con 0x7f0050071e60 2024-04-26T23:07:04.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.263+0000 7f004d400640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f003006ca90 0x7f003006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:04.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.265+0000 7f004d400640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f003006ca90 0x7f003006ef50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f003c034000 tx=0x7f003c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.267+0000 7f0046000640 1 -- 172.21.15.40:0/1052410688 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f00500b3bc0 con 0x7f0050071e60 2024-04-26T23:07:04.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.510+0000 7f0055200640 1 -- 172.21.15.40:0/1052410688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00500b6290 con 0x7f0050071e60 2024-04-26T23:07:04.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.512+0000 7f0046000640 1 -- 172.21.15.40:0/1052410688 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5183 (secure 0 0 0) 0x7f0040059c50 con 0x7f0050071e60 2024-04-26T23:07:04.512 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:04.512 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":74,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/755627718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":755627718},{"type":"v1","addr":"172.21.15.103:6837","nonce":755627718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":5965,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2914603370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2914603370},{"type":"v1","addr":"172.21.15.103:6836","nonce":2914603370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":7443,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/216893598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":216893598},{"type":"v1","addr":"172.21.15.40:6834","nonce":216893598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:06:57.853769+0000","modified":"2024-04-26T23:07:00.963881+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7446":{"gid":7446,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4227275373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":4227275373},{"type":"v1","addr":"172.21.15.40:6835","nonce":4227275373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7446,"qdb_cluster":[7446]},"id":5}]} 2024-04-26T23:07:04.512 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 74 2024-04-26T23:07:04.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.514+0000 7f0055200640 1 -- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f003006ca90 msgr2=0x7f003006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:04.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.514+0000 7f0055200640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f003006ca90 0x7f003006ef50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f003c034000 tx=0x7f003c02e000 comp rx=0 tx=0).stop 2024-04-26T23:07:04.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.514+0000 7f0055200640 1 -- 172.21.15.40:0/1052410688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071e60 msgr2=0x7f00500c86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:04.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.514+0000 7f0055200640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071e60 0x7f00500c86c0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f004000c460 tx=0x7f004000c920 comp rx=0 tx=0).stop 2024-04-26T23:07:04.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.514+0000 7f0055200640 1 -- 172.21.15.40:0/1052410688 shutdown_connections 2024-04-26T23:07:04.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.514+0000 7f0055200640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f003006ca90 0x7f003006ef50 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.514+0000 7f0055200640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00500cc480 0x7f00500b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.514+0000 7f0055200640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050072780 0x7f00501f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.514+0000 7f0055200640 1 --2- 172.21.15.40:0/1052410688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071e60 0x7f00500c86c0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:04.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.514+0000 7f0055200640 1 -- 172.21.15.40:0/1052410688 >> 172.21.15.40:0/1052410688 conn(0x7f00500d0be0 msgr2=0x7f00500b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:04.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.514+0000 7f0055200640 1 -- 172.21.15.40:0/1052410688 shutdown_connections 2024-04-26T23:07:04.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:04.514+0000 7f0055200640 1 -- 172.21.15.40:0/1052410688 wait complete. 2024-04-26T23:07:04.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 74, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:06:57.853769+0000', 'modified': '2024-04-26T23:07:00.963881+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7446}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7446': {'gid': 7446, 'name': 'c', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/4227275373', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 4227275373}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 4227275373}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7446, 'qdb_cluster': [7446]} 2024-04-26T23:07:04.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-26T23:07:04.938 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T23:07:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.158+0000 7f7925c00640 1 Processor -- start 2024-04-26T23:07:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.159+0000 7f7925c00640 1 -- start start 2024-04-26T23:07:05.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.160+0000 7f7925c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f79200c6f10 0x7f79200c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.160+0000 7f7925c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f79200cc980 0x7f79200cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.160+0000 7f7925c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79200cce10 con 0x7f79200c6010 2024-04-26T23:07:05.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.160+0000 7f7925c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f79200c72f0 con 0x7f79200c6f10 2024-04-26T23:07:05.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.160+0000 7f7925c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f79200cb860 con 0x7f79200cc980 2024-04-26T23:07:05.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.161+0000 7f791fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f79200c6f10 0x7f79200c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.161+0000 7f791fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f79200c6f10 0x7f79200c5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54272/0 (socket says 172.21.15.40:54272) 2024-04-26T23:07:05.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.161+0000 7f791fe00640 1 -- 172.21.15.40:0/3751431717 learned_addr learned my addr 172.21.15.40:0/3751431717 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:05.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.161+0000 7f791fe00640 1 -- 172.21.15.40:0/3751431717 >> v2:172.21.15.103:3300/0 conn(0x7f79200cc980 msgr2=0x7f79200cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:05.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.161+0000 7f791fe00640 1 --2- 172.21.15.40:0/3751431717 >> v2:172.21.15.103:3300/0 conn(0x7f79200cc980 0x7f79200cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.161+0000 7f791fe00640 1 -- 172.21.15.40:0/3751431717 >> v1:172.21.15.40:6789/0 conn(0x7f79200c6010 legacy=0x7f7920158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.161+0000 7f791fe00640 1 -- 172.21.15.40:0/3751431717 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79200cd0f0 con 0x7f79200c6f10 2024-04-26T23:07:05.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.162+0000 7f791fe00640 1 --2- 172.21.15.40:0/3751431717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79200c6f10 0x7f79200c5ad0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f79200b0460 tx=0x7f791002a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612acee6fe41afb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.162+0000 7f791ea00640 1 -- 172.21.15.40:0/3751431717 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7910036070 con 0x7f79200c6f10 2024-04-26T23:07:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.162+0000 7f791ea00640 1 -- 172.21.15.40:0/3751431717 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7910031600 con 0x7f79200c6f10 2024-04-26T23:07:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.162+0000 7f791ea00640 1 -- 172.21.15.40:0/3751431717 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79100319d0 con 0x7f79200c6f10 2024-04-26T23:07:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.163+0000 7f7925c00640 1 -- 172.21.15.40:0/3751431717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79200c6f10 msgr2=0x7f79200c5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.163+0000 7f7925c00640 1 --2- 172.21.15.40:0/3751431717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79200c6f10 0x7f79200c5ad0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f79200b0460 tx=0x7f791002a6b0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.163+0000 7f7925c00640 1 -- 172.21.15.40:0/3751431717 shutdown_connections 2024-04-26T23:07:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.163+0000 7f7925c00640 1 --2- 172.21.15.40:0/3751431717 >> v2:172.21.15.103:3300/0 conn(0x7f79200cc980 0x7f79200cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.163+0000 7f7925c00640 1 --2- 172.21.15.40:0/3751431717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79200c6f10 0x7f79200c5ad0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.163+0000 7f7925c00640 1 -- 172.21.15.40:0/3751431717 >> 172.21.15.40:0/3751431717 conn(0x7f79200d0a10 msgr2=0x7f79200d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.163+0000 7f7925c00640 1 -- 172.21.15.40:0/3751431717 shutdown_connections 2024-04-26T23:07:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.163+0000 7f7925c00640 1 -- 172.21.15.40:0/3751431717 wait complete. 2024-04-26T23:07:05.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.165+0000 7f7925c00640 1 Processor -- start 2024-04-26T23:07:05.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.165+0000 7f7925c00640 1 -- start start 2024-04-26T23:07:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.166+0000 7f7925c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79200c6010 0x7f79201443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.166+0000 7f7925c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79200c6f10 0x7f7920138cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.166+0000 7f7925c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201391f0 0x7f79200b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.166+0000 7f7925c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f79200c76a0 con 0x7f79201391f0 2024-04-26T23:07:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.166+0000 7f7925c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79200cb1a0 con 0x7f79200c6f10 2024-04-26T23:07:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.166+0000 7f7925c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f79200c7820 con 0x7f79200c6010 2024-04-26T23:07:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.166+0000 7f7925200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201391f0 0x7f79200b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.166+0000 7f7925200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201391f0 0x7f79200b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54278/0 (socket says 172.21.15.40:54278) 2024-04-26T23:07:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.166+0000 7f7925200640 1 -- 172.21.15.40:0/2516011460 learned_addr learned my addr 172.21.15.40:0/2516011460 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.167+0000 7f791f400640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79200c6f10 0x7f7920138cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.167+0000 7f7925200640 1 -- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79200c6010 msgr2=0x7f79201443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.167+0000 7f7925200640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79200c6010 0x7f79201443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.167+0000 7f7925200640 1 -- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79200c6f10 msgr2=0x7f7920138cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.167+0000 7f7925200640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79200c6f10 0x7f7920138cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.167+0000 7f7925200640 1 -- 172.21.15.40:0/2516011460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792015c2c0 con 0x7f79201391f0 2024-04-26T23:07:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.167+0000 7f791f400640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79200c6f10 0x7f7920138cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.167+0000 7f7925200640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201391f0 0x7f79200b3730 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f791800c390 tx=0x7f791800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.167+0000 7f7917e00640 1 -- 172.21.15.40:0/2516011460 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918017070 con 0x7f79201391f0 2024-04-26T23:07:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.167+0000 7f7917e00640 1 -- 172.21.15.40:0/2516011460 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79180052d0 con 0x7f79201391f0 2024-04-26T23:07:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.167+0000 7f7925c00640 1 -- 172.21.15.40:0/2516011460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f792015c5a0 con 0x7f79201391f0 2024-04-26T23:07:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.167+0000 7f7917e00640 1 -- 172.21.15.40:0/2516011460 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918003810 con 0x7f79201391f0 2024-04-26T23:07:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.167+0000 7f7925c00640 1 -- 172.21.15.40:0/2516011460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f792015cb60 con 0x7f79201391f0 2024-04-26T23:07:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.169+0000 7f7917e00640 1 -- 172.21.15.40:0/2516011460 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f791801a070 con 0x7f79201391f0 2024-04-26T23:07:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.170+0000 7f7917e00640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f78fc06c8f0 0x7f78fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.170+0000 7f7917e00640 1 -- 172.21.15.40:0/2516011460 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f791808afe0 con 0x7f79201391f0 2024-04-26T23:07:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.171+0000 7f791fe00640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f78fc06c8f0 0x7f78fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.171+0000 7f792a042740 1 Processor -- start 2024-04-26T23:07:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.171+0000 7f792a042740 1 -- start start 2024-04-26T23:07:05.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.172+0000 7f792a042740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4957b0 0x55bdac497910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.172+0000 7f792a042740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4d1e80 0x55bdac495b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.172+0000 7f792a042740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4c0270 0x55bdac4c06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.172+0000 7f792a042740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bdac3f9600 con 0x55bdac4957b0 2024-04-26T23:07:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.172+0000 7f792a042740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bdac396100 con 0x55bdac4c0270 2024-04-26T23:07:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.172+0000 7f792a042740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bdac3f55c0 con 0x55bdac4d1e80 2024-04-26T23:07:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.172+0000 7f791fe00640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f78fc06c8f0 0x7f78fc06edb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7910033000 tx=0x7f791002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.172+0000 7f791fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4957b0 0x55bdac497910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.172+0000 7f791fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4957b0 0x55bdac497910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54282/0 (socket says 172.21.15.40:54282) 2024-04-26T23:07:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.172+0000 7f791fe00640 1 -- 172.21.15.40:0/2234506054 learned_addr learned my addr 172.21.15.40:0/2234506054 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.173+0000 7f791f400640 1 --2- 172.21.15.40:0/2234506054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4d1e80 0x55bdac495b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.173+0000 7f791fe00640 1 -- 172.21.15.40:0/2234506054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4d1e80 msgr2=0x55bdac495b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.173+0000 7f7925200640 1 --2- 172.21.15.40:0/2234506054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4c0270 0x55bdac4c06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.173+0000 7f791fe00640 1 --2- 172.21.15.40:0/2234506054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4d1e80 0x55bdac495b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.173+0000 7f791fe00640 1 -- 172.21.15.40:0/2234506054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4c0270 msgr2=0x55bdac4c06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.173+0000 7f791fe00640 1 --2- 172.21.15.40:0/2234506054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4c0270 0x55bdac4c06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.173+0000 7f791fe00640 1 -- 172.21.15.40:0/2234506054 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdac10b2d0 con 0x55bdac4957b0 2024-04-26T23:07:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.173+0000 7f791fe00640 1 --2- 172.21.15.40:0/2234506054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4957b0 0x55bdac497910 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f791002aca0 tx=0x7f791002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16640ec4ac7b6c91 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.174+0000 7f7914c00640 1 -- 172.21.15.40:0/2234506054 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7910036040 con 0x55bdac4957b0 2024-04-26T23:07:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.174+0000 7f7914c00640 1 -- 172.21.15.40:0/2234506054 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7910041070 con 0x55bdac4957b0 2024-04-26T23:07:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.174+0000 7f7914c00640 1 -- 172.21.15.40:0/2234506054 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791003c9f0 con 0x55bdac4957b0 2024-04-26T23:07:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.174+0000 7f792a042740 1 -- 172.21.15.40:0/2234506054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4957b0 msgr2=0x55bdac497910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.174+0000 7f792a042740 1 --2- 172.21.15.40:0/2234506054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4957b0 0x55bdac497910 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f791002aca0 tx=0x7f791002f000 comp rx=0 tx=0).stop 2024-04-26T23:07:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.174+0000 7f792a042740 1 -- 172.21.15.40:0/2234506054 shutdown_connections 2024-04-26T23:07:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.174+0000 7f792a042740 1 --2- 172.21.15.40:0/2234506054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4c0270 0x55bdac4c06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.174+0000 7f792a042740 1 --2- 172.21.15.40:0/2234506054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4d1e80 0x55bdac495b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.174+0000 7f792a042740 1 --2- 172.21.15.40:0/2234506054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4957b0 0x55bdac497910 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.174+0000 7f792a042740 1 -- 172.21.15.40:0/2234506054 >> 172.21.15.40:0/2234506054 conn(0x55bdac429d10 msgr2=0x55bdac4db270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.175+0000 7f792a042740 1 -- 172.21.15.40:0/2234506054 shutdown_connections 2024-04-26T23:07:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.175+0000 7f792a042740 1 -- 172.21.15.40:0/2234506054 wait complete. 2024-04-26T23:07:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.175+0000 7f792a042740 1 -- start start 2024-04-26T23:07:05.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.175+0000 7f792a042740 1 Processor -- start 2024-04-26T23:07:05.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.176+0000 7f792a042740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 0x55bdac4c1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.176+0000 7f792a042740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4c0270 0x55bdac4bec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.176+0000 7f792a042740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 0x55bdac4bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.176+0000 7f792a042740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bdac380f70 con 0x55bdac4d1e80 2024-04-26T23:07:05.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.176+0000 7f792a042740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bdac4bddc0 con 0x55bdac4957b0 2024-04-26T23:07:05.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.176+0000 7f792a042740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bdac4ad8f0 con 0x55bdac4c0270 2024-04-26T23:07:05.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.177+0000 7f792a042740 10 client.0 _ll_register_callbacks cb 0x55bdac3ef480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:05.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.177+0000 7f7925200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 0x55bdac4bf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.177+0000 7f7925200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 0x55bdac4bf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54296/0 (socket says 172.21.15.40:54296) 2024-04-26T23:07:05.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.177+0000 7f7925200640 1 -- 172.21.15.40:0/3463045766 learned_addr learned my addr 172.21.15.40:0/3463045766 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:05.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.177+0000 7f791fe00640 1 --2- 172.21.15.40:0/3463045766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 0x55bdac4c1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.177+0000 7f791f400640 1 --2- 172.21.15.40:0/3463045766 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4c0270 0x55bdac4bec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.177+0000 7f7925200640 1 -- 172.21.15.40:0/3463045766 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4c0270 msgr2=0x55bdac4bec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:05.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.177+0000 7f7925200640 1 --2- 172.21.15.40:0/3463045766 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4c0270 0x55bdac4bec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.177+0000 7f7925200640 1 -- 172.21.15.40:0/3463045766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 msgr2=0x55bdac4c1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:05.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.177+0000 7f7925200640 1 --2- 172.21.15.40:0/3463045766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 0x55bdac4c1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.177+0000 7f7925200640 1 -- 172.21.15.40:0/3463045766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdac1d56b0 con 0x55bdac4d1e80 2024-04-26T23:07:05.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.177+0000 7f7925200640 1 --2- 172.21.15.40:0/3463045766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 0x55bdac4bf1b0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f791800cd50 tx=0x7f7918059370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c26def987588c56 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.177+0000 7f790d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f790d600640 1 -- 172.21.15.40:0/3463045766 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918071180 con 0x55bdac4d1e80 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f790d600640 1 -- 172.21.15.40:0/3463045766 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f7918058000 con 0x55bdac4d1e80 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f792a042740 1 -- 172.21.15.40:0/3463045766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7910041070 con 0x55bdac4d1e80 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f790d600640 1 -- 172.21.15.40:0/3463045766 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f7918061790 con 0x55bdac4d1e80 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f792a042740 10 client.7581 fetch_fsmap learned FSMap version 74 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f792a042740 1 -- 172.21.15.40:0/3463045766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55bdac4ad630 con 0x55bdac4d1e80 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f790d600640 1 -- 172.21.15.40:0/3463045766 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7910041070 con 0x55bdac4d1e80 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f790d600640 1 -- 172.21.15.40:0/3463045766 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f791805f160 con 0x55bdac4d1e80 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f792a042740 10 client.7581 fetch_fsmap finished waiting for FSMap version 74 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f792a042740 10 client.7581 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f792a042740 20 client.7581 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f792a042740 1 --2- 172.21.15.40:0/3463045766 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55bdac4d2660 0x55bdac4d2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f792a042740 4 client.7581 mds_command: new command op to 7446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:07:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.178+0000 7f792a042740 1 -- 172.21.15.40:0/3463045766 --> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdac407210 con 0x55bdac4d2660 2024-04-26T23:07:05.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.179+0000 7f791fe00640 1 --2- 172.21.15.40:0/3463045766 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55bdac4d2660 0x55bdac4d2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.179+0000 7f791fe00640 1 --2- 172.21.15.40:0/3463045766 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55bdac4d2660 0x55bdac4d2a40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b1a0907e8277ad4 server_cookie=2093c8ae8a69f2a8 in_seq=0 out_seq=0 2024-04-26T23:07:05.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.179+0000 7f790d600640 10 client.7581 ms_handle_connect on v2:172.21.15.40:6833/4227275373 2024-04-26T23:07:05.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.180+0000 7f790d600640 1 -- 172.21.15.40:0/3463045766 <== mds.0 v2:172.21.15.40:6833/4227275373 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bdac407210 con 0x55bdac4d2660 2024-04-26T23:07:05.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.180+0000 7f790d600640 10 client.7581 handle_command_reply: tid=0 2024-04-26T23:07:05.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.180+0000 7f792a042740 1 client.7581 shutdown 2024-04-26T23:07:05.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.181+0000 7f792a042740 1 -- 172.21.15.40:0/3463045766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 msgr2=0x55bdac4bf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.181+0000 7f792a042740 1 --2- 172.21.15.40:0/3463045766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 0x55bdac4bf1b0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f791800cd50 tx=0x7f7918059370 comp rx=0 tx=0).stop 2024-04-26T23:07:05.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.181+0000 7f792a042740 1 -- 172.21.15.40:0/3463045766 shutdown_connections 2024-04-26T23:07:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.181+0000 7f792a042740 1 --2- 172.21.15.40:0/3463045766 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55bdac4d2660 0x55bdac4d2a40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.181+0000 7f792a042740 1 --2- 172.21.15.40:0/3463045766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 0x55bdac4bf1b0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.181+0000 7f792a042740 1 --2- 172.21.15.40:0/3463045766 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4c0270 0x55bdac4bec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.181+0000 7f790d600640 0 client.7581 ms_handle_reset on v2:172.21.15.40:6833/4227275373 2024-04-26T23:07:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.181+0000 7f792a042740 1 --2- 172.21.15.40:0/3463045766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 0x55bdac4c1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.181+0000 7f792a042740 1 -- 172.21.15.40:0/3463045766 >> 172.21.15.40:0/3463045766 conn(0x55bdac429d10 msgr2=0x55bdac4de7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.181+0000 7f792a042740 1 -- 172.21.15.40:0/3463045766 shutdown_connections 2024-04-26T23:07:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.181+0000 7f792a042740 1 -- 172.21.15.40:0/3463045766 wait complete. 2024-04-26T23:07:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.181+0000 7f792a042740 20 client.7581 trim_cache size 0 max 16384 2024-04-26T23:07:05.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.201+0000 7f792a042740 1 Processor -- start 2024-04-26T23:07:05.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.201+0000 7f792a042740 1 -- start start 2024-04-26T23:07:05.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f792a042740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 0x55bdac4edf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f792a042740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4c0270 0x55bdac4d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f792a042740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 0x55bdac4ba200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f792a042740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7920200d50 con 0x55bdac4d1e80 2024-04-26T23:07:05.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f792a042740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79201f5370 con 0x55bdac4957b0 2024-04-26T23:07:05.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f792a042740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f79200b6310 con 0x55bdac4c0270 2024-04-26T23:07:05.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f7925200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 0x55bdac4ba200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f7925200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 0x55bdac4ba200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54310/0 (socket says 172.21.15.40:54310) 2024-04-26T23:07:05.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f7925200640 1 -- 172.21.15.40:0/3805672499 learned_addr learned my addr 172.21.15.40:0/3805672499 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:05.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f791fe00640 1 --2- 172.21.15.40:0/3805672499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 0x55bdac4edf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f791f400640 1 --2- 172.21.15.40:0/3805672499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4c0270 0x55bdac4d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f7925200640 1 -- 172.21.15.40:0/3805672499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4c0270 msgr2=0x55bdac4d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:05.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f7925200640 1 --2- 172.21.15.40:0/3805672499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4c0270 0x55bdac4d2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f7925200640 1 -- 172.21.15.40:0/3805672499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 msgr2=0x55bdac4edf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:05.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.202+0000 7f7925200640 1 --2- 172.21.15.40:0/3805672499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 0x55bdac4edf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.203+0000 7f7925200640 1 -- 172.21.15.40:0/3805672499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdac48b370 con 0x55bdac4d1e80 2024-04-26T23:07:05.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.203+0000 7f7925200640 1 --2- 172.21.15.40:0/3805672499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 0x55bdac4ba200 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f791808b280 tx=0x7f7918014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20608527400a16c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:05.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.203+0000 7f790fe00640 1 -- 172.21.15.40:0/3805672499 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918071180 con 0x55bdac4d1e80 2024-04-26T23:07:05.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.203+0000 7f790fe00640 1 -- 172.21.15.40:0/3805672499 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7918055220 con 0x55bdac4d1e80 2024-04-26T23:07:05.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.203+0000 7f790fe00640 1 -- 172.21.15.40:0/3805672499 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918058000 con 0x55bdac4d1e80 2024-04-26T23:07:05.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.204+0000 7f792a042740 1 -- 172.21.15.40:0/3805672499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 msgr2=0x55bdac4ba200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.204+0000 7f792a042740 1 --2- 172.21.15.40:0/3805672499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 0x55bdac4ba200 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f791808b280 tx=0x7f7918014000 comp rx=0 tx=0).stop 2024-04-26T23:07:05.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.204+0000 7f792a042740 1 -- 172.21.15.40:0/3805672499 shutdown_connections 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.204+0000 7f792a042740 1 --2- 172.21.15.40:0/3805672499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4d1e80 0x55bdac4ba200 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.204+0000 7f792a042740 1 --2- 172.21.15.40:0/3805672499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4c0270 0x55bdac4d2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.204+0000 7f792a042740 1 --2- 172.21.15.40:0/3805672499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 0x55bdac4edf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.204+0000 7f792a042740 1 -- 172.21.15.40:0/3805672499 >> 172.21.15.40:0/3805672499 conn(0x55bdac429d10 msgr2=0x55bdac48bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.204+0000 7f792a042740 1 -- 172.21.15.40:0/3805672499 shutdown_connections 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.204+0000 7f792a042740 1 -- 172.21.15.40:0/3805672499 wait complete. 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.205+0000 7f792a042740 1 -- start start 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.205+0000 7f792a042740 1 Processor -- start 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f792a042740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 0x55bdac555340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f792a042740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4c0270 0x55bdac4d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f792a042740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4d1e80 0x55bdac4ed9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f792a042740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bdac4aced0 con 0x55bdac4c0270 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f792a042740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bdac4ad340 con 0x55bdac4957b0 2024-04-26T23:07:05.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f792a042740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bdac4c0df0 con 0x55bdac4d1e80 2024-04-26T23:07:05.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f792a042740 10 client.0 _ll_register_callbacks cb 0x7f7918054ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:05.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f791f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4c0270 0x55bdac4d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f791f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4c0270 0x55bdac4d2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54326/0 (socket says 172.21.15.40:54326) 2024-04-26T23:07:05.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f791f400640 1 -- 172.21.15.40:0/567927020 learned_addr learned my addr 172.21.15.40:0/567927020 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:05.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f791fe00640 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 0x55bdac555340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f791f400640 1 -- 172.21.15.40:0/567927020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4d1e80 msgr2=0x55bdac4ed9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:05.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f7925200640 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4d1e80 0x55bdac4ed9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.206+0000 7f791f400640 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4d1e80 0x55bdac4ed9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.207+0000 7f791f400640 1 -- 172.21.15.40:0/567927020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 msgr2=0x55bdac555340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:05.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.207+0000 7f791f400640 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 0x55bdac555340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.207+0000 7f791f400640 1 -- 172.21.15.40:0/567927020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdac4bc760 con 0x55bdac4c0270 2024-04-26T23:07:05.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.207+0000 7f791fe00640 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 0x55bdac555340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:05.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.207+0000 7f791f400640 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4c0270 0x55bdac4d2c30 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f79201627a0 tx=0x7f790800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cffaf50909f1d56d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.207+0000 7f790d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.207+0000 7f790d600640 1 -- 172.21.15.40:0/567927020 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7908012830 con 0x55bdac4c0270 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.207+0000 7f790d600640 1 -- 172.21.15.40:0/567927020 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f79080129d0 con 0x55bdac4c0270 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.207+0000 7f792a042740 1 -- 172.21.15.40:0/567927020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7918055220 con 0x55bdac4c0270 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.207+0000 7f790d600640 1 -- 172.21.15.40:0/567927020 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f7908023050 con 0x55bdac4c0270 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.207+0000 7f792a042740 10 client.7587 fetch_fsmap learned FSMap version 74 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.207+0000 7f792a042740 1 -- 172.21.15.40:0/567927020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55bdac4d3170 con 0x55bdac4c0270 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.208+0000 7f790d600640 1 -- 172.21.15.40:0/567927020 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7918055220 con 0x55bdac4c0270 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.208+0000 7f790d600640 1 -- 172.21.15.40:0/567927020 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f7908026020 con 0x55bdac4c0270 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.208+0000 7f792a042740 10 client.7587 fetch_fsmap finished waiting for FSMap version 74 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.208+0000 7f792a042740 10 client.7587 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.208+0000 7f792a042740 20 client.7587 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.208+0000 7f792a042740 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55bdac494070 0x55bdac494450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.208+0000 7f792a042740 4 client.7587 mds_command: new command op to 7446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.208+0000 7f792a042740 1 -- 172.21.15.40:0/567927020 --> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] -- command(tid 0: {"prefix": "status"}) -- 0x55bdac401080 con 0x55bdac494070 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.208+0000 7f791fe00640 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55bdac494070 0x55bdac494450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f791fe00640 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55bdac494070 0x55bdac494450 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5aaa6420e67f3823 server_cookie=862d1c89dea6f16b in_seq=0 out_seq=0 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f790d600640 10 client.7587 ms_handle_connect on v2:172.21.15.40:6833/4227275373 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f790d600640 1 -- 172.21.15.40:0/567927020 <== mds.0 v2:172.21.15.40:6833/4227275373 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55bdac401080 con 0x55bdac494070 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f790d600640 10 client.7587 handle_command_reply: tid=0 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f792a042740 1 client.7587 shutdown 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f792a042740 1 -- 172.21.15.40:0/567927020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4c0270 msgr2=0x55bdac4d2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f792a042740 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4c0270 0x55bdac4d2c30 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f79201627a0 tx=0x7f790800f730 comp rx=0 tx=0).stop 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f792a042740 1 -- 172.21.15.40:0/567927020 shutdown_connections 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f792a042740 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bdac4d1e80 0x55bdac4ed9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f792a042740 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55bdac494070 0x55bdac494450 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f792a042740 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdac4c0270 0x55bdac4d2c30 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f792a042740 1 --2- 172.21.15.40:0/567927020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdac4957b0 0x55bdac555340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f790d600640 0 client.7587 ms_handle_reset on v2:172.21.15.40:6833/4227275373 2024-04-26T23:07:05.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f792a042740 1 -- 172.21.15.40:0/567927020 >> 172.21.15.40:0/567927020 conn(0x55bdac429d10 msgr2=0x55bdac4c1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:05.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f792a042740 1 -- 172.21.15.40:0/567927020 shutdown_connections 2024-04-26T23:07:05.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f792a042740 1 -- 172.21.15.40:0/567927020 wait complete. 2024-04-26T23:07:05.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.209+0000 7f792a042740 20 client.7587 trim_cache size 0 max 16384 2024-04-26T23:07:05.217 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:07:05.217 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:07:05.217 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:07:05.217 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 7446, 2024-04-26T23:07:05.217 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:07:05.218 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:07:05.218 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:07:05.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 6.8041731250000002, 2024-04-26T23:07:05.218 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 74, 2024-04-26T23:07:05.218 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 79, 2024-04-26T23:07:05.218 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 79, 2024-04-26T23:07:05.218 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 13.232598778 2024-04-26T23:07:05.218 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:07:05.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.210+0000 7f7925c00640 1 -- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f78fc06c8f0 msgr2=0x7f78fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.210+0000 7f7925c00640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f78fc06c8f0 0x7f78fc06edb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7910033000 tx=0x7f791002e000 comp rx=0 tx=0).stop 2024-04-26T23:07:05.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.210+0000 7f7925c00640 1 -- 172.21.15.40:0/2516011460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201391f0 msgr2=0x7f79200b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.210+0000 7f7925c00640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201391f0 0x7f79200b3730 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f791800c390 tx=0x7f791800c850 comp rx=0 tx=0).stop 2024-04-26T23:07:05.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.210+0000 7f7925c00640 1 -- 172.21.15.40:0/2516011460 shutdown_connections 2024-04-26T23:07:05.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.210+0000 7f7925c00640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f78fc06c8f0 0x7f78fc06edb0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.210+0000 7f7925c00640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79201391f0 0x7f79200b3730 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.210+0000 7f7925c00640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79200c6f10 0x7f7920138cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.210+0000 7f7925c00640 1 --2- 172.21.15.40:0/2516011460 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f79200c6010 0x7f79201443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.210+0000 7f7925c00640 1 -- 172.21.15.40:0/2516011460 >> 172.21.15.40:0/2516011460 conn(0x7f79200d0a10 msgr2=0x7f792006e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:05.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.210+0000 7f7925c00640 1 -- 172.21.15.40:0/2516011460 shutdown_connections 2024-04-26T23:07:05.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.210+0000 7f7925c00640 1 -- 172.21.15.40:0/2516011460 wait complete. 2024-04-26T23:07:05.226 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-26T23:07:05.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.418+0000 7f9aa8c00640 1 Processor -- start 2024-04-26T23:07:05.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.418+0000 7f9aa8c00640 1 -- start start 2024-04-26T23:07:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.419+0000 7f9aa8c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9aa4162af0 0x7f9aa40b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.420+0000 7f9aa8c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aa40d1fa0 0x7f9aa40b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.420+0000 7f9aa8c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aa40b44b0 con 0x7f9aa40b6450 2024-04-26T23:07:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.420+0000 7f9aa8c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f9aa40d0660 con 0x7f9aa4162af0 2024-04-26T23:07:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.420+0000 7f9aa8c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa40d5a70 con 0x7f9aa40d1fa0 2024-04-26T23:07:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.420+0000 7f9aa3400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9aa4162af0 0x7f9aa40b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.420+0000 7f9aa3400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9aa4162af0 0x7f9aa40b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54336/0 (socket says 172.21.15.40:54336) 2024-04-26T23:07:05.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.420+0000 7f9aa3400640 1 -- 172.21.15.40:0/3241243169 learned_addr learned my addr 172.21.15.40:0/3241243169 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:05.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.420+0000 7f9aa3e00640 1 --2- 172.21.15.40:0/3241243169 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aa40d1fa0 0x7f9aa40b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.420+0000 7f9aa3400640 1 -- 172.21.15.40:0/3241243169 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aa40d1fa0 msgr2=0x7f9aa40b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:05.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.421+0000 7f9aa3400640 1 --2- 172.21.15.40:0/3241243169 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aa40d1fa0 0x7f9aa40b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.421+0000 7f9aa3400640 1 -- 172.21.15.40:0/3241243169 >> v1:172.21.15.103:6789/0 conn(0x7f9aa40b6450 legacy=0x7f9aa40d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:05.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.421+0000 7f9aa3400640 1 -- 172.21.15.40:0/3241243169 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa40d0900 con 0x7f9aa4162af0 2024-04-26T23:07:05.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.421+0000 7f9aa3e00640 1 --2- 172.21.15.40:0/3241243169 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aa40d1fa0 0x7f9aa40b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:05.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.422+0000 7f9aa3400640 1 --2- 172.21.15.40:0/3241243169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4162af0 0x7f9aa40b3d40 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009e50 tx=0x7f9a9402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b67ff20a3d2d3f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:05.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.422+0000 7f9aa2a00640 1 -- 172.21.15.40:0/3241243169 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94036070 con 0x7f9aa4162af0 2024-04-26T23:07:05.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.422+0000 7f9aa2a00640 1 -- 172.21.15.40:0/3241243169 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a94031600 con 0x7f9aa4162af0 2024-04-26T23:07:05.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.422+0000 7f9aa2a00640 1 -- 172.21.15.40:0/3241243169 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a940319d0 con 0x7f9aa4162af0 2024-04-26T23:07:05.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.423+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3241243169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4162af0 msgr2=0x7f9aa40b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.423+0000 7f9aa8c00640 1 --2- 172.21.15.40:0/3241243169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4162af0 0x7f9aa40b3d40 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009e50 tx=0x7f9a9402a6c0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.423+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3241243169 shutdown_connections 2024-04-26T23:07:05.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.423+0000 7f9aa8c00640 1 --2- 172.21.15.40:0/3241243169 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aa40d1fa0 0x7f9aa40b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.423+0000 7f9aa8c00640 1 --2- 172.21.15.40:0/3241243169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4162af0 0x7f9aa40b3d40 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.423+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3241243169 >> 172.21.15.40:0/3241243169 conn(0x7f9aa409e210 msgr2=0x7f9aa40d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:05.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.423+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3241243169 shutdown_connections 2024-04-26T23:07:05.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.423+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3241243169 wait complete. 2024-04-26T23:07:05.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.425+0000 7f9aa8c00640 1 Processor -- start 2024-04-26T23:07:05.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.425+0000 7f9aa8c00640 1 -- start start 2024-04-26T23:07:05.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.426+0000 7f9aa8c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9aa40b6450 0x7f9aa415c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.426+0000 7f9aa8c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aa40d1fa0 0x7f9aa40b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.426+0000 7f9aa8c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4162af0 0x7f9aa40b3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.426+0000 7f9aa8c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa40d5d50 con 0x7f9aa4162af0 2024-04-26T23:07:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.426+0000 7f9aa8c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa40d0ec0 con 0x7f9aa40b6450 2024-04-26T23:07:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.426+0000 7f9aa8c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa40b4d70 con 0x7f9aa40d1fa0 2024-04-26T23:07:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.426+0000 7f9aa3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4162af0 0x7f9aa40b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.426+0000 7f9aa3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4162af0 0x7f9aa40b3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54340/0 (socket says 172.21.15.40:54340) 2024-04-26T23:07:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.426+0000 7f9aa3e00640 1 -- 172.21.15.40:0/3250184061 learned_addr learned my addr 172.21.15.40:0/3250184061 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.426+0000 7f9aa3400640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9aa40b6450 0x7f9aa415c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.426+0000 7f9a9aa00640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aa40d1fa0 0x7f9aa40b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.427+0000 7f9aa3e00640 1 -- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aa40d1fa0 msgr2=0x7f9aa40b2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.427+0000 7f9aa3e00640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aa40d1fa0 0x7f9aa40b2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.427+0000 7f9aa3e00640 1 -- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9aa40b6450 msgr2=0x7f9aa415c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.427+0000 7f9aa3e00640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9aa40b6450 0x7f9aa415c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.427+0000 7f9aa3e00640 1 -- 172.21.15.40:0/3250184061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa40b8510 con 0x7f9aa4162af0 2024-04-26T23:07:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.427+0000 7f9a9aa00640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aa40d1fa0 0x7f9aa40b2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.427+0000 7f9aa3e00640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4162af0 0x7f9aa40b3100 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00c7f0 tx=0x7f9a9c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.427+0000 7f9a9be00640 1 -- 172.21.15.40:0/3250184061 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c015070 con 0x7f9aa4162af0 2024-04-26T23:07:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.427+0000 7f9a9be00640 1 -- 172.21.15.40:0/3250184061 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a9c004240 con 0x7f9aa4162af0 2024-04-26T23:07:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.427+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3250184061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa40b87b0 con 0x7f9aa4162af0 2024-04-26T23:07:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.427+0000 7f9a9be00640 1 -- 172.21.15.40:0/3250184061 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c007620 con 0x7f9aa4162af0 2024-04-26T23:07:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.427+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3250184061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa415c980 con 0x7f9aa4162af0 2024-04-26T23:07:05.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.428+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3250184061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aa4069370 con 0x7f9aa4162af0 2024-04-26T23:07:05.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.429+0000 7f9a9be00640 1 -- 172.21.15.40:0/3250184061 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a9c0079d0 con 0x7f9aa4162af0 2024-04-26T23:07:05.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.430+0000 7f9a9be00640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a8006c8f0 0x7f9a8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:05.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.430+0000 7f9a9be00640 1 -- 172.21.15.40:0/3250184061 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f9a9c08bd00 con 0x7f9aa4162af0 2024-04-26T23:07:05.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.431+0000 7f9aa3400640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a8006c8f0 0x7f9a8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:05.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.432+0000 7f9aa3400640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a8006c8f0 0x7f9a8006edb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9a94033000 tx=0x7f9a94009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.435+0000 7f9a9be00640 1 -- 172.21.15.40:0/3250184061 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9a9c05a050 con 0x7f9aa4162af0 2024-04-26T23:07:05.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.680+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3250184061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9aa4074480 con 0x7f9aa4162af0 2024-04-26T23:07:05.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.681+0000 7f9a9be00640 1 -- 172.21.15.40:0/3250184061 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v74) ==== 53+0+73 (secure 0 0 0) 0x7f9a9c059950 con 0x7f9aa4162af0 2024-04-26T23:07:05.681 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-26T23:07:05.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.683+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a8006c8f0 msgr2=0x7f9a8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.683+0000 7f9aa8c00640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a8006c8f0 0x7f9a8006edb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9a94033000 tx=0x7f9a94009000 comp rx=0 tx=0).stop 2024-04-26T23:07:05.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.683+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3250184061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4162af0 msgr2=0x7f9aa40b3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:05.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.683+0000 7f9aa8c00640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4162af0 0x7f9aa40b3100 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00c7f0 tx=0x7f9a9c00ccb0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.683+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3250184061 shutdown_connections 2024-04-26T23:07:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.683+0000 7f9aa8c00640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a8006c8f0 0x7f9a8006edb0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.683+0000 7f9aa8c00640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aa4162af0 0x7f9aa40b3100 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.683+0000 7f9aa8c00640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aa40d1fa0 0x7f9aa40b2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.683+0000 7f9aa8c00640 1 --2- 172.21.15.40:0/3250184061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9aa40b6450 0x7f9aa415c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.683+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3250184061 >> 172.21.15.40:0/3250184061 conn(0x7f9aa409e210 msgr2=0x7f9aa415fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.683+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3250184061 shutdown_connections 2024-04-26T23:07:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05.683+0000 7f9aa8c00640 1 -- 172.21.15.40:0/3250184061 wait complete. 2024-04-26T23:07:05.702 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-26T23:07:05.703 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:07:05.703 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-04-26T23:07:05.703 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T23:07:05.703 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T23:07:05.703 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-26T23:07:05.703 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-26T23:07:05.703 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:07:05.703 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:07:05.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:05.703 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-04-26T23:07:05.753 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:07:05.753 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:07:05.753 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:07:05.753 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:05.753 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-26T23:07:05.753 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:05.753 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:07:05.753 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:07:05.753 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-04-26T23:07:05.753 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:07:05.753 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:07:05.754 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-04-26T23:07:05.754 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:07:05.754 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-04-26T23:07:05.754 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-04-26T23:07:05.754 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T23:07:05.754 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42288sec preferred_lft 42288sec 2024-04-26T23:07:05.754 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-26T23:07:05.754 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:05.754 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:07:05.754 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-04-26T23:07:05.754 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-04-26T23:07:05.754 INFO:teuthology.orchestra.run.smithi040.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-26T23:07:05.754 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 8e:8e:79:d7:cf:5d brd ff:ff:ff:ff:ff:ff 2024-04-26T23:07:05.755 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-26T23:07:05.755 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:05.755 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::4ce7:9bff:fe51:b1/64 scope link 2024-04-26T23:07:05.755 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:05.755 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:07:05.755 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether be:f1:28:e1:99:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T23:07:05.755 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::bcf1:28ff:fee1:9921/64 scope link 2024-04-26T23:07:05.755 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:05.755 INFO:teuthology.orchestra.run.smithi040.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:07:05.755 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 8e:8e:79:d7:cf:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-26T23:07:05.755 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::8c8e:79ff:fed7:cf5d/64 scope link 2024-04-26T23:07:05.755 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:05.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:05.757 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:07:05.833 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-04-26T23:07:05.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:05.836 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-04-26T23:07:05.888 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-04-26T23:07:05.888 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:07:05.888 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T23:07:05.888 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T23:07:05.888 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:07:05.889 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:07:05.889 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:07:05.889 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T23:07:05.889 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T23:07:05.889 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T23:07:05.889 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:07:05.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:07:06.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:07:06.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:06.106 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:07:06.158 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T23:07:06.158 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-26T23:07:06.159 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:07:06.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:06.159 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-04-26T23:07:06.248 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:07:06.248 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T23:07:06.303 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:07:06.303 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:07:06.303 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:07:06.356 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-26T23:07:06.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:06.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:07:06.500 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:07:06.500 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:07:06.501 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:07:06.522 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:07:06.522 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:07:06.522 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:07:06.522 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:07:06.522 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:07:06.549 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:07:06.549 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:07:06.549 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:07:06.549 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:07:06.549 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:07:06.549 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-26T23:07:06.549 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-26T23:07:06.549 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=815936ab-d4ac-471e-b069-1a0dc70078ee" will pass to kernel 2024-04-26T23:07:06.553 INFO:teuthology.orchestra.run.smithi040.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T23:07:06.553 INFO:teuthology.orchestra.run.smithi040.stdout: You just mounted a file system that supports labels which does not 2024-04-26T23:07:06.553 INFO:teuthology.orchestra.run.smithi040.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T23:07:06.553 INFO:teuthology.orchestra.run.smithi040.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T23:07:06.554 INFO:teuthology.orchestra.run.smithi040.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T23:07:06.557 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T23:07:06.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:06.558 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:07:06.638 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:07:06.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:06 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:07:06.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:07:06.772 INFO:teuthology.orchestra.run.smithi040.stdout:client7596 2024-04-26T23:07:06.773 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:07:06.773 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:07:06.826 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-26T23:07:07.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.090+0000 7fc4cbe00640 1 Processor -- start 2024-04-26T23:07:07.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.091+0000 7fc4cbe00640 1 -- start start 2024-04-26T23:07:07.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.092+0000 7fc4cbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4cc0cfaf0 0x7fc4cc073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.093+0000 7fc4cbe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4cc161880 con 0x7fc4cc06dd40 2024-04-26T23:07:07.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.093+0000 7fc4cbe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4cc0cff80 con 0x7fc4cc160a50 2024-04-26T23:07:07.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.093+0000 7fc4cbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc4cc071ce0 con 0x7fc4cc0cfaf0 2024-04-26T23:07:07.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.093+0000 7fc4cb400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc4cc06dd40 0x7fc4cc072730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43180/0 (socket says 172.21.15.40:43180) 2024-04-26T23:07:07.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.093+0000 7fc4cb400640 1 -- 172.21.15.40:0/1531616171 learned_addr learned my addr 172.21.15.40:0/1531616171 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.093+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/1531616171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4cc0cfaf0 0x7fc4cc073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.093+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2462214081 0 0) 0x7fc4cc161880 con 0x7fc4cc06dd40 2024-04-26T23:07:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.093+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4b0003520 con 0x7fc4cc06dd40 2024-04-26T23:07:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3421116791 0 0) 0x7fc4b0003520 con 0x7fc4cc06dd40 2024-04-26T23:07:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4cc0cfaf0 msgr2=0x7fc4cc073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4ca000640 1 --2- 172.21.15.40:0/1531616171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4cc0cfaf0 0x7fc4cc073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 >> v1:172.21.15.103:6789/0 conn(0x7fc4cc160a50 legacy=0x7fc4cc160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4cc0d0260 con 0x7fc4cc06dd40 2024-04-26T23:07:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fc4c40019f0 con 0x7fc4cc06dd40 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 >> v1:172.21.15.40:6789/0 conn(0x7fc4cc06dd40 legacy=0x7fc4cc072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4ca000640 1 --2- 172.21.15.40:0/1531616171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4b00039e0 0x7fc4b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4ca000640 1 --2- 172.21.15.40:0/1531616171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4b0004340 0x7fc4b0004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4ca000640 1 --2- 172.21.15.40:0/1531616171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b0005020 0x7fc4b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc071ce0 con 0x7fc4b0005020 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b000bf00 con 0x7fc4b0004340 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc4b000c0b0 con 0x7fc4b00039e0 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.094+0000 7fc4cb400640 1 --2- 172.21.15.40:0/1531616171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b0005020 0x7fc4b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.095+0000 7fc4caa00640 1 --2- 172.21.15.40:0/1531616171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4b00039e0 0x7fc4b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.095+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/1531616171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4b0004340 0x7fc4b0004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.095+0000 7fc4cb400640 1 -- 172.21.15.40:0/1531616171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4b00039e0 msgr2=0x7fc4b0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.095+0000 7fc4cb400640 1 --2- 172.21.15.40:0/1531616171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4b00039e0 0x7fc4b0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.095+0000 7fc4cb400640 1 -- 172.21.15.40:0/1531616171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4b0004340 msgr2=0x7fc4b0004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:07.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.095+0000 7fc4cb400640 1 --2- 172.21.15.40:0/1531616171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4b0004340 0x7fc4b0004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.095+0000 7fc4cb400640 1 -- 172.21.15.40:0/1531616171 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b0003520 con 0x7fc4b0005020 2024-04-26T23:07:07.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.095+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/1531616171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4b0004340 0x7fc4b0004820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:07.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.096+0000 7fc4cb400640 1 --2- 172.21.15.40:0/1531616171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b0005020 0x7fc4b000b9c0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fc4c400a850 tx=0x7fc4b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff687c6e4be45b90 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:07.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.096+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c400a4a0 con 0x7fc4b0005020 2024-04-26T23:07:07.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.096+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4c402f070 con 0x7fc4b0005020 2024-04-26T23:07:07.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.096+0000 7fc4ca000640 1 -- 172.21.15.40:0/1531616171 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c402bd20 con 0x7fc4b0005020 2024-04-26T23:07:07.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.097+0000 7fc4cbe00640 1 -- 172.21.15.40:0/1531616171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b0005020 msgr2=0x7fc4b000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:07.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.097+0000 7fc4cbe00640 1 --2- 172.21.15.40:0/1531616171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b0005020 0x7fc4b000b9c0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fc4c400a850 tx=0x7fc4b0005830 comp rx=0 tx=0).stop 2024-04-26T23:07:07.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.097+0000 7fc4cb400640 1 -- 172.21.15.40:0/1531616171 reap_dead start 2024-04-26T23:07:07.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.097+0000 7fc4cbe00640 1 -- 172.21.15.40:0/1531616171 shutdown_connections 2024-04-26T23:07:07.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.097+0000 7fc4cbe00640 1 -- 172.21.15.40:0/1531616171 >> 172.21.15.40:0/1531616171 conn(0x7fc4cc0b2720 msgr2=0x7fc4cc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:07.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.097+0000 7fc4cbe00640 1 -- 172.21.15.40:0/1531616171 shutdown_connections 2024-04-26T23:07:07.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.097+0000 7fc4cbe00640 1 -- 172.21.15.40:0/1531616171 wait complete. 2024-04-26T23:07:07.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.099+0000 7fc4cbe00640 1 Processor -- start 2024-04-26T23:07:07.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.099+0000 7fc4cbe00640 1 -- start start 2024-04-26T23:07:07.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.100+0000 7fc4cbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4b0005020 0x7fc4cc143f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.100+0000 7fc4cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4cc1404a0 0x7fc4cc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.100+0000 7fc4cbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4cc142560 0x7fc4cc144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.100+0000 7fc4cbe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc06e1f0 con 0x7fc4cc1404a0 2024-04-26T23:07:07.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.100+0000 7fc4cbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc0722a0 con 0x7fc4cc142560 2024-04-26T23:07:07.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.100+0000 7fc4cbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc4cc1468f0 con 0x7fc4b0005020 2024-04-26T23:07:07.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.101+0000 7fc4c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4cc1404a0 0x7fc4cc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.101+0000 7fc4c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4cc1404a0 0x7fc4cc15c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54354/0 (socket says 172.21.15.40:54354) 2024-04-26T23:07:07.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.101+0000 7fc4c3e00640 1 -- 172.21.15.40:0/120086356 learned_addr learned my addr 172.21.15.40:0/120086356 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:07.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.101+0000 7fc4cb400640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4cc142560 0x7fc4cc144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.101+0000 7fc4caa00640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4b0005020 0x7fc4cc143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.101+0000 7fc4c3e00640 1 -- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4b0005020 msgr2=0x7fc4cc143f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:07.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.101+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4b0005020 0x7fc4cc143f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.101+0000 7fc4c3e00640 1 -- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4cc142560 msgr2=0x7fc4cc144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:07.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.101+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4cc142560 0x7fc4cc144e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.101+0000 7fc4c3e00640 1 -- 172.21.15.40:0/120086356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4cc142e60 con 0x7fc4cc1404a0 2024-04-26T23:07:07.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.101+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4cc1404a0 0x7fc4cc15c600 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4010880 tx=0x7fc4b4010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:07.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.102+0000 7fc4c2a00640 1 -- 172.21.15.40:0/120086356 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b400f040 con 0x7fc4cc1404a0 2024-04-26T23:07:07.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.102+0000 7fc4c2a00640 1 -- 172.21.15.40:0/120086356 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4b400eb60 con 0x7fc4cc1404a0 2024-04-26T23:07:07.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.102+0000 7fc4c2a00640 1 -- 172.21.15.40:0/120086356 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b401c400 con 0x7fc4cc1404a0 2024-04-26T23:07:07.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.102+0000 7fc4cbe00640 1 -- 172.21.15.40:0/120086356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4cc143140 con 0x7fc4cc1404a0 2024-04-26T23:07:07.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.102+0000 7fc4cbe00640 1 -- 172.21.15.40:0/120086356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4cc143700 con 0x7fc4cc1404a0 2024-04-26T23:07:07.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.103+0000 7fc4cbe00640 1 -- 172.21.15.40:0/120086356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4cc069370 con 0x7fc4cc1404a0 2024-04-26T23:07:07.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.105+0000 7fc4c2a00640 1 -- 172.21.15.40:0/120086356 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc4b401c5a0 con 0x7fc4cc1404a0 2024-04-26T23:07:07.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.105+0000 7fc4c2a00640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc4a806c8f0 0x7fc4a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.105+0000 7fc4c2a00640 1 -- 172.21.15.40:0/120086356 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fc4b408c4b0 con 0x7fc4cc1404a0 2024-04-26T23:07:07.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.106+0000 7fc4caa00640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc4a806c8f0 0x7fc4a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.108+0000 7fc4caa00640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc4a806c8f0 0x7fc4a806edb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc007fd0 tx=0x7fc4bc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:07.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.110+0000 7fc4c2a00640 1 -- 172.21.15.40:0/120086356 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc4b4018030 con 0x7fc4cc1404a0 2024-04-26T23:07:07.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.356+0000 7fc4cbe00640 1 -- 172.21.15.40:0/120086356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fc4cc06fa60 con 0x7fc4cc1404a0 2024-04-26T23:07:07.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.502+0000 7fc4c2a00640 1 -- 172.21.15.40:0/120086356 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v75) ==== 105+0+0 (secure 0 0 0) 0x7fc4b405a6b0 con 0x7fc4cc1404a0 2024-04-26T23:07:07.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.506+0000 7fc4cbe00640 1 -- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc4a806c8f0 msgr2=0x7fc4a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:07.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.506+0000 7fc4cbe00640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc4a806c8f0 0x7fc4a806edb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc007fd0 tx=0x7fc4bc009040 comp rx=0 tx=0).stop 2024-04-26T23:07:07.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.506+0000 7fc4cbe00640 1 -- 172.21.15.40:0/120086356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4cc1404a0 msgr2=0x7fc4cc15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:07.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.506+0000 7fc4cbe00640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4cc1404a0 0x7fc4cc15c600 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4010880 tx=0x7fc4b4010d40 comp rx=0 tx=0).stop 2024-04-26T23:07:07.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.506+0000 7fc4cbe00640 1 -- 172.21.15.40:0/120086356 shutdown_connections 2024-04-26T23:07:07.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.506+0000 7fc4cbe00640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc4a806c8f0 0x7fc4a806edb0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.506+0000 7fc4cbe00640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4cc142560 0x7fc4cc144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.506+0000 7fc4cbe00640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4cc1404a0 0x7fc4cc15c600 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.506+0000 7fc4cbe00640 1 --2- 172.21.15.40:0/120086356 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc4b0005020 0x7fc4cc143f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.506+0000 7fc4cbe00640 1 -- 172.21.15.40:0/120086356 >> 172.21.15.40:0/120086356 conn(0x7fc4cc0b2720 msgr2=0x7fc4cc06f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:07.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.506+0000 7fc4cbe00640 1 -- 172.21.15.40:0/120086356 shutdown_connections 2024-04-26T23:07:07.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.506+0000 7fc4cbe00640 1 -- 172.21.15.40:0/120086356 wait complete. 2024-04-26T23:07:07.533 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:07.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.722+0000 7fe2ecc00640 1 Processor -- start 2024-04-26T23:07:07.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.722+0000 7fe2ecc00640 1 -- start start 2024-04-26T23:07:07.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.723+0000 7fe2ecc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2e80c41d0 0x7fe2e80cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.724+0000 7fe2ecc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2e81678f0 0x7fe2e80cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.724+0000 7fe2ecc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2e8163a50 con 0x7fe2e8073f30 2024-04-26T23:07:07.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.724+0000 7fe2ecc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe2e80cf7a0 con 0x7fe2e80c41d0 2024-04-26T23:07:07.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.724+0000 7fe2ecc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e80ceb90 con 0x7fe2e81678f0 2024-04-26T23:07:07.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.724+0000 7fe2e5e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2e80c41d0 0x7fe2e80cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.724+0000 7fe2e5400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe2e8073f30 0x7fe2e81671e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43186/0 (socket says 172.21.15.40:43186) 2024-04-26T23:07:07.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.724+0000 7fe2e5400640 1 -- 172.21.15.40:0/337300908 learned_addr learned my addr 172.21.15.40:0/337300908 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:07.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.724+0000 7fe2e5e00640 1 -- 172.21.15.40:0/337300908 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2e81678f0 msgr2=0x7fe2e80cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:07.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.724+0000 7fe2e6800640 1 --2- 172.21.15.40:0/337300908 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2e81678f0 0x7fe2e80cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.725+0000 7fe2e5e00640 1 --2- 172.21.15.40:0/337300908 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2e81678f0 0x7fe2e80cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.725+0000 7fe2e5e00640 1 -- 172.21.15.40:0/337300908 >> v1:172.21.15.40:6789/0 conn(0x7fe2e8073f30 legacy=0x7fe2e81671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:07.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.725+0000 7fe2e5e00640 1 -- 172.21.15.40:0/337300908 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e80cee70 con 0x7fe2e80c41d0 2024-04-26T23:07:07.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.725+0000 7fe2e6800640 1 --2- 172.21.15.40:0/337300908 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2e81678f0 0x7fe2e80cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:07.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.725+0000 7fe2e4a00640 1 -- 172.21.15.40:0/337300908 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 468684094 0 0) 0x7fe2e8163a50 con 0x7fe2e8073f30 2024-04-26T23:07:07.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.726+0000 7fe2e5e00640 1 --2- 172.21.15.40:0/337300908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2e80c41d0 0x7fe2e80cfc30 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80097a0 tx=0x7fe2d802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14335cfc9f6e6cdc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:07.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.726+0000 7fe2e4a00640 1 -- 172.21.15.40:0/337300908 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8037070 con 0x7fe2e80c41d0 2024-04-26T23:07:07.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.726+0000 7fe2e4a00640 1 -- 172.21.15.40:0/337300908 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d8032440 con 0x7fe2e80c41d0 2024-04-26T23:07:07.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.726+0000 7fe2e4a00640 1 -- 172.21.15.40:0/337300908 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8032810 con 0x7fe2e80c41d0 2024-04-26T23:07:07.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.727+0000 7fe2ecc00640 1 -- 172.21.15.40:0/337300908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2e80c41d0 msgr2=0x7fe2e80cfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:07.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.727+0000 7fe2ecc00640 1 --2- 172.21.15.40:0/337300908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2e80c41d0 0x7fe2e80cfc30 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80097a0 tx=0x7fe2d802a180 comp rx=0 tx=0).stop 2024-04-26T23:07:07.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.727+0000 7fe2ecc00640 1 -- 172.21.15.40:0/337300908 shutdown_connections 2024-04-26T23:07:07.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.727+0000 7fe2ecc00640 1 --2- 172.21.15.40:0/337300908 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2e81678f0 0x7fe2e80cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.727+0000 7fe2ecc00640 1 --2- 172.21.15.40:0/337300908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2e80c41d0 0x7fe2e80cfc30 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.727+0000 7fe2ecc00640 1 -- 172.21.15.40:0/337300908 >> 172.21.15.40:0/337300908 conn(0x7fe2e80d0be0 msgr2=0x7fe2e80d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:07.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.727+0000 7fe2ecc00640 1 -- 172.21.15.40:0/337300908 shutdown_connections 2024-04-26T23:07:07.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.727+0000 7fe2ecc00640 1 -- 172.21.15.40:0/337300908 wait complete. 2024-04-26T23:07:07.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.729+0000 7fe2ecc00640 1 Processor -- start 2024-04-26T23:07:07.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.729+0000 7fe2ecc00640 1 -- start start 2024-04-26T23:07:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.730+0000 7fe2ecc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2e8073f30 0x7fe2e80c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.730+0000 7fe2ecc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2e80c41d0 0x7fe2e80cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.730+0000 7fe2ecc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2e81678f0 0x7fe2e8154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.730+0000 7fe2ecc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e80d0170 con 0x7fe2e81678f0 2024-04-26T23:07:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.730+0000 7fe2ecc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e80cf430 con 0x7fe2e80c41d0 2024-04-26T23:07:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.730+0000 7fe2ecc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e80ce7b0 con 0x7fe2e8073f30 2024-04-26T23:07:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.730+0000 7fe2e6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2e81678f0 0x7fe2e8154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.730+0000 7fe2e6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2e81678f0 0x7fe2e8154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54368/0 (socket says 172.21.15.40:54368) 2024-04-26T23:07:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.730+0000 7fe2e6800640 1 -- 172.21.15.40:0/3720269168 learned_addr learned my addr 172.21.15.40:0/3720269168 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.731+0000 7fe2e5e00640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2e8073f30 0x7fe2e80c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.731+0000 7fe2e5400640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2e80c41d0 0x7fe2e80cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.731+0000 7fe2e6800640 1 -- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2e8073f30 msgr2=0x7fe2e80c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:07.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.731+0000 7fe2e6800640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2e8073f30 0x7fe2e80c8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.731+0000 7fe2e6800640 1 -- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2e80c41d0 msgr2=0x7fe2e80cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:07.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.731+0000 7fe2e6800640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2e80c41d0 0x7fe2e80cab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.731+0000 7fe2e6800640 1 -- 172.21.15.40:0/3720269168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e80cb090 con 0x7fe2e81678f0 2024-04-26T23:07:07.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.731+0000 7fe2e5400640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2e80c41d0 0x7fe2e80cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:07.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.731+0000 7fe2e6800640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2e81678f0 0x7fe2e8154f90 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000cd80 tx=0x7fe2e000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:07.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.731+0000 7fe2de000640 1 -- 172.21.15.40:0/3720269168 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e000f680 con 0x7fe2e81678f0 2024-04-26T23:07:07.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.731+0000 7fe2de000640 1 -- 172.21.15.40:0/3720269168 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e000f840 con 0x7fe2e81678f0 2024-04-26T23:07:07.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.731+0000 7fe2de000640 1 -- 172.21.15.40:0/3720269168 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e000fc30 con 0x7fe2e81678f0 2024-04-26T23:07:07.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.731+0000 7fe2ecc00640 1 -- 172.21.15.40:0/3720269168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2e80cb350 con 0x7fe2e81678f0 2024-04-26T23:07:07.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.732+0000 7fe2ecc00640 1 -- 172.21.15.40:0/3720269168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2e80cb910 con 0x7fe2e81678f0 2024-04-26T23:07:07.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.733+0000 7fe2ecc00640 1 -- 172.21.15.40:0/3720269168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2e80694f0 con 0x7fe2e81678f0 2024-04-26T23:07:07.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.734+0000 7fe2de000640 1 -- 172.21.15.40:0/3720269168 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2e00213f0 con 0x7fe2e81678f0 2024-04-26T23:07:07.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.735+0000 7fe2de000640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe2c406c8f0 0x7fe2c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:07.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.735+0000 7fe2de000640 1 -- 172.21.15.40:0/3720269168 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fe2e008caa0 con 0x7fe2e81678f0 2024-04-26T23:07:07.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.736+0000 7fe2e5e00640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe2c406c8f0 0x7fe2c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:07.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.737+0000 7fe2e5e00640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe2c406c8f0 0x7fe2c406edb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8034000 tx=0x7fe2d802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:07.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.740+0000 7fe2de000640 1 -- 172.21.15.40:0/3720269168 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe2e0014030 con 0x7fe2e81678f0 2024-04-26T23:07:07.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.984+0000 7fe2ecc00640 1 -- 172.21.15.40:0/3720269168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2e80d0430 con 0x7fe2e81678f0 2024-04-26T23:07:07.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.986+0000 7fe2de000640 1 -- 172.21.15.40:0/3720269168 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5199 (secure 0 0 0) 0x7fe2e005ad30 con 0x7fe2e81678f0 2024-04-26T23:07:07.986 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:07.986 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":76,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/755627718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":755627718},{"type":"v1","addr":"172.21.15.103:6837","nonce":755627718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":5965,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/2914603370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2914603370},{"type":"v1","addr":"172.21.15.103:6836","nonce":2914603370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:06:57.853769+0000","modified":"2024-04-26T23:07:07.506162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1],"up":{"mds_0":7446,"mds_1":7443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7443":{"gid":7443,"name":"a","rank":1,"incarnation":76,"state":"up:creating","state_seq":1,"addr":"172.21.15.40:6834/216893598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":216893598},{"type":"v1","addr":"172.21.15.40:6834","nonce":216893598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7446":{"gid":7446,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4227275373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":4227275373},{"type":"v1","addr":"172.21.15.40:6835","nonce":4227275373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7446,"qdb_cluster":[7446]},"id":5}]} 2024-04-26T23:07:07.987 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 76 2024-04-26T23:07:07.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.989+0000 7fe2ecc00640 1 -- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe2c406c8f0 msgr2=0x7fe2c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:07.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.989+0000 7fe2ecc00640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe2c406c8f0 0x7fe2c406edb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8034000 tx=0x7fe2d802e000 comp rx=0 tx=0).stop 2024-04-26T23:07:07.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.989+0000 7fe2ecc00640 1 -- 172.21.15.40:0/3720269168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2e81678f0 msgr2=0x7fe2e8154f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:07.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.989+0000 7fe2ecc00640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2e81678f0 0x7fe2e8154f90 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000cd80 tx=0x7fe2e000a650 comp rx=0 tx=0).stop 2024-04-26T23:07:07.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.989+0000 7fe2ecc00640 1 -- 172.21.15.40:0/3720269168 shutdown_connections 2024-04-26T23:07:07.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.989+0000 7fe2ecc00640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe2c406c8f0 0x7fe2c406edb0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.989+0000 7fe2ecc00640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2e81678f0 0x7fe2e8154f90 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.989+0000 7fe2ecc00640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2e80c41d0 0x7fe2e80cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.989+0000 7fe2ecc00640 1 --2- 172.21.15.40:0/3720269168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2e8073f30 0x7fe2e80c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:07.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.989+0000 7fe2ecc00640 1 -- 172.21.15.40:0/3720269168 >> 172.21.15.40:0/3720269168 conn(0x7fe2e80d0be0 msgr2=0x7fe2e8162940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:07.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.989+0000 7fe2ecc00640 1 -- 172.21.15.40:0/3720269168 shutdown_connections 2024-04-26T23:07:07.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:07.989+0000 7fe2ecc00640 1 -- 172.21.15.40:0/3720269168 wait complete. 2024-04-26T23:07:08.411 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:06:57.853769+0000', 'modified': '2024-04-26T23:07:07.506162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 7446, 'mds_1': 7443}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7443': {'gid': 7443, 'name': 'a', 'rank': 1, 'incarnation': 76, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.40:6834/216893598', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 216893598}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 216893598}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7446': {'gid': 7446, 'name': 'c', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/4227275373', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 4227275373}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 4227275373}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7446, 'qdb_cluster': [7446]} 2024-04-26T23:07:08.411 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7443:up:creating 2024-04-26T23:07:09.412 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:09.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.615+0000 7f049be00640 1 Processor -- start 2024-04-26T23:07:09.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.615+0000 7f049be00640 1 -- start start 2024-04-26T23:07:09.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.616+0000 7f049be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f049c161850 0x7f049c0d1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:09.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.617+0000 7f049be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f049c0b4770 0x7f049c0d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:09.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.617+0000 7f049be00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f049c0b4110 con 0x7f049c0d1ec0 2024-04-26T23:07:09.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.617+0000 7f049be00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f049c0d3850 con 0x7f049c161850 2024-04-26T23:07:09.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.617+0000 7f049be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f049c0d3b30 con 0x7f049c0b4770 2024-04-26T23:07:09.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.617+0000 7f049b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f049c0d1ec0 0x7f049c0d0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43192/0 (socket says 172.21.15.40:43192) 2024-04-26T23:07:09.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.617+0000 7f049b400640 1 -- 172.21.15.40:0/1827068543 learned_addr learned my addr 172.21.15.40:0/1827068543 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:09.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.617+0000 7f049aa00640 1 --2- 172.21.15.40:0/1827068543 >> v2:172.21.15.40:3300/0 conn(0x7f049c161850 0x7f049c0d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:09.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.617+0000 7f049a000640 1 --2- 172.21.15.40:0/1827068543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f049c0b4770 0x7f049c0d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:09.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.618+0000 7f049aa00640 1 -- 172.21.15.40:0/1827068543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f049c0b4770 msgr2=0x7f049c0d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:09.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.618+0000 7f049aa00640 1 --2- 172.21.15.40:0/1827068543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f049c0b4770 0x7f049c0d1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:09.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.618+0000 7f049aa00640 1 -- 172.21.15.40:0/1827068543 >> v1:172.21.15.40:6789/0 conn(0x7f049c0d1ec0 legacy=0x7f049c0d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:09.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.618+0000 7f049aa00640 1 -- 172.21.15.40:0/1827068543 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049c162c40 con 0x7f049c161850 2024-04-26T23:07:09.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.618+0000 7f0499600640 1 -- 172.21.15.40:0/1827068543 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1574299034 0 0) 0x7f049c0b4110 con 0x7f049c0d1ec0 2024-04-26T23:07:09.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.619+0000 7f049a000640 1 --2- 172.21.15.40:0/1827068543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f049c0b4770 0x7f049c0d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:09.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.619+0000 7f049aa00640 1 --2- 172.21.15.40:0/1827068543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f049c161850 0x7f049c0d1440 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f049c0b42e0 tx=0x7f048c02a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce87a4431c78654 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.619+0000 7f0499600640 1 -- 172.21.15.40:0/1827068543 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c02ae60 con 0x7f049c161850 2024-04-26T23:07:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.619+0000 7f0499600640 1 -- 172.21.15.40:0/1827068543 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f048c007740 con 0x7f049c161850 2024-04-26T23:07:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.619+0000 7f0499600640 1 -- 172.21.15.40:0/1827068543 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c007b10 con 0x7f049c161850 2024-04-26T23:07:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.620+0000 7f049be00640 1 -- 172.21.15.40:0/1827068543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f049c161850 msgr2=0x7f049c0d1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.620+0000 7f049be00640 1 --2- 172.21.15.40:0/1827068543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f049c161850 0x7f049c0d1440 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f049c0b42e0 tx=0x7f048c02a640 comp rx=0 tx=0).stop 2024-04-26T23:07:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.620+0000 7f049be00640 1 -- 172.21.15.40:0/1827068543 shutdown_connections 2024-04-26T23:07:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.620+0000 7f049be00640 1 --2- 172.21.15.40:0/1827068543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f049c0b4770 0x7f049c0d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.620+0000 7f049be00640 1 --2- 172.21.15.40:0/1827068543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f049c161850 0x7f049c0d1440 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.620+0000 7f049be00640 1 -- 172.21.15.40:0/1827068543 >> 172.21.15.40:0/1827068543 conn(0x7f049c0d4e10 msgr2=0x7f049c0d5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.620+0000 7f049be00640 1 -- 172.21.15.40:0/1827068543 shutdown_connections 2024-04-26T23:07:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.620+0000 7f049be00640 1 -- 172.21.15.40:0/1827068543 wait complete. 2024-04-26T23:07:09.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.623+0000 7f049be00640 1 Processor -- start 2024-04-26T23:07:09.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.623+0000 7f049be00640 1 -- start start 2024-04-26T23:07:09.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.624+0000 7f049be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c0b4770 0x7f049c1604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:09.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.624+0000 7f049be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f049c0d1ec0 0x7f049c0a27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:09.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.624+0000 7f049be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f049c161850 0x7f049c0b1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:09.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.624+0000 7f049be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f049c0d3f90 con 0x7f049c0d1ec0 2024-04-26T23:07:09.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.624+0000 7f049be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f049c0d3e10 con 0x7f049c0b4770 2024-04-26T23:07:09.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.624+0000 7f049be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f049c0b63e0 con 0x7f049c161850 2024-04-26T23:07:09.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.625+0000 7f049a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f049c0d1ec0 0x7f049c0a27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:09.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.625+0000 7f049a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f049c0d1ec0 0x7f049c0a27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54374/0 (socket says 172.21.15.40:54374) 2024-04-26T23:07:09.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.625+0000 7f049a000640 1 -- 172.21.15.40:0/1705674061 learned_addr learned my addr 172.21.15.40:0/1705674061 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:09.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.625+0000 7f049aa00640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c0b4770 0x7f049c1604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:09.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.625+0000 7f049a000640 1 -- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f049c161850 msgr2=0x7f049c0b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:09.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.625+0000 7f049a000640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f049c161850 0x7f049c0b1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.625+0000 7f049a000640 1 -- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c0b4770 msgr2=0x7f049c1604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.625+0000 7f049a000640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c0b4770 0x7f049c1604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.625+0000 7f049a000640 1 -- 172.21.15.40:0/1705674061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049c0a02e0 con 0x7f049c0d1ec0 2024-04-26T23:07:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.625+0000 7f049aa00640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c0b4770 0x7f049c1604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.626+0000 7f049a000640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f049c0d1ec0 0x7f049c0a27f0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f048800c820 tx=0x7f048800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.626+0000 7f0492a00640 1 -- 172.21.15.40:0/1705674061 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0488015070 con 0x7f049c0d1ec0 2024-04-26T23:07:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.626+0000 7f0492a00640 1 -- 172.21.15.40:0/1705674061 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0488007a30 con 0x7f049c0d1ec0 2024-04-26T23:07:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.626+0000 7f049be00640 1 -- 172.21.15.40:0/1705674061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f049c0a05c0 con 0x7f049c0d1ec0 2024-04-26T23:07:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.626+0000 7f0492a00640 1 -- 172.21.15.40:0/1705674061 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0488007e00 con 0x7f049c0d1ec0 2024-04-26T23:07:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.626+0000 7f049be00640 1 -- 172.21.15.40:0/1705674061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f049c0a1ff0 con 0x7f049c0d1ec0 2024-04-26T23:07:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.628+0000 7f049be00640 1 -- 172.21.15.40:0/1705674061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f049c069a80 con 0x7f049c0d1ec0 2024-04-26T23:07:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.628+0000 7f0492a00640 1 -- 172.21.15.40:0/1705674061 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0488018070 con 0x7f049c0d1ec0 2024-04-26T23:07:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.630+0000 7f0492a00640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f047c06c8f0 0x7f047c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.630+0000 7f0492a00640 1 -- 172.21.15.40:0/1705674061 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f048808bc60 con 0x7f049c0d1ec0 2024-04-26T23:07:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.631+0000 7f049aa00640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f047c06c8f0 0x7f047c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:09.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.632+0000 7f049aa00640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f047c06c8f0 0x7f047c06edb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f048c034000 tx=0x7f048c033040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:09.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.633+0000 7f0492a00640 1 -- 172.21.15.40:0/1705674061 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0488059f00 con 0x7f049c0d1ec0 2024-04-26T23:07:09.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.876+0000 7f049be00640 1 -- 172.21.15.40:0/1705674061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f049c0d3090 con 0x7f049c0d1ec0 2024-04-26T23:07:09.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.877+0000 7f0492a00640 1 -- 172.21.15.40:0/1705674061 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+5221 (secure 0 0 0) 0x7f04880600d0 con 0x7f049c0d1ec0 2024-04-26T23:07:09.878 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:09.878 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":79,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/755627718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":755627718},{"type":"v1","addr":"172.21.15.103:6837","nonce":755627718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:06:57.853769+0000","modified":"2024-04-26T23:07:09.523854+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7446,"mds_1":7443,"mds_2":5965},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5965":{"gid":5965,"name":"d","rank":2,"incarnation":78,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6836/2914603370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2914603370},{"type":"v1","addr":"172.21.15.103:6836","nonce":2914603370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7443":{"gid":7443,"name":"a","rank":1,"incarnation":76,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/216893598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":216893598},{"type":"v1","addr":"172.21.15.40:6834","nonce":216893598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7446":{"gid":7446,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4227275373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":4227275373},{"type":"v1","addr":"172.21.15.40:6835","nonce":4227275373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7446,"qdb_cluster":[5965,7443,7446]},"id":5}]} 2024-04-26T23:07:09.878 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 79 2024-04-26T23:07:09.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.880+0000 7f049be00640 1 -- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f047c06c8f0 msgr2=0x7f047c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:09.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.880+0000 7f049be00640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f047c06c8f0 0x7f047c06edb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f048c034000 tx=0x7f048c033040 comp rx=0 tx=0).stop 2024-04-26T23:07:09.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.880+0000 7f049be00640 1 -- 172.21.15.40:0/1705674061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f049c0d1ec0 msgr2=0x7f049c0a27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:09.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.880+0000 7f049be00640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f049c0d1ec0 0x7f049c0a27f0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f048800c820 tx=0x7f048800cce0 comp rx=0 tx=0).stop 2024-04-26T23:07:09.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.880+0000 7f049be00640 1 -- 172.21.15.40:0/1705674061 shutdown_connections 2024-04-26T23:07:09.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.880+0000 7f049be00640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f047c06c8f0 0x7f047c06edb0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:09.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.880+0000 7f049be00640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f049c161850 0x7f049c0b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:09.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.880+0000 7f049be00640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f049c0d1ec0 0x7f049c0a27f0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:09.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.880+0000 7f049be00640 1 --2- 172.21.15.40:0/1705674061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f049c0b4770 0x7f049c1604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:09.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.880+0000 7f049be00640 1 -- 172.21.15.40:0/1705674061 >> 172.21.15.40:0/1705674061 conn(0x7f049c0d4e10 msgr2=0x7f049c0b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:09.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.880+0000 7f049be00640 1 -- 172.21.15.40:0/1705674061 shutdown_connections 2024-04-26T23:07:09.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:09.880+0000 7f049be00640 1 -- 172.21.15.40:0/1705674061 wait complete. 2024-04-26T23:07:10.302 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:06:57.853769+0000', 'modified': '2024-04-26T23:07:09.523854+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 7446, 'mds_1': 7443, 'mds_2': 5965}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5965': {'gid': 5965, 'name': 'd', 'rank': 2, 'incarnation': 78, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.103:6836/2914603370', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6834', 'nonce': 2914603370}, {'type': 'v1', 'addr': '172.21.15.103:6836', 'nonce': 2914603370}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7443': {'gid': 7443, 'name': 'a', 'rank': 1, 'incarnation': 76, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.40:6834/216893598', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 216893598}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 216893598}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7446': {'gid': 7446, 'name': 'c', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/4227275373', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 4227275373}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 4227275373}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7446, 'qdb_cluster': [5965, 7443, 7446]} 2024-04-26T23:07:10.302 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T23:07:10.302 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-26T23:07:10.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.500+0000 7f40ba000640 1 Processor -- start 2024-04-26T23:07:10.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.500+0000 7f40ba000640 1 -- start start 2024-04-26T23:07:10.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.502+0000 7f40ba000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f40b4162810 0x7f40b4070350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.502+0000 7f40ba000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f40b40d1b30 0x7f40b40d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.502+0000 7f40ba000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40b40d10d0 con 0x7f40b40b2340 2024-04-26T23:07:10.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.502+0000 7f40ba000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f40b40d0750 con 0x7f40b4162810 2024-04-26T23:07:10.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.502+0000 7f40ba000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f40b40d16a0 con 0x7f40b40d1b30 2024-04-26T23:07:10.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.502+0000 7f40b2a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f40b40b2340 0x7f40b40cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43202/0 (socket says 172.21.15.40:43202) 2024-04-26T23:07:10.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.502+0000 7f40b2a00640 1 -- 172.21.15.40:0/1296342742 learned_addr learned my addr 172.21.15.40:0/1296342742 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:10.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.502+0000 7f40b3400640 1 --2- 172.21.15.40:0/1296342742 >> v2:172.21.15.40:3300/0 conn(0x7f40b4162810 0x7f40b4070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.503+0000 7f40b3e00640 1 --2- 172.21.15.40:0/1296342742 >> v2:172.21.15.103:3300/0 conn(0x7f40b40d1b30 0x7f40b40d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.503+0000 7f40b3400640 1 -- 172.21.15.40:0/1296342742 >> v2:172.21.15.103:3300/0 conn(0x7f40b40d1b30 msgr2=0x7f40b40d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.503+0000 7f40b3400640 1 --2- 172.21.15.40:0/1296342742 >> v2:172.21.15.103:3300/0 conn(0x7f40b40d1b30 0x7f40b40d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.503+0000 7f40b3400640 1 -- 172.21.15.40:0/1296342742 >> v1:172.21.15.40:6789/0 conn(0x7f40b40b2340 legacy=0x7f40b40cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.503+0000 7f40b3400640 1 -- 172.21.15.40:0/1296342742 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b40d1370 con 0x7f40b4162810 2024-04-26T23:07:10.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.503+0000 7f40b3e00640 1 --2- 172.21.15.40:0/1296342742 >> v2:172.21.15.103:3300/0 conn(0x7f40b40d1b30 0x7f40b40d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:10.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.504+0000 7f40b3400640 1 --2- 172.21.15.40:0/1296342742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40b4162810 0x7f40b4070350 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f40b4073330 tx=0x7f40a402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5048967fda0059f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.504+0000 7f40b2000640 1 -- 172.21.15.40:0/1296342742 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a4037070 con 0x7f40b4162810 2024-04-26T23:07:10.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.504+0000 7f40b2000640 1 -- 172.21.15.40:0/1296342742 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a4032440 con 0x7f40b4162810 2024-04-26T23:07:10.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.504+0000 7f40b2000640 1 -- 172.21.15.40:0/1296342742 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a4032810 con 0x7f40b4162810 2024-04-26T23:07:10.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.505+0000 7f40ba000640 1 -- 172.21.15.40:0/1296342742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40b4162810 msgr2=0x7f40b4070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.505+0000 7f40ba000640 1 --2- 172.21.15.40:0/1296342742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40b4162810 0x7f40b4070350 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f40b4073330 tx=0x7f40a402a1a0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.505+0000 7f40ba000640 1 -- 172.21.15.40:0/1296342742 shutdown_connections 2024-04-26T23:07:10.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.505+0000 7f40ba000640 1 --2- 172.21.15.40:0/1296342742 >> v2:172.21.15.103:3300/0 conn(0x7f40b40d1b30 0x7f40b40d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.505+0000 7f40ba000640 1 --2- 172.21.15.40:0/1296342742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40b4162810 0x7f40b4070350 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.505+0000 7f40ba000640 1 -- 172.21.15.40:0/1296342742 >> 172.21.15.40:0/1296342742 conn(0x7f40b40d2da0 msgr2=0x7f40b40d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:10.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.505+0000 7f40ba000640 1 -- 172.21.15.40:0/1296342742 shutdown_connections 2024-04-26T23:07:10.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.505+0000 7f40ba000640 1 -- 172.21.15.40:0/1296342742 wait complete. 2024-04-26T23:07:10.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.507+0000 7f40ba000640 1 Processor -- start 2024-04-26T23:07:10.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.507+0000 7f40ba000640 1 -- start start 2024-04-26T23:07:10.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40ba000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40b40b2340 0x7f40b40c9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40ba000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40b40d1b30 0x7f40b4160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40ba000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40b4162810 0x7f40b406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40ba000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f40b4070890 con 0x7f40b40b2340 2024-04-26T23:07:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40ba000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40b40d0470 con 0x7f40b40d1b30 2024-04-26T23:07:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40ba000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f40b40cdb00 con 0x7f40b4162810 2024-04-26T23:07:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40b3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40b40b2340 0x7f40b40c9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40b3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40b40b2340 0x7f40b40c9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54392/0 (socket says 172.21.15.40:54392) 2024-04-26T23:07:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40b3400640 1 -- 172.21.15.40:0/4157346052 learned_addr learned my addr 172.21.15.40:0/4157346052 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40b3e00640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40b4162810 0x7f40b406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40b2a00640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40b40d1b30 0x7f40b4160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40b3400640 1 -- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40b4162810 msgr2=0x7f40b406d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40b3400640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40b4162810 0x7f40b406d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40b3400640 1 -- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40b40d1b30 msgr2=0x7f40b4160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40b3400640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40b40d1b30 0x7f40b4160ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.508+0000 7f40b3400640 1 -- 172.21.15.40:0/4157346052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b4161b80 con 0x7f40b40b2340 2024-04-26T23:07:10.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.509+0000 7f40b3400640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40b40b2340 0x7f40b40c9020 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f40a4000c00 tx=0x7f40a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.509+0000 7f40ab400640 1 -- 172.21.15.40:0/4157346052 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a4037040 con 0x7f40b40b2340 2024-04-26T23:07:10.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.509+0000 7f40ab400640 1 -- 172.21.15.40:0/4157346052 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a4042070 con 0x7f40b40b2340 2024-04-26T23:07:10.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.509+0000 7f40ab400640 1 -- 172.21.15.40:0/4157346052 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a4030a50 con 0x7f40b40b2340 2024-04-26T23:07:10.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.509+0000 7f40ba000640 1 -- 172.21.15.40:0/4157346052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b41615c0 con 0x7f40b40b2340 2024-04-26T23:07:10.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.509+0000 7f40ba000640 1 -- 172.21.15.40:0/4157346052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b40c9f80 con 0x7f40b40b2340 2024-04-26T23:07:10.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.512+0000 7f40ab400640 1 -- 172.21.15.40:0/4157346052 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f40a4030bf0 con 0x7f40b40b2340 2024-04-26T23:07:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.513+0000 7f40ab400640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f409406c8f0 0x7f409406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.513+0000 7f40ab400640 1 -- 172.21.15.40:0/4157346052 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f40a40ac780 con 0x7f40b40b2340 2024-04-26T23:07:10.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.514+0000 7f40bcd99740 1 Processor -- start 2024-04-26T23:07:10.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.514+0000 7f40bcd99740 1 -- start start 2024-04-26T23:07:10.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.514+0000 7f40b2a00640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f409406c8f0 0x7f409406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.515+0000 7f40bcd99740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c1302c5e0 0x560c1302e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.515+0000 7f40bcd99740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c13068cb0 0x560c1302c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.515+0000 7f40bcd99740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c130570a0 0x560c13057520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.515+0000 7f40bcd99740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560c12f90600 con 0x560c1302c5e0 2024-04-26T23:07:10.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.515+0000 7f40bcd99740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560c12f2d100 con 0x560c13068cb0 2024-04-26T23:07:10.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.515+0000 7f40bcd99740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x560c12f8c5c0 con 0x560c130570a0 2024-04-26T23:07:10.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.515+0000 7f40b2a00640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f409406c8f0 0x7f409406edb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f40b406e550 tx=0x7f40a0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.515+0000 7f40b3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c1302c5e0 0x560c1302e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.515+0000 7f40b3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c1302c5e0 0x560c1302e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54408/0 (socket says 172.21.15.40:54408) 2024-04-26T23:07:10.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.515+0000 7f40b3400640 1 -- 172.21.15.40:0/3857704363 learned_addr learned my addr 172.21.15.40:0/3857704363 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:10.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.516+0000 7f40b3400640 1 -- 172.21.15.40:0/3857704363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c130570a0 msgr2=0x560c13057520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.516+0000 7f40b3400640 1 --2- 172.21.15.40:0/3857704363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c130570a0 0x560c13057520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.516+0000 7f40b3400640 1 -- 172.21.15.40:0/3857704363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c13068cb0 msgr2=0x560c1302c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.516+0000 7f40b3400640 1 --2- 172.21.15.40:0/3857704363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c13068cb0 0x560c1302c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.516+0000 7f40b3400640 1 -- 172.21.15.40:0/3857704363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c12ca22d0 con 0x560c1302c5e0 2024-04-26T23:07:10.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.516+0000 7f40b3400640 1 --2- 172.21.15.40:0/3857704363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c1302c5e0 0x560c1302e740 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f40a40aa9f0 tx=0x7f40a4009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46c82f5ae6589f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.516+0000 7f409f400640 1 -- 172.21.15.40:0/3857704363 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a407a280 con 0x560c1302c5e0 2024-04-26T23:07:10.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.516+0000 7f409f400640 1 -- 172.21.15.40:0/3857704363 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a407e630 con 0x560c1302c5e0 2024-04-26T23:07:10.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.516+0000 7f409f400640 1 -- 172.21.15.40:0/3857704363 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a4082e60 con 0x560c1302c5e0 2024-04-26T23:07:10.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.517+0000 7f40bcd99740 1 -- 172.21.15.40:0/3857704363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c1302c5e0 msgr2=0x560c1302e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.517+0000 7f40bcd99740 1 --2- 172.21.15.40:0/3857704363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c1302c5e0 0x560c1302e740 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f40a40aa9f0 tx=0x7f40a4009190 comp rx=0 tx=0).stop 2024-04-26T23:07:10.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.517+0000 7f40bcd99740 1 -- 172.21.15.40:0/3857704363 shutdown_connections 2024-04-26T23:07:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.517+0000 7f40bcd99740 1 --2- 172.21.15.40:0/3857704363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c130570a0 0x560c13057520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.517+0000 7f40bcd99740 1 --2- 172.21.15.40:0/3857704363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c13068cb0 0x560c1302c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.517+0000 7f40bcd99740 1 --2- 172.21.15.40:0/3857704363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c1302c5e0 0x560c1302e740 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.517+0000 7f40bcd99740 1 -- 172.21.15.40:0/3857704363 >> 172.21.15.40:0/3857704363 conn(0x560c12fc0d10 msgr2=0x560c130720a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.517+0000 7f40bcd99740 1 -- 172.21.15.40:0/3857704363 shutdown_connections 2024-04-26T23:07:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.517+0000 7f40bcd99740 1 -- 172.21.15.40:0/3857704363 wait complete. 2024-04-26T23:07:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.518+0000 7f40bcd99740 1 -- start start 2024-04-26T23:07:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.518+0000 7f40bcd99740 1 Processor -- start 2024-04-26T23:07:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.519+0000 7f40bcd99740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c1302c5e0 0x560c1306a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.519+0000 7f40bcd99740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c130570a0 0x560c13046930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.519+0000 7f40bcd99740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 0x560c13081cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.519+0000 7f40bcd99740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560c12f90600 con 0x560c13068cb0 2024-04-26T23:07:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.519+0000 7f40bcd99740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560c12f17f70 con 0x560c130570a0 2024-04-26T23:07:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.519+0000 7f40bcd99740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x560c13069630 con 0x560c1302c5e0 2024-04-26T23:07:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.519+0000 7f40bcd99740 10 client.0 _ll_register_callbacks cb 0x560c12f86480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.519+0000 7f40b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 0x560c13081cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.519+0000 7f40b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 0x560c13081cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54414/0 (socket says 172.21.15.40:54414) 2024-04-26T23:07:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.519+0000 7f40b3e00640 1 -- 172.21.15.40:0/1725976822 learned_addr learned my addr 172.21.15.40:0/1725976822 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.520+0000 7f40b3e00640 1 -- 172.21.15.40:0/1725976822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c1302c5e0 msgr2=0x560c1306a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.520+0000 7f40b3e00640 1 --2- 172.21.15.40:0/1725976822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c1302c5e0 0x560c1306a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.520+0000 7f40b3e00640 1 -- 172.21.15.40:0/1725976822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c130570a0 msgr2=0x560c13046930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.520+0000 7f40b3e00640 1 --2- 172.21.15.40:0/1725976822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c130570a0 0x560c13046930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.520+0000 7f40b3e00640 1 -- 172.21.15.40:0/1725976822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c12d6c6b0 con 0x560c13068cb0 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.520+0000 7f40b3e00640 1 --2- 172.21.15.40:0/1725976822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 0x560c13081cf0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00db00 tx=0x7f40ac00b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca18d09b9def3f3d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.520+0000 7f4093e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.520+0000 7f4093e00640 1 -- 172.21.15.40:0/1725976822 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ac01a060 con 0x560c13068cb0 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.520+0000 7f4093e00640 1 -- 172.21.15.40:0/1725976822 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f40ac00bd10 con 0x560c13068cb0 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.520+0000 7f40bcd99740 1 -- 172.21.15.40:0/1725976822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40a407e630 con 0x560c13068cb0 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.521+0000 7f4093e00640 1 -- 172.21.15.40:0/1725976822 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f40ac020050 con 0x560c13068cb0 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.521+0000 7f40bcd99740 10 client.7635 fetch_fsmap learned FSMap version 79 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.521+0000 7f40bcd99740 1 -- 172.21.15.40:0/1725976822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) -- 0x560c13082350 con 0x560c13068cb0 2024-04-26T23:07:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.521+0000 7f4093e00640 1 -- 172.21.15.40:0/1725976822 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a407e630 con 0x560c13068cb0 2024-04-26T23:07:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.521+0000 7f4093e00640 1 -- 172.21.15.40:0/1725976822 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 79) ==== 3534+0+0 (secure 0 0 0) 0x7f40ac023020 con 0x560c13068cb0 2024-04-26T23:07:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.521+0000 7f40bcd99740 10 client.7635 fetch_fsmap finished waiting for FSMap version 79 2024-04-26T23:07:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.521+0000 7f40bcd99740 10 client.7635 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-26T23:07:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.521+0000 7f40bcd99740 20 client.7635 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.521+0000 7f40bcd99740 1 --2- 172.21.15.40:0/1725976822 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x560c13057820 0x560c13057c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.521+0000 7f40bcd99740 4 client.7635 mds_command: new command op to 5965 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:07:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.521+0000 7f40bcd99740 1 -- 172.21.15.40:0/1725976822 --> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c12f9e210 con 0x560c13057820 2024-04-26T23:07:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.522+0000 7f40b3400640 1 --2- 172.21.15.40:0/1725976822 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x560c13057820 0x560c13057c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.523+0000 7f40b3400640 1 --2- 172.21.15.40:0/1725976822 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x560c13057820 0x560c13057c00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=274a0a0e3e8d410 server_cookie=be3ceb0029dffff9 in_seq=0 out_seq=0 2024-04-26T23:07:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.523+0000 7f4093e00640 10 client.7635 ms_handle_connect on v2:172.21.15.103:6834/2914603370 2024-04-26T23:07:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f4093e00640 1 -- 172.21.15.40:0/1725976822 <== mds.2 v2:172.21.15.103:6834/2914603370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560c12f9e210 con 0x560c13057820 2024-04-26T23:07:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f4093e00640 10 client.7635 handle_command_reply: tid=0 2024-04-26T23:07:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f40bcd99740 1 client.7635 shutdown 2024-04-26T23:07:10.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f40bcd99740 1 -- 172.21.15.40:0/1725976822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 msgr2=0x560c13081cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f40bcd99740 1 --2- 172.21.15.40:0/1725976822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 0x560c13081cf0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00db00 tx=0x7f40ac00b3b0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f40bcd99740 1 -- 172.21.15.40:0/1725976822 shutdown_connections 2024-04-26T23:07:10.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f40bcd99740 1 --2- 172.21.15.40:0/1725976822 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x560c13057820 0x560c13057c00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f40bcd99740 1 --2- 172.21.15.40:0/1725976822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 0x560c13081cf0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f40bcd99740 1 --2- 172.21.15.40:0/1725976822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c130570a0 0x560c13046930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f4093e00640 0 client.7635 ms_handle_reset on v2:172.21.15.103:6834/2914603370 2024-04-26T23:07:10.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f40bcd99740 1 --2- 172.21.15.40:0/1725976822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c1302c5e0 0x560c1306a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f40bcd99740 1 -- 172.21.15.40:0/1725976822 >> 172.21.15.40:0/1725976822 conn(0x560c12fc0d10 msgr2=0x560c130522a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:10.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f40bcd99740 1 -- 172.21.15.40:0/1725976822 shutdown_connections 2024-04-26T23:07:10.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.525+0000 7f40bcd99740 1 -- 172.21.15.40:0/1725976822 wait complete. 2024-04-26T23:07:10.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.526+0000 7f40bcd99740 20 client.7635 trim_cache size 0 max 16384 2024-04-26T23:07:10.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.545+0000 7f40bcd99740 1 Processor -- start 2024-04-26T23:07:10.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.545+0000 7f40bcd99740 1 -- start start 2024-04-26T23:07:10.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.546+0000 7f40bcd99740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c1302c5e0 0x560c13027ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.546+0000 7f40bcd99740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c130570a0 0x560c1308b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.546+0000 7f40bcd99740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 0x560c1308bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.546+0000 7f40bcd99740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f40b4201070 con 0x560c13068cb0 2024-04-26T23:07:10.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.546+0000 7f40bcd99740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40b4201740 con 0x560c130570a0 2024-04-26T23:07:10.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.546+0000 7f40bcd99740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f40b42001b0 con 0x560c1302c5e0 2024-04-26T23:07:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.546+0000 7f40b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 0x560c1308bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.546+0000 7f40b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 0x560c1308bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54420/0 (socket says 172.21.15.40:54420) 2024-04-26T23:07:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.546+0000 7f40b3e00640 1 -- 172.21.15.40:0/2075978812 learned_addr learned my addr 172.21.15.40:0/2075978812 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.547+0000 7f40b3e00640 1 -- 172.21.15.40:0/2075978812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c1302c5e0 msgr2=0x560c13027ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.547+0000 7f40b3400640 1 --2- 172.21.15.40:0/2075978812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c1302c5e0 0x560c13027ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.547+0000 7f40b2a00640 1 --2- 172.21.15.40:0/2075978812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c130570a0 0x560c1308b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.547+0000 7f40b3e00640 1 --2- 172.21.15.40:0/2075978812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c1302c5e0 0x560c13027ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.547+0000 7f40b3e00640 1 -- 172.21.15.40:0/2075978812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c130570a0 msgr2=0x560c1308b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.547+0000 7f40b3e00640 1 --2- 172.21.15.40:0/2075978812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c130570a0 0x560c1308b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.547+0000 7f40b3e00640 1 -- 172.21.15.40:0/2075978812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c130238a0 con 0x560c13068cb0 2024-04-26T23:07:10.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.547+0000 7f40b3e00640 1 --2- 172.21.15.40:0/2075978812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 0x560c1308bd10 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00d670 tx=0x7f40ac010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30eea9a445f52f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.547+0000 7f409ea00640 1 -- 172.21.15.40:0/2075978812 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ac028070 con 0x560c13068cb0 2024-04-26T23:07:10.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.547+0000 7f409ea00640 1 -- 172.21.15.40:0/2075978812 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40ac00bd10 con 0x560c13068cb0 2024-04-26T23:07:10.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.547+0000 7f409ea00640 1 -- 172.21.15.40:0/2075978812 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ac01a020 con 0x560c13068cb0 2024-04-26T23:07:10.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.548+0000 7f40bcd99740 1 -- 172.21.15.40:0/2075978812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 msgr2=0x560c1308bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.548+0000 7f40bcd99740 1 --2- 172.21.15.40:0/2075978812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 0x560c1308bd10 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00d670 tx=0x7f40ac010000 comp rx=0 tx=0).stop 2024-04-26T23:07:10.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.548+0000 7f40bcd99740 1 -- 172.21.15.40:0/2075978812 shutdown_connections 2024-04-26T23:07:10.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.548+0000 7f40bcd99740 1 --2- 172.21.15.40:0/2075978812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c13068cb0 0x560c1308bd10 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.548+0000 7f40bcd99740 1 --2- 172.21.15.40:0/2075978812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c130570a0 0x560c1308b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.548+0000 7f40bcd99740 1 --2- 172.21.15.40:0/2075978812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c1302c5e0 0x560c13027ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.548+0000 7f40bcd99740 1 -- 172.21.15.40:0/2075978812 >> 172.21.15.40:0/2075978812 conn(0x560c12fc0d10 msgr2=0x560c13022d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.548+0000 7f40bcd99740 1 -- 172.21.15.40:0/2075978812 shutdown_connections 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.548+0000 7f40bcd99740 1 -- 172.21.15.40:0/2075978812 wait complete. 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.549+0000 7f40bcd99740 1 -- start start 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.549+0000 7f40bcd99740 1 Processor -- start 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.550+0000 7f40bcd99740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c1302c5e0 0x560c13081190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.550+0000 7f40bcd99740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c130570a0 0x560c130eae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.550+0000 7f40bcd99740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c13068cb0 0x560c13044f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.550+0000 7f40bcd99740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560c1305a230 con 0x560c130570a0 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.550+0000 7f40bcd99740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560c13069110 con 0x560c1302c5e0 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.550+0000 7f40bcd99740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x560c13074ec0 con 0x560c13068cb0 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.550+0000 7f40bcd99740 10 client.0 _ll_register_callbacks cb 0x7f40ac0086a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.550+0000 7f40b2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c130570a0 0x560c130eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.550+0000 7f40b2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c130570a0 0x560c130eae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54428/0 (socket says 172.21.15.40:54428) 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.550+0000 7f40b2a00640 1 -- 172.21.15.40:0/2066854376 learned_addr learned my addr 172.21.15.40:0/2066854376 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.551+0000 7f40b3400640 1 --2- 172.21.15.40:0/2066854376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c1302c5e0 0x560c13081190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.551+0000 7f40b2a00640 1 -- 172.21.15.40:0/2066854376 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c13068cb0 msgr2=0x560c13044f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.551+0000 7f40b2a00640 1 --2- 172.21.15.40:0/2066854376 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c13068cb0 0x560c13044f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.551+0000 7f40b2a00640 1 -- 172.21.15.40:0/2066854376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c1302c5e0 msgr2=0x560c13081190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.551+0000 7f40b2a00640 1 --2- 172.21.15.40:0/2066854376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c1302c5e0 0x560c13081190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.551+0000 7f40b2a00640 1 -- 172.21.15.40:0/2066854376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c1308bc20 con 0x560c130570a0 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.551+0000 7f40b3400640 1 --2- 172.21.15.40:0/2066854376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c1302c5e0 0x560c13081190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.551+0000 7f40b2a00640 1 --2- 172.21.15.40:0/2066854376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c130570a0 0x560c130eae40 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f40a0007370 tx=0x7f40a0007770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf18e065981b6e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.551+0000 7f4093e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.551+0000 7f4093e00640 1 -- 172.21.15.40:0/2066854376 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a00047e0 con 0x560c130570a0 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.551+0000 7f4093e00640 1 -- 172.21.15.40:0/2066854376 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f40a001eb40 con 0x560c130570a0 2024-04-26T23:07:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.551+0000 7f40bcd99740 1 -- 172.21.15.40:0/2066854376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40ac00bd10 con 0x560c130570a0 2024-04-26T23:07:10.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.552+0000 7f4093e00640 1 -- 172.21.15.40:0/2066854376 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f40a001ed40 con 0x560c130570a0 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.552+0000 7f40bcd99740 10 client.7641 fetch_fsmap learned FSMap version 79 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.552+0000 7f40bcd99740 1 -- 172.21.15.40:0/2066854376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) -- 0x560c1308b990 con 0x560c130570a0 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.552+0000 7f4093e00640 1 -- 172.21.15.40:0/2066854376 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40ac00bd10 con 0x560c130570a0 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.552+0000 7f4093e00640 1 -- 172.21.15.40:0/2066854376 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 79) ==== 3534+0+0 (secure 0 0 0) 0x7f40a0004980 con 0x560c130570a0 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.552+0000 7f40bcd99740 10 client.7641 fetch_fsmap finished waiting for FSMap version 79 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.552+0000 7f40bcd99740 10 client.7641 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.552+0000 7f40bcd99740 20 client.7641 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.552+0000 7f40bcd99740 1 --2- 172.21.15.40:0/2066854376 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x560c130695a0 0x560c13069980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.552+0000 7f40bcd99740 4 client.7641 mds_command: new command op to 5965 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.552+0000 7f40bcd99740 1 -- 172.21.15.40:0/2066854376 --> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] -- command(tid 0: {"prefix": "status"}) -- 0x560c12f98080 con 0x560c130695a0 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.553+0000 7f40b3400640 1 --2- 172.21.15.40:0/2066854376 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x560c130695a0 0x560c13069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.554+0000 7f40b3400640 1 --2- 172.21.15.40:0/2066854376 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x560c130695a0 0x560c13069980 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=65ba52fd405524b9 server_cookie=7a2650aeea5ea520 in_seq=0 out_seq=0 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.554+0000 7f4093e00640 10 client.7641 ms_handle_connect on v2:172.21.15.103:6834/2914603370 2024-04-26T23:07:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.554+0000 7f4093e00640 1 -- 172.21.15.40:0/2066854376 <== mds.2 v2:172.21.15.103:6834/2914603370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x560c12f98080 con 0x560c130695a0 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.554+0000 7f4093e00640 10 client.7641 handle_command_reply: tid=0 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.554+0000 7f40bcd99740 1 client.7641 shutdown 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.554+0000 7f40bcd99740 1 -- 172.21.15.40:0/2066854376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c130570a0 msgr2=0x560c130eae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.554+0000 7f40bcd99740 1 --2- 172.21.15.40:0/2066854376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c130570a0 0x560c130eae40 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f40a0007370 tx=0x7f40a0007770 comp rx=0 tx=0).stop 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.555+0000 7f40bcd99740 1 -- 172.21.15.40:0/2066854376 shutdown_connections 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.555+0000 7f40bcd99740 1 --2- 172.21.15.40:0/2066854376 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x560c130695a0 0x560c13069980 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.555+0000 7f40bcd99740 1 --2- 172.21.15.40:0/2066854376 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560c13068cb0 0x560c13044f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.555+0000 7f40bcd99740 1 --2- 172.21.15.40:0/2066854376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560c130570a0 0x560c130eae40 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.555+0000 7f40bcd99740 1 --2- 172.21.15.40:0/2066854376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560c1302c5e0 0x560c13081190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.555+0000 7f4093e00640 0 client.7641 ms_handle_reset on v2:172.21.15.103:6834/2914603370 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.555+0000 7f40bcd99740 1 -- 172.21.15.40:0/2066854376 >> 172.21.15.40:0/2066854376 conn(0x560c12fc0d10 msgr2=0x560c13095920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.555+0000 7f40bcd99740 1 -- 172.21.15.40:0/2066854376 shutdown_connections 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.555+0000 7f40bcd99740 1 -- 172.21.15.40:0/2066854376 wait complete. 2024-04-26T23:07:10.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.555+0000 7f40bcd99740 20 client.7641 trim_cache size 0 max 16384 2024-04-26T23:07:10.558 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:07:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:07:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-04-26T23:07:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5965, 2024-04-26T23:07:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:07:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:07:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:07:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 2.0295723419999998, 2024-04-26T23:07:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 79, 2024-04-26T23:07:10.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 79, 2024-04-26T23:07:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 79, 2024-04-26T23:07:10.559 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 18.576374922999999 2024-04-26T23:07:10.559 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:07:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.555+0000 7f40ba000640 1 -- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f409406c8f0 msgr2=0x7f409406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.555+0000 7f40ba000640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f409406c8f0 0x7f409406edb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f40b406e550 tx=0x7f40a0009040 comp rx=0 tx=0).stop 2024-04-26T23:07:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.556+0000 7f40ba000640 1 -- 172.21.15.40:0/4157346052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40b40b2340 msgr2=0x7f40b40c9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.556+0000 7f40ba000640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40b40b2340 0x7f40b40c9020 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f40a4000c00 tx=0x7f40a40046c0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.556+0000 7f40ba000640 1 -- 172.21.15.40:0/4157346052 shutdown_connections 2024-04-26T23:07:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.556+0000 7f40ba000640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f409406c8f0 0x7f409406edb0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.556+0000 7f40ba000640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40b4162810 0x7f40b406d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.556+0000 7f40ba000640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40b40d1b30 0x7f40b4160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.556+0000 7f40ba000640 1 --2- 172.21.15.40:0/4157346052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40b40b2340 0x7f40b40c9020 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.556+0000 7f40ba000640 1 -- 172.21.15.40:0/4157346052 >> 172.21.15.40:0/4157346052 conn(0x7f40b40d2da0 msgr2=0x7f40b40cd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.556+0000 7f40ba000640 1 -- 172.21.15.40:0/4157346052 shutdown_connections 2024-04-26T23:07:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.556+0000 7f40ba000640 1 -- 172.21.15.40:0/4157346052 wait complete. 2024-04-26T23:07:10.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T23:07:10.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.769+0000 7f9abbe00640 1 Processor -- start 2024-04-26T23:07:10.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.770+0000 7f9abbe00640 1 -- start start 2024-04-26T23:07:10.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.771+0000 7f9abbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ab4071e60 0x7f9ab406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.771+0000 7f9abbe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9ab40cc480 0x7f9ab4166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.771+0000 7f9abbe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab415d7c0 con 0x7f9ab406fe20 2024-04-26T23:07:10.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.771+0000 7f9abbe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9ab415daa0 con 0x7f9ab40cc480 2024-04-26T23:07:10.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.771+0000 7f9abbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ab415dd80 con 0x7f9ab4071e60 2024-04-26T23:07:10.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.772+0000 7f9ab9e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9ab40cc480 0x7f9ab4166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.772+0000 7f9ab9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ab4071e60 0x7f9ab406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.772+0000 7f9ab9e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9ab40cc480 0x7f9ab4166ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49140/0 (socket says 172.21.15.40:49140) 2024-04-26T23:07:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.772+0000 7f9ab9e00640 1 -- 172.21.15.40:0/119102988 learned_addr learned my addr 172.21.15.40:0/119102988 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.772+0000 7f9ab9400640 1 -- 172.21.15.40:0/119102988 >> v2:172.21.15.103:3300/0 conn(0x7f9ab40cc480 msgr2=0x7f9ab4166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.772+0000 7f9ab9400640 1 --2- 172.21.15.40:0/119102988 >> v2:172.21.15.103:3300/0 conn(0x7f9ab40cc480 0x7f9ab4166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.773+0000 7f9ab9400640 1 -- 172.21.15.40:0/119102988 >> v1:172.21.15.103:6789/0 conn(0x7f9ab406fe20 legacy=0x7f9ab4072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.773+0000 7f9ab9400640 1 -- 172.21.15.40:0/119102988 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab415e060 con 0x7f9ab4071e60 2024-04-26T23:07:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.773+0000 7f9ab9e00640 1 --2- 172.21.15.40:0/119102988 >> v2:172.21.15.103:3300/0 conn(0x7f9ab40cc480 0x7f9ab4166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:07:10.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.774+0000 7f9ab9400640 1 --2- 172.21.15.40:0/119102988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ab4071e60 0x7f9ab406f8e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80097a0 tx=0x7f9aa802a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7869c388eda5b34 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.774+0000 7f9ab3e00640 1 -- 172.21.15.40:0/119102988 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8037070 con 0x7f9ab4071e60 2024-04-26T23:07:10.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.774+0000 7f9ab3e00640 1 -- 172.21.15.40:0/119102988 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa802ae40 con 0x7f9ab4071e60 2024-04-26T23:07:10.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.774+0000 7f9ab3e00640 1 -- 172.21.15.40:0/119102988 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8032660 con 0x7f9ab4071e60 2024-04-26T23:07:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.775+0000 7f9abbe00640 1 -- 172.21.15.40:0/119102988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ab4071e60 msgr2=0x7f9ab406f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.775+0000 7f9abbe00640 1 --2- 172.21.15.40:0/119102988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ab4071e60 0x7f9ab406f8e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80097a0 tx=0x7f9aa802a130 comp rx=0 tx=0).stop 2024-04-26T23:07:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.775+0000 7f9abbe00640 1 -- 172.21.15.40:0/119102988 shutdown_connections 2024-04-26T23:07:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.775+0000 7f9abbe00640 1 --2- 172.21.15.40:0/119102988 >> v2:172.21.15.103:3300/0 conn(0x7f9ab40cc480 0x7f9ab4166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.775+0000 7f9abbe00640 1 --2- 172.21.15.40:0/119102988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ab4071e60 0x7f9ab406f8e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.775+0000 7f9abbe00640 1 -- 172.21.15.40:0/119102988 >> 172.21.15.40:0/119102988 conn(0x7f9ab40d0be0 msgr2=0x7f9ab40d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.775+0000 7f9abbe00640 1 -- 172.21.15.40:0/119102988 shutdown_connections 2024-04-26T23:07:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.775+0000 7f9abbe00640 1 -- 172.21.15.40:0/119102988 wait complete. 2024-04-26T23:07:10.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.777+0000 7f9abbe00640 1 Processor -- start 2024-04-26T23:07:10.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.777+0000 7f9abbe00640 1 -- start start 2024-04-26T23:07:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.778+0000 7f9abbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab406fe20 0x7f9ab40ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.778+0000 7f9abbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ab4071e60 0x7f9ab414ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.778+0000 7f9abbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ab40cc480 0x7f9ab40b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.778+0000 7f9abbe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab415d570 con 0x7f9ab406fe20 2024-04-26T23:07:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.778+0000 7f9abbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab415d3f0 con 0x7f9ab40cc480 2024-04-26T23:07:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.778+0000 7f9abbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ab4072950 con 0x7f9ab4071e60 2024-04-26T23:07:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.779+0000 7f9ab9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab406fe20 0x7f9ab40ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.779+0000 7f9ab9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab406fe20 0x7f9ab40ca440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54430/0 (socket says 172.21.15.40:54430) 2024-04-26T23:07:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.779+0000 7f9ab9400640 1 -- 172.21.15.40:0/3912035062 learned_addr learned my addr 172.21.15.40:0/3912035062 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.779+0000 7f9ab8a00640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ab4071e60 0x7f9ab414ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.779+0000 7f9ab9e00640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ab40cc480 0x7f9ab40b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.779+0000 7f9ab9400640 1 -- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ab4071e60 msgr2=0x7f9ab414ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.779+0000 7f9ab9400640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ab4071e60 0x7f9ab414ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.779+0000 7f9ab9400640 1 -- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ab40cc480 msgr2=0x7f9ab40b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.779+0000 7f9ab9400640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ab40cc480 0x7f9ab40b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.779+0000 7f9ab9400640 1 -- 172.21.15.40:0/3912035062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab40caa20 con 0x7f9ab406fe20 2024-04-26T23:07:10.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.780+0000 7f9ab9400640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab406fe20 0x7f9ab40ca440 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8035040 tx=0x7f9aa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.780+0000 7f9ab1600640 1 -- 172.21.15.40:0/3912035062 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8041070 con 0x7f9ab406fe20 2024-04-26T23:07:10.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.780+0000 7f9ab1600640 1 -- 172.21.15.40:0/3912035062 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa803c460 con 0x7f9ab406fe20 2024-04-26T23:07:10.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.780+0000 7f9ab1600640 1 -- 172.21.15.40:0/3912035062 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8037040 con 0x7f9ab406fe20 2024-04-26T23:07:10.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.780+0000 7f9abbe00640 1 -- 172.21.15.40:0/3912035062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab40b5df0 con 0x7f9ab406fe20 2024-04-26T23:07:10.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.780+0000 7f9abbe00640 1 -- 172.21.15.40:0/3912035062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab40b63b0 con 0x7f9ab406fe20 2024-04-26T23:07:10.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.782+0000 7f9ab1600640 1 -- 172.21.15.40:0/3912035062 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9aa803a070 con 0x7f9ab406fe20 2024-04-26T23:07:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.784+0000 7f9ab1600640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a9406c8f0 0x7f9a9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.784+0000 7f9ab1600640 1 -- 172.21.15.40:0/3912035062 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f9aa80abff0 con 0x7f9ab406fe20 2024-04-26T23:07:10.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.785+0000 7f9abec1a740 1 Processor -- start 2024-04-26T23:07:10.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.785+0000 7f9ab8a00640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a9406c8f0 0x7f9a9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.785+0000 7f9abec1a740 1 -- start start 2024-04-26T23:07:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.786+0000 7f9abec1a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 0x5636601c6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.786+0000 7f9abec1a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563660200cb0 0x5636601c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.786+0000 7f9abec1a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636601ef0a0 0x5636601ef520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.786+0000 7f9abec1a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563660128600 con 0x5636601c45e0 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.786+0000 7f9abec1a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5636600c5100 con 0x563660200cb0 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.786+0000 7f9abec1a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5636601245c0 con 0x5636601ef0a0 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.786+0000 7f9ab8a00640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a9406c8f0 0x7f9a9406edb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9ab415ed20 tx=0x7f9aa0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.786+0000 7f9ab9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 0x5636601c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.786+0000 7f9ab9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 0x5636601c6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54442/0 (socket says 172.21.15.40:54442) 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.786+0000 7f9ab9400640 1 -- 172.21.15.40:0/3297426861 learned_addr learned my addr 172.21.15.40:0/3297426861 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.786+0000 7f9ab8a00640 1 --2- 172.21.15.40:0/3297426861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563660200cb0 0x5636601c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.786+0000 7f9ab9e00640 1 --2- 172.21.15.40:0/3297426861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636601ef0a0 0x5636601ef520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.787+0000 7f9ab9400640 1 -- 172.21.15.40:0/3297426861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636601ef0a0 msgr2=0x5636601ef520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.787+0000 7f9ab9400640 1 --2- 172.21.15.40:0/3297426861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636601ef0a0 0x5636601ef520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.787+0000 7f9ab9400640 1 -- 172.21.15.40:0/3297426861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563660200cb0 msgr2=0x5636601c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.787+0000 7f9ab9400640 1 --2- 172.21.15.40:0/3297426861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563660200cb0 0x5636601c49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.787+0000 7f9ab9400640 1 -- 172.21.15.40:0/3297426861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56365fe3a2d0 con 0x5636601c45e0 2024-04-26T23:07:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.787+0000 7f9ab9400640 1 --2- 172.21.15.40:0/3297426861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 0x5636601c6740 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8082200 tx=0x7f9aa8079690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58f63685b5503ba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.787+0000 7f9aa5c00640 1 -- 172.21.15.40:0/3297426861 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa804a040 con 0x5636601c45e0 2024-04-26T23:07:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.787+0000 7f9aa5c00640 1 -- 172.21.15.40:0/3297426861 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa8079b50 con 0x5636601c45e0 2024-04-26T23:07:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.787+0000 7f9aa5c00640 1 -- 172.21.15.40:0/3297426861 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa807df00 con 0x5636601c45e0 2024-04-26T23:07:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.788+0000 7f9abec1a740 1 -- 172.21.15.40:0/3297426861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 msgr2=0x5636601c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.788+0000 7f9abec1a740 1 --2- 172.21.15.40:0/3297426861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 0x5636601c6740 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8082200 tx=0x7f9aa8079690 comp rx=0 tx=0).stop 2024-04-26T23:07:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.788+0000 7f9abec1a740 1 -- 172.21.15.40:0/3297426861 shutdown_connections 2024-04-26T23:07:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.788+0000 7f9abec1a740 1 --2- 172.21.15.40:0/3297426861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5636601ef0a0 0x5636601ef520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.788+0000 7f9abec1a740 1 --2- 172.21.15.40:0/3297426861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563660200cb0 0x5636601c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.788+0000 7f9abec1a740 1 --2- 172.21.15.40:0/3297426861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 0x5636601c6740 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.788+0000 7f9abec1a740 1 -- 172.21.15.40:0/3297426861 >> 172.21.15.40:0/3297426861 conn(0x563660158d10 msgr2=0x56366020a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.788+0000 7f9abec1a740 1 -- 172.21.15.40:0/3297426861 shutdown_connections 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.788+0000 7f9abec1a740 1 -- 172.21.15.40:0/3297426861 wait complete. 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.789+0000 7f9abec1a740 1 -- start start 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.789+0000 7f9abec1a740 1 Processor -- start 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.790+0000 7f9abec1a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 0x5636601eb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.790+0000 7f9abec1a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601ef0a0 0x56366020aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.790+0000 7f9abec1a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 0x563660202900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.790+0000 7f9abec1a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563660128600 con 0x5636601c45e0 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.790+0000 7f9abec1a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5636600aff70 con 0x5636601ef0a0 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.790+0000 7f9abec1a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5636602048d0 con 0x563660200cb0 2024-04-26T23:07:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.790+0000 7f9abec1a740 10 client.0 _ll_register_callbacks cb 0x56366011e480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.790+0000 7f9ab9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 0x5636601eb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.790+0000 7f9ab9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 0x5636601eb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54454/0 (socket says 172.21.15.40:54454) 2024-04-26T23:07:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.790+0000 7f9ab9400640 1 -- 172.21.15.40:0/829749486 learned_addr learned my addr 172.21.15.40:0/829749486 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.791+0000 7f9ab9400640 1 -- 172.21.15.40:0/829749486 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 msgr2=0x563660202900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.791+0000 7f9ab9400640 1 --2- 172.21.15.40:0/829749486 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 0x563660202900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.791+0000 7f9ab9400640 1 -- 172.21.15.40:0/829749486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601ef0a0 msgr2=0x56366020aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.791+0000 7f9ab9400640 1 --2- 172.21.15.40:0/829749486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601ef0a0 0x56366020aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.791+0000 7f9ab9400640 1 -- 172.21.15.40:0/829749486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56365ff046b0 con 0x5636601c45e0 2024-04-26T23:07:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.791+0000 7f9ab9400640 1 --2- 172.21.15.40:0/829749486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 0x5636601eb9e0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8077e00 tx=0x7f9aa802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3add41a4bc8d2860 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.791+0000 7f9a9a000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.791+0000 7f9a9a000640 1 -- 172.21.15.40:0/829749486 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa804a040 con 0x5636601c45e0 2024-04-26T23:07:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.791+0000 7f9a9a000640 1 -- 172.21.15.40:0/829749486 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f9aa80791b0 con 0x5636601c45e0 2024-04-26T23:07:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.791+0000 7f9abec1a740 1 -- 172.21.15.40:0/829749486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9aa8079b50 con 0x5636601c45e0 2024-04-26T23:07:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.791+0000 7f9a9a000640 1 -- 172.21.15.40:0/829749486 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f9aa80991a0 con 0x5636601c45e0 2024-04-26T23:07:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.792+0000 7f9abec1a740 10 client.7650 fetch_fsmap learned FSMap version 79 2024-04-26T23:07:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.792+0000 7f9abec1a740 1 -- 172.21.15.40:0/829749486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) -- 0x5636601f1310 con 0x5636601c45e0 2024-04-26T23:07:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.792+0000 7f9a9a000640 1 -- 172.21.15.40:0/829749486 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa8079b50 con 0x5636601c45e0 2024-04-26T23:07:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.792+0000 7f9a9a000640 1 -- 172.21.15.40:0/829749486 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 79) ==== 3534+0+0 (secure 0 0 0) 0x7f9aa8080100 con 0x5636601c45e0 2024-04-26T23:07:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.792+0000 7f9abec1a740 10 client.7650 fetch_fsmap finished waiting for FSMap version 79 2024-04-26T23:07:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.792+0000 7f9abec1a740 10 client.7650 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:07:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.792+0000 7f9abec1a740 20 client.7650 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.792+0000 7f9abec1a740 1 --2- 172.21.15.40:0/829749486 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x5636602251d0 0x5636602275c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.792+0000 7f9abec1a740 4 client.7650 mds_command: new command op to 7443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:07:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.792+0000 7f9abec1a740 1 -- 172.21.15.40:0/829749486 --> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563660136210 con 0x5636602251d0 2024-04-26T23:07:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.792+0000 7f9ab8a00640 1 --2- 172.21.15.40:0/829749486 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x5636602251d0 0x5636602275c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.793+0000 7f9ab8a00640 1 --2- 172.21.15.40:0/829749486 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x5636602251d0 0x5636602275c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5a5f57b4e9725c9e server_cookie=52ec2e9fd422b5e8 in_seq=0 out_seq=0 2024-04-26T23:07:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.793+0000 7f9a9a000640 10 client.7650 ms_handle_connect on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9a9a000640 1 -- 172.21.15.40:0/829749486 <== mds.1 v2:172.21.15.40:6832/216893598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563660136210 con 0x5636602251d0 2024-04-26T23:07:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9a9a000640 10 client.7650 handle_command_reply: tid=0 2024-04-26T23:07:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9abec1a740 1 client.7650 shutdown 2024-04-26T23:07:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9abec1a740 1 -- 172.21.15.40:0/829749486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 msgr2=0x5636601eb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9abec1a740 1 --2- 172.21.15.40:0/829749486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 0x5636601eb9e0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8077e00 tx=0x7f9aa802a130 comp rx=0 tx=0).stop 2024-04-26T23:07:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9abec1a740 1 -- 172.21.15.40:0/829749486 shutdown_connections 2024-04-26T23:07:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9abec1a740 1 --2- 172.21.15.40:0/829749486 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x5636602251d0 0x5636602275c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9abec1a740 1 --2- 172.21.15.40:0/829749486 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 0x563660202900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9abec1a740 1 --2- 172.21.15.40:0/829749486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601ef0a0 0x56366020aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9abec1a740 1 --2- 172.21.15.40:0/829749486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601c45e0 0x5636601eb9e0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9abec1a740 1 -- 172.21.15.40:0/829749486 >> 172.21.15.40:0/829749486 conn(0x563660158d10 msgr2=0x56366020d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9a9a000640 0 client.7650 ms_handle_reset on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9abec1a740 1 -- 172.21.15.40:0/829749486 shutdown_connections 2024-04-26T23:07:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.794+0000 7f9abec1a740 1 -- 172.21.15.40:0/829749486 wait complete. 2024-04-26T23:07:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.795+0000 7f9abec1a740 20 client.7650 trim_cache size 0 max 16384 2024-04-26T23:07:10.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.814+0000 7f9abec1a740 1 Processor -- start 2024-04-26T23:07:10.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.814+0000 7f9abec1a740 1 -- start start 2024-04-26T23:07:10.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.815+0000 7f9abec1a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601c45e0 0x563660204cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.815+0000 7f9abec1a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 0x5636602051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.815+0000 7f9abec1a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 0x563660201090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.815+0000 7f9abec1a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab4201220 con 0x5636601ef0a0 2024-04-26T23:07:10.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.815+0000 7f9abec1a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab42018f0 con 0x5636601c45e0 2024-04-26T23:07:10.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.815+0000 7f9abec1a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ab42003e0 con 0x563660200cb0 2024-04-26T23:07:10.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.816+0000 7f9ab8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 0x5636602051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.816+0000 7f9ab8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 0x5636602051f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54470/0 (socket says 172.21.15.40:54470) 2024-04-26T23:07:10.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.816+0000 7f9ab8a00640 1 -- 172.21.15.40:0/2585106372 learned_addr learned my addr 172.21.15.40:0/2585106372 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:10.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.816+0000 7f9ab9e00640 1 --2- 172.21.15.40:0/2585106372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 0x563660201090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.816+0000 7f9ab9400640 1 --2- 172.21.15.40:0/2585106372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601c45e0 0x563660204cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.816+0000 7f9ab8a00640 1 -- 172.21.15.40:0/2585106372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 msgr2=0x563660201090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.816+0000 7f9ab8a00640 1 --2- 172.21.15.40:0/2585106372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 0x563660201090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.816+0000 7f9ab8a00640 1 -- 172.21.15.40:0/2585106372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601c45e0 msgr2=0x563660204cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.816+0000 7f9ab8a00640 1 --2- 172.21.15.40:0/2585106372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601c45e0 0x563660204cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.816+0000 7f9ab8a00640 1 -- 172.21.15.40:0/2585106372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56366020a020 con 0x5636601ef0a0 2024-04-26T23:07:10.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.816+0000 7f9ab9e00640 1 --2- 172.21.15.40:0/2585106372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 0x563660201090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:10.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.816+0000 7f9ab9400640 1 --2- 172.21.15.40:0/2585106372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601c45e0 0x563660204cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:10.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.817+0000 7f9ab8a00640 1 --2- 172.21.15.40:0/2585106372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 0x5636602051f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40c9c70 tx=0x7f9aa0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2bb3b1af961b1ec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.817+0000 7f9aa5200640 1 -- 172.21.15.40:0/2585106372 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0024420 con 0x5636601ef0a0 2024-04-26T23:07:10.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.817+0000 7f9aa5200640 1 -- 172.21.15.40:0/2585106372 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa0028030 con 0x5636601ef0a0 2024-04-26T23:07:10.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.817+0000 7f9aa5200640 1 -- 172.21.15.40:0/2585106372 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa00247e0 con 0x5636601ef0a0 2024-04-26T23:07:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.818+0000 7f9abec1a740 1 -- 172.21.15.40:0/2585106372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 msgr2=0x5636602051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.818+0000 7f9abec1a740 1 --2- 172.21.15.40:0/2585106372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 0x5636602051f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40c9c70 tx=0x7f9aa0011000 comp rx=0 tx=0).stop 2024-04-26T23:07:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.818+0000 7f9abec1a740 1 -- 172.21.15.40:0/2585106372 shutdown_connections 2024-04-26T23:07:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.818+0000 7f9abec1a740 1 --2- 172.21.15.40:0/2585106372 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 0x563660201090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.818+0000 7f9abec1a740 1 --2- 172.21.15.40:0/2585106372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 0x5636602051f0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.818+0000 7f9abec1a740 1 --2- 172.21.15.40:0/2585106372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601c45e0 0x563660204cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.818+0000 7f9abec1a740 1 -- 172.21.15.40:0/2585106372 >> 172.21.15.40:0/2585106372 conn(0x563660158d10 msgr2=0x563660205a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.818+0000 7f9abec1a740 1 -- 172.21.15.40:0/2585106372 shutdown_connections 2024-04-26T23:07:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.818+0000 7f9abec1a740 1 -- 172.21.15.40:0/2585106372 wait complete. 2024-04-26T23:07:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.818+0000 7f9abec1a740 1 -- start start 2024-04-26T23:07:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.819+0000 7f9abec1a740 1 Processor -- start 2024-04-26T23:07:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.819+0000 7f9abec1a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601c45e0 0x56366020d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.819+0000 7f9abec1a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 0x56366020d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.819+0000 7f9abec1a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 0x5636601ffa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9abec1a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563660128600 con 0x5636601ef0a0 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9abec1a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5636600aff70 con 0x5636601c45e0 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9abec1a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5636601ef730 con 0x563660200cb0 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9abec1a740 10 client.0 _ll_register_callbacks cb 0x7f9aa8079920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9ab8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 0x56366020d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9ab8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 0x56366020d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54476/0 (socket says 172.21.15.40:54476) 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9ab8a00640 1 -- 172.21.15.40:0/1346196999 learned_addr learned my addr 172.21.15.40:0/1346196999 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9ab9400640 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601c45e0 0x56366020d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9ab9e00640 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 0x5636601ffa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9ab8a00640 1 -- 172.21.15.40:0/1346196999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 msgr2=0x5636601ffa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9ab8a00640 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 0x5636601ffa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9ab8a00640 1 -- 172.21.15.40:0/1346196999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601c45e0 msgr2=0x56366020d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9ab8a00640 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601c45e0 0x56366020d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9ab8a00640 1 -- 172.21.15.40:0/1346196999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56366022bd10 con 0x5636601ef0a0 2024-04-26T23:07:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.820+0000 7f9ab9400640 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601c45e0 0x56366020d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.821+0000 7f9ab8a00640 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 0x56366020d720 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000e820 tx=0x7f9aa00036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7888d2f8db842adc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.821+0000 7f9a9a000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.821+0000 7f9a9a000640 1 -- 172.21.15.40:0/1346196999 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa00248f0 con 0x5636601ef0a0 2024-04-26T23:07:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.821+0000 7f9a9a000640 1 -- 172.21.15.40:0/1346196999 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f9aa0015900 con 0x5636601ef0a0 2024-04-26T23:07:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.821+0000 7f9abec1a740 1 -- 172.21.15.40:0/1346196999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9aa0028030 con 0x5636601ef0a0 2024-04-26T23:07:10.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.821+0000 7f9a9a000640 1 -- 172.21.15.40:0/1346196999 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f9aa0015b00 con 0x5636601ef0a0 2024-04-26T23:07:10.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.821+0000 7f9abec1a740 10 client.7656 fetch_fsmap learned FSMap version 79 2024-04-26T23:07:10.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.821+0000 7f9abec1a740 1 -- 172.21.15.40:0/1346196999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) -- 0x563660223c60 con 0x5636601ef0a0 2024-04-26T23:07:10.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.822+0000 7f9a9a000640 1 -- 172.21.15.40:0/1346196999 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa0028030 con 0x5636601ef0a0 2024-04-26T23:07:10.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.822+0000 7f9a9a000640 1 -- 172.21.15.40:0/1346196999 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 79) ==== 3534+0+0 (secure 0 0 0) 0x7f9aa0041020 con 0x5636601ef0a0 2024-04-26T23:07:10.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.822+0000 7f9abec1a740 10 client.7656 fetch_fsmap finished waiting for FSMap version 79 2024-04-26T23:07:10.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.822+0000 7f9abec1a740 10 client.7656 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:07:10.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.822+0000 7f9abec1a740 20 client.7656 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.822+0000 7f9abec1a740 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x563660282de0 0x5636602831c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.822+0000 7f9abec1a740 4 client.7656 mds_command: new command op to 7443 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.822+0000 7f9abec1a740 1 -- 172.21.15.40:0/1346196999 --> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] -- command(tid 0: {"prefix": "status"}) -- 0x563660130080 con 0x563660282de0 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.822+0000 7f9ab9400640 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x563660282de0 0x5636602831c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.822+0000 7f9ab9400640 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x563660282de0 0x5636602831c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67cc96c0b249c7dc server_cookie=3235e0a215868ba1 in_seq=0 out_seq=0 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.822+0000 7f9a9a000640 10 client.7656 ms_handle_connect on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9a9a000640 1 -- 172.21.15.40:0/1346196999 <== mds.1 v2:172.21.15.40:6832/216893598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x563660130080 con 0x563660282de0 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9a9a000640 10 client.7656 handle_command_reply: tid=0 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9abec1a740 1 client.7656 shutdown 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9abec1a740 1 -- 172.21.15.40:0/1346196999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 msgr2=0x56366020d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9abec1a740 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 0x56366020d720 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000e820 tx=0x7f9aa00036f0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9abec1a740 1 -- 172.21.15.40:0/1346196999 shutdown_connections 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9abec1a740 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x563660282de0 0x5636602831c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9abec1a740 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563660200cb0 0x5636601ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9abec1a740 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5636601ef0a0 0x56366020d720 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9abec1a740 1 --2- 172.21.15.40:0/1346196999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5636601c45e0 0x56366020d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9a9a000640 0 client.7656 ms_handle_reset on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:10.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9abec1a740 1 -- 172.21.15.40:0/1346196999 >> 172.21.15.40:0/1346196999 conn(0x563660158d10 msgr2=0x563660204c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:10.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9abec1a740 1 -- 172.21.15.40:0/1346196999 shutdown_connections 2024-04-26T23:07:10.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9abec1a740 1 -- 172.21.15.40:0/1346196999 wait complete. 2024-04-26T23:07:10.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.823+0000 7f9abec1a740 20 client.7656 trim_cache size 0 max 16384 2024-04-26T23:07:10.827 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:07:10.827 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:07:10.827 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-04-26T23:07:10.828 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 7443, 2024-04-26T23:07:10.828 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:07:10.828 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:07:10.828 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:07:10.828 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.3112854129999998, 2024-04-26T23:07:10.828 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 79, 2024-04-26T23:07:10.828 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 79, 2024-04-26T23:07:10.828 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 79, 2024-04-26T23:07:10.828 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 18.848479461 2024-04-26T23:07:10.828 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:07:10.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.824+0000 7f9abbe00640 1 -- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a9406c8f0 msgr2=0x7f9a9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.824+0000 7f9abbe00640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a9406c8f0 0x7f9a9406edb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9ab415ed20 tx=0x7f9aa0009040 comp rx=0 tx=0).stop 2024-04-26T23:07:10.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.824+0000 7f9abbe00640 1 -- 172.21.15.40:0/3912035062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab406fe20 msgr2=0x7f9ab40ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:10.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.824+0000 7f9abbe00640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab406fe20 0x7f9ab40ca440 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8035040 tx=0x7f9aa80046c0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.824+0000 7f9abbe00640 1 -- 172.21.15.40:0/3912035062 shutdown_connections 2024-04-26T23:07:10.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.824+0000 7f9abbe00640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a9406c8f0 0x7f9a9406edb0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.824+0000 7f9abbe00640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ab40cc480 0x7f9ab40b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.824+0000 7f9abbe00640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ab4071e60 0x7f9ab414ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.824+0000 7f9abbe00640 1 --2- 172.21.15.40:0/3912035062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab406fe20 0x7f9ab40ca440 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:10.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.824+0000 7f9abbe00640 1 -- 172.21.15.40:0/3912035062 >> 172.21.15.40:0/3912035062 conn(0x7f9ab40d0be0 msgr2=0x7f9ab40ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:10.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.824+0000 7f9abbe00640 1 -- 172.21.15.40:0/3912035062 shutdown_connections 2024-04-26T23:07:10.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:10.824+0000 7f9abbe00640 1 -- 172.21.15.40:0/3912035062 wait complete. 2024-04-26T23:07:10.841 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T23:07:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.046+0000 7fe399000640 1 Processor -- start 2024-04-26T23:07:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.046+0000 7fe399000640 1 -- start start 2024-04-26T23:07:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.047+0000 7fe399000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe39415ce60 0x7fe3940afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.047+0000 7fe399000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe394071fe0 0x7fe394073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.047+0000 7fe399000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3940d0820 con 0x7fe3940af7a0 2024-04-26T23:07:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.047+0000 7fe399000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe3940cc450 con 0x7fe39415ce60 2024-04-26T23:07:11.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.047+0000 7fe399000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe3941614c0 con 0x7fe394071fe0 2024-04-26T23:07:11.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.048+0000 7fe393400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe39415ce60 0x7fe3940afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.048+0000 7fe38aa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe3940af7a0 0x7fe3940ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43218/0 (socket says 172.21.15.40:43218) 2024-04-26T23:07:11.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.048+0000 7fe38aa00640 1 -- 172.21.15.40:0/4084151422 learned_addr learned my addr 172.21.15.40:0/4084151422 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:11.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.048+0000 7fe393e00640 1 --2- 172.21.15.40:0/4084151422 >> v2:172.21.15.103:3300/0 conn(0x7fe394071fe0 0x7fe394073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.048+0000 7fe392a00640 1 -- 172.21.15.40:0/4084151422 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1132822922 0 0) 0x7fe3940d0820 con 0x7fe3940af7a0 2024-04-26T23:07:11.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.048+0000 7fe393400640 1 -- 172.21.15.40:0/4084151422 >> v2:172.21.15.103:3300/0 conn(0x7fe394071fe0 msgr2=0x7fe394073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:11.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.048+0000 7fe393400640 1 --2- 172.21.15.40:0/4084151422 >> v2:172.21.15.103:3300/0 conn(0x7fe394071fe0 0x7fe394073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.048+0000 7fe393400640 1 -- 172.21.15.40:0/4084151422 >> v1:172.21.15.40:6789/0 conn(0x7fe3940af7a0 legacy=0x7fe3940ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.048+0000 7fe393400640 1 -- 172.21.15.40:0/4084151422 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe394161760 con 0x7fe39415ce60 2024-04-26T23:07:11.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.050+0000 7fe393400640 1 --2- 172.21.15.40:0/4084151422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39415ce60 0x7fe3940afc60 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fe3940d1af0 tx=0x7fe38402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4905775367a58a23 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:11.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.050+0000 7fe392a00640 1 -- 172.21.15.40:0/4084151422 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe384037070 con 0x7fe39415ce60 2024-04-26T23:07:11.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.050+0000 7fe392a00640 1 -- 172.21.15.40:0/4084151422 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe384032440 con 0x7fe39415ce60 2024-04-26T23:07:11.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.050+0000 7fe392a00640 1 -- 172.21.15.40:0/4084151422 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38402ae60 con 0x7fe39415ce60 2024-04-26T23:07:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.051+0000 7fe399000640 1 -- 172.21.15.40:0/4084151422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39415ce60 msgr2=0x7fe3940afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.051+0000 7fe399000640 1 --2- 172.21.15.40:0/4084151422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39415ce60 0x7fe3940afc60 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fe3940d1af0 tx=0x7fe38402a110 comp rx=0 tx=0).stop 2024-04-26T23:07:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.051+0000 7fe399000640 1 -- 172.21.15.40:0/4084151422 shutdown_connections 2024-04-26T23:07:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.051+0000 7fe399000640 1 --2- 172.21.15.40:0/4084151422 >> v2:172.21.15.103:3300/0 conn(0x7fe394071fe0 0x7fe394073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.051+0000 7fe399000640 1 --2- 172.21.15.40:0/4084151422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39415ce60 0x7fe3940afc60 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.051+0000 7fe399000640 1 -- 172.21.15.40:0/4084151422 >> 172.21.15.40:0/4084151422 conn(0x7fe3940d0a10 msgr2=0x7fe3940d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.051+0000 7fe399000640 1 -- 172.21.15.40:0/4084151422 shutdown_connections 2024-04-26T23:07:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.051+0000 7fe399000640 1 -- 172.21.15.40:0/4084151422 wait complete. 2024-04-26T23:07:11.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.053+0000 7fe399000640 1 Processor -- start 2024-04-26T23:07:11.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.053+0000 7fe399000640 1 -- start start 2024-04-26T23:07:11.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.054+0000 7fe399000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe394071fe0 0x7fe39415f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.054+0000 7fe399000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3940af7a0 0x7fe39415e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.054+0000 7fe399000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe39415ce60 0x7fe3940b4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.054+0000 7fe399000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe3940cc1f0 con 0x7fe394071fe0 2024-04-26T23:07:11.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.054+0000 7fe399000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe3940cc070 con 0x7fe3940af7a0 2024-04-26T23:07:11.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.054+0000 7fe399000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe394073ca0 con 0x7fe39415ce60 2024-04-26T23:07:11.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.054+0000 7fe393400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe394071fe0 0x7fe39415f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.054+0000 7fe393400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe394071fe0 0x7fe39415f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54498/0 (socket says 172.21.15.40:54498) 2024-04-26T23:07:11.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.054+0000 7fe393400640 1 -- 172.21.15.40:0/147683621 learned_addr learned my addr 172.21.15.40:0/147683621 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:11.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.055+0000 7fe393400640 1 -- 172.21.15.40:0/147683621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe39415ce60 msgr2=0x7fe3940b4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.055+0000 7fe393400640 1 --2- 172.21.15.40:0/147683621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe39415ce60 0x7fe3940b4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.055+0000 7fe393400640 1 -- 172.21.15.40:0/147683621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3940af7a0 msgr2=0x7fe39415e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.055+0000 7fe393400640 1 --2- 172.21.15.40:0/147683621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3940af7a0 0x7fe39415e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.055+0000 7fe393400640 1 -- 172.21.15.40:0/147683621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39415d640 con 0x7fe394071fe0 2024-04-26T23:07:11.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.055+0000 7fe393400640 1 --2- 172.21.15.40:0/147683621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe394071fe0 0x7fe39415f970 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fe384009890 tx=0x7fe3840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:11.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.055+0000 7fe38be00640 1 -- 172.21.15.40:0/147683621 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe384037040 con 0x7fe394071fe0 2024-04-26T23:07:11.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.055+0000 7fe38be00640 1 -- 172.21.15.40:0/147683621 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe384042070 con 0x7fe394071fe0 2024-04-26T23:07:11.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.055+0000 7fe38be00640 1 -- 172.21.15.40:0/147683621 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3840308a0 con 0x7fe394071fe0 2024-04-26T23:07:11.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.055+0000 7fe399000640 1 -- 172.21.15.40:0/147683621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe39415d920 con 0x7fe394071fe0 2024-04-26T23:07:11.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.056+0000 7fe399000640 1 -- 172.21.15.40:0/147683621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe39415dee0 con 0x7fe394071fe0 2024-04-26T23:07:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.058+0000 7fe38be00640 1 -- 172.21.15.40:0/147683621 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe384030a40 con 0x7fe394071fe0 2024-04-26T23:07:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.059+0000 7fe38be00640 1 --2- 172.21.15.40:0/147683621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe37006c8f0 0x7fe37006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.059+0000 7fe38be00640 1 -- 172.21.15.40:0/147683621 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fe3840ac640 con 0x7fe394071fe0 2024-04-26T23:07:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.060+0000 7fe39d5d7740 1 Processor -- start 2024-04-26T23:07:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.060+0000 7fe39d5d7740 1 -- start start 2024-04-26T23:07:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.060+0000 7fe38aa00640 1 --2- 172.21.15.40:0/147683621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe37006c8f0 0x7fe37006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe39d5d7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 0x55e554458910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe39d5d7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554492e80 0x55e554456b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe39d5d7740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554481270 0x55e5544816f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe39d5d7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e5543ba600 con 0x55e5544567b0 2024-04-26T23:07:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe39d5d7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e554357100 con 0x55e554492e80 2024-04-26T23:07:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe39d5d7740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e5543b65c0 con 0x55e554481270 2024-04-26T23:07:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe393400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 0x55e554458910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe393400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 0x55e554458910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54508/0 (socket says 172.21.15.40:54508) 2024-04-26T23:07:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe393400640 1 -- 172.21.15.40:0/1760770384 learned_addr learned my addr 172.21.15.40:0/1760770384 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe38aa00640 1 --2- 172.21.15.40:0/147683621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe37006c8f0 0x7fe37006edb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe37c007fd0 tx=0x7fe37c007920 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe393400640 1 -- 172.21.15.40:0/1760770384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554481270 msgr2=0x55e5544816f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe393400640 1 --2- 172.21.15.40:0/1760770384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554481270 0x55e5544816f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe393400640 1 -- 172.21.15.40:0/1760770384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554492e80 msgr2=0x55e554456b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe393400640 1 --2- 172.21.15.40:0/1760770384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554492e80 0x55e554456b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.061+0000 7fe393400640 1 -- 172.21.15.40:0/1760770384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5540cc2d0 con 0x55e5544567b0 2024-04-26T23:07:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.062+0000 7fe393400640 1 --2- 172.21.15.40:0/1760770384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 0x55e554458910 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fe38407a0d0 tx=0x7fe384032440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e19d5624baa6caf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.062+0000 7fe383e00640 1 -- 172.21.15.40:0/1760770384 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38407a140 con 0x55e5544567b0 2024-04-26T23:07:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.062+0000 7fe383e00640 1 -- 172.21.15.40:0/1760770384 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe38407e4f0 con 0x55e5544567b0 2024-04-26T23:07:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.062+0000 7fe383e00640 1 -- 172.21.15.40:0/1760770384 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe384082d20 con 0x55e5544567b0 2024-04-26T23:07:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.063+0000 7fe39d5d7740 1 -- 172.21.15.40:0/1760770384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 msgr2=0x55e554458910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.063+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/1760770384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 0x55e554458910 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fe38407a0d0 tx=0x7fe384032440 comp rx=0 tx=0).stop 2024-04-26T23:07:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.063+0000 7fe39d5d7740 1 -- 172.21.15.40:0/1760770384 shutdown_connections 2024-04-26T23:07:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.063+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/1760770384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554481270 0x55e5544816f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.063+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/1760770384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554492e80 0x55e554456b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.063+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/1760770384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 0x55e554458910 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.063+0000 7fe39d5d7740 1 -- 172.21.15.40:0/1760770384 >> 172.21.15.40:0/1760770384 conn(0x55e5543ead10 msgr2=0x55e55449c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:11.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.063+0000 7fe39d5d7740 1 -- 172.21.15.40:0/1760770384 shutdown_connections 2024-04-26T23:07:11.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.063+0000 7fe39d5d7740 1 -- 172.21.15.40:0/1760770384 wait complete. 2024-04-26T23:07:11.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.063+0000 7fe39d5d7740 1 -- start start 2024-04-26T23:07:11.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.064+0000 7fe39d5d7740 1 Processor -- start 2024-04-26T23:07:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe39d5d7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 0x55e554496780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe39d5d7740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554481270 0x55e554494710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe39d5d7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554492e80 0x55e554494c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe39d5d7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e5543ba600 con 0x55e5544567b0 2024-04-26T23:07:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe39d5d7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e554341f70 con 0x55e554492e80 2024-04-26T23:07:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe39d5d7740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e55447ee70 con 0x55e554481270 2024-04-26T23:07:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe39d5d7740 10 client.0 _ll_register_callbacks cb 0x55e5543b0480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe393400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 0x55e554496780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe393400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 0x55e554496780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54510/0 (socket says 172.21.15.40:54510) 2024-04-26T23:07:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe393400640 1 -- 172.21.15.40:0/52731738 learned_addr learned my addr 172.21.15.40:0/52731738 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe393400640 1 -- 172.21.15.40:0/52731738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554481270 msgr2=0x55e554494710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe393400640 1 --2- 172.21.15.40:0/52731738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554481270 0x55e554494710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe393400640 1 -- 172.21.15.40:0/52731738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554492e80 msgr2=0x55e554494c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe393400640 1 --2- 172.21.15.40:0/52731738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554492e80 0x55e554494c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.065+0000 7fe393400640 1 -- 172.21.15.40:0/52731738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5541966b0 con 0x55e5544567b0 2024-04-26T23:07:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.066+0000 7fe393400640 1 --2- 172.21.15.40:0/52731738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 0x55e554496780 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fe384079e40 tx=0x7fe384039fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535b20e33a2d485d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.066+0000 7fe380c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.066+0000 7fe380c00640 1 -- 172.21.15.40:0/52731738 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe384096020 con 0x55e5544567b0 2024-04-26T23:07:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.066+0000 7fe380c00640 1 -- 172.21.15.40:0/52731738 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fe3840767b0 con 0x55e5544567b0 2024-04-26T23:07:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.066+0000 7fe39d5d7740 1 -- 172.21.15.40:0/52731738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe38407e4f0 con 0x55e5544567b0 2024-04-26T23:07:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.066+0000 7fe380c00640 1 -- 172.21.15.40:0/52731738 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fe38407a140 con 0x55e5544567b0 2024-04-26T23:07:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.066+0000 7fe39d5d7740 10 client.7671 fetch_fsmap learned FSMap version 79 2024-04-26T23:07:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.066+0000 7fe39d5d7740 1 -- 172.21.15.40:0/52731738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) -- 0x55e5544845f0 con 0x55e5544567b0 2024-04-26T23:07:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.066+0000 7fe380c00640 1 -- 172.21.15.40:0/52731738 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe38407e4f0 con 0x55e5544567b0 2024-04-26T23:07:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.066+0000 7fe380c00640 1 -- 172.21.15.40:0/52731738 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 79) ==== 3534+0+0 (secure 0 0 0) 0x7fe38403a070 con 0x55e5544567b0 2024-04-26T23:07:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.066+0000 7fe39d5d7740 10 client.7671 fetch_fsmap finished waiting for FSMap version 79 2024-04-26T23:07:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.066+0000 7fe39d5d7740 10 client.7671 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:07:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.067+0000 7fe39d5d7740 20 client.7671 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.067+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/52731738 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55e5544b7070 0x55e5544b9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.067+0000 7fe39d5d7740 4 client.7671 mds_command: new command op to 7446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:07:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.067+0000 7fe39d5d7740 1 -- 172.21.15.40:0/52731738 --> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5543c8210 con 0x55e5544b7070 2024-04-26T23:07:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.067+0000 7fe38aa00640 1 --2- 172.21.15.40:0/52731738 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55e5544b7070 0x55e5544b9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.067+0000 7fe38aa00640 1 --2- 172.21.15.40:0/52731738 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55e5544b7070 0x55e5544b9460 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50cc1fbfcf44dad server_cookie=2189b407a8ee865e in_seq=0 out_seq=0 2024-04-26T23:07:11.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.067+0000 7fe380c00640 10 client.7671 ms_handle_connect on v2:172.21.15.40:6833/4227275373 2024-04-26T23:07:11.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.068+0000 7fe380c00640 1 -- 172.21.15.40:0/52731738 <== mds.0 v2:172.21.15.40:6833/4227275373 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5543c8210 con 0x55e5544b7070 2024-04-26T23:07:11.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.068+0000 7fe380c00640 10 client.7671 handle_command_reply: tid=0 2024-04-26T23:07:11.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.068+0000 7fe39d5d7740 1 client.7671 shutdown 2024-04-26T23:07:11.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.068+0000 7fe39d5d7740 1 -- 172.21.15.40:0/52731738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 msgr2=0x55e554496780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.068+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/52731738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 0x55e554496780 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fe384079e40 tx=0x7fe384039fe0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.068+0000 7fe39d5d7740 1 -- 172.21.15.40:0/52731738 shutdown_connections 2024-04-26T23:07:11.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.068+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/52731738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554492e80 0x55e554494c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.068+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/52731738 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554481270 0x55e554494710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.068+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/52731738 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55e5544b7070 0x55e5544b9460 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.069+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/52731738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e5544567b0 0x55e554496780 secure :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fe384079e40 tx=0x7fe384039fe0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.069+0000 7fe380c00640 0 client.7671 ms_handle_reset on v2:172.21.15.40:6833/4227275373 2024-04-26T23:07:11.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.069+0000 7fe39d5d7740 1 -- 172.21.15.40:0/52731738 >> 172.21.15.40:0/52731738 conn(0x55e5543ead10 msgr2=0x55e55449f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:11.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.069+0000 7fe39d5d7740 1 -- 172.21.15.40:0/52731738 shutdown_connections 2024-04-26T23:07:11.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.069+0000 7fe39d5d7740 1 -- 172.21.15.40:0/52731738 wait complete. 2024-04-26T23:07:11.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.069+0000 7fe39d5d7740 20 client.7671 trim_cache size 0 max 16384 2024-04-26T23:07:11.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.088+0000 7fe39d5d7740 1 Processor -- start 2024-04-26T23:07:11.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.089+0000 7fe39d5d7740 1 -- start start 2024-04-26T23:07:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.089+0000 7fe39d5d7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554481270 0x55e554495910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.089+0000 7fe39d5d7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 0x55e55447c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.089+0000 7fe39d5d7740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554495e50 0x55e55447ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.089+0000 7fe39d5d7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe394200cf0 con 0x55e554492e80 2024-04-26T23:07:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.089+0000 7fe39d5d7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe394201390 con 0x55e554481270 2024-04-26T23:07:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.089+0000 7fe39d5d7740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe3941f8a00 con 0x55e554495e50 2024-04-26T23:07:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.090+0000 7fe38aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 0x55e55447c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.090+0000 7fe38aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 0x55e55447c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54518/0 (socket says 172.21.15.40:54518) 2024-04-26T23:07:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.090+0000 7fe38aa00640 1 -- 172.21.15.40:0/2273164891 learned_addr learned my addr 172.21.15.40:0/2273164891 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.090+0000 7fe38aa00640 1 -- 172.21.15.40:0/2273164891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554495e50 msgr2=0x55e55447ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.090+0000 7fe38aa00640 1 --2- 172.21.15.40:0/2273164891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554495e50 0x55e55447ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.090+0000 7fe38aa00640 1 -- 172.21.15.40:0/2273164891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554481270 msgr2=0x55e554495910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.090+0000 7fe38aa00640 1 --2- 172.21.15.40:0/2273164891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554481270 0x55e554495910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.090+0000 7fe38aa00640 1 -- 172.21.15.40:0/2273164891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e55447ec10 con 0x55e554492e80 2024-04-26T23:07:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.090+0000 7fe38aa00640 1 --2- 172.21.15.40:0/2273164891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 0x55e55447c910 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fe37c007560 tx=0x7fe37c001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2ea1e82e5792e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.091+0000 7fe383400640 1 -- 172.21.15.40:0/2273164891 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37c026070 con 0x55e554492e80 2024-04-26T23:07:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.091+0000 7fe383400640 1 -- 172.21.15.40:0/2273164891 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe37c0203b0 con 0x55e554492e80 2024-04-26T23:07:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.091+0000 7fe383400640 1 -- 172.21.15.40:0/2273164891 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37c004e60 con 0x55e554492e80 2024-04-26T23:07:11.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.091+0000 7fe39d5d7740 1 -- 172.21.15.40:0/2273164891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 msgr2=0x55e55447c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.091+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/2273164891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 0x55e55447c910 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fe37c007560 tx=0x7fe37c001080 comp rx=0 tx=0).stop 2024-04-26T23:07:11.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.091+0000 7fe39d5d7740 1 -- 172.21.15.40:0/2273164891 shutdown_connections 2024-04-26T23:07:11.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.091+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/2273164891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554495e50 0x55e55447ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.091+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/2273164891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 0x55e55447c910 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.091+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/2273164891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554481270 0x55e554495910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.091+0000 7fe39d5d7740 1 -- 172.21.15.40:0/2273164891 >> 172.21.15.40:0/2273164891 conn(0x55e5543ead10 msgr2=0x55e554454570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:11.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.091+0000 7fe39d5d7740 1 -- 172.21.15.40:0/2273164891 shutdown_connections 2024-04-26T23:07:11.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.091+0000 7fe39d5d7740 1 -- 172.21.15.40:0/2273164891 wait complete. 2024-04-26T23:07:11.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.092+0000 7fe39d5d7740 1 -- start start 2024-04-26T23:07:11.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.092+0000 7fe39d5d7740 1 Processor -- start 2024-04-26T23:07:11.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.093+0000 7fe39d5d7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554481270 0x55e5544bd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.093+0000 7fe39d5d7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 0x55e5544bd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.093+0000 7fe39d5d7740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554495e50 0x55e554491c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.093+0000 7fe39d5d7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e5543ba600 con 0x55e554492e80 2024-04-26T23:07:11.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.093+0000 7fe39d5d7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e55447ee70 con 0x55e554481270 2024-04-26T23:07:11.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.093+0000 7fe39d5d7740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e554341f70 con 0x55e554495e50 2024-04-26T23:07:11.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.093+0000 7fe39d5d7740 10 client.0 _ll_register_callbacks cb 0x7fe384079d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:11.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.094+0000 7fe38aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 0x55e5544bd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.094+0000 7fe38aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 0x55e5544bd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54532/0 (socket says 172.21.15.40:54532) 2024-04-26T23:07:11.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.094+0000 7fe38aa00640 1 -- 172.21.15.40:0/1606758744 learned_addr learned my addr 172.21.15.40:0/1606758744 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:11.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.094+0000 7fe393400640 1 --2- 172.21.15.40:0/1606758744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554481270 0x55e5544bd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.094+0000 7fe38aa00640 1 -- 172.21.15.40:0/1606758744 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554495e50 msgr2=0x55e554491c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:11.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.094+0000 7fe38aa00640 1 --2- 172.21.15.40:0/1606758744 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554495e50 0x55e554491c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.094+0000 7fe38aa00640 1 -- 172.21.15.40:0/1606758744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554481270 msgr2=0x55e5544bd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:11.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.094+0000 7fe38aa00640 1 --2- 172.21.15.40:0/1606758744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554481270 0x55e5544bd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.094+0000 7fe38aa00640 1 -- 172.21.15.40:0/1606758744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5544aed30 con 0x55e554492e80 2024-04-26T23:07:11.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.094+0000 7fe393400640 1 --2- 172.21.15.40:0/1606758744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554481270 0x55e5544bd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:11.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.094+0000 7fe38aa00640 1 --2- 172.21.15.40:0/1606758744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 0x55e5544bd9c0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fe394157e70 tx=0x7fe37c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca111967290e116 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:11.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.094+0000 7fe380c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:11.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe380c00640 1 -- 172.21.15.40:0/1606758744 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37c026070 con 0x55e554492e80 2024-04-26T23:07:11.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe380c00640 1 -- 172.21.15.40:0/1606758744 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fe37c014430 con 0x55e554492e80 2024-04-26T23:07:11.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe39d5d7740 1 -- 172.21.15.40:0/1606758744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe37c0203b0 con 0x55e554492e80 2024-04-26T23:07:11.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe380c00640 1 -- 172.21.15.40:0/1606758744 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fe37c035050 con 0x55e554492e80 2024-04-26T23:07:11.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe39d5d7740 10 client.7677 fetch_fsmap learned FSMap version 79 2024-04-26T23:07:11.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe39d5d7740 1 -- 172.21.15.40:0/1606758744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) -- 0x55e5545165f0 con 0x55e554492e80 2024-04-26T23:07:11.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe380c00640 1 -- 172.21.15.40:0/1606758744 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe37c0203b0 con 0x55e554492e80 2024-04-26T23:07:11.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe380c00640 1 -- 172.21.15.40:0/1606758744 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 79) ==== 3534+0+0 (secure 0 0 0) 0x7fe37c038020 con 0x55e554492e80 2024-04-26T23:07:11.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe39d5d7740 10 client.7677 fetch_fsmap finished waiting for FSMap version 79 2024-04-26T23:07:11.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe39d5d7740 10 client.7677 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:07:11.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe39d5d7740 20 client.7677 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:11.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/1606758744 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55e554515330 0x55e554515710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe39d5d7740 4 client.7677 mds_command: new command op to 7446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:07:11.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.095+0000 7fe39d5d7740 1 -- 172.21.15.40:0/1606758744 --> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] -- command(tid 0: {"prefix": "status"}) -- 0x55e5543c2080 con 0x55e554515330 2024-04-26T23:07:11.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.096+0000 7fe393400640 1 --2- 172.21.15.40:0/1606758744 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55e554515330 0x55e554515710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.096+0000 7fe393400640 1 --2- 172.21.15.40:0/1606758744 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55e554515330 0x55e554515710 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35638a0d5e6b8a77 server_cookie=6e1a38613e6fa838 in_seq=0 out_seq=0 2024-04-26T23:07:11.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.096+0000 7fe380c00640 10 client.7677 ms_handle_connect on v2:172.21.15.40:6833/4227275373 2024-04-26T23:07:11.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.096+0000 7fe380c00640 1 -- 172.21.15.40:0/1606758744 <== mds.0 v2:172.21.15.40:6833/4227275373 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55e5543c2080 con 0x55e554515330 2024-04-26T23:07:11.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.096+0000 7fe380c00640 10 client.7677 handle_command_reply: tid=0 2024-04-26T23:07:11.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.096+0000 7fe39d5d7740 1 client.7677 shutdown 2024-04-26T23:07:11.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.097+0000 7fe39d5d7740 1 -- 172.21.15.40:0/1606758744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 msgr2=0x55e5544bd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.097+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/1606758744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 0x55e5544bd9c0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fe394157e70 tx=0x7fe37c00f000 comp rx=0 tx=0).stop 2024-04-26T23:07:11.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.097+0000 7fe39d5d7740 1 -- 172.21.15.40:0/1606758744 shutdown_connections 2024-04-26T23:07:11.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.097+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/1606758744 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e554495e50 0x55e554491c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.097+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/1606758744 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x55e554515330 0x55e554515710 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.097+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/1606758744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e554492e80 0x55e5544bd9c0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.097+0000 7fe39d5d7740 1 --2- 172.21.15.40:0/1606758744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e554481270 0x55e5544bd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.097+0000 7fe39d5d7740 1 -- 172.21.15.40:0/1606758744 >> 172.21.15.40:0/1606758744 conn(0x55e5543ead10 msgr2=0x55e554497f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:11.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.097+0000 7fe380c00640 0 client.7677 ms_handle_reset on v2:172.21.15.40:6833/4227275373 2024-04-26T23:07:11.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.097+0000 7fe39d5d7740 1 -- 172.21.15.40:0/1606758744 shutdown_connections 2024-04-26T23:07:11.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.097+0000 7fe39d5d7740 1 -- 172.21.15.40:0/1606758744 wait complete. 2024-04-26T23:07:11.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.097+0000 7fe39d5d7740 20 client.7677 trim_cache size 0 max 16384 2024-04-26T23:07:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:07:11.103 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:07:11.103 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:07:11.103 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 7446, 2024-04-26T23:07:11.103 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:07:11.103 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:07:11.103 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:07:11.103 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 12.691668813, 2024-04-26T23:07:11.103 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 79, 2024-04-26T23:07:11.103 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 79, 2024-04-26T23:07:11.103 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 79, 2024-04-26T23:07:11.103 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 19.120113224000001 2024-04-26T23:07:11.103 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:07:11.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.098+0000 7fe399000640 1 -- 172.21.15.40:0/147683621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe37006c8f0 msgr2=0x7fe37006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.098+0000 7fe399000640 1 --2- 172.21.15.40:0/147683621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe37006c8f0 0x7fe37006edb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe37c007fd0 tx=0x7fe37c007920 comp rx=0 tx=0).stop 2024-04-26T23:07:11.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.098+0000 7fe399000640 1 -- 172.21.15.40:0/147683621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe394071fe0 msgr2=0x7fe39415f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.098+0000 7fe399000640 1 --2- 172.21.15.40:0/147683621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe394071fe0 0x7fe39415f970 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fe384009890 tx=0x7fe3840046c0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.098+0000 7fe399000640 1 -- 172.21.15.40:0/147683621 shutdown_connections 2024-04-26T23:07:11.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.098+0000 7fe399000640 1 --2- 172.21.15.40:0/147683621 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe37006c8f0 0x7fe37006edb0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.098+0000 7fe399000640 1 --2- 172.21.15.40:0/147683621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe39415ce60 0x7fe3940b4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.098+0000 7fe399000640 1 --2- 172.21.15.40:0/147683621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3940af7a0 0x7fe39415e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.098+0000 7fe399000640 1 --2- 172.21.15.40:0/147683621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe394071fe0 0x7fe39415f970 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.098+0000 7fe399000640 1 -- 172.21.15.40:0/147683621 >> 172.21.15.40:0/147683621 conn(0x7fe3940d0a10 msgr2=0x7fe3940b05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:11.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.098+0000 7fe399000640 1 -- 172.21.15.40:0/147683621 shutdown_connections 2024-04-26T23:07:11.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.098+0000 7fe399000640 1 -- 172.21.15.40:0/147683621 wait complete. 2024-04-26T23:07:11.116 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:11.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.320+0000 7f71d0000640 1 Processor -- start 2024-04-26T23:07:11.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.320+0000 7f71d0000640 1 -- start start 2024-04-26T23:07:11.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.321+0000 7f71d0000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f71c8073b70 0x7f71c80aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.321+0000 7f71d0000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71c8072c00 con 0x7f71c80cbb50 2024-04-26T23:07:11.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.321+0000 7f71d0000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71c8072ee0 con 0x7f71c815f380 2024-04-26T23:07:11.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.321+0000 7f71d0000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f71c80729b0 con 0x7f71c8073b70 2024-04-26T23:07:11.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.322+0000 7f71ccc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f71c8073b70 0x7f71c80aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.322+0000 7f71ce000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f71c80cbb50 0x7f71c80af1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43230/0 (socket says 172.21.15.40:43230) 2024-04-26T23:07:11.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.322+0000 7f71ccc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f71c8073b70 0x7f71c80aecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54544/0 (socket says 172.21.15.40:54544) 2024-04-26T23:07:11.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.322+0000 7f71ce000640 1 -- 172.21.15.40:0/3575231270 learned_addr learned my addr 172.21.15.40:0/3575231270 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:11.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.322+0000 7f71ccc00640 1 -- 172.21.15.40:0/3575231270 >> v1:172.21.15.103:6789/0 conn(0x7f71c815f380 legacy=0x7f71c80cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.322+0000 7f71ccc00640 1 -- 172.21.15.40:0/3575231270 >> v1:172.21.15.40:6789/0 conn(0x7f71c80cbb50 legacy=0x7f71c80af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.322+0000 7f71ccc00640 1 -- 172.21.15.40:0/3575231270 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c80731c0 con 0x7f71c8073b70 2024-04-26T23:07:11.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.322+0000 7f71c7e00640 1 -- 172.21.15.40:0/3575231270 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3839064973 0 0) 0x7f71c8072c00 con 0x7f71c80cbb50 2024-04-26T23:07:11.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.324+0000 7f71ccc00640 1 --2- 172.21.15.40:0/3575231270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71c8073b70 0x7f71c80aecb0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f71b40097a0 tx=0x7f71b402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a99b15c5630fdc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:11.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.324+0000 7f71c7e00640 1 -- 172.21.15.40:0/3575231270 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b4037070 con 0x7f71c8073b70 2024-04-26T23:07:11.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.324+0000 7f71c7e00640 1 -- 172.21.15.40:0/3575231270 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71b4032440 con 0x7f71c8073b70 2024-04-26T23:07:11.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.324+0000 7f71c7e00640 1 -- 172.21.15.40:0/3575231270 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b4032810 con 0x7f71c8073b70 2024-04-26T23:07:11.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.325+0000 7f71d0000640 1 -- 172.21.15.40:0/3575231270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71c8073b70 msgr2=0x7f71c80aecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.325+0000 7f71d0000640 1 --2- 172.21.15.40:0/3575231270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71c8073b70 0x7f71c80aecb0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f71b40097a0 tx=0x7f71b402a230 comp rx=0 tx=0).stop 2024-04-26T23:07:11.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.325+0000 7f71d0000640 1 -- 172.21.15.40:0/3575231270 shutdown_connections 2024-04-26T23:07:11.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.325+0000 7f71d0000640 1 --2- 172.21.15.40:0/3575231270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71c8073b70 0x7f71c80aecb0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.325+0000 7f71d0000640 1 -- 172.21.15.40:0/3575231270 >> 172.21.15.40:0/3575231270 conn(0x7f71c80d0be0 msgr2=0x7f71c80d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:11.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.325+0000 7f71d0000640 1 -- 172.21.15.40:0/3575231270 shutdown_connections 2024-04-26T23:07:11.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.325+0000 7f71d0000640 1 -- 172.21.15.40:0/3575231270 wait complete. 2024-04-26T23:07:11.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.327+0000 7f71d0000640 1 Processor -- start 2024-04-26T23:07:11.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.327+0000 7f71d0000640 1 -- start start 2024-04-26T23:07:11.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71d0000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71c80cbb50 0x7f71c80c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71d0000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71c815f380 0x7f71c806f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71d0000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71c80c6100 0x7f71c80b5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71d0000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f71c8073620 con 0x7f71c80cbb50 2024-04-26T23:07:11.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71d0000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f71c80d08f0 con 0x7f71c815f380 2024-04-26T23:07:11.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71d0000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f71c80737a0 con 0x7f71c80c6100 2024-04-26T23:07:11.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71cd600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71c80cbb50 0x7f71c80c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71cd600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71c80cbb50 0x7f71c80c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54550/0 (socket says 172.21.15.40:54550) 2024-04-26T23:07:11.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71cd600640 1 -- 172.21.15.40:0/764564312 learned_addr learned my addr 172.21.15.40:0/764564312 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:11.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71cd600640 1 -- 172.21.15.40:0/764564312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71c80c6100 msgr2=0x7f71c80b5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71cd600640 1 --2- 172.21.15.40:0/764564312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71c80c6100 0x7f71c80b5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71cd600640 1 -- 172.21.15.40:0/764564312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71c815f380 msgr2=0x7f71c806f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71cd600640 1 --2- 172.21.15.40:0/764564312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71c815f380 0x7f71c806f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.328+0000 7f71cd600640 1 -- 172.21.15.40:0/764564312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c80c6630 con 0x7f71c80cbb50 2024-04-26T23:07:11.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.329+0000 7f71cd600640 1 --2- 172.21.15.40:0/764564312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71c80cbb50 0x7f71c80c88e0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f71bc00c450 tx=0x7f71bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:11.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.329+0000 7f71c5600640 1 -- 172.21.15.40:0/764564312 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc0052d0 con 0x7f71c80cbb50 2024-04-26T23:07:11.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.329+0000 7f71c5600640 1 -- 172.21.15.40:0/764564312 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71bc005470 con 0x7f71c80cbb50 2024-04-26T23:07:11.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.329+0000 7f71c5600640 1 -- 172.21.15.40:0/764564312 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc012870 con 0x7f71c80cbb50 2024-04-26T23:07:11.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.329+0000 7f71d0000640 1 -- 172.21.15.40:0/764564312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71c80c6910 con 0x7f71c80cbb50 2024-04-26T23:07:11.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.329+0000 7f71d0000640 1 -- 172.21.15.40:0/764564312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71c80c6ed0 con 0x7f71c80cbb50 2024-04-26T23:07:11.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.330+0000 7f71d0000640 1 -- 172.21.15.40:0/764564312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71c80694f0 con 0x7f71c80cbb50 2024-04-26T23:07:11.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.331+0000 7f71c5600640 1 -- 172.21.15.40:0/764564312 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f71bc012a10 con 0x7f71c80cbb50 2024-04-26T23:07:11.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.332+0000 7f71c5600640 1 --2- 172.21.15.40:0/764564312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f71a806c8f0 0x7f71a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:11.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.332+0000 7f71c5600640 1 -- 172.21.15.40:0/764564312 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f71bc08b980 con 0x7f71c80cbb50 2024-04-26T23:07:11.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.333+0000 7f71ccc00640 1 --2- 172.21.15.40:0/764564312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f71a806c8f0 0x7f71a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:11.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.334+0000 7f71ccc00640 1 --2- 172.21.15.40:0/764564312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f71a806c8f0 0x7f71a806edb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f71b4034000 tx=0x7f71b402c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:11.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.338+0000 7f71c5600640 1 -- 172.21.15.40:0/764564312 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f71bc059b80 con 0x7f71c80cbb50 2024-04-26T23:07:11.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.588+0000 7f71d0000640 1 -- 172.21.15.40:0/764564312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71c80ce290 con 0x7f71c80cbb50 2024-04-26T23:07:11.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.589+0000 7f71c5600640 1 -- 172.21.15.40:0/764564312 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+5221 (secure 0 0 0) 0x7f71bc059480 con 0x7f71c80cbb50 2024-04-26T23:07:11.589 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:11.589 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":79,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/755627718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":755627718},{"type":"v1","addr":"172.21.15.103:6837","nonce":755627718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:06:57.853769+0000","modified":"2024-04-26T23:07:09.523854+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7446,"mds_1":7443,"mds_2":5965},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5965":{"gid":5965,"name":"d","rank":2,"incarnation":78,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6836/2914603370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2914603370},{"type":"v1","addr":"172.21.15.103:6836","nonce":2914603370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7443":{"gid":7443,"name":"a","rank":1,"incarnation":76,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/216893598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":216893598},{"type":"v1","addr":"172.21.15.40:6834","nonce":216893598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7446":{"gid":7446,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4227275373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":4227275373},{"type":"v1","addr":"172.21.15.40:6835","nonce":4227275373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7446,"qdb_cluster":[5965,7443,7446]},"id":5}]} 2024-04-26T23:07:11.590 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 79 2024-04-26T23:07:11.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.591+0000 7f71d0000640 1 -- 172.21.15.40:0/764564312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f71a806c8f0 msgr2=0x7f71a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.591+0000 7f71d0000640 1 --2- 172.21.15.40:0/764564312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f71a806c8f0 0x7f71a806edb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f71b4034000 tx=0x7f71b402c000 comp rx=0 tx=0).stop 2024-04-26T23:07:11.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.591+0000 7f71d0000640 1 -- 172.21.15.40:0/764564312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71c80cbb50 msgr2=0x7f71c80c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:11.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.591+0000 7f71d0000640 1 --2- 172.21.15.40:0/764564312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71c80cbb50 0x7f71c80c88e0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f71bc00c450 tx=0x7f71bc00c910 comp rx=0 tx=0).stop 2024-04-26T23:07:11.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.592+0000 7f71d0000640 1 -- 172.21.15.40:0/764564312 shutdown_connections 2024-04-26T23:07:11.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.592+0000 7f71d0000640 1 --2- 172.21.15.40:0/764564312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f71a806c8f0 0x7f71a806edb0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.592+0000 7f71d0000640 1 --2- 172.21.15.40:0/764564312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71c80c6100 0x7f71c80b5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.592+0000 7f71d0000640 1 --2- 172.21.15.40:0/764564312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71c815f380 0x7f71c806f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.592+0000 7f71d0000640 1 --2- 172.21.15.40:0/764564312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71c80cbb50 0x7f71c80c88e0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:11.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.592+0000 7f71d0000640 1 -- 172.21.15.40:0/764564312 >> 172.21.15.40:0/764564312 conn(0x7f71c80d0be0 msgr2=0x7f71c80cd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:11.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.592+0000 7f71d0000640 1 -- 172.21.15.40:0/764564312 shutdown_connections 2024-04-26T23:07:11.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:11.592+0000 7f71d0000640 1 -- 172.21.15.40:0/764564312 wait complete. 2024-04-26T23:07:11.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:11.611 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p d1/d2/d3/d4/d5/d6/d7/d8) 2024-04-26T23:07:11.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:11.690 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec sync d1/d2/d3/d4/d5/d6/d7/d8) 2024-04-26T23:07:11.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:11.756 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 d1/d2) 2024-04-26T23:07:11.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:11.807 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 d1/d2/d3/d4) 2024-04-26T23:07:11.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:11.858 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 2 d1/d2/d3/d4/d5/d6) 2024-04-26T23:07:11.913 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:07:12.149 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"7","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:07:12.149 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:07:12.150 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:07:12.776 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:07:12.775+0000 7fdbb2a00640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T23:07:14.150 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:07:14.386 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"7","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:07:14.387 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:07:14.387 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:07:16.388 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:07:16.626 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"7","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:07:16.626 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:07:16.627 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:07:18.609 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:07:18.618 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:07:18.628 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:07:18.873 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"13","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/d1/d2","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-04-26T23:07:18.874 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000001", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/d1/d2/d3/d4", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-04-26T23:07:18.874 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:07:18.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:07:19.103 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:07:19.103 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:07:19.103 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:07:21.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:07:21.343 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:07:21.343 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:07:21.344 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:07:23.345 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:07:23.584 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1,"2":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":2,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/d1/d2/d3/d4/d5/d6","dirfrag":"0x10000000005","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:07:23.584 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1, "2": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "2", "dirfrag": "0x10000000005", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/d1/d2/d3/d4/d5/d6", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "6", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 2, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:07:23.584 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:07:23.585 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-04-26T23:07:23.815 INFO:teuthology.orchestra.run.smithi103.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"0","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dnwaiter":0,"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":2,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/d1/d2/d3/d4/d5/d6","dirfrag":"0x10000000005","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:07:23.816 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dnwaiter": 0, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "2", "dirfrag": "0x10000000005", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4/d5/d6", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 2, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:07:23.816 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:07:23.816 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 flush journal 2024-04-26T23:07:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.041+0000 7f5035e00640 1 Processor -- start 2024-04-26T23:07:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.041+0000 7f5035e00640 1 -- start start 2024-04-26T23:07:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.043+0000 7f5035e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5030071e60 0x7f503006f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.043+0000 7f5035e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f50300cc480 0x7f5030166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.043+0000 7f5035e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f503015d7c0 con 0x7f503006fe20 2024-04-26T23:07:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.043+0000 7f5035e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f503015daa0 con 0x7f50300cc480 2024-04-26T23:07:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.043+0000 7f5035e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f503015dd80 con 0x7f5030071e60 2024-04-26T23:07:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.043+0000 7f502e000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f503006fe20 0x7f5030072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33208/0 (socket says 172.21.15.40:33208) 2024-04-26T23:07:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.043+0000 7f502e000640 1 -- 172.21.15.40:0/1675383743 learned_addr learned my addr 172.21.15.40:0/1675383743 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.043+0000 7f502f400640 1 --2- 172.21.15.40:0/1675383743 >> v2:172.21.15.40:3300/0 conn(0x7f50300cc480 0x7f5030166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.043+0000 7f502ea00640 1 --2- 172.21.15.40:0/1675383743 >> v2:172.21.15.103:3300/0 conn(0x7f5030071e60 0x7f503006f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.043+0000 7f502d600640 1 -- 172.21.15.40:0/1675383743 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4021382146 0 0) 0x7f503015d7c0 con 0x7f503006fe20 2024-04-26T23:07:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.044+0000 7f502d600640 1 -- 172.21.15.40:0/1675383743 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5010003520 con 0x7f503006fe20 2024-04-26T23:07:24.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.044+0000 7f502f400640 1 -- 172.21.15.40:0/1675383743 >> v2:172.21.15.103:3300/0 conn(0x7f5030071e60 msgr2=0x7f503006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.044+0000 7f502f400640 1 --2- 172.21.15.40:0/1675383743 >> v2:172.21.15.103:3300/0 conn(0x7f5030071e60 0x7f503006f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.044+0000 7f502f400640 1 -- 172.21.15.40:0/1675383743 >> v1:172.21.15.40:6789/0 conn(0x7f503006fe20 legacy=0x7f5030072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.044+0000 7f502f400640 1 -- 172.21.15.40:0/1675383743 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f503015e060 con 0x7f50300cc480 2024-04-26T23:07:24.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.045+0000 7f502f400640 1 --2- 172.21.15.40:0/1675383743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50300cc480 0x7f5030166ad0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f50300b3700 tx=0x7f502802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=180827f11ecc38f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.045+0000 7f502d600640 1 -- 172.21.15.40:0/1675383743 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028007740 con 0x7f50300cc480 2024-04-26T23:07:24.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.045+0000 7f502d600640 1 -- 172.21.15.40:0/1675383743 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50280078e0 con 0x7f50300cc480 2024-04-26T23:07:24.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.045+0000 7f502d600640 1 -- 172.21.15.40:0/1675383743 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028007c90 con 0x7f50300cc480 2024-04-26T23:07:24.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.046+0000 7f5035e00640 1 -- 172.21.15.40:0/1675383743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50300cc480 msgr2=0x7f5030166ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.046+0000 7f5035e00640 1 --2- 172.21.15.40:0/1675383743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50300cc480 0x7f5030166ad0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f50300b3700 tx=0x7f502802a680 comp rx=0 tx=0).stop 2024-04-26T23:07:24.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.046+0000 7f5035e00640 1 -- 172.21.15.40:0/1675383743 shutdown_connections 2024-04-26T23:07:24.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.046+0000 7f5035e00640 1 --2- 172.21.15.40:0/1675383743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50300cc480 0x7f5030166ad0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.046+0000 7f5035e00640 1 --2- 172.21.15.40:0/1675383743 >> v2:172.21.15.103:3300/0 conn(0x7f5030071e60 0x7f503006f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.046+0000 7f5035e00640 1 -- 172.21.15.40:0/1675383743 >> 172.21.15.40:0/1675383743 conn(0x7f50300d0be0 msgr2=0x7f50300d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.046+0000 7f5035e00640 1 -- 172.21.15.40:0/1675383743 shutdown_connections 2024-04-26T23:07:24.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.046+0000 7f5035e00640 1 -- 172.21.15.40:0/1675383743 wait complete. 2024-04-26T23:07:24.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.048+0000 7f5035e00640 1 Processor -- start 2024-04-26T23:07:24.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.048+0000 7f5035e00640 1 -- start start 2024-04-26T23:07:24.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.049+0000 7f5035e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f503006fe20 0x7f50300c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.049+0000 7f5035e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5030071e60 0x7f50300c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.049+0000 7f5035e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50300cc480 0x7f50300c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.049+0000 7f5035e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f503015d570 con 0x7f503006fe20 2024-04-26T23:07:24.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.049+0000 7f5035e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f503015d3f0 con 0x7f5030071e60 2024-04-26T23:07:24.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.049+0000 7f5035e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5030072950 con 0x7f50300cc480 2024-04-26T23:07:24.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.049+0000 7f502ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f503006fe20 0x7f50300c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.049+0000 7f502ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f503006fe20 0x7f50300c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37758/0 (socket says 172.21.15.40:37758) 2024-04-26T23:07:24.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.049+0000 7f502ea00640 1 -- 172.21.15.40:0/2950175507 learned_addr learned my addr 172.21.15.40:0/2950175507 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.050+0000 7f502ea00640 1 -- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50300cc480 msgr2=0x7f50300c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.050+0000 7f502ea00640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50300cc480 0x7f50300c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.050+0000 7f502ea00640 1 -- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5030071e60 msgr2=0x7f50300c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.050+0000 7f502e000640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5030071e60 0x7f50300c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.050+0000 7f502ea00640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5030071e60 0x7f50300c9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.050+0000 7f502ea00640 1 -- 172.21.15.40:0/2950175507 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50300cfd90 con 0x7f503006fe20 2024-04-26T23:07:24.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.050+0000 7f502e000640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5030071e60 0x7f50300c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:24.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.050+0000 7f502ea00640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f503006fe20 0x7f50300c9150 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f502000d800 tx=0x7f502000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.051+0000 7f5027400640 1 -- 172.21.15.40:0/2950175507 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502000b6c0 con 0x7f503006fe20 2024-04-26T23:07:24.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.051+0000 7f5027400640 1 -- 172.21.15.40:0/2950175507 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5020019070 con 0x7f503006fe20 2024-04-26T23:07:24.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.051+0000 7f5027400640 1 -- 172.21.15.40:0/2950175507 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502000ba80 con 0x7f503006fe20 2024-04-26T23:07:24.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.051+0000 7f5035e00640 1 -- 172.21.15.40:0/2950175507 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50300d0070 con 0x7f503006fe20 2024-04-26T23:07:24.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.051+0000 7f5035e00640 1 -- 172.21.15.40:0/2950175507 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50300d0630 con 0x7f503006fe20 2024-04-26T23:07:24.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.054+0000 7f5027400640 1 -- 172.21.15.40:0/2950175507 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f502000bc20 con 0x7f503006fe20 2024-04-26T23:07:24.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.055+0000 7f5027400640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f500c06c8f0 0x7f500c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.055+0000 7f5027400640 1 -- 172.21.15.40:0/2950175507 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f502008c270 con 0x7f503006fe20 2024-04-26T23:07:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.055+0000 7f5038b95740 1 Processor -- start 2024-04-26T23:07:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.056+0000 7f502e000640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f500c06c8f0 0x7f500c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.056+0000 7f5038b95740 1 -- start start 2024-04-26T23:07:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.056+0000 7f5038b95740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3634540 0x5628c3634920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.056+0000 7f5038b95740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3651de0 0x5628c3652330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f5038b95740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c360a220 0x5628c36504a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f5038b95740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5628c351a030 con 0x5628c3651de0 2024-04-26T23:07:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f5038b95740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5628c35357f0 con 0x5628c360a220 2024-04-26T23:07:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f5038b95740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5628c34f07e0 con 0x5628c3634540 2024-04-26T23:07:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f502e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3651de0 0x5628c3652330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f502e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3651de0 0x5628c3652330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37770/0 (socket says 172.21.15.40:37770) 2024-04-26T23:07:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f502e000640 1 -- 172.21.15.40:0/3389075494 learned_addr learned my addr 172.21.15.40:0/3389075494 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f502e000640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f500c06c8f0 0x7f500c06edb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f50301f8010 tx=0x7f5018009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f502ea00640 1 --2- 172.21.15.40:0/3389075494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3634540 0x5628c3634920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f502f400640 1 --2- 172.21.15.40:0/3389075494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c360a220 0x5628c36504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f502e000640 1 -- 172.21.15.40:0/3389075494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3634540 msgr2=0x5628c3634920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f502e000640 1 --2- 172.21.15.40:0/3389075494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3634540 0x5628c3634920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f502e000640 1 -- 172.21.15.40:0/3389075494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c360a220 msgr2=0x5628c36504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f502e000640 1 --2- 172.21.15.40:0/3389075494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c360a220 0x5628c36504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.057+0000 7f502e000640 1 -- 172.21.15.40:0/3389075494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628c34fc050 con 0x5628c3651de0 2024-04-26T23:07:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.058+0000 7f502e000640 1 --2- 172.21.15.40:0/3389075494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3651de0 0x5628c3652330 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f5018005b90 tx=0x7f5018010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f02f39f418526509 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.058+0000 7f501f400640 1 -- 172.21.15.40:0/3389075494 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501800d9b0 con 0x5628c3651de0 2024-04-26T23:07:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.058+0000 7f501f400640 1 -- 172.21.15.40:0/3389075494 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f501800db50 con 0x5628c3651de0 2024-04-26T23:07:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.058+0000 7f501f400640 1 -- 172.21.15.40:0/3389075494 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5018020450 con 0x5628c3651de0 2024-04-26T23:07:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.059+0000 7f5038b95740 1 -- 172.21.15.40:0/3389075494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3651de0 msgr2=0x5628c3652330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.059+0000 7f5038b95740 1 --2- 172.21.15.40:0/3389075494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3651de0 0x5628c3652330 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f5018005b90 tx=0x7f5018010000 comp rx=0 tx=0).stop 2024-04-26T23:07:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.059+0000 7f5038b95740 1 -- 172.21.15.40:0/3389075494 shutdown_connections 2024-04-26T23:07:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.059+0000 7f5038b95740 1 --2- 172.21.15.40:0/3389075494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c360a220 0x5628c36504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.059+0000 7f5038b95740 1 --2- 172.21.15.40:0/3389075494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3651de0 0x5628c3652330 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.059+0000 7f5038b95740 1 --2- 172.21.15.40:0/3389075494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3634540 0x5628c3634920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.059+0000 7f5038b95740 1 -- 172.21.15.40:0/3389075494 >> 172.21.15.40:0/3389075494 conn(0x5628c3606b50 msgr2=0x5628c364f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.059+0000 7f5038b95740 1 -- 172.21.15.40:0/3389075494 shutdown_connections 2024-04-26T23:07:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.059+0000 7f5038b95740 1 -- 172.21.15.40:0/3389075494 wait complete. 2024-04-26T23:07:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.059+0000 7f5038b95740 1 -- start start 2024-04-26T23:07:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.060+0000 7f5038b95740 1 Processor -- start 2024-04-26T23:07:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f5038b95740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c3636f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f5038b95740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3651de0 0x5628c36037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f5038b95740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c3631060 0x5628c3603cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f5038b95740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5628c3572200 con 0x5628c3634540 2024-04-26T23:07:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f5038b95740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5628c356cee0 con 0x5628c3631060 2024-04-26T23:07:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f5038b95740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5628c3579e70 con 0x5628c3651de0 2024-04-26T23:07:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f5038b95740 10 client.0 _ll_register_callbacks cb 0x5628c3563270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f502ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c3636f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f502ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c3636f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37772/0 (socket says 172.21.15.40:37772) 2024-04-26T23:07:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f502ea00640 1 -- 172.21.15.40:0/2739121524 learned_addr learned my addr 172.21.15.40:0/2739121524 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f502e000640 1 --2- 172.21.15.40:0/2739121524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3651de0 0x5628c36037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f502ea00640 1 -- 172.21.15.40:0/2739121524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3651de0 msgr2=0x5628c36037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f502ea00640 1 --2- 172.21.15.40:0/2739121524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3651de0 0x5628c36037b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f502ea00640 1 -- 172.21.15.40:0/2739121524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c3631060 msgr2=0x5628c3603cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f502ea00640 1 --2- 172.21.15.40:0/2739121524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c3631060 0x5628c3603cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.061+0000 7f502ea00640 1 -- 172.21.15.40:0/2739121524 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628c33497d0 con 0x5628c3634540 2024-04-26T23:07:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.062+0000 7f502ea00640 1 --2- 172.21.15.40:0/2739121524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c3636f90 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f502000b4b0 tx=0x7f502008c190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256c093819f2fbdf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.062+0000 7f5017e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.062+0000 7f5017e00640 1 -- 172.21.15.40:0/2739121524 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5020088060 con 0x5628c3634540 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.062+0000 7f5017e00640 1 -- 172.21.15.40:0/2739121524 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f50200563e0 con 0x5628c3634540 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.062+0000 7f5038b95740 1 -- 172.21.15.40:0/2739121524 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f501800db50 con 0x5628c3634540 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.062+0000 7f5017e00640 1 -- 172.21.15.40:0/2739121524 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f5020077050 con 0x5628c3634540 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.062+0000 7f5038b95740 10 client.7701 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.062+0000 7f5038b95740 1 -- 172.21.15.40:0/2739121524 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x5628c327f3f0 con 0x5628c3634540 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.062+0000 7f5017e00640 1 -- 172.21.15.40:0/2739121524 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f501800db50 con 0x5628c3634540 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.063+0000 7f5017e00640 1 -- 172.21.15.40:0/2739121524 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7f5020013070 con 0x5628c3634540 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.063+0000 7f5038b95740 10 client.7701 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.063+0000 7f5038b95740 10 client.7701 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.063+0000 7f5038b95740 20 client.7701 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.063+0000 7f5038b95740 1 --2- 172.21.15.40:0/2739121524 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x5628c36484b0 0x5628c36488d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.063+0000 7f5038b95740 4 client.7701 mds_command: new command op to 7446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.063+0000 7f5038b95740 1 -- 172.21.15.40:0/2739121524 --> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628c357b7e0 con 0x5628c36484b0 2024-04-26T23:07:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.063+0000 7f502e000640 1 --2- 172.21.15.40:0/2739121524 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x5628c36484b0 0x5628c36488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.064+0000 7f502e000640 1 --2- 172.21.15.40:0/2739121524 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x5628c36484b0 0x5628c36488d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5924ac6988d53508 server_cookie=246ececc3bdad3a3 in_seq=0 out_seq=0 2024-04-26T23:07:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.064+0000 7f5017e00640 10 client.7701 ms_handle_connect on v2:172.21.15.40:6833/4227275373 2024-04-26T23:07:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.064+0000 7f5017e00640 1 -- 172.21.15.40:0/2739121524 <== mds.0 v2:172.21.15.40:6833/4227275373 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5628c357b7e0 con 0x5628c36484b0 2024-04-26T23:07:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.064+0000 7f5017e00640 10 client.7701 handle_command_reply: tid=0 2024-04-26T23:07:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.064+0000 7f5038b95740 1 client.7701 shutdown 2024-04-26T23:07:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.065+0000 7f5038b95740 1 -- 172.21.15.40:0/2739121524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 msgr2=0x5628c3636f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.065+0000 7f5038b95740 1 --2- 172.21.15.40:0/2739121524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c3636f90 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f502000b4b0 tx=0x7f502008c190 comp rx=0 tx=0).stop 2024-04-26T23:07:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.065+0000 7f5038b95740 1 -- 172.21.15.40:0/2739121524 shutdown_connections 2024-04-26T23:07:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.065+0000 7f5038b95740 1 --2- 172.21.15.40:0/2739121524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c3631060 0x5628c3603cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.065+0000 7f5038b95740 1 --2- 172.21.15.40:0/2739121524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3651de0 0x5628c36037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.065+0000 7f5038b95740 1 --2- 172.21.15.40:0/2739121524 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x5628c36484b0 0x5628c36488d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.065+0000 7f5038b95740 1 --2- 172.21.15.40:0/2739121524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c3636f90 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.065+0000 7f5038b95740 1 -- 172.21.15.40:0/2739121524 >> 172.21.15.40:0/2739121524 conn(0x5628c360a220 msgr2=0x5628c36337f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.065+0000 7f5017e00640 0 client.7701 ms_handle_reset on v2:172.21.15.40:6833/4227275373 2024-04-26T23:07:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.065+0000 7f5038b95740 1 -- 172.21.15.40:0/2739121524 shutdown_connections 2024-04-26T23:07:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.065+0000 7f5038b95740 1 -- 172.21.15.40:0/2739121524 wait complete. 2024-04-26T23:07:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.065+0000 7f5038b95740 20 client.7701 trim_cache size 0 max 16384 2024-04-26T23:07:24.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.082+0000 7f5038b95740 1 Processor -- start 2024-04-26T23:07:24.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.082+0000 7f5038b95740 1 -- start start 2024-04-26T23:07:24.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.083+0000 7f5038b95740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c360a220 0x5628c36334f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.083+0000 7f5038b95740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3631060 0x5628c3633a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.083+0000 7f5038b95740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c36080f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.083+0000 7f5038b95740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5030201cf0 con 0x5628c3634540 2024-04-26T23:07:24.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.083+0000 7f5038b95740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5030200f40 con 0x5628c360a220 2024-04-26T23:07:24.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.083+0000 7f5038b95740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f50302015c0 con 0x5628c3631060 2024-04-26T23:07:24.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.083+0000 7f502f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c36080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.083+0000 7f502f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c36080f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37782/0 (socket says 172.21.15.40:37782) 2024-04-26T23:07:24.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.083+0000 7f502f400640 1 -- 172.21.15.40:0/348235521 learned_addr learned my addr 172.21.15.40:0/348235521 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.084+0000 7f502f400640 1 -- 172.21.15.40:0/348235521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3631060 msgr2=0x5628c3633a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.084+0000 7f502f400640 1 --2- 172.21.15.40:0/348235521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3631060 0x5628c3633a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.084+0000 7f502ea00640 1 --2- 172.21.15.40:0/348235521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c360a220 0x5628c36334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.084+0000 7f502f400640 1 -- 172.21.15.40:0/348235521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c360a220 msgr2=0x5628c36334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.084+0000 7f502f400640 1 --2- 172.21.15.40:0/348235521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c360a220 0x5628c36334f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.084+0000 7f502f400640 1 -- 172.21.15.40:0/348235521 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628c323f0f0 con 0x5628c3634540 2024-04-26T23:07:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.084+0000 7f502f400640 1 --2- 172.21.15.40:0/348235521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c36080f0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f5028009e50 tx=0x7f5028004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac02a03710e4961 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.084+0000 7f501ea00640 1 -- 172.21.15.40:0/348235521 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028009450 con 0x5628c3634540 2024-04-26T23:07:24.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.084+0000 7f501ea00640 1 -- 172.21.15.40:0/348235521 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5028003b90 con 0x5628c3634540 2024-04-26T23:07:24.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.085+0000 7f501ea00640 1 -- 172.21.15.40:0/348235521 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028004040 con 0x5628c3634540 2024-04-26T23:07:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.085+0000 7f5038b95740 1 -- 172.21.15.40:0/348235521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 msgr2=0x5628c36080f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.085+0000 7f5038b95740 1 --2- 172.21.15.40:0/348235521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c36080f0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f5028009e50 tx=0x7f5028004ab0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.085+0000 7f5038b95740 1 -- 172.21.15.40:0/348235521 shutdown_connections 2024-04-26T23:07:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.085+0000 7f5038b95740 1 --2- 172.21.15.40:0/348235521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c36080f0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.085+0000 7f5038b95740 1 --2- 172.21.15.40:0/348235521 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c3631060 0x5628c3633a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.085+0000 7f5038b95740 1 --2- 172.21.15.40:0/348235521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c360a220 0x5628c36334f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.085+0000 7f5038b95740 1 -- 172.21.15.40:0/348235521 >> 172.21.15.40:0/348235521 conn(0x5628c3638610 msgr2=0x5628c3605310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.085+0000 7f5038b95740 1 -- 172.21.15.40:0/348235521 shutdown_connections 2024-04-26T23:07:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.086+0000 7f5038b95740 1 -- 172.21.15.40:0/348235521 wait complete. 2024-04-26T23:07:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.086+0000 7f5038b95740 1 -- start start 2024-04-26T23:07:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.086+0000 7f5038b95740 1 Processor -- start 2024-04-26T23:07:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.087+0000 7f5038b95740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c360a220 0x5628c36705b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.087+0000 7f5038b95740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c3631060 0x5628c3670af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.087+0000 7f5038b95740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c364e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.087+0000 7f5038b95740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5628c357afd0 con 0x5628c3634540 2024-04-26T23:07:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.087+0000 7f5038b95740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5030200000 con 0x5628c3631060 2024-04-26T23:07:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.087+0000 7f5038b95740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5628c36617f0 con 0x5628c360a220 2024-04-26T23:07:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.087+0000 7f5038b95740 10 client.0 _ll_register_callbacks cb 0x7f50200073a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.088+0000 7f502f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c364e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.088+0000 7f502f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c364e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37796/0 (socket says 172.21.15.40:37796) 2024-04-26T23:07:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.088+0000 7f502f400640 1 -- 172.21.15.40:0/3674087753 learned_addr learned my addr 172.21.15.40:0/3674087753 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.088+0000 7f502ea00640 1 --2- 172.21.15.40:0/3674087753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c360a220 0x5628c36705b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.088+0000 7f502e000640 1 --2- 172.21.15.40:0/3674087753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c3631060 0x5628c3670af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.088+0000 7f502f400640 1 -- 172.21.15.40:0/3674087753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c360a220 msgr2=0x5628c36705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.088+0000 7f502f400640 1 --2- 172.21.15.40:0/3674087753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c360a220 0x5628c36705b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.088+0000 7f502f400640 1 -- 172.21.15.40:0/3674087753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c3631060 msgr2=0x5628c3670af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.088+0000 7f502f400640 1 --2- 172.21.15.40:0/3674087753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c3631060 0x5628c3670af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.088+0000 7f502f400640 1 -- 172.21.15.40:0/3674087753 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628c3535a30 con 0x5628c3634540 2024-04-26T23:07:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.089+0000 7f502f400640 1 --2- 172.21.15.40:0/3674087753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c364e500 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f502802a6c0 tx=0x7f5028030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287642d2b661b526 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.089+0000 7f5017e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.089+0000 7f5017e00640 1 -- 172.21.15.40:0/3674087753 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028053060 con 0x5628c3634540 2024-04-26T23:07:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.089+0000 7f5017e00640 1 -- 172.21.15.40:0/3674087753 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f5028003990 con 0x5628c3634540 2024-04-26T23:07:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.089+0000 7f5038b95740 1 -- 172.21.15.40:0/3674087753 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5028003b90 con 0x5628c3634540 2024-04-26T23:07:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.089+0000 7f5017e00640 1 -- 172.21.15.40:0/3674087753 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f5028059050 con 0x5628c3634540 2024-04-26T23:07:24.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.089+0000 7f5038b95740 10 client.7707 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:24.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.089+0000 7f5038b95740 1 -- 172.21.15.40:0/3674087753 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x5628c365de70 con 0x5628c3634540 2024-04-26T23:07:24.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.090+0000 7f5017e00640 1 -- 172.21.15.40:0/3674087753 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5028003b90 con 0x5628c3634540 2024-04-26T23:07:24.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.090+0000 7f5017e00640 1 -- 172.21.15.40:0/3674087753 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7f502805c020 con 0x5628c3634540 2024-04-26T23:07:24.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.090+0000 7f5038b95740 10 client.7707 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:24.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.090+0000 7f5038b95740 10 client.7707 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-04-26T23:07:24.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.090+0000 7f5038b95740 20 client.7707 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:24.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.090+0000 7f5038b95740 1 --2- 172.21.15.40:0/3674087753 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x5628c3651de0 0x5628c364ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.090+0000 7f5038b95740 4 client.7707 mds_command: new command op to 7446 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:07:24.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.090+0000 7f5038b95740 1 -- 172.21.15.40:0/3674087753 --> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5628c3575650 con 0x5628c3651de0 2024-04-26T23:07:24.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.090+0000 7f502ea00640 1 --2- 172.21.15.40:0/3674087753 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x5628c3651de0 0x5628c364ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.091+0000 7f502ea00640 1 --2- 172.21.15.40:0/3674087753 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x5628c3651de0 0x5628c364ea40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e17e90017249dcb9 server_cookie=a4f16a783746e511 in_seq=0 out_seq=0 2024-04-26T23:07:24.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.091+0000 7f5017e00640 10 client.7707 ms_handle_connect on v2:172.21.15.40:6833/4227275373 2024-04-26T23:07:24.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.121+0000 7f5017e00640 1 -- 172.21.15.40:0/3674087753 <== mds.0 v2:172.21.15.40:6833/4227275373 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5628c3575650 con 0x5628c3651de0 2024-04-26T23:07:24.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.121+0000 7f5017e00640 10 client.7707 handle_command_reply: tid=0 2024-04-26T23:07:24.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.121+0000 7f5038b95740 1 client.7707 shutdown 2024-04-26T23:07:24.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.121+0000 7f5038b95740 1 -- 172.21.15.40:0/3674087753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 msgr2=0x5628c364e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.121+0000 7f5038b95740 1 --2- 172.21.15.40:0/3674087753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c364e500 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f502802a6c0 tx=0x7f5028030000 comp rx=0 tx=0).stop 2024-04-26T23:07:24.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.121+0000 7f5038b95740 1 -- 172.21.15.40:0/3674087753 shutdown_connections 2024-04-26T23:07:24.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.121+0000 7f5038b95740 1 --2- 172.21.15.40:0/3674087753 >> [v2:172.21.15.40:6833/4227275373,v1:172.21.15.40:6835/4227275373] conn(0x5628c3651de0 0x5628c364ea40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.121+0000 7f5038b95740 1 --2- 172.21.15.40:0/3674087753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c3634540 0x5628c364e500 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.121+0000 7f5038b95740 1 --2- 172.21.15.40:0/3674087753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628c3631060 0x5628c3670af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.121+0000 7f5038b95740 1 --2- 172.21.15.40:0/3674087753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5628c360a220 0x5628c36705b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.121+0000 7f5017e00640 0 client.7707 ms_handle_reset on v2:172.21.15.40:6833/4227275373 2024-04-26T23:07:24.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.121+0000 7f5038b95740 1 -- 172.21.15.40:0/3674087753 >> 172.21.15.40:0/3674087753 conn(0x5628c3638610 msgr2=0x5628c35ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.122+0000 7f5038b95740 1 -- 172.21.15.40:0/3674087753 shutdown_connections 2024-04-26T23:07:24.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.122+0000 7f5038b95740 1 -- 172.21.15.40:0/3674087753 wait complete. 2024-04-26T23:07:24.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.122+0000 7f5038b95740 20 client.7707 trim_cache size 0 max 16384 2024-04-26T23:07:24.123 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:07:24.123 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:07:24.123 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:07:24.123 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:07:24.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.123+0000 7f5035e00640 1 -- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f500c06c8f0 msgr2=0x7f500c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.123+0000 7f5035e00640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f500c06c8f0 0x7f500c06edb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f50301f8010 tx=0x7f5018009040 comp rx=0 tx=0).stop 2024-04-26T23:07:24.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.123+0000 7f5035e00640 1 -- 172.21.15.40:0/2950175507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f503006fe20 msgr2=0x7f50300c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.123+0000 7f5035e00640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f503006fe20 0x7f50300c9150 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f502000d800 tx=0x7f502000dcc0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.123+0000 7f5035e00640 1 -- 172.21.15.40:0/2950175507 shutdown_connections 2024-04-26T23:07:24.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.123+0000 7f5035e00640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f500c06c8f0 0x7f500c06edb0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.123+0000 7f5035e00640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f50300cc480 0x7f50300c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.123+0000 7f5035e00640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5030071e60 0x7f50300c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.123+0000 7f5035e00640 1 --2- 172.21.15.40:0/2950175507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f503006fe20 0x7f50300c9150 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.123+0000 7f5035e00640 1 -- 172.21.15.40:0/2950175507 >> 172.21.15.40:0/2950175507 conn(0x7f50300d0be0 msgr2=0x7f50300b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.123+0000 7f5035e00640 1 -- 172.21.15.40:0/2950175507 shutdown_connections 2024-04-26T23:07:24.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.123+0000 7f5035e00640 1 -- 172.21.15.40:0/2950175507 wait complete. 2024-04-26T23:07:24.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:1 flush journal 2024-04-26T23:07:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.360+0000 7f2d4ea00640 1 Processor -- start 2024-04-26T23:07:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.361+0000 7f2d4ea00640 1 -- start start 2024-04-26T23:07:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.362+0000 7f2d4ea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2d4806f790 0x7f2d4806fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.362+0000 7f2d4ea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2d48072c50 0x7f2d48073060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.362+0000 7f2d4ea00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d48156840 con 0x7f2d48156460 2024-04-26T23:07:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.362+0000 7f2d4ea00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f2d480cbf20 con 0x7f2d4806f790 2024-04-26T23:07:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.362+0000 7f2d4ea00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2d480cc200 con 0x7f2d48072c50 2024-04-26T23:07:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.362+0000 7f2d47e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2d48156460 0x7f2d480cb4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33214/0 (socket says 172.21.15.40:33214) 2024-04-26T23:07:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.362+0000 7f2d47e00640 1 -- 172.21.15.40:0/1980258512 learned_addr learned my addr 172.21.15.40:0/1980258512 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.363+0000 7f2d47400640 1 --2- 172.21.15.40:0/1980258512 >> v2:172.21.15.40:3300/0 conn(0x7f2d4806f790 0x7f2d4806fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.363+0000 7f2d46a00640 1 -- 172.21.15.40:0/1980258512 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1483795721 0 0) 0x7f2d48156840 con 0x7f2d48156460 2024-04-26T23:07:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.363+0000 7f2d4ca00640 1 --2- 172.21.15.40:0/1980258512 >> v2:172.21.15.103:3300/0 conn(0x7f2d48072c50 0x7f2d48073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.363+0000 7f2d46a00640 1 -- 172.21.15.40:0/1980258512 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d28003520 con 0x7f2d48156460 2024-04-26T23:07:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.363+0000 7f2d47400640 1 -- 172.21.15.40:0/1980258512 >> v2:172.21.15.103:3300/0 conn(0x7f2d48072c50 msgr2=0x7f2d48073060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.363+0000 7f2d47400640 1 --2- 172.21.15.40:0/1980258512 >> v2:172.21.15.103:3300/0 conn(0x7f2d48072c50 0x7f2d48073060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.363+0000 7f2d47400640 1 -- 172.21.15.40:0/1980258512 >> v1:172.21.15.40:6789/0 conn(0x7f2d48156460 legacy=0x7f2d480cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.363+0000 7f2d47400640 1 -- 172.21.15.40:0/1980258512 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d480cc4e0 con 0x7f2d4806f790 2024-04-26T23:07:24.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.364+0000 7f2d47400640 1 --2- 172.21.15.40:0/1980258512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d4806f790 0x7f2d4806fc40 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f2d480cb1c0 tx=0x7f2d3002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b1ebb6b50caef84 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.365+0000 7f2d46a00640 1 -- 172.21.15.40:0/1980258512 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d30007740 con 0x7f2d4806f790 2024-04-26T23:07:24.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.365+0000 7f2d46a00640 1 -- 172.21.15.40:0/1980258512 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d300078e0 con 0x7f2d4806f790 2024-04-26T23:07:24.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.365+0000 7f2d46a00640 1 -- 172.21.15.40:0/1980258512 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d30007cb0 con 0x7f2d4806f790 2024-04-26T23:07:24.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.365+0000 7f2d4ea00640 1 -- 172.21.15.40:0/1980258512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d4806f790 msgr2=0x7f2d4806fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.365+0000 7f2d4ea00640 1 --2- 172.21.15.40:0/1980258512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d4806f790 0x7f2d4806fc40 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f2d480cb1c0 tx=0x7f2d3002a680 comp rx=0 tx=0).stop 2024-04-26T23:07:24.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.365+0000 7f2d4ea00640 1 -- 172.21.15.40:0/1980258512 shutdown_connections 2024-04-26T23:07:24.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.365+0000 7f2d4ea00640 1 --2- 172.21.15.40:0/1980258512 >> v2:172.21.15.103:3300/0 conn(0x7f2d48072c50 0x7f2d48073060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.365+0000 7f2d4ea00640 1 --2- 172.21.15.40:0/1980258512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d4806f790 0x7f2d4806fc40 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.365+0000 7f2d4ea00640 1 -- 172.21.15.40:0/1980258512 >> 172.21.15.40:0/1980258512 conn(0x7f2d480d0be0 msgr2=0x7f2d480d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.366+0000 7f2d4ea00640 1 -- 172.21.15.40:0/1980258512 shutdown_connections 2024-04-26T23:07:24.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.366+0000 7f2d4ea00640 1 -- 172.21.15.40:0/1980258512 wait complete. 2024-04-26T23:07:24.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.367+0000 7f2d4ea00640 1 Processor -- start 2024-04-26T23:07:24.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.368+0000 7f2d4ea00640 1 -- start start 2024-04-26T23:07:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.368+0000 7f2d4ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d4806f790 0x7f2d481441e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.368+0000 7f2d4ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d48072c50 0x7f2d480b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d4ea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d48156460 0x7f2d480b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d4ea00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2d480cbbe0 con 0x7f2d4806f790 2024-04-26T23:07:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d4ea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d48072700 con 0x7f2d48072c50 2024-04-26T23:07:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d4ea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2d4806e260 con 0x7f2d48156460 2024-04-26T23:07:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d47e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d4806f790 0x7f2d481441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d47e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d4806f790 0x7f2d481441e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37804/0 (socket says 172.21.15.40:37804) 2024-04-26T23:07:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d47e00640 1 -- 172.21.15.40:0/1577624129 learned_addr learned my addr 172.21.15.40:0/1577624129 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d47400640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d48072c50 0x7f2d480b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d47e00640 1 -- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d48156460 msgr2=0x7f2d480b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d4ca00640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d48156460 0x7f2d480b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d47e00640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d48156460 0x7f2d480b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d47e00640 1 -- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d48072c50 msgr2=0x7f2d480b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d47e00640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d48072c50 0x7f2d480b3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.369+0000 7f2d47e00640 1 -- 172.21.15.40:0/1577624129 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d48143140 con 0x7f2d4806f790 2024-04-26T23:07:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.370+0000 7f2d47e00640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d4806f790 0x7f2d481441e0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f2d3800c390 tx=0x7f2d3800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.370+0000 7f2d3fe00640 1 -- 172.21.15.40:0/1577624129 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d38017070 con 0x7f2d4806f790 2024-04-26T23:07:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.370+0000 7f2d3fe00640 1 -- 172.21.15.40:0/1577624129 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d38005ce0 con 0x7f2d4806f790 2024-04-26T23:07:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.370+0000 7f2d3fe00640 1 -- 172.21.15.40:0/1577624129 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d380072f0 con 0x7f2d4806f790 2024-04-26T23:07:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.370+0000 7f2d4ea00640 1 -- 172.21.15.40:0/1577624129 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d48143420 con 0x7f2d4806f790 2024-04-26T23:07:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.370+0000 7f2d4ea00640 1 -- 172.21.15.40:0/1577624129 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d481439e0 con 0x7f2d4806f790 2024-04-26T23:07:24.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.372+0000 7f2d3fe00640 1 -- 172.21.15.40:0/1577624129 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d3801a070 con 0x7f2d4806f790 2024-04-26T23:07:24.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.374+0000 7f2d3fe00640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d2406c8f0 0x7f2d2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.374+0000 7f2d51871740 1 Processor -- start 2024-04-26T23:07:24.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.374+0000 7f2d3fe00640 1 -- 172.21.15.40:0/1577624129 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f2d3808b110 con 0x7f2d4806f790 2024-04-26T23:07:24.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.374+0000 7f2d51871740 1 -- start start 2024-04-26T23:07:24.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.375+0000 7f2d47400640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d2406c8f0 0x7f2d2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.375+0000 7f2d51871740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f82e540 0x558d2f82e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.375+0000 7f2d51871740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f84bde0 0x558d2f84c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.375+0000 7f2d51871740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f804220 0x558d2f84a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.375+0000 7f2d51871740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558d2f714030 con 0x558d2f804220 2024-04-26T23:07:24.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.375+0000 7f2d51871740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558d2f72f7f0 con 0x558d2f82e540 2024-04-26T23:07:24.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.375+0000 7f2d51871740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558d2f6ea7e0 con 0x558d2f84bde0 2024-04-26T23:07:24.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.375+0000 7f2d4ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f804220 0x558d2f84a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.376+0000 7f2d4ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f804220 0x558d2f84a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37806/0 (socket says 172.21.15.40:37806) 2024-04-26T23:07:24.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.376+0000 7f2d4ca00640 1 -- 172.21.15.40:0/2941929398 learned_addr learned my addr 172.21.15.40:0/2941929398 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.376+0000 7f2d47e00640 1 --2- 172.21.15.40:0/2941929398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f82e540 0x558d2f82e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.376+0000 7f2d47400640 1 --2- 172.21.15.40:0/2941929398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f84bde0 0x558d2f84c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.376+0000 7f2d4ca00640 1 -- 172.21.15.40:0/2941929398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f84bde0 msgr2=0x558d2f84c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.376+0000 7f2d4ca00640 1 --2- 172.21.15.40:0/2941929398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f84bde0 0x558d2f84c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.376+0000 7f2d4ca00640 1 -- 172.21.15.40:0/2941929398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f82e540 msgr2=0x558d2f82e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.376+0000 7f2d4ca00640 1 --2- 172.21.15.40:0/2941929398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f82e540 0x558d2f82e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.376+0000 7f2d4ca00640 1 -- 172.21.15.40:0/2941929398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d2f6f6050 con 0x558d2f804220 2024-04-26T23:07:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.376+0000 7f2d47400640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d2406c8f0 0x7f2d2406edb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2d30034000 tx=0x7f2d3002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.376+0000 7f2d4ca00640 1 --2- 172.21.15.40:0/2941929398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f804220 0x558d2f84a4a0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f2d400098e0 tx=0x7f2d40009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c81ca24081ce87 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.377+0000 7f2d37e00640 1 -- 172.21.15.40:0/2941929398 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d40017020 con 0x558d2f804220 2024-04-26T23:07:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.377+0000 7f2d37e00640 1 -- 172.21.15.40:0/2941929398 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d4000ea10 con 0x558d2f804220 2024-04-26T23:07:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.377+0000 7f2d37e00640 1 -- 172.21.15.40:0/2941929398 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4000f040 con 0x558d2f804220 2024-04-26T23:07:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.377+0000 7f2d51871740 1 -- 172.21.15.40:0/2941929398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f804220 msgr2=0x558d2f84a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.377+0000 7f2d51871740 1 --2- 172.21.15.40:0/2941929398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f804220 0x558d2f84a4a0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f2d400098e0 tx=0x7f2d40009da0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.377+0000 7f2d51871740 1 -- 172.21.15.40:0/2941929398 shutdown_connections 2024-04-26T23:07:24.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.377+0000 7f2d51871740 1 --2- 172.21.15.40:0/2941929398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f804220 0x558d2f84a4a0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.377+0000 7f2d51871740 1 --2- 172.21.15.40:0/2941929398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f84bde0 0x558d2f84c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.377+0000 7f2d51871740 1 --2- 172.21.15.40:0/2941929398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f82e540 0x558d2f82e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.377+0000 7f2d51871740 1 -- 172.21.15.40:0/2941929398 >> 172.21.15.40:0/2941929398 conn(0x558d2f800b50 msgr2=0x558d2f849350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.378+0000 7f2d51871740 1 -- 172.21.15.40:0/2941929398 shutdown_connections 2024-04-26T23:07:24.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.378+0000 7f2d51871740 1 -- 172.21.15.40:0/2941929398 wait complete. 2024-04-26T23:07:24.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.378+0000 7f2d51871740 1 -- start start 2024-04-26T23:07:24.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.379+0000 7f2d51871740 1 Processor -- start 2024-04-26T23:07:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d51871740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 0x558d2f826e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d51871740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f84bde0 0x558d2f831070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d51871740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f859680 0x558d2f7fd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d51871740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558d2f76c200 con 0x558d2f82e540 2024-04-26T23:07:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d51871740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558d2f766ee0 con 0x558d2f859680 2024-04-26T23:07:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d51871740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558d2f773e70 con 0x558d2f84bde0 2024-04-26T23:07:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d51871740 10 client.0 _ll_register_callbacks cb 0x558d2f75d270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d47e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 0x558d2f826e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d47e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 0x558d2f826e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37810/0 (socket says 172.21.15.40:37810) 2024-04-26T23:07:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d47e00640 1 -- 172.21.15.40:0/1301348396 learned_addr learned my addr 172.21.15.40:0/1301348396 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d47400640 1 --2- 172.21.15.40:0/1301348396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f84bde0 0x558d2f831070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d47e00640 1 -- 172.21.15.40:0/1301348396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f84bde0 msgr2=0x558d2f831070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d47e00640 1 --2- 172.21.15.40:0/1301348396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f84bde0 0x558d2f831070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d47e00640 1 -- 172.21.15.40:0/1301348396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f859680 msgr2=0x558d2f7fd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d4ca00640 1 --2- 172.21.15.40:0/1301348396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f859680 0x558d2f7fd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d47e00640 1 --2- 172.21.15.40:0/1301348396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f859680 0x558d2f7fd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.380+0000 7f2d47e00640 1 -- 172.21.15.40:0/1301348396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d2f5437d0 con 0x558d2f82e540 2024-04-26T23:07:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.381+0000 7f2d47e00640 1 --2- 172.21.15.40:0/1301348396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 0x558d2f826e60 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f2d380894a0 tx=0x7f2d380594e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377b7f61bbe2b48a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.381+0000 7f2d34c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.381+0000 7f2d34c00640 1 -- 172.21.15.40:0/1301348396 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d38056fa0 con 0x558d2f82e540 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.381+0000 7f2d34c00640 1 -- 172.21.15.40:0/1301348396 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f2d38007490 con 0x558d2f82e540 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.381+0000 7f2d51871740 1 -- 172.21.15.40:0/1301348396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d4000ea10 con 0x558d2f82e540 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.381+0000 7f2d34c00640 1 -- 172.21.15.40:0/1301348396 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f2d38087050 con 0x558d2f82e540 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.381+0000 7f2d51871740 10 client.7722 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.381+0000 7f2d51871740 1 -- 172.21.15.40:0/1301348396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x558d2f4793f0 con 0x558d2f82e540 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.382+0000 7f2d34c00640 1 -- 172.21.15.40:0/1301348396 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d4000ea10 con 0x558d2f82e540 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.382+0000 7f2d34c00640 1 -- 172.21.15.40:0/1301348396 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7f2d380580f0 con 0x558d2f82e540 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.382+0000 7f2d51871740 10 client.7722 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.382+0000 7f2d51871740 10 client.7722 resolve_mds: resolved 5:1 to role '5:1' aka daemon mds.a 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.382+0000 7f2d51871740 20 client.7722 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.382+0000 7f2d51871740 1 --2- 172.21.15.40:0/1301348396 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558d2f8315b0 0x558d2f830210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.382+0000 7f2d51871740 4 client.7722 mds_command: new command op to 7443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:07:24.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.382+0000 7f2d51871740 1 -- 172.21.15.40:0/1301348396 --> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d2f7757e0 con 0x558d2f8315b0 2024-04-26T23:07:24.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.382+0000 7f2d47400640 1 --2- 172.21.15.40:0/1301348396 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558d2f8315b0 0x558d2f830210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.383+0000 7f2d47400640 1 --2- 172.21.15.40:0/1301348396 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558d2f8315b0 0x558d2f830210 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9a35274ac67b2667 server_cookie=a8e3b7b3d8d6aed0 in_seq=0 out_seq=0 2024-04-26T23:07:24.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.383+0000 7f2d34c00640 10 client.7722 ms_handle_connect on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.383+0000 7f2d34c00640 1 -- 172.21.15.40:0/1301348396 <== mds.1 v2:172.21.15.40:6832/216893598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d2f7757e0 con 0x558d2f8315b0 2024-04-26T23:07:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.383+0000 7f2d34c00640 10 client.7722 handle_command_reply: tid=0 2024-04-26T23:07:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.383+0000 7f2d51871740 1 client.7722 shutdown 2024-04-26T23:07:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.384+0000 7f2d51871740 1 -- 172.21.15.40:0/1301348396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 msgr2=0x558d2f826e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.384+0000 7f2d51871740 1 --2- 172.21.15.40:0/1301348396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 0x558d2f826e60 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f2d380894a0 tx=0x7f2d380594e0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.384+0000 7f2d51871740 1 -- 172.21.15.40:0/1301348396 shutdown_connections 2024-04-26T23:07:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.384+0000 7f2d51871740 1 --2- 172.21.15.40:0/1301348396 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558d2f8315b0 0x558d2f830210 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.384+0000 7f2d51871740 1 --2- 172.21.15.40:0/1301348396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f859680 0x558d2f7fd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.384+0000 7f2d51871740 1 --2- 172.21.15.40:0/1301348396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f84bde0 0x558d2f831070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.384+0000 7f2d51871740 1 --2- 172.21.15.40:0/1301348396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 0x558d2f826e60 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.384+0000 7f2d34c00640 0 client.7722 ms_handle_reset on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.384+0000 7f2d51871740 1 -- 172.21.15.40:0/1301348396 >> 172.21.15.40:0/1301348396 conn(0x558d2f804220 msgr2=0x558d2f8020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.384+0000 7f2d51871740 1 -- 172.21.15.40:0/1301348396 shutdown_connections 2024-04-26T23:07:24.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.384+0000 7f2d51871740 1 -- 172.21.15.40:0/1301348396 wait complete. 2024-04-26T23:07:24.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.384+0000 7f2d51871740 20 client.7722 trim_cache size 0 max 16384 2024-04-26T23:07:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.401+0000 7f2d51871740 1 Processor -- start 2024-04-26T23:07:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.401+0000 7f2d51871740 1 -- start start 2024-04-26T23:07:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d51871740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 0x558d2f82a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d51871740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f84bde0 0x558d2f82a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d51871740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f86d3a0 0x558d2f849350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d51871740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2d48201be0 con 0x558d2f82e540 2024-04-26T23:07:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d51871740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d48200dd0 con 0x558d2f84bde0 2024-04-26T23:07:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d51871740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2d482014b0 con 0x558d2f86d3a0 2024-04-26T23:07:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d47e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 0x558d2f82a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d47e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 0x558d2f82a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37818/0 (socket says 172.21.15.40:37818) 2024-04-26T23:07:24.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d47e00640 1 -- 172.21.15.40:0/2982717963 learned_addr learned my addr 172.21.15.40:0/2982717963 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d47e00640 1 -- 172.21.15.40:0/2982717963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f86d3a0 msgr2=0x558d2f849350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d47e00640 1 --2- 172.21.15.40:0/2982717963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f86d3a0 0x558d2f849350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d47e00640 1 -- 172.21.15.40:0/2982717963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f84bde0 msgr2=0x558d2f82a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d47e00640 1 --2- 172.21.15.40:0/2982717963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f84bde0 0x558d2f82a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.402+0000 7f2d47e00640 1 -- 172.21.15.40:0/2982717963 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d2f7fdb20 con 0x558d2f82e540 2024-04-26T23:07:24.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.403+0000 7f2d47e00640 1 --2- 172.21.15.40:0/2982717963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 0x558d2f82a050 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f2d38059fd0 tx=0x7f2d3805a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f6dcfd3c717f12c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.403+0000 7f2d37400640 1 -- 172.21.15.40:0/2982717963 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3805d050 con 0x558d2f82e540 2024-04-26T23:07:24.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.403+0000 7f2d37400640 1 -- 172.21.15.40:0/2982717963 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d380711b0 con 0x558d2f82e540 2024-04-26T23:07:24.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.403+0000 7f2d37400640 1 -- 172.21.15.40:0/2982717963 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d380580f0 con 0x558d2f82e540 2024-04-26T23:07:24.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.404+0000 7f2d51871740 1 -- 172.21.15.40:0/2982717963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 msgr2=0x558d2f82a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.404+0000 7f2d51871740 1 --2- 172.21.15.40:0/2982717963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 0x558d2f82a050 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f2d38059fd0 tx=0x7f2d3805a000 comp rx=0 tx=0).stop 2024-04-26T23:07:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.404+0000 7f2d51871740 1 -- 172.21.15.40:0/2982717963 shutdown_connections 2024-04-26T23:07:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.404+0000 7f2d51871740 1 --2- 172.21.15.40:0/2982717963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f86d3a0 0x558d2f849350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.404+0000 7f2d51871740 1 --2- 172.21.15.40:0/2982717963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f84bde0 0x558d2f82a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.404+0000 7f2d51871740 1 --2- 172.21.15.40:0/2982717963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f82e540 0x558d2f82a050 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.404+0000 7f2d51871740 1 -- 172.21.15.40:0/2982717963 >> 172.21.15.40:0/2982717963 conn(0x558d2f804220 msgr2=0x558d2f826220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.404+0000 7f2d51871740 1 -- 172.21.15.40:0/2982717963 shutdown_connections 2024-04-26T23:07:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.404+0000 7f2d51871740 1 -- 172.21.15.40:0/2982717963 wait complete. 2024-04-26T23:07:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.404+0000 7f2d51871740 1 -- start start 2024-04-26T23:07:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.405+0000 7f2d51871740 1 Processor -- start 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d51871740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f82e540 0x558d2f85bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d51871740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f84bde0 0x558d2f85c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d51871740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f86d3a0 0x558d2f8c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d51871740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558d2f774fd0 con 0x558d2f84bde0 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d51871740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d48141e70 con 0x558d2f86d3a0 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d51871740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558d2f831320 con 0x558d2f82e540 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d51871740 10 client.0 _ll_register_callbacks cb 0x7f2d38054f40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d47400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f84bde0 0x558d2f85c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d47400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f84bde0 0x558d2f85c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37834/0 (socket says 172.21.15.40:37834) 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d47400640 1 -- 172.21.15.40:0/1918722906 learned_addr learned my addr 172.21.15.40:0/1918722906 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d4ca00640 1 --2- 172.21.15.40:0/1918722906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f86d3a0 0x558d2f8c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d47e00640 1 --2- 172.21.15.40:0/1918722906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f82e540 0x558d2f85bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d47400640 1 -- 172.21.15.40:0/1918722906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f82e540 msgr2=0x558d2f85bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d47400640 1 --2- 172.21.15.40:0/1918722906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f82e540 0x558d2f85bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d47400640 1 -- 172.21.15.40:0/1918722906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f86d3a0 msgr2=0x558d2f8c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.406+0000 7f2d47400640 1 --2- 172.21.15.40:0/1918722906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f86d3a0 0x558d2f8c2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.407+0000 7f2d47400640 1 -- 172.21.15.40:0/1918722906 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d2f4390f0 con 0x558d2f84bde0 2024-04-26T23:07:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.407+0000 7f2d47400640 1 --2- 172.21.15.40:0/1918722906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f84bde0 0x558d2f85c4f0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f2d30009e50 tx=0x7f2d30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209a70fdf4770684 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.407+0000 7f2d34c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.407+0000 7f2d34c00640 1 -- 172.21.15.40:0/1918722906 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d300505e0 con 0x558d2f84bde0 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.407+0000 7f2d34c00640 1 -- 172.21.15.40:0/1918722906 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f2d30050780 con 0x558d2f84bde0 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.407+0000 7f2d51871740 1 -- 172.21.15.40:0/1918722906 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d380711b0 con 0x558d2f84bde0 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.408+0000 7f2d34c00640 1 -- 172.21.15.40:0/1918722906 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f2d3004b440 con 0x558d2f84bde0 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.408+0000 7f2d51871740 10 client.7728 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.408+0000 7f2d51871740 1 -- 172.21.15.40:0/1918722906 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x558d2f72fa30 con 0x558d2f84bde0 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.408+0000 7f2d34c00640 1 -- 172.21.15.40:0/1918722906 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d380711b0 con 0x558d2f84bde0 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.408+0000 7f2d34c00640 1 -- 172.21.15.40:0/1918722906 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7f2d30049040 con 0x558d2f84bde0 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.408+0000 7f2d51871740 10 client.7728 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.408+0000 7f2d51871740 10 client.7728 resolve_mds: resolved 5:1 to role '5:1' aka daemon mds.a 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.408+0000 7f2d51871740 20 client.7728 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.408+0000 7f2d51871740 1 --2- 172.21.15.40:0/1918722906 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558d2f8402a0 0x558d2f8c3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.408+0000 7f2d51871740 4 client.7728 mds_command: new command op to 7443 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:07:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.408+0000 7f2d51871740 1 -- 172.21.15.40:0/1918722906 --> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558d2f76f650 con 0x558d2f8402a0 2024-04-26T23:07:24.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.409+0000 7f2d47e00640 1 --2- 172.21.15.40:0/1918722906 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558d2f8402a0 0x558d2f8c3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.409+0000 7f2d47e00640 1 --2- 172.21.15.40:0/1918722906 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558d2f8402a0 0x558d2f8c3520 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ffbf7ce2ea461b16 server_cookie=ce118a900afd217d in_seq=0 out_seq=0 2024-04-26T23:07:24.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.409+0000 7f2d34c00640 10 client.7728 ms_handle_connect on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:24.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d34c00640 1 -- 172.21.15.40:0/1918722906 <== mds.1 v2:172.21.15.40:6832/216893598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558d2f76f650 con 0x558d2f8402a0 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d34c00640 10 client.7728 handle_command_reply: tid=0 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d51871740 1 client.7728 shutdown 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d51871740 1 -- 172.21.15.40:0/1918722906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f84bde0 msgr2=0x558d2f85c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d51871740 1 --2- 172.21.15.40:0/1918722906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f84bde0 0x558d2f85c4f0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f2d30009e50 tx=0x7f2d30004a40 comp rx=0 tx=0).stop 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d51871740 1 -- 172.21.15.40:0/1918722906 shutdown_connections 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d51871740 1 --2- 172.21.15.40:0/1918722906 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558d2f8402a0 0x558d2f8c3520 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d51871740 1 --2- 172.21.15.40:0/1918722906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d2f86d3a0 0x558d2f8c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d51871740 1 --2- 172.21.15.40:0/1918722906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2f84bde0 0x558d2f85c4f0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d51871740 1 --2- 172.21.15.40:0/1918722906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558d2f82e540 0x558d2f85bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d51871740 1 -- 172.21.15.40:0/1918722906 >> 172.21.15.40:0/1918722906 conn(0x558d2f804220 msgr2=0x558d2f826220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d34c00640 0 client.7728 ms_handle_reset on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d51871740 1 -- 172.21.15.40:0/1918722906 shutdown_connections 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.432+0000 7f2d51871740 1 -- 172.21.15.40:0/1918722906 wait complete. 2024-04-26T23:07:24.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.433+0000 7f2d51871740 20 client.7728 trim_cache size 0 max 16384 2024-04-26T23:07:24.434 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:07:24.434 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:07:24.434 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:07:24.434 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:07:24.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.433+0000 7f2d4ea00640 1 -- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d2406c8f0 msgr2=0x7f2d2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.433+0000 7f2d4ea00640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d2406c8f0 0x7f2d2406edb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2d30034000 tx=0x7f2d3002e000 comp rx=0 tx=0).stop 2024-04-26T23:07:24.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.433+0000 7f2d4ea00640 1 -- 172.21.15.40:0/1577624129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d4806f790 msgr2=0x7f2d481441e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.434+0000 7f2d4ea00640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d4806f790 0x7f2d481441e0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f2d3800c390 tx=0x7f2d3800c850 comp rx=0 tx=0).stop 2024-04-26T23:07:24.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.434+0000 7f2d4ea00640 1 -- 172.21.15.40:0/1577624129 shutdown_connections 2024-04-26T23:07:24.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.434+0000 7f2d4ea00640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d2406c8f0 0x7f2d2406edb0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.434+0000 7f2d4ea00640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d48156460 0x7f2d480b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.434+0000 7f2d4ea00640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d48072c50 0x7f2d480b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.434+0000 7f2d4ea00640 1 --2- 172.21.15.40:0/1577624129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d4806f790 0x7f2d481441e0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.434+0000 7f2d4ea00640 1 -- 172.21.15.40:0/1577624129 >> 172.21.15.40:0/1577624129 conn(0x7f2d480d0be0 msgr2=0x7f2d480c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.434+0000 7f2d4ea00640 1 -- 172.21.15.40:0/1577624129 shutdown_connections 2024-04-26T23:07:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.434+0000 7f2d4ea00640 1 -- 172.21.15.40:0/1577624129 wait complete. 2024-04-26T23:07:24.454 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:2 flush journal 2024-04-26T23:07:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.657+0000 7fb502e00640 1 Processor -- start 2024-04-26T23:07:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.658+0000 7fb502e00640 1 -- start start 2024-04-26T23:07:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.659+0000 7fb502e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4fc073600 0x7fb4fc072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.659+0000 7fb502e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb4fc15fe80 0x7fb4fc0ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.659+0000 7fb502e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4fc1604e0 con 0x7fb4fc0739e0 2024-04-26T23:07:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.659+0000 7fb502e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb4fc073220 con 0x7fb4fc15fe80 2024-04-26T23:07:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.659+0000 7fb502e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb4fc072850 con 0x7fb4fc073600 2024-04-26T23:07:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.660+0000 7fb502400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb4fc15fe80 0x7fb4fc0ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.660+0000 7fb502400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb4fc15fe80 0x7fb4fc0ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37846/0 (socket says 172.21.15.40:37846) 2024-04-26T23:07:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.660+0000 7fb502400640 1 -- 172.21.15.40:0/3766558036 learned_addr learned my addr 172.21.15.40:0/3766558036 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.660+0000 7fb501a00640 1 --2- 172.21.15.40:0/3766558036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4fc073600 0x7fb4fc072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.660+0000 7fb502400640 1 -- 172.21.15.40:0/3766558036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4fc073600 msgr2=0x7fb4fc072ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.660+0000 7fb502400640 1 --2- 172.21.15.40:0/3766558036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4fc073600 0x7fb4fc072ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.660+0000 7fb502400640 1 -- 172.21.15.40:0/3766558036 >> v1:172.21.15.103:6789/0 conn(0x7fb4fc0739e0 legacy=0x7fb4fc15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.660+0000 7fb502400640 1 -- 172.21.15.40:0/3766558036 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4fc072200 con 0x7fb4fc15fe80 2024-04-26T23:07:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.662+0000 7fb502400640 1 --2- 172.21.15.40:0/3766558036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc15fe80 0x7fb4fc0ae8c0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4009e50 tx=0x7fb4f402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ce026de6cd6e49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.662+0000 7fb4fbe00640 1 -- 172.21.15.40:0/3766558036 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4036070 con 0x7fb4fc15fe80 2024-04-26T23:07:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.662+0000 7fb4fbe00640 1 -- 172.21.15.40:0/3766558036 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f4031600 con 0x7fb4fc15fe80 2024-04-26T23:07:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.662+0000 7fb4fbe00640 1 -- 172.21.15.40:0/3766558036 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f40319d0 con 0x7fb4fc15fe80 2024-04-26T23:07:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.663+0000 7fb502e00640 1 -- 172.21.15.40:0/3766558036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc15fe80 msgr2=0x7fb4fc0ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.663+0000 7fb502e00640 1 --2- 172.21.15.40:0/3766558036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc15fe80 0x7fb4fc0ae8c0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4009e50 tx=0x7fb4f402a6c0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.663+0000 7fb502e00640 1 -- 172.21.15.40:0/3766558036 shutdown_connections 2024-04-26T23:07:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.663+0000 7fb502e00640 1 --2- 172.21.15.40:0/3766558036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc15fe80 0x7fb4fc0ae8c0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.663+0000 7fb502e00640 1 --2- 172.21.15.40:0/3766558036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4fc073600 0x7fb4fc072ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.663+0000 7fb502e00640 1 -- 172.21.15.40:0/3766558036 >> 172.21.15.40:0/3766558036 conn(0x7fb4fc0b2720 msgr2=0x7fb4fc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.663+0000 7fb502e00640 1 -- 172.21.15.40:0/3766558036 shutdown_connections 2024-04-26T23:07:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.663+0000 7fb502e00640 1 -- 172.21.15.40:0/3766558036 wait complete. 2024-04-26T23:07:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.665+0000 7fb502e00640 1 Processor -- start 2024-04-26T23:07:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.665+0000 7fb502e00640 1 -- start start 2024-04-26T23:07:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.666+0000 7fb502e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc073600 0x7fb4fc14d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.666+0000 7fb502e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4fc0739e0 0x7fb4fc1f5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.666+0000 7fb502e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4fc15fe80 0x7fb4fc0b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.666+0000 7fb502e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc071e20 con 0x7fb4fc073600 2024-04-26T23:07:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.666+0000 7fb502e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc073e60 con 0x7fb4fc15fe80 2024-04-26T23:07:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.666+0000 7fb502e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb4fc071fa0 con 0x7fb4fc0739e0 2024-04-26T23:07:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.666+0000 7fb501a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc073600 0x7fb4fc14d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.666+0000 7fb501a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc073600 0x7fb4fc14d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37854/0 (socket says 172.21.15.40:37854) 2024-04-26T23:07:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.666+0000 7fb501a00640 1 -- 172.21.15.40:0/860748520 learned_addr learned my addr 172.21.15.40:0/860748520 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.667+0000 7fb502400640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4fc15fe80 0x7fb4fc0b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.667+0000 7fb501000640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4fc0739e0 0x7fb4fc1f5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.667+0000 7fb501a00640 1 -- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4fc0739e0 msgr2=0x7fb4fc1f5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.667+0000 7fb501a00640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4fc0739e0 0x7fb4fc1f5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.667+0000 7fb501a00640 1 -- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4fc15fe80 msgr2=0x7fb4fc0b14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.667+0000 7fb501a00640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4fc15fe80 0x7fb4fc0b14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.667+0000 7fb501a00640 1 -- 172.21.15.40:0/860748520 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4fc1f6050 con 0x7fb4fc073600 2024-04-26T23:07:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.667+0000 7fb501a00640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc073600 0x7fb4fc14d640 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000d800 tx=0x7fb4f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.667+0000 7fb4f9600640 1 -- 172.21.15.40:0/860748520 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f000b6c0 con 0x7fb4fc073600 2024-04-26T23:07:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.667+0000 7fb4f9600640 1 -- 172.21.15.40:0/860748520 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f0019070 con 0x7fb4fc073600 2024-04-26T23:07:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.667+0000 7fb502e00640 1 -- 172.21.15.40:0/860748520 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4fc0af2c0 con 0x7fb4fc073600 2024-04-26T23:07:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.668+0000 7fb4f9600640 1 -- 172.21.15.40:0/860748520 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f000ba80 con 0x7fb4fc073600 2024-04-26T23:07:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.668+0000 7fb502e00640 1 -- 172.21.15.40:0/860748520 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4fc161540 con 0x7fb4fc073600 2024-04-26T23:07:24.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.670+0000 7fb4f9600640 1 -- 172.21.15.40:0/860748520 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb4f000bc20 con 0x7fb4fc073600 2024-04-26T23:07:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.671+0000 7fb4f9600640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4e006c8f0 0x7fb4e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.671+0000 7fb4f9600640 1 -- 172.21.15.40:0/860748520 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fb4f008c270 con 0x7fb4fc073600 2024-04-26T23:07:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.671+0000 7fb5071b3740 1 Processor -- start 2024-04-26T23:07:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.672+0000 7fb5071b3740 1 -- start start 2024-04-26T23:07:24.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.672+0000 7fb501000640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4e006c8f0 0x7fb4e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.672+0000 7fb5071b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 0x561dc02deaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.672+0000 7fb5071b3740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc02fbfb0 0x561dc02fc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.672+0000 7fb5071b3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02b43f0 0x561dc02fa670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.672+0000 7fb5071b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561dc01c4030 con 0x561dc02de710 2024-04-26T23:07:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.672+0000 7fb5071b3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561dc01df7f0 con 0x561dc02b43f0 2024-04-26T23:07:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.672+0000 7fb5071b3740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561dc019a7e0 con 0x561dc02fbfb0 2024-04-26T23:07:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.673+0000 7fb501a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 0x561dc02deaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.673+0000 7fb501a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 0x561dc02deaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37866/0 (socket says 172.21.15.40:37866) 2024-04-26T23:07:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.673+0000 7fb501a00640 1 -- 172.21.15.40:0/4026895890 learned_addr learned my addr 172.21.15.40:0/4026895890 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.673+0000 7fb502400640 1 --2- 172.21.15.40:0/4026895890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02b43f0 0x561dc02fa670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.673+0000 7fb501000640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4e006c8f0 0x7fb4e006edb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc161050 tx=0x7fb4ec009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.673+0000 7fb501a00640 1 -- 172.21.15.40:0/4026895890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc02fbfb0 msgr2=0x561dc02fc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.673+0000 7fb501a00640 1 --2- 172.21.15.40:0/4026895890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc02fbfb0 0x561dc02fc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.673+0000 7fb501a00640 1 -- 172.21.15.40:0/4026895890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02b43f0 msgr2=0x561dc02fa670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.673+0000 7fb501a00640 1 --2- 172.21.15.40:0/4026895890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02b43f0 0x561dc02fa670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.673+0000 7fb501a00640 1 -- 172.21.15.40:0/4026895890 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dc01a6050 con 0x561dc02de710 2024-04-26T23:07:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.674+0000 7fb501a00640 1 --2- 172.21.15.40:0/4026895890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 0x561dc02deaf0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0062420 tx=0x7fb4f0059800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb993c05a405395b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.674+0000 7fb4ea000640 1 -- 172.21.15.40:0/4026895890 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0072020 con 0x561dc02de710 2024-04-26T23:07:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.674+0000 7fb4ea000640 1 -- 172.21.15.40:0/4026895890 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f0059d70 con 0x561dc02de710 2024-04-26T23:07:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.674+0000 7fb4ea000640 1 -- 172.21.15.40:0/4026895890 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f005e120 con 0x561dc02de710 2024-04-26T23:07:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.674+0000 7fb5071b3740 1 -- 172.21.15.40:0/4026895890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 msgr2=0x561dc02deaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.674+0000 7fb5071b3740 1 --2- 172.21.15.40:0/4026895890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 0x561dc02deaf0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0062420 tx=0x7fb4f0059800 comp rx=0 tx=0).stop 2024-04-26T23:07:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.674+0000 7fb5071b3740 1 -- 172.21.15.40:0/4026895890 shutdown_connections 2024-04-26T23:07:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.674+0000 7fb5071b3740 1 --2- 172.21.15.40:0/4026895890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02b43f0 0x561dc02fa670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.674+0000 7fb5071b3740 1 --2- 172.21.15.40:0/4026895890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc02fbfb0 0x561dc02fc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.674+0000 7fb5071b3740 1 --2- 172.21.15.40:0/4026895890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 0x561dc02deaf0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.674+0000 7fb5071b3740 1 -- 172.21.15.40:0/4026895890 >> 172.21.15.40:0/4026895890 conn(0x561dc02b0d20 msgr2=0x561dc02f9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.675+0000 7fb5071b3740 1 -- 172.21.15.40:0/4026895890 shutdown_connections 2024-04-26T23:07:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.675+0000 7fb5071b3740 1 -- 172.21.15.40:0/4026895890 wait complete. 2024-04-26T23:07:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.675+0000 7fb5071b3740 1 -- start start 2024-04-26T23:07:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.676+0000 7fb5071b3740 1 Processor -- start 2024-04-26T23:07:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb5071b3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02de710 0x561dc02e2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb5071b3740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc02fbfb0 0x561dc02f9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb5071b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 0x561dc03096f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb5071b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561dc021c200 con 0x561dc0309270 2024-04-26T23:07:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb5071b3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561dc0216ee0 con 0x561dc02de710 2024-04-26T23:07:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb5071b3740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561dc0223e70 con 0x561dc02fbfb0 2024-04-26T23:07:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb5071b3740 10 client.0 _ll_register_callbacks cb 0x561dc020d270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb502400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 0x561dc03096f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb502400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 0x561dc03096f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37872/0 (socket says 172.21.15.40:37872) 2024-04-26T23:07:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb502400640 1 -- 172.21.15.40:0/2357177990 learned_addr learned my addr 172.21.15.40:0/2357177990 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb501a00640 1 --2- 172.21.15.40:0/2357177990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02de710 0x561dc02e2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb501000640 1 --2- 172.21.15.40:0/2357177990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc02fbfb0 0x561dc02f9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb502400640 1 -- 172.21.15.40:0/2357177990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc02fbfb0 msgr2=0x561dc02f9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb502400640 1 --2- 172.21.15.40:0/2357177990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc02fbfb0 0x561dc02f9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb502400640 1 -- 172.21.15.40:0/2357177990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02de710 msgr2=0x561dc02e2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb502400640 1 --2- 172.21.15.40:0/2357177990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02de710 0x561dc02e2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.677+0000 7fb502400640 1 -- 172.21.15.40:0/2357177990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dbfff37d0 con 0x561dc0309270 2024-04-26T23:07:24.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.678+0000 7fb502400640 1 --2- 172.21.15.40:0/2357177990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 0x561dc03096f0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4009e50 tx=0x7fb4f4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae53cb234555df0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.678+0000 7fb4dea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:24.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.678+0000 7fb4dea00640 1 -- 172.21.15.40:0/2357177990 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4036040 con 0x561dc0309270 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.678+0000 7fb4dea00640 1 -- 172.21.15.40:0/2357177990 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fb4f4031d60 con 0x561dc0309270 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.678+0000 7fb5071b3740 1 -- 172.21.15.40:0/2357177990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4f0059d70 con 0x561dc0309270 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.678+0000 7fb4dea00640 1 -- 172.21.15.40:0/2357177990 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fb4f404d050 con 0x561dc0309270 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.678+0000 7fb5071b3740 10 client.7740 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.678+0000 7fb5071b3740 1 -- 172.21.15.40:0/2357177990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x561dbff293f0 con 0x561dc0309270 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.679+0000 7fb4dea00640 1 -- 172.21.15.40:0/2357177990 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f0059d70 con 0x561dc0309270 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.679+0000 7fb4dea00640 1 -- 172.21.15.40:0/2357177990 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7fb4f4050020 con 0x561dc0309270 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.679+0000 7fb5071b3740 10 client.7740 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.679+0000 7fb5071b3740 10 client.7740 resolve_mds: resolved 5:2 to role '5:2' aka daemon mds.d 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.679+0000 7fb5071b3740 20 client.7740 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.679+0000 7fb5071b3740 1 --2- 172.21.15.40:0/2357177990 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x561dc02cee90 0x561dc02e0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.679+0000 7fb5071b3740 4 client.7740 mds_command: new command op to 5965 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.679+0000 7fb5071b3740 1 -- 172.21.15.40:0/2357177990 --> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561dc02257e0 con 0x561dc02cee90 2024-04-26T23:07:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.680+0000 7fb501a00640 1 --2- 172.21.15.40:0/2357177990 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x561dc02cee90 0x561dc02e0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.681+0000 7fb501a00640 1 --2- 172.21.15.40:0/2357177990 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x561dc02cee90 0x561dc02e0760 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1797e8960dee5736 server_cookie=378e1de54861d120 in_seq=0 out_seq=0 2024-04-26T23:07:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.681+0000 7fb4dea00640 10 client.7740 ms_handle_connect on v2:172.21.15.103:6834/2914603370 2024-04-26T23:07:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.682+0000 7fb4dea00640 1 -- 172.21.15.40:0/2357177990 <== mds.2 v2:172.21.15.103:6834/2914603370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561dc02257e0 con 0x561dc02cee90 2024-04-26T23:07:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.682+0000 7fb4dea00640 10 client.7740 handle_command_reply: tid=0 2024-04-26T23:07:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.683+0000 7fb5071b3740 1 client.7740 shutdown 2024-04-26T23:07:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.683+0000 7fb5071b3740 1 -- 172.21.15.40:0/2357177990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 msgr2=0x561dc03096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.683+0000 7fb5071b3740 1 --2- 172.21.15.40:0/2357177990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 0x561dc03096f0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4009e50 tx=0x7fb4f4004ab0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.683+0000 7fb5071b3740 1 -- 172.21.15.40:0/2357177990 shutdown_connections 2024-04-26T23:07:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.683+0000 7fb5071b3740 1 --2- 172.21.15.40:0/2357177990 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x561dc02cee90 0x561dc02e0760 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.683+0000 7fb5071b3740 1 --2- 172.21.15.40:0/2357177990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 0x561dc03096f0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.683+0000 7fb5071b3740 1 --2- 172.21.15.40:0/2357177990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc02fbfb0 0x561dc02f9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.683+0000 7fb5071b3740 1 --2- 172.21.15.40:0/2357177990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02de710 0x561dc02e2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.683+0000 7fb5071b3740 1 -- 172.21.15.40:0/2357177990 >> 172.21.15.40:0/2357177990 conn(0x561dc02b43f0 msgr2=0x561dc02fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.683+0000 7fb4dea00640 0 client.7740 ms_handle_reset on v2:172.21.15.103:6834/2914603370 2024-04-26T23:07:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.683+0000 7fb5071b3740 1 -- 172.21.15.40:0/2357177990 shutdown_connections 2024-04-26T23:07:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.683+0000 7fb5071b3740 1 -- 172.21.15.40:0/2357177990 wait complete. 2024-04-26T23:07:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.683+0000 7fb5071b3740 20 client.7740 trim_cache size 0 max 16384 2024-04-26T23:07:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.699+0000 7fb5071b3740 1 Processor -- start 2024-04-26T23:07:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.700+0000 7fb5071b3740 1 -- start start 2024-04-26T23:07:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.700+0000 7fb5071b3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02de710 0x561dc02aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.700+0000 7fb5071b3740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc02fbfb0 0x561dc02af120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.700+0000 7fb5071b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 0x561dc02f6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.700+0000 7fb5071b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc201d80 con 0x561dc0309270 2024-04-26T23:07:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.700+0000 7fb5071b3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc200f50 con 0x561dc02de710 2024-04-26T23:07:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.700+0000 7fb5071b3740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb4fc201650 con 0x561dc02fbfb0 2024-04-26T23:07:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.701+0000 7fb502400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 0x561dc02f6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.701+0000 7fb502400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 0x561dc02f6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37886/0 (socket says 172.21.15.40:37886) 2024-04-26T23:07:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.701+0000 7fb502400640 1 -- 172.21.15.40:0/2348875031 learned_addr learned my addr 172.21.15.40:0/2348875031 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.701+0000 7fb502400640 1 -- 172.21.15.40:0/2348875031 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc02fbfb0 msgr2=0x561dc02af120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.701+0000 7fb502400640 1 --2- 172.21.15.40:0/2348875031 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc02fbfb0 0x561dc02af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.701+0000 7fb501a00640 1 --2- 172.21.15.40:0/2348875031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02de710 0x561dc02aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.701+0000 7fb502400640 1 -- 172.21.15.40:0/2348875031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02de710 msgr2=0x561dc02aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.701+0000 7fb502400640 1 --2- 172.21.15.40:0/2348875031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02de710 0x561dc02aebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.701+0000 7fb502400640 1 -- 172.21.15.40:0/2348875031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dbfee90f0 con 0x561dc0309270 2024-04-26T23:07:24.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.701+0000 7fb501a00640 1 --2- 172.21.15.40:0/2348875031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02de710 0x561dc02aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:24.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.702+0000 7fb502400640 1 --2- 172.21.15.40:0/2348875031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 0x561dc02f6ac0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4002230 tx=0x7fb4f403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456672d27c603622 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.702+0000 7fb4e9600640 1 -- 172.21.15.40:0/2348875031 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4036040 con 0x561dc0309270 2024-04-26T23:07:24.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.702+0000 7fb4e9600640 1 -- 172.21.15.40:0/2348875031 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f4031d60 con 0x561dc0309270 2024-04-26T23:07:24.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.702+0000 7fb4e9600640 1 -- 172.21.15.40:0/2348875031 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4003b90 con 0x561dc0309270 2024-04-26T23:07:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.702+0000 7fb5071b3740 1 -- 172.21.15.40:0/2348875031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 msgr2=0x561dc02f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.702+0000 7fb5071b3740 1 --2- 172.21.15.40:0/2348875031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 0x561dc02f6ac0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4002230 tx=0x7fb4f403d000 comp rx=0 tx=0).stop 2024-04-26T23:07:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.703+0000 7fb5071b3740 1 -- 172.21.15.40:0/2348875031 shutdown_connections 2024-04-26T23:07:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.703+0000 7fb5071b3740 1 --2- 172.21.15.40:0/2348875031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc0309270 0x561dc02f6ac0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.703+0000 7fb5071b3740 1 --2- 172.21.15.40:0/2348875031 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc02fbfb0 0x561dc02af120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.703+0000 7fb5071b3740 1 --2- 172.21.15.40:0/2348875031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02de710 0x561dc02aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.703+0000 7fb5071b3740 1 -- 172.21.15.40:0/2348875031 >> 172.21.15.40:0/2348875031 conn(0x561dc02b43f0 msgr2=0x561dc02d94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.703+0000 7fb5071b3740 1 -- 172.21.15.40:0/2348875031 shutdown_connections 2024-04-26T23:07:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.703+0000 7fb5071b3740 1 -- 172.21.15.40:0/2348875031 wait complete. 2024-04-26T23:07:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.703+0000 7fb5071b3740 1 -- start start 2024-04-26T23:07:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.704+0000 7fb5071b3740 1 Processor -- start 2024-04-26T23:07:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb5071b3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 0x561dc0372ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb5071b3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02fbfb0 0x561dc02cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb5071b3740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc0309270 0x561dc02eeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb5071b3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561dc0224fd0 con 0x561dc02de710 2024-04-26T23:07:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb5071b3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc2001d0 con 0x561dc02fbfb0 2024-04-26T23:07:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb5071b3740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561dc02fb220 con 0x561dc0309270 2024-04-26T23:07:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb5071b3740 10 client.0 _ll_register_callbacks cb 0x7fb4f4034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb501a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 0x561dc0372ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb501a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 0x561dc0372ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37902/0 (socket says 172.21.15.40:37902) 2024-04-26T23:07:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb501a00640 1 -- 172.21.15.40:0/607662960 learned_addr learned my addr 172.21.15.40:0/607662960 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb501000640 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02fbfb0 0x561dc02cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb502400640 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc0309270 0x561dc02eeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb501a00640 1 -- 172.21.15.40:0/607662960 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc0309270 msgr2=0x561dc02eeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb501a00640 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc0309270 0x561dc02eeeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb501a00640 1 -- 172.21.15.40:0/607662960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02fbfb0 msgr2=0x561dc02cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb501a00640 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02fbfb0 0x561dc02cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb501a00640 1 -- 172.21.15.40:0/607662960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dc01dfa30 con 0x561dc02de710 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.705+0000 7fb501000640 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02fbfb0 0x561dc02cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb501a00640 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 0x561dc0372ec0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fb4f008a570 tx=0x7fb4f0020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321aba2d28ff8eb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb4dea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb4dea00640 1 -- 172.21.15.40:0/607662960 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0072040 con 0x561dc02de710 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb4dea00640 1 -- 172.21.15.40:0/607662960 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fb4f008a240 con 0x561dc02de710 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb5071b3740 1 -- 172.21.15.40:0/607662960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4f4031d60 con 0x561dc02de710 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb4dea00640 1 -- 172.21.15.40:0/607662960 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fb4f0059d70 con 0x561dc02de710 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb5071b3740 10 client.7746 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb5071b3740 1 -- 172.21.15.40:0/607662960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x561dc02ef7f0 con 0x561dc02de710 2024-04-26T23:07:24.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb4dea00640 1 -- 172.21.15.40:0/607662960 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f4031d60 con 0x561dc02de710 2024-04-26T23:07:24.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb4dea00640 1 -- 172.21.15.40:0/607662960 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7fb4f0013070 con 0x561dc02de710 2024-04-26T23:07:24.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb5071b3740 10 client.7746 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:24.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb5071b3740 10 client.7746 resolve_mds: resolved 5:2 to role '5:2' aka daemon mds.d 2024-04-26T23:07:24.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb5071b3740 20 client.7746 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:24.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.706+0000 7fb5071b3740 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x561dc0313410 0x561dc02b2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.707+0000 7fb5071b3740 4 client.7746 mds_command: new command op to 5965 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:07:24.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.707+0000 7fb5071b3740 1 -- 172.21.15.40:0/607662960 --> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561dc021f650 con 0x561dc0313410 2024-04-26T23:07:24.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.707+0000 7fb501000640 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x561dc0313410 0x561dc02b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.708+0000 7fb501000640 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x561dc0313410 0x561dc02b2130 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dee0fa1e0d095d6d server_cookie=cd4e04fd451e4b36 in_seq=0 out_seq=0 2024-04-26T23:07:24.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.708+0000 7fb4dea00640 10 client.7746 ms_handle_connect on v2:172.21.15.103:6834/2914603370 2024-04-26T23:07:24.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb4dea00640 1 -- 172.21.15.40:0/607662960 <== mds.2 v2:172.21.15.103:6834/2914603370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561dc021f650 con 0x561dc0313410 2024-04-26T23:07:24.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb4dea00640 10 client.7746 handle_command_reply: tid=0 2024-04-26T23:07:24.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb5071b3740 1 client.7746 shutdown 2024-04-26T23:07:24.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb5071b3740 1 -- 172.21.15.40:0/607662960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 msgr2=0x561dc0372ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb5071b3740 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 0x561dc0372ec0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fb4f008a570 tx=0x7fb4f0020070 comp rx=0 tx=0).stop 2024-04-26T23:07:24.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb5071b3740 1 -- 172.21.15.40:0/607662960 shutdown_connections 2024-04-26T23:07:24.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb5071b3740 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.103:6834/2914603370,v1:172.21.15.103:6836/2914603370] conn(0x561dc0313410 0x561dc02b2130 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb5071b3740 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561dc0309270 0x561dc02eeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb5071b3740 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561dc02fbfb0 0x561dc02cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb5071b3740 1 --2- 172.21.15.40:0/607662960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561dc02de710 0x561dc0372ec0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb4dea00640 0 client.7746 ms_handle_reset on v2:172.21.15.103:6834/2914603370 2024-04-26T23:07:24.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb5071b3740 1 -- 172.21.15.40:0/607662960 >> 172.21.15.40:0/607662960 conn(0x561dc02b43f0 msgr2=0x561dc02aebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb5071b3740 1 -- 172.21.15.40:0/607662960 shutdown_connections 2024-04-26T23:07:24.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.728+0000 7fb5071b3740 1 -- 172.21.15.40:0/607662960 wait complete. 2024-04-26T23:07:24.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.729+0000 7fb5071b3740 20 client.7746 trim_cache size 0 max 16384 2024-04-26T23:07:24.730 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:07:24.730 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:07:24.730 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:07:24.730 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:07:24.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.729+0000 7fb502e00640 1 -- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4e006c8f0 msgr2=0x7fb4e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.729+0000 7fb502e00640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4e006c8f0 0x7fb4e006edb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc161050 tx=0x7fb4ec009040 comp rx=0 tx=0).stop 2024-04-26T23:07:24.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.729+0000 7fb502e00640 1 -- 172.21.15.40:0/860748520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc073600 msgr2=0x7fb4fc14d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.729+0000 7fb502e00640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc073600 0x7fb4fc14d640 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000d800 tx=0x7fb4f000dcc0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.729+0000 7fb502e00640 1 -- 172.21.15.40:0/860748520 shutdown_connections 2024-04-26T23:07:24.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.729+0000 7fb502e00640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4e006c8f0 0x7fb4e006edb0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.729+0000 7fb502e00640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4fc15fe80 0x7fb4fc0b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.730+0000 7fb502e00640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4fc0739e0 0x7fb4fc1f5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.730+0000 7fb502e00640 1 --2- 172.21.15.40:0/860748520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4fc073600 0x7fb4fc14d640 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.730+0000 7fb502e00640 1 -- 172.21.15.40:0/860748520 >> 172.21.15.40:0/860748520 conn(0x7fb4fc0b2720 msgr2=0x7fb4fc15e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.730+0000 7fb502e00640 1 -- 172.21.15.40:0/860748520 shutdown_connections 2024-04-26T23:07:24.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.730+0000 7fb502e00640 1 -- 172.21.15.40:0/860748520 wait complete. 2024-04-26T23:07:24.751 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:24.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.958+0000 7fcc10a00640 1 Processor -- start 2024-04-26T23:07:24.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.958+0000 7fcc10a00640 1 -- start start 2024-04-26T23:07:24.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.959+0000 7fcc10a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc0c162dc0 0x7fcc0c1631a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.959+0000 7fcc10a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcc0c0d0fb0 0x7fcc0c0d1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.959+0000 7fcc10a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc0c1636e0 con 0x7fcc0c0cf2d0 2024-04-26T23:07:24.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.959+0000 7fcc10a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fcc0c0cf7d0 con 0x7fcc0c162dc0 2024-04-26T23:07:24.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.959+0000 7fcc10a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcc0c0cfab0 con 0x7fcc0c0d0fb0 2024-04-26T23:07:24.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.960+0000 7fcc0b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc0c162dc0 0x7fcc0c1631a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.960+0000 7fcc0aa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcc0c0cf2d0 0x7fcc0c06d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34222/0 (socket says 172.21.15.40:34222) 2024-04-26T23:07:24.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.960+0000 7fcc0aa00640 1 -- 172.21.15.40:0/3486458642 learned_addr learned my addr 172.21.15.40:0/3486458642 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:24.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.960+0000 7fcc0be00640 1 --2- 172.21.15.40:0/3486458642 >> v2:172.21.15.103:3300/0 conn(0x7fcc0c0d0fb0 0x7fcc0c0d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.960+0000 7fcc0b400640 1 -- 172.21.15.40:0/3486458642 >> v2:172.21.15.103:3300/0 conn(0x7fcc0c0d0fb0 msgr2=0x7fcc0c0d1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:24.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.960+0000 7fcc0b400640 1 --2- 172.21.15.40:0/3486458642 >> v2:172.21.15.103:3300/0 conn(0x7fcc0c0d0fb0 0x7fcc0c0d1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.960+0000 7fcc0b400640 1 -- 172.21.15.40:0/3486458642 >> v1:172.21.15.40:6789/0 conn(0x7fcc0c0cf2d0 legacy=0x7fcc0c06d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.961+0000 7fcc0b400640 1 -- 172.21.15.40:0/3486458642 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0c0cfd90 con 0x7fcc0c162dc0 2024-04-26T23:07:24.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.960+0000 7fcc0a000640 1 -- 172.21.15.40:0/3486458642 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1520060127 0 0) 0x7fcc0c1636e0 con 0x7fcc0c0cf2d0 2024-04-26T23:07:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.962+0000 7fcc0b400640 1 --2- 172.21.15.40:0/3486458642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc0c162dc0 0x7fcc0c1631a0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0b8b00 tx=0x7fcbfc02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a145d6755d63d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.962+0000 7fcc0a000640 1 -- 172.21.15.40:0/3486458642 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc007740 con 0x7fcc0c162dc0 2024-04-26T23:07:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.962+0000 7fcc0a000640 1 -- 172.21.15.40:0/3486458642 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbfc0078e0 con 0x7fcc0c162dc0 2024-04-26T23:07:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.962+0000 7fcc0a000640 1 -- 172.21.15.40:0/3486458642 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc007cb0 con 0x7fcc0c162dc0 2024-04-26T23:07:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.962+0000 7fcc10a00640 1 -- 172.21.15.40:0/3486458642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc0c162dc0 msgr2=0x7fcc0c1631a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.962+0000 7fcc10a00640 1 --2- 172.21.15.40:0/3486458642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc0c162dc0 0x7fcc0c1631a0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0b8b00 tx=0x7fcbfc02a5f0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.963+0000 7fcc10a00640 1 -- 172.21.15.40:0/3486458642 shutdown_connections 2024-04-26T23:07:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.963+0000 7fcc10a00640 1 --2- 172.21.15.40:0/3486458642 >> v2:172.21.15.103:3300/0 conn(0x7fcc0c0d0fb0 0x7fcc0c0d1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.963+0000 7fcc10a00640 1 --2- 172.21.15.40:0/3486458642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc0c162dc0 0x7fcc0c1631a0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.963+0000 7fcc10a00640 1 -- 172.21.15.40:0/3486458642 >> 172.21.15.40:0/3486458642 conn(0x7fcc0c0d2bd0 msgr2=0x7fcc0c0d2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.963+0000 7fcc10a00640 1 -- 172.21.15.40:0/3486458642 shutdown_connections 2024-04-26T23:07:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.963+0000 7fcc10a00640 1 -- 172.21.15.40:0/3486458642 wait complete. 2024-04-26T23:07:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.964+0000 7fcc10a00640 1 Processor -- start 2024-04-26T23:07:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.965+0000 7fcc10a00640 1 -- start start 2024-04-26T23:07:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.965+0000 7fcc10a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc0c0cf2d0 0x7fcc0c0b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.965+0000 7fcc10a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc0c0d0fb0 0x7fcc0c0b2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.965+0000 7fcc10a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc0c162dc0 0x7fcc0c156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.966+0000 7fcc10a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c0cef90 con 0x7fcc0c0cf2d0 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.966+0000 7fcc10a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c0d0350 con 0x7fcc0c0d0fb0 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.966+0000 7fcc10a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcc0c0cf110 con 0x7fcc0c162dc0 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.966+0000 7fcc0b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc0c0cf2d0 0x7fcc0c0b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.966+0000 7fcc0b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc0c0cf2d0 0x7fcc0c0b7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41182/0 (socket says 172.21.15.40:41182) 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.966+0000 7fcc0b400640 1 -- 172.21.15.40:0/2554825833 learned_addr learned my addr 172.21.15.40:0/2554825833 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.966+0000 7fcc0aa00640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc0c0d0fb0 0x7fcc0c0b2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.966+0000 7fcc0be00640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc0c162dc0 0x7fcc0c156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.966+0000 7fcc0b400640 1 -- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc0c162dc0 msgr2=0x7fcc0c156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.966+0000 7fcc0b400640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc0c162dc0 0x7fcc0c156920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.966+0000 7fcc0b400640 1 -- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc0c0d0fb0 msgr2=0x7fcc0c0b2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.966+0000 7fcc0b400640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc0c0d0fb0 0x7fcc0c0b2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:24.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.966+0000 7fcc0b400640 1 -- 172.21.15.40:0/2554825833 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0c0b2a70 con 0x7fcc0c0cf2d0 2024-04-26T23:07:24.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.967+0000 7fcc0b400640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc0c0cf2d0 0x7fcc0c0b7140 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc000f80 tx=0x7fcbfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.967+0000 7fcc03400640 1 -- 172.21.15.40:0/2554825833 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc03f070 con 0x7fcc0c0cf2d0 2024-04-26T23:07:24.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.967+0000 7fcc03400640 1 -- 172.21.15.40:0/2554825833 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbfc031c40 con 0x7fcc0c0cf2d0 2024-04-26T23:07:24.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.967+0000 7fcc03400640 1 -- 172.21.15.40:0/2554825833 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc003990 con 0x7fcc0c0cf2d0 2024-04-26T23:07:24.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.967+0000 7fcc10a00640 1 -- 172.21.15.40:0/2554825833 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0c1f50d0 con 0x7fcc0c0cf2d0 2024-04-26T23:07:24.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.967+0000 7fcc10a00640 1 -- 172.21.15.40:0/2554825833 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc0c1f5690 con 0x7fcc0c0cf2d0 2024-04-26T23:07:24.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.969+0000 7fcc10a00640 1 -- 172.21.15.40:0/2554825833 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc0c069a80 con 0x7fcc0c0cf2d0 2024-04-26T23:07:24.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.970+0000 7fcc03400640 1 -- 172.21.15.40:0/2554825833 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcbfc007790 con 0x7fcc0c0cf2d0 2024-04-26T23:07:24.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.970+0000 7fcc03400640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcbec06c8f0 0x7fcbec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:24.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.971+0000 7fcc03400640 1 -- 172.21.15.40:0/2554825833 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fcbfc0acfc0 con 0x7fcc0c0cf2d0 2024-04-26T23:07:24.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.971+0000 7fcc0aa00640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcbec06c8f0 0x7fcbec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:24.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.972+0000 7fcc0aa00640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcbec06c8f0 0x7fcbec06edb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0b1a30 tx=0x7fcbf8007b90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:24.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:24.975+0000 7fcc03400640 1 -- 172.21.15.40:0/2554825833 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fcbfc07b250 con 0x7fcc0c0cf2d0 2024-04-26T23:07:25.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.222+0000 7fcc10a00640 1 -- 172.21.15.40:0/2554825833 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc0c068e00 con 0x7fcc0c0cf2d0 2024-04-26T23:07:25.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.223+0000 7fcc03400640 1 -- 172.21.15.40:0/2554825833 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5225 (secure 0 0 0) 0x7fcbfc07ab50 con 0x7fcc0c0cf2d0 2024-04-26T23:07:25.224 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:25.224 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":81,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/755627718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":755627718},{"type":"v1","addr":"172.21.15.103:6837","nonce":755627718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:06:57.853769+0000","modified":"2024-04-26T23:07:21.981294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7446,"mds_1":7443,"mds_2":5965},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5965":{"gid":5965,"name":"d","rank":2,"incarnation":78,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6836/2914603370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2914603370},{"type":"v1","addr":"172.21.15.103:6836","nonce":2914603370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7443":{"gid":7443,"name":"a","rank":1,"incarnation":76,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/216893598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":216893598},{"type":"v1","addr":"172.21.15.40:6834","nonce":216893598}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7446":{"gid":7446,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4227275373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":4227275373},{"type":"v1","addr":"172.21.15.40:6835","nonce":4227275373}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7446,"qdb_cluster":[5965,7443,7446]},"id":5}]} 2024-04-26T23:07:25.224 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 81 2024-04-26T23:07:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.226+0000 7fcc10a00640 1 -- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcbec06c8f0 msgr2=0x7fcbec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.226+0000 7fcc10a00640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcbec06c8f0 0x7fcbec06edb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0b1a30 tx=0x7fcbf8007b90 comp rx=0 tx=0).stop 2024-04-26T23:07:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.226+0000 7fcc10a00640 1 -- 172.21.15.40:0/2554825833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc0c0cf2d0 msgr2=0x7fcc0c0b7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.226+0000 7fcc10a00640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc0c0cf2d0 0x7fcc0c0b7140 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc000f80 tx=0x7fcbfc004a40 comp rx=0 tx=0).stop 2024-04-26T23:07:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.226+0000 7fcc10a00640 1 -- 172.21.15.40:0/2554825833 shutdown_connections 2024-04-26T23:07:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.226+0000 7fcc10a00640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcbec06c8f0 0x7fcbec06edb0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.226+0000 7fcc10a00640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc0c162dc0 0x7fcc0c156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.226+0000 7fcc10a00640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc0c0d0fb0 0x7fcc0c0b2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.226+0000 7fcc10a00640 1 --2- 172.21.15.40:0/2554825833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc0c0cf2d0 0x7fcc0c0b7140 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.226+0000 7fcc10a00640 1 -- 172.21.15.40:0/2554825833 >> 172.21.15.40:0/2554825833 conn(0x7fcc0c0d2bd0 msgr2=0x7fcc0c1631a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.226+0000 7fcc10a00640 1 -- 172.21.15.40:0/2554825833 shutdown_connections 2024-04-26T23:07:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.226+0000 7fcc10a00640 1 -- 172.21.15.40:0/2554825833 wait complete. 2024-04-26T23:07:25.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a scrub start /d1/d2/d3 2024-04-26T23:07:25.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.460+0000 7ff801000640 1 Processor -- start 2024-04-26T23:07:25.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.460+0000 7ff801000640 1 -- start start 2024-04-26T23:07:25.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.461+0000 7ff801000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7fc0b5120 0x7ff7fc0b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.461+0000 7ff801000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7fc0d3dc0 con 0x7ff7fc0b59b0 2024-04-26T23:07:25.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.462+0000 7ff801000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7fc0b75b0 con 0x7ff7fc0b5500 2024-04-26T23:07:25.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.462+0000 7ff801000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff7fc0d03b0 con 0x7ff7fc0b5120 2024-04-26T23:07:25.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.462+0000 7ff7fbe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff7fc0b59b0 0x7ff7fc0b7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34228/0 (socket says 172.21.15.40:34228) 2024-04-26T23:07:25.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.462+0000 7ff7fbe00640 1 -- 172.21.15.40:0/3773073345 learned_addr learned my addr 172.21.15.40:0/3773073345 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:25.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.462+0000 7ff7fb400640 1 --2- 172.21.15.40:0/3773073345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7fc0b5120 0x7ff7fc0b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.462+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2399072414 0 0) 0x7ff7fc0d3dc0 con 0x7ff7fc0b59b0 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7e0003520 con 0x7ff7fc0b59b0 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3670119231 0 0) 0x7ff7e0003520 con 0x7ff7fc0b59b0 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7fc0b5120 msgr2=0x7ff7fc0b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 --2- 172.21.15.40:0/3773073345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7fc0b5120 0x7ff7fc0b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 >> v1:172.21.15.103:6789/0 conn(0x7ff7fc0b5500 legacy=0x7ff7fc0b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7fc0d0600 con 0x7ff7fc0b59b0 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7ff7f40019f0 con 0x7ff7fc0b59b0 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 >> v1:172.21.15.40:6789/0 conn(0x7ff7fc0b59b0 legacy=0x7ff7fc0b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 --2- 172.21.15.40:0/3773073345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7e00039e0 0x7ff7e0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 --2- 172.21.15.40:0/3773073345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7fc0b5120 0x7ff7e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 --2- 172.21.15.40:0/3773073345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7e0004d40 0x7ff7e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc0d03b0 con 0x7ff7e00039e0 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e000baf0 con 0x7ff7e0004d40 2024-04-26T23:07:25.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff7e000bcf0 con 0x7ff7fc0b5120 2024-04-26T23:07:25.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.463+0000 7ff7fb400640 1 --2- 172.21.15.40:0/3773073345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7e00039e0 0x7ff7e0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.464+0000 7ff7fbe00640 1 --2- 172.21.15.40:0/3773073345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7e0004d40 0x7ff7e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.464+0000 7ff7fb400640 1 -- 172.21.15.40:0/3773073345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7fc0b5120 msgr2=0x7ff7e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.464+0000 7ff7faa00640 1 --2- 172.21.15.40:0/3773073345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7fc0b5120 0x7ff7e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.464+0000 7ff7fb400640 1 --2- 172.21.15.40:0/3773073345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7fc0b5120 0x7ff7e0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.464+0000 7ff7fb400640 1 -- 172.21.15.40:0/3773073345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7e0004d40 msgr2=0x7ff7e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.464+0000 7ff7fb400640 1 --2- 172.21.15.40:0/3773073345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7e0004d40 0x7ff7e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.464+0000 7ff7fb400640 1 -- 172.21.15.40:0/3773073345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7e0003520 con 0x7ff7e00039e0 2024-04-26T23:07:25.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.464+0000 7ff7faa00640 1 --2- 172.21.15.40:0/3773073345 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7fc0b5120 0x7ff7e0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:25.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.465+0000 7ff7fb400640 1 --2- 172.21.15.40:0/3773073345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7e00039e0 0x7ff7e0003e00 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009ac0 tx=0x7ff7e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164205d238134f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.465+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec036070 con 0x7ff7e00039e0 2024-04-26T23:07:25.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.466+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7ec031400 con 0x7ff7e00039e0 2024-04-26T23:07:25.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.466+0000 7ff7fa000640 1 -- 172.21.15.40:0/3773073345 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec009690 con 0x7ff7e00039e0 2024-04-26T23:07:25.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.466+0000 7ff801000640 1 -- 172.21.15.40:0/3773073345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7e00039e0 msgr2=0x7ff7e0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.466+0000 7ff801000640 1 --2- 172.21.15.40:0/3773073345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7e00039e0 0x7ff7e0003e00 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009ac0 tx=0x7ff7e0005830 comp rx=0 tx=0).stop 2024-04-26T23:07:25.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.466+0000 7ff7fbe00640 1 -- 172.21.15.40:0/3773073345 reap_dead start 2024-04-26T23:07:25.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.466+0000 7ff801000640 1 -- 172.21.15.40:0/3773073345 shutdown_connections 2024-04-26T23:07:25.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.466+0000 7ff801000640 1 -- 172.21.15.40:0/3773073345 >> 172.21.15.40:0/3773073345 conn(0x7ff7fc0d4db0 msgr2=0x7ff7fc0d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:25.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.466+0000 7ff801000640 1 -- 172.21.15.40:0/3773073345 shutdown_connections 2024-04-26T23:07:25.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.466+0000 7ff801000640 1 -- 172.21.15.40:0/3773073345 wait complete. 2024-04-26T23:07:25.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.468+0000 7ff801000640 1 Processor -- start 2024-04-26T23:07:25.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.469+0000 7ff801000640 1 -- start start 2024-04-26T23:07:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.469+0000 7ff801000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7e00039e0 0x7ff7fc1f4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.469+0000 7ff801000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7fc0b19a0 0x7ff7fc0b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff801000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7fc0cd000 0x7ff7fc0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff801000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc0b5ee0 con 0x7ff7fc0b19a0 2024-04-26T23:07:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff801000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc0d0bc0 con 0x7ff7e00039e0 2024-04-26T23:07:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff801000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff7fc0b6060 con 0x7ff7fc0cd000 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff7faa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7fc0b19a0 0x7ff7fc0b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff7faa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7fc0b19a0 0x7ff7fc0b1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41192/0 (socket says 172.21.15.40:41192) 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff7faa00640 1 -- 172.21.15.40:0/1521927803 learned_addr learned my addr 172.21.15.40:0/1521927803 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff7fbe00640 1 --2- 172.21.15.40:0/1521927803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7fc0cd000 0x7ff7fc0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff7fb400640 1 --2- 172.21.15.40:0/1521927803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7e00039e0 0x7ff7fc1f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff7faa00640 1 -- 172.21.15.40:0/1521927803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7fc0cd000 msgr2=0x7ff7fc0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff7faa00640 1 --2- 172.21.15.40:0/1521927803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7fc0cd000 0x7ff7fc0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff7faa00640 1 -- 172.21.15.40:0/1521927803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7e00039e0 msgr2=0x7ff7fc1f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff7faa00640 1 --2- 172.21.15.40:0/1521927803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7e00039e0 0x7ff7fc1f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.470+0000 7ff7faa00640 1 -- 172.21.15.40:0/1521927803 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7fc0cdd50 con 0x7ff7fc0b19a0 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.471+0000 7ff7faa00640 1 --2- 172.21.15.40:0/1521927803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7fc0b19a0 0x7ff7fc0b1e20 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7ff7e400d7a0 tx=0x7ff7e400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.471+0000 7ff7f3400640 1 -- 172.21.15.40:0/1521927803 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4005070 con 0x7ff7fc0b19a0 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.471+0000 7ff7f3400640 1 -- 172.21.15.40:0/1521927803 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e4005210 con 0x7ff7fc0b19a0 2024-04-26T23:07:25.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.471+0000 7ff7f3400640 1 -- 172.21.15.40:0/1521927803 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4007030 con 0x7ff7fc0b19a0 2024-04-26T23:07:25.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.471+0000 7ff801000640 1 -- 172.21.15.40:0/1521927803 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7fc0ce030 con 0x7ff7fc0b19a0 2024-04-26T23:07:25.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.471+0000 7ff801000640 1 -- 172.21.15.40:0/1521927803 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7fc0ce5f0 con 0x7ff7fc0b19a0 2024-04-26T23:07:25.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.474+0000 7ff7f3400640 1 -- 172.21.15.40:0/1521927803 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff7e401a020 con 0x7ff7fc0b19a0 2024-04-26T23:07:25.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.474+0000 7ff7f3400640 1 --2- 172.21.15.40:0/1521927803 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7d806c920 0x7ff7d806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.475+0000 7ff7f3400640 1 -- 172.21.15.40:0/1521927803 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7ff7e408b510 con 0x7ff7fc0b19a0 2024-04-26T23:07:25.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.475+0000 7ff7fb400640 1 --2- 172.21.15.40:0/1521927803 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7d806c920 0x7ff7d806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.475+0000 7ff805385740 1 Processor -- start 2024-04-26T23:07:25.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.475+0000 7ff805385740 1 -- start start 2024-04-26T23:07:25.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.476+0000 7ff805385740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891aaad80 0x55a891a5d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.476+0000 7ff805385740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891a8dea0 0x55a891a8d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.476+0000 7ff805385740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a8d760 0x55a891a62570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.476+0000 7ff805385740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a8919c68d0 con 0x55a891a8dea0 2024-04-26T23:07:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.476+0000 7ff805385740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a891963070 con 0x55a891aaad80 2024-04-26T23:07:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.476+0000 7ff805385740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a8919c2890 con 0x55a891a8d760 2024-04-26T23:07:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7faa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891a8dea0 0x55a891a8d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7faa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891a8dea0 0x55a891a8d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41198/0 (socket says 172.21.15.40:41198) 2024-04-26T23:07:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7faa00640 1 -- 172.21.15.40:0/2412044645 learned_addr learned my addr 172.21.15.40:0/2412044645 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7fb400640 1 --2- 172.21.15.40:0/1521927803 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7d806c920 0x7ff7d806ede0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec033000 tx=0x7ff7ec008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7fb400640 1 --2- 172.21.15.40:0/2412044645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891aaad80 0x55a891a5d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7fbe00640 1 --2- 172.21.15.40:0/2412044645 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a8d760 0x55a891a62570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7faa00640 1 -- 172.21.15.40:0/2412044645 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a8d760 msgr2=0x55a891a62570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7faa00640 1 --2- 172.21.15.40:0/2412044645 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a8d760 0x55a891a62570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7faa00640 1 -- 172.21.15.40:0/2412044645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891aaad80 msgr2=0x55a891a5d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7faa00640 1 --2- 172.21.15.40:0/2412044645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891aaad80 0x55a891a5d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7faa00640 1 -- 172.21.15.40:0/2412044645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a89198f7b0 con 0x55a891a8dea0 2024-04-26T23:07:25.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7fb400640 1 --2- 172.21.15.40:0/2412044645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891aaad80 0x55a891a5d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:25.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7fbe00640 1 --2- 172.21.15.40:0/2412044645 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a8d760 0x55a891a62570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:25.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.477+0000 7ff7faa00640 1 --2- 172.21.15.40:0/2412044645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891a8dea0 0x55a891a8d220 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7ff7e40555c0 tx=0x7ff7e4059920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b3df1d1b2bb4ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.478+0000 7ff7ebe00640 1 -- 172.21.15.40:0/2412044645 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4061d40 con 0x55a891a8dea0 2024-04-26T23:07:25.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.478+0000 7ff7ebe00640 1 -- 172.21.15.40:0/2412044645 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e4057460 con 0x55a891a8dea0 2024-04-26T23:07:25.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.478+0000 7ff7ebe00640 1 -- 172.21.15.40:0/2412044645 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e405d150 con 0x55a891a8dea0 2024-04-26T23:07:25.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.478+0000 7ff805385740 1 -- 172.21.15.40:0/2412044645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891a8dea0 msgr2=0x55a891a8d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.478+0000 7ff805385740 1 --2- 172.21.15.40:0/2412044645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891a8dea0 0x55a891a8d220 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7ff7e40555c0 tx=0x7ff7e4059920 comp rx=0 tx=0).stop 2024-04-26T23:07:25.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.479+0000 7ff805385740 1 -- 172.21.15.40:0/2412044645 shutdown_connections 2024-04-26T23:07:25.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.479+0000 7ff805385740 1 --2- 172.21.15.40:0/2412044645 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a8d760 0x55a891a62570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.479+0000 7ff805385740 1 --2- 172.21.15.40:0/2412044645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891a8dea0 0x55a891a8d220 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.479+0000 7ff805385740 1 --2- 172.21.15.40:0/2412044645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891aaad80 0x55a891a5d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.479+0000 7ff805385740 1 -- 172.21.15.40:0/2412044645 >> 172.21.15.40:0/2412044645 conn(0x55a891a5f980 msgr2=0x55a891aa8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:25.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.479+0000 7ff805385740 1 -- 172.21.15.40:0/2412044645 shutdown_connections 2024-04-26T23:07:25.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.479+0000 7ff805385740 1 -- 172.21.15.40:0/2412044645 wait complete. 2024-04-26T23:07:25.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.479+0000 7ff805385740 1 -- start start 2024-04-26T23:07:25.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.480+0000 7ff805385740 1 Processor -- start 2024-04-26T23:07:25.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.481+0000 7ff805385740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 0x55a891abbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.481+0000 7ff805385740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891aa8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.481+0000 7ff805385740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 0x55a891a77590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.481+0000 7ff805385740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a891a61b30 con 0x55a891aaad80 2024-04-26T23:07:25.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.481+0000 7ff805385740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a89194dec0 con 0x55a891a8dea0 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.481+0000 7ff805385740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a891a60f20 con 0x55a891a77110 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.481+0000 7ff805385740 10 client.0 _ll_register_callbacks cb 0x55a8919d3870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.481+0000 7ff7faa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891aa8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.481+0000 7ff7faa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891aa8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41208/0 (socket says 172.21.15.40:41208) 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.481+0000 7ff7faa00640 1 -- 172.21.15.40:0/1792854650 learned_addr learned my addr 172.21.15.40:0/1792854650 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.482+0000 7ff7faa00640 1 -- 172.21.15.40:0/1792854650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 msgr2=0x55a891a77590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.482+0000 7ff7faa00640 1 --2- 172.21.15.40:0/1792854650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 0x55a891a77590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.482+0000 7ff7faa00640 1 -- 172.21.15.40:0/1792854650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 msgr2=0x55a891abbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.482+0000 7ff7fb400640 1 --2- 172.21.15.40:0/1792854650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 0x55a891abbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.482+0000 7ff7faa00640 1 --2- 172.21.15.40:0/1792854650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 0x55a891abbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.482+0000 7ff7faa00640 1 -- 172.21.15.40:0/1792854650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8917a4690 con 0x55a891aaad80 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.482+0000 7ff7fb400640 1 --2- 172.21.15.40:0/1792854650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 0x55a891abbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.482+0000 7ff7faa00640 1 --2- 172.21.15.40:0/1792854650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891aa8220 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4061810 tx=0x7ff7e405c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e6763c791f7a14 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.482+0000 7ff7e8c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:25.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.482+0000 7ff7e8c00640 1 -- 172.21.15.40:0/1792854650 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e408b270 con 0x55a891aaad80 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.482+0000 7ff7e8c00640 1 -- 172.21.15.40:0/1792854650 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7ff7e40585b0 con 0x55a891aaad80 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.482+0000 7ff805385740 1 -- 172.21.15.40:0/1792854650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7e4057460 con 0x55a891aaad80 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.482+0000 7ff7e8c00640 1 -- 172.21.15.40:0/1792854650 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7ff7e4061d40 con 0x55a891aaad80 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.483+0000 7ff805385740 10 client.7767 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.483+0000 7ff805385740 1 -- 172.21.15.40:0/1792854650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x55a891804e20 con 0x55a891aaad80 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.483+0000 7ff7e8c00640 1 -- 172.21.15.40:0/1792854650 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e4057460 con 0x55a891aaad80 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.483+0000 7ff7e8c00640 1 -- 172.21.15.40:0/1792854650 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7ff7e4054eb0 con 0x55a891aaad80 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.483+0000 7ff805385740 10 client.7767 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.483+0000 7ff805385740 10 client.7767 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.483+0000 7ff805385740 20 client.7767 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.483+0000 7ff805385740 1 --2- 172.21.15.40:0/1792854650 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55a891a78120 0x55a891a78500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.483+0000 7ff805385740 4 client.7767 mds_command: new command op to 7443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.483+0000 7ff805385740 1 -- 172.21.15.40:0/1792854650 --> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8919d44e0 con 0x55a891a78120 2024-04-26T23:07:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.483+0000 7ff7fb400640 1 --2- 172.21.15.40:0/1792854650 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55a891a78120 0x55a891a78500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.484+0000 7ff7fb400640 1 --2- 172.21.15.40:0/1792854650 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55a891a78120 0x55a891a78500 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a781d889dd59ad41 server_cookie=964658d8efcf1f6a in_seq=0 out_seq=0 2024-04-26T23:07:25.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.484+0000 7ff7e8c00640 10 client.7767 ms_handle_connect on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:25.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff7e8c00640 1 -- 172.21.15.40:0/1792854650 <== mds.1 v2:172.21.15.40:6832/216893598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8919d44e0 con 0x55a891a78120 2024-04-26T23:07:25.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff7e8c00640 10 client.7767 handle_command_reply: tid=0 2024-04-26T23:07:25.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff805385740 1 client.7767 shutdown 2024-04-26T23:07:25.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff805385740 1 -- 172.21.15.40:0/1792854650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 msgr2=0x55a891aa8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff805385740 1 --2- 172.21.15.40:0/1792854650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891aa8220 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4061810 tx=0x7ff7e405c040 comp rx=0 tx=0).stop 2024-04-26T23:07:25.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff805385740 1 -- 172.21.15.40:0/1792854650 shutdown_connections 2024-04-26T23:07:25.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff805385740 1 --2- 172.21.15.40:0/1792854650 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55a891a78120 0x55a891a78500 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff805385740 1 --2- 172.21.15.40:0/1792854650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 0x55a891a77590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff805385740 1 --2- 172.21.15.40:0/1792854650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891aa8220 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff805385740 1 --2- 172.21.15.40:0/1792854650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 0x55a891abbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff7e8c00640 0 client.7767 ms_handle_reset on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:25.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff805385740 1 -- 172.21.15.40:0/1792854650 >> 172.21.15.40:0/1792854650 conn(0x55a891a8d760 msgr2=0x55a891a61100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:25.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff805385740 1 -- 172.21.15.40:0/1792854650 shutdown_connections 2024-04-26T23:07:25.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.485+0000 7ff805385740 1 -- 172.21.15.40:0/1792854650 wait complete. 2024-04-26T23:07:25.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.486+0000 7ff805385740 20 client.7767 trim_cache size 0 max 16384 2024-04-26T23:07:25.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.505+0000 7ff805385740 1 Processor -- start 2024-04-26T23:07:25.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.505+0000 7ff805385740 1 -- start start 2024-04-26T23:07:25.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.506+0000 7ff805385740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 0x55a891abba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.506+0000 7ff805385740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 0x55a891abbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.506+0000 7ff805385740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891a62cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.506+0000 7ff805385740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc201750 con 0x55a891aaad80 2024-04-26T23:07:25.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.506+0000 7ff805385740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc201e50 con 0x55a891a8dea0 2024-04-26T23:07:25.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.506+0000 7ff805385740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff7fc2007f0 con 0x55a891a77110 2024-04-26T23:07:25.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.506+0000 7ff7fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891a62cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.506+0000 7ff7fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891a62cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41210/0 (socket says 172.21.15.40:41210) 2024-04-26T23:07:25.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.506+0000 7ff7fbe00640 1 -- 172.21.15.40:0/1993899057 learned_addr learned my addr 172.21.15.40:0/1993899057 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.507+0000 7ff7fb400640 1 --2- 172.21.15.40:0/1993899057 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 0x55a891abba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.507+0000 7ff7fbe00640 1 -- 172.21.15.40:0/1993899057 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 msgr2=0x55a891abba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.507+0000 7ff7fbe00640 1 --2- 172.21.15.40:0/1993899057 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 0x55a891abba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.507+0000 7ff7fbe00640 1 -- 172.21.15.40:0/1993899057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 msgr2=0x55a891abbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.507+0000 7ff7faa00640 1 --2- 172.21.15.40:0/1993899057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 0x55a891abbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.507+0000 7ff7fbe00640 1 --2- 172.21.15.40:0/1993899057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 0x55a891abbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.507+0000 7ff7fbe00640 1 -- 172.21.15.40:0/1993899057 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a891697f80 con 0x55a891aaad80 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.507+0000 7ff7fbe00640 1 --2- 172.21.15.40:0/1993899057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891a62cc0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc08e270 tx=0x7ff7f4001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=186c298f6888ccaf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.507+0000 7ff7eb400640 1 -- 172.21.15.40:0/1993899057 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4007790 con 0x55a891aaad80 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.507+0000 7ff7eb400640 1 -- 172.21.15.40:0/1993899057 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f4019070 con 0x55a891aaad80 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.507+0000 7ff7eb400640 1 -- 172.21.15.40:0/1993899057 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4014400 con 0x55a891aaad80 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.508+0000 7ff805385740 1 -- 172.21.15.40:0/1993899057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 msgr2=0x55a891a62cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.508+0000 7ff805385740 1 --2- 172.21.15.40:0/1993899057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891a62cc0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc08e270 tx=0x7ff7f4001ae0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.508+0000 7ff805385740 1 -- 172.21.15.40:0/1993899057 shutdown_connections 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.508+0000 7ff805385740 1 --2- 172.21.15.40:0/1993899057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891a62cc0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.508+0000 7ff805385740 1 --2- 172.21.15.40:0/1993899057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 0x55a891abbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.508+0000 7ff805385740 1 --2- 172.21.15.40:0/1993899057 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 0x55a891abba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.508+0000 7ff805385740 1 -- 172.21.15.40:0/1993899057 >> 172.21.15.40:0/1993899057 conn(0x55a891a8d760 msgr2=0x55a891a60520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:25.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.508+0000 7ff805385740 1 -- 172.21.15.40:0/1993899057 shutdown_connections 2024-04-26T23:07:25.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.508+0000 7ff805385740 1 -- 172.21.15.40:0/1993899057 wait complete. 2024-04-26T23:07:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.509+0000 7ff805385740 1 -- start start 2024-04-26T23:07:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.509+0000 7ff805385740 1 Processor -- start 2024-04-26T23:07:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.510+0000 7ff805385740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 0x55a891b24320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.510+0000 7ff805385740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 0x55a891a89680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.510+0000 7ff805385740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891a89bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.510+0000 7ff805385740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a891a61b30 con 0x55a891aaad80 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.510+0000 7ff805385740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a891a60f20 con 0x55a891a8dea0 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.510+0000 7ff805385740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a89194dec0 con 0x55a891a77110 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.510+0000 7ff805385740 10 client.0 _ll_register_callbacks cb 0x7ff7e4058b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.510+0000 7ff7fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891a89bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.511+0000 7ff7fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891a89bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41216/0 (socket says 172.21.15.40:41216) 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.511+0000 7ff7fbe00640 1 -- 172.21.15.40:0/2265118699 learned_addr learned my addr 172.21.15.40:0/2265118699 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.511+0000 7ff7faa00640 1 --2- 172.21.15.40:0/2265118699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 0x55a891a89680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.511+0000 7ff7fb400640 1 --2- 172.21.15.40:0/2265118699 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 0x55a891b24320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.511+0000 7ff7fbe00640 1 -- 172.21.15.40:0/2265118699 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 msgr2=0x55a891b24320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.511+0000 7ff7fbe00640 1 --2- 172.21.15.40:0/2265118699 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 0x55a891b24320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.511+0000 7ff7fbe00640 1 -- 172.21.15.40:0/2265118699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 msgr2=0x55a891a89680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.511+0000 7ff7fbe00640 1 --2- 172.21.15.40:0/2265118699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 0x55a891a89680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.511+0000 7ff7fbe00640 1 -- 172.21.15.40:0/2265118699 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a891a7ab50 con 0x55a891aaad80 2024-04-26T23:07:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.511+0000 7ff7fbe00640 1 --2- 172.21.15.40:0/2265118699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891a89bc0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4016040 tx=0x7ff7f4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272bac362bc01516 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.511+0000 7ff7e8c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:25.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.512+0000 7ff7e8c00640 1 -- 172.21.15.40:0/2265118699 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4014bc0 con 0x55a891aaad80 2024-04-26T23:07:25.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.512+0000 7ff7e8c00640 1 -- 172.21.15.40:0/2265118699 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7ff7f4014d60 con 0x55a891aaad80 2024-04-26T23:07:25.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.512+0000 7ff805385740 1 -- 172.21.15.40:0/2265118699 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7f4019070 con 0x55a891aaad80 2024-04-26T23:07:25.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.512+0000 7ff7e8c00640 1 -- 172.21.15.40:0/2265118699 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7ff7f402e050 con 0x55a891aaad80 2024-04-26T23:07:25.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.512+0000 7ff805385740 10 client.7773 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:25.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.512+0000 7ff805385740 1 -- 172.21.15.40:0/2265118699 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x55a891a8a5a0 con 0x55a891aaad80 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.512+0000 7ff7e8c00640 1 -- 172.21.15.40:0/2265118699 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f4019070 con 0x55a891aaad80 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.513+0000 7ff7e8c00640 1 -- 172.21.15.40:0/2265118699 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7ff7f4031020 con 0x55a891aaad80 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.513+0000 7ff805385740 10 client.7773 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.513+0000 7ff805385740 10 client.7773 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.513+0000 7ff805385740 20 client.7773 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.513+0000 7ff805385740 1 --2- 172.21.15.40:0/2265118699 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55a891a621d0 0x55a891ab6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.513+0000 7ff805385740 4 client.7773 mds_command: new command op to 7443 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/d1/d2/d3"}] 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.513+0000 7ff805385740 1 -- 172.21.15.40:0/2265118699 --> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] -- command(tid 0: {"prefix": "scrub start", "path": "/d1/d2/d3"}) -- 0x55a8919ce350 con 0x55a891a621d0 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.513+0000 7ff7fb400640 1 --2- 172.21.15.40:0/2265118699 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55a891a621d0 0x55a891ab6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.513+0000 7ff7fb400640 1 --2- 172.21.15.40:0/2265118699 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55a891a621d0 0x55a891ab6920 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5647e22fb98750fa server_cookie=c815f0e510a342c in_seq=0 out_seq=0 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.513+0000 7ff7e8c00640 10 client.7773 ms_handle_connect on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff7e8c00640 1 -- 172.21.15.40:0/2265118699 <== mds.1 v2:172.21.15.40:6832/216893598 1 ==== command_reply(tid 0: -18 Not rank 0) ==== 18+0+0 (crc 0 0 0) 0x55a8919ce350 con 0x55a891a621d0 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff7e8c00640 10 client.7773 handle_command_reply: tid=0 2024-04-26T23:07:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff805385740 1 client.7773 shutdown 2024-04-26T23:07:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff805385740 1 -- 172.21.15.40:0/2265118699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 msgr2=0x55a891a89bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff805385740 1 --2- 172.21.15.40:0/2265118699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891a89bc0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4016040 tx=0x7ff7f4009040 comp rx=0 tx=0).stop 2024-04-26T23:07:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff805385740 1 -- 172.21.15.40:0/2265118699 shutdown_connections 2024-04-26T23:07:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff805385740 1 --2- 172.21.15.40:0/2265118699 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55a891a621d0 0x55a891ab6920 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff805385740 1 --2- 172.21.15.40:0/2265118699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a891aaad80 0x55a891a89bc0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff805385740 1 --2- 172.21.15.40:0/2265118699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a891a8dea0 0x55a891a89680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff805385740 1 --2- 172.21.15.40:0/2265118699 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a891a77110 0x55a891b24320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff805385740 1 -- 172.21.15.40:0/2265118699 >> 172.21.15.40:0/2265118699 conn(0x55a891a8d760 msgr2=0x55a891a60520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff7e8c00640 0 client.7773 ms_handle_reset on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff805385740 1 -- 172.21.15.40:0/2265118699 shutdown_connections 2024-04-26T23:07:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff805385740 1 -- 172.21.15.40:0/2265118699 wait complete. 2024-04-26T23:07:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.514+0000 7ff805385740 20 client.7773 trim_cache size 0 max 16384 2024-04-26T23:07:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:Error EXDEV: Not rank 0 2024-04-26T23:07:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 18 2024-04-26T23:07:25.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a scrub abort 2024-04-26T23:07:25.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.720+0000 7f7fb1c00640 1 Processor -- start 2024-04-26T23:07:25.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.720+0000 7f7fb1c00640 1 -- start start 2024-04-26T23:07:25.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.722+0000 7f7fb1c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7fac15efe0 0x7f7fac073930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.722+0000 7f7fb1c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7fac06cc00 0x7f7fac0b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.722+0000 7f7fb1c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fac0b4b10 0x7f7fac0b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.722+0000 7f7fb1c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f7fac0b54a0 con 0x7f7fac15efe0 2024-04-26T23:07:25.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.722+0000 7f7fb1c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7fac06d090 con 0x7f7fac06cc00 2024-04-26T23:07:25.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.722+0000 7f7fb1c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fac072e90 con 0x7f7fac0b4b10 2024-04-26T23:07:25.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.723+0000 7f7fabe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7fac15efe0 0x7f7fac073930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.723+0000 7f7fabe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7fac15efe0 0x7f7fac073930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41226/0 (socket says 172.21.15.40:41226) 2024-04-26T23:07:25.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.723+0000 7f7fabe00640 1 -- 172.21.15.40:0/2010517241 learned_addr learned my addr 172.21.15.40:0/2010517241 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.723+0000 7f7fb1200640 1 --2- 172.21.15.40:0/2010517241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fac0b4b10 0x7f7fac0b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.723+0000 7f7fa3400640 1 --2- 172.21.15.40:0/2010517241 >> v2:172.21.15.103:3300/0 conn(0x7f7fac06cc00 0x7f7fac0b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.723+0000 7f7fabe00640 1 -- 172.21.15.40:0/2010517241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fac0b4b10 msgr2=0x7f7fac0b4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.723+0000 7f7fabe00640 1 --2- 172.21.15.40:0/2010517241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fac0b4b10 0x7f7fac0b4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.723+0000 7f7fabe00640 1 -- 172.21.15.40:0/2010517241 >> v2:172.21.15.103:3300/0 conn(0x7f7fac06cc00 msgr2=0x7f7fac0b6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.723+0000 7f7fabe00640 1 --2- 172.21.15.40:0/2010517241 >> v2:172.21.15.103:3300/0 conn(0x7f7fac06cc00 0x7f7fac0b6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.723+0000 7f7fabe00640 1 -- 172.21.15.40:0/2010517241 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fac15f3c0 con 0x7f7fac15efe0 2024-04-26T23:07:25.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.724+0000 7f7fabe00640 1 --2- 172.21.15.40:0/2010517241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fac15efe0 0x7f7fac073930 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f7fac072a20 tx=0x7f7f9c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133c4a46e97022e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.725+0000 7f7fab400640 1 -- 172.21.15.40:0/2010517241 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c037070 con 0x7f7fac15efe0 2024-04-26T23:07:25.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.725+0000 7f7fab400640 1 -- 172.21.15.40:0/2010517241 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f9c032440 con 0x7f7fac15efe0 2024-04-26T23:07:25.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.725+0000 7f7fab400640 1 -- 172.21.15.40:0/2010517241 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c032810 con 0x7f7fac15efe0 2024-04-26T23:07:25.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.725+0000 7f7fb1c00640 1 -- 172.21.15.40:0/2010517241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fac15efe0 msgr2=0x7f7fac073930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.725+0000 7f7fb1c00640 1 --2- 172.21.15.40:0/2010517241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fac15efe0 0x7f7fac073930 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f7fac072a20 tx=0x7f7f9c02a140 comp rx=0 tx=0).stop 2024-04-26T23:07:25.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.726+0000 7f7fb1c00640 1 -- 172.21.15.40:0/2010517241 shutdown_connections 2024-04-26T23:07:25.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.726+0000 7f7fb1c00640 1 --2- 172.21.15.40:0/2010517241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fac0b4b10 0x7f7fac0b4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.726+0000 7f7fb1c00640 1 --2- 172.21.15.40:0/2010517241 >> v2:172.21.15.103:3300/0 conn(0x7f7fac06cc00 0x7f7fac0b6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.726+0000 7f7fb1c00640 1 --2- 172.21.15.40:0/2010517241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fac15efe0 0x7f7fac073930 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.726+0000 7f7fb1c00640 1 -- 172.21.15.40:0/2010517241 >> 172.21.15.40:0/2010517241 conn(0x7f7fac0b3810 msgr2=0x7f7fac0b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:25.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.726+0000 7f7fb1c00640 1 -- 172.21.15.40:0/2010517241 shutdown_connections 2024-04-26T23:07:25.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.726+0000 7f7fb1c00640 1 -- 172.21.15.40:0/2010517241 wait complete. 2024-04-26T23:07:25.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.727+0000 7f7fb1c00640 1 Processor -- start 2024-04-26T23:07:25.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.728+0000 7f7fb1c00640 1 -- start start 2024-04-26T23:07:25.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.729+0000 7f7fb1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fac06cc00 0x7f7fac14e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.729+0000 7f7fb1c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fac0b4b10 0x7f7fac14a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.729+0000 7f7fb1c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fac15efe0 0x7f7fac15ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.729+0000 7f7fb1c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac073410 con 0x7f7fac06cc00 2024-04-26T23:07:25.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.729+0000 7f7fb1c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac073130 con 0x7f7fac15efe0 2024-04-26T23:07:25.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.729+0000 7f7fb1c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fac0704f0 con 0x7f7fac0b4b10 2024-04-26T23:07:25.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.729+0000 7f7fabe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fac06cc00 0x7f7fac14e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.729+0000 7f7fabe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fac06cc00 0x7f7fac14e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41232/0 (socket says 172.21.15.40:41232) 2024-04-26T23:07:25.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.729+0000 7f7fabe00640 1 -- 172.21.15.40:0/2476614774 learned_addr learned my addr 172.21.15.40:0/2476614774 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.729+0000 7f7fa3400640 1 --2- 172.21.15.40:0/2476614774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fac0b4b10 0x7f7fac14a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.729+0000 7f7fabe00640 1 -- 172.21.15.40:0/2476614774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fac0b4b10 msgr2=0x7f7fac14a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.730+0000 7f7fabe00640 1 --2- 172.21.15.40:0/2476614774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fac0b4b10 0x7f7fac14a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.730+0000 7f7fabe00640 1 -- 172.21.15.40:0/2476614774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fac15efe0 msgr2=0x7f7fac15ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.730+0000 7f7fabe00640 1 --2- 172.21.15.40:0/2476614774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fac15efe0 0x7f7fac15ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.730+0000 7f7fabe00640 1 -- 172.21.15.40:0/2476614774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fac14aef0 con 0x7f7fac06cc00 2024-04-26T23:07:25.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.730+0000 7f7fabe00640 1 --2- 172.21.15.40:0/2476614774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fac06cc00 0x7f7fac14e340 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c02a180 tx=0x7f7f9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.730+0000 7f7fa8c00640 1 -- 172.21.15.40:0/2476614774 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c041020 con 0x7f7fac06cc00 2024-04-26T23:07:25.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.730+0000 7f7fa8c00640 1 -- 172.21.15.40:0/2476614774 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f9c0306c0 con 0x7f7fac06cc00 2024-04-26T23:07:25.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.730+0000 7f7fa8c00640 1 -- 172.21.15.40:0/2476614774 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c037040 con 0x7f7fac06cc00 2024-04-26T23:07:25.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.730+0000 7f7fb1c00640 1 -- 172.21.15.40:0/2476614774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fac14b190 con 0x7f7fac06cc00 2024-04-26T23:07:25.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.731+0000 7f7fb1c00640 1 -- 172.21.15.40:0/2476614774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fac14b750 con 0x7f7fac06cc00 2024-04-26T23:07:25.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.733+0000 7f7fa8c00640 1 -- 172.21.15.40:0/2476614774 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7f9c044020 con 0x7f7fac06cc00 2024-04-26T23:07:25.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.734+0000 7f7fa8c00640 1 --2- 172.21.15.40:0/2476614774 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7f8806c940 0x7f7f8806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.734+0000 7f7fa8c00640 1 -- 172.21.15.40:0/2476614774 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f7f9c0ac560 con 0x7f7fac06cc00 2024-04-26T23:07:25.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.735+0000 7f7fb61ee740 1 Processor -- start 2024-04-26T23:07:25.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.735+0000 7f7fa3400640 1 --2- 172.21.15.40:0/2476614774 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7f8806c940 0x7f7f8806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.735+0000 7f7fb61ee740 1 -- start start 2024-04-26T23:07:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.736+0000 7f7fb61ee740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848b9c0a0 0x558848b9c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.736+0000 7f7fb61ee740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b96f30 0x558848b9aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.736+0000 7f7fb61ee740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9b1e0 0x558848b9b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.736+0000 7f7fb61ee740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558848b00870 con 0x558848b96f30 2024-04-26T23:07:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.736+0000 7f7fb61ee740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558848a9ce40 con 0x558848b9c0a0 2024-04-26T23:07:25.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.736+0000 7f7fb61ee740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558848afc830 con 0x558848b9b1e0 2024-04-26T23:07:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.736+0000 7f7fa3400640 1 --2- 172.21.15.40:0/2476614774 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7f8806c940 0x7f7f8806ee00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7fac1f8200 tx=0x7f7f94008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.736+0000 7f7fa3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b96f30 0x558848b9aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.737+0000 7f7fa3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b96f30 0x558848b9aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41240/0 (socket says 172.21.15.40:41240) 2024-04-26T23:07:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.737+0000 7f7fa3400640 1 -- 172.21.15.40:0/1332182161 learned_addr learned my addr 172.21.15.40:0/1332182161 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.737+0000 7f7fabe00640 1 --2- 172.21.15.40:0/1332182161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848b9c0a0 0x558848b9c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.737+0000 7f7fb1200640 1 --2- 172.21.15.40:0/1332182161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9b1e0 0x558848b9b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.737+0000 7f7fa3400640 1 -- 172.21.15.40:0/1332182161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9b1e0 msgr2=0x558848b9b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.737+0000 7f7fa3400640 1 --2- 172.21.15.40:0/1332182161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9b1e0 0x558848b9b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.737+0000 7f7fa3400640 1 -- 172.21.15.40:0/1332182161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848b9c0a0 msgr2=0x558848b9c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.737+0000 7f7fa3400640 1 --2- 172.21.15.40:0/1332182161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848b9c0a0 0x558848b9c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.737+0000 7f7fa3400640 1 -- 172.21.15.40:0/1332182161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558848ac6080 con 0x558848b96f30 2024-04-26T23:07:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.737+0000 7f7fa3400640 1 --2- 172.21.15.40:0/1332182161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b96f30 0x558848b9aca0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400e800 tx=0x7f7f94011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4dbfbfc398b8123 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.738+0000 7f7fa0c00640 1 -- 172.21.15.40:0/1332182161 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94025070 con 0x558848b96f30 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.738+0000 7f7fa0c00640 1 -- 172.21.15.40:0/1332182161 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f94015640 con 0x558848b96f30 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.738+0000 7f7fa0c00640 1 -- 172.21.15.40:0/1332182161 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9400f040 con 0x558848b96f30 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.738+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1332182161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b96f30 msgr2=0x558848b9aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.738+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1332182161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b96f30 0x558848b9aca0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400e800 tx=0x7f7f94011000 comp rx=0 tx=0).stop 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.738+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1332182161 shutdown_connections 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.738+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1332182161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9b1e0 0x558848b9b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.738+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1332182161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b96f30 0x558848b9aca0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.738+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1332182161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848b9c0a0 0x558848b9c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.738+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1332182161 >> 172.21.15.40:0/1332182161 conn(0x558848b991a0 msgr2=0x558848be1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.739+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1332182161 shutdown_connections 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.739+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1332182161 wait complete. 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.739+0000 7f7fb61ee740 1 -- start start 2024-04-26T23:07:25.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.740+0000 7f7fb61ee740 1 Processor -- start 2024-04-26T23:07:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fb61ee740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 0x558848bbc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fb61ee740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 0x558848bbc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fb61ee740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848be5770 0x558848be5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fb61ee740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558848bc7c30 con 0x558848b9b1e0 2024-04-26T23:07:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fb61ee740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558848a87d90 con 0x558848be5770 2024-04-26T23:07:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fb61ee740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558848bc70b0 con 0x558848b9c0a0 2024-04-26T23:07:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fb61ee740 10 client.0 _ll_register_callbacks cb 0x558848a83680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fabe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 0x558848bbc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fabe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 0x558848bbc080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41244/0 (socket says 172.21.15.40:41244) 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fabe00640 1 -- 172.21.15.40:0/1397584076 learned_addr learned my addr 172.21.15.40:0/1397584076 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fa3400640 1 --2- 172.21.15.40:0/1397584076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 0x558848bbc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fabe00640 1 -- 172.21.15.40:0/1397584076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 msgr2=0x558848bbc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fabe00640 1 --2- 172.21.15.40:0/1397584076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 0x558848bbc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fabe00640 1 -- 172.21.15.40:0/1397584076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848be5770 msgr2=0x558848be5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fb1200640 1 --2- 172.21.15.40:0/1397584076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848be5770 0x558848be5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fabe00640 1 --2- 172.21.15.40:0/1397584076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848be5770 0x558848be5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.741+0000 7f7fabe00640 1 -- 172.21.15.40:0/1397584076 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588488dc500 con 0x558848b9b1e0 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.742+0000 7f7fabe00640 1 --2- 172.21.15.40:0/1397584076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 0x558848bbc080 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c076540 tx=0x7f7f9c07a8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ec082a2c6a3a56 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.742+0000 7f7f99600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.742+0000 7f7f99600640 1 -- 172.21.15.40:0/1397584076 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c07e490 con 0x558848b9b1e0 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.742+0000 7f7f99600640 1 -- 172.21.15.40:0/1397584076 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f7f9c082830 con 0x558848b9b1e0 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.742+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1397584076 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f94015640 con 0x558848b9b1e0 2024-04-26T23:07:25.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.742+0000 7f7f99600640 1 -- 172.21.15.40:0/1397584076 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f7f9c082cc0 con 0x558848b9b1e0 2024-04-26T23:07:25.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.742+0000 7f7fb61ee740 10 client.7785 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:25.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.742+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1397584076 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x55884893cc90 con 0x558848b9b1e0 2024-04-26T23:07:25.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.742+0000 7f7f99600640 1 -- 172.21.15.40:0/1397584076 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f94015640 con 0x558848b9b1e0 2024-04-26T23:07:25.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.742+0000 7f7f99600640 1 -- 172.21.15.40:0/1397584076 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7f7f9c03a030 con 0x558848b9b1e0 2024-04-26T23:07:25.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.743+0000 7f7fb61ee740 10 client.7785 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:25.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.743+0000 7f7fb61ee740 10 client.7785 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:07:25.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.743+0000 7f7fb61ee740 20 client.7785 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:25.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.743+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1397584076 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558848be37a0 0x558848be3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.743+0000 7f7fb61ee740 4 client.7785 mds_command: new command op to 7443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:07:25.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.743+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1397584076 --> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558848b0e480 con 0x558848be37a0 2024-04-26T23:07:25.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.743+0000 7f7fa3400640 1 --2- 172.21.15.40:0/1397584076 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558848be37a0 0x558848be3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.744+0000 7f7fa3400640 1 --2- 172.21.15.40:0/1397584076 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558848be37a0 0x558848be3ba0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c5a2a20f59fc155 server_cookie=45c4221c8827bfe6 in_seq=0 out_seq=0 2024-04-26T23:07:25.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.744+0000 7f7f99600640 10 client.7785 ms_handle_connect on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.744+0000 7f7f99600640 1 -- 172.21.15.40:0/1397584076 <== mds.1 v2:172.21.15.40:6832/216893598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558848b0e480 con 0x558848be37a0 2024-04-26T23:07:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.744+0000 7f7f99600640 10 client.7785 handle_command_reply: tid=0 2024-04-26T23:07:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.744+0000 7f7fb61ee740 1 client.7785 shutdown 2024-04-26T23:07:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.745+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1397584076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 msgr2=0x558848bbc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.745+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1397584076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 0x558848bbc080 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c076540 tx=0x7f7f9c07a8a0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.745+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1397584076 shutdown_connections 2024-04-26T23:07:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.745+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1397584076 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558848be37a0 0x558848be3ba0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.745+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1397584076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848be5770 0x558848be5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.745+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1397584076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 0x558848bbc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.745+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1397584076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 0x558848bbc080 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.745+0000 7f7f99600640 0 client.7785 ms_handle_reset on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.745+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1397584076 >> 172.21.15.40:0/1397584076 conn(0x558848b96f30 msgr2=0x558848bbdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.745+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1397584076 shutdown_connections 2024-04-26T23:07:25.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.745+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1397584076 wait complete. 2024-04-26T23:07:25.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.745+0000 7f7fb61ee740 20 client.7785 trim_cache size 0 max 16384 2024-04-26T23:07:25.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.764+0000 7f7fb61ee740 1 Processor -- start 2024-04-26T23:07:25.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.764+0000 7f7fb61ee740 1 -- start start 2024-04-26T23:07:25.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.765+0000 7f7fb61ee740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 0x558848be5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.765+0000 7f7fb61ee740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 0x558848be9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.765+0000 7f7fb61ee740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848be5770 0x558848bc6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.765+0000 7f7fb61ee740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac200c20 con 0x558848b9b1e0 2024-04-26T23:07:25.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.765+0000 7f7fb61ee740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac201320 con 0x558848be5770 2024-04-26T23:07:25.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.765+0000 7f7fb61ee740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fac1ffdf0 con 0x558848b9c0a0 2024-04-26T23:07:25.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.765+0000 7f7fabe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 0x558848be5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.765+0000 7f7fabe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 0x558848be5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41258/0 (socket says 172.21.15.40:41258) 2024-04-26T23:07:25.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.765+0000 7f7fabe00640 1 -- 172.21.15.40:0/1565449576 learned_addr learned my addr 172.21.15.40:0/1565449576 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.766+0000 7f7fabe00640 1 -- 172.21.15.40:0/1565449576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 msgr2=0x558848be9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.766+0000 7f7fabe00640 1 --2- 172.21.15.40:0/1565449576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 0x558848be9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.766+0000 7f7fabe00640 1 -- 172.21.15.40:0/1565449576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848be5770 msgr2=0x558848bc6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.766+0000 7f7fabe00640 1 --2- 172.21.15.40:0/1565449576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848be5770 0x558848bc6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.766+0000 7f7fabe00640 1 -- 172.21.15.40:0/1565449576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588487d1e20 con 0x558848b9b1e0 2024-04-26T23:07:25.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.766+0000 7f7fabe00640 1 --2- 172.21.15.40:0/1565449576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 0x558848be5200 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c02a730 tx=0x7f7f9c07d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229871ae023e107 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.766+0000 7f7f9be00640 1 -- 172.21.15.40:0/1565449576 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c079c20 con 0x558848b9b1e0 2024-04-26T23:07:25.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.766+0000 7f7f9be00640 1 -- 172.21.15.40:0/1565449576 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f9c07e490 con 0x558848b9b1e0 2024-04-26T23:07:25.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.766+0000 7f7f9be00640 1 -- 172.21.15.40:0/1565449576 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c0ac0b0 con 0x558848b9b1e0 2024-04-26T23:07:25.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.767+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1565449576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 msgr2=0x558848be5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.767+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1565449576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 0x558848be5200 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c02a730 tx=0x7f7f9c07d000 comp rx=0 tx=0).stop 2024-04-26T23:07:25.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.767+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1565449576 shutdown_connections 2024-04-26T23:07:25.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.767+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1565449576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848be5770 0x558848bc6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.767+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1565449576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 0x558848be9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.767+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1565449576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848b9b1e0 0x558848be5200 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.767+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1565449576 >> 172.21.15.40:0/1565449576 conn(0x558848b96f30 msgr2=0x558848be3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:25.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.767+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1565449576 shutdown_connections 2024-04-26T23:07:25.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.767+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1565449576 wait complete. 2024-04-26T23:07:25.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.768+0000 7f7fb61ee740 1 -- start start 2024-04-26T23:07:25.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.768+0000 7f7fb61ee740 1 Processor -- start 2024-04-26T23:07:25.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.769+0000 7f7fb61ee740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848b9b1e0 0x558848c5c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.769+0000 7f7fb61ee740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 0x558848b9c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.769+0000 7f7fb61ee740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848be5770 0x558848b9cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.769+0000 7f7fb61ee740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558848bc7c30 con 0x558848be5770 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.769+0000 7f7fb61ee740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558848a87d90 con 0x558848b9b1e0 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.769+0000 7f7fb61ee740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558848bbd320 con 0x558848b9c0a0 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.769+0000 7f7fb61ee740 10 client.0 _ll_register_callbacks cb 0x7f7f9c0095b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.769+0000 7f7fb1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848be5770 0x558848b9cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.769+0000 7f7fb1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848be5770 0x558848b9cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41272/0 (socket says 172.21.15.40:41272) 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.769+0000 7f7fb1200640 1 -- 172.21.15.40:0/1497880440 learned_addr learned my addr 172.21.15.40:0/1497880440 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.769+0000 7f7fa3400640 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 0x558848b9c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.770+0000 7f7fabe00640 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848b9b1e0 0x558848c5c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.770+0000 7f7fb1200640 1 -- 172.21.15.40:0/1497880440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 msgr2=0x558848b9c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.770+0000 7f7fb1200640 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 0x558848b9c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.770+0000 7f7fb1200640 1 -- 172.21.15.40:0/1497880440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848b9b1e0 msgr2=0x558848c5c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.770+0000 7f7fb1200640 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848b9b1e0 0x558848c5c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.770+0000 7f7fb1200640 1 -- 172.21.15.40:0/1497880440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558848be2c70 con 0x558848be5770 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.770+0000 7f7fabe00640 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848b9b1e0 0x558848c5c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.770+0000 7f7fb1200640 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848be5770 0x558848b9cca0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4009f50 tx=0x7f7fa40044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189faf9ed5077bdf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.770+0000 7f7f99600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.770+0000 7f7f99600640 1 -- 172.21.15.40:0/1497880440 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4007740 con 0x558848be5770 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.770+0000 7f7f99600640 1 -- 172.21.15.40:0/1497880440 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f7fa40078e0 con 0x558848be5770 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.770+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1497880440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f9c07e490 con 0x558848be5770 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.771+0000 7f7f99600640 1 -- 172.21.15.40:0/1497880440 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f7fa4022050 con 0x558848be5770 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.771+0000 7f7fb61ee740 10 client.7791 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.771+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1497880440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x558848bc7690 con 0x558848be5770 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.771+0000 7f7f99600640 1 -- 172.21.15.40:0/1497880440 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f9c07e490 con 0x558848be5770 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.771+0000 7f7f99600640 1 -- 172.21.15.40:0/1497880440 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7f7fa4025020 con 0x558848be5770 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.771+0000 7f7fb61ee740 10 client.7791 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:25.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.771+0000 7f7fb61ee740 10 client.7791 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:07:25.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.771+0000 7f7fb61ee740 20 client.7791 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:25.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.771+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558848c5f9c0 0x558848bf0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.771+0000 7f7fb61ee740 4 client.7791 mds_command: new command op to 7443 tid=0 multi_id=0 [{"prefix": "scrub abort"}] 2024-04-26T23:07:25.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.771+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1497880440 --> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] -- command(tid 0: {"prefix": "scrub abort"}) -- 0x558848b082f0 con 0x558848c5f9c0 2024-04-26T23:07:25.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.771+0000 7f7fabe00640 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558848c5f9c0 0x558848bf0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7fabe00640 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558848c5f9c0 0x558848bf0220 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=71e8e97e9cd0f727 server_cookie=72b2200ccde5a8a7 in_seq=0 out_seq=0 2024-04-26T23:07:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7f99600640 10 client.7791 ms_handle_connect on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7f99600640 1 -- 172.21.15.40:0/1497880440 <== mds.1 v2:172.21.15.40:6832/216893598 1 ==== command_reply(tid 0: -18 Not rank 0) ==== 18+0+0 (crc 0 0 0) 0x558848b082f0 con 0x558848c5f9c0 2024-04-26T23:07:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7f99600640 10 client.7791 handle_command_reply: tid=0 2024-04-26T23:07:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7fb61ee740 1 client.7791 shutdown 2024-04-26T23:07:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1497880440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848be5770 msgr2=0x558848b9cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848be5770 0x558848b9cca0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4009f50 tx=0x7f7fa40044d0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1497880440 shutdown_connections 2024-04-26T23:07:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x558848c5f9c0 0x558848bf0220 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558848be5770 0x558848b9cca0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558848b9c0a0 0x558848b9c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7f99600640 0 client.7791 ms_handle_reset on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7fb61ee740 1 --2- 172.21.15.40:0/1497880440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558848b9b1e0 0x558848c5c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.772+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1497880440 >> 172.21.15.40:0/1497880440 conn(0x558848b96f30 msgr2=0x558848c05a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.773+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1497880440 shutdown_connections 2024-04-26T23:07:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.773+0000 7f7fb61ee740 1 -- 172.21.15.40:0/1497880440 wait complete. 2024-04-26T23:07:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.773+0000 7f7fb61ee740 20 client.7791 trim_cache size 0 max 16384 2024-04-26T23:07:25.776 INFO:teuthology.orchestra.run.smithi040.stderr:Error EXDEV: Not rank 0 2024-04-26T23:07:25.780 DEBUG:teuthology.orchestra.run:got remote process result: 18 2024-04-26T23:07:25.781 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a scrub pause 2024-04-26T23:07:25.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.973+0000 7f16d9c00640 1 Processor -- start 2024-04-26T23:07:25.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.973+0000 7f16d9c00640 1 -- start start 2024-04-26T23:07:25.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.975+0000 7f16d9c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16d4071d50 0x7f16d40ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.975+0000 7f16d9c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f16d40ce5e0 0x7f16d40cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.975+0000 7f16d9c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d40b2470 con 0x7f16d40cf600 2024-04-26T23:07:25.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.975+0000 7f16d9c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f16d4071670 con 0x7f16d40ce5e0 2024-04-26T23:07:25.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.975+0000 7f16d9c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f16d40d3b90 con 0x7f16d4071d50 2024-04-26T23:07:25.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.975+0000 7f16d9200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f16d40ce5e0 0x7f16d40cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.975+0000 7f16d3400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16d4071d50 0x7f16d40ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.976+0000 7f16d9200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f16d40ce5e0 0x7f16d40cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:42186/0 (socket says 172.21.15.40:42186) 2024-04-26T23:07:25.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.976+0000 7f16d9200640 1 -- 172.21.15.40:0/2508308524 learned_addr learned my addr 172.21.15.40:0/2508308524 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.976+0000 7f16d3400640 1 -- 172.21.15.40:0/2508308524 >> v2:172.21.15.103:3300/0 conn(0x7f16d40ce5e0 msgr2=0x7f16d40cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.976+0000 7f16d3400640 1 --2- 172.21.15.40:0/2508308524 >> v2:172.21.15.103:3300/0 conn(0x7f16d40ce5e0 0x7f16d40cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.976+0000 7f16d3400640 1 -- 172.21.15.40:0/2508308524 >> v1:172.21.15.103:6789/0 conn(0x7f16d40cf600 legacy=0x7f16d40cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.976+0000 7f16d3400640 1 -- 172.21.15.40:0/2508308524 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d40cd620 con 0x7f16d4071d50 2024-04-26T23:07:25.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.978+0000 7f16d3400640 1 --2- 172.21.15.40:0/2508308524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16d4071d50 0x7f16d40ce0a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009d90 tx=0x7f16c002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=105d815c943b96a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.978+0000 7f16d2a00640 1 -- 172.21.15.40:0/2508308524 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0007740 con 0x7f16d4071d50 2024-04-26T23:07:25.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.978+0000 7f16d2a00640 1 -- 172.21.15.40:0/2508308524 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c00078e0 con 0x7f16d4071d50 2024-04-26T23:07:25.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.978+0000 7f16d2a00640 1 -- 172.21.15.40:0/2508308524 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0007cb0 con 0x7f16d4071d50 2024-04-26T23:07:25.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.979+0000 7f16d9c00640 1 -- 172.21.15.40:0/2508308524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16d4071d50 msgr2=0x7f16d40ce0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.979+0000 7f16d9c00640 1 --2- 172.21.15.40:0/2508308524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16d4071d50 0x7f16d40ce0a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009d90 tx=0x7f16c002a660 comp rx=0 tx=0).stop 2024-04-26T23:07:25.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.979+0000 7f16d9c00640 1 -- 172.21.15.40:0/2508308524 shutdown_connections 2024-04-26T23:07:25.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.979+0000 7f16d9c00640 1 --2- 172.21.15.40:0/2508308524 >> v2:172.21.15.103:3300/0 conn(0x7f16d40ce5e0 0x7f16d40cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.979+0000 7f16d9c00640 1 --2- 172.21.15.40:0/2508308524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16d4071d50 0x7f16d40ce0a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.979+0000 7f16d9c00640 1 -- 172.21.15.40:0/2508308524 >> 172.21.15.40:0/2508308524 conn(0x7f16d40d2bf0 msgr2=0x7f16d40d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:25.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.979+0000 7f16d9c00640 1 -- 172.21.15.40:0/2508308524 shutdown_connections 2024-04-26T23:07:25.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.980+0000 7f16d9c00640 1 -- 172.21.15.40:0/2508308524 wait complete. 2024-04-26T23:07:25.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.981+0000 7f16d9c00640 1 Processor -- start 2024-04-26T23:07:25.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.982+0000 7f16d9c00640 1 -- start start 2024-04-26T23:07:25.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.982+0000 7f16d9c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16d4071d50 0x7f16d409e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.982+0000 7f16d9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16d40ce5e0 0x7f16d41f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d9c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16d40cf600 0x7f16d41f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d9c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f16d40cdb80 con 0x7f16d40ce5e0 2024-04-26T23:07:25.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d9c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f16d40cd8a0 con 0x7f16d40cf600 2024-04-26T23:07:25.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d9c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f16d4160940 con 0x7f16d4071d50 2024-04-26T23:07:25.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16d40ce5e0 0x7f16d41f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16d40ce5e0 0x7f16d41f6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41274/0 (socket says 172.21.15.40:41274) 2024-04-26T23:07:25.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d3400640 1 -- 172.21.15.40:0/4049112663 learned_addr learned my addr 172.21.15.40:0/4049112663 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d3e00640 1 --2- 172.21.15.40:0/4049112663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16d4071d50 0x7f16d409e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d9200640 1 --2- 172.21.15.40:0/4049112663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16d40cf600 0x7f16d41f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d3400640 1 -- 172.21.15.40:0/4049112663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16d4071d50 msgr2=0x7f16d409e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d3400640 1 --2- 172.21.15.40:0/4049112663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16d4071d50 0x7f16d409e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d3400640 1 -- 172.21.15.40:0/4049112663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16d40cf600 msgr2=0x7f16d41f67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d3400640 1 --2- 172.21.15.40:0/4049112663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16d40cf600 0x7f16d41f67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.983+0000 7f16d3400640 1 -- 172.21.15.40:0/4049112663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d409f050 con 0x7f16d40ce5e0 2024-04-26T23:07:25.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.984+0000 7f16d3400640 1 --2- 172.21.15.40:0/4049112663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16d40ce5e0 0x7f16d41f6290 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009eb0 tx=0x7f16c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.984+0000 7f16cbe00640 1 -- 172.21.15.40:0/4049112663 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0040070 con 0x7f16d40ce5e0 2024-04-26T23:07:25.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.984+0000 7f16cbe00640 1 -- 172.21.15.40:0/4049112663 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c0031c40 con 0x7f16d40ce5e0 2024-04-26T23:07:25.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.984+0000 7f16cbe00640 1 -- 172.21.15.40:0/4049112663 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0045460 con 0x7f16d40ce5e0 2024-04-26T23:07:25.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.984+0000 7f16d9c00640 1 -- 172.21.15.40:0/4049112663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d409f330 con 0x7f16d40ce5e0 2024-04-26T23:07:25.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.984+0000 7f16d9c00640 1 -- 172.21.15.40:0/4049112663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d409f8f0 con 0x7f16d40ce5e0 2024-04-26T23:07:25.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.987+0000 7f16cbe00640 1 -- 172.21.15.40:0/4049112663 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f16c0039040 con 0x7f16d40ce5e0 2024-04-26T23:07:25.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.988+0000 7f16cbe00640 1 --2- 172.21.15.40:0/4049112663 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f16b406c940 0x7f16b406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.988+0000 7f16cbe00640 1 -- 172.21.15.40:0/4049112663 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f16c00ad550 con 0x7f16d40ce5e0 2024-04-26T23:07:25.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.988+0000 7f16ddfd7740 1 Processor -- start 2024-04-26T23:07:25.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.988+0000 7f16d3e00640 1 --2- 172.21.15.40:0/4049112663 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f16b406c940 0x7f16b406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.988+0000 7f16ddfd7740 1 -- start start 2024-04-26T23:07:25.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.989+0000 7f16ddfd7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 0x55bb5d315480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.989+0000 7f16ddfd7740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d30ff30 0x55bb5d313ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.989+0000 7f16ddfd7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3141e0 0x55bb5d314620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.989+0000 7f16ddfd7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bb5d279870 con 0x55bb5d3150a0 2024-04-26T23:07:25.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.989+0000 7f16ddfd7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bb5d215e40 con 0x55bb5d3141e0 2024-04-26T23:07:25.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.989+0000 7f16ddfd7740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bb5d275830 con 0x55bb5d30ff30 2024-04-26T23:07:25.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.990+0000 7f16d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 0x55bb5d315480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.990+0000 7f16d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 0x55bb5d315480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41288/0 (socket says 172.21.15.40:41288) 2024-04-26T23:07:25.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.990+0000 7f16d3e00640 1 -- 172.21.15.40:0/859406502 learned_addr learned my addr 172.21.15.40:0/859406502 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.990+0000 7f16d3e00640 1 --2- 172.21.15.40:0/4049112663 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f16b406c940 0x7f16b406ee00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f16c4007fd0 tx=0x7f16c4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.990+0000 7f16d9200640 1 --2- 172.21.15.40:0/859406502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3141e0 0x55bb5d314620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.990+0000 7f16d3400640 1 --2- 172.21.15.40:0/859406502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d30ff30 0x55bb5d313ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.990+0000 7f16d3e00640 1 -- 172.21.15.40:0/859406502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d30ff30 msgr2=0x55bb5d313ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.990+0000 7f16d3e00640 1 --2- 172.21.15.40:0/859406502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d30ff30 0x55bb5d313ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.990+0000 7f16d3e00640 1 -- 172.21.15.40:0/859406502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3141e0 msgr2=0x55bb5d314620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.990+0000 7f16d3e00640 1 --2- 172.21.15.40:0/859406502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3141e0 0x55bb5d314620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.990+0000 7f16d3e00640 1 -- 172.21.15.40:0/859406502 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb5d23f080 con 0x55bb5d3150a0 2024-04-26T23:07:25.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.990+0000 7f16d9200640 1 --2- 172.21.15.40:0/859406502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3141e0 0x55bb5d314620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:25.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.991+0000 7f16d3e00640 1 --2- 172.21.15.40:0/859406502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 0x55bb5d315480 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f16c40146f0 tx=0x7f16c4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded5ff85eba0e3cc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.991+0000 7f16c8c00640 1 -- 172.21.15.40:0/859406502 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4025070 con 0x55bb5d3150a0 2024-04-26T23:07:25.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.991+0000 7f16c8c00640 1 -- 172.21.15.40:0/859406502 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c4018030 con 0x55bb5d3150a0 2024-04-26T23:07:25.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.991+0000 7f16c8c00640 1 -- 172.21.15.40:0/859406502 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c401a050 con 0x55bb5d3150a0 2024-04-26T23:07:25.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.992+0000 7f16ddfd7740 1 -- 172.21.15.40:0/859406502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 msgr2=0x55bb5d315480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:25.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.992+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/859406502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 0x55bb5d315480 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f16c40146f0 tx=0x7f16c4011000 comp rx=0 tx=0).stop 2024-04-26T23:07:25.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.992+0000 7f16ddfd7740 1 -- 172.21.15.40:0/859406502 shutdown_connections 2024-04-26T23:07:25.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.992+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/859406502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3141e0 0x55bb5d314620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.992+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/859406502 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d30ff30 0x55bb5d313ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.992+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/859406502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 0x55bb5d315480 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.992+0000 7f16ddfd7740 1 -- 172.21.15.40:0/859406502 >> 172.21.15.40:0/859406502 conn(0x55bb5d3121a0 msgr2=0x55bb5d35ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:25.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.992+0000 7f16ddfd7740 1 -- 172.21.15.40:0/859406502 shutdown_connections 2024-04-26T23:07:25.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.992+0000 7f16ddfd7740 1 -- 172.21.15.40:0/859406502 wait complete. 2024-04-26T23:07:25.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.992+0000 7f16ddfd7740 1 -- start start 2024-04-26T23:07:25.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.993+0000 7f16ddfd7740 1 Processor -- start 2024-04-26T23:07:25.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.994+0000 7f16ddfd7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 0x55bb5d372250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.994+0000 7f16ddfd7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3150a0 0x55bb5d35be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.994+0000 7f16ddfd7740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d342550 0x55bb5d3429d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.994+0000 7f16ddfd7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bb5d340c30 con 0x55bb5d3141e0 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.994+0000 7f16ddfd7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bb5d200d90 con 0x55bb5d3150a0 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.994+0000 7f16ddfd7740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bb5d3400b0 con 0x55bb5d342550 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.994+0000 7f16ddfd7740 10 client.0 _ll_register_callbacks cb 0x55bb5d1fc680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.994+0000 7f16d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 0x55bb5d372250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.994+0000 7f16d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 0x55bb5d372250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41300/0 (socket says 172.21.15.40:41300) 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.994+0000 7f16d3e00640 1 -- 172.21.15.40:0/2757719577 learned_addr learned my addr 172.21.15.40:0/2757719577 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.994+0000 7f16d3400640 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3150a0 0x55bb5d35be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.995+0000 7f16d9200640 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d342550 0x55bb5d3429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.995+0000 7f16d3e00640 1 -- 172.21.15.40:0/2757719577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d342550 msgr2=0x55bb5d3429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.995+0000 7f16d3e00640 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d342550 0x55bb5d3429d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.995+0000 7f16d3e00640 1 -- 172.21.15.40:0/2757719577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3150a0 msgr2=0x55bb5d35be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.995+0000 7f16d3e00640 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3150a0 0x55bb5d35be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.995+0000 7f16d3e00640 1 -- 172.21.15.40:0/2757719577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb5d055500 con 0x55bb5d3141e0 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.995+0000 7f16d9200640 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d342550 0x55bb5d3429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.995+0000 7f16d3400640 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3150a0 0x55bb5d35be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.995+0000 7f16d3e00640 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 0x55bb5d372250 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f16c4023040 tx=0x7f16c4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d8c15fb19fd30a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.995+0000 7f16bd600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.995+0000 7f16bd600640 1 -- 172.21.15.40:0/2757719577 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4025040 con 0x55bb5d3141e0 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.995+0000 7f16bd600640 1 -- 172.21.15.40:0/2757719577 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f16c400eb60 con 0x55bb5d3141e0 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.995+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2757719577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16c4018030 con 0x55bb5d3141e0 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.996+0000 7f16bd600640 1 -- 172.21.15.40:0/2757719577 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f16c4033050 con 0x55bb5d3141e0 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.996+0000 7f16ddfd7740 10 client.7800 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.996+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2757719577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x55bb5d0b5c90 con 0x55bb5d3141e0 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.996+0000 7f16bd600640 1 -- 172.21.15.40:0/2757719577 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c4018030 con 0x55bb5d3141e0 2024-04-26T23:07:25.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.996+0000 7f16bd600640 1 -- 172.21.15.40:0/2757719577 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7f16c4036020 con 0x55bb5d3141e0 2024-04-26T23:07:25.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.996+0000 7f16ddfd7740 10 client.7800 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:25.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.996+0000 7f16ddfd7740 10 client.7800 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:07:25.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.996+0000 7f16ddfd7740 20 client.7800 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:25.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.996+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55bb5d335f30 0x55bb5d35e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:25.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.996+0000 7f16ddfd7740 4 client.7800 mds_command: new command op to 7443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:07:25.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.996+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2757719577 --> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb5d287480 con 0x55bb5d335f30 2024-04-26T23:07:25.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.997+0000 7f16d3400640 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55bb5d335f30 0x55bb5d35e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.997+0000 7f16d3400640 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55bb5d335f30 0x55bb5d35e930 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ff50e272b1462645 server_cookie=f84644b3b97bc8d6 in_seq=0 out_seq=0 2024-04-26T23:07:26.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.997+0000 7f16bd600640 10 client.7800 ms_handle_connect on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:26.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.998+0000 7f16bd600640 1 -- 172.21.15.40:0/2757719577 <== mds.1 v2:172.21.15.40:6832/216893598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb5d287480 con 0x55bb5d335f30 2024-04-26T23:07:26.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.998+0000 7f16bd600640 10 client.7800 handle_command_reply: tid=0 2024-04-26T23:07:26.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.998+0000 7f16ddfd7740 1 client.7800 shutdown 2024-04-26T23:07:26.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.998+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2757719577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 msgr2=0x55bb5d372250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.998+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 0x55bb5d372250 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f16c4023040 tx=0x7f16c4012000 comp rx=0 tx=0).stop 2024-04-26T23:07:26.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.998+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2757719577 shutdown_connections 2024-04-26T23:07:26.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.998+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55bb5d335f30 0x55bb5d35e930 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.998+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d342550 0x55bb5d3429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.998+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3150a0 0x55bb5d35be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.998+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/2757719577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 0x55bb5d372250 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.998+0000 7f16bd600640 0 client.7800 ms_handle_reset on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:26.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.998+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2757719577 >> 172.21.15.40:0/2757719577 conn(0x55bb5d30ff30 msgr2=0x55bb5d35a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:26.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.999+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2757719577 shutdown_connections 2024-04-26T23:07:26.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.999+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2757719577 wait complete. 2024-04-26T23:07:26.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:25.999+0000 7f16ddfd7740 20 client.7800 trim_cache size 0 max 16384 2024-04-26T23:07:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.018+0000 7f16ddfd7740 1 Processor -- start 2024-04-26T23:07:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.018+0000 7f16ddfd7740 1 -- start start 2024-04-26T23:07:26.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.019+0000 7f16ddfd7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 0x55bb5d35f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.019+0000 7f16ddfd7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3150a0 0x55bb5d3d2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.019+0000 7f16ddfd7740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d342550 0x55bb5d35d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.019+0000 7f16ddfd7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f16d40c8230 con 0x55bb5d3141e0 2024-04-26T23:07:26.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.019+0000 7f16ddfd7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f16d406f210 con 0x55bb5d3150a0 2024-04-26T23:07:26.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.019+0000 7f16ddfd7740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f16d40c7b00 con 0x55bb5d342550 2024-04-26T23:07:26.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.019+0000 7f16d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 0x55bb5d35f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.019+0000 7f16d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 0x55bb5d35f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41302/0 (socket says 172.21.15.40:41302) 2024-04-26T23:07:26.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.019+0000 7f16d3e00640 1 -- 172.21.15.40:0/1860526623 learned_addr learned my addr 172.21.15.40:0/1860526623 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:26.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.020+0000 7f16d3e00640 1 -- 172.21.15.40:0/1860526623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d342550 msgr2=0x55bb5d35d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.020+0000 7f16d3e00640 1 --2- 172.21.15.40:0/1860526623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d342550 0x55bb5d35d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.020+0000 7f16d3e00640 1 -- 172.21.15.40:0/1860526623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3150a0 msgr2=0x55bb5d3d2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.020+0000 7f16d3e00640 1 --2- 172.21.15.40:0/1860526623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3150a0 0x55bb5d3d2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.020+0000 7f16d3e00640 1 -- 172.21.15.40:0/1860526623 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb5cf4ae20 con 0x55bb5d3141e0 2024-04-26T23:07:26.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.020+0000 7f16d3e00640 1 --2- 172.21.15.40:0/1860526623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 0x55bb5d35f8b0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f16c4007b60 tx=0x7f16c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdfb50d32504b9d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:26.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.020+0000 7f16bfe00640 1 -- 172.21.15.40:0/1860526623 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c401a070 con 0x55bb5d3141e0 2024-04-26T23:07:26.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.020+0000 7f16bfe00640 1 -- 172.21.15.40:0/1860526623 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c4018040 con 0x55bb5d3141e0 2024-04-26T23:07:26.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.020+0000 7f16bfe00640 1 -- 172.21.15.40:0/1860526623 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4003900 con 0x55bb5d3141e0 2024-04-26T23:07:26.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.021+0000 7f16ddfd7740 1 -- 172.21.15.40:0/1860526623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 msgr2=0x55bb5d35f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.021+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/1860526623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 0x55bb5d35f8b0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f16c4007b60 tx=0x7f16c4017000 comp rx=0 tx=0).stop 2024-04-26T23:07:26.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.021+0000 7f16ddfd7740 1 -- 172.21.15.40:0/1860526623 shutdown_connections 2024-04-26T23:07:26.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.021+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/1860526623 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d342550 0x55bb5d35d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.021+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/1860526623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d3150a0 0x55bb5d3d2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.021+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/1860526623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3141e0 0x55bb5d35f8b0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.021+0000 7f16ddfd7740 1 -- 172.21.15.40:0/1860526623 >> 172.21.15.40:0/1860526623 conn(0x55bb5d30ff30 msgr2=0x55bb5d357510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:26.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.021+0000 7f16ddfd7740 1 -- 172.21.15.40:0/1860526623 shutdown_connections 2024-04-26T23:07:26.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.021+0000 7f16ddfd7740 1 -- 172.21.15.40:0/1860526623 wait complete. 2024-04-26T23:07:26.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.022+0000 7f16ddfd7740 1 -- start start 2024-04-26T23:07:26.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.022+0000 7f16ddfd7740 1 Processor -- start 2024-04-26T23:07:26.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.023+0000 7f16ddfd7740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d3141e0 0x55bb5d3d2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.023+0000 7f16ddfd7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 0x55bb5d30ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.023+0000 7f16ddfd7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d342550 0x55bb5d30f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.023+0000 7f16ddfd7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bb5d370700 con 0x55bb5d3150a0 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.023+0000 7f16ddfd7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f16d4152500 con 0x55bb5d342550 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.023+0000 7f16ddfd7740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bb5d3363d0 con 0x55bb5d3141e0 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.023+0000 7f16ddfd7740 10 client.0 _ll_register_callbacks cb 0x7f16c4023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.023+0000 7f16d3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 0x55bb5d30ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.023+0000 7f16d3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 0x55bb5d30ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41304/0 (socket says 172.21.15.40:41304) 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.023+0000 7f16d3400640 1 -- 172.21.15.40:0/2341341913 learned_addr learned my addr 172.21.15.40:0/2341341913 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.024+0000 7f16d3e00640 1 --2- 172.21.15.40:0/2341341913 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d3141e0 0x55bb5d3d2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.024+0000 7f16d3400640 1 -- 172.21.15.40:0/2341341913 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d3141e0 msgr2=0x55bb5d3d2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.024+0000 7f16d3400640 1 --2- 172.21.15.40:0/2341341913 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d3141e0 0x55bb5d3d2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.024+0000 7f16d3400640 1 -- 172.21.15.40:0/2341341913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d342550 msgr2=0x55bb5d30f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.024+0000 7f16d9200640 1 --2- 172.21.15.40:0/2341341913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d342550 0x55bb5d30f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.024+0000 7f16d3400640 1 --2- 172.21.15.40:0/2341341913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d342550 0x55bb5d30f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.024+0000 7f16d3400640 1 -- 172.21.15.40:0/2341341913 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb5d356c30 con 0x55bb5d3150a0 2024-04-26T23:07:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.024+0000 7f16d3400640 1 --2- 172.21.15.40:0/2341341913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 0x55bb5d30ef00 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f16c007ae60 tx=0x7f16c0009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc5e8d9775ef750 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.024+0000 7f16bd600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16bd600640 1 -- 172.21.15.40:0/2341341913 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c007f510 con 0x55bb5d3150a0 2024-04-26T23:07:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16bd600640 1 -- 172.21.15.40:0/2341341913 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f16c00ab9e0 con 0x55bb5d3150a0 2024-04-26T23:07:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2341341913 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16c4018040 con 0x55bb5d3150a0 2024-04-26T23:07:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16bd600640 1 -- 172.21.15.40:0/2341341913 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f16c0098050 con 0x55bb5d3150a0 2024-04-26T23:07:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16ddfd7740 10 client.7806 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2341341913 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x55bb5d33a080 con 0x55bb5d3150a0 2024-04-26T23:07:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16bd600640 1 -- 172.21.15.40:0/2341341913 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c4018040 con 0x55bb5d3150a0 2024-04-26T23:07:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16bd600640 1 -- 172.21.15.40:0/2341341913 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7f16c00777d0 con 0x55bb5d3150a0 2024-04-26T23:07:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16ddfd7740 10 client.7806 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16ddfd7740 10 client.7806 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16ddfd7740 20 client.7806 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/2341341913 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55bb5d339000 0x55bb5d3393e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16ddfd7740 4 client.7806 mds_command: new command op to 7443 tid=0 multi_id=0 [{"prefix": "scrub pause"}] 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.025+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2341341913 --> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] -- command(tid 0: {"prefix": "scrub pause"}) -- 0x55bb5d357fd0 con 0x55bb5d339000 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.026+0000 7f16d3e00640 1 --2- 172.21.15.40:0/2341341913 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55bb5d339000 0x55bb5d3393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.026+0000 7f16d3e00640 1 --2- 172.21.15.40:0/2341341913 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55bb5d339000 0x55bb5d3393e0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3da64b716aada5d8 server_cookie=55f05a80d5cb8b0a in_seq=0 out_seq=0 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.026+0000 7f16bd600640 10 client.7806 ms_handle_connect on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.026+0000 7f16bd600640 1 -- 172.21.15.40:0/2341341913 <== mds.1 v2:172.21.15.40:6832/216893598 1 ==== command_reply(tid 0: -18 Not rank 0) ==== 18+0+0 (crc 0 0 0) 0x55bb5d357fd0 con 0x55bb5d339000 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.026+0000 7f16bd600640 10 client.7806 handle_command_reply: tid=0 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.026+0000 7f16ddfd7740 1 client.7806 shutdown 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.026+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2341341913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 msgr2=0x55bb5d30ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.026+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/2341341913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 0x55bb5d30ef00 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f16c007ae60 tx=0x7f16c0009a70 comp rx=0 tx=0).stop 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.027+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2341341913 shutdown_connections 2024-04-26T23:07:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.027+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/2341341913 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x55bb5d339000 0x55bb5d3393e0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.027+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/2341341913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5d342550 0x55bb5d30f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.027+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/2341341913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb5d3150a0 0x55bb5d30ef00 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.027+0000 7f16ddfd7740 1 --2- 172.21.15.40:0/2341341913 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bb5d3141e0 0x55bb5d3d2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.027+0000 7f16bd600640 0 client.7806 ms_handle_reset on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:26.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.027+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2341341913 >> 172.21.15.40:0/2341341913 conn(0x55bb5d30ff30 msgr2=0x55bb5d35b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:26.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.027+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2341341913 shutdown_connections 2024-04-26T23:07:26.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.027+0000 7f16ddfd7740 1 -- 172.21.15.40:0/2341341913 wait complete. 2024-04-26T23:07:26.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.027+0000 7f16ddfd7740 20 client.7806 trim_cache size 0 max 16384 2024-04-26T23:07:26.030 INFO:teuthology.orchestra.run.smithi040.stderr:Error EXDEV: Not rank 0 2024-04-26T23:07:26.035 DEBUG:teuthology.orchestra.run:got remote process result: 18 2024-04-26T23:07:26.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a scrub resume 2024-04-26T23:07:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.230+0000 7f323fe00640 1 Processor -- start 2024-04-26T23:07:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.231+0000 7f323fe00640 1 -- start start 2024-04-26T23:07:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.232+0000 7f323fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3240160a50 0x7f3240073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.232+0000 7f323fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f32400cfaf0 0x7f3240160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.232+0000 7f323fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f324006dd40 0x7f3240072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.232+0000 7f323fe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f3240071ce0 con 0x7f324006dd40 2024-04-26T23:07:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.232+0000 7f323fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f32400722a0 con 0x7f32400cfaf0 2024-04-26T23:07:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.232+0000 7f323fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3240073f90 con 0x7f3240160a50 2024-04-26T23:07:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.232+0000 7f323f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f324006dd40 0x7f3240072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.232+0000 7f323f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f324006dd40 0x7f3240072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41308/0 (socket says 172.21.15.40:41308) 2024-04-26T23:07:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.233+0000 7f323f400640 1 -- 172.21.15.40:0/4124417415 learned_addr learned my addr 172.21.15.40:0/4124417415 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.233+0000 7f323e000640 1 --2- 172.21.15.40:0/4124417415 >> v2:172.21.15.103:3300/0 conn(0x7f32400cfaf0 0x7f3240160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.233+0000 7f323ea00640 1 --2- 172.21.15.40:0/4124417415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3240160a50 0x7f3240073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.233+0000 7f323f400640 1 -- 172.21.15.40:0/4124417415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3240160a50 msgr2=0x7f3240073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.233+0000 7f323f400640 1 --2- 172.21.15.40:0/4124417415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3240160a50 0x7f3240073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.233+0000 7f323f400640 1 -- 172.21.15.40:0/4124417415 >> v2:172.21.15.103:3300/0 conn(0x7f32400cfaf0 msgr2=0x7f3240160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.233+0000 7f323f400640 1 --2- 172.21.15.40:0/4124417415 >> v2:172.21.15.103:3300/0 conn(0x7f32400cfaf0 0x7f3240160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.233+0000 7f323f400640 1 -- 172.21.15.40:0/4124417415 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3240161880 con 0x7f324006dd40 2024-04-26T23:07:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.234+0000 7f323f400640 1 --2- 172.21.15.40:0/4124417415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f324006dd40 0x7f3240072730 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f3240074700 tx=0x7f323802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11024527140b455f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:26.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.234+0000 7f323d600640 1 -- 172.21.15.40:0/4124417415 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3238036070 con 0x7f324006dd40 2024-04-26T23:07:26.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.234+0000 7f323d600640 1 -- 172.21.15.40:0/4124417415 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3238031600 con 0x7f324006dd40 2024-04-26T23:07:26.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.234+0000 7f323d600640 1 -- 172.21.15.40:0/4124417415 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32380319d0 con 0x7f324006dd40 2024-04-26T23:07:26.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.235+0000 7f323fe00640 1 -- 172.21.15.40:0/4124417415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f324006dd40 msgr2=0x7f3240072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.235+0000 7f323fe00640 1 --2- 172.21.15.40:0/4124417415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f324006dd40 0x7f3240072730 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f3240074700 tx=0x7f323802a6a0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.235+0000 7f323fe00640 1 -- 172.21.15.40:0/4124417415 shutdown_connections 2024-04-26T23:07:26.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.235+0000 7f323fe00640 1 --2- 172.21.15.40:0/4124417415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f324006dd40 0x7f3240072730 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.235+0000 7f323fe00640 1 --2- 172.21.15.40:0/4124417415 >> v2:172.21.15.103:3300/0 conn(0x7f32400cfaf0 0x7f3240160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.235+0000 7f323fe00640 1 --2- 172.21.15.40:0/4124417415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3240160a50 0x7f3240073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.235+0000 7f323fe00640 1 -- 172.21.15.40:0/4124417415 >> 172.21.15.40:0/4124417415 conn(0x7f32400b2720 msgr2=0x7f32400b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:26.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.235+0000 7f323fe00640 1 -- 172.21.15.40:0/4124417415 shutdown_connections 2024-04-26T23:07:26.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.235+0000 7f323fe00640 1 -- 172.21.15.40:0/4124417415 wait complete. 2024-04-26T23:07:26.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.237+0000 7f323fe00640 1 Processor -- start 2024-04-26T23:07:26.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.237+0000 7f323fe00640 1 -- start start 2024-04-26T23:07:26.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.238+0000 7f323fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f324006dd40 0x7f324015b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.238+0000 7f323fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3240160a50 0x7f32400b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.238+0000 7f323fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f324013cd20 0x7f324015a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.238+0000 7f323fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32400d0260 con 0x7f324013cd20 2024-04-26T23:07:26.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.238+0000 7f323fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32400cff80 con 0x7f3240160a50 2024-04-26T23:07:26.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.238+0000 7f323fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3240071f30 con 0x7f324006dd40 2024-04-26T23:07:26.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.239+0000 7f323f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f324013cd20 0x7f324015a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.239+0000 7f323f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f324013cd20 0x7f324015a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41312/0 (socket says 172.21.15.40:41312) 2024-04-26T23:07:26.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.239+0000 7f323f400640 1 -- 172.21.15.40:0/2468331790 learned_addr learned my addr 172.21.15.40:0/2468331790 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:26.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.239+0000 7f323ea00640 1 --2- 172.21.15.40:0/2468331790 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f324006dd40 0x7f324015b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.239+0000 7f323e000640 1 --2- 172.21.15.40:0/2468331790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3240160a50 0x7f32400b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.239+0000 7f323f400640 1 -- 172.21.15.40:0/2468331790 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f324006dd40 msgr2=0x7f324015b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.239+0000 7f323f400640 1 --2- 172.21.15.40:0/2468331790 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f324006dd40 0x7f324015b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.239+0000 7f323f400640 1 -- 172.21.15.40:0/2468331790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3240160a50 msgr2=0x7f32400b1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.239+0000 7f323f400640 1 --2- 172.21.15.40:0/2468331790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3240160a50 0x7f32400b1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.239+0000 7f323f400640 1 -- 172.21.15.40:0/2468331790 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324015b960 con 0x7f324013cd20 2024-04-26T23:07:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.240+0000 7f323f400640 1 --2- 172.21.15.40:0/2468331790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f324013cd20 0x7f324015a470 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f3238034040 tx=0x7f3238004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.240+0000 7f3236a00640 1 -- 172.21.15.40:0/2468331790 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3238036050 con 0x7f324013cd20 2024-04-26T23:07:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.240+0000 7f3236a00640 1 -- 172.21.15.40:0/2468331790 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3238042070 con 0x7f324013cd20 2024-04-26T23:07:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.240+0000 7f3236a00640 1 -- 172.21.15.40:0/2468331790 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3238009510 con 0x7f324013cd20 2024-04-26T23:07:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.240+0000 7f323fe00640 1 -- 172.21.15.40:0/2468331790 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f324015bc40 con 0x7f324013cd20 2024-04-26T23:07:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.240+0000 7f323fe00640 1 -- 172.21.15.40:0/2468331790 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f324015c200 con 0x7f324013cd20 2024-04-26T23:07:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.244+0000 7f3236a00640 1 -- 172.21.15.40:0/2468331790 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3238007bd0 con 0x7f324013cd20 2024-04-26T23:07:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.245+0000 7f3236a00640 1 --2- 172.21.15.40:0/2468331790 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f322006c8f0 0x7f322006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.245+0000 7f3236a00640 1 -- 172.21.15.40:0/2468331790 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f32380acc40 con 0x7f324013cd20 2024-04-26T23:07:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.245+0000 7f32489ae740 1 Processor -- start 2024-04-26T23:07:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.245+0000 7f32489ae740 1 -- start start 2024-04-26T23:07:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.246+0000 7f323ea00640 1 --2- 172.21.15.40:0/2468331790 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f322006c8f0 0x7f322006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.246+0000 7f32489ae740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4b140 0x5571d4b4b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.246+0000 7f32489ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 0x5571d4b68f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.246+0000 7f32489ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b20e20 0x5571d4b670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.246+0000 7f32489ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5571d4a84900 con 0x5571d4b20e20 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.246+0000 7f32489ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5571d4a211d0 con 0x5571d4b689e0 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.247+0000 7f32489ae740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5571d4a808c0 con 0x5571d4b4b140 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.247+0000 7f323f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b20e20 0x5571d4b670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.247+0000 7f323f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b20e20 0x5571d4b670a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41314/0 (socket says 172.21.15.40:41314) 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.247+0000 7f323f400640 1 -- 172.21.15.40:0/827636357 learned_addr learned my addr 172.21.15.40:0/827636357 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.247+0000 7f323e000640 1 --2- 172.21.15.40:0/827636357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 0x5571d4b68f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.247+0000 7f323ea00640 1 --2- 172.21.15.40:0/827636357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4b140 0x5571d4b4b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.247+0000 7f323f400640 1 -- 172.21.15.40:0/827636357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4b140 msgr2=0x5571d4b4b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.247+0000 7f323f400640 1 --2- 172.21.15.40:0/827636357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4b140 0x5571d4b4b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.247+0000 7f323f400640 1 -- 172.21.15.40:0/827636357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 msgr2=0x5571d4b68f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.247+0000 7f323f400640 1 --2- 172.21.15.40:0/827636357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 0x5571d4b68f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.247+0000 7f323ea00640 1 --2- 172.21.15.40:0/2468331790 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f322006c8f0 0x7f322006edb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3230006fd0 tx=0x7f3230008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.247+0000 7f323f400640 1 -- 172.21.15.40:0/827636357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571d48607d0 con 0x5571d4b20e20 2024-04-26T23:07:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.248+0000 7f323f400640 1 --2- 172.21.15.40:0/827636357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b20e20 0x5571d4b670a0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f3238082df0 tx=0x7f3238076b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9356b0df7393e1c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.248+0000 7f322b400640 1 -- 172.21.15.40:0/827636357 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3238045360 con 0x5571d4b20e20 2024-04-26T23:07:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.248+0000 7f322b400640 1 -- 172.21.15.40:0/827636357 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3238037040 con 0x5571d4b20e20 2024-04-26T23:07:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.248+0000 7f322b400640 1 -- 172.21.15.40:0/827636357 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323807ae40 con 0x5571d4b20e20 2024-04-26T23:07:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.249+0000 7f32489ae740 1 -- 172.21.15.40:0/827636357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b20e20 msgr2=0x5571d4b670a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.249+0000 7f32489ae740 1 --2- 172.21.15.40:0/827636357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b20e20 0x5571d4b670a0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f3238082df0 tx=0x7f3238076b10 comp rx=0 tx=0).stop 2024-04-26T23:07:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.249+0000 7f32489ae740 1 -- 172.21.15.40:0/827636357 shutdown_connections 2024-04-26T23:07:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.249+0000 7f32489ae740 1 --2- 172.21.15.40:0/827636357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b20e20 0x5571d4b670a0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.249+0000 7f32489ae740 1 --2- 172.21.15.40:0/827636357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 0x5571d4b68f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.249+0000 7f32489ae740 1 --2- 172.21.15.40:0/827636357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4b140 0x5571d4b4b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.249+0000 7f32489ae740 1 -- 172.21.15.40:0/827636357 >> 172.21.15.40:0/827636357 conn(0x5571d4b1d750 msgr2=0x5571d4b65f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.249+0000 7f32489ae740 1 -- 172.21.15.40:0/827636357 shutdown_connections 2024-04-26T23:07:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.249+0000 7f32489ae740 1 -- 172.21.15.40:0/827636357 wait complete. 2024-04-26T23:07:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.249+0000 7f32489ae740 1 -- start start 2024-04-26T23:07:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.250+0000 7f32489ae740 1 Processor -- start 2024-04-26T23:07:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.251+0000 7f32489ae740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4b140 0x5571d4b4e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.251+0000 7f32489ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b689e0 0x5571d4b4a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.251+0000 7f32489ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b4a700 0x5571d4b76290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.251+0000 7f32489ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5571d4a211d0 con 0x5571d4b689e0 2024-04-26T23:07:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.251+0000 7f32489ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5571d4a0c120 con 0x5571d4b4a700 2024-04-26T23:07:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.251+0000 7f32489ae740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5571d4a84900 con 0x5571d4b4b140 2024-04-26T23:07:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.251+0000 7f32489ae740 10 client.0 _ll_register_callbacks cb 0x5571d4a44750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.251+0000 7f323e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b689e0 0x5571d4b4a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.251+0000 7f323e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b689e0 0x5571d4b4a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41328/0 (socket says 172.21.15.40:41328) 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.251+0000 7f323e000640 1 -- 172.21.15.40:0/1840884641 learned_addr learned my addr 172.21.15.40:0/1840884641 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.251+0000 7f323ea00640 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4b140 0x5571d4b4e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.251+0000 7f323f400640 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b4a700 0x5571d4b76290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.252+0000 7f323e000640 1 -- 172.21.15.40:0/1840884641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4b140 msgr2=0x5571d4b4e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.252+0000 7f323e000640 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4b140 0x5571d4b4e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.252+0000 7f323e000640 1 -- 172.21.15.40:0/1840884641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b4a700 msgr2=0x5571d4b76290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.252+0000 7f323e000640 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b4a700 0x5571d4b76290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.252+0000 7f323e000640 1 -- 172.21.15.40:0/1840884641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571d47560f0 con 0x5571d4b689e0 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.252+0000 7f323ea00640 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4b140 0x5571d4b4e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.252+0000 7f323e000640 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b689e0 0x5571d4b4a1c0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f322c011260 tx=0x7f322c011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a7450572af8e1a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.252+0000 7f321fe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:26.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.252+0000 7f321fe00640 1 -- 172.21.15.40:0/1840884641 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c019070 con 0x5571d4b689e0 2024-04-26T23:07:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.252+0000 7f321fe00640 1 -- 172.21.15.40:0/1840884641 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f322c015650 con 0x5571d4b689e0 2024-04-26T23:07:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.252+0000 7f32489ae740 1 -- 172.21.15.40:0/1840884641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3238037040 con 0x5571d4b689e0 2024-04-26T23:07:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.253+0000 7f321fe00640 1 -- 172.21.15.40:0/1840884641 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f322c024050 con 0x5571d4b689e0 2024-04-26T23:07:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.253+0000 7f32489ae740 10 client.7818 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.253+0000 7f32489ae740 1 -- 172.21.15.40:0/1840884641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x5571d48c0f60 con 0x5571d4b689e0 2024-04-26T23:07:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.253+0000 7f321fe00640 1 -- 172.21.15.40:0/1840884641 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3238037040 con 0x5571d4b689e0 2024-04-26T23:07:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.253+0000 7f321fe00640 1 -- 172.21.15.40:0/1840884641 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7f322c027020 con 0x5571d4b689e0 2024-04-26T23:07:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.253+0000 7f32489ae740 10 client.7818 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.253+0000 7f32489ae740 10 client.7818 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:07:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.253+0000 7f32489ae740 20 client.7818 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.253+0000 7f32489ae740 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x5571d4b67830 0x5571d4b84770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.253+0000 7f32489ae740 4 client.7818 mds_command: new command op to 7443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:07:26.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.253+0000 7f32489ae740 1 -- 172.21.15.40:0/1840884641 --> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571d4a92510 con 0x5571d4b67830 2024-04-26T23:07:26.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.253+0000 7f323ea00640 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x5571d4b67830 0x5571d4b84770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.254+0000 7f323ea00640 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x5571d4b67830 0x5571d4b84770 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bd86501b83d40a0b server_cookie=2a21529abae79eac in_seq=0 out_seq=0 2024-04-26T23:07:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.254+0000 7f321fe00640 10 client.7818 ms_handle_connect on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f321fe00640 1 -- 172.21.15.40:0/1840884641 <== mds.1 v2:172.21.15.40:6832/216893598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5571d4a92510 con 0x5571d4b67830 2024-04-26T23:07:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f321fe00640 10 client.7818 handle_command_reply: tid=0 2024-04-26T23:07:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f32489ae740 1 client.7818 shutdown 2024-04-26T23:07:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f32489ae740 1 -- 172.21.15.40:0/1840884641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b689e0 msgr2=0x5571d4b4a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f32489ae740 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b689e0 0x5571d4b4a1c0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f322c011260 tx=0x7f322c011720 comp rx=0 tx=0).stop 2024-04-26T23:07:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f32489ae740 1 -- 172.21.15.40:0/1840884641 shutdown_connections 2024-04-26T23:07:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f32489ae740 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x5571d4b67830 0x5571d4b84770 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f32489ae740 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b4a700 0x5571d4b76290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f32489ae740 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b689e0 0x5571d4b4a1c0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f32489ae740 1 --2- 172.21.15.40:0/1840884641 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4b140 0x5571d4b4e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f321fe00640 0 client.7818 ms_handle_reset on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:26.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f32489ae740 1 -- 172.21.15.40:0/1840884641 >> 172.21.15.40:0/1840884641 conn(0x5571d4b20e20 msgr2=0x5571d4b694d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:26.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f32489ae740 1 -- 172.21.15.40:0/1840884641 shutdown_connections 2024-04-26T23:07:26.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f32489ae740 1 -- 172.21.15.40:0/1840884641 wait complete. 2024-04-26T23:07:26.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.255+0000 7f32489ae740 20 client.7818 trim_cache size 0 max 16384 2024-04-26T23:07:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.274+0000 7f32489ae740 1 Processor -- start 2024-04-26T23:07:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.274+0000 7f32489ae740 1 -- start start 2024-04-26T23:07:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.275+0000 7f32489ae740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4a700 0x5571d4b16d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.275+0000 7f32489ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 0x5571d4b1b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.275+0000 7f32489ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 0x5571d4b4ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.275+0000 7f32489ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32401fda20 con 0x5571d4b4b140 2024-04-26T23:07:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.275+0000 7f32489ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32401fcbf0 con 0x5571d4b689e0 2024-04-26T23:07:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.275+0000 7f32489ae740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f32401fd2f0 con 0x5571d4b4a700 2024-04-26T23:07:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.276+0000 7f323e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 0x5571d4b1b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.276+0000 7f323e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 0x5571d4b1b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41332/0 (socket says 172.21.15.40:41332) 2024-04-26T23:07:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.276+0000 7f323e000640 1 -- 172.21.15.40:0/42685778 learned_addr learned my addr 172.21.15.40:0/42685778 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.276+0000 7f323e000640 1 -- 172.21.15.40:0/42685778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4a700 msgr2=0x5571d4b16d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.276+0000 7f323e000640 1 --2- 172.21.15.40:0/42685778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4a700 0x5571d4b16d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.276+0000 7f323e000640 1 -- 172.21.15.40:0/42685778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 msgr2=0x5571d4b4ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.276+0000 7f323e000640 1 --2- 172.21.15.40:0/42685778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 0x5571d4b4ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.276+0000 7f323e000640 1 -- 172.21.15.40:0/42685778 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571d472f1b0 con 0x5571d4b4b140 2024-04-26T23:07:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.276+0000 7f323e000640 1 --2- 172.21.15.40:0/42685778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 0x5571d4b1b5e0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f322c011c20 tx=0x7f322c009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a4d68e1d740731 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.277+0000 7f322aa00640 1 -- 172.21.15.40:0/42685778 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c019040 con 0x5571d4b4b140 2024-04-26T23:07:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.277+0000 7f322aa00640 1 -- 172.21.15.40:0/42685778 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f322c02d070 con 0x5571d4b4b140 2024-04-26T23:07:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.277+0000 7f322aa00640 1 -- 172.21.15.40:0/42685778 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c015a30 con 0x5571d4b4b140 2024-04-26T23:07:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.277+0000 7f32489ae740 1 -- 172.21.15.40:0/42685778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 msgr2=0x5571d4b1b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.277+0000 7f32489ae740 1 --2- 172.21.15.40:0/42685778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 0x5571d4b1b5e0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f322c011c20 tx=0x7f322c009000 comp rx=0 tx=0).stop 2024-04-26T23:07:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.277+0000 7f32489ae740 1 -- 172.21.15.40:0/42685778 shutdown_connections 2024-04-26T23:07:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.277+0000 7f32489ae740 1 --2- 172.21.15.40:0/42685778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 0x5571d4b4ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.277+0000 7f32489ae740 1 --2- 172.21.15.40:0/42685778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 0x5571d4b1b5e0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.277+0000 7f32489ae740 1 --2- 172.21.15.40:0/42685778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4a700 0x5571d4b16d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.277+0000 7f32489ae740 1 -- 172.21.15.40:0/42685778 >> 172.21.15.40:0/42685778 conn(0x5571d4b20e20 msgr2=0x5571d4b78c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.277+0000 7f32489ae740 1 -- 172.21.15.40:0/42685778 shutdown_connections 2024-04-26T23:07:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.277+0000 7f32489ae740 1 -- 172.21.15.40:0/42685778 wait complete. 2024-04-26T23:07:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.278+0000 7f32489ae740 1 -- start start 2024-04-26T23:07:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.278+0000 7f32489ae740 1 Processor -- start 2024-04-26T23:07:26.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.279+0000 7f32489ae740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4a700 0x5571d4bdfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.279+0000 7f32489ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 0x5571d4b4ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.279+0000 7f32489ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 0x5571d4b4ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.279+0000 7f32489ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5571d4b5d7c0 con 0x5571d4b4b140 2024-04-26T23:07:26.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.279+0000 7f32489ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32400b4bd0 con 0x5571d4b689e0 2024-04-26T23:07:26.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.279+0000 7f32489ae740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5571d4b77450 con 0x5571d4b4a700 2024-04-26T23:07:26.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.279+0000 7f32489ae740 10 client.0 _ll_register_callbacks cb 0x7f322c00b300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:26.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.280+0000 7f323e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 0x5571d4b4ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.280+0000 7f323e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 0x5571d4b4ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41338/0 (socket says 172.21.15.40:41338) 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.280+0000 7f323e000640 1 -- 172.21.15.40:0/3136101699 learned_addr learned my addr 172.21.15.40:0/3136101699 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.280+0000 7f323ea00640 1 --2- 172.21.15.40:0/3136101699 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4a700 0x5571d4bdfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.280+0000 7f323e000640 1 -- 172.21.15.40:0/3136101699 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4a700 msgr2=0x5571d4bdfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.280+0000 7f323e000640 1 --2- 172.21.15.40:0/3136101699 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4a700 0x5571d4bdfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.280+0000 7f323e000640 1 -- 172.21.15.40:0/3136101699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 msgr2=0x5571d4b4ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.280+0000 7f323f400640 1 --2- 172.21.15.40:0/3136101699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 0x5571d4b4ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.280+0000 7f323e000640 1 --2- 172.21.15.40:0/3136101699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 0x5571d4b4ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.280+0000 7f323e000640 1 -- 172.21.15.40:0/3136101699 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571d4b1c200 con 0x5571d4b4b140 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.280+0000 7f323e000640 1 --2- 172.21.15.40:0/3136101699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 0x5571d4b4ea20 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f322c006fd0 tx=0x7f322c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ba1b1d05acf86d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.280+0000 7f321fe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f321fe00640 1 -- 172.21.15.40:0/3136101699 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c036060 con 0x5571d4b4b140 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f321fe00640 1 -- 172.21.15.40:0/3136101699 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f322c003610 con 0x5571d4b4b140 2024-04-26T23:07:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f32489ae740 1 -- 172.21.15.40:0/3136101699 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f322c02d070 con 0x5571d4b4b140 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f321fe00640 1 -- 172.21.15.40:0/3136101699 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f322c03c050 con 0x5571d4b4b140 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f32489ae740 10 client.7824 fetch_fsmap learned FSMap version 81 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f32489ae740 1 -- 172.21.15.40:0/3136101699 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x5571d4b1bab0 con 0x5571d4b4b140 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f321fe00640 1 -- 172.21.15.40:0/3136101699 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f322c02d070 con 0x5571d4b4b140 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f321fe00640 1 -- 172.21.15.40:0/3136101699 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 81) ==== 3546+0+0 (secure 0 0 0) 0x7f322c03f020 con 0x5571d4b4b140 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f32489ae740 10 client.7824 fetch_fsmap finished waiting for FSMap version 81 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f32489ae740 10 client.7824 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f32489ae740 20 client.7824 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f32489ae740 1 --2- 172.21.15.40:0/3136101699 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x5571d4b5c320 0x5571d4b88790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f32489ae740 4 client.7824 mds_command: new command op to 7443 tid=0 multi_id=0 [{"prefix": "scrub resume"}] 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.281+0000 7f32489ae740 1 -- 172.21.15.40:0/3136101699 --> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] -- command(tid 0: {"prefix": "scrub resume"}) -- 0x5571d4a8c380 con 0x5571d4b5c320 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.282+0000 7f323ea00640 1 --2- 172.21.15.40:0/3136101699 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x5571d4b5c320 0x5571d4b88790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.282+0000 7f323ea00640 1 --2- 172.21.15.40:0/3136101699 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x5571d4b5c320 0x5571d4b88790 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8a6846bbde58ffdc server_cookie=16bd724c044046f2 in_seq=0 out_seq=0 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.282+0000 7f321fe00640 10 client.7824 ms_handle_connect on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:26.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.282+0000 7f321fe00640 1 -- 172.21.15.40:0/3136101699 <== mds.1 v2:172.21.15.40:6832/216893598 1 ==== command_reply(tid 0: -18 Not rank 0) ==== 18+0+0 (crc 0 0 0) 0x5571d4a8c380 con 0x5571d4b5c320 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.282+0000 7f321fe00640 10 client.7824 handle_command_reply: tid=0 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.282+0000 7f32489ae740 1 client.7824 shutdown 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.283+0000 7f32489ae740 1 -- 172.21.15.40:0/3136101699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 msgr2=0x5571d4b4ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.283+0000 7f32489ae740 1 --2- 172.21.15.40:0/3136101699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 0x5571d4b4ea20 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f322c006fd0 tx=0x7f322c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.283+0000 7f32489ae740 1 -- 172.21.15.40:0/3136101699 shutdown_connections 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.283+0000 7f32489ae740 1 --2- 172.21.15.40:0/3136101699 >> [v2:172.21.15.40:6832/216893598,v1:172.21.15.40:6834/216893598] conn(0x5571d4b5c320 0x5571d4b88790 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.283+0000 7f32489ae740 1 --2- 172.21.15.40:0/3136101699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5571d4b689e0 0x5571d4b4ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.283+0000 7f32489ae740 1 --2- 172.21.15.40:0/3136101699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5571d4b4b140 0x5571d4b4ea20 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.283+0000 7f32489ae740 1 --2- 172.21.15.40:0/3136101699 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5571d4b4a700 0x5571d4bdfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.283+0000 7f321fe00640 0 client.7824 ms_handle_reset on v2:172.21.15.40:6832/216893598 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.283+0000 7f32489ae740 1 -- 172.21.15.40:0/3136101699 >> 172.21.15.40:0/3136101699 conn(0x5571d4b20e20 msgr2=0x5571d4b22270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.283+0000 7f32489ae740 1 -- 172.21.15.40:0/3136101699 shutdown_connections 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.283+0000 7f32489ae740 1 -- 172.21.15.40:0/3136101699 wait complete. 2024-04-26T23:07:26.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.283+0000 7f32489ae740 20 client.7824 trim_cache size 0 max 16384 2024-04-26T23:07:26.287 INFO:teuthology.orchestra.run.smithi040.stderr:Error EXDEV: Not rank 0 2024-04-26T23:07:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 18 2024-04-26T23:07:26.291 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-26T23:07:26.291 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:07:26.380 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:07:26.492 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-26T23:07:26.492 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:07:26.492 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:07:26.492 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:07:26.553 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:07:26.553 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:07:26.609 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:07:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.828+0000 7f626de00640 1 Processor -- start 2024-04-26T23:07:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.829+0000 7f626de00640 1 -- start start 2024-04-26T23:07:26.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.830+0000 7f626de00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f626815f5a0 0x7f62680d5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.830+0000 7f626de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6268163610 0x7f62680cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.830+0000 7f626de00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62680d1a70 con 0x7f62680d2830 2024-04-26T23:07:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.830+0000 7f626de00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6268163d30 con 0x7f626815f5a0 2024-04-26T23:07:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.830+0000 7f626de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f62680d1d50 con 0x7f6268163610 2024-04-26T23:07:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.831+0000 7f625e000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f62680d2830 0x7f6268162f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34240/0 (socket says 172.21.15.40:34240) 2024-04-26T23:07:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.831+0000 7f625e000640 1 -- 172.21.15.40:0/819334946 learned_addr learned my addr 172.21.15.40:0/819334946 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.831+0000 7f6267400640 1 --2- 172.21.15.40:0/819334946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6268163610 0x7f62680cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.831+0000 7f6266a00640 1 --2- 172.21.15.40:0/819334946 >> v2:172.21.15.103:3300/0 conn(0x7f626815f5a0 0x7f62680d5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.831+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3164309424 0 0) 0x7f62680d1a70 con 0x7f62680d2830 2024-04-26T23:07:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.831+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f624c003520 con 0x7f62680d2830 2024-04-26T23:07:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3087012035 0 0) 0x7f624c003520 con 0x7f62680d2830 2024-04-26T23:07:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6268163610 msgr2=0x7f62680cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 --2- 172.21.15.40:0/819334946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6268163610 0x7f62680cfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 >> v2:172.21.15.103:3300/0 conn(0x7f626815f5a0 msgr2=0x7f62680d5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 --2- 172.21.15.40:0/819334946 >> v2:172.21.15.103:3300/0 conn(0x7f626815f5a0 0x7f62680d5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62680d2030 con 0x7f62680d2830 2024-04-26T23:07:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f6250001670 con 0x7f62680d2830 2024-04-26T23:07:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 >> v1:172.21.15.40:6789/0 conn(0x7f62680d2830 legacy=0x7f6268162f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 --2- 172.21.15.40:0/819334946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f624c0039b0 0x7f624c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 --2- 172.21.15.40:0/819334946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6268163610 0x7f624c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 --2- 172.21.15.40:0/819334946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f624c004d40 0x7f624c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f62680d1d50 con 0x7f624c0039b0 2024-04-26T23:07:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f624c00b9c0 con 0x7f624c004d40 2024-04-26T23:07:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f624c00bb70 con 0x7f6268163610 2024-04-26T23:07:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266a00640 1 --2- 172.21.15.40:0/819334946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f624c0039b0 0x7f624c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266a00640 1 -- 172.21.15.40:0/819334946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6268163610 msgr2=0x7f624c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266a00640 1 --2- 172.21.15.40:0/819334946 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6268163610 0x7f624c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266a00640 1 -- 172.21.15.40:0/819334946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f624c004d40 msgr2=0x7f624c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266a00640 1 --2- 172.21.15.40:0/819334946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f624c004d40 0x7f624c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.832+0000 7f6266a00640 1 -- 172.21.15.40:0/819334946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f624c003520 con 0x7f624c0039b0 2024-04-26T23:07:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.834+0000 7f6266a00640 1 --2- 172.21.15.40:0/819334946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f624c0039b0 0x7f624c003dd0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f6258009fc0 tx=0x7f624c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1547a299ceea25ff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.834+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258009c10 con 0x7f624c0039b0 2024-04-26T23:07:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.834+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6258036070 con 0x7f624c0039b0 2024-04-26T23:07:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.834+0000 7f6266000640 1 -- 172.21.15.40:0/819334946 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625802d070 con 0x7f624c0039b0 2024-04-26T23:07:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.834+0000 7f626de00640 1 -- 172.21.15.40:0/819334946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f624c0039b0 msgr2=0x7f624c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.834+0000 7f626de00640 1 --2- 172.21.15.40:0/819334946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f624c0039b0 0x7f624c003dd0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f6258009fc0 tx=0x7f624c005830 comp rx=0 tx=0).stop 2024-04-26T23:07:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.834+0000 7f6267400640 1 -- 172.21.15.40:0/819334946 reap_dead start 2024-04-26T23:07:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.835+0000 7f626de00640 1 -- 172.21.15.40:0/819334946 shutdown_connections 2024-04-26T23:07:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.835+0000 7f626de00640 1 -- 172.21.15.40:0/819334946 >> 172.21.15.40:0/819334946 conn(0x7f62680d5190 msgr2=0x7f62680d55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.835+0000 7f626de00640 1 -- 172.21.15.40:0/819334946 shutdown_connections 2024-04-26T23:07:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.835+0000 7f626de00640 1 -- 172.21.15.40:0/819334946 wait complete. 2024-04-26T23:07:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.836+0000 7f626de00640 1 Processor -- start 2024-04-26T23:07:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.837+0000 7f626de00640 1 -- start start 2024-04-26T23:07:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f626de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f624c0039b0 0x7f62680ba7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f626de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6268157fe0 0x7f62680b9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f626de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62680ba080 0x7f62680c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f626de00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f62680d3c20 con 0x7f624c0039b0 2024-04-26T23:07:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f626de00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f62680d4190 con 0x7f62680ba080 2024-04-26T23:07:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f626de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f62680d3da0 con 0x7f6268157fe0 2024-04-26T23:07:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f6266a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f624c0039b0 0x7f62680ba7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f6266a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f624c0039b0 0x7f62680ba7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41356/0 (socket says 172.21.15.40:41356) 2024-04-26T23:07:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f6266a00640 1 -- 172.21.15.40:0/3416999068 learned_addr learned my addr 172.21.15.40:0/3416999068 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f6267400640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62680ba080 0x7f62680c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f625e000640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6268157fe0 0x7f62680b9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f6266a00640 1 -- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6268157fe0 msgr2=0x7f62680b9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f6266a00640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6268157fe0 0x7f62680b9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f6266a00640 1 -- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62680ba080 msgr2=0x7f62680c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f6266a00640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62680ba080 0x7f62680c9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f6266a00640 1 -- 172.21.15.40:0/3416999068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62681eb440 con 0x7f624c0039b0 2024-04-26T23:07:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f6267400640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62680ba080 0x7f62680c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.838+0000 7f625e000640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6268157fe0 0x7f62680b9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.839+0000 7f6266a00640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f624c0039b0 0x7f62680ba7e0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f625800e230 tx=0x7f6258009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.839+0000 7f625fe00640 1 -- 172.21.15.40:0/3416999068 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258009c10 con 0x7f624c0039b0 2024-04-26T23:07:26.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.839+0000 7f625fe00640 1 -- 172.21.15.40:0/3416999068 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6258003bb0 con 0x7f624c0039b0 2024-04-26T23:07:26.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.839+0000 7f626de00640 1 -- 172.21.15.40:0/3416999068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62680c8e60 con 0x7f624c0039b0 2024-04-26T23:07:26.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.839+0000 7f625fe00640 1 -- 172.21.15.40:0/3416999068 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258004020 con 0x7f624c0039b0 2024-04-26T23:07:26.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.839+0000 7f626de00640 1 -- 172.21.15.40:0/3416999068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62680ba530 con 0x7f624c0039b0 2024-04-26T23:07:26.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.841+0000 7f626de00640 1 -- 172.21.15.40:0/3416999068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62680bad20 con 0x7f624c0039b0 2024-04-26T23:07:26.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.842+0000 7f625fe00640 1 -- 172.21.15.40:0/3416999068 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6258005d10 con 0x7f624c0039b0 2024-04-26T23:07:26.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.842+0000 7f625fe00640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f624406c8f0 0x7f624406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:26.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.843+0000 7f625fe00640 1 -- 172.21.15.40:0/3416999068 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f62580ad230 con 0x7f624c0039b0 2024-04-26T23:07:26.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.843+0000 7f625e000640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f624406c8f0 0x7f624406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:26.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.844+0000 7f625e000640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f624406c8f0 0x7f624406edb0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6250001eb0 tx=0x7f6250009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:26.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:26.847+0000 7f625fe00640 1 -- 172.21.15.40:0/3416999068 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f62680bad20 con 0x7f624c0039b0 2024-04-26T23:07:27.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.056+0000 7f626de00640 1 -- 172.21.15.40:0/3416999068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62680d4310 con 0x7f624c0039b0 2024-04-26T23:07:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.059+0000 7f625fe00640 1 -- 172.21.15.40:0/3416999068 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+16909 (secure 0 0 0) 0x7f62680d4310 con 0x7f624c0039b0 2024-04-26T23:07:27.060 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:27.060 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":79,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:06:59.420463+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:06:55.738277+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-26T23:06:56.838617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:07:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.062+0000 7f626de00640 1 -- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f624406c8f0 msgr2=0x7f624406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.062+0000 7f626de00640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f624406c8f0 0x7f624406edb0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6250001eb0 tx=0x7f6250009040 comp rx=0 tx=0).stop 2024-04-26T23:07:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.062+0000 7f626de00640 1 -- 172.21.15.40:0/3416999068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f624c0039b0 msgr2=0x7f62680ba7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.062+0000 7f626de00640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f624c0039b0 0x7f62680ba7e0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f625800e230 tx=0x7f6258009750 comp rx=0 tx=0).stop 2024-04-26T23:07:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.062+0000 7f626de00640 1 -- 172.21.15.40:0/3416999068 shutdown_connections 2024-04-26T23:07:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.062+0000 7f626de00640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f624406c8f0 0x7f624406edb0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.062+0000 7f626de00640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62680ba080 0x7f62680c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.062+0000 7f626de00640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6268157fe0 0x7f62680b9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.062+0000 7f626de00640 1 --2- 172.21.15.40:0/3416999068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f624c0039b0 0x7f62680ba7e0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.062+0000 7f626de00640 1 -- 172.21.15.40:0/3416999068 >> 172.21.15.40:0/3416999068 conn(0x7f62680d5190 msgr2=0x7f626806dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.063+0000 7f626de00640 1 -- 172.21.15.40:0/3416999068 shutdown_connections 2024-04-26T23:07:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.063+0000 7f626de00640 1 -- 172.21.15.40:0/3416999068 wait complete. 2024-04-26T23:07:27.084 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:06:55.738277+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:06:56.838617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:07:27.084 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:07:27.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.300+0000 7efcab200640 1 Processor -- start 2024-04-26T23:07:27.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.300+0000 7efcab200640 1 -- start start 2024-04-26T23:07:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.302+0000 7efcab200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efcac073600 0x7efcac072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.302+0000 7efcab200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efcac0739e0 0x7efcac15f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.302+0000 7efcab200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcac1604e0 con 0x7efcac15fcb0 2024-04-26T23:07:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.302+0000 7efcab200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7efcac073220 con 0x7efcac073600 2024-04-26T23:07:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.302+0000 7efcab200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efcac072850 con 0x7efcac0739e0 2024-04-26T23:07:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.302+0000 7efca9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efcac073600 0x7efcac072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.302+0000 7efcaa800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efcac15fcb0 0x7efcac0ae8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34254/0 (socket says 172.21.15.40:34254) 2024-04-26T23:07:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.302+0000 7efcaa800640 1 -- 172.21.15.40:0/247680144 learned_addr learned my addr 172.21.15.40:0/247680144 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.303+0000 7efca9400640 1 --2- 172.21.15.40:0/247680144 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efcac0739e0 0x7efcac15f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.303+0000 7efca9e00640 1 -- 172.21.15.40:0/247680144 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efcac0739e0 msgr2=0x7efcac15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.303+0000 7efca9e00640 1 --2- 172.21.15.40:0/247680144 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efcac0739e0 0x7efcac15f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.303+0000 7efca9e00640 1 -- 172.21.15.40:0/247680144 >> v1:172.21.15.40:6789/0 conn(0x7efcac15fcb0 legacy=0x7efcac0ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:27.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.303+0000 7efca9e00640 1 -- 172.21.15.40:0/247680144 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcac072200 con 0x7efcac073600 2024-04-26T23:07:27.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.304+0000 7efca9e00640 1 --2- 172.21.15.40:0/247680144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcac073600 0x7efcac072ce0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0097a0 tx=0x7efc9c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d279a76317d57400 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:27.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.304+0000 7efca8a00640 1 -- 172.21.15.40:0/247680144 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c037070 con 0x7efcac073600 2024-04-26T23:07:27.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.304+0000 7efca8a00640 1 -- 172.21.15.40:0/247680144 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc9c032440 con 0x7efcac073600 2024-04-26T23:07:27.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.304+0000 7efca8a00640 1 -- 172.21.15.40:0/247680144 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c032810 con 0x7efcac073600 2024-04-26T23:07:27.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.305+0000 7efcab200640 1 -- 172.21.15.40:0/247680144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcac073600 msgr2=0x7efcac072ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:27.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.305+0000 7efcab200640 1 --2- 172.21.15.40:0/247680144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcac073600 0x7efcac072ce0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0097a0 tx=0x7efc9c02a180 comp rx=0 tx=0).stop 2024-04-26T23:07:27.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.305+0000 7efcab200640 1 -- 172.21.15.40:0/247680144 shutdown_connections 2024-04-26T23:07:27.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.305+0000 7efcab200640 1 --2- 172.21.15.40:0/247680144 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efcac0739e0 0x7efcac15f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.305+0000 7efcab200640 1 --2- 172.21.15.40:0/247680144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcac073600 0x7efcac072ce0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.305+0000 7efcab200640 1 -- 172.21.15.40:0/247680144 >> 172.21.15.40:0/247680144 conn(0x7efcac0b2720 msgr2=0x7efcac0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:27.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.305+0000 7efcab200640 1 -- 172.21.15.40:0/247680144 shutdown_connections 2024-04-26T23:07:27.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.305+0000 7efcab200640 1 -- 172.21.15.40:0/247680144 wait complete. 2024-04-26T23:07:27.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.307+0000 7efcab200640 1 Processor -- start 2024-04-26T23:07:27.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.308+0000 7efcab200640 1 -- start start 2024-04-26T23:07:27.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.309+0000 7efcab200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efcac073600 0x7efcac158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:27.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.309+0000 7efcab200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcac0739e0 0x7efcac151470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:27.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.309+0000 7efcab200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcac15fcb0 0x7efcac154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:27.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.309+0000 7efcab200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efcac071e20 con 0x7efcac15fcb0 2024-04-26T23:07:27.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.309+0000 7efcab200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efcac073e60 con 0x7efcac0739e0 2024-04-26T23:07:27.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.309+0000 7efcab200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efcac071fa0 con 0x7efcac073600 2024-04-26T23:07:27.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.309+0000 7efcaa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcac15fcb0 0x7efcac154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:27.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.309+0000 7efcaa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcac15fcb0 0x7efcac154ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41366/0 (socket says 172.21.15.40:41366) 2024-04-26T23:07:27.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.309+0000 7efcaa800640 1 -- 172.21.15.40:0/1067775217 learned_addr learned my addr 172.21.15.40:0/1067775217 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:27.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.310+0000 7efca9e00640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efcac073600 0x7efcac158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:27.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.310+0000 7efca9400640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcac0739e0 0x7efcac151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:27.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.310+0000 7efcaa800640 1 -- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efcac073600 msgr2=0x7efcac158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:27.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.310+0000 7efcaa800640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efcac073600 0x7efcac158250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.310+0000 7efcaa800640 1 -- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcac0739e0 msgr2=0x7efcac151470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:27.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.310+0000 7efcaa800640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcac0739e0 0x7efcac151470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.310+0000 7efcaa800640 1 -- 172.21.15.40:0/1067775217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcac1571b0 con 0x7efcac15fcb0 2024-04-26T23:07:27.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.310+0000 7efcaa800640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcac15fcb0 0x7efcac154ad0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7efca400cab0 tx=0x7efca400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:27.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.311+0000 7efca2000640 1 -- 172.21.15.40:0/1067775217 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca4017070 con 0x7efcac15fcb0 2024-04-26T23:07:27.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.311+0000 7efca2000640 1 -- 172.21.15.40:0/1067775217 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca400ac30 con 0x7efcac15fcb0 2024-04-26T23:07:27.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.311+0000 7efca2000640 1 -- 172.21.15.40:0/1067775217 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca4012400 con 0x7efcac15fcb0 2024-04-26T23:07:27.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.311+0000 7efcab200640 1 -- 172.21.15.40:0/1067775217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcac157490 con 0x7efcac15fcb0 2024-04-26T23:07:27.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.311+0000 7efcab200640 1 -- 172.21.15.40:0/1067775217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcac157a50 con 0x7efcac15fcb0 2024-04-26T23:07:27.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.312+0000 7efcab200640 1 -- 172.21.15.40:0/1067775217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcac0b5620 con 0x7efcac15fcb0 2024-04-26T23:07:27.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.313+0000 7efca2000640 1 -- 172.21.15.40:0/1067775217 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efca401a070 con 0x7efcac15fcb0 2024-04-26T23:07:27.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.314+0000 7efca2000640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc8c06c920 0x7efc8c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:27.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.314+0000 7efca2000640 1 -- 172.21.15.40:0/1067775217 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7efca408b9b0 con 0x7efcac15fcb0 2024-04-26T23:07:27.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.315+0000 7efca9e00640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc8c06c920 0x7efc8c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:27.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.316+0000 7efca9e00640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc8c06c920 0x7efc8c06ede0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7efc9c034000 tx=0x7efc9c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:27.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.318+0000 7efca2000640 1 -- 172.21.15.40:0/1067775217 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7efcac0b5620 con 0x7efcac15fcb0 2024-04-26T23:07:27.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.525+0000 7efcab200640 1 -- 172.21.15.40:0/1067775217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efcac151aa0 con 0x7efcac15fcb0 2024-04-26T23:07:27.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.526+0000 7efca2000640 1 -- 172.21.15.40:0/1067775217 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) ==== 93+0+10 (secure 0 0 0) 0x7efca4059560 con 0x7efcac15fcb0 2024-04-26T23:07:27.526 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:07:27.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.528+0000 7efcab200640 1 -- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc8c06c920 msgr2=0x7efc8c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:27.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.528+0000 7efcab200640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc8c06c920 0x7efc8c06ede0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7efc9c034000 tx=0x7efc9c02e000 comp rx=0 tx=0).stop 2024-04-26T23:07:27.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.529+0000 7efcab200640 1 -- 172.21.15.40:0/1067775217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcac15fcb0 msgr2=0x7efcac154ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:27.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.529+0000 7efcab200640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcac15fcb0 0x7efcac154ad0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7efca400cab0 tx=0x7efca400cf70 comp rx=0 tx=0).stop 2024-04-26T23:07:27.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.529+0000 7efcab200640 1 -- 172.21.15.40:0/1067775217 shutdown_connections 2024-04-26T23:07:27.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.529+0000 7efcab200640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc8c06c920 0x7efc8c06ede0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.529+0000 7efcab200640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcac15fcb0 0x7efcac154ad0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.529+0000 7efcab200640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcac0739e0 0x7efcac151470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.529+0000 7efcab200640 1 --2- 172.21.15.40:0/1067775217 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efcac073600 0x7efcac158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.529+0000 7efcab200640 1 -- 172.21.15.40:0/1067775217 >> 172.21.15.40:0/1067775217 conn(0x7efcac0b2720 msgr2=0x7efcac06e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:27.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.529+0000 7efcab200640 1 -- 172.21.15.40:0/1067775217 shutdown_connections 2024-04-26T23:07:27.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.529+0000 7efcab200640 1 -- 172.21.15.40:0/1067775217 wait complete. 2024-04-26T23:07:27.552 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:07:27.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.753+0000 7faed5400640 1 Processor -- start 2024-04-26T23:07:27.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.753+0000 7faed5400640 1 -- start start 2024-04-26T23:07:27.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.754+0000 7faed5400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faed0160a50 0x7faed0073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:27.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faed5400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faed00cfaf0 0x7faed0160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:27.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faed5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed006dd40 0x7faed0072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:27.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faed5400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7faed0071ce0 con 0x7faed00cfaf0 2024-04-26T23:07:27.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faed5400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7faed00722a0 con 0x7faed0160a50 2024-04-26T23:07:27.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faed5400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faed0073f90 con 0x7faed006dd40 2024-04-26T23:07:27.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faec7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faed00cfaf0 0x7faed0160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:27.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faec7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faed00cfaf0 0x7faed0160e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41370/0 (socket says 172.21.15.40:41370) 2024-04-26T23:07:27.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faec7e00640 1 -- 172.21.15.40:0/821542511 learned_addr learned my addr 172.21.15.40:0/821542511 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:27.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faecfe00640 1 --2- 172.21.15.40:0/821542511 >> v2:172.21.15.103:3300/0 conn(0x7faed0160a50 0x7faed0073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:27.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faed4a00640 1 --2- 172.21.15.40:0/821542511 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed006dd40 0x7faed0072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:27.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faec7e00640 1 -- 172.21.15.40:0/821542511 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed006dd40 msgr2=0x7faed0072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:27.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faec7e00640 1 --2- 172.21.15.40:0/821542511 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed006dd40 0x7faed0072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faec7e00640 1 -- 172.21.15.40:0/821542511 >> v2:172.21.15.103:3300/0 conn(0x7faed0160a50 msgr2=0x7faed0073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:27.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faec7e00640 1 --2- 172.21.15.40:0/821542511 >> v2:172.21.15.103:3300/0 conn(0x7faed0160a50 0x7faed0073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faec7e00640 1 -- 172.21.15.40:0/821542511 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed0161880 con 0x7faed00cfaf0 2024-04-26T23:07:27.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.755+0000 7faecfe00640 1 --2- 172.21.15.40:0/821542511 >> v2:172.21.15.103:3300/0 conn(0x7faed0160a50 0x7faed0073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.757+0000 7faec7e00640 1 --2- 172.21.15.40:0/821542511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00cfaf0 0x7faed0160e30 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7faed0161b70 tx=0x7faeb802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6366ae6647256c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.757+0000 7faecf400640 1 -- 172.21.15.40:0/821542511 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb8037070 con 0x7faed00cfaf0 2024-04-26T23:07:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.757+0000 7faecf400640 1 -- 172.21.15.40:0/821542511 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeb8032440 con 0x7faed00cfaf0 2024-04-26T23:07:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.757+0000 7faecf400640 1 -- 172.21.15.40:0/821542511 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb8032810 con 0x7faed00cfaf0 2024-04-26T23:07:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.758+0000 7faed5400640 1 -- 172.21.15.40:0/821542511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00cfaf0 msgr2=0x7faed0160e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.758+0000 7faed5400640 1 --2- 172.21.15.40:0/821542511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00cfaf0 0x7faed0160e30 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7faed0161b70 tx=0x7faeb802a1e0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.758+0000 7faed5400640 1 -- 172.21.15.40:0/821542511 shutdown_connections 2024-04-26T23:07:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.758+0000 7faed5400640 1 --2- 172.21.15.40:0/821542511 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed006dd40 0x7faed0072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.758+0000 7faed5400640 1 --2- 172.21.15.40:0/821542511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00cfaf0 0x7faed0160e30 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.758+0000 7faed5400640 1 --2- 172.21.15.40:0/821542511 >> v2:172.21.15.103:3300/0 conn(0x7faed0160a50 0x7faed0073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.758+0000 7faed5400640 1 -- 172.21.15.40:0/821542511 >> 172.21.15.40:0/821542511 conn(0x7faed00b2720 msgr2=0x7faed00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.758+0000 7faed5400640 1 -- 172.21.15.40:0/821542511 shutdown_connections 2024-04-26T23:07:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.758+0000 7faed5400640 1 -- 172.21.15.40:0/821542511 wait complete. 2024-04-26T23:07:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.760+0000 7faed5400640 1 Processor -- start 2024-04-26T23:07:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.760+0000 7faed5400640 1 -- start start 2024-04-26T23:07:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.761+0000 7faed5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed006dd40 0x7faed015a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.761+0000 7faed5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed0160a50 0x7faed015d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.761+0000 7faed5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed015d8a0 0x7faed015f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.761+0000 7faed5400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faed00d0260 con 0x7faed015d8a0 2024-04-26T23:07:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.761+0000 7faed5400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faed00cff80 con 0x7faed0160a50 2024-04-26T23:07:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.761+0000 7faed5400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faed0071f30 con 0x7faed006dd40 2024-04-26T23:07:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.761+0000 7faed4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed015d8a0 0x7faed015f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.761+0000 7faed4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed015d8a0 0x7faed015f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41386/0 (socket says 172.21.15.40:41386) 2024-04-26T23:07:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.761+0000 7faed4a00640 1 -- 172.21.15.40:0/2550007343 learned_addr learned my addr 172.21.15.40:0/2550007343 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.761+0000 7faecfe00640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed006dd40 0x7faed015a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.762+0000 7faec7e00640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed0160a50 0x7faed015d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.762+0000 7faed4a00640 1 -- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed006dd40 msgr2=0x7faed015a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.762+0000 7faed4a00640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed006dd40 0x7faed015a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.762+0000 7faed4a00640 1 -- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed0160a50 msgr2=0x7faed015d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.762+0000 7faed4a00640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed0160a50 0x7faed015d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.762+0000 7faed4a00640 1 -- 172.21.15.40:0/2550007343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed015dde0 con 0x7faed015d8a0 2024-04-26T23:07:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.762+0000 7faec7e00640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed0160a50 0x7faed015d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.762+0000 7faed4a00640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed015d8a0 0x7faed015f7d0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7faec800cd20 tx=0x7faec800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.762+0000 7faeccc00640 1 -- 172.21.15.40:0/2550007343 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8003b90 con 0x7faed015d8a0 2024-04-26T23:07:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.762+0000 7faeccc00640 1 -- 172.21.15.40:0/2550007343 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec8003d30 con 0x7faed015d8a0 2024-04-26T23:07:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.762+0000 7faeccc00640 1 -- 172.21.15.40:0/2550007343 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8004240 con 0x7faed015d8a0 2024-04-26T23:07:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.762+0000 7faed5400640 1 -- 172.21.15.40:0/2550007343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed015e0c0 con 0x7faed015d8a0 2024-04-26T23:07:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.763+0000 7faed5400640 1 -- 172.21.15.40:0/2550007343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed015e680 con 0x7faed015d8a0 2024-04-26T23:07:27.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.764+0000 7faed5400640 1 -- 172.21.15.40:0/2550007343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed0069370 con 0x7faed015d8a0 2024-04-26T23:07:27.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.765+0000 7faeccc00640 1 -- 172.21.15.40:0/2550007343 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faec80043e0 con 0x7faed015d8a0 2024-04-26T23:07:27.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.766+0000 7faeccc00640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeac06c8f0 0x7faeac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:27.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.766+0000 7faeccc00640 1 -- 172.21.15.40:0/2550007343 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7faec808c830 con 0x7faed015d8a0 2024-04-26T23:07:27.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.766+0000 7faecfe00640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeac06c8f0 0x7faeac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:27.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.768+0000 7faecfe00640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeac06c8f0 0x7faeac06edb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faec000a950 tx=0x7faec0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:27.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.774+0000 7faeccc00640 1 -- 172.21.15.40:0/2550007343 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7faec805aa30 con 0x7faed015d8a0 2024-04-26T23:07:27.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.980+0000 7faed5400640 1 -- 172.21.15.40:0/2550007343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7faed0161150 con 0x7faed015d8a0 2024-04-26T23:07:27.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.981+0000 7faeccc00640 1 -- 172.21.15.40:0/2550007343 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) ==== 104+0+11 (secure 0 0 0) 0x7faec805a330 con 0x7faed015d8a0 2024-04-26T23:07:27.981 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:07:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.983+0000 7faed5400640 1 -- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeac06c8f0 msgr2=0x7faeac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.983+0000 7faed5400640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeac06c8f0 0x7faeac06edb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faec000a950 tx=0x7faec0008040 comp rx=0 tx=0).stop 2024-04-26T23:07:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.984+0000 7faed5400640 1 -- 172.21.15.40:0/2550007343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed015d8a0 msgr2=0x7faed015f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.984+0000 7faed5400640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed015d8a0 0x7faed015f7d0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7faec800cd20 tx=0x7faec800a660 comp rx=0 tx=0).stop 2024-04-26T23:07:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.984+0000 7faed5400640 1 -- 172.21.15.40:0/2550007343 shutdown_connections 2024-04-26T23:07:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.984+0000 7faed5400640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeac06c8f0 0x7faeac06edb0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.984+0000 7faed5400640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed015d8a0 0x7faed015f7d0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.984+0000 7faed5400640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed0160a50 0x7faed015d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.984+0000 7faed5400640 1 --2- 172.21.15.40:0/2550007343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed006dd40 0x7faed015a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:27.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.984+0000 7faed5400640 1 -- 172.21.15.40:0/2550007343 >> 172.21.15.40:0/2550007343 conn(0x7faed00b2720 msgr2=0x7faed00b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:27.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.984+0000 7faed5400640 1 -- 172.21.15.40:0/2550007343 shutdown_connections 2024-04-26T23:07:27.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:27.984+0000 7faed5400640 1 -- 172.21.15.40:0/2550007343 wait complete. 2024-04-26T23:07:28.003 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:07:28.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.199+0000 7f2f63e00640 1 Processor -- start 2024-04-26T23:07:28.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.199+0000 7f2f63e00640 1 -- start start 2024-04-26T23:07:28.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.200+0000 7f2f63e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2f641486b0 0x7f2f640d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:28.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.200+0000 7f2f63e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f6406ebf0 0x7f2f6406f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:28.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.200+0000 7f2f63e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f640d19f0 con 0x7f2f640d2710 2024-04-26T23:07:28.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.200+0000 7f2f63e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f2f6415ff80 con 0x7f2f6406ebf0 2024-04-26T23:07:28.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.200+0000 7f2f63e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2f64160260 con 0x7f2f641486b0 2024-04-26T23:07:28.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.201+0000 7f2f63400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f6406ebf0 0x7f2f6406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:28.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.201+0000 7f2f5be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2f640d2710 0x7f2f6415f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34256/0 (socket says 172.21.15.40:34256) 2024-04-26T23:07:28.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.201+0000 7f2f5be00640 1 -- 172.21.15.40:0/483746131 learned_addr learned my addr 172.21.15.40:0/483746131 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:28.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.201+0000 7f2f62a00640 1 --2- 172.21.15.40:0/483746131 >> v2:172.21.15.103:3300/0 conn(0x7f2f641486b0 0x7f2f640d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:28.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.201+0000 7f2f62000640 1 -- 172.21.15.40:0/483746131 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560724205 0 0) 0x7f2f640d19f0 con 0x7f2f640d2710 2024-04-26T23:07:28.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.201+0000 7f2f63400640 1 -- 172.21.15.40:0/483746131 >> v2:172.21.15.103:3300/0 conn(0x7f2f641486b0 msgr2=0x7f2f640d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:28.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.201+0000 7f2f63400640 1 --2- 172.21.15.40:0/483746131 >> v2:172.21.15.103:3300/0 conn(0x7f2f641486b0 0x7f2f640d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.201+0000 7f2f63400640 1 -- 172.21.15.40:0/483746131 >> v1:172.21.15.40:6789/0 conn(0x7f2f640d2710 legacy=0x7f2f6415f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:28.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.202+0000 7f2f63400640 1 -- 172.21.15.40:0/483746131 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f640d1cb0 con 0x7f2f6406ebf0 2024-04-26T23:07:28.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.203+0000 7f2f63400640 1 --2- 172.21.15.40:0/483746131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6406ebf0 0x7f2f6406f080 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009d90 tx=0x7f2f5c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bcaaf6da05e1916 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:28.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.203+0000 7f2f62000640 1 -- 172.21.15.40:0/483746131 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c031600 con 0x7f2f6406ebf0 2024-04-26T23:07:28.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.203+0000 7f2f62000640 1 -- 172.21.15.40:0/483746131 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f5c0317a0 con 0x7f2f6406ebf0 2024-04-26T23:07:28.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.203+0000 7f2f62000640 1 -- 172.21.15.40:0/483746131 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c031b70 con 0x7f2f6406ebf0 2024-04-26T23:07:28.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.204+0000 7f2f63e00640 1 -- 172.21.15.40:0/483746131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6406ebf0 msgr2=0x7f2f6406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:28.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.204+0000 7f2f63e00640 1 --2- 172.21.15.40:0/483746131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6406ebf0 0x7f2f6406f080 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009d90 tx=0x7f2f5c02a610 comp rx=0 tx=0).stop 2024-04-26T23:07:28.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.204+0000 7f2f63e00640 1 -- 172.21.15.40:0/483746131 shutdown_connections 2024-04-26T23:07:28.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.204+0000 7f2f63e00640 1 --2- 172.21.15.40:0/483746131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6406ebf0 0x7f2f6406f080 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.204+0000 7f2f63e00640 1 --2- 172.21.15.40:0/483746131 >> v2:172.21.15.103:3300/0 conn(0x7f2f641486b0 0x7f2f640d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.204+0000 7f2f63e00640 1 -- 172.21.15.40:0/483746131 >> 172.21.15.40:0/483746131 conn(0x7f2f64068090 msgr2=0x7f2f64068470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:28.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.204+0000 7f2f63e00640 1 -- 172.21.15.40:0/483746131 shutdown_connections 2024-04-26T23:07:28.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.204+0000 7f2f63e00640 1 -- 172.21.15.40:0/483746131 wait complete. 2024-04-26T23:07:28.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.206+0000 7f2f63e00640 1 Processor -- start 2024-04-26T23:07:28.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.206+0000 7f2f63e00640 1 -- start start 2024-04-26T23:07:28.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.207+0000 7f2f63e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f6406ebf0 0x7f2f641e79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:28.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.207+0000 7f2f63e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f640d2710 0x7f2f64146390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:28.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.207+0000 7f2f63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f641486b0 0x7f2f6415e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:28.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.207+0000 7f2f63e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f64148c10 con 0x7f2f641486b0 2024-04-26T23:07:28.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.207+0000 7f2f63e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2f64148a90 con 0x7f2f6406ebf0 2024-04-26T23:07:28.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.207+0000 7f2f63e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2f640b6e40 con 0x7f2f640d2710 2024-04-26T23:07:28.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.207+0000 7f2f63400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f641486b0 0x7f2f6415e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:28.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.207+0000 7f2f63400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f641486b0 0x7f2f6415e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41404/0 (socket says 172.21.15.40:41404) 2024-04-26T23:07:28.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.207+0000 7f2f63400640 1 -- 172.21.15.40:0/2951518396 learned_addr learned my addr 172.21.15.40:0/2951518396 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:28.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.208+0000 7f2f5be00640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f640d2710 0x7f2f64146390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:28.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.208+0000 7f2f62a00640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f6406ebf0 0x7f2f641e79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:28.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.208+0000 7f2f63400640 1 -- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f640d2710 msgr2=0x7f2f64146390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:28.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.208+0000 7f2f63400640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f640d2710 0x7f2f64146390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.208+0000 7f2f63400640 1 -- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f6406ebf0 msgr2=0x7f2f641e79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:28.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.208+0000 7f2f63400640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f6406ebf0 0x7f2f641e79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.208+0000 7f2f63400640 1 -- 172.21.15.40:0/2951518396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f640cf4c0 con 0x7f2f641486b0 2024-04-26T23:07:28.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.208+0000 7f2f63400640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f641486b0 0x7f2f6415e3a0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c02ac00 tx=0x7f2f5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:28.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.209+0000 7f2f5aa00640 1 -- 172.21.15.40:0/2951518396 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c03f070 con 0x7f2f641486b0 2024-04-26T23:07:28.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.209+0000 7f2f5aa00640 1 -- 172.21.15.40:0/2951518396 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f5c030b60 con 0x7f2f641486b0 2024-04-26T23:07:28.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.209+0000 7f2f5aa00640 1 -- 172.21.15.40:0/2951518396 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c009450 con 0x7f2f641486b0 2024-04-26T23:07:28.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.209+0000 7f2f63e00640 1 -- 172.21.15.40:0/2951518396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f640cf7a0 con 0x7f2f641486b0 2024-04-26T23:07:28.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.209+0000 7f2f63e00640 1 -- 172.21.15.40:0/2951518396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f640cfd60 con 0x7f2f641486b0 2024-04-26T23:07:28.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.210+0000 7f2f63e00640 1 -- 172.21.15.40:0/2951518396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f64069a80 con 0x7f2f641486b0 2024-04-26T23:07:28.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.211+0000 7f2f5aa00640 1 -- 172.21.15.40:0/2951518396 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2f5c03b410 con 0x7f2f641486b0 2024-04-26T23:07:28.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.212+0000 7f2f5aa00640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f4006c8f0 0x7f2f4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:28.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.212+0000 7f2f5aa00640 1 -- 172.21.15.40:0/2951518396 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f2f5c0ad340 con 0x7f2f641486b0 2024-04-26T23:07:28.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.213+0000 7f2f62a00640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f4006c8f0 0x7f2f4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:28.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.214+0000 7f2f62a00640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f4006c8f0 0x7f2f4006edb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2f54006fd0 tx=0x7f2f54008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:28.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.217+0000 7f2f5aa00640 1 -- 172.21.15.40:0/2951518396 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2f5c07aef0 con 0x7f2f641486b0 2024-04-26T23:07:28.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.417+0000 7f2f63e00640 1 -- 172.21.15.40:0/2951518396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2f64073a80 con 0x7f2f641486b0 2024-04-26T23:07:28.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.418+0000 7f2f5aa00640 1 -- 172.21.15.40:0/2951518396 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) ==== 100+0+11 (secure 0 0 0) 0x7f2f5c07f1f0 con 0x7f2f641486b0 2024-04-26T23:07:28.418 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:07:28.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.420+0000 7f2f63e00640 1 -- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f4006c8f0 msgr2=0x7f2f4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:28.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.420+0000 7f2f63e00640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f4006c8f0 0x7f2f4006edb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2f54006fd0 tx=0x7f2f54008040 comp rx=0 tx=0).stop 2024-04-26T23:07:28.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.420+0000 7f2f63e00640 1 -- 172.21.15.40:0/2951518396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f641486b0 msgr2=0x7f2f6415e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:28.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.420+0000 7f2f63e00640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f641486b0 0x7f2f6415e3a0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c02ac00 tx=0x7f2f5c004a40 comp rx=0 tx=0).stop 2024-04-26T23:07:28.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.420+0000 7f2f63e00640 1 -- 172.21.15.40:0/2951518396 shutdown_connections 2024-04-26T23:07:28.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.420+0000 7f2f63e00640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f4006c8f0 0x7f2f4006edb0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.420+0000 7f2f63e00640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f641486b0 0x7f2f6415e3a0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.420+0000 7f2f63e00640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f640d2710 0x7f2f64146390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.420+0000 7f2f63e00640 1 --2- 172.21.15.40:0/2951518396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f6406ebf0 0x7f2f641e79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.420+0000 7f2f63e00640 1 -- 172.21.15.40:0/2951518396 >> 172.21.15.40:0/2951518396 conn(0x7f2f64068090 msgr2=0x7f2f640b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:28.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.421+0000 7f2f63e00640 1 -- 172.21.15.40:0/2951518396 shutdown_connections 2024-04-26T23:07:28.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.421+0000 7f2f63e00640 1 -- 172.21.15.40:0/2951518396 wait complete. 2024-04-26T23:07:28.439 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:07:28.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.617+0000 7ff651200640 1 Processor -- start 2024-04-26T23:07:28.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.618+0000 7ff651200640 1 -- start start 2024-04-26T23:07:28.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.619+0000 7ff651200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff64c070900 0x7ff64c0d3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:28.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.619+0000 7ff651200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff64c0d0b10 0x7ff64c0d0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:28.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.619+0000 7ff651200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff64c070de0 con 0x7ff64c15cfb0 2024-04-26T23:07:28.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.619+0000 7ff651200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff64c0cdf50 con 0x7ff64c0d0b10 2024-04-26T23:07:28.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.619+0000 7ff651200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff64c0710c0 con 0x7ff64c070900 2024-04-26T23:07:28.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.620+0000 7ff64be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff64c0d0b10 0x7ff64c0d0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:28.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.620+0000 7ff643e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff64c070900 0x7ff64c0d3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:28.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.620+0000 7ff64be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff64c0d0b10 0x7ff64c0d0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:42370/0 (socket says 172.21.15.40:42370) 2024-04-26T23:07:28.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.620+0000 7ff64be00640 1 -- 172.21.15.40:0/508461916 learned_addr learned my addr 172.21.15.40:0/508461916 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:28.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.621+0000 7ff643e00640 1 -- 172.21.15.40:0/508461916 >> v2:172.21.15.103:3300/0 conn(0x7ff64c0d0b10 msgr2=0x7ff64c0d0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:28.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.621+0000 7ff643e00640 1 --2- 172.21.15.40:0/508461916 >> v2:172.21.15.103:3300/0 conn(0x7ff64c0d0b10 0x7ff64c0d0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.621+0000 7ff643e00640 1 -- 172.21.15.40:0/508461916 >> v1:172.21.15.103:6789/0 conn(0x7ff64c15cfb0 legacy=0x7ff64c0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:28.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.621+0000 7ff643e00640 1 -- 172.21.15.40:0/508461916 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64c0cdbe0 con 0x7ff64c070900 2024-04-26T23:07:28.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.621+0000 7ff64be00640 1 --2- 172.21.15.40:0/508461916 >> v2:172.21.15.103:3300/0 conn(0x7ff64c0d0b10 0x7ff64c0d0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:07:28.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.622+0000 7ff643e00640 1 --2- 172.21.15.40:0/508461916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff64c070900 0x7ff64c0d3730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff6340097a0 tx=0x7ff63402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=548b72d3cc130a9a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:28.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.622+0000 7ff64aa00640 1 -- 172.21.15.40:0/508461916 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff634037070 con 0x7ff64c070900 2024-04-26T23:07:28.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.622+0000 7ff64aa00640 1 -- 172.21.15.40:0/508461916 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff634032440 con 0x7ff64c070900 2024-04-26T23:07:28.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.622+0000 7ff64aa00640 1 -- 172.21.15.40:0/508461916 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff634032810 con 0x7ff64c070900 2024-04-26T23:07:28.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.623+0000 7ff651200640 1 -- 172.21.15.40:0/508461916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff64c070900 msgr2=0x7ff64c0d3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:28.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.623+0000 7ff651200640 1 --2- 172.21.15.40:0/508461916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff64c070900 0x7ff64c0d3730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff6340097a0 tx=0x7ff63402a180 comp rx=0 tx=0).stop 2024-04-26T23:07:28.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.623+0000 7ff651200640 1 -- 172.21.15.40:0/508461916 shutdown_connections 2024-04-26T23:07:28.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.623+0000 7ff651200640 1 --2- 172.21.15.40:0/508461916 >> v2:172.21.15.103:3300/0 conn(0x7ff64c0d0b10 0x7ff64c0d0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.623+0000 7ff651200640 1 --2- 172.21.15.40:0/508461916 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff64c070900 0x7ff64c0d3730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.623+0000 7ff651200640 1 -- 172.21.15.40:0/508461916 >> 172.21.15.40:0/508461916 conn(0x7ff64c0d2c10 msgr2=0x7ff64c0d3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:28.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.623+0000 7ff651200640 1 -- 172.21.15.40:0/508461916 shutdown_connections 2024-04-26T23:07:28.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.623+0000 7ff651200640 1 -- 172.21.15.40:0/508461916 wait complete. 2024-04-26T23:07:28.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.625+0000 7ff651200640 1 Processor -- start 2024-04-26T23:07:28.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.625+0000 7ff651200640 1 -- start start 2024-04-26T23:07:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.626+0000 7ff651200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff64c070900 0x7ff64c15d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.626+0000 7ff651200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff64c0d0b10 0x7ff64c150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:28.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.626+0000 7ff651200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff64c15cfb0 0x7ff64c0c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:28.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.626+0000 7ff651200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c0cd9a0 con 0x7ff64c15cfb0 2024-04-26T23:07:28.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.626+0000 7ff651200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c0cd820 con 0x7ff64c0d0b10 2024-04-26T23:07:28.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.626+0000 7ff651200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff64c0ceaf0 con 0x7ff64c070900 2024-04-26T23:07:28.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.626+0000 7ff64be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff64c15cfb0 0x7ff64c0c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:28.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.627+0000 7ff64be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff64c15cfb0 0x7ff64c0c9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41416/0 (socket says 172.21.15.40:41416) 2024-04-26T23:07:28.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.627+0000 7ff64be00640 1 -- 172.21.15.40:0/1280475695 learned_addr learned my addr 172.21.15.40:0/1280475695 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:28.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.627+0000 7ff643e00640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff64c0d0b10 0x7ff64c150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:28.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.627+0000 7ff64b400640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff64c070900 0x7ff64c15d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:28.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.627+0000 7ff64be00640 1 -- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff64c070900 msgr2=0x7ff64c15d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:28.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.627+0000 7ff64be00640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff64c070900 0x7ff64c15d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.627+0000 7ff64be00640 1 -- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff64c0d0b10 msgr2=0x7ff64c150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:28.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.627+0000 7ff64be00640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff64c0d0b10 0x7ff64c150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.627+0000 7ff64be00640 1 -- 172.21.15.40:0/1280475695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64c15df80 con 0x7ff64c15cfb0 2024-04-26T23:07:28.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.627+0000 7ff64be00640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff64c15cfb0 0x7ff64c0c9ce0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7ff644009e70 tx=0x7ff64400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:28.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.627+0000 7ff643400640 1 -- 172.21.15.40:0/1280475695 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64400cbe0 con 0x7ff64c15cfb0 2024-04-26T23:07:28.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.628+0000 7ff643400640 1 -- 172.21.15.40:0/1280475695 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff64400cd80 con 0x7ff64c15cfb0 2024-04-26T23:07:28.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.628+0000 7ff651200640 1 -- 172.21.15.40:0/1280475695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64c15e260 con 0x7ff64c15cfb0 2024-04-26T23:07:28.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.628+0000 7ff643400640 1 -- 172.21.15.40:0/1280475695 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff644003bb0 con 0x7ff64c15cfb0 2024-04-26T23:07:28.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.628+0000 7ff651200640 1 -- 172.21.15.40:0/1280475695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff64c15e820 con 0x7ff64c15cfb0 2024-04-26T23:07:28.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.629+0000 7ff651200640 1 -- 172.21.15.40:0/1280475695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff64c069a80 con 0x7ff64c15cfb0 2024-04-26T23:07:28.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.630+0000 7ff643400640 1 -- 172.21.15.40:0/1280475695 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff644005d10 con 0x7ff64c15cfb0 2024-04-26T23:07:28.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.631+0000 7ff643400640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff62806c920 0x7ff62806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:28.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.631+0000 7ff643400640 1 -- 172.21.15.40:0/1280475695 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7ff64408c080 con 0x7ff64c15cfb0 2024-04-26T23:07:28.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.632+0000 7ff64b400640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff62806c920 0x7ff62806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:28.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.633+0000 7ff64b400640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff62806c920 0x7ff62806ede0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff63c001500 tx=0x7ff63c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:28.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.636+0000 7ff643400640 1 -- 172.21.15.40:0/1280475695 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff644022020 con 0x7ff64c15cfb0 2024-04-26T23:07:28.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.851+0000 7ff651200640 1 -- 172.21.15.40:0/1280475695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff64c15c7b0 con 0x7ff64c15cfb0 2024-04-26T23:07:28.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.852+0000 7ff643400640 1 -- 172.21.15.40:0/1280475695 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v79) ==== 80+0+0 (secure 0 0 0) 0x7ff644010040 con 0x7ff64c15cfb0 2024-04-26T23:07:28.852 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-04-26T23:07:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.854+0000 7ff651200640 1 -- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff62806c920 msgr2=0x7ff62806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.854+0000 7ff651200640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff62806c920 0x7ff62806ede0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff63c001500 tx=0x7ff63c009040 comp rx=0 tx=0).stop 2024-04-26T23:07:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.854+0000 7ff651200640 1 -- 172.21.15.40:0/1280475695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff64c15cfb0 msgr2=0x7ff64c0c9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.854+0000 7ff651200640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff64c15cfb0 0x7ff64c0c9ce0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7ff644009e70 tx=0x7ff64400c490 comp rx=0 tx=0).stop 2024-04-26T23:07:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.855+0000 7ff651200640 1 -- 172.21.15.40:0/1280475695 shutdown_connections 2024-04-26T23:07:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.855+0000 7ff651200640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff62806c920 0x7ff62806ede0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.855+0000 7ff651200640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff64c15cfb0 0x7ff64c0c9ce0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.855+0000 7ff651200640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff64c0d0b10 0x7ff64c150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.855+0000 7ff651200640 1 --2- 172.21.15.40:0/1280475695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff64c070900 0x7ff64c15d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.855+0000 7ff651200640 1 -- 172.21.15.40:0/1280475695 >> 172.21.15.40:0/1280475695 conn(0x7ff64c0d2c10 msgr2=0x7ff64c0b5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.855+0000 7ff651200640 1 -- 172.21.15.40:0/1280475695 shutdown_connections 2024-04-26T23:07:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:28.855+0000 7ff651200640 1 -- 172.21.15.40:0/1280475695 wait complete. 2024-04-26T23:07:28.874 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T23:07:28.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:28.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-26T23:07:28.974 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:07:28.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:28.974 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T23:07:29.024 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:07:29.024 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:07:29.024 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:07:29.083 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:07:29.083 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:07:29.083 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:07:29.084 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:07:29.145 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:07:29.145 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:07:29.146 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:07:29.146 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:07:29.203 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:07:29.203 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:29.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.426+0000 7fe526000640 1 Processor -- start 2024-04-26T23:07:29.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.426+0000 7fe526000640 1 -- start start 2024-04-26T23:07:29.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.428+0000 7fe526000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe520145970 0x7fe5201612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:29.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.428+0000 7fe526000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe520146060 0x7fe5200b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:29.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.428+0000 7fe526000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5200b59f0 con 0x7fe5200b3fb0 2024-04-26T23:07:29.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.428+0000 7fe526000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe5200ceed0 con 0x7fe520146060 2024-04-26T23:07:29.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.428+0000 7fe526000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe5201617f0 con 0x7fe520145970 2024-04-26T23:07:29.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.429+0000 7fe517e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe520146060 0x7fe5200b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:29.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.429+0000 7fe517e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe520146060 0x7fe5200b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:42380/0 (socket says 172.21.15.40:42380) 2024-04-26T23:07:29.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.429+0000 7fe517e00640 1 -- 172.21.15.40:0/2298547376 learned_addr learned my addr 172.21.15.40:0/2298547376 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:29.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.429+0000 7fe524c00640 1 --2- 172.21.15.40:0/2298547376 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe520145970 0x7fe5201612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:29.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.430+0000 7fe517e00640 1 -- 172.21.15.40:0/2298547376 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe520145970 msgr2=0x7fe5201612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:29.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.430+0000 7fe517e00640 1 --2- 172.21.15.40:0/2298547376 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe520145970 0x7fe5201612b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:29.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.430+0000 7fe517e00640 1 -- 172.21.15.40:0/2298547376 >> v1:172.21.15.103:6789/0 conn(0x7fe5200b3fb0 legacy=0x7fe5200d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:29.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.430+0000 7fe517e00640 1 -- 172.21.15.40:0/2298547376 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5200b5f40 con 0x7fe520146060 2024-04-26T23:07:29.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.431+0000 7fe517e00640 1 --2- 172.21.15.40:0/2298547376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520146060 0x7fe5200b3a70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe508009870 tx=0x7fe50802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=245d8068dbbd404e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:29.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.431+0000 7fe51fe00640 1 -- 172.21.15.40:0/2298547376 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508037070 con 0x7fe520146060 2024-04-26T23:07:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.432+0000 7fe51fe00640 1 -- 172.21.15.40:0/2298547376 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe508032440 con 0x7fe520146060 2024-04-26T23:07:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.432+0000 7fe51fe00640 1 -- 172.21.15.40:0/2298547376 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508032810 con 0x7fe520146060 2024-04-26T23:07:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.432+0000 7fe526000640 1 -- 172.21.15.40:0/2298547376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520146060 msgr2=0x7fe5200b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.432+0000 7fe526000640 1 --2- 172.21.15.40:0/2298547376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520146060 0x7fe5200b3a70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe508009870 tx=0x7fe50802a270 comp rx=0 tx=0).stop 2024-04-26T23:07:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.432+0000 7fe526000640 1 -- 172.21.15.40:0/2298547376 shutdown_connections 2024-04-26T23:07:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.432+0000 7fe526000640 1 --2- 172.21.15.40:0/2298547376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520146060 0x7fe5200b3a70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.432+0000 7fe526000640 1 --2- 172.21.15.40:0/2298547376 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe520145970 0x7fe5201612b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.432+0000 7fe526000640 1 -- 172.21.15.40:0/2298547376 >> 172.21.15.40:0/2298547376 conn(0x7fe520075eb0 msgr2=0x7fe520068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.432+0000 7fe526000640 1 -- 172.21.15.40:0/2298547376 shutdown_connections 2024-04-26T23:07:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.432+0000 7fe526000640 1 -- 172.21.15.40:0/2298547376 wait complete. 2024-04-26T23:07:29.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.434+0000 7fe526000640 1 Processor -- start 2024-04-26T23:07:29.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.434+0000 7fe526000640 1 -- start start 2024-04-26T23:07:29.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.435+0000 7fe526000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520145970 0x7fe52013cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:29.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.435+0000 7fe526000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe520146060 0x7fe520159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:29.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.435+0000 7fe526000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe520158190 0x7fe52015ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:29.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.435+0000 7fe526000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe5200cf3e0 con 0x7fe520146060 2024-04-26T23:07:29.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.435+0000 7fe526000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe5200cf100 con 0x7fe520145970 2024-04-26T23:07:29.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.435+0000 7fe526000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe520145d50 con 0x7fe520158190 2024-04-26T23:07:29.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.435+0000 7fe517e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe520146060 0x7fe520159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:29.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.435+0000 7fe517e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe520146060 0x7fe520159d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41424/0 (socket says 172.21.15.40:41424) 2024-04-26T23:07:29.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.435+0000 7fe517e00640 1 -- 172.21.15.40:0/716810575 learned_addr learned my addr 172.21.15.40:0/716810575 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.436+0000 7fe525600640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe520158190 0x7fe52015ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.436+0000 7fe524c00640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520145970 0x7fe52013cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.436+0000 7fe517e00640 1 -- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe520158190 msgr2=0x7fe52015ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.436+0000 7fe517e00640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe520158190 0x7fe52015ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.436+0000 7fe517e00640 1 -- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520145970 msgr2=0x7fe52013cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.436+0000 7fe517e00640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520145970 0x7fe52013cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.436+0000 7fe517e00640 1 -- 172.21.15.40:0/716810575 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52013d450 con 0x7fe520146060 2024-04-26T23:07:29.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.436+0000 7fe517e00640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe520146060 0x7fe520159d50 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fe508009990 tx=0x7fe5080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.437+0000 7fe51d600640 1 -- 172.21.15.40:0/716810575 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508043070 con 0x7fe520146060 2024-04-26T23:07:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.437+0000 7fe51d600640 1 -- 172.21.15.40:0/716810575 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe508038040 con 0x7fe520146060 2024-04-26T23:07:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.437+0000 7fe526000640 1 -- 172.21.15.40:0/716810575 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5201586a0 con 0x7fe520146060 2024-04-26T23:07:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.437+0000 7fe51d600640 1 -- 172.21.15.40:0/716810575 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508037050 con 0x7fe520146060 2024-04-26T23:07:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.437+0000 7fe526000640 1 -- 172.21.15.40:0/716810575 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe520158c60 con 0x7fe520146060 2024-04-26T23:07:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.438+0000 7fe526000640 1 -- 172.21.15.40:0/716810575 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe520069370 con 0x7fe520146060 2024-04-26T23:07:29.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.439+0000 7fe51d600640 1 -- 172.21.15.40:0/716810575 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe508046070 con 0x7fe520146060 2024-04-26T23:07:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.440+0000 7fe51d600640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe4fc06c8f0 0x7fe4fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.440+0000 7fe51d600640 1 -- 172.21.15.40:0/716810575 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fe5080abcf0 con 0x7fe520146060 2024-04-26T23:07:29.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.440+0000 7fe524c00640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe4fc06c8f0 0x7fe4fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:29.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.442+0000 7fe524c00640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe4fc06c8f0 0x7fe4fc06edb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe510009fd0 tx=0x7fe51000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.444+0000 7fe51d600640 1 -- 172.21.15.40:0/716810575 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe508079f80 con 0x7fe520146060 2024-04-26T23:07:29.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.693+0000 7fe526000640 1 -- 172.21.15.40:0/716810575 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5200d1400 con 0x7fe520146060 2024-04-26T23:07:29.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.695+0000 7fe51d600640 1 -- 172.21.15.40:0/716810575 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5227 (secure 0 0 0) 0x7fe508079880 con 0x7fe520146060 2024-04-26T23:07:29.695 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:29.695 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/755627718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":755627718},{"type":"v1","addr":"172.21.15.103:6837","nonce":755627718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:06:57.853769+0000","modified":"2024-04-26T23:07:26.981321+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7446,"mds_1":7443,"mds_2":5965},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5965":{"gid":5965,"name":"d","rank":2,"incarnation":78,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6836/2914603370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2914603370},{"type":"v1","addr":"172.21.15.103:6836","nonce":2914603370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7443":{"gid":7443,"name":"a","rank":1,"incarnation":76,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/216893598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":216893598},{"type":"v1","addr":"172.21.15.40:6834","nonce":216893598}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7446":{"gid":7446,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4227275373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":4227275373},{"type":"v1","addr":"172.21.15.40:6835","nonce":4227275373}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7446,"qdb_cluster":[7446,7443,5965]},"id":5}]} 2024-04-26T23:07:29.695 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 82 2024-04-26T23:07:29.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.697+0000 7fe526000640 1 -- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe4fc06c8f0 msgr2=0x7fe4fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:29.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.697+0000 7fe526000640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe4fc06c8f0 0x7fe4fc06edb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe510009fd0 tx=0x7fe51000b040 comp rx=0 tx=0).stop 2024-04-26T23:07:29.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.697+0000 7fe526000640 1 -- 172.21.15.40:0/716810575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe520146060 msgr2=0x7fe520159d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:29.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.697+0000 7fe526000640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe520146060 0x7fe520159d50 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fe508009990 tx=0x7fe5080046c0 comp rx=0 tx=0).stop 2024-04-26T23:07:29.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.697+0000 7fe526000640 1 -- 172.21.15.40:0/716810575 shutdown_connections 2024-04-26T23:07:29.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.697+0000 7fe526000640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe4fc06c8f0 0x7fe4fc06edb0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:29.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.697+0000 7fe526000640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe520158190 0x7fe52015ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:29.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.697+0000 7fe526000640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe520146060 0x7fe520159d50 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:29.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.698+0000 7fe526000640 1 --2- 172.21.15.40:0/716810575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520145970 0x7fe52013cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:29.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.698+0000 7fe526000640 1 -- 172.21.15.40:0/716810575 >> 172.21.15.40:0/716810575 conn(0x7fe520075eb0 msgr2=0x7fe520071d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.698+0000 7fe526000640 1 -- 172.21.15.40:0/716810575 shutdown_connections 2024-04-26T23:07:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.698+0000 7fe526000640 1 -- 172.21.15.40:0/716810575 wait complete. 2024-04-26T23:07:29.717 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:07:29.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.929+0000 7fb165c00640 1 Processor -- start 2024-04-26T23:07:29.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.929+0000 7fb165c00640 1 -- start start 2024-04-26T23:07:29.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.931+0000 7fb165c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb160071fe0 0x7fb1600afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:29.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.931+0000 7fb165c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1600d0820 con 0x7fb1600af7a0 2024-04-26T23:07:29.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.931+0000 7fb165c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1601614c0 con 0x7fb16015ce60 2024-04-26T23:07:29.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.931+0000 7fb165c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb1600cc450 con 0x7fb160071fe0 2024-04-26T23:07:29.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.931+0000 7fb15f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb1600af7a0 0x7fb160073760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34262/0 (socket says 172.21.15.40:34262) 2024-04-26T23:07:29.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.931+0000 7fb15f400640 1 -- 172.21.15.40:0/3855032427 learned_addr learned my addr 172.21.15.40:0/3855032427 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:29.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.931+0000 7fb165200640 1 --2- 172.21.15.40:0/3855032427 >> v2:172.21.15.40:3300/0 conn(0x7fb160071fe0 0x7fb1600afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:29.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.932+0000 7fb15ea00640 1 -- 172.21.15.40:0/3855032427 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2944267128 0 0) 0x7fb1600d0820 con 0x7fb1600af7a0 2024-04-26T23:07:29.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.932+0000 7fb165200640 1 -- 172.21.15.40:0/3855032427 >> v1:172.21.15.103:6789/0 conn(0x7fb16015ce60 legacy=0x7fb1600ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:29.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.932+0000 7fb165200640 1 -- 172.21.15.40:0/3855032427 >> v1:172.21.15.40:6789/0 conn(0x7fb1600af7a0 legacy=0x7fb160073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:29.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.932+0000 7fb165200640 1 -- 172.21.15.40:0/3855032427 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb160161760 con 0x7fb160071fe0 2024-04-26T23:07:29.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.933+0000 7fb165200640 1 --2- 172.21.15.40:0/3855032427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb160071fe0 0x7fb1600afc60 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fb158009d90 tx=0x7fb15802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95e9ad65e984705 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:29.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.933+0000 7fb15ea00640 1 -- 172.21.15.40:0/3855032427 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb158031600 con 0x7fb160071fe0 2024-04-26T23:07:29.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.933+0000 7fb15ea00640 1 -- 172.21.15.40:0/3855032427 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1580317a0 con 0x7fb160071fe0 2024-04-26T23:07:29.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.933+0000 7fb15ea00640 1 -- 172.21.15.40:0/3855032427 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb158031b70 con 0x7fb160071fe0 2024-04-26T23:07:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.934+0000 7fb165c00640 1 -- 172.21.15.40:0/3855032427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb160071fe0 msgr2=0x7fb1600afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.934+0000 7fb165c00640 1 --2- 172.21.15.40:0/3855032427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb160071fe0 0x7fb1600afc60 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fb158009d90 tx=0x7fb15802a610 comp rx=0 tx=0).stop 2024-04-26T23:07:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.934+0000 7fb165c00640 1 -- 172.21.15.40:0/3855032427 shutdown_connections 2024-04-26T23:07:29.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.934+0000 7fb165c00640 1 --2- 172.21.15.40:0/3855032427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb160071fe0 0x7fb1600afc60 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:29.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.934+0000 7fb165c00640 1 -- 172.21.15.40:0/3855032427 >> 172.21.15.40:0/3855032427 conn(0x7fb1600d0a10 msgr2=0x7fb1600d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:29.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.934+0000 7fb165c00640 1 -- 172.21.15.40:0/3855032427 shutdown_connections 2024-04-26T23:07:29.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.934+0000 7fb165c00640 1 -- 172.21.15.40:0/3855032427 wait complete. 2024-04-26T23:07:29.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.936+0000 7fb165c00640 1 Processor -- start 2024-04-26T23:07:29.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.936+0000 7fb165c00640 1 -- start start 2024-04-26T23:07:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.937+0000 7fb165c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb160071fe0 0x7fb16014e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.937+0000 7fb165c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1600af7a0 0x7fb1601559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.937+0000 7fb165c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb16015ce60 0x7fb16015d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.937+0000 7fb165c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb1600cc1f0 con 0x7fb160071fe0 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.937+0000 7fb165c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb1600cc070 con 0x7fb16015ce60 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.937+0000 7fb165c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb160073e70 con 0x7fb1600af7a0 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.937+0000 7fb15fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb160071fe0 0x7fb16014e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.938+0000 7fb15fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb160071fe0 0x7fb16014e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41452/0 (socket says 172.21.15.40:41452) 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.938+0000 7fb15fe00640 1 -- 172.21.15.40:0/3739963198 learned_addr learned my addr 172.21.15.40:0/3739963198 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.938+0000 7fb15f400640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1600af7a0 0x7fb1601559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.938+0000 7fb165200640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb16015ce60 0x7fb16015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.938+0000 7fb15fe00640 1 -- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1600af7a0 msgr2=0x7fb1601559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.938+0000 7fb15fe00640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1600af7a0 0x7fb1601559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.938+0000 7fb15fe00640 1 -- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb16015ce60 msgr2=0x7fb16015d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.938+0000 7fb15fe00640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb16015ce60 0x7fb16015d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.938+0000 7fb15fe00640 1 -- 172.21.15.40:0/3739963198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1601563b0 con 0x7fb160071fe0 2024-04-26T23:07:29.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.938+0000 7fb15f400640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1600af7a0 0x7fb1601559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:29.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.938+0000 7fb15fe00640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb160071fe0 0x7fb16014e580 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fb15000b3f0 tx=0x7fb15000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:29.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.939+0000 7fb157e00640 1 -- 172.21.15.40:0/3739963198 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb150015020 con 0x7fb160071fe0 2024-04-26T23:07:29.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.939+0000 7fb157e00640 1 -- 172.21.15.40:0/3739963198 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb150011830 con 0x7fb160071fe0 2024-04-26T23:07:29.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.939+0000 7fb157e00640 1 -- 172.21.15.40:0/3739963198 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb150011c20 con 0x7fb160071fe0 2024-04-26T23:07:29.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.939+0000 7fb165c00640 1 -- 172.21.15.40:0/3739963198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb160156690 con 0x7fb160071fe0 2024-04-26T23:07:29.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.939+0000 7fb165c00640 1 -- 172.21.15.40:0/3739963198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb160155f10 con 0x7fb160071fe0 2024-04-26T23:07:29.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.940+0000 7fb165c00640 1 -- 172.21.15.40:0/3739963198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1600c58b0 con 0x7fb160071fe0 2024-04-26T23:07:29.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.941+0000 7fb157e00640 1 -- 172.21.15.40:0/3739963198 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb150018020 con 0x7fb160071fe0 2024-04-26T23:07:29.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.942+0000 7fb157e00640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb14006c8f0 0x7fb14006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:29.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.942+0000 7fb157e00640 1 -- 172.21.15.40:0/3739963198 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fb15008ba20 con 0x7fb160071fe0 2024-04-26T23:07:29.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.943+0000 7fb15f400640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb14006c8f0 0x7fb14006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:29.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.944+0000 7fb15f400640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb14006c8f0 0x7fb14006edb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb1600c7a90 tx=0x7fb14c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:29.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:29.947+0000 7fb157e00640 1 -- 172.21.15.40:0/3739963198 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fb15001f030 con 0x7fb160071fe0 2024-04-26T23:07:30.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.154+0000 7fb165c00640 1 -- 172.21.15.40:0/3739963198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb160069370 con 0x7fb160071fe0 2024-04-26T23:07:30.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.157+0000 7fb157e00640 1 -- 172.21.15.40:0/3739963198 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+16909 (secure 0 0 0) 0x7fb1500172c0 con 0x7fb160071fe0 2024-04-26T23:07:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:30.158 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":79,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:06:59.420463+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:06:55.738277+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-26T23:06:56.838617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:07:30.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.159+0000 7fb165c00640 1 -- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb14006c8f0 msgr2=0x7fb14006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:30.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.159+0000 7fb165c00640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb14006c8f0 0x7fb14006edb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb1600c7a90 tx=0x7fb14c009040 comp rx=0 tx=0).stop 2024-04-26T23:07:30.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.159+0000 7fb165c00640 1 -- 172.21.15.40:0/3739963198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb160071fe0 msgr2=0x7fb16014e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:30.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.159+0000 7fb165c00640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb160071fe0 0x7fb16014e580 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fb15000b3f0 tx=0x7fb15000b8b0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.160+0000 7fb165c00640 1 -- 172.21.15.40:0/3739963198 shutdown_connections 2024-04-26T23:07:30.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.160+0000 7fb165c00640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb14006c8f0 0x7fb14006edb0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.160+0000 7fb165c00640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb16015ce60 0x7fb16015d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.160+0000 7fb165c00640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb1600af7a0 0x7fb1601559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.160+0000 7fb165c00640 1 --2- 172.21.15.40:0/3739963198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb160071fe0 0x7fb16014e580 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.160+0000 7fb165c00640 1 -- 172.21.15.40:0/3739963198 >> 172.21.15.40:0/3739963198 conn(0x7fb1600d0a10 msgr2=0x7fb1600b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:30.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.160+0000 7fb165c00640 1 -- 172.21.15.40:0/3739963198 shutdown_connections 2024-04-26T23:07:30.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.160+0000 7fb165c00640 1 -- 172.21.15.40:0/3739963198 wait complete. 2024-04-26T23:07:30.180 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:06:55.738277+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:06:56.838617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:07:30.180 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:07:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.386+0000 7f13bb200640 1 Processor -- start 2024-04-26T23:07:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.386+0000 7f13bb200640 1 -- start start 2024-04-26T23:07:30.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.387+0000 7f13bb200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f13b415f060 0x7f13b415f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:30.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.388+0000 7f13bb200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f13b40d1f70 0x7f13b40d3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:30.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.388+0000 7f13bb200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13b40d5c20 con 0x7f13b40d3640 2024-04-26T23:07:30.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.388+0000 7f13bb200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f13b40c99b0 con 0x7f13b40d1f70 2024-04-26T23:07:30.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.388+0000 7f13bb200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f13b40cfc40 con 0x7f13b415f060 2024-04-26T23:07:30.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.388+0000 7f13ba800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f13b40d1f70 0x7f13b40d3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:30.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.388+0000 7f13b9400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f13b40d3640 0x7f13b40d1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34276/0 (socket says 172.21.15.40:34276) 2024-04-26T23:07:30.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.388+0000 7f13b9400640 1 -- 172.21.15.40:0/2369369679 learned_addr learned my addr 172.21.15.40:0/2369369679 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:30.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.388+0000 7f13ba800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f13b40d1f70 0x7f13b40d3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41458/0 (socket says 172.21.15.40:41458) 2024-04-26T23:07:30.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.388+0000 7f13b9e00640 1 --2- 172.21.15.40:0/2369369679 >> v2:172.21.15.103:3300/0 conn(0x7f13b415f060 0x7f13b415f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:30.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.388+0000 7f13ba800640 1 -- 172.21.15.40:0/2369369679 >> v2:172.21.15.103:3300/0 conn(0x7f13b415f060 msgr2=0x7f13b415f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:30.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.388+0000 7f13ba800640 1 --2- 172.21.15.40:0/2369369679 >> v2:172.21.15.103:3300/0 conn(0x7f13b415f060 0x7f13b415f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.388+0000 7f13ba800640 1 -- 172.21.15.40:0/2369369679 >> v1:172.21.15.40:6789/0 conn(0x7f13b40d3640 legacy=0x7f13b40d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:30.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.388+0000 7f13ba800640 1 -- 172.21.15.40:0/2369369679 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b40cff20 con 0x7f13b40d1f70 2024-04-26T23:07:30.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.390+0000 7f13ba800640 1 --2- 172.21.15.40:0/2369369679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b40d1f70 0x7f13b40d3a80 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f13b408d7b0 tx=0x7f13b002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d1764b4c16904b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:30.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.390+0000 7f13b8a00640 1 -- 172.21.15.40:0/2369369679 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b0037070 con 0x7f13b40d1f70 2024-04-26T23:07:30.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.390+0000 7f13b8a00640 1 -- 172.21.15.40:0/2369369679 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13b0032440 con 0x7f13b40d1f70 2024-04-26T23:07:30.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.390+0000 7f13b8a00640 1 -- 172.21.15.40:0/2369369679 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b0032810 con 0x7f13b40d1f70 2024-04-26T23:07:30.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.390+0000 7f13bb200640 1 -- 172.21.15.40:0/2369369679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b40d1f70 msgr2=0x7f13b40d3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:30.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.390+0000 7f13bb200640 1 --2- 172.21.15.40:0/2369369679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b40d1f70 0x7f13b40d3a80 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f13b408d7b0 tx=0x7f13b002a1a0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.391+0000 7f13bb200640 1 -- 172.21.15.40:0/2369369679 shutdown_connections 2024-04-26T23:07:30.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.391+0000 7f13bb200640 1 --2- 172.21.15.40:0/2369369679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b40d1f70 0x7f13b40d3a80 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.391+0000 7f13bb200640 1 --2- 172.21.15.40:0/2369369679 >> v2:172.21.15.103:3300/0 conn(0x7f13b415f060 0x7f13b415f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.391+0000 7f13bb200640 1 -- 172.21.15.40:0/2369369679 >> 172.21.15.40:0/2369369679 conn(0x7f13b40d4e50 msgr2=0x7f13b40d5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:30.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.391+0000 7f13bb200640 1 -- 172.21.15.40:0/2369369679 shutdown_connections 2024-04-26T23:07:30.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.391+0000 7f13bb200640 1 -- 172.21.15.40:0/2369369679 wait complete. 2024-04-26T23:07:30.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.393+0000 7f13bb200640 1 Processor -- start 2024-04-26T23:07:30.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.393+0000 7f13bb200640 1 -- start start 2024-04-26T23:07:30.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.395+0000 7f13bb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b40d1f70 0x7f13b415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:30.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.395+0000 7f13bb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b40d3640 0x7f13b41f6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:30.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.395+0000 7f13bb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13b415f060 0x7f13b40cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:30.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.395+0000 7f13bb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f13b40d0ce0 con 0x7f13b40d1f70 2024-04-26T23:07:30.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.395+0000 7f13bb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13b40d04e0 con 0x7f13b40d3640 2024-04-26T23:07:30.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.395+0000 7f13bb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f13b40cc420 con 0x7f13b415f060 2024-04-26T23:07:30.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.395+0000 7f13b9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b40d1f70 0x7f13b415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:30.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.395+0000 7f13b9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b40d1f70 0x7f13b415ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41464/0 (socket says 172.21.15.40:41464) 2024-04-26T23:07:30.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.395+0000 7f13b9e00640 1 -- 172.21.15.40:0/2456472691 learned_addr learned my addr 172.21.15.40:0/2456472691 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:30.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.395+0000 7f13b9e00640 1 -- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13b415f060 msgr2=0x7f13b40cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:30.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.395+0000 7f13b9e00640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13b415f060 0x7f13b40cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.396+0000 7f13b9400640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b40d3640 0x7f13b41f6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:30.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.396+0000 7f13b9e00640 1 -- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b40d3640 msgr2=0x7f13b41f6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:30.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.396+0000 7f13b9e00640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b40d3640 0x7f13b41f6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.396+0000 7f13b9e00640 1 -- 172.21.15.40:0/2456472691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b40cc6c0 con 0x7f13b40d1f70 2024-04-26T23:07:30.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.396+0000 7f13b9400640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b40d3640 0x7f13b41f6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:30.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.396+0000 7f13b9e00640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b40d1f70 0x7f13b415ccb0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f13a800c7f0 tx=0x7f13a800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:30.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.396+0000 7f13ae000640 1 -- 172.21.15.40:0/2456472691 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8015070 con 0x7f13b40d1f70 2024-04-26T23:07:30.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.396+0000 7f13ae000640 1 -- 172.21.15.40:0/2456472691 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a8004220 con 0x7f13b40d1f70 2024-04-26T23:07:30.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.396+0000 7f13ae000640 1 -- 172.21.15.40:0/2456472691 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8007620 con 0x7f13b40d1f70 2024-04-26T23:07:30.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.396+0000 7f13bb200640 1 -- 172.21.15.40:0/2456472691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b40cc9a0 con 0x7f13b40d1f70 2024-04-26T23:07:30.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.397+0000 7f13bb200640 1 -- 172.21.15.40:0/2456472691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b40ccf60 con 0x7f13b40d1f70 2024-04-26T23:07:30.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.398+0000 7f13bb200640 1 -- 172.21.15.40:0/2456472691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13b4069a80 con 0x7f13b40d1f70 2024-04-26T23:07:30.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.399+0000 7f13ae000640 1 -- 172.21.15.40:0/2456472691 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f13a80079d0 con 0x7f13b40d1f70 2024-04-26T23:07:30.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.400+0000 7f13ae000640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f139806c940 0x7f139806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:30.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.400+0000 7f13ae000640 1 -- 172.21.15.40:0/2456472691 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f13a808c090 con 0x7f13b40d1f70 2024-04-26T23:07:30.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.401+0000 7f13b9400640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f139806c940 0x7f139806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:30.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.402+0000 7f13b9400640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f139806c940 0x7f139806ee00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f13b40ce250 tx=0x7f13a4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:30.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.405+0000 7f13ae000640 1 -- 172.21.15.40:0/2456472691 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f13a805a310 con 0x7f13b40d1f70 2024-04-26T23:07:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.622+0000 7f13bb200640 1 -- 172.21.15.40:0/2456472691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f13b4068e00 con 0x7f13b40d1f70 2024-04-26T23:07:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.623+0000 7f13ae000640 1 -- 172.21.15.40:0/2456472691 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) ==== 93+0+10 (secure 0 0 0) 0x7f13a8059c10 con 0x7f13b40d1f70 2024-04-26T23:07:30.624 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:07:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.626+0000 7f13bb200640 1 -- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f139806c940 msgr2=0x7f139806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.626+0000 7f13bb200640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f139806c940 0x7f139806ee00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f13b40ce250 tx=0x7f13a4009040 comp rx=0 tx=0).stop 2024-04-26T23:07:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.626+0000 7f13bb200640 1 -- 172.21.15.40:0/2456472691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b40d1f70 msgr2=0x7f13b415ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.626+0000 7f13bb200640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b40d1f70 0x7f13b415ccb0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f13a800c7f0 tx=0x7f13a800ccb0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.626+0000 7f13bb200640 1 -- 172.21.15.40:0/2456472691 shutdown_connections 2024-04-26T23:07:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.626+0000 7f13bb200640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f139806c940 0x7f139806ee00 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.626+0000 7f13bb200640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13b415f060 0x7f13b40cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.626+0000 7f13bb200640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b40d3640 0x7f13b41f6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.626+0000 7f13bb200640 1 --2- 172.21.15.40:0/2456472691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b40d1f70 0x7f13b415ccb0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.626+0000 7f13bb200640 1 -- 172.21.15.40:0/2456472691 >> 172.21.15.40:0/2456472691 conn(0x7f13b40d4e50 msgr2=0x7f13b406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.627+0000 7f13bb200640 1 -- 172.21.15.40:0/2456472691 shutdown_connections 2024-04-26T23:07:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.627+0000 7f13bb200640 1 -- 172.21.15.40:0/2456472691 wait complete. 2024-04-26T23:07:30.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:07:30.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.861+0000 7f9444a00640 1 Processor -- start 2024-04-26T23:07:30.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.862+0000 7f9444a00640 1 -- start start 2024-04-26T23:07:30.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.863+0000 7f9444a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94400ceff0 0x7f94400cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:30.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.863+0000 7f9444a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9440071300 0x7f94400cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:30.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.863+0000 7f9444a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94400d3950 con 0x7f9440071d40 2024-04-26T23:07:30.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.863+0000 7f9444a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f9440160ab0 con 0x7f94400ceff0 2024-04-26T23:07:30.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.863+0000 7f9444a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f94400cf3d0 con 0x7f9440071300 2024-04-26T23:07:30.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.863+0000 7f943e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94400ceff0 0x7f94400cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:30.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.864+0000 7f943e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94400ceff0 0x7f94400cf6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41466/0 (socket says 172.21.15.40:41466) 2024-04-26T23:07:30.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.864+0000 7f943e800640 1 -- 172.21.15.40:0/1832425254 learned_addr learned my addr 172.21.15.40:0/1832425254 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:30.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.864+0000 7f943f200640 1 --2- 172.21.15.40:0/1832425254 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9440071300 0x7f94400cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:30.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.864+0000 7f943e800640 1 -- 172.21.15.40:0/1832425254 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9440071300 msgr2=0x7f94400cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:30.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.864+0000 7f943e800640 1 --2- 172.21.15.40:0/1832425254 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9440071300 0x7f94400cfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.864+0000 7f943e800640 1 -- 172.21.15.40:0/1832425254 >> v1:172.21.15.103:6789/0 conn(0x7f9440071d40 legacy=0x7f9440070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:30.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.864+0000 7f943e800640 1 -- 172.21.15.40:0/1832425254 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9440070110 con 0x7f94400ceff0 2024-04-26T23:07:30.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.865+0000 7f943e800640 1 --2- 172.21.15.40:0/1832425254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94400ceff0 0x7f94400cf6a0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f94300098a0 tx=0x7f943002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba286aa27e59161 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:30.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.866+0000 7f943d400640 1 -- 172.21.15.40:0/1832425254 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9430037070 con 0x7f94400ceff0 2024-04-26T23:07:30.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.866+0000 7f943d400640 1 -- 172.21.15.40:0/1832425254 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9430032440 con 0x7f94400ceff0 2024-04-26T23:07:30.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.866+0000 7f943d400640 1 -- 172.21.15.40:0/1832425254 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9430032810 con 0x7f94400ceff0 2024-04-26T23:07:30.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.866+0000 7f9444a00640 1 -- 172.21.15.40:0/1832425254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94400ceff0 msgr2=0x7f94400cf6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:30.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.866+0000 7f9444a00640 1 --2- 172.21.15.40:0/1832425254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94400ceff0 0x7f94400cf6a0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f94300098a0 tx=0x7f943002a250 comp rx=0 tx=0).stop 2024-04-26T23:07:30.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.867+0000 7f9444a00640 1 -- 172.21.15.40:0/1832425254 shutdown_connections 2024-04-26T23:07:30.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.867+0000 7f9444a00640 1 --2- 172.21.15.40:0/1832425254 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9440071300 0x7f94400cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.867+0000 7f9444a00640 1 --2- 172.21.15.40:0/1832425254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94400ceff0 0x7f94400cf6a0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.867+0000 7f9444a00640 1 -- 172.21.15.40:0/1832425254 >> 172.21.15.40:0/1832425254 conn(0x7f94400d2c90 msgr2=0x7f94400d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:30.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.867+0000 7f9444a00640 1 -- 172.21.15.40:0/1832425254 shutdown_connections 2024-04-26T23:07:30.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.867+0000 7f9444a00640 1 -- 172.21.15.40:0/1832425254 wait complete. 2024-04-26T23:07:30.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.868+0000 7f9444a00640 1 Processor -- start 2024-04-26T23:07:30.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.869+0000 7f9444a00640 1 -- start start 2024-04-26T23:07:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.869+0000 7f9444a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440071300 0x7f9440150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f9444a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9440071d40 0x7f94401514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f9444a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94400ceff0 0x7f94401f5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f9444a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f94400719b0 con 0x7f9440071300 2024-04-26T23:07:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f9444a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f94400706d0 con 0x7f94400ceff0 2024-04-26T23:07:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f9444a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f94400cd6c0 con 0x7f9440071d40 2024-04-26T23:07:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f943e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440071300 0x7f9440150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f943e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440071300 0x7f9440150cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41478/0 (socket says 172.21.15.40:41478) 2024-04-26T23:07:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f943e800640 1 -- 172.21.15.40:0/1865113369 learned_addr learned my addr 172.21.15.40:0/1865113369 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:30.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f943e800640 1 -- 172.21.15.40:0/1865113369 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9440071d40 msgr2=0x7f94401514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:30.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f943e800640 1 --2- 172.21.15.40:0/1865113369 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9440071d40 0x7f94401514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f943f200640 1 --2- 172.21.15.40:0/1865113369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94400ceff0 0x7f94401f5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f943e800640 1 -- 172.21.15.40:0/1865113369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94400ceff0 msgr2=0x7f94401f5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f943e800640 1 --2- 172.21.15.40:0/1865113369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94400ceff0 0x7f94401f5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.870+0000 7f943e800640 1 -- 172.21.15.40:0/1865113369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9440151200 con 0x7f9440071300 2024-04-26T23:07:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.871+0000 7f943e800640 1 --2- 172.21.15.40:0/1865113369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440071300 0x7f9440150cc0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f9430000c00 tx=0x7f94300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.871+0000 7f9436a00640 1 -- 172.21.15.40:0/1865113369 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9430037040 con 0x7f9440071300 2024-04-26T23:07:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.871+0000 7f9436a00640 1 -- 172.21.15.40:0/1865113369 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9430004230 con 0x7f9440071300 2024-04-26T23:07:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.871+0000 7f9436a00640 1 -- 172.21.15.40:0/1865113369 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9430003610 con 0x7f9440071300 2024-04-26T23:07:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.871+0000 7f9444a00640 1 -- 172.21.15.40:0/1865113369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94400ca430 con 0x7f9440071300 2024-04-26T23:07:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.871+0000 7f9444a00640 1 -- 172.21.15.40:0/1865113369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94400ca9f0 con 0x7f9440071300 2024-04-26T23:07:30.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.872+0000 7f9444a00640 1 -- 172.21.15.40:0/1865113369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9440069370 con 0x7f9440071300 2024-04-26T23:07:30.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.873+0000 7f9436a00640 1 -- 172.21.15.40:0/1865113369 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f94300037b0 con 0x7f9440071300 2024-04-26T23:07:30.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.874+0000 7f9436a00640 1 --2- 172.21.15.40:0/1865113369 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f941c06c8f0 0x7f941c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:30.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.874+0000 7f9436a00640 1 -- 172.21.15.40:0/1865113369 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f94300aca60 con 0x7f9440071300 2024-04-26T23:07:30.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.875+0000 7f943de00640 1 --2- 172.21.15.40:0/1865113369 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f941c06c8f0 0x7f941c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:30.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.876+0000 7f943de00640 1 --2- 172.21.15.40:0/1865113369 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f941c06c8f0 0x7f941c06edb0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f94400cbed0 tx=0x7f94280079b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:30.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:30.879+0000 7f9436a00640 1 -- 172.21.15.40:0/1865113369 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9440151200 con 0x7f9440071300 2024-04-26T23:07:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.082+0000 7f9444a00640 1 -- 172.21.15.40:0/1865113369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f94400741b0 con 0x7f9440071300 2024-04-26T23:07:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.083+0000 7f9436a00640 1 -- 172.21.15.40:0/1865113369 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) ==== 104+0+11 (secure 0 0 0) 0x7f943007acf0 con 0x7f9440071300 2024-04-26T23:07:31.084 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:07:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.086+0000 7f9444a00640 1 -- 172.21.15.40:0/1865113369 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f941c06c8f0 msgr2=0x7f941c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.086+0000 7f9444a00640 1 --2- 172.21.15.40:0/1865113369 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f941c06c8f0 0x7f941c06edb0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f94400cbed0 tx=0x7f94280079b0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.086+0000 7f9444a00640 1 -- 172.21.15.40:0/1865113369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440071300 msgr2=0x7f9440150cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.086+0000 7f9444a00640 1 --2- 172.21.15.40:0/1865113369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440071300 0x7f9440150cc0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f9430000c00 tx=0x7f94300046c0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.086+0000 7f9444a00640 1 -- 172.21.15.40:0/1865113369 shutdown_connections 2024-04-26T23:07:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.086+0000 7f9444a00640 1 --2- 172.21.15.40:0/1865113369 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f941c06c8f0 0x7f941c06edb0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.086+0000 7f9444a00640 1 --2- 172.21.15.40:0/1865113369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94400ceff0 0x7f94401f5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.086+0000 7f9444a00640 1 --2- 172.21.15.40:0/1865113369 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9440071d40 0x7f94401514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.086+0000 7f9444a00640 1 --2- 172.21.15.40:0/1865113369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440071300 0x7f9440150cc0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.086+0000 7f9444a00640 1 -- 172.21.15.40:0/1865113369 >> 172.21.15.40:0/1865113369 conn(0x7f94400d2c90 msgr2=0x7f944006e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.086+0000 7f9444a00640 1 -- 172.21.15.40:0/1865113369 shutdown_connections 2024-04-26T23:07:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.086+0000 7f9444a00640 1 -- 172.21.15.40:0/1865113369 wait complete. 2024-04-26T23:07:31.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:07:31.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.328+0000 7f327d400640 1 Processor -- start 2024-04-26T23:07:31.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.329+0000 7f327d400640 1 -- start start 2024-04-26T23:07:31.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.330+0000 7f327d400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f327815f380 0x7f32780cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:31.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.330+0000 7f327d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f32780d0620 0x7f32780cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:31.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.330+0000 7f327d400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3278072c00 con 0x7f3278073b70 2024-04-26T23:07:31.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.330+0000 7f327d400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f32780729b0 con 0x7f32780d0620 2024-04-26T23:07:31.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.330+0000 7f327d400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3278072ee0 con 0x7f327815f380 2024-04-26T23:07:31.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.331+0000 7f3277200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f32780d0620 0x7f32780cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:31.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.331+0000 7f3277200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f32780d0620 0x7f32780cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41486/0 (socket says 172.21.15.40:41486) 2024-04-26T23:07:31.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.331+0000 7f3277200640 1 -- 172.21.15.40:0/375510583 learned_addr learned my addr 172.21.15.40:0/375510583 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:31.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.331+0000 7f3276800640 1 --2- 172.21.15.40:0/375510583 >> v2:172.21.15.103:3300/0 conn(0x7f327815f380 0x7f32780cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:31.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.331+0000 7f3277200640 1 -- 172.21.15.40:0/375510583 >> v2:172.21.15.103:3300/0 conn(0x7f327815f380 msgr2=0x7f32780cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:31.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.331+0000 7f3277200640 1 --2- 172.21.15.40:0/375510583 >> v2:172.21.15.103:3300/0 conn(0x7f327815f380 0x7f32780cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.331+0000 7f3277200640 1 -- 172.21.15.40:0/375510583 >> v1:172.21.15.103:6789/0 conn(0x7f3278073b70 legacy=0x7f32780cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:31.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.331+0000 7f3277200640 1 -- 172.21.15.40:0/375510583 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32780731c0 con 0x7f32780d0620 2024-04-26T23:07:31.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.332+0000 7f3277200640 1 --2- 172.21.15.40:0/375510583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32780d0620 0x7f32780cb980 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f32780b3820 tx=0x7f327002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f3fb764037ba94 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:31.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.333+0000 7f3275e00640 1 -- 172.21.15.40:0/375510583 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270036070 con 0x7f32780d0620 2024-04-26T23:07:31.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.333+0000 7f3275e00640 1 -- 172.21.15.40:0/375510583 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3270031600 con 0x7f32780d0620 2024-04-26T23:07:31.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.333+0000 7f3275e00640 1 -- 172.21.15.40:0/375510583 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32700319d0 con 0x7f32780d0620 2024-04-26T23:07:31.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.333+0000 7f327d400640 1 -- 172.21.15.40:0/375510583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32780d0620 msgr2=0x7f32780cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:31.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.333+0000 7f327d400640 1 --2- 172.21.15.40:0/375510583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32780d0620 0x7f32780cb980 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f32780b3820 tx=0x7f327002a6f0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.334+0000 7f327d400640 1 -- 172.21.15.40:0/375510583 shutdown_connections 2024-04-26T23:07:31.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.334+0000 7f327d400640 1 --2- 172.21.15.40:0/375510583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32780d0620 0x7f32780cb980 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.334+0000 7f327d400640 1 --2- 172.21.15.40:0/375510583 >> v2:172.21.15.103:3300/0 conn(0x7f327815f380 0x7f32780cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.334+0000 7f327d400640 1 -- 172.21.15.40:0/375510583 >> 172.21.15.40:0/375510583 conn(0x7f32780d0be0 msgr2=0x7f32780d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:31.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.334+0000 7f327d400640 1 -- 172.21.15.40:0/375510583 shutdown_connections 2024-04-26T23:07:31.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.334+0000 7f327d400640 1 -- 172.21.15.40:0/375510583 wait complete. 2024-04-26T23:07:31.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.336+0000 7f327d400640 1 Processor -- start 2024-04-26T23:07:31.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.336+0000 7f327d400640 1 -- start start 2024-04-26T23:07:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.337+0000 7f327d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32780d0620 0x7f32780b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.337+0000 7f327d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f327815f380 0x7f32780b5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.337+0000 7f327d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f327806f500 0x7f327806f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.337+0000 7f327d400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3278073620 con 0x7f327815f380 2024-04-26T23:07:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.337+0000 7f327d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32780aff50 con 0x7f327806f500 2024-04-26T23:07:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.337+0000 7f327d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f32780737a0 con 0x7f32780d0620 2024-04-26T23:07:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.337+0000 7f326de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f327815f380 0x7f32780b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.337+0000 7f326de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f327815f380 0x7f32780b5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41494/0 (socket says 172.21.15.40:41494) 2024-04-26T23:07:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.337+0000 7f326de00640 1 -- 172.21.15.40:0/1120402004 learned_addr learned my addr 172.21.15.40:0/1120402004 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.338+0000 7f3277200640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f327806f500 0x7f327806f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.338+0000 7f3276800640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32780d0620 0x7f32780b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.338+0000 7f326de00640 1 -- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32780d0620 msgr2=0x7f32780b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.338+0000 7f326de00640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32780d0620 0x7f32780b47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.338+0000 7f326de00640 1 -- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f327806f500 msgr2=0x7f327806f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:31.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.338+0000 7f326de00640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f327806f500 0x7f327806f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.338+0000 7f326de00640 1 -- 172.21.15.40:0/1120402004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32780b3a20 con 0x7f327815f380 2024-04-26T23:07:31.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.338+0000 7f326de00640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f327815f380 0x7f32780b5470 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f326000c460 tx=0x7f326000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:31.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.338+0000 7f326f400640 1 -- 172.21.15.40:0/1120402004 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260016020 con 0x7f327815f380 2024-04-26T23:07:31.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.338+0000 7f326f400640 1 -- 172.21.15.40:0/1120402004 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3260012870 con 0x7f327815f380 2024-04-26T23:07:31.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.338+0000 7f327d400640 1 -- 172.21.15.40:0/1120402004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32780b3d00 con 0x7f327815f380 2024-04-26T23:07:31.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.338+0000 7f326f400640 1 -- 172.21.15.40:0/1120402004 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260012c60 con 0x7f327815f380 2024-04-26T23:07:31.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.339+0000 7f327d400640 1 -- 172.21.15.40:0/1120402004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32780b42c0 con 0x7f327815f380 2024-04-26T23:07:31.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.340+0000 7f327d400640 1 -- 172.21.15.40:0/1120402004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32780b44c0 con 0x7f327815f380 2024-04-26T23:07:31.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.341+0000 7f326f400640 1 -- 172.21.15.40:0/1120402004 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3260019020 con 0x7f327815f380 2024-04-26T23:07:31.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.342+0000 7f326f400640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f325406c8f0 0x7f325406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:31.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.342+0000 7f326f400640 1 -- 172.21.15.40:0/1120402004 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f326008bdd0 con 0x7f327815f380 2024-04-26T23:07:31.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.342+0000 7f3276800640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f325406c8f0 0x7f325406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:31.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.344+0000 7f3276800640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f325406c8f0 0x7f325406edb0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3268006fd0 tx=0x7f3268008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:31.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.346+0000 7f326f400640 1 -- 172.21.15.40:0/1120402004 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f32780b44c0 con 0x7f327815f380 2024-04-26T23:07:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.557+0000 7f327d400640 1 -- 172.21.15.40:0/1120402004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f32780694f0 con 0x7f327815f380 2024-04-26T23:07:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.558+0000 7f326f400640 1 -- 172.21.15.40:0/1120402004 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) ==== 100+0+11 (secure 0 0 0) 0x7f3260020020 con 0x7f327815f380 2024-04-26T23:07:31.558 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:07:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.561+0000 7f327d400640 1 -- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f325406c8f0 msgr2=0x7f325406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.561+0000 7f327d400640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f325406c8f0 0x7f325406edb0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3268006fd0 tx=0x7f3268008040 comp rx=0 tx=0).stop 2024-04-26T23:07:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.561+0000 7f327d400640 1 -- 172.21.15.40:0/1120402004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f327815f380 msgr2=0x7f32780b5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.561+0000 7f327d400640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f327815f380 0x7f32780b5470 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f326000c460 tx=0x7f326000c920 comp rx=0 tx=0).stop 2024-04-26T23:07:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.561+0000 7f327d400640 1 -- 172.21.15.40:0/1120402004 shutdown_connections 2024-04-26T23:07:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.561+0000 7f327d400640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f325406c8f0 0x7f325406edb0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.561+0000 7f327d400640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f327806f500 0x7f327806f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.561+0000 7f327d400640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f327815f380 0x7f32780b5470 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.561+0000 7f327d400640 1 --2- 172.21.15.40:0/1120402004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32780d0620 0x7f32780b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.561+0000 7f327d400640 1 -- 172.21.15.40:0/1120402004 >> 172.21.15.40:0/1120402004 conn(0x7f32780d0be0 msgr2=0x7f32780cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.562+0000 7f327d400640 1 -- 172.21.15.40:0/1120402004 shutdown_connections 2024-04-26T23:07:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.562+0000 7f327d400640 1 -- 172.21.15.40:0/1120402004 wait complete. 2024-04-26T23:07:31.579 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.789+0000 7f1c61e00640 1 Processor -- start 2024-04-26T23:07:31.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.789+0000 7f1c61e00640 1 -- start start 2024-04-26T23:07:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.790+0000 7f1c61e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c5c071e60 0x7f1c5c06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.790+0000 7f1c61e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1c5c06fe20 0x7f1c5c072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.790+0000 7f1c61e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c5c15d7c0 con 0x7f1c5c072780 2024-04-26T23:07:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.790+0000 7f1c61e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1c5c15daa0 con 0x7f1c5c06fe20 2024-04-26T23:07:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.790+0000 7f1c61e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c5c15dd80 con 0x7f1c5c071e60 2024-04-26T23:07:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.791+0000 7f1c53e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1c5c06fe20 0x7f1c5c072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.791+0000 7f1c53e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1c5c06fe20 0x7f1c5c072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41510/0 (socket says 172.21.15.40:41510) 2024-04-26T23:07:31.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.791+0000 7f1c53e00640 1 -- 172.21.15.40:0/2235155477 learned_addr learned my addr 172.21.15.40:0/2235155477 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:31.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.791+0000 7f1c5aa00640 1 --2- 172.21.15.40:0/2235155477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c5c071e60 0x7f1c5c06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:31.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.791+0000 7f1c53e00640 1 -- 172.21.15.40:0/2235155477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c5c071e60 msgr2=0x7f1c5c06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:31.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.791+0000 7f1c53e00640 1 --2- 172.21.15.40:0/2235155477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c5c071e60 0x7f1c5c06f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.791+0000 7f1c53e00640 1 -- 172.21.15.40:0/2235155477 >> v1:172.21.15.103:6789/0 conn(0x7f1c5c072780 legacy=0x7f1c5c0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:31.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.791+0000 7f1c53e00640 1 -- 172.21.15.40:0/2235155477 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c5c15e060 con 0x7f1c5c06fe20 2024-04-26T23:07:31.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.792+0000 7f1c53e00640 1 --2- 172.21.15.40:0/2235155477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c5c06fe20 0x7f1c5c072240 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f1c44009870 tx=0x7f1c4402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1fc5405f1f2d407 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:31.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.793+0000 7f1c5a000640 1 -- 172.21.15.40:0/2235155477 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c44037070 con 0x7f1c5c06fe20 2024-04-26T23:07:31.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.793+0000 7f1c5a000640 1 -- 172.21.15.40:0/2235155477 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c44032440 con 0x7f1c5c06fe20 2024-04-26T23:07:31.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.793+0000 7f1c5a000640 1 -- 172.21.15.40:0/2235155477 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c44032810 con 0x7f1c5c06fe20 2024-04-26T23:07:31.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.793+0000 7f1c61e00640 1 -- 172.21.15.40:0/2235155477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c5c06fe20 msgr2=0x7f1c5c072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:31.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.793+0000 7f1c61e00640 1 --2- 172.21.15.40:0/2235155477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c5c06fe20 0x7f1c5c072240 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f1c44009870 tx=0x7f1c4402a210 comp rx=0 tx=0).stop 2024-04-26T23:07:31.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.793+0000 7f1c61e00640 1 -- 172.21.15.40:0/2235155477 shutdown_connections 2024-04-26T23:07:31.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.793+0000 7f1c61e00640 1 --2- 172.21.15.40:0/2235155477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c5c06fe20 0x7f1c5c072240 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.794+0000 7f1c61e00640 1 --2- 172.21.15.40:0/2235155477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c5c071e60 0x7f1c5c06f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.794+0000 7f1c61e00640 1 -- 172.21.15.40:0/2235155477 >> 172.21.15.40:0/2235155477 conn(0x7f1c5c0d0be0 msgr2=0x7f1c5c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:31.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.794+0000 7f1c61e00640 1 -- 172.21.15.40:0/2235155477 shutdown_connections 2024-04-26T23:07:31.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.794+0000 7f1c61e00640 1 -- 172.21.15.40:0/2235155477 wait complete. 2024-04-26T23:07:31.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.795+0000 7f1c61e00640 1 Processor -- start 2024-04-26T23:07:31.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.796+0000 7f1c61e00640 1 -- start start 2024-04-26T23:07:31.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.796+0000 7f1c61e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c5c06fe20 0x7f1c5c158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:31.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.796+0000 7f1c61e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c5c071e60 0x7f1c5c0b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:31.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.796+0000 7f1c61e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c5c072780 0x7f1c5c0af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:31.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.797+0000 7f1c61e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5c15d3f0 con 0x7f1c5c06fe20 2024-04-26T23:07:31.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.797+0000 7f1c61e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5c072c10 con 0x7f1c5c071e60 2024-04-26T23:07:31.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.797+0000 7f1c61e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c5c15d570 con 0x7f1c5c072780 2024-04-26T23:07:31.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.797+0000 7f1c5aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c5c06fe20 0x7f1c5c158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:31.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.797+0000 7f1c5aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c5c06fe20 0x7f1c5c158310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41520/0 (socket says 172.21.15.40:41520) 2024-04-26T23:07:31.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.797+0000 7f1c5aa00640 1 -- 172.21.15.40:0/3600172110 learned_addr learned my addr 172.21.15.40:0/3600172110 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:31.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.797+0000 7f1c5b400640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c5c072780 0x7f1c5c0af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:31.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.797+0000 7f1c53e00640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c5c071e60 0x7f1c5c0b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:31.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.797+0000 7f1c5aa00640 1 -- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c5c072780 msgr2=0x7f1c5c0af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:31.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.797+0000 7f1c5aa00640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c5c072780 0x7f1c5c0af400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.797+0000 7f1c5aa00640 1 -- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c5c071e60 msgr2=0x7f1c5c0b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:31.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.797+0000 7f1c5aa00640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c5c071e60 0x7f1c5c0b4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:31.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.797+0000 7f1c5aa00640 1 -- 172.21.15.40:0/3600172110 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c5c1f8ee0 con 0x7f1c5c06fe20 2024-04-26T23:07:31.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.798+0000 7f1c5aa00640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c5c06fe20 0x7f1c5c158310 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00d800 tx=0x7f1c4c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:31.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.798+0000 7f1c53400640 1 -- 172.21.15.40:0/3600172110 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c00b660 con 0x7f1c5c06fe20 2024-04-26T23:07:31.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.798+0000 7f1c53400640 1 -- 172.21.15.40:0/3600172110 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c4c019070 con 0x7f1c5c06fe20 2024-04-26T23:07:31.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.798+0000 7f1c53400640 1 -- 172.21.15.40:0/3600172110 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c00ba20 con 0x7f1c5c06fe20 2024-04-26T23:07:31.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.798+0000 7f1c61e00640 1 -- 172.21.15.40:0/3600172110 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c5c1f91c0 con 0x7f1c5c06fe20 2024-04-26T23:07:31.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.798+0000 7f1c61e00640 1 -- 172.21.15.40:0/3600172110 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c5c1f9780 con 0x7f1c5c06fe20 2024-04-26T23:07:31.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.800+0000 7f1c61e00640 1 -- 172.21.15.40:0/3600172110 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c5c06ec80 con 0x7f1c5c06fe20 2024-04-26T23:07:31.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.800+0000 7f1c53400640 1 -- 172.21.15.40:0/3600172110 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1c4c00bbc0 con 0x7f1c5c06fe20 2024-04-26T23:07:31.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.801+0000 7f1c53400640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c3806c8f0 0x7f1c3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:31.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.801+0000 7f1c53400640 1 -- 172.21.15.40:0/3600172110 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f1c4c08c230 con 0x7f1c5c06fe20 2024-04-26T23:07:31.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.802+0000 7f1c53e00640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c3806c8f0 0x7f1c3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:31.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.803+0000 7f1c53e00640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c3806c8f0 0x7f1c3806edb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1c44034000 tx=0x7f1c4402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:31.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:31.806+0000 7f1c53400640 1 -- 172.21.15.40:0/3600172110 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1c5c06ec80 con 0x7f1c5c06fe20 2024-04-26T23:07:32.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.054+0000 7f1c61e00640 1 -- 172.21.15.40:0/3600172110 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c5c0694f0 con 0x7f1c5c06fe20 2024-04-26T23:07:32.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.055+0000 7f1c53400640 1 -- 172.21.15.40:0/3600172110 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5227 (secure 0 0 0) 0x7f1c4c05a430 con 0x7f1c5c06fe20 2024-04-26T23:07:32.055 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:32.056 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/755627718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":755627718},{"type":"v1","addr":"172.21.15.103:6837","nonce":755627718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:06:57.853769+0000","modified":"2024-04-26T23:07:26.981321+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7446,"mds_1":7443,"mds_2":5965},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5965":{"gid":5965,"name":"d","rank":2,"incarnation":78,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6836/2914603370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2914603370},{"type":"v1","addr":"172.21.15.103:6836","nonce":2914603370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7443":{"gid":7443,"name":"a","rank":1,"incarnation":76,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/216893598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":216893598},{"type":"v1","addr":"172.21.15.40:6834","nonce":216893598}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7446":{"gid":7446,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4227275373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":4227275373},{"type":"v1","addr":"172.21.15.40:6835","nonce":4227275373}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7446,"qdb_cluster":[7446,7443,5965]},"id":5}]} 2024-04-26T23:07:32.056 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 82 2024-04-26T23:07:32.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.058+0000 7f1c61e00640 1 -- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c3806c8f0 msgr2=0x7f1c3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.058+0000 7f1c61e00640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c3806c8f0 0x7f1c3806edb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1c44034000 tx=0x7f1c4402e000 comp rx=0 tx=0).stop 2024-04-26T23:07:32.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.058+0000 7f1c61e00640 1 -- 172.21.15.40:0/3600172110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c5c06fe20 msgr2=0x7f1c5c158310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.058+0000 7f1c61e00640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c5c06fe20 0x7f1c5c158310 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00d800 tx=0x7f1c4c00dcc0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.058+0000 7f1c61e00640 1 -- 172.21.15.40:0/3600172110 shutdown_connections 2024-04-26T23:07:32.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.058+0000 7f1c61e00640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c3806c8f0 0x7f1c3806edb0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.058+0000 7f1c61e00640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c5c072780 0x7f1c5c0af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.058+0000 7f1c61e00640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c5c071e60 0x7f1c5c0b4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.058+0000 7f1c61e00640 1 --2- 172.21.15.40:0/3600172110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c5c06fe20 0x7f1c5c158310 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.058+0000 7f1c61e00640 1 -- 172.21.15.40:0/3600172110 >> 172.21.15.40:0/3600172110 conn(0x7f1c5c0d0be0 msgr2=0x7f1c5c158930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:32.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.058+0000 7f1c61e00640 1 -- 172.21.15.40:0/3600172110 shutdown_connections 2024-04-26T23:07:32.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.058+0000 7f1c61e00640 1 -- 172.21.15.40:0/3600172110 wait complete. 2024-04-26T23:07:32.078 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:07:32.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.282+0000 7f9891200640 1 Processor -- start 2024-04-26T23:07:32.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.282+0000 7f9891200640 1 -- start start 2024-04-26T23:07:32.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.283+0000 7f9891200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f988c156460 0x7f988c0cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.283+0000 7f9891200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f988c06fea0 0x7f988c072c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.283+0000 7f9891200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f988c156840 con 0x7f988c0cba10 2024-04-26T23:07:32.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.283+0000 7f9891200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f988c0cbf20 con 0x7f988c06fea0 2024-04-26T23:07:32.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.283+0000 7f9891200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f988c0cc200 con 0x7f988c156460 2024-04-26T23:07:32.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.284+0000 7f988a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f988c06fea0 0x7f988c072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:32.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.284+0000 7f988a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f988c06fea0 0x7f988c072c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41532/0 (socket says 172.21.15.40:41532) 2024-04-26T23:07:32.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.284+0000 7f988a800640 1 -- 172.21.15.40:0/2884641541 learned_addr learned my addr 172.21.15.40:0/2884641541 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:32.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.284+0000 7f9889e00640 1 --2- 172.21.15.40:0/2884641541 >> v2:172.21.15.103:3300/0 conn(0x7f988c156460 0x7f988c0cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:32.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.284+0000 7f988a800640 1 -- 172.21.15.40:0/2884641541 >> v2:172.21.15.103:3300/0 conn(0x7f988c156460 msgr2=0x7f988c0cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:32.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.284+0000 7f988a800640 1 --2- 172.21.15.40:0/2884641541 >> v2:172.21.15.103:3300/0 conn(0x7f988c156460 0x7f988c0cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.284+0000 7f988a800640 1 -- 172.21.15.40:0/2884641541 >> v1:172.21.15.103:6789/0 conn(0x7f988c0cba10 legacy=0x7f988c06f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:32.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.284+0000 7f988a800640 1 -- 172.21.15.40:0/2884641541 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c0cc4e0 con 0x7f988c06fea0 2024-04-26T23:07:32.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.286+0000 7f988a800640 1 --2- 172.21.15.40:0/2884641541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c06fea0 0x7f988c072c50 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f988c0b05d0 tx=0x7f988402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e766125b9ee9e071 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:32.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.286+0000 7f9888a00640 1 -- 172.21.15.40:0/2884641541 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884037070 con 0x7f988c06fea0 2024-04-26T23:07:32.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.286+0000 7f9888a00640 1 -- 172.21.15.40:0/2884641541 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9884032440 con 0x7f988c06fea0 2024-04-26T23:07:32.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.286+0000 7f9888a00640 1 -- 172.21.15.40:0/2884641541 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884032810 con 0x7f988c06fea0 2024-04-26T23:07:32.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.287+0000 7f9891200640 1 -- 172.21.15.40:0/2884641541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c06fea0 msgr2=0x7f988c072c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.287+0000 7f9891200640 1 --2- 172.21.15.40:0/2884641541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c06fea0 0x7f988c072c50 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f988c0b05d0 tx=0x7f988402a230 comp rx=0 tx=0).stop 2024-04-26T23:07:32.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.287+0000 7f9891200640 1 -- 172.21.15.40:0/2884641541 shutdown_connections 2024-04-26T23:07:32.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.287+0000 7f9891200640 1 --2- 172.21.15.40:0/2884641541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c06fea0 0x7f988c072c50 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.287+0000 7f9891200640 1 --2- 172.21.15.40:0/2884641541 >> v2:172.21.15.103:3300/0 conn(0x7f988c156460 0x7f988c0cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.287+0000 7f9891200640 1 -- 172.21.15.40:0/2884641541 >> 172.21.15.40:0/2884641541 conn(0x7f988c0d0be0 msgr2=0x7f988c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:32.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.287+0000 7f9891200640 1 -- 172.21.15.40:0/2884641541 shutdown_connections 2024-04-26T23:07:32.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.287+0000 7f9891200640 1 -- 172.21.15.40:0/2884641541 wait complete. 2024-04-26T23:07:32.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.289+0000 7f9891200640 1 Processor -- start 2024-04-26T23:07:32.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.289+0000 7f9891200640 1 -- start start 2024-04-26T23:07:32.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.290+0000 7f9891200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f988c06fea0 0x7f988c0c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.290+0000 7f9891200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c0cba10 0x7f988c15e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.290+0000 7f9891200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c156460 0x7f988c15edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.290+0000 7f9891200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f988c06e260 con 0x7f988c156460 2024-04-26T23:07:32.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.290+0000 7f9891200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f988c072700 con 0x7f988c06fea0 2024-04-26T23:07:32.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.290+0000 7f9891200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f988c06e3e0 con 0x7f988c0cba10 2024-04-26T23:07:32.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.290+0000 7f988a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c156460 0x7f988c15edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:32.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.290+0000 7f988a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c156460 0x7f988c15edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41542/0 (socket says 172.21.15.40:41542) 2024-04-26T23:07:32.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.290+0000 7f988a800640 1 -- 172.21.15.40:0/1439678017 learned_addr learned my addr 172.21.15.40:0/1439678017 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:32.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.291+0000 7f988a800640 1 -- 172.21.15.40:0/1439678017 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c0cba10 msgr2=0x7f988c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.291+0000 7f988a800640 1 --2- 172.21.15.40:0/1439678017 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c0cba10 0x7f988c15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.291+0000 7f988a800640 1 -- 172.21.15.40:0/1439678017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f988c06fea0 msgr2=0x7f988c0c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.291+0000 7f988a800640 1 --2- 172.21.15.40:0/1439678017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f988c06fea0 0x7f988c0c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.291+0000 7f988a800640 1 -- 172.21.15.40:0/1439678017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c0ca600 con 0x7f988c156460 2024-04-26T23:07:32.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.291+0000 7f988a800640 1 --2- 172.21.15.40:0/1439678017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c156460 0x7f988c15edb0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f9884009960 tx=0x7f98840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:32.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.291+0000 7f9882000640 1 -- 172.21.15.40:0/1439678017 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884041020 con 0x7f988c156460 2024-04-26T23:07:32.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.291+0000 7f9882000640 1 -- 172.21.15.40:0/1439678017 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9884030700 con 0x7f988c156460 2024-04-26T23:07:32.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.291+0000 7f9882000640 1 -- 172.21.15.40:0/1439678017 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884037040 con 0x7f988c156460 2024-04-26T23:07:32.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.291+0000 7f9891200640 1 -- 172.21.15.40:0/1439678017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988c0c53b0 con 0x7f988c156460 2024-04-26T23:07:32.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.291+0000 7f9891200640 1 -- 172.21.15.40:0/1439678017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f988c0c5970 con 0x7f988c156460 2024-04-26T23:07:32.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.293+0000 7f9891200640 1 -- 172.21.15.40:0/1439678017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f988c06f550 con 0x7f988c156460 2024-04-26T23:07:32.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.294+0000 7f9882000640 1 -- 172.21.15.40:0/1439678017 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9884044020 con 0x7f988c156460 2024-04-26T23:07:32.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.295+0000 7f9882000640 1 --2- 172.21.15.40:0/1439678017 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f986c06c8f0 0x7f986c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.295+0000 7f9882000640 1 -- 172.21.15.40:0/1439678017 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f98840ac5d0 con 0x7f988c156460 2024-04-26T23:07:32.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.295+0000 7f9889e00640 1 --2- 172.21.15.40:0/1439678017 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f986c06c8f0 0x7f986c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:32.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.296+0000 7f9889e00640 1 --2- 172.21.15.40:0/1439678017 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f986c06c8f0 0x7f986c06edb0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f987c0070b0 tx=0x7f987c007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:32.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.299+0000 7f9882000640 1 -- 172.21.15.40:0/1439678017 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f988c06f550 con 0x7f988c156460 2024-04-26T23:07:32.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.505+0000 7f9891200640 1 -- 172.21.15.40:0/1439678017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f988c0c5e50 con 0x7f988c156460 2024-04-26T23:07:32.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.508+0000 7f9882000640 1 -- 172.21.15.40:0/1439678017 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+16909 (secure 0 0 0) 0x7f988c0c5e50 con 0x7f988c156460 2024-04-26T23:07:32.508 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:32.508 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":79,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:06:59.420463+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:06:55.738277+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-26T23:06:56.838617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:07:32.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.510+0000 7f9891200640 1 -- 172.21.15.40:0/1439678017 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f986c06c8f0 msgr2=0x7f986c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.510+0000 7f9891200640 1 --2- 172.21.15.40:0/1439678017 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f986c06c8f0 0x7f986c06edb0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f987c0070b0 tx=0x7f987c007040 comp rx=0 tx=0).stop 2024-04-26T23:07:32.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.510+0000 7f9891200640 1 -- 172.21.15.40:0/1439678017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c156460 msgr2=0x7f988c15edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.510+0000 7f9891200640 1 --2- 172.21.15.40:0/1439678017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c156460 0x7f988c15edb0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f9884009960 tx=0x7f98840046c0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.511+0000 7f9891200640 1 -- 172.21.15.40:0/1439678017 shutdown_connections 2024-04-26T23:07:32.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.511+0000 7f9891200640 1 --2- 172.21.15.40:0/1439678017 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f986c06c8f0 0x7f986c06edb0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.511+0000 7f9891200640 1 --2- 172.21.15.40:0/1439678017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f988c156460 0x7f988c15edb0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.511+0000 7f9891200640 1 --2- 172.21.15.40:0/1439678017 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f988c0cba10 0x7f988c15e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.511+0000 7f9891200640 1 --2- 172.21.15.40:0/1439678017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f988c06fea0 0x7f988c0c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.511+0000 7f9891200640 1 -- 172.21.15.40:0/1439678017 >> 172.21.15.40:0/1439678017 conn(0x7f988c0d0be0 msgr2=0x7f988c0cf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:32.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.511+0000 7f9891200640 1 -- 172.21.15.40:0/1439678017 shutdown_connections 2024-04-26T23:07:32.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.511+0000 7f9891200640 1 -- 172.21.15.40:0/1439678017 wait complete. 2024-04-26T23:07:32.529 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T23:07:32.530 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:32.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.736+0000 7f0434400640 1 Processor -- start 2024-04-26T23:07:32.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.736+0000 7f0434400640 1 -- start start 2024-04-26T23:07:32.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.737+0000 7f0434400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f042c072950 0x7f042c167ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.737+0000 7f0434400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f042c15d7c0 con 0x7f042c0cc480 2024-04-26T23:07:32.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.737+0000 7f0434400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f042c15daa0 con 0x7f042c071e60 2024-04-26T23:07:32.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.737+0000 7f0434400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f042c15dd80 con 0x7f042c072950 2024-04-26T23:07:32.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.737+0000 7f0431000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f042c0cc480 0x7f042c072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34284/0 (socket says 172.21.15.40:34284) 2024-04-26T23:07:32.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.737+0000 7f0431000640 1 -- 172.21.15.40:0/2644794420 learned_addr learned my addr 172.21.15.40:0/2644794420 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:32.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.738+0000 7f0432400640 1 --2- 172.21.15.40:0/2644794420 >> v2:172.21.15.103:3300/0 conn(0x7f042c072950 0x7f042c167ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:32.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.738+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2453512578 0 0) 0x7f042c15d7c0 con 0x7f042c0cc480 2024-04-26T23:07:32.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.738+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0414003520 con 0x7f042c0cc480 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.738+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1174835738 0 0) 0x7f0414003520 con 0x7f042c0cc480 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.738+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 >> v2:172.21.15.103:3300/0 conn(0x7f042c072950 msgr2=0x7f042c167ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.738+0000 7f042be00640 1 --2- 172.21.15.40:0/2644794420 >> v2:172.21.15.103:3300/0 conn(0x7f042c072950 0x7f042c167ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.738+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 >> v1:172.21.15.103:6789/0 conn(0x7f042c071e60 legacy=0x7f042c06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.738+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f042c15e060 con 0x7f042c0cc480 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.738+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f0418001670 con 0x7f042c0cc480 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.738+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 >> v1:172.21.15.40:6789/0 conn(0x7f042c0cc480 legacy=0x7f042c072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.739+0000 7f042be00640 1 --2- 172.21.15.40:0/2644794420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414003a00 0x7f0414003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.739+0000 7f042be00640 1 --2- 172.21.15.40:0/2644794420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414004800 0x7f0414004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.739+0000 7f042be00640 1 --2- 172.21.15.40:0/2644794420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04140051c0 0x7f041400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.739+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f042c15dd80 con 0x7f0414004800 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.739+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04140104d0 con 0x7f04140051c0 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.739+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f04140106d0 con 0x7f0414003a00 2024-04-26T23:07:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.739+0000 7f0431000640 1 --2- 172.21.15.40:0/2644794420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414004800 0x7f0414004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:32.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.739+0000 7f0431000640 1 -- 172.21.15.40:0/2644794420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414003a00 msgr2=0x7f0414003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:32.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.739+0000 7f0431000640 1 --2- 172.21.15.40:0/2644794420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414003a00 0x7f0414003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.739+0000 7f0431000640 1 -- 172.21.15.40:0/2644794420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04140051c0 msgr2=0x7f041400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:32.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.739+0000 7f0431000640 1 --2- 172.21.15.40:0/2644794420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04140051c0 0x7f041400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.739+0000 7f0431000640 1 -- 172.21.15.40:0/2644794420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0414003520 con 0x7f0414004800 2024-04-26T23:07:32.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.739+0000 7f0432400640 1 -- 172.21.15.40:0/2644794420 reap_dead start 2024-04-26T23:07:32.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.740+0000 7f0431000640 1 --2- 172.21.15.40:0/2644794420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414004800 0x7f0414004c80 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f041800c0e0 tx=0x7f0414005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b23f11bbd03f5d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:32.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.740+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0418007c20 con 0x7f0414004800 2024-04-26T23:07:32.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.740+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0418005af0 con 0x7f0414004800 2024-04-26T23:07:32.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.740+0000 7f042be00640 1 -- 172.21.15.40:0/2644794420 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0418007070 con 0x7f0414004800 2024-04-26T23:07:32.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.741+0000 7f0434400640 1 -- 172.21.15.40:0/2644794420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414004800 msgr2=0x7f0414004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.741+0000 7f0434400640 1 --2- 172.21.15.40:0/2644794420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414004800 0x7f0414004c80 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f041800c0e0 tx=0x7f0414005830 comp rx=0 tx=0).stop 2024-04-26T23:07:32.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.741+0000 7f0434400640 1 -- 172.21.15.40:0/2644794420 shutdown_connections 2024-04-26T23:07:32.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.741+0000 7f0434400640 1 --2- 172.21.15.40:0/2644794420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414004800 0x7f0414004c80 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.741+0000 7f0434400640 1 -- 172.21.15.40:0/2644794420 >> 172.21.15.40:0/2644794420 conn(0x7f042c0d0be0 msgr2=0x7f042c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:32.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.741+0000 7f0434400640 1 -- 172.21.15.40:0/2644794420 shutdown_connections 2024-04-26T23:07:32.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.741+0000 7f0434400640 1 -- 172.21.15.40:0/2644794420 wait complete. 2024-04-26T23:07:32.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.743+0000 7f0434400640 1 Processor -- start 2024-04-26T23:07:32.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.743+0000 7f0434400640 1 -- start start 2024-04-26T23:07:32.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.744+0000 7f0434400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414004800 0x7f042c0c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.744+0000 7f0434400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f042c15a8c0 0x7f042c0cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.744+0000 7f0434400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f042c0cdb30 0x7f042c1f9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.744+0000 7f0434400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f042c15d570 con 0x7f042c0cdb30 2024-04-26T23:07:32.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.744+0000 7f0434400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f042c15d3f0 con 0x7f042c15a8c0 2024-04-26T23:07:32.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.744+0000 7f0434400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f042c072240 con 0x7f0414004800 2024-04-26T23:07:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.744+0000 7f0432400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f042c0cdb30 0x7f042c1f9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.744+0000 7f0432400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f042c0cdb30 0x7f042c1f9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41556/0 (socket says 172.21.15.40:41556) 2024-04-26T23:07:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.744+0000 7f0432400640 1 -- 172.21.15.40:0/1185555910 learned_addr learned my addr 172.21.15.40:0/1185555910 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.745+0000 7f0431000640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f042c15a8c0 0x7f042c0cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.745+0000 7f0431a00640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414004800 0x7f042c0c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.745+0000 7f0432400640 1 -- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414004800 msgr2=0x7f042c0c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.745+0000 7f0432400640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414004800 0x7f042c0c88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.745+0000 7f0432400640 1 -- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f042c15a8c0 msgr2=0x7f042c0cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.745+0000 7f0432400640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f042c15a8c0 0x7f042c0cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.745+0000 7f0432400640 1 -- 172.21.15.40:0/1185555910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f042c1fa1a0 con 0x7f042c0cdb30 2024-04-26T23:07:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.745+0000 7f0431000640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f042c15a8c0 0x7f042c0cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.745+0000 7f0431a00640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414004800 0x7f042c0c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.745+0000 7f0432400640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f042c0cdb30 0x7f042c1f9190 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f042c0cc910 tx=0x7f042400ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:32.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.745+0000 7f0429600640 1 -- 172.21.15.40:0/1185555910 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042400bb10 con 0x7f042c0cdb30 2024-04-26T23:07:32.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.746+0000 7f0429600640 1 -- 172.21.15.40:0/1185555910 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f042401a070 con 0x7f042c0cdb30 2024-04-26T23:07:32.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.746+0000 7f0429600640 1 -- 172.21.15.40:0/1185555910 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0424014400 con 0x7f042c0cdb30 2024-04-26T23:07:32.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.746+0000 7f0434400640 1 -- 172.21.15.40:0/1185555910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f042c1f9ec0 con 0x7f042c0cdb30 2024-04-26T23:07:32.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.746+0000 7f0434400640 1 -- 172.21.15.40:0/1185555910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f042c0c80a0 con 0x7f042c0cdb30 2024-04-26T23:07:32.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.747+0000 7f0434400640 1 -- 172.21.15.40:0/1185555910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f042c0694f0 con 0x7f042c0cdb30 2024-04-26T23:07:32.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.748+0000 7f0429600640 1 -- 172.21.15.40:0/1185555910 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0424013070 con 0x7f042c0cdb30 2024-04-26T23:07:32.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.749+0000 7f0429600640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f040c06c920 0x7f040c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:32.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.749+0000 7f0429600640 1 -- 172.21.15.40:0/1185555910 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f042408bf40 con 0x7f042c0cdb30 2024-04-26T23:07:32.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.750+0000 7f0431a00640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f040c06c920 0x7f040c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:32.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.751+0000 7f0431a00640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f040c06c920 0x7f040c06ede0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f042c06b230 tx=0x7f0420009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:32.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.753+0000 7f0429600640 1 -- 172.21.15.40:0/1185555910 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f042405a1e0 con 0x7f042c0cdb30 2024-04-26T23:07:32.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.989+0000 7f0434400640 1 -- 172.21.15.40:0/1185555910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f042c0cf3b0 con 0x7f042c0cdb30 2024-04-26T23:07:32.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.990+0000 7f0429600640 1 -- 172.21.15.40:0/1185555910 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5227 (secure 0 0 0) 0x7f0424059ae0 con 0x7f042c0cdb30 2024-04-26T23:07:32.991 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:32.991 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/755627718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":755627718},{"type":"v1","addr":"172.21.15.103:6837","nonce":755627718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:06:57.853769+0000","modified":"2024-04-26T23:07:26.981321+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7446,"mds_1":7443,"mds_2":5965},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5965":{"gid":5965,"name":"d","rank":2,"incarnation":78,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6836/2914603370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2914603370},{"type":"v1","addr":"172.21.15.103:6836","nonce":2914603370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7443":{"gid":7443,"name":"a","rank":1,"incarnation":76,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/216893598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":216893598},{"type":"v1","addr":"172.21.15.40:6834","nonce":216893598}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7446":{"gid":7446,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4227275373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":4227275373},{"type":"v1","addr":"172.21.15.40:6835","nonce":4227275373}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7446,"qdb_cluster":[7446,7443,5965]},"id":5}]} 2024-04-26T23:07:32.991 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 82 2024-04-26T23:07:32.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.993+0000 7f0434400640 1 -- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f040c06c920 msgr2=0x7f040c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.993+0000 7f0434400640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f040c06c920 0x7f040c06ede0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f042c06b230 tx=0x7f0420009040 comp rx=0 tx=0).stop 2024-04-26T23:07:32.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.993+0000 7f0434400640 1 -- 172.21.15.40:0/1185555910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f042c0cdb30 msgr2=0x7f042c1f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:32.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.993+0000 7f0434400640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f042c0cdb30 0x7f042c1f9190 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f042c0cc910 tx=0x7f042400ef50 comp rx=0 tx=0).stop 2024-04-26T23:07:32.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.993+0000 7f0434400640 1 -- 172.21.15.40:0/1185555910 shutdown_connections 2024-04-26T23:07:32.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.993+0000 7f0434400640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f040c06c920 0x7f040c06ede0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.993+0000 7f0434400640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f042c0cdb30 0x7f042c1f9190 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.993+0000 7f0434400640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f042c15a8c0 0x7f042c0cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.993+0000 7f0434400640 1 --2- 172.21.15.40:0/1185555910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0414004800 0x7f042c0c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:32.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.993+0000 7f0434400640 1 -- 172.21.15.40:0/1185555910 >> 172.21.15.40:0/1185555910 conn(0x7f042c0d0be0 msgr2=0x7f042c0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:32.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.993+0000 7f0434400640 1 -- 172.21.15.40:0/1185555910 shutdown_connections 2024-04-26T23:07:32.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:32.993+0000 7f0434400640 1 -- 172.21.15.40:0/1185555910 wait complete. 2024-04-26T23:07:33.013 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.207+0000 7fc365e00640 1 Processor -- start 2024-04-26T23:07:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.207+0000 7fc365e00640 1 -- start start 2024-04-26T23:07:33.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.209+0000 7fc365e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc3600d32e0 0x7fc3600d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.209+0000 7fc365e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc360162c40 con 0x7fc3600b6220 2024-04-26T23:07:33.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.209+0000 7fc365e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3600d2d80 con 0x7fc3600b8d20 2024-04-26T23:07:33.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.209+0000 7fc365e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc3600d3c80 con 0x7fc3600d32e0 2024-04-26T23:07:33.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.209+0000 7fc364a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc3600b6220 0x7fc3600d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34292/0 (socket says 172.21.15.40:34292) 2024-04-26T23:07:33.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.209+0000 7fc364a00640 1 -- 172.21.15.40:0/2971162601 learned_addr learned my addr 172.21.15.40:0/2971162601 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:33.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35fe00640 1 --2- 172.21.15.40:0/2971162601 >> v2:172.21.15.103:3300/0 conn(0x7fc3600d32e0 0x7fc3600d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:33.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3916134063 0 0) 0x7fc360162c40 con 0x7fc3600b6220 2024-04-26T23:07:33.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc344003520 con 0x7fc3600b6220 2024-04-26T23:07:33.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3858249619 0 0) 0x7fc344003520 con 0x7fc3600b6220 2024-04-26T23:07:33.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 >> v2:172.21.15.103:3300/0 conn(0x7fc3600d32e0 msgr2=0x7fc3600d3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:33.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 --2- 172.21.15.40:0/2971162601 >> v2:172.21.15.103:3300/0 conn(0x7fc3600d32e0 0x7fc3600d3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 >> v1:172.21.15.103:6789/0 conn(0x7fc3600b8d20 legacy=0x7fc3600b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:33.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3600b7c80 con 0x7fc3600b6220 2024-04-26T23:07:33.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fc350001670 con 0x7fc3600b6220 2024-04-26T23:07:33.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 >> v1:172.21.15.40:6789/0 conn(0x7fc3600b6220 legacy=0x7fc3600d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:33.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 --2- 172.21.15.40:0/2971162601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc344003a00 0x7fc344003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 --2- 172.21.15.40:0/2971162601 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc344004800 0x7fc344004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 --2- 172.21.15.40:0/2971162601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3440051c0 0x7fc34400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc34400c090 con 0x7fc344003a00 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc34400c240 con 0x7fc3440051c0 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.210+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc34400c440 con 0x7fc344004800 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.211+0000 7fc364a00640 1 --2- 172.21.15.40:0/2971162601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc344003a00 0x7fc344003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.211+0000 7fc365400640 1 --2- 172.21.15.40:0/2971162601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3440051c0 0x7fc34400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.211+0000 7fc35fe00640 1 --2- 172.21.15.40:0/2971162601 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc344004800 0x7fc344004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.211+0000 7fc364a00640 1 -- 172.21.15.40:0/2971162601 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc344004800 msgr2=0x7fc344004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.211+0000 7fc364a00640 1 --2- 172.21.15.40:0/2971162601 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc344004800 0x7fc344004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.211+0000 7fc364a00640 1 -- 172.21.15.40:0/2971162601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3440051c0 msgr2=0x7fc34400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.211+0000 7fc364a00640 1 --2- 172.21.15.40:0/2971162601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3440051c0 0x7fc34400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.211+0000 7fc364a00640 1 -- 172.21.15.40:0/2971162601 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc344003520 con 0x7fc344003a00 2024-04-26T23:07:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.211+0000 7fc365400640 1 -- 172.21.15.40:0/2971162601 reap_dead start 2024-04-26T23:07:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.212+0000 7fc364a00640 1 --2- 172.21.15.40:0/2971162601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc344003a00 0x7fc344003e20 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fc35000c110 tx=0x7fc344005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159f41cfbeb6cdf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.212+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc350007c40 con 0x7fc344003a00 2024-04-26T23:07:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.213+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc350005af0 con 0x7fc344003a00 2024-04-26T23:07:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.213+0000 7fc35f400640 1 -- 172.21.15.40:0/2971162601 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc350007070 con 0x7fc344003a00 2024-04-26T23:07:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.213+0000 7fc365e00640 1 -- 172.21.15.40:0/2971162601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc344003a00 msgr2=0x7fc344003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.213+0000 7fc365e00640 1 --2- 172.21.15.40:0/2971162601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc344003a00 0x7fc344003e20 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fc35000c110 tx=0x7fc344005830 comp rx=0 tx=0).stop 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.213+0000 7fc365e00640 1 -- 172.21.15.40:0/2971162601 shutdown_connections 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.213+0000 7fc365e00640 1 --2- 172.21.15.40:0/2971162601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc344003a00 0x7fc344003e20 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.213+0000 7fc365e00640 1 -- 172.21.15.40:0/2971162601 >> 172.21.15.40:0/2971162601 conn(0x7fc3600d4d90 msgr2=0x7fc3600d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.213+0000 7fc365e00640 1 -- 172.21.15.40:0/2971162601 shutdown_connections 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.213+0000 7fc365e00640 1 -- 172.21.15.40:0/2971162601 wait complete. 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.215+0000 7fc365e00640 1 Processor -- start 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.215+0000 7fc365e00640 1 -- start start 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.216+0000 7fc365e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc344003a00 0x7fc36015b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.216+0000 7fc365e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36015b910 0x7fc36014cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.216+0000 7fc365e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3600cad30 0x7fc36014d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.216+0000 7fc365e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc3600b8520 con 0x7fc3600cad30 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.216+0000 7fc365e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc3600b8240 con 0x7fc36015b910 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.216+0000 7fc365e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc3600d3740 con 0x7fc344003a00 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.216+0000 7fc365400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3600cad30 0x7fc36014d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.216+0000 7fc365400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3600cad30 0x7fc36014d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41570/0 (socket says 172.21.15.40:41570) 2024-04-26T23:07:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.216+0000 7fc365400640 1 -- 172.21.15.40:0/237896978 learned_addr learned my addr 172.21.15.40:0/237896978 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.217+0000 7fc364a00640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc344003a00 0x7fc36015b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.217+0000 7fc365400640 1 -- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc344003a00 msgr2=0x7fc36015b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.217+0000 7fc35fe00640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36015b910 0x7fc36014cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.217+0000 7fc365400640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc344003a00 0x7fc36015b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.217+0000 7fc365400640 1 -- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36015b910 msgr2=0x7fc36014cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.217+0000 7fc365400640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36015b910 0x7fc36014cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.217+0000 7fc365400640 1 -- 172.21.15.40:0/237896978 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36015a330 con 0x7fc3600cad30 2024-04-26T23:07:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.217+0000 7fc365400640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3600cad30 0x7fc36014d010 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fc36008da10 tx=0x7fc35800d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.217+0000 7fc35cc00640 1 -- 172.21.15.40:0/237896978 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc358018070 con 0x7fc3600cad30 2024-04-26T23:07:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.217+0000 7fc35cc00640 1 -- 172.21.15.40:0/237896978 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc358003d80 con 0x7fc3600cad30 2024-04-26T23:07:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.217+0000 7fc365e00640 1 -- 172.21.15.40:0/237896978 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc36015a610 con 0x7fc3600cad30 2024-04-26T23:07:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.217+0000 7fc35cc00640 1 -- 172.21.15.40:0/237896978 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3580052f0 con 0x7fc3600cad30 2024-04-26T23:07:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.217+0000 7fc365e00640 1 -- 172.21.15.40:0/237896978 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc36015abd0 con 0x7fc3600cad30 2024-04-26T23:07:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.218+0000 7fc365e00640 1 -- 172.21.15.40:0/237896978 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc360069370 con 0x7fc3600cad30 2024-04-26T23:07:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.220+0000 7fc35cc00640 1 -- 172.21.15.40:0/237896978 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc35801b070 con 0x7fc3600cad30 2024-04-26T23:07:33.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.221+0000 7fc35cc00640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc34006c920 0x7fc34006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.221+0000 7fc35cc00640 1 -- 172.21.15.40:0/237896978 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fc35808b300 con 0x7fc3600cad30 2024-04-26T23:07:33.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.221+0000 7fc364a00640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc34006c920 0x7fc34006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.223+0000 7fc364a00640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc34006c920 0x7fc34006ede0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc350032000 tx=0x7fc350009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:33.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.225+0000 7fc35cc00640 1 -- 172.21.15.40:0/237896978 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc358058f40 con 0x7fc3600cad30 2024-04-26T23:07:33.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.462+0000 7fc365e00640 1 -- 172.21.15.40:0/237896978 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3600d30c0 con 0x7fc3600cad30 2024-04-26T23:07:33.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.463+0000 7fc35cc00640 1 -- 172.21.15.40:0/237896978 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5227 (secure 0 0 0) 0x7fc35808b0c0 con 0x7fc3600cad30 2024-04-26T23:07:33.463 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:33.463 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/755627718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":755627718},{"type":"v1","addr":"172.21.15.103:6837","nonce":755627718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:06:57.853769+0000","modified":"2024-04-26T23:07:26.981321+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7446,"mds_1":7443,"mds_2":5965},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5965":{"gid":5965,"name":"d","rank":2,"incarnation":78,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6836/2914603370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2914603370},{"type":"v1","addr":"172.21.15.103:6836","nonce":2914603370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7443":{"gid":7443,"name":"a","rank":1,"incarnation":76,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/216893598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":216893598},{"type":"v1","addr":"172.21.15.40:6834","nonce":216893598}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7446":{"gid":7446,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4227275373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":4227275373},{"type":"v1","addr":"172.21.15.40:6835","nonce":4227275373}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7446,"qdb_cluster":[7446,7443,5965]},"id":5}]} 2024-04-26T23:07:33.464 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 82 2024-04-26T23:07:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.465+0000 7fc365e00640 1 -- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc34006c920 msgr2=0x7fc34006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.465+0000 7fc365e00640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc34006c920 0x7fc34006ede0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc350032000 tx=0x7fc350009030 comp rx=0 tx=0).stop 2024-04-26T23:07:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.465+0000 7fc365e00640 1 -- 172.21.15.40:0/237896978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3600cad30 msgr2=0x7fc36014d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.465+0000 7fc365e00640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3600cad30 0x7fc36014d010 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fc36008da10 tx=0x7fc35800d9f0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.466+0000 7fc365e00640 1 -- 172.21.15.40:0/237896978 shutdown_connections 2024-04-26T23:07:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.466+0000 7fc365e00640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc34006c920 0x7fc34006ede0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.466+0000 7fc365e00640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3600cad30 0x7fc36014d010 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.466+0000 7fc365e00640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36015b910 0x7fc36014cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.466+0000 7fc365e00640 1 --2- 172.21.15.40:0/237896978 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc344003a00 0x7fc36015b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.466+0000 7fc365e00640 1 -- 172.21.15.40:0/237896978 >> 172.21.15.40:0/237896978 conn(0x7fc3600d4d90 msgr2=0x7fc3600cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:33.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.466+0000 7fc365e00640 1 -- 172.21.15.40:0/237896978 shutdown_connections 2024-04-26T23:07:33.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.466+0000 7fc365e00640 1 -- 172.21.15.40:0/237896978 wait complete. 2024-04-26T23:07:33.485 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:07:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.692+0000 7fc9e1e00640 1 Processor -- start 2024-04-26T23:07:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.692+0000 7fc9e1e00640 1 -- start start 2024-04-26T23:07:33.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.693+0000 7fc9e1e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc9dc15f380 0x7fc9dc0cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.693+0000 7fc9e1e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9dc0d0620 0x7fc9dc0cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.693+0000 7fc9e1e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9dc072c00 con 0x7fc9dc073b70 2024-04-26T23:07:33.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.693+0000 7fc9e1e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc9dc0729b0 con 0x7fc9dc0d0620 2024-04-26T23:07:33.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.693+0000 7fc9e1e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc9dc072ee0 con 0x7fc9dc15f380 2024-04-26T23:07:33.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.694+0000 7fc9db400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9dc0d0620 0x7fc9dc0cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.694+0000 7fc9db400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9dc0d0620 0x7fc9dc0cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41578/0 (socket says 172.21.15.40:41578) 2024-04-26T23:07:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.694+0000 7fc9db400640 1 -- 172.21.15.40:0/804583889 learned_addr learned my addr 172.21.15.40:0/804583889 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.694+0000 7fc9daa00640 1 --2- 172.21.15.40:0/804583889 >> v2:172.21.15.103:3300/0 conn(0x7fc9dc15f380 0x7fc9dc0cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.694+0000 7fc9db400640 1 -- 172.21.15.40:0/804583889 >> v2:172.21.15.103:3300/0 conn(0x7fc9dc15f380 msgr2=0x7fc9dc0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.694+0000 7fc9db400640 1 --2- 172.21.15.40:0/804583889 >> v2:172.21.15.103:3300/0 conn(0x7fc9dc15f380 0x7fc9dc0cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.694+0000 7fc9db400640 1 -- 172.21.15.40:0/804583889 >> v1:172.21.15.103:6789/0 conn(0x7fc9dc073b70 legacy=0x7fc9dc0cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.694+0000 7fc9db400640 1 -- 172.21.15.40:0/804583889 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9dc0731c0 con 0x7fc9dc0d0620 2024-04-26T23:07:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.696+0000 7fc9db400640 1 --2- 172.21.15.40:0/804583889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9dc0d0620 0x7fc9dc0cb980 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0b3820 tx=0x7fc9d402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb5abdb5775611a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.696+0000 7fc9d9600640 1 -- 172.21.15.40:0/804583889 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d4037070 con 0x7fc9dc0d0620 2024-04-26T23:07:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.696+0000 7fc9d9600640 1 -- 172.21.15.40:0/804583889 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9d4032440 con 0x7fc9dc0d0620 2024-04-26T23:07:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.696+0000 7fc9d9600640 1 -- 172.21.15.40:0/804583889 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d4032810 con 0x7fc9dc0d0620 2024-04-26T23:07:33.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.697+0000 7fc9e1e00640 1 -- 172.21.15.40:0/804583889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9dc0d0620 msgr2=0x7fc9dc0cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:33.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.697+0000 7fc9e1e00640 1 --2- 172.21.15.40:0/804583889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9dc0d0620 0x7fc9dc0cb980 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0b3820 tx=0x7fc9d402a230 comp rx=0 tx=0).stop 2024-04-26T23:07:33.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.697+0000 7fc9e1e00640 1 -- 172.21.15.40:0/804583889 shutdown_connections 2024-04-26T23:07:33.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.697+0000 7fc9e1e00640 1 --2- 172.21.15.40:0/804583889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9dc0d0620 0x7fc9dc0cb980 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.697+0000 7fc9e1e00640 1 --2- 172.21.15.40:0/804583889 >> v2:172.21.15.103:3300/0 conn(0x7fc9dc15f380 0x7fc9dc0cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.697+0000 7fc9e1e00640 1 -- 172.21.15.40:0/804583889 >> 172.21.15.40:0/804583889 conn(0x7fc9dc0d0be0 msgr2=0x7fc9dc0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:33.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.697+0000 7fc9e1e00640 1 -- 172.21.15.40:0/804583889 shutdown_connections 2024-04-26T23:07:33.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.697+0000 7fc9e1e00640 1 -- 172.21.15.40:0/804583889 wait complete. 2024-04-26T23:07:33.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.699+0000 7fc9e1e00640 1 Processor -- start 2024-04-26T23:07:33.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.699+0000 7fc9e1e00640 1 -- start start 2024-04-26T23:07:33.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.700+0000 7fc9e1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9dc0d0620 0x7fc9dc1f8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.700+0000 7fc9e1e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9dc15f380 0x7fc9dc0c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.700+0000 7fc9e1e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9dc0c6ca0 0x7fc9dc154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.700+0000 7fc9e1e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc073620 con 0x7fc9dc0d0620 2024-04-26T23:07:33.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.700+0000 7fc9e1e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc0aff50 con 0x7fc9dc15f380 2024-04-26T23:07:33.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.700+0000 7fc9e1e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc9dc0737a0 con 0x7fc9dc0c6ca0 2024-04-26T23:07:33.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.700+0000 7fc9daa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9dc0d0620 0x7fc9dc1f8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:33.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.700+0000 7fc9daa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9dc0d0620 0x7fc9dc1f8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41594/0 (socket says 172.21.15.40:41594) 2024-04-26T23:07:33.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.700+0000 7fc9daa00640 1 -- 172.21.15.40:0/2463477105 learned_addr learned my addr 172.21.15.40:0/2463477105 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:33.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.701+0000 7fc9daa00640 1 -- 172.21.15.40:0/2463477105 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9dc0c6ca0 msgr2=0x7fc9dc154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:33.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.701+0000 7fc9daa00640 1 --2- 172.21.15.40:0/2463477105 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9dc0c6ca0 0x7fc9dc154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.701+0000 7fc9daa00640 1 -- 172.21.15.40:0/2463477105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9dc15f380 msgr2=0x7fc9dc0c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:33.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.701+0000 7fc9daa00640 1 --2- 172.21.15.40:0/2463477105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9dc15f380 0x7fc9dc0c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.701+0000 7fc9daa00640 1 -- 172.21.15.40:0/2463477105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9dc0c7490 con 0x7fc9dc0d0620 2024-04-26T23:07:33.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.701+0000 7fc9daa00640 1 --2- 172.21.15.40:0/2463477105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9dc0d0620 0x7fc9dc1f8760 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc00d5f0 tx=0x7fc9cc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:33.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.701+0000 7fc9d3400640 1 -- 172.21.15.40:0/2463477105 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc0125f0 con 0x7fc9dc0d0620 2024-04-26T23:07:33.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.701+0000 7fc9d3400640 1 -- 172.21.15.40:0/2463477105 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9cc019070 con 0x7fc9dc0d0620 2024-04-26T23:07:33.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.701+0000 7fc9d3400640 1 -- 172.21.15.40:0/2463477105 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc0129d0 con 0x7fc9dc0d0620 2024-04-26T23:07:33.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.701+0000 7fc9e1e00640 1 -- 172.21.15.40:0/2463477105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9dc0c7770 con 0x7fc9dc0d0620 2024-04-26T23:07:33.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.701+0000 7fc9e1e00640 1 -- 172.21.15.40:0/2463477105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9dc0c8da0 con 0x7fc9dc0d0620 2024-04-26T23:07:33.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.702+0000 7fc9e1e00640 1 -- 172.21.15.40:0/2463477105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9dc069af0 con 0x7fc9dc0d0620 2024-04-26T23:07:33.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.703+0000 7fc9d3400640 1 -- 172.21.15.40:0/2463477105 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc9cc012b70 con 0x7fc9dc0d0620 2024-04-26T23:07:33.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.705+0000 7fc9d3400640 1 --2- 172.21.15.40:0/2463477105 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc9b806c8f0 0x7fc9b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:33.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.705+0000 7fc9d3400640 1 -- 172.21.15.40:0/2463477105 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7fc9cc08c140 con 0x7fc9dc0d0620 2024-04-26T23:07:33.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.706+0000 7fc9da000640 1 --2- 172.21.15.40:0/2463477105 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc9b806c8f0 0x7fc9b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:33.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.707+0000 7fc9da000640 1 --2- 172.21.15.40:0/2463477105 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc9b806c8f0 0x7fc9b806edb0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc152960 tx=0x7fc9c4007d30 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:33.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.709+0000 7fc9d3400640 1 -- 172.21.15.40:0/2463477105 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc9dc069af0 con 0x7fc9dc0d0620 2024-04-26T23:07:33.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.915+0000 7fc9e1e00640 1 -- 172.21.15.40:0/2463477105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc9dc0c3da0 con 0x7fc9dc0d0620 2024-04-26T23:07:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.918+0000 7fc9d3400640 1 -- 172.21.15.40:0/2463477105 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+16909 (secure 0 0 0) 0x7fc9cc05a3d0 con 0x7fc9dc0d0620 2024-04-26T23:07:33.918 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:33.919 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":79,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:06:59.420463+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:06:55.738277+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-26T23:06:56.838617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:07:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.921+0000 7fc9e1e00640 1 -- 172.21.15.40:0/2463477105 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc9b806c8f0 msgr2=0x7fc9b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.921+0000 7fc9e1e00640 1 --2- 172.21.15.40:0/2463477105 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc9b806c8f0 0x7fc9b806edb0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc152960 tx=0x7fc9c4007d30 comp rx=0 tx=0).stop 2024-04-26T23:07:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.921+0000 7fc9e1e00640 1 -- 172.21.15.40:0/2463477105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9dc0d0620 msgr2=0x7fc9dc1f8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.921+0000 7fc9e1e00640 1 --2- 172.21.15.40:0/2463477105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9dc0d0620 0x7fc9dc1f8760 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc00d5f0 tx=0x7fc9cc00dab0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.921+0000 7fc9e1e00640 1 -- 172.21.15.40:0/2463477105 shutdown_connections 2024-04-26T23:07:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.921+0000 7fc9e1e00640 1 --2- 172.21.15.40:0/2463477105 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc9b806c8f0 0x7fc9b806edb0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.921+0000 7fc9e1e00640 1 --2- 172.21.15.40:0/2463477105 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9dc0c6ca0 0x7fc9dc154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.921+0000 7fc9e1e00640 1 --2- 172.21.15.40:0/2463477105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9dc15f380 0x7fc9dc0c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.921+0000 7fc9e1e00640 1 --2- 172.21.15.40:0/2463477105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9dc0d0620 0x7fc9dc1f8760 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:33.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.921+0000 7fc9e1e00640 1 -- 172.21.15.40:0/2463477105 >> 172.21.15.40:0/2463477105 conn(0x7fc9dc0d0be0 msgr2=0x7fc9dc0cf120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:33.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.921+0000 7fc9e1e00640 1 -- 172.21.15.40:0/2463477105 shutdown_connections 2024-04-26T23:07:33.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:33.921+0000 7fc9e1e00640 1 -- 172.21.15.40:0/2463477105 wait complete. 2024-04-26T23:07:33.941 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-26T23:07:34.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.150+0000 7f7279c00640 1 Processor -- start 2024-04-26T23:07:34.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.150+0000 7f7279c00640 1 -- start start 2024-04-26T23:07:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.151+0000 7f7279c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7274162810 0x7f7274070350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.151+0000 7f7279c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72740d10d0 con 0x7f72740b2340 2024-04-26T23:07:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.151+0000 7f7279c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72740d1370 con 0x7f72740d1b30 2024-04-26T23:07:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.151+0000 7f7279c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f72740d0750 con 0x7f7274162810 2024-04-26T23:07:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.152+0000 7f7272000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f72740b2340 0x7f72740cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34308/0 (socket says 172.21.15.40:34308) 2024-04-26T23:07:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.152+0000 7f7272000640 1 -- 172.21.15.40:0/1023591716 learned_addr learned my addr 172.21.15.40:0/1023591716 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.152+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2680733839 0 0) 0x7f72740d10d0 con 0x7f72740b2340 2024-04-26T23:07:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.152+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7258003520 con 0x7f72740b2340 2024-04-26T23:07:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.152+0000 7f7272a00640 1 --2- 172.21.15.40:0/1023591716 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7274162810 0x7f7274070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2933716456 0 0) 0x7f7258003520 con 0x7f72740b2340 2024-04-26T23:07:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7274162810 msgr2=0x7f7274070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7271600640 1 --2- 172.21.15.40:0/1023591716 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7274162810 0x7f7274070350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 >> v1:172.21.15.103:6789/0 conn(0x7f72740d1b30 legacy=0x7f7274070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72740d01b0 con 0x7f72740b2340 2024-04-26T23:07:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f7260001670 con 0x7f72740b2340 2024-04-26T23:07:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 >> v1:172.21.15.40:6789/0 conn(0x7f72740b2340 legacy=0x7f72740cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:34.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7271600640 1 --2- 172.21.15.40:0/1023591716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72580039e0 0x7f7258003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7271600640 1 --2- 172.21.15.40:0/1023591716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7258004800 0x7f7258004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7271600640 1 --2- 172.21.15.40:0/1023591716 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7274162810 0x7f7258005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72740d0750 con 0x7f72580039e0 2024-04-26T23:07:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f725800baf0 con 0x7f7258004800 2024-04-26T23:07:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f725800bcf0 con 0x7f7274162810 2024-04-26T23:07:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.153+0000 7f7272a00640 1 --2- 172.21.15.40:0/1023591716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72580039e0 0x7f7258003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.154+0000 7f7272000640 1 --2- 172.21.15.40:0/1023591716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7258004800 0x7f7258004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.154+0000 7f7272a00640 1 -- 172.21.15.40:0/1023591716 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7274162810 msgr2=0x7f7258005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.154+0000 7f7272a00640 1 --2- 172.21.15.40:0/1023591716 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7274162810 0x7f7258005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.154+0000 7f7272a00640 1 -- 172.21.15.40:0/1023591716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7258004800 msgr2=0x7f7258004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:34.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.154+0000 7f7272a00640 1 --2- 172.21.15.40:0/1023591716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7258004800 0x7f7258004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.154+0000 7f7272a00640 1 -- 172.21.15.40:0/1023591716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7258003520 con 0x7f72580039e0 2024-04-26T23:07:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.154+0000 7f7272000640 1 --2- 172.21.15.40:0/1023591716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7258004800 0x7f7258004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:34.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.155+0000 7f7272a00640 1 --2- 172.21.15.40:0/1023591716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72580039e0 0x7f7258003e00 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f7264007360 tx=0x7f7258005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d3a6a74775d14de server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.155+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7264009be0 con 0x7f72580039e0 2024-04-26T23:07:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.155+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f726402e620 con 0x7f72580039e0 2024-04-26T23:07:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.155+0000 7f7271600640 1 -- 172.21.15.40:0/1023591716 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726402e9f0 con 0x7f72580039e0 2024-04-26T23:07:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.156+0000 7f7279c00640 1 -- 172.21.15.40:0/1023591716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72580039e0 msgr2=0x7f7258003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.156+0000 7f7279c00640 1 --2- 172.21.15.40:0/1023591716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72580039e0 0x7f7258003e00 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f7264007360 tx=0x7f7258005830 comp rx=0 tx=0).stop 2024-04-26T23:07:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.156+0000 7f7273400640 1 -- 172.21.15.40:0/1023591716 reap_dead start 2024-04-26T23:07:34.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.156+0000 7f7279c00640 1 -- 172.21.15.40:0/1023591716 shutdown_connections 2024-04-26T23:07:34.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.156+0000 7f7279c00640 1 -- 172.21.15.40:0/1023591716 >> 172.21.15.40:0/1023591716 conn(0x7f72740d2da0 msgr2=0x7f72740d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:34.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.156+0000 7f7279c00640 1 -- 172.21.15.40:0/1023591716 shutdown_connections 2024-04-26T23:07:34.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.156+0000 7f7279c00640 1 -- 172.21.15.40:0/1023591716 wait complete. 2024-04-26T23:07:34.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.158+0000 7f7279c00640 1 Processor -- start 2024-04-26T23:07:34.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.158+0000 7f7279c00640 1 -- start start 2024-04-26T23:07:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.159+0000 7f7279c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72580039e0 0x7f72741616c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.159+0000 7f7279c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7274160180 0x7f727415f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.159+0000 7f7279c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72740c85e0 0x7f727415b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.159+0000 7f7279c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72740cdb00 con 0x7f72740c85e0 2024-04-26T23:07:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.159+0000 7f7279c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72740d16a0 con 0x7f7274160180 2024-04-26T23:07:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.159+0000 7f7279c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7274071260 con 0x7f72580039e0 2024-04-26T23:07:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.159+0000 7f7273400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72740c85e0 0x7f727415b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.159+0000 7f7273400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72740c85e0 0x7f727415b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41598/0 (socket says 172.21.15.40:41598) 2024-04-26T23:07:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.159+0000 7f7273400640 1 -- 172.21.15.40:0/3835131858 learned_addr learned my addr 172.21.15.40:0/3835131858 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.160+0000 7f7272000640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7274160180 0x7f727415f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.160+0000 7f7272a00640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72580039e0 0x7f72741616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.160+0000 7f7273400640 1 -- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72580039e0 msgr2=0x7f72741616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.160+0000 7f7273400640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72580039e0 0x7f72741616c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.160+0000 7f7273400640 1 -- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7274160180 msgr2=0x7f727415f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.160+0000 7f7273400640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7274160180 0x7f727415f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.160+0000 7f7273400640 1 -- 172.21.15.40:0/3835131858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7274160620 con 0x7f72740c85e0 2024-04-26T23:07:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.160+0000 7f7273400640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72740c85e0 0x7f727415b030 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f7274073d90 tx=0x7f726c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.161+0000 7f726aa00640 1 -- 172.21.15.40:0/3835131858 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c004220 con 0x7f72740c85e0 2024-04-26T23:07:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.161+0000 7f726aa00640 1 -- 172.21.15.40:0/3835131858 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f726c0043e0 con 0x7f72740c85e0 2024-04-26T23:07:34.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.161+0000 7f726aa00640 1 -- 172.21.15.40:0/3835131858 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c007310 con 0x7f72740c85e0 2024-04-26T23:07:34.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.161+0000 7f7279c00640 1 -- 172.21.15.40:0/3835131858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7274160900 con 0x7f72740c85e0 2024-04-26T23:07:34.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.161+0000 7f7279c00640 1 -- 172.21.15.40:0/3835131858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7274160ec0 con 0x7f72740c85e0 2024-04-26T23:07:34.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.162+0000 7f7279c00640 1 -- 172.21.15.40:0/3835131858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72740694f0 con 0x7f72740c85e0 2024-04-26T23:07:34.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.164+0000 7f726aa00640 1 -- 172.21.15.40:0/3835131858 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f726c0074b0 con 0x7f72740c85e0 2024-04-26T23:07:34.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.165+0000 7f726aa00640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f725406c8f0 0x7f725406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.165+0000 7f726aa00640 1 -- 172.21.15.40:0/3835131858 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6101+0+0 (secure 0 0 0) 0x7f726c08aee0 con 0x7f72740c85e0 2024-04-26T23:07:34.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.166+0000 7f7272a00640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f725406c8f0 0x7f725406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.167+0000 7f7272a00640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f725406c8f0 0x7f725406edb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7264035000 tx=0x7f726400b030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:34.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.170+0000 7f726aa00640 1 -- 172.21.15.40:0/3835131858 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f726c0590e0 con 0x7f72740c85e0 2024-04-26T23:07:34.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.417+0000 7f7279c00640 1 -- 172.21.15.40:0/3835131858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f727406e770 con 0x7f72740c85e0 2024-04-26T23:07:34.419 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:07:34.419+0000 7f36fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T23:07:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.423+0000 7f726aa00640 1 -- 172.21.15.40:0/3835131858 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v83) ==== 161+0+0 (secure 0 0 0) 0x7f726c0589e0 con 0x7f72740c85e0 2024-04-26T23:07:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T23:07:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.427+0000 7f7279c00640 1 -- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f725406c8f0 msgr2=0x7f725406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.427+0000 7f7279c00640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f725406c8f0 0x7f725406edb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7264035000 tx=0x7f726400b030 comp rx=0 tx=0).stop 2024-04-26T23:07:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.427+0000 7f7279c00640 1 -- 172.21.15.40:0/3835131858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72740c85e0 msgr2=0x7f727415b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.427+0000 7f7279c00640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72740c85e0 0x7f727415b030 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f7274073d90 tx=0x7f726c00c820 comp rx=0 tx=0).stop 2024-04-26T23:07:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.431+0000 7f7279c00640 1 -- 172.21.15.40:0/3835131858 shutdown_connections 2024-04-26T23:07:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.431+0000 7f7279c00640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f725406c8f0 0x7f725406edb0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.431+0000 7f7279c00640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72740c85e0 0x7f727415b030 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.431+0000 7f7279c00640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7274160180 0x7f727415f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.431+0000 7f7279c00640 1 --2- 172.21.15.40:0/3835131858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72580039e0 0x7f72741616c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.431+0000 7f7279c00640 1 -- 172.21.15.40:0/3835131858 >> 172.21.15.40:0/3835131858 conn(0x7f72740d2da0 msgr2=0x7f72740b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.431+0000 7f7279c00640 1 -- 172.21.15.40:0/3835131858 shutdown_connections 2024-04-26T23:07:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.431+0000 7f7279c00640 1 -- 172.21.15.40:0/3835131858 wait complete. 2024-04-26T23:07:34.451 INFO:tasks.ceph.mds.c.smithi040.stderr: -9637> 2024-04-26T23:07:02.774+0000 7fdbb2a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T23:07:34.455 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T23:07:34.460 INFO:tasks.ceph.mds.c.smithi040.stderr: -4770> 2024-04-26T23:07:12.775+0000 7fdbb2a00640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T23:07:34.500 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-04-26T23:07:34.518 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:07:34.519 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-26T23:07:34.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.644+0000 7f458fe00640 1 Processor -- start 2024-04-26T23:07:34.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.644+0000 7f458fe00640 1 -- start start 2024-04-26T23:07:34.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.645+0000 7f458fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4590160a50 0x7f4590073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.645+0000 7f458fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45900cfaf0 0x7f4590160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.645+0000 7f458fe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4590161880 con 0x7f459006dd40 2024-04-26T23:07:34.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.645+0000 7f458fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4590071ce0 con 0x7f4590160a50 2024-04-26T23:07:34.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.645+0000 7f458fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f45900722a0 con 0x7f45900cfaf0 2024-04-26T23:07:34.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.646+0000 7f458f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f459006dd40 0x7f4590072730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34322/0 (socket says 172.21.15.40:34322) 2024-04-26T23:07:34.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.646+0000 7f458f400640 1 -- 172.21.15.40:0/2005434621 learned_addr learned my addr 172.21.15.40:0/2005434621 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:34.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.646+0000 7f4587e00640 1 --2- 172.21.15.40:0/2005434621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45900cfaf0 0x7f4590160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.646+0000 7f458ea00640 1 --2- 172.21.15.40:0/2005434621 >> v2:172.21.15.103:3300/0 conn(0x7f4590160a50 0x7f4590073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.646+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4165538519 0 0) 0x7f4590161880 con 0x7f459006dd40 2024-04-26T23:07:34.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.646+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4574003520 con 0x7f459006dd40 2024-04-26T23:07:34.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4001962912 0 0) 0x7f4574003520 con 0x7f459006dd40 2024-04-26T23:07:34.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45900cfaf0 msgr2=0x7f4590160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:34.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 --2- 172.21.15.40:0/2005434621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45900cfaf0 0x7f4590160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 >> v2:172.21.15.103:3300/0 conn(0x7f4590160a50 msgr2=0x7f4590073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:34.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 --2- 172.21.15.40:0/2005434621 >> v2:172.21.15.103:3300/0 conn(0x7f4590160a50 0x7f4590073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4590161370 con 0x7f459006dd40 2024-04-26T23:07:34.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f45880019f0 con 0x7f459006dd40 2024-04-26T23:07:34.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 >> v1:172.21.15.40:6789/0 conn(0x7f459006dd40 legacy=0x7f4590072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:34.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 --2- 172.21.15.40:0/2005434621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45740039b0 0x7f4574003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 --2- 172.21.15.40:0/2005434621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4574004360 0x7f4574004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 --2- 172.21.15.40:0/2005434621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4574004eb0 0x7f457400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4590071ce0 con 0x7f4574004360 2024-04-26T23:07:34.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f45900722a0 con 0x7f4574004eb0 2024-04-26T23:07:34.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f457400bf00 con 0x7f45740039b0 2024-04-26T23:07:34.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f4587e00640 1 --2- 172.21.15.40:0/2005434621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4574004360 0x7f4574004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f4587e00640 1 -- 172.21.15.40:0/2005434621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45740039b0 msgr2=0x7f4574003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:34.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f4587e00640 1 --2- 172.21.15.40:0/2005434621 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45740039b0 0x7f4574003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f4587e00640 1 -- 172.21.15.40:0/2005434621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4574004eb0 msgr2=0x7f457400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:34.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f4587e00640 1 --2- 172.21.15.40:0/2005434621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4574004eb0 0x7f457400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.647+0000 7f4587e00640 1 -- 172.21.15.40:0/2005434621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4574003520 con 0x7f4574004360 2024-04-26T23:07:34.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.648+0000 7f4587e00640 1 --2- 172.21.15.40:0/2005434621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4574004360 0x7f4574004840 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f4578009ac0 tx=0x7f4574005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f04eff45e38d1d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:34.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.649+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4578036070 con 0x7f4574004360 2024-04-26T23:07:34.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.649+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4578031400 con 0x7f4574004360 2024-04-26T23:07:34.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.649+0000 7f458e000640 1 -- 172.21.15.40:0/2005434621 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4578009690 con 0x7f4574004360 2024-04-26T23:07:34.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.649+0000 7f458fe00640 1 -- 172.21.15.40:0/2005434621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4574004360 msgr2=0x7f4574004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:34.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.649+0000 7f458fe00640 1 --2- 172.21.15.40:0/2005434621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4574004360 0x7f4574004840 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f4578009ac0 tx=0x7f4574005830 comp rx=0 tx=0).stop 2024-04-26T23:07:34.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.649+0000 7f458f400640 1 -- 172.21.15.40:0/2005434621 reap_dead start 2024-04-26T23:07:34.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.649+0000 7f458fe00640 1 -- 172.21.15.40:0/2005434621 shutdown_connections 2024-04-26T23:07:34.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.649+0000 7f458fe00640 1 -- 172.21.15.40:0/2005434621 >> 172.21.15.40:0/2005434621 conn(0x7f45900b2720 msgr2=0x7f45900b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:34.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.649+0000 7f458fe00640 1 -- 172.21.15.40:0/2005434621 shutdown_connections 2024-04-26T23:07:34.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.649+0000 7f458fe00640 1 -- 172.21.15.40:0/2005434621 wait complete. 2024-04-26T23:07:34.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.651+0000 7f458fe00640 1 Processor -- start 2024-04-26T23:07:34.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.651+0000 7f458fe00640 1 -- start start 2024-04-26T23:07:34.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.652+0000 7f458fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4574004360 0x7f459013c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.652+0000 7f458fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f459013aca0 0x7f4590140760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.652+0000 7f458fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45901424b0 0x7f4590142960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.652+0000 7f458fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f45900d0260 con 0x7f45901424b0 2024-04-26T23:07:34.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.652+0000 7f458fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4590073f90 con 0x7f459013aca0 2024-04-26T23:07:34.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.652+0000 7f458fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4590071f30 con 0x7f4574004360 2024-04-26T23:07:34.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.653+0000 7f458f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45901424b0 0x7f4590142960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.653+0000 7f458f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45901424b0 0x7f4590142960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41636/0 (socket says 172.21.15.40:41636) 2024-04-26T23:07:34.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.653+0000 7f458f400640 1 -- 172.21.15.40:0/1824034056 learned_addr learned my addr 172.21.15.40:0/1824034056 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:34.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.653+0000 7f458ea00640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4574004360 0x7f459013c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.653+0000 7f4587e00640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f459013aca0 0x7f4590140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.653+0000 7f458f400640 1 -- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4574004360 msgr2=0x7f459013c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.653+0000 7f458f400640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4574004360 0x7f459013c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.653+0000 7f458f400640 1 -- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f459013aca0 msgr2=0x7f4590140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.653+0000 7f458f400640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f459013aca0 0x7f4590140760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.653+0000 7f458f400640 1 -- 172.21.15.40:0/1824034056 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f459013f3e0 con 0x7f45901424b0 2024-04-26T23:07:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.653+0000 7f458f400640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45901424b0 0x7f4590142960 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f458800ca70 tx=0x7f458800ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.654+0000 7f4586a00640 1 -- 172.21.15.40:0/1824034056 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458800a5e0 con 0x7f45901424b0 2024-04-26T23:07:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.654+0000 7f4586a00640 1 -- 172.21.15.40:0/1824034056 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f458800a780 con 0x7f45901424b0 2024-04-26T23:07:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.654+0000 7f4586a00640 1 -- 172.21.15.40:0/1824034056 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458800ab50 con 0x7f45901424b0 2024-04-26T23:07:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.654+0000 7f458fe00640 1 -- 172.21.15.40:0/1824034056 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f459013f6c0 con 0x7f45901424b0 2024-04-26T23:07:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.654+0000 7f458fe00640 1 -- 172.21.15.40:0/1824034056 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f459013fc80 con 0x7f45901424b0 2024-04-26T23:07:34.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.655+0000 7f458fe00640 1 -- 172.21.15.40:0/1824034056 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4590069370 con 0x7f45901424b0 2024-04-26T23:07:34.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.656+0000 7f4586a00640 1 -- 172.21.15.40:0/1824034056 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f458800acf0 con 0x7f45901424b0 2024-04-26T23:07:34.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.657+0000 7f4586a00640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f456c06c8f0 0x7f456c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:34.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.657+0000 7f4586a00640 1 -- 172.21.15.40:0/1824034056 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6359+0+0 (secure 0 0 0) 0x7f458808b3e0 con 0x7f45901424b0 2024-04-26T23:07:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.658+0000 7f458ea00640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f456c06c8f0 0x7f456c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:34.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.659+0000 7f458ea00640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f456c06c8f0 0x7f456c06edb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f45800047b0 tx=0x7f4580008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:34.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.662+0000 7f4586a00640 1 -- 172.21.15.40:0/1824034056 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f45880594f0 con 0x7f45901424b0 2024-04-26T23:07:34.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:34.915+0000 7f458fe00640 1 -- 172.21.15.40:0/1824034056 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f4590146040 con 0x7f45901424b0 2024-04-26T23:07:35.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.427+0000 7f4586a00640 1 -- 172.21.15.40:0/1824034056 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v84) ==== 104+0+0 (secure 0 0 0) 0x7f4588058df0 con 0x7f45901424b0 2024-04-26T23:07:35.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.430+0000 7f458fe00640 1 -- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f456c06c8f0 msgr2=0x7f456c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:35.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.430+0000 7f458fe00640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f456c06c8f0 0x7f456c06edb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f45800047b0 tx=0x7f4580008040 comp rx=0 tx=0).stop 2024-04-26T23:07:35.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.430+0000 7f458fe00640 1 -- 172.21.15.40:0/1824034056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45901424b0 msgr2=0x7f4590142960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:35.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.430+0000 7f458fe00640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45901424b0 0x7f4590142960 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f458800ca70 tx=0x7f458800ce70 comp rx=0 tx=0).stop 2024-04-26T23:07:35.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.431+0000 7f458fe00640 1 -- 172.21.15.40:0/1824034056 shutdown_connections 2024-04-26T23:07:35.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.431+0000 7f458fe00640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f456c06c8f0 0x7f456c06edb0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:35.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.431+0000 7f458fe00640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45901424b0 0x7f4590142960 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:35.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.431+0000 7f458fe00640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f459013aca0 0x7f4590140760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:35.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.431+0000 7f458fe00640 1 --2- 172.21.15.40:0/1824034056 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4574004360 0x7f459013c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:35.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.431+0000 7f458fe00640 1 -- 172.21.15.40:0/1824034056 >> 172.21.15.40:0/1824034056 conn(0x7f45900b2720 msgr2=0x7f459015d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:35.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.431+0000 7f458fe00640 1 -- 172.21.15.40:0/1824034056 shutdown_connections 2024-04-26T23:07:35.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.431+0000 7f458fe00640 1 -- 172.21.15.40:0/1824034056 wait complete. 2024-04-26T23:07:35.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T23:07:35.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.713+0000 7f5285200640 1 Processor -- start 2024-04-26T23:07:35.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.713+0000 7f5285200640 1 -- start start 2024-04-26T23:07:35.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.714+0000 7f5285200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52800704c0 0x7f528015f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:35.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.714+0000 7f5285200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f528006e5e0 con 0x7f52800d03d0 2024-04-26T23:07:35.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.714+0000 7f5285200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f528006eda0 con 0x7f52800cf8e0 2024-04-26T23:07:35.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.714+0000 7f5285200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f5280070e40 con 0x7f52800704c0 2024-04-26T23:07:35.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.714+0000 7f527d400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f52800d03d0 0x7f528015eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49980/0 (socket says 172.21.15.40:49980) 2024-04-26T23:07:35.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.715+0000 7f527d400640 1 -- 172.21.15.40:0/1440563412 learned_addr learned my addr 172.21.15.40:0/1440563412 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:35.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.715+0000 7f527e800640 1 --2- 172.21.15.40:0/1440563412 >> v2:172.21.15.40:3300/0 conn(0x7f52800704c0 0x7f528015f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:35.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.715+0000 7f527e800640 1 -- 172.21.15.40:0/1440563412 >> v1:172.21.15.103:6789/0 conn(0x7f52800cf8e0 legacy=0x7f52800cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:35.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.715+0000 7f527e800640 1 -- 172.21.15.40:0/1440563412 >> v1:172.21.15.40:6789/0 conn(0x7f52800d03d0 legacy=0x7f528015eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:35.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.715+0000 7f527e800640 1 -- 172.21.15.40:0/1440563412 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5280071120 con 0x7f52800704c0 2024-04-26T23:07:35.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.715+0000 7f527ca00640 1 -- 172.21.15.40:0/1440563412 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3839201477 0 0) 0x7f528006e5e0 con 0x7f52800d03d0 2024-04-26T23:07:35.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.716+0000 7f527e800640 1 --2- 172.21.15.40:0/1440563412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52800704c0 0x7f528015f280 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f52780097a0 tx=0x7f527802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283b3d29716c9710 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:35.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.716+0000 7f527ca00640 1 -- 172.21.15.40:0/1440563412 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5278037070 con 0x7f52800704c0 2024-04-26T23:07:35.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.717+0000 7f527ca00640 1 -- 172.21.15.40:0/1440563412 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5278032440 con 0x7f52800704c0 2024-04-26T23:07:35.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.717+0000 7f527ca00640 1 -- 172.21.15.40:0/1440563412 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5278032810 con 0x7f52800704c0 2024-04-26T23:07:35.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.717+0000 7f5285200640 1 -- 172.21.15.40:0/1440563412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52800704c0 msgr2=0x7f528015f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:35.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.717+0000 7f5285200640 1 --2- 172.21.15.40:0/1440563412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52800704c0 0x7f528015f280 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f52780097a0 tx=0x7f527802a1e0 comp rx=0 tx=0).stop 2024-04-26T23:07:35.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.717+0000 7f5285200640 1 -- 172.21.15.40:0/1440563412 shutdown_connections 2024-04-26T23:07:35.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.717+0000 7f5285200640 1 --2- 172.21.15.40:0/1440563412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52800704c0 0x7f528015f280 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:35.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.717+0000 7f5285200640 1 -- 172.21.15.40:0/1440563412 >> 172.21.15.40:0/1440563412 conn(0x7f52800b4a50 msgr2=0x7f52800b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:35.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.717+0000 7f5285200640 1 -- 172.21.15.40:0/1440563412 shutdown_connections 2024-04-26T23:07:35.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.717+0000 7f5285200640 1 -- 172.21.15.40:0/1440563412 wait complete. 2024-04-26T23:07:35.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.719+0000 7f5285200640 1 Processor -- start 2024-04-26T23:07:35.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.719+0000 7f5285200640 1 -- start start 2024-04-26T23:07:35.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.720+0000 7f5285200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52800704c0 0x7f528015e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:35.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.720+0000 7f5285200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52800cf8e0 0x7f52801554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:35.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.720+0000 7f5285200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52800d03d0 0x7f52800b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:35.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.720+0000 7f5285200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f52800ce560 con 0x7f52800d03d0 2024-04-26T23:07:35.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.720+0000 7f5285200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f528006cfc0 con 0x7f52800cf8e0 2024-04-26T23:07:35.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.720+0000 7f5285200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f52800cf270 con 0x7f52800704c0 2024-04-26T23:07:35.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.720+0000 7f527e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52800d03d0 0x7f52800b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:35.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.720+0000 7f527e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52800d03d0 0x7f52800b7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53362/0 (socket says 172.21.15.40:53362) 2024-04-26T23:07:35.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.720+0000 7f527e800640 1 -- 172.21.15.40:0/1573153251 learned_addr learned my addr 172.21.15.40:0/1573153251 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:35.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.721+0000 7f527de00640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52800704c0 0x7f528015e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:35.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.721+0000 7f527d400640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52800cf8e0 0x7f52801554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:35.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.721+0000 7f527e800640 1 -- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52800704c0 msgr2=0x7f528015e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:35.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.721+0000 7f527e800640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52800704c0 0x7f528015e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:35.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.721+0000 7f527e800640 1 -- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52800cf8e0 msgr2=0x7f52801554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:35.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.721+0000 7f527e800640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52800cf8e0 0x7f52801554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:35.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.721+0000 7f527e800640 1 -- 172.21.15.40:0/1573153251 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528015d580 con 0x7f52800d03d0 2024-04-26T23:07:35.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.721+0000 7f527d400640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52800cf8e0 0x7f52801554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:35.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.721+0000 7f527e800640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52800d03d0 0x7f52800b7410 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f52780097a0 tx=0x7f52780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:35.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.722+0000 7f5276000640 1 -- 172.21.15.40:0/1573153251 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5278041070 con 0x7f52800d03d0 2024-04-26T23:07:35.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.722+0000 7f5276000640 1 -- 172.21.15.40:0/1573153251 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f527803c870 con 0x7f52800d03d0 2024-04-26T23:07:35.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.722+0000 7f5276000640 1 -- 172.21.15.40:0/1573153251 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5278037040 con 0x7f52800d03d0 2024-04-26T23:07:35.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.722+0000 7f5285200640 1 -- 172.21.15.40:0/1573153251 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f528015d860 con 0x7f52800d03d0 2024-04-26T23:07:35.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.722+0000 7f5285200640 1 -- 172.21.15.40:0/1573153251 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f528015de20 con 0x7f52800d03d0 2024-04-26T23:07:35.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.723+0000 7f5285200640 1 -- 172.21.15.40:0/1573153251 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5280069600 con 0x7f52800d03d0 2024-04-26T23:07:35.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.724+0000 7f5276000640 1 -- 172.21.15.40:0/1573153251 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5278044070 con 0x7f52800d03d0 2024-04-26T23:07:35.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.725+0000 7f5276000640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f526006c8f0 0x7f526006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:35.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.726+0000 7f5276000640 1 -- 172.21.15.40:0/1573153251 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6359+0+0 (secure 0 0 0) 0x7f52780aca30 con 0x7f52800d03d0 2024-04-26T23:07:35.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.726+0000 7f527de00640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f526006c8f0 0x7f526006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:35.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.727+0000 7f527de00640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f526006c8f0 0x7f526006edb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5270009f20 tx=0x7f52700094d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:35.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.731+0000 7f5276000640 1 -- 172.21.15.40:0/1573153251 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f527807ac50 con 0x7f52800d03d0 2024-04-26T23:07:35.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:35.939+0000 7f5285200640 1 -- 172.21.15.40:0/1573153251 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5280070aa0 con 0x7f52800d03d0 2024-04-26T23:07:36.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.673+0000 7f5276000640 1 -- 172.21.15.40:0/1573153251 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v81) ==== 181+0+0 (secure 0 0 0) 0x7f527807a550 con 0x7f52800d03d0 2024-04-26T23:07:36.674 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-04-26T23:07:36.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.677+0000 7f5285200640 1 -- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f526006c8f0 msgr2=0x7f526006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:36.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.677+0000 7f5285200640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f526006c8f0 0x7f526006edb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5270009f20 tx=0x7f52700094d0 comp rx=0 tx=0).stop 2024-04-26T23:07:36.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.677+0000 7f5285200640 1 -- 172.21.15.40:0/1573153251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52800d03d0 msgr2=0x7f52800b7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:36.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.677+0000 7f5285200640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52800d03d0 0x7f52800b7410 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f52780097a0 tx=0x7f52780046c0 comp rx=0 tx=0).stop 2024-04-26T23:07:36.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.678+0000 7f5285200640 1 -- 172.21.15.40:0/1573153251 shutdown_connections 2024-04-26T23:07:36.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.678+0000 7f5285200640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f526006c8f0 0x7f526006edb0 secure :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5270009f20 tx=0x7f52700094d0 comp rx=0 tx=0).stop 2024-04-26T23:07:36.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.678+0000 7f5285200640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52800d03d0 0x7f52800b7410 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:36.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.678+0000 7f5285200640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52800cf8e0 0x7f52801554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:36.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.678+0000 7f5285200640 1 --2- 172.21.15.40:0/1573153251 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f52800704c0 0x7f528015e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:36.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.678+0000 7f5285200640 1 -- 172.21.15.40:0/1573153251 >> 172.21.15.40:0/1573153251 conn(0x7f52800b4a50 msgr2=0x7f52800cfcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:36.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.679+0000 7f5285200640 1 -- 172.21.15.40:0/1573153251 shutdown_connections 2024-04-26T23:07:36.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.680+0000 7f5285200640 1 -- 172.21.15.40:0/1573153251 wait complete. 2024-04-26T23:07:36.703 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T23:07:36.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.920+0000 7f2b10200640 1 Processor -- start 2024-04-26T23:07:36.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.920+0000 7f2b10200640 1 -- start start 2024-04-26T23:07:36.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.922+0000 7f2b10200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b0806eae0 0x7f2b080b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:36.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.922+0000 7f2b10200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2b0815a4a0 0x7f2b0815a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:36.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.922+0000 7f2b10200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b0815d8d0 con 0x7f2b0806e700 2024-04-26T23:07:36.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.922+0000 7f2b10200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2b0815ae50 con 0x7f2b0815a4a0 2024-04-26T23:07:36.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.922+0000 7f2b10200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2b0815dbb0 con 0x7f2b0806eae0 2024-04-26T23:07:36.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.922+0000 7f2b0d800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2b0806e700 0x7f2b080ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49986/0 (socket says 172.21.15.40:49986) 2024-04-26T23:07:36.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.922+0000 7f2b0d800640 1 -- 172.21.15.40:0/745716180 learned_addr learned my addr 172.21.15.40:0/745716180 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:36.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b0e200640 1 --2- 172.21.15.40:0/745716180 >> v2:172.21.15.103:3300/0 conn(0x7f2b0815a4a0 0x7f2b0815a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:36.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b0ce00640 1 --2- 172.21.15.40:0/745716180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b0806eae0 0x7f2b080b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:36.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3444890464 0 0) 0x7f2b0815d8d0 con 0x7f2b0806e700 2024-04-26T23:07:36.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2af0003520 con 0x7f2b0806e700 2024-04-26T23:07:36.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2221672469 0 0) 0x7f2af0003520 con 0x7f2b0806e700 2024-04-26T23:07:36.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b0806eae0 msgr2=0x7f2b080b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:36.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b07e00640 1 --2- 172.21.15.40:0/745716180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b0806eae0 0x7f2b080b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:36.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 >> v2:172.21.15.103:3300/0 conn(0x7f2b0815a4a0 msgr2=0x7f2b0815a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:36.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b07e00640 1 --2- 172.21.15.40:0/745716180 >> v2:172.21.15.103:3300/0 conn(0x7f2b0815a4a0 0x7f2b0815a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:36.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0815de90 con 0x7f2b0806e700 2024-04-26T23:07:36.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f2afc001670 con 0x7f2b0806e700 2024-04-26T23:07:36.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 >> v1:172.21.15.40:6789/0 conn(0x7f2b0806e700 legacy=0x7f2b080ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:36.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b07e00640 1 --2- 172.21.15.40:0/745716180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2af00039b0 0x7f2af0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:36.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b07e00640 1 --2- 172.21.15.40:0/745716180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b0806eae0 0x7f2af0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:36.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.923+0000 7f2b07e00640 1 --2- 172.21.15.40:0/745716180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af0004d40 0x7f2af00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:36.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.924+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0815ae50 con 0x7f2af0004d40 2024-04-26T23:07:36.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.924+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0815dbb0 con 0x7f2af00039b0 2024-04-26T23:07:36.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.924+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2af000b9c0 con 0x7f2b0806eae0 2024-04-26T23:07:36.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.924+0000 7f2b0e200640 1 --2- 172.21.15.40:0/745716180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af0004d40 0x7f2af00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:36.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.924+0000 7f2b0d800640 1 --2- 172.21.15.40:0/745716180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2af00039b0 0x7f2af0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:36.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.924+0000 7f2b0ce00640 1 --2- 172.21.15.40:0/745716180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b0806eae0 0x7f2af0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:36.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.924+0000 7f2b0e200640 1 -- 172.21.15.40:0/745716180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b0806eae0 msgr2=0x7f2af0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:36.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.924+0000 7f2b0e200640 1 --2- 172.21.15.40:0/745716180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b0806eae0 0x7f2af0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:36.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.924+0000 7f2b0e200640 1 -- 172.21.15.40:0/745716180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2af00039b0 msgr2=0x7f2af0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:36.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.924+0000 7f2b0e200640 1 --2- 172.21.15.40:0/745716180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2af00039b0 0x7f2af0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:36.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.924+0000 7f2b0e200640 1 -- 172.21.15.40:0/745716180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af0003520 con 0x7f2af0004d40 2024-04-26T23:07:36.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.924+0000 7f2b0ce00640 1 --2- 172.21.15.40:0/745716180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b0806eae0 0x7f2af0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:36.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.926+0000 7f2b0e200640 1 --2- 172.21.15.40:0/745716180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af0004d40 0x7f2af00051c0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f2b0000e170 tx=0x7f2af0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11702b4770cef1c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:36.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.926+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b00034020 con 0x7f2af0004d40 2024-04-26T23:07:36.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.926+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b00030430 con 0x7f2af0004d40 2024-04-26T23:07:36.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.926+0000 7f2b07e00640 1 -- 172.21.15.40:0/745716180 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b00009c10 con 0x7f2af0004d40 2024-04-26T23:07:36.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.926+0000 7f2b10200640 1 -- 172.21.15.40:0/745716180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af0004d40 msgr2=0x7f2af00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:36.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.926+0000 7f2b10200640 1 --2- 172.21.15.40:0/745716180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af0004d40 0x7f2af00051c0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f2b0000e170 tx=0x7f2af0005830 comp rx=0 tx=0).stop 2024-04-26T23:07:36.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.926+0000 7f2b0e200640 1 -- 172.21.15.40:0/745716180 reap_dead start 2024-04-26T23:07:36.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.927+0000 7f2b10200640 1 -- 172.21.15.40:0/745716180 shutdown_connections 2024-04-26T23:07:36.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.927+0000 7f2b10200640 1 -- 172.21.15.40:0/745716180 >> 172.21.15.40:0/745716180 conn(0x7f2b0806f310 msgr2=0x7f2b0806f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:36.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.927+0000 7f2b10200640 1 -- 172.21.15.40:0/745716180 shutdown_connections 2024-04-26T23:07:36.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.927+0000 7f2b10200640 1 -- 172.21.15.40:0/745716180 wait complete. 2024-04-26T23:07:36.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.929+0000 7f2b10200640 1 Processor -- start 2024-04-26T23:07:36.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.929+0000 7f2b10200640 1 -- start start 2024-04-26T23:07:36.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b10200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af0004d40 0x7f2b0815b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b10200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b08157c10 0x7f2b081f4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b10200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b081f53f0 0x7f2b0815ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b10200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2b080d0a40 con 0x7f2af0004d40 2024-04-26T23:07:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b10200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2b080d08c0 con 0x7f2b08157c10 2024-04-26T23:07:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b10200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2b0815d480 con 0x7f2b081f53f0 2024-04-26T23:07:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b0d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af0004d40 0x7f2b0815b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b0d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af0004d40 0x7f2b0815b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53388/0 (socket says 172.21.15.40:53388) 2024-04-26T23:07:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b0d800640 1 -- 172.21.15.40:0/2407049063 learned_addr learned my addr 172.21.15.40:0/2407049063 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b0d800640 1 -- 172.21.15.40:0/2407049063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b081f53f0 msgr2=0x7f2b0815ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b0d800640 1 --2- 172.21.15.40:0/2407049063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b081f53f0 0x7f2b0815ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b0d800640 1 -- 172.21.15.40:0/2407049063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b08157c10 msgr2=0x7f2b081f4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:36.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b0d800640 1 --2- 172.21.15.40:0/2407049063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b08157c10 0x7f2b081f4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:36.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.930+0000 7f2b0d800640 1 -- 172.21.15.40:0/2407049063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0815eef0 con 0x7f2af0004d40 2024-04-26T23:07:36.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.931+0000 7f2b0d800640 1 --2- 172.21.15.40:0/2407049063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af0004d40 0x7f2b0815b280 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00e850 tx=0x7f2afc00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:36.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.931+0000 7f2b05600640 1 -- 172.21.15.40:0/2407049063 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc00c870 con 0x7f2af0004d40 2024-04-26T23:07:36.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.931+0000 7f2b05600640 1 -- 172.21.15.40:0/2407049063 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2afc018070 con 0x7f2af0004d40 2024-04-26T23:07:36.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.931+0000 7f2b10200640 1 -- 172.21.15.40:0/2407049063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b0815f1d0 con 0x7f2af0004d40 2024-04-26T23:07:36.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.931+0000 7f2b05600640 1 -- 172.21.15.40:0/2407049063 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc00cc50 con 0x7f2af0004d40 2024-04-26T23:07:36.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.931+0000 7f2b10200640 1 -- 172.21.15.40:0/2407049063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b0815fa70 con 0x7f2af0004d40 2024-04-26T23:07:36.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.932+0000 7f2b10200640 1 -- 172.21.15.40:0/2407049063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b08069600 con 0x7f2af0004d40 2024-04-26T23:07:36.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.933+0000 7f2b05600640 1 -- 172.21.15.40:0/2407049063 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2afc01d460 con 0x7f2af0004d40 2024-04-26T23:07:36.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.934+0000 7f2b05600640 1 --2- 172.21.15.40:0/2407049063 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2ae806c8f0 0x7f2ae806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:36.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.934+0000 7f2b05600640 1 -- 172.21.15.40:0/2407049063 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5921+0+0 (secure 0 0 0) 0x7f2afc08bbb0 con 0x7f2af0004d40 2024-04-26T23:07:36.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.935+0000 7f2b0ce00640 1 --2- 172.21.15.40:0/2407049063 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2ae806c8f0 0x7f2ae806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:36.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.936+0000 7f2b0ce00640 1 --2- 172.21.15.40:0/2407049063 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2ae806c8f0 0x7f2ae806edb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2b0815d120 tx=0x7f2af4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:36.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:36.939+0000 7f2b05600640 1 -- 172.21.15.40:0/2407049063 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2afc059e60 con 0x7f2af0004d40 2024-04-26T23:07:37.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.158+0000 7f2b10200640 1 -- 172.21.15.40:0/2407049063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2b080cda00 con 0x7f2af0004d40 2024-04-26T23:07:37.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.686+0000 7f2b05600640 1 -- 172.21.15.40:0/2407049063 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v82) ==== 169+0+0 (secure 0 0 0) 0x7f2afc059760 con 0x7f2af0004d40 2024-04-26T23:07:37.696 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-04-26T23:07:37.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.689+0000 7f2b10200640 1 -- 172.21.15.40:0/2407049063 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2ae806c8f0 msgr2=0x7f2ae806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.689+0000 7f2b10200640 1 --2- 172.21.15.40:0/2407049063 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2ae806c8f0 0x7f2ae806edb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2b0815d120 tx=0x7f2af4008040 comp rx=0 tx=0).stop 2024-04-26T23:07:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.690+0000 7f2b10200640 1 -- 172.21.15.40:0/2407049063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af0004d40 msgr2=0x7f2b0815b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.690+0000 7f2b10200640 1 --2- 172.21.15.40:0/2407049063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af0004d40 0x7f2b0815b280 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00e850 tx=0x7f2afc00ed10 comp rx=0 tx=0).stop 2024-04-26T23:07:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.690+0000 7f2b10200640 1 -- 172.21.15.40:0/2407049063 shutdown_connections 2024-04-26T23:07:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.690+0000 7f2b10200640 1 --2- 172.21.15.40:0/2407049063 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2ae806c8f0 0x7f2ae806edb0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.690+0000 7f2b10200640 1 --2- 172.21.15.40:0/2407049063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b081f53f0 0x7f2b0815ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.690+0000 7f2b10200640 1 --2- 172.21.15.40:0/2407049063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b08157c10 0x7f2b081f4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.690+0000 7f2b10200640 1 --2- 172.21.15.40:0/2407049063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af0004d40 0x7f2b0815b280 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.690+0000 7f2b10200640 1 -- 172.21.15.40:0/2407049063 >> 172.21.15.40:0/2407049063 conn(0x7f2b0806f310 msgr2=0x7f2b08159960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.690+0000 7f2b10200640 1 -- 172.21.15.40:0/2407049063 shutdown_connections 2024-04-26T23:07:37.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.690+0000 7f2b10200640 1 -- 172.21.15.40:0/2407049063 wait complete. 2024-04-26T23:07:37.718 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-26T23:07:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.934+0000 7feb17e00640 1 Processor -- start 2024-04-26T23:07:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.934+0000 7feb17e00640 1 -- start start 2024-04-26T23:07:37.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.936+0000 7feb17e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb18163290 0x7feb180d7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:37.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.936+0000 7feb17e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feb18072360 0x7feb180d4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:37.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.936+0000 7feb17e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb180d8320 con 0x7feb180d7cc0 2024-04-26T23:07:37.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.936+0000 7feb17e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7feb180d8100 con 0x7feb18072360 2024-04-26T23:07:37.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.936+0000 7feb17e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feb180d3d70 con 0x7feb18163290 2024-04-26T23:07:37.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.937+0000 7feb16000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7feb180d7cc0 0x7feb18071c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50002/0 (socket says 172.21.15.40:50002) 2024-04-26T23:07:37.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.937+0000 7feb16000640 1 -- 172.21.15.40:0/3514791244 learned_addr learned my addr 172.21.15.40:0/3514791244 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:37.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.937+0000 7feb17400640 1 --2- 172.21.15.40:0/3514791244 >> v2:172.21.15.40:3300/0 conn(0x7feb18072360 0x7feb180d4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:37.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.938+0000 7feb15600640 1 -- 172.21.15.40:0/3514791244 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 48199107 0 0) 0x7feb180d8320 con 0x7feb180d7cc0 2024-04-26T23:07:37.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.938+0000 7feb15600640 1 -- 172.21.15.40:0/3514791244 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feafc003520 con 0x7feb180d7cc0 2024-04-26T23:07:37.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.938+0000 7feb17400640 1 -- 172.21.15.40:0/3514791244 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb18163290 msgr2=0x7feb180d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:37.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.938+0000 7feb17400640 1 --2- 172.21.15.40:0/3514791244 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb18163290 0x7feb180d7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:37.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.938+0000 7feb17400640 1 -- 172.21.15.40:0/3514791244 >> v1:172.21.15.40:6789/0 conn(0x7feb180d7cc0 legacy=0x7feb18071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:37.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.938+0000 7feb17400640 1 -- 172.21.15.40:0/3514791244 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb180d8600 con 0x7feb18072360 2024-04-26T23:07:37.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.940+0000 7feb17400640 1 --2- 172.21.15.40:0/3514791244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb18072360 0x7feb180d4200 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7feb10009d90 tx=0x7feb1002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a8b65aec1810e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:37.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.941+0000 7feb15600640 1 -- 172.21.15.40:0/3514791244 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb10007740 con 0x7feb18072360 2024-04-26T23:07:37.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.941+0000 7feb15600640 1 -- 172.21.15.40:0/3514791244 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb100078e0 con 0x7feb18072360 2024-04-26T23:07:37.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.941+0000 7feb15600640 1 -- 172.21.15.40:0/3514791244 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb10007cb0 con 0x7feb18072360 2024-04-26T23:07:37.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.942+0000 7feb17e00640 1 -- 172.21.15.40:0/3514791244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb18072360 msgr2=0x7feb180d4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:37.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.942+0000 7feb17e00640 1 --2- 172.21.15.40:0/3514791244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb18072360 0x7feb180d4200 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7feb10009d90 tx=0x7feb1002a660 comp rx=0 tx=0).stop 2024-04-26T23:07:37.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.942+0000 7feb17e00640 1 -- 172.21.15.40:0/3514791244 shutdown_connections 2024-04-26T23:07:37.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.942+0000 7feb17e00640 1 --2- 172.21.15.40:0/3514791244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb18072360 0x7feb180d4200 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:37.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.942+0000 7feb17e00640 1 --2- 172.21.15.40:0/3514791244 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb18163290 0x7feb180d7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:37.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.942+0000 7feb17e00640 1 -- 172.21.15.40:0/3514791244 >> 172.21.15.40:0/3514791244 conn(0x7feb180d91a0 msgr2=0x7feb180d9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:37.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.942+0000 7feb17e00640 1 -- 172.21.15.40:0/3514791244 shutdown_connections 2024-04-26T23:07:37.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.943+0000 7feb17e00640 1 -- 172.21.15.40:0/3514791244 wait complete. 2024-04-26T23:07:37.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.945+0000 7feb17e00640 1 Processor -- start 2024-04-26T23:07:37.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.946+0000 7feb17e00640 1 -- start start 2024-04-26T23:07:37.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.947+0000 7feb17e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb18072360 0x7feb180d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:37.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.947+0000 7feb17e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb180d7cc0 0x7feb180b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:37.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.947+0000 7feb17e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb18163290 0x7feb180b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:37.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.947+0000 7feb17e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb18163670 con 0x7feb180d7cc0 2024-04-26T23:07:37.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.947+0000 7feb17e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb180b8730 con 0x7feb18072360 2024-04-26T23:07:37.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.947+0000 7feb17e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feb181637f0 con 0x7feb18163290 2024-04-26T23:07:37.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.947+0000 7feb16000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb180d7cc0 0x7feb180b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:37.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.947+0000 7feb16000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb180d7cc0 0x7feb180b37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53406/0 (socket says 172.21.15.40:53406) 2024-04-26T23:07:37.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.947+0000 7feb16000640 1 -- 172.21.15.40:0/2371005955 learned_addr learned my addr 172.21.15.40:0/2371005955 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:37.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.948+0000 7feb16a00640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb18072360 0x7feb180d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:37.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.948+0000 7feb17400640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb18163290 0x7feb180b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:37.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.948+0000 7feb16000640 1 -- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb18163290 msgr2=0x7feb180b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:37.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.948+0000 7feb16000640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb18163290 0x7feb180b51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:37.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.948+0000 7feb16000640 1 -- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb18072360 msgr2=0x7feb180d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:37.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.948+0000 7feb16000640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb18072360 0x7feb180d12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:37.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.948+0000 7feb16000640 1 -- 172.21.15.40:0/2371005955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb180d1970 con 0x7feb180d7cc0 2024-04-26T23:07:37.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.948+0000 7feb16000640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb180d7cc0 0x7feb180b37a0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7feb0400c500 tx=0x7feb0400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:37.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.948+0000 7feb0ea00640 1 -- 172.21.15.40:0/2371005955 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb04017070 con 0x7feb180d7cc0 2024-04-26T23:07:37.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.949+0000 7feb0ea00640 1 -- 172.21.15.40:0/2371005955 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb04005ce0 con 0x7feb180d7cc0 2024-04-26T23:07:37.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.949+0000 7feb17e00640 1 -- 172.21.15.40:0/2371005955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb180d1c50 con 0x7feb180d7cc0 2024-04-26T23:07:37.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.949+0000 7feb0ea00640 1 -- 172.21.15.40:0/2371005955 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb04007330 con 0x7feb180d7cc0 2024-04-26T23:07:37.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.949+0000 7feb17e00640 1 -- 172.21.15.40:0/2371005955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb180d2210 con 0x7feb180d7cc0 2024-04-26T23:07:37.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.950+0000 7feb17e00640 1 -- 172.21.15.40:0/2371005955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb18069370 con 0x7feb180d7cc0 2024-04-26T23:07:37.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.951+0000 7feb0ea00640 1 -- 172.21.15.40:0/2371005955 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feb0401a070 con 0x7feb180d7cc0 2024-04-26T23:07:37.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.952+0000 7feb0ea00640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7feaf806c8f0 0x7feaf806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:37.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.952+0000 7feb0ea00640 1 -- 172.21.15.40:0/2371005955 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5507+0+0 (secure 0 0 0) 0x7feb0408afd0 con 0x7feb180d7cc0 2024-04-26T23:07:37.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.953+0000 7feb16a00640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7feaf806c8f0 0x7feaf806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:37.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.954+0000 7feb16a00640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7feaf806c8f0 0x7feaf806edb0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7feb0800b810 tx=0x7feb08008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:37.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:37.956+0000 7feb0ea00640 1 -- 172.21.15.40:0/2371005955 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7feb04059570 con 0x7feb180d7cc0 2024-04-26T23:07:38.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.161+0000 7feb17e00640 1 -- 172.21.15.40:0/2371005955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7feb180ba040 con 0x7feb180d7cc0 2024-04-26T23:07:38.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.168+0000 7feb0ea00640 1 -- 172.21.15.40:0/2371005955 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) ==== 104+0+0 (secure 0 0 0) 0x7feb04058e70 con 0x7feb180d7cc0 2024-04-26T23:07:38.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.171+0000 7feb17e00640 1 -- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7feaf806c8f0 msgr2=0x7feaf806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:38.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.171+0000 7feb17e00640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7feaf806c8f0 0x7feaf806edb0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7feb0800b810 tx=0x7feb08008040 comp rx=0 tx=0).stop 2024-04-26T23:07:38.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.171+0000 7feb17e00640 1 -- 172.21.15.40:0/2371005955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb180d7cc0 msgr2=0x7feb180b37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:38.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.171+0000 7feb17e00640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb180d7cc0 0x7feb180b37a0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7feb0400c500 tx=0x7feb0400c9c0 comp rx=0 tx=0).stop 2024-04-26T23:07:38.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.171+0000 7feb17e00640 1 -- 172.21.15.40:0/2371005955 shutdown_connections 2024-04-26T23:07:38.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.171+0000 7feb17e00640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7feaf806c8f0 0x7feaf806edb0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:38.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.171+0000 7feb17e00640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb18163290 0x7feb180b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:38.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.171+0000 7feb17e00640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb180d7cc0 0x7feb180b37a0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:38.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.171+0000 7feb17e00640 1 --2- 172.21.15.40:0/2371005955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb18072360 0x7feb180d12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:38.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.171+0000 7feb17e00640 1 -- 172.21.15.40:0/2371005955 >> 172.21.15.40:0/2371005955 conn(0x7feb180d91a0 msgr2=0x7feb1806f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:38.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.171+0000 7feb17e00640 1 -- 172.21.15.40:0/2371005955 shutdown_connections 2024-04-26T23:07:38.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.171+0000 7feb17e00640 1 -- 172.21.15.40:0/2371005955 wait complete. 2024-04-26T23:07:38.190 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_non_mds0' 2024-04-26T23:07:38.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.387+0000 7f90f4e00640 1 Processor -- start 2024-04-26T23:07:38.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.387+0000 7f90f4e00640 1 -- start start 2024-04-26T23:07:38.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.389+0000 7f90f4e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f90f0071e60 0x7f90f006f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:38.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.389+0000 7f90f4e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90f015d7c0 con 0x7f90f00cc480 2024-04-26T23:07:38.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.389+0000 7f90f4e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90f015daa0 con 0x7f90f006fe20 2024-04-26T23:07:38.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.389+0000 7f90f4e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f90f015dd80 con 0x7f90f0071e60 2024-04-26T23:07:38.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.389+0000 7f90ee800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f90f00cc480 0x7f90f0166ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50012/0 (socket says 172.21.15.40:50012) 2024-04-26T23:07:38.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.389+0000 7f90ee800640 1 -- 172.21.15.40:0/4276636380 learned_addr learned my addr 172.21.15.40:0/4276636380 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.390+0000 7f90ede00640 1 --2- 172.21.15.40:0/4276636380 >> v2:172.21.15.103:3300/0 conn(0x7f90f0071e60 0x7f90f006f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.390+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3655021586 0 0) 0x7f90f015d7c0 con 0x7f90f00cc480 2024-04-26T23:07:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.390+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90d4003520 con 0x7f90f00cc480 2024-04-26T23:07:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.390+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 997060183 0 0) 0x7f90d4003520 con 0x7f90f00cc480 2024-04-26T23:07:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.390+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 >> v2:172.21.15.103:3300/0 conn(0x7f90f0071e60 msgr2=0x7f90f006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.390+0000 7f90ed400640 1 --2- 172.21.15.40:0/4276636380 >> v2:172.21.15.103:3300/0 conn(0x7f90f0071e60 0x7f90f006f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.390+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 >> v1:172.21.15.103:6789/0 conn(0x7f90f006fe20 legacy=0x7f90f0072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.390+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f015e060 con 0x7f90f00cc480 2024-04-26T23:07:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.390+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f90e80019f0 con 0x7f90f00cc480 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.390+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 >> v1:172.21.15.40:6789/0 conn(0x7f90f00cc480 legacy=0x7f90f0166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.390+0000 7f90ed400640 1 --2- 172.21.15.40:0/4276636380 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90d40039b0 0x7f90d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90ed400640 1 --2- 172.21.15.40:0/4276636380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90d4004800 0x7f90d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90ed400640 1 --2- 172.21.15.40:0/4276636380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90d40051c0 0x7f90d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f90f015dd80 con 0x7f90d40051c0 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f90d400c060 con 0x7f90d4004800 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f90d400c260 con 0x7f90d40039b0 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90ee800640 1 --2- 172.21.15.40:0/4276636380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90d40051c0 0x7f90d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90e7e00640 1 --2- 172.21.15.40:0/4276636380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90d4004800 0x7f90d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90ede00640 1 --2- 172.21.15.40:0/4276636380 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90d40039b0 0x7f90d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90ee800640 1 -- 172.21.15.40:0/4276636380 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90d40039b0 msgr2=0x7f90d4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90ee800640 1 --2- 172.21.15.40:0/4276636380 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90d40039b0 0x7f90d4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90ee800640 1 -- 172.21.15.40:0/4276636380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90d4004800 msgr2=0x7f90d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90ee800640 1 --2- 172.21.15.40:0/4276636380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90d4004800 0x7f90d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:38.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90ee800640 1 -- 172.21.15.40:0/4276636380 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d4003520 con 0x7f90d40051c0 2024-04-26T23:07:38.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.391+0000 7f90e7e00640 1 --2- 172.21.15.40:0/4276636380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90d4004800 0x7f90d4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:07:38.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.392+0000 7f90ee800640 1 -- 172.21.15.40:0/4276636380 reap_dead start 2024-04-26T23:07:38.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.393+0000 7f90ee800640 1 --2- 172.21.15.40:0/4276636380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90d40051c0 0x7f90d400b9c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f90e800a840 tx=0x7f90d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83d310af1cb87f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:38.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.393+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e800a460 con 0x7f90d40051c0 2024-04-26T23:07:38.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.393+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90e802f070 con 0x7f90d40051c0 2024-04-26T23:07:38.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.393+0000 7f90ed400640 1 -- 172.21.15.40:0/4276636380 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e802bcb0 con 0x7f90d40051c0 2024-04-26T23:07:38.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.393+0000 7f90f4e00640 1 -- 172.21.15.40:0/4276636380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90d40051c0 msgr2=0x7f90d400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:38.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.393+0000 7f90f4e00640 1 --2- 172.21.15.40:0/4276636380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90d40051c0 0x7f90d400b9c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f90e800a840 tx=0x7f90d4005830 comp rx=0 tx=0).stop 2024-04-26T23:07:38.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.394+0000 7f90f4e00640 1 -- 172.21.15.40:0/4276636380 shutdown_connections 2024-04-26T23:07:38.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.394+0000 7f90f4e00640 1 --2- 172.21.15.40:0/4276636380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90d40051c0 0x7f90d400b9c0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:38.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.394+0000 7f90f4e00640 1 -- 172.21.15.40:0/4276636380 >> 172.21.15.40:0/4276636380 conn(0x7f90f00d0be0 msgr2=0x7f90f00d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:38.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.394+0000 7f90f4e00640 1 -- 172.21.15.40:0/4276636380 shutdown_connections 2024-04-26T23:07:38.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.394+0000 7f90f4e00640 1 -- 172.21.15.40:0/4276636380 wait complete. 2024-04-26T23:07:38.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.396+0000 7f90f4e00640 1 Processor -- start 2024-04-26T23:07:38.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.396+0000 7f90f4e00640 1 -- start start 2024-04-26T23:07:38.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.397+0000 7f90f4e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90d40051c0 0x7f90f01f61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:38.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.397+0000 7f90f4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f015c770 0x7f90f00b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:38.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.397+0000 7f90f4e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90f01f7950 0x7f90f00cf850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:38.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.397+0000 7f90f4e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f90f015d570 con 0x7f90f015c770 2024-04-26T23:07:38.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.397+0000 7f90f4e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f90f015d3f0 con 0x7f90d40051c0 2024-04-26T23:07:38.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.397+0000 7f90f4e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f90f006f8e0 con 0x7f90f01f7950 2024-04-26T23:07:38.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.397+0000 7f90e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f015c770 0x7f90f00b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:38.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.397+0000 7f90e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f015c770 0x7f90f00b1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53412/0 (socket says 172.21.15.40:53412) 2024-04-26T23:07:38.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.397+0000 7f90e7e00640 1 -- 172.21.15.40:0/3241072549 learned_addr learned my addr 172.21.15.40:0/3241072549 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:38.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.397+0000 7f90ede00640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90d40051c0 0x7f90f01f61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:38.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.397+0000 7f90ee800640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90f01f7950 0x7f90f00cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:38.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.397+0000 7f90e7e00640 1 -- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90f01f7950 msgr2=0x7f90f00cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.398+0000 7f90e7e00640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90f01f7950 0x7f90f00cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.398+0000 7f90e7e00640 1 -- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90d40051c0 msgr2=0x7f90f01f61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.398+0000 7f90e7e00640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90d40051c0 0x7f90f01f61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.398+0000 7f90e7e00640 1 -- 172.21.15.40:0/3241072549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f01f68b0 con 0x7f90f015c770 2024-04-26T23:07:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.398+0000 7f90ee800640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90f01f7950 0x7f90f00cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.398+0000 7f90ede00640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90d40051c0 0x7f90f01f61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.398+0000 7f90e7e00640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f015c770 0x7f90f00b1aa0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f90d800e4d0 tx=0x7f90d800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.398+0000 7f90e6000640 1 -- 172.21.15.40:0/3241072549 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d80052e0 con 0x7f90f015c770 2024-04-26T23:07:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.398+0000 7f90e6000640 1 -- 172.21.15.40:0/3241072549 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90d8018070 con 0x7f90f015c770 2024-04-26T23:07:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.398+0000 7f90e6000640 1 -- 172.21.15.40:0/3241072549 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d8003810 con 0x7f90f015c770 2024-04-26T23:07:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.398+0000 7f90f4e00640 1 -- 172.21.15.40:0/3241072549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90f01f6b90 con 0x7f90f015c770 2024-04-26T23:07:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.398+0000 7f90f4e00640 1 -- 172.21.15.40:0/3241072549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90f01f7150 con 0x7f90f015c770 2024-04-26T23:07:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.400+0000 7f90f4e00640 1 -- 172.21.15.40:0/3241072549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90f00694f0 con 0x7f90f015c770 2024-04-26T23:07:38.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.401+0000 7f90e6000640 1 -- 172.21.15.40:0/3241072549 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f90d80039b0 con 0x7f90f015c770 2024-04-26T23:07:38.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.402+0000 7f90e6000640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f90cc06c8f0 0x7f90cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:38.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.402+0000 7f90e6000640 1 -- 172.21.15.40:0/3241072549 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5507+0+0 (secure 0 0 0) 0x7f90d808b810 con 0x7f90f015c770 2024-04-26T23:07:38.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.403+0000 7f90ede00640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f90cc06c8f0 0x7f90cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:38.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.404+0000 7f90ede00640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f90cc06c8f0 0x7f90cc06edb0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009980 tx=0x7f90e0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:38.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.407+0000 7f90e6000640 1 -- 172.21.15.40:0/3241072549 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f90d8059c60 con 0x7f90f015c770 2024-04-26T23:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:38.686+0000 7f90f4e00640 1 -- 172.21.15.40:0/3241072549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_non_mds0"]} v 0) -- 0x7f90f015f5f0 con 0x7f90f015c770 2024-04-26T23:07:39.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.699+0000 7f90e6000640 1 -- 172.21.15.40:0/3241072549 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_non_mds0"]}]=0 v348) ==== 141+0+0 (secure 0 0 0) 0x7f90d8059560 con 0x7f90f015c770 2024-04-26T23:07:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.703+0000 7f90f4e00640 1 -- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f90cc06c8f0 msgr2=0x7f90cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.703+0000 7f90f4e00640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f90cc06c8f0 0x7f90cc06edb0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009980 tx=0x7f90e0008040 comp rx=0 tx=0).stop 2024-04-26T23:07:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.703+0000 7f90f4e00640 1 -- 172.21.15.40:0/3241072549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f015c770 msgr2=0x7f90f00b1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:39.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.703+0000 7f90f4e00640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f015c770 0x7f90f00b1aa0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f90d800e4d0 tx=0x7f90d800e990 comp rx=0 tx=0).stop 2024-04-26T23:07:39.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.703+0000 7f90f4e00640 1 -- 172.21.15.40:0/3241072549 shutdown_connections 2024-04-26T23:07:39.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.703+0000 7f90f4e00640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f90cc06c8f0 0x7f90cc06edb0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:39.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.703+0000 7f90f4e00640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90f01f7950 0x7f90f00cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:39.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.703+0000 7f90f4e00640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f015c770 0x7f90f00b1aa0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:39.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.703+0000 7f90f4e00640 1 --2- 172.21.15.40:0/3241072549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90d40051c0 0x7f90f01f61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:39.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.703+0000 7f90f4e00640 1 -- 172.21.15.40:0/3241072549 >> 172.21.15.40:0/3241072549 conn(0x7f90f00d0be0 msgr2=0x7f90f00ce7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:39.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.704+0000 7f90f4e00640 1 -- 172.21.15.40:0/3241072549 shutdown_connections 2024-04-26T23:07:39.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.704+0000 7f90f4e00640 1 -- 172.21.15.40:0/3241072549 wait complete. 2024-04-26T23:07:39.729 INFO:tasks.cephfs_test_runner:test_scrub_non_mds0 (tasks.cephfs.test_multimds_misc.TestScrub2) ... ok 2024-04-26T23:07:39.729 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_pause_and_resume_mds0 (tasks.cephfs.test_multimds_misc.TestScrub2) 2024-04-26T23:07:39.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:07:39.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.947+0000 7f32adc00640 1 Processor -- start 2024-04-26T23:07:39.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.947+0000 7f32adc00640 1 -- start start 2024-04-26T23:07:39.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.949+0000 7f32adc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f32a815ce60 0x7f32a80afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:39.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.949+0000 7f32adc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32a80af7a0 0x7f32a80ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.949+0000 7f32adc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32a80d0820 con 0x7f32a8071fe0 2024-04-26T23:07:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.949+0000 7f32adc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f32a80cc450 con 0x7f32a815ce60 2024-04-26T23:07:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.949+0000 7f32adc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f32a81614c0 con 0x7f32a80af7a0 2024-04-26T23:07:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.950+0000 7f32ad200640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f32a8071fe0 0x7f32a8073760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:55506/0 (socket says 172.21.15.40:55506) 2024-04-26T23:07:39.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.950+0000 7f32a7400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32a80af7a0 0x7f32a80ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.950+0000 7f32ad200640 1 -- 172.21.15.40:0/1553920546 learned_addr learned my addr 172.21.15.40:0/1553920546 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.950+0000 7f32a7e00640 1 --2- 172.21.15.40:0/1553920546 >> v2:172.21.15.103:3300/0 conn(0x7f32a815ce60 0x7f32a80afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.951+0000 7f32a7400640 1 -- 172.21.15.40:0/1553920546 >> v2:172.21.15.103:3300/0 conn(0x7f32a815ce60 msgr2=0x7f32a80afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.951+0000 7f32a7400640 1 --2- 172.21.15.40:0/1553920546 >> v2:172.21.15.103:3300/0 conn(0x7f32a815ce60 0x7f32a80afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:39.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.951+0000 7f32a7400640 1 -- 172.21.15.40:0/1553920546 >> v1:172.21.15.103:6789/0 conn(0x7f32a8071fe0 legacy=0x7f32a8073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.951+0000 7f32a7400640 1 -- 172.21.15.40:0/1553920546 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a8161760 con 0x7f32a80af7a0 2024-04-26T23:07:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.951+0000 7f32a6a00640 1 -- 172.21.15.40:0/1553920546 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 458241741 0 0) 0x7f32a80d0820 con 0x7f32a8071fe0 2024-04-26T23:07:39.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.952+0000 7f32a7400640 1 --2- 172.21.15.40:0/1553920546 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32a80af7a0 0x7f32a80ceff0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f32940097a0 tx=0x7f329402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dadcc6843d6aa0ca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:39.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.953+0000 7f32a6a00640 1 -- 172.21.15.40:0/1553920546 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3294037070 con 0x7f32a80af7a0 2024-04-26T23:07:39.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.953+0000 7f32a6a00640 1 -- 172.21.15.40:0/1553920546 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f329402ae50 con 0x7f32a80af7a0 2024-04-26T23:07:39.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.953+0000 7f32a6a00640 1 -- 172.21.15.40:0/1553920546 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3294032660 con 0x7f32a80af7a0 2024-04-26T23:07:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.953+0000 7f32adc00640 1 -- 172.21.15.40:0/1553920546 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32a80af7a0 msgr2=0x7f32a80ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.953+0000 7f32adc00640 1 --2- 172.21.15.40:0/1553920546 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32a80af7a0 0x7f32a80ceff0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f32940097a0 tx=0x7f329402a100 comp rx=0 tx=0).stop 2024-04-26T23:07:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.953+0000 7f32adc00640 1 -- 172.21.15.40:0/1553920546 shutdown_connections 2024-04-26T23:07:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.953+0000 7f32adc00640 1 --2- 172.21.15.40:0/1553920546 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32a80af7a0 0x7f32a80ceff0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.953+0000 7f32adc00640 1 --2- 172.21.15.40:0/1553920546 >> v2:172.21.15.103:3300/0 conn(0x7f32a815ce60 0x7f32a80afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:39.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.953+0000 7f32adc00640 1 -- 172.21.15.40:0/1553920546 >> 172.21.15.40:0/1553920546 conn(0x7f32a80d0a10 msgr2=0x7f32a80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:39.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.954+0000 7f32adc00640 1 -- 172.21.15.40:0/1553920546 shutdown_connections 2024-04-26T23:07:39.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.954+0000 7f32adc00640 1 -- 172.21.15.40:0/1553920546 wait complete. 2024-04-26T23:07:39.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.955+0000 7f32adc00640 1 Processor -- start 2024-04-26T23:07:39.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.956+0000 7f32adc00640 1 -- start start 2024-04-26T23:07:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.956+0000 7f32adc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32a8071fe0 0x7f32a815b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.956+0000 7f32adc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32a815ce60 0x7f32a80c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:39.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32adc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32a815b910 0x7f32a815bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32adc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32a8073e70 con 0x7f32a8071fe0 2024-04-26T23:07:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32adc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32a80cf530 con 0x7f32a815b910 2024-04-26T23:07:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32adc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f32a8073ff0 con 0x7f32a815ce60 2024-04-26T23:07:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32a8071fe0 0x7f32a815b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32a8071fe0 0x7f32a815b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53414/0 (socket says 172.21.15.40:53414) 2024-04-26T23:07:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32a7e00640 1 -- 172.21.15.40:0/2823084824 learned_addr learned my addr 172.21.15.40:0/2823084824 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32a7400640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32a815ce60 0x7f32a80c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32ad200640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32a815b910 0x7f32a815bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32a7e00640 1 -- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32a815ce60 msgr2=0x7f32a80c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32a7e00640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32a815ce60 0x7f32a80c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32a7e00640 1 -- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32a815b910 msgr2=0x7f32a815bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32a7e00640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32a815b910 0x7f32a815bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:39.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32a7e00640 1 -- 172.21.15.40:0/2823084824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a815a450 con 0x7f32a8071fe0 2024-04-26T23:07:39.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.957+0000 7f32a7400640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32a815ce60 0x7f32a80c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:39.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.958+0000 7f32a7e00640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32a8071fe0 0x7f32a815b1f0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f329800e870 tx=0x7f329800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.958+0000 7f329fe00640 1 -- 172.21.15.40:0/2823084824 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3298017070 con 0x7f32a8071fe0 2024-04-26T23:07:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.958+0000 7f329fe00640 1 -- 172.21.15.40:0/2823084824 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f329800c940 con 0x7f32a8071fe0 2024-04-26T23:07:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.958+0000 7f329fe00640 1 -- 172.21.15.40:0/2823084824 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329800cd30 con 0x7f32a8071fe0 2024-04-26T23:07:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.958+0000 7f32adc00640 1 -- 172.21.15.40:0/2823084824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32a81f9610 con 0x7f32a8071fe0 2024-04-26T23:07:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.958+0000 7f32adc00640 1 -- 172.21.15.40:0/2823084824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32a815a9f0 con 0x7f32a8071fe0 2024-04-26T23:07:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.959+0000 7f32adc00640 1 -- 172.21.15.40:0/2823084824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32a815ae20 con 0x7f32a8071fe0 2024-04-26T23:07:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.961+0000 7f329fe00640 1 -- 172.21.15.40:0/2823084824 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f329801a070 con 0x7f32a8071fe0 2024-04-26T23:07:39.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.961+0000 7f329fe00640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f328806c8f0 0x7f328806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:39.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.962+0000 7f329fe00640 1 -- 172.21.15.40:0/2823084824 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5507+0+0 (secure 0 0 0) 0x7f329808ba00 con 0x7f32a8071fe0 2024-04-26T23:07:39.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.962+0000 7f32a7400640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f328806c8f0 0x7f328806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:39.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.963+0000 7f32a7400640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f328806c8f0 0x7f328806edb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f3294034000 tx=0x7f329402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:39.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:39.966+0000 7f329fe00640 1 -- 172.21.15.40:0/2823084824 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3298059e50 con 0x7f32a8071fe0 2024-04-26T23:07:40.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.173+0000 7f32adc00640 1 -- 172.21.15.40:0/2823084824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32a80d01f0 con 0x7f32a8071fe0 2024-04-26T23:07:40.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.175+0000 7f329fe00640 1 -- 172.21.15.40:0/2823084824 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+13518 (secure 0 0 0) 0x7f3298059750 con 0x7f32a8071fe0 2024-04-26T23:07:40.175 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:40.175 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:07:37.675172+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6834/216893598":"2024-04-27T23:07:34.419695+0000","172.21.15.40:6832/216893598":"2024-04-27T23:07:34.419695+0000","172.21.15.40:6835/4227275373":"2024-04-27T23:07:34.419717+0000","172.21.15.40:6833/4227275373":"2024-04-27T23:07:34.419717+0000","172.21.15.103:6836/2914603370":"2024-04-27T23:07:34.419659+0000","172.21.15.103:6834/2914603370":"2024-04-27T23:07:34.419659+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:07:40.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.178+0000 7f32adc00640 1 -- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f328806c8f0 msgr2=0x7f328806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:40.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.178+0000 7f32adc00640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f328806c8f0 0x7f328806edb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f3294034000 tx=0x7f329402e000 comp rx=0 tx=0).stop 2024-04-26T23:07:40.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.178+0000 7f32adc00640 1 -- 172.21.15.40:0/2823084824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32a8071fe0 msgr2=0x7f32a815b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:40.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.178+0000 7f32adc00640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32a8071fe0 0x7f32a815b1f0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f329800e870 tx=0x7f329800ed30 comp rx=0 tx=0).stop 2024-04-26T23:07:40.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.178+0000 7f32adc00640 1 -- 172.21.15.40:0/2823084824 shutdown_connections 2024-04-26T23:07:40.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.178+0000 7f32adc00640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f328806c8f0 0x7f328806edb0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.178+0000 7f32adc00640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32a815b910 0x7f32a815bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.178+0000 7f32adc00640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32a815ce60 0x7f32a80c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.178+0000 7f32adc00640 1 --2- 172.21.15.40:0/2823084824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32a8071fe0 0x7f32a815b1f0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.178+0000 7f32adc00640 1 -- 172.21.15.40:0/2823084824 >> 172.21.15.40:0/2823084824 conn(0x7f32a80d0a10 msgr2=0x7f32a8155480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:40.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.178+0000 7f32adc00640 1 -- 172.21.15.40:0/2823084824 shutdown_connections 2024-04-26T23:07:40.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.178+0000 7f32adc00640 1 -- 172.21.15.40:0/2823084824 wait complete. 2024-04-26T23:07:40.197 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:07:40.197 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:07:40.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.415+0000 7f7319e00640 1 Processor -- start 2024-04-26T23:07:40.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.415+0000 7f7319e00640 1 -- start start 2024-04-26T23:07:40.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.415+0000 7f7319e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f73140738d0 0x7f7314073cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:40.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.415+0000 7f7319e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f73140ae150 0x7f73140b4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:40.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.415+0000 7f7319e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73140741f0 con 0x7f73140b2140 2024-04-26T23:07:40.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.415+0000 7f7319e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f73140b16f0 con 0x7f73140738d0 2024-04-26T23:07:40.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.415+0000 7f7319e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f73140b1cb0 con 0x7f73140ae150 2024-04-26T23:07:40.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.416+0000 7f7318a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f73140738d0 0x7f7314073cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:40.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.416+0000 7f7318a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f73140738d0 0x7f7314073cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53418/0 (socket says 172.21.15.40:53418) 2024-04-26T23:07:40.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.416+0000 7f7318a00640 1 -- 172.21.15.40:0/1079762503 learned_addr learned my addr 172.21.15.40:0/1079762503 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:40.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.416+0000 7f7313e00640 1 --2- 172.21.15.40:0/1079762503 >> v2:172.21.15.103:3300/0 conn(0x7f73140ae150 0x7f73140b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:40.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.416+0000 7f7318a00640 1 -- 172.21.15.40:0/1079762503 >> v2:172.21.15.103:3300/0 conn(0x7f73140ae150 msgr2=0x7f73140b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:40.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.416+0000 7f7318a00640 1 --2- 172.21.15.40:0/1079762503 >> v2:172.21.15.103:3300/0 conn(0x7f73140ae150 0x7f73140b4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.416+0000 7f7318a00640 1 -- 172.21.15.40:0/1079762503 >> v1:172.21.15.103:6789/0 conn(0x7f73140b2140 legacy=0x7f73140b25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:40.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.416+0000 7f7318a00640 1 -- 172.21.15.40:0/1079762503 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73140b5fc0 con 0x7f73140738d0 2024-04-26T23:07:40.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.417+0000 7f7318a00640 1 --2- 172.21.15.40:0/1079762503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73140738d0 0x7f7314073cb0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f73140720a0 tx=0x7f730802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efcb312a4ae8bdf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:40.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.418+0000 7f7313400640 1 -- 172.21.15.40:0/1079762503 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7308036070 con 0x7f73140738d0 2024-04-26T23:07:40.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.418+0000 7f7313400640 1 -- 172.21.15.40:0/1079762503 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7308031600 con 0x7f73140738d0 2024-04-26T23:07:40.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.418+0000 7f7313400640 1 -- 172.21.15.40:0/1079762503 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73080319d0 con 0x7f73140738d0 2024-04-26T23:07:40.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.418+0000 7f7319e00640 1 -- 172.21.15.40:0/1079762503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73140738d0 msgr2=0x7f7314073cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:40.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.418+0000 7f7319e00640 1 --2- 172.21.15.40:0/1079762503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73140738d0 0x7f7314073cb0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f73140720a0 tx=0x7f730802a6f0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.418+0000 7f7319e00640 1 -- 172.21.15.40:0/1079762503 shutdown_connections 2024-04-26T23:07:40.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.418+0000 7f7319e00640 1 --2- 172.21.15.40:0/1079762503 >> v2:172.21.15.103:3300/0 conn(0x7f73140ae150 0x7f73140b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.418+0000 7f7319e00640 1 --2- 172.21.15.40:0/1079762503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73140738d0 0x7f7314073cb0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.418+0000 7f7319e00640 1 -- 172.21.15.40:0/1079762503 >> 172.21.15.40:0/1079762503 conn(0x7f73140b2e70 msgr2=0x7f73140b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:40.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.418+0000 7f7319e00640 1 -- 172.21.15.40:0/1079762503 shutdown_connections 2024-04-26T23:07:40.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.419+0000 7f7319e00640 1 -- 172.21.15.40:0/1079762503 wait complete. 2024-04-26T23:07:40.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.420+0000 7f7319e00640 1 Processor -- start 2024-04-26T23:07:40.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.420+0000 7f7319e00640 1 -- start start 2024-04-26T23:07:40.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.421+0000 7f7319e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73140738d0 0x7f731409dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:40.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.421+0000 7f7319e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73140ae150 0x7f731409e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:40.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.421+0000 7f7319e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73140b2140 0x7f73141f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:40.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.421+0000 7f7319e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f73140b46b0 con 0x7f73140ae150 2024-04-26T23:07:40.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.421+0000 7f7319e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f73140b1940 con 0x7f73140738d0 2024-04-26T23:07:40.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.421+0000 7f7319e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f73140b3b20 con 0x7f73140b2140 2024-04-26T23:07:40.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.422+0000 7f7313e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73140ae150 0x7f731409e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:40.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.422+0000 7f7313e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73140ae150 0x7f731409e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53422/0 (socket says 172.21.15.40:53422) 2024-04-26T23:07:40.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.422+0000 7f7313e00640 1 -- 172.21.15.40:0/1544093588 learned_addr learned my addr 172.21.15.40:0/1544093588 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:40.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.422+0000 7f7318a00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73140738d0 0x7f731409dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:40.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.422+0000 7f7319400640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73140b2140 0x7f73141f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:40.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.422+0000 7f7313e00640 1 -- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73140b2140 msgr2=0x7f73141f4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:40.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.422+0000 7f7313e00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73140b2140 0x7f73141f4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.422+0000 7f7313e00640 1 -- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73140738d0 msgr2=0x7f731409dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:40.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.422+0000 7f7313e00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73140738d0 0x7f731409dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.422+0000 7f7313e00640 1 -- 172.21.15.40:0/1544093588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f731415d4c0 con 0x7f73140ae150 2024-04-26T23:07:40.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.422+0000 7f7319400640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73140b2140 0x7f73141f4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:40.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.422+0000 7f7318a00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73140738d0 0x7f731409dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:07:40.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.423+0000 7f7313e00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73140ae150 0x7f731409e280 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f730000d7d0 tx=0x7f730000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:40.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.423+0000 7f7310c00640 1 -- 172.21.15.40:0/1544093588 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7300004020 con 0x7f73140ae150 2024-04-26T23:07:40.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.423+0000 7f7310c00640 1 -- 172.21.15.40:0/1544093588 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73000041e0 con 0x7f73140ae150 2024-04-26T23:07:40.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.423+0000 7f7310c00640 1 -- 172.21.15.40:0/1544093588 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73000043d0 con 0x7f73140ae150 2024-04-26T23:07:40.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.423+0000 7f7319e00640 1 -- 172.21.15.40:0/1544093588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f731415d7a0 con 0x7f73140ae150 2024-04-26T23:07:40.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.423+0000 7f7319e00640 1 -- 172.21.15.40:0/1544093588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73141f4770 con 0x7f73140ae150 2024-04-26T23:07:40.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.424+0000 7f7319e00640 1 -- 172.21.15.40:0/1544093588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7314069a80 con 0x7f73140ae150 2024-04-26T23:07:40.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.425+0000 7f7310c00640 1 -- 172.21.15.40:0/1544093588 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f730001b020 con 0x7f73140ae150 2024-04-26T23:07:40.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.426+0000 7f7310c00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f72f406c8f0 0x7f72f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:40.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.427+0000 7f7310c00640 1 -- 172.21.15.40:0/1544093588 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5507+0+0 (secure 0 0 0) 0x7f730008b990 con 0x7f73140ae150 2024-04-26T23:07:40.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.427+0000 7f7318a00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f72f406c8f0 0x7f72f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:40.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.428+0000 7f7318a00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f72f406c8f0 0x7f72f406edb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7308033000 tx=0x7f7308009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:40.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.432+0000 7f7310c00640 1 -- 172.21.15.40:0/1544093588 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7300059de0 con 0x7f73140ae150 2024-04-26T23:07:40.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.638+0000 7f7319e00640 1 -- 172.21.15.40:0/1544093588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f73140b2970 con 0x7f73140ae150 2024-04-26T23:07:40.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.639+0000 7f7310c00640 1 -- 172.21.15.40:0/1544093588 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f73000596e0 con 0x7f73140ae150 2024-04-26T23:07:40.640 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:07:40.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.641+0000 7f7319e00640 1 -- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f72f406c8f0 msgr2=0x7f72f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:40.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.641+0000 7f7319e00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f72f406c8f0 0x7f72f406edb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7308033000 tx=0x7f7308009000 comp rx=0 tx=0).stop 2024-04-26T23:07:40.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.641+0000 7f7319e00640 1 -- 172.21.15.40:0/1544093588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73140ae150 msgr2=0x7f731409e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:40.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.641+0000 7f7319e00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73140ae150 0x7f731409e280 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f730000d7d0 tx=0x7f730000dc90 comp rx=0 tx=0).stop 2024-04-26T23:07:40.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.642+0000 7f7319e00640 1 -- 172.21.15.40:0/1544093588 shutdown_connections 2024-04-26T23:07:40.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.642+0000 7f7319e00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f72f406c8f0 0x7f72f406edb0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.642+0000 7f7319e00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73140b2140 0x7f73141f4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.642+0000 7f7319e00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73140ae150 0x7f731409e280 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.642+0000 7f7319e00640 1 --2- 172.21.15.40:0/1544093588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73140738d0 0x7f731409dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.642+0000 7f7319e00640 1 -- 172.21.15.40:0/1544093588 >> 172.21.15.40:0/1544093588 conn(0x7f73140b2e70 msgr2=0x7f731415a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:40.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.642+0000 7f7319e00640 1 -- 172.21.15.40:0/1544093588 shutdown_connections 2024-04-26T23:07:40.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.642+0000 7f7319e00640 1 -- 172.21.15.40:0/1544093588 wait complete. 2024-04-26T23:07:40.660 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_pause_and_resume_mds0' 2024-04-26T23:07:40.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.871+0000 7fb6e1200640 1 Processor -- start 2024-04-26T23:07:40.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.872+0000 7fb6e1200640 1 -- start start 2024-04-26T23:07:40.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.873+0000 7fb6e1200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb6dc072950 0x7fb6dc167ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:40.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.873+0000 7fb6e1200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6dc15d7c0 con 0x7fb6dc0cc480 2024-04-26T23:07:40.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.873+0000 7fb6e1200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6dc15daa0 con 0x7fb6dc071e60 2024-04-26T23:07:40.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.873+0000 7fb6e1200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb6dc15dd80 con 0x7fb6dc072950 2024-04-26T23:07:40.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.873+0000 7fb6d9400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb6dc0cc480 0x7fb6dc072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50016/0 (socket says 172.21.15.40:50016) 2024-04-26T23:07:40.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.873+0000 7fb6d9400640 1 -- 172.21.15.40:0/2913928501 learned_addr learned my addr 172.21.15.40:0/2913928501 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:40.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.873+0000 7fb6da800640 1 --2- 172.21.15.40:0/2913928501 >> v2:172.21.15.40:3300/0 conn(0x7fb6dc072950 0x7fb6dc167ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.873+0000 7fb6da800640 1 -- 172.21.15.40:0/2913928501 >> v1:172.21.15.103:6789/0 conn(0x7fb6dc071e60 legacy=0x7fb6dc06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.873+0000 7fb6d8a00640 1 -- 172.21.15.40:0/2913928501 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1979373823 0 0) 0x7fb6dc15d7c0 con 0x7fb6dc0cc480 2024-04-26T23:07:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.873+0000 7fb6da800640 1 -- 172.21.15.40:0/2913928501 >> v1:172.21.15.40:6789/0 conn(0x7fb6dc0cc480 legacy=0x7fb6dc072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.873+0000 7fb6da800640 1 -- 172.21.15.40:0/2913928501 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6dc15e060 con 0x7fb6dc072950 2024-04-26T23:07:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.874+0000 7fb6da800640 1 --2- 172.21.15.40:0/2913928501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6dc072950 0x7fb6dc167ae0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4009d90 tx=0x7fb6d402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c8c2a073a2c707 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.875+0000 7fb6d8a00640 1 -- 172.21.15.40:0/2913928501 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4007740 con 0x7fb6dc072950 2024-04-26T23:07:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.875+0000 7fb6d8a00640 1 -- 172.21.15.40:0/2913928501 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d40078e0 con 0x7fb6dc072950 2024-04-26T23:07:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.875+0000 7fb6d8a00640 1 -- 172.21.15.40:0/2913928501 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4007cb0 con 0x7fb6dc072950 2024-04-26T23:07:40.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.875+0000 7fb6e1200640 1 -- 172.21.15.40:0/2913928501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6dc072950 msgr2=0x7fb6dc167ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:40.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.875+0000 7fb6e1200640 1 --2- 172.21.15.40:0/2913928501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6dc072950 0x7fb6dc167ae0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4009d90 tx=0x7fb6d402a6f0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.875+0000 7fb6e1200640 1 -- 172.21.15.40:0/2913928501 shutdown_connections 2024-04-26T23:07:40.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.875+0000 7fb6e1200640 1 --2- 172.21.15.40:0/2913928501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6dc072950 0x7fb6dc167ae0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.875+0000 7fb6e1200640 1 -- 172.21.15.40:0/2913928501 >> 172.21.15.40:0/2913928501 conn(0x7fb6dc0d0be0 msgr2=0x7fb6dc0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:40.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.877+0000 7fb6e1200640 1 -- 172.21.15.40:0/2913928501 shutdown_connections 2024-04-26T23:07:40.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.877+0000 7fb6e1200640 1 -- 172.21.15.40:0/2913928501 wait complete. 2024-04-26T23:07:40.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.877+0000 7fb6e1200640 1 Processor -- start 2024-04-26T23:07:40.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.878+0000 7fb6e1200640 1 -- start start 2024-04-26T23:07:40.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.878+0000 7fb6e1200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6dc071e60 0x7fb6dc14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.878+0000 7fb6e1200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6dc072950 0x7fb6dc0c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6e1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6dc0cc480 0x7fb6dc0ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6e1200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb6dc15d570 con 0x7fb6dc0cc480 2024-04-26T23:07:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6e1200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb6dc15d3f0 con 0x7fb6dc072950 2024-04-26T23:07:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6e1200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6dc06fff0 con 0x7fb6dc071e60 2024-04-26T23:07:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6da800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6dc0cc480 0x7fb6dc0ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6da800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6dc0cc480 0x7fb6dc0ca3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53454/0 (socket says 172.21.15.40:53454) 2024-04-26T23:07:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6da800640 1 -- 172.21.15.40:0/586092731 learned_addr learned my addr 172.21.15.40:0/586092731 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6d9e00640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6dc071e60 0x7fb6dc14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6d9400640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6dc072950 0x7fb6dc0c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6da800640 1 -- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6dc071e60 msgr2=0x7fb6dc14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6da800640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6dc071e60 0x7fb6dc14efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6da800640 1 -- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6dc072950 msgr2=0x7fb6dc0c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:40.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6da800640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6dc072950 0x7fb6dc0c9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:40.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6da800640 1 -- 172.21.15.40:0/586092731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6dc1f5160 con 0x7fb6dc0cc480 2024-04-26T23:07:40.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6da800640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6dc0cc480 0x7fb6dc0ca3c0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4009d90 tx=0x7fb6d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:40.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6d2000640 1 -- 172.21.15.40:0/586092731 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4040070 con 0x7fb6dc0cc480 2024-04-26T23:07:40.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6d2000640 1 -- 172.21.15.40:0/586092731 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d4031c90 con 0x7fb6dc0cc480 2024-04-26T23:07:40.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6d2000640 1 -- 172.21.15.40:0/586092731 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d403c640 con 0x7fb6dc0cc480 2024-04-26T23:07:40.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6e1200640 1 -- 172.21.15.40:0/586092731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6dc1f5440 con 0x7fb6dc0cc480 2024-04-26T23:07:40.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.880+0000 7fb6e1200640 1 -- 172.21.15.40:0/586092731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6dc1f5a00 con 0x7fb6dc0cc480 2024-04-26T23:07:40.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.881+0000 7fb6e1200640 1 -- 172.21.15.40:0/586092731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6dc0694f0 con 0x7fb6dc0cc480 2024-04-26T23:07:40.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.882+0000 7fb6d2000640 1 -- 172.21.15.40:0/586092731 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb6d4039040 con 0x7fb6dc0cc480 2024-04-26T23:07:40.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.883+0000 7fb6d2000640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb6bc06c8f0 0x7fb6bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:40.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.883+0000 7fb6d2000640 1 -- 172.21.15.40:0/586092731 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5507+0+0 (secure 0 0 0) 0x7fb6d40ad1c0 con 0x7fb6dc0cc480 2024-04-26T23:07:40.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.884+0000 7fb6d9e00640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb6bc06c8f0 0x7fb6bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:40.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.886+0000 7fb6d9e00640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb6bc06c8f0 0x7fb6bc06edb0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc000f60 tx=0x7fb6cc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:40.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:40.889+0000 7fb6d2000640 1 -- 172.21.15.40:0/586092731 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fb6d407b6d0 con 0x7fb6dc0cc480 2024-04-26T23:07:41.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.169+0000 7fb6e1200640 1 -- 172.21.15.40:0/586092731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_pause_and_resume_mds0"]} v 0) -- 0x7fb6dc06e6f0 con 0x7fb6dc0cc480 2024-04-26T23:07:41.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.716+0000 7fb6d2000640 1 -- 172.21.15.40:0/586092731 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_pause_and_resume_mds0"]}]=0 v350) ==== 157+0+0 (secure 0 0 0) 0x7fb6d407afd0 con 0x7fb6dc0cc480 2024-04-26T23:07:41.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.719+0000 7fb6e1200640 1 -- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb6bc06c8f0 msgr2=0x7fb6bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:41.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.719+0000 7fb6e1200640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb6bc06c8f0 0x7fb6bc06edb0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc000f60 tx=0x7fb6cc009040 comp rx=0 tx=0).stop 2024-04-26T23:07:41.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.719+0000 7fb6e1200640 1 -- 172.21.15.40:0/586092731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6dc0cc480 msgr2=0x7fb6dc0ca3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:41.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.719+0000 7fb6e1200640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6dc0cc480 0x7fb6dc0ca3c0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4009d90 tx=0x7fb6d4004a40 comp rx=0 tx=0).stop 2024-04-26T23:07:41.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.720+0000 7fb6e1200640 1 -- 172.21.15.40:0/586092731 shutdown_connections 2024-04-26T23:07:41.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.720+0000 7fb6e1200640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb6bc06c8f0 0x7fb6bc06edb0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:41.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.720+0000 7fb6e1200640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6dc0cc480 0x7fb6dc0ca3c0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:41.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.720+0000 7fb6e1200640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6dc072950 0x7fb6dc0c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:41.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.720+0000 7fb6e1200640 1 --2- 172.21.15.40:0/586092731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6dc071e60 0x7fb6dc14efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:41.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.720+0000 7fb6e1200640 1 -- 172.21.15.40:0/586092731 >> 172.21.15.40:0/586092731 conn(0x7fb6dc0d0be0 msgr2=0x7fb6dc0ce7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:41.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.720+0000 7fb6e1200640 1 -- 172.21.15.40:0/586092731 shutdown_connections 2024-04-26T23:07:41.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.720+0000 7fb6e1200640 1 -- 172.21.15.40:0/586092731 wait complete. 2024-04-26T23:07:41.741 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-26T23:07:41.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.968+0000 7fc966200640 1 Processor -- start 2024-04-26T23:07:41.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.969+0000 7fc966200640 1 -- start start 2024-04-26T23:07:41.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.970+0000 7fc966200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc9600b10b0 0x7fc9600ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:41.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.970+0000 7fc966200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc96006c950 0x7fc96015e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:41.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.970+0000 7fc966200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9600cfd80 con 0x7fc9600b0380 2024-04-26T23:07:41.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.970+0000 7fc966200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc9600af8e0 con 0x7fc96006c950 2024-04-26T23:07:41.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.970+0000 7fc966200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc9600d0020 con 0x7fc9600b10b0 2024-04-26T23:07:41.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.970+0000 7fc95fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc96006c950 0x7fc96015e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:41.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.970+0000 7fc95fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc96006c950 0x7fc96015e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53462/0 (socket says 172.21.15.40:53462) 2024-04-26T23:07:41.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.970+0000 7fc95fe00640 1 -- 172.21.15.40:0/1014470474 learned_addr learned my addr 172.21.15.40:0/1014470474 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:41.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.971+0000 7fc95fe00640 1 -- 172.21.15.40:0/1014470474 >> v2:172.21.15.103:3300/0 conn(0x7fc9600b10b0 msgr2=0x7fc9600ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:41.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.971+0000 7fc95fe00640 1 --2- 172.21.15.40:0/1014470474 >> v2:172.21.15.103:3300/0 conn(0x7fc9600b10b0 0x7fc9600ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:41.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.971+0000 7fc95fe00640 1 -- 172.21.15.40:0/1014470474 >> v1:172.21.15.103:6789/0 conn(0x7fc9600b0380 legacy=0x7fc96006c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:41.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.971+0000 7fc95fe00640 1 -- 172.21.15.40:0/1014470474 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9600d0300 con 0x7fc96006c950 2024-04-26T23:07:41.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.972+0000 7fc95fe00640 1 --2- 172.21.15.40:0/1014470474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96006c950 0x7fc96015e970 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fc960072f00 tx=0x7fc95802a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=601af88494f94ea0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:41.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.972+0000 7fc95ea00640 1 -- 172.21.15.40:0/1014470474 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc958036070 con 0x7fc96006c950 2024-04-26T23:07:41.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.972+0000 7fc95ea00640 1 -- 172.21.15.40:0/1014470474 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc958031600 con 0x7fc96006c950 2024-04-26T23:07:41.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.972+0000 7fc95ea00640 1 -- 172.21.15.40:0/1014470474 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9580319d0 con 0x7fc96006c950 2024-04-26T23:07:41.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.973+0000 7fc966200640 1 -- 172.21.15.40:0/1014470474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96006c950 msgr2=0x7fc96015e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:41.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.973+0000 7fc966200640 1 --2- 172.21.15.40:0/1014470474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96006c950 0x7fc96015e970 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fc960072f00 tx=0x7fc95802a630 comp rx=0 tx=0).stop 2024-04-26T23:07:41.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.973+0000 7fc966200640 1 -- 172.21.15.40:0/1014470474 shutdown_connections 2024-04-26T23:07:41.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.973+0000 7fc966200640 1 --2- 172.21.15.40:0/1014470474 >> v2:172.21.15.103:3300/0 conn(0x7fc9600b10b0 0x7fc9600ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:41.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.973+0000 7fc966200640 1 --2- 172.21.15.40:0/1014470474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc96006c950 0x7fc96015e970 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:41.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.973+0000 7fc966200640 1 -- 172.21.15.40:0/1014470474 >> 172.21.15.40:0/1014470474 conn(0x7fc9600d1cf0 msgr2=0x7fc9600d20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:41.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.973+0000 7fc966200640 1 -- 172.21.15.40:0/1014470474 shutdown_connections 2024-04-26T23:07:41.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.973+0000 7fc966200640 1 -- 172.21.15.40:0/1014470474 wait complete. 2024-04-26T23:07:41.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.975+0000 7fc966200640 1 Processor -- start 2024-04-26T23:07:41.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.975+0000 7fc966200640 1 -- start start 2024-04-26T23:07:41.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.976+0000 7fc966200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc96006c950 0x7fc9600c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.976+0000 7fc966200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9600b0380 0x7fc9601620b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.976+0000 7fc966200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9600b10b0 0x7fc9601f4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.976+0000 7fc966200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc9600b0760 con 0x7fc9600b10b0 2024-04-26T23:07:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.976+0000 7fc966200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9600d1980 con 0x7fc9600b0380 2024-04-26T23:07:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.976+0000 7fc966200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc9600d15c0 con 0x7fc96006c950 2024-04-26T23:07:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.976+0000 7fc95fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9600b10b0 0x7fc9601f4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.976+0000 7fc95fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9600b10b0 0x7fc9601f4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53476/0 (socket says 172.21.15.40:53476) 2024-04-26T23:07:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.976+0000 7fc95fe00640 1 -- 172.21.15.40:0/1831604769 learned_addr learned my addr 172.21.15.40:0/1831604769 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.976+0000 7fc95fe00640 1 -- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc96006c950 msgr2=0x7fc9600c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.976+0000 7fc95fe00640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc96006c950 0x7fc9600c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.977+0000 7fc95fe00640 1 -- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9600b0380 msgr2=0x7fc9601620b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:41.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.977+0000 7fc957e00640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9600b0380 0x7fc9601620b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:41.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.977+0000 7fc95fe00640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9600b0380 0x7fc9601620b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:41.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.977+0000 7fc95fe00640 1 -- 172.21.15.40:0/1831604769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9600c7320 con 0x7fc9600b10b0 2024-04-26T23:07:41.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.977+0000 7fc957e00640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9600b0380 0x7fc9601620b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:41.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.977+0000 7fc95fe00640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9600b10b0 0x7fc9601f4d80 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fc958034040 tx=0x7fc958004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:41.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.977+0000 7fc957400640 1 -- 172.21.15.40:0/1831604769 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc958041020 con 0x7fc9600b10b0 2024-04-26T23:07:41.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.977+0000 7fc957400640 1 -- 172.21.15.40:0/1831604769 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc958007780 con 0x7fc9600b10b0 2024-04-26T23:07:41.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.977+0000 7fc957400640 1 -- 172.21.15.40:0/1831604769 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc958036040 con 0x7fc9600b10b0 2024-04-26T23:07:41.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.977+0000 7fc966200640 1 -- 172.21.15.40:0/1831604769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9600cb760 con 0x7fc9600b10b0 2024-04-26T23:07:41.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.978+0000 7fc966200640 1 -- 172.21.15.40:0/1831604769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9600c75c0 con 0x7fc9600b10b0 2024-04-26T23:07:41.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.979+0000 7fc966200640 1 -- 172.21.15.40:0/1831604769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9600694f0 con 0x7fc9600b10b0 2024-04-26T23:07:41.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.980+0000 7fc957400640 1 -- 172.21.15.40:0/1831604769 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc958044020 con 0x7fc9600b10b0 2024-04-26T23:07:41.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.981+0000 7fc957400640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc93c06c8f0 0x7fc93c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:41.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.981+0000 7fc957400640 1 -- 172.21.15.40:0/1831604769 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5507+0+0 (secure 0 0 0) 0x7fc9580ac8b0 con 0x7fc9600b10b0 2024-04-26T23:07:41.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.982+0000 7fc95f400640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc93c06c8f0 0x7fc93c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:41.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.983+0000 7fc95f400640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc93c06c8f0 0x7fc93c06edb0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc950003cc0 tx=0x7fc9500079b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:41.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:41.985+0000 7fc957400640 1 -- 172.21.15.40:0/1831604769 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc95804b030 con 0x7fc9600b10b0 2024-04-26T23:07:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.189+0000 7fc966200640 1 -- 172.21.15.40:0/1831604769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fc9600c8300 con 0x7fc9600b10b0 2024-04-26T23:07:42.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.195+0000 7fc957400640 1 -- 172.21.15.40:0/1831604769 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7fc95807ae50 con 0x7fc9600b10b0 2024-04-26T23:07:42.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.198+0000 7fc966200640 1 -- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc93c06c8f0 msgr2=0x7fc93c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:42.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.198+0000 7fc966200640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc93c06c8f0 0x7fc93c06edb0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc950003cc0 tx=0x7fc9500079b0 comp rx=0 tx=0).stop 2024-04-26T23:07:42.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.198+0000 7fc966200640 1 -- 172.21.15.40:0/1831604769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9600b10b0 msgr2=0x7fc9601f4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:42.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.198+0000 7fc966200640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9600b10b0 0x7fc9601f4d80 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fc958034040 tx=0x7fc958004a40 comp rx=0 tx=0).stop 2024-04-26T23:07:42.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.198+0000 7fc966200640 1 -- 172.21.15.40:0/1831604769 shutdown_connections 2024-04-26T23:07:42.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.198+0000 7fc966200640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc93c06c8f0 0x7fc93c06edb0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:42.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.198+0000 7fc966200640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9600b10b0 0x7fc9601f4d80 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:42.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.198+0000 7fc966200640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9600b0380 0x7fc9601620b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:42.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.198+0000 7fc966200640 1 --2- 172.21.15.40:0/1831604769 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc96006c950 0x7fc9600c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:42.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.198+0000 7fc966200640 1 -- 172.21.15.40:0/1831604769 >> 172.21.15.40:0/1831604769 conn(0x7fc9600d1cf0 msgr2=0x7fc9600b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:42.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.199+0000 7fc966200640 1 -- 172.21.15.40:0/1831604769 shutdown_connections 2024-04-26T23:07:42.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.199+0000 7fc966200640 1 -- 172.21.15.40:0/1831604769 wait complete. 2024-04-26T23:07:42.220 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-26T23:07:42.220 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:07:42.319 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:07:42.434 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-26T23:07:42.434 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:07:42.435 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:07:42.498 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:07:42.499 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:07:42.499 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:07:42.557 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:07:42.558 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:07:42.558 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:07:42.622 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:07:42.623 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:07:42.623 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:07:42.681 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:07:42.681 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-26T23:07:42.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:42.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.902+0000 7fc947e00640 1 Processor -- start 2024-04-26T23:07:42.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.902+0000 7fc947e00640 1 -- start start 2024-04-26T23:07:42.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.903+0000 7fc947e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9480b6220 0x7fc9480d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:42.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.904+0000 7fc947e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9480d08d0 0x7fc9480d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:42.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.904+0000 7fc947e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc948162c40 con 0x7fc9480d3820 2024-04-26T23:07:42.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.904+0000 7fc947e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc9480d2d80 con 0x7fc9480b6220 2024-04-26T23:07:42.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.904+0000 7fc947e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc9480b7c80 con 0x7fc9480d08d0 2024-04-26T23:07:42.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.904+0000 7fc946a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9480b6220 0x7fc9480d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:42.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.904+0000 7fc946a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9480b6220 0x7fc9480d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53482/0 (socket says 172.21.15.40:53482) 2024-04-26T23:07:42.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.904+0000 7fc946a00640 1 -- 172.21.15.40:0/2148578373 learned_addr learned my addr 172.21.15.40:0/2148578373 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:42.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.904+0000 7fc946a00640 1 -- 172.21.15.40:0/2148578373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9480d08d0 msgr2=0x7fc9480d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:42.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.904+0000 7fc946a00640 1 --2- 172.21.15.40:0/2148578373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9480d08d0 0x7fc9480d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:42.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.905+0000 7fc946a00640 1 -- 172.21.15.40:0/2148578373 >> v1:172.21.15.103:6789/0 conn(0x7fc9480d3820 legacy=0x7fc9480b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:42.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.905+0000 7fc946a00640 1 -- 172.21.15.40:0/2148578373 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9480b7f60 con 0x7fc9480b6220 2024-04-26T23:07:42.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.906+0000 7fc946a00640 1 --2- 172.21.15.40:0/2148578373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9480b6220 0x7fc9480d0390 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc938009870 tx=0x7fc93802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b33e453aad7bc4f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:42.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.906+0000 7fc945600640 1 -- 172.21.15.40:0/2148578373 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc938037070 con 0x7fc9480b6220 2024-04-26T23:07:42.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.906+0000 7fc945600640 1 -- 172.21.15.40:0/2148578373 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc938032440 con 0x7fc9480b6220 2024-04-26T23:07:42.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.906+0000 7fc945600640 1 -- 172.21.15.40:0/2148578373 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc938032810 con 0x7fc9480b6220 2024-04-26T23:07:42.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.907+0000 7fc947e00640 1 -- 172.21.15.40:0/2148578373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9480b6220 msgr2=0x7fc9480d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:42.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.907+0000 7fc947e00640 1 --2- 172.21.15.40:0/2148578373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9480b6220 0x7fc9480d0390 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc938009870 tx=0x7fc93802a200 comp rx=0 tx=0).stop 2024-04-26T23:07:42.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.907+0000 7fc947e00640 1 -- 172.21.15.40:0/2148578373 shutdown_connections 2024-04-26T23:07:42.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.907+0000 7fc947e00640 1 --2- 172.21.15.40:0/2148578373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9480d08d0 0x7fc9480d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:42.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.907+0000 7fc947e00640 1 --2- 172.21.15.40:0/2148578373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9480b6220 0x7fc9480d0390 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:42.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.907+0000 7fc947e00640 1 -- 172.21.15.40:0/2148578373 >> 172.21.15.40:0/2148578373 conn(0x7fc9480d4d90 msgr2=0x7fc9480d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:42.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.907+0000 7fc947e00640 1 -- 172.21.15.40:0/2148578373 shutdown_connections 2024-04-26T23:07:42.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.907+0000 7fc947e00640 1 -- 172.21.15.40:0/2148578373 wait complete. 2024-04-26T23:07:42.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.909+0000 7fc947e00640 1 Processor -- start 2024-04-26T23:07:42.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.909+0000 7fc947e00640 1 -- start start 2024-04-26T23:07:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.910+0000 7fc947e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9480b6220 0x7fc948153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.910+0000 7fc947e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9480d08d0 0x7fc9480c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.910+0000 7fc947e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9480d3820 0x7fc94815eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.910+0000 7fc947e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc9480b9d30 con 0x7fc9480d3820 2024-04-26T23:07:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.910+0000 7fc947e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9480b8520 con 0x7fc9480d08d0 2024-04-26T23:07:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.910+0000 7fc947e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc9480b9eb0 con 0x7fc9480b6220 2024-04-26T23:07:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.910+0000 7fc947400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9480d3820 0x7fc94815eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.910+0000 7fc947400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9480d3820 0x7fc94815eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53492/0 (socket says 172.21.15.40:53492) 2024-04-26T23:07:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.910+0000 7fc947400640 1 -- 172.21.15.40:0/2654057999 learned_addr learned my addr 172.21.15.40:0/2654057999 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.911+0000 7fc946a00640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9480b6220 0x7fc948153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.911+0000 7fc946000640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9480d08d0 0x7fc9480c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.911+0000 7fc947400640 1 -- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9480b6220 msgr2=0x7fc948153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:42.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.911+0000 7fc947400640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9480b6220 0x7fc948153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:42.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.911+0000 7fc947400640 1 -- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9480d08d0 msgr2=0x7fc9480c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:42.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.911+0000 7fc947400640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9480d08d0 0x7fc9480c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:42.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.911+0000 7fc947400640 1 -- 172.21.15.40:0/2654057999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9480ca310 con 0x7fc9480d3820 2024-04-26T23:07:42.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.911+0000 7fc947400640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9480d3820 0x7fc94815eb10 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fc94000b990 tx=0x7fc94000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:42.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.911+0000 7fc93ea00640 1 -- 172.21.15.40:0/2654057999 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940019070 con 0x7fc9480d3820 2024-04-26T23:07:42.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.911+0000 7fc93ea00640 1 -- 172.21.15.40:0/2654057999 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc94000cba0 con 0x7fc9480d3820 2024-04-26T23:07:42.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.911+0000 7fc947e00640 1 -- 172.21.15.40:0/2654057999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9480c8d30 con 0x7fc9480d3820 2024-04-26T23:07:42.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.911+0000 7fc93ea00640 1 -- 172.21.15.40:0/2654057999 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940014440 con 0x7fc9480d3820 2024-04-26T23:07:42.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.912+0000 7fc947e00640 1 -- 172.21.15.40:0/2654057999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9480c92f0 con 0x7fc9480d3820 2024-04-26T23:07:42.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.913+0000 7fc947e00640 1 -- 172.21.15.40:0/2654057999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc948069370 con 0x7fc9480d3820 2024-04-26T23:07:42.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.914+0000 7fc93ea00640 1 -- 172.21.15.40:0/2654057999 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc94001c070 con 0x7fc9480d3820 2024-04-26T23:07:42.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.915+0000 7fc93ea00640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc92406c8f0 0x7fc92406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:42.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.915+0000 7fc93ea00640 1 -- 172.21.15.40:0/2654057999 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5507+0+0 (secure 0 0 0) 0x7fc94008b8e0 con 0x7fc9480d3820 2024-04-26T23:07:42.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.915+0000 7fc946a00640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc92406c8f0 0x7fc92406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:42.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.916+0000 7fc946a00640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc92406c8f0 0x7fc92406edb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc938034000 tx=0x7fc93802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:42.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:42.919+0000 7fc93ea00640 1 -- 172.21.15.40:0/2654057999 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc940059d30 con 0x7fc9480d3820 2024-04-26T23:07:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.167+0000 7fc947e00640 1 -- 172.21.15.40:0/2654057999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9480d34f0 con 0x7fc9480d3820 2024-04-26T23:07:43.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.168+0000 7fc93ea00640 1 -- 172.21.15.40:0/2654057999 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+3872 (secure 0 0 0) 0x7fc940059630 con 0x7fc9480d3820 2024-04-26T23:07:43.168 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:43.168 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":84,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/755627718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":755627718},{"type":"v1","addr":"172.21.15.103:6837","nonce":755627718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6202,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/1910399388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1910399388},{"type":"v1","addr":"172.21.15.103:6836","nonce":1910399388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":7932,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3362256069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3362256069},{"type":"v1","addr":"172.21.15.40:6833","nonce":3362256069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":7941,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/4148036905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":4148036905},{"type":"v1","addr":"172.21.15.40:6835","nonce":4148036905}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[]} 2024-04-26T23:07:43.169 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 84 2024-04-26T23:07:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.170+0000 7fc947e00640 1 -- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc92406c8f0 msgr2=0x7fc92406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.170+0000 7fc947e00640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc92406c8f0 0x7fc92406edb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc938034000 tx=0x7fc93802e000 comp rx=0 tx=0).stop 2024-04-26T23:07:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.171+0000 7fc947e00640 1 -- 172.21.15.40:0/2654057999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9480d3820 msgr2=0x7fc94815eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.171+0000 7fc947e00640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9480d3820 0x7fc94815eb10 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fc94000b990 tx=0x7fc94000be50 comp rx=0 tx=0).stop 2024-04-26T23:07:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.171+0000 7fc947e00640 1 -- 172.21.15.40:0/2654057999 shutdown_connections 2024-04-26T23:07:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.171+0000 7fc947e00640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc92406c8f0 0x7fc92406edb0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.171+0000 7fc947e00640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9480d3820 0x7fc94815eb10 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.171+0000 7fc947e00640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9480d08d0 0x7fc9480c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.171+0000 7fc947e00640 1 --2- 172.21.15.40:0/2654057999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9480b6220 0x7fc948153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.171+0000 7fc947e00640 1 -- 172.21.15.40:0/2654057999 >> 172.21.15.40:0/2654057999 conn(0x7fc9480d4d90 msgr2=0x7fc9481623b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:43.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.171+0000 7fc947e00640 1 -- 172.21.15.40:0/2654057999 shutdown_connections 2024-04-26T23:07:43.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.171+0000 7fc947e00640 1 -- 172.21.15.40:0/2654057999 wait complete. 2024-04-26T23:07:43.192 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-26T23:07:43.192 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-26T23:07:43.193 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-26T23:07:43.193 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:07:43.193 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-26T23:07:43.193 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-26T23:07:43.193 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-26T23:07:43.193 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:07:43.193 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-26T23:07:43.194 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-26T23:07:43.194 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-26T23:07:43.194 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:07:43.194 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-26T23:07:43.194 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-26T23:07:43.194 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-26T23:07:43.194 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:07:43.209 INFO:tasks.ceph.mds.a:Stopped 2024-04-26T23:07:43.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-26T23:07:43.214 INFO:tasks.ceph.mds.d:Stopped 2024-04-26T23:07:43.214 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-26T23:07:43.215 INFO:tasks.ceph.mds.b:Stopped 2024-04-26T23:07:43.215 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-26T23:07:43.216 INFO:tasks.ceph.mds.c:Stopped 2024-04-26T23:07:43.216 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-26T23:07:43.265 INFO:tasks.ceph.mds.d:Started 2024-04-26T23:07:43.265 DEBUG:teuthology.parallel:result is None 2024-04-26T23:07:43.266 INFO:tasks.ceph.mds.a:Started 2024-04-26T23:07:43.267 DEBUG:teuthology.parallel:result is None 2024-04-26T23:07:43.273 INFO:tasks.ceph.mds.b:Started 2024-04-26T23:07:43.273 DEBUG:teuthology.parallel:result is None 2024-04-26T23:07:43.274 INFO:tasks.ceph.mds.c:Started 2024-04-26T23:07:43.274 DEBUG:teuthology.parallel:result is None 2024-04-26T23:07:43.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-26T23:07:43.418 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-26T23:07:43.419 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-04-26T23:07:43.421 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-26T23:07:43.425 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:07:43.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.463+0000 7f32b8400640 1 Processor -- start 2024-04-26T23:07:43.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.463+0000 7f32b8400640 1 -- start start 2024-04-26T23:07:43.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.465+0000 7f32b8400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b0072950 0x7f32b0167ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.465+0000 7f32b8400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32b015d7c0 con 0x7f32b00cc480 2024-04-26T23:07:43.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.465+0000 7f32b8400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32b015daa0 con 0x7f32b0071e60 2024-04-26T23:07:43.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.465+0000 7f32b8400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f32b015dd80 con 0x7f32b0072950 2024-04-26T23:07:43.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.465+0000 7f32b5000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f32b00cc480 0x7f32b0072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50032/0 (socket says 172.21.15.40:50032) 2024-04-26T23:07:43.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.465+0000 7f32b5000640 1 -- 172.21.15.40:0/386115421 learned_addr learned my addr 172.21.15.40:0/386115421 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:43.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.465+0000 7f32b6400640 1 --2- 172.21.15.40:0/386115421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b0072950 0x7f32b0167ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.465+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3797173360 0 0) 0x7f32b015d7c0 con 0x7f32b00cc480 2024-04-26T23:07:43.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.465+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3298003520 con 0x7f32b00cc480 2024-04-26T23:07:43.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 632737598 0 0) 0x7f32b015daa0 con 0x7f32b0071e60 2024-04-26T23:07:43.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32b015d7c0 con 0x7f32b0071e60 2024-04-26T23:07:43.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3483469981 0 0) 0x7f3298003520 con 0x7f32b00cc480 2024-04-26T23:07:43.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b0072950 msgr2=0x7f32b0167ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:43.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 --2- 172.21.15.40:0/386115421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b0072950 0x7f32b0167ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 >> v1:172.21.15.103:6789/0 conn(0x7f32b0071e60 legacy=0x7f32b006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:43.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b015e060 con 0x7f32b00cc480 2024-04-26T23:07:43.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f329c001670 con 0x7f32b00cc480 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 >> v1:172.21.15.40:6789/0 conn(0x7f32b00cc480 legacy=0x7f32b0072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 --2- 172.21.15.40:0/386115421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298003b70 0x7f3298003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 --2- 172.21.15.40:0/386115421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298004800 0x7f3298004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 --2- 172.21.15.40:0/386115421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b0072950 0x7f32980051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32b015dd80 con 0x7f3298003b70 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f329800bad0 con 0x7f3298004800 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f329800bcd0 con 0x7f32b0072950 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32b5a00640 1 --2- 172.21.15.40:0/386115421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298003b70 0x7f3298003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32b5000640 1 --2- 172.21.15.40:0/386115421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298004800 0x7f3298004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.466+0000 7f32b6400640 1 --2- 172.21.15.40:0/386115421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b0072950 0x7f32980051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.467+0000 7f32b5a00640 1 -- 172.21.15.40:0/386115421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b0072950 msgr2=0x7f32980051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.467+0000 7f32b5a00640 1 --2- 172.21.15.40:0/386115421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b0072950 0x7f32980051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.467+0000 7f32b5a00640 1 -- 172.21.15.40:0/386115421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298004800 msgr2=0x7f3298004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.467+0000 7f32b5a00640 1 --2- 172.21.15.40:0/386115421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298004800 0x7f3298004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.467+0000 7f32b5a00640 1 -- 172.21.15.40:0/386115421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3298003520 con 0x7f3298003b70 2024-04-26T23:07:43.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.467+0000 7f32b5000640 1 --2- 172.21.15.40:0/386115421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298004800 0x7f3298004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:07:43.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.468+0000 7f32b5a00640 1 --2- 172.21.15.40:0/386115421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298003b70 0x7f3298003f70 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f32a40018d0 tx=0x7f3298005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34dc307674735f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:43.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.468+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a40014f0 con 0x7f3298003b70 2024-04-26T23:07:43.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.468+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32a40052d0 con 0x7f3298003b70 2024-04-26T23:07:43.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.468+0000 7f32afe00640 1 -- 172.21.15.40:0/386115421 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a402aca0 con 0x7f3298003b70 2024-04-26T23:07:43.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.469+0000 7f32b8400640 1 -- 172.21.15.40:0/386115421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298003b70 msgr2=0x7f3298003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:43.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.469+0000 7f32b8400640 1 --2- 172.21.15.40:0/386115421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298003b70 0x7f3298003f70 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f32a40018d0 tx=0x7f3298005830 comp rx=0 tx=0).stop 2024-04-26T23:07:43.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.469+0000 7f32b6400640 1 -- 172.21.15.40:0/386115421 reap_dead start 2024-04-26T23:07:43.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.469+0000 7f32b8400640 1 -- 172.21.15.40:0/386115421 shutdown_connections 2024-04-26T23:07:43.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.469+0000 7f32b8400640 1 -- 172.21.15.40:0/386115421 >> 172.21.15.40:0/386115421 conn(0x7f32b00d0be0 msgr2=0x7f32b00d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:43.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.469+0000 7f32b8400640 1 -- 172.21.15.40:0/386115421 shutdown_connections 2024-04-26T23:07:43.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.469+0000 7f32b8400640 1 -- 172.21.15.40:0/386115421 wait complete. 2024-04-26T23:07:43.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.471+0000 7f32b8400640 1 Processor -- start 2024-04-26T23:07:43.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.471+0000 7f32b8400640 1 -- start start 2024-04-26T23:07:43.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b8400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298003b70 0x7f32b00b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b8400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b015a120 0x7f32b00c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b8400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b01512a0 0x7f32b00ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b8400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32b015d570 con 0x7f3298003b70 2024-04-26T23:07:43.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b8400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32b015d3f0 con 0x7f32b015a120 2024-04-26T23:07:43.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b8400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f32b006fff0 con 0x7f32b01512a0 2024-04-26T23:07:43.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298003b70 0x7f32b00b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298003b70 0x7f32b00b62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53548/0 (socket says 172.21.15.40:53548) 2024-04-26T23:07:43.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b5a00640 1 -- 172.21.15.40:0/1096722882 learned_addr learned my addr 172.21.15.40:0/1096722882 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:43.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b6400640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b01512a0 0x7f32b00ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b5000640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b015a120 0x7f32b00c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b5a00640 1 -- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b01512a0 msgr2=0x7f32b00ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:43.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b5a00640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b01512a0 0x7f32b00ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b5a00640 1 -- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b015a120 msgr2=0x7f32b00c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:43.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b5a00640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b015a120 0x7f32b00c9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.472+0000 7f32b5a00640 1 -- 172.21.15.40:0/1096722882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b00b3a20 con 0x7f3298003b70 2024-04-26T23:07:43.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.473+0000 7f32b5a00640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298003b70 0x7f32b00b62b0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f32a40018d0 tx=0x7f32a40012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:43.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.473+0000 7f32ad600640 1 -- 172.21.15.40:0/1096722882 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a403f020 con 0x7f3298003b70 2024-04-26T23:07:43.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.473+0000 7f32ad600640 1 -- 172.21.15.40:0/1096722882 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32a4006e50 con 0x7f3298003b70 2024-04-26T23:07:43.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.473+0000 7f32ad600640 1 -- 172.21.15.40:0/1096722882 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a4037040 con 0x7f3298003b70 2024-04-26T23:07:43.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.474+0000 7f32b8400640 1 -- 172.21.15.40:0/1096722882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32b00c8f70 con 0x7f3298003b70 2024-04-26T23:07:43.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.474+0000 7f32b8400640 1 -- 172.21.15.40:0/1096722882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32b00b3cc0 con 0x7f3298003b70 2024-04-26T23:07:43.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.475+0000 7f32b8400640 1 -- 172.21.15.40:0/1096722882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32b00694f0 con 0x7f3298003b70 2024-04-26T23:07:43.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.476+0000 7f32ad600640 1 -- 172.21.15.40:0/1096722882 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f32a4042020 con 0x7f3298003b70 2024-04-26T23:07:43.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.477+0000 7f32ad600640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f329006c8f0 0x7f329006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.477+0000 7f32ad600640 1 -- 172.21.15.40:0/1096722882 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5507+0+0 (secure 0 0 0) 0x7f32a40ab710 con 0x7f3298003b70 2024-04-26T23:07:43.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.478+0000 7f32b5000640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f329006c8f0 0x7f329006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.478+0000 7f32b5000640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f329006c8f0 0x7f329006edb0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f329c001200 tx=0x7f329c001e30 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:43.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.481+0000 7f32ad600640 1 -- 172.21.15.40:0/1096722882 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f32a4049030 con 0x7f3298003b70 2024-04-26T23:07:43.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.696+0000 7f32b8400640 1 -- 172.21.15.40:0/1096722882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f32b00c9490 con 0x7f3298003b70 2024-04-26T23:07:43.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.740+0000 7f32ad600640 1 -- 172.21.15.40:0/1096722882 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v84) ==== 97+0+0 (secure 0 0 0) 0x7f32a40795a0 con 0x7f3298003b70 2024-04-26T23:07:43.741 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-04-26T23:07:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.743+0000 7f32b8400640 1 -- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f329006c8f0 msgr2=0x7f329006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.743+0000 7f32b8400640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f329006c8f0 0x7f329006edb0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f329c001200 tx=0x7f329c001e30 comp rx=0 tx=0).stop 2024-04-26T23:07:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.743+0000 7f32b8400640 1 -- 172.21.15.40:0/1096722882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298003b70 msgr2=0x7f32b00b62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.743+0000 7f32b8400640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298003b70 0x7f32b00b62b0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f32a40018d0 tx=0x7f32a40012d0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.743+0000 7f32b8400640 1 -- 172.21.15.40:0/1096722882 shutdown_connections 2024-04-26T23:07:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.743+0000 7f32b8400640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f329006c8f0 0x7f329006edb0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.744+0000 7f32b8400640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b01512a0 0x7f32b00ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.744+0000 7f32b8400640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b015a120 0x7f32b00c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.744+0000 7f32b8400640 1 --2- 172.21.15.40:0/1096722882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3298003b70 0x7f32b00b62b0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.744+0000 7f32b8400640 1 -- 172.21.15.40:0/1096722882 >> 172.21.15.40:0/1096722882 conn(0x7f32b00d0be0 msgr2=0x7f32b00d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.744+0000 7f32b8400640 1 -- 172.21.15.40:0/1096722882 shutdown_connections 2024-04-26T23:07:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.744+0000 7f32b8400640 1 -- 172.21.15.40:0/1096722882 wait complete. 2024-04-26T23:07:43.768 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-26T23:07:43.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.975+0000 7f85f6600640 1 Processor -- start 2024-04-26T23:07:43.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.976+0000 7f85f6600640 1 -- start start 2024-04-26T23:07:43.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.977+0000 7f85f6600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f00b2e70 0x7f85f00b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.977+0000 7f85f6600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f85f00b3250 0x7f85f00b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.977+0000 7f85f6600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85f0160270 0x7f85f01606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.977+0000 7f85f6600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f85f00b5c40 con 0x7f85f0160270 2024-04-26T23:07:43.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.977+0000 7f85f6600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f85f00b2bf0 con 0x7f85f00b3250 2024-04-26T23:07:43.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.977+0000 7f85f6600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f85f00b1fc0 con 0x7f85f00b2e70 2024-04-26T23:07:43.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.978+0000 7f85f5c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85f0160270 0x7f85f01606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.978+0000 7f85f5c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85f0160270 0x7f85f01606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53564/0 (socket says 172.21.15.40:53564) 2024-04-26T23:07:43.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.978+0000 7f85f5c00640 1 -- 172.21.15.40:0/154163748 learned_addr learned my addr 172.21.15.40:0/154163748 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:43.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.978+0000 7f85f5200640 1 --2- 172.21.15.40:0/154163748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f00b2e70 0x7f85f00b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.978+0000 7f85f5c00640 1 -- 172.21.15.40:0/154163748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f00b2e70 msgr2=0x7f85f00b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:43.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.978+0000 7f85f5c00640 1 --2- 172.21.15.40:0/154163748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f00b2e70 0x7f85f00b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.978+0000 7f85f5c00640 1 -- 172.21.15.40:0/154163748 >> v2:172.21.15.103:3300/0 conn(0x7f85f00b3250 msgr2=0x7f85f00b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:43.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.978+0000 7f85f5c00640 1 --2- 172.21.15.40:0/154163748 >> v2:172.21.15.103:3300/0 conn(0x7f85f00b3250 0x7f85f00b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.978+0000 7f85f5c00640 1 -- 172.21.15.40:0/154163748 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f00b2910 con 0x7f85f0160270 2024-04-26T23:07:43.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.979+0000 7f85f5c00640 1 --2- 172.21.15.40:0/154163748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f0160270 0x7f85f01606a0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f85f00d2390 tx=0x7f85e802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fccb978cc16acb33 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:43.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.980+0000 7f85ef400640 1 -- 172.21.15.40:0/154163748 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e8036070 con 0x7f85f0160270 2024-04-26T23:07:43.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.980+0000 7f85ef400640 1 -- 172.21.15.40:0/154163748 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e8031600 con 0x7f85f0160270 2024-04-26T23:07:43.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.980+0000 7f85ef400640 1 -- 172.21.15.40:0/154163748 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e80319d0 con 0x7f85f0160270 2024-04-26T23:07:43.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.980+0000 7f85f6600640 1 -- 172.21.15.40:0/154163748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f0160270 msgr2=0x7f85f01606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:43.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.980+0000 7f85f6600640 1 --2- 172.21.15.40:0/154163748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f0160270 0x7f85f01606a0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f85f00d2390 tx=0x7f85e802a6a0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.981+0000 7f85f6600640 1 -- 172.21.15.40:0/154163748 shutdown_connections 2024-04-26T23:07:43.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.981+0000 7f85f6600640 1 --2- 172.21.15.40:0/154163748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f0160270 0x7f85f01606a0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.981+0000 7f85f6600640 1 --2- 172.21.15.40:0/154163748 >> v2:172.21.15.103:3300/0 conn(0x7f85f00b3250 0x7f85f00b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.981+0000 7f85f6600640 1 --2- 172.21.15.40:0/154163748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f00b2e70 0x7f85f00b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.981+0000 7f85f6600640 1 -- 172.21.15.40:0/154163748 >> 172.21.15.40:0/154163748 conn(0x7f85f00b6700 msgr2=0x7f85f00b6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:43.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.981+0000 7f85f6600640 1 -- 172.21.15.40:0/154163748 shutdown_connections 2024-04-26T23:07:43.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.981+0000 7f85f6600640 1 -- 172.21.15.40:0/154163748 wait complete. 2024-04-26T23:07:43.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.983+0000 7f85f6600640 1 Processor -- start 2024-04-26T23:07:43.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.983+0000 7f85f6600640 1 -- start start 2024-04-26T23:07:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.985+0000 7f85f6600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f00b2e70 0x7f85f015e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.985+0000 7f85f6600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f00b3250 0x7f85f015d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.985+0000 7f85f6600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85f0160270 0x7f85f01f88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.985+0000 7f85f6600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f85f00b6430 con 0x7f85f00b3250 2024-04-26T23:07:43.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.985+0000 7f85f6600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f85f00b6150 con 0x7f85f0160270 2024-04-26T23:07:43.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.985+0000 7f85f6600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f85f0148af0 con 0x7f85f00b2e70 2024-04-26T23:07:43.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.985+0000 7f85efe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f00b3250 0x7f85f015d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.985+0000 7f85efe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f00b3250 0x7f85f015d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53568/0 (socket says 172.21.15.40:53568) 2024-04-26T23:07:43.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.985+0000 7f85efe00640 1 -- 172.21.15.40:0/1980582705 learned_addr learned my addr 172.21.15.40:0/1980582705 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:43.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.985+0000 7f85f5200640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f00b2e70 0x7f85f015e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.985+0000 7f85efe00640 1 -- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f00b2e70 msgr2=0x7f85f015e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:43.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.986+0000 7f85efe00640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f00b2e70 0x7f85f015e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.986+0000 7f85efe00640 1 -- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85f0160270 msgr2=0x7f85f01f88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:43.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.986+0000 7f85f5c00640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85f0160270 0x7f85f01f88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.986+0000 7f85efe00640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85f0160270 0x7f85f01f88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:43.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.986+0000 7f85efe00640 1 -- 172.21.15.40:0/1980582705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f0158d70 con 0x7f85f00b3250 2024-04-26T23:07:43.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.986+0000 7f85efe00640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f00b3250 0x7f85f015d670 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00b4d0 tx=0x7f85dc00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:43.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.986+0000 7f85ecc00640 1 -- 172.21.15.40:0/1980582705 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85dc014070 con 0x7f85f00b3250 2024-04-26T23:07:43.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.986+0000 7f85ecc00640 1 -- 172.21.15.40:0/1980582705 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85dc010870 con 0x7f85f00b3250 2024-04-26T23:07:43.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.986+0000 7f85ecc00640 1 -- 172.21.15.40:0/1980582705 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85dc010c60 con 0x7f85f00b3250 2024-04-26T23:07:43.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.986+0000 7f85f6600640 1 -- 172.21.15.40:0/1980582705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85f0161ea0 con 0x7f85f00b3250 2024-04-26T23:07:43.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.987+0000 7f85f6600640 1 -- 172.21.15.40:0/1980582705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85f00cc8e0 con 0x7f85f00b3250 2024-04-26T23:07:43.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.988+0000 7f85f6600640 1 -- 172.21.15.40:0/1980582705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85f0069370 con 0x7f85f00b3250 2024-04-26T23:07:43.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.989+0000 7f85ecc00640 1 -- 172.21.15.40:0/1980582705 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f85dc017070 con 0x7f85f00b3250 2024-04-26T23:07:43.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.990+0000 7f85ecc00640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85d006c8f0 0x7f85d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:43.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.990+0000 7f85ecc00640 1 -- 172.21.15.40:0/1980582705 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5249+0+0 (secure 0 0 0) 0x7f85dc08b830 con 0x7f85f00b3250 2024-04-26T23:07:43.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.990+0000 7f85f5200640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85d006c8f0 0x7f85d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.992+0000 7f85f5200640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85d006c8f0 0x7f85d006edb0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f85e4006fd0 tx=0x7f85e4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:43.994+0000 7f85ecc00640 1 -- 172.21.15.40:0/1980582705 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f85dc059d80 con 0x7f85f00b3250 2024-04-26T23:07:44.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.283+0000 7f85f6600640 1 -- 172.21.15.40:0/1980582705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f85f0069bb0 con 0x7f85f00b3250 2024-04-26T23:07:44.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.284+0000 7f85ecc00640 1 -- 172.21.15.40:0/1980582705 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) ==== 80+0+5304 (secure 0 0 0) 0x7f85dc059680 con 0x7f85f00b3250 2024-04-26T23:07:44.284 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:44.284 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:07:44.284 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-04-26T23:07:44.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmF/UQHBAACdb8iskp2MuBMV6S5tYFJg==", 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmniCyMxAAT/rVbIKj9EnvwDiblKMCeg==", 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.285 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:07:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:07:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:07:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-04-26T23:07:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmLGZhJxAAg/fsrRrNoF/Jx1PnBVjjeQ==", 2024-04-26T23:07:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:07:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:07:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:07:44.286 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxms3DnAxAAxJj/rybrv/0cLK8PmQ619A==", 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-04-26T23:07:44.287 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDrMSxmbeONCxAATB67ujBDn2tVxpbMju7Mfg==", 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDuMSxmqE/PJhAAsNNu5ar8tHWQauxaXOvagA==", 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDyMSxmd2trBhAAbnl7EBldxgtOd1s0Ll0A9w==", 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD1MSxmqdtKEhAA9oxBxOu/musCKyladrJgYA==", 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.289 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD7MSxm+kl7IxAAlPAGIemdC8gcvp+koeULaQ==", 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.290 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD+MSxmzKiaLRAA2UPS1wwJHgMxQf/zLMjMVQ==", 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQACMixmluHrCxAAmbmb33f5MVSTWNuy1D0Zsg==", 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.291 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAFMixm6GQjJRAArmdnX7ioekT8GSdsbeB1Ow==", 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg==", 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.292 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow r", 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmVhn4IhAA94XsDk1x5Oax5smXPoFQKg==", 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxm3qhbGRAAK62blgPVShIYskDsMcaffQ==", 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmxof/KxAAUmXVV4Vdp2DZnRjQzQ0Inw==", 2024-04-26T23:07:44.294 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDlMSxmmVwcGRAAqgHymPPTPuS6XXC6LuOVPg==", 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.295 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmVf+9EhAAcpk8NgFqgCLXeHA2z7oHmA==", 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmjjpcCBAAjDQRUBn/TK9hF3T2WBObyw==", 2024-04-26T23:07:44.296 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:07:44.297 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:07:44.297 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-04-26T23:07:44.297 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:07:44.297 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.297 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:44.297 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-26T23:07:44.297 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:07:44.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.287+0000 7f85f6600640 1 -- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85d006c8f0 msgr2=0x7f85d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:44.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.287+0000 7f85f6600640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85d006c8f0 0x7f85d006edb0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f85e4006fd0 tx=0x7f85e4008040 comp rx=0 tx=0).stop 2024-04-26T23:07:44.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.287+0000 7f85f6600640 1 -- 172.21.15.40:0/1980582705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f00b3250 msgr2=0x7f85f015d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:44.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.287+0000 7f85f6600640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f00b3250 0x7f85f015d670 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00b4d0 tx=0x7f85dc00b990 comp rx=0 tx=0).stop 2024-04-26T23:07:44.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.287+0000 7f85f6600640 1 -- 172.21.15.40:0/1980582705 shutdown_connections 2024-04-26T23:07:44.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.287+0000 7f85f6600640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85d006c8f0 0x7f85d006edb0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.287+0000 7f85f6600640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85f0160270 0x7f85f01f88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.287+0000 7f85f6600640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f00b3250 0x7f85f015d670 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.287+0000 7f85f6600640 1 --2- 172.21.15.40:0/1980582705 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f00b2e70 0x7f85f015e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.287+0000 7f85f6600640 1 -- 172.21.15.40:0/1980582705 >> 172.21.15.40:0/1980582705 conn(0x7f85f00b6700 msgr2=0x7f85f015dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:44.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.287+0000 7f85f6600640 1 -- 172.21.15.40:0/1980582705 shutdown_connections 2024-04-26T23:07:44.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.287+0000 7f85f6600640 1 -- 172.21.15.40:0/1980582705 wait complete. 2024-04-26T23:07:44.305 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:07:44.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.505+0000 7fa49cc00640 1 Processor -- start 2024-04-26T23:07:44.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.505+0000 7fa49cc00640 1 -- start start 2024-04-26T23:07:44.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.507+0000 7fa49cc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4980cc480 0x7fa498072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.507+0000 7fa49cc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa498072780 0x7fa498166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.507+0000 7fa49cc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa49815d7c0 con 0x7fa498071e60 2024-04-26T23:07:44.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.507+0000 7fa49cc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa49815daa0 con 0x7fa498072780 2024-04-26T23:07:44.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.507+0000 7fa49cc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa49815dd80 con 0x7fa4980cc480 2024-04-26T23:07:44.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.507+0000 7fa495400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4980cc480 0x7fa498072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:44.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.507+0000 7fa496800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa498072780 0x7fa498166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:44.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.508+0000 7fa495400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4980cc480 0x7fa498072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:42074/0 (socket says 172.21.15.40:42074) 2024-04-26T23:07:44.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.508+0000 7fa495400640 1 -- 172.21.15.40:0/2303202199 learned_addr learned my addr 172.21.15.40:0/2303202199 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:44.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.508+0000 7fa495400640 1 -- 172.21.15.40:0/2303202199 >> v2:172.21.15.103:3300/0 conn(0x7fa498072780 msgr2=0x7fa498166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:44.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.508+0000 7fa495400640 1 --2- 172.21.15.40:0/2303202199 >> v2:172.21.15.103:3300/0 conn(0x7fa498072780 0x7fa498166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.508+0000 7fa495400640 1 -- 172.21.15.40:0/2303202199 >> v1:172.21.15.103:6789/0 conn(0x7fa498071e60 legacy=0x7fa49806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:44.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.508+0000 7fa495400640 1 -- 172.21.15.40:0/2303202199 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49815e060 con 0x7fa4980cc480 2024-04-26T23:07:44.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.510+0000 7fa495400640 1 --2- 172.21.15.40:0/2303202199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4980cc480 0x7fa498072240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa484009870 tx=0x7fa48402a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebd57b6ee9e33f21 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:44.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.510+0000 7fa494a00640 1 -- 172.21.15.40:0/2303202199 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa484037070 con 0x7fa4980cc480 2024-04-26T23:07:44.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.510+0000 7fa494a00640 1 -- 172.21.15.40:0/2303202199 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa484032440 con 0x7fa4980cc480 2024-04-26T23:07:44.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.510+0000 7fa494a00640 1 -- 172.21.15.40:0/2303202199 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa484032810 con 0x7fa4980cc480 2024-04-26T23:07:44.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.511+0000 7fa49cc00640 1 -- 172.21.15.40:0/2303202199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4980cc480 msgr2=0x7fa498072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:44.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.511+0000 7fa49cc00640 1 --2- 172.21.15.40:0/2303202199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4980cc480 0x7fa498072240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa484009870 tx=0x7fa48402a250 comp rx=0 tx=0).stop 2024-04-26T23:07:44.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.511+0000 7fa49cc00640 1 -- 172.21.15.40:0/2303202199 shutdown_connections 2024-04-26T23:07:44.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.511+0000 7fa49cc00640 1 --2- 172.21.15.40:0/2303202199 >> v2:172.21.15.103:3300/0 conn(0x7fa498072780 0x7fa498166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.511+0000 7fa49cc00640 1 --2- 172.21.15.40:0/2303202199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4980cc480 0x7fa498072240 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.511+0000 7fa49cc00640 1 -- 172.21.15.40:0/2303202199 >> 172.21.15.40:0/2303202199 conn(0x7fa4980d0be0 msgr2=0x7fa4980d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:44.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.511+0000 7fa49cc00640 1 -- 172.21.15.40:0/2303202199 shutdown_connections 2024-04-26T23:07:44.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.511+0000 7fa49cc00640 1 -- 172.21.15.40:0/2303202199 wait complete. 2024-04-26T23:07:44.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.513+0000 7fa49cc00640 1 Processor -- start 2024-04-26T23:07:44.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.513+0000 7fa49cc00640 1 -- start start 2024-04-26T23:07:44.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.514+0000 7fa49cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa498071e60 0x7fa49814e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.514+0000 7fa49cc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa498072780 0x7fa4980b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.514+0000 7fa49cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4980cc480 0x7fa498148c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.514+0000 7fa49cc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa49815d3f0 con 0x7fa4980cc480 2024-04-26T23:07:44.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.514+0000 7fa49cc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa498072c10 con 0x7fa498071e60 2024-04-26T23:07:44.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.514+0000 7fa49cc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa49815d570 con 0x7fa498072780 2024-04-26T23:07:44.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.514+0000 7fa496800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4980cc480 0x7fa498148c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.515+0000 7fa496800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4980cc480 0x7fa498148c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53572/0 (socket says 172.21.15.40:53572) 2024-04-26T23:07:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.515+0000 7fa496800640 1 -- 172.21.15.40:0/2245987312 learned_addr learned my addr 172.21.15.40:0/2245987312 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.515+0000 7fa495e00640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa498071e60 0x7fa49814e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.515+0000 7fa495400640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa498072780 0x7fa4980b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.515+0000 7fa496800640 1 -- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa498072780 msgr2=0x7fa4980b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.515+0000 7fa496800640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa498072780 0x7fa4980b1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.515+0000 7fa496800640 1 -- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa498071e60 msgr2=0x7fa49814e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.515+0000 7fa496800640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa498071e60 0x7fa49814e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.515+0000 7fa496800640 1 -- 172.21.15.40:0/2245987312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4980ce9c0 con 0x7fa4980cc480 2024-04-26T23:07:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.515+0000 7fa496800640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4980cc480 0x7fa498148c80 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fa49000efc0 tx=0x7fa49000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.516+0000 7fa48e000640 1 -- 172.21.15.40:0/2245987312 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490018020 con 0x7fa4980cc480 2024-04-26T23:07:44.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.516+0000 7fa48e000640 1 -- 172.21.15.40:0/2245987312 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa490014480 con 0x7fa4980cc480 2024-04-26T23:07:44.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.516+0000 7fa48e000640 1 -- 172.21.15.40:0/2245987312 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490014870 con 0x7fa4980cc480 2024-04-26T23:07:44.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.516+0000 7fa49cc00640 1 -- 172.21.15.40:0/2245987312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4980ceca0 con 0x7fa4980cc480 2024-04-26T23:07:44.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.516+0000 7fa49cc00640 1 -- 172.21.15.40:0/2245987312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa49814edc0 con 0x7fa4980cc480 2024-04-26T23:07:44.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.517+0000 7fa49cc00640 1 -- 172.21.15.40:0/2245987312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4980694f0 con 0x7fa4980cc480 2024-04-26T23:07:44.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.518+0000 7fa48e000640 1 -- 172.21.15.40:0/2245987312 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa49001b020 con 0x7fa4980cc480 2024-04-26T23:07:44.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.519+0000 7fa48e000640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa47806c8f0 0x7fa47806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.520+0000 7fa48e000640 1 -- 172.21.15.40:0/2245987312 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5249+0+0 (secure 0 0 0) 0x7fa49008c580 con 0x7fa4980cc480 2024-04-26T23:07:44.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.520+0000 7fa495e00640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa47806c8f0 0x7fa47806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:44.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.521+0000 7fa495e00640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa47806c8f0 0x7fa47806edb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa488000f60 tx=0x7fa488009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:44.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.524+0000 7fa48e000640 1 -- 172.21.15.40:0/2245987312 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa490022030 con 0x7fa4980cc480 2024-04-26T23:07:44.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.729+0000 7fa49cc00640 1 -- 172.21.15.40:0/2245987312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa49806ce60 con 0x7fa4980cc480 2024-04-26T23:07:44.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.730+0000 7fa48e000640 1 -- 172.21.15.40:0/2245987312 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+13129 (secure 0 0 0) 0x7fa490010040 con 0x7fa4980cc480 2024-04-26T23:07:44.731 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:44.731 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":86,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:07:43.755230+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:07:44.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.733+0000 7fa49cc00640 1 -- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa47806c8f0 msgr2=0x7fa47806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:44.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.733+0000 7fa49cc00640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa47806c8f0 0x7fa47806edb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa488000f60 tx=0x7fa488009040 comp rx=0 tx=0).stop 2024-04-26T23:07:44.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.733+0000 7fa49cc00640 1 -- 172.21.15.40:0/2245987312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4980cc480 msgr2=0x7fa498148c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:44.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.733+0000 7fa49cc00640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4980cc480 0x7fa498148c80 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fa49000efc0 tx=0x7fa49000c490 comp rx=0 tx=0).stop 2024-04-26T23:07:44.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.733+0000 7fa49cc00640 1 -- 172.21.15.40:0/2245987312 shutdown_connections 2024-04-26T23:07:44.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.733+0000 7fa49cc00640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa47806c8f0 0x7fa47806edb0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.733+0000 7fa49cc00640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4980cc480 0x7fa498148c80 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.733+0000 7fa49cc00640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa498072780 0x7fa4980b1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.733+0000 7fa49cc00640 1 --2- 172.21.15.40:0/2245987312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa498071e60 0x7fa49814e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.734+0000 7fa49cc00640 1 -- 172.21.15.40:0/2245987312 >> 172.21.15.40:0/2245987312 conn(0x7fa4980d0be0 msgr2=0x7fa498072d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.734+0000 7fa49cc00640 1 -- 172.21.15.40:0/2245987312 shutdown_connections 2024-04-26T23:07:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.734+0000 7fa49cc00640 1 -- 172.21.15.40:0/2245987312 wait complete. 2024-04-26T23:07:44.757 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:07:44.757 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:07:44.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.967+0000 7f945f000640 1 Processor -- start 2024-04-26T23:07:44.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.967+0000 7f945f000640 1 -- start start 2024-04-26T23:07:44.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.969+0000 7f945f000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9458071e60 0x7f945806f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.969+0000 7f945f000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f945815d7c0 con 0x7f94580cc480 2024-04-26T23:07:44.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.969+0000 7f945f000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f945815daa0 con 0x7f945806fe20 2024-04-26T23:07:44.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.969+0000 7f945f000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f945815dd80 con 0x7f9458071e60 2024-04-26T23:07:44.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.969+0000 7f945d000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f94580cc480 0x7f9458166ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33408/0 (socket says 172.21.15.40:33408) 2024-04-26T23:07:44.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.969+0000 7f945d000640 1 -- 172.21.15.40:0/4136335394 learned_addr learned my addr 172.21.15.40:0/4136335394 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:44.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.969+0000 7f9457e00640 1 --2- 172.21.15.40:0/4136335394 >> v2:172.21.15.103:3300/0 conn(0x7f9458071e60 0x7f945806f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:44.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1691718797 0 0) 0x7f945815d7c0 con 0x7f94580cc480 2024-04-26T23:07:44.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f943c003520 con 0x7f94580cc480 2024-04-26T23:07:44.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3029118838 0 0) 0x7f943c003520 con 0x7f94580cc480 2024-04-26T23:07:44.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 >> v2:172.21.15.103:3300/0 conn(0x7f9458071e60 msgr2=0x7f945806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 --2- 172.21.15.40:0/4136335394 >> v2:172.21.15.103:3300/0 conn(0x7f9458071e60 0x7f945806f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 >> v1:172.21.15.103:6789/0 conn(0x7f945806fe20 legacy=0x7f9458072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945815e060 con 0x7f94580cc480 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f9450001670 con 0x7f94580cc480 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 >> v1:172.21.15.40:6789/0 conn(0x7f94580cc480 legacy=0x7f9458166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 --2- 172.21.15.40:0/4136335394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f943c0039b0 0x7f943c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 --2- 172.21.15.40:0/4136335394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f943c004800 0x7f943c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 --2- 172.21.15.40:0/4136335394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f943c0051c0 0x7f943c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f945815dd80 con 0x7f943c0039b0 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f943c00c060 con 0x7f943c0051c0 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.970+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f943c00c260 con 0x7f943c004800 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.971+0000 7f9457e00640 1 --2- 172.21.15.40:0/4136335394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f943c0039b0 0x7f943c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.971+0000 7f945d000640 1 --2- 172.21.15.40:0/4136335394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f943c0051c0 0x7f943c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:44.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.971+0000 7f9457e00640 1 -- 172.21.15.40:0/4136335394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f943c004800 msgr2=0x7f943c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:44.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.971+0000 7f9457e00640 1 --2- 172.21.15.40:0/4136335394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f943c004800 0x7f943c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.971+0000 7f9457e00640 1 -- 172.21.15.40:0/4136335394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f943c0051c0 msgr2=0x7f943c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:44.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.971+0000 7f9457e00640 1 --2- 172.21.15.40:0/4136335394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f943c0051c0 0x7f943c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.971+0000 7f945d000640 1 -- 172.21.15.40:0/4136335394 reap_dead start 2024-04-26T23:07:44.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.971+0000 7f9457e00640 1 -- 172.21.15.40:0/4136335394 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943c003520 con 0x7f943c0039b0 2024-04-26T23:07:44.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.972+0000 7f9457e00640 1 --2- 172.21.15.40:0/4136335394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f943c0039b0 0x7f943c003dd0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f9448009fc0 tx=0x7f943c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c49bb28884803 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:44.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.972+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9448009c10 con 0x7f943c0039b0 2024-04-26T23:07:44.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.972+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9448036070 con 0x7f943c0039b0 2024-04-26T23:07:44.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.972+0000 7f9456a00640 1 -- 172.21.15.40:0/4136335394 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9448031880 con 0x7f943c0039b0 2024-04-26T23:07:44.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.973+0000 7f945f000640 1 -- 172.21.15.40:0/4136335394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f943c0039b0 msgr2=0x7f943c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.973+0000 7f945f000640 1 --2- 172.21.15.40:0/4136335394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f943c0039b0 0x7f943c003dd0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f9448009fc0 tx=0x7f943c005830 comp rx=0 tx=0).stop 2024-04-26T23:07:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.973+0000 7f945f000640 1 -- 172.21.15.40:0/4136335394 shutdown_connections 2024-04-26T23:07:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.973+0000 7f945f000640 1 --2- 172.21.15.40:0/4136335394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f943c0039b0 0x7f943c003dd0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.973+0000 7f945f000640 1 -- 172.21.15.40:0/4136335394 >> 172.21.15.40:0/4136335394 conn(0x7f94580d0be0 msgr2=0x7f94580d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.973+0000 7f945f000640 1 -- 172.21.15.40:0/4136335394 shutdown_connections 2024-04-26T23:07:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.973+0000 7f945f000640 1 -- 172.21.15.40:0/4136335394 wait complete. 2024-04-26T23:07:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.975+0000 7f945f000640 1 Processor -- start 2024-04-26T23:07:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.975+0000 7f945f000640 1 -- start start 2024-04-26T23:07:44.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.976+0000 7f945f000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f943c0039b0 0x7f945814bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.976+0000 7f945f000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94581f8b60 0x7f94581f8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.976+0000 7f945f000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94580ca4b0 0x7f94580ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.976+0000 7f945f000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f945815d570 con 0x7f94581f8b60 2024-04-26T23:07:44.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.976+0000 7f945f000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f945815d3f0 con 0x7f94580ca4b0 2024-04-26T23:07:44.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.976+0000 7f945f000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f945806f8e0 con 0x7f943c0039b0 2024-04-26T23:07:44.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.976+0000 7f9457400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94581f8b60 0x7f94581f8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:44.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.977+0000 7f9457400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94581f8b60 0x7f94581f8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44418/0 (socket says 172.21.15.40:44418) 2024-04-26T23:07:44.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.977+0000 7f9457400640 1 -- 172.21.15.40:0/3915450561 learned_addr learned my addr 172.21.15.40:0/3915450561 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:44.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.977+0000 7f9457400640 1 -- 172.21.15.40:0/3915450561 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f943c0039b0 msgr2=0x7f945814bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:44.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.977+0000 7f9457400640 1 --2- 172.21.15.40:0/3915450561 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f943c0039b0 0x7f945814bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.977+0000 7f9457400640 1 -- 172.21.15.40:0/3915450561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94580ca4b0 msgr2=0x7f94580ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:44.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.977+0000 7f945d000640 1 --2- 172.21.15.40:0/3915450561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94580ca4b0 0x7f94580ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:44.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.977+0000 7f9457400640 1 --2- 172.21.15.40:0/3915450561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94580ca4b0 0x7f94580ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:44.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.977+0000 7f9457400640 1 -- 172.21.15.40:0/3915450561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945814c040 con 0x7f94581f8b60 2024-04-26T23:07:44.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.977+0000 7f9457400640 1 --2- 172.21.15.40:0/3915450561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94581f8b60 0x7f94581f8fe0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f944000c500 tx=0x7f944000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:44.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.978+0000 7f944fe00640 1 -- 172.21.15.40:0/3915450561 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94400052d0 con 0x7f94581f8b60 2024-04-26T23:07:44.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.978+0000 7f944fe00640 1 -- 172.21.15.40:0/3915450561 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9440005470 con 0x7f94581f8b60 2024-04-26T23:07:44.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.978+0000 7f944fe00640 1 -- 172.21.15.40:0/3915450561 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9440012850 con 0x7f94581f8b60 2024-04-26T23:07:44.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.978+0000 7f945f000640 1 -- 172.21.15.40:0/3915450561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94580c9410 con 0x7f94581f8b60 2024-04-26T23:07:44.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.978+0000 7f945f000640 1 -- 172.21.15.40:0/3915450561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94580c99d0 con 0x7f94581f8b60 2024-04-26T23:07:44.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.979+0000 7f945f000640 1 -- 172.21.15.40:0/3915450561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94580694f0 con 0x7f94581f8b60 2024-04-26T23:07:44.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.980+0000 7f944fe00640 1 -- 172.21.15.40:0/3915450561 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f94400129f0 con 0x7f94581f8b60 2024-04-26T23:07:44.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.981+0000 7f944fe00640 1 --2- 172.21.15.40:0/3915450561 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f943406c8f0 0x7f943406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:44.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.981+0000 7f944fe00640 1 -- 172.21.15.40:0/3915450561 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5249+0+0 (secure 0 0 0) 0x7f944008b6c0 con 0x7f94581f8b60 2024-04-26T23:07:44.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.982+0000 7f9457e00640 1 --2- 172.21.15.40:0/3915450561 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f943406c8f0 0x7f943406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:44.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.983+0000 7f9457e00640 1 --2- 172.21.15.40:0/3915450561 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f943406c8f0 0x7f943406edb0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f9448035000 tx=0x7f944802d000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:44.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:44.985+0000 7f944fe00640 1 -- 172.21.15.40:0/3915450561 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9440059ca0 con 0x7f94581f8b60 2024-04-26T23:07:45.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.188+0000 7f945f000640 1 -- 172.21.15.40:0/3915450561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f945806de90 con 0x7f94581f8b60 2024-04-26T23:07:45.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.189+0000 7f944fe00640 1 -- 172.21.15.40:0/3915450561 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) ==== 93+0+10 (secure 0 0 0) 0x7f94400595a0 con 0x7f94581f8b60 2024-04-26T23:07:45.189 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:07:45.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.192+0000 7f945f000640 1 -- 172.21.15.40:0/3915450561 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f943406c8f0 msgr2=0x7f943406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:45.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.192+0000 7f945f000640 1 --2- 172.21.15.40:0/3915450561 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f943406c8f0 0x7f943406edb0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f9448035000 tx=0x7f944802d000 comp rx=0 tx=0).stop 2024-04-26T23:07:45.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.192+0000 7f945f000640 1 -- 172.21.15.40:0/3915450561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94581f8b60 msgr2=0x7f94581f8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:45.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.192+0000 7f945f000640 1 --2- 172.21.15.40:0/3915450561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94581f8b60 0x7f94581f8fe0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f944000c500 tx=0x7f944000c9c0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.192+0000 7f945f000640 1 -- 172.21.15.40:0/3915450561 shutdown_connections 2024-04-26T23:07:45.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.192+0000 7f945f000640 1 --2- 172.21.15.40:0/3915450561 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f943406c8f0 0x7f943406edb0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.192+0000 7f945f000640 1 --2- 172.21.15.40:0/3915450561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94580ca4b0 0x7f94580ca930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.192+0000 7f945f000640 1 --2- 172.21.15.40:0/3915450561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94581f8b60 0x7f94581f8fe0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.192+0000 7f945f000640 1 --2- 172.21.15.40:0/3915450561 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f943c0039b0 0x7f945814bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.192+0000 7f945f000640 1 -- 172.21.15.40:0/3915450561 >> 172.21.15.40:0/3915450561 conn(0x7f94580d0be0 msgr2=0x7f94580ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:45.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.192+0000 7f945f000640 1 -- 172.21.15.40:0/3915450561 shutdown_connections 2024-04-26T23:07:45.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.193+0000 7f945f000640 1 -- 172.21.15.40:0/3915450561 wait complete. 2024-04-26T23:07:45.212 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T23:07:45.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.425+0000 7fa7f2400640 1 Processor -- start 2024-04-26T23:07:45.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.426+0000 7fa7f2400640 1 -- start start 2024-04-26T23:07:45.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.427+0000 7fa7f2400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa7ec071e60 0x7fa7ec06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:45.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.427+0000 7fa7f2400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7ec06fe20 0x7fa7ec072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:45.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.427+0000 7fa7f2400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7ec15d7c0 con 0x7fa7ec072780 2024-04-26T23:07:45.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.427+0000 7fa7f2400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa7ec15daa0 con 0x7fa7ec06fe20 2024-04-26T23:07:45.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.427+0000 7fa7f2400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa7ec15dd80 con 0x7fa7ec071e60 2024-04-26T23:07:45.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.427+0000 7fa7eaa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7ec06fe20 0x7fa7ec072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:45.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.428+0000 7fa7eaa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7ec06fe20 0x7fa7ec072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44430/0 (socket says 172.21.15.40:44430) 2024-04-26T23:07:45.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.428+0000 7fa7eaa00640 1 -- 172.21.15.40:0/1529793319 learned_addr learned my addr 172.21.15.40:0/1529793319 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:45.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.428+0000 7fa7ebe00640 1 --1- 172.21.15.40:0/1529793319 >> v1:172.21.15.40:6789/0 conn(0x7fa7ec072780 0x7fa7ec0cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33424/0 (socket says 172.21.15.40:33424) 2024-04-26T23:07:45.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.428+0000 7fa7eb400640 1 --2- 172.21.15.40:0/1529793319 >> v2:172.21.15.103:3300/0 conn(0x7fa7ec071e60 0x7fa7ec06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:45.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.428+0000 7fa7eaa00640 1 -- 172.21.15.40:0/1529793319 >> v2:172.21.15.103:3300/0 conn(0x7fa7ec071e60 msgr2=0x7fa7ec06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:45.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.428+0000 7fa7eaa00640 1 --2- 172.21.15.40:0/1529793319 >> v2:172.21.15.103:3300/0 conn(0x7fa7ec071e60 0x7fa7ec06f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.428+0000 7fa7eaa00640 1 -- 172.21.15.40:0/1529793319 >> v1:172.21.15.40:6789/0 conn(0x7fa7ec072780 legacy=0x7fa7ec0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:45.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.428+0000 7fa7eaa00640 1 -- 172.21.15.40:0/1529793319 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ec15e060 con 0x7fa7ec06fe20 2024-04-26T23:07:45.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.428+0000 7fa7ea000640 1 -- 172.21.15.40:0/1529793319 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4121151621 0 0) 0x7fa7ec15d7c0 con 0x7fa7ec072780 2024-04-26T23:07:45.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.429+0000 7fa7eaa00640 1 --2- 172.21.15.40:0/1529793319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec06fe20 0x7fa7ec072240 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec0b3700 tx=0x7fa7d802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a659ddb61270f06f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:45.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.430+0000 7fa7ea000640 1 -- 172.21.15.40:0/1529793319 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8037070 con 0x7fa7ec06fe20 2024-04-26T23:07:45.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.430+0000 7fa7ea000640 1 -- 172.21.15.40:0/1529793319 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d8032440 con 0x7fa7ec06fe20 2024-04-26T23:07:45.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.430+0000 7fa7ea000640 1 -- 172.21.15.40:0/1529793319 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8032810 con 0x7fa7ec06fe20 2024-04-26T23:07:45.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.430+0000 7fa7f2400640 1 -- 172.21.15.40:0/1529793319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec06fe20 msgr2=0x7fa7ec072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:45.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.430+0000 7fa7f2400640 1 --2- 172.21.15.40:0/1529793319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec06fe20 0x7fa7ec072240 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec0b3700 tx=0x7fa7d802a270 comp rx=0 tx=0).stop 2024-04-26T23:07:45.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.430+0000 7fa7f2400640 1 -- 172.21.15.40:0/1529793319 shutdown_connections 2024-04-26T23:07:45.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.430+0000 7fa7f2400640 1 --2- 172.21.15.40:0/1529793319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec06fe20 0x7fa7ec072240 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.430+0000 7fa7f2400640 1 --2- 172.21.15.40:0/1529793319 >> v2:172.21.15.103:3300/0 conn(0x7fa7ec071e60 0x7fa7ec06f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.430+0000 7fa7f2400640 1 -- 172.21.15.40:0/1529793319 >> 172.21.15.40:0/1529793319 conn(0x7fa7ec0d0be0 msgr2=0x7fa7ec0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:45.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.430+0000 7fa7f2400640 1 -- 172.21.15.40:0/1529793319 shutdown_connections 2024-04-26T23:07:45.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.430+0000 7fa7f2400640 1 -- 172.21.15.40:0/1529793319 wait complete. 2024-04-26T23:07:45.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.432+0000 7fa7f2400640 1 Processor -- start 2024-04-26T23:07:45.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.432+0000 7fa7f2400640 1 -- start start 2024-04-26T23:07:45.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.433+0000 7fa7f2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec06fe20 0x7fa7ec15ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:45.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.433+0000 7fa7f2400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7ec071e60 0x7fa7ec15bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:45.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.433+0000 7fa7f2400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7ec072780 0x7fa7ec0c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:45.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.433+0000 7fa7f2400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec15d3f0 con 0x7fa7ec06fe20 2024-04-26T23:07:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.433+0000 7fa7f2400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec072bd0 con 0x7fa7ec072780 2024-04-26T23:07:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.433+0000 7fa7f2400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa7ec15d570 con 0x7fa7ec071e60 2024-04-26T23:07:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.434+0000 7fa7eb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec06fe20 0x7fa7ec15ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.434+0000 7fa7eb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec06fe20 0x7fa7ec15ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44440/0 (socket says 172.21.15.40:44440) 2024-04-26T23:07:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.434+0000 7fa7eb400640 1 -- 172.21.15.40:0/2016914337 learned_addr learned my addr 172.21.15.40:0/2016914337 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:45.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.434+0000 7fa7eb400640 1 -- 172.21.15.40:0/2016914337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7ec071e60 msgr2=0x7fa7ec15bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.434+0000 7fa7eb400640 1 --2- 172.21.15.40:0/2016914337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7ec071e60 0x7fa7ec15bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.434+0000 7fa7eb400640 1 -- 172.21.15.40:0/2016914337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7ec072780 msgr2=0x7fa7ec0c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.434+0000 7fa7eb400640 1 --2- 172.21.15.40:0/2016914337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7ec072780 0x7fa7ec0c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.434+0000 7fa7eb400640 1 -- 172.21.15.40:0/2016914337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ec1f5b20 con 0x7fa7ec06fe20 2024-04-26T23:07:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.435+0000 7fa7eb400640 1 --2- 172.21.15.40:0/2016914337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec06fe20 0x7fa7ec15ba30 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc00d830 tx=0x7fa7dc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.435+0000 7fa7e3400640 1 -- 172.21.15.40:0/2016914337 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc00b6c0 con 0x7fa7ec06fe20 2024-04-26T23:07:45.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.435+0000 7fa7e3400640 1 -- 172.21.15.40:0/2016914337 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7dc019070 con 0x7fa7ec06fe20 2024-04-26T23:07:45.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.435+0000 7fa7e3400640 1 -- 172.21.15.40:0/2016914337 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc00ba80 con 0x7fa7ec06fe20 2024-04-26T23:07:45.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.435+0000 7fa7f2400640 1 -- 172.21.15.40:0/2016914337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7ec0ca6e0 con 0x7fa7ec06fe20 2024-04-26T23:07:45.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.435+0000 7fa7f2400640 1 -- 172.21.15.40:0/2016914337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7ec1f5040 con 0x7fa7ec06fe20 2024-04-26T23:07:45.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.436+0000 7fa7f2400640 1 -- 172.21.15.40:0/2016914337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7ec0694f0 con 0x7fa7ec06fe20 2024-04-26T23:07:45.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.437+0000 7fa7e3400640 1 -- 172.21.15.40:0/2016914337 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa7dc01e460 con 0x7fa7ec06fe20 2024-04-26T23:07:45.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.438+0000 7fa7e3400640 1 --2- 172.21.15.40:0/2016914337 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7cc06c8f0 0x7fa7cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:45.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.438+0000 7fa7e3400640 1 -- 172.21.15.40:0/2016914337 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5249+0+0 (secure 0 0 0) 0x7fa7dc08c110 con 0x7fa7ec06fe20 2024-04-26T23:07:45.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.439+0000 7fa7eaa00640 1 --2- 172.21.15.40:0/2016914337 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7cc06c8f0 0x7fa7cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:45.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.440+0000 7fa7eaa00640 1 --2- 172.21.15.40:0/2016914337 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7cc06c8f0 0x7fa7cc06edb0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8034000 tx=0x7fa7d802c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:45.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.444+0000 7fa7e3400640 1 -- 172.21.15.40:0/2016914337 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa7dc05a660 con 0x7fa7ec06fe20 2024-04-26T23:07:45.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.651+0000 7fa7f2400640 1 -- 172.21.15.40:0/2016914337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa7ec0cbd70 con 0x7fa7ec06fe20 2024-04-26T23:07:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.652+0000 7fa7e3400640 1 -- 172.21.15.40:0/2016914337 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v86) ==== 84+0+65 (secure 0 0 0) 0x7fa7dc059f60 con 0x7fa7ec06fe20 2024-04-26T23:07:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:45.652 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-04-26T23:07:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:07:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-04-26T23:07:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-04-26T23:07:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:07:45.653 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-04-26T23:07:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.655+0000 7fa7f2400640 1 -- 172.21.15.40:0/2016914337 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7cc06c8f0 msgr2=0x7fa7cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.655+0000 7fa7f2400640 1 --2- 172.21.15.40:0/2016914337 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7cc06c8f0 0x7fa7cc06edb0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8034000 tx=0x7fa7d802c000 comp rx=0 tx=0).stop 2024-04-26T23:07:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.655+0000 7fa7f2400640 1 -- 172.21.15.40:0/2016914337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec06fe20 msgr2=0x7fa7ec15ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.655+0000 7fa7f2400640 1 --2- 172.21.15.40:0/2016914337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec06fe20 0x7fa7ec15ba30 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc00d830 tx=0x7fa7dc00dcf0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.655+0000 7fa7f2400640 1 -- 172.21.15.40:0/2016914337 shutdown_connections 2024-04-26T23:07:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.655+0000 7fa7f2400640 1 --2- 172.21.15.40:0/2016914337 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7cc06c8f0 0x7fa7cc06edb0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.655+0000 7fa7f2400640 1 --2- 172.21.15.40:0/2016914337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7ec072780 0x7fa7ec0c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.655+0000 7fa7f2400640 1 --2- 172.21.15.40:0/2016914337 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7ec071e60 0x7fa7ec15bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.655+0000 7fa7f2400640 1 --2- 172.21.15.40:0/2016914337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec06fe20 0x7fa7ec15ba30 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.655+0000 7fa7f2400640 1 -- 172.21.15.40:0/2016914337 >> 172.21.15.40:0/2016914337 conn(0x7fa7ec0d0be0 msgr2=0x7fa7ec0cb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.655+0000 7fa7f2400640 1 -- 172.21.15.40:0/2016914337 shutdown_connections 2024-04-26T23:07:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.655+0000 7fa7f2400640 1 -- 172.21.15.40:0/2016914337 wait complete. 2024-04-26T23:07:45.676 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T23:07:45.676 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T23:07:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.888+0000 7fc712200640 1 Processor -- start 2024-04-26T23:07:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.888+0000 7fc712200640 1 -- start start 2024-04-26T23:07:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.889+0000 7fc712200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc70c15c160 0x7fc70c0b4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.889+0000 7fc712200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc70c15c5c0 0x7fc70c0b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.889+0000 7fc712200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc70c15cb90 con 0x7fc70c073e10 2024-04-26T23:07:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.889+0000 7fc712200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc70c153040 con 0x7fc70c15c5c0 2024-04-26T23:07:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.889+0000 7fc712200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc70c15fc40 con 0x7fc70c15c160 2024-04-26T23:07:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.890+0000 7fc711800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc70c15c5c0 0x7fc70c0b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.890+0000 7fc711800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc70c15c5c0 0x7fc70c0b4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:35238/0 (socket says 172.21.15.40:35238) 2024-04-26T23:07:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.890+0000 7fc711800640 1 -- 172.21.15.40:0/1618688547 learned_addr learned my addr 172.21.15.40:0/1618688547 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.890+0000 7fc70be00640 1 --2- 172.21.15.40:0/1618688547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc70c15c160 0x7fc70c0b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.891+0000 7fc711800640 1 -- 172.21.15.40:0/1618688547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc70c15c160 msgr2=0x7fc70c0b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.891+0000 7fc711800640 1 --2- 172.21.15.40:0/1618688547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc70c15c160 0x7fc70c0b4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.891+0000 7fc711800640 1 -- 172.21.15.40:0/1618688547 >> v1:172.21.15.103:6789/0 conn(0x7fc70c073e10 legacy=0x7fc70c1600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.891+0000 7fc711800640 1 -- 172.21.15.40:0/1618688547 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70c15ce70 con 0x7fc70c15c5c0 2024-04-26T23:07:45.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.893+0000 7fc711800640 1 --2- 172.21.15.40:0/1618688547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc70c15c5c0 0x7fc70c0b4d80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc704009e50 tx=0x7fc70402c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=519e0679f4ffa795 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:45.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.893+0000 7fc70b400640 1 -- 172.21.15.40:0/1618688547 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704032610 con 0x7fc70c15c5c0 2024-04-26T23:07:45.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.894+0000 7fc70b400640 1 -- 172.21.15.40:0/1618688547 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc704037070 con 0x7fc70c15c5c0 2024-04-26T23:07:45.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.894+0000 7fc70b400640 1 -- 172.21.15.40:0/1618688547 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7040329d0 con 0x7fc70c15c5c0 2024-04-26T23:07:45.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.894+0000 7fc712200640 1 -- 172.21.15.40:0/1618688547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc70c15c5c0 msgr2=0x7fc70c0b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:45.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.894+0000 7fc712200640 1 --2- 172.21.15.40:0/1618688547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc70c15c5c0 0x7fc70c0b4d80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc704009e50 tx=0x7fc70402c920 comp rx=0 tx=0).stop 2024-04-26T23:07:45.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.894+0000 7fc712200640 1 -- 172.21.15.40:0/1618688547 shutdown_connections 2024-04-26T23:07:45.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.894+0000 7fc712200640 1 --2- 172.21.15.40:0/1618688547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc70c15c5c0 0x7fc70c0b4d80 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.894+0000 7fc712200640 1 --2- 172.21.15.40:0/1618688547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc70c15c160 0x7fc70c0b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.894+0000 7fc712200640 1 -- 172.21.15.40:0/1618688547 >> 172.21.15.40:0/1618688547 conn(0x7fc70c075eb0 msgr2=0x7fc70c068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:45.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.894+0000 7fc712200640 1 -- 172.21.15.40:0/1618688547 shutdown_connections 2024-04-26T23:07:45.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.894+0000 7fc712200640 1 -- 172.21.15.40:0/1618688547 wait complete. 2024-04-26T23:07:45.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.896+0000 7fc712200640 1 Processor -- start 2024-04-26T23:07:45.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.896+0000 7fc712200640 1 -- start start 2024-04-26T23:07:45.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.897+0000 7fc712200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc70c073e10 0x7fc70c0cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:45.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.897+0000 7fc712200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc70c15c160 0x7fc70c0ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:45.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.897+0000 7fc712200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc70c15c5c0 0x7fc70c0cf650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:45.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.897+0000 7fc712200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc70c073540 con 0x7fc70c15c160 2024-04-26T23:07:45.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.897+0000 7fc712200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc70c0733c0 con 0x7fc70c15c5c0 2024-04-26T23:07:45.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.897+0000 7fc712200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc70c15d150 con 0x7fc70c073e10 2024-04-26T23:07:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.898+0000 7fc70be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc70c15c160 0x7fc70c0ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.898+0000 7fc70be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc70c15c160 0x7fc70c0ca110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44446/0 (socket says 172.21.15.40:44446) 2024-04-26T23:07:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.898+0000 7fc70be00640 1 -- 172.21.15.40:0/1097086421 learned_addr learned my addr 172.21.15.40:0/1097086421 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.898+0000 7fc710e00640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc70c073e10 0x7fc70c0cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.898+0000 7fc711800640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc70c15c5c0 0x7fc70c0cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.898+0000 7fc70be00640 1 -- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc70c073e10 msgr2=0x7fc70c0cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.898+0000 7fc70be00640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc70c073e10 0x7fc70c0cddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.898+0000 7fc70be00640 1 -- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc70c15c5c0 msgr2=0x7fc70c0cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.898+0000 7fc70be00640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc70c15c5c0 0x7fc70c0cf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.898+0000 7fc70be00640 1 -- 172.21.15.40:0/1097086421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70c0ce310 con 0x7fc70c15c160 2024-04-26T23:07:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.898+0000 7fc711800640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc70c15c5c0 0x7fc70c0cf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.898+0000 7fc70be00640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc70c15c160 0x7fc70c0ca110 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800e9b0 tx=0x7fc6f800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.899+0000 7fc708c00640 1 -- 172.21.15.40:0/1097086421 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8017070 con 0x7fc70c15c160 2024-04-26T23:07:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.899+0000 7fc708c00640 1 -- 172.21.15.40:0/1097086421 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6f800c9a0 con 0x7fc70c15c160 2024-04-26T23:07:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.899+0000 7fc708c00640 1 -- 172.21.15.40:0/1097086421 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f800cd90 con 0x7fc70c15c160 2024-04-26T23:07:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.899+0000 7fc712200640 1 -- 172.21.15.40:0/1097086421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc70c151390 con 0x7fc70c15c160 2024-04-26T23:07:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.899+0000 7fc712200640 1 -- 172.21.15.40:0/1097086421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc70c0ceb70 con 0x7fc70c15c160 2024-04-26T23:07:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.900+0000 7fc712200640 1 -- 172.21.15.40:0/1097086421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc70c069370 con 0x7fc70c15c160 2024-04-26T23:07:45.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.901+0000 7fc708c00640 1 -- 172.21.15.40:0/1097086421 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc6f801a070 con 0x7fc70c15c160 2024-04-26T23:07:45.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.902+0000 7fc708c00640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc6ec06c8f0 0x7fc6ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.902+0000 7fc708c00640 1 -- 172.21.15.40:0/1097086421 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5249+0+0 (secure 0 0 0) 0x7fc6f808ba40 con 0x7fc70c15c160 2024-04-26T23:07:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.903+0000 7fc710e00640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc6ec06c8f0 0x7fc6ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.904+0000 7fc710e00640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc6ec06c8f0 0x7fc6ec06edb0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc700001640 tx=0x7fc700009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:45.907+0000 7fc708c00640 1 -- 172.21.15.40:0/1097086421 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc6f8059f90 con 0x7fc70c15c160 2024-04-26T23:07:46.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.119+0000 7fc712200640 1 -- 172.21.15.40:0/1097086421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc70c06f4e0 con 0x7fc70c15c160 2024-04-26T23:07:46.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.770+0000 7fc708c00640 1 -- 172.21.15.40:0/1097086421 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v87) ==== 138+0+0 (secure 0 0 0) 0x7fc6f8059890 con 0x7fc70c15c160 2024-04-26T23:07:46.770 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-04-26T23:07:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.774+0000 7fc712200640 1 -- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc6ec06c8f0 msgr2=0x7fc6ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.774+0000 7fc712200640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc6ec06c8f0 0x7fc6ec06edb0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc700001640 tx=0x7fc700009040 comp rx=0 tx=0).stop 2024-04-26T23:07:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.774+0000 7fc712200640 1 -- 172.21.15.40:0/1097086421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc70c15c160 msgr2=0x7fc70c0ca110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.774+0000 7fc712200640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc70c15c160 0x7fc70c0ca110 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800e9b0 tx=0x7fc6f800ee70 comp rx=0 tx=0).stop 2024-04-26T23:07:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.774+0000 7fc712200640 1 -- 172.21.15.40:0/1097086421 shutdown_connections 2024-04-26T23:07:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.774+0000 7fc712200640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc6ec06c8f0 0x7fc6ec06edb0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:46.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.774+0000 7fc712200640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc70c15c5c0 0x7fc70c0cf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:46.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.774+0000 7fc712200640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc70c15c160 0x7fc70c0ca110 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:46.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.774+0000 7fc712200640 1 --2- 172.21.15.40:0/1097086421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc70c073e10 0x7fc70c0cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:46.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.774+0000 7fc712200640 1 -- 172.21.15.40:0/1097086421 >> 172.21.15.40:0/1097086421 conn(0x7fc70c075eb0 msgr2=0x7fc70c15b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:46.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.774+0000 7fc712200640 1 -- 172.21.15.40:0/1097086421 shutdown_connections 2024-04-26T23:07:46.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:46.775+0000 7fc712200640 1 -- 172.21.15.40:0/1097086421 wait complete. 2024-04-26T23:07:46.804 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T23:07:47.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.020+0000 7fe0baa00640 1 Processor -- start 2024-04-26T23:07:47.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.020+0000 7fe0baa00640 1 -- start start 2024-04-26T23:07:47.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.022+0000 7fe0baa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0b4145970 0x7fe0b41612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:47.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.022+0000 7fe0baa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe0b4146060 0x7fe0b40b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:47.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.022+0000 7fe0baa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe0b40b3fb0 0x7fe0b40d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:47.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.022+0000 7fe0baa00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe0b40ceed0 con 0x7fe0b4145970 2024-04-26T23:07:47.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.022+0000 7fe0baa00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe0b40b59f0 con 0x7fe0b40b3fb0 2024-04-26T23:07:47.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.022+0000 7fe0baa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe0b41617f0 con 0x7fe0b4146060 2024-04-26T23:07:47.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.023+0000 7fe0b9600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0b4145970 0x7fe0b41612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:47.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.023+0000 7fe0b9600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0b4145970 0x7fe0b41612b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44450/0 (socket says 172.21.15.40:44450) 2024-04-26T23:07:47.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.023+0000 7fe0b9600640 1 -- 172.21.15.40:0/1182022953 learned_addr learned my addr 172.21.15.40:0/1182022953 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:47.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.023+0000 7fe0b9600640 1 -- 172.21.15.40:0/1182022953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe0b4146060 msgr2=0x7fe0b40b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:47.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.023+0000 7fe0b8c00640 1 --2- 172.21.15.40:0/1182022953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe0b4146060 0x7fe0b40b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:47.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.023+0000 7fe0ba000640 1 --2- 172.21.15.40:0/1182022953 >> v2:172.21.15.103:3300/0 conn(0x7fe0b40b3fb0 0x7fe0b40d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:47.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.023+0000 7fe0b9600640 1 --2- 172.21.15.40:0/1182022953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe0b4146060 0x7fe0b40b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:47.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.023+0000 7fe0b9600640 1 -- 172.21.15.40:0/1182022953 >> v2:172.21.15.103:3300/0 conn(0x7fe0b40b3fb0 msgr2=0x7fe0b40d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:47.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.023+0000 7fe0b9600640 1 --2- 172.21.15.40:0/1182022953 >> v2:172.21.15.103:3300/0 conn(0x7fe0b40b3fb0 0x7fe0b40d06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:47.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.023+0000 7fe0b9600640 1 -- 172.21.15.40:0/1182022953 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b40b5f40 con 0x7fe0b4145970 2024-04-26T23:07:47.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.023+0000 7fe0b8c00640 1 --2- 172.21.15.40:0/1182022953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe0b4146060 0x7fe0b40b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:47.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.023+0000 7fe0ba000640 1 --2- 172.21.15.40:0/1182022953 >> v2:172.21.15.103:3300/0 conn(0x7fe0b40b3fb0 0x7fe0b40d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:47.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.024+0000 7fe0b9600640 1 --2- 172.21.15.40:0/1182022953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b4145970 0x7fe0b41612b0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8009870 tx=0x7fe0a802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a81c91f73861f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:47.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.025+0000 7fe0b3e00640 1 -- 172.21.15.40:0/1182022953 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8037070 con 0x7fe0b4145970 2024-04-26T23:07:47.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.025+0000 7fe0b3e00640 1 -- 172.21.15.40:0/1182022953 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0a8032440 con 0x7fe0b4145970 2024-04-26T23:07:47.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.025+0000 7fe0b3e00640 1 -- 172.21.15.40:0/1182022953 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8032810 con 0x7fe0b4145970 2024-04-26T23:07:47.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.025+0000 7fe0baa00640 1 -- 172.21.15.40:0/1182022953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b4145970 msgr2=0x7fe0b41612b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:47.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.025+0000 7fe0baa00640 1 --2- 172.21.15.40:0/1182022953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b4145970 0x7fe0b41612b0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8009870 tx=0x7fe0a802a210 comp rx=0 tx=0).stop 2024-04-26T23:07:47.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.025+0000 7fe0baa00640 1 -- 172.21.15.40:0/1182022953 shutdown_connections 2024-04-26T23:07:47.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.025+0000 7fe0baa00640 1 --2- 172.21.15.40:0/1182022953 >> v2:172.21.15.103:3300/0 conn(0x7fe0b40b3fb0 0x7fe0b40d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:47.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.025+0000 7fe0baa00640 1 --2- 172.21.15.40:0/1182022953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe0b4146060 0x7fe0b40b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:47.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.025+0000 7fe0baa00640 1 --2- 172.21.15.40:0/1182022953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b4145970 0x7fe0b41612b0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:47.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.025+0000 7fe0baa00640 1 -- 172.21.15.40:0/1182022953 >> 172.21.15.40:0/1182022953 conn(0x7fe0b4075eb0 msgr2=0x7fe0b4068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:47.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.026+0000 7fe0baa00640 1 -- 172.21.15.40:0/1182022953 shutdown_connections 2024-04-26T23:07:47.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.026+0000 7fe0baa00640 1 -- 172.21.15.40:0/1182022953 wait complete. 2024-04-26T23:07:47.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.027+0000 7fe0baa00640 1 Processor -- start 2024-04-26T23:07:47.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.027+0000 7fe0baa00640 1 -- start start 2024-04-26T23:07:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.028+0000 7fe0baa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe0b4145970 0x7fe0b41f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.028+0000 7fe0baa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0b4146060 0x7fe0b415d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.028+0000 7fe0baa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b415c340 0x7fe0b415fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:47.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.028+0000 7fe0baa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b40cf3e0 con 0x7fe0b415c340 2024-04-26T23:07:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.028+0000 7fe0baa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b40cf100 con 0x7fe0b4146060 2024-04-26T23:07:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.028+0000 7fe0baa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe0b4145d50 con 0x7fe0b4145970 2024-04-26T23:07:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.029+0000 7fe0ba000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b415c340 0x7fe0b415fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.029+0000 7fe0ba000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b415c340 0x7fe0b415fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44456/0 (socket says 172.21.15.40:44456) 2024-04-26T23:07:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.029+0000 7fe0ba000640 1 -- 172.21.15.40:0/2654452533 learned_addr learned my addr 172.21.15.40:0/2654452533 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:47.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.029+0000 7fe0b8c00640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0b4146060 0x7fe0b415d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:47.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.029+0000 7fe0b9600640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe0b4145970 0x7fe0b41f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:47.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.030+0000 7fe0ba000640 1 -- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe0b4145970 msgr2=0x7fe0b41f57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:47.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.030+0000 7fe0ba000640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe0b4145970 0x7fe0b41f57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:47.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.030+0000 7fe0ba000640 1 -- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0b4146060 msgr2=0x7fe0b415d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:47.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.030+0000 7fe0ba000640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0b4146060 0x7fe0b415d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:47.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.030+0000 7fe0ba000640 1 -- 172.21.15.40:0/2654452533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b415e170 con 0x7fe0b415c340 2024-04-26T23:07:47.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.030+0000 7fe0ba000640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b415c340 0x7fe0b415fda0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00cf50 tx=0x7fe0ac00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:47.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.030+0000 7fe0b1600640 1 -- 172.21.15.40:0/2654452533 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac01a070 con 0x7fe0b415c340 2024-04-26T23:07:47.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.030+0000 7fe0b1600640 1 -- 172.21.15.40:0/2654452533 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0ac00d840 con 0x7fe0b415c340 2024-04-26T23:07:47.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.030+0000 7fe0b1600640 1 -- 172.21.15.40:0/2654452533 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac00dc30 con 0x7fe0b415c340 2024-04-26T23:07:47.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.030+0000 7fe0baa00640 1 -- 172.21.15.40:0/2654452533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b415e450 con 0x7fe0b415c340 2024-04-26T23:07:47.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.030+0000 7fe0baa00640 1 -- 172.21.15.40:0/2654452533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b415ea10 con 0x7fe0b415c340 2024-04-26T23:07:47.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.032+0000 7fe0baa00640 1 -- 172.21.15.40:0/2654452533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0b40ce210 con 0x7fe0b415c340 2024-04-26T23:07:47.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.033+0000 7fe0b1600640 1 -- 172.21.15.40:0/2654452533 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe0ac01d070 con 0x7fe0b415c340 2024-04-26T23:07:47.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.034+0000 7fe0b1600640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe09406c8f0 0x7fe09406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:47.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.034+0000 7fe0b1600640 1 -- 172.21.15.40:0/2654452533 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5619+0+0 (secure 0 0 0) 0x7fe0ac08c240 con 0x7fe0b415c340 2024-04-26T23:07:47.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.035+0000 7fe0b9600640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe09406c8f0 0x7fe09406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:47.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.036+0000 7fe0b9600640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe09406c8f0 0x7fe09406edb0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8034000 tx=0x7fe0a802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:47.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.040+0000 7fe0b1600640 1 -- 172.21.15.40:0/2654452533 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe0b40ce210 con 0x7fe0b415c340 2024-04-26T23:07:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.242+0000 7fe0baa00640 1 -- 172.21.15.40:0/2654452533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe0b4076420 con 0x7fe0b415c340 2024-04-26T23:07:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.786+0000 7fe0b1600640 1 -- 172.21.15.40:0/2654452533 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v88) ==== 170+0+0 (secure 0 0 0) 0x7fe0b4076420 con 0x7fe0b415c340 2024-04-26T23:07:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-04-26T23:07:47.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.793+0000 7fe0baa00640 1 -- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe09406c8f0 msgr2=0x7fe09406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:47.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.793+0000 7fe0baa00640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe09406c8f0 0x7fe09406edb0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8034000 tx=0x7fe0a802e000 comp rx=0 tx=0).stop 2024-04-26T23:07:47.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.793+0000 7fe0baa00640 1 -- 172.21.15.40:0/2654452533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b415c340 msgr2=0x7fe0b415fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:47.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.793+0000 7fe0baa00640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b415c340 0x7fe0b415fda0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00cf50 tx=0x7fe0ac00a7d0 comp rx=0 tx=0).stop 2024-04-26T23:07:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.794+0000 7fe0baa00640 1 -- 172.21.15.40:0/2654452533 shutdown_connections 2024-04-26T23:07:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.794+0000 7fe0baa00640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe09406c8f0 0x7fe09406edb0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.794+0000 7fe0baa00640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b415c340 0x7fe0b415fda0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.794+0000 7fe0baa00640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0b4146060 0x7fe0b415d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.794+0000 7fe0baa00640 1 --2- 172.21.15.40:0/2654452533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe0b4145970 0x7fe0b41f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.794+0000 7fe0baa00640 1 -- 172.21.15.40:0/2654452533 >> 172.21.15.40:0/2654452533 conn(0x7fe0b4075eb0 msgr2=0x7fe0b4141820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.795+0000 7fe0baa00640 1 -- 172.21.15.40:0/2654452533 shutdown_connections 2024-04-26T23:07:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:47.795+0000 7fe0baa00640 1 -- 172.21.15.40:0/2654452533 wait complete. 2024-04-26T23:07:47.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T23:07:48.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.011+0000 7f37cd000640 1 Processor -- start 2024-04-26T23:07:48.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.012+0000 7f37cd000640 1 -- start start 2024-04-26T23:07:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.013+0000 7f37cd000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f37c80b4140 0x7f37c80c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.013+0000 7f37cd000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37c80b3580 con 0x7f37c806e3f0 2024-04-26T23:07:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.013+0000 7f37cd000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37c8167310 con 0x7f37c80d0d00 2024-04-26T23:07:48.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.013+0000 7f37cd000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f37c81675f0 con 0x7f37c80b4140 2024-04-26T23:07:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.014+0000 7f37c6800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f37c806e3f0 0x7f37c80b2e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33432/0 (socket says 172.21.15.40:33432) 2024-04-26T23:07:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.014+0000 7f37c6800640 1 -- 172.21.15.40:0/2417573592 learned_addr learned my addr 172.21.15.40:0/2417573592 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:48.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.014+0000 7f37c5e00640 1 --2- 172.21.15.40:0/2417573592 >> v2:172.21.15.103:3300/0 conn(0x7f37c80b4140 0x7f37c80c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:48.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.014+0000 7f37c4a00640 1 -- 172.21.15.40:0/2417573592 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3901923836 0 0) 0x7f37c80b3580 con 0x7f37c806e3f0 2024-04-26T23:07:48.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.014+0000 7f37c4a00640 1 -- 172.21.15.40:0/2417573592 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37ac003520 con 0x7f37c806e3f0 2024-04-26T23:07:48.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.015+0000 7f37c5e00640 1 -- 172.21.15.40:0/2417573592 >> v1:172.21.15.103:6789/0 conn(0x7f37c80d0d00 legacy=0x7f37c80b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:48.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.015+0000 7f37c5e00640 1 -- 172.21.15.40:0/2417573592 >> v1:172.21.15.40:6789/0 conn(0x7f37c806e3f0 legacy=0x7f37c80b2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:48.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.015+0000 7f37c5e00640 1 -- 172.21.15.40:0/2417573592 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c81678d0 con 0x7f37c80b4140 2024-04-26T23:07:48.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.016+0000 7f37c5e00640 1 --2- 172.21.15.40:0/2417573592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c80b4140 0x7f37c80c5dd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f37b8009d90 tx=0x7f37b802a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c3f1571d237e45b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:48.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.017+0000 7f37c4a00640 1 -- 172.21.15.40:0/2417573592 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b8031600 con 0x7f37c80b4140 2024-04-26T23:07:48.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.017+0000 7f37c4a00640 1 -- 172.21.15.40:0/2417573592 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b80317a0 con 0x7f37c80b4140 2024-04-26T23:07:48.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.017+0000 7f37c4a00640 1 -- 172.21.15.40:0/2417573592 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b8031b50 con 0x7f37c80b4140 2024-04-26T23:07:48.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.017+0000 7f37cd000640 1 -- 172.21.15.40:0/2417573592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c80b4140 msgr2=0x7f37c80c5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:48.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.017+0000 7f37cd000640 1 --2- 172.21.15.40:0/2417573592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c80b4140 0x7f37c80c5dd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f37b8009d90 tx=0x7f37b802a5d0 comp rx=0 tx=0).stop 2024-04-26T23:07:48.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.017+0000 7f37cd000640 1 -- 172.21.15.40:0/2417573592 shutdown_connections 2024-04-26T23:07:48.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.017+0000 7f37cd000640 1 --2- 172.21.15.40:0/2417573592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c80b4140 0x7f37c80c5dd0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:48.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.017+0000 7f37cd000640 1 -- 172.21.15.40:0/2417573592 >> 172.21.15.40:0/2417573592 conn(0x7f37c80d2e00 msgr2=0x7f37c80d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:48.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.017+0000 7f37cd000640 1 -- 172.21.15.40:0/2417573592 shutdown_connections 2024-04-26T23:07:48.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.018+0000 7f37cd000640 1 -- 172.21.15.40:0/2417573592 wait complete. 2024-04-26T23:07:48.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.019+0000 7f37cd000640 1 Processor -- start 2024-04-26T23:07:48.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.019+0000 7f37cd000640 1 -- start start 2024-04-26T23:07:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.020+0000 7f37cd000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f37c806e3f0 0x7f37c80d0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.020+0000 7f37cd000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37c80b4140 0x7f37c815f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.020+0000 7f37cd000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c80d0d00 0x7f37c80cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.020+0000 7f37cd000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f37c80b46a0 con 0x7f37c80b4140 2024-04-26T23:07:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.020+0000 7f37cd000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f37c80b4520 con 0x7f37c80d0d00 2024-04-26T23:07:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.020+0000 7f37cd000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f37c80d2780 con 0x7f37c806e3f0 2024-04-26T23:07:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.021+0000 7f37c5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37c80b4140 0x7f37c815f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:48.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.021+0000 7f37c5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37c80b4140 0x7f37c815f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44472/0 (socket says 172.21.15.40:44472) 2024-04-26T23:07:48.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.021+0000 7f37c5400640 1 -- 172.21.15.40:0/1340301570 learned_addr learned my addr 172.21.15.40:0/1340301570 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:48.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.021+0000 7f37c5e00640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f37c806e3f0 0x7f37c80d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:48.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.021+0000 7f37c6800640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c80d0d00 0x7f37c80cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:48.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.021+0000 7f37c5400640 1 -- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f37c806e3f0 msgr2=0x7f37c80d0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:48.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.021+0000 7f37c5400640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f37c806e3f0 0x7f37c80d0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:48.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.021+0000 7f37c5400640 1 -- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c80d0d00 msgr2=0x7f37c80cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:48.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.021+0000 7f37c5400640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c80d0d00 0x7f37c80cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:48.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.021+0000 7f37c5400640 1 -- 172.21.15.40:0/1340301570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c80ce110 con 0x7f37c80b4140 2024-04-26T23:07:48.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.022+0000 7f37c5400640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37c80b4140 0x7f37c815f3f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f37b000c3c0 tx=0x7f37b000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:48.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.022+0000 7f37be000640 1 -- 172.21.15.40:0/1340301570 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b00052d0 con 0x7f37c80b4140 2024-04-26T23:07:48.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.022+0000 7f37be000640 1 -- 172.21.15.40:0/1340301570 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b0005490 con 0x7f37c80b4140 2024-04-26T23:07:48.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.022+0000 7f37cd000640 1 -- 172.21.15.40:0/1340301570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37c80ce950 con 0x7f37c80b4140 2024-04-26T23:07:48.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.022+0000 7f37be000640 1 -- 172.21.15.40:0/1340301570 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b0007260 con 0x7f37c80b4140 2024-04-26T23:07:48.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.022+0000 7f37cd000640 1 -- 172.21.15.40:0/1340301570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37c80ce670 con 0x7f37c80b4140 2024-04-26T23:07:48.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.023+0000 7f37cd000640 1 -- 172.21.15.40:0/1340301570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37c80694f0 con 0x7f37c80b4140 2024-04-26T23:07:48.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.025+0000 7f37be000640 1 -- 172.21.15.40:0/1340301570 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f37b0007400 con 0x7f37c80b4140 2024-04-26T23:07:48.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.026+0000 7f37be000640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f37a406c8f0 0x7f37a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:48.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.026+0000 7f37be000640 1 -- 172.21.15.40:0/1340301570 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6001+0+0 (secure 0 0 0) 0x7f37b008b180 con 0x7f37c80b4140 2024-04-26T23:07:48.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.027+0000 7f37c5e00640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f37a406c8f0 0x7f37a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:48.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.028+0000 7f37c5e00640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f37a406c8f0 0x7f37a406edb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f37b8034000 tx=0x7f37b802f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:48.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.031+0000 7f37be000640 1 -- 172.21.15.40:0/1340301570 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f37c80ce110 con 0x7f37c80b4140 2024-04-26T23:07:48.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.277+0000 7f37cd000640 1 -- 172.21.15.40:0/1340301570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f37c80b8d80 con 0x7f37c80b4140 2024-04-26T23:07:48.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:07:48.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:07:48.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.801+0000 7f37be000640 1 -- 172.21.15.40:0/1340301570 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v87) ==== 174+0+0 (secure 0 0 0) 0x7f37b0059480 con 0x7f37c80b4140 2024-04-26T23:07:48.804 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 13 and data pool 14 2024-04-26T23:07:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.820+0000 7f37cd000640 1 -- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f37a406c8f0 msgr2=0x7f37a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.820+0000 7f37cd000640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f37a406c8f0 0x7f37a406edb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f37b8034000 tx=0x7f37b802f000 comp rx=0 tx=0).stop 2024-04-26T23:07:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.820+0000 7f37cd000640 1 -- 172.21.15.40:0/1340301570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37c80b4140 msgr2=0x7f37c815f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.820+0000 7f37cd000640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37c80b4140 0x7f37c815f3f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f37b000c3c0 tx=0x7f37b000c880 comp rx=0 tx=0).stop 2024-04-26T23:07:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.821+0000 7f37cd000640 1 -- 172.21.15.40:0/1340301570 shutdown_connections 2024-04-26T23:07:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.821+0000 7f37cd000640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f37a406c8f0 0x7f37a406edb0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.821+0000 7f37cd000640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c80d0d00 0x7f37c80cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.821+0000 7f37cd000640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37c80b4140 0x7f37c815f3f0 secure :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f37b000c3c0 tx=0x7f37b000c880 comp rx=0 tx=0).stop 2024-04-26T23:07:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.821+0000 7f37cd000640 1 --2- 172.21.15.40:0/1340301570 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f37c806e3f0 0x7f37c80d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.821+0000 7f37cd000640 1 -- 172.21.15.40:0/1340301570 >> 172.21.15.40:0/1340301570 conn(0x7f37c80d2e00 msgr2=0x7f37c806d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.822+0000 7f37cd000640 1 -- 172.21.15.40:0/1340301570 shutdown_connections 2024-04-26T23:07:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:48.822+0000 7f37cd000640 1 -- 172.21.15.40:0/1340301570 wait complete. 2024-04-26T23:07:48.855 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:07:49.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.030+0000 7fe29b200640 1 Processor -- start 2024-04-26T23:07:49.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.030+0000 7fe29b200640 1 -- start start 2024-04-26T23:07:49.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.031+0000 7fe29b200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe29c15ce60 0x7fe29c0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.031+0000 7fe29b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe29c071fe0 0x7fe29c073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.032+0000 7fe29b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe29c0d0820 con 0x7fe29c0af7a0 2024-04-26T23:07:49.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.032+0000 7fe29b200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe29c0cc450 con 0x7fe29c15ce60 2024-04-26T23:07:49.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.032+0000 7fe29b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe29c1614c0 con 0x7fe29c071fe0 2024-04-26T23:07:49.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.032+0000 7fe299e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe29c15ce60 0x7fe29c0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.032+0000 7fe299e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe29c15ce60 0x7fe29c0afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44476/0 (socket says 172.21.15.40:44476) 2024-04-26T23:07:49.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.032+0000 7fe299e00640 1 -- 172.21.15.40:0/2420946501 learned_addr learned my addr 172.21.15.40:0/2420946501 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:49.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.032+0000 7fe29a800640 1 --2- 172.21.15.40:0/2420946501 >> v2:172.21.15.103:3300/0 conn(0x7fe29c071fe0 0x7fe29c073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.032+0000 7fe299e00640 1 -- 172.21.15.40:0/2420946501 >> v2:172.21.15.103:3300/0 conn(0x7fe29c071fe0 msgr2=0x7fe29c073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:49.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.032+0000 7fe299e00640 1 --2- 172.21.15.40:0/2420946501 >> v2:172.21.15.103:3300/0 conn(0x7fe29c071fe0 0x7fe29c073760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.032+0000 7fe299e00640 1 -- 172.21.15.40:0/2420946501 >> v1:172.21.15.103:6789/0 conn(0x7fe29c0af7a0 legacy=0x7fe29c0ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:49.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.032+0000 7fe299e00640 1 -- 172.21.15.40:0/2420946501 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c161760 con 0x7fe29c15ce60 2024-04-26T23:07:49.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.033+0000 7fe299e00640 1 --2- 172.21.15.40:0/2420946501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe29c15ce60 0x7fe29c0afc60 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fe29c0d1af0 tx=0x7fe28c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ce5e62bdf45906 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:49.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.034+0000 7fe299400640 1 -- 172.21.15.40:0/2420946501 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c037070 con 0x7fe29c15ce60 2024-04-26T23:07:49.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.034+0000 7fe299400640 1 -- 172.21.15.40:0/2420946501 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe28c032440 con 0x7fe29c15ce60 2024-04-26T23:07:49.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.034+0000 7fe299400640 1 -- 172.21.15.40:0/2420946501 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c032810 con 0x7fe29c15ce60 2024-04-26T23:07:49.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.034+0000 7fe29b200640 1 -- 172.21.15.40:0/2420946501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe29c15ce60 msgr2=0x7fe29c0afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.034+0000 7fe29b200640 1 --2- 172.21.15.40:0/2420946501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe29c15ce60 0x7fe29c0afc60 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fe29c0d1af0 tx=0x7fe28c02a1a0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.035+0000 7fe29b200640 1 -- 172.21.15.40:0/2420946501 shutdown_connections 2024-04-26T23:07:49.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.035+0000 7fe29b200640 1 --2- 172.21.15.40:0/2420946501 >> v2:172.21.15.103:3300/0 conn(0x7fe29c071fe0 0x7fe29c073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.035+0000 7fe29b200640 1 --2- 172.21.15.40:0/2420946501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe29c15ce60 0x7fe29c0afc60 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.035+0000 7fe29b200640 1 -- 172.21.15.40:0/2420946501 >> 172.21.15.40:0/2420946501 conn(0x7fe29c0d0a10 msgr2=0x7fe29c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:49.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.035+0000 7fe29b200640 1 -- 172.21.15.40:0/2420946501 shutdown_connections 2024-04-26T23:07:49.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.035+0000 7fe29b200640 1 -- 172.21.15.40:0/2420946501 wait complete. 2024-04-26T23:07:49.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.036+0000 7fe29b200640 1 Processor -- start 2024-04-26T23:07:49.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.037+0000 7fe29b200640 1 -- start start 2024-04-26T23:07:49.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.037+0000 7fe29b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe29c071fe0 0x7fe29c0ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.037+0000 7fe29b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe29c0af7a0 0x7fe29c14e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.038+0000 7fe29b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe29c15ce60 0x7fe29c0b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.038+0000 7fe29b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c0cc1f0 con 0x7fe29c0af7a0 2024-04-26T23:07:49.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.038+0000 7fe29b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c0cc070 con 0x7fe29c071fe0 2024-04-26T23:07:49.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.038+0000 7fe29b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe29c073ca0 con 0x7fe29c15ce60 2024-04-26T23:07:49.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.038+0000 7fe293e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe29c0af7a0 0x7fe29c14e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.038+0000 7fe293e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe29c0af7a0 0x7fe29c14e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44484/0 (socket says 172.21.15.40:44484) 2024-04-26T23:07:49.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.038+0000 7fe293e00640 1 -- 172.21.15.40:0/2848784858 learned_addr learned my addr 172.21.15.40:0/2848784858 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:49.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.038+0000 7fe299e00640 1 --2- 172.21.15.40:0/2848784858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe29c071fe0 0x7fe29c0ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.038+0000 7fe293e00640 1 -- 172.21.15.40:0/2848784858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe29c15ce60 msgr2=0x7fe29c0b4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.038+0000 7fe293e00640 1 --2- 172.21.15.40:0/2848784858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe29c15ce60 0x7fe29c0b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.038+0000 7fe293e00640 1 -- 172.21.15.40:0/2848784858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe29c071fe0 msgr2=0x7fe29c0ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.038+0000 7fe293e00640 1 --2- 172.21.15.40:0/2848784858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe29c071fe0 0x7fe29c0ca3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.038+0000 7fe293e00640 1 -- 172.21.15.40:0/2848784858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c0b5c00 con 0x7fe29c0af7a0 2024-04-26T23:07:49.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.039+0000 7fe293e00640 1 --2- 172.21.15.40:0/2848784858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe29c0af7a0 0x7fe29c14e270 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fe28400c410 tx=0x7fe28400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:49.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.039+0000 7fe292000640 1 -- 172.21.15.40:0/2848784858 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284017070 con 0x7fe29c0af7a0 2024-04-26T23:07:49.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.039+0000 7fe292000640 1 -- 172.21.15.40:0/2848784858 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2840035f0 con 0x7fe29c0af7a0 2024-04-26T23:07:49.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.039+0000 7fe292000640 1 -- 172.21.15.40:0/2848784858 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284003a20 con 0x7fe29c0af7a0 2024-04-26T23:07:49.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.039+0000 7fe29b200640 1 -- 172.21.15.40:0/2848784858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29c0b5ee0 con 0x7fe29c0af7a0 2024-04-26T23:07:49.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.039+0000 7fe29b200640 1 -- 172.21.15.40:0/2848784858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe29c0b64a0 con 0x7fe29c0af7a0 2024-04-26T23:07:49.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.040+0000 7fe29b200640 1 -- 172.21.15.40:0/2848784858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe29c0c8270 con 0x7fe29c0af7a0 2024-04-26T23:07:49.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.041+0000 7fe292000640 1 -- 172.21.15.40:0/2848784858 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe284003bc0 con 0x7fe29c0af7a0 2024-04-26T23:07:49.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.043+0000 7fe292000640 1 --2- 172.21.15.40:0/2848784858 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe27806c920 0x7fe27806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.043+0000 7fe292000640 1 -- 172.21.15.40:0/2848784858 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6101+0+0 (secure 0 0 0) 0x7fe28408aff0 con 0x7fe29c0af7a0 2024-04-26T23:07:49.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.043+0000 7fe299e00640 1 --2- 172.21.15.40:0/2848784858 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe27806c920 0x7fe27806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.045+0000 7fe299e00640 1 --2- 172.21.15.40:0/2848784858 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe27806c920 0x7fe27806ede0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe28c034000 tx=0x7fe28c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:49.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.047+0000 7fe292000640 1 -- 172.21.15.40:0/2848784858 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe29c0c8270 con 0x7fe29c0af7a0 2024-04-26T23:07:49.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.257+0000 7fe29b200640 1 -- 172.21.15.40:0/2848784858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe29c069370 con 0x7fe29c0af7a0 2024-04-26T23:07:49.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.260+0000 7fe292000640 1 -- 172.21.15.40:0/2848784858 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+16862 (secure 0 0 0) 0x7fe284059250 con 0x7fe29c0af7a0 2024-04-26T23:07:49.260 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:49.260 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":89,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:07:48.279120+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:07:46.137238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-26T23:07:47.258518+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:07:49.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.262+0000 7fe29b200640 1 -- 172.21.15.40:0/2848784858 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe27806c920 msgr2=0x7fe27806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.262+0000 7fe29b200640 1 --2- 172.21.15.40:0/2848784858 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe27806c920 0x7fe27806ede0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe28c034000 tx=0x7fe28c02e000 comp rx=0 tx=0).stop 2024-04-26T23:07:49.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.262+0000 7fe29b200640 1 -- 172.21.15.40:0/2848784858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe29c0af7a0 msgr2=0x7fe29c14e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.262+0000 7fe29b200640 1 --2- 172.21.15.40:0/2848784858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe29c0af7a0 0x7fe29c14e270 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fe28400c410 tx=0x7fe28400c8d0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.263+0000 7fe29b200640 1 -- 172.21.15.40:0/2848784858 shutdown_connections 2024-04-26T23:07:49.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.263+0000 7fe29b200640 1 --2- 172.21.15.40:0/2848784858 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe27806c920 0x7fe27806ede0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.263+0000 7fe29b200640 1 --2- 172.21.15.40:0/2848784858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe29c15ce60 0x7fe29c0b4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.263+0000 7fe29b200640 1 --2- 172.21.15.40:0/2848784858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe29c0af7a0 0x7fe29c14e270 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.263+0000 7fe29b200640 1 --2- 172.21.15.40:0/2848784858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe29c071fe0 0x7fe29c0ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.263+0000 7fe29b200640 1 -- 172.21.15.40:0/2848784858 >> 172.21.15.40:0/2848784858 conn(0x7fe29c0d0a10 msgr2=0x7fe29c1581f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:49.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.263+0000 7fe29b200640 1 -- 172.21.15.40:0/2848784858 shutdown_connections 2024-04-26T23:07:49.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.263+0000 7fe29b200640 1 -- 172.21.15.40:0/2848784858 wait complete. 2024-04-26T23:07:49.282 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:07:49.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.486+0000 7fa913000640 1 Processor -- start 2024-04-26T23:07:49.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.486+0000 7fa913000640 1 -- start start 2024-04-26T23:07:49.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.487+0000 7fa913000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa90c163940 0x7fa90c071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.487+0000 7fa913000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90c06d690 con 0x7fa90c0cfc70 2024-04-26T23:07:49.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.487+0000 7fa913000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90c06d970 con 0x7fa90c1634e0 2024-04-26T23:07:49.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.487+0000 7fa913000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa90c06dd60 con 0x7fa90c163940 2024-04-26T23:07:49.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.488+0000 7fa911c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa90c0cfc70 0x7fa90c162dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33434/0 (socket says 172.21.15.40:33434) 2024-04-26T23:07:49.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.488+0000 7fa911c00640 1 -- 172.21.15.40:0/3344347358 learned_addr learned my addr 172.21.15.40:0/3344347358 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:49.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.488+0000 7fa912600640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa90c163940 0x7fa90c071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.488+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2649909947 0 0) 0x7fa90c06d690 con 0x7fa90c0cfc70 2024-04-26T23:07:49.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.488+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8f0003520 con 0x7fa90c0cfc70 2024-04-26T23:07:49.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 53649361 0 0) 0x7fa8f0003520 con 0x7fa90c0cfc70 2024-04-26T23:07:49.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa90c163940 msgr2=0x7fa90c071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:49.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa90be00640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa90c163940 0x7fa90c071620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 >> v1:172.21.15.103:6789/0 conn(0x7fa90c1634e0 legacy=0x7fa90c0d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c06e4d0 con 0x7fa90c0cfc70 2024-04-26T23:07:49.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fa8fc0019f0 con 0x7fa90c0cfc70 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 >> v1:172.21.15.40:6789/0 conn(0x7fa90c0cfc70 legacy=0x7fa90c162dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa90be00640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8f00039e0 0x7fa8f0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa90be00640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa90c163940 0x7fa8f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa90be00640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8f0004d40 0x7fa8f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa90c06dd60 con 0x7fa8f00039e0 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f000bc70 con 0x7fa8f0004d40 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa8f000be70 con 0x7fa90c163940 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.489+0000 7fa911c00640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8f00039e0 0x7fa8f0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.490+0000 7fa912600640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8f0004d40 0x7fa8f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.490+0000 7fa911c00640 1 -- 172.21.15.40:0/3344347358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa90c163940 msgr2=0x7fa8f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.490+0000 7fa911200640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa90c163940 0x7fa8f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.490+0000 7fa911c00640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa90c163940 0x7fa8f0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.490+0000 7fa911c00640 1 -- 172.21.15.40:0/3344347358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8f0004d40 msgr2=0x7fa8f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:49.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.490+0000 7fa911c00640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8f0004d40 0x7fa8f00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.490+0000 7fa911c00640 1 -- 172.21.15.40:0/3344347358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f0003520 con 0x7fa8f00039e0 2024-04-26T23:07:49.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.490+0000 7fa911200640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa90c163940 0x7fa8f0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:49.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.490+0000 7fa912600640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8f0004d40 0x7fa8f00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:49.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.491+0000 7fa911c00640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8f00039e0 0x7fa8f0003e00 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00a7b0 tx=0x7fa8f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e62149a5909beb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:49.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.491+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc00a3d0 con 0x7fa8f00039e0 2024-04-26T23:07:49.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.491+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8fc02f070 con 0x7fa8f00039e0 2024-04-26T23:07:49.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.491+0000 7fa90be00640 1 -- 172.21.15.40:0/3344347358 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc02bc40 con 0x7fa8f00039e0 2024-04-26T23:07:49.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.492+0000 7fa913000640 1 -- 172.21.15.40:0/3344347358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8f00039e0 msgr2=0x7fa8f0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.492+0000 7fa913000640 1 --2- 172.21.15.40:0/3344347358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8f00039e0 0x7fa8f0003e00 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00a7b0 tx=0x7fa8f0005830 comp rx=0 tx=0).stop 2024-04-26T23:07:49.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.492+0000 7fa912600640 1 -- 172.21.15.40:0/3344347358 reap_dead start 2024-04-26T23:07:49.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.492+0000 7fa913000640 1 -- 172.21.15.40:0/3344347358 shutdown_connections 2024-04-26T23:07:49.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.492+0000 7fa913000640 1 -- 172.21.15.40:0/3344347358 >> 172.21.15.40:0/3344347358 conn(0x7fa90c0d2c10 msgr2=0x7fa90c0d3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:49.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.492+0000 7fa913000640 1 -- 172.21.15.40:0/3344347358 shutdown_connections 2024-04-26T23:07:49.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.492+0000 7fa913000640 1 -- 172.21.15.40:0/3344347358 wait complete. 2024-04-26T23:07:49.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.493+0000 7fa913000640 1 Processor -- start 2024-04-26T23:07:49.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.494+0000 7fa913000640 1 -- start start 2024-04-26T23:07:49.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa913000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8f00039e0 0x7fa90c15de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa913000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa90c0cbca0 0x7fa90c0b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa913000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa90c0b4f00 0x7fa90c0b5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa913000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa90c071620 con 0x7fa90c0b4f00 2024-04-26T23:07:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa913000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa90c06e040 con 0x7fa8f00039e0 2024-04-26T23:07:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa913000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa90c0717a0 con 0x7fa90c0cbca0 2024-04-26T23:07:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa912600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa90c0b4f00 0x7fa90c0b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa912600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa90c0b4f00 0x7fa90c0b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44494/0 (socket says 172.21.15.40:44494) 2024-04-26T23:07:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa912600640 1 -- 172.21.15.40:0/1726376687 learned_addr learned my addr 172.21.15.40:0/1726376687 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa911c00640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8f00039e0 0x7fa90c15de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa912600640 1 -- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa90c0cbca0 msgr2=0x7fa90c0b7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa912600640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa90c0cbca0 0x7fa90c0b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa912600640 1 -- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8f00039e0 msgr2=0x7fa90c15de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa912600640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8f00039e0 0x7fa90c15de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa912600640 1 -- 172.21.15.40:0/1726376687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c0b7680 con 0x7fa90c0b4f00 2024-04-26T23:07:49.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.495+0000 7fa911c00640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8f00039e0 0x7fa90c15de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:49.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.496+0000 7fa912600640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa90c0b4f00 0x7fa90c0b5380 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fa904006fd0 tx=0x7fa90400e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:49.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.496+0000 7fa909600640 1 -- 172.21.15.40:0/1726376687 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa904017020 con 0x7fa90c0b4f00 2024-04-26T23:07:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.496+0000 7fa909600640 1 -- 172.21.15.40:0/1726376687 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa904013630 con 0x7fa90c0b4f00 2024-04-26T23:07:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.496+0000 7fa913000640 1 -- 172.21.15.40:0/1726376687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa90c0cc260 con 0x7fa90c0b4f00 2024-04-26T23:07:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.496+0000 7fa909600640 1 -- 172.21.15.40:0/1726376687 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90400f040 con 0x7fa90c0b4f00 2024-04-26T23:07:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.496+0000 7fa913000640 1 -- 172.21.15.40:0/1726376687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa90c0cc820 con 0x7fa90c0b4f00 2024-04-26T23:07:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.497+0000 7fa913000640 1 -- 172.21.15.40:0/1726376687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa90c069a80 con 0x7fa90c0b4f00 2024-04-26T23:07:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.498+0000 7fa909600640 1 -- 172.21.15.40:0/1726376687 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa90401a020 con 0x7fa90c0b4f00 2024-04-26T23:07:49.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.499+0000 7fa909600640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa8ec06c8f0 0x7fa8ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.499+0000 7fa909600640 1 -- 172.21.15.40:0/1726376687 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6101+0+0 (secure 0 0 0) 0x7fa90408b660 con 0x7fa90c0b4f00 2024-04-26T23:07:49.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.500+0000 7fa911c00640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa8ec06c8f0 0x7fa8ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.501+0000 7fa911c00640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa8ec06c8f0 0x7fa8ec06edb0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00a780 tx=0x7fa8fc009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:49.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.504+0000 7fa909600640 1 -- 172.21.15.40:0/1726376687 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa904021020 con 0x7fa90c0b4f00 2024-04-26T23:07:49.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.700+0000 7fa913000640 1 -- 172.21.15.40:0/1726376687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa90c0b8b90 con 0x7fa90c0b4f00 2024-04-26T23:07:49.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.702+0000 7fa909600640 1 -- 172.21.15.40:0/1726376687 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+16862 (secure 0 0 0) 0x7fa9040598f0 con 0x7fa90c0b4f00 2024-04-26T23:07:49.703 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:49.703 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":89,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:07:48.279120+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:07:46.137238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-26T23:07:47.258518+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:07:49.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.705+0000 7fa913000640 1 -- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa8ec06c8f0 msgr2=0x7fa8ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.705+0000 7fa913000640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa8ec06c8f0 0x7fa8ec06edb0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00a780 tx=0x7fa8fc009030 comp rx=0 tx=0).stop 2024-04-26T23:07:49.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.705+0000 7fa913000640 1 -- 172.21.15.40:0/1726376687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa90c0b4f00 msgr2=0x7fa90c0b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.705+0000 7fa913000640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa90c0b4f00 0x7fa90c0b5380 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fa904006fd0 tx=0x7fa90400e8b0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.705+0000 7fa913000640 1 -- 172.21.15.40:0/1726376687 shutdown_connections 2024-04-26T23:07:49.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.705+0000 7fa913000640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa8ec06c8f0 0x7fa8ec06edb0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.705+0000 7fa913000640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa90c0b4f00 0x7fa90c0b5380 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.705+0000 7fa913000640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa90c0cbca0 0x7fa90c0b7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.705+0000 7fa913000640 1 --2- 172.21.15.40:0/1726376687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8f00039e0 0x7fa90c15de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.705+0000 7fa913000640 1 -- 172.21.15.40:0/1726376687 >> 172.21.15.40:0/1726376687 conn(0x7fa90c0d2c10 msgr2=0x7fa90c161ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:49.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.705+0000 7fa913000640 1 -- 172.21.15.40:0/1726376687 shutdown_connections 2024-04-26T23:07:49.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.705+0000 7fa913000640 1 -- 172.21.15.40:0/1726376687 wait complete. 2024-04-26T23:07:49.725 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T23:07:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.931+0000 7ff8f6c00640 1 Processor -- start 2024-04-26T23:07:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.931+0000 7ff8f6c00640 1 -- start start 2024-04-26T23:07:49.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.932+0000 7ff8f6c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8f0160a50 0x7ff8f0073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.933+0000 7ff8f6c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8f0161880 con 0x7ff8f00cfaf0 2024-04-26T23:07:49.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.933+0000 7ff8f6c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8f00cff80 con 0x7ff8f006dd40 2024-04-26T23:07:49.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.933+0000 7ff8f6c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff8f0071ce0 con 0x7ff8f0160a50 2024-04-26T23:07:49.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.933+0000 7ff8f4e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff8f00cfaf0 0x7ff8f0160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33448/0 (socket says 172.21.15.40:33448) 2024-04-26T23:07:49.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.933+0000 7ff8f4e00640 1 -- 172.21.15.40:0/2629661353 learned_addr learned my addr 172.21.15.40:0/2629661353 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:49.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.933+0000 7ff8f5800640 1 --2- 172.21.15.40:0/2629661353 >> v2:172.21.15.40:3300/0 conn(0x7ff8f0160a50 0x7ff8f0073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.934+0000 7ff8f5800640 1 -- 172.21.15.40:0/2629661353 >> v1:172.21.15.103:6789/0 conn(0x7ff8f006dd40 legacy=0x7ff8f0072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:49.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.934+0000 7ff8f5800640 1 -- 172.21.15.40:0/2629661353 >> v1:172.21.15.40:6789/0 conn(0x7ff8f00cfaf0 legacy=0x7ff8f0160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.934+0000 7ff8f5800640 1 -- 172.21.15.40:0/2629661353 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f00d0260 con 0x7ff8f0160a50 2024-04-26T23:07:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.935+0000 7ff8f5800640 1 --2- 172.21.15.40:0/2629661353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f0160a50 0x7ff8f0073a50 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40097a0 tx=0x7ff8e402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3223904ab1583eb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.935+0000 7ff8efe00640 1 -- 172.21.15.40:0/2629661353 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4037070 con 0x7ff8f0160a50 2024-04-26T23:07:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.935+0000 7ff8efe00640 1 -- 172.21.15.40:0/2629661353 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8e4032440 con 0x7ff8f0160a50 2024-04-26T23:07:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.935+0000 7ff8efe00640 1 -- 172.21.15.40:0/2629661353 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4032810 con 0x7ff8f0160a50 2024-04-26T23:07:49.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.936+0000 7ff8f6c00640 1 -- 172.21.15.40:0/2629661353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f0160a50 msgr2=0x7ff8f0073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.936+0000 7ff8f6c00640 1 --2- 172.21.15.40:0/2629661353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f0160a50 0x7ff8f0073a50 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40097a0 tx=0x7ff8e402a1e0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.936+0000 7ff8f6c00640 1 -- 172.21.15.40:0/2629661353 shutdown_connections 2024-04-26T23:07:49.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.936+0000 7ff8f6c00640 1 --2- 172.21.15.40:0/2629661353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f0160a50 0x7ff8f0073a50 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.936+0000 7ff8f6c00640 1 -- 172.21.15.40:0/2629661353 >> 172.21.15.40:0/2629661353 conn(0x7ff8f00b2720 msgr2=0x7ff8f00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:49.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.936+0000 7ff8f6c00640 1 -- 172.21.15.40:0/2629661353 shutdown_connections 2024-04-26T23:07:49.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.936+0000 7ff8f6c00640 1 -- 172.21.15.40:0/2629661353 wait complete. 2024-04-26T23:07:49.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.938+0000 7ff8f6c00640 1 Processor -- start 2024-04-26T23:07:49.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.938+0000 7ff8f6c00640 1 -- start start 2024-04-26T23:07:49.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.939+0000 7ff8f6c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f006dd40 0x7ff8f01f81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.939+0000 7ff8f6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8f0160a50 0x7ff8f015f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.939+0000 7ff8f6c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f00af2d0 0x7ff8f015f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.939+0000 7ff8f6c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f0073f90 con 0x7ff8f00af2d0 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.939+0000 7ff8f6c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f00722a0 con 0x7ff8f006dd40 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.939+0000 7ff8f6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff8f006e1f0 con 0x7ff8f0160a50 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.939+0000 7ff8f6200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f00af2d0 0x7ff8f015f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.939+0000 7ff8f6200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f00af2d0 0x7ff8f015f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44518/0 (socket says 172.21.15.40:44518) 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.939+0000 7ff8f6200640 1 -- 172.21.15.40:0/902548907 learned_addr learned my addr 172.21.15.40:0/902548907 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.939+0000 7ff8f5800640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f006dd40 0x7ff8f01f81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.940+0000 7ff8f6200640 1 -- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8f0160a50 msgr2=0x7ff8f015f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.940+0000 7ff8f6200640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8f0160a50 0x7ff8f015f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.940+0000 7ff8f6200640 1 -- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f006dd40 msgr2=0x7ff8f01f81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.940+0000 7ff8f6200640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f006dd40 0x7ff8f01f81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.940+0000 7ff8f6200640 1 -- 172.21.15.40:0/902548907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f00af7d0 con 0x7ff8f00af2d0 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.940+0000 7ff8f5800640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f006dd40 0x7ff8f01f81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:49.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.940+0000 7ff8f6200640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f00af2d0 0x7ff8f015f610 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800c780 tx=0x7ff8e800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:49.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.940+0000 7ff8ed600640 1 -- 172.21.15.40:0/902548907 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8004b60 con 0x7ff8f00af2d0 2024-04-26T23:07:49.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.940+0000 7ff8ed600640 1 -- 172.21.15.40:0/902548907 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8e8004d00 con 0x7ff8f00af2d0 2024-04-26T23:07:49.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.940+0000 7ff8ed600640 1 -- 172.21.15.40:0/902548907 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e800aa90 con 0x7ff8f00af2d0 2024-04-26T23:07:49.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.940+0000 7ff8f6c00640 1 -- 172.21.15.40:0/902548907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f00afab0 con 0x7ff8f00af2d0 2024-04-26T23:07:49.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.940+0000 7ff8f6c00640 1 -- 172.21.15.40:0/902548907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f00b0070 con 0x7ff8f00af2d0 2024-04-26T23:07:49.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.942+0000 7ff8f6c00640 1 -- 172.21.15.40:0/902548907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8f0069370 con 0x7ff8f00af2d0 2024-04-26T23:07:49.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.943+0000 7ff8ed600640 1 -- 172.21.15.40:0/902548907 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff8e800ac30 con 0x7ff8f00af2d0 2024-04-26T23:07:49.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.944+0000 7ff8ed600640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8d006c8f0 0x7ff8d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:49.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.944+0000 7ff8ed600640 1 -- 172.21.15.40:0/902548907 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7ff8e808c0c0 con 0x7ff8f00af2d0 2024-04-26T23:07:49.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.944+0000 7ff8f5800640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8d006c8f0 0x7ff8d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:49.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.946+0000 7ff8f5800640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8d006c8f0 0x7ff8d006edb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4034000 tx=0x7ff8e402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:49.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:49.948+0000 7ff8ed600640 1 -- 172.21.15.40:0/902548907 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff8e8020070 con 0x7ff8f00af2d0 2024-04-26T23:07:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.194+0000 7ff8f6c00640 1 -- 172.21.15.40:0/902548907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff8f00d1860 con 0x7ff8f00af2d0 2024-04-26T23:07:50.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.814+0000 7ff8ed600640 1 -- 172.21.15.40:0/902548907 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v90) ==== 118+0+0 (secure 0 0 0) 0x7ff8e805a230 con 0x7ff8f00af2d0 2024-04-26T23:07:50.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.818+0000 7ff8f6c00640 1 -- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8d006c8f0 msgr2=0x7ff8d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:50.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.818+0000 7ff8f6c00640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8d006c8f0 0x7ff8d006edb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4034000 tx=0x7ff8e402e000 comp rx=0 tx=0).stop 2024-04-26T23:07:50.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.818+0000 7ff8f6c00640 1 -- 172.21.15.40:0/902548907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f00af2d0 msgr2=0x7ff8f015f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:50.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.818+0000 7ff8f6c00640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f00af2d0 0x7ff8f015f610 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800c780 tx=0x7ff8e800cc40 comp rx=0 tx=0).stop 2024-04-26T23:07:50.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.818+0000 7ff8f6c00640 1 -- 172.21.15.40:0/902548907 shutdown_connections 2024-04-26T23:07:50.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.818+0000 7ff8f6c00640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff8d006c8f0 0x7ff8d006edb0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:50.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.818+0000 7ff8f6c00640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f00af2d0 0x7ff8f015f610 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:50.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.818+0000 7ff8f6c00640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff8f0160a50 0x7ff8f015f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:50.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.818+0000 7ff8f6c00640 1 --2- 172.21.15.40:0/902548907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f006dd40 0x7ff8f01f81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:50.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.818+0000 7ff8f6c00640 1 -- 172.21.15.40:0/902548907 >> 172.21.15.40:0/902548907 conn(0x7ff8f00b2720 msgr2=0x7ff8f015b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:50.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.818+0000 7ff8f6c00640 1 -- 172.21.15.40:0/902548907 shutdown_connections 2024-04-26T23:07:50.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:50.819+0000 7ff8f6c00640 1 -- 172.21.15.40:0/902548907 wait complete. 2024-04-26T23:07:50.841 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-26T23:07:50.842 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-26T23:07:51.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.060+0000 7f07d0c00640 1 Processor -- start 2024-04-26T23:07:51.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.060+0000 7f07d0c00640 1 -- start start 2024-04-26T23:07:51.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.062+0000 7f07d0c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f07cc0b5120 0x7f07cc0b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:51.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.062+0000 7f07d0c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07cc0b59b0 0x7f07cc0b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:51.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.062+0000 7f07d0c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07cc0d3dc0 con 0x7f07cc0b5500 2024-04-26T23:07:51.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.062+0000 7f07d0c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f07cc0d03b0 con 0x7f07cc0b5120 2024-04-26T23:07:51.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.062+0000 7f07d0c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f07cc0b75b0 con 0x7f07cc0b59b0 2024-04-26T23:07:51.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.062+0000 7f07cb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f07cc0b5120 0x7f07cc0b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:51.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.062+0000 7f07cb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f07cc0b5120 0x7f07cc0b5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44534/0 (socket says 172.21.15.40:44534) 2024-04-26T23:07:51.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.062+0000 7f07cb400640 1 -- 172.21.15.40:0/13587845 learned_addr learned my addr 172.21.15.40:0/13587845 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:51.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.063+0000 7f07cb400640 1 -- 172.21.15.40:0/13587845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07cc0b59b0 msgr2=0x7f07cc0b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:51.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.063+0000 7f07cb400640 1 --2- 172.21.15.40:0/13587845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07cc0b59b0 0x7f07cc0b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:51.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.063+0000 7f07cb400640 1 -- 172.21.15.40:0/13587845 >> v1:172.21.15.40:6789/0 conn(0x7f07cc0b5500 legacy=0x7f07cc0b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:51.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.063+0000 7f07cb400640 1 -- 172.21.15.40:0/13587845 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07cc0d0600 con 0x7f07cc0b5120 2024-04-26T23:07:51.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.064+0000 7f07cb400640 1 --2- 172.21.15.40:0/13587845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc0b5120 0x7f07cc0b5ee0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f07bc009e80 tx=0x7f07bc02a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537f1cadae6c6ee5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:51.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.064+0000 7f07ca000640 1 -- 172.21.15.40:0/13587845 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07bc036070 con 0x7f07cc0b5120 2024-04-26T23:07:51.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.064+0000 7f07ca000640 1 -- 172.21.15.40:0/13587845 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07bc031600 con 0x7f07cc0b5120 2024-04-26T23:07:51.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.064+0000 7f07ca000640 1 -- 172.21.15.40:0/13587845 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07bc0319d0 con 0x7f07cc0b5120 2024-04-26T23:07:51.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.065+0000 7f07d0c00640 1 -- 172.21.15.40:0/13587845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc0b5120 msgr2=0x7f07cc0b5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:51.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.065+0000 7f07d0c00640 1 --2- 172.21.15.40:0/13587845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc0b5120 0x7f07cc0b5ee0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f07bc009e80 tx=0x7f07bc02a630 comp rx=0 tx=0).stop 2024-04-26T23:07:51.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.065+0000 7f07d0c00640 1 -- 172.21.15.40:0/13587845 shutdown_connections 2024-04-26T23:07:51.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.065+0000 7f07d0c00640 1 --2- 172.21.15.40:0/13587845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07cc0b59b0 0x7f07cc0b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:51.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.065+0000 7f07d0c00640 1 --2- 172.21.15.40:0/13587845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc0b5120 0x7f07cc0b5ee0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:51.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.065+0000 7f07d0c00640 1 -- 172.21.15.40:0/13587845 >> 172.21.15.40:0/13587845 conn(0x7f07cc0d4db0 msgr2=0x7f07cc0d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:51.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.065+0000 7f07d0c00640 1 -- 172.21.15.40:0/13587845 shutdown_connections 2024-04-26T23:07:51.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.065+0000 7f07d0c00640 1 -- 172.21.15.40:0/13587845 wait complete. 2024-04-26T23:07:51.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.067+0000 7f07d0c00640 1 Processor -- start 2024-04-26T23:07:51.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.067+0000 7f07d0c00640 1 -- start start 2024-04-26T23:07:51.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.068+0000 7f07d0c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07cc0b5120 0x7f07cc14cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:51.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.068+0000 7f07d0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc0b5500 0x7f07cc0ce310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:51.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.068+0000 7f07d0c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07cc0b59b0 0x7f07cc0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:51.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.068+0000 7f07d0c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc0b7c00 con 0x7f07cc0b5500 2024-04-26T23:07:51.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.068+0000 7f07d0c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc0d0bc0 con 0x7f07cc0b59b0 2024-04-26T23:07:51.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.068+0000 7f07d0c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f07cc0b7d80 con 0x7f07cc0b5120 2024-04-26T23:07:51.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.068+0000 7f07caa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc0b5500 0x7f07cc0ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:51.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.068+0000 7f07caa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc0b5500 0x7f07cc0ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44540/0 (socket says 172.21.15.40:44540) 2024-04-26T23:07:51.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.068+0000 7f07caa00640 1 -- 172.21.15.40:0/1047865783 learned_addr learned my addr 172.21.15.40:0/1047865783 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.069+0000 7f07caa00640 1 -- 172.21.15.40:0/1047865783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07cc0b5120 msgr2=0x7f07cc14cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.069+0000 7f07cbe00640 1 --2- 172.21.15.40:0/1047865783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07cc0b59b0 0x7f07cc0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.069+0000 7f07caa00640 1 --2- 172.21.15.40:0/1047865783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07cc0b5120 0x7f07cc14cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.069+0000 7f07caa00640 1 -- 172.21.15.40:0/1047865783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07cc0b59b0 msgr2=0x7f07cc0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.069+0000 7f07caa00640 1 --2- 172.21.15.40:0/1047865783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07cc0b59b0 0x7f07cc0ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:51.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.069+0000 7f07caa00640 1 -- 172.21.15.40:0/1047865783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07cc1503f0 con 0x7f07cc0b5500 2024-04-26T23:07:51.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.069+0000 7f07caa00640 1 --2- 172.21.15.40:0/1047865783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc0b5500 0x7f07cc0ce310 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f07b400c400 tx=0x7f07b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:51.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.069+0000 7f07c3400640 1 -- 172.21.15.40:0/1047865783 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b40052d0 con 0x7f07cc0b5500 2024-04-26T23:07:51.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.069+0000 7f07c3400640 1 -- 172.21.15.40:0/1047865783 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07b4005490 con 0x7f07cc0b5500 2024-04-26T23:07:51.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.069+0000 7f07c3400640 1 -- 172.21.15.40:0/1047865783 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b4007260 con 0x7f07cc0b5500 2024-04-26T23:07:51.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.069+0000 7f07d0c00640 1 -- 172.21.15.40:0/1047865783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07cc15cd30 con 0x7f07cc0b5500 2024-04-26T23:07:51.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.070+0000 7f07d0c00640 1 -- 172.21.15.40:0/1047865783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07cc15d2f0 con 0x7f07cc0b5500 2024-04-26T23:07:51.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.071+0000 7f07d0c00640 1 -- 172.21.15.40:0/1047865783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07cc069370 con 0x7f07cc0b5500 2024-04-26T23:07:51.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.072+0000 7f07c3400640 1 -- 172.21.15.40:0/1047865783 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f07b4007400 con 0x7f07cc0b5500 2024-04-26T23:07:51.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.073+0000 7f07c3400640 1 --2- 172.21.15.40:0/1047865783 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f07a806c8f0 0x7f07a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:51.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.073+0000 7f07c3400640 1 -- 172.21.15.40:0/1047865783 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f07b408b210 con 0x7f07cc0b5500 2024-04-26T23:07:51.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.074+0000 7f07cb400640 1 --2- 172.21.15.40:0/1047865783 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f07a806c8f0 0x7f07a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:51.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.075+0000 7f07cb400640 1 --2- 172.21.15.40:0/1047865783 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f07a806c8f0 0x7f07a806edb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f07bc033000 tx=0x7f07bc02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:51.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.079+0000 7f07c3400640 1 -- 172.21.15.40:0/1047865783 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f07b40594b0 con 0x7f07cc0b5500 2024-04-26T23:07:51.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.329+0000 7f07d0c00640 1 -- 172.21.15.40:0/1047865783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f07cc06cbc0 con 0x7f07cc0b5500 2024-04-26T23:07:51.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.824+0000 7f07c3400640 1 -- 172.21.15.40:0/1047865783 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v91) ==== 122+0+0 (secure 0 0 0) 0x7f07b4058db0 con 0x7f07cc0b5500 2024-04-26T23:07:51.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.828+0000 7f07d0c00640 1 -- 172.21.15.40:0/1047865783 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f07a806c8f0 msgr2=0x7f07a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:51.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.828+0000 7f07d0c00640 1 --2- 172.21.15.40:0/1047865783 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f07a806c8f0 0x7f07a806edb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f07bc033000 tx=0x7f07bc02e000 comp rx=0 tx=0).stop 2024-04-26T23:07:51.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.828+0000 7f07d0c00640 1 -- 172.21.15.40:0/1047865783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc0b5500 msgr2=0x7f07cc0ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:51.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.828+0000 7f07d0c00640 1 --2- 172.21.15.40:0/1047865783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc0b5500 0x7f07cc0ce310 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f07b400c400 tx=0x7f07b400c8c0 comp rx=0 tx=0).stop 2024-04-26T23:07:51.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.828+0000 7f07d0c00640 1 -- 172.21.15.40:0/1047865783 shutdown_connections 2024-04-26T23:07:51.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.828+0000 7f07d0c00640 1 --2- 172.21.15.40:0/1047865783 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f07a806c8f0 0x7f07a806edb0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:51.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.828+0000 7f07d0c00640 1 --2- 172.21.15.40:0/1047865783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07cc0b59b0 0x7f07cc0ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:51.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.828+0000 7f07d0c00640 1 --2- 172.21.15.40:0/1047865783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc0b5500 0x7f07cc0ce310 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:51.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.828+0000 7f07d0c00640 1 --2- 172.21.15.40:0/1047865783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07cc0b5120 0x7f07cc14cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:51.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.828+0000 7f07d0c00640 1 -- 172.21.15.40:0/1047865783 >> 172.21.15.40:0/1047865783 conn(0x7f07cc0d4db0 msgr2=0x7f07cc0d10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:51.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.829+0000 7f07d0c00640 1 -- 172.21.15.40:0/1047865783 shutdown_connections 2024-04-26T23:07:51.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:51.829+0000 7f07d0c00640 1 -- 172.21.15.40:0/1047865783 wait complete. 2024-04-26T23:07:51.852 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-26T23:07:51.853 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.064+0000 7f928f200640 1 Processor -- start 2024-04-26T23:07:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.064+0000 7f928f200640 1 -- start start 2024-04-26T23:07:52.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.065+0000 7f928f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9288163430 0x7f9288166f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:52.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.066+0000 7f928f200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f92881674a0 0x7f9288167950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:52.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.066+0000 7f928f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9288163810 con 0x7f92880c62c0 2024-04-26T23:07:52.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.066+0000 7f928f200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f9288167e90 con 0x7f92881674a0 2024-04-26T23:07:52.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.066+0000 7f928f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f92880cb400 con 0x7f9288163430 2024-04-26T23:07:52.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.066+0000 7f928d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f92881674a0 0x7f9288167950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:52.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.066+0000 7f928d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f92881674a0 0x7f9288167950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44544/0 (socket says 172.21.15.40:44544) 2024-04-26T23:07:52.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.066+0000 7f928d400640 1 -- 172.21.15.40:0/775504701 learned_addr learned my addr 172.21.15.40:0/775504701 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:52.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.066+0000 7f928de00640 1 --2- 172.21.15.40:0/775504701 >> v2:172.21.15.103:3300/0 conn(0x7f9288163430 0x7f9288166f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.067+0000 7f928d400640 1 -- 172.21.15.40:0/775504701 >> v2:172.21.15.103:3300/0 conn(0x7f9288163430 msgr2=0x7f9288166f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.067+0000 7f928d400640 1 --2- 172.21.15.40:0/775504701 >> v2:172.21.15.103:3300/0 conn(0x7f9288163430 0x7f9288166f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.067+0000 7f928d400640 1 -- 172.21.15.40:0/775504701 >> v1:172.21.15.103:6789/0 conn(0x7f92880c62c0 legacy=0x7f92880c66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.067+0000 7f928d400640 1 -- 172.21.15.40:0/775504701 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92880cb100 con 0x7f92881674a0 2024-04-26T23:07:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.067+0000 7f928de00640 1 --2- 172.21.15.40:0/775504701 >> v2:172.21.15.103:3300/0 conn(0x7f9288163430 0x7f9288166f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.068+0000 7f928d400640 1 --2- 172.21.15.40:0/775504701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92881674a0 0x7f9288167950 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f928806d670 tx=0x7f927802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b7a74fbdb438527 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.068+0000 7f928ca00640 1 -- 172.21.15.40:0/775504701 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9278037070 con 0x7f92881674a0 2024-04-26T23:07:52.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.068+0000 7f928ca00640 1 -- 172.21.15.40:0/775504701 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9278032440 con 0x7f92881674a0 2024-04-26T23:07:52.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.068+0000 7f928ca00640 1 -- 172.21.15.40:0/775504701 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9278032810 con 0x7f92881674a0 2024-04-26T23:07:52.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.069+0000 7f928f200640 1 -- 172.21.15.40:0/775504701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92881674a0 msgr2=0x7f9288167950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:52.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.069+0000 7f928f200640 1 --2- 172.21.15.40:0/775504701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92881674a0 0x7f9288167950 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f928806d670 tx=0x7f927802a230 comp rx=0 tx=0).stop 2024-04-26T23:07:52.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.069+0000 7f928f200640 1 -- 172.21.15.40:0/775504701 shutdown_connections 2024-04-26T23:07:52.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.069+0000 7f928f200640 1 --2- 172.21.15.40:0/775504701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92881674a0 0x7f9288167950 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.069+0000 7f928f200640 1 --2- 172.21.15.40:0/775504701 >> v2:172.21.15.103:3300/0 conn(0x7f9288163430 0x7f9288166f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.069+0000 7f928f200640 1 -- 172.21.15.40:0/775504701 >> 172.21.15.40:0/775504701 conn(0x7f92880d0a10 msgr2=0x7f92880d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:52.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.069+0000 7f928f200640 1 -- 172.21.15.40:0/775504701 shutdown_connections 2024-04-26T23:07:52.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.069+0000 7f928f200640 1 -- 172.21.15.40:0/775504701 wait complete. 2024-04-26T23:07:52.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.071+0000 7f928f200640 1 Processor -- start 2024-04-26T23:07:52.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.071+0000 7f928f200640 1 -- start start 2024-04-26T23:07:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.072+0000 7f928f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92880c62c0 0x7f9288165030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.072+0000 7f928f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9288163430 0x7f92880c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.072+0000 7f928f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92881674a0 0x7f92881f9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.072+0000 7f928f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f92880afe80 con 0x7f92880c62c0 2024-04-26T23:07:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.072+0000 7f928f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f92880ccc60 con 0x7f9288163430 2024-04-26T23:07:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.072+0000 7f928f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f92880cd1e0 con 0x7f92881674a0 2024-04-26T23:07:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.072+0000 7f928de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92880c62c0 0x7f9288165030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:52.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.073+0000 7f928de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92880c62c0 0x7f9288165030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44554/0 (socket says 172.21.15.40:44554) 2024-04-26T23:07:52.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.073+0000 7f928de00640 1 -- 172.21.15.40:0/2472203283 learned_addr learned my addr 172.21.15.40:0/2472203283 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:52.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.073+0000 7f928de00640 1 -- 172.21.15.40:0/2472203283 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92881674a0 msgr2=0x7f92881f9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:52.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.073+0000 7f928de00640 1 --2- 172.21.15.40:0/2472203283 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92881674a0 0x7f92881f9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.073+0000 7f928de00640 1 -- 172.21.15.40:0/2472203283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9288163430 msgr2=0x7f92880c84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:52.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.073+0000 7f928de00640 1 --2- 172.21.15.40:0/2472203283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9288163430 0x7f92880c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.073+0000 7f928de00640 1 -- 172.21.15.40:0/2472203283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92881f9d00 con 0x7f92880c62c0 2024-04-26T23:07:52.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.073+0000 7f928de00640 1 --2- 172.21.15.40:0/2472203283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92880c62c0 0x7f9288165030 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f927c00c7f0 tx=0x7f927c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.073+0000 7f9282000640 1 -- 172.21.15.40:0/2472203283 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927c015070 con 0x7f92880c62c0 2024-04-26T23:07:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.074+0000 7f9282000640 1 -- 172.21.15.40:0/2472203283 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f927c007a30 con 0x7f92880c62c0 2024-04-26T23:07:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.074+0000 7f9282000640 1 -- 172.21.15.40:0/2472203283 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927c007e00 con 0x7f92880c62c0 2024-04-26T23:07:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.074+0000 7f928f200640 1 -- 172.21.15.40:0/2472203283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92881f9fe0 con 0x7f92880c62c0 2024-04-26T23:07:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.074+0000 7f928f200640 1 -- 172.21.15.40:0/2472203283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9288166d30 con 0x7f92880c62c0 2024-04-26T23:07:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.075+0000 7f928f200640 1 -- 172.21.15.40:0/2472203283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92880b0c80 con 0x7f92880c62c0 2024-04-26T23:07:52.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.076+0000 7f9282000640 1 -- 172.21.15.40:0/2472203283 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f927c018070 con 0x7f92880c62c0 2024-04-26T23:07:52.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.077+0000 7f9282000640 1 --2- 172.21.15.40:0/2472203283 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f926c06c8f0 0x7f926c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:52.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.077+0000 7f9282000640 1 -- 172.21.15.40:0/2472203283 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f927c08bd00 con 0x7f92880c62c0 2024-04-26T23:07:52.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.078+0000 7f928d400640 1 --2- 172.21.15.40:0/2472203283 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f926c06c8f0 0x7f926c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:52.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.079+0000 7f928d400640 1 --2- 172.21.15.40:0/2472203283 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f926c06c8f0 0x7f926c06edb0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f9278034000 tx=0x7f927802c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:52.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.082+0000 7f9282000640 1 -- 172.21.15.40:0/2472203283 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f92880b0c80 con 0x7f92880c62c0 2024-04-26T23:07:52.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.332+0000 7f928f200640 1 -- 172.21.15.40:0/2472203283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9288069370 con 0x7f92880c62c0 2024-04-26T23:07:52.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.333+0000 7f9282000640 1 -- 172.21.15.40:0/2472203283 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5183 (secure 0 0 0) 0x7f927c059f90 con 0x7f92880c62c0 2024-04-26T23:07:52.334 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:52.334 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":91,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:07:51.330852+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8040},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[8040]},"id":6}]} 2024-04-26T23:07:52.334 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 91 2024-04-26T23:07:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.336+0000 7f928f200640 1 -- 172.21.15.40:0/2472203283 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f926c06c8f0 msgr2=0x7f926c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.336+0000 7f928f200640 1 --2- 172.21.15.40:0/2472203283 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f926c06c8f0 0x7f926c06edb0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f9278034000 tx=0x7f927802c000 comp rx=0 tx=0).stop 2024-04-26T23:07:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.336+0000 7f928f200640 1 -- 172.21.15.40:0/2472203283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92880c62c0 msgr2=0x7f9288165030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.336+0000 7f928f200640 1 --2- 172.21.15.40:0/2472203283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92880c62c0 0x7f9288165030 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f927c00c7f0 tx=0x7f927c00cbf0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.336+0000 7f928f200640 1 -- 172.21.15.40:0/2472203283 shutdown_connections 2024-04-26T23:07:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.336+0000 7f928f200640 1 --2- 172.21.15.40:0/2472203283 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f926c06c8f0 0x7f926c06edb0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.336+0000 7f928f200640 1 --2- 172.21.15.40:0/2472203283 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92881674a0 0x7f92881f9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.336+0000 7f928f200640 1 --2- 172.21.15.40:0/2472203283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9288163430 0x7f92880c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.336+0000 7f928f200640 1 --2- 172.21.15.40:0/2472203283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92880c62c0 0x7f9288165030 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.336+0000 7f928f200640 1 -- 172.21.15.40:0/2472203283 >> 172.21.15.40:0/2472203283 conn(0x7f92880d0a10 msgr2=0x7f92880b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.336+0000 7f928f200640 1 -- 172.21.15.40:0/2472203283 shutdown_connections 2024-04-26T23:07:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.336+0000 7f928f200640 1 -- 172.21.15.40:0/2472203283 wait complete. 2024-04-26T23:07:52.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:07:52.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.572+0000 7fbd5aa00640 1 Processor -- start 2024-04-26T23:07:52.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.573+0000 7fbd5aa00640 1 -- start start 2024-04-26T23:07:52.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.574+0000 7fbd5aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbd54071e60 0x7fbd5406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.574+0000 7fbd5aa00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd5415d7c0 con 0x7fbd540cc480 2024-04-26T23:07:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.574+0000 7fbd5aa00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd5415daa0 con 0x7fbd5406fe20 2024-04-26T23:07:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.574+0000 7fbd5aa00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fbd5415dd80 con 0x7fbd54071e60 2024-04-26T23:07:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.574+0000 7fbd53e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbd54071e60 0x7fbd5406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.575+0000 7fbd53e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbd54071e60 0x7fbd5406f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44570/0 (socket says 172.21.15.40:44570) 2024-04-26T23:07:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.575+0000 7fbd53e00640 1 -- 172.21.15.40:0/3746191381 learned_addr learned my addr 172.21.15.40:0/3746191381 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.575+0000 7fbd53e00640 1 -- 172.21.15.40:0/3746191381 >> v1:172.21.15.103:6789/0 conn(0x7fbd5406fe20 legacy=0x7fbd54072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.575+0000 7fbd53e00640 1 -- 172.21.15.40:0/3746191381 >> v1:172.21.15.40:6789/0 conn(0x7fbd540cc480 legacy=0x7fbd54166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:52.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.575+0000 7fbd53e00640 1 -- 172.21.15.40:0/3746191381 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd5415e060 con 0x7fbd54071e60 2024-04-26T23:07:52.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.576+0000 7fbd53e00640 1 --2- 172.21.15.40:0/3746191381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54071e60 0x7fbd5406f8e0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009870 tx=0x7fbd4402a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74979c0748e7286 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:52.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.577+0000 7fbd53400640 1 -- 172.21.15.40:0/3746191381 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd44037070 con 0x7fbd54071e60 2024-04-26T23:07:52.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.577+0000 7fbd53400640 1 -- 172.21.15.40:0/3746191381 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd44032440 con 0x7fbd54071e60 2024-04-26T23:07:52.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.577+0000 7fbd53400640 1 -- 172.21.15.40:0/3746191381 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd44032810 con 0x7fbd54071e60 2024-04-26T23:07:52.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.577+0000 7fbd5aa00640 1 -- 172.21.15.40:0/3746191381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54071e60 msgr2=0x7fbd5406f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:52.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.577+0000 7fbd5aa00640 1 --2- 172.21.15.40:0/3746191381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54071e60 0x7fbd5406f8e0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009870 tx=0x7fbd4402a2a0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.577+0000 7fbd5aa00640 1 -- 172.21.15.40:0/3746191381 shutdown_connections 2024-04-26T23:07:52.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.577+0000 7fbd5aa00640 1 --2- 172.21.15.40:0/3746191381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd54071e60 0x7fbd5406f8e0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.577+0000 7fbd5aa00640 1 -- 172.21.15.40:0/3746191381 >> 172.21.15.40:0/3746191381 conn(0x7fbd540d0be0 msgr2=0x7fbd540d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:52.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.578+0000 7fbd5aa00640 1 -- 172.21.15.40:0/3746191381 shutdown_connections 2024-04-26T23:07:52.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.578+0000 7fbd5aa00640 1 -- 172.21.15.40:0/3746191381 wait complete. 2024-04-26T23:07:52.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.579+0000 7fbd5aa00640 1 Processor -- start 2024-04-26T23:07:52.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.579+0000 7fbd5aa00640 1 -- start start 2024-04-26T23:07:52.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.580+0000 7fbd5aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd5406fe20 0x7fbd540b6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.581+0000 7fbd5aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd54071e60 0x7fbd5415c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.581+0000 7fbd5aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd540cc480 0x7fbd5415f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.581+0000 7fbd5aa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5415d570 con 0x7fbd540cc480 2024-04-26T23:07:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.581+0000 7fbd5aa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5415d3f0 con 0x7fbd5406fe20 2024-04-26T23:07:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.581+0000 7fbd5aa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbd54072950 con 0x7fbd54071e60 2024-04-26T23:07:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.581+0000 7fbd58a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd540cc480 0x7fbd5415f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.581+0000 7fbd58a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd540cc480 0x7fbd5415f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44584/0 (socket says 172.21.15.40:44584) 2024-04-26T23:07:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.581+0000 7fbd58a00640 1 -- 172.21.15.40:0/858819298 learned_addr learned my addr 172.21.15.40:0/858819298 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.581+0000 7fbd4be00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd54071e60 0x7fbd5415c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.581+0000 7fbd53e00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd5406fe20 0x7fbd540b6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.581+0000 7fbd58a00640 1 -- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd54071e60 msgr2=0x7fbd5415c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.582+0000 7fbd58a00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd54071e60 0x7fbd5415c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.582+0000 7fbd58a00640 1 -- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd5406fe20 msgr2=0x7fbd540b6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.582+0000 7fbd58a00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd5406fe20 0x7fbd540b6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.582+0000 7fbd58a00640 1 -- 172.21.15.40:0/858819298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd541f6170 con 0x7fbd540cc480 2024-04-26T23:07:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.582+0000 7fbd58a00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd540cc480 0x7fbd5415f550 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00c6e0 tx=0x7fbd4c00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.582+0000 7fbd50c00640 1 -- 172.21.15.40:0/858819298 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c003b70 con 0x7fbd540cc480 2024-04-26T23:07:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.582+0000 7fbd50c00640 1 -- 172.21.15.40:0/858819298 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd4c003d30 con 0x7fbd540cc480 2024-04-26T23:07:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.582+0000 7fbd50c00640 1 -- 172.21.15.40:0/858819298 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c005310 con 0x7fbd540cc480 2024-04-26T23:07:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.582+0000 7fbd5aa00640 1 -- 172.21.15.40:0/858819298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd540b5990 con 0x7fbd540cc480 2024-04-26T23:07:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.582+0000 7fbd5aa00640 1 -- 172.21.15.40:0/858819298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd540b5f50 con 0x7fbd540cc480 2024-04-26T23:07:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.584+0000 7fbd5aa00640 1 -- 172.21.15.40:0/858819298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd540b5780 con 0x7fbd540cc480 2024-04-26T23:07:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.584+0000 7fbd50c00640 1 -- 172.21.15.40:0/858819298 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbd4c0054b0 con 0x7fbd540cc480 2024-04-26T23:07:52.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.585+0000 7fbd50c00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd3006c940 0x7fbd3006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:52.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.586+0000 7fbd50c00640 1 -- 172.21.15.40:0/858819298 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fbd4c08bb60 con 0x7fbd540cc480 2024-04-26T23:07:52.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.586+0000 7fbd53e00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd3006c940 0x7fbd3006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:52.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.587+0000 7fbd53e00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd3006c940 0x7fbd3006ee00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fbd44034000 tx=0x7fbd4402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:52.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.590+0000 7fbd50c00640 1 -- 172.21.15.40:0/858819298 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fbd4c059de0 con 0x7fbd540cc480 2024-04-26T23:07:52.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.796+0000 7fbd5aa00640 1 -- 172.21.15.40:0/858819298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd540b6120 con 0x7fbd540cc480 2024-04-26T23:07:52.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.798+0000 7fbd50c00640 1 -- 172.21.15.40:0/858819298 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+16849 (secure 0 0 0) 0x7fbd4c0596e0 con 0x7fbd540cc480 2024-04-26T23:07:52.799 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:52.799 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:07:49.778062+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:07:46.137238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-26T23:07:47.258518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:07:52.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.801+0000 7fbd5aa00640 1 -- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd3006c940 msgr2=0x7fbd3006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.801+0000 7fbd5aa00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd3006c940 0x7fbd3006ee00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fbd44034000 tx=0x7fbd4402e000 comp rx=0 tx=0).stop 2024-04-26T23:07:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.801+0000 7fbd5aa00640 1 -- 172.21.15.40:0/858819298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd540cc480 msgr2=0x7fbd5415f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.801+0000 7fbd5aa00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd540cc480 0x7fbd5415f550 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00c6e0 tx=0x7fbd4c00cba0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.801+0000 7fbd5aa00640 1 -- 172.21.15.40:0/858819298 shutdown_connections 2024-04-26T23:07:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.801+0000 7fbd5aa00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbd3006c940 0x7fbd3006ee00 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.801+0000 7fbd5aa00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd540cc480 0x7fbd5415f550 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.801+0000 7fbd5aa00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd54071e60 0x7fbd5415c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.801+0000 7fbd5aa00640 1 --2- 172.21.15.40:0/858819298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd5406fe20 0x7fbd540b6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.801+0000 7fbd5aa00640 1 -- 172.21.15.40:0/858819298 >> 172.21.15.40:0/858819298 conn(0x7fbd540d0be0 msgr2=0x7fbd540b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.801+0000 7fbd5aa00640 1 -- 172.21.15.40:0/858819298 shutdown_connections 2024-04-26T23:07:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:52.801+0000 7fbd5aa00640 1 -- 172.21.15.40:0/858819298 wait complete. 2024-04-26T23:07:52.822 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T23:07:52.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:07:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.035+0000 7f35c4a00640 1 Processor -- start 2024-04-26T23:07:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.036+0000 7f35c4a00640 1 -- start start 2024-04-26T23:07:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.037+0000 7f35c4a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35c00b6220 0x7f35c00d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.037+0000 7f35c4a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f35c00d39f0 0x7f35c00b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.037+0000 7f35c4a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c0162c40 con 0x7f35c00d08d0 2024-04-26T23:07:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.037+0000 7f35c4a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f35c00d2d80 con 0x7f35c00b6220 2024-04-26T23:07:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.037+0000 7f35c4a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f35c00b7c80 con 0x7f35c00d39f0 2024-04-26T23:07:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.038+0000 7f35be800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35c00b6220 0x7f35c00d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.038+0000 7f35be800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35c00b6220 0x7f35c00d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44600/0 (socket says 172.21.15.40:44600) 2024-04-26T23:07:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.038+0000 7f35be800640 1 -- 172.21.15.40:0/1219484008 learned_addr learned my addr 172.21.15.40:0/1219484008 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.038+0000 7f35bf200640 1 --2- 172.21.15.40:0/1219484008 >> v2:172.21.15.103:3300/0 conn(0x7f35c00d39f0 0x7f35c00b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.038+0000 7f35be800640 1 -- 172.21.15.40:0/1219484008 >> v2:172.21.15.103:3300/0 conn(0x7f35c00d39f0 msgr2=0x7f35c00b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.038+0000 7f35be800640 1 --2- 172.21.15.40:0/1219484008 >> v2:172.21.15.103:3300/0 conn(0x7f35c00d39f0 0x7f35c00b8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.038+0000 7f35be800640 1 -- 172.21.15.40:0/1219484008 >> v1:172.21.15.103:6789/0 conn(0x7f35c00d08d0 legacy=0x7f35c00d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.038+0000 7f35be800640 1 -- 172.21.15.40:0/1219484008 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c00b7f60 con 0x7f35c00b6220 2024-04-26T23:07:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.038+0000 7f35bf200640 1 --2- 172.21.15.40:0/1219484008 >> v2:172.21.15.103:3300/0 conn(0x7f35c00d39f0 0x7f35c00b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.040+0000 7f35be800640 1 --2- 172.21.15.40:0/1219484008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35c00b6220 0x7f35c00d0390 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f35c00d5dd0 tx=0x7f35b002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536d24cff764704c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.040+0000 7f35bd400640 1 -- 172.21.15.40:0/1219484008 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b0037070 con 0x7f35c00b6220 2024-04-26T23:07:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.040+0000 7f35bd400640 1 -- 172.21.15.40:0/1219484008 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35b0032440 con 0x7f35c00b6220 2024-04-26T23:07:53.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.040+0000 7f35bd400640 1 -- 172.21.15.40:0/1219484008 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b0032810 con 0x7f35c00b6220 2024-04-26T23:07:53.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.041+0000 7f35c4a00640 1 -- 172.21.15.40:0/1219484008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35c00b6220 msgr2=0x7f35c00d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:53.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.041+0000 7f35c4a00640 1 --2- 172.21.15.40:0/1219484008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35c00b6220 0x7f35c00d0390 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f35c00d5dd0 tx=0x7f35b002a230 comp rx=0 tx=0).stop 2024-04-26T23:07:53.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.041+0000 7f35c4a00640 1 -- 172.21.15.40:0/1219484008 shutdown_connections 2024-04-26T23:07:53.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.041+0000 7f35c4a00640 1 --2- 172.21.15.40:0/1219484008 >> v2:172.21.15.103:3300/0 conn(0x7f35c00d39f0 0x7f35c00b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.041+0000 7f35c4a00640 1 --2- 172.21.15.40:0/1219484008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35c00b6220 0x7f35c00d0390 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.041+0000 7f35c4a00640 1 -- 172.21.15.40:0/1219484008 >> 172.21.15.40:0/1219484008 conn(0x7f35c00d4d90 msgr2=0x7f35c00d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:53.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.041+0000 7f35c4a00640 1 -- 172.21.15.40:0/1219484008 shutdown_connections 2024-04-26T23:07:53.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.041+0000 7f35c4a00640 1 -- 172.21.15.40:0/1219484008 wait complete. 2024-04-26T23:07:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.043+0000 7f35c4a00640 1 Processor -- start 2024-04-26T23:07:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.043+0000 7f35c4a00640 1 -- start start 2024-04-26T23:07:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.044+0000 7f35c4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35c00b6220 0x7f35c006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.044+0000 7f35c4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35c00d08d0 0x7f35c0152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.044+0000 7f35c4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35c00d39f0 0x7f35c0153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.044+0000 7f35c4a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f35c00b9260 con 0x7f35c00d39f0 2024-04-26T23:07:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.044+0000 7f35c4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f35c00b8520 con 0x7f35c00b6220 2024-04-26T23:07:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.044+0000 7f35c4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f35c00b93e0 con 0x7f35c00d08d0 2024-04-26T23:07:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.045+0000 7f35bf200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35c00d39f0 0x7f35c0153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.045+0000 7f35bf200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35c00d39f0 0x7f35c0153290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44616/0 (socket says 172.21.15.40:44616) 2024-04-26T23:07:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.045+0000 7f35bf200640 1 -- 172.21.15.40:0/44936439 learned_addr learned my addr 172.21.15.40:0/44936439 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.045+0000 7f35be800640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35c00b6220 0x7f35c006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.045+0000 7f35bde00640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35c00d08d0 0x7f35c0152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.045+0000 7f35bf200640 1 -- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35c00d08d0 msgr2=0x7f35c0152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.045+0000 7f35bf200640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35c00d08d0 0x7f35c0152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.045+0000 7f35bf200640 1 -- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35c00b6220 msgr2=0x7f35c006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.045+0000 7f35bf200640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35c00b6220 0x7f35c006d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.045+0000 7f35bf200640 1 -- 172.21.15.40:0/44936439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c0153da0 con 0x7f35c00d39f0 2024-04-26T23:07:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.046+0000 7f35bf200640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35c00d39f0 0x7f35c0153290 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f35b800cd50 tx=0x7f35b800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.046+0000 7f35b6a00640 1 -- 172.21.15.40:0/44936439 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b800f680 con 0x7f35c00d39f0 2024-04-26T23:07:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.046+0000 7f35b6a00640 1 -- 172.21.15.40:0/44936439 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35b800f840 con 0x7f35c00d39f0 2024-04-26T23:07:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.046+0000 7f35b6a00640 1 -- 172.21.15.40:0/44936439 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b800fc30 con 0x7f35c00d39f0 2024-04-26T23:07:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.046+0000 7f35c4a00640 1 -- 172.21.15.40:0/44936439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c0154080 con 0x7f35c00d39f0 2024-04-26T23:07:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.046+0000 7f35c4a00640 1 -- 172.21.15.40:0/44936439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c0154920 con 0x7f35c00d39f0 2024-04-26T23:07:53.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.047+0000 7f35c4a00640 1 -- 172.21.15.40:0/44936439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35c00d5be0 con 0x7f35c00d39f0 2024-04-26T23:07:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.049+0000 7f35b6a00640 1 -- 172.21.15.40:0/44936439 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f35b80213f0 con 0x7f35c00d39f0 2024-04-26T23:07:53.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.050+0000 7f35b6a00640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f359c06c8f0 0x7f359c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:53.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.050+0000 7f35b6a00640 1 -- 172.21.15.40:0/44936439 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f35b808cb90 con 0x7f35c00d39f0 2024-04-26T23:07:53.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.051+0000 7f35be800640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f359c06c8f0 0x7f359c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:53.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.052+0000 7f35be800640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f359c06c8f0 0x7f359c06edb0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f35b0034000 tx=0x7f35b002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.055+0000 7f35b6a00640 1 -- 172.21.15.40:0/44936439 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f35c00d5be0 con 0x7f35c00d39f0 2024-04-26T23:07:53.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.256+0000 7f35c4a00640 1 -- 172.21.15.40:0/44936439 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f35c00cb9f0 con 0x7f359c06c8f0 2024-04-26T23:07:53.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.276+0000 7f35b6a00640 1 -- 172.21.15.40:0/44936439 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246766 (secure 0 0 0) 0x7f35b007e400 con 0x7f359c06c8f0 2024-04-26T23:07:53.280 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:53.281 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:07:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.282+0000 7f35c4a00640 1 -- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f359c06c8f0 msgr2=0x7f359c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.282+0000 7f35c4a00640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f359c06c8f0 0x7f359c06edb0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f35b0034000 tx=0x7f35b002e000 comp rx=0 tx=0).stop 2024-04-26T23:07:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.282+0000 7f35c4a00640 1 -- 172.21.15.40:0/44936439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35c00d39f0 msgr2=0x7f35c0153290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.282+0000 7f35c4a00640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35c00d39f0 0x7f35c0153290 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f35b800cd50 tx=0x7f35b800a650 comp rx=0 tx=0).stop 2024-04-26T23:07:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.282+0000 7f35c4a00640 1 -- 172.21.15.40:0/44936439 shutdown_connections 2024-04-26T23:07:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.282+0000 7f35c4a00640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f359c06c8f0 0x7f359c06edb0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.282+0000 7f35c4a00640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35c00d39f0 0x7f35c0153290 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.282+0000 7f35c4a00640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f35c00d08d0 0x7f35c0152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.282+0000 7f35c4a00640 1 --2- 172.21.15.40:0/44936439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35c00b6220 0x7f35c006d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.282+0000 7f35c4a00640 1 -- 172.21.15.40:0/44936439 >> 172.21.15.40:0/44936439 conn(0x7f35c00d4d90 msgr2=0x7f35c00d32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.283+0000 7f35c4a00640 1 -- 172.21.15.40:0/44936439 shutdown_connections 2024-04-26T23:07:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.283+0000 7f35c4a00640 1 -- 172.21.15.40:0/44936439 wait complete. 2024-04-26T23:07:53.304 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-04-26T23:07:51.295479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259020,"kb_used_data":9568,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715700,"statfs":{"total":773094113280,"available":772828876800,"internally_reserved":0,"allocated":9797632,"data_stored":5160894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.522199"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.805814+0000","last_change":"2024-04-26T23:07:47.805814+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.805814+0000","last_undegraded":"2024-04-26T23:07:47.805814+0000","last_fullsized":"2024-04-26T23:07:47.805814+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.814942+0000","last_change":"2024-04-26T23:07:47.814942+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.814942+0000","last_undegraded":"2024-04-26T23:07:47.814942+0000","last_fullsized":"2024-04-26T23:07:47.814942+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.803438+0000","last_change":"2024-04-26T23:07:47.803438+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.803438+0000","last_undegraded":"2024-04-26T23:07:47.803438+0000","last_fullsized":"2024-04-26T23:07:47.803438+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.805828+0000","last_change":"2024-04-26T23:07:47.805828+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.805828+0000","last_undegraded":"2024-04-26T23:07:47.805828+0000","last_fullsized":"2024-04-26T23:07:47.805828+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.803123+0000","last_change":"2024-04-26T23:07:47.803123+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.803123+0000","last_undegraded":"2024-04-26T23:07:47.803123+0000","last_fullsized":"2024-04-26T23:07:47.803123+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.798868+0000","last_change":"2024-04-26T23:07:47.798868+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.798868+0000","last_undegraded":"2024-04-26T23:07:47.798868+0000","last_fullsized":"2024-04-26T23:07:47.798868+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.811325+0000","last_change":"2024-04-26T23:07:47.811325+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.811325+0000","last_undegraded":"2024-04-26T23:07:47.811325+0000","last_fullsized":"2024-04-26T23:07:47.811325+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.808634+0000","last_change":"2024-04-26T23:07:47.808634+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.808634+0000","last_undegraded":"2024-04-26T23:07:47.808634+0000","last_fullsized":"2024-04-26T23:07:47.808634+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.807950+0000","last_change":"2024-04-26T23:07:47.807950+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.807950+0000","last_undegraded":"2024-04-26T23:07:47.807950+0000","last_fullsized":"2024-04-26T23:07:47.807950+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.802654+0000","last_change":"2024-04-26T23:07:47.802654+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.802654+0000","last_undegraded":"2024-04-26T23:07:47.802654+0000","last_fullsized":"2024-04-26T23:07:47.802654+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.810499+0000","last_change":"2024-04-26T23:07:47.810499+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.810499+0000","last_undegraded":"2024-04-26T23:07:47.810499+0000","last_fullsized":"2024-04-26T23:07:47.810499+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.799836+0000","last_change":"2024-04-26T23:07:47.799836+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.799836+0000","last_undegraded":"2024-04-26T23:07:47.799836+0000","last_fullsized":"2024-04-26T23:07:47.799836+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.806323+0000","last_change":"2024-04-26T23:07:47.806323+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.806323+0000","last_undegraded":"2024-04-26T23:07:47.806323+0000","last_fullsized":"2024-04-26T23:07:47.806323+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.803189+0000","last_change":"2024-04-26T23:07:47.803189+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.803189+0000","last_undegraded":"2024-04-26T23:07:47.803189+0000","last_fullsized":"2024-04-26T23:07:47.803189+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.804982+0000","last_change":"2024-04-26T23:07:47.804982+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.804982+0000","last_undegraded":"2024-04-26T23:07:47.804982+0000","last_fullsized":"2024-04-26T23:07:47.804982+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.799040+0000","last_change":"2024-04-26T23:07:47.799040+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.799040+0000","last_undegraded":"2024-04-26T23:07:47.799040+0000","last_fullsized":"2024-04-26T23:07:47.799040+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.799527+0000","last_change":"2024-04-26T23:07:47.799527+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.799527+0000","last_undegraded":"2024-04-26T23:07:47.799527+0000","last_fullsized":"2024-04-26T23:07:47.799527+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.811731+0000","last_change":"2024-04-26T23:07:47.811731+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.811731+0000","last_undegraded":"2024-04-26T23:07:47.811731+0000","last_fullsized":"2024-04-26T23:07:47.811731+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.802252+0000","last_change":"2024-04-26T23:07:47.802252+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.802252+0000","last_undegraded":"2024-04-26T23:07:47.802252+0000","last_fullsized":"2024-04-26T23:07:47.802252+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.805308+0000","last_change":"2024-04-26T23:07:47.805308+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.805308+0000","last_undegraded":"2024-04-26T23:07:47.805308+0000","last_fullsized":"2024-04-26T23:07:47.805308+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.805550+0000","last_change":"2024-04-26T23:07:47.805550+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.805550+0000","last_undegraded":"2024-04-26T23:07:47.805550+0000","last_fullsized":"2024-04-26T23:07:47.805550+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.801963+0000","last_change":"2024-04-26T23:07:47.801963+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.801963+0000","last_undegraded":"2024-04-26T23:07:47.801963+0000","last_fullsized":"2024-04-26T23:07:47.801963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.799039+0000","last_change":"2024-04-26T23:07:47.799039+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.799039+0000","last_undegraded":"2024-04-26T23:07:47.799039+0000","last_fullsized":"2024-04-26T23:07:47.799039+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.800911+0000","last_change":"2024-04-26T23:07:47.800911+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.800911+0000","last_undegraded":"2024-04-26T23:07:47.800911+0000","last_fullsized":"2024-04-26T23:07:47.800911+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.815180+0000","last_change":"2024-04-26T23:07:47.815180+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.815180+0000","last_undegraded":"2024-04-26T23:07:47.815180+0000","last_fullsized":"2024-04-26T23:07:47.815180+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.806660+0000","last_change":"2024-04-26T23:07:47.806660+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.806660+0000","last_undegraded":"2024-04-26T23:07:47.806660+0000","last_fullsized":"2024-04-26T23:07:47.806660+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.798941+0000","last_change":"2024-04-26T23:07:47.798941+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.798941+0000","last_undegraded":"2024-04-26T23:07:47.798941+0000","last_fullsized":"2024-04-26T23:07:47.798941+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.804684+0000","last_change":"2024-04-26T23:07:47.804684+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.804684+0000","last_undegraded":"2024-04-26T23:07:47.804684+0000","last_fullsized":"2024-04-26T23:07:47.804684+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.812460+0000","last_change":"2024-04-26T23:07:47.812460+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.812460+0000","last_undegraded":"2024-04-26T23:07:47.812460+0000","last_fullsized":"2024-04-26T23:07:47.812460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.800719+0000","last_change":"2024-04-26T23:07:47.800719+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.800719+0000","last_undegraded":"2024-04-26T23:07:47.800719+0000","last_fullsized":"2024-04-26T23:07:47.800719+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.816873+0000","last_change":"2024-04-26T23:07:47.816873+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.816873+0000","last_undegraded":"2024-04-26T23:07:47.816873+0000","last_fullsized":"2024-04-26T23:07:47.816873+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.798593+0000","last_change":"2024-04-26T23:07:47.798593+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.798593+0000","last_undegraded":"2024-04-26T23:07:47.798593+0000","last_fullsized":"2024-04-26T23:07:47.798593+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.800163+0000","last_change":"2024-04-26T23:07:47.800163+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.800163+0000","last_undegraded":"2024-04-26T23:07:47.800163+0000","last_fullsized":"2024-04-26T23:07:47.800163+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.820976+0000","last_change":"2024-04-26T23:07:47.809702+0000","last_active":"2024-04-26T23:07:47.820976+0000","last_peered":"2024-04-26T23:07:47.820976+0000","last_clean":"2024-04-26T23:07:47.820976+0000","last_became_active":"2024-04-26T23:07:47.807055+0000","last_became_peered":"2024-04-26T23:07:47.807055+0000","last_unstale":"2024-04-26T23:07:47.820976+0000","last_undegraded":"2024-04-26T23:07:47.820976+0000","last_fullsized":"2024-04-26T23:07:47.820976+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:21.116738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.799552+0000","last_change":"2024-04-26T23:07:47.799552+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.799552+0000","last_undegraded":"2024-04-26T23:07:47.799552+0000","last_fullsized":"2024-04-26T23:07:47.799552+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.827078+0000","last_change":"2024-04-26T23:07:47.821601+0000","last_active":"2024-04-26T23:07:47.827078+0000","last_peered":"2024-04-26T23:07:47.827078+0000","last_clean":"2024-04-26T23:07:47.827078+0000","last_became_active":"2024-04-26T23:07:47.821310+0000","last_became_peered":"2024-04-26T23:07:47.821310+0000","last_unstale":"2024-04-26T23:07:47.827078+0000","last_undegraded":"2024-04-26T23:07:47.827078+0000","last_fullsized":"2024-04-26T23:07:47.827078+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:57:04.001569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.814021+0000","last_change":"2024-04-26T23:07:47.814021+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.814021+0000","last_undegraded":"2024-04-26T23:07:47.814021+0000","last_fullsized":"2024-04-26T23:07:47.814021+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.832020+0000","last_change":"2024-04-26T23:07:47.827222+0000","last_active":"2024-04-26T23:07:47.832020+0000","last_peered":"2024-04-26T23:07:47.832020+0000","last_clean":"2024-04-26T23:07:47.832020+0000","last_became_active":"2024-04-26T23:07:47.826726+0000","last_became_peered":"2024-04-26T23:07:47.826726+0000","last_unstale":"2024-04-26T23:07:47.832020+0000","last_undegraded":"2024-04-26T23:07:47.832020+0000","last_fullsized":"2024-04-26T23:07:47.832020+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:33:01.607220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.798875+0000","last_change":"2024-04-26T23:07:47.798875+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.798875+0000","last_undegraded":"2024-04-26T23:07:47.798875+0000","last_fullsized":"2024-04-26T23:07:47.798875+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.821542+0000","last_change":"2024-04-26T23:07:47.817334+0000","last_active":"2024-04-26T23:07:47.821542+0000","last_peered":"2024-04-26T23:07:47.821542+0000","last_clean":"2024-04-26T23:07:47.821542+0000","last_became_active":"2024-04-26T23:07:47.816968+0000","last_became_peered":"2024-04-26T23:07:47.816968+0000","last_unstale":"2024-04-26T23:07:47.821542+0000","last_undegraded":"2024-04-26T23:07:47.821542+0000","last_fullsized":"2024-04-26T23:07:47.821542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:51.328973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.803581+0000","last_change":"2024-04-26T23:07:47.803581+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.803581+0000","last_undegraded":"2024-04-26T23:07:47.803581+0000","last_fullsized":"2024-04-26T23:07:47.803581+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.827181+0000","last_change":"2024-04-26T23:07:47.820427+0000","last_active":"2024-04-26T23:07:47.827181+0000","last_peered":"2024-04-26T23:07:47.827181+0000","last_clean":"2024-04-26T23:07:47.827181+0000","last_became_active":"2024-04-26T23:07:47.820082+0000","last_became_peered":"2024-04-26T23:07:47.820082+0000","last_unstale":"2024-04-26T23:07:47.827181+0000","last_undegraded":"2024-04-26T23:07:47.827181+0000","last_fullsized":"2024-04-26T23:07:47.827181+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:39.447983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.797606+0000","last_change":"2024-04-26T23:07:47.797606+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.797606+0000","last_undegraded":"2024-04-26T23:07:47.797606+0000","last_fullsized":"2024-04-26T23:07:47.797606+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.826920+0000","last_change":"2024-04-26T23:07:47.823971+0000","last_active":"2024-04-26T23:07:47.826920+0000","last_peered":"2024-04-26T23:07:47.826920+0000","last_clean":"2024-04-26T23:07:47.826920+0000","last_became_active":"2024-04-26T23:07:47.821435+0000","last_became_peered":"2024-04-26T23:07:47.821435+0000","last_unstale":"2024-04-26T23:07:47.826920+0000","last_undegraded":"2024-04-26T23:07:47.826920+0000","last_fullsized":"2024-04-26T23:07:47.826920+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:24:29.558386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.798424+0000","last_change":"2024-04-26T23:07:47.798424+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.798424+0000","last_undegraded":"2024-04-26T23:07:47.798424+0000","last_fullsized":"2024-04-26T23:07:47.798424+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.825520+0000","last_change":"2024-04-26T23:07:47.822575+0000","last_active":"2024-04-26T23:07:47.825520+0000","last_peered":"2024-04-26T23:07:47.825520+0000","last_clean":"2024-04-26T23:07:47.825520+0000","last_became_active":"2024-04-26T23:07:47.820316+0000","last_became_peered":"2024-04-26T23:07:47.820316+0000","last_unstale":"2024-04-26T23:07:47.825520+0000","last_undegraded":"2024-04-26T23:07:47.825520+0000","last_fullsized":"2024-04-26T23:07:47.825520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:10:52.592001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.805813+0000","last_change":"2024-04-26T23:07:47.805813+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.805813+0000","last_undegraded":"2024-04-26T23:07:47.805813+0000","last_fullsized":"2024-04-26T23:07:47.805813+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.828206+0000","last_change":"2024-04-26T23:07:47.821897+0000","last_active":"2024-04-26T23:07:47.828206+0000","last_peered":"2024-04-26T23:07:47.828206+0000","last_clean":"2024-04-26T23:07:47.828206+0000","last_became_active":"2024-04-26T23:07:47.820410+0000","last_became_peered":"2024-04-26T23:07:47.820410+0000","last_unstale":"2024-04-26T23:07:47.828206+0000","last_undegraded":"2024-04-26T23:07:47.828206+0000","last_fullsized":"2024-04-26T23:07:47.828206+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:36:02.362889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.809963+0000","last_change":"2024-04-26T23:07:47.809963+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.809963+0000","last_undegraded":"2024-04-26T23:07:47.809963+0000","last_fullsized":"2024-04-26T23:07:47.809963+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.832194+0000","last_change":"2024-04-26T23:07:47.827901+0000","last_active":"2024-04-26T23:07:47.832194+0000","last_peered":"2024-04-26T23:07:47.832194+0000","last_clean":"2024-04-26T23:07:47.832194+0000","last_became_active":"2024-04-26T23:07:47.827572+0000","last_became_peered":"2024-04-26T23:07:47.827572+0000","last_unstale":"2024-04-26T23:07:47.832194+0000","last_undegraded":"2024-04-26T23:07:47.832194+0000","last_fullsized":"2024-04-26T23:07:47.832194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:52:06.881525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.812710+0000","last_change":"2024-04-26T23:07:47.812710+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.812710+0000","last_undegraded":"2024-04-26T23:07:47.812710+0000","last_fullsized":"2024-04-26T23:07:47.812710+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.824383+0000","last_change":"2024-04-26T23:07:47.821054+0000","last_active":"2024-04-26T23:07:47.824383+0000","last_peered":"2024-04-26T23:07:47.824383+0000","last_clean":"2024-04-26T23:07:47.824383+0000","last_became_active":"2024-04-26T23:07:47.820581+0000","last_became_peered":"2024-04-26T23:07:47.820581+0000","last_unstale":"2024-04-26T23:07:47.824383+0000","last_undegraded":"2024-04-26T23:07:47.824383+0000","last_fullsized":"2024-04-26T23:07:47.824383+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:46.908552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.809175+0000","last_change":"2024-04-26T23:07:47.809175+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.809175+0000","last_undegraded":"2024-04-26T23:07:47.809175+0000","last_fullsized":"2024-04-26T23:07:47.809175+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.827048+0000","last_change":"2024-04-26T23:07:47.820922+0000","last_active":"2024-04-26T23:07:47.827048+0000","last_peered":"2024-04-26T23:07:47.827048+0000","last_clean":"2024-04-26T23:07:47.827048+0000","last_became_active":"2024-04-26T23:07:47.820591+0000","last_became_peered":"2024-04-26T23:07:47.820591+0000","last_unstale":"2024-04-26T23:07:47.827048+0000","last_undegraded":"2024-04-26T23:07:47.827048+0000","last_fullsized":"2024-04-26T23:07:47.827048+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:24:33.218143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.804153+0000","last_change":"2024-04-26T23:07:47.804153+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.804153+0000","last_undegraded":"2024-04-26T23:07:47.804153+0000","last_fullsized":"2024-04-26T23:07:47.804153+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.819304+0000","last_change":"2024-04-26T23:07:47.815166+0000","last_active":"2024-04-26T23:07:47.819304+0000","last_peered":"2024-04-26T23:07:47.819304+0000","last_clean":"2024-04-26T23:07:47.819304+0000","last_became_active":"2024-04-26T23:07:47.814809+0000","last_became_peered":"2024-04-26T23:07:47.814809+0000","last_unstale":"2024-04-26T23:07:47.819304+0000","last_undegraded":"2024-04-26T23:07:47.819304+0000","last_fullsized":"2024-04-26T23:07:47.819304+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:15:58.115631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.802796+0000","last_change":"2024-04-26T23:07:47.802796+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.802796+0000","last_undegraded":"2024-04-26T23:07:47.802796+0000","last_fullsized":"2024-04-26T23:07:47.802796+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.827422+0000","last_change":"2024-04-26T23:07:47.822437+0000","last_active":"2024-04-26T23:07:47.827422+0000","last_peered":"2024-04-26T23:07:47.827422+0000","last_clean":"2024-04-26T23:07:47.827422+0000","last_became_active":"2024-04-26T23:07:47.822143+0000","last_became_peered":"2024-04-26T23:07:47.822143+0000","last_unstale":"2024-04-26T23:07:47.827422+0000","last_undegraded":"2024-04-26T23:07:47.827422+0000","last_fullsized":"2024-04-26T23:07:47.827422+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:44.048319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.799494+0000","last_change":"2024-04-26T23:07:47.799494+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.799494+0000","last_undegraded":"2024-04-26T23:07:47.799494+0000","last_fullsized":"2024-04-26T23:07:47.799494+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.824170+0000","last_change":"2024-04-26T23:07:47.818118+0000","last_active":"2024-04-26T23:07:47.824170+0000","last_peered":"2024-04-26T23:07:47.824170+0000","last_clean":"2024-04-26T23:07:47.824170+0000","last_became_active":"2024-04-26T23:07:47.816576+0000","last_became_peered":"2024-04-26T23:07:47.816576+0000","last_unstale":"2024-04-26T23:07:47.824170+0000","last_undegraded":"2024-04-26T23:07:47.824170+0000","last_fullsized":"2024-04-26T23:07:47.824170+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:10:52.592001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.797456+0000","last_change":"2024-04-26T23:07:47.797456+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.797456+0000","last_undegraded":"2024-04-26T23:07:47.797456+0000","last_fullsized":"2024-04-26T23:07:47.797456+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.825014+0000","last_change":"2024-04-26T23:07:47.821873+0000","last_active":"2024-04-26T23:07:47.825014+0000","last_peered":"2024-04-26T23:07:47.825014+0000","last_clean":"2024-04-26T23:07:47.825014+0000","last_became_active":"2024-04-26T23:07:47.821521+0000","last_became_peered":"2024-04-26T23:07:47.821521+0000","last_unstale":"2024-04-26T23:07:47.825014+0000","last_undegraded":"2024-04-26T23:07:47.825014+0000","last_fullsized":"2024-04-26T23:07:47.825014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:14.811668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.801066+0000","last_change":"2024-04-26T23:07:47.801066+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.801066+0000","last_undegraded":"2024-04-26T23:07:47.801066+0000","last_fullsized":"2024-04-26T23:07:47.801066+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.827033+0000","last_change":"2024-04-26T23:07:47.822478+0000","last_active":"2024-04-26T23:07:47.827033+0000","last_peered":"2024-04-26T23:07:47.827033+0000","last_clean":"2024-04-26T23:07:47.827033+0000","last_became_active":"2024-04-26T23:07:47.821881+0000","last_became_peered":"2024-04-26T23:07:47.821881+0000","last_unstale":"2024-04-26T23:07:47.827033+0000","last_undegraded":"2024-04-26T23:07:47.827033+0000","last_fullsized":"2024-04-26T23:07:47.827033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:07:57.468174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.799293+0000","last_change":"2024-04-26T23:07:47.799293+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.799293+0000","last_undegraded":"2024-04-26T23:07:47.799293+0000","last_fullsized":"2024-04-26T23:07:47.799293+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.829760+0000","last_change":"2024-04-26T23:07:47.823873+0000","last_active":"2024-04-26T23:07:47.829760+0000","last_peered":"2024-04-26T23:07:47.829760+0000","last_clean":"2024-04-26T23:07:47.829760+0000","last_became_active":"2024-04-26T23:07:47.823123+0000","last_became_peered":"2024-04-26T23:07:47.823123+0000","last_unstale":"2024-04-26T23:07:47.829760+0000","last_undegraded":"2024-04-26T23:07:47.829760+0000","last_fullsized":"2024-04-26T23:07:47.829760+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:13:16.015174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.799823+0000","last_change":"2024-04-26T23:07:47.799823+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.799823+0000","last_undegraded":"2024-04-26T23:07:47.799823+0000","last_fullsized":"2024-04-26T23:07:47.799823+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.829694+0000","last_change":"2024-04-26T23:07:47.825792+0000","last_active":"2024-04-26T23:07:47.829694+0000","last_peered":"2024-04-26T23:07:47.829694+0000","last_clean":"2024-04-26T23:07:47.829694+0000","last_became_active":"2024-04-26T23:07:47.824554+0000","last_became_peered":"2024-04-26T23:07:47.824554+0000","last_unstale":"2024-04-26T23:07:47.829694+0000","last_undegraded":"2024-04-26T23:07:47.829694+0000","last_fullsized":"2024-04-26T23:07:47.829694+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:52.749464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.801520+0000","last_change":"2024-04-26T23:07:47.801520+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.801520+0000","last_undegraded":"2024-04-26T23:07:47.801520+0000","last_fullsized":"2024-04-26T23:07:47.801520+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.829903+0000","last_change":"2024-04-26T23:07:47.828834+0000","last_active":"2024-04-26T23:07:47.829903+0000","last_peered":"2024-04-26T23:07:47.829903+0000","last_clean":"2024-04-26T23:07:47.829903+0000","last_became_active":"2024-04-26T23:07:47.828446+0000","last_became_peered":"2024-04-26T23:07:47.828446+0000","last_unstale":"2024-04-26T23:07:47.829903+0000","last_undegraded":"2024-04-26T23:07:47.829903+0000","last_fullsized":"2024-04-26T23:07:47.829903+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:14.811668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.797882+0000","last_change":"2024-04-26T23:07:47.797882+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.797882+0000","last_undegraded":"2024-04-26T23:07:47.797882+0000","last_fullsized":"2024-04-26T23:07:47.797882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.832012+0000","last_change":"2024-04-26T23:07:47.825672+0000","last_active":"2024-04-26T23:07:47.832012+0000","last_peered":"2024-04-26T23:07:47.832012+0000","last_clean":"2024-04-26T23:07:47.832012+0000","last_became_active":"2024-04-26T23:07:47.825386+0000","last_became_peered":"2024-04-26T23:07:47.825386+0000","last_unstale":"2024-04-26T23:07:47.832012+0000","last_undegraded":"2024-04-26T23:07:47.832012+0000","last_fullsized":"2024-04-26T23:07:47.832012+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:46.079465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.816471+0000","last_change":"2024-04-26T23:07:47.816471+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.816471+0000","last_undegraded":"2024-04-26T23:07:47.816471+0000","last_fullsized":"2024-04-26T23:07:47.816471+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.819573+0000","last_change":"2024-04-26T23:07:47.813255+0000","last_active":"2024-04-26T23:07:47.819573+0000","last_peered":"2024-04-26T23:07:47.819573+0000","last_clean":"2024-04-26T23:07:47.819573+0000","last_became_active":"2024-04-26T23:07:47.812932+0000","last_became_peered":"2024-04-26T23:07:47.812932+0000","last_unstale":"2024-04-26T23:07:47.819573+0000","last_undegraded":"2024-04-26T23:07:47.819573+0000","last_fullsized":"2024-04-26T23:07:47.819573+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:14.614466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.812775+0000","last_change":"2024-04-26T23:07:47.812775+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.812775+0000","last_undegraded":"2024-04-26T23:07:47.812775+0000","last_fullsized":"2024-04-26T23:07:47.812775+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.832192+0000","last_change":"2024-04-26T23:07:47.827803+0000","last_active":"2024-04-26T23:07:47.832192+0000","last_peered":"2024-04-26T23:07:47.832192+0000","last_clean":"2024-04-26T23:07:47.832192+0000","last_became_active":"2024-04-26T23:07:47.827474+0000","last_became_peered":"2024-04-26T23:07:47.827474+0000","last_unstale":"2024-04-26T23:07:47.832192+0000","last_undegraded":"2024-04-26T23:07:47.832192+0000","last_fullsized":"2024-04-26T23:07:47.832192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:54:39.836546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.801073+0000","last_change":"2024-04-26T23:07:47.801073+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.801073+0000","last_undegraded":"2024-04-26T23:07:47.801073+0000","last_fullsized":"2024-04-26T23:07:47.801073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.825600+0000","last_change":"2024-04-26T23:07:47.822273+0000","last_active":"2024-04-26T23:07:47.825600+0000","last_peered":"2024-04-26T23:07:47.825600+0000","last_clean":"2024-04-26T23:07:47.825600+0000","last_became_active":"2024-04-26T23:07:47.821964+0000","last_became_peered":"2024-04-26T23:07:47.821964+0000","last_unstale":"2024-04-26T23:07:47.825600+0000","last_undegraded":"2024-04-26T23:07:47.825600+0000","last_fullsized":"2024-04-26T23:07:47.825600+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:50:42.831548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.799528+0000","last_change":"2024-04-26T23:07:47.799528+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.799528+0000","last_undegraded":"2024-04-26T23:07:47.799528+0000","last_fullsized":"2024-04-26T23:07:47.799528+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.832024+0000","last_change":"2024-04-26T23:07:47.828282+0000","last_active":"2024-04-26T23:07:47.832024+0000","last_peered":"2024-04-26T23:07:47.832024+0000","last_clean":"2024-04-26T23:07:47.832024+0000","last_became_active":"2024-04-26T23:07:47.827934+0000","last_became_peered":"2024-04-26T23:07:47.827934+0000","last_unstale":"2024-04-26T23:07:47.832024+0000","last_undegraded":"2024-04-26T23:07:47.832024+0000","last_fullsized":"2024-04-26T23:07:47.832024+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:19:40.570836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.797735+0000","last_change":"2024-04-26T23:07:47.797735+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.797735+0000","last_undegraded":"2024-04-26T23:07:47.797735+0000","last_fullsized":"2024-04-26T23:07:47.797735+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.829366+0000","last_change":"2024-04-26T23:07:47.825275+0000","last_active":"2024-04-26T23:07:47.829366+0000","last_peered":"2024-04-26T23:07:47.829366+0000","last_clean":"2024-04-26T23:07:47.829366+0000","last_became_active":"2024-04-26T23:07:47.824983+0000","last_became_peered":"2024-04-26T23:07:47.824983+0000","last_unstale":"2024-04-26T23:07:47.829366+0000","last_undegraded":"2024-04-26T23:07:47.829366+0000","last_fullsized":"2024-04-26T23:07:47.829366+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:57:05.659949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.797756+0000","last_change":"2024-04-26T23:07:47.797756+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.797756+0000","last_undegraded":"2024-04-26T23:07:47.797756+0000","last_fullsized":"2024-04-26T23:07:47.797756+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.824526+0000","last_change":"2024-04-26T23:07:47.817958+0000","last_active":"2024-04-26T23:07:47.824526+0000","last_peered":"2024-04-26T23:07:47.824526+0000","last_clean":"2024-04-26T23:07:47.824526+0000","last_became_active":"2024-04-26T23:07:47.816225+0000","last_became_peered":"2024-04-26T23:07:47.816225+0000","last_unstale":"2024-04-26T23:07:47.824526+0000","last_undegraded":"2024-04-26T23:07:47.824526+0000","last_fullsized":"2024-04-26T23:07:47.824526+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:09:14.023306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.814640+0000","last_change":"2024-04-26T23:07:47.814640+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.814640+0000","last_undegraded":"2024-04-26T23:07:47.814640+0000","last_fullsized":"2024-04-26T23:07:47.814640+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.817491+0000","last_change":"2024-04-26T23:07:47.806304+0000","last_active":"2024-04-26T23:07:47.817491+0000","last_peered":"2024-04-26T23:07:47.817491+0000","last_clean":"2024-04-26T23:07:47.817491+0000","last_became_active":"2024-04-26T23:07:47.805537+0000","last_became_peered":"2024-04-26T23:07:47.805537+0000","last_unstale":"2024-04-26T23:07:47.817491+0000","last_undegraded":"2024-04-26T23:07:47.817491+0000","last_fullsized":"2024-04-26T23:07:47.817491+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:12:56.841587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.795838+0000","last_change":"2024-04-26T23:07:47.795838+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.795838+0000","last_undegraded":"2024-04-26T23:07:47.795838+0000","last_fullsized":"2024-04-26T23:07:47.795838+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.819889+0000","last_change":"2024-04-26T23:07:47.816254+0000","last_active":"2024-04-26T23:07:47.819889+0000","last_peered":"2024-04-26T23:07:47.819889+0000","last_clean":"2024-04-26T23:07:47.819889+0000","last_became_active":"2024-04-26T23:07:47.815567+0000","last_became_peered":"2024-04-26T23:07:47.815567+0000","last_unstale":"2024-04-26T23:07:47.819889+0000","last_undegraded":"2024-04-26T23:07:47.819889+0000","last_fullsized":"2024-04-26T23:07:47.819889+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:46.908552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.813165+0000","last_change":"2024-04-26T23:07:47.813165+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.813165+0000","last_undegraded":"2024-04-26T23:07:47.813165+0000","last_fullsized":"2024-04-26T23:07:47.813165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.831979+0000","last_change":"2024-04-26T23:07:47.825330+0000","last_active":"2024-04-26T23:07:47.831979+0000","last_peered":"2024-04-26T23:07:47.831979+0000","last_clean":"2024-04-26T23:07:47.831979+0000","last_became_active":"2024-04-26T23:07:47.825036+0000","last_became_peered":"2024-04-26T23:07:47.825036+0000","last_unstale":"2024-04-26T23:07:47.831979+0000","last_undegraded":"2024-04-26T23:07:47.831979+0000","last_fullsized":"2024-04-26T23:07:47.831979+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:13:16.015174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.797788+0000","last_change":"2024-04-26T23:07:47.797788+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.797788+0000","last_undegraded":"2024-04-26T23:07:47.797788+0000","last_fullsized":"2024-04-26T23:07:47.797788+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.827669+0000","last_change":"2024-04-26T23:07:47.822015+0000","last_active":"2024-04-26T23:07:47.827669+0000","last_peered":"2024-04-26T23:07:47.827669+0000","last_clean":"2024-04-26T23:07:47.827669+0000","last_became_active":"2024-04-26T23:07:47.820758+0000","last_became_peered":"2024-04-26T23:07:47.820758+0000","last_unstale":"2024-04-26T23:07:47.827669+0000","last_undegraded":"2024-04-26T23:07:47.827669+0000","last_fullsized":"2024-04-26T23:07:47.827669+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:33:27.821052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.798215+0000","last_change":"2024-04-26T23:07:47.798215+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.798215+0000","last_undegraded":"2024-04-26T23:07:47.798215+0000","last_fullsized":"2024-04-26T23:07:47.798215+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.829730+0000","last_change":"2024-04-26T23:07:47.825258+0000","last_active":"2024-04-26T23:07:47.829730+0000","last_peered":"2024-04-26T23:07:47.829730+0000","last_clean":"2024-04-26T23:07:47.829730+0000","last_became_active":"2024-04-26T23:07:47.824989+0000","last_became_peered":"2024-04-26T23:07:47.824989+0000","last_unstale":"2024-04-26T23:07:47.829730+0000","last_undegraded":"2024-04-26T23:07:47.829730+0000","last_fullsized":"2024-04-26T23:07:47.829730+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:13.303513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":88,"state":"creating+peering","last_fresh":"2024-04-26T23:07:47.813399+0000","last_change":"2024-04-26T23:07:47.813399+0000","last_active":"2024-04-26T23:07:47.771893+0000","last_peered":"2024-04-26T23:07:47.771893+0000","last_clean":"2024-04-26T23:07:47.771893+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T23:07:47.813399+0000","last_undegraded":"2024-04-26T23:07:47.813399+0000","last_fullsized":"2024-04-26T23:07:47.813399+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.827013+0000","last_change":"2024-04-26T23:07:47.822209+0000","last_active":"2024-04-26T23:07:47.827013+0000","last_peered":"2024-04-26T23:07:47.827013+0000","last_clean":"2024-04-26T23:07:47.827013+0000","last_became_active":"2024-04-26T23:07:47.821920+0000","last_became_peered":"2024-04-26T23:07:47.821920+0000","last_unstale":"2024-04-26T23:07:47.827013+0000","last_undegraded":"2024-04-26T23:07:47.827013+0000","last_fullsized":"2024-04-26T23:07:47.827013+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:46:06.085106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":300,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-26T23:07:47.813025+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:07:47.813025+0000","last_peered":"2024-04-26T23:07:47.813025+0000","last_clean":"2024-04-26T23:07:47.813025+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:07:47.813025+0000","last_undegraded":"2024-04-26T23:07:47.813025+0000","last_fullsized":"2024-04-26T23:07:47.813025+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32812,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339028,"statfs":{"total":96636764160,"available":96603164672,"internally_reserved":0,"allocated":1667072,"data_stored":1086221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339596,"statfs":{"total":96636764160,"available":96603746304,"internally_reserved":0,"allocated":1085440,"data_stored":502520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574933,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339612,"statfs":{"total":96636764160,"available":96603762688,"internally_reserved":0,"allocated":1073152,"data_stored":495853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574933,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339596,"statfs":{"total":96636764160,"available":96603746304,"internally_reserved":0,"allocated":1085440,"data_stored":502520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339612,"statfs":{"total":96636764160,"available":96603762688,"internally_reserved":0,"allocated":1073152,"data_stored":495853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339612,"statfs":{"total":96636764160,"available":96603762688,"internally_reserved":0,"allocated":1073152,"data_stored":495853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339612,"statfs":{"total":96636764160,"available":96603762688,"internally_reserved":0,"allocated":1073152,"data_stored":495853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1667072,"data_stored":1086221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:07:53.309 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:07:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.562+0000 7f336ae00640 1 Processor -- start 2024-04-26T23:07:53.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.562+0000 7f336ae00640 1 -- start start 2024-04-26T23:07:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.564+0000 7f336ae00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3364158cd0 0x7f3364159160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.564+0000 7f336ae00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33640cce10 con 0x7f33640cc980 2024-04-26T23:07:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.564+0000 7f336ae00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33640cd0f0 con 0x7f33640c6f10 2024-04-26T23:07:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.564+0000 7f336ae00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f33640c72f0 con 0x7f3364158cd0 2024-04-26T23:07:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.565+0000 7f336a400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3364158cd0 0x7f3364159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.565+0000 7f336a400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3364158cd0 0x7f3364159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44618/0 (socket says 172.21.15.40:44618) 2024-04-26T23:07:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.565+0000 7f3369000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f33640cc980 0x7f33640cbcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33486/0 (socket says 172.21.15.40:33486) 2024-04-26T23:07:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.565+0000 7f336a400640 1 -- 172.21.15.40:0/3313516553 learned_addr learned my addr 172.21.15.40:0/3313516553 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.565+0000 7f336a400640 1 -- 172.21.15.40:0/3313516553 >> v1:172.21.15.103:6789/0 conn(0x7f33640c6f10 legacy=0x7f33640c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.565+0000 7f336a400640 1 -- 172.21.15.40:0/3313516553 >> v1:172.21.15.40:6789/0 conn(0x7f33640cc980 legacy=0x7f33640cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.565+0000 7f336a400640 1 -- 172.21.15.40:0/3313516553 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33640cb4f0 con 0x7f3364158cd0 2024-04-26T23:07:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.566+0000 7f336a400640 1 --2- 172.21.15.40:0/3313516553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3364158cd0 0x7f3364159160 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f33640c7470 tx=0x7f335c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a86f8505c16ae49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.567+0000 7f3363e00640 1 -- 172.21.15.40:0/3313516553 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c036070 con 0x7f3364158cd0 2024-04-26T23:07:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.567+0000 7f3363e00640 1 -- 172.21.15.40:0/3313516553 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f335c031600 con 0x7f3364158cd0 2024-04-26T23:07:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.567+0000 7f3363e00640 1 -- 172.21.15.40:0/3313516553 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c0319d0 con 0x7f3364158cd0 2024-04-26T23:07:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.567+0000 7f336ae00640 1 -- 172.21.15.40:0/3313516553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3364158cd0 msgr2=0x7f3364159160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.567+0000 7f336ae00640 1 --2- 172.21.15.40:0/3313516553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3364158cd0 0x7f3364159160 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f33640c7470 tx=0x7f335c02a710 comp rx=0 tx=0).stop 2024-04-26T23:07:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.568+0000 7f336ae00640 1 -- 172.21.15.40:0/3313516553 shutdown_connections 2024-04-26T23:07:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.568+0000 7f336ae00640 1 --2- 172.21.15.40:0/3313516553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3364158cd0 0x7f3364159160 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.568+0000 7f336ae00640 1 -- 172.21.15.40:0/3313516553 >> 172.21.15.40:0/3313516553 conn(0x7f33640d0a10 msgr2=0x7f33640d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.568+0000 7f336ae00640 1 -- 172.21.15.40:0/3313516553 shutdown_connections 2024-04-26T23:07:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.568+0000 7f336ae00640 1 -- 172.21.15.40:0/3313516553 wait complete. 2024-04-26T23:07:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.570+0000 7f336ae00640 1 Processor -- start 2024-04-26T23:07:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.570+0000 7f336ae00640 1 -- start start 2024-04-26T23:07:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.571+0000 7f336ae00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33640c6f10 0x7f336414b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.571+0000 7f336ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3364158cd0 0x7f33640b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.571+0000 7f336ae00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33640b1e00 0x7f33640c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.571+0000 7f336ae00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f33640c61e0 con 0x7f33640c6f10 2024-04-26T23:07:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.571+0000 7f336ae00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33640cb860 con 0x7f33640b1e00 2024-04-26T23:07:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.571+0000 7f336ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f33640c6360 con 0x7f3364158cd0 2024-04-26T23:07:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.572+0000 7f3369a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33640c6f10 0x7f336414b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.572+0000 7f3369a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33640c6f10 0x7f336414b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44620/0 (socket says 172.21.15.40:44620) 2024-04-26T23:07:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.572+0000 7f3369a00640 1 -- 172.21.15.40:0/431352123 learned_addr learned my addr 172.21.15.40:0/431352123 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.572+0000 7f3369a00640 1 -- 172.21.15.40:0/431352123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3364158cd0 msgr2=0x7f33640b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.572+0000 7f3369a00640 1 --2- 172.21.15.40:0/431352123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3364158cd0 0x7f33640b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.572+0000 7f3369a00640 1 -- 172.21.15.40:0/431352123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33640b1e00 msgr2=0x7f33640c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.572+0000 7f336a400640 1 --2- 172.21.15.40:0/431352123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33640b1e00 0x7f33640c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.572+0000 7f3369a00640 1 --2- 172.21.15.40:0/431352123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33640b1e00 0x7f33640c7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.572+0000 7f3369a00640 1 -- 172.21.15.40:0/431352123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3364156690 con 0x7f33640c6f10 2024-04-26T23:07:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.573+0000 7f3369a00640 1 --2- 172.21.15.40:0/431352123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33640c6f10 0x7f336414b870 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f335800c450 tx=0x7f335800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.573+0000 7f3361600640 1 -- 172.21.15.40:0/431352123 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33580052d0 con 0x7f33640c6f10 2024-04-26T23:07:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.573+0000 7f3361600640 1 -- 172.21.15.40:0/431352123 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3358005470 con 0x7f33640c6f10 2024-04-26T23:07:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.573+0000 7f3361600640 1 -- 172.21.15.40:0/431352123 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3358012870 con 0x7f33640c6f10 2024-04-26T23:07:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.573+0000 7f336ae00640 1 -- 172.21.15.40:0/431352123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f336414bdb0 con 0x7f33640c6f10 2024-04-26T23:07:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.573+0000 7f336ae00640 1 -- 172.21.15.40:0/431352123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33640b0b00 con 0x7f33640c6f10 2024-04-26T23:07:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.574+0000 7f336ae00640 1 -- 172.21.15.40:0/431352123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3364069370 con 0x7f33640c6f10 2024-04-26T23:07:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.576+0000 7f3361600640 1 -- 172.21.15.40:0/431352123 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3358012a10 con 0x7f33640c6f10 2024-04-26T23:07:53.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.577+0000 7f3361600640 1 --2- 172.21.15.40:0/431352123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f334406c8f0 0x7f334406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:53.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.577+0000 7f3361600640 1 -- 172.21.15.40:0/431352123 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f335808b920 con 0x7f33640c6f10 2024-04-26T23:07:53.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.578+0000 7f3369000640 1 --2- 172.21.15.40:0/431352123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f334406c8f0 0x7f334406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:53.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.579+0000 7f3369000640 1 --2- 172.21.15.40:0/431352123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f334406c8f0 0x7f334406edb0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3364155f00 tx=0x7f33500079c0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.582+0000 7f3361600640 1 -- 172.21.15.40:0/431352123 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3358059bb0 con 0x7f33640c6f10 2024-04-26T23:07:53.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.782+0000 7f336ae00640 1 -- 172.21.15.40:0/431352123 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f33640c58b0 con 0x7f334406c8f0 2024-04-26T23:07:53.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.801+0000 7f3361600640 1 -- 172.21.15.40:0/431352123 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252079 (secure 0 0 0) 0x7f33640c58b0 con 0x7f334406c8f0 2024-04-26T23:07:53.805 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:53.806 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:07:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.807+0000 7f336ae00640 1 -- 172.21.15.40:0/431352123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f334406c8f0 msgr2=0x7f334406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.807+0000 7f336ae00640 1 --2- 172.21.15.40:0/431352123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f334406c8f0 0x7f334406edb0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3364155f00 tx=0x7f33500079c0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.807+0000 7f336ae00640 1 -- 172.21.15.40:0/431352123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33640c6f10 msgr2=0x7f336414b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.807+0000 7f336ae00640 1 --2- 172.21.15.40:0/431352123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33640c6f10 0x7f336414b870 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f335800c450 tx=0x7f335800c910 comp rx=0 tx=0).stop 2024-04-26T23:07:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.808+0000 7f336ae00640 1 -- 172.21.15.40:0/431352123 shutdown_connections 2024-04-26T23:07:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.808+0000 7f336ae00640 1 --2- 172.21.15.40:0/431352123 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f334406c8f0 0x7f334406edb0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.808+0000 7f336ae00640 1 --2- 172.21.15.40:0/431352123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33640b1e00 0x7f33640c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.808+0000 7f336ae00640 1 --2- 172.21.15.40:0/431352123 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3364158cd0 0x7f33640b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.808+0000 7f336ae00640 1 --2- 172.21.15.40:0/431352123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33640c6f10 0x7f336414b870 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.808+0000 7f336ae00640 1 -- 172.21.15.40:0/431352123 >> 172.21.15.40:0/431352123 conn(0x7f33640d0a10 msgr2=0x7f33640af550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.808+0000 7f336ae00640 1 -- 172.21.15.40:0/431352123 shutdown_connections 2024-04-26T23:07:53.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:53.808+0000 7f336ae00640 1 -- 172.21.15.40:0/431352123 wait complete. 2024-04-26T23:07:53.826 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-04-26T23:07:53.296101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259364,"kb_used_data":9896,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715356,"statfs":{"total":773094113280,"available":772828524544,"internally_reserved":0,"allocated":10133504,"data_stored":5316588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001810"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.784633+0000","last_change":"2024-04-26T23:07:48.829109+0000","last_active":"2024-04-26T23:07:49.784633+0000","last_peered":"2024-04-26T23:07:49.784633+0000","last_clean":"2024-04-26T23:07:49.784633+0000","last_became_active":"2024-04-26T23:07:48.828808+0000","last_became_peered":"2024-04-26T23:07:48.828808+0000","last_unstale":"2024-04-26T23:07:49.784633+0000","last_undegraded":"2024-04-26T23:07:49.784633+0000","last_fullsized":"2024-04-26T23:07:49.784633+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:10.646343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.829140+0000","last_change":"2024-04-26T23:07:48.815423+0000","last_active":"2024-04-26T23:07:49.829140+0000","last_peered":"2024-04-26T23:07:49.829140+0000","last_clean":"2024-04-26T23:07:49.829140+0000","last_became_active":"2024-04-26T23:07:48.814935+0000","last_became_peered":"2024-04-26T23:07:48.814935+0000","last_unstale":"2024-04-26T23:07:49.829140+0000","last_undegraded":"2024-04-26T23:07:49.829140+0000","last_fullsized":"2024-04-26T23:07:49.829140+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:22:19.868778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.784101+0000","last_change":"2024-04-26T23:07:48.810694+0000","last_active":"2024-04-26T23:07:49.784101+0000","last_peered":"2024-04-26T23:07:49.784101+0000","last_clean":"2024-04-26T23:07:49.784101+0000","last_became_active":"2024-04-26T23:07:48.810298+0000","last_became_peered":"2024-04-26T23:07:48.810298+0000","last_unstale":"2024-04-26T23:07:49.784101+0000","last_undegraded":"2024-04-26T23:07:49.784101+0000","last_fullsized":"2024-04-26T23:07:49.784101+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:38:14.958205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.331599+0000","last_change":"2024-04-26T23:07:48.816887+0000","last_active":"2024-04-26T23:07:50.331599+0000","last_peered":"2024-04-26T23:07:50.331599+0000","last_clean":"2024-04-26T23:07:50.331599+0000","last_became_active":"2024-04-26T23:07:48.816600+0000","last_became_peered":"2024-04-26T23:07:48.816600+0000","last_unstale":"2024-04-26T23:07:50.331599+0000","last_undegraded":"2024-04-26T23:07:50.331599+0000","last_fullsized":"2024-04-26T23:07:50.331599+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:26:40.227555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.822751+0000","last_change":"2024-04-26T23:07:48.808425+0000","last_active":"2024-04-26T23:07:49.822751+0000","last_peered":"2024-04-26T23:07:49.822751+0000","last_clean":"2024-04-26T23:07:49.822751+0000","last_became_active":"2024-04-26T23:07:48.808115+0000","last_became_peered":"2024-04-26T23:07:48.808115+0000","last_unstale":"2024-04-26T23:07:49.822751+0000","last_undegraded":"2024-04-26T23:07:49.822751+0000","last_fullsized":"2024-04-26T23:07:49.822751+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:14.319444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.326512+0000","last_change":"2024-04-26T23:07:48.828350+0000","last_active":"2024-04-26T23:07:50.326512+0000","last_peered":"2024-04-26T23:07:50.326512+0000","last_clean":"2024-04-26T23:07:50.326512+0000","last_became_active":"2024-04-26T23:07:48.827700+0000","last_became_peered":"2024-04-26T23:07:48.827700+0000","last_unstale":"2024-04-26T23:07:50.326512+0000","last_undegraded":"2024-04-26T23:07:50.326512+0000","last_fullsized":"2024-04-26T23:07:50.326512+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:11:03.573840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.787573+0000","last_change":"2024-04-26T23:07:48.803399+0000","last_active":"2024-04-26T23:07:49.787573+0000","last_peered":"2024-04-26T23:07:49.787573+0000","last_clean":"2024-04-26T23:07:49.787573+0000","last_became_active":"2024-04-26T23:07:48.802236+0000","last_became_peered":"2024-04-26T23:07:48.802236+0000","last_unstale":"2024-04-26T23:07:49.787573+0000","last_undegraded":"2024-04-26T23:07:49.787573+0000","last_fullsized":"2024-04-26T23:07:49.787573+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:24:37.083947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.328866+0000","last_change":"2024-04-26T23:07:48.827381+0000","last_active":"2024-04-26T23:07:50.328866+0000","last_peered":"2024-04-26T23:07:50.328866+0000","last_clean":"2024-04-26T23:07:50.328866+0000","last_became_active":"2024-04-26T23:07:48.822878+0000","last_became_peered":"2024-04-26T23:07:48.822878+0000","last_unstale":"2024-04-26T23:07:50.328866+0000","last_undegraded":"2024-04-26T23:07:50.328866+0000","last_fullsized":"2024-04-26T23:07:50.328866+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:04:49.501126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.823404+0000","last_change":"2024-04-26T23:07:48.813919+0000","last_active":"2024-04-26T23:07:49.823404+0000","last_peered":"2024-04-26T23:07:49.823404+0000","last_clean":"2024-04-26T23:07:49.823404+0000","last_became_active":"2024-04-26T23:07:48.813484+0000","last_became_peered":"2024-04-26T23:07:48.813484+0000","last_unstale":"2024-04-26T23:07:49.823404+0000","last_undegraded":"2024-04-26T23:07:49.823404+0000","last_fullsized":"2024-04-26T23:07:49.823404+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:15:59.131561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.332147+0000","last_change":"2024-04-26T23:07:48.828720+0000","last_active":"2024-04-26T23:07:50.332147+0000","last_peered":"2024-04-26T23:07:50.332147+0000","last_clean":"2024-04-26T23:07:50.332147+0000","last_became_active":"2024-04-26T23:07:48.825805+0000","last_became_peered":"2024-04-26T23:07:48.825805+0000","last_unstale":"2024-04-26T23:07:50.332147+0000","last_undegraded":"2024-04-26T23:07:50.332147+0000","last_fullsized":"2024-04-26T23:07:50.332147+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:42.522405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.823296+0000","last_change":"2024-04-26T23:07:48.834488+0000","last_active":"2024-04-26T23:07:49.823296+0000","last_peered":"2024-04-26T23:07:49.823296+0000","last_clean":"2024-04-26T23:07:49.823296+0000","last_became_active":"2024-04-26T23:07:48.834081+0000","last_became_peered":"2024-04-26T23:07:48.834081+0000","last_unstale":"2024-04-26T23:07:49.823296+0000","last_undegraded":"2024-04-26T23:07:49.823296+0000","last_fullsized":"2024-04-26T23:07:49.823296+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:26:40.227555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.331163+0000","last_change":"2024-04-26T23:07:48.804754+0000","last_active":"2024-04-26T23:07:50.331163+0000","last_peered":"2024-04-26T23:07:50.331163+0000","last_clean":"2024-04-26T23:07:50.331163+0000","last_became_active":"2024-04-26T23:07:48.804105+0000","last_became_peered":"2024-04-26T23:07:48.804105+0000","last_unstale":"2024-04-26T23:07:50.331163+0000","last_undegraded":"2024-04-26T23:07:50.331163+0000","last_fullsized":"2024-04-26T23:07:50.331163+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:45:29.677720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.328775+0000","last_change":"2024-04-26T23:07:48.826575+0000","last_active":"2024-04-26T23:07:50.328775+0000","last_peered":"2024-04-26T23:07:50.328775+0000","last_clean":"2024-04-26T23:07:50.328775+0000","last_became_active":"2024-04-26T23:07:48.825378+0000","last_became_peered":"2024-04-26T23:07:48.825378+0000","last_unstale":"2024-04-26T23:07:50.328775+0000","last_undegraded":"2024-04-26T23:07:50.328775+0000","last_fullsized":"2024-04-26T23:07:50.328775+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:10:06.703880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.822328+0000","last_change":"2024-04-26T23:07:48.825104+0000","last_active":"2024-04-26T23:07:49.822328+0000","last_peered":"2024-04-26T23:07:49.822328+0000","last_clean":"2024-04-26T23:07:49.822328+0000","last_became_active":"2024-04-26T23:07:48.824751+0000","last_became_peered":"2024-04-26T23:07:48.824751+0000","last_unstale":"2024-04-26T23:07:49.822328+0000","last_undegraded":"2024-04-26T23:07:49.822328+0000","last_fullsized":"2024-04-26T23:07:49.822328+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:00:14.910605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.328978+0000","last_change":"2024-04-26T23:07:48.802358+0000","last_active":"2024-04-26T23:07:50.328978+0000","last_peered":"2024-04-26T23:07:50.328978+0000","last_clean":"2024-04-26T23:07:50.328978+0000","last_became_active":"2024-04-26T23:07:48.801876+0000","last_became_peered":"2024-04-26T23:07:48.801876+0000","last_unstale":"2024-04-26T23:07:50.328978+0000","last_undegraded":"2024-04-26T23:07:50.328978+0000","last_fullsized":"2024-04-26T23:07:50.328978+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:00:14.910605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.326175+0000","last_change":"2024-04-26T23:07:48.805423+0000","last_active":"2024-04-26T23:07:50.326175+0000","last_peered":"2024-04-26T23:07:50.326175+0000","last_clean":"2024-04-26T23:07:50.326175+0000","last_became_active":"2024-04-26T23:07:48.802769+0000","last_became_peered":"2024-04-26T23:07:48.802769+0000","last_unstale":"2024-04-26T23:07:50.326175+0000","last_undegraded":"2024-04-26T23:07:50.326175+0000","last_fullsized":"2024-04-26T23:07:50.326175+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:33:42.345870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.328954+0000","last_change":"2024-04-26T23:07:48.814063+0000","last_active":"2024-04-26T23:07:50.328954+0000","last_peered":"2024-04-26T23:07:50.328954+0000","last_clean":"2024-04-26T23:07:50.328954+0000","last_became_active":"2024-04-26T23:07:48.813780+0000","last_became_peered":"2024-04-26T23:07:48.813780+0000","last_unstale":"2024-04-26T23:07:50.328954+0000","last_undegraded":"2024-04-26T23:07:50.328954+0000","last_fullsized":"2024-04-26T23:07:50.328954+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:16:56.768664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.784300+0000","last_change":"2024-04-26T23:07:48.812247+0000","last_active":"2024-04-26T23:07:49.784300+0000","last_peered":"2024-04-26T23:07:49.784300+0000","last_clean":"2024-04-26T23:07:49.784300+0000","last_became_active":"2024-04-26T23:07:48.811835+0000","last_became_peered":"2024-04-26T23:07:48.811835+0000","last_unstale":"2024-04-26T23:07:49.784300+0000","last_undegraded":"2024-04-26T23:07:49.784300+0000","last_fullsized":"2024-04-26T23:07:49.784300+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:37:43.232658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.823416+0000","last_change":"2024-04-26T23:07:48.825471+0000","last_active":"2024-04-26T23:07:49.823416+0000","last_peered":"2024-04-26T23:07:49.823416+0000","last_clean":"2024-04-26T23:07:49.823416+0000","last_became_active":"2024-04-26T23:07:48.825167+0000","last_became_peered":"2024-04-26T23:07:48.825167+0000","last_unstale":"2024-04-26T23:07:49.823416+0000","last_undegraded":"2024-04-26T23:07:49.823416+0000","last_fullsized":"2024-04-26T23:07:49.823416+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:32:34.278788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.331103+0000","last_change":"2024-04-26T23:07:48.816643+0000","last_active":"2024-04-26T23:07:50.331103+0000","last_peered":"2024-04-26T23:07:50.331103+0000","last_clean":"2024-04-26T23:07:50.331103+0000","last_became_active":"2024-04-26T23:07:48.816337+0000","last_became_peered":"2024-04-26T23:07:48.816337+0000","last_unstale":"2024-04-26T23:07:50.331103+0000","last_undegraded":"2024-04-26T23:07:50.331103+0000","last_fullsized":"2024-04-26T23:07:50.331103+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:53.765394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.822546+0000","last_change":"2024-04-26T23:07:48.813961+0000","last_active":"2024-04-26T23:07:49.822546+0000","last_peered":"2024-04-26T23:07:49.822546+0000","last_clean":"2024-04-26T23:07:49.822546+0000","last_became_active":"2024-04-26T23:07:48.812994+0000","last_became_peered":"2024-04-26T23:07:48.812994+0000","last_unstale":"2024-04-26T23:07:49.822546+0000","last_undegraded":"2024-04-26T23:07:49.822546+0000","last_fullsized":"2024-04-26T23:07:49.822546+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:45.064250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.822330+0000","last_change":"2024-04-26T23:07:48.824980+0000","last_active":"2024-04-26T23:07:49.822330+0000","last_peered":"2024-04-26T23:07:49.822330+0000","last_clean":"2024-04-26T23:07:49.822330+0000","last_became_active":"2024-04-26T23:07:48.824590+0000","last_became_peered":"2024-04-26T23:07:48.824590+0000","last_unstale":"2024-04-26T23:07:49.822330+0000","last_undegraded":"2024-04-26T23:07:49.822330+0000","last_fullsized":"2024-04-26T23:07:49.822330+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:36:03.378820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.788767+0000","last_change":"2024-04-26T23:07:48.811514+0000","last_active":"2024-04-26T23:07:49.788767+0000","last_peered":"2024-04-26T23:07:49.788767+0000","last_clean":"2024-04-26T23:07:49.788767+0000","last_became_active":"2024-04-26T23:07:48.811233+0000","last_became_peered":"2024-04-26T23:07:48.811233+0000","last_unstale":"2024-04-26T23:07:49.788767+0000","last_undegraded":"2024-04-26T23:07:49.788767+0000","last_fullsized":"2024-04-26T23:07:49.788767+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:36:03.378820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.784085+0000","last_change":"2024-04-26T23:07:48.814756+0000","last_active":"2024-04-26T23:07:49.784085+0000","last_peered":"2024-04-26T23:07:49.784085+0000","last_clean":"2024-04-26T23:07:49.784085+0000","last_became_active":"2024-04-26T23:07:48.814497+0000","last_became_peered":"2024-04-26T23:07:48.814497+0000","last_unstale":"2024-04-26T23:07:49.784085+0000","last_undegraded":"2024-04-26T23:07:49.784085+0000","last_fullsized":"2024-04-26T23:07:49.784085+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:36.407430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.829382+0000","last_change":"2024-04-26T23:07:48.810274+0000","last_active":"2024-04-26T23:07:49.829382+0000","last_peered":"2024-04-26T23:07:49.829382+0000","last_clean":"2024-04-26T23:07:49.829382+0000","last_became_active":"2024-04-26T23:07:48.807989+0000","last_became_peered":"2024-04-26T23:07:48.807989+0000","last_unstale":"2024-04-26T23:07:49.829382+0000","last_undegraded":"2024-04-26T23:07:49.829382+0000","last_fullsized":"2024-04-26T23:07:49.829382+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:32:34.278788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.823311+0000","last_change":"2024-04-26T23:07:48.825292+0000","last_active":"2024-04-26T23:07:49.823311+0000","last_peered":"2024-04-26T23:07:49.823311+0000","last_clean":"2024-04-26T23:07:49.823311+0000","last_became_active":"2024-04-26T23:07:48.824789+0000","last_became_peered":"2024-04-26T23:07:48.824789+0000","last_unstale":"2024-04-26T23:07:49.823311+0000","last_undegraded":"2024-04-26T23:07:49.823311+0000","last_fullsized":"2024-04-26T23:07:49.823311+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:22:19.868778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.331305+0000","last_change":"2024-04-26T23:07:48.808304+0000","last_active":"2024-04-26T23:07:50.331305+0000","last_peered":"2024-04-26T23:07:50.331305+0000","last_clean":"2024-04-26T23:07:50.331305+0000","last_became_active":"2024-04-26T23:07:48.808023+0000","last_became_peered":"2024-04-26T23:07:48.808023+0000","last_unstale":"2024-04-26T23:07:50.331305+0000","last_undegraded":"2024-04-26T23:07:50.331305+0000","last_fullsized":"2024-04-26T23:07:50.331305+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:18.751358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.331517+0000","last_change":"2024-04-26T23:07:48.827591+0000","last_active":"2024-04-26T23:07:50.331517+0000","last_peered":"2024-04-26T23:07:50.331517+0000","last_clean":"2024-04-26T23:07:50.331517+0000","last_became_active":"2024-04-26T23:07:48.827297+0000","last_became_peered":"2024-04-26T23:07:48.827297+0000","last_unstale":"2024-04-26T23:07:50.331517+0000","last_undegraded":"2024-04-26T23:07:50.331517+0000","last_fullsized":"2024-04-26T23:07:50.331517+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:19:41.586767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.789170+0000","last_change":"2024-04-26T23:07:48.807397+0000","last_active":"2024-04-26T23:07:49.789170+0000","last_peered":"2024-04-26T23:07:49.789170+0000","last_clean":"2024-04-26T23:07:49.789170+0000","last_became_active":"2024-04-26T23:07:48.807045+0000","last_became_peered":"2024-04-26T23:07:48.807045+0000","last_unstale":"2024-04-26T23:07:49.789170+0000","last_undegraded":"2024-04-26T23:07:49.789170+0000","last_fullsized":"2024-04-26T23:07:49.789170+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:16.828842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.823137+0000","last_change":"2024-04-26T23:07:48.814565+0000","last_active":"2024-04-26T23:07:49.823137+0000","last_peered":"2024-04-26T23:07:49.823137+0000","last_clean":"2024-04-26T23:07:49.823137+0000","last_became_active":"2024-04-26T23:07:48.814243+0000","last_became_peered":"2024-04-26T23:07:48.814243+0000","last_unstale":"2024-04-26T23:07:49.823137+0000","last_undegraded":"2024-04-26T23:07:49.823137+0000","last_fullsized":"2024-04-26T23:07:49.823137+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:19:41.586767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.828380+0000","last_change":"2024-04-26T23:07:48.824953+0000","last_active":"2024-04-26T23:07:49.828380+0000","last_peered":"2024-04-26T23:07:49.828380+0000","last_clean":"2024-04-26T23:07:49.828380+0000","last_became_active":"2024-04-26T23:07:48.824560+0000","last_became_peered":"2024-04-26T23:07:48.824560+0000","last_unstale":"2024-04-26T23:07:49.828380+0000","last_undegraded":"2024-04-26T23:07:49.828380+0000","last_fullsized":"2024-04-26T23:07:49.828380+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:33:02.623151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.326359+0000","last_change":"2024-04-26T23:07:48.803059+0000","last_active":"2024-04-26T23:07:50.326359+0000","last_peered":"2024-04-26T23:07:50.326359+0000","last_clean":"2024-04-26T23:07:50.326359+0000","last_became_active":"2024-04-26T23:07:48.802415+0000","last_became_peered":"2024-04-26T23:07:48.802415+0000","last_unstale":"2024-04-26T23:07:50.326359+0000","last_undegraded":"2024-04-26T23:07:50.326359+0000","last_fullsized":"2024-04-26T23:07:50.326359+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:41:09.667932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.326935+0000","last_change":"2024-04-26T23:07:48.811635+0000","last_active":"2024-04-26T23:07:50.326935+0000","last_peered":"2024-04-26T23:07:50.326935+0000","last_clean":"2024-04-26T23:07:50.326935+0000","last_became_active":"2024-04-26T23:07:48.811135+0000","last_became_peered":"2024-04-26T23:07:48.811135+0000","last_unstale":"2024-04-26T23:07:50.326935+0000","last_undegraded":"2024-04-26T23:07:50.326935+0000","last_fullsized":"2024-04-26T23:07:50.326935+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:06:24.519190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"89'4","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.790551+0000","last_change":"2024-04-26T23:07:47.809702+0000","last_active":"2024-04-26T23:07:49.790551+0000","last_peered":"2024-04-26T23:07:49.790551+0000","last_clean":"2024-04-26T23:07:49.790551+0000","last_became_active":"2024-04-26T23:07:47.807055+0000","last_became_peered":"2024-04-26T23:07:47.807055+0000","last_unstale":"2024-04-26T23:07:49.790551+0000","last_undegraded":"2024-04-26T23:07:49.790551+0000","last_fullsized":"2024-04-26T23:07:49.790551+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:53.581190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.328545+0000","last_change":"2024-04-26T23:07:48.807376+0000","last_active":"2024-04-26T23:07:50.328545+0000","last_peered":"2024-04-26T23:07:50.328545+0000","last_clean":"2024-04-26T23:07:50.328545+0000","last_became_active":"2024-04-26T23:07:48.807094+0000","last_became_peered":"2024-04-26T23:07:48.807094+0000","last_unstale":"2024-04-26T23:07:50.328545+0000","last_undegraded":"2024-04-26T23:07:50.328545+0000","last_fullsized":"2024-04-26T23:07:50.328545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:02:52.154681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"89'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.823582+0000","last_change":"2024-04-26T23:07:47.821601+0000","last_active":"2024-04-26T23:07:49.823582+0000","last_peered":"2024-04-26T23:07:49.823582+0000","last_clean":"2024-04-26T23:07:49.823582+0000","last_became_active":"2024-04-26T23:07:47.821310+0000","last_became_peered":"2024-04-26T23:07:47.821310+0000","last_unstale":"2024-04-26T23:07:49.823582+0000","last_undegraded":"2024-04-26T23:07:49.823582+0000","last_fullsized":"2024-04-26T23:07:49.823582+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:07:57.468174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.789992+0000","last_change":"2024-04-26T23:07:48.816604+0000","last_active":"2024-04-26T23:07:49.789992+0000","last_peered":"2024-04-26T23:07:49.789992+0000","last_clean":"2024-04-26T23:07:49.789992+0000","last_became_active":"2024-04-26T23:07:48.815996+0000","last_became_peered":"2024-04-26T23:07:48.815996+0000","last_unstale":"2024-04-26T23:07:49.789992+0000","last_undegraded":"2024-04-26T23:07:49.789992+0000","last_fullsized":"2024-04-26T23:07:49.789992+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:36.407430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"89'2","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.789776+0000","last_change":"2024-04-26T23:07:47.827222+0000","last_active":"2024-04-26T23:07:49.789776+0000","last_peered":"2024-04-26T23:07:49.789776+0000","last_clean":"2024-04-26T23:07:49.789776+0000","last_became_active":"2024-04-26T23:07:47.826726+0000","last_became_peered":"2024-04-26T23:07:47.826726+0000","last_unstale":"2024-04-26T23:07:49.789776+0000","last_undegraded":"2024-04-26T23:07:49.789776+0000","last_fullsized":"2024-04-26T23:07:49.789776+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:46.908552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.329406+0000","last_change":"2024-04-26T23:07:48.799410+0000","last_active":"2024-04-26T23:07:50.329406+0000","last_peered":"2024-04-26T23:07:50.329406+0000","last_clean":"2024-04-26T23:07:50.329406+0000","last_became_active":"2024-04-26T23:07:48.799074+0000","last_became_peered":"2024-04-26T23:07:48.799074+0000","last_unstale":"2024-04-26T23:07:50.329406+0000","last_undegraded":"2024-04-26T23:07:50.329406+0000","last_fullsized":"2024-04-26T23:07:50.329406+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:11:14.226560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"89'2","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.789999+0000","last_change":"2024-04-26T23:07:47.817334+0000","last_active":"2024-04-26T23:07:49.789999+0000","last_peered":"2024-04-26T23:07:49.789999+0000","last_clean":"2024-04-26T23:07:49.789999+0000","last_became_active":"2024-04-26T23:07:47.816968+0000","last_became_peered":"2024-04-26T23:07:47.816968+0000","last_unstale":"2024-04-26T23:07:49.789999+0000","last_undegraded":"2024-04-26T23:07:49.789999+0000","last_fullsized":"2024-04-26T23:07:49.789999+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:54:39.836546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.329352+0000","last_change":"2024-04-26T23:07:48.815609+0000","last_active":"2024-04-26T23:07:50.329352+0000","last_peered":"2024-04-26T23:07:50.329352+0000","last_clean":"2024-04-26T23:07:50.329352+0000","last_became_active":"2024-04-26T23:07:48.813901+0000","last_became_peered":"2024-04-26T23:07:48.813901+0000","last_unstale":"2024-04-26T23:07:50.329352+0000","last_undegraded":"2024-04-26T23:07:50.329352+0000","last_fullsized":"2024-04-26T23:07:50.329352+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:36:03.378820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"89'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.823394+0000","last_change":"2024-04-26T23:07:47.820427+0000","last_active":"2024-04-26T23:07:49.823394+0000","last_peered":"2024-04-26T23:07:49.823394+0000","last_clean":"2024-04-26T23:07:49.823394+0000","last_became_active":"2024-04-26T23:07:47.820082+0000","last_became_peered":"2024-04-26T23:07:47.820082+0000","last_unstale":"2024-04-26T23:07:49.823394+0000","last_undegraded":"2024-04-26T23:07:49.823394+0000","last_fullsized":"2024-04-26T23:07:49.823394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:21.116738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.789245+0000","last_change":"2024-04-26T23:07:48.800796+0000","last_active":"2024-04-26T23:07:49.789245+0000","last_peered":"2024-04-26T23:07:49.789245+0000","last_clean":"2024-04-26T23:07:49.789245+0000","last_became_active":"2024-04-26T23:07:48.800495+0000","last_became_peered":"2024-04-26T23:07:48.800495+0000","last_unstale":"2024-04-26T23:07:49.789245+0000","last_undegraded":"2024-04-26T23:07:49.789245+0000","last_fullsized":"2024-04-26T23:07:49.789245+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:14.319444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"89'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.330474+0000","last_change":"2024-04-26T23:07:47.823971+0000","last_active":"2024-04-26T23:07:50.330474+0000","last_peered":"2024-04-26T23:07:50.330474+0000","last_clean":"2024-04-26T23:07:50.330474+0000","last_became_active":"2024-04-26T23:07:47.821435+0000","last_became_peered":"2024-04-26T23:07:47.821435+0000","last_unstale":"2024-04-26T23:07:50.330474+0000","last_undegraded":"2024-04-26T23:07:50.330474+0000","last_fullsized":"2024-04-26T23:07:50.330474+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:21.116738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.326985+0000","last_change":"2024-04-26T23:07:48.827984+0000","last_active":"2024-04-26T23:07:50.326985+0000","last_peered":"2024-04-26T23:07:50.326985+0000","last_clean":"2024-04-26T23:07:50.326985+0000","last_became_active":"2024-04-26T23:07:48.827179+0000","last_became_peered":"2024-04-26T23:07:48.827179+0000","last_unstale":"2024-04-26T23:07:50.326985+0000","last_undegraded":"2024-04-26T23:07:50.326985+0000","last_fullsized":"2024-04-26T23:07:50.326985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:57.388694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.784292+0000","last_change":"2024-04-26T23:07:47.822575+0000","last_active":"2024-04-26T23:07:49.784292+0000","last_peered":"2024-04-26T23:07:49.784292+0000","last_clean":"2024-04-26T23:07:49.784292+0000","last_became_active":"2024-04-26T23:07:47.820316+0000","last_became_peered":"2024-04-26T23:07:47.820316+0000","last_unstale":"2024-04-26T23:07:49.784292+0000","last_undegraded":"2024-04-26T23:07:49.784292+0000","last_fullsized":"2024-04-26T23:07:49.784292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:16:08.827822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.330554+0000","last_change":"2024-04-26T23:07:48.808633+0000","last_active":"2024-04-26T23:07:50.330554+0000","last_peered":"2024-04-26T23:07:50.330554+0000","last_clean":"2024-04-26T23:07:50.330554+0000","last_became_active":"2024-04-26T23:07:48.808256+0000","last_became_peered":"2024-04-26T23:07:48.808256+0000","last_unstale":"2024-04-26T23:07:50.330554+0000","last_undegraded":"2024-04-26T23:07:50.330554+0000","last_fullsized":"2024-04-26T23:07:50.330554+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:14.319444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.787509+0000","last_change":"2024-04-26T23:07:47.821897+0000","last_active":"2024-04-26T23:07:49.787509+0000","last_peered":"2024-04-26T23:07:49.787509+0000","last_clean":"2024-04-26T23:07:49.787509+0000","last_became_active":"2024-04-26T23:07:47.820410+0000","last_became_peered":"2024-04-26T23:07:47.820410+0000","last_unstale":"2024-04-26T23:07:49.787509+0000","last_undegraded":"2024-04-26T23:07:49.787509+0000","last_fullsized":"2024-04-26T23:07:49.787509+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:26:39.211624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.823077+0000","last_change":"2024-04-26T23:07:48.811538+0000","last_active":"2024-04-26T23:07:49.823077+0000","last_peered":"2024-04-26T23:07:49.823077+0000","last_clean":"2024-04-26T23:07:49.823077+0000","last_became_active":"2024-04-26T23:07:48.811263+0000","last_became_peered":"2024-04-26T23:07:48.811263+0000","last_unstale":"2024-04-26T23:07:49.823077+0000","last_undegraded":"2024-04-26T23:07:49.823077+0000","last_fullsized":"2024-04-26T23:07:49.823077+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:33:02.623151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.829599+0000","last_change":"2024-04-26T23:07:47.827901+0000","last_active":"2024-04-26T23:07:49.829599+0000","last_peered":"2024-04-26T23:07:49.829599+0000","last_clean":"2024-04-26T23:07:49.829599+0000","last_became_active":"2024-04-26T23:07:47.827572+0000","last_became_peered":"2024-04-26T23:07:47.827572+0000","last_unstale":"2024-04-26T23:07:49.829599+0000","last_undegraded":"2024-04-26T23:07:49.829599+0000","last_fullsized":"2024-04-26T23:07:49.829599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:10:05.687949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.789117+0000","last_change":"2024-04-26T23:07:48.813042+0000","last_active":"2024-04-26T23:07:49.789117+0000","last_peered":"2024-04-26T23:07:49.789117+0000","last_clean":"2024-04-26T23:07:49.789117+0000","last_became_active":"2024-04-26T23:07:48.812645+0000","last_became_peered":"2024-04-26T23:07:48.812645+0000","last_unstale":"2024-04-26T23:07:49.789117+0000","last_undegraded":"2024-04-26T23:07:49.789117+0000","last_fullsized":"2024-04-26T23:07:49.789117+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:51:21.877621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.784433+0000","last_change":"2024-04-26T23:07:47.821054+0000","last_active":"2024-04-26T23:07:49.784433+0000","last_peered":"2024-04-26T23:07:49.784433+0000","last_clean":"2024-04-26T23:07:49.784433+0000","last_became_active":"2024-04-26T23:07:47.820581+0000","last_became_peered":"2024-04-26T23:07:47.820581+0000","last_unstale":"2024-04-26T23:07:49.784433+0000","last_undegraded":"2024-04-26T23:07:49.784433+0000","last_fullsized":"2024-04-26T23:07:49.784433+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:29.710976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.784340+0000","last_change":"2024-04-26T23:07:48.801851+0000","last_active":"2024-04-26T23:07:49.784340+0000","last_peered":"2024-04-26T23:07:49.784340+0000","last_clean":"2024-04-26T23:07:49.784340+0000","last_became_active":"2024-04-26T23:07:48.800995+0000","last_became_peered":"2024-04-26T23:07:48.800995+0000","last_unstale":"2024-04-26T23:07:49.784340+0000","last_undegraded":"2024-04-26T23:07:49.784340+0000","last_fullsized":"2024-04-26T23:07:49.784340+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:07.806976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"89'3","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.823676+0000","last_change":"2024-04-26T23:07:47.820922+0000","last_active":"2024-04-26T23:07:49.823676+0000","last_peered":"2024-04-26T23:07:49.823676+0000","last_clean":"2024-04-26T23:07:49.823676+0000","last_became_active":"2024-04-26T23:07:47.820591+0000","last_became_peered":"2024-04-26T23:07:47.820591+0000","last_unstale":"2024-04-26T23:07:49.823676+0000","last_undegraded":"2024-04-26T23:07:49.823676+0000","last_fullsized":"2024-04-26T23:07:49.823676+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:07:06.356926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.331171+0000","last_change":"2024-04-26T23:07:48.808067+0000","last_active":"2024-04-26T23:07:50.331171+0000","last_peered":"2024-04-26T23:07:50.331171+0000","last_clean":"2024-04-26T23:07:50.331171+0000","last_became_active":"2024-04-26T23:07:48.807707+0000","last_became_peered":"2024-04-26T23:07:48.807707+0000","last_unstale":"2024-04-26T23:07:50.331171+0000","last_undegraded":"2024-04-26T23:07:50.331171+0000","last_fullsized":"2024-04-26T23:07:50.331171+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:33:02.623151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"89'6","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.327592+0000","last_change":"2024-04-26T23:07:47.815166+0000","last_active":"2024-04-26T23:07:50.327592+0000","last_peered":"2024-04-26T23:07:50.327592+0000","last_clean":"2024-04-26T23:07:50.327592+0000","last_became_active":"2024-04-26T23:07:47.814809+0000","last_became_peered":"2024-04-26T23:07:47.814809+0000","last_unstale":"2024-04-26T23:07:50.327592+0000","last_undegraded":"2024-04-26T23:07:50.327592+0000","last_fullsized":"2024-04-26T23:07:50.327592+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:01:31.495005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.329772+0000","last_change":"2024-04-26T23:07:48.806752+0000","last_active":"2024-04-26T23:07:50.329772+0000","last_peered":"2024-04-26T23:07:50.329772+0000","last_clean":"2024-04-26T23:07:50.329772+0000","last_became_active":"2024-04-26T23:07:48.806286+0000","last_became_peered":"2024-04-26T23:07:48.806286+0000","last_unstale":"2024-04-26T23:07:50.329772+0000","last_undegraded":"2024-04-26T23:07:50.329772+0000","last_fullsized":"2024-04-26T23:07:50.329772+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:45.064250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"89'2","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.789396+0000","last_change":"2024-04-26T23:07:47.822437+0000","last_active":"2024-04-26T23:07:49.789396+0000","last_peered":"2024-04-26T23:07:49.789396+0000","last_clean":"2024-04-26T23:07:49.789396+0000","last_became_active":"2024-04-26T23:07:47.822143+0000","last_became_peered":"2024-04-26T23:07:47.822143+0000","last_unstale":"2024-04-26T23:07:49.789396+0000","last_undegraded":"2024-04-26T23:07:49.789396+0000","last_fullsized":"2024-04-26T23:07:49.789396+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:32:33.262857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.822451+0000","last_change":"2024-04-26T23:07:48.808704+0000","last_active":"2024-04-26T23:07:49.822451+0000","last_peered":"2024-04-26T23:07:49.822451+0000","last_clean":"2024-04-26T23:07:49.822451+0000","last_became_active":"2024-04-26T23:07:48.808379+0000","last_became_peered":"2024-04-26T23:07:48.808379+0000","last_unstale":"2024-04-26T23:07:49.822451+0000","last_undegraded":"2024-04-26T23:07:49.822451+0000","last_fullsized":"2024-04-26T23:07:49.822451+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:53.765394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.327287+0000","last_change":"2024-04-26T23:07:47.818118+0000","last_active":"2024-04-26T23:07:50.327287+0000","last_peered":"2024-04-26T23:07:50.327287+0000","last_clean":"2024-04-26T23:07:50.327287+0000","last_became_active":"2024-04-26T23:07:47.816576+0000","last_became_peered":"2024-04-26T23:07:47.816576+0000","last_unstale":"2024-04-26T23:07:50.327287+0000","last_undegraded":"2024-04-26T23:07:50.327287+0000","last_fullsized":"2024-04-26T23:07:50.327287+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:20:29.710976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.330156+0000","last_change":"2024-04-26T23:07:48.827278+0000","last_active":"2024-04-26T23:07:50.330156+0000","last_peered":"2024-04-26T23:07:50.330156+0000","last_clean":"2024-04-26T23:07:50.330156+0000","last_became_active":"2024-04-26T23:07:48.826998+0000","last_became_peered":"2024-04-26T23:07:48.826998+0000","last_unstale":"2024-04-26T23:07:50.330156+0000","last_undegraded":"2024-04-26T23:07:50.330156+0000","last_fullsized":"2024-04-26T23:07:50.330156+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:02.531726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.329129+0000","last_change":"2024-04-26T23:07:47.821873+0000","last_active":"2024-04-26T23:07:50.329129+0000","last_peered":"2024-04-26T23:07:50.329129+0000","last_clean":"2024-04-26T23:07:50.329129+0000","last_became_active":"2024-04-26T23:07:47.821521+0000","last_became_peered":"2024-04-26T23:07:47.821521+0000","last_unstale":"2024-04-26T23:07:50.329129+0000","last_undegraded":"2024-04-26T23:07:50.329129+0000","last_fullsized":"2024-04-26T23:07:50.329129+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:53:09.423166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.331348+0000","last_change":"2024-04-26T23:07:48.813340+0000","last_active":"2024-04-26T23:07:50.331348+0000","last_peered":"2024-04-26T23:07:50.331348+0000","last_clean":"2024-04-26T23:07:50.331348+0000","last_became_active":"2024-04-26T23:07:48.813073+0000","last_became_peered":"2024-04-26T23:07:48.813073+0000","last_unstale":"2024-04-26T23:07:50.331348+0000","last_undegraded":"2024-04-26T23:07:50.331348+0000","last_fullsized":"2024-04-26T23:07:50.331348+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:42.670329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"89'8","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.330080+0000","last_change":"2024-04-26T23:07:47.822478+0000","last_active":"2024-04-26T23:07:50.330080+0000","last_peered":"2024-04-26T23:07:50.330080+0000","last_clean":"2024-04-26T23:07:50.330080+0000","last_became_active":"2024-04-26T23:07:47.821881+0000","last_became_peered":"2024-04-26T23:07:47.821881+0000","last_unstale":"2024-04-26T23:07:50.330080+0000","last_undegraded":"2024-04-26T23:07:50.330080+0000","last_fullsized":"2024-04-26T23:07:50.330080+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:07:06.356926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.327402+0000","last_change":"2024-04-26T23:07:48.811398+0000","last_active":"2024-04-26T23:07:50.327402+0000","last_peered":"2024-04-26T23:07:50.327402+0000","last_clean":"2024-04-26T23:07:50.327402+0000","last_became_active":"2024-04-26T23:07:48.810664+0000","last_became_peered":"2024-04-26T23:07:48.810664+0000","last_unstale":"2024-04-26T23:07:50.327402+0000","last_undegraded":"2024-04-26T23:07:50.327402+0000","last_fullsized":"2024-04-26T23:07:50.327402+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:43.895589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"89'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.790817+0000","last_change":"2024-04-26T23:07:47.823873+0000","last_active":"2024-04-26T23:07:49.790817+0000","last_peered":"2024-04-26T23:07:49.790817+0000","last_clean":"2024-04-26T23:07:49.790817+0000","last_became_active":"2024-04-26T23:07:47.823123+0000","last_became_peered":"2024-04-26T23:07:47.823123+0000","last_unstale":"2024-04-26T23:07:49.790817+0000","last_undegraded":"2024-04-26T23:07:49.790817+0000","last_fullsized":"2024-04-26T23:07:49.790817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:32:39.604734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.789553+0000","last_change":"2024-04-26T23:07:48.806573+0000","last_active":"2024-04-26T23:07:49.789553+0000","last_peered":"2024-04-26T23:07:49.789553+0000","last_clean":"2024-04-26T23:07:49.789553+0000","last_became_active":"2024-04-26T23:07:48.805144+0000","last_became_peered":"2024-04-26T23:07:48.805144+0000","last_unstale":"2024-04-26T23:07:49.789553+0000","last_undegraded":"2024-04-26T23:07:49.789553+0000","last_fullsized":"2024-04-26T23:07:49.789553+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:53.765394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"89'2","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.790146+0000","last_change":"2024-04-26T23:07:47.825792+0000","last_active":"2024-04-26T23:07:49.790146+0000","last_peered":"2024-04-26T23:07:49.790146+0000","last_clean":"2024-04-26T23:07:49.790146+0000","last_became_active":"2024-04-26T23:07:47.824554+0000","last_became_peered":"2024-04-26T23:07:47.824554+0000","last_unstale":"2024-04-26T23:07:49.790146+0000","last_undegraded":"2024-04-26T23:07:49.790146+0000","last_fullsized":"2024-04-26T23:07:49.790146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:10:52.592001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.823431+0000","last_change":"2024-04-26T23:07:48.807982+0000","last_active":"2024-04-26T23:07:49.823431+0000","last_peered":"2024-04-26T23:07:49.823431+0000","last_clean":"2024-04-26T23:07:49.823431+0000","last_became_active":"2024-04-26T23:07:48.807660+0000","last_became_peered":"2024-04-26T23:07:48.807660+0000","last_unstale":"2024-04-26T23:07:49.823431+0000","last_undegraded":"2024-04-26T23:07:49.823431+0000","last_fullsized":"2024-04-26T23:07:49.823431+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:47.924483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"89'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.823852+0000","last_change":"2024-04-26T23:07:47.828834+0000","last_active":"2024-04-26T23:07:49.823852+0000","last_peered":"2024-04-26T23:07:49.823852+0000","last_clean":"2024-04-26T23:07:49.823852+0000","last_became_active":"2024-04-26T23:07:47.828446+0000","last_became_peered":"2024-04-26T23:07:47.828446+0000","last_unstale":"2024-04-26T23:07:49.823852+0000","last_undegraded":"2024-04-26T23:07:49.823852+0000","last_fullsized":"2024-04-26T23:07:49.823852+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:53:09.423166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.326602+0000","last_change":"2024-04-26T23:07:48.828333+0000","last_active":"2024-04-26T23:07:50.326602+0000","last_peered":"2024-04-26T23:07:50.326602+0000","last_clean":"2024-04-26T23:07:50.326602+0000","last_became_active":"2024-04-26T23:07:48.828035+0000","last_became_peered":"2024-04-26T23:07:48.828035+0000","last_unstale":"2024-04-26T23:07:50.326602+0000","last_undegraded":"2024-04-26T23:07:50.326602+0000","last_fullsized":"2024-04-26T23:07:50.326602+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:35:48.339371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"89'2","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.828511+0000","last_change":"2024-04-26T23:07:47.825672+0000","last_active":"2024-04-26T23:07:49.828511+0000","last_peered":"2024-04-26T23:07:49.828511+0000","last_clean":"2024-04-26T23:07:49.828511+0000","last_became_active":"2024-04-26T23:07:47.825386+0000","last_became_peered":"2024-04-26T23:07:47.825386+0000","last_unstale":"2024-04-26T23:07:49.828511+0000","last_undegraded":"2024-04-26T23:07:49.828511+0000","last_fullsized":"2024-04-26T23:07:49.828511+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:38:13.410883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.828488+0000","last_change":"2024-04-26T23:07:48.805998+0000","last_active":"2024-04-26T23:07:49.828488+0000","last_peered":"2024-04-26T23:07:49.828488+0000","last_clean":"2024-04-26T23:07:49.828488+0000","last_became_active":"2024-04-26T23:07:48.805524+0000","last_became_peered":"2024-04-26T23:07:48.805524+0000","last_unstale":"2024-04-26T23:07:49.828488+0000","last_undegraded":"2024-04-26T23:07:49.828488+0000","last_fullsized":"2024-04-26T23:07:49.828488+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:26:40.227555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.329881+0000","last_change":"2024-04-26T23:07:47.813255+0000","last_active":"2024-04-26T23:07:50.329881+0000","last_peered":"2024-04-26T23:07:50.329881+0000","last_clean":"2024-04-26T23:07:50.329881+0000","last_became_active":"2024-04-26T23:07:47.812932+0000","last_became_peered":"2024-04-26T23:07:47.812932+0000","last_unstale":"2024-04-26T23:07:50.329881+0000","last_undegraded":"2024-04-26T23:07:50.329881+0000","last_fullsized":"2024-04-26T23:07:50.329881+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:32:46.442669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.828468+0000","last_change":"2024-04-26T23:07:48.826176+0000","last_active":"2024-04-26T23:07:49.828468+0000","last_peered":"2024-04-26T23:07:49.828468+0000","last_clean":"2024-04-26T23:07:49.828468+0000","last_became_active":"2024-04-26T23:07:48.825865+0000","last_became_peered":"2024-04-26T23:07:48.825865+0000","last_unstale":"2024-04-26T23:07:49.828468+0000","last_undegraded":"2024-04-26T23:07:49.828468+0000","last_fullsized":"2024-04-26T23:07:49.828468+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:19:41.586767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.828594+0000","last_change":"2024-04-26T23:07:47.827803+0000","last_active":"2024-04-26T23:07:49.828594+0000","last_peered":"2024-04-26T23:07:49.828594+0000","last_clean":"2024-04-26T23:07:49.828594+0000","last_became_active":"2024-04-26T23:07:47.827474+0000","last_became_peered":"2024-04-26T23:07:47.827474+0000","last_unstale":"2024-04-26T23:07:49.828594+0000","last_undegraded":"2024-04-26T23:07:49.828594+0000","last_fullsized":"2024-04-26T23:07:49.828594+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:20.447920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.326983+0000","last_change":"2024-04-26T23:07:48.809634+0000","last_active":"2024-04-26T23:07:50.326983+0000","last_peered":"2024-04-26T23:07:50.326983+0000","last_clean":"2024-04-26T23:07:50.326983+0000","last_became_active":"2024-04-26T23:07:48.809355+0000","last_became_peered":"2024-04-26T23:07:48.809355+0000","last_unstale":"2024-04-26T23:07:50.326983+0000","last_undegraded":"2024-04-26T23:07:50.326983+0000","last_fullsized":"2024-04-26T23:07:50.326983+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:54:40.795033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.327774+0000","last_change":"2024-04-26T23:07:47.822273+0000","last_active":"2024-04-26T23:07:50.327774+0000","last_peered":"2024-04-26T23:07:50.327774+0000","last_clean":"2024-04-26T23:07:50.327774+0000","last_became_active":"2024-04-26T23:07:47.821964+0000","last_became_peered":"2024-04-26T23:07:47.821964+0000","last_unstale":"2024-04-26T23:07:50.327774+0000","last_undegraded":"2024-04-26T23:07:50.327774+0000","last_fullsized":"2024-04-26T23:07:50.327774+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:12.180660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.790637+0000","last_change":"2024-04-26T23:07:48.799235+0000","last_active":"2024-04-26T23:07:49.790637+0000","last_peered":"2024-04-26T23:07:49.790637+0000","last_clean":"2024-04-26T23:07:49.790637+0000","last_became_active":"2024-04-26T23:07:48.798914+0000","last_became_peered":"2024-04-26T23:07:48.798914+0000","last_unstale":"2024-04-26T23:07:49.790637+0000","last_undegraded":"2024-04-26T23:07:49.790637+0000","last_fullsized":"2024-04-26T23:07:49.790637+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:33:02.623151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"89'2","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.790651+0000","last_change":"2024-04-26T23:07:47.828282+0000","last_active":"2024-04-26T23:07:49.790651+0000","last_peered":"2024-04-26T23:07:49.790651+0000","last_clean":"2024-04-26T23:07:49.790651+0000","last_became_active":"2024-04-26T23:07:47.827934+0000","last_became_peered":"2024-04-26T23:07:47.827934+0000","last_unstale":"2024-04-26T23:07:49.790651+0000","last_undegraded":"2024-04-26T23:07:49.790651+0000","last_fullsized":"2024-04-26T23:07:49.790651+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:15:58.115631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.329161+0000","last_change":"2024-04-26T23:07:48.813125+0000","last_active":"2024-04-26T23:07:50.329161+0000","last_peered":"2024-04-26T23:07:50.329161+0000","last_clean":"2024-04-26T23:07:50.329161+0000","last_became_active":"2024-04-26T23:07:48.812912+0000","last_became_peered":"2024-04-26T23:07:48.812912+0000","last_unstale":"2024-04-26T23:07:50.329161+0000","last_undegraded":"2024-04-26T23:07:50.329161+0000","last_fullsized":"2024-04-26T23:07:50.329161+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:49.731627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"90'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.823677+0000","last_change":"2024-04-26T23:07:47.825275+0000","last_active":"2024-04-26T23:07:49.823677+0000","last_peered":"2024-04-26T23:07:49.823677+0000","last_clean":"2024-04-26T23:07:49.823677+0000","last_became_active":"2024-04-26T23:07:47.824983+0000","last_became_peered":"2024-04-26T23:07:47.824983+0000","last_unstale":"2024-04-26T23:07:49.823677+0000","last_undegraded":"2024-04-26T23:07:49.823677+0000","last_fullsized":"2024-04-26T23:07:49.823677+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:24:29.558386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.329414+0000","last_change":"2024-04-26T23:07:48.805969+0000","last_active":"2024-04-26T23:07:50.329414+0000","last_peered":"2024-04-26T23:07:50.329414+0000","last_clean":"2024-04-26T23:07:50.329414+0000","last_became_active":"2024-04-26T23:07:48.803208+0000","last_became_peered":"2024-04-26T23:07:48.803208+0000","last_unstale":"2024-04-26T23:07:50.329414+0000","last_undegraded":"2024-04-26T23:07:50.329414+0000","last_fullsized":"2024-04-26T23:07:50.329414+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:29:39.723678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.823866+0000","last_change":"2024-04-26T23:07:47.817958+0000","last_active":"2024-04-26T23:07:49.823866+0000","last_peered":"2024-04-26T23:07:49.823866+0000","last_clean":"2024-04-26T23:07:49.823866+0000","last_became_active":"2024-04-26T23:07:47.816225+0000","last_became_peered":"2024-04-26T23:07:47.816225+0000","last_unstale":"2024-04-26T23:07:49.823866+0000","last_undegraded":"2024-04-26T23:07:49.823866+0000","last_fullsized":"2024-04-26T23:07:49.823866+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:12:56.841587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.829412+0000","last_change":"2024-04-26T23:07:48.817598+0000","last_active":"2024-04-26T23:07:49.829412+0000","last_peered":"2024-04-26T23:07:49.829412+0000","last_clean":"2024-04-26T23:07:49.829412+0000","last_became_active":"2024-04-26T23:07:48.816979+0000","last_became_peered":"2024-04-26T23:07:48.816979+0000","last_unstale":"2024-04-26T23:07:49.829412+0000","last_undegraded":"2024-04-26T23:07:49.829412+0000","last_fullsized":"2024-04-26T23:07:49.829412+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:15:59.131561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.790322+0000","last_change":"2024-04-26T23:07:47.806304+0000","last_active":"2024-04-26T23:07:49.790322+0000","last_peered":"2024-04-26T23:07:49.790322+0000","last_clean":"2024-04-26T23:07:49.790322+0000","last_became_active":"2024-04-26T23:07:47.805537+0000","last_became_peered":"2024-04-26T23:07:47.805537+0000","last_unstale":"2024-04-26T23:07:49.790322+0000","last_undegraded":"2024-04-26T23:07:49.790322+0000","last_fullsized":"2024-04-26T23:07:49.790322+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:52:06.881525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.327890+0000","last_change":"2024-04-26T23:07:48.812464+0000","last_active":"2024-04-26T23:07:50.327890+0000","last_peered":"2024-04-26T23:07:50.327890+0000","last_clean":"2024-04-26T23:07:50.327890+0000","last_became_active":"2024-04-26T23:07:48.812131+0000","last_became_peered":"2024-04-26T23:07:48.812131+0000","last_unstale":"2024-04-26T23:07:50.327890+0000","last_undegraded":"2024-04-26T23:07:50.327890+0000","last_fullsized":"2024-04-26T23:07:50.327890+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:56:15.055094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.327317+0000","last_change":"2024-04-26T23:07:47.816254+0000","last_active":"2024-04-26T23:07:50.327317+0000","last_peered":"2024-04-26T23:07:50.327317+0000","last_clean":"2024-04-26T23:07:50.327317+0000","last_became_active":"2024-04-26T23:07:47.815567+0000","last_became_peered":"2024-04-26T23:07:47.815567+0000","last_unstale":"2024-04-26T23:07:50.327317+0000","last_undegraded":"2024-04-26T23:07:50.327317+0000","last_fullsized":"2024-04-26T23:07:50.327317+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:10:12.192944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.784278+0000","last_change":"2024-04-26T23:07:48.821568+0000","last_active":"2024-04-26T23:07:49.784278+0000","last_peered":"2024-04-26T23:07:49.784278+0000","last_clean":"2024-04-26T23:07:49.784278+0000","last_became_active":"2024-04-26T23:07:48.821260+0000","last_became_peered":"2024-04-26T23:07:48.821260+0000","last_unstale":"2024-04-26T23:07:49.784278+0000","last_undegraded":"2024-04-26T23:07:49.784278+0000","last_fullsized":"2024-04-26T23:07:49.784278+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:58:36.954405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.828532+0000","last_change":"2024-04-26T23:07:47.825330+0000","last_active":"2024-04-26T23:07:49.828532+0000","last_peered":"2024-04-26T23:07:49.828532+0000","last_clean":"2024-04-26T23:07:49.828532+0000","last_became_active":"2024-04-26T23:07:47.825036+0000","last_became_peered":"2024-04-26T23:07:47.825036+0000","last_unstale":"2024-04-26T23:07:49.828532+0000","last_undegraded":"2024-04-26T23:07:49.828532+0000","last_fullsized":"2024-04-26T23:07:49.828532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:32:39.604734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.788917+0000","last_change":"2024-04-26T23:07:48.826864+0000","last_active":"2024-04-26T23:07:49.788917+0000","last_peered":"2024-04-26T23:07:49.788917+0000","last_clean":"2024-04-26T23:07:49.788917+0000","last_became_active":"2024-04-26T23:07:48.825972+0000","last_became_peered":"2024-04-26T23:07:48.825972+0000","last_unstale":"2024-04-26T23:07:49.788917+0000","last_undegraded":"2024-04-26T23:07:49.788917+0000","last_fullsized":"2024-04-26T23:07:49.788917+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:45.064250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.789090+0000","last_change":"2024-04-26T23:07:47.822015+0000","last_active":"2024-04-26T23:07:49.789090+0000","last_peered":"2024-04-26T23:07:49.789090+0000","last_clean":"2024-04-26T23:07:49.789090+0000","last_became_active":"2024-04-26T23:07:47.820758+0000","last_became_peered":"2024-04-26T23:07:47.820758+0000","last_unstale":"2024-04-26T23:07:49.789090+0000","last_undegraded":"2024-04-26T23:07:49.789090+0000","last_fullsized":"2024-04-26T23:07:49.789090+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:46.079465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.327328+0000","last_change":"2024-04-26T23:07:48.810840+0000","last_active":"2024-04-26T23:07:50.327328+0000","last_peered":"2024-04-26T23:07:50.327328+0000","last_clean":"2024-04-26T23:07:50.327328+0000","last_became_active":"2024-04-26T23:07:48.801361+0000","last_became_peered":"2024-04-26T23:07:48.801361+0000","last_unstale":"2024-04-26T23:07:50.327328+0000","last_undegraded":"2024-04-26T23:07:50.327328+0000","last_fullsized":"2024-04-26T23:07:50.327328+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:56.616074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.787575+0000","last_change":"2024-04-26T23:07:47.825258+0000","last_active":"2024-04-26T23:07:49.787575+0000","last_peered":"2024-04-26T23:07:49.787575+0000","last_clean":"2024-04-26T23:07:49.787575+0000","last_became_active":"2024-04-26T23:07:47.824989+0000","last_became_peered":"2024-04-26T23:07:47.824989+0000","last_unstale":"2024-04-26T23:07:49.787575+0000","last_undegraded":"2024-04-26T23:07:49.787575+0000","last_fullsized":"2024-04-26T23:07:49.787575+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:22:18.852847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.790407+0000","last_change":"2024-04-26T23:07:48.804064+0000","last_active":"2024-04-26T23:07:49.790407+0000","last_peered":"2024-04-26T23:07:49.790407+0000","last_clean":"2024-04-26T23:07:49.790407+0000","last_became_active":"2024-04-26T23:07:48.803258+0000","last_became_peered":"2024-04-26T23:07:48.803258+0000","last_unstale":"2024-04-26T23:07:49.790407+0000","last_undegraded":"2024-04-26T23:07:49.790407+0000","last_fullsized":"2024-04-26T23:07:49.790407+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:47.771893+0000","last_clean_scrub_stamp":"2024-04-26T23:07:47.771893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:38:14.958205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:50.331798+0000","last_change":"2024-04-26T23:07:47.822209+0000","last_active":"2024-04-26T23:07:50.331798+0000","last_peered":"2024-04-26T23:07:50.331798+0000","last_clean":"2024-04-26T23:07:50.331798+0000","last_became_active":"2024-04-26T23:07:47.821920+0000","last_became_peered":"2024-04-26T23:07:47.821920+0000","last_unstale":"2024-04-26T23:07:50.331798+0000","last_undegraded":"2024-04-26T23:07:50.331798+0000","last_fullsized":"2024-04-26T23:07:50.331798+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:07:46.755962+0000","last_clean_scrub_stamp":"2024-04-26T23:07:46.755962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:33:27.821052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":304,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:07:49.787788+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:07:49.787788+0000","last_peered":"2024-04-26T23:07:49.787788+0000","last_clean":"2024-04-26T23:07:49.787788+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:07:49.787788+0000","last_undegraded":"2024-04-26T23:07:49.787788+0000","last_fullsized":"2024-04-26T23:07:49.787788+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574934,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32860,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338980,"statfs":{"total":96636764160,"available":96603115520,"internally_reserved":0,"allocated":1712128,"data_stored":1106803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574934,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1130496,"data_stored":518595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574934,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1110016,"data_stored":516379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574934,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1118208,"data_stored":516539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574934,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1110016,"data_stored":516379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574936,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1110016,"data_stored":516379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1114112,"data_stored":516449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574935,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32868,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338972,"statfs":{"total":96636764160,"available":96603107328,"internally_reserved":0,"allocated":1728512,"data_stored":1109065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:07:53.830 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T23:07:53.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-26T23:07:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.092+0000 7fe13f200640 1 Processor -- start 2024-04-26T23:07:54.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.092+0000 7fe13f200640 1 -- start start 2024-04-26T23:07:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.093+0000 7fe13f200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe1380d3bb0 0x7fe1380b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.093+0000 7fe13f200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1380b7df0 con 0x7fe13815f4f0 2024-04-26T23:07:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.093+0000 7fe13f200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1380b80d0 con 0x7fe13815f110 2024-04-26T23:07:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.093+0000 7fe13f200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe1380b83b0 con 0x7fe1380d3bb0 2024-04-26T23:07:54.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.094+0000 7fe13d400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe13815f4f0 0x7fe1380d34a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33500/0 (socket says 172.21.15.40:33500) 2024-04-26T23:07:54.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.094+0000 7fe13d400640 1 -- 172.21.15.40:0/3268892288 learned_addr learned my addr 172.21.15.40:0/3268892288 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:54.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.094+0000 7fe13e800640 1 --2- 172.21.15.40:0/3268892288 >> v2:172.21.15.40:3300/0 conn(0x7fe1380d3bb0 0x7fe1380b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:54.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.094+0000 7fe13ca00640 1 -- 172.21.15.40:0/3268892288 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288208592 0 0) 0x7fe1380b7df0 con 0x7fe13815f4f0 2024-04-26T23:07:54.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.094+0000 7fe13ca00640 1 -- 172.21.15.40:0/3268892288 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe124003520 con 0x7fe13815f4f0 2024-04-26T23:07:54.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.095+0000 7fe13e800640 1 -- 172.21.15.40:0/3268892288 >> v1:172.21.15.103:6789/0 conn(0x7fe13815f110 legacy=0x7fe1380d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:54.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.095+0000 7fe13e800640 1 -- 172.21.15.40:0/3268892288 >> v1:172.21.15.40:6789/0 conn(0x7fe13815f4f0 legacy=0x7fe1380d34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:54.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.095+0000 7fe13e800640 1 -- 172.21.15.40:0/3268892288 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1380b8690 con 0x7fe1380d3bb0 2024-04-26T23:07:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.096+0000 7fe13e800640 1 --2- 172.21.15.40:0/3268892288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1380d3bb0 0x7fe1380b5b40 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fe1340097a0 tx=0x7fe13402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692861120f59f950 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.097+0000 7fe13ca00640 1 -- 172.21.15.40:0/3268892288 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134037070 con 0x7fe1380d3bb0 2024-04-26T23:07:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.097+0000 7fe13ca00640 1 -- 172.21.15.40:0/3268892288 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe134032440 con 0x7fe1380d3bb0 2024-04-26T23:07:54.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.097+0000 7fe13ca00640 1 -- 172.21.15.40:0/3268892288 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1340327f0 con 0x7fe1380d3bb0 2024-04-26T23:07:54.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.097+0000 7fe13f200640 1 -- 172.21.15.40:0/3268892288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1380d3bb0 msgr2=0x7fe1380b5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:54.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.097+0000 7fe13f200640 1 --2- 172.21.15.40:0/3268892288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1380d3bb0 0x7fe1380b5b40 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fe1340097a0 tx=0x7fe13402a230 comp rx=0 tx=0).stop 2024-04-26T23:07:54.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.097+0000 7fe13f200640 1 -- 172.21.15.40:0/3268892288 shutdown_connections 2024-04-26T23:07:54.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.097+0000 7fe13f200640 1 --2- 172.21.15.40:0/3268892288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1380d3bb0 0x7fe1380b5b40 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.097+0000 7fe13f200640 1 -- 172.21.15.40:0/3268892288 >> 172.21.15.40:0/3268892288 conn(0x7fe13809e210 msgr2=0x7fe1380d4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:54.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.097+0000 7fe13f200640 1 -- 172.21.15.40:0/3268892288 shutdown_connections 2024-04-26T23:07:54.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.097+0000 7fe13f200640 1 -- 172.21.15.40:0/3268892288 wait complete. 2024-04-26T23:07:54.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.099+0000 7fe13f200640 1 Processor -- start 2024-04-26T23:07:54.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.099+0000 7fe13f200640 1 -- start start 2024-04-26T23:07:54.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.100+0000 7fe13f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1380d3bb0 0x7fe1381f5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:54.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.100+0000 7fe13f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe13815f110 0x7fe1381f4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:54.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.100+0000 7fe13f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13815f4f0 0x7fe1380cac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:54.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.100+0000 7fe13f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe1380d0c30 con 0x7fe13815f4f0 2024-04-26T23:07:54.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.100+0000 7fe13f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe1380d2fa0 con 0x7fe13815f110 2024-04-26T23:07:54.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.100+0000 7fe13f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe1380d0db0 con 0x7fe1380d3bb0 2024-04-26T23:07:54.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.101+0000 7fe13e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13815f4f0 0x7fe1380cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:54.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.101+0000 7fe13e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13815f4f0 0x7fe1380cac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44646/0 (socket says 172.21.15.40:44646) 2024-04-26T23:07:54.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.101+0000 7fe13e800640 1 -- 172.21.15.40:0/2589569128 learned_addr learned my addr 172.21.15.40:0/2589569128 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:54.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.101+0000 7fe13d400640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe13815f110 0x7fe1381f4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:54.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.101+0000 7fe13de00640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1380d3bb0 0x7fe1381f5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:54.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.101+0000 7fe13e800640 1 -- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1380d3bb0 msgr2=0x7fe1381f5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:54.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.101+0000 7fe13e800640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1380d3bb0 0x7fe1381f5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.101+0000 7fe13e800640 1 -- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe13815f110 msgr2=0x7fe1381f4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:54.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.101+0000 7fe13e800640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe13815f110 0x7fe1381f4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.101+0000 7fe13e800640 1 -- 172.21.15.40:0/2589569128 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13809ef40 con 0x7fe13815f4f0 2024-04-26T23:07:54.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.102+0000 7fe13e800640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13815f4f0 0x7fe1380cac00 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fe1340097a0 tx=0x7fe1340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:54.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.102+0000 7fe12e000640 1 -- 172.21.15.40:0/2589569128 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134037050 con 0x7fe13815f4f0 2024-04-26T23:07:54.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.102+0000 7fe12e000640 1 -- 172.21.15.40:0/2589569128 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe134043070 con 0x7fe13815f4f0 2024-04-26T23:07:54.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.102+0000 7fe12e000640 1 -- 172.21.15.40:0/2589569128 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134032810 con 0x7fe13815f4f0 2024-04-26T23:07:54.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.102+0000 7fe13f200640 1 -- 172.21.15.40:0/2589569128 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe13809f220 con 0x7fe13815f4f0 2024-04-26T23:07:54.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.102+0000 7fe13f200640 1 -- 172.21.15.40:0/2589569128 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe13809f7e0 con 0x7fe13815f4f0 2024-04-26T23:07:54.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.103+0000 7fe13f200640 1 -- 172.21.15.40:0/2589569128 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe138069370 con 0x7fe13815f4f0 2024-04-26T23:07:54.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.105+0000 7fe12e000640 1 -- 172.21.15.40:0/2589569128 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe1340329b0 con 0x7fe13815f4f0 2024-04-26T23:07:54.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.106+0000 7fe12e000640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe11c06c8f0 0x7fe11c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:54.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.106+0000 7fe12e000640 1 -- 172.21.15.40:0/2589569128 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fe1340ac860 con 0x7fe13815f4f0 2024-04-26T23:07:54.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.107+0000 7fe13de00640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe11c06c8f0 0x7fe11c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:54.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.108+0000 7fe13de00640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe11c06c8f0 0x7fe11c06edb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe130001d40 tx=0x7fe130009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:54.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.110+0000 7fe12e000640 1 -- 172.21.15.40:0/2589569128 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe13407aa60 con 0x7fe13815f4f0 2024-04-26T23:07:54.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.379+0000 7fe13f200640 1 -- 172.21.15.40:0/2589569128 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fe1380cbd90 con 0x7fe13815f4f0 2024-04-26T23:07:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.384+0000 7fe12e000640 1 -- 172.21.15.40:0/2589569128 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) ==== 190+0+147 (secure 0 0 0) 0x7fe13407a360 con 0x7fe13815f4f0 2024-04-26T23:07:54.385 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-04-26T23:07:54.385 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg== 2024-04-26T23:07:54.385 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-04-26T23:07:54.385 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-04-26T23:07:54.385 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-26T23:07:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-04-26T23:07:54.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.387+0000 7fe13f200640 1 -- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe11c06c8f0 msgr2=0x7fe11c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:54.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.387+0000 7fe13f200640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe11c06c8f0 0x7fe11c06edb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe130001d40 tx=0x7fe130009040 comp rx=0 tx=0).stop 2024-04-26T23:07:54.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.387+0000 7fe13f200640 1 -- 172.21.15.40:0/2589569128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13815f4f0 msgr2=0x7fe1380cac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:54.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.387+0000 7fe13f200640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13815f4f0 0x7fe1380cac00 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fe1340097a0 tx=0x7fe1340046c0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.387+0000 7fe13f200640 1 -- 172.21.15.40:0/2589569128 shutdown_connections 2024-04-26T23:07:54.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.387+0000 7fe13f200640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe11c06c8f0 0x7fe11c06edb0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.387+0000 7fe13f200640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13815f4f0 0x7fe1380cac00 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.387+0000 7fe13f200640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe13815f110 0x7fe1381f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.387+0000 7fe13f200640 1 --2- 172.21.15.40:0/2589569128 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1380d3bb0 0x7fe1381f5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.387+0000 7fe13f200640 1 -- 172.21.15.40:0/2589569128 >> 172.21.15.40:0/2589569128 conn(0x7fe13809e210 msgr2=0x7fe13806dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:54.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.387+0000 7fe13f200640 1 -- 172.21.15.40:0/2589569128 shutdown_connections 2024-04-26T23:07:54.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.387+0000 7fe13f200640 1 -- 172.21.15.40:0/2589569128 wait complete. 2024-04-26T23:07:54.405 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:54.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.610+0000 7ff7de600640 1 Processor -- start 2024-04-26T23:07:54.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.610+0000 7ff7de600640 1 -- start start 2024-04-26T23:07:54.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.612+0000 7ff7de600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff7d80c41d0 0x7ff7d80cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:54.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.612+0000 7ff7de600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7d8073f30 0x7ff7d81671e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:54.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.612+0000 7ff7de600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff7d8167720 0x7ff7d806ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:54.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.612+0000 7ff7de600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff7d80cf7a0 con 0x7ff7d80c41d0 2024-04-26T23:07:54.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.612+0000 7ff7de600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff7d8163a50 con 0x7ff7d8167720 2024-04-26T23:07:54.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.612+0000 7ff7de600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff7d80ceb90 con 0x7ff7d8073f30 2024-04-26T23:07:54.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.612+0000 7ff7d7400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff7d80c41d0 0x7ff7d80cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:54.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.612+0000 7ff7d7400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff7d80c41d0 0x7ff7d80cfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44648/0 (socket says 172.21.15.40:44648) 2024-04-26T23:07:54.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.612+0000 7ff7d7400640 1 -- 172.21.15.40:0/998971832 learned_addr learned my addr 172.21.15.40:0/998971832 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:54.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.612+0000 7ff7d7e00640 1 --2- 172.21.15.40:0/998971832 >> v2:172.21.15.103:3300/0 conn(0x7ff7d8167720 0x7ff7d806ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:54.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.612+0000 7ff7d6a00640 1 --2- 172.21.15.40:0/998971832 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7d8073f30 0x7ff7d81671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:54.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.613+0000 7ff7d7400640 1 -- 172.21.15.40:0/998971832 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7d8073f30 msgr2=0x7ff7d81671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:54.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.613+0000 7ff7d7400640 1 --2- 172.21.15.40:0/998971832 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7d8073f30 0x7ff7d81671e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.613+0000 7ff7d7400640 1 -- 172.21.15.40:0/998971832 >> v2:172.21.15.103:3300/0 conn(0x7ff7d8167720 msgr2=0x7ff7d806ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:54.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.613+0000 7ff7d7400640 1 --2- 172.21.15.40:0/998971832 >> v2:172.21.15.103:3300/0 conn(0x7ff7d8167720 0x7ff7d806ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.613+0000 7ff7d7400640 1 -- 172.21.15.40:0/998971832 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d80cee70 con 0x7ff7d80c41d0 2024-04-26T23:07:54.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.613+0000 7ff7d7e00640 1 --2- 172.21.15.40:0/998971832 >> v2:172.21.15.103:3300/0 conn(0x7ff7d8167720 0x7ff7d806ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:54.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.614+0000 7ff7d7400640 1 --2- 172.21.15.40:0/998971832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7d80c41d0 0x7ff7d80cfc30 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7ff7d80713f0 tx=0x7ff7c802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de5fc2094808755 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:54.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.614+0000 7ff7d6000640 1 -- 172.21.15.40:0/998971832 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8037070 con 0x7ff7d80c41d0 2024-04-26T23:07:54.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.614+0000 7ff7d6000640 1 -- 172.21.15.40:0/998971832 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c8032440 con 0x7ff7d80c41d0 2024-04-26T23:07:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.614+0000 7ff7d6000640 1 -- 172.21.15.40:0/998971832 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8032810 con 0x7ff7d80c41d0 2024-04-26T23:07:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.615+0000 7ff7de600640 1 -- 172.21.15.40:0/998971832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7d80c41d0 msgr2=0x7ff7d80cfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.615+0000 7ff7de600640 1 --2- 172.21.15.40:0/998971832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7d80c41d0 0x7ff7d80cfc30 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7ff7d80713f0 tx=0x7ff7c802a1e0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.615+0000 7ff7de600640 1 -- 172.21.15.40:0/998971832 shutdown_connections 2024-04-26T23:07:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.615+0000 7ff7de600640 1 --2- 172.21.15.40:0/998971832 >> v2:172.21.15.103:3300/0 conn(0x7ff7d8167720 0x7ff7d806ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.615+0000 7ff7de600640 1 --2- 172.21.15.40:0/998971832 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7d8073f30 0x7ff7d81671e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.615+0000 7ff7de600640 1 --2- 172.21.15.40:0/998971832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7d80c41d0 0x7ff7d80cfc30 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.615+0000 7ff7de600640 1 -- 172.21.15.40:0/998971832 >> 172.21.15.40:0/998971832 conn(0x7ff7d80d0be0 msgr2=0x7ff7d80d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.615+0000 7ff7de600640 1 -- 172.21.15.40:0/998971832 shutdown_connections 2024-04-26T23:07:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.615+0000 7ff7de600640 1 -- 172.21.15.40:0/998971832 wait complete. 2024-04-26T23:07:54.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.616+0000 7ff7de600640 1 Processor -- start 2024-04-26T23:07:54.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.617+0000 7ff7de600640 1 -- start start 2024-04-26T23:07:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.617+0000 7ff7de600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7d8073f30 0x7ff7d8163c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.617+0000 7ff7de600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7d80c41d0 0x7ff7d81641a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.617+0000 7ff7de600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7d8167720 0x7ff7d81fcf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.617+0000 7ff7de600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d80d0170 con 0x7ff7d80c41d0 2024-04-26T23:07:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.617+0000 7ff7de600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d80cf430 con 0x7ff7d8073f30 2024-04-26T23:07:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.617+0000 7ff7de600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff7d80ce7b0 con 0x7ff7d8167720 2024-04-26T23:07:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.618+0000 7ff7d6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7d80c41d0 0x7ff7d81641a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.618+0000 7ff7d6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7d80c41d0 0x7ff7d81641a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44656/0 (socket says 172.21.15.40:44656) 2024-04-26T23:07:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.618+0000 7ff7d6a00640 1 -- 172.21.15.40:0/1011275734 learned_addr learned my addr 172.21.15.40:0/1011275734 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.618+0000 7ff7d7e00640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7d8167720 0x7ff7d81fcf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.618+0000 7ff7d7400640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7d8073f30 0x7ff7d8163c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.618+0000 7ff7d6a00640 1 -- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7d8167720 msgr2=0x7ff7d81fcf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.618+0000 7ff7d6a00640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7d8167720 0x7ff7d81fcf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.618+0000 7ff7d6a00640 1 -- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7d8073f30 msgr2=0x7ff7d8163c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.618+0000 7ff7d6a00640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7d8073f30 0x7ff7d8163c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.618+0000 7ff7d6a00640 1 -- 172.21.15.40:0/1011275734 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d814b100 con 0x7ff7d80c41d0 2024-04-26T23:07:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.618+0000 7ff7d7400640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7d8073f30 0x7ff7d8163c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:54.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.619+0000 7ff7d6a00640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7d80c41d0 0x7ff7d81641a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400c610 tx=0x7ff7c400cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:54.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.619+0000 7ff7cf400640 1 -- 172.21.15.40:0/1011275734 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4004230 con 0x7ff7d80c41d0 2024-04-26T23:07:54.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.619+0000 7ff7cf400640 1 -- 172.21.15.40:0/1011275734 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c40043d0 con 0x7ff7d80c41d0 2024-04-26T23:07:54.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.619+0000 7ff7cf400640 1 -- 172.21.15.40:0/1011275734 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4003830 con 0x7ff7d80c41d0 2024-04-26T23:07:54.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.619+0000 7ff7de600640 1 -- 172.21.15.40:0/1011275734 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d814b3e0 con 0x7ff7d80c41d0 2024-04-26T23:07:54.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.619+0000 7ff7de600640 1 -- 172.21.15.40:0/1011275734 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d814b9a0 con 0x7ff7d80c41d0 2024-04-26T23:07:54.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.620+0000 7ff7de600640 1 -- 172.21.15.40:0/1011275734 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7d80694f0 con 0x7ff7d80c41d0 2024-04-26T23:07:54.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.621+0000 7ff7cf400640 1 -- 172.21.15.40:0/1011275734 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff7c40039d0 con 0x7ff7d80c41d0 2024-04-26T23:07:54.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.622+0000 7ff7cf400640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7b806c8f0 0x7ff7b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:54.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.622+0000 7ff7cf400640 1 -- 172.21.15.40:0/1011275734 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7ff7c408bc50 con 0x7ff7d80c41d0 2024-04-26T23:07:54.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.623+0000 7ff7d7400640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7b806c8f0 0x7ff7b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:54.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.624+0000 7ff7d7400640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7b806c8f0 0x7ff7b806edb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8034000 tx=0x7ff7c802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:54.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.627+0000 7ff7cf400640 1 -- 172.21.15.40:0/1011275734 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff7c4015020 con 0x7ff7d80c41d0 2024-04-26T23:07:54.754 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:07:54.754+0000 7fd9e1000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T23:07:54.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.873+0000 7ff7de600640 1 -- 172.21.15.40:0/1011275734 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7d80729c0 con 0x7ff7d80c41d0 2024-04-26T23:07:54.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.874+0000 7ff7cf400640 1 -- 172.21.15.40:0/1011275734 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5183 (secure 0 0 0) 0x7ff7c4059ee0 con 0x7ff7d80c41d0 2024-04-26T23:07:54.875 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:54.875 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":91,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:07:51.330852+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8040},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[8040]},"id":6}]} 2024-04-26T23:07:54.875 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 91 2024-04-26T23:07:54.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.877+0000 7ff7de600640 1 -- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7b806c8f0 msgr2=0x7ff7b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:54.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.877+0000 7ff7de600640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7b806c8f0 0x7ff7b806edb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8034000 tx=0x7ff7c802e000 comp rx=0 tx=0).stop 2024-04-26T23:07:54.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.877+0000 7ff7de600640 1 -- 172.21.15.40:0/1011275734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7d80c41d0 msgr2=0x7ff7d81641a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:54.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.877+0000 7ff7de600640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7d80c41d0 0x7ff7d81641a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400c610 tx=0x7ff7c400cad0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.877+0000 7ff7de600640 1 -- 172.21.15.40:0/1011275734 shutdown_connections 2024-04-26T23:07:54.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.877+0000 7ff7de600640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7b806c8f0 0x7ff7b806edb0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.877+0000 7ff7de600640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7d8167720 0x7ff7d81fcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.877+0000 7ff7de600640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7d80c41d0 0x7ff7d81641a0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.877+0000 7ff7de600640 1 --2- 172.21.15.40:0/1011275734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7d8073f30 0x7ff7d8163c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:54.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.877+0000 7ff7de600640 1 -- 172.21.15.40:0/1011275734 >> 172.21.15.40:0/1011275734 conn(0x7ff7d80d0be0 msgr2=0x7ff7d80b1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:54.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.877+0000 7ff7de600640 1 -- 172.21.15.40:0/1011275734 shutdown_connections 2024-04-26T23:07:54.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:54.877+0000 7ff7de600640 1 -- 172.21.15.40:0/1011275734 wait complete. 2024-04-26T23:07:55.298 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 91, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:07:48.279070+0000', 'modified': '2024-04-26T23:07:51.330852+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8040}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8040': {'gid': 8040, 'name': 'c', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/88401598', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 88401598}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 88401598}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8040, 'qdb_cluster': [8040]} 2024-04-26T23:07:55.298 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-26T23:07:55.299 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T23:07:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.522+0000 7f45cb400640 1 Processor -- start 2024-04-26T23:07:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.522+0000 7f45cb400640 1 -- start start 2024-04-26T23:07:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.524+0000 7f45cb400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b0380 0x7f45c40ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.524+0000 7f45cb400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45c40cfd80 con 0x7f45c406c950 2024-04-26T23:07:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.524+0000 7f45cb400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45c40d0020 con 0x7f45c40b10b0 2024-04-26T23:07:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.524+0000 7f45cb400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f45c40af8e0 con 0x7f45c40b0380 2024-04-26T23:07:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.524+0000 7f45c9400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f45c406c950 0x7f45c415e970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41982/0 (socket says 172.21.15.40:41982) 2024-04-26T23:07:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.524+0000 7f45c9400640 1 -- 172.21.15.40:0/2861969137 learned_addr learned my addr 172.21.15.40:0/2861969137 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c8a00640 1 --2- 172.21.15.40:0/2861969137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b0380 0x7f45c40ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 235548333 0 0) 0x7f45c40cfd80 con 0x7f45c406c950 2024-04-26T23:07:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45ac003520 con 0x7f45c406c950 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3268038031 0 0) 0x7f45ac003520 con 0x7f45c406c950 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b0380 msgr2=0x7f45c40ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c3400640 1 --2- 172.21.15.40:0/2861969137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b0380 0x7f45c40ccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 >> v1:172.21.15.103:6789/0 conn(0x7f45c40b10b0 legacy=0x7f45c406c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c40d0300 con 0x7f45c406c950 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f45bc0019f0 con 0x7f45c406c950 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 >> v1:172.21.15.40:6789/0 conn(0x7f45c406c950 legacy=0x7f45c415e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c3400640 1 --2- 172.21.15.40:0/2861969137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45ac0039e0 0x7f45ac003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c3400640 1 --2- 172.21.15.40:0/2861969137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b0380 0x7f45ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c3400640 1 --2- 172.21.15.40:0/2861969137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45ac004d40 0x7f45ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.525+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f45c40af8e0 con 0x7f45ac004d40 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.526+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac00bb50 con 0x7f45ac0039e0 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.526+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f45ac00bd50 con 0x7f45c40b0380 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.526+0000 7f45c9400640 1 --2- 172.21.15.40:0/2861969137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45ac004d40 0x7f45ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.526+0000 7f45c8a00640 1 --2- 172.21.15.40:0/2861969137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45ac0039e0 0x7f45ac003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.526+0000 7f45c3e00640 1 --2- 172.21.15.40:0/2861969137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b0380 0x7f45ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.526+0000 7f45c9400640 1 -- 172.21.15.40:0/2861969137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b0380 msgr2=0x7f45ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.526+0000 7f45c9400640 1 --2- 172.21.15.40:0/2861969137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b0380 0x7f45ac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.526+0000 7f45c9400640 1 -- 172.21.15.40:0/2861969137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45ac0039e0 msgr2=0x7f45ac003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.526+0000 7f45c9400640 1 --2- 172.21.15.40:0/2861969137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45ac0039e0 0x7f45ac003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.526+0000 7f45c9400640 1 -- 172.21.15.40:0/2861969137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac003520 con 0x7f45ac004d40 2024-04-26T23:07:55.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.526+0000 7f45c3e00640 1 --2- 172.21.15.40:0/2861969137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b0380 0x7f45ac004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:55.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.527+0000 7f45c9400640 1 --2- 172.21.15.40:0/2861969137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45ac004d40 0x7f45ac0051c0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00a740 tx=0x7f45ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b53bc42df8b641c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:55.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.528+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc00a390 con 0x7f45ac004d40 2024-04-26T23:07:55.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.528+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45bc02f070 con 0x7f45ac004d40 2024-04-26T23:07:55.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.528+0000 7f45c3400640 1 -- 172.21.15.40:0/2861969137 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc02bbc0 con 0x7f45ac004d40 2024-04-26T23:07:55.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.528+0000 7f45cb400640 1 -- 172.21.15.40:0/2861969137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45ac004d40 msgr2=0x7f45ac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.528+0000 7f45cb400640 1 --2- 172.21.15.40:0/2861969137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45ac004d40 0x7f45ac0051c0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00a740 tx=0x7f45ac005830 comp rx=0 tx=0).stop 2024-04-26T23:07:55.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.528+0000 7f45c9400640 1 -- 172.21.15.40:0/2861969137 reap_dead start 2024-04-26T23:07:55.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.528+0000 7f45cb400640 1 -- 172.21.15.40:0/2861969137 shutdown_connections 2024-04-26T23:07:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.528+0000 7f45cb400640 1 -- 172.21.15.40:0/2861969137 >> 172.21.15.40:0/2861969137 conn(0x7f45c40d1cf0 msgr2=0x7f45c40d20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.528+0000 7f45cb400640 1 -- 172.21.15.40:0/2861969137 shutdown_connections 2024-04-26T23:07:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.528+0000 7f45cb400640 1 -- 172.21.15.40:0/2861969137 wait complete. 2024-04-26T23:07:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.531+0000 7f45cb400640 1 Processor -- start 2024-04-26T23:07:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.531+0000 7f45cb400640 1 -- start start 2024-04-26T23:07:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.532+0000 7f45cb400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45ac004d40 0x7f45c40b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.532+0000 7f45cb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45c40b4a50 0x7f45c40b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.532+0000 7f45cb400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b8370 0x7f45c40b87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.532+0000 7f45cb400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f45c40b0760 con 0x7f45c40b4a50 2024-04-26T23:07:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.532+0000 7f45cb400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f45c40d1980 con 0x7f45ac004d40 2024-04-26T23:07:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.532+0000 7f45cb400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f45c40d15c0 con 0x7f45c40b8370 2024-04-26T23:07:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.532+0000 7f45c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45c40b4a50 0x7f45c40b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.532+0000 7f45c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45c40b4a50 0x7f45c40b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44710/0 (socket says 172.21.15.40:44710) 2024-04-26T23:07:55.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.532+0000 7f45c3e00640 1 -- 172.21.15.40:0/1258186909 learned_addr learned my addr 172.21.15.40:0/1258186909 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:55.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.532+0000 7f45c8a00640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45ac004d40 0x7f45c40b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.532+0000 7f45c3e00640 1 -- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b8370 msgr2=0x7f45c40b87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.532+0000 7f45c9400640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b8370 0x7f45c40b87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.533+0000 7f45c3e00640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b8370 0x7f45c40b87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.533+0000 7f45c3e00640 1 -- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45ac004d40 msgr2=0x7f45c40b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.533+0000 7f45c3e00640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45ac004d40 0x7f45c40b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.533+0000 7f45c3e00640 1 -- 172.21.15.40:0/1258186909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c40b7e90 con 0x7f45c40b4a50 2024-04-26T23:07:55.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.533+0000 7f45c8a00640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45ac004d40 0x7f45c40b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:55.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.533+0000 7f45c3e00640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45c40b4a50 0x7f45c40b4ed0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f45b000d7a0 tx=0x7f45b000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:55.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.533+0000 7f45c0c00640 1 -- 172.21.15.40:0/1258186909 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b0005070 con 0x7f45c40b4a50 2024-04-26T23:07:55.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.533+0000 7f45c0c00640 1 -- 172.21.15.40:0/1258186909 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b0005210 con 0x7f45c40b4a50 2024-04-26T23:07:55.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.533+0000 7f45c0c00640 1 -- 172.21.15.40:0/1258186909 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b0007030 con 0x7f45c40b4a50 2024-04-26T23:07:55.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.533+0000 7f45cb400640 1 -- 172.21.15.40:0/1258186909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c40b5800 con 0x7f45c40b4a50 2024-04-26T23:07:55.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.533+0000 7f45cb400640 1 -- 172.21.15.40:0/1258186909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c40b5dc0 con 0x7f45c40b4a50 2024-04-26T23:07:55.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.536+0000 7f45c0c00640 1 -- 172.21.15.40:0/1258186909 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f45b001a030 con 0x7f45c40b4a50 2024-04-26T23:07:55.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.537+0000 7f45c0c00640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f45a406c920 0x7f45a406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.538+0000 7f45c0c00640 1 -- 172.21.15.40:0/1258186909 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f45b008b460 con 0x7f45c40b4a50 2024-04-26T23:07:55.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.538+0000 7f45c8a00640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f45a406c920 0x7f45a406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.538+0000 7f45ce16c740 1 Processor -- start 2024-04-26T23:07:55.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.538+0000 7f45ce16c740 1 -- start start 2024-04-26T23:07:55.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.539+0000 7f45c8a00640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f45a406c920 0x7f45a406ede0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f45b80099e0 tx=0x7f45b8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:55.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.539+0000 7f45ce16c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260715e0 0x55d726073740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.539+0000 7f45ce16c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 0x55d7260719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.539+0000 7f45ce16c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d72609c0a0 0x55d72609c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.539+0000 7f45ce16c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d725fd5600 con 0x55d7260715e0 2024-04-26T23:07:55.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.539+0000 7f45ce16c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d725f72100 con 0x55d72609c0a0 2024-04-26T23:07:55.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.539+0000 7f45ce16c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d725fd15c0 con 0x55d7260adcb0 2024-04-26T23:07:55.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.540+0000 7f45c8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260715e0 0x55d726073740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.540+0000 7f45c8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260715e0 0x55d726073740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44722/0 (socket says 172.21.15.40:44722) 2024-04-26T23:07:55.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.540+0000 7f45c8a00640 1 -- 172.21.15.40:0/863020252 learned_addr learned my addr 172.21.15.40:0/863020252 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:55.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.540+0000 7f45c8a00640 1 -- 172.21.15.40:0/863020252 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 msgr2=0x55d7260719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.540+0000 7f45c8a00640 1 --2- 172.21.15.40:0/863020252 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 0x55d7260719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.540+0000 7f45c8a00640 1 -- 172.21.15.40:0/863020252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d72609c0a0 msgr2=0x55d72609c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.540+0000 7f45c8a00640 1 --2- 172.21.15.40:0/863020252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d72609c0a0 0x55d72609c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.540+0000 7f45c8a00640 1 -- 172.21.15.40:0/863020252 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d725ce72d0 con 0x55d7260715e0 2024-04-26T23:07:55.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.540+0000 7f45c8a00640 1 --2- 172.21.15.40:0/863020252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260715e0 0x55d726073740 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f45b8011ce0 tx=0x7f45b8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64dc1713c76f55d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:55.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.541+0000 7f45b5200640 1 -- 172.21.15.40:0/863020252 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8011d10 con 0x55d7260715e0 2024-04-26T23:07:55.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.541+0000 7f45b5200640 1 -- 172.21.15.40:0/863020252 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b8026070 con 0x55d7260715e0 2024-04-26T23:07:55.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.541+0000 7f45b5200640 1 -- 172.21.15.40:0/863020252 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8021660 con 0x55d7260715e0 2024-04-26T23:07:55.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.541+0000 7f45ce16c740 1 -- 172.21.15.40:0/863020252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260715e0 msgr2=0x55d726073740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.541+0000 7f45ce16c740 1 --2- 172.21.15.40:0/863020252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260715e0 0x55d726073740 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f45b8011ce0 tx=0x7f45b8014000 comp rx=0 tx=0).stop 2024-04-26T23:07:55.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.542+0000 7f45ce16c740 1 -- 172.21.15.40:0/863020252 shutdown_connections 2024-04-26T23:07:55.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.542+0000 7f45ce16c740 1 --2- 172.21.15.40:0/863020252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d72609c0a0 0x55d72609c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.542+0000 7f45ce16c740 1 --2- 172.21.15.40:0/863020252 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 0x55d7260719c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.542+0000 7f45ce16c740 1 --2- 172.21.15.40:0/863020252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260715e0 0x55d726073740 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.542+0000 7f45ce16c740 1 -- 172.21.15.40:0/863020252 >> 172.21.15.40:0/863020252 conn(0x55d726005d10 msgr2=0x55d7260b70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:55.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.542+0000 7f45ce16c740 1 -- 172.21.15.40:0/863020252 shutdown_connections 2024-04-26T23:07:55.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.542+0000 7f45ce16c740 1 -- 172.21.15.40:0/863020252 wait complete. 2024-04-26T23:07:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.542+0000 7f45ce16c740 1 -- start start 2024-04-26T23:07:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.543+0000 7f45ce16c740 1 Processor -- start 2024-04-26T23:07:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45ce16c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 0x55d72609f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45ce16c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 0x55d7260b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45ce16c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 0x55d72608c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45ce16c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d725f5cf70 con 0x55d72609c0a0 2024-04-26T23:07:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45ce16c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d726099bf0 con 0x55d7260715e0 2024-04-26T23:07:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45ce16c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d7260c6e60 con 0x55d7260adcb0 2024-04-26T23:07:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45ce16c740 10 client.0 _ll_register_callbacks cb 0x55d725fcb480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 0x55d7260b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 0x55d7260b27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44738/0 (socket says 172.21.15.40:44738) 2024-04-26T23:07:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45c3e00640 1 -- 172.21.15.40:0/4059801509 learned_addr learned my addr 172.21.15.40:0/4059801509 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45c9400640 1 --2- 172.21.15.40:0/4059801509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 0x55d72608c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45c3e00640 1 -- 172.21.15.40:0/4059801509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 msgr2=0x55d72608c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45c8a00640 1 --2- 172.21.15.40:0/4059801509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 0x55d72609f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45c3e00640 1 --2- 172.21.15.40:0/4059801509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 0x55d72608c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45c3e00640 1 -- 172.21.15.40:0/4059801509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 msgr2=0x55d72609f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45c3e00640 1 --2- 172.21.15.40:0/4059801509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 0x55d72609f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.544+0000 7f45c3e00640 1 -- 172.21.15.40:0/4059801509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d725db16b0 con 0x55d72609c0a0 2024-04-26T23:07:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.545+0000 7f45c3e00640 1 --2- 172.21.15.40:0/4059801509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 0x55d7260b27a0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f45b0000fb0 tx=0x7f45b0059830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b5db2c7fb92b598 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.545+0000 7f45a9600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.545+0000 7f45a9600640 1 -- 172.21.15.40:0/4059801509 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b008b070 con 0x55d72609c0a0 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.545+0000 7f45a9600640 1 -- 172.21.15.40:0/4059801509 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f45b0089bc0 con 0x55d72609c0a0 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.545+0000 7f45ce16c740 1 -- 172.21.15.40:0/4059801509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45b8026070 con 0x55d72609c0a0 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.545+0000 7f45a9600640 1 -- 172.21.15.40:0/4059801509 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f45b008b210 con 0x55d72609c0a0 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.545+0000 7f45ce16c740 10 client.8172 fetch_fsmap learned FSMap version 91 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.545+0000 7f45ce16c740 1 -- 172.21.15.40:0/4059801509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) -- 0x55d72609c9a0 con 0x55d72609c0a0 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.545+0000 7f45a9600640 1 -- 172.21.15.40:0/4059801509 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b8026070 con 0x55d72609c0a0 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.546+0000 7f45a9600640 1 -- 172.21.15.40:0/4059801509 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 91) ==== 3494+0+0 (secure 0 0 0) 0x7f45b0058440 con 0x55d72609c0a0 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.546+0000 7f45ce16c740 10 client.8172 fetch_fsmap finished waiting for FSMap version 91 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.546+0000 7f45ce16c740 10 client.8172 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.546+0000 7f45ce16c740 20 client.8172 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.546+0000 7f45ce16c740 1 --2- 172.21.15.40:0/4059801509 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d7260ca3d0 0x55d72609cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.546+0000 7f45ce16c740 4 client.8172 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.546+0000 7f45ce16c740 1 -- 172.21.15.40:0/4059801509 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d725fe3210 con 0x55d7260ca3d0 2024-04-26T23:07:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.546+0000 7f45c8a00640 1 --2- 172.21.15.40:0/4059801509 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d7260ca3d0 0x55d72609cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.547+0000 7f45c8a00640 1 --2- 172.21.15.40:0/4059801509 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d7260ca3d0 0x55d72609cfb0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b0225d187f9d279 server_cookie=1dab9f42661b70cf in_seq=0 out_seq=0 2024-04-26T23:07:55.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.547+0000 7f45a9600640 10 client.8172 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:07:55.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45a9600640 1 -- 172.21.15.40:0/4059801509 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d725fe3210 con 0x55d7260ca3d0 2024-04-26T23:07:55.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45a9600640 10 client.8172 handle_command_reply: tid=0 2024-04-26T23:07:55.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45ce16c740 1 client.8172 shutdown 2024-04-26T23:07:55.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45ce16c740 1 -- 172.21.15.40:0/4059801509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 msgr2=0x55d7260b27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45ce16c740 1 --2- 172.21.15.40:0/4059801509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 0x55d7260b27a0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f45b0000fb0 tx=0x7f45b0059830 comp rx=0 tx=0).stop 2024-04-26T23:07:55.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45ce16c740 1 -- 172.21.15.40:0/4059801509 shutdown_connections 2024-04-26T23:07:55.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45ce16c740 1 --2- 172.21.15.40:0/4059801509 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d7260ca3d0 0x55d72609cfb0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45ce16c740 1 --2- 172.21.15.40:0/4059801509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 0x55d72608c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45ce16c740 1 --2- 172.21.15.40:0/4059801509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 0x55d7260b27a0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45a9600640 0 client.8172 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:07:55.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45ce16c740 1 --2- 172.21.15.40:0/4059801509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 0x55d72609f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45ce16c740 1 -- 172.21.15.40:0/4059801509 >> 172.21.15.40:0/4059801509 conn(0x55d726005d10 msgr2=0x55d7260ba590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:55.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45ce16c740 1 -- 172.21.15.40:0/4059801509 shutdown_connections 2024-04-26T23:07:55.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45ce16c740 1 -- 172.21.15.40:0/4059801509 wait complete. 2024-04-26T23:07:55.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.548+0000 7f45ce16c740 20 client.8172 trim_cache size 0 max 16384 2024-04-26T23:07:55.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.569+0000 7f45ce16c740 1 Processor -- start 2024-04-26T23:07:55.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.569+0000 7f45ce16c740 1 -- start start 2024-04-26T23:07:55.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.570+0000 7f45ce16c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 0x55d7260b9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.570+0000 7f45ce16c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d72609c0a0 0x55d72608b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.570+0000 7f45ce16c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260adcb0 0x55d7261341e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.570+0000 7f45ce16c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f45c4201620 con 0x55d7260adcb0 2024-04-26T23:07:55.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.570+0000 7f45ce16c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f45c4200ad0 con 0x55d7260715e0 2024-04-26T23:07:55.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.570+0000 7f45ce16c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f45c4201000 con 0x55d72609c0a0 2024-04-26T23:07:55.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.571+0000 7f45c9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260adcb0 0x55d7261341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.571+0000 7f45c9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260adcb0 0x55d7261341e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44748/0 (socket says 172.21.15.40:44748) 2024-04-26T23:07:55.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.571+0000 7f45c9400640 1 -- 172.21.15.40:0/2123885212 learned_addr learned my addr 172.21.15.40:0/2123885212 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.571+0000 7f45c8a00640 1 --2- 172.21.15.40:0/2123885212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 0x55d7260b9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.571+0000 7f45c3e00640 1 --2- 172.21.15.40:0/2123885212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d72609c0a0 0x55d72608b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.571+0000 7f45c9400640 1 -- 172.21.15.40:0/2123885212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d72609c0a0 msgr2=0x55d72608b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.571+0000 7f45c9400640 1 --2- 172.21.15.40:0/2123885212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d72609c0a0 0x55d72608b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.571+0000 7f45c9400640 1 -- 172.21.15.40:0/2123885212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 msgr2=0x55d7260b9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.571+0000 7f45c9400640 1 --2- 172.21.15.40:0/2123885212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 0x55d7260b9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.571+0000 7f45c9400640 1 -- 172.21.15.40:0/2123885212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7260684b0 con 0x55d7260adcb0 2024-04-26T23:07:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.572+0000 7f45c9400640 1 --2- 172.21.15.40:0/2123885212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260adcb0 0x55d7261341e0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00a860 tx=0x7f45bc02d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad1a049659e3e7f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.572+0000 7f45abe00640 1 -- 172.21.15.40:0/2123885212 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc044070 con 0x55d7260adcb0 2024-04-26T23:07:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.572+0000 7f45abe00640 1 -- 172.21.15.40:0/2123885212 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45bc02f040 con 0x55d7260adcb0 2024-04-26T23:07:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.572+0000 7f45abe00640 1 -- 172.21.15.40:0/2123885212 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc0019f0 con 0x55d7260adcb0 2024-04-26T23:07:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.573+0000 7f45ce16c740 1 -- 172.21.15.40:0/2123885212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260adcb0 msgr2=0x55d7261341e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.573+0000 7f45ce16c740 1 --2- 172.21.15.40:0/2123885212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260adcb0 0x55d7261341e0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00a860 tx=0x7f45bc02d070 comp rx=0 tx=0).stop 2024-04-26T23:07:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.573+0000 7f45ce16c740 1 -- 172.21.15.40:0/2123885212 shutdown_connections 2024-04-26T23:07:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.573+0000 7f45ce16c740 1 --2- 172.21.15.40:0/2123885212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7260adcb0 0x55d7261341e0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.573+0000 7f45ce16c740 1 --2- 172.21.15.40:0/2123885212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d72609c0a0 0x55d72608b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.573+0000 7f45ce16c740 1 --2- 172.21.15.40:0/2123885212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 0x55d7260b9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.573+0000 7f45ce16c740 1 -- 172.21.15.40:0/2123885212 >> 172.21.15.40:0/2123885212 conn(0x55d726005d10 msgr2=0x55d72609b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.573+0000 7f45ce16c740 1 -- 172.21.15.40:0/2123885212 shutdown_connections 2024-04-26T23:07:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.573+0000 7f45ce16c740 1 -- 172.21.15.40:0/2123885212 wait complete. 2024-04-26T23:07:55.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.574+0000 7f45ce16c740 1 -- start start 2024-04-26T23:07:55.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.574+0000 7f45ce16c740 1 Processor -- start 2024-04-26T23:07:55.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.575+0000 7f45ce16c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 0x55d726098dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.575+0000 7f45ce16c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 0x55d7260c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.575+0000 7f45ce16c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 0x55d7260b9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.575+0000 7f45ce16c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d7260b18c0 con 0x55d72609c0a0 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.575+0000 7f45ce16c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d72609f930 con 0x55d7260715e0 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.575+0000 7f45ce16c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d7260ca820 con 0x55d7260adcb0 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.575+0000 7f45ce16c740 10 client.0 _ll_register_callbacks cb 0x7f45b00589e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.575+0000 7f45c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 0x55d7260c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.575+0000 7f45c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 0x55d7260c57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44762/0 (socket says 172.21.15.40:44762) 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.575+0000 7f45c3e00640 1 -- 172.21.15.40:0/4106557577 learned_addr learned my addr 172.21.15.40:0/4106557577 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.576+0000 7f45c9400640 1 --2- 172.21.15.40:0/4106557577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 0x55d7260b9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.576+0000 7f45c8a00640 1 --2- 172.21.15.40:0/4106557577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 0x55d726098dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.576+0000 7f45c3e00640 1 -- 172.21.15.40:0/4106557577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 msgr2=0x55d7260b9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.576+0000 7f45c3e00640 1 --2- 172.21.15.40:0/4106557577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 0x55d7260b9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.576+0000 7f45c3e00640 1 -- 172.21.15.40:0/4106557577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 msgr2=0x55d726098dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.576+0000 7f45c3e00640 1 --2- 172.21.15.40:0/4106557577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 0x55d726098dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.576+0000 7f45c3e00640 1 -- 172.21.15.40:0/4106557577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d72609db60 con 0x55d72609c0a0 2024-04-26T23:07:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.576+0000 7f45c3e00640 1 --2- 172.21.15.40:0/4106557577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 0x55d7260c57c0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f45b00616a0 tx=0x7f45b000d6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237e0190bd3a0cf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:55.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.576+0000 7f45a9600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:07:55.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.576+0000 7f45a9600640 1 -- 172.21.15.40:0/4106557577 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b007c070 con 0x55d72609c0a0 2024-04-26T23:07:55.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.576+0000 7f45a9600640 1 -- 172.21.15.40:0/4106557577 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f45b0054d40 con 0x55d72609c0a0 2024-04-26T23:07:55.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.576+0000 7f45ce16c740 1 -- 172.21.15.40:0/4106557577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45bc02f040 con 0x55d72609c0a0 2024-04-26T23:07:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.577+0000 7f45a9600640 1 -- 172.21.15.40:0/4106557577 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f45b0054f40 con 0x55d72609c0a0 2024-04-26T23:07:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.577+0000 7f45ce16c740 10 client.8178 fetch_fsmap learned FSMap version 91 2024-04-26T23:07:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.577+0000 7f45ce16c740 1 -- 172.21.15.40:0/4106557577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) -- 0x55d7260c61b0 con 0x55d72609c0a0 2024-04-26T23:07:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.577+0000 7f45a9600640 1 -- 172.21.15.40:0/4106557577 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45bc02f040 con 0x55d72609c0a0 2024-04-26T23:07:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.577+0000 7f45a9600640 1 -- 172.21.15.40:0/4106557577 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 91) ==== 3494+0+0 (secure 0 0 0) 0x7f45b007f020 con 0x55d72609c0a0 2024-04-26T23:07:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.577+0000 7f45ce16c740 10 client.8178 fetch_fsmap finished waiting for FSMap version 91 2024-04-26T23:07:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.577+0000 7f45ce16c740 10 client.8178 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:07:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.577+0000 7f45ce16c740 20 client.8178 populate_metadata read hostname 'smithi040' 2024-04-26T23:07:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.577+0000 7f45ce16c740 1 --2- 172.21.15.40:0/4106557577 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d7260ba4f0 0x55d7260970a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.577+0000 7f45ce16c740 4 client.8178 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.577+0000 7f45ce16c740 1 -- 172.21.15.40:0/4106557577 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "status"}) -- 0x55d725fdd080 con 0x55d7260ba4f0 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.577+0000 7f45c8a00640 1 --2- 172.21.15.40:0/4106557577 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d7260ba4f0 0x55d7260970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.578+0000 7f45a9600640 10 client.8178 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.578+0000 7f45c8a00640 1 --2- 172.21.15.40:0/4106557577 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d7260ba4f0 0x55d7260970a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3450f2548b7dd0 server_cookie=c4cb4ea5ec06c6b0 in_seq=0 out_seq=0 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.578+0000 7f45a9600640 1 -- 172.21.15.40:0/4106557577 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55d725fdd080 con 0x55d7260ba4f0 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.578+0000 7f45a9600640 10 client.8178 handle_command_reply: tid=0 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.578+0000 7f45ce16c740 1 client.8178 shutdown 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.578+0000 7f45ce16c740 1 -- 172.21.15.40:0/4106557577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 msgr2=0x55d7260c57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.578+0000 7f45ce16c740 1 --2- 172.21.15.40:0/4106557577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 0x55d7260c57c0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f45b00616a0 tx=0x7f45b000d6f0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.579+0000 7f45ce16c740 1 -- 172.21.15.40:0/4106557577 shutdown_connections 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.579+0000 7f45ce16c740 1 --2- 172.21.15.40:0/4106557577 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d7260ba4f0 0x55d7260970a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.579+0000 7f45ce16c740 1 --2- 172.21.15.40:0/4106557577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d7260adcb0 0x55d7260b9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.579+0000 7f45ce16c740 1 --2- 172.21.15.40:0/4106557577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d72609c0a0 0x55d7260c57c0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.579+0000 7f45a9600640 0 client.8178 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:07:55.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.579+0000 7f45ce16c740 1 --2- 172.21.15.40:0/4106557577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7260715e0 0x55d726098dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.579+0000 7f45ce16c740 1 -- 172.21.15.40:0/4106557577 >> 172.21.15.40:0/4106557577 conn(0x55d726005d10 msgr2=0x55d72609e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:55.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.579+0000 7f45ce16c740 1 -- 172.21.15.40:0/4106557577 shutdown_connections 2024-04-26T23:07:55.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.579+0000 7f45ce16c740 1 -- 172.21.15.40:0/4106557577 wait complete. 2024-04-26T23:07:55.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.579+0000 7f45ce16c740 20 client.8178 trim_cache size 0 max 16384 2024-04-26T23:07:55.582 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:07:55.582 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:07:55.582 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:07:55.582 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8040, 2024-04-26T23:07:55.582 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:07:55.582 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:07:55.583 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:07:55.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 6.7776861640000003, 2024-04-26T23:07:55.583 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 91, 2024-04-26T23:07:55.583 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 90, 2024-04-26T23:07:55.583 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-26T23:07:55.583 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 12.14757723 2024-04-26T23:07:55.583 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:07:55.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.580+0000 7f45cb400640 1 -- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f45a406c920 msgr2=0x7f45a406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.580+0000 7f45cb400640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f45a406c920 0x7f45a406ede0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f45b80099e0 tx=0x7f45b8008040 comp rx=0 tx=0).stop 2024-04-26T23:07:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.580+0000 7f45cb400640 1 -- 172.21.15.40:0/1258186909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45c40b4a50 msgr2=0x7f45c40b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.580+0000 7f45cb400640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45c40b4a50 0x7f45c40b4ed0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f45b000d7a0 tx=0x7f45b000dc60 comp rx=0 tx=0).stop 2024-04-26T23:07:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.580+0000 7f45cb400640 1 -- 172.21.15.40:0/1258186909 shutdown_connections 2024-04-26T23:07:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.580+0000 7f45cb400640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f45a406c920 0x7f45a406ede0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.580+0000 7f45cb400640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45c40b8370 0x7f45c40b87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.580+0000 7f45cb400640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45c40b4a50 0x7f45c40b4ed0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.580+0000 7f45cb400640 1 --2- 172.21.15.40:0/1258186909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45ac004d40 0x7f45c40b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.580+0000 7f45cb400640 1 -- 172.21.15.40:0/1258186909 >> 172.21.15.40:0/1258186909 conn(0x7f45c40d1cf0 msgr2=0x7f45c406ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.580+0000 7f45cb400640 1 -- 172.21.15.40:0/1258186909 shutdown_connections 2024-04-26T23:07:55.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.580+0000 7f45cb400640 1 -- 172.21.15.40:0/1258186909 wait complete. 2024-04-26T23:07:55.597 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-26T23:07:55.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.795+0000 7f3d4ca00640 1 Processor -- start 2024-04-26T23:07:55.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.795+0000 7f3d4ca00640 1 -- start start 2024-04-26T23:07:55.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.797+0000 7f3d4ca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3d480d06b0 0x7f3d480d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.797+0000 7f3d4ca00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d480b59f0 con 0x7f3d48145970 2024-04-26T23:07:55.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.797+0000 7f3d4ca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d480b4180 con 0x7f3d48146060 2024-04-26T23:07:55.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.797+0000 7f3d4ca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3d480ceed0 con 0x7f3d480d06b0 2024-04-26T23:07:55.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.797+0000 7f3d47400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3d48145970 0x7f3d481612b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41996/0 (socket says 172.21.15.40:41996) 2024-04-26T23:07:55.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.797+0000 7f3d47400640 1 -- 172.21.15.40:0/1348356518 learned_addr learned my addr 172.21.15.40:0/1348356518 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:07:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.798+0000 7f3d47e00640 1 --2- 172.21.15.40:0/1348356518 >> v2:172.21.15.103:3300/0 conn(0x7f3d480d06b0 0x7f3d480d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.798+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1820495959 0 0) 0x7f3d480b59f0 con 0x7f3d48145970 2024-04-26T23:07:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.798+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d2c003520 con 0x7f3d48145970 2024-04-26T23:07:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.798+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3044818687 0 0) 0x7f3d2c003520 con 0x7f3d48145970 2024-04-26T23:07:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.798+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 >> v2:172.21.15.103:3300/0 conn(0x7f3d480d06b0 msgr2=0x7f3d480d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.798+0000 7f3d46000640 1 --2- 172.21.15.40:0/1348356518 >> v2:172.21.15.103:3300/0 conn(0x7f3d480d06b0 0x7f3d480d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.798+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 >> v1:172.21.15.103:6789/0 conn(0x7f3d48146060 legacy=0x7f3d480b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.798+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d480b5f40 con 0x7f3d48145970 2024-04-26T23:07:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.798+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f3d38001670 con 0x7f3d48145970 2024-04-26T23:07:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.798+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 >> v1:172.21.15.40:6789/0 conn(0x7f3d48145970 legacy=0x7f3d481612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.798+0000 7f3d46000640 1 --2- 172.21.15.40:0/1348356518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d2c0039b0 0x7f3d2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d46000640 1 --2- 172.21.15.40:0/1348356518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d2c004800 0x7f3d2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d46000640 1 --2- 172.21.15.40:0/1348356518 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d2c0051c0 0x7f3d2c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c00c030 con 0x7f3d2c0039b0 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c00c1e0 con 0x7f3d2c004800 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2c00c3e0 con 0x7f3d2c0051c0 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d47400640 1 --2- 172.21.15.40:0/1348356518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d2c0039b0 0x7f3d2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d46a00640 1 --2- 172.21.15.40:0/1348356518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d2c004800 0x7f3d2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d47e00640 1 --2- 172.21.15.40:0/1348356518 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d2c0051c0 0x7f3d2c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d47400640 1 -- 172.21.15.40:0/1348356518 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d2c0051c0 msgr2=0x7f3d2c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d47400640 1 --2- 172.21.15.40:0/1348356518 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d2c0051c0 0x7f3d2c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d47400640 1 -- 172.21.15.40:0/1348356518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d2c004800 msgr2=0x7f3d2c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d47400640 1 --2- 172.21.15.40:0/1348356518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d2c004800 0x7f3d2c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d47400640 1 -- 172.21.15.40:0/1348356518 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2c003520 con 0x7f3d2c0039b0 2024-04-26T23:07:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.799+0000 7f3d47e00640 1 -- 172.21.15.40:0/1348356518 reap_dead start 2024-04-26T23:07:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.800+0000 7f3d47400640 1 --2- 172.21.15.40:0/1348356518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d2c0039b0 0x7f3d2c003dd0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800c210 tx=0x7f3d2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50b3813b2f076f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.801+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38007d30 con 0x7f3d2c0039b0 2024-04-26T23:07:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.801+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d38005af0 con 0x7f3d2c0039b0 2024-04-26T23:07:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.801+0000 7f3d46000640 1 -- 172.21.15.40:0/1348356518 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38007070 con 0x7f3d2c0039b0 2024-04-26T23:07:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.801+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1348356518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d2c0039b0 msgr2=0x7f3d2c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.801+0000 7f3d4ca00640 1 --2- 172.21.15.40:0/1348356518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d2c0039b0 0x7f3d2c003dd0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800c210 tx=0x7f3d2c005830 comp rx=0 tx=0).stop 2024-04-26T23:07:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.801+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1348356518 shutdown_connections 2024-04-26T23:07:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.801+0000 7f3d4ca00640 1 --2- 172.21.15.40:0/1348356518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d2c0039b0 0x7f3d2c003dd0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.801+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1348356518 >> 172.21.15.40:0/1348356518 conn(0x7f3d48075eb0 msgr2=0x7f3d48068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.801+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1348356518 shutdown_connections 2024-04-26T23:07:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.802+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1348356518 wait complete. 2024-04-26T23:07:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.804+0000 7f3d4ca00640 1 Processor -- start 2024-04-26T23:07:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.804+0000 7f3d4ca00640 1 -- start start 2024-04-26T23:07:55.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.805+0000 7f3d4ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d2c0039b0 0x7f3d480ab590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.805+0000 7f3d4ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d4815c660 0x7f3d4813dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.805+0000 7f3d4ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d48143fb0 0x7f3d48144440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.805+0000 7f3d4ca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3d480cf3e0 con 0x7f3d2c0039b0 2024-04-26T23:07:55.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.805+0000 7f3d4ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3d480cf100 con 0x7f3d4815c660 2024-04-26T23:07:55.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.805+0000 7f3d4ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3d48145d50 con 0x7f3d48143fb0 2024-04-26T23:07:55.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.805+0000 7f3d47400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d2c0039b0 0x7f3d480ab590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.805+0000 7f3d47400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d2c0039b0 0x7f3d480ab590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44776/0 (socket says 172.21.15.40:44776) 2024-04-26T23:07:55.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.805+0000 7f3d47400640 1 -- 172.21.15.40:0/1419722694 learned_addr learned my addr 172.21.15.40:0/1419722694 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:55.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.806+0000 7f3d46a00640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d4815c660 0x7f3d4813dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.806+0000 7f3d47e00640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d48143fb0 0x7f3d48144440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.806+0000 7f3d47400640 1 -- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d48143fb0 msgr2=0x7f3d48144440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.806+0000 7f3d47400640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d48143fb0 0x7f3d48144440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.806+0000 7f3d47400640 1 -- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d4815c660 msgr2=0x7f3d4813dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:55.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.806+0000 7f3d47400640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d4815c660 0x7f3d4813dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:55.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.806+0000 7f3d47400640 1 -- 172.21.15.40:0/1419722694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d4813e140 con 0x7f3d2c0039b0 2024-04-26T23:07:55.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.806+0000 7f3d47400640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d2c0039b0 0x7f3d480ab590 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800c330 tx=0x7f3d38001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:55.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.807+0000 7f3d3f400640 1 -- 172.21.15.40:0/1419722694 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38037040 con 0x7f3d2c0039b0 2024-04-26T23:07:55.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.807+0000 7f3d3f400640 1 -- 172.21.15.40:0/1419722694 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d38041070 con 0x7f3d2c0039b0 2024-04-26T23:07:55.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.807+0000 7f3d3f400640 1 -- 172.21.15.40:0/1419722694 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38001670 con 0x7f3d2c0039b0 2024-04-26T23:07:55.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.807+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1419722694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d4813e410 con 0x7f3d2c0039b0 2024-04-26T23:07:55.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.807+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1419722694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d4813e9d0 con 0x7f3d2c0039b0 2024-04-26T23:07:55.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.808+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1419722694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d48069370 con 0x7f3d2c0039b0 2024-04-26T23:07:55.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.809+0000 7f3d3f400640 1 -- 172.21.15.40:0/1419722694 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d380049e0 con 0x7f3d2c0039b0 2024-04-26T23:07:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.810+0000 7f3d3f400640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3d2806c8f0 0x7f3d2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.810+0000 7f3d3f400640 1 -- 172.21.15.40:0/1419722694 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f3d380ab9b0 con 0x7f3d2c0039b0 2024-04-26T23:07:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.811+0000 7f3d46a00640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3d2806c8f0 0x7f3d2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:55.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.812+0000 7f3d46a00640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3d2806c8f0 0x7f3d2806edb0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3d34007fd0 tx=0x7f3d34009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:55.815+0000 7f3d3f400640 1 -- 172.21.15.40:0/1419722694 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3d38079bb0 con 0x7f3d2c0039b0 2024-04-26T23:07:56.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.071+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1419722694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3d480b55e0 con 0x7f3d2c0039b0 2024-04-26T23:07:56.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.072+0000 7f3d3f400640 1 -- 172.21.15.40:0/1419722694 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v91) ==== 53+0+73 (secure 0 0 0) 0x7f3d380794b0 con 0x7f3d2c0039b0 2024-04-26T23:07:56.072 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-26T23:07:56.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.075+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3d2806c8f0 msgr2=0x7f3d2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:56.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.075+0000 7f3d4ca00640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3d2806c8f0 0x7f3d2806edb0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3d34007fd0 tx=0x7f3d34009040 comp rx=0 tx=0).stop 2024-04-26T23:07:56.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.075+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1419722694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d2c0039b0 msgr2=0x7f3d480ab590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:56.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.075+0000 7f3d4ca00640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d2c0039b0 0x7f3d480ab590 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800c330 tx=0x7f3d38001a40 comp rx=0 tx=0).stop 2024-04-26T23:07:56.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.075+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1419722694 shutdown_connections 2024-04-26T23:07:56.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.075+0000 7f3d4ca00640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3d2806c8f0 0x7f3d2806edb0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:56.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.075+0000 7f3d4ca00640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d48143fb0 0x7f3d48144440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:56.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.075+0000 7f3d4ca00640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d4815c660 0x7f3d4813dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:56.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.075+0000 7f3d4ca00640 1 --2- 172.21.15.40:0/1419722694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d2c0039b0 0x7f3d480ab590 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:56.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.075+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1419722694 >> 172.21.15.40:0/1419722694 conn(0x7f3d48075eb0 msgr2=0x7f3d480b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:56.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.075+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1419722694 shutdown_connections 2024-04-26T23:07:56.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56.075+0000 7f3d4ca00640 1 -- 172.21.15.40:0/1419722694 wait complete. 2024-04-26T23:07:56.094 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-26T23:07:56.095 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:07:56.095 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-04-26T23:07:56.095 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T23:07:56.095 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T23:07:56.095 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-26T23:07:56.095 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-26T23:07:56.095 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:07:56.095 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:07:56.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:56.095 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-04-26T23:07:56.144 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:07:56.145 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:07:56.145 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:07:56.145 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:56.145 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-26T23:07:56.145 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:56.145 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:07:56.145 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:07:56.145 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-04-26T23:07:56.145 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:07:56.145 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:07:56.145 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-04-26T23:07:56.145 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:07:56.145 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42238sec preferred_lft 42238sec 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 8e:8e:79:d7:cf:5d brd ff:ff:ff:ff:ff:ff 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::4ce7:9bff:fe51:b1/64 scope link 2024-04-26T23:07:56.146 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:56.147 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:07:56.147 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether be:f1:28:e1:99:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T23:07:56.147 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::bcf1:28ff:fee1:9921/64 scope link 2024-04-26T23:07:56.147 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:56.147 INFO:teuthology.orchestra.run.smithi040.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:07:56.147 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 8e:8e:79:d7:cf:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-26T23:07:56.147 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::8c8e:79ff:fed7:cf5d/64 scope link 2024-04-26T23:07:56.147 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:07:56.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:56.148 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:07:56.233 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-04-26T23:07:56.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:56.237 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-04-26T23:07:56.293 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-04-26T23:07:56.293 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:07:56.293 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T23:07:56.294 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T23:07:56.294 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:07:56.295 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:07:56.295 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:07:56.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T23:07:56.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T23:07:56.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T23:07:56.295 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:07:56.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:07:56.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:07:56.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:56.524 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:07:56.577 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T23:07:56.577 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-26T23:07:56.578 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:07:56.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:56.578 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-04-26T23:07:56.662 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:07:56.662 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T23:07:56.719 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:07:56.720 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:07:56.720 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:07:56.773 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-26T23:07:56.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:56.774 DEBUG:teuthology.orchestra.run.smithi040:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:07:56.903 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:07:56.903 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:07:56.904 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:07:56.926 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:07:56.926 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:07:56.927 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:07:56.927 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:07:56.927 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:07:56.955 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:07:56.955 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:07:56.955 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:07:56.955 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:07:56.955 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:07:56.956 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-26T23:07:56.956 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-26T23:07:56.956 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=815936ab-d4ac-471e-b069-1a0dc70078ee" will pass to kernel 2024-04-26T23:07:56.958 INFO:teuthology.orchestra.run.smithi040.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T23:07:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: You just mounted a file system that supports labels which does not 2024-04-26T23:07:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T23:07:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T23:07:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T23:07:56.963 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T23:07:56.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:07:56.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:07:57.036 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:07:57.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:07:57.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:07:57.181 INFO:teuthology.orchestra.run.smithi040.stdout:client6328 2024-04-26T23:07:57.181 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:07:57.182 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:07:57.236 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-26T23:07:57.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.446+0000 7f0055c00640 1 Processor -- start 2024-04-26T23:07:57.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.447+0000 7f0055c00640 1 -- start start 2024-04-26T23:07:57.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.448+0000 7f0055c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0050070350 0x7f0050070b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:57.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.448+0000 7f0055c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050071090 0x7f005015e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:57.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.448+0000 7f0055c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00500d2390 con 0x7f00500cfde0 2024-04-26T23:07:57.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.448+0000 7f0055c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f0050071500 con 0x7f0050070350 2024-04-26T23:07:57.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.448+0000 7f0055c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f00500afab0 con 0x7f0050071090 2024-04-26T23:07:57.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.449+0000 7f004e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0050070350 0x7f0050070b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:57.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.449+0000 7f004e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0050070350 0x7f0050070b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44788/0 (socket says 172.21.15.40:44788) 2024-04-26T23:07:57.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.449+0000 7f004e000640 1 -- 172.21.15.40:0/2617177100 learned_addr learned my addr 172.21.15.40:0/2617177100 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:57.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.449+0000 7f004f400640 1 --2- 172.21.15.40:0/2617177100 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050071090 0x7f005015e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:57.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.449+0000 7f004e000640 1 -- 172.21.15.40:0/2617177100 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050071090 msgr2=0x7f005015e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:57.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.449+0000 7f004e000640 1 --2- 172.21.15.40:0/2617177100 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050071090 0x7f005015e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.449+0000 7f004d600640 1 -- 172.21.15.40:0/2617177100 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1038322632 0 0) 0x7f00500d2390 con 0x7f00500cfde0 2024-04-26T23:07:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.449+0000 7f004e000640 1 -- 172.21.15.40:0/2617177100 >> v1:172.21.15.40:6789/0 conn(0x7f00500cfde0 legacy=0x7f00500d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.449+0000 7f004e000640 1 -- 172.21.15.40:0/2617177100 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005006d3f0 con 0x7f0050070350 2024-04-26T23:07:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.449+0000 7f004f400640 1 --2- 172.21.15.40:0/2617177100 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050071090 0x7f005015e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.450+0000 7f004e000640 1 --2- 172.21.15.40:0/2617177100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050070350 0x7f0050070b50 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f003c009870 tx=0x7f003c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536833a1ee8c2efb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.451+0000 7f004d600640 1 -- 172.21.15.40:0/2617177100 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c037070 con 0x7f0050070350 2024-04-26T23:07:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.451+0000 7f004d600640 1 -- 172.21.15.40:0/2617177100 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f003c032440 con 0x7f0050070350 2024-04-26T23:07:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.451+0000 7f004d600640 1 -- 172.21.15.40:0/2617177100 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c032810 con 0x7f0050070350 2024-04-26T23:07:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.451+0000 7f0055c00640 1 -- 172.21.15.40:0/2617177100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050070350 msgr2=0x7f0050070b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.451+0000 7f0055c00640 1 --2- 172.21.15.40:0/2617177100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050070350 0x7f0050070b50 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f003c009870 tx=0x7f003c02a1c0 comp rx=0 tx=0).stop 2024-04-26T23:07:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.451+0000 7f0055c00640 1 -- 172.21.15.40:0/2617177100 shutdown_connections 2024-04-26T23:07:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.451+0000 7f0055c00640 1 --2- 172.21.15.40:0/2617177100 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0050071090 0x7f005015e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.451+0000 7f0055c00640 1 --2- 172.21.15.40:0/2617177100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050070350 0x7f0050070b50 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.451+0000 7f0055c00640 1 -- 172.21.15.40:0/2617177100 >> 172.21.15.40:0/2617177100 conn(0x7f00500d2d80 msgr2=0x7f00500d3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.452+0000 7f0055c00640 1 -- 172.21.15.40:0/2617177100 shutdown_connections 2024-04-26T23:07:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.452+0000 7f0055c00640 1 -- 172.21.15.40:0/2617177100 wait complete. 2024-04-26T23:07:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.453+0000 7f0055c00640 1 Processor -- start 2024-04-26T23:07:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.453+0000 7f0055c00640 1 -- start start 2024-04-26T23:07:57.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.454+0000 7f0055c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0050070350 0x7f005009d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:57.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.454+0000 7f0055c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071090 0x7f00501f89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:57.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.454+0000 7f0055c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00500cfde0 0x7f0050158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:57.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.454+0000 7f0055c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f005006d9b0 con 0x7f0050071090 2024-04-26T23:07:57.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.454+0000 7f0055c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f005006dd20 con 0x7f0050070350 2024-04-26T23:07:57.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.454+0000 7f0055c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f005015ef30 con 0x7f00500cfde0 2024-04-26T23:07:57.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.455+0000 7f004e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071090 0x7f00501f89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:57.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.455+0000 7f004e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071090 0x7f00501f89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44802/0 (socket says 172.21.15.40:44802) 2024-04-26T23:07:57.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.455+0000 7f004e000640 1 -- 172.21.15.40:0/3431089259 learned_addr learned my addr 172.21.15.40:0/3431089259 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.455+0000 7f004f400640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00500cfde0 0x7f0050158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.455+0000 7f004e000640 1 -- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00500cfde0 msgr2=0x7f0050158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.455+0000 7f004e000640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00500cfde0 0x7f0050158c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.455+0000 7f004ea00640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0050070350 0x7f005009d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.455+0000 7f004e000640 1 -- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0050070350 msgr2=0x7f005009d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.455+0000 7f004e000640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0050070350 0x7f005009d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.455+0000 7f004e000640 1 -- 172.21.15.40:0/3431089259 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005009f240 con 0x7f0050071090 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.455+0000 7f004ea00640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0050070350 0x7f005009d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.456+0000 7f004e000640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071090 0x7f00501f89e0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f003c035040 tx=0x7f003c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.456+0000 7f0046a00640 1 -- 172.21.15.40:0/3431089259 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c037040 con 0x7f0050071090 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.456+0000 7f0046a00640 1 -- 172.21.15.40:0/3431089259 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f003c004230 con 0x7f0050071090 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.456+0000 7f0046a00640 1 -- 172.21.15.40:0/3431089259 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c003610 con 0x7f0050071090 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.456+0000 7f0055c00640 1 -- 172.21.15.40:0/3431089259 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f005009f520 con 0x7f0050071090 2024-04-26T23:07:57.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.456+0000 7f0055c00640 1 -- 172.21.15.40:0/3431089259 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f005009fae0 con 0x7f0050071090 2024-04-26T23:07:57.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.457+0000 7f0055c00640 1 -- 172.21.15.40:0/3431089259 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00500694f0 con 0x7f0050071090 2024-04-26T23:07:57.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.458+0000 7f0046a00640 1 -- 172.21.15.40:0/3431089259 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f003c02ad10 con 0x7f0050071090 2024-04-26T23:07:57.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.459+0000 7f0046a00640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f003006c8f0 0x7f003006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:57.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.459+0000 7f0046a00640 1 -- 172.21.15.40:0/3431089259 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f003c0aca70 con 0x7f0050071090 2024-04-26T23:07:57.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.460+0000 7f004ea00640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f003006c8f0 0x7f003006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:57.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.461+0000 7f004ea00640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f003006c8f0 0x7f003006edb0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0040000f60 tx=0x7f0040009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.464+0000 7f0046a00640 1 -- 172.21.15.40:0/3431089259 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f003c07ad90 con 0x7f0050071090 2024-04-26T23:07:57.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.711+0000 7f0055c00640 1 -- 172.21.15.40:0/3431089259 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f00500d24a0 con 0x7f0050071090 2024-04-26T23:07:57.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.859+0000 7f0046a00640 1 -- 172.21.15.40:0/3431089259 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v92) ==== 105+0+0 (secure 0 0 0) 0x7f003c07a690 con 0x7f0050071090 2024-04-26T23:07:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.862+0000 7f0055c00640 1 -- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f003006c8f0 msgr2=0x7f003006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.862+0000 7f0055c00640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f003006c8f0 0x7f003006edb0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0040000f60 tx=0x7f0040009040 comp rx=0 tx=0).stop 2024-04-26T23:07:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.862+0000 7f0055c00640 1 -- 172.21.15.40:0/3431089259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071090 msgr2=0x7f00501f89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.862+0000 7f0055c00640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071090 0x7f00501f89e0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f003c035040 tx=0x7f003c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:07:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.863+0000 7f0055c00640 1 -- 172.21.15.40:0/3431089259 shutdown_connections 2024-04-26T23:07:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.863+0000 7f0055c00640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f003006c8f0 0x7f003006edb0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.863+0000 7f0055c00640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00500cfde0 0x7f0050158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.863+0000 7f0055c00640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0050071090 0x7f00501f89e0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.863+0000 7f0055c00640 1 --2- 172.21.15.40:0/3431089259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0050070350 0x7f005009d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.863+0000 7f0055c00640 1 -- 172.21.15.40:0/3431089259 >> 172.21.15.40:0/3431089259 conn(0x7f00500d2d80 msgr2=0x7f00500cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.863+0000 7f0055c00640 1 -- 172.21.15.40:0/3431089259 shutdown_connections 2024-04-26T23:07:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:57.863+0000 7f0055c00640 1 -- 172.21.15.40:0/3431089259 wait complete. 2024-04-26T23:07:57.890 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.073+0000 7f85fd400640 1 Processor -- start 2024-04-26T23:07:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.074+0000 7f85fd400640 1 -- start start 2024-04-26T23:07:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.075+0000 7f85fd400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f85f815ce60 0x7f85f80afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.075+0000 7f85fd400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f80af7a0 0x7f85f80ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.075+0000 7f85fd400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85f8071fe0 0x7f85f8073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.075+0000 7f85fd400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f85f80cc450 con 0x7f85f8071fe0 2024-04-26T23:07:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.075+0000 7f85fd400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f85f80d0820 con 0x7f85f815ce60 2024-04-26T23:07:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.075+0000 7f85fd400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f85f81614c0 con 0x7f85f80af7a0 2024-04-26T23:07:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.075+0000 7f85fca00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85f8071fe0 0x7f85f8073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.076+0000 7f85fca00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85f8071fe0 0x7f85f8073760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44808/0 (socket says 172.21.15.40:44808) 2024-04-26T23:07:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.076+0000 7f85fca00640 1 -- 172.21.15.40:0/3959711814 learned_addr learned my addr 172.21.15.40:0/3959711814 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.076+0000 7f85f7400640 1 --2- 172.21.15.40:0/3959711814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f80af7a0 0x7f85f80ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.076+0000 7f85f7e00640 1 --2- 172.21.15.40:0/3959711814 >> v2:172.21.15.103:3300/0 conn(0x7f85f815ce60 0x7f85f80afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.076+0000 7f85fca00640 1 -- 172.21.15.40:0/3959711814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f80af7a0 msgr2=0x7f85f80ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.076+0000 7f85fca00640 1 --2- 172.21.15.40:0/3959711814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f80af7a0 0x7f85f80ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.076+0000 7f85fca00640 1 -- 172.21.15.40:0/3959711814 >> v2:172.21.15.103:3300/0 conn(0x7f85f815ce60 msgr2=0x7f85f80afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.076+0000 7f85fca00640 1 --2- 172.21.15.40:0/3959711814 >> v2:172.21.15.103:3300/0 conn(0x7f85f815ce60 0x7f85f80afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.076+0000 7f85fca00640 1 -- 172.21.15.40:0/3959711814 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f8161760 con 0x7f85f8071fe0 2024-04-26T23:07:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.077+0000 7f85fca00640 1 --2- 172.21.15.40:0/3959711814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f8071fe0 0x7f85f8073760 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f85f80741b0 tx=0x7f85f002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e559827219a3c858 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.078+0000 7f85f6a00640 1 -- 172.21.15.40:0/3959711814 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f0036070 con 0x7f85f8071fe0 2024-04-26T23:07:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.078+0000 7f85f6a00640 1 -- 172.21.15.40:0/3959711814 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85f0031600 con 0x7f85f8071fe0 2024-04-26T23:07:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.078+0000 7f85f6a00640 1 -- 172.21.15.40:0/3959711814 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f00319d0 con 0x7f85f8071fe0 2024-04-26T23:07:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.078+0000 7f85fd400640 1 -- 172.21.15.40:0/3959711814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f8071fe0 msgr2=0x7f85f8073760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.078+0000 7f85fd400640 1 --2- 172.21.15.40:0/3959711814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f8071fe0 0x7f85f8073760 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f85f80741b0 tx=0x7f85f002a610 comp rx=0 tx=0).stop 2024-04-26T23:07:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.078+0000 7f85fd400640 1 -- 172.21.15.40:0/3959711814 shutdown_connections 2024-04-26T23:07:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.078+0000 7f85fd400640 1 --2- 172.21.15.40:0/3959711814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f8071fe0 0x7f85f8073760 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.078+0000 7f85fd400640 1 --2- 172.21.15.40:0/3959711814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f80af7a0 0x7f85f80ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.078+0000 7f85fd400640 1 --2- 172.21.15.40:0/3959711814 >> v2:172.21.15.103:3300/0 conn(0x7f85f815ce60 0x7f85f80afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.078+0000 7f85fd400640 1 -- 172.21.15.40:0/3959711814 >> 172.21.15.40:0/3959711814 conn(0x7f85f80d0a10 msgr2=0x7f85f80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.079+0000 7f85fd400640 1 -- 172.21.15.40:0/3959711814 shutdown_connections 2024-04-26T23:07:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.079+0000 7f85fd400640 1 -- 172.21.15.40:0/3959711814 wait complete. 2024-04-26T23:07:58.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.080+0000 7f85fd400640 1 Processor -- start 2024-04-26T23:07:58.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.080+0000 7f85fd400640 1 -- start start 2024-04-26T23:07:58.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.081+0000 7f85fd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f8071fe0 0x7f85f815ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:58.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.081+0000 7f85fd400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f80af7a0 0x7f85f8150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:58.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.081+0000 7f85fd400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85f815ce60 0x7f85f80b1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:58.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.081+0000 7f85fd400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f85f8073ca0 con 0x7f85f8071fe0 2024-04-26T23:07:58.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.081+0000 7f85fd400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f85f80cf530 con 0x7f85f815ce60 2024-04-26T23:07:58.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.081+0000 7f85fd400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f85f8073e20 con 0x7f85f80af7a0 2024-04-26T23:07:58.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.082+0000 7f85f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f8071fe0 0x7f85f815ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:58.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.082+0000 7f85f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f8071fe0 0x7f85f815ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44816/0 (socket says 172.21.15.40:44816) 2024-04-26T23:07:58.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.082+0000 7f85f7e00640 1 -- 172.21.15.40:0/1562750116 learned_addr learned my addr 172.21.15.40:0/1562750116 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:58.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.082+0000 7f85f7400640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f80af7a0 0x7f85f8150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:58.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.082+0000 7f85fca00640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85f815ce60 0x7f85f80b1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:58.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.082+0000 7f85f7e00640 1 -- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f80af7a0 msgr2=0x7f85f8150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:58.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.082+0000 7f85f7e00640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f80af7a0 0x7f85f8150ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:58.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.082+0000 7f85f7e00640 1 -- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85f815ce60 msgr2=0x7f85f80b1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:58.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.082+0000 7f85f7e00640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85f815ce60 0x7f85f80b1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:58.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.082+0000 7f85f7e00640 1 -- 172.21.15.40:0/1562750116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f81f4b60 con 0x7f85f8071fe0 2024-04-26T23:07:58.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.082+0000 7f85f7400640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f80af7a0 0x7f85f8150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:58.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.082+0000 7f85fca00640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85f815ce60 0x7f85f80b1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:07:58.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.083+0000 7f85f7e00640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f8071fe0 0x7f85f815ae10 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f85e800d5f0 tx=0x7f85e800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:58.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.083+0000 7f85efe00640 1 -- 172.21.15.40:0/1562750116 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e80125f0 con 0x7f85f8071fe0 2024-04-26T23:07:58.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.083+0000 7f85efe00640 1 -- 172.21.15.40:0/1562750116 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e8019070 con 0x7f85f8071fe0 2024-04-26T23:07:58.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.083+0000 7f85efe00640 1 -- 172.21.15.40:0/1562750116 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e80129d0 con 0x7f85f8071fe0 2024-04-26T23:07:58.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.083+0000 7f85fd400640 1 -- 172.21.15.40:0/1562750116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85f81f4e40 con 0x7f85f8071fe0 2024-04-26T23:07:58.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.083+0000 7f85fd400640 1 -- 172.21.15.40:0/1562750116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85f81f5400 con 0x7f85f8071fe0 2024-04-26T23:07:58.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.084+0000 7f85fd400640 1 -- 172.21.15.40:0/1562750116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85f8069370 con 0x7f85f8071fe0 2024-04-26T23:07:58.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.085+0000 7f85efe00640 1 -- 172.21.15.40:0/1562750116 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f85e8012b70 con 0x7f85f8071fe0 2024-04-26T23:07:58.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.087+0000 7f85efe00640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85d406c8f0 0x7f85d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:58.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.087+0000 7f85efe00640 1 -- 172.21.15.40:0/1562750116 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f85e808c040 con 0x7f85f8071fe0 2024-04-26T23:07:58.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.087+0000 7f85f7400640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85d406c8f0 0x7f85d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:58.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.089+0000 7f85f7400640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85d406c8f0 0x7f85d406edb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f85f80cd0b0 tx=0x7f85e0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.091+0000 7f85efe00640 1 -- 172.21.15.40:0/1562750116 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f85e805a2e0 con 0x7f85f8071fe0 2024-04-26T23:07:58.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.333+0000 7f85fd400640 1 -- 172.21.15.40:0/1562750116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85f80d01f0 con 0x7f85f8071fe0 2024-04-26T23:07:58.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.335+0000 7f85efe00640 1 -- 172.21.15.40:0/1562750116 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+5199 (secure 0 0 0) 0x7f85e8059be0 con 0x7f85f8071fe0 2024-04-26T23:07:58.335 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:07:58.335 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":93,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:07:57.865195+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1],"up":{"mds_0":8040,"mds_1":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:creating","state_seq":1,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[8040]},"id":6}]} 2024-04-26T23:07:58.335 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 93 2024-04-26T23:07:58.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.337+0000 7f85fd400640 1 -- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85d406c8f0 msgr2=0x7f85d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:58.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.337+0000 7f85fd400640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85d406c8f0 0x7f85d406edb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f85f80cd0b0 tx=0x7f85e0008040 comp rx=0 tx=0).stop 2024-04-26T23:07:58.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.337+0000 7f85fd400640 1 -- 172.21.15.40:0/1562750116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f8071fe0 msgr2=0x7f85f815ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:58.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.337+0000 7f85fd400640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f8071fe0 0x7f85f815ae10 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f85e800d5f0 tx=0x7f85e800dab0 comp rx=0 tx=0).stop 2024-04-26T23:07:58.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.337+0000 7f85fd400640 1 -- 172.21.15.40:0/1562750116 shutdown_connections 2024-04-26T23:07:58.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.337+0000 7f85fd400640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f85d406c8f0 0x7f85d406edb0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:58.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.337+0000 7f85fd400640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85f815ce60 0x7f85f80b1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:58.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.337+0000 7f85fd400640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85f80af7a0 0x7f85f8150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:58.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.337+0000 7f85fd400640 1 --2- 172.21.15.40:0/1562750116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85f8071fe0 0x7f85f815ae10 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:58.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.338+0000 7f85fd400640 1 -- 172.21.15.40:0/1562750116 >> 172.21.15.40:0/1562750116 conn(0x7f85f80d0a10 msgr2=0x7f85f80b0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:58.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.338+0000 7f85fd400640 1 -- 172.21.15.40:0/1562750116 shutdown_connections 2024-04-26T23:07:58.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:58.338+0000 7f85fd400640 1 -- 172.21.15.40:0/1562750116 wait complete. 2024-04-26T23:07:58.758 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 93, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:07:48.279070+0000', 'modified': '2024-04-26T23:07:57.865195+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 8040, 'mds_1': 8031}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8031': {'gid': 8031, 'name': 'a', 'rank': 1, 'incarnation': 93, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.40:6833/1811310743', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 1811310743}, {'type': 'v1', 'addr': '172.21.15.40:6833', 'nonce': 1811310743}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8040': {'gid': 8040, 'name': 'c', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/88401598', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 88401598}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 88401598}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8040, 'qdb_cluster': [8040]} 2024-04-26T23:07:58.758 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8031:up:creating 2024-04-26T23:07:59.754 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:07:59.754+0000 7fd9e1000640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T23:07:59.759 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:07:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.957+0000 7faecf200640 1 Processor -- start 2024-04-26T23:07:59.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.957+0000 7faecf200640 1 -- start start 2024-04-26T23:07:59.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.958+0000 7faecf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed00b6220 0x7faed00d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:59.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.958+0000 7faecf200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faed00d39f0 0x7faed00b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:59.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.959+0000 7faecf200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faed0162c40 con 0x7faed00d08d0 2024-04-26T23:07:59.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.959+0000 7faecf200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7faed00d2d80 con 0x7faed00d39f0 2024-04-26T23:07:59.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.959+0000 7faecf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faed00b7c80 con 0x7faed00b6220 2024-04-26T23:07:59.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.959+0000 7faece800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faed00d39f0 0x7faed00b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:59.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.959+0000 7faece800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faed00d39f0 0x7faed00b8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44832/0 (socket says 172.21.15.40:44832) 2024-04-26T23:07:59.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.959+0000 7faece800640 1 -- 172.21.15.40:0/926929346 learned_addr learned my addr 172.21.15.40:0/926929346 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:59.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.959+0000 7faecde00640 1 --2- 172.21.15.40:0/926929346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed00b6220 0x7faed00d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:59.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.959+0000 7faece800640 1 -- 172.21.15.40:0/926929346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed00b6220 msgr2=0x7faed00d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:59.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.959+0000 7faece800640 1 --2- 172.21.15.40:0/926929346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed00b6220 0x7faed00d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:59.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.959+0000 7faece800640 1 -- 172.21.15.40:0/926929346 >> v1:172.21.15.103:6789/0 conn(0x7faed00d08d0 legacy=0x7faed00d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:07:59.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.959+0000 7faece800640 1 -- 172.21.15.40:0/926929346 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed00b7f60 con 0x7faed00d39f0 2024-04-26T23:07:59.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.961+0000 7faece800640 1 --2- 172.21.15.40:0/926929346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00d39f0 0x7faed00b8d20 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7faec8009e50 tx=0x7faec802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2902b4d1bd50266c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:59.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.961+0000 7faecca00640 1 -- 172.21.15.40:0/926929346 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8036070 con 0x7faed00d39f0 2024-04-26T23:07:59.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.961+0000 7faecca00640 1 -- 172.21.15.40:0/926929346 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec8031600 con 0x7faed00d39f0 2024-04-26T23:07:59.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.961+0000 7faecca00640 1 -- 172.21.15.40:0/926929346 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec80319d0 con 0x7faed00d39f0 2024-04-26T23:07:59.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.962+0000 7faecf200640 1 -- 172.21.15.40:0/926929346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00d39f0 msgr2=0x7faed00b8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:59.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.962+0000 7faecf200640 1 --2- 172.21.15.40:0/926929346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00d39f0 0x7faed00b8d20 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7faec8009e50 tx=0x7faec802a6c0 comp rx=0 tx=0).stop 2024-04-26T23:07:59.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.962+0000 7faecf200640 1 -- 172.21.15.40:0/926929346 shutdown_connections 2024-04-26T23:07:59.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.962+0000 7faecf200640 1 --2- 172.21.15.40:0/926929346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00d39f0 0x7faed00b8d20 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:59.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.962+0000 7faecf200640 1 --2- 172.21.15.40:0/926929346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed00b6220 0x7faed00d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:59.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.962+0000 7faecf200640 1 -- 172.21.15.40:0/926929346 >> 172.21.15.40:0/926929346 conn(0x7faed00d4d90 msgr2=0x7faed00d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:07:59.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.962+0000 7faecf200640 1 -- 172.21.15.40:0/926929346 shutdown_connections 2024-04-26T23:07:59.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.962+0000 7faecf200640 1 -- 172.21.15.40:0/926929346 wait complete. 2024-04-26T23:07:59.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.964+0000 7faecf200640 1 Processor -- start 2024-04-26T23:07:59.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.964+0000 7faecf200640 1 -- start start 2024-04-26T23:07:59.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.965+0000 7faecf200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed00b6220 0x7faed01f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:59.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.965+0000 7faecf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed00d08d0 0x7faed00cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:59.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.965+0000 7faecf200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00d39f0 0x7faed00cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:59.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.965+0000 7faecf200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faed00b9260 con 0x7faed00d39f0 2024-04-26T23:07:59.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.965+0000 7faecf200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faed00b8520 con 0x7faed00b6220 2024-04-26T23:07:59.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.965+0000 7faecf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faed00b93e0 con 0x7faed00d08d0 2024-04-26T23:07:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.965+0000 7faece800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00d39f0 0x7faed00cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.965+0000 7faece800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00d39f0 0x7faed00cdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44834/0 (socket says 172.21.15.40:44834) 2024-04-26T23:07:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.965+0000 7faece800640 1 -- 172.21.15.40:0/673219477 learned_addr learned my addr 172.21.15.40:0/673219477 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:07:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.965+0000 7faecd400640 1 --2- 172.21.15.40:0/673219477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed00d08d0 0x7faed00cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.966+0000 7faece800640 1 -- 172.21.15.40:0/673219477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed00d08d0 msgr2=0x7faed00cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.966+0000 7faece800640 1 --2- 172.21.15.40:0/673219477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed00d08d0 0x7faed00cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.966+0000 7faece800640 1 -- 172.21.15.40:0/673219477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed00b6220 msgr2=0x7faed01f4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:07:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.966+0000 7faece800640 1 --2- 172.21.15.40:0/673219477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed00b6220 0x7faed01f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:07:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.966+0000 7faece800640 1 -- 172.21.15.40:0/673219477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed00c99b0 con 0x7faed00d39f0 2024-04-26T23:07:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.966+0000 7faece800640 1 --2- 172.21.15.40:0/673219477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00d39f0 0x7faed00cdde0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7faec8009f70 tx=0x7faec8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:59.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.966+0000 7faec6000640 1 -- 172.21.15.40:0/673219477 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8041020 con 0x7faed00d39f0 2024-04-26T23:07:59.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.966+0000 7faec6000640 1 -- 172.21.15.40:0/673219477 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec8037040 con 0x7faed00d39f0 2024-04-26T23:07:59.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.966+0000 7faec6000640 1 -- 172.21.15.40:0/673219477 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8036070 con 0x7faed00d39f0 2024-04-26T23:07:59.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.966+0000 7faecf200640 1 -- 172.21.15.40:0/673219477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed00c9c90 con 0x7faed00d39f0 2024-04-26T23:07:59.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.966+0000 7faecf200640 1 -- 172.21.15.40:0/673219477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed00ca250 con 0x7faed00d39f0 2024-04-26T23:07:59.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.967+0000 7faecf200640 1 -- 172.21.15.40:0/673219477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed0069370 con 0x7faed00d39f0 2024-04-26T23:07:59.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.969+0000 7faec6000640 1 -- 172.21.15.40:0/673219477 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faec8044020 con 0x7faed00d39f0 2024-04-26T23:07:59.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.970+0000 7faec6000640 1 --2- 172.21.15.40:0/673219477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeb006c8f0 0x7faeb006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:07:59.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.970+0000 7faec6000640 1 -- 172.21.15.40:0/673219477 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7faec80ac890 con 0x7faed00d39f0 2024-04-26T23:07:59.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.971+0000 7faecde00640 1 --2- 172.21.15.40:0/673219477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeb006c8f0 0x7faeb006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:07:59.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.972+0000 7faecde00640 1 --2- 172.21.15.40:0/673219477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeb006c8f0 0x7faeb006edb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7faec00098f0 tx=0x7faec0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:07:59.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:07:59.979+0000 7faec6000640 1 -- 172.21.15.40:0/673219477 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7faec804b030 con 0x7faed00d39f0 2024-04-26T23:08:00.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.219+0000 7faecf200640 1 -- 172.21.15.40:0/673219477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faed00a2ae0 con 0x7faed00d39f0 2024-04-26T23:08:00.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.220+0000 7faec6000640 1 -- 172.21.15.40:0/673219477 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5221 (secure 0 0 0) 0x7faec807ab20 con 0x7faed00d39f0 2024-04-26T23:08:00.220 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:00.220 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":96,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:07:59.880947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:00.220 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 96 2024-04-26T23:08:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.222+0000 7faecf200640 1 -- 172.21.15.40:0/673219477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeb006c8f0 msgr2=0x7faeb006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.222+0000 7faecf200640 1 --2- 172.21.15.40:0/673219477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeb006c8f0 0x7faeb006edb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7faec00098f0 tx=0x7faec0008040 comp rx=0 tx=0).stop 2024-04-26T23:08:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.222+0000 7faecf200640 1 -- 172.21.15.40:0/673219477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00d39f0 msgr2=0x7faed00cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.222+0000 7faecf200640 1 --2- 172.21.15.40:0/673219477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00d39f0 0x7faed00cdde0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7faec8009f70 tx=0x7faec8004a40 comp rx=0 tx=0).stop 2024-04-26T23:08:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.222+0000 7faecf200640 1 -- 172.21.15.40:0/673219477 shutdown_connections 2024-04-26T23:08:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.222+0000 7faecf200640 1 --2- 172.21.15.40:0/673219477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faeb006c8f0 0x7faeb006edb0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.222+0000 7faecf200640 1 --2- 172.21.15.40:0/673219477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed00d39f0 0x7faed00cdde0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.222+0000 7faecf200640 1 --2- 172.21.15.40:0/673219477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faed00d08d0 0x7faed00cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.222+0000 7faecf200640 1 --2- 172.21.15.40:0/673219477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faed00b6220 0x7faed01f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.222+0000 7faecf200640 1 -- 172.21.15.40:0/673219477 >> 172.21.15.40:0/673219477 conn(0x7faed00d4d90 msgr2=0x7faed006cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.223+0000 7faecf200640 1 -- 172.21.15.40:0/673219477 shutdown_connections 2024-04-26T23:08:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.223+0000 7faecf200640 1 -- 172.21.15.40:0/673219477 wait complete. 2024-04-26T23:08:00.643 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 96, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:07:48.279070+0000', 'modified': '2024-04-26T23:07:59.880947+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 8040, 'mds_1': 8031, 'mds_2': 6253}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6253': {'gid': 6253, 'name': 'd', 'rank': 2, 'incarnation': 95, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.103:6837/2728939257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6836', 'nonce': 2728939257}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 2728939257}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8031': {'gid': 8031, 'name': 'a', 'rank': 1, 'incarnation': 93, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.40:6833/1811310743', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 1811310743}, {'type': 'v1', 'addr': '172.21.15.40:6833', 'nonce': 1811310743}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8040': {'gid': 8040, 'name': 'c', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/88401598', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 88401598}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 88401598}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8040, 'qdb_cluster': [6253, 8031, 8040]} 2024-04-26T23:08:00.643 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T23:08:00.644 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-26T23:08:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.850+0000 7fc22b200640 1 Processor -- start 2024-04-26T23:08:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.851+0000 7fc22b200640 1 -- start start 2024-04-26T23:08:00.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.852+0000 7fc22b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc22c0b4770 0x7fc22c0d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.852+0000 7fc22b200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc22c0b63e0 0x7fc22c161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.852+0000 7fc22b200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22c0b4110 con 0x7fc22c161850 2024-04-26T23:08:00.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.852+0000 7fc22b200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc22c0d3850 con 0x7fc22c0b63e0 2024-04-26T23:08:00.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.852+0000 7fc22b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc22c0d3b30 con 0x7fc22c0b4770 2024-04-26T23:08:00.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.853+0000 7fc229e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc22c161850 0x7fc22c0d1440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42004/0 (socket says 172.21.15.40:42004) 2024-04-26T23:08:00.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.853+0000 7fc229e00640 1 -- 172.21.15.40:0/231171866 learned_addr learned my addr 172.21.15.40:0/231171866 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:00.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.853+0000 7fc22a800640 1 --2- 172.21.15.40:0/231171866 >> v2:172.21.15.40:3300/0 conn(0x7fc22c0b63e0 0x7fc22c161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.853+0000 7fc229400640 1 --2- 172.21.15.40:0/231171866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc22c0b4770 0x7fc22c0d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.853+0000 7fc228a00640 1 -- 172.21.15.40:0/231171866 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292125095 0 0) 0x7fc22c0b4110 con 0x7fc22c161850 2024-04-26T23:08:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.853+0000 7fc228a00640 1 -- 172.21.15.40:0/231171866 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc210003520 con 0x7fc22c161850 2024-04-26T23:08:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.853+0000 7fc22a800640 1 -- 172.21.15.40:0/231171866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc22c0b4770 msgr2=0x7fc22c0d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.853+0000 7fc22a800640 1 --2- 172.21.15.40:0/231171866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc22c0b4770 0x7fc22c0d1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.853+0000 7fc22a800640 1 -- 172.21.15.40:0/231171866 >> v1:172.21.15.40:6789/0 conn(0x7fc22c161850 legacy=0x7fc22c0d1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:00.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.853+0000 7fc22a800640 1 -- 172.21.15.40:0/231171866 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22c0b3d70 con 0x7fc22c0b63e0 2024-04-26T23:08:00.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.854+0000 7fc22a800640 1 --2- 172.21.15.40:0/231171866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c0b63e0 0x7fc22c161c30 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fc224009d90 tx=0x7fc22402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b92305bc70379b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.855+0000 7fc228a00640 1 -- 172.21.15.40:0/231171866 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224031600 con 0x7fc22c0b63e0 2024-04-26T23:08:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.855+0000 7fc228a00640 1 -- 172.21.15.40:0/231171866 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2240317a0 con 0x7fc22c0b63e0 2024-04-26T23:08:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.855+0000 7fc228a00640 1 -- 172.21.15.40:0/231171866 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224031b70 con 0x7fc22c0b63e0 2024-04-26T23:08:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.855+0000 7fc22b200640 1 -- 172.21.15.40:0/231171866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c0b63e0 msgr2=0x7fc22c161c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.855+0000 7fc22b200640 1 --2- 172.21.15.40:0/231171866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c0b63e0 0x7fc22c161c30 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fc224009d90 tx=0x7fc22402a5d0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.855+0000 7fc22b200640 1 -- 172.21.15.40:0/231171866 shutdown_connections 2024-04-26T23:08:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.855+0000 7fc22b200640 1 --2- 172.21.15.40:0/231171866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c0b63e0 0x7fc22c161c30 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.855+0000 7fc22b200640 1 --2- 172.21.15.40:0/231171866 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc22c0b4770 0x7fc22c0d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.855+0000 7fc22b200640 1 -- 172.21.15.40:0/231171866 >> 172.21.15.40:0/231171866 conn(0x7fc22c0d4e10 msgr2=0x7fc22c0d5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.855+0000 7fc22b200640 1 -- 172.21.15.40:0/231171866 shutdown_connections 2024-04-26T23:08:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.855+0000 7fc22b200640 1 -- 172.21.15.40:0/231171866 wait complete. 2024-04-26T23:08:00.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.857+0000 7fc22b200640 1 Processor -- start 2024-04-26T23:08:00.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.857+0000 7fc22b200640 1 -- start start 2024-04-26T23:08:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.858+0000 7fc22b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc22c0b4770 0x7fc22c153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.858+0000 7fc22b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c0b63e0 0x7fc22c09fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.858+0000 7fc22b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc22c161850 0x7fc22c0b9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.858+0000 7fc22b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c0d2210 con 0x7fc22c0b63e0 2024-04-26T23:08:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.858+0000 7fc22b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c0d2090 con 0x7fc22c0b4770 2024-04-26T23:08:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.858+0000 7fc22b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc22c162170 con 0x7fc22c161850 2024-04-26T23:08:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.858+0000 7fc229400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c0b63e0 0x7fc22c09fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.858+0000 7fc229400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c0b63e0 0x7fc22c09fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44850/0 (socket says 172.21.15.40:44850) 2024-04-26T23:08:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.858+0000 7fc229400640 1 -- 172.21.15.40:0/1530903598 learned_addr learned my addr 172.21.15.40:0/1530903598 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:00.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.859+0000 7fc22a800640 1 --2- 172.21.15.40:0/1530903598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc22c161850 0x7fc22c0b9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.859+0000 7fc229400640 1 -- 172.21.15.40:0/1530903598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc22c161850 msgr2=0x7fc22c0b9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:00.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.859+0000 7fc229400640 1 --2- 172.21.15.40:0/1530903598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc22c161850 0x7fc22c0b9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.859+0000 7fc229400640 1 -- 172.21.15.40:0/1530903598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc22c0b4770 msgr2=0x7fc22c153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:00.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.859+0000 7fc229400640 1 --2- 172.21.15.40:0/1530903598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc22c0b4770 0x7fc22c153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.859+0000 7fc229400640 1 -- 172.21.15.40:0/1530903598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22c09ee00 con 0x7fc22c0b63e0 2024-04-26T23:08:00.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.859+0000 7fc229400640 1 --2- 172.21.15.40:0/1530903598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c0b63e0 0x7fc22c09fbc0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fc21800d7d0 tx=0x7fc21800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:00.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.859+0000 7fc222000640 1 -- 172.21.15.40:0/1530903598 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21800b6c0 con 0x7fc22c0b63e0 2024-04-26T23:08:00.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.859+0000 7fc222000640 1 -- 172.21.15.40:0/1530903598 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc218019070 con 0x7fc22c0b63e0 2024-04-26T23:08:00.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.859+0000 7fc22b200640 1 -- 172.21.15.40:0/1530903598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc22c09f0e0 con 0x7fc22c0b63e0 2024-04-26T23:08:00.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.859+0000 7fc222000640 1 -- 172.21.15.40:0/1530903598 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21800ba80 con 0x7fc22c0b63e0 2024-04-26T23:08:00.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.860+0000 7fc22b200640 1 -- 172.21.15.40:0/1530903598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc22c09f6a0 con 0x7fc22c0b63e0 2024-04-26T23:08:00.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.862+0000 7fc222000640 1 -- 172.21.15.40:0/1530903598 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc21800bc20 con 0x7fc22c0b63e0 2024-04-26T23:08:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.863+0000 7fc222000640 1 --2- 172.21.15.40:0/1530903598 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc20c06c8f0 0x7fc20c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.863+0000 7fc222000640 1 -- 172.21.15.40:0/1530903598 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fc21808c1b0 con 0x7fc22c0b63e0 2024-04-26T23:08:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.864+0000 7fc229e00640 1 --2- 172.21.15.40:0/1530903598 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc20c06c8f0 0x7fc20c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.865+0000 7fc233dc5740 1 Processor -- start 2024-04-26T23:08:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.865+0000 7fc229e00640 1 --2- 172.21.15.40:0/1530903598 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc20c06c8f0 0x7fc20c06edb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc21c000f60 tx=0x7fc21c0078f0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.865+0000 7fc233dc5740 1 -- start start 2024-04-26T23:08:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.866+0000 7fc233dc5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51cf7b0 0x558fe51d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc233dc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe51cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc233dc5740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51fa270 0x558fe51fa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc233dc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558fe5133600 con 0x558fe520be80 2024-04-26T23:08:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc233dc5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558fe50d0100 con 0x558fe51cf7b0 2024-04-26T23:08:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc233dc5740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558fe512f5c0 con 0x558fe51fa270 2024-04-26T23:08:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc229400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe51cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc229400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe51cfb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44864/0 (socket says 172.21.15.40:44864) 2024-04-26T23:08:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc229400640 1 -- 172.21.15.40:0/2656250271 learned_addr learned my addr 172.21.15.40:0/2656250271 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc229e00640 1 --2- 172.21.15.40:0/2656250271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51cf7b0 0x558fe51d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc22a800640 1 --2- 172.21.15.40:0/2656250271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51fa270 0x558fe51fa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc229400640 1 -- 172.21.15.40:0/2656250271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51fa270 msgr2=0x558fe51fa6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc229400640 1 --2- 172.21.15.40:0/2656250271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51fa270 0x558fe51fa6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc229400640 1 -- 172.21.15.40:0/2656250271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51cf7b0 msgr2=0x558fe51d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc229400640 1 --2- 172.21.15.40:0/2656250271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51cf7b0 0x558fe51d1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc229400640 1 -- 172.21.15.40:0/2656250271 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fe4e452d0 con 0x558fe520be80 2024-04-26T23:08:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.867+0000 7fc229e00640 1 --2- 172.21.15.40:0/2656250271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51cf7b0 0x558fe51d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.868+0000 7fc229400640 1 --2- 172.21.15.40:0/2656250271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe51cfb90 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fc218059ce0 tx=0x7fc21805a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2c7745bbe5c75c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.868+0000 7fc216a00640 1 -- 172.21.15.40:0/2656250271 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc218062930 con 0x558fe520be80 2024-04-26T23:08:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.868+0000 7fc216a00640 1 -- 172.21.15.40:0/2656250271 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc218058050 con 0x558fe520be80 2024-04-26T23:08:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.868+0000 7fc216a00640 1 -- 172.21.15.40:0/2656250271 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2180563c0 con 0x558fe520be80 2024-04-26T23:08:00.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.869+0000 7fc233dc5740 1 -- 172.21.15.40:0/2656250271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 msgr2=0x558fe51cfb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:00.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.869+0000 7fc233dc5740 1 --2- 172.21.15.40:0/2656250271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe51cfb90 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fc218059ce0 tx=0x7fc21805a590 comp rx=0 tx=0).stop 2024-04-26T23:08:00.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.869+0000 7fc233dc5740 1 -- 172.21.15.40:0/2656250271 shutdown_connections 2024-04-26T23:08:00.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.869+0000 7fc233dc5740 1 --2- 172.21.15.40:0/2656250271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51fa270 0x558fe51fa6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.869+0000 7fc233dc5740 1 --2- 172.21.15.40:0/2656250271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe51cfb90 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.869+0000 7fc233dc5740 1 --2- 172.21.15.40:0/2656250271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51cf7b0 0x558fe51d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.869+0000 7fc233dc5740 1 -- 172.21.15.40:0/2656250271 >> 172.21.15.40:0/2656250271 conn(0x558fe5163d10 msgr2=0x558fe5215270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.869+0000 7fc233dc5740 1 -- 172.21.15.40:0/2656250271 shutdown_connections 2024-04-26T23:08:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.869+0000 7fc233dc5740 1 -- 172.21.15.40:0/2656250271 wait complete. 2024-04-26T23:08:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.870+0000 7fc233dc5740 1 -- start start 2024-04-26T23:08:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.870+0000 7fc233dc5740 1 Processor -- start 2024-04-26T23:08:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.872+0000 7fc233dc5740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51cf7b0 0x558fe51fb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.872+0000 7fc233dc5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 0x558fe51f53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.872+0000 7fc233dc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe51fcca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.872+0000 7fc233dc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558fe50baf70 con 0x558fe520be80 2024-04-26T23:08:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.872+0000 7fc233dc5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558fe51f7dc0 con 0x558fe51fa270 2024-04-26T23:08:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.872+0000 7fc233dc5740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558fe51f5930 con 0x558fe51cf7b0 2024-04-26T23:08:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.872+0000 7fc233dc5740 10 client.0 _ll_register_callbacks cb 0x558fe5129480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.872+0000 7fc22a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe51fcca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.872+0000 7fc22a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe51fcca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44876/0 (socket says 172.21.15.40:44876) 2024-04-26T23:08:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.872+0000 7fc22a800640 1 -- 172.21.15.40:0/497614989 learned_addr learned my addr 172.21.15.40:0/497614989 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.872+0000 7fc229e00640 1 --2- 172.21.15.40:0/497614989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51cf7b0 0x558fe51fb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.873+0000 7fc22a800640 1 -- 172.21.15.40:0/497614989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51cf7b0 msgr2=0x558fe51fb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.873+0000 7fc22a800640 1 --2- 172.21.15.40:0/497614989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51cf7b0 0x558fe51fb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.873+0000 7fc22a800640 1 -- 172.21.15.40:0/497614989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 msgr2=0x558fe51f53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.873+0000 7fc229400640 1 --2- 172.21.15.40:0/497614989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 0x558fe51f53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.873+0000 7fc22a800640 1 --2- 172.21.15.40:0/497614989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 0x558fe51f53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.873+0000 7fc22a800640 1 -- 172.21.15.40:0/497614989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fe4f0f6b0 con 0x558fe520be80 2024-04-26T23:08:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.873+0000 7fc22a800640 1 --2- 172.21.15.40:0/497614989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe51fcca0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fc224009d90 tx=0x7fc224004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c9726f63cb2c30 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.873+0000 7fc20b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.873+0000 7fc20b400640 1 -- 172.21.15.40:0/497614989 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224009450 con 0x558fe520be80 2024-04-26T23:08:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.873+0000 7fc20b400640 1 -- 172.21.15.40:0/497614989 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fc224031400 con 0x558fe520be80 2024-04-26T23:08:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.873+0000 7fc233dc5740 1 -- 172.21.15.40:0/497614989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc218058050 con 0x558fe520be80 2024-04-26T23:08:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.874+0000 7fc20b400640 1 -- 172.21.15.40:0/497614989 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fc22404e050 con 0x558fe520be80 2024-04-26T23:08:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.874+0000 7fc233dc5740 10 client.8220 fetch_fsmap learned FSMap version 96 2024-04-26T23:08:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.874+0000 7fc233dc5740 1 -- 172.21.15.40:0/497614989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) -- 0x558fe520e0a0 con 0x558fe520be80 2024-04-26T23:08:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.874+0000 7fc20b400640 1 -- 172.21.15.40:0/497614989 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc218058050 con 0x558fe520be80 2024-04-26T23:08:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.874+0000 7fc20b400640 1 -- 172.21.15.40:0/497614989 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 96) ==== 3534+0+0 (secure 0 0 0) 0x7fc224051020 con 0x558fe520be80 2024-04-26T23:08:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.874+0000 7fc233dc5740 10 client.8220 fetch_fsmap finished waiting for FSMap version 96 2024-04-26T23:08:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.874+0000 7fc233dc5740 10 client.8220 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-26T23:08:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.874+0000 7fc233dc5740 20 client.8220 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.874+0000 7fc233dc5740 1 --2- 172.21.15.40:0/497614989 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x558fe522e3a0 0x558fe52326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.874+0000 7fc233dc5740 4 client.8220 mds_command: new command op to 6253 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.874+0000 7fc233dc5740 1 -- 172.21.15.40:0/497614989 --> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fe5141210 con 0x558fe522e3a0 2024-04-26T23:08:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.875+0000 7fc229e00640 1 --2- 172.21.15.40:0/497614989 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x558fe522e3a0 0x558fe52326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.876+0000 7fc229e00640 1 --2- 172.21.15.40:0/497614989 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x558fe522e3a0 0x558fe52326a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e75c1fe704f9f8d server_cookie=33454e24c9063d96 in_seq=0 out_seq=0 2024-04-26T23:08:00.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.876+0000 7fc20b400640 10 client.8220 ms_handle_connect on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:00.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc20b400640 1 -- 172.21.15.40:0/497614989 <== mds.2 v2:172.21.15.103:6836/2728939257 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558fe5141210 con 0x558fe522e3a0 2024-04-26T23:08:00.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc20b400640 10 client.8220 handle_command_reply: tid=0 2024-04-26T23:08:00.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc233dc5740 1 client.8220 shutdown 2024-04-26T23:08:00.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc233dc5740 1 -- 172.21.15.40:0/497614989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 msgr2=0x558fe51fcca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:00.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc233dc5740 1 --2- 172.21.15.40:0/497614989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe51fcca0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fc224009d90 tx=0x7fc224004ab0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc233dc5740 1 -- 172.21.15.40:0/497614989 shutdown_connections 2024-04-26T23:08:00.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc233dc5740 1 --2- 172.21.15.40:0/497614989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe51fcca0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc233dc5740 1 --2- 172.21.15.40:0/497614989 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x558fe522e3a0 0x558fe52326a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc233dc5740 1 --2- 172.21.15.40:0/497614989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 0x558fe51f53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc233dc5740 1 --2- 172.21.15.40:0/497614989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51cf7b0 0x558fe51fb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc233dc5740 1 -- 172.21.15.40:0/497614989 >> 172.21.15.40:0/497614989 conn(0x558fe5163d10 msgr2=0x558fe51f81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:00.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc20b400640 0 client.8220 ms_handle_reset on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:00.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc233dc5740 1 -- 172.21.15.40:0/497614989 shutdown_connections 2024-04-26T23:08:00.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc233dc5740 1 -- 172.21.15.40:0/497614989 wait complete. 2024-04-26T23:08:00.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.878+0000 7fc233dc5740 20 client.8220 trim_cache size 0 max 16384 2024-04-26T23:08:00.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.898+0000 7fc233dc5740 1 Processor -- start 2024-04-26T23:08:00.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.898+0000 7fc233dc5740 1 -- start start 2024-04-26T23:08:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.899+0000 7fc233dc5740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51cf7b0 0x558fe51fbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.899+0000 7fc233dc5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 0x558fe51fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.899+0000 7fc233dc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe5226c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.899+0000 7fc233dc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c200d80 con 0x558fe520be80 2024-04-26T23:08:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.899+0000 7fc233dc5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c201450 con 0x558fe51fa270 2024-04-26T23:08:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.899+0000 7fc233dc5740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc22c0cd840 con 0x558fe51cf7b0 2024-04-26T23:08:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.900+0000 7fc22a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe5226c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.900+0000 7fc22a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe5226c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44888/0 (socket says 172.21.15.40:44888) 2024-04-26T23:08:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.900+0000 7fc22a800640 1 -- 172.21.15.40:0/2774495220 learned_addr learned my addr 172.21.15.40:0/2774495220 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.900+0000 7fc22a800640 1 -- 172.21.15.40:0/2774495220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51cf7b0 msgr2=0x558fe51fbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.900+0000 7fc229400640 1 --2- 172.21.15.40:0/2774495220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 0x558fe51fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.900+0000 7fc229e00640 1 --2- 172.21.15.40:0/2774495220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51cf7b0 0x558fe51fbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.900+0000 7fc22a800640 1 --2- 172.21.15.40:0/2774495220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51cf7b0 0x558fe51fbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.900+0000 7fc22a800640 1 -- 172.21.15.40:0/2774495220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 msgr2=0x558fe51fc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.900+0000 7fc22a800640 1 --2- 172.21.15.40:0/2774495220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 0x558fe51fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.900+0000 7fc22a800640 1 -- 172.21.15.40:0/2774495220 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fe51fd010 con 0x558fe520be80 2024-04-26T23:08:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.900+0000 7fc22a800640 1 --2- 172.21.15.40:0/2774495220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe5226c30 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fc224007130 tx=0x7fc22403e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57222d262cfd209b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:00.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.901+0000 7fc216000640 1 -- 172.21.15.40:0/2774495220 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2240316d0 con 0x558fe520be80 2024-04-26T23:08:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.901+0000 7fc216000640 1 -- 172.21.15.40:0/2774495220 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2240319d0 con 0x558fe520be80 2024-04-26T23:08:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.901+0000 7fc216000640 1 -- 172.21.15.40:0/2774495220 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224031dc0 con 0x558fe520be80 2024-04-26T23:08:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.901+0000 7fc233dc5740 1 -- 172.21.15.40:0/2774495220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 msgr2=0x558fe5226c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.901+0000 7fc233dc5740 1 --2- 172.21.15.40:0/2774495220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe5226c30 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fc224007130 tx=0x7fc22403e040 comp rx=0 tx=0).stop 2024-04-26T23:08:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.901+0000 7fc233dc5740 1 -- 172.21.15.40:0/2774495220 shutdown_connections 2024-04-26T23:08:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.901+0000 7fc233dc5740 1 --2- 172.21.15.40:0/2774495220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe520be80 0x558fe5226c30 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.901+0000 7fc233dc5740 1 --2- 172.21.15.40:0/2774495220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 0x558fe51fc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.901+0000 7fc233dc5740 1 --2- 172.21.15.40:0/2774495220 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe51cf7b0 0x558fe51fbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.901+0000 7fc233dc5740 1 -- 172.21.15.40:0/2774495220 >> 172.21.15.40:0/2774495220 conn(0x558fe5163d10 msgr2=0x558fe51c5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.901+0000 7fc233dc5740 1 -- 172.21.15.40:0/2774495220 shutdown_connections 2024-04-26T23:08:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.901+0000 7fc233dc5740 1 -- 172.21.15.40:0/2774495220 wait complete. 2024-04-26T23:08:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.902+0000 7fc233dc5740 1 -- start start 2024-04-26T23:08:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.902+0000 7fc233dc5740 1 Processor -- start 2024-04-26T23:08:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.903+0000 7fc233dc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe51cf7b0 0x558fe51fbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.903+0000 7fc233dc5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 0x558fe51fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.903+0000 7fc233dc5740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe520be80 0x558fe520ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.903+0000 7fc233dc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558fe522e220 con 0x558fe51cf7b0 2024-04-26T23:08:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.903+0000 7fc233dc5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558fe51f5d60 con 0x558fe51fa270 2024-04-26T23:08:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.903+0000 7fc233dc5740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558fe522e840 con 0x558fe520be80 2024-04-26T23:08:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.903+0000 7fc233dc5740 10 client.0 _ll_register_callbacks cb 0x7fc2100036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.903+0000 7fc229e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe51cf7b0 0x558fe51fbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.904+0000 7fc229e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe51cf7b0 0x558fe51fbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44894/0 (socket says 172.21.15.40:44894) 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.904+0000 7fc229e00640 1 -- 172.21.15.40:0/269442532 learned_addr learned my addr 172.21.15.40:0/269442532 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.904+0000 7fc229e00640 1 -- 172.21.15.40:0/269442532 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe520be80 msgr2=0x558fe520ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.904+0000 7fc229e00640 1 --2- 172.21.15.40:0/269442532 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe520be80 0x558fe520ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.904+0000 7fc229e00640 1 -- 172.21.15.40:0/269442532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 msgr2=0x558fe51fc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.904+0000 7fc229400640 1 --2- 172.21.15.40:0/269442532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 0x558fe51fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.904+0000 7fc229e00640 1 --2- 172.21.15.40:0/269442532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 0x558fe51fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.904+0000 7fc229e00640 1 -- 172.21.15.40:0/269442532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fe520bca0 con 0x558fe51cf7b0 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.904+0000 7fc229e00640 1 --2- 172.21.15.40:0/269442532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe51cf7b0 0x558fe51fbf90 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fc21c001d70 tx=0x7fc21c001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6573ddd79e39677 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.904+0000 7fc20b400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.904+0000 7fc20b400640 1 -- 172.21.15.40:0/269442532 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21c023660 con 0x558fe51cf7b0 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.904+0000 7fc20b400640 1 -- 172.21.15.40:0/269442532 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fc21c023800 con 0x558fe51cf7b0 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.905+0000 7fc233dc5740 1 -- 172.21.15.40:0/269442532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2240319d0 con 0x558fe51cf7b0 2024-04-26T23:08:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.905+0000 7fc20b400640 1 -- 172.21.15.40:0/269442532 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fc21c031050 con 0x558fe51cf7b0 2024-04-26T23:08:00.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.905+0000 7fc233dc5740 10 client.8226 fetch_fsmap learned FSMap version 96 2024-04-26T23:08:00.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.905+0000 7fc233dc5740 1 -- 172.21.15.40:0/269442532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) -- 0x558fe527fe70 con 0x558fe51cf7b0 2024-04-26T23:08:00.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.905+0000 7fc20b400640 1 -- 172.21.15.40:0/269442532 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2240319d0 con 0x558fe51cf7b0 2024-04-26T23:08:00.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.905+0000 7fc20b400640 1 -- 172.21.15.40:0/269442532 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 96) ==== 3534+0+0 (secure 0 0 0) 0x7fc21c034020 con 0x558fe51cf7b0 2024-04-26T23:08:00.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.905+0000 7fc233dc5740 10 client.8226 fetch_fsmap finished waiting for FSMap version 96 2024-04-26T23:08:00.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.905+0000 7fc233dc5740 10 client.8226 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-26T23:08:00.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.905+0000 7fc233dc5740 20 client.8226 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:00.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.905+0000 7fc233dc5740 1 --2- 172.21.15.40:0/269442532 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x558fe52366c0 0x558fe5236aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:00.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.905+0000 7fc233dc5740 4 client.8226 mds_command: new command op to 6253 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:08:00.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.905+0000 7fc233dc5740 1 -- 172.21.15.40:0/269442532 --> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] -- command(tid 0: {"prefix": "status"}) -- 0x558fe513b080 con 0x558fe52366c0 2024-04-26T23:08:00.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.906+0000 7fc229400640 1 --2- 172.21.15.40:0/269442532 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x558fe52366c0 0x558fe5236aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:00.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.907+0000 7fc229400640 1 --2- 172.21.15.40:0/269442532 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x558fe52366c0 0x558fe5236aa0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8fb7f979afd9a895 server_cookie=ca28531e1233c086 in_seq=0 out_seq=0 2024-04-26T23:08:00.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.907+0000 7fc20b400640 10 client.8226 ms_handle_connect on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc20b400640 1 -- 172.21.15.40:0/269442532 <== mds.2 v2:172.21.15.103:6836/2728939257 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x558fe513b080 con 0x558fe52366c0 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc20b400640 10 client.8226 handle_command_reply: tid=0 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc233dc5740 1 client.8226 shutdown 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc233dc5740 1 -- 172.21.15.40:0/269442532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe51cf7b0 msgr2=0x558fe51fbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc233dc5740 1 --2- 172.21.15.40:0/269442532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe51cf7b0 0x558fe51fbf90 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fc21c001d70 tx=0x7fc21c001150 comp rx=0 tx=0).stop 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc233dc5740 1 -- 172.21.15.40:0/269442532 shutdown_connections 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc233dc5740 1 --2- 172.21.15.40:0/269442532 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558fe520be80 0x558fe520ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc233dc5740 1 --2- 172.21.15.40:0/269442532 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x558fe52366c0 0x558fe5236aa0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc233dc5740 1 --2- 172.21.15.40:0/269442532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558fe51fa270 0x558fe51fc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc233dc5740 1 --2- 172.21.15.40:0/269442532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558fe51cf7b0 0x558fe51fbf90 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc20b400640 0 client.8226 ms_handle_reset on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc233dc5740 1 -- 172.21.15.40:0/269442532 >> 172.21.15.40:0/269442532 conn(0x558fe5163d10 msgr2=0x558fe5237b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc233dc5740 1 -- 172.21.15.40:0/269442532 shutdown_connections 2024-04-26T23:08:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc233dc5740 1 -- 172.21.15.40:0/269442532 wait complete. 2024-04-26T23:08:00.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.908+0000 7fc233dc5740 20 client.8226 trim_cache size 0 max 16384 2024-04-26T23:08:00.911 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:00.911 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:08:00.911 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-04-26T23:08:00.911 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6253, 2024-04-26T23:08:00.911 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:08:00.911 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:08:00.911 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:08:00.911 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 2.0252562209999998, 2024-04-26T23:08:00.911 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 96, 2024-04-26T23:08:00.911 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 90, 2024-04-26T23:08:00.912 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-26T23:08:00.912 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 17.48064325 2024-04-26T23:08:00.912 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.909+0000 7fc22b200640 1 -- 172.21.15.40:0/1530903598 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc20c06c8f0 msgr2=0x7fc20c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.909+0000 7fc22b200640 1 --2- 172.21.15.40:0/1530903598 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc20c06c8f0 0x7fc20c06edb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc21c000f60 tx=0x7fc21c0078f0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.909+0000 7fc22b200640 1 -- 172.21.15.40:0/1530903598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c0b63e0 msgr2=0x7fc22c09fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.909+0000 7fc22b200640 1 --2- 172.21.15.40:0/1530903598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c0b63e0 0x7fc22c09fbc0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fc21800d7d0 tx=0x7fc21800dc90 comp rx=0 tx=0).stop 2024-04-26T23:08:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.909+0000 7fc22b200640 1 -- 172.21.15.40:0/1530903598 shutdown_connections 2024-04-26T23:08:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.909+0000 7fc22b200640 1 --2- 172.21.15.40:0/1530903598 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc20c06c8f0 0x7fc20c06edb0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.909+0000 7fc22b200640 1 --2- 172.21.15.40:0/1530903598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc22c161850 0x7fc22c0b9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.909+0000 7fc22b200640 1 --2- 172.21.15.40:0/1530903598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c0b63e0 0x7fc22c09fbc0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.909+0000 7fc22b200640 1 --2- 172.21.15.40:0/1530903598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc22c0b4770 0x7fc22c153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.909+0000 7fc22b200640 1 -- 172.21.15.40:0/1530903598 >> 172.21.15.40:0/1530903598 conn(0x7fc22c0d4e10 msgr2=0x7fc22c0b67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.909+0000 7fc22b200640 1 -- 172.21.15.40:0/1530903598 shutdown_connections 2024-04-26T23:08:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:00.909+0000 7fc22b200640 1 -- 172.21.15.40:0/1530903598 wait complete. 2024-04-26T23:08:00.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T23:08:01.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.113+0000 7f9af3200640 1 Processor -- start 2024-04-26T23:08:01.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.114+0000 7f9af3200640 1 -- start start 2024-04-26T23:08:01.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.115+0000 7f9af3200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9af40d32e0 0x7f9af40d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.115+0000 7f9af3200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9af4162c40 con 0x7f9af40b8d20 2024-04-26T23:08:01.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.115+0000 7f9af3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9af40d2d80 con 0x7f9af40b6220 2024-04-26T23:08:01.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.115+0000 7f9af3200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f9af40d3c80 con 0x7f9af40d32e0 2024-04-26T23:08:01.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.115+0000 7f9af1400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9af40d32e0 0x7f9af40d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.115+0000 7f9af2800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9af40b8d20 0x7f9af40b9160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42016/0 (socket says 172.21.15.40:42016) 2024-04-26T23:08:01.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.115+0000 7f9af2800640 1 -- 172.21.15.40:0/3479709985 learned_addr learned my addr 172.21.15.40:0/3479709985 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:01.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.116+0000 7f9af1400640 1 -- 172.21.15.40:0/3479709985 >> v1:172.21.15.103:6789/0 conn(0x7f9af40b6220 legacy=0x7f9af40d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.116+0000 7f9af1400640 1 -- 172.21.15.40:0/3479709985 >> v1:172.21.15.40:6789/0 conn(0x7f9af40b8d20 legacy=0x7f9af40b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.116+0000 7f9af1400640 1 -- 172.21.15.40:0/3479709985 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af40b7c80 con 0x7f9af40d32e0 2024-04-26T23:08:01.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.117+0000 7f9af1400640 1 --2- 172.21.15.40:0/3479709985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d32e0 0x7f9af40d3740 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009d90 tx=0x7f9adc02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9079d6539be70e60 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.117+0000 7f9af0a00640 1 -- 172.21.15.40:0/3479709985 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc007740 con 0x7f9af40d32e0 2024-04-26T23:08:01.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.117+0000 7f9af0a00640 1 -- 172.21.15.40:0/3479709985 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9adc0078e0 con 0x7f9af40d32e0 2024-04-26T23:08:01.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.117+0000 7f9af0a00640 1 -- 172.21.15.40:0/3479709985 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc007cb0 con 0x7f9af40d32e0 2024-04-26T23:08:01.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.118+0000 7f9af3200640 1 -- 172.21.15.40:0/3479709985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d32e0 msgr2=0x7f9af40d3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.118+0000 7f9af3200640 1 --2- 172.21.15.40:0/3479709985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d32e0 0x7f9af40d3740 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009d90 tx=0x7f9adc02a6f0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.118+0000 7f9af3200640 1 -- 172.21.15.40:0/3479709985 shutdown_connections 2024-04-26T23:08:01.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.118+0000 7f9af3200640 1 --2- 172.21.15.40:0/3479709985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d32e0 0x7f9af40d3740 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.118+0000 7f9af3200640 1 -- 172.21.15.40:0/3479709985 >> 172.21.15.40:0/3479709985 conn(0x7f9af40d4d90 msgr2=0x7f9af40d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.118+0000 7f9af3200640 1 -- 172.21.15.40:0/3479709985 shutdown_connections 2024-04-26T23:08:01.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.118+0000 7f9af3200640 1 -- 172.21.15.40:0/3479709985 wait complete. 2024-04-26T23:08:01.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.120+0000 7f9af3200640 1 Processor -- start 2024-04-26T23:08:01.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.120+0000 7f9af3200640 1 -- start start 2024-04-26T23:08:01.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.121+0000 7f9af3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9af40b6220 0x7f9af40b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.121+0000 7f9af3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af40b8d20 0x7f9af4160190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.121+0000 7f9af3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d32e0 0x7f9af40b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.121+0000 7f9af3200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9af40b8520 con 0x7f9af40d32e0 2024-04-26T23:08:01.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.121+0000 7f9af3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9af40b8240 con 0x7f9af40b8d20 2024-04-26T23:08:01.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.121+0000 7f9af3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9af40d0aa0 con 0x7f9af40b6220 2024-04-26T23:08:01.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.121+0000 7f9af2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d32e0 0x7f9af40b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.121+0000 7f9af2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d32e0 0x7f9af40b44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44920/0 (socket says 172.21.15.40:44920) 2024-04-26T23:08:01.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.121+0000 7f9af2800640 1 -- 172.21.15.40:0/583072059 learned_addr learned my addr 172.21.15.40:0/583072059 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:01.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9af1400640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af40b8d20 0x7f9af4160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9af1e00640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9af40b6220 0x7f9af40b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9af2800640 1 -- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9af40b6220 msgr2=0x7f9af40b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9af2800640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9af40b6220 0x7f9af40b1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9af2800640 1 -- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af40b8d20 msgr2=0x7f9af4160190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9af2800640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af40b8d20 0x7f9af4160190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9af2800640 1 -- 172.21.15.40:0/583072059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af40b2580 con 0x7f9af40d32e0 2024-04-26T23:08:01.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9af1e00640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9af40b6220 0x7f9af40b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:01.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9af1400640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af40b8d20 0x7f9af4160190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:01.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9af2800640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d32e0 0x7f9af40b44e0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00c450 tx=0x7f9aec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9aea000640 1 -- 172.21.15.40:0/583072059 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec017070 con 0x7f9af40d32e0 2024-04-26T23:08:01.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9aea000640 1 -- 172.21.15.40:0/583072059 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aec005ce0 con 0x7f9af40d32e0 2024-04-26T23:08:01.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9aea000640 1 -- 172.21.15.40:0/583072059 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec007270 con 0x7f9af40d32e0 2024-04-26T23:08:01.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.122+0000 7f9af3200640 1 -- 172.21.15.40:0/583072059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9af40b2860 con 0x7f9af40d32e0 2024-04-26T23:08:01.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.123+0000 7f9af3200640 1 -- 172.21.15.40:0/583072059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9af40b2e20 con 0x7f9af40d32e0 2024-04-26T23:08:01.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.126+0000 7f9aea000640 1 -- 172.21.15.40:0/583072059 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9aec01a070 con 0x7f9af40d32e0 2024-04-26T23:08:01.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.127+0000 7f9aea000640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9ad006c8f0 0x7f9ad006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.127+0000 7f9afbff0740 1 Processor -- start 2024-04-26T23:08:01.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.127+0000 7f9afbff0740 1 -- start start 2024-04-26T23:08:01.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.127+0000 7f9aea000640 1 -- 172.21.15.40:0/583072059 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f9aec08aff0 con 0x7f9af40d32e0 2024-04-26T23:08:01.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.128+0000 7f9af1e00640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9ad006c8f0 0x7f9ad006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.128+0000 7f9afbff0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890ed07b0 0x55c890ed2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.128+0000 7f9afbff0740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890f0ce80 0x55c890ed0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.128+0000 7f9afbff0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890efb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.128+0000 7f9afbff0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c890e34600 con 0x55c890efb270 2024-04-26T23:08:01.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.128+0000 7f9afbff0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c890dd1100 con 0x55c890ed07b0 2024-04-26T23:08:01.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.128+0000 7f9afbff0740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c890e305c0 con 0x55c890f0ce80 2024-04-26T23:08:01.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.129+0000 7f9af2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890efb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.129+0000 7f9af2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890efb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44922/0 (socket says 172.21.15.40:44922) 2024-04-26T23:08:01.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.129+0000 7f9af2800640 1 -- 172.21.15.40:0/3335138511 learned_addr learned my addr 172.21.15.40:0/3335138511 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:01.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.129+0000 7f9af2800640 1 -- 172.21.15.40:0/3335138511 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890f0ce80 msgr2=0x55c890ed0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.129+0000 7f9af2800640 1 --2- 172.21.15.40:0/3335138511 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890f0ce80 0x55c890ed0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.129+0000 7f9af2800640 1 -- 172.21.15.40:0/3335138511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890ed07b0 msgr2=0x55c890ed2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.129+0000 7f9af2800640 1 --2- 172.21.15.40:0/3335138511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890ed07b0 0x55c890ed2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.129+0000 7f9af2800640 1 -- 172.21.15.40:0/3335138511 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c890b462d0 con 0x55c890efb270 2024-04-26T23:08:01.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.129+0000 7f9af2800640 1 --2- 172.21.15.40:0/3335138511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890efb6f0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f9aec058ac0 tx=0x7f9aec058c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36ad893e9b9d08b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.129+0000 7f9af1e00640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9ad006c8f0 0x7f9ad006edb0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4000f60 tx=0x7f9ae4001b60 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.129+0000 7f9ae2a00640 1 -- 172.21.15.40:0/3335138511 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec056e30 con 0x55c890efb270 2024-04-26T23:08:01.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.129+0000 7f9ae2a00640 1 -- 172.21.15.40:0/3335138511 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aec0551a0 con 0x55c890efb270 2024-04-26T23:08:01.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.129+0000 7f9ae2a00640 1 -- 172.21.15.40:0/3335138511 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec061260 con 0x55c890efb270 2024-04-26T23:08:01.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.130+0000 7f9afbff0740 1 -- 172.21.15.40:0/3335138511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 msgr2=0x55c890efb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.130+0000 7f9afbff0740 1 --2- 172.21.15.40:0/3335138511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890efb6f0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f9aec058ac0 tx=0x7f9aec058c70 comp rx=0 tx=0).stop 2024-04-26T23:08:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.130+0000 7f9afbff0740 1 -- 172.21.15.40:0/3335138511 shutdown_connections 2024-04-26T23:08:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.130+0000 7f9afbff0740 1 --2- 172.21.15.40:0/3335138511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890efb6f0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.130+0000 7f9afbff0740 1 --2- 172.21.15.40:0/3335138511 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890f0ce80 0x55c890ed0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.130+0000 7f9afbff0740 1 --2- 172.21.15.40:0/3335138511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890ed07b0 0x55c890ed2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.130+0000 7f9afbff0740 1 -- 172.21.15.40:0/3335138511 >> 172.21.15.40:0/3335138511 conn(0x55c890e64d10 msgr2=0x55c890f16270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.130+0000 7f9afbff0740 1 -- 172.21.15.40:0/3335138511 shutdown_connections 2024-04-26T23:08:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.131+0000 7f9afbff0740 1 -- 172.21.15.40:0/3335138511 wait complete. 2024-04-26T23:08:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.131+0000 7f9afbff0740 1 -- start start 2024-04-26T23:08:01.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.131+0000 7f9afbff0740 1 Processor -- start 2024-04-26T23:08:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.132+0000 7f9afbff0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890ed07b0 0x55c890f2a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.132+0000 7f9afbff0740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890efb270 0x55c890f264a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.132+0000 7f9afbff0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 0x55c890ef2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.132+0000 7f9afbff0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c890dbbf70 con 0x55c890ed07b0 2024-04-26T23:08:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.132+0000 7f9afbff0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c890ef8dc0 con 0x55c890f0ce80 2024-04-26T23:08:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.132+0000 7f9afbff0740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c890ef3600 con 0x55c890efb270 2024-04-26T23:08:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.133+0000 7f9afbff0740 10 client.0 _ll_register_callbacks cb 0x55c890e2a480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.133+0000 7f9af1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890ed07b0 0x55c890f2a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.133+0000 7f9af1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890ed07b0 0x55c890f2a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44938/0 (socket says 172.21.15.40:44938) 2024-04-26T23:08:01.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.133+0000 7f9af1e00640 1 -- 172.21.15.40:0/2366429276 learned_addr learned my addr 172.21.15.40:0/2366429276 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.133+0000 7f9af1400640 1 --2- 172.21.15.40:0/2366429276 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890efb270 0x55c890f264a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.133+0000 7f9af1e00640 1 -- 172.21.15.40:0/2366429276 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890efb270 msgr2=0x55c890f264a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.133+0000 7f9af1e00640 1 --2- 172.21.15.40:0/2366429276 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890efb270 0x55c890f264a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.133+0000 7f9af1e00640 1 -- 172.21.15.40:0/2366429276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 msgr2=0x55c890ef2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.133+0000 7f9af2800640 1 --2- 172.21.15.40:0/2366429276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 0x55c890ef2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.133+0000 7f9af1e00640 1 --2- 172.21.15.40:0/2366429276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 0x55c890ef2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.133+0000 7f9af1e00640 1 -- 172.21.15.40:0/2366429276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c890c106b0 con 0x55c890ed07b0 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.133+0000 7f9af1e00640 1 --2- 172.21.15.40:0/2366429276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890ed07b0 0x55c890f2a010 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400def0 tx=0x7f9ae4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ebba3ea4758a6a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.133+0000 7f9adb400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9adb400640 1 -- 172.21.15.40:0/2366429276 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4026060 con 0x55c890ed07b0 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9adb400640 1 -- 172.21.15.40:0/2366429276 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f9ae4016610 con 0x55c890ed07b0 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9afbff0740 1 -- 172.21.15.40:0/2366429276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9aec0551a0 con 0x55c890ed07b0 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9adb400640 1 -- 172.21.15.40:0/2366429276 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f9ae4019e50 con 0x55c890ed07b0 2024-04-26T23:08:01.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9afbff0740 10 client.8241 fetch_fsmap learned FSMap version 96 2024-04-26T23:08:01.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9afbff0740 1 -- 172.21.15.40:0/2366429276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) -- 0x55c890ef3340 con 0x55c890ed07b0 2024-04-26T23:08:01.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9adb400640 1 -- 172.21.15.40:0/2366429276 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aec0551a0 con 0x55c890ed07b0 2024-04-26T23:08:01.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9adb400640 1 -- 172.21.15.40:0/2366429276 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 96) ==== 3534+0+0 (secure 0 0 0) 0x7f9ae402f080 con 0x55c890ed07b0 2024-04-26T23:08:01.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9afbff0740 10 client.8241 fetch_fsmap finished waiting for FSMap version 96 2024-04-26T23:08:01.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9afbff0740 10 client.8241 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:08:01.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9afbff0740 20 client.8241 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:01.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9afbff0740 1 --2- 172.21.15.40:0/2366429276 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55c890efd780 0x55c890efdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9afbff0740 4 client.8241 mds_command: new command op to 8031 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:01.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.134+0000 7f9afbff0740 1 -- 172.21.15.40:0/2366429276 --> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c890e42210 con 0x55c890efd780 2024-04-26T23:08:01.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.135+0000 7f9af1400640 1 --2- 172.21.15.40:0/2366429276 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55c890efd780 0x55c890efdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.135+0000 7f9af1400640 1 --2- 172.21.15.40:0/2366429276 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55c890efd780 0x55c890efdb60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=58a4d5c7790c43b1 server_cookie=439f431b0b6182c4 in_seq=0 out_seq=0 2024-04-26T23:08:01.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.135+0000 7f9adb400640 10 client.8241 ms_handle_connect on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:01.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.136+0000 7f9adb400640 1 -- 172.21.15.40:0/2366429276 <== mds.1 v2:172.21.15.40:6832/1811310743 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c890e42210 con 0x55c890efd780 2024-04-26T23:08:01.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.136+0000 7f9adb400640 10 client.8241 handle_command_reply: tid=0 2024-04-26T23:08:01.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.136+0000 7f9afbff0740 1 client.8241 shutdown 2024-04-26T23:08:01.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.137+0000 7f9afbff0740 1 -- 172.21.15.40:0/2366429276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890ed07b0 msgr2=0x55c890f2a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.137+0000 7f9afbff0740 1 --2- 172.21.15.40:0/2366429276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890ed07b0 0x55c890f2a010 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400def0 tx=0x7f9ae4001120 comp rx=0 tx=0).stop 2024-04-26T23:08:01.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.137+0000 7f9afbff0740 1 -- 172.21.15.40:0/2366429276 shutdown_connections 2024-04-26T23:08:01.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.137+0000 7f9afbff0740 1 --2- 172.21.15.40:0/2366429276 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55c890efd780 0x55c890efdb60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.137+0000 7f9afbff0740 1 --2- 172.21.15.40:0/2366429276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 0x55c890ef2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.137+0000 7f9afbff0740 1 --2- 172.21.15.40:0/2366429276 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890efb270 0x55c890f264a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.137+0000 7f9afbff0740 1 --2- 172.21.15.40:0/2366429276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890ed07b0 0x55c890f2a010 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.137+0000 7f9adb400640 0 client.8241 ms_handle_reset on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:01.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.137+0000 7f9afbff0740 1 -- 172.21.15.40:0/2366429276 >> 172.21.15.40:0/2366429276 conn(0x55c890e64d10 msgr2=0x55c890ef6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.137+0000 7f9afbff0740 1 -- 172.21.15.40:0/2366429276 shutdown_connections 2024-04-26T23:08:01.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.137+0000 7f9afbff0740 1 -- 172.21.15.40:0/2366429276 wait complete. 2024-04-26T23:08:01.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.137+0000 7f9afbff0740 20 client.8241 trim_cache size 0 max 16384 2024-04-26T23:08:01.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.157+0000 7f9afbff0740 1 Processor -- start 2024-04-26T23:08:01.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.157+0000 7f9afbff0740 1 -- start start 2024-04-26T23:08:01.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.158+0000 7f9afbff0740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890ed07b0 0x55c890ef4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.158+0000 7f9afbff0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890ef5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.158+0000 7f9afbff0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 0x55c890ecef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.158+0000 7f9afbff0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9af4201090 con 0x55c890efb270 2024-04-26T23:08:01.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.158+0000 7f9afbff0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9af4201760 con 0x55c890f0ce80 2024-04-26T23:08:01.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.158+0000 7f9afbff0740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9af40a1a30 con 0x55c890ed07b0 2024-04-26T23:08:01.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.158+0000 7f9af1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890ef5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.158+0000 7f9af1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890ef5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44952/0 (socket says 172.21.15.40:44952) 2024-04-26T23:08:01.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.158+0000 7f9af1400640 1 -- 172.21.15.40:0/2217338222 learned_addr learned my addr 172.21.15.40:0/2217338222 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.159+0000 7f9af1400640 1 -- 172.21.15.40:0/2217338222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890ed07b0 msgr2=0x55c890ef4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.159+0000 7f9af1400640 1 --2- 172.21.15.40:0/2217338222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890ed07b0 0x55c890ef4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.159+0000 7f9af1400640 1 -- 172.21.15.40:0/2217338222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 msgr2=0x55c890ecef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.159+0000 7f9af1400640 1 --2- 172.21.15.40:0/2217338222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 0x55c890ecef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.159+0000 7f9af1400640 1 -- 172.21.15.40:0/2217338222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c890ef8d30 con 0x55c890efb270 2024-04-26T23:08:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.159+0000 7f9af1400640 1 --2- 172.21.15.40:0/2217338222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890ef5420 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f9adc02ace0 tx=0x7f9adc02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97d10dd7dcaf392 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.159+0000 7f9ae2000640 1 -- 172.21.15.40:0/2217338222 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc045660 con 0x55c890efb270 2024-04-26T23:08:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.159+0000 7f9ae2000640 1 -- 172.21.15.40:0/2217338222 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9adc04b070 con 0x55c890efb270 2024-04-26T23:08:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.159+0000 7f9ae2000640 1 -- 172.21.15.40:0/2217338222 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc007690 con 0x55c890efb270 2024-04-26T23:08:01.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.160+0000 7f9afbff0740 1 -- 172.21.15.40:0/2217338222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 msgr2=0x55c890ef5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.160+0000 7f9afbff0740 1 --2- 172.21.15.40:0/2217338222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890ef5420 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f9adc02ace0 tx=0x7f9adc02e000 comp rx=0 tx=0).stop 2024-04-26T23:08:01.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.160+0000 7f9afbff0740 1 -- 172.21.15.40:0/2217338222 shutdown_connections 2024-04-26T23:08:01.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.160+0000 7f9afbff0740 1 --2- 172.21.15.40:0/2217338222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 0x55c890ecef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.160+0000 7f9afbff0740 1 --2- 172.21.15.40:0/2217338222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890ef5420 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.160+0000 7f9afbff0740 1 --2- 172.21.15.40:0/2217338222 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890ed07b0 0x55c890ef4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.160+0000 7f9afbff0740 1 -- 172.21.15.40:0/2217338222 >> 172.21.15.40:0/2217338222 conn(0x55c890e64d10 msgr2=0x55c890ecb290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.160+0000 7f9afbff0740 1 -- 172.21.15.40:0/2217338222 shutdown_connections 2024-04-26T23:08:01.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.160+0000 7f9afbff0740 1 -- 172.21.15.40:0/2217338222 wait complete. 2024-04-26T23:08:01.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.161+0000 7f9afbff0740 1 -- start start 2024-04-26T23:08:01.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.161+0000 7f9afbff0740 1 Processor -- start 2024-04-26T23:08:01.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.162+0000 7f9afbff0740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890ed07b0 0x55c890f395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.162+0000 7f9afbff0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890ecf770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.162+0000 7f9afbff0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 0x55c890ef3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.162+0000 7f9afbff0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c890efc800 con 0x55c890efb270 2024-04-26T23:08:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.162+0000 7f9afbff0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c890ef30b0 con 0x55c890f0ce80 2024-04-26T23:08:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.162+0000 7f9afbff0740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c890ef1a30 con 0x55c890ed07b0 2024-04-26T23:08:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.162+0000 7f9afbff0740 10 client.0 _ll_register_callbacks cb 0x7f9ae4001e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.163+0000 7f9af1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890ecf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.163+0000 7f9af1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890ecf770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44968/0 (socket says 172.21.15.40:44968) 2024-04-26T23:08:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.163+0000 7f9af1400640 1 -- 172.21.15.40:0/178051240 learned_addr learned my addr 172.21.15.40:0/178051240 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.163+0000 7f9af1400640 1 -- 172.21.15.40:0/178051240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890ed07b0 msgr2=0x55c890f395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.163+0000 7f9af1e00640 1 --2- 172.21.15.40:0/178051240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890ed07b0 0x55c890f395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.163+0000 7f9af1400640 1 --2- 172.21.15.40:0/178051240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890ed07b0 0x55c890f395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.163+0000 7f9af2800640 1 --2- 172.21.15.40:0/178051240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 0x55c890ef3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.163+0000 7f9af1400640 1 -- 172.21.15.40:0/178051240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 msgr2=0x55c890ef3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.163+0000 7f9af1400640 1 --2- 172.21.15.40:0/178051240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 0x55c890ef3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.163+0000 7f9af1400640 1 -- 172.21.15.40:0/178051240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c890ecfe10 con 0x55c890efb270 2024-04-26T23:08:01.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.163+0000 7f9af1400640 1 --2- 172.21.15.40:0/178051240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890ecf770 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f9adc02ace0 tx=0x7f9adc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d757b8fa230c77f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.163+0000 7f9adb400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:01.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.164+0000 7f9adb400640 1 -- 172.21.15.40:0/178051240 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc045660 con 0x55c890efb270 2024-04-26T23:08:01.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.164+0000 7f9adb400640 1 -- 172.21.15.40:0/178051240 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f9adc045800 con 0x55c890efb270 2024-04-26T23:08:01.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.164+0000 7f9afbff0740 1 -- 172.21.15.40:0/178051240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9adc04b070 con 0x55c890efb270 2024-04-26T23:08:01.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.164+0000 7f9adb400640 1 -- 172.21.15.40:0/178051240 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f9adc05f050 con 0x55c890efb270 2024-04-26T23:08:01.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.164+0000 7f9afbff0740 10 client.8247 fetch_fsmap learned FSMap version 96 2024-04-26T23:08:01.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.164+0000 7f9afbff0740 1 -- 172.21.15.40:0/178051240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) -- 0x55c890f39cb0 con 0x55c890efb270 2024-04-26T23:08:01.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.164+0000 7f9adb400640 1 -- 172.21.15.40:0/178051240 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9adc04b070 con 0x55c890efb270 2024-04-26T23:08:01.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.164+0000 7f9adb400640 1 -- 172.21.15.40:0/178051240 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 96) ==== 3534+0+0 (secure 0 0 0) 0x7f9adc062020 con 0x55c890efb270 2024-04-26T23:08:01.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.164+0000 7f9afbff0740 10 client.8247 fetch_fsmap finished waiting for FSMap version 96 2024-04-26T23:08:01.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.164+0000 7f9afbff0740 10 client.8247 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:08:01.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.164+0000 7f9afbff0740 20 client.8247 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:01.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.165+0000 7f9afbff0740 1 --2- 172.21.15.40:0/178051240 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55c890f0ef90 0x55c890f0f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.165+0000 7f9afbff0740 4 client.8247 mds_command: new command op to 8031 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:08:01.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.165+0000 7f9afbff0740 1 -- 172.21.15.40:0/178051240 --> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] -- command(tid 0: {"prefix": "status"}) -- 0x55c890e3c080 con 0x55c890f0ef90 2024-04-26T23:08:01.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.165+0000 7f9af1e00640 1 --2- 172.21.15.40:0/178051240 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55c890f0ef90 0x55c890f0f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.165+0000 7f9af1e00640 1 --2- 172.21.15.40:0/178051240 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55c890f0ef90 0x55c890f0f370 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2439ef819e4f2864 server_cookie=dfeadae6e7eaf1f2 in_seq=0 out_seq=0 2024-04-26T23:08:01.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.165+0000 7f9adb400640 10 client.8247 ms_handle_connect on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:01.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9adb400640 1 -- 172.21.15.40:0/178051240 <== mds.1 v2:172.21.15.40:6832/1811310743 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55c890e3c080 con 0x55c890f0ef90 2024-04-26T23:08:01.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9adb400640 10 client.8247 handle_command_reply: tid=0 2024-04-26T23:08:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9afbff0740 1 client.8247 shutdown 2024-04-26T23:08:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9afbff0740 1 -- 172.21.15.40:0/178051240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 msgr2=0x55c890ecf770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9afbff0740 1 --2- 172.21.15.40:0/178051240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890ecf770 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f9adc02ace0 tx=0x7f9adc004ab0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9afbff0740 1 -- 172.21.15.40:0/178051240 shutdown_connections 2024-04-26T23:08:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9afbff0740 1 --2- 172.21.15.40:0/178051240 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55c890f0ef90 0x55c890f0f370 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9afbff0740 1 --2- 172.21.15.40:0/178051240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c890f0ce80 0x55c890ef3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9afbff0740 1 --2- 172.21.15.40:0/178051240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c890efb270 0x55c890ecf770 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9afbff0740 1 --2- 172.21.15.40:0/178051240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c890ed07b0 0x55c890f395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9adb400640 0 client.8247 ms_handle_reset on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9afbff0740 1 -- 172.21.15.40:0/178051240 >> 172.21.15.40:0/178051240 conn(0x55c890e64d10 msgr2=0x55c890ef6770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9afbff0740 1 -- 172.21.15.40:0/178051240 shutdown_connections 2024-04-26T23:08:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9afbff0740 1 -- 172.21.15.40:0/178051240 wait complete. 2024-04-26T23:08:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.166+0000 7f9afbff0740 20 client.8247 trim_cache size 0 max 16384 2024-04-26T23:08:01.170 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:08:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-04-26T23:08:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8031, 2024-04-26T23:08:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:08:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:08:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:08:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.2970197830000001, 2024-04-26T23:08:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 96, 2024-04-26T23:08:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 90, 2024-04-26T23:08:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-26T23:08:01.170 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 17.743046422999999 2024-04-26T23:08:01.171 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.167+0000 7f9af3200640 1 -- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9ad006c8f0 msgr2=0x7f9ad006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.167+0000 7f9af3200640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9ad006c8f0 0x7f9ad006edb0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4000f60 tx=0x7f9ae4001b60 comp rx=0 tx=0).stop 2024-04-26T23:08:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.167+0000 7f9af3200640 1 -- 172.21.15.40:0/583072059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d32e0 msgr2=0x7f9af40b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.167+0000 7f9af3200640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d32e0 0x7f9af40b44e0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00c450 tx=0x7f9aec00c910 comp rx=0 tx=0).stop 2024-04-26T23:08:01.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.167+0000 7f9af3200640 1 -- 172.21.15.40:0/583072059 shutdown_connections 2024-04-26T23:08:01.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.167+0000 7f9af3200640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9ad006c8f0 0x7f9ad006edb0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.167+0000 7f9af3200640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d32e0 0x7f9af40b44e0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.167+0000 7f9af3200640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9af40b8d20 0x7f9af4160190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.167+0000 7f9af3200640 1 --2- 172.21.15.40:0/583072059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9af40b6220 0x7f9af40b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.167+0000 7f9af3200640 1 -- 172.21.15.40:0/583072059 >> 172.21.15.40:0/583072059 conn(0x7f9af40d4d90 msgr2=0x7f9af406cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.167+0000 7f9af3200640 1 -- 172.21.15.40:0/583072059 shutdown_connections 2024-04-26T23:08:01.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.167+0000 7f9af3200640 1 -- 172.21.15.40:0/583072059 wait complete. 2024-04-26T23:08:01.184 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T23:08:01.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.379+0000 7f3ad6800640 1 Processor -- start 2024-04-26T23:08:01.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.380+0000 7f3ad6800640 1 -- start start 2024-04-26T23:08:01.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.381+0000 7f3ad6800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3ad015f2d0 0x7f3ad00afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.381+0000 7f3ad6800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ad00ce5e0 0x7f3ad00c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.381+0000 7f3ad6800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ad00af7b0 con 0x7f3ad00c5df0 2024-04-26T23:08:01.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.381+0000 7f3ad6800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3ad00afa90 con 0x7f3ad015f2d0 2024-04-26T23:08:01.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.381+0000 7f3ad6800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3ad00cbd90 con 0x7f3ad00ce5e0 2024-04-26T23:08:01.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.382+0000 7f3ad4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ad00ce5e0 0x7f3ad00c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.382+0000 7f3ad4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ad00ce5e0 0x7f3ad00c58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:58048/0 (socket says 172.21.15.40:58048) 2024-04-26T23:08:01.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.382+0000 7f3ad4a00640 1 -- 172.21.15.40:0/55051666 learned_addr learned my addr 172.21.15.40:0/55051666 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:01.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.382+0000 7f3ad5400640 1 --2- 172.21.15.40:0/55051666 >> v2:172.21.15.103:3300/0 conn(0x7f3ad015f2d0 0x7f3ad00afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.383+0000 7f3ad4a00640 1 -- 172.21.15.40:0/55051666 >> v2:172.21.15.103:3300/0 conn(0x7f3ad015f2d0 msgr2=0x7f3ad00afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.383+0000 7f3ad4a00640 1 --2- 172.21.15.40:0/55051666 >> v2:172.21.15.103:3300/0 conn(0x7f3ad015f2d0 0x7f3ad00afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.383+0000 7f3ad4a00640 1 -- 172.21.15.40:0/55051666 >> v1:172.21.15.103:6789/0 conn(0x7f3ad00c5df0 legacy=0x7f3ad00cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.383+0000 7f3ad4a00640 1 -- 172.21.15.40:0/55051666 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad00cc070 con 0x7f3ad00ce5e0 2024-04-26T23:08:01.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.384+0000 7f3ad4a00640 1 --2- 172.21.15.40:0/55051666 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ad00ce5e0 0x7f3ad00c58b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009870 tx=0x7f3abc02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e89f03881cc95e0f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.384+0000 7f3acfe00640 1 -- 172.21.15.40:0/55051666 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc037070 con 0x7f3ad00ce5e0 2024-04-26T23:08:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.384+0000 7f3acfe00640 1 -- 172.21.15.40:0/55051666 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3abc032440 con 0x7f3ad00ce5e0 2024-04-26T23:08:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.384+0000 7f3acfe00640 1 -- 172.21.15.40:0/55051666 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc032810 con 0x7f3ad00ce5e0 2024-04-26T23:08:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.385+0000 7f3ad6800640 1 -- 172.21.15.40:0/55051666 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ad00ce5e0 msgr2=0x7f3ad00c58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.385+0000 7f3ad6800640 1 --2- 172.21.15.40:0/55051666 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ad00ce5e0 0x7f3ad00c58b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009870 tx=0x7f3abc02a250 comp rx=0 tx=0).stop 2024-04-26T23:08:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.385+0000 7f3ad6800640 1 -- 172.21.15.40:0/55051666 shutdown_connections 2024-04-26T23:08:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.385+0000 7f3ad6800640 1 --2- 172.21.15.40:0/55051666 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ad00ce5e0 0x7f3ad00c58b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.385+0000 7f3ad6800640 1 --2- 172.21.15.40:0/55051666 >> v2:172.21.15.103:3300/0 conn(0x7f3ad015f2d0 0x7f3ad00afdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.385+0000 7f3ad6800640 1 -- 172.21.15.40:0/55051666 >> 172.21.15.40:0/55051666 conn(0x7f3ad00d0a10 msgr2=0x7f3ad00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.385+0000 7f3ad6800640 1 -- 172.21.15.40:0/55051666 shutdown_connections 2024-04-26T23:08:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.385+0000 7f3ad6800640 1 -- 172.21.15.40:0/55051666 wait complete. 2024-04-26T23:08:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.387+0000 7f3ad6800640 1 Processor -- start 2024-04-26T23:08:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.387+0000 7f3ad6800640 1 -- start start 2024-04-26T23:08:01.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.388+0000 7f3ad6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad00c5df0 0x7f3ad00b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.388+0000 7f3ad6800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ad00ce5e0 0x7f3ad01f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.388+0000 7f3ad6800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad015f2d0 0x7f3ad00b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.388+0000 7f3ad6800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad00ccca0 con 0x7f3ad00c5df0 2024-04-26T23:08:01.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.388+0000 7f3ad6800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad006e270 con 0x7f3ad015f2d0 2024-04-26T23:08:01.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.388+0000 7f3ad6800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3ad00cce20 con 0x7f3ad00ce5e0 2024-04-26T23:08:01.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.388+0000 7f3ad5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad00c5df0 0x7f3ad00b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.388+0000 7f3ad5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad00c5df0 0x7f3ad00b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44978/0 (socket says 172.21.15.40:44978) 2024-04-26T23:08:01.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.388+0000 7f3ad5400640 1 -- 172.21.15.40:0/530060914 learned_addr learned my addr 172.21.15.40:0/530060914 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:01.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.389+0000 7f3ad4a00640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ad00ce5e0 0x7f3ad01f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.389+0000 7f3ad5400640 1 -- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ad00ce5e0 msgr2=0x7f3ad01f56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.389+0000 7f3ad5400640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ad00ce5e0 0x7f3ad01f56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.389+0000 7f3ad5400640 1 -- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad015f2d0 msgr2=0x7f3ad00b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.389+0000 7f3ad5e00640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad015f2d0 0x7f3ad00b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.389+0000 7f3ad5400640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad015f2d0 0x7f3ad00b3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.389+0000 7f3ad5400640 1 -- 172.21.15.40:0/530060914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad00b4e60 con 0x7f3ad00c5df0 2024-04-26T23:08:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.389+0000 7f3ad5e00640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad015f2d0 0x7f3ad00b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.389+0000 7f3ad5400640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad00c5df0 0x7f3ad00b61e0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e9e0 tx=0x7f3ac400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.389+0000 7f3acd600640 1 -- 172.21.15.40:0/530060914 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4017070 con 0x7f3ad00c5df0 2024-04-26T23:08:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.389+0000 7f3acd600640 1 -- 172.21.15.40:0/530060914 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ac400c9d0 con 0x7f3ad00c5df0 2024-04-26T23:08:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.389+0000 7f3ad6800640 1 -- 172.21.15.40:0/530060914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad00b5140 con 0x7f3ad00c5df0 2024-04-26T23:08:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.390+0000 7f3acd600640 1 -- 172.21.15.40:0/530060914 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac400cdc0 con 0x7f3ad00c5df0 2024-04-26T23:08:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.390+0000 7f3ad6800640 1 -- 172.21.15.40:0/530060914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ad00b5700 con 0x7f3ad00c5df0 2024-04-26T23:08:01.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.393+0000 7f3acd600640 1 -- 172.21.15.40:0/530060914 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3ac401a070 con 0x7f3ad00c5df0 2024-04-26T23:08:01.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.394+0000 7f3acd600640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3ab006c8f0 0x7f3ab006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.394+0000 7f3acd600640 1 -- 172.21.15.40:0/530060914 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f3ac408c050 con 0x7f3ad00c5df0 2024-04-26T23:08:01.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.394+0000 7f3adada1740 1 Processor -- start 2024-04-26T23:08:01.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.394+0000 7f3adada1740 1 -- start start 2024-04-26T23:08:01.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.394+0000 7f3ad4a00640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3ab006c8f0 0x7f3ab006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.395+0000 7f3adada1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff6437910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.395+0000 7f3adada1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6471e80 0x560ff6435b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.395+0000 7f3adada1740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6460270 0x560ff64606f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.395+0000 7f3adada1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560ff6399600 con 0x560ff64357b0 2024-04-26T23:08:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.395+0000 7f3adada1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560ff6336100 con 0x560ff6471e80 2024-04-26T23:08:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.395+0000 7f3adada1740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x560ff63955c0 con 0x560ff6460270 2024-04-26T23:08:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.395+0000 7f3ad5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff6437910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.396+0000 7f3ad5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff6437910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44980/0 (socket says 172.21.15.40:44980) 2024-04-26T23:08:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.396+0000 7f3ad5400640 1 -- 172.21.15.40:0/387609212 learned_addr learned my addr 172.21.15.40:0/387609212 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.396+0000 7f3ad4a00640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3ab006c8f0 0x7f3ab006edb0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f3abc034000 tx=0x7f3abc02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.396+0000 7f3ad5400640 1 -- 172.21.15.40:0/387609212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6460270 msgr2=0x560ff64606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.396+0000 7f3ad5e00640 1 --2- 172.21.15.40:0/387609212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6460270 0x560ff64606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.396+0000 7f3ad5400640 1 --2- 172.21.15.40:0/387609212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6460270 0x560ff64606f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.396+0000 7f3ad5400640 1 -- 172.21.15.40:0/387609212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6471e80 msgr2=0x560ff6435b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.396+0000 7f3ad5400640 1 --2- 172.21.15.40:0/387609212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6471e80 0x560ff6435b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.396+0000 7f3ad5400640 1 -- 172.21.15.40:0/387609212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ff60ab2d0 con 0x560ff64357b0 2024-04-26T23:08:01.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.396+0000 7f3ad5400640 1 --2- 172.21.15.40:0/387609212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff6437910 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4062280 tx=0x7f3ac4059bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3dc2cb4a88915f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.396+0000 7f3ac1c00640 1 -- 172.21.15.40:0/387609212 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4021040 con 0x560ff64357b0 2024-04-26T23:08:01.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.396+0000 7f3ac1c00640 1 -- 172.21.15.40:0/387609212 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ac4072180 con 0x560ff64357b0 2024-04-26T23:08:01.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.396+0000 7f3ac1c00640 1 -- 172.21.15.40:0/387609212 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac40627b0 con 0x560ff64357b0 2024-04-26T23:08:01.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.397+0000 7f3adada1740 1 -- 172.21.15.40:0/387609212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 msgr2=0x560ff6437910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.397+0000 7f3adada1740 1 --2- 172.21.15.40:0/387609212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff6437910 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4062280 tx=0x7f3ac4059bd0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.397+0000 7f3adada1740 1 -- 172.21.15.40:0/387609212 shutdown_connections 2024-04-26T23:08:01.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.397+0000 7f3adada1740 1 --2- 172.21.15.40:0/387609212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6460270 0x560ff64606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.397+0000 7f3adada1740 1 --2- 172.21.15.40:0/387609212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6471e80 0x560ff6435b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.397+0000 7f3adada1740 1 --2- 172.21.15.40:0/387609212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff6437910 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.397+0000 7f3adada1740 1 -- 172.21.15.40:0/387609212 >> 172.21.15.40:0/387609212 conn(0x560ff63c9d10 msgr2=0x560ff647b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.397+0000 7f3adada1740 1 -- 172.21.15.40:0/387609212 shutdown_connections 2024-04-26T23:08:01.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.397+0000 7f3adada1740 1 -- 172.21.15.40:0/387609212 wait complete. 2024-04-26T23:08:01.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.398+0000 7f3adada1740 1 -- start start 2024-04-26T23:08:01.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.398+0000 7f3adada1740 1 Processor -- start 2024-04-26T23:08:01.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.399+0000 7f3adada1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff64357b0 0x560ff6463680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.399+0000 7f3adada1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff6460270 0x560ff644c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.399+0000 7f3adada1740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6471e80 0x560ff64759b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.399+0000 7f3adada1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560ff6320f70 con 0x560ff6460270 2024-04-26T23:08:01.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.399+0000 7f3adada1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560ff645ddc0 con 0x560ff64357b0 2024-04-26T23:08:01.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.399+0000 7f3adada1740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x560ff6463bc0 con 0x560ff6471e80 2024-04-26T23:08:01.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.399+0000 7f3adada1740 10 client.0 _ll_register_callbacks cb 0x560ff638f480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:01.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.400+0000 7f3ad4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff6460270 0x560ff644c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.400+0000 7f3ad4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff6460270 0x560ff644c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44988/0 (socket says 172.21.15.40:44988) 2024-04-26T23:08:01.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.400+0000 7f3ad4a00640 1 -- 172.21.15.40:0/16776789 learned_addr learned my addr 172.21.15.40:0/16776789 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:01.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.400+0000 7f3ad5e00640 1 --2- 172.21.15.40:0/16776789 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6471e80 0x560ff64759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.400+0000 7f3ad5400640 1 --2- 172.21.15.40:0/16776789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff64357b0 0x560ff6463680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.400+0000 7f3ad4a00640 1 -- 172.21.15.40:0/16776789 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6471e80 msgr2=0x560ff64759b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.400+0000 7f3ad4a00640 1 --2- 172.21.15.40:0/16776789 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6471e80 0x560ff64759b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.400+0000 7f3ad4a00640 1 -- 172.21.15.40:0/16776789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff64357b0 msgr2=0x560ff6463680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.400+0000 7f3ad4a00640 1 --2- 172.21.15.40:0/16776789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff64357b0 0x560ff6463680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.400+0000 7f3ad4a00640 1 -- 172.21.15.40:0/16776789 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ff61756b0 con 0x560ff6460270 2024-04-26T23:08:01.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.400+0000 7f3ad4a00640 1 --2- 172.21.15.40:0/16776789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff6460270 0x560ff644c920 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009990 tx=0x7f3abc02c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96414e6cda411b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.400+0000 7f3ab6000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:01.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3ab6000640 1 -- 172.21.15.40:0/16776789 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc04e060 con 0x560ff6460270 2024-04-26T23:08:01.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3adada1740 1 -- 172.21.15.40:0/16776789 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ac4072180 con 0x560ff6460270 2024-04-26T23:08:01.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3ab6000640 1 -- 172.21.15.40:0/16776789 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f3abc03e930 con 0x560ff6460270 2024-04-26T23:08:01.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3ab6000640 1 -- 172.21.15.40:0/16776789 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f3abc054050 con 0x560ff6460270 2024-04-26T23:08:01.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3adada1740 10 client.8256 fetch_fsmap learned FSMap version 96 2024-04-26T23:08:01.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3adada1740 1 -- 172.21.15.40:0/16776789 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) -- 0x560ff6463160 con 0x560ff6460270 2024-04-26T23:08:01.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3ab6000640 1 -- 172.21.15.40:0/16776789 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ac4072180 con 0x560ff6460270 2024-04-26T23:08:01.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3ab6000640 1 -- 172.21.15.40:0/16776789 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 96) ==== 3534+0+0 (secure 0 0 0) 0x7f3abc057020 con 0x560ff6460270 2024-04-26T23:08:01.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3adada1740 10 client.8256 fetch_fsmap finished waiting for FSMap version 96 2024-04-26T23:08:01.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3adada1740 10 client.8256 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:08:01.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3adada1740 20 client.8256 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:01.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3adada1740 1 --2- 172.21.15.40:0/16776789 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x560ff644ed90 0x560ff644f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3adada1740 4 client.8256 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:01.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.401+0000 7f3adada1740 1 -- 172.21.15.40:0/16776789 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ff63a7210 con 0x560ff644ed90 2024-04-26T23:08:01.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.402+0000 7f3ad5400640 1 --2- 172.21.15.40:0/16776789 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x560ff644ed90 0x560ff644f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.402+0000 7f3ad5400640 1 --2- 172.21.15.40:0/16776789 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x560ff644ed90 0x560ff644f170 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54e7f1493b85b970 server_cookie=69c49bedaea14fe3 in_seq=0 out_seq=0 2024-04-26T23:08:01.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.402+0000 7f3ab6000640 10 client.8256 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:01.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.403+0000 7f3ab6000640 1 -- 172.21.15.40:0/16776789 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560ff63a7210 con 0x560ff644ed90 2024-04-26T23:08:01.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.403+0000 7f3ab6000640 10 client.8256 handle_command_reply: tid=0 2024-04-26T23:08:01.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.403+0000 7f3adada1740 1 client.8256 shutdown 2024-04-26T23:08:01.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.403+0000 7f3adada1740 1 -- 172.21.15.40:0/16776789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff6460270 msgr2=0x560ff644c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.403+0000 7f3adada1740 1 --2- 172.21.15.40:0/16776789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff6460270 0x560ff644c920 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009990 tx=0x7f3abc02c070 comp rx=0 tx=0).stop 2024-04-26T23:08:01.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.403+0000 7f3adada1740 1 -- 172.21.15.40:0/16776789 shutdown_connections 2024-04-26T23:08:01.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.403+0000 7f3adada1740 1 --2- 172.21.15.40:0/16776789 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x560ff644ed90 0x560ff644f170 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.403+0000 7f3adada1740 1 --2- 172.21.15.40:0/16776789 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6471e80 0x560ff64759b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.403+0000 7f3adada1740 1 --2- 172.21.15.40:0/16776789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff6460270 0x560ff644c920 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.403+0000 7f3adada1740 1 --2- 172.21.15.40:0/16776789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff64357b0 0x560ff6463680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.403+0000 7f3ab6000640 0 client.8256 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:01.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.403+0000 7f3adada1740 1 -- 172.21.15.40:0/16776789 >> 172.21.15.40:0/16776789 conn(0x560ff63c9d10 msgr2=0x560ff645e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.404+0000 7f3adada1740 1 -- 172.21.15.40:0/16776789 shutdown_connections 2024-04-26T23:08:01.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.404+0000 7f3adada1740 1 -- 172.21.15.40:0/16776789 wait complete. 2024-04-26T23:08:01.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.404+0000 7f3adada1740 20 client.8256 trim_cache size 0 max 16384 2024-04-26T23:08:01.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.423+0000 7f3adada1740 1 Processor -- start 2024-04-26T23:08:01.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.423+0000 7f3adada1740 1 -- start start 2024-04-26T23:08:01.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.424+0000 7f3adada1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff644b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.424+0000 7f3adada1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6460270 0x560ff644bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.424+0000 7f3adada1740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6471e80 0x560ff645f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.424+0000 7f3adada1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad0200e70 con 0x560ff64357b0 2024-04-26T23:08:01.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.424+0000 7f3adada1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad0201540 con 0x560ff6460270 2024-04-26T23:08:01.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.424+0000 7f3adada1740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3ad0156710 con 0x560ff6471e80 2024-04-26T23:08:01.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.425+0000 7f3ad5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff644b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.425+0000 7f3ad5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff644b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45000/0 (socket says 172.21.15.40:45000) 2024-04-26T23:08:01.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.425+0000 7f3ad5400640 1 -- 172.21.15.40:0/2651871523 learned_addr learned my addr 172.21.15.40:0/2651871523 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:01.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.425+0000 7f3ad5e00640 1 --2- 172.21.15.40:0/2651871523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6471e80 0x560ff645f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.425+0000 7f3ad4a00640 1 --2- 172.21.15.40:0/2651871523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6460270 0x560ff644bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.425+0000 7f3ad5400640 1 -- 172.21.15.40:0/2651871523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6471e80 msgr2=0x560ff645f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.425+0000 7f3ad5400640 1 --2- 172.21.15.40:0/2651871523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6471e80 0x560ff645f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.425+0000 7f3ad5400640 1 -- 172.21.15.40:0/2651871523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6460270 msgr2=0x560ff644bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.425+0000 7f3ad5400640 1 --2- 172.21.15.40:0/2651871523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6460270 0x560ff644bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.425+0000 7f3ad5400640 1 -- 172.21.15.40:0/2651871523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ff647b870 con 0x560ff64357b0 2024-04-26T23:08:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.425+0000 7f3ad4a00640 1 --2- 172.21.15.40:0/2651871523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6460270 0x560ff644bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.426+0000 7f3ad5400640 1 --2- 172.21.15.40:0/2651871523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff644b830 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f3ac405e140 tx=0x7f3ac401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ca541e0b8801a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.426+0000 7f3ac1200640 1 -- 172.21.15.40:0/2651871523 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac407e070 con 0x560ff64357b0 2024-04-26T23:08:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.426+0000 7f3ac1200640 1 -- 172.21.15.40:0/2651871523 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ac4077070 con 0x560ff64357b0 2024-04-26T23:08:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.426+0000 7f3ac1200640 1 -- 172.21.15.40:0/2651871523 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4089040 con 0x560ff64357b0 2024-04-26T23:08:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.427+0000 7f3adada1740 1 -- 172.21.15.40:0/2651871523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 msgr2=0x560ff644b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.427+0000 7f3adada1740 1 --2- 172.21.15.40:0/2651871523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff644b830 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f3ac405e140 tx=0x7f3ac401e000 comp rx=0 tx=0).stop 2024-04-26T23:08:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.427+0000 7f3adada1740 1 -- 172.21.15.40:0/2651871523 shutdown_connections 2024-04-26T23:08:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.427+0000 7f3adada1740 1 --2- 172.21.15.40:0/2651871523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6471e80 0x560ff645f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.427+0000 7f3adada1740 1 --2- 172.21.15.40:0/2651871523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6460270 0x560ff644bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.427+0000 7f3adada1740 1 --2- 172.21.15.40:0/2651871523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff644b830 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.427+0000 7f3adada1740 1 -- 172.21.15.40:0/2651871523 >> 172.21.15.40:0/2651871523 conn(0x560ff63c9d10 msgr2=0x560ff6436690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.427+0000 7f3adada1740 1 -- 172.21.15.40:0/2651871523 shutdown_connections 2024-04-26T23:08:01.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.427+0000 7f3adada1740 1 -- 172.21.15.40:0/2651871523 wait complete. 2024-04-26T23:08:01.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.427+0000 7f3adada1740 1 -- start start 2024-04-26T23:08:01.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.428+0000 7f3adada1740 1 Processor -- start 2024-04-26T23:08:01.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3adada1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff644b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3adada1740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6460270 0x560ff644bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3adada1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6471e80 0x560ff6433e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3adada1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560ff644ce60 con 0x560ff64357b0 2024-04-26T23:08:01.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3adada1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560ff6452c60 con 0x560ff6471e80 2024-04-26T23:08:01.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3adada1740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x560ff647ada0 con 0x560ff6460270 2024-04-26T23:08:01.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3adada1740 10 client.0 _ll_register_callbacks cb 0x7f3abc035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:01.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3ad5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff644b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3ad5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff644b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45008/0 (socket says 172.21.15.40:45008) 2024-04-26T23:08:01.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3ad5400640 1 -- 172.21.15.40:0/621059175 learned_addr learned my addr 172.21.15.40:0/621059175 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:01.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3ad5400640 1 -- 172.21.15.40:0/621059175 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6460270 msgr2=0x560ff644bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3ad5400640 1 --2- 172.21.15.40:0/621059175 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6460270 0x560ff644bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3ad5400640 1 -- 172.21.15.40:0/621059175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6471e80 msgr2=0x560ff6433e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3ad5400640 1 --2- 172.21.15.40:0/621059175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6471e80 0x560ff6433e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.429+0000 7f3ad5400640 1 -- 172.21.15.40:0/621059175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ff6470fd0 con 0x560ff64357b0 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3ad5400640 1 --2- 172.21.15.40:0/621059175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff644b830 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f3ac40598d0 tx=0x7f3ac4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf81038d0054ea0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3ab6000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3ab6000640 1 -- 172.21.15.40:0/621059175 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac407e030 con 0x560ff64357b0 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3ab6000640 1 -- 172.21.15.40:0/621059175 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f3ac4057c70 con 0x560ff64357b0 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3adada1740 1 -- 172.21.15.40:0/621059175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ac4077070 con 0x560ff64357b0 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3ab6000640 1 -- 172.21.15.40:0/621059175 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f3ac4021040 con 0x560ff64357b0 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3adada1740 10 client.8262 fetch_fsmap learned FSMap version 96 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3adada1740 1 -- 172.21.15.40:0/621059175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) -- 0x560ff6470d30 con 0x560ff64357b0 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3ab6000640 1 -- 172.21.15.40:0/621059175 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ac4077070 con 0x560ff64357b0 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3ab6000640 1 -- 172.21.15.40:0/621059175 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 96) ==== 3534+0+0 (secure 0 0 0) 0x7f3ac4056240 con 0x560ff64357b0 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3adada1740 10 client.8262 fetch_fsmap finished waiting for FSMap version 96 2024-04-26T23:08:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3adada1740 10 client.8262 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3adada1740 20 client.8262 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3adada1740 1 --2- 172.21.15.40:0/621059175 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x560ff6434580 0x560ff647e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3adada1740 4 client.8262 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.430+0000 7f3adada1740 1 -- 172.21.15.40:0/621059175 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "status"}) -- 0x560ff63a1080 con 0x560ff6434580 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.431+0000 7f3ad4a00640 1 --2- 172.21.15.40:0/621059175 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x560ff6434580 0x560ff647e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.431+0000 7f3ad4a00640 1 --2- 172.21.15.40:0/621059175 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x560ff6434580 0x560ff647e440 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=856510024669faf3 server_cookie=8e8a94ea1e466fb in_seq=0 out_seq=0 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.431+0000 7f3ab6000640 10 client.8262 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3ab6000640 1 -- 172.21.15.40:0/621059175 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+319 (crc 0 0 0) 0x560ff63a1080 con 0x560ff6434580 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3ab6000640 10 client.8262 handle_command_reply: tid=0 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3adada1740 1 client.8262 shutdown 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3adada1740 1 -- 172.21.15.40:0/621059175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 msgr2=0x560ff644b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3adada1740 1 --2- 172.21.15.40:0/621059175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff644b830 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f3ac40598d0 tx=0x7f3ac4000bc0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3adada1740 1 -- 172.21.15.40:0/621059175 shutdown_connections 2024-04-26T23:08:01.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3adada1740 1 --2- 172.21.15.40:0/621059175 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x560ff6434580 0x560ff647e440 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3adada1740 1 --2- 172.21.15.40:0/621059175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ff6471e80 0x560ff6433e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3adada1740 1 --2- 172.21.15.40:0/621059175 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ff6460270 0x560ff644bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3adada1740 1 --2- 172.21.15.40:0/621059175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ff64357b0 0x560ff644b830 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3ab6000640 0 client.8262 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:01.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3adada1740 1 -- 172.21.15.40:0/621059175 >> 172.21.15.40:0/621059175 conn(0x560ff63c9d10 msgr2=0x560ff6436690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3adada1740 1 -- 172.21.15.40:0/621059175 shutdown_connections 2024-04-26T23:08:01.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3adada1740 1 -- 172.21.15.40:0/621059175 wait complete. 2024-04-26T23:08:01.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.432+0000 7f3adada1740 20 client.8262 trim_cache size 0 max 16384 2024-04-26T23:08:01.436 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:01.436 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:08:01.436 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:08:01.437 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8040, 2024-04-26T23:08:01.437 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:08:01.437 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:08:01.437 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:08:01.437 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 12.630910987, 2024-04-26T23:08:01.437 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 96, 2024-04-26T23:08:01.437 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 90, 2024-04-26T23:08:01.437 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-26T23:08:01.437 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 18.000801958 2024-04-26T23:08:01.437 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:01.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.433+0000 7f3ad6800640 1 -- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3ab006c8f0 msgr2=0x7f3ab006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.433+0000 7f3ad6800640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3ab006c8f0 0x7f3ab006edb0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f3abc034000 tx=0x7f3abc02e000 comp rx=0 tx=0).stop 2024-04-26T23:08:01.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.433+0000 7f3ad6800640 1 -- 172.21.15.40:0/530060914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad00c5df0 msgr2=0x7f3ad00b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.433+0000 7f3ad6800640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad00c5df0 0x7f3ad00b61e0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e9e0 tx=0x7f3ac400eea0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.433+0000 7f3ad6800640 1 -- 172.21.15.40:0/530060914 shutdown_connections 2024-04-26T23:08:01.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.433+0000 7f3ad6800640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3ab006c8f0 0x7f3ab006edb0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.433+0000 7f3ad6800640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad015f2d0 0x7f3ad00b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.433+0000 7f3ad6800640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ad00ce5e0 0x7f3ad01f56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.433+0000 7f3ad6800640 1 --2- 172.21.15.40:0/530060914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ad00c5df0 0x7f3ad00b61e0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.433+0000 7f3ad6800640 1 -- 172.21.15.40:0/530060914 >> 172.21.15.40:0/530060914 conn(0x7f3ad00d0a10 msgr2=0x7f3ad006e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.433+0000 7f3ad6800640 1 -- 172.21.15.40:0/530060914 shutdown_connections 2024-04-26T23:08:01.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.433+0000 7f3ad6800640 1 -- 172.21.15.40:0/530060914 wait complete. 2024-04-26T23:08:01.449 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:01.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.646+0000 7fa0baa00640 1 Processor -- start 2024-04-26T23:08:01.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.647+0000 7fa0baa00640 1 -- start start 2024-04-26T23:08:01.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.648+0000 7fa0baa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa0b406f790 0x7fa0b406fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.648+0000 7fa0baa00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0b4156840 con 0x7fa0b4156460 2024-04-26T23:08:01.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.648+0000 7fa0baa00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0b40cbf20 con 0x7fa0b4072c50 2024-04-26T23:08:01.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.648+0000 7fa0baa00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b40cc200 con 0x7fa0b406f790 2024-04-26T23:08:01.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.648+0000 7fa0b3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa0b4156460 0x7fa0b40cb4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42030/0 (socket says 172.21.15.40:42030) 2024-04-26T23:08:01.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.648+0000 7fa0b3e00640 1 -- 172.21.15.40:0/3153378954 learned_addr learned my addr 172.21.15.40:0/3153378954 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.649+0000 7fa0b3400640 1 --2- 172.21.15.40:0/3153378954 >> v2:172.21.15.103:3300/0 conn(0x7fa0b406f790 0x7fa0b406fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.649+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4102265655 0 0) 0x7fa0b4156840 con 0x7fa0b4156460 2024-04-26T23:08:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.649+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa098003520 con 0x7fa0b4156460 2024-04-26T23:08:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.649+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3428557169 0 0) 0x7fa098003520 con 0x7fa0b4156460 2024-04-26T23:08:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.649+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 >> v2:172.21.15.103:3300/0 conn(0x7fa0b406f790 msgr2=0x7fa0b406fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.649+0000 7fa0b2a00640 1 --2- 172.21.15.40:0/3153378954 >> v2:172.21.15.103:3300/0 conn(0x7fa0b406f790 0x7fa0b406fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.649+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 >> v1:172.21.15.103:6789/0 conn(0x7fa0b4072c50 legacy=0x7fa0b4073060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.649+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b40cc4e0 con 0x7fa0b4156460 2024-04-26T23:08:01.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.649+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fa0a40019f0 con 0x7fa0b4156460 2024-04-26T23:08:01.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.649+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 >> v1:172.21.15.40:6789/0 conn(0x7fa0b4156460 legacy=0x7fa0b40cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.649+0000 7fa0b2a00640 1 --2- 172.21.15.40:0/3153378954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0980039b0 0x7fa098003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b2a00640 1 --2- 172.21.15.40:0/3153378954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa098004800 0x7fa098004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b2a00640 1 --2- 172.21.15.40:0/3153378954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0980051c0 0x7fa09800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b40cc200 con 0x7fa0980039b0 2024-04-26T23:08:01.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa09800c060 con 0x7fa098004800 2024-04-26T23:08:01.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa09800c260 con 0x7fa0980051c0 2024-04-26T23:08:01.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b3e00640 1 --2- 172.21.15.40:0/3153378954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0980039b0 0x7fa098003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b3400640 1 --2- 172.21.15.40:0/3153378954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa098004800 0x7fa098004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b8a00640 1 --2- 172.21.15.40:0/3153378954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0980051c0 0x7fa09800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b3e00640 1 -- 172.21.15.40:0/3153378954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0980051c0 msgr2=0x7fa09800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b3e00640 1 --2- 172.21.15.40:0/3153378954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0980051c0 0x7fa09800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b3e00640 1 -- 172.21.15.40:0/3153378954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa098004800 msgr2=0x7fa098004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b3e00640 1 --2- 172.21.15.40:0/3153378954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa098004800 0x7fa098004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b3e00640 1 -- 172.21.15.40:0/3153378954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa098003520 con 0x7fa0980039b0 2024-04-26T23:08:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.650+0000 7fa0b8a00640 1 -- 172.21.15.40:0/3153378954 reap_dead start 2024-04-26T23:08:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.651+0000 7fa0b3e00640 1 --2- 172.21.15.40:0/3153378954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0980039b0 0x7fa098003dd0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400a840 tx=0x7fa098005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c370cbf53700c73 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.652+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a400a460 con 0x7fa0980039b0 2024-04-26T23:08:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.652+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a402f070 con 0x7fa0980039b0 2024-04-26T23:08:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.652+0000 7fa0b2a00640 1 -- 172.21.15.40:0/3153378954 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a402bcf0 con 0x7fa0980039b0 2024-04-26T23:08:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.652+0000 7fa0baa00640 1 -- 172.21.15.40:0/3153378954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0980039b0 msgr2=0x7fa098003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.652+0000 7fa0baa00640 1 --2- 172.21.15.40:0/3153378954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0980039b0 0x7fa098003dd0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400a840 tx=0x7fa098005830 comp rx=0 tx=0).stop 2024-04-26T23:08:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.652+0000 7fa0baa00640 1 -- 172.21.15.40:0/3153378954 shutdown_connections 2024-04-26T23:08:01.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.652+0000 7fa0baa00640 1 --2- 172.21.15.40:0/3153378954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0980039b0 0x7fa098003dd0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.652+0000 7fa0baa00640 1 -- 172.21.15.40:0/3153378954 >> 172.21.15.40:0/3153378954 conn(0x7fa0b40d0be0 msgr2=0x7fa0b40d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.653+0000 7fa0baa00640 1 -- 172.21.15.40:0/3153378954 shutdown_connections 2024-04-26T23:08:01.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.653+0000 7fa0baa00640 1 -- 172.21.15.40:0/3153378954 wait complete. 2024-04-26T23:08:01.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.654+0000 7fa0baa00640 1 Processor -- start 2024-04-26T23:08:01.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.654+0000 7fa0baa00640 1 -- start start 2024-04-26T23:08:01.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.655+0000 7fa0baa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0980039b0 0x7fa0b41f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.655+0000 7fa0baa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b415dc70 0x7fa0b40ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.655+0000 7fa0baa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0b415e0f0 0x7fa0b40cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.655+0000 7fa0baa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b406e260 con 0x7fa0980039b0 2024-04-26T23:08:01.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.655+0000 7fa0baa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b4072700 con 0x7fa0b415dc70 2024-04-26T23:08:01.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.655+0000 7fa0baa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b406e3e0 con 0x7fa0b415e0f0 2024-04-26T23:08:01.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.656+0000 7fa0b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0980039b0 0x7fa0b41f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.656+0000 7fa0b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0980039b0 0x7fa0b41f5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45014/0 (socket says 172.21.15.40:45014) 2024-04-26T23:08:01.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.656+0000 7fa0b3e00640 1 -- 172.21.15.40:0/2095549416 learned_addr learned my addr 172.21.15.40:0/2095549416 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:01.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.656+0000 7fa0b3400640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b415dc70 0x7fa0b40ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.656+0000 7fa0b8a00640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0b415e0f0 0x7fa0b40cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.656+0000 7fa0b3e00640 1 -- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0b415e0f0 msgr2=0x7fa0b40cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.656+0000 7fa0b3e00640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0b415e0f0 0x7fa0b40cecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.656+0000 7fa0b3e00640 1 -- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b415dc70 msgr2=0x7fa0b40ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.656+0000 7fa0b3e00640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b415dc70 0x7fa0b40ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.656+0000 7fa0b3e00640 1 -- 172.21.15.40:0/2095549416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b415c730 con 0x7fa0980039b0 2024-04-26T23:08:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.656+0000 7fa0b3400640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b415dc70 0x7fa0b40ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.656+0000 7fa0b3e00640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0980039b0 0x7fa0b41f5130 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4001cf0 tx=0x7fa0b40ccaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.657+0000 7fa0abe00640 1 -- 172.21.15.40:0/2095549416 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4044070 con 0x7fa0980039b0 2024-04-26T23:08:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.657+0000 7fa0abe00640 1 -- 172.21.15.40:0/2095549416 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a4034670 con 0x7fa0980039b0 2024-04-26T23:08:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.657+0000 7fa0abe00640 1 -- 172.21.15.40:0/2095549416 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a400a460 con 0x7fa0980039b0 2024-04-26T23:08:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.657+0000 7fa0baa00640 1 -- 172.21.15.40:0/2095549416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b415ca10 con 0x7fa0980039b0 2024-04-26T23:08:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.657+0000 7fa0baa00640 1 -- 172.21.15.40:0/2095549416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b415cfd0 con 0x7fa0980039b0 2024-04-26T23:08:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.658+0000 7fa0baa00640 1 -- 172.21.15.40:0/2095549416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0b40694f0 con 0x7fa0980039b0 2024-04-26T23:08:01.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.659+0000 7fa0abe00640 1 -- 172.21.15.40:0/2095549416 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa0a4047070 con 0x7fa0980039b0 2024-04-26T23:08:01.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.660+0000 7fa0abe00640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa09006c8f0 0x7fa09006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:01.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.660+0000 7fa0abe00640 1 -- 172.21.15.40:0/2095549416 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fa0a40abc40 con 0x7fa0980039b0 2024-04-26T23:08:01.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.661+0000 7fa0b3400640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa09006c8f0 0x7fa09006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:01.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.662+0000 7fa0b3400640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa09006c8f0 0x7fa09006edb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa0b414d830 tx=0x7fa09c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.665+0000 7fa0abe00640 1 -- 172.21.15.40:0/2095549416 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa0a4079e40 con 0x7fa0980039b0 2024-04-26T23:08:01.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.911+0000 7fa0baa00640 1 -- 172.21.15.40:0/2095549416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0b406de80 con 0x7fa0980039b0 2024-04-26T23:08:01.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.912+0000 7fa0abe00640 1 -- 172.21.15.40:0/2095549416 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5221 (secure 0 0 0) 0x7fa0a4079740 con 0x7fa0980039b0 2024-04-26T23:08:01.913 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:01.913 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":96,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:07:59.880947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:01.913 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 96 2024-04-26T23:08:01.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.915+0000 7fa0baa00640 1 -- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa09006c8f0 msgr2=0x7fa09006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.915+0000 7fa0baa00640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa09006c8f0 0x7fa09006edb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa0b414d830 tx=0x7fa09c008040 comp rx=0 tx=0).stop 2024-04-26T23:08:01.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.915+0000 7fa0baa00640 1 -- 172.21.15.40:0/2095549416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0980039b0 msgr2=0x7fa0b41f5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:01.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.915+0000 7fa0baa00640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0980039b0 0x7fa0b41f5130 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4001cf0 tx=0x7fa0b40ccaa0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.915+0000 7fa0baa00640 1 -- 172.21.15.40:0/2095549416 shutdown_connections 2024-04-26T23:08:01.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.915+0000 7fa0baa00640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa09006c8f0 0x7fa09006edb0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.915+0000 7fa0baa00640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0b415e0f0 0x7fa0b40cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.915+0000 7fa0baa00640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b415dc70 0x7fa0b40ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.915+0000 7fa0baa00640 1 --2- 172.21.15.40:0/2095549416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0980039b0 0x7fa0b41f5130 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:01.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.915+0000 7fa0baa00640 1 -- 172.21.15.40:0/2095549416 >> 172.21.15.40:0/2095549416 conn(0x7fa0b40d0be0 msgr2=0x7fa0b406ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:01.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.916+0000 7fa0baa00640 1 -- 172.21.15.40:0/2095549416 shutdown_connections 2024-04-26T23:08:01.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:01.916+0000 7fa0baa00640 1 -- 172.21.15.40:0/2095549416 wait complete. 2024-04-26T23:08:01.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:08:01.936 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p d1/d2/d3/d4/d5/d6/d7/d8) 2024-04-26T23:08:02.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:08:02.006 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec sync d1/d2/d3/d4/d5/d6/d7/d8) 2024-04-26T23:08:02.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:08:02.067 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 d1/d2) 2024-04-26T23:08:02.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:08:02.119 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 d1/d2/d3/d4) 2024-04-26T23:08:02.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:08:02.175 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 2 d1/d2/d3/d4/d5/d6) 2024-04-26T23:08:02.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:08:02.461 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"5","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:08:02.461 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:08:02.461 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:08:04.463 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:08:04.702 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"7","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:08:04.702 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:08:04.702 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:08:04.755 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:08:04.755+0000 7fd9e1000640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T23:08:06.703 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:08:06.946 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"7","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:08:06.946 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:08:06.946 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:08:08.947 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:08:09.184 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"13","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/d1/d2","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-04-26T23:08:09.184 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000001", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/d1/d2/d3/d4", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-04-26T23:08:09.185 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:08:09.185 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:08:09.418 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:08:09.419 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:08:09.419 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:08:11.420 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:08:11.661 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:08:11.662 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:08:11.662 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:08:13.663 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:08:13.905 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:08:13.906 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:08:13.906 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:08:15.907 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:08:16.142 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1,"2":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":2,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/d1/d2/d3/d4/d5/d6","dirfrag":"0x10000000005","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-04-26T23:08:16.143 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1, "2": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "2", "dirfrag": "0x10000000005", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/d1/d2/d3/d4/d5/d6", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "6", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 2, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false } ] 2024-04-26T23:08:16.143 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:08:16.143 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-04-26T23:08:16.374 INFO:teuthology.orchestra.run.smithi103.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"0","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dnwaiter":0,"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":2,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/d1/d2/d3/d4/d5/d6","dirfrag":"0x10000000005","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:08:16.375 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dnwaiter": 0, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "2", "dirfrag": "0x10000000005", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4/d5/d6", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 2, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:08:16.375 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:08:16.375 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 flush journal 2024-04-26T23:08:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.606+0000 7f7b65a00640 1 Processor -- start 2024-04-26T23:08:16.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.606+0000 7f7b65a00640 1 -- start start 2024-04-26T23:08:16.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.608+0000 7f7b65a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7b6015bc70 0x7f7b600cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.609+0000 7f7b65a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7b60071ef0 0x7f7b60072990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.609+0000 7f7b65a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b6006dde0 con 0x7f7b60072440 2024-04-26T23:08:16.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.609+0000 7f7b65a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f7b6006e3a0 con 0x7f7b6015bc70 2024-04-26T23:08:16.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.609+0000 7f7b65a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7b600b4840 con 0x7f7b60071ef0 2024-04-26T23:08:16.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.609+0000 7f7b5fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7b6015bc70 0x7f7b600cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.609+0000 7f7b65000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7b60072440 0x7f7b6015f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60574/0 (socket says 172.21.15.40:60574) 2024-04-26T23:08:16.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.609+0000 7f7b5fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7b6015bc70 0x7f7b600cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54494/0 (socket says 172.21.15.40:54494) 2024-04-26T23:08:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.609+0000 7f7b65000640 1 -- 172.21.15.40:0/482197846 learned_addr learned my addr 172.21.15.40:0/482197846 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.609+0000 7f7b5f400640 1 --2- 172.21.15.40:0/482197846 >> v2:172.21.15.103:3300/0 conn(0x7f7b60071ef0 0x7f7b60072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.610+0000 7f7b5fe00640 1 -- 172.21.15.40:0/482197846 >> v2:172.21.15.103:3300/0 conn(0x7f7b60071ef0 msgr2=0x7f7b60072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.610+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/482197846 >> v2:172.21.15.103:3300/0 conn(0x7f7b60071ef0 0x7f7b60072990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.610+0000 7f7b5fe00640 1 -- 172.21.15.40:0/482197846 >> v1:172.21.15.40:6789/0 conn(0x7f7b60072440 legacy=0x7f7b6015f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.610+0000 7f7b5fe00640 1 -- 172.21.15.40:0/482197846 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6006e0c0 con 0x7f7b6015bc70 2024-04-26T23:08:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.610+0000 7f7b5ea00640 1 -- 172.21.15.40:0/482197846 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 408786589 0 0) 0x7f7b6006dde0 con 0x7f7b60072440 2024-04-26T23:08:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.611+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/482197846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6015bc70 0x7f7b600cdf30 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f7b60071670 tx=0x7f7b5002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2c2b88bf0c0f27 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.611+0000 7f7b5ea00640 1 -- 172.21.15.40:0/482197846 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b50007740 con 0x7f7b6015bc70 2024-04-26T23:08:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.611+0000 7f7b5ea00640 1 -- 172.21.15.40:0/482197846 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b500078e0 con 0x7f7b6015bc70 2024-04-26T23:08:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.611+0000 7f7b5ea00640 1 -- 172.21.15.40:0/482197846 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b50007cb0 con 0x7f7b6015bc70 2024-04-26T23:08:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.612+0000 7f7b65a00640 1 -- 172.21.15.40:0/482197846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6015bc70 msgr2=0x7f7b600cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.612+0000 7f7b65a00640 1 --2- 172.21.15.40:0/482197846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6015bc70 0x7f7b600cdf30 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f7b60071670 tx=0x7f7b5002a680 comp rx=0 tx=0).stop 2024-04-26T23:08:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.612+0000 7f7b65a00640 1 -- 172.21.15.40:0/482197846 shutdown_connections 2024-04-26T23:08:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.612+0000 7f7b65a00640 1 --2- 172.21.15.40:0/482197846 >> v2:172.21.15.103:3300/0 conn(0x7f7b60071ef0 0x7f7b60072990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.612+0000 7f7b65a00640 1 --2- 172.21.15.40:0/482197846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6015bc70 0x7f7b600cdf30 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.612+0000 7f7b65a00640 1 -- 172.21.15.40:0/482197846 >> 172.21.15.40:0/482197846 conn(0x7f7b600b2720 msgr2=0x7f7b600b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.612+0000 7f7b65a00640 1 -- 172.21.15.40:0/482197846 shutdown_connections 2024-04-26T23:08:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.612+0000 7f7b65a00640 1 -- 172.21.15.40:0/482197846 wait complete. 2024-04-26T23:08:16.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.614+0000 7f7b65a00640 1 Processor -- start 2024-04-26T23:08:16.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.614+0000 7f7b65a00640 1 -- start start 2024-04-26T23:08:16.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.615+0000 7f7b65a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b60071ef0 0x7f7b60141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.615+0000 7f7b65a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b60072440 0x7f7b60141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.615+0000 7f7b65a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b6015bc70 0x7f7b6015e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.615+0000 7f7b65a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6015c050 con 0x7f7b60071ef0 2024-04-26T23:08:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.615+0000 7f7b65a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6006fad0 con 0x7f7b60072440 2024-04-26T23:08:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.615+0000 7f7b65a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7b6015c1d0 con 0x7f7b6015bc70 2024-04-26T23:08:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.615+0000 7f7b5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b60071ef0 0x7f7b60141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.615+0000 7f7b5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b60071ef0 0x7f7b60141510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54508/0 (socket says 172.21.15.40:54508) 2024-04-26T23:08:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.615+0000 7f7b5fe00640 1 -- 172.21.15.40:0/2694982639 learned_addr learned my addr 172.21.15.40:0/2694982639 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.616+0000 7f7b5fe00640 1 -- 172.21.15.40:0/2694982639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b6015bc70 msgr2=0x7f7b6015e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.616+0000 7f7b5f400640 1 --2- 172.21.15.40:0/2694982639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b60072440 0x7f7b60141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.616+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/2694982639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b6015bc70 0x7f7b6015e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.616+0000 7f7b5fe00640 1 -- 172.21.15.40:0/2694982639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b60072440 msgr2=0x7f7b60141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.616+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/2694982639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b60072440 0x7f7b60141dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.616+0000 7f7b5fe00640 1 -- 172.21.15.40:0/2694982639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b60144c40 con 0x7f7b60071ef0 2024-04-26T23:08:16.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.616+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/2694982639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b60071ef0 0x7f7b60141510 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f7b50000f80 tx=0x7f7b50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.616+0000 7f7b57e00640 1 -- 172.21.15.40:0/2694982639 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b50040070 con 0x7f7b60071ef0 2024-04-26T23:08:16.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.616+0000 7f7b57e00640 1 -- 172.21.15.40:0/2694982639 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b50031c40 con 0x7f7b60071ef0 2024-04-26T23:08:16.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.616+0000 7f7b57e00640 1 -- 172.21.15.40:0/2694982639 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b50009450 con 0x7f7b60071ef0 2024-04-26T23:08:16.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.616+0000 7f7b65a00640 1 -- 172.21.15.40:0/2694982639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b6014e5e0 con 0x7f7b60071ef0 2024-04-26T23:08:16.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.616+0000 7f7b65a00640 1 -- 172.21.15.40:0/2694982639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b6014eba0 con 0x7f7b60071ef0 2024-04-26T23:08:16.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.619+0000 7f7b57e00640 1 -- 172.21.15.40:0/2694982639 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7b50039040 con 0x7f7b60071ef0 2024-04-26T23:08:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.619+0000 7f7b57e00640 1 --2- 172.21.15.40:0/2694982639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7b4006c920 0x7f7b4006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.620+0000 7f7b57e00640 1 -- 172.21.15.40:0/2694982639 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f7b500acff0 con 0x7f7b60071ef0 2024-04-26T23:08:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.620+0000 7f7b69d6b740 1 Processor -- start 2024-04-26T23:08:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.620+0000 7f7b5f400640 1 --2- 172.21.15.40:0/2694982639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7b4006c920 0x7f7b4006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.620+0000 7f7b69d6b740 1 -- start start 2024-04-26T23:08:16.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.621+0000 7f7b69d6b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 0x55d1c065ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.621+0000 7f7b69d6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c067c3c0 0x55d1c067c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b69d6b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c0634800 0x55d1c067aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b69d6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d1c0540030 con 0x55d1c067c3c0 2024-04-26T23:08:16.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b69d6b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d1c055b7f0 con 0x55d1c0634800 2024-04-26T23:08:16.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b69d6b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d1c05167e0 con 0x55d1c065eb20 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b5f400640 1 --2- 172.21.15.40:0/2694982639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7b4006c920 0x7f7b4006ede0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c006fd0 tx=0x7f7b4c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b5f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c067c3c0 0x55d1c067c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b5f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c067c3c0 0x55d1c067c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54522/0 (socket says 172.21.15.40:54522) 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b5f400640 1 -- 172.21.15.40:0/3732106371 learned_addr learned my addr 172.21.15.40:0/3732106371 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/3732106371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 0x55d1c065ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b5f400640 1 -- 172.21.15.40:0/3732106371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 msgr2=0x55d1c065ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b5f400640 1 --2- 172.21.15.40:0/3732106371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 0x55d1c065ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b5f400640 1 -- 172.21.15.40:0/3732106371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c0634800 msgr2=0x55d1c067aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b65000640 1 --2- 172.21.15.40:0/3732106371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c0634800 0x55d1c067aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b5f400640 1 --2- 172.21.15.40:0/3732106371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c0634800 0x55d1c067aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b5f400640 1 -- 172.21.15.40:0/3732106371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1c0522050 con 0x55d1c067c3c0 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.622+0000 7f7b65000640 1 --2- 172.21.15.40:0/3732106371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c0634800 0x55d1c067aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.623+0000 7f7b5f400640 1 --2- 172.21.15.40:0/3732106371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c067c3c0 0x55d1c067c910 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c0043d0 tx=0x7f7b4c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792c8aa251d2ff03 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.623+0000 7f7b54c00640 1 -- 172.21.15.40:0/3732106371 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c00f040 con 0x55d1c067c3c0 2024-04-26T23:08:16.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.623+0000 7f7b54c00640 1 -- 172.21.15.40:0/3732106371 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b4c0044c0 con 0x55d1c067c3c0 2024-04-26T23:08:16.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.623+0000 7f7b54c00640 1 -- 172.21.15.40:0/3732106371 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c00e6c0 con 0x55d1c067c3c0 2024-04-26T23:08:16.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.624+0000 7f7b69d6b740 1 -- 172.21.15.40:0/3732106371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c067c3c0 msgr2=0x55d1c067c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.624+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/3732106371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c067c3c0 0x55d1c067c910 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c0043d0 tx=0x7f7b4c011000 comp rx=0 tx=0).stop 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.624+0000 7f7b69d6b740 1 -- 172.21.15.40:0/3732106371 shutdown_connections 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.624+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/3732106371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c0634800 0x55d1c067aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.624+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/3732106371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c067c3c0 0x55d1c067c910 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.624+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/3732106371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 0x55d1c065ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.624+0000 7f7b69d6b740 1 -- 172.21.15.40:0/3732106371 >> 172.21.15.40:0/3732106371 conn(0x55d1c0631130 msgr2=0x55d1c0679930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.624+0000 7f7b69d6b740 1 -- 172.21.15.40:0/3732106371 shutdown_connections 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.624+0000 7f7b69d6b740 1 -- 172.21.15.40:0/3732106371 wait complete. 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.625+0000 7f7b69d6b740 1 -- start start 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.625+0000 7f7b69d6b740 1 Processor -- start 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b69d6b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c065eb20 0x55d1c0673d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b69d6b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c067c3c0 0x55d1c06742c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b69d6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06776c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b69d6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d1c0598200 con 0x55d1c065ddc0 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b69d6b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d1c0592ee0 con 0x55d1c065eb20 2024-04-26T23:08:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b69d6b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d1c059fe70 con 0x55d1c067c3c0 2024-04-26T23:08:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b69d6b740 10 client.0 _ll_register_callbacks cb 0x55d1c0589270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b65000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06776c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b65000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06776c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54528/0 (socket says 172.21.15.40:54528) 2024-04-26T23:08:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b65000640 1 -- 172.21.15.40:0/745427444 learned_addr learned my addr 172.21.15.40:0/745427444 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b65000640 1 -- 172.21.15.40:0/745427444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c067c3c0 msgr2=0x55d1c06742c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b65000640 1 --2- 172.21.15.40:0/745427444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c067c3c0 0x55d1c06742c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b65000640 1 -- 172.21.15.40:0/745427444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c065eb20 msgr2=0x55d1c0673d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b65000640 1 --2- 172.21.15.40:0/745427444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c065eb20 0x55d1c0673d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.626+0000 7f7b65000640 1 -- 172.21.15.40:0/745427444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1c036f7d0 con 0x55d1c065ddc0 2024-04-26T23:08:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.627+0000 7f7b65000640 1 --2- 172.21.15.40:0/745427444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06776c0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f7b58001e30 tx=0x7f7b5800d5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6854c6a58a29cf9a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.627+0000 7f7b49600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.627+0000 7f7b49600640 1 -- 172.21.15.40:0/745427444 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b580072d0 con 0x55d1c065ddc0 2024-04-26T23:08:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.627+0000 7f7b49600640 1 -- 172.21.15.40:0/745427444 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f7b58007470 con 0x55d1c065ddc0 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.627+0000 7f7b69d6b740 1 -- 172.21.15.40:0/745427444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d1c05a17e0 con 0x55d1c065ddc0 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.627+0000 7f7b49600640 1 -- 172.21.15.40:0/745427444 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7f7b58007670 con 0x55d1c065ddc0 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.627+0000 7f7b69d6b740 10 client.8283 fetch_fsmap learned FSMap version 98 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.627+0000 7f7b69d6b740 1 -- 172.21.15.40:0/745427444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) -- 0x55d1c02a53f0 con 0x55d1c065ddc0 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.628+0000 7f7b49600640 1 -- 172.21.15.40:0/745427444 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d1c05a17e0 con 0x55d1c065ddc0 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.628+0000 7f7b49600640 1 -- 172.21.15.40:0/745427444 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 98) ==== 3546+0+0 (secure 0 0 0) 0x7f7b580047e0 con 0x55d1c065ddc0 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.628+0000 7f7b69d6b740 10 client.8283 fetch_fsmap finished waiting for FSMap version 98 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.628+0000 7f7b69d6b740 10 client.8283 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.628+0000 7f7b69d6b740 20 client.8283 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.628+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/745427444 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d1c0675210 0x55d1c06756c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.628+0000 7f7b69d6b740 4 client.8283 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.628+0000 7f7b69d6b740 1 -- 172.21.15.40:0/745427444 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1c059b650 con 0x55d1c0675210 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.628+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/745427444 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d1c0675210 0x55d1c06756c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.628+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/745427444 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d1c0675210 0x55d1c06756c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34f94629def67fc7 server_cookie=4d2cdf99c56584a1 in_seq=0 out_seq=0 2024-04-26T23:08:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.628+0000 7f7b49600640 10 client.8283 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:16.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.629+0000 7f7b49600640 1 -- 172.21.15.40:0/745427444 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d1c059b650 con 0x55d1c0675210 2024-04-26T23:08:16.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.629+0000 7f7b49600640 10 client.8283 handle_command_reply: tid=0 2024-04-26T23:08:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.629+0000 7f7b69d6b740 1 client.8283 shutdown 2024-04-26T23:08:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.629+0000 7f7b69d6b740 1 -- 172.21.15.40:0/745427444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 msgr2=0x55d1c06776c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.629+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/745427444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06776c0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f7b58001e30 tx=0x7f7b5800d5e0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.630+0000 7f7b69d6b740 1 -- 172.21.15.40:0/745427444 shutdown_connections 2024-04-26T23:08:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.630+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/745427444 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d1c0675210 0x55d1c06756c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.630+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/745427444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06776c0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.630+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/745427444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c067c3c0 0x55d1c06742c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.630+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/745427444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c065eb20 0x55d1c0673d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.630+0000 7f7b49600640 0 client.8283 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.630+0000 7f7b69d6b740 1 -- 172.21.15.40:0/745427444 >> 172.21.15.40:0/745427444 conn(0x55d1c0634800 msgr2=0x55d1c062a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.630+0000 7f7b69d6b740 1 -- 172.21.15.40:0/745427444 shutdown_connections 2024-04-26T23:08:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.630+0000 7f7b69d6b740 1 -- 172.21.15.40:0/745427444 wait complete. 2024-04-26T23:08:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.630+0000 7f7b69d6b740 20 client.8283 trim_cache size 0 max 16384 2024-04-26T23:08:16.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.646+0000 7f7b69d6b740 1 Processor -- start 2024-04-26T23:08:16.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.646+0000 7f7b69d6b740 1 -- start start 2024-04-26T23:08:16.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.646+0000 7f7b69d6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06ea720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b69d6b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 0x55d1c0670de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b69d6b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c067c3c0 0x55d1c0671320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b69d6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7b601fd810 con 0x55d1c065ddc0 2024-04-26T23:08:16.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b69d6b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6015c4f0 con 0x55d1c067c3c0 2024-04-26T23:08:16.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b69d6b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7b6015cbf0 con 0x55d1c065eb20 2024-04-26T23:08:16.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06ea720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06ea720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54540/0 (socket says 172.21.15.40:54540) 2024-04-26T23:08:16.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b5fe00640 1 -- 172.21.15.40:0/2616952530 learned_addr learned my addr 172.21.15.40:0/2616952530 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:16.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b5f400640 1 --2- 172.21.15.40:0/2616952530 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 0x55d1c0670de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b5fe00640 1 -- 172.21.15.40:0/2616952530 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 msgr2=0x55d1c0670de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/2616952530 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 0x55d1c0670de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b65000640 1 --2- 172.21.15.40:0/2616952530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c067c3c0 0x55d1c0671320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b5fe00640 1 -- 172.21.15.40:0/2616952530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c067c3c0 msgr2=0x55d1c0671320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.647+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/2616952530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c067c3c0 0x55d1c0671320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.648+0000 7f7b5fe00640 1 -- 172.21.15.40:0/2616952530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1c02650f0 con 0x55d1c065ddc0 2024-04-26T23:08:16.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.648+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/2616952530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06ea720 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f7b500ab330 tx=0x7f7b500ab200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3866e3eef728c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.648+0000 7f7b4be00640 1 -- 172.21.15.40:0/2616952530 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b500aa070 con 0x55d1c065ddc0 2024-04-26T23:08:16.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.648+0000 7f7b4be00640 1 -- 172.21.15.40:0/2616952530 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b50093100 con 0x55d1c065ddc0 2024-04-26T23:08:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.648+0000 7f7b4be00640 1 -- 172.21.15.40:0/2616952530 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b50049070 con 0x55d1c065ddc0 2024-04-26T23:08:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.648+0000 7f7b69d6b740 1 -- 172.21.15.40:0/2616952530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 msgr2=0x55d1c06ea720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.648+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/2616952530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06ea720 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f7b500ab330 tx=0x7f7b500ab200 comp rx=0 tx=0).stop 2024-04-26T23:08:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.649+0000 7f7b69d6b740 1 -- 172.21.15.40:0/2616952530 shutdown_connections 2024-04-26T23:08:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.649+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/2616952530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c067c3c0 0x55d1c0671320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.649+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/2616952530 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 0x55d1c0670de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.649+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/2616952530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06ea720 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.649+0000 7f7b69d6b740 1 -- 172.21.15.40:0/2616952530 >> 172.21.15.40:0/2616952530 conn(0x55d1c0634800 msgr2=0x55d1c0634be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.649+0000 7f7b69d6b740 1 -- 172.21.15.40:0/2616952530 shutdown_connections 2024-04-26T23:08:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.649+0000 7f7b69d6b740 1 -- 172.21.15.40:0/2616952530 wait complete. 2024-04-26T23:08:16.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.649+0000 7f7b69d6b740 1 -- start start 2024-04-26T23:08:16.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.650+0000 7f7b69d6b740 1 Processor -- start 2024-04-26T23:08:16.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.650+0000 7f7b69d6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06617f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.650+0000 7f7b69d6b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 0x55d1c0661d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.650+0000 7f7b69d6b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c067c3c0 0x55d1c0673c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.650+0000 7f7b69d6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d1c05a0fd0 con 0x55d1c065ddc0 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.650+0000 7f7b69d6b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7b600cf590 con 0x55d1c067c3c0 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.650+0000 7f7b69d6b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d1c069c6b0 con 0x55d1c065eb20 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.651+0000 7f7b69d6b740 10 client.0 _ll_register_callbacks cb 0x7f7b58007c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.651+0000 7f7b5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06617f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.651+0000 7f7b5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06617f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54550/0 (socket says 172.21.15.40:54550) 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.651+0000 7f7b5fe00640 1 -- 172.21.15.40:0/1095371723 learned_addr learned my addr 172.21.15.40:0/1095371723 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.651+0000 7f7b5fe00640 1 -- 172.21.15.40:0/1095371723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 msgr2=0x55d1c0661d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.651+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/1095371723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 0x55d1c0661d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.651+0000 7f7b5fe00640 1 -- 172.21.15.40:0/1095371723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c067c3c0 msgr2=0x55d1c0673c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.651+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/1095371723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c067c3c0 0x55d1c0673c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.651+0000 7f7b5fe00640 1 -- 172.21.15.40:0/1095371723 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1c055ba30 con 0x55d1c065ddc0 2024-04-26T23:08:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.651+0000 7f7b5fe00640 1 --2- 172.21.15.40:0/1095371723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06617f0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f7b5002a6c0 tx=0x7f7b58001720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb41bbbd6449acf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.651+0000 7f7b49600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b49600640 1 -- 172.21.15.40:0/1095371723 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b500aa040 con 0x55d1c065ddc0 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b49600640 1 -- 172.21.15.40:0/1095371723 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f7b50078e30 con 0x55d1c065ddc0 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b69d6b740 1 -- 172.21.15.40:0/1095371723 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b50093100 con 0x55d1c065ddc0 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b49600640 1 -- 172.21.15.40:0/1095371723 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7f7b5007be60 con 0x55d1c065ddc0 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b69d6b740 10 client.8289 fetch_fsmap learned FSMap version 98 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b69d6b740 1 -- 172.21.15.40:0/1095371723 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) -- 0x55d1c06326d0 con 0x55d1c065ddc0 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b49600640 1 -- 172.21.15.40:0/1095371723 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b50093100 con 0x55d1c065ddc0 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b49600640 1 -- 172.21.15.40:0/1095371723 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 98) ==== 3546+0+0 (secure 0 0 0) 0x7f7b500771a0 con 0x55d1c065ddc0 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b69d6b740 10 client.8289 fetch_fsmap finished waiting for FSMap version 98 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b69d6b740 10 client.8289 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b69d6b740 20 client.8289 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/1095371723 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d1c066f970 0x55d1c06741a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b69d6b740 4 client.8289 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:08:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.652+0000 7f7b69d6b740 1 -- 172.21.15.40:0/1095371723 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d1c065f420 con 0x55d1c066f970 2024-04-26T23:08:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.653+0000 7f7b5f400640 1 --2- 172.21.15.40:0/1095371723 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d1c066f970 0x55d1c06741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.653+0000 7f7b5f400640 1 --2- 172.21.15.40:0/1095371723 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d1c066f970 0x55d1c06741a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb6aab21426b7036 server_cookie=33beeae2276243b4 in_seq=0 out_seq=0 2024-04-26T23:08:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.653+0000 7f7b49600640 10 client.8289 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.678+0000 7f7b49600640 1 -- 172.21.15.40:0/1095371723 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d1c065f420 con 0x55d1c066f970 2024-04-26T23:08:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.678+0000 7f7b49600640 10 client.8289 handle_command_reply: tid=0 2024-04-26T23:08:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.678+0000 7f7b69d6b740 1 client.8289 shutdown 2024-04-26T23:08:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.678+0000 7f7b69d6b740 1 -- 172.21.15.40:0/1095371723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 msgr2=0x55d1c06617f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.678+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/1095371723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06617f0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f7b5002a6c0 tx=0x7f7b58001720 comp rx=0 tx=0).stop 2024-04-26T23:08:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.679+0000 7f7b69d6b740 1 -- 172.21.15.40:0/1095371723 shutdown_connections 2024-04-26T23:08:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.679+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/1095371723 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55d1c066f970 0x55d1c06741a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.679+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/1095371723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1c067c3c0 0x55d1c0673c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.679+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/1095371723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1c065eb20 0x55d1c0661d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.679+0000 7f7b69d6b740 1 --2- 172.21.15.40:0/1095371723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1c065ddc0 0x55d1c06617f0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.679+0000 7f7b49600640 0 client.8289 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.679+0000 7f7b69d6b740 1 -- 172.21.15.40:0/1095371723 >> 172.21.15.40:0/1095371723 conn(0x55d1c0634800 msgr2=0x55d1c0675a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.679+0000 7f7b69d6b740 1 -- 172.21.15.40:0/1095371723 shutdown_connections 2024-04-26T23:08:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.679+0000 7f7b69d6b740 1 -- 172.21.15.40:0/1095371723 wait complete. 2024-04-26T23:08:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.679+0000 7f7b69d6b740 20 client.8289 trim_cache size 0 max 16384 2024-04-26T23:08:16.681 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:16.681 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:08:16.681 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:08:16.681 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.680+0000 7f7b65a00640 1 -- 172.21.15.40:0/2694982639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7b4006c920 msgr2=0x7f7b4006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.680+0000 7f7b65a00640 1 --2- 172.21.15.40:0/2694982639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7b4006c920 0x7f7b4006ede0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c006fd0 tx=0x7f7b4c008040 comp rx=0 tx=0).stop 2024-04-26T23:08:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.680+0000 7f7b65a00640 1 -- 172.21.15.40:0/2694982639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b60071ef0 msgr2=0x7f7b60141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.680+0000 7f7b65a00640 1 --2- 172.21.15.40:0/2694982639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b60071ef0 0x7f7b60141510 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f7b50000f80 tx=0x7f7b50004a40 comp rx=0 tx=0).stop 2024-04-26T23:08:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.680+0000 7f7b65a00640 1 -- 172.21.15.40:0/2694982639 shutdown_connections 2024-04-26T23:08:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.680+0000 7f7b65a00640 1 --2- 172.21.15.40:0/2694982639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7b4006c920 0x7f7b4006ede0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.680+0000 7f7b65a00640 1 --2- 172.21.15.40:0/2694982639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b6015bc70 0x7f7b6015e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.680+0000 7f7b65a00640 1 --2- 172.21.15.40:0/2694982639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b60072440 0x7f7b60141dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.680+0000 7f7b65a00640 1 --2- 172.21.15.40:0/2694982639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b60071ef0 0x7f7b60141510 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.680+0000 7f7b65a00640 1 -- 172.21.15.40:0/2694982639 >> 172.21.15.40:0/2694982639 conn(0x7f7b600b2720 msgr2=0x7f7b600b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.680+0000 7f7b65a00640 1 -- 172.21.15.40:0/2694982639 shutdown_connections 2024-04-26T23:08:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.680+0000 7f7b65a00640 1 -- 172.21.15.40:0/2694982639 wait complete. 2024-04-26T23:08:16.699 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:1 flush journal 2024-04-26T23:08:16.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.896+0000 7f0689600640 1 Processor -- start 2024-04-26T23:08:16.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.896+0000 7f0689600640 1 -- start start 2024-04-26T23:08:16.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.897+0000 7f0689600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f06840b6470 0x7f06840cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.897+0000 7f0689600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06840d5d30 con 0x7f068415f680 2024-04-26T23:08:16.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.897+0000 7f0689600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06840d5a70 con 0x7f06840d1770 2024-04-26T23:08:16.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.897+0000 7f0689600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f06840d3760 con 0x7f06840b6470 2024-04-26T23:08:16.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.898+0000 7f067be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f06840b6470 0x7f06840cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.898+0000 7f0688c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f068415f680 0x7f06840cff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60582/0 (socket says 172.21.15.40:60582) 2024-04-26T23:08:16.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.898+0000 7f0688c00640 1 -- 172.21.15.40:0/2555923947 learned_addr learned my addr 172.21.15.40:0/2555923947 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:16.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.898+0000 7f067be00640 1 --2- 172.21.15.40:0/2555923947 >> v2:172.21.15.40:3300/0 conn(0x7f06840b6470 0x7f06840cf9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54560/0 (socket says 172.21.15.40:54560) 2024-04-26T23:08:16.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.898+0000 7f067be00640 1 -- 172.21.15.40:0/2555923947 >> v1:172.21.15.103:6789/0 conn(0x7f06840d1770 legacy=0x7f068415ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.898+0000 7f067be00640 1 -- 172.21.15.40:0/2555923947 >> v1:172.21.15.40:6789/0 conn(0x7f068415f680 legacy=0x7f06840cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.898+0000 7f067be00640 1 -- 172.21.15.40:0/2555923947 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06840b5b00 con 0x7f06840b6470 2024-04-26T23:08:16.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.898+0000 7f0683400640 1 -- 172.21.15.40:0/2555923947 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2516490279 0 0) 0x7f06840d5d30 con 0x7f068415f680 2024-04-26T23:08:16.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.899+0000 7f067be00640 1 --2- 172.21.15.40:0/2555923947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840b6470 0x7f06840cf9e0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f066c0097a0 tx=0x7f066c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e800faa990ad26f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.900+0000 7f0683400640 1 -- 172.21.15.40:0/2555923947 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c037070 con 0x7f06840b6470 2024-04-26T23:08:16.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.900+0000 7f0683400640 1 -- 172.21.15.40:0/2555923947 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f066c032440 con 0x7f06840b6470 2024-04-26T23:08:16.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.900+0000 7f0683400640 1 -- 172.21.15.40:0/2555923947 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c032810 con 0x7f06840b6470 2024-04-26T23:08:16.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.900+0000 7f0689600640 1 -- 172.21.15.40:0/2555923947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840b6470 msgr2=0x7f06840cf9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.900+0000 7f0689600640 1 --2- 172.21.15.40:0/2555923947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840b6470 0x7f06840cf9e0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f066c0097a0 tx=0x7f066c02a230 comp rx=0 tx=0).stop 2024-04-26T23:08:16.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.900+0000 7f0689600640 1 -- 172.21.15.40:0/2555923947 shutdown_connections 2024-04-26T23:08:16.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.900+0000 7f0689600640 1 --2- 172.21.15.40:0/2555923947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840b6470 0x7f06840cf9e0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.900+0000 7f0689600640 1 -- 172.21.15.40:0/2555923947 >> 172.21.15.40:0/2555923947 conn(0x7f06840d4dd0 msgr2=0x7f06840d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:16.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.900+0000 7f0689600640 1 -- 172.21.15.40:0/2555923947 shutdown_connections 2024-04-26T23:08:16.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.900+0000 7f0689600640 1 -- 172.21.15.40:0/2555923947 wait complete. 2024-04-26T23:08:16.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.902+0000 7f0689600640 1 Processor -- start 2024-04-26T23:08:16.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.902+0000 7f0689600640 1 -- start start 2024-04-26T23:08:16.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.903+0000 7f0689600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06840b6470 0x7f06841f5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.903+0000 7f0689600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840d1770 0x7f06840ce160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.903+0000 7f0689600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f068415f680 0x7f06840ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.903+0000 7f0689600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f06840d3d70 con 0x7f06840d1770 2024-04-26T23:08:16.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.903+0000 7f0689600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f06840d3bf0 con 0x7f06840b6470 2024-04-26T23:08:16.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.903+0000 7f0689600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f06840d1b50 con 0x7f068415f680 2024-04-26T23:08:16.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.903+0000 7f067be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840d1770 0x7f06840ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.904+0000 7f067be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840d1770 0x7f06840ce160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54576/0 (socket says 172.21.15.40:54576) 2024-04-26T23:08:16.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.904+0000 7f067be00640 1 -- 172.21.15.40:0/2506709315 learned_addr learned my addr 172.21.15.40:0/2506709315 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:16.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.904+0000 7f067be00640 1 -- 172.21.15.40:0/2506709315 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f068415f680 msgr2=0x7f06840ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.904+0000 7f067be00640 1 --2- 172.21.15.40:0/2506709315 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f068415f680 0x7f06840ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.904+0000 7f067be00640 1 -- 172.21.15.40:0/2506709315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06840b6470 msgr2=0x7f06841f5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.904+0000 7f067be00640 1 --2- 172.21.15.40:0/2506709315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06840b6470 0x7f06841f5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.904+0000 7f067be00640 1 -- 172.21.15.40:0/2506709315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06840b8960 con 0x7f06840d1770 2024-04-26T23:08:16.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.904+0000 7f067be00640 1 --2- 172.21.15.40:0/2506709315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840d1770 0x7f06840ce160 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f066c000c00 tx=0x7f066c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.905+0000 7f0680c00640 1 -- 172.21.15.40:0/2506709315 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c037040 con 0x7f06840d1770 2024-04-26T23:08:16.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.905+0000 7f0680c00640 1 -- 172.21.15.40:0/2506709315 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f066c042070 con 0x7f06840d1770 2024-04-26T23:08:16.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.905+0000 7f0680c00640 1 -- 172.21.15.40:0/2506709315 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c030a70 con 0x7f06840d1770 2024-04-26T23:08:16.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.905+0000 7f0689600640 1 -- 172.21.15.40:0/2506709315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06840b7ba0 con 0x7f06840d1770 2024-04-26T23:08:16.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.905+0000 7f0689600640 1 -- 172.21.15.40:0/2506709315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06840b8160 con 0x7f06840d1770 2024-04-26T23:08:16.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.907+0000 7f0680c00640 1 -- 172.21.15.40:0/2506709315 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f066c030c10 con 0x7f06840d1770 2024-04-26T23:08:16.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.908+0000 7f0680c00640 1 --2- 172.21.15.40:0/2506709315 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f066006c8f0 0x7f066006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.908+0000 7f0680c00640 1 -- 172.21.15.40:0/2506709315 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f066c0ac6a0 con 0x7f06840d1770 2024-04-26T23:08:16.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.909+0000 7f0683e00640 1 --2- 172.21.15.40:0/2506709315 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f066006c8f0 0x7f066006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.909+0000 7f068dbc6740 1 Processor -- start 2024-04-26T23:08:16.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.909+0000 7f068dbc6740 1 -- start start 2024-04-26T23:08:16.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.910+0000 7f0683e00640 1 --2- 172.21.15.40:0/2506709315 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f066006c8f0 0x7f066006edb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0674000f60 tx=0x7f06740079b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.910+0000 7f068dbc6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f53710 0x55e128f53af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.910+0000 7f068dbc6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f70fb0 0x55e128f71500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.910+0000 7f068dbc6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f293f0 0x55e128f6f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.910+0000 7f068dbc6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e128e39030 con 0x55e128f293f0 2024-04-26T23:08:16.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.910+0000 7f068dbc6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e128e547f0 con 0x55e128f70fb0 2024-04-26T23:08:16.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.910+0000 7f068dbc6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e128e0f7e0 con 0x55e128f53710 2024-04-26T23:08:16.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.910+0000 7f0688c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f293f0 0x55e128f6f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.910+0000 7f0688c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f293f0 0x55e128f6f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54580/0 (socket says 172.21.15.40:54580) 2024-04-26T23:08:16.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.910+0000 7f0688c00640 1 -- 172.21.15.40:0/2841745201 learned_addr learned my addr 172.21.15.40:0/2841745201 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:16.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.911+0000 7f0683e00640 1 --2- 172.21.15.40:0/2841745201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f53710 0x55e128f53af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.911+0000 7f0688c00640 1 -- 172.21.15.40:0/2841745201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f53710 msgr2=0x55e128f53af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.911+0000 7f0688c00640 1 --2- 172.21.15.40:0/2841745201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f53710 0x55e128f53af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.911+0000 7f0688c00640 1 -- 172.21.15.40:0/2841745201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f70fb0 msgr2=0x55e128f71500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.911+0000 7f0688c00640 1 --2- 172.21.15.40:0/2841745201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f70fb0 0x55e128f71500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.911+0000 7f0688c00640 1 -- 172.21.15.40:0/2841745201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e128e1b050 con 0x55e128f293f0 2024-04-26T23:08:16.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.911+0000 7f0688c00640 1 --2- 172.21.15.40:0/2841745201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f293f0 0x55e128f6f670 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f067c001cd0 tx=0x7f067c002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b465bc5e3692a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.911+0000 7f0673e00640 1 -- 172.21.15.40:0/2841745201 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c00c840 con 0x55e128f293f0 2024-04-26T23:08:16.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.911+0000 7f0673e00640 1 -- 172.21.15.40:0/2841745201 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f067c00ca00 con 0x55e128f293f0 2024-04-26T23:08:16.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.911+0000 7f0673e00640 1 -- 172.21.15.40:0/2841745201 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c00cdf0 con 0x55e128f293f0 2024-04-26T23:08:16.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.912+0000 7f068dbc6740 1 -- 172.21.15.40:0/2841745201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f293f0 msgr2=0x55e128f6f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.912+0000 7f068dbc6740 1 --2- 172.21.15.40:0/2841745201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f293f0 0x55e128f6f670 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f067c001cd0 tx=0x7f067c002190 comp rx=0 tx=0).stop 2024-04-26T23:08:16.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.912+0000 7f068dbc6740 1 -- 172.21.15.40:0/2841745201 shutdown_connections 2024-04-26T23:08:16.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.912+0000 7f068dbc6740 1 --2- 172.21.15.40:0/2841745201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f293f0 0x55e128f6f670 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.912+0000 7f068dbc6740 1 --2- 172.21.15.40:0/2841745201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f70fb0 0x55e128f71500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.912+0000 7f068dbc6740 1 --2- 172.21.15.40:0/2841745201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f53710 0x55e128f53af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.912+0000 7f068dbc6740 1 -- 172.21.15.40:0/2841745201 >> 172.21.15.40:0/2841745201 conn(0x55e128f25d20 msgr2=0x55e128f6e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:16.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.912+0000 7f068dbc6740 1 -- 172.21.15.40:0/2841745201 shutdown_connections 2024-04-26T23:08:16.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.912+0000 7f068dbc6740 1 -- 172.21.15.40:0/2841745201 wait complete. 2024-04-26T23:08:16.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.913+0000 7f068dbc6740 1 -- start start 2024-04-26T23:08:16.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.913+0000 7f068dbc6740 1 Processor -- start 2024-04-26T23:08:16.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.914+0000 7f068dbc6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f53710 0x55e128f44f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.914+0000 7f068dbc6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f70fb0 0x55e128f668c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.914+0000 7f068dbc6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f7e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.914+0000 7f068dbc6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e128e91200 con 0x55e128f70fb0 2024-04-26T23:08:16.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.914+0000 7f068dbc6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e128e8bee0 con 0x55e128f573f0 2024-04-26T23:08:16.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.914+0000 7f068dbc6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e128e98e70 con 0x55e128f53710 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.914+0000 7f068dbc6740 10 client.0 _ll_register_callbacks cb 0x55e128e82270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.914+0000 7f067be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f70fb0 0x55e128f668c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.914+0000 7f067be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f70fb0 0x55e128f668c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54588/0 (socket says 172.21.15.40:54588) 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.914+0000 7f067be00640 1 -- 172.21.15.40:0/4229374812 learned_addr learned my addr 172.21.15.40:0/4229374812 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.915+0000 7f0688c00640 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f7e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.915+0000 7f067be00640 1 -- 172.21.15.40:0/4229374812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f53710 msgr2=0x55e128f44f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.915+0000 7f0683e00640 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f53710 0x55e128f44f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.915+0000 7f067be00640 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f53710 0x55e128f44f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.915+0000 7f067be00640 1 -- 172.21.15.40:0/4229374812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 msgr2=0x55e128f7e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.915+0000 7f067be00640 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f7e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.915+0000 7f067be00640 1 -- 172.21.15.40:0/4229374812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e128c687d0 con 0x55e128f70fb0 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.915+0000 7f0688c00640 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f7e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.915+0000 7f067be00640 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f70fb0 0x55e128f668c0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f066c082980 tx=0x7f066c079cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df03d53ded3dd32b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.915+0000 7f0670c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.915+0000 7f0670c00640 1 -- 172.21.15.40:0/4229374812 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c0aae60 con 0x55e128f70fb0 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.915+0000 7f0670c00640 1 -- 172.21.15.40:0/4229374812 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f066c07a730 con 0x55e128f70fb0 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.915+0000 7f068dbc6740 1 -- 172.21.15.40:0/4229374812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f067c00ca00 con 0x55e128f70fb0 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.916+0000 7f0670c00640 1 -- 172.21.15.40:0/4229374812 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7f066c07a9d0 con 0x55e128f70fb0 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.916+0000 7f068dbc6740 10 client.8304 fetch_fsmap learned FSMap version 98 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.916+0000 7f068dbc6740 1 -- 172.21.15.40:0/4229374812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) -- 0x55e128b9e3f0 con 0x55e128f70fb0 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.916+0000 7f0670c00640 1 -- 172.21.15.40:0/4229374812 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f067c00ca00 con 0x55e128f70fb0 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.916+0000 7f0670c00640 1 -- 172.21.15.40:0/4229374812 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 98) ==== 3546+0+0 (secure 0 0 0) 0x7f066c08ece0 con 0x55e128f70fb0 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.916+0000 7f068dbc6740 10 client.8304 fetch_fsmap finished waiting for FSMap version 98 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.916+0000 7f068dbc6740 10 client.8304 resolve_mds: resolved 6:1 to role '6:1' aka daemon mds.a 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.916+0000 7f068dbc6740 20 client.8304 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.916+0000 7f068dbc6740 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55e128f272c0 0x55e128f27730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.916+0000 7f068dbc6740 4 client.8304 mds_command: new command op to 8031 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.916+0000 7f068dbc6740 1 -- 172.21.15.40:0/4229374812 --> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e128e9a7e0 con 0x55e128f272c0 2024-04-26T23:08:16.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.916+0000 7f0683e00640 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55e128f272c0 0x55e128f27730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.917+0000 7f0683e00640 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55e128f272c0 0x55e128f27730 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=df5e5cca5ac5e7a5 server_cookie=cf201968cc4b784d in_seq=0 out_seq=0 2024-04-26T23:08:16.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.917+0000 7f0670c00640 10 client.8304 ms_handle_connect on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:16.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f0670c00640 1 -- 172.21.15.40:0/4229374812 <== mds.1 v2:172.21.15.40:6832/1811310743 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e128e9a7e0 con 0x55e128f272c0 2024-04-26T23:08:16.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f0670c00640 10 client.8304 handle_command_reply: tid=0 2024-04-26T23:08:16.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f068dbc6740 1 client.8304 shutdown 2024-04-26T23:08:16.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f068dbc6740 1 -- 172.21.15.40:0/4229374812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f70fb0 msgr2=0x55e128f668c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f068dbc6740 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f70fb0 0x55e128f668c0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f066c082980 tx=0x7f066c079cf0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f068dbc6740 1 -- 172.21.15.40:0/4229374812 shutdown_connections 2024-04-26T23:08:16.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f068dbc6740 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55e128f272c0 0x55e128f27730 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f068dbc6740 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f7e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f068dbc6740 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f70fb0 0x55e128f668c0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f068dbc6740 1 --2- 172.21.15.40:0/4229374812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f53710 0x55e128f44f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f0670c00640 0 client.8304 ms_handle_reset on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f068dbc6740 1 -- 172.21.15.40:0/4229374812 >> 172.21.15.40:0/4229374812 conn(0x55e128f293f0 msgr2=0x55e128f52790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f068dbc6740 1 -- 172.21.15.40:0/4229374812 shutdown_connections 2024-04-26T23:08:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f068dbc6740 1 -- 172.21.15.40:0/4229374812 wait complete. 2024-04-26T23:08:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.918+0000 7f068dbc6740 20 client.8304 trim_cache size 0 max 16384 2024-04-26T23:08:16.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.935+0000 7f068dbc6740 1 Processor -- start 2024-04-26T23:08:16.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.935+0000 7f068dbc6740 1 -- start start 2024-04-26T23:08:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.935+0000 7f068dbc6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 0x55e128f567d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.935+0000 7f068dbc6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f56d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.935+0000 7f068dbc6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f70fb0 0x55e128f22960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.935+0000 7f068dbc6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0684201ca0 con 0x55e128f53710 2024-04-26T23:08:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.935+0000 7f068dbc6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0684200e70 con 0x55e128f573f0 2024-04-26T23:08:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.935+0000 7f068dbc6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0684201570 con 0x55e128f70fb0 2024-04-26T23:08:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.936+0000 7f0683e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 0x55e128f567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.936+0000 7f0683e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 0x55e128f567d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54594/0 (socket says 172.21.15.40:54594) 2024-04-26T23:08:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.936+0000 7f0683e00640 1 -- 172.21.15.40:0/1134820768 learned_addr learned my addr 172.21.15.40:0/1134820768 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.936+0000 7f067be00640 1 --2- 172.21.15.40:0/1134820768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f56d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.936+0000 7f0688c00640 1 --2- 172.21.15.40:0/1134820768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f70fb0 0x55e128f22960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.936+0000 7f0683e00640 1 -- 172.21.15.40:0/1134820768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f70fb0 msgr2=0x55e128f22960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.936+0000 7f0683e00640 1 --2- 172.21.15.40:0/1134820768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f70fb0 0x55e128f22960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.936+0000 7f0683e00640 1 -- 172.21.15.40:0/1134820768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 msgr2=0x55e128f56d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.936+0000 7f0683e00640 1 --2- 172.21.15.40:0/1134820768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f56d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.936+0000 7f0683e00640 1 -- 172.21.15.40:0/1134820768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e128b5e0f0 con 0x55e128f53710 2024-04-26T23:08:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.936+0000 7f067be00640 1 --2- 172.21.15.40:0/1134820768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f56d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.937+0000 7f0683e00640 1 --2- 172.21.15.40:0/1134820768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 0x55e128f567d0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f0674014710 tx=0x7f0674012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=323f7ff504d7195 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.937+0000 7f0673400640 1 -- 172.21.15.40:0/1134820768 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674007220 con 0x55e128f53710 2024-04-26T23:08:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.937+0000 7f0673400640 1 -- 172.21.15.40:0/1134820768 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06740073c0 con 0x55e128f53710 2024-04-26T23:08:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.937+0000 7f0673400640 1 -- 172.21.15.40:0/1134820768 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674015e40 con 0x55e128f53710 2024-04-26T23:08:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.937+0000 7f068dbc6740 1 -- 172.21.15.40:0/1134820768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 msgr2=0x55e128f567d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.937+0000 7f068dbc6740 1 --2- 172.21.15.40:0/1134820768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 0x55e128f567d0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f0674014710 tx=0x7f0674012000 comp rx=0 tx=0).stop 2024-04-26T23:08:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.937+0000 7f068dbc6740 1 -- 172.21.15.40:0/1134820768 shutdown_connections 2024-04-26T23:08:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.937+0000 7f068dbc6740 1 --2- 172.21.15.40:0/1134820768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f70fb0 0x55e128f22960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.937+0000 7f068dbc6740 1 --2- 172.21.15.40:0/1134820768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f56d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.937+0000 7f068dbc6740 1 --2- 172.21.15.40:0/1134820768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 0x55e128f567d0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.937+0000 7f068dbc6740 1 -- 172.21.15.40:0/1134820768 >> 172.21.15.40:0/1134820768 conn(0x55e128f293f0 msgr2=0x55e128f1f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.937+0000 7f068dbc6740 1 -- 172.21.15.40:0/1134820768 shutdown_connections 2024-04-26T23:08:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.938+0000 7f068dbc6740 1 -- 172.21.15.40:0/1134820768 wait complete. 2024-04-26T23:08:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.938+0000 7f068dbc6740 1 -- start start 2024-04-26T23:08:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.938+0000 7f068dbc6740 1 Processor -- start 2024-04-26T23:08:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.939+0000 7f068dbc6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 0x55e128fe5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.939+0000 7f068dbc6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f22960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.939+0000 7f068dbc6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f70fb0 0x55e128f22ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.939+0000 7f068dbc6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e128e99fd0 con 0x55e128f53710 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.939+0000 7f068dbc6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f068415c350 con 0x55e128f573f0 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.939+0000 7f068dbc6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e128f27c80 con 0x55e128f70fb0 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.939+0000 7f068dbc6740 10 client.0 _ll_register_callbacks cb 0x7f066c030f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.940+0000 7f0683e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 0x55e128fe5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.940+0000 7f0683e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 0x55e128fe5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54596/0 (socket says 172.21.15.40:54596) 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.940+0000 7f0683e00640 1 -- 172.21.15.40:0/705905328 learned_addr learned my addr 172.21.15.40:0/705905328 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.940+0000 7f067be00640 1 --2- 172.21.15.40:0/705905328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f22960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.940+0000 7f0688c00640 1 --2- 172.21.15.40:0/705905328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f70fb0 0x55e128f22ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.940+0000 7f0683e00640 1 -- 172.21.15.40:0/705905328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f70fb0 msgr2=0x55e128f22ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.940+0000 7f0683e00640 1 --2- 172.21.15.40:0/705905328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f70fb0 0x55e128f22ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.940+0000 7f0683e00640 1 -- 172.21.15.40:0/705905328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 msgr2=0x55e128f22960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.940+0000 7f0683e00640 1 --2- 172.21.15.40:0/705905328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f22960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.940+0000 7f0683e00640 1 -- 172.21.15.40:0/705905328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e128e54a30 con 0x55e128f53710 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.940+0000 7f0683e00640 1 --2- 172.21.15.40:0/705905328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 0x55e128fe5140 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f0674001e30 tx=0x7f0674001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f65be4fe3f269d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.940+0000 7f0670c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.941+0000 7f0670c00640 1 -- 172.21.15.40:0/705905328 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06740037f0 con 0x55e128f53710 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.941+0000 7f0670c00640 1 -- 172.21.15.40:0/705905328 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f0674003990 con 0x55e128f53710 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.941+0000 7f068dbc6740 1 -- 172.21.15.40:0/705905328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06740073c0 con 0x55e128f53710 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.941+0000 7f0670c00640 1 -- 172.21.15.40:0/705905328 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7f0674003b90 con 0x55e128f53710 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.941+0000 7f068dbc6740 10 client.8310 fetch_fsmap learned FSMap version 98 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.941+0000 7f068dbc6740 1 -- 172.21.15.40:0/705905328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) -- 0x55e128f44640 con 0x55e128f53710 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.941+0000 7f0670c00640 1 -- 172.21.15.40:0/705905328 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06740073c0 con 0x55e128f53710 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.941+0000 7f0670c00640 1 -- 172.21.15.40:0/705905328 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 98) ==== 3546+0+0 (secure 0 0 0) 0x7f0674022020 con 0x55e128f53710 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.941+0000 7f068dbc6740 10 client.8310 fetch_fsmap finished waiting for FSMap version 98 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.942+0000 7f068dbc6740 10 client.8310 resolve_mds: resolved 6:1 to role '6:1' aka daemon mds.a 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.942+0000 7f068dbc6740 20 client.8310 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.942+0000 7f068dbc6740 1 --2- 172.21.15.40:0/705905328 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55e128f28970 0x55e128fe7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.942+0000 7f068dbc6740 4 client.8310 mds_command: new command op to 8031 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:08:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.942+0000 7f068dbc6740 1 -- 172.21.15.40:0/705905328 --> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e128e94650 con 0x55e128f28970 2024-04-26T23:08:16.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.942+0000 7f067be00640 1 --2- 172.21.15.40:0/705905328 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55e128f28970 0x55e128fe7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:16.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.942+0000 7f067be00640 1 --2- 172.21.15.40:0/705905328 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55e128f28970 0x55e128fe7d20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=98f0ae5bdecde304 server_cookie=2d6ec0e6cebaa9ad in_seq=0 out_seq=0 2024-04-26T23:08:16.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.942+0000 7f0670c00640 10 client.8310 ms_handle_connect on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f0670c00640 1 -- 172.21.15.40:0/705905328 <== mds.1 v2:172.21.15.40:6832/1811310743 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e128e94650 con 0x55e128f28970 2024-04-26T23:08:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f0670c00640 10 client.8310 handle_command_reply: tid=0 2024-04-26T23:08:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f068dbc6740 1 client.8310 shutdown 2024-04-26T23:08:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f068dbc6740 1 -- 172.21.15.40:0/705905328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 msgr2=0x55e128fe5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f068dbc6740 1 --2- 172.21.15.40:0/705905328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 0x55e128fe5140 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f0674001e30 tx=0x7f0674001190 comp rx=0 tx=0).stop 2024-04-26T23:08:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f068dbc6740 1 -- 172.21.15.40:0/705905328 shutdown_connections 2024-04-26T23:08:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f068dbc6740 1 --2- 172.21.15.40:0/705905328 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x55e128f28970 0x55e128fe7d20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f068dbc6740 1 --2- 172.21.15.40:0/705905328 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e128f70fb0 0x55e128f22ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f068dbc6740 1 --2- 172.21.15.40:0/705905328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e128f573f0 0x55e128f22960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f068dbc6740 1 --2- 172.21.15.40:0/705905328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e128f53710 0x55e128fe5140 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f0670c00640 0 client.8310 ms_handle_reset on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f068dbc6740 1 -- 172.21.15.40:0/705905328 >> 172.21.15.40:0/705905328 conn(0x55e128f293f0 msgr2=0x55e128f91e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f068dbc6740 1 -- 172.21.15.40:0/705905328 shutdown_connections 2024-04-26T23:08:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.963+0000 7f068dbc6740 1 -- 172.21.15.40:0/705905328 wait complete. 2024-04-26T23:08:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.964+0000 7f068dbc6740 20 client.8310 trim_cache size 0 max 16384 2024-04-26T23:08:16.965 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:16.965 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:08:16.965 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:08:16.965 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.964+0000 7f0689600640 1 -- 172.21.15.40:0/2506709315 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f066006c8f0 msgr2=0x7f066006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.964+0000 7f0689600640 1 --2- 172.21.15.40:0/2506709315 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f066006c8f0 0x7f066006edb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0674000f60 tx=0x7f06740079b0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.964+0000 7f0689600640 1 -- 172.21.15.40:0/2506709315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840d1770 msgr2=0x7f06840ce160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.964+0000 7f0689600640 1 --2- 172.21.15.40:0/2506709315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840d1770 0x7f06840ce160 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f066c000c00 tx=0x7f066c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.964+0000 7f0689600640 1 -- 172.21.15.40:0/2506709315 shutdown_connections 2024-04-26T23:08:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.964+0000 7f0689600640 1 --2- 172.21.15.40:0/2506709315 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f066006c8f0 0x7f066006edb0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.964+0000 7f0689600640 1 --2- 172.21.15.40:0/2506709315 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f068415f680 0x7f06840ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.964+0000 7f0689600640 1 --2- 172.21.15.40:0/2506709315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840d1770 0x7f06840ce160 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.964+0000 7f0689600640 1 --2- 172.21.15.40:0/2506709315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06840b6470 0x7f06841f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.964+0000 7f0689600640 1 -- 172.21.15.40:0/2506709315 >> 172.21.15.40:0/2506709315 conn(0x7f06840d4dd0 msgr2=0x7f068406e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.965+0000 7f0689600640 1 -- 172.21.15.40:0/2506709315 shutdown_connections 2024-04-26T23:08:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:16.965+0000 7f0689600640 1 -- 172.21.15.40:0/2506709315 wait complete. 2024-04-26T23:08:16.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:2 flush journal 2024-04-26T23:08:17.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.184+0000 7f5134c00640 1 Processor -- start 2024-04-26T23:08:17.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.184+0000 7f5134c00640 1 -- start start 2024-04-26T23:08:17.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.186+0000 7f5134c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5130160a50 0x7f5130073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.186+0000 7f5134c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f51300cfaf0 0x7f5130160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.186+0000 7f5134c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f513006dd40 0x7f5130072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.186+0000 7f5134c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f5130071ce0 con 0x7f513006dd40 2024-04-26T23:08:17.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.186+0000 7f5134c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f51300722a0 con 0x7f51300cfaf0 2024-04-26T23:08:17.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.186+0000 7f5134c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5130073f90 con 0x7f5130160a50 2024-04-26T23:08:17.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.186+0000 7f512fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f513006dd40 0x7f5130072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.187+0000 7f512fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f513006dd40 0x7f5130072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54612/0 (socket says 172.21.15.40:54612) 2024-04-26T23:08:17.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.187+0000 7f512fe00640 1 -- 172.21.15.40:0/3096075654 learned_addr learned my addr 172.21.15.40:0/3096075654 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:17.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.187+0000 7f512f400640 1 --2- 172.21.15.40:0/3096075654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5130160a50 0x7f5130073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.187+0000 7f5126a00640 1 --2- 172.21.15.40:0/3096075654 >> v2:172.21.15.103:3300/0 conn(0x7f51300cfaf0 0x7f5130160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.187+0000 7f512fe00640 1 -- 172.21.15.40:0/3096075654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5130160a50 msgr2=0x7f5130073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:17.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.187+0000 7f512fe00640 1 --2- 172.21.15.40:0/3096075654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5130160a50 0x7f5130073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.187+0000 7f512fe00640 1 -- 172.21.15.40:0/3096075654 >> v2:172.21.15.103:3300/0 conn(0x7f51300cfaf0 msgr2=0x7f5130160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:17.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.187+0000 7f512fe00640 1 --2- 172.21.15.40:0/3096075654 >> v2:172.21.15.103:3300/0 conn(0x7f51300cfaf0 0x7f5130160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.187+0000 7f512fe00640 1 -- 172.21.15.40:0/3096075654 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5130161880 con 0x7f513006dd40 2024-04-26T23:08:17.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.187+0000 7f5126a00640 1 --2- 172.21.15.40:0/3096075654 >> v2:172.21.15.103:3300/0 conn(0x7f51300cfaf0 0x7f5130160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:17.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.188+0000 7f512fe00640 1 --2- 172.21.15.40:0/3096075654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f513006dd40 0x7f5130072730 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f5130074700 tx=0x7f512802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd2cce89dcdfaca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:17.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.189+0000 7f512ea00640 1 -- 172.21.15.40:0/3096075654 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128036070 con 0x7f513006dd40 2024-04-26T23:08:17.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.189+0000 7f512ea00640 1 -- 172.21.15.40:0/3096075654 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5128031600 con 0x7f513006dd40 2024-04-26T23:08:17.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.189+0000 7f512ea00640 1 -- 172.21.15.40:0/3096075654 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51280319d0 con 0x7f513006dd40 2024-04-26T23:08:17.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.189+0000 7f5134c00640 1 -- 172.21.15.40:0/3096075654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f513006dd40 msgr2=0x7f5130072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:17.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.189+0000 7f5134c00640 1 --2- 172.21.15.40:0/3096075654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f513006dd40 0x7f5130072730 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f5130074700 tx=0x7f512802a6a0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.189+0000 7f5134c00640 1 -- 172.21.15.40:0/3096075654 shutdown_connections 2024-04-26T23:08:17.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.189+0000 7f5134c00640 1 --2- 172.21.15.40:0/3096075654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f513006dd40 0x7f5130072730 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.189+0000 7f5134c00640 1 --2- 172.21.15.40:0/3096075654 >> v2:172.21.15.103:3300/0 conn(0x7f51300cfaf0 0x7f5130160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.189+0000 7f5134c00640 1 --2- 172.21.15.40:0/3096075654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5130160a50 0x7f5130073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.189+0000 7f5134c00640 1 -- 172.21.15.40:0/3096075654 >> 172.21.15.40:0/3096075654 conn(0x7f51300b2720 msgr2=0x7f51300b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:17.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.190+0000 7f5134c00640 1 -- 172.21.15.40:0/3096075654 shutdown_connections 2024-04-26T23:08:17.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.190+0000 7f5134c00640 1 -- 172.21.15.40:0/3096075654 wait complete. 2024-04-26T23:08:17.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.191+0000 7f5134c00640 1 Processor -- start 2024-04-26T23:08:17.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.191+0000 7f5134c00640 1 -- start start 2024-04-26T23:08:17.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.192+0000 7f5134c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f513006dd40 0x7f513013e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.192+0000 7f5134c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5130160a50 0x7f51301f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.192+0000 7f5134c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51301f41d0 0x7f51301439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.192+0000 7f5134c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f51300d0260 con 0x7f5130160a50 2024-04-26T23:08:17.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.192+0000 7f5134c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f51300cff80 con 0x7f51301f41d0 2024-04-26T23:08:17.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.192+0000 7f5134c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5130071f30 con 0x7f513006dd40 2024-04-26T23:08:17.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.193+0000 7f5126a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5130160a50 0x7f51301f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.193+0000 7f5126a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5130160a50 0x7f51301f3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54620/0 (socket says 172.21.15.40:54620) 2024-04-26T23:08:17.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.193+0000 7f5126a00640 1 -- 172.21.15.40:0/3322664619 learned_addr learned my addr 172.21.15.40:0/3322664619 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:17.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.193+0000 7f512f400640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f513006dd40 0x7f513013e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.193+0000 7f512fe00640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51301f41d0 0x7f51301439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.193+0000 7f5126a00640 1 -- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f513006dd40 msgr2=0x7f513013e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:17.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.193+0000 7f5126a00640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f513006dd40 0x7f513013e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.193+0000 7f5126a00640 1 -- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51301f41d0 msgr2=0x7f51301439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:17.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.193+0000 7f5126a00640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51301f41d0 0x7f51301439b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.193+0000 7f5126a00640 1 -- 172.21.15.40:0/3322664619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5130142910 con 0x7f5130160a50 2024-04-26T23:08:17.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.193+0000 7f512fe00640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51301f41d0 0x7f51301439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:17.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.194+0000 7f5126a00640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5130160a50 0x7f51301f3c90 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f511800c7c0 tx=0x7f511800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:17.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.194+0000 7f5127e00640 1 -- 172.21.15.40:0/3322664619 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5118015070 con 0x7f5130160a50 2024-04-26T23:08:17.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.194+0000 7f5127e00640 1 -- 172.21.15.40:0/3322664619 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51180079d0 con 0x7f5130160a50 2024-04-26T23:08:17.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.194+0000 7f5127e00640 1 -- 172.21.15.40:0/3322664619 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5118007da0 con 0x7f5130160a50 2024-04-26T23:08:17.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.194+0000 7f5134c00640 1 -- 172.21.15.40:0/3322664619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5130142bf0 con 0x7f5130160a50 2024-04-26T23:08:17.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.194+0000 7f5134c00640 1 -- 172.21.15.40:0/3322664619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51301431b0 con 0x7f5130160a50 2024-04-26T23:08:17.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.196+0000 7f5127e00640 1 -- 172.21.15.40:0/3322664619 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5118018070 con 0x7f5130160a50 2024-04-26T23:08:17.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.198+0000 7f5127e00640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f510c06c920 0x7f510c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.198+0000 7f5127e00640 1 -- 172.21.15.40:0/3322664619 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f511808bb00 con 0x7f5130160a50 2024-04-26T23:08:17.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.199+0000 7f512f400640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f510c06c920 0x7f510c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.199+0000 7f5138fe6740 1 Processor -- start 2024-04-26T23:08:17.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.199+0000 7f5138fe6740 1 -- start start 2024-04-26T23:08:17.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.200+0000 7f512f400640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f510c06c920 0x7f510c06ede0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5120006fd0 tx=0x7f5120008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:17.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.200+0000 7f5138fe6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 0x563d5d638af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.200+0000 7f5138fe6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d655fb0 0x563d5d656500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.200+0000 7f5138fe6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d60e3f0 0x563d5d654670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.200+0000 7f5138fe6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563d5d51e030 con 0x563d5d638710 2024-04-26T23:08:17.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.200+0000 7f5138fe6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563d5d5397f0 con 0x563d5d60e3f0 2024-04-26T23:08:17.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.200+0000 7f5138fe6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x563d5d4f47e0 con 0x563d5d655fb0 2024-04-26T23:08:17.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.200+0000 7f512f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 0x563d5d638af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.200+0000 7f512f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 0x563d5d638af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54630/0 (socket says 172.21.15.40:54630) 2024-04-26T23:08:17.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.200+0000 7f512f400640 1 -- 172.21.15.40:0/2412894030 learned_addr learned my addr 172.21.15.40:0/2412894030 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:17.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.201+0000 7f512f400640 1 -- 172.21.15.40:0/2412894030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d655fb0 msgr2=0x563d5d656500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:17.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.201+0000 7f512f400640 1 --2- 172.21.15.40:0/2412894030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d655fb0 0x563d5d656500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.201+0000 7f512f400640 1 -- 172.21.15.40:0/2412894030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d60e3f0 msgr2=0x563d5d654670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:17.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.201+0000 7f512f400640 1 --2- 172.21.15.40:0/2412894030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d60e3f0 0x563d5d654670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.201+0000 7f512f400640 1 -- 172.21.15.40:0/2412894030 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d5d500050 con 0x563d5d638710 2024-04-26T23:08:17.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.201+0000 7f512f400640 1 --2- 172.21.15.40:0/2412894030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 0x563d5d638af0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f512000f9c0 tx=0x7f5120012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c5efe948fbc974c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:17.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.201+0000 7f511f400640 1 -- 172.21.15.40:0/2412894030 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512001b050 con 0x563d5d638710 2024-04-26T23:08:17.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.201+0000 7f511f400640 1 -- 172.21.15.40:0/2412894030 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5120025070 con 0x563d5d638710 2024-04-26T23:08:17.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.201+0000 7f511f400640 1 -- 172.21.15.40:0/2412894030 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120014de0 con 0x563d5d638710 2024-04-26T23:08:17.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.202+0000 7f5138fe6740 1 -- 172.21.15.40:0/2412894030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 msgr2=0x563d5d638af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:17.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.202+0000 7f5138fe6740 1 --2- 172.21.15.40:0/2412894030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 0x563d5d638af0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f512000f9c0 tx=0x7f5120012000 comp rx=0 tx=0).stop 2024-04-26T23:08:17.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.202+0000 7f5138fe6740 1 -- 172.21.15.40:0/2412894030 shutdown_connections 2024-04-26T23:08:17.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.202+0000 7f5138fe6740 1 --2- 172.21.15.40:0/2412894030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d60e3f0 0x563d5d654670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.202+0000 7f5138fe6740 1 --2- 172.21.15.40:0/2412894030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d655fb0 0x563d5d656500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.202+0000 7f5138fe6740 1 --2- 172.21.15.40:0/2412894030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 0x563d5d638af0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.202+0000 7f5138fe6740 1 -- 172.21.15.40:0/2412894030 >> 172.21.15.40:0/2412894030 conn(0x563d5d60ad20 msgr2=0x563d5d653520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:17.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.202+0000 7f5138fe6740 1 -- 172.21.15.40:0/2412894030 shutdown_connections 2024-04-26T23:08:17.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.202+0000 7f5138fe6740 1 -- 172.21.15.40:0/2412894030 wait complete. 2024-04-26T23:08:17.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.203+0000 7f5138fe6740 1 -- start start 2024-04-26T23:08:17.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.203+0000 7f5138fe6740 1 Processor -- start 2024-04-26T23:08:17.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.204+0000 7f5138fe6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d638710 0x563d5d637c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.204+0000 7f5138fe6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 0x563d5d64be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.204+0000 7f5138fe6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 0x563d5d64ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.204+0000 7f5138fe6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563d5d576200 con 0x563d5d64e9e0 2024-04-26T23:08:17.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.204+0000 7f5138fe6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563d5d570ee0 con 0x563d5d655fb0 2024-04-26T23:08:17.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.204+0000 7f5138fe6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x563d5d57de70 con 0x563d5d638710 2024-04-26T23:08:17.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.204+0000 7f5138fe6740 10 client.0 _ll_register_callbacks cb 0x563d5d567270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:17.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.204+0000 7f512fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 0x563d5d64ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.205+0000 7f512fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 0x563d5d64ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54634/0 (socket says 172.21.15.40:54634) 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.205+0000 7f512fe00640 1 -- 172.21.15.40:0/3588393972 learned_addr learned my addr 172.21.15.40:0/3588393972 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.205+0000 7f512f400640 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d638710 0x563d5d637c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.205+0000 7f5126a00640 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 0x563d5d64be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.205+0000 7f512fe00640 1 -- 172.21.15.40:0/3588393972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d638710 msgr2=0x563d5d637c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.205+0000 7f512fe00640 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d638710 0x563d5d637c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.205+0000 7f512fe00640 1 -- 172.21.15.40:0/3588393972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 msgr2=0x563d5d64be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.205+0000 7f512fe00640 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 0x563d5d64be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.205+0000 7f512fe00640 1 -- 172.21.15.40:0/3588393972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d5d34d7d0 con 0x563d5d64e9e0 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.205+0000 7f512f400640 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d638710 0x563d5d637c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.206+0000 7f512fe00640 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 0x563d5d64ee40 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f512802a6e0 tx=0x7f5128004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c5e142e57870649 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.206+0000 7f5113e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.206+0000 7f5113e00640 1 -- 172.21.15.40:0/3588393972 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128041020 con 0x563d5d64e9e0 2024-04-26T23:08:17.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.206+0000 7f5138fe6740 1 -- 172.21.15.40:0/3588393972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5120025070 con 0x563d5d64e9e0 2024-04-26T23:08:17.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.206+0000 7f5113e00640 1 -- 172.21.15.40:0/3588393972 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f5128031dc0 con 0x563d5d64e9e0 2024-04-26T23:08:17.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.206+0000 7f5113e00640 1 -- 172.21.15.40:0/3588393972 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7f512804b050 con 0x563d5d64e9e0 2024-04-26T23:08:17.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.206+0000 7f5138fe6740 10 client.8322 fetch_fsmap learned FSMap version 98 2024-04-26T23:08:17.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.206+0000 7f5138fe6740 1 -- 172.21.15.40:0/3588393972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) -- 0x563d5d2833f0 con 0x563d5d64e9e0 2024-04-26T23:08:17.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.207+0000 7f5113e00640 1 -- 172.21.15.40:0/3588393972 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5120025070 con 0x563d5d64e9e0 2024-04-26T23:08:17.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.207+0000 7f5113e00640 1 -- 172.21.15.40:0/3588393972 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 98) ==== 3546+0+0 (secure 0 0 0) 0x7f512804e020 con 0x563d5d64e9e0 2024-04-26T23:08:17.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.207+0000 7f5138fe6740 10 client.8322 fetch_fsmap finished waiting for FSMap version 98 2024-04-26T23:08:17.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.207+0000 7f5138fe6740 10 client.8322 resolve_mds: resolved 6:2 to role '6:2' aka daemon mds.d 2024-04-26T23:08:17.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.207+0000 7f5138fe6740 20 client.8322 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:17.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.207+0000 7f5138fe6740 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x563d5d64fcb0 0x563d5d6500c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.207+0000 7f5138fe6740 4 client.8322 mds_command: new command op to 6253 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:17.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.207+0000 7f5138fe6740 1 -- 172.21.15.40:0/3588393972 --> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d5d57f7e0 con 0x563d5d64fcb0 2024-04-26T23:08:17.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.208+0000 7f512f400640 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x563d5d64fcb0 0x563d5d6500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.209+0000 7f512f400640 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x563d5d64fcb0 0x563d5d6500c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6a6a7569bcdca265 server_cookie=e7a3523f084fc2f3 in_seq=0 out_seq=0 2024-04-26T23:08:17.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.209+0000 7f5113e00640 10 client.8322 ms_handle_connect on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:17.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.210+0000 7f5113e00640 1 -- 172.21.15.40:0/3588393972 <== mds.2 v2:172.21.15.103:6836/2728939257 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563d5d57f7e0 con 0x563d5d64fcb0 2024-04-26T23:08:17.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.210+0000 7f5113e00640 10 client.8322 handle_command_reply: tid=0 2024-04-26T23:08:17.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.211+0000 7f5138fe6740 1 client.8322 shutdown 2024-04-26T23:08:17.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.211+0000 7f5138fe6740 1 -- 172.21.15.40:0/3588393972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 msgr2=0x563d5d64ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:17.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.211+0000 7f5138fe6740 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 0x563d5d64ee40 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f512802a6e0 tx=0x7f5128004a40 comp rx=0 tx=0).stop 2024-04-26T23:08:17.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.211+0000 7f5138fe6740 1 -- 172.21.15.40:0/3588393972 shutdown_connections 2024-04-26T23:08:17.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.211+0000 7f5138fe6740 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 0x563d5d64ee40 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.211+0000 7f5138fe6740 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x563d5d64fcb0 0x563d5d6500c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.211+0000 7f5138fe6740 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 0x563d5d64be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.211+0000 7f5138fe6740 1 --2- 172.21.15.40:0/3588393972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d638710 0x563d5d637c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.211+0000 7f5138fe6740 1 -- 172.21.15.40:0/3588393972 >> 172.21.15.40:0/3588393972 conn(0x563d5d60e3f0 msgr2=0x563d5d6081f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:17.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.211+0000 7f5113e00640 0 client.8322 ms_handle_reset on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:17.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.211+0000 7f5138fe6740 1 -- 172.21.15.40:0/3588393972 shutdown_connections 2024-04-26T23:08:17.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.211+0000 7f5138fe6740 1 -- 172.21.15.40:0/3588393972 wait complete. 2024-04-26T23:08:17.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.211+0000 7f5138fe6740 20 client.8322 trim_cache size 0 max 16384 2024-04-26T23:08:17.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.228+0000 7f5138fe6740 1 Processor -- start 2024-04-26T23:08:17.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.228+0000 7f5138fe6740 1 -- start start 2024-04-26T23:08:17.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.229+0000 7f5138fe6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 0x563d5d6c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.229+0000 7f5138fe6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d64e9e0 0x563d5d650860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.229+0000 7f5138fe6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 0x563d5d64dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.229+0000 7f5138fe6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f51301ffb80 con 0x563d5d638710 2024-04-26T23:08:17.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.229+0000 7f5138fe6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f51301fed50 con 0x563d5d655fb0 2024-04-26T23:08:17.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.229+0000 7f5138fe6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f51301ff450 con 0x563d5d64e9e0 2024-04-26T23:08:17.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.229+0000 7f512f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 0x563d5d6c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.229+0000 7f512f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 0x563d5d6c5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54638/0 (socket says 172.21.15.40:54638) 2024-04-26T23:08:17.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.229+0000 7f512f400640 1 -- 172.21.15.40:0/89539778 learned_addr learned my addr 172.21.15.40:0/89539778 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:17.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.230+0000 7f512f400640 1 -- 172.21.15.40:0/89539778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d64e9e0 msgr2=0x563d5d650860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:17.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.230+0000 7f512f400640 1 --2- 172.21.15.40:0/89539778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d64e9e0 0x563d5d650860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.230+0000 7f512f400640 1 -- 172.21.15.40:0/89539778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 msgr2=0x563d5d64dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:17.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.230+0000 7f512f400640 1 --2- 172.21.15.40:0/89539778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 0x563d5d64dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.230+0000 7f512f400640 1 -- 172.21.15.40:0/89539778 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d5d2430f0 con 0x563d5d638710 2024-04-26T23:08:17.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.230+0000 7f512f400640 1 --2- 172.21.15.40:0/89539778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 0x563d5d6c5010 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f5120006e80 tx=0x7f5120014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32feb0ef922a6b56 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:17.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.230+0000 7f511ea00640 1 -- 172.21.15.40:0/89539778 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512001b050 con 0x563d5d638710 2024-04-26T23:08:17.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.230+0000 7f511ea00640 1 -- 172.21.15.40:0/89539778 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5120025040 con 0x563d5d638710 2024-04-26T23:08:17.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.230+0000 7f511ea00640 1 -- 172.21.15.40:0/89539778 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120033b50 con 0x563d5d638710 2024-04-26T23:08:17.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.231+0000 7f5138fe6740 1 -- 172.21.15.40:0/89539778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 msgr2=0x563d5d6c5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:17.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.231+0000 7f5138fe6740 1 --2- 172.21.15.40:0/89539778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 0x563d5d6c5010 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f5120006e80 tx=0x7f5120014000 comp rx=0 tx=0).stop 2024-04-26T23:08:17.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.231+0000 7f5138fe6740 1 -- 172.21.15.40:0/89539778 shutdown_connections 2024-04-26T23:08:17.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.231+0000 7f5138fe6740 1 --2- 172.21.15.40:0/89539778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 0x563d5d64dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.231+0000 7f5138fe6740 1 --2- 172.21.15.40:0/89539778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d64e9e0 0x563d5d650860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.231+0000 7f5138fe6740 1 --2- 172.21.15.40:0/89539778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d638710 0x563d5d6c5010 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.231+0000 7f5138fe6740 1 -- 172.21.15.40:0/89539778 >> 172.21.15.40:0/89539778 conn(0x563d5d60e3f0 msgr2=0x563d5d6316b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:17.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.231+0000 7f5138fe6740 1 -- 172.21.15.40:0/89539778 shutdown_connections 2024-04-26T23:08:17.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.231+0000 7f5138fe6740 1 -- 172.21.15.40:0/89539778 wait complete. 2024-04-26T23:08:17.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.232+0000 7f5138fe6740 1 -- start start 2024-04-26T23:08:17.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.232+0000 7f5138fe6740 1 Processor -- start 2024-04-26T23:08:17.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.233+0000 7f5138fe6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d638710 0x563d5d654e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.233+0000 7f5138fe6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 0x563d5d6c12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.233+0000 7f5138fe6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 0x563d5d648eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.233+0000 7f5138fe6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563d5d57efd0 con 0x563d5d64e9e0 2024-04-26T23:08:17.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.233+0000 7f5138fe6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5130140990 con 0x563d5d655fb0 2024-04-26T23:08:17.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.233+0000 7f5138fe6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x563d5d650350 con 0x563d5d638710 2024-04-26T23:08:17.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.233+0000 7f5138fe6740 10 client.0 _ll_register_callbacks cb 0x7f5128034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:17.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.233+0000 7f5126a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 0x563d5d6c12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.233+0000 7f5126a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 0x563d5d6c12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54654/0 (socket says 172.21.15.40:54654) 2024-04-26T23:08:17.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.233+0000 7f5126a00640 1 -- 172.21.15.40:0/2253475771 learned_addr learned my addr 172.21.15.40:0/2253475771 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:17.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.234+0000 7f512f400640 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d638710 0x563d5d654e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.234+0000 7f512fe00640 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 0x563d5d648eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.234+0000 7f5126a00640 1 -- 172.21.15.40:0/2253475771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d638710 msgr2=0x563d5d654e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:17.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.234+0000 7f5126a00640 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d638710 0x563d5d654e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.234+0000 7f5126a00640 1 -- 172.21.15.40:0/2253475771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 msgr2=0x563d5d648eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:17.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.234+0000 7f5126a00640 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 0x563d5d648eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.234+0000 7f5126a00640 1 -- 172.21.15.40:0/2253475771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d5d539a30 con 0x563d5d64e9e0 2024-04-26T23:08:17.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.234+0000 7f512f400640 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d638710 0x563d5d654e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:17.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.234+0000 7f5126a00640 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 0x563d5d6c12f0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f5118007410 tx=0x7f5118059330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91cc383225cc358c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:17.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.234+0000 7f5113e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:17.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.234+0000 7f5113e00640 1 -- 172.21.15.40:0/2253475771 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511801f050 con 0x563d5d64e9e0 2024-04-26T23:08:17.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.234+0000 7f5113e00640 1 -- 172.21.15.40:0/2253475771 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f5118058d80 con 0x563d5d64e9e0 2024-04-26T23:08:17.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.234+0000 7f5138fe6740 1 -- 172.21.15.40:0/2253475771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5120025040 con 0x563d5d64e9e0 2024-04-26T23:08:17.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.235+0000 7f5113e00640 1 -- 172.21.15.40:0/2253475771 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (secure 0 0 0) 0x7f5118059730 con 0x563d5d64e9e0 2024-04-26T23:08:17.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.235+0000 7f5138fe6740 10 client.8328 fetch_fsmap learned FSMap version 98 2024-04-26T23:08:17.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.235+0000 7f5138fe6740 1 -- 172.21.15.40:0/2253475771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) -- 0x563d5d649450 con 0x563d5d64e9e0 2024-04-26T23:08:17.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.235+0000 7f5113e00640 1 -- 172.21.15.40:0/2253475771 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5120025040 con 0x563d5d64e9e0 2024-04-26T23:08:17.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.235+0000 7f5113e00640 1 -- 172.21.15.40:0/2253475771 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 98) ==== 3546+0+0 (secure 0 0 0) 0x7f5118055480 con 0x563d5d64e9e0 2024-04-26T23:08:17.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.235+0000 7f5138fe6740 10 client.8328 fetch_fsmap finished waiting for FSMap version 98 2024-04-26T23:08:17.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.235+0000 7f5138fe6740 10 client.8328 resolve_mds: resolved 6:2 to role '6:2' aka daemon mds.d 2024-04-26T23:08:17.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.235+0000 7f5138fe6740 20 client.8328 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:17.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.235+0000 7f5138fe6740 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x563d5d676090 0x563d5d634440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:17.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.235+0000 7f5138fe6740 4 client.8328 mds_command: new command op to 6253 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:08:17.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.235+0000 7f5138fe6740 1 -- 172.21.15.40:0/2253475771 --> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563d5d651e60 con 0x563d5d676090 2024-04-26T23:08:17.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.236+0000 7f512f400640 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x563d5d676090 0x563d5d634440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:17.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.237+0000 7f512f400640 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x563d5d676090 0x563d5d634440 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eb394387cc1f9f15 server_cookie=d43fa9a1ee38dc6 in_seq=0 out_seq=0 2024-04-26T23:08:17.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.237+0000 7f5113e00640 10 client.8328 ms_handle_connect on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:17.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.257+0000 7f5113e00640 1 -- 172.21.15.40:0/2253475771 <== mds.2 v2:172.21.15.103:6836/2728939257 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563d5d651e60 con 0x563d5d676090 2024-04-26T23:08:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.257+0000 7f5113e00640 10 client.8328 handle_command_reply: tid=0 2024-04-26T23:08:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.257+0000 7f5138fe6740 1 client.8328 shutdown 2024-04-26T23:08:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.257+0000 7f5138fe6740 1 -- 172.21.15.40:0/2253475771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 msgr2=0x563d5d6c12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.257+0000 7f5138fe6740 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 0x563d5d6c12f0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f5118007410 tx=0x7f5118059330 comp rx=0 tx=0).stop 2024-04-26T23:08:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.257+0000 7f5138fe6740 1 -- 172.21.15.40:0/2253475771 shutdown_connections 2024-04-26T23:08:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.257+0000 7f5138fe6740 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x563d5d676090 0x563d5d634440 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.257+0000 7f5138fe6740 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d5d655fb0 0x563d5d648eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.257+0000 7f5138fe6740 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d5d64e9e0 0x563d5d6c12f0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.257+0000 7f5138fe6740 1 --2- 172.21.15.40:0/2253475771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563d5d638710 0x563d5d654e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.257+0000 7f5113e00640 0 client.8328 ms_handle_reset on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.257+0000 7f5138fe6740 1 -- 172.21.15.40:0/2253475771 >> 172.21.15.40:0/2253475771 conn(0x563d5d60e3f0 msgr2=0x563d5d6507a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:17.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.257+0000 7f5138fe6740 1 -- 172.21.15.40:0/2253475771 shutdown_connections 2024-04-26T23:08:17.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.258+0000 7f5138fe6740 1 -- 172.21.15.40:0/2253475771 wait complete. 2024-04-26T23:08:17.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.258+0000 7f5138fe6740 20 client.8328 trim_cache size 0 max 16384 2024-04-26T23:08:17.259 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:17.259 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:08:17.259 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:08:17.259 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.259+0000 7f5134c00640 1 -- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f510c06c920 msgr2=0x7f510c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.259+0000 7f5134c00640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f510c06c920 0x7f510c06ede0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5120006fd0 tx=0x7f5120008040 comp rx=0 tx=0).stop 2024-04-26T23:08:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.259+0000 7f5134c00640 1 -- 172.21.15.40:0/3322664619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5130160a50 msgr2=0x7f51301f3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.259+0000 7f5134c00640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5130160a50 0x7f51301f3c90 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f511800c7c0 tx=0x7f511800cc80 comp rx=0 tx=0).stop 2024-04-26T23:08:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.259+0000 7f5134c00640 1 -- 172.21.15.40:0/3322664619 shutdown_connections 2024-04-26T23:08:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.259+0000 7f5134c00640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f510c06c920 0x7f510c06ede0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.259+0000 7f5134c00640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51301f41d0 0x7f51301439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.259+0000 7f5134c00640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5130160a50 0x7f51301f3c90 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.259+0000 7f5134c00640 1 --2- 172.21.15.40:0/3322664619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f513006dd40 0x7f513013e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.259+0000 7f5134c00640 1 -- 172.21.15.40:0/3322664619 >> 172.21.15.40:0/3322664619 conn(0x7f51300b2720 msgr2=0x7f513015eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.259+0000 7f5134c00640 1 -- 172.21.15.40:0/3322664619 shutdown_connections 2024-04-26T23:08:17.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17.259+0000 7f5134c00640 1 -- 172.21.15.40:0/3322664619 wait complete. 2024-04-26T23:08:17.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:08:17.278 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec find d1/d2/d3/) 2024-04-26T23:08:17.349 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/ 2024-04-26T23:08:17.349 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4 2024-04-26T23:08:17.349 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5 2024-04-26T23:08:17.349 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6 2024-04-26T23:08:17.349 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6/d7 2024-04-26T23:08:17.349 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6/d7/d8 2024-04-26T23:08:17.350 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:08:17.351 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:08:17.351 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:08:17.351 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:08:17.351 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/").st_ino) 2024-04-26T23:08:17.351 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:08:17.496 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627778 2024-04-26T23:08:17.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:08:17.511 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:08:17.511 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:08:17.511 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:08:17.511 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:08:17.511 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4").st_ino) 2024-04-26T23:08:17.511 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:08:17.699 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627779 2024-04-26T23:08:17.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:08:17.713 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:08:17.713 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:08:17.713 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:08:17.713 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:08:17.713 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5").st_ino) 2024-04-26T23:08:17.713 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:08:17.907 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627780 2024-04-26T23:08:17.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:08:17.924 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:08:17.924 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:08:17.924 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:08:17.924 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:08:17.924 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6").st_ino) 2024-04-26T23:08:17.924 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:08:18.121 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627781 2024-04-26T23:08:18.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:08:18.135 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:08:18.135 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:08:18.135 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:08:18.135 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:08:18.135 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6/d7").st_ino) 2024-04-26T23:08:18.135 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:08:18.331 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627782 2024-04-26T23:08:18.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:08:18.345 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:08:18.345 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:08:18.346 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:08:18.346 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:08:18.346 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6/d7/d8").st_ino) 2024-04-26T23:08:18.346 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:08:18.540 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627783 2024-04-26T23:08:18.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:08:18.554 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000002.00000000 parent 2024-04-26T23:08:18.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.716+0000 7fdb963cbf40 1 Processor -- start 2024-04-26T23:08:18.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.716+0000 7fdb963cbf40 1 -- start start 2024-04-26T23:08:18.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.717+0000 7fdb963cbf40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b4e89850a0 0x55b4e89559d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.717+0000 7fdb963cbf40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e8985480 0x55b4e8a239f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.717+0000 7fdb963cbf40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e899f290 0x55b4e8a247e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.717+0000 7fdb963cbf40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b4e899a880 con 0x55b4e8985480 2024-04-26T23:08:18.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.717+0000 7fdb963cbf40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b4e88fcd20 con 0x55b4e899f290 2024-04-26T23:08:18.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.717+0000 7fdb963cbf40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b4e899ab20 con 0x55b4e89850a0 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.717+0000 7fdb8ac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e8985480 0x55b4e8a239f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.717+0000 7fdb8ac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e8985480 0x55b4e8a239f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54682/0 (socket says 172.21.15.40:54682) 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.717+0000 7fdb8ac00640 1 -- 172.21.15.40:0/2916999731 learned_addr learned my addr 172.21.15.40:0/2916999731 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.718+0000 7fdb93600640 1 --2- 172.21.15.40:0/2916999731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b4e89850a0 0x55b4e89559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.718+0000 7fdb94000640 1 --2- 172.21.15.40:0/2916999731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e899f290 0x55b4e8a247e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.718+0000 7fdb8ac00640 1 -- 172.21.15.40:0/2916999731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b4e89850a0 msgr2=0x55b4e89559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.718+0000 7fdb8ac00640 1 --2- 172.21.15.40:0/2916999731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b4e89850a0 0x55b4e89559d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.718+0000 7fdb8ac00640 1 -- 172.21.15.40:0/2916999731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e899f290 msgr2=0x55b4e8a247e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.718+0000 7fdb8ac00640 1 --2- 172.21.15.40:0/2916999731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e899f290 0x55b4e8a247e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.718+0000 7fdb8ac00640 1 -- 172.21.15.40:0/2916999731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4e8955670 con 0x55b4e8985480 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.718+0000 7fdb94000640 1 --2- 172.21.15.40:0/2916999731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e899f290 0x55b4e8a247e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.718+0000 7fdb8ac00640 1 --2- 172.21.15.40:0/2916999731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e8985480 0x55b4e8a239f0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00a7a0 tx=0x7fdb7c00a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b530d52a5fe218 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.718+0000 7fdb92200640 1 -- 172.21.15.40:0/2916999731 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c019070 con 0x55b4e8985480 2024-04-26T23:08:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.718+0000 7fdb92200640 1 -- 172.21.15.40:0/2916999731 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb7c007600 con 0x55b4e8985480 2024-04-26T23:08:18.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.718+0000 7fdb92200640 1 -- 172.21.15.40:0/2916999731 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c0079d0 con 0x55b4e8985480 2024-04-26T23:08:18.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.719+0000 7fdb963cbf40 1 -- 172.21.15.40:0/2916999731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e8985480 msgr2=0x55b4e8a239f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.719+0000 7fdb963cbf40 1 --2- 172.21.15.40:0/2916999731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e8985480 0x55b4e8a239f0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00a7a0 tx=0x7fdb7c00a890 comp rx=0 tx=0).stop 2024-04-26T23:08:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.719+0000 7fdb963cbf40 1 -- 172.21.15.40:0/2916999731 shutdown_connections 2024-04-26T23:08:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.719+0000 7fdb963cbf40 1 --2- 172.21.15.40:0/2916999731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e899f290 0x55b4e8a247e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.719+0000 7fdb963cbf40 1 --2- 172.21.15.40:0/2916999731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e8985480 0x55b4e8a239f0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.719+0000 7fdb963cbf40 1 --2- 172.21.15.40:0/2916999731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b4e89850a0 0x55b4e89559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.719+0000 7fdb963cbf40 1 -- 172.21.15.40:0/2916999731 >> 172.21.15.40:0/2916999731 conn(0x55b4e8982c00 msgr2=0x55b4e899cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.719+0000 7fdb963cbf40 1 -- 172.21.15.40:0/2916999731 shutdown_connections 2024-04-26T23:08:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.720+0000 7fdb963cbf40 1 -- 172.21.15.40:0/2916999731 wait complete. 2024-04-26T23:08:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.720+0000 7fdb963cbf40 1 Processor -- start 2024-04-26T23:08:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.720+0000 7fdb963cbf40 1 -- start start 2024-04-26T23:08:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.721+0000 7fdb963cbf40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e89850a0 0x55b4e8a2f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.721+0000 7fdb963cbf40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b4e8985480 0x55b4e8954210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.721+0000 7fdb963cbf40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e899f290 0x55b4e8954750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.721+0000 7fdb963cbf40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b4e899d710 con 0x55b4e899f290 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.721+0000 7fdb963cbf40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b4e8a23f30 con 0x55b4e89850a0 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.721+0000 7fdb963cbf40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b4e8958310 con 0x55b4e8985480 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.722+0000 7fdb94000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e899f290 0x55b4e8954750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.722+0000 7fdb94000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e899f290 0x55b4e8954750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54696/0 (socket says 172.21.15.40:54696) 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.722+0000 7fdb94000640 1 -- 172.21.15.40:0/62335081 learned_addr learned my addr 172.21.15.40:0/62335081 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.722+0000 7fdb8ac00640 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b4e8985480 0x55b4e8954210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.722+0000 7fdb93600640 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e89850a0 0x55b4e8a2f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.722+0000 7fdb94000640 1 -- 172.21.15.40:0/62335081 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b4e8985480 msgr2=0x55b4e8954210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.722+0000 7fdb94000640 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b4e8985480 0x55b4e8954210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.722+0000 7fdb94000640 1 -- 172.21.15.40:0/62335081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e89850a0 msgr2=0x55b4e8a2f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.722+0000 7fdb94000640 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e89850a0 0x55b4e8a2f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.722+0000 7fdb94000640 1 -- 172.21.15.40:0/62335081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4e8955ae0 con 0x55b4e899f290 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.722+0000 7fdb8ac00640 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b4e8985480 0x55b4e8954210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.722+0000 7fdb93600640 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e89850a0 0x55b4e8a2f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.723+0000 7fdb94000640 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e899f290 0x55b4e8954750 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c00bfc0 tx=0x7fdb8c009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.723+0000 7fdb90e00640 1 -- 172.21.15.40:0/62335081 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c0067b0 con 0x55b4e899f290 2024-04-26T23:08:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.723+0000 7fdb90e00640 1 -- 172.21.15.40:0/62335081 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb8c006950 con 0x55b4e899f290 2024-04-26T23:08:18.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.723+0000 7fdb90e00640 1 -- 172.21.15.40:0/62335081 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c006d20 con 0x55b4e899f290 2024-04-26T23:08:18.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.723+0000 7fdb963cbf40 1 -- 172.21.15.40:0/62335081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b4e8955dc0 con 0x55b4e899f290 2024-04-26T23:08:18.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.723+0000 7fdb963cbf40 1 -- 172.21.15.40:0/62335081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4e8a23400 con 0x55b4e899f290 2024-04-26T23:08:18.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.726+0000 7fdb90e00640 1 -- 172.21.15.40:0/62335081 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb8c01d440 con 0x55b4e899f290 2024-04-26T23:08:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.726+0000 7fdb90e00640 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb7006b2d0 0x7fdb7006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.727+0000 7fdb90e00640 1 -- 172.21.15.40:0/62335081 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fdb8c08ac00 con 0x55b4e899f290 2024-04-26T23:08:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.727+0000 7fdb963cbf40 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55b4e8957560 0x55b4e8a286a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.727+0000 7fdb963cbf40 1 -- 172.21.15.40:0/62335081 --> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] -- osd_op(unknown.0.0:1 13.12 13:4bec3886:::10000000002.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55b4e8a20300 con 0x55b4e8957560 2024-04-26T23:08:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.727+0000 7fdb93600640 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb7006b2d0 0x7fdb7006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.727+0000 7fdb8ac00640 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55b4e8957560 0x55b4e8a286a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.728+0000 7fdb8ac00640 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55b4e8957560 0x55b4e8a286a0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:18.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.728+0000 7fdb93600640 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb7006b2d0 0x7fdb7006d790 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fdb84056000 tx=0x7fdb84007030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.731+0000 7fdb8ac00640 1 -- 172.21.15.40:0/62335081 <== osd.0 v2:172.21.15.40:6800/2897145808 1 ==== osd_op_reply(1 10000000002.00000000 [rmxattr] v90'8 uv8 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fdb7c007b30 con 0x55b4e8957560 2024-04-26T23:08:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.731+0000 7fdb963cbf40 1 -- 172.21.15.40:0/62335081 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55b4e8957560 msgr2=0x55b4e8a286a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.731+0000 7fdb963cbf40 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55b4e8957560 0x55b4e8a286a0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.731+0000 7fdb963cbf40 1 -- 172.21.15.40:0/62335081 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb7006b2d0 msgr2=0x7fdb7006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.731+0000 7fdb963cbf40 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdb7006b2d0 0x7fdb7006d790 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fdb84056000 tx=0x7fdb84007030 comp rx=0 tx=0).stop 2024-04-26T23:08:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.731+0000 7fdb963cbf40 1 -- 172.21.15.40:0/62335081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e899f290 msgr2=0x55b4e8954750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.731+0000 7fdb963cbf40 1 --2- 172.21.15.40:0/62335081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e899f290 0x55b4e8954750 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c00bfc0 tx=0x7fdb8c009800 comp rx=0 tx=0).stop 2024-04-26T23:08:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.731+0000 7fdb94000640 1 -- 172.21.15.40:0/62335081 reap_dead start 2024-04-26T23:08:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.731+0000 7fdb963cbf40 1 -- 172.21.15.40:0/62335081 shutdown_connections 2024-04-26T23:08:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.731+0000 7fdb963cbf40 1 -- 172.21.15.40:0/62335081 >> 172.21.15.40:0/62335081 conn(0x55b4e8982c00 msgr2=0x55b4e899cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.732+0000 7fdb963cbf40 1 -- 172.21.15.40:0/62335081 shutdown_connections 2024-04-26T23:08:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.732+0000 7fdb963cbf40 1 -- 172.21.15.40:0/62335081 wait complete. 2024-04-26T23:08:18.736 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000003.00000000 parent 2024-04-26T23:08:18.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.854+0000 7f3c6b40ef40 1 Processor -- start 2024-04-26T23:08:18.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.855+0000 7f3c6b40ef40 1 -- start start 2024-04-26T23:08:18.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.855+0000 7f3c6b40ef40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d40536fe90 0x55d405418760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.855+0000 7f3c6b40ef40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d405371070 0x55d405417b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.855+0000 7f3c6b40ef40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d40538a590 0x55d40536cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.855+0000 7f3c6b40ef40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d405388ef0 con 0x55d405371070 2024-04-26T23:08:18.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.855+0000 7f3c6b40ef40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d4052e5d20 con 0x55d40536fe90 2024-04-26T23:08:18.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.855+0000 7f3c6b40ef40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d405416030 con 0x55d40538a590 2024-04-26T23:08:18.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.856+0000 7f3c5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d405371070 0x55d405417b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.856+0000 7f3c69000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d40538a590 0x55d40536cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.856+0000 7f3c5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d405371070 0x55d405417b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54704/0 (socket says 172.21.15.40:54704) 2024-04-26T23:08:18.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.856+0000 7f3c5fe00640 1 -- 172.21.15.40:0/909664387 learned_addr learned my addr 172.21.15.40:0/909664387 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:18.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.856+0000 7f3c68600640 1 --2- 172.21.15.40:0/909664387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d40536fe90 0x55d405418760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.857+0000 7f3c5fe00640 1 -- 172.21.15.40:0/909664387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d40538a590 msgr2=0x55d40536cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:18.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.857+0000 7f3c5fe00640 1 --2- 172.21.15.40:0/909664387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d40538a590 0x55d40536cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.857+0000 7f3c5fe00640 1 -- 172.21.15.40:0/909664387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d40536fe90 msgr2=0x55d405418760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:18.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.857+0000 7f3c5fe00640 1 --2- 172.21.15.40:0/909664387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d40536fe90 0x55d405418760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.857+0000 7f3c5fe00640 1 -- 172.21.15.40:0/909664387 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d40538c410 con 0x55d405371070 2024-04-26T23:08:18.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.857+0000 7f3c5fe00640 1 --2- 172.21.15.40:0/909664387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d405371070 0x55d405417b20 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f3c50002de0 tx=0x7f3c5000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7ad6b5f767256 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:18.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.857+0000 7f3c67200640 1 -- 172.21.15.40:0/909664387 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5000f400 con 0x55d405371070 2024-04-26T23:08:18.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.857+0000 7f3c67200640 1 -- 172.21.15.40:0/909664387 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c50014070 con 0x55d405371070 2024-04-26T23:08:18.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.857+0000 7f3c67200640 1 -- 172.21.15.40:0/909664387 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5000f7e0 con 0x55d405371070 2024-04-26T23:08:18.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.858+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/909664387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d405371070 msgr2=0x55d405417b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.858+0000 7f3c6b40ef40 1 --2- 172.21.15.40:0/909664387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d405371070 0x55d405417b20 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f3c50002de0 tx=0x7f3c5000b5b0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.858+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/909664387 shutdown_connections 2024-04-26T23:08:18.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.858+0000 7f3c6b40ef40 1 --2- 172.21.15.40:0/909664387 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d40538a590 0x55d40536cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.858+0000 7f3c6b40ef40 1 --2- 172.21.15.40:0/909664387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d405371070 0x55d405417b20 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.858+0000 7f3c6b40ef40 1 --2- 172.21.15.40:0/909664387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d40536fe90 0x55d405418760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.858+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/909664387 >> 172.21.15.40:0/909664387 conn(0x55d40538bdf0 msgr2=0x55d40538b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.858+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/909664387 shutdown_connections 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.858+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/909664387 wait complete. 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.859+0000 7f3c6b40ef40 1 Processor -- start 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.859+0000 7f3c6b40ef40 1 -- start start 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.860+0000 7f3c6b40ef40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d40536fe90 0x55d4053f97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.860+0000 7f3c6b40ef40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d405371070 0x55d40540e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.860+0000 7f3c6b40ef40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d40538a590 0x55d40540e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.860+0000 7f3c6b40ef40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d40540f990 con 0x55d405371070 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.860+0000 7f3c6b40ef40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d405419d30 con 0x55d40536fe90 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.860+0000 7f3c6b40ef40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d40540f4e0 con 0x55d40538a590 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.860+0000 7f3c5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d405371070 0x55d40540e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.860+0000 7f3c5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d405371070 0x55d40540e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54712/0 (socket says 172.21.15.40:54712) 2024-04-26T23:08:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.860+0000 7f3c5fe00640 1 -- 172.21.15.40:0/3379677070 learned_addr learned my addr 172.21.15.40:0/3379677070 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.861+0000 7f3c69000640 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d40538a590 0x55d40540e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.861+0000 7f3c5fe00640 1 -- 172.21.15.40:0/3379677070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d40538a590 msgr2=0x55d40540e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.861+0000 7f3c5fe00640 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d40538a590 0x55d40540e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.861+0000 7f3c68600640 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d40536fe90 0x55d4053f97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.861+0000 7f3c5fe00640 1 -- 172.21.15.40:0/3379677070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d40536fe90 msgr2=0x55d4053f97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.861+0000 7f3c5fe00640 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d40536fe90 0x55d4053f97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.861+0000 7f3c5fe00640 1 -- 172.21.15.40:0/3379677070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4053f9d20 con 0x55d405371070 2024-04-26T23:08:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.861+0000 7f3c5fe00640 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d405371070 0x55d40540e370 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f3c50010000 tx=0x7f3c50001fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.861+0000 7f3c65e00640 1 -- 172.21.15.40:0/3379677070 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c50028070 con 0x55d405371070 2024-04-26T23:08:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.861+0000 7f3c65e00640 1 -- 172.21.15.40:0/3379677070 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c50019480 con 0x55d405371070 2024-04-26T23:08:18.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.861+0000 7f3c65e00640 1 -- 172.21.15.40:0/3379677070 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c50019870 con 0x55d405371070 2024-04-26T23:08:18.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.861+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/3379677070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d40540ac90 con 0x55d405371070 2024-04-26T23:08:18.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.862+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/3379677070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d40540b250 con 0x55d405371070 2024-04-26T23:08:18.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.864+0000 7f3c65e00640 1 -- 172.21.15.40:0/3379677070 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3c5002b070 con 0x55d405371070 2024-04-26T23:08:18.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.865+0000 7f3c65e00640 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c4406b2d0 0x7f3c4406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.865+0000 7f3c65e00640 1 -- 172.21.15.40:0/3379677070 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f3c5008be10 con 0x55d405371070 2024-04-26T23:08:18.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.865+0000 7f3c6b40ef40 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55d405413180 0x55d40540bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.865+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/3379677070 --> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] -- osd_op(unknown.0.0:1 13.12 13:4def72d4:::10000000003.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55d40540c270 con 0x55d405413180 2024-04-26T23:08:18.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.865+0000 7f3c68600640 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c4406b2d0 0x7f3c4406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.865+0000 7f3c69000640 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55d405413180 0x55d40540bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.866+0000 7f3c69000640 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55d405413180 0x55d40540bd30 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:18.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.866+0000 7f3c68600640 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c4406b2d0 0x7f3c4406d790 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3c58009600 tx=0x7f3c58005310 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:18.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.869+0000 7f3c69000640 1 -- 172.21.15.40:0/3379677070 <== osd.0 v2:172.21.15.40:6800/2897145808 1 ==== osd_op_reply(1 10000000003.00000000 [rmxattr] v90'9 uv9 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f3c60051860 con 0x55d405413180 2024-04-26T23:08:18.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.869+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/3379677070 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55d405413180 msgr2=0x55d40540bd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.869+0000 7f3c6b40ef40 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x55d405413180 0x55d40540bd30 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.869+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/3379677070 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c4406b2d0 msgr2=0x7f3c4406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.869+0000 7f3c6b40ef40 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3c4406b2d0 0x7f3c4406d790 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3c58009600 tx=0x7f3c58005310 comp rx=0 tx=0).stop 2024-04-26T23:08:18.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.869+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/3379677070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d405371070 msgr2=0x55d40540e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.869+0000 7f3c6b40ef40 1 --2- 172.21.15.40:0/3379677070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d405371070 0x55d40540e370 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f3c50010000 tx=0x7f3c50001fe0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.869+0000 7f3c69000640 1 -- 172.21.15.40:0/3379677070 reap_dead start 2024-04-26T23:08:18.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.869+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/3379677070 shutdown_connections 2024-04-26T23:08:18.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.869+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/3379677070 >> 172.21.15.40:0/3379677070 conn(0x55d40538bdf0 msgr2=0x55d405418760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:18.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.869+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/3379677070 shutdown_connections 2024-04-26T23:08:18.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.869+0000 7f3c6b40ef40 1 -- 172.21.15.40:0/3379677070 wait complete. 2024-04-26T23:08:18.875 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000004.00000000 parent 2024-04-26T23:08:18.897 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:08:18.932 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:08:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.988+0000 7f9b9dd21f40 1 Processor -- start 2024-04-26T23:08:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.988+0000 7f9b9dd21f40 1 -- start start 2024-04-26T23:08:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.990+0000 7f9b9dd21f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fe5e3cf990 0x55fe5e49e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.990+0000 7f9b9dd21f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe5e4a8830 0x55fe5e4a70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.990+0000 7f9b9dd21f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fe5e41c960 0x55fe5e4a5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.990+0000 7f9b9dd21f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fe5e4a9fa0 con 0x55fe5e4a8830 2024-04-26T23:08:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.990+0000 7f9b9dd21f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fe5e378d20 con 0x55fe5e41c960 2024-04-26T23:08:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.990+0000 7f9b9dd21f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55fe5e41ea60 con 0x55fe5e3cf990 2024-04-26T23:08:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.990+0000 7f9b9a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe5e4a8830 0x55fe5e4a70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.990+0000 7f9b9a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe5e4a8830 0x55fe5e4a70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54728/0 (socket says 172.21.15.40:54728) 2024-04-26T23:08:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.990+0000 7f9b9a600640 1 -- 172.21.15.40:0/4078189100 learned_addr learned my addr 172.21.15.40:0/4078189100 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.990+0000 7f9b9b000640 1 --2- 172.21.15.40:0/4078189100 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fe5e3cf990 0x55fe5e49e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.990+0000 7f9b9ba00640 1 --2- 172.21.15.40:0/4078189100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fe5e41c960 0x55fe5e4a5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.991+0000 7f9b9a600640 1 -- 172.21.15.40:0/4078189100 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fe5e3cf990 msgr2=0x55fe5e49e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.991+0000 7f9b9a600640 1 --2- 172.21.15.40:0/4078189100 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fe5e3cf990 0x55fe5e49e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.991+0000 7f9b9a600640 1 -- 172.21.15.40:0/4078189100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fe5e41c960 msgr2=0x55fe5e4a5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.991+0000 7f9b9a600640 1 --2- 172.21.15.40:0/4078189100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fe5e41c960 0x55fe5e4a5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.991+0000 7f9b9a600640 1 -- 172.21.15.40:0/4078189100 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe5e49dde0 con 0x55fe5e4a8830 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.991+0000 7f9b9a600640 1 --2- 172.21.15.40:0/4078189100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe5e4a8830 0x55fe5e4a70c0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f9b88001ce0 tx=0x7f9b8800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e5d74ed39e6158 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.991+0000 7f9b99200640 1 -- 172.21.15.40:0/4078189100 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8800bd80 con 0x55fe5e4a8830 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.992+0000 7f9b99200640 1 -- 172.21.15.40:0/4078189100 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b8800f400 con 0x55fe5e4a8830 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.992+0000 7f9b99200640 1 -- 172.21.15.40:0/4078189100 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8800f7f0 con 0x55fe5e4a8830 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.992+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/4078189100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe5e4a8830 msgr2=0x55fe5e4a70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.992+0000 7f9b9dd21f40 1 --2- 172.21.15.40:0/4078189100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe5e4a8830 0x55fe5e4a70c0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f9b88001ce0 tx=0x7f9b8800b5b0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.993+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/4078189100 shutdown_connections 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.993+0000 7f9b9dd21f40 1 --2- 172.21.15.40:0/4078189100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fe5e41c960 0x55fe5e4a5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.993+0000 7f9b9dd21f40 1 --2- 172.21.15.40:0/4078189100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe5e4a8830 0x55fe5e4a70c0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.993+0000 7f9b9dd21f40 1 --2- 172.21.15.40:0/4078189100 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fe5e3cf990 0x55fe5e49e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.993+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/4078189100 >> 172.21.15.40:0/4078189100 conn(0x55fe5e41edb0 msgr2=0x55fe5e3d0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.993+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/4078189100 shutdown_connections 2024-04-26T23:08:18.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.993+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/4078189100 wait complete. 2024-04-26T23:08:18.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.994+0000 7f9b9dd21f40 1 Processor -- start 2024-04-26T23:08:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.994+0000 7f9b9dd21f40 1 -- start start 2024-04-26T23:08:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.995+0000 7f9b9dd21f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe5e3cf990 0x55fe5e4a04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.995+0000 7f9b9dd21f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fe5e4a8830 0x55fe5e49c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.995+0000 7f9b9dd21f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fe5e535020 0x55fe5e5354a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.995+0000 7f9b9dd21f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fe5e404830 con 0x55fe5e3cf990 2024-04-26T23:08:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.995+0000 7f9b9dd21f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fe5e4146c0 con 0x55fe5e535020 2024-04-26T23:08:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.995+0000 7f9b9dd21f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55fe5e4a35b0 con 0x55fe5e4a8830 2024-04-26T23:08:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.995+0000 7f9b9b000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe5e3cf990 0x55fe5e4a04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.995+0000 7f9b9b000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe5e3cf990 0x55fe5e4a04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54736/0 (socket says 172.21.15.40:54736) 2024-04-26T23:08:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.995+0000 7f9b9b000640 1 -- 172.21.15.40:0/1740554213 learned_addr learned my addr 172.21.15.40:0/1740554213 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.995+0000 7f9b9ba00640 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fe5e535020 0x55fe5e5354a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.995+0000 7f9b9a600640 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fe5e4a8830 0x55fe5e49c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.996+0000 7f9b9b000640 1 -- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fe5e4a8830 msgr2=0x55fe5e49c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.996+0000 7f9b9b000640 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fe5e4a8830 0x55fe5e49c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.996+0000 7f9b9b000640 1 -- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fe5e535020 msgr2=0x55fe5e5354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.996+0000 7f9b9b000640 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fe5e535020 0x55fe5e5354a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.996+0000 7f9b9b000640 1 -- 172.21.15.40:0/1740554213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe5e4a7990 con 0x55fe5e3cf990 2024-04-26T23:08:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.996+0000 7f9b9a600640 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fe5e4a8830 0x55fe5e49c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:08:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.996+0000 7f9b9b000640 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe5e3cf990 0x55fe5e4a04b0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c008840 tx=0x7f9b8c008930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.996+0000 7f9b93400640 1 -- 172.21.15.40:0/1740554213 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0147e0 con 0x55fe5e3cf990 2024-04-26T23:08:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.996+0000 7f9b93400640 1 -- 172.21.15.40:0/1740554213 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b8c01a070 con 0x55fe5e3cf990 2024-04-26T23:08:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.996+0000 7f9b93400640 1 -- 172.21.15.40:0/1740554213 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c014bc0 con 0x55fe5e3cf990 2024-04-26T23:08:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.996+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/1740554213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fe5e53fb10 con 0x55fe5e3cf990 2024-04-26T23:08:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.997+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/1740554213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe5e53f810 con 0x55fe5e3cf990 2024-04-26T23:08:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:18.999+0000 7f9b93400640 1 -- 172.21.15.40:0/1740554213 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9b8c015470 con 0x55fe5e3cf990 2024-04-26T23:08:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.000+0000 7f9b93400640 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9b7c06b2d0 0x7f9b7c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.000+0000 7f9b93400640 1 -- 172.21.15.40:0/1740554213 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f9b8c08b380 con 0x55fe5e3cf990 2024-04-26T23:08:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.000+0000 7f9b9a600640 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9b7c06b2d0 0x7f9b7c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.000+0000 7f9b9dd21f40 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x55fe5e540fb0 0x55fe5e5413d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.000+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/1740554213 --> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] -- osd_op(unknown.0.0:1 13.5 13:a367a3e8:::10000000004.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55fe5e54be10 con 0x55fe5e540fb0 2024-04-26T23:08:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.001+0000 7f9b9ba00640 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x55fe5e540fb0 0x55fe5e5413d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.002+0000 7f9b9a600640 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9b7c06b2d0 0x7f9b7c06d790 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800b5f0 tx=0x7f9b88009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.002+0000 7f9b9ba00640 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x55fe5e540fb0 0x55fe5e5413d0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.004+0000 7f9b9ba00640 1 -- 172.21.15.40:0/1740554213 <== osd.4 v2:172.21.15.103:6803/801343304 1 ==== osd_op_reply(1 10000000004.00000000 [rmxattr] v90'6 uv6 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9b940518b0 con 0x55fe5e540fb0 2024-04-26T23:08:19.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.004+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x55fe5e540fb0 msgr2=0x55fe5e5413d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.004+0000 7f9b9dd21f40 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x55fe5e540fb0 0x55fe5e5413d0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.004+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9b7c06b2d0 msgr2=0x7f9b7c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.004+0000 7f9b9dd21f40 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9b7c06b2d0 0x7f9b7c06d790 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800b5f0 tx=0x7f9b88009000 comp rx=0 tx=0).stop 2024-04-26T23:08:19.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.004+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/1740554213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe5e3cf990 msgr2=0x55fe5e4a04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.004+0000 7f9b9dd21f40 1 --2- 172.21.15.40:0/1740554213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe5e3cf990 0x55fe5e4a04b0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c008840 tx=0x7f9b8c008930 comp rx=0 tx=0).stop 2024-04-26T23:08:19.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.004+0000 7f9b9ba00640 1 -- 172.21.15.40:0/1740554213 reap_dead start 2024-04-26T23:08:19.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.004+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/1740554213 shutdown_connections 2024-04-26T23:08:19.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.004+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/1740554213 >> 172.21.15.40:0/1740554213 conn(0x55fe5e41edb0 msgr2=0x55fe5e413c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.004+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/1740554213 shutdown_connections 2024-04-26T23:08:19.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.004+0000 7f9b9dd21f40 1 -- 172.21.15.40:0/1740554213 wait complete. 2024-04-26T23:08:19.010 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000005.00000000 parent 2024-04-26T23:08:19.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.095+0000 7fc6bd5b7f40 1 Processor -- start 2024-04-26T23:08:19.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.095+0000 7fc6bd5b7f40 1 -- start start 2024-04-26T23:08:19.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.096+0000 7fc6bd5b7f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584f44c0e90 0x5584f45654c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.096+0000 7fc6bd5b7f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584f44c2070 0x5584f4563e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.096+0000 7fc6bd5b7f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584f44d93d0 0x5584f4563490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.096+0000 7fc6bd5b7f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5584f4567030 con 0x5584f44c2070 2024-04-26T23:08:19.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.096+0000 7fc6bd5b7f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5584f4436d20 con 0x5584f44c0e90 2024-04-26T23:08:19.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.096+0000 7fc6bd5b7f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5584f4567310 con 0x5584f44d93d0 2024-04-26T23:08:19.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.097+0000 7fc6b9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584f44c2070 0x5584f4563e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.097+0000 7fc6b9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584f44c2070 0x5584f4563e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54764/0 (socket says 172.21.15.40:54764) 2024-04-26T23:08:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.097+0000 7fc6b9e00640 1 -- 172.21.15.40:0/21658364 learned_addr learned my addr 172.21.15.40:0/21658364 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.097+0000 7fc6ba800640 1 --2- 172.21.15.40:0/21658364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584f44c0e90 0x5584f45654c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.097+0000 7fc6bb200640 1 --2- 172.21.15.40:0/21658364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584f44d93d0 0x5584f4563490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.097+0000 7fc6b9e00640 1 -- 172.21.15.40:0/21658364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584f44d93d0 msgr2=0x5584f4563490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.097+0000 7fc6b9e00640 1 --2- 172.21.15.40:0/21658364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584f44d93d0 0x5584f4563490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.097+0000 7fc6b9e00640 1 -- 172.21.15.40:0/21658364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584f44c0e90 msgr2=0x5584f45654c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.097+0000 7fc6b9e00640 1 --2- 172.21.15.40:0/21658364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584f44c0e90 0x5584f45654c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.097+0000 7fc6b9e00640 1 -- 172.21.15.40:0/21658364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584f448f930 con 0x5584f44c2070 2024-04-26T23:08:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.097+0000 7fc6b9e00640 1 --2- 172.21.15.40:0/21658364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584f44c2070 0x5584f4563e80 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0460b0 tx=0x7fc6ac050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb843f66e0b78e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.098+0000 7fc6b8a00640 1 -- 172.21.15.40:0/21658364 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac055030 con 0x5584f44c2070 2024-04-26T23:08:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.098+0000 7fc6b8a00640 1 -- 172.21.15.40:0/21658364 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6ac051af0 con 0x5584f44c2070 2024-04-26T23:08:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.098+0000 7fc6b8a00640 1 -- 172.21.15.40:0/21658364 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac065480 con 0x5584f44c2070 2024-04-26T23:08:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.098+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/21658364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584f44c2070 msgr2=0x5584f4563e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.098+0000 7fc6bd5b7f40 1 --2- 172.21.15.40:0/21658364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584f44c2070 0x5584f4563e80 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0460b0 tx=0x7fc6ac050000 comp rx=0 tx=0).stop 2024-04-26T23:08:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.099+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/21658364 shutdown_connections 2024-04-26T23:08:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.099+0000 7fc6bd5b7f40 1 --2- 172.21.15.40:0/21658364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584f44d93d0 0x5584f4563490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.099+0000 7fc6bd5b7f40 1 --2- 172.21.15.40:0/21658364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584f44c2070 0x5584f4563e80 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.099+0000 7fc6bd5b7f40 1 --2- 172.21.15.40:0/21658364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584f44c0e90 0x5584f45654c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.099+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/21658364 >> 172.21.15.40:0/21658364 conn(0x5584f44dcdf0 msgr2=0x5584f44c0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.099+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/21658364 shutdown_connections 2024-04-26T23:08:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.099+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/21658364 wait complete. 2024-04-26T23:08:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.099+0000 7fc6bd5b7f40 1 Processor -- start 2024-04-26T23:08:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.100+0000 7fc6bd5b7f40 1 -- start start 2024-04-26T23:08:19.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.100+0000 7fc6bd5b7f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584f44c0e90 0x5584f45584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.100+0000 7fc6bd5b7f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584f44c2070 0x5584f45589e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.100+0000 7fc6bd5b7f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584f44d93d0 0x5584f44bfd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.100+0000 7fc6bd5b7f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5584f4566200 con 0x5584f44d93d0 2024-04-26T23:08:19.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.100+0000 7fc6bd5b7f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5584f4566450 con 0x5584f44c2070 2024-04-26T23:08:19.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.100+0000 7fc6bd5b7f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5584f45643c0 con 0x5584f44c0e90 2024-04-26T23:08:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.101+0000 7fc6bb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584f44d93d0 0x5584f44bfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.101+0000 7fc6bb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584f44d93d0 0x5584f44bfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54768/0 (socket says 172.21.15.40:54768) 2024-04-26T23:08:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.101+0000 7fc6bb200640 1 -- 172.21.15.40:0/3285570092 learned_addr learned my addr 172.21.15.40:0/3285570092 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.101+0000 7fc6b9e00640 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584f44c2070 0x5584f45589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.101+0000 7fc6ba800640 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584f44c0e90 0x5584f45584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.101+0000 7fc6bb200640 1 -- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584f44c0e90 msgr2=0x5584f45584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.101+0000 7fc6bb200640 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584f44c0e90 0x5584f45584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.101+0000 7fc6bb200640 1 -- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584f44c2070 msgr2=0x5584f45589e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.101+0000 7fc6bb200640 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584f44c2070 0x5584f45589e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.101+0000 7fc6bb200640 1 -- 172.21.15.40:0/3285570092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584f455c8c0 con 0x5584f44d93d0 2024-04-26T23:08:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.102+0000 7fc6bb200640 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584f44d93d0 0x5584f44bfd40 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400cc70 tx=0x7fc6b400a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.102+0000 7fc6b3400640 1 -- 172.21.15.40:0/3285570092 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4008040 con 0x5584f44d93d0 2024-04-26T23:08:19.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.102+0000 7fc6b3400640 1 -- 172.21.15.40:0/3285570092 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6b4014020 con 0x5584f44d93d0 2024-04-26T23:08:19.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.102+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/3285570092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5584f4563f20 con 0x5584f44d93d0 2024-04-26T23:08:19.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.102+0000 7fc6b3400640 1 -- 172.21.15.40:0/3285570092 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b400f400 con 0x5584f44d93d0 2024-04-26T23:08:19.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.102+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/3285570092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584f455f7a0 con 0x5584f44d93d0 2024-04-26T23:08:19.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.104+0000 7fc6b3400640 1 -- 172.21.15.40:0/3285570092 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc6b400f5a0 con 0x5584f44d93d0 2024-04-26T23:08:19.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.105+0000 7fc6b3400640 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc69806b2d0 0x7fc69806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.105+0000 7fc6b3400640 1 -- 172.21.15.40:0/3285570092 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fc6b408a930 con 0x5584f44d93d0 2024-04-26T23:08:19.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.105+0000 7fc6bd5b7f40 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x5584f455cb30 0x5584f45f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.105+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/3285570092 --> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] -- osd_op(unknown.0.0:1 13.9 13:9258b67d:::10000000005.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x5584f455fcc0 con 0x5584f455cb30 2024-04-26T23:08:19.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.106+0000 7fc6ba800640 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc69806b2d0 0x7fc69806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.106+0000 7fc6b9e00640 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x5584f455cb30 0x5584f45f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.107+0000 7fc6b9e00640 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x5584f455cb30 0x5584f45f59d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.107+0000 7fc6ba800640 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc69806b2d0 0x7fc69806d790 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4005400 tx=0x7fc6a4005310 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.110+0000 7fc6b9e00640 1 -- 172.21.15.40:0/3285570092 <== osd.5 v2:172.21.15.103:6800/157083760 1 ==== osd_op_reply(1 10000000005.00000000 [rmxattr] v90'6 uv6 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc6ac065480 con 0x5584f455cb30 2024-04-26T23:08:19.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.110+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x5584f455cb30 msgr2=0x5584f45f59d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.110+0000 7fc6bd5b7f40 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x5584f455cb30 0x5584f45f59d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.110+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc69806b2d0 msgr2=0x7fc69806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.110+0000 7fc6bd5b7f40 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc69806b2d0 0x7fc69806d790 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4005400 tx=0x7fc6a4005310 comp rx=0 tx=0).stop 2024-04-26T23:08:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.110+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/3285570092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584f44d93d0 msgr2=0x5584f44bfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.110+0000 7fc6bd5b7f40 1 --2- 172.21.15.40:0/3285570092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584f44d93d0 0x5584f44bfd40 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400cc70 tx=0x7fc6b400a5d0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.110+0000 7fc6bb200640 1 -- 172.21.15.40:0/3285570092 reap_dead start 2024-04-26T23:08:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.110+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/3285570092 shutdown_connections 2024-04-26T23:08:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.110+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/3285570092 >> 172.21.15.40:0/3285570092 conn(0x5584f44dcdf0 msgr2=0x5584f4565290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.110+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/3285570092 shutdown_connections 2024-04-26T23:08:19.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.110+0000 7fc6bd5b7f40 1 -- 172.21.15.40:0/3285570092 wait complete. 2024-04-26T23:08:19.115 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000006.00000000 parent 2024-04-26T23:08:19.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.220+0000 7fe1abef6f40 1 Processor -- start 2024-04-26T23:08:19.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.220+0000 7fe1abef6f40 1 -- start start 2024-04-26T23:08:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.221+0000 7fe1abef6f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe18c0022a0 0x55cf429ec660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.221+0000 7fe1abef6f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf42948be0 0x55cf42961ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.221+0000 7fe1abef6f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf429ee090 0x55cf42a7b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.221+0000 7fe1abef6f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cf428c0d20 con 0x55cf429ee090 2024-04-26T23:08:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.221+0000 7fe1abef6f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cf428c4d60 con 0x55cf42948be0 2024-04-26T23:08:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.221+0000 7fe1abef6f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55cf429ef420 con 0x7fe18c0022a0 2024-04-26T23:08:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.222+0000 7fe1a9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf429ee090 0x55cf42a7b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.222+0000 7fe1a9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf429ee090 0x55cf42a7b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54792/0 (socket says 172.21.15.40:54792) 2024-04-26T23:08:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.222+0000 7fe1a9a00640 1 -- 172.21.15.40:0/3667109404 learned_addr learned my addr 172.21.15.40:0/3667109404 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.222+0000 7fe1a9000640 1 --2- 172.21.15.40:0/3667109404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe18c0022a0 0x55cf429ec660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.222+0000 7fe1a0600640 1 --2- 172.21.15.40:0/3667109404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf42948be0 0x55cf42961ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.222+0000 7fe1a9a00640 1 -- 172.21.15.40:0/3667109404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe18c0022a0 msgr2=0x55cf429ec660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.222+0000 7fe1a9a00640 1 --2- 172.21.15.40:0/3667109404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe18c0022a0 0x55cf429ec660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.222+0000 7fe1a9a00640 1 -- 172.21.15.40:0/3667109404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf42948be0 msgr2=0x55cf42961ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.222+0000 7fe1a9a00640 1 --2- 172.21.15.40:0/3667109404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf42948be0 0x55cf42961ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.222+0000 7fe1a9a00640 1 -- 172.21.15.40:0/3667109404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf429ed100 con 0x55cf429ee090 2024-04-26T23:08:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.222+0000 7fe1a9a00640 1 --2- 172.21.15.40:0/3667109404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf429ee090 0x55cf42a7b700 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4006e60 tx=0x7fe1a40028a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a46f8e749e37a8f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.223+0000 7fe1a3400640 1 -- 172.21.15.40:0/3667109404 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4011610 con 0x55cf429ee090 2024-04-26T23:08:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.223+0000 7fe1a3400640 1 -- 172.21.15.40:0/3667109404 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a4016070 con 0x55cf429ee090 2024-04-26T23:08:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.223+0000 7fe1a3400640 1 -- 172.21.15.40:0/3667109404 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a40119f0 con 0x55cf429ee090 2024-04-26T23:08:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.223+0000 7fe1abef6f40 1 -- 172.21.15.40:0/3667109404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf429ee090 msgr2=0x55cf42a7b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.223+0000 7fe1abef6f40 1 --2- 172.21.15.40:0/3667109404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf429ee090 0x55cf42a7b700 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4006e60 tx=0x7fe1a40028a0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.223+0000 7fe1abef6f40 1 -- 172.21.15.40:0/3667109404 shutdown_connections 2024-04-26T23:08:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.223+0000 7fe1abef6f40 1 --2- 172.21.15.40:0/3667109404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf429ee090 0x55cf42a7b700 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.223+0000 7fe1abef6f40 1 --2- 172.21.15.40:0/3667109404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf42948be0 0x55cf42961ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.223+0000 7fe1abef6f40 1 --2- 172.21.15.40:0/3667109404 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe18c0022a0 0x55cf429ec660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.223+0000 7fe1abef6f40 1 -- 172.21.15.40:0/3667109404 >> 172.21.15.40:0/3667109404 conn(0x55cf42946c00 msgr2=0x55cf42946210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.224+0000 7fe1abef6f40 1 -- 172.21.15.40:0/3667109404 shutdown_connections 2024-04-26T23:08:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.224+0000 7fe1abef6f40 1 -- 172.21.15.40:0/3667109404 wait complete. 2024-04-26T23:08:19.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.224+0000 7fe1abef6f40 1 Processor -- start 2024-04-26T23:08:19.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.224+0000 7fe1abef6f40 1 -- start start 2024-04-26T23:08:19.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.225+0000 7fe1abef6f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf42948be0 0x55cf429e5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.225+0000 7fe1abef6f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf429ee090 0x55cf4295e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.225+0000 7fe1abef6f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe18c0022a0 0x55cf42a78360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.225+0000 7fe1abef6f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cf429ed720 con 0x55cf429ee090 2024-04-26T23:08:19.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.225+0000 7fe1abef6f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cf42a7c050 con 0x55cf42948be0 2024-04-26T23:08:19.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.225+0000 7fe1abef6f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55cf42a7bc40 con 0x7fe18c0022a0 2024-04-26T23:08:19.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.226+0000 7fe1a0600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf429ee090 0x55cf4295e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.226+0000 7fe1a0600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf429ee090 0x55cf4295e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54796/0 (socket says 172.21.15.40:54796) 2024-04-26T23:08:19.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.226+0000 7fe1a0600640 1 -- 172.21.15.40:0/419035212 learned_addr learned my addr 172.21.15.40:0/419035212 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.226+0000 7fe1a9000640 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf42948be0 0x55cf429e5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.226+0000 7fe1a9a00640 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe18c0022a0 0x55cf42a78360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.226+0000 7fe1a0600640 1 -- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe18c0022a0 msgr2=0x55cf42a78360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.226+0000 7fe1a0600640 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe18c0022a0 0x55cf42a78360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.226+0000 7fe1a0600640 1 -- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf42948be0 msgr2=0x55cf429e5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.226+0000 7fe1a0600640 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf42948be0 0x55cf429e5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.226+0000 7fe1a0600640 1 -- 172.21.15.40:0/419035212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf4295ec30 con 0x55cf429ee090 2024-04-26T23:08:19.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.226+0000 7fe1a9000640 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf42948be0 0x55cf429e5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:08:19.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.226+0000 7fe1a0600640 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf429ee090 0x55cf4295e6f0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fe19000b4f0 tx=0x7fe19000b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.227+0000 7fe1a2000640 1 -- 172.21.15.40:0/419035212 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe190014020 con 0x55cf429ee090 2024-04-26T23:08:19.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.227+0000 7fe1a2000640 1 -- 172.21.15.40:0/419035212 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe190010620 con 0x55cf429ee090 2024-04-26T23:08:19.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.227+0000 7fe1a2000640 1 -- 172.21.15.40:0/419035212 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe190008030 con 0x55cf429ee090 2024-04-26T23:08:19.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.227+0000 7fe1abef6f40 1 -- 172.21.15.40:0/419035212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cf4291a3c0 con 0x55cf429ee090 2024-04-26T23:08:19.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.227+0000 7fe1abef6f40 1 -- 172.21.15.40:0/419035212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf42945e80 con 0x55cf429ee090 2024-04-26T23:08:19.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.229+0000 7fe1a2000640 1 -- 172.21.15.40:0/419035212 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe190017020 con 0x55cf429ee090 2024-04-26T23:08:19.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.230+0000 7fe1a2000640 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe18406b2d0 0x7fe18406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.230+0000 7fe1a2000640 1 -- 172.21.15.40:0/419035212 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fe19008a9e0 con 0x55cf429ee090 2024-04-26T23:08:19.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.230+0000 7fe1abef6f40 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x55cf429e6a80 0x55cf429e6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.230+0000 7fe1a9000640 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe18406b2d0 0x7fe18406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.230+0000 7fe1abef6f40 1 -- 172.21.15.40:0/419035212 --> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] -- osd_op(unknown.0.0:1 13.5 13:a5feb6f3:::10000000006.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55cf42a7b700 con 0x55cf429e6a80 2024-04-26T23:08:19.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.231+0000 7fe1a9a00640 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x55cf429e6a80 0x55cf429e6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.232+0000 7fe1a9000640 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe18406b2d0 0x7fe18406d790 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe198058000 tx=0x7fe198053000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.232+0000 7fe1a9a00640 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x55cf429e6a80 0x55cf429e6ef0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.235+0000 7fe1a9a00640 1 -- 172.21.15.40:0/419035212 <== osd.4 v2:172.21.15.103:6803/801343304 1 ==== osd_op_reply(1 10000000006.00000000 [rmxattr] v90'7 uv7 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe1a4011870 con 0x55cf429e6a80 2024-04-26T23:08:19.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.235+0000 7fe1abef6f40 1 -- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x55cf429e6a80 msgr2=0x55cf429e6ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.235+0000 7fe1abef6f40 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x55cf429e6a80 0x55cf429e6ef0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.235+0000 7fe1abef6f40 1 -- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe18406b2d0 msgr2=0x7fe18406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.235+0000 7fe1abef6f40 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe18406b2d0 0x7fe18406d790 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe198058000 tx=0x7fe198053000 comp rx=0 tx=0).stop 2024-04-26T23:08:19.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.235+0000 7fe1abef6f40 1 -- 172.21.15.40:0/419035212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf429ee090 msgr2=0x55cf4295e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.235+0000 7fe1abef6f40 1 --2- 172.21.15.40:0/419035212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf429ee090 0x55cf4295e6f0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fe19000b4f0 tx=0x7fe19000b9b0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.235+0000 7fe1a9a00640 1 -- 172.21.15.40:0/419035212 reap_dead start 2024-04-26T23:08:19.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.235+0000 7fe1abef6f40 1 -- 172.21.15.40:0/419035212 shutdown_connections 2024-04-26T23:08:19.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.235+0000 7fe1abef6f40 1 -- 172.21.15.40:0/419035212 >> 172.21.15.40:0/419035212 conn(0x55cf42946c00 msgr2=0x55cf429626d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.235+0000 7fe1abef6f40 1 -- 172.21.15.40:0/419035212 shutdown_connections 2024-04-26T23:08:19.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.235+0000 7fe1abef6f40 1 -- 172.21.15.40:0/419035212 wait complete. 2024-04-26T23:08:19.243 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000007.00000000 parent 2024-04-26T23:08:19.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.335+0000 7f6e55f40f40 1 Processor -- start 2024-04-26T23:08:19.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.335+0000 7f6e55f40f40 1 -- start start 2024-04-26T23:08:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.336+0000 7f6e55f40f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fbb409080 0x562fbb44d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.336+0000 7f6e55f40f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562fbb437360 0x562fbb433130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.336+0000 7f6e55f40f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562fbb4da280 0x562fbb44f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.336+0000 7f6e55f40f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562fbb4377d0 con 0x562fbb409080 2024-04-26T23:08:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.336+0000 7f6e55f40f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562fbb3aed20 con 0x562fbb4da280 2024-04-26T23:08:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.336+0000 7f6e55f40f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x562fbb452a60 con 0x562fbb437360 2024-04-26T23:08:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.336+0000 7f6e53200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fbb409080 0x562fbb44d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.336+0000 7f6e53200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fbb409080 0x562fbb44d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54802/0 (socket says 172.21.15.40:54802) 2024-04-26T23:08:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.336+0000 7f6e53200640 1 -- 172.21.15.40:0/2192976566 learned_addr learned my addr 172.21.15.40:0/2192976566 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.337+0000 7f6e4be00640 1 --2- 172.21.15.40:0/2192976566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562fbb437360 0x562fbb433130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.337+0000 7f6e53200640 1 -- 172.21.15.40:0/2192976566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562fbb437360 msgr2=0x562fbb433130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.337+0000 7f6e53200640 1 --2- 172.21.15.40:0/2192976566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562fbb437360 0x562fbb433130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.337+0000 7f6e53200640 1 -- 172.21.15.40:0/2192976566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562fbb4da280 msgr2=0x562fbb44f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.337+0000 7f6e53200640 1 --2- 172.21.15.40:0/2192976566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562fbb4da280 0x562fbb44f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.337+0000 7f6e53200640 1 -- 172.21.15.40:0/2192976566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fbb40b470 con 0x562fbb409080 2024-04-26T23:08:19.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.337+0000 7f6e4be00640 1 --2- 172.21.15.40:0/2192976566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562fbb437360 0x562fbb433130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:19.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.337+0000 7f6e53200640 1 --2- 172.21.15.40:0/2192976566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fbb409080 0x562fbb44d840 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f6e44006ec0 tx=0x7f6e4400b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbef76dbdd0dbd8f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.337+0000 7f6e51e00640 1 -- 172.21.15.40:0/2192976566 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e44012070 con 0x562fbb409080 2024-04-26T23:08:19.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.338+0000 7f6e51e00640 1 -- 172.21.15.40:0/2192976566 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e4400e690 con 0x562fbb409080 2024-04-26T23:08:19.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.338+0000 7f6e51e00640 1 -- 172.21.15.40:0/2192976566 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4400eac0 con 0x562fbb409080 2024-04-26T23:08:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.338+0000 7f6e55f40f40 1 -- 172.21.15.40:0/2192976566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fbb409080 msgr2=0x562fbb44d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.338+0000 7f6e55f40f40 1 --2- 172.21.15.40:0/2192976566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fbb409080 0x562fbb44d840 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f6e44006ec0 tx=0x7f6e4400b6c0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.338+0000 7f6e55f40f40 1 -- 172.21.15.40:0/2192976566 shutdown_connections 2024-04-26T23:08:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.338+0000 7f6e55f40f40 1 --2- 172.21.15.40:0/2192976566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562fbb4da280 0x562fbb44f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.338+0000 7f6e55f40f40 1 --2- 172.21.15.40:0/2192976566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562fbb437360 0x562fbb433130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.338+0000 7f6e55f40f40 1 --2- 172.21.15.40:0/2192976566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fbb409080 0x562fbb44d840 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.338+0000 7f6e55f40f40 1 -- 172.21.15.40:0/2192976566 >> 172.21.15.40:0/2192976566 conn(0x562fbb434c00 msgr2=0x562fbb452350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.338+0000 7f6e55f40f40 1 -- 172.21.15.40:0/2192976566 shutdown_connections 2024-04-26T23:08:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.338+0000 7f6e55f40f40 1 -- 172.21.15.40:0/2192976566 wait complete. 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.339+0000 7f6e55f40f40 1 Processor -- start 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.339+0000 7f6e55f40f40 1 -- start start 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.340+0000 7f6e55f40f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fbb409080 0x562fbb429d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.340+0000 7f6e55f40f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562fbb437360 0x562fbb4504f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.340+0000 7f6e55f40f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562fbb4da280 0x562fbb4d4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.340+0000 7f6e55f40f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562fbb4d66d0 con 0x562fbb409080 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.340+0000 7f6e55f40f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562fbb4d6920 con 0x562fbb4da280 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.340+0000 7f6e55f40f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x562fbb4344d0 con 0x562fbb437360 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.340+0000 7f6e53200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fbb409080 0x562fbb429d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.341+0000 7f6e53200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fbb409080 0x562fbb429d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54806/0 (socket says 172.21.15.40:54806) 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.341+0000 7f6e53200640 1 -- 172.21.15.40:0/349039822 learned_addr learned my addr 172.21.15.40:0/349039822 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.341+0000 7f6e4be00640 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562fbb437360 0x562fbb4504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.341+0000 7f6e53c00640 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562fbb4da280 0x562fbb4d4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.341+0000 7f6e53200640 1 -- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562fbb437360 msgr2=0x562fbb4504f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.341+0000 7f6e53200640 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562fbb437360 0x562fbb4504f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.341+0000 7f6e53200640 1 -- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562fbb4da280 msgr2=0x562fbb4d4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.341+0000 7f6e53200640 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562fbb4da280 0x562fbb4d4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.341+0000 7f6e53200640 1 -- 172.21.15.40:0/349039822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fbb44dd80 con 0x562fbb409080 2024-04-26T23:08:19.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.341+0000 7f6e53c00640 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562fbb4da280 0x562fbb4d4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:19.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.341+0000 7f6e53200640 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fbb409080 0x562fbb429d30 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400bbc0 tx=0x7f6e44000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.342+0000 7f6e50a00640 1 -- 172.21.15.40:0/349039822 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e44012050 con 0x562fbb409080 2024-04-26T23:08:19.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.342+0000 7f6e50a00640 1 -- 172.21.15.40:0/349039822 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e44017900 con 0x562fbb409080 2024-04-26T23:08:19.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.342+0000 7f6e50a00640 1 -- 172.21.15.40:0/349039822 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e44017cf0 con 0x562fbb409080 2024-04-26T23:08:19.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.342+0000 7f6e55f40f40 1 -- 172.21.15.40:0/349039822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562fbb4d5830 con 0x562fbb409080 2024-04-26T23:08:19.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.342+0000 7f6e55f40f40 1 -- 172.21.15.40:0/349039822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fbb42a270 con 0x562fbb409080 2024-04-26T23:08:19.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.344+0000 7f6e50a00640 1 -- 172.21.15.40:0/349039822 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6e44021070 con 0x562fbb409080 2024-04-26T23:08:19.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.345+0000 7f6e50a00640 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6e3006b2d0 0x7f6e3006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.345+0000 7f6e50a00640 1 -- 172.21.15.40:0/349039822 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f6e4408cbc0 con 0x562fbb409080 2024-04-26T23:08:19.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.345+0000 7f6e55f40f40 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x562fbb433ac0 0x562fbb4d7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.345+0000 7f6e55f40f40 1 -- 172.21.15.40:0/349039822 --> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] -- osd_op(unknown.0.0:1 13.1d 13:bbf5850a:::10000000007.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x562fbb4d8010 con 0x562fbb433ac0 2024-04-26T23:08:19.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.346+0000 7f6e4be00640 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6e3006b2d0 0x7f6e3006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.346+0000 7f6e53c00640 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x562fbb433ac0 0x562fbb4d7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.347+0000 7f6e4be00640 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6e3006b2d0 0x7f6e3006d790 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c056000 tx=0x7f6e3c050000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.347+0000 7f6e53c00640 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x562fbb433ac0 0x562fbb4d7ad0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.350+0000 7f6e53c00640 1 -- 172.21.15.40:0/349039822 <== osd.7 v2:172.21.15.103:6819/3769540391 1 ==== osd_op_reply(1 10000000007.00000000 [rmxattr] v90'7 uv7 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6e4c0028f0 con 0x562fbb433ac0 2024-04-26T23:08:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.350+0000 7f6e55f40f40 1 -- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x562fbb433ac0 msgr2=0x562fbb4d7ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.350+0000 7f6e55f40f40 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x562fbb433ac0 0x562fbb4d7ad0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.350+0000 7f6e55f40f40 1 -- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6e3006b2d0 msgr2=0x7f6e3006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.350+0000 7f6e55f40f40 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6e3006b2d0 0x7f6e3006d790 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c056000 tx=0x7f6e3c050000 comp rx=0 tx=0).stop 2024-04-26T23:08:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.350+0000 7f6e55f40f40 1 -- 172.21.15.40:0/349039822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fbb409080 msgr2=0x562fbb429d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.350+0000 7f6e55f40f40 1 --2- 172.21.15.40:0/349039822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fbb409080 0x562fbb429d30 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400bbc0 tx=0x7f6e44000bc0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.350+0000 7f6e53c00640 1 -- 172.21.15.40:0/349039822 reap_dead start 2024-04-26T23:08:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.350+0000 7f6e55f40f40 1 -- 172.21.15.40:0/349039822 shutdown_connections 2024-04-26T23:08:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.350+0000 7f6e55f40f40 1 -- 172.21.15.40:0/349039822 >> 172.21.15.40:0/349039822 conn(0x562fbb434c00 msgr2=0x562fbb452350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.350+0000 7f6e55f40f40 1 -- 172.21.15.40:0/349039822 shutdown_connections 2024-04-26T23:08:19.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.350+0000 7f6e55f40f40 1 -- 172.21.15.40:0/349039822 wait complete. 2024-04-26T23:08:19.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:08:19.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /d1/d2/d3 recursive,force 2024-04-26T23:08:19.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.558+0000 7f476a800640 1 Processor -- start 2024-04-26T23:08:19.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.558+0000 7f476a800640 1 -- start start 2024-04-26T23:08:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.560+0000 7f476a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f476415f420 0x7f47640d0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.560+0000 7f476a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f47640d3b60 0x7f47640b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.560+0000 7f476a800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47640d0d50 con 0x7f47640b3d90 2024-04-26T23:08:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.560+0000 7f476a800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f47640d0a70 con 0x7f47640d3b60 2024-04-26T23:08:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.560+0000 7f476a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f47640b1bf0 con 0x7f476415f420 2024-04-26T23:08:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.560+0000 7f4769e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f47640d3b60 0x7f47640b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.560+0000 7f4769e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f47640d3b60 0x7f47640b26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54808/0 (socket says 172.21.15.40:54808) 2024-04-26T23:08:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.560+0000 7f4769e00640 1 -- 172.21.15.40:0/4063221812 learned_addr learned my addr 172.21.15.40:0/4063221812 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.560+0000 7f4769400640 1 --2- 172.21.15.40:0/4063221812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f476415f420 0x7f47640d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.561+0000 7f4769e00640 1 -- 172.21.15.40:0/4063221812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f476415f420 msgr2=0x7f47640d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.561+0000 7f4769e00640 1 --2- 172.21.15.40:0/4063221812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f476415f420 0x7f47640d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.561+0000 7f4769e00640 1 -- 172.21.15.40:0/4063221812 >> v1:172.21.15.103:6789/0 conn(0x7f47640b3d90 legacy=0x7f47640d3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.561+0000 7f4769e00640 1 -- 172.21.15.40:0/4063221812 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47640b1ed0 con 0x7f47640d3b60 2024-04-26T23:08:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.562+0000 7f4769e00640 1 --2- 172.21.15.40:0/4063221812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640d3b60 0x7f47640b26d0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f475c009e50 tx=0x7f475c02a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d350f4f18631a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.562+0000 7f4763e00640 1 -- 172.21.15.40:0/4063221812 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c036070 con 0x7f47640d3b60 2024-04-26T23:08:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.562+0000 7f4763e00640 1 -- 172.21.15.40:0/4063221812 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f475c031600 con 0x7f47640d3b60 2024-04-26T23:08:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.562+0000 7f4763e00640 1 -- 172.21.15.40:0/4063221812 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c0319d0 con 0x7f47640d3b60 2024-04-26T23:08:19.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.563+0000 7f476a800640 1 -- 172.21.15.40:0/4063221812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640d3b60 msgr2=0x7f47640b26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.563+0000 7f476a800640 1 --2- 172.21.15.40:0/4063221812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640d3b60 0x7f47640b26d0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f475c009e50 tx=0x7f475c02a630 comp rx=0 tx=0).stop 2024-04-26T23:08:19.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.563+0000 7f476a800640 1 -- 172.21.15.40:0/4063221812 shutdown_connections 2024-04-26T23:08:19.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.563+0000 7f476a800640 1 --2- 172.21.15.40:0/4063221812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640d3b60 0x7f47640b26d0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.563+0000 7f476a800640 1 --2- 172.21.15.40:0/4063221812 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f476415f420 0x7f47640d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.563+0000 7f476a800640 1 -- 172.21.15.40:0/4063221812 >> 172.21.15.40:0/4063221812 conn(0x7f476409e1d0 msgr2=0x7f47640d4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.563+0000 7f476a800640 1 -- 172.21.15.40:0/4063221812 shutdown_connections 2024-04-26T23:08:19.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.563+0000 7f476a800640 1 -- 172.21.15.40:0/4063221812 wait complete. 2024-04-26T23:08:19.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.565+0000 7f476a800640 1 Processor -- start 2024-04-26T23:08:19.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.565+0000 7f476a800640 1 -- start start 2024-04-26T23:08:19.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.566+0000 7f476a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47640b3d90 0x7f47640cc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.566+0000 7f476a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47640d3b60 0x7f47640b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.566+0000 7f476a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f476415f420 0x7f47640ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.566+0000 7f476a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f47640b2c10 con 0x7f476415f420 2024-04-26T23:08:19.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.566+0000 7f476a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f47640d2e50 con 0x7f47640b3d90 2024-04-26T23:08:19.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.566+0000 7f476a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f47640b2d90 con 0x7f47640d3b60 2024-04-26T23:08:19.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.566+0000 7f4769e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f476415f420 0x7f47640ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.566+0000 7f4769e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f476415f420 0x7f47640ce2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54810/0 (socket says 172.21.15.40:54810) 2024-04-26T23:08:19.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.566+0000 7f4769e00640 1 -- 172.21.15.40:0/376525831 learned_addr learned my addr 172.21.15.40:0/376525831 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.567+0000 7f4769400640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47640b3d90 0x7f47640cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.567+0000 7f4768a00640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47640d3b60 0x7f47640b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.567+0000 7f4769e00640 1 -- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47640d3b60 msgr2=0x7f47640b8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.567+0000 7f4769e00640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47640d3b60 0x7f47640b8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.567+0000 7f4769e00640 1 -- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47640b3d90 msgr2=0x7f47640cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.567+0000 7f4769e00640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47640b3d90 0x7f47640cc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.567+0000 7f4769e00640 1 -- 172.21.15.40:0/376525831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47640b8e50 con 0x7f476415f420 2024-04-26T23:08:19.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.567+0000 7f4769e00640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f476415f420 0x7f47640ce2a0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f475c009e50 tx=0x7f475c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.568+0000 7f4761600640 1 -- 172.21.15.40:0/376525831 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c041020 con 0x7f476415f420 2024-04-26T23:08:19.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.568+0000 7f4761600640 1 -- 172.21.15.40:0/376525831 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f475c008690 con 0x7f476415f420 2024-04-26T23:08:19.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.568+0000 7f4761600640 1 -- 172.21.15.40:0/376525831 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c036040 con 0x7f476415f420 2024-04-26T23:08:19.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.568+0000 7f476a800640 1 -- 172.21.15.40:0/376525831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47640b9130 con 0x7f476415f420 2024-04-26T23:08:19.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.568+0000 7f476a800640 1 -- 172.21.15.40:0/376525831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47640b96f0 con 0x7f476415f420 2024-04-26T23:08:19.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.570+0000 7f4761600640 1 -- 172.21.15.40:0/376525831 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f475c044020 con 0x7f476415f420 2024-04-26T23:08:19.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.571+0000 7f4761600640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f474406c8f0 0x7f474406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.571+0000 7f4761600640 1 -- 172.21.15.40:0/376525831 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f475c0ac990 con 0x7f476415f420 2024-04-26T23:08:19.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.572+0000 7f4769400640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f474406c8f0 0x7f474406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.573+0000 7f4769400640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f474406c8f0 0x7f474406edb0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4758006fd0 tx=0x7f4758008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.573+0000 7f476eb77740 1 Processor -- start 2024-04-26T23:08:19.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.574+0000 7f476eb77740 1 -- start start 2024-04-26T23:08:19.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.575+0000 7f476eb77740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 0x5632dcd329a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.575+0000 7f476eb77740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 0x5632dcd624b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.575+0000 7f476eb77740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd629f0 0x5632dcd37800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.575+0000 7f476eb77740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5632dcc9cb80 con 0x5632dcd629f0 2024-04-26T23:08:19.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.575+0000 7f476eb77740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5632dcc39420 con 0x5632dcd80010 2024-04-26T23:08:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.575+0000 7f476eb77740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5632dcc98b40 con 0x5632dcd63130 2024-04-26T23:08:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.575+0000 7f4769e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd629f0 0x5632dcd37800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.576+0000 7f4769e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd629f0 0x5632dcd37800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54824/0 (socket says 172.21.15.40:54824) 2024-04-26T23:08:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.576+0000 7f4769e00640 1 -- 172.21.15.40:0/49041682 learned_addr learned my addr 172.21.15.40:0/49041682 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.576+0000 7f4769400640 1 --2- 172.21.15.40:0/49041682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 0x5632dcd329a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.576+0000 7f4769e00640 1 -- 172.21.15.40:0/49041682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 msgr2=0x5632dcd624b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.576+0000 7f4769e00640 1 --2- 172.21.15.40:0/49041682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 0x5632dcd624b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.576+0000 7f4769e00640 1 -- 172.21.15.40:0/49041682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 msgr2=0x5632dcd329a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.576+0000 7f4769e00640 1 --2- 172.21.15.40:0/49041682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 0x5632dcd329a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.576+0000 7f4769e00640 1 -- 172.21.15.40:0/49041682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632dca76a50 con 0x5632dcd629f0 2024-04-26T23:08:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.576+0000 7f4769400640 1 --2- 172.21.15.40:0/49041682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 0x5632dcd329a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.576+0000 7f4769e00640 1 --2- 172.21.15.40:0/49041682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd629f0 0x5632dcd37800 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f475c07a160 tx=0x7f475c009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc256479856448a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.576+0000 7f4755c00640 1 -- 172.21.15.40:0/49041682 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c092020 con 0x5632dcd629f0 2024-04-26T23:08:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.577+0000 7f4755c00640 1 -- 172.21.15.40:0/49041682 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f475c07e8d0 con 0x5632dcd629f0 2024-04-26T23:08:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.577+0000 7f4755c00640 1 -- 172.21.15.40:0/49041682 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c083100 con 0x5632dcd629f0 2024-04-26T23:08:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.577+0000 7f476eb77740 1 -- 172.21.15.40:0/49041682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd629f0 msgr2=0x5632dcd37800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.577+0000 7f476eb77740 1 --2- 172.21.15.40:0/49041682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd629f0 0x5632dcd37800 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f475c07a160 tx=0x7f475c009810 comp rx=0 tx=0).stop 2024-04-26T23:08:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.577+0000 7f476eb77740 1 -- 172.21.15.40:0/49041682 shutdown_connections 2024-04-26T23:08:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.577+0000 7f476eb77740 1 --2- 172.21.15.40:0/49041682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd629f0 0x5632dcd37800 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.578+0000 7f476eb77740 1 --2- 172.21.15.40:0/49041682 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 0x5632dcd624b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.578+0000 7f476eb77740 1 --2- 172.21.15.40:0/49041682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 0x5632dcd329a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.578+0000 7f476eb77740 1 -- 172.21.15.40:0/49041682 >> 172.21.15.40:0/49041682 conn(0x5632dcd34c10 msgr2=0x5632dcd7d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.578+0000 7f476eb77740 1 -- 172.21.15.40:0/49041682 shutdown_connections 2024-04-26T23:08:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.578+0000 7f476eb77740 1 -- 172.21.15.40:0/49041682 wait complete. 2024-04-26T23:08:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.578+0000 7f476eb77740 1 -- start start 2024-04-26T23:08:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.579+0000 7f476eb77740 1 Processor -- start 2024-04-26T23:08:19.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.579+0000 7f476eb77740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 0x5632dcd8d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.579+0000 7f476eb77740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 0x5632dcd37250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f476eb77740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 0x5632dcd37790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f476eb77740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5632dcc9cb80 con 0x5632dcd4ffe0 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f476eb77740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5632dcc24370 con 0x5632dcd80010 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f476eb77740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5632dcd36dc0 con 0x5632dcd63130 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f476eb77740 10 client.0 _ll_register_callbacks cb 0x5632dcc93890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f4769e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 0x5632dcd37790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f4769e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 0x5632dcd37790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54836/0 (socket says 172.21.15.40:54836) 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f4769e00640 1 -- 172.21.15.40:0/848179661 learned_addr learned my addr 172.21.15.40:0/848179661 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f4768a00640 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 0x5632dcd37250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f4769400640 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 0x5632dcd8d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f4769e00640 1 -- 172.21.15.40:0/848179661 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 msgr2=0x5632dcd8d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f4769e00640 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 0x5632dcd8d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f4769e00640 1 -- 172.21.15.40:0/848179661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 msgr2=0x5632dcd37250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f4769e00640 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 0x5632dcd37250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f4769e00640 1 -- 172.21.15.40:0/848179661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632dc96e3e0 con 0x5632dcd4ffe0 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.580+0000 7f4769400640 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 0x5632dcd8d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.581+0000 7f4769e00640 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 0x5632dcd37790 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f475c083070 tx=0x7f475c02a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab06b897e07d5de server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.581+0000 7f474a000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.581+0000 7f474a000640 1 -- 172.21.15.40:0/848179661 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c097070 con 0x5632dcd4ffe0 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.581+0000 7f474a000640 1 -- 172.21.15.40:0/848179661 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f475c079970 con 0x5632dcd4ffe0 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.581+0000 7f476eb77740 1 -- 172.21.15.40:0/848179661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f475c07e8d0 con 0x5632dcd4ffe0 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.581+0000 7f474a000640 1 -- 172.21.15.40:0/848179661 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f475c092040 con 0x5632dcd4ffe0 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.581+0000 7f476eb77740 10 client.8394 fetch_fsmap learned FSMap version 99 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.581+0000 7f476eb77740 1 -- 172.21.15.40:0/848179661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x5632dcad71e0 con 0x5632dcd4ffe0 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.581+0000 7f474a000640 1 -- 172.21.15.40:0/848179661 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f475c07e8d0 con 0x5632dcd4ffe0 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.582+0000 7f474a000640 1 -- 172.21.15.40:0/848179661 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 99) ==== 3550+0+0 (secure 0 0 0) 0x7f475c076270 con 0x5632dcd4ffe0 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.582+0000 7f476eb77740 10 client.8394 fetch_fsmap finished waiting for FSMap version 99 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.582+0000 7f476eb77740 10 client.8394 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.582+0000 7f476eb77740 20 client.8394 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.582+0000 7f476eb77740 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x5632dcd7b530 0x5632dcd7b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.582+0000 7f476eb77740 4 client.8394 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.582+0000 7f476eb77740 1 -- 172.21.15.40:0/848179661 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632dccaa790 con 0x5632dcd7b530 2024-04-26T23:08:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.582+0000 7f4769400640 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x5632dcd7b530 0x5632dcd7b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.583+0000 7f4769400640 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x5632dcd7b530 0x5632dcd7b960 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20cb8852a1984449 server_cookie=12040f68c90f837c in_seq=0 out_seq=0 2024-04-26T23:08:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.583+0000 7f474a000640 10 client.8394 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.583+0000 7f474a000640 1 -- 172.21.15.40:0/848179661 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632dccaa790 con 0x5632dcd7b530 2024-04-26T23:08:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.583+0000 7f474a000640 10 client.8394 handle_command_reply: tid=0 2024-04-26T23:08:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.583+0000 7f476eb77740 1 client.8394 shutdown 2024-04-26T23:08:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.584+0000 7f476eb77740 1 -- 172.21.15.40:0/848179661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 msgr2=0x5632dcd37790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.584+0000 7f476eb77740 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 0x5632dcd37790 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f475c083070 tx=0x7f475c02a630 comp rx=0 tx=0).stop 2024-04-26T23:08:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.584+0000 7f476eb77740 1 -- 172.21.15.40:0/848179661 shutdown_connections 2024-04-26T23:08:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.584+0000 7f476eb77740 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x5632dcd7b530 0x5632dcd7b960 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.584+0000 7f476eb77740 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 0x5632dcd37790 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.584+0000 7f476eb77740 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 0x5632dcd37250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.584+0000 7f474a000640 0 client.8394 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.584+0000 7f476eb77740 1 --2- 172.21.15.40:0/848179661 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 0x5632dcd8d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.584+0000 7f476eb77740 1 -- 172.21.15.40:0/848179661 >> 172.21.15.40:0/848179661 conn(0x5632dcd629f0 msgr2=0x5632dcd320c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.584+0000 7f476eb77740 1 -- 172.21.15.40:0/848179661 shutdown_connections 2024-04-26T23:08:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.584+0000 7f476eb77740 1 -- 172.21.15.40:0/848179661 wait complete. 2024-04-26T23:08:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.584+0000 7f476eb77740 20 client.8394 trim_cache size 0 max 16384 2024-04-26T23:08:19.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.622+0000 7f476eb77740 1 Processor -- start 2024-04-26T23:08:19.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.622+0000 7f476eb77740 1 -- start start 2024-04-26T23:08:19.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.623+0000 7f476eb77740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 0x5632dcd388d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.623+0000 7f476eb77740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 0x5632dcd8cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.623+0000 7f476eb77740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 0x5632dcd8d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.623+0000 7f476eb77740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4764201dc0 con 0x5632dcd4ffe0 2024-04-26T23:08:19.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.623+0000 7f476eb77740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4764200f60 con 0x5632dcd80010 2024-04-26T23:08:19.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.623+0000 7f476eb77740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4764201690 con 0x5632dcd63130 2024-04-26T23:08:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.624+0000 7f4769400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 0x5632dcd388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.624+0000 7f4769400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 0x5632dcd388d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54842/0 (socket says 172.21.15.40:54842) 2024-04-26T23:08:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.624+0000 7f4769400640 1 -- 172.21.15.40:0/1057609578 learned_addr learned my addr 172.21.15.40:0/1057609578 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.624+0000 7f4769e00640 1 --2- 172.21.15.40:0/1057609578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 0x5632dcd8d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.624+0000 7f4768a00640 1 --2- 172.21.15.40:0/1057609578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 0x5632dcd8cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.624+0000 7f4769400640 1 -- 172.21.15.40:0/1057609578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 msgr2=0x5632dcd8cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.624+0000 7f4769400640 1 --2- 172.21.15.40:0/1057609578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 0x5632dcd8cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.624+0000 7f4769400640 1 -- 172.21.15.40:0/1057609578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 msgr2=0x5632dcd8d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.624+0000 7f4769400640 1 --2- 172.21.15.40:0/1057609578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 0x5632dcd8d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.624+0000 7f4769400640 1 -- 172.21.15.40:0/1057609578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632dc9474a0 con 0x5632dcd4ffe0 2024-04-26T23:08:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.624+0000 7f4768a00640 1 --2- 172.21.15.40:0/1057609578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 0x5632dcd8cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.624+0000 7f4769400640 1 --2- 172.21.15.40:0/1057609578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 0x5632dcd388d0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f475800f910 tx=0x7f4758015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b1209c5ce69e6a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.625+0000 7f4755200640 1 -- 172.21.15.40:0/1057609578 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475802d070 con 0x5632dcd4ffe0 2024-04-26T23:08:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.625+0000 7f4755200640 1 -- 172.21.15.40:0/1057609578 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4758027070 con 0x5632dcd4ffe0 2024-04-26T23:08:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.625+0000 7f4755200640 1 -- 172.21.15.40:0/1057609578 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4758027220 con 0x5632dcd4ffe0 2024-04-26T23:08:19.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.625+0000 7f476eb77740 1 -- 172.21.15.40:0/1057609578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 msgr2=0x5632dcd388d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.625+0000 7f476eb77740 1 --2- 172.21.15.40:0/1057609578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 0x5632dcd388d0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f475800f910 tx=0x7f4758015000 comp rx=0 tx=0).stop 2024-04-26T23:08:19.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.626+0000 7f476eb77740 1 -- 172.21.15.40:0/1057609578 shutdown_connections 2024-04-26T23:08:19.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.626+0000 7f476eb77740 1 --2- 172.21.15.40:0/1057609578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd80010 0x5632dcd8d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.626+0000 7f476eb77740 1 --2- 172.21.15.40:0/1057609578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd63130 0x5632dcd8cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.626+0000 7f476eb77740 1 --2- 172.21.15.40:0/1057609578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd4ffe0 0x5632dcd388d0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.626+0000 7f476eb77740 1 -- 172.21.15.40:0/1057609578 >> 172.21.15.40:0/1057609578 conn(0x5632dcd629f0 msgr2=0x5632dcd66230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.626+0000 7f476eb77740 1 -- 172.21.15.40:0/1057609578 shutdown_connections 2024-04-26T23:08:19.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.626+0000 7f476eb77740 1 -- 172.21.15.40:0/1057609578 wait complete. 2024-04-26T23:08:19.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.626+0000 7f476eb77740 1 -- start start 2024-04-26T23:08:19.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.627+0000 7f476eb77740 1 Processor -- start 2024-04-26T23:08:19.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f476eb77740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd63130 0x5632dcd5ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f476eb77740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd80010 0x5632dcd808d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f476eb77740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd81500 0x5632dcd80e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f476eb77740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5632dcd2e3c0 con 0x5632dcd63130 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f476eb77740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4764150450 con 0x5632dcd81500 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f476eb77740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5632dcd32dc0 con 0x5632dcd80010 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f476eb77740 10 client.0 _ll_register_callbacks cb 0x7f475c07a060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f4769400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd63130 0x5632dcd5ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f4769400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd63130 0x5632dcd5ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54844/0 (socket says 172.21.15.40:54844) 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f4769400640 1 -- 172.21.15.40:0/1352022607 learned_addr learned my addr 172.21.15.40:0/1352022607 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f4768a00640 1 --2- 172.21.15.40:0/1352022607 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd80010 0x5632dcd808d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f4769400640 1 -- 172.21.15.40:0/1352022607 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd80010 msgr2=0x5632dcd808d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f4769e00640 1 --2- 172.21.15.40:0/1352022607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd81500 0x5632dcd80e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f4769400640 1 --2- 172.21.15.40:0/1352022607 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd80010 0x5632dcd808d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f4769400640 1 -- 172.21.15.40:0/1352022607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd81500 msgr2=0x5632dcd80e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f4769400640 1 --2- 172.21.15.40:0/1352022607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd81500 0x5632dcd80e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.628+0000 7f4769400640 1 -- 172.21.15.40:0/1352022607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632dcd65740 con 0x5632dcd63130 2024-04-26T23:08:19.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.629+0000 7f4769400640 1 --2- 172.21.15.40:0/1352022607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd63130 0x5632dcd5ce10 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f475802b040 tx=0x7f4758004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9795d1f17bf8995 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.629+0000 7f474a000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.629+0000 7f474a000640 1 -- 172.21.15.40:0/1352022607 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4758038060 con 0x5632dcd63130 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.629+0000 7f474a000640 1 -- 172.21.15.40:0/1352022607 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f47580276b0 con 0x5632dcd63130 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.629+0000 7f476eb77740 1 -- 172.21.15.40:0/1352022607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4758027070 con 0x5632dcd63130 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.629+0000 7f474a000640 1 -- 172.21.15.40:0/1352022607 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f475803e050 con 0x5632dcd63130 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.629+0000 7f476eb77740 10 client.8400 fetch_fsmap learned FSMap version 99 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.629+0000 7f476eb77740 1 -- 172.21.15.40:0/1352022607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x5632dcd7eae0 con 0x5632dcd63130 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.630+0000 7f474a000640 1 -- 172.21.15.40:0/1352022607 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4758027070 con 0x5632dcd63130 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.630+0000 7f474a000640 1 -- 172.21.15.40:0/1352022607 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 99) ==== 3550+0+0 (secure 0 0 0) 0x7f4758041020 con 0x5632dcd63130 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.630+0000 7f476eb77740 10 client.8400 fetch_fsmap finished waiting for FSMap version 99 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.630+0000 7f476eb77740 10 client.8400 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.630+0000 7f476eb77740 20 client.8400 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.630+0000 7f476eb77740 1 --2- 172.21.15.40:0/1352022607 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x5632dcd5d350 0x5632dcd72e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.630+0000 7f476eb77740 4 client.8400 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/d1/d2/d3", "scrubops": ["recursive", "force"]}] 2024-04-26T23:08:19.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.630+0000 7f476eb77740 1 -- 172.21.15.40:0/1352022607 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "scrub start", "path": "/d1/d2/d3", "scrubops": ["recursive", "force"]}) -- 0x5632dcca4600 con 0x5632dcd5d350 2024-04-26T23:08:19.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.630+0000 7f4768a00640 1 --2- 172.21.15.40:0/1352022607 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x5632dcd5d350 0x5632dcd72e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.631+0000 7f4768a00640 1 --2- 172.21.15.40:0/1352022607 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x5632dcd5d350 0x5632dcd72e40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51ae96a8f391623c server_cookie=a1d35819ed2f370d in_seq=0 out_seq=0 2024-04-26T23:08:19.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.631+0000 7f474a000640 10 client.8400 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:19.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f474a000640 1 -- 172.21.15.40:0/1352022607 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5632dcca4600 con 0x5632dcd5d350 2024-04-26T23:08:19.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f474a000640 10 client.8400 handle_command_reply: tid=0 2024-04-26T23:08:19.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f476eb77740 1 client.8400 shutdown 2024-04-26T23:08:19.634 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:08:19.632+0000 7fd9dc000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000002 [...2,head] /d1/d2/d3/ auth{1=1,2=1} v19 ap=1 f(v0 m2024-04-26T23:08:01.996726+0000 1=0+1) n(v3 rc2024-04-26T23:08:02.226723+0000 6=0+6) (inest mix dirty) (ifile excl) caps={6328=pAsLsXsFsx/pAsLsXsFsxcral@4},l=6328 | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 0x564f1e1c6580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000002:[<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:08:01.996726+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:08:02.226723+0000 6=0+6)","memory_value.dirstat":"f(v0 m2024-04-26T23:08:01.996726+0000 1=0+1)","memory_value.rstat":"n(v3 rc2024-04-26T23:08:02.226723+0000 6=0+6)","error_str":""},"return_code":-61} 2024-04-26T23:08:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f476eb77740 1 -- 172.21.15.40:0/1352022607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd63130 msgr2=0x5632dcd5ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f476eb77740 1 --2- 172.21.15.40:0/1352022607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd63130 0x5632dcd5ce10 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f475802b040 tx=0x7f4758004730 comp rx=0 tx=0).stop 2024-04-26T23:08:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f476eb77740 1 -- 172.21.15.40:0/1352022607 shutdown_connections 2024-04-26T23:08:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f476eb77740 1 --2- 172.21.15.40:0/1352022607 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x5632dcd5d350 0x5632dcd72e40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f476eb77740 1 --2- 172.21.15.40:0/1352022607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632dcd81500 0x5632dcd80e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f476eb77740 1 --2- 172.21.15.40:0/1352022607 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632dcd80010 0x5632dcd808d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f476eb77740 1 --2- 172.21.15.40:0/1352022607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632dcd63130 0x5632dcd5ce10 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f474a000640 0 client.8400 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f476eb77740 1 -- 172.21.15.40:0/1352022607 >> 172.21.15.40:0/1352022607 conn(0x5632dcd629f0 msgr2=0x5632dcd37190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f476eb77740 1 -- 172.21.15.40:0/1352022607 shutdown_connections 2024-04-26T23:08:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.632+0000 7f476eb77740 1 -- 172.21.15.40:0/1352022607 wait complete. 2024-04-26T23:08:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.633+0000 7f476eb77740 20 client.8400 trim_cache size 0 max 16384 2024-04-26T23:08:19.636 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:19.636 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-04-26T23:08:19.636 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "b17426f8-9d25-4fbb-a99f-95207d3f0c9f", 2024-04-26T23:08:19.636 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-04-26T23:08:19.636 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:19.637 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-04-26T23:08:19.633+0000 7f9394e00640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000004 [...2,head] /d1/d2/d3/d4/d5/ auth{0=1,2=1} v15 ap=1 f(v0 m2024-04-26T23:08:02.000726+0000 1=0+1) n(v2 rc2024-04-26T23:08:02.226723+0000 4=0+4) (inest lock) (ifile excl) caps={6328=pAsLsXsFsx/pAsLsXsFsxcral@4},l=6328 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 authpin=1 scrubqueue=0 0x55fe21ffec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000004:[<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:08:02.000726+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:08:02.226723+0000 4=0+4)","memory_value.dirstat":"f(v0 m2024-04-26T23:08:02.000726+0000 1=0+1)","memory_value.rstat":"n(v2 rc2024-04-26T23:08:02.226723+0000 4=0+4)","error_str":""},"return_code":-61} 2024-04-26T23:08:19.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.633+0000 7f476a800640 1 -- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f474406c8f0 msgr2=0x7f474406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.633+0000 7f476a800640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f474406c8f0 0x7f474406edb0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4758006fd0 tx=0x7f4758008040 comp rx=0 tx=0).stop 2024-04-26T23:08:19.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.633+0000 7f476a800640 1 -- 172.21.15.40:0/376525831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f476415f420 msgr2=0x7f47640ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.633+0000 7f476a800640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f476415f420 0x7f47640ce2a0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f475c009e50 tx=0x7f475c004a40 comp rx=0 tx=0).stop 2024-04-26T23:08:19.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.633+0000 7f476a800640 1 -- 172.21.15.40:0/376525831 shutdown_connections 2024-04-26T23:08:19.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.633+0000 7f476a800640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f474406c8f0 0x7f474406edb0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.633+0000 7f476a800640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f476415f420 0x7f47640ce2a0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.633+0000 7f476a800640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f47640d3b60 0x7f47640b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.633+0000 7f476a800640 1 --2- 172.21.15.40:0/376525831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47640b3d90 0x7f47640cc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.633+0000 7f476a800640 1 -- 172.21.15.40:0/376525831 >> 172.21.15.40:0/376525831 conn(0x7f476409e1d0 msgr2=0x7f476406ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.634+0000 7f476a800640 1 -- 172.21.15.40:0/376525831 shutdown_connections 2024-04-26T23:08:19.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.634+0000 7f476a800640 1 -- 172.21.15.40:0/376525831 wait complete. 2024-04-26T23:08:19.638 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:08:19.634+0000 7fd9e4200640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000003 [...2,head] /d1/d2/d3/d4/ auth{1=1,2=1} v19 ap=2 f(v0 m2024-04-26T23:08:01.998726+0000 1=0+1) n(v4 rc2024-04-26T23:08:02.226723+0000 5=0+5) (inest sync->mix g=1,2) caps={6328=pAsLsXsFs/pAsLsXsFsxcral@8} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 export_pin=1 0x564f1e1c8680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000003:[<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:08:01.998726+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:08:02.226723+0000 5=0+5)","memory_value.dirstat":"f(v0 m2024-04-26T23:08:01.998726+0000 1=0+1)","memory_value.rstat":"n(v4 rc2024-04-26T23:08:02.226723+0000 5=0+5)","error_str":""},"return_code":-61} 2024-04-26T23:08:19.639 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-04-26T23:08:19.636+0000 7f939d000640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000005 [...2,head] /d1/d2/d3/d4/d5/d6/ auth{0=1,2=1} v11 ap=2 f(v0 m2024-04-26T23:08:02.001726+0000 1=0+1) n(v1 rc2024-04-26T23:08:02.226723+0000 3=0+3) (inest sync->mix g=0,2 scatter_wanted) caps={6328=pAsLsXsFs/pAsLsXsFsxcral@7} | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 export_pin=2 0x55fe21fff700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000005:[<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:08:02.001726+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:08:02.003726+0000 3=0+3)","memory_value.dirstat":"f(v0 m2024-04-26T23:08:02.001726+0000 1=0+1)","memory_value.rstat":"n(v1 rc2024-04-26T23:08:02.226723+0000 3=0+3)","error_str":""},"return_code":-61} 2024-04-26T23:08:19.639 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-26T23:08:19.634+0000 7f8047e00640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000006 [...2,head] /d1/d2/d3/d4/d5/d6/d7/ auth v4 ap=1 f(v0 m2024-04-26T23:08:02.003726+0000 1=0+1) n(v0 rc2024-04-26T23:08:02.003726+0000 2=0+2) (inest lock) (ifile excl) caps={6328=pAsLsXsFsx/pAsLsXsFsxcral@4},l=6328 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x560ec42a2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000006:[<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:08:02.003726+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:08:02.003726+0000 2=0+2)","memory_value.dirstat":"f(v0 m2024-04-26T23:08:02.003726+0000 1=0+1)","memory_value.rstat":"n(v0 rc2024-04-26T23:08:02.003726+0000 2=0+2)","error_str":""},"return_code":-61} 2024-04-26T23:08:19.639 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-26T23:08:19.635+0000 7f8047e00640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000007 [...2,head] /d1/d2/d3/d4/d5/d6/d7/d8/ auth v2 ap=1 f() n(v0 1=0+1) (inest lock) (ifile excl) caps={6328=pAsLsXsFsx/AsLsXsFsx@2},l=6328 | importingcaps=0 dirfrag=0 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x560ec42a2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000007:[<0x10000000006/d8 v2>,<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 1=0+1)","error_str":""},"return_code":-61} 2024-04-26T23:08:19.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:08:19.653 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub pause 2024-04-26T23:08:19.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.839+0000 7f8f60c00640 1 Processor -- start 2024-04-26T23:08:19.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.839+0000 7f8f60c00640 1 -- start start 2024-04-26T23:08:19.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.841+0000 7f8f60c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f5c1467e0 0x7f8f5c0c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.841+0000 7f8f60c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8f5c06c240 0x7f8f5c06c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.841+0000 7f8f60c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f5c0c6e40 con 0x7f8f5c06ee30 2024-04-26T23:08:19.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.841+0000 7f8f60c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f8f5c0c70e0 con 0x7f8f5c1467e0 2024-04-26T23:08:19.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.841+0000 7f8f60c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8f5c0c73c0 con 0x7f8f5c06c240 2024-04-26T23:08:19.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.841+0000 7f8f5b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f5c1467e0 0x7f8f5c0c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.841+0000 7f8f5b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f5c1467e0 0x7f8f5c0c7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54846/0 (socket says 172.21.15.40:54846) 2024-04-26T23:08:19.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.841+0000 7f8f5b400640 1 -- 172.21.15.40:0/4254503067 learned_addr learned my addr 172.21.15.40:0/4254503067 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.842+0000 7f8f5b400640 1 -- 172.21.15.40:0/4254503067 >> v2:172.21.15.103:3300/0 conn(0x7f8f5c06c240 msgr2=0x7f8f5c06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.842+0000 7f8f5b400640 1 --2- 172.21.15.40:0/4254503067 >> v2:172.21.15.103:3300/0 conn(0x7f8f5c06c240 0x7f8f5c06c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.842+0000 7f8f5b400640 1 -- 172.21.15.40:0/4254503067 >> v1:172.21.15.103:6789/0 conn(0x7f8f5c06ee30 legacy=0x7f8f5c06f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.842+0000 7f8f5b400640 1 -- 172.21.15.40:0/4254503067 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5c0c76a0 con 0x7f8f5c1467e0 2024-04-26T23:08:19.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.843+0000 7f8f5b400640 1 --2- 172.21.15.40:0/4254503067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f5c1467e0 0x7f8f5c0c7bc0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009870 tx=0x7f8f4c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39bd8c1c2f7dd444 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.843+0000 7f8f5a000640 1 -- 172.21.15.40:0/4254503067 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c037070 con 0x7f8f5c1467e0 2024-04-26T23:08:19.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.843+0000 7f8f5a000640 1 -- 172.21.15.40:0/4254503067 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f4c032440 con 0x7f8f5c1467e0 2024-04-26T23:08:19.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.843+0000 7f8f5a000640 1 -- 172.21.15.40:0/4254503067 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c032810 con 0x7f8f5c1467e0 2024-04-26T23:08:19.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.843+0000 7f8f60c00640 1 -- 172.21.15.40:0/4254503067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f5c1467e0 msgr2=0x7f8f5c0c7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.843+0000 7f8f60c00640 1 --2- 172.21.15.40:0/4254503067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f5c1467e0 0x7f8f5c0c7bc0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009870 tx=0x7f8f4c02a270 comp rx=0 tx=0).stop 2024-04-26T23:08:19.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.844+0000 7f8f60c00640 1 -- 172.21.15.40:0/4254503067 shutdown_connections 2024-04-26T23:08:19.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.844+0000 7f8f60c00640 1 --2- 172.21.15.40:0/4254503067 >> v2:172.21.15.103:3300/0 conn(0x7f8f5c06c240 0x7f8f5c06c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.844+0000 7f8f60c00640 1 --2- 172.21.15.40:0/4254503067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f5c1467e0 0x7f8f5c0c7bc0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.844+0000 7f8f60c00640 1 -- 172.21.15.40:0/4254503067 >> 172.21.15.40:0/4254503067 conn(0x7f8f5c0d1ae0 msgr2=0x7f8f5c0d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.844+0000 7f8f60c00640 1 -- 172.21.15.40:0/4254503067 shutdown_connections 2024-04-26T23:08:19.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.844+0000 7f8f60c00640 1 -- 172.21.15.40:0/4254503067 wait complete. 2024-04-26T23:08:19.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.845+0000 7f8f60c00640 1 Processor -- start 2024-04-26T23:08:19.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.846+0000 7f8f60c00640 1 -- start start 2024-04-26T23:08:19.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.846+0000 7f8f60c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f5c06c240 0x7f8f5c15b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.846+0000 7f8f60c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8f5c06ee30 0x7f8f5c15d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.846+0000 7f8f60c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f5c1467e0 0x7f8f5c0b0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.847+0000 7f8f60c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c0c8280 con 0x7f8f5c06c240 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.847+0000 7f8f60c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c0c8100 con 0x7f8f5c1467e0 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.847+0000 7f8f60c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8f5c160ad0 con 0x7f8f5c06ee30 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.847+0000 7f8f5b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f5c06c240 0x7f8f5c15b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.847+0000 7f8f5b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f5c06c240 0x7f8f5c15b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54858/0 (socket says 172.21.15.40:54858) 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.847+0000 7f8f5b400640 1 -- 172.21.15.40:0/3025017397 learned_addr learned my addr 172.21.15.40:0/3025017397 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.847+0000 7f8f5b400640 1 -- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8f5c06ee30 msgr2=0x7f8f5c15d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.847+0000 7f8f5aa00640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8f5c06ee30 0x7f8f5c15d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.847+0000 7f8f5b400640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8f5c06ee30 0x7f8f5c15d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.847+0000 7f8f5be00640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f5c1467e0 0x7f8f5c0b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.847+0000 7f8f5b400640 1 -- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f5c1467e0 msgr2=0x7f8f5c0b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.847+0000 7f8f5b400640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f5c1467e0 0x7f8f5c0b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.847+0000 7f8f5b400640 1 -- 172.21.15.40:0/3025017397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5c0b0c50 con 0x7f8f5c06c240 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.848+0000 7f8f5b400640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f5c06c240 0x7f8f5c15b680 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009870 tx=0x7f8f4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.848+0000 7f8f53400640 1 -- 172.21.15.40:0/3025017397 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c037040 con 0x7f8f5c06c240 2024-04-26T23:08:19.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.848+0000 7f8f53400640 1 -- 172.21.15.40:0/3025017397 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f4c004230 con 0x7f8f5c06c240 2024-04-26T23:08:19.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.848+0000 7f8f60c00640 1 -- 172.21.15.40:0/3025017397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f5c15f900 con 0x7f8f5c06c240 2024-04-26T23:08:19.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.848+0000 7f8f53400640 1 -- 172.21.15.40:0/3025017397 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c03c670 con 0x7f8f5c06c240 2024-04-26T23:08:19.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.848+0000 7f8f60c00640 1 -- 172.21.15.40:0/3025017397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f5c0b11f0 con 0x7f8f5c06c240 2024-04-26T23:08:19.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.851+0000 7f8f53400640 1 -- 172.21.15.40:0/3025017397 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8f4c03a070 con 0x7f8f5c06c240 2024-04-26T23:08:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.851+0000 7f8f53400640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f3c06c8f0 0x7f8f3c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.852+0000 7f8f53400640 1 -- 172.21.15.40:0/3025017397 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f8f4c0ac740 con 0x7f8f5c06c240 2024-04-26T23:08:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.852+0000 7f8f65018740 1 Processor -- start 2024-04-26T23:08:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.852+0000 7f8f65018740 1 -- start start 2024-04-26T23:08:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.852+0000 7f8f5aa00640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f3c06c8f0 0x7f8f3c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.853+0000 7f8f65018740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bd5140 0x55bce9bd5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.853+0000 7f8f65018740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 0x55bce9bf2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.853+0000 7f8f65018740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9baae20 0x55bce9bf10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.853+0000 7f8f65018740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bce9b0e900 con 0x55bce9bf29e0 2024-04-26T23:08:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.853+0000 7f8f65018740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bce9aab1d0 con 0x55bce9baae20 2024-04-26T23:08:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.853+0000 7f8f65018740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bce9b0a8c0 con 0x55bce9bd5140 2024-04-26T23:08:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.853+0000 7f8f5aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 0x55bce9bf2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.853+0000 7f8f5aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 0x55bce9bf2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54866/0 (socket says 172.21.15.40:54866) 2024-04-26T23:08:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.853+0000 7f8f5b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bd5140 0x55bce9bd5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.853+0000 7f8f5aa00640 1 -- 172.21.15.40:0/1484296190 learned_addr learned my addr 172.21.15.40:0/1484296190 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.854+0000 7f8f5aa00640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f3c06c8f0 0x7f8f3c06edb0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0b14f0 tx=0x7f8f48008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.854+0000 7f8f5be00640 1 --2- 172.21.15.40:0/1484296190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9baae20 0x55bce9bf10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.854+0000 7f8f5aa00640 1 -- 172.21.15.40:0/1484296190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bd5140 msgr2=0x55bce9bd5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.854+0000 7f8f5aa00640 1 --2- 172.21.15.40:0/1484296190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bd5140 0x55bce9bd5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.854+0000 7f8f5aa00640 1 -- 172.21.15.40:0/1484296190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9baae20 msgr2=0x55bce9bf10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.854+0000 7f8f5aa00640 1 --2- 172.21.15.40:0/1484296190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9baae20 0x55bce9bf10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.854+0000 7f8f5aa00640 1 -- 172.21.15.40:0/1484296190 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bce98ea7d0 con 0x55bce9bf29e0 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.854+0000 7f8f5aa00640 1 --2- 172.21.15.40:0/1484296190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 0x55bce9bf2f30 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800ca30 tx=0x7f8f4800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc1e6ae29197019 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.855+0000 7f8f47400640 1 -- 172.21.15.40:0/1484296190 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f48023070 con 0x55bce9bf29e0 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.855+0000 7f8f47400640 1 -- 172.21.15.40:0/1484296190 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f48013db0 con 0x55bce9bf29e0 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.855+0000 7f8f47400640 1 -- 172.21.15.40:0/1484296190 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4801e620 con 0x55bce9bf29e0 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.855+0000 7f8f65018740 1 -- 172.21.15.40:0/1484296190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 msgr2=0x55bce9bf2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.855+0000 7f8f65018740 1 --2- 172.21.15.40:0/1484296190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 0x55bce9bf2f30 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800ca30 tx=0x7f8f4800f000 comp rx=0 tx=0).stop 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.855+0000 7f8f65018740 1 -- 172.21.15.40:0/1484296190 shutdown_connections 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.855+0000 7f8f65018740 1 --2- 172.21.15.40:0/1484296190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9baae20 0x55bce9bf10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.855+0000 7f8f65018740 1 --2- 172.21.15.40:0/1484296190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 0x55bce9bf2f30 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.855+0000 7f8f65018740 1 --2- 172.21.15.40:0/1484296190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bd5140 0x55bce9bd5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.855+0000 7f8f65018740 1 -- 172.21.15.40:0/1484296190 >> 172.21.15.40:0/1484296190 conn(0x55bce9ba7750 msgr2=0x55bce9beff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.855+0000 7f8f65018740 1 -- 172.21.15.40:0/1484296190 shutdown_connections 2024-04-26T23:08:19.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.855+0000 7f8f65018740 1 -- 172.21.15.40:0/1484296190 wait complete. 2024-04-26T23:08:19.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.856+0000 7f8f65018740 1 -- start start 2024-04-26T23:08:19.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.856+0000 7f8f65018740 1 Processor -- start 2024-04-26T23:08:19.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.857+0000 7f8f65018740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9bd5140 0x55bce9be7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.857+0000 7f8f65018740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bf29e0 0x55bce9bd3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.857+0000 7f8f65018740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9be7000 0x55bce9bebd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.857+0000 7f8f65018740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bce9babbc0 con 0x55bce9be7000 2024-04-26T23:08:19.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.857+0000 7f8f65018740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bce9a96120 con 0x55bce9bd5140 2024-04-26T23:08:19.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.857+0000 7f8f65018740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bce9ba5680 con 0x55bce9bf29e0 2024-04-26T23:08:19.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.857+0000 7f8f65018740 10 client.0 _ll_register_callbacks cb 0x55bce9ace750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.858+0000 7f8f5be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9be7000 0x55bce9bebd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.858+0000 7f8f5be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9be7000 0x55bce9bebd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54874/0 (socket says 172.21.15.40:54874) 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.858+0000 7f8f5be00640 1 -- 172.21.15.40:0/642818624 learned_addr learned my addr 172.21.15.40:0/642818624 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.858+0000 7f8f5b400640 1 --2- 172.21.15.40:0/642818624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9bd5140 0x55bce9be7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.858+0000 7f8f5aa00640 1 --2- 172.21.15.40:0/642818624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bf29e0 0x55bce9bd3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.858+0000 7f8f5be00640 1 -- 172.21.15.40:0/642818624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bf29e0 msgr2=0x55bce9bd3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.858+0000 7f8f5be00640 1 --2- 172.21.15.40:0/642818624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bf29e0 0x55bce9bd3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.858+0000 7f8f5be00640 1 -- 172.21.15.40:0/642818624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9bd5140 msgr2=0x55bce9be7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.858+0000 7f8f5be00640 1 --2- 172.21.15.40:0/642818624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9bd5140 0x55bce9be7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.858+0000 7f8f5be00640 1 -- 172.21.15.40:0/642818624 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bce97e00f0 con 0x55bce9be7000 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.858+0000 7f8f5be00640 1 --2- 172.21.15.40:0/642818624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9be7000 0x55bce9bebd70 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400f560 tx=0x7f8f5400fa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3699839d56ba5f55 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.859+0000 7f8f3be00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.859+0000 7f8f3be00640 1 -- 172.21.15.40:0/642818624 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f54003b80 con 0x55bce9be7000 2024-04-26T23:08:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.859+0000 7f8f3be00640 1 -- 172.21.15.40:0/642818624 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f8f54003d20 con 0x55bce9be7000 2024-04-26T23:08:19.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.859+0000 7f8f65018740 1 -- 172.21.15.40:0/642818624 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f48013db0 con 0x55bce9be7000 2024-04-26T23:08:19.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.859+0000 7f8f3be00640 1 -- 172.21.15.40:0/642818624 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f8f540050d0 con 0x55bce9be7000 2024-04-26T23:08:19.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.859+0000 7f8f65018740 10 client.8412 fetch_fsmap learned FSMap version 99 2024-04-26T23:08:19.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.859+0000 7f8f65018740 1 -- 172.21.15.40:0/642818624 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x55bce994af60 con 0x55bce9be7000 2024-04-26T23:08:19.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.859+0000 7f8f3be00640 1 -- 172.21.15.40:0/642818624 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f48013db0 con 0x55bce9be7000 2024-04-26T23:08:19.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.859+0000 7f8f3be00640 1 -- 172.21.15.40:0/642818624 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 99) ==== 3550+0+0 (secure 0 0 0) 0x7f8f54023020 con 0x55bce9be7000 2024-04-26T23:08:19.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.860+0000 7f8f65018740 10 client.8412 fetch_fsmap finished waiting for FSMap version 99 2024-04-26T23:08:19.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.860+0000 7f8f65018740 10 client.8412 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-04-26T23:08:19.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.860+0000 7f8f65018740 20 client.8412 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:19.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.860+0000 7f8f65018740 1 --2- 172.21.15.40:0/642818624 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55bce9c03ea0 0x55bce9c042d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.860+0000 7f8f65018740 4 client.8412 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:19.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.860+0000 7f8f65018740 1 -- 172.21.15.40:0/642818624 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bce9b1c510 con 0x55bce9c03ea0 2024-04-26T23:08:19.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.860+0000 7f8f5b400640 1 --2- 172.21.15.40:0/642818624 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55bce9c03ea0 0x55bce9c042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.860+0000 7f8f5b400640 1 --2- 172.21.15.40:0/642818624 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55bce9c03ea0 0x55bce9c042d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b98ea380a0e911d1 server_cookie=72b6bbc3aae90eec in_seq=0 out_seq=0 2024-04-26T23:08:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.860+0000 7f8f3be00640 10 client.8412 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.861+0000 7f8f3be00640 1 -- 172.21.15.40:0/642818624 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bce9b1c510 con 0x55bce9c03ea0 2024-04-26T23:08:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.861+0000 7f8f3be00640 10 client.8412 handle_command_reply: tid=0 2024-04-26T23:08:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.861+0000 7f8f65018740 1 client.8412 shutdown 2024-04-26T23:08:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.861+0000 7f8f65018740 1 -- 172.21.15.40:0/642818624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9be7000 msgr2=0x55bce9bebd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.861+0000 7f8f65018740 1 --2- 172.21.15.40:0/642818624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9be7000 0x55bce9bebd70 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400f560 tx=0x7f8f5400fa20 comp rx=0 tx=0).stop 2024-04-26T23:08:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.862+0000 7f8f65018740 1 -- 172.21.15.40:0/642818624 shutdown_connections 2024-04-26T23:08:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.862+0000 7f8f65018740 1 --2- 172.21.15.40:0/642818624 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55bce9c03ea0 0x55bce9c042d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.862+0000 7f8f65018740 1 --2- 172.21.15.40:0/642818624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9be7000 0x55bce9bebd70 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.862+0000 7f8f65018740 1 --2- 172.21.15.40:0/642818624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bf29e0 0x55bce9bd3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.862+0000 7f8f65018740 1 --2- 172.21.15.40:0/642818624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9bd5140 0x55bce9be7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.862+0000 7f8f65018740 1 -- 172.21.15.40:0/642818624 >> 172.21.15.40:0/642818624 conn(0x55bce9baae20 msgr2=0x55bce9bf3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.862+0000 7f8f3be00640 0 client.8412 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.862+0000 7f8f65018740 1 -- 172.21.15.40:0/642818624 shutdown_connections 2024-04-26T23:08:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.862+0000 7f8f65018740 1 -- 172.21.15.40:0/642818624 wait complete. 2024-04-26T23:08:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.862+0000 7f8f65018740 20 client.8412 trim_cache size 0 max 16384 2024-04-26T23:08:19.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.881+0000 7f8f65018740 1 Processor -- start 2024-04-26T23:08:19.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.881+0000 7f8f65018740 1 -- start start 2024-04-26T23:08:19.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f65018740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bd5140 0x55bce9bf0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f65018740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9be7000 0x55bce9be9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f65018740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 0x55bce9bfe670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f65018740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c201df0 con 0x55bce9bf29e0 2024-04-26T23:08:19.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f65018740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c200f90 con 0x55bce9be7000 2024-04-26T23:08:19.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f65018740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8f5c2016c0 con 0x55bce9bd5140 2024-04-26T23:08:19.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f5be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 0x55bce9bfe670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f5be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 0x55bce9bfe670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54890/0 (socket says 172.21.15.40:54890) 2024-04-26T23:08:19.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f5be00640 1 -- 172.21.15.40:0/2076307425 learned_addr learned my addr 172.21.15.40:0/2076307425 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f5be00640 1 -- 172.21.15.40:0/2076307425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bd5140 msgr2=0x55bce9bf0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f5be00640 1 --2- 172.21.15.40:0/2076307425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bd5140 0x55bce9bf0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f5be00640 1 -- 172.21.15.40:0/2076307425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9be7000 msgr2=0x55bce9be9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f5be00640 1 --2- 172.21.15.40:0/2076307425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9be7000 0x55bce9be9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.882+0000 7f8f5be00640 1 -- 172.21.15.40:0/2076307425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bce97b91b0 con 0x55bce9bf29e0 2024-04-26T23:08:19.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.883+0000 7f8f5be00640 1 --2- 172.21.15.40:0/2076307425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 0x55bce9bfe670 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f8f54005960 tx=0x7f8f5400d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99af4e1ea1a8db26 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.883+0000 7f8f46a00640 1 -- 172.21.15.40:0/2076307425 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f54005470 con 0x55bce9bf29e0 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.883+0000 7f8f46a00640 1 -- 172.21.15.40:0/2076307425 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f54028030 con 0x55bce9bf29e0 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.883+0000 7f8f46a00640 1 -- 172.21.15.40:0/2076307425 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f54003da0 con 0x55bce9bf29e0 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.884+0000 7f8f65018740 1 -- 172.21.15.40:0/2076307425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 msgr2=0x55bce9bfe670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.884+0000 7f8f65018740 1 --2- 172.21.15.40:0/2076307425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 0x55bce9bfe670 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f8f54005960 tx=0x7f8f5400d000 comp rx=0 tx=0).stop 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.884+0000 7f8f65018740 1 -- 172.21.15.40:0/2076307425 shutdown_connections 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.884+0000 7f8f65018740 1 --2- 172.21.15.40:0/2076307425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bf29e0 0x55bce9bfe670 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.884+0000 7f8f65018740 1 --2- 172.21.15.40:0/2076307425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9be7000 0x55bce9be9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.884+0000 7f8f65018740 1 --2- 172.21.15.40:0/2076307425 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9bd5140 0x55bce9bf0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.884+0000 7f8f65018740 1 -- 172.21.15.40:0/2076307425 >> 172.21.15.40:0/2076307425 conn(0x55bce9baae20 msgr2=0x55bce9beb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.884+0000 7f8f65018740 1 -- 172.21.15.40:0/2076307425 shutdown_connections 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.884+0000 7f8f65018740 1 -- 172.21.15.40:0/2076307425 wait complete. 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.884+0000 7f8f65018740 1 -- start start 2024-04-26T23:08:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.885+0000 7f8f65018740 1 Processor -- start 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f65018740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bd5140 0x55bce9c13c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f65018740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9be7000 0x55bce9ba4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f65018740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9bf29e0 0x55bce9ba4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f65018740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bce9beb300 con 0x55bce9bd5140 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f65018740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c0af4e0 con 0x55bce9bf29e0 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f65018740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bce9bec110 con 0x55bce9be7000 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f65018740 10 client.0 _ll_register_callbacks cb 0x7f8f540014e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f5b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bd5140 0x55bce9c13c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f5b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bd5140 0x55bce9c13c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54892/0 (socket says 172.21.15.40:54892) 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f5b400640 1 -- 172.21.15.40:0/1189335583 learned_addr learned my addr 172.21.15.40:0/1189335583 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f5b400640 1 -- 172.21.15.40:0/1189335583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9be7000 msgr2=0x55bce9ba4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f5b400640 1 --2- 172.21.15.40:0/1189335583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9be7000 0x55bce9ba4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f5b400640 1 -- 172.21.15.40:0/1189335583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9bf29e0 msgr2=0x55bce9ba4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f5b400640 1 --2- 172.21.15.40:0/1189335583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9bf29e0 0x55bce9ba4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.886+0000 7f8f5b400640 1 -- 172.21.15.40:0/1189335583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bce9c73320 con 0x55bce9bd5140 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f5b400640 1 --2- 172.21.15.40:0/1189335583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bd5140 0x55bce9c13c20 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c076610 tx=0x7f8f4c0764d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1136c880827009f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f3be00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f3be00640 1 -- 172.21.15.40:0/1189335583 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c04a040 con 0x55bce9bd5140 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f3be00640 1 -- 172.21.15.40:0/1189335583 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f8f4c076820 con 0x55bce9bd5140 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f65018740 1 -- 172.21.15.40:0/1189335583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f54028030 con 0x55bce9bd5140 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f3be00640 1 -- 172.21.15.40:0/1189335583 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f8f4c082e20 con 0x55bce9bd5140 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f65018740 10 client.8418 fetch_fsmap learned FSMap version 99 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f65018740 1 -- 172.21.15.40:0/1189335583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x55bce9c740a0 con 0x55bce9bd5140 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f3be00640 1 -- 172.21.15.40:0/1189335583 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f54028030 con 0x55bce9bd5140 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f3be00640 1 -- 172.21.15.40:0/1189335583 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 99) ==== 3550+0+0 (secure 0 0 0) 0x7f8f4c07e130 con 0x55bce9bd5140 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f65018740 10 client.8418 fetch_fsmap finished waiting for FSMap version 99 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f65018740 10 client.8418 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f65018740 20 client.8418 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f65018740 1 --2- 172.21.15.40:0/1189335583 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55bce9bf3470 0x55bce9bee330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:19.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f65018740 4 client.8418 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "scrub pause"}] 2024-04-26T23:08:19.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.887+0000 7f8f65018740 1 -- 172.21.15.40:0/1189335583 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "scrub pause"}) -- 0x55bce9b16380 con 0x55bce9bf3470 2024-04-26T23:08:19.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.888+0000 7f8f5aa00640 1 --2- 172.21.15.40:0/1189335583 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55bce9bf3470 0x55bce9bee330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.888+0000 7f8f5aa00640 1 --2- 172.21.15.40:0/1189335583 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55bce9bf3470 0x55bce9bee330 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb20dba867e98558 server_cookie=1445dbd31baeda70 in_seq=0 out_seq=0 2024-04-26T23:08:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.888+0000 7f8f3be00640 10 client.8418 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.888+0000 7f8f3be00640 1 -- 172.21.15.40:0/1189335583 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55bce9b16380 con 0x55bce9bf3470 2024-04-26T23:08:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.888+0000 7f8f3be00640 10 client.8418 handle_command_reply: tid=0 2024-04-26T23:08:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.888+0000 7f8f65018740 1 client.8418 shutdown 2024-04-26T23:08:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.889+0000 7f8f65018740 1 -- 172.21.15.40:0/1189335583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bd5140 msgr2=0x55bce9c13c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.889+0000 7f8f65018740 1 --2- 172.21.15.40:0/1189335583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bd5140 0x55bce9c13c20 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c076610 tx=0x7f8f4c0764d0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.889+0000 7f8f65018740 1 -- 172.21.15.40:0/1189335583 shutdown_connections 2024-04-26T23:08:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.889+0000 7f8f65018740 1 --2- 172.21.15.40:0/1189335583 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x55bce9bf3470 0x55bce9bee330 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.889+0000 7f8f65018740 1 --2- 172.21.15.40:0/1189335583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bce9bf29e0 0x55bce9ba4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.889+0000 7f8f65018740 1 --2- 172.21.15.40:0/1189335583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bce9be7000 0x55bce9ba4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.889+0000 7f8f3be00640 0 client.8418 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.889+0000 7f8f65018740 1 --2- 172.21.15.40:0/1189335583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bce9bd5140 0x55bce9c13c20 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.889+0000 7f8f65018740 1 -- 172.21.15.40:0/1189335583 >> 172.21.15.40:0/1189335583 conn(0x55bce9baae20 msgr2=0x55bce9bf0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.889+0000 7f8f65018740 1 -- 172.21.15.40:0/1189335583 shutdown_connections 2024-04-26T23:08:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.889+0000 7f8f65018740 1 -- 172.21.15.40:0/1189335583 wait complete. 2024-04-26T23:08:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.889+0000 7f8f65018740 20 client.8418 trim_cache size 0 max 16384 2024-04-26T23:08:19.892 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:19.892 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:08:19.892 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:19.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.890+0000 7f8f60c00640 1 -- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f3c06c8f0 msgr2=0x7f8f3c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.890+0000 7f8f60c00640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f3c06c8f0 0x7f8f3c06edb0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0b14f0 tx=0x7f8f48008040 comp rx=0 tx=0).stop 2024-04-26T23:08:19.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.890+0000 7f8f60c00640 1 -- 172.21.15.40:0/3025017397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f5c06c240 msgr2=0x7f8f5c15b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:19.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.890+0000 7f8f60c00640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f5c06c240 0x7f8f5c15b680 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009870 tx=0x7f8f4c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.890+0000 7f8f60c00640 1 -- 172.21.15.40:0/3025017397 shutdown_connections 2024-04-26T23:08:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.890+0000 7f8f60c00640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8f3c06c8f0 0x7f8f3c06edb0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.890+0000 7f8f60c00640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f5c1467e0 0x7f8f5c0b0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.890+0000 7f8f60c00640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8f5c06ee30 0x7f8f5c15d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.890+0000 7f8f60c00640 1 --2- 172.21.15.40:0/3025017397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f5c06c240 0x7f8f5c15b680 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.890+0000 7f8f60c00640 1 -- 172.21.15.40:0/3025017397 >> 172.21.15.40:0/3025017397 conn(0x7f8f5c0d1ae0 msgr2=0x7f8f5c0b1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.890+0000 7f8f60c00640 1 -- 172.21.15.40:0/3025017397 shutdown_connections 2024-04-26T23:08:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:19.890+0000 7f8f60c00640 1 -- 172.21.15.40:0/3025017397 wait complete. 2024-04-26T23:08:19.913 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:08:19.913 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.6:1 scrub status 2024-04-26T23:08:20.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.113+0000 7f2996200640 1 Processor -- start 2024-04-26T23:08:20.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.113+0000 7f2996200640 1 -- start start 2024-04-26T23:08:20.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.115+0000 7f2996200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2990160a50 0x7f2990073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.115+0000 7f2996200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f299006dd40 0x7f2990072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.115+0000 7f2996200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2990161880 con 0x7f29900cfaf0 2024-04-26T23:08:20.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.115+0000 7f2996200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2990071ce0 con 0x7f2990160a50 2024-04-26T23:08:20.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.115+0000 7f2996200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f29900722a0 con 0x7f299006dd40 2024-04-26T23:08:20.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.116+0000 7f2995800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f299006dd40 0x7f2990072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.116+0000 7f2995800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f299006dd40 0x7f2990072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:55876/0 (socket says 172.21.15.40:55876) 2024-04-26T23:08:20.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.116+0000 7f2995800640 1 -- 172.21.15.40:0/1923079652 learned_addr learned my addr 172.21.15.40:0/1923079652 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.116+0000 7f298fe00640 1 --1- 172.21.15.40:0/1923079652 >> v1:172.21.15.103:6789/0 conn(0x7f29900cfaf0 0x7f2990160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:35244/0 (socket says 172.21.15.40:35244) 2024-04-26T23:08:20.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.116+0000 7f2994e00640 1 --2- 172.21.15.40:0/1923079652 >> v2:172.21.15.103:3300/0 conn(0x7f2990160a50 0x7f2990073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.117+0000 7f2995800640 1 -- 172.21.15.40:0/1923079652 >> v2:172.21.15.103:3300/0 conn(0x7f2990160a50 msgr2=0x7f2990073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.117+0000 7f2995800640 1 --2- 172.21.15.40:0/1923079652 >> v2:172.21.15.103:3300/0 conn(0x7f2990160a50 0x7f2990073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.117+0000 7f298f400640 1 -- 172.21.15.40:0/1923079652 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 305565046 0 0) 0x7f2990161880 con 0x7f29900cfaf0 2024-04-26T23:08:20.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.117+0000 7f2995800640 1 -- 172.21.15.40:0/1923079652 >> v1:172.21.15.103:6789/0 conn(0x7f29900cfaf0 legacy=0x7f2990160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.117+0000 7f2995800640 1 -- 172.21.15.40:0/1923079652 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29900cff80 con 0x7f299006dd40 2024-04-26T23:08:20.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.118+0000 7f2995800640 1 --2- 172.21.15.40:0/1923079652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f299006dd40 0x7f2990072730 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2988009870 tx=0x7f298802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12efe4fcbb85fd36 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.119+0000 7f298f400640 1 -- 172.21.15.40:0/1923079652 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988037070 con 0x7f299006dd40 2024-04-26T23:08:20.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.119+0000 7f298f400640 1 -- 172.21.15.40:0/1923079652 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2988032440 con 0x7f299006dd40 2024-04-26T23:08:20.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.119+0000 7f298f400640 1 -- 172.21.15.40:0/1923079652 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988032810 con 0x7f299006dd40 2024-04-26T23:08:20.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.119+0000 7f2996200640 1 -- 172.21.15.40:0/1923079652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f299006dd40 msgr2=0x7f2990072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.119+0000 7f2996200640 1 --2- 172.21.15.40:0/1923079652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f299006dd40 0x7f2990072730 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2988009870 tx=0x7f298802a270 comp rx=0 tx=0).stop 2024-04-26T23:08:20.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.119+0000 7f2996200640 1 -- 172.21.15.40:0/1923079652 shutdown_connections 2024-04-26T23:08:20.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.119+0000 7f2996200640 1 --2- 172.21.15.40:0/1923079652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f299006dd40 0x7f2990072730 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.119+0000 7f2996200640 1 --2- 172.21.15.40:0/1923079652 >> v2:172.21.15.103:3300/0 conn(0x7f2990160a50 0x7f2990073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.119+0000 7f2996200640 1 -- 172.21.15.40:0/1923079652 >> 172.21.15.40:0/1923079652 conn(0x7f29900b2720 msgr2=0x7f29900b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.119+0000 7f2996200640 1 -- 172.21.15.40:0/1923079652 shutdown_connections 2024-04-26T23:08:20.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.119+0000 7f2996200640 1 -- 172.21.15.40:0/1923079652 wait complete. 2024-04-26T23:08:20.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.121+0000 7f2996200640 1 Processor -- start 2024-04-26T23:08:20.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.121+0000 7f2996200640 1 -- start start 2024-04-26T23:08:20.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.122+0000 7f2996200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299006dd40 0x7f299015b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.122+0000 7f2996200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2990160a50 0x7f29900b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.122+0000 7f2996200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299015a700 0x7f2990159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.122+0000 7f2996200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2990071f30 con 0x7f299015a700 2024-04-26T23:08:20.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.122+0000 7f2996200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2990073f90 con 0x7f299006dd40 2024-04-26T23:08:20.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.122+0000 7f2996200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f299006e1f0 con 0x7f2990160a50 2024-04-26T23:08:20.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.123+0000 7f2995800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299015a700 0x7f2990159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.123+0000 7f2995800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299015a700 0x7f2990159e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54898/0 (socket says 172.21.15.40:54898) 2024-04-26T23:08:20.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.123+0000 7f2995800640 1 -- 172.21.15.40:0/3269630710 learned_addr learned my addr 172.21.15.40:0/3269630710 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.123+0000 7f2994e00640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299006dd40 0x7f299015b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.123+0000 7f298fe00640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2990160a50 0x7f29900b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.123+0000 7f2995800640 1 -- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2990160a50 msgr2=0x7f29900b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.123+0000 7f2995800640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2990160a50 0x7f29900b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.123+0000 7f2995800640 1 -- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299006dd40 msgr2=0x7f299015b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.123+0000 7f2995800640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299006dd40 0x7f299015b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.123+0000 7f2995800640 1 -- 172.21.15.40:0/3269630710 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299013c3e0 con 0x7f299015a700 2024-04-26T23:08:20.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.123+0000 7f2995800640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299015a700 0x7f2990159e20 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f2988009870 tx=0x7f29880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.124+0000 7f298cc00640 1 -- 172.21.15.40:0/3269630710 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988037040 con 0x7f299015a700 2024-04-26T23:08:20.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.124+0000 7f298cc00640 1 -- 172.21.15.40:0/3269630710 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2988004030 con 0x7f299015a700 2024-04-26T23:08:20.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.124+0000 7f298cc00640 1 -- 172.21.15.40:0/3269630710 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988004400 con 0x7f299015a700 2024-04-26T23:08:20.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.124+0000 7f2996200640 1 -- 172.21.15.40:0/3269630710 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299013c6c0 con 0x7f299015a700 2024-04-26T23:08:20.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.124+0000 7f2996200640 1 -- 172.21.15.40:0/3269630710 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f299013c0f0 con 0x7f299015a700 2024-04-26T23:08:20.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.127+0000 7f298cc00640 1 -- 172.21.15.40:0/3269630710 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f298803a070 con 0x7f299015a700 2024-04-26T23:08:20.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.128+0000 7f298cc00640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f297006c920 0x7f297006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.128+0000 7f298cc00640 1 -- 172.21.15.40:0/3269630710 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f29880ac110 con 0x7f299015a700 2024-04-26T23:08:20.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.128+0000 7f299a679740 1 Processor -- start 2024-04-26T23:08:20.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.128+0000 7f299a679740 1 -- start start 2024-04-26T23:08:20.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.128+0000 7f2994e00640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f297006c920 0x7f297006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.129+0000 7f299a679740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b978af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.129+0000 7f299a679740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b995fb0 0x561e7b996500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.129+0000 7f299a679740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b94e3f0 0x561e7b994670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.129+0000 7f299a679740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561e7b85e030 con 0x561e7b978710 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.129+0000 7f299a679740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561e7b8797f0 con 0x561e7b995fb0 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.129+0000 7f299a679740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561e7b8347e0 con 0x561e7b94e3f0 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.129+0000 7f2994e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b978af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.129+0000 7f2994e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b978af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54900/0 (socket says 172.21.15.40:54900) 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.130+0000 7f2994e00640 1 -- 172.21.15.40:0/3350381671 learned_addr learned my addr 172.21.15.40:0/3350381671 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.130+0000 7f298fe00640 1 --2- 172.21.15.40:0/3350381671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b995fb0 0x561e7b996500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.130+0000 7f2995800640 1 --2- 172.21.15.40:0/3350381671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b94e3f0 0x561e7b994670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.130+0000 7f2994e00640 1 -- 172.21.15.40:0/3350381671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b94e3f0 msgr2=0x561e7b994670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.130+0000 7f2994e00640 1 --2- 172.21.15.40:0/3350381671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b94e3f0 0x561e7b994670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.130+0000 7f2994e00640 1 -- 172.21.15.40:0/3350381671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b995fb0 msgr2=0x561e7b996500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.130+0000 7f2994e00640 1 --2- 172.21.15.40:0/3350381671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b995fb0 0x561e7b996500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.130+0000 7f2994e00640 1 -- 172.21.15.40:0/3350381671 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e7b840050 con 0x561e7b978710 2024-04-26T23:08:20.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.130+0000 7f2994e00640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f297006c920 0x7f297006ede0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2980011340 tx=0x7f298000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.130+0000 7f2994e00640 1 --2- 172.21.15.40:0/3350381671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b978af0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f298001b000 tx=0x7f298001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7efeabec07998260 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.130+0000 7f2985600640 1 -- 172.21.15.40:0/3350381671 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298001d070 con 0x561e7b978710 2024-04-26T23:08:20.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.131+0000 7f2985600640 1 -- 172.21.15.40:0/3350381671 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2980017b40 con 0x561e7b978710 2024-04-26T23:08:20.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.131+0000 7f2985600640 1 -- 172.21.15.40:0/3350381671 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980015450 con 0x561e7b978710 2024-04-26T23:08:20.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.131+0000 7f299a679740 1 -- 172.21.15.40:0/3350381671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 msgr2=0x561e7b978af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.131+0000 7f299a679740 1 --2- 172.21.15.40:0/3350381671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b978af0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f298001b000 tx=0x7f298001c000 comp rx=0 tx=0).stop 2024-04-26T23:08:20.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.131+0000 7f299a679740 1 -- 172.21.15.40:0/3350381671 shutdown_connections 2024-04-26T23:08:20.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.131+0000 7f299a679740 1 --2- 172.21.15.40:0/3350381671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b94e3f0 0x561e7b994670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.131+0000 7f299a679740 1 --2- 172.21.15.40:0/3350381671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b995fb0 0x561e7b996500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.131+0000 7f299a679740 1 --2- 172.21.15.40:0/3350381671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b978af0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.131+0000 7f299a679740 1 -- 172.21.15.40:0/3350381671 >> 172.21.15.40:0/3350381671 conn(0x561e7b94ad20 msgr2=0x561e7b993520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.131+0000 7f299a679740 1 -- 172.21.15.40:0/3350381671 shutdown_connections 2024-04-26T23:08:20.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.131+0000 7f299a679740 1 -- 172.21.15.40:0/3350381671 wait complete. 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.132+0000 7f299a679740 1 -- start start 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.132+0000 7f299a679740 1 Processor -- start 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.133+0000 7f299a679740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b978710 0x561e7b98c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.133+0000 7f299a679740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b995fb0 0x561e7b98ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.133+0000 7f299a679740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b977a20 0x561e7b98f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.133+0000 7f299a679740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561e7b8b6200 con 0x561e7b977a20 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.133+0000 7f299a679740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561e7b8b0ee0 con 0x561e7b978710 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.133+0000 7f299a679740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561e7b8bde70 con 0x561e7b995fb0 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.133+0000 7f299a679740 10 client.0 _ll_register_callbacks cb 0x561e7b8a7270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.134+0000 7f2995800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b977a20 0x561e7b98f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.134+0000 7f2995800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b977a20 0x561e7b98f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54906/0 (socket says 172.21.15.40:54906) 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.134+0000 7f2995800640 1 -- 172.21.15.40:0/4085810245 learned_addr learned my addr 172.21.15.40:0/4085810245 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.134+0000 7f2994e00640 1 --2- 172.21.15.40:0/4085810245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b978710 0x561e7b98c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.134+0000 7f2995800640 1 -- 172.21.15.40:0/4085810245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b995fb0 msgr2=0x561e7b98ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.134+0000 7f2995800640 1 --2- 172.21.15.40:0/4085810245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b995fb0 0x561e7b98ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.134+0000 7f2995800640 1 -- 172.21.15.40:0/4085810245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b978710 msgr2=0x561e7b98c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.134+0000 7f2995800640 1 --2- 172.21.15.40:0/4085810245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b978710 0x561e7b98c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.134+0000 7f2995800640 1 -- 172.21.15.40:0/4085810245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e7b68d7d0 con 0x561e7b977a20 2024-04-26T23:08:20.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.134+0000 7f2995800640 1 --2- 172.21.15.40:0/4085810245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b977a20 0x561e7b98f360 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f29880097c0 tx=0x7f2988032500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701efc4b44114b94 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.134+0000 7f297a000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:20.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f297a000640 1 -- 172.21.15.40:0/4085810245 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298804a040 con 0x561e7b977a20 2024-04-26T23:08:20.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f297a000640 1 -- 172.21.15.40:0/4085810245 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f2988032e00 con 0x561e7b977a20 2024-04-26T23:08:20.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f299a679740 1 -- 172.21.15.40:0/4085810245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2980017b40 con 0x561e7b977a20 2024-04-26T23:08:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f297a000640 1 -- 172.21.15.40:0/4085810245 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f29880aa050 con 0x561e7b977a20 2024-04-26T23:08:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f299a679740 10 client.8427 fetch_fsmap learned FSMap version 99 2024-04-26T23:08:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f299a679740 1 -- 172.21.15.40:0/4085810245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x561e7b5c33f0 con 0x561e7b977a20 2024-04-26T23:08:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f297a000640 1 -- 172.21.15.40:0/4085810245 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2980017b40 con 0x561e7b977a20 2024-04-26T23:08:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f297a000640 1 -- 172.21.15.40:0/4085810245 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 99) ==== 3550+0+0 (secure 0 0 0) 0x7f29880823f0 con 0x561e7b977a20 2024-04-26T23:08:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f299a679740 10 client.8427 fetch_fsmap finished waiting for FSMap version 99 2024-04-26T23:08:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f299a679740 10 client.8427 resolve_mds: resolved 6:1 to role '6:1' aka daemon mds.a 2024-04-26T23:08:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f299a679740 20 client.8427 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f299a679740 1 --2- 172.21.15.40:0/4085810245 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x561e7b98a5d0 0x561e7b98a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f299a679740 4 client.8427 mds_command: new command op to 8031 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:20.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.135+0000 7f299a679740 1 -- 172.21.15.40:0/4085810245 --> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e7b8bf7e0 con 0x561e7b98a5d0 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.136+0000 7f2994e00640 1 --2- 172.21.15.40:0/4085810245 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x561e7b98a5d0 0x561e7b98a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.136+0000 7f2994e00640 1 --2- 172.21.15.40:0/4085810245 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x561e7b98a5d0 0x561e7b98a9e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6218f1499eb0089d server_cookie=b4a9a702898354ba in_seq=0 out_seq=0 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.136+0000 7f297a000640 10 client.8427 ms_handle_connect on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.137+0000 7f297a000640 1 -- 172.21.15.40:0/4085810245 <== mds.1 v2:172.21.15.40:6832/1811310743 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561e7b8bf7e0 con 0x561e7b98a5d0 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.137+0000 7f297a000640 10 client.8427 handle_command_reply: tid=0 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.137+0000 7f299a679740 1 client.8427 shutdown 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.137+0000 7f299a679740 1 -- 172.21.15.40:0/4085810245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b977a20 msgr2=0x561e7b98f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.137+0000 7f299a679740 1 --2- 172.21.15.40:0/4085810245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b977a20 0x561e7b98f360 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f29880097c0 tx=0x7f2988032500 comp rx=0 tx=0).stop 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.137+0000 7f299a679740 1 -- 172.21.15.40:0/4085810245 shutdown_connections 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.137+0000 7f299a679740 1 --2- 172.21.15.40:0/4085810245 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x561e7b98a5d0 0x561e7b98a9e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.137+0000 7f299a679740 1 --2- 172.21.15.40:0/4085810245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b977a20 0x561e7b98f360 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.137+0000 7f299a679740 1 --2- 172.21.15.40:0/4085810245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b995fb0 0x561e7b98ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.137+0000 7f299a679740 1 --2- 172.21.15.40:0/4085810245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b978710 0x561e7b98c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.137+0000 7f299a679740 1 -- 172.21.15.40:0/4085810245 >> 172.21.15.40:0/4085810245 conn(0x561e7b94e3f0 msgr2=0x561e7b9481f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.137+0000 7f297a000640 0 client.8427 ms_handle_reset on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:20.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.138+0000 7f299a679740 1 -- 172.21.15.40:0/4085810245 shutdown_connections 2024-04-26T23:08:20.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.138+0000 7f299a679740 1 -- 172.21.15.40:0/4085810245 wait complete. 2024-04-26T23:08:20.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.138+0000 7f299a679740 20 client.8427 trim_cache size 0 max 16384 2024-04-26T23:08:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.158+0000 7f299a679740 1 Processor -- start 2024-04-26T23:08:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.158+0000 7f299a679740 1 -- start start 2024-04-26T23:08:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f299a679740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b977a20 0x561e7b988eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f299a679740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b944080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f299a679740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b995fb0 0x561e7b94d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f299a679740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561e7b990910 con 0x561e7b978710 2024-04-26T23:08:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f299a679740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2990201c80 con 0x561e7b977a20 2024-04-26T23:08:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f299a679740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2990200e40 con 0x561e7b995fb0 2024-04-26T23:08:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f298fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b944080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f298fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b944080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54910/0 (socket says 172.21.15.40:54910) 2024-04-26T23:08:20.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f298fe00640 1 -- 172.21.15.40:0/144656035 learned_addr learned my addr 172.21.15.40:0/144656035 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f2994e00640 1 --2- 172.21.15.40:0/144656035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b977a20 0x561e7b988eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f2995800640 1 --2- 172.21.15.40:0/144656035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b995fb0 0x561e7b94d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f298fe00640 1 -- 172.21.15.40:0/144656035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b995fb0 msgr2=0x561e7b94d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f298fe00640 1 --2- 172.21.15.40:0/144656035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b995fb0 0x561e7b94d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f298fe00640 1 -- 172.21.15.40:0/144656035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b977a20 msgr2=0x561e7b988eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f298fe00640 1 --2- 172.21.15.40:0/144656035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b977a20 0x561e7b988eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.159+0000 7f298fe00640 1 -- 172.21.15.40:0/144656035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e7b5830f0 con 0x561e7b978710 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.160+0000 7f298fe00640 1 --2- 172.21.15.40:0/144656035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b944080 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f297c0072e0 tx=0x7f297c0077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d9e88c28ff92c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.160+0000 7f2984c00640 1 -- 172.21.15.40:0/144656035 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c01b070 con 0x561e7b978710 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.160+0000 7f2984c00640 1 -- 172.21.15.40:0/144656035 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f297c001370 con 0x561e7b978710 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.160+0000 7f2984c00640 1 -- 172.21.15.40:0/144656035 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c001740 con 0x561e7b978710 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.161+0000 7f299a679740 1 -- 172.21.15.40:0/144656035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 msgr2=0x561e7b944080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.161+0000 7f299a679740 1 --2- 172.21.15.40:0/144656035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b944080 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f297c0072e0 tx=0x7f297c0077a0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.161+0000 7f299a679740 1 -- 172.21.15.40:0/144656035 shutdown_connections 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.161+0000 7f299a679740 1 --2- 172.21.15.40:0/144656035 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b995fb0 0x561e7b94d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.161+0000 7f299a679740 1 --2- 172.21.15.40:0/144656035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b944080 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.161+0000 7f299a679740 1 --2- 172.21.15.40:0/144656035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b977a20 0x561e7b988eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.161+0000 7f299a679740 1 -- 172.21.15.40:0/144656035 >> 172.21.15.40:0/144656035 conn(0x561e7b94e3f0 msgr2=0x561e7b97c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.161+0000 7f299a679740 1 -- 172.21.15.40:0/144656035 shutdown_connections 2024-04-26T23:08:20.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.161+0000 7f299a679740 1 -- 172.21.15.40:0/144656035 wait complete. 2024-04-26T23:08:20.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.162+0000 7f299a679740 1 -- start start 2024-04-26T23:08:20.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.162+0000 7f299a679740 1 Processor -- start 2024-04-26T23:08:20.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.163+0000 7f299a679740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b977a20 0x561e7b94cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.163+0000 7f299a679740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b9fbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.163+0000 7f299a679740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b995fb0 0x561e7b9761e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.163+0000 7f299a679740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f29901f9300 con 0x561e7b978710 2024-04-26T23:08:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.163+0000 7f299a679740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2990201550 con 0x561e7b977a20 2024-04-26T23:08:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.163+0000 7f299a679740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561e7b8befd0 con 0x561e7b995fb0 2024-04-26T23:08:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.163+0000 7f299a679740 10 client.0 _ll_register_callbacks cb 0x7f29880069c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.163+0000 7f298fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b9fbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.163+0000 7f298fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b9fbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54916/0 (socket says 172.21.15.40:54916) 2024-04-26T23:08:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.163+0000 7f298fe00640 1 -- 172.21.15.40:0/907140816 learned_addr learned my addr 172.21.15.40:0/907140816 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.163+0000 7f298fe00640 1 -- 172.21.15.40:0/907140816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b995fb0 msgr2=0x561e7b9761e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.163+0000 7f298fe00640 1 --2- 172.21.15.40:0/907140816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b995fb0 0x561e7b9761e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.164+0000 7f2994e00640 1 --2- 172.21.15.40:0/907140816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b977a20 0x561e7b94cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.164+0000 7f298fe00640 1 -- 172.21.15.40:0/907140816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b977a20 msgr2=0x561e7b94cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.164+0000 7f298fe00640 1 --2- 172.21.15.40:0/907140816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b977a20 0x561e7b94cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.164+0000 7f298fe00640 1 -- 172.21.15.40:0/907140816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e7b879a30 con 0x561e7b978710 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.164+0000 7f2994e00640 1 --2- 172.21.15.40:0/907140816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b977a20 0x561e7b94cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.164+0000 7f298fe00640 1 --2- 172.21.15.40:0/907140816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b9fbf20 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f297c009010 tx=0x7f297c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c0cf580bd7b45d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.164+0000 7f297a000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.164+0000 7f297a000640 1 -- 172.21.15.40:0/907140816 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c01b040 con 0x561e7b978710 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.164+0000 7f297a000640 1 -- 172.21.15.40:0/907140816 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f297c001c40 con 0x561e7b978710 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.164+0000 7f299a679740 1 -- 172.21.15.40:0/907140816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f297c001370 con 0x561e7b978710 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.165+0000 7f297a000640 1 -- 172.21.15.40:0/907140816 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f297c015460 con 0x561e7b978710 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.165+0000 7f299a679740 10 client.8433 fetch_fsmap learned FSMap version 99 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.165+0000 7f299a679740 1 -- 172.21.15.40:0/907140816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x561e7b94d0c0 con 0x561e7b978710 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.165+0000 7f297a000640 1 -- 172.21.15.40:0/907140816 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f297c001370 con 0x561e7b978710 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.165+0000 7f297a000640 1 -- 172.21.15.40:0/907140816 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 99) ==== 3550+0+0 (secure 0 0 0) 0x7f297c015600 con 0x561e7b978710 2024-04-26T23:08:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.165+0000 7f299a679740 10 client.8433 fetch_fsmap finished waiting for FSMap version 99 2024-04-26T23:08:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.165+0000 7f299a679740 10 client.8433 resolve_mds: resolved 6:1 to role '6:1' aka daemon mds.a 2024-04-26T23:08:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.165+0000 7f299a679740 20 client.8433 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.165+0000 7f299a679740 1 --2- 172.21.15.40:0/907140816 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x561e7b9fb7f0 0x561e7b976720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.165+0000 7f299a679740 4 client.8433 mds_command: new command op to 8031 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:08:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.165+0000 7f299a679740 1 -- 172.21.15.40:0/907140816 --> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561e7b8b9650 con 0x561e7b9fb7f0 2024-04-26T23:08:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.165+0000 7f2994e00640 1 --2- 172.21.15.40:0/907140816 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x561e7b9fb7f0 0x561e7b976720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f2994e00640 1 --2- 172.21.15.40:0/907140816 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x561e7b9fb7f0 0x561e7b976720 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=30b71284c3661eb7 server_cookie=be2eee45fd818281 in_seq=0 out_seq=0 2024-04-26T23:08:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f297a000640 10 client.8433 ms_handle_connect on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f297a000640 1 -- 172.21.15.40:0/907140816 <== mds.1 v2:172.21.15.40:6832/1811310743 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x561e7b8b9650 con 0x561e7b9fb7f0 2024-04-26T23:08:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f297a000640 10 client.8433 handle_command_reply: tid=0 2024-04-26T23:08:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f299a679740 1 client.8433 shutdown 2024-04-26T23:08:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f299a679740 1 -- 172.21.15.40:0/907140816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 msgr2=0x561e7b9fbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f299a679740 1 --2- 172.21.15.40:0/907140816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b9fbf20 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f297c009010 tx=0x7f297c000ed0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f299a679740 1 -- 172.21.15.40:0/907140816 shutdown_connections 2024-04-26T23:08:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f299a679740 1 --2- 172.21.15.40:0/907140816 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x561e7b9fb7f0 0x561e7b976720 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f299a679740 1 --2- 172.21.15.40:0/907140816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561e7b995fb0 0x561e7b9761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f299a679740 1 --2- 172.21.15.40:0/907140816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e7b978710 0x561e7b9fbf20 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f297a000640 0 client.8433 ms_handle_reset on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f299a679740 1 --2- 172.21.15.40:0/907140816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561e7b977a20 0x561e7b94cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f299a679740 1 -- 172.21.15.40:0/907140816 >> 172.21.15.40:0/907140816 conn(0x561e7b94e3f0 msgr2=0x561e7b947960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.166+0000 7f299a679740 1 -- 172.21.15.40:0/907140816 shutdown_connections 2024-04-26T23:08:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.167+0000 7f299a679740 1 -- 172.21.15.40:0/907140816 wait complete. 2024-04-26T23:08:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.167+0000 7f299a679740 20 client.8433 trim_cache size 0 max 16384 2024-04-26T23:08:20.171 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-26T23:08:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:08:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: "b17426f8-9d25-4fbb-a99f-95207d3f0c9f": { 2024-04-26T23:08:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:08:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "b17426f8-9d25-4fbb-a99f-95207d3f0c9f", 2024-04-26T23:08:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:08:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:08:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:08:20.171 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:20.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.167+0000 7f2996200640 1 -- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f297006c920 msgr2=0x7f297006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.167+0000 7f2996200640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f297006c920 0x7f297006ede0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2980011340 tx=0x7f298000b040 comp rx=0 tx=0).stop 2024-04-26T23:08:20.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.167+0000 7f2996200640 1 -- 172.21.15.40:0/3269630710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299015a700 msgr2=0x7f2990159e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.167+0000 7f2996200640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299015a700 0x7f2990159e20 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f2988009870 tx=0x7f29880046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.167+0000 7f2996200640 1 -- 172.21.15.40:0/3269630710 shutdown_connections 2024-04-26T23:08:20.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.167+0000 7f2996200640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f297006c920 0x7f297006ede0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.167+0000 7f2996200640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299015a700 0x7f2990159e20 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.167+0000 7f2996200640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2990160a50 0x7f29900b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.167+0000 7f2996200640 1 --2- 172.21.15.40:0/3269630710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299006dd40 0x7f299015b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.167+0000 7f2996200640 1 -- 172.21.15.40:0/3269630710 >> 172.21.15.40:0/3269630710 conn(0x7f29900b2720 msgr2=0x7f2990141270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.167+0000 7f2996200640 1 -- 172.21.15.40:0/3269630710 shutdown_connections 2024-04-26T23:08:20.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.168+0000 7f2996200640 1 -- 172.21.15.40:0/3269630710 wait complete. 2024-04-26T23:08:20.186 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:08:20.186 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:08:20.186 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.6:2 scrub status 2024-04-26T23:08:20.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.394+0000 7f1a84e00640 1 Processor -- start 2024-04-26T23:08:20.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.394+0000 7f1a84e00640 1 -- start start 2024-04-26T23:08:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.395+0000 7f1a84e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a8015f380 0x7f1a800cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.395+0000 7f1a84e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a80073b70 0x7f1a800cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.395+0000 7f1a84e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a80072c00 con 0x7f1a800d0620 2024-04-26T23:08:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.396+0000 7f1a84e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1a800729b0 con 0x7f1a80073b70 2024-04-26T23:08:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.396+0000 7f1a84e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1a80072ee0 con 0x7f1a8015f380 2024-04-26T23:08:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.396+0000 7f1a7d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a80073b70 0x7f1a800cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.396+0000 7f1a7d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a80073b70 0x7f1a800cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54930/0 (socket says 172.21.15.40:54930) 2024-04-26T23:08:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.396+0000 7f1a7d400640 1 -- 172.21.15.40:0/1056937649 learned_addr learned my addr 172.21.15.40:0/1056937649 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.396+0000 7f1a7de00640 1 --2- 172.21.15.40:0/1056937649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a8015f380 0x7f1a800cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.396+0000 7f1a7d400640 1 -- 172.21.15.40:0/1056937649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a8015f380 msgr2=0x7f1a800cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.397+0000 7f1a7d400640 1 --2- 172.21.15.40:0/1056937649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a8015f380 0x7f1a800cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.397+0000 7f1a7d400640 1 -- 172.21.15.40:0/1056937649 >> v1:172.21.15.40:6789/0 conn(0x7f1a800d0620 legacy=0x7f1a800cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.397+0000 7f1a7d400640 1 -- 172.21.15.40:0/1056937649 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a800731c0 con 0x7f1a80073b70 2024-04-26T23:08:20.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.398+0000 7f1a7d400640 1 --2- 172.21.15.40:0/1056937649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a80073b70 0x7f1a800cb980 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009870 tx=0x7f1a6c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78a4ed9f457b62f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.398+0000 7f1a7ca00640 1 -- 172.21.15.40:0/1056937649 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c037070 con 0x7f1a80073b70 2024-04-26T23:08:20.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.398+0000 7f1a7ca00640 1 -- 172.21.15.40:0/1056937649 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a6c032440 con 0x7f1a80073b70 2024-04-26T23:08:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.398+0000 7f1a7ca00640 1 -- 172.21.15.40:0/1056937649 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c032810 con 0x7f1a80073b70 2024-04-26T23:08:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.399+0000 7f1a84e00640 1 -- 172.21.15.40:0/1056937649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a80073b70 msgr2=0x7f1a800cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.399+0000 7f1a84e00640 1 --2- 172.21.15.40:0/1056937649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a80073b70 0x7f1a800cb980 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009870 tx=0x7f1a6c02a250 comp rx=0 tx=0).stop 2024-04-26T23:08:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.399+0000 7f1a84e00640 1 -- 172.21.15.40:0/1056937649 shutdown_connections 2024-04-26T23:08:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.399+0000 7f1a84e00640 1 --2- 172.21.15.40:0/1056937649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a80073b70 0x7f1a800cb980 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.399+0000 7f1a84e00640 1 --2- 172.21.15.40:0/1056937649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a8015f380 0x7f1a800cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.399+0000 7f1a84e00640 1 -- 172.21.15.40:0/1056937649 >> 172.21.15.40:0/1056937649 conn(0x7f1a800d0be0 msgr2=0x7f1a800d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.399+0000 7f1a84e00640 1 -- 172.21.15.40:0/1056937649 shutdown_connections 2024-04-26T23:08:20.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.399+0000 7f1a84e00640 1 -- 172.21.15.40:0/1056937649 wait complete. 2024-04-26T23:08:20.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.401+0000 7f1a84e00640 1 Processor -- start 2024-04-26T23:08:20.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.401+0000 7f1a84e00640 1 -- start start 2024-04-26T23:08:20.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.402+0000 7f1a84e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a800d0620 0x7f1a80152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.402+0000 7f1a84e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8015f380 0x7f1a8015cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.402+0000 7f1a84e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a8015d270 0x7f1a800c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.402+0000 7f1a84e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a80073620 con 0x7f1a8015f380 2024-04-26T23:08:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.402+0000 7f1a84e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a800aff10 con 0x7f1a800d0620 2024-04-26T23:08:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.402+0000 7f1a84e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1a800737a0 con 0x7f1a8015d270 2024-04-26T23:08:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.402+0000 7f1a7d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8015f380 0x7f1a8015cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.402+0000 7f1a7d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8015f380 0x7f1a8015cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54940/0 (socket says 172.21.15.40:54940) 2024-04-26T23:08:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.402+0000 7f1a7d400640 1 -- 172.21.15.40:0/1075991096 learned_addr learned my addr 172.21.15.40:0/1075991096 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.403+0000 7f1a7d400640 1 -- 172.21.15.40:0/1075991096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a8015d270 msgr2=0x7f1a800c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.403+0000 7f1a7d400640 1 --2- 172.21.15.40:0/1075991096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a8015d270 0x7f1a800c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.403+0000 7f1a7d400640 1 -- 172.21.15.40:0/1075991096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a800d0620 msgr2=0x7f1a80152020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.403+0000 7f1a7d400640 1 --2- 172.21.15.40:0/1075991096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a800d0620 0x7f1a80152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.403+0000 7f1a7d400640 1 -- 172.21.15.40:0/1075991096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a801f8440 con 0x7f1a8015f380 2024-04-26T23:08:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.403+0000 7f1a7d400640 1 --2- 172.21.15.40:0/1075991096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8015f380 0x7f1a8015cd30 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c000c00 tx=0x7f1a6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.404+0000 7f1a76000640 1 -- 172.21.15.40:0/1075991096 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c041020 con 0x7f1a8015f380 2024-04-26T23:08:20.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.404+0000 7f1a76000640 1 -- 172.21.15.40:0/1075991096 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a6c0306c0 con 0x7f1a8015f380 2024-04-26T23:08:20.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.404+0000 7f1a76000640 1 -- 172.21.15.40:0/1075991096 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c037040 con 0x7f1a8015f380 2024-04-26T23:08:20.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.404+0000 7f1a84e00640 1 -- 172.21.15.40:0/1075991096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a801f8720 con 0x7f1a8015f380 2024-04-26T23:08:20.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.404+0000 7f1a84e00640 1 -- 172.21.15.40:0/1075991096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a801f8ce0 con 0x7f1a8015f380 2024-04-26T23:08:20.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.406+0000 7f1a76000640 1 -- 172.21.15.40:0/1075991096 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1a6c044020 con 0x7f1a8015f380 2024-04-26T23:08:20.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.407+0000 7f1a76000640 1 --2- 172.21.15.40:0/1075991096 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a6006c920 0x7f1a6006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.407+0000 7f1a76000640 1 -- 172.21.15.40:0/1075991096 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f1a6c0ac4f0 con 0x7f1a8015f380 2024-04-26T23:08:20.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.408+0000 7f1a7de00640 1 --2- 172.21.15.40:0/1075991096 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a6006c920 0x7f1a6006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.408+0000 7f1a87dcf740 1 Processor -- start 2024-04-26T23:08:20.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.408+0000 7f1a87dcf740 1 -- start start 2024-04-26T23:08:20.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.409+0000 7f1a7de00640 1 --2- 172.21.15.40:0/1075991096 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a6006c920 0x7f1a6006ede0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1a70006fd0 tx=0x7f1a70008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.409+0000 7f1a87dcf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00cf540 0x55d4e00cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.409+0000 7f1a87dcf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00ecde0 0x55d4e00ed330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.409+0000 7f1a87dcf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00a5220 0x55d4e00eb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.409+0000 7f1a87dcf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d4dffb5030 con 0x55d4e00ecde0 2024-04-26T23:08:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.409+0000 7f1a87dcf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d4dffd07f0 con 0x55d4e00cf540 2024-04-26T23:08:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.409+0000 7f1a87dcf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d4dff8b7e0 con 0x55d4e00a5220 2024-04-26T23:08:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.409+0000 7f1a7d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00ecde0 0x55d4e00ed330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.410+0000 7f1a7de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00cf540 0x55d4e00cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.410+0000 7f1a7de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00cf540 0x55d4e00cf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:33996/0 (socket says 172.21.15.40:33996) 2024-04-26T23:08:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.410+0000 7f1a7de00640 1 -- 172.21.15.40:0/1171800443 learned_addr learned my addr 172.21.15.40:0/1171800443 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.410+0000 7f1a7e800640 1 --2- 172.21.15.40:0/1171800443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00a5220 0x55d4e00eb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.410+0000 7f1a7d400640 1 -- 172.21.15.40:0/1171800443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00a5220 msgr2=0x55d4e00eb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.410+0000 7f1a7d400640 1 --2- 172.21.15.40:0/1171800443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00a5220 0x55d4e00eb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.410+0000 7f1a7d400640 1 -- 172.21.15.40:0/1171800443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00cf540 msgr2=0x55d4e00cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.410+0000 7f1a7d400640 1 --2- 172.21.15.40:0/1171800443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00cf540 0x55d4e00cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.410+0000 7f1a7d400640 1 -- 172.21.15.40:0/1171800443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4dff97050 con 0x55d4e00ecde0 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.410+0000 7f1a7e800640 1 --2- 172.21.15.40:0/1171800443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00a5220 0x55d4e00eb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.410+0000 7f1a7d400640 1 --2- 172.21.15.40:0/1171800443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00ecde0 0x55d4e00ed330 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c079cf0 tx=0x7f1a6c07a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce69c977ccc823e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.411+0000 7f1a6aa00640 1 -- 172.21.15.40:0/1171800443 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c082bd0 con 0x55d4e00ecde0 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.411+0000 7f1a6aa00640 1 -- 172.21.15.40:0/1171800443 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a6c031e30 con 0x55d4e00ecde0 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.411+0000 7f1a6aa00640 1 -- 172.21.15.40:0/1171800443 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c076660 con 0x55d4e00ecde0 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.411+0000 7f1a87dcf740 1 -- 172.21.15.40:0/1171800443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00ecde0 msgr2=0x55d4e00ed330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.411+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/1171800443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00ecde0 0x55d4e00ed330 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c079cf0 tx=0x7f1a6c07a7b0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.411+0000 7f1a87dcf740 1 -- 172.21.15.40:0/1171800443 shutdown_connections 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.411+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/1171800443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00a5220 0x55d4e00eb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.411+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/1171800443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00ecde0 0x55d4e00ed330 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.411+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/1171800443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00cf540 0x55d4e00cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.411+0000 7f1a87dcf740 1 -- 172.21.15.40:0/1171800443 >> 172.21.15.40:0/1171800443 conn(0x55d4e00a1b50 msgr2=0x55d4e00ea350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.411+0000 7f1a87dcf740 1 -- 172.21.15.40:0/1171800443 shutdown_connections 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.412+0000 7f1a87dcf740 1 -- 172.21.15.40:0/1171800443 wait complete. 2024-04-26T23:08:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.412+0000 7f1a87dcf740 1 -- start start 2024-04-26T23:08:20.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.412+0000 7f1a87dcf740 1 Processor -- start 2024-04-26T23:08:20.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.413+0000 7f1a87dcf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00cf540 0x55d4e00c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.413+0000 7f1a87dcf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00ecde0 0x55d4e00d2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.413+0000 7f1a87dcf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00fd890 0x55d4e00cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.413+0000 7f1a87dcf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d4e000d200 con 0x55d4e00cf540 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.413+0000 7f1a87dcf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d4e0007ee0 con 0x55d4e00fd890 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.413+0000 7f1a87dcf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d4e0014e70 con 0x55d4e00ecde0 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.413+0000 7f1a87dcf740 10 client.0 _ll_register_callbacks cb 0x55d4dfffe270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.414+0000 7f1a7de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00cf540 0x55d4e00c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.414+0000 7f1a7de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00cf540 0x55d4e00c3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54970/0 (socket says 172.21.15.40:54970) 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.414+0000 7f1a7de00640 1 -- 172.21.15.40:0/2913544204 learned_addr learned my addr 172.21.15.40:0/2913544204 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.414+0000 7f1a7d400640 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00ecde0 0x55d4e00d2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.414+0000 7f1a7e800640 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00fd890 0x55d4e00cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.414+0000 7f1a7de00640 1 -- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00ecde0 msgr2=0x55d4e00d2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.414+0000 7f1a7de00640 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00ecde0 0x55d4e00d2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.414+0000 7f1a7de00640 1 -- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00fd890 msgr2=0x55d4e00cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.414+0000 7f1a7de00640 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00fd890 0x55d4e00cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.414+0000 7f1a7de00640 1 -- 172.21.15.40:0/2913544204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4dfde47d0 con 0x55d4e00cf540 2024-04-26T23:08:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.414+0000 7f1a7e800640 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00fd890 0x55d4e00cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.414+0000 7f1a7d400640 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00ecde0 0x55d4e00d2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.415+0000 7f1a7de00640 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00cf540 0x55d4e00c3e20 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f1a70006fa0 tx=0x7f1a700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db071853f1935bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.415+0000 7f1a5f400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.415+0000 7f1a5f400640 1 -- 172.21.15.40:0/2913544204 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70024020 con 0x55d4e00cf540 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.415+0000 7f1a5f400640 1 -- 172.21.15.40:0/2913544204 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f1a70020630 con 0x55d4e00cf540 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.415+0000 7f1a87dcf740 1 -- 172.21.15.40:0/2913544204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a6c031e30 con 0x55d4e00cf540 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.415+0000 7f1a5f400640 1 -- 172.21.15.40:0/2913544204 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f1a7002e050 con 0x55d4e00cf540 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.415+0000 7f1a87dcf740 10 client.8448 fetch_fsmap learned FSMap version 99 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.415+0000 7f1a87dcf740 1 -- 172.21.15.40:0/2913544204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x55d4dfd1a3f0 con 0x55d4e00cf540 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.415+0000 7f1a5f400640 1 -- 172.21.15.40:0/2913544204 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a6c031e30 con 0x55d4e00cf540 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.416+0000 7f1a5f400640 1 -- 172.21.15.40:0/2913544204 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 99) ==== 3550+0+0 (secure 0 0 0) 0x7f1a70031020 con 0x55d4e00cf540 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.416+0000 7f1a87dcf740 10 client.8448 fetch_fsmap finished waiting for FSMap version 99 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.416+0000 7f1a87dcf740 10 client.8448 resolve_mds: resolved 6:2 to role '6:2' aka daemon mds.d 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.416+0000 7f1a87dcf740 20 client.8448 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.416+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x55d4e00d0200 0x55d4e00d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.416+0000 7f1a87dcf740 4 client.8448 mds_command: new command op to 6253 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:20.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.416+0000 7f1a87dcf740 1 -- 172.21.15.40:0/2913544204 --> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4e00167e0 con 0x55d4e00d0200 2024-04-26T23:08:20.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.417+0000 7f1a7d400640 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x55d4e00d0200 0x55d4e00d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.418+0000 7f1a7d400640 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x55d4e00d0200 0x55d4e00d0690 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=851170bc63bd1e61 server_cookie=976c0040095d884 in_seq=0 out_seq=0 2024-04-26T23:08:20.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.418+0000 7f1a5f400640 10 client.8448 ms_handle_connect on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:20.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a5f400640 1 -- 172.21.15.40:0/2913544204 <== mds.2 v2:172.21.15.103:6836/2728939257 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d4e00167e0 con 0x55d4e00d0200 2024-04-26T23:08:20.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a5f400640 10 client.8448 handle_command_reply: tid=0 2024-04-26T23:08:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a87dcf740 1 client.8448 shutdown 2024-04-26T23:08:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a87dcf740 1 -- 172.21.15.40:0/2913544204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00cf540 msgr2=0x55d4e00c3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00cf540 0x55d4e00c3e20 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f1a70006fa0 tx=0x7f1a700046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a87dcf740 1 -- 172.21.15.40:0/2913544204 shutdown_connections 2024-04-26T23:08:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x55d4e00d0200 0x55d4e00d0690 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00fd890 0x55d4e00cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00ecde0 0x55d4e00d2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/2913544204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00cf540 0x55d4e00c3e20 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a5f400640 0 client.8448 ms_handle_reset on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a87dcf740 1 -- 172.21.15.40:0/2913544204 >> 172.21.15.40:0/2913544204 conn(0x55d4e00a5220 msgr2=0x55d4e00ee160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a87dcf740 1 -- 172.21.15.40:0/2913544204 shutdown_connections 2024-04-26T23:08:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a87dcf740 1 -- 172.21.15.40:0/2913544204 wait complete. 2024-04-26T23:08:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.420+0000 7f1a87dcf740 20 client.8448 trim_cache size 0 max 16384 2024-04-26T23:08:20.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.439+0000 7f1a87dcf740 1 Processor -- start 2024-04-26T23:08:20.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.439+0000 7f1a87dcf740 1 -- start start 2024-04-26T23:08:20.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.440+0000 7f1a87dcf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00ecde0 0x55d4e00c7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.440+0000 7f1a87dcf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00fd890 0x55d4e00c34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.440+0000 7f1a87dcf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00d1600 0x55d4e00a57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.440+0000 7f1a87dcf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a80202090 con 0x55d4e00fd890 2024-04-26T23:08:20.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.440+0000 7f1a87dcf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a80201230 con 0x55d4e00ecde0 2024-04-26T23:08:20.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.440+0000 7f1a87dcf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1a80201960 con 0x55d4e00d1600 2024-04-26T23:08:20.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.440+0000 7f1a7d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00fd890 0x55d4e00c34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.441+0000 7f1a7d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00fd890 0x55d4e00c34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54978/0 (socket says 172.21.15.40:54978) 2024-04-26T23:08:20.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.441+0000 7f1a7d400640 1 -- 172.21.15.40:0/986252667 learned_addr learned my addr 172.21.15.40:0/986252667 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.441+0000 7f1a7de00640 1 --2- 172.21.15.40:0/986252667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00ecde0 0x55d4e00c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.441+0000 7f1a7e800640 1 --2- 172.21.15.40:0/986252667 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00d1600 0x55d4e00a57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.441+0000 7f1a7d400640 1 -- 172.21.15.40:0/986252667 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00d1600 msgr2=0x55d4e00a57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.441+0000 7f1a7d400640 1 --2- 172.21.15.40:0/986252667 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00d1600 0x55d4e00a57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.441+0000 7f1a7d400640 1 -- 172.21.15.40:0/986252667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00ecde0 msgr2=0x55d4e00c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.441+0000 7f1a7d400640 1 --2- 172.21.15.40:0/986252667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00ecde0 0x55d4e00c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.441+0000 7f1a7d400640 1 -- 172.21.15.40:0/986252667 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4dfcda0f0 con 0x55d4e00fd890 2024-04-26T23:08:20.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.441+0000 7f1a7d400640 1 --2- 172.21.15.40:0/986252667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00fd890 0x55d4e00c34c0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c07a8a0 tx=0x7f1a6c048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3431988c7767ff4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.442+0000 7f1a6a000640 1 -- 172.21.15.40:0/986252667 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c031e30 con 0x55d4e00fd890 2024-04-26T23:08:20.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.442+0000 7f1a6a000640 1 -- 172.21.15.40:0/986252667 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a6c082bd0 con 0x55d4e00fd890 2024-04-26T23:08:20.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.442+0000 7f1a6a000640 1 -- 172.21.15.40:0/986252667 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c0782f0 con 0x55d4e00fd890 2024-04-26T23:08:20.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.442+0000 7f1a87dcf740 1 -- 172.21.15.40:0/986252667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00fd890 msgr2=0x55d4e00c34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.442+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/986252667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00fd890 0x55d4e00c34c0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c07a8a0 tx=0x7f1a6c048000 comp rx=0 tx=0).stop 2024-04-26T23:08:20.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.442+0000 7f1a87dcf740 1 -- 172.21.15.40:0/986252667 shutdown_connections 2024-04-26T23:08:20.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.442+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/986252667 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00d1600 0x55d4e00a57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.442+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/986252667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00fd890 0x55d4e00c34c0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.442+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/986252667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00ecde0 0x55d4e00c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.442+0000 7f1a87dcf740 1 -- 172.21.15.40:0/986252667 >> 172.21.15.40:0/986252667 conn(0x55d4e00a5220 msgr2=0x55d4e00c3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.442+0000 7f1a87dcf740 1 -- 172.21.15.40:0/986252667 shutdown_connections 2024-04-26T23:08:20.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.442+0000 7f1a87dcf740 1 -- 172.21.15.40:0/986252667 wait complete. 2024-04-26T23:08:20.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.443+0000 7f1a87dcf740 1 -- start start 2024-04-26T23:08:20.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.443+0000 7f1a87dcf740 1 Processor -- start 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.444+0000 7f1a87dcf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00d1600 0x55d4e00fcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.444+0000 7f1a87dcf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00ecde0 0x55d4e00cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.444+0000 7f1a87dcf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00fd890 0x55d4e00d2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.444+0000 7f1a87dcf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d4e0015fd0 con 0x55d4e00ecde0 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.444+0000 7f1a87dcf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a800b5fb0 con 0x55d4e00d1600 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.444+0000 7f1a87dcf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d4e00cdfd0 con 0x55d4e00fd890 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.444+0000 7f1a87dcf740 10 client.0 _ll_register_callbacks cb 0x7f1a7000bea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.445+0000 7f1a7d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00ecde0 0x55d4e00cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.445+0000 7f1a7d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00ecde0 0x55d4e00cd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54988/0 (socket says 172.21.15.40:54988) 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.445+0000 7f1a7d400640 1 -- 172.21.15.40:0/4133924417 learned_addr learned my addr 172.21.15.40:0/4133924417 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.445+0000 7f1a7de00640 1 --2- 172.21.15.40:0/4133924417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00d1600 0x55d4e00fcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.445+0000 7f1a7e800640 1 --2- 172.21.15.40:0/4133924417 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00fd890 0x55d4e00d2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.445+0000 7f1a7d400640 1 -- 172.21.15.40:0/4133924417 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00fd890 msgr2=0x55d4e00d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.445+0000 7f1a7d400640 1 --2- 172.21.15.40:0/4133924417 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00fd890 0x55d4e00d2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.445+0000 7f1a7d400640 1 -- 172.21.15.40:0/4133924417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00d1600 msgr2=0x55d4e00fcd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.445+0000 7f1a7d400640 1 --2- 172.21.15.40:0/4133924417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00d1600 0x55d4e00fcd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.445+0000 7f1a7d400640 1 -- 172.21.15.40:0/4133924417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4dffd0a30 con 0x55d4e00ecde0 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a7d400640 1 --2- 172.21.15.40:0/4133924417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00ecde0 0x55d4e00cd2b0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c035040 tx=0x7f1a6c008070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c617775044d9c8ce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a5f400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a5f400640 1 -- 172.21.15.40:0/4133924417 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c097060 con 0x55d4e00ecde0 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a5f400640 1 -- 172.21.15.40:0/4133924417 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f1a6c0782f0 con 0x55d4e00ecde0 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a87dcf740 1 -- 172.21.15.40:0/4133924417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a6c082bd0 con 0x55d4e00ecde0 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a5f400640 1 -- 172.21.15.40:0/4133924417 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f1a6c09e050 con 0x55d4e00ecde0 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a87dcf740 10 client.8454 fetch_fsmap learned FSMap version 99 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a87dcf740 1 -- 172.21.15.40:0/4133924417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x55d4e00fc820 con 0x55d4e00ecde0 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a5f400640 1 -- 172.21.15.40:0/4133924417 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a6c082bd0 con 0x55d4e00ecde0 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a5f400640 1 -- 172.21.15.40:0/4133924417 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 99) ==== 3550+0+0 (secure 0 0 0) 0x7f1a6c03a030 con 0x55d4e00ecde0 2024-04-26T23:08:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a87dcf740 10 client.8454 fetch_fsmap finished waiting for FSMap version 99 2024-04-26T23:08:20.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a87dcf740 10 client.8454 resolve_mds: resolved 6:2 to role '6:2' aka daemon mds.d 2024-04-26T23:08:20.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a87dcf740 20 client.8454 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:20.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/4133924417 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x55d4e014d220 0x55d4e00d2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a87dcf740 4 client.8454 mds_command: new command op to 6253 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:08:20.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.446+0000 7f1a87dcf740 1 -- 172.21.15.40:0/4133924417 --> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d4e0010650 con 0x55d4e014d220 2024-04-26T23:08:20.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.447+0000 7f1a7de00640 1 --2- 172.21.15.40:0/4133924417 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x55d4e014d220 0x55d4e00d2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.448+0000 7f1a7de00640 1 --2- 172.21.15.40:0/4133924417 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x55d4e014d220 0x55d4e00d2750 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=14fc10ca0d748f91 server_cookie=8909ee5db1db17a3 in_seq=0 out_seq=0 2024-04-26T23:08:20.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.448+0000 7f1a5f400640 10 client.8454 ms_handle_connect on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a5f400640 1 -- 172.21.15.40:0/4133924417 <== mds.2 v2:172.21.15.103:6836/2728939257 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55d4e0010650 con 0x55d4e014d220 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a5f400640 10 client.8454 handle_command_reply: tid=0 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a87dcf740 1 client.8454 shutdown 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a87dcf740 1 -- 172.21.15.40:0/4133924417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00ecde0 msgr2=0x55d4e00cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/4133924417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00ecde0 0x55d4e00cd2b0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c035040 tx=0x7f1a6c008070 comp rx=0 tx=0).stop 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a87dcf740 1 -- 172.21.15.40:0/4133924417 shutdown_connections 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/4133924417 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d4e00fd890 0x55d4e00d2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/4133924417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4e00ecde0 0x55d4e00cd2b0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/4133924417 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x55d4e014d220 0x55d4e00d2750 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a87dcf740 1 --2- 172.21.15.40:0/4133924417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4e00d1600 0x55d4e00fcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a5f400640 0 client.8454 ms_handle_reset on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a87dcf740 1 -- 172.21.15.40:0/4133924417 >> 172.21.15.40:0/4133924417 conn(0x55d4e00a5220 msgr2=0x55d4e00ce610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a87dcf740 1 -- 172.21.15.40:0/4133924417 shutdown_connections 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a87dcf740 1 -- 172.21.15.40:0/4133924417 wait complete. 2024-04-26T23:08:20.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.449+0000 7f1a87dcf740 20 client.8454 trim_cache size 0 max 16384 2024-04-26T23:08:20.452 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-26T23:08:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:08:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "b17426f8-9d25-4fbb-a99f-95207d3f0c9f": { 2024-04-26T23:08:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:08:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "b17426f8-9d25-4fbb-a99f-95207d3f0c9f", 2024-04-26T23:08:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:08:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:08:20.452 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:08:20.452 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:20.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.450+0000 7f1a84e00640 1 -- 172.21.15.40:0/1075991096 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a6006c920 msgr2=0x7f1a6006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.450+0000 7f1a84e00640 1 --2- 172.21.15.40:0/1075991096 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a6006c920 0x7f1a6006ede0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1a70006fd0 tx=0x7f1a70008040 comp rx=0 tx=0).stop 2024-04-26T23:08:20.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.450+0000 7f1a84e00640 1 -- 172.21.15.40:0/1075991096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8015f380 msgr2=0x7f1a8015cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.450+0000 7f1a84e00640 1 --2- 172.21.15.40:0/1075991096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8015f380 0x7f1a8015cd30 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c000c00 tx=0x7f1a6c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.450+0000 7f1a84e00640 1 -- 172.21.15.40:0/1075991096 shutdown_connections 2024-04-26T23:08:20.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.450+0000 7f1a84e00640 1 --2- 172.21.15.40:0/1075991096 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1a6006c920 0x7f1a6006ede0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.450+0000 7f1a84e00640 1 --2- 172.21.15.40:0/1075991096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a8015d270 0x7f1a800c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.450+0000 7f1a84e00640 1 --2- 172.21.15.40:0/1075991096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8015f380 0x7f1a8015cd30 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.450+0000 7f1a84e00640 1 --2- 172.21.15.40:0/1075991096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a800d0620 0x7f1a80152020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.450+0000 7f1a84e00640 1 -- 172.21.15.40:0/1075991096 >> 172.21.15.40:0/1075991096 conn(0x7f1a800d0be0 msgr2=0x7f1a800cdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.450+0000 7f1a84e00640 1 -- 172.21.15.40:0/1075991096 shutdown_connections 2024-04-26T23:08:20.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.450+0000 7f1a84e00640 1 -- 172.21.15.40:0/1075991096 wait complete. 2024-04-26T23:08:20.468 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:08:20.468 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:08:20.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-26T23:08:20.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.675+0000 7fd901c00640 1 Processor -- start 2024-04-26T23:08:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.675+0000 7fd901c00640 1 -- start start 2024-04-26T23:08:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.677+0000 7fd901c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd8fc0d1770 0x7fd8fc15ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.677+0000 7fd901c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8fc0d1b50 0x7fd8fc15f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.677+0000 7fd901c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8fc0d5d30 con 0x7fd8fc0b6470 2024-04-26T23:08:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.677+0000 7fd901c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd8fc0d3760 con 0x7fd8fc0d1770 2024-04-26T23:08:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.677+0000 7fd901c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd8fc0d09f0 con 0x7fd8fc0d1b50 2024-04-26T23:08:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.678+0000 7fd8f3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd8fc0b6470 0x7fd8fc0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60600/0 (socket says 172.21.15.40:60600) 2024-04-26T23:08:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.678+0000 7fd8f3e00640 1 -- 172.21.15.40:0/1523487471 learned_addr learned my addr 172.21.15.40:0/1523487471 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.678+0000 7fd8fbe00640 1 --2- 172.21.15.40:0/1523487471 >> v2:172.21.15.103:3300/0 conn(0x7fd8fc0d1770 0x7fd8fc15ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.678+0000 7fd901200640 1 --2- 172.21.15.40:0/1523487471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8fc0d1b50 0x7fd8fc15f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.678+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 653914774 0 0) 0x7fd8fc0d5d30 con 0x7fd8fc0b6470 2024-04-26T23:08:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.678+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8e0003520 con 0x7fd8fc0b6470 2024-04-26T23:08:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.678+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1044716891 0 0) 0x7fd8e0003520 con 0x7fd8fc0b6470 2024-04-26T23:08:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.678+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8fc0d1b50 msgr2=0x7fd8fc15f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.678+0000 7fd8fb400640 1 --2- 172.21.15.40:0/1523487471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8fc0d1b50 0x7fd8fc15f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.678+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 >> v2:172.21.15.103:3300/0 conn(0x7fd8fc0d1770 msgr2=0x7fd8fc15ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.678+0000 7fd8fb400640 1 --2- 172.21.15.40:0/1523487471 >> v2:172.21.15.103:3300/0 conn(0x7fd8fc0d1770 0x7fd8fc15ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.678+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8fc0d5a70 con 0x7fd8fc0b6470 2024-04-26T23:08:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fd8e4001670 con 0x7fd8fc0b6470 2024-04-26T23:08:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 >> v1:172.21.15.40:6789/0 conn(0x7fd8fc0b6470 legacy=0x7fd8fc0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8fb400640 1 --2- 172.21.15.40:0/1523487471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8e00039b0 0x7fd8e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8fb400640 1 --2- 172.21.15.40:0/1523487471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8e0004800 0x7fd8e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8fb400640 1 --2- 172.21.15.40:0/1523487471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8fc0d1b50 0x7fd8e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd8fc0d3760 con 0x7fd8e0004800 2024-04-26T23:08:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd8fc0d09f0 con 0x7fd8e00039b0 2024-04-26T23:08:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd8e000b9c0 con 0x7fd8fc0d1b50 2024-04-26T23:08:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/1523487471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8e0004800 0x7fd8e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8fbe00640 1 --2- 172.21.15.40:0/1523487471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8e00039b0 0x7fd8e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8f3e00640 1 -- 172.21.15.40:0/1523487471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8fc0d1b50 msgr2=0x7fd8e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/1523487471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8fc0d1b50 0x7fd8e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8f3e00640 1 -- 172.21.15.40:0/1523487471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8e00039b0 msgr2=0x7fd8e0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/1523487471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8e00039b0 0x7fd8e0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8f3e00640 1 -- 172.21.15.40:0/1523487471 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e0003520 con 0x7fd8e0004800 2024-04-26T23:08:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.679+0000 7fd8fbe00640 1 --2- 172.21.15.40:0/1523487471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8e00039b0 0x7fd8e0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:20.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.680+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/1523487471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8e0004800 0x7fd8e0004c80 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400c160 tx=0x7fd8e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6198643b31863b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.681+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4007c80 con 0x7fd8e0004800 2024-04-26T23:08:20.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.681+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e4005ad0 con 0x7fd8e0004800 2024-04-26T23:08:20.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.681+0000 7fd8fb400640 1 -- 172.21.15.40:0/1523487471 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4007070 con 0x7fd8e0004800 2024-04-26T23:08:20.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.681+0000 7fd901c00640 1 -- 172.21.15.40:0/1523487471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8e0004800 msgr2=0x7fd8e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.681+0000 7fd901c00640 1 --2- 172.21.15.40:0/1523487471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8e0004800 0x7fd8e0004c80 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400c160 tx=0x7fd8e0005830 comp rx=0 tx=0).stop 2024-04-26T23:08:20.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.681+0000 7fd901200640 1 -- 172.21.15.40:0/1523487471 reap_dead start 2024-04-26T23:08:20.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.681+0000 7fd901c00640 1 -- 172.21.15.40:0/1523487471 shutdown_connections 2024-04-26T23:08:20.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.681+0000 7fd901c00640 1 -- 172.21.15.40:0/1523487471 >> 172.21.15.40:0/1523487471 conn(0x7fd8fc0d4dd0 msgr2=0x7fd8fc0d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.682+0000 7fd901c00640 1 -- 172.21.15.40:0/1523487471 shutdown_connections 2024-04-26T23:08:20.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.682+0000 7fd901c00640 1 -- 172.21.15.40:0/1523487471 wait complete. 2024-04-26T23:08:20.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.683+0000 7fd901c00640 1 Processor -- start 2024-04-26T23:08:20.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.683+0000 7fd901c00640 1 -- start start 2024-04-26T23:08:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.684+0000 7fd901c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8e0004800 0x7fd8fc0cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.684+0000 7fd901c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8fc1576f0 0x7fd8fc155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.684+0000 7fd901c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8fc1e8ae0 0x7fd8fc0b99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.684+0000 7fd901c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd8fc0d3bf0 con 0x7fd8fc1576f0 2024-04-26T23:08:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.684+0000 7fd901c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd8fc0d2670 con 0x7fd8fc1e8ae0 2024-04-26T23:08:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.684+0000 7fd901c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd8fc0d3d70 con 0x7fd8e0004800 2024-04-26T23:08:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.685+0000 7fd8f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8fc1576f0 0x7fd8fc155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.685+0000 7fd8f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8fc1576f0 0x7fd8fc155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55004/0 (socket says 172.21.15.40:55004) 2024-04-26T23:08:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.685+0000 7fd8f3e00640 1 -- 172.21.15.40:0/2359022308 learned_addr learned my addr 172.21.15.40:0/2359022308 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.685+0000 7fd901200640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8fc1e8ae0 0x7fd8fc0b99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.685+0000 7fd8fbe00640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8e0004800 0x7fd8fc0cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.685+0000 7fd8f3e00640 1 -- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8e0004800 msgr2=0x7fd8fc0cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.685+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8e0004800 0x7fd8fc0cc860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.685+0000 7fd8f3e00640 1 -- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8fc1e8ae0 msgr2=0x7fd8fc0b99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.685+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8fc1e8ae0 0x7fd8fc0b99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.685+0000 7fd8f3e00640 1 -- 172.21.15.40:0/2359022308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8fc1574d0 con 0x7fd8fc1576f0 2024-04-26T23:08:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.685+0000 7fd901200640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8fc1e8ae0 0x7fd8fc0b99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.685+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8fc1576f0 0x7fd8fc155990 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400c280 tx=0x7fd8e4001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.686+0000 7fd8f8c00640 1 -- 172.21.15.40:0/2359022308 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4041020 con 0x7fd8fc1576f0 2024-04-26T23:08:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.686+0000 7fd8f8c00640 1 -- 172.21.15.40:0/2359022308 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e4001430 con 0x7fd8fc1576f0 2024-04-26T23:08:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.686+0000 7fd8f8c00640 1 -- 172.21.15.40:0/2359022308 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4007070 con 0x7fd8fc1576f0 2024-04-26T23:08:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.686+0000 7fd901c00640 1 -- 172.21.15.40:0/2359022308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8fc15b1c0 con 0x7fd8fc1576f0 2024-04-26T23:08:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.686+0000 7fd901c00640 1 -- 172.21.15.40:0/2359022308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8fc15aee0 con 0x7fd8fc1576f0 2024-04-26T23:08:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.688+0000 7fd8f8c00640 1 -- 172.21.15.40:0/2359022308 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd8e4044020 con 0x7fd8fc1576f0 2024-04-26T23:08:20.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.689+0000 7fd8f8c00640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd8d806c920 0x7fd8d806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.689+0000 7fd8f8c00640 1 -- 172.21.15.40:0/2359022308 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fd8e40ac0b0 con 0x7fd8fc1576f0 2024-04-26T23:08:20.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.690+0000 7fd906004740 1 Processor -- start 2024-04-26T23:08:20.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.690+0000 7fd906004740 1 -- start start 2024-04-26T23:08:20.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.690+0000 7fd8fbe00640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd8d806c920 0x7fd8d806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd906004740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e2d710 0x555ec8e490f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd906004740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 0x555ec8e2daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd906004740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 0x555ec8e3ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd906004740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555ec8d13030 con 0x555ec8e4b550 2024-04-26T23:08:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd906004740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555ec8d2e7f0 con 0x555ec8e2d710 2024-04-26T23:08:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd906004740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x555ec8ce97e0 con 0x555ec8e3e8d0 2024-04-26T23:08:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd8f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 0x555ec8e2daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd8f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 0x555ec8e2daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55012/0 (socket says 172.21.15.40:55012) 2024-04-26T23:08:20.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd8f3e00640 1 -- 172.21.15.40:0/312722725 learned_addr learned my addr 172.21.15.40:0/312722725 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd8fbe00640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd8d806c920 0x7fd8d806ede0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0099c0 tx=0x7fd8ec008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd8fbe00640 1 --2- 172.21.15.40:0/312722725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e2d710 0x555ec8e490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd901200640 1 --2- 172.21.15.40:0/312722725 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 0x555ec8e3ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd8f3e00640 1 -- 172.21.15.40:0/312722725 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 msgr2=0x555ec8e3ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/312722725 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 0x555ec8e3ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd8f3e00640 1 -- 172.21.15.40:0/312722725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e2d710 msgr2=0x555ec8e490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.691+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/312722725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e2d710 0x555ec8e490f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.692+0000 7fd8f3e00640 1 -- 172.21.15.40:0/312722725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ec8cf5050 con 0x555ec8e4b550 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.692+0000 7fd901200640 1 --2- 172.21.15.40:0/312722725 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 0x555ec8e3ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.692+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/312722725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 0x555ec8e2daf0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4079b40 tx=0x7fd8e4030e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a607a94beb5400 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.692+0000 7fd8f0c00640 1 -- 172.21.15.40:0/312722725 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e407df60 con 0x555ec8e4b550 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.692+0000 7fd8f0c00640 1 -- 172.21.15.40:0/312722725 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e4082790 con 0x555ec8e4b550 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.692+0000 7fd8f0c00640 1 -- 172.21.15.40:0/312722725 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4077eb0 con 0x555ec8e4b550 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.693+0000 7fd906004740 1 -- 172.21.15.40:0/312722725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 msgr2=0x555ec8e2daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.693+0000 7fd906004740 1 --2- 172.21.15.40:0/312722725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 0x555ec8e2daf0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4079b40 tx=0x7fd8e4030e60 comp rx=0 tx=0).stop 2024-04-26T23:08:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.693+0000 7fd906004740 1 -- 172.21.15.40:0/312722725 shutdown_connections 2024-04-26T23:08:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.693+0000 7fd906004740 1 --2- 172.21.15.40:0/312722725 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 0x555ec8e3ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.693+0000 7fd906004740 1 --2- 172.21.15.40:0/312722725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 0x555ec8e2daf0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.693+0000 7fd906004740 1 --2- 172.21.15.40:0/312722725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e2d710 0x555ec8e490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.693+0000 7fd906004740 1 -- 172.21.15.40:0/312722725 >> 172.21.15.40:0/312722725 conn(0x555ec8dffd20 msgr2=0x555ec8e48520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.693+0000 7fd906004740 1 -- 172.21.15.40:0/312722725 shutdown_connections 2024-04-26T23:08:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.693+0000 7fd906004740 1 -- 172.21.15.40:0/312722725 wait complete. 2024-04-26T23:08:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.694+0000 7fd906004740 1 -- start start 2024-04-26T23:08:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.694+0000 7fd906004740 1 Processor -- start 2024-04-26T23:08:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd906004740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e3e8d0 0x555ec8e53310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd906004740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e4b550 0x555ec8e2fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd906004740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e63320 0x555ec8e30400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd906004740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555ec8d6b200 con 0x555ec8e3e8d0 2024-04-26T23:08:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd906004740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555ec8d65ee0 con 0x555ec8e4b550 2024-04-26T23:08:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd906004740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x555ec8d72e70 con 0x555ec8e63320 2024-04-26T23:08:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd906004740 10 client.0 _ll_register_callbacks cb 0x555ec8d5c270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd8fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e3e8d0 0x555ec8e53310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd8fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e3e8d0 0x555ec8e53310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55024/0 (socket says 172.21.15.40:55024) 2024-04-26T23:08:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd8fbe00640 1 -- 172.21.15.40:0/747928656 learned_addr learned my addr 172.21.15.40:0/747928656 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd8fbe00640 1 -- 172.21.15.40:0/747928656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e63320 msgr2=0x555ec8e30400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd8fbe00640 1 --2- 172.21.15.40:0/747928656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e63320 0x555ec8e30400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd8fbe00640 1 -- 172.21.15.40:0/747928656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e4b550 msgr2=0x555ec8e2fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd8fbe00640 1 --2- 172.21.15.40:0/747928656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e4b550 0x555ec8e2fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.695+0000 7fd8fbe00640 1 -- 172.21.15.40:0/747928656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ec8b427d0 con 0x555ec8e3e8d0 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.696+0000 7fd8fbe00640 1 --2- 172.21.15.40:0/747928656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e3e8d0 0x555ec8e53310 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0156f0 tx=0x7fd8ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9939f18af1893c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.696+0000 7fd8e9600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.696+0000 7fd8e9600640 1 -- 172.21.15.40:0/747928656 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec026070 con 0x555ec8e3e8d0 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.696+0000 7fd8e9600640 1 -- 172.21.15.40:0/747928656 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fd8ec014730 con 0x555ec8e3e8d0 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.696+0000 7fd906004740 1 -- 172.21.15.40:0/747928656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8e4082790 con 0x555ec8e3e8d0 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.696+0000 7fd8e9600640 1 -- 172.21.15.40:0/747928656 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7fd8ec031050 con 0x555ec8e3e8d0 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.696+0000 7fd906004740 10 client.8466 fetch_fsmap learned FSMap version 99 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.696+0000 7fd906004740 1 -- 172.21.15.40:0/747928656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x555ec8a783f0 con 0x555ec8e3e8d0 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.697+0000 7fd8e9600640 1 -- 172.21.15.40:0/747928656 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e4082790 con 0x555ec8e3e8d0 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.697+0000 7fd8e9600640 1 -- 172.21.15.40:0/747928656 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 99) ==== 3550+0+0 (secure 0 0 0) 0x7fd8ec034020 con 0x555ec8e3e8d0 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.697+0000 7fd906004740 10 client.8466 fetch_fsmap finished waiting for FSMap version 99 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.697+0000 7fd906004740 10 client.8466 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.697+0000 7fd906004740 20 client.8466 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.697+0000 7fd906004740 1 --2- 172.21.15.40:0/747928656 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x555ec8e61810 0x555ec8e61cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.697+0000 7fd906004740 4 client.8466 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.697+0000 7fd906004740 1 -- 172.21.15.40:0/747928656 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ec8d747e0 con 0x555ec8e61810 2024-04-26T23:08:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.697+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/747928656 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x555ec8e61810 0x555ec8e61cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.698+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/747928656 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x555ec8e61810 0x555ec8e61cc0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66b685d81741ea0a server_cookie=e26b6c5feb85ec7d in_seq=0 out_seq=0 2024-04-26T23:08:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.698+0000 7fd8e9600640 10 client.8466 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.698+0000 7fd8e9600640 1 -- 172.21.15.40:0/747928656 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555ec8d747e0 con 0x555ec8e61810 2024-04-26T23:08:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.698+0000 7fd8e9600640 10 client.8466 handle_command_reply: tid=0 2024-04-26T23:08:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.698+0000 7fd906004740 1 client.8466 shutdown 2024-04-26T23:08:20.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.699+0000 7fd906004740 1 -- 172.21.15.40:0/747928656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e3e8d0 msgr2=0x555ec8e53310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.699+0000 7fd906004740 1 --2- 172.21.15.40:0/747928656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e3e8d0 0x555ec8e53310 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0156f0 tx=0x7fd8ec004730 comp rx=0 tx=0).stop 2024-04-26T23:08:20.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.699+0000 7fd906004740 1 -- 172.21.15.40:0/747928656 shutdown_connections 2024-04-26T23:08:20.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.699+0000 7fd906004740 1 --2- 172.21.15.40:0/747928656 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x555ec8e61810 0x555ec8e61cc0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.699+0000 7fd906004740 1 --2- 172.21.15.40:0/747928656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e63320 0x555ec8e30400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.699+0000 7fd906004740 1 --2- 172.21.15.40:0/747928656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e4b550 0x555ec8e2fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.699+0000 7fd906004740 1 --2- 172.21.15.40:0/747928656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e3e8d0 0x555ec8e53310 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.699+0000 7fd8e9600640 0 client.8466 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:20.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.699+0000 7fd906004740 1 -- 172.21.15.40:0/747928656 >> 172.21.15.40:0/747928656 conn(0x555ec8e2d710 msgr2=0x555ec8dfd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.699+0000 7fd906004740 1 -- 172.21.15.40:0/747928656 shutdown_connections 2024-04-26T23:08:20.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.699+0000 7fd906004740 1 -- 172.21.15.40:0/747928656 wait complete. 2024-04-26T23:08:20.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.699+0000 7fd906004740 20 client.8466 trim_cache size 0 max 16384 2024-04-26T23:08:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.718+0000 7fd906004740 1 Processor -- start 2024-04-26T23:08:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.718+0000 7fd906004740 1 -- start start 2024-04-26T23:08:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.719+0000 7fd906004740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 0x555ec8e011c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.719+0000 7fd906004740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e4b550 0x555ec8e027e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.719+0000 7fd906004740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e63320 0x555ec8e3cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.719+0000 7fd906004740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd8fc071e80 con 0x555ec8e63320 2024-04-26T23:08:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.719+0000 7fd906004740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd8fc071020 con 0x555ec8e4b550 2024-04-26T23:08:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.719+0000 7fd906004740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd8fc071750 con 0x555ec8e3e8d0 2024-04-26T23:08:20.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.719+0000 7fd901200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e63320 0x555ec8e3cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.719+0000 7fd901200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e63320 0x555ec8e3cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55028/0 (socket says 172.21.15.40:55028) 2024-04-26T23:08:20.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.719+0000 7fd901200640 1 -- 172.21.15.40:0/2877999293 learned_addr learned my addr 172.21.15.40:0/2877999293 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.720+0000 7fd901200640 1 -- 172.21.15.40:0/2877999293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 msgr2=0x555ec8e011c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.720+0000 7fd901200640 1 --2- 172.21.15.40:0/2877999293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 0x555ec8e011c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.720+0000 7fd901200640 1 -- 172.21.15.40:0/2877999293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e4b550 msgr2=0x555ec8e027e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.720+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/2877999293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e4b550 0x555ec8e027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.720+0000 7fd901200640 1 --2- 172.21.15.40:0/2877999293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e4b550 0x555ec8e027e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.720+0000 7fd901200640 1 -- 172.21.15.40:0/2877999293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ec8a380f0 con 0x555ec8e63320 2024-04-26T23:08:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.720+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/2877999293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e4b550 0x555ec8e027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.720+0000 7fd901200640 1 --2- 172.21.15.40:0/2877999293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e63320 0x555ec8e3cca0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009f60 tx=0x7fd8f4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5fc9bb57eb63e8a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.720+0000 7fd8ebe00640 1 -- 172.21.15.40:0/2877999293 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f401b070 con 0x555ec8e63320 2024-04-26T23:08:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.720+0000 7fd8ebe00640 1 -- 172.21.15.40:0/2877999293 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8f4014730 con 0x555ec8e63320 2024-04-26T23:08:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.720+0000 7fd8ebe00640 1 -- 172.21.15.40:0/2877999293 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4014b20 con 0x555ec8e63320 2024-04-26T23:08:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.721+0000 7fd906004740 1 -- 172.21.15.40:0/2877999293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e63320 msgr2=0x555ec8e3cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.721+0000 7fd906004740 1 --2- 172.21.15.40:0/2877999293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e63320 0x555ec8e3cca0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009f60 tx=0x7fd8f4009f90 comp rx=0 tx=0).stop 2024-04-26T23:08:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.721+0000 7fd906004740 1 -- 172.21.15.40:0/2877999293 shutdown_connections 2024-04-26T23:08:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.721+0000 7fd906004740 1 --2- 172.21.15.40:0/2877999293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e63320 0x555ec8e3cca0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.721+0000 7fd906004740 1 --2- 172.21.15.40:0/2877999293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e4b550 0x555ec8e027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.721+0000 7fd906004740 1 --2- 172.21.15.40:0/2877999293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 0x555ec8e011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.721+0000 7fd906004740 1 -- 172.21.15.40:0/2877999293 >> 172.21.15.40:0/2877999293 conn(0x555ec8e2d710 msgr2=0x555ec8e58fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.721+0000 7fd906004740 1 -- 172.21.15.40:0/2877999293 shutdown_connections 2024-04-26T23:08:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.721+0000 7fd906004740 1 -- 172.21.15.40:0/2877999293 wait complete. 2024-04-26T23:08:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.722+0000 7fd906004740 1 -- start start 2024-04-26T23:08:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.722+0000 7fd906004740 1 Processor -- start 2024-04-26T23:08:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.723+0000 7fd906004740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 0x555ec8e72b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.723+0000 7fd906004740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 0x555ec8e730b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.723+0000 7fd906004740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e63320 0x555ec8e69730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.723+0000 7fd906004740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555ec8d73fd0 con 0x555ec8e4b550 2024-04-26T23:08:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.723+0000 7fd906004740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd8fc156aa0 con 0x555ec8e63320 2024-04-26T23:08:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.723+0000 7fd906004740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x555ec8ecb080 con 0x555ec8e3e8d0 2024-04-26T23:08:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.723+0000 7fd906004740 10 client.0 _ll_register_callbacks cb 0x7fd8ec0096d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.723+0000 7fd8f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 0x555ec8e730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.723+0000 7fd8f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 0x555ec8e730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55038/0 (socket says 172.21.15.40:55038) 2024-04-26T23:08:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.723+0000 7fd8f3e00640 1 -- 172.21.15.40:0/1568227864 learned_addr learned my addr 172.21.15.40:0/1568227864 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.724+0000 7fd8f3e00640 1 -- 172.21.15.40:0/1568227864 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 msgr2=0x555ec8e72b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.724+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/1568227864 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 0x555ec8e72b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.724+0000 7fd8f3e00640 1 -- 172.21.15.40:0/1568227864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e63320 msgr2=0x555ec8e69730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.724+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/1568227864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e63320 0x555ec8e69730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.724+0000 7fd8f3e00640 1 -- 172.21.15.40:0/1568227864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ec8d2ea30 con 0x555ec8e4b550 2024-04-26T23:08:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.724+0000 7fd8f3e00640 1 --2- 172.21.15.40:0/1568227864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 0x555ec8e730b0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4079af0 tx=0x7fd8e407d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c31111921ef0a62 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.724+0000 7fd8e9600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.724+0000 7fd8e9600640 1 -- 172.21.15.40:0/1568227864 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4075e50 con 0x555ec8e4b550 2024-04-26T23:08:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.724+0000 7fd8e9600640 1 -- 172.21.15.40:0/1568227864 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fd8e4077eb0 con 0x555ec8e4b550 2024-04-26T23:08:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.724+0000 7fd906004740 1 -- 172.21.15.40:0/1568227864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8f4014730 con 0x555ec8e4b550 2024-04-26T23:08:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.724+0000 7fd8e9600640 1 -- 172.21.15.40:0/1568227864 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7fd8e4082790 con 0x555ec8e4b550 2024-04-26T23:08:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.724+0000 7fd906004740 10 client.8472 fetch_fsmap learned FSMap version 99 2024-04-26T23:08:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.724+0000 7fd906004740 1 -- 172.21.15.40:0/1568227864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x555ec8ecae00 con 0x555ec8e4b550 2024-04-26T23:08:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.725+0000 7fd8e9600640 1 -- 172.21.15.40:0/1568227864 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8f4014730 con 0x555ec8e4b550 2024-04-26T23:08:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.725+0000 7fd8e9600640 1 -- 172.21.15.40:0/1568227864 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 99) ==== 3550+0+0 (secure 0 0 0) 0x7fd8e404e060 con 0x555ec8e4b550 2024-04-26T23:08:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.725+0000 7fd906004740 10 client.8472 fetch_fsmap finished waiting for FSMap version 99 2024-04-26T23:08:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.725+0000 7fd906004740 10 client.8472 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-04-26T23:08:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.725+0000 7fd906004740 20 client.8472 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.725+0000 7fd906004740 1 --2- 172.21.15.40:0/1568227864 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x555ec8e69c70 0x555ec8e46cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.725+0000 7fd906004740 4 client.8472 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:08:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.725+0000 7fd906004740 1 -- 172.21.15.40:0/1568227864 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555ec8d6e650 con 0x555ec8e69c70 2024-04-26T23:08:20.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.725+0000 7fd8fbe00640 1 --2- 172.21.15.40:0/1568227864 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x555ec8e69c70 0x555ec8e46cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd8fbe00640 1 --2- 172.21.15.40:0/1568227864 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x555ec8e69c70 0x555ec8e46cf0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66fd5e8eeca23439 server_cookie=212b363e33400d3e in_seq=0 out_seq=0 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd8e9600640 10 client.8472 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd8e9600640 1 -- 172.21.15.40:0/1568227864 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x555ec8d6e650 con 0x555ec8e69c70 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd8e9600640 10 client.8472 handle_command_reply: tid=0 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd906004740 1 client.8472 shutdown 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd906004740 1 -- 172.21.15.40:0/1568227864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 msgr2=0x555ec8e730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd906004740 1 --2- 172.21.15.40:0/1568227864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 0x555ec8e730b0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4079af0 tx=0x7fd8e407d040 comp rx=0 tx=0).stop 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd906004740 1 -- 172.21.15.40:0/1568227864 shutdown_connections 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd906004740 1 --2- 172.21.15.40:0/1568227864 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x555ec8e69c70 0x555ec8e46cf0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd906004740 1 --2- 172.21.15.40:0/1568227864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555ec8e63320 0x555ec8e69730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd906004740 1 --2- 172.21.15.40:0/1568227864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555ec8e4b550 0x555ec8e730b0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd906004740 1 --2- 172.21.15.40:0/1568227864 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555ec8e3e8d0 0x555ec8e72b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd8e9600640 0 client.8472 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd906004740 1 -- 172.21.15.40:0/1568227864 >> 172.21.15.40:0/1568227864 conn(0x555ec8e2d710 msgr2=0x555ec8e01bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd906004740 1 -- 172.21.15.40:0/1568227864 shutdown_connections 2024-04-26T23:08:20.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd906004740 1 -- 172.21.15.40:0/1568227864 wait complete. 2024-04-26T23:08:20.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.726+0000 7fd906004740 20 client.8472 trim_cache size 0 max 16384 2024-04-26T23:08:20.730 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:20.730 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-26T23:08:20.730 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:08:20.730 INFO:teuthology.orchestra.run.smithi040.stdout: "b17426f8-9d25-4fbb-a99f-95207d3f0c9f": { 2024-04-26T23:08:20.730 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:08:20.730 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "b17426f8-9d25-4fbb-a99f-95207d3f0c9f", 2024-04-26T23:08:20.730 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:08:20.730 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:08:20.730 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:08:20.731 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:20.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.727+0000 7fd901c00640 1 -- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd8d806c920 msgr2=0x7fd8d806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.727+0000 7fd901c00640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd8d806c920 0x7fd8d806ede0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0099c0 tx=0x7fd8ec008040 comp rx=0 tx=0).stop 2024-04-26T23:08:20.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.727+0000 7fd901c00640 1 -- 172.21.15.40:0/2359022308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8fc1576f0 msgr2=0x7fd8fc155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.727+0000 7fd901c00640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8fc1576f0 0x7fd8fc155990 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400c280 tx=0x7fd8e4001970 comp rx=0 tx=0).stop 2024-04-26T23:08:20.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.727+0000 7fd901c00640 1 -- 172.21.15.40:0/2359022308 shutdown_connections 2024-04-26T23:08:20.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.727+0000 7fd901c00640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd8d806c920 0x7fd8d806ede0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.727+0000 7fd901c00640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8fc1e8ae0 0x7fd8fc0b99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.727+0000 7fd901c00640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8fc1576f0 0x7fd8fc155990 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.727+0000 7fd901c00640 1 --2- 172.21.15.40:0/2359022308 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8e0004800 0x7fd8fc0cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.727+0000 7fd901c00640 1 -- 172.21.15.40:0/2359022308 >> 172.21.15.40:0/2359022308 conn(0x7fd8fc0d4dd0 msgr2=0x7fd8fc0d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.727+0000 7fd901c00640 1 -- 172.21.15.40:0/2359022308 shutdown_connections 2024-04-26T23:08:20.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.727+0000 7fd901c00640 1 -- 172.21.15.40:0/2359022308 wait complete. 2024-04-26T23:08:20.743 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:08:20.743 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:20.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.935+0000 7f1ef6200640 1 Processor -- start 2024-04-26T23:08:20.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.935+0000 7f1ef6200640 1 -- start start 2024-04-26T23:08:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.937+0000 7f1ef6200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1ef0071e60 0x7f1ef006f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.937+0000 7f1ef6200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ef006fe20 0x7f1ef0072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.937+0000 7f1ef6200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1ef0072780 0x7f1ef00cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.937+0000 7f1ef6200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1ef015d7c0 con 0x7f1ef0072780 2024-04-26T23:08:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.937+0000 7f1ef6200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1ef015daa0 con 0x7f1ef0071e60 2024-04-26T23:08:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.937+0000 7f1ef6200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1ef015dd80 con 0x7f1ef006fe20 2024-04-26T23:08:20.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.937+0000 7f1eefe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1ef0072780 0x7f1ef00cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.938+0000 7f1eefe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1ef0072780 0x7f1ef00cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55054/0 (socket says 172.21.15.40:55054) 2024-04-26T23:08:20.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.938+0000 7f1eefe00640 1 -- 172.21.15.40:0/3796483750 learned_addr learned my addr 172.21.15.40:0/3796483750 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.938+0000 7f1eef400640 1 --2- 172.21.15.40:0/3796483750 >> v2:172.21.15.103:3300/0 conn(0x7f1ef0071e60 0x7f1ef006f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.938+0000 7f1eeea00640 1 --2- 172.21.15.40:0/3796483750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ef006fe20 0x7f1ef0072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.938+0000 7f1eefe00640 1 -- 172.21.15.40:0/3796483750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ef006fe20 msgr2=0x7f1ef0072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.938+0000 7f1eefe00640 1 --2- 172.21.15.40:0/3796483750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ef006fe20 0x7f1ef0072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.938+0000 7f1eefe00640 1 -- 172.21.15.40:0/3796483750 >> v2:172.21.15.103:3300/0 conn(0x7f1ef0071e60 msgr2=0x7f1ef006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:20.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.938+0000 7f1eefe00640 1 --2- 172.21.15.40:0/3796483750 >> v2:172.21.15.103:3300/0 conn(0x7f1ef0071e60 0x7f1ef006f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.938+0000 7f1eefe00640 1 -- 172.21.15.40:0/3796483750 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ef015e060 con 0x7f1ef0072780 2024-04-26T23:08:20.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.939+0000 7f1eefe00640 1 --2- 172.21.15.40:0/3796483750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef0072780 0x7f1ef00cb470 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00b3700 tx=0x7f1ee802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8b963c223d85b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.940+0000 7f1eee000640 1 -- 172.21.15.40:0/3796483750 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8036070 con 0x7f1ef0072780 2024-04-26T23:08:20.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.940+0000 7f1eee000640 1 -- 172.21.15.40:0/3796483750 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ee8031600 con 0x7f1ef0072780 2024-04-26T23:08:20.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.940+0000 7f1eee000640 1 -- 172.21.15.40:0/3796483750 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee80319d0 con 0x7f1ef0072780 2024-04-26T23:08:20.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.940+0000 7f1ef6200640 1 -- 172.21.15.40:0/3796483750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef0072780 msgr2=0x7f1ef00cb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.940+0000 7f1ef6200640 1 --2- 172.21.15.40:0/3796483750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef0072780 0x7f1ef00cb470 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00b3700 tx=0x7f1ee802a6a0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.940+0000 7f1ef6200640 1 -- 172.21.15.40:0/3796483750 shutdown_connections 2024-04-26T23:08:20.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.940+0000 7f1ef6200640 1 --2- 172.21.15.40:0/3796483750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef0072780 0x7f1ef00cb470 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.940+0000 7f1ef6200640 1 --2- 172.21.15.40:0/3796483750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ef006fe20 0x7f1ef0072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.940+0000 7f1ef6200640 1 --2- 172.21.15.40:0/3796483750 >> v2:172.21.15.103:3300/0 conn(0x7f1ef0071e60 0x7f1ef006f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.940+0000 7f1ef6200640 1 -- 172.21.15.40:0/3796483750 >> 172.21.15.40:0/3796483750 conn(0x7f1ef00d0be0 msgr2=0x7f1ef00d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:20.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.940+0000 7f1ef6200640 1 -- 172.21.15.40:0/3796483750 shutdown_connections 2024-04-26T23:08:20.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.941+0000 7f1ef6200640 1 -- 172.21.15.40:0/3796483750 wait complete. 2024-04-26T23:08:20.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.942+0000 7f1ef6200640 1 Processor -- start 2024-04-26T23:08:20.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.942+0000 7f1ef6200640 1 -- start start 2024-04-26T23:08:20.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.943+0000 7f1ef6200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ef006fe20 0x7f1ef0150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.943+0000 7f1ef6200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef0071e60 0x7f1ef015a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.943+0000 7f1ef6200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef0072780 0x7f1ef00b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.943+0000 7f1ef6200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef015d3f0 con 0x7f1ef0071e60 2024-04-26T23:08:20.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.943+0000 7f1ef6200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef0072c10 con 0x7f1ef0072780 2024-04-26T23:08:20.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.943+0000 7f1ef6200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1ef015d570 con 0x7f1ef006fe20 2024-04-26T23:08:20.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.944+0000 7f1eeea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef0071e60 0x7f1ef015a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.944+0000 7f1eeea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef0071e60 0x7f1ef015a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55064/0 (socket says 172.21.15.40:55064) 2024-04-26T23:08:20.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.944+0000 7f1eeea00640 1 -- 172.21.15.40:0/2227526434 learned_addr learned my addr 172.21.15.40:0/2227526434 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:20.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.944+0000 7f1eef400640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ef006fe20 0x7f1ef0150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.944+0000 7f1eefe00640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef0072780 0x7f1ef00b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.944+0000 7f1eeea00640 1 -- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ef006fe20 msgr2=0x7f1ef0150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.944+0000 7f1eeea00640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ef006fe20 0x7f1ef0150f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.944+0000 7f1eeea00640 1 -- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef0072780 msgr2=0x7f1ef00b5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.944+0000 7f1eeea00640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef0072780 0x7f1ef00b5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.944+0000 7f1eeea00640 1 -- 172.21.15.40:0/2227526434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ef00b6480 con 0x7f1ef0071e60 2024-04-26T23:08:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.945+0000 7f1eeea00640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef0071e60 0x7f1ef015a450 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f1edc00d830 tx=0x7f1edc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.945+0000 7f1ee7400640 1 -- 172.21.15.40:0/2227526434 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc018020 con 0x7f1ef0071e60 2024-04-26T23:08:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.945+0000 7f1ee7400640 1 -- 172.21.15.40:0/2227526434 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1edc0096a0 con 0x7f1ef0071e60 2024-04-26T23:08:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.945+0000 7f1ee7400640 1 -- 172.21.15.40:0/2227526434 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc009a70 con 0x7f1ef0071e60 2024-04-26T23:08:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.945+0000 7f1ef6200640 1 -- 172.21.15.40:0/2227526434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ef00b4e70 con 0x7f1ef0071e60 2024-04-26T23:08:20.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.945+0000 7f1ef6200640 1 -- 172.21.15.40:0/2227526434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ef00c8360 con 0x7f1ef0071e60 2024-04-26T23:08:20.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.946+0000 7f1ef6200640 1 -- 172.21.15.40:0/2227526434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ef00694f0 con 0x7f1ef0071e60 2024-04-26T23:08:20.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.947+0000 7f1ee7400640 1 -- 172.21.15.40:0/2227526434 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1edc01b020 con 0x7f1ef0071e60 2024-04-26T23:08:20.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.948+0000 7f1ee7400640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ed006c8f0 0x7f1ed006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:20.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.948+0000 7f1ee7400640 1 -- 172.21.15.40:0/2227526434 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f1edc08c140 con 0x7f1ef0071e60 2024-04-26T23:08:20.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.949+0000 7f1eef400640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ed006c8f0 0x7f1ed006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:20.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.950+0000 7f1eef400640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ed006c8f0 0x7f1ed006edb0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0009ad0 tx=0x7f1ee0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:20.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:20.953+0000 7f1ee7400640 1 -- 172.21.15.40:0/2227526434 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1edc022030 con 0x7f1ef0071e60 2024-04-26T23:08:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.203+0000 7f1ef6200640 1 -- 172.21.15.40:0/2227526434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ef0073b80 con 0x7f1ef0071e60 2024-04-26T23:08:21.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.205+0000 7f1ee7400640 1 -- 172.21.15.40:0/2227526434 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5227 (secure 0 0 0) 0x7f1edc0102e0 con 0x7f1ef0071e60 2024-04-26T23:08:21.205 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:21.205 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":99,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:18.428829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[8040,8031,6253]},"id":6}]} 2024-04-26T23:08:21.205 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 99 2024-04-26T23:08:21.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.207+0000 7f1ef6200640 1 -- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ed006c8f0 msgr2=0x7f1ed006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:21.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.207+0000 7f1ef6200640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ed006c8f0 0x7f1ed006edb0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0009ad0 tx=0x7f1ee0008040 comp rx=0 tx=0).stop 2024-04-26T23:08:21.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.207+0000 7f1ef6200640 1 -- 172.21.15.40:0/2227526434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef0071e60 msgr2=0x7f1ef015a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:21.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.207+0000 7f1ef6200640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef0071e60 0x7f1ef015a450 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f1edc00d830 tx=0x7f1edc00dcf0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.208+0000 7f1ef6200640 1 -- 172.21.15.40:0/2227526434 shutdown_connections 2024-04-26T23:08:21.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.208+0000 7f1ef6200640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ed006c8f0 0x7f1ed006edb0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.208+0000 7f1ef6200640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef0072780 0x7f1ef00b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.208+0000 7f1ef6200640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef0071e60 0x7f1ef015a450 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.208+0000 7f1ef6200640 1 --2- 172.21.15.40:0/2227526434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ef006fe20 0x7f1ef0150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.208+0000 7f1ef6200640 1 -- 172.21.15.40:0/2227526434 >> 172.21.15.40:0/2227526434 conn(0x7f1ef00d0be0 msgr2=0x7f1ef00b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:21.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.208+0000 7f1ef6200640 1 -- 172.21.15.40:0/2227526434 shutdown_connections 2024-04-26T23:08:21.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.208+0000 7f1ef6200640 1 -- 172.21.15.40:0/2227526434 wait complete. 2024-04-26T23:08:21.228 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'd'] 2024-04-26T23:08:21.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:08:21.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.440+0000 7fbb5d400640 1 Processor -- start 2024-04-26T23:08:21.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.440+0000 7fbb5d400640 1 -- start start 2024-04-26T23:08:21.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.442+0000 7fbb5d400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbb580d2400 0x7fbb580d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:21.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.442+0000 7fbb5d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbb580d0c80 0x7fbb580b5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:21.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.442+0000 7fbb5d400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb580d20f0 con 0x7fbb580d17d0 2024-04-26T23:08:21.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.442+0000 7fbb5d400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fbb580d37b0 con 0x7fbb580d0c80 2024-04-26T23:08:21.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.442+0000 7fbb5d400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbb580d3d70 con 0x7fbb580d2400 2024-04-26T23:08:21.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.443+0000 7fbb5ca00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbb580d0c80 0x7fbb580b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:21.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.443+0000 7fbb5ca00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbb580d0c80 0x7fbb580b5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55068/0 (socket says 172.21.15.40:55068) 2024-04-26T23:08:21.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.443+0000 7fbb5ca00640 1 -- 172.21.15.40:0/1741456696 learned_addr learned my addr 172.21.15.40:0/1741456696 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:21.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.443+0000 7fbb5ca00640 1 -- 172.21.15.40:0/1741456696 >> v2:172.21.15.103:3300/0 conn(0x7fbb580d2400 msgr2=0x7fbb580d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.443+0000 7fbb5ca00640 1 --2- 172.21.15.40:0/1741456696 >> v2:172.21.15.103:3300/0 conn(0x7fbb580d2400 0x7fbb580d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.443+0000 7fbb5ca00640 1 -- 172.21.15.40:0/1741456696 >> v1:172.21.15.40:6789/0 conn(0x7fbb580d17d0 legacy=0x7fbb580b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.443+0000 7fbb5ca00640 1 -- 172.21.15.40:0/1741456696 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5815f9c0 con 0x7fbb580d0c80 2024-04-26T23:08:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.444+0000 7fbb5ca00640 1 --2- 172.21.15.40:0/1741456696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb580d0c80 0x7fbb580b5210 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fbb580b6920 tx=0x7fbb5002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6355ffedb8fb2be server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.444+0000 7fbb56a00640 1 -- 172.21.15.40:0/1741456696 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb50036070 con 0x7fbb580d0c80 2024-04-26T23:08:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.445+0000 7fbb56a00640 1 -- 172.21.15.40:0/1741456696 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb50031600 con 0x7fbb580d0c80 2024-04-26T23:08:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.445+0000 7fbb56a00640 1 -- 172.21.15.40:0/1741456696 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb500319d0 con 0x7fbb580d0c80 2024-04-26T23:08:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.445+0000 7fbb5d400640 1 -- 172.21.15.40:0/1741456696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb580d0c80 msgr2=0x7fbb580b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.445+0000 7fbb5d400640 1 --2- 172.21.15.40:0/1741456696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb580d0c80 0x7fbb580b5210 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fbb580b6920 tx=0x7fbb5002a6a0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.445+0000 7fbb5d400640 1 -- 172.21.15.40:0/1741456696 shutdown_connections 2024-04-26T23:08:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.445+0000 7fbb5d400640 1 --2- 172.21.15.40:0/1741456696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb580d0c80 0x7fbb580b5210 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.445+0000 7fbb5d400640 1 --2- 172.21.15.40:0/1741456696 >> v2:172.21.15.103:3300/0 conn(0x7fbb580d2400 0x7fbb580d1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.445+0000 7fbb5d400640 1 -- 172.21.15.40:0/1741456696 >> 172.21.15.40:0/1741456696 conn(0x7fbb5809e210 msgr2=0x7fbb580d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.445+0000 7fbb5d400640 1 -- 172.21.15.40:0/1741456696 shutdown_connections 2024-04-26T23:08:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.445+0000 7fbb5d400640 1 -- 172.21.15.40:0/1741456696 wait complete. 2024-04-26T23:08:21.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.447+0000 7fbb5d400640 1 Processor -- start 2024-04-26T23:08:21.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.447+0000 7fbb5d400640 1 -- start start 2024-04-26T23:08:21.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.448+0000 7fbb5d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb580d0c80 0x7fbb580cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:21.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.448+0000 7fbb5d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb580d17d0 0x7fbb5815dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:21.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.448+0000 7fbb5d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb580d2400 0x7fbb58158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:21.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.448+0000 7fbb5d400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbb580d3a00 con 0x7fbb580d0c80 2024-04-26T23:08:21.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.448+0000 7fbb5d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbb580d5a10 con 0x7fbb580d2400 2024-04-26T23:08:21.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.448+0000 7fbb5d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbb580d10d0 con 0x7fbb580d17d0 2024-04-26T23:08:21.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.449+0000 7fbb57e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb580d0c80 0x7fbb580cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:21.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.449+0000 7fbb57e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb580d0c80 0x7fbb580cbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55072/0 (socket says 172.21.15.40:55072) 2024-04-26T23:08:21.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.449+0000 7fbb57e00640 1 -- 172.21.15.40:0/673718903 learned_addr learned my addr 172.21.15.40:0/673718903 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:21.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.449+0000 7fbb5ca00640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb580d2400 0x7fbb58158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:21.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.449+0000 7fbb57400640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb580d17d0 0x7fbb5815dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:21.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.449+0000 7fbb57e00640 1 -- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb580d17d0 msgr2=0x7fbb5815dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:21.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.449+0000 7fbb57e00640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb580d17d0 0x7fbb5815dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.449+0000 7fbb57e00640 1 -- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb580d2400 msgr2=0x7fbb58158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:21.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.449+0000 7fbb57e00640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb580d2400 0x7fbb58158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.449+0000 7fbb57e00640 1 -- 172.21.15.40:0/673718903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb580cc7c0 con 0x7fbb580d0c80 2024-04-26T23:08:21.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.449+0000 7fbb5ca00640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb580d2400 0x7fbb58158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:08:21.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.450+0000 7fbb57e00640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb580d0c80 0x7fbb580cbf70 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800c430 tx=0x7fbb4800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:21.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.450+0000 7fbb4fe00640 1 -- 172.21.15.40:0/673718903 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480052d0 con 0x7fbb580d0c80 2024-04-26T23:08:21.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.450+0000 7fbb4fe00640 1 -- 172.21.15.40:0/673718903 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb48005490 con 0x7fbb580d0c80 2024-04-26T23:08:21.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.450+0000 7fbb5d400640 1 -- 172.21.15.40:0/673718903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb580ccaa0 con 0x7fbb580d0c80 2024-04-26T23:08:21.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.450+0000 7fbb4fe00640 1 -- 172.21.15.40:0/673718903 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48007260 con 0x7fbb580d0c80 2024-04-26T23:08:21.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.450+0000 7fbb5d400640 1 -- 172.21.15.40:0/673718903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb580cd060 con 0x7fbb580d0c80 2024-04-26T23:08:21.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.451+0000 7fbb5d400640 1 -- 172.21.15.40:0/673718903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb58069370 con 0x7fbb580d0c80 2024-04-26T23:08:21.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.452+0000 7fbb4fe00640 1 -- 172.21.15.40:0/673718903 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbb48007400 con 0x7fbb580d0c80 2024-04-26T23:08:21.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.453+0000 7fbb4fe00640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbb3806c8f0 0x7fbb3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:21.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.453+0000 7fbb4fe00640 1 -- 172.21.15.40:0/673718903 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fbb4808b230 con 0x7fbb580d0c80 2024-04-26T23:08:21.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.454+0000 7fbb57400640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbb3806c8f0 0x7fbb3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:21.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.455+0000 7fbb57400640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbb3806c8f0 0x7fbb3806edb0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbb581f8d30 tx=0x7fbb44008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:21.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.458+0000 7fbb4fe00640 1 -- 172.21.15.40:0/673718903 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fbb480594c0 con 0x7fbb580d0c80 2024-04-26T23:08:21.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.735+0000 7fbb5d400640 1 -- 172.21.15.40:0/673718903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fbb580d5470 con 0x7fbb580d0c80 2024-04-26T23:08:21.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.736+0000 7fbb4fe00640 1 -- 172.21.15.40:0/673718903 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1339 (secure 0 0 0) 0x7fbb48058dc0 con 0x7fbb580d0c80 2024-04-26T23:08:21.736 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":461,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":90,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":700185,"bytes_used":265830400,"bytes_avail":772828282880,"bytes_total":773094113280,"read_bytes_sec":0,"write_bytes_sec":10662,"read_op_per_sec":0,"write_op_per_sec":8},"fsmap":{"epoch":99,"id":6,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":6,"rank":2,"name":"d","status":"up:active","gid":6253},{"filesystem_id":6,"rank":1,"name":"a","status":"up:active","gid":8031},{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8040}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-26T23:07:35.290664+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T23:08:21.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.739+0000 7fbb5d400640 1 -- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbb3806c8f0 msgr2=0x7fbb3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:21.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.739+0000 7fbb5d400640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbb3806c8f0 0x7fbb3806edb0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbb581f8d30 tx=0x7fbb44008040 comp rx=0 tx=0).stop 2024-04-26T23:08:21.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.739+0000 7fbb5d400640 1 -- 172.21.15.40:0/673718903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb580d0c80 msgr2=0x7fbb580cbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:21.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.739+0000 7fbb5d400640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb580d0c80 0x7fbb580cbf70 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800c430 tx=0x7fbb4800c8f0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.739+0000 7fbb5d400640 1 -- 172.21.15.40:0/673718903 shutdown_connections 2024-04-26T23:08:21.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.739+0000 7fbb5d400640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbb3806c8f0 0x7fbb3806edb0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.739+0000 7fbb5d400640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb580d2400 0x7fbb58158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.739+0000 7fbb5d400640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb580d17d0 0x7fbb5815dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.739+0000 7fbb5d400640 1 --2- 172.21.15.40:0/673718903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb580d0c80 0x7fbb580cbf70 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:21.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.739+0000 7fbb5d400640 1 -- 172.21.15.40:0/673718903 >> 172.21.15.40:0/673718903 conn(0x7fbb5809e210 msgr2=0x7fbb580d4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:21.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.739+0000 7fbb5d400640 1 -- 172.21.15.40:0/673718903 shutdown_connections 2024-04-26T23:08:21.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:21.739+0000 7fbb5d400640 1 -- 172.21.15.40:0/673718903 wait complete. 2024-04-26T23:08:21.757 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-26T23:08:22.759 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:22.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.972+0000 7f039be00640 1 Processor -- start 2024-04-26T23:08:22.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.973+0000 7f039be00640 1 -- start start 2024-04-26T23:08:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.974+0000 7f039be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f039c0cc980 0x7f039c0cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.974+0000 7f039be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f039c0cc230 0x7f039c158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.974+0000 7f039be00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f039c0cce10 con 0x7f039c0c6f10 2024-04-26T23:08:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.974+0000 7f039be00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f039c0c72f0 con 0x7f039c0cc230 2024-04-26T23:08:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.974+0000 7f039be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f039c0cb860 con 0x7f039c0cc980 2024-04-26T23:08:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.974+0000 7f039b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f039c0cc230 0x7f039c158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.974+0000 7f039aa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f039c0c6f10 0x7f039c0c5ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60622/0 (socket says 172.21.15.40:60622) 2024-04-26T23:08:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.974+0000 7f039aa00640 1 -- 172.21.15.40:0/2322389377 learned_addr learned my addr 172.21.15.40:0/2322389377 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.975+0000 7f039a000640 1 --2- 172.21.15.40:0/2322389377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f039c0cc980 0x7f039c0cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.975+0000 7f039b400640 1 -- 172.21.15.40:0/2322389377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f039c0cc980 msgr2=0x7f039c0cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:22.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.975+0000 7f039b400640 1 --2- 172.21.15.40:0/2322389377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f039c0cc980 0x7f039c0cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:22.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.975+0000 7f039b400640 1 -- 172.21.15.40:0/2322389377 >> v1:172.21.15.40:6789/0 conn(0x7f039c0c6f10 legacy=0x7f039c0c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:22.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.975+0000 7f039b400640 1 -- 172.21.15.40:0/2322389377 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f039c0cd0f0 con 0x7f039c0cc230 2024-04-26T23:08:22.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.976+0000 7f039b400640 1 --2- 172.21.15.40:0/2322389377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0cc230 0x7f039c158cd0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f0394009d90 tx=0x7f039402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6c016e47f506bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:22.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.976+0000 7f0399600640 1 -- 172.21.15.40:0/2322389377 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394031600 con 0x7f039c0cc230 2024-04-26T23:08:22.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.976+0000 7f0399600640 1 -- 172.21.15.40:0/2322389377 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03940317a0 con 0x7f039c0cc230 2024-04-26T23:08:22.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.976+0000 7f0399600640 1 -- 172.21.15.40:0/2322389377 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394031b70 con 0x7f039c0cc230 2024-04-26T23:08:22.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.977+0000 7f039be00640 1 -- 172.21.15.40:0/2322389377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0cc230 msgr2=0x7f039c158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:22.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.977+0000 7f039be00640 1 --2- 172.21.15.40:0/2322389377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0cc230 0x7f039c158cd0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f0394009d90 tx=0x7f039402a5d0 comp rx=0 tx=0).stop 2024-04-26T23:08:22.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.977+0000 7f039be00640 1 -- 172.21.15.40:0/2322389377 shutdown_connections 2024-04-26T23:08:22.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.977+0000 7f039be00640 1 --2- 172.21.15.40:0/2322389377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0cc230 0x7f039c158cd0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:22.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.977+0000 7f039be00640 1 --2- 172.21.15.40:0/2322389377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f039c0cc980 0x7f039c0cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:22.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.977+0000 7f039be00640 1 -- 172.21.15.40:0/2322389377 >> 172.21.15.40:0/2322389377 conn(0x7f039c0d0a10 msgr2=0x7f039c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:22.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.977+0000 7f039be00640 1 -- 172.21.15.40:0/2322389377 shutdown_connections 2024-04-26T23:08:22.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.977+0000 7f039be00640 1 -- 172.21.15.40:0/2322389377 wait complete. 2024-04-26T23:08:22.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.979+0000 7f039be00640 1 Processor -- start 2024-04-26T23:08:22.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.979+0000 7f039be00640 1 -- start start 2024-04-26T23:08:22.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.980+0000 7f039be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f039c0c6f10 0x7f039c1f8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:22.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.980+0000 7f039be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f039c0cc230 0x7f039c0c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:22.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.980+0000 7f039be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c15c570 0x7f039c1505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:22.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.980+0000 7f039be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f039c0c61e0 con 0x7f039c15c570 2024-04-26T23:08:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.980+0000 7f039be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f039c0cb1a0 con 0x7f039c0cc230 2024-04-26T23:08:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.980+0000 7f039be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f039c0c6360 con 0x7f039c0c6f10 2024-04-26T23:08:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.980+0000 7f039b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c15c570 0x7f039c1505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.981+0000 7f039b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c15c570 0x7f039c1505e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55088/0 (socket says 172.21.15.40:55088) 2024-04-26T23:08:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.981+0000 7f039b400640 1 -- 172.21.15.40:0/3430974597 learned_addr learned my addr 172.21.15.40:0/3430974597 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.981+0000 7f039aa00640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f039c0c6f10 0x7f039c1f8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.981+0000 7f039a000640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f039c0cc230 0x7f039c0c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.981+0000 7f039b400640 1 -- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f039c0c6f10 msgr2=0x7f039c1f8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.981+0000 7f039b400640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f039c0c6f10 0x7f039c1f8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.981+0000 7f039b400640 1 -- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f039c0cc230 msgr2=0x7f039c0c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:22.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.981+0000 7f039b400640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f039c0cc230 0x7f039c0c8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:22.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.981+0000 7f039b400640 1 -- 172.21.15.40:0/3430974597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f039c1f4b80 con 0x7f039c15c570 2024-04-26T23:08:22.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.981+0000 7f039a000640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f039c0cc230 0x7f039c0c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:22.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.981+0000 7f039aa00640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f039c0c6f10 0x7f039c1f8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:08:22.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.982+0000 7f039b400640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c15c570 0x7f039c1505e0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f039402abc0 tx=0x7f0394004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:22.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.982+0000 7f0392a00640 1 -- 172.21.15.40:0/3430974597 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394040070 con 0x7f039c15c570 2024-04-26T23:08:22.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.982+0000 7f0392a00640 1 -- 172.21.15.40:0/3430974597 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0394030c00 con 0x7f039c15c570 2024-04-26T23:08:22.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.982+0000 7f0392a00640 1 -- 172.21.15.40:0/3430974597 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394009450 con 0x7f039c15c570 2024-04-26T23:08:22.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.982+0000 7f039be00640 1 -- 172.21.15.40:0/3430974597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f039c0c8550 con 0x7f039c15c570 2024-04-26T23:08:22.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.982+0000 7f039be00640 1 -- 172.21.15.40:0/3430974597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f039c0c8b00 con 0x7f039c15c570 2024-04-26T23:08:22.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.983+0000 7f039be00640 1 -- 172.21.15.40:0/3430974597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f039c069370 con 0x7f039c15c570 2024-04-26T23:08:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.984+0000 7f0392a00640 1 -- 172.21.15.40:0/3430974597 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0394039040 con 0x7f039c15c570 2024-04-26T23:08:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.985+0000 7f0392a00640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f037806c920 0x7f037806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.985+0000 7f0392a00640 1 -- 172.21.15.40:0/3430974597 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f03940acc20 con 0x7f039c15c570 2024-04-26T23:08:22.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.986+0000 7f039aa00640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f037806c920 0x7f037806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.987+0000 7f039aa00640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f037806c920 0x7f037806ede0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f038c000f60 tx=0x7f038c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:22.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:22.990+0000 7f0392a00640 1 -- 172.21.15.40:0/3430974597 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f039407aeb0 con 0x7f039c15c570 2024-04-26T23:08:23.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.234+0000 7f039be00640 1 -- 172.21.15.40:0/3430974597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f039c070020 con 0x7f039c15c570 2024-04-26T23:08:23.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.235+0000 7f0392a00640 1 -- 172.21.15.40:0/3430974597 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5227 (secure 0 0 0) 0x7f039407a7b0 con 0x7f039c15c570 2024-04-26T23:08:23.236 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:23.236 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":99,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:18.428829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[8040,8031,6253]},"id":6}]} 2024-04-26T23:08:23.237 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 99 2024-04-26T23:08:23.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.238+0000 7f039be00640 1 -- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f037806c920 msgr2=0x7f037806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:23.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.238+0000 7f039be00640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f037806c920 0x7f037806ede0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f038c000f60 tx=0x7f038c009040 comp rx=0 tx=0).stop 2024-04-26T23:08:23.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.238+0000 7f039be00640 1 -- 172.21.15.40:0/3430974597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c15c570 msgr2=0x7f039c1505e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:23.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.238+0000 7f039be00640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c15c570 0x7f039c1505e0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f039402abc0 tx=0x7f0394004a40 comp rx=0 tx=0).stop 2024-04-26T23:08:23.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.238+0000 7f039be00640 1 -- 172.21.15.40:0/3430974597 shutdown_connections 2024-04-26T23:08:23.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.238+0000 7f039be00640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f037806c920 0x7f037806ede0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.238+0000 7f039be00640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c15c570 0x7f039c1505e0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.239+0000 7f039be00640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f039c0cc230 0x7f039c0c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.239+0000 7f039be00640 1 --2- 172.21.15.40:0/3430974597 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f039c0c6f10 0x7f039c1f8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.239+0000 7f039be00640 1 -- 172.21.15.40:0/3430974597 >> 172.21.15.40:0/3430974597 conn(0x7f039c0d0a10 msgr2=0x7f039c06dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:23.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.239+0000 7f039be00640 1 -- 172.21.15.40:0/3430974597 shutdown_connections 2024-04-26T23:08:23.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.239+0000 7f039be00640 1 -- 172.21.15.40:0/3430974597 wait complete. 2024-04-26T23:08:23.258 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'd'] 2024-04-26T23:08:23.259 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:08:23.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.459+0000 7f9128600640 1 Processor -- start 2024-04-26T23:08:23.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.459+0000 7f9128600640 1 -- start start 2024-04-26T23:08:23.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.461+0000 7f9128600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f912015f1d0 0x7f912006f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:23.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.461+0000 7f9128600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f912006c240 0x7f912006c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:23.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.461+0000 7f9128600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91200b4480 con 0x7f91200726e0 2024-04-26T23:08:23.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.461+0000 7f9128600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f912006cbe0 con 0x7f912006c240 2024-04-26T23:08:23.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.461+0000 7f9128600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f91200ae8c0 con 0x7f912015f1d0 2024-04-26T23:08:23.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.461+0000 7f9125200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f912006c240 0x7f912006c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:23.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.461+0000 7f9125200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f912006c240 0x7f912006c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55102/0 (socket says 172.21.15.40:55102) 2024-04-26T23:08:23.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.461+0000 7f9125200640 1 -- 172.21.15.40:0/2106044179 learned_addr learned my addr 172.21.15.40:0/2106044179 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:23.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.461+0000 7f9125c00640 1 --2- 172.21.15.40:0/2106044179 >> v2:172.21.15.103:3300/0 conn(0x7f912015f1d0 0x7f912006f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.462+0000 7f9125200640 1 -- 172.21.15.40:0/2106044179 >> v2:172.21.15.103:3300/0 conn(0x7f912015f1d0 msgr2=0x7f912006f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.462+0000 7f9125200640 1 --2- 172.21.15.40:0/2106044179 >> v2:172.21.15.103:3300/0 conn(0x7f912015f1d0 0x7f912006f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.462+0000 7f9125200640 1 -- 172.21.15.40:0/2106044179 >> v1:172.21.15.103:6789/0 conn(0x7f91200726e0 legacy=0x7f9120072b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.462+0000 7f9125200640 1 -- 172.21.15.40:0/2106044179 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91200b4720 con 0x7f912006c240 2024-04-26T23:08:23.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.462+0000 7f9125c00640 1 --2- 172.21.15.40:0/2106044179 >> v2:172.21.15.103:3300/0 conn(0x7f912015f1d0 0x7f912006f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.463+0000 7f9125200640 1 --2- 172.21.15.40:0/2106044179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f912006c240 0x7f912006c6a0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f912006cfb0 tx=0x7f910c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81fe5f5cbe8c94a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.463+0000 7f911fe00640 1 -- 172.21.15.40:0/2106044179 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910c037070 con 0x7f912006c240 2024-04-26T23:08:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.463+0000 7f911fe00640 1 -- 172.21.15.40:0/2106044179 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f910c032440 con 0x7f912006c240 2024-04-26T23:08:23.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.463+0000 7f911fe00640 1 -- 172.21.15.40:0/2106044179 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910c032810 con 0x7f912006c240 2024-04-26T23:08:23.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.464+0000 7f9128600640 1 -- 172.21.15.40:0/2106044179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f912006c240 msgr2=0x7f912006c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:23.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.464+0000 7f9128600640 1 --2- 172.21.15.40:0/2106044179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f912006c240 0x7f912006c6a0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f912006cfb0 tx=0x7f910c02a230 comp rx=0 tx=0).stop 2024-04-26T23:08:23.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.464+0000 7f9128600640 1 -- 172.21.15.40:0/2106044179 shutdown_connections 2024-04-26T23:08:23.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.464+0000 7f9128600640 1 --2- 172.21.15.40:0/2106044179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f912006c240 0x7f912006c6a0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.464+0000 7f9128600640 1 --2- 172.21.15.40:0/2106044179 >> v2:172.21.15.103:3300/0 conn(0x7f912015f1d0 0x7f912006f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.464+0000 7f9128600640 1 -- 172.21.15.40:0/2106044179 >> 172.21.15.40:0/2106044179 conn(0x7f91200b3180 msgr2=0x7f91200b3560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:23.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.464+0000 7f9128600640 1 -- 172.21.15.40:0/2106044179 shutdown_connections 2024-04-26T23:08:23.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.464+0000 7f9128600640 1 -- 172.21.15.40:0/2106044179 wait complete. 2024-04-26T23:08:23.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.466+0000 7f9128600640 1 Processor -- start 2024-04-26T23:08:23.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.466+0000 7f9128600640 1 -- start start 2024-04-26T23:08:23.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.467+0000 7f9128600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f912006c240 0x7f912015d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:23.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.467+0000 7f9128600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91200726e0 0x7f91200c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:23.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.467+0000 7f9128600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f912015f1d0 0x7f91201588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:23.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.467+0000 7f9128600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f91200b4fc0 con 0x7f91200726e0 2024-04-26T23:08:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.467+0000 7f9128600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f91200b4ce0 con 0x7f912006c240 2024-04-26T23:08:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.467+0000 7f9128600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f912006f6b0 con 0x7f912015f1d0 2024-04-26T23:08:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.467+0000 7f9125200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91200726e0 0x7f91200c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.467+0000 7f9125200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91200726e0 0x7f91200c76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55114/0 (socket says 172.21.15.40:55114) 2024-04-26T23:08:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.467+0000 7f9125200640 1 -- 172.21.15.40:0/3976698890 learned_addr learned my addr 172.21.15.40:0/3976698890 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.468+0000 7f9125c00640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f912006c240 0x7f912015d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.468+0000 7f9125200640 1 -- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f912015f1d0 msgr2=0x7f91201588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.468+0000 7f9126600640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f912015f1d0 0x7f91201588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.468+0000 7f9125200640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f912015f1d0 0x7f91201588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.468+0000 7f9125200640 1 -- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f912006c240 msgr2=0x7f912015d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.468+0000 7f9125200640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f912006c240 0x7f912015d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.468+0000 7f9125200640 1 -- 172.21.15.40:0/3976698890 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91201f4d10 con 0x7f91200726e0 2024-04-26T23:08:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.468+0000 7f9125c00640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f912006c240 0x7f912015d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.468+0000 7f9125200640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91200726e0 0x7f91200c76f0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f910c000c00 tx=0x7f910c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.469+0000 7f911d600640 1 -- 172.21.15.40:0/3976698890 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910c041070 con 0x7f91200726e0 2024-04-26T23:08:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.469+0000 7f911d600640 1 -- 172.21.15.40:0/3976698890 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f910c03c870 con 0x7f91200726e0 2024-04-26T23:08:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.469+0000 7f9128600640 1 -- 172.21.15.40:0/3976698890 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f912015acb0 con 0x7f91200726e0 2024-04-26T23:08:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.469+0000 7f911d600640 1 -- 172.21.15.40:0/3976698890 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910c037040 con 0x7f91200726e0 2024-04-26T23:08:23.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.469+0000 7f9128600640 1 -- 172.21.15.40:0/3976698890 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f912015b4f0 con 0x7f91200726e0 2024-04-26T23:08:23.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.470+0000 7f9128600640 1 -- 172.21.15.40:0/3976698890 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9120068fc0 con 0x7f91200726e0 2024-04-26T23:08:23.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.471+0000 7f911d600640 1 -- 172.21.15.40:0/3976698890 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f910c044070 con 0x7f91200726e0 2024-04-26T23:08:23.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.472+0000 7f911d600640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f910006c8f0 0x7f910006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:23.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.472+0000 7f911d600640 1 -- 172.21.15.40:0/3976698890 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f910c0aca00 con 0x7f91200726e0 2024-04-26T23:08:23.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.473+0000 7f9125c00640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f910006c8f0 0x7f910006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:23.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.474+0000 7f9125c00640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f910006c8f0 0x7f910006edb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9114006fd0 tx=0x7f9114008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:23.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.477+0000 7f911d600640 1 -- 172.21.15.40:0/3976698890 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f912015acb0 con 0x7f91200726e0 2024-04-26T23:08:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.756+0000 7f9128600640 1 -- 172.21.15.40:0/3976698890 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f91201629f0 con 0x7f91200726e0 2024-04-26T23:08:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.757+0000 7f911d600640 1 -- 172.21.15.40:0/3976698890 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1339 (secure 0 0 0) 0x7f910c07ac00 con 0x7f91200726e0 2024-04-26T23:08:23.757 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:23.758 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":463,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":90,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":700185,"bytes_used":265830400,"bytes_avail":772828282880,"bytes_total":773094113280,"read_bytes_sec":0,"write_bytes_sec":10065,"read_op_per_sec":0,"write_op_per_sec":8},"fsmap":{"epoch":99,"id":6,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":6,"rank":2,"name":"d","status":"up:active","gid":6253},{"filesystem_id":6,"rank":1,"name":"a","status":"up:active","gid":8031},{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8040}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-26T23:07:35.290664+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T23:08:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.760+0000 7f9128600640 1 -- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f910006c8f0 msgr2=0x7f910006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.760+0000 7f9128600640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f910006c8f0 0x7f910006edb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9114006fd0 tx=0x7f9114008040 comp rx=0 tx=0).stop 2024-04-26T23:08:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.760+0000 7f9128600640 1 -- 172.21.15.40:0/3976698890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91200726e0 msgr2=0x7f91200c76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.760+0000 7f9128600640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91200726e0 0x7f91200c76f0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f910c000c00 tx=0x7f910c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.760+0000 7f9128600640 1 -- 172.21.15.40:0/3976698890 shutdown_connections 2024-04-26T23:08:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.760+0000 7f9128600640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f910006c8f0 0x7f910006edb0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.760+0000 7f9128600640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f912015f1d0 0x7f91201588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.760+0000 7f9128600640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91200726e0 0x7f91200c76f0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.760+0000 7f9128600640 1 --2- 172.21.15.40:0/3976698890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f912006c240 0x7f912015d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.760+0000 7f9128600640 1 -- 172.21.15.40:0/3976698890 >> 172.21.15.40:0/3976698890 conn(0x7f91200b3180 msgr2=0x7f9120161a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.761+0000 7f9128600640 1 -- 172.21.15.40:0/3976698890 shutdown_connections 2024-04-26T23:08:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:23.761+0000 7f9128600640 1 -- 172.21.15.40:0/3976698890 wait complete. 2024-04-26T23:08:23.779 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-26T23:08:24.780 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:25.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.005+0000 7f1dace00640 1 Processor -- start 2024-04-26T23:08:25.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.006+0000 7f1dace00640 1 -- start start 2024-04-26T23:08:25.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.007+0000 7f1dace00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1da815ce60 0x7f1da80afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:25.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.007+0000 7f1dace00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1da80af7a0 0x7f1da80ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:25.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.007+0000 7f1dace00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1da80d0820 con 0x7f1da8071fe0 2024-04-26T23:08:25.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.007+0000 7f1dace00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1da80cc450 con 0x7f1da815ce60 2024-04-26T23:08:25.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.007+0000 7f1dace00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1da81614c0 con 0x7f1da80af7a0 2024-04-26T23:08:25.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.008+0000 7f1da7e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1da8071fe0 0x7f1da8073760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39920/0 (socket says 172.21.15.40:39920) 2024-04-26T23:08:25.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.008+0000 7f1da7e00640 1 -- 172.21.15.40:0/2920621204 learned_addr learned my addr 172.21.15.40:0/2920621204 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:25.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.008+0000 7f1da7400640 1 --2- 172.21.15.40:0/2920621204 >> v2:172.21.15.40:3300/0 conn(0x7f1da815ce60 0x7f1da80afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:25.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.008+0000 7f1d9fe00640 1 --2- 172.21.15.40:0/2920621204 >> v2:172.21.15.103:3300/0 conn(0x7f1da80af7a0 0x7f1da80ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:25.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.008+0000 7f1da6a00640 1 -- 172.21.15.40:0/2920621204 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3065046963 0 0) 0x7f1da80d0820 con 0x7f1da8071fe0 2024-04-26T23:08:25.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.008+0000 7f1da6a00640 1 -- 172.21.15.40:0/2920621204 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d8c003520 con 0x7f1da8071fe0 2024-04-26T23:08:25.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.008+0000 7f1da7400640 1 -- 172.21.15.40:0/2920621204 >> v2:172.21.15.103:3300/0 conn(0x7f1da80af7a0 msgr2=0x7f1da80ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:25.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.008+0000 7f1da7400640 1 --2- 172.21.15.40:0/2920621204 >> v2:172.21.15.103:3300/0 conn(0x7f1da80af7a0 0x7f1da80ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.008+0000 7f1da7400640 1 -- 172.21.15.40:0/2920621204 >> v1:172.21.15.40:6789/0 conn(0x7f1da8071fe0 legacy=0x7f1da8073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:25.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.008+0000 7f1da7400640 1 -- 172.21.15.40:0/2920621204 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da8161760 con 0x7f1da815ce60 2024-04-26T23:08:25.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.009+0000 7f1da7400640 1 --2- 172.21.15.40:0/2920621204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da815ce60 0x7f1da80afc60 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f1da80d1940 tx=0x7f1d9802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f050e7c6de019d2a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:25.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.010+0000 7f1da6a00640 1 -- 172.21.15.40:0/2920621204 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98037070 con 0x7f1da815ce60 2024-04-26T23:08:25.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.010+0000 7f1da6a00640 1 -- 172.21.15.40:0/2920621204 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d98032440 con 0x7f1da815ce60 2024-04-26T23:08:25.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.010+0000 7f1da6a00640 1 -- 172.21.15.40:0/2920621204 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9802ae60 con 0x7f1da815ce60 2024-04-26T23:08:25.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.010+0000 7f1dace00640 1 -- 172.21.15.40:0/2920621204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da815ce60 msgr2=0x7f1da80afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:25.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.010+0000 7f1dace00640 1 --2- 172.21.15.40:0/2920621204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da815ce60 0x7f1da80afc60 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f1da80d1940 tx=0x7f1d9802a110 comp rx=0 tx=0).stop 2024-04-26T23:08:25.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.010+0000 7f1dace00640 1 -- 172.21.15.40:0/2920621204 shutdown_connections 2024-04-26T23:08:25.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.010+0000 7f1dace00640 1 --2- 172.21.15.40:0/2920621204 >> v2:172.21.15.103:3300/0 conn(0x7f1da80af7a0 0x7f1da80ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.010+0000 7f1dace00640 1 --2- 172.21.15.40:0/2920621204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da815ce60 0x7f1da80afc60 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.010+0000 7f1dace00640 1 -- 172.21.15.40:0/2920621204 >> 172.21.15.40:0/2920621204 conn(0x7f1da80d0a10 msgr2=0x7f1da80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:25.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.011+0000 7f1dace00640 1 -- 172.21.15.40:0/2920621204 shutdown_connections 2024-04-26T23:08:25.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.011+0000 7f1dace00640 1 -- 172.21.15.40:0/2920621204 wait complete. 2024-04-26T23:08:25.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.012+0000 7f1dace00640 1 Processor -- start 2024-04-26T23:08:25.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.012+0000 7f1dace00640 1 -- start start 2024-04-26T23:08:25.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.013+0000 7f1dace00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da8071fe0 0x7f1da8153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:25.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.013+0000 7f1dace00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1da80af7a0 0x7f1da815dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:25.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.013+0000 7f1dace00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da815ce60 0x7f1da81f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:25.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.013+0000 7f1dace00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1da8073e70 con 0x7f1da8071fe0 2024-04-26T23:08:25.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.013+0000 7f1dace00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1da80cf530 con 0x7f1da815ce60 2024-04-26T23:08:25.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.013+0000 7f1dace00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1da8073ff0 con 0x7f1da80af7a0 2024-04-26T23:08:25.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.014+0000 7f1da7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da8071fe0 0x7f1da8153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:25.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.014+0000 7f1da7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da8071fe0 0x7f1da8153880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52814/0 (socket says 172.21.15.40:52814) 2024-04-26T23:08:25.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.014+0000 7f1da7400640 1 -- 172.21.15.40:0/1698234367 learned_addr learned my addr 172.21.15.40:0/1698234367 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:25.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.014+0000 7f1da7400640 1 -- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1da80af7a0 msgr2=0x7f1da815dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:25.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.014+0000 7f1da7400640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1da80af7a0 0x7f1da815dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.014+0000 7f1da7400640 1 -- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da815ce60 msgr2=0x7f1da81f5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:25.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.014+0000 7f1da7e00640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da815ce60 0x7f1da81f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:25.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.014+0000 7f1da7400640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da815ce60 0x7f1da81f5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.014+0000 7f1da7400640 1 -- 172.21.15.40:0/1698234367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da80c7e50 con 0x7f1da8071fe0 2024-04-26T23:08:25.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.014+0000 7f1da7e00640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da815ce60 0x7f1da81f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:25.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.015+0000 7f1da7400640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da8071fe0 0x7f1da8153880 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f1d98000c00 tx=0x7f1d980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:25.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.015+0000 7f1d9f400640 1 -- 172.21.15.40:0/1698234367 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98037040 con 0x7f1da8071fe0 2024-04-26T23:08:25.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.015+0000 7f1d9f400640 1 -- 172.21.15.40:0/1698234367 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d98042070 con 0x7f1da8071fe0 2024-04-26T23:08:25.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.015+0000 7f1d9f400640 1 -- 172.21.15.40:0/1698234367 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98030920 con 0x7f1da8071fe0 2024-04-26T23:08:25.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.015+0000 7f1dace00640 1 -- 172.21.15.40:0/1698234367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da80c8130 con 0x7f1da8071fe0 2024-04-26T23:08:25.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.015+0000 7f1dace00640 1 -- 172.21.15.40:0/1698234367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da80c86f0 con 0x7f1da8071fe0 2024-04-26T23:08:25.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.016+0000 7f1dace00640 1 -- 172.21.15.40:0/1698234367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da8069370 con 0x7f1da8071fe0 2024-04-26T23:08:25.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.017+0000 7f1d9f400640 1 -- 172.21.15.40:0/1698234367 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1d98030ac0 con 0x7f1da8071fe0 2024-04-26T23:08:25.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.019+0000 7f1d9f400640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1d8406c8f0 0x7f1d8406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:25.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.019+0000 7f1d9f400640 1 -- 172.21.15.40:0/1698234367 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f1d980ac720 con 0x7f1da8071fe0 2024-04-26T23:08:25.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.019+0000 7f1d9fe00640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1d8406c8f0 0x7f1d8406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:25.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.021+0000 7f1d9fe00640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1d8406c8f0 0x7f1d8406edb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1da815c4d0 tx=0x7f1d90008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:25.023 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:08:25.023+0000 7f36ffc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-26T23:08:25.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.023+0000 7f1d9f400640 1 -- 172.21.15.40:0/1698234367 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1d9807a2d0 con 0x7f1da8071fe0 2024-04-26T23:08:25.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.287+0000 7f1dace00640 1 -- 172.21.15.40:0/1698234367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1da80af000 con 0x7f1da8071fe0 2024-04-26T23:08:25.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.288+0000 7f1d9f400640 1 -- 172.21.15.40:0/1698234367 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5230 (secure 0 0 0) 0x7f1d9807e5d0 con 0x7f1da8071fe0 2024-04-26T23:08:25.289 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:25.289 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":100,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:24.848549+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:25.289 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 100 2024-04-26T23:08:25.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.292+0000 7f1dace00640 1 -- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1d8406c8f0 msgr2=0x7f1d8406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:25.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.292+0000 7f1dace00640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1d8406c8f0 0x7f1d8406edb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1da815c4d0 tx=0x7f1d90008040 comp rx=0 tx=0).stop 2024-04-26T23:08:25.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.292+0000 7f1dace00640 1 -- 172.21.15.40:0/1698234367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da8071fe0 msgr2=0x7f1da8153880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:25.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.292+0000 7f1dace00640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da8071fe0 0x7f1da8153880 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f1d98000c00 tx=0x7f1d980046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.292+0000 7f1dace00640 1 -- 172.21.15.40:0/1698234367 shutdown_connections 2024-04-26T23:08:25.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.292+0000 7f1dace00640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1d8406c8f0 0x7f1d8406edb0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.292+0000 7f1dace00640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da815ce60 0x7f1da81f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.292+0000 7f1dace00640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1da80af7a0 0x7f1da815dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.292+0000 7f1dace00640 1 --2- 172.21.15.40:0/1698234367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da8071fe0 0x7f1da8153880 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.292+0000 7f1dace00640 1 -- 172.21.15.40:0/1698234367 >> 172.21.15.40:0/1698234367 conn(0x7f1da80d0a10 msgr2=0x7f1da81574b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:25.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.293+0000 7f1dace00640 1 -- 172.21.15.40:0/1698234367 shutdown_connections 2024-04-26T23:08:25.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.293+0000 7f1dace00640 1 -- 172.21.15.40:0/1698234367 wait complete. 2024-04-26T23:08:25.312 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'd'] 2024-04-26T23:08:25.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:08:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.511+0000 7fe264e00640 1 Processor -- start 2024-04-26T23:08:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.511+0000 7fe264e00640 1 -- start start 2024-04-26T23:08:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.512+0000 7fe264e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2601605d0 0x7fe260071c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.512+0000 7fe264e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2600706f0 0x7fe2600b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.512+0000 7fe264e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe260071900 con 0x7fe2600d1890 2024-04-26T23:08:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.512+0000 7fe264e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe260071340 con 0x7fe2601605d0 2024-04-26T23:08:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.512+0000 7fe264e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe260071620 con 0x7fe2600706f0 2024-04-26T23:08:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.512+0000 7fe25f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2601605d0 0x7fe260071c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.512+0000 7fe256a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe2600d1890 0x7fe26006ffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39924/0 (socket says 172.21.15.40:39924) 2024-04-26T23:08:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.512+0000 7fe256a00640 1 -- 172.21.15.40:0/3648295754 learned_addr learned my addr 172.21.15.40:0/3648295754 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.513+0000 7fe25fe00640 1 --2- 172.21.15.40:0/3648295754 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2600706f0 0x7fe2600b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.513+0000 7fe25f400640 1 -- 172.21.15.40:0/3648295754 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2600706f0 msgr2=0x7fe2600b13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.513+0000 7fe25f400640 1 --2- 172.21.15.40:0/3648295754 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2600706f0 0x7fe2600b13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.513+0000 7fe25f400640 1 -- 172.21.15.40:0/3648295754 >> v1:172.21.15.40:6789/0 conn(0x7fe2600d1890 legacy=0x7fe26006ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.513+0000 7fe25f400640 1 -- 172.21.15.40:0/3648295754 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2600d1370 con 0x7fe2601605d0 2024-04-26T23:08:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.513+0000 7fe25ea00640 1 -- 172.21.15.40:0/3648295754 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4052859868 0 0) 0x7fe260071900 con 0x7fe2600d1890 2024-04-26T23:08:25.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.514+0000 7fe25f400640 1 --2- 172.21.15.40:0/3648295754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2601605d0 0x7fe260071c40 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fe2500097a0 tx=0x7fe25002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ccb6d6bebcd4b00 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:25.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.514+0000 7fe25ea00640 1 -- 172.21.15.40:0/3648295754 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe250037070 con 0x7fe2601605d0 2024-04-26T23:08:25.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.514+0000 7fe25ea00640 1 -- 172.21.15.40:0/3648295754 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe250032440 con 0x7fe2601605d0 2024-04-26T23:08:25.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.514+0000 7fe25ea00640 1 -- 172.21.15.40:0/3648295754 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe250032810 con 0x7fe2601605d0 2024-04-26T23:08:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.515+0000 7fe264e00640 1 -- 172.21.15.40:0/3648295754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2601605d0 msgr2=0x7fe260071c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.515+0000 7fe264e00640 1 --2- 172.21.15.40:0/3648295754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2601605d0 0x7fe260071c40 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fe2500097a0 tx=0x7fe25002a180 comp rx=0 tx=0).stop 2024-04-26T23:08:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.515+0000 7fe264e00640 1 -- 172.21.15.40:0/3648295754 shutdown_connections 2024-04-26T23:08:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.515+0000 7fe264e00640 1 --2- 172.21.15.40:0/3648295754 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2600706f0 0x7fe2600b13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.515+0000 7fe264e00640 1 --2- 172.21.15.40:0/3648295754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2601605d0 0x7fe260071c40 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.515+0000 7fe264e00640 1 -- 172.21.15.40:0/3648295754 >> 172.21.15.40:0/3648295754 conn(0x7fe2600d2c60 msgr2=0x7fe2600d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.515+0000 7fe264e00640 1 -- 172.21.15.40:0/3648295754 shutdown_connections 2024-04-26T23:08:25.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.515+0000 7fe264e00640 1 -- 172.21.15.40:0/3648295754 wait complete. 2024-04-26T23:08:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.517+0000 7fe264e00640 1 Processor -- start 2024-04-26T23:08:25.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.517+0000 7fe264e00640 1 -- start start 2024-04-26T23:08:25.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.518+0000 7fe264e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2600706f0 0x7fe2600c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:25.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.518+0000 7fe264e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2601605d0 0x7fe260147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:25.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.518+0000 7fe264e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2600b7ed0 0x7fe2600cf900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:25.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.518+0000 7fe264e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe260070ce0 con 0x7fe2600b7ed0 2024-04-26T23:08:25.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.518+0000 7fe264e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe260070b60 con 0x7fe2601605d0 2024-04-26T23:08:25.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.518+0000 7fe264e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2600b1930 con 0x7fe2600706f0 2024-04-26T23:08:25.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.518+0000 7fe25fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2600b7ed0 0x7fe2600cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:25.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.518+0000 7fe25fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2600b7ed0 0x7fe2600cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52840/0 (socket says 172.21.15.40:52840) 2024-04-26T23:08:25.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.518+0000 7fe25fe00640 1 -- 172.21.15.40:0/3000304639 learned_addr learned my addr 172.21.15.40:0/3000304639 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:25.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.518+0000 7fe25f400640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2600706f0 0x7fe2600c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:25.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.519+0000 7fe25fe00640 1 -- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2600706f0 msgr2=0x7fe2600c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:25.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.519+0000 7fe25fe00640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2600706f0 0x7fe2600c92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.519+0000 7fe256a00640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2601605d0 0x7fe260147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:25.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.519+0000 7fe25fe00640 1 -- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2601605d0 msgr2=0x7fe260147840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:25.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.519+0000 7fe25fe00640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2601605d0 0x7fe260147840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.519+0000 7fe25fe00640 1 -- 172.21.15.40:0/3000304639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe260147fe0 con 0x7fe2600b7ed0 2024-04-26T23:08:25.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.519+0000 7fe256a00640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2601605d0 0x7fe260147840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:25.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.519+0000 7fe25fe00640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2600b7ed0 0x7fe2600cf900 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fe25800cd70 tx=0x7fe25800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:25.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.519+0000 7fe257e00640 1 -- 172.21.15.40:0/3000304639 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258003b90 con 0x7fe2600b7ed0 2024-04-26T23:08:25.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.519+0000 7fe257e00640 1 -- 172.21.15.40:0/3000304639 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe258003d30 con 0x7fe2600b7ed0 2024-04-26T23:08:25.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.519+0000 7fe257e00640 1 -- 172.21.15.40:0/3000304639 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258004240 con 0x7fe2600b7ed0 2024-04-26T23:08:25.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.519+0000 7fe264e00640 1 -- 172.21.15.40:0/3000304639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2601482c0 con 0x7fe2600b7ed0 2024-04-26T23:08:25.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.520+0000 7fe264e00640 1 -- 172.21.15.40:0/3000304639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe260148880 con 0x7fe2600b7ed0 2024-04-26T23:08:25.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.521+0000 7fe264e00640 1 -- 172.21.15.40:0/3000304639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe260069a80 con 0x7fe2600b7ed0 2024-04-26T23:08:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.523+0000 7fe257e00640 1 -- 172.21.15.40:0/3000304639 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2580043e0 con 0x7fe2600b7ed0 2024-04-26T23:08:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.524+0000 7fe257e00640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe23c06c8f0 0x7fe23c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.524+0000 7fe257e00640 1 -- 172.21.15.40:0/3000304639 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fe25808c830 con 0x7fe2600b7ed0 2024-04-26T23:08:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.524+0000 7fe25f400640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe23c06c8f0 0x7fe23c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:25.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.526+0000 7fe25f400640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe23c06c8f0 0x7fe23c06edb0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe250034000 tx=0x7fe25002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.528+0000 7fe257e00640 1 -- 172.21.15.40:0/3000304639 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe25805aa30 con 0x7fe2600b7ed0 2024-04-26T23:08:25.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.804+0000 7fe264e00640 1 -- 172.21.15.40:0/3000304639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fe2600cd770 con 0x7fe2600b7ed0 2024-04-26T23:08:25.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.805+0000 7fe257e00640 1 -- 172.21.15.40:0/3000304639 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1458 (secure 0 0 0) 0x7fe25805a330 con 0x7fe2600b7ed0 2024-04-26T23:08:25.805 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:25.805 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":465,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":90,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":700759,"bytes_used":265830400,"bytes_avail":772828282880,"bytes_total":773094113280,"read_bytes_sec":0,"write_bytes_sec":10150,"read_op_per_sec":0,"write_op_per_sec":8},"fsmap":{"epoch":100,"id":6,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":6,"rank":2,"name":"d","status":"up:active","gid":6253},{"filesystem_id":6,"rank":1,"name":"a","status":"up:active","gid":8031},{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8040}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-26T23:07:35.290664+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T23:08:25.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.808+0000 7fe264e00640 1 -- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe23c06c8f0 msgr2=0x7fe23c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:25.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.808+0000 7fe264e00640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe23c06c8f0 0x7fe23c06edb0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe250034000 tx=0x7fe25002e000 comp rx=0 tx=0).stop 2024-04-26T23:08:25.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.808+0000 7fe264e00640 1 -- 172.21.15.40:0/3000304639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2600b7ed0 msgr2=0x7fe2600cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:25.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.808+0000 7fe264e00640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2600b7ed0 0x7fe2600cf900 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fe25800cd70 tx=0x7fe25800a660 comp rx=0 tx=0).stop 2024-04-26T23:08:25.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.808+0000 7fe264e00640 1 -- 172.21.15.40:0/3000304639 shutdown_connections 2024-04-26T23:08:25.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.808+0000 7fe264e00640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe23c06c8f0 0x7fe23c06edb0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.808+0000 7fe264e00640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2600b7ed0 0x7fe2600cf900 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.808+0000 7fe264e00640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2601605d0 0x7fe260147840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.808+0000 7fe264e00640 1 --2- 172.21.15.40:0/3000304639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2600706f0 0x7fe2600c92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:25.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.808+0000 7fe264e00640 1 -- 172.21.15.40:0/3000304639 >> 172.21.15.40:0/3000304639 conn(0x7fe2600d2c60 msgr2=0x7fe2600b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:25.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.808+0000 7fe264e00640 1 -- 172.21.15.40:0/3000304639 shutdown_connections 2024-04-26T23:08:25.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:25.808+0000 7fe264e00640 1 -- 172.21.15.40:0/3000304639 wait complete. 2024-04-26T23:08:25.824 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-26T23:08:26.825 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.027+0000 7fb22dc00640 1 Processor -- start 2024-04-26T23:08:27.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.027+0000 7fb22dc00640 1 -- start start 2024-04-26T23:08:27.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.029+0000 7fb22dc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb2280b6220 0x7fb2280d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.029+0000 7fb22dc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2280d08d0 0x7fb2280d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.029+0000 7fb22dc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb228162c40 con 0x7fb2280d3820 2024-04-26T23:08:27.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.029+0000 7fb22dc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb2280d3c60 con 0x7fb2280d08d0 2024-04-26T23:08:27.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.029+0000 7fb22dc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb2280d2d80 con 0x7fb2280b6220 2024-04-26T23:08:27.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.029+0000 7fb227400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2280d08d0 0x7fb2280d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:27.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.029+0000 7fb227400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2280d08d0 0x7fb2280d32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52846/0 (socket says 172.21.15.40:52846) 2024-04-26T23:08:27.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.029+0000 7fb227400640 1 -- 172.21.15.40:0/4254318735 learned_addr learned my addr 172.21.15.40:0/4254318735 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:27.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.030+0000 7fb227400640 1 -- 172.21.15.40:0/4254318735 >> v2:172.21.15.103:3300/0 conn(0x7fb2280b6220 msgr2=0x7fb2280d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.030+0000 7fb227e00640 1 --2- 172.21.15.40:0/4254318735 >> v2:172.21.15.103:3300/0 conn(0x7fb2280b6220 0x7fb2280d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.030+0000 7fb227400640 1 --2- 172.21.15.40:0/4254318735 >> v2:172.21.15.103:3300/0 conn(0x7fb2280b6220 0x7fb2280d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.030+0000 7fb227400640 1 -- 172.21.15.40:0/4254318735 >> v1:172.21.15.103:6789/0 conn(0x7fb2280d3820 legacy=0x7fb2280b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.030+0000 7fb227400640 1 -- 172.21.15.40:0/4254318735 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2280b7c80 con 0x7fb2280d08d0 2024-04-26T23:08:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.030+0000 7fb227e00640 1 --2- 172.21.15.40:0/4254318735 >> v2:172.21.15.103:3300/0 conn(0x7fb2280b6220 0x7fb2280d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.031+0000 7fb227400640 1 --2- 172.21.15.40:0/4254318735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2280d08d0 0x7fb2280d32e0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fb2280d4cf0 tx=0x7fb21402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e4d1b5b57519bf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:27.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.031+0000 7fb226a00640 1 -- 172.21.15.40:0/4254318735 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb214037070 con 0x7fb2280d08d0 2024-04-26T23:08:27.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.031+0000 7fb226a00640 1 -- 172.21.15.40:0/4254318735 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb214032440 con 0x7fb2280d08d0 2024-04-26T23:08:27.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.031+0000 7fb226a00640 1 -- 172.21.15.40:0/4254318735 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb214032810 con 0x7fb2280d08d0 2024-04-26T23:08:27.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.032+0000 7fb22dc00640 1 -- 172.21.15.40:0/4254318735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2280d08d0 msgr2=0x7fb2280d32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:27.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.032+0000 7fb22dc00640 1 --2- 172.21.15.40:0/4254318735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2280d08d0 0x7fb2280d32e0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fb2280d4cf0 tx=0x7fb21402a230 comp rx=0 tx=0).stop 2024-04-26T23:08:27.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.032+0000 7fb22dc00640 1 -- 172.21.15.40:0/4254318735 shutdown_connections 2024-04-26T23:08:27.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.032+0000 7fb22dc00640 1 --2- 172.21.15.40:0/4254318735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2280d08d0 0x7fb2280d32e0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.032+0000 7fb22dc00640 1 --2- 172.21.15.40:0/4254318735 >> v2:172.21.15.103:3300/0 conn(0x7fb2280b6220 0x7fb2280d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.032+0000 7fb22dc00640 1 -- 172.21.15.40:0/4254318735 >> 172.21.15.40:0/4254318735 conn(0x7fb2280d4d90 msgr2=0x7fb2280d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:27.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.032+0000 7fb22dc00640 1 -- 172.21.15.40:0/4254318735 shutdown_connections 2024-04-26T23:08:27.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.032+0000 7fb22dc00640 1 -- 172.21.15.40:0/4254318735 wait complete. 2024-04-26T23:08:27.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.034+0000 7fb22dc00640 1 Processor -- start 2024-04-26T23:08:27.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.034+0000 7fb22dc00640 1 -- start start 2024-04-26T23:08:27.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.035+0000 7fb22dc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2280b6220 0x7fb2280b5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.035+0000 7fb22dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2280d08d0 0x7fb2280ce050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.036+0000 7fb22dc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2280d3820 0x7fb2280a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.036+0000 7fb22dc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2280b8520 con 0x7fb2280d08d0 2024-04-26T23:08:27.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.036+0000 7fb22dc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2280b8240 con 0x7fb2280d3820 2024-04-26T23:08:27.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.036+0000 7fb22dc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb2280b9d30 con 0x7fb2280b6220 2024-04-26T23:08:27.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.036+0000 7fb227400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2280d08d0 0x7fb2280ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:27.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.036+0000 7fb227400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2280d08d0 0x7fb2280ce050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52862/0 (socket says 172.21.15.40:52862) 2024-04-26T23:08:27.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.036+0000 7fb227400640 1 -- 172.21.15.40:0/2985766835 learned_addr learned my addr 172.21.15.40:0/2985766835 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:27.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.036+0000 7fb227e00640 1 --2- 172.21.15.40:0/2985766835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2280b6220 0x7fb2280b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:27.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.036+0000 7fb227400640 1 -- 172.21.15.40:0/2985766835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2280b6220 msgr2=0x7fb2280b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:27.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.036+0000 7fb227400640 1 --2- 172.21.15.40:0/2985766835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2280b6220 0x7fb2280b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.036+0000 7fb227400640 1 -- 172.21.15.40:0/2985766835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2280d3820 msgr2=0x7fb2280a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:27.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.036+0000 7fb227400640 1 --2- 172.21.15.40:0/2985766835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2280d3820 0x7fb2280a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.036+0000 7fb227400640 1 -- 172.21.15.40:0/2985766835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2280a1ec0 con 0x7fb2280d08d0 2024-04-26T23:08:27.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.037+0000 7fb227400640 1 --2- 172.21.15.40:0/2985766835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2280d08d0 0x7fb2280ce050 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fb214000c00 tx=0x7fb2140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:27.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.037+0000 7fb21fe00640 1 -- 172.21.15.40:0/2985766835 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb214041020 con 0x7fb2280d08d0 2024-04-26T23:08:27.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.037+0000 7fb21fe00640 1 -- 172.21.15.40:0/2985766835 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb214030970 con 0x7fb2280d08d0 2024-04-26T23:08:27.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.037+0000 7fb21fe00640 1 -- 172.21.15.40:0/2985766835 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb214037040 con 0x7fb2280d08d0 2024-04-26T23:08:27.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.037+0000 7fb22dc00640 1 -- 172.21.15.40:0/2985766835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2280a21a0 con 0x7fb2280d08d0 2024-04-26T23:08:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.037+0000 7fb22dc00640 1 -- 172.21.15.40:0/2985766835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2280a2760 con 0x7fb2280d08d0 2024-04-26T23:08:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.038+0000 7fb22dc00640 1 -- 172.21.15.40:0/2985766835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb228069370 con 0x7fb2280d08d0 2024-04-26T23:08:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.039+0000 7fb21fe00640 1 -- 172.21.15.40:0/2985766835 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb214044020 con 0x7fb2280d08d0 2024-04-26T23:08:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.040+0000 7fb21fe00640 1 --2- 172.21.15.40:0/2985766835 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb20806c940 0x7fb20806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.040+0000 7fb21fe00640 1 -- 172.21.15.40:0/2985766835 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fb2140ac630 con 0x7fb2280d08d0 2024-04-26T23:08:27.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.041+0000 7fb227e00640 1 --2- 172.21.15.40:0/2985766835 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb20806c940 0x7fb20806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:27.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.042+0000 7fb227e00640 1 --2- 172.21.15.40:0/2985766835 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb20806c940 0x7fb20806ee00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb218008850 tx=0x7fb218008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:27.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.045+0000 7fb21fe00640 1 -- 172.21.15.40:0/2985766835 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fb21404b030 con 0x7fb2280d08d0 2024-04-26T23:08:27.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.288+0000 7fb22dc00640 1 -- 172.21.15.40:0/2985766835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb228074360 con 0x7fb2280d08d0 2024-04-26T23:08:27.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.289+0000 7fb21fe00640 1 -- 172.21.15.40:0/2985766835 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5232 (secure 0 0 0) 0x7fb21407a9b0 con 0x7fb2280d08d0 2024-04-26T23:08:27.290 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:27.290 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:26.911596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:27.290 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-04-26T23:08:27.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.292+0000 7fb22dc00640 1 -- 172.21.15.40:0/2985766835 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb20806c940 msgr2=0x7fb20806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:27.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.292+0000 7fb22dc00640 1 --2- 172.21.15.40:0/2985766835 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb20806c940 0x7fb20806ee00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb218008850 tx=0x7fb218008040 comp rx=0 tx=0).stop 2024-04-26T23:08:27.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.292+0000 7fb22dc00640 1 -- 172.21.15.40:0/2985766835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2280d08d0 msgr2=0x7fb2280ce050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:27.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.292+0000 7fb22dc00640 1 --2- 172.21.15.40:0/2985766835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2280d08d0 0x7fb2280ce050 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fb214000c00 tx=0x7fb2140046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.292+0000 7fb22dc00640 1 -- 172.21.15.40:0/2985766835 shutdown_connections 2024-04-26T23:08:27.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.292+0000 7fb22dc00640 1 --2- 172.21.15.40:0/2985766835 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb20806c940 0x7fb20806ee00 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.292+0000 7fb22dc00640 1 --2- 172.21.15.40:0/2985766835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2280d3820 0x7fb2280a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.292+0000 7fb22dc00640 1 --2- 172.21.15.40:0/2985766835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2280d08d0 0x7fb2280ce050 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.292+0000 7fb22dc00640 1 --2- 172.21.15.40:0/2985766835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb2280b6220 0x7fb2280b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.292+0000 7fb22dc00640 1 -- 172.21.15.40:0/2985766835 >> 172.21.15.40:0/2985766835 conn(0x7fb2280d4d90 msgr2=0x7fb22806e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:27.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.292+0000 7fb22dc00640 1 -- 172.21.15.40:0/2985766835 shutdown_connections 2024-04-26T23:08:27.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.292+0000 7fb22dc00640 1 -- 172.21.15.40:0/2985766835 wait complete. 2024-04-26T23:08:27.312 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'd'] 2024-04-26T23:08:27.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:08:27.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.507+0000 7f5b4ca00640 1 Processor -- start 2024-04-26T23:08:27.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.507+0000 7f5b4ca00640 1 -- start start 2024-04-26T23:08:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.509+0000 7f5b4ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b48146180 0x7f5b4806cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.509+0000 7f5b4ca00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b4815d340 con 0x7f5b480b7280 2024-04-26T23:08:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.509+0000 7f5b4ca00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b480cfd30 con 0x7f5b480b7660 2024-04-26T23:08:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.509+0000 7f5b4ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5b48146930 con 0x7f5b48146180 2024-04-26T23:08:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.509+0000 7f5b47400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5b480b7280 0x7f5b48145a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39934/0 (socket says 172.21.15.40:39934) 2024-04-26T23:08:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.509+0000 7f5b47400640 1 -- 172.21.15.40:0/1149279538 learned_addr learned my addr 172.21.15.40:0/1149279538 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46a00640 1 --2- 172.21.15.40:0/1149279538 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b48146180 0x7f5b4806cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 799802717 0 0) 0x7f5b4815d340 con 0x7f5b480b7280 2024-04-26T23:08:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b2c003520 con 0x7f5b480b7280 2024-04-26T23:08:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1824182119 0 0) 0x7f5b2c003520 con 0x7f5b480b7280 2024-04-26T23:08:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b48146180 msgr2=0x7f5b4806cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 --2- 172.21.15.40:0/1149279538 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b48146180 0x7f5b4806cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 >> v1:172.21.15.103:6789/0 conn(0x7f5b480b7660 legacy=0x7f5b48160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b480d19b0 con 0x7f5b480b7280 2024-04-26T23:08:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f5b38001670 con 0x7f5b480b7280 2024-04-26T23:08:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 >> v1:172.21.15.40:6789/0 conn(0x7f5b480b7280 legacy=0x7f5b48145a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 --2- 172.21.15.40:0/1149279538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b2c0039e0 0x7f5b2c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 --2- 172.21.15.40:0/1149279538 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b48146180 0x7f5b2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 --2- 172.21.15.40:0/1149279538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b2c004d40 0x7f5b2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c00bba0 con 0x7f5b2c0039e0 2024-04-26T23:08:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c00bd20 con 0x7f5b2c004d40 2024-04-26T23:08:27.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.510+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5b2c00bf20 con 0x7f5b48146180 2024-04-26T23:08:27.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.511+0000 7f5b47400640 1 --2- 172.21.15.40:0/1149279538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b2c0039e0 0x7f5b2c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:27.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.511+0000 7f5b47e00640 1 --2- 172.21.15.40:0/1149279538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b2c004d40 0x7f5b2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:27.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.511+0000 7f5b47400640 1 -- 172.21.15.40:0/1149279538 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b48146180 msgr2=0x7f5b2c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:27.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.511+0000 7f5b47400640 1 --2- 172.21.15.40:0/1149279538 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b48146180 0x7f5b2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.511+0000 7f5b47400640 1 -- 172.21.15.40:0/1149279538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b2c004d40 msgr2=0x7f5b2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:27.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.511+0000 7f5b47400640 1 --2- 172.21.15.40:0/1149279538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b2c004d40 0x7f5b2c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.511+0000 7f5b47400640 1 -- 172.21.15.40:0/1149279538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2c003520 con 0x7f5b2c0039e0 2024-04-26T23:08:27.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.511+0000 7f5b47e00640 1 --2- 172.21.15.40:0/1149279538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b2c004d40 0x7f5b2c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:27.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.512+0000 7f5b47400640 1 --2- 172.21.15.40:0/1149279538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b2c0039e0 0x7f5b2c003e00 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800c190 tx=0x7f5b2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d1e4a056adb41f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:27.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.512+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38007c60 con 0x7f5b2c0039e0 2024-04-26T23:08:27.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.512+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b38005ad0 con 0x7f5b2c0039e0 2024-04-26T23:08:27.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.512+0000 7f5b46000640 1 -- 172.21.15.40:0/1149279538 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38007070 con 0x7f5b2c0039e0 2024-04-26T23:08:27.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.513+0000 7f5b4ca00640 1 -- 172.21.15.40:0/1149279538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b2c0039e0 msgr2=0x7f5b2c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:27.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.513+0000 7f5b4ca00640 1 --2- 172.21.15.40:0/1149279538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b2c0039e0 0x7f5b2c003e00 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800c190 tx=0x7f5b2c005830 comp rx=0 tx=0).stop 2024-04-26T23:08:27.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.513+0000 7f5b47e00640 1 -- 172.21.15.40:0/1149279538 reap_dead start 2024-04-26T23:08:27.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.513+0000 7f5b4ca00640 1 -- 172.21.15.40:0/1149279538 shutdown_connections 2024-04-26T23:08:27.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.513+0000 7f5b4ca00640 1 -- 172.21.15.40:0/1149279538 >> 172.21.15.40:0/1149279538 conn(0x7f5b48072ee0 msgr2=0x7f5b480732c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:27.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.513+0000 7f5b4ca00640 1 -- 172.21.15.40:0/1149279538 shutdown_connections 2024-04-26T23:08:27.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.513+0000 7f5b4ca00640 1 -- 172.21.15.40:0/1149279538 wait complete. 2024-04-26T23:08:27.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.515+0000 7f5b4ca00640 1 Processor -- start 2024-04-26T23:08:27.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.515+0000 7f5b4ca00640 1 -- start start 2024-04-26T23:08:27.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.516+0000 7f5b4ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b2c0039e0 0x7f5b480b58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.516+0000 7f5b4ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b481f48e0 0x7f5b4815bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.516+0000 7f5b4ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b481f5a60 0x7f5b48144200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.516+0000 7f5b4ca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b480cf250 con 0x7f5b481f48e0 2024-04-26T23:08:27.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.516+0000 7f5b4ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b480cf8a0 con 0x7f5b2c0039e0 2024-04-26T23:08:27.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.516+0000 7f5b4ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5b480cf530 con 0x7f5b481f5a60 2024-04-26T23:08:27.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.516+0000 7f5b46a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b481f48e0 0x7f5b4815bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:27.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.516+0000 7f5b46a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b481f48e0 0x7f5b4815bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52872/0 (socket says 172.21.15.40:52872) 2024-04-26T23:08:27.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.516+0000 7f5b46a00640 1 -- 172.21.15.40:0/3025932640 learned_addr learned my addr 172.21.15.40:0/3025932640 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:27.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.516+0000 7f5b47400640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b2c0039e0 0x7f5b480b58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:27.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.517+0000 7f5b47e00640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b481f5a60 0x7f5b48144200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:27.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.517+0000 7f5b46a00640 1 -- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b481f5a60 msgr2=0x7f5b48144200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:27.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.517+0000 7f5b46a00640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b481f5a60 0x7f5b48144200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.517+0000 7f5b46a00640 1 -- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b2c0039e0 msgr2=0x7f5b480b58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:27.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.517+0000 7f5b46a00640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b2c0039e0 0x7f5b480b58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.517+0000 7f5b46a00640 1 -- 172.21.15.40:0/3025932640 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4815c290 con 0x7f5b481f48e0 2024-04-26T23:08:27.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.517+0000 7f5b46a00640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b481f48e0 0x7f5b4815bd50 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400d5f0 tx=0x7f5b3400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:27.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.517+0000 7f5b3f400640 1 -- 172.21.15.40:0/3025932640 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b340125f0 con 0x7f5b481f48e0 2024-04-26T23:08:27.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.517+0000 7f5b3f400640 1 -- 172.21.15.40:0/3025932640 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b34019070 con 0x7f5b481f48e0 2024-04-26T23:08:27.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.517+0000 7f5b3f400640 1 -- 172.21.15.40:0/3025932640 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b340129d0 con 0x7f5b481f48e0 2024-04-26T23:08:27.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.517+0000 7f5b4ca00640 1 -- 172.21.15.40:0/3025932640 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b481f50a0 con 0x7f5b481f48e0 2024-04-26T23:08:27.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.518+0000 7f5b4ca00640 1 -- 172.21.15.40:0/3025932640 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b481f5660 con 0x7f5b481f48e0 2024-04-26T23:08:27.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.519+0000 7f5b4ca00640 1 -- 172.21.15.40:0/3025932640 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b481f5860 con 0x7f5b481f48e0 2024-04-26T23:08:27.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.519+0000 7f5b3f400640 1 -- 172.21.15.40:0/3025932640 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5b34012b70 con 0x7f5b481f48e0 2024-04-26T23:08:27.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.520+0000 7f5b3f400640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5b2806c920 0x7f5b2806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:27.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.520+0000 7f5b3f400640 1 -- 172.21.15.40:0/3025932640 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f5b3408c070 con 0x7f5b481f48e0 2024-04-26T23:08:27.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.521+0000 7f5b47400640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5b2806c920 0x7f5b2806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:27.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.522+0000 7f5b47400640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5b2806c920 0x7f5b2806ede0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800c160 tx=0x7f5b38009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:27.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.525+0000 7f5b3f400640 1 -- 172.21.15.40:0/3025932640 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5b481f5860 con 0x7f5b481f48e0 2024-04-26T23:08:27.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.798+0000 7f5b4ca00640 1 -- 172.21.15.40:0/3025932640 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f5b4815f400 con 0x7f5b481f48e0 2024-04-26T23:08:27.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.799+0000 7f5b3f400640 1 -- 172.21.15.40:0/3025932640 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1967 (secure 0 0 0) 0x7f5b4815c290 con 0x7f5b481f48e0 2024-04-26T23:08:27.800 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:27.800 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":467,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":90,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":700759,"bytes_used":265830400,"bytes_avail":772828282880,"bytes_total":773094113280,"read_bytes_sec":0,"write_bytes_sec":6738,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":102,"id":6,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":6,"rank":2,"name":"d","status":"up:active","gid":6253},{"filesystem_id":6,"rank":1,"name":"a","status":"up:active","gid":8031},{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8040}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T23:08:25.308784+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:08:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.801+0000 7f5b4ca00640 1 -- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5b2806c920 msgr2=0x7f5b2806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.801+0000 7f5b4ca00640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5b2806c920 0x7f5b2806ede0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800c160 tx=0x7f5b38009030 comp rx=0 tx=0).stop 2024-04-26T23:08:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.801+0000 7f5b4ca00640 1 -- 172.21.15.40:0/3025932640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b481f48e0 msgr2=0x7f5b4815bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.801+0000 7f5b4ca00640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b481f48e0 0x7f5b4815bd50 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400d5f0 tx=0x7f5b3400dab0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.802+0000 7f5b4ca00640 1 -- 172.21.15.40:0/3025932640 shutdown_connections 2024-04-26T23:08:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.802+0000 7f5b4ca00640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5b2806c920 0x7f5b2806ede0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.802+0000 7f5b4ca00640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b481f5a60 0x7f5b48144200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.802+0000 7f5b4ca00640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b481f48e0 0x7f5b4815bd50 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.802+0000 7f5b4ca00640 1 --2- 172.21.15.40:0/3025932640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b2c0039e0 0x7f5b480b58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.802+0000 7f5b4ca00640 1 -- 172.21.15.40:0/3025932640 >> 172.21.15.40:0/3025932640 conn(0x7f5b48072ee0 msgr2=0x7f5b48141a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.802+0000 7f5b4ca00640 1 -- 172.21.15.40:0/3025932640 shutdown_connections 2024-04-26T23:08:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:27.802+0000 7f5b4ca00640 1 -- 172.21.15.40:0/3025932640 wait complete. 2024-04-26T23:08:27.820 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'paused paths [/d1/d2/d3]', 'c': 'paused paths [/d1/d2/d3]', 'd': 'paused paths [/d1/d2/d3]'} 2024-04-26T23:08:27.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:08:27.820 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub resume 2024-04-26T23:08:28.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.016+0000 7f04c7a00640 1 Processor -- start 2024-04-26T23:08:28.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.016+0000 7f04c7a00640 1 -- start start 2024-04-26T23:08:28.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.018+0000 7f04c7a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f04c0072950 0x7f04c0167ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.018+0000 7f04c7a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04c015d7c0 con 0x7f04c0071e60 2024-04-26T23:08:28.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.018+0000 7f04c7a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04c015daa0 con 0x7f04c00cc480 2024-04-26T23:08:28.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.018+0000 7f04c7a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f04c015dd80 con 0x7f04c0072950 2024-04-26T23:08:28.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.018+0000 7f04c5000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f04c0071e60 0x7f04c006f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39946/0 (socket says 172.21.15.40:39946) 2024-04-26T23:08:28.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.018+0000 7f04c5000640 1 -- 172.21.15.40:0/2884786669 learned_addr learned my addr 172.21.15.40:0/2884786669 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:28.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.019+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 923184638 0 0) 0x7f04c015d7c0 con 0x7f04c0071e60 2024-04-26T23:08:28.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.019+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04a8003520 con 0x7f04c0071e60 2024-04-26T23:08:28.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.019+0000 7f04c5a00640 1 --2- 172.21.15.40:0/2884786669 >> v2:172.21.15.103:3300/0 conn(0x7f04c0072950 0x7f04c0167ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.019+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 571164055 0 0) 0x7f04a8003520 con 0x7f04c0071e60 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.019+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 >> v2:172.21.15.103:3300/0 conn(0x7f04c0072950 msgr2=0x7f04c0167ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.019+0000 7f04bf400640 1 --2- 172.21.15.40:0/2884786669 >> v2:172.21.15.103:3300/0 conn(0x7f04c0072950 0x7f04c0167ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.019+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 >> v1:172.21.15.103:6789/0 conn(0x7f04c00cc480 legacy=0x7f04c0072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.019+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c015e060 con 0x7f04c0071e60 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.019+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f04b4001670 con 0x7f04c0071e60 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.019+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 >> v1:172.21.15.40:6789/0 conn(0x7f04c0071e60 legacy=0x7f04c006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.019+0000 7f04bf400640 1 --2- 172.21.15.40:0/2884786669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a8003a00 0x7f04a8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.019+0000 7f04bf400640 1 --2- 172.21.15.40:0/2884786669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04a8004800 0x7f04a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.019+0000 7f04bf400640 1 --2- 172.21.15.40:0/2884786669 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04a8005210 0x7f04a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.020+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f04c015dd80 con 0x7f04a8003a00 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.020+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04a800c160 con 0x7f04a8004800 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.020+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f04a800c360 con 0x7f04a8005210 2024-04-26T23:08:28.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.020+0000 7f04c5000640 1 --2- 172.21.15.40:0/2884786669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a8003a00 0x7f04a8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.020+0000 7f04c5000640 1 -- 172.21.15.40:0/2884786669 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04a8005210 msgr2=0x7f04a800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.020+0000 7f04c5000640 1 --2- 172.21.15.40:0/2884786669 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04a8005210 0x7f04a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.020+0000 7f04bfe00640 1 --2- 172.21.15.40:0/2884786669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04a8004800 0x7f04a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.020+0000 7f04c5000640 1 -- 172.21.15.40:0/2884786669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04a8004800 msgr2=0x7f04a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.020+0000 7f04c5000640 1 --2- 172.21.15.40:0/2884786669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04a8004800 0x7f04a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.020+0000 7f04c5000640 1 -- 172.21.15.40:0/2884786669 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a8003520 con 0x7f04a8003a00 2024-04-26T23:08:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.020+0000 7f04c5a00640 1 -- 172.21.15.40:0/2884786669 reap_dead start 2024-04-26T23:08:28.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.020+0000 7f04bfe00640 1 --2- 172.21.15.40:0/2884786669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04a8004800 0x7f04a8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:28.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.021+0000 7f04c5000640 1 --2- 172.21.15.40:0/2884786669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a8003a00 0x7f04a8003e20 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f04b400c0e0 tx=0x7f04a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ddddfcc6bbfec2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.022+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b4007c20 con 0x7f04a8003a00 2024-04-26T23:08:28.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.022+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b4005af0 con 0x7f04a8003a00 2024-04-26T23:08:28.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.022+0000 7f04bf400640 1 -- 172.21.15.40:0/2884786669 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b4007070 con 0x7f04a8003a00 2024-04-26T23:08:28.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.022+0000 7f04c7a00640 1 -- 172.21.15.40:0/2884786669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a8003a00 msgr2=0x7f04a8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.022+0000 7f04c7a00640 1 --2- 172.21.15.40:0/2884786669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a8003a00 0x7f04a8003e20 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f04b400c0e0 tx=0x7f04a8005830 comp rx=0 tx=0).stop 2024-04-26T23:08:28.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.022+0000 7f04c7a00640 1 -- 172.21.15.40:0/2884786669 shutdown_connections 2024-04-26T23:08:28.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.022+0000 7f04c7a00640 1 --2- 172.21.15.40:0/2884786669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a8003a00 0x7f04a8003e20 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.022+0000 7f04c7a00640 1 -- 172.21.15.40:0/2884786669 >> 172.21.15.40:0/2884786669 conn(0x7f04c00d0be0 msgr2=0x7f04c00d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.022+0000 7f04c7a00640 1 -- 172.21.15.40:0/2884786669 shutdown_connections 2024-04-26T23:08:28.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.022+0000 7f04c7a00640 1 -- 172.21.15.40:0/2884786669 wait complete. 2024-04-26T23:08:28.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.024+0000 7f04c7a00640 1 Processor -- start 2024-04-26T23:08:28.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.024+0000 7f04c7a00640 1 -- start start 2024-04-26T23:08:28.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.025+0000 7f04c7a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a8003a00 0x7f04c0156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.025+0000 7f04c7a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04c00cab20 0x7f04c01579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.025+0000 7f04c7a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c0157f30 0x7f04c00ae8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.025+0000 7f04c7a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f04c015d570 con 0x7f04a8003a00 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.025+0000 7f04c7a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04c015d3f0 con 0x7f04c0157f30 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.025+0000 7f04c7a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f04c0072240 con 0x7f04c00cab20 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.026+0000 7f04c5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a8003a00 0x7f04c0156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.026+0000 7f04c5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a8003a00 0x7f04c0156160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52888/0 (socket says 172.21.15.40:52888) 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.026+0000 7f04c5000640 1 -- 172.21.15.40:0/2538375352 learned_addr learned my addr 172.21.15.40:0/2538375352 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.026+0000 7f04bfe00640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04c00cab20 0x7f04c01579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.026+0000 7f04c5a00640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c0157f30 0x7f04c00ae8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.026+0000 7f04c5000640 1 -- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04c00cab20 msgr2=0x7f04c01579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.026+0000 7f04c5000640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04c00cab20 0x7f04c01579f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.026+0000 7f04c5000640 1 -- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c0157f30 msgr2=0x7f04c00ae8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.026+0000 7f04c5000640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c0157f30 0x7f04c00ae8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.026+0000 7f04c5000640 1 -- 172.21.15.40:0/2538375352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c00b4800 con 0x7f04a8003a00 2024-04-26T23:08:28.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.026+0000 7f04c5000640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a8003a00 0x7f04c0156160 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f04b400c1d0 tx=0x7f04c015e230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.027+0000 7f04bcc00640 1 -- 172.21.15.40:0/2538375352 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b4042070 con 0x7f04a8003a00 2024-04-26T23:08:28.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.027+0000 7f04bcc00640 1 -- 172.21.15.40:0/2538375352 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b4001630 con 0x7f04a8003a00 2024-04-26T23:08:28.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.027+0000 7f04c7a00640 1 -- 172.21.15.40:0/2538375352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c00b3bf0 con 0x7f04a8003a00 2024-04-26T23:08:28.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.027+0000 7f04bcc00640 1 -- 172.21.15.40:0/2538375352 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b4037040 con 0x7f04a8003a00 2024-04-26T23:08:28.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.027+0000 7f04c7a00640 1 -- 172.21.15.40:0/2538375352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c00b41b0 con 0x7f04a8003a00 2024-04-26T23:08:28.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.030+0000 7f04bcc00640 1 -- 172.21.15.40:0/2538375352 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f04b4045070 con 0x7f04a8003a00 2024-04-26T23:08:28.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.031+0000 7f04bcc00640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f04a006c8f0 0x7f04a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.031+0000 7f04bcc00640 1 -- 172.21.15.40:0/2538375352 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f04b40ab2c0 con 0x7f04a8003a00 2024-04-26T23:08:28.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.031+0000 7f04ca989740 1 Processor -- start 2024-04-26T23:08:28.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.031+0000 7f04ca989740 1 -- start start 2024-04-26T23:08:28.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.031+0000 7f04bfe00640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f04a006c8f0 0x7f04a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.032+0000 7f04ca989740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 0x561181920920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.032+0000 7f04ca989740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 0x56118193e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.032+0000 7f04ca989740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611818f6220 0x56118193c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.032+0000 7f04ca989740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561181806030 con 0x5611818f6220 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.032+0000 7f04ca989740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5611818217f0 con 0x561181920540 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.032+0000 7f04ca989740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5611817dc7e0 con 0x56118193dde0 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.032+0000 7f04c5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611818f6220 0x56118193c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.032+0000 7f04c5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611818f6220 0x56118193c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52892/0 (socket says 172.21.15.40:52892) 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.032+0000 7f04c5a00640 1 -- 172.21.15.40:0/3628465804 learned_addr learned my addr 172.21.15.40:0/3628465804 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.033+0000 7f04c5000640 1 --2- 172.21.15.40:0/3628465804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 0x561181920920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.033+0000 7f04c5a00640 1 -- 172.21.15.40:0/3628465804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 msgr2=0x56118193e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.033+0000 7f04c5a00640 1 --2- 172.21.15.40:0/3628465804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 0x56118193e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.033+0000 7f04c5a00640 1 -- 172.21.15.40:0/3628465804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 msgr2=0x561181920920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.033+0000 7f04c5a00640 1 --2- 172.21.15.40:0/3628465804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 0x561181920920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.033+0000 7f04c5a00640 1 -- 172.21.15.40:0/3628465804 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611817e8050 con 0x5611818f6220 2024-04-26T23:08:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.033+0000 7f04bfe00640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f04a006c8f0 0x7f04a006edb0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f04ac000c00 tx=0x7f04ac009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.033+0000 7f04c5000640 1 --2- 172.21.15.40:0/3628465804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 0x561181920920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.033+0000 7f04c5a00640 1 --2- 172.21.15.40:0/3628465804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611818f6220 0x56118193c4a0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f04c00cc910 tx=0x7f04b8007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b47a2667acf8d6bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.033+0000 7f04b1600640 1 -- 172.21.15.40:0/3628465804 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8007e20 con 0x5611818f6220 2024-04-26T23:08:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.033+0000 7f04b1600640 1 -- 172.21.15.40:0/3628465804 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b801a070 con 0x5611818f6220 2024-04-26T23:08:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.033+0000 7f04b1600640 1 -- 172.21.15.40:0/3628465804 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8015640 con 0x5611818f6220 2024-04-26T23:08:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.034+0000 7f04ca989740 1 -- 172.21.15.40:0/3628465804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611818f6220 msgr2=0x56118193c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.034+0000 7f04ca989740 1 --2- 172.21.15.40:0/3628465804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611818f6220 0x56118193c4a0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f04c00cc910 tx=0x7f04b8007620 comp rx=0 tx=0).stop 2024-04-26T23:08:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.034+0000 7f04ca989740 1 -- 172.21.15.40:0/3628465804 shutdown_connections 2024-04-26T23:08:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.034+0000 7f04ca989740 1 --2- 172.21.15.40:0/3628465804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611818f6220 0x56118193c4a0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.034+0000 7f04ca989740 1 --2- 172.21.15.40:0/3628465804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 0x56118193e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.034+0000 7f04ca989740 1 --2- 172.21.15.40:0/3628465804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 0x561181920920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.034+0000 7f04ca989740 1 -- 172.21.15.40:0/3628465804 >> 172.21.15.40:0/3628465804 conn(0x5611818f2b50 msgr2=0x56118193b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.034+0000 7f04ca989740 1 -- 172.21.15.40:0/3628465804 shutdown_connections 2024-04-26T23:08:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.034+0000 7f04ca989740 1 -- 172.21.15.40:0/3628465804 wait complete. 2024-04-26T23:08:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.035+0000 7f04ca989740 1 -- start start 2024-04-26T23:08:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.035+0000 7f04ca989740 1 Processor -- start 2024-04-26T23:08:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.036+0000 7f04ca989740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 0x56118190ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.036+0000 7f04ca989740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 0x5611819233f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.036+0000 7f04ca989740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 0x56118193e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.036+0000 7f04ca989740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56118185e200 con 0x561181923930 2024-04-26T23:08:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.036+0000 7f04ca989740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561181858ee0 con 0x561181920540 2024-04-26T23:08:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.036+0000 7f04ca989740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561181865e70 con 0x56118193dde0 2024-04-26T23:08:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.036+0000 7f04ca989740 10 client.0 _ll_register_callbacks cb 0x56118184f270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.036+0000 7f04c5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 0x56118193e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.036+0000 7f04c5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 0x56118193e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52906/0 (socket says 172.21.15.40:52906) 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.036+0000 7f04c5a00640 1 -- 172.21.15.40:0/666872103 learned_addr learned my addr 172.21.15.40:0/666872103 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.036+0000 7f04c5a00640 1 -- 172.21.15.40:0/666872103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 msgr2=0x5611819233f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.036+0000 7f04c5a00640 1 --2- 172.21.15.40:0/666872103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 0x5611819233f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.037+0000 7f04c5a00640 1 -- 172.21.15.40:0/666872103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 msgr2=0x56118190ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.037+0000 7f04c5a00640 1 --2- 172.21.15.40:0/666872103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 0x56118190ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.037+0000 7f04c5a00640 1 -- 172.21.15.40:0/666872103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611816357d0 con 0x561181923930 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.037+0000 7f04c5a00640 1 --2- 172.21.15.40:0/666872103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 0x56118193e8d0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f04b8017040 tx=0x7f04b8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142ef196db327101 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.037+0000 7f04a6000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.037+0000 7f04a6000640 1 -- 172.21.15.40:0/666872103 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8015960 con 0x561181923930 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.037+0000 7f04a6000640 1 -- 172.21.15.40:0/666872103 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f04b8015b00 con 0x561181923930 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.037+0000 7f04ca989740 1 -- 172.21.15.40:0/666872103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04b801a070 con 0x561181923930 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.037+0000 7f04a6000640 1 -- 172.21.15.40:0/666872103 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f04b8015d00 con 0x561181923930 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.037+0000 7f04ca989740 10 client.8544 fetch_fsmap learned FSMap version 102 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.037+0000 7f04ca989740 1 -- 172.21.15.40:0/666872103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) -- 0x56118156b3f0 con 0x561181923930 2024-04-26T23:08:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.038+0000 7f04a6000640 1 -- 172.21.15.40:0/666872103 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b801a070 con 0x561181923930 2024-04-26T23:08:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.038+0000 7f04a6000640 1 -- 172.21.15.40:0/666872103 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 102) ==== 3550+0+0 (secure 0 0 0) 0x7f04b8033020 con 0x561181923930 2024-04-26T23:08:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.038+0000 7f04ca989740 10 client.8544 fetch_fsmap finished waiting for FSMap version 102 2024-04-26T23:08:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.038+0000 7f04ca989740 10 client.8544 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-04-26T23:08:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.038+0000 7f04ca989740 20 client.8544 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.038+0000 7f04ca989740 1 --2- 172.21.15.40:0/666872103 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x56118191bbc0 0x56118191c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.038+0000 7f04ca989740 4 client.8544 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.038+0000 7f04ca989740 1 -- 172.21.15.40:0/666872103 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611818677e0 con 0x56118191bbc0 2024-04-26T23:08:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.038+0000 7f04c5000640 1 --2- 172.21.15.40:0/666872103 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x56118191bbc0 0x56118191c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.038+0000 7f04c5000640 1 --2- 172.21.15.40:0/666872103 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x56118191bbc0 0x56118191c050 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5eb79b8455a9298e server_cookie=662c50a536156fbe in_seq=0 out_seq=0 2024-04-26T23:08:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.038+0000 7f04a6000640 10 client.8544 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.039+0000 7f04a6000640 1 -- 172.21.15.40:0/666872103 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5611818677e0 con 0x56118191bbc0 2024-04-26T23:08:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.039+0000 7f04a6000640 10 client.8544 handle_command_reply: tid=0 2024-04-26T23:08:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.039+0000 7f04ca989740 1 client.8544 shutdown 2024-04-26T23:08:28.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.039+0000 7f04ca989740 1 -- 172.21.15.40:0/666872103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 msgr2=0x56118193e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.039+0000 7f04ca989740 1 --2- 172.21.15.40:0/666872103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 0x56118193e8d0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f04b8017040 tx=0x7f04b8004ab0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.039+0000 7f04ca989740 1 -- 172.21.15.40:0/666872103 shutdown_connections 2024-04-26T23:08:28.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.039+0000 7f04ca989740 1 --2- 172.21.15.40:0/666872103 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x56118191bbc0 0x56118191c050 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.039+0000 7f04ca989740 1 --2- 172.21.15.40:0/666872103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 0x56118193e8d0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.039+0000 7f04ca989740 1 --2- 172.21.15.40:0/666872103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 0x5611819233f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.039+0000 7f04ca989740 1 --2- 172.21.15.40:0/666872103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 0x56118190ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.039+0000 7f04ca989740 1 -- 172.21.15.40:0/666872103 >> 172.21.15.40:0/666872103 conn(0x5611818f6220 msgr2=0x5611818f5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.039+0000 7f04a6000640 0 client.8544 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:28.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.040+0000 7f04ca989740 1 -- 172.21.15.40:0/666872103 shutdown_connections 2024-04-26T23:08:28.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.040+0000 7f04ca989740 1 -- 172.21.15.40:0/666872103 wait complete. 2024-04-26T23:08:28.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.040+0000 7f04ca989740 20 client.8544 trim_cache size 0 max 16384 2024-04-26T23:08:28.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.061+0000 7f04ca989740 1 Processor -- start 2024-04-26T23:08:28.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.061+0000 7f04ca989740 1 -- start start 2024-04-26T23:08:28.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.062+0000 7f04ca989740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181920540 0x561181923f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.062+0000 7f04ca989740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181923930 0x561181921df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.062+0000 7f04ca989740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 0x5611819abaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.062+0000 7f04ca989740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f04c02026c0 con 0x561181920540 2024-04-26T23:08:28.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.062+0000 7f04ca989740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04c0201880 con 0x561181923930 2024-04-26T23:08:28.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.062+0000 7f04ca989740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f04c0201f90 con 0x56118193dde0 2024-04-26T23:08:28.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.062+0000 7f04c5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181920540 0x561181923f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.062+0000 7f04c5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181920540 0x561181923f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52910/0 (socket says 172.21.15.40:52910) 2024-04-26T23:08:28.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.062+0000 7f04c5000640 1 -- 172.21.15.40:0/2111789082 learned_addr learned my addr 172.21.15.40:0/2111789082 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.063+0000 7f04c5000640 1 -- 172.21.15.40:0/2111789082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 msgr2=0x5611819abaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.063+0000 7f04c5a00640 1 --2- 172.21.15.40:0/2111789082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 0x5611819abaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.063+0000 7f04c5000640 1 --2- 172.21.15.40:0/2111789082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 0x5611819abaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.063+0000 7f04c5000640 1 -- 172.21.15.40:0/2111789082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181923930 msgr2=0x561181921df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.063+0000 7f04c5000640 1 --2- 172.21.15.40:0/2111789082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181923930 0x561181921df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.063+0000 7f04c5000640 1 -- 172.21.15.40:0/2111789082 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56118152b0f0 con 0x561181920540 2024-04-26T23:08:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.063+0000 7f04c5000640 1 --2- 172.21.15.40:0/2111789082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181920540 0x561181923f20 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f04b4078c00 tx=0x7f04b4075220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd9f2771414a1dc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.063+0000 7f04b0c00640 1 -- 172.21.15.40:0/2111789082 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b4091020 con 0x561181920540 2024-04-26T23:08:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.063+0000 7f04b0c00640 1 -- 172.21.15.40:0/2111789082 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b407b040 con 0x561181920540 2024-04-26T23:08:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.063+0000 7f04b0c00640 1 -- 172.21.15.40:0/2111789082 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b40ab070 con 0x561181920540 2024-04-26T23:08:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.064+0000 7f04ca989740 1 -- 172.21.15.40:0/2111789082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181920540 msgr2=0x561181923f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.064+0000 7f04ca989740 1 --2- 172.21.15.40:0/2111789082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181920540 0x561181923f20 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f04b4078c00 tx=0x7f04b4075220 comp rx=0 tx=0).stop 2024-04-26T23:08:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.064+0000 7f04ca989740 1 -- 172.21.15.40:0/2111789082 shutdown_connections 2024-04-26T23:08:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.064+0000 7f04ca989740 1 --2- 172.21.15.40:0/2111789082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 0x5611819abaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.064+0000 7f04ca989740 1 --2- 172.21.15.40:0/2111789082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181923930 0x561181921df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.064+0000 7f04ca989740 1 --2- 172.21.15.40:0/2111789082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181920540 0x561181923f20 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.064+0000 7f04ca989740 1 -- 172.21.15.40:0/2111789082 >> 172.21.15.40:0/2111789082 conn(0x5611818f6220 msgr2=0x56118191fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.064+0000 7f04ca989740 1 -- 172.21.15.40:0/2111789082 shutdown_connections 2024-04-26T23:08:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.064+0000 7f04ca989740 1 -- 172.21.15.40:0/2111789082 wait complete. 2024-04-26T23:08:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.065+0000 7f04ca989740 1 -- start start 2024-04-26T23:08:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.065+0000 7f04ca989740 1 Processor -- start 2024-04-26T23:08:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.066+0000 7f04ca989740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 0x56118195dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.066+0000 7f04ca989740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 0x56118195e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.066+0000 7f04ca989740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 0x56118191bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.066+0000 7f04ca989740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561181866fd0 con 0x561181923930 2024-04-26T23:08:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.066+0000 7f04ca989740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04c00cfcd0 con 0x561181920540 2024-04-26T23:08:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.066+0000 7f04ca989740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5611818f7790 con 0x56118193dde0 2024-04-26T23:08:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.066+0000 7f04ca989740 10 client.0 _ll_register_callbacks cb 0x7f04b8009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.066+0000 7f04bfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 0x56118195e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.066+0000 7f04bfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 0x56118195e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52914/0 (socket says 172.21.15.40:52914) 2024-04-26T23:08:28.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.066+0000 7f04bfe00640 1 -- 172.21.15.40:0/3008420371 learned_addr learned my addr 172.21.15.40:0/3008420371 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.067+0000 7f04c5000640 1 --2- 172.21.15.40:0/3008420371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 0x56118195dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.067+0000 7f04bfe00640 1 -- 172.21.15.40:0/3008420371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 msgr2=0x56118191bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.067+0000 7f04bfe00640 1 --2- 172.21.15.40:0/3008420371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 0x56118191bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.067+0000 7f04bfe00640 1 -- 172.21.15.40:0/3008420371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 msgr2=0x56118195dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.067+0000 7f04bfe00640 1 --2- 172.21.15.40:0/3008420371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 0x56118195dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.067+0000 7f04bfe00640 1 -- 172.21.15.40:0/3008420371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561181821a30 con 0x561181923930 2024-04-26T23:08:28.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.067+0000 7f04bfe00640 1 --2- 172.21.15.40:0/3008420371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 0x56118195e1a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f04c00b0620 tx=0x7f04ac00e810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=455722a121e9b343 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.067+0000 7f04a6000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:28.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.067+0000 7f04a6000640 1 -- 172.21.15.40:0/3008420371 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac005310 con 0x561181923930 2024-04-26T23:08:28.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.067+0000 7f04a6000640 1 -- 172.21.15.40:0/3008420371 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f04ac021c60 con 0x561181923930 2024-04-26T23:08:28.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.067+0000 7f04ca989740 1 -- 172.21.15.40:0/3008420371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04b407b040 con 0x561181923930 2024-04-26T23:08:28.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.068+0000 7f04a6000640 1 -- 172.21.15.40:0/3008420371 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f04ac021e60 con 0x561181923930 2024-04-26T23:08:28.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.068+0000 7f04ca989740 10 client.8550 fetch_fsmap learned FSMap version 102 2024-04-26T23:08:28.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.068+0000 7f04ca989740 1 -- 172.21.15.40:0/3008420371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) -- 0x56118191cf90 con 0x561181923930 2024-04-26T23:08:28.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.068+0000 7f04a6000640 1 -- 172.21.15.40:0/3008420371 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b407b040 con 0x561181923930 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.068+0000 7f04a6000640 1 -- 172.21.15.40:0/3008420371 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 102) ==== 3550+0+0 (secure 0 0 0) 0x7f04ac028050 con 0x561181923930 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.068+0000 7f04ca989740 10 client.8550 fetch_fsmap finished waiting for FSMap version 102 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.068+0000 7f04ca989740 10 client.8550 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.068+0000 7f04ca989740 20 client.8550 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.068+0000 7f04ca989740 1 --2- 172.21.15.40:0/3008420371 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x56118191ed70 0x56118191c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.068+0000 7f04ca989740 4 client.8550 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "scrub resume"}] 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.068+0000 7f04ca989740 1 -- 172.21.15.40:0/3008420371 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "scrub resume"}) -- 0x561181861650 con 0x56118191ed70 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.068+0000 7f04c5000640 1 --2- 172.21.15.40:0/3008420371 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x56118191ed70 0x56118191c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.069+0000 7f04c5000640 1 --2- 172.21.15.40:0/3008420371 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x56118191ed70 0x56118191c4e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4e9ee7300a79399 server_cookie=e6b141c6e3e955a5 in_seq=0 out_seq=0 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.069+0000 7f04a6000640 10 client.8550 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.069+0000 7f04a6000640 1 -- 172.21.15.40:0/3008420371 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x561181861650 con 0x56118191ed70 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.069+0000 7f04a6000640 10 client.8550 handle_command_reply: tid=0 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.069+0000 7f04ca989740 1 client.8550 shutdown 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.069+0000 7f04ca989740 1 -- 172.21.15.40:0/3008420371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 msgr2=0x56118195e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.069+0000 7f04ca989740 1 --2- 172.21.15.40:0/3008420371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 0x56118195e1a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f04c00b0620 tx=0x7f04ac00e810 comp rx=0 tx=0).stop 2024-04-26T23:08:28.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.069+0000 7f04ca989740 1 -- 172.21.15.40:0/3008420371 shutdown_connections 2024-04-26T23:08:28.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.069+0000 7f04ca989740 1 --2- 172.21.15.40:0/3008420371 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x56118191ed70 0x56118191c4e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.070+0000 7f04ca989740 1 --2- 172.21.15.40:0/3008420371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56118193dde0 0x56118191bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.070+0000 7f04ca989740 1 --2- 172.21.15.40:0/3008420371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561181923930 0x56118195e1a0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.070+0000 7f04ca989740 1 --2- 172.21.15.40:0/3008420371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561181920540 0x56118195dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.070+0000 7f04ca989740 1 -- 172.21.15.40:0/3008420371 >> 172.21.15.40:0/3008420371 conn(0x5611818f6220 msgr2=0x56118195ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.070+0000 7f04a6000640 0 client.8550 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:28.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.070+0000 7f04ca989740 1 -- 172.21.15.40:0/3008420371 shutdown_connections 2024-04-26T23:08:28.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.070+0000 7f04ca989740 1 -- 172.21.15.40:0/3008420371 wait complete. 2024-04-26T23:08:28.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.070+0000 7f04ca989740 20 client.8550 trim_cache size 0 max 16384 2024-04-26T23:08:28.072 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:28.073 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:08:28.073 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:28.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.070+0000 7f04c7a00640 1 -- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f04a006c8f0 msgr2=0x7f04a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.070+0000 7f04c7a00640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f04a006c8f0 0x7f04a006edb0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f04ac000c00 tx=0x7f04ac009040 comp rx=0 tx=0).stop 2024-04-26T23:08:28.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.070+0000 7f04c7a00640 1 -- 172.21.15.40:0/2538375352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a8003a00 msgr2=0x7f04c0156160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.070+0000 7f04c7a00640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a8003a00 0x7f04c0156160 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f04b400c1d0 tx=0x7f04c015e230 comp rx=0 tx=0).stop 2024-04-26T23:08:28.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.071+0000 7f04c7a00640 1 -- 172.21.15.40:0/2538375352 shutdown_connections 2024-04-26T23:08:28.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.071+0000 7f04c7a00640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f04a006c8f0 0x7f04a006edb0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.071+0000 7f04c7a00640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c0157f30 0x7f04c00ae8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.071+0000 7f04c7a00640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f04c00cab20 0x7f04c01579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.071+0000 7f04c7a00640 1 --2- 172.21.15.40:0/2538375352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04a8003a00 0x7f04c0156160 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.071+0000 7f04c7a00640 1 -- 172.21.15.40:0/2538375352 >> 172.21.15.40:0/2538375352 conn(0x7f04c00d0be0 msgr2=0x7f04c00b1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.071+0000 7f04c7a00640 1 -- 172.21.15.40:0/2538375352 shutdown_connections 2024-04-26T23:08:28.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.071+0000 7f04c7a00640 1 -- 172.21.15.40:0/2538375352 wait complete. 2024-04-26T23:08:28.086 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:08:28.087 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.6:1 scrub status 2024-04-26T23:08:28.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.282+0000 7fe17b200640 1 Processor -- start 2024-04-26T23:08:28.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.283+0000 7fe17b200640 1 -- start start 2024-04-26T23:08:28.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.284+0000 7fe17b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17c0b6470 0x7fe17c0cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.284+0000 7fe17b200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c0d5d30 con 0x7fe17c0d1770 2024-04-26T23:08:28.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.284+0000 7fe17b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c0d5a70 con 0x7fe17c15f680 2024-04-26T23:08:28.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.284+0000 7fe17b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe17c0d3760 con 0x7fe17c0b6470 2024-04-26T23:08:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.284+0000 7fe179e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe17c0d1770 0x7fe17c15ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39954/0 (socket says 172.21.15.40:39954) 2024-04-26T23:08:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.284+0000 7fe179e00640 1 -- 172.21.15.40:0/240041457 learned_addr learned my addr 172.21.15.40:0/240041457 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.285+0000 7fe179400640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17c0b6470 0x7fe17c0cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.285+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1989643682 0 0) 0x7fe17c0d5d30 con 0x7fe17c0d1770 2024-04-26T23:08:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.285+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe15c003520 con 0x7fe17c0d1770 2024-04-26T23:08:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.285+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3851391018 0 0) 0x7fe15c003520 con 0x7fe17c0d1770 2024-04-26T23:08:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.285+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17c0b6470 msgr2=0x7fe17c0cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.285+0000 7fe178a00640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17c0b6470 0x7fe17c0cf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 >> v1:172.21.15.103:6789/0 conn(0x7fe17c15f680 legacy=0x7fe17c0cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c0b5b00 con 0x7fe17c0d1770 2024-04-26T23:08:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fe16c001670 con 0x7fe17c0d1770 2024-04-26T23:08:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 >> v1:172.21.15.40:6789/0 conn(0x7fe17c0d1770 legacy=0x7fe17c15ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe178a00640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe15c0039e0 0x7fe15c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe178a00640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe15c004800 0x7fe15c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe178a00640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17c0b6470 0x7fe15c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c0d3760 con 0x7fe15c004800 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c00ba70 con 0x7fe15c0039e0 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe15c00bc70 con 0x7fe17c0b6470 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe179400640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe15c004800 0x7fe15c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe179e00640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe15c0039e0 0x7fe15c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe179400640 1 -- 172.21.15.40:0/240041457 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17c0b6470 msgr2=0x7fe15c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.286+0000 7fe17a800640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17c0b6470 0x7fe15c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.287+0000 7fe179400640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17c0b6470 0x7fe15c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.287+0000 7fe179400640 1 -- 172.21.15.40:0/240041457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe15c0039e0 msgr2=0x7fe15c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.287+0000 7fe179400640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe15c0039e0 0x7fe15c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.287+0000 7fe179400640 1 -- 172.21.15.40:0/240041457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe15c003520 con 0x7fe15c004800 2024-04-26T23:08:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.287+0000 7fe179e00640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe15c0039e0 0x7fe15c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:28.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.287+0000 7fe17a800640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe17c0b6470 0x7fe15c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.288+0000 7fe179400640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe15c004800 0x7fe15c004c80 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fe17c0b6890 tx=0x7fe15c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26991e1182a65073 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.288+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164037070 con 0x7fe15c004800 2024-04-26T23:08:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.288+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1640325f0 con 0x7fe15c004800 2024-04-26T23:08:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.288+0000 7fe178a00640 1 -- 172.21.15.40:0/240041457 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164009c10 con 0x7fe15c004800 2024-04-26T23:08:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.289+0000 7fe17b200640 1 -- 172.21.15.40:0/240041457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe15c004800 msgr2=0x7fe15c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.289+0000 7fe17b200640 1 --2- 172.21.15.40:0/240041457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe15c004800 0x7fe15c004c80 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fe17c0b6890 tx=0x7fe15c005830 comp rx=0 tx=0).stop 2024-04-26T23:08:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.289+0000 7fe17a800640 1 -- 172.21.15.40:0/240041457 reap_dead start 2024-04-26T23:08:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.289+0000 7fe17b200640 1 -- 172.21.15.40:0/240041457 shutdown_connections 2024-04-26T23:08:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.289+0000 7fe17b200640 1 -- 172.21.15.40:0/240041457 >> 172.21.15.40:0/240041457 conn(0x7fe17c0d4dd0 msgr2=0x7fe17c0d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.289+0000 7fe17b200640 1 -- 172.21.15.40:0/240041457 shutdown_connections 2024-04-26T23:08:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.289+0000 7fe17b200640 1 -- 172.21.15.40:0/240041457 wait complete. 2024-04-26T23:08:28.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.291+0000 7fe17b200640 1 Processor -- start 2024-04-26T23:08:28.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.291+0000 7fe17b200640 1 -- start start 2024-04-26T23:08:28.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.293+0000 7fe17b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe15c004800 0x7fe17c152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.293+0000 7fe17b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe17c0cc460 0x7fe17c0ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.293+0000 7fe17b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe17c0b2d60 0x7fe17c0b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.293+0000 7fe17b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c0d3d70 con 0x7fe17c0b2d60 2024-04-26T23:08:28.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.293+0000 7fe17b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c0d3bf0 con 0x7fe17c0cc460 2024-04-26T23:08:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.293+0000 7fe17b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe17c0d1b50 con 0x7fe15c004800 2024-04-26T23:08:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.293+0000 7fe17a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe17c0b2d60 0x7fe17c0b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.293+0000 7fe17a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe17c0b2d60 0x7fe17c0b31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52934/0 (socket says 172.21.15.40:52934) 2024-04-26T23:08:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.293+0000 7fe17a800640 1 -- 172.21.15.40:0/1505482443 learned_addr learned my addr 172.21.15.40:0/1505482443 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.293+0000 7fe179e00640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe15c004800 0x7fe17c152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.293+0000 7fe179400640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe17c0cc460 0x7fe17c0ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.294+0000 7fe17a800640 1 -- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe15c004800 msgr2=0x7fe17c152800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.294+0000 7fe17a800640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe15c004800 0x7fe17c152800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.294+0000 7fe17a800640 1 -- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe17c0cc460 msgr2=0x7fe17c0ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.294+0000 7fe17a800640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe17c0cc460 0x7fe17c0ceb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.294+0000 7fe17a800640 1 -- 172.21.15.40:0/1505482443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c1f8790 con 0x7fe17c0b2d60 2024-04-26T23:08:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.294+0000 7fe17a800640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe17c0b2d60 0x7fe17c0b31e0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fe17c0bae80 tx=0x7fe17400d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.294+0000 7fe172000640 1 -- 172.21.15.40:0/1505482443 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe174007030 con 0x7fe17c0b2d60 2024-04-26T23:08:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.294+0000 7fe172000640 1 -- 172.21.15.40:0/1505482443 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1740071f0 con 0x7fe17c0b2d60 2024-04-26T23:08:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.294+0000 7fe17b200640 1 -- 172.21.15.40:0/1505482443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17c1f8a70 con 0x7fe17c0b2d60 2024-04-26T23:08:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.294+0000 7fe172000640 1 -- 172.21.15.40:0/1505482443 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1740075e0 con 0x7fe17c0b2d60 2024-04-26T23:08:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.295+0000 7fe17b200640 1 -- 172.21.15.40:0/1505482443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe17c1f9030 con 0x7fe17c0b2d60 2024-04-26T23:08:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.297+0000 7fe172000640 1 -- 172.21.15.40:0/1505482443 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe17401b020 con 0x7fe17c0b2d60 2024-04-26T23:08:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.298+0000 7fe172000640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe15806c8f0 0x7fe15806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.298+0000 7fe172000640 1 -- 172.21.15.40:0/1505482443 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fe17408b6c0 con 0x7fe17c0b2d60 2024-04-26T23:08:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.298+0000 7fe183832740 1 Processor -- start 2024-04-26T23:08:28.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.298+0000 7fe179e00640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe15806c8f0 0x7fe15806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.298+0000 7fe183832740 1 -- start start 2024-04-26T23:08:28.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.299+0000 7fe183832740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7aa1710 0x5577b7aa1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.299+0000 7fe183832740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7abefb0 0x5577b7abf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.299+0000 7fe183832740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7a773f0 0x5577b7abd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.299+0000 7fe183832740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5577b7987030 con 0x5577b7aa1710 2024-04-26T23:08:28.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.299+0000 7fe183832740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5577b79a27f0 con 0x5577b7a773f0 2024-04-26T23:08:28.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.299+0000 7fe183832740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5577b795d7e0 con 0x5577b7abefb0 2024-04-26T23:08:28.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.300+0000 7fe179e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7aa1710 0x5577b7aa1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.300+0000 7fe179e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7aa1710 0x5577b7aa1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52950/0 (socket says 172.21.15.40:52950) 2024-04-26T23:08:28.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.300+0000 7fe179e00640 1 -- 172.21.15.40:0/3122825606 learned_addr learned my addr 172.21.15.40:0/3122825606 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.300+0000 7fe179e00640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe15806c8f0 0x7fe15806edb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0017e0 tx=0x7fe16c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.300+0000 7fe179400640 1 --2- 172.21.15.40:0/3122825606 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7abefb0 0x5577b7abf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.300+0000 7fe17a800640 1 --2- 172.21.15.40:0/3122825606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7a773f0 0x5577b7abd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.300+0000 7fe179e00640 1 -- 172.21.15.40:0/3122825606 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7abefb0 msgr2=0x5577b7abf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.300+0000 7fe179e00640 1 --2- 172.21.15.40:0/3122825606 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7abefb0 0x5577b7abf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.300+0000 7fe179e00640 1 -- 172.21.15.40:0/3122825606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7a773f0 msgr2=0x5577b7abd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.300+0000 7fe179e00640 1 --2- 172.21.15.40:0/3122825606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7a773f0 0x5577b7abd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.300+0000 7fe179e00640 1 -- 172.21.15.40:0/3122825606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577b7969050 con 0x5577b7aa1710 2024-04-26T23:08:28.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.300+0000 7fe17a800640 1 --2- 172.21.15.40:0/3122825606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7a773f0 0x5577b7abd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:28.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.301+0000 7fe179e00640 1 --2- 172.21.15.40:0/3122825606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7aa1710 0x5577b7aa1af0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00fe00 tx=0x7fe16c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e993b2a1991e08e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.301+0000 7fe16a600640 1 -- 172.21.15.40:0/3122825606 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c026070 con 0x5577b7aa1710 2024-04-26T23:08:28.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.301+0000 7fe16a600640 1 -- 172.21.15.40:0/3122825606 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16c01c040 con 0x5577b7aa1710 2024-04-26T23:08:28.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.301+0000 7fe16a600640 1 -- 172.21.15.40:0/3122825606 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c01b070 con 0x5577b7aa1710 2024-04-26T23:08:28.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.301+0000 7fe183832740 1 -- 172.21.15.40:0/3122825606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7aa1710 msgr2=0x5577b7aa1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.301+0000 7fe183832740 1 --2- 172.21.15.40:0/3122825606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7aa1710 0x5577b7aa1af0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00fe00 tx=0x7fe16c012000 comp rx=0 tx=0).stop 2024-04-26T23:08:28.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.302+0000 7fe183832740 1 -- 172.21.15.40:0/3122825606 shutdown_connections 2024-04-26T23:08:28.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.302+0000 7fe183832740 1 --2- 172.21.15.40:0/3122825606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7a773f0 0x5577b7abd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.302+0000 7fe183832740 1 --2- 172.21.15.40:0/3122825606 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7abefb0 0x5577b7abf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.302+0000 7fe183832740 1 --2- 172.21.15.40:0/3122825606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7aa1710 0x5577b7aa1af0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.302+0000 7fe183832740 1 -- 172.21.15.40:0/3122825606 >> 172.21.15.40:0/3122825606 conn(0x5577b7a73d20 msgr2=0x5577b7abc520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.302+0000 7fe183832740 1 -- 172.21.15.40:0/3122825606 shutdown_connections 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.302+0000 7fe183832740 1 -- 172.21.15.40:0/3122825606 wait complete. 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.302+0000 7fe183832740 1 -- start start 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.303+0000 7fe183832740 1 Processor -- start 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe183832740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7aa1710 0x5577b7a9f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe183832740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7abefb0 0x5577b7ab6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe183832740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7ab6ac0 0x5577b7abfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe183832740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5577b79df200 con 0x5577b7abefb0 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe183832740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5577b79d9ee0 con 0x5577b7ab6ac0 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe183832740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5577b79e6e70 con 0x5577b7aa1710 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe183832740 10 client.0 _ll_register_callbacks cb 0x5577b79d0270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe179400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7abefb0 0x5577b7ab6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe179400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7abefb0 0x5577b7ab6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52960/0 (socket says 172.21.15.40:52960) 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe179400640 1 -- 172.21.15.40:0/3934189614 learned_addr learned my addr 172.21.15.40:0/3934189614 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe179e00640 1 --2- 172.21.15.40:0/3934189614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7aa1710 0x5577b7a9f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe179400640 1 -- 172.21.15.40:0/3934189614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7aa1710 msgr2=0x5577b7a9f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe179400640 1 --2- 172.21.15.40:0/3934189614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7aa1710 0x5577b7a9f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe179400640 1 -- 172.21.15.40:0/3934189614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7ab6ac0 msgr2=0x5577b7abfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe179400640 1 --2- 172.21.15.40:0/3934189614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7ab6ac0 0x5577b7abfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.304+0000 7fe179400640 1 -- 172.21.15.40:0/3934189614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577b77b67d0 con 0x5577b7abefb0 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.305+0000 7fe179400640 1 --2- 172.21.15.40:0/3934189614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7abefb0 0x5577b7ab6580 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fe16402ae30 tx=0x7fe164004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e1dcf2bd5a2bd72 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.305+0000 7fe162a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.305+0000 7fe162a00640 1 -- 172.21.15.40:0/3934189614 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164037040 con 0x5577b7abefb0 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.305+0000 7fe162a00640 1 -- 172.21.15.40:0/3934189614 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fe164047c70 con 0x5577b7abefb0 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.305+0000 7fe183832740 1 -- 172.21.15.40:0/3934189614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe16c01c040 con 0x5577b7abefb0 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.305+0000 7fe162a00640 1 -- 172.21.15.40:0/3934189614 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7fe164009cd0 con 0x5577b7abefb0 2024-04-26T23:08:28.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.305+0000 7fe183832740 10 client.8562 fetch_fsmap learned FSMap version 102 2024-04-26T23:08:28.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.305+0000 7fe183832740 1 -- 172.21.15.40:0/3934189614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) -- 0x5577b76ec3f0 con 0x5577b7abefb0 2024-04-26T23:08:28.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.306+0000 7fe162a00640 1 -- 172.21.15.40:0/3934189614 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16c01c040 con 0x5577b7abefb0 2024-04-26T23:08:28.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.306+0000 7fe162a00640 1 -- 172.21.15.40:0/3934189614 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 102) ==== 3550+0+0 (secure 0 0 0) 0x7fe164031070 con 0x5577b7abefb0 2024-04-26T23:08:28.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.306+0000 7fe183832740 10 client.8562 fetch_fsmap finished waiting for FSMap version 102 2024-04-26T23:08:28.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.306+0000 7fe183832740 10 client.8562 resolve_mds: resolved 6:1 to role '6:1' aka daemon mds.a 2024-04-26T23:08:28.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.306+0000 7fe183832740 20 client.8562 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:28.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.306+0000 7fe183832740 1 --2- 172.21.15.40:0/3934189614 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x5577b7ab4c40 0x5577b7ab5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.306+0000 7fe183832740 4 client.8562 mds_command: new command op to 8031 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:28.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.306+0000 7fe183832740 1 -- 172.21.15.40:0/3934189614 --> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577b79e87e0 con 0x5577b7ab4c40 2024-04-26T23:08:28.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.306+0000 7fe179e00640 1 --2- 172.21.15.40:0/3934189614 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x5577b7ab4c40 0x5577b7ab5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.307+0000 7fe179e00640 1 --2- 172.21.15.40:0/3934189614 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x5577b7ab4c40 0x5577b7ab5090 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1996211f388a4b server_cookie=647bcf59e6e66baf in_seq=0 out_seq=0 2024-04-26T23:08:28.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.307+0000 7fe162a00640 10 client.8562 ms_handle_connect on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:28.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.307+0000 7fe162a00640 1 -- 172.21.15.40:0/3934189614 <== mds.1 v2:172.21.15.40:6832/1811310743 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5577b79e87e0 con 0x5577b7ab4c40 2024-04-26T23:08:28.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.307+0000 7fe162a00640 10 client.8562 handle_command_reply: tid=0 2024-04-26T23:08:28.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.307+0000 7fe183832740 1 client.8562 shutdown 2024-04-26T23:08:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.308+0000 7fe183832740 1 -- 172.21.15.40:0/3934189614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7abefb0 msgr2=0x5577b7ab6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.308+0000 7fe183832740 1 --2- 172.21.15.40:0/3934189614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7abefb0 0x5577b7ab6580 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fe16402ae30 tx=0x7fe164004ab0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.308+0000 7fe183832740 1 -- 172.21.15.40:0/3934189614 shutdown_connections 2024-04-26T23:08:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.308+0000 7fe183832740 1 --2- 172.21.15.40:0/3934189614 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x5577b7ab4c40 0x5577b7ab5090 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.308+0000 7fe183832740 1 --2- 172.21.15.40:0/3934189614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7ab6ac0 0x5577b7abfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.308+0000 7fe183832740 1 --2- 172.21.15.40:0/3934189614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7abefb0 0x5577b7ab6580 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.308+0000 7fe162a00640 0 client.8562 ms_handle_reset on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.308+0000 7fe183832740 1 --2- 172.21.15.40:0/3934189614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7aa1710 0x5577b7a9f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.308+0000 7fe183832740 1 -- 172.21.15.40:0/3934189614 >> 172.21.15.40:0/3934189614 conn(0x5577b7a773f0 msgr2=0x5577b7a76970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.308+0000 7fe183832740 1 -- 172.21.15.40:0/3934189614 shutdown_connections 2024-04-26T23:08:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.308+0000 7fe183832740 1 -- 172.21.15.40:0/3934189614 wait complete. 2024-04-26T23:08:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.308+0000 7fe183832740 20 client.8562 trim_cache size 0 max 16384 2024-04-26T23:08:28.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.329+0000 7fe183832740 1 Processor -- start 2024-04-26T23:08:28.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.329+0000 7fe183832740 1 -- start start 2024-04-26T23:08:28.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.330+0000 7fe183832740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7aa1710 0x5577b7a752c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.330+0000 7fe183832740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 0x5577b7abf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.330+0000 7fe183832740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7abefb0 0x5577b7abf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.330+0000 7fe183832740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c2025c0 con 0x5577b7ab6ac0 2024-04-26T23:08:28.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.330+0000 7fe183832740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c201760 con 0x5577b7abefb0 2024-04-26T23:08:28.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.330+0000 7fe183832740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe17c201e90 con 0x5577b7aa1710 2024-04-26T23:08:28.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.330+0000 7fe179400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 0x5577b7abf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.330+0000 7fe179400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 0x5577b7abf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52964/0 (socket says 172.21.15.40:52964) 2024-04-26T23:08:28.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.330+0000 7fe179400640 1 -- 172.21.15.40:0/3397646814 learned_addr learned my addr 172.21.15.40:0/3397646814 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.331+0000 7fe179400640 1 -- 172.21.15.40:0/3397646814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7aa1710 msgr2=0x5577b7a752c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.331+0000 7fe179400640 1 --2- 172.21.15.40:0/3397646814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7aa1710 0x5577b7a752c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.331+0000 7fe179400640 1 -- 172.21.15.40:0/3397646814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7abefb0 msgr2=0x5577b7abf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.331+0000 7fe179400640 1 --2- 172.21.15.40:0/3397646814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7abefb0 0x5577b7abf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.331+0000 7fe179400640 1 -- 172.21.15.40:0/3397646814 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577b76ac0f0 con 0x5577b7ab6ac0 2024-04-26T23:08:28.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.331+0000 7fe179400640 1 --2- 172.21.15.40:0/3397646814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 0x5577b7abf460 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fe16402ad60 tx=0x7fe16403f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de5bc2bb3b3490a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.331+0000 7fe169c00640 1 -- 172.21.15.40:0/3397646814 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164037040 con 0x5577b7ab6ac0 2024-04-26T23:08:28.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.331+0000 7fe169c00640 1 -- 172.21.15.40:0/3397646814 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe164046630 con 0x5577b7ab6ac0 2024-04-26T23:08:28.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.331+0000 7fe169c00640 1 -- 172.21.15.40:0/3397646814 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1640469e0 con 0x5577b7ab6ac0 2024-04-26T23:08:28.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.332+0000 7fe183832740 1 -- 172.21.15.40:0/3397646814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 msgr2=0x5577b7abf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.332+0000 7fe183832740 1 --2- 172.21.15.40:0/3397646814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 0x5577b7abf460 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fe16402ad60 tx=0x7fe16403f070 comp rx=0 tx=0).stop 2024-04-26T23:08:28.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.332+0000 7fe183832740 1 -- 172.21.15.40:0/3397646814 shutdown_connections 2024-04-26T23:08:28.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.332+0000 7fe183832740 1 --2- 172.21.15.40:0/3397646814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7abefb0 0x5577b7abf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.332+0000 7fe183832740 1 --2- 172.21.15.40:0/3397646814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 0x5577b7abf460 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.332+0000 7fe183832740 1 --2- 172.21.15.40:0/3397646814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7aa1710 0x5577b7a752c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.332+0000 7fe183832740 1 -- 172.21.15.40:0/3397646814 >> 172.21.15.40:0/3397646814 conn(0x5577b7a773f0 msgr2=0x5577b7a77810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.332+0000 7fe183832740 1 -- 172.21.15.40:0/3397646814 shutdown_connections 2024-04-26T23:08:28.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.332+0000 7fe183832740 1 -- 172.21.15.40:0/3397646814 wait complete. 2024-04-26T23:08:28.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.333+0000 7fe183832740 1 -- start start 2024-04-26T23:08:28.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.333+0000 7fe183832740 1 Processor -- start 2024-04-26T23:08:28.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.334+0000 7fe183832740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7aa1710 0x5577b7aa4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.334+0000 7fe183832740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 0x5577b7a9f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.334+0000 7fe183832740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7abefb0 0x5577b7ab72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.334+0000 7fe183832740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5577b79e7fd0 con 0x5577b7ab6ac0 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.334+0000 7fe183832740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c2008d0 con 0x5577b7aa1710 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.334+0000 7fe183832740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5577b7a9fe10 con 0x5577b7abefb0 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.334+0000 7fe183832740 10 client.0 _ll_register_callbacks cb 0x7fe164035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.334+0000 7fe179400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 0x5577b7a9f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.334+0000 7fe179400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 0x5577b7a9f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52968/0 (socket says 172.21.15.40:52968) 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.334+0000 7fe179400640 1 -- 172.21.15.40:0/3334087677 learned_addr learned my addr 172.21.15.40:0/3334087677 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.335+0000 7fe17a800640 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7abefb0 0x5577b7ab72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.335+0000 7fe179e00640 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7aa1710 0x5577b7aa4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.335+0000 7fe179400640 1 -- 172.21.15.40:0/3334087677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7abefb0 msgr2=0x5577b7ab72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.335+0000 7fe179400640 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7abefb0 0x5577b7ab72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.335+0000 7fe179400640 1 -- 172.21.15.40:0/3334087677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7aa1710 msgr2=0x5577b7aa4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.335+0000 7fe179400640 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7aa1710 0x5577b7aa4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.335+0000 7fe179400640 1 -- 172.21.15.40:0/3334087677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577b79a2a30 con 0x5577b7ab6ac0 2024-04-26T23:08:28.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.335+0000 7fe179e00640 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7aa1710 0x5577b7aa4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:28.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.335+0000 7fe179400640 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 0x5577b7a9f270 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fe16400c090 tx=0x7fe16400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4fc2ca15bd003d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.335+0000 7fe162a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:28.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe162a00640 1 -- 172.21.15.40:0/3334087677 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16405d020 con 0x5577b7ab6ac0 2024-04-26T23:08:28.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe162a00640 1 -- 172.21.15.40:0/3334087677 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fe164046b90 con 0x5577b7ab6ac0 2024-04-26T23:08:28.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe183832740 1 -- 172.21.15.40:0/3334087677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe164046630 con 0x5577b7ab6ac0 2024-04-26T23:08:28.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe162a00640 1 -- 172.21.15.40:0/3334087677 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7fe164054040 con 0x5577b7ab6ac0 2024-04-26T23:08:28.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe183832740 10 client.8568 fetch_fsmap learned FSMap version 102 2024-04-26T23:08:28.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe183832740 1 -- 172.21.15.40:0/3334087677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) -- 0x5577b7a76ad0 con 0x5577b7ab6ac0 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe162a00640 1 -- 172.21.15.40:0/3334087677 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe164046630 con 0x5577b7ab6ac0 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe162a00640 1 -- 172.21.15.40:0/3334087677 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 102) ==== 3550+0+0 (secure 0 0 0) 0x7fe164031070 con 0x5577b7ab6ac0 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe183832740 10 client.8568 fetch_fsmap finished waiting for FSMap version 102 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe183832740 10 client.8568 resolve_mds: resolved 6:1 to role '6:1' aka daemon mds.a 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe183832740 20 client.8568 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe183832740 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x5577b7abbe10 0x5577b7ab77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe183832740 4 client.8568 mds_command: new command op to 8031 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.336+0000 7fe183832740 1 -- 172.21.15.40:0/3334087677 --> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5577b79e2650 con 0x5577b7abbe10 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.337+0000 7fe179e00640 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x5577b7abbe10 0x5577b7ab77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.337+0000 7fe179e00640 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x5577b7abbe10 0x5577b7ab77f0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e85f5681f4e2d24 server_cookie=22f5a133cc7e9a8b in_seq=0 out_seq=0 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.337+0000 7fe162a00640 10 client.8568 ms_handle_connect on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe162a00640 1 -- 172.21.15.40:0/3334087677 <== mds.1 v2:172.21.15.40:6832/1811310743 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5577b79e2650 con 0x5577b7abbe10 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe162a00640 10 client.8568 handle_command_reply: tid=0 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe183832740 1 client.8568 shutdown 2024-04-26T23:08:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe183832740 1 -- 172.21.15.40:0/3334087677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 msgr2=0x5577b7a9f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe183832740 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 0x5577b7a9f270 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fe16400c090 tx=0x7fe16400b040 comp rx=0 tx=0).stop 2024-04-26T23:08:28.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe183832740 1 -- 172.21.15.40:0/3334087677 shutdown_connections 2024-04-26T23:08:28.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe183832740 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.40:6832/1811310743,v1:172.21.15.40:6833/1811310743] conn(0x5577b7abbe10 0x5577b7ab77f0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe162a00640 0 client.8568 ms_handle_reset on v2:172.21.15.40:6832/1811310743 2024-04-26T23:08:28.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe183832740 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5577b7abefb0 0x5577b7ab72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe183832740 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5577b7ab6ac0 0x5577b7a9f270 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe183832740 1 --2- 172.21.15.40:0/3334087677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5577b7aa1710 0x5577b7aa4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe183832740 1 -- 172.21.15.40:0/3334087677 >> 172.21.15.40:0/3334087677 conn(0x5577b7a773f0 msgr2=0x5577b7abf460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe183832740 1 -- 172.21.15.40:0/3334087677 shutdown_connections 2024-04-26T23:08:28.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe183832740 1 -- 172.21.15.40:0/3334087677 wait complete. 2024-04-26T23:08:28.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.338+0000 7fe183832740 20 client.8568 trim_cache size 0 max 16384 2024-04-26T23:08:28.342 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:28.342 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-26T23:08:28.342 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:08:28.342 INFO:teuthology.orchestra.run.smithi040.stdout: "b17426f8-9d25-4fbb-a99f-95207d3f0c9f": { 2024-04-26T23:08:28.342 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:08:28.342 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "b17426f8-9d25-4fbb-a99f-95207d3f0c9f", 2024-04-26T23:08:28.342 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:08:28.342 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:08:28.342 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:08:28.342 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:28.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.339+0000 7fe17b200640 1 -- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe15806c8f0 msgr2=0x7fe15806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.339+0000 7fe17b200640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe15806c8f0 0x7fe15806edb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0017e0 tx=0x7fe16c009040 comp rx=0 tx=0).stop 2024-04-26T23:08:28.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.339+0000 7fe17b200640 1 -- 172.21.15.40:0/1505482443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe17c0b2d60 msgr2=0x7fe17c0b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.339+0000 7fe17b200640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe17c0b2d60 0x7fe17c0b31e0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fe17c0bae80 tx=0x7fe17400d890 comp rx=0 tx=0).stop 2024-04-26T23:08:28.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.339+0000 7fe17b200640 1 -- 172.21.15.40:0/1505482443 shutdown_connections 2024-04-26T23:08:28.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.339+0000 7fe17b200640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe15806c8f0 0x7fe15806edb0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.339+0000 7fe17b200640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe17c0b2d60 0x7fe17c0b31e0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.339+0000 7fe17b200640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe17c0cc460 0x7fe17c0ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.339+0000 7fe17b200640 1 --2- 172.21.15.40:0/1505482443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe15c004800 0x7fe17c152800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.339+0000 7fe17b200640 1 -- 172.21.15.40:0/1505482443 >> 172.21.15.40:0/1505482443 conn(0x7fe17c0d4dd0 msgr2=0x7fe17c0d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.339+0000 7fe17b200640 1 -- 172.21.15.40:0/1505482443 shutdown_connections 2024-04-26T23:08:28.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.339+0000 7fe17b200640 1 -- 172.21.15.40:0/1505482443 wait complete. 2024-04-26T23:08:28.356 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:08:28.356 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:08:28.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.6:2 scrub status 2024-04-26T23:08:28.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.544+0000 7f1c72800640 1 Processor -- start 2024-04-26T23:08:28.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.544+0000 7f1c72800640 1 -- start start 2024-04-26T23:08:28.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.546+0000 7f1c72800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c6c0cfaf0 0x7f1c6c073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.546+0000 7f1c72800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1c6c06dd40 0x7f1c6c072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.546+0000 7f1c72800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c6c161880 con 0x7f1c6c160a50 2024-04-26T23:08:28.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.546+0000 7f1c72800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1c6c071ce0 con 0x7f1c6c06dd40 2024-04-26T23:08:28.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.546+0000 7f1c72800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c6c0722a0 con 0x7f1c6c0cfaf0 2024-04-26T23:08:28.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.546+0000 7f1c71400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1c6c160a50 0x7f1c6c160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39964/0 (socket says 172.21.15.40:39964) 2024-04-26T23:08:28.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.546+0000 7f1c71400640 1 -- 172.21.15.40:0/3747062279 learned_addr learned my addr 172.21.15.40:0/3747062279 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:28.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.547+0000 7f1c70a00640 1 --2- 172.21.15.40:0/3747062279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c6c0cfaf0 0x7f1c6c073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.547+0000 7f1c71e00640 1 --2- 172.21.15.40:0/3747062279 >> v2:172.21.15.103:3300/0 conn(0x7f1c6c06dd40 0x7f1c6c072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.547+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 984965144 0 0) 0x7f1c6c161880 con 0x7f1c6c160a50 2024-04-26T23:08:28.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.547+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c54003520 con 0x7f1c6c160a50 2024-04-26T23:08:28.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.547+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 384094982 0 0) 0x7f1c54003520 con 0x7f1c6c160a50 2024-04-26T23:08:28.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.547+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c6c0cfaf0 msgr2=0x7f1c6c073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.547+0000 7f1c6be00640 1 --2- 172.21.15.40:0/3747062279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c6c0cfaf0 0x7f1c6c073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.547+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 >> v2:172.21.15.103:3300/0 conn(0x7f1c6c06dd40 msgr2=0x7f1c6c072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c6be00640 1 --2- 172.21.15.40:0/3747062279 >> v2:172.21.15.103:3300/0 conn(0x7f1c6c06dd40 0x7f1c6c072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c6c0cff80 con 0x7f1c6c160a50 2024-04-26T23:08:28.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f1c60001670 con 0x7f1c6c160a50 2024-04-26T23:08:28.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 >> v1:172.21.15.40:6789/0 conn(0x7f1c6c160a50 legacy=0x7f1c6c160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c6be00640 1 --2- 172.21.15.40:0/3747062279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c540039b0 0x7f1c54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c6be00640 1 --2- 172.21.15.40:0/3747062279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c54004800 0x7f1c54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c6be00640 1 --2- 172.21.15.40:0/3747062279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c54005180 0x7f1c5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c071ce0 con 0x7f1c540039b0 2024-04-26T23:08:28.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c0722a0 con 0x7f1c54005180 2024-04-26T23:08:28.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c5400bf00 con 0x7f1c54004800 2024-04-26T23:08:28.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c71400640 1 --2- 172.21.15.40:0/3747062279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c540039b0 0x7f1c54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c71400640 1 -- 172.21.15.40:0/3747062279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c54004800 msgr2=0x7f1c54004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c71400640 1 --2- 172.21.15.40:0/3747062279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c54004800 0x7f1c54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c71400640 1 -- 172.21.15.40:0/3747062279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c54005180 msgr2=0x7f1c5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c71400640 1 --2- 172.21.15.40:0/3747062279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c54005180 0x7f1c5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.548+0000 7f1c71400640 1 -- 172.21.15.40:0/3747062279 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c54003520 con 0x7f1c540039b0 2024-04-26T23:08:28.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.549+0000 7f1c71400640 1 --2- 172.21.15.40:0/3747062279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c540039b0 0x7f1c54003dd0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f1c6000c190 tx=0x7f1c54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f425c96fea6b9d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.550+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c60007c80 con 0x7f1c540039b0 2024-04-26T23:08:28.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.550+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c60005ad0 con 0x7f1c540039b0 2024-04-26T23:08:28.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.550+0000 7f1c6be00640 1 -- 172.21.15.40:0/3747062279 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c60007070 con 0x7f1c540039b0 2024-04-26T23:08:28.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.550+0000 7f1c72800640 1 -- 172.21.15.40:0/3747062279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c540039b0 msgr2=0x7f1c54003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.550+0000 7f1c72800640 1 --2- 172.21.15.40:0/3747062279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c540039b0 0x7f1c54003dd0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f1c6000c190 tx=0x7f1c54005830 comp rx=0 tx=0).stop 2024-04-26T23:08:28.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.550+0000 7f1c71e00640 1 -- 172.21.15.40:0/3747062279 reap_dead start 2024-04-26T23:08:28.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.550+0000 7f1c72800640 1 -- 172.21.15.40:0/3747062279 shutdown_connections 2024-04-26T23:08:28.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.550+0000 7f1c72800640 1 -- 172.21.15.40:0/3747062279 >> 172.21.15.40:0/3747062279 conn(0x7f1c6c0b2720 msgr2=0x7f1c6c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.551+0000 7f1c72800640 1 -- 172.21.15.40:0/3747062279 shutdown_connections 2024-04-26T23:08:28.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.551+0000 7f1c72800640 1 -- 172.21.15.40:0/3747062279 wait complete. 2024-04-26T23:08:28.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.552+0000 7f1c72800640 1 Processor -- start 2024-04-26T23:08:28.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.553+0000 7f1c72800640 1 -- start start 2024-04-26T23:08:28.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.553+0000 7f1c72800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c540039b0 0x7f1c6c15b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.553+0000 7f1c72800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c6c15c790 0x7f1c6c13bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.553+0000 7f1c72800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c6c13c090 0x7f1c6c13da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.553+0000 7f1c72800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c06e1f0 con 0x7f1c6c15c790 2024-04-26T23:08:28.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.553+0000 7f1c72800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c071f30 con 0x7f1c540039b0 2024-04-26T23:08:28.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.554+0000 7f1c72800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c6c1468f0 con 0x7f1c6c13c090 2024-04-26T23:08:28.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.554+0000 7f1c70a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c6c15c790 0x7f1c6c13bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.554+0000 7f1c70a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c6c15c790 0x7f1c6c13bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52984/0 (socket says 172.21.15.40:52984) 2024-04-26T23:08:28.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.554+0000 7f1c70a00640 1 -- 172.21.15.40:0/3871200712 learned_addr learned my addr 172.21.15.40:0/3871200712 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.554+0000 7f1c71e00640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c6c13c090 0x7f1c6c13da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.554+0000 7f1c70a00640 1 -- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c6c13c090 msgr2=0x7f1c6c13da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.554+0000 7f1c70a00640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c6c13c090 0x7f1c6c13da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.554+0000 7f1c71400640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c540039b0 0x7f1c6c15b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.554+0000 7f1c70a00640 1 -- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c540039b0 msgr2=0x7f1c6c15b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.554+0000 7f1c70a00640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c540039b0 0x7f1c6c15b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.554+0000 7f1c70a00640 1 -- 172.21.15.40:0/3871200712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c6c15b730 con 0x7f1c6c15c790 2024-04-26T23:08:28.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.555+0000 7f1c70a00640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c6c15c790 0x7f1c6c13bb50 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f1c5800e7c0 tx=0x7f1c5800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.555+0000 7f1c69600640 1 -- 172.21.15.40:0/3871200712 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58017070 con 0x7f1c6c15c790 2024-04-26T23:08:28.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.555+0000 7f1c69600640 1 -- 172.21.15.40:0/3871200712 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c5800c940 con 0x7f1c6c15c790 2024-04-26T23:08:28.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.555+0000 7f1c69600640 1 -- 172.21.15.40:0/3871200712 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5800cd10 con 0x7f1c6c15c790 2024-04-26T23:08:28.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.555+0000 7f1c72800640 1 -- 172.21.15.40:0/3871200712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c6c15b9d0 con 0x7f1c6c15c790 2024-04-26T23:08:28.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.555+0000 7f1c72800640 1 -- 172.21.15.40:0/3871200712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c6c15bf90 con 0x7f1c6c15c790 2024-04-26T23:08:28.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.557+0000 7f1c69600640 1 -- 172.21.15.40:0/3871200712 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1c5801a070 con 0x7f1c6c15c790 2024-04-26T23:08:28.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.558+0000 7f1c69600640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c4c06c920 0x7f1c4c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.558+0000 7f1c69600640 1 -- 172.21.15.40:0/3871200712 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f1c5808bb70 con 0x7f1c6c15c790 2024-04-26T23:08:28.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.559+0000 7f1c76c4e740 1 Processor -- start 2024-04-26T23:08:28.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.559+0000 7f1c71400640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c4c06c920 0x7f1c4c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.559+0000 7f1c76c4e740 1 -- start start 2024-04-26T23:08:28.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.560+0000 7f1c76c4e740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648e1710 0x557e648e1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.560+0000 7f1c76c4e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648fefb0 0x557e648ff500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.560+0000 7f1c76c4e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648b73f0 0x557e648fd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.560+0000 7f1c76c4e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557e647c7030 con 0x557e648b73f0 2024-04-26T23:08:28.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.560+0000 7f1c76c4e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557e647e27f0 con 0x557e648fefb0 2024-04-26T23:08:28.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.560+0000 7f1c76c4e740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x557e6479d7e0 con 0x557e648e1710 2024-04-26T23:08:28.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.561+0000 7f1c71400640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c4c06c920 0x7f1c4c06ede0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1c60032000 tx=0x7f1c60009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.561+0000 7f1c71e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648b73f0 0x557e648fd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.561+0000 7f1c71e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648b73f0 0x557e648fd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52990/0 (socket says 172.21.15.40:52990) 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.561+0000 7f1c71e00640 1 -- 172.21.15.40:0/2923799462 learned_addr learned my addr 172.21.15.40:0/2923799462 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.561+0000 7f1c70a00640 1 --2- 172.21.15.40:0/2923799462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648fefb0 0x557e648ff500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.561+0000 7f1c71400640 1 --2- 172.21.15.40:0/2923799462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648e1710 0x557e648e1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.561+0000 7f1c71e00640 1 -- 172.21.15.40:0/2923799462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648e1710 msgr2=0x557e648e1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.561+0000 7f1c71e00640 1 --2- 172.21.15.40:0/2923799462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648e1710 0x557e648e1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.561+0000 7f1c71e00640 1 -- 172.21.15.40:0/2923799462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648fefb0 msgr2=0x557e648ff500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.561+0000 7f1c71e00640 1 --2- 172.21.15.40:0/2923799462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648fefb0 0x557e648ff500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.561+0000 7f1c71e00640 1 -- 172.21.15.40:0/2923799462 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e647a9050 con 0x557e648b73f0 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.561+0000 7f1c70a00640 1 --2- 172.21.15.40:0/2923799462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648fefb0 0x557e648ff500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.562+0000 7f1c71e00640 1 --2- 172.21.15.40:0/2923799462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648b73f0 0x557e648fd670 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c146cc0 tx=0x7f1c6400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860bd0cfcb46df85 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.562+0000 7f1c5dc00640 1 -- 172.21.15.40:0/2923799462 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c64015600 con 0x557e648b73f0 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.562+0000 7f1c5dc00640 1 -- 172.21.15.40:0/2923799462 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c6401a070 con 0x557e648b73f0 2024-04-26T23:08:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.562+0000 7f1c5dc00640 1 -- 172.21.15.40:0/2923799462 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c640159e0 con 0x557e648b73f0 2024-04-26T23:08:28.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.562+0000 7f1c76c4e740 1 -- 172.21.15.40:0/2923799462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648b73f0 msgr2=0x557e648fd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.562+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/2923799462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648b73f0 0x557e648fd670 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c146cc0 tx=0x7f1c6400b040 comp rx=0 tx=0).stop 2024-04-26T23:08:28.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.563+0000 7f1c76c4e740 1 -- 172.21.15.40:0/2923799462 shutdown_connections 2024-04-26T23:08:28.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.563+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/2923799462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648b73f0 0x557e648fd670 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.563+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/2923799462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648fefb0 0x557e648ff500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.563+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/2923799462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648e1710 0x557e648e1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.563+0000 7f1c76c4e740 1 -- 172.21.15.40:0/2923799462 >> 172.21.15.40:0/2923799462 conn(0x557e648b3d20 msgr2=0x557e648fc520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.563+0000 7f1c76c4e740 1 -- 172.21.15.40:0/2923799462 shutdown_connections 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.563+0000 7f1c76c4e740 1 -- 172.21.15.40:0/2923799462 wait complete. 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.563+0000 7f1c76c4e740 1 -- start start 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.564+0000 7f1c76c4e740 1 Processor -- start 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c76c4e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648e1710 0x557e648f4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c76c4e740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648fefb0 0x557e648f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c76c4e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648ffaa0 0x557e648fff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c76c4e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557e6481f200 con 0x557e648e1710 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c76c4e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557e64819ee0 con 0x557e648ffaa0 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c76c4e740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x557e64826e70 con 0x557e648fefb0 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c76c4e740 10 client.0 _ll_register_callbacks cb 0x557e64810270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c71400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648e1710 0x557e648f4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c71400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648e1710 0x557e648f4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52994/0 (socket says 172.21.15.40:52994) 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c71400640 1 -- 172.21.15.40:0/2542824607 learned_addr learned my addr 172.21.15.40:0/2542824607 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c70a00640 1 --2- 172.21.15.40:0/2542824607 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648fefb0 0x557e648f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c71400640 1 -- 172.21.15.40:0/2542824607 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648fefb0 msgr2=0x557e648f3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c71400640 1 --2- 172.21.15.40:0/2542824607 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648fefb0 0x557e648f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c71400640 1 -- 172.21.15.40:0/2542824607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648ffaa0 msgr2=0x557e648fff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c71400640 1 --2- 172.21.15.40:0/2542824607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648ffaa0 0x557e648fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.565+0000 7f1c71400640 1 -- 172.21.15.40:0/2542824607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e645f67d0 con 0x557e648e1710 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.566+0000 7f1c71400640 1 --2- 172.21.15.40:0/2542824607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648e1710 0x557e648f4880 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f1c60001600 tx=0x7f1c600019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7640b6b94c70f0f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.566+0000 7f1c52000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.566+0000 7f1c52000640 1 -- 172.21.15.40:0/2542824607 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c60041030 con 0x557e648e1710 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.566+0000 7f1c52000640 1 -- 172.21.15.40:0/2542824607 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f1c600075f0 con 0x557e648e1710 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.566+0000 7f1c76c4e740 1 -- 172.21.15.40:0/2542824607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c6401a070 con 0x557e648e1710 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.566+0000 7f1c52000640 1 -- 172.21.15.40:0/2542824607 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f1c60048430 con 0x557e648e1710 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.566+0000 7f1c76c4e740 10 client.8580 fetch_fsmap learned FSMap version 102 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.566+0000 7f1c76c4e740 1 -- 172.21.15.40:0/2542824607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) -- 0x557e6452c3f0 con 0x557e648e1710 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.567+0000 7f1c52000640 1 -- 172.21.15.40:0/2542824607 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c6401a070 con 0x557e648e1710 2024-04-26T23:08:28.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.567+0000 7f1c52000640 1 -- 172.21.15.40:0/2542824607 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 102) ==== 3550+0+0 (secure 0 0 0) 0x7f1c60058080 con 0x557e648e1710 2024-04-26T23:08:28.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.567+0000 7f1c76c4e740 10 client.8580 fetch_fsmap finished waiting for FSMap version 102 2024-04-26T23:08:28.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.567+0000 7f1c76c4e740 10 client.8580 resolve_mds: resolved 6:2 to role '6:2' aka daemon mds.d 2024-04-26T23:08:28.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.567+0000 7f1c76c4e740 20 client.8580 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:28.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.567+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/2542824607 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x557e648cf1e0 0x557e648cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.567+0000 7f1c76c4e740 4 client.8580 mds_command: new command op to 6253 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:28.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.567+0000 7f1c76c4e740 1 -- 172.21.15.40:0/2542824607 --> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e648287e0 con 0x557e648cf1e0 2024-04-26T23:08:28.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.568+0000 7f1c70a00640 1 --2- 172.21.15.40:0/2542824607 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x557e648cf1e0 0x557e648cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.568+0000 7f1c70a00640 1 --2- 172.21.15.40:0/2542824607 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x557e648cf1e0 0x557e648cf5c0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5278715bf8d433c9 server_cookie=c477543cda9ca900 in_seq=0 out_seq=0 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.568+0000 7f1c52000640 10 client.8580 ms_handle_connect on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.570+0000 7f1c52000640 1 -- 172.21.15.40:0/2542824607 <== mds.2 v2:172.21.15.103:6836/2728939257 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557e648287e0 con 0x557e648cf1e0 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.570+0000 7f1c52000640 10 client.8580 handle_command_reply: tid=0 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.570+0000 7f1c76c4e740 1 client.8580 shutdown 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.571+0000 7f1c76c4e740 1 -- 172.21.15.40:0/2542824607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648e1710 msgr2=0x557e648f4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.571+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/2542824607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648e1710 0x557e648f4880 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f1c60001600 tx=0x7f1c600019e0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.571+0000 7f1c76c4e740 1 -- 172.21.15.40:0/2542824607 shutdown_connections 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.571+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/2542824607 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x557e648cf1e0 0x557e648cf5c0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.571+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/2542824607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648ffaa0 0x557e648fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.571+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/2542824607 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648fefb0 0x557e648f3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.571+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/2542824607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648e1710 0x557e648f4880 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.571+0000 7f1c52000640 0 client.8580 ms_handle_reset on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.571+0000 7f1c76c4e740 1 -- 172.21.15.40:0/2542824607 >> 172.21.15.40:0/2542824607 conn(0x557e648b73f0 msgr2=0x557e648b0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.571+0000 7f1c76c4e740 1 -- 172.21.15.40:0/2542824607 shutdown_connections 2024-04-26T23:08:28.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.571+0000 7f1c76c4e740 1 -- 172.21.15.40:0/2542824607 wait complete. 2024-04-26T23:08:28.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.571+0000 7f1c76c4e740 20 client.8580 trim_cache size 0 max 16384 2024-04-26T23:08:28.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.590+0000 7f1c76c4e740 1 Processor -- start 2024-04-26T23:08:28.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.590+0000 7f1c76c4e740 1 -- start start 2024-04-26T23:08:28.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.591+0000 7f1c76c4e740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648e1710 0x557e648fd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.591+0000 7f1c76c4e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648fefb0 0x557e648fac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.591+0000 7f1c76c4e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648ffaa0 0x557e648cd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.591+0000 7f1c76c4e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c2020f0 con 0x557e648ffaa0 2024-04-26T23:08:28.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.591+0000 7f1c76c4e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c201290 con 0x557e648fefb0 2024-04-26T23:08:28.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.591+0000 7f1c76c4e740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c6c2019c0 con 0x557e648e1710 2024-04-26T23:08:28.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.591+0000 7f1c71e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648ffaa0 0x557e648cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.592+0000 7f1c71e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648ffaa0 0x557e648cd580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52998/0 (socket says 172.21.15.40:52998) 2024-04-26T23:08:28.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.592+0000 7f1c71e00640 1 -- 172.21.15.40:0/4083741930 learned_addr learned my addr 172.21.15.40:0/4083741930 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.592+0000 7f1c70a00640 1 --2- 172.21.15.40:0/4083741930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648fefb0 0x557e648fac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.592+0000 7f1c71e00640 1 -- 172.21.15.40:0/4083741930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648e1710 msgr2=0x557e648fd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.592+0000 7f1c71e00640 1 --2- 172.21.15.40:0/4083741930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648e1710 0x557e648fd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.592+0000 7f1c71e00640 1 -- 172.21.15.40:0/4083741930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648fefb0 msgr2=0x557e648fac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.592+0000 7f1c71e00640 1 --2- 172.21.15.40:0/4083741930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648fefb0 0x557e648fac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.592+0000 7f1c71e00640 1 -- 172.21.15.40:0/4083741930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e644ec0f0 con 0x557e648ffaa0 2024-04-26T23:08:28.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.592+0000 7f1c71e00640 1 --2- 172.21.15.40:0/4083741930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648ffaa0 0x557e648cd580 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f1c64009320 tx=0x7f1c64004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5a962eeb0adefe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.593+0000 7f1c5d200640 1 -- 172.21.15.40:0/4083741930 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c64023070 con 0x557e648ffaa0 2024-04-26T23:08:28.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.593+0000 7f1c5d200640 1 -- 172.21.15.40:0/4083741930 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c6401a070 con 0x557e648ffaa0 2024-04-26T23:08:28.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.593+0000 7f1c5d200640 1 -- 172.21.15.40:0/4083741930 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c640158a0 con 0x557e648ffaa0 2024-04-26T23:08:28.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.593+0000 7f1c76c4e740 1 -- 172.21.15.40:0/4083741930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648ffaa0 msgr2=0x557e648cd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.593+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/4083741930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648ffaa0 0x557e648cd580 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f1c64009320 tx=0x7f1c64004ab0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.593+0000 7f1c76c4e740 1 -- 172.21.15.40:0/4083741930 shutdown_connections 2024-04-26T23:08:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.593+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/4083741930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648ffaa0 0x557e648cd580 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.593+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/4083741930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648fefb0 0x557e648fac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.593+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/4083741930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648e1710 0x557e648fd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.593+0000 7f1c76c4e740 1 -- 172.21.15.40:0/4083741930 >> 172.21.15.40:0/4083741930 conn(0x557e648b73f0 msgr2=0x557e648f56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.593+0000 7f1c76c4e740 1 -- 172.21.15.40:0/4083741930 shutdown_connections 2024-04-26T23:08:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.593+0000 7f1c76c4e740 1 -- 172.21.15.40:0/4083741930 wait complete. 2024-04-26T23:08:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.594+0000 7f1c76c4e740 1 -- start start 2024-04-26T23:08:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.594+0000 7f1c76c4e740 1 Processor -- start 2024-04-26T23:08:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.595+0000 7f1c76c4e740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648e1710 0x557e6495ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.595+0000 7f1c76c4e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648fefb0 0x557e648f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.595+0000 7f1c76c4e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648ffaa0 0x557e648e43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.595+0000 7f1c76c4e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557e64827fd0 con 0x557e648fefb0 2024-04-26T23:08:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.595+0000 7f1c76c4e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c2003d0 con 0x557e648ffaa0 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.595+0000 7f1c76c4e740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x557e648fc170 con 0x557e648e1710 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.595+0000 7f1c76c4e740 10 client.0 _ll_register_callbacks cb 0x7f1c540036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.595+0000 7f1c70a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648fefb0 0x557e648f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.595+0000 7f1c70a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648fefb0 0x557e648f4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53004/0 (socket says 172.21.15.40:53004) 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.595+0000 7f1c70a00640 1 -- 172.21.15.40:0/1576627202 learned_addr learned my addr 172.21.15.40:0/1576627202 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.596+0000 7f1c70a00640 1 -- 172.21.15.40:0/1576627202 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648e1710 msgr2=0x557e6495ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.596+0000 7f1c70a00640 1 --2- 172.21.15.40:0/1576627202 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648e1710 0x557e6495ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.596+0000 7f1c70a00640 1 -- 172.21.15.40:0/1576627202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648ffaa0 msgr2=0x557e648e43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.596+0000 7f1c70a00640 1 --2- 172.21.15.40:0/1576627202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648ffaa0 0x557e648e43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.596+0000 7f1c70a00640 1 -- 172.21.15.40:0/1576627202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e647e2a30 con 0x557e648fefb0 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.596+0000 7f1c70a00640 1 --2- 172.21.15.40:0/1576627202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648fefb0 0x557e648f4a20 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f1c5800c520 tx=0x7f1c58009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e560d5a2dc8761 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.596+0000 7f1c52000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.596+0000 7f1c52000640 1 -- 172.21.15.40:0/1576627202 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58075020 con 0x557e648fefb0 2024-04-26T23:08:28.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.596+0000 7f1c52000640 1 -- 172.21.15.40:0/1576627202 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f1c58055d70 con 0x557e648fefb0 2024-04-26T23:08:28.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.596+0000 7f1c76c4e740 1 -- 172.21.15.40:0/1576627202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c6401a070 con 0x557e648fefb0 2024-04-26T23:08:28.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.597+0000 7f1c52000640 1 -- 172.21.15.40:0/1576627202 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f1c58021040 con 0x557e648fefb0 2024-04-26T23:08:28.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.597+0000 7f1c76c4e740 10 client.8586 fetch_fsmap learned FSMap version 102 2024-04-26T23:08:28.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.597+0000 7f1c76c4e740 1 -- 172.21.15.40:0/1576627202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) -- 0x557e648ad190 con 0x557e648fefb0 2024-04-26T23:08:28.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.597+0000 7f1c52000640 1 -- 172.21.15.40:0/1576627202 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c6401a070 con 0x557e648fefb0 2024-04-26T23:08:28.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.597+0000 7f1c52000640 1 -- 172.21.15.40:0/1576627202 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 102) ==== 3550+0+0 (secure 0 0 0) 0x7f1c58024060 con 0x557e648fefb0 2024-04-26T23:08:28.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.597+0000 7f1c76c4e740 10 client.8586 fetch_fsmap finished waiting for FSMap version 102 2024-04-26T23:08:28.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.597+0000 7f1c76c4e740 10 client.8586 resolve_mds: resolved 6:2 to role '6:2' aka daemon mds.d 2024-04-26T23:08:28.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.597+0000 7f1c76c4e740 20 client.8586 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:28.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.597+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/1576627202 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x557e648e4920 0x557e648fc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.597+0000 7f1c76c4e740 4 client.8586 mds_command: new command op to 6253 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:08:28.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.597+0000 7f1c76c4e740 1 -- 172.21.15.40:0/1576627202 --> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557e64822650 con 0x557e648e4920 2024-04-26T23:08:28.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.598+0000 7f1c71400640 1 --2- 172.21.15.40:0/1576627202 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x557e648e4920 0x557e648fc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.599+0000 7f1c71400640 1 --2- 172.21.15.40:0/1576627202 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x557e648e4920 0x557e648fc7b0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=34e39c7dc2e06ff9 server_cookie=9508589ad9c35238 in_seq=0 out_seq=0 2024-04-26T23:08:28.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.599+0000 7f1c52000640 10 client.8586 ms_handle_connect on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:28.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.599+0000 7f1c52000640 1 -- 172.21.15.40:0/1576627202 <== mds.2 v2:172.21.15.103:6836/2728939257 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x557e64822650 con 0x557e648e4920 2024-04-26T23:08:28.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c52000640 10 client.8586 handle_command_reply: tid=0 2024-04-26T23:08:28.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c76c4e740 1 client.8586 shutdown 2024-04-26T23:08:28.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c76c4e740 1 -- 172.21.15.40:0/1576627202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648fefb0 msgr2=0x557e648f4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/1576627202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648fefb0 0x557e648f4a20 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f1c5800c520 tx=0x7f1c58009190 comp rx=0 tx=0).stop 2024-04-26T23:08:28.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c76c4e740 1 -- 172.21.15.40:0/1576627202 shutdown_connections 2024-04-26T23:08:28.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/1576627202 >> [v2:172.21.15.103:6836/2728939257,v1:172.21.15.103:6837/2728939257] conn(0x557e648e4920 0x557e648fc7b0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/1576627202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e648ffaa0 0x557e648e43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/1576627202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e648fefb0 0x557e648f4a20 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c52000640 0 client.8586 ms_handle_reset on v2:172.21.15.103:6836/2728939257 2024-04-26T23:08:28.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c76c4e740 1 --2- 172.21.15.40:0/1576627202 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x557e648e1710 0x557e6495ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c76c4e740 1 -- 172.21.15.40:0/1576627202 >> 172.21.15.40:0/1576627202 conn(0x557e648b73f0 msgr2=0x557e648fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c76c4e740 1 -- 172.21.15.40:0/1576627202 shutdown_connections 2024-04-26T23:08:28.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c76c4e740 1 -- 172.21.15.40:0/1576627202 wait complete. 2024-04-26T23:08:28.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.600+0000 7f1c76c4e740 20 client.8586 trim_cache size 0 max 16384 2024-04-26T23:08:28.603 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:28.603 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-26T23:08:28.603 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:08:28.604 INFO:teuthology.orchestra.run.smithi040.stdout: "b17426f8-9d25-4fbb-a99f-95207d3f0c9f": { 2024-04-26T23:08:28.604 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:08:28.604 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "b17426f8-9d25-4fbb-a99f-95207d3f0c9f", 2024-04-26T23:08:28.604 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:08:28.604 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:08:28.604 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:08:28.604 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.601+0000 7f1c72800640 1 -- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c4c06c920 msgr2=0x7f1c4c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.601+0000 7f1c72800640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c4c06c920 0x7f1c4c06ede0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1c60032000 tx=0x7f1c60009030 comp rx=0 tx=0).stop 2024-04-26T23:08:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.601+0000 7f1c72800640 1 -- 172.21.15.40:0/3871200712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c6c15c790 msgr2=0x7f1c6c13bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.601+0000 7f1c72800640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c6c15c790 0x7f1c6c13bb50 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f1c5800e7c0 tx=0x7f1c5800ec80 comp rx=0 tx=0).stop 2024-04-26T23:08:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.601+0000 7f1c72800640 1 -- 172.21.15.40:0/3871200712 shutdown_connections 2024-04-26T23:08:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.601+0000 7f1c72800640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c4c06c920 0x7f1c4c06ede0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.601+0000 7f1c72800640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c6c13c090 0x7f1c6c13da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.601+0000 7f1c72800640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c6c15c790 0x7f1c6c13bb50 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.601+0000 7f1c72800640 1 --2- 172.21.15.40:0/3871200712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c540039b0 0x7f1c6c15b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.601+0000 7f1c72800640 1 -- 172.21.15.40:0/3871200712 >> 172.21.15.40:0/3871200712 conn(0x7f1c6c0b2720 msgr2=0x7f1c6c15e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.601+0000 7f1c72800640 1 -- 172.21.15.40:0/3871200712 shutdown_connections 2024-04-26T23:08:28.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.601+0000 7f1c72800640 1 -- 172.21.15.40:0/3871200712 wait complete. 2024-04-26T23:08:28.618 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:08:28.618 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:08:28.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-26T23:08:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.830+0000 7f1e79000640 1 Processor -- start 2024-04-26T23:08:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.831+0000 7f1e79000640 1 -- start start 2024-04-26T23:08:28.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.832+0000 7f1e79000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1e740b52b0 0x7f1e7406c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.832+0000 7f1e79000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e74163620 0x7f1e7406f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.832+0000 7f1e79000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e7406ce50 con 0x7f1e74145710 2024-04-26T23:08:28.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.833+0000 7f1e79000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1e7406f1b0 con 0x7f1e74163620 2024-04-26T23:08:28.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.833+0000 7f1e79000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1e74145af0 con 0x7f1e740b52b0 2024-04-26T23:08:28.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.833+0000 7f1e73e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e74163620 0x7f1e7406f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.833+0000 7f1e73e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e74163620 0x7f1e7406f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53014/0 (socket says 172.21.15.40:53014) 2024-04-26T23:08:28.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.833+0000 7f1e73e00640 1 -- 172.21.15.40:0/3087862820 learned_addr learned my addr 172.21.15.40:0/3087862820 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.833+0000 7f1e6be00640 1 --2- 172.21.15.40:0/3087862820 >> v2:172.21.15.103:3300/0 conn(0x7f1e740b52b0 0x7f1e7406c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.833+0000 7f1e73e00640 1 -- 172.21.15.40:0/3087862820 >> v2:172.21.15.103:3300/0 conn(0x7f1e740b52b0 msgr2=0x7f1e7406c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.833+0000 7f1e73e00640 1 --2- 172.21.15.40:0/3087862820 >> v2:172.21.15.103:3300/0 conn(0x7f1e740b52b0 0x7f1e7406c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.833+0000 7f1e73e00640 1 -- 172.21.15.40:0/3087862820 >> v1:172.21.15.103:6789/0 conn(0x7f1e74145710 legacy=0x7f1e740b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.833+0000 7f1e73e00640 1 -- 172.21.15.40:0/3087862820 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e7406c890 con 0x7f1e74163620 2024-04-26T23:08:28.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.835+0000 7f1e73e00640 1 --2- 172.21.15.40:0/3087862820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e74163620 0x7f1e7406f640 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f1e74162b70 tx=0x7f1e6c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad591144afd40db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.835+0000 7f1e72a00640 1 -- 172.21.15.40:0/3087862820 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c036070 con 0x7f1e74163620 2024-04-26T23:08:28.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.835+0000 7f1e72a00640 1 -- 172.21.15.40:0/3087862820 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6c031600 con 0x7f1e74163620 2024-04-26T23:08:28.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.835+0000 7f1e72a00640 1 -- 172.21.15.40:0/3087862820 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c0319d0 con 0x7f1e74163620 2024-04-26T23:08:28.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.836+0000 7f1e79000640 1 -- 172.21.15.40:0/3087862820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e74163620 msgr2=0x7f1e7406f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.836+0000 7f1e79000640 1 --2- 172.21.15.40:0/3087862820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e74163620 0x7f1e7406f640 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f1e74162b70 tx=0x7f1e6c02a660 comp rx=0 tx=0).stop 2024-04-26T23:08:28.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.836+0000 7f1e79000640 1 -- 172.21.15.40:0/3087862820 shutdown_connections 2024-04-26T23:08:28.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.836+0000 7f1e79000640 1 --2- 172.21.15.40:0/3087862820 >> v2:172.21.15.103:3300/0 conn(0x7f1e740b52b0 0x7f1e7406c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.836+0000 7f1e79000640 1 --2- 172.21.15.40:0/3087862820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e74163620 0x7f1e7406f640 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.836+0000 7f1e79000640 1 -- 172.21.15.40:0/3087862820 >> 172.21.15.40:0/3087862820 conn(0x7f1e740b38a0 msgr2=0x7f1e740b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.836+0000 7f1e79000640 1 -- 172.21.15.40:0/3087862820 shutdown_connections 2024-04-26T23:08:28.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.836+0000 7f1e79000640 1 -- 172.21.15.40:0/3087862820 wait complete. 2024-04-26T23:08:28.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.838+0000 7f1e79000640 1 Processor -- start 2024-04-26T23:08:28.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.838+0000 7f1e79000640 1 -- start start 2024-04-26T23:08:28.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e79000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e740b52b0 0x7f1e74142190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e79000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e74145710 0x7f1e741e95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e79000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e74163620 0x7f1e740af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e79000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1e74163ab0 con 0x7f1e74163620 2024-04-26T23:08:28.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e79000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7406ee40 con 0x7f1e74145710 2024-04-26T23:08:28.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e79000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1e7406fb80 con 0x7f1e740b52b0 2024-04-26T23:08:28.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e73e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e74163620 0x7f1e740af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e73e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e74163620 0x7f1e740af910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53022/0 (socket says 172.21.15.40:53022) 2024-04-26T23:08:28.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e73e00640 1 -- 172.21.15.40:0/3722336324 learned_addr learned my addr 172.21.15.40:0/3722336324 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e73400640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e740b52b0 0x7f1e74142190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e6be00640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e74145710 0x7f1e741e95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e73e00640 1 -- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e740b52b0 msgr2=0x7f1e74142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e73e00640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e740b52b0 0x7f1e74142190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e73e00640 1 -- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e74145710 msgr2=0x7f1e741e95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e73e00640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e74145710 0x7f1e741e95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.840+0000 7f1e73e00640 1 -- 172.21.15.40:0/3722336324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e741426d0 con 0x7f1e74163620 2024-04-26T23:08:28.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.841+0000 7f1e73e00640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e74163620 0x7f1e740af910 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c02ac50 tx=0x7f1e6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.841+0000 7f1e6b400640 1 -- 172.21.15.40:0/3722336324 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c041030 con 0x7f1e74163620 2024-04-26T23:08:28.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.841+0000 7f1e6b400640 1 -- 172.21.15.40:0/3722336324 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6c031400 con 0x7f1e74163620 2024-04-26T23:08:28.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.841+0000 7f1e6b400640 1 -- 172.21.15.40:0/3722336324 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c036040 con 0x7f1e74163620 2024-04-26T23:08:28.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.841+0000 7f1e79000640 1 -- 172.21.15.40:0/3722336324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e741f4d30 con 0x7f1e74163620 2024-04-26T23:08:28.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.841+0000 7f1e79000640 1 -- 172.21.15.40:0/3722336324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e740d0b60 con 0x7f1e74163620 2024-04-26T23:08:28.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.844+0000 7f1e6b400640 1 -- 172.21.15.40:0/3722336324 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1e6c044070 con 0x7f1e74163620 2024-04-26T23:08:28.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.845+0000 7f1e6b400640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e5006c8f0 0x7f1e5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.845+0000 7f1e6b400640 1 -- 172.21.15.40:0/3722336324 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f1e6c0ac750 con 0x7f1e74163620 2024-04-26T23:08:28.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.845+0000 7f1e7d365740 1 Processor -- start 2024-04-26T23:08:28.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.845+0000 7f1e7d365740 1 -- start start 2024-04-26T23:08:28.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.846+0000 7f1e73400640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e5006c8f0 0x7f1e5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.846+0000 7f1e7d365740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf73710 0x558eabf73af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.846+0000 7f1e7d365740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf90fb0 0x558eabf91500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.846+0000 7f1e7d365740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf493f0 0x558eabf8f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.846+0000 7f1e7d365740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558eabe59030 con 0x558eabf73710 2024-04-26T23:08:28.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.846+0000 7f1e7d365740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558eabe747f0 con 0x558eabf90fb0 2024-04-26T23:08:28.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.846+0000 7f1e7d365740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558eabe2f7e0 con 0x558eabf493f0 2024-04-26T23:08:28.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.847+0000 7f1e73400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf73710 0x558eabf73af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.847+0000 7f1e73400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf73710 0x558eabf73af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53032/0 (socket says 172.21.15.40:53032) 2024-04-26T23:08:28.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.847+0000 7f1e73400640 1 -- 172.21.15.40:0/4149111028 learned_addr learned my addr 172.21.15.40:0/4149111028 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.847+0000 7f1e73400640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e5006c8f0 0x7f1e5006edb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1e741eae60 tx=0x7f1e64009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.847+0000 7f1e6be00640 1 --2- 172.21.15.40:0/4149111028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf90fb0 0x558eabf91500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.847+0000 7f1e73e00640 1 --2- 172.21.15.40:0/4149111028 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf493f0 0x558eabf8f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.847+0000 7f1e73400640 1 -- 172.21.15.40:0/4149111028 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf493f0 msgr2=0x558eabf8f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.847+0000 7f1e73400640 1 --2- 172.21.15.40:0/4149111028 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf493f0 0x558eabf8f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.847+0000 7f1e73400640 1 -- 172.21.15.40:0/4149111028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf90fb0 msgr2=0x558eabf91500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.847+0000 7f1e73400640 1 --2- 172.21.15.40:0/4149111028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf90fb0 0x558eabf91500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.847+0000 7f1e73400640 1 -- 172.21.15.40:0/4149111028 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eabe3b050 con 0x558eabf73710 2024-04-26T23:08:28.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.848+0000 7f1e73400640 1 --2- 172.21.15.40:0/4149111028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf73710 0x558eabf73af0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400e830 tx=0x7f1e64011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=217ec3c68cd3d089 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.848+0000 7f1e63e00640 1 -- 172.21.15.40:0/4149111028 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e640159b0 con 0x558eabf73710 2024-04-26T23:08:28.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.848+0000 7f1e63e00640 1 -- 172.21.15.40:0/4149111028 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e64015b50 con 0x558eabf73710 2024-04-26T23:08:28.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.848+0000 7f1e63e00640 1 -- 172.21.15.40:0/4149111028 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e64007020 con 0x558eabf73710 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.848+0000 7f1e7d365740 1 -- 172.21.15.40:0/4149111028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf73710 msgr2=0x558eabf73af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.848+0000 7f1e7d365740 1 --2- 172.21.15.40:0/4149111028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf73710 0x558eabf73af0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400e830 tx=0x7f1e64011000 comp rx=0 tx=0).stop 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.849+0000 7f1e7d365740 1 -- 172.21.15.40:0/4149111028 shutdown_connections 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.849+0000 7f1e7d365740 1 --2- 172.21.15.40:0/4149111028 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf493f0 0x558eabf8f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.849+0000 7f1e7d365740 1 --2- 172.21.15.40:0/4149111028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf90fb0 0x558eabf91500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.849+0000 7f1e7d365740 1 --2- 172.21.15.40:0/4149111028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf73710 0x558eabf73af0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.849+0000 7f1e7d365740 1 -- 172.21.15.40:0/4149111028 >> 172.21.15.40:0/4149111028 conn(0x558eabf45d20 msgr2=0x558eabf8e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.849+0000 7f1e7d365740 1 -- 172.21.15.40:0/4149111028 shutdown_connections 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.849+0000 7f1e7d365740 1 -- 172.21.15.40:0/4149111028 wait complete. 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.849+0000 7f1e7d365740 1 -- start start 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.850+0000 7f1e7d365740 1 Processor -- start 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e7d365740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf73710 0x558eabfa13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e7d365740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf90fb0 0x558eabf75ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e7d365740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 0x558eabf8e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e7d365740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558eabeb1200 con 0x558eabf8dcc0 2024-04-26T23:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e7d365740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558eabeabee0 con 0x558eabf73710 2024-04-26T23:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e7d365740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558eabeb8e70 con 0x558eabf90fb0 2024-04-26T23:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e7d365740 10 client.0 _ll_register_callbacks cb 0x558eabea2270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e73e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 0x558eabf8e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e73e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 0x558eabf8e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53042/0 (socket says 172.21.15.40:53042) 2024-04-26T23:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e73e00640 1 -- 172.21.15.40:0/2244593672 learned_addr learned my addr 172.21.15.40:0/2244593672 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e6be00640 1 --2- 172.21.15.40:0/2244593672 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf90fb0 0x558eabf75ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e73e00640 1 -- 172.21.15.40:0/2244593672 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf90fb0 msgr2=0x558eabf75ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e73e00640 1 --2- 172.21.15.40:0/2244593672 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf90fb0 0x558eabf75ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e73e00640 1 -- 172.21.15.40:0/2244593672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf73710 msgr2=0x558eabfa13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e73e00640 1 --2- 172.21.15.40:0/2244593672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf73710 0x558eabfa13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.851+0000 7f1e73e00640 1 -- 172.21.15.40:0/2244593672 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eabc887d0 con 0x558eabf8dcc0 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.852+0000 7f1e73e00640 1 --2- 172.21.15.40:0/2244593672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 0x558eabf8e140 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c079f50 tx=0x7f1e6c07aa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be06de3271cf593 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.852+0000 7f1e60c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.852+0000 7f1e60c00640 1 -- 172.21.15.40:0/2244593672 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c07e600 con 0x558eabf8dcc0 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.852+0000 7f1e60c00640 1 -- 172.21.15.40:0/2244593672 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f1e6c0a6c90 con 0x558eabf8dcc0 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.852+0000 7f1e7d365740 1 -- 172.21.15.40:0/2244593672 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e64015b50 con 0x558eabf8dcc0 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.852+0000 7f1e60c00640 1 -- 172.21.15.40:0/2244593672 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f1e6c082e30 con 0x558eabf8dcc0 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.852+0000 7f1e7d365740 10 client.8598 fetch_fsmap learned FSMap version 102 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.852+0000 7f1e7d365740 1 -- 172.21.15.40:0/2244593672 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) -- 0x558eabbbe3f0 con 0x558eabf8dcc0 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.852+0000 7f1e60c00640 1 -- 172.21.15.40:0/2244593672 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e64015b50 con 0x558eabf8dcc0 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.852+0000 7f1e60c00640 1 -- 172.21.15.40:0/2244593672 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 102) ==== 3550+0+0 (secure 0 0 0) 0x7f1e6c07e160 con 0x558eabf8dcc0 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.853+0000 7f1e7d365740 10 client.8598 fetch_fsmap finished waiting for FSMap version 102 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.853+0000 7f1e7d365740 10 client.8598 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.853+0000 7f1e7d365740 20 client.8598 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.853+0000 7f1e7d365740 1 --2- 172.21.15.40:0/2244593672 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x558eabf748a0 0x558eabf74cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.853+0000 7f1e7d365740 4 client.8598 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.853+0000 7f1e7d365740 1 -- 172.21.15.40:0/2244593672 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558eabeba7e0 con 0x558eabf748a0 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.853+0000 7f1e73400640 1 --2- 172.21.15.40:0/2244593672 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x558eabf748a0 0x558eabf74cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.853+0000 7f1e73400640 1 --2- 172.21.15.40:0/2244593672 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x558eabf748a0 0x558eabf74cd0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ace9a16309210e2c server_cookie=5f000321779c743a in_seq=0 out_seq=0 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.853+0000 7f1e60c00640 10 client.8598 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.854+0000 7f1e60c00640 1 -- 172.21.15.40:0/2244593672 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558eabeba7e0 con 0x558eabf748a0 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.854+0000 7f1e60c00640 10 client.8598 handle_command_reply: tid=0 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.854+0000 7f1e7d365740 1 client.8598 shutdown 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.854+0000 7f1e7d365740 1 -- 172.21.15.40:0/2244593672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 msgr2=0x558eabf8e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.854+0000 7f1e7d365740 1 --2- 172.21.15.40:0/2244593672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 0x558eabf8e140 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c079f50 tx=0x7f1e6c07aa10 comp rx=0 tx=0).stop 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.854+0000 7f1e7d365740 1 -- 172.21.15.40:0/2244593672 shutdown_connections 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.854+0000 7f1e7d365740 1 --2- 172.21.15.40:0/2244593672 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x558eabf748a0 0x558eabf74cd0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.855+0000 7f1e7d365740 1 --2- 172.21.15.40:0/2244593672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 0x558eabf8e140 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.855+0000 7f1e7d365740 1 --2- 172.21.15.40:0/2244593672 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf90fb0 0x558eabf75ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.855+0000 7f1e7d365740 1 --2- 172.21.15.40:0/2244593672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf73710 0x558eabfa13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.855+0000 7f1e7d365740 1 -- 172.21.15.40:0/2244593672 >> 172.21.15.40:0/2244593672 conn(0x558eabf493f0 msgr2=0x558eabf92330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.855+0000 7f1e60c00640 0 client.8598 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.855+0000 7f1e7d365740 1 -- 172.21.15.40:0/2244593672 shutdown_connections 2024-04-26T23:08:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.855+0000 7f1e7d365740 1 -- 172.21.15.40:0/2244593672 wait complete. 2024-04-26T23:08:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.855+0000 7f1e7d365740 20 client.8598 trim_cache size 0 max 16384 2024-04-26T23:08:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.874+0000 7f1e7d365740 1 Processor -- start 2024-04-26T23:08:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.874+0000 7f1e7d365740 1 -- start start 2024-04-26T23:08:28.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.875+0000 7f1e7d365740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf73710 0x558eabfb21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.875+0000 7f1e7d365740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf8dcc0 0x558eabf70bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.875+0000 7f1e7d365740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf90fb0 0x558eabf71110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.875+0000 7f1e7d365740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1e74201c60 con 0x558eabf90fb0 2024-04-26T23:08:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.875+0000 7f1e7d365740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e74200e00 con 0x558eabf73710 2024-04-26T23:08:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.875+0000 7f1e7d365740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1e74201530 con 0x558eabf8dcc0 2024-04-26T23:08:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.875+0000 7f1e73e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf90fb0 0x558eabf71110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.875+0000 7f1e73e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf90fb0 0x558eabf71110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53050/0 (socket says 172.21.15.40:53050) 2024-04-26T23:08:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.875+0000 7f1e73e00640 1 -- 172.21.15.40:0/370311037 learned_addr learned my addr 172.21.15.40:0/370311037 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.876+0000 7f1e6be00640 1 --2- 172.21.15.40:0/370311037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf8dcc0 0x558eabf70bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.876+0000 7f1e73e00640 1 -- 172.21.15.40:0/370311037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf8dcc0 msgr2=0x558eabf70bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.876+0000 7f1e73400640 1 --2- 172.21.15.40:0/370311037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf73710 0x558eabfb21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.876+0000 7f1e73e00640 1 --2- 172.21.15.40:0/370311037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf8dcc0 0x558eabf70bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.876+0000 7f1e73e00640 1 -- 172.21.15.40:0/370311037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf73710 msgr2=0x558eabfb21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.876+0000 7f1e73e00640 1 --2- 172.21.15.40:0/370311037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf73710 0x558eabfb21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.876+0000 7f1e73e00640 1 -- 172.21.15.40:0/370311037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eabb7e0f0 con 0x558eabf90fb0 2024-04-26T23:08:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.876+0000 7f1e6be00640 1 --2- 172.21.15.40:0/370311037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf8dcc0 0x558eabf70bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.876+0000 7f1e73e00640 1 --2- 172.21.15.40:0/370311037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf90fb0 0x558eabf71110 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c07e570 tx=0x7f1e6c07d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73609337c541f4c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.876+0000 7f1e63400640 1 -- 172.21.15.40:0/370311037 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c07e600 con 0x558eabf90fb0 2024-04-26T23:08:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.876+0000 7f1e63400640 1 -- 172.21.15.40:0/370311037 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6c07e160 con 0x558eabf90fb0 2024-04-26T23:08:28.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.876+0000 7f1e63400640 1 -- 172.21.15.40:0/370311037 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c099d70 con 0x558eabf90fb0 2024-04-26T23:08:28.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.877+0000 7f1e7d365740 1 -- 172.21.15.40:0/370311037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf90fb0 msgr2=0x558eabf71110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.877+0000 7f1e7d365740 1 --2- 172.21.15.40:0/370311037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf90fb0 0x558eabf71110 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c07e570 tx=0x7f1e6c07d000 comp rx=0 tx=0).stop 2024-04-26T23:08:28.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.877+0000 7f1e7d365740 1 -- 172.21.15.40:0/370311037 shutdown_connections 2024-04-26T23:08:28.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.877+0000 7f1e7d365740 1 --2- 172.21.15.40:0/370311037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf90fb0 0x558eabf71110 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.877+0000 7f1e7d365740 1 --2- 172.21.15.40:0/370311037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf8dcc0 0x558eabf70bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.877+0000 7f1e7d365740 1 --2- 172.21.15.40:0/370311037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf73710 0x558eabfb21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.877+0000 7f1e7d365740 1 -- 172.21.15.40:0/370311037 >> 172.21.15.40:0/370311037 conn(0x558eabf493f0 msgr2=0x558eabf87100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.877+0000 7f1e7d365740 1 -- 172.21.15.40:0/370311037 shutdown_connections 2024-04-26T23:08:28.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.877+0000 7f1e7d365740 1 -- 172.21.15.40:0/370311037 wait complete. 2024-04-26T23:08:28.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.878+0000 7f1e7d365740 1 -- start start 2024-04-26T23:08:28.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.878+0000 7f1e7d365740 1 Processor -- start 2024-04-26T23:08:28.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.879+0000 7f1e7d365740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf73710 0x558eabf9e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.879+0000 7f1e7d365740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 0x558eac006bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.879+0000 7f1e7d365740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf90fb0 0x558eac0070f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.879+0000 7f1e7d365740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558eabeb9fd0 con 0x558eabf8dcc0 2024-04-26T23:08:28.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.879+0000 7f1e7d365740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e74160270 con 0x558eabf90fb0 2024-04-26T23:08:28.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.879+0000 7f1e7d365740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558eabf75f30 con 0x558eabf73710 2024-04-26T23:08:28.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.879+0000 7f1e7d365740 10 client.0 _ll_register_callbacks cb 0x7f1e6c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:08:28.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.879+0000 7f1e6be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 0x558eac006bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.879+0000 7f1e6be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 0x558eac006bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53054/0 (socket says 172.21.15.40:53054) 2024-04-26T23:08:28.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.879+0000 7f1e6be00640 1 -- 172.21.15.40:0/4013021275 learned_addr learned my addr 172.21.15.40:0/4013021275 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:28.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.880+0000 7f1e6be00640 1 -- 172.21.15.40:0/4013021275 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf73710 msgr2=0x558eabf9e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.880+0000 7f1e73400640 1 --2- 172.21.15.40:0/4013021275 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf73710 0x558eabf9e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.880+0000 7f1e73e00640 1 --2- 172.21.15.40:0/4013021275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf90fb0 0x558eac0070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.880+0000 7f1e6be00640 1 --2- 172.21.15.40:0/4013021275 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf73710 0x558eabf9e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.880+0000 7f1e6be00640 1 -- 172.21.15.40:0/4013021275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf90fb0 msgr2=0x558eac0070f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.880+0000 7f1e6be00640 1 --2- 172.21.15.40:0/4013021275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf90fb0 0x558eac0070f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.880+0000 7f1e6be00640 1 -- 172.21.15.40:0/4013021275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eabe74a30 con 0x558eabf8dcc0 2024-04-26T23:08:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.880+0000 7f1e6be00640 1 --2- 172.21.15.40:0/4013021275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 0x558eac006bb0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f1e74144cf0 tx=0x7f1e5c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d93ee0d3faaa98 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.880+0000 7f1e60c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:08:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.880+0000 7f1e60c00640 1 -- 172.21.15.40:0/4013021275 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c013870 con 0x558eabf8dcc0 2024-04-26T23:08:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.880+0000 7f1e60c00640 1 -- 172.21.15.40:0/4013021275 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f1e5c013a10 con 0x558eabf8dcc0 2024-04-26T23:08:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.880+0000 7f1e7d365740 1 -- 172.21.15.40:0/4013021275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e6c07e160 con 0x558eabf8dcc0 2024-04-26T23:08:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.881+0000 7f1e60c00640 1 -- 172.21.15.40:0/4013021275 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f1e5c025050 con 0x558eabf8dcc0 2024-04-26T23:08:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.881+0000 7f1e7d365740 10 client.8604 fetch_fsmap learned FSMap version 102 2024-04-26T23:08:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.881+0000 7f1e7d365740 1 -- 172.21.15.40:0/4013021275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) -- 0x558eac007630 con 0x558eabf8dcc0 2024-04-26T23:08:28.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.881+0000 7f1e60c00640 1 -- 172.21.15.40:0/4013021275 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6c07e160 con 0x558eabf8dcc0 2024-04-26T23:08:28.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.881+0000 7f1e60c00640 1 -- 172.21.15.40:0/4013021275 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 102) ==== 3550+0+0 (secure 0 0 0) 0x7f1e5c028020 con 0x558eabf8dcc0 2024-04-26T23:08:28.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.881+0000 7f1e7d365740 10 client.8604 fetch_fsmap finished waiting for FSMap version 102 2024-04-26T23:08:28.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.881+0000 7f1e7d365740 10 client.8604 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-04-26T23:08:28.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.881+0000 7f1e7d365740 20 client.8604 populate_metadata read hostname 'smithi040' 2024-04-26T23:08:28.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.881+0000 7f1e7d365740 1 --2- 172.21.15.40:0/4013021275 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x558eabf8e330 0x558eabf8ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:28.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.881+0000 7f1e7d365740 4 client.8604 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:08:28.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.881+0000 7f1e7d365740 1 -- 172.21.15.40:0/4013021275 --> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558eabeb4650 con 0x558eabf8e330 2024-04-26T23:08:28.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.881+0000 7f1e73400640 1 --2- 172.21.15.40:0/4013021275 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x558eabf8e330 0x558eabf8ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:28.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e73400640 1 --2- 172.21.15.40:0/4013021275 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x558eabf8e330 0x558eabf8ef90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdc35cda73000a36 server_cookie=5f4757bd0c86bd9c in_seq=0 out_seq=0 2024-04-26T23:08:28.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e60c00640 10 client.8604 ms_handle_connect on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:28.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e60c00640 1 -- 172.21.15.40:0/4013021275 <== mds.0 v2:172.21.15.40:6834/88401598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x558eabeb4650 con 0x558eabf8e330 2024-04-26T23:08:28.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e60c00640 10 client.8604 handle_command_reply: tid=0 2024-04-26T23:08:28.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e7d365740 1 client.8604 shutdown 2024-04-26T23:08:28.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e7d365740 1 -- 172.21.15.40:0/4013021275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 msgr2=0x558eac006bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e7d365740 1 --2- 172.21.15.40:0/4013021275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 0x558eac006bb0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f1e74144cf0 tx=0x7f1e5c011a50 comp rx=0 tx=0).stop 2024-04-26T23:08:28.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e7d365740 1 -- 172.21.15.40:0/4013021275 shutdown_connections 2024-04-26T23:08:28.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e7d365740 1 --2- 172.21.15.40:0/4013021275 >> [v2:172.21.15.40:6834/88401598,v1:172.21.15.40:6835/88401598] conn(0x558eabf8e330 0x558eabf8ef90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e7d365740 1 --2- 172.21.15.40:0/4013021275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558eabf90fb0 0x558eac0070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e7d365740 1 --2- 172.21.15.40:0/4013021275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558eabf8dcc0 0x558eac006bb0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e7d365740 1 --2- 172.21.15.40:0/4013021275 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558eabf73710 0x558eabf9e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e60c00640 0 client.8604 ms_handle_reset on v2:172.21.15.40:6834/88401598 2024-04-26T23:08:28.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e7d365740 1 -- 172.21.15.40:0/4013021275 >> 172.21.15.40:0/4013021275 conn(0x558eabf493f0 msgr2=0x558eabf70bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.882+0000 7f1e7d365740 1 -- 172.21.15.40:0/4013021275 shutdown_connections 2024-04-26T23:08:28.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e7d365740 1 -- 172.21.15.40:0/4013021275 wait complete. 2024-04-26T23:08:28.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e7d365740 20 client.8604 trim_cache size 0 max 16384 2024-04-26T23:08:28.890 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:08:28.890 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-26T23:08:28.890 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:08:28.890 INFO:teuthology.orchestra.run.smithi040.stdout: "b17426f8-9d25-4fbb-a99f-95207d3f0c9f": { 2024-04-26T23:08:28.890 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:08:28.890 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "b17426f8-9d25-4fbb-a99f-95207d3f0c9f", 2024-04-26T23:08:28.891 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:08:28.891 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:08:28.891 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:08:28.891 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:08:28.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e79000640 1 -- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e5006c8f0 msgr2=0x7f1e5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e79000640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e5006c8f0 0x7f1e5006edb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1e741eae60 tx=0x7f1e64009040 comp rx=0 tx=0).stop 2024-04-26T23:08:28.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e79000640 1 -- 172.21.15.40:0/3722336324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e74163620 msgr2=0x7f1e740af910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:28.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e79000640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e74163620 0x7f1e740af910 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c02ac50 tx=0x7f1e6c004a40 comp rx=0 tx=0).stop 2024-04-26T23:08:28.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e79000640 1 -- 172.21.15.40:0/3722336324 shutdown_connections 2024-04-26T23:08:28.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e79000640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e5006c8f0 0x7f1e5006edb0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e79000640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e74163620 0x7f1e740af910 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e79000640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e74145710 0x7f1e741e95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e79000640 1 --2- 172.21.15.40:0/3722336324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e740b52b0 0x7f1e74142190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:28.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e79000640 1 -- 172.21.15.40:0/3722336324 >> 172.21.15.40:0/3722336324 conn(0x7f1e740b38a0 msgr2=0x7f1e7406ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:28.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e79000640 1 -- 172.21.15.40:0/3722336324 shutdown_connections 2024-04-26T23:08:28.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:28.883+0000 7f1e79000640 1 -- 172.21.15.40:0/3722336324 wait complete. 2024-04-26T23:08:28.899 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:08:28.900 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:29.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.094+0000 7fcdd7000640 1 Processor -- start 2024-04-26T23:08:29.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.094+0000 7fcdd7000640 1 -- start start 2024-04-26T23:08:29.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.096+0000 7fcdd7000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcdd015f2d0 0x7fcdd00afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:29.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.096+0000 7fcdd7000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcdd00ce5e0 0x7fcdd00c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:29.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.096+0000 7fcdd7000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdd00af7b0 con 0x7fcdd00c5df0 2024-04-26T23:08:29.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.096+0000 7fcdd7000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fcdd00afa90 con 0x7fcdd015f2d0 2024-04-26T23:08:29.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.096+0000 7fcdd7000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcdd00cbd90 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.096+0000 7fcdd5c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcdd015f2d0 0x7fcdd00afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:29.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.096+0000 7fcdd5c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcdd015f2d0 0x7fcdd00afdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53058/0 (socket says 172.21.15.40:53058) 2024-04-26T23:08:29.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.096+0000 7fcdd6600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcdd00c5df0 0x7fcdd00cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39970/0 (socket says 172.21.15.40:39970) 2024-04-26T23:08:29.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.096+0000 7fcdd5c00640 1 -- 172.21.15.40:0/805652551 learned_addr learned my addr 172.21.15.40:0/805652551 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:29.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.097+0000 7fcdd5200640 1 --2- 172.21.15.40:0/805652551 >> v2:172.21.15.103:3300/0 conn(0x7fcdd00ce5e0 0x7fcdd00c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:29.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.097+0000 7fcdd5c00640 1 -- 172.21.15.40:0/805652551 >> v2:172.21.15.103:3300/0 conn(0x7fcdd00ce5e0 msgr2=0x7fcdd00c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:29.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.097+0000 7fcdd5c00640 1 --2- 172.21.15.40:0/805652551 >> v2:172.21.15.103:3300/0 conn(0x7fcdd00ce5e0 0x7fcdd00c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.097+0000 7fcdd5c00640 1 -- 172.21.15.40:0/805652551 >> v1:172.21.15.40:6789/0 conn(0x7fcdd00c5df0 legacy=0x7fcdd00cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:29.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.097+0000 7fcdcfe00640 1 -- 172.21.15.40:0/805652551 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 420012799 0 0) 0x7fcdd00af7b0 con 0x7fcdd00c5df0 2024-04-26T23:08:29.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.097+0000 7fcdd5c00640 1 -- 172.21.15.40:0/805652551 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd00cc070 con 0x7fcdd015f2d0 2024-04-26T23:08:29.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.097+0000 7fcdd5200640 1 --2- 172.21.15.40:0/805652551 >> v2:172.21.15.103:3300/0 conn(0x7fcdd00ce5e0 0x7fcdd00c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:29.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.098+0000 7fcdd5c00640 1 --2- 172.21.15.40:0/805652551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd015f2d0 0x7fcdd00afdf0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00ce520 tx=0x7fcdc402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3c8059bf2c9da3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:29.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.098+0000 7fcdcfe00640 1 -- 172.21.15.40:0/805652551 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4037070 con 0x7fcdd015f2d0 2024-04-26T23:08:29.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.098+0000 7fcdcfe00640 1 -- 172.21.15.40:0/805652551 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdc4032440 con 0x7fcdd015f2d0 2024-04-26T23:08:29.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.098+0000 7fcdcfe00640 1 -- 172.21.15.40:0/805652551 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4032810 con 0x7fcdd015f2d0 2024-04-26T23:08:29.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.099+0000 7fcdd7000640 1 -- 172.21.15.40:0/805652551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd015f2d0 msgr2=0x7fcdd00afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:29.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.099+0000 7fcdd7000640 1 --2- 172.21.15.40:0/805652551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd015f2d0 0x7fcdd00afdf0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00ce520 tx=0x7fcdc402a270 comp rx=0 tx=0).stop 2024-04-26T23:08:29.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.099+0000 7fcdd7000640 1 -- 172.21.15.40:0/805652551 shutdown_connections 2024-04-26T23:08:29.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.099+0000 7fcdd7000640 1 --2- 172.21.15.40:0/805652551 >> v2:172.21.15.103:3300/0 conn(0x7fcdd00ce5e0 0x7fcdd00c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.099+0000 7fcdd7000640 1 --2- 172.21.15.40:0/805652551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd015f2d0 0x7fcdd00afdf0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.099+0000 7fcdd7000640 1 -- 172.21.15.40:0/805652551 >> 172.21.15.40:0/805652551 conn(0x7fcdd00d0a10 msgr2=0x7fcdd00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:29.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.099+0000 7fcdd7000640 1 -- 172.21.15.40:0/805652551 shutdown_connections 2024-04-26T23:08:29.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.099+0000 7fcdd7000640 1 -- 172.21.15.40:0/805652551 wait complete. 2024-04-26T23:08:29.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.101+0000 7fcdd7000640 1 Processor -- start 2024-04-26T23:08:29.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.101+0000 7fcdd7000640 1 -- start start 2024-04-26T23:08:29.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.102+0000 7fcdd7000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd00c5df0 0x7fcdd00c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:29.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.102+0000 7fcdd7000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd00ce5e0 0x7fcdd014b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:29.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.102+0000 7fcdd7000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdd015f2d0 0x7fcdd00b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:29.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.102+0000 7fcdd7000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd00ccca0 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.102+0000 7fcdd7000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd006e270 con 0x7fcdd00c5df0 2024-04-26T23:08:29.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.102+0000 7fcdd7000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd00cce20 con 0x7fcdd015f2d0 2024-04-26T23:08:29.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.102+0000 7fcdd5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd00ce5e0 0x7fcdd014b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:29.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.102+0000 7fcdd5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd00ce5e0 0x7fcdd014b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53074/0 (socket says 172.21.15.40:53074) 2024-04-26T23:08:29.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.102+0000 7fcdd5200640 1 -- 172.21.15.40:0/1711825639 learned_addr learned my addr 172.21.15.40:0/1711825639 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:29.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.103+0000 7fcdd5c00640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd00c5df0 0x7fcdd00c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:29.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.103+0000 7fcdd5200640 1 -- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdd015f2d0 msgr2=0x7fcdd00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:29.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.103+0000 7fcdd5200640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdd015f2d0 0x7fcdd00b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.103+0000 7fcdd5200640 1 -- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd00c5df0 msgr2=0x7fcdd00c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:29.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.103+0000 7fcdd5200640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd00c5df0 0x7fcdd00c7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.103+0000 7fcdd5200640 1 -- 172.21.15.40:0/1711825639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd015b520 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.103+0000 7fcdd5c00640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd00c5df0 0x7fcdd00c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:29.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.103+0000 7fcdd5200640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd00ce5e0 0x7fcdd014b7f0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc00c7f0 tx=0x7fcdbc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:29.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.103+0000 7fcdcd600640 1 -- 172.21.15.40:0/1711825639 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc015070 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.103+0000 7fcdcd600640 1 -- 172.21.15.40:0/1711825639 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdbc007a30 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.103+0000 7fcdcd600640 1 -- 172.21.15.40:0/1711825639 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc007e00 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.103+0000 7fcdd7000640 1 -- 172.21.15.40:0/1711825639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd015b800 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.104+0000 7fcdd7000640 1 -- 172.21.15.40:0/1711825639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd015bdc0 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.105+0000 7fcdd7000640 1 -- 172.21.15.40:0/1711825639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd0069370 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.106+0000 7fcdcd600640 1 -- 172.21.15.40:0/1711825639 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcdbc018070 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.106+0000 7fcdcd600640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdb006c8f0 0x7fcdb006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:29.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.107+0000 7fcdcd600640 1 -- 172.21.15.40:0/1711825639 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fcdbc08bc60 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.107+0000 7fcdd5c00640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdb006c8f0 0x7fcdb006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:29.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.108+0000 7fcdd5c00640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdb006c8f0 0x7fcdb006edb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4034000 tx=0x7fcdc402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:29.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.112+0000 7fcdcd600640 1 -- 172.21.15.40:0/1711825639 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fcdbc059f90 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.359+0000 7fcdd7000640 1 -- 172.21.15.40:0/1711825639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcdd006de10 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.359+0000 7fcdcd600640 1 -- 172.21.15.40:0/1711825639 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5232 (secure 0 0 0) 0x7fcdbc059890 con 0x7fcdd00ce5e0 2024-04-26T23:08:29.360 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:29.360 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:26.911596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:29.360 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-04-26T23:08:29.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.362+0000 7fcdd7000640 1 -- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdb006c8f0 msgr2=0x7fcdb006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:29.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.362+0000 7fcdd7000640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdb006c8f0 0x7fcdb006edb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4034000 tx=0x7fcdc402e000 comp rx=0 tx=0).stop 2024-04-26T23:08:29.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.362+0000 7fcdd7000640 1 -- 172.21.15.40:0/1711825639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd00ce5e0 msgr2=0x7fcdd014b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:29.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.362+0000 7fcdd7000640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd00ce5e0 0x7fcdd014b7f0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc00c7f0 tx=0x7fcdbc00cbf0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.362+0000 7fcdd7000640 1 -- 172.21.15.40:0/1711825639 shutdown_connections 2024-04-26T23:08:29.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.362+0000 7fcdd7000640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcdb006c8f0 0x7fcdb006edb0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.362+0000 7fcdd7000640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdd015f2d0 0x7fcdd00b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.362+0000 7fcdd7000640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd00ce5e0 0x7fcdd014b7f0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.362+0000 7fcdd7000640 1 --2- 172.21.15.40:0/1711825639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd00c5df0 0x7fcdd00c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.362+0000 7fcdd7000640 1 -- 172.21.15.40:0/1711825639 >> 172.21.15.40:0/1711825639 conn(0x7fcdd00d0a10 msgr2=0x7fcdd0155690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:29.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.363+0000 7fcdd7000640 1 -- 172.21.15.40:0/1711825639 shutdown_connections 2024-04-26T23:08:29.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.363+0000 7fcdd7000640 1 -- 172.21.15.40:0/1711825639 wait complete. 2024-04-26T23:08:29.382 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'd'] 2024-04-26T23:08:29.383 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:08:29.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.594+0000 7f6bedc00640 1 Processor -- start 2024-04-26T23:08:29.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.595+0000 7f6bedc00640 1 -- start start 2024-04-26T23:08:29.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.596+0000 7f6bedc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6be80cc480 0x7f6be8072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:29.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.596+0000 7f6bedc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6be8072780 0x7f6be8166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:29.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.596+0000 7f6bedc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6be815d7c0 con 0x7f6be8071e60 2024-04-26T23:08:29.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.596+0000 7f6bedc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f6be815daa0 con 0x7f6be8072780 2024-04-26T23:08:29.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.596+0000 7f6bedc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6be815dd80 con 0x7f6be80cc480 2024-04-26T23:08:29.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.597+0000 7f6be7400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6be8072780 0x7f6be8166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:29.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.597+0000 7f6be7400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6be8072780 0x7f6be8166ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53078/0 (socket says 172.21.15.40:53078) 2024-04-26T23:08:29.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.597+0000 7f6be7400640 1 -- 172.21.15.40:0/704542471 learned_addr learned my addr 172.21.15.40:0/704542471 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:29.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.597+0000 7f6bdfe00640 1 --2- 172.21.15.40:0/704542471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6be80cc480 0x7f6be8072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:29.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.597+0000 7f6be7400640 1 -- 172.21.15.40:0/704542471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6be80cc480 msgr2=0x7f6be8072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:29.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.597+0000 7f6be7400640 1 --2- 172.21.15.40:0/704542471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6be80cc480 0x7f6be8072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.597+0000 7f6be7400640 1 -- 172.21.15.40:0/704542471 >> v1:172.21.15.103:6789/0 conn(0x7f6be8071e60 legacy=0x7f6be806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:29.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.597+0000 7f6be7400640 1 -- 172.21.15.40:0/704542471 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be815e060 con 0x7f6be8072780 2024-04-26T23:08:29.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.598+0000 7f6be7400640 1 --2- 172.21.15.40:0/704542471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be8072780 0x7f6be8166ad0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009e50 tx=0x7f6be002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52be19f73273158 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:29.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.599+0000 7f6be6000640 1 -- 172.21.15.40:0/704542471 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be0036070 con 0x7f6be8072780 2024-04-26T23:08:29.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.599+0000 7f6be6000640 1 -- 172.21.15.40:0/704542471 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be0031600 con 0x7f6be8072780 2024-04-26T23:08:29.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.599+0000 7f6be6000640 1 -- 172.21.15.40:0/704542471 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be00319d0 con 0x7f6be8072780 2024-04-26T23:08:29.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.599+0000 7f6bedc00640 1 -- 172.21.15.40:0/704542471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be8072780 msgr2=0x7f6be8166ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:29.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.599+0000 7f6bedc00640 1 --2- 172.21.15.40:0/704542471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be8072780 0x7f6be8166ad0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009e50 tx=0x7f6be002a6c0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.599+0000 7f6bedc00640 1 -- 172.21.15.40:0/704542471 shutdown_connections 2024-04-26T23:08:29.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.599+0000 7f6bedc00640 1 --2- 172.21.15.40:0/704542471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6be80cc480 0x7f6be8072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.599+0000 7f6bedc00640 1 --2- 172.21.15.40:0/704542471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be8072780 0x7f6be8166ad0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.599+0000 7f6bedc00640 1 -- 172.21.15.40:0/704542471 >> 172.21.15.40:0/704542471 conn(0x7f6be80d0be0 msgr2=0x7f6be80d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:29.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.600+0000 7f6bedc00640 1 -- 172.21.15.40:0/704542471 shutdown_connections 2024-04-26T23:08:29.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.600+0000 7f6bedc00640 1 -- 172.21.15.40:0/704542471 wait complete. 2024-04-26T23:08:29.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.601+0000 7f6bedc00640 1 Processor -- start 2024-04-26T23:08:29.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.601+0000 7f6bedc00640 1 -- start start 2024-04-26T23:08:29.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.602+0000 7f6bedc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6be8071e60 0x7f6be80cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:29.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.602+0000 7f6bedc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be8072780 0x7f6be81561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.602+0000 7f6bedc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6be80cc480 0x7f6be81f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.602+0000 7f6bedc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6be815d3f0 con 0x7f6be8072780 2024-04-26T23:08:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.602+0000 7f6bedc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6be8072c10 con 0x7f6be8071e60 2024-04-26T23:08:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.602+0000 7f6bedc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6be815d570 con 0x7f6be80cc480 2024-04-26T23:08:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.603+0000 7f6bdfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be8072780 0x7f6be81561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.603+0000 7f6bdfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be8072780 0x7f6be81561d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53084/0 (socket says 172.21.15.40:53084) 2024-04-26T23:08:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.603+0000 7f6bdfe00640 1 -- 172.21.15.40:0/2719590315 learned_addr learned my addr 172.21.15.40:0/2719590315 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.603+0000 7f6be6a00640 1 --2- 172.21.15.40:0/2719590315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6be8071e60 0x7f6be80cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.603+0000 7f6bdfe00640 1 -- 172.21.15.40:0/2719590315 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6be80cc480 msgr2=0x7f6be81f5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.603+0000 7f6bdfe00640 1 --2- 172.21.15.40:0/2719590315 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6be80cc480 0x7f6be81f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.603+0000 7f6bdfe00640 1 -- 172.21.15.40:0/2719590315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6be8071e60 msgr2=0x7f6be80cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:29.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.603+0000 7f6bdfe00640 1 --2- 172.21.15.40:0/2719590315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6be8071e60 0x7f6be80cab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.603+0000 7f6bdfe00640 1 -- 172.21.15.40:0/2719590315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be81f2a80 con 0x7f6be8072780 2024-04-26T23:08:29.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.604+0000 7f6bdfe00640 1 --2- 172.21.15.40:0/2719590315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be8072780 0x7f6be81561d0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f6bd000d5f0 tx=0x7f6bd000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:29.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.604+0000 7f6bdea00640 1 -- 172.21.15.40:0/2719590315 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd00125f0 con 0x7f6be8072780 2024-04-26T23:08:29.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.604+0000 7f6bdea00640 1 -- 172.21.15.40:0/2719590315 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd0019070 con 0x7f6be8072780 2024-04-26T23:08:29.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.604+0000 7f6bdea00640 1 -- 172.21.15.40:0/2719590315 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd00129d0 con 0x7f6be8072780 2024-04-26T23:08:29.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.604+0000 7f6bedc00640 1 -- 172.21.15.40:0/2719590315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6be81f67b0 con 0x7f6be8072780 2024-04-26T23:08:29.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.604+0000 7f6bedc00640 1 -- 172.21.15.40:0/2719590315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6be8156710 con 0x7f6be8072780 2024-04-26T23:08:29.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.605+0000 7f6bedc00640 1 -- 172.21.15.40:0/2719590315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6be80694f0 con 0x7f6be8072780 2024-04-26T23:08:29.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.606+0000 7f6bdea00640 1 -- 172.21.15.40:0/2719590315 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6bd0012b70 con 0x7f6be8072780 2024-04-26T23:08:29.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.607+0000 7f6bdea00640 1 --2- 172.21.15.40:0/2719590315 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6bc406c920 0x7f6bc406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:29.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.607+0000 7f6bdea00640 1 -- 172.21.15.40:0/2719590315 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f6bd008c0a0 con 0x7f6be8072780 2024-04-26T23:08:29.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.608+0000 7f6be6a00640 1 --2- 172.21.15.40:0/2719590315 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6bc406c920 0x7f6bc406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:29.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.609+0000 7f6be6a00640 1 --2- 172.21.15.40:0/2719590315 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6bc406c920 0x7f6bc406ede0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8004ea0 tx=0x7f6bd8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:29.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.612+0000 7f6bdea00640 1 -- 172.21.15.40:0/2719590315 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f6bd005a330 con 0x7f6be8072780 2024-04-26T23:08:29.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.897+0000 7f6bedc00640 1 -- 172.21.15.40:0/2719590315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f6be81f64a0 con 0x7f6be8072780 2024-04-26T23:08:29.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.899+0000 7f6bdea00640 1 -- 172.21.15.40:0/2719590315 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1967 (secure 0 0 0) 0x7f6bd0059c30 con 0x7f6be8072780 2024-04-26T23:08:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:29.899 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":469,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":90,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":700759,"bytes_used":265830400,"bytes_avail":772828282880,"bytes_total":773094113280,"read_bytes_sec":0,"write_bytes_sec":6738,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":102,"id":6,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":6,"rank":2,"name":"d","status":"up:active","gid":6253},{"filesystem_id":6,"rank":1,"name":"a","status":"up:active","gid":8031},{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8040}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T23:08:25.308784+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:08:29.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.901+0000 7f6bedc00640 1 -- 172.21.15.40:0/2719590315 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6bc406c920 msgr2=0x7f6bc406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:29.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.901+0000 7f6bedc00640 1 --2- 172.21.15.40:0/2719590315 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6bc406c920 0x7f6bc406ede0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8004ea0 tx=0x7f6bd8009040 comp rx=0 tx=0).stop 2024-04-26T23:08:29.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.902+0000 7f6bedc00640 1 -- 172.21.15.40:0/2719590315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be8072780 msgr2=0x7f6be81561d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:29.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.902+0000 7f6bedc00640 1 --2- 172.21.15.40:0/2719590315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be8072780 0x7f6be81561d0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f6bd000d5f0 tx=0x7f6bd000dab0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.902+0000 7f6bedc00640 1 -- 172.21.15.40:0/2719590315 shutdown_connections 2024-04-26T23:08:29.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.902+0000 7f6bedc00640 1 --2- 172.21.15.40:0/2719590315 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6bc406c920 0x7f6bc406ede0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.902+0000 7f6bedc00640 1 --2- 172.21.15.40:0/2719590315 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6be80cc480 0x7f6be81f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.902+0000 7f6bedc00640 1 --2- 172.21.15.40:0/2719590315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be8072780 0x7f6be81561d0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.902+0000 7f6bedc00640 1 --2- 172.21.15.40:0/2719590315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6be8071e60 0x7f6be80cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:29.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.902+0000 7f6bedc00640 1 -- 172.21.15.40:0/2719590315 >> 172.21.15.40:0/2719590315 conn(0x7f6be80d0be0 msgr2=0x7f6be80b16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:29.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.902+0000 7f6bedc00640 1 -- 172.21.15.40:0/2719590315 shutdown_connections 2024-04-26T23:08:29.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:29.902+0000 7f6bedc00640 1 -- 172.21.15.40:0/2719590315 wait complete. 2024-04-26T23:08:29.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'paused paths [/d1/d2/d3]', 'c': 'paused paths [/d1/d2/d3]', 'd': 'paused paths [/d1/d2/d3]'} 2024-04-26T23:08:30.060 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:08:30.060+0000 7f36ffc00640 -1 log_channel(cluster) log [ERR] : Health check update: 3 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-26T23:08:30.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:31.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.146+0000 7f5b0d400640 1 Processor -- start 2024-04-26T23:08:31.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.146+0000 7f5b0d400640 1 -- start start 2024-04-26T23:08:31.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.147+0000 7f5b0d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b080af7a0 0x7f5b080afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.147+0000 7f5b0d400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b080d0820 con 0x7f5b08071fe0 2024-04-26T23:08:31.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.148+0000 7f5b0d400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b081614c0 con 0x7f5b0815ce60 2024-04-26T23:08:31.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.148+0000 7f5b0d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5b080cc450 con 0x7f5b080af7a0 2024-04-26T23:08:31.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.148+0000 7f5b0ca00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5b08071fe0 0x7f5b08073760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39984/0 (socket says 172.21.15.40:39984) 2024-04-26T23:08:31.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.148+0000 7f5b0ca00640 1 -- 172.21.15.40:0/1005097496 learned_addr learned my addr 172.21.15.40:0/1005097496 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:31.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.148+0000 7f5b07400640 1 --2- 172.21.15.40:0/1005097496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b080af7a0 0x7f5b080afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:31.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.148+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2997078621 0 0) 0x7f5b080d0820 con 0x7f5b08071fe0 2024-04-26T23:08:31.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.148+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5aec003520 con 0x7f5b08071fe0 2024-04-26T23:08:31.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2517218239 0 0) 0x7f5aec003520 con 0x7f5b08071fe0 2024-04-26T23:08:31.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b080af7a0 msgr2=0x7f5b080afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:31.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b06a00640 1 --2- 172.21.15.40:0/1005097496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b080af7a0 0x7f5b080afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 >> v1:172.21.15.103:6789/0 conn(0x7f5b0815ce60 legacy=0x7f5b080ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:31.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b08161760 con 0x7f5b08071fe0 2024-04-26T23:08:31.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f5b000019f0 con 0x7f5b08071fe0 2024-04-26T23:08:31.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 >> v1:172.21.15.40:6789/0 conn(0x7f5b08071fe0 legacy=0x7f5b08073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:31.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b06a00640 1 --2- 172.21.15.40:0/1005097496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aec0039e0 0x7f5aec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b06a00640 1 --2- 172.21.15.40:0/1005097496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aec004800 0x7f5aec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b06a00640 1 --2- 172.21.15.40:0/1005097496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b080af7a0 0x7f5aec005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b080cc450 con 0x7f5aec0039e0 2024-04-26T23:08:31.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec00baf0 con 0x7f5aec004800 2024-04-26T23:08:31.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5aec00bcf0 con 0x7f5b080af7a0 2024-04-26T23:08:31.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.149+0000 7f5b07e00640 1 --2- 172.21.15.40:0/1005097496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aec0039e0 0x7f5aec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:31.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.150+0000 7f5b07400640 1 --2- 172.21.15.40:0/1005097496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aec004800 0x7f5aec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:31.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.150+0000 7f5b07e00640 1 -- 172.21.15.40:0/1005097496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b080af7a0 msgr2=0x7f5aec005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:31.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.150+0000 7f5b0ca00640 1 --2- 172.21.15.40:0/1005097496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b080af7a0 0x7f5aec005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:31.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.150+0000 7f5b07e00640 1 --2- 172.21.15.40:0/1005097496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b080af7a0 0x7f5aec005200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.150+0000 7f5b07e00640 1 -- 172.21.15.40:0/1005097496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aec004800 msgr2=0x7f5aec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:31.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.150+0000 7f5b07e00640 1 --2- 172.21.15.40:0/1005097496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aec004800 0x7f5aec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.150+0000 7f5b07e00640 1 -- 172.21.15.40:0/1005097496 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec003520 con 0x7f5aec0039e0 2024-04-26T23:08:31.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.150+0000 7f5b07400640 1 --2- 172.21.15.40:0/1005097496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5aec004800 0x7f5aec004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:31.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.151+0000 7f5b07e00640 1 --2- 172.21.15.40:0/1005097496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aec0039e0 0x7f5aec003e00 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f5af8001750 tx=0x7f5aec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30eda3f2bf9772bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:31.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.151+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8001370 con 0x7f5aec0039e0 2024-04-26T23:08:31.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.151+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af80052d0 con 0x7f5aec0039e0 2024-04-26T23:08:31.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.151+0000 7f5b06a00640 1 -- 172.21.15.40:0/1005097496 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af802abe0 con 0x7f5aec0039e0 2024-04-26T23:08:31.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.152+0000 7f5b0d400640 1 -- 172.21.15.40:0/1005097496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aec0039e0 msgr2=0x7f5aec003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:31.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.152+0000 7f5b0d400640 1 --2- 172.21.15.40:0/1005097496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aec0039e0 0x7f5aec003e00 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f5af8001750 tx=0x7f5aec005830 comp rx=0 tx=0).stop 2024-04-26T23:08:31.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.152+0000 7f5b0ca00640 1 -- 172.21.15.40:0/1005097496 reap_dead start 2024-04-26T23:08:31.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.152+0000 7f5b0d400640 1 -- 172.21.15.40:0/1005097496 shutdown_connections 2024-04-26T23:08:31.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.152+0000 7f5b0d400640 1 -- 172.21.15.40:0/1005097496 >> 172.21.15.40:0/1005097496 conn(0x7f5b080d0a10 msgr2=0x7f5b080d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:31.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.152+0000 7f5b0d400640 1 -- 172.21.15.40:0/1005097496 shutdown_connections 2024-04-26T23:08:31.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.152+0000 7f5b0d400640 1 -- 172.21.15.40:0/1005097496 wait complete. 2024-04-26T23:08:31.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.154+0000 7f5b0d400640 1 Processor -- start 2024-04-26T23:08:31.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.154+0000 7f5b0d400640 1 -- start start 2024-04-26T23:08:31.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.155+0000 7f5b0d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aec0039e0 0x7f5b081569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.155+0000 7f5b0d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b081f65e0 0x7f5b080c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.155+0000 7f5b0d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b081f6a60 0x7f5b080c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.155+0000 7f5b0d400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b080cc1f0 con 0x7f5aec0039e0 2024-04-26T23:08:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.155+0000 7f5b0d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b080cc070 con 0x7f5b081f65e0 2024-04-26T23:08:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.155+0000 7f5b0d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5b08073e70 con 0x7f5b081f6a60 2024-04-26T23:08:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.155+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aec0039e0 0x7f5b081569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.155+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aec0039e0 0x7f5b081569b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53096/0 (socket says 172.21.15.40:53096) 2024-04-26T23:08:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.155+0000 7f5b07e00640 1 -- 172.21.15.40:0/1030001735 learned_addr learned my addr 172.21.15.40:0/1030001735 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.155+0000 7f5b07e00640 1 -- 172.21.15.40:0/1030001735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b081f6a60 msgr2=0x7f5b080c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.155+0000 7f5b07e00640 1 --2- 172.21.15.40:0/1030001735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b081f6a60 0x7f5b080c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.155+0000 7f5b07e00640 1 -- 172.21.15.40:0/1030001735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b081f65e0 msgr2=0x7f5b080c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:31.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.155+0000 7f5b07e00640 1 --2- 172.21.15.40:0/1030001735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b081f65e0 0x7f5b080c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.156+0000 7f5b07e00640 1 -- 172.21.15.40:0/1030001735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b081f6fc0 con 0x7f5aec0039e0 2024-04-26T23:08:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.156+0000 7f5b07e00640 1 --2- 172.21.15.40:0/1030001735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aec0039e0 0x7f5b081569b0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f5af8008fd0 tx=0x7f5af8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.156+0000 7f5affe00640 1 -- 172.21.15.40:0/1030001735 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8042070 con 0x7f5aec0039e0 2024-04-26T23:08:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.156+0000 7f5affe00640 1 -- 172.21.15.40:0/1030001735 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af802be50 con 0x7f5aec0039e0 2024-04-26T23:08:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.156+0000 7f5affe00640 1 -- 172.21.15.40:0/1030001735 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8037040 con 0x7f5aec0039e0 2024-04-26T23:08:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.156+0000 7f5b0d400640 1 -- 172.21.15.40:0/1030001735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b081f72a0 con 0x7f5aec0039e0 2024-04-26T23:08:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.156+0000 7f5b0d400640 1 -- 172.21.15.40:0/1030001735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b080b3ad0 con 0x7f5aec0039e0 2024-04-26T23:08:31.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.158+0000 7f5b0d400640 1 -- 172.21.15.40:0/1030001735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b08069370 con 0x7f5aec0039e0 2024-04-26T23:08:31.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.159+0000 7f5affe00640 1 -- 172.21.15.40:0/1030001735 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5af8045070 con 0x7f5aec0039e0 2024-04-26T23:08:31.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.159+0000 7f5affe00640 1 --2- 172.21.15.40:0/1030001735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ae806c8f0 0x7f5ae806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.159+0000 7f5affe00640 1 -- 172.21.15.40:0/1030001735 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f5af80ab460 con 0x7f5aec0039e0 2024-04-26T23:08:31.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.160+0000 7f5b07400640 1 --2- 172.21.15.40:0/1030001735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ae806c8f0 0x7f5ae806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:31.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.161+0000 7f5b07400640 1 --2- 172.21.15.40:0/1030001735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ae806c8f0 0x7f5ae806edb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5b0815c710 tx=0x7f5af4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:31.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.164+0000 7f5affe00640 1 -- 172.21.15.40:0/1030001735 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5af8079700 con 0x7f5aec0039e0 2024-04-26T23:08:31.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.405+0000 7f5b0d400640 1 -- 172.21.15.40:0/1030001735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b081f74a0 con 0x7f5aec0039e0 2024-04-26T23:08:31.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.406+0000 7f5affe00640 1 -- 172.21.15.40:0/1030001735 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5232 (secure 0 0 0) 0x7f5af8079000 con 0x7f5aec0039e0 2024-04-26T23:08:31.406 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:31.407 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:26.911596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:31.407 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-04-26T23:08:31.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.409+0000 7f5b0d400640 1 -- 172.21.15.40:0/1030001735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ae806c8f0 msgr2=0x7f5ae806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:31.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.409+0000 7f5b0d400640 1 --2- 172.21.15.40:0/1030001735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ae806c8f0 0x7f5ae806edb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5b0815c710 tx=0x7f5af4008040 comp rx=0 tx=0).stop 2024-04-26T23:08:31.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.409+0000 7f5b0d400640 1 -- 172.21.15.40:0/1030001735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aec0039e0 msgr2=0x7f5b081569b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:31.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.409+0000 7f5b0d400640 1 --2- 172.21.15.40:0/1030001735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aec0039e0 0x7f5b081569b0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f5af8008fd0 tx=0x7f5af8001120 comp rx=0 tx=0).stop 2024-04-26T23:08:31.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.409+0000 7f5b0d400640 1 -- 172.21.15.40:0/1030001735 shutdown_connections 2024-04-26T23:08:31.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.409+0000 7f5b0d400640 1 --2- 172.21.15.40:0/1030001735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5ae806c8f0 0x7f5ae806edb0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.409+0000 7f5b0d400640 1 --2- 172.21.15.40:0/1030001735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5b081f6a60 0x7f5b080c9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.409+0000 7f5b0d400640 1 --2- 172.21.15.40:0/1030001735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b081f65e0 0x7f5b080c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.409+0000 7f5b0d400640 1 --2- 172.21.15.40:0/1030001735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5aec0039e0 0x7f5b081569b0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.409+0000 7f5b0d400640 1 -- 172.21.15.40:0/1030001735 >> 172.21.15.40:0/1030001735 conn(0x7f5b080d0a10 msgr2=0x7f5b08160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:31.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.409+0000 7f5b0d400640 1 -- 172.21.15.40:0/1030001735 shutdown_connections 2024-04-26T23:08:31.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.409+0000 7f5b0d400640 1 -- 172.21.15.40:0/1030001735 wait complete. 2024-04-26T23:08:31.429 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'd'] 2024-04-26T23:08:31.429 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:08:31.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.633+0000 7f2da3200640 1 Processor -- start 2024-04-26T23:08:31.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.633+0000 7f2da3200640 1 -- start start 2024-04-26T23:08:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.635+0000 7f2da3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2da4073e10 0x7f2da41600d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.635+0000 7f2da3200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2da40b4f50 0x7f2da415d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.635+0000 7f2da3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da4160610 con 0x7f2da415c160 2024-04-26T23:08:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.635+0000 7f2da3200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f2da4153040 con 0x7f2da40b4f50 2024-04-26T23:08:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.635+0000 7f2da3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2da415fc40 con 0x7f2da4073e10 2024-04-26T23:08:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.635+0000 7f2da2800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2da40b4f50 0x7f2da415d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.635+0000 7f2da2800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2da40b4f50 0x7f2da415d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53110/0 (socket says 172.21.15.40:53110) 2024-04-26T23:08:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.635+0000 7f2da2800640 1 -- 172.21.15.40:0/1993888331 learned_addr learned my addr 172.21.15.40:0/1993888331 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.635+0000 7f2da1e00640 1 --2- 172.21.15.40:0/1993888331 >> v2:172.21.15.103:3300/0 conn(0x7f2da4073e10 0x7f2da41600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.636+0000 7f2da2800640 1 -- 172.21.15.40:0/1993888331 >> v2:172.21.15.103:3300/0 conn(0x7f2da4073e10 msgr2=0x7f2da41600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.636+0000 7f2da2800640 1 --2- 172.21.15.40:0/1993888331 >> v2:172.21.15.103:3300/0 conn(0x7f2da4073e10 0x7f2da41600d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.636+0000 7f2da2800640 1 -- 172.21.15.40:0/1993888331 >> v1:172.21.15.103:6789/0 conn(0x7f2da415c160 legacy=0x7f2da40b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.636+0000 7f2da2800640 1 -- 172.21.15.40:0/1993888331 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da415cb90 con 0x7f2da40b4f50 2024-04-26T23:08:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.637+0000 7f2da2800640 1 --2- 172.21.15.40:0/1993888331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2da40b4f50 0x7f2da415d150 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f2da4159ca0 tx=0x7f2d9c02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ddb804914192270 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.637+0000 7f2da0a00640 1 -- 172.21.15.40:0/1993888331 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c036070 con 0x7f2da40b4f50 2024-04-26T23:08:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.637+0000 7f2da0a00640 1 -- 172.21.15.40:0/1993888331 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d9c031600 con 0x7f2da40b4f50 2024-04-26T23:08:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.637+0000 7f2da0a00640 1 -- 172.21.15.40:0/1993888331 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c0319d0 con 0x7f2da40b4f50 2024-04-26T23:08:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.638+0000 7f2da3200640 1 -- 172.21.15.40:0/1993888331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2da40b4f50 msgr2=0x7f2da415d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.638+0000 7f2da3200640 1 --2- 172.21.15.40:0/1993888331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2da40b4f50 0x7f2da415d150 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f2da4159ca0 tx=0x7f2d9c02a6c0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.638+0000 7f2da3200640 1 -- 172.21.15.40:0/1993888331 shutdown_connections 2024-04-26T23:08:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.638+0000 7f2da3200640 1 --2- 172.21.15.40:0/1993888331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2da40b4f50 0x7f2da415d150 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.638+0000 7f2da3200640 1 --2- 172.21.15.40:0/1993888331 >> v2:172.21.15.103:3300/0 conn(0x7f2da4073e10 0x7f2da41600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.638+0000 7f2da3200640 1 -- 172.21.15.40:0/1993888331 >> 172.21.15.40:0/1993888331 conn(0x7f2da4075eb0 msgr2=0x7f2da4068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.638+0000 7f2da3200640 1 -- 172.21.15.40:0/1993888331 shutdown_connections 2024-04-26T23:08:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.638+0000 7f2da3200640 1 -- 172.21.15.40:0/1993888331 wait complete. 2024-04-26T23:08:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.640+0000 7f2da3200640 1 Processor -- start 2024-04-26T23:08:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.640+0000 7f2da3200640 1 -- start start 2024-04-26T23:08:31.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2da4073e10 0x7f2da414ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2da40b4f50 0x7f2da40ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da415c160 0x7f2da414d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da3200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2da40733c0 con 0x7f2da4073e10 2024-04-26T23:08:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2da415f8d0 con 0x7f2da40b4f50 2024-04-26T23:08:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2da4073540 con 0x7f2da415c160 2024-04-26T23:08:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2da4073e10 0x7f2da414ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2da4073e10 0x7f2da414ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53116/0 (socket says 172.21.15.40:53116) 2024-04-26T23:08:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da1e00640 1 -- 172.21.15.40:0/1072359045 learned_addr learned my addr 172.21.15.40:0/1072359045 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da1e00640 1 -- 172.21.15.40:0/1072359045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da415c160 msgr2=0x7f2da414d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da1e00640 1 --2- 172.21.15.40:0/1072359045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da415c160 0x7f2da414d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da1e00640 1 -- 172.21.15.40:0/1072359045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2da40b4f50 msgr2=0x7f2da40ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da1e00640 1 --2- 172.21.15.40:0/1072359045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2da40b4f50 0x7f2da40ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.641+0000 7f2da1e00640 1 -- 172.21.15.40:0/1072359045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da414f4d0 con 0x7f2da4073e10 2024-04-26T23:08:31.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.642+0000 7f2da1e00640 1 --2- 172.21.15.40:0/1072359045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2da4073e10 0x7f2da414ef90 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400d620 tx=0x7f2d9400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:31.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.642+0000 7f2d9a000640 1 -- 172.21.15.40:0/1072359045 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d940125f0 con 0x7f2da4073e10 2024-04-26T23:08:31.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.642+0000 7f2d9a000640 1 -- 172.21.15.40:0/1072359045 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d94019070 con 0x7f2da4073e10 2024-04-26T23:08:31.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.642+0000 7f2d9a000640 1 -- 172.21.15.40:0/1072359045 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d940129d0 con 0x7f2da4073e10 2024-04-26T23:08:31.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.642+0000 7f2da3200640 1 -- 172.21.15.40:0/1072359045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da4139050 con 0x7f2da4073e10 2024-04-26T23:08:31.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.642+0000 7f2da3200640 1 -- 172.21.15.40:0/1072359045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da41392f0 con 0x7f2da4073e10 2024-04-26T23:08:31.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.643+0000 7f2da3200640 1 -- 172.21.15.40:0/1072359045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da4069370 con 0x7f2da4073e10 2024-04-26T23:08:31.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.644+0000 7f2d9a000640 1 -- 172.21.15.40:0/1072359045 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d94012b70 con 0x7f2da4073e10 2024-04-26T23:08:31.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.645+0000 7f2d9a000640 1 --2- 172.21.15.40:0/1072359045 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d8406c8f0 0x7f2d8406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:31.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.645+0000 7f2d9a000640 1 -- 172.21.15.40:0/1072359045 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f2d9408c160 con 0x7f2da4073e10 2024-04-26T23:08:31.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.646+0000 7f2da1400640 1 --2- 172.21.15.40:0/1072359045 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d8406c8f0 0x7f2d8406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.647+0000 7f2da1400640 1 --2- 172.21.15.40:0/1072359045 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d8406c8f0 0x7f2d8406edb0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2da40b36c0 tx=0x7f2d900079b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.650+0000 7f2d9a000640 1 -- 172.21.15.40:0/1072359045 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2da414f4d0 con 0x7f2da4073e10 2024-04-26T23:08:31.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.928+0000 7f2da3200640 1 -- 172.21.15.40:0/1072359045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f2da415d300 con 0x7f2da4073e10 2024-04-26T23:08:31.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.929+0000 7f2d9a000640 1 -- 172.21.15.40:0/1072359045 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1967 (secure 0 0 0) 0x7f2d9405a360 con 0x7f2da4073e10 2024-04-26T23:08:31.929 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:31.929 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":471,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":90,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":706274,"bytes_used":265838592,"bytes_avail":772828274688,"bytes_total":773094113280,"read_bytes_sec":0,"write_bytes_sec":7421,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":102,"id":6,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":6,"rank":2,"name":"d","status":"up:active","gid":6253},{"filesystem_id":6,"rank":1,"name":"a","status":"up:active","gid":8031},{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8040}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-26T23:08:25.308784+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:08:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.932+0000 7f2da3200640 1 -- 172.21.15.40:0/1072359045 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d8406c8f0 msgr2=0x7f2d8406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.932+0000 7f2da3200640 1 --2- 172.21.15.40:0/1072359045 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d8406c8f0 0x7f2d8406edb0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2da40b36c0 tx=0x7f2d900079b0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.932+0000 7f2da3200640 1 -- 172.21.15.40:0/1072359045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2da4073e10 msgr2=0x7f2da414ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.932+0000 7f2da3200640 1 --2- 172.21.15.40:0/1072359045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2da4073e10 0x7f2da414ef90 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400d620 tx=0x7f2d9400dae0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.932+0000 7f2da3200640 1 -- 172.21.15.40:0/1072359045 shutdown_connections 2024-04-26T23:08:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.932+0000 7f2da3200640 1 --2- 172.21.15.40:0/1072359045 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2d8406c8f0 0x7f2d8406edb0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.932+0000 7f2da3200640 1 --2- 172.21.15.40:0/1072359045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2da415c160 0x7f2da414d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.932+0000 7f2da3200640 1 --2- 172.21.15.40:0/1072359045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2da40b4f50 0x7f2da40ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.932+0000 7f2da3200640 1 --2- 172.21.15.40:0/1072359045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2da4073e10 0x7f2da414ef90 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:31.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.932+0000 7f2da3200640 1 -- 172.21.15.40:0/1072359045 >> 172.21.15.40:0/1072359045 conn(0x7f2da4075eb0 msgr2=0x7f2da415d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:31.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.932+0000 7f2da3200640 1 -- 172.21.15.40:0/1072359045 shutdown_connections 2024-04-26T23:08:31.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:31.932+0000 7f2da3200640 1 -- 172.21.15.40:0/1072359045 wait complete. 2024-04-26T23:08:31.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'paused paths [/d1/d2/d3]', 'c': 'paused paths [/d1/d2/d3]', 'd': 'paused paths [/d1/d2/d3]'} 2024-04-26T23:08:32.953 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:33.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.175+0000 7f2060e00640 1 Processor -- start 2024-04-26T23:08:33.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.175+0000 7f2060e00640 1 -- start start 2024-04-26T23:08:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.176+0000 7f2060e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f205c15d150 0x7f205c070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.176+0000 7f2060e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f205c0cf570 0x7f205c0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.177+0000 7f2060e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f205c0713b0 con 0x7f205c0d3810 2024-04-26T23:08:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.177+0000 7f2060e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f205c15d530 con 0x7f205c15d150 2024-04-26T23:08:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.177+0000 7f2060e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f205c0709e0 con 0x7f205c0cf570 2024-04-26T23:08:33.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.177+0000 7f205aa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f205c0d3810 0x7f205c0ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39986/0 (socket says 172.21.15.40:39986) 2024-04-26T23:08:33.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.177+0000 7f205aa00640 1 -- 172.21.15.40:0/3966927550 learned_addr learned my addr 172.21.15.40:0/3966927550 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:33.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.177+0000 7f205b400640 1 --2- 172.21.15.40:0/3966927550 >> v2:172.21.15.40:3300/0 conn(0x7f205c15d150 0x7f205c070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:33.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.177+0000 7f205be00640 1 --2- 172.21.15.40:0/3966927550 >> v2:172.21.15.103:3300/0 conn(0x7f205c0cf570 0x7f205c0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:33.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.178+0000 7f205b400640 1 -- 172.21.15.40:0/3966927550 >> v2:172.21.15.103:3300/0 conn(0x7f205c0cf570 msgr2=0x7f205c0cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:33.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.178+0000 7f205b400640 1 --2- 172.21.15.40:0/3966927550 >> v2:172.21.15.103:3300/0 conn(0x7f205c0cf570 0x7f205c0cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.178+0000 7f205b400640 1 -- 172.21.15.40:0/3966927550 >> v1:172.21.15.40:6789/0 conn(0x7f205c0d3810 legacy=0x7f205c0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:33.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.177+0000 7f205a000640 1 -- 172.21.15.40:0/3966927550 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1400081504 0 0) 0x7f2044001110 con 0x7f205c0d3810 2024-04-26T23:08:33.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.178+0000 7f205b400640 1 -- 172.21.15.40:0/3966927550 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f205c0700b0 con 0x7f205c15d150 2024-04-26T23:08:33.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.179+0000 7f205b400640 1 --2- 172.21.15.40:0/3966927550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f205c15d150 0x7f205c070e70 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f205c073e00 tx=0x7f204c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232aef6922ec96b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:33.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.179+0000 7f205a000640 1 -- 172.21.15.40:0/3966927550 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c037070 con 0x7f205c15d150 2024-04-26T23:08:33.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.179+0000 7f205a000640 1 -- 172.21.15.40:0/3966927550 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f204c032440 con 0x7f205c15d150 2024-04-26T23:08:33.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.179+0000 7f205a000640 1 -- 172.21.15.40:0/3966927550 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c032810 con 0x7f205c15d150 2024-04-26T23:08:33.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.180+0000 7f2060e00640 1 -- 172.21.15.40:0/3966927550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f205c15d150 msgr2=0x7f205c070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:33.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.180+0000 7f2060e00640 1 --2- 172.21.15.40:0/3966927550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f205c15d150 0x7f205c070e70 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f205c073e00 tx=0x7f204c02a1a0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.180+0000 7f2060e00640 1 -- 172.21.15.40:0/3966927550 shutdown_connections 2024-04-26T23:08:33.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.180+0000 7f2060e00640 1 --2- 172.21.15.40:0/3966927550 >> v2:172.21.15.103:3300/0 conn(0x7f205c0cf570 0x7f205c0cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.180+0000 7f2060e00640 1 --2- 172.21.15.40:0/3966927550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f205c15d150 0x7f205c070e70 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.180+0000 7f2060e00640 1 -- 172.21.15.40:0/3966927550 >> 172.21.15.40:0/3966927550 conn(0x7f205c0d2c10 msgr2=0x7f205c0d3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:33.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.180+0000 7f2060e00640 1 -- 172.21.15.40:0/3966927550 shutdown_connections 2024-04-26T23:08:33.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.180+0000 7f2060e00640 1 -- 172.21.15.40:0/3966927550 wait complete. 2024-04-26T23:08:33.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.182+0000 7f2060e00640 1 Processor -- start 2024-04-26T23:08:33.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.182+0000 7f2060e00640 1 -- start start 2024-04-26T23:08:33.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f2060e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f205c0cf570 0x7f205c15a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:33.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f2060e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f205c0d3810 0x7f205c0caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:33.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f2060e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f205c15d150 0x7f205c06e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:33.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f2060e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f205c070670 con 0x7f205c15d150 2024-04-26T23:08:33.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f2060e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f205c0ceaf0 con 0x7f205c0cf570 2024-04-26T23:08:33.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f2060e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f205c0d0b10 con 0x7f205c0d3810 2024-04-26T23:08:33.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f205be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f205c15d150 0x7f205c06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:33.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f205be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f205c15d150 0x7f205c06e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53144/0 (socket says 172.21.15.40:53144) 2024-04-26T23:08:33.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f205be00640 1 -- 172.21.15.40:0/2165676818 learned_addr learned my addr 172.21.15.40:0/2165676818 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:33.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f205b400640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f205c0cf570 0x7f205c15a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:33.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f205be00640 1 -- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f205c0d3810 msgr2=0x7f205c0caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:33.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f205be00640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f205c0d3810 0x7f205c0caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f205be00640 1 -- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f205c0cf570 msgr2=0x7f205c15a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:33.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.183+0000 7f205be00640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f205c0cf570 0x7f205c15a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.184+0000 7f205be00640 1 -- 172.21.15.40:0/2165676818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f205c0c9790 con 0x7f205c15d150 2024-04-26T23:08:33.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.184+0000 7f205b400640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f205c0cf570 0x7f205c15a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:33.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.184+0000 7f205be00640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f205c15d150 0x7f205c06e040 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f2054007e40 tx=0x7f205400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:33.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.184+0000 7f2053400640 1 -- 172.21.15.40:0/2165676818 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2054014620 con 0x7f205c15d150 2024-04-26T23:08:33.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.184+0000 7f2053400640 1 -- 172.21.15.40:0/2165676818 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2054019070 con 0x7f205c15d150 2024-04-26T23:08:33.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.184+0000 7f2060e00640 1 -- 172.21.15.40:0/2165676818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f205c0c68a0 con 0x7f205c15d150 2024-04-26T23:08:33.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.184+0000 7f2053400640 1 -- 172.21.15.40:0/2165676818 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2054014a20 con 0x7f205c15d150 2024-04-26T23:08:33.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.184+0000 7f2060e00640 1 -- 172.21.15.40:0/2165676818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f205c0c6e60 con 0x7f205c15d150 2024-04-26T23:08:33.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.186+0000 7f2060e00640 1 -- 172.21.15.40:0/2165676818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f205c0c7060 con 0x7f205c15d150 2024-04-26T23:08:33.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.187+0000 7f2053400640 1 -- 172.21.15.40:0/2165676818 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2054014bc0 con 0x7f205c15d150 2024-04-26T23:08:33.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.187+0000 7f2053400640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f203806c8f0 0x7f203806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:33.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.188+0000 7f2053400640 1 -- 172.21.15.40:0/2165676818 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f205408c650 con 0x7f205c15d150 2024-04-26T23:08:33.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.188+0000 7f205b400640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f203806c8f0 0x7f203806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:33.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.189+0000 7f205b400640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f203806c8f0 0x7f203806edb0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f204c034000 tx=0x7f204c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:33.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.192+0000 7f2053400640 1 -- 172.21.15.40:0/2165676818 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f205c0c7060 con 0x7f205c15d150 2024-04-26T23:08:33.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.442+0000 7f2060e00640 1 -- 172.21.15.40:0/2165676818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f205c0b8b90 con 0x7f205c15d150 2024-04-26T23:08:33.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.445+0000 7f2053400640 1 -- 172.21.15.40:0/2165676818 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5232 (secure 0 0 0) 0x7f2054011260 con 0x7f205c15d150 2024-04-26T23:08:33.445 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:33.445 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:26.911596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:33.446 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-04-26T23:08:33.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.448+0000 7f2060e00640 1 -- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f203806c8f0 msgr2=0x7f203806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:33.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.448+0000 7f2060e00640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f203806c8f0 0x7f203806edb0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f204c034000 tx=0x7f204c02e000 comp rx=0 tx=0).stop 2024-04-26T23:08:33.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.448+0000 7f2060e00640 1 -- 172.21.15.40:0/2165676818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f205c15d150 msgr2=0x7f205c06e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:33.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.448+0000 7f2060e00640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f205c15d150 0x7f205c06e040 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f2054007e40 tx=0x7f205400e4b0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.448+0000 7f2060e00640 1 -- 172.21.15.40:0/2165676818 shutdown_connections 2024-04-26T23:08:33.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.448+0000 7f2060e00640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f203806c8f0 0x7f203806edb0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.448+0000 7f2060e00640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f205c15d150 0x7f205c06e040 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.448+0000 7f2060e00640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f205c0d3810 0x7f205c0caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.448+0000 7f2060e00640 1 --2- 172.21.15.40:0/2165676818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f205c0cf570 0x7f205c15a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.448+0000 7f2060e00640 1 -- 172.21.15.40:0/2165676818 >> 172.21.15.40:0/2165676818 conn(0x7f205c0d2c10 msgr2=0x7f205c15bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:33.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.448+0000 7f2060e00640 1 -- 172.21.15.40:0/2165676818 shutdown_connections 2024-04-26T23:08:33.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.448+0000 7f2060e00640 1 -- 172.21.15.40:0/2165676818 wait complete. 2024-04-26T23:08:33.469 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'd'] 2024-04-26T23:08:33.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:08:33.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.680+0000 7f24c1600640 1 Processor -- start 2024-04-26T23:08:33.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.681+0000 7f24c1600640 1 -- start start 2024-04-26T23:08:33.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.682+0000 7f24c1600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f24bc0d0c80 0x7f24bc0b5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:33.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.682+0000 7f24c1600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24bc15f9c0 con 0x7f24bc0d17d0 2024-04-26T23:08:33.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.682+0000 7f24c1600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24bc0d5730 con 0x7f24bc0d2400 2024-04-26T23:08:33.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.682+0000 7f24c1600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f24bc0d37b0 con 0x7f24bc0d0c80 2024-04-26T23:08:33.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.683+0000 7f24bbe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f24bc0d17d0 0x7f24bc0b6180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39990/0 (socket says 172.21.15.40:39990) 2024-04-26T23:08:33.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.683+0000 7f24bbe00640 1 -- 172.21.15.40:0/2818291870 learned_addr learned my addr 172.21.15.40:0/2818291870 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:33.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.683+0000 7f24c0c00640 1 --2- 172.21.15.40:0/2818291870 >> v2:172.21.15.40:3300/0 conn(0x7f24bc0d0c80 0x7f24bc0b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:33.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.683+0000 7f24baa00640 1 -- 172.21.15.40:0/2818291870 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 318527508 0 0) 0x7f24bc15f9c0 con 0x7f24bc0d17d0 2024-04-26T23:08:33.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.683+0000 7f24baa00640 1 -- 172.21.15.40:0/2818291870 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24a0003520 con 0x7f24bc0d17d0 2024-04-26T23:08:33.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.683+0000 7f24c0c00640 1 -- 172.21.15.40:0/2818291870 >> v1:172.21.15.103:6789/0 conn(0x7f24bc0d2400 legacy=0x7f24bc0d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:33.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.683+0000 7f24c0c00640 1 -- 172.21.15.40:0/2818291870 >> v1:172.21.15.40:6789/0 conn(0x7f24bc0d17d0 legacy=0x7f24bc0b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:33.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.683+0000 7f24c0c00640 1 -- 172.21.15.40:0/2818291870 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24bc0d5a10 con 0x7f24bc0d0c80 2024-04-26T23:08:33.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.685+0000 7f24c0c00640 1 --2- 172.21.15.40:0/2818291870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24bc0d0c80 0x7f24bc0b5210 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f24b4009dc0 tx=0x7f24b402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b116b55ecf62cd4d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:33.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.685+0000 7f24baa00640 1 -- 172.21.15.40:0/2818291870 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4007740 con 0x7f24bc0d0c80 2024-04-26T23:08:33.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.685+0000 7f24baa00640 1 -- 172.21.15.40:0/2818291870 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24b40078e0 con 0x7f24bc0d0c80 2024-04-26T23:08:33.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.685+0000 7f24baa00640 1 -- 172.21.15.40:0/2818291870 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4007cb0 con 0x7f24bc0d0c80 2024-04-26T23:08:33.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.686+0000 7f24c1600640 1 -- 172.21.15.40:0/2818291870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24bc0d0c80 msgr2=0x7f24bc0b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:33.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.686+0000 7f24c1600640 1 --2- 172.21.15.40:0/2818291870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24bc0d0c80 0x7f24bc0b5210 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f24b4009dc0 tx=0x7f24b402a6a0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.686+0000 7f24c1600640 1 -- 172.21.15.40:0/2818291870 shutdown_connections 2024-04-26T23:08:33.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.686+0000 7f24c1600640 1 --2- 172.21.15.40:0/2818291870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24bc0d0c80 0x7f24bc0b5210 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.686+0000 7f24c1600640 1 -- 172.21.15.40:0/2818291870 >> 172.21.15.40:0/2818291870 conn(0x7f24bc09e210 msgr2=0x7f24bc0d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:33.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.686+0000 7f24c1600640 1 -- 172.21.15.40:0/2818291870 shutdown_connections 2024-04-26T23:08:33.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.686+0000 7f24c1600640 1 -- 172.21.15.40:0/2818291870 wait complete. 2024-04-26T23:08:33.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.688+0000 7f24c1600640 1 Processor -- start 2024-04-26T23:08:33.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.688+0000 7f24c1600640 1 -- start start 2024-04-26T23:08:33.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.689+0000 7f24c1600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24bc0d0c80 0x7f24bc0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:33.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.689+0000 7f24c1600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24bc0d17d0 0x7f24bc0ba3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:33.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.689+0000 7f24c1600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24bc0d2400 0x7f24bc0ba930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:33.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.689+0000 7f24c1600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f24bc0d10f0 con 0x7f24bc0d17d0 2024-04-26T23:08:33.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.689+0000 7f24c1600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24bc0d3d70 con 0x7f24bc0d0c80 2024-04-26T23:08:33.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.689+0000 7f24c1600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f24bc0d1270 con 0x7f24bc0d2400 2024-04-26T23:08:33.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.689+0000 7f24bb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24bc0d17d0 0x7f24bc0ba3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:33.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.689+0000 7f24bb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24bc0d17d0 0x7f24bc0ba3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53158/0 (socket says 172.21.15.40:53158) 2024-04-26T23:08:33.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.689+0000 7f24bb400640 1 -- 172.21.15.40:0/2585949856 learned_addr learned my addr 172.21.15.40:0/2585949856 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:33.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.689+0000 7f24bbe00640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24bc0d0c80 0x7f24bc0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:33.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.690+0000 7f24c0c00640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24bc0d2400 0x7f24bc0ba930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:33.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.690+0000 7f24bb400640 1 -- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24bc0d2400 msgr2=0x7f24bc0ba930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:33.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.690+0000 7f24bb400640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24bc0d2400 0x7f24bc0ba930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.690+0000 7f24bb400640 1 -- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24bc0d0c80 msgr2=0x7f24bc0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:33.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.690+0000 7f24bb400640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24bc0d0c80 0x7f24bc0cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.690+0000 7f24bb400640 1 -- 172.21.15.40:0/2585949856 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24bc1601c0 con 0x7f24bc0d17d0 2024-04-26T23:08:33.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.690+0000 7f24c0c00640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24bc0d2400 0x7f24bc0ba930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:33.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.690+0000 7f24bb400640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24bc0d17d0 0x7f24bc0ba3f0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f24a800c440 tx=0x7f24a800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.690+0000 7f24b3e00640 1 -- 172.21.15.40:0/2585949856 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a80052d0 con 0x7f24bc0d17d0 2024-04-26T23:08:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.690+0000 7f24b3e00640 1 -- 172.21.15.40:0/2585949856 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a8005470 con 0x7f24bc0d17d0 2024-04-26T23:08:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.690+0000 7f24b3e00640 1 -- 172.21.15.40:0/2585949856 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8007260 con 0x7f24bc0d17d0 2024-04-26T23:08:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.690+0000 7f24c1600640 1 -- 172.21.15.40:0/2585949856 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24bc1604a0 con 0x7f24bc0d17d0 2024-04-26T23:08:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.690+0000 7f24c1600640 1 -- 172.21.15.40:0/2585949856 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24bc160a60 con 0x7f24bc0d17d0 2024-04-26T23:08:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.692+0000 7f24c1600640 1 -- 172.21.15.40:0/2585949856 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24bc069370 con 0x7f24bc0d17d0 2024-04-26T23:08:33.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.693+0000 7f24b3e00640 1 -- 172.21.15.40:0/2585949856 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f24a8007400 con 0x7f24bc0d17d0 2024-04-26T23:08:33.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.694+0000 7f24b3e00640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f249c06c920 0x7f249c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:33.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.694+0000 7f24b3e00640 1 -- 172.21.15.40:0/2585949856 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f24a808b130 con 0x7f24bc0d17d0 2024-04-26T23:08:33.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.694+0000 7f24bbe00640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f249c06c920 0x7f249c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.696+0000 7f24bbe00640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f249c06c920 0x7f249c06ede0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0011e0 tx=0x7f24ac009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:33.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.698+0000 7f24b3e00640 1 -- 172.21.15.40:0/2585949856 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f24a80593c0 con 0x7f24bc0d17d0 2024-04-26T23:08:33.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.985+0000 7f24c1600640 1 -- 172.21.15.40:0/2585949856 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f24bc0cf360 con 0x7f24bc0d17d0 2024-04-26T23:08:33.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.986+0000 7f24b3e00640 1 -- 172.21.15.40:0/2585949856 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1959 (secure 0 0 0) 0x7f24a8058cc0 con 0x7f24bc0d17d0 2024-04-26T23:08:33.987 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:33.987 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":473,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":90,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":706274,"bytes_used":265838592,"bytes_avail":772828274688,"bytes_total":773094113280,"write_bytes_sec":767,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":102,"id":6,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":6,"rank":2,"name":"d","status":"up:active","gid":6253},{"filesystem_id":6,"rank":1,"name":"a","status":"up:active","gid":8031},{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8040}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":23,"modified":"2024-04-26T23:08:31.311023+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:08:33.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.989+0000 7f24c1600640 1 -- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f249c06c920 msgr2=0x7f249c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:33.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.989+0000 7f24c1600640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f249c06c920 0x7f249c06ede0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0011e0 tx=0x7f24ac009040 comp rx=0 tx=0).stop 2024-04-26T23:08:33.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.989+0000 7f24c1600640 1 -- 172.21.15.40:0/2585949856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24bc0d17d0 msgr2=0x7f24bc0ba3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:33.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.989+0000 7f24c1600640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24bc0d17d0 0x7f24bc0ba3f0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f24a800c440 tx=0x7f24a800c900 comp rx=0 tx=0).stop 2024-04-26T23:08:33.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.989+0000 7f24c1600640 1 -- 172.21.15.40:0/2585949856 shutdown_connections 2024-04-26T23:08:33.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.989+0000 7f24c1600640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f249c06c920 0x7f249c06ede0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.989+0000 7f24c1600640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24bc0d2400 0x7f24bc0ba930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.989+0000 7f24c1600640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24bc0d17d0 0x7f24bc0ba3f0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.989+0000 7f24c1600640 1 --2- 172.21.15.40:0/2585949856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24bc0d0c80 0x7f24bc0cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:33.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.989+0000 7f24c1600640 1 -- 172.21.15.40:0/2585949856 >> 172.21.15.40:0/2585949856 conn(0x7f24bc09e210 msgr2=0x7f24bc074ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:33.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.990+0000 7f24c1600640 1 -- 172.21.15.40:0/2585949856 shutdown_connections 2024-04-26T23:08:33.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:33.990+0000 7f24c1600640 1 -- 172.21.15.40:0/2585949856 wait complete. 2024-04-26T23:08:34.010 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'paused paths [/d1/d2/d3]', 'c': 'idle+waiting paths [/d1/d2/d3]', 'd': 'idle+waiting paths [/d1/d2/d3]'} 2024-04-26T23:08:35.011 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:35.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.233+0000 7fed71800640 1 Processor -- start 2024-04-26T23:08:35.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.234+0000 7fed71800640 1 -- start start 2024-04-26T23:08:35.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.235+0000 7fed71800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fed6c071e60 0x7fed6c06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:35.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.235+0000 7fed71800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed6c06fe20 0x7fed6c072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:35.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.235+0000 7fed71800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed6c15d7c0 con 0x7fed6c072780 2024-04-26T23:08:35.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.235+0000 7fed71800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fed6c15daa0 con 0x7fed6c071e60 2024-04-26T23:08:35.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.235+0000 7fed71800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fed6c15dd80 con 0x7fed6c06fe20 2024-04-26T23:08:35.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.236+0000 7fed6aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fed6c071e60 0x7fed6c06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:35.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.236+0000 7fed6aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fed6c071e60 0x7fed6c06f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53818/0 (socket says 172.21.15.40:53818) 2024-04-26T23:08:35.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.236+0000 7fed6aa00640 1 -- 172.21.15.40:0/2231181228 learned_addr learned my addr 172.21.15.40:0/2231181228 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:35.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.236+0000 7fed62000640 1 --2- 172.21.15.40:0/2231181228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed6c06fe20 0x7fed6c072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:35.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.236+0000 7fed6aa00640 1 -- 172.21.15.40:0/2231181228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed6c06fe20 msgr2=0x7fed6c072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:35.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.236+0000 7fed6aa00640 1 --2- 172.21.15.40:0/2231181228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed6c06fe20 0x7fed6c072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.236+0000 7fed6aa00640 1 -- 172.21.15.40:0/2231181228 >> v1:172.21.15.103:6789/0 conn(0x7fed6c072780 legacy=0x7fed6c0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:35.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.236+0000 7fed6aa00640 1 -- 172.21.15.40:0/2231181228 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed6c15e060 con 0x7fed6c071e60 2024-04-26T23:08:35.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.236+0000 7fed62000640 1 --2- 172.21.15.40:0/2231181228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed6c06fe20 0x7fed6c072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:35.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.237+0000 7fed6aa00640 1 --2- 172.21.15.40:0/2231181228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c071e60 0x7fed6c06f8e0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009870 tx=0x7fed5c02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=924e36e150b3a34d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:35.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.238+0000 7fed6a000640 1 -- 172.21.15.40:0/2231181228 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5c037070 con 0x7fed6c071e60 2024-04-26T23:08:35.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.238+0000 7fed6a000640 1 -- 172.21.15.40:0/2231181228 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed5c032440 con 0x7fed6c071e60 2024-04-26T23:08:35.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.238+0000 7fed6a000640 1 -- 172.21.15.40:0/2231181228 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5c032810 con 0x7fed6c071e60 2024-04-26T23:08:35.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.238+0000 7fed71800640 1 -- 172.21.15.40:0/2231181228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c071e60 msgr2=0x7fed6c06f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:35.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.238+0000 7fed71800640 1 --2- 172.21.15.40:0/2231181228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c071e60 0x7fed6c06f8e0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009870 tx=0x7fed5c02a200 comp rx=0 tx=0).stop 2024-04-26T23:08:35.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.238+0000 7fed71800640 1 -- 172.21.15.40:0/2231181228 shutdown_connections 2024-04-26T23:08:35.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.238+0000 7fed71800640 1 --2- 172.21.15.40:0/2231181228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed6c06fe20 0x7fed6c072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.238+0000 7fed71800640 1 --2- 172.21.15.40:0/2231181228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c071e60 0x7fed6c06f8e0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.238+0000 7fed71800640 1 -- 172.21.15.40:0/2231181228 >> 172.21.15.40:0/2231181228 conn(0x7fed6c0d0be0 msgr2=0x7fed6c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:35.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.238+0000 7fed71800640 1 -- 172.21.15.40:0/2231181228 shutdown_connections 2024-04-26T23:08:35.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.238+0000 7fed71800640 1 -- 172.21.15.40:0/2231181228 wait complete. 2024-04-26T23:08:35.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.240+0000 7fed71800640 1 Processor -- start 2024-04-26T23:08:35.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.240+0000 7fed71800640 1 -- start start 2024-04-26T23:08:35.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.241+0000 7fed71800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed6c06fe20 0x7fed6c0ae8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:35.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.241+0000 7fed71800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c071e60 0x7fed6c1f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:35.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.241+0000 7fed71800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed6c072780 0x7fed6c1f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:35.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.241+0000 7fed71800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c15d3f0 con 0x7fed6c071e60 2024-04-26T23:08:35.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.241+0000 7fed71800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c072c10 con 0x7fed6c072780 2024-04-26T23:08:35.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.241+0000 7fed71800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fed6c15d570 con 0x7fed6c06fe20 2024-04-26T23:08:35.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.242+0000 7fed62000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c071e60 0x7fed6c1f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:35.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.242+0000 7fed62000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c071e60 0x7fed6c1f4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53824/0 (socket says 172.21.15.40:53824) 2024-04-26T23:08:35.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.242+0000 7fed62000640 1 -- 172.21.15.40:0/2367378835 learned_addr learned my addr 172.21.15.40:0/2367378835 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:35.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.242+0000 7fed6aa00640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed6c06fe20 0x7fed6c0ae8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:35.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.242+0000 7fed6b400640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed6c072780 0x7fed6c1f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:35.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.242+0000 7fed62000640 1 -- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed6c06fe20 msgr2=0x7fed6c0ae8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:35.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.242+0000 7fed62000640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed6c06fe20 0x7fed6c0ae8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.242+0000 7fed62000640 1 -- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed6c072780 msgr2=0x7fed6c1f5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:35.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.242+0000 7fed62000640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed6c072780 0x7fed6c1f5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.242+0000 7fed62000640 1 -- 172.21.15.40:0/2367378835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed6c0af1b0 con 0x7fed6c071e60 2024-04-26T23:08:35.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.242+0000 7fed6b400640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed6c072780 0x7fed6c1f5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:35.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.243+0000 7fed62000640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c071e60 0x7fed6c1f4dd0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fed5400c7f0 tx=0x7fed5400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:35.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.243+0000 7fed63400640 1 -- 172.21.15.40:0/2367378835 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed54015070 con 0x7fed6c071e60 2024-04-26T23:08:35.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.243+0000 7fed63400640 1 -- 172.21.15.40:0/2367378835 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed54007a30 con 0x7fed6c071e60 2024-04-26T23:08:35.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.243+0000 7fed63400640 1 -- 172.21.15.40:0/2367378835 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed54007e00 con 0x7fed6c071e60 2024-04-26T23:08:35.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.243+0000 7fed71800640 1 -- 172.21.15.40:0/2367378835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed6c0af490 con 0x7fed6c071e60 2024-04-26T23:08:35.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.243+0000 7fed71800640 1 -- 172.21.15.40:0/2367378835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed6c0afa50 con 0x7fed6c071e60 2024-04-26T23:08:35.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.244+0000 7fed71800640 1 -- 172.21.15.40:0/2367378835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed6c0694f0 con 0x7fed6c071e60 2024-04-26T23:08:35.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.245+0000 7fed63400640 1 -- 172.21.15.40:0/2367378835 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fed54018070 con 0x7fed6c071e60 2024-04-26T23:08:35.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.246+0000 7fed63400640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fed4806c8f0 0x7fed4806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:35.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.246+0000 7fed63400640 1 -- 172.21.15.40:0/2367378835 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fed5408bc60 con 0x7fed6c071e60 2024-04-26T23:08:35.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.247+0000 7fed6aa00640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fed4806c8f0 0x7fed4806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:35.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.248+0000 7fed6aa00640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fed4806c8f0 0x7fed4806edb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fed5c034000 tx=0x7fed5c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:35.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.251+0000 7fed63400640 1 -- 172.21.15.40:0/2367378835 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fed54059f90 con 0x7fed6c071e60 2024-04-26T23:08:35.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.499+0000 7fed71800640 1 -- 172.21.15.40:0/2367378835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed6c073e60 con 0x7fed6c071e60 2024-04-26T23:08:35.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.501+0000 7fed63400640 1 -- 172.21.15.40:0/2367378835 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5232 (secure 0 0 0) 0x7fed54059890 con 0x7fed6c071e60 2024-04-26T23:08:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:35.501 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:26.911596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:35.501 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-04-26T23:08:35.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.503+0000 7fed71800640 1 -- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fed4806c8f0 msgr2=0x7fed4806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:35.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.503+0000 7fed71800640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fed4806c8f0 0x7fed4806edb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fed5c034000 tx=0x7fed5c02e000 comp rx=0 tx=0).stop 2024-04-26T23:08:35.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.503+0000 7fed71800640 1 -- 172.21.15.40:0/2367378835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c071e60 msgr2=0x7fed6c1f4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:35.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.503+0000 7fed71800640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c071e60 0x7fed6c1f4dd0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fed5400c7f0 tx=0x7fed5400cbf0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.504+0000 7fed71800640 1 -- 172.21.15.40:0/2367378835 shutdown_connections 2024-04-26T23:08:35.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.504+0000 7fed71800640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fed4806c8f0 0x7fed4806edb0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.504+0000 7fed71800640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed6c072780 0x7fed6c1f5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.504+0000 7fed71800640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed6c071e60 0x7fed6c1f4dd0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.504+0000 7fed71800640 1 --2- 172.21.15.40:0/2367378835 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fed6c06fe20 0x7fed6c0ae8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.504+0000 7fed71800640 1 -- 172.21.15.40:0/2367378835 >> 172.21.15.40:0/2367378835 conn(0x7fed6c0d0be0 msgr2=0x7fed6c0b5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:35.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.504+0000 7fed71800640 1 -- 172.21.15.40:0/2367378835 shutdown_connections 2024-04-26T23:08:35.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.504+0000 7fed71800640 1 -- 172.21.15.40:0/2367378835 wait complete. 2024-04-26T23:08:35.524 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'd'] 2024-04-26T23:08:35.524 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:08:35.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.728+0000 7f1b3e600640 1 Processor -- start 2024-04-26T23:08:35.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.728+0000 7f1b3e600640 1 -- start start 2024-04-26T23:08:35.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.729+0000 7f1b3e600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1b380d17d0 0x7f1b380b6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:35.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.730+0000 7f1b3e600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1b380d0c80 0x7f1b380b5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:35.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.730+0000 7f1b3e600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b380b66c0 con 0x7f1b380d2400 2024-04-26T23:08:35.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.730+0000 7f1b3e600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1b380d37b0 con 0x7f1b380d0c80 2024-04-26T23:08:35.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.730+0000 7f1b3e600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1b380d3d70 con 0x7f1b380d17d0 2024-04-26T23:08:35.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.730+0000 7f1b3dc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1b380d0c80 0x7f1b380b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:35.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.730+0000 7f1b3dc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1b380d0c80 0x7f1b380b5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53826/0 (socket says 172.21.15.40:53826) 2024-04-26T23:08:35.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.730+0000 7f1b3dc00640 1 -- 172.21.15.40:0/1928438799 learned_addr learned my addr 172.21.15.40:0/1928438799 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:35.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.730+0000 7f1b3d200640 1 --2- 172.21.15.40:0/1928438799 >> v2:172.21.15.103:3300/0 conn(0x7f1b380d17d0 0x7f1b380b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:35.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.731+0000 7f1b3dc00640 1 -- 172.21.15.40:0/1928438799 >> v2:172.21.15.103:3300/0 conn(0x7f1b380d17d0 msgr2=0x7f1b380b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:35.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.731+0000 7f1b3dc00640 1 --2- 172.21.15.40:0/1928438799 >> v2:172.21.15.103:3300/0 conn(0x7f1b380d17d0 0x7f1b380b6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.731+0000 7f1b3dc00640 1 -- 172.21.15.40:0/1928438799 >> v1:172.21.15.103:6789/0 conn(0x7f1b380d2400 legacy=0x7f1b380d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:35.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.731+0000 7f1b3dc00640 1 -- 172.21.15.40:0/1928438799 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3815f9c0 con 0x7f1b380d0c80 2024-04-26T23:08:35.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.732+0000 7f1b3dc00640 1 --2- 172.21.15.40:0/1928438799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b380d0c80 0x7f1b380b5210 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f1b30009e50 tx=0x7f1b3002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11d9c0d396d7bce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:35.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.732+0000 7f1b37400640 1 -- 172.21.15.40:0/1928438799 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b30036070 con 0x7f1b380d0c80 2024-04-26T23:08:35.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.732+0000 7f1b37400640 1 -- 172.21.15.40:0/1928438799 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b30031600 con 0x7f1b380d0c80 2024-04-26T23:08:35.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.732+0000 7f1b37400640 1 -- 172.21.15.40:0/1928438799 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3002ae60 con 0x7f1b380d0c80 2024-04-26T23:08:35.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.733+0000 7f1b3e600640 1 -- 172.21.15.40:0/1928438799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b380d0c80 msgr2=0x7f1b380b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:35.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.733+0000 7f1b3e600640 1 --2- 172.21.15.40:0/1928438799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b380d0c80 0x7f1b380b5210 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f1b30009e50 tx=0x7f1b3002a680 comp rx=0 tx=0).stop 2024-04-26T23:08:35.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.733+0000 7f1b3e600640 1 -- 172.21.15.40:0/1928438799 shutdown_connections 2024-04-26T23:08:35.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.733+0000 7f1b3e600640 1 --2- 172.21.15.40:0/1928438799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b380d0c80 0x7f1b380b5210 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.733+0000 7f1b3e600640 1 --2- 172.21.15.40:0/1928438799 >> v2:172.21.15.103:3300/0 conn(0x7f1b380d17d0 0x7f1b380b6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.733+0000 7f1b3e600640 1 -- 172.21.15.40:0/1928438799 >> 172.21.15.40:0/1928438799 conn(0x7f1b3809e210 msgr2=0x7f1b380d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:35.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.733+0000 7f1b3e600640 1 -- 172.21.15.40:0/1928438799 shutdown_connections 2024-04-26T23:08:35.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.733+0000 7f1b3e600640 1 -- 172.21.15.40:0/1928438799 wait complete. 2024-04-26T23:08:35.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.735+0000 7f1b3e600640 1 Processor -- start 2024-04-26T23:08:35.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.735+0000 7f1b3e600640 1 -- start start 2024-04-26T23:08:35.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.736+0000 7f1b3e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b380d0c80 0x7f1b380b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:35.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3e600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b380d17d0 0x7f1b380b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:35.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3e600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b380d2400 0x7f1b381f8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:35.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3e600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1b380d3a00 con 0x7f1b380d0c80 2024-04-26T23:08:35.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3e600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1b380d5a10 con 0x7f1b380d2400 2024-04-26T23:08:35.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3e600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1b380d10f0 con 0x7f1b380d17d0 2024-04-26T23:08:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b380d0c80 0x7f1b380b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b380d0c80 0x7f1b380b90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53830/0 (socket says 172.21.15.40:53830) 2024-04-26T23:08:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3d200640 1 -- 172.21.15.40:0/1016454296 learned_addr learned my addr 172.21.15.40:0/1016454296 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b37e00640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b380d17d0 0x7f1b380b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3d200640 1 -- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b380d17d0 msgr2=0x7f1b380b95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3d200640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b380d17d0 0x7f1b380b95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3d200640 1 -- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b380d2400 msgr2=0x7f1b381f8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3dc00640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b380d2400 0x7f1b381f8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3d200640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b380d2400 0x7f1b381f8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.737+0000 7f1b3d200640 1 -- 172.21.15.40:0/1016454296 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b380bace0 con 0x7f1b380d0c80 2024-04-26T23:08:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.738+0000 7f1b3d200640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b380d0c80 0x7f1b380b90a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00d800 tx=0x7f1b2c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.738+0000 7f1b34c00640 1 -- 172.21.15.40:0/1016454296 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c00b6c0 con 0x7f1b380d0c80 2024-04-26T23:08:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.738+0000 7f1b34c00640 1 -- 172.21.15.40:0/1016454296 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b2c019070 con 0x7f1b380d0c80 2024-04-26T23:08:35.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.738+0000 7f1b3e600640 1 -- 172.21.15.40:0/1016454296 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b380b9c40 con 0x7f1b380d0c80 2024-04-26T23:08:35.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.738+0000 7f1b34c00640 1 -- 172.21.15.40:0/1016454296 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c00ba80 con 0x7f1b380d0c80 2024-04-26T23:08:35.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.738+0000 7f1b3e600640 1 -- 172.21.15.40:0/1016454296 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b380ba200 con 0x7f1b380d0c80 2024-04-26T23:08:35.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.739+0000 7f1b3e600640 1 -- 172.21.15.40:0/1016454296 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b38069370 con 0x7f1b380d0c80 2024-04-26T23:08:35.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.740+0000 7f1b34c00640 1 -- 172.21.15.40:0/1016454296 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1b2c00bc20 con 0x7f1b380d0c80 2024-04-26T23:08:35.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.741+0000 7f1b34c00640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1b1806c8f0 0x7f1b1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:35.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.741+0000 7f1b34c00640 1 -- 172.21.15.40:0/1016454296 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f1b2c08c270 con 0x7f1b380d0c80 2024-04-26T23:08:35.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.742+0000 7f1b37e00640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1b1806c8f0 0x7f1b1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:35.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.743+0000 7f1b37e00640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1b1806c8f0 0x7f1b1806edb0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1b381f8a00 tx=0x7f1b24009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:35.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:35.746+0000 7f1b34c00640 1 -- 172.21.15.40:0/1016454296 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1b2c01c360 con 0x7f1b380d0c80 2024-04-26T23:08:36.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.015+0000 7f1b3e600640 1 -- 172.21.15.40:0/1016454296 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f1b380d5db0 con 0x7f1b380d0c80 2024-04-26T23:08:36.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.016+0000 7f1b34c00640 1 -- 172.21.15.40:0/1016454296 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1959 (secure 0 0 0) 0x7f1b2c05a470 con 0x7f1b380d0c80 2024-04-26T23:08:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:36.016 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":476,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":90,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":706274,"bytes_used":265838592,"bytes_avail":772828274688,"bytes_total":773094113280,"write_bytes_sec":767,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":102,"id":6,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":6,"rank":2,"name":"d","status":"up:active","gid":6253},{"filesystem_id":6,"rank":1,"name":"a","status":"up:active","gid":8031},{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8040}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":23,"modified":"2024-04-26T23:08:31.311023+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:08:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.018+0000 7f1b3e600640 1 -- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1b1806c8f0 msgr2=0x7f1b1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.018+0000 7f1b3e600640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1b1806c8f0 0x7f1b1806edb0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1b381f8a00 tx=0x7f1b24009040 comp rx=0 tx=0).stop 2024-04-26T23:08:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.019+0000 7f1b3e600640 1 -- 172.21.15.40:0/1016454296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b380d0c80 msgr2=0x7f1b380b90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.019+0000 7f1b3e600640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b380d0c80 0x7f1b380b90a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00d800 tx=0x7f1b2c00dcc0 comp rx=0 tx=0).stop 2024-04-26T23:08:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.019+0000 7f1b3e600640 1 -- 172.21.15.40:0/1016454296 shutdown_connections 2024-04-26T23:08:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.019+0000 7f1b3e600640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1b1806c8f0 0x7f1b1806edb0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:36.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.019+0000 7f1b3e600640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b380d2400 0x7f1b381f8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.019+0000 7f1b3e600640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1b380d17d0 0x7f1b380b95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.019+0000 7f1b3e600640 1 --2- 172.21.15.40:0/1016454296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b380d0c80 0x7f1b380b90a0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.019+0000 7f1b3e600640 1 -- 172.21.15.40:0/1016454296 >> 172.21.15.40:0/1016454296 conn(0x7f1b3809e210 msgr2=0x7f1b380b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.019+0000 7f1b3e600640 1 -- 172.21.15.40:0/1016454296 shutdown_connections 2024-04-26T23:08:36.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:36.019+0000 7f1b3e600640 1 -- 172.21.15.40:0/1016454296 wait complete. 2024-04-26T23:08:36.036 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'paused paths [/d1/d2/d3]', 'c': 'idle+waiting paths [/d1/d2/d3]', 'd': 'idle+waiting paths [/d1/d2/d3]'} 2024-04-26T23:08:37.037 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:37.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.254+0000 7fe23fe00640 1 Processor -- start 2024-04-26T23:08:37.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.254+0000 7fe23fe00640 1 -- start start 2024-04-26T23:08:37.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.256+0000 7fe23fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2400c6f10 0x7fe2400c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:37.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.256+0000 7fe23fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe2400cc980 0x7fe2400cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:37.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.256+0000 7fe23fe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2400cce10 con 0x7fe2400c6010 2024-04-26T23:08:37.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.256+0000 7fe23fe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe2400c72f0 con 0x7fe2400c6f10 2024-04-26T23:08:37.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.256+0000 7fe23fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe2400cb860 con 0x7fe2400cc980 2024-04-26T23:08:37.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.256+0000 7fe23ea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2400c6f10 0x7fe2400c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:37.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.256+0000 7fe23ea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2400c6f10 0x7fe2400c5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53836/0 (socket says 172.21.15.40:53836) 2024-04-26T23:08:37.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.256+0000 7fe23ea00640 1 -- 172.21.15.40:0/3598494937 learned_addr learned my addr 172.21.15.40:0/3598494937 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:37.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.256+0000 7fe23f400640 1 --1- 172.21.15.40:0/3598494937 >> v1:172.21.15.40:6789/0 conn(0x7fe2400c6010 0x7fe240158cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39038/0 (socket says 172.21.15.40:39038) 2024-04-26T23:08:37.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.257+0000 7fe23ea00640 1 -- 172.21.15.40:0/3598494937 >> v2:172.21.15.103:3300/0 conn(0x7fe2400cc980 msgr2=0x7fe2400cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:37.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.257+0000 7fe23ea00640 1 --2- 172.21.15.40:0/3598494937 >> v2:172.21.15.103:3300/0 conn(0x7fe2400cc980 0x7fe2400cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.257+0000 7fe23ea00640 1 -- 172.21.15.40:0/3598494937 >> v1:172.21.15.40:6789/0 conn(0x7fe2400c6010 legacy=0x7fe240158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:37.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.257+0000 7fe23ea00640 1 -- 172.21.15.40:0/3598494937 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2400cd0f0 con 0x7fe2400c6f10 2024-04-26T23:08:37.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.258+0000 7fe23ea00640 1 --2- 172.21.15.40:0/3598494937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2400c6f10 0x7fe2400c5ad0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fe2400b0460 tx=0x7fe23002a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587b4edc2d010299 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:37.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.258+0000 7fe23d600640 1 -- 172.21.15.40:0/3598494937 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe230037070 con 0x7fe2400c6f10 2024-04-26T23:08:37.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.258+0000 7fe23d600640 1 -- 172.21.15.40:0/3598494937 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe230032440 con 0x7fe2400c6f10 2024-04-26T23:08:37.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.258+0000 7fe23d600640 1 -- 172.21.15.40:0/3598494937 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe230032810 con 0x7fe2400c6f10 2024-04-26T23:08:37.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.259+0000 7fe23fe00640 1 -- 172.21.15.40:0/3598494937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2400c6f10 msgr2=0x7fe2400c5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:37.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.259+0000 7fe23fe00640 1 --2- 172.21.15.40:0/3598494937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2400c6f10 0x7fe2400c5ad0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fe2400b0460 tx=0x7fe23002a1f0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.259+0000 7fe23fe00640 1 -- 172.21.15.40:0/3598494937 shutdown_connections 2024-04-26T23:08:37.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.259+0000 7fe23fe00640 1 --2- 172.21.15.40:0/3598494937 >> v2:172.21.15.103:3300/0 conn(0x7fe2400cc980 0x7fe2400cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.259+0000 7fe23fe00640 1 --2- 172.21.15.40:0/3598494937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2400c6f10 0x7fe2400c5ad0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.259+0000 7fe23fe00640 1 -- 172.21.15.40:0/3598494937 >> 172.21.15.40:0/3598494937 conn(0x7fe2400d0a10 msgr2=0x7fe2400d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:37.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.259+0000 7fe23fe00640 1 -- 172.21.15.40:0/3598494937 shutdown_connections 2024-04-26T23:08:37.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.259+0000 7fe23fe00640 1 -- 172.21.15.40:0/3598494937 wait complete. 2024-04-26T23:08:37.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.260+0000 7fe23fe00640 1 Processor -- start 2024-04-26T23:08:37.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.261+0000 7fe23fe00640 1 -- start start 2024-04-26T23:08:37.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.261+0000 7fe23fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2400c6010 0x7fe2401534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:37.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2400c6f10 0x7fe2400c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:37.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2400c9e70 0x7fe24006e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:37.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe2400c76a0 con 0x7fe2400c6010 2024-04-26T23:08:37.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2400cb1a0 con 0x7fe2400c9e70 2024-04-26T23:08:37.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2400c7820 con 0x7fe2400c6f10 2024-04-26T23:08:37.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2400c6010 0x7fe2401534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:37.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2400c6010 0x7fe2401534f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53846/0 (socket says 172.21.15.40:53846) 2024-04-26T23:08:37.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23ea00640 1 -- 172.21.15.40:0/2147147653 learned_addr learned my addr 172.21.15.40:0/2147147653 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:37.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23e000640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2400c6f10 0x7fe2400c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23ea00640 1 -- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2400c6f10 msgr2=0x7fe2400c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23ea00640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2400c6f10 0x7fe2400c9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23ea00640 1 -- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2400c9e70 msgr2=0x7fe24006e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23ea00640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2400c9e70 0x7fe24006e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23ea00640 1 -- 172.21.15.40:0/2147147653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2400b3980 con 0x7fe2400c6010 2024-04-26T23:08:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.262+0000 7fe23e000640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2400c6f10 0x7fe2400c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.263+0000 7fe23ea00640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2400c6010 0x7fe2401534f0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fe23002a230 tx=0x7fe2300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.263+0000 7fe237400640 1 -- 172.21.15.40:0/2147147653 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe230041070 con 0x7fe2400c6010 2024-04-26T23:08:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.263+0000 7fe237400640 1 -- 172.21.15.40:0/2147147653 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe23003c870 con 0x7fe2400c6010 2024-04-26T23:08:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.263+0000 7fe237400640 1 -- 172.21.15.40:0/2147147653 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe230037040 con 0x7fe2400c6010 2024-04-26T23:08:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.263+0000 7fe23fe00640 1 -- 172.21.15.40:0/2147147653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2400b3c60 con 0x7fe2400c6010 2024-04-26T23:08:37.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.263+0000 7fe23fe00640 1 -- 172.21.15.40:0/2147147653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2400b4220 con 0x7fe2400c6010 2024-04-26T23:08:37.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.264+0000 7fe23fe00640 1 -- 172.21.15.40:0/2147147653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe240069370 con 0x7fe2400c6010 2024-04-26T23:08:37.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.265+0000 7fe237400640 1 -- 172.21.15.40:0/2147147653 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe230044070 con 0x7fe2400c6010 2024-04-26T23:08:37.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.266+0000 7fe237400640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe22006c8f0 0x7fe22006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:37.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.266+0000 7fe237400640 1 -- 172.21.15.40:0/2147147653 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fe2300ac9e0 con 0x7fe2400c6010 2024-04-26T23:08:37.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.267+0000 7fe23e000640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe22006c8f0 0x7fe22006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:37.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.268+0000 7fe23e000640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe22006c8f0 0x7fe22006edb0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe22c006fd0 tx=0x7fe22c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:37.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.271+0000 7fe237400640 1 -- 172.21.15.40:0/2147147653 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe23007abe0 con 0x7fe2400c6010 2024-04-26T23:08:37.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.516+0000 7fe23fe00640 1 -- 172.21.15.40:0/2147147653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe240070020 con 0x7fe2400c6010 2024-04-26T23:08:37.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.518+0000 7fe237400640 1 -- 172.21.15.40:0/2147147653 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5232 (secure 0 0 0) 0x7fe23007a4e0 con 0x7fe2400c6010 2024-04-26T23:08:37.518 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:37.519 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:26.911596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:37.519 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-04-26T23:08:37.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.521+0000 7fe23fe00640 1 -- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe22006c8f0 msgr2=0x7fe22006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.521+0000 7fe23fe00640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe22006c8f0 0x7fe22006edb0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe22c006fd0 tx=0x7fe22c008040 comp rx=0 tx=0).stop 2024-04-26T23:08:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.521+0000 7fe23fe00640 1 -- 172.21.15.40:0/2147147653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2400c6010 msgr2=0x7fe2401534f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.521+0000 7fe23fe00640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2400c6010 0x7fe2401534f0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fe23002a230 tx=0x7fe2300046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.521+0000 7fe23fe00640 1 -- 172.21.15.40:0/2147147653 shutdown_connections 2024-04-26T23:08:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.521+0000 7fe23fe00640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe22006c8f0 0x7fe22006edb0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.521+0000 7fe23fe00640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2400c9e70 0x7fe24006e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.521+0000 7fe23fe00640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2400c6f10 0x7fe2400c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.521+0000 7fe23fe00640 1 --2- 172.21.15.40:0/2147147653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2400c6010 0x7fe2401534f0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.521+0000 7fe23fe00640 1 -- 172.21.15.40:0/2147147653 >> 172.21.15.40:0/2147147653 conn(0x7fe2400d0a10 msgr2=0x7fe2400cbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:37.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.521+0000 7fe23fe00640 1 -- 172.21.15.40:0/2147147653 shutdown_connections 2024-04-26T23:08:37.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.521+0000 7fe23fe00640 1 -- 172.21.15.40:0/2147147653 wait complete. 2024-04-26T23:08:37.540 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'd'] 2024-04-26T23:08:37.541 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:08:37.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.744+0000 7ff6cfe00640 1 Processor -- start 2024-04-26T23:08:37.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.744+0000 7ff6cfe00640 1 -- start start 2024-04-26T23:08:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.746+0000 7ff6cfe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6d0146060 0x7ff6d00b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.746+0000 7ff6cfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6d00b3fb0 0x7ff6d00d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.746+0000 7ff6cfe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d00b59f0 con 0x7ff6d0145970 2024-04-26T23:08:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.746+0000 7ff6cfe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff6d00ceed0 con 0x7ff6d00b3fb0 2024-04-26T23:08:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.746+0000 7ff6cfe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d00b5f40 con 0x7ff6d0146060 2024-04-26T23:08:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.746+0000 7ff6cf400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6d00b3fb0 0x7ff6d00d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.746+0000 7ff6cf400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6d00b3fb0 0x7ff6d00d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53858/0 (socket says 172.21.15.40:53858) 2024-04-26T23:08:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.746+0000 7ff6cf400640 1 -- 172.21.15.40:0/1009480240 learned_addr learned my addr 172.21.15.40:0/1009480240 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.746+0000 7ff6cf400640 1 -- 172.21.15.40:0/1009480240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6d0146060 msgr2=0x7ff6d00b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.746+0000 7ff6cf400640 1 --2- 172.21.15.40:0/1009480240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6d0146060 0x7ff6d00b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.746+0000 7ff6cf400640 1 -- 172.21.15.40:0/1009480240 >> v1:172.21.15.103:6789/0 conn(0x7ff6d0145970 legacy=0x7ff6d01612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.747+0000 7ff6cf400640 1 -- 172.21.15.40:0/1009480240 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d00b5c60 con 0x7ff6d00b3fb0 2024-04-26T23:08:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.748+0000 7ff6cf400640 1 --2- 172.21.15.40:0/1009480240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6d00b3fb0 0x7ff6d00d06b0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009870 tx=0x7ff6c802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86480b07aeaf2f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.748+0000 7ff6cd600640 1 -- 172.21.15.40:0/1009480240 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8037070 con 0x7ff6d00b3fb0 2024-04-26T23:08:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.748+0000 7ff6cd600640 1 -- 172.21.15.40:0/1009480240 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c8032440 con 0x7ff6d00b3fb0 2024-04-26T23:08:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.748+0000 7ff6cd600640 1 -- 172.21.15.40:0/1009480240 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8032810 con 0x7ff6d00b3fb0 2024-04-26T23:08:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.749+0000 7ff6cfe00640 1 -- 172.21.15.40:0/1009480240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6d00b3fb0 msgr2=0x7ff6d00d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.749+0000 7ff6cfe00640 1 --2- 172.21.15.40:0/1009480240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6d00b3fb0 0x7ff6d00d06b0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009870 tx=0x7ff6c802a210 comp rx=0 tx=0).stop 2024-04-26T23:08:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.749+0000 7ff6cfe00640 1 -- 172.21.15.40:0/1009480240 shutdown_connections 2024-04-26T23:08:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.749+0000 7ff6cfe00640 1 --2- 172.21.15.40:0/1009480240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6d0146060 0x7ff6d00b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.749+0000 7ff6cfe00640 1 --2- 172.21.15.40:0/1009480240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6d00b3fb0 0x7ff6d00d06b0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.749+0000 7ff6cfe00640 1 -- 172.21.15.40:0/1009480240 >> 172.21.15.40:0/1009480240 conn(0x7ff6d0075eb0 msgr2=0x7ff6d0068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:37.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.749+0000 7ff6cfe00640 1 -- 172.21.15.40:0/1009480240 shutdown_connections 2024-04-26T23:08:37.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.749+0000 7ff6cfe00640 1 -- 172.21.15.40:0/1009480240 wait complete. 2024-04-26T23:08:37.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.751+0000 7ff6cfe00640 1 Processor -- start 2024-04-26T23:08:37.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.751+0000 7ff6cfe00640 1 -- start start 2024-04-26T23:08:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6cfe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6d0145970 0x7ff6d0152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6cfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6d0146060 0x7ff6d0140080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6cfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6d01405c0 0x7ff6d0154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6cfe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d0145d50 con 0x7ff6d0146060 2024-04-26T23:08:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6cfe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d00cf3e0 con 0x7ff6d01405c0 2024-04-26T23:08:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6cfe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d00d0bf0 con 0x7ff6d0145970 2024-04-26T23:08:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6ce000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6d0146060 0x7ff6d0140080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6ce000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6d0146060 0x7ff6d0140080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53874/0 (socket says 172.21.15.40:53874) 2024-04-26T23:08:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6ce000640 1 -- 172.21.15.40:0/942028607 learned_addr learned my addr 172.21.15.40:0/942028607 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6ce000640 1 -- 172.21.15.40:0/942028607 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6d0145970 msgr2=0x7ff6d0152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6cf400640 1 --2- 172.21.15.40:0/942028607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6d01405c0 0x7ff6d0154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6ce000640 1 --2- 172.21.15.40:0/942028607 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6d0145970 0x7ff6d0152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6ce000640 1 -- 172.21.15.40:0/942028607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6d01405c0 msgr2=0x7ff6d0154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.752+0000 7ff6ce000640 1 --2- 172.21.15.40:0/942028607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6d01405c0 0x7ff6d0154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.753+0000 7ff6ce000640 1 -- 172.21.15.40:0/942028607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d0155930 con 0x7ff6d0146060 2024-04-26T23:08:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.753+0000 7ff6ce000640 1 --2- 172.21.15.40:0/942028607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6d0146060 0x7ff6d0140080 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc00b410 tx=0x7ff6bc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.753+0000 7ff6c6a00640 1 -- 172.21.15.40:0/942028607 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc015020 con 0x7ff6d0146060 2024-04-26T23:08:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.753+0000 7ff6c6a00640 1 -- 172.21.15.40:0/942028607 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6bc011850 con 0x7ff6d0146060 2024-04-26T23:08:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.753+0000 7ff6cfe00640 1 -- 172.21.15.40:0/942028607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d0155c10 con 0x7ff6d0146060 2024-04-26T23:08:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.753+0000 7ff6c6a00640 1 -- 172.21.15.40:0/942028607 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc011c40 con 0x7ff6d0146060 2024-04-26T23:08:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.753+0000 7ff6cfe00640 1 -- 172.21.15.40:0/942028607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d01561d0 con 0x7ff6d0146060 2024-04-26T23:08:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.754+0000 7ff6cfe00640 1 -- 172.21.15.40:0/942028607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6d01409d0 con 0x7ff6d0146060 2024-04-26T23:08:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.755+0000 7ff6c6a00640 1 -- 172.21.15.40:0/942028607 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff6bc018020 con 0x7ff6d0146060 2024-04-26T23:08:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.757+0000 7ff6c6a00640 1 --2- 172.21.15.40:0/942028607 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff6b006c8f0 0x7ff6b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.757+0000 7ff6c6a00640 1 -- 172.21.15.40:0/942028607 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7ff6bc08bcf0 con 0x7ff6d0146060 2024-04-26T23:08:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.757+0000 7ff6cea00640 1 --2- 172.21.15.40:0/942028607 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff6b006c8f0 0x7ff6b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.759+0000 7ff6cea00640 1 --2- 172.21.15.40:0/942028607 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff6b006c8f0 0x7ff6b006edb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff6d015c670 tx=0x7ff6c0007d30 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:37.762+0000 7ff6c6a00640 1 -- 172.21.15.40:0/942028607 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff6d01409d0 con 0x7ff6d0146060 2024-04-26T23:08:38.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.046+0000 7ff6cfe00640 1 -- 172.21.15.40:0/942028607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7ff6d00d1400 con 0x7ff6d0146060 2024-04-26T23:08:38.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.047+0000 7ff6c6a00640 1 -- 172.21.15.40:0/942028607 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1965 (secure 0 0 0) 0x7ff6bc01f020 con 0x7ff6d0146060 2024-04-26T23:08:38.047 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:38.048 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":478,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":90,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":706274,"bytes_used":265838592,"bytes_avail":772828274688,"bytes_total":773094113280,"write_bytes_sec":682,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":102,"id":6,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":6,"rank":2,"name":"d","status":"up:active","gid":6253},{"filesystem_id":6,"rank":1,"name":"a","status":"up:active","gid":8031},{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8040}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":24,"modified":"2024-04-26T23:08:35.312595+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:08:38.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.050+0000 7ff6cfe00640 1 -- 172.21.15.40:0/942028607 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff6b006c8f0 msgr2=0x7ff6b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:38.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.050+0000 7ff6cfe00640 1 --2- 172.21.15.40:0/942028607 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff6b006c8f0 0x7ff6b006edb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff6d015c670 tx=0x7ff6c0007d30 comp rx=0 tx=0).stop 2024-04-26T23:08:38.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.050+0000 7ff6cfe00640 1 -- 172.21.15.40:0/942028607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6d0146060 msgr2=0x7ff6d0140080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:38.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.050+0000 7ff6cfe00640 1 --2- 172.21.15.40:0/942028607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6d0146060 0x7ff6d0140080 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc00b410 tx=0x7ff6bc00b8d0 comp rx=0 tx=0).stop 2024-04-26T23:08:38.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.050+0000 7ff6cfe00640 1 -- 172.21.15.40:0/942028607 shutdown_connections 2024-04-26T23:08:38.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.050+0000 7ff6cfe00640 1 --2- 172.21.15.40:0/942028607 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff6b006c8f0 0x7ff6b006edb0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:38.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.050+0000 7ff6cfe00640 1 --2- 172.21.15.40:0/942028607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6d01405c0 0x7ff6d0154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:38.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.050+0000 7ff6cfe00640 1 --2- 172.21.15.40:0/942028607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6d0146060 0x7ff6d0140080 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:38.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.050+0000 7ff6cfe00640 1 --2- 172.21.15.40:0/942028607 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6d0145970 0x7ff6d0152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:38.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.050+0000 7ff6cfe00640 1 -- 172.21.15.40:0/942028607 >> 172.21.15.40:0/942028607 conn(0x7ff6d0075eb0 msgr2=0x7ff6d015c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:38.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.050+0000 7ff6cfe00640 1 -- 172.21.15.40:0/942028607 shutdown_connections 2024-04-26T23:08:38.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:38.050+0000 7ff6cfe00640 1 -- 172.21.15.40:0/942028607 wait complete. 2024-04-26T23:08:38.070 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'idle+waiting paths [/d1/d2/d3]', 'c': 'idle+waiting paths [/d1/d2/d3]', 'd': 'idle+waiting paths [/d1/d2/d3]'} 2024-04-26T23:08:39.071 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:39.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.293+0000 7ff1d7e00640 1 Processor -- start 2024-04-26T23:08:39.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.293+0000 7ff1d7e00640 1 -- start start 2024-04-26T23:08:39.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.294+0000 7ff1d7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff1d80af7a0 0x7ff1d80afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:39.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.294+0000 7ff1d7e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d80d0820 con 0x7ff1d815ce60 2024-04-26T23:08:39.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.295+0000 7ff1d7e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d81614c0 con 0x7ff1d8071fe0 2024-04-26T23:08:39.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.295+0000 7ff1d7e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff1d80cc450 con 0x7ff1d80af7a0 2024-04-26T23:08:39.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.295+0000 7ff1cfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff1d80af7a0 0x7ff1d80afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:39.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.295+0000 7ff1cfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff1d80af7a0 0x7ff1d80afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53876/0 (socket says 172.21.15.40:53876) 2024-04-26T23:08:39.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.295+0000 7ff1cfe00640 1 -- 172.21.15.40:0/471737692 learned_addr learned my addr 172.21.15.40:0/471737692 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:39.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.295+0000 7ff1d6a00640 1 --1- 172.21.15.40:0/471737692 >> v1:172.21.15.40:6789/0 conn(0x7ff1d815ce60 0x7ff1d80ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39044/0 (socket says 172.21.15.40:39044) 2024-04-26T23:08:39.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.295+0000 7ff1cfe00640 1 -- 172.21.15.40:0/471737692 >> v1:172.21.15.103:6789/0 conn(0x7ff1d8071fe0 legacy=0x7ff1d8073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:39.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.295+0000 7ff1cfe00640 1 -- 172.21.15.40:0/471737692 >> v1:172.21.15.40:6789/0 conn(0x7ff1d815ce60 legacy=0x7ff1d80ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:39.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.295+0000 7ff1cfe00640 1 -- 172.21.15.40:0/471737692 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d8161760 con 0x7ff1d80af7a0 2024-04-26T23:08:39.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.297+0000 7ff1cfe00640 1 --2- 172.21.15.40:0/471737692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1d80af7a0 0x7ff1d80afc60 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0009870 tx=0x7ff1c002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f5c49344a3b003 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:39.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.297+0000 7ff1d6000640 1 -- 172.21.15.40:0/471737692 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0037070 con 0x7ff1d80af7a0 2024-04-26T23:08:39.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.297+0000 7ff1d6000640 1 -- 172.21.15.40:0/471737692 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1c0032440 con 0x7ff1d80af7a0 2024-04-26T23:08:39.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.297+0000 7ff1d6000640 1 -- 172.21.15.40:0/471737692 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0032810 con 0x7ff1d80af7a0 2024-04-26T23:08:39.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.298+0000 7ff1d7e00640 1 -- 172.21.15.40:0/471737692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1d80af7a0 msgr2=0x7ff1d80afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:39.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.298+0000 7ff1d7e00640 1 --2- 172.21.15.40:0/471737692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1d80af7a0 0x7ff1d80afc60 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0009870 tx=0x7ff1c002a210 comp rx=0 tx=0).stop 2024-04-26T23:08:39.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.298+0000 7ff1d7e00640 1 -- 172.21.15.40:0/471737692 shutdown_connections 2024-04-26T23:08:39.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.298+0000 7ff1d7e00640 1 --2- 172.21.15.40:0/471737692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1d80af7a0 0x7ff1d80afc60 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:39.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.298+0000 7ff1d7e00640 1 -- 172.21.15.40:0/471737692 >> 172.21.15.40:0/471737692 conn(0x7ff1d80d0a10 msgr2=0x7ff1d80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:39.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.298+0000 7ff1d7e00640 1 -- 172.21.15.40:0/471737692 shutdown_connections 2024-04-26T23:08:39.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.298+0000 7ff1d7e00640 1 -- 172.21.15.40:0/471737692 wait complete. 2024-04-26T23:08:39.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.300+0000 7ff1d7e00640 1 Processor -- start 2024-04-26T23:08:39.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.300+0000 7ff1d7e00640 1 -- start start 2024-04-26T23:08:39.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.301+0000 7ff1d7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1d8071fe0 0x7ff1d8153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:39.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.301+0000 7ff1d7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1d80af7a0 0x7ff1d806f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:39.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.301+0000 7ff1d7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1d815ce60 0x7ff1d806f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:39.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.301+0000 7ff1d7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d80cc1f0 con 0x7ff1d8071fe0 2024-04-26T23:08:39.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.301+0000 7ff1d7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d80cc070 con 0x7ff1d815ce60 2024-04-26T23:08:39.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.301+0000 7ff1d7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff1d8073e70 con 0x7ff1d80af7a0 2024-04-26T23:08:39.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.301+0000 7ff1d6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1d8071fe0 0x7ff1d8153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:39.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.301+0000 7ff1d6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1d8071fe0 0x7ff1d8153660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53890/0 (socket says 172.21.15.40:53890) 2024-04-26T23:08:39.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.301+0000 7ff1d6a00640 1 -- 172.21.15.40:0/475816702 learned_addr learned my addr 172.21.15.40:0/475816702 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.302+0000 7ff1d6a00640 1 -- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1d80af7a0 msgr2=0x7ff1d806f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.302+0000 7ff1d7400640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1d815ce60 0x7ff1d806f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.302+0000 7ff1cfe00640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1d80af7a0 0x7ff1d806f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.302+0000 7ff1d6a00640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1d80af7a0 0x7ff1d806f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.302+0000 7ff1d6a00640 1 -- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1d815ce60 msgr2=0x7ff1d806f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.302+0000 7ff1d6a00640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1d815ce60 0x7ff1d806f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.302+0000 7ff1d6a00640 1 -- 172.21.15.40:0/475816702 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d80c8f30 con 0x7ff1d8071fe0 2024-04-26T23:08:39.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.302+0000 7ff1d6a00640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1d8071fe0 0x7ff1d8153660 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800c3c0 tx=0x7ff1c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:39.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.302+0000 7ff1cea00640 1 -- 172.21.15.40:0/475816702 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c80074e0 con 0x7ff1d8071fe0 2024-04-26T23:08:39.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.302+0000 7ff1cea00640 1 -- 172.21.15.40:0/475816702 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1c8007680 con 0x7ff1d8071fe0 2024-04-26T23:08:39.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.302+0000 7ff1d7e00640 1 -- 172.21.15.40:0/475816702 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d80c9210 con 0x7ff1d8071fe0 2024-04-26T23:08:39.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.302+0000 7ff1cea00640 1 -- 172.21.15.40:0/475816702 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8004a00 con 0x7ff1d8071fe0 2024-04-26T23:08:39.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.303+0000 7ff1d7e00640 1 -- 172.21.15.40:0/475816702 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d8153ba0 con 0x7ff1d8071fe0 2024-04-26T23:08:39.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.304+0000 7ff1d7e00640 1 -- 172.21.15.40:0/475816702 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1d8069370 con 0x7ff1d8071fe0 2024-04-26T23:08:39.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.305+0000 7ff1cea00640 1 -- 172.21.15.40:0/475816702 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff1c801a020 con 0x7ff1d8071fe0 2024-04-26T23:08:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.306+0000 7ff1cea00640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff1b406c8f0 0x7ff1b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.306+0000 7ff1cea00640 1 -- 172.21.15.40:0/475816702 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7ff1c808b4b0 con 0x7ff1d8071fe0 2024-04-26T23:08:39.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.307+0000 7ff1cfe00640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff1b406c8f0 0x7ff1b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:39.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.308+0000 7ff1cfe00640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff1b406c8f0 0x7ff1b406edb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0034000 tx=0x7ff1c002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:39.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.311+0000 7ff1cea00640 1 -- 172.21.15.40:0/475816702 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff1c8059740 con 0x7ff1d8071fe0 2024-04-26T23:08:39.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.555+0000 7ff1d7e00640 1 -- 172.21.15.40:0/475816702 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1d80d01f0 con 0x7ff1d8071fe0 2024-04-26T23:08:39.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.557+0000 7ff1cea00640 1 -- 172.21.15.40:0/475816702 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5232 (secure 0 0 0) 0x7ff1c8059040 con 0x7ff1d8071fe0 2024-04-26T23:08:39.557 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:39.557 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:26.911596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:39.557 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-04-26T23:08:39.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.559+0000 7ff1d7e00640 1 -- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff1b406c8f0 msgr2=0x7ff1b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:39.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.559+0000 7ff1d7e00640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff1b406c8f0 0x7ff1b406edb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0034000 tx=0x7ff1c002e000 comp rx=0 tx=0).stop 2024-04-26T23:08:39.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.559+0000 7ff1d7e00640 1 -- 172.21.15.40:0/475816702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1d8071fe0 msgr2=0x7ff1d8153660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:39.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.559+0000 7ff1d7e00640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1d8071fe0 0x7ff1d8153660 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800c3c0 tx=0x7ff1c800c880 comp rx=0 tx=0).stop 2024-04-26T23:08:39.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.560+0000 7ff1d7e00640 1 -- 172.21.15.40:0/475816702 shutdown_connections 2024-04-26T23:08:39.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.560+0000 7ff1d7e00640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff1b406c8f0 0x7ff1b406edb0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:39.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.560+0000 7ff1d7e00640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1d815ce60 0x7ff1d806f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:39.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.560+0000 7ff1d7e00640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1d80af7a0 0x7ff1d806f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:39.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.560+0000 7ff1d7e00640 1 --2- 172.21.15.40:0/475816702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1d8071fe0 0x7ff1d8153660 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:39.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.560+0000 7ff1d7e00640 1 -- 172.21.15.40:0/475816702 >> 172.21.15.40:0/475816702 conn(0x7ff1d80d0a10 msgr2=0x7ff1d80ae8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:39.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.560+0000 7ff1d7e00640 1 -- 172.21.15.40:0/475816702 shutdown_connections 2024-04-26T23:08:39.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.560+0000 7ff1d7e00640 1 -- 172.21.15.40:0/475816702 wait complete. 2024-04-26T23:08:39.580 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'd'] 2024-04-26T23:08:39.580 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:08:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.785+0000 7f562c000640 1 Processor -- start 2024-04-26T23:08:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.785+0000 7f562c000640 1 -- start start 2024-04-26T23:08:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.787+0000 7f562c000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56240cc480 0x7f5624072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.787+0000 7f562c000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5624072780 0x7f5624166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.787+0000 7f562c000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f562415d7c0 con 0x7f5624071e60 2024-04-26T23:08:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.787+0000 7f562c000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f562415daa0 con 0x7f5624072780 2024-04-26T23:08:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.787+0000 7f562c000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f562415dd80 con 0x7f56240cc480 2024-04-26T23:08:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.787+0000 7f562a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5624072780 0x7f5624166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.787+0000 7f562a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5624072780 0x7f5624166ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53898/0 (socket says 172.21.15.40:53898) 2024-04-26T23:08:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.787+0000 7f562a000640 1 -- 172.21.15.40:0/818773257 learned_addr learned my addr 172.21.15.40:0/818773257 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:39.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.787+0000 7f5628c00640 1 --2- 172.21.15.40:0/818773257 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56240cc480 0x7f5624072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:39.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.788+0000 7f562a000640 1 -- 172.21.15.40:0/818773257 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56240cc480 msgr2=0x7f5624072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:39.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.788+0000 7f562a000640 1 --2- 172.21.15.40:0/818773257 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56240cc480 0x7f5624072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:39.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.788+0000 7f562a000640 1 -- 172.21.15.40:0/818773257 >> v1:172.21.15.103:6789/0 conn(0x7f5624071e60 legacy=0x7f562406f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.788+0000 7f562a000640 1 -- 172.21.15.40:0/818773257 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f562415e060 con 0x7f5624072780 2024-04-26T23:08:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.789+0000 7f562a000640 1 --2- 172.21.15.40:0/818773257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5624072780 0x7f5624166ad0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f561c009e50 tx=0x7f561c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea5c30b0ebc66a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.789+0000 7f5623e00640 1 -- 172.21.15.40:0/818773257 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561c036070 con 0x7f5624072780 2024-04-26T23:08:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.789+0000 7f5623e00640 1 -- 172.21.15.40:0/818773257 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f561c031600 con 0x7f5624072780 2024-04-26T23:08:39.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.789+0000 7f5623e00640 1 -- 172.21.15.40:0/818773257 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561c0319d0 con 0x7f5624072780 2024-04-26T23:08:39.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.790+0000 7f562c000640 1 -- 172.21.15.40:0/818773257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5624072780 msgr2=0x7f5624166ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:39.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.790+0000 7f562c000640 1 --2- 172.21.15.40:0/818773257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5624072780 0x7f5624166ad0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f561c009e50 tx=0x7f561c02a710 comp rx=0 tx=0).stop 2024-04-26T23:08:39.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.790+0000 7f562c000640 1 -- 172.21.15.40:0/818773257 shutdown_connections 2024-04-26T23:08:39.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.790+0000 7f562c000640 1 --2- 172.21.15.40:0/818773257 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56240cc480 0x7f5624072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.790+0000 7f562c000640 1 --2- 172.21.15.40:0/818773257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5624072780 0x7f5624166ad0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.790+0000 7f562c000640 1 -- 172.21.15.40:0/818773257 >> 172.21.15.40:0/818773257 conn(0x7f56240d0be0 msgr2=0x7f56240d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.790+0000 7f562c000640 1 -- 172.21.15.40:0/818773257 shutdown_connections 2024-04-26T23:08:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.790+0000 7f562c000640 1 -- 172.21.15.40:0/818773257 wait complete. 2024-04-26T23:08:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.792+0000 7f562c000640 1 Processor -- start 2024-04-26T23:08:39.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.792+0000 7f562c000640 1 -- start start 2024-04-26T23:08:39.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.793+0000 7f562c000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5624071e60 0x7f562414e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:39.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.793+0000 7f562c000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5624072780 0x7f56240b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:39.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.793+0000 7f562c000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56240cc480 0x7f56240b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:39.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.793+0000 7f562c000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f562415d3f0 con 0x7f5624072780 2024-04-26T23:08:39.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.793+0000 7f562c000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5624072c10 con 0x7f56240cc480 2024-04-26T23:08:39.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.793+0000 7f562c000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f562415d570 con 0x7f5624071e60 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.793+0000 7f5628c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5624072780 0x7f56240b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.793+0000 7f5628c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5624072780 0x7f56240b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53906/0 (socket says 172.21.15.40:53906) 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.793+0000 7f5628c00640 1 -- 172.21.15.40:0/700255583 learned_addr learned my addr 172.21.15.40:0/700255583 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.793+0000 7f5629600640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5624071e60 0x7f562414e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.794+0000 7f5628c00640 1 -- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5624071e60 msgr2=0x7f562414e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.794+0000 7f5628c00640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5624071e60 0x7f562414e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.794+0000 7f562a000640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56240cc480 0x7f56240b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.794+0000 7f5628c00640 1 -- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56240cc480 msgr2=0x7f56240b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.794+0000 7f5628c00640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56240cc480 0x7f56240b5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.794+0000 7f5628c00640 1 -- 172.21.15.40:0/700255583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56240ca360 con 0x7f5624072780 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.794+0000 7f5629600640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5624071e60 0x7f562414e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.794+0000 7f5628c00640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5624072780 0x7f56240b5a60 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f561000d5d0 tx=0x7f561000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.794+0000 7f5621600640 1 -- 172.21.15.40:0/700255583 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56100125f0 con 0x7f5624072780 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.794+0000 7f5621600640 1 -- 172.21.15.40:0/700255583 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5610019070 con 0x7f5624072780 2024-04-26T23:08:39.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.794+0000 7f5621600640 1 -- 172.21.15.40:0/700255583 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56100129f0 con 0x7f5624072780 2024-04-26T23:08:39.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.794+0000 7f562c000640 1 -- 172.21.15.40:0/700255583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56240ca640 con 0x7f5624072780 2024-04-26T23:08:39.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.794+0000 7f562c000640 1 -- 172.21.15.40:0/700255583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56240c9710 con 0x7f5624072780 2024-04-26T23:08:39.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.796+0000 7f562c000640 1 -- 172.21.15.40:0/700255583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f562406fba0 con 0x7f5624072780 2024-04-26T23:08:39.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.797+0000 7f5621600640 1 -- 172.21.15.40:0/700255583 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5610012b90 con 0x7f5624072780 2024-04-26T23:08:39.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.798+0000 7f5621600640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f560406c920 0x7f560406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:39.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.798+0000 7f5621600640 1 -- 172.21.15.40:0/700255583 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f561008c010 con 0x7f5624072780 2024-04-26T23:08:39.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.798+0000 7f5629600640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f560406c920 0x7f560406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:39.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.800+0000 7f5629600640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f560406c920 0x7f560406ede0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5618000f60 tx=0x7f5618009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:39.802+0000 7f5621600640 1 -- 172.21.15.40:0/700255583 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f562406fba0 con 0x7f5624072780 2024-04-26T23:08:40.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.078+0000 7f562c000640 1 -- 172.21.15.40:0/700255583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f56240694f0 con 0x7f5624072780 2024-04-26T23:08:40.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.079+0000 7f5621600640 1 -- 172.21.15.40:0/700255583 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1965 (secure 0 0 0) 0x7f561005a340 con 0x7f5624072780 2024-04-26T23:08:40.080 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:40.080 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":480,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":90,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":706274,"bytes_used":265838592,"bytes_avail":772828274688,"bytes_total":773094113280,"write_bytes_sec":682,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":102,"id":6,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":6,"rank":2,"name":"d","status":"up:active","gid":6253},{"filesystem_id":6,"rank":1,"name":"a","status":"up:active","gid":8031},{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8040}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":24,"modified":"2024-04-26T23:08:35.312595+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:08:40.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.082+0000 7f562c000640 1 -- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f560406c920 msgr2=0x7f560406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:40.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.082+0000 7f562c000640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f560406c920 0x7f560406ede0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5618000f60 tx=0x7f5618009040 comp rx=0 tx=0).stop 2024-04-26T23:08:40.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.082+0000 7f562c000640 1 -- 172.21.15.40:0/700255583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5624072780 msgr2=0x7f56240b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:40.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.082+0000 7f562c000640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5624072780 0x7f56240b5a60 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f561000d5d0 tx=0x7f561000da90 comp rx=0 tx=0).stop 2024-04-26T23:08:40.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.082+0000 7f562c000640 1 -- 172.21.15.40:0/700255583 shutdown_connections 2024-04-26T23:08:40.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.082+0000 7f562c000640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f560406c920 0x7f560406ede0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:40.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.082+0000 7f562c000640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56240cc480 0x7f56240b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:40.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.082+0000 7f562c000640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5624072780 0x7f56240b5a60 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:40.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.082+0000 7f562c000640 1 --2- 172.21.15.40:0/700255583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5624071e60 0x7f562414e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:40.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.082+0000 7f562c000640 1 -- 172.21.15.40:0/700255583 >> 172.21.15.40:0/700255583 conn(0x7f56240d0be0 msgr2=0x7f56240c7060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:40.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.082+0000 7f562c000640 1 -- 172.21.15.40:0/700255583 shutdown_connections 2024-04-26T23:08:40.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:40.083+0000 7f562c000640 1 -- 172.21.15.40:0/700255583 wait complete. 2024-04-26T23:08:40.100 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'idle+waiting paths [/d1/d2/d3]', 'c': 'idle+waiting paths [/d1/d2/d3]', 'd': 'idle+waiting paths [/d1/d2/d3]'} 2024-04-26T23:08:41.101 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:41.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.327+0000 7f9aa2e00640 1 Processor -- start 2024-04-26T23:08:41.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.327+0000 7f9aa2e00640 1 -- start start 2024-04-26T23:08:41.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.329+0000 7f9aa2e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9a9c0c6f10 0x7f9a9c0c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:41.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.329+0000 7f9aa2e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9a9c0cc980 0x7f9a9c0cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:41.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.329+0000 7f9aa2e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a9c0c6010 0x7f9a9c0cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:41.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.329+0000 7f9aa2e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f9a9c0c72f0 con 0x7f9a9c0cc980 2024-04-26T23:08:41.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.329+0000 7f9aa2e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9a9c0cb860 con 0x7f9a9c0c6f10 2024-04-26T23:08:41.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.329+0000 7f9aa2e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9a9c0cce10 con 0x7f9a9c0c6010 2024-04-26T23:08:41.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.329+0000 7f9aa1000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9a9c0cc980 0x7f9a9c0cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:41.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.329+0000 7f9aa1000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9a9c0cc980 0x7f9a9c0cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53918/0 (socket says 172.21.15.40:53918) 2024-04-26T23:08:41.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.329+0000 7f9aa1000640 1 -- 172.21.15.40:0/3168252122 learned_addr learned my addr 172.21.15.40:0/3168252122 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:41.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.330+0000 7f9aa1a00640 1 --2- 172.21.15.40:0/3168252122 >> v2:172.21.15.103:3300/0 conn(0x7f9a9c0c6f10 0x7f9a9c0c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:41.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.330+0000 7f9aa1000640 1 -- 172.21.15.40:0/3168252122 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a9c0c6010 msgr2=0x7f9a9c0cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:41.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.330+0000 7f9aa1000640 1 --2- 172.21.15.40:0/3168252122 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a9c0c6010 0x7f9a9c0cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.330+0000 7f9aa1000640 1 -- 172.21.15.40:0/3168252122 >> v2:172.21.15.103:3300/0 conn(0x7f9a9c0c6f10 msgr2=0x7f9a9c0c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:41.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.330+0000 7f9aa1000640 1 --2- 172.21.15.40:0/3168252122 >> v2:172.21.15.103:3300/0 conn(0x7f9a9c0c6f10 0x7f9a9c0c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.330+0000 7f9aa1000640 1 -- 172.21.15.40:0/3168252122 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9c0cd0f0 con 0x7f9a9c0cc980 2024-04-26T23:08:41.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.331+0000 7f9aa1000640 1 --2- 172.21.15.40:0/3168252122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a9c0cc980 0x7f9a9c0cbcf0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0ce590 tx=0x7f9a8802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74667108117583f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:41.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.331+0000 7f9a9be00640 1 -- 172.21.15.40:0/3168252122 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88037070 con 0x7f9a9c0cc980 2024-04-26T23:08:41.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.332+0000 7f9a9be00640 1 -- 172.21.15.40:0/3168252122 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a88032440 con 0x7f9a9c0cc980 2024-04-26T23:08:41.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.332+0000 7f9a9be00640 1 -- 172.21.15.40:0/3168252122 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88032810 con 0x7f9a9c0cc980 2024-04-26T23:08:41.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.332+0000 7f9aa2e00640 1 -- 172.21.15.40:0/3168252122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a9c0cc980 msgr2=0x7f9a9c0cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:41.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.332+0000 7f9aa2e00640 1 --2- 172.21.15.40:0/3168252122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a9c0cc980 0x7f9a9c0cbcf0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0ce590 tx=0x7f9a8802a140 comp rx=0 tx=0).stop 2024-04-26T23:08:41.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.332+0000 7f9aa2e00640 1 -- 172.21.15.40:0/3168252122 shutdown_connections 2024-04-26T23:08:41.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.332+0000 7f9aa2e00640 1 --2- 172.21.15.40:0/3168252122 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a9c0c6010 0x7f9a9c0cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.332+0000 7f9aa2e00640 1 --2- 172.21.15.40:0/3168252122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a9c0cc980 0x7f9a9c0cbcf0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.332+0000 7f9aa2e00640 1 --2- 172.21.15.40:0/3168252122 >> v2:172.21.15.103:3300/0 conn(0x7f9a9c0c6f10 0x7f9a9c0c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.332+0000 7f9aa2e00640 1 -- 172.21.15.40:0/3168252122 >> 172.21.15.40:0/3168252122 conn(0x7f9a9c0d0a10 msgr2=0x7f9a9c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:41.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.332+0000 7f9aa2e00640 1 -- 172.21.15.40:0/3168252122 shutdown_connections 2024-04-26T23:08:41.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.333+0000 7f9aa2e00640 1 -- 172.21.15.40:0/3168252122 wait complete. 2024-04-26T23:08:41.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.334+0000 7f9aa2e00640 1 Processor -- start 2024-04-26T23:08:41.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.335+0000 7f9aa2e00640 1 -- start start 2024-04-26T23:08:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.335+0000 7f9aa2e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a9c0c6010 0x7f9a9c1504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.335+0000 7f9aa2e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a9c0c6f10 0x7f9a9c0b5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa2e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a9c0b5790 0x7f9a9c0c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa2e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c0c76a0 con 0x7f9a9c0c6010 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa2e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c0cb1a0 con 0x7f9a9c0c6f10 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa2e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9a9c0c7820 con 0x7f9a9c0b5790 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a9c0c6010 0x7f9a9c1504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a9c0c6010 0x7f9a9c1504c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53934/0 (socket says 172.21.15.40:53934) 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa1a00640 1 -- 172.21.15.40:0/2468898533 learned_addr learned my addr 172.21.15.40:0/2468898533 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa1000640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a9c0c6f10 0x7f9a9c0b5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa2400640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a9c0b5790 0x7f9a9c0c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa1a00640 1 -- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a9c0b5790 msgr2=0x7f9a9c0c8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa1a00640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a9c0b5790 0x7f9a9c0c8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa1a00640 1 -- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a9c0c6f10 msgr2=0x7f9a9c0b5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa1a00640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a9c0c6f10 0x7f9a9c0b5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa1a00640 1 -- 172.21.15.40:0/2468898533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9c1f50a0 con 0x7f9a9c0c6010 2024-04-26T23:08:41.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa2400640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a9c0b5790 0x7f9a9c0c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:41.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.336+0000 7f9aa1000640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a9c0c6f10 0x7f9a9c0b5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:41.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.337+0000 7f9aa1a00640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a9c0c6010 0x7f9a9c1504c0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f9a9000d830 tx=0x7f9a9000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:41.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.337+0000 7f9a99600640 1 -- 172.21.15.40:0/2468898533 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9000b6c0 con 0x7f9a9c0c6010 2024-04-26T23:08:41.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.337+0000 7f9a99600640 1 -- 172.21.15.40:0/2468898533 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a90019070 con 0x7f9a9c0c6010 2024-04-26T23:08:41.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.337+0000 7f9a99600640 1 -- 172.21.15.40:0/2468898533 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9000ba80 con 0x7f9a9c0c6010 2024-04-26T23:08:41.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.337+0000 7f9aa2e00640 1 -- 172.21.15.40:0/2468898533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9c0c7da0 con 0x7f9a9c0c6010 2024-04-26T23:08:41.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.337+0000 7f9aa2e00640 1 -- 172.21.15.40:0/2468898533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a9c0c8360 con 0x7f9a9c0c6010 2024-04-26T23:08:41.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.338+0000 7f9aa2e00640 1 -- 172.21.15.40:0/2468898533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a9c069370 con 0x7f9a9c0c6010 2024-04-26T23:08:41.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.339+0000 7f9a99600640 1 -- 172.21.15.40:0/2468898533 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a9001e460 con 0x7f9a9c0c6010 2024-04-26T23:08:41.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.340+0000 7f9a99600640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a7c06c920 0x7f9a7c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:41.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.340+0000 7f9a99600640 1 -- 172.21.15.40:0/2468898533 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f9a9008c3a0 con 0x7f9a9c0c6010 2024-04-26T23:08:41.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.341+0000 7f9aa1000640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a7c06c920 0x7f9a7c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:41.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.342+0000 7f9aa1000640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a7c06c920 0x7f9a7c06ede0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9a88034000 tx=0x7f9a8802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:41.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.345+0000 7f9a99600640 1 -- 172.21.15.40:0/2468898533 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9a9005a630 con 0x7f9a9c0c6010 2024-04-26T23:08:41.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.591+0000 7f9aa2e00640 1 -- 172.21.15.40:0/2468898533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a9c15e940 con 0x7f9a9c0c6010 2024-04-26T23:08:41.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.592+0000 7f9a99600640 1 -- 172.21.15.40:0/2468898533 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5232 (secure 0 0 0) 0x7f9a90059f30 con 0x7f9a9c0c6010 2024-04-26T23:08:41.593 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:41.593 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:26.911596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:41.593 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-04-26T23:08:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.595+0000 7f9aa2e00640 1 -- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a7c06c920 msgr2=0x7f9a7c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.595+0000 7f9aa2e00640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a7c06c920 0x7f9a7c06ede0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9a88034000 tx=0x7f9a8802e000 comp rx=0 tx=0).stop 2024-04-26T23:08:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.595+0000 7f9aa2e00640 1 -- 172.21.15.40:0/2468898533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a9c0c6010 msgr2=0x7f9a9c1504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.595+0000 7f9aa2e00640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a9c0c6010 0x7f9a9c1504c0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f9a9000d830 tx=0x7f9a9000dcf0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.595+0000 7f9aa2e00640 1 -- 172.21.15.40:0/2468898533 shutdown_connections 2024-04-26T23:08:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.595+0000 7f9aa2e00640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a7c06c920 0x7f9a7c06ede0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.595+0000 7f9aa2e00640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a9c0b5790 0x7f9a9c0c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.595+0000 7f9aa2e00640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a9c0c6f10 0x7f9a9c0b5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.595+0000 7f9aa2e00640 1 --2- 172.21.15.40:0/2468898533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a9c0c6010 0x7f9a9c1504c0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.595+0000 7f9aa2e00640 1 -- 172.21.15.40:0/2468898533 >> 172.21.15.40:0/2468898533 conn(0x7f9a9c0d0a10 msgr2=0x7f9a9c06e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.595+0000 7f9aa2e00640 1 -- 172.21.15.40:0/2468898533 shutdown_connections 2024-04-26T23:08:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.595+0000 7f9aa2e00640 1 -- 172.21.15.40:0/2468898533 wait complete. 2024-04-26T23:08:41.614 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'd'] 2024-04-26T23:08:41.614 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:08:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.821+0000 7f13fbe00640 1 Processor -- start 2024-04-26T23:08:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.822+0000 7f13fbe00640 1 -- start start 2024-04-26T23:08:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.823+0000 7f13fbe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f13fc0d17d0 0x7f13fc0b6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.823+0000 7f13fbe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13fc0b66c0 con 0x7f13fc0d0c80 2024-04-26T23:08:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.823+0000 7f13fbe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13fc15f9c0 con 0x7f13fc0d2400 2024-04-26T23:08:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.823+0000 7f13fbe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f13fc0d37b0 con 0x7f13fc0d17d0 2024-04-26T23:08:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.823+0000 7f13fb400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f13fc0d0c80 0x7f13fc0b5210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39054/0 (socket says 172.21.15.40:39054) 2024-04-26T23:08:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.823+0000 7f13fb400640 1 -- 172.21.15.40:0/1987535158 learned_addr learned my addr 172.21.15.40:0/1987535158 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.823+0000 7f13faa00640 1 --2- 172.21.15.40:0/1987535158 >> v2:172.21.15.40:3300/0 conn(0x7f13fc0d17d0 0x7f13fc0b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.824+0000 7f13faa00640 1 -- 172.21.15.40:0/1987535158 >> v1:172.21.15.103:6789/0 conn(0x7f13fc0d2400 legacy=0x7f13fc0d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.824+0000 7f13faa00640 1 -- 172.21.15.40:0/1987535158 >> v1:172.21.15.40:6789/0 conn(0x7f13fc0d0c80 legacy=0x7f13fc0b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.824+0000 7f13fa000640 1 -- 172.21.15.40:0/1987535158 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2360462190 0 0) 0x7f13f4001490 con 0x7f13fc0d0c80 2024-04-26T23:08:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.824+0000 7f13faa00640 1 -- 172.21.15.40:0/1987535158 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13fc0d5730 con 0x7f13fc0d17d0 2024-04-26T23:08:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.825+0000 7f13faa00640 1 --2- 172.21.15.40:0/1987535158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13fc0d17d0 0x7f13fc0b6180 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0097d0 tx=0x7f13ec02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=281ea905495b16e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.825+0000 7f13fa000640 1 -- 172.21.15.40:0/1987535158 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec037070 con 0x7f13fc0d17d0 2024-04-26T23:08:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.826+0000 7f13fa000640 1 -- 172.21.15.40:0/1987535158 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13ec032440 con 0x7f13fc0d17d0 2024-04-26T23:08:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.826+0000 7f13fa000640 1 -- 172.21.15.40:0/1987535158 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec032810 con 0x7f13fc0d17d0 2024-04-26T23:08:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.826+0000 7f13fbe00640 1 -- 172.21.15.40:0/1987535158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13fc0d17d0 msgr2=0x7f13fc0b6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.826+0000 7f13fbe00640 1 --2- 172.21.15.40:0/1987535158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13fc0d17d0 0x7f13fc0b6180 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0097d0 tx=0x7f13ec02a1c0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.826+0000 7f13fbe00640 1 -- 172.21.15.40:0/1987535158 shutdown_connections 2024-04-26T23:08:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.826+0000 7f13fbe00640 1 --2- 172.21.15.40:0/1987535158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13fc0d17d0 0x7f13fc0b6180 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.826+0000 7f13fbe00640 1 -- 172.21.15.40:0/1987535158 >> 172.21.15.40:0/1987535158 conn(0x7f13fc09e210 msgr2=0x7f13fc0d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.826+0000 7f13fbe00640 1 -- 172.21.15.40:0/1987535158 shutdown_connections 2024-04-26T23:08:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.826+0000 7f13fbe00640 1 -- 172.21.15.40:0/1987535158 wait complete. 2024-04-26T23:08:41.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.828+0000 7f13fbe00640 1 Processor -- start 2024-04-26T23:08:41.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.828+0000 7f13fbe00640 1 -- start start 2024-04-26T23:08:41.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.829+0000 7f13fbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13fc0d0c80 0x7f13fc162720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:41.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.829+0000 7f13fbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13fc0d17d0 0x7f13fc0b9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:41.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.829+0000 7f13fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13fc0d2400 0x7f13fc1522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:41.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.829+0000 7f13fbe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f13fc0d3a00 con 0x7f13fc0d2400 2024-04-26T23:08:41.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.829+0000 7f13fbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13fc0d3d70 con 0x7f13fc0d17d0 2024-04-26T23:08:41.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.829+0000 7f13fbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f13fc0d10f0 con 0x7f13fc0d0c80 2024-04-26T23:08:41.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.830+0000 7f13fb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13fc0d2400 0x7f13fc1522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:41.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.830+0000 7f13fb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13fc0d2400 0x7f13fc1522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53944/0 (socket says 172.21.15.40:53944) 2024-04-26T23:08:41.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.830+0000 7f13fb400640 1 -- 172.21.15.40:0/1886549473 learned_addr learned my addr 172.21.15.40:0/1886549473 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.830+0000 7f13faa00640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13fc0d0c80 0x7f13fc162720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.830+0000 7f13f3e00640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13fc0d17d0 0x7f13fc0b9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.830+0000 7f13fb400640 1 -- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13fc0d0c80 msgr2=0x7f13fc162720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.830+0000 7f13fb400640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13fc0d0c80 0x7f13fc162720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.830+0000 7f13fb400640 1 -- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13fc0d17d0 msgr2=0x7f13fc0b9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.830+0000 7f13fb400640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13fc0d17d0 0x7f13fc0b9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.830+0000 7f13fb400640 1 -- 172.21.15.40:0/1886549473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13fc0b9d60 con 0x7f13fc0d2400 2024-04-26T23:08:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.830+0000 7f13f3e00640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13fc0d17d0 0x7f13fc0b9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.830+0000 7f13fb400640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13fc0d2400 0x7f13fc1522a0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f13f400c700 tx=0x7f13f400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.831+0000 7f13f2a00640 1 -- 172.21.15.40:0/1886549473 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4018070 con 0x7f13fc0d2400 2024-04-26T23:08:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.831+0000 7f13f2a00640 1 -- 172.21.15.40:0/1886549473 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13f400ce10 con 0x7f13fc0d2400 2024-04-26T23:08:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.831+0000 7f13fbe00640 1 -- 172.21.15.40:0/1886549473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13fc0ba000 con 0x7f13fc0d2400 2024-04-26T23:08:41.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.831+0000 7f13f2a00640 1 -- 172.21.15.40:0/1886549473 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f400a860 con 0x7f13fc0d2400 2024-04-26T23:08:41.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.831+0000 7f13fbe00640 1 -- 172.21.15.40:0/1886549473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13fc0ba5c0 con 0x7f13fc0d2400 2024-04-26T23:08:41.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.832+0000 7f13fbe00640 1 -- 172.21.15.40:0/1886549473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13fc0ba800 con 0x7f13fc0d2400 2024-04-26T23:08:41.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.833+0000 7f13f2a00640 1 -- 172.21.15.40:0/1886549473 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f13f401b070 con 0x7f13fc0d2400 2024-04-26T23:08:41.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.834+0000 7f13f2a00640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f13d806c8f0 0x7f13d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:41.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.834+0000 7f13f2a00640 1 -- 172.21.15.40:0/1886549473 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f13f408b260 con 0x7f13fc0d2400 2024-04-26T23:08:41.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.835+0000 7f13faa00640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f13d806c8f0 0x7f13d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:41.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.836+0000 7f13faa00640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f13d806c8f0 0x7f13d806edb0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f13ec034000 tx=0x7f13ec02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:41.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:41.839+0000 7f13f2a00640 1 -- 172.21.15.40:0/1886549473 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f13fc0b66c0 con 0x7f13fc0d2400 2024-04-26T23:08:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.122+0000 7f13fbe00640 1 -- 172.21.15.40:0/1886549473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f13fc069370 con 0x7f13fc0d2400 2024-04-26T23:08:42.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.123+0000 7f13f2a00640 1 -- 172.21.15.40:0/1886549473 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1919 (secure 0 0 0) 0x7f13f40594f0 con 0x7f13fc0d2400 2024-04-26T23:08:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:42.123 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":482,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":90,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":712040,"bytes_used":265846784,"bytes_avail":772828266496,"bytes_total":773094113280,"write_bytes_sec":1279,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":102,"id":6,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":6,"rank":2,"name":"d","status":"up:active","gid":6253},{"filesystem_id":6,"rank":1,"name":"a","status":"up:active","gid":8031},{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8040}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":25,"modified":"2024-04-26T23:08:39.314288+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:08:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.125+0000 7f13fbe00640 1 -- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f13d806c8f0 msgr2=0x7f13d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.125+0000 7f13fbe00640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f13d806c8f0 0x7f13d806edb0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f13ec034000 tx=0x7f13ec02e000 comp rx=0 tx=0).stop 2024-04-26T23:08:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.125+0000 7f13fbe00640 1 -- 172.21.15.40:0/1886549473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13fc0d2400 msgr2=0x7f13fc1522a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.125+0000 7f13fbe00640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13fc0d2400 0x7f13fc1522a0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f13f400c700 tx=0x7f13f400cbc0 comp rx=0 tx=0).stop 2024-04-26T23:08:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.126+0000 7f13fbe00640 1 -- 172.21.15.40:0/1886549473 shutdown_connections 2024-04-26T23:08:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.126+0000 7f13fbe00640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f13d806c8f0 0x7f13d806edb0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.126+0000 7f13fbe00640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13fc0d2400 0x7f13fc1522a0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.126+0000 7f13fbe00640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13fc0d17d0 0x7f13fc0b9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.126+0000 7f13fbe00640 1 --2- 172.21.15.40:0/1886549473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f13fc0d0c80 0x7f13fc162720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.126+0000 7f13fbe00640 1 -- 172.21.15.40:0/1886549473 >> 172.21.15.40:0/1886549473 conn(0x7f13fc09e210 msgr2=0x7f13fc15eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.126+0000 7f13fbe00640 1 -- 172.21.15.40:0/1886549473 shutdown_connections 2024-04-26T23:08:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:42.126+0000 7f13fbe00640 1 -- 172.21.15.40:0/1886549473 wait complete. 2024-04-26T23:08:42.146 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'idle+waiting paths [/d1/d2/d3]', 'd': 'idle+waiting paths [/d1/d2/d3]'} 2024-04-26T23:08:43.148 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:43.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.364+0000 7fd7cc600640 1 Processor -- start 2024-04-26T23:08:43.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.365+0000 7fd7cc600640 1 -- start start 2024-04-26T23:08:43.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.366+0000 7fd7cc600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd7c415d240 0x7fd7c4072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:43.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.366+0000 7fd7cc600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd7c4073510 0x7fd7c40d1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:43.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.366+0000 7fd7cc600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7c40d0390 con 0x7fd7c406c240 2024-04-26T23:08:43.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.366+0000 7fd7cc600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fd7c40d2b80 con 0x7fd7c4073510 2024-04-26T23:08:43.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.366+0000 7fd7cc600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd7c40d25c0 con 0x7fd7c415d240 2024-04-26T23:08:43.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.367+0000 7fd7ca600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd7c4073510 0x7fd7c40d1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:43.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.367+0000 7fd7ca600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd7c4073510 0x7fd7c40d1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53960/0 (socket says 172.21.15.40:53960) 2024-04-26T23:08:43.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.367+0000 7fd7ca600640 1 -- 172.21.15.40:0/759341367 learned_addr learned my addr 172.21.15.40:0/759341367 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:43.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.367+0000 7fd7c9c00640 1 --1- 172.21.15.40:0/759341367 >> v1:172.21.15.40:6789/0 conn(0x7fd7c406c240 0x7fd7c406c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39066/0 (socket says 172.21.15.40:39066) 2024-04-26T23:08:43.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.367+0000 7fd7ca600640 1 -- 172.21.15.40:0/759341367 >> v2:172.21.15.103:3300/0 conn(0x7fd7c415d240 msgr2=0x7fd7c4072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:43.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.367+0000 7fd7ca600640 1 --2- 172.21.15.40:0/759341367 >> v2:172.21.15.103:3300/0 conn(0x7fd7c415d240 0x7fd7c4072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:43.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.367+0000 7fd7ca600640 1 -- 172.21.15.40:0/759341367 >> v1:172.21.15.40:6789/0 conn(0x7fd7c406c240 legacy=0x7fd7c406c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:43.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.367+0000 7fd7ca600640 1 -- 172.21.15.40:0/759341367 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c40d28a0 con 0x7fd7c4073510 2024-04-26T23:08:43.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.367+0000 7fd7c3e00640 1 -- 172.21.15.40:0/759341367 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3300764918 0 0) 0x7fd7c40d0390 con 0x7fd7c406c240 2024-04-26T23:08:43.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.369+0000 7fd7ca600640 1 --2- 172.21.15.40:0/759341367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c4073510 0x7fd7c40d1b70 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40b7050 tx=0x7fd7bc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c5e86e5b2821a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.369+0000 7fd7c3e00640 1 -- 172.21.15.40:0/759341367 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc037070 con 0x7fd7c4073510 2024-04-26T23:08:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.369+0000 7fd7c3e00640 1 -- 172.21.15.40:0/759341367 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7bc032440 con 0x7fd7c4073510 2024-04-26T23:08:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.369+0000 7fd7c3e00640 1 -- 172.21.15.40:0/759341367 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc032810 con 0x7fd7c4073510 2024-04-26T23:08:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.369+0000 7fd7cc600640 1 -- 172.21.15.40:0/759341367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c4073510 msgr2=0x7fd7c40d1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.369+0000 7fd7cc600640 1 --2- 172.21.15.40:0/759341367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c4073510 0x7fd7c40d1b70 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40b7050 tx=0x7fd7bc02a270 comp rx=0 tx=0).stop 2024-04-26T23:08:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.370+0000 7fd7cc600640 1 -- 172.21.15.40:0/759341367 shutdown_connections 2024-04-26T23:08:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.370+0000 7fd7cc600640 1 --2- 172.21.15.40:0/759341367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c4073510 0x7fd7c40d1b70 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.370+0000 7fd7cc600640 1 --2- 172.21.15.40:0/759341367 >> v2:172.21.15.103:3300/0 conn(0x7fd7c415d240 0x7fd7c4072fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.370+0000 7fd7cc600640 1 -- 172.21.15.40:0/759341367 >> 172.21.15.40:0/759341367 conn(0x7fd7c4068090 msgr2=0x7fd7c40684a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.370+0000 7fd7cc600640 1 -- 172.21.15.40:0/759341367 shutdown_connections 2024-04-26T23:08:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.370+0000 7fd7cc600640 1 -- 172.21.15.40:0/759341367 wait complete. 2024-04-26T23:08:43.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.372+0000 7fd7cc600640 1 Processor -- start 2024-04-26T23:08:43.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.372+0000 7fd7cc600640 1 -- start start 2024-04-26T23:08:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.373+0000 7fd7cc600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c406c240 0x7fd7c415a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.373+0000 7fd7cc600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7c4073510 0x7fd7c40b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.373+0000 7fd7cc600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7c415d240 0x7fd7c40b2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.373+0000 7fd7cc600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c40cf6a0 con 0x7fd7c406c240 2024-04-26T23:08:43.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.373+0000 7fd7cc600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c40d0910 con 0x7fd7c4073510 2024-04-26T23:08:43.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.373+0000 7fd7cc600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c40cf820 con 0x7fd7c415d240 2024-04-26T23:08:43.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.373+0000 7fd7c9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c406c240 0x7fd7c415a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:43.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.373+0000 7fd7c9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c406c240 0x7fd7c415a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53972/0 (socket says 172.21.15.40:53972) 2024-04-26T23:08:43.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.373+0000 7fd7c9c00640 1 -- 172.21.15.40:0/1100470802 learned_addr learned my addr 172.21.15.40:0/1100470802 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:43.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.374+0000 7fd7c9c00640 1 -- 172.21.15.40:0/1100470802 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7c415d240 msgr2=0x7fd7c40b2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:43.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.374+0000 7fd7c9c00640 1 --2- 172.21.15.40:0/1100470802 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7c415d240 0x7fd7c40b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:43.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.374+0000 7fd7c9c00640 1 -- 172.21.15.40:0/1100470802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7c4073510 msgr2=0x7fd7c40b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:43.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.374+0000 7fd7c9c00640 1 --2- 172.21.15.40:0/1100470802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7c4073510 0x7fd7c40b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:43.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.374+0000 7fd7c9c00640 1 -- 172.21.15.40:0/1100470802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c415ad60 con 0x7fd7c406c240 2024-04-26T23:08:43.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.374+0000 7fd7c9c00640 1 --2- 172.21.15.40:0/1100470802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c406c240 0x7fd7c415a820 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800c500 tx=0x7fd7b800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:43.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.374+0000 7fd7c1600640 1 -- 172.21.15.40:0/1100470802 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8018070 con 0x7fd7c406c240 2024-04-26T23:08:43.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.374+0000 7fd7c1600640 1 -- 172.21.15.40:0/1100470802 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b8007580 con 0x7fd7c406c240 2024-04-26T23:08:43.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.374+0000 7fd7c1600640 1 -- 172.21.15.40:0/1100470802 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8004a20 con 0x7fd7c406c240 2024-04-26T23:08:43.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.374+0000 7fd7cc600640 1 -- 172.21.15.40:0/1100470802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c41f6e60 con 0x7fd7c406c240 2024-04-26T23:08:43.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.374+0000 7fd7cc600640 1 -- 172.21.15.40:0/1100470802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c41f2ae0 con 0x7fd7c406c240 2024-04-26T23:08:43.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.375+0000 7fd7cc600640 1 -- 172.21.15.40:0/1100470802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7c40694f0 con 0x7fd7c406c240 2024-04-26T23:08:43.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.376+0000 7fd7c1600640 1 -- 172.21.15.40:0/1100470802 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd7b8004bc0 con 0x7fd7c406c240 2024-04-26T23:08:43.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.378+0000 7fd7c1600640 1 --2- 172.21.15.40:0/1100470802 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd7a806c8f0 0x7fd7a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:43.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.378+0000 7fd7c1600640 1 -- 172.21.15.40:0/1100470802 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fd7b808af50 con 0x7fd7c406c240 2024-04-26T23:08:43.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.379+0000 7fd7c9200640 1 --2- 172.21.15.40:0/1100470802 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd7a806c8f0 0x7fd7a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:43.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.380+0000 7fd7c9200640 1 --2- 172.21.15.40:0/1100470802 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd7a806c8f0 0x7fd7a806edb0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd7c41f05c0 tx=0x7fd7b4009250 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:43.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.382+0000 7fd7c1600640 1 -- 172.21.15.40:0/1100470802 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd7b80591e0 con 0x7fd7c406c240 2024-04-26T23:08:43.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.631+0000 7fd7cc600640 1 -- 172.21.15.40:0/1100470802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7c41f3470 con 0x7fd7c406c240 2024-04-26T23:08:43.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.633+0000 7fd7c1600640 1 -- 172.21.15.40:0/1100470802 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5232 (secure 0 0 0) 0x7fd7b8058ae0 con 0x7fd7c406c240 2024-04-26T23:08:43.633 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:43.634 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:26.911596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:43.634 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-04-26T23:08:43.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.636+0000 7fd7cc600640 1 -- 172.21.15.40:0/1100470802 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd7a806c8f0 msgr2=0x7fd7a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:43.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.636+0000 7fd7cc600640 1 --2- 172.21.15.40:0/1100470802 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd7a806c8f0 0x7fd7a806edb0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd7c41f05c0 tx=0x7fd7b4009250 comp rx=0 tx=0).stop 2024-04-26T23:08:43.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.636+0000 7fd7cc600640 1 -- 172.21.15.40:0/1100470802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c406c240 msgr2=0x7fd7c415a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:43.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.636+0000 7fd7cc600640 1 --2- 172.21.15.40:0/1100470802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c406c240 0x7fd7c415a820 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800c500 tx=0x7fd7b800c9c0 comp rx=0 tx=0).stop 2024-04-26T23:08:43.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.636+0000 7fd7cc600640 1 -- 172.21.15.40:0/1100470802 shutdown_connections 2024-04-26T23:08:43.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.636+0000 7fd7cc600640 1 --2- 172.21.15.40:0/1100470802 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd7a806c8f0 0x7fd7a806edb0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:43.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.636+0000 7fd7cc600640 1 --2- 172.21.15.40:0/1100470802 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7c415d240 0x7fd7c40b2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:43.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.636+0000 7fd7cc600640 1 --2- 172.21.15.40:0/1100470802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7c4073510 0x7fd7c40b1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:43.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.636+0000 7fd7cc600640 1 --2- 172.21.15.40:0/1100470802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7c406c240 0x7fd7c415a820 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:43.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.636+0000 7fd7cc600640 1 -- 172.21.15.40:0/1100470802 >> 172.21.15.40:0/1100470802 conn(0x7fd7c4068090 msgr2=0x7fd7c40d0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:43.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.636+0000 7fd7cc600640 1 -- 172.21.15.40:0/1100470802 shutdown_connections 2024-04-26T23:08:43.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.636+0000 7fd7cc600640 1 -- 172.21.15.40:0/1100470802 wait complete. 2024-04-26T23:08:43.655 DEBUG:tasks.cephfs.test_multimds_misc:current active=['c', 'a', 'd'] 2024-04-26T23:08:43.655 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:08:43.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.864+0000 7f8edb200640 1 Processor -- start 2024-04-26T23:08:43.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.864+0000 7f8edb200640 1 -- start start 2024-04-26T23:08:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.865+0000 7f8edb200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8edc0d2400 0x7f8edc0d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.865+0000 7f8edb200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8edc0d20f0 con 0x7f8edc0d0c80 2024-04-26T23:08:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.865+0000 7f8edb200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8edc15f9c0 con 0x7f8edc0d17d0 2024-04-26T23:08:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.865+0000 7f8edb200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f8edc0d37b0 con 0x7f8edc0d2400 2024-04-26T23:08:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.866+0000 7f8ed9400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8edc0d2400 0x7f8edc0d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.866+0000 7f8ed9400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8edc0d2400 0x7f8edc0d1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53988/0 (socket says 172.21.15.40:53988) 2024-04-26T23:08:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.866+0000 7f8eda800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8edc0d0c80 0x7f8edc0b5210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39070/0 (socket says 172.21.15.40:39070) 2024-04-26T23:08:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.866+0000 7f8ed9400640 1 -- 172.21.15.40:0/1631704255 learned_addr learned my addr 172.21.15.40:0/1631704255 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.866+0000 7f8ed9400640 1 -- 172.21.15.40:0/1631704255 >> v1:172.21.15.103:6789/0 conn(0x7f8edc0d17d0 legacy=0x7f8edc0b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.866+0000 7f8ed9400640 1 -- 172.21.15.40:0/1631704255 >> v1:172.21.15.40:6789/0 conn(0x7f8edc0d0c80 legacy=0x7f8edc0b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.866+0000 7f8ed8a00640 1 -- 172.21.15.40:0/1631704255 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3553924019 0 0) 0x7f8edc0d20f0 con 0x7f8edc0d0c80 2024-04-26T23:08:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.866+0000 7f8ed9400640 1 -- 172.21.15.40:0/1631704255 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8edc0d5730 con 0x7f8edc0d2400 2024-04-26T23:08:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.867+0000 7f8ed9400640 1 --2- 172.21.15.40:0/1631704255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8edc0d2400 0x7f8edc0d1bb0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4009e50 tx=0x7f8ec402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d6ed2c2b80678c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.868+0000 7f8ed8a00640 1 -- 172.21.15.40:0/1631704255 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4036070 con 0x7f8edc0d2400 2024-04-26T23:08:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.868+0000 7f8ed8a00640 1 -- 172.21.15.40:0/1631704255 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ec4031600 con 0x7f8edc0d2400 2024-04-26T23:08:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.868+0000 7f8ed8a00640 1 -- 172.21.15.40:0/1631704255 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec40319d0 con 0x7f8edc0d2400 2024-04-26T23:08:43.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.868+0000 7f8edb200640 1 -- 172.21.15.40:0/1631704255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8edc0d2400 msgr2=0x7f8edc0d1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:43.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.868+0000 7f8edb200640 1 --2- 172.21.15.40:0/1631704255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8edc0d2400 0x7f8edc0d1bb0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4009e50 tx=0x7f8ec402a730 comp rx=0 tx=0).stop 2024-04-26T23:08:43.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.868+0000 7f8edb200640 1 -- 172.21.15.40:0/1631704255 shutdown_connections 2024-04-26T23:08:43.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.868+0000 7f8edb200640 1 --2- 172.21.15.40:0/1631704255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8edc0d2400 0x7f8edc0d1bb0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:43.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.868+0000 7f8edb200640 1 -- 172.21.15.40:0/1631704255 >> 172.21.15.40:0/1631704255 conn(0x7f8edc09e210 msgr2=0x7f8edc0d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:43.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.869+0000 7f8edb200640 1 -- 172.21.15.40:0/1631704255 shutdown_connections 2024-04-26T23:08:43.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.869+0000 7f8edb200640 1 -- 172.21.15.40:0/1631704255 wait complete. 2024-04-26T23:08:43.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.870+0000 7f8edb200640 1 Processor -- start 2024-04-26T23:08:43.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.871+0000 7f8edb200640 1 -- start start 2024-04-26T23:08:43.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8edb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8edc0d0c80 0x7f8edc1511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:43.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8edb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8edc0d17d0 0x7f8edc1617a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:43.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8edb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8edc0d2400 0x7f8edc0b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:43.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8edb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc0d3a00 con 0x7f8edc0d0c80 2024-04-26T23:08:43.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8edb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc0d3d70 con 0x7f8edc0d2400 2024-04-26T23:08:43.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8edb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8edc0d10d0 con 0x7f8edc0d17d0 2024-04-26T23:08:43.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8ed9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8edc0d0c80 0x7f8edc1511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:43.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8ed9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8edc0d0c80 0x7f8edc1511d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54004/0 (socket says 172.21.15.40:54004) 2024-04-26T23:08:43.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8ed9e00640 1 -- 172.21.15.40:0/1702202330 learned_addr learned my addr 172.21.15.40:0/1702202330 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:43.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8ed9e00640 1 -- 172.21.15.40:0/1702202330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8edc0d17d0 msgr2=0x7f8edc1617a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:43.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8ed9e00640 1 --2- 172.21.15.40:0/1702202330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8edc0d17d0 0x7f8edc1617a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:43.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8ed9e00640 1 -- 172.21.15.40:0/1702202330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8edc0d2400 msgr2=0x7f8edc0b4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:43.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8ed9e00640 1 --2- 172.21.15.40:0/1702202330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8edc0d2400 0x7f8edc0b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:43.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.872+0000 7f8ed9e00640 1 -- 172.21.15.40:0/1702202330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8edc151710 con 0x7f8edc0d0c80 2024-04-26T23:08:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.873+0000 7f8ed9e00640 1 --2- 172.21.15.40:0/1702202330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8edc0d0c80 0x7f8edc1511d0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00c480 tx=0x7f8ecc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.873+0000 7f8ed2000640 1 -- 172.21.15.40:0/1702202330 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc0052d0 con 0x7f8edc0d0c80 2024-04-26T23:08:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.873+0000 7f8ed2000640 1 -- 172.21.15.40:0/1702202330 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ecc005470 con 0x7f8edc0d0c80 2024-04-26T23:08:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.873+0000 7f8ed2000640 1 -- 172.21.15.40:0/1702202330 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc012870 con 0x7f8edc0d0c80 2024-04-26T23:08:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.873+0000 7f8edb200640 1 -- 172.21.15.40:0/1702202330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8edc162200 con 0x7f8edc0d0c80 2024-04-26T23:08:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.873+0000 7f8edb200640 1 -- 172.21.15.40:0/1702202330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8edc0cc1c0 con 0x7f8edc0d0c80 2024-04-26T23:08:43.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.874+0000 7f8edb200640 1 -- 172.21.15.40:0/1702202330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8edc069370 con 0x7f8edc0d0c80 2024-04-26T23:08:43.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.875+0000 7f8ed2000640 1 -- 172.21.15.40:0/1702202330 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8ecc012a10 con 0x7f8edc0d0c80 2024-04-26T23:08:43.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.876+0000 7f8ed2000640 1 --2- 172.21.15.40:0/1702202330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8eb806c920 0x7f8eb806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:43.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.876+0000 7f8ed2000640 1 -- 172.21.15.40:0/1702202330 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f8ecc08b9a0 con 0x7f8edc0d0c80 2024-04-26T23:08:43.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.877+0000 7f8ed9400640 1 --2- 172.21.15.40:0/1702202330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8eb806c920 0x7f8eb806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:43.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.878+0000 7f8ed9400640 1 --2- 172.21.15.40:0/1702202330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8eb806c920 0x7f8eb806ede0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4033000 tx=0x7f8ec402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:43.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:43.881+0000 7f8ed2000640 1 -- 172.21.15.40:0/1702202330 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8ecc059c30 con 0x7f8edc0d0c80 2024-04-26T23:08:44.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.165+0000 7f8edb200640 1 -- 172.21.15.40:0/1702202330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f8edc0d5db0 con 0x7f8edc0d0c80 2024-04-26T23:08:44.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.166+0000 7f8ed2000640 1 -- 172.21.15.40:0/1702202330 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1824 (secure 0 0 0) 0x7f8ecc059530 con 0x7f8edc0d0c80 2024-04-26T23:08:44.166 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:44.166 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":484,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":90,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":712040,"bytes_used":265846784,"bytes_avail":772828266496,"bytes_total":773094113280,"write_bytes_sec":597,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":102,"id":6,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":6,"rank":2,"name":"d","status":"up:active","gid":6253},{"filesystem_id":6,"rank":1,"name":"a","status":"up:active","gid":8031},{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8040}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":26,"modified":"2024-04-26T23:08:41.314915+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T23:08:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.169+0000 7f8edb200640 1 -- 172.21.15.40:0/1702202330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8eb806c920 msgr2=0x7f8eb806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.169+0000 7f8edb200640 1 --2- 172.21.15.40:0/1702202330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8eb806c920 0x7f8eb806ede0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4033000 tx=0x7f8ec402e000 comp rx=0 tx=0).stop 2024-04-26T23:08:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.169+0000 7f8edb200640 1 -- 172.21.15.40:0/1702202330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8edc0d0c80 msgr2=0x7f8edc1511d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.169+0000 7f8edb200640 1 --2- 172.21.15.40:0/1702202330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8edc0d0c80 0x7f8edc1511d0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00c480 tx=0x7f8ecc00c940 comp rx=0 tx=0).stop 2024-04-26T23:08:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.169+0000 7f8edb200640 1 -- 172.21.15.40:0/1702202330 shutdown_connections 2024-04-26T23:08:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.169+0000 7f8edb200640 1 --2- 172.21.15.40:0/1702202330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8eb806c920 0x7f8eb806ede0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.169+0000 7f8edb200640 1 --2- 172.21.15.40:0/1702202330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8edc0d2400 0x7f8edc0b4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.169+0000 7f8edb200640 1 --2- 172.21.15.40:0/1702202330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8edc0d17d0 0x7f8edc1617a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.169+0000 7f8edb200640 1 --2- 172.21.15.40:0/1702202330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8edc0d0c80 0x7f8edc1511d0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.169+0000 7f8edb200640 1 -- 172.21.15.40:0/1702202330 >> 172.21.15.40:0/1702202330 conn(0x7f8edc09e210 msgr2=0x7f8edc06e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.169+0000 7f8edb200640 1 -- 172.21.15.40:0/1702202330 shutdown_connections 2024-04-26T23:08:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.169+0000 7f8edb200640 1 -- 172.21.15.40:0/1702202330 wait complete. 2024-04-26T23:08:44.191 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-26T23:08:44.191 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-26T23:08:44.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:08:44.295 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:08:44.401 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-26T23:08:44.401 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:08:44.401 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:08:44.401 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:08:44.463 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:08:44.463 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:08:44.518 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:08:44.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.730+0000 7f552b600640 1 Processor -- start 2024-04-26T23:08:44.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.731+0000 7f552b600640 1 -- start start 2024-04-26T23:08:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.732+0000 7f552b600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5524156460 0x7f55240cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.732+0000 7f552b600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5524156840 con 0x7f552406fea0 2024-04-26T23:08:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.732+0000 7f552b600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55240cbf20 con 0x7f55240cba10 2024-04-26T23:08:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.732+0000 7f552b600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f55240cc200 con 0x7f5524156460 2024-04-26T23:08:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.732+0000 7f5528c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5524156460 0x7f55240cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.732+0000 7f5529600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f552406fea0 0x7f5524072c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39078/0 (socket says 172.21.15.40:39078) 2024-04-26T23:08:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.732+0000 7f5529600640 1 -- 172.21.15.40:0/1872652709 learned_addr learned my addr 172.21.15.40:0/1872652709 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.733+0000 7f5528c00640 1 -- 172.21.15.40:0/1872652709 >> v1:172.21.15.103:6789/0 conn(0x7f55240cba10 legacy=0x7f552406f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.733+0000 7f5528c00640 1 -- 172.21.15.40:0/1872652709 >> v1:172.21.15.40:6789/0 conn(0x7f552406fea0 legacy=0x7f5524072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:44.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.733+0000 7f5528c00640 1 -- 172.21.15.40:0/1872652709 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55240cc4e0 con 0x7f5524156460 2024-04-26T23:08:44.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.734+0000 7f5528c00640 1 --2- 172.21.15.40:0/1872652709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5524156460 0x7f55240cb4d0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f5514009d90 tx=0x7f551402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=897487cddd300416 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:44.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.734+0000 7f5523400640 1 -- 172.21.15.40:0/1872652709 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5514007740 con 0x7f5524156460 2024-04-26T23:08:44.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.734+0000 7f5523400640 1 -- 172.21.15.40:0/1872652709 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55140078e0 con 0x7f5524156460 2024-04-26T23:08:44.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.734+0000 7f5523400640 1 -- 172.21.15.40:0/1872652709 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5514007cb0 con 0x7f5524156460 2024-04-26T23:08:44.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.735+0000 7f552b600640 1 -- 172.21.15.40:0/1872652709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5524156460 msgr2=0x7f55240cb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:44.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.735+0000 7f552b600640 1 --2- 172.21.15.40:0/1872652709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5524156460 0x7f55240cb4d0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f5514009d90 tx=0x7f551402a6a0 comp rx=0 tx=0).stop 2024-04-26T23:08:44.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.735+0000 7f552b600640 1 -- 172.21.15.40:0/1872652709 shutdown_connections 2024-04-26T23:08:44.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.735+0000 7f552b600640 1 --2- 172.21.15.40:0/1872652709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5524156460 0x7f55240cb4d0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:44.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.735+0000 7f552b600640 1 -- 172.21.15.40:0/1872652709 >> 172.21.15.40:0/1872652709 conn(0x7f55240d0be0 msgr2=0x7f55240d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:44.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.735+0000 7f552b600640 1 -- 172.21.15.40:0/1872652709 shutdown_connections 2024-04-26T23:08:44.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.735+0000 7f552b600640 1 -- 172.21.15.40:0/1872652709 wait complete. 2024-04-26T23:08:44.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.737+0000 7f552b600640 1 Processor -- start 2024-04-26T23:08:44.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.737+0000 7f552b600640 1 -- start start 2024-04-26T23:08:44.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f552b600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f552406fea0 0x7f55241f0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:44.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f552b600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55240cba10 0x7f552415e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:44.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f552b600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55240c72b0 0x7f5524152870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:44.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f552b600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f552406e260 con 0x7f55240c72b0 2024-04-26T23:08:44.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f552b600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5524072700 con 0x7f55240cba10 2024-04-26T23:08:44.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f552b600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f552406e3e0 con 0x7f552406fea0 2024-04-26T23:08:44.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f5529600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55240c72b0 0x7f5524152870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f5529600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55240c72b0 0x7f5524152870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54016/0 (socket says 172.21.15.40:54016) 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f5529600640 1 -- 172.21.15.40:0/2120196823 learned_addr learned my addr 172.21.15.40:0/2120196823 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f5529600640 1 -- 172.21.15.40:0/2120196823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f552406fea0 msgr2=0x7f55241f0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f5529600640 1 --2- 172.21.15.40:0/2120196823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f552406fea0 0x7f55241f0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f5529600640 1 -- 172.21.15.40:0/2120196823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55240cba10 msgr2=0x7f552415e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f5529600640 1 --2- 172.21.15.40:0/2120196823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55240cba10 0x7f552415e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.738+0000 7f5529600640 1 -- 172.21.15.40:0/2120196823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552415d450 con 0x7f55240c72b0 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.739+0000 7f5529600640 1 --2- 172.21.15.40:0/2120196823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55240c72b0 0x7f5524152870 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f551c00c420 tx=0x7f551c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.739+0000 7f5520c00640 1 -- 172.21.15.40:0/2120196823 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c017070 con 0x7f55240c72b0 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.739+0000 7f5520c00640 1 -- 172.21.15.40:0/2120196823 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f551c005ce0 con 0x7f55240c72b0 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.739+0000 7f5520c00640 1 -- 172.21.15.40:0/2120196823 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c0072d0 con 0x7f55240c72b0 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.739+0000 7f552b600640 1 -- 172.21.15.40:0/2120196823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f552415d730 con 0x7f55240c72b0 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.739+0000 7f552b600640 1 -- 172.21.15.40:0/2120196823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f552415dcf0 con 0x7f55240c72b0 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.740+0000 7f552b600640 1 -- 172.21.15.40:0/2120196823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55240694f0 con 0x7f55240c72b0 2024-04-26T23:08:44.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.741+0000 7f5520c00640 1 -- 172.21.15.40:0/2120196823 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f551c01a070 con 0x7f55240c72b0 2024-04-26T23:08:44.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.742+0000 7f5520c00640 1 --2- 172.21.15.40:0/2120196823 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f550006c920 0x7f550006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:44.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.742+0000 7f5520c00640 1 -- 172.21.15.40:0/2120196823 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f551c08af50 con 0x7f55240c72b0 2024-04-26T23:08:44.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.743+0000 7f5528c00640 1 --2- 172.21.15.40:0/2120196823 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f550006c920 0x7f550006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:44.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.745+0000 7f5528c00640 1 --2- 172.21.15.40:0/2120196823 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f550006c920 0x7f550006ede0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5514034000 tx=0x7f551402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:44.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.747+0000 7f5520c00640 1 -- 172.21.15.40:0/2120196823 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f551c058b90 con 0x7f55240c72b0 2024-04-26T23:08:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.960+0000 7f552b600640 1 -- 172.21.15.40:0/2120196823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55240c3e50 con 0x7f55240c72b0 2024-04-26T23:08:44.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.961+0000 7f5520c00640 1 -- 172.21.15.40:0/2120196823 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+16849 (secure 0 0 0) 0x7f551c05ce90 con 0x7f55240c72b0 2024-04-26T23:08:44.962 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:44.962 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:07:49.778062+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:07:46.137238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-26T23:07:47.258518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:08:44.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.963+0000 7f552b600640 1 -- 172.21.15.40:0/2120196823 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f550006c920 msgr2=0x7f550006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:44.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.963+0000 7f552b600640 1 --2- 172.21.15.40:0/2120196823 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f550006c920 0x7f550006ede0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5514034000 tx=0x7f551402e000 comp rx=0 tx=0).stop 2024-04-26T23:08:44.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.963+0000 7f552b600640 1 -- 172.21.15.40:0/2120196823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55240c72b0 msgr2=0x7f5524152870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:44.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.963+0000 7f552b600640 1 --2- 172.21.15.40:0/2120196823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55240c72b0 0x7f5524152870 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f551c00c420 tx=0x7f551c00c8e0 comp rx=0 tx=0).stop 2024-04-26T23:08:44.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.964+0000 7f552b600640 1 -- 172.21.15.40:0/2120196823 shutdown_connections 2024-04-26T23:08:44.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.964+0000 7f552b600640 1 --2- 172.21.15.40:0/2120196823 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f550006c920 0x7f550006ede0 secure :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5514034000 tx=0x7f551402e000 comp rx=0 tx=0).stop 2024-04-26T23:08:44.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.964+0000 7f552b600640 1 --2- 172.21.15.40:0/2120196823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55240c72b0 0x7f5524152870 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:44.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.964+0000 7f552b600640 1 --2- 172.21.15.40:0/2120196823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55240cba10 0x7f552415e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:44.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.964+0000 7f552b600640 1 --2- 172.21.15.40:0/2120196823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f552406fea0 0x7f55241f0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:44.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.964+0000 7f552b600640 1 -- 172.21.15.40:0/2120196823 >> 172.21.15.40:0/2120196823 conn(0x7f55240d0be0 msgr2=0x7f55240ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:44.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.964+0000 7f552b600640 1 -- 172.21.15.40:0/2120196823 shutdown_connections 2024-04-26T23:08:44.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:44.964+0000 7f552b600640 1 -- 172.21.15.40:0/2120196823 wait complete. 2024-04-26T23:08:44.985 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:07:46.137238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:07:47.258518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:08:44.986 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:08:45.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.203+0000 7f7d95600640 1 Processor -- start 2024-04-26T23:08:45.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.203+0000 7f7d95600640 1 -- start start 2024-04-26T23:08:45.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.204+0000 7f7d95600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7d90073600 0x7f7d90072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.204+0000 7f7d95600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d901604e0 con 0x7f7d9015fe80 2024-04-26T23:08:45.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.204+0000 7f7d95600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d90072200 con 0x7f7d900739e0 2024-04-26T23:08:45.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.204+0000 7f7d95600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7d90073220 con 0x7f7d90073600 2024-04-26T23:08:45.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.205+0000 7f7d94c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7d9015fe80 0x7f7d900ae8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60456/0 (socket says 172.21.15.40:60456) 2024-04-26T23:08:45.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.205+0000 7f7d94c00640 1 -- 172.21.15.40:0/3022442312 learned_addr learned my addr 172.21.15.40:0/3022442312 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:45.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.205+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 893899209 0 0) 0x7f7d901604e0 con 0x7f7d9015fe80 2024-04-26T23:08:45.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.205+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d74003520 con 0x7f7d9015fe80 2024-04-26T23:08:45.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.205+0000 7f7d8fe00640 1 --2- 172.21.15.40:0/3022442312 >> v2:172.21.15.103:3300/0 conn(0x7f7d90073600 0x7f7d90072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:45.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.205+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2924807318 0 0) 0x7f7d74003520 con 0x7f7d9015fe80 2024-04-26T23:08:45.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.205+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 >> v2:172.21.15.103:3300/0 conn(0x7f7d90073600 msgr2=0x7f7d90072ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:45.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.205+0000 7f7d8f400640 1 --2- 172.21.15.40:0/3022442312 >> v2:172.21.15.103:3300/0 conn(0x7f7d90073600 0x7f7d90072ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.205+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 >> v1:172.21.15.103:6789/0 conn(0x7f7d900739e0 legacy=0x7f7d9015f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:45.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d900724e0 con 0x7f7d9015fe80 2024-04-26T23:08:45.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f7d880019f0 con 0x7f7d9015fe80 2024-04-26T23:08:45.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 >> v1:172.21.15.40:6789/0 conn(0x7f7d9015fe80 legacy=0x7f7d900ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:45.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d8f400640 1 --2- 172.21.15.40:0/3022442312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d740039b0 0x7f7d74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d8f400640 1 --2- 172.21.15.40:0/3022442312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d74004800 0x7f7d74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d8f400640 1 --2- 172.21.15.40:0/3022442312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740051c0 0x7f7d7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7d90073220 con 0x7f7d740051c0 2024-04-26T23:08:45.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7400c110 con 0x7f7d740039b0 2024-04-26T23:08:45.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7d7400c310 con 0x7f7d74004800 2024-04-26T23:08:45.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d94c00640 1 --2- 172.21.15.40:0/3022442312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740051c0 0x7f7d7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d8fe00640 1 --2- 172.21.15.40:0/3022442312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d740039b0 0x7f7d74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d94c00640 1 -- 172.21.15.40:0/3022442312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d74004800 msgr2=0x7f7d74004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d94c00640 1 --2- 172.21.15.40:0/3022442312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d74004800 0x7f7d74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d94c00640 1 -- 172.21.15.40:0/3022442312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d740039b0 msgr2=0x7f7d74003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d94c00640 1 --2- 172.21.15.40:0/3022442312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d740039b0 0x7f7d74003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.206+0000 7f7d94c00640 1 -- 172.21.15.40:0/3022442312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d74003520 con 0x7f7d740051c0 2024-04-26T23:08:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.207+0000 7f7d94c00640 1 -- 172.21.15.40:0/3022442312 reap_dead start 2024-04-26T23:08:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.208+0000 7f7d94c00640 1 --2- 172.21.15.40:0/3022442312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740051c0 0x7f7d7400b9c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800a850 tx=0x7f7d74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351452ea3e4eb108 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.208+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8800a4a0 con 0x7f7d740051c0 2024-04-26T23:08:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.208+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d8802f070 con 0x7f7d740051c0 2024-04-26T23:08:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.208+0000 7f7d8f400640 1 -- 172.21.15.40:0/3022442312 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8802bce0 con 0x7f7d740051c0 2024-04-26T23:08:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.208+0000 7f7d95600640 1 -- 172.21.15.40:0/3022442312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740051c0 msgr2=0x7f7d7400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.208+0000 7f7d95600640 1 --2- 172.21.15.40:0/3022442312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740051c0 0x7f7d7400b9c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800a850 tx=0x7f7d74005830 comp rx=0 tx=0).stop 2024-04-26T23:08:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.209+0000 7f7d95600640 1 -- 172.21.15.40:0/3022442312 shutdown_connections 2024-04-26T23:08:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.209+0000 7f7d95600640 1 --2- 172.21.15.40:0/3022442312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740051c0 0x7f7d7400b9c0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.209+0000 7f7d95600640 1 -- 172.21.15.40:0/3022442312 >> 172.21.15.40:0/3022442312 conn(0x7f7d900b2720 msgr2=0x7f7d900b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.209+0000 7f7d95600640 1 -- 172.21.15.40:0/3022442312 shutdown_connections 2024-04-26T23:08:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.209+0000 7f7d95600640 1 -- 172.21.15.40:0/3022442312 wait complete. 2024-04-26T23:08:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.211+0000 7f7d95600640 1 Processor -- start 2024-04-26T23:08:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.211+0000 7f7d95600640 1 -- start start 2024-04-26T23:08:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.212+0000 7f7d95600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740051c0 0x7f7d9014a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.212+0000 7f7d95600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d90147d20 0x7f7d90148180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.212+0000 7f7d95600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d90161810 0x7f7d90161ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.212+0000 7f7d95600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7d90160290 con 0x7f7d740051c0 2024-04-26T23:08:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.212+0000 7f7d95600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7d90072850 con 0x7f7d90147d20 2024-04-26T23:08:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.212+0000 7f7d95600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7d90071e20 con 0x7f7d90161810 2024-04-26T23:08:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.212+0000 7f7d8fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740051c0 0x7f7d9014a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.213+0000 7f7d8fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740051c0 0x7f7d9014a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55756/0 (socket says 172.21.15.40:55756) 2024-04-26T23:08:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.213+0000 7f7d8fe00640 1 -- 172.21.15.40:0/4169232190 learned_addr learned my addr 172.21.15.40:0/4169232190 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.213+0000 7f7d87400640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d90147d20 0x7f7d90148180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.213+0000 7f7d94c00640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d90161810 0x7f7d90161ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.213+0000 7f7d8fe00640 1 -- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d90161810 msgr2=0x7f7d90161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.213+0000 7f7d8fe00640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d90161810 0x7f7d90161ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.213+0000 7f7d8fe00640 1 -- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d90147d20 msgr2=0x7f7d90148180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.213+0000 7f7d8fe00640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d90147d20 0x7f7d90148180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.213+0000 7f7d8fe00640 1 -- 172.21.15.40:0/4169232190 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d901486c0 con 0x7f7d740051c0 2024-04-26T23:08:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.213+0000 7f7d94c00640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d90161810 0x7f7d90161ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.213+0000 7f7d8fe00640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740051c0 0x7f7d9014a020 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000f580 tx=0x7f7d8000fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.214+0000 7f7d8cc00640 1 -- 172.21.15.40:0/4169232190 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80013810 con 0x7f7d740051c0 2024-04-26T23:08:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.214+0000 7f7d8cc00640 1 -- 172.21.15.40:0/4169232190 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d8001a070 con 0x7f7d740051c0 2024-04-26T23:08:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.214+0000 7f7d8cc00640 1 -- 172.21.15.40:0/4169232190 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80013bb0 con 0x7f7d740051c0 2024-04-26T23:08:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.214+0000 7f7d95600640 1 -- 172.21.15.40:0/4169232190 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d90148950 con 0x7f7d740051c0 2024-04-26T23:08:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.214+0000 7f7d95600640 1 -- 172.21.15.40:0/4169232190 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d901491f0 con 0x7f7d740051c0 2024-04-26T23:08:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.215+0000 7f7d95600640 1 -- 172.21.15.40:0/4169232190 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d90069370 con 0x7f7d740051c0 2024-04-26T23:08:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.216+0000 7f7d8cc00640 1 -- 172.21.15.40:0/4169232190 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7d80012070 con 0x7f7d740051c0 2024-04-26T23:08:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.218+0000 7f7d8cc00640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7d6c06c8f0 0x7f7d6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.218+0000 7f7d8cc00640 1 -- 172.21.15.40:0/4169232190 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7f7d8008be10 con 0x7f7d740051c0 2024-04-26T23:08:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.218+0000 7f7d87400640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7d6c06c8f0 0x7f7d6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.220+0000 7f7d87400640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7d6c06c8f0 0x7f7d6c06edb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7d90163410 tx=0x7f7d78009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:45.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.223+0000 7f7d8cc00640 1 -- 172.21.15.40:0/4169232190 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7d8008f030 con 0x7f7d740051c0 2024-04-26T23:08:45.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.431+0000 7f7d95600640 1 -- 172.21.15.40:0/4169232190 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7d900b3480 con 0x7f7d740051c0 2024-04-26T23:08:45.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.432+0000 7f7d8cc00640 1 -- 172.21.15.40:0/4169232190 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f7d8001c310 con 0x7f7d740051c0 2024-04-26T23:08:45.432 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:08:45.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.435+0000 7f7d95600640 1 -- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7d6c06c8f0 msgr2=0x7f7d6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:45.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.435+0000 7f7d95600640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7d6c06c8f0 0x7f7d6c06edb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7d90163410 tx=0x7f7d78009040 comp rx=0 tx=0).stop 2024-04-26T23:08:45.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.435+0000 7f7d95600640 1 -- 172.21.15.40:0/4169232190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740051c0 msgr2=0x7f7d9014a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:45.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.435+0000 7f7d95600640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740051c0 0x7f7d9014a020 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000f580 tx=0x7f7d8000fa40 comp rx=0 tx=0).stop 2024-04-26T23:08:45.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.435+0000 7f7d95600640 1 -- 172.21.15.40:0/4169232190 shutdown_connections 2024-04-26T23:08:45.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.435+0000 7f7d95600640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7d6c06c8f0 0x7f7d6c06edb0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.435+0000 7f7d95600640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7d90161810 0x7f7d90161ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.435+0000 7f7d95600640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d90147d20 0x7f7d90148180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.435+0000 7f7d95600640 1 --2- 172.21.15.40:0/4169232190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740051c0 0x7f7d9014a020 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.435+0000 7f7d95600640 1 -- 172.21.15.40:0/4169232190 >> 172.21.15.40:0/4169232190 conn(0x7f7d900b2720 msgr2=0x7f7d900b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:45.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.435+0000 7f7d95600640 1 -- 172.21.15.40:0/4169232190 shutdown_connections 2024-04-26T23:08:45.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.435+0000 7f7d95600640 1 -- 172.21.15.40:0/4169232190 wait complete. 2024-04-26T23:08:45.456 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:08:45.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.672+0000 7faa83e00640 1 Processor -- start 2024-04-26T23:08:45.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.673+0000 7faa83e00640 1 -- start start 2024-04-26T23:08:45.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.674+0000 7faa83e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faa8406f420 0x7faa841644c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.674+0000 7faa83e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa840b4e60 0x7faa84071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.674+0000 7faa83e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8406f800 con 0x7faa84072220 2024-04-26T23:08:45.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.674+0000 7faa83e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7faa84072670 con 0x7faa8406f420 2024-04-26T23:08:45.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.674+0000 7faa83e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faa84164a00 con 0x7faa840b4e60 2024-04-26T23:08:45.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.675+0000 7faa82a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faa8406f420 0x7faa841644c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:45.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.675+0000 7faa82a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faa8406f420 0x7faa841644c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55764/0 (socket says 172.21.15.40:55764) 2024-04-26T23:08:45.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.675+0000 7faa82a00640 1 -- 172.21.15.40:0/1840169231 learned_addr learned my addr 172.21.15.40:0/1840169231 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:45.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.675+0000 7faa82000640 1 --2- 172.21.15.40:0/1840169231 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa840b4e60 0x7faa84071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:45.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.675+0000 7faa82a00640 1 -- 172.21.15.40:0/1840169231 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa840b4e60 msgr2=0x7faa84071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:45.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.675+0000 7faa82a00640 1 --2- 172.21.15.40:0/1840169231 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa840b4e60 0x7faa84071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.675+0000 7faa82a00640 1 -- 172.21.15.40:0/1840169231 >> v1:172.21.15.103:6789/0 conn(0x7faa84072220 legacy=0x7faa8406dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:45.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.675+0000 7faa82a00640 1 -- 172.21.15.40:0/1840169231 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa840b3dc0 con 0x7faa8406f420 2024-04-26T23:08:45.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.676+0000 7faa82a00640 1 --2- 172.21.15.40:0/1840169231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8406f420 0x7faa841644c0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7faa74009870 tx=0x7faa7402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17238c24e0fdcb15 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:45.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.677+0000 7faa81600640 1 -- 172.21.15.40:0/1840169231 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa74037070 con 0x7faa8406f420 2024-04-26T23:08:45.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.677+0000 7faa81600640 1 -- 172.21.15.40:0/1840169231 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa74032440 con 0x7faa8406f420 2024-04-26T23:08:45.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.677+0000 7faa81600640 1 -- 172.21.15.40:0/1840169231 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa74032810 con 0x7faa8406f420 2024-04-26T23:08:45.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.677+0000 7faa83e00640 1 -- 172.21.15.40:0/1840169231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8406f420 msgr2=0x7faa841644c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:45.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.677+0000 7faa83e00640 1 --2- 172.21.15.40:0/1840169231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8406f420 0x7faa841644c0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7faa74009870 tx=0x7faa7402a210 comp rx=0 tx=0).stop 2024-04-26T23:08:45.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.678+0000 7faa83e00640 1 -- 172.21.15.40:0/1840169231 shutdown_connections 2024-04-26T23:08:45.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.678+0000 7faa83e00640 1 --2- 172.21.15.40:0/1840169231 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa840b4e60 0x7faa84071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.678+0000 7faa83e00640 1 --2- 172.21.15.40:0/1840169231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8406f420 0x7faa841644c0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.678+0000 7faa83e00640 1 -- 172.21.15.40:0/1840169231 >> 172.21.15.40:0/1840169231 conn(0x7faa840b2720 msgr2=0x7faa840b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:45.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.678+0000 7faa83e00640 1 -- 172.21.15.40:0/1840169231 shutdown_connections 2024-04-26T23:08:45.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.678+0000 7faa83e00640 1 -- 172.21.15.40:0/1840169231 wait complete. 2024-04-26T23:08:45.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.679+0000 7faa83e00640 1 Processor -- start 2024-04-26T23:08:45.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.680+0000 7faa83e00640 1 -- start start 2024-04-26T23:08:45.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.680+0000 7faa83e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8406f420 0x7faa840ab180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.680+0000 7faa83e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa84072220 0x7faa8415cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.680+0000 7faa83e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa840b4e60 0x7faa84160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.680+0000 7faa83e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa840b4660 con 0x7faa8406f420 2024-04-26T23:08:45.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.680+0000 7faa83e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa840b4380 con 0x7faa84072220 2024-04-26T23:08:45.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.680+0000 7faa83e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faa840b4940 con 0x7faa840b4e60 2024-04-26T23:08:45.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.681+0000 7faa82a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8406f420 0x7faa840ab180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:45.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.681+0000 7faa82a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8406f420 0x7faa840ab180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55766/0 (socket says 172.21.15.40:55766) 2024-04-26T23:08:45.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.681+0000 7faa82a00640 1 -- 172.21.15.40:0/1847420791 learned_addr learned my addr 172.21.15.40:0/1847420791 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:45.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.681+0000 7faa82000640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa84072220 0x7faa8415cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:45.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.681+0000 7faa83400640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa840b4e60 0x7faa84160cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:45.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.681+0000 7faa82a00640 1 -- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa840b4e60 msgr2=0x7faa84160cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:45.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.681+0000 7faa82a00640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa840b4e60 0x7faa84160cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.681+0000 7faa82a00640 1 -- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa84072220 msgr2=0x7faa8415cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:45.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.681+0000 7faa82a00640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa84072220 0x7faa8415cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.681+0000 7faa82a00640 1 -- 172.21.15.40:0/1847420791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa84161510 con 0x7faa8406f420 2024-04-26T23:08:45.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.681+0000 7faa82000640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa84072220 0x7faa8415cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:45.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.682+0000 7faa82a00640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8406f420 0x7faa840ab180 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7faa74009870 tx=0x7faa740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:45.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.682+0000 7faa7b400640 1 -- 172.21.15.40:0/1847420791 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa74041020 con 0x7faa8406f420 2024-04-26T23:08:45.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.682+0000 7faa7b400640 1 -- 172.21.15.40:0/1847420791 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa740306c0 con 0x7faa8406f420 2024-04-26T23:08:45.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.682+0000 7faa7b400640 1 -- 172.21.15.40:0/1847420791 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa74037040 con 0x7faa8406f420 2024-04-26T23:08:45.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.682+0000 7faa83e00640 1 -- 172.21.15.40:0/1847420791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa841617f0 con 0x7faa8406f420 2024-04-26T23:08:45.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.682+0000 7faa83e00640 1 -- 172.21.15.40:0/1847420791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa84161db0 con 0x7faa8406f420 2024-04-26T23:08:45.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.683+0000 7faa83e00640 1 -- 172.21.15.40:0/1847420791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa84069370 con 0x7faa8406f420 2024-04-26T23:08:45.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.684+0000 7faa7b400640 1 -- 172.21.15.40:0/1847420791 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faa74044020 con 0x7faa8406f420 2024-04-26T23:08:45.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.686+0000 7faa7b400640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faa6406c8f0 0x7faa6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:45.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.686+0000 7faa7b400640 1 -- 172.21.15.40:0/1847420791 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7faa740ac5b0 con 0x7faa8406f420 2024-04-26T23:08:45.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.687+0000 7faa82000640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faa6406c8f0 0x7faa6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:45.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.688+0000 7faa82000640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faa6406c8f0 0x7faa6406edb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7faa70006fd0 tx=0x7faa70008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:45.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.691+0000 7faa7b400640 1 -- 172.21.15.40:0/1847420791 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7faa7404b030 con 0x7faa8406f420 2024-04-26T23:08:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.903+0000 7faa83e00640 1 -- 172.21.15.40:0/1847420791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7faa84164310 con 0x7faa8406f420 2024-04-26T23:08:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.904+0000 7faa7b400640 1 -- 172.21.15.40:0/1847420791 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7faa7407a7b0 con 0x7faa8406f420 2024-04-26T23:08:45.904 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:08:45.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.907+0000 7faa83e00640 1 -- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faa6406c8f0 msgr2=0x7faa6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.907+0000 7faa83e00640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faa6406c8f0 0x7faa6406edb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7faa70006fd0 tx=0x7faa70008040 comp rx=0 tx=0).stop 2024-04-26T23:08:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.907+0000 7faa83e00640 1 -- 172.21.15.40:0/1847420791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8406f420 msgr2=0x7faa840ab180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.907+0000 7faa83e00640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8406f420 0x7faa840ab180 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7faa74009870 tx=0x7faa740046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.907+0000 7faa83e00640 1 -- 172.21.15.40:0/1847420791 shutdown_connections 2024-04-26T23:08:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.907+0000 7faa83e00640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faa6406c8f0 0x7faa6406edb0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.907+0000 7faa83e00640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa840b4e60 0x7faa84160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.907+0000 7faa83e00640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa84072220 0x7faa8415cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.907+0000 7faa83e00640 1 --2- 172.21.15.40:0/1847420791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa8406f420 0x7faa840ab180 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.907+0000 7faa83e00640 1 -- 172.21.15.40:0/1847420791 >> 172.21.15.40:0/1847420791 conn(0x7faa840b2720 msgr2=0x7faa840b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.907+0000 7faa83e00640 1 -- 172.21.15.40:0/1847420791 shutdown_connections 2024-04-26T23:08:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:45.907+0000 7faa83e00640 1 -- 172.21.15.40:0/1847420791 wait complete. 2024-04-26T23:08:45.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:08:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.144+0000 7fd496a00640 1 Processor -- start 2024-04-26T23:08:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.145+0000 7fd496a00640 1 -- start start 2024-04-26T23:08:46.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.146+0000 7fd496a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4900d39d0 0x7fd4900b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:46.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.146+0000 7fd496a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4900b8bd0 con 0x7fd4900d2ae0 2024-04-26T23:08:46.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.146+0000 7fd496a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4900b8eb0 con 0x7fd49015f1a0 2024-04-26T23:08:46.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.146+0000 7fd496a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fd4900b9190 con 0x7fd4900d39d0 2024-04-26T23:08:46.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.147+0000 7fd494c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4900d39d0 0x7fd4900b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:46.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.147+0000 7fd496000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd4900d2ae0 0x7fd4900b9c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60472/0 (socket says 172.21.15.40:60472) 2024-04-26T23:08:46.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.147+0000 7fd496000640 1 -- 172.21.15.40:0/3323324597 learned_addr learned my addr 172.21.15.40:0/3323324597 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:46.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.147+0000 7fd48fe00640 1 -- 172.21.15.40:0/3323324597 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3431462227 0 0) 0x7fd4900b8bd0 con 0x7fd4900d2ae0 2024-04-26T23:08:46.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.147+0000 7fd48fe00640 1 -- 172.21.15.40:0/3323324597 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd478003520 con 0x7fd4900d2ae0 2024-04-26T23:08:46.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.147+0000 7fd494c00640 1 -- 172.21.15.40:0/3323324597 >> v1:172.21.15.103:6789/0 conn(0x7fd49015f1a0 legacy=0x7fd4900d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:46.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.147+0000 7fd494c00640 1 -- 172.21.15.40:0/3323324597 >> v1:172.21.15.40:6789/0 conn(0x7fd4900d2ae0 legacy=0x7fd4900b9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:46.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.147+0000 7fd494c00640 1 -- 172.21.15.40:0/3323324597 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4900b9470 con 0x7fd4900d39d0 2024-04-26T23:08:46.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.149+0000 7fd494c00640 1 --2- 172.21.15.40:0/3323324597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4900d39d0 0x7fd4900b85f0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fd484009d90 tx=0x7fd48402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0c1d37d876e94b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:46.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.149+0000 7fd48fe00640 1 -- 172.21.15.40:0/3323324597 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd484007740 con 0x7fd4900d39d0 2024-04-26T23:08:46.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.149+0000 7fd48fe00640 1 -- 172.21.15.40:0/3323324597 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4840078e0 con 0x7fd4900d39d0 2024-04-26T23:08:46.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.149+0000 7fd48fe00640 1 -- 172.21.15.40:0/3323324597 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd484007c90 con 0x7fd4900d39d0 2024-04-26T23:08:46.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.150+0000 7fd496a00640 1 -- 172.21.15.40:0/3323324597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4900d39d0 msgr2=0x7fd4900b85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:46.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.150+0000 7fd496a00640 1 --2- 172.21.15.40:0/3323324597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4900d39d0 0x7fd4900b85f0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fd484009d90 tx=0x7fd48402a6f0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.150+0000 7fd496a00640 1 -- 172.21.15.40:0/3323324597 shutdown_connections 2024-04-26T23:08:46.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.150+0000 7fd496a00640 1 --2- 172.21.15.40:0/3323324597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4900d39d0 0x7fd4900b85f0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.150+0000 7fd496a00640 1 -- 172.21.15.40:0/3323324597 >> 172.21.15.40:0/3323324597 conn(0x7fd4900d4d90 msgr2=0x7fd4900d5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:46.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.150+0000 7fd496a00640 1 -- 172.21.15.40:0/3323324597 shutdown_connections 2024-04-26T23:08:46.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.150+0000 7fd496a00640 1 -- 172.21.15.40:0/3323324597 wait complete. 2024-04-26T23:08:46.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.152+0000 7fd496a00640 1 Processor -- start 2024-04-26T23:08:46.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.152+0000 7fd496a00640 1 -- start start 2024-04-26T23:08:46.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.153+0000 7fd496a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4900d2ae0 0x7fd49009ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:46.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.153+0000 7fd496a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4900d39d0 0x7fd49015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:46.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.153+0000 7fd496a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd49015f1a0 0x7fd4901615a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:46.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.153+0000 7fd496a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd49015f580 con 0x7fd4900d39d0 2024-04-26T23:08:46.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.153+0000 7fd496a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd4900d0bb0 con 0x7fd4900d2ae0 2024-04-26T23:08:46.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.153+0000 7fd496a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd49015f700 con 0x7fd49015f1a0 2024-04-26T23:08:46.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.153+0000 7fd494c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4900d39d0 0x7fd49015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:46.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.153+0000 7fd494c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4900d39d0 0x7fd49015ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55778/0 (socket says 172.21.15.40:55778) 2024-04-26T23:08:46.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.153+0000 7fd494c00640 1 -- 172.21.15.40:0/1257623764 learned_addr learned my addr 172.21.15.40:0/1257623764 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:46.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.153+0000 7fd495600640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4900d2ae0 0x7fd49009ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:46.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.153+0000 7fd496000640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd49015f1a0 0x7fd4901615a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:46.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.153+0000 7fd494c00640 1 -- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd49015f1a0 msgr2=0x7fd4901615a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:46.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.153+0000 7fd494c00640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd49015f1a0 0x7fd4901615a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.154+0000 7fd494c00640 1 -- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4900d2ae0 msgr2=0x7fd49009ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:46.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.154+0000 7fd494c00640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4900d2ae0 0x7fd49009ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.154+0000 7fd494c00640 1 -- 172.21.15.40:0/1257623764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4900a1ed0 con 0x7fd4900d39d0 2024-04-26T23:08:46.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.154+0000 7fd494c00640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4900d39d0 0x7fd49015ae40 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fd484009d90 tx=0x7fd484004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:46.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.154+0000 7fd48d600640 1 -- 172.21.15.40:0/1257623764 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd484009450 con 0x7fd4900d39d0 2024-04-26T23:08:46.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.154+0000 7fd48d600640 1 -- 172.21.15.40:0/1257623764 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd484004240 con 0x7fd4900d39d0 2024-04-26T23:08:46.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.154+0000 7fd496a00640 1 -- 172.21.15.40:0/1257623764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4900a2170 con 0x7fd4900d39d0 2024-04-26T23:08:46.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.154+0000 7fd48d600640 1 -- 172.21.15.40:0/1257623764 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4840045f0 con 0x7fd4900d39d0 2024-04-26T23:08:46.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.154+0000 7fd496a00640 1 -- 172.21.15.40:0/1257623764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4900a2730 con 0x7fd4900d39d0 2024-04-26T23:08:46.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.156+0000 7fd496a00640 1 -- 172.21.15.40:0/1257623764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd490069370 con 0x7fd4900d39d0 2024-04-26T23:08:46.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.157+0000 7fd48d600640 1 -- 172.21.15.40:0/1257623764 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd484039040 con 0x7fd4900d39d0 2024-04-26T23:08:46.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.159+0000 7fd48d600640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd47406c8f0 0x7fd47406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:46.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.159+0000 7fd48d600640 1 -- 172.21.15.40:0/1257623764 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fd4840ad110 con 0x7fd4900d39d0 2024-04-26T23:08:46.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.160+0000 7fd495600640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd47406c8f0 0x7fd47406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:46.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.161+0000 7fd495600640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd47406c8f0 0x7fd47406edb0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd480007fd0 tx=0x7fd480009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:46.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.164+0000 7fd48d600640 1 -- 172.21.15.40:0/1257623764 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd484040030 con 0x7fd4900d39d0 2024-04-26T23:08:46.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.373+0000 7fd496a00640 1 -- 172.21.15.40:0/1257623764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd490075140 con 0x7fd4900d39d0 2024-04-26T23:08:46.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.374+0000 7fd48d600640 1 -- 172.21.15.40:0/1257623764 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fd48407b3b0 con 0x7fd4900d39d0 2024-04-26T23:08:46.374 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:08:46.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.376+0000 7fd496a00640 1 -- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd47406c8f0 msgr2=0x7fd47406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:46.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.376+0000 7fd496a00640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd47406c8f0 0x7fd47406edb0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd480007fd0 tx=0x7fd480009040 comp rx=0 tx=0).stop 2024-04-26T23:08:46.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.376+0000 7fd496a00640 1 -- 172.21.15.40:0/1257623764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4900d39d0 msgr2=0x7fd49015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:46.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.376+0000 7fd496a00640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4900d39d0 0x7fd49015ae40 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fd484009d90 tx=0x7fd484004a40 comp rx=0 tx=0).stop 2024-04-26T23:08:46.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.376+0000 7fd496a00640 1 -- 172.21.15.40:0/1257623764 shutdown_connections 2024-04-26T23:08:46.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.376+0000 7fd496a00640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd47406c8f0 0x7fd47406edb0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.376+0000 7fd496a00640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd49015f1a0 0x7fd4901615a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.376+0000 7fd496a00640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4900d39d0 0x7fd49015ae40 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.376+0000 7fd496a00640 1 --2- 172.21.15.40:0/1257623764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4900d2ae0 0x7fd49009ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.376+0000 7fd496a00640 1 -- 172.21.15.40:0/1257623764 >> 172.21.15.40:0/1257623764 conn(0x7fd4900d4d90 msgr2=0x7fd4900d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:46.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.376+0000 7fd496a00640 1 -- 172.21.15.40:0/1257623764 shutdown_connections 2024-04-26T23:08:46.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.377+0000 7fd496a00640 1 -- 172.21.15.40:0/1257623764 wait complete. 2024-04-26T23:08:46.396 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:08:46.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.603+0000 7fbad7200640 1 Processor -- start 2024-04-26T23:08:46.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.604+0000 7fbad7200640 1 -- start start 2024-04-26T23:08:46.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.605+0000 7fbad7200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbad815ce60 0x7fbad80afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:46.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.605+0000 7fbad7200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbad80af7a0 0x7fbad80ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:46.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.605+0000 7fbad7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad8071fe0 0x7fbad8073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:46.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.605+0000 7fbad7200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fbad80cc450 con 0x7fbad80af7a0 2024-04-26T23:08:46.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.605+0000 7fbad7200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbad80d0820 con 0x7fbad815ce60 2024-04-26T23:08:46.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.605+0000 7fbad7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbad81614c0 con 0x7fbad8071fe0 2024-04-26T23:08:46.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.606+0000 7fbacfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbad80af7a0 0x7fbad80ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:46.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.606+0000 7fbacfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbad80af7a0 0x7fbad80ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55786/0 (socket says 172.21.15.40:55786) 2024-04-26T23:08:46.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.606+0000 7fbacfe00640 1 -- 172.21.15.40:0/4085925168 learned_addr learned my addr 172.21.15.40:0/4085925168 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:46.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.606+0000 7fbad5e00640 1 --2- 172.21.15.40:0/4085925168 >> v2:172.21.15.103:3300/0 conn(0x7fbad815ce60 0x7fbad80afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:46.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.606+0000 7fbad6800640 1 --2- 172.21.15.40:0/4085925168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad8071fe0 0x7fbad8073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:46.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.606+0000 7fbacfe00640 1 -- 172.21.15.40:0/4085925168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad8071fe0 msgr2=0x7fbad8073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:46.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.606+0000 7fbacfe00640 1 --2- 172.21.15.40:0/4085925168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad8071fe0 0x7fbad8073760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.606+0000 7fbacfe00640 1 -- 172.21.15.40:0/4085925168 >> v2:172.21.15.103:3300/0 conn(0x7fbad815ce60 msgr2=0x7fbad80afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.606+0000 7fbacfe00640 1 --2- 172.21.15.40:0/4085925168 >> v2:172.21.15.103:3300/0 conn(0x7fbad815ce60 0x7fbad80afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.606+0000 7fbacfe00640 1 -- 172.21.15.40:0/4085925168 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad8161760 con 0x7fbad80af7a0 2024-04-26T23:08:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.608+0000 7fbacfe00640 1 --2- 172.21.15.40:0/4085925168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80af7a0 0x7fbad80ceff0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fbad80741b0 tx=0x7fbac002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee645d0844a1c2fa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.608+0000 7fbad5400640 1 -- 172.21.15.40:0/4085925168 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0037070 con 0x7fbad80af7a0 2024-04-26T23:08:46.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.608+0000 7fbad5400640 1 -- 172.21.15.40:0/4085925168 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac0032440 con 0x7fbad80af7a0 2024-04-26T23:08:46.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.608+0000 7fbad5400640 1 -- 172.21.15.40:0/4085925168 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0032810 con 0x7fbad80af7a0 2024-04-26T23:08:46.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.609+0000 7fbad7200640 1 -- 172.21.15.40:0/4085925168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80af7a0 msgr2=0x7fbad80ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:46.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.609+0000 7fbad7200640 1 --2- 172.21.15.40:0/4085925168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80af7a0 0x7fbad80ceff0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fbad80741b0 tx=0x7fbac002a140 comp rx=0 tx=0).stop 2024-04-26T23:08:46.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.609+0000 7fbad7200640 1 -- 172.21.15.40:0/4085925168 shutdown_connections 2024-04-26T23:08:46.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.609+0000 7fbad7200640 1 --2- 172.21.15.40:0/4085925168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad8071fe0 0x7fbad8073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.609+0000 7fbad7200640 1 --2- 172.21.15.40:0/4085925168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80af7a0 0x7fbad80ceff0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.609+0000 7fbad7200640 1 --2- 172.21.15.40:0/4085925168 >> v2:172.21.15.103:3300/0 conn(0x7fbad815ce60 0x7fbad80afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.609+0000 7fbad7200640 1 -- 172.21.15.40:0/4085925168 >> 172.21.15.40:0/4085925168 conn(0x7fbad80d0a10 msgr2=0x7fbad80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:46.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.609+0000 7fbad7200640 1 -- 172.21.15.40:0/4085925168 shutdown_connections 2024-04-26T23:08:46.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.609+0000 7fbad7200640 1 -- 172.21.15.40:0/4085925168 wait complete. 2024-04-26T23:08:46.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.611+0000 7fbad7200640 1 Processor -- start 2024-04-26T23:08:46.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.611+0000 7fbad7200640 1 -- start start 2024-04-26T23:08:46.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.612+0000 7fbad7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad8071fe0 0x7fbad80b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:46.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.612+0000 7fbad7200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80af7a0 0x7fbad80c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:46.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.612+0000 7fbad7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad815ce60 0x7fbad80c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:46.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.612+0000 7fbad7200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbad8073ca0 con 0x7fbad80af7a0 2024-04-26T23:08:46.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.612+0000 7fbad7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbad80cf530 con 0x7fbad8071fe0 2024-04-26T23:08:46.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.612+0000 7fbad7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbad8073e20 con 0x7fbad815ce60 2024-04-26T23:08:46.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.612+0000 7fbacfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80af7a0 0x7fbad80c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:46.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.612+0000 7fbacfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80af7a0 0x7fbad80c7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55788/0 (socket says 172.21.15.40:55788) 2024-04-26T23:08:46.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.612+0000 7fbacfe00640 1 -- 172.21.15.40:0/2118691670 learned_addr learned my addr 172.21.15.40:0/2118691670 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:46.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.612+0000 7fbad6800640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad815ce60 0x7fbad80c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:46.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.612+0000 7fbad5e00640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad8071fe0 0x7fbad80b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:46.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.613+0000 7fbacfe00640 1 -- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad815ce60 msgr2=0x7fbad80c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:46.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.613+0000 7fbacfe00640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad815ce60 0x7fbad80c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.613+0000 7fbacfe00640 1 -- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad8071fe0 msgr2=0x7fbad80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:46.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.613+0000 7fbacfe00640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad8071fe0 0x7fbad80b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.613+0000 7fbacfe00640 1 -- 172.21.15.40:0/2118691670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad80b51f0 con 0x7fbad80af7a0 2024-04-26T23:08:46.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.613+0000 7fbacfe00640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80af7a0 0x7fbad80c7ae0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fbac00097c0 tx=0x7fbac00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:46.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.613+0000 7fbace000640 1 -- 172.21.15.40:0/2118691670 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0037040 con 0x7fbad80af7a0 2024-04-26T23:08:46.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.614+0000 7fbace000640 1 -- 172.21.15.40:0/2118691670 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac0042070 con 0x7fbad80af7a0 2024-04-26T23:08:46.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.614+0000 7fbad7200640 1 -- 172.21.15.40:0/2118691670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad80b54d0 con 0x7fbad80af7a0 2024-04-26T23:08:46.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.614+0000 7fbace000640 1 -- 172.21.15.40:0/2118691670 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac00309c0 con 0x7fbad80af7a0 2024-04-26T23:08:46.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.614+0000 7fbad7200640 1 -- 172.21.15.40:0/2118691670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad814abb0 con 0x7fbad80af7a0 2024-04-26T23:08:46.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.615+0000 7fbad7200640 1 -- 172.21.15.40:0/2118691670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbad80748d0 con 0x7fbad80af7a0 2024-04-26T23:08:46.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.616+0000 7fbace000640 1 -- 172.21.15.40:0/2118691670 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbac0030b60 con 0x7fbad80af7a0 2024-04-26T23:08:46.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.617+0000 7fbace000640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbab406c8f0 0x7fbab406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:46.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.617+0000 7fbace000640 1 -- 172.21.15.40:0/2118691670 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fbac00ac740 con 0x7fbad80af7a0 2024-04-26T23:08:46.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.618+0000 7fbad5e00640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbab406c8f0 0x7fbab406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:46.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.619+0000 7fbad5e00640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbab406c8f0 0x7fbab406edb0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fbac8006fd0 tx=0x7fbac8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.621+0000 7fbace000640 1 -- 172.21.15.40:0/2118691670 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fbad80748d0 con 0x7fbad80af7a0 2024-04-26T23:08:46.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.840+0000 7fbad7200640 1 -- 172.21.15.40:0/2118691670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbad80d01f0 con 0x7fbad80af7a0 2024-04-26T23:08:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.841+0000 7fbace000640 1 -- 172.21.15.40:0/2118691670 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v90) ==== 80+0+0 (secure 0 0 0) 0x7fbac007a8b0 con 0x7fbad80af7a0 2024-04-26T23:08:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-04-26T23:08:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.844+0000 7fbad7200640 1 -- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbab406c8f0 msgr2=0x7fbab406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.844+0000 7fbad7200640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbab406c8f0 0x7fbab406edb0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fbac8006fd0 tx=0x7fbac8008040 comp rx=0 tx=0).stop 2024-04-26T23:08:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.844+0000 7fbad7200640 1 -- 172.21.15.40:0/2118691670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80af7a0 msgr2=0x7fbad80c7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.844+0000 7fbad7200640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80af7a0 0x7fbad80c7ae0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fbac00097c0 tx=0x7fbac00046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.844+0000 7fbad7200640 1 -- 172.21.15.40:0/2118691670 shutdown_connections 2024-04-26T23:08:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.844+0000 7fbad7200640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbab406c8f0 0x7fbab406edb0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.844+0000 7fbad7200640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad815ce60 0x7fbad80c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.844+0000 7fbad7200640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad80af7a0 0x7fbad80c7ae0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.844+0000 7fbad7200640 1 --2- 172.21.15.40:0/2118691670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad8071fe0 0x7fbad80b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.844+0000 7fbad7200640 1 -- 172.21.15.40:0/2118691670 >> 172.21.15.40:0/2118691670 conn(0x7fbad80d0a10 msgr2=0x7fbad80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.844+0000 7fbad7200640 1 -- 172.21.15.40:0/2118691670 shutdown_connections 2024-04-26T23:08:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:46.844+0000 7fbad7200640 1 -- 172.21.15.40:0/2118691670 wait complete. 2024-04-26T23:08:46.864 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T23:08:46.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:08:46.864 DEBUG:teuthology.orchestra.run.smithi040:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-26T23:08:46.965 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:08:46.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:08:46.965 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T23:08:47.021 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:08:47.021 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:08:47.021 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:08:47.079 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:08:47.079 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:08:47.079 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:08:47.080 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:08:47.140 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:08:47.140 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:08:47.140 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:08:47.140 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:08:47.198 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:08:47.198 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:47.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.418+0000 7ffa52c00640 1 Processor -- start 2024-04-26T23:08:47.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.418+0000 7ffa52c00640 1 -- start start 2024-04-26T23:08:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.420+0000 7ffa52c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ffa4c15ce60 0x7ffa4c0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.420+0000 7ffa52c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c071fe0 0x7ffa4c073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.420+0000 7ffa52c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa4c0d0820 con 0x7ffa4c0af7a0 2024-04-26T23:08:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.420+0000 7ffa52c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ffa4c0cc450 con 0x7ffa4c15ce60 2024-04-26T23:08:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.420+0000 7ffa52c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa4c1614c0 con 0x7ffa4c071fe0 2024-04-26T23:08:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.420+0000 7ffa50e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ffa4c0af7a0 0x7ffa4c0ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60478/0 (socket says 172.21.15.40:60478) 2024-04-26T23:08:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.420+0000 7ffa50e00640 1 -- 172.21.15.40:0/2747975969 learned_addr learned my addr 172.21.15.40:0/2747975969 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.420+0000 7ffa51800640 1 --2- 172.21.15.40:0/2747975969 >> v2:172.21.15.103:3300/0 conn(0x7ffa4c15ce60 0x7ffa4c0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa52200640 1 --2- 172.21.15.40:0/2747975969 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c071fe0 0x7ffa4c073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1083159568 0 0) 0x7ffa4c0d0820 con 0x7ffa4c0af7a0 2024-04-26T23:08:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa34003520 con 0x7ffa4c0af7a0 2024-04-26T23:08:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 617668984 0 0) 0x7ffa34003520 con 0x7ffa4c0af7a0 2024-04-26T23:08:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c071fe0 msgr2=0x7ffa4c073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 --2- 172.21.15.40:0/2747975969 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c071fe0 0x7ffa4c073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 >> v2:172.21.15.103:3300/0 conn(0x7ffa4c15ce60 msgr2=0x7ffa4c0afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 --2- 172.21.15.40:0/2747975969 >> v2:172.21.15.103:3300/0 conn(0x7ffa4c15ce60 0x7ffa4c0afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4c161760 con 0x7ffa4c0af7a0 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7ffa38001670 con 0x7ffa4c0af7a0 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 >> v1:172.21.15.40:6789/0 conn(0x7ffa4c0af7a0 legacy=0x7ffa4c0ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 --2- 172.21.15.40:0/2747975969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa340039b0 0x7ffa34003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 --2- 172.21.15.40:0/2747975969 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c071fe0 0x7ffa34004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 --2- 172.21.15.40:0/2747975969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34004d40 0x7ffa340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c0cc450 con 0x7ffa34004d40 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c1614c0 con 0x7ffa340039b0 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.421+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa3400ba40 con 0x7ffa4c071fe0 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.422+0000 7ffa52200640 1 --2- 172.21.15.40:0/2747975969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34004d40 0x7ffa340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.422+0000 7ffa51800640 1 --2- 172.21.15.40:0/2747975969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa340039b0 0x7ffa34003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.422+0000 7ffa50e00640 1 --2- 172.21.15.40:0/2747975969 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c071fe0 0x7ffa34004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.422+0000 7ffa52200640 1 -- 172.21.15.40:0/2747975969 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c071fe0 msgr2=0x7ffa34004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.422+0000 7ffa52200640 1 --2- 172.21.15.40:0/2747975969 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c071fe0 0x7ffa34004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.422+0000 7ffa52200640 1 -- 172.21.15.40:0/2747975969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa340039b0 msgr2=0x7ffa34003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.422+0000 7ffa52200640 1 --2- 172.21.15.40:0/2747975969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa340039b0 0x7ffa34003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.422+0000 7ffa52200640 1 -- 172.21.15.40:0/2747975969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa34003520 con 0x7ffa34004d40 2024-04-26T23:08:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.422+0000 7ffa50e00640 1 --2- 172.21.15.40:0/2747975969 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c071fe0 0x7ffa34004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.423+0000 7ffa52200640 1 --2- 172.21.15.40:0/2747975969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34004d40 0x7ffa340051c0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7ffa44009fc0 tx=0x7ffa34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e05d17ae80a494 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.423+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa44009c10 con 0x7ffa34004d40 2024-04-26T23:08:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.423+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa44036070 con 0x7ffa34004d40 2024-04-26T23:08:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.423+0000 7ffa4be00640 1 -- 172.21.15.40:0/2747975969 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa44031880 con 0x7ffa34004d40 2024-04-26T23:08:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.424+0000 7ffa52c00640 1 -- 172.21.15.40:0/2747975969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34004d40 msgr2=0x7ffa340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.424+0000 7ffa52c00640 1 --2- 172.21.15.40:0/2747975969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa34004d40 0x7ffa340051c0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7ffa44009fc0 tx=0x7ffa34005830 comp rx=0 tx=0).stop 2024-04-26T23:08:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.424+0000 7ffa52200640 1 -- 172.21.15.40:0/2747975969 reap_dead start 2024-04-26T23:08:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.424+0000 7ffa52c00640 1 -- 172.21.15.40:0/2747975969 shutdown_connections 2024-04-26T23:08:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.424+0000 7ffa52c00640 1 -- 172.21.15.40:0/2747975969 >> 172.21.15.40:0/2747975969 conn(0x7ffa4c0d0a10 msgr2=0x7ffa4c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.424+0000 7ffa52c00640 1 -- 172.21.15.40:0/2747975969 shutdown_connections 2024-04-26T23:08:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.424+0000 7ffa52c00640 1 -- 172.21.15.40:0/2747975969 wait complete. 2024-04-26T23:08:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.426+0000 7ffa52c00640 1 Processor -- start 2024-04-26T23:08:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.426+0000 7ffa52c00640 1 -- start start 2024-04-26T23:08:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.427+0000 7ffa52c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa34004d40 0x7ffa4c14d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.427+0000 7ffa52c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c0c9550 0x7ffa4c0c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.427+0000 7ffa52c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0c6940 0x7ffa4c0c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.427+0000 7ffa52c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c0cc1f0 con 0x7ffa4c0c6940 2024-04-26T23:08:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.427+0000 7ffa52c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c0cc070 con 0x7ffa34004d40 2024-04-26T23:08:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.427+0000 7ffa52c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa4c073760 con 0x7ffa4c0c9550 2024-04-26T23:08:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.428+0000 7ffa52200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0c6940 0x7ffa4c0c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.428+0000 7ffa52200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0c6940 0x7ffa4c0c6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55814/0 (socket says 172.21.15.40:55814) 2024-04-26T23:08:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.428+0000 7ffa52200640 1 -- 172.21.15.40:0/3998569745 learned_addr learned my addr 172.21.15.40:0/3998569745 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.428+0000 7ffa51800640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa34004d40 0x7ffa4c14d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.428+0000 7ffa50e00640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c0c9550 0x7ffa4c0c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.428+0000 7ffa52200640 1 -- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c0c9550 msgr2=0x7ffa4c0c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.428+0000 7ffa52200640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c0c9550 0x7ffa4c0c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.428+0000 7ffa52200640 1 -- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa34004d40 msgr2=0x7ffa4c14d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.428+0000 7ffa52200640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa34004d40 0x7ffa4c14d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.428+0000 7ffa52200640 1 -- 172.21.15.40:0/3998569745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4c06fd00 con 0x7ffa4c0c6940 2024-04-26T23:08:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.428+0000 7ffa50e00640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c0c9550 0x7ffa4c0c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.428+0000 7ffa52200640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0c6940 0x7ffa4c0c6dc0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7ffa4400e200 tx=0x7ffa44009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.429+0000 7ffa49600640 1 -- 172.21.15.40:0/3998569745 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa44009c10 con 0x7ffa4c0c6940 2024-04-26T23:08:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.429+0000 7ffa49600640 1 -- 172.21.15.40:0/3998569745 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa44040030 con 0x7ffa4c0c6940 2024-04-26T23:08:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.429+0000 7ffa49600640 1 -- 172.21.15.40:0/3998569745 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa44003bd0 con 0x7ffa4c0c6940 2024-04-26T23:08:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.429+0000 7ffa52c00640 1 -- 172.21.15.40:0/3998569745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4c0ca130 con 0x7ffa4c0c6940 2024-04-26T23:08:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.429+0000 7ffa52c00640 1 -- 172.21.15.40:0/3998569745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa4c0ca6f0 con 0x7ffa4c0c6940 2024-04-26T23:08:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.430+0000 7ffa52c00640 1 -- 172.21.15.40:0/3998569745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa4c069370 con 0x7ffa4c0c6940 2024-04-26T23:08:47.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.431+0000 7ffa49600640 1 -- 172.21.15.40:0/3998569745 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffa44005d10 con 0x7ffa4c0c6940 2024-04-26T23:08:47.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.432+0000 7ffa49600640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ffa2c06c920 0x7ffa2c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.432+0000 7ffa49600640 1 -- 172.21.15.40:0/3998569745 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7ffa440ace50 con 0x7ffa4c0c6940 2024-04-26T23:08:47.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.433+0000 7ffa51800640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ffa2c06c920 0x7ffa2c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.434+0000 7ffa51800640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ffa2c06c920 0x7ffa2c06ede0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ffa40009a10 tx=0x7ffa40008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:47.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.437+0000 7ffa49600640 1 -- 172.21.15.40:0/3998569745 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ffa4404a020 con 0x7ffa4c0c6940 2024-04-26T23:08:47.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.683+0000 7ffa52c00640 1 -- 172.21.15.40:0/3998569745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa4c0d01f0 con 0x7ffa4c0c6940 2024-04-26T23:08:47.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.684+0000 7ffa49600640 1 -- 172.21.15.40:0/3998569745 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5232 (secure 0 0 0) 0x7ffa4407b0e0 con 0x7ffa4c0c6940 2024-04-26T23:08:47.685 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:47.685 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:26.911596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:47.686 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-04-26T23:08:47.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.687+0000 7ffa52c00640 1 -- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ffa2c06c920 msgr2=0x7ffa2c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:47.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.687+0000 7ffa52c00640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ffa2c06c920 0x7ffa2c06ede0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ffa40009a10 tx=0x7ffa40008040 comp rx=0 tx=0).stop 2024-04-26T23:08:47.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.687+0000 7ffa52c00640 1 -- 172.21.15.40:0/3998569745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0c6940 msgr2=0x7ffa4c0c6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:47.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.687+0000 7ffa52c00640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0c6940 0x7ffa4c0c6dc0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7ffa4400e200 tx=0x7ffa44009750 comp rx=0 tx=0).stop 2024-04-26T23:08:47.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.687+0000 7ffa52c00640 1 -- 172.21.15.40:0/3998569745 shutdown_connections 2024-04-26T23:08:47.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.687+0000 7ffa52c00640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ffa2c06c920 0x7ffa2c06ede0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.687+0000 7ffa52c00640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0c6940 0x7ffa4c0c6dc0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.687+0000 7ffa52c00640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa4c0c9550 0x7ffa4c0c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.687+0000 7ffa52c00640 1 --2- 172.21.15.40:0/3998569745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa34004d40 0x7ffa4c14d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.687+0000 7ffa52c00640 1 -- 172.21.15.40:0/3998569745 >> 172.21.15.40:0/3998569745 conn(0x7ffa4c0d0a10 msgr2=0x7ffa4c0ce0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:47.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.688+0000 7ffa52c00640 1 -- 172.21.15.40:0/3998569745 shutdown_connections 2024-04-26T23:08:47.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.688+0000 7ffa52c00640 1 -- 172.21.15.40:0/3998569745 wait complete. 2024-04-26T23:08:47.710 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:08:47.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.926+0000 7fd463200640 1 Processor -- start 2024-04-26T23:08:47.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.927+0000 7fd463200640 1 -- start start 2024-04-26T23:08:47.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.928+0000 7fd463200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c0b6eb0 0x7fd45c164b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.928+0000 7fd463200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd45c071220 0x7fd45c167080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.928+0000 7fd463200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd45c068e60 con 0x7fd45c15d010 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.928+0000 7fd463200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fd45c15d3f0 con 0x7fd45c071220 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.928+0000 7fd463200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd45c0b11d0 con 0x7fd45c0b6eb0 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.929+0000 7fd462800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd45c071220 0x7fd45c167080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.929+0000 7fd461e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd45c15d010 0x7fd45c0b67a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60486/0 (socket says 172.21.15.40:60486) 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.929+0000 7fd461e00640 1 -- 172.21.15.40:0/2042028163 learned_addr learned my addr 172.21.15.40:0/2042028163 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.929+0000 7fd461400640 1 --2- 172.21.15.40:0/2042028163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c0b6eb0 0x7fd45c164b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.929+0000 7fd462800640 1 -- 172.21.15.40:0/2042028163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c0b6eb0 msgr2=0x7fd45c164b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.929+0000 7fd462800640 1 --2- 172.21.15.40:0/2042028163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c0b6eb0 0x7fd45c164b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.929+0000 7fd462800640 1 -- 172.21.15.40:0/2042028163 >> v1:172.21.15.40:6789/0 conn(0x7fd45c15d010 legacy=0x7fd45c0b67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.929+0000 7fd460a00640 1 -- 172.21.15.40:0/2042028163 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 187053209 0 0) 0x7fd45c068e60 con 0x7fd45c15d010 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.929+0000 7fd462800640 1 -- 172.21.15.40:0/2042028163 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45c0b14b0 con 0x7fd45c071220 2024-04-26T23:08:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.929+0000 7fd461400640 1 --2- 172.21.15.40:0/2042028163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c0b6eb0 0x7fd45c164b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:47.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.931+0000 7fd462800640 1 --2- 172.21.15.40:0/2042028163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd45c071220 0x7fd45c167080 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fd458009d90 tx=0x7fd45802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99a75b590e5a8b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:47.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.931+0000 7fd460a00640 1 -- 172.21.15.40:0/2042028163 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd458031600 con 0x7fd45c071220 2024-04-26T23:08:47.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.931+0000 7fd460a00640 1 -- 172.21.15.40:0/2042028163 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4580317a0 con 0x7fd45c071220 2024-04-26T23:08:47.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.931+0000 7fd460a00640 1 -- 172.21.15.40:0/2042028163 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd458031b70 con 0x7fd45c071220 2024-04-26T23:08:47.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.932+0000 7fd463200640 1 -- 172.21.15.40:0/2042028163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd45c071220 msgr2=0x7fd45c167080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:47.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.932+0000 7fd463200640 1 --2- 172.21.15.40:0/2042028163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd45c071220 0x7fd45c167080 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fd458009d90 tx=0x7fd45802a5d0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.932+0000 7fd463200640 1 -- 172.21.15.40:0/2042028163 shutdown_connections 2024-04-26T23:08:47.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.932+0000 7fd463200640 1 --2- 172.21.15.40:0/2042028163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd45c071220 0x7fd45c167080 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.932+0000 7fd463200640 1 --2- 172.21.15.40:0/2042028163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c0b6eb0 0x7fd45c164b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.932+0000 7fd463200640 1 -- 172.21.15.40:0/2042028163 >> 172.21.15.40:0/2042028163 conn(0x7fd45c09ba50 msgr2=0x7fd45c09be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.932+0000 7fd463200640 1 -- 172.21.15.40:0/2042028163 shutdown_connections 2024-04-26T23:08:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.932+0000 7fd463200640 1 -- 172.21.15.40:0/2042028163 wait complete. 2024-04-26T23:08:47.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.934+0000 7fd463200640 1 Processor -- start 2024-04-26T23:08:47.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.934+0000 7fd463200640 1 -- start start 2024-04-26T23:08:47.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.935+0000 7fd463200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd45c071220 0x7fd45c158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.935+0000 7fd463200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c0b6eb0 0x7fd45c0b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.935+0000 7fd463200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd45c15d010 0x7fd45c1f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.935+0000 7fd463200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd45c0b1d50 con 0x7fd45c15d010 2024-04-26T23:08:47.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.935+0000 7fd463200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd45c0b1a70 con 0x7fd45c071220 2024-04-26T23:08:47.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.935+0000 7fd463200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd45c0b0e60 con 0x7fd45c0b6eb0 2024-04-26T23:08:47.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.935+0000 7fd462800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd45c15d010 0x7fd45c1f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.935+0000 7fd462800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd45c15d010 0x7fd45c1f3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55830/0 (socket says 172.21.15.40:55830) 2024-04-26T23:08:47.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.935+0000 7fd462800640 1 -- 172.21.15.40:0/2422035625 learned_addr learned my addr 172.21.15.40:0/2422035625 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:47.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.936+0000 7fd461e00640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd45c071220 0x7fd45c158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.936+0000 7fd461400640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c0b6eb0 0x7fd45c0b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.936+0000 7fd462800640 1 -- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c0b6eb0 msgr2=0x7fd45c0b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:47.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.936+0000 7fd462800640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c0b6eb0 0x7fd45c0b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.936+0000 7fd462800640 1 -- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd45c071220 msgr2=0x7fd45c158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:47.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.936+0000 7fd462800640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd45c071220 0x7fd45c158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:47.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.936+0000 7fd462800640 1 -- 172.21.15.40:0/2422035625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45c157510 con 0x7fd45c15d010 2024-04-26T23:08:47.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.936+0000 7fd461e00640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd45c071220 0x7fd45c158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:47.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.936+0000 7fd462800640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd45c15d010 0x7fd45c1f3d60 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fd458009d90 tx=0x7fd458004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:47.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.936+0000 7fd452000640 1 -- 172.21.15.40:0/2422035625 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd458040070 con 0x7fd45c15d010 2024-04-26T23:08:47.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.936+0000 7fd452000640 1 -- 172.21.15.40:0/2422035625 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd458030be0 con 0x7fd45c15d010 2024-04-26T23:08:47.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.936+0000 7fd452000640 1 -- 172.21.15.40:0/2422035625 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd458009450 con 0x7fd45c15d010 2024-04-26T23:08:47.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.937+0000 7fd463200640 1 -- 172.21.15.40:0/2422035625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd45c1577f0 con 0x7fd45c15d010 2024-04-26T23:08:47.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.937+0000 7fd463200640 1 -- 172.21.15.40:0/2422035625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd45c157db0 con 0x7fd45c15d010 2024-04-26T23:08:47.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.938+0000 7fd463200640 1 -- 172.21.15.40:0/2422035625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd45c1f7df0 con 0x7fd45c15d010 2024-04-26T23:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.939+0000 7fd452000640 1 -- 172.21.15.40:0/2422035625 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd458039040 con 0x7fd45c15d010 2024-04-26T23:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.940+0000 7fd452000640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd44006c8f0 0x7fd44006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.940+0000 7fd452000640 1 -- 172.21.15.40:0/2422035625 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6101+0+0 (secure 0 0 0) 0x7fd4580acf30 con 0x7fd45c15d010 2024-04-26T23:08:47.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.940+0000 7fd461e00640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd44006c8f0 0x7fd44006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:47.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.942+0000 7fd461e00640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd44006c8f0 0x7fd44006edb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd454000f60 tx=0x7fd454009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:47.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:47.944+0000 7fd452000640 1 -- 172.21.15.40:0/2422035625 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd45c1f7df0 con 0x7fd45c15d010 2024-04-26T23:08:48.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.148+0000 7fd463200640 1 -- 172.21.15.40:0/2422035625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd45c0b6370 con 0x7fd45c15d010 2024-04-26T23:08:48.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.152+0000 7fd452000640 1 -- 172.21.15.40:0/2422035625 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+16849 (secure 0 0 0) 0x7fd45807b130 con 0x7fd45c15d010 2024-04-26T23:08:48.152 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:48.153 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:07:49.778062+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:07:46.137238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-26T23:07:47.258518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:08:48.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.155+0000 7fd463200640 1 -- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd44006c8f0 msgr2=0x7fd44006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:48.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.155+0000 7fd463200640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd44006c8f0 0x7fd44006edb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd454000f60 tx=0x7fd454009040 comp rx=0 tx=0).stop 2024-04-26T23:08:48.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.155+0000 7fd463200640 1 -- 172.21.15.40:0/2422035625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd45c15d010 msgr2=0x7fd45c1f3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:48.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.155+0000 7fd463200640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd45c15d010 0x7fd45c1f3d60 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fd458009d90 tx=0x7fd458004a40 comp rx=0 tx=0).stop 2024-04-26T23:08:48.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.155+0000 7fd463200640 1 -- 172.21.15.40:0/2422035625 shutdown_connections 2024-04-26T23:08:48.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.155+0000 7fd463200640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd44006c8f0 0x7fd44006edb0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.155+0000 7fd463200640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd45c15d010 0x7fd45c1f3d60 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.155+0000 7fd463200640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c0b6eb0 0x7fd45c0b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.155+0000 7fd463200640 1 --2- 172.21.15.40:0/2422035625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd45c071220 0x7fd45c158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.155+0000 7fd463200640 1 -- 172.21.15.40:0/2422035625 >> 172.21.15.40:0/2422035625 conn(0x7fd45c09ba50 msgr2=0x7fd45c06ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.155+0000 7fd463200640 1 -- 172.21.15.40:0/2422035625 shutdown_connections 2024-04-26T23:08:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.155+0000 7fd463200640 1 -- 172.21.15.40:0/2422035625 wait complete. 2024-04-26T23:08:48.176 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:07:46.137238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:07:47.258518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:08:48.177 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:08:48.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.390+0000 7fa863200640 1 Processor -- start 2024-04-26T23:08:48.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.390+0000 7fa863200640 1 -- start start 2024-04-26T23:08:48.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.392+0000 7fa863200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa864160a50 0x7fa864073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.392+0000 7fa863200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8640cfaf0 0x7fa864160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.392+0000 7fa863200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa86406dd40 0x7fa864072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.392+0000 7fa863200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa864071ce0 con 0x7fa86406dd40 2024-04-26T23:08:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.392+0000 7fa863200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa8640722a0 con 0x7fa864160a50 2024-04-26T23:08:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.392+0000 7fa863200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa864073f90 con 0x7fa8640cfaf0 2024-04-26T23:08:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.393+0000 7fa862800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa86406dd40 0x7fa864072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.393+0000 7fa862800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa86406dd40 0x7fa864072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55834/0 (socket says 172.21.15.40:55834) 2024-04-26T23:08:48.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.393+0000 7fa862800640 1 -- 172.21.15.40:0/805277439 learned_addr learned my addr 172.21.15.40:0/805277439 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.393+0000 7fa859400640 1 --2- 172.21.15.40:0/805277439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8640cfaf0 0x7fa864160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.393+0000 7fa861e00640 1 --2- 172.21.15.40:0/805277439 >> v2:172.21.15.103:3300/0 conn(0x7fa864160a50 0x7fa864073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.393+0000 7fa862800640 1 -- 172.21.15.40:0/805277439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8640cfaf0 msgr2=0x7fa864160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.393+0000 7fa862800640 1 --2- 172.21.15.40:0/805277439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8640cfaf0 0x7fa864160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.393+0000 7fa862800640 1 -- 172.21.15.40:0/805277439 >> v2:172.21.15.103:3300/0 conn(0x7fa864160a50 msgr2=0x7fa864073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.393+0000 7fa862800640 1 --2- 172.21.15.40:0/805277439 >> v2:172.21.15.103:3300/0 conn(0x7fa864160a50 0x7fa864073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.393+0000 7fa862800640 1 -- 172.21.15.40:0/805277439 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa864161880 con 0x7fa86406dd40 2024-04-26T23:08:48.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.394+0000 7fa862800640 1 --2- 172.21.15.40:0/805277439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa86406dd40 0x7fa864072730 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fa864161b70 tx=0x7fa85c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136c911aaba2d6bc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:48.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.395+0000 7fa861400640 1 -- 172.21.15.40:0/805277439 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c037070 con 0x7fa86406dd40 2024-04-26T23:08:48.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.395+0000 7fa861400640 1 -- 172.21.15.40:0/805277439 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa85c032440 con 0x7fa86406dd40 2024-04-26T23:08:48.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.395+0000 7fa861400640 1 -- 172.21.15.40:0/805277439 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c032810 con 0x7fa86406dd40 2024-04-26T23:08:48.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.395+0000 7fa863200640 1 -- 172.21.15.40:0/805277439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa86406dd40 msgr2=0x7fa864072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:48.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.395+0000 7fa863200640 1 --2- 172.21.15.40:0/805277439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa86406dd40 0x7fa864072730 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fa864161b70 tx=0x7fa85c02a1e0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.395+0000 7fa863200640 1 -- 172.21.15.40:0/805277439 shutdown_connections 2024-04-26T23:08:48.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.395+0000 7fa863200640 1 --2- 172.21.15.40:0/805277439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa86406dd40 0x7fa864072730 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.395+0000 7fa863200640 1 --2- 172.21.15.40:0/805277439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8640cfaf0 0x7fa864160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.395+0000 7fa863200640 1 --2- 172.21.15.40:0/805277439 >> v2:172.21.15.103:3300/0 conn(0x7fa864160a50 0x7fa864073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.395+0000 7fa863200640 1 -- 172.21.15.40:0/805277439 >> 172.21.15.40:0/805277439 conn(0x7fa8640b2720 msgr2=0x7fa8640b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:48.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.396+0000 7fa863200640 1 -- 172.21.15.40:0/805277439 shutdown_connections 2024-04-26T23:08:48.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.396+0000 7fa863200640 1 -- 172.21.15.40:0/805277439 wait complete. 2024-04-26T23:08:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.397+0000 7fa863200640 1 Processor -- start 2024-04-26T23:08:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.398+0000 7fa863200640 1 -- start start 2024-04-26T23:08:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.398+0000 7fa863200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa86406dd40 0x7fa8641f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.398+0000 7fa863200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa864160a50 0x7fa864140490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa863200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa86415c760 0x7fa8640aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa863200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa8640d0260 con 0x7fa864160a50 2024-04-26T23:08:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa863200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa8640cff80 con 0x7fa86406dd40 2024-04-26T23:08:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa863200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa864071f30 con 0x7fa86415c760 2024-04-26T23:08:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa859400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa864160a50 0x7fa864140490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa859400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa864160a50 0x7fa864140490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55844/0 (socket says 172.21.15.40:55844) 2024-04-26T23:08:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa859400640 1 -- 172.21.15.40:0/2568246551 learned_addr learned my addr 172.21.15.40:0/2568246551 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa861e00640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa86406dd40 0x7fa8641f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa859400640 1 -- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa86415c760 msgr2=0x7fa8640aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa862800640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa86415c760 0x7fa8640aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa859400640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa86415c760 0x7fa8640aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa859400640 1 -- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa86406dd40 msgr2=0x7fa8641f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa859400640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa86406dd40 0x7fa8641f2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.399+0000 7fa859400640 1 -- 172.21.15.40:0/2568246551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa864144770 con 0x7fa864160a50 2024-04-26T23:08:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.400+0000 7fa859400640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa864160a50 0x7fa864140490 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00d800 tx=0x7fa84c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.400+0000 7fa85aa00640 1 -- 172.21.15.40:0/2568246551 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c018020 con 0x7fa864160a50 2024-04-26T23:08:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.400+0000 7fa85aa00640 1 -- 172.21.15.40:0/2568246551 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa84c0096a0 con 0x7fa864160a50 2024-04-26T23:08:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.400+0000 7fa863200640 1 -- 172.21.15.40:0/2568246551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8641f31b0 con 0x7fa864160a50 2024-04-26T23:08:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.400+0000 7fa85aa00640 1 -- 172.21.15.40:0/2568246551 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c009a70 con 0x7fa864160a50 2024-04-26T23:08:48.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.400+0000 7fa863200640 1 -- 172.21.15.40:0/2568246551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8641f3770 con 0x7fa864160a50 2024-04-26T23:08:48.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.401+0000 7fa863200640 1 -- 172.21.15.40:0/2568246551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa864069370 con 0x7fa864160a50 2024-04-26T23:08:48.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.403+0000 7fa85aa00640 1 -- 172.21.15.40:0/2568246551 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa84c01b020 con 0x7fa864160a50 2024-04-26T23:08:48.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.404+0000 7fa85aa00640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa84006c8f0 0x7fa84006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:48.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.404+0000 7fa85aa00640 1 -- 172.21.15.40:0/2568246551 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7fa84c08c7a0 con 0x7fa864160a50 2024-04-26T23:08:48.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.405+0000 7fa861e00640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa84006c8f0 0x7fa84006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:48.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.406+0000 7fa861e00640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa84006c8f0 0x7fa84006edb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa864143820 tx=0x7fa8540039b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:48.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.408+0000 7fa85aa00640 1 -- 172.21.15.40:0/2568246551 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa84c022030 con 0x7fa864160a50 2024-04-26T23:08:48.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.618+0000 7fa863200640 1 -- 172.21.15.40:0/2568246551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa86406fa60 con 0x7fa864160a50 2024-04-26T23:08:48.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.618+0000 7fa85aa00640 1 -- 172.21.15.40:0/2568246551 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fa84c011340 con 0x7fa864160a50 2024-04-26T23:08:48.619 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:08:48.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.621+0000 7fa863200640 1 -- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa84006c8f0 msgr2=0x7fa84006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:48.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.621+0000 7fa863200640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa84006c8f0 0x7fa84006edb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa864143820 tx=0x7fa8540039b0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.621+0000 7fa863200640 1 -- 172.21.15.40:0/2568246551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa864160a50 msgr2=0x7fa864140490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:48.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.621+0000 7fa863200640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa864160a50 0x7fa864140490 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00d800 tx=0x7fa84c00dcc0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.621+0000 7fa863200640 1 -- 172.21.15.40:0/2568246551 shutdown_connections 2024-04-26T23:08:48.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.621+0000 7fa863200640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa84006c8f0 0x7fa84006edb0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.621+0000 7fa863200640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa86415c760 0x7fa8640aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.621+0000 7fa863200640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa864160a50 0x7fa864140490 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.621+0000 7fa863200640 1 --2- 172.21.15.40:0/2568246551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa86406dd40 0x7fa8641f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.621+0000 7fa863200640 1 -- 172.21.15.40:0/2568246551 >> 172.21.15.40:0/2568246551 conn(0x7fa8640b2720 msgr2=0x7fa86415e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.621+0000 7fa863200640 1 -- 172.21.15.40:0/2568246551 shutdown_connections 2024-04-26T23:08:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.621+0000 7fa863200640 1 -- 172.21.15.40:0/2568246551 wait complete. 2024-04-26T23:08:48.641 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:08:48.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.856+0000 7f1323200640 1 Processor -- start 2024-04-26T23:08:48.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.856+0000 7f1323200640 1 -- start start 2024-04-26T23:08:48.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.858+0000 7f1323200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f131c071e60 0x7f131c06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:48.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.858+0000 7f1323200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f131c15d7c0 con 0x7f131c06fe20 2024-04-26T23:08:48.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.858+0000 7f1323200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f131c15daa0 con 0x7f131c0cc480 2024-04-26T23:08:48.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.858+0000 7f1323200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f131c15dd80 con 0x7f131c071e60 2024-04-26T23:08:48.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.858+0000 7f131be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f131c071e60 0x7f131c06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:48.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.858+0000 7f131b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f131c06fe20 0x7f131c072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60490/0 (socket says 172.21.15.40:60490) 2024-04-26T23:08:48.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.858+0000 7f131b400640 1 -- 172.21.15.40:0/513701954 learned_addr learned my addr 172.21.15.40:0/513701954 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:48.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.858+0000 7f131be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f131c071e60 0x7f131c06f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55860/0 (socket says 172.21.15.40:55860) 2024-04-26T23:08:48.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.859+0000 7f131be00640 1 -- 172.21.15.40:0/513701954 >> v1:172.21.15.103:6789/0 conn(0x7f131c0cc480 legacy=0x7f131c166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:48.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.859+0000 7f131be00640 1 -- 172.21.15.40:0/513701954 >> v1:172.21.15.40:6789/0 conn(0x7f131c06fe20 legacy=0x7f131c072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:48.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.859+0000 7f131be00640 1 -- 172.21.15.40:0/513701954 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131c15e060 con 0x7f131c071e60 2024-04-26T23:08:48.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.860+0000 7f131be00640 1 --2- 172.21.15.40:0/513701954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f131c071e60 0x7f131c06f8e0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f130c009d90 tx=0x7f130c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6da4bf9cffe5124 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:48.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.861+0000 7f131aa00640 1 -- 172.21.15.40:0/513701954 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c007740 con 0x7f131c071e60 2024-04-26T23:08:48.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.861+0000 7f131aa00640 1 -- 172.21.15.40:0/513701954 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f130c0078e0 con 0x7f131c071e60 2024-04-26T23:08:48.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.861+0000 7f131aa00640 1 -- 172.21.15.40:0/513701954 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c007cb0 con 0x7f131c071e60 2024-04-26T23:08:48.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.861+0000 7f1323200640 1 -- 172.21.15.40:0/513701954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f131c071e60 msgr2=0x7f131c06f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:48.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.861+0000 7f1323200640 1 --2- 172.21.15.40:0/513701954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f131c071e60 0x7f131c06f8e0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f130c009d90 tx=0x7f130c02a6a0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.862+0000 7f1323200640 1 -- 172.21.15.40:0/513701954 shutdown_connections 2024-04-26T23:08:48.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.862+0000 7f1323200640 1 --2- 172.21.15.40:0/513701954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f131c071e60 0x7f131c06f8e0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.862+0000 7f1323200640 1 -- 172.21.15.40:0/513701954 >> 172.21.15.40:0/513701954 conn(0x7f131c0d0be0 msgr2=0x7f131c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:48.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.862+0000 7f1323200640 1 -- 172.21.15.40:0/513701954 shutdown_connections 2024-04-26T23:08:48.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.862+0000 7f1323200640 1 -- 172.21.15.40:0/513701954 wait complete. 2024-04-26T23:08:48.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.864+0000 7f1323200640 1 Processor -- start 2024-04-26T23:08:48.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.864+0000 7f1323200640 1 -- start start 2024-04-26T23:08:48.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.865+0000 7f1323200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f131c06fe20 0x7f131c155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f1323200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f131c071e60 0x7f131c0b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f1323200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f131c0cc480 0x7f131c0b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f1323200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f131c15d570 con 0x7f131c071e60 2024-04-26T23:08:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f1323200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f131c15d3f0 con 0x7f131c06fe20 2024-04-26T23:08:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f1323200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f131c072950 con 0x7f131c0cc480 2024-04-26T23:08:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f131b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f131c071e60 0x7f131c0b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f131b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f131c071e60 0x7f131c0b4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55868/0 (socket says 172.21.15.40:55868) 2024-04-26T23:08:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f131b400640 1 -- 172.21.15.40:0/1902485041 learned_addr learned my addr 172.21.15.40:0/1902485041 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f131be00640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f131c06fe20 0x7f131c155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f1321200640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f131c0cc480 0x7f131c0b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f131b400640 1 -- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f131c0cc480 msgr2=0x7f131c0b53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f131b400640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f131c0cc480 0x7f131c0b53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f131b400640 1 -- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f131c06fe20 msgr2=0x7f131c155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f131b400640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f131c06fe20 0x7f131c155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.866+0000 7f131b400640 1 -- 172.21.15.40:0/1902485041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131c159820 con 0x7f131c071e60 2024-04-26T23:08:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.867+0000 7f131b400640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f131c071e60 0x7f131c0b4e70 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f130400c400 tx=0x7f130400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:48.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.867+0000 7f1313e00640 1 -- 172.21.15.40:0/1902485041 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304017070 con 0x7f131c071e60 2024-04-26T23:08:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.867+0000 7f1313e00640 1 -- 172.21.15.40:0/1902485041 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1304005ce0 con 0x7f131c071e60 2024-04-26T23:08:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.867+0000 7f1323200640 1 -- 172.21.15.40:0/1902485041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f131c159b00 con 0x7f131c071e60 2024-04-26T23:08:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.867+0000 7f1313e00640 1 -- 172.21.15.40:0/1902485041 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13040072f0 con 0x7f131c071e60 2024-04-26T23:08:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.867+0000 7f1323200640 1 -- 172.21.15.40:0/1902485041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f131c15a0c0 con 0x7f131c071e60 2024-04-26T23:08:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.869+0000 7f1323200640 1 -- 172.21.15.40:0/1902485041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f131c0694f0 con 0x7f131c071e60 2024-04-26T23:08:48.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.869+0000 7f1313e00640 1 -- 172.21.15.40:0/1902485041 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f130401a070 con 0x7f131c071e60 2024-04-26T23:08:48.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.870+0000 7f1313e00640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f12f806c8f0 0x7f12f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.871+0000 7f1313e00640 1 -- 172.21.15.40:0/1902485041 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6217+0+0 (secure 0 0 0) 0x7f130408b1d0 con 0x7f131c071e60 2024-04-26T23:08:48.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.871+0000 7f131be00640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f12f806c8f0 0x7f12f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.872+0000 7f131be00640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f12f806c8f0 0x7f12f806edb0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f130c034000 tx=0x7f130c02f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:48.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:48.875+0000 7f1313e00640 1 -- 172.21.15.40:0/1902485041 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1304059360 con 0x7f131c071e60 2024-04-26T23:08:49.053 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:08:49.060 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:08:49.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.088+0000 7f1323200640 1 -- 172.21.15.40:0/1902485041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f131c160220 con 0x7f131c071e60 2024-04-26T23:08:49.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.089+0000 7f1313e00640 1 -- 172.21.15.40:0/1902485041 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f1304058c60 con 0x7f131c071e60 2024-04-26T23:08:49.089 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:08:49.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.092+0000 7f1323200640 1 -- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f12f806c8f0 msgr2=0x7f12f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:49.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.092+0000 7f1323200640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f12f806c8f0 0x7f12f806edb0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f130c034000 tx=0x7f130c02f000 comp rx=0 tx=0).stop 2024-04-26T23:08:49.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.092+0000 7f1323200640 1 -- 172.21.15.40:0/1902485041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f131c071e60 msgr2=0x7f131c0b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:49.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.092+0000 7f1323200640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f131c071e60 0x7f131c0b4e70 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f130400c400 tx=0x7f130400c8c0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.092+0000 7f1323200640 1 -- 172.21.15.40:0/1902485041 shutdown_connections 2024-04-26T23:08:49.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.092+0000 7f1323200640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f12f806c8f0 0x7f12f806edb0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.092+0000 7f1323200640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f131c0cc480 0x7f131c0b53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.092+0000 7f1323200640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f131c071e60 0x7f131c0b4e70 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.092+0000 7f1323200640 1 --2- 172.21.15.40:0/1902485041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f131c06fe20 0x7f131c155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.092+0000 7f1323200640 1 -- 172.21.15.40:0/1902485041 >> 172.21.15.40:0/1902485041 conn(0x7f131c0d0be0 msgr2=0x7f131c072d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:49.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.092+0000 7f1323200640 1 -- 172.21.15.40:0/1902485041 shutdown_connections 2024-04-26T23:08:49.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.092+0000 7f1323200640 1 -- 172.21.15.40:0/1902485041 wait complete. 2024-04-26T23:08:49.111 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:08:49.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.373+0000 7f1c3e200640 1 Processor -- start 2024-04-26T23:08:49.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.373+0000 7f1c3e200640 1 -- start start 2024-04-26T23:08:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.375+0000 7f1c3e200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1c38071e60 0x7f1c3806f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.375+0000 7f1c3e200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1c3806fe20 0x7f1c38072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.375+0000 7f1c3e200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c3815d7c0 con 0x7f1c38072780 2024-04-26T23:08:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.375+0000 7f1c3e200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1c3815daa0 con 0x7f1c38071e60 2024-04-26T23:08:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.375+0000 7f1c3e200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1c3815dd80 con 0x7f1c3806fe20 2024-04-26T23:08:49.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.375+0000 7f1c37400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1c38071e60 0x7f1c3806f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:49.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.376+0000 7f1c37e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1c38072780 0x7f1c380cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60506/0 (socket says 172.21.15.40:60506) 2024-04-26T23:08:49.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.376+0000 7f1c37400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1c38071e60 0x7f1c3806f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55884/0 (socket says 172.21.15.40:55884) 2024-04-26T23:08:49.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.376+0000 7f1c37e00640 1 -- 172.21.15.40:0/1965022479 learned_addr learned my addr 172.21.15.40:0/1965022479 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:49.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.376+0000 7f1c36a00640 1 --2- 172.21.15.40:0/1965022479 >> v2:172.21.15.103:3300/0 conn(0x7f1c3806fe20 0x7f1c38072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:49.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.376+0000 7f1c37400640 1 -- 172.21.15.40:0/1965022479 >> v2:172.21.15.103:3300/0 conn(0x7f1c3806fe20 msgr2=0x7f1c38072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:49.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.376+0000 7f1c37400640 1 --2- 172.21.15.40:0/1965022479 >> v2:172.21.15.103:3300/0 conn(0x7f1c3806fe20 0x7f1c38072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.376+0000 7f1c37400640 1 -- 172.21.15.40:0/1965022479 >> v1:172.21.15.40:6789/0 conn(0x7f1c38072780 legacy=0x7f1c380cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:49.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.376+0000 7f1c37400640 1 -- 172.21.15.40:0/1965022479 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c3815e060 con 0x7f1c38071e60 2024-04-26T23:08:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.377+0000 7f1c37400640 1 --2- 172.21.15.40:0/1965022479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c38071e60 0x7f1c3806f8e0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f1c380b3820 tx=0x7f1c2802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f829899f0019c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.377+0000 7f1c36000640 1 -- 172.21.15.40:0/1965022479 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28037070 con 0x7f1c38071e60 2024-04-26T23:08:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.378+0000 7f1c36000640 1 -- 172.21.15.40:0/1965022479 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c28032440 con 0x7f1c38071e60 2024-04-26T23:08:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.378+0000 7f1c36000640 1 -- 172.21.15.40:0/1965022479 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28032810 con 0x7f1c38071e60 2024-04-26T23:08:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.378+0000 7f1c3e200640 1 -- 172.21.15.40:0/1965022479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c38071e60 msgr2=0x7f1c3806f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.378+0000 7f1c3e200640 1 --2- 172.21.15.40:0/1965022479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c38071e60 0x7f1c3806f8e0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f1c380b3820 tx=0x7f1c2802a1a0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.378+0000 7f1c3e200640 1 -- 172.21.15.40:0/1965022479 shutdown_connections 2024-04-26T23:08:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.378+0000 7f1c3e200640 1 --2- 172.21.15.40:0/1965022479 >> v2:172.21.15.103:3300/0 conn(0x7f1c3806fe20 0x7f1c38072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.378+0000 7f1c3e200640 1 --2- 172.21.15.40:0/1965022479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c38071e60 0x7f1c3806f8e0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.378+0000 7f1c3e200640 1 -- 172.21.15.40:0/1965022479 >> 172.21.15.40:0/1965022479 conn(0x7f1c380d0be0 msgr2=0x7f1c380d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.378+0000 7f1c3e200640 1 -- 172.21.15.40:0/1965022479 shutdown_connections 2024-04-26T23:08:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.378+0000 7f1c3e200640 1 -- 172.21.15.40:0/1965022479 wait complete. 2024-04-26T23:08:49.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.380+0000 7f1c3e200640 1 Processor -- start 2024-04-26T23:08:49.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.380+0000 7f1c3e200640 1 -- start start 2024-04-26T23:08:49.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.381+0000 7f1c3e200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c3806fe20 0x7f1c381f8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:49.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.381+0000 7f1c3e200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c38071e60 0x7f1c380b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:49.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.381+0000 7f1c3e200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c38072780 0x7f1c38157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:49.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.381+0000 7f1c3e200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3815d3f0 con 0x7f1c3806fe20 2024-04-26T23:08:49.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.381+0000 7f1c3e200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c38072bd0 con 0x7f1c38072780 2024-04-26T23:08:49.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.381+0000 7f1c3e200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c3815d570 con 0x7f1c38071e60 2024-04-26T23:08:49.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.381+0000 7f1c37400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c3806fe20 0x7f1c381f8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:49.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.381+0000 7f1c37400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c3806fe20 0x7f1c381f8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55898/0 (socket says 172.21.15.40:55898) 2024-04-26T23:08:49.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.381+0000 7f1c37400640 1 -- 172.21.15.40:0/211445427 learned_addr learned my addr 172.21.15.40:0/211445427 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:49.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.382+0000 7f1c37e00640 1 --2- 172.21.15.40:0/211445427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c38072780 0x7f1c38157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.382+0000 7f1c37400640 1 -- 172.21.15.40:0/211445427 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c38071e60 msgr2=0x7f1c380b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.382+0000 7f1c37400640 1 --2- 172.21.15.40:0/211445427 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c38071e60 0x7f1c380b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.382+0000 7f1c37400640 1 -- 172.21.15.40:0/211445427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c38072780 msgr2=0x7f1c38157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.382+0000 7f1c37400640 1 --2- 172.21.15.40:0/211445427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c38072780 0x7f1c38157b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.382+0000 7f1c37400640 1 -- 172.21.15.40:0/211445427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c380b4f30 con 0x7f1c3806fe20 2024-04-26T23:08:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.382+0000 7f1c37400640 1 --2- 172.21.15.40:0/211445427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c3806fe20 0x7f1c381f8a40 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1c28000c00 tx=0x7f1c280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.383+0000 7f1c2f400640 1 -- 172.21.15.40:0/211445427 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28037040 con 0x7f1c3806fe20 2024-04-26T23:08:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.383+0000 7f1c2f400640 1 -- 172.21.15.40:0/211445427 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c28042070 con 0x7f1c3806fe20 2024-04-26T23:08:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.383+0000 7f1c2f400640 1 -- 172.21.15.40:0/211445427 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28030bd0 con 0x7f1c3806fe20 2024-04-26T23:08:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.383+0000 7f1c3e200640 1 -- 172.21.15.40:0/211445427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c380b4cb0 con 0x7f1c3806fe20 2024-04-26T23:08:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.383+0000 7f1c3e200640 1 -- 172.21.15.40:0/211445427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c380c83a0 con 0x7f1c3806fe20 2024-04-26T23:08:49.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.384+0000 7f1c3e200640 1 -- 172.21.15.40:0/211445427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c380c7d40 con 0x7f1c3806fe20 2024-04-26T23:08:49.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.385+0000 7f1c2f400640 1 -- 172.21.15.40:0/211445427 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1c28047460 con 0x7f1c3806fe20 2024-04-26T23:08:49.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.386+0000 7f1c2f400640 1 --2- 172.21.15.40:0/211445427 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c1406c8f0 0x7f1c1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:49.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.386+0000 7f1c2f400640 1 -- 172.21.15.40:0/211445427 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6217+0+0 (secure 0 0 0) 0x7f1c280ac9c0 con 0x7f1c3806fe20 2024-04-26T23:08:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.387+0000 7f1c36a00640 1 --2- 172.21.15.40:0/211445427 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c1406c8f0 0x7f1c1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.388+0000 7f1c36a00640 1 --2- 172.21.15.40:0/211445427 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c1406c8f0 0x7f1c1406edb0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1c381590a0 tx=0x7f1c20008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.391+0000 7f1c2f400640 1 -- 172.21.15.40:0/211445427 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1c380b4cb0 con 0x7f1c3806fe20 2024-04-26T23:08:49.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.592+0000 7f1c3e200640 1 -- 172.21.15.40:0/211445427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1c380b1600 con 0x7f1c3806fe20 2024-04-26T23:08:49.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.593+0000 7f1c2f400640 1 -- 172.21.15.40:0/211445427 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f1c28032e20 con 0x7f1c3806fe20 2024-04-26T23:08:49.593 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:08:49.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.596+0000 7f1c3e200640 1 -- 172.21.15.40:0/211445427 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c1406c8f0 msgr2=0x7f1c1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:49.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.596+0000 7f1c3e200640 1 --2- 172.21.15.40:0/211445427 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c1406c8f0 0x7f1c1406edb0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1c381590a0 tx=0x7f1c20008040 comp rx=0 tx=0).stop 2024-04-26T23:08:49.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.596+0000 7f1c3e200640 1 -- 172.21.15.40:0/211445427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c3806fe20 msgr2=0x7f1c381f8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:49.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.596+0000 7f1c3e200640 1 --2- 172.21.15.40:0/211445427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c3806fe20 0x7f1c381f8a40 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1c28000c00 tx=0x7f1c280046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.596+0000 7f1c3e200640 1 -- 172.21.15.40:0/211445427 shutdown_connections 2024-04-26T23:08:49.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.596+0000 7f1c3e200640 1 --2- 172.21.15.40:0/211445427 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1c1406c8f0 0x7f1c1406edb0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.596+0000 7f1c3e200640 1 --2- 172.21.15.40:0/211445427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c38072780 0x7f1c38157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.596+0000 7f1c3e200640 1 --2- 172.21.15.40:0/211445427 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c38071e60 0x7f1c380b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.596+0000 7f1c3e200640 1 --2- 172.21.15.40:0/211445427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c3806fe20 0x7f1c381f8a40 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.596+0000 7f1c3e200640 1 -- 172.21.15.40:0/211445427 >> 172.21.15.40:0/211445427 conn(0x7f1c380d0be0 msgr2=0x7f1c380ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:49.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.596+0000 7f1c3e200640 1 -- 172.21.15.40:0/211445427 shutdown_connections 2024-04-26T23:08:49.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.596+0000 7f1c3e200640 1 -- 172.21.15.40:0/211445427 wait complete. 2024-04-26T23:08:49.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:49.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.832+0000 7f0187c00640 1 Processor -- start 2024-04-26T23:08:49.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.833+0000 7f0187c00640 1 -- start start 2024-04-26T23:08:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.835+0000 7f0187c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f01800d18f0 0x7f01800b2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.835+0000 7f0187c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01800d0710 con 0x7f0180160720 2024-04-26T23:08:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.835+0000 7f0187c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01800d0430 con 0x7f01800d0d00 2024-04-26T23:08:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.835+0000 7f0187c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f01800d01e0 con 0x7f01800d18f0 2024-04-26T23:08:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.835+0000 7f0185200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0180160720 0x7f01800ce620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60516/0 (socket says 172.21.15.40:60516) 2024-04-26T23:08:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.835+0000 7f0185200640 1 -- 172.21.15.40:0/1681666043 learned_addr learned my addr 172.21.15.40:0/1681666043 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:49.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.835+0000 7f0185c00640 1 --2- 172.21.15.40:0/1681666043 >> v2:172.21.15.40:3300/0 conn(0x7f01800d18f0 0x7f01800b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:49.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.835+0000 7f017f400640 1 -- 172.21.15.40:0/1681666043 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 819665339 0 0) 0x7f01800d0710 con 0x7f0180160720 2024-04-26T23:08:49.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.836+0000 7f0185c00640 1 -- 172.21.15.40:0/1681666043 >> v1:172.21.15.103:6789/0 conn(0x7f01800d0d00 legacy=0x7f01800d11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:49.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.836+0000 7f0185c00640 1 -- 172.21.15.40:0/1681666043 >> v1:172.21.15.40:6789/0 conn(0x7f0180160720 legacy=0x7f01800ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:49.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.836+0000 7f0185c00640 1 -- 172.21.15.40:0/1681666043 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01800cdb40 con 0x7f01800d18f0 2024-04-26T23:08:49.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.837+0000 7f0185c00640 1 --2- 172.21.15.40:0/1681666043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01800d18f0 0x7f01800b2570 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f0178009d90 tx=0x7f017802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b168ea567840c3ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:49.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.837+0000 7f017f400640 1 -- 172.21.15.40:0/1681666043 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178007740 con 0x7f01800d18f0 2024-04-26T23:08:49.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.837+0000 7f017f400640 1 -- 172.21.15.40:0/1681666043 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01780078e0 con 0x7f01800d18f0 2024-04-26T23:08:49.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.837+0000 7f017f400640 1 -- 172.21.15.40:0/1681666043 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178007cb0 con 0x7f01800d18f0 2024-04-26T23:08:49.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.837+0000 7f0187c00640 1 -- 172.21.15.40:0/1681666043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01800d18f0 msgr2=0x7f01800b2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:49.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.837+0000 7f0187c00640 1 --2- 172.21.15.40:0/1681666043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01800d18f0 0x7f01800b2570 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f0178009d90 tx=0x7f017802a6a0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.838+0000 7f0187c00640 1 -- 172.21.15.40:0/1681666043 shutdown_connections 2024-04-26T23:08:49.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.838+0000 7f0187c00640 1 --2- 172.21.15.40:0/1681666043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01800d18f0 0x7f01800b2570 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.838+0000 7f0187c00640 1 -- 172.21.15.40:0/1681666043 >> 172.21.15.40:0/1681666043 conn(0x7f01800d2e00 msgr2=0x7f01800d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:49.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.838+0000 7f0187c00640 1 -- 172.21.15.40:0/1681666043 shutdown_connections 2024-04-26T23:08:49.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.838+0000 7f0187c00640 1 -- 172.21.15.40:0/1681666043 wait complete. 2024-04-26T23:08:49.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.840+0000 7f0187c00640 1 Processor -- start 2024-04-26T23:08:49.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.840+0000 7f0187c00640 1 -- start start 2024-04-26T23:08:49.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.841+0000 7f0187c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01800d0d00 0x7f01800c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:49.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.841+0000 7f0187c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01800d18f0 0x7f01800a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:49.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.841+0000 7f0187c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0180160720 0x7f01800cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:49.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.841+0000 7f0187c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01800ced30 con 0x7f0180160720 2024-04-26T23:08:49.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.841+0000 7f0187c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01800ce100 con 0x7f01800d0d00 2024-04-26T23:08:49.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.841+0000 7f0187c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0180160b00 con 0x7f01800d18f0 2024-04-26T23:08:49.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.841+0000 7f0185c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0180160720 0x7f01800cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:49.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.841+0000 7f0185c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0180160720 0x7f01800cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55914/0 (socket says 172.21.15.40:55914) 2024-04-26T23:08:49.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.841+0000 7f0185c00640 1 -- 172.21.15.40:0/955659652 learned_addr learned my addr 172.21.15.40:0/955659652 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:49.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.841+0000 7f0185200640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01800d0d00 0x7f01800c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:49.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.842+0000 7f017fe00640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01800d18f0 0x7f01800a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:49.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.842+0000 7f0185c00640 1 -- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01800d18f0 msgr2=0x7f01800a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:49.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.842+0000 7f0185c00640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01800d18f0 0x7f01800a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.842+0000 7f0185c00640 1 -- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01800d0d00 msgr2=0x7f01800c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.842+0000 7f0185c00640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01800d0d00 0x7f01800c9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.842+0000 7f0185c00640 1 -- 172.21.15.40:0/955659652 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01800ca520 con 0x7f0180160720 2024-04-26T23:08:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.842+0000 7f0185200640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01800d0d00 0x7f01800c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:08:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.842+0000 7f0185c00640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0180160720 0x7f01800cc960 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f0178009eb0 tx=0x7f0178004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.843+0000 7f017cc00640 1 -- 172.21.15.40:0/955659652 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178040070 con 0x7f0180160720 2024-04-26T23:08:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.843+0000 7f017cc00640 1 -- 172.21.15.40:0/955659652 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0178031c90 con 0x7f0180160720 2024-04-26T23:08:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.843+0000 7f017cc00640 1 -- 172.21.15.40:0/955659652 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178045680 con 0x7f0180160720 2024-04-26T23:08:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.843+0000 7f0187c00640 1 -- 172.21.15.40:0/955659652 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01800ca800 con 0x7f0180160720 2024-04-26T23:08:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.843+0000 7f0187c00640 1 -- 172.21.15.40:0/955659652 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01800cadc0 con 0x7f0180160720 2024-04-26T23:08:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.844+0000 7f0187c00640 1 -- 172.21.15.40:0/955659652 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01800694f0 con 0x7f0180160720 2024-04-26T23:08:49.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.845+0000 7f017cc00640 1 -- 172.21.15.40:0/955659652 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0178039040 con 0x7f0180160720 2024-04-26T23:08:49.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.847+0000 7f017cc00640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f016406c920 0x7f016406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:49.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.847+0000 7f017cc00640 1 -- 172.21.15.40:0/955659652 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6217+0+0 (secure 0 0 0) 0x7f01780ad770 con 0x7f0180160720 2024-04-26T23:08:49.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.848+0000 7f0185200640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f016406c920 0x7f016406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:49.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.849+0000 7f0185200640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f016406c920 0x7f016406ede0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f0174000f60 tx=0x7f0174007cc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:49.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:49.852+0000 7f017cc00640 1 -- 172.21.15.40:0/955659652 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f017807bb80 con 0x7f0180160720 2024-04-26T23:08:50.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.098+0000 7f0187c00640 1 -- 172.21.15.40:0/955659652 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01800b8d80 con 0x7f0180160720 2024-04-26T23:08:50.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.099+0000 7f017cc00640 1 -- 172.21.15.40:0/955659652 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5232 (secure 0 0 0) 0x7f017807b480 con 0x7f0180160720 2024-04-26T23:08:50.099 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:50.100 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:26.911596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:50.100 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-04-26T23:08:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.101+0000 7f0187c00640 1 -- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f016406c920 msgr2=0x7f016406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.101+0000 7f0187c00640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f016406c920 0x7f016406ede0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f0174000f60 tx=0x7f0174007cc0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.101+0000 7f0187c00640 1 -- 172.21.15.40:0/955659652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0180160720 msgr2=0x7f01800cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.101+0000 7f0187c00640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0180160720 0x7f01800cc960 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f0178009eb0 tx=0x7f0178004a40 comp rx=0 tx=0).stop 2024-04-26T23:08:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.101+0000 7f0187c00640 1 -- 172.21.15.40:0/955659652 shutdown_connections 2024-04-26T23:08:50.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.101+0000 7f0187c00640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f016406c920 0x7f016406ede0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.101+0000 7f0187c00640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0180160720 0x7f01800cc960 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.101+0000 7f0187c00640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01800d18f0 0x7f01800a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.101+0000 7f0187c00640 1 --2- 172.21.15.40:0/955659652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01800d0d00 0x7f01800c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.101+0000 7f0187c00640 1 -- 172.21.15.40:0/955659652 >> 172.21.15.40:0/955659652 conn(0x7f01800d2e00 msgr2=0x7f01800d11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:50.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.102+0000 7f0187c00640 1 -- 172.21.15.40:0/955659652 shutdown_connections 2024-04-26T23:08:50.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.102+0000 7f0187c00640 1 -- 172.21.15.40:0/955659652 wait complete. 2024-04-26T23:08:50.122 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:08:50.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.340+0000 7ff36f200640 1 Processor -- start 2024-04-26T23:08:50.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.340+0000 7ff36f200640 1 -- start start 2024-04-26T23:08:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.342+0000 7ff36f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff3680c41d0 0x7ff3680cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.342+0000 7ff36f200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff368073f30 0x7ff3681671e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.342+0000 7ff36f200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff368163a50 con 0x7ff368167720 2024-04-26T23:08:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.342+0000 7ff36f200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff3680cf7a0 con 0x7ff368073f30 2024-04-26T23:08:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.342+0000 7ff36f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff368167b60 con 0x7ff3680c41d0 2024-04-26T23:08:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.342+0000 7ff35fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff368073f30 0x7ff3681671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.342+0000 7ff36d200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff368167720 0x7ff36806ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60518/0 (socket says 172.21.15.40:60518) 2024-04-26T23:08:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.342+0000 7ff36d200640 1 -- 172.21.15.40:0/1919870124 learned_addr learned my addr 172.21.15.40:0/1919870124 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.343+0000 7ff367e00640 1 --2- 172.21.15.40:0/1919870124 >> v2:172.21.15.103:3300/0 conn(0x7ff3680c41d0 0x7ff3680cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.343+0000 7ff35fe00640 1 -- 172.21.15.40:0/1919870124 >> v2:172.21.15.103:3300/0 conn(0x7ff3680c41d0 msgr2=0x7ff3680cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.343+0000 7ff35fe00640 1 --2- 172.21.15.40:0/1919870124 >> v2:172.21.15.103:3300/0 conn(0x7ff3680c41d0 0x7ff3680cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.343+0000 7ff35fe00640 1 -- 172.21.15.40:0/1919870124 >> v1:172.21.15.40:6789/0 conn(0x7ff368167720 legacy=0x7ff36806ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.343+0000 7ff35fe00640 1 -- 172.21.15.40:0/1919870124 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3680ceb90 con 0x7ff368073f30 2024-04-26T23:08:50.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.344+0000 7ff35fe00640 1 --2- 172.21.15.40:0/1919870124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff368073f30 0x7ff3681671e0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7ff3680b3730 tx=0x7ff35002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a66086bffcddeb5a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:50.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.344+0000 7ff367400640 1 -- 172.21.15.40:0/1919870124 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff350037070 con 0x7ff368073f30 2024-04-26T23:08:50.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.344+0000 7ff367400640 1 -- 172.21.15.40:0/1919870124 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff350032440 con 0x7ff368073f30 2024-04-26T23:08:50.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.344+0000 7ff367400640 1 -- 172.21.15.40:0/1919870124 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff350032810 con 0x7ff368073f30 2024-04-26T23:08:50.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.345+0000 7ff36f200640 1 -- 172.21.15.40:0/1919870124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff368073f30 msgr2=0x7ff3681671e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:50.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.345+0000 7ff36f200640 1 --2- 172.21.15.40:0/1919870124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff368073f30 0x7ff3681671e0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7ff3680b3730 tx=0x7ff35002a1a0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.345+0000 7ff36f200640 1 -- 172.21.15.40:0/1919870124 shutdown_connections 2024-04-26T23:08:50.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.345+0000 7ff36f200640 1 --2- 172.21.15.40:0/1919870124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff368073f30 0x7ff3681671e0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.345+0000 7ff36f200640 1 --2- 172.21.15.40:0/1919870124 >> v2:172.21.15.103:3300/0 conn(0x7ff3680c41d0 0x7ff3680cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.345+0000 7ff36f200640 1 -- 172.21.15.40:0/1919870124 >> 172.21.15.40:0/1919870124 conn(0x7ff3680d0be0 msgr2=0x7ff3680d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:50.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.345+0000 7ff36f200640 1 -- 172.21.15.40:0/1919870124 shutdown_connections 2024-04-26T23:08:50.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.345+0000 7ff36f200640 1 -- 172.21.15.40:0/1919870124 wait complete. 2024-04-26T23:08:50.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.347+0000 7ff36f200640 1 Processor -- start 2024-04-26T23:08:50.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.347+0000 7ff36f200640 1 -- start start 2024-04-26T23:08:50.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.348+0000 7ff36f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff368073f30 0x7ff3681510f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.348+0000 7ff36f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3680c41d0 0x7ff368166230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.348+0000 7ff36f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff368167720 0x7ff3680c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.348+0000 7ff36f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff3680cf430 con 0x7ff368073f30 2024-04-26T23:08:50.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.348+0000 7ff36f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff3680cf150 con 0x7ff3680c41d0 2024-04-26T23:08:50.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.348+0000 7ff36f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff3680d0170 con 0x7ff368167720 2024-04-26T23:08:50.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.349+0000 7ff367e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff368073f30 0x7ff3681510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:50.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.349+0000 7ff367e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff368073f30 0x7ff3681510f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55940/0 (socket says 172.21.15.40:55940) 2024-04-26T23:08:50.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.349+0000 7ff367e00640 1 -- 172.21.15.40:0/3235427353 learned_addr learned my addr 172.21.15.40:0/3235427353 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:50.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.349+0000 7ff367e00640 1 -- 172.21.15.40:0/3235427353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff368167720 msgr2=0x7ff3680c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:50.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.349+0000 7ff367e00640 1 --2- 172.21.15.40:0/3235427353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff368167720 0x7ff3680c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.349+0000 7ff367e00640 1 -- 172.21.15.40:0/3235427353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3680c41d0 msgr2=0x7ff368166230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:50.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.349+0000 7ff367e00640 1 --2- 172.21.15.40:0/3235427353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3680c41d0 0x7ff368166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.349+0000 7ff367e00640 1 -- 172.21.15.40:0/3235427353 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff368165470 con 0x7ff368073f30 2024-04-26T23:08:50.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.349+0000 7ff367e00640 1 --2- 172.21.15.40:0/3235427353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff368073f30 0x7ff3681510f0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7ff35800d800 tx=0x7ff35800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:50.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.350+0000 7ff364c00640 1 -- 172.21.15.40:0/3235427353 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35800b6c0 con 0x7ff368073f30 2024-04-26T23:08:50.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.350+0000 7ff364c00640 1 -- 172.21.15.40:0/3235427353 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff358019070 con 0x7ff368073f30 2024-04-26T23:08:50.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.350+0000 7ff364c00640 1 -- 172.21.15.40:0/3235427353 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35800ba80 con 0x7ff368073f30 2024-04-26T23:08:50.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.350+0000 7ff36f200640 1 -- 172.21.15.40:0/3235427353 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff368165750 con 0x7ff368073f30 2024-04-26T23:08:50.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.350+0000 7ff36f200640 1 -- 172.21.15.40:0/3235427353 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff368165d10 con 0x7ff368073f30 2024-04-26T23:08:50.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.351+0000 7ff36f200640 1 -- 172.21.15.40:0/3235427353 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3680694f0 con 0x7ff368073f30 2024-04-26T23:08:50.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.352+0000 7ff364c00640 1 -- 172.21.15.40:0/3235427353 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff35800bc20 con 0x7ff368073f30 2024-04-26T23:08:50.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.353+0000 7ff364c00640 1 --2- 172.21.15.40:0/3235427353 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff34406c8f0 0x7ff34406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.353+0000 7ff364c00640 1 -- 172.21.15.40:0/3235427353 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6217+0+0 (secure 0 0 0) 0x7ff35808c170 con 0x7ff368073f30 2024-04-26T23:08:50.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.354+0000 7ff35fe00640 1 --2- 172.21.15.40:0/3235427353 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff34406c8f0 0x7ff34406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:50.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.355+0000 7ff35fe00640 1 --2- 172.21.15.40:0/3235427353 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff34406c8f0 0x7ff34406edb0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff350034000 tx=0x7ff35002c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:50.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.358+0000 7ff364c00640 1 -- 172.21.15.40:0/3235427353 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff358011340 con 0x7ff368073f30 2024-04-26T23:08:50.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.565+0000 7ff36f200640 1 -- 172.21.15.40:0/3235427353 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3680d0430 con 0x7ff368073f30 2024-04-26T23:08:50.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.567+0000 7ff364c00640 1 -- 172.21.15.40:0/3235427353 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+17042 (secure 0 0 0) 0x7ff35805a430 con 0x7ff368073f30 2024-04-26T23:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:50.568 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":92,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:08:49.208604+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:07:46.137238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-26T23:07:47.258518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":5}]},{"pgid":"14.1b","mappings":[{"from":1,"to":2}]},{"pgid":"14.21","mappings":[{"from":1,"to":7}]},{"pgid":"14.36","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:08:50.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.569+0000 7ff36f200640 1 -- 172.21.15.40:0/3235427353 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff34406c8f0 msgr2=0x7ff34406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:50.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.569+0000 7ff36f200640 1 --2- 172.21.15.40:0/3235427353 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff34406c8f0 0x7ff34406edb0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff350034000 tx=0x7ff35002c000 comp rx=0 tx=0).stop 2024-04-26T23:08:50.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.569+0000 7ff36f200640 1 -- 172.21.15.40:0/3235427353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff368073f30 msgr2=0x7ff3681510f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:50.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.569+0000 7ff36f200640 1 --2- 172.21.15.40:0/3235427353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff368073f30 0x7ff3681510f0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7ff35800d800 tx=0x7ff35800dcc0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.569+0000 7ff36f200640 1 -- 172.21.15.40:0/3235427353 shutdown_connections 2024-04-26T23:08:50.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.569+0000 7ff36f200640 1 --2- 172.21.15.40:0/3235427353 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff34406c8f0 0x7ff34406edb0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.569+0000 7ff36f200640 1 --2- 172.21.15.40:0/3235427353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff368167720 0x7ff3680c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.569+0000 7ff36f200640 1 --2- 172.21.15.40:0/3235427353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3680c41d0 0x7ff368166230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.569+0000 7ff36f200640 1 --2- 172.21.15.40:0/3235427353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff368073f30 0x7ff3681510f0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.569+0000 7ff36f200640 1 -- 172.21.15.40:0/3235427353 >> 172.21.15.40:0/3235427353 conn(0x7ff3680d0be0 msgr2=0x7ff368162220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:50.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.570+0000 7ff36f200640 1 -- 172.21.15.40:0/3235427353 shutdown_connections 2024-04-26T23:08:50.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.570+0000 7ff36f200640 1 -- 172.21.15.40:0/3235427353 wait complete. 2024-04-26T23:08:50.591 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T23:08:50.591 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:50.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.804+0000 7effd7e00640 1 Processor -- start 2024-04-26T23:08:50.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.805+0000 7effd7e00640 1 -- start start 2024-04-26T23:08:50.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.806+0000 7effd7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7effd80d3740 0x7effd80b9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.806+0000 7effd7e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effd8162c40 con 0x7effd80d32e0 2024-04-26T23:08:50.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.806+0000 7effd7e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effd80d2d80 con 0x7effd80b6220 2024-04-26T23:08:50.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.806+0000 7effd7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7effd80b7c80 con 0x7effd80d3740 2024-04-26T23:08:50.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.807+0000 7effd6000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7effd80d32e0 0x7effd80b8d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60530/0 (socket says 172.21.15.40:60530) 2024-04-26T23:08:50.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.807+0000 7effd6000640 1 -- 172.21.15.40:0/304633210 learned_addr learned my addr 172.21.15.40:0/304633210 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:50.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.807+0000 7effd7400640 1 --2- 172.21.15.40:0/304633210 >> v2:172.21.15.103:3300/0 conn(0x7effd80d3740 0x7effd80b9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.807+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1190921001 0 0) 0x7effd8162c40 con 0x7effd80d32e0 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.807+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effbc003520 con 0x7effd80d32e0 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3824500886 0 0) 0x7effbc003520 con 0x7effd80d32e0 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 >> v2:172.21.15.103:3300/0 conn(0x7effd80d3740 msgr2=0x7effd80b9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd5600640 1 --2- 172.21.15.40:0/304633210 >> v2:172.21.15.103:3300/0 conn(0x7effd80d3740 0x7effd80b9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 >> v1:172.21.15.103:6789/0 conn(0x7effd80b6220 legacy=0x7effd80d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd80b7f60 con 0x7effd80d32e0 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7effc0001670 con 0x7effd80d32e0 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 >> v1:172.21.15.40:6789/0 conn(0x7effd80d32e0 legacy=0x7effd80b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd5600640 1 --2- 172.21.15.40:0/304633210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effbc003a00 0x7effbc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd5600640 1 --2- 172.21.15.40:0/304633210 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effbc004800 0x7effbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd5600640 1 --2- 172.21.15.40:0/304633210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effbc0051c0 0x7effbc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7effd80b7c80 con 0x7effbc0051c0 2024-04-26T23:08:50.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7effbc00c0c0 con 0x7effbc003a00 2024-04-26T23:08:50.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7effbc00c2c0 con 0x7effbc004800 2024-04-26T23:08:50.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd7400640 1 --2- 172.21.15.40:0/304633210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effbc0051c0 0x7effbc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:50.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd6a00640 1 --2- 172.21.15.40:0/304633210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effbc003a00 0x7effbc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:50.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.808+0000 7effd6000640 1 --2- 172.21.15.40:0/304633210 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effbc004800 0x7effbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:50.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.809+0000 7effd7400640 1 -- 172.21.15.40:0/304633210 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effbc004800 msgr2=0x7effbc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:50.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.809+0000 7effd7400640 1 --2- 172.21.15.40:0/304633210 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effbc004800 0x7effbc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.809+0000 7effd7400640 1 -- 172.21.15.40:0/304633210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effbc003a00 msgr2=0x7effbc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:50.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.809+0000 7effd7400640 1 --2- 172.21.15.40:0/304633210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effbc003a00 0x7effbc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.809+0000 7effd7400640 1 -- 172.21.15.40:0/304633210 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effbc003520 con 0x7effbc0051c0 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.809+0000 7effd6a00640 1 --2- 172.21.15.40:0/304633210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effbc003a00 0x7effbc003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.810+0000 7effd7400640 1 -- 172.21.15.40:0/304633210 reap_dead start 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.810+0000 7effd7400640 1 --2- 172.21.15.40:0/304633210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effbc0051c0 0x7effbc00b9c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7effd0009ac0 tx=0x7effbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974ec4550b3ea651 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.810+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0009690 con 0x7effbc0051c0 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.810+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effd0004020 con 0x7effbc0051c0 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.810+0000 7effd5600640 1 -- 172.21.15.40:0/304633210 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd00043d0 con 0x7effbc0051c0 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.811+0000 7effd7e00640 1 -- 172.21.15.40:0/304633210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effbc0051c0 msgr2=0x7effbc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.811+0000 7effd7e00640 1 --2- 172.21.15.40:0/304633210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effbc0051c0 0x7effbc00b9c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7effd0009ac0 tx=0x7effbc005830 comp rx=0 tx=0).stop 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.811+0000 7effd7e00640 1 -- 172.21.15.40:0/304633210 shutdown_connections 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.811+0000 7effd7e00640 1 --2- 172.21.15.40:0/304633210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effbc0051c0 0x7effbc00b9c0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.811+0000 7effd7e00640 1 -- 172.21.15.40:0/304633210 >> 172.21.15.40:0/304633210 conn(0x7effd80d4d90 msgr2=0x7effd80d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.811+0000 7effd7e00640 1 -- 172.21.15.40:0/304633210 shutdown_connections 2024-04-26T23:08:50.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.811+0000 7effd7e00640 1 -- 172.21.15.40:0/304633210 wait complete. 2024-04-26T23:08:50.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.813+0000 7effd7e00640 1 Processor -- start 2024-04-26T23:08:50.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.813+0000 7effd7e00640 1 -- start start 2024-04-26T23:08:50.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.814+0000 7effd7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effbc0051c0 0x7effd80caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.814+0000 7effd7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd815e700 0x7effd815eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.814+0000 7effd7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effd815d540 0x7effd815d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.814+0000 7effd7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7effd80d3b80 con 0x7effd815e700 2024-04-26T23:08:50.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.814+0000 7effd7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7effd80b8520 con 0x7effd815d540 2024-04-26T23:08:50.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.814+0000 7effd7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7effd80d0390 con 0x7effbc0051c0 2024-04-26T23:08:50.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.815+0000 7effd6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd815e700 0x7effd815eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:50.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.815+0000 7effd6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd815e700 0x7effd815eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55952/0 (socket says 172.21.15.40:55952) 2024-04-26T23:08:50.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.815+0000 7effd6000640 1 -- 172.21.15.40:0/2540331352 learned_addr learned my addr 172.21.15.40:0/2540331352 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:50.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.815+0000 7effd6a00640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effbc0051c0 0x7effd80caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:50.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.815+0000 7effd7400640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effd815d540 0x7effd815d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:50.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.815+0000 7effd6000640 1 -- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effbc0051c0 msgr2=0x7effd80caad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:50.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.815+0000 7effd6000640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effbc0051c0 0x7effd80caad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.815+0000 7effd6000640 1 -- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effd815d540 msgr2=0x7effd815d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:50.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.815+0000 7effd6000640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effd815d540 0x7effd815d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:50.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.815+0000 7effd6000640 1 -- 172.21.15.40:0/2540331352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd8150a00 con 0x7effd815e700 2024-04-26T23:08:50.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.815+0000 7effd7400640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effd815d540 0x7effd815d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:50.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.815+0000 7effd6000640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd815e700 0x7effd815eb80 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7effc000e800 tx=0x7effc000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:50.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.816+0000 7effcea00640 1 -- 172.21.15.40:0/2540331352 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc000c660 con 0x7effd815e700 2024-04-26T23:08:50.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.816+0000 7effcea00640 1 -- 172.21.15.40:0/2540331352 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effc0018070 con 0x7effd815e700 2024-04-26T23:08:50.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.816+0000 7effcea00640 1 -- 172.21.15.40:0/2540331352 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc000ca40 con 0x7effd815e700 2024-04-26T23:08:50.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.816+0000 7effd7e00640 1 -- 172.21.15.40:0/2540331352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effd80cb160 con 0x7effd815e700 2024-04-26T23:08:50.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.816+0000 7effd7e00640 1 -- 172.21.15.40:0/2540331352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effd80cb720 con 0x7effd815e700 2024-04-26T23:08:50.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.817+0000 7effd7e00640 1 -- 172.21.15.40:0/2540331352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effd8069370 con 0x7effd815e700 2024-04-26T23:08:50.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.818+0000 7effcea00640 1 -- 172.21.15.40:0/2540331352 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7effc000cbe0 con 0x7effd815e700 2024-04-26T23:08:50.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.819+0000 7effcea00640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7effb406c920 0x7effb406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:50.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.819+0000 7effcea00640 1 -- 172.21.15.40:0/2540331352 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6217+0+0 (secure 0 0 0) 0x7effc008baf0 con 0x7effd815e700 2024-04-26T23:08:50.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.820+0000 7effd6a00640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7effb406c920 0x7effb406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:50.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.821+0000 7effd6a00640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7effb406c920 0x7effb406ede0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7effc8007fd0 tx=0x7effc8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:50.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:50.823+0000 7effcea00640 1 -- 172.21.15.40:0/2540331352 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7effc0059c80 con 0x7effd815e700 2024-04-26T23:08:51.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.065+0000 7effd7e00640 1 -- 172.21.15.40:0/2540331352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effd80b9120 con 0x7effd815e700 2024-04-26T23:08:51.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.067+0000 7effcea00640 1 -- 172.21.15.40:0/2540331352 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5232 (secure 0 0 0) 0x7effc0059580 con 0x7effd815e700 2024-04-26T23:08:51.067 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:51.067 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:26.911596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:51.067 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-04-26T23:08:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.069+0000 7effd7e00640 1 -- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7effb406c920 msgr2=0x7effb406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.069+0000 7effd7e00640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7effb406c920 0x7effb406ede0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7effc8007fd0 tx=0x7effc8009040 comp rx=0 tx=0).stop 2024-04-26T23:08:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.069+0000 7effd7e00640 1 -- 172.21.15.40:0/2540331352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd815e700 msgr2=0x7effd815eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.069+0000 7effd7e00640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd815e700 0x7effd815eb80 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7effc000e800 tx=0x7effc000ecc0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.069+0000 7effd7e00640 1 -- 172.21.15.40:0/2540331352 shutdown_connections 2024-04-26T23:08:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.069+0000 7effd7e00640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7effb406c920 0x7effb406ede0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.069+0000 7effd7e00640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effd815d540 0x7effd815d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.069+0000 7effd7e00640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effd815e700 0x7effd815eb80 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.069+0000 7effd7e00640 1 --2- 172.21.15.40:0/2540331352 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7effbc0051c0 0x7effd80caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.069+0000 7effd7e00640 1 -- 172.21.15.40:0/2540331352 >> 172.21.15.40:0/2540331352 conn(0x7effd80d4d90 msgr2=0x7effd80b6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:51.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.070+0000 7effd7e00640 1 -- 172.21.15.40:0/2540331352 shutdown_connections 2024-04-26T23:08:51.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.070+0000 7effd7e00640 1 -- 172.21.15.40:0/2540331352 wait complete. 2024-04-26T23:08:51.089 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:08:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.284+0000 7f0f6e000640 1 Processor -- start 2024-04-26T23:08:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.284+0000 7f0f6e000640 1 -- start start 2024-04-26T23:08:51.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.285+0000 7f0f6e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f68162880 0x7f0f680d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:51.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.285+0000 7f0f6e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0f680b5f20 0x7f0f680d34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:51.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.285+0000 7f0f6e000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f680b7e50 con 0x7f0f680d39f0 2024-04-26T23:08:51.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.285+0000 7f0f6e000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f0f680d0580 con 0x7f0f680b5f20 2024-04-26T23:08:51.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.285+0000 7f0f6e000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0f680d1800 con 0x7f0f68162880 2024-04-26T23:08:51.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.286+0000 7f0f66a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0f680b5f20 0x7f0f680d34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:51.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.286+0000 7f0f66a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0f680b5f20 0x7f0f680d34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55968/0 (socket says 172.21.15.40:55968) 2024-04-26T23:08:51.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.286+0000 7f0f66a00640 1 -- 172.21.15.40:0/2535165805 learned_addr learned my addr 172.21.15.40:0/2535165805 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:51.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.286+0000 7f0f67400640 1 --2- 172.21.15.40:0/2535165805 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f68162880 0x7f0f680d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:51.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.286+0000 7f0f66a00640 1 -- 172.21.15.40:0/2535165805 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f68162880 msgr2=0x7f0f680d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:51.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.286+0000 7f0f66a00640 1 --2- 172.21.15.40:0/2535165805 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f68162880 0x7f0f680d2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.286+0000 7f0f66a00640 1 -- 172.21.15.40:0/2535165805 >> v1:172.21.15.40:6789/0 conn(0x7f0f680d39f0 legacy=0x7f0f680b69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:51.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.286+0000 7f0f66a00640 1 -- 172.21.15.40:0/2535165805 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f680d07d0 con 0x7f0f680b5f20 2024-04-26T23:08:51.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.286+0000 7f0f67400640 1 --2- 172.21.15.40:0/2535165805 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f68162880 0x7f0f680d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:51.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.287+0000 7f0f66a00640 1 --2- 172.21.15.40:0/2535165805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f680b5f20 0x7f0f680d34b0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f0f50009870 tx=0x7f0f5002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7415b0e0a7c9c1b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:51.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.287+0000 7f0f66000640 1 -- 172.21.15.40:0/2535165805 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f50037070 con 0x7f0f680b5f20 2024-04-26T23:08:51.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.288+0000 7f0f66000640 1 -- 172.21.15.40:0/2535165805 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f50032440 con 0x7f0f680b5f20 2024-04-26T23:08:51.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.288+0000 7f0f66000640 1 -- 172.21.15.40:0/2535165805 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f50032810 con 0x7f0f680b5f20 2024-04-26T23:08:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.288+0000 7f0f6e000640 1 -- 172.21.15.40:0/2535165805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f680b5f20 msgr2=0x7f0f680d34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.288+0000 7f0f6e000640 1 --2- 172.21.15.40:0/2535165805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f680b5f20 0x7f0f680d34b0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f0f50009870 tx=0x7f0f5002a250 comp rx=0 tx=0).stop 2024-04-26T23:08:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.288+0000 7f0f6e000640 1 -- 172.21.15.40:0/2535165805 shutdown_connections 2024-04-26T23:08:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.288+0000 7f0f6e000640 1 --2- 172.21.15.40:0/2535165805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f680b5f20 0x7f0f680d34b0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.288+0000 7f0f6e000640 1 --2- 172.21.15.40:0/2535165805 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f68162880 0x7f0f680d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.288+0000 7f0f6e000640 1 -- 172.21.15.40:0/2535165805 >> 172.21.15.40:0/2535165805 conn(0x7f0f680d4f60 msgr2=0x7f0f680d5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.288+0000 7f0f6e000640 1 -- 172.21.15.40:0/2535165805 shutdown_connections 2024-04-26T23:08:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.288+0000 7f0f6e000640 1 -- 172.21.15.40:0/2535165805 wait complete. 2024-04-26T23:08:51.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.290+0000 7f0f6e000640 1 Processor -- start 2024-04-26T23:08:51.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.290+0000 7f0f6e000640 1 -- start start 2024-04-26T23:08:51.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.291+0000 7f0f6e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f680b5f20 0x7f0f6815f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:51.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.291+0000 7f0f6e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f680d39f0 0x7f0f6815c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:51.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.291+0000 7f0f6e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f68162880 0x7f0f6815c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.291+0000 7f0f6e000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0f68162c60 con 0x7f0f68162880 2024-04-26T23:08:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.291+0000 7f0f6e000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f680d0d90 con 0x7f0f680d39f0 2024-04-26T23:08:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.291+0000 7f0f6e000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0f68162de0 con 0x7f0f680b5f20 2024-04-26T23:08:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.291+0000 7f0f67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f68162880 0x7f0f6815c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.291+0000 7f0f67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f68162880 0x7f0f6815c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55982/0 (socket says 172.21.15.40:55982) 2024-04-26T23:08:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.291+0000 7f0f67e00640 1 -- 172.21.15.40:0/3469514364 learned_addr learned my addr 172.21.15.40:0/3469514364 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.292+0000 7f0f67400640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f680b5f20 0x7f0f6815f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.292+0000 7f0f66a00640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f680d39f0 0x7f0f6815c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.292+0000 7f0f67e00640 1 -- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f680b5f20 msgr2=0x7f0f6815f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.292+0000 7f0f67e00640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f680b5f20 0x7f0f6815f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.292+0000 7f0f67e00640 1 -- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f680d39f0 msgr2=0x7f0f6815c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.292+0000 7f0f67e00640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f680d39f0 0x7f0f6815c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.292+0000 7f0f67e00640 1 -- 172.21.15.40:0/3469514364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f680b3450 con 0x7f0f68162880 2024-04-26T23:08:51.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.292+0000 7f0f67e00640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f68162880 0x7f0f6815c5a0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f0f60001eb0 tx=0x7f0f6000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:51.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.292+0000 7f0f5f400640 1 -- 172.21.15.40:0/3469514364 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f60017070 con 0x7f0f68162880 2024-04-26T23:08:51.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.293+0000 7f0f5f400640 1 -- 172.21.15.40:0/3469514364 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f600052d0 con 0x7f0f68162880 2024-04-26T23:08:51.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.293+0000 7f0f5f400640 1 -- 172.21.15.40:0/3469514364 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f60003810 con 0x7f0f68162880 2024-04-26T23:08:51.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.293+0000 7f0f6e000640 1 -- 172.21.15.40:0/3469514364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f680b4e80 con 0x7f0f68162880 2024-04-26T23:08:51.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.293+0000 7f0f6e000640 1 -- 172.21.15.40:0/3469514364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f680b5900 con 0x7f0f68162880 2024-04-26T23:08:51.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.294+0000 7f0f6e000640 1 -- 172.21.15.40:0/3469514364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f680694f0 con 0x7f0f68162880 2024-04-26T23:08:51.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.295+0000 7f0f5f400640 1 -- 172.21.15.40:0/3469514364 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0f6001a070 con 0x7f0f68162880 2024-04-26T23:08:51.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.296+0000 7f0f5f400640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0f4406c8f0 0x7f0f4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:51.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.296+0000 7f0f5f400640 1 -- 172.21.15.40:0/3469514364 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6217+0+0 (secure 0 0 0) 0x7f0f6008b150 con 0x7f0f68162880 2024-04-26T23:08:51.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.297+0000 7f0f67400640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0f4406c8f0 0x7f0f4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:51.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.298+0000 7f0f67400640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0f4406c8f0 0x7f0f4406edb0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0f58008f80 tx=0x7f0f580084d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:51.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.300+0000 7f0f5f400640 1 -- 172.21.15.40:0/3469514364 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0f680b4e80 con 0x7f0f68162880 2024-04-26T23:08:51.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.550+0000 7f0f6e000640 1 -- 172.21.15.40:0/3469514364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f680cb430 con 0x7f0f68162880 2024-04-26T23:08:51.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.551+0000 7f0f5f400640 1 -- 172.21.15.40:0/3469514364 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5232 (secure 0 0 0) 0x7f0f600592e0 con 0x7f0f68162880 2024-04-26T23:08:51.553 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:51.553 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:07:48.279070+0000","modified":"2024-04-26T23:08:26.911596+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8040,"mds_1":8031,"mds_2":6253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6253":{"gid":6253,"name":"d","rank":2,"incarnation":95,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2728939257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2728939257},{"type":"v1","addr":"172.21.15.103:6837","nonce":2728939257}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8031":{"gid":8031,"name":"a","rank":1,"incarnation":93,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/1811310743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1811310743},{"type":"v1","addr":"172.21.15.40:6833","nonce":1811310743}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/88401598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":88401598},{"type":"v1","addr":"172.21.15.40:6835","nonce":88401598}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8040,"qdb_cluster":[6253,8031,8040]},"id":6}]} 2024-04-26T23:08:51.553 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-04-26T23:08:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.554+0000 7f0f6e000640 1 -- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0f4406c8f0 msgr2=0x7f0f4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.554+0000 7f0f6e000640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0f4406c8f0 0x7f0f4406edb0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0f58008f80 tx=0x7f0f580084d0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.554+0000 7f0f6e000640 1 -- 172.21.15.40:0/3469514364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f68162880 msgr2=0x7f0f6815c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.554+0000 7f0f6e000640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f68162880 0x7f0f6815c5a0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f0f60001eb0 tx=0x7f0f6000c890 comp rx=0 tx=0).stop 2024-04-26T23:08:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.554+0000 7f0f6e000640 1 -- 172.21.15.40:0/3469514364 shutdown_connections 2024-04-26T23:08:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.554+0000 7f0f6e000640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0f4406c8f0 0x7f0f4406edb0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.554+0000 7f0f6e000640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f68162880 0x7f0f6815c5a0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.554+0000 7f0f6e000640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f680d39f0 0x7f0f6815c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.554+0000 7f0f6e000640 1 --2- 172.21.15.40:0/3469514364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f680b5f20 0x7f0f6815f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.554+0000 7f0f6e000640 1 -- 172.21.15.40:0/3469514364 >> 172.21.15.40:0/3469514364 conn(0x7f0f680d4f60 msgr2=0x7f0f680d5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:51.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.554+0000 7f0f6e000640 1 -- 172.21.15.40:0/3469514364 shutdown_connections 2024-04-26T23:08:51.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.554+0000 7f0f6e000640 1 -- 172.21.15.40:0/3469514364 wait complete. 2024-04-26T23:08:51.574 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:08:51.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.798+0000 7f5812200640 1 Processor -- start 2024-04-26T23:08:51.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.799+0000 7f5812200640 1 -- start start 2024-04-26T23:08:51.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.800+0000 7f5812200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f580c0b6220 0x7f580c0d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:51.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.800+0000 7f5812200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f580c162c40 con 0x7f580c0d08d0 2024-04-26T23:08:51.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.800+0000 7f5812200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f580c0d2d80 con 0x7f580c0d39f0 2024-04-26T23:08:51.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.800+0000 7f5812200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f580c0b7c80 con 0x7f580c0b6220 2024-04-26T23:08:51.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.801+0000 7f5810e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f580c0b6220 0x7f580c0d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:51.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.801+0000 7f580be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f580c0d08d0 0x7f580c0d32e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60546/0 (socket says 172.21.15.40:60546) 2024-04-26T23:08:51.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.801+0000 7f580be00640 1 -- 172.21.15.40:0/1319205716 learned_addr learned my addr 172.21.15.40:0/1319205716 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:51.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.801+0000 7f5810e00640 1 -- 172.21.15.40:0/1319205716 >> v1:172.21.15.103:6789/0 conn(0x7f580c0d39f0 legacy=0x7f580c0b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:51.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.801+0000 7f5810e00640 1 -- 172.21.15.40:0/1319205716 >> v1:172.21.15.40:6789/0 conn(0x7f580c0d08d0 legacy=0x7f580c0d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:51.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.801+0000 7f5810e00640 1 -- 172.21.15.40:0/1319205716 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f580c0b7f60 con 0x7f580c0b6220 2024-04-26T23:08:51.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.801+0000 7f580b400640 1 -- 172.21.15.40:0/1319205716 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 438405753 0 0) 0x7f580c162c40 con 0x7f580c0d08d0 2024-04-26T23:08:51.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.802+0000 7f5810e00640 1 --2- 172.21.15.40:0/1319205716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580c0b6220 0x7f580c0d0390 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f57fc0097a0 tx=0x7f57fc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc2b661d1a05457 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:51.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.803+0000 7f580b400640 1 -- 172.21.15.40:0/1319205716 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc037070 con 0x7f580c0b6220 2024-04-26T23:08:51.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.803+0000 7f580b400640 1 -- 172.21.15.40:0/1319205716 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57fc032440 con 0x7f580c0b6220 2024-04-26T23:08:51.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.803+0000 7f580b400640 1 -- 172.21.15.40:0/1319205716 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc032810 con 0x7f580c0b6220 2024-04-26T23:08:51.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.803+0000 7f5812200640 1 -- 172.21.15.40:0/1319205716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580c0b6220 msgr2=0x7f580c0d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:51.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.803+0000 7f5812200640 1 --2- 172.21.15.40:0/1319205716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580c0b6220 0x7f580c0d0390 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f57fc0097a0 tx=0x7f57fc02a1e0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.803+0000 7f5812200640 1 -- 172.21.15.40:0/1319205716 shutdown_connections 2024-04-26T23:08:51.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.803+0000 7f5812200640 1 --2- 172.21.15.40:0/1319205716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580c0b6220 0x7f580c0d0390 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.803+0000 7f5812200640 1 -- 172.21.15.40:0/1319205716 >> 172.21.15.40:0/1319205716 conn(0x7f580c0d4d90 msgr2=0x7f580c0d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:51.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.803+0000 7f5812200640 1 -- 172.21.15.40:0/1319205716 shutdown_connections 2024-04-26T23:08:51.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.804+0000 7f5812200640 1 -- 172.21.15.40:0/1319205716 wait complete. 2024-04-26T23:08:51.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.805+0000 7f5812200640 1 Processor -- start 2024-04-26T23:08:51.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.805+0000 7f5812200640 1 -- start start 2024-04-26T23:08:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.806+0000 7f5812200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580c0b6220 0x7f580c161410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.806+0000 7f5812200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f580c0d08d0 0x7f580c0b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.806+0000 7f5812200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f580c0d39f0 0x7f580c0b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.806+0000 7f5812200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f580c0b9430 con 0x7f580c0b6220 2024-04-26T23:08:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.806+0000 7f5812200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f580c0b8520 con 0x7f580c0d08d0 2024-04-26T23:08:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.806+0000 7f5812200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f580c0b95b0 con 0x7f580c0d39f0 2024-04-26T23:08:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.807+0000 7f5810e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580c0b6220 0x7f580c161410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.807+0000 7f5810e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580c0b6220 0x7f580c161410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56000/0 (socket says 172.21.15.40:56000) 2024-04-26T23:08:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.807+0000 7f5810e00640 1 -- 172.21.15.40:0/1220841999 learned_addr learned my addr 172.21.15.40:0/1220841999 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.807+0000 7f580be00640 1 --2- 172.21.15.40:0/1220841999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f580c0d08d0 0x7f580c0b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.807+0000 7f5810e00640 1 -- 172.21.15.40:0/1220841999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f580c0d39f0 msgr2=0x7f580c0b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:51.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.807+0000 7f5810e00640 1 --2- 172.21.15.40:0/1220841999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f580c0d39f0 0x7f580c0b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.807+0000 7f5810e00640 1 -- 172.21.15.40:0/1220841999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f580c0d08d0 msgr2=0x7f580c0b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:51.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.807+0000 7f5810e00640 1 --2- 172.21.15.40:0/1220841999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f580c0d08d0 0x7f580c0b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:51.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.807+0000 7f5810e00640 1 -- 172.21.15.40:0/1220841999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f580c1f4af0 con 0x7f580c0b6220 2024-04-26T23:08:51.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.807+0000 7f5810e00640 1 --2- 172.21.15.40:0/1220841999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580c0b6220 0x7f580c161410 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f57fc0097a0 tx=0x7f57fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:51.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.808+0000 7f5808c00640 1 -- 172.21.15.40:0/1220841999 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc041070 con 0x7f580c0b6220 2024-04-26T23:08:51.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.808+0000 7f5808c00640 1 -- 172.21.15.40:0/1220841999 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57fc03c870 con 0x7f580c0b6220 2024-04-26T23:08:51.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.808+0000 7f5808c00640 1 -- 172.21.15.40:0/1220841999 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc037040 con 0x7f580c0b6220 2024-04-26T23:08:51.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.808+0000 7f5812200640 1 -- 172.21.15.40:0/1220841999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f580c1f4db0 con 0x7f580c0b6220 2024-04-26T23:08:51.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.808+0000 7f5812200640 1 -- 172.21.15.40:0/1220841999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f580c1f5370 con 0x7f580c0b6220 2024-04-26T23:08:51.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.809+0000 7f5812200640 1 -- 172.21.15.40:0/1220841999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f580c069370 con 0x7f580c0b6220 2024-04-26T23:08:51.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.810+0000 7f5808c00640 1 -- 172.21.15.40:0/1220841999 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f57fc044070 con 0x7f580c0b6220 2024-04-26T23:08:51.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.811+0000 7f5808c00640 1 --2- 172.21.15.40:0/1220841999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f57ec06c920 0x7f57ec06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:51.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.811+0000 7f5808c00640 1 -- 172.21.15.40:0/1220841999 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6217+0+0 (secure 0 0 0) 0x7f57fc0aca50 con 0x7f580c0b6220 2024-04-26T23:08:51.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.812+0000 7f580be00640 1 --2- 172.21.15.40:0/1220841999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f57ec06c920 0x7f57ec06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:51.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.813+0000 7f580be00640 1 --2- 172.21.15.40:0/1220841999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f57ec06c920 0x7f57ec06ede0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f57f8007fd0 tx=0x7f57f8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:51.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:51.815+0000 7f5808c00640 1 -- 172.21.15.40:0/1220841999 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f57fc07ad00 con 0x7f580c0b6220 2024-04-26T23:08:52.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.018+0000 7f5812200640 1 -- 172.21.15.40:0/1220841999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f580c0d30c0 con 0x7f580c0b6220 2024-04-26T23:08:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.020+0000 7f5808c00640 1 -- 172.21.15.40:0/1220841999 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+17042 (secure 0 0 0) 0x7f57fc07a600 con 0x7f580c0b6220 2024-04-26T23:08:52.021 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:52.021 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":92,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:08:49.208604+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:07:46.137238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-26T23:07:47.258518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":5}]},{"pgid":"14.1b","mappings":[{"from":1,"to":2}]},{"pgid":"14.21","mappings":[{"from":1,"to":7}]},{"pgid":"14.36","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:08:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.023+0000 7f5812200640 1 -- 172.21.15.40:0/1220841999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f57ec06c920 msgr2=0x7f57ec06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.023+0000 7f5812200640 1 --2- 172.21.15.40:0/1220841999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f57ec06c920 0x7f57ec06ede0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f57f8007fd0 tx=0x7f57f8009040 comp rx=0 tx=0).stop 2024-04-26T23:08:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.023+0000 7f5812200640 1 -- 172.21.15.40:0/1220841999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580c0b6220 msgr2=0x7f580c161410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.023+0000 7f5812200640 1 --2- 172.21.15.40:0/1220841999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580c0b6220 0x7f580c161410 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f57fc0097a0 tx=0x7f57fc0046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.023+0000 7f5812200640 1 -- 172.21.15.40:0/1220841999 shutdown_connections 2024-04-26T23:08:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.023+0000 7f5812200640 1 --2- 172.21.15.40:0/1220841999 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f57ec06c920 0x7f57ec06ede0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.023+0000 7f5812200640 1 --2- 172.21.15.40:0/1220841999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f580c0d39f0 0x7f580c0b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.023+0000 7f5812200640 1 --2- 172.21.15.40:0/1220841999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f580c0d08d0 0x7f580c0b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.023+0000 7f5812200640 1 --2- 172.21.15.40:0/1220841999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f580c0b6220 0x7f580c161410 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.023+0000 7f5812200640 1 -- 172.21.15.40:0/1220841999 >> 172.21.15.40:0/1220841999 conn(0x7f580c0d4d90 msgr2=0x7f580c06e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:52.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.023+0000 7f5812200640 1 -- 172.21.15.40:0/1220841999 shutdown_connections 2024-04-26T23:08:52.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.023+0000 7f5812200640 1 -- 172.21.15.40:0/1220841999 wait complete. 2024-04-26T23:08:52.043 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-26T23:08:52.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.252+0000 7f3ac8a00640 1 Processor -- start 2024-04-26T23:08:52.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.252+0000 7f3ac8a00640 1 -- start start 2024-04-26T23:08:52.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.253+0000 7f3ac8a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ac4071e60 0x7f3ac406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:52.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.253+0000 7f3ac8a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ac415d7c0 con 0x7f3ac40cc480 2024-04-26T23:08:52.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.254+0000 7f3ac8a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ac415daa0 con 0x7f3ac406fe20 2024-04-26T23:08:52.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.254+0000 7f3ac8a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f3ac415dd80 con 0x7f3ac4071e60 2024-04-26T23:08:52.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.254+0000 7f3ac1e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ac4071e60 0x7f3ac406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:52.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.254+0000 7f3ac2800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3ac40cc480 0x7f3ac4166ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60558/0 (socket says 172.21.15.40:60558) 2024-04-26T23:08:52.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.254+0000 7f3ac2800640 1 -- 172.21.15.40:0/1903019691 learned_addr learned my addr 172.21.15.40:0/1903019691 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:52.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.254+0000 7f3ac1e00640 1 -- 172.21.15.40:0/1903019691 >> v1:172.21.15.103:6789/0 conn(0x7f3ac406fe20 legacy=0x7f3ac4072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:52.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.254+0000 7f3ac1e00640 1 -- 172.21.15.40:0/1903019691 >> v1:172.21.15.40:6789/0 conn(0x7f3ac40cc480 legacy=0x7f3ac4166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:52.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.254+0000 7f3ac0a00640 1 -- 172.21.15.40:0/1903019691 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1596801905 0 0) 0x7f3ac415d7c0 con 0x7f3ac40cc480 2024-04-26T23:08:52.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.254+0000 7f3ac1e00640 1 -- 172.21.15.40:0/1903019691 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac415e060 con 0x7f3ac4071e60 2024-04-26T23:08:52.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.255+0000 7f3ac1e00640 1 --2- 172.21.15.40:0/1903019691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ac4071e60 0x7f3ac406f8e0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4009d90 tx=0x7f3ab402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ad7f678a5a7e09 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:52.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.256+0000 7f3ac0a00640 1 -- 172.21.15.40:0/1903019691 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4007740 con 0x7f3ac4071e60 2024-04-26T23:08:52.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.256+0000 7f3ac0a00640 1 -- 172.21.15.40:0/1903019691 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ab40078e0 con 0x7f3ac4071e60 2024-04-26T23:08:52.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.256+0000 7f3ac0a00640 1 -- 172.21.15.40:0/1903019691 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4007cb0 con 0x7f3ac4071e60 2024-04-26T23:08:52.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.256+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1903019691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ac4071e60 msgr2=0x7f3ac406f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:52.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.256+0000 7f3ac8a00640 1 --2- 172.21.15.40:0/1903019691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ac4071e60 0x7f3ac406f8e0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4009d90 tx=0x7f3ab402a6a0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.256+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1903019691 shutdown_connections 2024-04-26T23:08:52.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.256+0000 7f3ac8a00640 1 --2- 172.21.15.40:0/1903019691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ac4071e60 0x7f3ac406f8e0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.256+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1903019691 >> 172.21.15.40:0/1903019691 conn(0x7f3ac40d0be0 msgr2=0x7f3ac40d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:52.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.256+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1903019691 shutdown_connections 2024-04-26T23:08:52.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.257+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1903019691 wait complete. 2024-04-26T23:08:52.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.258+0000 7f3ac8a00640 1 Processor -- start 2024-04-26T23:08:52.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.258+0000 7f3ac8a00640 1 -- start start 2024-04-26T23:08:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.259+0000 7f3ac8a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ac406fe20 0x7f3ac414e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.259+0000 7f3ac8a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac4071e60 0x7f3ac40b5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.259+0000 7f3ac8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ac40cc480 0x7f3ac40b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.259+0000 7f3ac8a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac415d570 con 0x7f3ac40cc480 2024-04-26T23:08:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.259+0000 7f3ac8a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac415d3f0 con 0x7f3ac4071e60 2024-04-26T23:08:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.259+0000 7f3ac8a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3ac4072950 con 0x7f3ac406fe20 2024-04-26T23:08:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.260+0000 7f3ac2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ac40cc480 0x7f3ac40b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.260+0000 7f3ac2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ac40cc480 0x7f3ac40b5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56016/0 (socket says 172.21.15.40:56016) 2024-04-26T23:08:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.260+0000 7f3ac2800640 1 -- 172.21.15.40:0/1014278694 learned_addr learned my addr 172.21.15.40:0/1014278694 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.260+0000 7f3ac1400640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac4071e60 0x7f3ac40b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.260+0000 7f3ac2800640 1 -- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ac406fe20 msgr2=0x7f3ac414e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.260+0000 7f3ac2800640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ac406fe20 0x7f3ac414e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.260+0000 7f3ac2800640 1 -- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac4071e60 msgr2=0x7f3ac40b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.260+0000 7f3ac2800640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac4071e60 0x7f3ac40b5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.260+0000 7f3ac2800640 1 -- 172.21.15.40:0/1014278694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac40cf780 con 0x7f3ac40cc480 2024-04-26T23:08:52.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.260+0000 7f3ac1400640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac4071e60 0x7f3ac40b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:52.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.260+0000 7f3ac2800640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ac40cc480 0x7f3ac40b5d50 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00c440 tx=0x7f3abc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:52.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.261+0000 7f3aba000640 1 -- 172.21.15.40:0/1014278694 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc0074e0 con 0x7f3ac40cc480 2024-04-26T23:08:52.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.261+0000 7f3aba000640 1 -- 172.21.15.40:0/1014278694 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3abc0047e0 con 0x7f3ac40cc480 2024-04-26T23:08:52.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.261+0000 7f3aba000640 1 -- 172.21.15.40:0/1014278694 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc004bd0 con 0x7f3ac40cc480 2024-04-26T23:08:52.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.261+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1014278694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac40cfa60 con 0x7f3ac40cc480 2024-04-26T23:08:52.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.261+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1014278694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac40d0020 con 0x7f3ac40cc480 2024-04-26T23:08:52.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.262+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1014278694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ac40694f0 con 0x7f3ac40cc480 2024-04-26T23:08:52.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.263+0000 7f3aba000640 1 -- 172.21.15.40:0/1014278694 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3abc011070 con 0x7f3ac40cc480 2024-04-26T23:08:52.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.264+0000 7f3aba000640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3aa006c8f0 0x7f3aa006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:52.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.264+0000 7f3aba000640 1 -- 172.21.15.40:0/1014278694 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6217+0+0 (secure 0 0 0) 0x7f3abc08b9e0 con 0x7f3ac40cc480 2024-04-26T23:08:52.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.265+0000 7f3ac1e00640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3aa006c8f0 0x7f3aa006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:52.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.266+0000 7f3ac1e00640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3aa006c8f0 0x7f3aa006edb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4034000 tx=0x7f3ab402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:52.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.268+0000 7f3aba000640 1 -- 172.21.15.40:0/1014278694 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3abc059b70 con 0x7f3ac40cc480 2024-04-26T23:08:52.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.505+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1014278694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3ac4073c80 con 0x7f3ac40cc480 2024-04-26T23:08:52.506 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:08:52.506+0000 7f36fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T23:08:52.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.510+0000 7f3aba000640 1 -- 172.21.15.40:0/1014278694 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v103) ==== 161+0+0 (secure 0 0 0) 0x7f3abc059470 con 0x7f3ac40cc480 2024-04-26T23:08:52.510 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T23:08:52.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.512+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3aa006c8f0 msgr2=0x7f3aa006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:52.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.512+0000 7f3ac8a00640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3aa006c8f0 0x7f3aa006edb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4034000 tx=0x7f3ab402e000 comp rx=0 tx=0).stop 2024-04-26T23:08:52.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.512+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1014278694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ac40cc480 msgr2=0x7f3ac40b5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:52.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.512+0000 7f3ac8a00640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ac40cc480 0x7f3ac40b5d50 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00c440 tx=0x7f3abc00c900 comp rx=0 tx=0).stop 2024-04-26T23:08:52.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.513+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1014278694 shutdown_connections 2024-04-26T23:08:52.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.513+0000 7f3ac8a00640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3aa006c8f0 0x7f3aa006edb0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.513+0000 7f3ac8a00640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ac40cc480 0x7f3ac40b5d50 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.513+0000 7f3ac8a00640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac4071e60 0x7f3ac40b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.513+0000 7f3ac8a00640 1 --2- 172.21.15.40:0/1014278694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ac406fe20 0x7f3ac414e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.513+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1014278694 >> 172.21.15.40:0/1014278694 conn(0x7f3ac40d0be0 msgr2=0x7f3ac40b06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:52.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.514+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1014278694 shutdown_connections 2024-04-26T23:08:52.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.514+0000 7f3ac8a00640 1 -- 172.21.15.40:0/1014278694 wait complete. 2024-04-26T23:08:52.530 INFO:tasks.ceph.mds.a.smithi040.stderr: -1596> 2024-04-26T23:08:19.633+0000 7f9394e00640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000004 [...2,head] /d1/d2/d3/d4/d5/ auth{0=1,2=1} v15 ap=1 f(v0 m2024-04-26T23:08:02.000726+0000 1=0+1) n(v2 rc2024-04-26T23:08:02.226723+0000 4=0+4) (inest lock) (ifile excl) caps={6328=pAsLsXsFsx/pAsLsXsFsxcral@4},l=6328 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 authpin=1 scrubqueue=0 0x55fe21ffec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000004:[<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:08:02.000726+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:08:02.226723+0000 4=0+4)","memory_value.dirstat":"f(v0 m2024-04-26T23:08:02.000726+0000 1=0+1)","memory_value.rstat":"n(v2 rc2024-04-26T23:08:02.226723+0000 4=0+4)","error_str":""},"return_code":-61} 2024-04-26T23:08:52.530 INFO:tasks.ceph.mds.a.smithi040.stderr: -1447> 2024-04-26T23:08:19.636+0000 7f939d000640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000005 [...2,head] /d1/d2/d3/d4/d5/d6/ auth{0=1,2=1} v11 ap=2 f(v0 m2024-04-26T23:08:02.001726+0000 1=0+1) n(v1 rc2024-04-26T23:08:02.226723+0000 3=0+3) (inest sync->mix g=0,2 scatter_wanted) caps={6328=pAsLsXsFs/pAsLsXsFsxcral@7} | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 export_pin=2 0x55fe21fff700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000005:[<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:08:02.001726+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:08:02.003726+0000 3=0+3)","memory_value.dirstat":"f(v0 m2024-04-26T23:08:02.001726+0000 1=0+1)","memory_value.rstat":"n(v1 rc2024-04-26T23:08:02.226723+0000 3=0+3)","error_str":""},"return_code":-61} 2024-04-26T23:08:52.537 INFO:tasks.ceph.mds.c.smithi040.stderr: -8002> 2024-04-26T23:08:04.755+0000 7fd9e1000640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T23:08:52.540 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T23:08:52.544 INFO:tasks.ceph.mds.c.smithi040.stderr: -3408> 2024-04-26T23:08:19.632+0000 7fd9dc000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000002 [...2,head] /d1/d2/d3/ auth{1=1,2=1} v19 ap=1 f(v0 m2024-04-26T23:08:01.996726+0000 1=0+1) n(v3 rc2024-04-26T23:08:02.226723+0000 6=0+6) (inest mix dirty) (ifile excl) caps={6328=pAsLsXsFsx/pAsLsXsFsxcral@4},l=6328 | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 0x564f1e1c6580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000002:[<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:08:01.996726+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:08:02.226723+0000 6=0+6)","memory_value.dirstat":"f(v0 m2024-04-26T23:08:01.996726+0000 1=0+1)","memory_value.rstat":"n(v3 rc2024-04-26T23:08:02.226723+0000 6=0+6)","error_str":""},"return_code":-61} 2024-04-26T23:08:52.544 INFO:tasks.ceph.mds.c.smithi040.stderr: -3280> 2024-04-26T23:08:19.634+0000 7fd9e4200640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000003 [...2,head] /d1/d2/d3/d4/ auth{1=1,2=1} v19 ap=2 f(v0 m2024-04-26T23:08:01.998726+0000 1=0+1) n(v4 rc2024-04-26T23:08:02.226723+0000 5=0+5) (inest sync->mix g=1,2) caps={6328=pAsLsXsFs/pAsLsXsFsxcral@8} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 export_pin=1 0x564f1e1c8680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000003:[<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:08:01.998726+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:08:02.226723+0000 5=0+5)","memory_value.dirstat":"f(v0 m2024-04-26T23:08:01.998726+0000 1=0+1)","memory_value.rstat":"n(v4 rc2024-04-26T23:08:02.226723+0000 5=0+5)","error_str":""},"return_code":-61} 2024-04-26T23:08:52.554 INFO:tasks.ceph.mds.d.smithi103.stderr: -1496> 2024-04-26T23:08:19.634+0000 7f8047e00640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000006 [...2,head] /d1/d2/d3/d4/d5/d6/d7/ auth v4 ap=1 f(v0 m2024-04-26T23:08:02.003726+0000 1=0+1) n(v0 rc2024-04-26T23:08:02.003726+0000 2=0+2) (inest lock) (ifile excl) caps={6328=pAsLsXsFsx/pAsLsXsFsxcral@4},l=6328 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x560ec42a2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000006:[<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:08:02.003726+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:08:02.003726+0000 2=0+2)","memory_value.dirstat":"f(v0 m2024-04-26T23:08:02.003726+0000 1=0+1)","memory_value.rstat":"n(v0 rc2024-04-26T23:08:02.003726+0000 2=0+2)","error_str":""},"return_code":-61} 2024-04-26T23:08:52.554 INFO:tasks.ceph.mds.d.smithi103.stderr: -1475> 2024-04-26T23:08:19.635+0000 7f8047e00640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000007 [...2,head] /d1/d2/d3/d4/d5/d6/d7/d8/ auth v2 ap=1 f() n(v0 1=0+1) (inest lock) (ifile excl) caps={6328=pAsLsXsFsx/AsLsXsFsx@2},l=6328 | importingcaps=0 dirfrag=0 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x560ec42a2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000007:[<0x10000000006/d8 v2>,<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 1=0+1)","error_str":""},"return_code":-61} 2024-04-26T23:08:52.588 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-04-26T23:08:52.608 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:08:52.614 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-26T23:08:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.724+0000 7f1ea6400640 1 Processor -- start 2024-04-26T23:08:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.725+0000 7f1ea6400640 1 -- start start 2024-04-26T23:08:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.726+0000 7f1ea6400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1ea015f3a0 0x7f1ea00d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.726+0000 7f1ea6400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1ea00d2330 0x7f1ea00b2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.726+0000 7f1ea6400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ea00d0e20 con 0x7f1ea00d1770 2024-04-26T23:08:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.726+0000 7f1ea6400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1ea00b1d70 con 0x7f1ea015f3a0 2024-04-26T23:08:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.726+0000 7f1ea6400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1ea00b2050 con 0x7f1ea00d2330 2024-04-26T23:08:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.726+0000 7f1e9f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1ea015f3a0 0x7f1ea00d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.726+0000 7f1e9ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1ea00d1770 0x7f1ea00d1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60576/0 (socket says 172.21.15.40:60576) 2024-04-26T23:08:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.726+0000 7f1e9ea00640 1 -- 172.21.15.40:0/417991149 learned_addr learned my addr 172.21.15.40:0/417991149 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.727+0000 7f1e9f400640 1 -- 172.21.15.40:0/417991149 >> v2:172.21.15.103:3300/0 conn(0x7f1ea00d2330 msgr2=0x7f1ea00b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.727+0000 7f1e9fe00640 1 --2- 172.21.15.40:0/417991149 >> v2:172.21.15.103:3300/0 conn(0x7f1ea00d2330 0x7f1ea00b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.727+0000 7f1e9f400640 1 --2- 172.21.15.40:0/417991149 >> v2:172.21.15.103:3300/0 conn(0x7f1ea00d2330 0x7f1ea00b2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.727+0000 7f1e9f400640 1 -- 172.21.15.40:0/417991149 >> v1:172.21.15.40:6789/0 conn(0x7f1ea00d1770 legacy=0x7f1ea00d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:52.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.727+0000 7f1e9e000640 1 -- 172.21.15.40:0/417991149 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3257062847 0 0) 0x7f1ea00d0e20 con 0x7f1ea00d1770 2024-04-26T23:08:52.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.727+0000 7f1e9f400640 1 -- 172.21.15.40:0/417991149 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea00b2330 con 0x7f1ea015f3a0 2024-04-26T23:08:52.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.727+0000 7f1e9fe00640 1 --2- 172.21.15.40:0/417991149 >> v2:172.21.15.103:3300/0 conn(0x7f1ea00d2330 0x7f1ea00b2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:52.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.728+0000 7f1e9f400640 1 --2- 172.21.15.40:0/417991149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea015f3a0 0x7f1ea00d06f0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f1ea00c89f0 tx=0x7f1e9002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69ce1e7f0f6c3c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:52.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.728+0000 7f1e9e000640 1 -- 172.21.15.40:0/417991149 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e90037070 con 0x7f1ea015f3a0 2024-04-26T23:08:52.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.728+0000 7f1e9e000640 1 -- 172.21.15.40:0/417991149 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e90032440 con 0x7f1ea015f3a0 2024-04-26T23:08:52.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.728+0000 7f1e9e000640 1 -- 172.21.15.40:0/417991149 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e90032810 con 0x7f1ea015f3a0 2024-04-26T23:08:52.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.728+0000 7f1ea6400640 1 -- 172.21.15.40:0/417991149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea015f3a0 msgr2=0x7f1ea00d06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:52.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.728+0000 7f1ea6400640 1 --2- 172.21.15.40:0/417991149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea015f3a0 0x7f1ea00d06f0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f1ea00c89f0 tx=0x7f1e9002a1a0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.729+0000 7f1ea6400640 1 -- 172.21.15.40:0/417991149 shutdown_connections 2024-04-26T23:08:52.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.729+0000 7f1ea6400640 1 --2- 172.21.15.40:0/417991149 >> v2:172.21.15.103:3300/0 conn(0x7f1ea00d2330 0x7f1ea00b2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.729+0000 7f1ea6400640 1 --2- 172.21.15.40:0/417991149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea015f3a0 0x7f1ea00d06f0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.729+0000 7f1ea6400640 1 -- 172.21.15.40:0/417991149 >> 172.21.15.40:0/417991149 conn(0x7f1ea00d4f90 msgr2=0x7f1ea00d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:52.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.729+0000 7f1ea6400640 1 -- 172.21.15.40:0/417991149 shutdown_connections 2024-04-26T23:08:52.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.729+0000 7f1ea6400640 1 -- 172.21.15.40:0/417991149 wait complete. 2024-04-26T23:08:52.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.730+0000 7f1ea6400640 1 Processor -- start 2024-04-26T23:08:52.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.731+0000 7f1ea6400640 1 -- start start 2024-04-26T23:08:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.731+0000 7f1ea6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea00d1770 0x7f1ea01f8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1ea6400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea00d2330 0x7f1ea01f4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1ea6400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea015f3a0 0x7f1ea015cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1ea6400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea00b2f10 con 0x7f1ea00d1770 2024-04-26T23:08:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1ea6400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea00b2d90 con 0x7f1ea00d2330 2024-04-26T23:08:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1ea6400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea00b3090 con 0x7f1ea015f3a0 2024-04-26T23:08:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1e9f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea00d1770 0x7f1ea01f8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1e9f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea00d1770 0x7f1ea01f8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56074/0 (socket says 172.21.15.40:56074) 2024-04-26T23:08:52.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1e9f400640 1 -- 172.21.15.40:0/2130192013 learned_addr learned my addr 172.21.15.40:0/2130192013 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:52.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1e9ea00640 1 --2- 172.21.15.40:0/2130192013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea00d2330 0x7f1ea01f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:52.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1e9f400640 1 -- 172.21.15.40:0/2130192013 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea015f3a0 msgr2=0x7f1ea015cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:52.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1e9f400640 1 --2- 172.21.15.40:0/2130192013 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea015f3a0 0x7f1ea015cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1e9f400640 1 -- 172.21.15.40:0/2130192013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea00d2330 msgr2=0x7f1ea01f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:52.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1e9f400640 1 --2- 172.21.15.40:0/2130192013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea00d2330 0x7f1ea01f4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:52.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.732+0000 7f1e9f400640 1 -- 172.21.15.40:0/2130192013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea015f990 con 0x7f1ea00d1770 2024-04-26T23:08:52.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.733+0000 7f1e9f400640 1 --2- 172.21.15.40:0/2130192013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea00d1770 0x7f1ea01f8b10 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f1e90000c00 tx=0x7f1e900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:52.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.733+0000 7f1e97400640 1 -- 172.21.15.40:0/2130192013 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e90037050 con 0x7f1ea00d1770 2024-04-26T23:08:52.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.733+0000 7f1e97400640 1 -- 172.21.15.40:0/2130192013 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e90038040 con 0x7f1ea00d1770 2024-04-26T23:08:52.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.733+0000 7f1ea6400640 1 -- 172.21.15.40:0/2130192013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea015fc30 con 0x7f1ea00d1770 2024-04-26T23:08:52.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.733+0000 7f1e97400640 1 -- 172.21.15.40:0/2130192013 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e90004450 con 0x7f1ea00d1770 2024-04-26T23:08:52.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.733+0000 7f1ea6400640 1 -- 172.21.15.40:0/2130192013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea01601f0 con 0x7f1ea00d1770 2024-04-26T23:08:52.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.734+0000 7f1ea6400640 1 -- 172.21.15.40:0/2130192013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ea00694f0 con 0x7f1ea00d1770 2024-04-26T23:08:52.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.735+0000 7f1e97400640 1 -- 172.21.15.40:0/2130192013 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1e90030980 con 0x7f1ea00d1770 2024-04-26T23:08:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.736+0000 7f1e97400640 1 --2- 172.21.15.40:0/2130192013 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7c06c8f0 0x7f1e7c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.736+0000 7f1e97400640 1 -- 172.21.15.40:0/2130192013 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6475+0+0 (secure 0 0 0) 0x7f1e900acb40 con 0x7f1ea00d1770 2024-04-26T23:08:52.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.737+0000 7f1e9ea00640 1 --2- 172.21.15.40:0/2130192013 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7c06c8f0 0x7f1e7c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:52.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.738+0000 7f1e9ea00640 1 --2- 172.21.15.40:0/2130192013 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7c06c8f0 0x7f1e7c06edb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1ea01f8060 tx=0x7f1e88009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:52.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.741+0000 7f1e97400640 1 -- 172.21.15.40:0/2130192013 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1e9007abd0 con 0x7f1ea00d1770 2024-04-26T23:08:52.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:52.993+0000 7f1ea6400640 1 -- 172.21.15.40:0/2130192013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1ea00b6860 con 0x7f1ea00d1770 2024-04-26T23:08:53.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.513+0000 7f1e97400640 1 -- 172.21.15.40:0/2130192013 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v104) ==== 104+0+0 (secure 0 0 0) 0x7f1e9007a4d0 con 0x7f1ea00d1770 2024-04-26T23:08:53.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.517+0000 7f1ea6400640 1 -- 172.21.15.40:0/2130192013 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7c06c8f0 msgr2=0x7f1e7c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:53.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.517+0000 7f1ea6400640 1 --2- 172.21.15.40:0/2130192013 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7c06c8f0 0x7f1e7c06edb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1ea01f8060 tx=0x7f1e88009040 comp rx=0 tx=0).stop 2024-04-26T23:08:53.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.517+0000 7f1ea6400640 1 -- 172.21.15.40:0/2130192013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea00d1770 msgr2=0x7f1ea01f8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:53.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.517+0000 7f1ea6400640 1 --2- 172.21.15.40:0/2130192013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea00d1770 0x7f1ea01f8b10 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f1e90000c00 tx=0x7f1e900046c0 comp rx=0 tx=0).stop 2024-04-26T23:08:53.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.517+0000 7f1ea6400640 1 -- 172.21.15.40:0/2130192013 shutdown_connections 2024-04-26T23:08:53.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.517+0000 7f1ea6400640 1 --2- 172.21.15.40:0/2130192013 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7c06c8f0 0x7f1e7c06edb0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:53.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.517+0000 7f1ea6400640 1 --2- 172.21.15.40:0/2130192013 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1ea015f3a0 0x7f1ea015cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:53.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.517+0000 7f1ea6400640 1 --2- 172.21.15.40:0/2130192013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ea00d2330 0x7f1ea01f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:53.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.517+0000 7f1ea6400640 1 --2- 172.21.15.40:0/2130192013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea00d1770 0x7f1ea01f8b10 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:53.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.517+0000 7f1ea6400640 1 -- 172.21.15.40:0/2130192013 >> 172.21.15.40:0/2130192013 conn(0x7f1ea00d4f90 msgr2=0x7f1ea00d0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:53.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.517+0000 7f1ea6400640 1 -- 172.21.15.40:0/2130192013 shutdown_connections 2024-04-26T23:08:53.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.517+0000 7f1ea6400640 1 -- 172.21.15.40:0/2130192013 wait complete. 2024-04-26T23:08:53.542 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T23:08:53.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.809+0000 7f7178a00640 1 Processor -- start 2024-04-26T23:08:53.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.809+0000 7f7178a00640 1 -- start start 2024-04-26T23:08:53.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.810+0000 7f7178a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f717415f0f0 0x7f71740d55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:53.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.810+0000 7f7178a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71740d21e0 0x7f71740c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:53.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.811+0000 7f7178a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71740d0580 con 0x7f71740d1690 2024-04-26T23:08:53.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.811+0000 7f7178a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f71740d0e20 con 0x7f717415f0f0 2024-04-26T23:08:53.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.811+0000 7f7178a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f71740d0860 con 0x7f71740d21e0 2024-04-26T23:08:53.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.811+0000 7f7173400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f717415f0f0 0x7f71740d55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:53.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.811+0000 7f7172a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f71740d1690 0x7f71740d1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60588/0 (socket says 172.21.15.40:60588) 2024-04-26T23:08:53.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.811+0000 7f7172a00640 1 -- 172.21.15.40:0/526196132 learned_addr learned my addr 172.21.15.40:0/526196132 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:53.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.811+0000 7f7173e00640 1 --2- 172.21.15.40:0/526196132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71740d21e0 0x7f71740c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:53.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.811+0000 7f7173400640 1 -- 172.21.15.40:0/526196132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71740d21e0 msgr2=0x7f71740c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:53.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.811+0000 7f7173400640 1 --2- 172.21.15.40:0/526196132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71740d21e0 0x7f71740c8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:53.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.811+0000 7f7173400640 1 -- 172.21.15.40:0/526196132 >> v1:172.21.15.40:6789/0 conn(0x7f71740d1690 legacy=0x7f71740d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:53.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.811+0000 7f7172000640 1 -- 172.21.15.40:0/526196132 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 80585142 0 0) 0x7f71740d0580 con 0x7f71740d1690 2024-04-26T23:08:53.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.811+0000 7f7173400640 1 -- 172.21.15.40:0/526196132 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71740d0b40 con 0x7f717415f0f0 2024-04-26T23:08:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.813+0000 7f7173400640 1 --2- 172.21.15.40:0/526196132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f717415f0f0 0x7f71740d55f0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f7164009d90 tx=0x7f716402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75573a298e24d4e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.813+0000 7f7172000640 1 -- 172.21.15.40:0/526196132 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7164007740 con 0x7f717415f0f0 2024-04-26T23:08:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.813+0000 7f7172000640 1 -- 172.21.15.40:0/526196132 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71640078e0 con 0x7f717415f0f0 2024-04-26T23:08:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.813+0000 7f7172000640 1 -- 172.21.15.40:0/526196132 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7164007cb0 con 0x7f717415f0f0 2024-04-26T23:08:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.813+0000 7f7178a00640 1 -- 172.21.15.40:0/526196132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f717415f0f0 msgr2=0x7f71740d55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.813+0000 7f7178a00640 1 --2- 172.21.15.40:0/526196132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f717415f0f0 0x7f71740d55f0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f7164009d90 tx=0x7f716402a6f0 comp rx=0 tx=0).stop 2024-04-26T23:08:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.813+0000 7f7178a00640 1 -- 172.21.15.40:0/526196132 shutdown_connections 2024-04-26T23:08:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.814+0000 7f7178a00640 1 --2- 172.21.15.40:0/526196132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f71740d21e0 0x7f71740c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.814+0000 7f7178a00640 1 --2- 172.21.15.40:0/526196132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f717415f0f0 0x7f71740d55f0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.814+0000 7f7178a00640 1 -- 172.21.15.40:0/526196132 >> 172.21.15.40:0/526196132 conn(0x7f717409e1b0 msgr2=0x7f71740d4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.814+0000 7f7178a00640 1 -- 172.21.15.40:0/526196132 shutdown_connections 2024-04-26T23:08:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.814+0000 7f7178a00640 1 -- 172.21.15.40:0/526196132 wait complete. 2024-04-26T23:08:53.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.815+0000 7f7178a00640 1 Processor -- start 2024-04-26T23:08:53.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.815+0000 7f7178a00640 1 -- start start 2024-04-26T23:08:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.816+0000 7f7178a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71740d1690 0x7f7174153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.816+0000 7f7178a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71740d21e0 0x7f717415a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.816+0000 7f7178a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f717415f0f0 0x7f717415a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.816+0000 7f7178a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f71740c9280 con 0x7f71740d21e0 2024-04-26T23:08:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.816+0000 7f7178a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f71740d5d40 con 0x7f71740d1690 2024-04-26T23:08:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.816+0000 7f7178a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f71740c9400 con 0x7f717415f0f0 2024-04-26T23:08:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.817+0000 7f7172a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71740d21e0 0x7f717415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.817+0000 7f7172a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71740d21e0 0x7f717415a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56090/0 (socket says 172.21.15.40:56090) 2024-04-26T23:08:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.817+0000 7f7172a00640 1 -- 172.21.15.40:0/3271821057 learned_addr learned my addr 172.21.15.40:0/3271821057 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:53.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.817+0000 7f7173e00640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f717415f0f0 0x7f717415a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:53.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.817+0000 7f7173400640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71740d1690 0x7f7174153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.817+0000 7f7172a00640 1 -- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f717415f0f0 msgr2=0x7f717415a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.817+0000 7f7172a00640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f717415f0f0 0x7f717415a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.817+0000 7f7172a00640 1 -- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71740d1690 msgr2=0x7f7174153f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.817+0000 7f7172a00640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71740d1690 0x7f7174153f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.817+0000 7f7172a00640 1 -- 172.21.15.40:0/3271821057 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7174158800 con 0x7f71740d21e0 2024-04-26T23:08:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.817+0000 7f7173e00640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f717415f0f0 0x7f717415a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.817+0000 7f7172a00640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71740d21e0 0x7f717415a350 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f715c00c4d0 tx=0x7f715c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.818+0000 7f716b400640 1 -- 172.21.15.40:0/3271821057 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c018070 con 0x7f71740d21e0 2024-04-26T23:08:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.818+0000 7f716b400640 1 -- 172.21.15.40:0/3271821057 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f715c007500 con 0x7f71740d21e0 2024-04-26T23:08:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.818+0000 7f716b400640 1 -- 172.21.15.40:0/3271821057 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c004a20 con 0x7f71740d21e0 2024-04-26T23:08:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.818+0000 7f7178a00640 1 -- 172.21.15.40:0/3271821057 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7174158ae0 con 0x7f71740d21e0 2024-04-26T23:08:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.818+0000 7f7178a00640 1 -- 172.21.15.40:0/3271821057 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71741590a0 con 0x7f71740d21e0 2024-04-26T23:08:53.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.819+0000 7f7178a00640 1 -- 172.21.15.40:0/3271821057 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7174069370 con 0x7f71740d21e0 2024-04-26T23:08:53.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.820+0000 7f716b400640 1 -- 172.21.15.40:0/3271821057 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f715c004bc0 con 0x7f71740d21e0 2024-04-26T23:08:53.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.822+0000 7f716b400640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f715006c920 0x7f715006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:53.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.822+0000 7f716b400640 1 -- 172.21.15.40:0/3271821057 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6475+0+0 (secure 0 0 0) 0x7f715c08b100 con 0x7f71740d21e0 2024-04-26T23:08:53.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.822+0000 7f7173400640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f715006c920 0x7f715006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:53.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.824+0000 7f7173400640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f715006c920 0x7f715006ede0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7164034000 tx=0x7f7164033040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:53.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:53.826+0000 7f716b400640 1 -- 172.21.15.40:0/3271821057 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f715c059220 con 0x7f71740d21e0 2024-04-26T23:08:54.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.028+0000 7f7178a00640 1 -- 172.21.15.40:0/3271821057 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f71740d0430 con 0x7f71740d21e0 2024-04-26T23:08:54.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.517+0000 7f716b400640 1 -- 172.21.15.40:0/3271821057 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v94) ==== 181+0+0 (secure 0 0 0) 0x7f715c058b20 con 0x7f71740d21e0 2024-04-26T23:08:54.518 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-04-26T23:08:54.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.521+0000 7f7178a00640 1 -- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f715006c920 msgr2=0x7f715006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:54.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.521+0000 7f7178a00640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f715006c920 0x7f715006ede0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7164034000 tx=0x7f7164033040 comp rx=0 tx=0).stop 2024-04-26T23:08:54.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.521+0000 7f7178a00640 1 -- 172.21.15.40:0/3271821057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71740d21e0 msgr2=0x7f717415a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:54.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.521+0000 7f7178a00640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71740d21e0 0x7f717415a350 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f715c00c4d0 tx=0x7f715c00c990 comp rx=0 tx=0).stop 2024-04-26T23:08:54.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.521+0000 7f7178a00640 1 -- 172.21.15.40:0/3271821057 shutdown_connections 2024-04-26T23:08:54.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.521+0000 7f7178a00640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f715006c920 0x7f715006ede0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:54.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.521+0000 7f7178a00640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f717415f0f0 0x7f717415a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:54.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.521+0000 7f7178a00640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71740d21e0 0x7f717415a350 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:54.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.521+0000 7f7178a00640 1 --2- 172.21.15.40:0/3271821057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71740d1690 0x7f7174153f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:54.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.521+0000 7f7178a00640 1 -- 172.21.15.40:0/3271821057 >> 172.21.15.40:0/3271821057 conn(0x7f717409e1b0 msgr2=0x7f71740d4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:54.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.521+0000 7f7178a00640 1 -- 172.21.15.40:0/3271821057 shutdown_connections 2024-04-26T23:08:54.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.522+0000 7f7178a00640 1 -- 172.21.15.40:0/3271821057 wait complete. 2024-04-26T23:08:54.549 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T23:08:54.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.765+0000 7f41b8000640 1 Processor -- start 2024-04-26T23:08:54.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.765+0000 7f41b8000640 1 -- start start 2024-04-26T23:08:54.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.768+0000 7f41b8000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f41b014d090 0x7f41b0164620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:54.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.768+0000 7f41b8000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f41b00b5320 0x7f41b00b5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:54.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.768+0000 7f41b8000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41b0164b60 con 0x7f41b00b3df0 2024-04-26T23:08:54.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.768+0000 7f41b8000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f41b00b5c60 con 0x7f41b00b5320 2024-04-26T23:08:54.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.768+0000 7f41b8000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f41b00b4280 con 0x7f41b014d090 2024-04-26T23:08:54.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.768+0000 7f41b6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f41b00b5320 0x7f41b00b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:54.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.768+0000 7f41b6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f41b00b5320 0x7f41b00b5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56100/0 (socket says 172.21.15.40:56100) 2024-04-26T23:08:54.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.768+0000 7f41b6000640 1 -- 172.21.15.40:0/4260232376 learned_addr learned my addr 172.21.15.40:0/4260232376 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:54.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.768+0000 7f41b5600640 1 --2- 172.21.15.40:0/4260232376 >> v2:172.21.15.103:3300/0 conn(0x7f41b014d090 0x7f41b0164620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:54.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.768+0000 7f41b6000640 1 -- 172.21.15.40:0/4260232376 >> v2:172.21.15.103:3300/0 conn(0x7f41b014d090 msgr2=0x7f41b0164620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:54.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.768+0000 7f41b6000640 1 --2- 172.21.15.40:0/4260232376 >> v2:172.21.15.103:3300/0 conn(0x7f41b014d090 0x7f41b0164620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:54.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.768+0000 7f41b6000640 1 -- 172.21.15.40:0/4260232376 >> v1:172.21.15.103:6789/0 conn(0x7f41b00b3df0 legacy=0x7f41b0164d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:54.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.768+0000 7f41b6000640 1 -- 172.21.15.40:0/4260232376 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b00b4560 con 0x7f41b00b5320 2024-04-26T23:08:54.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.769+0000 7f41b6000640 1 --2- 172.21.15.40:0/4260232376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41b00b5320 0x7f41b00b5720 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f41b00726c0 tx=0x7f41a802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70bb8125094aff93 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:54.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.769+0000 7f41afe00640 1 -- 172.21.15.40:0/4260232376 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8036070 con 0x7f41b00b5320 2024-04-26T23:08:54.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.769+0000 7f41afe00640 1 -- 172.21.15.40:0/4260232376 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a8031600 con 0x7f41b00b5320 2024-04-26T23:08:54.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.769+0000 7f41afe00640 1 -- 172.21.15.40:0/4260232376 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a80319d0 con 0x7f41b00b5320 2024-04-26T23:08:54.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.770+0000 7f41b8000640 1 -- 172.21.15.40:0/4260232376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41b00b5320 msgr2=0x7f41b00b5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:54.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.770+0000 7f41b8000640 1 --2- 172.21.15.40:0/4260232376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41b00b5320 0x7f41b00b5720 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f41b00726c0 tx=0x7f41a802a6f0 comp rx=0 tx=0).stop 2024-04-26T23:08:54.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.770+0000 7f41b8000640 1 -- 172.21.15.40:0/4260232376 shutdown_connections 2024-04-26T23:08:54.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.770+0000 7f41b8000640 1 --2- 172.21.15.40:0/4260232376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41b00b5320 0x7f41b00b5720 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:54.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.770+0000 7f41b8000640 1 --2- 172.21.15.40:0/4260232376 >> v2:172.21.15.103:3300/0 conn(0x7f41b014d090 0x7f41b0164620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:54.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.770+0000 7f41b8000640 1 -- 172.21.15.40:0/4260232376 >> 172.21.15.40:0/4260232376 conn(0x7f41b006f310 msgr2=0x7f41b006f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:54.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.770+0000 7f41b8000640 1 -- 172.21.15.40:0/4260232376 shutdown_connections 2024-04-26T23:08:54.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.770+0000 7f41b8000640 1 -- 172.21.15.40:0/4260232376 wait complete. 2024-04-26T23:08:54.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.772+0000 7f41b8000640 1 Processor -- start 2024-04-26T23:08:54.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.772+0000 7f41b8000640 1 -- start start 2024-04-26T23:08:54.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.773+0000 7f41b8000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41b00b5320 0x7f41b0163140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:54.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.773+0000 7f41b8000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41b014d090 0x7f41b014a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:54.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.773+0000 7f41b8000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41b00afe40 0x7f41b0145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:54.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.773+0000 7f41b8000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f41b00737a0 con 0x7f41b00b5320 2024-04-26T23:08:54.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.773+0000 7f41b8000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f41b00b4b20 con 0x7f41b014d090 2024-04-26T23:08:54.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.773+0000 7f41b8000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f41b0165480 con 0x7f41b00afe40 2024-04-26T23:08:54.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.773+0000 7f41b5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41b00b5320 0x7f41b0163140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:54.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.773+0000 7f41b5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41b00b5320 0x7f41b0163140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56116/0 (socket says 172.21.15.40:56116) 2024-04-26T23:08:54.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.773+0000 7f41b5600640 1 -- 172.21.15.40:0/1678330199 learned_addr learned my addr 172.21.15.40:0/1678330199 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:54.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.774+0000 7f41b5600640 1 -- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41b00afe40 msgr2=0x7f41b0145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:54.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.774+0000 7f41b5600640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41b00afe40 0x7f41b0145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:54.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.774+0000 7f41b5600640 1 -- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41b014d090 msgr2=0x7f41b014a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:54.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.774+0000 7f41b4c00640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41b014d090 0x7f41b014a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:54.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.774+0000 7f41b5600640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41b014d090 0x7f41b014a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:54.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.774+0000 7f41b5600640 1 -- 172.21.15.40:0/1678330199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b00b03b0 con 0x7f41b00b5320 2024-04-26T23:08:54.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.774+0000 7f41b4c00640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41b014d090 0x7f41b014a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:54.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.774+0000 7f41b5600640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41b00b5320 0x7f41b0163140 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f41a400c7f0 tx=0x7f41a400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:54.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.774+0000 7f41ad600640 1 -- 172.21.15.40:0/1678330199 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4015070 con 0x7f41b00b5320 2024-04-26T23:08:54.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.774+0000 7f41ad600640 1 -- 172.21.15.40:0/1678330199 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a4004240 con 0x7f41b00b5320 2024-04-26T23:08:54.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.774+0000 7f41ad600640 1 -- 172.21.15.40:0/1678330199 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4007620 con 0x7f41b00b5320 2024-04-26T23:08:54.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.774+0000 7f41b8000640 1 -- 172.21.15.40:0/1678330199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b0144830 con 0x7f41b00b5320 2024-04-26T23:08:54.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.775+0000 7f41b8000640 1 -- 172.21.15.40:0/1678330199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b0144df0 con 0x7f41b00b5320 2024-04-26T23:08:54.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.777+0000 7f41b8000640 1 -- 172.21.15.40:0/1678330199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b0069600 con 0x7f41b00b5320 2024-04-26T23:08:54.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.778+0000 7f41ad600640 1 -- 172.21.15.40:0/1678330199 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f41a40079d0 con 0x7f41b00b5320 2024-04-26T23:08:54.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.778+0000 7f41ad600640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f419006c8f0 0x7f419006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:54.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.778+0000 7f41ad600640 1 -- 172.21.15.40:0/1678330199 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6037+0+0 (secure 0 0 0) 0x7f41a408bd60 con 0x7f41b00b5320 2024-04-26T23:08:54.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.779+0000 7f41b4c00640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f419006c8f0 0x7f419006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:54.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.780+0000 7f41b4c00640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f419006c8f0 0x7f419006edb0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f41b014bd10 tx=0x7f419c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:54.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.782+0000 7f41ad600640 1 -- 172.21.15.40:0/1678330199 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f41a405a040 con 0x7f41b00b5320 2024-04-26T23:08:54.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:54.997+0000 7f41b8000640 1 -- 172.21.15.40:0/1678330199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f41b00b3540 con 0x7f41b00b5320 2024-04-26T23:08:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.522+0000 7f41ad600640 1 -- 172.21.15.40:0/1678330199 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v95) ==== 169+0+0 (secure 0 0 0) 0x7f41a4059940 con 0x7f41b00b5320 2024-04-26T23:08:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-04-26T23:08:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.525+0000 7f41b8000640 1 -- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f419006c8f0 msgr2=0x7f419006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.525+0000 7f41b8000640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f419006c8f0 0x7f419006edb0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f41b014bd10 tx=0x7f419c008040 comp rx=0 tx=0).stop 2024-04-26T23:08:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.525+0000 7f41b8000640 1 -- 172.21.15.40:0/1678330199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41b00b5320 msgr2=0x7f41b0163140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.525+0000 7f41b8000640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41b00b5320 0x7f41b0163140 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f41a400c7f0 tx=0x7f41a400ccb0 comp rx=0 tx=0).stop 2024-04-26T23:08:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.526+0000 7f41b8000640 1 -- 172.21.15.40:0/1678330199 shutdown_connections 2024-04-26T23:08:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.526+0000 7f41b8000640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f419006c8f0 0x7f419006edb0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.526+0000 7f41b8000640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41b00afe40 0x7f41b0145bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.526+0000 7f41b8000640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41b014d090 0x7f41b014a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.526+0000 7f41b8000640 1 --2- 172.21.15.40:0/1678330199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41b00b5320 0x7f41b0163140 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.526+0000 7f41b8000640 1 -- 172.21.15.40:0/1678330199 >> 172.21.15.40:0/1678330199 conn(0x7f41b006f310 msgr2=0x7f41b00cd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.526+0000 7f41b8000640 1 -- 172.21.15.40:0/1678330199 shutdown_connections 2024-04-26T23:08:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.526+0000 7f41b8000640 1 -- 172.21.15.40:0/1678330199 wait complete. 2024-04-26T23:08:55.550 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-26T23:08:55.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.759+0000 7f4135200640 1 Processor -- start 2024-04-26T23:08:55.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.759+0000 7f4135200640 1 -- start start 2024-04-26T23:08:55.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.760+0000 7f4135200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f413006f300 0x7f413015f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:55.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.760+0000 7f4135200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41300d19f0 con 0x7f41301486b0 2024-04-26T23:08:55.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.760+0000 7f4135200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f413015ff80 con 0x7f413015f4c0 2024-04-26T23:08:55.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.760+0000 7f4135200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4130160260 con 0x7f413006f300 2024-04-26T23:08:55.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.760+0000 7f412f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f41301486b0 0x7f41300d21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45106/0 (socket says 172.21.15.40:45106) 2024-04-26T23:08:55.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.760+0000 7f412f400640 1 -- 172.21.15.40:0/3063133922 learned_addr learned my addr 172.21.15.40:0/3063133922 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:55.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.761+0000 7f412fe00640 1 --2- 172.21.15.40:0/3063133922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f413006f300 0x7f413015f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:55.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.761+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588951923 0 0) 0x7f41300d19f0 con 0x7f41301486b0 2024-04-26T23:08:55.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.761+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4114003520 con 0x7f41301486b0 2024-04-26T23:08:55.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.761+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3803200574 0 0) 0x7f4114003520 con 0x7f41301486b0 2024-04-26T23:08:55.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f413006f300 msgr2=0x7f413015f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:55.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412e000640 1 --2- 172.21.15.40:0/3063133922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f413006f300 0x7f413015f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:55.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 >> v1:172.21.15.103:6789/0 conn(0x7f413015f4c0 legacy=0x7f413006ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:55.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41300d1cb0 con 0x7f41301486b0 2024-04-26T23:08:55.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f41200019f0 con 0x7f41301486b0 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 >> v1:172.21.15.40:6789/0 conn(0x7f41301486b0 legacy=0x7f41300d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412e000640 1 --2- 172.21.15.40:0/3063133922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41140039e0 0x7f4114003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412e000640 1 --2- 172.21.15.40:0/3063133922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4114004800 0x7f4114004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412e000640 1 --2- 172.21.15.40:0/3063133922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41140051c0 0x7f411400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4130160260 con 0x7f41140039e0 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f411400bf00 con 0x7f4114004800 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f411400c080 con 0x7f41140051c0 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412f400640 1 --2- 172.21.15.40:0/3063133922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41140039e0 0x7f4114003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412ea00640 1 --2- 172.21.15.40:0/3063133922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4114004800 0x7f4114004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412f400640 1 -- 172.21.15.40:0/3063133922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41140051c0 msgr2=0x7f411400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412f400640 1 --2- 172.21.15.40:0/3063133922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41140051c0 0x7f411400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412f400640 1 -- 172.21.15.40:0/3063133922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4114004800 msgr2=0x7f4114004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412f400640 1 --2- 172.21.15.40:0/3063133922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4114004800 0x7f4114004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:55.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.762+0000 7f412f400640 1 -- 172.21.15.40:0/3063133922 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4114003520 con 0x7f41140039e0 2024-04-26T23:08:55.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.763+0000 7f412f400640 1 --2- 172.21.15.40:0/3063133922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41140039e0 0x7f4114003e00 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f412000a840 tx=0x7f4114005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9589520efeb1ca44 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:55.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.764+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412000a460 con 0x7f41140039e0 2024-04-26T23:08:55.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.764+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f412002f070 con 0x7f41140039e0 2024-04-26T23:08:55.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.764+0000 7f412e000640 1 -- 172.21.15.40:0/3063133922 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412002bcb0 con 0x7f41140039e0 2024-04-26T23:08:55.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.764+0000 7f4135200640 1 -- 172.21.15.40:0/3063133922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41140039e0 msgr2=0x7f4114003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:55.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.764+0000 7f4135200640 1 --2- 172.21.15.40:0/3063133922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41140039e0 0x7f4114003e00 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f412000a840 tx=0x7f4114005830 comp rx=0 tx=0).stop 2024-04-26T23:08:55.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.764+0000 7f412fe00640 1 -- 172.21.15.40:0/3063133922 reap_dead start 2024-04-26T23:08:55.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.764+0000 7f4135200640 1 -- 172.21.15.40:0/3063133922 shutdown_connections 2024-04-26T23:08:55.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.764+0000 7f4135200640 1 -- 172.21.15.40:0/3063133922 >> 172.21.15.40:0/3063133922 conn(0x7f4130068090 msgr2=0x7f4130068470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:55.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.765+0000 7f4135200640 1 -- 172.21.15.40:0/3063133922 shutdown_connections 2024-04-26T23:08:55.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.765+0000 7f4135200640 1 -- 172.21.15.40:0/3063133922 wait complete. 2024-04-26T23:08:55.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.766+0000 7f4135200640 1 Processor -- start 2024-04-26T23:08:55.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.766+0000 7f4135200640 1 -- start start 2024-04-26T23:08:55.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.767+0000 7f4135200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41140039e0 0x7f41300d0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:55.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.767+0000 7f4135200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41301f5820 0x7f4130146af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:55.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.767+0000 7f4135200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41301e7420 0x7f41301e7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:55.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.767+0000 7f4135200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4130148c10 con 0x7f41140039e0 2024-04-26T23:08:55.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.768+0000 7f4135200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4130148a90 con 0x7f41301f5820 2024-04-26T23:08:55.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.768+0000 7f4135200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f41300b6e40 con 0x7f41301e7420 2024-04-26T23:08:55.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.768+0000 7f412f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41140039e0 0x7f41300d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:55.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.768+0000 7f412f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41140039e0 0x7f41300d0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42988/0 (socket says 172.21.15.40:42988) 2024-04-26T23:08:55.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.768+0000 7f412f400640 1 -- 172.21.15.40:0/1080812547 learned_addr learned my addr 172.21.15.40:0/1080812547 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:55.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.768+0000 7f412ea00640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41301f5820 0x7f4130146af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:55.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.768+0000 7f412f400640 1 -- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41301e7420 msgr2=0x7f41301e7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:55.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.768+0000 7f412f400640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41301e7420 0x7f41301e7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:55.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.768+0000 7f412f400640 1 -- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41301f5820 msgr2=0x7f4130146af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:55.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.768+0000 7f412f400640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41301f5820 0x7f4130146af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:55.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.768+0000 7f412f400640 1 -- 172.21.15.40:0/1080812547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4130147030 con 0x7f41140039e0 2024-04-26T23:08:55.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.768+0000 7f412ea00640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41301f5820 0x7f4130146af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:55.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.769+0000 7f412f400640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41140039e0 0x7f41300d0670 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f412000a840 tx=0x7f41300d1e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:55.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.769+0000 7f4127400640 1 -- 172.21.15.40:0/1080812547 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412000a460 con 0x7f41140039e0 2024-04-26T23:08:55.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.769+0000 7f4127400640 1 -- 172.21.15.40:0/1080812547 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4120004d60 con 0x7f41140039e0 2024-04-26T23:08:55.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.769+0000 7f4127400640 1 -- 172.21.15.40:0/1080812547 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4120001fb0 con 0x7f41140039e0 2024-04-26T23:08:55.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.769+0000 7f4135200640 1 -- 172.21.15.40:0/1080812547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41300cf8b0 con 0x7f41140039e0 2024-04-26T23:08:55.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.769+0000 7f4135200640 1 -- 172.21.15.40:0/1080812547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41300cfe70 con 0x7f41140039e0 2024-04-26T23:08:55.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.770+0000 7f4135200640 1 -- 172.21.15.40:0/1080812547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4130069a80 con 0x7f41140039e0 2024-04-26T23:08:55.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.771+0000 7f4127400640 1 -- 172.21.15.40:0/1080812547 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4120034690 con 0x7f41140039e0 2024-04-26T23:08:55.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.772+0000 7f4127400640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f410c06c920 0x7f410c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:55.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.772+0000 7f4127400640 1 -- 172.21.15.40:0/1080812547 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5507+0+0 (secure 0 0 0) 0x7f41200abfe0 con 0x7f41140039e0 2024-04-26T23:08:55.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.773+0000 7f412ea00640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f410c06c920 0x7f410c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:55.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.774+0000 7f412ea00640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f410c06c920 0x7f410c06ede0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f41300ce720 tx=0x7f4118009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:55.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:55.777+0000 7f4127400640 1 -- 172.21.15.40:0/1080812547 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f412002ce20 con 0x7f41140039e0 2024-04-26T23:08:56.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.001+0000 7f4135200640 1 -- 172.21.15.40:0/1080812547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f4130073a80 con 0x7f41140039e0 2024-04-26T23:08:56.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.005+0000 7f4127400640 1 -- 172.21.15.40:0/1080812547 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f412007a4c0 con 0x7f41140039e0 2024-04-26T23:08:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.008+0000 7f4135200640 1 -- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f410c06c920 msgr2=0x7f410c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.008+0000 7f4135200640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f410c06c920 0x7f410c06ede0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f41300ce720 tx=0x7f4118009040 comp rx=0 tx=0).stop 2024-04-26T23:08:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.008+0000 7f4135200640 1 -- 172.21.15.40:0/1080812547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41140039e0 msgr2=0x7f41300d0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.008+0000 7f4135200640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41140039e0 0x7f41300d0670 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f412000a840 tx=0x7f41300d1e80 comp rx=0 tx=0).stop 2024-04-26T23:08:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.008+0000 7f4135200640 1 -- 172.21.15.40:0/1080812547 shutdown_connections 2024-04-26T23:08:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.008+0000 7f4135200640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f410c06c920 0x7f410c06ede0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:56.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.008+0000 7f4135200640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f41301e7420 0x7f41301e7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:56.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.008+0000 7f4135200640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41301f5820 0x7f4130146af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:56.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.008+0000 7f4135200640 1 --2- 172.21.15.40:0/1080812547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41140039e0 0x7f41300d0670 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:56.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.008+0000 7f4135200640 1 -- 172.21.15.40:0/1080812547 >> 172.21.15.40:0/1080812547 conn(0x7f4130068090 msgr2=0x7f41301617e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:56.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.008+0000 7f4135200640 1 -- 172.21.15.40:0/1080812547 shutdown_connections 2024-04-26T23:08:56.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.008+0000 7f4135200640 1 -- 172.21.15.40:0/1080812547 wait complete. 2024-04-26T23:08:56.026 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_pause_and_resume_mds0' 2024-04-26T23:08:56.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.208+0000 7f5298200640 1 Processor -- start 2024-04-26T23:08:56.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.208+0000 7f5298200640 1 -- start start 2024-04-26T23:08:56.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.209+0000 7f5298200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5290156460 0x7f52900cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:56.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.209+0000 7f5298200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5290156840 con 0x7f52900cba10 2024-04-26T23:08:56.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.209+0000 7f5298200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52900cbf20 con 0x7f529006fea0 2024-04-26T23:08:56.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.209+0000 7f5298200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f52900cc200 con 0x7f5290156460 2024-04-26T23:08:56.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.210+0000 7f5295800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5290156460 0x7f52900cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:56.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.210+0000 7f5294e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f52900cba10 0x7f529006f790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45120/0 (socket says 172.21.15.40:45120) 2024-04-26T23:08:56.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.210+0000 7f5294e00640 1 -- 172.21.15.40:0/290126864 learned_addr learned my addr 172.21.15.40:0/290126864 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:56.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.210+0000 7f5295800640 1 -- 172.21.15.40:0/290126864 >> v1:172.21.15.103:6789/0 conn(0x7f529006fea0 legacy=0x7f5290072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:56.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.210+0000 7f5295800640 1 -- 172.21.15.40:0/290126864 >> v1:172.21.15.40:6789/0 conn(0x7f52900cba10 legacy=0x7f529006f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:56.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.210+0000 7f5295800640 1 -- 172.21.15.40:0/290126864 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52900cc4e0 con 0x7f5290156460 2024-04-26T23:08:56.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.212+0000 7f5295800640 1 --2- 172.21.15.40:0/290126864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290156460 0x7f52900cb4d0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f52840097a0 tx=0x7f528402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc84fed10cfc089 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:56.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.212+0000 7f528fe00640 1 -- 172.21.15.40:0/290126864 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5284037070 con 0x7f5290156460 2024-04-26T23:08:56.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.212+0000 7f528fe00640 1 -- 172.21.15.40:0/290126864 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5284032440 con 0x7f5290156460 2024-04-26T23:08:56.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.212+0000 7f528fe00640 1 -- 172.21.15.40:0/290126864 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5284032810 con 0x7f5290156460 2024-04-26T23:08:56.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.213+0000 7f5298200640 1 -- 172.21.15.40:0/290126864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290156460 msgr2=0x7f52900cb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:56.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.213+0000 7f5298200640 1 --2- 172.21.15.40:0/290126864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290156460 0x7f52900cb4d0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f52840097a0 tx=0x7f528402a1e0 comp rx=0 tx=0).stop 2024-04-26T23:08:56.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.213+0000 7f5298200640 1 -- 172.21.15.40:0/290126864 shutdown_connections 2024-04-26T23:08:56.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.213+0000 7f5298200640 1 --2- 172.21.15.40:0/290126864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290156460 0x7f52900cb4d0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:56.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.213+0000 7f5298200640 1 -- 172.21.15.40:0/290126864 >> 172.21.15.40:0/290126864 conn(0x7f52900d0be0 msgr2=0x7f52900d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:56.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.213+0000 7f5298200640 1 -- 172.21.15.40:0/290126864 shutdown_connections 2024-04-26T23:08:56.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.213+0000 7f5298200640 1 -- 172.21.15.40:0/290126864 wait complete. 2024-04-26T23:08:56.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.215+0000 7f5298200640 1 Processor -- start 2024-04-26T23:08:56.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.215+0000 7f5298200640 1 -- start start 2024-04-26T23:08:56.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.216+0000 7f5298200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f529006fea0 0x7f52901f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:56.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.216+0000 7f5298200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52900cba10 0x7f5290150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:56.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.216+0000 7f5298200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290156460 0x7f52900c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:56.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.216+0000 7f5298200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f529006e260 con 0x7f5290156460 2024-04-26T23:08:56.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.216+0000 7f5298200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5290072700 con 0x7f52900cba10 2024-04-26T23:08:56.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.216+0000 7f5298200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f529006e3e0 con 0x7f529006fea0 2024-04-26T23:08:56.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.216+0000 7f5296200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290156460 0x7f52900c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:56.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.217+0000 7f5296200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290156460 0x7f52900c5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43020/0 (socket says 172.21.15.40:43020) 2024-04-26T23:08:56.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.217+0000 7f5296200640 1 -- 172.21.15.40:0/83631489 learned_addr learned my addr 172.21.15.40:0/83631489 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:56.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.217+0000 7f5294e00640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52900cba10 0x7f5290150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:56.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.217+0000 7f5295800640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f529006fea0 0x7f52901f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:56.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.217+0000 7f5296200640 1 -- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f529006fea0 msgr2=0x7f52901f88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:56.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.217+0000 7f5296200640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f529006fea0 0x7f52901f88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:56.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.217+0000 7f5296200640 1 -- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52900cba10 msgr2=0x7f5290150490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:56.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.217+0000 7f5296200640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52900cba10 0x7f5290150490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:56.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.217+0000 7f5296200640 1 -- 172.21.15.40:0/83631489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52900c8410 con 0x7f5290156460 2024-04-26T23:08:56.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.217+0000 7f5296200640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290156460 0x7f52900c5df0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f528800c780 tx=0x7f528800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:56.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.218+0000 7f528d600640 1 -- 172.21.15.40:0/83631489 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288004b60 con 0x7f5290156460 2024-04-26T23:08:56.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.218+0000 7f528d600640 1 -- 172.21.15.40:0/83631489 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5288004d00 con 0x7f5290156460 2024-04-26T23:08:56.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.218+0000 7f5298200640 1 -- 172.21.15.40:0/83631489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52901f9110 con 0x7f5290156460 2024-04-26T23:08:56.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.218+0000 7f528d600640 1 -- 172.21.15.40:0/83631489 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528800aa90 con 0x7f5290156460 2024-04-26T23:08:56.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.218+0000 7f5298200640 1 -- 172.21.15.40:0/83631489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52901f96d0 con 0x7f5290156460 2024-04-26T23:08:56.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.219+0000 7f5298200640 1 -- 172.21.15.40:0/83631489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52900694f0 con 0x7f5290156460 2024-04-26T23:08:56.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.221+0000 7f528d600640 1 -- 172.21.15.40:0/83631489 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f528800ac30 con 0x7f5290156460 2024-04-26T23:08:56.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.222+0000 7f528d600640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f527006c920 0x7f527006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:56.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.222+0000 7f528d600640 1 -- 172.21.15.40:0/83631489 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5507+0+0 (secure 0 0 0) 0x7f528808bff0 con 0x7f5290156460 2024-04-26T23:08:56.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.223+0000 7f5295800640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f527006c920 0x7f527006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:56.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.224+0000 7f5295800640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f527006c920 0x7f527006ede0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5284034000 tx=0x7f528402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:56.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.227+0000 7f528d600640 1 -- 172.21.15.40:0/83631489 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f528805a4a0 con 0x7f5290156460 2024-04-26T23:08:56.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:56.520+0000 7f5298200640 1 -- 172.21.15.40:0/83631489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_pause_and_resume_mds0"]} v 0) -- 0x7f5290151980 con 0x7f5290156460 2024-04-26T23:08:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.535+0000 7f528d600640 1 -- 172.21.15.40:0/83631489 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_pause_and_resume_mds0"]}]=0 v416) ==== 154+0+0 (secure 0 0 0) 0x7f5288059da0 con 0x7f5290156460 2024-04-26T23:08:57.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.539+0000 7f5298200640 1 -- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f527006c920 msgr2=0x7f527006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:57.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.539+0000 7f5298200640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f527006c920 0x7f527006ede0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5284034000 tx=0x7f528402e000 comp rx=0 tx=0).stop 2024-04-26T23:08:57.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.539+0000 7f5298200640 1 -- 172.21.15.40:0/83631489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290156460 msgr2=0x7f52900c5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:57.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.539+0000 7f5298200640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290156460 0x7f52900c5df0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f528800c780 tx=0x7f528800cc40 comp rx=0 tx=0).stop 2024-04-26T23:08:57.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.539+0000 7f5298200640 1 -- 172.21.15.40:0/83631489 shutdown_connections 2024-04-26T23:08:57.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.539+0000 7f5298200640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f527006c920 0x7f527006ede0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:57.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.539+0000 7f5298200640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5290156460 0x7f52900c5df0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:57.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.539+0000 7f5298200640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52900cba10 0x7f5290150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:57.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.539+0000 7f5298200640 1 --2- 172.21.15.40:0/83631489 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f529006fea0 0x7f52901f88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:57.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.539+0000 7f5298200640 1 -- 172.21.15.40:0/83631489 >> 172.21.15.40:0/83631489 conn(0x7f52900d0be0 msgr2=0x7f52900b0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:57.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.539+0000 7f5298200640 1 -- 172.21.15.40:0/83631489 shutdown_connections 2024-04-26T23:08:57.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.539+0000 7f5298200640 1 -- 172.21.15.40:0/83631489 wait complete. 2024-04-26T23:08:57.564 INFO:tasks.cephfs_test_runner:test_scrub_pause_and_resume_mds0 (tasks.cephfs.test_multimds_misc.TestScrub2) ... ok 2024-04-26T23:08:57.565 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_pause_and_resume_with_abort_mds0 (tasks.cephfs.test_multimds_misc.TestScrub2) 2024-04-26T23:08:57.565 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:08:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.776+0000 7fd1e5800640 1 Processor -- start 2024-04-26T23:08:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.776+0000 7fd1e5800640 1 -- start start 2024-04-26T23:08:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.777+0000 7fd1e5800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1e00af7a0 0x7fd1e00afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.777+0000 7fd1e5800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e00d0820 con 0x7fd1e015ce60 2024-04-26T23:08:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.778+0000 7fd1e5800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e01614c0 con 0x7fd1e0071fe0 2024-04-26T23:08:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.778+0000 7fd1e5800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e00cc450 con 0x7fd1e00af7a0 2024-04-26T23:08:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.778+0000 7fd1dfe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd1e015ce60 0x7fd1e00ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45134/0 (socket says 172.21.15.40:45134) 2024-04-26T23:08:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.778+0000 7fd1dfe00640 1 -- 172.21.15.40:0/2107620583 learned_addr learned my addr 172.21.15.40:0/2107620583 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.778+0000 7fd1d7e00640 1 --2- 172.21.15.40:0/2107620583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1e00af7a0 0x7fd1e00afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.778+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2547786125 0 0) 0x7fd1e00d0820 con 0x7fd1e015ce60 2024-04-26T23:08:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.778+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1c4003520 con 0x7fd1e015ce60 2024-04-26T23:08:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3541120159 0 0) 0x7fd1c4003520 con 0x7fd1e015ce60 2024-04-26T23:08:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1e00af7a0 msgr2=0x7fd1e00afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1df400640 1 --2- 172.21.15.40:0/2107620583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1e00af7a0 0x7fd1e00afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 >> v1:172.21.15.103:6789/0 conn(0x7fd1e0071fe0 legacy=0x7fd1e0073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e0161760 con 0x7fd1e015ce60 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fd1d0001670 con 0x7fd1e015ce60 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 >> v1:172.21.15.40:6789/0 conn(0x7fd1e015ce60 legacy=0x7fd1e00ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1df400640 1 --2- 172.21.15.40:0/2107620583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c40039e0 0x7fd1c4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1df400640 1 --2- 172.21.15.40:0/2107620583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1c4004800 0x7fd1c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1df400640 1 --2- 172.21.15.40:0/2107620583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1c4005290 0x7fd1c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e00cc450 con 0x7fd1c40039e0 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c400c160 con 0x7fd1c4004800 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c400c360 con 0x7fd1c4005290 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1d7e00640 1 --2- 172.21.15.40:0/2107620583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1e00af7a0 0x7fd1e00afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.779+0000 7fd1dfe00640 1 --2- 172.21.15.40:0/2107620583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c40039e0 0x7fd1c4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.780+0000 7fd1dfe00640 1 -- 172.21.15.40:0/2107620583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1c4005290 msgr2=0x7fd1c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.780+0000 7fd1dfe00640 1 --2- 172.21.15.40:0/2107620583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1c4005290 0x7fd1c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.780+0000 7fd1d7e00640 1 --2- 172.21.15.40:0/2107620583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1c4004800 0x7fd1c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.780+0000 7fd1dfe00640 1 -- 172.21.15.40:0/2107620583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1c4004800 msgr2=0x7fd1c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.780+0000 7fd1dfe00640 1 --2- 172.21.15.40:0/2107620583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1c4004800 0x7fd1c4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.780+0000 7fd1dfe00640 1 -- 172.21.15.40:0/2107620583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c4003520 con 0x7fd1c40039e0 2024-04-26T23:08:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.781+0000 7fd1dfe00640 1 --2- 172.21.15.40:0/2107620583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c40039e0 0x7fd1c4003e00 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000c190 tx=0x7fd1c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8283e3b937a9cfe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.781+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0007c60 con 0x7fd1c40039e0 2024-04-26T23:08:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.781+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d0005ad0 con 0x7fd1c40039e0 2024-04-26T23:08:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.781+0000 7fd1df400640 1 -- 172.21.15.40:0/2107620583 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0007070 con 0x7fd1c40039e0 2024-04-26T23:08:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.782+0000 7fd1e5800640 1 -- 172.21.15.40:0/2107620583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c40039e0 msgr2=0x7fd1c4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.782+0000 7fd1e5800640 1 --2- 172.21.15.40:0/2107620583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1c40039e0 0x7fd1c4003e00 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000c190 tx=0x7fd1c4005830 comp rx=0 tx=0).stop 2024-04-26T23:08:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.782+0000 7fd1e4e00640 1 -- 172.21.15.40:0/2107620583 reap_dead start 2024-04-26T23:08:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.782+0000 7fd1e5800640 1 -- 172.21.15.40:0/2107620583 shutdown_connections 2024-04-26T23:08:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.782+0000 7fd1e5800640 1 -- 172.21.15.40:0/2107620583 >> 172.21.15.40:0/2107620583 conn(0x7fd1e00d0a10 msgr2=0x7fd1e00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.782+0000 7fd1e5800640 1 -- 172.21.15.40:0/2107620583 shutdown_connections 2024-04-26T23:08:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.782+0000 7fd1e5800640 1 -- 172.21.15.40:0/2107620583 wait complete. 2024-04-26T23:08:57.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.784+0000 7fd1e5800640 1 Processor -- start 2024-04-26T23:08:57.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.784+0000 7fd1e5800640 1 -- start start 2024-04-26T23:08:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e5800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1c40039e0 0x7fd1e00c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e5800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e014e0c0 0x7fd1e015df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1e00b1910 0x7fd1e00b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e5800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e00cc1f0 con 0x7fd1e00b1910 2024-04-26T23:08:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e5800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e00cc070 con 0x7fd1e014e0c0 2024-04-26T23:08:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e5800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e0073e70 con 0x7fd1c40039e0 2024-04-26T23:08:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1e00b1910 0x7fd1e00b1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1e00b1910 0x7fd1e00b1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43042/0 (socket says 172.21.15.40:43042) 2024-04-26T23:08:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e4e00640 1 -- 172.21.15.40:0/2895922367 learned_addr learned my addr 172.21.15.40:0/2895922367 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1dfe00640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1c40039e0 0x7fd1e00c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1d7e00640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e014e0c0 0x7fd1e015df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e4e00640 1 -- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1c40039e0 msgr2=0x7fd1e00c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e4e00640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1c40039e0 0x7fd1e00c8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e4e00640 1 -- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e014e0c0 msgr2=0x7fd1e015df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e4e00640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e014e0c0 0x7fd1e015df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.785+0000 7fd1e4e00640 1 -- 172.21.15.40:0/2895922367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e014e540 con 0x7fd1e00b1910 2024-04-26T23:08:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.786+0000 7fd1e4e00640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1e00b1910 0x7fd1e00b1dc0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fd1e006c2f0 tx=0x7fd1d800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.786+0000 7fd1dcc00640 1 -- 172.21.15.40:0/2895922367 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8017070 con 0x7fd1e00b1910 2024-04-26T23:08:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.786+0000 7fd1dcc00640 1 -- 172.21.15.40:0/2895922367 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d8005d60 con 0x7fd1e00b1910 2024-04-26T23:08:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.786+0000 7fd1dcc00640 1 -- 172.21.15.40:0/2895922367 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d800a850 con 0x7fd1e00b1910 2024-04-26T23:08:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.786+0000 7fd1e5800640 1 -- 172.21.15.40:0/2895922367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e00c93b0 con 0x7fd1e00b1910 2024-04-26T23:08:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.786+0000 7fd1e5800640 1 -- 172.21.15.40:0/2895922367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e00c9970 con 0x7fd1e00b1910 2024-04-26T23:08:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.787+0000 7fd1e5800640 1 -- 172.21.15.40:0/2895922367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e0069370 con 0x7fd1e00b1910 2024-04-26T23:08:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.788+0000 7fd1dcc00640 1 -- 172.21.15.40:0/2895922367 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd1d801a070 con 0x7fd1e00b1910 2024-04-26T23:08:57.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.789+0000 7fd1dcc00640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd1bc06c8f0 0x7fd1bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:57.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.789+0000 7fd1dcc00640 1 -- 172.21.15.40:0/2895922367 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5507+0+0 (secure 0 0 0) 0x7fd1d808ae10 con 0x7fd1e00b1910 2024-04-26T23:08:57.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.790+0000 7fd1dfe00640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd1bc06c8f0 0x7fd1bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:57.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.791+0000 7fd1dfe00640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd1bc06c8f0 0x7fd1bc06edb0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000c160 tx=0x7fd1d0009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:57.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.794+0000 7fd1dcc00640 1 -- 172.21.15.40:0/2895922367 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd1d808e030 con 0x7fd1e00b1910 2024-04-26T23:08:57.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.995+0000 7fd1e5800640 1 -- 172.21.15.40:0/2895922367 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd1e00d01f0 con 0x7fd1e00b1910 2024-04-26T23:08:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:57.997+0000 7fd1dcc00640 1 -- 172.21.15.40:0/2895922367 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+13516 (secure 0 0 0) 0x7fd1d8059260 con 0x7fd1e00b1910 2024-04-26T23:08:57.998 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:08:57.998 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":95,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:08:55.518135+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6837/2728939257":"2024-04-27T23:08:52.507025+0000","172.21.15.103:6836/2728939257":"2024-04-27T23:08:52.507025+0000","172.21.15.40:6835/88401598":"2024-04-27T23:08:52.507043+0000","172.21.15.40:6834/88401598":"2024-04-27T23:08:52.507043+0000","172.21.15.40:6833/1811310743":"2024-04-27T23:08:52.507039+0000","172.21.15.40:6832/1811310743":"2024-04-27T23:08:52.507039+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:08:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.000+0000 7fd1e5800640 1 -- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd1bc06c8f0 msgr2=0x7fd1bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.000+0000 7fd1e5800640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd1bc06c8f0 0x7fd1bc06edb0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000c160 tx=0x7fd1d0009030 comp rx=0 tx=0).stop 2024-04-26T23:08:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.000+0000 7fd1e5800640 1 -- 172.21.15.40:0/2895922367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1e00b1910 msgr2=0x7fd1e00b1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.000+0000 7fd1e5800640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1e00b1910 0x7fd1e00b1dc0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fd1e006c2f0 tx=0x7fd1d800cb90 comp rx=0 tx=0).stop 2024-04-26T23:08:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.000+0000 7fd1e5800640 1 -- 172.21.15.40:0/2895922367 shutdown_connections 2024-04-26T23:08:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.000+0000 7fd1e5800640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd1bc06c8f0 0x7fd1bc06edb0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.000+0000 7fd1e5800640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1e00b1910 0x7fd1e00b1dc0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.000+0000 7fd1e5800640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e014e0c0 0x7fd1e015df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.000+0000 7fd1e5800640 1 --2- 172.21.15.40:0/2895922367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd1c40039e0 0x7fd1e00c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.000+0000 7fd1e5800640 1 -- 172.21.15.40:0/2895922367 >> 172.21.15.40:0/2895922367 conn(0x7fd1e00d0a10 msgr2=0x7fd1e00caed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.000+0000 7fd1e5800640 1 -- 172.21.15.40:0/2895922367 shutdown_connections 2024-04-26T23:08:58.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.001+0000 7fd1e5800640 1 -- 172.21.15.40:0/2895922367 wait complete. 2024-04-26T23:08:58.020 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:08:58.021 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:08:58.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.237+0000 7fe2b2000640 1 Processor -- start 2024-04-26T23:08:58.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.237+0000 7fe2b2000640 1 -- start start 2024-04-26T23:08:58.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.239+0000 7fe2b2000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac15bc70 0x7fe2ac0cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.239+0000 7fe2b2000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2ac071ef0 0x7fe2ac072990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.239+0000 7fe2b2000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe2ac072440 0x7fe2ac072ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.239+0000 7fe2b2000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe2ac06e3a0 con 0x7fe2ac071ef0 2024-04-26T23:08:58.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.239+0000 7fe2b2000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe2ac0b4840 con 0x7fe2ac072440 2024-04-26T23:08:58.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.239+0000 7fe2b2000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2ac06dde0 con 0x7fe2ac15bc70 2024-04-26T23:08:58.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.239+0000 7fe2abe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2ac071ef0 0x7fe2ac072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:58.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.239+0000 7fe2abe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2ac071ef0 0x7fe2ac072990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43054/0 (socket says 172.21.15.40:43054) 2024-04-26T23:08:58.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.239+0000 7fe2abe00640 1 -- 172.21.15.40:0/3363217087 learned_addr learned my addr 172.21.15.40:0/3363217087 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:58.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.240+0000 7fe2b0c00640 1 --2- 172.21.15.40:0/3363217087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac15bc70 0x7fe2ac0cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:58.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.240+0000 7fe2b1600640 1 --2- 172.21.15.40:0/3363217087 >> v2:172.21.15.103:3300/0 conn(0x7fe2ac072440 0x7fe2ac072ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:58.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.240+0000 7fe2abe00640 1 -- 172.21.15.40:0/3363217087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac15bc70 msgr2=0x7fe2ac0cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:58.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.240+0000 7fe2abe00640 1 --2- 172.21.15.40:0/3363217087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac15bc70 0x7fe2ac0cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.240+0000 7fe2abe00640 1 -- 172.21.15.40:0/3363217087 >> v2:172.21.15.103:3300/0 conn(0x7fe2ac072440 msgr2=0x7fe2ac072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:58.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.240+0000 7fe2abe00640 1 --2- 172.21.15.40:0/3363217087 >> v2:172.21.15.103:3300/0 conn(0x7fe2ac072440 0x7fe2ac072ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.240+0000 7fe2abe00640 1 -- 172.21.15.40:0/3363217087 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ac06e0c0 con 0x7fe2ac071ef0 2024-04-26T23:08:58.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.240+0000 7fe2b0c00640 1 --2- 172.21.15.40:0/3363217087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac15bc70 0x7fe2ac0cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:58.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.241+0000 7fe2abe00640 1 --2- 172.21.15.40:0/3363217087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2ac071ef0 0x7fe2ac072990 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac06daf0 tx=0x7fe29802c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c31eff4f478088 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:58.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.241+0000 7fe2ab400640 1 -- 172.21.15.40:0/3363217087 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe298032790 con 0x7fe2ac071ef0 2024-04-26T23:08:58.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.241+0000 7fe2ab400640 1 -- 172.21.15.40:0/3363217087 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe298038070 con 0x7fe2ac071ef0 2024-04-26T23:08:58.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.241+0000 7fe2ab400640 1 -- 172.21.15.40:0/3363217087 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe298032b50 con 0x7fe2ac071ef0 2024-04-26T23:08:58.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.242+0000 7fe2b2000640 1 -- 172.21.15.40:0/3363217087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2ac071ef0 msgr2=0x7fe2ac072990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:58.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.242+0000 7fe2b2000640 1 --2- 172.21.15.40:0/3363217087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2ac071ef0 0x7fe2ac072990 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac06daf0 tx=0x7fe29802c890 comp rx=0 tx=0).stop 2024-04-26T23:08:58.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.242+0000 7fe2b2000640 1 -- 172.21.15.40:0/3363217087 shutdown_connections 2024-04-26T23:08:58.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.242+0000 7fe2b2000640 1 --2- 172.21.15.40:0/3363217087 >> v2:172.21.15.103:3300/0 conn(0x7fe2ac072440 0x7fe2ac072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.242+0000 7fe2b2000640 1 --2- 172.21.15.40:0/3363217087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2ac071ef0 0x7fe2ac072990 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.242+0000 7fe2b2000640 1 --2- 172.21.15.40:0/3363217087 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac15bc70 0x7fe2ac0cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.242+0000 7fe2b2000640 1 -- 172.21.15.40:0/3363217087 >> 172.21.15.40:0/3363217087 conn(0x7fe2ac0b2720 msgr2=0x7fe2ac0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:58.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.242+0000 7fe2b2000640 1 -- 172.21.15.40:0/3363217087 shutdown_connections 2024-04-26T23:08:58.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.243+0000 7fe2b2000640 1 -- 172.21.15.40:0/3363217087 wait complete. 2024-04-26T23:08:58.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.244+0000 7fe2b2000640 1 Processor -- start 2024-04-26T23:08:58.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.244+0000 7fe2b2000640 1 -- start start 2024-04-26T23:08:58.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.245+0000 7fe2b2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2ac072440 0x7fe2ac159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.245+0000 7fe2b2000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac15bc70 0x7fe2ac0aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.245+0000 7fe2b2000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2ac150140 0x7fe2ac1f8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.245+0000 7fe2b2000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac15c050 con 0x7fe2ac072440 2024-04-26T23:08:58.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.245+0000 7fe2b2000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac06fad0 con 0x7fe2ac150140 2024-04-26T23:08:58.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.245+0000 7fe2b2000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2ac15c1d0 con 0x7fe2ac15bc70 2024-04-26T23:08:58.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.246+0000 7fe2b0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2ac072440 0x7fe2ac159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:58.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.246+0000 7fe2b0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2ac072440 0x7fe2ac159b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43068/0 (socket says 172.21.15.40:43068) 2024-04-26T23:08:58.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.246+0000 7fe2b0c00640 1 -- 172.21.15.40:0/120262499 learned_addr learned my addr 172.21.15.40:0/120262499 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:58.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.246+0000 7fe2abe00640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac15bc70 0x7fe2ac0aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:58.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.246+0000 7fe2b1600640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2ac150140 0x7fe2ac1f8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:58.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.246+0000 7fe2b0c00640 1 -- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac15bc70 msgr2=0x7fe2ac0aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:58.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.246+0000 7fe2b0c00640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac15bc70 0x7fe2ac0aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.246+0000 7fe2b0c00640 1 -- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2ac150140 msgr2=0x7fe2ac1f8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:58.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.246+0000 7fe2b0c00640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2ac150140 0x7fe2ac1f8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.246+0000 7fe2b0c00640 1 -- 172.21.15.40:0/120262499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ac15a0a0 con 0x7fe2ac072440 2024-04-26T23:08:58.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.246+0000 7fe2abe00640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac15bc70 0x7fe2ac0aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:58.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.246+0000 7fe2b1600640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2ac150140 0x7fe2ac1f8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:08:58.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.247+0000 7fe2b0c00640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2ac072440 0x7fe2ac159b60 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fe2a000c7f0 tx=0x7fe2a000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:58.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.247+0000 7fe2a8c00640 1 -- 172.21.15.40:0/120262499 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0015070 con 0x7fe2ac072440 2024-04-26T23:08:58.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.247+0000 7fe2a8c00640 1 -- 172.21.15.40:0/120262499 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2a0007820 con 0x7fe2ac072440 2024-04-26T23:08:58.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.247+0000 7fe2a8c00640 1 -- 172.21.15.40:0/120262499 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0007bd0 con 0x7fe2ac072440 2024-04-26T23:08:58.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.247+0000 7fe2b2000640 1 -- 172.21.15.40:0/120262499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2ac159380 con 0x7fe2ac072440 2024-04-26T23:08:58.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.247+0000 7fe2b2000640 1 -- 172.21.15.40:0/120262499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2ac15ad20 con 0x7fe2ac072440 2024-04-26T23:08:58.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.248+0000 7fe2b2000640 1 -- 172.21.15.40:0/120262499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2ac069370 con 0x7fe2ac072440 2024-04-26T23:08:58.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.249+0000 7fe2a8c00640 1 -- 172.21.15.40:0/120262499 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2a0018070 con 0x7fe2ac072440 2024-04-26T23:08:58.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.250+0000 7fe2a8c00640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe28c06c8f0 0x7fe28c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.250+0000 7fe2a8c00640 1 -- 172.21.15.40:0/120262499 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5507+0+0 (secure 0 0 0) 0x7fe2a008bba0 con 0x7fe2ac072440 2024-04-26T23:08:58.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.251+0000 7fe2abe00640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe28c06c8f0 0x7fe28c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:58.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.252+0000 7fe2abe00640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe28c06c8f0 0x7fe28c06edb0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe298037000 tx=0x7fe298008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.255+0000 7fe2a8c00640 1 -- 172.21.15.40:0/120262499 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe2a005a090 con 0x7fe2ac072440 2024-04-26T23:08:58.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.460+0000 7fe2b2000640 1 -- 172.21.15.40:0/120262499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe2ac0cd1f0 con 0x7fe2ac072440 2024-04-26T23:08:58.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.460+0000 7fe2a8c00640 1 -- 172.21.15.40:0/120262499 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7fe2a0059990 con 0x7fe2ac072440 2024-04-26T23:08:58.461 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:08:58.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.463+0000 7fe2b2000640 1 -- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe28c06c8f0 msgr2=0x7fe28c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:58.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.463+0000 7fe2b2000640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe28c06c8f0 0x7fe28c06edb0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe298037000 tx=0x7fe298008040 comp rx=0 tx=0).stop 2024-04-26T23:08:58.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.463+0000 7fe2b2000640 1 -- 172.21.15.40:0/120262499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2ac072440 msgr2=0x7fe2ac159b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:58.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.463+0000 7fe2b2000640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2ac072440 0x7fe2ac159b60 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fe2a000c7f0 tx=0x7fe2a000ccb0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.463+0000 7fe2b2000640 1 -- 172.21.15.40:0/120262499 shutdown_connections 2024-04-26T23:08:58.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.463+0000 7fe2b2000640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe28c06c8f0 0x7fe28c06edb0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.463+0000 7fe2b2000640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2ac150140 0x7fe2ac1f8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.463+0000 7fe2b2000640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2ac15bc70 0x7fe2ac0aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.463+0000 7fe2b2000640 1 --2- 172.21.15.40:0/120262499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2ac072440 0x7fe2ac159b60 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.463+0000 7fe2b2000640 1 -- 172.21.15.40:0/120262499 >> 172.21.15.40:0/120262499 conn(0x7fe2ac0b2720 msgr2=0x7fe2ac0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:58.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.463+0000 7fe2b2000640 1 -- 172.21.15.40:0/120262499 shutdown_connections 2024-04-26T23:08:58.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.464+0000 7fe2b2000640 1 -- 172.21.15.40:0/120262499 wait complete. 2024-04-26T23:08:58.483 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_pause_and_resume_with_abort_mds0' 2024-04-26T23:08:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.698+0000 7fc015400640 1 Processor -- start 2024-04-26T23:08:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.698+0000 7fc015400640 1 -- start start 2024-04-26T23:08:58.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.699+0000 7fc015400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100cc480 0x7fc010072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.699+0000 7fc015400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc01015d7c0 con 0x7fc010071e60 2024-04-26T23:08:58.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.699+0000 7fc015400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc01015daa0 con 0x7fc010072780 2024-04-26T23:08:58.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.699+0000 7fc015400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc01015dd80 con 0x7fc0100cc480 2024-04-26T23:08:58.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.700+0000 7fc00e800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc010071e60 0x7fc01006f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45146/0 (socket says 172.21.15.40:45146) 2024-04-26T23:08:58.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.700+0000 7fc00e800640 1 -- 172.21.15.40:0/1515252660 learned_addr learned my addr 172.21.15.40:0/1515252660 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:58.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.700+0000 7fc005e00640 1 --2- 172.21.15.40:0/1515252660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100cc480 0x7fc010072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.700+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3756175186 0 0) 0x7fc01015d7c0 con 0x7fc010071e60 2024-04-26T23:08:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.700+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbff4003520 con 0x7fc010071e60 2024-04-26T23:08:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3797736562 0 0) 0x7fbff4003520 con 0x7fc010071e60 2024-04-26T23:08:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100cc480 msgr2=0x7fc010072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00de00640 1 --2- 172.21.15.40:0/1515252660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100cc480 0x7fc010072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 >> v1:172.21.15.103:6789/0 conn(0x7fc010072780 legacy=0x7fc010166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01015e060 con 0x7fc010071e60 2024-04-26T23:08:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fc000001670 con 0x7fc010071e60 2024-04-26T23:08:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 >> v1:172.21.15.40:6789/0 conn(0x7fc010071e60 legacy=0x7fc01006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00de00640 1 --2- 172.21.15.40:0/1515252660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff40039e0 0x7fbff4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00de00640 1 --2- 172.21.15.40:0/1515252660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff4004800 0x7fbff4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00de00640 1 --2- 172.21.15.40:0/1515252660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100cc480 0x7fbff40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc01015dd80 con 0x7fbff40039e0 2024-04-26T23:08:58.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbff400ba70 con 0x7fbff4004800 2024-04-26T23:08:58.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbff400bc70 con 0x7fc0100cc480 2024-04-26T23:08:58.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00e800640 1 --2- 172.21.15.40:0/1515252660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff40039e0 0x7fbff4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:58.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00e800640 1 -- 172.21.15.40:0/1515252660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100cc480 msgr2=0x7fbff40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:58.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00e800640 1 --2- 172.21.15.40:0/1515252660 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100cc480 0x7fbff40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00e800640 1 -- 172.21.15.40:0/1515252660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff4004800 msgr2=0x7fbff4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:58.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00e800640 1 --2- 172.21.15.40:0/1515252660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff4004800 0x7fbff4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.701+0000 7fc00e800640 1 -- 172.21.15.40:0/1515252660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff4003520 con 0x7fbff40039e0 2024-04-26T23:08:58.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.702+0000 7fc00e800640 1 --2- 172.21.15.40:0/1515252660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff40039e0 0x7fbff4003e00 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fc00000c210 tx=0x7fbff4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9921b1d881f4e47 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:58.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.703+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc000007cf0 con 0x7fbff40039e0 2024-04-26T23:08:58.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.703+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc000005ad0 con 0x7fbff40039e0 2024-04-26T23:08:58.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.703+0000 7fc00de00640 1 -- 172.21.15.40:0/1515252660 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc000007070 con 0x7fbff40039e0 2024-04-26T23:08:58.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.703+0000 7fc015400640 1 -- 172.21.15.40:0/1515252660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff40039e0 msgr2=0x7fbff4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:58.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.703+0000 7fc015400640 1 --2- 172.21.15.40:0/1515252660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff40039e0 0x7fbff4003e00 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fc00000c210 tx=0x7fbff4005830 comp rx=0 tx=0).stop 2024-04-26T23:08:58.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.703+0000 7fc00f200640 1 -- 172.21.15.40:0/1515252660 reap_dead start 2024-04-26T23:08:58.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.703+0000 7fc015400640 1 -- 172.21.15.40:0/1515252660 shutdown_connections 2024-04-26T23:08:58.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.703+0000 7fc015400640 1 -- 172.21.15.40:0/1515252660 >> 172.21.15.40:0/1515252660 conn(0x7fc0100d0be0 msgr2=0x7fc0100d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:58.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.704+0000 7fc015400640 1 -- 172.21.15.40:0/1515252660 shutdown_connections 2024-04-26T23:08:58.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.704+0000 7fc015400640 1 -- 172.21.15.40:0/1515252660 wait complete. 2024-04-26T23:08:58.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.705+0000 7fc015400640 1 Processor -- start 2024-04-26T23:08:58.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.705+0000 7fc015400640 1 -- start start 2024-04-26T23:08:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.706+0000 7fc015400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff40039e0 0x7fc0101510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.706+0000 7fc015400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100b5050 0x7fc0100b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.706+0000 7fc015400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0100b3b90 0x7fc0100b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.706+0000 7fc015400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc01015d3f0 con 0x7fbff40039e0 2024-04-26T23:08:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.706+0000 7fc015400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc010072c10 con 0x7fc0100b3b90 2024-04-26T23:08:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.706+0000 7fc015400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc01015d570 con 0x7fc0100b5050 2024-04-26T23:08:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.707+0000 7fc00e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff40039e0 0x7fc0101510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.707+0000 7fc00e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff40039e0 0x7fc0101510d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43092/0 (socket says 172.21.15.40:43092) 2024-04-26T23:08:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.707+0000 7fc00e800640 1 -- 172.21.15.40:0/2360868726 learned_addr learned my addr 172.21.15.40:0/2360868726 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.707+0000 7fc00f200640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0100b3b90 0x7fc0100b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.707+0000 7fc005e00640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100b5050 0x7fc0100b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.707+0000 7fc00e800640 1 -- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100b5050 msgr2=0x7fc0100b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.707+0000 7fc00e800640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100b5050 0x7fc0100b54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.707+0000 7fc00e800640 1 -- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0100b3b90 msgr2=0x7fc0100b4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.707+0000 7fc00e800640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0100b3b90 0x7fc0100b4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.707+0000 7fc00e800640 1 -- 172.21.15.40:0/2360868726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc010151610 con 0x7fbff40039e0 2024-04-26T23:08:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.708+0000 7fc00e800640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff40039e0 0x7fc0101510d0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fc00000c330 tx=0x7fc01015dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.708+0000 7fc007400640 1 -- 172.21.15.40:0/2360868726 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc000001630 con 0x7fbff40039e0 2024-04-26T23:08:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.708+0000 7fc007400640 1 -- 172.21.15.40:0/2360868726 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc000038040 con 0x7fbff40039e0 2024-04-26T23:08:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.708+0000 7fc007400640 1 -- 172.21.15.40:0/2360868726 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc000007290 con 0x7fbff40039e0 2024-04-26T23:08:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.708+0000 7fc015400640 1 -- 172.21.15.40:0/2360868726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc01015bcb0 con 0x7fbff40039e0 2024-04-26T23:08:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.708+0000 7fc015400640 1 -- 172.21.15.40:0/2360868726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc01015c270 con 0x7fbff40039e0 2024-04-26T23:08:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.709+0000 7fc015400640 1 -- 172.21.15.40:0/2360868726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0100694f0 con 0x7fbff40039e0 2024-04-26T23:08:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.711+0000 7fc007400640 1 -- 172.21.15.40:0/2360868726 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc000007430 con 0x7fbff40039e0 2024-04-26T23:08:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.712+0000 7fc007400640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfec06c8f0 0x7fbfec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.712+0000 7fc007400640 1 -- 172.21.15.40:0/2360868726 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5507+0+0 (secure 0 0 0) 0x7fc0000abec0 con 0x7fbff40039e0 2024-04-26T23:08:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.712+0000 7fc005e00640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfec06c8f0 0x7fbfec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:58.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.713+0000 7fc005e00640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfec06c8f0 0x7fbfec06edb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc0100b4a60 tx=0x7fbff8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:58.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.716+0000 7fc007400640 1 -- 172.21.15.40:0/2360868726 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc000005aa0 con 0x7fbff40039e0 2024-04-26T23:08:58.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:58.994+0000 7fc015400640 1 -- 172.21.15.40:0/2360868726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_pause_and_resume_with_abort_mds0"]} v 0) -- 0x7fc010069cd0 con 0x7fbff40039e0 2024-04-26T23:08:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.547+0000 7fc007400640 1 -- 172.21.15.40:0/2360868726 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_pause_and_resume_with_abort_mds0"]}]=0 v418) ==== 168+0+0 (secure 0 0 0) 0x7fc00007a310 con 0x7fbff40039e0 2024-04-26T23:08:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.552+0000 7fc015400640 1 -- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfec06c8f0 msgr2=0x7fbfec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.552+0000 7fc015400640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfec06c8f0 0x7fbfec06edb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc0100b4a60 tx=0x7fbff8008040 comp rx=0 tx=0).stop 2024-04-26T23:08:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.552+0000 7fc015400640 1 -- 172.21.15.40:0/2360868726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff40039e0 msgr2=0x7fc0101510d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.552+0000 7fc015400640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff40039e0 0x7fc0101510d0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fc00000c330 tx=0x7fc01015dc70 comp rx=0 tx=0).stop 2024-04-26T23:08:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.552+0000 7fc015400640 1 -- 172.21.15.40:0/2360868726 shutdown_connections 2024-04-26T23:08:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.552+0000 7fc015400640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfec06c8f0 0x7fbfec06edb0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.552+0000 7fc015400640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0100b3b90 0x7fc0100b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.552+0000 7fc015400640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc0100b5050 0x7fc0100b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.552+0000 7fc015400640 1 --2- 172.21.15.40:0/2360868726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff40039e0 0x7fc0101510d0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.552+0000 7fc015400640 1 -- 172.21.15.40:0/2360868726 >> 172.21.15.40:0/2360868726 conn(0x7fc0100d0be0 msgr2=0x7fc010073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.552+0000 7fc015400640 1 -- 172.21.15.40:0/2360868726 shutdown_connections 2024-04-26T23:08:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.553+0000 7fc015400640 1 -- 172.21.15.40:0/2360868726 wait complete. 2024-04-26T23:08:59.576 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-26T23:08:59.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.796+0000 7fe6a5400640 1 Processor -- start 2024-04-26T23:08:59.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.796+0000 7fe6a5400640 1 -- start start 2024-04-26T23:08:59.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.797+0000 7fe6a5400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6a00730a0 0x7fe6a00cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:59.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.798+0000 7fe6a5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe6a0071ef0 0x7fe6a00b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:59.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.798+0000 7fe6a5400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6a006dde0 con 0x7fe6a015bc70 2024-04-26T23:08:59.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.798+0000 7fe6a5400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe6a006e3a0 con 0x7fe6a00730a0 2024-04-26T23:08:59.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.798+0000 7fe6a5400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe6a00b4840 con 0x7fe6a0071ef0 2024-04-26T23:08:59.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.798+0000 7fe69fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe6a015bc70 0x7fe6a0072990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45158/0 (socket says 172.21.15.40:45158) 2024-04-26T23:08:59.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.798+0000 7fe69fe00640 1 -- 172.21.15.40:0/3122167492 learned_addr learned my addr 172.21.15.40:0/3122167492 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:08:59.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.798+0000 7fe69f400640 1 --2- 172.21.15.40:0/3122167492 >> v2:172.21.15.40:3300/0 conn(0x7fe6a00730a0 0x7fe6a00cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:59.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.798+0000 7fe6a4a00640 1 --2- 172.21.15.40:0/3122167492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe6a0071ef0 0x7fe6a00b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:59.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.798+0000 7fe69f400640 1 -- 172.21.15.40:0/3122167492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe6a0071ef0 msgr2=0x7fe6a00b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:08:59.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.798+0000 7fe69f400640 1 --2- 172.21.15.40:0/3122167492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe6a0071ef0 0x7fe6a00b4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:59.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.798+0000 7fe69f400640 1 -- 172.21.15.40:0/3122167492 >> v1:172.21.15.40:6789/0 conn(0x7fe6a015bc70 legacy=0x7fe6a0072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:59.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.798+0000 7fe69f400640 1 -- 172.21.15.40:0/3122167492 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a006e0c0 con 0x7fe6a00730a0 2024-04-26T23:08:59.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.800+0000 7fe69f400640 1 --2- 172.21.15.40:0/3122167492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6a00730a0 0x7fe6a00cdf30 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0097a0 tx=0x7fe68c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a1b92b4081d887 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:59.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.800+0000 7fe69ea00640 1 -- 172.21.15.40:0/3122167492 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c037070 con 0x7fe6a00730a0 2024-04-26T23:08:59.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.800+0000 7fe69ea00640 1 -- 172.21.15.40:0/3122167492 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe68c032440 con 0x7fe6a00730a0 2024-04-26T23:08:59.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.800+0000 7fe69ea00640 1 -- 172.21.15.40:0/3122167492 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c032810 con 0x7fe6a00730a0 2024-04-26T23:08:59.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.800+0000 7fe6a5400640 1 -- 172.21.15.40:0/3122167492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6a00730a0 msgr2=0x7fe6a00cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:59.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.800+0000 7fe6a5400640 1 --2- 172.21.15.40:0/3122167492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6a00730a0 0x7fe6a00cdf30 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0097a0 tx=0x7fe68c02a180 comp rx=0 tx=0).stop 2024-04-26T23:08:59.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.801+0000 7fe6a5400640 1 -- 172.21.15.40:0/3122167492 shutdown_connections 2024-04-26T23:08:59.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.801+0000 7fe6a5400640 1 --2- 172.21.15.40:0/3122167492 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe6a0071ef0 0x7fe6a00b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.801+0000 7fe6a5400640 1 --2- 172.21.15.40:0/3122167492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6a00730a0 0x7fe6a00cdf30 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.801+0000 7fe6a5400640 1 -- 172.21.15.40:0/3122167492 >> 172.21.15.40:0/3122167492 conn(0x7fe6a00b2720 msgr2=0x7fe6a00b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.801+0000 7fe6a5400640 1 -- 172.21.15.40:0/3122167492 shutdown_connections 2024-04-26T23:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.801+0000 7fe6a5400640 1 -- 172.21.15.40:0/3122167492 wait complete. 2024-04-26T23:08:59.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.803+0000 7fe6a5400640 1 Processor -- start 2024-04-26T23:08:59.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.803+0000 7fe6a5400640 1 -- start start 2024-04-26T23:08:59.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.804+0000 7fe6a5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6a0071ef0 0x7fe6a0149080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:59.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.804+0000 7fe6a5400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe6a00730a0 0x7fe6a0145c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:59.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.804+0000 7fe6a5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6a015bc70 0x7fe6a0159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:59.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.804+0000 7fe6a5400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a015c050 con 0x7fe6a015bc70 2024-04-26T23:08:59.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.804+0000 7fe6a5400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a006fad0 con 0x7fe6a0071ef0 2024-04-26T23:08:59.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.804+0000 7fe6a5400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe6a015c1d0 con 0x7fe6a00730a0 2024-04-26T23:08:59.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.804+0000 7fe6a4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6a015bc70 0x7fe6a0159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:59.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.804+0000 7fe6a4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6a015bc70 0x7fe6a0159e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43116/0 (socket says 172.21.15.40:43116) 2024-04-26T23:08:59.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.804+0000 7fe6a4a00640 1 -- 172.21.15.40:0/3639289537 learned_addr learned my addr 172.21.15.40:0/3639289537 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:08:59.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.804+0000 7fe69fe00640 1 --2- 172.21.15.40:0/3639289537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6a0071ef0 0x7fe6a0149080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:59.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.804+0000 7fe6a4a00640 1 -- 172.21.15.40:0/3639289537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe6a00730a0 msgr2=0x7fe6a0145c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:59.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.804+0000 7fe6a4a00640 1 --2- 172.21.15.40:0/3639289537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe6a00730a0 0x7fe6a0145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:59.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.804+0000 7fe6a4a00640 1 -- 172.21.15.40:0/3639289537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6a0071ef0 msgr2=0x7fe6a0149080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:08:59.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.805+0000 7fe6a4a00640 1 --2- 172.21.15.40:0/3639289537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6a0071ef0 0x7fe6a0149080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:08:59.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.805+0000 7fe6a4a00640 1 -- 172.21.15.40:0/3639289537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a0146140 con 0x7fe6a015bc70 2024-04-26T23:08:59.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.805+0000 7fe6a4a00640 1 --2- 172.21.15.40:0/3639289537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6a015bc70 0x7fe6a0159e20 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fe69800cc60 tx=0x7fe69800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:59.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.805+0000 7fe697e00640 1 -- 172.21.15.40:0/3639289537 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698003990 con 0x7fe6a015bc70 2024-04-26T23:08:59.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.805+0000 7fe697e00640 1 -- 172.21.15.40:0/3639289537 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe698003b30 con 0x7fe6a015bc70 2024-04-26T23:08:59.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.805+0000 7fe697e00640 1 -- 172.21.15.40:0/3639289537 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698004020 con 0x7fe6a015bc70 2024-04-26T23:08:59.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.805+0000 7fe6a5400640 1 -- 172.21.15.40:0/3639289537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6a015a770 con 0x7fe6a015bc70 2024-04-26T23:08:59.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.805+0000 7fe6a5400640 1 -- 172.21.15.40:0/3639289537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6a015ad30 con 0x7fe6a015bc70 2024-04-26T23:08:59.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.806+0000 7fe6a5400640 1 -- 172.21.15.40:0/3639289537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6a0069370 con 0x7fe6a015bc70 2024-04-26T23:08:59.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.807+0000 7fe697e00640 1 -- 172.21.15.40:0/3639289537 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe6980041c0 con 0x7fe6a015bc70 2024-04-26T23:08:59.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.808+0000 7fe697e00640 1 --2- 172.21.15.40:0/3639289537 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe68006c8f0 0x7fe68006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:08:59.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.809+0000 7fe697e00640 1 -- 172.21.15.40:0/3639289537 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5507+0+0 (secure 0 0 0) 0x7fe69808bfd0 con 0x7fe6a015bc70 2024-04-26T23:08:59.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.809+0000 7fe69fe00640 1 --2- 172.21.15.40:0/3639289537 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe68006c8f0 0x7fe68006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:08:59.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.810+0000 7fe69fe00640 1 --2- 172.21.15.40:0/3639289537 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe68006c8f0 0x7fe68006edb0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe690000f60 tx=0x7fe690009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:08:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:08:59.813+0000 7fe697e00640 1 -- 172.21.15.40:0/3639289537 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe698014030 con 0x7fe6a015bc70 2024-04-26T23:09:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.016+0000 7fe6a5400640 1 -- 172.21.15.40:0/3639289537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe6a00b5860 con 0x7fe6a015bc70 2024-04-26T23:09:00.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.024+0000 7fe697e00640 1 -- 172.21.15.40:0/3639289537 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) ==== 122+0+0 (secure 0 0 0) 0x7fe69805a420 con 0x7fe6a015bc70 2024-04-26T23:09:00.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.026+0000 7fe6a5400640 1 -- 172.21.15.40:0/3639289537 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe68006c8f0 msgr2=0x7fe68006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:00.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.026+0000 7fe6a5400640 1 --2- 172.21.15.40:0/3639289537 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe68006c8f0 0x7fe68006edb0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe690000f60 tx=0x7fe690009040 comp rx=0 tx=0).stop 2024-04-26T23:09:00.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.027+0000 7fe6a5400640 1 -- 172.21.15.40:0/3639289537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6a015bc70 msgr2=0x7fe6a0159e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:00.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.027+0000 7fe6a5400640 1 --2- 172.21.15.40:0/3639289537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6a015bc70 0x7fe6a0159e20 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fe69800cc60 tx=0x7fe69800a5c0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.027+0000 7fe6a5400640 1 -- 172.21.15.40:0/3639289537 shutdown_connections 2024-04-26T23:09:00.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.027+0000 7fe6a5400640 1 --2- 172.21.15.40:0/3639289537 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe68006c8f0 0x7fe68006edb0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.027+0000 7fe6a5400640 1 --2- 172.21.15.40:0/3639289537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6a015bc70 0x7fe6a0159e20 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.027+0000 7fe6a5400640 1 --2- 172.21.15.40:0/3639289537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe6a00730a0 0x7fe6a0145c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.027+0000 7fe6a5400640 1 --2- 172.21.15.40:0/3639289537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6a0071ef0 0x7fe6a0149080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.027+0000 7fe6a5400640 1 -- 172.21.15.40:0/3639289537 >> 172.21.15.40:0/3639289537 conn(0x7fe6a00b2720 msgr2=0x7fe6a00af250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:00.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.027+0000 7fe6a5400640 1 -- 172.21.15.40:0/3639289537 shutdown_connections 2024-04-26T23:09:00.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.027+0000 7fe6a5400640 1 -- 172.21.15.40:0/3639289537 wait complete. 2024-04-26T23:09:00.045 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-26T23:09:00.046 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:09:00.153 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:09:00.254 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-26T23:09:00.254 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:09:00.255 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:09:00.317 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:09:00.317 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:09:00.318 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:09:00.375 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:09:00.375 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:09:00.376 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:09:00.436 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:09:00.437 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:09:00.437 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:09:00.494 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:09:00.494 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-26T23:09:00.494 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:00.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.712+0000 7f9182800640 1 Processor -- start 2024-04-26T23:09:00.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.712+0000 7f9182800640 1 -- start start 2024-04-26T23:09:00.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.713+0000 7f9182800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f917c071e60 0x7f917c06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:00.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.713+0000 7f9182800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f917c06fe20 0x7f917c072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:00.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f9182800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c072780 0x7f917c0cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:00.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f9182800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f917c15d7c0 con 0x7f917c071e60 2024-04-26T23:09:00.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f9182800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f917c15daa0 con 0x7f917c06fe20 2024-04-26T23:09:00.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f9182800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f917c15dd80 con 0x7f917c072780 2024-04-26T23:09:00.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f917b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f917c071e60 0x7f917c06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:00.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f917b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f917c071e60 0x7f917c06f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43124/0 (socket says 172.21.15.40:43124) 2024-04-26T23:09:00.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f917b400640 1 -- 172.21.15.40:0/2468086278 learned_addr learned my addr 172.21.15.40:0/2468086278 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:00.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f917be00640 1 --2- 172.21.15.40:0/2468086278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c072780 0x7f917c0cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:00.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f917aa00640 1 --2- 172.21.15.40:0/2468086278 >> v2:172.21.15.103:3300/0 conn(0x7f917c06fe20 0x7f917c072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:00.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f917b400640 1 -- 172.21.15.40:0/2468086278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c072780 msgr2=0x7f917c0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:00.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f917b400640 1 --2- 172.21.15.40:0/2468086278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c072780 0x7f917c0cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f917b400640 1 -- 172.21.15.40:0/2468086278 >> v2:172.21.15.103:3300/0 conn(0x7f917c06fe20 msgr2=0x7f917c072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:00.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f917b400640 1 --2- 172.21.15.40:0/2468086278 >> v2:172.21.15.103:3300/0 conn(0x7f917c06fe20 0x7f917c072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.714+0000 7f917b400640 1 -- 172.21.15.40:0/2468086278 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c15e060 con 0x7f917c071e60 2024-04-26T23:09:00.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.716+0000 7f917b400640 1 --2- 172.21.15.40:0/2468086278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c071e60 0x7f917c06f8e0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f917c0b3820 tx=0x7f916c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af53e15c9aa9842 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:00.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.716+0000 7f917a000640 1 -- 172.21.15.40:0/2468086278 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c037070 con 0x7f917c071e60 2024-04-26T23:09:00.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.716+0000 7f917a000640 1 -- 172.21.15.40:0/2468086278 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f916c032440 con 0x7f917c071e60 2024-04-26T23:09:00.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.716+0000 7f917a000640 1 -- 172.21.15.40:0/2468086278 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c032810 con 0x7f917c071e60 2024-04-26T23:09:00.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.717+0000 7f9182800640 1 -- 172.21.15.40:0/2468086278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c071e60 msgr2=0x7f917c06f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:00.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.717+0000 7f9182800640 1 --2- 172.21.15.40:0/2468086278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c071e60 0x7f917c06f8e0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f917c0b3820 tx=0x7f916c02a1e0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.717+0000 7f9182800640 1 -- 172.21.15.40:0/2468086278 shutdown_connections 2024-04-26T23:09:00.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.717+0000 7f9182800640 1 --2- 172.21.15.40:0/2468086278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c072780 0x7f917c0cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.717+0000 7f9182800640 1 --2- 172.21.15.40:0/2468086278 >> v2:172.21.15.103:3300/0 conn(0x7f917c06fe20 0x7f917c072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.717+0000 7f9182800640 1 --2- 172.21.15.40:0/2468086278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c071e60 0x7f917c06f8e0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.717+0000 7f9182800640 1 -- 172.21.15.40:0/2468086278 >> 172.21.15.40:0/2468086278 conn(0x7f917c0d0be0 msgr2=0x7f917c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:00.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.717+0000 7f9182800640 1 -- 172.21.15.40:0/2468086278 shutdown_connections 2024-04-26T23:09:00.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.717+0000 7f9182800640 1 -- 172.21.15.40:0/2468086278 wait complete. 2024-04-26T23:09:00.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.719+0000 7f9182800640 1 Processor -- start 2024-04-26T23:09:00.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.719+0000 7f9182800640 1 -- start start 2024-04-26T23:09:00.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.720+0000 7f9182800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f917c06fe20 0x7f917c15c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:00.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.720+0000 7f9182800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c071e60 0x7f917c150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:00.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.720+0000 7f9182800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c072780 0x7f917c15a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:00.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.720+0000 7f9182800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f917c15d3f0 con 0x7f917c072780 2024-04-26T23:09:00.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.720+0000 7f9182800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f917c072c10 con 0x7f917c06fe20 2024-04-26T23:09:00.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.720+0000 7f9182800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f917c15d570 con 0x7f917c071e60 2024-04-26T23:09:00.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.721+0000 7f917be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c072780 0x7f917c15a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:00.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.721+0000 7f917be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c072780 0x7f917c15a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43128/0 (socket says 172.21.15.40:43128) 2024-04-26T23:09:00.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.721+0000 7f917be00640 1 -- 172.21.15.40:0/2584808374 learned_addr learned my addr 172.21.15.40:0/2584808374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:00.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.721+0000 7f917aa00640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c071e60 0x7f917c150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:00.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.721+0000 7f917b400640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f917c06fe20 0x7f917c15c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:00.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.721+0000 7f917be00640 1 -- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c071e60 msgr2=0x7f917c150cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:00.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.721+0000 7f917be00640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c071e60 0x7f917c150cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.721+0000 7f917be00640 1 -- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f917c06fe20 msgr2=0x7f917c15c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:00.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.721+0000 7f917be00640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f917c06fe20 0x7f917c15c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.721+0000 7f917be00640 1 -- 172.21.15.40:0/2584808374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c15b3b0 con 0x7f917c072780 2024-04-26T23:09:00.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.722+0000 7f917be00640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c072780 0x7f917c15a940 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f917400cc30 tx=0x7f917400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:00.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.722+0000 7f9173400640 1 -- 172.21.15.40:0/2584808374 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917400ce40 con 0x7f917c072780 2024-04-26T23:09:00.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.722+0000 7f9173400640 1 -- 172.21.15.40:0/2584808374 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9174003990 con 0x7f917c072780 2024-04-26T23:09:00.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.722+0000 7f9173400640 1 -- 172.21.15.40:0/2584808374 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9174003d40 con 0x7f917c072780 2024-04-26T23:09:00.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.722+0000 7f9182800640 1 -- 172.21.15.40:0/2584808374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f917c15b690 con 0x7f917c072780 2024-04-26T23:09:00.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.722+0000 7f9182800640 1 -- 172.21.15.40:0/2584808374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f917c15bc50 con 0x7f917c072780 2024-04-26T23:09:00.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.723+0000 7f9182800640 1 -- 172.21.15.40:0/2584808374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f917c0694f0 con 0x7f917c072780 2024-04-26T23:09:00.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.724+0000 7f9173400640 1 -- 172.21.15.40:0/2584808374 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9174003ee0 con 0x7f917c072780 2024-04-26T23:09:00.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.725+0000 7f9173400640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f915806c8f0 0x7f915806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:00.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.725+0000 7f9173400640 1 -- 172.21.15.40:0/2584808374 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5507+0+0 (secure 0 0 0) 0x7f917408bfb0 con 0x7f917c072780 2024-04-26T23:09:00.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.726+0000 7f917b400640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f915806c8f0 0x7f915806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:00.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.727+0000 7f917b400640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f915806c8f0 0x7f915806edb0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f916c034000 tx=0x7f916c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:00.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.730+0000 7f9173400640 1 -- 172.21.15.40:0/2584808374 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9174014030 con 0x7f917c072780 2024-04-26T23:09:00.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.979+0000 7f9182800640 1 -- 172.21.15.40:0/2584808374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f917c0cc6a0 con 0x7f917c072780 2024-04-26T23:09:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.980+0000 7f9173400640 1 -- 172.21.15.40:0/2584808374 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) ==== 95+0+3879 (secure 0 0 0) 0x7f917405a400 con 0x7f917c072780 2024-04-26T23:09:00.981 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:00.981 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":104,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3701451901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3701451901},{"type":"v1","addr":"172.21.15.103:6835","nonce":3701451901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3595679441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3595679441},{"type":"v1","addr":"172.21.15.103:6837","nonce":3595679441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104},{"gid":8856,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1660223851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1660223851},{"type":"v1","addr":"172.21.15.40:6833","nonce":1660223851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104},{"gid":8865,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2139196513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2139196513},{"type":"v1","addr":"172.21.15.40:6835","nonce":2139196513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104}],"filesystems":[]} 2024-04-26T23:09:00.981 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 104 2024-04-26T23:09:00.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.983+0000 7f9182800640 1 -- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f915806c8f0 msgr2=0x7f915806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:00.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.983+0000 7f9182800640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f915806c8f0 0x7f915806edb0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f916c034000 tx=0x7f916c02e000 comp rx=0 tx=0).stop 2024-04-26T23:09:00.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.983+0000 7f9182800640 1 -- 172.21.15.40:0/2584808374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c072780 msgr2=0x7f917c15a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:00.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.983+0000 7f9182800640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c072780 0x7f917c15a940 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f917400cc30 tx=0x7f917400a5c0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.983+0000 7f9182800640 1 -- 172.21.15.40:0/2584808374 shutdown_connections 2024-04-26T23:09:00.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.983+0000 7f9182800640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f915806c8f0 0x7f915806edb0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.983+0000 7f9182800640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f917c072780 0x7f917c15a940 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.983+0000 7f9182800640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c071e60 0x7f917c150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.983+0000 7f9182800640 1 --2- 172.21.15.40:0/2584808374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f917c06fe20 0x7f917c15c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:00.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.983+0000 7f9182800640 1 -- 172.21.15.40:0/2584808374 >> 172.21.15.40:0/2584808374 conn(0x7f917c0d0be0 msgr2=0x7f917c0ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:00.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.983+0000 7f9182800640 1 -- 172.21.15.40:0/2584808374 shutdown_connections 2024-04-26T23:09:00.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:00.984+0000 7f9182800640 1 -- 172.21.15.40:0/2584808374 wait complete. 2024-04-26T23:09:01.002 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-26T23:09:01.003 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-26T23:09:01.003 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-26T23:09:01.003 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:09:01.003 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-26T23:09:01.003 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-26T23:09:01.003 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-26T23:09:01.003 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:09:01.004 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-26T23:09:01.004 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-26T23:09:01.004 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-26T23:09:01.004 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:09:01.004 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-26T23:09:01.004 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-26T23:09:01.004 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-26T23:09:01.005 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:09:01.020 INFO:tasks.ceph.mds.c:Stopped 2024-04-26T23:09:01.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-26T23:09:01.021 INFO:tasks.ceph.mds.a:Stopped 2024-04-26T23:09:01.021 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-26T23:09:01.026 INFO:tasks.ceph.mds.d:Stopped 2024-04-26T23:09:01.026 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-26T23:09:01.030 INFO:tasks.ceph.mds.b:Stopped 2024-04-26T23:09:01.030 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-26T23:09:01.068 INFO:tasks.ceph.mds.c:Started 2024-04-26T23:09:01.068 DEBUG:teuthology.parallel:result is None 2024-04-26T23:09:01.074 INFO:tasks.ceph.mds.a:Started 2024-04-26T23:09:01.074 DEBUG:teuthology.parallel:result is None 2024-04-26T23:09:01.076 INFO:tasks.ceph.mds.d:Started 2024-04-26T23:09:01.077 DEBUG:teuthology.parallel:result is None 2024-04-26T23:09:01.083 INFO:tasks.ceph.mds.b:Started 2024-04-26T23:09:01.083 DEBUG:teuthology.parallel:result is None 2024-04-26T23:09:01.083 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-26T23:09:01.218 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:09:01.222 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-04-26T23:09:01.233 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-26T23:09:01.233 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-26T23:09:01.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.269+0000 7fcd8be00640 1 Processor -- start 2024-04-26T23:09:01.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.269+0000 7fcd8be00640 1 -- start start 2024-04-26T23:09:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.270+0000 7fcd8be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd84163610 0x7fcd840d5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.270+0000 7fcd8be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd840d2830 0x7fcd840cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.270+0000 7fcd8be00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd840d1a70 con 0x7fcd8415f5a0 2024-04-26T23:09:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.270+0000 7fcd8be00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcd84163d30 con 0x7fcd840d2830 2024-04-26T23:09:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.270+0000 7fcd8be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcd840d1d50 con 0x7fcd84163610 2024-04-26T23:09:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.271+0000 7fcd89400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcd8415f5a0 0x7fcd84162f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45196/0 (socket says 172.21.15.40:45196) 2024-04-26T23:09:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.271+0000 7fcd89400640 1 -- 172.21.15.40:0/2014300301 learned_addr learned my addr 172.21.15.40:0/2014300301 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.271+0000 7fcd89e00640 1 --2- 172.21.15.40:0/2014300301 >> v2:172.21.15.103:3300/0 conn(0x7fcd840d2830 0x7fcd840cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.271+0000 7fcd88a00640 1 --2- 172.21.15.40:0/2014300301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd84163610 0x7fcd840d5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:01.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.271+0000 7fcd83e00640 1 -- 172.21.15.40:0/2014300301 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3461732159 0 0) 0x7fcd840d1a70 con 0x7fcd8415f5a0 2024-04-26T23:09:01.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.271+0000 7fcd89e00640 1 -- 172.21.15.40:0/2014300301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd84163610 msgr2=0x7fcd840d5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:01.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.271+0000 7fcd89e00640 1 --2- 172.21.15.40:0/2014300301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd84163610 0x7fcd840d5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.271+0000 7fcd89e00640 1 -- 172.21.15.40:0/2014300301 >> v1:172.21.15.40:6789/0 conn(0x7fcd8415f5a0 legacy=0x7fcd84162f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:01.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.271+0000 7fcd89e00640 1 -- 172.21.15.40:0/2014300301 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd840d2030 con 0x7fcd840d2830 2024-04-26T23:09:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.272+0000 7fcd89e00640 1 --2- 172.21.15.40:0/2014300301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd840d2830 0x7fcd840cfda0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c009d90 tx=0x7fcd7c02a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1a5b411250e89eb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.273+0000 7fcd83e00640 1 -- 172.21.15.40:0/2014300301 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c031600 con 0x7fcd840d2830 2024-04-26T23:09:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.273+0000 7fcd83e00640 1 -- 172.21.15.40:0/2014300301 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd7c0317a0 con 0x7fcd840d2830 2024-04-26T23:09:01.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.273+0000 7fcd83e00640 1 -- 172.21.15.40:0/2014300301 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c031b70 con 0x7fcd840d2830 2024-04-26T23:09:01.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.273+0000 7fcd8be00640 1 -- 172.21.15.40:0/2014300301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd840d2830 msgr2=0x7fcd840cfda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:01.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.273+0000 7fcd8be00640 1 --2- 172.21.15.40:0/2014300301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd840d2830 0x7fcd840cfda0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c009d90 tx=0x7fcd7c02a5d0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.273+0000 7fcd8be00640 1 -- 172.21.15.40:0/2014300301 shutdown_connections 2024-04-26T23:09:01.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.273+0000 7fcd8be00640 1 --2- 172.21.15.40:0/2014300301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd840d2830 0x7fcd840cfda0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.273+0000 7fcd8be00640 1 --2- 172.21.15.40:0/2014300301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd84163610 0x7fcd840d5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.273+0000 7fcd8be00640 1 -- 172.21.15.40:0/2014300301 >> 172.21.15.40:0/2014300301 conn(0x7fcd840d5190 msgr2=0x7fcd840d55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:01.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.273+0000 7fcd8be00640 1 -- 172.21.15.40:0/2014300301 shutdown_connections 2024-04-26T23:09:01.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.273+0000 7fcd8be00640 1 -- 172.21.15.40:0/2014300301 wait complete. 2024-04-26T23:09:01.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.275+0000 7fcd8be00640 1 Processor -- start 2024-04-26T23:09:01.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.275+0000 7fcd8be00640 1 -- start start 2024-04-26T23:09:01.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.276+0000 7fcd8be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd840d2830 0x7fcd8415a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:01.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.276+0000 7fcd8be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd84163610 0x7fcd841f85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:01.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.276+0000 7fcd8be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd840cb780 0x7fcd840ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:01.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.276+0000 7fcd8be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcd840d3c20 con 0x7fcd840d2830 2024-04-26T23:09:01.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.276+0000 7fcd8be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcd840d4190 con 0x7fcd84163610 2024-04-26T23:09:01.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.276+0000 7fcd8be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcd840d3da0 con 0x7fcd840cb780 2024-04-26T23:09:01.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.277+0000 7fcd89400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd840d2830 0x7fcd8415a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:01.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.277+0000 7fcd89400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd840d2830 0x7fcd8415a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43148/0 (socket says 172.21.15.40:43148) 2024-04-26T23:09:01.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.277+0000 7fcd89400640 1 -- 172.21.15.40:0/3581662238 learned_addr learned my addr 172.21.15.40:0/3581662238 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.277+0000 7fcd89e00640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd840cb780 0x7fcd840ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.277+0000 7fcd88a00640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd84163610 0x7fcd841f85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.277+0000 7fcd89400640 1 -- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd840cb780 msgr2=0x7fcd840ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.277+0000 7fcd89400640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd840cb780 0x7fcd840ccdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.277+0000 7fcd89400640 1 -- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd84163610 msgr2=0x7fcd841f85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.277+0000 7fcd89400640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd84163610 0x7fcd841f85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.277+0000 7fcd89400640 1 -- 172.21.15.40:0/3581662238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd8415ac30 con 0x7fcd840d2830 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.278+0000 7fcd89400640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd840d2830 0x7fcd8415a6f0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800c4d0 tx=0x7fcd7800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.278+0000 7fcd81600640 1 -- 172.21.15.40:0/3581662238 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd78018070 con 0x7fcd840d2830 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.278+0000 7fcd81600640 1 -- 172.21.15.40:0/3581662238 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd78007500 con 0x7fcd840d2830 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.278+0000 7fcd81600640 1 -- 172.21.15.40:0/3581662238 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd78004a20 con 0x7fcd840d2830 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.278+0000 7fcd8be00640 1 -- 172.21.15.40:0/3581662238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd841f8df0 con 0x7fcd840d2830 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.278+0000 7fcd8be00640 1 -- 172.21.15.40:0/3581662238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd840b7e20 con 0x7fcd840d2830 2024-04-26T23:09:01.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.279+0000 7fcd8be00640 1 -- 172.21.15.40:0/3581662238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd840694f0 con 0x7fcd840d2830 2024-04-26T23:09:01.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.280+0000 7fcd81600640 1 -- 172.21.15.40:0/3581662238 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcd78004bc0 con 0x7fcd840d2830 2024-04-26T23:09:01.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.281+0000 7fcd81600640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd6406c920 0x7fcd6406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.281+0000 7fcd81600640 1 -- 172.21.15.40:0/3581662238 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5507+0+0 (secure 0 0 0) 0x7fcd7808acc0 con 0x7fcd840d2830 2024-04-26T23:09:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.282+0000 7fcd88a00640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd6406c920 0x7fcd6406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:01.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.282+0000 7fcd88a00640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd6406c920 0x7fcd6406ede0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fcd840d11d0 tx=0x7fcd70009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:01.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.286+0000 7fcd81600640 1 -- 172.21.15.40:0/3581662238 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fcd78091080 con 0x7fcd840d2830 2024-04-26T23:09:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.506+0000 7fcd8be00640 1 -- 172.21.15.40:0/3581662238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fcd84068d30 con 0x7fcd840d2830 2024-04-26T23:09:01.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.575+0000 7fcd81600640 1 -- 172.21.15.40:0/3581662238 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v97) ==== 97+0+0 (secure 0 0 0) 0x7fcd8415ac30 con 0x7fcd840d2830 2024-04-26T23:09:01.576 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-04-26T23:09:01.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.579+0000 7fcd8be00640 1 -- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd6406c920 msgr2=0x7fcd6406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:01.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.579+0000 7fcd8be00640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd6406c920 0x7fcd6406ede0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fcd840d11d0 tx=0x7fcd70009190 comp rx=0 tx=0).stop 2024-04-26T23:09:01.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.579+0000 7fcd8be00640 1 -- 172.21.15.40:0/3581662238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd840d2830 msgr2=0x7fcd8415a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:01.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.579+0000 7fcd8be00640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd840d2830 0x7fcd8415a6f0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800c4d0 tx=0x7fcd7800c990 comp rx=0 tx=0).stop 2024-04-26T23:09:01.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.579+0000 7fcd8be00640 1 -- 172.21.15.40:0/3581662238 shutdown_connections 2024-04-26T23:09:01.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.579+0000 7fcd8be00640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcd6406c920 0x7fcd6406ede0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.579+0000 7fcd8be00640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd840cb780 0x7fcd840ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.579+0000 7fcd8be00640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd84163610 0x7fcd841f85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.579+0000 7fcd8be00640 1 --2- 172.21.15.40:0/3581662238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd840d2830 0x7fcd8415a6f0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.579+0000 7fcd8be00640 1 -- 172.21.15.40:0/3581662238 >> 172.21.15.40:0/3581662238 conn(0x7fcd840d5190 msgr2=0x7fcd8406e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:01.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.579+0000 7fcd8be00640 1 -- 172.21.15.40:0/3581662238 shutdown_connections 2024-04-26T23:09:01.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.580+0000 7fcd8be00640 1 -- 172.21.15.40:0/3581662238 wait complete. 2024-04-26T23:09:01.599 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-26T23:09:01.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.797+0000 7f916e000640 1 Processor -- start 2024-04-26T23:09:01.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.797+0000 7f916e000640 1 -- start start 2024-04-26T23:09:01.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.798+0000 7f916e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9168163610 0x7f91680d5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:01.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.798+0000 7f916e000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f91680d2830 0x7f91680cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:01.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.798+0000 7f916e000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91680d1a70 con 0x7f916815f5a0 2024-04-26T23:09:01.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.798+0000 7f916e000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f9168163d30 con 0x7f9168163610 2024-04-26T23:09:01.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.798+0000 7f916e000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f91680d1d50 con 0x7f91680d2830 2024-04-26T23:09:01.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.798+0000 7f9166a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9168163610 0x7f91680d5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:01.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.798+0000 7f9166a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9168163610 0x7f91680d5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43164/0 (socket says 172.21.15.40:43164) 2024-04-26T23:09:01.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.798+0000 7f9166a00640 1 -- 172.21.15.40:0/1907876175 learned_addr learned my addr 172.21.15.40:0/1907876175 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:01.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.799+0000 7f9166a00640 1 -- 172.21.15.40:0/1907876175 >> v2:172.21.15.103:3300/0 conn(0x7f91680d2830 msgr2=0x7f91680cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:01.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.799+0000 7f9166a00640 1 --2- 172.21.15.40:0/1907876175 >> v2:172.21.15.103:3300/0 conn(0x7f91680d2830 0x7f91680cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.799+0000 7f9166a00640 1 -- 172.21.15.40:0/1907876175 >> v1:172.21.15.103:6789/0 conn(0x7f916815f5a0 legacy=0x7f9168162f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:01.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.799+0000 7f9166a00640 1 -- 172.21.15.40:0/1907876175 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91680d2030 con 0x7f9168163610 2024-04-26T23:09:01.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.800+0000 7f9166a00640 1 --2- 172.21.15.40:0/1907876175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9168163610 0x7f91680d5cb0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f916808ea00 tx=0x7f915402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597f0275dcfa7a82 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:01.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.800+0000 7f9166000640 1 -- 172.21.15.40:0/1907876175 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9154036070 con 0x7f9168163610 2024-04-26T23:09:01.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.800+0000 7f9166000640 1 -- 172.21.15.40:0/1907876175 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9154031600 con 0x7f9168163610 2024-04-26T23:09:01.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.800+0000 7f9166000640 1 -- 172.21.15.40:0/1907876175 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91540319d0 con 0x7f9168163610 2024-04-26T23:09:01.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.801+0000 7f916e000640 1 -- 172.21.15.40:0/1907876175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9168163610 msgr2=0x7f91680d5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:01.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.801+0000 7f916e000640 1 --2- 172.21.15.40:0/1907876175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9168163610 0x7f91680d5cb0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f916808ea00 tx=0x7f915402a660 comp rx=0 tx=0).stop 2024-04-26T23:09:01.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.801+0000 7f916e000640 1 -- 172.21.15.40:0/1907876175 shutdown_connections 2024-04-26T23:09:01.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.801+0000 7f916e000640 1 --2- 172.21.15.40:0/1907876175 >> v2:172.21.15.103:3300/0 conn(0x7f91680d2830 0x7f91680cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.801+0000 7f916e000640 1 --2- 172.21.15.40:0/1907876175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9168163610 0x7f91680d5cb0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.801+0000 7f916e000640 1 -- 172.21.15.40:0/1907876175 >> 172.21.15.40:0/1907876175 conn(0x7f91680d5190 msgr2=0x7f91680d55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:01.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.801+0000 7f916e000640 1 -- 172.21.15.40:0/1907876175 shutdown_connections 2024-04-26T23:09:01.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.801+0000 7f916e000640 1 -- 172.21.15.40:0/1907876175 wait complete. 2024-04-26T23:09:01.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.803+0000 7f916e000640 1 Processor -- start 2024-04-26T23:09:01.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.803+0000 7f916e000640 1 -- start start 2024-04-26T23:09:01.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.804+0000 7f916e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91680d2830 0x7f916814af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:01.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.804+0000 7f916e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9168163610 0x7f91681613f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:01.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.804+0000 7f916e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f916815e800 0x7f916814d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:01.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.804+0000 7f916e000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f91680d3c20 con 0x7f91680d2830 2024-04-26T23:09:01.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.804+0000 7f916e000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f91680d4190 con 0x7f9168163610 2024-04-26T23:09:01.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.804+0000 7f916e000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f91680d3da0 con 0x7f916815e800 2024-04-26T23:09:01.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.804+0000 7f9167400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91680d2830 0x7f916814af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:01.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.804+0000 7f9167400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91680d2830 0x7f916814af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43172/0 (socket says 172.21.15.40:43172) 2024-04-26T23:09:01.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.804+0000 7f9167400640 1 -- 172.21.15.40:0/731493256 learned_addr learned my addr 172.21.15.40:0/731493256 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:01.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.805+0000 7f9167400640 1 -- 172.21.15.40:0/731493256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f916815e800 msgr2=0x7f916814d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:01.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.805+0000 7f9167400640 1 --2- 172.21.15.40:0/731493256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f916815e800 0x7f916814d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.805+0000 7f9167400640 1 -- 172.21.15.40:0/731493256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9168163610 msgr2=0x7f91681613f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:01.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.805+0000 7f9167400640 1 --2- 172.21.15.40:0/731493256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9168163610 0x7f91681613f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:01.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.805+0000 7f9167400640 1 -- 172.21.15.40:0/731493256 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91681f50a0 con 0x7f91680d2830 2024-04-26T23:09:01.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.805+0000 7f9167400640 1 --2- 172.21.15.40:0/731493256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91680d2830 0x7f916814af10 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f915800c410 tx=0x7f915800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:01.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.805+0000 7f915f400640 1 -- 172.21.15.40:0/731493256 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9158017070 con 0x7f91680d2830 2024-04-26T23:09:01.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.805+0000 7f915f400640 1 -- 172.21.15.40:0/731493256 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91580035f0 con 0x7f91680d2830 2024-04-26T23:09:01.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.805+0000 7f915f400640 1 -- 172.21.15.40:0/731493256 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9158003a20 con 0x7f91680d2830 2024-04-26T23:09:01.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.805+0000 7f916e000640 1 -- 172.21.15.40:0/731493256 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9168161d60 con 0x7f91680d2830 2024-04-26T23:09:01.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.806+0000 7f916e000640 1 -- 172.21.15.40:0/731493256 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f916814b6e0 con 0x7f91680d2830 2024-04-26T23:09:01.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.807+0000 7f916e000640 1 -- 172.21.15.40:0/731493256 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91680694f0 con 0x7f91680d2830 2024-04-26T23:09:01.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.808+0000 7f915f400640 1 -- 172.21.15.40:0/731493256 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9158003bc0 con 0x7f91680d2830 2024-04-26T23:09:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.809+0000 7f915f400640 1 --2- 172.21.15.40:0/731493256 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f914806c8f0 0x7f914806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.810+0000 7f915f400640 1 -- 172.21.15.40:0/731493256 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5249+0+0 (secure 0 0 0) 0x7f915808acc0 con 0x7f91680d2830 2024-04-26T23:09:01.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.810+0000 7f9166a00640 1 --2- 172.21.15.40:0/731493256 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f914806c8f0 0x7f914806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:01.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.811+0000 7f9166a00640 1 --2- 172.21.15.40:0/731493256 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f914806c8f0 0x7f914806edb0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9154033000 tx=0x7f915402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:01.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:01.814+0000 7f915f400640 1 -- 172.21.15.40:0/731493256 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9158091080 con 0x7f91680d2830 2024-04-26T23:09:02.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.092+0000 7f916e000640 1 -- 172.21.15.40:0/731493256 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f91680bb110 con 0x7f91680d2830 2024-04-26T23:09:02.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.093+0000 7f915f400640 1 -- 172.21.15.40:0/731493256 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) ==== 80+0+5304 (secure 0 0 0) 0x7f91580592a0 con 0x7f91680d2830 2024-04-26T23:09:02.094 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:02.094 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:02.094 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-04-26T23:09:02.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.094 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmF/UQHBAACdb8iskp2MuBMV6S5tYFJg==", 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmniCyMxAAT/rVbIKj9EnvwDiblKMCeg==", 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:09:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmLGZhJxAAg/fsrRrNoF/Jx1PnBVjjeQ==", 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxms3DnAxAAxJj/rybrv/0cLK8PmQ619A==", 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDrMSxmbeONCxAATB67ujBDn2tVxpbMju7Mfg==", 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDuMSxmqE/PJhAAsNNu5ar8tHWQauxaXOvagA==", 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDyMSxmd2trBhAAbnl7EBldxgtOd1s0Ll0A9w==", 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD1MSxmqdtKEhAA9oxBxOu/musCKyladrJgYA==", 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:09:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD7MSxm+kl7IxAAlPAGIemdC8gcvp+koeULaQ==", 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-04-26T23:09:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQD+MSxmzKiaLRAA2UPS1wwJHgMxQf/zLMjMVQ==", 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQACMixmluHrCxAAmbmb33f5MVSTWNuy1D0Zsg==", 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-04-26T23:09:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAFMixm6GQjJRAArmdnX7ioekT8GSdsbeB1Ow==", 2024-04-26T23:09:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-04-26T23:09:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-04-26T23:09:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-04-26T23:09:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg==", 2024-04-26T23:09:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow r", 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmVhn4IhAA94XsDk1x5Oax5smXPoFQKg==", 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:09:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxm3qhbGRAAK62blgPVShIYskDsMcaffQ==", 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDnMSxmxof/KxAAUmXVV4Vdp2DZnRjQzQ0Inw==", 2024-04-26T23:09:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDlMSxmmVwcGRAAqgHymPPTPuS6XXC6LuOVPg==", 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmVf+9EhAAcpk8NgFqgCLXeHA2z7oHmA==", 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQDmMSxmjjpcCBAAjDQRUBn/TK9hF3T2WBObyw==", 2024-04-26T23:09:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-04-26T23:09:02.107 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-04-26T23:09:02.107 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-04-26T23:09:02.107 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-04-26T23:09:02.107 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.107 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:02.107 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-26T23:09:02.107 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.096+0000 7f916e000640 1 -- 172.21.15.40:0/731493256 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f914806c8f0 msgr2=0x7f914806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.096+0000 7f916e000640 1 --2- 172.21.15.40:0/731493256 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f914806c8f0 0x7f914806edb0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9154033000 tx=0x7f915402e000 comp rx=0 tx=0).stop 2024-04-26T23:09:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.096+0000 7f916e000640 1 -- 172.21.15.40:0/731493256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91680d2830 msgr2=0x7f916814af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.096+0000 7f916e000640 1 --2- 172.21.15.40:0/731493256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91680d2830 0x7f916814af10 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f915800c410 tx=0x7f915800c8d0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.096+0000 7f916e000640 1 -- 172.21.15.40:0/731493256 shutdown_connections 2024-04-26T23:09:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.096+0000 7f916e000640 1 --2- 172.21.15.40:0/731493256 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f914806c8f0 0x7f914806edb0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.096+0000 7f916e000640 1 --2- 172.21.15.40:0/731493256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f916815e800 0x7f916814d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.096+0000 7f916e000640 1 --2- 172.21.15.40:0/731493256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9168163610 0x7f91681613f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.096+0000 7f916e000640 1 --2- 172.21.15.40:0/731493256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91680d2830 0x7f916814af10 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.096+0000 7f916e000640 1 -- 172.21.15.40:0/731493256 >> 172.21.15.40:0/731493256 conn(0x7f91680d5190 msgr2=0x7f91680cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.096+0000 7f916e000640 1 -- 172.21.15.40:0/731493256 shutdown_connections 2024-04-26T23:09:02.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.096+0000 7f916e000640 1 -- 172.21.15.40:0/731493256 wait complete. 2024-04-26T23:09:02.115 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:09:02.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.330+0000 7f0de6c00640 1 Processor -- start 2024-04-26T23:09:02.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.331+0000 7f0de6c00640 1 -- start start 2024-04-26T23:09:02.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.332+0000 7f0de6c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0de00d32e0 0x7f0de00d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:02.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.332+0000 7f0de6c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de0162c40 con 0x7f0de00b8d20 2024-04-26T23:09:02.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.332+0000 7f0de6c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de00d2d80 con 0x7f0de00b6220 2024-04-26T23:09:02.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.332+0000 7f0de6c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f0de00d3c80 con 0x7f0de00d32e0 2024-04-26T23:09:02.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.332+0000 7f0de4e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0de00d32e0 0x7f0de00d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:02.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.333+0000 7f0de4e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0de00d32e0 0x7f0de00d3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43182/0 (socket says 172.21.15.40:43182) 2024-04-26T23:09:02.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.333+0000 7f0de4e00640 1 -- 172.21.15.40:0/135985261 learned_addr learned my addr 172.21.15.40:0/135985261 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:02.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.333+0000 7f0de4e00640 1 -- 172.21.15.40:0/135985261 >> v1:172.21.15.103:6789/0 conn(0x7f0de00b6220 legacy=0x7f0de00d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:02.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.333+0000 7f0de4e00640 1 -- 172.21.15.40:0/135985261 >> v1:172.21.15.40:6789/0 conn(0x7f0de00b8d20 legacy=0x7f0de00b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:02.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.333+0000 7f0de4e00640 1 -- 172.21.15.40:0/135985261 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de00b7c80 con 0x7f0de00d32e0 2024-04-26T23:09:02.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.334+0000 7f0de4e00640 1 --2- 172.21.15.40:0/135985261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de00d32e0 0x7f0de00d3740 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009870 tx=0x7f0dcc02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ad41e31acee69f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:02.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.335+0000 7f0ddfe00640 1 -- 172.21.15.40:0/135985261 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc037070 con 0x7f0de00d32e0 2024-04-26T23:09:02.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.335+0000 7f0ddfe00640 1 -- 172.21.15.40:0/135985261 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dcc032440 con 0x7f0de00d32e0 2024-04-26T23:09:02.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.335+0000 7f0ddfe00640 1 -- 172.21.15.40:0/135985261 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc032810 con 0x7f0de00d32e0 2024-04-26T23:09:02.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.335+0000 7f0de6c00640 1 -- 172.21.15.40:0/135985261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de00d32e0 msgr2=0x7f0de00d3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:02.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.335+0000 7f0de6c00640 1 --2- 172.21.15.40:0/135985261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de00d32e0 0x7f0de00d3740 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009870 tx=0x7f0dcc02a2a0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.336+0000 7f0de6c00640 1 -- 172.21.15.40:0/135985261 shutdown_connections 2024-04-26T23:09:02.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.336+0000 7f0de6c00640 1 --2- 172.21.15.40:0/135985261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de00d32e0 0x7f0de00d3740 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.336+0000 7f0de6c00640 1 -- 172.21.15.40:0/135985261 >> 172.21.15.40:0/135985261 conn(0x7f0de00d4d90 msgr2=0x7f0de00d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:02.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.336+0000 7f0de6c00640 1 -- 172.21.15.40:0/135985261 shutdown_connections 2024-04-26T23:09:02.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.336+0000 7f0de6c00640 1 -- 172.21.15.40:0/135985261 wait complete. 2024-04-26T23:09:02.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.337+0000 7f0de6c00640 1 Processor -- start 2024-04-26T23:09:02.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.338+0000 7f0de6c00640 1 -- start start 2024-04-26T23:09:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de6c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0de00b6220 0x7f0de00b5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de6c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de00b8d20 0x7f0de01f8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de00d32e0 0x7f0de00cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de6c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0de00b8520 con 0x7f0de00b8d20 2024-04-26T23:09:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de6c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0de00b8240 con 0x7f0de00b6220 2024-04-26T23:09:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0de00d0aa0 con 0x7f0de00d32e0 2024-04-26T23:09:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de00b8d20 0x7f0de01f8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de00b8d20 0x7f0de01f8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43190/0 (socket says 172.21.15.40:43190) 2024-04-26T23:09:02.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de4e00640 1 -- 172.21.15.40:0/3093385862 learned_addr learned my addr 172.21.15.40:0/3093385862 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de4e00640 1 -- 172.21.15.40:0/3093385862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de00d32e0 msgr2=0x7f0de00cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de4e00640 1 --2- 172.21.15.40:0/3093385862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de00d32e0 0x7f0de00cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de4e00640 1 -- 172.21.15.40:0/3093385862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0de00b6220 msgr2=0x7f0de00b5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de4e00640 1 --2- 172.21.15.40:0/3093385862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0de00b6220 0x7f0de00b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.339+0000 7f0de4e00640 1 -- 172.21.15.40:0/3093385862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de01f9260 con 0x7f0de00b8d20 2024-04-26T23:09:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.340+0000 7f0de4e00640 1 --2- 172.21.15.40:0/3093385862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de00b8d20 0x7f0de01f8d20 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc0090b0 tx=0x7f0dcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.340+0000 7f0ddd600640 1 -- 172.21.15.40:0/3093385862 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc037050 con 0x7f0de00b8d20 2024-04-26T23:09:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.340+0000 7f0ddd600640 1 -- 172.21.15.40:0/3093385862 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dcc043070 con 0x7f0de00b8d20 2024-04-26T23:09:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.340+0000 7f0ddd600640 1 -- 172.21.15.40:0/3093385862 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc030c70 con 0x7f0de00b8d20 2024-04-26T23:09:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.340+0000 7f0de6c00640 1 -- 172.21.15.40:0/3093385862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de01f5030 con 0x7f0de00b8d20 2024-04-26T23:09:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.340+0000 7f0de6c00640 1 -- 172.21.15.40:0/3093385862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de01f55f0 con 0x7f0de00b8d20 2024-04-26T23:09:02.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.341+0000 7f0de6c00640 1 -- 172.21.15.40:0/3093385862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de0069370 con 0x7f0de00b8d20 2024-04-26T23:09:02.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.342+0000 7f0ddd600640 1 -- 172.21.15.40:0/3093385862 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0dcc048460 con 0x7f0de00b8d20 2024-04-26T23:09:02.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.343+0000 7f0ddd600640 1 --2- 172.21.15.40:0/3093385862 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0dc006c8f0 0x7f0dc006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:02.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.344+0000 7f0ddd600640 1 -- 172.21.15.40:0/3093385862 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5249+0+0 (secure 0 0 0) 0x7f0dcc0ac880 con 0x7f0de00b8d20 2024-04-26T23:09:02.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.344+0000 7f0de5800640 1 --2- 172.21.15.40:0/3093385862 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0dc006c8f0 0x7f0dc006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:02.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.345+0000 7f0de5800640 1 --2- 172.21.15.40:0/3093385862 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0dc006c8f0 0x7f0dc006edb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0dd00043c0 tx=0x7f0dd0007d30 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:02.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.348+0000 7f0ddd600640 1 -- 172.21.15.40:0/3093385862 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0de01f9260 con 0x7f0de00b8d20 2024-04-26T23:09:02.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.552+0000 7f0de6c00640 1 -- 172.21.15.40:0/3093385862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0de00d30c0 con 0x7f0de00b8d20 2024-04-26T23:09:02.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.554+0000 7f0ddd600640 1 -- 172.21.15.40:0/3093385862 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+13129 (secure 0 0 0) 0x7f0dcc046360 con 0x7f0de00b8d20 2024-04-26T23:09:02.555 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:02.555 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":99,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:09:01.595733+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:09:02.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.557+0000 7f0de6c00640 1 -- 172.21.15.40:0/3093385862 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0dc006c8f0 msgr2=0x7f0dc006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.557+0000 7f0de6c00640 1 --2- 172.21.15.40:0/3093385862 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0dc006c8f0 0x7f0dc006edb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0dd00043c0 tx=0x7f0dd0007d30 comp rx=0 tx=0).stop 2024-04-26T23:09:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.557+0000 7f0de6c00640 1 -- 172.21.15.40:0/3093385862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de00b8d20 msgr2=0x7f0de01f8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.557+0000 7f0de6c00640 1 --2- 172.21.15.40:0/3093385862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de00b8d20 0x7f0de01f8d20 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc0090b0 tx=0x7f0dcc0046c0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.557+0000 7f0de6c00640 1 -- 172.21.15.40:0/3093385862 shutdown_connections 2024-04-26T23:09:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.557+0000 7f0de6c00640 1 --2- 172.21.15.40:0/3093385862 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0dc006c8f0 0x7f0dc006edb0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.557+0000 7f0de6c00640 1 --2- 172.21.15.40:0/3093385862 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de00d32e0 0x7f0de00cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.557+0000 7f0de6c00640 1 --2- 172.21.15.40:0/3093385862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0de00b8d20 0x7f0de01f8d20 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.557+0000 7f0de6c00640 1 --2- 172.21.15.40:0/3093385862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0de00b6220 0x7f0de00b5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.557+0000 7f0de6c00640 1 -- 172.21.15.40:0/3093385862 >> 172.21.15.40:0/3093385862 conn(0x7f0de00d4d90 msgr2=0x7f0de00ba410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.557+0000 7f0de6c00640 1 -- 172.21.15.40:0/3093385862 shutdown_connections 2024-04-26T23:09:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.557+0000 7f0de6c00640 1 -- 172.21.15.40:0/3093385862 wait complete. 2024-04-26T23:09:02.578 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:09:02.578 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:09:02.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.779+0000 7f8856600640 1 Processor -- start 2024-04-26T23:09:02.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.779+0000 7f8856600640 1 -- start start 2024-04-26T23:09:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.780+0000 7f8856600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8850071e60 0x7f885006f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.781+0000 7f8856600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f885015d7c0 con 0x7f885006fe20 2024-04-26T23:09:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.781+0000 7f8856600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f885015daa0 con 0x7f88500cc480 2024-04-26T23:09:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.781+0000 7f8856600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f885015dd80 con 0x7f8850071e60 2024-04-26T23:09:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.781+0000 7f884ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f885006fe20 0x7f8850072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45220/0 (socket says 172.21.15.40:45220) 2024-04-26T23:09:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.781+0000 7f884ea00640 1 -- 172.21.15.40:0/1047852413 learned_addr learned my addr 172.21.15.40:0/1047852413 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.781+0000 7f884f400640 1 --2- 172.21.15.40:0/1047852413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8850071e60 0x7f885006f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1217225649 0 0) 0x7f885015d7c0 con 0x7f885006fe20 2024-04-26T23:09:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8834003520 con 0x7f885006fe20 2024-04-26T23:09:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3384647681 0 0) 0x7f8834003520 con 0x7f885006fe20 2024-04-26T23:09:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8850071e60 msgr2=0x7f885006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 --2- 172.21.15.40:0/1047852413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8850071e60 0x7f885006f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 >> v1:172.21.15.103:6789/0 conn(0x7f88500cc480 legacy=0x7f8850166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885015e060 con 0x7f885006fe20 2024-04-26T23:09:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f8838001670 con 0x7f885006fe20 2024-04-26T23:09:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 >> v1:172.21.15.40:6789/0 conn(0x7f885006fe20 legacy=0x7f8850072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 --2- 172.21.15.40:0/1047852413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88340039e0 0x7f8834003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 --2- 172.21.15.40:0/1047852413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8834004800 0x7f8834004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 --2- 172.21.15.40:0/1047852413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8850071e60 0x7f88340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f885015dd80 con 0x7f88340039e0 2024-04-26T23:09:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f883400baa0 con 0x7f8834004800 2024-04-26T23:09:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f883400bca0 con 0x7f8850071e60 2024-04-26T23:09:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.782+0000 7f884f400640 1 --2- 172.21.15.40:0/1047852413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88340039e0 0x7f8834003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.783+0000 7f884fe00640 1 --2- 172.21.15.40:0/1047852413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8850071e60 0x7f88340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:02.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.783+0000 7f884ea00640 1 --2- 172.21.15.40:0/1047852413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8834004800 0x7f8834004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:02.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.783+0000 7f884f400640 1 -- 172.21.15.40:0/1047852413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8850071e60 msgr2=0x7f88340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:02.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.783+0000 7f884f400640 1 --2- 172.21.15.40:0/1047852413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8850071e60 0x7f88340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.783+0000 7f884f400640 1 -- 172.21.15.40:0/1047852413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8834004800 msgr2=0x7f8834004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:02.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.783+0000 7f884f400640 1 --2- 172.21.15.40:0/1047852413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8834004800 0x7f8834004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.783+0000 7f884f400640 1 -- 172.21.15.40:0/1047852413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8834003520 con 0x7f88340039e0 2024-04-26T23:09:02.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.783+0000 7f884fe00640 1 --2- 172.21.15.40:0/1047852413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8850071e60 0x7f88340051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:02.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.784+0000 7f884f400640 1 --2- 172.21.15.40:0/1047852413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88340039e0 0x7f8834003e00 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f8840009f90 tx=0x7f8834005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3afe8a94ce7f0de2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:02.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.784+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8840009c10 con 0x7f88340039e0 2024-04-26T23:09:02.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.784+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8840036070 con 0x7f88340039e0 2024-04-26T23:09:02.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.784+0000 7f884e000640 1 -- 172.21.15.40:0/1047852413 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8840031830 con 0x7f88340039e0 2024-04-26T23:09:02.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.785+0000 7f8856600640 1 -- 172.21.15.40:0/1047852413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88340039e0 msgr2=0x7f8834003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:02.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.785+0000 7f8856600640 1 --2- 172.21.15.40:0/1047852413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88340039e0 0x7f8834003e00 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f8840009f90 tx=0x7f8834005830 comp rx=0 tx=0).stop 2024-04-26T23:09:02.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.785+0000 7f884fe00640 1 -- 172.21.15.40:0/1047852413 reap_dead start 2024-04-26T23:09:02.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.785+0000 7f8856600640 1 -- 172.21.15.40:0/1047852413 shutdown_connections 2024-04-26T23:09:02.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.785+0000 7f8856600640 1 -- 172.21.15.40:0/1047852413 >> 172.21.15.40:0/1047852413 conn(0x7f88500d0be0 msgr2=0x7f88500d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:02.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.785+0000 7f8856600640 1 -- 172.21.15.40:0/1047852413 shutdown_connections 2024-04-26T23:09:02.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.785+0000 7f8856600640 1 -- 172.21.15.40:0/1047852413 wait complete. 2024-04-26T23:09:02.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.787+0000 7f8856600640 1 Processor -- start 2024-04-26T23:09:02.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.787+0000 7f8856600640 1 -- start start 2024-04-26T23:09:02.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.788+0000 7f8856600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88340039e0 0x7f88500afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:02.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.788+0000 7f8856600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88500ae8d0 0x7f88500b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:02.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.788+0000 7f8856600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f88500cec30 0x7f88500cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.788+0000 7f8856600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f885015d570 con 0x7f88500ae8d0 2024-04-26T23:09:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.788+0000 7f8856600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f885015d3f0 con 0x7f88340039e0 2024-04-26T23:09:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.788+0000 7f8856600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8850072950 con 0x7f88500cec30 2024-04-26T23:09:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.788+0000 7f884ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88500ae8d0 0x7f88500b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.788+0000 7f884ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88500ae8d0 0x7f88500b49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43216/0 (socket says 172.21.15.40:43216) 2024-04-26T23:09:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.788+0000 7f884ea00640 1 -- 172.21.15.40:0/1297243537 learned_addr learned my addr 172.21.15.40:0/1297243537 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:02.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.789+0000 7f884f400640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88340039e0 0x7f88500afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:02.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.789+0000 7f884ea00640 1 -- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f88500cec30 msgr2=0x7f88500cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:02.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.789+0000 7f884fe00640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f88500cec30 0x7f88500cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:02.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.789+0000 7f884ea00640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f88500cec30 0x7f88500cf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.789+0000 7f884ea00640 1 -- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88340039e0 msgr2=0x7f88500afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:02.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.789+0000 7f884ea00640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88340039e0 0x7f88500afe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.789+0000 7f884ea00640 1 -- 172.21.15.40:0/1297243537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88500aed60 con 0x7f88500ae8d0 2024-04-26T23:09:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.789+0000 7f884fe00640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f88500cec30 0x7f88500cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.789+0000 7f884ea00640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88500ae8d0 0x7f88500b49b0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f883800d7a0 tx=0x7f883800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.789+0000 7f8847400640 1 -- 172.21.15.40:0/1297243537 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8838019070 con 0x7f88500ae8d0 2024-04-26T23:09:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.789+0000 7f8847400640 1 -- 172.21.15.40:0/1297243537 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88380074e0 con 0x7f88500ae8d0 2024-04-26T23:09:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.789+0000 7f8856600640 1 -- 172.21.15.40:0/1297243537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88500af040 con 0x7f88500ae8d0 2024-04-26T23:09:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.790+0000 7f8847400640 1 -- 172.21.15.40:0/1297243537 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8838004a00 con 0x7f88500ae8d0 2024-04-26T23:09:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.790+0000 7f8856600640 1 -- 172.21.15.40:0/1297243537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88500af600 con 0x7f88500ae8d0 2024-04-26T23:09:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.791+0000 7f8856600640 1 -- 172.21.15.40:0/1297243537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88500694f0 con 0x7f88500ae8d0 2024-04-26T23:09:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.792+0000 7f8847400640 1 -- 172.21.15.40:0/1297243537 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f883801c070 con 0x7f88500ae8d0 2024-04-26T23:09:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.793+0000 7f8847400640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f882c06c8f0 0x7f882c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.793+0000 7f8847400640 1 -- 172.21.15.40:0/1297243537 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5249+0+0 (secure 0 0 0) 0x7f883808b940 con 0x7f88500ae8d0 2024-04-26T23:09:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.794+0000 7f884f400640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f882c06c8f0 0x7f882c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.795+0000 7f884f400640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f882c06c8f0 0x7f882c06edb0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8840035000 tx=0x7f884002f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:02.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:02.797+0000 7f8847400640 1 -- 172.21.15.40:0/1297243537 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8838023020 con 0x7f88500ae8d0 2024-04-26T23:09:03.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.003+0000 7f8856600640 1 -- 172.21.15.40:0/1297243537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f885006fb80 con 0x7f88500ae8d0 2024-04-26T23:09:03.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.004+0000 7f8847400640 1 -- 172.21.15.40:0/1297243537 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) ==== 93+0+10 (secure 0 0 0) 0x7f8838059f10 con 0x7f88500ae8d0 2024-04-26T23:09:03.005 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:09:03.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.007+0000 7f8856600640 1 -- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f882c06c8f0 msgr2=0x7f882c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:03.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.007+0000 7f8856600640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f882c06c8f0 0x7f882c06edb0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8840035000 tx=0x7f884002f000 comp rx=0 tx=0).stop 2024-04-26T23:09:03.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.007+0000 7f8856600640 1 -- 172.21.15.40:0/1297243537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88500ae8d0 msgr2=0x7f88500b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:03.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.007+0000 7f8856600640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88500ae8d0 0x7f88500b49b0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f883800d7a0 tx=0x7f883800dc60 comp rx=0 tx=0).stop 2024-04-26T23:09:03.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.007+0000 7f8856600640 1 -- 172.21.15.40:0/1297243537 shutdown_connections 2024-04-26T23:09:03.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.007+0000 7f8856600640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f882c06c8f0 0x7f882c06edb0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.007+0000 7f8856600640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f88500cec30 0x7f88500cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.007+0000 7f8856600640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88500ae8d0 0x7f88500b49b0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.007+0000 7f8856600640 1 --2- 172.21.15.40:0/1297243537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88340039e0 0x7f88500afe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.007+0000 7f8856600640 1 -- 172.21.15.40:0/1297243537 >> 172.21.15.40:0/1297243537 conn(0x7f88500d0be0 msgr2=0x7f885015f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:03.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.007+0000 7f8856600640 1 -- 172.21.15.40:0/1297243537 shutdown_connections 2024-04-26T23:09:03.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.008+0000 7f8856600640 1 -- 172.21.15.40:0/1297243537 wait complete. 2024-04-26T23:09:03.026 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T23:09:03.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.238+0000 7faa77400640 1 Processor -- start 2024-04-26T23:09:03.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.238+0000 7faa77400640 1 -- start start 2024-04-26T23:09:03.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.239+0000 7faa77400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faa70071e60 0x7faa7006f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.239+0000 7faa77400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa7006fe20 0x7faa70072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.240+0000 7faa77400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa7015d7c0 con 0x7faa70072780 2024-04-26T23:09:03.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.240+0000 7faa77400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7faa7015daa0 con 0x7faa70071e60 2024-04-26T23:09:03.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.240+0000 7faa77400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faa7015dd80 con 0x7faa7006fe20 2024-04-26T23:09:03.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.240+0000 7faa75400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7faa70072780 0x7faa700cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45232/0 (socket says 172.21.15.40:45232) 2024-04-26T23:09:03.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.240+0000 7faa75400640 1 -- 172.21.15.40:0/1839121092 learned_addr learned my addr 172.21.15.40:0/1839121092 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:03.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.240+0000 7faa6fe00640 1 --2- 172.21.15.40:0/1839121092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa7006fe20 0x7faa70072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.240+0000 7faa74a00640 1 --2- 172.21.15.40:0/1839121092 >> v2:172.21.15.103:3300/0 conn(0x7faa70071e60 0x7faa7006f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1659822532 0 0) 0x7faa7015d7c0 con 0x7faa70072780 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa54003520 con 0x7faa70072780 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3072670557 0 0) 0x7faa54003520 con 0x7faa70072780 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa7006fe20 msgr2=0x7faa70072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 --2- 172.21.15.40:0/1839121092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa7006fe20 0x7faa70072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 >> v2:172.21.15.103:3300/0 conn(0x7faa70071e60 msgr2=0x7faa7006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 --2- 172.21.15.40:0/1839121092 >> v2:172.21.15.103:3300/0 conn(0x7faa70071e60 0x7faa7006f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7015e060 con 0x7faa70072780 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7faa68001670 con 0x7faa70072780 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 >> v1:172.21.15.40:6789/0 conn(0x7faa70072780 legacy=0x7faa700cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 --2- 172.21.15.40:0/1839121092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa540039b0 0x7faa54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 --2- 172.21.15.40:0/1839121092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa7006fe20 0x7faa54004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 --2- 172.21.15.40:0/1839121092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa54004d40 0x7faa540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa7015daa0 con 0x7faa54004d40 2024-04-26T23:09:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa7015dd80 con 0x7faa540039b0 2024-04-26T23:09:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.241+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faa5400fdd0 con 0x7faa7006fe20 2024-04-26T23:09:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.242+0000 7faa75400640 1 --2- 172.21.15.40:0/1839121092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa54004d40 0x7faa540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.242+0000 7faa6fe00640 1 --2- 172.21.15.40:0/1839121092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa7006fe20 0x7faa54004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.242+0000 7faa74a00640 1 --2- 172.21.15.40:0/1839121092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa540039b0 0x7faa54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.242+0000 7faa75400640 1 -- 172.21.15.40:0/1839121092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa7006fe20 msgr2=0x7faa54004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.242+0000 7faa75400640 1 --2- 172.21.15.40:0/1839121092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa7006fe20 0x7faa54004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.242+0000 7faa75400640 1 -- 172.21.15.40:0/1839121092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa540039b0 msgr2=0x7faa54003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.242+0000 7faa75400640 1 --2- 172.21.15.40:0/1839121092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa540039b0 0x7faa54003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.242+0000 7faa75400640 1 -- 172.21.15.40:0/1839121092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa54003520 con 0x7faa54004d40 2024-04-26T23:09:03.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.242+0000 7faa6fe00640 1 --2- 172.21.15.40:0/1839121092 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa7006fe20 0x7faa54004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:09:03.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.243+0000 7faa75400640 1 --2- 172.21.15.40:0/1839121092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa54004d40 0x7faa540051c0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7faa6800c240 tx=0x7faa54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb6cd7ce2023384 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:03.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.243+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6802e070 con 0x7faa54004d40 2024-04-26T23:09:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.243+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa68005af0 con 0x7faa54004d40 2024-04-26T23:09:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.243+0000 7faa6f400640 1 -- 172.21.15.40:0/1839121092 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa68007d30 con 0x7faa54004d40 2024-04-26T23:09:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.244+0000 7faa77400640 1 -- 172.21.15.40:0/1839121092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa54004d40 msgr2=0x7faa540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.244+0000 7faa77400640 1 --2- 172.21.15.40:0/1839121092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa54004d40 0x7faa540051c0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7faa6800c240 tx=0x7faa54005830 comp rx=0 tx=0).stop 2024-04-26T23:09:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.244+0000 7faa75400640 1 -- 172.21.15.40:0/1839121092 reap_dead start 2024-04-26T23:09:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.244+0000 7faa77400640 1 -- 172.21.15.40:0/1839121092 shutdown_connections 2024-04-26T23:09:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.244+0000 7faa77400640 1 -- 172.21.15.40:0/1839121092 >> 172.21.15.40:0/1839121092 conn(0x7faa700d0be0 msgr2=0x7faa700d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.244+0000 7faa77400640 1 -- 172.21.15.40:0/1839121092 shutdown_connections 2024-04-26T23:09:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.244+0000 7faa77400640 1 -- 172.21.15.40:0/1839121092 wait complete. 2024-04-26T23:09:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.246+0000 7faa77400640 1 Processor -- start 2024-04-26T23:09:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.246+0000 7faa77400640 1 -- start start 2024-04-26T23:09:03.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.247+0000 7faa77400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa54004d40 0x7faa700b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.247+0000 7faa77400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa7015ac10 0x7faa7015b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.247+0000 7faa77400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa70155a10 0x7faa70155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.247+0000 7faa77400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa7015d3f0 con 0x7faa7015ac10 2024-04-26T23:09:03.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.247+0000 7faa77400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa70072c10 con 0x7faa70155a10 2024-04-26T23:09:03.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.247+0000 7faa77400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faa7015d570 con 0x7faa54004d40 2024-04-26T23:09:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.247+0000 7faa6fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa7015ac10 0x7faa7015b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.248+0000 7faa6fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa7015ac10 0x7faa7015b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43234/0 (socket says 172.21.15.40:43234) 2024-04-26T23:09:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.248+0000 7faa6fe00640 1 -- 172.21.15.40:0/3694290617 learned_addr learned my addr 172.21.15.40:0/3694290617 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.248+0000 7faa75400640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa70155a10 0x7faa70155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.248+0000 7faa74a00640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa54004d40 0x7faa700b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.248+0000 7faa6fe00640 1 -- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa54004d40 msgr2=0x7faa700b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.248+0000 7faa6fe00640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa54004d40 0x7faa700b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.248+0000 7faa6fe00640 1 -- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa70155a10 msgr2=0x7faa70155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.248+0000 7faa6fe00640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa70155a10 0x7faa70155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.248+0000 7faa6fe00640 1 -- 172.21.15.40:0/3694290617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa70159190 con 0x7faa7015ac10 2024-04-26T23:09:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.248+0000 7faa6fe00640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa7015ac10 0x7faa7015b090 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7faa58011600 tx=0x7faa58011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:03.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.249+0000 7faa6cc00640 1 -- 172.21.15.40:0/3694290617 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5800f920 con 0x7faa7015ac10 2024-04-26T23:09:03.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.249+0000 7faa6cc00640 1 -- 172.21.15.40:0/3694290617 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa5800fac0 con 0x7faa7015ac10 2024-04-26T23:09:03.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.249+0000 7faa6cc00640 1 -- 172.21.15.40:0/3694290617 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5800fe90 con 0x7faa7015ac10 2024-04-26T23:09:03.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.249+0000 7faa77400640 1 -- 172.21.15.40:0/3694290617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa70159470 con 0x7faa7015ac10 2024-04-26T23:09:03.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.249+0000 7faa77400640 1 -- 172.21.15.40:0/3694290617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa70159a30 con 0x7faa7015ac10 2024-04-26T23:09:03.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.250+0000 7faa77400640 1 -- 172.21.15.40:0/3694290617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa700694f0 con 0x7faa7015ac10 2024-04-26T23:09:03.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.251+0000 7faa6cc00640 1 -- 172.21.15.40:0/3694290617 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faa58015460 con 0x7faa7015ac10 2024-04-26T23:09:03.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.252+0000 7faa6cc00640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faa4c06ca90 0x7faa4c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.252+0000 7faa6cc00640 1 -- 172.21.15.40:0/3694290617 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5249+0+0 (secure 0 0 0) 0x7faa5808c340 con 0x7faa7015ac10 2024-04-26T23:09:03.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.253+0000 7faa74a00640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faa4c06ca90 0x7faa4c06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.254+0000 7faa74a00640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faa4c06ca90 0x7faa4c06ef50 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7faa60009fd0 tx=0x7faa6000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:03.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.257+0000 7faa6cc00640 1 -- 172.21.15.40:0/3694290617 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7faa58019030 con 0x7faa7015ac10 2024-04-26T23:09:03.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.468+0000 7faa77400640 1 -- 172.21.15.40:0/3694290617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7faa700cf440 con 0x7faa7015ac10 2024-04-26T23:09:03.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.468+0000 7faa6cc00640 1 -- 172.21.15.40:0/3694290617 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v99) ==== 84+0+65 (secure 0 0 0) 0x7faa5805a920 con 0x7faa7015ac10 2024-04-26T23:09:03.469 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:03.469 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-04-26T23:09:03.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:09:03.469 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-04-26T23:09:03.469 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-04-26T23:09:03.469 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:03.469 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-04-26T23:09:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.471+0000 7faa77400640 1 -- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faa4c06ca90 msgr2=0x7faa4c06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.471+0000 7faa77400640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faa4c06ca90 0x7faa4c06ef50 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7faa60009fd0 tx=0x7faa6000b040 comp rx=0 tx=0).stop 2024-04-26T23:09:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.471+0000 7faa77400640 1 -- 172.21.15.40:0/3694290617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa7015ac10 msgr2=0x7faa7015b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.471+0000 7faa77400640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa7015ac10 0x7faa7015b090 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7faa58011600 tx=0x7faa58011ac0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.471+0000 7faa77400640 1 -- 172.21.15.40:0/3694290617 shutdown_connections 2024-04-26T23:09:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.471+0000 7faa77400640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faa4c06ca90 0x7faa4c06ef50 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.471+0000 7faa77400640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa70155a10 0x7faa70155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.471+0000 7faa77400640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa7015ac10 0x7faa7015b090 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.471+0000 7faa77400640 1 --2- 172.21.15.40:0/3694290617 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faa54004d40 0x7faa700b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.471+0000 7faa77400640 1 -- 172.21.15.40:0/3694290617 >> 172.21.15.40:0/3694290617 conn(0x7faa700d0be0 msgr2=0x7faa700ceaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.471+0000 7faa77400640 1 -- 172.21.15.40:0/3694290617 shutdown_connections 2024-04-26T23:09:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.471+0000 7faa77400640 1 -- 172.21.15.40:0/3694290617 wait complete. 2024-04-26T23:09:03.491 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T23:09:03.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T23:09:03.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.719+0000 7f3b89800640 1 Processor -- start 2024-04-26T23:09:03.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.719+0000 7f3b89800640 1 -- start start 2024-04-26T23:09:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.721+0000 7f3b89800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3b8406eae0 0x7f3b840b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.721+0000 7f3b89800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3b8415a4a0 0x7f3b8415a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.721+0000 7f3b89800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b8415d8d0 con 0x7f3b8406e700 2024-04-26T23:09:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.721+0000 7f3b89800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f3b8415ae90 con 0x7f3b8415a4a0 2024-04-26T23:09:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.721+0000 7f3b89800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3b8415dbb0 con 0x7f3b8406eae0 2024-04-26T23:09:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.722+0000 7f3b83400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3b8415a4a0 0x7f3b8415a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.722+0000 7f3b82a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3b8406e700 0x7f3b840ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45234/0 (socket says 172.21.15.40:45234) 2024-04-26T23:09:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.722+0000 7f3b82a00640 1 -- 172.21.15.40:0/3863254868 learned_addr learned my addr 172.21.15.40:0/3863254868 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:03.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.722+0000 7f3b82000640 1 --2- 172.21.15.40:0/3863254868 >> v2:172.21.15.103:3300/0 conn(0x7f3b8406eae0 0x7f3b840b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.722+0000 7f3b83400640 1 -- 172.21.15.40:0/3863254868 >> v2:172.21.15.103:3300/0 conn(0x7f3b8406eae0 msgr2=0x7f3b840b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.722+0000 7f3b83400640 1 --2- 172.21.15.40:0/3863254868 >> v2:172.21.15.103:3300/0 conn(0x7f3b8406eae0 0x7f3b840b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.722+0000 7f3b83400640 1 -- 172.21.15.40:0/3863254868 >> v1:172.21.15.40:6789/0 conn(0x7f3b8406e700 legacy=0x7f3b840ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.722+0000 7f3b83400640 1 -- 172.21.15.40:0/3863254868 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8415de90 con 0x7f3b8415a4a0 2024-04-26T23:09:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.722+0000 7f3b81600640 1 -- 172.21.15.40:0/3863254868 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3373558156 0 0) 0x7f3b8415d8d0 con 0x7f3b8406e700 2024-04-26T23:09:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.723+0000 7f3b83400640 1 --2- 172.21.15.40:0/3863254868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8415a4a0 0x7f3b8415a950 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009d90 tx=0x7f3b7c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9028aad67b2f84eb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.724+0000 7f3b81600640 1 -- 172.21.15.40:0/3863254868 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c007740 con 0x7f3b8415a4a0 2024-04-26T23:09:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.724+0000 7f3b81600640 1 -- 172.21.15.40:0/3863254868 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b7c0078e0 con 0x7f3b8415a4a0 2024-04-26T23:09:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.724+0000 7f3b81600640 1 -- 172.21.15.40:0/3863254868 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c007cb0 con 0x7f3b8415a4a0 2024-04-26T23:09:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.724+0000 7f3b89800640 1 -- 172.21.15.40:0/3863254868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8415a4a0 msgr2=0x7f3b8415a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:03.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.724+0000 7f3b89800640 1 --2- 172.21.15.40:0/3863254868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8415a4a0 0x7f3b8415a950 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009d90 tx=0x7f3b7c02a6a0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.724+0000 7f3b89800640 1 -- 172.21.15.40:0/3863254868 shutdown_connections 2024-04-26T23:09:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.724+0000 7f3b89800640 1 --2- 172.21.15.40:0/3863254868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8415a4a0 0x7f3b8415a950 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.724+0000 7f3b89800640 1 --2- 172.21.15.40:0/3863254868 >> v2:172.21.15.103:3300/0 conn(0x7f3b8406eae0 0x7f3b840b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.724+0000 7f3b89800640 1 -- 172.21.15.40:0/3863254868 >> 172.21.15.40:0/3863254868 conn(0x7f3b8406f310 msgr2=0x7f3b8406f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.725+0000 7f3b89800640 1 -- 172.21.15.40:0/3863254868 shutdown_connections 2024-04-26T23:09:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.725+0000 7f3b89800640 1 -- 172.21.15.40:0/3863254868 wait complete. 2024-04-26T23:09:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.726+0000 7f3b89800640 1 Processor -- start 2024-04-26T23:09:03.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.726+0000 7f3b89800640 1 -- start start 2024-04-26T23:09:03.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.727+0000 7f3b89800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8406e700 0x7f3b84156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.727+0000 7f3b89800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b8406eae0 0x7f3b841f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.727+0000 7f3b89800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b8415a4a0 0x7f3b840b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.727+0000 7f3b89800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3b840d0a40 con 0x7f3b8406e700 2024-04-26T23:09:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.727+0000 7f3b89800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3b840d08c0 con 0x7f3b8406eae0 2024-04-26T23:09:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.727+0000 7f3b89800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3b8415d480 con 0x7f3b8415a4a0 2024-04-26T23:09:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.728+0000 7f3b82a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8406e700 0x7f3b84156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.728+0000 7f3b82a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8406e700 0x7f3b84156170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43258/0 (socket says 172.21.15.40:43258) 2024-04-26T23:09:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.728+0000 7f3b82a00640 1 -- 172.21.15.40:0/1600862981 learned_addr learned my addr 172.21.15.40:0/1600862981 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.728+0000 7f3b82000640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b8406eae0 0x7f3b841f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.728+0000 7f3b83400640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b8415a4a0 0x7f3b840b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.728+0000 7f3b82a00640 1 -- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b8415a4a0 msgr2=0x7f3b840b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.728+0000 7f3b82a00640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b8415a4a0 0x7f3b840b1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.728+0000 7f3b82a00640 1 -- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b8406eae0 msgr2=0x7f3b841f56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.728+0000 7f3b82a00640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b8406eae0 0x7f3b841f56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.728+0000 7f3b82a00640 1 -- 172.21.15.40:0/1600862981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b84154710 con 0x7f3b8406e700 2024-04-26T23:09:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.728+0000 7f3b83400640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b8415a4a0 0x7f3b840b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.728+0000 7f3b82a00640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8406e700 0x7f3b84156170 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400c4d0 tx=0x7f3b7400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.729+0000 7f3b7aa00640 1 -- 172.21.15.40:0/1600862981 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74018070 con 0x7f3b8406e700 2024-04-26T23:09:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.729+0000 7f3b7aa00640 1 -- 172.21.15.40:0/1600862981 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b74007500 con 0x7f3b8406e700 2024-04-26T23:09:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.729+0000 7f3b7aa00640 1 -- 172.21.15.40:0/1600862981 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74004a20 con 0x7f3b8406e700 2024-04-26T23:09:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.729+0000 7f3b89800640 1 -- 172.21.15.40:0/1600862981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b841f5c10 con 0x7f3b8406e700 2024-04-26T23:09:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.729+0000 7f3b89800640 1 -- 172.21.15.40:0/1600862981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b84154cb0 con 0x7f3b8406e700 2024-04-26T23:09:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.730+0000 7f3b89800640 1 -- 172.21.15.40:0/1600862981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b840cd530 con 0x7f3b8406e700 2024-04-26T23:09:03.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.731+0000 7f3b7aa00640 1 -- 172.21.15.40:0/1600862981 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3b74004bc0 con 0x7f3b8406e700 2024-04-26T23:09:03.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.732+0000 7f3b7aa00640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3b6406c8f0 0x7f3b6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:03.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.732+0000 7f3b7aa00640 1 -- 172.21.15.40:0/1600862981 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5249+0+0 (secure 0 0 0) 0x7f3b7408acd0 con 0x7f3b8406e700 2024-04-26T23:09:03.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.733+0000 7f3b82000640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3b6406c8f0 0x7f3b6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:03.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.734+0000 7f3b82000640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3b6406c8f0 0x7f3b6406edb0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3b840cd110 tx=0x7f3b70008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:03.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.737+0000 7f3b7aa00640 1 -- 172.21.15.40:0/1600862981 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3b74059220 con 0x7f3b8406e700 2024-04-26T23:09:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:03.940+0000 7f3b89800640 1 -- 172.21.15.40:0/1600862981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3b84069600 con 0x7f3b8406e700 2024-04-26T23:09:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.605+0000 7f3b7aa00640 1 -- 172.21.15.40:0/1600862981 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v100) ==== 138+0+0 (secure 0 0 0) 0x7f3b74058b20 con 0x7f3b8406e700 2024-04-26T23:09:04.606 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-04-26T23:09:04.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.608+0000 7f3b89800640 1 -- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3b6406c8f0 msgr2=0x7f3b6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:04.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.608+0000 7f3b89800640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3b6406c8f0 0x7f3b6406edb0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3b840cd110 tx=0x7f3b70008040 comp rx=0 tx=0).stop 2024-04-26T23:09:04.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.608+0000 7f3b89800640 1 -- 172.21.15.40:0/1600862981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8406e700 msgr2=0x7f3b84156170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:04.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.608+0000 7f3b89800640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8406e700 0x7f3b84156170 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400c4d0 tx=0x7f3b7400c990 comp rx=0 tx=0).stop 2024-04-26T23:09:04.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.609+0000 7f3b89800640 1 -- 172.21.15.40:0/1600862981 shutdown_connections 2024-04-26T23:09:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.609+0000 7f3b89800640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3b6406c8f0 0x7f3b6406edb0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.609+0000 7f3b89800640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3b8415a4a0 0x7f3b840b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.609+0000 7f3b89800640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b8406eae0 0x7f3b841f56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.609+0000 7f3b89800640 1 --2- 172.21.15.40:0/1600862981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8406e700 0x7f3b84156170 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.609+0000 7f3b89800640 1 -- 172.21.15.40:0/1600862981 >> 172.21.15.40:0/1600862981 conn(0x7f3b8406f310 msgr2=0x7f3b8415b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.609+0000 7f3b89800640 1 -- 172.21.15.40:0/1600862981 shutdown_connections 2024-04-26T23:09:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.609+0000 7f3b89800640 1 -- 172.21.15.40:0/1600862981 wait complete. 2024-04-26T23:09:04.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T23:09:04.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.835+0000 7fe94a000640 1 Processor -- start 2024-04-26T23:09:04.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.835+0000 7fe94a000640 1 -- start start 2024-04-26T23:09:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.837+0000 7fe94a000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe944073e10 0x7fe9441600d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.837+0000 7fe94a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9440b4f50 0x7fe94415d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.837+0000 7fe94a000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe944160610 con 0x7fe94415c160 2024-04-26T23:09:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.837+0000 7fe94a000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe944153040 con 0x7fe9440b4f50 2024-04-26T23:09:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.837+0000 7fe94a000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe94415fc40 con 0x7fe944073e10 2024-04-26T23:09:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.838+0000 7fe949600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9440b4f50 0x7fe94415d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.838+0000 7fe949600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9440b4f50 0x7fe94415d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43262/0 (socket says 172.21.15.40:43262) 2024-04-26T23:09:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.838+0000 7fe949600640 1 -- 172.21.15.40:0/629324960 learned_addr learned my addr 172.21.15.40:0/629324960 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.838+0000 7fe949600640 1 -- 172.21.15.40:0/629324960 >> v2:172.21.15.103:3300/0 conn(0x7fe944073e10 msgr2=0x7fe9441600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.838+0000 7fe949600640 1 --2- 172.21.15.40:0/629324960 >> v2:172.21.15.103:3300/0 conn(0x7fe944073e10 0x7fe9441600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.838+0000 7fe949600640 1 -- 172.21.15.40:0/629324960 >> v1:172.21.15.103:6789/0 conn(0x7fe94415c160 legacy=0x7fe9440b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.838+0000 7fe949600640 1 -- 172.21.15.40:0/629324960 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe94415cb90 con 0x7fe9440b4f50 2024-04-26T23:09:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.839+0000 7fe949600640 1 --2- 172.21.15.40:0/629324960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9440b4f50 0x7fe94415d150 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fe944159ca0 tx=0x7fe93c02a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2071c3ac1804384c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.840+0000 7fe943400640 1 -- 172.21.15.40:0/629324960 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c036070 con 0x7fe9440b4f50 2024-04-26T23:09:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.840+0000 7fe943400640 1 -- 172.21.15.40:0/629324960 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe93c031600 con 0x7fe9440b4f50 2024-04-26T23:09:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.840+0000 7fe943400640 1 -- 172.21.15.40:0/629324960 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c0319d0 con 0x7fe9440b4f50 2024-04-26T23:09:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.840+0000 7fe94a000640 1 -- 172.21.15.40:0/629324960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9440b4f50 msgr2=0x7fe94415d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.840+0000 7fe94a000640 1 --2- 172.21.15.40:0/629324960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9440b4f50 0x7fe94415d150 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fe944159ca0 tx=0x7fe93c02a6d0 comp rx=0 tx=0).stop 2024-04-26T23:09:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.840+0000 7fe94a000640 1 -- 172.21.15.40:0/629324960 shutdown_connections 2024-04-26T23:09:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.840+0000 7fe94a000640 1 --2- 172.21.15.40:0/629324960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9440b4f50 0x7fe94415d150 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.840+0000 7fe94a000640 1 --2- 172.21.15.40:0/629324960 >> v2:172.21.15.103:3300/0 conn(0x7fe944073e10 0x7fe9441600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.840+0000 7fe94a000640 1 -- 172.21.15.40:0/629324960 >> 172.21.15.40:0/629324960 conn(0x7fe944075eb0 msgr2=0x7fe944068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.841+0000 7fe94a000640 1 -- 172.21.15.40:0/629324960 shutdown_connections 2024-04-26T23:09:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.841+0000 7fe94a000640 1 -- 172.21.15.40:0/629324960 wait complete. 2024-04-26T23:09:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.843+0000 7fe94a000640 1 Processor -- start 2024-04-26T23:09:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.843+0000 7fe94a000640 1 -- start start 2024-04-26T23:09:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe94a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe944073e10 0x7fe9440cf370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe94a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9440b4f50 0x7fe94415aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe94a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe94415c160 0x7fe94415b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe94a000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe9440733c0 con 0x7fe944073e10 2024-04-26T23:09:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe94a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe94415f8d0 con 0x7fe9440b4f50 2024-04-26T23:09:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe94a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe944073540 con 0x7fe94415c160 2024-04-26T23:09:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe948c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe944073e10 0x7fe9440cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe948c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe944073e10 0x7fe9440cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43278/0 (socket says 172.21.15.40:43278) 2024-04-26T23:09:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe948c00640 1 -- 172.21.15.40:0/2246378549 learned_addr learned my addr 172.21.15.40:0/2246378549 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe948c00640 1 -- 172.21.15.40:0/2246378549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe94415c160 msgr2=0x7fe94415b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe948c00640 1 --2- 172.21.15.40:0/2246378549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe94415c160 0x7fe94415b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe948c00640 1 -- 172.21.15.40:0/2246378549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9440b4f50 msgr2=0x7fe94415aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe948c00640 1 --2- 172.21.15.40:0/2246378549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9440b4f50 0x7fe94415aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.845+0000 7fe948c00640 1 -- 172.21.15.40:0/2246378549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9440cfba0 con 0x7fe944073e10 2024-04-26T23:09:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.846+0000 7fe948c00640 1 --2- 172.21.15.40:0/2246378549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe944073e10 0x7fe9440cf370 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fe93800b410 tx=0x7fe93800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.846+0000 7fe940c00640 1 -- 172.21.15.40:0/2246378549 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938015020 con 0x7fe944073e10 2024-04-26T23:09:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.846+0000 7fe940c00640 1 -- 172.21.15.40:0/2246378549 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe938011850 con 0x7fe944073e10 2024-04-26T23:09:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.846+0000 7fe940c00640 1 -- 172.21.15.40:0/2246378549 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938011c40 con 0x7fe944073e10 2024-04-26T23:09:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.846+0000 7fe94a000640 1 -- 172.21.15.40:0/2246378549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9440cfe80 con 0x7fe944073e10 2024-04-26T23:09:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.846+0000 7fe94a000640 1 -- 172.21.15.40:0/2246378549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9440d0440 con 0x7fe944073e10 2024-04-26T23:09:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.848+0000 7fe94a000640 1 -- 172.21.15.40:0/2246378549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9440cd9f0 con 0x7fe944073e10 2024-04-26T23:09:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.850+0000 7fe940c00640 1 -- 172.21.15.40:0/2246378549 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe938018020 con 0x7fe944073e10 2024-04-26T23:09:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.851+0000 7fe940c00640 1 --2- 172.21.15.40:0/2246378549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe92406c8f0 0x7fe92406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.851+0000 7fe940c00640 1 -- 172.21.15.40:0/2246378549 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5619+0+0 (secure 0 0 0) 0x7fe93808bff0 con 0x7fe944073e10 2024-04-26T23:09:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.852+0000 7fe943e00640 1 --2- 172.21.15.40:0/2246378549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe92406c8f0 0x7fe92406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:04.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.853+0000 7fe943e00640 1 --2- 172.21.15.40:0/2246378549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe92406c8f0 0x7fe92406edb0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe94415bcb0 tx=0x7fe930007990 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:04.854+0000 7fe940c00640 1 -- 172.21.15.40:0/2246378549 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe9440cd9f0 con 0x7fe944073e10 2024-04-26T23:09:05.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.054+0000 7fe94a000640 1 -- 172.21.15.40:0/2246378549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe94406f4e0 con 0x7fe944073e10 2024-04-26T23:09:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.623+0000 7fe940c00640 1 -- 172.21.15.40:0/2246378549 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v101) ==== 170+0+0 (secure 0 0 0) 0x7fe93801f020 con 0x7fe944073e10 2024-04-26T23:09:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-04-26T23:09:05.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.627+0000 7fe94a000640 1 -- 172.21.15.40:0/2246378549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe92406c8f0 msgr2=0x7fe92406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:05.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.627+0000 7fe94a000640 1 --2- 172.21.15.40:0/2246378549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe92406c8f0 0x7fe92406edb0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe94415bcb0 tx=0x7fe930007990 comp rx=0 tx=0).stop 2024-04-26T23:09:05.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.627+0000 7fe94a000640 1 -- 172.21.15.40:0/2246378549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe944073e10 msgr2=0x7fe9440cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:05.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.627+0000 7fe94a000640 1 --2- 172.21.15.40:0/2246378549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe944073e10 0x7fe9440cf370 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fe93800b410 tx=0x7fe93800b8d0 comp rx=0 tx=0).stop 2024-04-26T23:09:05.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.627+0000 7fe94a000640 1 -- 172.21.15.40:0/2246378549 shutdown_connections 2024-04-26T23:09:05.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.627+0000 7fe94a000640 1 --2- 172.21.15.40:0/2246378549 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe92406c8f0 0x7fe92406edb0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:05.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.627+0000 7fe94a000640 1 --2- 172.21.15.40:0/2246378549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe94415c160 0x7fe94415b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.627+0000 7fe94a000640 1 --2- 172.21.15.40:0/2246378549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9440b4f50 0x7fe94415aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.627+0000 7fe94a000640 1 --2- 172.21.15.40:0/2246378549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe944073e10 0x7fe9440cf370 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.627+0000 7fe94a000640 1 -- 172.21.15.40:0/2246378549 >> 172.21.15.40:0/2246378549 conn(0x7fe944075eb0 msgr2=0x7fe944155790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.628+0000 7fe94a000640 1 -- 172.21.15.40:0/2246378549 shutdown_connections 2024-04-26T23:09:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.628+0000 7fe94a000640 1 -- 172.21.15.40:0/2246378549 wait complete. 2024-04-26T23:09:05.666 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T23:09:05.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.847+0000 7fd369600640 1 Processor -- start 2024-04-26T23:09:05.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.847+0000 7fd369600640 1 -- start start 2024-04-26T23:09:05.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.849+0000 7fd369600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b5120 0x7fd3640b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:05.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.849+0000 7fd369600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd3640b5500 0x7fd3640b6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:05.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.849+0000 7fd369600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3640d3dc0 con 0x7fd3640b59b0 2024-04-26T23:09:05.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.849+0000 7fd369600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd3640d03b0 con 0x7fd3640b5500 2024-04-26T23:09:05.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.849+0000 7fd369600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd3640b75b0 con 0x7fd3640b5120 2024-04-26T23:09:05.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.849+0000 7fd368c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd3640b59b0 0x7fd3640b6d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49956/0 (socket says 172.21.15.40:49956) 2024-04-26T23:09:05.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.849+0000 7fd368c00640 1 -- 172.21.15.40:0/115952258 learned_addr learned my addr 172.21.15.40:0/115952258 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:05.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd363e00640 1 --2- 172.21.15.40:0/115952258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b5120 0x7fd3640b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:05.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd363400640 1 --2- 172.21.15.40:0/115952258 >> v2:172.21.15.103:3300/0 conn(0x7fd3640b5500 0x7fd3640b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:05.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1464735188 0 0) 0x7fd3640d3dc0 con 0x7fd3640b59b0 2024-04-26T23:09:05.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd344003520 con 0x7fd3640b59b0 2024-04-26T23:09:05.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 625279956 0 0) 0x7fd344003520 con 0x7fd3640b59b0 2024-04-26T23:09:05.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b5120 msgr2=0x7fd3640b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 --2- 172.21.15.40:0/115952258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b5120 0x7fd3640b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 >> v2:172.21.15.103:3300/0 conn(0x7fd3640b5500 msgr2=0x7fd3640b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 --2- 172.21.15.40:0/115952258 >> v2:172.21.15.103:3300/0 conn(0x7fd3640b5500 0x7fd3640b6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3640d0600 con 0x7fd3640b59b0 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fd35c001670 con 0x7fd3640b59b0 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 >> v1:172.21.15.40:6789/0 conn(0x7fd3640b59b0 legacy=0x7fd3640b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 --2- 172.21.15.40:0/115952258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3440039b0 0x7fd344003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 --2- 172.21.15.40:0/115952258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd344004800 0x7fd344004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 --2- 172.21.15.40:0/115952258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b5120 0x7fd3440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd3640d03b0 con 0x7fd3440039b0 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd3640b75b0 con 0x7fd344004800 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.850+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd34400baa0 con 0x7fd3640b5120 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.851+0000 7fd363e00640 1 --2- 172.21.15.40:0/115952258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3440039b0 0x7fd344003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.851+0000 7fd368c00640 1 --2- 172.21.15.40:0/115952258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b5120 0x7fd3440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:05.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.851+0000 7fd363400640 1 --2- 172.21.15.40:0/115952258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd344004800 0x7fd344004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:05.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.851+0000 7fd363e00640 1 -- 172.21.15.40:0/115952258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b5120 msgr2=0x7fd3440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:05.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.851+0000 7fd363e00640 1 --2- 172.21.15.40:0/115952258 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b5120 0x7fd3440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:05.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.851+0000 7fd363e00640 1 -- 172.21.15.40:0/115952258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd344004800 msgr2=0x7fd344004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:05.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.851+0000 7fd363e00640 1 --2- 172.21.15.40:0/115952258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd344004800 0x7fd344004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:05.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.851+0000 7fd363e00640 1 -- 172.21.15.40:0/115952258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd344003520 con 0x7fd3440039b0 2024-04-26T23:09:05.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.853+0000 7fd363e00640 1 --2- 172.21.15.40:0/115952258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3440039b0 0x7fd344003dd0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fd354009a90 tx=0x7fd344005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf9083b6534283a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:05.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.853+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd354036070 con 0x7fd3440039b0 2024-04-26T23:09:05.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.853+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd354031400 con 0x7fd3440039b0 2024-04-26T23:09:05.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.853+0000 7fd362a00640 1 -- 172.21.15.40:0/115952258 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd354009690 con 0x7fd3440039b0 2024-04-26T23:09:05.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.854+0000 7fd369600640 1 -- 172.21.15.40:0/115952258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3440039b0 msgr2=0x7fd344003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:05.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.854+0000 7fd369600640 1 --2- 172.21.15.40:0/115952258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3440039b0 0x7fd344003dd0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fd354009a90 tx=0x7fd344005830 comp rx=0 tx=0).stop 2024-04-26T23:09:05.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.854+0000 7fd368c00640 1 -- 172.21.15.40:0/115952258 reap_dead start 2024-04-26T23:09:05.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.854+0000 7fd369600640 1 -- 172.21.15.40:0/115952258 shutdown_connections 2024-04-26T23:09:05.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.854+0000 7fd369600640 1 -- 172.21.15.40:0/115952258 >> 172.21.15.40:0/115952258 conn(0x7fd3640d4db0 msgr2=0x7fd3640d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:05.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.854+0000 7fd369600640 1 -- 172.21.15.40:0/115952258 shutdown_connections 2024-04-26T23:09:05.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.854+0000 7fd369600640 1 -- 172.21.15.40:0/115952258 wait complete. 2024-04-26T23:09:05.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.856+0000 7fd369600640 1 Processor -- start 2024-04-26T23:09:05.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.856+0000 7fd369600640 1 -- start start 2024-04-26T23:09:05.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.857+0000 7fd369600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3440039b0 0x7fd3640b9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:05.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.857+0000 7fd369600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3640ca100 0x7fd3640ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:05.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.857+0000 7fd369600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b19a0 0x7fd3640b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:05.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.857+0000 7fd369600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd3640b5ee0 con 0x7fd3440039b0 2024-04-26T23:09:05.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.857+0000 7fd369600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd3640d0bc0 con 0x7fd3640ca100 2024-04-26T23:09:05.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.857+0000 7fd369600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd3640b6060 con 0x7fd3640b19a0 2024-04-26T23:09:05.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.857+0000 7fd363e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3440039b0 0x7fd3640b9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:05.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.857+0000 7fd363e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3440039b0 0x7fd3640b9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37820/0 (socket says 172.21.15.40:37820) 2024-04-26T23:09:05.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.857+0000 7fd363e00640 1 -- 172.21.15.40:0/1523163768 learned_addr learned my addr 172.21.15.40:0/1523163768 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:05.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.857+0000 7fd363400640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3640ca100 0x7fd3640ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.858+0000 7fd368c00640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b19a0 0x7fd3640b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.858+0000 7fd363e00640 1 -- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b19a0 msgr2=0x7fd3640b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.858+0000 7fd363e00640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b19a0 0x7fd3640b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.858+0000 7fd363e00640 1 -- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3640ca100 msgr2=0x7fd3640ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.858+0000 7fd363e00640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3640ca100 0x7fd3640ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.858+0000 7fd363e00640 1 -- 172.21.15.40:0/1523163768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3640caba0 con 0x7fd3440039b0 2024-04-26T23:09:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.858+0000 7fd368c00640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b19a0 0x7fd3640b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.858+0000 7fd363e00640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3440039b0 0x7fd3640b9be0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fd354009bb0 tx=0x7fd354004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.858+0000 7fd35be00640 1 -- 172.21.15.40:0/1523163768 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd354040020 con 0x7fd3440039b0 2024-04-26T23:09:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.858+0000 7fd35be00640 1 -- 172.21.15.40:0/1523163768 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd354003940 con 0x7fd3440039b0 2024-04-26T23:09:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.858+0000 7fd35be00640 1 -- 172.21.15.40:0/1523163768 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd354036040 con 0x7fd3440039b0 2024-04-26T23:09:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.858+0000 7fd369600640 1 -- 172.21.15.40:0/1523163768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3640cae80 con 0x7fd3440039b0 2024-04-26T23:09:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.859+0000 7fd369600640 1 -- 172.21.15.40:0/1523163768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3640b7a70 con 0x7fd3440039b0 2024-04-26T23:09:05.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.860+0000 7fd369600640 1 -- 172.21.15.40:0/1523163768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd364069370 con 0x7fd3440039b0 2024-04-26T23:09:05.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.861+0000 7fd35be00640 1 -- 172.21.15.40:0/1523163768 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd354043020 con 0x7fd3440039b0 2024-04-26T23:09:05.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.862+0000 7fd35be00640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd34006c920 0x7fd34006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:05.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.862+0000 7fd35be00640 1 -- 172.21.15.40:0/1523163768 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6001+0+0 (secure 0 0 0) 0x7fd3540abf20 con 0x7fd3440039b0 2024-04-26T23:09:05.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.863+0000 7fd363400640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd34006c920 0x7fd34006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:05.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.864+0000 7fd363400640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd34006c920 0x7fd34006ede0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd36406dec0 tx=0x7fd34c00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:05.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:05.867+0000 7fd35be00640 1 -- 172.21.15.40:0/1523163768 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd35404a070 con 0x7fd3440039b0 2024-04-26T23:09:06.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.114+0000 7fd369600640 1 -- 172.21.15.40:0/1523163768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd364161d30 con 0x7fd3440039b0 2024-04-26T23:09:06.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.628+0000 7fd35be00640 1 -- 172.21.15.40:0/1523163768 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v107) ==== 174+0+0 (secure 0 0 0) 0x7fd35407a190 con 0x7fd3440039b0 2024-04-26T23:09:06.628 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 15 and data pool 16 2024-04-26T23:09:06.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.632+0000 7fd369600640 1 -- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd34006c920 msgr2=0x7fd34006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:06.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.632+0000 7fd369600640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd34006c920 0x7fd34006ede0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd36406dec0 tx=0x7fd34c00b040 comp rx=0 tx=0).stop 2024-04-26T23:09:06.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.632+0000 7fd369600640 1 -- 172.21.15.40:0/1523163768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3440039b0 msgr2=0x7fd3640b9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:06.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.632+0000 7fd369600640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3440039b0 0x7fd3640b9be0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fd354009bb0 tx=0x7fd354004730 comp rx=0 tx=0).stop 2024-04-26T23:09:06.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.632+0000 7fd369600640 1 -- 172.21.15.40:0/1523163768 shutdown_connections 2024-04-26T23:09:06.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.632+0000 7fd369600640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd34006c920 0x7fd34006ede0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:06.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.632+0000 7fd369600640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3640b19a0 0x7fd3640b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:06.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.632+0000 7fd369600640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3640ca100 0x7fd3640ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:06.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.632+0000 7fd369600640 1 --2- 172.21.15.40:0/1523163768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3440039b0 0x7fd3640b9be0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:06.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.632+0000 7fd369600640 1 -- 172.21.15.40:0/1523163768 >> 172.21.15.40:0/1523163768 conn(0x7fd3640d4db0 msgr2=0x7fd3640cf880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:06.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.633+0000 7fd369600640 1 -- 172.21.15.40:0/1523163768 shutdown_connections 2024-04-26T23:09:06.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.633+0000 7fd369600640 1 -- 172.21.15.40:0/1523163768 wait complete. 2024-04-26T23:09:06.680 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:09:06.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.853+0000 7f4fb3e00640 1 Processor -- start 2024-04-26T23:09:06.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.853+0000 7f4fb3e00640 1 -- start start 2024-04-26T23:09:06.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.855+0000 7f4fb3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4fb4163430 0x7f4fb4166f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:06.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.855+0000 7f4fb3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fb41674a0 0x7f4fb4167920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:06.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.855+0000 7f4fb3e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4fb40c62c0 0x7f4fb40c6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:06.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.855+0000 7f4fb3e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f4fb40cb400 con 0x7f4fb4163430 2024-04-26T23:09:06.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.855+0000 7f4fb3e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4fb4167e60 con 0x7f4fb40c62c0 2024-04-26T23:09:06.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.855+0000 7f4fb3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb4163810 con 0x7f4fb41674a0 2024-04-26T23:09:06.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.855+0000 7f4fb2a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4fb4163430 0x7f4fb4166f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:06.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.856+0000 7f4fb2a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4fb4163430 0x7f4fb4166f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37834/0 (socket says 172.21.15.40:37834) 2024-04-26T23:09:06.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.856+0000 7f4fb2a00640 1 -- 172.21.15.40:0/2466518673 learned_addr learned my addr 172.21.15.40:0/2466518673 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:06.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.856+0000 7f4fb3400640 1 --2- 172.21.15.40:0/2466518673 >> v2:172.21.15.103:3300/0 conn(0x7f4fb40c62c0 0x7f4fb40c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:06.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.856+0000 7f4fabe00640 1 --2- 172.21.15.40:0/2466518673 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fb41674a0 0x7f4fb4167920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:06.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.856+0000 7f4fb2a00640 1 -- 172.21.15.40:0/2466518673 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fb41674a0 msgr2=0x7f4fb4167920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:06.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.856+0000 7f4fb2a00640 1 --2- 172.21.15.40:0/2466518673 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fb41674a0 0x7f4fb4167920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:06.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.856+0000 7f4fb2a00640 1 -- 172.21.15.40:0/2466518673 >> v2:172.21.15.103:3300/0 conn(0x7f4fb40c62c0 msgr2=0x7f4fb40c6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:06.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.856+0000 7f4fb2a00640 1 --2- 172.21.15.40:0/2466518673 >> v2:172.21.15.103:3300/0 conn(0x7f4fb40c62c0 0x7f4fb40c6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:06.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.856+0000 7f4fb2a00640 1 -- 172.21.15.40:0/2466518673 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb40cb100 con 0x7f4fb4163430 2024-04-26T23:09:06.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.858+0000 7f4fb2a00640 1 --2- 172.21.15.40:0/2466518673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4163430 0x7f4fb4166f60 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4071ac0 tx=0x7f4fa402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7178308b4d972776 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:06.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.858+0000 7f4fb2000640 1 -- 172.21.15.40:0/2466518673 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4037070 con 0x7f4fb4163430 2024-04-26T23:09:06.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.858+0000 7f4fb2000640 1 -- 172.21.15.40:0/2466518673 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa4032440 con 0x7f4fb4163430 2024-04-26T23:09:06.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.858+0000 7f4fb2000640 1 -- 172.21.15.40:0/2466518673 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4032810 con 0x7f4fb4163430 2024-04-26T23:09:06.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.859+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2466518673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4163430 msgr2=0x7f4fb4166f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:06.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.859+0000 7f4fb3e00640 1 --2- 172.21.15.40:0/2466518673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4163430 0x7f4fb4166f60 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4071ac0 tx=0x7f4fa402a1e0 comp rx=0 tx=0).stop 2024-04-26T23:09:06.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.859+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2466518673 shutdown_connections 2024-04-26T23:09:06.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.859+0000 7f4fb3e00640 1 --2- 172.21.15.40:0/2466518673 >> v2:172.21.15.103:3300/0 conn(0x7f4fb40c62c0 0x7f4fb40c6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:06.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.859+0000 7f4fb3e00640 1 --2- 172.21.15.40:0/2466518673 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fb41674a0 0x7f4fb4167920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:06.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.859+0000 7f4fb3e00640 1 --2- 172.21.15.40:0/2466518673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4163430 0x7f4fb4166f60 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:06.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.859+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2466518673 >> 172.21.15.40:0/2466518673 conn(0x7f4fb40d0a10 msgr2=0x7f4fb40d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:06.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.859+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2466518673 shutdown_connections 2024-04-26T23:09:06.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.859+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2466518673 wait complete. 2024-04-26T23:09:06.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.861+0000 7f4fb3e00640 1 Processor -- start 2024-04-26T23:09:06.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.861+0000 7f4fb3e00640 1 -- start start 2024-04-26T23:09:06.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.862+0000 7f4fb3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fb40c62c0 0x7f4fb415e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:06.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.862+0000 7f4fb3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4163430 0x7f4fb40c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:06.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.862+0000 7f4fb3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fb41674a0 0x7f4fb40c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:06.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.862+0000 7f4fb3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb40afe80 con 0x7f4fb4163430 2024-04-26T23:09:06.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.862+0000 7f4fb3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb40ccc60 con 0x7f4fb40c62c0 2024-04-26T23:09:06.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.862+0000 7f4fb3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb40cd1e0 con 0x7f4fb41674a0 2024-04-26T23:09:06.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.863+0000 7f4fabe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4163430 0x7f4fb40c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:06.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.863+0000 7f4fabe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4163430 0x7f4fb40c9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37840/0 (socket says 172.21.15.40:37840) 2024-04-26T23:09:06.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.863+0000 7f4fabe00640 1 -- 172.21.15.40:0/2792957664 learned_addr learned my addr 172.21.15.40:0/2792957664 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:06.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.863+0000 7f4fabe00640 1 -- 172.21.15.40:0/2792957664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fb41674a0 msgr2=0x7f4fb40c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:06.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.863+0000 7f4fabe00640 1 --2- 172.21.15.40:0/2792957664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fb41674a0 0x7f4fb40c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:06.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.863+0000 7f4fabe00640 1 -- 172.21.15.40:0/2792957664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fb40c62c0 msgr2=0x7f4fb415e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:06.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.863+0000 7f4fabe00640 1 --2- 172.21.15.40:0/2792957664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fb40c62c0 0x7f4fb415e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:06.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.863+0000 7f4fabe00640 1 -- 172.21.15.40:0/2792957664 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb40b4cb0 con 0x7f4fb4163430 2024-04-26T23:09:06.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.863+0000 7f4fabe00640 1 --2- 172.21.15.40:0/2792957664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4163430 0x7f4fb40c9260 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00d830 tx=0x7f4f9c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:06.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.864+0000 7f4faaa00640 1 -- 172.21.15.40:0/2792957664 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c00b6c0 con 0x7f4fb4163430 2024-04-26T23:09:06.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.864+0000 7f4faaa00640 1 -- 172.21.15.40:0/2792957664 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f9c019070 con 0x7f4fb4163430 2024-04-26T23:09:06.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.864+0000 7f4faaa00640 1 -- 172.21.15.40:0/2792957664 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c00ba80 con 0x7f4fb4163430 2024-04-26T23:09:06.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.864+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2792957664 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb4164df0 con 0x7f4fb4163430 2024-04-26T23:09:06.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.864+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2792957664 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb41653b0 con 0x7f4fb4163430 2024-04-26T23:09:06.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.865+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2792957664 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb4069370 con 0x7f4fb4163430 2024-04-26T23:09:06.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.866+0000 7f4faaa00640 1 -- 172.21.15.40:0/2792957664 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4f9c01e460 con 0x7f4fb4163430 2024-04-26T23:09:06.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.867+0000 7f4faaa00640 1 --2- 172.21.15.40:0/2792957664 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4f9006c8f0 0x7f4f9006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:06.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.867+0000 7f4faaa00640 1 -- 172.21.15.40:0/2792957664 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6101+0+0 (secure 0 0 0) 0x7f4f9c08c460 con 0x7f4fb4163430 2024-04-26T23:09:06.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.868+0000 7f4fb2a00640 1 --2- 172.21.15.40:0/2792957664 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4f9006c8f0 0x7f4f9006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:06.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.869+0000 7f4fb2a00640 1 --2- 172.21.15.40:0/2792957664 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4f9006c8f0 0x7f4f9006edb0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4034000 tx=0x7f4fa402c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:06.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:06.872+0000 7f4faaa00640 1 -- 172.21.15.40:0/2792957664 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4f9c05a660 con 0x7f4fb4163430 2024-04-26T23:09:07.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.067+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2792957664 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4fb4167b50 con 0x7f4fb4163430 2024-04-26T23:09:07.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.069+0000 7f4faaa00640 1 -- 172.21.15.40:0/2792957664 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+16861 (secure 0 0 0) 0x7f4f9c059f60 con 0x7f4fb4163430 2024-04-26T23:09:07.070 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:07.070 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:09:06.116569+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:09:03.955148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-26T23:09:05.070627+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:09:07.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.072+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2792957664 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4f9006c8f0 msgr2=0x7f4f9006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:07.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.072+0000 7f4fb3e00640 1 --2- 172.21.15.40:0/2792957664 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4f9006c8f0 0x7f4f9006edb0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4034000 tx=0x7f4fa402c000 comp rx=0 tx=0).stop 2024-04-26T23:09:07.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.072+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2792957664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4163430 msgr2=0x7f4fb40c9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:07.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.072+0000 7f4fb3e00640 1 --2- 172.21.15.40:0/2792957664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4163430 0x7f4fb40c9260 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00d830 tx=0x7f4f9c00dcf0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.072+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2792957664 shutdown_connections 2024-04-26T23:09:07.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.072+0000 7f4fb3e00640 1 --2- 172.21.15.40:0/2792957664 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4f9006c8f0 0x7f4f9006edb0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.072+0000 7f4fb3e00640 1 --2- 172.21.15.40:0/2792957664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fb41674a0 0x7f4fb40c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.072+0000 7f4fb3e00640 1 --2- 172.21.15.40:0/2792957664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4163430 0x7f4fb40c9260 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.072+0000 7f4fb3e00640 1 --2- 172.21.15.40:0/2792957664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fb40c62c0 0x7f4fb415e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.072+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2792957664 >> 172.21.15.40:0/2792957664 conn(0x7f4fb40d0a10 msgr2=0x7f4fb40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:07.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.072+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2792957664 shutdown_connections 2024-04-26T23:09:07.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.072+0000 7f4fb3e00640 1 -- 172.21.15.40:0/2792957664 wait complete. 2024-04-26T23:09:07.091 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:09:07.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.303+0000 7f9c53e00640 1 Processor -- start 2024-04-26T23:09:07.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.303+0000 7f9c53e00640 1 -- start start 2024-04-26T23:09:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.305+0000 7f9c53e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9c540ce5e0 0x7f9c540afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.305+0000 7f9c53e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9c540cc590 0x7f9c540cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.305+0000 7f9c53e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c540af7b0 con 0x7f9c5415f2d0 2024-04-26T23:09:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.305+0000 7f9c53e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f9c540afa90 con 0x7f9c540ce5e0 2024-04-26T23:09:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.305+0000 7f9c53e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9c540cbd90 con 0x7f9c540cc590 2024-04-26T23:09:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.305+0000 7f9c4be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9c540ce5e0 0x7f9c540afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.305+0000 7f9c4be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9c540ce5e0 0x7f9c540afdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37850/0 (socket says 172.21.15.40:37850) 2024-04-26T23:09:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.305+0000 7f9c4be00640 1 -- 172.21.15.40:0/1784954899 learned_addr learned my addr 172.21.15.40:0/1784954899 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:07.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.305+0000 7f9c4be00640 1 -- 172.21.15.40:0/1784954899 >> v2:172.21.15.103:3300/0 conn(0x7f9c540cc590 msgr2=0x7f9c540cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:07.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.305+0000 7f9c53400640 1 --2- 172.21.15.40:0/1784954899 >> v2:172.21.15.103:3300/0 conn(0x7f9c540cc590 0x7f9c540cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:07.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.306+0000 7f9c4be00640 1 --2- 172.21.15.40:0/1784954899 >> v2:172.21.15.103:3300/0 conn(0x7f9c540cc590 0x7f9c540cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.306+0000 7f9c4be00640 1 -- 172.21.15.40:0/1784954899 >> v1:172.21.15.40:6789/0 conn(0x7f9c5415f2d0 legacy=0x7f9c540c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:07.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.306+0000 7f9c4be00640 1 -- 172.21.15.40:0/1784954899 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c540cc070 con 0x7f9c540ce5e0 2024-04-26T23:09:07.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.306+0000 7f9c52000640 1 -- 172.21.15.40:0/1784954899 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 728623345 0 0) 0x7f9c540af7b0 con 0x7f9c5415f2d0 2024-04-26T23:09:07.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.307+0000 7f9c4be00640 1 --2- 172.21.15.40:0/1784954899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c540ce5e0 0x7f9c540afdf0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f9c540c7a50 tx=0x7f9c3c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408a81470831fc53 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:07.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.307+0000 7f9c52000640 1 -- 172.21.15.40:0/1784954899 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c037070 con 0x7f9c540ce5e0 2024-04-26T23:09:07.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.307+0000 7f9c52000640 1 -- 172.21.15.40:0/1784954899 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c3c032440 con 0x7f9c540ce5e0 2024-04-26T23:09:07.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.307+0000 7f9c52000640 1 -- 172.21.15.40:0/1784954899 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c032810 con 0x7f9c540ce5e0 2024-04-26T23:09:07.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.307+0000 7f9c53e00640 1 -- 172.21.15.40:0/1784954899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c540ce5e0 msgr2=0x7f9c540afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:07.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.307+0000 7f9c53e00640 1 --2- 172.21.15.40:0/1784954899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c540ce5e0 0x7f9c540afdf0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f9c540c7a50 tx=0x7f9c3c02a2a0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.308+0000 7f9c53e00640 1 -- 172.21.15.40:0/1784954899 shutdown_connections 2024-04-26T23:09:07.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.308+0000 7f9c53e00640 1 --2- 172.21.15.40:0/1784954899 >> v2:172.21.15.103:3300/0 conn(0x7f9c540cc590 0x7f9c540cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.308+0000 7f9c53e00640 1 --2- 172.21.15.40:0/1784954899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c540ce5e0 0x7f9c540afdf0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.308+0000 7f9c53e00640 1 -- 172.21.15.40:0/1784954899 >> 172.21.15.40:0/1784954899 conn(0x7f9c540d0a10 msgr2=0x7f9c540d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:07.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.308+0000 7f9c53e00640 1 -- 172.21.15.40:0/1784954899 shutdown_connections 2024-04-26T23:09:07.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.308+0000 7f9c53e00640 1 -- 172.21.15.40:0/1784954899 wait complete. 2024-04-26T23:09:07.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.309+0000 7f9c53e00640 1 Processor -- start 2024-04-26T23:09:07.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.310+0000 7f9c53e00640 1 -- start start 2024-04-26T23:09:07.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.310+0000 7f9c53e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c540cc590 0x7f9c54155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:07.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.310+0000 7f9c53e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c540ce5e0 0x7f9c5414b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:07.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c53e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c5415f2d0 0x7f9c54072c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:07.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c53e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9c540c5fc0 con 0x7f9c5415f2d0 2024-04-26T23:09:07.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c53e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9c5406e270 con 0x7f9c540cc590 2024-04-26T23:09:07.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c53e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9c540b0240 con 0x7f9c540ce5e0 2024-04-26T23:09:07.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c53400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c5415f2d0 0x7f9c54072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:07.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c53400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c5415f2d0 0x7f9c54072c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37862/0 (socket says 172.21.15.40:37862) 2024-04-26T23:09:07.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c53400640 1 -- 172.21.15.40:0/1684639474 learned_addr learned my addr 172.21.15.40:0/1684639474 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c52a00640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c540cc590 0x7f9c54155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c4be00640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c540ce5e0 0x7f9c5414b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c53400640 1 -- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c540ce5e0 msgr2=0x7f9c5414b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c53400640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c540ce5e0 0x7f9c5414b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c53400640 1 -- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c540cc590 msgr2=0x7f9c54155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c53400640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c540cc590 0x7f9c54155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c53400640 1 -- 172.21.15.40:0/1684639474 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c5414ba20 con 0x7f9c5415f2d0 2024-04-26T23:09:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.311+0000 7f9c52a00640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c540cc590 0x7f9c54155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.312+0000 7f9c53400640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c5415f2d0 0x7f9c54072c50 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00cd70 tx=0x7f9c4c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.312+0000 7f9c4aa00640 1 -- 172.21.15.40:0/1684639474 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c003b90 con 0x7f9c5415f2d0 2024-04-26T23:09:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.312+0000 7f9c4aa00640 1 -- 172.21.15.40:0/1684639474 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c4c003d30 con 0x7f9c5415f2d0 2024-04-26T23:09:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.312+0000 7f9c4aa00640 1 -- 172.21.15.40:0/1684639474 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c004240 con 0x7f9c5415f2d0 2024-04-26T23:09:07.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.312+0000 7f9c53e00640 1 -- 172.21.15.40:0/1684639474 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c541565a0 con 0x7f9c5415f2d0 2024-04-26T23:09:07.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.312+0000 7f9c53e00640 1 -- 172.21.15.40:0/1684639474 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c540b0490 con 0x7f9c5415f2d0 2024-04-26T23:09:07.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.313+0000 7f9c53e00640 1 -- 172.21.15.40:0/1684639474 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c54069370 con 0x7f9c5415f2d0 2024-04-26T23:09:07.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.315+0000 7f9c4aa00640 1 -- 172.21.15.40:0/1684639474 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9c4c0043e0 con 0x7f9c5415f2d0 2024-04-26T23:09:07.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.316+0000 7f9c4aa00640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9c3006c920 0x7f9c3006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:07.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.316+0000 7f9c4aa00640 1 -- 172.21.15.40:0/1684639474 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6101+0+0 (secure 0 0 0) 0x7f9c4c08c7a0 con 0x7f9c5415f2d0 2024-04-26T23:09:07.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.316+0000 7f9c52a00640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9c3006c920 0x7f9c3006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:07.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.317+0000 7f9c52a00640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9c3006c920 0x7f9c3006ede0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9c44000f60 tx=0x7f9c44009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:07.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.322+0000 7f9c4aa00640 1 -- 172.21.15.40:0/1684639474 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9c4c05aa30 con 0x7f9c5415f2d0 2024-04-26T23:09:07.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.535+0000 7f9c53e00640 1 -- 172.21.15.40:0/1684639474 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c5406de10 con 0x7f9c5415f2d0 2024-04-26T23:09:07.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.537+0000 7f9c4aa00640 1 -- 172.21.15.40:0/1684639474 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+16861 (secure 0 0 0) 0x7f9c4c05a330 con 0x7f9c5415f2d0 2024-04-26T23:09:07.538 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:07.538 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:09:06.116569+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:09:03.955148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-26T23:09:05.070627+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:09:07.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.540+0000 7f9c53e00640 1 -- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9c3006c920 msgr2=0x7f9c3006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:07.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.540+0000 7f9c53e00640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9c3006c920 0x7f9c3006ede0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9c44000f60 tx=0x7f9c44009040 comp rx=0 tx=0).stop 2024-04-26T23:09:07.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.540+0000 7f9c53e00640 1 -- 172.21.15.40:0/1684639474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c5415f2d0 msgr2=0x7f9c54072c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:07.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.540+0000 7f9c53e00640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c5415f2d0 0x7f9c54072c50 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00cd70 tx=0x7f9c4c00a660 comp rx=0 tx=0).stop 2024-04-26T23:09:07.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.540+0000 7f9c53e00640 1 -- 172.21.15.40:0/1684639474 shutdown_connections 2024-04-26T23:09:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.540+0000 7f9c53e00640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9c3006c920 0x7f9c3006ede0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.540+0000 7f9c53e00640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c5415f2d0 0x7f9c54072c50 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.540+0000 7f9c53e00640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9c540ce5e0 0x7f9c5414b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.540+0000 7f9c53e00640 1 --2- 172.21.15.40:0/1684639474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c540cc590 0x7f9c54155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.540+0000 7f9c53e00640 1 -- 172.21.15.40:0/1684639474 >> 172.21.15.40:0/1684639474 conn(0x7f9c540d0a10 msgr2=0x7f9c540ae8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.540+0000 7f9c53e00640 1 -- 172.21.15.40:0/1684639474 shutdown_connections 2024-04-26T23:09:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.540+0000 7f9c53e00640 1 -- 172.21.15.40:0/1684639474 wait complete. 2024-04-26T23:09:07.560 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T23:09:07.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.762+0000 7f0c17600640 1 Processor -- start 2024-04-26T23:09:07.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.763+0000 7f0c17600640 1 -- start start 2024-04-26T23:09:07.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.764+0000 7f0c17600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0c100d1a00 0x7f0c100d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:07.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.764+0000 7f0c17600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0c100b40d0 0x7f0c100b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:07.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.764+0000 7f0c17600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c100d0a10 con 0x7f0c100d2320 2024-04-26T23:09:07.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.764+0000 7f0c17600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f0c100d0cf0 con 0x7f0c100b40d0 2024-04-26T23:09:07.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.764+0000 7f0c17600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0c100d0fd0 con 0x7f0c100d1a00 2024-04-26T23:09:07.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.764+0000 7f0c0fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0c100d2320 0x7f0c101625f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49974/0 (socket says 172.21.15.40:49974) 2024-04-26T23:09:07.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.764+0000 7f0c0fe00640 1 -- 172.21.15.40:0/1190138339 learned_addr learned my addr 172.21.15.40:0/1190138339 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:07.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.764+0000 7f0c15600640 1 --2- 172.21.15.40:0/1190138339 >> v2:172.21.15.40:3300/0 conn(0x7f0c100b40d0 0x7f0c100b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:07.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.765+0000 7f0c14c00640 1 --2- 172.21.15.40:0/1190138339 >> v2:172.21.15.103:3300/0 conn(0x7f0c100d1a00 0x7f0c100d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:07.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.765+0000 7f0c15600640 1 -- 172.21.15.40:0/1190138339 >> v2:172.21.15.103:3300/0 conn(0x7f0c100d1a00 msgr2=0x7f0c100d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:07.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.765+0000 7f0c15600640 1 --2- 172.21.15.40:0/1190138339 >> v2:172.21.15.103:3300/0 conn(0x7f0c100d1a00 0x7f0c100d1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.765+0000 7f0c15600640 1 -- 172.21.15.40:0/1190138339 >> v1:172.21.15.40:6789/0 conn(0x7f0c100d2320 legacy=0x7f0c101625f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:07.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.765+0000 7f0c15600640 1 -- 172.21.15.40:0/1190138339 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c100d40a0 con 0x7f0c100b40d0 2024-04-26T23:09:07.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.766+0000 7f0c15600640 1 --2- 172.21.15.40:0/1190138339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c100b40d0 0x7f0c100b4b00 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f0c1006c650 tx=0x7f0c0802a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a2eced8e595b3d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:07.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.766+0000 7f0c0f400640 1 -- 172.21.15.40:0/1190138339 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c08037070 con 0x7f0c100b40d0 2024-04-26T23:09:07.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.766+0000 7f0c0f400640 1 -- 172.21.15.40:0/1190138339 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c0802add0 con 0x7f0c100b40d0 2024-04-26T23:09:07.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.766+0000 7f0c0f400640 1 -- 172.21.15.40:0/1190138339 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c08032660 con 0x7f0c100b40d0 2024-04-26T23:09:07.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.767+0000 7f0c17600640 1 -- 172.21.15.40:0/1190138339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c100b40d0 msgr2=0x7f0c100b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:07.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.767+0000 7f0c17600640 1 --2- 172.21.15.40:0/1190138339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c100b40d0 0x7f0c100b4b00 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f0c1006c650 tx=0x7f0c0802a080 comp rx=0 tx=0).stop 2024-04-26T23:09:07.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.767+0000 7f0c17600640 1 -- 172.21.15.40:0/1190138339 shutdown_connections 2024-04-26T23:09:07.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.767+0000 7f0c17600640 1 --2- 172.21.15.40:0/1190138339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c100b40d0 0x7f0c100b4b00 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.767+0000 7f0c17600640 1 --2- 172.21.15.40:0/1190138339 >> v2:172.21.15.103:3300/0 conn(0x7f0c100d1a00 0x7f0c100d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.767+0000 7f0c17600640 1 -- 172.21.15.40:0/1190138339 >> 172.21.15.40:0/1190138339 conn(0x7f0c100d4fd0 msgr2=0x7f0c100d53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:07.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.767+0000 7f0c17600640 1 -- 172.21.15.40:0/1190138339 shutdown_connections 2024-04-26T23:09:07.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.767+0000 7f0c17600640 1 -- 172.21.15.40:0/1190138339 wait complete. 2024-04-26T23:09:07.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.769+0000 7f0c17600640 1 Processor -- start 2024-04-26T23:09:07.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.769+0000 7f0c17600640 1 -- start start 2024-04-26T23:09:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.770+0000 7f0c17600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c100b40d0 0x7f0c101523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.770+0000 7f0c17600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c100d1a00 0x7f0c100cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.770+0000 7f0c17600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c100d2320 0x7f0c100c86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.770+0000 7f0c17600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0c100b5040 con 0x7f0c100d1a00 2024-04-26T23:09:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.770+0000 7f0c17600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0c10162f50 con 0x7f0c100d2320 2024-04-26T23:09:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.770+0000 7f0c17600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0c100b51c0 con 0x7f0c100b40d0 2024-04-26T23:09:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.770+0000 7f0c0fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c100d1a00 0x7f0c100cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.770+0000 7f0c0fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c100d1a00 0x7f0c100cdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37878/0 (socket says 172.21.15.40:37878) 2024-04-26T23:09:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.770+0000 7f0c0fe00640 1 -- 172.21.15.40:0/3356784837 learned_addr learned my addr 172.21.15.40:0/3356784837 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.771+0000 7f0c0fe00640 1 -- 172.21.15.40:0/3356784837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c100b40d0 msgr2=0x7f0c101523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.771+0000 7f0c0fe00640 1 --2- 172.21.15.40:0/3356784837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c100b40d0 0x7f0c101523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.771+0000 7f0c0fe00640 1 -- 172.21.15.40:0/3356784837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c100d2320 msgr2=0x7f0c100c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:07.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.771+0000 7f0c0fe00640 1 --2- 172.21.15.40:0/3356784837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c100d2320 0x7f0c100c86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.771+0000 7f0c0fe00640 1 -- 172.21.15.40:0/3356784837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c100cea30 con 0x7f0c100d1a00 2024-04-26T23:09:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.771+0000 7f0c0fe00640 1 --2- 172.21.15.40:0/3356784837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c100d1a00 0x7f0c100cdb00 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800c3c0 tx=0x7f0bf800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:07.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.771+0000 7f0c0cc00640 1 -- 172.21.15.40:0/3356784837 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8017070 con 0x7f0c100d1a00 2024-04-26T23:09:07.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.771+0000 7f0c0cc00640 1 -- 172.21.15.40:0/3356784837 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf8005ce0 con 0x7f0c100d1a00 2024-04-26T23:09:07.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.771+0000 7f0c0cc00640 1 -- 172.21.15.40:0/3356784837 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80072f0 con 0x7f0c100d1a00 2024-04-26T23:09:07.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.771+0000 7f0c17600640 1 -- 172.21.15.40:0/3356784837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c100ce470 con 0x7f0c100d1a00 2024-04-26T23:09:07.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.771+0000 7f0c17600640 1 -- 172.21.15.40:0/3356784837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c100b88f0 con 0x7f0c100d1a00 2024-04-26T23:09:07.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.772+0000 7f0c17600640 1 -- 172.21.15.40:0/3356784837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c100694f0 con 0x7f0c100d1a00 2024-04-26T23:09:07.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.773+0000 7f0c0cc00640 1 -- 172.21.15.40:0/3356784837 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0bf801a070 con 0x7f0c100d1a00 2024-04-26T23:09:07.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.774+0000 7f0c0cc00640 1 --2- 172.21.15.40:0/3356784837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0bec06c940 0x7f0bec06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:07.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.774+0000 7f0c0cc00640 1 -- 172.21.15.40:0/3356784837 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f0bf808b140 con 0x7f0c100d1a00 2024-04-26T23:09:07.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.775+0000 7f0c14c00640 1 --2- 172.21.15.40:0/3356784837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0bec06c940 0x7f0bec06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:07.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.776+0000 7f0c14c00640 1 --2- 172.21.15.40:0/3356784837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0bec06c940 0x7f0bec06ee00 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0c00007e60 tx=0x7f0c00007390 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:07.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:07.779+0000 7f0c0cc00640 1 -- 172.21.15.40:0/3356784837 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0bf80593c0 con 0x7f0c100d1a00 2024-04-26T23:09:08.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.039+0000 7f0c17600640 1 -- 172.21.15.40:0/3356784837 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0c100b5f50 con 0x7f0c100d1a00 2024-04-26T23:09:08.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.676+0000 7f0c0cc00640 1 -- 172.21.15.40:0/3356784837 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v110) ==== 118+0+0 (secure 0 0 0) 0x7f0bf8058cc0 con 0x7f0c100d1a00 2024-04-26T23:09:08.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.680+0000 7f0c17600640 1 -- 172.21.15.40:0/3356784837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0bec06c940 msgr2=0x7f0bec06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:08.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.680+0000 7f0c17600640 1 --2- 172.21.15.40:0/3356784837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0bec06c940 0x7f0bec06ee00 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0c00007e60 tx=0x7f0c00007390 comp rx=0 tx=0).stop 2024-04-26T23:09:08.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.680+0000 7f0c17600640 1 -- 172.21.15.40:0/3356784837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c100d1a00 msgr2=0x7f0c100cdb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:08.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.680+0000 7f0c17600640 1 --2- 172.21.15.40:0/3356784837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c100d1a00 0x7f0c100cdb00 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800c3c0 tx=0x7f0bf800c880 comp rx=0 tx=0).stop 2024-04-26T23:09:08.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.680+0000 7f0c17600640 1 -- 172.21.15.40:0/3356784837 shutdown_connections 2024-04-26T23:09:08.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.680+0000 7f0c17600640 1 --2- 172.21.15.40:0/3356784837 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0bec06c940 0x7f0bec06ee00 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:08.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.680+0000 7f0c17600640 1 --2- 172.21.15.40:0/3356784837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c100d2320 0x7f0c100c86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:08.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.680+0000 7f0c17600640 1 --2- 172.21.15.40:0/3356784837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c100d1a00 0x7f0c100cdb00 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:08.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.680+0000 7f0c17600640 1 --2- 172.21.15.40:0/3356784837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c100b40d0 0x7f0c101523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:08.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.680+0000 7f0c17600640 1 -- 172.21.15.40:0/3356784837 >> 172.21.15.40:0/3356784837 conn(0x7f0c100d4fd0 msgr2=0x7f0c10161660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:08.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.680+0000 7f0c17600640 1 -- 172.21.15.40:0/3356784837 shutdown_connections 2024-04-26T23:09:08.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.680+0000 7f0c17600640 1 -- 172.21.15.40:0/3356784837 wait complete. 2024-04-26T23:09:08.704 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-26T23:09:08.705 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-26T23:09:08.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.926+0000 7fd57c800640 1 Processor -- start 2024-04-26T23:09:08.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.927+0000 7fd57c800640 1 -- start start 2024-04-26T23:09:08.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.928+0000 7fd57c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5740cfde0 0x7fd5740d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:08.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.928+0000 7fd57c800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd574071260 0x7fd57415e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:08.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.928+0000 7fd57c800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5740d2390 con 0x7fd574070350 2024-04-26T23:09:08.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.928+0000 7fd57c800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd5740afab0 con 0x7fd574071260 2024-04-26T23:09:08.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.928+0000 7fd57c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd57406dd20 con 0x7fd5740cfde0 2024-04-26T23:09:08.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.929+0000 7fd579e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5740cfde0 0x7fd5740d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:08.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.929+0000 7fd579400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd574070350 0x7fd574070b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:55096/0 (socket says 172.21.15.40:55096) 2024-04-26T23:09:08.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.929+0000 7fd579400640 1 -- 172.21.15.40:0/323531101 learned_addr learned my addr 172.21.15.40:0/323531101 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:08.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.929+0000 7fd57a800640 1 --2- 172.21.15.40:0/323531101 >> v2:172.21.15.103:3300/0 conn(0x7fd574071260 0x7fd57415e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:08.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.930+0000 7fd579e00640 1 -- 172.21.15.40:0/323531101 >> v2:172.21.15.103:3300/0 conn(0x7fd574071260 msgr2=0x7fd57415e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:08.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.930+0000 7fd579e00640 1 --2- 172.21.15.40:0/323531101 >> v2:172.21.15.103:3300/0 conn(0x7fd574071260 0x7fd57415e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:08.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.930+0000 7fd579e00640 1 -- 172.21.15.40:0/323531101 >> v1:172.21.15.103:6789/0 conn(0x7fd574070350 legacy=0x7fd574070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:08.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.930+0000 7fd579e00640 1 -- 172.21.15.40:0/323531101 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd57406d3f0 con 0x7fd5740cfde0 2024-04-26T23:09:08.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.930+0000 7fd578a00640 1 -- 172.21.15.40:0/323531101 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 557363584 0 0) 0x7fd5740d2390 con 0x7fd574070350 2024-04-26T23:09:08.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.930+0000 7fd57a800640 1 --2- 172.21.15.40:0/323531101 >> v2:172.21.15.103:3300/0 conn(0x7fd574071260 0x7fd57415e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:09:08.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.932+0000 7fd579e00640 1 --2- 172.21.15.40:0/323531101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5740cfde0 0x7fd5740d1b60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd5680097a0 tx=0x7fd56802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59fd49129f9d3b66 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:08.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.932+0000 7fd578a00640 1 -- 172.21.15.40:0/323531101 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd568037070 con 0x7fd5740cfde0 2024-04-26T23:09:08.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.933+0000 7fd578a00640 1 -- 172.21.15.40:0/323531101 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd56802ae50 con 0x7fd5740cfde0 2024-04-26T23:09:08.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.933+0000 7fd578a00640 1 -- 172.21.15.40:0/323531101 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd568032660 con 0x7fd5740cfde0 2024-04-26T23:09:08.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.933+0000 7fd57c800640 1 -- 172.21.15.40:0/323531101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5740cfde0 msgr2=0x7fd5740d1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:08.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.933+0000 7fd57c800640 1 --2- 172.21.15.40:0/323531101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5740cfde0 0x7fd5740d1b60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd5680097a0 tx=0x7fd56802a100 comp rx=0 tx=0).stop 2024-04-26T23:09:08.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.933+0000 7fd57c800640 1 -- 172.21.15.40:0/323531101 shutdown_connections 2024-04-26T23:09:08.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.933+0000 7fd57c800640 1 --2- 172.21.15.40:0/323531101 >> v2:172.21.15.103:3300/0 conn(0x7fd574071260 0x7fd57415e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:08.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.933+0000 7fd57c800640 1 --2- 172.21.15.40:0/323531101 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5740cfde0 0x7fd5740d1b60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:08.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.933+0000 7fd57c800640 1 -- 172.21.15.40:0/323531101 >> 172.21.15.40:0/323531101 conn(0x7fd5740d2d80 msgr2=0x7fd5740d3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:08.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.933+0000 7fd57c800640 1 -- 172.21.15.40:0/323531101 shutdown_connections 2024-04-26T23:09:08.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.934+0000 7fd57c800640 1 -- 172.21.15.40:0/323531101 wait complete. 2024-04-26T23:09:08.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.935+0000 7fd57c800640 1 Processor -- start 2024-04-26T23:09:08.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.935+0000 7fd57c800640 1 -- start start 2024-04-26T23:09:08.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.936+0000 7fd57c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd574070350 0x7fd5741582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:08.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.936+0000 7fd57c800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd574071260 0x7fd574162060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:08.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.936+0000 7fd57c800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5740cfde0 0x7fd5741625a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:08.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.936+0000 7fd57c800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd5740d20a0 con 0x7fd5740cfde0 2024-04-26T23:09:08.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.936+0000 7fd57c800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd57406d9b0 con 0x7fd574071260 2024-04-26T23:09:08.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.936+0000 7fd57c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd57415ef30 con 0x7fd574070350 2024-04-26T23:09:08.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.937+0000 7fd57a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5740cfde0 0x7fd5741625a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:08.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.937+0000 7fd57a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5740cfde0 0x7fd5741625a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37890/0 (socket says 172.21.15.40:37890) 2024-04-26T23:09:08.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.937+0000 7fd57a800640 1 -- 172.21.15.40:0/2218763423 learned_addr learned my addr 172.21.15.40:0/2218763423 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:08.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.937+0000 7fd579e00640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd574070350 0x7fd5741582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:08.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.937+0000 7fd579400640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd574071260 0x7fd574162060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:08.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.937+0000 7fd57a800640 1 -- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd574070350 msgr2=0x7fd5741582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:08.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.937+0000 7fd57a800640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd574070350 0x7fd5741582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:08.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.937+0000 7fd57a800640 1 -- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd574071260 msgr2=0x7fd574162060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:08.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.937+0000 7fd57a800640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd574071260 0x7fd574162060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:08.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.937+0000 7fd57a800640 1 -- 172.21.15.40:0/2218763423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5740c6b00 con 0x7fd5740cfde0 2024-04-26T23:09:08.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.938+0000 7fd57a800640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5740cfde0 0x7fd5741625a0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fd57000efc0 tx=0x7fd57000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:08.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.938+0000 7fd56e000640 1 -- 172.21.15.40:0/2218763423 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd570018020 con 0x7fd5740cfde0 2024-04-26T23:09:08.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.938+0000 7fd56e000640 1 -- 172.21.15.40:0/2218763423 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd570014480 con 0x7fd5740cfde0 2024-04-26T23:09:08.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.938+0000 7fd56e000640 1 -- 172.21.15.40:0/2218763423 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd570014850 con 0x7fd5740cfde0 2024-04-26T23:09:08.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.938+0000 7fd57c800640 1 -- 172.21.15.40:0/2218763423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5740c6de0 con 0x7fd5740cfde0 2024-04-26T23:09:08.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.938+0000 7fd57c800640 1 -- 172.21.15.40:0/2218763423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5740c73a0 con 0x7fd5740cfde0 2024-04-26T23:09:08.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.939+0000 7fd57c800640 1 -- 172.21.15.40:0/2218763423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5740694f0 con 0x7fd5740cfde0 2024-04-26T23:09:08.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.941+0000 7fd56e000640 1 -- 172.21.15.40:0/2218763423 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd57001b020 con 0x7fd5740cfde0 2024-04-26T23:09:08.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.941+0000 7fd56e000640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd55806c8f0 0x7fd55806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:08.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.942+0000 7fd56e000640 1 -- 172.21.15.40:0/2218763423 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fd57008cb70 con 0x7fd5740cfde0 2024-04-26T23:09:08.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.942+0000 7fd579e00640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd55806c8f0 0x7fd55806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:08.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.943+0000 7fd579e00640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd55806c8f0 0x7fd55806edb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd568034000 tx=0x7fd56802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:08.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:08.946+0000 7fd56e000640 1 -- 172.21.15.40:0/2218763423 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd570022020 con 0x7fd5740cfde0 2024-04-26T23:09:09.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.193+0000 7fd57c800640 1 -- 172.21.15.40:0/2218763423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd5740afde0 con 0x7fd5740cfde0 2024-04-26T23:09:09.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.680+0000 7fd56e000640 1 -- 172.21.15.40:0/2218763423 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v111) ==== 122+0+0 (secure 0 0 0) 0x7fd570010040 con 0x7fd5740cfde0 2024-04-26T23:09:09.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.683+0000 7fd57c800640 1 -- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd55806c8f0 msgr2=0x7fd55806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:09.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.683+0000 7fd57c800640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd55806c8f0 0x7fd55806edb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd568034000 tx=0x7fd56802e000 comp rx=0 tx=0).stop 2024-04-26T23:09:09.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.683+0000 7fd57c800640 1 -- 172.21.15.40:0/2218763423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5740cfde0 msgr2=0x7fd5741625a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:09.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.683+0000 7fd57c800640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5740cfde0 0x7fd5741625a0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fd57000efc0 tx=0x7fd57000c490 comp rx=0 tx=0).stop 2024-04-26T23:09:09.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.684+0000 7fd57c800640 1 -- 172.21.15.40:0/2218763423 shutdown_connections 2024-04-26T23:09:09.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.684+0000 7fd57c800640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd55806c8f0 0x7fd55806edb0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:09.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.684+0000 7fd57c800640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5740cfde0 0x7fd5741625a0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:09.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.684+0000 7fd57c800640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd574071260 0x7fd574162060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:09.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.684+0000 7fd57c800640 1 --2- 172.21.15.40:0/2218763423 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd574070350 0x7fd5741582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:09.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.684+0000 7fd57c800640 1 -- 172.21.15.40:0/2218763423 >> 172.21.15.40:0/2218763423 conn(0x7fd5740d2d80 msgr2=0x7fd57415e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:09.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.684+0000 7fd57c800640 1 -- 172.21.15.40:0/2218763423 shutdown_connections 2024-04-26T23:09:09.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.684+0000 7fd57c800640 1 -- 172.21.15.40:0/2218763423 wait complete. 2024-04-26T23:09:09.707 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-26T23:09:09.707 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:09.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.904+0000 7fde40c00640 1 Processor -- start 2024-04-26T23:09:09.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.904+0000 7fde40c00640 1 -- start start 2024-04-26T23:09:09.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.905+0000 7fde40c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fde3c0cc480 0x7fde3c072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:09.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.905+0000 7fde40c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fde3c072780 0x7fde3c166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:09.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.905+0000 7fde40c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde3c15d7c0 con 0x7fde3c071e60 2024-04-26T23:09:09.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.905+0000 7fde40c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fde3c15daa0 con 0x7fde3c072780 2024-04-26T23:09:09.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.905+0000 7fde40c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fde3c15dd80 con 0x7fde3c0cc480 2024-04-26T23:09:09.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.905+0000 7fde3a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fde3c072780 0x7fde3c166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:09.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.905+0000 7fde3a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fde3c072780 0x7fde3c166ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37904/0 (socket says 172.21.15.40:37904) 2024-04-26T23:09:09.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.905+0000 7fde3a800640 1 -- 172.21.15.40:0/1753241944 learned_addr learned my addr 172.21.15.40:0/1753241944 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:09.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.906+0000 7fde33e00640 1 --2- 172.21.15.40:0/1753241944 >> v2:172.21.15.103:3300/0 conn(0x7fde3c0cc480 0x7fde3c072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:09.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.906+0000 7fde3a800640 1 -- 172.21.15.40:0/1753241944 >> v2:172.21.15.103:3300/0 conn(0x7fde3c0cc480 msgr2=0x7fde3c072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:09.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.906+0000 7fde3a800640 1 --2- 172.21.15.40:0/1753241944 >> v2:172.21.15.103:3300/0 conn(0x7fde3c0cc480 0x7fde3c072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:09.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.906+0000 7fde3a800640 1 -- 172.21.15.40:0/1753241944 >> v1:172.21.15.103:6789/0 conn(0x7fde3c071e60 legacy=0x7fde3c06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:09.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.906+0000 7fde3a800640 1 -- 172.21.15.40:0/1753241944 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c15e060 con 0x7fde3c072780 2024-04-26T23:09:09.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.907+0000 7fde3a800640 1 --2- 172.21.15.40:0/1753241944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3c072780 0x7fde3c166ad0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0b3820 tx=0x7fde3402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf3cd25cc768bf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:09.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.907+0000 7fde39400640 1 -- 172.21.15.40:0/1753241944 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34036070 con 0x7fde3c072780 2024-04-26T23:09:09.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.907+0000 7fde39400640 1 -- 172.21.15.40:0/1753241944 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde34031600 con 0x7fde3c072780 2024-04-26T23:09:09.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.907+0000 7fde39400640 1 -- 172.21.15.40:0/1753241944 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde340319d0 con 0x7fde3c072780 2024-04-26T23:09:09.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.908+0000 7fde40c00640 1 -- 172.21.15.40:0/1753241944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3c072780 msgr2=0x7fde3c166ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:09.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.908+0000 7fde40c00640 1 --2- 172.21.15.40:0/1753241944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3c072780 0x7fde3c166ad0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0b3820 tx=0x7fde3402a6f0 comp rx=0 tx=0).stop 2024-04-26T23:09:09.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.908+0000 7fde40c00640 1 -- 172.21.15.40:0/1753241944 shutdown_connections 2024-04-26T23:09:09.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.908+0000 7fde40c00640 1 --2- 172.21.15.40:0/1753241944 >> v2:172.21.15.103:3300/0 conn(0x7fde3c0cc480 0x7fde3c072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:09.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.908+0000 7fde40c00640 1 --2- 172.21.15.40:0/1753241944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3c072780 0x7fde3c166ad0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:09.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.908+0000 7fde40c00640 1 -- 172.21.15.40:0/1753241944 >> 172.21.15.40:0/1753241944 conn(0x7fde3c0d0be0 msgr2=0x7fde3c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:09.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.908+0000 7fde40c00640 1 -- 172.21.15.40:0/1753241944 shutdown_connections 2024-04-26T23:09:09.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.908+0000 7fde40c00640 1 -- 172.21.15.40:0/1753241944 wait complete. 2024-04-26T23:09:09.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.910+0000 7fde40c00640 1 Processor -- start 2024-04-26T23:09:09.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.910+0000 7fde40c00640 1 -- start start 2024-04-26T23:09:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.911+0000 7fde40c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde3c071e60 0x7fde3c0c6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.911+0000 7fde40c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde3c072780 0x7fde3c1ea8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.911+0000 7fde40c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3c0cc480 0x7fde3c1eade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.911+0000 7fde40c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c15d3f0 con 0x7fde3c0cc480 2024-04-26T23:09:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.911+0000 7fde40c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c072bd0 con 0x7fde3c072780 2024-04-26T23:09:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.911+0000 7fde40c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fde3c15d570 con 0x7fde3c071e60 2024-04-26T23:09:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.912+0000 7fde3a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3c0cc480 0x7fde3c1eade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.912+0000 7fde3a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3c0cc480 0x7fde3c1eade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37916/0 (socket says 172.21.15.40:37916) 2024-04-26T23:09:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.912+0000 7fde3a800640 1 -- 172.21.15.40:0/1768275981 learned_addr learned my addr 172.21.15.40:0/1768275981 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.912+0000 7fde33e00640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde3c072780 0x7fde3c1ea8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.912+0000 7fde39e00640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde3c071e60 0x7fde3c0c6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.912+0000 7fde3a800640 1 -- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde3c071e60 msgr2=0x7fde3c0c6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.912+0000 7fde3a800640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde3c071e60 0x7fde3c0c6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.912+0000 7fde3a800640 1 -- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde3c072780 msgr2=0x7fde3c1ea8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.912+0000 7fde3a800640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde3c072780 0x7fde3c1ea8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.912+0000 7fde3a800640 1 -- 172.21.15.40:0/1768275981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c0c6cd0 con 0x7fde3c0cc480 2024-04-26T23:09:09.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.913+0000 7fde3a800640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3c0cc480 0x7fde3c1eade0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fde3402ace0 tx=0x7fde34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:09.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.913+0000 7fde32000640 1 -- 172.21.15.40:0/1768275981 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34036050 con 0x7fde3c0cc480 2024-04-26T23:09:09.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.913+0000 7fde32000640 1 -- 172.21.15.40:0/1768275981 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde34042070 con 0x7fde3c0cc480 2024-04-26T23:09:09.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.913+0000 7fde40c00640 1 -- 172.21.15.40:0/1768275981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3c0b6830 con 0x7fde3c0cc480 2024-04-26T23:09:09.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.913+0000 7fde32000640 1 -- 172.21.15.40:0/1768275981 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34009510 con 0x7fde3c0cc480 2024-04-26T23:09:09.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.913+0000 7fde40c00640 1 -- 172.21.15.40:0/1768275981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde3c0b6550 con 0x7fde3c0cc480 2024-04-26T23:09:09.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.914+0000 7fde40c00640 1 -- 172.21.15.40:0/1768275981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde3c0694f0 con 0x7fde3c0cc480 2024-04-26T23:09:09.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.915+0000 7fde32000640 1 -- 172.21.15.40:0/1768275981 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fde34007bd0 con 0x7fde3c0cc480 2024-04-26T23:09:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.916+0000 7fde32000640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fde1806c8f0 0x7fde1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.916+0000 7fde32000640 1 -- 172.21.15.40:0/1768275981 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fde340acb90 con 0x7fde3c0cc480 2024-04-26T23:09:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.917+0000 7fde39e00640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fde1806c8f0 0x7fde1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:09.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.918+0000 7fde39e00640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fde1806c8f0 0x7fde1806edb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fde2c004ea0 tx=0x7fde2c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:09.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:09.921+0000 7fde32000640 1 -- 172.21.15.40:0/1768275981 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fde34045360 con 0x7fde3c0cc480 2024-04-26T23:09:10.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.165+0000 7fde40c00640 1 -- 172.21.15.40:0/1768275981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde3c06ce60 con 0x7fde3c0cc480 2024-04-26T23:09:10.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.167+0000 7fde32000640 1 -- 172.21.15.40:0/1768275981 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5192 (secure 0 0 0) 0x7fde3407ae20 con 0x7fde3c0cc480 2024-04-26T23:09:10.167 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:10.167 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":111,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":6598,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":8949,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:09.194635+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955]},"id":7}]} 2024-04-26T23:09:10.168 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 111 2024-04-26T23:09:10.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.170+0000 7fde40c00640 1 -- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fde1806c8f0 msgr2=0x7fde1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.170+0000 7fde40c00640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fde1806c8f0 0x7fde1806edb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fde2c004ea0 tx=0x7fde2c009040 comp rx=0 tx=0).stop 2024-04-26T23:09:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.170+0000 7fde40c00640 1 -- 172.21.15.40:0/1768275981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3c0cc480 msgr2=0x7fde3c1eade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.170+0000 7fde40c00640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3c0cc480 0x7fde3c1eade0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fde3402ace0 tx=0x7fde34004a40 comp rx=0 tx=0).stop 2024-04-26T23:09:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.170+0000 7fde40c00640 1 -- 172.21.15.40:0/1768275981 shutdown_connections 2024-04-26T23:09:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.170+0000 7fde40c00640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fde1806c8f0 0x7fde1806edb0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.170+0000 7fde40c00640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3c0cc480 0x7fde3c1eade0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.170+0000 7fde40c00640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde3c072780 0x7fde3c1ea8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.170+0000 7fde40c00640 1 --2- 172.21.15.40:0/1768275981 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fde3c071e60 0x7fde3c0c6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.170+0000 7fde40c00640 1 -- 172.21.15.40:0/1768275981 >> 172.21.15.40:0/1768275981 conn(0x7fde3c0d0be0 msgr2=0x7fde3c0cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.170+0000 7fde40c00640 1 -- 172.21.15.40:0/1768275981 shutdown_connections 2024-04-26T23:09:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.170+0000 7fde40c00640 1 -- 172.21.15.40:0/1768275981 wait complete. 2024-04-26T23:09:10.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:09:10.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.403+0000 7fccc0200640 1 Processor -- start 2024-04-26T23:09:10.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.403+0000 7fccc0200640 1 -- start start 2024-04-26T23:09:10.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.405+0000 7fccc0200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fccb815f380 0x7fccb80cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.405+0000 7fccc0200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccb80cfc00 con 0x7fccb8073b70 2024-04-26T23:09:10.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.405+0000 7fccc0200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccb8072c00 con 0x7fccb80d0620 2024-04-26T23:09:10.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.405+0000 7fccc0200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fccb80729b0 con 0x7fccb815f380 2024-04-26T23:09:10.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.405+0000 7fccbce00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fccb8073b70 0x7fccb80cede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49990/0 (socket says 172.21.15.40:49990) 2024-04-26T23:09:10.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.405+0000 7fccbce00640 1 -- 172.21.15.40:0/3701079388 learned_addr learned my addr 172.21.15.40:0/3701079388 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:10.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccbd800640 1 --2- 172.21.15.40:0/3701079388 >> v2:172.21.15.103:3300/0 conn(0x7fccb815f380 0x7fccb80cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:10.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1075943088 0 0) 0x7fccb80cfc00 con 0x7fccb8073b70 2024-04-26T23:09:10.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcca0003520 con 0x7fccb8073b70 2024-04-26T23:09:10.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 702023200 0 0) 0x7fcca0003520 con 0x7fccb8073b70 2024-04-26T23:09:10.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 >> v2:172.21.15.103:3300/0 conn(0x7fccb815f380 msgr2=0x7fccb80cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 --2- 172.21.15.40:0/3701079388 >> v2:172.21.15.103:3300/0 conn(0x7fccb815f380 0x7fccb80cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 >> v1:172.21.15.103:6789/0 conn(0x7fccb80d0620 legacy=0x7fccb80cf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb8072ee0 con 0x7fccb8073b70 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fcca4001670 con 0x7fccb8073b70 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 >> v1:172.21.15.40:6789/0 conn(0x7fccb8073b70 legacy=0x7fccb80cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 --2- 172.21.15.40:0/3701079388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcca00039b0 0x7fcca0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 --2- 172.21.15.40:0/3701079388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcca0004800 0x7fcca0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 --2- 172.21.15.40:0/3701079388 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcca00051c0 0x7fcca000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fccb80729b0 con 0x7fcca0004800 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcca000c060 con 0x7fcca00039b0 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.406+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcca000c260 con 0x7fcca00051c0 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.407+0000 7fccbce00640 1 --2- 172.21.15.40:0/3701079388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcca0004800 0x7fcca0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.407+0000 7fccbd800640 1 --2- 172.21.15.40:0/3701079388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcca00039b0 0x7fcca0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.407+0000 7fccbce00640 1 -- 172.21.15.40:0/3701079388 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcca00051c0 msgr2=0x7fcca000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:10.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.407+0000 7fccbce00640 1 --2- 172.21.15.40:0/3701079388 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcca00051c0 0x7fcca000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.407+0000 7fccbce00640 1 -- 172.21.15.40:0/3701079388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcca00039b0 msgr2=0x7fcca0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:10.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.407+0000 7fccbce00640 1 --2- 172.21.15.40:0/3701079388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcca00039b0 0x7fcca0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.407+0000 7fccbce00640 1 -- 172.21.15.40:0/3701079388 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca0003520 con 0x7fcca0004800 2024-04-26T23:09:10.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.407+0000 7fccbd800640 1 --2- 172.21.15.40:0/3701079388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcca00039b0 0x7fcca0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:10.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.407+0000 7fccbe200640 1 -- 172.21.15.40:0/3701079388 reap_dead start 2024-04-26T23:09:10.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.408+0000 7fccbce00640 1 --2- 172.21.15.40:0/3701079388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcca0004800 0x7fcca0004c80 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fcca400c240 tx=0x7fcca0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efcc3d5f54756437 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:10.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.408+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca4007cf0 con 0x7fcca0004800 2024-04-26T23:09:10.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.408+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcca40058d0 con 0x7fcca0004800 2024-04-26T23:09:10.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.408+0000 7fccb7e00640 1 -- 172.21.15.40:0/3701079388 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca4005d20 con 0x7fcca0004800 2024-04-26T23:09:10.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.409+0000 7fccc0200640 1 -- 172.21.15.40:0/3701079388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcca0004800 msgr2=0x7fcca0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:10.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.409+0000 7fccc0200640 1 --2- 172.21.15.40:0/3701079388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcca0004800 0x7fcca0004c80 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fcca400c240 tx=0x7fcca0005830 comp rx=0 tx=0).stop 2024-04-26T23:09:10.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.409+0000 7fccc0200640 1 -- 172.21.15.40:0/3701079388 shutdown_connections 2024-04-26T23:09:10.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.409+0000 7fccc0200640 1 --2- 172.21.15.40:0/3701079388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcca0004800 0x7fcca0004c80 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.409+0000 7fccc0200640 1 -- 172.21.15.40:0/3701079388 >> 172.21.15.40:0/3701079388 conn(0x7fccb80d0be0 msgr2=0x7fccb80d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:10.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.409+0000 7fccc0200640 1 -- 172.21.15.40:0/3701079388 shutdown_connections 2024-04-26T23:09:10.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.409+0000 7fccc0200640 1 -- 172.21.15.40:0/3701079388 wait complete. 2024-04-26T23:09:10.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.411+0000 7fccc0200640 1 Processor -- start 2024-04-26T23:09:10.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.411+0000 7fccc0200640 1 -- start start 2024-04-26T23:09:10.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.412+0000 7fccc0200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcca0004800 0x7fccb80b4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.412+0000 7fccc0200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccb80ca8b0 0x7fccb814fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.412+0000 7fccc0200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb81501a0 0x7fccb8150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.412+0000 7fccc0200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fccb80737a0 con 0x7fccb81501a0 2024-04-26T23:09:10.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.412+0000 7fccc0200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fccb8073620 con 0x7fcca0004800 2024-04-26T23:09:10.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.412+0000 7fccc0200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fccb8073920 con 0x7fccb80ca8b0 2024-04-26T23:09:10.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.412+0000 7fccbe200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb81501a0 0x7fccb8150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:10.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.412+0000 7fccbe200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb81501a0 0x7fccb8150ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37938/0 (socket says 172.21.15.40:37938) 2024-04-26T23:09:10.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.412+0000 7fccbe200640 1 -- 172.21.15.40:0/273756602 learned_addr learned my addr 172.21.15.40:0/273756602 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:10.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.413+0000 7fccbd800640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcca0004800 0x7fccb80b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:10.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.413+0000 7fccbce00640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccb80ca8b0 0x7fccb814fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:10.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.413+0000 7fccbe200640 1 -- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccb80ca8b0 msgr2=0x7fccb814fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:10.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.413+0000 7fccbe200640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccb80ca8b0 0x7fccb814fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.413+0000 7fccbe200640 1 -- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcca0004800 msgr2=0x7fccb80b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:10.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.413+0000 7fccbe200640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcca0004800 0x7fccb80b4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.413+0000 7fccbe200640 1 -- 172.21.15.40:0/273756602 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb80b3b30 con 0x7fccb81501a0 2024-04-26T23:09:10.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.413+0000 7fccbe200640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb81501a0 0x7fccb8150ac0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fccb8071b50 tx=0x7fccb000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:10.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.413+0000 7fccb5600640 1 -- 172.21.15.40:0/273756602 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0017070 con 0x7fccb81501a0 2024-04-26T23:09:10.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.413+0000 7fccb5600640 1 -- 172.21.15.40:0/273756602 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccb0005d40 con 0x7fccb81501a0 2024-04-26T23:09:10.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.413+0000 7fccc0200640 1 -- 172.21.15.40:0/273756602 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccb80c9810 con 0x7fccb81501a0 2024-04-26T23:09:10.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.414+0000 7fccb5600640 1 -- 172.21.15.40:0/273756602 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0007310 con 0x7fccb81501a0 2024-04-26T23:09:10.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.414+0000 7fccc0200640 1 -- 172.21.15.40:0/273756602 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccb80c9dd0 con 0x7fccb81501a0 2024-04-26T23:09:10.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.415+0000 7fccc0200640 1 -- 172.21.15.40:0/273756602 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccb80694f0 con 0x7fccb81501a0 2024-04-26T23:09:10.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.416+0000 7fccb5600640 1 -- 172.21.15.40:0/273756602 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fccb001a070 con 0x7fccb81501a0 2024-04-26T23:09:10.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.417+0000 7fccb5600640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcc9806c8f0 0x7fcc9806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.417+0000 7fccb5600640 1 -- 172.21.15.40:0/273756602 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fccb008afb0 con 0x7fccb81501a0 2024-04-26T23:09:10.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.418+0000 7fccbd800640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcc9806c8f0 0x7fcc9806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:10.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.419+0000 7fccbd800640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcc9806c8f0 0x7fcc9806edb0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fccb81f83e0 tx=0x7fccac008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:10.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.422+0000 7fccb5600640 1 -- 172.21.15.40:0/273756602 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fccb00591b0 con 0x7fccb81501a0 2024-04-26T23:09:10.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.630+0000 7fccc0200640 1 -- 172.21.15.40:0/273756602 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fccb80b1940 con 0x7fccb81501a0 2024-04-26T23:09:10.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.633+0000 7fccb5600640 1 -- 172.21.15.40:0/273756602 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16848 (secure 0 0 0) 0x7fccb0058ab0 con 0x7fccb81501a0 2024-04-26T23:09:10.633 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:10.634 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":103,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:09:07.651462+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:09:03.955148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-26T23:09:05.070627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:09:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.635+0000 7fccc0200640 1 -- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcc9806c8f0 msgr2=0x7fcc9806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.635+0000 7fccc0200640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcc9806c8f0 0x7fcc9806edb0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fccb81f83e0 tx=0x7fccac008040 comp rx=0 tx=0).stop 2024-04-26T23:09:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.635+0000 7fccc0200640 1 -- 172.21.15.40:0/273756602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb81501a0 msgr2=0x7fccb8150ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.635+0000 7fccc0200640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb81501a0 0x7fccb8150ac0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fccb8071b50 tx=0x7fccb000cb90 comp rx=0 tx=0).stop 2024-04-26T23:09:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.636+0000 7fccc0200640 1 -- 172.21.15.40:0/273756602 shutdown_connections 2024-04-26T23:09:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.636+0000 7fccc0200640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcc9806c8f0 0x7fcc9806edb0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.636+0000 7fccc0200640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb81501a0 0x7fccb8150ac0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.636+0000 7fccc0200640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fccb80ca8b0 0x7fccb814fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.636+0000 7fccc0200640 1 --2- 172.21.15.40:0/273756602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcca0004800 0x7fccb80b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.636+0000 7fccc0200640 1 -- 172.21.15.40:0/273756602 >> 172.21.15.40:0/273756602 conn(0x7fccb80d0be0 msgr2=0x7fccb80cde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:10.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.636+0000 7fccc0200640 1 -- 172.21.15.40:0/273756602 shutdown_connections 2024-04-26T23:09:10.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.636+0000 7fccc0200640 1 -- 172.21.15.40:0/273756602 wait complete. 2024-04-26T23:09:10.656 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T23:09:10.656 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:09:10.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.863+0000 7feaec000640 1 Processor -- start 2024-04-26T23:09:10.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.863+0000 7feaec000640 1 -- start start 2024-04-26T23:09:10.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.864+0000 7feaec000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feae4071e60 0x7feae406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.864+0000 7feaec000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feae40cc480 0x7feae4166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.864+0000 7feaec000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feae415d7c0 con 0x7feae406fe20 2024-04-26T23:09:10.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.864+0000 7feaec000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7feae415daa0 con 0x7feae4071e60 2024-04-26T23:09:10.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.864+0000 7feaec000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feae415dd80 con 0x7feae40cc480 2024-04-26T23:09:10.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.865+0000 7feae9600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feae4071e60 0x7feae406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.865+0000 7feae8c00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7feae406fe20 0x7feae4072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:55126/0 (socket says 172.21.15.40:55126) 2024-04-26T23:09:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.865+0000 7feae8c00640 1 -- 172.21.15.40:0/1222491264 learned_addr learned my addr 172.21.15.40:0/1222491264 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.865+0000 7feaea000640 1 --2- 172.21.15.40:0/1222491264 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feae40cc480 0x7feae4166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.866+0000 7feae9600640 1 -- 172.21.15.40:0/1222491264 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feae40cc480 msgr2=0x7feae4166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.866+0000 7feae9600640 1 --2- 172.21.15.40:0/1222491264 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feae40cc480 0x7feae4166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.866+0000 7feae9600640 1 -- 172.21.15.40:0/1222491264 >> v1:172.21.15.103:6789/0 conn(0x7feae406fe20 legacy=0x7feae4072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:10.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.866+0000 7feae9600640 1 -- 172.21.15.40:0/1222491264 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae415e060 con 0x7feae4071e60 2024-04-26T23:09:10.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.868+0000 7feae9600640 1 --2- 172.21.15.40:0/1222491264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feae4071e60 0x7feae406f8e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fead80097a0 tx=0x7fead802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=477ce32d925b4394 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:10.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.868+0000 7feae3e00640 1 -- 172.21.15.40:0/1222491264 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead8037070 con 0x7feae4071e60 2024-04-26T23:09:10.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.868+0000 7feae3e00640 1 -- 172.21.15.40:0/1222491264 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fead8032440 con 0x7feae4071e60 2024-04-26T23:09:10.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.868+0000 7feae3e00640 1 -- 172.21.15.40:0/1222491264 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead8032810 con 0x7feae4071e60 2024-04-26T23:09:10.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.869+0000 7feaec000640 1 -- 172.21.15.40:0/1222491264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feae4071e60 msgr2=0x7feae406f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:10.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.869+0000 7feaec000640 1 --2- 172.21.15.40:0/1222491264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feae4071e60 0x7feae406f8e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fead80097a0 tx=0x7fead802a180 comp rx=0 tx=0).stop 2024-04-26T23:09:10.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.869+0000 7feaec000640 1 -- 172.21.15.40:0/1222491264 shutdown_connections 2024-04-26T23:09:10.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.869+0000 7feaec000640 1 --2- 172.21.15.40:0/1222491264 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feae40cc480 0x7feae4166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.869+0000 7feaec000640 1 --2- 172.21.15.40:0/1222491264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feae4071e60 0x7feae406f8e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.869+0000 7feaec000640 1 -- 172.21.15.40:0/1222491264 >> 172.21.15.40:0/1222491264 conn(0x7feae40d0be0 msgr2=0x7feae40d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:10.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.869+0000 7feaec000640 1 -- 172.21.15.40:0/1222491264 shutdown_connections 2024-04-26T23:09:10.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.869+0000 7feaec000640 1 -- 172.21.15.40:0/1222491264 wait complete. 2024-04-26T23:09:10.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.871+0000 7feaec000640 1 Processor -- start 2024-04-26T23:09:10.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.871+0000 7feaec000640 1 -- start start 2024-04-26T23:09:10.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.872+0000 7feaec000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feae406fe20 0x7feae40ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.872+0000 7feaec000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feae4071e60 0x7feae414e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.872+0000 7feaec000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feae40cc480 0x7feae40cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.872+0000 7feaec000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feae415d570 con 0x7feae40cc480 2024-04-26T23:09:10.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.872+0000 7feaec000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feae415d3f0 con 0x7feae406fe20 2024-04-26T23:09:10.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.872+0000 7feaec000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feae4072950 con 0x7feae4071e60 2024-04-26T23:09:10.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.873+0000 7feaea000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feae40cc480 0x7feae40cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:10.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.873+0000 7feaea000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feae40cc480 0x7feae40cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37940/0 (socket says 172.21.15.40:37940) 2024-04-26T23:09:10.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.873+0000 7feaea000640 1 -- 172.21.15.40:0/1767274377 learned_addr learned my addr 172.21.15.40:0/1767274377 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:10.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.873+0000 7feae8c00640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feae4071e60 0x7feae414e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:10.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.873+0000 7feae9600640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feae406fe20 0x7feae40ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:10.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.873+0000 7feaea000640 1 -- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feae4071e60 msgr2=0x7feae414e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:10.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.873+0000 7feaea000640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feae4071e60 0x7feae414e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.873+0000 7feaea000640 1 -- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feae406fe20 msgr2=0x7feae40ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:10.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.873+0000 7feaea000640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feae406fe20 0x7feae40ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:10.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.873+0000 7feaea000640 1 -- 172.21.15.40:0/1767274377 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae40c98b0 con 0x7feae40cc480 2024-04-26T23:09:10.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.873+0000 7feae9600640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feae406fe20 0x7feae40ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:10.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.874+0000 7feaea000640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feae40cc480 0x7feae40cf8d0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7feadc009cb0 tx=0x7feadc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:10.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.874+0000 7feae1600640 1 -- 172.21.15.40:0/1767274377 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feadc018020 con 0x7feae40cc480 2024-04-26T23:09:10.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.874+0000 7feae1600640 1 -- 172.21.15.40:0/1767274377 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feadc014480 con 0x7feae40cc480 2024-04-26T23:09:10.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.874+0000 7feae1600640 1 -- 172.21.15.40:0/1767274377 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feadc014870 con 0x7feae40cc480 2024-04-26T23:09:10.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.874+0000 7feaec000640 1 -- 172.21.15.40:0/1767274377 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae40c9b90 con 0x7feae40cc480 2024-04-26T23:09:10.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.874+0000 7feaec000640 1 -- 172.21.15.40:0/1767274377 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae40ca150 con 0x7feae40cc480 2024-04-26T23:09:10.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.875+0000 7feaec000640 1 -- 172.21.15.40:0/1767274377 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feae40694f0 con 0x7feae40cc480 2024-04-26T23:09:10.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.876+0000 7feae1600640 1 -- 172.21.15.40:0/1767274377 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feadc01b020 con 0x7feae40cc480 2024-04-26T23:09:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.877+0000 7feae1600640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7feac406c8f0 0x7feac406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.877+0000 7feae1600640 1 -- 172.21.15.40:0/1767274377 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7feadc08ca50 con 0x7feae40cc480 2024-04-26T23:09:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.878+0000 7feae9600640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7feac406c8f0 0x7feac406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:10.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.879+0000 7feae9600640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7feac406c8f0 0x7feac406edb0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fead8034000 tx=0x7fead802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:10.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:10.882+0000 7feae1600640 1 -- 172.21.15.40:0/1767274377 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7feadc022020 con 0x7feae40cc480 2024-04-26T23:09:11.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.084+0000 7feaec000640 1 -- 172.21.15.40:0/1767274377 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7feae40b0b80 con 0x7feac406c8f0 2024-04-26T23:09:11.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.104+0000 7feae1600640 1 -- 172.21.15.40:0/1767274377 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252466 (secure 0 0 0) 0x7feae40b0b80 con 0x7feac406c8f0 2024-04-26T23:09:11.106 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:11.110 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:09:11.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.111+0000 7feaec000640 1 -- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7feac406c8f0 msgr2=0x7feac406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:11.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.111+0000 7feaec000640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7feac406c8f0 0x7feac406edb0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fead8034000 tx=0x7fead802e000 comp rx=0 tx=0).stop 2024-04-26T23:09:11.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.111+0000 7feaec000640 1 -- 172.21.15.40:0/1767274377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feae40cc480 msgr2=0x7feae40cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:11.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.111+0000 7feaec000640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feae40cc480 0x7feae40cf8d0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7feadc009cb0 tx=0x7feadc00c490 comp rx=0 tx=0).stop 2024-04-26T23:09:11.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.111+0000 7feaec000640 1 -- 172.21.15.40:0/1767274377 shutdown_connections 2024-04-26T23:09:11.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.111+0000 7feaec000640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7feac406c8f0 0x7feac406edb0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.111+0000 7feaec000640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feae40cc480 0x7feae40cf8d0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.111+0000 7feaec000640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feae4071e60 0x7feae414e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.111+0000 7feaec000640 1 --2- 172.21.15.40:0/1767274377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feae406fe20 0x7feae40ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.111+0000 7feaec000640 1 -- 172.21.15.40:0/1767274377 >> 172.21.15.40:0/1767274377 conn(0x7feae40d0be0 msgr2=0x7feae40c7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:11.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.111+0000 7feaec000640 1 -- 172.21.15.40:0/1767274377 shutdown_connections 2024-04-26T23:09:11.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.111+0000 7feaec000640 1 -- 172.21.15.40:0/1767274377 wait complete. 2024-04-26T23:09:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-04-26T23:09:09.324249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260604,"kb_used_data":11148,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714116,"statfs":{"total":773094113280,"available":772827254784,"internally_reserved":0,"allocated":11415552,"data_stored":5937416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.715490"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666954+0000","last_change":"2024-04-26T23:09:06.669170+0000","last_active":"2024-04-26T23:09:07.666954+0000","last_peered":"2024-04-26T23:09:07.666954+0000","last_clean":"2024-04-26T23:09:07.666954+0000","last_became_active":"2024-04-26T23:09:06.668925+0000","last_became_peered":"2024-04-26T23:09:06.668925+0000","last_unstale":"2024-04-26T23:09:07.666954+0000","last_undegraded":"2024-04-26T23:09:07.666954+0000","last_fullsized":"2024-04-26T23:09:07.666954+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:42:49.224734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669919+0000","last_change":"2024-04-26T23:09:06.665749+0000","last_active":"2024-04-26T23:09:07.669919+0000","last_peered":"2024-04-26T23:09:07.669919+0000","last_clean":"2024-04-26T23:09:07.669919+0000","last_became_active":"2024-04-26T23:09:06.664461+0000","last_became_peered":"2024-04-26T23:09:06.664461+0000","last_unstale":"2024-04-26T23:09:07.669919+0000","last_undegraded":"2024-04-26T23:09:07.669919+0000","last_fullsized":"2024-04-26T23:09:07.669919+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:28:13.915388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666494+0000","last_change":"2024-04-26T23:09:06.679574+0000","last_active":"2024-04-26T23:09:07.666494+0000","last_peered":"2024-04-26T23:09:07.666494+0000","last_clean":"2024-04-26T23:09:07.666494+0000","last_became_active":"2024-04-26T23:09:06.678077+0000","last_became_peered":"2024-04-26T23:09:06.678077+0000","last_unstale":"2024-04-26T23:09:07.666494+0000","last_undegraded":"2024-04-26T23:09:07.666494+0000","last_fullsized":"2024-04-26T23:09:07.666494+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:16:58.028346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.666017+0000","last_change":"2024-04-26T23:09:06.663224+0000","last_active":"2024-04-26T23:09:06.666017+0000","last_peered":"2024-04-26T23:09:06.666017+0000","last_clean":"2024-04-26T23:09:06.666017+0000","last_became_active":"2024-04-26T23:09:06.662610+0000","last_became_peered":"2024-04-26T23:09:06.662610+0000","last_unstale":"2024-04-26T23:09:06.666017+0000","last_undegraded":"2024-04-26T23:09:06.666017+0000","last_fullsized":"2024-04-26T23:09:06.666017+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:56:12.324814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970315+0000","last_change":"2024-04-26T23:09:06.662796+0000","last_active":"2024-04-26T23:09:07.970315+0000","last_peered":"2024-04-26T23:09:07.970315+0000","last_clean":"2024-04-26T23:09:07.970315+0000","last_became_active":"2024-04-26T23:09:06.662415+0000","last_became_peered":"2024-04-26T23:09:06.662415+0000","last_unstale":"2024-04-26T23:09:07.970315+0000","last_undegraded":"2024-04-26T23:09:07.970315+0000","last_fullsized":"2024-04-26T23:09:07.970315+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:48:50.599486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668192+0000","last_change":"2024-04-26T23:09:06.656830+0000","last_active":"2024-04-26T23:09:07.668192+0000","last_peered":"2024-04-26T23:09:07.668192+0000","last_clean":"2024-04-26T23:09:07.668192+0000","last_became_active":"2024-04-26T23:09:06.656524+0000","last_became_peered":"2024-04-26T23:09:06.656524+0000","last_unstale":"2024-04-26T23:09:07.668192+0000","last_undegraded":"2024-04-26T23:09:07.668192+0000","last_fullsized":"2024-04-26T23:09:07.668192+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:02:01.858467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668004+0000","last_change":"2024-04-26T23:09:06.657369+0000","last_active":"2024-04-26T23:09:07.668004+0000","last_peered":"2024-04-26T23:09:07.668004+0000","last_clean":"2024-04-26T23:09:07.668004+0000","last_became_active":"2024-04-26T23:09:06.655004+0000","last_became_peered":"2024-04-26T23:09:06.655004+0000","last_unstale":"2024-04-26T23:09:07.668004+0000","last_undegraded":"2024-04-26T23:09:07.668004+0000","last_fullsized":"2024-04-26T23:09:07.668004+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:24.705194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874244+0000","last_change":"2024-04-26T23:09:06.672513+0000","last_active":"2024-04-26T23:09:07.874244+0000","last_peered":"2024-04-26T23:09:07.874244+0000","last_clean":"2024-04-26T23:09:07.874244+0000","last_became_active":"2024-04-26T23:09:06.672082+0000","last_became_peered":"2024-04-26T23:09:06.672082+0000","last_unstale":"2024-04-26T23:09:07.874244+0000","last_undegraded":"2024-04-26T23:09:07.874244+0000","last_fullsized":"2024-04-26T23:09:07.874244+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:15:34.664013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668927+0000","last_change":"2024-04-26T23:09:06.666071+0000","last_active":"2024-04-26T23:09:07.668927+0000","last_peered":"2024-04-26T23:09:07.668927+0000","last_clean":"2024-04-26T23:09:07.668927+0000","last_became_active":"2024-04-26T23:09:06.665749+0000","last_became_peered":"2024-04-26T23:09:06.665749+0000","last_unstale":"2024-04-26T23:09:07.668927+0000","last_undegraded":"2024-04-26T23:09:07.668927+0000","last_fullsized":"2024-04-26T23:09:07.668927+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:51.040958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.971187+0000","last_change":"2024-04-26T23:09:06.669131+0000","last_active":"2024-04-26T23:09:07.971187+0000","last_peered":"2024-04-26T23:09:07.971187+0000","last_clean":"2024-04-26T23:09:07.971187+0000","last_became_active":"2024-04-26T23:09:06.667784+0000","last_became_peered":"2024-04-26T23:09:06.667784+0000","last_unstale":"2024-04-26T23:09:07.971187+0000","last_undegraded":"2024-04-26T23:09:07.971187+0000","last_fullsized":"2024-04-26T23:09:07.971187+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:51.891505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.683321+0000","last_change":"2024-04-26T23:09:06.677164+0000","last_active":"2024-04-26T23:09:06.683321+0000","last_peered":"2024-04-26T23:09:06.683321+0000","last_clean":"2024-04-26T23:09:06.683321+0000","last_became_active":"2024-04-26T23:09:06.676801+0000","last_became_peered":"2024-04-26T23:09:06.676801+0000","last_unstale":"2024-04-26T23:09:06.683321+0000","last_undegraded":"2024-04-26T23:09:06.683321+0000","last_fullsized":"2024-04-26T23:09:06.683321+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:33.003760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.671769+0000","last_change":"2024-04-26T23:09:06.664832+0000","last_active":"2024-04-26T23:09:07.671769+0000","last_peered":"2024-04-26T23:09:07.671769+0000","last_clean":"2024-04-26T23:09:07.671769+0000","last_became_active":"2024-04-26T23:09:06.662967+0000","last_became_peered":"2024-04-26T23:09:06.662967+0000","last_unstale":"2024-04-26T23:09:07.671769+0000","last_undegraded":"2024-04-26T23:09:07.671769+0000","last_fullsized":"2024-04-26T23:09:07.671769+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:28:13.915388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.681991+0000","last_change":"2024-04-26T23:09:06.674961+0000","last_active":"2024-04-26T23:09:06.681991+0000","last_peered":"2024-04-26T23:09:06.681991+0000","last_clean":"2024-04-26T23:09:06.681991+0000","last_became_active":"2024-04-26T23:09:06.674684+0000","last_became_peered":"2024-04-26T23:09:06.674684+0000","last_unstale":"2024-04-26T23:09:06.681991+0000","last_undegraded":"2024-04-26T23:09:06.681991+0000","last_fullsized":"2024-04-26T23:09:06.681991+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:01.579191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.916619+0000","last_change":"2024-04-26T23:09:06.666045+0000","last_active":"2024-04-26T23:09:07.916619+0000","last_peered":"2024-04-26T23:09:07.916619+0000","last_clean":"2024-04-26T23:09:07.916619+0000","last_became_active":"2024-04-26T23:09:06.665751+0000","last_became_peered":"2024-04-26T23:09:06.665751+0000","last_unstale":"2024-04-26T23:09:07.916619+0000","last_undegraded":"2024-04-26T23:09:07.916619+0000","last_fullsized":"2024-04-26T23:09:07.916619+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:45.467614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.917043+0000","last_change":"2024-04-26T23:09:06.677462+0000","last_active":"2024-04-26T23:09:07.917043+0000","last_peered":"2024-04-26T23:09:07.917043+0000","last_clean":"2024-04-26T23:09:07.917043+0000","last_became_active":"2024-04-26T23:09:06.675457+0000","last_became_peered":"2024-04-26T23:09:06.675457+0000","last_unstale":"2024-04-26T23:09:07.917043+0000","last_undegraded":"2024-04-26T23:09:07.917043+0000","last_fullsized":"2024-04-26T23:09:07.917043+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:34:00.415090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.916617+0000","last_change":"2024-04-26T23:09:06.669110+0000","last_active":"2024-04-26T23:09:07.916617+0000","last_peered":"2024-04-26T23:09:07.916617+0000","last_clean":"2024-04-26T23:09:07.916617+0000","last_became_active":"2024-04-26T23:09:06.667973+0000","last_became_peered":"2024-04-26T23:09:06.667973+0000","last_unstale":"2024-04-26T23:09:07.916617+0000","last_undegraded":"2024-04-26T23:09:07.916617+0000","last_fullsized":"2024-04-26T23:09:07.916617+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:01.579191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.667058+0000","last_change":"2024-04-26T23:09:06.678151+0000","last_active":"2024-04-26T23:09:07.667058+0000","last_peered":"2024-04-26T23:09:07.667058+0000","last_clean":"2024-04-26T23:09:07.667058+0000","last_became_active":"2024-04-26T23:09:06.677841+0000","last_became_peered":"2024-04-26T23:09:06.677841+0000","last_unstale":"2024-04-26T23:09:07.667058+0000","last_undegraded":"2024-04-26T23:09:07.667058+0000","last_fullsized":"2024-04-26T23:09:07.667058+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:38:08.941438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.671135+0000","last_change":"2024-04-26T23:09:06.660183+0000","last_active":"2024-04-26T23:09:07.671135+0000","last_peered":"2024-04-26T23:09:07.671135+0000","last_clean":"2024-04-26T23:09:07.671135+0000","last_became_active":"2024-04-26T23:09:06.659849+0000","last_became_peered":"2024-04-26T23:09:06.659849+0000","last_unstale":"2024-04-26T23:09:07.671135+0000","last_undegraded":"2024-04-26T23:09:07.671135+0000","last_fullsized":"2024-04-26T23:09:07.671135+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:55:50.893638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666578+0000","last_change":"2024-04-26T23:09:06.660084+0000","last_active":"2024-04-26T23:09:07.666578+0000","last_peered":"2024-04-26T23:09:07.666578+0000","last_clean":"2024-04-26T23:09:07.666578+0000","last_became_active":"2024-04-26T23:09:06.659755+0000","last_became_peered":"2024-04-26T23:09:06.659755+0000","last_unstale":"2024-04-26T23:09:07.666578+0000","last_undegraded":"2024-04-26T23:09:07.666578+0000","last_fullsized":"2024-04-26T23:09:07.666578+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:02:08.796726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.671390+0000","last_change":"2024-04-26T23:09:06.666561+0000","last_active":"2024-04-26T23:09:07.671390+0000","last_peered":"2024-04-26T23:09:07.671390+0000","last_clean":"2024-04-26T23:09:07.671390+0000","last_became_active":"2024-04-26T23:09:06.662601+0000","last_became_peered":"2024-04-26T23:09:06.662601+0000","last_unstale":"2024-04-26T23:09:07.671390+0000","last_undegraded":"2024-04-26T23:09:07.671390+0000","last_fullsized":"2024-04-26T23:09:07.671390+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:58:11.926588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874311+0000","last_change":"2024-04-26T23:09:06.674879+0000","last_active":"2024-04-26T23:09:07.874311+0000","last_peered":"2024-04-26T23:09:07.874311+0000","last_clean":"2024-04-26T23:09:07.874311+0000","last_became_active":"2024-04-26T23:09:06.674618+0000","last_became_peered":"2024-04-26T23:09:06.674618+0000","last_unstale":"2024-04-26T23:09:07.874311+0000","last_undegraded":"2024-04-26T23:09:07.874311+0000","last_fullsized":"2024-04-26T23:09:07.874311+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:11:54.242600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970996+0000","last_change":"2024-04-26T23:09:06.674880+0000","last_active":"2024-04-26T23:09:07.970996+0000","last_peered":"2024-04-26T23:09:07.970996+0000","last_clean":"2024-04-26T23:09:07.970996+0000","last_became_active":"2024-04-26T23:09:06.674000+0000","last_became_peered":"2024-04-26T23:09:06.674000+0000","last_unstale":"2024-04-26T23:09:07.970996+0000","last_undegraded":"2024-04-26T23:09:07.970996+0000","last_fullsized":"2024-04-26T23:09:07.970996+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:02:01.858467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.665303+0000","last_change":"2024-04-26T23:09:06.666473+0000","last_active":"2024-04-26T23:09:07.665303+0000","last_peered":"2024-04-26T23:09:07.665303+0000","last_clean":"2024-04-26T23:09:07.665303+0000","last_became_active":"2024-04-26T23:09:06.666200+0000","last_became_peered":"2024-04-26T23:09:06.666200+0000","last_unstale":"2024-04-26T23:09:07.665303+0000","last_undegraded":"2024-04-26T23:09:07.665303+0000","last_fullsized":"2024-04-26T23:09:07.665303+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:58:09.666061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970934+0000","last_change":"2024-04-26T23:09:06.656044+0000","last_active":"2024-04-26T23:09:07.970934+0000","last_peered":"2024-04-26T23:09:07.970934+0000","last_clean":"2024-04-26T23:09:07.970934+0000","last_became_active":"2024-04-26T23:09:06.655744+0000","last_became_peered":"2024-04-26T23:09:06.655744+0000","last_unstale":"2024-04-26T23:09:07.970934+0000","last_undegraded":"2024-04-26T23:09:07.970934+0000","last_fullsized":"2024-04-26T23:09:07.970934+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:24.705194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.672386+0000","last_change":"2024-04-26T23:09:06.676459+0000","last_active":"2024-04-26T23:09:07.672386+0000","last_peered":"2024-04-26T23:09:07.672386+0000","last_clean":"2024-04-26T23:09:07.672386+0000","last_became_active":"2024-04-26T23:09:06.676174+0000","last_became_peered":"2024-04-26T23:09:06.676174+0000","last_unstale":"2024-04-26T23:09:07.672386+0000","last_undegraded":"2024-04-26T23:09:07.672386+0000","last_fullsized":"2024-04-26T23:09:07.672386+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:55:50.893638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.672068+0000","last_change":"2024-04-26T23:09:06.663998+0000","last_active":"2024-04-26T23:09:07.672068+0000","last_peered":"2024-04-26T23:09:07.672068+0000","last_clean":"2024-04-26T23:09:07.672068+0000","last_became_active":"2024-04-26T23:09:06.663686+0000","last_became_peered":"2024-04-26T23:09:06.663686+0000","last_unstale":"2024-04-26T23:09:07.672068+0000","last_undegraded":"2024-04-26T23:09:07.672068+0000","last_fullsized":"2024-04-26T23:09:07.672068+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:56:12.324814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.681030+0000","last_change":"2024-04-26T23:09:06.673082+0000","last_active":"2024-04-26T23:09:06.681030+0000","last_peered":"2024-04-26T23:09:06.681030+0000","last_clean":"2024-04-26T23:09:06.681030+0000","last_became_active":"2024-04-26T23:09:06.672753+0000","last_became_peered":"2024-04-26T23:09:06.672753+0000","last_unstale":"2024-04-26T23:09:06.681030+0000","last_undegraded":"2024-04-26T23:09:06.681030+0000","last_fullsized":"2024-04-26T23:09:06.681030+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:14.603835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669175+0000","last_change":"2024-04-26T23:09:06.679743+0000","last_active":"2024-04-26T23:09:07.669175+0000","last_peered":"2024-04-26T23:09:07.669175+0000","last_clean":"2024-04-26T23:09:07.669175+0000","last_became_active":"2024-04-26T23:09:06.679486+0000","last_became_peered":"2024-04-26T23:09:06.679486+0000","last_unstale":"2024-04-26T23:09:07.669175+0000","last_undegraded":"2024-04-26T23:09:07.669175+0000","last_fullsized":"2024-04-26T23:09:07.669175+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:50.209129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874708+0000","last_change":"2024-04-26T23:09:06.661913+0000","last_active":"2024-04-26T23:09:07.874708+0000","last_peered":"2024-04-26T23:09:07.874708+0000","last_clean":"2024-04-26T23:09:07.874708+0000","last_became_active":"2024-04-26T23:09:06.661615+0000","last_became_peered":"2024-04-26T23:09:06.661615+0000","last_unstale":"2024-04-26T23:09:07.874708+0000","last_undegraded":"2024-04-26T23:09:07.874708+0000","last_fullsized":"2024-04-26T23:09:07.874708+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:04:25.642147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.670667+0000","last_change":"2024-04-26T23:09:06.674052+0000","last_active":"2024-04-26T23:09:07.670667+0000","last_peered":"2024-04-26T23:09:07.670667+0000","last_clean":"2024-04-26T23:09:07.670667+0000","last_became_active":"2024-04-26T23:09:06.673669+0000","last_became_peered":"2024-04-26T23:09:06.673669+0000","last_unstale":"2024-04-26T23:09:07.670667+0000","last_undegraded":"2024-04-26T23:09:07.670667+0000","last_fullsized":"2024-04-26T23:09:07.670667+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:51.040958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668432+0000","last_change":"2024-04-26T23:09:06.656264+0000","last_active":"2024-04-26T23:09:07.668432+0000","last_peered":"2024-04-26T23:09:07.668432+0000","last_clean":"2024-04-26T23:09:07.668432+0000","last_became_active":"2024-04-26T23:09:06.653460+0000","last_became_peered":"2024-04-26T23:09:06.653460+0000","last_unstale":"2024-04-26T23:09:07.668432+0000","last_undegraded":"2024-04-26T23:09:07.668432+0000","last_fullsized":"2024-04-26T23:09:07.668432+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:54.748365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874070+0000","last_change":"2024-04-26T23:09:06.676934+0000","last_active":"2024-04-26T23:09:07.874070+0000","last_peered":"2024-04-26T23:09:07.874070+0000","last_clean":"2024-04-26T23:09:07.874070+0000","last_became_active":"2024-04-26T23:09:06.676678+0000","last_became_peered":"2024-04-26T23:09:06.676678+0000","last_unstale":"2024-04-26T23:09:07.874070+0000","last_undegraded":"2024-04-26T23:09:07.874070+0000","last_fullsized":"2024-04-26T23:09:07.874070+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:25:54.919118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.665539+0000","last_change":"2024-04-26T23:09:06.652396+0000","last_active":"2024-04-26T23:09:07.665539+0000","last_peered":"2024-04-26T23:09:07.665539+0000","last_clean":"2024-04-26T23:09:07.665539+0000","last_became_active":"2024-04-26T23:09:06.652099+0000","last_became_peered":"2024-04-26T23:09:06.652099+0000","last_unstale":"2024-04-26T23:09:07.665539+0000","last_undegraded":"2024-04-26T23:09:07.665539+0000","last_fullsized":"2024-04-26T23:09:07.665539+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:37:02.056219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"102'4","reported_seq":16,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.683469+0000","last_change":"2024-04-26T23:09:05.651659+0000","last_active":"2024-04-26T23:09:06.683469+0000","last_peered":"2024-04-26T23:09:06.683469+0000","last_clean":"2024-04-26T23:09:06.683469+0000","last_became_active":"2024-04-26T23:09:05.651111+0000","last_became_peered":"2024-04-26T23:09:05.651111+0000","last_unstale":"2024-04-26T23:09:06.683469+0000","last_undegraded":"2024-04-26T23:09:06.683469+0000","last_fullsized":"2024-04-26T23:09:06.683469+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:49.192120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874062+0000","last_change":"2024-04-26T23:09:06.675929+0000","last_active":"2024-04-26T23:09:07.874062+0000","last_peered":"2024-04-26T23:09:07.874062+0000","last_clean":"2024-04-26T23:09:07.874062+0000","last_became_active":"2024-04-26T23:09:06.675214+0000","last_became_peered":"2024-04-26T23:09:06.675214+0000","last_unstale":"2024-04-26T23:09:07.874062+0000","last_undegraded":"2024-04-26T23:09:07.874062+0000","last_fullsized":"2024-04-26T23:09:07.874062+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:05.404801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874794+0000","last_change":"2024-04-26T23:09:05.657155+0000","last_active":"2024-04-26T23:09:07.874794+0000","last_peered":"2024-04-26T23:09:07.874794+0000","last_clean":"2024-04-26T23:09:07.874794+0000","last_became_active":"2024-04-26T23:09:05.656853+0000","last_became_peered":"2024-04-26T23:09:05.656853+0000","last_unstale":"2024-04-26T23:09:07.874794+0000","last_undegraded":"2024-04-26T23:09:07.874794+0000","last_fullsized":"2024-04-26T23:09:07.874794+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:06.319287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.671094+0000","last_change":"2024-04-26T23:09:06.657417+0000","last_active":"2024-04-26T23:09:07.671094+0000","last_peered":"2024-04-26T23:09:07.671094+0000","last_clean":"2024-04-26T23:09:07.671094+0000","last_became_active":"2024-04-26T23:09:06.657081+0000","last_became_peered":"2024-04-26T23:09:06.657081+0000","last_unstale":"2024-04-26T23:09:07.671094+0000","last_undegraded":"2024-04-26T23:09:07.671094+0000","last_fullsized":"2024-04-26T23:09:07.671094+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:50.209129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.670993+0000","last_change":"2024-04-26T23:09:05.657829+0000","last_active":"2024-04-26T23:09:07.670993+0000","last_peered":"2024-04-26T23:09:07.670993+0000","last_clean":"2024-04-26T23:09:07.670993+0000","last_became_active":"2024-04-26T23:09:05.657537+0000","last_became_peered":"2024-04-26T23:09:05.657537+0000","last_unstale":"2024-04-26T23:09:07.670993+0000","last_undegraded":"2024-04-26T23:09:07.670993+0000","last_fullsized":"2024-04-26T23:09:07.670993+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:56.541839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.681505+0000","last_change":"2024-04-26T23:09:06.673645+0000","last_active":"2024-04-26T23:09:06.681505+0000","last_peered":"2024-04-26T23:09:06.681505+0000","last_clean":"2024-04-26T23:09:06.681505+0000","last_became_active":"2024-04-26T23:09:06.673364+0000","last_became_peered":"2024-04-26T23:09:06.673364+0000","last_unstale":"2024-04-26T23:09:06.681505+0000","last_undegraded":"2024-04-26T23:09:06.681505+0000","last_fullsized":"2024-04-26T23:09:06.681505+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:07.566798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.971379+0000","last_change":"2024-04-26T23:09:05.657725+0000","last_active":"2024-04-26T23:09:07.971379+0000","last_peered":"2024-04-26T23:09:07.971379+0000","last_clean":"2024-04-26T23:09:07.971379+0000","last_became_active":"2024-04-26T23:09:05.655539+0000","last_became_peered":"2024-04-26T23:09:05.655539+0000","last_unstale":"2024-04-26T23:09:07.971379+0000","last_undegraded":"2024-04-26T23:09:07.971379+0000","last_fullsized":"2024-04-26T23:09:07.971379+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:41.337351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.677643+0000","last_change":"2024-04-26T23:09:06.667432+0000","last_active":"2024-04-26T23:09:06.677643+0000","last_peered":"2024-04-26T23:09:06.677643+0000","last_clean":"2024-04-26T23:09:06.677643+0000","last_became_active":"2024-04-26T23:09:06.665746+0000","last_became_peered":"2024-04-26T23:09:06.665746+0000","last_unstale":"2024-04-26T23:09:06.677643+0000","last_undegraded":"2024-04-26T23:09:06.677643+0000","last_fullsized":"2024-04-26T23:09:06.677643+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:55:50.893638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.916841+0000","last_change":"2024-04-26T23:09:05.658546+0000","last_active":"2024-04-26T23:09:07.916841+0000","last_peered":"2024-04-26T23:09:07.916841+0000","last_clean":"2024-04-26T23:09:07.916841+0000","last_became_active":"2024-04-26T23:09:05.658110+0000","last_became_peered":"2024-04-26T23:09:05.658110+0000","last_unstale":"2024-04-26T23:09:07.916841+0000","last_undegraded":"2024-04-26T23:09:07.916841+0000","last_fullsized":"2024-04-26T23:09:07.916841+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:50.874496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666921+0000","last_change":"2024-04-26T23:09:06.660056+0000","last_active":"2024-04-26T23:09:07.666921+0000","last_peered":"2024-04-26T23:09:07.666921+0000","last_clean":"2024-04-26T23:09:07.666921+0000","last_became_active":"2024-04-26T23:09:06.659328+0000","last_became_peered":"2024-04-26T23:09:06.659328+0000","last_unstale":"2024-04-26T23:09:07.666921+0000","last_undegraded":"2024-04-26T23:09:07.666921+0000","last_fullsized":"2024-04-26T23:09:07.666921+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:09.802401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"102'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.678691+0000","last_change":"2024-04-26T23:09:05.663211+0000","last_active":"2024-04-26T23:09:06.678691+0000","last_peered":"2024-04-26T23:09:06.678691+0000","last_clean":"2024-04-26T23:09:06.678691+0000","last_became_active":"2024-04-26T23:09:05.662769+0000","last_became_peered":"2024-04-26T23:09:05.662769+0000","last_unstale":"2024-04-26T23:09:06.678691+0000","last_undegraded":"2024-04-26T23:09:06.678691+0000","last_fullsized":"2024-04-26T23:09:06.678691+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:04:08.972842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.670334+0000","last_change":"2024-04-26T23:09:06.675356+0000","last_active":"2024-04-26T23:09:07.670334+0000","last_peered":"2024-04-26T23:09:07.670334+0000","last_clean":"2024-04-26T23:09:07.670334+0000","last_became_active":"2024-04-26T23:09:06.674810+0000","last_became_peered":"2024-04-26T23:09:06.674810+0000","last_unstale":"2024-04-26T23:09:07.670334+0000","last_undegraded":"2024-04-26T23:09:07.670334+0000","last_fullsized":"2024-04-26T23:09:07.670334+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:56:12.324814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669818+0000","last_change":"2024-04-26T23:09:05.662055+0000","last_active":"2024-04-26T23:09:07.669818+0000","last_peered":"2024-04-26T23:09:07.669818+0000","last_clean":"2024-04-26T23:09:07.669818+0000","last_became_active":"2024-04-26T23:09:05.661716+0000","last_became_peered":"2024-04-26T23:09:05.661716+0000","last_unstale":"2024-04-26T23:09:07.669818+0000","last_undegraded":"2024-04-26T23:09:07.669818+0000","last_fullsized":"2024-04-26T23:09:07.669818+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:13.586825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.663424+0000","last_change":"2024-04-26T23:09:06.660257+0000","last_active":"2024-04-26T23:09:06.663424+0000","last_peered":"2024-04-26T23:09:06.663424+0000","last_clean":"2024-04-26T23:09:06.663424+0000","last_became_active":"2024-04-26T23:09:06.659964+0000","last_became_peered":"2024-04-26T23:09:06.659964+0000","last_unstale":"2024-04-26T23:09:06.663424+0000","last_undegraded":"2024-04-26T23:09:06.663424+0000","last_fullsized":"2024-04-26T23:09:06.663424+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:28:13.915388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.652242+0000","last_change":"2024-04-26T23:09:05.653029+0000","last_active":"2024-04-26T23:09:06.652242+0000","last_peered":"2024-04-26T23:09:06.652242+0000","last_clean":"2024-04-26T23:09:06.652242+0000","last_became_active":"2024-04-26T23:09:05.652717+0000","last_became_peered":"2024-04-26T23:09:05.652717+0000","last_unstale":"2024-04-26T23:09:06.652242+0000","last_undegraded":"2024-04-26T23:09:06.652242+0000","last_fullsized":"2024-04-26T23:09:06.652242+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:50.023949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.665650+0000","last_change":"2024-04-26T23:09:06.659026+0000","last_active":"2024-04-26T23:09:07.665650+0000","last_peered":"2024-04-26T23:09:07.665650+0000","last_clean":"2024-04-26T23:09:07.665650+0000","last_became_active":"2024-04-26T23:09:06.656679+0000","last_became_peered":"2024-04-26T23:09:06.656679+0000","last_unstale":"2024-04-26T23:09:07.665650+0000","last_undegraded":"2024-04-26T23:09:07.665650+0000","last_fullsized":"2024-04-26T23:09:07.665650+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:49:28.936175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.672287+0000","last_change":"2024-04-26T23:09:05.659767+0000","last_active":"2024-04-26T23:09:07.672287+0000","last_peered":"2024-04-26T23:09:07.672287+0000","last_clean":"2024-04-26T23:09:07.672287+0000","last_became_active":"2024-04-26T23:09:05.658427+0000","last_became_peered":"2024-04-26T23:09:05.658427+0000","last_unstale":"2024-04-26T23:09:07.672287+0000","last_undegraded":"2024-04-26T23:09:07.672287+0000","last_fullsized":"2024-04-26T23:09:07.672287+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:31.986750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970681+0000","last_change":"2024-04-26T23:09:06.658098+0000","last_active":"2024-04-26T23:09:07.970681+0000","last_peered":"2024-04-26T23:09:07.970681+0000","last_clean":"2024-04-26T23:09:07.970681+0000","last_became_active":"2024-04-26T23:09:06.657814+0000","last_became_peered":"2024-04-26T23:09:06.657814+0000","last_unstale":"2024-04-26T23:09:07.970681+0000","last_undegraded":"2024-04-26T23:09:07.970681+0000","last_fullsized":"2024-04-26T23:09:07.970681+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:23:06.112828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666879+0000","last_change":"2024-04-26T23:09:05.655699+0000","last_active":"2024-04-26T23:09:07.666879+0000","last_peered":"2024-04-26T23:09:07.666879+0000","last_clean":"2024-04-26T23:09:07.666879+0000","last_became_active":"2024-04-26T23:09:05.655426+0000","last_became_peered":"2024-04-26T23:09:05.655426+0000","last_unstale":"2024-04-26T23:09:07.666879+0000","last_undegraded":"2024-04-26T23:09:07.666879+0000","last_fullsized":"2024-04-26T23:09:07.666879+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:24.064319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874111+0000","last_change":"2024-04-26T23:09:06.663392+0000","last_active":"2024-04-26T23:09:07.874111+0000","last_peered":"2024-04-26T23:09:07.874111+0000","last_clean":"2024-04-26T23:09:07.874111+0000","last_became_active":"2024-04-26T23:09:06.663109+0000","last_became_peered":"2024-04-26T23:09:06.663109+0000","last_unstale":"2024-04-26T23:09:07.874111+0000","last_undegraded":"2024-04-26T23:09:07.874111+0000","last_fullsized":"2024-04-26T23:09:07.874111+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:39.712791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"102'3","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.917142+0000","last_change":"2024-04-26T23:09:05.660656+0000","last_active":"2024-04-26T23:09:07.917142+0000","last_peered":"2024-04-26T23:09:07.917142+0000","last_clean":"2024-04-26T23:09:07.917142+0000","last_became_active":"2024-04-26T23:09:05.659934+0000","last_became_peered":"2024-04-26T23:09:05.659934+0000","last_unstale":"2024-04-26T23:09:07.917142+0000","last_undegraded":"2024-04-26T23:09:07.917142+0000","last_fullsized":"2024-04-26T23:09:07.917142+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:50.023949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970494+0000","last_change":"2024-04-26T23:09:06.656814+0000","last_active":"2024-04-26T23:09:07.970494+0000","last_peered":"2024-04-26T23:09:07.970494+0000","last_clean":"2024-04-26T23:09:07.970494+0000","last_became_active":"2024-04-26T23:09:06.656548+0000","last_became_peered":"2024-04-26T23:09:06.656548+0000","last_unstale":"2024-04-26T23:09:07.970494+0000","last_undegraded":"2024-04-26T23:09:07.970494+0000","last_fullsized":"2024-04-26T23:09:07.970494+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:32.801517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"102'6","reported_seq":20,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669454+0000","last_change":"2024-04-26T23:09:05.659052+0000","last_active":"2024-04-26T23:09:07.669454+0000","last_peered":"2024-04-26T23:09:07.669454+0000","last_clean":"2024-04-26T23:09:07.669454+0000","last_became_active":"2024-04-26T23:09:05.658758+0000","last_became_peered":"2024-04-26T23:09:05.658758+0000","last_unstale":"2024-04-26T23:09:07.669454+0000","last_undegraded":"2024-04-26T23:09:07.669454+0000","last_fullsized":"2024-04-26T23:09:07.669454+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:57.613194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.685791+0000","last_change":"2024-04-26T23:09:06.681695+0000","last_active":"2024-04-26T23:09:06.685791+0000","last_peered":"2024-04-26T23:09:06.685791+0000","last_clean":"2024-04-26T23:09:06.685791+0000","last_became_active":"2024-04-26T23:09:06.679733+0000","last_became_peered":"2024-04-26T23:09:06.679733+0000","last_unstale":"2024-04-26T23:09:06.685791+0000","last_undegraded":"2024-04-26T23:09:06.685791+0000","last_fullsized":"2024-04-26T23:09:06.685791+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:06:16.800754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970619+0000","last_change":"2024-04-26T23:09:05.657500+0000","last_active":"2024-04-26T23:09:07.970619+0000","last_peered":"2024-04-26T23:09:07.970619+0000","last_clean":"2024-04-26T23:09:07.970619+0000","last_became_active":"2024-04-26T23:09:05.656871+0000","last_became_peered":"2024-04-26T23:09:05.656871+0000","last_unstale":"2024-04-26T23:09:07.970619+0000","last_undegraded":"2024-04-26T23:09:07.970619+0000","last_fullsized":"2024-04-26T23:09:07.970619+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:57.613194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669348+0000","last_change":"2024-04-26T23:09:06.659893+0000","last_active":"2024-04-26T23:09:07.669348+0000","last_peered":"2024-04-26T23:09:07.669348+0000","last_clean":"2024-04-26T23:09:07.669348+0000","last_became_active":"2024-04-26T23:09:06.659535+0000","last_became_peered":"2024-04-26T23:09:06.659535+0000","last_unstale":"2024-04-26T23:09:07.669348+0000","last_undegraded":"2024-04-26T23:09:07.669348+0000","last_fullsized":"2024-04-26T23:09:07.669348+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:58:11.926588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.653977+0000","last_change":"2024-04-26T23:09:05.653759+0000","last_active":"2024-04-26T23:09:06.653977+0000","last_peered":"2024-04-26T23:09:06.653977+0000","last_clean":"2024-04-26T23:09:06.653977+0000","last_became_active":"2024-04-26T23:09:05.653448+0000","last_became_peered":"2024-04-26T23:09:05.653448+0000","last_unstale":"2024-04-26T23:09:06.653977+0000","last_undegraded":"2024-04-26T23:09:06.653977+0000","last_fullsized":"2024-04-26T23:09:06.653977+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:58:10.909579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874674+0000","last_change":"2024-04-26T23:09:06.675912+0000","last_active":"2024-04-26T23:09:07.874674+0000","last_peered":"2024-04-26T23:09:07.874674+0000","last_clean":"2024-04-26T23:09:07.874674+0000","last_became_active":"2024-04-26T23:09:06.675587+0000","last_became_peered":"2024-04-26T23:09:06.675587+0000","last_unstale":"2024-04-26T23:09:07.874674+0000","last_undegraded":"2024-04-26T23:09:07.874674+0000","last_fullsized":"2024-04-26T23:09:07.874674+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:01.067828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874412+0000","last_change":"2024-04-26T23:09:05.658978+0000","last_active":"2024-04-26T23:09:07.874412+0000","last_peered":"2024-04-26T23:09:07.874412+0000","last_clean":"2024-04-26T23:09:07.874412+0000","last_became_active":"2024-04-26T23:09:05.657667+0000","last_became_peered":"2024-04-26T23:09:05.657667+0000","last_unstale":"2024-04-26T23:09:07.874412+0000","last_undegraded":"2024-04-26T23:09:07.874412+0000","last_fullsized":"2024-04-26T23:09:07.874412+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:19:31.137605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.667100+0000","last_change":"2024-04-26T23:09:06.678771+0000","last_active":"2024-04-26T23:09:07.667100+0000","last_peered":"2024-04-26T23:09:07.667100+0000","last_clean":"2024-04-26T23:09:07.667100+0000","last_became_active":"2024-04-26T23:09:06.678470+0000","last_became_peered":"2024-04-26T23:09:06.678470+0000","last_unstale":"2024-04-26T23:09:07.667100+0000","last_undegraded":"2024-04-26T23:09:07.667100+0000","last_fullsized":"2024-04-26T23:09:07.667100+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:58:31.193256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"102'8","reported_seq":22,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668418+0000","last_change":"2024-04-26T23:09:05.660538+0000","last_active":"2024-04-26T23:09:07.668418+0000","last_peered":"2024-04-26T23:09:07.668418+0000","last_clean":"2024-04-26T23:09:07.668418+0000","last_became_active":"2024-04-26T23:09:05.660211+0000","last_became_peered":"2024-04-26T23:09:05.660211+0000","last_unstale":"2024-04-26T23:09:07.668418+0000","last_undegraded":"2024-04-26T23:09:07.668418+0000","last_fullsized":"2024-04-26T23:09:07.668418+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:12:31.044722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874542+0000","last_change":"2024-04-26T23:09:06.654675+0000","last_active":"2024-04-26T23:09:07.874542+0000","last_peered":"2024-04-26T23:09:07.874542+0000","last_clean":"2024-04-26T23:09:07.874542+0000","last_became_active":"2024-04-26T23:09:06.654369+0000","last_became_peered":"2024-04-26T23:09:06.654369+0000","last_unstale":"2024-04-26T23:09:07.874542+0000","last_undegraded":"2024-04-26T23:09:07.874542+0000","last_fullsized":"2024-04-26T23:09:07.874542+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:54.789576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669194+0000","last_change":"2024-04-26T23:09:05.656875+0000","last_active":"2024-04-26T23:09:07.669194+0000","last_peered":"2024-04-26T23:09:07.669194+0000","last_clean":"2024-04-26T23:09:07.669194+0000","last_became_active":"2024-04-26T23:09:05.656569+0000","last_became_peered":"2024-04-26T23:09:05.656569+0000","last_unstale":"2024-04-26T23:09:07.669194+0000","last_undegraded":"2024-04-26T23:09:07.669194+0000","last_fullsized":"2024-04-26T23:09:07.669194+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:00.713750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668596+0000","last_change":"2024-04-26T23:09:06.654678+0000","last_active":"2024-04-26T23:09:07.668596+0000","last_peered":"2024-04-26T23:09:07.668596+0000","last_clean":"2024-04-26T23:09:07.668596+0000","last_became_active":"2024-04-26T23:09:06.654326+0000","last_became_peered":"2024-04-26T23:09:06.654326+0000","last_unstale":"2024-04-26T23:09:07.668596+0000","last_undegraded":"2024-04-26T23:09:07.668596+0000","last_fullsized":"2024-04-26T23:09:07.668596+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:51.891505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.917127+0000","last_change":"2024-04-26T23:09:05.661198+0000","last_active":"2024-04-26T23:09:07.917127+0000","last_peered":"2024-04-26T23:09:07.917127+0000","last_clean":"2024-04-26T23:09:07.917127+0000","last_became_active":"2024-04-26T23:09:05.660742+0000","last_became_peered":"2024-04-26T23:09:05.660742+0000","last_unstale":"2024-04-26T23:09:07.917127+0000","last_undegraded":"2024-04-26T23:09:07.917127+0000","last_fullsized":"2024-04-26T23:09:07.917127+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:49.192120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.971157+0000","last_change":"2024-04-26T23:09:06.662581+0000","last_active":"2024-04-26T23:09:07.971157+0000","last_peered":"2024-04-26T23:09:07.971157+0000","last_clean":"2024-04-26T23:09:07.971157+0000","last_became_active":"2024-04-26T23:09:06.661964+0000","last_became_peered":"2024-04-26T23:09:06.661964+0000","last_unstale":"2024-04-26T23:09:07.971157+0000","last_undegraded":"2024-04-26T23:09:07.971157+0000","last_fullsized":"2024-04-26T23:09:07.971157+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:54.748365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.668275+0000","last_change":"2024-04-26T23:09:05.658524+0000","last_active":"2024-04-26T23:09:06.668275+0000","last_peered":"2024-04-26T23:09:06.668275+0000","last_clean":"2024-04-26T23:09:06.668275+0000","last_became_active":"2024-04-26T23:09:05.658233+0000","last_became_peered":"2024-04-26T23:09:05.658233+0000","last_unstale":"2024-04-26T23:09:06.668275+0000","last_undegraded":"2024-04-26T23:09:06.668275+0000","last_fullsized":"2024-04-26T23:09:06.668275+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:00.713750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666047+0000","last_change":"2024-04-26T23:09:06.664326+0000","last_active":"2024-04-26T23:09:07.666047+0000","last_peered":"2024-04-26T23:09:07.666047+0000","last_clean":"2024-04-26T23:09:07.666047+0000","last_became_active":"2024-04-26T23:09:06.663467+0000","last_became_peered":"2024-04-26T23:09:06.663467+0000","last_unstale":"2024-04-26T23:09:07.666047+0000","last_undegraded":"2024-04-26T23:09:07.666047+0000","last_fullsized":"2024-04-26T23:09:07.666047+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:49:42.578048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874744+0000","last_change":"2024-04-26T23:09:05.660355+0000","last_active":"2024-04-26T23:09:07.874744+0000","last_peered":"2024-04-26T23:09:07.874744+0000","last_clean":"2024-04-26T23:09:07.874744+0000","last_became_active":"2024-04-26T23:09:05.660077+0000","last_became_peered":"2024-04-26T23:09:05.660077+0000","last_unstale":"2024-04-26T23:09:07.874744+0000","last_undegraded":"2024-04-26T23:09:07.874744+0000","last_fullsized":"2024-04-26T23:09:07.874744+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:11:10.583556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666915+0000","last_change":"2024-04-26T23:09:06.662267+0000","last_active":"2024-04-26T23:09:07.666915+0000","last_peered":"2024-04-26T23:09:07.666915+0000","last_clean":"2024-04-26T23:09:07.666915+0000","last_became_active":"2024-04-26T23:09:06.661952+0000","last_became_peered":"2024-04-26T23:09:06.661952+0000","last_unstale":"2024-04-26T23:09:07.666915+0000","last_undegraded":"2024-04-26T23:09:07.666915+0000","last_fullsized":"2024-04-26T23:09:07.666915+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:37.386095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.873796+0000","last_change":"2024-04-26T23:09:05.657945+0000","last_active":"2024-04-26T23:09:07.873796+0000","last_peered":"2024-04-26T23:09:07.873796+0000","last_clean":"2024-04-26T23:09:07.873796+0000","last_became_active":"2024-04-26T23:09:05.657447+0000","last_became_peered":"2024-04-26T23:09:05.657447+0000","last_unstale":"2024-04-26T23:09:07.873796+0000","last_undegraded":"2024-04-26T23:09:07.873796+0000","last_fullsized":"2024-04-26T23:09:07.873796+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:23.522041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.678162+0000","last_change":"2024-04-26T23:09:06.670171+0000","last_active":"2024-04-26T23:09:06.678162+0000","last_peered":"2024-04-26T23:09:06.678162+0000","last_clean":"2024-04-26T23:09:06.678162+0000","last_became_active":"2024-04-26T23:09:06.669702+0000","last_became_peered":"2024-04-26T23:09:06.669702+0000","last_unstale":"2024-04-26T23:09:06.678162+0000","last_undegraded":"2024-04-26T23:09:06.678162+0000","last_fullsized":"2024-04-26T23:09:06.678162+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:00:38.519071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.670419+0000","last_change":"2024-04-26T23:09:05.657685+0000","last_active":"2024-04-26T23:09:07.670419+0000","last_peered":"2024-04-26T23:09:07.670419+0000","last_clean":"2024-04-26T23:09:07.670419+0000","last_became_active":"2024-04-26T23:09:05.657388+0000","last_became_peered":"2024-04-26T23:09:05.657388+0000","last_unstale":"2024-04-26T23:09:07.670419+0000","last_undegraded":"2024-04-26T23:09:07.670419+0000","last_fullsized":"2024-04-26T23:09:07.670419+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:41.337351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874751+0000","last_change":"2024-04-26T23:09:06.675924+0000","last_active":"2024-04-26T23:09:07.874751+0000","last_peered":"2024-04-26T23:09:07.874751+0000","last_clean":"2024-04-26T23:09:07.874751+0000","last_became_active":"2024-04-26T23:09:06.671516+0000","last_became_peered":"2024-04-26T23:09:06.671516+0000","last_unstale":"2024-04-26T23:09:07.874751+0000","last_undegraded":"2024-04-26T23:09:07.874751+0000","last_fullsized":"2024-04-26T23:09:07.874751+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:28.481514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874504+0000","last_change":"2024-04-26T23:09:05.658201+0000","last_active":"2024-04-26T23:09:07.874504+0000","last_peered":"2024-04-26T23:09:07.874504+0000","last_clean":"2024-04-26T23:09:07.874504+0000","last_became_active":"2024-04-26T23:09:05.657874+0000","last_became_peered":"2024-04-26T23:09:05.657874+0000","last_unstale":"2024-04-26T23:09:07.874504+0000","last_undegraded":"2024-04-26T23:09:07.874504+0000","last_fullsized":"2024-04-26T23:09:07.874504+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:01:31.728766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.665282+0000","last_change":"2024-04-26T23:09:06.663889+0000","last_active":"2024-04-26T23:09:07.665282+0000","last_peered":"2024-04-26T23:09:07.665282+0000","last_clean":"2024-04-26T23:09:07.665282+0000","last_became_active":"2024-04-26T23:09:06.663592+0000","last_became_peered":"2024-04-26T23:09:06.663592+0000","last_unstale":"2024-04-26T23:09:07.665282+0000","last_undegraded":"2024-04-26T23:09:07.665282+0000","last_fullsized":"2024-04-26T23:09:07.665282+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:02:43.649622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970649+0000","last_change":"2024-04-26T23:09:05.663246+0000","last_active":"2024-04-26T23:09:07.970649+0000","last_peered":"2024-04-26T23:09:07.970649+0000","last_clean":"2024-04-26T23:09:07.970649+0000","last_became_active":"2024-04-26T23:09:05.662964+0000","last_became_peered":"2024-04-26T23:09:05.662964+0000","last_unstale":"2024-04-26T23:09:07.970649+0000","last_undegraded":"2024-04-26T23:09:07.970649+0000","last_fullsized":"2024-04-26T23:09:07.970649+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:13.586825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669614+0000","last_change":"2024-04-26T23:09:06.663379+0000","last_active":"2024-04-26T23:09:07.669614+0000","last_peered":"2024-04-26T23:09:07.669614+0000","last_clean":"2024-04-26T23:09:07.669614+0000","last_became_active":"2024-04-26T23:09:06.663076+0000","last_became_peered":"2024-04-26T23:09:06.663076+0000","last_unstale":"2024-04-26T23:09:07.669614+0000","last_undegraded":"2024-04-26T23:09:07.669614+0000","last_fullsized":"2024-04-26T23:09:07.669614+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:51.891505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666890+0000","last_change":"2024-04-26T23:09:05.667415+0000","last_active":"2024-04-26T23:09:07.666890+0000","last_peered":"2024-04-26T23:09:07.666890+0000","last_clean":"2024-04-26T23:09:07.666890+0000","last_became_active":"2024-04-26T23:09:05.667214+0000","last_became_peered":"2024-04-26T23:09:05.667214+0000","last_unstale":"2024-04-26T23:09:07.666890+0000","last_undegraded":"2024-04-26T23:09:07.666890+0000","last_fullsized":"2024-04-26T23:09:07.666890+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:35:08.878677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.681988+0000","last_change":"2024-04-26T23:09:06.675212+0000","last_active":"2024-04-26T23:09:06.681988+0000","last_peered":"2024-04-26T23:09:06.681988+0000","last_clean":"2024-04-26T23:09:06.681988+0000","last_became_active":"2024-04-26T23:09:06.674932+0000","last_became_peered":"2024-04-26T23:09:06.674932+0000","last_unstale":"2024-04-26T23:09:06.681988+0000","last_undegraded":"2024-04-26T23:09:06.681988+0000","last_fullsized":"2024-04-26T23:09:06.681988+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:45.467614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669398+0000","last_change":"2024-04-26T23:09:05.656432+0000","last_active":"2024-04-26T23:09:07.669398+0000","last_peered":"2024-04-26T23:09:07.669398+0000","last_clean":"2024-04-26T23:09:07.669398+0000","last_became_active":"2024-04-26T23:09:05.656106+0000","last_became_peered":"2024-04-26T23:09:05.656106+0000","last_unstale":"2024-04-26T23:09:07.669398+0000","last_undegraded":"2024-04-26T23:09:07.669398+0000","last_fullsized":"2024-04-26T23:09:07.669398+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:22:13.434235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.683198+0000","last_change":"2024-04-26T23:09:06.675010+0000","last_active":"2024-04-26T23:09:06.683198+0000","last_peered":"2024-04-26T23:09:06.683198+0000","last_clean":"2024-04-26T23:09:06.683198+0000","last_became_active":"2024-04-26T23:09:06.674649+0000","last_became_peered":"2024-04-26T23:09:06.674649+0000","last_unstale":"2024-04-26T23:09:06.683198+0000","last_undegraded":"2024-04-26T23:09:06.683198+0000","last_fullsized":"2024-04-26T23:09:06.683198+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:12:32.061731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.971523+0000","last_change":"2024-04-26T23:09:05.664895+0000","last_active":"2024-04-26T23:09:07.971523+0000","last_peered":"2024-04-26T23:09:07.971523+0000","last_clean":"2024-04-26T23:09:07.971523+0000","last_became_active":"2024-04-26T23:09:05.664554+0000","last_became_peered":"2024-04-26T23:09:05.664554+0000","last_unstale":"2024-04-26T23:09:07.971523+0000","last_undegraded":"2024-04-26T23:09:07.971523+0000","last_fullsized":"2024-04-26T23:09:07.971523+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:24:14.206855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874500+0000","last_change":"2024-04-26T23:09:06.667186+0000","last_active":"2024-04-26T23:09:07.874500+0000","last_peered":"2024-04-26T23:09:07.874500+0000","last_clean":"2024-04-26T23:09:07.874500+0000","last_became_active":"2024-04-26T23:09:06.666897+0000","last_became_peered":"2024-04-26T23:09:06.666897+0000","last_unstale":"2024-04-26T23:09:07.874500+0000","last_undegraded":"2024-04-26T23:09:07.874500+0000","last_fullsized":"2024-04-26T23:09:07.874500+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:39:32.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.671663+0000","last_change":"2024-04-26T23:09:05.651495+0000","last_active":"2024-04-26T23:09:07.671663+0000","last_peered":"2024-04-26T23:09:07.671663+0000","last_clean":"2024-04-26T23:09:07.671663+0000","last_became_active":"2024-04-26T23:09:05.651177+0000","last_became_peered":"2024-04-26T23:09:05.651177+0000","last_unstale":"2024-04-26T23:09:07.671663+0000","last_undegraded":"2024-04-26T23:09:07.671663+0000","last_fullsized":"2024-04-26T23:09:07.671663+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:06.549789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874902+0000","last_change":"2024-04-26T23:09:06.669767+0000","last_active":"2024-04-26T23:09:07.874902+0000","last_peered":"2024-04-26T23:09:07.874902+0000","last_clean":"2024-04-26T23:09:07.874902+0000","last_became_active":"2024-04-26T23:09:06.669484+0000","last_became_peered":"2024-04-26T23:09:06.669484+0000","last_unstale":"2024-04-26T23:09:07.874902+0000","last_undegraded":"2024-04-26T23:09:07.874902+0000","last_fullsized":"2024-04-26T23:09:07.874902+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:09:27.722749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.873915+0000","last_change":"2024-04-26T23:09:05.653829+0000","last_active":"2024-04-26T23:09:07.873915+0000","last_peered":"2024-04-26T23:09:07.873915+0000","last_clean":"2024-04-26T23:09:07.873915+0000","last_became_active":"2024-04-26T23:09:05.653554+0000","last_became_peered":"2024-04-26T23:09:05.653554+0000","last_unstale":"2024-04-26T23:09:07.873915+0000","last_undegraded":"2024-04-26T23:09:07.873915+0000","last_fullsized":"2024-04-26T23:09:07.873915+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:10:01.882411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.917108+0000","last_change":"2024-04-26T23:09:06.680120+0000","last_active":"2024-04-26T23:09:07.917108+0000","last_peered":"2024-04-26T23:09:07.917108+0000","last_clean":"2024-04-26T23:09:07.917108+0000","last_became_active":"2024-04-26T23:09:06.679083+0000","last_became_peered":"2024-04-26T23:09:06.679083+0000","last_unstale":"2024-04-26T23:09:07.917108+0000","last_undegraded":"2024-04-26T23:09:07.917108+0000","last_fullsized":"2024-04-26T23:09:07.917108+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:17.381118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668821+0000","last_change":"2024-04-26T23:09:05.657447+0000","last_active":"2024-04-26T23:09:07.668821+0000","last_peered":"2024-04-26T23:09:07.668821+0000","last_clean":"2024-04-26T23:09:07.668821+0000","last_became_active":"2024-04-26T23:09:05.656875+0000","last_became_peered":"2024-04-26T23:09:05.656875+0000","last_unstale":"2024-04-26T23:09:07.668821+0000","last_undegraded":"2024-04-26T23:09:07.668821+0000","last_fullsized":"2024-04-26T23:09:07.668821+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:24:14.206855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.683233+0000","last_change":"2024-04-26T23:09:06.675016+0000","last_active":"2024-04-26T23:09:06.683233+0000","last_peered":"2024-04-26T23:09:06.683233+0000","last_clean":"2024-04-26T23:09:06.683233+0000","last_became_active":"2024-04-26T23:09:06.674609+0000","last_became_peered":"2024-04-26T23:09:06.674609+0000","last_unstale":"2024-04-26T23:09:06.683233+0000","last_undegraded":"2024-04-26T23:09:06.683233+0000","last_fullsized":"2024-04-26T23:09:06.683233+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:57.558848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874772+0000","last_change":"2024-04-26T23:09:05.653955+0000","last_active":"2024-04-26T23:09:07.874772+0000","last_peered":"2024-04-26T23:09:07.874772+0000","last_clean":"2024-04-26T23:09:07.874772+0000","last_became_active":"2024-04-26T23:09:05.653654+0000","last_became_peered":"2024-04-26T23:09:05.653654+0000","last_unstale":"2024-04-26T23:09:07.874772+0000","last_undegraded":"2024-04-26T23:09:07.874772+0000","last_fullsized":"2024-04-26T23:09:07.874772+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:37:20.196982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.675790+0000","last_change":"2024-04-26T23:09:06.666471+0000","last_active":"2024-04-26T23:09:06.675790+0000","last_peered":"2024-04-26T23:09:06.675790+0000","last_clean":"2024-04-26T23:09:06.675790+0000","last_became_active":"2024-04-26T23:09:06.666198+0000","last_became_peered":"2024-04-26T23:09:06.666198+0000","last_unstale":"2024-04-26T23:09:06.675790+0000","last_undegraded":"2024-04-26T23:09:06.675790+0000","last_fullsized":"2024-04-26T23:09:06.675790+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:22:14.451245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874912+0000","last_change":"2024-04-26T23:09:05.657794+0000","last_active":"2024-04-26T23:09:07.874912+0000","last_peered":"2024-04-26T23:09:07.874912+0000","last_clean":"2024-04-26T23:09:07.874912+0000","last_became_active":"2024-04-26T23:09:05.657429+0000","last_became_peered":"2024-04-26T23:09:05.657429+0000","last_unstale":"2024-04-26T23:09:07.874912+0000","last_undegraded":"2024-04-26T23:09:07.874912+0000","last_fullsized":"2024-04-26T23:09:07.874912+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:19.441312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":330,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.916279+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:09:07.916279+0000","last_peered":"2024-04-26T23:09:07.916279+0000","last_clean":"2024-04-26T23:09:07.916279+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:09:07.916279+0000","last_undegraded":"2024-04-26T23:09:07.916279+0000","last_fullsized":"2024-04-26T23:09:07.916279+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33008,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1867776,"data_stored":1182001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1298432,"data_stored":600381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1265664,"data_stored":591473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32436,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339404,"statfs":{"total":96636764160,"available":96603549696,"internally_reserved":0,"allocated":1286144,"data_stored":598257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1269760,"data_stored":592055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574951,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1269760,"data_stored":591507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1269760,"data_stored":591507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1888256,"data_stored":1190235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:09:11.135 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:09:11.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.378+0000 7fa7f0e00640 1 Processor -- start 2024-04-26T23:09:11.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.379+0000 7fa7f0e00640 1 -- start start 2024-04-26T23:09:11.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.380+0000 7fa7f0e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7ec162810 0x7fa7ec070350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.380+0000 7fa7f0e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa7ec0b2340 0x7fa7ec0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.380+0000 7fa7f0e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7ec0d10d0 con 0x7fa7ec0cd930 2024-04-26T23:09:11.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.380+0000 7fa7f0e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa7ec0d0750 con 0x7fa7ec162810 2024-04-26T23:09:11.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.380+0000 7fa7f0e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa7ec0d16a0 con 0x7fa7ec0b2340 2024-04-26T23:09:11.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.380+0000 7fa7e9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7ec162810 0x7fa7ec070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:11.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.380+0000 7fa7e9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7ec162810 0x7fa7ec070350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37942/0 (socket says 172.21.15.40:37942) 2024-04-26T23:09:11.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.380+0000 7fa7e9e00640 1 -- 172.21.15.40:0/364927201 learned_addr learned my addr 172.21.15.40:0/364927201 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:11.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.381+0000 7fa7e9400640 1 --2- 172.21.15.40:0/364927201 >> v2:172.21.15.103:3300/0 conn(0x7fa7ec0b2340 0x7fa7ec0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:11.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.381+0000 7fa7e9e00640 1 -- 172.21.15.40:0/364927201 >> v2:172.21.15.103:3300/0 conn(0x7fa7ec0b2340 msgr2=0x7fa7ec0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:11.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.381+0000 7fa7e9e00640 1 --2- 172.21.15.40:0/364927201 >> v2:172.21.15.103:3300/0 conn(0x7fa7ec0b2340 0x7fa7ec0cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.381+0000 7fa7e9e00640 1 -- 172.21.15.40:0/364927201 >> v1:172.21.15.40:6789/0 conn(0x7fa7ec0cd930 legacy=0x7fa7ec0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:11.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.381+0000 7fa7e9e00640 1 -- 172.21.15.40:0/364927201 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ec0d1370 con 0x7fa7ec162810 2024-04-26T23:09:11.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.381+0000 7fa7e8a00640 1 -- 172.21.15.40:0/364927201 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2261679946 0 0) 0x7fa7ec0d10d0 con 0x7fa7ec0cd930 2024-04-26T23:09:11.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.382+0000 7fa7e9e00640 1 --2- 172.21.15.40:0/364927201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec162810 0x7fa7ec070350 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec0cd0a0 tx=0x7fa7dc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af4fa58e6edaf18 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.382+0000 7fa7e8a00640 1 -- 172.21.15.40:0/364927201 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc037070 con 0x7fa7ec162810 2024-04-26T23:09:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.382+0000 7fa7e8a00640 1 -- 172.21.15.40:0/364927201 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7dc032440 con 0x7fa7ec162810 2024-04-26T23:09:11.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.382+0000 7fa7e8a00640 1 -- 172.21.15.40:0/364927201 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc032810 con 0x7fa7ec162810 2024-04-26T23:09:11.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.383+0000 7fa7f0e00640 1 -- 172.21.15.40:0/364927201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec162810 msgr2=0x7fa7ec070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:11.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.383+0000 7fa7f0e00640 1 --2- 172.21.15.40:0/364927201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec162810 0x7fa7ec070350 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec0cd0a0 tx=0x7fa7dc02a270 comp rx=0 tx=0).stop 2024-04-26T23:09:11.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.383+0000 7fa7f0e00640 1 -- 172.21.15.40:0/364927201 shutdown_connections 2024-04-26T23:09:11.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.383+0000 7fa7f0e00640 1 --2- 172.21.15.40:0/364927201 >> v2:172.21.15.103:3300/0 conn(0x7fa7ec0b2340 0x7fa7ec0cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.383+0000 7fa7f0e00640 1 --2- 172.21.15.40:0/364927201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec162810 0x7fa7ec070350 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.383+0000 7fa7f0e00640 1 -- 172.21.15.40:0/364927201 >> 172.21.15.40:0/364927201 conn(0x7fa7ec0d2da0 msgr2=0x7fa7ec0d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:11.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.383+0000 7fa7f0e00640 1 -- 172.21.15.40:0/364927201 shutdown_connections 2024-04-26T23:09:11.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.383+0000 7fa7f0e00640 1 -- 172.21.15.40:0/364927201 wait complete. 2024-04-26T23:09:11.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.385+0000 7fa7f0e00640 1 Processor -- start 2024-04-26T23:09:11.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.385+0000 7fa7f0e00640 1 -- start start 2024-04-26T23:09:11.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.386+0000 7fa7f0e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7ec0b2340 0x7fa7ec1521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.386+0000 7fa7f0e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec0cd930 0x7fa7ec0c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.386+0000 7fa7f0e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7ec162810 0x7fa7ec0cf410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.386+0000 7fa7f0e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec070890 con 0x7fa7ec0cd930 2024-04-26T23:09:11.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.386+0000 7fa7f0e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec0d0470 con 0x7fa7ec162810 2024-04-26T23:09:11.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.386+0000 7fa7f0e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa7ec0d2240 con 0x7fa7ec0b2340 2024-04-26T23:09:11.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.386+0000 7fa7e9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec0cd930 0x7fa7ec0c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:11.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.386+0000 7fa7e9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec0cd930 0x7fa7ec0c8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37954/0 (socket says 172.21.15.40:37954) 2024-04-26T23:09:11.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.386+0000 7fa7e9400640 1 -- 172.21.15.40:0/430695383 learned_addr learned my addr 172.21.15.40:0/430695383 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:11.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.387+0000 7fa7e9e00640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7ec0b2340 0x7fa7ec1521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:11.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.387+0000 7fa7ea800640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7ec162810 0x7fa7ec0cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:11.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.387+0000 7fa7e9400640 1 -- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7ec0b2340 msgr2=0x7fa7ec1521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:11.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.387+0000 7fa7e9400640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7ec0b2340 0x7fa7ec1521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.387+0000 7fa7e9400640 1 -- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7ec162810 msgr2=0x7fa7ec0cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.387+0000 7fa7e9400640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7ec162810 0x7fa7ec0cf410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.387+0000 7fa7e9400640 1 -- 172.21.15.40:0/430695383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ec0ca1e0 con 0x7fa7ec0cd930 2024-04-26T23:09:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.387+0000 7fa7e9e00640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7ec0b2340 0x7fa7ec1521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.387+0000 7fa7e9400640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec0cd930 0x7fa7ec0c8300 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4007ef0 tx=0x7fa7d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.387+0000 7fa7e2000640 1 -- 172.21.15.40:0/430695383 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4014620 con 0x7fa7ec0cd930 2024-04-26T23:09:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.387+0000 7fa7e2000640 1 -- 172.21.15.40:0/430695383 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d4019070 con 0x7fa7ec0cd930 2024-04-26T23:09:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.387+0000 7fa7e2000640 1 -- 172.21.15.40:0/430695383 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4014a00 con 0x7fa7ec0cd930 2024-04-26T23:09:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.388+0000 7fa7f0e00640 1 -- 172.21.15.40:0/430695383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7ec0ca4c0 con 0x7fa7ec0cd930 2024-04-26T23:09:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.388+0000 7fa7f0e00640 1 -- 172.21.15.40:0/430695383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7ec0caa80 con 0x7fa7ec0cd930 2024-04-26T23:09:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.389+0000 7fa7f0e00640 1 -- 172.21.15.40:0/430695383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7ec0694f0 con 0x7fa7ec0cd930 2024-04-26T23:09:11.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.390+0000 7fa7e2000640 1 -- 172.21.15.40:0/430695383 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa7d4014ba0 con 0x7fa7ec0cd930 2024-04-26T23:09:11.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.391+0000 7fa7e2000640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7c806c8f0 0x7fa7c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.392+0000 7fa7e2000640 1 -- 172.21.15.40:0/430695383 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fa7d408c780 con 0x7fa7ec0cd930 2024-04-26T23:09:11.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.392+0000 7fa7e9e00640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7c806c8f0 0x7fa7c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:11.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.393+0000 7fa7e9e00640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7c806c8f0 0x7fa7c806edb0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc034000 tx=0x7fa7dc02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:11.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.396+0000 7fa7e2000640 1 -- 172.21.15.40:0/430695383 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa7d405a8f0 con 0x7fa7ec0cd930 2024-04-26T23:09:11.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.598+0000 7fa7f0e00640 1 -- 172.21.15.40:0/430695383 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa7ec0cec60 con 0x7fa7c806c8f0 2024-04-26T23:09:11.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.617+0000 7fa7e2000640 1 -- 172.21.15.40:0/430695383 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252466 (secure 0 0 0) 0x7fa7ec0cec60 con 0x7fa7c806c8f0 2024-04-26T23:09:11.621 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:11.622 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:09:11.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.622+0000 7fa7f0e00640 1 -- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7c806c8f0 msgr2=0x7fa7c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:11.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.622+0000 7fa7f0e00640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7c806c8f0 0x7fa7c806edb0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc034000 tx=0x7fa7dc02e000 comp rx=0 tx=0).stop 2024-04-26T23:09:11.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.622+0000 7fa7f0e00640 1 -- 172.21.15.40:0/430695383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec0cd930 msgr2=0x7fa7ec0c8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:11.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.622+0000 7fa7f0e00640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec0cd930 0x7fa7ec0c8300 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4007ef0 tx=0x7fa7d400e4b0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.623+0000 7fa7f0e00640 1 -- 172.21.15.40:0/430695383 shutdown_connections 2024-04-26T23:09:11.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.623+0000 7fa7f0e00640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa7c806c8f0 0x7fa7c806edb0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.623+0000 7fa7f0e00640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7ec162810 0x7fa7ec0cf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.623+0000 7fa7f0e00640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7ec0cd930 0x7fa7ec0c8300 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.623+0000 7fa7f0e00640 1 --2- 172.21.15.40:0/430695383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7ec0b2340 0x7fa7ec1521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.623+0000 7fa7f0e00640 1 -- 172.21.15.40:0/430695383 >> 172.21.15.40:0/430695383 conn(0x7fa7ec0d2da0 msgr2=0x7fa7ec0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:11.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.623+0000 7fa7f0e00640 1 -- 172.21.15.40:0/430695383 shutdown_connections 2024-04-26T23:09:11.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.623+0000 7fa7f0e00640 1 -- 172.21.15.40:0/430695383 wait complete. 2024-04-26T23:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-04-26T23:09:11.325194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260604,"kb_used_data":11148,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714116,"statfs":{"total":773094113280,"available":772827254784,"internally_reserved":0,"allocated":11415552,"data_stored":5937416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002281"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666954+0000","last_change":"2024-04-26T23:09:06.669170+0000","last_active":"2024-04-26T23:09:07.666954+0000","last_peered":"2024-04-26T23:09:07.666954+0000","last_clean":"2024-04-26T23:09:07.666954+0000","last_became_active":"2024-04-26T23:09:06.668925+0000","last_became_peered":"2024-04-26T23:09:06.668925+0000","last_unstale":"2024-04-26T23:09:07.666954+0000","last_undegraded":"2024-04-26T23:09:07.666954+0000","last_fullsized":"2024-04-26T23:09:07.666954+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:42:49.224734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669919+0000","last_change":"2024-04-26T23:09:06.665749+0000","last_active":"2024-04-26T23:09:07.669919+0000","last_peered":"2024-04-26T23:09:07.669919+0000","last_clean":"2024-04-26T23:09:07.669919+0000","last_became_active":"2024-04-26T23:09:06.664461+0000","last_became_peered":"2024-04-26T23:09:06.664461+0000","last_unstale":"2024-04-26T23:09:07.669919+0000","last_undegraded":"2024-04-26T23:09:07.669919+0000","last_fullsized":"2024-04-26T23:09:07.669919+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:28:13.915388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666494+0000","last_change":"2024-04-26T23:09:06.679574+0000","last_active":"2024-04-26T23:09:07.666494+0000","last_peered":"2024-04-26T23:09:07.666494+0000","last_clean":"2024-04-26T23:09:07.666494+0000","last_became_active":"2024-04-26T23:09:06.678077+0000","last_became_peered":"2024-04-26T23:09:06.678077+0000","last_unstale":"2024-04-26T23:09:07.666494+0000","last_undegraded":"2024-04-26T23:09:07.666494+0000","last_fullsized":"2024-04-26T23:09:07.666494+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:16:58.028346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.666017+0000","last_change":"2024-04-26T23:09:06.663224+0000","last_active":"2024-04-26T23:09:06.666017+0000","last_peered":"2024-04-26T23:09:06.666017+0000","last_clean":"2024-04-26T23:09:06.666017+0000","last_became_active":"2024-04-26T23:09:06.662610+0000","last_became_peered":"2024-04-26T23:09:06.662610+0000","last_unstale":"2024-04-26T23:09:06.666017+0000","last_undegraded":"2024-04-26T23:09:06.666017+0000","last_fullsized":"2024-04-26T23:09:06.666017+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:56:12.324814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970315+0000","last_change":"2024-04-26T23:09:06.662796+0000","last_active":"2024-04-26T23:09:07.970315+0000","last_peered":"2024-04-26T23:09:07.970315+0000","last_clean":"2024-04-26T23:09:07.970315+0000","last_became_active":"2024-04-26T23:09:06.662415+0000","last_became_peered":"2024-04-26T23:09:06.662415+0000","last_unstale":"2024-04-26T23:09:07.970315+0000","last_undegraded":"2024-04-26T23:09:07.970315+0000","last_fullsized":"2024-04-26T23:09:07.970315+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:48:50.599486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668192+0000","last_change":"2024-04-26T23:09:06.656830+0000","last_active":"2024-04-26T23:09:07.668192+0000","last_peered":"2024-04-26T23:09:07.668192+0000","last_clean":"2024-04-26T23:09:07.668192+0000","last_became_active":"2024-04-26T23:09:06.656524+0000","last_became_peered":"2024-04-26T23:09:06.656524+0000","last_unstale":"2024-04-26T23:09:07.668192+0000","last_undegraded":"2024-04-26T23:09:07.668192+0000","last_fullsized":"2024-04-26T23:09:07.668192+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:02:01.858467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668004+0000","last_change":"2024-04-26T23:09:06.657369+0000","last_active":"2024-04-26T23:09:07.668004+0000","last_peered":"2024-04-26T23:09:07.668004+0000","last_clean":"2024-04-26T23:09:07.668004+0000","last_became_active":"2024-04-26T23:09:06.655004+0000","last_became_peered":"2024-04-26T23:09:06.655004+0000","last_unstale":"2024-04-26T23:09:07.668004+0000","last_undegraded":"2024-04-26T23:09:07.668004+0000","last_fullsized":"2024-04-26T23:09:07.668004+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:24.705194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874244+0000","last_change":"2024-04-26T23:09:06.672513+0000","last_active":"2024-04-26T23:09:07.874244+0000","last_peered":"2024-04-26T23:09:07.874244+0000","last_clean":"2024-04-26T23:09:07.874244+0000","last_became_active":"2024-04-26T23:09:06.672082+0000","last_became_peered":"2024-04-26T23:09:06.672082+0000","last_unstale":"2024-04-26T23:09:07.874244+0000","last_undegraded":"2024-04-26T23:09:07.874244+0000","last_fullsized":"2024-04-26T23:09:07.874244+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:15:34.664013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668927+0000","last_change":"2024-04-26T23:09:06.666071+0000","last_active":"2024-04-26T23:09:07.668927+0000","last_peered":"2024-04-26T23:09:07.668927+0000","last_clean":"2024-04-26T23:09:07.668927+0000","last_became_active":"2024-04-26T23:09:06.665749+0000","last_became_peered":"2024-04-26T23:09:06.665749+0000","last_unstale":"2024-04-26T23:09:07.668927+0000","last_undegraded":"2024-04-26T23:09:07.668927+0000","last_fullsized":"2024-04-26T23:09:07.668927+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:51.040958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.971187+0000","last_change":"2024-04-26T23:09:06.669131+0000","last_active":"2024-04-26T23:09:07.971187+0000","last_peered":"2024-04-26T23:09:07.971187+0000","last_clean":"2024-04-26T23:09:07.971187+0000","last_became_active":"2024-04-26T23:09:06.667784+0000","last_became_peered":"2024-04-26T23:09:06.667784+0000","last_unstale":"2024-04-26T23:09:07.971187+0000","last_undegraded":"2024-04-26T23:09:07.971187+0000","last_fullsized":"2024-04-26T23:09:07.971187+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:51.891505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.683321+0000","last_change":"2024-04-26T23:09:06.677164+0000","last_active":"2024-04-26T23:09:06.683321+0000","last_peered":"2024-04-26T23:09:06.683321+0000","last_clean":"2024-04-26T23:09:06.683321+0000","last_became_active":"2024-04-26T23:09:06.676801+0000","last_became_peered":"2024-04-26T23:09:06.676801+0000","last_unstale":"2024-04-26T23:09:06.683321+0000","last_undegraded":"2024-04-26T23:09:06.683321+0000","last_fullsized":"2024-04-26T23:09:06.683321+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:33.003760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.671769+0000","last_change":"2024-04-26T23:09:06.664832+0000","last_active":"2024-04-26T23:09:07.671769+0000","last_peered":"2024-04-26T23:09:07.671769+0000","last_clean":"2024-04-26T23:09:07.671769+0000","last_became_active":"2024-04-26T23:09:06.662967+0000","last_became_peered":"2024-04-26T23:09:06.662967+0000","last_unstale":"2024-04-26T23:09:07.671769+0000","last_undegraded":"2024-04-26T23:09:07.671769+0000","last_fullsized":"2024-04-26T23:09:07.671769+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:28:13.915388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.681991+0000","last_change":"2024-04-26T23:09:06.674961+0000","last_active":"2024-04-26T23:09:06.681991+0000","last_peered":"2024-04-26T23:09:06.681991+0000","last_clean":"2024-04-26T23:09:06.681991+0000","last_became_active":"2024-04-26T23:09:06.674684+0000","last_became_peered":"2024-04-26T23:09:06.674684+0000","last_unstale":"2024-04-26T23:09:06.681991+0000","last_undegraded":"2024-04-26T23:09:06.681991+0000","last_fullsized":"2024-04-26T23:09:06.681991+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:01.579191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.916619+0000","last_change":"2024-04-26T23:09:06.666045+0000","last_active":"2024-04-26T23:09:07.916619+0000","last_peered":"2024-04-26T23:09:07.916619+0000","last_clean":"2024-04-26T23:09:07.916619+0000","last_became_active":"2024-04-26T23:09:06.665751+0000","last_became_peered":"2024-04-26T23:09:06.665751+0000","last_unstale":"2024-04-26T23:09:07.916619+0000","last_undegraded":"2024-04-26T23:09:07.916619+0000","last_fullsized":"2024-04-26T23:09:07.916619+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:45.467614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.917043+0000","last_change":"2024-04-26T23:09:06.677462+0000","last_active":"2024-04-26T23:09:07.917043+0000","last_peered":"2024-04-26T23:09:07.917043+0000","last_clean":"2024-04-26T23:09:07.917043+0000","last_became_active":"2024-04-26T23:09:06.675457+0000","last_became_peered":"2024-04-26T23:09:06.675457+0000","last_unstale":"2024-04-26T23:09:07.917043+0000","last_undegraded":"2024-04-26T23:09:07.917043+0000","last_fullsized":"2024-04-26T23:09:07.917043+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:34:00.415090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.916617+0000","last_change":"2024-04-26T23:09:06.669110+0000","last_active":"2024-04-26T23:09:07.916617+0000","last_peered":"2024-04-26T23:09:07.916617+0000","last_clean":"2024-04-26T23:09:07.916617+0000","last_became_active":"2024-04-26T23:09:06.667973+0000","last_became_peered":"2024-04-26T23:09:06.667973+0000","last_unstale":"2024-04-26T23:09:07.916617+0000","last_undegraded":"2024-04-26T23:09:07.916617+0000","last_fullsized":"2024-04-26T23:09:07.916617+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:01.579191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.667058+0000","last_change":"2024-04-26T23:09:06.678151+0000","last_active":"2024-04-26T23:09:07.667058+0000","last_peered":"2024-04-26T23:09:07.667058+0000","last_clean":"2024-04-26T23:09:07.667058+0000","last_became_active":"2024-04-26T23:09:06.677841+0000","last_became_peered":"2024-04-26T23:09:06.677841+0000","last_unstale":"2024-04-26T23:09:07.667058+0000","last_undegraded":"2024-04-26T23:09:07.667058+0000","last_fullsized":"2024-04-26T23:09:07.667058+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:38:08.941438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.671135+0000","last_change":"2024-04-26T23:09:06.660183+0000","last_active":"2024-04-26T23:09:07.671135+0000","last_peered":"2024-04-26T23:09:07.671135+0000","last_clean":"2024-04-26T23:09:07.671135+0000","last_became_active":"2024-04-26T23:09:06.659849+0000","last_became_peered":"2024-04-26T23:09:06.659849+0000","last_unstale":"2024-04-26T23:09:07.671135+0000","last_undegraded":"2024-04-26T23:09:07.671135+0000","last_fullsized":"2024-04-26T23:09:07.671135+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:55:50.893638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666578+0000","last_change":"2024-04-26T23:09:06.660084+0000","last_active":"2024-04-26T23:09:07.666578+0000","last_peered":"2024-04-26T23:09:07.666578+0000","last_clean":"2024-04-26T23:09:07.666578+0000","last_became_active":"2024-04-26T23:09:06.659755+0000","last_became_peered":"2024-04-26T23:09:06.659755+0000","last_unstale":"2024-04-26T23:09:07.666578+0000","last_undegraded":"2024-04-26T23:09:07.666578+0000","last_fullsized":"2024-04-26T23:09:07.666578+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:02:08.796726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.671390+0000","last_change":"2024-04-26T23:09:06.666561+0000","last_active":"2024-04-26T23:09:07.671390+0000","last_peered":"2024-04-26T23:09:07.671390+0000","last_clean":"2024-04-26T23:09:07.671390+0000","last_became_active":"2024-04-26T23:09:06.662601+0000","last_became_peered":"2024-04-26T23:09:06.662601+0000","last_unstale":"2024-04-26T23:09:07.671390+0000","last_undegraded":"2024-04-26T23:09:07.671390+0000","last_fullsized":"2024-04-26T23:09:07.671390+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:58:11.926588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874311+0000","last_change":"2024-04-26T23:09:06.674879+0000","last_active":"2024-04-26T23:09:07.874311+0000","last_peered":"2024-04-26T23:09:07.874311+0000","last_clean":"2024-04-26T23:09:07.874311+0000","last_became_active":"2024-04-26T23:09:06.674618+0000","last_became_peered":"2024-04-26T23:09:06.674618+0000","last_unstale":"2024-04-26T23:09:07.874311+0000","last_undegraded":"2024-04-26T23:09:07.874311+0000","last_fullsized":"2024-04-26T23:09:07.874311+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:11:54.242600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970996+0000","last_change":"2024-04-26T23:09:06.674880+0000","last_active":"2024-04-26T23:09:07.970996+0000","last_peered":"2024-04-26T23:09:07.970996+0000","last_clean":"2024-04-26T23:09:07.970996+0000","last_became_active":"2024-04-26T23:09:06.674000+0000","last_became_peered":"2024-04-26T23:09:06.674000+0000","last_unstale":"2024-04-26T23:09:07.970996+0000","last_undegraded":"2024-04-26T23:09:07.970996+0000","last_fullsized":"2024-04-26T23:09:07.970996+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:02:01.858467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.665303+0000","last_change":"2024-04-26T23:09:06.666473+0000","last_active":"2024-04-26T23:09:07.665303+0000","last_peered":"2024-04-26T23:09:07.665303+0000","last_clean":"2024-04-26T23:09:07.665303+0000","last_became_active":"2024-04-26T23:09:06.666200+0000","last_became_peered":"2024-04-26T23:09:06.666200+0000","last_unstale":"2024-04-26T23:09:07.665303+0000","last_undegraded":"2024-04-26T23:09:07.665303+0000","last_fullsized":"2024-04-26T23:09:07.665303+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:58:09.666061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970934+0000","last_change":"2024-04-26T23:09:06.656044+0000","last_active":"2024-04-26T23:09:07.970934+0000","last_peered":"2024-04-26T23:09:07.970934+0000","last_clean":"2024-04-26T23:09:07.970934+0000","last_became_active":"2024-04-26T23:09:06.655744+0000","last_became_peered":"2024-04-26T23:09:06.655744+0000","last_unstale":"2024-04-26T23:09:07.970934+0000","last_undegraded":"2024-04-26T23:09:07.970934+0000","last_fullsized":"2024-04-26T23:09:07.970934+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:24.705194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.672386+0000","last_change":"2024-04-26T23:09:06.676459+0000","last_active":"2024-04-26T23:09:07.672386+0000","last_peered":"2024-04-26T23:09:07.672386+0000","last_clean":"2024-04-26T23:09:07.672386+0000","last_became_active":"2024-04-26T23:09:06.676174+0000","last_became_peered":"2024-04-26T23:09:06.676174+0000","last_unstale":"2024-04-26T23:09:07.672386+0000","last_undegraded":"2024-04-26T23:09:07.672386+0000","last_fullsized":"2024-04-26T23:09:07.672386+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:55:50.893638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.672068+0000","last_change":"2024-04-26T23:09:06.663998+0000","last_active":"2024-04-26T23:09:07.672068+0000","last_peered":"2024-04-26T23:09:07.672068+0000","last_clean":"2024-04-26T23:09:07.672068+0000","last_became_active":"2024-04-26T23:09:06.663686+0000","last_became_peered":"2024-04-26T23:09:06.663686+0000","last_unstale":"2024-04-26T23:09:07.672068+0000","last_undegraded":"2024-04-26T23:09:07.672068+0000","last_fullsized":"2024-04-26T23:09:07.672068+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:56:12.324814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.681030+0000","last_change":"2024-04-26T23:09:06.673082+0000","last_active":"2024-04-26T23:09:06.681030+0000","last_peered":"2024-04-26T23:09:06.681030+0000","last_clean":"2024-04-26T23:09:06.681030+0000","last_became_active":"2024-04-26T23:09:06.672753+0000","last_became_peered":"2024-04-26T23:09:06.672753+0000","last_unstale":"2024-04-26T23:09:06.681030+0000","last_undegraded":"2024-04-26T23:09:06.681030+0000","last_fullsized":"2024-04-26T23:09:06.681030+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:14.603835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669175+0000","last_change":"2024-04-26T23:09:06.679743+0000","last_active":"2024-04-26T23:09:07.669175+0000","last_peered":"2024-04-26T23:09:07.669175+0000","last_clean":"2024-04-26T23:09:07.669175+0000","last_became_active":"2024-04-26T23:09:06.679486+0000","last_became_peered":"2024-04-26T23:09:06.679486+0000","last_unstale":"2024-04-26T23:09:07.669175+0000","last_undegraded":"2024-04-26T23:09:07.669175+0000","last_fullsized":"2024-04-26T23:09:07.669175+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:50.209129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874708+0000","last_change":"2024-04-26T23:09:06.661913+0000","last_active":"2024-04-26T23:09:07.874708+0000","last_peered":"2024-04-26T23:09:07.874708+0000","last_clean":"2024-04-26T23:09:07.874708+0000","last_became_active":"2024-04-26T23:09:06.661615+0000","last_became_peered":"2024-04-26T23:09:06.661615+0000","last_unstale":"2024-04-26T23:09:07.874708+0000","last_undegraded":"2024-04-26T23:09:07.874708+0000","last_fullsized":"2024-04-26T23:09:07.874708+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:04:25.642147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.670667+0000","last_change":"2024-04-26T23:09:06.674052+0000","last_active":"2024-04-26T23:09:07.670667+0000","last_peered":"2024-04-26T23:09:07.670667+0000","last_clean":"2024-04-26T23:09:07.670667+0000","last_became_active":"2024-04-26T23:09:06.673669+0000","last_became_peered":"2024-04-26T23:09:06.673669+0000","last_unstale":"2024-04-26T23:09:07.670667+0000","last_undegraded":"2024-04-26T23:09:07.670667+0000","last_fullsized":"2024-04-26T23:09:07.670667+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:51.040958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668432+0000","last_change":"2024-04-26T23:09:06.656264+0000","last_active":"2024-04-26T23:09:07.668432+0000","last_peered":"2024-04-26T23:09:07.668432+0000","last_clean":"2024-04-26T23:09:07.668432+0000","last_became_active":"2024-04-26T23:09:06.653460+0000","last_became_peered":"2024-04-26T23:09:06.653460+0000","last_unstale":"2024-04-26T23:09:07.668432+0000","last_undegraded":"2024-04-26T23:09:07.668432+0000","last_fullsized":"2024-04-26T23:09:07.668432+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:54.748365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874070+0000","last_change":"2024-04-26T23:09:06.676934+0000","last_active":"2024-04-26T23:09:07.874070+0000","last_peered":"2024-04-26T23:09:07.874070+0000","last_clean":"2024-04-26T23:09:07.874070+0000","last_became_active":"2024-04-26T23:09:06.676678+0000","last_became_peered":"2024-04-26T23:09:06.676678+0000","last_unstale":"2024-04-26T23:09:07.874070+0000","last_undegraded":"2024-04-26T23:09:07.874070+0000","last_fullsized":"2024-04-26T23:09:07.874070+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:25:54.919118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.665539+0000","last_change":"2024-04-26T23:09:06.652396+0000","last_active":"2024-04-26T23:09:07.665539+0000","last_peered":"2024-04-26T23:09:07.665539+0000","last_clean":"2024-04-26T23:09:07.665539+0000","last_became_active":"2024-04-26T23:09:06.652099+0000","last_became_peered":"2024-04-26T23:09:06.652099+0000","last_unstale":"2024-04-26T23:09:07.665539+0000","last_undegraded":"2024-04-26T23:09:07.665539+0000","last_fullsized":"2024-04-26T23:09:07.665539+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:37:02.056219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"102'4","reported_seq":16,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.683469+0000","last_change":"2024-04-26T23:09:05.651659+0000","last_active":"2024-04-26T23:09:06.683469+0000","last_peered":"2024-04-26T23:09:06.683469+0000","last_clean":"2024-04-26T23:09:06.683469+0000","last_became_active":"2024-04-26T23:09:05.651111+0000","last_became_peered":"2024-04-26T23:09:05.651111+0000","last_unstale":"2024-04-26T23:09:06.683469+0000","last_undegraded":"2024-04-26T23:09:06.683469+0000","last_fullsized":"2024-04-26T23:09:06.683469+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:49.192120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874062+0000","last_change":"2024-04-26T23:09:06.675929+0000","last_active":"2024-04-26T23:09:07.874062+0000","last_peered":"2024-04-26T23:09:07.874062+0000","last_clean":"2024-04-26T23:09:07.874062+0000","last_became_active":"2024-04-26T23:09:06.675214+0000","last_became_peered":"2024-04-26T23:09:06.675214+0000","last_unstale":"2024-04-26T23:09:07.874062+0000","last_undegraded":"2024-04-26T23:09:07.874062+0000","last_fullsized":"2024-04-26T23:09:07.874062+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:05.404801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874794+0000","last_change":"2024-04-26T23:09:05.657155+0000","last_active":"2024-04-26T23:09:07.874794+0000","last_peered":"2024-04-26T23:09:07.874794+0000","last_clean":"2024-04-26T23:09:07.874794+0000","last_became_active":"2024-04-26T23:09:05.656853+0000","last_became_peered":"2024-04-26T23:09:05.656853+0000","last_unstale":"2024-04-26T23:09:07.874794+0000","last_undegraded":"2024-04-26T23:09:07.874794+0000","last_fullsized":"2024-04-26T23:09:07.874794+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:06:06.319287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.671094+0000","last_change":"2024-04-26T23:09:06.657417+0000","last_active":"2024-04-26T23:09:07.671094+0000","last_peered":"2024-04-26T23:09:07.671094+0000","last_clean":"2024-04-26T23:09:07.671094+0000","last_became_active":"2024-04-26T23:09:06.657081+0000","last_became_peered":"2024-04-26T23:09:06.657081+0000","last_unstale":"2024-04-26T23:09:07.671094+0000","last_undegraded":"2024-04-26T23:09:07.671094+0000","last_fullsized":"2024-04-26T23:09:07.671094+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:50.209129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.670993+0000","last_change":"2024-04-26T23:09:05.657829+0000","last_active":"2024-04-26T23:09:07.670993+0000","last_peered":"2024-04-26T23:09:07.670993+0000","last_clean":"2024-04-26T23:09:07.670993+0000","last_became_active":"2024-04-26T23:09:05.657537+0000","last_became_peered":"2024-04-26T23:09:05.657537+0000","last_unstale":"2024-04-26T23:09:07.670993+0000","last_undegraded":"2024-04-26T23:09:07.670993+0000","last_fullsized":"2024-04-26T23:09:07.670993+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:56.541839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.681505+0000","last_change":"2024-04-26T23:09:06.673645+0000","last_active":"2024-04-26T23:09:06.681505+0000","last_peered":"2024-04-26T23:09:06.681505+0000","last_clean":"2024-04-26T23:09:06.681505+0000","last_became_active":"2024-04-26T23:09:06.673364+0000","last_became_peered":"2024-04-26T23:09:06.673364+0000","last_unstale":"2024-04-26T23:09:06.681505+0000","last_undegraded":"2024-04-26T23:09:06.681505+0000","last_fullsized":"2024-04-26T23:09:06.681505+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:07.566798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.971379+0000","last_change":"2024-04-26T23:09:05.657725+0000","last_active":"2024-04-26T23:09:07.971379+0000","last_peered":"2024-04-26T23:09:07.971379+0000","last_clean":"2024-04-26T23:09:07.971379+0000","last_became_active":"2024-04-26T23:09:05.655539+0000","last_became_peered":"2024-04-26T23:09:05.655539+0000","last_unstale":"2024-04-26T23:09:07.971379+0000","last_undegraded":"2024-04-26T23:09:07.971379+0000","last_fullsized":"2024-04-26T23:09:07.971379+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:41.337351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.677643+0000","last_change":"2024-04-26T23:09:06.667432+0000","last_active":"2024-04-26T23:09:06.677643+0000","last_peered":"2024-04-26T23:09:06.677643+0000","last_clean":"2024-04-26T23:09:06.677643+0000","last_became_active":"2024-04-26T23:09:06.665746+0000","last_became_peered":"2024-04-26T23:09:06.665746+0000","last_unstale":"2024-04-26T23:09:06.677643+0000","last_undegraded":"2024-04-26T23:09:06.677643+0000","last_fullsized":"2024-04-26T23:09:06.677643+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:55:50.893638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.916841+0000","last_change":"2024-04-26T23:09:05.658546+0000","last_active":"2024-04-26T23:09:07.916841+0000","last_peered":"2024-04-26T23:09:07.916841+0000","last_clean":"2024-04-26T23:09:07.916841+0000","last_became_active":"2024-04-26T23:09:05.658110+0000","last_became_peered":"2024-04-26T23:09:05.658110+0000","last_unstale":"2024-04-26T23:09:07.916841+0000","last_undegraded":"2024-04-26T23:09:07.916841+0000","last_fullsized":"2024-04-26T23:09:07.916841+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:50.874496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666921+0000","last_change":"2024-04-26T23:09:06.660056+0000","last_active":"2024-04-26T23:09:07.666921+0000","last_peered":"2024-04-26T23:09:07.666921+0000","last_clean":"2024-04-26T23:09:07.666921+0000","last_became_active":"2024-04-26T23:09:06.659328+0000","last_became_peered":"2024-04-26T23:09:06.659328+0000","last_unstale":"2024-04-26T23:09:07.666921+0000","last_undegraded":"2024-04-26T23:09:07.666921+0000","last_fullsized":"2024-04-26T23:09:07.666921+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:09.802401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"102'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.678691+0000","last_change":"2024-04-26T23:09:05.663211+0000","last_active":"2024-04-26T23:09:06.678691+0000","last_peered":"2024-04-26T23:09:06.678691+0000","last_clean":"2024-04-26T23:09:06.678691+0000","last_became_active":"2024-04-26T23:09:05.662769+0000","last_became_peered":"2024-04-26T23:09:05.662769+0000","last_unstale":"2024-04-26T23:09:06.678691+0000","last_undegraded":"2024-04-26T23:09:06.678691+0000","last_fullsized":"2024-04-26T23:09:06.678691+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:04:08.972842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.670334+0000","last_change":"2024-04-26T23:09:06.675356+0000","last_active":"2024-04-26T23:09:07.670334+0000","last_peered":"2024-04-26T23:09:07.670334+0000","last_clean":"2024-04-26T23:09:07.670334+0000","last_became_active":"2024-04-26T23:09:06.674810+0000","last_became_peered":"2024-04-26T23:09:06.674810+0000","last_unstale":"2024-04-26T23:09:07.670334+0000","last_undegraded":"2024-04-26T23:09:07.670334+0000","last_fullsized":"2024-04-26T23:09:07.670334+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:56:12.324814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669818+0000","last_change":"2024-04-26T23:09:05.662055+0000","last_active":"2024-04-26T23:09:07.669818+0000","last_peered":"2024-04-26T23:09:07.669818+0000","last_clean":"2024-04-26T23:09:07.669818+0000","last_became_active":"2024-04-26T23:09:05.661716+0000","last_became_peered":"2024-04-26T23:09:05.661716+0000","last_unstale":"2024-04-26T23:09:07.669818+0000","last_undegraded":"2024-04-26T23:09:07.669818+0000","last_fullsized":"2024-04-26T23:09:07.669818+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:13.586825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.663424+0000","last_change":"2024-04-26T23:09:06.660257+0000","last_active":"2024-04-26T23:09:06.663424+0000","last_peered":"2024-04-26T23:09:06.663424+0000","last_clean":"2024-04-26T23:09:06.663424+0000","last_became_active":"2024-04-26T23:09:06.659964+0000","last_became_peered":"2024-04-26T23:09:06.659964+0000","last_unstale":"2024-04-26T23:09:06.663424+0000","last_undegraded":"2024-04-26T23:09:06.663424+0000","last_fullsized":"2024-04-26T23:09:06.663424+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:28:13.915388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.652242+0000","last_change":"2024-04-26T23:09:05.653029+0000","last_active":"2024-04-26T23:09:06.652242+0000","last_peered":"2024-04-26T23:09:06.652242+0000","last_clean":"2024-04-26T23:09:06.652242+0000","last_became_active":"2024-04-26T23:09:05.652717+0000","last_became_peered":"2024-04-26T23:09:05.652717+0000","last_unstale":"2024-04-26T23:09:06.652242+0000","last_undegraded":"2024-04-26T23:09:06.652242+0000","last_fullsized":"2024-04-26T23:09:06.652242+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:50.023949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.665650+0000","last_change":"2024-04-26T23:09:06.659026+0000","last_active":"2024-04-26T23:09:07.665650+0000","last_peered":"2024-04-26T23:09:07.665650+0000","last_clean":"2024-04-26T23:09:07.665650+0000","last_became_active":"2024-04-26T23:09:06.656679+0000","last_became_peered":"2024-04-26T23:09:06.656679+0000","last_unstale":"2024-04-26T23:09:07.665650+0000","last_undegraded":"2024-04-26T23:09:07.665650+0000","last_fullsized":"2024-04-26T23:09:07.665650+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:49:28.936175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.672287+0000","last_change":"2024-04-26T23:09:05.659767+0000","last_active":"2024-04-26T23:09:07.672287+0000","last_peered":"2024-04-26T23:09:07.672287+0000","last_clean":"2024-04-26T23:09:07.672287+0000","last_became_active":"2024-04-26T23:09:05.658427+0000","last_became_peered":"2024-04-26T23:09:05.658427+0000","last_unstale":"2024-04-26T23:09:07.672287+0000","last_undegraded":"2024-04-26T23:09:07.672287+0000","last_fullsized":"2024-04-26T23:09:07.672287+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:31.986750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970681+0000","last_change":"2024-04-26T23:09:06.658098+0000","last_active":"2024-04-26T23:09:07.970681+0000","last_peered":"2024-04-26T23:09:07.970681+0000","last_clean":"2024-04-26T23:09:07.970681+0000","last_became_active":"2024-04-26T23:09:06.657814+0000","last_became_peered":"2024-04-26T23:09:06.657814+0000","last_unstale":"2024-04-26T23:09:07.970681+0000","last_undegraded":"2024-04-26T23:09:07.970681+0000","last_fullsized":"2024-04-26T23:09:07.970681+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:23:06.112828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666879+0000","last_change":"2024-04-26T23:09:05.655699+0000","last_active":"2024-04-26T23:09:07.666879+0000","last_peered":"2024-04-26T23:09:07.666879+0000","last_clean":"2024-04-26T23:09:07.666879+0000","last_became_active":"2024-04-26T23:09:05.655426+0000","last_became_peered":"2024-04-26T23:09:05.655426+0000","last_unstale":"2024-04-26T23:09:07.666879+0000","last_undegraded":"2024-04-26T23:09:07.666879+0000","last_fullsized":"2024-04-26T23:09:07.666879+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:24.064319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874111+0000","last_change":"2024-04-26T23:09:06.663392+0000","last_active":"2024-04-26T23:09:07.874111+0000","last_peered":"2024-04-26T23:09:07.874111+0000","last_clean":"2024-04-26T23:09:07.874111+0000","last_became_active":"2024-04-26T23:09:06.663109+0000","last_became_peered":"2024-04-26T23:09:06.663109+0000","last_unstale":"2024-04-26T23:09:07.874111+0000","last_undegraded":"2024-04-26T23:09:07.874111+0000","last_fullsized":"2024-04-26T23:09:07.874111+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:39.712791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"102'3","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.917142+0000","last_change":"2024-04-26T23:09:05.660656+0000","last_active":"2024-04-26T23:09:07.917142+0000","last_peered":"2024-04-26T23:09:07.917142+0000","last_clean":"2024-04-26T23:09:07.917142+0000","last_became_active":"2024-04-26T23:09:05.659934+0000","last_became_peered":"2024-04-26T23:09:05.659934+0000","last_unstale":"2024-04-26T23:09:07.917142+0000","last_undegraded":"2024-04-26T23:09:07.917142+0000","last_fullsized":"2024-04-26T23:09:07.917142+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:50.023949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970494+0000","last_change":"2024-04-26T23:09:06.656814+0000","last_active":"2024-04-26T23:09:07.970494+0000","last_peered":"2024-04-26T23:09:07.970494+0000","last_clean":"2024-04-26T23:09:07.970494+0000","last_became_active":"2024-04-26T23:09:06.656548+0000","last_became_peered":"2024-04-26T23:09:06.656548+0000","last_unstale":"2024-04-26T23:09:07.970494+0000","last_undegraded":"2024-04-26T23:09:07.970494+0000","last_fullsized":"2024-04-26T23:09:07.970494+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:32.801517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"102'6","reported_seq":20,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669454+0000","last_change":"2024-04-26T23:09:05.659052+0000","last_active":"2024-04-26T23:09:07.669454+0000","last_peered":"2024-04-26T23:09:07.669454+0000","last_clean":"2024-04-26T23:09:07.669454+0000","last_became_active":"2024-04-26T23:09:05.658758+0000","last_became_peered":"2024-04-26T23:09:05.658758+0000","last_unstale":"2024-04-26T23:09:07.669454+0000","last_undegraded":"2024-04-26T23:09:07.669454+0000","last_fullsized":"2024-04-26T23:09:07.669454+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:57.613194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.685791+0000","last_change":"2024-04-26T23:09:06.681695+0000","last_active":"2024-04-26T23:09:06.685791+0000","last_peered":"2024-04-26T23:09:06.685791+0000","last_clean":"2024-04-26T23:09:06.685791+0000","last_became_active":"2024-04-26T23:09:06.679733+0000","last_became_peered":"2024-04-26T23:09:06.679733+0000","last_unstale":"2024-04-26T23:09:06.685791+0000","last_undegraded":"2024-04-26T23:09:06.685791+0000","last_fullsized":"2024-04-26T23:09:06.685791+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:06:16.800754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970619+0000","last_change":"2024-04-26T23:09:05.657500+0000","last_active":"2024-04-26T23:09:07.970619+0000","last_peered":"2024-04-26T23:09:07.970619+0000","last_clean":"2024-04-26T23:09:07.970619+0000","last_became_active":"2024-04-26T23:09:05.656871+0000","last_became_peered":"2024-04-26T23:09:05.656871+0000","last_unstale":"2024-04-26T23:09:07.970619+0000","last_undegraded":"2024-04-26T23:09:07.970619+0000","last_fullsized":"2024-04-26T23:09:07.970619+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:57.613194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669348+0000","last_change":"2024-04-26T23:09:06.659893+0000","last_active":"2024-04-26T23:09:07.669348+0000","last_peered":"2024-04-26T23:09:07.669348+0000","last_clean":"2024-04-26T23:09:07.669348+0000","last_became_active":"2024-04-26T23:09:06.659535+0000","last_became_peered":"2024-04-26T23:09:06.659535+0000","last_unstale":"2024-04-26T23:09:07.669348+0000","last_undegraded":"2024-04-26T23:09:07.669348+0000","last_fullsized":"2024-04-26T23:09:07.669348+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:58:11.926588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.653977+0000","last_change":"2024-04-26T23:09:05.653759+0000","last_active":"2024-04-26T23:09:06.653977+0000","last_peered":"2024-04-26T23:09:06.653977+0000","last_clean":"2024-04-26T23:09:06.653977+0000","last_became_active":"2024-04-26T23:09:05.653448+0000","last_became_peered":"2024-04-26T23:09:05.653448+0000","last_unstale":"2024-04-26T23:09:06.653977+0000","last_undegraded":"2024-04-26T23:09:06.653977+0000","last_fullsized":"2024-04-26T23:09:06.653977+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:58:10.909579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874674+0000","last_change":"2024-04-26T23:09:06.675912+0000","last_active":"2024-04-26T23:09:07.874674+0000","last_peered":"2024-04-26T23:09:07.874674+0000","last_clean":"2024-04-26T23:09:07.874674+0000","last_became_active":"2024-04-26T23:09:06.675587+0000","last_became_peered":"2024-04-26T23:09:06.675587+0000","last_unstale":"2024-04-26T23:09:07.874674+0000","last_undegraded":"2024-04-26T23:09:07.874674+0000","last_fullsized":"2024-04-26T23:09:07.874674+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:01.067828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874412+0000","last_change":"2024-04-26T23:09:05.658978+0000","last_active":"2024-04-26T23:09:07.874412+0000","last_peered":"2024-04-26T23:09:07.874412+0000","last_clean":"2024-04-26T23:09:07.874412+0000","last_became_active":"2024-04-26T23:09:05.657667+0000","last_became_peered":"2024-04-26T23:09:05.657667+0000","last_unstale":"2024-04-26T23:09:07.874412+0000","last_undegraded":"2024-04-26T23:09:07.874412+0000","last_fullsized":"2024-04-26T23:09:07.874412+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:19:31.137605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.667100+0000","last_change":"2024-04-26T23:09:06.678771+0000","last_active":"2024-04-26T23:09:07.667100+0000","last_peered":"2024-04-26T23:09:07.667100+0000","last_clean":"2024-04-26T23:09:07.667100+0000","last_became_active":"2024-04-26T23:09:06.678470+0000","last_became_peered":"2024-04-26T23:09:06.678470+0000","last_unstale":"2024-04-26T23:09:07.667100+0000","last_undegraded":"2024-04-26T23:09:07.667100+0000","last_fullsized":"2024-04-26T23:09:07.667100+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:58:31.193256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"102'8","reported_seq":22,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668418+0000","last_change":"2024-04-26T23:09:05.660538+0000","last_active":"2024-04-26T23:09:07.668418+0000","last_peered":"2024-04-26T23:09:07.668418+0000","last_clean":"2024-04-26T23:09:07.668418+0000","last_became_active":"2024-04-26T23:09:05.660211+0000","last_became_peered":"2024-04-26T23:09:05.660211+0000","last_unstale":"2024-04-26T23:09:07.668418+0000","last_undegraded":"2024-04-26T23:09:07.668418+0000","last_fullsized":"2024-04-26T23:09:07.668418+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:12:31.044722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874542+0000","last_change":"2024-04-26T23:09:06.654675+0000","last_active":"2024-04-26T23:09:07.874542+0000","last_peered":"2024-04-26T23:09:07.874542+0000","last_clean":"2024-04-26T23:09:07.874542+0000","last_became_active":"2024-04-26T23:09:06.654369+0000","last_became_peered":"2024-04-26T23:09:06.654369+0000","last_unstale":"2024-04-26T23:09:07.874542+0000","last_undegraded":"2024-04-26T23:09:07.874542+0000","last_fullsized":"2024-04-26T23:09:07.874542+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:54.789576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669194+0000","last_change":"2024-04-26T23:09:05.656875+0000","last_active":"2024-04-26T23:09:07.669194+0000","last_peered":"2024-04-26T23:09:07.669194+0000","last_clean":"2024-04-26T23:09:07.669194+0000","last_became_active":"2024-04-26T23:09:05.656569+0000","last_became_peered":"2024-04-26T23:09:05.656569+0000","last_unstale":"2024-04-26T23:09:07.669194+0000","last_undegraded":"2024-04-26T23:09:07.669194+0000","last_fullsized":"2024-04-26T23:09:07.669194+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:00.713750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668596+0000","last_change":"2024-04-26T23:09:06.654678+0000","last_active":"2024-04-26T23:09:07.668596+0000","last_peered":"2024-04-26T23:09:07.668596+0000","last_clean":"2024-04-26T23:09:07.668596+0000","last_became_active":"2024-04-26T23:09:06.654326+0000","last_became_peered":"2024-04-26T23:09:06.654326+0000","last_unstale":"2024-04-26T23:09:07.668596+0000","last_undegraded":"2024-04-26T23:09:07.668596+0000","last_fullsized":"2024-04-26T23:09:07.668596+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:51.891505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.917127+0000","last_change":"2024-04-26T23:09:05.661198+0000","last_active":"2024-04-26T23:09:07.917127+0000","last_peered":"2024-04-26T23:09:07.917127+0000","last_clean":"2024-04-26T23:09:07.917127+0000","last_became_active":"2024-04-26T23:09:05.660742+0000","last_became_peered":"2024-04-26T23:09:05.660742+0000","last_unstale":"2024-04-26T23:09:07.917127+0000","last_undegraded":"2024-04-26T23:09:07.917127+0000","last_fullsized":"2024-04-26T23:09:07.917127+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:49.192120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.971157+0000","last_change":"2024-04-26T23:09:06.662581+0000","last_active":"2024-04-26T23:09:07.971157+0000","last_peered":"2024-04-26T23:09:07.971157+0000","last_clean":"2024-04-26T23:09:07.971157+0000","last_became_active":"2024-04-26T23:09:06.661964+0000","last_became_peered":"2024-04-26T23:09:06.661964+0000","last_unstale":"2024-04-26T23:09:07.971157+0000","last_undegraded":"2024-04-26T23:09:07.971157+0000","last_fullsized":"2024-04-26T23:09:07.971157+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:54.748365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.668275+0000","last_change":"2024-04-26T23:09:05.658524+0000","last_active":"2024-04-26T23:09:06.668275+0000","last_peered":"2024-04-26T23:09:06.668275+0000","last_clean":"2024-04-26T23:09:06.668275+0000","last_became_active":"2024-04-26T23:09:05.658233+0000","last_became_peered":"2024-04-26T23:09:05.658233+0000","last_unstale":"2024-04-26T23:09:06.668275+0000","last_undegraded":"2024-04-26T23:09:06.668275+0000","last_fullsized":"2024-04-26T23:09:06.668275+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:00.713750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666047+0000","last_change":"2024-04-26T23:09:06.664326+0000","last_active":"2024-04-26T23:09:07.666047+0000","last_peered":"2024-04-26T23:09:07.666047+0000","last_clean":"2024-04-26T23:09:07.666047+0000","last_became_active":"2024-04-26T23:09:06.663467+0000","last_became_peered":"2024-04-26T23:09:06.663467+0000","last_unstale":"2024-04-26T23:09:07.666047+0000","last_undegraded":"2024-04-26T23:09:07.666047+0000","last_fullsized":"2024-04-26T23:09:07.666047+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:49:42.578048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874744+0000","last_change":"2024-04-26T23:09:05.660355+0000","last_active":"2024-04-26T23:09:07.874744+0000","last_peered":"2024-04-26T23:09:07.874744+0000","last_clean":"2024-04-26T23:09:07.874744+0000","last_became_active":"2024-04-26T23:09:05.660077+0000","last_became_peered":"2024-04-26T23:09:05.660077+0000","last_unstale":"2024-04-26T23:09:07.874744+0000","last_undegraded":"2024-04-26T23:09:07.874744+0000","last_fullsized":"2024-04-26T23:09:07.874744+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:11:10.583556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666915+0000","last_change":"2024-04-26T23:09:06.662267+0000","last_active":"2024-04-26T23:09:07.666915+0000","last_peered":"2024-04-26T23:09:07.666915+0000","last_clean":"2024-04-26T23:09:07.666915+0000","last_became_active":"2024-04-26T23:09:06.661952+0000","last_became_peered":"2024-04-26T23:09:06.661952+0000","last_unstale":"2024-04-26T23:09:07.666915+0000","last_undegraded":"2024-04-26T23:09:07.666915+0000","last_fullsized":"2024-04-26T23:09:07.666915+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:37.386095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.873796+0000","last_change":"2024-04-26T23:09:05.657945+0000","last_active":"2024-04-26T23:09:07.873796+0000","last_peered":"2024-04-26T23:09:07.873796+0000","last_clean":"2024-04-26T23:09:07.873796+0000","last_became_active":"2024-04-26T23:09:05.657447+0000","last_became_peered":"2024-04-26T23:09:05.657447+0000","last_unstale":"2024-04-26T23:09:07.873796+0000","last_undegraded":"2024-04-26T23:09:07.873796+0000","last_fullsized":"2024-04-26T23:09:07.873796+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:23.522041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.678162+0000","last_change":"2024-04-26T23:09:06.670171+0000","last_active":"2024-04-26T23:09:06.678162+0000","last_peered":"2024-04-26T23:09:06.678162+0000","last_clean":"2024-04-26T23:09:06.678162+0000","last_became_active":"2024-04-26T23:09:06.669702+0000","last_became_peered":"2024-04-26T23:09:06.669702+0000","last_unstale":"2024-04-26T23:09:06.678162+0000","last_undegraded":"2024-04-26T23:09:06.678162+0000","last_fullsized":"2024-04-26T23:09:06.678162+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:00:38.519071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.670419+0000","last_change":"2024-04-26T23:09:05.657685+0000","last_active":"2024-04-26T23:09:07.670419+0000","last_peered":"2024-04-26T23:09:07.670419+0000","last_clean":"2024-04-26T23:09:07.670419+0000","last_became_active":"2024-04-26T23:09:05.657388+0000","last_became_peered":"2024-04-26T23:09:05.657388+0000","last_unstale":"2024-04-26T23:09:07.670419+0000","last_undegraded":"2024-04-26T23:09:07.670419+0000","last_fullsized":"2024-04-26T23:09:07.670419+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:41.337351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874751+0000","last_change":"2024-04-26T23:09:06.675924+0000","last_active":"2024-04-26T23:09:07.874751+0000","last_peered":"2024-04-26T23:09:07.874751+0000","last_clean":"2024-04-26T23:09:07.874751+0000","last_became_active":"2024-04-26T23:09:06.671516+0000","last_became_peered":"2024-04-26T23:09:06.671516+0000","last_unstale":"2024-04-26T23:09:07.874751+0000","last_undegraded":"2024-04-26T23:09:07.874751+0000","last_fullsized":"2024-04-26T23:09:07.874751+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:28.481514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874504+0000","last_change":"2024-04-26T23:09:05.658201+0000","last_active":"2024-04-26T23:09:07.874504+0000","last_peered":"2024-04-26T23:09:07.874504+0000","last_clean":"2024-04-26T23:09:07.874504+0000","last_became_active":"2024-04-26T23:09:05.657874+0000","last_became_peered":"2024-04-26T23:09:05.657874+0000","last_unstale":"2024-04-26T23:09:07.874504+0000","last_undegraded":"2024-04-26T23:09:07.874504+0000","last_fullsized":"2024-04-26T23:09:07.874504+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:01:31.728766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.665282+0000","last_change":"2024-04-26T23:09:06.663889+0000","last_active":"2024-04-26T23:09:07.665282+0000","last_peered":"2024-04-26T23:09:07.665282+0000","last_clean":"2024-04-26T23:09:07.665282+0000","last_became_active":"2024-04-26T23:09:06.663592+0000","last_became_peered":"2024-04-26T23:09:06.663592+0000","last_unstale":"2024-04-26T23:09:07.665282+0000","last_undegraded":"2024-04-26T23:09:07.665282+0000","last_fullsized":"2024-04-26T23:09:07.665282+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:02:43.649622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.970649+0000","last_change":"2024-04-26T23:09:05.663246+0000","last_active":"2024-04-26T23:09:07.970649+0000","last_peered":"2024-04-26T23:09:07.970649+0000","last_clean":"2024-04-26T23:09:07.970649+0000","last_became_active":"2024-04-26T23:09:05.662964+0000","last_became_peered":"2024-04-26T23:09:05.662964+0000","last_unstale":"2024-04-26T23:09:07.970649+0000","last_undegraded":"2024-04-26T23:09:07.970649+0000","last_fullsized":"2024-04-26T23:09:07.970649+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:13.586825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669614+0000","last_change":"2024-04-26T23:09:06.663379+0000","last_active":"2024-04-26T23:09:07.669614+0000","last_peered":"2024-04-26T23:09:07.669614+0000","last_clean":"2024-04-26T23:09:07.669614+0000","last_became_active":"2024-04-26T23:09:06.663076+0000","last_became_peered":"2024-04-26T23:09:06.663076+0000","last_unstale":"2024-04-26T23:09:07.669614+0000","last_undegraded":"2024-04-26T23:09:07.669614+0000","last_fullsized":"2024-04-26T23:09:07.669614+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:51.891505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.666890+0000","last_change":"2024-04-26T23:09:05.667415+0000","last_active":"2024-04-26T23:09:07.666890+0000","last_peered":"2024-04-26T23:09:07.666890+0000","last_clean":"2024-04-26T23:09:07.666890+0000","last_became_active":"2024-04-26T23:09:05.667214+0000","last_became_peered":"2024-04-26T23:09:05.667214+0000","last_unstale":"2024-04-26T23:09:07.666890+0000","last_undegraded":"2024-04-26T23:09:07.666890+0000","last_fullsized":"2024-04-26T23:09:07.666890+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:35:08.878677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.681988+0000","last_change":"2024-04-26T23:09:06.675212+0000","last_active":"2024-04-26T23:09:06.681988+0000","last_peered":"2024-04-26T23:09:06.681988+0000","last_clean":"2024-04-26T23:09:06.681988+0000","last_became_active":"2024-04-26T23:09:06.674932+0000","last_became_peered":"2024-04-26T23:09:06.674932+0000","last_unstale":"2024-04-26T23:09:06.681988+0000","last_undegraded":"2024-04-26T23:09:06.681988+0000","last_fullsized":"2024-04-26T23:09:06.681988+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:45.467614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.669398+0000","last_change":"2024-04-26T23:09:05.656432+0000","last_active":"2024-04-26T23:09:07.669398+0000","last_peered":"2024-04-26T23:09:07.669398+0000","last_clean":"2024-04-26T23:09:07.669398+0000","last_became_active":"2024-04-26T23:09:05.656106+0000","last_became_peered":"2024-04-26T23:09:05.656106+0000","last_unstale":"2024-04-26T23:09:07.669398+0000","last_undegraded":"2024-04-26T23:09:07.669398+0000","last_fullsized":"2024-04-26T23:09:07.669398+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:22:13.434235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.683198+0000","last_change":"2024-04-26T23:09:06.675010+0000","last_active":"2024-04-26T23:09:06.683198+0000","last_peered":"2024-04-26T23:09:06.683198+0000","last_clean":"2024-04-26T23:09:06.683198+0000","last_became_active":"2024-04-26T23:09:06.674649+0000","last_became_peered":"2024-04-26T23:09:06.674649+0000","last_unstale":"2024-04-26T23:09:06.683198+0000","last_undegraded":"2024-04-26T23:09:06.683198+0000","last_fullsized":"2024-04-26T23:09:06.683198+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:12:32.061731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.971523+0000","last_change":"2024-04-26T23:09:05.664895+0000","last_active":"2024-04-26T23:09:07.971523+0000","last_peered":"2024-04-26T23:09:07.971523+0000","last_clean":"2024-04-26T23:09:07.971523+0000","last_became_active":"2024-04-26T23:09:05.664554+0000","last_became_peered":"2024-04-26T23:09:05.664554+0000","last_unstale":"2024-04-26T23:09:07.971523+0000","last_undegraded":"2024-04-26T23:09:07.971523+0000","last_fullsized":"2024-04-26T23:09:07.971523+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:24:14.206855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874500+0000","last_change":"2024-04-26T23:09:06.667186+0000","last_active":"2024-04-26T23:09:07.874500+0000","last_peered":"2024-04-26T23:09:07.874500+0000","last_clean":"2024-04-26T23:09:07.874500+0000","last_became_active":"2024-04-26T23:09:06.666897+0000","last_became_peered":"2024-04-26T23:09:06.666897+0000","last_unstale":"2024-04-26T23:09:07.874500+0000","last_undegraded":"2024-04-26T23:09:07.874500+0000","last_fullsized":"2024-04-26T23:09:07.874500+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:39:32.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.671663+0000","last_change":"2024-04-26T23:09:05.651495+0000","last_active":"2024-04-26T23:09:07.671663+0000","last_peered":"2024-04-26T23:09:07.671663+0000","last_clean":"2024-04-26T23:09:07.671663+0000","last_became_active":"2024-04-26T23:09:05.651177+0000","last_became_peered":"2024-04-26T23:09:05.651177+0000","last_unstale":"2024-04-26T23:09:07.671663+0000","last_undegraded":"2024-04-26T23:09:07.671663+0000","last_fullsized":"2024-04-26T23:09:07.671663+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:06.549789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874902+0000","last_change":"2024-04-26T23:09:06.669767+0000","last_active":"2024-04-26T23:09:07.874902+0000","last_peered":"2024-04-26T23:09:07.874902+0000","last_clean":"2024-04-26T23:09:07.874902+0000","last_became_active":"2024-04-26T23:09:06.669484+0000","last_became_peered":"2024-04-26T23:09:06.669484+0000","last_unstale":"2024-04-26T23:09:07.874902+0000","last_undegraded":"2024-04-26T23:09:07.874902+0000","last_fullsized":"2024-04-26T23:09:07.874902+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:09:27.722749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.873915+0000","last_change":"2024-04-26T23:09:05.653829+0000","last_active":"2024-04-26T23:09:07.873915+0000","last_peered":"2024-04-26T23:09:07.873915+0000","last_clean":"2024-04-26T23:09:07.873915+0000","last_became_active":"2024-04-26T23:09:05.653554+0000","last_became_peered":"2024-04-26T23:09:05.653554+0000","last_unstale":"2024-04-26T23:09:07.873915+0000","last_undegraded":"2024-04-26T23:09:07.873915+0000","last_fullsized":"2024-04-26T23:09:07.873915+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:10:01.882411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.917108+0000","last_change":"2024-04-26T23:09:06.680120+0000","last_active":"2024-04-26T23:09:07.917108+0000","last_peered":"2024-04-26T23:09:07.917108+0000","last_clean":"2024-04-26T23:09:07.917108+0000","last_became_active":"2024-04-26T23:09:06.679083+0000","last_became_peered":"2024-04-26T23:09:06.679083+0000","last_unstale":"2024-04-26T23:09:07.917108+0000","last_undegraded":"2024-04-26T23:09:07.917108+0000","last_fullsized":"2024-04-26T23:09:07.917108+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:17.381118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.668821+0000","last_change":"2024-04-26T23:09:05.657447+0000","last_active":"2024-04-26T23:09:07.668821+0000","last_peered":"2024-04-26T23:09:07.668821+0000","last_clean":"2024-04-26T23:09:07.668821+0000","last_became_active":"2024-04-26T23:09:05.656875+0000","last_became_peered":"2024-04-26T23:09:05.656875+0000","last_unstale":"2024-04-26T23:09:07.668821+0000","last_undegraded":"2024-04-26T23:09:07.668821+0000","last_fullsized":"2024-04-26T23:09:07.668821+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:24:14.206855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.683233+0000","last_change":"2024-04-26T23:09:06.675016+0000","last_active":"2024-04-26T23:09:06.683233+0000","last_peered":"2024-04-26T23:09:06.683233+0000","last_clean":"2024-04-26T23:09:06.683233+0000","last_became_active":"2024-04-26T23:09:06.674609+0000","last_became_peered":"2024-04-26T23:09:06.674609+0000","last_unstale":"2024-04-26T23:09:06.683233+0000","last_undegraded":"2024-04-26T23:09:06.683233+0000","last_fullsized":"2024-04-26T23:09:06.683233+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:57.558848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874772+0000","last_change":"2024-04-26T23:09:05.653955+0000","last_active":"2024-04-26T23:09:07.874772+0000","last_peered":"2024-04-26T23:09:07.874772+0000","last_clean":"2024-04-26T23:09:07.874772+0000","last_became_active":"2024-04-26T23:09:05.653654+0000","last_became_peered":"2024-04-26T23:09:05.653654+0000","last_unstale":"2024-04-26T23:09:07.874772+0000","last_undegraded":"2024-04-26T23:09:07.874772+0000","last_fullsized":"2024-04-26T23:09:07.874772+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:37:20.196982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T23:09:06.675790+0000","last_change":"2024-04-26T23:09:06.666471+0000","last_active":"2024-04-26T23:09:06.675790+0000","last_peered":"2024-04-26T23:09:06.675790+0000","last_clean":"2024-04-26T23:09:06.675790+0000","last_became_active":"2024-04-26T23:09:06.666198+0000","last_became_peered":"2024-04-26T23:09:06.666198+0000","last_unstale":"2024-04-26T23:09:06.675790+0000","last_undegraded":"2024-04-26T23:09:06.675790+0000","last_fullsized":"2024-04-26T23:09:06.675790+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:05.607064+0000","last_clean_scrub_stamp":"2024-04-26T23:09:05.607064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:22:14.451245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.874912+0000","last_change":"2024-04-26T23:09:05.657794+0000","last_active":"2024-04-26T23:09:07.874912+0000","last_peered":"2024-04-26T23:09:07.874912+0000","last_clean":"2024-04-26T23:09:07.874912+0000","last_became_active":"2024-04-26T23:09:05.657429+0000","last_became_peered":"2024-04-26T23:09:05.657429+0000","last_unstale":"2024-04-26T23:09:07.874912+0000","last_undegraded":"2024-04-26T23:09:07.874912+0000","last_fullsized":"2024-04-26T23:09:07.874912+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:09:04.590054+0000","last_clean_scrub_stamp":"2024-04-26T23:09:04.590054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:19.441312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":330,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-04-26T23:09:07.916279+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:09:07.916279+0000","last_peered":"2024-04-26T23:09:07.916279+0000","last_clean":"2024-04-26T23:09:07.916279+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:09:07.916279+0000","last_undegraded":"2024-04-26T23:09:07.916279+0000","last_fullsized":"2024-04-26T23:09:07.916279+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33008,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1867776,"data_stored":1182001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1298432,"data_stored":600381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1265664,"data_stored":591473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32436,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339404,"statfs":{"total":96636764160,"available":96603549696,"internally_reserved":0,"allocated":1286144,"data_stored":598257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1269760,"data_stored":592055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574951,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1269760,"data_stored":591507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1269760,"data_stored":591507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1888256,"data_stored":1190235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:09:11.648 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T23:09:11.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-26T23:09:11.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.913+0000 7fee90600640 1 Processor -- start 2024-04-26T23:09:11.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.913+0000 7fee90600640 1 -- start start 2024-04-26T23:09:11.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.915+0000 7fee90600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fee88072950 0x7fee88167ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.915+0000 7fee90600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee8815d7c0 con 0x7fee88071e60 2024-04-26T23:09:11.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.915+0000 7fee90600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee8815daa0 con 0x7fee880cc480 2024-04-26T23:09:11.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.915+0000 7fee90600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fee8815dd80 con 0x7fee88072950 2024-04-26T23:09:11.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.915+0000 7fee8dc00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fee88071e60 0x7fee8806f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50004/0 (socket says 172.21.15.40:50004) 2024-04-26T23:09:11.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.915+0000 7fee8dc00640 1 -- 172.21.15.40:0/3406429139 learned_addr learned my addr 172.21.15.40:0/3406429139 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 191663986 0 0) 0x7fee8815d7c0 con 0x7fee88071e60 2024-04-26T23:09:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee6c003520 con 0x7fee88071e60 2024-04-26T23:09:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee8e600640 1 --2- 172.21.15.40:0/3406429139 >> v2:172.21.15.103:3300/0 conn(0x7fee88072950 0x7fee88167ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 422040671 0 0) 0x7fee6c003520 con 0x7fee88071e60 2024-04-26T23:09:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 >> v2:172.21.15.103:3300/0 conn(0x7fee88072950 msgr2=0x7fee88167ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 --2- 172.21.15.40:0/3406429139 >> v2:172.21.15.103:3300/0 conn(0x7fee88072950 0x7fee88167ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 >> v1:172.21.15.103:6789/0 conn(0x7fee880cc480 legacy=0x7fee88072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8815e060 con 0x7fee88071e60 2024-04-26T23:09:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fee780019f0 con 0x7fee88071e60 2024-04-26T23:09:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 >> v1:172.21.15.40:6789/0 conn(0x7fee88071e60 legacy=0x7fee8806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 --2- 172.21.15.40:0/3406429139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee6c003a00 0x7fee6c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 --2- 172.21.15.40:0/3406429139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c004800 0x7fee6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 --2- 172.21.15.40:0/3406429139 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee6c005210 0x7fee6c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee8815dd80 con 0x7fee6c004800 2024-04-26T23:09:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c00c160 con 0x7fee6c003a00 2024-04-26T23:09:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.916+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fee6c00c360 con 0x7fee6c005210 2024-04-26T23:09:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.917+0000 7fee8d200640 1 --2- 172.21.15.40:0/3406429139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c004800 0x7fee6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.917+0000 7fee8dc00640 1 --2- 172.21.15.40:0/3406429139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee6c003a00 0x7fee6c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.917+0000 7fee8e600640 1 --2- 172.21.15.40:0/3406429139 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee6c005210 0x7fee6c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.917+0000 7fee8d200640 1 -- 172.21.15.40:0/3406429139 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee6c005210 msgr2=0x7fee6c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.917+0000 7fee8d200640 1 --2- 172.21.15.40:0/3406429139 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee6c005210 0x7fee6c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.917+0000 7fee8d200640 1 -- 172.21.15.40:0/3406429139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee6c003a00 msgr2=0x7fee6c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.917+0000 7fee8d200640 1 --2- 172.21.15.40:0/3406429139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee6c003a00 0x7fee6c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.917+0000 7fee8d200640 1 -- 172.21.15.40:0/3406429139 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6c003520 con 0x7fee6c004800 2024-04-26T23:09:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.917+0000 7fee8e600640 1 -- 172.21.15.40:0/3406429139 reap_dead start 2024-04-26T23:09:11.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.918+0000 7fee8d200640 1 --2- 172.21.15.40:0/3406429139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c004800 0x7fee6c004c80 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fee74001750 tx=0x7fee6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351dd3ca1e00502c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:11.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.919+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee74001370 con 0x7fee6c004800 2024-04-26T23:09:11.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.919+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee740052d0 con 0x7fee6c004800 2024-04-26T23:09:11.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.919+0000 7fee87e00640 1 -- 172.21.15.40:0/3406429139 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7402abe0 con 0x7fee6c004800 2024-04-26T23:09:11.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.919+0000 7fee90600640 1 -- 172.21.15.40:0/3406429139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c004800 msgr2=0x7fee6c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:11.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.919+0000 7fee90600640 1 --2- 172.21.15.40:0/3406429139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c004800 0x7fee6c004c80 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fee74001750 tx=0x7fee6c005830 comp rx=0 tx=0).stop 2024-04-26T23:09:11.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.919+0000 7fee90600640 1 -- 172.21.15.40:0/3406429139 shutdown_connections 2024-04-26T23:09:11.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.919+0000 7fee90600640 1 --2- 172.21.15.40:0/3406429139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c004800 0x7fee6c004c80 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.919+0000 7fee90600640 1 -- 172.21.15.40:0/3406429139 >> 172.21.15.40:0/3406429139 conn(0x7fee880d0be0 msgr2=0x7fee880d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:11.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.919+0000 7fee90600640 1 -- 172.21.15.40:0/3406429139 shutdown_connections 2024-04-26T23:09:11.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.919+0000 7fee90600640 1 -- 172.21.15.40:0/3406429139 wait complete. 2024-04-26T23:09:11.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.921+0000 7fee90600640 1 Processor -- start 2024-04-26T23:09:11.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.921+0000 7fee90600640 1 -- start start 2024-04-26T23:09:11.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.922+0000 7fee90600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee6c004800 0x7fee88151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.922+0000 7fee90600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee880cfc40 0x7fee880d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.922+0000 7fee90600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee880b5f50 0x7fee880b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.922+0000 7fee90600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee8815d570 con 0x7fee880b5f50 2024-04-26T23:09:11.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.922+0000 7fee90600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee8815d3f0 con 0x7fee880cfc40 2024-04-26T23:09:11.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.922+0000 7fee90600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fee88072240 con 0x7fee6c004800 2024-04-26T23:09:11.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.923+0000 7fee8e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee880b5f50 0x7fee880b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:11.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.923+0000 7fee8e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee880b5f50 0x7fee880b63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37978/0 (socket says 172.21.15.40:37978) 2024-04-26T23:09:11.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.923+0000 7fee8e600640 1 -- 172.21.15.40:0/1711514585 learned_addr learned my addr 172.21.15.40:0/1711514585 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:11.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.923+0000 7fee8dc00640 1 --2- 172.21.15.40:0/1711514585 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee6c004800 0x7fee88151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:11.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.923+0000 7fee8e600640 1 -- 172.21.15.40:0/1711514585 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee6c004800 msgr2=0x7fee88151510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:11.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.923+0000 7fee8e600640 1 --2- 172.21.15.40:0/1711514585 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee6c004800 0x7fee88151510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.923+0000 7fee8e600640 1 -- 172.21.15.40:0/1711514585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee880cfc40 msgr2=0x7fee880d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:11.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.923+0000 7fee8e600640 1 --2- 172.21.15.40:0/1711514585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee880cfc40 0x7fee880d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:11.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.923+0000 7fee8e600640 1 -- 172.21.15.40:0/1711514585 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8814da00 con 0x7fee880b5f50 2024-04-26T23:09:11.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.923+0000 7fee8e600640 1 --2- 172.21.15.40:0/1711514585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee880b5f50 0x7fee880b63d0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fee88074290 tx=0x7fee8000f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:11.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.924+0000 7fee85600640 1 -- 172.21.15.40:0/1711514585 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee80017020 con 0x7fee880b5f50 2024-04-26T23:09:11.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.924+0000 7fee85600640 1 -- 172.21.15.40:0/1711514585 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee80004460 con 0x7fee880b5f50 2024-04-26T23:09:11.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.924+0000 7fee85600640 1 -- 172.21.15.40:0/1711514585 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee80003850 con 0x7fee880b5f50 2024-04-26T23:09:11.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.924+0000 7fee90600640 1 -- 172.21.15.40:0/1711514585 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee8814dce0 con 0x7fee880b5f50 2024-04-26T23:09:11.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.924+0000 7fee90600640 1 -- 172.21.15.40:0/1711514585 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee8814e2a0 con 0x7fee880b5f50 2024-04-26T23:09:11.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.925+0000 7fee90600640 1 -- 172.21.15.40:0/1711514585 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee880694f0 con 0x7fee880b5f50 2024-04-26T23:09:11.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.926+0000 7fee85600640 1 -- 172.21.15.40:0/1711514585 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fee8001a020 con 0x7fee880b5f50 2024-04-26T23:09:11.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.927+0000 7fee85600640 1 --2- 172.21.15.40:0/1711514585 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee6806c920 0x7fee6806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:11.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.927+0000 7fee85600640 1 -- 172.21.15.40:0/1711514585 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fee8008bbf0 con 0x7fee880b5f50 2024-04-26T23:09:11.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.928+0000 7fee8dc00640 1 --2- 172.21.15.40:0/1711514585 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee6806c920 0x7fee6806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:11.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.929+0000 7fee8dc00640 1 --2- 172.21.15.40:0/1711514585 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee6806c920 0x7fee6806ede0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fee780013d0 tx=0x7fee78009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:11.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:11.932+0000 7fee85600640 1 -- 172.21.15.40:0/1711514585 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fee80021020 con 0x7fee880b5f50 2024-04-26T23:09:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.207+0000 7fee90600640 1 -- 172.21.15.40:0/1711514585 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fee88159cd0 con 0x7fee880b5f50 2024-04-26T23:09:12.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.213+0000 7fee85600640 1 -- 172.21.15.40:0/1711514585 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) ==== 190+0+147 (secure 0 0 0) 0x7fee80059e80 con 0x7fee880b5f50 2024-04-26T23:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-04-26T23:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDnMSxmxsphEBAAnnWEW05bCxzga46VFiRmwg== 2024-04-26T23:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-04-26T23:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-04-26T23:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-26T23:09:12.214 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-04-26T23:09:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.216+0000 7fee90600640 1 -- 172.21.15.40:0/1711514585 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee6806c920 msgr2=0x7fee6806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.216+0000 7fee90600640 1 --2- 172.21.15.40:0/1711514585 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee6806c920 0x7fee6806ede0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fee780013d0 tx=0x7fee78009040 comp rx=0 tx=0).stop 2024-04-26T23:09:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.216+0000 7fee90600640 1 -- 172.21.15.40:0/1711514585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee880b5f50 msgr2=0x7fee880b63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.216+0000 7fee90600640 1 --2- 172.21.15.40:0/1711514585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee880b5f50 0x7fee880b63d0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fee88074290 tx=0x7fee8000f920 comp rx=0 tx=0).stop 2024-04-26T23:09:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.216+0000 7fee90600640 1 -- 172.21.15.40:0/1711514585 shutdown_connections 2024-04-26T23:09:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.216+0000 7fee90600640 1 --2- 172.21.15.40:0/1711514585 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee6806c920 0x7fee6806ede0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.216+0000 7fee90600640 1 --2- 172.21.15.40:0/1711514585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee880b5f50 0x7fee880b63d0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.216+0000 7fee90600640 1 --2- 172.21.15.40:0/1711514585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee880cfc40 0x7fee880d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.216+0000 7fee90600640 1 --2- 172.21.15.40:0/1711514585 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee6c004800 0x7fee88151510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.216+0000 7fee90600640 1 -- 172.21.15.40:0/1711514585 >> 172.21.15.40:0/1711514585 conn(0x7fee880d0be0 msgr2=0x7fee880ae960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.216+0000 7fee90600640 1 -- 172.21.15.40:0/1711514585 shutdown_connections 2024-04-26T23:09:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.216+0000 7fee90600640 1 -- 172.21.15.40:0/1711514585 wait complete. 2024-04-26T23:09:12.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:12.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.430+0000 7f8c95800640 1 Processor -- start 2024-04-26T23:09:12.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.430+0000 7f8c95800640 1 -- start start 2024-04-26T23:09:12.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.432+0000 7f8c95800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c90158cd0 0x7f8c90159160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:12.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.432+0000 7f8c95800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c900cce10 con 0x7f8c900cc980 2024-04-26T23:09:12.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.432+0000 7f8c95800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c900cd0f0 con 0x7f8c900c6f10 2024-04-26T23:09:12.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.432+0000 7f8c95800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8c900c72f0 con 0x7f8c90158cd0 2024-04-26T23:09:12.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.433+0000 7f8c87e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8c900cc980 0x7f8c900cbcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50014/0 (socket says 172.21.15.40:50014) 2024-04-26T23:09:12.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.433+0000 7f8c87e00640 1 -- 172.21.15.40:0/3082721346 learned_addr learned my addr 172.21.15.40:0/3082721346 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:12.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.433+0000 7f8c94e00640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c90158cd0 0x7f8c90159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:12.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.433+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2007078791 0 0) 0x7f8c900cce10 con 0x7f8c900cc980 2024-04-26T23:09:12.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.433+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c74003520 con 0x7f8c900cc980 2024-04-26T23:09:12.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.433+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1858765901 0 0) 0x7f8c74003520 con 0x7f8c900cc980 2024-04-26T23:09:12.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.433+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c90158cd0 msgr2=0x7f8c90159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:12.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.433+0000 7f8c8f400640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c90158cd0 0x7f8c90159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:12.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.434+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 >> v1:172.21.15.103:6789/0 conn(0x7f8c900c6f10 legacy=0x7f8c900c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:12.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.434+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c900cb4f0 con 0x7f8c900cc980 2024-04-26T23:09:12.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.434+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f8c78001670 con 0x7f8c900cc980 2024-04-26T23:09:12.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.434+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 >> v1:172.21.15.40:6789/0 conn(0x7f8c900cc980 legacy=0x7f8c900cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.434+0000 7f8c8f400640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c740039e0 0x7f8c74003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.434+0000 7f8c8f400640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c90158cd0 0x7f8c74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.434+0000 7f8c8f400640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c74004d40 0x7f8c740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.434+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8c900c72f0 con 0x7f8c74004d40 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.434+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7400baf0 con 0x7f8c740039e0 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.434+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8c7400bcf0 con 0x7f8c90158cd0 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.434+0000 7f8c94e00640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c74004d40 0x7f8c740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.434+0000 7f8c8fe00640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c740039e0 0x7f8c74003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.434+0000 7f8c87e00640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c90158cd0 0x7f8c74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.435+0000 7f8c94e00640 1 -- 172.21.15.40:0/3082721346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c90158cd0 msgr2=0x7f8c74004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.435+0000 7f8c94e00640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c90158cd0 0x7f8c74004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.435+0000 7f8c94e00640 1 -- 172.21.15.40:0/3082721346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c740039e0 msgr2=0x7f8c74003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.435+0000 7f8c94e00640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c740039e0 0x7f8c74003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:12.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.435+0000 7f8c94e00640 1 -- 172.21.15.40:0/3082721346 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c74003520 con 0x7f8c74004d40 2024-04-26T23:09:12.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.435+0000 7f8c8fe00640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c740039e0 0x7f8c74003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:12.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.435+0000 7f8c87e00640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c90158cd0 0x7f8c74004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:12.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.436+0000 7f8c94e00640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c74004d40 0x7f8c740051c0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f8c88009fc0 tx=0x7f8c74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72799f3353a11d0e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:12.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.436+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c88009c10 con 0x7f8c74004d40 2024-04-26T23:09:12.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.436+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c88036070 con 0x7f8c74004d40 2024-04-26T23:09:12.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.436+0000 7f8c8f400640 1 -- 172.21.15.40:0/3082721346 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c88031880 con 0x7f8c74004d40 2024-04-26T23:09:12.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.436+0000 7f8c95800640 1 -- 172.21.15.40:0/3082721346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c74004d40 msgr2=0x7f8c740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:12.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.436+0000 7f8c95800640 1 --2- 172.21.15.40:0/3082721346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c74004d40 0x7f8c740051c0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f8c88009fc0 tx=0x7f8c74005830 comp rx=0 tx=0).stop 2024-04-26T23:09:12.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.436+0000 7f8c94e00640 1 -- 172.21.15.40:0/3082721346 reap_dead start 2024-04-26T23:09:12.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.437+0000 7f8c95800640 1 -- 172.21.15.40:0/3082721346 shutdown_connections 2024-04-26T23:09:12.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.437+0000 7f8c95800640 1 -- 172.21.15.40:0/3082721346 >> 172.21.15.40:0/3082721346 conn(0x7f8c900d0a10 msgr2=0x7f8c900d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:12.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.437+0000 7f8c95800640 1 -- 172.21.15.40:0/3082721346 shutdown_connections 2024-04-26T23:09:12.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.437+0000 7f8c95800640 1 -- 172.21.15.40:0/3082721346 wait complete. 2024-04-26T23:09:12.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.438+0000 7f8c95800640 1 Processor -- start 2024-04-26T23:09:12.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.439+0000 7f8c95800640 1 -- start start 2024-04-26T23:09:12.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.439+0000 7f8c95800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c900b3730 0x7f8c900b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:12.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.439+0000 7f8c95800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c90156fa0 0x7f8c900b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.439+0000 7f8c95800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c900b4050 0x7f8c900b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.439+0000 7f8c95800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8c900c61e0 con 0x7f8c900b4050 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.439+0000 7f8c95800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8c900cb860 con 0x7f8c90156fa0 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.439+0000 7f8c95800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8c900c6360 con 0x7f8c900b3730 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.440+0000 7f8c94e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c900b4050 0x7f8c900b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.440+0000 7f8c94e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c900b4050 0x7f8c900b0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37992/0 (socket says 172.21.15.40:37992) 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.440+0000 7f8c94e00640 1 -- 172.21.15.40:0/3371931503 learned_addr learned my addr 172.21.15.40:0/3371931503 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.440+0000 7f8c87e00640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c90156fa0 0x7f8c900b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.440+0000 7f8c8fe00640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c900b3730 0x7f8c900b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.440+0000 7f8c94e00640 1 -- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c900b3730 msgr2=0x7f8c900b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.440+0000 7f8c94e00640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c900b3730 0x7f8c900b4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.440+0000 7f8c94e00640 1 -- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c90156fa0 msgr2=0x7f8c900b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.440+0000 7f8c94e00640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c90156fa0 0x7f8c900b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:12.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.440+0000 7f8c94e00640 1 -- 172.21.15.40:0/3371931503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c90147980 con 0x7f8c900b4050 2024-04-26T23:09:12.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.440+0000 7f8c8fe00640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c900b3730 0x7f8c900b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:12.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.441+0000 7f8c94e00640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c900b4050 0x7f8c900b0940 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800e230 tx=0x7f8c88009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:12.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.441+0000 7f8c8cc00640 1 -- 172.21.15.40:0/3371931503 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c88009c10 con 0x7f8c900b4050 2024-04-26T23:09:12.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.441+0000 7f8c8cc00640 1 -- 172.21.15.40:0/3371931503 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c88031ca0 con 0x7f8c900b4050 2024-04-26T23:09:12.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.441+0000 7f8c8cc00640 1 -- 172.21.15.40:0/3371931503 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c88003bd0 con 0x7f8c900b4050 2024-04-26T23:09:12.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.441+0000 7f8c95800640 1 -- 172.21.15.40:0/3371931503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c90157860 con 0x7f8c900b4050 2024-04-26T23:09:12.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.441+0000 7f8c95800640 1 -- 172.21.15.40:0/3371931503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c90157e20 con 0x7f8c900b4050 2024-04-26T23:09:12.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.442+0000 7f8c95800640 1 -- 172.21.15.40:0/3371931503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c90069370 con 0x7f8c900b4050 2024-04-26T23:09:12.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.443+0000 7f8c8cc00640 1 -- 172.21.15.40:0/3371931503 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8c88005d10 con 0x7f8c900b4050 2024-04-26T23:09:12.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.444+0000 7f8c8cc00640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8c6c06c920 0x7f8c6c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:12.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.444+0000 7f8c8cc00640 1 -- 172.21.15.40:0/3371931503 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f8c880ad1d0 con 0x7f8c900b4050 2024-04-26T23:09:12.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.445+0000 7f8c8fe00640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8c6c06c920 0x7f8c6c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:12.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.446+0000 7f8c8fe00640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8c6c06c920 0x7f8c6c06ede0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8c800016f0 tx=0x7f8c80009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:12.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.449+0000 7f8c8cc00640 1 -- 172.21.15.40:0/3371931503 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8c8804a060 con 0x7f8c900b4050 2024-04-26T23:09:12.592 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-04-26T23:09:12.591+0000 7f928e600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T23:09:12.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.701+0000 7f8c95800640 1 -- 172.21.15.40:0/3371931503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c900c58b0 con 0x7f8c900b4050 2024-04-26T23:09:12.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.703+0000 7f8c8cc00640 1 -- 172.21.15.40:0/3371931503 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5192 (secure 0 0 0) 0x7f8c8807b3d0 con 0x7f8c900b4050 2024-04-26T23:09:12.703 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:12.703 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":111,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":6598,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":8949,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:09.194635+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955]},"id":7}]} 2024-04-26T23:09:12.703 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 111 2024-04-26T23:09:12.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.705+0000 7f8c95800640 1 -- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8c6c06c920 msgr2=0x7f8c6c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:12.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.705+0000 7f8c95800640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8c6c06c920 0x7f8c6c06ede0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8c800016f0 tx=0x7f8c80009040 comp rx=0 tx=0).stop 2024-04-26T23:09:12.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.705+0000 7f8c95800640 1 -- 172.21.15.40:0/3371931503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c900b4050 msgr2=0x7f8c900b0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:12.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.705+0000 7f8c95800640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c900b4050 0x7f8c900b0940 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800e230 tx=0x7f8c88009750 comp rx=0 tx=0).stop 2024-04-26T23:09:12.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.705+0000 7f8c95800640 1 -- 172.21.15.40:0/3371931503 shutdown_connections 2024-04-26T23:09:12.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.705+0000 7f8c95800640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8c6c06c920 0x7f8c6c06ede0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:12.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.705+0000 7f8c95800640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c900b4050 0x7f8c900b0940 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:12.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.705+0000 7f8c95800640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c90156fa0 0x7f8c900b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:12.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.705+0000 7f8c95800640 1 --2- 172.21.15.40:0/3371931503 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c900b3730 0x7f8c900b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:12.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.705+0000 7f8c95800640 1 -- 172.21.15.40:0/3371931503 >> 172.21.15.40:0/3371931503 conn(0x7f8c900d0a10 msgr2=0x7f8c90162ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:12.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.706+0000 7f8c95800640 1 -- 172.21.15.40:0/3371931503 shutdown_connections 2024-04-26T23:09:12.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:12.706+0000 7f8c95800640 1 -- 172.21.15.40:0/3371931503 wait complete. 2024-04-26T23:09:13.128 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 111, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:09:06.116498+0000', 'modified': '2024-04-26T23:09:09.194635+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8955}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8955': {'gid': 8955, 'name': 'a', 'rank': 0, 'incarnation': 107, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/1138762604', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 1138762604}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 1138762604}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8955, 'qdb_cluster': [8955]} 2024-04-26T23:09:13.129 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-26T23:09:13.129 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T23:09:13.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.355+0000 7f1e9ee00640 1 Processor -- start 2024-04-26T23:09:13.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.355+0000 7f1e9ee00640 1 -- start start 2024-04-26T23:09:13.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.356+0000 7f1e9ee00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e980af7a0 0x7f1e980afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.356+0000 7f1e9ee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e98071fe0 0x7f1e98073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.356+0000 7f1e9ee00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e980d0820 con 0x7f1e9815ce60 2024-04-26T23:09:13.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.357+0000 7f1e9ee00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1e980cc450 con 0x7f1e980af7a0 2024-04-26T23:09:13.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.357+0000 7f1e9ee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1e981614c0 con 0x7f1e98071fe0 2024-04-26T23:09:13.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.357+0000 7f1e9d000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e980af7a0 0x7f1e980afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.357+0000 7f1e9d000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e980af7a0 0x7f1e980afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38000/0 (socket says 172.21.15.40:38000) 2024-04-26T23:09:13.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.357+0000 7f1e9d000640 1 -- 172.21.15.40:0/2476818592 learned_addr learned my addr 172.21.15.40:0/2476818592 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:13.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.357+0000 7f1e9e400640 1 --2- 172.21.15.40:0/2476818592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e98071fe0 0x7f1e98073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.357+0000 7f1e9d000640 1 -- 172.21.15.40:0/2476818592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e98071fe0 msgr2=0x7f1e98073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:13.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.357+0000 7f1e9d000640 1 --2- 172.21.15.40:0/2476818592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e98071fe0 0x7f1e98073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.357+0000 7f1e9d000640 1 -- 172.21.15.40:0/2476818592 >> v1:172.21.15.103:6789/0 conn(0x7f1e9815ce60 legacy=0x7f1e980ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:13.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.357+0000 7f1e9d000640 1 -- 172.21.15.40:0/2476818592 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e98161760 con 0x7f1e980af7a0 2024-04-26T23:09:13.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.357+0000 7f1e9e400640 1 --2- 172.21.15.40:0/2476818592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e98071fe0 0x7f1e98073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:13.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.358+0000 7f1e9d000640 1 --2- 172.21.15.40:0/2476818592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e980af7a0 0x7f1e980afc60 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009870 tx=0x7f1e8402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c31de53ab6def7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:13.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.359+0000 7f1e97e00640 1 -- 172.21.15.40:0/2476818592 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e84037070 con 0x7f1e980af7a0 2024-04-26T23:09:13.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.359+0000 7f1e97e00640 1 -- 172.21.15.40:0/2476818592 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e84032440 con 0x7f1e980af7a0 2024-04-26T23:09:13.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.359+0000 7f1e97e00640 1 -- 172.21.15.40:0/2476818592 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e84032810 con 0x7f1e980af7a0 2024-04-26T23:09:13.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.359+0000 7f1e9ee00640 1 -- 172.21.15.40:0/2476818592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e980af7a0 msgr2=0x7f1e980afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.359+0000 7f1e9ee00640 1 --2- 172.21.15.40:0/2476818592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e980af7a0 0x7f1e980afc60 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009870 tx=0x7f1e8402a180 comp rx=0 tx=0).stop 2024-04-26T23:09:13.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.359+0000 7f1e9ee00640 1 -- 172.21.15.40:0/2476818592 shutdown_connections 2024-04-26T23:09:13.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.359+0000 7f1e9ee00640 1 --2- 172.21.15.40:0/2476818592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e98071fe0 0x7f1e98073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.359+0000 7f1e9ee00640 1 --2- 172.21.15.40:0/2476818592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e980af7a0 0x7f1e980afc60 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.359+0000 7f1e9ee00640 1 -- 172.21.15.40:0/2476818592 >> 172.21.15.40:0/2476818592 conn(0x7f1e980d0a10 msgr2=0x7f1e980d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:13.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.360+0000 7f1e9ee00640 1 -- 172.21.15.40:0/2476818592 shutdown_connections 2024-04-26T23:09:13.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.360+0000 7f1e9ee00640 1 -- 172.21.15.40:0/2476818592 wait complete. 2024-04-26T23:09:13.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.361+0000 7f1e9ee00640 1 Processor -- start 2024-04-26T23:09:13.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.361+0000 7f1e9ee00640 1 -- start start 2024-04-26T23:09:13.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.362+0000 7f1e9ee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e98071fe0 0x7f1e9814b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.362+0000 7f1e9ee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e980af7a0 0x7f1e98153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.362+0000 7f1e9ee00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9815ce60 0x7f1e980ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.362+0000 7f1e9ee00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1e980cc1f0 con 0x7f1e9815ce60 2024-04-26T23:09:13.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.362+0000 7f1e9ee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e980cc070 con 0x7f1e98071fe0 2024-04-26T23:09:13.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.362+0000 7f1e9ee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1e98073ca0 con 0x7f1e980af7a0 2024-04-26T23:09:13.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.363+0000 7f1e9e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9815ce60 0x7f1e980ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.363+0000 7f1e9e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9815ce60 0x7f1e980ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38016/0 (socket says 172.21.15.40:38016) 2024-04-26T23:09:13.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.363+0000 7f1e9e400640 1 -- 172.21.15.40:0/1440436094 learned_addr learned my addr 172.21.15.40:0/1440436094 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:13.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.363+0000 7f1e9d000640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e980af7a0 0x7f1e98153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.363+0000 7f1e9da00640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e98071fe0 0x7f1e9814b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.363+0000 7f1e9e400640 1 -- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e980af7a0 msgr2=0x7f1e98153110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.363+0000 7f1e9e400640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e980af7a0 0x7f1e98153110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.363+0000 7f1e9e400640 1 -- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e98071fe0 msgr2=0x7f1e9814b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.363+0000 7f1e9e400640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e98071fe0 0x7f1e9814b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.363+0000 7f1e9e400640 1 -- 172.21.15.40:0/1440436094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9814ba20 con 0x7f1e9815ce60 2024-04-26T23:09:13.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.363+0000 7f1e9e400640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9815ce60 0x7f1e980ae8c0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f1e9000cf20 tx=0x7f1e9000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:13.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.364+0000 7f1e95600640 1 -- 172.21.15.40:0/1440436094 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9001a070 con 0x7f1e9815ce60 2024-04-26T23:09:13.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.364+0000 7f1e95600640 1 -- 172.21.15.40:0/1440436094 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e9000d840 con 0x7f1e9815ce60 2024-04-26T23:09:13.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.364+0000 7f1e95600640 1 -- 172.21.15.40:0/1440436094 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9000dc30 con 0x7f1e9815ce60 2024-04-26T23:09:13.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.364+0000 7f1e9ee00640 1 -- 172.21.15.40:0/1440436094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e98153bb0 con 0x7f1e9815ce60 2024-04-26T23:09:13.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.364+0000 7f1e9ee00640 1 -- 172.21.15.40:0/1440436094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e98154170 con 0x7f1e9815ce60 2024-04-26T23:09:13.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.366+0000 7f1e95600640 1 -- 172.21.15.40:0/1440436094 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1e9001d070 con 0x7f1e9815ce60 2024-04-26T23:09:13.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.367+0000 7f1e95600640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7806c8f0 0x7f1e7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.367+0000 7f1e95600640 1 -- 172.21.15.40:0/1440436094 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f1e9008c5b0 con 0x7f1e9815ce60 2024-04-26T23:09:13.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.368+0000 7f1e9da00640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7806c8f0 0x7f1e7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.368+0000 7f1ea3233740 1 Processor -- start 2024-04-26T23:09:13.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.368+0000 7f1ea3233740 1 -- start start 2024-04-26T23:09:13.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.369+0000 7f1e9da00640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7806c8f0 0x7f1e7806edb0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00a750 tx=0x7f1e8c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:13.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.369+0000 7f1ea3233740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 0x55beb8a29910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.369+0000 7f1ea3233740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 0x55beb8a27b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.369+0000 7f1ea3233740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a52270 0x55beb8a526f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.369+0000 7f1ea3233740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55beb898b600 con 0x55beb8a277b0 2024-04-26T23:09:13.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.369+0000 7f1ea3233740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55beb8928100 con 0x55beb8a52270 2024-04-26T23:09:13.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.369+0000 7f1ea3233740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55beb89875c0 con 0x55beb8a63e80 2024-04-26T23:09:13.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.370+0000 7f1e9da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 0x55beb8a29910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.370+0000 7f1e9da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 0x55beb8a29910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38020/0 (socket says 172.21.15.40:38020) 2024-04-26T23:09:13.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.370+0000 7f1e9da00640 1 -- 172.21.15.40:0/753262753 learned_addr learned my addr 172.21.15.40:0/753262753 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:13.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.370+0000 7f1e9d000640 1 --2- 172.21.15.40:0/753262753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 0x55beb8a27b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.370+0000 7f1e9da00640 1 -- 172.21.15.40:0/753262753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 msgr2=0x55beb8a27b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:13.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.370+0000 7f1e9da00640 1 --2- 172.21.15.40:0/753262753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 0x55beb8a27b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.370+0000 7f1e9da00640 1 -- 172.21.15.40:0/753262753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a52270 msgr2=0x55beb8a526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:13.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.370+0000 7f1e9e400640 1 --2- 172.21.15.40:0/753262753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a52270 0x55beb8a526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.370+0000 7f1e9da00640 1 --2- 172.21.15.40:0/753262753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a52270 0x55beb8a526f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.370+0000 7f1e9da00640 1 -- 172.21.15.40:0/753262753 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55beb869d2d0 con 0x55beb8a277b0 2024-04-26T23:09:13.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.370+0000 7f1e9da00640 1 --2- 172.21.15.40:0/753262753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 0x55beb8a29910 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00c960 tx=0x7f1e8c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10470b0a8f2fe2e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:13.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.371+0000 7f1e8a000640 1 -- 172.21.15.40:0/753262753 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c019040 con 0x55beb8a277b0 2024-04-26T23:09:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.371+0000 7f1e8a000640 1 -- 172.21.15.40:0/753262753 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e8c022070 con 0x55beb8a277b0 2024-04-26T23:09:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.371+0000 7f1e8a000640 1 -- 172.21.15.40:0/753262753 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c013dc0 con 0x55beb8a277b0 2024-04-26T23:09:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.371+0000 7f1ea3233740 1 -- 172.21.15.40:0/753262753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 msgr2=0x55beb8a29910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.371+0000 7f1ea3233740 1 --2- 172.21.15.40:0/753262753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 0x55beb8a29910 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00c960 tx=0x7f1e8c00f000 comp rx=0 tx=0).stop 2024-04-26T23:09:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.371+0000 7f1ea3233740 1 -- 172.21.15.40:0/753262753 shutdown_connections 2024-04-26T23:09:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.371+0000 7f1ea3233740 1 --2- 172.21.15.40:0/753262753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a52270 0x55beb8a526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.371+0000 7f1ea3233740 1 --2- 172.21.15.40:0/753262753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 0x55beb8a27b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.371+0000 7f1ea3233740 1 --2- 172.21.15.40:0/753262753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 0x55beb8a29910 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.371+0000 7f1ea3233740 1 -- 172.21.15.40:0/753262753 >> 172.21.15.40:0/753262753 conn(0x55beb89bbd10 msgr2=0x55beb8a6d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.372+0000 7f1ea3233740 1 -- 172.21.15.40:0/753262753 shutdown_connections 2024-04-26T23:09:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.372+0000 7f1ea3233740 1 -- 172.21.15.40:0/753262753 wait complete. 2024-04-26T23:09:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.372+0000 7f1ea3233740 1 -- start start 2024-04-26T23:09:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.372+0000 7f1ea3233740 1 Processor -- start 2024-04-26T23:09:13.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.373+0000 7f1ea3233740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 0x55beb8a6ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.373+0000 7f1ea3233740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a52270 0x55beb8a65de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.373+0000 7f1ea3233740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a63e80 0x55beb8a66320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.373+0000 7f1ea3233740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55beb8912f70 con 0x55beb8a277b0 2024-04-26T23:09:13.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.373+0000 7f1ea3233740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55beb8a4fdc0 con 0x55beb8a63e80 2024-04-26T23:09:13.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.373+0000 7f1ea3233740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55beb8a7d0d0 con 0x55beb8a52270 2024-04-26T23:09:13.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.373+0000 7f1ea3233740 10 client.0 _ll_register_callbacks cb 0x55beb8981480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:13.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.374+0000 7f1e9da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 0x55beb8a6ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.374+0000 7f1e9da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 0x55beb8a6ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38030/0 (socket says 172.21.15.40:38030) 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.374+0000 7f1e9da00640 1 -- 172.21.15.40:0/1860631788 learned_addr learned my addr 172.21.15.40:0/1860631788 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.374+0000 7f1e9d000640 1 --2- 172.21.15.40:0/1860631788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a52270 0x55beb8a65de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.374+0000 7f1e9da00640 1 -- 172.21.15.40:0/1860631788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a52270 msgr2=0x55beb8a65de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.374+0000 7f1e9da00640 1 --2- 172.21.15.40:0/1860631788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a52270 0x55beb8a65de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.374+0000 7f1e9e400640 1 --2- 172.21.15.40:0/1860631788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a63e80 0x55beb8a66320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.374+0000 7f1e9da00640 1 -- 172.21.15.40:0/1860631788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a63e80 msgr2=0x55beb8a66320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.374+0000 7f1e9da00640 1 --2- 172.21.15.40:0/1860631788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a63e80 0x55beb8a66320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.374+0000 7f1e9da00640 1 -- 172.21.15.40:0/1860631788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55beb87676b0 con 0x55beb8a277b0 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.375+0000 7f1e9da00640 1 --2- 172.21.15.40:0/1860631788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 0x55beb8a6ddd0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c0137d0 tx=0x7f1e8c004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ba6823f92f48e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.375+0000 7f1e7ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.375+0000 7f1e7ea00640 1 -- 172.21.15.40:0/1860631788 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c013980 con 0x55beb8a277b0 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.375+0000 7f1e7ea00640 1 -- 172.21.15.40:0/1860631788 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f1e8c013b20 con 0x55beb8a277b0 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.375+0000 7f1ea3233740 1 -- 172.21.15.40:0/1860631788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e8c022070 con 0x55beb8a277b0 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.375+0000 7f1e7ea00640 1 -- 172.21.15.40:0/1860631788 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7f1e8c013d20 con 0x55beb8a277b0 2024-04-26T23:09:13.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.375+0000 7f1ea3233740 10 client.9090 fetch_fsmap learned FSMap version 111 2024-04-26T23:09:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.375+0000 7f1ea3233740 1 -- 172.21.15.40:0/1860631788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=2+}) -- 0x55beb8a7ce20 con 0x55beb8a277b0 2024-04-26T23:09:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.376+0000 7f1e7ea00640 1 -- 172.21.15.40:0/1860631788 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e8c022070 con 0x55beb8a277b0 2024-04-26T23:09:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.376+0000 7f1e7ea00640 1 -- 172.21.15.40:0/1860631788 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 111) ==== 3494+0+0 (secure 0 0 0) 0x7f1e8c01c030 con 0x55beb8a277b0 2024-04-26T23:09:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.376+0000 7f1ea3233740 10 client.9090 fetch_fsmap finished waiting for FSMap version 111 2024-04-26T23:09:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.376+0000 7f1ea3233740 10 client.9090 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:09:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.376+0000 7f1ea3233740 20 client.9090 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.376+0000 7f1ea3233740 1 --2- 172.21.15.40:0/1860631788 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55beb8a52870 0x55beb8a550b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.376+0000 7f1ea3233740 4 client.9090 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.376+0000 7f1ea3233740 1 -- 172.21.15.40:0/1860631788 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55beb8999210 con 0x55beb8a52870 2024-04-26T23:09:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.376+0000 7f1e9d000640 1 --2- 172.21.15.40:0/1860631788 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55beb8a52870 0x55beb8a550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.377+0000 7f1e9d000640 1 --2- 172.21.15.40:0/1860631788 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55beb8a52870 0x55beb8a550b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8028820d46448dc2 server_cookie=f9a71bdf2d953f53 in_seq=0 out_seq=0 2024-04-26T23:09:13.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.377+0000 7f1e7ea00640 10 client.9090 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:13.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1e7ea00640 1 -- 172.21.15.40:0/1860631788 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55beb8999210 con 0x55beb8a52870 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1e7ea00640 10 client.9090 handle_command_reply: tid=0 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1ea3233740 1 client.9090 shutdown 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1ea3233740 1 -- 172.21.15.40:0/1860631788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 msgr2=0x55beb8a6ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1ea3233740 1 --2- 172.21.15.40:0/1860631788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 0x55beb8a6ddd0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c0137d0 tx=0x7f1e8c004680 comp rx=0 tx=0).stop 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1ea3233740 1 -- 172.21.15.40:0/1860631788 shutdown_connections 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1ea3233740 1 --2- 172.21.15.40:0/1860631788 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55beb8a52870 0x55beb8a550b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1ea3233740 1 --2- 172.21.15.40:0/1860631788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a63e80 0x55beb8a66320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1ea3233740 1 --2- 172.21.15.40:0/1860631788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a52270 0x55beb8a65de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1ea3233740 1 --2- 172.21.15.40:0/1860631788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a277b0 0x55beb8a6ddd0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1e7ea00640 0 client.9090 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1ea3233740 1 -- 172.21.15.40:0/1860631788 >> 172.21.15.40:0/1860631788 conn(0x55beb89bbd10 msgr2=0x55beb8a70780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1ea3233740 1 -- 172.21.15.40:0/1860631788 shutdown_connections 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1ea3233740 1 -- 172.21.15.40:0/1860631788 wait complete. 2024-04-26T23:09:13.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.378+0000 7f1ea3233740 20 client.9090 trim_cache size 0 max 16384 2024-04-26T23:09:13.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.398+0000 7f1ea3233740 1 Processor -- start 2024-04-26T23:09:13.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.398+0000 7f1ea3233740 1 -- start start 2024-04-26T23:09:13.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.399+0000 7f1ea3233740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a277b0 0x55beb8a56080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.399+0000 7f1ea3233740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a52270 0x55beb8a6d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.399+0000 7f1ea3233740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 0x55beb8a550b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.399+0000 7f1ea3233740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1e98200c60 con 0x55beb8a52270 2024-04-26T23:09:13.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.399+0000 7f1ea3233740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e980c6c40 con 0x55beb8a277b0 2024-04-26T23:09:13.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.399+0000 7f1ea3233740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1e98200730 con 0x55beb8a63e80 2024-04-26T23:09:13.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.400+0000 7f1e9d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a52270 0x55beb8a6d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.400+0000 7f1e9d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a52270 0x55beb8a6d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38046/0 (socket says 172.21.15.40:38046) 2024-04-26T23:09:13.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.400+0000 7f1e9d000640 1 -- 172.21.15.40:0/2318393827 learned_addr learned my addr 172.21.15.40:0/2318393827 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:13.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.400+0000 7f1e9e400640 1 --2- 172.21.15.40:0/2318393827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 0x55beb8a550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.400+0000 7f1e9d000640 1 -- 172.21.15.40:0/2318393827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 msgr2=0x55beb8a550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:13.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.400+0000 7f1e9d000640 1 --2- 172.21.15.40:0/2318393827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 0x55beb8a550b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.400+0000 7f1e9da00640 1 --2- 172.21.15.40:0/2318393827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a277b0 0x55beb8a56080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.400+0000 7f1e9d000640 1 -- 172.21.15.40:0/2318393827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a277b0 msgr2=0x55beb8a56080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:13.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.400+0000 7f1e9d000640 1 --2- 172.21.15.40:0/2318393827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a277b0 0x55beb8a56080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.400+0000 7f1e9d000640 1 -- 172.21.15.40:0/2318393827 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55beb8a69270 con 0x55beb8a52270 2024-04-26T23:09:13.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.401+0000 7f1e9d000640 1 --2- 172.21.15.40:0/2318393827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a52270 0x55beb8a6d6d0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f1e840097c0 tx=0x7f1e8402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54da634847f02a18 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:13.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.401+0000 7f1e89600640 1 -- 172.21.15.40:0/2318393827 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e84037040 con 0x55beb8a52270 2024-04-26T23:09:13.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.401+0000 7f1e89600640 1 -- 172.21.15.40:0/2318393827 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e84040020 con 0x55beb8a52270 2024-04-26T23:09:13.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.401+0000 7f1e89600640 1 -- 172.21.15.40:0/2318393827 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e84032900 con 0x55beb8a52270 2024-04-26T23:09:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.401+0000 7f1ea3233740 1 -- 172.21.15.40:0/2318393827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a52270 msgr2=0x55beb8a6d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.401+0000 7f1ea3233740 1 --2- 172.21.15.40:0/2318393827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a52270 0x55beb8a6d6d0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f1e840097c0 tx=0x7f1e8402c000 comp rx=0 tx=0).stop 2024-04-26T23:09:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.402+0000 7f1ea3233740 1 -- 172.21.15.40:0/2318393827 shutdown_connections 2024-04-26T23:09:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.402+0000 7f1ea3233740 1 --2- 172.21.15.40:0/2318393827 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 0x55beb8a550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.402+0000 7f1ea3233740 1 --2- 172.21.15.40:0/2318393827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a52270 0x55beb8a6d6d0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.402+0000 7f1ea3233740 1 --2- 172.21.15.40:0/2318393827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a277b0 0x55beb8a56080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.402+0000 7f1ea3233740 1 -- 172.21.15.40:0/2318393827 >> 172.21.15.40:0/2318393827 conn(0x55beb89bbd10 msgr2=0x55beb8a29790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.402+0000 7f1ea3233740 1 -- 172.21.15.40:0/2318393827 shutdown_connections 2024-04-26T23:09:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.402+0000 7f1ea3233740 1 -- 172.21.15.40:0/2318393827 wait complete. 2024-04-26T23:09:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.402+0000 7f1ea3233740 1 -- start start 2024-04-26T23:09:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.402+0000 7f1ea3233740 1 Processor -- start 2024-04-26T23:09:13.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.403+0000 7f1ea3233740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a277b0 0x55beb8aef460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.403+0000 7f1ea3233740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 0x55beb8a54f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.403+0000 7f1ea3233740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a6d6d0 0x55beb8a554d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.403+0000 7f1ea3233740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55beb8a526f0 con 0x55beb8a6d6d0 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.403+0000 7f1ea3233740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55beb8a4f4d0 con 0x55beb8a277b0 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.403+0000 7f1ea3233740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55beb8a52ce0 con 0x55beb8a63e80 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.403+0000 7f1ea3233740 10 client.0 _ll_register_callbacks cb 0x7f1e8c00c800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.404+0000 7f1e9e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a6d6d0 0x55beb8a554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.404+0000 7f1e9e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a6d6d0 0x55beb8a554d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38054/0 (socket says 172.21.15.40:38054) 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.404+0000 7f1e9e400640 1 -- 172.21.15.40:0/625414437 learned_addr learned my addr 172.21.15.40:0/625414437 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.404+0000 7f1e9d000640 1 --2- 172.21.15.40:0/625414437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 0x55beb8a54f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.404+0000 7f1e9da00640 1 --2- 172.21.15.40:0/625414437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a277b0 0x55beb8aef460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.404+0000 7f1e9e400640 1 -- 172.21.15.40:0/625414437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 msgr2=0x55beb8a54f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.404+0000 7f1e9e400640 1 --2- 172.21.15.40:0/625414437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 0x55beb8a54f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.404+0000 7f1e9e400640 1 -- 172.21.15.40:0/625414437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a277b0 msgr2=0x55beb8aef460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.404+0000 7f1e9e400640 1 --2- 172.21.15.40:0/625414437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a277b0 0x55beb8aef460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.404+0000 7f1e9e400640 1 -- 172.21.15.40:0/625414437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55beb8a56490 con 0x55beb8a6d6d0 2024-04-26T23:09:13.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.405+0000 7f1e9e400640 1 --2- 172.21.15.40:0/625414437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a6d6d0 0x55beb8a554d0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f1e9005a0d0 tx=0x7f1e9005a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e336b3472b90e2b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:13.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.405+0000 7f1e7ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:13.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.405+0000 7f1e7ea00640 1 -- 172.21.15.40:0/625414437 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e90062d20 con 0x55beb8a6d6d0 2024-04-26T23:09:13.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.405+0000 7f1e7ea00640 1 -- 172.21.15.40:0/625414437 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f1e9005e170 con 0x55beb8a6d6d0 2024-04-26T23:09:13.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.405+0000 7f1ea3233740 1 -- 172.21.15.40:0/625414437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e84040020 con 0x55beb8a6d6d0 2024-04-26T23:09:13.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.405+0000 7f1e7ea00640 1 -- 172.21.15.40:0/625414437 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7f1e90062890 con 0x55beb8a6d6d0 2024-04-26T23:09:13.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.405+0000 7f1ea3233740 10 client.9096 fetch_fsmap learned FSMap version 111 2024-04-26T23:09:13.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.405+0000 7f1ea3233740 1 -- 172.21.15.40:0/625414437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=2+}) -- 0x55beb8a8f640 con 0x55beb8a6d6d0 2024-04-26T23:09:13.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.405+0000 7f1e7ea00640 1 -- 172.21.15.40:0/625414437 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e84040020 con 0x55beb8a6d6d0 2024-04-26T23:09:13.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.406+0000 7f1e7ea00640 1 -- 172.21.15.40:0/625414437 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 111) ==== 3494+0+0 (secure 0 0 0) 0x7f1e9008c0b0 con 0x55beb8a6d6d0 2024-04-26T23:09:13.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.406+0000 7f1ea3233740 10 client.9096 fetch_fsmap finished waiting for FSMap version 111 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.406+0000 7f1ea3233740 10 client.9096 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.406+0000 7f1ea3233740 20 client.9096 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.406+0000 7f1ea3233740 1 --2- 172.21.15.40:0/625414437 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55beb8a67d60 0x55beb8a68140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.406+0000 7f1ea3233740 4 client.9096 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.406+0000 7f1ea3233740 1 -- 172.21.15.40:0/625414437 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "status"}) -- 0x55beb8993080 con 0x55beb8a67d60 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.406+0000 7f1e9da00640 1 --2- 172.21.15.40:0/625414437 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55beb8a67d60 0x55beb8a68140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1e9da00640 1 --2- 172.21.15.40:0/625414437 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55beb8a67d60 0x55beb8a68140 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61e912d69208795 server_cookie=b2bb140e6559fc37 in_seq=0 out_seq=0 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1e7ea00640 10 client.9096 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1e7ea00640 1 -- 172.21.15.40:0/625414437 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x55beb8993080 con 0x55beb8a67d60 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1e7ea00640 10 client.9096 handle_command_reply: tid=0 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1ea3233740 1 client.9096 shutdown 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1ea3233740 1 -- 172.21.15.40:0/625414437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a6d6d0 msgr2=0x55beb8a554d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1ea3233740 1 --2- 172.21.15.40:0/625414437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a6d6d0 0x55beb8a554d0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f1e9005a0d0 tx=0x7f1e9005a140 comp rx=0 tx=0).stop 2024-04-26T23:09:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1ea3233740 1 -- 172.21.15.40:0/625414437 shutdown_connections 2024-04-26T23:09:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1ea3233740 1 --2- 172.21.15.40:0/625414437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55beb8a6d6d0 0x55beb8a554d0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1ea3233740 1 --2- 172.21.15.40:0/625414437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55beb8a63e80 0x55beb8a54f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1ea3233740 1 --2- 172.21.15.40:0/625414437 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55beb8a67d60 0x55beb8a68140 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1ea3233740 1 --2- 172.21.15.40:0/625414437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55beb8a277b0 0x55beb8aef460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1ea3233740 1 -- 172.21.15.40:0/625414437 >> 172.21.15.40:0/625414437 conn(0x55beb89bbd10 msgr2=0x55beb8a62c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1e7ea00640 0 client.9096 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1ea3233740 1 -- 172.21.15.40:0/625414437 shutdown_connections 2024-04-26T23:09:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.407+0000 7f1ea3233740 1 -- 172.21.15.40:0/625414437 wait complete. 2024-04-26T23:09:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.408+0000 7f1ea3233740 20 client.9096 trim_cache size 0 max 16384 2024-04-26T23:09:13.411 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:09:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:09:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8955, 2024-04-26T23:09:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:09:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:09:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:09:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 6.7771520819999997, 2024-04-26T23:09:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 111, 2024-04-26T23:09:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 103, 2024-04-26T23:09:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 103, 2024-04-26T23:09:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 12.179894787 2024-04-26T23:09:13.412 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.408+0000 7f1e9ee00640 1 -- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7806c8f0 msgr2=0x7f1e7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.408+0000 7f1e9ee00640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7806c8f0 0x7f1e7806edb0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00a750 tx=0x7f1e8c008040 comp rx=0 tx=0).stop 2024-04-26T23:09:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.408+0000 7f1e9ee00640 1 -- 172.21.15.40:0/1440436094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9815ce60 msgr2=0x7f1e980ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.408+0000 7f1e9ee00640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9815ce60 0x7f1e980ae8c0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f1e9000cf20 tx=0x7f1e9000a7d0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.408+0000 7f1e9ee00640 1 -- 172.21.15.40:0/1440436094 shutdown_connections 2024-04-26T23:09:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.408+0000 7f1e9ee00640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1e7806c8f0 0x7f1e7806edb0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.408+0000 7f1e9ee00640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9815ce60 0x7f1e980ae8c0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.408+0000 7f1e9ee00640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e980af7a0 0x7f1e98153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.408+0000 7f1e9ee00640 1 --2- 172.21.15.40:0/1440436094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e98071fe0 0x7f1e9814b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.408+0000 7f1e9ee00640 1 -- 172.21.15.40:0/1440436094 >> 172.21.15.40:0/1440436094 conn(0x7f1e980d0a10 msgr2=0x7f1e98155d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:13.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.408+0000 7f1e9ee00640 1 -- 172.21.15.40:0/1440436094 shutdown_connections 2024-04-26T23:09:13.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.409+0000 7f1e9ee00640 1 -- 172.21.15.40:0/1440436094 wait complete. 2024-04-26T23:09:13.427 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-26T23:09:13.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.627+0000 7ff24e400640 1 Processor -- start 2024-04-26T23:09:13.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.627+0000 7ff24e400640 1 -- start start 2024-04-26T23:09:13.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.629+0000 7ff24e400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff2480d06b0 0x7ff2480d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.629+0000 7ff24e400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2480b59f0 con 0x7ff248146060 2024-04-26T23:09:13.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.629+0000 7ff24e400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2480b4180 con 0x7ff248145970 2024-04-26T23:09:13.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.629+0000 7ff24e400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff2480ceed0 con 0x7ff2480d06b0 2024-04-26T23:09:13.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.629+0000 7ff24da00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff2480d06b0 0x7ff2480d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.629+0000 7ff247e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff248146060 0x7ff2480b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50030/0 (socket says 172.21.15.40:50030) 2024-04-26T23:09:13.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.629+0000 7ff247e00640 1 -- 172.21.15.40:0/934980400 learned_addr learned my addr 172.21.15.40:0/934980400 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:13.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.630+0000 7ff247400640 1 -- 172.21.15.40:0/934980400 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3216549342 0 0) 0x7ff2480b59f0 con 0x7ff248146060 2024-04-26T23:09:13.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.630+0000 7ff247400640 1 -- 172.21.15.40:0/934980400 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff230003520 con 0x7ff248146060 2024-04-26T23:09:13.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.630+0000 7ff24da00640 1 -- 172.21.15.40:0/934980400 >> v1:172.21.15.103:6789/0 conn(0x7ff248145970 legacy=0x7ff2481612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.630+0000 7ff24da00640 1 -- 172.21.15.40:0/934980400 >> v1:172.21.15.40:6789/0 conn(0x7ff248146060 legacy=0x7ff2480b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.630+0000 7ff24da00640 1 -- 172.21.15.40:0/934980400 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2480b5f40 con 0x7ff2480d06b0 2024-04-26T23:09:13.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.631+0000 7ff24da00640 1 --2- 172.21.15.40:0/934980400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2480d06b0 0x7ff2480d0ae0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7ff240009dc0 tx=0x7ff24002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a6efd41cfd1eca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:13.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.631+0000 7ff247400640 1 -- 172.21.15.40:0/934980400 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240007740 con 0x7ff2480d06b0 2024-04-26T23:09:13.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.631+0000 7ff247400640 1 -- 172.21.15.40:0/934980400 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2400078e0 con 0x7ff2480d06b0 2024-04-26T23:09:13.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.631+0000 7ff247400640 1 -- 172.21.15.40:0/934980400 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240007cb0 con 0x7ff2480d06b0 2024-04-26T23:09:13.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.632+0000 7ff24e400640 1 -- 172.21.15.40:0/934980400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2480d06b0 msgr2=0x7ff2480d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.632+0000 7ff24e400640 1 --2- 172.21.15.40:0/934980400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2480d06b0 0x7ff2480d0ae0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7ff240009dc0 tx=0x7ff24002a730 comp rx=0 tx=0).stop 2024-04-26T23:09:13.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.632+0000 7ff24e400640 1 -- 172.21.15.40:0/934980400 shutdown_connections 2024-04-26T23:09:13.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.632+0000 7ff24e400640 1 --2- 172.21.15.40:0/934980400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2480d06b0 0x7ff2480d0ae0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.632+0000 7ff24e400640 1 -- 172.21.15.40:0/934980400 >> 172.21.15.40:0/934980400 conn(0x7ff248075eb0 msgr2=0x7ff248068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:13.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.632+0000 7ff24e400640 1 -- 172.21.15.40:0/934980400 shutdown_connections 2024-04-26T23:09:13.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.632+0000 7ff24e400640 1 -- 172.21.15.40:0/934980400 wait complete. 2024-04-26T23:09:13.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.634+0000 7ff24e400640 1 Processor -- start 2024-04-26T23:09:13.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.634+0000 7ff24e400640 1 -- start start 2024-04-26T23:09:13.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.635+0000 7ff24e400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2480d06b0 0x7ff2481406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.635+0000 7ff24e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff248145970 0x7ff248140bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.635+0000 7ff24e400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff248146060 0x7ff248143140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.635+0000 7ff24e400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff2480cf100 con 0x7ff248145970 2024-04-26T23:09:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.635+0000 7ff24e400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff2480d1020 con 0x7ff2480d06b0 2024-04-26T23:09:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.635+0000 7ff24e400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff2480cf3e0 con 0x7ff248146060 2024-04-26T23:09:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.635+0000 7ff247e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff248145970 0x7ff248140bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.635+0000 7ff247e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff248145970 0x7ff248140bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38072/0 (socket says 172.21.15.40:38072) 2024-04-26T23:09:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.635+0000 7ff247e00640 1 -- 172.21.15.40:0/1172357540 learned_addr learned my addr 172.21.15.40:0/1172357540 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.636+0000 7ff24da00640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff248146060 0x7ff248143140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.636+0000 7ff24d000640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2480d06b0 0x7ff2481406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.636+0000 7ff247e00640 1 -- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff248146060 msgr2=0x7ff248143140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.636+0000 7ff247e00640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff248146060 0x7ff248143140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.636+0000 7ff247e00640 1 -- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2480d06b0 msgr2=0x7ff2481406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.636+0000 7ff247e00640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2480d06b0 0x7ff2481406b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.636+0000 7ff247e00640 1 -- 172.21.15.40:0/1172357540 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24815daf0 con 0x7ff248145970 2024-04-26T23:09:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.636+0000 7ff24d000640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2480d06b0 0x7ff2481406b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.636+0000 7ff247e00640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff248145970 0x7ff248140bf0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ff23400c410 tx=0x7ff23400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.636+0000 7ff244c00640 1 -- 172.21.15.40:0/1172357540 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff234017070 con 0x7ff248145970 2024-04-26T23:09:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.636+0000 7ff244c00640 1 -- 172.21.15.40:0/1172357540 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff234005ce0 con 0x7ff248145970 2024-04-26T23:09:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.636+0000 7ff244c00640 1 -- 172.21.15.40:0/1172357540 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2340072f0 con 0x7ff248145970 2024-04-26T23:09:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.636+0000 7ff24e400640 1 -- 172.21.15.40:0/1172357540 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff24815ddd0 con 0x7ff248145970 2024-04-26T23:09:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.637+0000 7ff24e400640 1 -- 172.21.15.40:0/1172357540 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff24815e390 con 0x7ff248145970 2024-04-26T23:09:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.638+0000 7ff24e400640 1 -- 172.21.15.40:0/1172357540 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff248069370 con 0x7ff248145970 2024-04-26T23:09:13.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.639+0000 7ff244c00640 1 -- 172.21.15.40:0/1172357540 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff23401a070 con 0x7ff248145970 2024-04-26T23:09:13.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.640+0000 7ff244c00640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff22806c8f0 0x7ff22806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:13.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.640+0000 7ff244c00640 1 -- 172.21.15.40:0/1172357540 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7ff23408b170 con 0x7ff248145970 2024-04-26T23:09:13.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.641+0000 7ff24d000640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff22806c8f0 0x7ff22806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:13.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.642+0000 7ff24d000640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff22806c8f0 0x7ff22806edb0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff23c000f60 tx=0x7ff23c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:13.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.644+0000 7ff244c00640 1 -- 172.21.15.40:0/1172357540 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff2340594c0 con 0x7ff248145970 2024-04-26T23:09:13.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.899+0000 7ff24e400640 1 -- 172.21.15.40:0/1172357540 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff24815d7a0 con 0x7ff248145970 2024-04-26T23:09:13.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.900+0000 7ff244c00640 1 -- 172.21.15.40:0/1172357540 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v111) ==== 53+0+73 (secure 0 0 0) 0x7ff234058dc0 con 0x7ff248145970 2024-04-26T23:09:13.900 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-26T23:09:13.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.903+0000 7ff24e400640 1 -- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff22806c8f0 msgr2=0x7ff22806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.903+0000 7ff24e400640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff22806c8f0 0x7ff22806edb0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff23c000f60 tx=0x7ff23c009040 comp rx=0 tx=0).stop 2024-04-26T23:09:13.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.903+0000 7ff24e400640 1 -- 172.21.15.40:0/1172357540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff248145970 msgr2=0x7ff248140bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:13.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.903+0000 7ff24e400640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff248145970 0x7ff248140bf0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ff23400c410 tx=0x7ff23400c8d0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.903+0000 7ff24e400640 1 -- 172.21.15.40:0/1172357540 shutdown_connections 2024-04-26T23:09:13.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.903+0000 7ff24e400640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff22806c8f0 0x7ff22806edb0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.903+0000 7ff24e400640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff248146060 0x7ff248143140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.903+0000 7ff24e400640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff248145970 0x7ff248140bf0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.903+0000 7ff24e400640 1 --2- 172.21.15.40:0/1172357540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2480d06b0 0x7ff2481406b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:13.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.903+0000 7ff24e400640 1 -- 172.21.15.40:0/1172357540 >> 172.21.15.40:0/1172357540 conn(0x7ff248075eb0 msgr2=0x7ff2480b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:13.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.903+0000 7ff24e400640 1 -- 172.21.15.40:0/1172357540 shutdown_connections 2024-04-26T23:09:13.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:13.904+0000 7ff24e400640 1 -- 172.21.15.40:0/1172357540 wait complete. 2024-04-26T23:09:13.923 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-26T23:09:13.923 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:09:13.923 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-04-26T23:09:13.924 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T23:09:13.924 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T23:09:13.924 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-26T23:09:13.924 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-26T23:09:13.924 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:09:13.924 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:09:13.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:13.924 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-04-26T23:09:13.977 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:09:13.977 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:09:13.977 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:09:13.977 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:09:13.977 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-26T23:09:13.977 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:09:13.977 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:09:13.977 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:09:13.978 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-04-26T23:09:13.978 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:09:13.978 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:09:13.978 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-04-26T23:09:13.978 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:09:13.978 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-04-26T23:09:13.978 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-04-26T23:09:13.978 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T23:09:13.978 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42160sec preferred_lft 42160sec 2024-04-26T23:09:13.978 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-26T23:09:13.979 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:09:13.979 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:09:13.979 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-04-26T23:09:13.979 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-04-26T23:09:13.979 INFO:teuthology.orchestra.run.smithi040.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-26T23:09:13.979 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 8e:8e:79:d7:cf:5d brd ff:ff:ff:ff:ff:ff 2024-04-26T23:09:13.979 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-26T23:09:13.979 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:09:13.979 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::4ce7:9bff:fe51:b1/64 scope link 2024-04-26T23:09:13.979 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:09:13.979 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:09:13.979 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether be:f1:28:e1:99:21 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T23:09:13.979 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::bcf1:28ff:fee1:9921/64 scope link 2024-04-26T23:09:13.980 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:09:13.980 INFO:teuthology.orchestra.run.smithi040.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-26T23:09:13.980 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 8e:8e:79:d7:cf:5d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-26T23:09:13.980 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::8c8e:79ff:fed7:cf5d/64 scope link 2024-04-26T23:09:13.980 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:09:13.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:13.981 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:09:14.064 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-04-26T23:09:14.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:14.069 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-04-26T23:09:14.122 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-04-26T23:09:14.122 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:09:14.122 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T23:09:14.122 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T23:09:14.122 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:09:14.123 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:09:14.123 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:09:14.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T23:09:14.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T23:09:14.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T23:09:14.124 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:09:14.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:09:14.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:09:14.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:14.351 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:09:14.402 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T23:09:14.402 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-26T23:09:14.403 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:09:14.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:14.403 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-04-26T23:09:14.483 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:09:14.484 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T23:09:14.536 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:09:14.536 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:09:14.537 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:09:14.591 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-26T23:09:14.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:14.591 DEBUG:teuthology.orchestra.run.smithi040:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:09:14.734 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:09:14.734 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:09:14.734 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:09:14.756 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:09:14.756 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:09:14.756 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:09:14.756 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:09:14.756 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:09:14.784 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-26T23:09:14.888 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-26T23:09:14.888 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-26T23:09:14.888 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-26T23:09:14.888 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-26T23:09:14.888 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-26T23:09:14.888 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.40:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-26T23:09:14.888 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=815936ab-d4ac-471e-b069-1a0dc70078ee" will pass to kernel 2024-04-26T23:09:14.889 INFO:teuthology.orchestra.run.smithi040.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T23:09:14.889 INFO:teuthology.orchestra.run.smithi040.stdout: You just mounted a file system that supports labels which does not 2024-04-26T23:09:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T23:09:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T23:09:14.890 INFO:teuthology.orchestra.run.smithi040.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T23:09:14.890 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T23:09:14.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:14.890 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:09:14.987 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:09:15.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:09:15.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:09:15.128 INFO:teuthology.orchestra.run.smithi040.stdout:client6092 2024-04-26T23:09:15.129 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:09:15.129 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:09:15.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-26T23:09:15.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.414+0000 7fa76a200640 1 Processor -- start 2024-04-26T23:09:15.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.414+0000 7fa76a200640 1 -- start start 2024-04-26T23:09:15.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.416+0000 7fa76a200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa764071e60 0x7fa76406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:15.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.416+0000 7fa76a200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7640cc480 0x7fa764166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:15.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.416+0000 7fa76a200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa76415d7c0 con 0x7fa76406fe20 2024-04-26T23:09:15.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.416+0000 7fa76a200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa76415daa0 con 0x7fa7640cc480 2024-04-26T23:09:15.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.416+0000 7fa76a200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa76415dd80 con 0x7fa764071e60 2024-04-26T23:09:15.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.416+0000 7fa763e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7640cc480 0x7fa764166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:15.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.416+0000 7fa763e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7640cc480 0x7fa764166ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45074/0 (socket says 172.21.15.40:45074) 2024-04-26T23:09:15.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.416+0000 7fa763e00640 1 -- 172.21.15.40:0/2707657908 learned_addr learned my addr 172.21.15.40:0/2707657908 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:15.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.417+0000 7fa763e00640 1 -- 172.21.15.40:0/2707657908 >> v2:172.21.15.103:3300/0 conn(0x7fa764071e60 msgr2=0x7fa76406f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:15.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.417+0000 7fa763e00640 1 --2- 172.21.15.40:0/2707657908 >> v2:172.21.15.103:3300/0 conn(0x7fa764071e60 0x7fa76406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.417+0000 7fa763e00640 1 -- 172.21.15.40:0/2707657908 >> v1:172.21.15.103:6789/0 conn(0x7fa76406fe20 legacy=0x7fa764072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:15.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.417+0000 7fa763e00640 1 -- 172.21.15.40:0/2707657908 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76415e060 con 0x7fa7640cc480 2024-04-26T23:09:15.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.418+0000 7fa763e00640 1 --2- 172.21.15.40:0/2707657908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7640cc480 0x7fa764166ad0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fa7640b3700 tx=0x7fa75c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb454a3b373e6b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:15.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.419+0000 7fa762000640 1 -- 172.21.15.40:0/2707657908 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c036070 con 0x7fa7640cc480 2024-04-26T23:09:15.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.419+0000 7fa762000640 1 -- 172.21.15.40:0/2707657908 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa75c031600 con 0x7fa7640cc480 2024-04-26T23:09:15.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.419+0000 7fa762000640 1 -- 172.21.15.40:0/2707657908 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c0319d0 con 0x7fa7640cc480 2024-04-26T23:09:15.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.420+0000 7fa76a200640 1 -- 172.21.15.40:0/2707657908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7640cc480 msgr2=0x7fa764166ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:15.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.420+0000 7fa76a200640 1 --2- 172.21.15.40:0/2707657908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7640cc480 0x7fa764166ad0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fa7640b3700 tx=0x7fa75c02a6f0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.420+0000 7fa76a200640 1 -- 172.21.15.40:0/2707657908 shutdown_connections 2024-04-26T23:09:15.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.420+0000 7fa76a200640 1 --2- 172.21.15.40:0/2707657908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7640cc480 0x7fa764166ad0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.420+0000 7fa76a200640 1 --2- 172.21.15.40:0/2707657908 >> v2:172.21.15.103:3300/0 conn(0x7fa764071e60 0x7fa76406f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.420+0000 7fa76a200640 1 -- 172.21.15.40:0/2707657908 >> 172.21.15.40:0/2707657908 conn(0x7fa7640d0be0 msgr2=0x7fa7640d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:15.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.420+0000 7fa76a200640 1 -- 172.21.15.40:0/2707657908 shutdown_connections 2024-04-26T23:09:15.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.420+0000 7fa76a200640 1 -- 172.21.15.40:0/2707657908 wait complete. 2024-04-26T23:09:15.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.422+0000 7fa76a200640 1 Processor -- start 2024-04-26T23:09:15.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.423+0000 7fa76a200640 1 -- start start 2024-04-26T23:09:15.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.424+0000 7fa76a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa76406fe20 0x7fa76414d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:15.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.424+0000 7fa76a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa764071e60 0x7fa7640b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:15.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.424+0000 7fa76a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7640cc480 0x7fa7640b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:15.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.424+0000 7fa76a200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa76415d570 con 0x7fa764071e60 2024-04-26T23:09:15.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.424+0000 7fa76a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa76415d3f0 con 0x7fa76406fe20 2024-04-26T23:09:15.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.424+0000 7fa76a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa764072950 con 0x7fa7640cc480 2024-04-26T23:09:15.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.424+0000 7fa762a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa764071e60 0x7fa7640b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:15.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.424+0000 7fa762a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa764071e60 0x7fa7640b3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45078/0 (socket says 172.21.15.40:45078) 2024-04-26T23:09:15.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.424+0000 7fa762a00640 1 -- 172.21.15.40:0/1501822356 learned_addr learned my addr 172.21.15.40:0/1501822356 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:15.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.425+0000 7fa763400640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa76406fe20 0x7fa76414d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:15.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.425+0000 7fa763e00640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7640cc480 0x7fa7640b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:15.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.425+0000 7fa762a00640 1 -- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7640cc480 msgr2=0x7fa7640b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:15.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.425+0000 7fa762a00640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7640cc480 0x7fa7640b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.425+0000 7fa762a00640 1 -- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa76406fe20 msgr2=0x7fa76414d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:15.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.425+0000 7fa762a00640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa76406fe20 0x7fa76414d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.425+0000 7fa762a00640 1 -- 172.21.15.40:0/1501822356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7640b4340 con 0x7fa764071e60 2024-04-26T23:09:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.425+0000 7fa762a00640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa764071e60 0x7fa7640b3880 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fa75000c430 tx=0x7fa75000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.426+0000 7fa75b400640 1 -- 172.21.15.40:0/1501822356 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750011790 con 0x7fa764071e60 2024-04-26T23:09:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.426+0000 7fa75b400640 1 -- 172.21.15.40:0/1501822356 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa750017070 con 0x7fa764071e60 2024-04-26T23:09:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.426+0000 7fa75b400640 1 -- 172.21.15.40:0/1501822356 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750011bb0 con 0x7fa764071e60 2024-04-26T23:09:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.426+0000 7fa76a200640 1 -- 172.21.15.40:0/1501822356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7640b4620 con 0x7fa764071e60 2024-04-26T23:09:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.426+0000 7fa76a200640 1 -- 172.21.15.40:0/1501822356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7640b4be0 con 0x7fa764071e60 2024-04-26T23:09:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.427+0000 7fa76a200640 1 -- 172.21.15.40:0/1501822356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7640694f0 con 0x7fa764071e60 2024-04-26T23:09:15.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.428+0000 7fa75b400640 1 -- 172.21.15.40:0/1501822356 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa75001c460 con 0x7fa764071e60 2024-04-26T23:09:15.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.429+0000 7fa75b400640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa74406c920 0x7fa74406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:15.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.429+0000 7fa75b400640 1 -- 172.21.15.40:0/1501822356 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fa75008bc00 con 0x7fa764071e60 2024-04-26T23:09:15.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.430+0000 7fa763400640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa74406c920 0x7fa74406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:15.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.431+0000 7fa763400640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa74406c920 0x7fa74406ede0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa754009890 tx=0x7fa754008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:15.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.434+0000 7fa75b400640 1 -- 172.21.15.40:0/1501822356 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa750059e00 con 0x7fa764071e60 2024-04-26T23:09:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.680+0000 7fa76a200640 1 -- 172.21.15.40:0/1501822356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fa764072520 con 0x7fa764071e60 2024-04-26T23:09:15.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.736+0000 7fa75b400640 1 -- 172.21.15.40:0/1501822356 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v112) ==== 105+0+0 (secure 0 0 0) 0x7fa750059700 con 0x7fa764071e60 2024-04-26T23:09:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.740+0000 7fa76a200640 1 -- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa74406c920 msgr2=0x7fa74406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.740+0000 7fa76a200640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa74406c920 0x7fa74406ede0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa754009890 tx=0x7fa754008040 comp rx=0 tx=0).stop 2024-04-26T23:09:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.740+0000 7fa76a200640 1 -- 172.21.15.40:0/1501822356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa764071e60 msgr2=0x7fa7640b3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.740+0000 7fa76a200640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa764071e60 0x7fa7640b3880 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fa75000c430 tx=0x7fa75000c8f0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.741+0000 7fa76a200640 1 -- 172.21.15.40:0/1501822356 shutdown_connections 2024-04-26T23:09:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.741+0000 7fa76a200640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa74406c920 0x7fa74406ede0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.741+0000 7fa76a200640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7640cc480 0x7fa7640b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.741+0000 7fa76a200640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa764071e60 0x7fa7640b3880 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.741+0000 7fa76a200640 1 --2- 172.21.15.40:0/1501822356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa76406fe20 0x7fa76414d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.741+0000 7fa76a200640 1 -- 172.21.15.40:0/1501822356 >> 172.21.15.40:0/1501822356 conn(0x7fa7640d0be0 msgr2=0x7fa76415f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:15.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.741+0000 7fa76a200640 1 -- 172.21.15.40:0/1501822356 shutdown_connections 2024-04-26T23:09:15.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.741+0000 7fa76a200640 1 -- 172.21.15.40:0/1501822356 wait complete. 2024-04-26T23:09:15.766 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:15.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.948+0000 7fa5c6400640 1 Processor -- start 2024-04-26T23:09:15.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.948+0000 7fa5c6400640 1 -- start start 2024-04-26T23:09:15.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.950+0000 7fa5c6400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5c015ce60 0x7fa5c00afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:15.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.950+0000 7fa5c6400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5c0071fe0 0x7fa5c0073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:15.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.950+0000 7fa5c6400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5c00d0820 con 0x7fa5c00af7a0 2024-04-26T23:09:15.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.950+0000 7fa5c6400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fa5c00cc450 con 0x7fa5c0071fe0 2024-04-26T23:09:15.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.950+0000 7fa5c6400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c01614c0 con 0x7fa5c015ce60 2024-04-26T23:09:15.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.950+0000 7fa5c5a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5c0071fe0 0x7fa5c0073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:15.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.950+0000 7fa5c5a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5c0071fe0 0x7fa5c0073760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45088/0 (socket says 172.21.15.40:45088) 2024-04-26T23:09:15.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.950+0000 7fa5c5a00640 1 -- 172.21.15.40:0/2047591504 learned_addr learned my addr 172.21.15.40:0/2047591504 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:15.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.951+0000 7fa5c5000640 1 --2- 172.21.15.40:0/2047591504 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5c015ce60 0x7fa5c00afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:15.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.951+0000 7fa5c5a00640 1 -- 172.21.15.40:0/2047591504 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5c015ce60 msgr2=0x7fa5c00afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:15.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.951+0000 7fa5c5a00640 1 --2- 172.21.15.40:0/2047591504 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5c015ce60 0x7fa5c00afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.951+0000 7fa5c5a00640 1 -- 172.21.15.40:0/2047591504 >> v1:172.21.15.103:6789/0 conn(0x7fa5c00af7a0 legacy=0x7fa5c00ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:15.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.951+0000 7fa5c5a00640 1 -- 172.21.15.40:0/2047591504 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c0161760 con 0x7fa5c0071fe0 2024-04-26T23:09:15.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.952+0000 7fa5c5a00640 1 --2- 172.21.15.40:0/2047591504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c0071fe0 0x7fa5c0073760 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009e50 tx=0x7fa5b802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f93123782d65ea4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:15.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.953+0000 7fa5bf400640 1 -- 172.21.15.40:0/2047591504 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8036070 con 0x7fa5c0071fe0 2024-04-26T23:09:15.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.953+0000 7fa5bf400640 1 -- 172.21.15.40:0/2047591504 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b8031600 con 0x7fa5c0071fe0 2024-04-26T23:09:15.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.953+0000 7fa5bf400640 1 -- 172.21.15.40:0/2047591504 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b80319d0 con 0x7fa5c0071fe0 2024-04-26T23:09:15.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.953+0000 7fa5c6400640 1 -- 172.21.15.40:0/2047591504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c0071fe0 msgr2=0x7fa5c0073760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:15.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.953+0000 7fa5c6400640 1 --2- 172.21.15.40:0/2047591504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c0071fe0 0x7fa5c0073760 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009e50 tx=0x7fa5b802a680 comp rx=0 tx=0).stop 2024-04-26T23:09:15.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.953+0000 7fa5c6400640 1 -- 172.21.15.40:0/2047591504 shutdown_connections 2024-04-26T23:09:15.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.953+0000 7fa5c6400640 1 --2- 172.21.15.40:0/2047591504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c0071fe0 0x7fa5c0073760 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.953+0000 7fa5c6400640 1 --2- 172.21.15.40:0/2047591504 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5c015ce60 0x7fa5c00afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.953+0000 7fa5c6400640 1 -- 172.21.15.40:0/2047591504 >> 172.21.15.40:0/2047591504 conn(0x7fa5c00d0a10 msgr2=0x7fa5c00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:15.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.954+0000 7fa5c6400640 1 -- 172.21.15.40:0/2047591504 shutdown_connections 2024-04-26T23:09:15.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.954+0000 7fa5c6400640 1 -- 172.21.15.40:0/2047591504 wait complete. 2024-04-26T23:09:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.955+0000 7fa5c6400640 1 Processor -- start 2024-04-26T23:09:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.955+0000 7fa5c6400640 1 -- start start 2024-04-26T23:09:15.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.956+0000 7fa5c6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c0071fe0 0x7fa5c0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:15.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.956+0000 7fa5c6400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5c00af7a0 0x7fa5c0158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:15.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.956+0000 7fa5c6400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5c015ce60 0x7fa5c00c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:15.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.956+0000 7fa5c6400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c00cc1f0 con 0x7fa5c0071fe0 2024-04-26T23:09:15.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.956+0000 7fa5c6400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c00cc070 con 0x7fa5c00af7a0 2024-04-26T23:09:15.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.956+0000 7fa5c6400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c0073ca0 con 0x7fa5c015ce60 2024-04-26T23:09:15.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.956+0000 7fa5c5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c0071fe0 0x7fa5c0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:15.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.957+0000 7fa5c5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c0071fe0 0x7fa5c0156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45090/0 (socket says 172.21.15.40:45090) 2024-04-26T23:09:15.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.957+0000 7fa5c5000640 1 -- 172.21.15.40:0/3323616314 learned_addr learned my addr 172.21.15.40:0/3323616314 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:15.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.957+0000 7fa5bfe00640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5c00af7a0 0x7fa5c0158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:15.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.957+0000 7fa5c5a00640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5c015ce60 0x7fa5c00c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:15.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.957+0000 7fa5c5000640 1 -- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5c015ce60 msgr2=0x7fa5c00c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:15.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.957+0000 7fa5c5000640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5c015ce60 0x7fa5c00c6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.957+0000 7fa5c5000640 1 -- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5c00af7a0 msgr2=0x7fa5c0158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:15.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.957+0000 7fa5c5000640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5c00af7a0 0x7fa5c0158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:15.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.957+0000 7fa5c5000640 1 -- 172.21.15.40:0/3323616314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c01575b0 con 0x7fa5c0071fe0 2024-04-26T23:09:15.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.957+0000 7fa5c5000640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c0071fe0 0x7fa5c0156ed0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400d5f0 tx=0x7fa5b400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:15.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.958+0000 7fa5bcc00640 1 -- 172.21.15.40:0/3323616314 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40125f0 con 0x7fa5c0071fe0 2024-04-26T23:09:15.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.958+0000 7fa5bcc00640 1 -- 172.21.15.40:0/3323616314 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b4019070 con 0x7fa5c0071fe0 2024-04-26T23:09:15.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.958+0000 7fa5bcc00640 1 -- 172.21.15.40:0/3323616314 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40129d0 con 0x7fa5c0071fe0 2024-04-26T23:09:15.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.958+0000 7fa5c6400640 1 -- 172.21.15.40:0/3323616314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c0157890 con 0x7fa5c0071fe0 2024-04-26T23:09:15.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.958+0000 7fa5c6400640 1 -- 172.21.15.40:0/3323616314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c0157e50 con 0x7fa5c0071fe0 2024-04-26T23:09:15.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.959+0000 7fa5c6400640 1 -- 172.21.15.40:0/3323616314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5c0069370 con 0x7fa5c0071fe0 2024-04-26T23:09:15.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.960+0000 7fa5bcc00640 1 -- 172.21.15.40:0/3323616314 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa5b4012b70 con 0x7fa5c0071fe0 2024-04-26T23:09:15.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.961+0000 7fa5bcc00640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa5a006c8f0 0x7fa5a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:15.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.961+0000 7fa5bcc00640 1 -- 172.21.15.40:0/3323616314 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fa5b408c0a0 con 0x7fa5c0071fe0 2024-04-26T23:09:15.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.962+0000 7fa5bfe00640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa5a006c8f0 0x7fa5a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.963+0000 7fa5bfe00640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa5a006c8f0 0x7fa5a006edb0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa5c01eb890 tx=0x7fa5ac009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:15.965+0000 7fa5bcc00640 1 -- 172.21.15.40:0/3323616314 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fa5b405a3d0 con 0x7fa5c0071fe0 2024-04-26T23:09:16.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.211+0000 7fa5c6400640 1 -- 172.21.15.40:0/3323616314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5c00d01f0 con 0x7fa5c0071fe0 2024-04-26T23:09:16.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.212+0000 7fa5bcc00640 1 -- 172.21.15.40:0/3323616314 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) ==== 95+0+5208 (secure 0 0 0) 0x7fa5b4059cd0 con 0x7fa5c0071fe0 2024-04-26T23:09:16.212 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:16.212 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":113,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":6598,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:15.740961+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1],"up":{"mds_0":8955,"mds_1":8949},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:creating","state_seq":1,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955]},"id":7}]} 2024-04-26T23:09:16.212 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 113 2024-04-26T23:09:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.214+0000 7fa5c6400640 1 -- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa5a006c8f0 msgr2=0x7fa5a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.214+0000 7fa5c6400640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa5a006c8f0 0x7fa5a006edb0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa5c01eb890 tx=0x7fa5ac009040 comp rx=0 tx=0).stop 2024-04-26T23:09:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.214+0000 7fa5c6400640 1 -- 172.21.15.40:0/3323616314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c0071fe0 msgr2=0x7fa5c0156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.214+0000 7fa5c6400640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c0071fe0 0x7fa5c0156ed0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400d5f0 tx=0x7fa5b400dab0 comp rx=0 tx=0).stop 2024-04-26T23:09:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.214+0000 7fa5c6400640 1 -- 172.21.15.40:0/3323616314 shutdown_connections 2024-04-26T23:09:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.214+0000 7fa5c6400640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa5a006c8f0 0x7fa5a006edb0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.214+0000 7fa5c6400640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa5c015ce60 0x7fa5c00c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.214+0000 7fa5c6400640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5c00af7a0 0x7fa5c0158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:16.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.214+0000 7fa5c6400640 1 --2- 172.21.15.40:0/3323616314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5c0071fe0 0x7fa5c0156ed0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:16.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.214+0000 7fa5c6400640 1 -- 172.21.15.40:0/3323616314 >> 172.21.15.40:0/3323616314 conn(0x7fa5c00d0a10 msgr2=0x7fa5c0160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:16.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.215+0000 7fa5c6400640 1 -- 172.21.15.40:0/3323616314 shutdown_connections 2024-04-26T23:09:16.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:16.215+0000 7fa5c6400640 1 -- 172.21.15.40:0/3323616314 wait complete. 2024-04-26T23:09:16.637 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 113, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:09:06.116498+0000', 'modified': '2024-04-26T23:09:15.740961+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 8955, 'mds_1': 8949}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8949': {'gid': 8949, 'name': 'c', 'rank': 1, 'incarnation': 113, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.40:6833/3250599834', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 3250599834}, {'type': 'v1', 'addr': '172.21.15.40:6833', 'nonce': 3250599834}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8955': {'gid': 8955, 'name': 'a', 'rank': 0, 'incarnation': 107, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/1138762604', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 1138762604}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 1138762604}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8955, 'qdb_cluster': [8955]} 2024-04-26T23:09:16.637 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8949:up:creating 2024-04-26T23:09:17.592 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-04-26T23:09:17.592+0000 7f928e600640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T23:09:17.638 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.838+0000 7f7e2a600640 1 Processor -- start 2024-04-26T23:09:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.839+0000 7f7e2a600640 1 -- start start 2024-04-26T23:09:17.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.840+0000 7f7e2a600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7e240d3340 0x7f7e24163610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:17.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.840+0000 7f7e2a600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e240701e0 con 0x7f7e2415f1d0 2024-04-26T23:09:17.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.840+0000 7f7e2a600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e240704c0 con 0x7f7e24070a80 2024-04-26T23:09:17.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.840+0000 7f7e2a600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7e2415f5b0 con 0x7f7e240d3340 2024-04-26T23:09:17.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.841+0000 7f7e29200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7e2415f1d0 0x7f7e240d0e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35780/0 (socket says 172.21.15.40:35780) 2024-04-26T23:09:17.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.841+0000 7f7e29200640 1 -- 172.21.15.40:0/1738692375 learned_addr learned my addr 172.21.15.40:0/1738692375 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:17.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.841+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1992498675 0 0) 0x7f7e240701e0 con 0x7f7e2415f1d0 2024-04-26T23:09:17.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.841+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e0c003520 con 0x7f7e2415f1d0 2024-04-26T23:09:17.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.841+0000 7f7e29c00640 1 --2- 172.21.15.40:0/1738692375 >> v2:172.21.15.103:3300/0 conn(0x7f7e240d3340 0x7f7e24163610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:17.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.841+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1849767833 0 0) 0x7f7e0c003520 con 0x7f7e2415f1d0 2024-04-26T23:09:17.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.841+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 >> v2:172.21.15.103:3300/0 conn(0x7f7e240d3340 msgr2=0x7f7e24163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:17.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.841+0000 7f7e23400640 1 --2- 172.21.15.40:0/1738692375 >> v2:172.21.15.103:3300/0 conn(0x7f7e240d3340 0x7f7e24163610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:17.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.841+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 >> v1:172.21.15.103:6789/0 conn(0x7f7e24070a80 legacy=0x7f7e240d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:17.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.841+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e240707a0 con 0x7f7e2415f1d0 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.841+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f7e18001670 con 0x7f7e2415f1d0 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.841+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 >> v1:172.21.15.40:6789/0 conn(0x7f7e2415f1d0 legacy=0x7f7e240d0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e23400640 1 --2- 172.21.15.40:0/1738692375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e0c003a00 0x7f7e0c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e23400640 1 --2- 172.21.15.40:0/1738692375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c004800 0x7f7e0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e23400640 1 --2- 172.21.15.40:0/1738692375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e0c0052e0 0x7f7e0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2415f5b0 con 0x7f7e0c004800 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0c00c200 con 0x7f7e0c003a00 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7e0c00c400 con 0x7f7e0c0052e0 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e23e00640 1 --2- 172.21.15.40:0/1738692375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c004800 0x7f7e0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e29200640 1 --2- 172.21.15.40:0/1738692375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e0c003a00 0x7f7e0c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e23e00640 1 -- 172.21.15.40:0/1738692375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e0c0052e0 msgr2=0x7f7e0c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e23e00640 1 --2- 172.21.15.40:0/1738692375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e0c0052e0 0x7f7e0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e23e00640 1 -- 172.21.15.40:0/1738692375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e0c003a00 msgr2=0x7f7e0c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e23e00640 1 --2- 172.21.15.40:0/1738692375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e0c003a00 0x7f7e0c003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:17.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e23e00640 1 -- 172.21.15.40:0/1738692375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0c003520 con 0x7f7e0c004800 2024-04-26T23:09:17.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.842+0000 7f7e29c00640 1 -- 172.21.15.40:0/1738692375 reap_dead start 2024-04-26T23:09:17.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.843+0000 7f7e23e00640 1 --2- 172.21.15.40:0/1738692375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c004800 0x7f7e0c004c80 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f7e10001750 tx=0x7f7e0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93a21f62a316de2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:17.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.843+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10001370 con 0x7f7e0c004800 2024-04-26T23:09:17.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.843+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e100052d0 con 0x7f7e0c004800 2024-04-26T23:09:17.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.843+0000 7f7e23400640 1 -- 172.21.15.40:0/1738692375 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1002abc0 con 0x7f7e0c004800 2024-04-26T23:09:17.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.844+0000 7f7e2a600640 1 -- 172.21.15.40:0/1738692375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c004800 msgr2=0x7f7e0c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:17.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.844+0000 7f7e2a600640 1 --2- 172.21.15.40:0/1738692375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c004800 0x7f7e0c004c80 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f7e10001750 tx=0x7f7e0c005830 comp rx=0 tx=0).stop 2024-04-26T23:09:17.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.844+0000 7f7e2a600640 1 -- 172.21.15.40:0/1738692375 shutdown_connections 2024-04-26T23:09:17.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.844+0000 7f7e2a600640 1 --2- 172.21.15.40:0/1738692375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0c004800 0x7f7e0c004c80 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:17.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.844+0000 7f7e2a600640 1 -- 172.21.15.40:0/1738692375 >> 172.21.15.40:0/1738692375 conn(0x7f7e2409ba50 msgr2=0x7f7e2409be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:17.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.844+0000 7f7e2a600640 1 -- 172.21.15.40:0/1738692375 shutdown_connections 2024-04-26T23:09:17.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.844+0000 7f7e2a600640 1 -- 172.21.15.40:0/1738692375 wait complete. 2024-04-26T23:09:17.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.846+0000 7f7e2a600640 1 Processor -- start 2024-04-26T23:09:17.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.846+0000 7f7e2a600640 1 -- start start 2024-04-26T23:09:17.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.847+0000 7f7e2a600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e0c004800 0x7f7e241ea410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:17.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.847+0000 7f7e2a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e241e97c0 0x7f7e2415a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:17.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.847+0000 7f7e2a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e2415d400 0x7f7e2415d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:17.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.847+0000 7f7e2a600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e24163610 con 0x7f7e2415d400 2024-04-26T23:09:17.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.847+0000 7f7e2a600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2406da80 con 0x7f7e241e97c0 2024-04-26T23:09:17.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.847+0000 7f7e2a600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7e24163790 con 0x7f7e0c004800 2024-04-26T23:09:17.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.847+0000 7f7e29c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e2415d400 0x7f7e2415d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:17.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.847+0000 7f7e29c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e2415d400 0x7f7e2415d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45104/0 (socket says 172.21.15.40:45104) 2024-04-26T23:09:17.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.847+0000 7f7e29c00640 1 -- 172.21.15.40:0/1500472365 learned_addr learned my addr 172.21.15.40:0/1500472365 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:17.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e29200640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e0c004800 0x7f7e241ea410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:17.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e29c00640 1 -- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e0c004800 msgr2=0x7f7e241ea410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:17.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e23e00640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e241e97c0 0x7f7e2415a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:17.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e29c00640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e0c004800 0x7f7e241ea410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:17.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e29c00640 1 -- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e241e97c0 msgr2=0x7f7e2415a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:17.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e29c00640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e241e97c0 0x7f7e2415a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:17.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e29c00640 1 -- 172.21.15.40:0/1500472365 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e241e9c40 con 0x7f7e2415d400 2024-04-26T23:09:17.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e29200640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e0c004800 0x7f7e241ea410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:17.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e29c00640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e2415d400 0x7f7e2415d880 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f7e240d3b20 tx=0x7f7e1c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:17.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e20c00640 1 -- 172.21.15.40:0/1500472365 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c00ed90 con 0x7f7e2415d400 2024-04-26T23:09:17.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e20c00640 1 -- 172.21.15.40:0/1500472365 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e1c019070 con 0x7f7e2415d400 2024-04-26T23:09:17.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e20c00640 1 -- 172.21.15.40:0/1500472365 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c014660 con 0x7f7e2415d400 2024-04-26T23:09:17.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e2a600640 1 -- 172.21.15.40:0/1500472365 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e241eaaf0 con 0x7f7e2415d400 2024-04-26T23:09:17.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.848+0000 7f7e2a600640 1 -- 172.21.15.40:0/1500472365 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e241eb0b0 con 0x7f7e2415d400 2024-04-26T23:09:17.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.850+0000 7f7e2a600640 1 -- 172.21.15.40:0/1500472365 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e24069a80 con 0x7f7e2415d400 2024-04-26T23:09:17.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.851+0000 7f7e20c00640 1 -- 172.21.15.40:0/1500472365 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7e1c014800 con 0x7f7e2415d400 2024-04-26T23:09:17.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.851+0000 7f7e20c00640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e0406c8f0 0x7f7e0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:17.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.851+0000 7f7e20c00640 1 -- 172.21.15.40:0/1500472365 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f7e1c08c100 con 0x7f7e2415d400 2024-04-26T23:09:17.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.852+0000 7f7e29200640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e0406c8f0 0x7f7e0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:17.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.853+0000 7f7e29200640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e0406c8f0 0x7f7e0406edb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7e180017e0 tx=0x7f7e18009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:17.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:17.856+0000 7f7e20c00640 1 -- 172.21.15.40:0/1500472365 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7e1c05a390 con 0x7f7e2415d400 2024-04-26T23:09:18.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.096+0000 7f7e2a600640 1 -- 172.21.15.40:0/1500472365 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e2406d350 con 0x7f7e2415d400 2024-04-26T23:09:18.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.097+0000 7f7e20c00640 1 -- 172.21.15.40:0/1500472365 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+5230 (secure 0 0 0) 0x7f7e1c059c90 con 0x7f7e2415d400 2024-04-26T23:09:18.098 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:18.098 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":116,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:17.762683+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[6598,8949,8955]},"id":7}]} 2024-04-26T23:09:18.098 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 116 2024-04-26T23:09:18.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.100+0000 7f7e2a600640 1 -- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e0406c8f0 msgr2=0x7f7e0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.100+0000 7f7e2a600640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e0406c8f0 0x7f7e0406edb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7e180017e0 tx=0x7f7e18009040 comp rx=0 tx=0).stop 2024-04-26T23:09:18.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.100+0000 7f7e2a600640 1 -- 172.21.15.40:0/1500472365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e2415d400 msgr2=0x7f7e2415d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.100+0000 7f7e2a600640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e2415d400 0x7f7e2415d880 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f7e240d3b20 tx=0x7f7e1c00e4b0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.100+0000 7f7e2a600640 1 -- 172.21.15.40:0/1500472365 shutdown_connections 2024-04-26T23:09:18.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.100+0000 7f7e2a600640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7e0406c8f0 0x7f7e0406edb0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.100+0000 7f7e2a600640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e2415d400 0x7f7e2415d880 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.100+0000 7f7e2a600640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e241e97c0 0x7f7e2415a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.100+0000 7f7e2a600640 1 --2- 172.21.15.40:0/1500472365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7e0c004800 0x7f7e241ea410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.100+0000 7f7e2a600640 1 -- 172.21.15.40:0/1500472365 >> 172.21.15.40:0/1500472365 conn(0x7f7e2409ba50 msgr2=0x7f7e240d01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:18.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.100+0000 7f7e2a600640 1 -- 172.21.15.40:0/1500472365 shutdown_connections 2024-04-26T23:09:18.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.101+0000 7f7e2a600640 1 -- 172.21.15.40:0/1500472365 wait complete. 2024-04-26T23:09:18.521 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:09:06.116498+0000', 'modified': '2024-04-26T23:09:17.762683+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 8955, 'mds_1': 8949, 'mds_2': 6598}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6598': {'gid': 6598, 'name': 'd', 'rank': 2, 'incarnation': 115, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.103:6837/2022020922', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6835', 'nonce': 2022020922}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 2022020922}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8949': {'gid': 8949, 'name': 'c', 'rank': 1, 'incarnation': 113, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.40:6833/3250599834', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 3250599834}, {'type': 'v1', 'addr': '172.21.15.40:6833', 'nonce': 3250599834}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8955': {'gid': 8955, 'name': 'a', 'rank': 0, 'incarnation': 107, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/1138762604', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 1138762604}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 1138762604}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8955, 'qdb_cluster': [6598, 8949, 8955]} 2024-04-26T23:09:18.522 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T23:09:18.522 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-26T23:09:18.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.721+0000 7fb355e00640 1 Processor -- start 2024-04-26T23:09:18.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.721+0000 7fb355e00640 1 -- start start 2024-04-26T23:09:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.723+0000 7fb355e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3500d2490 0x7fb3500cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.723+0000 7fb355e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3500d0200 0x7fb350070900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.723+0000 7fb355e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3500cd9c0 con 0x7fb350070e40 2024-04-26T23:09:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.723+0000 7fb355e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb3500d1b20 con 0x7fb3500d0200 2024-04-26T23:09:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.723+0000 7fb355e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb3500cdca0 con 0x7fb3500d2490 2024-04-26T23:09:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.723+0000 7fb34fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3500d0200 0x7fb350070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.723+0000 7fb34fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3500d0200 0x7fb350070900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45118/0 (socket says 172.21.15.40:45118) 2024-04-26T23:09:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.723+0000 7fb34fe00640 1 -- 172.21.15.40:0/3204864678 learned_addr learned my addr 172.21.15.40:0/3204864678 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.724+0000 7fb354a00640 1 --2- 172.21.15.40:0/3204864678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3500d2490 0x7fb3500cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.724+0000 7fb34fe00640 1 -- 172.21.15.40:0/3204864678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3500d2490 msgr2=0x7fb3500cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.724+0000 7fb34fe00640 1 --2- 172.21.15.40:0/3204864678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3500d2490 0x7fb3500cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.724+0000 7fb34fe00640 1 -- 172.21.15.40:0/3204864678 >> v1:172.21.15.103:6789/0 conn(0x7fb350070e40 legacy=0x7fb35015c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.724+0000 7fb34fe00640 1 -- 172.21.15.40:0/3204864678 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3500cdf80 con 0x7fb3500d0200 2024-04-26T23:09:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.724+0000 7fb354a00640 1 --2- 172.21.15.40:0/3204864678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3500d2490 0x7fb3500cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.725+0000 7fb34fe00640 1 --2- 172.21.15.40:0/3204864678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3500d0200 0x7fb350070900 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fb340009870 tx=0x7fb34002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524467f9c8fc5740 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:18.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.725+0000 7fb34f400640 1 -- 172.21.15.40:0/3204864678 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb340037070 con 0x7fb3500d0200 2024-04-26T23:09:18.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.725+0000 7fb34f400640 1 -- 172.21.15.40:0/3204864678 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb340032440 con 0x7fb3500d0200 2024-04-26T23:09:18.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.725+0000 7fb34f400640 1 -- 172.21.15.40:0/3204864678 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb340032810 con 0x7fb3500d0200 2024-04-26T23:09:18.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.726+0000 7fb355e00640 1 -- 172.21.15.40:0/3204864678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3500d0200 msgr2=0x7fb350070900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.726+0000 7fb355e00640 1 --2- 172.21.15.40:0/3204864678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3500d0200 0x7fb350070900 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fb340009870 tx=0x7fb34002a210 comp rx=0 tx=0).stop 2024-04-26T23:09:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.726+0000 7fb355e00640 1 -- 172.21.15.40:0/3204864678 shutdown_connections 2024-04-26T23:09:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.726+0000 7fb355e00640 1 --2- 172.21.15.40:0/3204864678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3500d0200 0x7fb350070900 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.726+0000 7fb355e00640 1 --2- 172.21.15.40:0/3204864678 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3500d2490 0x7fb3500cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.726+0000 7fb355e00640 1 -- 172.21.15.40:0/3204864678 >> 172.21.15.40:0/3204864678 conn(0x7fb3500d2c10 msgr2=0x7fb3500d3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.726+0000 7fb355e00640 1 -- 172.21.15.40:0/3204864678 shutdown_connections 2024-04-26T23:09:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.726+0000 7fb355e00640 1 -- 172.21.15.40:0/3204864678 wait complete. 2024-04-26T23:09:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.728+0000 7fb355e00640 1 Processor -- start 2024-04-26T23:09:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.728+0000 7fb355e00640 1 -- start start 2024-04-26T23:09:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb350070e40 0x7fb3501f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3500d0200 0x7fb35015f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3500d2490 0x7fb3500b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb350071290 con 0x7fb3500d2490 2024-04-26T23:09:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb3500ce540 con 0x7fb350070e40 2024-04-26T23:09:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb3500702b0 con 0x7fb3500d0200 2024-04-26T23:09:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3500d2490 0x7fb3500b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3500d2490 0x7fb3500b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45128/0 (socket says 172.21.15.40:45128) 2024-04-26T23:09:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355400640 1 -- 172.21.15.40:0/692422096 learned_addr learned my addr 172.21.15.40:0/692422096 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb354a00640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb350070e40 0x7fb3501f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355400640 1 -- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3500d0200 msgr2=0x7fb35015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355400640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3500d0200 0x7fb35015f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355400640 1 -- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb350070e40 msgr2=0x7fb3501f50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355400640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb350070e40 0x7fb3501f50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb355400640 1 -- 172.21.15.40:0/692422096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3500cb270 con 0x7fb3500d2490 2024-04-26T23:09:18.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.729+0000 7fb354a00640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb350070e40 0x7fb3501f50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:18.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.730+0000 7fb355400640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3500d2490 0x7fb3500b4ab0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fb34800c7b0 tx=0x7fb34800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:18.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.730+0000 7fb34cc00640 1 -- 172.21.15.40:0/692422096 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348004b60 con 0x7fb3500d2490 2024-04-26T23:09:18.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.730+0000 7fb34cc00640 1 -- 172.21.15.40:0/692422096 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb348004d00 con 0x7fb3500d2490 2024-04-26T23:09:18.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.730+0000 7fb34cc00640 1 -- 172.21.15.40:0/692422096 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34800aa90 con 0x7fb3500d2490 2024-04-26T23:09:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.730+0000 7fb355e00640 1 -- 172.21.15.40:0/692422096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3500cb530 con 0x7fb3500d2490 2024-04-26T23:09:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.730+0000 7fb355e00640 1 -- 172.21.15.40:0/692422096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3500cbaf0 con 0x7fb3500d2490 2024-04-26T23:09:18.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.733+0000 7fb34cc00640 1 -- 172.21.15.40:0/692422096 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb34800ac30 con 0x7fb3500d2490 2024-04-26T23:09:18.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.734+0000 7fb34cc00640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb33406c8f0 0x7fb33406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.734+0000 7fb34cc00640 1 -- 172.21.15.40:0/692422096 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fb34808bfe0 con 0x7fb3500d2490 2024-04-26T23:09:18.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.734+0000 7fb35a1ed740 1 Processor -- start 2024-04-26T23:09:18.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.735+0000 7fb35a1ed740 1 -- start start 2024-04-26T23:09:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.735+0000 7fb354a00640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb33406c8f0 0x7fb33406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.735+0000 7fb35a1ed740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736867b0 0x55d173688910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.735+0000 7fb35a1ed740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736c2e80 0x55d173686b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.735+0000 7fb35a1ed740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736b1270 0x55d1736b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.735+0000 7fb35a1ed740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d1735ea600 con 0x55d1736867b0 2024-04-26T23:09:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.735+0000 7fb35a1ed740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d173587100 con 0x55d1736c2e80 2024-04-26T23:09:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.735+0000 7fb35a1ed740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d1735e65c0 con 0x55d1736b1270 2024-04-26T23:09:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.736+0000 7fb354a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736867b0 0x55d173688910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.736+0000 7fb354a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736867b0 0x55d173688910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45144/0 (socket says 172.21.15.40:45144) 2024-04-26T23:09:18.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.736+0000 7fb354a00640 1 -- 172.21.15.40:0/3967400469 learned_addr learned my addr 172.21.15.40:0/3967400469 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:18.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.736+0000 7fb34fe00640 1 --2- 172.21.15.40:0/3967400469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736c2e80 0x55d173686b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.736+0000 7fb355400640 1 --2- 172.21.15.40:0/3967400469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736b1270 0x55d1736b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.736+0000 7fb354a00640 1 -- 172.21.15.40:0/3967400469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736b1270 msgr2=0x55d1736b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.736+0000 7fb354a00640 1 --2- 172.21.15.40:0/3967400469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736b1270 0x55d1736b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.736+0000 7fb354a00640 1 -- 172.21.15.40:0/3967400469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736c2e80 msgr2=0x55d173686b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.736+0000 7fb354a00640 1 --2- 172.21.15.40:0/3967400469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736c2e80 0x55d173686b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.736+0000 7fb354a00640 1 -- 172.21.15.40:0/3967400469 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1732fc2d0 con 0x55d1736867b0 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.736+0000 7fb354a00640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb33406c8f0 0x7fb33406edb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb344006fd0 tx=0x7fb344008d10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.737+0000 7fb354a00640 1 --2- 172.21.15.40:0/3967400469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736867b0 0x55d173688910 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fb34400db10 tx=0x7fb344018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a40f93f965449e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.737+0000 7fb33d200640 1 -- 172.21.15.40:0/3967400469 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34401a040 con 0x55d1736867b0 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.737+0000 7fb33d200640 1 -- 172.21.15.40:0/3967400469 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb344023070 con 0x55d1736867b0 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.737+0000 7fb33d200640 1 -- 172.21.15.40:0/3967400469 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34401e430 con 0x55d1736867b0 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.737+0000 7fb35a1ed740 1 -- 172.21.15.40:0/3967400469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736867b0 msgr2=0x55d173688910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.737+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/3967400469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736867b0 0x55d173688910 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fb34400db10 tx=0x7fb344018000 comp rx=0 tx=0).stop 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.738+0000 7fb35a1ed740 1 -- 172.21.15.40:0/3967400469 shutdown_connections 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.738+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/3967400469 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736b1270 0x55d1736b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.738+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/3967400469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736c2e80 0x55d173686b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.738+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/3967400469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736867b0 0x55d173688910 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.738+0000 7fb35a1ed740 1 -- 172.21.15.40:0/3967400469 >> 172.21.15.40:0/3967400469 conn(0x55d17361ad10 msgr2=0x55d1736cc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:18.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.738+0000 7fb35a1ed740 1 -- 172.21.15.40:0/3967400469 shutdown_connections 2024-04-26T23:09:18.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.738+0000 7fb35a1ed740 1 -- 172.21.15.40:0/3967400469 wait complete. 2024-04-26T23:09:18.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.738+0000 7fb35a1ed740 1 -- start start 2024-04-26T23:09:18.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.739+0000 7fb35a1ed740 1 Processor -- start 2024-04-26T23:09:18.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb35a1ed740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736867b0 0x55d1736b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb35a1ed740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736b1270 0x55d1736c8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb35a1ed740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb35a1ed740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d173571f70 con 0x55d1736c2e80 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb35a1ed740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d1736aedc0 con 0x55d1736867b0 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb35a1ed740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d1736c7da0 con 0x55d1736b1270 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb35a1ed740 10 client.0 _ll_register_callbacks cb 0x55d1735e0480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb355400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb355400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45158/0 (socket says 172.21.15.40:45158) 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb355400640 1 -- 172.21.15.40:0/3362499360 learned_addr learned my addr 172.21.15.40:0/3362499360 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb354a00640 1 --2- 172.21.15.40:0/3362499360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736867b0 0x55d1736b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb355400640 1 -- 172.21.15.40:0/3362499360 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736b1270 msgr2=0x55d1736c8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb355400640 1 --2- 172.21.15.40:0/3362499360 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736b1270 0x55d1736c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb355400640 1 -- 172.21.15.40:0/3362499360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736867b0 msgr2=0x55d1736b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb355400640 1 --2- 172.21.15.40:0/3362499360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736867b0 0x55d1736b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.740+0000 7fb355400640 1 -- 172.21.15.40:0/3362499360 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1733c66b0 con 0x55d1736c2e80 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb355400640 1 --2- 172.21.15.40:0/3362499360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c8d70 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fb348059a90 tx=0x7fb348059c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c919dc445d48dda server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb331600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb331600640 1 -- 172.21.15.40:0/3362499360 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348087020 con 0x55d1736c2e80 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb331600640 1 -- 172.21.15.40:0/3362499360 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fb348059d60 con 0x55d1736c2e80 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb35a1ed740 1 -- 172.21.15.40:0/3362499360 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb344023070 con 0x55d1736c2e80 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb331600640 1 -- 172.21.15.40:0/3362499360 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7fb348059f60 con 0x55d1736c2e80 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb35a1ed740 10 client.9135 fetch_fsmap learned FSMap version 116 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb35a1ed740 1 -- 172.21.15.40:0/3362499360 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) -- 0x55d1736c7ae0 con 0x55d1736c2e80 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb331600640 1 -- 172.21.15.40:0/3362499360 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb344023070 con 0x55d1736c2e80 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb331600640 1 -- 172.21.15.40:0/3362499360 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 116) ==== 3534+0+0 (secure 0 0 0) 0x7fb348062250 con 0x55d1736c2e80 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb35a1ed740 10 client.9135 fetch_fsmap finished waiting for FSMap version 116 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb35a1ed740 10 client.9135 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb35a1ed740 20 client.9135 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.741+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/3362499360 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55d1736c6ee0 0x55d1736c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.742+0000 7fb35a1ed740 4 client.9135 mds_command: new command op to 6598 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:18.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.742+0000 7fb35a1ed740 1 -- 172.21.15.40:0/3362499360 --> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1735f8210 con 0x55d1736c6ee0 2024-04-26T23:09:18.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.742+0000 7fb354a00640 1 --2- 172.21.15.40:0/3362499360 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55d1736c6ee0 0x55d1736c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.743+0000 7fb354a00640 1 --2- 172.21.15.40:0/3362499360 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55d1736c6ee0 0x55d1736c72c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3d8d530d1c5aacd7 server_cookie=d479d0d4c116c702 in_seq=0 out_seq=0 2024-04-26T23:09:18.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.743+0000 7fb331600640 10 client.9135 ms_handle_connect on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:18.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.745+0000 7fb331600640 1 -- 172.21.15.40:0/3362499360 <== mds.2 v2:172.21.15.103:6835/2022020922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d1735f8210 con 0x55d1736c6ee0 2024-04-26T23:09:18.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.745+0000 7fb331600640 10 client.9135 handle_command_reply: tid=0 2024-04-26T23:09:18.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.745+0000 7fb35a1ed740 1 client.9135 shutdown 2024-04-26T23:09:18.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.746+0000 7fb35a1ed740 1 -- 172.21.15.40:0/3362499360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 msgr2=0x55d1736c8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.746+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/3362499360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c8d70 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fb348059a90 tx=0x7fb348059c40 comp rx=0 tx=0).stop 2024-04-26T23:09:18.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.746+0000 7fb35a1ed740 1 -- 172.21.15.40:0/3362499360 shutdown_connections 2024-04-26T23:09:18.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.746+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/3362499360 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55d1736c6ee0 0x55d1736c72c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.746+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/3362499360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c8d70 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.746+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/3362499360 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736b1270 0x55d1736c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.746+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/3362499360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736867b0 0x55d1736b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.746+0000 7fb331600640 0 client.9135 ms_handle_reset on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:18.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.746+0000 7fb35a1ed740 1 -- 172.21.15.40:0/3362499360 >> 172.21.15.40:0/3362499360 conn(0x55d17361ad10 msgr2=0x55d1736aefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:18.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.746+0000 7fb35a1ed740 1 -- 172.21.15.40:0/3362499360 shutdown_connections 2024-04-26T23:09:18.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.746+0000 7fb35a1ed740 1 -- 172.21.15.40:0/3362499360 wait complete. 2024-04-26T23:09:18.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.746+0000 7fb35a1ed740 20 client.9135 trim_cache size 0 max 16384 2024-04-26T23:09:18.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.766+0000 7fb35a1ed740 1 Processor -- start 2024-04-26T23:09:18.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.766+0000 7fb35a1ed740 1 -- start start 2024-04-26T23:09:18.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.767+0000 7fb35a1ed740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736867b0 0x55d1736cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.767+0000 7fb35a1ed740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736b1270 0x55d1736cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.767+0000 7fb35a1ed740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.767+0000 7fb35a1ed740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb350201020 con 0x55d1736c2e80 2024-04-26T23:09:18.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.767+0000 7fb35a1ed740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb3502016f0 con 0x55d1736867b0 2024-04-26T23:09:18.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.767+0000 7fb35a1ed740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb3500ca5f0 con 0x55d1736b1270 2024-04-26T23:09:18.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.767+0000 7fb355400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.767+0000 7fb355400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45170/0 (socket says 172.21.15.40:45170) 2024-04-26T23:09:18.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.767+0000 7fb355400640 1 -- 172.21.15.40:0/2204557448 learned_addr learned my addr 172.21.15.40:0/2204557448 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.768+0000 7fb355400640 1 -- 172.21.15.40:0/2204557448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736b1270 msgr2=0x55d1736cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.768+0000 7fb354a00640 1 --2- 172.21.15.40:0/2204557448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736867b0 0x55d1736cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.768+0000 7fb355400640 1 --2- 172.21.15.40:0/2204557448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736b1270 0x55d1736cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.768+0000 7fb355400640 1 -- 172.21.15.40:0/2204557448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736867b0 msgr2=0x55d1736cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.768+0000 7fb355400640 1 --2- 172.21.15.40:0/2204557448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736867b0 0x55d1736cea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.768+0000 7fb355400640 1 -- 172.21.15.40:0/2204557448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1736b1f60 con 0x55d1736c2e80 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.768+0000 7fb355400640 1 --2- 172.21.15.40:0/2204557448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c9990 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fb348000f80 tx=0x7fb348071000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd738d0b84970b0e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.768+0000 7fb333e00640 1 -- 172.21.15.40:0/2204557448 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348081070 con 0x55d1736c2e80 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.768+0000 7fb333e00640 1 -- 172.21.15.40:0/2204557448 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb348056170 con 0x55d1736c2e80 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.768+0000 7fb333e00640 1 -- 172.21.15.40:0/2204557448 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348087040 con 0x55d1736c2e80 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.769+0000 7fb35a1ed740 1 -- 172.21.15.40:0/2204557448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 msgr2=0x55d1736c9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.769+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/2204557448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c9990 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fb348000f80 tx=0x7fb348071000 comp rx=0 tx=0).stop 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.769+0000 7fb35a1ed740 1 -- 172.21.15.40:0/2204557448 shutdown_connections 2024-04-26T23:09:18.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.769+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/2204557448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c9990 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.769+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/2204557448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736b1270 0x55d1736cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.769+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/2204557448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736867b0 0x55d1736cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.769+0000 7fb35a1ed740 1 -- 172.21.15.40:0/2204557448 >> 172.21.15.40:0/2204557448 conn(0x55d17361ad10 msgr2=0x55d17367cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:18.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.769+0000 7fb35a1ed740 1 -- 172.21.15.40:0/2204557448 shutdown_connections 2024-04-26T23:09:18.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.769+0000 7fb35a1ed740 1 -- 172.21.15.40:0/2204557448 wait complete. 2024-04-26T23:09:18.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.770+0000 7fb35a1ed740 1 -- start start 2024-04-26T23:09:18.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.770+0000 7fb35a1ed740 1 Processor -- start 2024-04-26T23:09:18.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.771+0000 7fb35a1ed740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736867b0 0x55d1737452d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.771+0000 7fb35a1ed740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736b1270 0x55d1736c9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.771+0000 7fb35a1ed740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.771+0000 7fb35a1ed740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d1736db530 con 0x55d1736c2e80 2024-04-26T23:09:18.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.771+0000 7fb35a1ed740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d1736a3000 con 0x55d1736b1270 2024-04-26T23:09:18.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.771+0000 7fb35a1ed740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d1736df550 con 0x55d1736867b0 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.771+0000 7fb35a1ed740 10 client.0 _ll_register_callbacks cb 0x7fb34805a200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.771+0000 7fb355400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.771+0000 7fb355400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45174/0 (socket says 172.21.15.40:45174) 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.771+0000 7fb355400640 1 -- 172.21.15.40:0/2060960518 learned_addr learned my addr 172.21.15.40:0/2060960518 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.772+0000 7fb354a00640 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736867b0 0x55d1737452d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.772+0000 7fb355400640 1 -- 172.21.15.40:0/2060960518 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736867b0 msgr2=0x55d1737452d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.772+0000 7fb355400640 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736867b0 0x55d1737452d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.772+0000 7fb34fe00640 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736b1270 0x55d1736c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.772+0000 7fb355400640 1 -- 172.21.15.40:0/2060960518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736b1270 msgr2=0x55d1736c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.772+0000 7fb355400640 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736b1270 0x55d1736c9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.772+0000 7fb355400640 1 -- 172.21.15.40:0/2060960518 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d173745810 con 0x55d1736c2e80 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.772+0000 7fb34fe00640 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736b1270 0x55d1736c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.772+0000 7fb355400640 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c9ed0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fb34800c780 tx=0x7fb34808a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed74e66375d9c04 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:18.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.772+0000 7fb331600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:18.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.776+0000 7fb331600640 1 -- 172.21.15.40:0/2060960518 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348081030 con 0x55d1736c2e80 2024-04-26T23:09:18.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.776+0000 7fb331600640 1 -- 172.21.15.40:0/2060960518 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fb348059d60 con 0x55d1736c2e80 2024-04-26T23:09:18.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.776+0000 7fb35a1ed740 1 -- 172.21.15.40:0/2060960518 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb348056170 con 0x55d1736c2e80 2024-04-26T23:09:18.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.776+0000 7fb331600640 1 -- 172.21.15.40:0/2060960518 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7fb348087040 con 0x55d1736c2e80 2024-04-26T23:09:18.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.776+0000 7fb35a1ed740 10 client.9141 fetch_fsmap learned FSMap version 116 2024-04-26T23:09:18.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.776+0000 7fb35a1ed740 1 -- 172.21.15.40:0/2060960518 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) -- 0x55d173745810 con 0x55d1736c2e80 2024-04-26T23:09:18.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.776+0000 7fb331600640 1 -- 172.21.15.40:0/2060960518 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb348056170 con 0x55d1736c2e80 2024-04-26T23:09:18.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.777+0000 7fb331600640 1 -- 172.21.15.40:0/2060960518 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 116) ==== 3534+0+0 (secure 0 0 0) 0x7fb348006b50 con 0x55d1736c2e80 2024-04-26T23:09:18.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.777+0000 7fb35a1ed740 10 client.9141 fetch_fsmap finished waiting for FSMap version 116 2024-04-26T23:09:18.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.777+0000 7fb35a1ed740 10 client.9141 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-26T23:09:18.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.777+0000 7fb35a1ed740 20 client.9141 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:18.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.777+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55d173685490 0x55d173685870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.777+0000 7fb35a1ed740 4 client.9141 mds_command: new command op to 6598 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:09:18.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.777+0000 7fb35a1ed740 1 -- 172.21.15.40:0/2060960518 --> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] -- command(tid 0: {"prefix": "status"}) -- 0x55d1735f2080 con 0x55d173685490 2024-04-26T23:09:18.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.777+0000 7fb354a00640 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55d173685490 0x55d173685870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.779+0000 7fb354a00640 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55d173685490 0x55d173685870 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5e78f6bdb53bf46d server_cookie=dff6c27c02dd63b6 in_seq=0 out_seq=0 2024-04-26T23:09:18.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.779+0000 7fb331600640 10 client.9141 ms_handle_connect on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:18.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.779+0000 7fb331600640 1 -- 172.21.15.40:0/2060960518 <== mds.2 v2:172.21.15.103:6835/2022020922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x55d1735f2080 con 0x55d173685490 2024-04-26T23:09:18.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.779+0000 7fb331600640 10 client.9141 handle_command_reply: tid=0 2024-04-26T23:09:18.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.779+0000 7fb35a1ed740 1 client.9141 shutdown 2024-04-26T23:09:18.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.780+0000 7fb35a1ed740 1 -- 172.21.15.40:0/2060960518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 msgr2=0x55d1736c9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.780+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c9ed0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fb34800c780 tx=0x7fb34808a1d0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.780+0000 7fb35a1ed740 1 -- 172.21.15.40:0/2060960518 shutdown_connections 2024-04-26T23:09:18.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.780+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55d173685490 0x55d173685870 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.780+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1736c2e80 0x55d1736c9ed0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.780+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1736b1270 0x55d1736c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.780+0000 7fb35a1ed740 1 --2- 172.21.15.40:0/2060960518 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d1736867b0 0x55d1737452d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.780+0000 7fb331600640 0 client.9141 ms_handle_reset on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:18.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.780+0000 7fb35a1ed740 1 -- 172.21.15.40:0/2060960518 >> 172.21.15.40:0/2060960518 conn(0x55d17361ad10 msgr2=0x55d1736806d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:18.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.780+0000 7fb35a1ed740 1 -- 172.21.15.40:0/2060960518 shutdown_connections 2024-04-26T23:09:18.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.780+0000 7fb35a1ed740 1 -- 172.21.15.40:0/2060960518 wait complete. 2024-04-26T23:09:18.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.780+0000 7fb35a1ed740 20 client.9141 trim_cache size 0 max 16384 2024-04-26T23:09:18.781 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:18.781 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:09:18.782 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-04-26T23:09:18.782 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6598, 2024-04-26T23:09:18.782 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:09:18.782 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:09:18.782 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:09:18.782 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 2.015157871, 2024-04-26T23:09:18.782 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 116, 2024-04-26T23:09:18.782 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 103, 2024-04-26T23:09:18.782 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 103, 2024-04-26T23:09:18.782 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 17.542443832 2024-04-26T23:09:18.782 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:18.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.781+0000 7fb355e00640 1 -- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb33406c8f0 msgr2=0x7fb33406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.781+0000 7fb355e00640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb33406c8f0 0x7fb33406edb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb344006fd0 tx=0x7fb344008d10 comp rx=0 tx=0).stop 2024-04-26T23:09:18.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.781+0000 7fb355e00640 1 -- 172.21.15.40:0/692422096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3500d2490 msgr2=0x7fb3500b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.781+0000 7fb355e00640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3500d2490 0x7fb3500b4ab0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fb34800c7b0 tx=0x7fb34800cc70 comp rx=0 tx=0).stop 2024-04-26T23:09:18.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.781+0000 7fb355e00640 1 -- 172.21.15.40:0/692422096 shutdown_connections 2024-04-26T23:09:18.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.781+0000 7fb355e00640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb33406c8f0 0x7fb33406edb0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.781+0000 7fb355e00640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3500d2490 0x7fb3500b4ab0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.781+0000 7fb355e00640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3500d0200 0x7fb35015f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.781+0000 7fb355e00640 1 --2- 172.21.15.40:0/692422096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb350070e40 0x7fb3501f50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.781+0000 7fb355e00640 1 -- 172.21.15.40:0/692422096 >> 172.21.15.40:0/692422096 conn(0x7fb3500d2c10 msgr2=0x7fb3500d2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:18.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.781+0000 7fb355e00640 1 -- 172.21.15.40:0/692422096 shutdown_connections 2024-04-26T23:09:18.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.781+0000 7fb355e00640 1 -- 172.21.15.40:0/692422096 wait complete. 2024-04-26T23:09:18.797 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T23:09:18.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.976+0000 7f5f32a00640 1 Processor -- start 2024-04-26T23:09:18.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.976+0000 7f5f32a00640 1 -- start start 2024-04-26T23:09:18.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.978+0000 7f5f32a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5f2c15f380 0x7f5f2c0cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.978+0000 7f5f32a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5f2c0d0620 0x7f5f2c0cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.978+0000 7f5f32a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f2c072c00 con 0x7f5f2c073b70 2024-04-26T23:09:18.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.978+0000 7f5f32a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f5f2c0729b0 con 0x7f5f2c15f380 2024-04-26T23:09:18.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.978+0000 7f5f32a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5f2c072ee0 con 0x7f5f2c0d0620 2024-04-26T23:09:18.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.979+0000 7f5f2be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5f2c15f380 0x7f5f2c0cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.979+0000 7f5f2be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5f2c15f380 0x7f5f2c0cb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45182/0 (socket says 172.21.15.40:45182) 2024-04-26T23:09:18.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.979+0000 7f5f2be00640 1 -- 172.21.15.40:0/528173795 learned_addr learned my addr 172.21.15.40:0/528173795 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:18.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.979+0000 7f5f30a00640 1 --2- 172.21.15.40:0/528173795 >> v2:172.21.15.103:3300/0 conn(0x7f5f2c0d0620 0x7f5f2c0cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.979+0000 7f5f2be00640 1 -- 172.21.15.40:0/528173795 >> v2:172.21.15.103:3300/0 conn(0x7f5f2c0d0620 msgr2=0x7f5f2c0cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.979+0000 7f5f2be00640 1 --2- 172.21.15.40:0/528173795 >> v2:172.21.15.103:3300/0 conn(0x7f5f2c0d0620 0x7f5f2c0cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.979+0000 7f5f2be00640 1 -- 172.21.15.40:0/528173795 >> v1:172.21.15.103:6789/0 conn(0x7f5f2c073b70 legacy=0x7f5f2c0cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.979+0000 7f5f2be00640 1 -- 172.21.15.40:0/528173795 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f2c0731c0 con 0x7f5f2c15f380 2024-04-26T23:09:18.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.979+0000 7f5f30a00640 1 --2- 172.21.15.40:0/528173795 >> v2:172.21.15.103:3300/0 conn(0x7f5f2c0d0620 0x7f5f2c0cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:18.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.980+0000 7f5f2be00640 1 --2- 172.21.15.40:0/528173795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f2c15f380 0x7f5f2c0cb440 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0735d0 tx=0x7f5f1c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610a08612f31bad2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:18.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.981+0000 7f5f2b400640 1 -- 172.21.15.40:0/528173795 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c037070 con 0x7f5f2c15f380 2024-04-26T23:09:18.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.981+0000 7f5f2b400640 1 -- 172.21.15.40:0/528173795 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f1c032440 con 0x7f5f2c15f380 2024-04-26T23:09:18.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.981+0000 7f5f2b400640 1 -- 172.21.15.40:0/528173795 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c032810 con 0x7f5f2c15f380 2024-04-26T23:09:18.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.981+0000 7f5f32a00640 1 -- 172.21.15.40:0/528173795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f2c15f380 msgr2=0x7f5f2c0cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.981+0000 7f5f32a00640 1 --2- 172.21.15.40:0/528173795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f2c15f380 0x7f5f2c0cb440 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0735d0 tx=0x7f5f1c02a230 comp rx=0 tx=0).stop 2024-04-26T23:09:18.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.981+0000 7f5f32a00640 1 -- 172.21.15.40:0/528173795 shutdown_connections 2024-04-26T23:09:18.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.981+0000 7f5f32a00640 1 --2- 172.21.15.40:0/528173795 >> v2:172.21.15.103:3300/0 conn(0x7f5f2c0d0620 0x7f5f2c0cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.981+0000 7f5f32a00640 1 --2- 172.21.15.40:0/528173795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f2c15f380 0x7f5f2c0cb440 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.981+0000 7f5f32a00640 1 -- 172.21.15.40:0/528173795 >> 172.21.15.40:0/528173795 conn(0x7f5f2c0d0be0 msgr2=0x7f5f2c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:18.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.982+0000 7f5f32a00640 1 -- 172.21.15.40:0/528173795 shutdown_connections 2024-04-26T23:09:18.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.982+0000 7f5f32a00640 1 -- 172.21.15.40:0/528173795 wait complete. 2024-04-26T23:09:18.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.983+0000 7f5f32a00640 1 Processor -- start 2024-04-26T23:09:18.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.983+0000 7f5f32a00640 1 -- start start 2024-04-26T23:09:18.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.984+0000 7f5f32a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f2c0d0620 0x7f5f2c1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.984+0000 7f5f32a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f2c15f380 0x7f5f2c0b5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.984+0000 7f5f32a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f2c0c70c0 0x7f5f2c0b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.984+0000 7f5f32a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2c073620 con 0x7f5f2c0c70c0 2024-04-26T23:09:18.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.984+0000 7f5f32a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2c0aff50 con 0x7f5f2c15f380 2024-04-26T23:09:18.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.984+0000 7f5f32a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5f2c0737a0 con 0x7f5f2c0d0620 2024-04-26T23:09:18.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.985+0000 7f5f30a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f2c0c70c0 0x7f5f2c0b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.985+0000 7f5f30a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f2c0c70c0 0x7f5f2c0b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45186/0 (socket says 172.21.15.40:45186) 2024-04-26T23:09:18.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.985+0000 7f5f30a00640 1 -- 172.21.15.40:0/26449476 learned_addr learned my addr 172.21.15.40:0/26449476 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:18.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.985+0000 7f5f2be00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f2c0d0620 0x7f5f2c1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.985+0000 7f5f23e00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f2c15f380 0x7f5f2c0b5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.985+0000 7f5f30a00640 1 -- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f2c0d0620 msgr2=0x7f5f2c1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.985+0000 7f5f30a00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f2c0d0620 0x7f5f2c1554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.985+0000 7f5f30a00640 1 -- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f2c15f380 msgr2=0x7f5f2c0b5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.985+0000 7f5f30a00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f2c15f380 0x7f5f2c0b5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.985+0000 7f5f30a00640 1 -- 172.21.15.40:0/26449476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f2c0c50c0 con 0x7f5f2c0c70c0 2024-04-26T23:09:18.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.985+0000 7f5f2be00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f2c0d0620 0x7f5f2c1554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:09:18.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.986+0000 7f5f30a00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f2c0c70c0 0x7f5f2c0b3ec0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400cef0 tx=0x7f5f2400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:18.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.986+0000 7f5f28c00640 1 -- 172.21.15.40:0/26449476 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2401a070 con 0x7f5f2c0c70c0 2024-04-26T23:09:18.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.986+0000 7f5f28c00640 1 -- 172.21.15.40:0/26449476 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f2400d860 con 0x7f5f2c0c70c0 2024-04-26T23:09:18.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.986+0000 7f5f28c00640 1 -- 172.21.15.40:0/26449476 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2400dc50 con 0x7f5f2c0c70c0 2024-04-26T23:09:18.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.986+0000 7f5f32a00640 1 -- 172.21.15.40:0/26449476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f2c0c53a0 con 0x7f5f2c0c70c0 2024-04-26T23:09:18.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.986+0000 7f5f32a00640 1 -- 172.21.15.40:0/26449476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f2c0c5960 con 0x7f5f2c0c70c0 2024-04-26T23:09:18.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.988+0000 7f5f28c00640 1 -- 172.21.15.40:0/26449476 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5f2401d070 con 0x7f5f2c0c70c0 2024-04-26T23:09:18.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.989+0000 7f5f28c00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5f0806c8f0 0x7f5f0806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.989+0000 7f5f28c00640 1 -- 172.21.15.40:0/26449476 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f5f2408c660 con 0x7f5f2c0c70c0 2024-04-26T23:09:18.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.990+0000 7f5f3585c740 1 Processor -- start 2024-04-26T23:09:18.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.990+0000 7f5f2be00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5f0806c8f0 0x7f5f0806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.990+0000 7f5f3585c740 1 -- start start 2024-04-26T23:09:18.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.991+0000 7f5f3585c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12647740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.991+0000 7f5f3585c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b12681cb0 0x561b126459c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.991+0000 7f5f3585c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b126700a0 0x561b12670520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.991+0000 7f5f3585c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561b125a9600 con 0x561b126455e0 2024-04-26T23:09:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.991+0000 7f5f3585c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561b12546100 con 0x561b126700a0 2024-04-26T23:09:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.991+0000 7f5f3585c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561b125a55c0 con 0x561b12681cb0 2024-04-26T23:09:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.991+0000 7f5f2be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12647740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.991+0000 7f5f2be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12647740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45198/0 (socket says 172.21.15.40:45198) 2024-04-26T23:09:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.991+0000 7f5f2be00640 1 -- 172.21.15.40:0/4165698412 learned_addr learned my addr 172.21.15.40:0/4165698412 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.992+0000 7f5f2be00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5f0806c8f0 0x7f5f0806edb0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c034000 tx=0x7f5f1c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.992+0000 7f5f23e00640 1 --2- 172.21.15.40:0/4165698412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b12681cb0 0x561b126459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.992+0000 7f5f30a00640 1 --2- 172.21.15.40:0/4165698412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b126700a0 0x561b12670520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.992+0000 7f5f2be00640 1 -- 172.21.15.40:0/4165698412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b12681cb0 msgr2=0x561b126459c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.992+0000 7f5f2be00640 1 --2- 172.21.15.40:0/4165698412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b12681cb0 0x561b126459c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.992+0000 7f5f2be00640 1 -- 172.21.15.40:0/4165698412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b126700a0 msgr2=0x561b12670520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.992+0000 7f5f2be00640 1 --2- 172.21.15.40:0/4165698412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b126700a0 0x561b12670520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.992+0000 7f5f2be00640 1 -- 172.21.15.40:0/4165698412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b122bb2d0 con 0x561b126455e0 2024-04-26T23:09:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.992+0000 7f5f23e00640 1 --2- 172.21.15.40:0/4165698412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b12681cb0 0x561b126459c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.992+0000 7f5f2be00640 1 --2- 172.21.15.40:0/4165698412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12647740 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0090b0 tx=0x7f5f1c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48face0d7023481 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.992+0000 7f5f1be00640 1 -- 172.21.15.40:0/4165698412 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c04e070 con 0x561b126455e0 2024-04-26T23:09:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.992+0000 7f5f1be00640 1 -- 172.21.15.40:0/4165698412 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f1c03e720 con 0x561b126455e0 2024-04-26T23:09:18.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.992+0000 7f5f1be00640 1 -- 172.21.15.40:0/4165698412 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c037040 con 0x561b126455e0 2024-04-26T23:09:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.993+0000 7f5f3585c740 1 -- 172.21.15.40:0/4165698412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 msgr2=0x561b12647740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.993+0000 7f5f3585c740 1 --2- 172.21.15.40:0/4165698412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12647740 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0090b0 tx=0x7f5f1c009240 comp rx=0 tx=0).stop 2024-04-26T23:09:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.993+0000 7f5f3585c740 1 -- 172.21.15.40:0/4165698412 shutdown_connections 2024-04-26T23:09:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.993+0000 7f5f3585c740 1 --2- 172.21.15.40:0/4165698412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b126700a0 0x561b12670520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.993+0000 7f5f3585c740 1 --2- 172.21.15.40:0/4165698412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b12681cb0 0x561b126459c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.993+0000 7f5f3585c740 1 --2- 172.21.15.40:0/4165698412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12647740 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:18.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.993+0000 7f5f3585c740 1 -- 172.21.15.40:0/4165698412 >> 172.21.15.40:0/4165698412 conn(0x561b125d9d10 msgr2=0x561b1268b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.993+0000 7f5f3585c740 1 -- 172.21.15.40:0/4165698412 shutdown_connections 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.993+0000 7f5f3585c740 1 -- 172.21.15.40:0/4165698412 wait complete. 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.994+0000 7f5f3585c740 1 -- start start 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.994+0000 7f5f3585c740 1 Processor -- start 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.995+0000 7f5f3585c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12683c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.995+0000 7f5f3585c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b126700a0 0x561b12684160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.995+0000 7f5f3585c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 0x561b12671970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.995+0000 7f5f3585c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561b12530f70 con 0x561b126455e0 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.995+0000 7f5f3585c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561b1266dbf0 con 0x561b12681cb0 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.995+0000 7f5f3585c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561b12672890 con 0x561b126700a0 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.995+0000 7f5f3585c740 10 client.0 _ll_register_callbacks cb 0x561b1259f480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.995+0000 7f5f2be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12683c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.995+0000 7f5f2be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12683c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45204/0 (socket says 172.21.15.40:45204) 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.995+0000 7f5f2be00640 1 -- 172.21.15.40:0/3745591549 learned_addr learned my addr 172.21.15.40:0/3745591549 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:18.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.996+0000 7f5f23e00640 1 --2- 172.21.15.40:0/3745591549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b126700a0 0x561b12684160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.996+0000 7f5f2be00640 1 -- 172.21.15.40:0/3745591549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b126700a0 msgr2=0x561b12684160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.996+0000 7f5f2be00640 1 --2- 172.21.15.40:0/3745591549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b126700a0 0x561b12684160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.996+0000 7f5f2be00640 1 -- 172.21.15.40:0/3745591549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 msgr2=0x561b12671970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.996+0000 7f5f30a00640 1 --2- 172.21.15.40:0/3745591549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 0x561b12671970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.996+0000 7f5f2be00640 1 --2- 172.21.15.40:0/3745591549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 0x561b12671970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.996+0000 7f5f2be00640 1 -- 172.21.15.40:0/3745591549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b123856b0 con 0x561b126455e0 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.996+0000 7f5f2be00640 1 --2- 172.21.15.40:0/3745591549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12683c20 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c04c040 tx=0x7f5f1c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5d5dc0aff8411b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.996+0000 7f5f18c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.996+0000 7f5f18c00640 1 -- 172.21.15.40:0/3745591549 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c057060 con 0x561b126455e0 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.996+0000 7f5f18c00640 1 -- 172.21.15.40:0/3745591549 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f5f1c03ead0 con 0x561b126455e0 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.996+0000 7f5f3585c740 1 -- 172.21.15.40:0/3745591549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f1c03e720 con 0x561b126455e0 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.997+0000 7f5f18c00640 1 -- 172.21.15.40:0/3745591549 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f5f1c037030 con 0x561b126455e0 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.997+0000 7f5f3585c740 10 client.9153 fetch_fsmap learned FSMap version 116 2024-04-26T23:09:19.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.997+0000 7f5f3585c740 1 -- 172.21.15.40:0/3745591549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) -- 0x561b126725d0 con 0x561b126455e0 2024-04-26T23:09:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.997+0000 7f5f18c00640 1 -- 172.21.15.40:0/3745591549 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f1c03e720 con 0x561b126455e0 2024-04-26T23:09:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.997+0000 7f5f18c00640 1 -- 172.21.15.40:0/3745591549 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 116) ==== 3534+0+0 (secure 0 0 0) 0x7f5f1c048070 con 0x561b126455e0 2024-04-26T23:09:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.997+0000 7f5f3585c740 10 client.9153 fetch_fsmap finished waiting for FSMap version 116 2024-04-26T23:09:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.997+0000 7f5f3585c740 10 client.9153 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:09:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.997+0000 7f5f3585c740 20 client.9153 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.997+0000 7f5f3585c740 1 --2- 172.21.15.40:0/3745591549 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x561b1269e3c0 0x561b1269e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.997+0000 7f5f3585c740 4 client.9153 mds_command: new command op to 8949 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.997+0000 7f5f3585c740 1 -- 172.21.15.40:0/3745591549 --> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b125b7210 con 0x561b1269e3c0 2024-04-26T23:09:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.997+0000 7f5f23e00640 1 --2- 172.21.15.40:0/3745591549 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x561b1269e3c0 0x561b1269e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.998+0000 7f5f23e00640 1 --2- 172.21.15.40:0/3745591549 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x561b1269e3c0 0x561b1269e7a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=35f6bc07db8b2cf7 server_cookie=fe61af3884ea66e6 in_seq=0 out_seq=0 2024-04-26T23:09:19.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.998+0000 7f5f18c00640 10 client.9153 ms_handle_connect on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:19.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f18c00640 1 -- 172.21.15.40:0/3745591549 <== mds.1 v2:172.21.15.40:6832/3250599834 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b125b7210 con 0x561b1269e3c0 2024-04-26T23:09:19.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f18c00640 10 client.9153 handle_command_reply: tid=0 2024-04-26T23:09:19.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f3585c740 1 client.9153 shutdown 2024-04-26T23:09:19.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f3585c740 1 -- 172.21.15.40:0/3745591549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 msgr2=0x561b12683c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f3585c740 1 --2- 172.21.15.40:0/3745591549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12683c20 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c04c040 tx=0x7f5f1c004730 comp rx=0 tx=0).stop 2024-04-26T23:09:19.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f3585c740 1 -- 172.21.15.40:0/3745591549 shutdown_connections 2024-04-26T23:09:19.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f3585c740 1 --2- 172.21.15.40:0/3745591549 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x561b1269e3c0 0x561b1269e7a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f3585c740 1 --2- 172.21.15.40:0/3745591549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 0x561b12671970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f3585c740 1 --2- 172.21.15.40:0/3745591549 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b126700a0 0x561b12684160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f3585c740 1 --2- 172.21.15.40:0/3745591549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12683c20 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f18c00640 0 client.9153 ms_handle_reset on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:19.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f3585c740 1 -- 172.21.15.40:0/3745591549 >> 172.21.15.40:0/3745591549 conn(0x561b125d9d10 msgr2=0x561b1266ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:19.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f3585c740 1 -- 172.21.15.40:0/3745591549 shutdown_connections 2024-04-26T23:09:19.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:18.999+0000 7f5f3585c740 1 -- 172.21.15.40:0/3745591549 wait complete. 2024-04-26T23:09:19.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.000+0000 7f5f3585c740 20 client.9153 trim_cache size 0 max 16384 2024-04-26T23:09:19.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.019+0000 7f5f3585c740 1 Processor -- start 2024-04-26T23:09:19.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.019+0000 7f5f3585c740 1 -- start start 2024-04-26T23:09:19.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f3585c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b126455e0 0x561b126a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f3585c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126700a0 0x561b12683b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f3585c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 0x561b126840a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f3585c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2c200ec0 con 0x561b126700a0 2024-04-26T23:09:19.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f3585c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2c201590 con 0x561b12681cb0 2024-04-26T23:09:19.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f3585c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5f2c1522a0 con 0x561b126455e0 2024-04-26T23:09:19.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f23e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126700a0 0x561b12683b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f23e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126700a0 0x561b12683b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45218/0 (socket says 172.21.15.40:45218) 2024-04-26T23:09:19.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f23e00640 1 -- 172.21.15.40:0/4088596976 learned_addr learned my addr 172.21.15.40:0/4088596976 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:19.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f30a00640 1 --2- 172.21.15.40:0/4088596976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 0x561b126840a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f2be00640 1 --2- 172.21.15.40:0/4088596976 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b126455e0 0x561b126a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f23e00640 1 -- 172.21.15.40:0/4088596976 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b126455e0 msgr2=0x561b126a4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f23e00640 1 --2- 172.21.15.40:0/4088596976 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b126455e0 0x561b126a4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f23e00640 1 -- 172.21.15.40:0/4088596976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 msgr2=0x561b126840a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f23e00640 1 --2- 172.21.15.40:0/4088596976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 0x561b126840a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.020+0000 7f5f23e00640 1 -- 172.21.15.40:0/4088596976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b12685560 con 0x561b126700a0 2024-04-26T23:09:19.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.021+0000 7f5f23e00640 1 --2- 172.21.15.40:0/4088596976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126700a0 0x561b12683b60 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f5f14004e00 tx=0x7f5f14007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161a37e64e2f7474 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:19.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.021+0000 7f5f1b400640 1 -- 172.21.15.40:0/4088596976 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f14007030 con 0x561b126700a0 2024-04-26T23:09:19.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.021+0000 7f5f1b400640 1 -- 172.21.15.40:0/4088596976 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f1401e020 con 0x561b126700a0 2024-04-26T23:09:19.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.021+0000 7f5f1b400640 1 -- 172.21.15.40:0/4088596976 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f14007230 con 0x561b126700a0 2024-04-26T23:09:19.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.022+0000 7f5f3585c740 1 -- 172.21.15.40:0/4088596976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126700a0 msgr2=0x561b12683b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.022+0000 7f5f3585c740 1 --2- 172.21.15.40:0/4088596976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126700a0 0x561b12683b60 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f5f14004e00 tx=0x7f5f14007d50 comp rx=0 tx=0).stop 2024-04-26T23:09:19.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.022+0000 7f5f3585c740 1 -- 172.21.15.40:0/4088596976 shutdown_connections 2024-04-26T23:09:19.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.022+0000 7f5f3585c740 1 --2- 172.21.15.40:0/4088596976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 0x561b126840a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.022+0000 7f5f3585c740 1 --2- 172.21.15.40:0/4088596976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126700a0 0x561b12683b60 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.022+0000 7f5f3585c740 1 --2- 172.21.15.40:0/4088596976 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b126455e0 0x561b126a4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.022+0000 7f5f3585c740 1 -- 172.21.15.40:0/4088596976 >> 172.21.15.40:0/4088596976 conn(0x561b125d9d10 msgr2=0x561b126736e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:19.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.022+0000 7f5f3585c740 1 -- 172.21.15.40:0/4088596976 shutdown_connections 2024-04-26T23:09:19.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.022+0000 7f5f3585c740 1 -- 172.21.15.40:0/4088596976 wait complete. 2024-04-26T23:09:19.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.023+0000 7f5f3585c740 1 -- start start 2024-04-26T23:09:19.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.023+0000 7f5f3585c740 1 Processor -- start 2024-04-26T23:09:19.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f3585c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12705510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f3585c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 0x561b12686140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f3585c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b1266ec30 0x561b12686680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f3585c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561b126712e0 con 0x561b126455e0 2024-04-26T23:09:19.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f3585c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561b1266d470 con 0x561b12681cb0 2024-04-26T23:09:19.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f3585c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561b1266ce90 con 0x561b1266ec30 2024-04-26T23:09:19.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f3585c740 10 client.0 _ll_register_callbacks cb 0x7f5f1c04c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:19.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f2be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12705510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f2be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12705510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45228/0 (socket says 172.21.15.40:45228) 2024-04-26T23:09:19.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f2be00640 1 -- 172.21.15.40:0/759559061 learned_addr learned my addr 172.21.15.40:0/759559061 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:19.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f2be00640 1 -- 172.21.15.40:0/759559061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b1266ec30 msgr2=0x561b12686680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f2be00640 1 --2- 172.21.15.40:0/759559061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b1266ec30 0x561b12686680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f2be00640 1 -- 172.21.15.40:0/759559061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 msgr2=0x561b12686140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f2be00640 1 --2- 172.21.15.40:0/759559061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 0x561b12686140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.024+0000 7f5f2be00640 1 -- 172.21.15.40:0/759559061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b12672380 con 0x561b126455e0 2024-04-26T23:09:19.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.025+0000 7f5f2be00640 1 --2- 172.21.15.40:0/759559061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12705510 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c004450 tx=0x7f5f1c040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5c9f612256bf85 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:19.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.025+0000 7f5f18c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:19.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.025+0000 7f5f18c00640 1 -- 172.21.15.40:0/759559061 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c066060 con 0x561b126455e0 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.025+0000 7f5f18c00640 1 -- 172.21.15.40:0/759559061 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f5f1c003a20 con 0x561b126455e0 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.025+0000 7f5f3585c740 1 -- 172.21.15.40:0/759559061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f1401e020 con 0x561b126455e0 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.025+0000 7f5f18c00640 1 -- 172.21.15.40:0/759559061 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f5f1c06c050 con 0x561b126455e0 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.025+0000 7f5f3585c740 10 client.9159 fetch_fsmap learned FSMap version 116 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.025+0000 7f5f3585c740 1 -- 172.21.15.40:0/759559061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) -- 0x561b12672380 con 0x561b126455e0 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.025+0000 7f5f18c00640 1 -- 172.21.15.40:0/759559061 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f1401e020 con 0x561b126455e0 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.026+0000 7f5f18c00640 1 -- 172.21.15.40:0/759559061 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 116) ==== 3534+0+0 (secure 0 0 0) 0x7f5f1c06f020 con 0x561b126455e0 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.026+0000 7f5f3585c740 10 client.9159 fetch_fsmap finished waiting for FSMap version 116 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.026+0000 7f5f3585c740 10 client.9159 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.026+0000 7f5f3585c740 20 client.9159 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.026+0000 7f5f3585c740 1 --2- 172.21.15.40:0/759559061 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x561b1269d930 0x561b12682610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.026+0000 7f5f3585c740 4 client.9159 mds_command: new command op to 8949 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.026+0000 7f5f3585c740 1 -- 172.21.15.40:0/759559061 --> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] -- command(tid 0: {"prefix": "status"}) -- 0x561b125b1080 con 0x561b1269d930 2024-04-26T23:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.026+0000 7f5f23e00640 1 --2- 172.21.15.40:0/759559061 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x561b1269d930 0x561b12682610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.026+0000 7f5f18c00640 10 client.9159 ms_handle_connect on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.026+0000 7f5f23e00640 1 --2- 172.21.15.40:0/759559061 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x561b1269d930 0x561b12682610 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c09cf80a80506899 server_cookie=88137e24cf317b2c in_seq=0 out_seq=0 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f18c00640 1 -- 172.21.15.40:0/759559061 <== mds.1 v2:172.21.15.40:6832/3250599834 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x561b125b1080 con 0x561b1269d930 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f18c00640 10 client.9159 handle_command_reply: tid=0 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f3585c740 1 client.9159 shutdown 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f3585c740 1 -- 172.21.15.40:0/759559061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 msgr2=0x561b12705510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f3585c740 1 --2- 172.21.15.40:0/759559061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12705510 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c004450 tx=0x7f5f1c040040 comp rx=0 tx=0).stop 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f3585c740 1 -- 172.21.15.40:0/759559061 shutdown_connections 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f3585c740 1 --2- 172.21.15.40:0/759559061 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x561b1269d930 0x561b12682610 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f3585c740 1 --2- 172.21.15.40:0/759559061 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b1266ec30 0x561b12686680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f3585c740 1 --2- 172.21.15.40:0/759559061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b12681cb0 0x561b12686140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f3585c740 1 --2- 172.21.15.40:0/759559061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b126455e0 0x561b12705510 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f18c00640 0 client.9159 ms_handle_reset on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:19.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f3585c740 1 -- 172.21.15.40:0/759559061 >> 172.21.15.40:0/759559061 conn(0x561b125d9d10 msgr2=0x561b12683b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:19.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f3585c740 1 -- 172.21.15.40:0/759559061 shutdown_connections 2024-04-26T23:09:19.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f3585c740 1 -- 172.21.15.40:0/759559061 wait complete. 2024-04-26T23:09:19.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.027+0000 7f5f3585c740 20 client.9159 trim_cache size 0 max 16384 2024-04-26T23:09:19.033 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:09:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-04-26T23:09:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8949, 2024-04-26T23:09:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:09:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:09:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:09:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.2778181000000002, 2024-04-26T23:09:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 116, 2024-04-26T23:09:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 103, 2024-04-26T23:09:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 103, 2024-04-26T23:09:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 17.802767847999998 2024-04-26T23:09:19.034 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:19.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.028+0000 7f5f32a00640 1 -- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5f0806c8f0 msgr2=0x7f5f0806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.028+0000 7f5f32a00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5f0806c8f0 0x7f5f0806edb0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c034000 tx=0x7f5f1c02e000 comp rx=0 tx=0).stop 2024-04-26T23:09:19.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.028+0000 7f5f32a00640 1 -- 172.21.15.40:0/26449476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f2c0c70c0 msgr2=0x7f5f2c0b3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.028+0000 7f5f32a00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f2c0c70c0 0x7f5f2c0b3ec0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400cef0 tx=0x7f5f2400a710 comp rx=0 tx=0).stop 2024-04-26T23:09:19.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.028+0000 7f5f32a00640 1 -- 172.21.15.40:0/26449476 shutdown_connections 2024-04-26T23:09:19.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.028+0000 7f5f32a00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5f0806c8f0 0x7f5f0806edb0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.028+0000 7f5f32a00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f2c0c70c0 0x7f5f2c0b3ec0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.028+0000 7f5f32a00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f2c15f380 0x7f5f2c0b5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.028+0000 7f5f32a00640 1 --2- 172.21.15.40:0/26449476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5f2c0d0620 0x7f5f2c1554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.028+0000 7f5f32a00640 1 -- 172.21.15.40:0/26449476 >> 172.21.15.40:0/26449476 conn(0x7f5f2c0d0be0 msgr2=0x7f5f2c071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:19.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.028+0000 7f5f32a00640 1 -- 172.21.15.40:0/26449476 shutdown_connections 2024-04-26T23:09:19.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.028+0000 7f5f32a00640 1 -- 172.21.15.40:0/26449476 wait complete. 2024-04-26T23:09:19.044 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T23:09:19.223 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:09:19.231 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:09:19.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.238+0000 7f0662800640 1 Processor -- start 2024-04-26T23:09:19.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.238+0000 7f0662800640 1 -- start start 2024-04-26T23:09:19.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.239+0000 7f0662800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f065c0b6220 0x7f065c0d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.239+0000 7f0662800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f065c0d08d0 0x7f065c0d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.239+0000 7f0662800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f065c162c40 con 0x7f065c0d3820 2024-04-26T23:09:19.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.239+0000 7f0662800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f065c0d3c60 con 0x7f065c0b6220 2024-04-26T23:09:19.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.239+0000 7f0662800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f065c0d2d80 con 0x7f065c0d08d0 2024-04-26T23:09:19.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.239+0000 7f0661400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f065c0b6220 0x7f065c0d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.239+0000 7f0661400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f065c0b6220 0x7f065c0d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45236/0 (socket says 172.21.15.40:45236) 2024-04-26T23:09:19.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.239+0000 7f0661400640 1 -- 172.21.15.40:0/248294679 learned_addr learned my addr 172.21.15.40:0/248294679 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:19.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.240+0000 7f0661400640 1 -- 172.21.15.40:0/248294679 >> v2:172.21.15.103:3300/0 conn(0x7f065c0d08d0 msgr2=0x7f065c0d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.240+0000 7f0661400640 1 --2- 172.21.15.40:0/248294679 >> v2:172.21.15.103:3300/0 conn(0x7f065c0d08d0 0x7f065c0d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.240+0000 7f0661400640 1 -- 172.21.15.40:0/248294679 >> v1:172.21.15.103:6789/0 conn(0x7f065c0d3820 legacy=0x7f065c0b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.240+0000 7f0661400640 1 -- 172.21.15.40:0/248294679 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f065c0b7c80 con 0x7f065c0b6220 2024-04-26T23:09:19.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.241+0000 7f0661400640 1 --2- 172.21.15.40:0/248294679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c0b6220 0x7f065c0d0390 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f065c08f300 tx=0x7f065002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab65047a3c87a47 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:19.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.241+0000 7f065be00640 1 -- 172.21.15.40:0/248294679 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0650037070 con 0x7f065c0b6220 2024-04-26T23:09:19.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.241+0000 7f065be00640 1 -- 172.21.15.40:0/248294679 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0650032440 con 0x7f065c0b6220 2024-04-26T23:09:19.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.241+0000 7f065be00640 1 -- 172.21.15.40:0/248294679 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0650032810 con 0x7f065c0b6220 2024-04-26T23:09:19.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.242+0000 7f0662800640 1 -- 172.21.15.40:0/248294679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c0b6220 msgr2=0x7f065c0d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.242+0000 7f0662800640 1 --2- 172.21.15.40:0/248294679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c0b6220 0x7f065c0d0390 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f065c08f300 tx=0x7f065002a230 comp rx=0 tx=0).stop 2024-04-26T23:09:19.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.242+0000 7f0662800640 1 -- 172.21.15.40:0/248294679 shutdown_connections 2024-04-26T23:09:19.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.242+0000 7f0662800640 1 --2- 172.21.15.40:0/248294679 >> v2:172.21.15.103:3300/0 conn(0x7f065c0d08d0 0x7f065c0d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.242+0000 7f0662800640 1 --2- 172.21.15.40:0/248294679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c0b6220 0x7f065c0d0390 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.242+0000 7f0662800640 1 -- 172.21.15.40:0/248294679 >> 172.21.15.40:0/248294679 conn(0x7f065c0d4d90 msgr2=0x7f065c0d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:19.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.242+0000 7f0662800640 1 -- 172.21.15.40:0/248294679 shutdown_connections 2024-04-26T23:09:19.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.242+0000 7f0662800640 1 -- 172.21.15.40:0/248294679 wait complete. 2024-04-26T23:09:19.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.244+0000 7f0662800640 1 Processor -- start 2024-04-26T23:09:19.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.244+0000 7f0662800640 1 -- start start 2024-04-26T23:09:19.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0662800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f065c0b6220 0x7f065c1530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0662800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c0d08d0 0x7f065c0b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0662800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c0d3820 0x7f065c0cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0662800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f065c0b8520 con 0x7f065c0d3820 2024-04-26T23:09:19.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0662800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f065c0b8240 con 0x7f065c0d08d0 2024-04-26T23:09:19.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0662800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f065c0b9d30 con 0x7f065c0b6220 2024-04-26T23:09:19.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0661e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c0d3820 0x7f065c0cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0661e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c0d3820 0x7f065c0cdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45238/0 (socket says 172.21.15.40:45238) 2024-04-26T23:09:19.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0661e00640 1 -- 172.21.15.40:0/315853489 learned_addr learned my addr 172.21.15.40:0/315853489 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:19.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0660a00640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c0d08d0 0x7f065c0b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0661400640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f065c0b6220 0x7f065c1530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0661e00640 1 -- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f065c0b6220 msgr2=0x7f065c1530e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0661e00640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f065c0b6220 0x7f065c1530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.245+0000 7f0661e00640 1 -- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c0d08d0 msgr2=0x7f065c0b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.246+0000 7f0661e00640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c0d08d0 0x7f065c0b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.246+0000 7f0661e00640 1 -- 172.21.15.40:0/315853489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f065c153620 con 0x7f065c0d3820 2024-04-26T23:09:19.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.246+0000 7f0660a00640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c0d08d0 0x7f065c0b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:19.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.246+0000 7f0661e00640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c0d3820 0x7f065c0cdde0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f065400b990 tx=0x7f065400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:19.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.246+0000 7f0659600640 1 -- 172.21.15.40:0/315853489 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0654019070 con 0x7f065c0d3820 2024-04-26T23:09:19.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.246+0000 7f0659600640 1 -- 172.21.15.40:0/315853489 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f065400cba0 con 0x7f065c0d3820 2024-04-26T23:09:19.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.246+0000 7f0659600640 1 -- 172.21.15.40:0/315853489 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0654014440 con 0x7f065c0d3820 2024-04-26T23:09:19.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.246+0000 7f0662800640 1 -- 172.21.15.40:0/315853489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f065c1f8730 con 0x7f065c0d3820 2024-04-26T23:09:19.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.246+0000 7f0662800640 1 -- 172.21.15.40:0/315853489 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f065c1f8cf0 con 0x7f065c0d3820 2024-04-26T23:09:19.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.249+0000 7f0659600640 1 -- 172.21.15.40:0/315853489 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f065401c070 con 0x7f065c0d3820 2024-04-26T23:09:19.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.250+0000 7f0659600640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f064006c8f0 0x7f064006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.250+0000 7f0659600640 1 -- 172.21.15.40:0/315853489 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f065408ba60 con 0x7f065c0d3820 2024-04-26T23:09:19.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.250+0000 7f0666b86740 1 Processor -- start 2024-04-26T23:09:19.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.250+0000 7f0666b86740 1 -- start start 2024-04-26T23:09:19.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.250+0000 7f0661400640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f064006c8f0 0x7f064006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.251+0000 7f0666b86740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd847137b0 0x55bd84715910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.251+0000 7f0666b86740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8474fe80 0x55bd84713b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.251+0000 7f0666b86740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8473e270 0x55bd8473e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.251+0000 7f0666b86740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bd84677600 con 0x55bd8473e270 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.251+0000 7f0666b86740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bd84614100 con 0x55bd847137b0 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.251+0000 7f0666b86740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bd846735c0 con 0x55bd8474fe80 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.251+0000 7f0661e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8473e270 0x55bd8473e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.251+0000 7f0661e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8473e270 0x55bd8473e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45254/0 (socket says 172.21.15.40:45254) 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.251+0000 7f0661e00640 1 -- 172.21.15.40:0/1009933082 learned_addr learned my addr 172.21.15.40:0/1009933082 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.251+0000 7f0660a00640 1 --2- 172.21.15.40:0/1009933082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8474fe80 0x55bd84713b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.252+0000 7f0661400640 1 --2- 172.21.15.40:0/1009933082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd847137b0 0x55bd84715910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.252+0000 7f0661e00640 1 -- 172.21.15.40:0/1009933082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8474fe80 msgr2=0x55bd84713b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.252+0000 7f0661e00640 1 --2- 172.21.15.40:0/1009933082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8474fe80 0x55bd84713b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.252+0000 7f0661e00640 1 -- 172.21.15.40:0/1009933082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd847137b0 msgr2=0x55bd84715910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.252+0000 7f0661e00640 1 --2- 172.21.15.40:0/1009933082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd847137b0 0x55bd84715910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.252+0000 7f0661e00640 1 -- 172.21.15.40:0/1009933082 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd843892d0 con 0x55bd8473e270 2024-04-26T23:09:19.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.252+0000 7f0661400640 1 --2- 172.21.15.40:0/1009933082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd847137b0 0x55bd84715910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:19.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.252+0000 7f0661400640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f064006c8f0 0x7f064006edb0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0650034000 tx=0x7f065002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:19.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.252+0000 7f0661e00640 1 --2- 172.21.15.40:0/1009933082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8473e270 0x55bd8473e6f0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f0654089da0 tx=0x7f06540596c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3f8c092f08bc78 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:19.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.252+0000 7f0649c00640 1 -- 172.21.15.40:0/1009933082 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06540621e0 con 0x55bd8473e270 2024-04-26T23:09:19.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.252+0000 7f0649c00640 1 -- 172.21.15.40:0/1009933082 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0654057900 con 0x55bd8473e270 2024-04-26T23:09:19.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.252+0000 7f0649c00640 1 -- 172.21.15.40:0/1009933082 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0654061d50 con 0x55bd8473e270 2024-04-26T23:09:19.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.253+0000 7f0666b86740 1 -- 172.21.15.40:0/1009933082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8473e270 msgr2=0x55bd8473e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.253+0000 7f0666b86740 1 --2- 172.21.15.40:0/1009933082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8473e270 0x55bd8473e6f0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f0654089da0 tx=0x7f06540596c0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.253+0000 7f0666b86740 1 -- 172.21.15.40:0/1009933082 shutdown_connections 2024-04-26T23:09:19.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.253+0000 7f0666b86740 1 --2- 172.21.15.40:0/1009933082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8473e270 0x55bd8473e6f0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.253+0000 7f0666b86740 1 --2- 172.21.15.40:0/1009933082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8474fe80 0x55bd84713b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.253+0000 7f0666b86740 1 --2- 172.21.15.40:0/1009933082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd847137b0 0x55bd84715910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.253+0000 7f0666b86740 1 -- 172.21.15.40:0/1009933082 >> 172.21.15.40:0/1009933082 conn(0x55bd846a7d10 msgr2=0x55bd84759270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:19.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.253+0000 7f0666b86740 1 -- 172.21.15.40:0/1009933082 shutdown_connections 2024-04-26T23:09:19.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.253+0000 7f0666b86740 1 -- 172.21.15.40:0/1009933082 wait complete. 2024-04-26T23:09:19.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.254+0000 7f0666b86740 1 -- start start 2024-04-26T23:09:19.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.254+0000 7f0666b86740 1 Processor -- start 2024-04-26T23:09:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.255+0000 7f0666b86740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 0x55bd8473f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.255+0000 7f0666b86740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd8473e270 0x55bd8472ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.255+0000 7f0666b86740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8474fe80 0x55bd84754000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.255+0000 7f0666b86740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bd845fef70 con 0x55bd847137b0 2024-04-26T23:09:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.255+0000 7f0666b86740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bd8473bdc0 con 0x55bd8473e270 2024-04-26T23:09:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.255+0000 7f0666b86740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bd84742030 con 0x55bd8474fe80 2024-04-26T23:09:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.256+0000 7f0666b86740 10 client.0 _ll_register_callbacks cb 0x55bd8466d480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.256+0000 7f0661400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 0x55bd8473f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.256+0000 7f0661400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 0x55bd8473f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45266/0 (socket says 172.21.15.40:45266) 2024-04-26T23:09:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.256+0000 7f0661400640 1 -- 172.21.15.40:0/566892077 learned_addr learned my addr 172.21.15.40:0/566892077 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.256+0000 7f0660a00640 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd8473e270 0x55bd8472ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.256+0000 7f0661e00640 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8474fe80 0x55bd84754000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.256+0000 7f0661400640 1 -- 172.21.15.40:0/566892077 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8474fe80 msgr2=0x55bd84754000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.256+0000 7f0661400640 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8474fe80 0x55bd84754000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.256+0000 7f0661400640 1 -- 172.21.15.40:0/566892077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd8473e270 msgr2=0x55bd8472ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.256+0000 7f0661400640 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd8473e270 0x55bd8472ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.256+0000 7f0661400640 1 -- 172.21.15.40:0/566892077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd844536b0 con 0x55bd847137b0 2024-04-26T23:09:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.256+0000 7f0660a00640 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd8473e270 0x55bd8472ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.256+0000 7f0661e00640 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8474fe80 0x55bd84754000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.257+0000 7f0661400640 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 0x55bd8473f0c0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f0650035040 tx=0x7f0650004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28f9fda5c546d22 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.257+0000 7f063e000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.257+0000 7f063e000640 1 -- 172.21.15.40:0/566892077 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065004e060 con 0x55bd847137b0 2024-04-26T23:09:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.257+0000 7f063e000640 1 -- 172.21.15.40:0/566892077 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f0650032dd0 con 0x55bd847137b0 2024-04-26T23:09:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.257+0000 7f0666b86740 1 -- 172.21.15.40:0/566892077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0654057900 con 0x55bd847137b0 2024-04-26T23:09:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.257+0000 7f063e000640 1 -- 172.21.15.40:0/566892077 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f0650054050 con 0x55bd847137b0 2024-04-26T23:09:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.257+0000 7f0666b86740 10 client.9171 fetch_fsmap learned FSMap version 116 2024-04-26T23:09:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.257+0000 7f0666b86740 1 -- 172.21.15.40:0/566892077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) -- 0x55bd847407b0 con 0x55bd847137b0 2024-04-26T23:09:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.258+0000 7f063e000640 1 -- 172.21.15.40:0/566892077 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0654057900 con 0x55bd847137b0 2024-04-26T23:09:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.258+0000 7f063e000640 1 -- 172.21.15.40:0/566892077 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 116) ==== 3534+0+0 (secure 0 0 0) 0x7f0650057020 con 0x55bd847137b0 2024-04-26T23:09:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.258+0000 7f0666b86740 10 client.9171 fetch_fsmap finished waiting for FSMap version 116 2024-04-26T23:09:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.258+0000 7f0666b86740 10 client.9171 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:09:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.258+0000 7f0666b86740 20 client.9171 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.258+0000 7f0666b86740 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55bd8476c5a0 0x55bd847410f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.258+0000 7f0666b86740 4 client.9171 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.258+0000 7f0666b86740 1 -- 172.21.15.40:0/566892077 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd84685210 con 0x55bd8476c5a0 2024-04-26T23:09:19.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.258+0000 7f0660a00640 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55bd8476c5a0 0x55bd847410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.259+0000 7f0660a00640 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55bd8476c5a0 0x55bd847410f0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bf99cfe117f647b server_cookie=8f98ff56cce5b2fd in_seq=0 out_seq=0 2024-04-26T23:09:19.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.259+0000 7f063e000640 10 client.9171 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:19.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.259+0000 7f063e000640 1 -- 172.21.15.40:0/566892077 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bd84685210 con 0x55bd8476c5a0 2024-04-26T23:09:19.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.259+0000 7f063e000640 10 client.9171 handle_command_reply: tid=0 2024-04-26T23:09:19.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.260+0000 7f0666b86740 1 client.9171 shutdown 2024-04-26T23:09:19.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.260+0000 7f0666b86740 1 -- 172.21.15.40:0/566892077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 msgr2=0x55bd8473f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.260+0000 7f0666b86740 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 0x55bd8473f0c0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f0650035040 tx=0x7f0650004730 comp rx=0 tx=0).stop 2024-04-26T23:09:19.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.260+0000 7f0666b86740 1 -- 172.21.15.40:0/566892077 shutdown_connections 2024-04-26T23:09:19.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.260+0000 7f0666b86740 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8474fe80 0x55bd84754000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.260+0000 7f0666b86740 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55bd8476c5a0 0x55bd847410f0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.260+0000 7f0666b86740 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd8473e270 0x55bd8472ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.260+0000 7f0666b86740 1 --2- 172.21.15.40:0/566892077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 0x55bd8473f0c0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.260+0000 7f0666b86740 1 -- 172.21.15.40:0/566892077 >> 172.21.15.40:0/566892077 conn(0x55bd846a7d10 msgr2=0x55bd8475c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:19.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.260+0000 7f063e000640 0 client.9171 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:19.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.260+0000 7f0666b86740 1 -- 172.21.15.40:0/566892077 shutdown_connections 2024-04-26T23:09:19.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.260+0000 7f0666b86740 1 -- 172.21.15.40:0/566892077 wait complete. 2024-04-26T23:09:19.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.260+0000 7f0666b86740 20 client.9171 trim_cache size 0 max 16384 2024-04-26T23:09:19.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.280+0000 7f0666b86740 1 Processor -- start 2024-04-26T23:09:19.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.280+0000 7f0666b86740 1 -- start start 2024-04-26T23:09:19.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.281+0000 7f0666b86740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd847137b0 0x55bd847412b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.281+0000 7f0666b86740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8473e270 0x55bd84738a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.281+0000 7f0666b86740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8474fe80 0x55bd84738f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.281+0000 7f0666b86740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f065c200f40 con 0x55bd8474fe80 2024-04-26T23:09:19.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.281+0000 7f0666b86740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f065c201610 con 0x55bd847137b0 2024-04-26T23:09:19.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.281+0000 7f0666b86740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f065c0b27a0 con 0x55bd8473e270 2024-04-26T23:09:19.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.282+0000 7f0661e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8474fe80 0x55bd84738f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.282+0000 7f0661e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8474fe80 0x55bd84738f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45272/0 (socket says 172.21.15.40:45272) 2024-04-26T23:09:19.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.282+0000 7f0661e00640 1 -- 172.21.15.40:0/3631284664 learned_addr learned my addr 172.21.15.40:0/3631284664 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:19.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.282+0000 7f0660a00640 1 --2- 172.21.15.40:0/3631284664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8473e270 0x55bd84738a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.282+0000 7f0661400640 1 --2- 172.21.15.40:0/3631284664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd847137b0 0x55bd847412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.282+0000 7f0661e00640 1 -- 172.21.15.40:0/3631284664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8473e270 msgr2=0x55bd84738a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.282+0000 7f0661e00640 1 --2- 172.21.15.40:0/3631284664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8473e270 0x55bd84738a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.282+0000 7f0661e00640 1 -- 172.21.15.40:0/3631284664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd847137b0 msgr2=0x55bd847412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.282+0000 7f0661e00640 1 --2- 172.21.15.40:0/3631284664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd847137b0 0x55bd847412b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.282+0000 7f0661e00640 1 -- 172.21.15.40:0/3631284664 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd847117a0 con 0x55bd8474fe80 2024-04-26T23:09:19.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.282+0000 7f0661e00640 1 --2- 172.21.15.40:0/3631284664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8474fe80 0x55bd84738f50 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f0654089e50 tx=0x7f065400e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5f0b43003de054 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:19.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.283+0000 7f0649200640 1 -- 172.21.15.40:0/3631284664 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0654076070 con 0x55bd8474fe80 2024-04-26T23:09:19.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.283+0000 7f0649200640 1 -- 172.21.15.40:0/3631284664 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06540621e0 con 0x55bd8474fe80 2024-04-26T23:09:19.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.283+0000 7f0649200640 1 -- 172.21.15.40:0/3631284664 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0654055c70 con 0x55bd8474fe80 2024-04-26T23:09:19.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.283+0000 7f0666b86740 1 -- 172.21.15.40:0/3631284664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8474fe80 msgr2=0x55bd84738f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.283+0000 7f0666b86740 1 --2- 172.21.15.40:0/3631284664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8474fe80 0x55bd84738f50 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f0654089e50 tx=0x7f065400e070 comp rx=0 tx=0).stop 2024-04-26T23:09:19.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.283+0000 7f0666b86740 1 -- 172.21.15.40:0/3631284664 shutdown_connections 2024-04-26T23:09:19.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.283+0000 7f0666b86740 1 --2- 172.21.15.40:0/3631284664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd8474fe80 0x55bd84738f50 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.283+0000 7f0666b86740 1 --2- 172.21.15.40:0/3631284664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8473e270 0x55bd84738a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.283+0000 7f0666b86740 1 --2- 172.21.15.40:0/3631284664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd847137b0 0x55bd847412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.283+0000 7f0666b86740 1 -- 172.21.15.40:0/3631284664 >> 172.21.15.40:0/3631284664 conn(0x55bd846a7d10 msgr2=0x55bd8476bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:19.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.284+0000 7f0666b86740 1 -- 172.21.15.40:0/3631284664 shutdown_connections 2024-04-26T23:09:19.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.284+0000 7f0666b86740 1 -- 172.21.15.40:0/3631284664 wait complete. 2024-04-26T23:09:19.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.284+0000 7f0666b86740 1 -- start start 2024-04-26T23:09:19.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.285+0000 7f0666b86740 1 Processor -- start 2024-04-26T23:09:19.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0666b86740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 0x55bd847c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0666b86740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8473e270 0x55bd84739270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0666b86740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd8474fe80 0x55bd847c0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0666b86740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bd8473d5a0 con 0x55bd847137b0 2024-04-26T23:09:19.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0666b86740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bd84740a90 con 0x55bd8474fe80 2024-04-26T23:09:19.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0666b86740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55bd8476c9c0 con 0x55bd8473e270 2024-04-26T23:09:19.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0666b86740 10 client.0 _ll_register_callbacks cb 0x7f0650035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:19.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0661400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 0x55bd847c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0661400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 0x55bd847c3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45274/0 (socket says 172.21.15.40:45274) 2024-04-26T23:09:19.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0661400640 1 -- 172.21.15.40:0/2029647932 learned_addr learned my addr 172.21.15.40:0/2029647932 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:19.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0661e00640 1 --2- 172.21.15.40:0/2029647932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd8474fe80 0x55bd847c0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0661400640 1 -- 172.21.15.40:0/2029647932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8473e270 msgr2=0x55bd84739270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0661400640 1 --2- 172.21.15.40:0/2029647932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8473e270 0x55bd84739270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0661400640 1 -- 172.21.15.40:0/2029647932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd8474fe80 msgr2=0x55bd847c0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0661400640 1 --2- 172.21.15.40:0/2029647932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd8474fe80 0x55bd847c0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.286+0000 7f0661400640 1 -- 172.21.15.40:0/2029647932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd847397b0 con 0x55bd847137b0 2024-04-26T23:09:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.287+0000 7f0661400640 1 --2- 172.21.15.40:0/2029647932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 0x55bd847c3ed0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f065003eab0 tx=0x7f065003b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df2b07ab8770976 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:19.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.287+0000 7f063e000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:19.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.287+0000 7f063e000640 1 -- 172.21.15.40:0/2029647932 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0650037040 con 0x55bd847137b0 2024-04-26T23:09:19.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.287+0000 7f063e000640 1 -- 172.21.15.40:0/2029647932 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f0650004310 con 0x55bd847137b0 2024-04-26T23:09:19.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.287+0000 7f0666b86740 1 -- 172.21.15.40:0/2029647932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06540621e0 con 0x55bd847137b0 2024-04-26T23:09:19.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.287+0000 7f063e000640 1 -- 172.21.15.40:0/2029647932 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f0650065050 con 0x55bd847137b0 2024-04-26T23:09:19.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.287+0000 7f0666b86740 10 client.9177 fetch_fsmap learned FSMap version 116 2024-04-26T23:09:19.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.287+0000 7f0666b86740 1 -- 172.21.15.40:0/2029647932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) -- 0x55bd84711fc0 con 0x55bd847137b0 2024-04-26T23:09:19.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.288+0000 7f063e000640 1 -- 172.21.15.40:0/2029647932 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06540621e0 con 0x55bd847137b0 2024-04-26T23:09:19.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.288+0000 7f063e000640 1 -- 172.21.15.40:0/2029647932 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 116) ==== 3534+0+0 (secure 0 0 0) 0x7f0650068020 con 0x55bd847137b0 2024-04-26T23:09:19.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.288+0000 7f0666b86740 10 client.9177 fetch_fsmap finished waiting for FSMap version 116 2024-04-26T23:09:19.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.288+0000 7f0666b86740 10 client.9177 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:09:19.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.288+0000 7f0666b86740 20 client.9177 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:19.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.288+0000 7f0666b86740 1 --2- 172.21.15.40:0/2029647932 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55bd84767910 0x55bd847c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.288+0000 7f0666b86740 4 client.9177 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.288+0000 7f0666b86740 1 -- 172.21.15.40:0/2029647932 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "status"}) -- 0x55bd8467f080 con 0x55bd84767910 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.288+0000 7f0660a00640 1 --2- 172.21.15.40:0/2029647932 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55bd84767910 0x55bd847c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.288+0000 7f0660a00640 1 --2- 172.21.15.40:0/2029647932 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55bd84767910 0x55bd847c10e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e43d5bb4dfad9bed server_cookie=2a738fdfe807e598 in_seq=0 out_seq=0 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f063e000640 10 client.9177 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f063e000640 1 -- 172.21.15.40:0/2029647932 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x55bd8467f080 con 0x55bd84767910 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f063e000640 10 client.9177 handle_command_reply: tid=0 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f0666b86740 1 client.9177 shutdown 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f0666b86740 1 -- 172.21.15.40:0/2029647932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 msgr2=0x55bd847c3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f0666b86740 1 --2- 172.21.15.40:0/2029647932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 0x55bd847c3ed0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f065003eab0 tx=0x7f065003b000 comp rx=0 tx=0).stop 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f0666b86740 1 -- 172.21.15.40:0/2029647932 shutdown_connections 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f0666b86740 1 --2- 172.21.15.40:0/2029647932 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55bd84767910 0x55bd847c10e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f0666b86740 1 --2- 172.21.15.40:0/2029647932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd8474fe80 0x55bd847c0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f0666b86740 1 --2- 172.21.15.40:0/2029647932 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55bd8473e270 0x55bd84739270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f0666b86740 1 --2- 172.21.15.40:0/2029647932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bd847137b0 0x55bd847c3ed0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f063e000640 0 client.9177 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:19.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f0666b86740 1 -- 172.21.15.40:0/2029647932 >> 172.21.15.40:0/2029647932 conn(0x55bd846a7d10 msgr2=0x55bd8470d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:19.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f0666b86740 1 -- 172.21.15.40:0/2029647932 shutdown_connections 2024-04-26T23:09:19.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.289+0000 7f0666b86740 1 -- 172.21.15.40:0/2029647932 wait complete. 2024-04-26T23:09:19.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.290+0000 7f0666b86740 20 client.9177 trim_cache size 0 max 16384 2024-04-26T23:09:19.293 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:19.294 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "815936ab-d4ac-471e-b069-1a0dc70078ee", 2024-04-26T23:09:19.294 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:09:19.294 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8955, 2024-04-26T23:09:19.294 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:09:19.294 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:09:19.294 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:09:19.294 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 12.659076317, 2024-04-26T23:09:19.294 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 116, 2024-04-26T23:09:19.294 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 103, 2024-04-26T23:09:19.294 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 103, 2024-04-26T23:09:19.294 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 18.061819461999999 2024-04-26T23:09:19.294 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:19.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.290+0000 7f0662800640 1 -- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f064006c8f0 msgr2=0x7f064006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.290+0000 7f0662800640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f064006c8f0 0x7f064006edb0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0650034000 tx=0x7f065002e000 comp rx=0 tx=0).stop 2024-04-26T23:09:19.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.290+0000 7f0662800640 1 -- 172.21.15.40:0/315853489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c0d3820 msgr2=0x7f065c0cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.290+0000 7f0662800640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c0d3820 0x7f065c0cdde0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f065400b990 tx=0x7f065400be50 comp rx=0 tx=0).stop 2024-04-26T23:09:19.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.290+0000 7f0662800640 1 -- 172.21.15.40:0/315853489 shutdown_connections 2024-04-26T23:09:19.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.290+0000 7f0662800640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f064006c8f0 0x7f064006edb0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.290+0000 7f0662800640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f065c0d3820 0x7f065c0cdde0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.290+0000 7f0662800640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c0d08d0 0x7f065c0b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.290+0000 7f0662800640 1 --2- 172.21.15.40:0/315853489 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f065c0b6220 0x7f065c1530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.290+0000 7f0662800640 1 -- 172.21.15.40:0/315853489 >> 172.21.15.40:0/315853489 conn(0x7f065c0d4d90 msgr2=0x7f065c0b6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:19.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.291+0000 7f0662800640 1 -- 172.21.15.40:0/315853489 shutdown_connections 2024-04-26T23:09:19.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.291+0000 7f0662800640 1 -- 172.21.15.40:0/315853489 wait complete. 2024-04-26T23:09:19.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:19.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.575+0000 7fad13200640 1 Processor -- start 2024-04-26T23:09:19.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.575+0000 7fad13200640 1 -- start start 2024-04-26T23:09:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.577+0000 7fad13200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad0c15ce60 0x7fad0c0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.577+0000 7fad13200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad0c0af7a0 0x7fad0c0ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.577+0000 7fad13200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad0c0d0820 con 0x7fad0c071fe0 2024-04-26T23:09:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.577+0000 7fad13200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fad0c0cc450 con 0x7fad0c0af7a0 2024-04-26T23:09:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.577+0000 7fad13200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fad0c1614c0 con 0x7fad0c15ce60 2024-04-26T23:09:19.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.577+0000 7fad11400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad0c0af7a0 0x7fad0c0ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.577+0000 7fad11400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad0c0af7a0 0x7fad0c0ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45278/0 (socket says 172.21.15.40:45278) 2024-04-26T23:09:19.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.577+0000 7fad11400640 1 -- 172.21.15.40:0/1020663856 learned_addr learned my addr 172.21.15.40:0/1020663856 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:19.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.578+0000 7fad11e00640 1 --2- 172.21.15.40:0/1020663856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad0c15ce60 0x7fad0c0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.578+0000 7fad11400640 1 -- 172.21.15.40:0/1020663856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad0c15ce60 msgr2=0x7fad0c0afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.578+0000 7fad11400640 1 --2- 172.21.15.40:0/1020663856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad0c15ce60 0x7fad0c0afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.578+0000 7fad11400640 1 -- 172.21.15.40:0/1020663856 >> v1:172.21.15.103:6789/0 conn(0x7fad0c071fe0 legacy=0x7fad0c073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.578+0000 7fad11400640 1 -- 172.21.15.40:0/1020663856 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c161760 con 0x7fad0c0af7a0 2024-04-26T23:09:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.578+0000 7fad11e00640 1 --2- 172.21.15.40:0/1020663856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad0c15ce60 0x7fad0c0afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.579+0000 7fad11400640 1 --2- 172.21.15.40:0/1020663856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c0af7a0 0x7fad0c0ceff0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7facfc009870 tx=0x7facfc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b599b5b226fc05 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.579+0000 7fad10a00640 1 -- 172.21.15.40:0/1020663856 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc037070 con 0x7fad0c0af7a0 2024-04-26T23:09:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.580+0000 7fad10a00640 1 -- 172.21.15.40:0/1020663856 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facfc032440 con 0x7fad0c0af7a0 2024-04-26T23:09:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.580+0000 7fad10a00640 1 -- 172.21.15.40:0/1020663856 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc032810 con 0x7fad0c0af7a0 2024-04-26T23:09:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.580+0000 7fad13200640 1 -- 172.21.15.40:0/1020663856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c0af7a0 msgr2=0x7fad0c0ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.580+0000 7fad13200640 1 --2- 172.21.15.40:0/1020663856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c0af7a0 0x7fad0c0ceff0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7facfc009870 tx=0x7facfc02a180 comp rx=0 tx=0).stop 2024-04-26T23:09:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.580+0000 7fad13200640 1 -- 172.21.15.40:0/1020663856 shutdown_connections 2024-04-26T23:09:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.580+0000 7fad13200640 1 --2- 172.21.15.40:0/1020663856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c0af7a0 0x7fad0c0ceff0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.580+0000 7fad13200640 1 --2- 172.21.15.40:0/1020663856 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad0c15ce60 0x7fad0c0afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.580+0000 7fad13200640 1 -- 172.21.15.40:0/1020663856 >> 172.21.15.40:0/1020663856 conn(0x7fad0c0d0a10 msgr2=0x7fad0c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.580+0000 7fad13200640 1 -- 172.21.15.40:0/1020663856 shutdown_connections 2024-04-26T23:09:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.580+0000 7fad13200640 1 -- 172.21.15.40:0/1020663856 wait complete. 2024-04-26T23:09:19.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.582+0000 7fad13200640 1 Processor -- start 2024-04-26T23:09:19.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.582+0000 7fad13200640 1 -- start start 2024-04-26T23:09:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.583+0000 7fad13200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad0c071fe0 0x7fad0c15ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.583+0000 7fad13200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c0af7a0 0x7fad0c15b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.583+0000 7fad13200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad0c15ce60 0x7fad0c15b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.583+0000 7fad13200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c073e70 con 0x7fad0c0af7a0 2024-04-26T23:09:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.583+0000 7fad13200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c0cf530 con 0x7fad0c071fe0 2024-04-26T23:09:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.583+0000 7fad13200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fad0c073ff0 con 0x7fad0c15ce60 2024-04-26T23:09:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.584+0000 7fad11400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c0af7a0 0x7fad0c15b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.584+0000 7fad11400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c0af7a0 0x7fad0c15b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45280/0 (socket says 172.21.15.40:45280) 2024-04-26T23:09:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.584+0000 7fad11400640 1 -- 172.21.15.40:0/2072873839 learned_addr learned my addr 172.21.15.40:0/2072873839 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.584+0000 7fad11e00640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad0c071fe0 0x7fad0c15ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.584+0000 7fad12800640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad0c15ce60 0x7fad0c15b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.584+0000 7fad11400640 1 -- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad0c15ce60 msgr2=0x7fad0c15b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.584+0000 7fad11400640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad0c15ce60 0x7fad0c15b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.584+0000 7fad11400640 1 -- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad0c071fe0 msgr2=0x7fad0c15ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.584+0000 7fad11400640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad0c071fe0 0x7fad0c15ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.584+0000 7fad11400640 1 -- 172.21.15.40:0/2072873839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c15f130 con 0x7fad0c0af7a0 2024-04-26T23:09:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.584+0000 7fad11e00640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad0c071fe0 0x7fad0c15ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.584+0000 7fad12800640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad0c15ce60 0x7fad0c15b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.584+0000 7fad11400640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c0af7a0 0x7fad0c15b3a0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7facfc000c00 tx=0x7facfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.585+0000 7fad06000640 1 -- 172.21.15.40:0/2072873839 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc037040 con 0x7fad0c0af7a0 2024-04-26T23:09:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.585+0000 7fad06000640 1 -- 172.21.15.40:0/2072873839 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facfc032e50 con 0x7fad0c0af7a0 2024-04-26T23:09:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.585+0000 7fad06000640 1 -- 172.21.15.40:0/2072873839 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc0309b0 con 0x7fad0c0af7a0 2024-04-26T23:09:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.585+0000 7fad13200640 1 -- 172.21.15.40:0/2072873839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad0c15de30 con 0x7fad0c0af7a0 2024-04-26T23:09:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.585+0000 7fad13200640 1 -- 172.21.15.40:0/2072873839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad0c0c7d30 con 0x7fad0c0af7a0 2024-04-26T23:09:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.586+0000 7fad13200640 1 -- 172.21.15.40:0/2072873839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad0c0b6af0 con 0x7fad0c0af7a0 2024-04-26T23:09:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.588+0000 7fad06000640 1 -- 172.21.15.40:0/2072873839 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7facfc044020 con 0x7fad0c0af7a0 2024-04-26T23:09:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.588+0000 7fad06000640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7facf006c8f0 0x7facf006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.589+0000 7fad06000640 1 -- 172.21.15.40:0/2072873839 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7facfc0abdf0 con 0x7fad0c0af7a0 2024-04-26T23:09:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.589+0000 7fad11e00640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7facf006c8f0 0x7facf006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:19.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.591+0000 7fad11e00640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7facf006c8f0 0x7facf006edb0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fad00006fd0 tx=0x7fad00008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:19.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.593+0000 7fad06000640 1 -- 172.21.15.40:0/2072873839 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fad0c0b6af0 con 0x7fad0c0af7a0 2024-04-26T23:09:19.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.838+0000 7fad13200640 1 -- 172.21.15.40:0/2072873839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad0c069370 con 0x7fad0c0af7a0 2024-04-26T23:09:19.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.839+0000 7fad06000640 1 -- 172.21.15.40:0/2072873839 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+5230 (secure 0 0 0) 0x7facfc04b020 con 0x7fad0c0af7a0 2024-04-26T23:09:19.839 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:19.839 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":116,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:17.762683+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[6598,8949,8955]},"id":7}]} 2024-04-26T23:09:19.840 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 116 2024-04-26T23:09:19.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.842+0000 7fad13200640 1 -- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7facf006c8f0 msgr2=0x7facf006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.842+0000 7fad13200640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7facf006c8f0 0x7facf006edb0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fad00006fd0 tx=0x7fad00008040 comp rx=0 tx=0).stop 2024-04-26T23:09:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.842+0000 7fad13200640 1 -- 172.21.15.40:0/2072873839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c0af7a0 msgr2=0x7fad0c15b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.842+0000 7fad13200640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c0af7a0 0x7fad0c15b3a0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7facfc000c00 tx=0x7facfc0046c0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.842+0000 7fad13200640 1 -- 172.21.15.40:0/2072873839 shutdown_connections 2024-04-26T23:09:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.842+0000 7fad13200640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7facf006c8f0 0x7facf006edb0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.842+0000 7fad13200640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fad0c15ce60 0x7fad0c15b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.842+0000 7fad13200640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad0c0af7a0 0x7fad0c15b3a0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.842+0000 7fad13200640 1 --2- 172.21.15.40:0/2072873839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad0c071fe0 0x7fad0c15ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.842+0000 7fad13200640 1 -- 172.21.15.40:0/2072873839 >> 172.21.15.40:0/2072873839 conn(0x7fad0c0d0a10 msgr2=0x7fad0c0cb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.842+0000 7fad13200640 1 -- 172.21.15.40:0/2072873839 shutdown_connections 2024-04-26T23:09:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:19.842+0000 7fad13200640 1 -- 172.21.15.40:0/2072873839 wait complete. 2024-04-26T23:09:19.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:19.861 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p d1/d2/d3/d4/d5/d6/d7/d8) 2024-04-26T23:09:19.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:19.926 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec sync d1/d2/d3/d4/d5/d6/d7/d8) 2024-04-26T23:09:19.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:19.989 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 d1/d2) 2024-04-26T23:09:20.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:20.043 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 d1/d2/d3/d4) 2024-04-26T23:09:20.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:20.097 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 2 d1/d2/d3/d4/d5/d6) 2024-04-26T23:09:20.164 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:09:20.402 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"5","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:09:20.402 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:09:20.402 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:09:22.404 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:09:22.593 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-04-26T23:09:22.592+0000 7f928e600640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T23:09:22.645 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"7","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:09:22.645 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:09:22.646 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:09:24.646 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:09:24.890 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"7","version":"7","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-04-26T23:09:24.891 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:09:24.891 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:09:26.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-04-26T23:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"13","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/d1/d2","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":0,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1,0","states":["auth","dirty","complete","frozentree"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[1,0],"replica_nonce":0},"auth_pins":0,"is_frozen":true,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":1,"subtree":1,"replicated":1,"dirty":1,"waiter":0,"authpin":0,"tempexporting":1},"nref":6}}] 2024-04-26T23:09:27.140 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000001", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": 0, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1,0", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": true, "is_rep": false, "nref": 6, "path": "/d1/d2/d3/d4", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 1, "ptrwaiter": 0, "replicated": 1, "request": 0, "subtree": 1, "tempexporting": 1, "waiter": 0 }, "projected_version": "14", "replica_state": { "authority": [ 1, 0 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete", "frozentree" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:09:27.140 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] =?= [('/d1/d2', 0), ('/d1/d2/d3/d4', 1)] 2024-04-26T23:09:27.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:09:27.361 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:09:27.361 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:09:27.362 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:09:29.362 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:09:29.600 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:09:29.601 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:09:29.601 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:09:31.603 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:09:31.843 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:09:31.844 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:09:31.844 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:09:33.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-04-26T23:09:34.085 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"14","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1,"2":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":2,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/d1/d2/d3/d4/d5/d6","dirfrag":"0x10000000005","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-04-26T23:09:34.086 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1, "2": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "14", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "2", "dirfrag": "0x10000000005", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/d1/d2/d3/d4/d5/d6", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "6", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 2, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false } ] 2024-04-26T23:09:34.086 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:09:34.086 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-04-26T23:09:34.314 INFO:teuthology.orchestra.run.smithi103.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"13","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1/d2/d3/d4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"0","version":"14","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dnwaiter":0,"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":2,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/d1/d2/d3/d4/d5/d6","dirfrag":"0x10000000005","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-04-26T23:09:34.314 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000003", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/d1/d2/d3/d4", "pins": { "child": 1, "dnwaiter": 0, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "2", "dirfrag": "0x10000000005", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d1/d2/d3/d4/d5/d6", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 2, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false } ] 2024-04-26T23:09:34.314 INFO:tasks.cephfs.cephfs_test_case:[('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] =?= [('/d1/d2/d3/d4', 1), ('/d1/d2/d3/d4/d5/d6', 2)] 2024-04-26T23:09:34.315 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 flush journal 2024-04-26T23:09:34.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.542+0000 7fc516a00640 1 Processor -- start 2024-04-26T23:09:34.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.542+0000 7fc516a00640 1 -- start start 2024-04-26T23:09:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.544+0000 7fc516a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5100d1770 0x7fc51015ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.544+0000 7fc516a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc5100d1b50 0x7fc51015f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.544+0000 7fc516a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5100d5d30 con 0x7fc5100b6470 2024-04-26T23:09:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.544+0000 7fc516a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc5100d3760 con 0x7fc5100d1b50 2024-04-26T23:09:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.544+0000 7fc516a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc5100d09f0 con 0x7fc5100d1770 2024-04-26T23:09:34.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.545+0000 7fc516000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc5100d1b50 0x7fc51015f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.545+0000 7fc514c00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc5100b6470 0x7fc5100b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:59648/0 (socket says 172.21.15.40:59648) 2024-04-26T23:09:34.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.545+0000 7fc514c00640 1 -- 172.21.15.40:0/3900063095 learned_addr learned my addr 172.21.15.40:0/3900063095 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:34.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.545+0000 7fc515600640 1 --2- 172.21.15.40:0/3900063095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5100d1770 0x7fc51015ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.546+0000 7fc516000640 1 -- 172.21.15.40:0/3900063095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5100d1770 msgr2=0x7fc51015ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.546+0000 7fc516000640 1 --2- 172.21.15.40:0/3900063095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5100d1770 0x7fc51015ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.546+0000 7fc516000640 1 -- 172.21.15.40:0/3900063095 >> v1:172.21.15.103:6789/0 conn(0x7fc5100b6470 legacy=0x7fc5100b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.546+0000 7fc516000640 1 -- 172.21.15.40:0/3900063095 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5100d5a70 con 0x7fc5100d1b50 2024-04-26T23:09:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.547+0000 7fc516000640 1 --2- 172.21.15.40:0/3900063095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5100d1b50 0x7fc51015f4b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc508009d90 tx=0x7fc50802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ff270a3abba98f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.549+0000 7fc50fe00640 1 -- 172.21.15.40:0/3900063095 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508007740 con 0x7fc5100d1b50 2024-04-26T23:09:34.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.549+0000 7fc50fe00640 1 -- 172.21.15.40:0/3900063095 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5080078e0 con 0x7fc5100d1b50 2024-04-26T23:09:34.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.549+0000 7fc50fe00640 1 -- 172.21.15.40:0/3900063095 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508007cb0 con 0x7fc5100d1b50 2024-04-26T23:09:34.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.549+0000 7fc516a00640 1 -- 172.21.15.40:0/3900063095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5100d1b50 msgr2=0x7fc51015f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.549+0000 7fc516a00640 1 --2- 172.21.15.40:0/3900063095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5100d1b50 0x7fc51015f4b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc508009d90 tx=0x7fc50802a660 comp rx=0 tx=0).stop 2024-04-26T23:09:34.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.550+0000 7fc516a00640 1 -- 172.21.15.40:0/3900063095 shutdown_connections 2024-04-26T23:09:34.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.550+0000 7fc516a00640 1 --2- 172.21.15.40:0/3900063095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5100d1b50 0x7fc51015f4b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.550+0000 7fc516a00640 1 --2- 172.21.15.40:0/3900063095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5100d1770 0x7fc51015ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.550+0000 7fc516a00640 1 -- 172.21.15.40:0/3900063095 >> 172.21.15.40:0/3900063095 conn(0x7fc5100d4dd0 msgr2=0x7fc5100d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:34.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.550+0000 7fc516a00640 1 -- 172.21.15.40:0/3900063095 shutdown_connections 2024-04-26T23:09:34.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.550+0000 7fc516a00640 1 -- 172.21.15.40:0/3900063095 wait complete. 2024-04-26T23:09:34.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.552+0000 7fc516a00640 1 Processor -- start 2024-04-26T23:09:34.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.552+0000 7fc516a00640 1 -- start start 2024-04-26T23:09:34.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.553+0000 7fc516a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5100b6470 0x7fc5100b7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.553+0000 7fc516a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5100d1770 0x7fc5100b8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.553+0000 7fc516a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5100d1b50 0x7fc5100cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.553+0000 7fc516a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc5100d3bf0 con 0x7fc5100d1770 2024-04-26T23:09:34.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.553+0000 7fc516a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc5100d2670 con 0x7fc5100d1b50 2024-04-26T23:09:34.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.553+0000 7fc516a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc5100d3d70 con 0x7fc5100b6470 2024-04-26T23:09:34.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.553+0000 7fc514c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5100d1770 0x7fc5100b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.553+0000 7fc514c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5100d1770 0x7fc5100b8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47406/0 (socket says 172.21.15.40:47406) 2024-04-26T23:09:34.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.553+0000 7fc514c00640 1 -- 172.21.15.40:0/4176429525 learned_addr learned my addr 172.21.15.40:0/4176429525 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:34.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.553+0000 7fc515600640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5100b6470 0x7fc5100b7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.553+0000 7fc516000640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5100d1b50 0x7fc5100cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.554+0000 7fc514c00640 1 -- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5100b6470 msgr2=0x7fc5100b7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.554+0000 7fc514c00640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5100b6470 0x7fc5100b7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.554+0000 7fc514c00640 1 -- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5100d1b50 msgr2=0x7fc5100cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.554+0000 7fc514c00640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5100d1b50 0x7fc5100cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.554+0000 7fc514c00640 1 -- 172.21.15.40:0/4176429525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5100b8ca0 con 0x7fc5100d1770 2024-04-26T23:09:34.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.554+0000 7fc514c00640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5100d1770 0x7fc5100b8760 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00c440 tx=0x7fc4fc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.554+0000 7fc50d600640 1 -- 172.21.15.40:0/4176429525 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc017070 con 0x7fc5100d1770 2024-04-26T23:09:34.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.554+0000 7fc50d600640 1 -- 172.21.15.40:0/4176429525 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4fc005ce0 con 0x7fc5100d1770 2024-04-26T23:09:34.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.554+0000 7fc50d600640 1 -- 172.21.15.40:0/4176429525 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc0072f0 con 0x7fc5100d1770 2024-04-26T23:09:34.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.554+0000 7fc516a00640 1 -- 172.21.15.40:0/4176429525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5101f8670 con 0x7fc5100d1770 2024-04-26T23:09:34.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.555+0000 7fc516a00640 1 -- 172.21.15.40:0/4176429525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5101f8c30 con 0x7fc5100d1770 2024-04-26T23:09:34.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.557+0000 7fc50d600640 1 -- 172.21.15.40:0/4176429525 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc4fc01a070 con 0x7fc5100d1770 2024-04-26T23:09:34.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.558+0000 7fc50d600640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc4f006c8f0 0x7fc4f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.558+0000 7fc50d600640 1 -- 172.21.15.40:0/4176429525 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fc4fc08afe0 con 0x7fc5100d1770 2024-04-26T23:09:34.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.558+0000 7fc51ae1b740 1 Processor -- start 2024-04-26T23:09:34.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.558+0000 7fc515600640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc4f006c8f0 0x7fc4f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.558+0000 7fc51ae1b740 1 -- start start 2024-04-26T23:09:34.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.559+0000 7fc51ae1b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ad710 0x55df371adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.559+0000 7fc51ae1b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371cafb0 0x55df371cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.559+0000 7fc51ae1b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371833f0 0x55df371c9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.559+0000 7fc51ae1b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55df37093030 con 0x55df371833f0 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.559+0000 7fc51ae1b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55df370ae7f0 con 0x55df371cafb0 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.559+0000 7fc51ae1b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55df370697e0 con 0x55df371ad710 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.560+0000 7fc515600640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc4f006c8f0 0x7fc4f006edb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fc5100cebc0 tx=0x7fc504008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.560+0000 7fc516000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371833f0 0x55df371c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.560+0000 7fc516000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371833f0 0x55df371c9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47418/0 (socket says 172.21.15.40:47418) 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.560+0000 7fc516000640 1 -- 172.21.15.40:0/3122917508 learned_addr learned my addr 172.21.15.40:0/3122917508 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.560+0000 7fc515600640 1 --2- 172.21.15.40:0/3122917508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ad710 0x55df371adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.560+0000 7fc514c00640 1 --2- 172.21.15.40:0/3122917508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371cafb0 0x55df371cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.560+0000 7fc516000640 1 -- 172.21.15.40:0/3122917508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ad710 msgr2=0x55df371adaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.560+0000 7fc516000640 1 --2- 172.21.15.40:0/3122917508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ad710 0x55df371adaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.560+0000 7fc516000640 1 -- 172.21.15.40:0/3122917508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371cafb0 msgr2=0x55df371cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.560+0000 7fc516000640 1 --2- 172.21.15.40:0/3122917508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371cafb0 0x55df371cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.560+0000 7fc516000640 1 -- 172.21.15.40:0/3122917508 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df37075050 con 0x55df371833f0 2024-04-26T23:09:34.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.560+0000 7fc515600640 1 --2- 172.21.15.40:0/3122917508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ad710 0x55df371adaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:09:34.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.561+0000 7fc516000640 1 --2- 172.21.15.40:0/3122917508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371833f0 0x55df371c9670 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fc508009eb0 tx=0x7fc50802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c4f67e6314be94 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.561+0000 7fc501c00640 1 -- 172.21.15.40:0/3122917508 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508007840 con 0x55df371833f0 2024-04-26T23:09:34.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.561+0000 7fc501c00640 1 -- 172.21.15.40:0/3122917508 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5080079e0 con 0x55df371833f0 2024-04-26T23:09:34.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.561+0000 7fc501c00640 1 -- 172.21.15.40:0/3122917508 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508007d90 con 0x55df371833f0 2024-04-26T23:09:34.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.562+0000 7fc51ae1b740 1 -- 172.21.15.40:0/3122917508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371833f0 msgr2=0x55df371c9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.562+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/3122917508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371833f0 0x55df371c9670 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fc508009eb0 tx=0x7fc50802e000 comp rx=0 tx=0).stop 2024-04-26T23:09:34.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.562+0000 7fc51ae1b740 1 -- 172.21.15.40:0/3122917508 shutdown_connections 2024-04-26T23:09:34.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.562+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/3122917508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371833f0 0x55df371c9670 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.562+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/3122917508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371cafb0 0x55df371cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.562+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/3122917508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ad710 0x55df371adaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.562+0000 7fc51ae1b740 1 -- 172.21.15.40:0/3122917508 >> 172.21.15.40:0/3122917508 conn(0x55df3717fd20 msgr2=0x55df371c8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:34.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.562+0000 7fc51ae1b740 1 -- 172.21.15.40:0/3122917508 shutdown_connections 2024-04-26T23:09:34.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.562+0000 7fc51ae1b740 1 -- 172.21.15.40:0/3122917508 wait complete. 2024-04-26T23:09:34.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.563+0000 7fc51ae1b740 1 -- start start 2024-04-26T23:09:34.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.563+0000 7fc51ae1b740 1 Processor -- start 2024-04-26T23:09:34.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc51ae1b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371ad710 0x55df371a8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc51ae1b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371cafb0 0x55df371a93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc51ae1b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ac790 0x55df371acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc51ae1b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55df370eb200 con 0x55df371ac790 2024-04-26T23:09:34.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc51ae1b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55df370e5ee0 con 0x55df371ad710 2024-04-26T23:09:34.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc51ae1b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55df370f2e70 con 0x55df371cafb0 2024-04-26T23:09:34.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc51ae1b740 10 client.0 _ll_register_callbacks cb 0x55df370dc270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:34.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc516000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ac790 0x55df371acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc516000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ac790 0x55df371acbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47434/0 (socket says 172.21.15.40:47434) 2024-04-26T23:09:34.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc516000640 1 -- 172.21.15.40:0/2812429659 learned_addr learned my addr 172.21.15.40:0/2812429659 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:34.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc514c00640 1 --2- 172.21.15.40:0/2812429659 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371cafb0 0x55df371a93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc516000640 1 -- 172.21.15.40:0/2812429659 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371cafb0 msgr2=0x55df371a93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc515600640 1 --2- 172.21.15.40:0/2812429659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371ad710 0x55df371a8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.564+0000 7fc516000640 1 --2- 172.21.15.40:0/2812429659 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371cafb0 0x55df371a93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.565+0000 7fc516000640 1 -- 172.21.15.40:0/2812429659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371ad710 msgr2=0x55df371a8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.565+0000 7fc516000640 1 --2- 172.21.15.40:0/2812429659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371ad710 0x55df371a8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.565+0000 7fc516000640 1 -- 172.21.15.40:0/2812429659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df36ec27d0 con 0x55df371ac790 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.565+0000 7fc516000640 1 --2- 172.21.15.40:0/2812429659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ac790 0x55df371acbf0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fc50802a6a0 tx=0x7fc508004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f443f7f568243 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.565+0000 7fc4f6000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.565+0000 7fc4f6000640 1 -- 172.21.15.40:0/2812429659 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508054060 con 0x55df371ac790 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.565+0000 7fc4f6000640 1 -- 172.21.15.40:0/2812429659 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fc508007da0 con 0x55df371ac790 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.565+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2812429659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5080079e0 con 0x55df371ac790 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.566+0000 7fc4f6000640 1 -- 172.21.15.40:0/2812429659 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7fc50805a050 con 0x55df371ac790 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.566+0000 7fc51ae1b740 10 client.9192 fetch_fsmap learned FSMap version 118 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.566+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2812429659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) -- 0x55df36df83f0 con 0x55df371ac790 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.566+0000 7fc4f6000640 1 -- 172.21.15.40:0/2812429659 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5080079e0 con 0x55df371ac790 2024-04-26T23:09:34.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.566+0000 7fc4f6000640 1 -- 172.21.15.40:0/2812429659 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 118) ==== 3546+0+0 (secure 0 0 0) 0x7fc50805d020 con 0x55df371ac790 2024-04-26T23:09:34.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.566+0000 7fc51ae1b740 10 client.9192 fetch_fsmap finished waiting for FSMap version 118 2024-04-26T23:09:34.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.566+0000 7fc51ae1b740 10 client.9192 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:34.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.566+0000 7fc51ae1b740 20 client.9192 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:34.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.566+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/2812429659 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55df371cbf70 0x55df371cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.566+0000 7fc51ae1b740 4 client.9192 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:34.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.566+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2812429659 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df370f47e0 con 0x55df371cbf70 2024-04-26T23:09:34.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.566+0000 7fc515600640 1 --2- 172.21.15.40:0/2812429659 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55df371cbf70 0x55df371cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.567+0000 7fc515600640 1 --2- 172.21.15.40:0/2812429659 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55df371cbf70 0x55df371cc3e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca48d4ac3de4ddb3 server_cookie=3a8ab82b68583e6c in_seq=0 out_seq=0 2024-04-26T23:09:34.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.567+0000 7fc4f6000640 10 client.9192 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:34.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc4f6000640 1 -- 172.21.15.40:0/2812429659 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55df370f47e0 con 0x55df371cbf70 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc4f6000640 10 client.9192 handle_command_reply: tid=0 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc51ae1b740 1 client.9192 shutdown 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2812429659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ac790 msgr2=0x55df371acbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/2812429659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ac790 0x55df371acbf0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fc50802a6a0 tx=0x7fc508004ab0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2812429659 shutdown_connections 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/2812429659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ac790 0x55df371acbf0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/2812429659 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371cafb0 0x55df371a93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/2812429659 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55df371cbf70 0x55df371cc3e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/2812429659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371ad710 0x55df371a8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2812429659 >> 172.21.15.40:0/2812429659 conn(0x55df371833f0 msgr2=0x55df371828e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc4f6000640 0 client.9192 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2812429659 shutdown_connections 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2812429659 wait complete. 2024-04-26T23:09:34.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.568+0000 7fc51ae1b740 20 client.9192 trim_cache size 0 max 16384 2024-04-26T23:09:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.585+0000 7fc51ae1b740 1 Processor -- start 2024-04-26T23:09:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.586+0000 7fc51ae1b740 1 -- start start 2024-04-26T23:09:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.586+0000 7fc51ae1b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ac790 0x55df371c7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.586+0000 7fc51ae1b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ad710 0x55df371bfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.586+0000 7fc51ae1b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371cafb0 0x55df3717c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.586+0000 7fc51ae1b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc510201db0 con 0x55df371ad710 2024-04-26T23:09:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.586+0000 7fc51ae1b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc510200fd0 con 0x55df371cafb0 2024-04-26T23:09:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.586+0000 7fc51ae1b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc510201680 con 0x55df371ac790 2024-04-26T23:09:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.587+0000 7fc514c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ad710 0x55df371bfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.587+0000 7fc514c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ad710 0x55df371bfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47450/0 (socket says 172.21.15.40:47450) 2024-04-26T23:09:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.587+0000 7fc514c00640 1 -- 172.21.15.40:0/3250875242 learned_addr learned my addr 172.21.15.40:0/3250875242 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.587+0000 7fc515600640 1 --2- 172.21.15.40:0/3250875242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ac790 0x55df371c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.587+0000 7fc516000640 1 --2- 172.21.15.40:0/3250875242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371cafb0 0x55df3717c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.587+0000 7fc514c00640 1 -- 172.21.15.40:0/3250875242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ac790 msgr2=0x55df371c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.587+0000 7fc514c00640 1 --2- 172.21.15.40:0/3250875242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ac790 0x55df371c7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.587+0000 7fc514c00640 1 -- 172.21.15.40:0/3250875242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371cafb0 msgr2=0x55df3717c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.587+0000 7fc514c00640 1 --2- 172.21.15.40:0/3250875242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371cafb0 0x55df3717c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.587+0000 7fc514c00640 1 -- 172.21.15.40:0/3250875242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df36db80f0 con 0x55df371ad710 2024-04-26T23:09:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.587+0000 7fc516000640 1 --2- 172.21.15.40:0/3250875242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371cafb0 0x55df3717c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.587+0000 7fc514c00640 1 --2- 172.21.15.40:0/3250875242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ad710 0x55df371bfad0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc061260 tx=0x7fc4fc058640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b16863abb4165c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.588+0000 7fc501200640 1 -- 172.21.15.40:0/3250875242 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc071020 con 0x55df371ad710 2024-04-26T23:09:34.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.588+0000 7fc501200640 1 -- 172.21.15.40:0/3250875242 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4fc061790 con 0x55df371ad710 2024-04-26T23:09:34.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.588+0000 7fc501200640 1 -- 172.21.15.40:0/3250875242 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc055220 con 0x55df371ad710 2024-04-26T23:09:34.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.588+0000 7fc51ae1b740 1 -- 172.21.15.40:0/3250875242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ad710 msgr2=0x55df371bfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.588+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/3250875242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ad710 0x55df371bfad0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc061260 tx=0x7fc4fc058640 comp rx=0 tx=0).stop 2024-04-26T23:09:34.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.588+0000 7fc51ae1b740 1 -- 172.21.15.40:0/3250875242 shutdown_connections 2024-04-26T23:09:34.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.588+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/3250875242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371cafb0 0x55df3717c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.588+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/3250875242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371ad710 0x55df371bfad0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.588+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/3250875242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ac790 0x55df371c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.588+0000 7fc51ae1b740 1 -- 172.21.15.40:0/3250875242 >> 172.21.15.40:0/3250875242 conn(0x55df371833f0 msgr2=0x55df371acb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:34.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.588+0000 7fc51ae1b740 1 -- 172.21.15.40:0/3250875242 shutdown_connections 2024-04-26T23:09:34.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.589+0000 7fc51ae1b740 1 -- 172.21.15.40:0/3250875242 wait complete. 2024-04-26T23:09:34.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.589+0000 7fc51ae1b740 1 -- start start 2024-04-26T23:09:34.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.589+0000 7fc51ae1b740 1 Processor -- start 2024-04-26T23:09:34.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.590+0000 7fc51ae1b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ac790 0x55df37241200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.590+0000 7fc51ae1b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371ad710 0x55df3717c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.590+0000 7fc51ae1b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371cafb0 0x55df3717cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.590+0000 7fc51ae1b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55df370f3fd0 con 0x55df371cafb0 2024-04-26T23:09:34.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.590+0000 7fc51ae1b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc5101f5860 con 0x55df371ad710 2024-04-26T23:09:34.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.590+0000 7fc51ae1b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55df37242eb0 con 0x55df371ac790 2024-04-26T23:09:34.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.590+0000 7fc51ae1b740 10 client.0 _ll_register_callbacks cb 0x7fc508009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:34.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc516000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371cafb0 0x55df3717cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc516000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371cafb0 0x55df3717cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47466/0 (socket says 172.21.15.40:47466) 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc516000640 1 -- 172.21.15.40:0/2964495196 learned_addr learned my addr 172.21.15.40:0/2964495196 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc514c00640 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371ad710 0x55df3717c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc515600640 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ac790 0x55df37241200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc516000640 1 -- 172.21.15.40:0/2964495196 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ac790 msgr2=0x55df37241200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc516000640 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ac790 0x55df37241200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc516000640 1 -- 172.21.15.40:0/2964495196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371ad710 msgr2=0x55df3717c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc516000640 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371ad710 0x55df3717c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc516000640 1 -- 172.21.15.40:0/2964495196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df370aea30 con 0x55df371cafb0 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc514c00640 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371ad710 0x55df3717c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc515600640 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ac790 0x55df37241200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc516000640 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371cafb0 0x55df3717cea0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fc50802a6a0 tx=0x7fc50802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4333908ecfe20038 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.591+0000 7fc4f6000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.592+0000 7fc4f6000640 1 -- 172.21.15.40:0/2964495196 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50804f480 con 0x55df371cafb0 2024-04-26T23:09:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.592+0000 7fc4f6000640 1 -- 172.21.15.40:0/2964495196 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fc5080549b0 con 0x55df371cafb0 2024-04-26T23:09:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.592+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2964495196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4fc061790 con 0x55df371cafb0 2024-04-26T23:09:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.592+0000 7fc4f6000640 1 -- 172.21.15.40:0/2964495196 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7fc508054bb0 con 0x55df371cafb0 2024-04-26T23:09:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.592+0000 7fc51ae1b740 10 client.9198 fetch_fsmap learned FSMap version 118 2024-04-26T23:09:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.592+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2964495196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) -- 0x55df37242bf0 con 0x55df371cafb0 2024-04-26T23:09:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.592+0000 7fc4f6000640 1 -- 172.21.15.40:0/2964495196 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4fc061790 con 0x55df371cafb0 2024-04-26T23:09:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.592+0000 7fc4f6000640 1 -- 172.21.15.40:0/2964495196 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 118) ==== 3546+0+0 (secure 0 0 0) 0x7fc50805d070 con 0x55df371cafb0 2024-04-26T23:09:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.592+0000 7fc51ae1b740 10 client.9198 fetch_fsmap finished waiting for FSMap version 118 2024-04-26T23:09:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.592+0000 7fc51ae1b740 10 client.9198 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.592+0000 7fc51ae1b740 20 client.9198 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.593+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55df371c09b0 0x55df371ea970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.593+0000 7fc51ae1b740 4 client.9198 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:09:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.593+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2964495196 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55df370ee650 con 0x55df371c09b0 2024-04-26T23:09:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.593+0000 7fc515600640 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55df371c09b0 0x55df371ea970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.593+0000 7fc515600640 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55df371c09b0 0x55df371ea970 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa031ad2bba233ed server_cookie=575f8f3827eab00e in_seq=0 out_seq=0 2024-04-26T23:09:34.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.593+0000 7fc4f6000640 10 client.9198 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:34.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc4f6000640 1 -- 172.21.15.40:0/2964495196 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55df370ee650 con 0x55df371c09b0 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc4f6000640 10 client.9198 handle_command_reply: tid=0 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc51ae1b740 1 client.9198 shutdown 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2964495196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371cafb0 msgr2=0x55df3717cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371cafb0 0x55df3717cea0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fc50802a6a0 tx=0x7fc50802f000 comp rx=0 tx=0).stop 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2964495196 shutdown_connections 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df371cafb0 0x55df3717cea0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55df371c09b0 0x55df371ea970 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55df371ad710 0x55df3717c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc51ae1b740 1 --2- 172.21.15.40:0/2964495196 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55df371ac790 0x55df37241200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2964495196 >> 172.21.15.40:0/2964495196 conn(0x55df371833f0 msgr2=0x55df371ebb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc4f6000640 0 client.9198 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2964495196 shutdown_connections 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.619+0000 7fc51ae1b740 1 -- 172.21.15.40:0/2964495196 wait complete. 2024-04-26T23:09:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.620+0000 7fc51ae1b740 20 client.9198 trim_cache size 0 max 16384 2024-04-26T23:09:34.621 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:34.621 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:09:34.621 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:09:34.621 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.620+0000 7fc516a00640 1 -- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc4f006c8f0 msgr2=0x7fc4f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.620+0000 7fc516a00640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc4f006c8f0 0x7fc4f006edb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fc5100cebc0 tx=0x7fc504008040 comp rx=0 tx=0).stop 2024-04-26T23:09:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.620+0000 7fc516a00640 1 -- 172.21.15.40:0/4176429525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5100d1770 msgr2=0x7fc5100b8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.620+0000 7fc516a00640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5100d1770 0x7fc5100b8760 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00c440 tx=0x7fc4fc00c900 comp rx=0 tx=0).stop 2024-04-26T23:09:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.621+0000 7fc516a00640 1 -- 172.21.15.40:0/4176429525 shutdown_connections 2024-04-26T23:09:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.621+0000 7fc516a00640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc4f006c8f0 0x7fc4f006edb0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.621+0000 7fc516a00640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5100d1b50 0x7fc5100cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.621+0000 7fc516a00640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5100d1770 0x7fc5100b8760 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.621+0000 7fc516a00640 1 --2- 172.21.15.40:0/4176429525 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc5100b6470 0x7fc5100b7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.621+0000 7fc516a00640 1 -- 172.21.15.40:0/4176429525 >> 172.21.15.40:0/4176429525 conn(0x7fc5100d4dd0 msgr2=0x7fc5100d51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:34.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.621+0000 7fc516a00640 1 -- 172.21.15.40:0/4176429525 shutdown_connections 2024-04-26T23:09:34.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.621+0000 7fc516a00640 1 -- 172.21.15.40:0/4176429525 wait complete. 2024-04-26T23:09:34.638 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:1 flush journal 2024-04-26T23:09:34.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.829+0000 7f4ed7e00640 1 Processor -- start 2024-04-26T23:09:34.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.829+0000 7f4ed7e00640 1 -- start start 2024-04-26T23:09:34.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.830+0000 7f4ed7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4ed80d1770 0x7f4ed815ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.830+0000 7f4ed7e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ed80d5d30 con 0x7f4ed80b6470 2024-04-26T23:09:34.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.830+0000 7f4ed7e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ed80b5550 con 0x7f4ed80d1b50 2024-04-26T23:09:34.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.830+0000 7f4ed7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed80d3760 con 0x7f4ed80d1770 2024-04-26T23:09:34.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.831+0000 7f4ed6000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4ed80b6470 0x7f4ed80b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45568/0 (socket says 172.21.15.40:45568) 2024-04-26T23:09:34.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.831+0000 7f4ed6000640 1 -- 172.21.15.40:0/100002193 learned_addr learned my addr 172.21.15.40:0/100002193 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:34.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.831+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2022581461 0 0) 0x7f4ed80d5d30 con 0x7f4ed80b6470 2024-04-26T23:09:34.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.831+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ebc003520 con 0x7f4ed80b6470 2024-04-26T23:09:34.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.831+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/100002193 >> v2:172.21.15.103:3300/0 conn(0x7f4ed80d1770 0x7f4ed815ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3970353692 0 0) 0x7f4ebc003520 con 0x7f4ed80b6470 2024-04-26T23:09:34.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 >> v2:172.21.15.103:3300/0 conn(0x7f4ed80d1770 msgr2=0x7f4ed815ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed5600640 1 --2- 172.21.15.40:0/100002193 >> v2:172.21.15.103:3300/0 conn(0x7f4ed80d1770 0x7f4ed815ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 >> v1:172.21.15.103:6789/0 conn(0x7f4ed80d1b50 legacy=0x7f4ed80b4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed80d09f0 con 0x7f4ed80b6470 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f4ec0001670 con 0x7f4ed80b6470 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 >> v1:172.21.15.40:6789/0 conn(0x7f4ed80b6470 legacy=0x7f4ed80b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed5600640 1 --2- 172.21.15.40:0/100002193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ebc003a00 0x7f4ebc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed5600640 1 --2- 172.21.15.40:0/100002193 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ebc004800 0x7f4ebc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed5600640 1 --2- 172.21.15.40:0/100002193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ebc005290 0x7f4ebc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed80d3760 con 0x7f4ebc005290 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc00c1e0 con 0x7f4ebc003a00 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ebc00c3e0 con 0x7f4ebc004800 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed7400640 1 --2- 172.21.15.40:0/100002193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ebc005290 0x7f4ebc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.832+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/100002193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ebc003a00 0x7f4ebc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.833+0000 7f4ed6000640 1 --2- 172.21.15.40:0/100002193 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ebc004800 0x7f4ebc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.833+0000 7f4ed7400640 1 -- 172.21.15.40:0/100002193 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ebc004800 msgr2=0x7f4ebc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.833+0000 7f4ed7400640 1 --2- 172.21.15.40:0/100002193 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ebc004800 0x7f4ebc004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.833+0000 7f4ed7400640 1 -- 172.21.15.40:0/100002193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ebc003a00 msgr2=0x7f4ebc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.833+0000 7f4ed7400640 1 --2- 172.21.15.40:0/100002193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ebc003a00 0x7f4ebc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.833+0000 7f4ed7400640 1 -- 172.21.15.40:0/100002193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ebc003520 con 0x7f4ebc005290 2024-04-26T23:09:34.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.834+0000 7f4ed7400640 1 -- 172.21.15.40:0/100002193 reap_dead start 2024-04-26T23:09:34.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.834+0000 7f4ed7400640 1 --2- 172.21.15.40:0/100002193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ebc005290 0x7f4ebc00b9c0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0003db0 tx=0x7f4ebc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5366d7f53c3e699d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.834+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0037070 con 0x7f4ebc005290 2024-04-26T23:09:34.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.835+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ed002aa00 con 0x7f4ebc005290 2024-04-26T23:09:34.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.835+0000 7f4ed5600640 1 -- 172.21.15.40:0/100002193 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0003980 con 0x7f4ebc005290 2024-04-26T23:09:34.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.835+0000 7f4ed7e00640 1 -- 172.21.15.40:0/100002193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ebc005290 msgr2=0x7f4ebc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.835+0000 7f4ed7e00640 1 --2- 172.21.15.40:0/100002193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ebc005290 0x7f4ebc00b9c0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0003db0 tx=0x7f4ebc005830 comp rx=0 tx=0).stop 2024-04-26T23:09:34.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.835+0000 7f4ed7e00640 1 -- 172.21.15.40:0/100002193 shutdown_connections 2024-04-26T23:09:34.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.835+0000 7f4ed7e00640 1 --2- 172.21.15.40:0/100002193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ebc005290 0x7f4ebc00b9c0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.835+0000 7f4ed7e00640 1 -- 172.21.15.40:0/100002193 >> 172.21.15.40:0/100002193 conn(0x7f4ed80d4dd0 msgr2=0x7f4ed80d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:34.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.835+0000 7f4ed7e00640 1 -- 172.21.15.40:0/100002193 shutdown_connections 2024-04-26T23:09:34.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.835+0000 7f4ed7e00640 1 -- 172.21.15.40:0/100002193 wait complete. 2024-04-26T23:09:34.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.837+0000 7f4ed7e00640 1 Processor -- start 2024-04-26T23:09:34.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.837+0000 7f4ed7e00640 1 -- start start 2024-04-26T23:09:34.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.838+0000 7f4ed7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ebc005290 0x7f4ed80b9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.838+0000 7f4ed7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed81593f0 0x7f4ed80b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.838+0000 7f4ed7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed80b7f10 0x7f4ed8150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.838+0000 7f4ed7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed80d2670 con 0x7f4ed80b7f10 2024-04-26T23:09:34.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.838+0000 7f4ed7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed80b5b00 con 0x7f4ed81593f0 2024-04-26T23:09:34.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.838+0000 7f4ed7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed80d3bf0 con 0x7f4ebc005290 2024-04-26T23:09:34.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.839+0000 7f4ed7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed80b7f10 0x7f4ed8150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.839+0000 7f4ed7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed80b7f10 0x7f4ed8150690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47488/0 (socket says 172.21.15.40:47488) 2024-04-26T23:09:34.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.839+0000 7f4ed7400640 1 -- 172.21.15.40:0/3083641308 learned_addr learned my addr 172.21.15.40:0/3083641308 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:34.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.839+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ebc005290 0x7f4ed80b9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.839+0000 7f4ed7400640 1 -- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ebc005290 msgr2=0x7f4ed80b9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.839+0000 7f4ed6000640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed81593f0 0x7f4ed80b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.839+0000 7f4ed7400640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ebc005290 0x7f4ed80b9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.839+0000 7f4ed7400640 1 -- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed81593f0 msgr2=0x7f4ed80b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.839+0000 7f4ed7400640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed81593f0 0x7f4ed80b79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.839+0000 7f4ed7400640 1 -- 172.21.15.40:0/3083641308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed80b85e0 con 0x7f4ed80b7f10 2024-04-26T23:09:34.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.839+0000 7f4ed6000640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed81593f0 0x7f4ed80b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:34.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.839+0000 7f4ed7400640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed80b7f10 0x7f4ed8150690 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0003ea0 tx=0x7f4ed00036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.840+0000 7f4ecf400640 1 -- 172.21.15.40:0/3083641308 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0042070 con 0x7f4ed80b7f10 2024-04-26T23:09:34.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.840+0000 7f4ecf400640 1 -- 172.21.15.40:0/3083641308 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ed0032a70 con 0x7f4ed80b7f10 2024-04-26T23:09:34.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.840+0000 7f4ecf400640 1 -- 172.21.15.40:0/3083641308 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0037040 con 0x7f4ed80b7f10 2024-04-26T23:09:34.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.840+0000 7f4ed7e00640 1 -- 172.21.15.40:0/3083641308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed80b88c0 con 0x7f4ed80b7f10 2024-04-26T23:09:34.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.840+0000 7f4ed7e00640 1 -- 172.21.15.40:0/3083641308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed80b9d50 con 0x7f4ed80b7f10 2024-04-26T23:09:34.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.842+0000 7f4ecf400640 1 -- 172.21.15.40:0/3083641308 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4ed0045070 con 0x7f4ed80b7f10 2024-04-26T23:09:34.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.843+0000 7f4ecf400640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4eb406c8f0 0x7f4eb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.843+0000 7f4ecf400640 1 -- 172.21.15.40:0/3083641308 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f4ed00aabf0 con 0x7f4ed80b7f10 2024-04-26T23:09:34.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.844+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4eb406c8f0 0x7f4eb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.844+0000 7f4ee0b95740 1 Processor -- start 2024-04-26T23:09:34.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.844+0000 7f4ee0b95740 1 -- start start 2024-04-26T23:09:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.845+0000 7f4ee0b95740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8af710 0x560d9a8afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.845+0000 7f4ee0b95740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8ccfb0 0x560d9a8cd500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.845+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4eb406c8f0 0x7f4eb406edb0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4ec80041e0 tx=0x7f4ec800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.845+0000 7f4ee0b95740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8853f0 0x560d9a8cb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.845+0000 7f4ee0b95740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560d9a795030 con 0x560d9a8853f0 2024-04-26T23:09:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.845+0000 7f4ee0b95740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560d9a7b07f0 con 0x560d9a8af710 2024-04-26T23:09:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.845+0000 7f4ee0b95740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x560d9a76b7e0 con 0x560d9a8ccfb0 2024-04-26T23:09:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.845+0000 7f4ed7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8853f0 0x560d9a8cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.845+0000 7f4ed7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8853f0 0x560d9a8cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47500/0 (socket says 172.21.15.40:47500) 2024-04-26T23:09:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.845+0000 7f4ed7400640 1 -- 172.21.15.40:0/1145735536 learned_addr learned my addr 172.21.15.40:0/1145735536 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.846+0000 7f4ed6000640 1 --2- 172.21.15.40:0/1145735536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8ccfb0 0x560d9a8cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.846+0000 7f4ed7400640 1 -- 172.21.15.40:0/1145735536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8ccfb0 msgr2=0x560d9a8cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.846+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/1145735536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8af710 0x560d9a8afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.846+0000 7f4ed7400640 1 --2- 172.21.15.40:0/1145735536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8ccfb0 0x560d9a8cd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.846+0000 7f4ed7400640 1 -- 172.21.15.40:0/1145735536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8af710 msgr2=0x560d9a8afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.846+0000 7f4ed7400640 1 --2- 172.21.15.40:0/1145735536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8af710 0x560d9a8afaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.846+0000 7f4ed7400640 1 -- 172.21.15.40:0/1145735536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d9a777050 con 0x560d9a8853f0 2024-04-26T23:09:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.846+0000 7f4ed6000640 1 --2- 172.21.15.40:0/1145735536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8ccfb0 0x560d9a8cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.846+0000 7f4ed7400640 1 --2- 172.21.15.40:0/1145735536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8853f0 0x560d9a8cb670 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00784a0 tx=0x7f4ed00781c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d327036400068ffc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.846+0000 7f4ec7400640 1 -- 172.21.15.40:0/1145735536 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed007caa0 con 0x560d9a8853f0 2024-04-26T23:09:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.846+0000 7f4ec7400640 1 -- 172.21.15.40:0/1145735536 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ed00812d0 con 0x560d9a8853f0 2024-04-26T23:09:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.846+0000 7f4ec7400640 1 -- 172.21.15.40:0/1145735536 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0003a10 con 0x560d9a8853f0 2024-04-26T23:09:34.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.847+0000 7f4ee0b95740 1 -- 172.21.15.40:0/1145735536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8853f0 msgr2=0x560d9a8cb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.847+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/1145735536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8853f0 0x560d9a8cb670 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00784a0 tx=0x7f4ed00781c0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.847+0000 7f4ee0b95740 1 -- 172.21.15.40:0/1145735536 shutdown_connections 2024-04-26T23:09:34.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.847+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/1145735536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8853f0 0x560d9a8cb670 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.847+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/1145735536 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8ccfb0 0x560d9a8cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.847+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/1145735536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8af710 0x560d9a8afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.847+0000 7f4ee0b95740 1 -- 172.21.15.40:0/1145735536 >> 172.21.15.40:0/1145735536 conn(0x560d9a881d20 msgr2=0x560d9a8ca520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:34.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.847+0000 7f4ee0b95740 1 -- 172.21.15.40:0/1145735536 shutdown_connections 2024-04-26T23:09:34.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.847+0000 7f4ee0b95740 1 -- 172.21.15.40:0/1145735536 wait complete. 2024-04-26T23:09:34.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.848+0000 7f4ee0b95740 1 -- start start 2024-04-26T23:09:34.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.848+0000 7f4ee0b95740 1 Processor -- start 2024-04-26T23:09:34.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.849+0000 7f4ee0b95740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8af710 0x560d9a8cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.849+0000 7f4ee0b95740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8ccfb0 0x560d9a8cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.849+0000 7f4ee0b95740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ae850 0x560d9a8aac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.849+0000 7f4ee0b95740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560d9a7ed200 con 0x560d9a8af710 2024-04-26T23:09:34.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.849+0000 7f4ee0b95740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560d9a7e7ee0 con 0x560d9a8ae850 2024-04-26T23:09:34.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.849+0000 7f4ee0b95740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x560d9a7f4e70 con 0x560d9a8ccfb0 2024-04-26T23:09:34.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.849+0000 7f4ee0b95740 10 client.0 _ll_register_callbacks cb 0x560d9a7de270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:34.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.849+0000 7f4ed6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8af710 0x560d9a8cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.849+0000 7f4ed6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8af710 0x560d9a8cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47508/0 (socket says 172.21.15.40:47508) 2024-04-26T23:09:34.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.849+0000 7f4ed6a00640 1 -- 172.21.15.40:0/3307858918 learned_addr learned my addr 172.21.15.40:0/3307858918 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:34.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.850+0000 7f4ed6a00640 1 -- 172.21.15.40:0/3307858918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8ccfb0 msgr2=0x560d9a8cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.850+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/3307858918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8ccfb0 0x560d9a8cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.850+0000 7f4ed6a00640 1 -- 172.21.15.40:0/3307858918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ae850 msgr2=0x560d9a8aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.850+0000 7f4ed7400640 1 --2- 172.21.15.40:0/3307858918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ae850 0x560d9a8aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.850+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/3307858918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ae850 0x560d9a8aac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.850+0000 7f4ed6a00640 1 -- 172.21.15.40:0/3307858918 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d9a5c47d0 con 0x560d9a8af710 2024-04-26T23:09:34.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.850+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/3307858918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8af710 0x560d9a8cdaa0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f4ec80176e0 tx=0x7f4ec8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a8c864e95c303 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.850+0000 7f4ebbe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:34.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.850+0000 7f4ebbe00640 1 -- 172.21.15.40:0/3307858918 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8015730 con 0x560d9a8af710 2024-04-26T23:09:34.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.850+0000 7f4ebbe00640 1 -- 172.21.15.40:0/3307858918 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f4ec80158d0 con 0x560d9a8af710 2024-04-26T23:09:34.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.850+0000 7f4ee0b95740 1 -- 172.21.15.40:0/3307858918 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ed00812d0 con 0x560d9a8af710 2024-04-26T23:09:34.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.851+0000 7f4ebbe00640 1 -- 172.21.15.40:0/3307858918 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7f4ec8015ad0 con 0x560d9a8af710 2024-04-26T23:09:34.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.851+0000 7f4ee0b95740 10 client.9210 fetch_fsmap learned FSMap version 118 2024-04-26T23:09:34.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.851+0000 7f4ee0b95740 1 -- 172.21.15.40:0/3307858918 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) -- 0x560d9a4fa3f0 con 0x560d9a8af710 2024-04-26T23:09:34.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.851+0000 7f4ebbe00640 1 -- 172.21.15.40:0/3307858918 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ed00812d0 con 0x560d9a8af710 2024-04-26T23:09:34.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.851+0000 7f4ebbe00640 1 -- 172.21.15.40:0/3307858918 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 118) ==== 3546+0+0 (secure 0 0 0) 0x7f4ec8032080 con 0x560d9a8af710 2024-04-26T23:09:34.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.851+0000 7f4ee0b95740 10 client.9210 fetch_fsmap finished waiting for FSMap version 118 2024-04-26T23:09:34.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.851+0000 7f4ee0b95740 10 client.9210 resolve_mds: resolved 7:1 to role '7:1' aka daemon mds.c 2024-04-26T23:09:34.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.851+0000 7f4ee0b95740 20 client.9210 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:34.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.851+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/3307858918 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x560d9a8ab1b0 0x560d9a8ab620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.851+0000 7f4ee0b95740 4 client.9210 mds_command: new command op to 8949 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:34.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.851+0000 7f4ee0b95740 1 -- 172.21.15.40:0/3307858918 --> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d9a7f67e0 con 0x560d9a8ab1b0 2024-04-26T23:09:34.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.852+0000 7f4ed6000640 1 --2- 172.21.15.40:0/3307858918 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x560d9a8ab1b0 0x560d9a8ab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.852+0000 7f4ed6000640 1 --2- 172.21.15.40:0/3307858918 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x560d9a8ab1b0 0x560d9a8ab620 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=adf0b4e1f426aef0 server_cookie=6b2b225e0d0f7dc in_seq=0 out_seq=0 2024-04-26T23:09:34.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.852+0000 7f4ebbe00640 10 client.9210 ms_handle_connect on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:34.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ebbe00640 1 -- 172.21.15.40:0/3307858918 <== mds.1 v2:172.21.15.40:6832/3250599834 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560d9a7f67e0 con 0x560d9a8ab1b0 2024-04-26T23:09:34.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ebbe00640 10 client.9210 handle_command_reply: tid=0 2024-04-26T23:09:34.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ee0b95740 1 client.9210 shutdown 2024-04-26T23:09:34.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ee0b95740 1 -- 172.21.15.40:0/3307858918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8af710 msgr2=0x560d9a8cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/3307858918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8af710 0x560d9a8cdaa0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f4ec80176e0 tx=0x7f4ec8004a40 comp rx=0 tx=0).stop 2024-04-26T23:09:34.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ee0b95740 1 -- 172.21.15.40:0/3307858918 shutdown_connections 2024-04-26T23:09:34.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/3307858918 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x560d9a8ab1b0 0x560d9a8ab620 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/3307858918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ae850 0x560d9a8aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/3307858918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8ccfb0 0x560d9a8cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/3307858918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8af710 0x560d9a8cdaa0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ebbe00640 0 client.9210 ms_handle_reset on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:34.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ee0b95740 1 -- 172.21.15.40:0/3307858918 >> 172.21.15.40:0/3307858918 conn(0x560d9a8853f0 msgr2=0x560d9a87e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:34.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ee0b95740 1 -- 172.21.15.40:0/3307858918 shutdown_connections 2024-04-26T23:09:34.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.853+0000 7f4ee0b95740 1 -- 172.21.15.40:0/3307858918 wait complete. 2024-04-26T23:09:34.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.854+0000 7f4ee0b95740 20 client.9210 trim_cache size 0 max 16384 2024-04-26T23:09:34.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.869+0000 7f4ee0b95740 1 Processor -- start 2024-04-26T23:09:34.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.869+0000 7f4ee0b95740 1 -- start start 2024-04-26T23:09:34.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.870+0000 7f4ee0b95740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ae850 0x560d9a87eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.870+0000 7f4ee0b95740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8af710 0x560d9a8a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.870+0000 7f4ee0b95740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ccfb0 0x560d9a8a5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.870+0000 7f4ee0b95740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed82021f0 con 0x560d9a8ccfb0 2024-04-26T23:09:34.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.870+0000 7f4ee0b95740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed82013c0 con 0x560d9a8ae850 2024-04-26T23:09:34.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.870+0000 7f4ee0b95740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed8201ac0 con 0x560d9a8af710 2024-04-26T23:09:34.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.871+0000 7f4ed7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ccfb0 0x560d9a8a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.871+0000 7f4ed7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ccfb0 0x560d9a8a5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42898/0 (socket says 172.21.15.40:42898) 2024-04-26T23:09:34.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.871+0000 7f4ed7400640 1 -- 172.21.15.40:0/2527640153 learned_addr learned my addr 172.21.15.40:0/2527640153 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:34.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.871+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/2527640153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ae850 0x560d9a87eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.871+0000 7f4ed6000640 1 --2- 172.21.15.40:0/2527640153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8af710 0x560d9a8a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.871+0000 7f4ed7400640 1 -- 172.21.15.40:0/2527640153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8af710 msgr2=0x560d9a8a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.871+0000 7f4ed7400640 1 --2- 172.21.15.40:0/2527640153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8af710 0x560d9a8a53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.871+0000 7f4ed7400640 1 -- 172.21.15.40:0/2527640153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ae850 msgr2=0x560d9a87eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.871+0000 7f4ed7400640 1 --2- 172.21.15.40:0/2527640153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ae850 0x560d9a87eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.871+0000 7f4ed7400640 1 -- 172.21.15.40:0/2527640153 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d9a4ba0f0 con 0x560d9a8ccfb0 2024-04-26T23:09:34.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.871+0000 7f4ed6000640 1 --2- 172.21.15.40:0/2527640153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8af710 0x560d9a8a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:34.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.871+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/2527640153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ae850 0x560d9a87eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:34.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.871+0000 7f4ed7400640 1 --2- 172.21.15.40:0/2527640153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ccfb0 0x560d9a8a5920 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0078960 tx=0x7f4ed007c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e13eb8083468dcd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.872+0000 7f4ec6a00640 1 -- 172.21.15.40:0/2527640153 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0003a10 con 0x560d9a8ccfb0 2024-04-26T23:09:34.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.872+0000 7f4ec6a00640 1 -- 172.21.15.40:0/2527640153 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ed007caa0 con 0x560d9a8ccfb0 2024-04-26T23:09:34.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.872+0000 7f4ec6a00640 1 -- 172.21.15.40:0/2527640153 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0074e10 con 0x560d9a8ccfb0 2024-04-26T23:09:34.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.872+0000 7f4ee0b95740 1 -- 172.21.15.40:0/2527640153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ccfb0 msgr2=0x560d9a8a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.872+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/2527640153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ccfb0 0x560d9a8a5920 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0078960 tx=0x7f4ed007c000 comp rx=0 tx=0).stop 2024-04-26T23:09:34.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.872+0000 7f4ee0b95740 1 -- 172.21.15.40:0/2527640153 shutdown_connections 2024-04-26T23:09:34.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.872+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/2527640153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ccfb0 0x560d9a8a5920 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.872+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/2527640153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8af710 0x560d9a8a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.872+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/2527640153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ae850 0x560d9a87eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.872+0000 7f4ee0b95740 1 -- 172.21.15.40:0/2527640153 >> 172.21.15.40:0/2527640153 conn(0x560d9a8853f0 msgr2=0x560d9a8ab590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:34.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.872+0000 7f4ee0b95740 1 -- 172.21.15.40:0/2527640153 shutdown_connections 2024-04-26T23:09:34.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.872+0000 7f4ee0b95740 1 -- 172.21.15.40:0/2527640153 wait complete. 2024-04-26T23:09:34.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.873+0000 7f4ee0b95740 1 -- start start 2024-04-26T23:09:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.873+0000 7f4ee0b95740 1 Processor -- start 2024-04-26T23:09:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.874+0000 7f4ee0b95740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ae850 0x560d9a8cc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.874+0000 7f4ee0b95740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8af710 0x560d9a8d8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.874+0000 7f4ee0b95740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ccfb0 0x560d9a8d8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.874+0000 7f4ee0b95740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560d9a7f5fd0 con 0x560d9a8ae850 2024-04-26T23:09:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.874+0000 7f4ee0b95740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed80cd840 con 0x560d9a8ccfb0 2024-04-26T23:09:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.874+0000 7f4ee0b95740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x560d9a87b570 con 0x560d9a8af710 2024-04-26T23:09:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.874+0000 7f4ee0b95740 10 client.0 _ll_register_callbacks cb 0x7f4ec8003db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.874+0000 7f4ed6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ae850 0x560d9a8cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.874+0000 7f4ed6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ae850 0x560d9a8cc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42900/0 (socket says 172.21.15.40:42900) 2024-04-26T23:09:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.875+0000 7f4ed6a00640 1 -- 172.21.15.40:0/1909133448 learned_addr learned my addr 172.21.15.40:0/1909133448 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.875+0000 7f4ed6a00640 1 -- 172.21.15.40:0/1909133448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8af710 msgr2=0x560d9a8d8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.875+0000 7f4ed6000640 1 --2- 172.21.15.40:0/1909133448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8af710 0x560d9a8d8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.875+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/1909133448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8af710 0x560d9a8d8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.875+0000 7f4ed7400640 1 --2- 172.21.15.40:0/1909133448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ccfb0 0x560d9a8d8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.875+0000 7f4ed6a00640 1 -- 172.21.15.40:0/1909133448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ccfb0 msgr2=0x560d9a8d8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.875+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/1909133448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ccfb0 0x560d9a8d8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.875+0000 7f4ed6a00640 1 -- 172.21.15.40:0/1909133448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d9a7b0a30 con 0x560d9a8ae850 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.875+0000 7f4ed6a00640 1 --2- 172.21.15.40:0/1909133448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ae850 0x560d9a8cc290 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f4ec80176b0 tx=0x7f4ec8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c556639043500d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.875+0000 7f4ebbe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ebbe00640 1 -- 172.21.15.40:0/1909133448 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8022960 con 0x560d9a8ae850 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ebbe00640 1 -- 172.21.15.40:0/1909133448 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f4ec803ac50 con 0x560d9a8ae850 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ee0b95740 1 -- 172.21.15.40:0/1909133448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ed007caa0 con 0x560d9a8ae850 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ebbe00640 1 -- 172.21.15.40:0/1909133448 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7f4ec803ae50 con 0x560d9a8ae850 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ee0b95740 10 client.9216 fetch_fsmap learned FSMap version 118 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ee0b95740 1 -- 172.21.15.40:0/1909133448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) -- 0x560d9a8a61d0 con 0x560d9a8ae850 2024-04-26T23:09:34.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ebbe00640 1 -- 172.21.15.40:0/1909133448 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ed007caa0 con 0x560d9a8ae850 2024-04-26T23:09:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ebbe00640 1 -- 172.21.15.40:0/1909133448 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 118) ==== 3546+0+0 (secure 0 0 0) 0x7f4ec8016070 con 0x560d9a8ae850 2024-04-26T23:09:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ee0b95740 10 client.9216 fetch_fsmap finished waiting for FSMap version 118 2024-04-26T23:09:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ee0b95740 10 client.9216 resolve_mds: resolved 7:1 to role '7:1' aka daemon mds.c 2024-04-26T23:09:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ee0b95740 20 client.9216 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/1909133448 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x560d9a8cb740 0x560d9a8cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ee0b95740 4 client.9216 mds_command: new command op to 8949 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:09:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.876+0000 7f4ee0b95740 1 -- 172.21.15.40:0/1909133448 --> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560d9a7f0650 con 0x560d9a8cb740 2024-04-26T23:09:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.877+0000 7f4ed6000640 1 --2- 172.21.15.40:0/1909133448 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x560d9a8cb740 0x560d9a8cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.877+0000 7f4ed6000640 1 --2- 172.21.15.40:0/1909133448 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x560d9a8cb740 0x560d9a8cd6a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ba3db6656a2f7611 server_cookie=5b642c55ada63734 in_seq=0 out_seq=0 2024-04-26T23:09:34.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.877+0000 7f4ebbe00640 10 client.9216 ms_handle_connect on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:34.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.897+0000 7f4ebbe00640 1 -- 172.21.15.40:0/1909133448 <== mds.1 v2:172.21.15.40:6832/3250599834 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560d9a7f0650 con 0x560d9a8cb740 2024-04-26T23:09:34.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.897+0000 7f4ebbe00640 10 client.9216 handle_command_reply: tid=0 2024-04-26T23:09:34.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.897+0000 7f4ee0b95740 1 client.9216 shutdown 2024-04-26T23:09:34.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.897+0000 7f4ee0b95740 1 -- 172.21.15.40:0/1909133448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ae850 msgr2=0x560d9a8cc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.897+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/1909133448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ae850 0x560d9a8cc290 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f4ec80176b0 tx=0x7f4ec8015000 comp rx=0 tx=0).stop 2024-04-26T23:09:34.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.897+0000 7f4ee0b95740 1 -- 172.21.15.40:0/1909133448 shutdown_connections 2024-04-26T23:09:34.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.897+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/1909133448 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x560d9a8cb740 0x560d9a8cd6a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.897+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/1909133448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d9a8ccfb0 0x560d9a8d8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.897+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/1909133448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560d9a8af710 0x560d9a8d8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.898+0000 7f4ee0b95740 1 --2- 172.21.15.40:0/1909133448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d9a8ae850 0x560d9a8cc290 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.898+0000 7f4ee0b95740 1 -- 172.21.15.40:0/1909133448 >> 172.21.15.40:0/1909133448 conn(0x560d9a8853f0 msgr2=0x560d9a8848f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:34.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.898+0000 7f4ebbe00640 0 client.9216 ms_handle_reset on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:34.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.898+0000 7f4ee0b95740 1 -- 172.21.15.40:0/1909133448 shutdown_connections 2024-04-26T23:09:34.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.898+0000 7f4ee0b95740 1 -- 172.21.15.40:0/1909133448 wait complete. 2024-04-26T23:09:34.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.898+0000 7f4ee0b95740 20 client.9216 trim_cache size 0 max 16384 2024-04-26T23:09:34.899 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:34.900 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:09:34.900 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:09:34.900 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:34.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.899+0000 7f4ed7e00640 1 -- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4eb406c8f0 msgr2=0x7f4eb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.899+0000 7f4ed7e00640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4eb406c8f0 0x7f4eb406edb0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4ec80041e0 tx=0x7f4ec800b040 comp rx=0 tx=0).stop 2024-04-26T23:09:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.899+0000 7f4ed7e00640 1 -- 172.21.15.40:0/3083641308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed80b7f10 msgr2=0x7f4ed8150690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.899+0000 7f4ed7e00640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed80b7f10 0x7f4ed8150690 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0003ea0 tx=0x7f4ed00036f0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.899+0000 7f4ed7e00640 1 -- 172.21.15.40:0/3083641308 shutdown_connections 2024-04-26T23:09:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.899+0000 7f4ed7e00640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4eb406c8f0 0x7f4eb406edb0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.899+0000 7f4ed7e00640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed80b7f10 0x7f4ed8150690 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.899+0000 7f4ed7e00640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed81593f0 0x7f4ed80b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.899+0000 7f4ed7e00640 1 --2- 172.21.15.40:0/3083641308 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ebc005290 0x7f4ed80b9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.899+0000 7f4ed7e00640 1 -- 172.21.15.40:0/3083641308 >> 172.21.15.40:0/3083641308 conn(0x7f4ed80d4dd0 msgr2=0x7f4ed81583c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.899+0000 7f4ed7e00640 1 -- 172.21.15.40:0/3083641308 shutdown_connections 2024-04-26T23:09:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:34.899+0000 7f4ed7e00640 1 -- 172.21.15.40:0/3083641308 wait complete. 2024-04-26T23:09:34.920 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:2 flush journal 2024-04-26T23:09:35.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.137+0000 7f9199c00640 1 Processor -- start 2024-04-26T23:09:35.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.137+0000 7f9199c00640 1 -- start start 2024-04-26T23:09:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.139+0000 7f9199c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f91941678f0 0x7f91940cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.139+0000 7f9199c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9194163a50 con 0x7f91940c41d0 2024-04-26T23:09:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.139+0000 7f9199c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91940ceb90 con 0x7f9194073f30 2024-04-26T23:09:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.139+0000 7f9199c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f91940cf7a0 con 0x7f91941678f0 2024-04-26T23:09:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.140+0000 7f9192a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f91940c41d0 0x7f91940cfc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44804/0 (socket says 172.21.15.40:44804) 2024-04-26T23:09:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.140+0000 7f9192a00640 1 -- 172.21.15.40:0/634031262 learned_addr learned my addr 172.21.15.40:0/634031262 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:35.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.140+0000 7f9193400640 1 --2- 172.21.15.40:0/634031262 >> v2:172.21.15.40:3300/0 conn(0x7f91941678f0 0x7f91940cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:35.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.140+0000 7f9193400640 1 -- 172.21.15.40:0/634031262 >> v1:172.21.15.103:6789/0 conn(0x7f9194073f30 legacy=0x7f91941671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:35.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.140+0000 7f9193400640 1 -- 172.21.15.40:0/634031262 >> v1:172.21.15.40:6789/0 conn(0x7f91940c41d0 legacy=0x7f91940cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:35.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.140+0000 7f9193400640 1 -- 172.21.15.40:0/634031262 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91940cee70 con 0x7f91941678f0 2024-04-26T23:09:35.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.141+0000 7f9193400640 1 --2- 172.21.15.40:0/634031262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91941678f0 0x7f91940cc240 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f918c009d90 tx=0x7f918c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b270d38550e1956 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:35.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.142+0000 7f9191600640 1 -- 172.21.15.40:0/634031262 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c007740 con 0x7f91941678f0 2024-04-26T23:09:35.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.142+0000 7f9191600640 1 -- 172.21.15.40:0/634031262 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f918c0078e0 con 0x7f91941678f0 2024-04-26T23:09:35.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.142+0000 7f9191600640 1 -- 172.21.15.40:0/634031262 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c007cb0 con 0x7f91941678f0 2024-04-26T23:09:35.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.143+0000 7f9199c00640 1 -- 172.21.15.40:0/634031262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91941678f0 msgr2=0x7f91940cc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:35.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.143+0000 7f9199c00640 1 --2- 172.21.15.40:0/634031262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91941678f0 0x7f91940cc240 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f918c009d90 tx=0x7f918c02a6a0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.143+0000 7f9199c00640 1 -- 172.21.15.40:0/634031262 shutdown_connections 2024-04-26T23:09:35.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.143+0000 7f9199c00640 1 --2- 172.21.15.40:0/634031262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91941678f0 0x7f91940cc240 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.143+0000 7f9199c00640 1 -- 172.21.15.40:0/634031262 >> 172.21.15.40:0/634031262 conn(0x7f91940d0be0 msgr2=0x7f91940d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:35.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.143+0000 7f9199c00640 1 -- 172.21.15.40:0/634031262 shutdown_connections 2024-04-26T23:09:35.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.144+0000 7f9199c00640 1 -- 172.21.15.40:0/634031262 wait complete. 2024-04-26T23:09:35.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.146+0000 7f9199c00640 1 Processor -- start 2024-04-26T23:09:35.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.146+0000 7f9199c00640 1 -- start start 2024-04-26T23:09:35.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.147+0000 7f9199c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9194073f30 0x7f919414f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.148+0000 7f9199c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91940c41d0 0x7f91940c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.148+0000 7f9199c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91941678f0 0x7f91940c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.148+0000 7f9199c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f91940ce7b0 con 0x7f9194073f30 2024-04-26T23:09:35.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.148+0000 7f9199c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f91940cf430 con 0x7f91940c41d0 2024-04-26T23:09:35.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.148+0000 7f9199c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f91940ce930 con 0x7f91941678f0 2024-04-26T23:09:35.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.148+0000 7f9192a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9194073f30 0x7f919414f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:35.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.148+0000 7f9192a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9194073f30 0x7f919414f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42924/0 (socket says 172.21.15.40:42924) 2024-04-26T23:09:35.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.148+0000 7f9192a00640 1 -- 172.21.15.40:0/201404183 learned_addr learned my addr 172.21.15.40:0/201404183 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:35.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.148+0000 7f9193400640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91941678f0 0x7f91940c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:35.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.148+0000 7f9192000640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91940c41d0 0x7f91940c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:35.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.148+0000 7f9192a00640 1 -- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91941678f0 msgr2=0x7f91940c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:35.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.149+0000 7f9192a00640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91941678f0 0x7f91940c6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.149+0000 7f9192a00640 1 -- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91940c41d0 msgr2=0x7f91940c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:35.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.149+0000 7f9192a00640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91940c41d0 0x7f91940c68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.149+0000 7f9192a00640 1 -- 172.21.15.40:0/201404183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919414f630 con 0x7f9194073f30 2024-04-26T23:09:35.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.149+0000 7f9192000640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91940c41d0 0x7f91940c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:35.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.149+0000 7f9192a00640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9194073f30 0x7f919414f0f0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f918400c400 tx=0x7f918400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:35.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.149+0000 7f918aa00640 1 -- 172.21.15.40:0/201404183 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9184017070 con 0x7f9194073f30 2024-04-26T23:09:35.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.149+0000 7f918aa00640 1 -- 172.21.15.40:0/201404183 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9184005ce0 con 0x7f9194073f30 2024-04-26T23:09:35.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.149+0000 7f918aa00640 1 -- 172.21.15.40:0/201404183 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91840072f0 con 0x7f9194073f30 2024-04-26T23:09:35.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.149+0000 7f9199c00640 1 -- 172.21.15.40:0/201404183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91941ed980 con 0x7f9194073f30 2024-04-26T23:09:35.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.150+0000 7f9199c00640 1 -- 172.21.15.40:0/201404183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91941edf40 con 0x7f9194073f30 2024-04-26T23:09:35.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.153+0000 7f918aa00640 1 -- 172.21.15.40:0/201404183 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f918401a070 con 0x7f9194073f30 2024-04-26T23:09:35.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.154+0000 7f918aa00640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f917406c8f0 0x7f917406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.154+0000 7f918aa00640 1 -- 172.21.15.40:0/201404183 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f918408b190 con 0x7f9194073f30 2024-04-26T23:09:35.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.154+0000 7f919cbce740 1 Processor -- start 2024-04-26T23:09:35.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.154+0000 7f919cbce740 1 -- start start 2024-04-26T23:09:35.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.154+0000 7f9192000640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f917406c8f0 0x7f917406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:35.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f919cbce740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3dc540 0x56380c3dc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f9192000640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f917406c8f0 0x7f917406edb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f91940b6b50 tx=0x7f9180009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:35.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f919cbce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 0x56380c3fa330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f919cbce740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3b2220 0x56380c3f84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f919cbce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56380c2c2030 con 0x56380c3f9de0 2024-04-26T23:09:35.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f919cbce740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56380c2dd7f0 con 0x56380c3dc540 2024-04-26T23:09:35.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f919cbce740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x56380c2987e0 con 0x56380c3b2220 2024-04-26T23:09:35.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f9192000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 0x56380c3fa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:35.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f9192000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 0x56380c3fa330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42928/0 (socket says 172.21.15.40:42928) 2024-04-26T23:09:35.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f9192000640 1 -- 172.21.15.40:0/169055851 learned_addr learned my addr 172.21.15.40:0/169055851 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:35.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f9193400640 1 --2- 172.21.15.40:0/169055851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3b2220 0x56380c3f84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:35.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f9192a00640 1 --2- 172.21.15.40:0/169055851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3dc540 0x56380c3dc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:35.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f9192000640 1 -- 172.21.15.40:0/169055851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3b2220 msgr2=0x56380c3f84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:35.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f9192000640 1 --2- 172.21.15.40:0/169055851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3b2220 0x56380c3f84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.156+0000 7f9192000640 1 -- 172.21.15.40:0/169055851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3dc540 msgr2=0x56380c3dc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:35.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.157+0000 7f9192000640 1 --2- 172.21.15.40:0/169055851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3dc540 0x56380c3dc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.157+0000 7f9192000640 1 -- 172.21.15.40:0/169055851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56380c2a4050 con 0x56380c3f9de0 2024-04-26T23:09:35.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.157+0000 7f9193400640 1 --2- 172.21.15.40:0/169055851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3b2220 0x56380c3f84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:35.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.157+0000 7f9192000640 1 --2- 172.21.15.40:0/169055851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 0x56380c3fa330 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f918000da30 tx=0x7f9180010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa4995d4012515c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:35.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.157+0000 7f917f400640 1 -- 172.21.15.40:0/169055851 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9180024070 con 0x56380c3f9de0 2024-04-26T23:09:35.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.157+0000 7f917f400640 1 -- 172.21.15.40:0/169055851 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9180005480 con 0x56380c3f9de0 2024-04-26T23:09:35.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.157+0000 7f917f400640 1 -- 172.21.15.40:0/169055851 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9180014bc0 con 0x56380c3f9de0 2024-04-26T23:09:35.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.158+0000 7f919cbce740 1 -- 172.21.15.40:0/169055851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 msgr2=0x56380c3fa330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:35.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.158+0000 7f919cbce740 1 --2- 172.21.15.40:0/169055851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 0x56380c3fa330 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f918000da30 tx=0x7f9180010000 comp rx=0 tx=0).stop 2024-04-26T23:09:35.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.158+0000 7f919cbce740 1 -- 172.21.15.40:0/169055851 shutdown_connections 2024-04-26T23:09:35.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.158+0000 7f919cbce740 1 --2- 172.21.15.40:0/169055851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3b2220 0x56380c3f84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.158+0000 7f919cbce740 1 --2- 172.21.15.40:0/169055851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 0x56380c3fa330 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.158+0000 7f919cbce740 1 --2- 172.21.15.40:0/169055851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3dc540 0x56380c3dc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.158+0000 7f919cbce740 1 -- 172.21.15.40:0/169055851 >> 172.21.15.40:0/169055851 conn(0x56380c3aeb50 msgr2=0x56380c3f7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.158+0000 7f919cbce740 1 -- 172.21.15.40:0/169055851 shutdown_connections 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.158+0000 7f919cbce740 1 -- 172.21.15.40:0/169055851 wait complete. 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.159+0000 7f919cbce740 1 -- start start 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.159+0000 7f919cbce740 1 Processor -- start 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.160+0000 7f919cbce740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3dc540 0x56380c407650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.160+0000 7f919cbce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 0x56380c3ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.160+0000 7f919cbce740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3d82d0 0x56380c3abcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.160+0000 7f919cbce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56380c31a200 con 0x56380c3f9de0 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.160+0000 7f919cbce740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56380c314ee0 con 0x56380c3dc540 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.160+0000 7f919cbce740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x56380c321e70 con 0x56380c3d82d0 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.160+0000 7f919cbce740 10 client.0 _ll_register_callbacks cb 0x56380c30b270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:35.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.160+0000 7f9192000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 0x56380c3ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.160+0000 7f9192000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 0x56380c3ab7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42932/0 (socket says 172.21.15.40:42932) 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.160+0000 7f9192000640 1 -- 172.21.15.40:0/2641788430 learned_addr learned my addr 172.21.15.40:0/2641788430 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.161+0000 7f9192000640 1 -- 172.21.15.40:0/2641788430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3d82d0 msgr2=0x56380c3abcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.161+0000 7f9192000640 1 --2- 172.21.15.40:0/2641788430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3d82d0 0x56380c3abcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.161+0000 7f9192000640 1 -- 172.21.15.40:0/2641788430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3dc540 msgr2=0x56380c407650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.161+0000 7f9192000640 1 --2- 172.21.15.40:0/2641788430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3dc540 0x56380c407650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.161+0000 7f9192000640 1 -- 172.21.15.40:0/2641788430 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56380c0f17d0 con 0x56380c3f9de0 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.161+0000 7f9192000640 1 --2- 172.21.15.40:0/2641788430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 0x56380c3ab7b0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f918001e000 tx=0x7f91800036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c54c4f7514d5dd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.161+0000 7f9173e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.161+0000 7f9173e00640 1 -- 172.21.15.40:0/2641788430 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918002f020 con 0x56380c3f9de0 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.161+0000 7f9173e00640 1 -- 172.21.15.40:0/2641788430 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f9180015d30 con 0x56380c3f9de0 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.161+0000 7f919cbce740 1 -- 172.21.15.40:0/2641788430 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9180005480 con 0x56380c3f9de0 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.161+0000 7f9173e00640 1 -- 172.21.15.40:0/2641788430 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7f9180039050 con 0x56380c3f9de0 2024-04-26T23:09:35.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.161+0000 7f919cbce740 10 client.9228 fetch_fsmap learned FSMap version 118 2024-04-26T23:09:35.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.162+0000 7f919cbce740 1 -- 172.21.15.40:0/2641788430 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) -- 0x56380c0273f0 con 0x56380c3f9de0 2024-04-26T23:09:35.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.162+0000 7f9173e00640 1 -- 172.21.15.40:0/2641788430 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9180005480 con 0x56380c3f9de0 2024-04-26T23:09:35.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.162+0000 7f9173e00640 1 -- 172.21.15.40:0/2641788430 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 118) ==== 3546+0+0 (secure 0 0 0) 0x7f91800067d0 con 0x56380c3f9de0 2024-04-26T23:09:35.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.162+0000 7f919cbce740 10 client.9228 fetch_fsmap finished waiting for FSMap version 118 2024-04-26T23:09:35.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.162+0000 7f919cbce740 10 client.9228 resolve_mds: resolved 7:2 to role '7:2' aka daemon mds.d 2024-04-26T23:09:35.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.162+0000 7f919cbce740 20 client.9228 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:35.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.162+0000 7f919cbce740 1 --2- 172.21.15.40:0/2641788430 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x56380c3db050 0x56380c3db4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.162+0000 7f919cbce740 4 client.9228 mds_command: new command op to 6598 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:35.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.162+0000 7f919cbce740 1 -- 172.21.15.40:0/2641788430 --> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56380c3237e0 con 0x56380c3db050 2024-04-26T23:09:35.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.163+0000 7f9192a00640 1 --2- 172.21.15.40:0/2641788430 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x56380c3db050 0x56380c3db4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:35.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.164+0000 7f9192a00640 1 --2- 172.21.15.40:0/2641788430 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x56380c3db050 0x56380c3db4b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=51568ebfce12dd88 server_cookie=970eeb744ccf13ab in_seq=0 out_seq=0 2024-04-26T23:09:35.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.164+0000 7f9173e00640 10 client.9228 ms_handle_connect on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:35.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f9173e00640 1 -- 172.21.15.40:0/2641788430 <== mds.2 v2:172.21.15.103:6835/2022020922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56380c3237e0 con 0x56380c3db050 2024-04-26T23:09:35.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f9173e00640 10 client.9228 handle_command_reply: tid=0 2024-04-26T23:09:35.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f919cbce740 1 client.9228 shutdown 2024-04-26T23:09:35.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f919cbce740 1 -- 172.21.15.40:0/2641788430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 msgr2=0x56380c3ab7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:35.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f919cbce740 1 --2- 172.21.15.40:0/2641788430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 0x56380c3ab7b0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f918001e000 tx=0x7f91800036f0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f919cbce740 1 -- 172.21.15.40:0/2641788430 shutdown_connections 2024-04-26T23:09:35.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f919cbce740 1 --2- 172.21.15.40:0/2641788430 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x56380c3db050 0x56380c3db4b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f919cbce740 1 --2- 172.21.15.40:0/2641788430 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3d82d0 0x56380c3abcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f919cbce740 1 --2- 172.21.15.40:0/2641788430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3f9de0 0x56380c3ab7b0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f919cbce740 1 --2- 172.21.15.40:0/2641788430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3dc540 0x56380c407650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f919cbce740 1 -- 172.21.15.40:0/2641788430 >> 172.21.15.40:0/2641788430 conn(0x56380c3b2220 msgr2=0x56380c3fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:35.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f9173e00640 0 client.9228 ms_handle_reset on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:35.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f919cbce740 1 -- 172.21.15.40:0/2641788430 shutdown_connections 2024-04-26T23:09:35.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f919cbce740 1 -- 172.21.15.40:0/2641788430 wait complete. 2024-04-26T23:09:35.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.166+0000 7f919cbce740 20 client.9228 trim_cache size 0 max 16384 2024-04-26T23:09:35.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.183+0000 7f919cbce740 1 Processor -- start 2024-04-26T23:09:35.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.183+0000 7f919cbce740 1 -- start start 2024-04-26T23:09:35.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f919cbce740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3d82d0 0x56380c41b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f919cbce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3dc540 0x56380c3db240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f919cbce740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3f9de0 0x56380c3d6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f919cbce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9194206330 con 0x56380c3dc540 2024-04-26T23:09:35.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f919cbce740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9194205500 con 0x56380c3f9de0 2024-04-26T23:09:35.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f919cbce740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9194205c00 con 0x56380c3d82d0 2024-04-26T23:09:35.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f9192000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3dc540 0x56380c3db240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:35.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f9192000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3dc540 0x56380c3db240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42940/0 (socket says 172.21.15.40:42940) 2024-04-26T23:09:35.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f9192000640 1 -- 172.21.15.40:0/569529340 learned_addr learned my addr 172.21.15.40:0/569529340 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:35.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f9192000640 1 -- 172.21.15.40:0/569529340 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3d82d0 msgr2=0x56380c41b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:35.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f9192000640 1 --2- 172.21.15.40:0/569529340 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3d82d0 0x56380c41b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f9192000640 1 -- 172.21.15.40:0/569529340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3f9de0 msgr2=0x56380c3d6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:35.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f9192000640 1 --2- 172.21.15.40:0/569529340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3f9de0 0x56380c3d6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.184+0000 7f9192000640 1 -- 172.21.15.40:0/569529340 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56380bfe70f0 con 0x56380c3dc540 2024-04-26T23:09:35.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.185+0000 7f9192000640 1 --2- 172.21.15.40:0/569529340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3dc540 0x56380c3db240 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f918000e010 tx=0x7f9180028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30bfaddc2d890f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:35.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.185+0000 7f917ea00640 1 -- 172.21.15.40:0/569529340 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918002f070 con 0x56380c3dc540 2024-04-26T23:09:35.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.185+0000 7f917ea00640 1 -- 172.21.15.40:0/569529340 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9180015ac0 con 0x56380c3dc540 2024-04-26T23:09:35.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.185+0000 7f917ea00640 1 -- 172.21.15.40:0/569529340 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9180024040 con 0x56380c3dc540 2024-04-26T23:09:35.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.185+0000 7f919cbce740 1 -- 172.21.15.40:0/569529340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3dc540 msgr2=0x56380c3db240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:35.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.185+0000 7f919cbce740 1 --2- 172.21.15.40:0/569529340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3dc540 0x56380c3db240 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f918000e010 tx=0x7f9180028000 comp rx=0 tx=0).stop 2024-04-26T23:09:35.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.185+0000 7f919cbce740 1 -- 172.21.15.40:0/569529340 shutdown_connections 2024-04-26T23:09:35.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.185+0000 7f919cbce740 1 --2- 172.21.15.40:0/569529340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3f9de0 0x56380c3d6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.185+0000 7f919cbce740 1 --2- 172.21.15.40:0/569529340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3dc540 0x56380c3db240 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.185+0000 7f919cbce740 1 --2- 172.21.15.40:0/569529340 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3d82d0 0x56380c41b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.185+0000 7f919cbce740 1 -- 172.21.15.40:0/569529340 >> 172.21.15.40:0/569529340 conn(0x56380c3b2220 msgr2=0x56380c3aba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:35.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.186+0000 7f919cbce740 1 -- 172.21.15.40:0/569529340 shutdown_connections 2024-04-26T23:09:35.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.186+0000 7f919cbce740 1 -- 172.21.15.40:0/569529340 wait complete. 2024-04-26T23:09:35.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.186+0000 7f919cbce740 1 -- start start 2024-04-26T23:09:35.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.186+0000 7f919cbce740 1 Processor -- start 2024-04-26T23:09:35.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.187+0000 7f919cbce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3d82d0 0x56380c3f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.187+0000 7f919cbce740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3dc540 0x56380c3d6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.187+0000 7f919cbce740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3f9de0 0x56380c46f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f919cbce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56380c322fd0 con 0x56380c3d82d0 2024-04-26T23:09:35.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f919cbce740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9194150980 con 0x56380c3dc540 2024-04-26T23:09:35.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f919cbce740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x56380c3d80e0 con 0x56380c3f9de0 2024-04-26T23:09:35.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f919cbce740 10 client.0 _ll_register_callbacks cb 0x7f9180022070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:35.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f9192a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3d82d0 0x56380c3f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:35.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f9192a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3d82d0 0x56380c3f5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42950/0 (socket says 172.21.15.40:42950) 2024-04-26T23:09:35.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f9192a00640 1 -- 172.21.15.40:0/4067702124 learned_addr learned my addr 172.21.15.40:0/4067702124 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:35.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f9192a00640 1 -- 172.21.15.40:0/4067702124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3f9de0 msgr2=0x56380c46f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:35.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f9192a00640 1 --2- 172.21.15.40:0/4067702124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3f9de0 0x56380c46f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f9192a00640 1 -- 172.21.15.40:0/4067702124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3dc540 msgr2=0x56380c3d6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:35.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f9192a00640 1 --2- 172.21.15.40:0/4067702124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3dc540 0x56380c3d6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f9192a00640 1 -- 172.21.15.40:0/4067702124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56380c2dda30 con 0x56380c3d82d0 2024-04-26T23:09:35.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f9192a00640 1 --2- 172.21.15.40:0/4067702124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3d82d0 0x56380c3f5f80 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f9184089490 tx=0x7f918400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f92aab1edc0388e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:35.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.188+0000 7f9173e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:35.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f9173e00640 1 -- 172.21.15.40:0/4067702124 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9184087060 con 0x56380c3d82d0 2024-04-26T23:09:35.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f9173e00640 1 -- 172.21.15.40:0/4067702124 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f9184071210 con 0x56380c3d82d0 2024-04-26T23:09:35.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f919cbce740 1 -- 172.21.15.40:0/4067702124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9180015ac0 con 0x56380c3d82d0 2024-04-26T23:09:35.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f9173e00640 1 -- 172.21.15.40:0/4067702124 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7f9184076050 con 0x56380c3d82d0 2024-04-26T23:09:35.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f919cbce740 10 client.9234 fetch_fsmap learned FSMap version 118 2024-04-26T23:09:35.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f919cbce740 1 -- 172.21.15.40:0/4067702124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) -- 0x56380c3eda10 con 0x56380c3d82d0 2024-04-26T23:09:35.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f9173e00640 1 -- 172.21.15.40:0/4067702124 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9180015ac0 con 0x56380c3d82d0 2024-04-26T23:09:35.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f9173e00640 1 -- 172.21.15.40:0/4067702124 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 118) ==== 3546+0+0 (secure 0 0 0) 0x7f9184055300 con 0x56380c3d82d0 2024-04-26T23:09:35.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f919cbce740 10 client.9234 fetch_fsmap finished waiting for FSMap version 118 2024-04-26T23:09:35.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f919cbce740 10 client.9234 resolve_mds: resolved 7:2 to role '7:2' aka daemon mds.d 2024-04-26T23:09:35.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f919cbce740 20 client.9234 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:35.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f919cbce740 1 --2- 172.21.15.40:0/4067702124 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x56380c4099e0 0x56380c46f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:35.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f919cbce740 4 client.9234 mds_command: new command op to 6598 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T23:09:35.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.189+0000 7f919cbce740 1 -- 172.21.15.40:0/4067702124 --> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56380c31d650 con 0x56380c4099e0 2024-04-26T23:09:35.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.190+0000 7f9192000640 1 --2- 172.21.15.40:0/4067702124 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x56380c4099e0 0x56380c46f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:35.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.191+0000 7f9192000640 1 --2- 172.21.15.40:0/4067702124 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x56380c4099e0 0x56380c46f670 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a41c9e97957013d2 server_cookie=16c62ca2fe20a921 in_seq=0 out_seq=0 2024-04-26T23:09:35.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.191+0000 7f9173e00640 10 client.9234 ms_handle_connect on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:35.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.213+0000 7f9173e00640 1 -- 172.21.15.40:0/4067702124 <== mds.2 v2:172.21.15.103:6835/2022020922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56380c31d650 con 0x56380c4099e0 2024-04-26T23:09:35.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.213+0000 7f9173e00640 10 client.9234 handle_command_reply: tid=0 2024-04-26T23:09:35.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.213+0000 7f919cbce740 1 client.9234 shutdown 2024-04-26T23:09:35.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.213+0000 7f919cbce740 1 -- 172.21.15.40:0/4067702124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3d82d0 msgr2=0x56380c3f5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:35.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.213+0000 7f919cbce740 1 --2- 172.21.15.40:0/4067702124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3d82d0 0x56380c3f5f80 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f9184089490 tx=0x7f918400cf70 comp rx=0 tx=0).stop 2024-04-26T23:09:35.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.213+0000 7f919cbce740 1 -- 172.21.15.40:0/4067702124 shutdown_connections 2024-04-26T23:09:35.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.213+0000 7f919cbce740 1 --2- 172.21.15.40:0/4067702124 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x56380c4099e0 0x56380c46f670 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.213+0000 7f919cbce740 1 --2- 172.21.15.40:0/4067702124 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56380c3f9de0 0x56380c46f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.213+0000 7f919cbce740 1 --2- 172.21.15.40:0/4067702124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c3dc540 0x56380c3d6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.213+0000 7f919cbce740 1 --2- 172.21.15.40:0/4067702124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56380c3d82d0 0x56380c3f5f80 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.213+0000 7f919cbce740 1 -- 172.21.15.40:0/4067702124 >> 172.21.15.40:0/4067702124 conn(0x56380c3b2220 msgr2=0x56380c3db240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:35.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.213+0000 7f9173e00640 0 client.9234 ms_handle_reset on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:35.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.213+0000 7f919cbce740 1 -- 172.21.15.40:0/4067702124 shutdown_connections 2024-04-26T23:09:35.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.214+0000 7f919cbce740 1 -- 172.21.15.40:0/4067702124 wait complete. 2024-04-26T23:09:35.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.214+0000 7f919cbce740 20 client.9234 trim_cache size 0 max 16384 2024-04-26T23:09:35.215 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "message": "", 2024-04-26T23:09:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:09:35.216 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:35.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.215+0000 7f9199c00640 1 -- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f917406c8f0 msgr2=0x7f917406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:35.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.215+0000 7f9199c00640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f917406c8f0 0x7f917406edb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f91940b6b50 tx=0x7f9180009040 comp rx=0 tx=0).stop 2024-04-26T23:09:35.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.215+0000 7f9199c00640 1 -- 172.21.15.40:0/201404183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9194073f30 msgr2=0x7f919414f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:35.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.215+0000 7f9199c00640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9194073f30 0x7f919414f0f0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f918400c400 tx=0x7f918400c8c0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.215+0000 7f9199c00640 1 -- 172.21.15.40:0/201404183 shutdown_connections 2024-04-26T23:09:35.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.215+0000 7f9199c00640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f917406c8f0 0x7f917406edb0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.215+0000 7f9199c00640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91941678f0 0x7f91940c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.215+0000 7f9199c00640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91940c41d0 0x7f91940c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.215+0000 7f9199c00640 1 --2- 172.21.15.40:0/201404183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9194073f30 0x7f919414f0f0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:35.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.215+0000 7f9199c00640 1 -- 172.21.15.40:0/201404183 >> 172.21.15.40:0/201404183 conn(0x7f91940d0be0 msgr2=0x7f919406de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:35.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.215+0000 7f9199c00640 1 -- 172.21.15.40:0/201404183 shutdown_connections 2024-04-26T23:09:35.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35.215+0000 7f9199c00640 1 -- 172.21.15.40:0/201404183 wait complete. 2024-04-26T23:09:35.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:35.236 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec find d1/d2/d3/) 2024-04-26T23:09:35.302 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/ 2024-04-26T23:09:35.303 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4 2024-04-26T23:09:35.303 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5 2024-04-26T23:09:35.303 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6 2024-04-26T23:09:35.303 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6/d7 2024-04-26T23:09:35.303 INFO:teuthology.orchestra.run.smithi040.stdout:d1/d2/d3/d4/d5/d6/d7/d8 2024-04-26T23:09:35.304 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:09:35.304 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:09:35.304 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:09:35.304 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:09:35.304 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/").st_ino) 2024-04-26T23:09:35.304 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:09:35.444 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627778 2024-04-26T23:09:35.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:09:35.459 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:09:35.459 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:09:35.459 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:09:35.459 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:09:35.459 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4").st_ino) 2024-04-26T23:09:35.459 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:09:35.653 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627779 2024-04-26T23:09:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:09:35.668 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:09:35.668 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:09:35.668 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:09:35.668 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:09:35.668 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5").st_ino) 2024-04-26T23:09:35.668 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:09:35.861 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627780 2024-04-26T23:09:35.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:09:35.875 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:09:35.875 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:09:35.875 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:09:35.875 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:09:35.876 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6").st_ino) 2024-04-26T23:09:35.876 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:09:36.070 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627781 2024-04-26T23:09:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:09:36.084 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:09:36.084 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:09:36.084 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:09:36.084 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:09:36.085 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6/d7").st_ino) 2024-04-26T23:09:36.085 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:09:36.278 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627782 2024-04-26T23:09:36.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:09:36.293 DEBUG:teuthology.orchestra.run.smithi040:> stdin-killer -- python3 -c ' 2024-04-26T23:09:36.293 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:09:36.293 DEBUG:teuthology.orchestra.run.smithi040:> import stat 2024-04-26T23:09:36.293 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:09:36.293 DEBUG:teuthology.orchestra.run.smithi040:> print(os.stat("/home/ubuntu/cephtest/mnt.0/d1/d2/d3/d4/d5/d6/d7/d8").st_ino) 2024-04-26T23:09:36.293 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:09:36.485 INFO:teuthology.orchestra.run.smithi040.stdout:1099511627783 2024-04-26T23:09:36.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:09:36.499 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000002.00000000 parent 2024-04-26T23:09:36.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.659+0000 7f14c5e11f40 1 Processor -- start 2024-04-26T23:09:36.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.659+0000 7f14c5e11f40 1 -- start start 2024-04-26T23:09:36.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.660+0000 7f14c5e11f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a34f2dee90 0x55a34f386c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.660+0000 7f14c5e11f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a34f2e0070 0x55a34f37e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.660+0000 7f14c5e11f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a34f2ddb50 0x55a34f2de000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.660+0000 7f14c5e11f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a34f385310 con 0x55a34f2e0070 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.660+0000 7f14c5e11f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a34f385030 con 0x55a34f2ddb50 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.660+0000 7f14c5e11f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a34f3855f0 con 0x55a34f2dee90 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.660+0000 7f14c1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a34f2e0070 0x55a34f37e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.660+0000 7f14c1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a34f2e0070 0x55a34f37e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42966/0 (socket says 172.21.15.40:42966) 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.660+0000 7f14c1a00640 1 -- 172.21.15.40:0/1642645703 learned_addr learned my addr 172.21.15.40:0/1642645703 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.660+0000 7f14c1a00640 1 -- 172.21.15.40:0/1642645703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a34f2dee90 msgr2=0x55a34f386c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.660+0000 7f14c2400640 1 --2- 172.21.15.40:0/1642645703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a34f2dee90 0x55a34f386c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.661+0000 7f14c1a00640 1 --2- 172.21.15.40:0/1642645703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a34f2dee90 0x55a34f386c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.661+0000 7f14c2e00640 1 --2- 172.21.15.40:0/1642645703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a34f2ddb50 0x55a34f2de000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.661+0000 7f14c1a00640 1 -- 172.21.15.40:0/1642645703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a34f2ddb50 msgr2=0x55a34f2de000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.661+0000 7f14c1a00640 1 --2- 172.21.15.40:0/1642645703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a34f2ddb50 0x55a34f2de000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.661+0000 7f14c1a00640 1 -- 172.21.15.40:0/1642645703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a34f1d7b30 con 0x55a34f2e0070 2024-04-26T23:09:36.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.661+0000 7f14c2400640 1 --2- 172.21.15.40:0/1642645703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a34f2dee90 0x55a34f386c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:36.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.661+0000 7f14c1a00640 1 --2- 172.21.15.40:0/1642645703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a34f2e0070 0x55a34f37e460 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f14ac00a5e0 tx=0x7f14ac00aaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124378704e22b668 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:36.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.661+0000 7f14bbe00640 1 -- 172.21.15.40:0/1642645703 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ac008a40 con 0x55a34f2e0070 2024-04-26T23:09:36.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.661+0000 7f14bbe00640 1 -- 172.21.15.40:0/1642645703 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14ac008c00 con 0x55a34f2e0070 2024-04-26T23:09:36.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.661+0000 7f14bbe00640 1 -- 172.21.15.40:0/1642645703 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ac013440 con 0x55a34f2e0070 2024-04-26T23:09:36.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.662+0000 7f14c5e11f40 1 -- 172.21.15.40:0/1642645703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a34f2e0070 msgr2=0x55a34f37e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.662+0000 7f14c5e11f40 1 --2- 172.21.15.40:0/1642645703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a34f2e0070 0x55a34f37e460 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f14ac00a5e0 tx=0x7f14ac00aaa0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.662+0000 7f14c5e11f40 1 -- 172.21.15.40:0/1642645703 shutdown_connections 2024-04-26T23:09:36.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.662+0000 7f14c5e11f40 1 --2- 172.21.15.40:0/1642645703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a34f2ddb50 0x55a34f2de000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.662+0000 7f14c5e11f40 1 --2- 172.21.15.40:0/1642645703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a34f2e0070 0x55a34f37e460 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.662+0000 7f14c5e11f40 1 --2- 172.21.15.40:0/1642645703 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a34f2dee90 0x55a34f386c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.662+0000 7f14c5e11f40 1 -- 172.21.15.40:0/1642645703 >> 172.21.15.40:0/1642645703 conn(0x55a34f2fadf0 msgr2=0x55a34f384460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:36.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.662+0000 7f14c5e11f40 1 -- 172.21.15.40:0/1642645703 shutdown_connections 2024-04-26T23:09:36.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.662+0000 7f14c5e11f40 1 -- 172.21.15.40:0/1642645703 wait complete. 2024-04-26T23:09:36.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.663+0000 7f14c5e11f40 1 Processor -- start 2024-04-26T23:09:36.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.663+0000 7f14c5e11f40 1 -- start start 2024-04-26T23:09:36.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.664+0000 7f14c5e11f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a34f2ddb50 0x55a34f41b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.664+0000 7f14c5e11f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a34f2dee90 0x55a34f4187c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.664+0000 7f14c5e11f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a34f2e0070 0x55a34f37c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.664+0000 7f14c5e11f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a34f2dc660 con 0x55a34f2e0070 2024-04-26T23:09:36.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.664+0000 7f14c5e11f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a34f2dece0 con 0x55a34f2ddb50 2024-04-26T23:09:36.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.664+0000 7f14c5e11f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a34f388c40 con 0x55a34f2dee90 2024-04-26T23:09:36.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.664+0000 7f14c2e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a34f2e0070 0x55a34f37c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.664+0000 7f14c2e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a34f2e0070 0x55a34f37c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42968/0 (socket says 172.21.15.40:42968) 2024-04-26T23:09:36.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.664+0000 7f14c2e00640 1 -- 172.21.15.40:0/2863716754 learned_addr learned my addr 172.21.15.40:0/2863716754 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:36.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.665+0000 7f14c2400640 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a34f2ddb50 0x55a34f41b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.665+0000 7f14c2e00640 1 -- 172.21.15.40:0/2863716754 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a34f2dee90 msgr2=0x55a34f4187c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.665+0000 7f14c1a00640 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a34f2dee90 0x55a34f4187c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.665+0000 7f14c2e00640 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a34f2dee90 0x55a34f4187c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.665+0000 7f14c2e00640 1 -- 172.21.15.40:0/2863716754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a34f2ddb50 msgr2=0x55a34f41b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.665+0000 7f14c2e00640 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a34f2ddb50 0x55a34f41b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.665+0000 7f14c2e00640 1 -- 172.21.15.40:0/2863716754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a34f41b880 con 0x55a34f2e0070 2024-04-26T23:09:36.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.665+0000 7f14c2400640 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a34f2ddb50 0x55a34f41b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:36.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.665+0000 7f14c2e00640 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a34f2e0070 0x55a34f37c970 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f14bc055040 tx=0x7f14bc002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:36.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.666+0000 7f14baa00640 1 -- 172.21.15.40:0/2863716754 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc06b070 con 0x55a34f2e0070 2024-04-26T23:09:36.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.666+0000 7f14baa00640 1 -- 172.21.15.40:0/2863716754 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14bc052890 con 0x55a34f2e0070 2024-04-26T23:09:36.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.666+0000 7f14c5e11f40 1 -- 172.21.15.40:0/2863716754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a34f41bb60 con 0x55a34f2e0070 2024-04-26T23:09:36.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.666+0000 7f14baa00640 1 -- 172.21.15.40:0/2863716754 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc052c80 con 0x55a34f2e0070 2024-04-26T23:09:36.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.666+0000 7f14c5e11f40 1 -- 172.21.15.40:0/2863716754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a34f41c120 con 0x55a34f2e0070 2024-04-26T23:09:36.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.668+0000 7f14baa00640 1 -- 172.21.15.40:0/2863716754 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f14bc06e070 con 0x55a34f2e0070 2024-04-26T23:09:36.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.669+0000 7f14baa00640 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f14a006b2d0 0x7f14a006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.669+0000 7f14baa00640 1 -- 172.21.15.40:0/2863716754 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f14bc0ce540 con 0x55a34f2e0070 2024-04-26T23:09:36.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.669+0000 7f14c5e11f40 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x55a34f37d3c0 0x55a34f4268d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.669+0000 7f14c5e11f40 1 -- 172.21.15.40:0/2863716754 --> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] -- osd_op(unknown.0.0:1 15.12 15:4bec3886:::10000000002.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a34f426e10 con 0x55a34f37d3c0 2024-04-26T23:09:36.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.670+0000 7f14c2400640 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f14a006b2d0 0x7f14a006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.670+0000 7f14c1a00640 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x55a34f37d3c0 0x55a34f4268d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.670+0000 7f14c1a00640 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x55a34f37d3c0 0x55a34f4268d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:36.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.671+0000 7f14c2400640 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f14a006b2d0 0x7f14a006d790 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f14b400a530 tx=0x7f14b4007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:36.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.673+0000 7f14c1a00640 1 -- 172.21.15.40:0/2863716754 <== osd.3 v2:172.21.15.40:6823/378872239 1 ==== osd_op_reply(1 10000000002.00000000 [rmxattr] v103'8 uv8 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f14ac008840 con 0x55a34f37d3c0 2024-04-26T23:09:36.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.673+0000 7f14c5e11f40 1 -- 172.21.15.40:0/2863716754 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x55a34f37d3c0 msgr2=0x55a34f4268d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.673+0000 7f14c5e11f40 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x55a34f37d3c0 0x55a34f4268d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.673+0000 7f14c5e11f40 1 -- 172.21.15.40:0/2863716754 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f14a006b2d0 msgr2=0x7f14a006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.673+0000 7f14c5e11f40 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f14a006b2d0 0x7f14a006d790 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f14b400a530 tx=0x7f14b4007040 comp rx=0 tx=0).stop 2024-04-26T23:09:36.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.673+0000 7f14c5e11f40 1 -- 172.21.15.40:0/2863716754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a34f2e0070 msgr2=0x55a34f37c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.673+0000 7f14c5e11f40 1 --2- 172.21.15.40:0/2863716754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a34f2e0070 0x55a34f37c970 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f14bc055040 tx=0x7f14bc002fb0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.673+0000 7f14c2e00640 1 -- 172.21.15.40:0/2863716754 reap_dead start 2024-04-26T23:09:36.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.673+0000 7f14c5e11f40 1 -- 172.21.15.40:0/2863716754 shutdown_connections 2024-04-26T23:09:36.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.673+0000 7f14c5e11f40 1 -- 172.21.15.40:0/2863716754 >> 172.21.15.40:0/2863716754 conn(0x55a34f2fadf0 msgr2=0x55a34f2ddf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:36.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.673+0000 7f14c5e11f40 1 -- 172.21.15.40:0/2863716754 shutdown_connections 2024-04-26T23:09:36.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.673+0000 7f14c5e11f40 1 -- 172.21.15.40:0/2863716754 wait complete. 2024-04-26T23:09:36.678 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000003.00000000 parent 2024-04-26T23:09:36.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.766+0000 7f72397cbf40 1 Processor -- start 2024-04-26T23:09:36.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.766+0000 7f72397cbf40 1 -- start start 2024-04-26T23:09:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.767+0000 7f72397cbf40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c6d114ae90 0x55c6d11482e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.767+0000 7f72397cbf40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6d114c070 0x55c6d11e8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.767+0000 7f72397cbf40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c6d114bc60 0x55c6d11471d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.767+0000 7f72397cbf40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c6d11f1310 con 0x55c6d114c070 2024-04-26T23:09:36.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.767+0000 7f72397cbf40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c6d11f1030 con 0x55c6d114ae90 2024-04-26T23:09:36.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.767+0000 7f72397cbf40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c6d11f15f0 con 0x55c6d114bc60 2024-04-26T23:09:36.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.767+0000 7f722e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6d114c070 0x55c6d11e8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.767+0000 7f722e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6d114c070 0x55c6d11e8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42994/0 (socket says 172.21.15.40:42994) 2024-04-26T23:09:36.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.767+0000 7f722e000640 1 -- 172.21.15.40:0/127813407 learned_addr learned my addr 172.21.15.40:0/127813407 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:36.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.768+0000 7f7236a00640 1 --2- 172.21.15.40:0/127813407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c6d114ae90 0x55c6d11482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.768+0000 7f722e000640 1 -- 172.21.15.40:0/127813407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c6d114bc60 msgr2=0x55c6d11471d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:36.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.768+0000 7f722e000640 1 --2- 172.21.15.40:0/127813407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c6d114bc60 0x55c6d11471d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.768+0000 7f722e000640 1 -- 172.21.15.40:0/127813407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c6d114ae90 msgr2=0x55c6d11482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:36.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.768+0000 7f722e000640 1 --2- 172.21.15.40:0/127813407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c6d114ae90 0x55c6d11482e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.768+0000 7f722e000640 1 -- 172.21.15.40:0/127813407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6d1043b30 con 0x55c6d114c070 2024-04-26T23:09:36.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.768+0000 7f722e000640 1 --2- 172.21.15.40:0/127813407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6d114c070 0x55c6d11e8ba0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f7220009790 tx=0x7f7220009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffad52201924374b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:36.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.769+0000 7f7235600640 1 -- 172.21.15.40:0/127813407 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7220017070 con 0x55c6d114c070 2024-04-26T23:09:36.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.769+0000 7f7235600640 1 -- 172.21.15.40:0/127813407 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72200090d0 con 0x55c6d114c070 2024-04-26T23:09:36.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.769+0000 7f7235600640 1 -- 172.21.15.40:0/127813407 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7220006ae0 con 0x55c6d114c070 2024-04-26T23:09:36.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.769+0000 7f72397cbf40 1 -- 172.21.15.40:0/127813407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6d114c070 msgr2=0x55c6d11e8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.769+0000 7f72397cbf40 1 --2- 172.21.15.40:0/127813407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6d114c070 0x55c6d11e8ba0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f7220009790 tx=0x7f7220009c50 comp rx=0 tx=0).stop 2024-04-26T23:09:36.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.769+0000 7f72397cbf40 1 -- 172.21.15.40:0/127813407 shutdown_connections 2024-04-26T23:09:36.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.769+0000 7f72397cbf40 1 --2- 172.21.15.40:0/127813407 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c6d114bc60 0x55c6d11471d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.769+0000 7f72397cbf40 1 --2- 172.21.15.40:0/127813407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6d114c070 0x55c6d11e8ba0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.769+0000 7f72397cbf40 1 --2- 172.21.15.40:0/127813407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c6d114ae90 0x55c6d11482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.769+0000 7f72397cbf40 1 -- 172.21.15.40:0/127813407 >> 172.21.15.40:0/127813407 conn(0x55c6d1166df0 msgr2=0x55c6d11671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:36.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.769+0000 7f72397cbf40 1 -- 172.21.15.40:0/127813407 shutdown_connections 2024-04-26T23:09:36.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.770+0000 7f72397cbf40 1 -- 172.21.15.40:0/127813407 wait complete. 2024-04-26T23:09:36.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.770+0000 7f72397cbf40 1 Processor -- start 2024-04-26T23:09:36.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.770+0000 7f72397cbf40 1 -- start start 2024-04-26T23:09:36.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.771+0000 7f72397cbf40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c6d114ae90 0x55c6d1288540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.771+0000 7f72397cbf40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6d114bc60 0x55c6d1149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.771+0000 7f72397cbf40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c6d114c070 0x55c6d114a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.771+0000 7f72397cbf40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c6d11e90e0 con 0x55c6d114bc60 2024-04-26T23:09:36.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.771+0000 7f72397cbf40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c6d1146a80 con 0x55c6d114ae90 2024-04-26T23:09:36.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.771+0000 7f72397cbf40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c6d1119800 con 0x55c6d114c070 2024-04-26T23:09:36.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.772+0000 7f722e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6d114bc60 0x55c6d1149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.772+0000 7f722e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6d114bc60 0x55c6d1149b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42996/0 (socket says 172.21.15.40:42996) 2024-04-26T23:09:36.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.772+0000 7f722e000640 1 -- 172.21.15.40:0/1715953919 learned_addr learned my addr 172.21.15.40:0/1715953919 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:36.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.772+0000 7f722e000640 1 -- 172.21.15.40:0/1715953919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c6d114c070 msgr2=0x55c6d114a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.772+0000 7f722e000640 1 --2- 172.21.15.40:0/1715953919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c6d114c070 0x55c6d114a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.772+0000 7f722e000640 1 -- 172.21.15.40:0/1715953919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c6d114ae90 msgr2=0x55c6d1288540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.772+0000 7f722e000640 1 --2- 172.21.15.40:0/1715953919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c6d114ae90 0x55c6d1288540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.772+0000 7f722e000640 1 -- 172.21.15.40:0/1715953919 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6d1287cf0 con 0x55c6d114bc60 2024-04-26T23:09:36.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.772+0000 7f722e000640 1 --2- 172.21.15.40:0/1715953919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6d114bc60 0x55c6d1149b50 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f7220009fd0 tx=0x7f7220003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:36.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.773+0000 7f722fe00640 1 -- 172.21.15.40:0/1715953919 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7220020020 con 0x55c6d114bc60 2024-04-26T23:09:36.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.773+0000 7f722fe00640 1 -- 172.21.15.40:0/1715953919 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7220012610 con 0x55c6d114bc60 2024-04-26T23:09:36.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.773+0000 7f72397cbf40 1 -- 172.21.15.40:0/1715953919 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c6d1287fd0 con 0x55c6d114bc60 2024-04-26T23:09:36.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.773+0000 7f722fe00640 1 -- 172.21.15.40:0/1715953919 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7220017040 con 0x55c6d114bc60 2024-04-26T23:09:36.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.773+0000 7f72397cbf40 1 -- 172.21.15.40:0/1715953919 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6d11e3110 con 0x55c6d114bc60 2024-04-26T23:09:36.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.775+0000 7f722fe00640 1 -- 172.21.15.40:0/1715953919 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f722001c410 con 0x55c6d114bc60 2024-04-26T23:09:36.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.776+0000 7f722fe00640 1 --2- 172.21.15.40:0/1715953919 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f721406b2d0 0x7f721406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.776+0000 7f722fe00640 1 -- 172.21.15.40:0/1715953919 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f722008b2e0 con 0x55c6d114bc60 2024-04-26T23:09:36.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.776+0000 7f72397cbf40 1 --2- 172.21.15.40:0/1715953919 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x55c6d11e6050 0x55c6d1286b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.776+0000 7f72397cbf40 1 -- 172.21.15.40:0/1715953919 --> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] -- osd_op(unknown.0.0:1 15.12 15:4def72d4:::10000000003.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55c6d12870c0 con 0x55c6d11e6050 2024-04-26T23:09:36.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.777+0000 7f7236a00640 1 --2- 172.21.15.40:0/1715953919 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f721406b2d0 0x7f721406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.777+0000 7f7237400640 1 --2- 172.21.15.40:0/1715953919 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x55c6d11e6050 0x55c6d1286b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.777+0000 7f7237400640 1 --2- 172.21.15.40:0/1715953919 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x55c6d11e6050 0x55c6d1286b80 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:36.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.778+0000 7f7236a00640 1 --2- 172.21.15.40:0/1715953919 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f721406b2d0 0x7f721406d790 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f7228005700 tx=0x7f7228005690 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:36.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.779+0000 7f7237400640 1 -- 172.21.15.40:0/1715953919 <== osd.3 v2:172.21.15.40:6823/378872239 1 ==== osd_op_reply(1 10000000003.00000000 [rmxattr] v103'9 uv9 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7230054420 con 0x55c6d11e6050 2024-04-26T23:09:36.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.780+0000 7f72397cbf40 1 -- 172.21.15.40:0/1715953919 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x55c6d11e6050 msgr2=0x55c6d1286b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.780+0000 7f72397cbf40 1 --2- 172.21.15.40:0/1715953919 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x55c6d11e6050 0x55c6d1286b80 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.780+0000 7f72397cbf40 1 -- 172.21.15.40:0/1715953919 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f721406b2d0 msgr2=0x7f721406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.780+0000 7f72397cbf40 1 --2- 172.21.15.40:0/1715953919 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f721406b2d0 0x7f721406d790 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f7228005700 tx=0x7f7228005690 comp rx=0 tx=0).stop 2024-04-26T23:09:36.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.780+0000 7f72397cbf40 1 -- 172.21.15.40:0/1715953919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6d114bc60 msgr2=0x55c6d1149b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.780+0000 7f72397cbf40 1 --2- 172.21.15.40:0/1715953919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6d114bc60 0x55c6d1149b50 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f7220009fd0 tx=0x7f7220003020 comp rx=0 tx=0).stop 2024-04-26T23:09:36.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.780+0000 7f7237400640 1 -- 172.21.15.40:0/1715953919 reap_dead start 2024-04-26T23:09:36.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.780+0000 7f72397cbf40 1 -- 172.21.15.40:0/1715953919 shutdown_connections 2024-04-26T23:09:36.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.780+0000 7f72397cbf40 1 -- 172.21.15.40:0/1715953919 >> 172.21.15.40:0/1715953919 conn(0x55c6d1166df0 msgr2=0x55c6d11671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:36.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.780+0000 7f72397cbf40 1 -- 172.21.15.40:0/1715953919 shutdown_connections 2024-04-26T23:09:36.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.780+0000 7f72397cbf40 1 -- 172.21.15.40:0/1715953919 wait complete. 2024-04-26T23:09:36.785 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000004.00000000 parent 2024-04-26T23:09:36.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.883+0000 7f44b018cf40 1 Processor -- start 2024-04-26T23:09:36.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.883+0000 7f44b018cf40 1 -- start start 2024-04-26T23:09:36.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.884+0000 7f44b018cf40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645ac267c00 0x5645ac267210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.884+0000 7f44b018cf40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645ac2980a0 0x5645ac33f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.884+0000 7f44b018cf40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645ac298480 0x5645ac33fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.884+0000 7f44b018cf40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5645ac3451e0 con 0x5645ac267c00 2024-04-26T23:09:36.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.884+0000 7f44b018cf40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5645ac20fd20 con 0x5645ac298480 2024-04-26T23:09:36.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.884+0000 7f44b018cf40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5645ac2b1a10 con 0x5645ac2980a0 2024-04-26T23:09:36.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.884+0000 7f44ad400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645ac267c00 0x5645ac267210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.884+0000 7f44ad400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645ac267c00 0x5645ac267210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43028/0 (socket says 172.21.15.40:43028) 2024-04-26T23:09:36.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.884+0000 7f44ad400640 1 -- 172.21.15.40:0/1359414154 learned_addr learned my addr 172.21.15.40:0/1359414154 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:36.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.885+0000 7f44ad400640 1 -- 172.21.15.40:0/1359414154 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645ac2980a0 msgr2=0x5645ac33f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.885+0000 7f44ad400640 1 --2- 172.21.15.40:0/1359414154 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645ac2980a0 0x5645ac33f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.885+0000 7f44ad400640 1 -- 172.21.15.40:0/1359414154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645ac298480 msgr2=0x5645ac33fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.885+0000 7f44ad400640 1 --2- 172.21.15.40:0/1359414154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645ac298480 0x5645ac33fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.885+0000 7f44ad400640 1 -- 172.21.15.40:0/1359414154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645ac295170 con 0x5645ac267c00 2024-04-26T23:09:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.885+0000 7f44ad400640 1 --2- 172.21.15.40:0/1359414154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645ac267c00 0x5645ac267210 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f44a0025d20 tx=0x7f44a0051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1484142987256e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.885+0000 7f44a7400640 1 -- 172.21.15.40:0/1359414154 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0053880 con 0x5645ac267c00 2024-04-26T23:09:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.885+0000 7f44a7400640 1 -- 172.21.15.40:0/1359414154 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a0053a20 con 0x5645ac267c00 2024-04-26T23:09:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.885+0000 7f44a7400640 1 -- 172.21.15.40:0/1359414154 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0053e10 con 0x5645ac267c00 2024-04-26T23:09:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.886+0000 7f44b018cf40 1 -- 172.21.15.40:0/1359414154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645ac267c00 msgr2=0x5645ac267210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.886+0000 7f44b018cf40 1 --2- 172.21.15.40:0/1359414154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645ac267c00 0x5645ac267210 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f44a0025d20 tx=0x7f44a0051000 comp rx=0 tx=0).stop 2024-04-26T23:09:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.886+0000 7f44b018cf40 1 -- 172.21.15.40:0/1359414154 shutdown_connections 2024-04-26T23:09:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.886+0000 7f44b018cf40 1 --2- 172.21.15.40:0/1359414154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645ac298480 0x5645ac33fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.886+0000 7f44b018cf40 1 --2- 172.21.15.40:0/1359414154 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645ac2980a0 0x5645ac33f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.886+0000 7f44b018cf40 1 --2- 172.21.15.40:0/1359414154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645ac267c00 0x5645ac267210 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.886+0000 7f44b018cf40 1 -- 172.21.15.40:0/1359414154 >> 172.21.15.40:0/1359414154 conn(0x5645ac295c00 msgr2=0x5645ac33ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:36.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.886+0000 7f44b018cf40 1 -- 172.21.15.40:0/1359414154 shutdown_connections 2024-04-26T23:09:36.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.886+0000 7f44b018cf40 1 -- 172.21.15.40:0/1359414154 wait complete. 2024-04-26T23:09:36.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.887+0000 7f44b018cf40 1 Processor -- start 2024-04-26T23:09:36.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.887+0000 7f44b018cf40 1 -- start start 2024-04-26T23:09:36.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.888+0000 7f44b018cf40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645ac267c00 0x5645ac341820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.888+0000 7f44b018cf40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645ac2980a0 0x5645ac333170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.888+0000 7f44b018cf40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645ac298480 0x5645ac33e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.888+0000 7f44b018cf40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5645ac295410 con 0x5645ac298480 2024-04-26T23:09:36.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.888+0000 7f44b018cf40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5645ac3402f0 con 0x5645ac267c00 2024-04-26T23:09:36.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.888+0000 7f44b018cf40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5645ac26b6d0 con 0x5645ac2980a0 2024-04-26T23:09:36.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.888+0000 7f44ade00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645ac298480 0x5645ac33e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.888+0000 7f44ade00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645ac298480 0x5645ac33e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43036/0 (socket says 172.21.15.40:43036) 2024-04-26T23:09:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.888+0000 7f44ade00640 1 -- 172.21.15.40:0/3308029477 learned_addr learned my addr 172.21.15.40:0/3308029477 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.889+0000 7f44aca00640 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645ac2980a0 0x5645ac333170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.889+0000 7f44ad400640 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645ac267c00 0x5645ac341820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.889+0000 7f44ade00640 1 -- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645ac2980a0 msgr2=0x5645ac333170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.889+0000 7f44ade00640 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645ac2980a0 0x5645ac333170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.889+0000 7f44ade00640 1 -- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645ac267c00 msgr2=0x5645ac341820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.889+0000 7f44ade00640 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645ac267c00 0x5645ac341820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.889+0000 7f44ade00640 1 -- 172.21.15.40:0/3308029477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645ac33cf70 con 0x5645ac298480 2024-04-26T23:09:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.889+0000 7f44aca00640 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5645ac2980a0 0x5645ac333170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.889+0000 7f44ade00640 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645ac298480 0x5645ac33e010 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f44a800a0e0 tx=0x7f44a800a5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.889+0000 7f44a6000640 1 -- 172.21.15.40:0/3308029477 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8013430 con 0x5645ac298480 2024-04-26T23:09:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.889+0000 7f44a6000640 1 -- 172.21.15.40:0/3308029477 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a8018070 con 0x5645ac298480 2024-04-26T23:09:36.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.889+0000 7f44a6000640 1 -- 172.21.15.40:0/3308029477 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8013810 con 0x5645ac298480 2024-04-26T23:09:36.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.890+0000 7f44b018cf40 1 -- 172.21.15.40:0/3308029477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5645ac33d250 con 0x5645ac298480 2024-04-26T23:09:36.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.890+0000 7f44b018cf40 1 -- 172.21.15.40:0/3308029477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645ac33d810 con 0x5645ac298480 2024-04-26T23:09:36.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.892+0000 7f44a6000640 1 -- 172.21.15.40:0/3308029477 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f44a80139b0 con 0x5645ac298480 2024-04-26T23:09:36.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.893+0000 7f44a6000640 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f449006b2d0 0x7f449006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.893+0000 7f44a6000640 1 -- 172.21.15.40:0/3308029477 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f44a808abf0 con 0x5645ac298480 2024-04-26T23:09:36.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.893+0000 7f44b018cf40 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x5645ac33cab0 0x5645ac3e6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.893+0000 7f44b018cf40 1 -- 172.21.15.40:0/3308029477 --> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] -- osd_op(unknown.0.0:1 15.5 15:a367a3e8:::10000000004.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5645ac33ea00 con 0x5645ac33cab0 2024-04-26T23:09:36.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.894+0000 7f44ad400640 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f449006b2d0 0x7f449006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.894+0000 7f44aca00640 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x5645ac33cab0 0x5645ac3e6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.895+0000 7f44aca00640 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x5645ac33cab0 0x5645ac3e6890 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:36.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.895+0000 7f44ad400640 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f449006b2d0 0x7f449006d790 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f44a0055040 tx=0x7f44a0053000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.898+0000 7f44aca00640 1 -- 172.21.15.40:0/3308029477 <== osd.6 v2:172.21.15.103:6816/293480649 1 ==== osd_op_reply(1 10000000004.00000000 [rmxattr] v103'6 uv6 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f449c00e360 con 0x5645ac33cab0 2024-04-26T23:09:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.898+0000 7f44b018cf40 1 -- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x5645ac33cab0 msgr2=0x5645ac3e6890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.898+0000 7f44b018cf40 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x5645ac33cab0 0x5645ac3e6890 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.898+0000 7f44b018cf40 1 -- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f449006b2d0 msgr2=0x7f449006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.898+0000 7f44b018cf40 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f449006b2d0 0x7f449006d790 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f44a0055040 tx=0x7f44a0053000 comp rx=0 tx=0).stop 2024-04-26T23:09:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.898+0000 7f44b018cf40 1 -- 172.21.15.40:0/3308029477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645ac298480 msgr2=0x5645ac33e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.898+0000 7f44b018cf40 1 --2- 172.21.15.40:0/3308029477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645ac298480 0x5645ac33e010 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f44a800a0e0 tx=0x7f44a800a5a0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.898+0000 7f44ade00640 1 -- 172.21.15.40:0/3308029477 reap_dead start 2024-04-26T23:09:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.898+0000 7f44b018cf40 1 -- 172.21.15.40:0/3308029477 shutdown_connections 2024-04-26T23:09:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.898+0000 7f44b018cf40 1 -- 172.21.15.40:0/3308029477 >> 172.21.15.40:0/3308029477 conn(0x5645ac295c00 msgr2=0x5645ac33f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.898+0000 7f44b018cf40 1 -- 172.21.15.40:0/3308029477 shutdown_connections 2024-04-26T23:09:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.898+0000 7f44b018cf40 1 -- 172.21.15.40:0/3308029477 wait complete. 2024-04-26T23:09:36.904 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000005.00000000 parent 2024-04-26T23:09:36.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.995+0000 7f927011df40 1 Processor -- start 2024-04-26T23:09:36.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.995+0000 7f927011df40 1 -- start start 2024-04-26T23:09:36.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f927011df40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558216025e90 0x5582160c7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f927011df40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558216027070 0x558216040780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f927011df40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582160cd5a0 0x55821603df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:36.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f927011df40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5582160cc310 con 0x558216025e90 2024-04-26T23:09:36.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f927011df40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558215f9bd20 con 0x5582160cd5a0 2024-04-26T23:09:36.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f927011df40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5582160cc5f0 con 0x558216027070 2024-04-26T23:09:36.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f926c800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558216025e90 0x5582160c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:36.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f926c800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558216025e90 0x5582160c7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43056/0 (socket says 172.21.15.40:43056) 2024-04-26T23:09:36.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f926c800640 1 -- 172.21.15.40:0/2010561751 learned_addr learned my addr 172.21.15.40:0/2010561751 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:36.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f926c800640 1 -- 172.21.15.40:0/2010561751 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558216027070 msgr2=0x558216040780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:36.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f926c800640 1 --2- 172.21.15.40:0/2010561751 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558216027070 0x558216040780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f926c800640 1 -- 172.21.15.40:0/2010561751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582160cd5a0 msgr2=0x55821603df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:36.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f926c800640 1 --2- 172.21.15.40:0/2010561751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582160cd5a0 0x55821603df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:36.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.996+0000 7f926c800640 1 -- 172.21.15.40:0/2010561751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558216041160 con 0x558216025e90 2024-04-26T23:09:36.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.997+0000 7f926c800640 1 --2- 172.21.15.40:0/2010561751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558216025e90 0x5582160c7220 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f926000a490 tx=0x7f926000a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d441fd7f1fceadec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:36.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.997+0000 7f926aa00640 1 -- 172.21.15.40:0/2010561751 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926000f7b0 con 0x558216025e90 2024-04-26T23:09:36.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.997+0000 7f926aa00640 1 -- 172.21.15.40:0/2010561751 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9260015070 con 0x558216025e90 2024-04-26T23:09:36.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.997+0000 7f926aa00640 1 -- 172.21.15.40:0/2010561751 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926000fbb0 con 0x558216025e90 2024-04-26T23:09:36.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.997+0000 7f927011df40 1 -- 172.21.15.40:0/2010561751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558216025e90 msgr2=0x5582160c7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:36.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.997+0000 7f927011df40 1 --2- 172.21.15.40:0/2010561751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558216025e90 0x5582160c7220 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f926000a490 tx=0x7f926000a950 comp rx=0 tx=0).stop 2024-04-26T23:09:37.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.998+0000 7f927011df40 1 -- 172.21.15.40:0/2010561751 shutdown_connections 2024-04-26T23:09:37.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.998+0000 7f927011df40 1 --2- 172.21.15.40:0/2010561751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582160cd5a0 0x55821603df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.998+0000 7f927011df40 1 --2- 172.21.15.40:0/2010561751 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558216027070 0x558216040780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.998+0000 7f927011df40 1 --2- 172.21.15.40:0/2010561751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558216025e90 0x5582160c7220 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.998+0000 7f927011df40 1 -- 172.21.15.40:0/2010561751 >> 172.21.15.40:0/2010561751 conn(0x558216041df0 msgr2=0x558216026270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.998+0000 7f927011df40 1 -- 172.21.15.40:0/2010561751 shutdown_connections 2024-04-26T23:09:37.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.998+0000 7f927011df40 1 -- 172.21.15.40:0/2010561751 wait complete. 2024-04-26T23:09:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.998+0000 7f927011df40 1 Processor -- start 2024-04-26T23:09:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.999+0000 7f927011df40 1 -- start start 2024-04-26T23:09:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.999+0000 7f927011df40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558216025e90 0x5582160c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.999+0000 7f927011df40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558216027070 0x5582160c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.999+0000 7f927011df40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582160cd5a0 0x558216165e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.999+0000 7f927011df40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5582160cd0a0 con 0x558216027070 2024-04-26T23:09:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.999+0000 7f927011df40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5582160413d0 con 0x5582160cd5a0 2024-04-26T23:09:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:36.999+0000 7f927011df40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5582160259b0 con 0x558216025e90 2024-04-26T23:09:37.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.000+0000 7f926be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558216027070 0x5582160c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.000+0000 7f926be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558216027070 0x5582160c89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43060/0 (socket says 172.21.15.40:43060) 2024-04-26T23:09:37.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.000+0000 7f926be00640 1 -- 172.21.15.40:0/217565763 learned_addr learned my addr 172.21.15.40:0/217565763 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.000+0000 7f926c800640 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558216025e90 0x5582160c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.000+0000 7f926d200640 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582160cd5a0 0x558216165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.000+0000 7f926be00640 1 -- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558216025e90 msgr2=0x5582160c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.000+0000 7f926be00640 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558216025e90 0x5582160c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.000+0000 7f926be00640 1 -- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582160cd5a0 msgr2=0x558216165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.000+0000 7f926be00640 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582160cd5a0 0x558216165e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.000+0000 7f926be00640 1 -- 172.21.15.40:0/217565763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582160c8ff0 con 0x558216027070 2024-04-26T23:09:37.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.000+0000 7f926c800640 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558216025e90 0x5582160c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:37.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.001+0000 7f926be00640 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558216027070 0x5582160c89a0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f9258000c00 tx=0x7f92580058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.001+0000 7f9269600640 1 -- 172.21.15.40:0/217565763 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258062070 con 0x558216027070 2024-04-26T23:09:37.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.001+0000 7f9269600640 1 -- 172.21.15.40:0/217565763 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9258054b90 con 0x558216027070 2024-04-26T23:09:37.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.001+0000 7f9269600640 1 -- 172.21.15.40:0/217565763 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258067460 con 0x558216027070 2024-04-26T23:09:37.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.001+0000 7f927011df40 1 -- 172.21.15.40:0/217565763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5582160beb60 con 0x558216027070 2024-04-26T23:09:37.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.001+0000 7f927011df40 1 -- 172.21.15.40:0/217565763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558216022130 con 0x558216027070 2024-04-26T23:09:37.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.003+0000 7f9269600640 1 -- 172.21.15.40:0/217565763 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f925805e410 con 0x558216027070 2024-04-26T23:09:37.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.004+0000 7f9269600640 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f924c06b2d0 0x7f924c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.004+0000 7f9269600640 1 -- 172.21.15.40:0/217565763 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f92580cee50 con 0x558216027070 2024-04-26T23:09:37.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.004+0000 7f927011df40 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x5582160b5180 0x558216166890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.004+0000 7f927011df40 1 -- 172.21.15.40:0/217565763 --> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] -- osd_op(unknown.0.0:1 15.9 15:9258b67d:::10000000005.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55821615ab10 con 0x5582160b5180 2024-04-26T23:09:37.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.005+0000 7f926c800640 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f924c06b2d0 0x7f924c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.005+0000 7f926d200640 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x5582160b5180 0x558216166890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.006+0000 7f926d200640 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x5582160b5180 0x558216166890 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.006+0000 7f926c800640 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f924c06b2d0 0x7f924c06d790 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f926000a990 tx=0x7f9260007030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.009+0000 7f926d200640 1 -- 172.21.15.40:0/217565763 <== osd.6 v2:172.21.15.103:6816/293480649 1 ==== osd_op_reply(1 10000000005.00000000 [rmxattr] v103'6 uv6 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f92640097f0 con 0x5582160b5180 2024-04-26T23:09:37.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.009+0000 7f927011df40 1 -- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x5582160b5180 msgr2=0x558216166890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.009+0000 7f927011df40 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x5582160b5180 0x558216166890 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.009+0000 7f927011df40 1 -- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f924c06b2d0 msgr2=0x7f924c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.009+0000 7f927011df40 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f924c06b2d0 0x7f924c06d790 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f926000a990 tx=0x7f9260007030 comp rx=0 tx=0).stop 2024-04-26T23:09:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.009+0000 7f927011df40 1 -- 172.21.15.40:0/217565763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558216027070 msgr2=0x5582160c89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.009+0000 7f927011df40 1 --2- 172.21.15.40:0/217565763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558216027070 0x5582160c89a0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f9258000c00 tx=0x7f92580058b0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.009+0000 7f926d200640 1 -- 172.21.15.40:0/217565763 reap_dead start 2024-04-26T23:09:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.009+0000 7f927011df40 1 -- 172.21.15.40:0/217565763 shutdown_connections 2024-04-26T23:09:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.009+0000 7f927011df40 1 -- 172.21.15.40:0/217565763 >> 172.21.15.40:0/217565763 conn(0x558216041df0 msgr2=0x5582160c51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.009+0000 7f927011df40 1 -- 172.21.15.40:0/217565763 shutdown_connections 2024-04-26T23:09:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.009+0000 7f927011df40 1 -- 172.21.15.40:0/217565763 wait complete. 2024-04-26T23:09:37.014 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000006.00000000 parent 2024-04-26T23:09:37.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.109+0000 7f0ae7f66f40 1 Processor -- start 2024-04-26T23:09:37.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.109+0000 7f0ae7f66f40 1 -- start start 2024-04-26T23:09:37.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.110+0000 7f0ae7f66f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564955b3ce90 0x564955bda920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.110+0000 7f0ae7f66f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564955b3e070 0x564955b3bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.110+0000 7f0ae7f66f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564955b57590 0x564955b3c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.110+0000 7f0ae7f66f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564955b55ef0 con 0x564955b57590 2024-04-26T23:09:37.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.110+0000 7f0ae7f66f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564955ab2d20 con 0x564955b3e070 2024-04-26T23:09:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.110+0000 7f0ae7f66f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564955be3030 con 0x564955b3ce90 2024-04-26T23:09:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.110+0000 7f0ae5c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564955b57590 0x564955b3c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.111+0000 7f0ae5c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564955b57590 0x564955b3c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43070/0 (socket says 172.21.15.40:43070) 2024-04-26T23:09:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.111+0000 7f0ae5c00640 1 -- 172.21.15.40:0/2595388201 learned_addr learned my addr 172.21.15.40:0/2595388201 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.111+0000 7f0ae5200640 1 --2- 172.21.15.40:0/2595388201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564955b3ce90 0x564955bda920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.111+0000 7f0adc800640 1 --2- 172.21.15.40:0/2595388201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564955b3e070 0x564955b3bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.111+0000 7f0ae5c00640 1 -- 172.21.15.40:0/2595388201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564955b3ce90 msgr2=0x564955bda920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.111+0000 7f0ae5c00640 1 --2- 172.21.15.40:0/2595388201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564955b3ce90 0x564955bda920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.111+0000 7f0ae5c00640 1 -- 172.21.15.40:0/2595388201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564955b3e070 msgr2=0x564955b3bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.111+0000 7f0ae5c00640 1 --2- 172.21.15.40:0/2595388201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564955b3e070 0x564955b3bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.111+0000 7f0ae5c00640 1 -- 172.21.15.40:0/2595388201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564955b38c00 con 0x564955b57590 2024-04-26T23:09:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.111+0000 7f0ae5c00640 1 --2- 172.21.15.40:0/2595388201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564955b57590 0x564955b3c090 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009540 tx=0x7f0ae0009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20efffd4d54a5602 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.112+0000 7f0adf400640 1 -- 172.21.15.40:0/2595388201 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0017070 con 0x564955b57590 2024-04-26T23:09:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.112+0000 7f0adf400640 1 -- 172.21.15.40:0/2595388201 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae0006600 con 0x564955b57590 2024-04-26T23:09:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.112+0000 7f0adf400640 1 -- 172.21.15.40:0/2595388201 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae00069f0 con 0x564955b57590 2024-04-26T23:09:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.112+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2595388201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564955b57590 msgr2=0x564955b3c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.112+0000 7f0ae7f66f40 1 --2- 172.21.15.40:0/2595388201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564955b57590 0x564955b3c090 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009540 tx=0x7f0ae0009a00 comp rx=0 tx=0).stop 2024-04-26T23:09:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.112+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2595388201 shutdown_connections 2024-04-26T23:09:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.112+0000 7f0ae7f66f40 1 --2- 172.21.15.40:0/2595388201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564955b57590 0x564955b3c090 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.112+0000 7f0ae7f66f40 1 --2- 172.21.15.40:0/2595388201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564955b3e070 0x564955b3bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.112+0000 7f0ae7f66f40 1 --2- 172.21.15.40:0/2595388201 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564955b3ce90 0x564955bda920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.112+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2595388201 >> 172.21.15.40:0/2595388201 conn(0x564955b58df0 msgr2=0x564955b38fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.112+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2595388201 shutdown_connections 2024-04-26T23:09:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.112+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2595388201 wait complete. 2024-04-26T23:09:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.113+0000 7f0ae7f66f40 1 Processor -- start 2024-04-26T23:09:37.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.113+0000 7f0ae7f66f40 1 -- start start 2024-04-26T23:09:37.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.114+0000 7f0ae7f66f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564955b3ce90 0x564955bdab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.114+0000 7f0ae7f66f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564955b3e070 0x564955bdb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.114+0000 7f0ae7f66f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564955b57590 0x564955bdbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.114+0000 7f0ae7f66f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564955b3a560 con 0x564955b3e070 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.114+0000 7f0ae7f66f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564955b384b0 con 0x564955b57590 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.114+0000 7f0ae7f66f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564955b3a0c0 con 0x564955b3ce90 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.114+0000 7f0adc800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564955b3e070 0x564955bdb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.114+0000 7f0adc800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564955b3e070 0x564955bdb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43080/0 (socket says 172.21.15.40:43080) 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.114+0000 7f0adc800640 1 -- 172.21.15.40:0/2425666767 learned_addr learned my addr 172.21.15.40:0/2425666767 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.115+0000 7f0ae5200640 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564955b3ce90 0x564955bdab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.115+0000 7f0adc800640 1 -- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564955b3ce90 msgr2=0x564955bdab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.115+0000 7f0ae5c00640 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564955b57590 0x564955bdbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.115+0000 7f0adc800640 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564955b3ce90 0x564955bdab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.115+0000 7f0adc800640 1 -- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564955b57590 msgr2=0x564955bdbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.115+0000 7f0adc800640 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564955b57590 0x564955bdbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.115+0000 7f0adc800640 1 -- 172.21.15.40:0/2425666767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564955be5710 con 0x564955b3e070 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.115+0000 7f0adc800640 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564955b3e070 0x564955bdb660 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00011e0 tx=0x7f0ad000b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.115+0000 7f0ade000640 1 -- 172.21.15.40:0/2425666767 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0014020 con 0x564955b3e070 2024-04-26T23:09:37.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.115+0000 7f0ade000640 1 -- 172.21.15.40:0/2425666767 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad000be40 con 0x564955b3e070 2024-04-26T23:09:37.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.115+0000 7f0ade000640 1 -- 172.21.15.40:0/2425666767 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0008030 con 0x564955b3e070 2024-04-26T23:09:37.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.115+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2425666767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564955bd53b0 con 0x564955b3e070 2024-04-26T23:09:37.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.116+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2425666767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564955bd3b50 con 0x564955b3e070 2024-04-26T23:09:37.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.118+0000 7f0ade000640 1 -- 172.21.15.40:0/2425666767 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ad0017020 con 0x564955b3e070 2024-04-26T23:09:37.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.118+0000 7f0ade000640 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ac406b2d0 0x7f0ac406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.119+0000 7f0ade000640 1 -- 172.21.15.40:0/2425666767 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f0ad008aa80 con 0x564955b3e070 2024-04-26T23:09:37.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.119+0000 7f0ae7f66f40 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x564955bdb050 0x564955bd21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.119+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2425666767 --> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] -- osd_op(unknown.0.0:1 15.5 15:a5feb6f3:::10000000006.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x564955bd3500 con 0x564955bdb050 2024-04-26T23:09:37.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.119+0000 7f0ae5200640 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ac406b2d0 0x7f0ac406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.120+0000 7f0ae5c00640 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x564955bdb050 0x564955bd21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.120+0000 7f0ae5200640 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ac406b2d0 0x7f0ac406d790 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4057000 tx=0x7f0ad405a040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.121+0000 7f0ae5c00640 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x564955bdb050 0x564955bd21f0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.124+0000 7f0ae5c00640 1 -- 172.21.15.40:0/2425666767 <== osd.6 v2:172.21.15.103:6816/293480649 1 ==== osd_op_reply(1 10000000006.00000000 [rmxattr] v103'7 uv7 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0ae00068f0 con 0x564955bdb050 2024-04-26T23:09:37.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.124+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x564955bdb050 msgr2=0x564955bd21f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.124+0000 7f0ae7f66f40 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x564955bdb050 0x564955bd21f0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.124+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ac406b2d0 msgr2=0x7f0ac406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.124+0000 7f0ae7f66f40 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0ac406b2d0 0x7f0ac406d790 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4057000 tx=0x7f0ad405a040 comp rx=0 tx=0).stop 2024-04-26T23:09:37.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.124+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2425666767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564955b3e070 msgr2=0x564955bdb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.124+0000 7f0ae7f66f40 1 --2- 172.21.15.40:0/2425666767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564955b3e070 0x564955bdb660 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00011e0 tx=0x7f0ad000b6a0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.124+0000 7f0ae5c00640 1 -- 172.21.15.40:0/2425666767 reap_dead start 2024-04-26T23:09:37.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.124+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2425666767 shutdown_connections 2024-04-26T23:09:37.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.124+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2425666767 >> 172.21.15.40:0/2425666767 conn(0x564955b58df0 msgr2=0x564955bd41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.124+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2425666767 shutdown_connections 2024-04-26T23:09:37.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.124+0000 7f0ae7f66f40 1 -- 172.21.15.40:0/2425666767 wait complete. 2024-04-26T23:09:37.129 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmxattr 10000000007.00000000 parent 2024-04-26T23:09:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.219+0000 7f3ddd791f40 1 Processor -- start 2024-04-26T23:09:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.219+0000 7f3ddd791f40 1 -- start start 2024-04-26T23:09:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.220+0000 7f3ddd791f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dc00022a0 0x562a6be96200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.220+0000 7f3ddd791f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6be99030 0x562a6be9b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.220+0000 7f3ddd791f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6be9b6d0 0x562a6bdc2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.220+0000 7f3ddd791f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562a6bd6bd20 con 0x562a6be99030 2024-04-26T23:09:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.220+0000 7f3ddd791f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562a6bd6fd60 con 0x562a6be9b6d0 2024-04-26T23:09:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.220+0000 7f3ddd791f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x562a6be9c310 con 0x7f3dc00022a0 2024-04-26T23:09:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.220+0000 7f3dd3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6be99030 0x562a6be9b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.220+0000 7f3dd3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6be99030 0x562a6be9b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43090/0 (socket says 172.21.15.40:43090) 2024-04-26T23:09:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.220+0000 7f3dd3e00640 1 -- 172.21.15.40:0/928009542 learned_addr learned my addr 172.21.15.40:0/928009542 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.221+0000 7f3dd3e00640 1 -- 172.21.15.40:0/928009542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dc00022a0 msgr2=0x562a6be96200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.221+0000 7f3dd3e00640 1 --2- 172.21.15.40:0/928009542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dc00022a0 0x562a6be96200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.221+0000 7f3dd3e00640 1 -- 172.21.15.40:0/928009542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6be9b6d0 msgr2=0x562a6bdc2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.221+0000 7f3ddb400640 1 --2- 172.21.15.40:0/928009542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6be9b6d0 0x562a6bdc2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.221+0000 7f3dd3e00640 1 --2- 172.21.15.40:0/928009542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6be9b6d0 0x562a6bdc2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.221+0000 7f3dd3e00640 1 -- 172.21.15.40:0/928009542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a6be9e0b0 con 0x562a6be99030 2024-04-26T23:09:37.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.221+0000 7f3dd3e00640 1 --2- 172.21.15.40:0/928009542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6be99030 0x562a6be9b190 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400c300 tx=0x7f3dc400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b7d9254344ba70 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.221+0000 7f3dd9600640 1 -- 172.21.15.40:0/928009542 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc400e540 con 0x562a6be99030 2024-04-26T23:09:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.221+0000 7f3dd9600640 1 -- 172.21.15.40:0/928009542 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc400e6e0 con 0x562a6be99030 2024-04-26T23:09:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.221+0000 7f3dd9600640 1 -- 172.21.15.40:0/928009542 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc400e8d0 con 0x562a6be99030 2024-04-26T23:09:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.222+0000 7f3ddd791f40 1 -- 172.21.15.40:0/928009542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6be99030 msgr2=0x562a6be9b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.222+0000 7f3ddd791f40 1 --2- 172.21.15.40:0/928009542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6be99030 0x562a6be9b190 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400c300 tx=0x7f3dc400c7c0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.222+0000 7f3ddd791f40 1 -- 172.21.15.40:0/928009542 shutdown_connections 2024-04-26T23:09:37.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.222+0000 7f3ddd791f40 1 --2- 172.21.15.40:0/928009542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6be9b6d0 0x562a6bdc2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.222+0000 7f3ddd791f40 1 --2- 172.21.15.40:0/928009542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6be99030 0x562a6be9b190 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.222+0000 7f3ddd791f40 1 --2- 172.21.15.40:0/928009542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3dc00022a0 0x562a6be96200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.222+0000 7f3ddd791f40 1 -- 172.21.15.40:0/928009542 >> 172.21.15.40:0/928009542 conn(0x562a6be11df0 msgr2=0x562a6be121d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.222+0000 7f3ddd791f40 1 -- 172.21.15.40:0/928009542 shutdown_connections 2024-04-26T23:09:37.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.222+0000 7f3ddd791f40 1 -- 172.21.15.40:0/928009542 wait complete. 2024-04-26T23:09:37.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.223+0000 7f3ddd791f40 1 Processor -- start 2024-04-26T23:09:37.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.223+0000 7f3ddd791f40 1 -- start start 2024-04-26T23:09:37.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.224+0000 7f3ddd791f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a6be99030 0x562a6be94520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.224+0000 7f3ddd791f40 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6be9b6d0 0x562a6bdf3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.224+0000 7f3ddd791f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dc00022a0 0x562a6be8e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.224+0000 7f3ddd791f40 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562a6be9e640 con 0x562a6be9b6d0 2024-04-26T23:09:37.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.224+0000 7f3ddd791f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562a6bea3b30 con 0x7f3dc00022a0 2024-04-26T23:09:37.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.224+0000 7f3ddd791f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x562a6bea41b0 con 0x562a6be99030 2024-04-26T23:09:37.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.224+0000 7f3dd3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6be9b6d0 0x562a6bdf3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.224+0000 7f3dd3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6be9b6d0 0x562a6bdf3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43102/0 (socket says 172.21.15.40:43102) 2024-04-26T23:09:37.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.224+0000 7f3dd3e00640 1 -- 172.21.15.40:0/1013106726 learned_addr learned my addr 172.21.15.40:0/1013106726 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.224+0000 7f3ddaa00640 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a6be99030 0x562a6be94520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.224+0000 7f3ddb400640 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dc00022a0 0x562a6be8e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.224+0000 7f3dd3e00640 1 -- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a6be99030 msgr2=0x562a6be94520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.224+0000 7f3dd3e00640 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562a6be99030 0x562a6be94520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.225+0000 7f3dd3e00640 1 -- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dc00022a0 msgr2=0x562a6be8e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.225+0000 7f3dd3e00640 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3dc00022a0 0x562a6be8e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.225+0000 7f3dd3e00640 1 -- 172.21.15.40:0/1013106726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a6be95b70 con 0x562a6be9b6d0 2024-04-26T23:09:37.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.225+0000 7f3dd3e00640 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6be9b6d0 0x562a6bdf3380 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x562a6be11510 tx=0x7f3dc4001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.225+0000 7f3dd3400640 1 -- 172.21.15.40:0/1013106726 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc400e540 con 0x562a6be9b6d0 2024-04-26T23:09:37.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.225+0000 7f3dd3400640 1 -- 172.21.15.40:0/1013106726 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc4022020 con 0x562a6be9b6d0 2024-04-26T23:09:37.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.225+0000 7f3ddd791f40 1 -- 172.21.15.40:0/1013106726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562a6be95e50 con 0x562a6be9b6d0 2024-04-26T23:09:37.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.225+0000 7f3dd3400640 1 -- 172.21.15.40:0/1013106726 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc400ee60 con 0x562a6be9b6d0 2024-04-26T23:09:37.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.225+0000 7f3ddd791f40 1 -- 172.21.15.40:0/1013106726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a6bdf2e60 con 0x562a6be9b6d0 2024-04-26T23:09:37.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.227+0000 7f3dd3400640 1 -- 172.21.15.40:0/1013106726 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3dc4018070 con 0x562a6be9b6d0 2024-04-26T23:09:37.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.228+0000 7f3dd3400640 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3db806b2d0 0x7f3db806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.228+0000 7f3dd3400640 1 -- 172.21.15.40:0/1013106726 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f3dc408cde0 con 0x562a6be9b6d0 2024-04-26T23:09:37.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.229+0000 7f3ddd791f40 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x562a6bdc2a20 0x562a6bea1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.229+0000 7f3ddd791f40 1 -- 172.21.15.40:0/1013106726 --> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] -- osd_op(unknown.0.0:1 15.1d 15:bbf5850a:::10000000007.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x562a6bea1ea0 con 0x562a6bdc2a20 2024-04-26T23:09:37.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.229+0000 7f3ddaa00640 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3db806b2d0 0x7f3db806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.229+0000 7f3ddb400640 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x562a6bdc2a20 0x562a6bea1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.230+0000 7f3ddb400640 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x562a6bdc2a20 0x562a6bea1960 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.231+0000 7f3ddaa00640 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3db806b2d0 0x7f3db806d790 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc056000 tx=0x7f3dcc050000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.234+0000 7f3ddb400640 1 -- 172.21.15.40:0/1013106726 <== osd.4 v2:172.21.15.103:6803/801343304 1 ==== osd_op_reply(1 10000000007.00000000 [rmxattr] v103'7 uv7 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f3dd4011040 con 0x562a6bdc2a20 2024-04-26T23:09:37.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.234+0000 7f3ddd791f40 1 -- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x562a6bdc2a20 msgr2=0x562a6bea1960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.234+0000 7f3ddd791f40 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x562a6bdc2a20 0x562a6bea1960 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.234+0000 7f3ddd791f40 1 -- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3db806b2d0 msgr2=0x7f3db806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.234+0000 7f3ddd791f40 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3db806b2d0 0x7f3db806d790 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc056000 tx=0x7f3dcc050000 comp rx=0 tx=0).stop 2024-04-26T23:09:37.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.234+0000 7f3ddd791f40 1 -- 172.21.15.40:0/1013106726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6be9b6d0 msgr2=0x562a6bdf3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.234+0000 7f3ddd791f40 1 --2- 172.21.15.40:0/1013106726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6be9b6d0 0x562a6bdf3380 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x562a6be11510 tx=0x7f3dc4001190 comp rx=0 tx=0).stop 2024-04-26T23:09:37.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.234+0000 7f3ddb400640 1 -- 172.21.15.40:0/1013106726 reap_dead start 2024-04-26T23:09:37.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.234+0000 7f3ddd791f40 1 -- 172.21.15.40:0/1013106726 shutdown_connections 2024-04-26T23:09:37.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.234+0000 7f3ddd791f40 1 -- 172.21.15.40:0/1013106726 >> 172.21.15.40:0/1013106726 conn(0x562a6be11df0 msgr2=0x562a6be121d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.235+0000 7f3ddd791f40 1 -- 172.21.15.40:0/1013106726 shutdown_connections 2024-04-26T23:09:37.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.235+0000 7f3ddd791f40 1 -- 172.21.15.40:0/1013106726 wait complete. 2024-04-26T23:09:37.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:37.240 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub start /d1/d2/d3 recursive,force 2024-04-26T23:09:37.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.443+0000 7f9066e00640 1 Processor -- start 2024-04-26T23:09:37.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.443+0000 7f9066e00640 1 -- start start 2024-04-26T23:09:37.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.444+0000 7f9066e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90600d3b60 0x7f90600d0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.445+0000 7f9066e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90600b3d90 0x7f90600b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.445+0000 7f9066e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90600d0d50 con 0x7f906015f420 2024-04-26T23:09:37.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.445+0000 7f9066e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f90600d0a70 con 0x7f90600b3d90 2024-04-26T23:09:37.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.445+0000 7f9066e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f90600b1bf0 con 0x7f90600d3b60 2024-04-26T23:09:37.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.445+0000 7f9066400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90600b3d90 0x7f90600b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.445+0000 7f9066400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90600b3d90 0x7f90600b26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43118/0 (socket says 172.21.15.40:43118) 2024-04-26T23:09:37.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.445+0000 7f9066400640 1 -- 172.21.15.40:0/3981636804 learned_addr learned my addr 172.21.15.40:0/3981636804 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.445+0000 7f9065000640 1 --2- 172.21.15.40:0/3981636804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90600d3b60 0x7f90600d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.445+0000 7f9066400640 1 -- 172.21.15.40:0/3981636804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90600d3b60 msgr2=0x7f90600d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.445+0000 7f9066400640 1 --2- 172.21.15.40:0/3981636804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90600d3b60 0x7f90600d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.446+0000 7f9066400640 1 -- 172.21.15.40:0/3981636804 >> v1:172.21.15.103:6789/0 conn(0x7f906015f420 legacy=0x7f90600d3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.446+0000 7f9066400640 1 -- 172.21.15.40:0/3981636804 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90600b1ed0 con 0x7f90600b3d90 2024-04-26T23:09:37.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.447+0000 7f9066400640 1 --2- 172.21.15.40:0/3981636804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90600b3d90 0x7f90600b26d0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f9058009e50 tx=0x7f905802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1cee366e442456 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.447+0000 7f905fe00640 1 -- 172.21.15.40:0/3981636804 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058036070 con 0x7f90600b3d90 2024-04-26T23:09:37.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.447+0000 7f905fe00640 1 -- 172.21.15.40:0/3981636804 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9058031600 con 0x7f90600b3d90 2024-04-26T23:09:37.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.447+0000 7f905fe00640 1 -- 172.21.15.40:0/3981636804 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90580319d0 con 0x7f90600b3d90 2024-04-26T23:09:37.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.448+0000 7f9066e00640 1 -- 172.21.15.40:0/3981636804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90600b3d90 msgr2=0x7f90600b26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.448+0000 7f9066e00640 1 --2- 172.21.15.40:0/3981636804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90600b3d90 0x7f90600b26d0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f9058009e50 tx=0x7f905802a680 comp rx=0 tx=0).stop 2024-04-26T23:09:37.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.448+0000 7f9066e00640 1 -- 172.21.15.40:0/3981636804 shutdown_connections 2024-04-26T23:09:37.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.448+0000 7f9066e00640 1 --2- 172.21.15.40:0/3981636804 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90600d3b60 0x7f90600d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.448+0000 7f9066e00640 1 --2- 172.21.15.40:0/3981636804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90600b3d90 0x7f90600b26d0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.448+0000 7f9066e00640 1 -- 172.21.15.40:0/3981636804 >> 172.21.15.40:0/3981636804 conn(0x7f906009e1d0 msgr2=0x7f90600d4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.448+0000 7f9066e00640 1 -- 172.21.15.40:0/3981636804 shutdown_connections 2024-04-26T23:09:37.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.448+0000 7f9066e00640 1 -- 172.21.15.40:0/3981636804 wait complete. 2024-04-26T23:09:37.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.450+0000 7f9066e00640 1 Processor -- start 2024-04-26T23:09:37.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.450+0000 7f9066e00640 1 -- start start 2024-04-26T23:09:37.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.451+0000 7f9066e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90600b3d90 0x7f90600b9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.451+0000 7f9066e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90600d3b60 0x7f9060151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.451+0000 7f9066e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f906015f420 0x7f9060151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.451+0000 7f9066e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f90600b2c10 con 0x7f90600b3d90 2024-04-26T23:09:37.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.451+0000 7f9066e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f90600d2e50 con 0x7f906015f420 2024-04-26T23:09:37.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.451+0000 7f9066e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f90600b2d90 con 0x7f90600d3b60 2024-04-26T23:09:37.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.451+0000 7f9065a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90600b3d90 0x7f90600b9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.451+0000 7f9065a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90600b3d90 0x7f90600b9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43132/0 (socket says 172.21.15.40:43132) 2024-04-26T23:09:37.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.451+0000 7f9065a00640 1 -- 172.21.15.40:0/1848521948 learned_addr learned my addr 172.21.15.40:0/1848521948 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f9065a00640 1 -- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90600d3b60 msgr2=0x7f9060151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f9066400640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f906015f420 0x7f9060151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f9065000640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90600d3b60 0x7f9060151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f9065a00640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90600d3b60 0x7f9060151100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f9065a00640 1 -- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f906015f420 msgr2=0x7f9060151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f9065a00640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f906015f420 0x7f9060151640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f9065a00640 1 -- 172.21.15.40:0/1848521948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90600cc7e0 con 0x7f90600b3d90 2024-04-26T23:09:37.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f9066400640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f906015f420 0x7f9060151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:37.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f9065a00640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90600b3d90 0x7f90600b9e30 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f905400c400 tx=0x7f905400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f905d600640 1 -- 172.21.15.40:0/1848521948 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9054017070 con 0x7f90600b3d90 2024-04-26T23:09:37.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f905d600640 1 -- 172.21.15.40:0/1848521948 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90540052d0 con 0x7f90600b3d90 2024-04-26T23:09:37.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f9066e00640 1 -- 172.21.15.40:0/1848521948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90600cbf40 con 0x7f90600b3d90 2024-04-26T23:09:37.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f905d600640 1 -- 172.21.15.40:0/1848521948 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9054003830 con 0x7f90600b3d90 2024-04-26T23:09:37.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.452+0000 7f9066e00640 1 -- 172.21.15.40:0/1848521948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90600cc500 con 0x7f90600b3d90 2024-04-26T23:09:37.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.455+0000 7f905d600640 1 -- 172.21.15.40:0/1848521948 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f905401a070 con 0x7f90600b3d90 2024-04-26T23:09:37.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.456+0000 7f905d600640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f904006c8f0 0x7f904006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.456+0000 7f905d600640 1 -- 172.21.15.40:0/1848521948 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f905408aff0 con 0x7f90600b3d90 2024-04-26T23:09:37.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.456+0000 7f906b395740 1 Processor -- start 2024-04-26T23:09:37.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.456+0000 7f9065000640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f904006c8f0 0x7f904006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.456+0000 7f906b395740 1 -- start start 2024-04-26T23:09:37.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.457+0000 7f906b395740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb6626d010 0x55fb6621f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.457+0000 7f906b395740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 0x55fb6624f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.457+0000 7f906b395740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6624f9f0 0x55fb66224800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.457+0000 7f906b395740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fb66189b80 con 0x55fb6624f9f0 2024-04-26T23:09:37.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.457+0000 7f906b395740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fb66126420 con 0x55fb6626d010 2024-04-26T23:09:37.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.457+0000 7f906b395740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55fb66185b40 con 0x55fb66250130 2024-04-26T23:09:37.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.458+0000 7f9065000640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f904006c8f0 0x7f904006edb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f906006d740 tx=0x7f904c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.458+0000 7f9066400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6624f9f0 0x55fb66224800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.458+0000 7f9066400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6624f9f0 0x55fb66224800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43134/0 (socket says 172.21.15.40:43134) 2024-04-26T23:09:37.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.458+0000 7f9066400640 1 -- 172.21.15.40:0/1329398310 learned_addr learned my addr 172.21.15.40:0/1329398310 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.458+0000 7f9065a00640 1 --2- 172.21.15.40:0/1329398310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb6626d010 0x55fb6621f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.458+0000 7f9065000640 1 --2- 172.21.15.40:0/1329398310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 0x55fb6624f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.458+0000 7f9066400640 1 -- 172.21.15.40:0/1329398310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 msgr2=0x55fb6624f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.458+0000 7f9066400640 1 --2- 172.21.15.40:0/1329398310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 0x55fb6624f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.458+0000 7f9066400640 1 -- 172.21.15.40:0/1329398310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb6626d010 msgr2=0x55fb6621f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.458+0000 7f9066400640 1 --2- 172.21.15.40:0/1329398310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb6626d010 0x55fb6621f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.458+0000 7f9066400640 1 -- 172.21.15.40:0/1329398310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb65f63a50 con 0x55fb6624f9f0 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.459+0000 7f9066400640 1 --2- 172.21.15.40:0/1329398310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6624f9f0 0x55fb66224800 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f905802ac70 tx=0x7f905802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b271b8e9c68e450 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.459+0000 7f9052000640 1 -- 172.21.15.40:0/1329398310 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058036040 con 0x55fb6624f9f0 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.459+0000 7f9052000640 1 -- 172.21.15.40:0/1329398310 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9058042070 con 0x55fb6624f9f0 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.459+0000 7f9052000640 1 -- 172.21.15.40:0/1329398310 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905803c870 con 0x55fb6624f9f0 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.460+0000 7f906b395740 1 -- 172.21.15.40:0/1329398310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6624f9f0 msgr2=0x55fb66224800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.460+0000 7f906b395740 1 --2- 172.21.15.40:0/1329398310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6624f9f0 0x55fb66224800 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f905802ac70 tx=0x7f905802e000 comp rx=0 tx=0).stop 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.460+0000 7f906b395740 1 -- 172.21.15.40:0/1329398310 shutdown_connections 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.460+0000 7f906b395740 1 --2- 172.21.15.40:0/1329398310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6624f9f0 0x55fb66224800 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.460+0000 7f906b395740 1 --2- 172.21.15.40:0/1329398310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 0x55fb6624f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.460+0000 7f906b395740 1 --2- 172.21.15.40:0/1329398310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb6626d010 0x55fb6621f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.460+0000 7f906b395740 1 -- 172.21.15.40:0/1329398310 >> 172.21.15.40:0/1329398310 conn(0x55fb66221c10 msgr2=0x55fb6626a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.460+0000 7f906b395740 1 -- 172.21.15.40:0/1329398310 shutdown_connections 2024-04-26T23:09:37.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.460+0000 7f906b395740 1 -- 172.21.15.40:0/1329398310 wait complete. 2024-04-26T23:09:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.461+0000 7f906b395740 1 -- start start 2024-04-26T23:09:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.461+0000 7f906b395740 1 Processor -- start 2024-04-26T23:09:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.462+0000 7f906b395740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 0x55fb6623fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.462+0000 7f906b395740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb6626d010 0x55fb662516e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.462+0000 7f906b395740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb66251c20 0x55fb6626a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.462+0000 7f906b395740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fb66223dc0 con 0x55fb66251c20 2024-04-26T23:09:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.462+0000 7f906b395740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fb66111370 con 0x55fb6626d010 2024-04-26T23:09:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.462+0000 7f906b395740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55fb662231b0 con 0x55fb66250130 2024-04-26T23:09:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.462+0000 7f906b395740 10 client.0 _ll_register_callbacks cb 0x55fb66180890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.462+0000 7f9066400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb66251c20 0x55fb6626a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.462+0000 7f9066400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb66251c20 0x55fb6626a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43142/0 (socket says 172.21.15.40:43142) 2024-04-26T23:09:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.462+0000 7f9066400640 1 -- 172.21.15.40:0/1711607692 learned_addr learned my addr 172.21.15.40:0/1711607692 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.463+0000 7f9065000640 1 --2- 172.21.15.40:0/1711607692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb6626d010 0x55fb662516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.463+0000 7f9065a00640 1 --2- 172.21.15.40:0/1711607692 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 0x55fb6623fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.463+0000 7f9066400640 1 -- 172.21.15.40:0/1711607692 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 msgr2=0x55fb6623fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.463+0000 7f9066400640 1 --2- 172.21.15.40:0/1711607692 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 0x55fb6623fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.463+0000 7f9066400640 1 -- 172.21.15.40:0/1711607692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb6626d010 msgr2=0x55fb662516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.463+0000 7f9066400640 1 --2- 172.21.15.40:0/1711607692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb6626d010 0x55fb662516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.463+0000 7f9066400640 1 -- 172.21.15.40:0/1711607692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb65e5b3e0 con 0x55fb66251c20 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.463+0000 7f9066400640 1 --2- 172.21.15.40:0/1711607692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb66251c20 0x55fb6626a4b0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f905803e000 tx=0x7f9058004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b659e172345b8ad6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.463+0000 7f9046a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.464+0000 7f9046a00640 1 -- 172.21.15.40:0/1711607692 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905804a070 con 0x55fb66251c20 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.464+0000 7f906b395740 1 -- 172.21.15.40:0/1711607692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9058042070 con 0x55fb66251c20 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.464+0000 7f9046a00640 1 -- 172.21.15.40:0/1711607692 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f9058004450 con 0x55fb66251c20 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.464+0000 7f9046a00640 1 -- 172.21.15.40:0/1711607692 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f9058056050 con 0x55fb66251c20 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.464+0000 7f906b395740 10 client.9306 fetch_fsmap learned FSMap version 119 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.464+0000 7f906b395740 1 -- 172.21.15.40:0/1711607692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) -- 0x55fb65fc41e0 con 0x55fb66251c20 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.464+0000 7f9046a00640 1 -- 172.21.15.40:0/1711607692 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9058042070 con 0x55fb66251c20 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.464+0000 7f9046a00640 1 -- 172.21.15.40:0/1711607692 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 119) ==== 3550+0+0 (secure 0 0 0) 0x7f9058059020 con 0x55fb66251c20 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.464+0000 7f906b395740 10 client.9306 fetch_fsmap finished waiting for FSMap version 119 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.464+0000 7f906b395740 10 client.9306 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.464+0000 7f906b395740 20 client.9306 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:37.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.465+0000 7f906b395740 1 --2- 172.21.15.40:0/1711607692 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55fb6627a0f0 0x55fb6627a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.465+0000 7f906b395740 4 client.9306 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:37.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.465+0000 7f906b395740 1 -- 172.21.15.40:0/1711607692 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb66197790 con 0x55fb6627a0f0 2024-04-26T23:09:37.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.465+0000 7f9065a00640 1 --2- 172.21.15.40:0/1711607692 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55fb6627a0f0 0x55fb6627a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.465+0000 7f9065a00640 1 --2- 172.21.15.40:0/1711607692 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55fb6627a0f0 0x55fb6627a4f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=322349da8cfedf63 server_cookie=5912429b975e3569 in_seq=0 out_seq=0 2024-04-26T23:09:37.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.465+0000 7f9046a00640 10 client.9306 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:37.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.466+0000 7f9046a00640 1 -- 172.21.15.40:0/1711607692 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fb66197790 con 0x55fb6627a0f0 2024-04-26T23:09:37.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.466+0000 7f9046a00640 10 client.9306 handle_command_reply: tid=0 2024-04-26T23:09:37.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.466+0000 7f906b395740 1 client.9306 shutdown 2024-04-26T23:09:37.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.466+0000 7f906b395740 1 -- 172.21.15.40:0/1711607692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb66251c20 msgr2=0x55fb6626a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.466+0000 7f906b395740 1 --2- 172.21.15.40:0/1711607692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb66251c20 0x55fb6626a4b0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f905803e000 tx=0x7f9058004ab0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.466+0000 7f906b395740 1 -- 172.21.15.40:0/1711607692 shutdown_connections 2024-04-26T23:09:37.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.466+0000 7f906b395740 1 --2- 172.21.15.40:0/1711607692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb66251c20 0x55fb6626a4b0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.466+0000 7f906b395740 1 --2- 172.21.15.40:0/1711607692 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55fb6627a0f0 0x55fb6627a4f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.466+0000 7f906b395740 1 --2- 172.21.15.40:0/1711607692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb6626d010 0x55fb662516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.466+0000 7f906b395740 1 --2- 172.21.15.40:0/1711607692 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 0x55fb6623fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.466+0000 7f906b395740 1 -- 172.21.15.40:0/1711607692 >> 172.21.15.40:0/1711607692 conn(0x55fb6624f9f0 msgr2=0x55fb66220900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.466+0000 7f9046a00640 0 client.9306 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:37.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.467+0000 7f906b395740 1 -- 172.21.15.40:0/1711607692 shutdown_connections 2024-04-26T23:09:37.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.467+0000 7f906b395740 1 -- 172.21.15.40:0/1711607692 wait complete. 2024-04-26T23:09:37.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.467+0000 7f906b395740 20 client.9306 trim_cache size 0 max 16384 2024-04-26T23:09:37.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.505+0000 7f906b395740 1 Processor -- start 2024-04-26T23:09:37.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.505+0000 7f906b395740 1 -- start start 2024-04-26T23:09:37.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.506+0000 7f906b395740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb66250130 0x55fb66226890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.506+0000 7f906b395740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66251c20 0x55fb6627a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.506+0000 7f906b395740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 0x55fb6627aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.506+0000 7f906b395740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9060201d20 con 0x55fb6626d010 2024-04-26T23:09:37.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.506+0000 7f906b395740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9060200ec0 con 0x55fb66250130 2024-04-26T23:09:37.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.506+0000 7f906b395740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f90602015f0 con 0x55fb66251c20 2024-04-26T23:09:37.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.506+0000 7f9066400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 0x55fb6627aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.507+0000 7f9066400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 0x55fb6627aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43158/0 (socket says 172.21.15.40:43158) 2024-04-26T23:09:37.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.507+0000 7f9066400640 1 -- 172.21.15.40:0/3295795854 learned_addr learned my addr 172.21.15.40:0/3295795854 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.507+0000 7f9065a00640 1 --2- 172.21.15.40:0/3295795854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb66250130 0x55fb66226890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.507+0000 7f9065000640 1 --2- 172.21.15.40:0/3295795854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66251c20 0x55fb6627a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.507+0000 7f9066400640 1 -- 172.21.15.40:0/3295795854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66251c20 msgr2=0x55fb6627a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.507+0000 7f9066400640 1 --2- 172.21.15.40:0/3295795854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66251c20 0x55fb6627a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.507+0000 7f9066400640 1 -- 172.21.15.40:0/3295795854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb66250130 msgr2=0x55fb66226890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.507+0000 7f9066400640 1 --2- 172.21.15.40:0/3295795854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb66250130 0x55fb66226890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.507+0000 7f9066400640 1 -- 172.21.15.40:0/3295795854 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb6626ce40 con 0x55fb6626d010 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.507+0000 7f9065a00640 1 --2- 172.21.15.40:0/3295795854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb66250130 0x55fb66226890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.507+0000 7f9066400640 1 --2- 172.21.15.40:0/3295795854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 0x55fb6627aa10 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f90580048d0 tx=0x7f9058033070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6eb7ac5bfe745ff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.508+0000 7f9051600640 1 -- 172.21.15.40:0/3295795854 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905804a040 con 0x55fb6626d010 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.508+0000 7f9051600640 1 -- 172.21.15.40:0/3295795854 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9058042040 con 0x55fb6626d010 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.508+0000 7f9051600640 1 -- 172.21.15.40:0/3295795854 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058036050 con 0x55fb6626d010 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.508+0000 7f906b395740 1 -- 172.21.15.40:0/3295795854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 msgr2=0x55fb6627aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.508+0000 7f906b395740 1 --2- 172.21.15.40:0/3295795854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 0x55fb6627aa10 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f90580048d0 tx=0x7f9058033070 comp rx=0 tx=0).stop 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.509+0000 7f906b395740 1 -- 172.21.15.40:0/3295795854 shutdown_connections 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.509+0000 7f906b395740 1 --2- 172.21.15.40:0/3295795854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 0x55fb6627aa10 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.509+0000 7f906b395740 1 --2- 172.21.15.40:0/3295795854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66251c20 0x55fb6627a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.509+0000 7f906b395740 1 --2- 172.21.15.40:0/3295795854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb66250130 0x55fb66226890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.509+0000 7f906b395740 1 -- 172.21.15.40:0/3295795854 >> 172.21.15.40:0/3295795854 conn(0x55fb6624f9f0 msgr2=0x55fb66223670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.509+0000 7f906b395740 1 -- 172.21.15.40:0/3295795854 shutdown_connections 2024-04-26T23:09:37.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.509+0000 7f906b395740 1 -- 172.21.15.40:0/3295795854 wait complete. 2024-04-26T23:09:37.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.509+0000 7f906b395740 1 -- start start 2024-04-26T23:09:37.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.510+0000 7f906b395740 1 Processor -- start 2024-04-26T23:09:37.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f906b395740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 0x55fb662e1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f906b395740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb66251c20 0x55fb662834c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f906b395740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 0x55fb6627a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f906b395740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fb66251950 con 0x55fb6626d010 2024-04-26T23:09:37.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f906b395740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f90600cef50 con 0x55fb66251c20 2024-04-26T23:09:37.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f906b395740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55fb6624c320 con 0x55fb66250130 2024-04-26T23:09:37.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f906b395740 10 client.0 _ll_register_callbacks cb 0x7f90540589b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:37.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f9066400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 0x55fb6627a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f9066400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 0x55fb6627a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43170/0 (socket says 172.21.15.40:43170) 2024-04-26T23:09:37.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f9066400640 1 -- 172.21.15.40:0/3997513834 learned_addr learned my addr 172.21.15.40:0/3997513834 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f9065000640 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb66251c20 0x55fb662834c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f9065a00640 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 0x55fb662e1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f9066400640 1 -- 172.21.15.40:0/3997513834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 msgr2=0x55fb662e1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.511+0000 7f9066400640 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 0x55fb662e1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.512+0000 7f9066400640 1 -- 172.21.15.40:0/3997513834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb66251c20 msgr2=0x55fb662834c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.512+0000 7f9066400640 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb66251c20 0x55fb662834c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.512+0000 7f9066400640 1 -- 172.21.15.40:0/3997513834 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb65e344a0 con 0x55fb6626d010 2024-04-26T23:09:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.512+0000 7f9065000640 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb66251c20 0x55fb662834c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.512+0000 7f9066400640 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 0x55fb6627a4d0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f9058009680 tx=0x7f9058009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d854e39bf92b7e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.512+0000 7f9046a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.512+0000 7f9046a00640 1 -- 172.21.15.40:0/3997513834 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058036030 con 0x55fb6626d010 2024-04-26T23:09:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.512+0000 7f9046a00640 1 -- 172.21.15.40:0/3997513834 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f905804ebd0 con 0x55fb6626d010 2024-04-26T23:09:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.512+0000 7f906b395740 1 -- 172.21.15.40:0/3997513834 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9058042040 con 0x55fb6626d010 2024-04-26T23:09:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.512+0000 7f9046a00640 1 -- 172.21.15.40:0/3997513834 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f9058069050 con 0x55fb6626d010 2024-04-26T23:09:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.512+0000 7f906b395740 10 client.9312 fetch_fsmap learned FSMap version 119 2024-04-26T23:09:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.512+0000 7f906b395740 1 -- 172.21.15.40:0/3997513834 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) -- 0x55fb662e46d0 con 0x55fb6626d010 2024-04-26T23:09:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.513+0000 7f9046a00640 1 -- 172.21.15.40:0/3997513834 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9058042040 con 0x55fb6626d010 2024-04-26T23:09:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.513+0000 7f9046a00640 1 -- 172.21.15.40:0/3997513834 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 119) ==== 3550+0+0 (secure 0 0 0) 0x7f905806c020 con 0x55fb6626d010 2024-04-26T23:09:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.513+0000 7f906b395740 10 client.9312 fetch_fsmap finished waiting for FSMap version 119 2024-04-26T23:09:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.513+0000 7f906b395740 10 client.9312 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.513+0000 7f906b395740 20 client.9312 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.513+0000 7f906b395740 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55fb662e4220 0x55fb6627aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.513+0000 7f906b395740 4 client.9312 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/d1/d2/d3", "scrubops": ["recursive", "force"]}] 2024-04-26T23:09:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.513+0000 7f906b395740 1 -- 172.21.15.40:0/3997513834 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "scrub start", "path": "/d1/d2/d3", "scrubops": ["recursive", "force"]}) -- 0x55fb66191600 con 0x55fb662e4220 2024-04-26T23:09:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.513+0000 7f9065a00640 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55fb662e4220 0x55fb6627aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.514+0000 7f9065a00640 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55fb662e4220 0x55fb6627aa10 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f753fe94f991a5e7 server_cookie=77204bd77626f53b in_seq=0 out_seq=0 2024-04-26T23:09:37.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.514+0000 7f9046a00640 10 client.9312 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:37.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.514+0000 7f9046a00640 1 -- 172.21.15.40:0/3997513834 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fb66191600 con 0x55fb662e4220 2024-04-26T23:09:37.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.514+0000 7f9046a00640 10 client.9312 handle_command_reply: tid=0 2024-04-26T23:09:37.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.514+0000 7f906b395740 1 client.9312 shutdown 2024-04-26T23:09:37.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.515+0000 7f906b395740 1 -- 172.21.15.40:0/3997513834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 msgr2=0x55fb6627a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.515+0000 7f906b395740 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 0x55fb6627a4d0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f9058009680 tx=0x7f9058009000 comp rx=0 tx=0).stop 2024-04-26T23:09:37.518 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-04-26T23:09:37.515+0000 7f9289600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000002 [...2,head] /d1/d2/d3/ auth{1=1,2=1} v19 ap=1 f(v0 m2024-04-26T23:09:19.919671+0000 1=0+1) n(v3 rc2024-04-26T23:09:20.161667+0000 6=0+6) (inest mix dirty) (ifile excl) caps={6092=pAsLsXsFsx/pAsLsXsFsxcral@4},l=6092 | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 0x557fb37e1080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(15)0x10000000002:[<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:09:19.919671+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:09:20.161667+0000 6=0+6)","memory_value.dirstat":"f(v0 m2024-04-26T23:09:19.919671+0000 1=0+1)","memory_value.rstat":"n(v3 rc2024-04-26T23:09:20.161667+0000 6=0+6)","error_str":""},"return_code":-61} 2024-04-26T23:09:37.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.515+0000 7f906b395740 1 -- 172.21.15.40:0/3997513834 shutdown_connections 2024-04-26T23:09:37.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.515+0000 7f906b395740 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fb6626d010 0x55fb6627a4d0 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.515+0000 7f906b395740 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55fb662e4220 0x55fb6627aa10 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.515+0000 7f906b395740 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb66251c20 0x55fb662834c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.515+0000 7f906b395740 1 --2- 172.21.15.40:0/3997513834 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fb66250130 0x55fb662e1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.515+0000 7f9046a00640 0 client.9312 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:37.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.515+0000 7f906b395740 1 -- 172.21.15.40:0/3997513834 >> 172.21.15.40:0/3997513834 conn(0x55fb6624f9f0 msgr2=0x55fb66252e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.515+0000 7f906b395740 1 -- 172.21.15.40:0/3997513834 shutdown_connections 2024-04-26T23:09:37.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.515+0000 7f906b395740 1 -- 172.21.15.40:0/3997513834 wait complete. 2024-04-26T23:09:37.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.516+0000 7f906b395740 20 client.9312 trim_cache size 0 max 16384 2024-04-26T23:09:37.520 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-04-26T23:09:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "a9827531-70a9-4d6f-b12a-2d71fa886d03", 2024-04-26T23:09:37.520 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-04-26T23:09:37.520 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:37.521 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:09:37.516+0000 7fa211000640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000004 [...2,head] /d1/d2/d3/d4/d5/ auth{0=1,2=1} v15 ap=1 f(v0 m2024-04-26T23:09:19.921671+0000 1=0+1) n(v2 rc2024-04-26T23:09:20.161667+0000 4=0+4) (inest lock) (ifile excl) caps={6092=pAsLsXsFsx/pAsLsXsFsxcral@4},l=6092 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 authpin=1 scrubqueue=0 0x5638bd7b0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(15)0x10000000004:[<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:09:19.921671+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:09:20.161667+0000 4=0+4)","memory_value.dirstat":"f(v0 m2024-04-26T23:09:19.921671+0000 1=0+1)","memory_value.rstat":"n(v2 rc2024-04-26T23:09:20.161667+0000 4=0+4)","error_str":""},"return_code":-61} 2024-04-26T23:09:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.516+0000 7f9066e00640 1 -- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f904006c8f0 msgr2=0x7f904006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.516+0000 7f9066e00640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f904006c8f0 0x7f904006edb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f906006d740 tx=0x7f904c008040 comp rx=0 tx=0).stop 2024-04-26T23:09:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.516+0000 7f9066e00640 1 -- 172.21.15.40:0/1848521948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90600b3d90 msgr2=0x7f90600b9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.516+0000 7f9066e00640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90600b3d90 0x7f90600b9e30 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f905400c400 tx=0x7f905400c8c0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.516+0000 7f9066e00640 1 -- 172.21.15.40:0/1848521948 shutdown_connections 2024-04-26T23:09:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.516+0000 7f9066e00640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f904006c8f0 0x7f904006edb0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.516+0000 7f9066e00640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f906015f420 0x7f9060151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.516+0000 7f9066e00640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90600d3b60 0x7f9060151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.516+0000 7f9066e00640 1 --2- 172.21.15.40:0/1848521948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90600b3d90 0x7f90600b9e30 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.516+0000 7f9066e00640 1 -- 172.21.15.40:0/1848521948 >> 172.21.15.40:0/1848521948 conn(0x7f906009e1d0 msgr2=0x7f906006d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.517+0000 7f9066e00640 1 -- 172.21.15.40:0/1848521948 shutdown_connections 2024-04-26T23:09:37.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.517+0000 7f9066e00640 1 -- 172.21.15.40:0/1848521948 wait complete. 2024-04-26T23:09:37.523 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-04-26T23:09:37.517+0000 7f9291800640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000003 [...2,head] /d1/d2/d3/d4/ auth{1=1,2=1} v19 ap=2 f(v0 m2024-04-26T23:09:19.920671+0000 1=0+1) n(v4 rc2024-04-26T23:09:20.161667+0000 5=0+5) (inest sync->mix g=1,2) caps={6092=pAsLsXsFs/pAsLsXsFsxcral@8} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 export_pin=1 0x557fb37e0b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(15)0x10000000003:[<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:09:19.920671+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:09:20.161667+0000 5=0+5)","memory_value.dirstat":"f(v0 m2024-04-26T23:09:19.920671+0000 1=0+1)","memory_value.rstat":"n(v4 rc2024-04-26T23:09:20.161667+0000 5=0+5)","error_str":""},"return_code":-61} 2024-04-26T23:09:37.523 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-04-26T23:09:37.519+0000 7fa219200640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000005 [...2,head] /d1/d2/d3/d4/d5/d6/ auth{0=1,2=1} v11 ap=2 f(v0 m2024-04-26T23:09:19.922671+0000 1=0+1) n(v1 rc2024-04-26T23:09:20.161667+0000 3=0+3) (inest sync->mix g=0,2 scatter_wanted) caps={6092=pAsLsXsFs/pAsLsXsFsxcral@7} | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 export_pin=2 0x5638bd7b1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(15)0x10000000005:[<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:09:19.922671+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:09:19.923671+0000 3=0+3)","memory_value.dirstat":"f(v0 m2024-04-26T23:09:19.922671+0000 1=0+1)","memory_value.rstat":"n(v1 rc2024-04-26T23:09:20.161667+0000 3=0+3)","error_str":""},"return_code":-61} 2024-04-26T23:09:37.523 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-26T23:09:37.517+0000 7fddf5000640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000006 [...2,head] /d1/d2/d3/d4/d5/d6/d7/ auth v4 ap=1 f(v0 m2024-04-26T23:09:19.923671+0000 1=0+1) n(v0 rc2024-04-26T23:09:19.923671+0000 2=0+2) (inest lock) (ifile excl) caps={6092=pAsLsXsFsx/pAsLsXsFsxcral@4},l=6092 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x55cd1e12c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(15)0x10000000006:[<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:09:19.923671+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:09:19.923671+0000 2=0+2)","memory_value.dirstat":"f(v0 m2024-04-26T23:09:19.923671+0000 1=0+1)","memory_value.rstat":"n(v0 rc2024-04-26T23:09:19.923671+0000 2=0+2)","error_str":""},"return_code":-61} 2024-04-26T23:09:37.524 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-26T23:09:37.518+0000 7fddf5000640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000007 [...2,head] /d1/d2/d3/d4/d5/d6/d7/d8/ auth v2 ap=1 f() n(v0 1=0+1) (inest lock) (ifile excl) caps={6092=pAsLsXsFsx/AsLsXsFsx@2},l=6092 | importingcaps=0 dirfrag=0 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x55cd1e12c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(15)0x10000000007:[<0x10000000006/d8 v2>,<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 1=0+1)","error_str":""},"return_code":-61} 2024-04-26T23:09:37.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:37.536 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub pause 2024-04-26T23:09:37.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.736+0000 7fc09c000640 1 Processor -- start 2024-04-26T23:09:37.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.737+0000 7fc09c000640 1 -- start start 2024-04-26T23:09:37.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.738+0000 7fc09c000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc09415f3a0 0x7fc0940d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.738+0000 7fc09c000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc0940d1770 0x7fc0940d1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.738+0000 7fc09c000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0940d2590 con 0x7fc0940d2160 2024-04-26T23:09:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.738+0000 7fc09c000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc0940d0e20 con 0x7fc09415f3a0 2024-04-26T23:09:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.738+0000 7fc09c000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc0940b1d70 con 0x7fc0940d1770 2024-04-26T23:09:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.738+0000 7fc099600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc09415f3a0 0x7fc0940d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.738+0000 7fc099600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc09415f3a0 0x7fc0940d06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43176/0 (socket says 172.21.15.40:43176) 2024-04-26T23:09:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.738+0000 7fc099600640 1 -- 172.21.15.40:0/3547205895 learned_addr learned my addr 172.21.15.40:0/3547205895 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.738+0000 7fc09a000640 1 --1- 172.21.15.40:0/3547205895 >> v1:172.21.15.40:6789/0 conn(0x7fc0940d2160 0x7fc0940d2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44836/0 (socket says 172.21.15.40:44836) 2024-04-26T23:09:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.739+0000 7fc098c00640 1 --2- 172.21.15.40:0/3547205895 >> v2:172.21.15.103:3300/0 conn(0x7fc0940d1770 0x7fc0940d1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.739+0000 7fc099600640 1 -- 172.21.15.40:0/3547205895 >> v2:172.21.15.103:3300/0 conn(0x7fc0940d1770 msgr2=0x7fc0940d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.739+0000 7fc099600640 1 --2- 172.21.15.40:0/3547205895 >> v2:172.21.15.103:3300/0 conn(0x7fc0940d1770 0x7fc0940d1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.739+0000 7fc099600640 1 -- 172.21.15.40:0/3547205895 >> v1:172.21.15.40:6789/0 conn(0x7fc0940d2160 legacy=0x7fc0940d2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.739+0000 7fc093e00640 1 -- 172.21.15.40:0/3547205895 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2012713617 0 0) 0x7fc0940d2590 con 0x7fc0940d2160 2024-04-26T23:09:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.739+0000 7fc099600640 1 -- 172.21.15.40:0/3547205895 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0940b2050 con 0x7fc09415f3a0 2024-04-26T23:09:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.740+0000 7fc099600640 1 --2- 172.21.15.40:0/3547205895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc09415f3a0 0x7fc0940d06f0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fc0940b7b80 tx=0x7fc08802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32e1cf7c42ac135 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.740+0000 7fc093e00640 1 -- 172.21.15.40:0/3547205895 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc088037070 con 0x7fc09415f3a0 2024-04-26T23:09:37.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.740+0000 7fc093e00640 1 -- 172.21.15.40:0/3547205895 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc088032440 con 0x7fc09415f3a0 2024-04-26T23:09:37.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.740+0000 7fc093e00640 1 -- 172.21.15.40:0/3547205895 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc088032810 con 0x7fc09415f3a0 2024-04-26T23:09:37.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.741+0000 7fc09c000640 1 -- 172.21.15.40:0/3547205895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc09415f3a0 msgr2=0x7fc0940d06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.741+0000 7fc09c000640 1 --2- 172.21.15.40:0/3547205895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc09415f3a0 0x7fc0940d06f0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fc0940b7b80 tx=0x7fc08802a270 comp rx=0 tx=0).stop 2024-04-26T23:09:37.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.741+0000 7fc09c000640 1 -- 172.21.15.40:0/3547205895 shutdown_connections 2024-04-26T23:09:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.741+0000 7fc09c000640 1 --2- 172.21.15.40:0/3547205895 >> v2:172.21.15.103:3300/0 conn(0x7fc0940d1770 0x7fc0940d1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.741+0000 7fc09c000640 1 --2- 172.21.15.40:0/3547205895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc09415f3a0 0x7fc0940d06f0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.741+0000 7fc09c000640 1 -- 172.21.15.40:0/3547205895 >> 172.21.15.40:0/3547205895 conn(0x7fc0940d4f90 msgr2=0x7fc0940d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.741+0000 7fc09c000640 1 -- 172.21.15.40:0/3547205895 shutdown_connections 2024-04-26T23:09:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.741+0000 7fc09c000640 1 -- 172.21.15.40:0/3547205895 wait complete. 2024-04-26T23:09:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.742+0000 7fc09c000640 1 Processor -- start 2024-04-26T23:09:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.743+0000 7fc09c000640 1 -- start start 2024-04-26T23:09:37.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.743+0000 7fc09c000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0940d1770 0x7fc094151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.743+0000 7fc09c000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0940d2160 0x7fc0941f8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.743+0000 7fc09c000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc09415f3a0 0x7fc0941f8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.743+0000 7fc09c000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc09415f900 con 0x7fc0940d2160 2024-04-26T23:09:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.743+0000 7fc09c000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc09415f780 con 0x7fc0940d1770 2024-04-26T23:09:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.743+0000 7fc09c000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc0940b3860 con 0x7fc09415f3a0 2024-04-26T23:09:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.744+0000 7fc098c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0940d2160 0x7fc0941f8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.744+0000 7fc098c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0940d2160 0x7fc0941f8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43184/0 (socket says 172.21.15.40:43184) 2024-04-26T23:09:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.744+0000 7fc098c00640 1 -- 172.21.15.40:0/1872471870 learned_addr learned my addr 172.21.15.40:0/1872471870 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.744+0000 7fc099600640 1 --2- 172.21.15.40:0/1872471870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0940d1770 0x7fc094151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.744+0000 7fc098c00640 1 -- 172.21.15.40:0/1872471870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc09415f3a0 msgr2=0x7fc0941f8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.744+0000 7fc098c00640 1 --2- 172.21.15.40:0/1872471870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc09415f3a0 0x7fc0941f8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.744+0000 7fc098c00640 1 -- 172.21.15.40:0/1872471870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0940d1770 msgr2=0x7fc094151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.744+0000 7fc098c00640 1 --2- 172.21.15.40:0/1872471870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0940d1770 0x7fc094151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.744+0000 7fc098c00640 1 -- 172.21.15.40:0/1872471870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0940b9b50 con 0x7fc0940d2160 2024-04-26T23:09:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.744+0000 7fc098c00640 1 --2- 172.21.15.40:0/1872471870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0940d2160 0x7fc0941f8050 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fc08000d800 tx=0x7fc08000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.745+0000 7fc091600640 1 -- 172.21.15.40:0/1872471870 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08000b6c0 con 0x7fc0940d2160 2024-04-26T23:09:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.745+0000 7fc091600640 1 -- 172.21.15.40:0/1872471870 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc080019070 con 0x7fc0940d2160 2024-04-26T23:09:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.745+0000 7fc091600640 1 -- 172.21.15.40:0/1872471870 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08000ba80 con 0x7fc0940d2160 2024-04-26T23:09:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.745+0000 7fc09c000640 1 -- 172.21.15.40:0/1872471870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0940b9e30 con 0x7fc0940d2160 2024-04-26T23:09:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.745+0000 7fc09c000640 1 -- 172.21.15.40:0/1872471870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0940ba3f0 con 0x7fc0940d2160 2024-04-26T23:09:37.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.747+0000 7fc091600640 1 -- 172.21.15.40:0/1872471870 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc08000bc20 con 0x7fc0940d2160 2024-04-26T23:09:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.748+0000 7fc091600640 1 --2- 172.21.15.40:0/1872471870 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc07406c920 0x7fc07406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.748+0000 7fc091600640 1 -- 172.21.15.40:0/1872471870 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fc08008c210 con 0x7fc0940d2160 2024-04-26T23:09:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.749+0000 7fc09ee4b740 1 Processor -- start 2024-04-26T23:09:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.749+0000 7fc09ee4b740 1 -- start start 2024-04-26T23:09:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.749+0000 7fc099600640 1 --2- 172.21.15.40:0/1872471870 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc07406c920 0x7fc07406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc09ee4b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19d9e2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc09ee4b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc09ee4b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9b7c30 0x55d19d9fdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc09ee4b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d19d91b900 con 0x55d19d9b7c30 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc09ee4b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d19d8b81d0 con 0x55d19d9ff7f0 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc09ee4b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d19d9178c0 con 0x55d19d9e1f50 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc09a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9b7c30 0x55d19d9fdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc09a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9b7c30 0x55d19d9fdeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43192/0 (socket says 172.21.15.40:43192) 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc09a000640 1 -- 172.21.15.40:0/2781817904 learned_addr learned my addr 172.21.15.40:0/2781817904 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc099600640 1 --2- 172.21.15.40:0/1872471870 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc07406c920 0x7fc07406ede0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc088034000 tx=0x7fc08802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc099600640 1 --2- 172.21.15.40:0/2781817904 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19d9e2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc09a000640 1 -- 172.21.15.40:0/2781817904 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 msgr2=0x55d19d9e2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc098c00640 1 --2- 172.21.15.40:0/2781817904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc09a000640 1 --2- 172.21.15.40:0/2781817904 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19d9e2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.750+0000 7fc09a000640 1 -- 172.21.15.40:0/2781817904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 msgr2=0x55d19d9ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.751+0000 7fc09a000640 1 --2- 172.21.15.40:0/2781817904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9ffd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.751+0000 7fc09a000640 1 -- 172.21.15.40:0/2781817904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d19d6f77d0 con 0x55d19d9b7c30 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.751+0000 7fc099600640 1 --2- 172.21.15.40:0/2781817904 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19d9e2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.751+0000 7fc098c00640 1 --2- 172.21.15.40:0/2781817904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.751+0000 7fc09a000640 1 --2- 172.21.15.40:0/2781817904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9b7c30 0x55d19d9fdeb0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fc08c001d60 tx=0x7fc08c00c0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df728ac0342012d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.751+0000 7fc085c00640 1 -- 172.21.15.40:0/2781817904 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c00c910 con 0x55d19d9b7c30 2024-04-26T23:09:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.751+0000 7fc085c00640 1 -- 172.21.15.40:0/2781817904 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc08c00cab0 con 0x55d19d9b7c30 2024-04-26T23:09:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.751+0000 7fc085c00640 1 -- 172.21.15.40:0/2781817904 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c00ce60 con 0x55d19d9b7c30 2024-04-26T23:09:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.752+0000 7fc09ee4b740 1 -- 172.21.15.40:0/2781817904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9b7c30 msgr2=0x55d19d9fdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.752+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/2781817904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9b7c30 0x55d19d9fdeb0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fc08c001d60 tx=0x7fc08c00c0e0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.752+0000 7fc09ee4b740 1 -- 172.21.15.40:0/2781817904 shutdown_connections 2024-04-26T23:09:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.752+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/2781817904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9b7c30 0x55d19d9fdeb0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.752+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/2781817904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.752+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/2781817904 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19d9e2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.752+0000 7fc09ee4b740 1 -- 172.21.15.40:0/2781817904 >> 172.21.15.40:0/2781817904 conn(0x55d19d9b4560 msgr2=0x55d19d9fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.752+0000 7fc09ee4b740 1 -- 172.21.15.40:0/2781817904 shutdown_connections 2024-04-26T23:09:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.752+0000 7fc09ee4b740 1 -- 172.21.15.40:0/2781817904 wait complete. 2024-04-26T23:09:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.753+0000 7fc09ee4b740 1 -- start start 2024-04-26T23:09:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.753+0000 7fc09ee4b740 1 Processor -- start 2024-04-26T23:09:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.754+0000 7fc09ee4b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19d9b1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.754+0000 7fc09ee4b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.754+0000 7fc09ee4b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 0x55d19d9e3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.754+0000 7fc09ee4b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d19d9b89d0 con 0x55d19da109b0 2024-04-26T23:09:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.754+0000 7fc09ee4b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d19d8a3120 con 0x55d19d9ff7f0 2024-04-26T23:09:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.754+0000 7fc09ee4b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d19d9b2490 con 0x55d19d9e1f50 2024-04-26T23:09:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.754+0000 7fc09ee4b740 10 client.0 _ll_register_callbacks cb 0x55d19d8db750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.754+0000 7fc09a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 0x55d19d9e3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.754+0000 7fc09a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 0x55d19d9e3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43204/0 (socket says 172.21.15.40:43204) 2024-04-26T23:09:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.754+0000 7fc09a000640 1 -- 172.21.15.40:0/2423695931 learned_addr learned my addr 172.21.15.40:0/2423695931 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc098c00640 1 --2- 172.21.15.40:0/2423695931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc09a000640 1 -- 172.21.15.40:0/2423695931 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 msgr2=0x55d19d9b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc09a000640 1 --2- 172.21.15.40:0/2423695931 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19d9b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc09a000640 1 -- 172.21.15.40:0/2423695931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 msgr2=0x55d19d9b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc09a000640 1 --2- 172.21.15.40:0/2423695931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9b1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc09a000640 1 -- 172.21.15.40:0/2423695931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d19d5ed0f0 con 0x55d19da109b0 2024-04-26T23:09:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc09a000640 1 --2- 172.21.15.40:0/2423695931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 0x55d19d9e3ee0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00c120 tx=0x7fc08c001400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67369eef491ff033 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc07a000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc07a000640 1 -- 172.21.15.40:0/2423695931 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c024060 con 0x55d19da109b0 2024-04-26T23:09:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc09ee4b740 1 -- 172.21.15.40:0/2423695931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc08c00cab0 con 0x55d19da109b0 2024-04-26T23:09:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc07a000640 1 -- 172.21.15.40:0/2423695931 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fc08c0072d0 con 0x55d19da109b0 2024-04-26T23:09:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc07a000640 1 -- 172.21.15.40:0/2423695931 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fc08c019040 con 0x55d19da109b0 2024-04-26T23:09:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc09ee4b740 10 client.9327 fetch_fsmap learned FSMap version 119 2024-04-26T23:09:37.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.755+0000 7fc09ee4b740 1 -- 172.21.15.40:0/2423695931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) -- 0x55d19d757f60 con 0x55d19da109b0 2024-04-26T23:09:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.756+0000 7fc07a000640 1 -- 172.21.15.40:0/2423695931 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc08c00cab0 con 0x55d19da109b0 2024-04-26T23:09:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.756+0000 7fc07a000640 1 -- 172.21.15.40:0/2423695931 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 119) ==== 3550+0+0 (secure 0 0 0) 0x7fc08c0074d0 con 0x55d19da109b0 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.756+0000 7fc09ee4b740 10 client.9327 fetch_fsmap finished waiting for FSMap version 119 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.756+0000 7fc09ee4b740 10 client.9327 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.756+0000 7fc09ee4b740 20 client.9327 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.756+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/2423695931 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55d19d9e48f0 0x55d19d9f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.756+0000 7fc09ee4b740 4 client.9327 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.756+0000 7fc09ee4b740 1 -- 172.21.15.40:0/2423695931 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d19d929510 con 0x55d19d9e48f0 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.756+0000 7fc099600640 1 --2- 172.21.15.40:0/2423695931 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55d19d9e48f0 0x55d19d9f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.757+0000 7fc099600640 1 --2- 172.21.15.40:0/2423695931 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55d19d9e48f0 0x55d19d9f44d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd995034625f700f server_cookie=402fedb5947fdf67 in_seq=0 out_seq=0 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.757+0000 7fc07a000640 10 client.9327 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc07a000640 1 -- 172.21.15.40:0/2423695931 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d19d929510 con 0x55d19d9e48f0 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc07a000640 10 client.9327 handle_command_reply: tid=0 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc09ee4b740 1 client.9327 shutdown 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc09ee4b740 1 -- 172.21.15.40:0/2423695931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 msgr2=0x55d19d9e3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/2423695931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 0x55d19d9e3ee0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00c120 tx=0x7fc08c001400 comp rx=0 tx=0).stop 2024-04-26T23:09:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc09ee4b740 1 -- 172.21.15.40:0/2423695931 shutdown_connections 2024-04-26T23:09:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/2423695931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 0x55d19d9e3ee0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/2423695931 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55d19d9e48f0 0x55d19d9f44d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/2423695931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/2423695931 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19d9b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc09ee4b740 1 -- 172.21.15.40:0/2423695931 >> 172.21.15.40:0/2423695931 conn(0x55d19d9b7c30 msgr2=0x55d19d9fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc07a000640 0 client.9327 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc09ee4b740 1 -- 172.21.15.40:0/2423695931 shutdown_connections 2024-04-26T23:09:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc09ee4b740 1 -- 172.21.15.40:0/2423695931 wait complete. 2024-04-26T23:09:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.758+0000 7fc09ee4b740 20 client.9327 trim_cache size 0 max 16384 2024-04-26T23:09:37.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.790+0000 7fc09ee4b740 1 Processor -- start 2024-04-26T23:09:37.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.790+0000 7fc09ee4b740 1 -- start start 2024-04-26T23:09:37.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.790+0000 7fc09ee4b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19da21000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.790+0000 7fc09ee4b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9f5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.790+0000 7fc09ee4b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 0x55d19d9b8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.790+0000 7fc09ee4b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc094201c90 con 0x55d19da109b0 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.790+0000 7fc09ee4b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc094200e30 con 0x55d19d9ff7f0 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.791+0000 7fc09ee4b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc094201560 con 0x55d19d9e1f50 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.791+0000 7fc09a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 0x55d19d9b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.791+0000 7fc09a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 0x55d19d9b8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43208/0 (socket says 172.21.15.40:43208) 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.791+0000 7fc09a000640 1 -- 172.21.15.40:0/3050395375 learned_addr learned my addr 172.21.15.40:0/3050395375 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.791+0000 7fc099600640 1 --2- 172.21.15.40:0/3050395375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19da21000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.791+0000 7fc098c00640 1 --2- 172.21.15.40:0/3050395375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.791+0000 7fc09a000640 1 -- 172.21.15.40:0/3050395375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 msgr2=0x55d19da21000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.791+0000 7fc09a000640 1 --2- 172.21.15.40:0/3050395375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19da21000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.791+0000 7fc09a000640 1 -- 172.21.15.40:0/3050395375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 msgr2=0x55d19d9f5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.791+0000 7fc09a000640 1 --2- 172.21.15.40:0/3050395375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9f5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.791+0000 7fc09a000640 1 -- 172.21.15.40:0/3050395375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d19d5c61b0 con 0x55d19da109b0 2024-04-26T23:09:37.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.791+0000 7fc098c00640 1 --2- 172.21.15.40:0/3050395375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9f5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:37.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.792+0000 7fc09a000640 1 --2- 172.21.15.40:0/3050395375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 0x55d19d9b8010 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00cb30 tx=0x7fc08c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e113a77e2225ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.792+0000 7fc085200640 1 -- 172.21.15.40:0/3050395375 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c024040 con 0x55d19da109b0 2024-04-26T23:09:37.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.792+0000 7fc085200640 1 -- 172.21.15.40:0/3050395375 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc08c033020 con 0x55d19da109b0 2024-04-26T23:09:37.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.792+0000 7fc085200640 1 -- 172.21.15.40:0/3050395375 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c0070d0 con 0x55d19da109b0 2024-04-26T23:09:37.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.792+0000 7fc09ee4b740 1 -- 172.21.15.40:0/3050395375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 msgr2=0x55d19d9b8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.792+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/3050395375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 0x55d19d9b8010 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00cb30 tx=0x7fc08c01d000 comp rx=0 tx=0).stop 2024-04-26T23:09:37.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.793+0000 7fc09ee4b740 1 -- 172.21.15.40:0/3050395375 shutdown_connections 2024-04-26T23:09:37.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.793+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/3050395375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19da109b0 0x55d19d9b8010 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.793+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/3050395375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19d9ff7f0 0x55d19d9f5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.793+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/3050395375 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19da21000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.793+0000 7fc09ee4b740 1 -- 172.21.15.40:0/3050395375 >> 172.21.15.40:0/3050395375 conn(0x55d19d9b7c30 msgr2=0x55d19da0ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.793+0000 7fc09ee4b740 1 -- 172.21.15.40:0/3050395375 shutdown_connections 2024-04-26T23:09:37.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.793+0000 7fc09ee4b740 1 -- 172.21.15.40:0/3050395375 wait complete. 2024-04-26T23:09:37.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.793+0000 7fc09ee4b740 1 -- start start 2024-04-26T23:09:37.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.793+0000 7fc09ee4b740 1 Processor -- start 2024-04-26T23:09:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.794+0000 7fc09ee4b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19d9f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.794+0000 7fc09ee4b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9ff7f0 0x55d19d9b8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.794+0000 7fc09ee4b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19da109b0 0x55d19d9b8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.794+0000 7fc09ee4b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d19d9e1850 con 0x55d19d9ff7f0 2024-04-26T23:09:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.794+0000 7fc09ee4b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc094152e90 con 0x55d19da109b0 2024-04-26T23:09:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.794+0000 7fc09ee4b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d19d917ab0 con 0x55d19d9e1f50 2024-04-26T23:09:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.794+0000 7fc09ee4b740 10 client.0 _ll_register_callbacks cb 0x7fc08c001bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.795+0000 7fc098c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9ff7f0 0x55d19d9b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.795+0000 7fc098c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9ff7f0 0x55d19d9b8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43224/0 (socket says 172.21.15.40:43224) 2024-04-26T23:09:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.795+0000 7fc098c00640 1 -- 172.21.15.40:0/938559432 learned_addr learned my addr 172.21.15.40:0/938559432 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.795+0000 7fc099600640 1 --2- 172.21.15.40:0/938559432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19d9f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.795+0000 7fc098c00640 1 -- 172.21.15.40:0/938559432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 msgr2=0x55d19d9f4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.795+0000 7fc098c00640 1 --2- 172.21.15.40:0/938559432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19d9f4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.795+0000 7fc098c00640 1 -- 172.21.15.40:0/938559432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19da109b0 msgr2=0x55d19d9b8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.795+0000 7fc09a000640 1 --2- 172.21.15.40:0/938559432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19da109b0 0x55d19d9b8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.795+0000 7fc098c00640 1 --2- 172.21.15.40:0/938559432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19da109b0 0x55d19d9b8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.795+0000 7fc098c00640 1 -- 172.21.15.40:0/938559432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d19da0c400 con 0x55d19d9ff7f0 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.795+0000 7fc098c00640 1 --2- 172.21.15.40:0/938559432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9ff7f0 0x55d19d9b8010 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fc0800623c0 tx=0x7fc080011340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c6a1c48434b787 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.795+0000 7fc07a000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc07a000640 1 -- 172.21.15.40:0/938559432 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080088060 con 0x55d19d9ff7f0 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc07a000640 1 -- 172.21.15.40:0/938559432 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fc080056380 con 0x55d19d9ff7f0 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc09ee4b740 1 -- 172.21.15.40:0/938559432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc08c033020 con 0x55d19d9ff7f0 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc07a000640 1 -- 172.21.15.40:0/938559432 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fc080077050 con 0x55d19d9ff7f0 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc09ee4b740 10 client.9333 fetch_fsmap learned FSMap version 119 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc09ee4b740 1 -- 172.21.15.40:0/938559432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) -- 0x55d19d9b6120 con 0x55d19d9ff7f0 2024-04-26T23:09:37.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc07a000640 1 -- 172.21.15.40:0/938559432 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc08c033020 con 0x55d19d9ff7f0 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc07a000640 1 -- 172.21.15.40:0/938559432 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 119) ==== 3550+0+0 (secure 0 0 0) 0x7fc080013070 con 0x55d19d9ff7f0 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc09ee4b740 10 client.9333 fetch_fsmap finished waiting for FSMap version 119 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc09ee4b740 10 client.9333 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc09ee4b740 20 client.9333 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/938559432 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55d19da762c0 0x55d19d9ffbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc09ee4b740 4 client.9333 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "scrub pause"}] 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.796+0000 7fc09ee4b740 1 -- 172.21.15.40:0/938559432 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "scrub pause"}) -- 0x55d19d923380 con 0x55d19da762c0 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.797+0000 7fc099600640 1 --2- 172.21.15.40:0/938559432 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55d19da762c0 0x55d19d9ffbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.797+0000 7fc099600640 1 --2- 172.21.15.40:0/938559432 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55d19da762c0 0x55d19d9ffbd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5025bc5c7aad4c server_cookie=b045807e84dd744 in_seq=0 out_seq=0 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.797+0000 7fc07a000640 10 client.9333 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.797+0000 7fc07a000640 1 -- 172.21.15.40:0/938559432 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55d19d923380 con 0x55d19da762c0 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.797+0000 7fc07a000640 10 client.9333 handle_command_reply: tid=0 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.797+0000 7fc09ee4b740 1 client.9333 shutdown 2024-04-26T23:09:37.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc09ee4b740 1 -- 172.21.15.40:0/938559432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9ff7f0 msgr2=0x55d19d9b8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/938559432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9ff7f0 0x55d19d9b8010 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fc0800623c0 tx=0x7fc080011340 comp rx=0 tx=0).stop 2024-04-26T23:09:37.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc09ee4b740 1 -- 172.21.15.40:0/938559432 shutdown_connections 2024-04-26T23:09:37.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/938559432 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55d19da762c0 0x55d19d9ffbd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/938559432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d19da109b0 0x55d19d9b8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/938559432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d19d9ff7f0 0x55d19d9b8010 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc09ee4b740 1 --2- 172.21.15.40:0/938559432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d19d9e1f50 0x55d19d9f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc09ee4b740 1 -- 172.21.15.40:0/938559432 >> 172.21.15.40:0/938559432 conn(0x55d19d9b7c30 msgr2=0x55d19d9f5650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc07a000640 0 client.9333 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:37.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc09ee4b740 1 -- 172.21.15.40:0/938559432 shutdown_connections 2024-04-26T23:09:37.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc09ee4b740 1 -- 172.21.15.40:0/938559432 wait complete. 2024-04-26T23:09:37.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc09ee4b740 20 client.9333 trim_cache size 0 max 16384 2024-04-26T23:09:37.803 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:37.803 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:09:37.803 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc09c000640 1 -- 172.21.15.40:0/1872471870 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc07406c920 msgr2=0x7fc07406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.798+0000 7fc09c000640 1 --2- 172.21.15.40:0/1872471870 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc07406c920 0x7fc07406ede0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc088034000 tx=0x7fc08802e000 comp rx=0 tx=0).stop 2024-04-26T23:09:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.799+0000 7fc09c000640 1 -- 172.21.15.40:0/1872471870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0940d2160 msgr2=0x7fc0941f8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.799+0000 7fc09c000640 1 --2- 172.21.15.40:0/1872471870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0940d2160 0x7fc0941f8050 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fc08000d800 tx=0x7fc08000dcc0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.799+0000 7fc09c000640 1 -- 172.21.15.40:0/1872471870 shutdown_connections 2024-04-26T23:09:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.799+0000 7fc09c000640 1 --2- 172.21.15.40:0/1872471870 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc07406c920 0x7fc07406ede0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.799+0000 7fc09c000640 1 --2- 172.21.15.40:0/1872471870 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc09415f3a0 0x7fc0941f8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.799+0000 7fc09c000640 1 --2- 172.21.15.40:0/1872471870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0940d2160 0x7fc0941f8050 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.799+0000 7fc09c000640 1 --2- 172.21.15.40:0/1872471870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0940d1770 0x7fc094151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.799+0000 7fc09c000640 1 -- 172.21.15.40:0/1872471870 >> 172.21.15.40:0/1872471870 conn(0x7fc0940d4f90 msgr2=0x7fc0940d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.799+0000 7fc09c000640 1 -- 172.21.15.40:0/1872471870 shutdown_connections 2024-04-26T23:09:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:37.799+0000 7fc09c000640 1 -- 172.21.15.40:0/1872471870 wait complete. 2024-04-26T23:09:37.815 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:09:37.816 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.7:1 scrub status 2024-04-26T23:09:38.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.011+0000 7fdf36e00640 1 Processor -- start 2024-04-26T23:09:38.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.011+0000 7fdf36e00640 1 -- start start 2024-04-26T23:09:38.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.013+0000 7fdf36e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf300ce210 0x7fdf300d3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.013+0000 7fdf36e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf300ce780 con 0x7fdf300b2d00 2024-04-26T23:09:38.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.013+0000 7fdf36e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf300cea60 con 0x7fdf300d11d0 2024-04-26T23:09:38.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.013+0000 7fdf36e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdf300b2220 con 0x7fdf300ce210 2024-04-26T23:09:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.014+0000 7fdf36400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdf300b2d00 0x7fdf300b3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44850/0 (socket says 172.21.15.40:44850) 2024-04-26T23:09:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.014+0000 7fdf36400640 1 -- 172.21.15.40:0/451133778 learned_addr learned my addr 172.21.15.40:0/451133778 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.014+0000 7fdf35000640 1 --2- 172.21.15.40:0/451133778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf300ce210 0x7fdf300d3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.014+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3230696859 0 0) 0x7fdf300ce780 con 0x7fdf300b2d00 2024-04-26T23:09:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.014+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf18003520 con 0x7fdf300b2d00 2024-04-26T23:09:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.014+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1554608159 0 0) 0x7fdf18003520 con 0x7fdf300b2d00 2024-04-26T23:09:38.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.014+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf300ce210 msgr2=0x7fdf300d3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf2fe00640 1 --2- 172.21.15.40:0/451133778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf300ce210 0x7fdf300d3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 >> v1:172.21.15.103:6789/0 conn(0x7fdf300d11d0 legacy=0x7fdf300d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf300b2500 con 0x7fdf300b2d00 2024-04-26T23:09:38.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fdf280019f0 con 0x7fdf300b2d00 2024-04-26T23:09:38.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 >> v1:172.21.15.40:6789/0 conn(0x7fdf300b2d00 legacy=0x7fdf300b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf2fe00640 1 --2- 172.21.15.40:0/451133778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf180039e0 0x7fdf18003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf2fe00640 1 --2- 172.21.15.40:0/451133778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf300ce210 0x7fdf18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf2fe00640 1 --2- 172.21.15.40:0/451133778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf18004d40 0x7fdf180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf300b2220 con 0x7fdf18004d40 2024-04-26T23:09:38.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1800ba70 con 0x7fdf180039e0 2024-04-26T23:09:38.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdf1800bc70 con 0x7fdf300ce210 2024-04-26T23:09:38.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf36400640 1 --2- 172.21.15.40:0/451133778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf18004d40 0x7fdf180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf35a00640 1 --2- 172.21.15.40:0/451133778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf180039e0 0x7fdf18003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf36400640 1 -- 172.21.15.40:0/451133778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf300ce210 msgr2=0x7fdf18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf36400640 1 --2- 172.21.15.40:0/451133778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf300ce210 0x7fdf18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf36400640 1 -- 172.21.15.40:0/451133778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf180039e0 msgr2=0x7fdf18003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.015+0000 7fdf36400640 1 --2- 172.21.15.40:0/451133778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf180039e0 0x7fdf18003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.016+0000 7fdf36400640 1 -- 172.21.15.40:0/451133778 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf18003520 con 0x7fdf18004d40 2024-04-26T23:09:38.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.016+0000 7fdf35a00640 1 --2- 172.21.15.40:0/451133778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf180039e0 0x7fdf18003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:38.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.016+0000 7fdf36400640 1 --2- 172.21.15.40:0/451133778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf18004d40 0x7fdf180051c0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fdf28002230 tx=0x7fdf18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ff703cb0487396 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.017+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2800a4a0 con 0x7fdf18004d40 2024-04-26T23:09:38.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.017+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf2802f070 con 0x7fdf18004d40 2024-04-26T23:09:38.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.017+0000 7fdf2fe00640 1 -- 172.21.15.40:0/451133778 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2802bcb0 con 0x7fdf18004d40 2024-04-26T23:09:38.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.017+0000 7fdf36e00640 1 -- 172.21.15.40:0/451133778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf18004d40 msgr2=0x7fdf180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.017+0000 7fdf36e00640 1 --2- 172.21.15.40:0/451133778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf18004d40 0x7fdf180051c0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fdf28002230 tx=0x7fdf18005830 comp rx=0 tx=0).stop 2024-04-26T23:09:38.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.017+0000 7fdf36400640 1 -- 172.21.15.40:0/451133778 reap_dead start 2024-04-26T23:09:38.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.017+0000 7fdf36e00640 1 -- 172.21.15.40:0/451133778 shutdown_connections 2024-04-26T23:09:38.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.017+0000 7fdf36e00640 1 -- 172.21.15.40:0/451133778 >> 172.21.15.40:0/451133778 conn(0x7fdf300d2bf0 msgr2=0x7fdf300d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.018+0000 7fdf36e00640 1 -- 172.21.15.40:0/451133778 shutdown_connections 2024-04-26T23:09:38.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.018+0000 7fdf36e00640 1 -- 172.21.15.40:0/451133778 wait complete. 2024-04-26T23:09:38.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.019+0000 7fdf36e00640 1 Processor -- start 2024-04-26T23:09:38.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.019+0000 7fdf36e00640 1 -- start start 2024-04-26T23:09:38.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.020+0000 7fdf36e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf18004d40 0x7fdf3009cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.020+0000 7fdf36e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf30156900 0x7fdf3014fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.020+0000 7fdf36e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf300cb860 0x7fdf300cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.020+0000 7fdf36e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3015d450 con 0x7fdf300cb860 2024-04-26T23:09:38.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.020+0000 7fdf36e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf300d0c80 con 0x7fdf18004d40 2024-04-26T23:09:38.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.020+0000 7fdf36e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdf300d3710 con 0x7fdf30156900 2024-04-26T23:09:38.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.021+0000 7fdf36400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf300cb860 0x7fdf300cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.021+0000 7fdf36400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf300cb860 0x7fdf300cbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43250/0 (socket says 172.21.15.40:43250) 2024-04-26T23:09:38.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.021+0000 7fdf36400640 1 -- 172.21.15.40:0/311644245 learned_addr learned my addr 172.21.15.40:0/311644245 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.021+0000 7fdf35a00640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf18004d40 0x7fdf3009cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.021+0000 7fdf35000640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf30156900 0x7fdf3014fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.021+0000 7fdf36400640 1 -- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf30156900 msgr2=0x7fdf3014fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.021+0000 7fdf36400640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf30156900 0x7fdf3014fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.021+0000 7fdf36400640 1 -- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf18004d40 msgr2=0x7fdf3009cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.021+0000 7fdf36400640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf18004d40 0x7fdf3009cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.021+0000 7fdf36400640 1 -- 172.21.15.40:0/311644245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf30150400 con 0x7fdf300cb860 2024-04-26T23:09:38.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.022+0000 7fdf36400640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf300cb860 0x7fdf300cbce0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fdf2802b500 tx=0x7fdf28001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.022+0000 7fdf2d600640 1 -- 172.21.15.40:0/311644245 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2802bc60 con 0x7fdf300cb860 2024-04-26T23:09:38.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.022+0000 7fdf2d600640 1 -- 172.21.15.40:0/311644245 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf28001250 con 0x7fdf300cb860 2024-04-26T23:09:38.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.022+0000 7fdf2d600640 1 -- 172.21.15.40:0/311644245 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2800a480 con 0x7fdf300cb860 2024-04-26T23:09:38.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.022+0000 7fdf36e00640 1 -- 172.21.15.40:0/311644245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf30157810 con 0x7fdf300cb860 2024-04-26T23:09:38.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.022+0000 7fdf36e00640 1 -- 172.21.15.40:0/311644245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf30157250 con 0x7fdf300cb860 2024-04-26T23:09:38.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.024+0000 7fdf2d600640 1 -- 172.21.15.40:0/311644245 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdf28034470 con 0x7fdf300cb860 2024-04-26T23:09:38.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.025+0000 7fdf2d600640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdf1006c920 0x7fdf1006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.025+0000 7fdf2d600640 1 -- 172.21.15.40:0/311644245 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fdf280abf20 con 0x7fdf300cb860 2024-04-26T23:09:38.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.026+0000 7fdf35a00640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdf1006c920 0x7fdf1006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.027+0000 7fdf35a00640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdf1006c920 0x7fdf1006ede0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fdf301568b0 tx=0x7fdf24009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.027+0000 7fdf3b3d4740 1 Processor -- start 2024-04-26T23:09:38.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.028+0000 7fdf3b3d4740 1 -- start start 2024-04-26T23:09:38.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.029+0000 7fdf3b3d4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1718710 0x55e3f1718af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.029+0000 7fdf3b3d4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 0x55e3f1736500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.029+0000 7fdf3b3d4740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f16ee3f0 0x55e3f1734670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.029+0000 7fdf3b3d4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e3f15fe030 con 0x55e3f1735fb0 2024-04-26T23:09:38.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.029+0000 7fdf3b3d4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e3f16197f0 con 0x55e3f1718710 2024-04-26T23:09:38.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.029+0000 7fdf3b3d4740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e3f15d47e0 con 0x55e3f16ee3f0 2024-04-26T23:09:38.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.029+0000 7fdf35000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 0x55e3f1736500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.029+0000 7fdf35000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 0x55e3f1736500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43264/0 (socket says 172.21.15.40:43264) 2024-04-26T23:09:38.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.029+0000 7fdf35000640 1 -- 172.21.15.40:0/1293825495 learned_addr learned my addr 172.21.15.40:0/1293825495 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.030+0000 7fdf35a00640 1 --2- 172.21.15.40:0/1293825495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1718710 0x55e3f1718af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.030+0000 7fdf35000640 1 -- 172.21.15.40:0/1293825495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f16ee3f0 msgr2=0x55e3f1734670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.030+0000 7fdf35000640 1 --2- 172.21.15.40:0/1293825495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f16ee3f0 0x55e3f1734670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.030+0000 7fdf35000640 1 -- 172.21.15.40:0/1293825495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1718710 msgr2=0x55e3f1718af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.030+0000 7fdf35000640 1 --2- 172.21.15.40:0/1293825495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1718710 0x55e3f1718af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.030+0000 7fdf35000640 1 -- 172.21.15.40:0/1293825495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3f15e0050 con 0x55e3f1735fb0 2024-04-26T23:09:38.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.030+0000 7fdf35a00640 1 --2- 172.21.15.40:0/1293825495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1718710 0x55e3f1718af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:38.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.030+0000 7fdf35000640 1 --2- 172.21.15.40:0/1293825495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 0x55e3f1736500 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009a60 tx=0x7fdf1c009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74bc0ef97a07c09b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.030+0000 7fdf22000640 1 -- 172.21.15.40:0/1293825495 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c018070 con 0x55e3f1735fb0 2024-04-26T23:09:38.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.030+0000 7fdf22000640 1 -- 172.21.15.40:0/1293825495 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf1c010d60 con 0x55e3f1735fb0 2024-04-26T23:09:38.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.030+0000 7fdf22000640 1 -- 172.21.15.40:0/1293825495 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c00f040 con 0x55e3f1735fb0 2024-04-26T23:09:38.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.031+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/1293825495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 msgr2=0x55e3f1736500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.031+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/1293825495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 0x55e3f1736500 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009a60 tx=0x7fdf1c009f20 comp rx=0 tx=0).stop 2024-04-26T23:09:38.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.031+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/1293825495 shutdown_connections 2024-04-26T23:09:38.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.031+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/1293825495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f16ee3f0 0x55e3f1734670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.031+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/1293825495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 0x55e3f1736500 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.031+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/1293825495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1718710 0x55e3f1718af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.031+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/1293825495 >> 172.21.15.40:0/1293825495 conn(0x55e3f16ead20 msgr2=0x55e3f1733520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.031+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/1293825495 shutdown_connections 2024-04-26T23:09:38.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.031+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/1293825495 wait complete. 2024-04-26T23:09:38.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.032+0000 7fdf3b3d4740 1 -- start start 2024-04-26T23:09:38.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.032+0000 7fdf3b3d4740 1 Processor -- start 2024-04-26T23:09:38.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.033+0000 7fdf3b3d4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1718710 0x55e3f1716d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.033+0000 7fdf3b3d4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 0x55e3f17061f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.033+0000 7fdf3b3d4740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f171b580 0x55e3f16e7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.033+0000 7fdf3b3d4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e3f1656200 con 0x55e3f1735fb0 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.033+0000 7fdf3b3d4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e3f1650ee0 con 0x55e3f1718710 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.033+0000 7fdf3b3d4740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e3f165de70 con 0x55e3f171b580 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.033+0000 7fdf3b3d4740 10 client.0 _ll_register_callbacks cb 0x55e3f1647270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.034+0000 7fdf35000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 0x55e3f17061f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.034+0000 7fdf35000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 0x55e3f17061f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43268/0 (socket says 172.21.15.40:43268) 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.034+0000 7fdf35000640 1 -- 172.21.15.40:0/3226406731 learned_addr learned my addr 172.21.15.40:0/3226406731 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.034+0000 7fdf35a00640 1 --2- 172.21.15.40:0/3226406731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1718710 0x55e3f1716d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.034+0000 7fdf36400640 1 --2- 172.21.15.40:0/3226406731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f171b580 0x55e3f16e7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.034+0000 7fdf35000640 1 -- 172.21.15.40:0/3226406731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f171b580 msgr2=0x55e3f16e7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.034+0000 7fdf35000640 1 --2- 172.21.15.40:0/3226406731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f171b580 0x55e3f16e7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.034+0000 7fdf35000640 1 -- 172.21.15.40:0/3226406731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1718710 msgr2=0x55e3f1716d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.034+0000 7fdf35000640 1 --2- 172.21.15.40:0/3226406731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1718710 0x55e3f1716d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.034+0000 7fdf35000640 1 -- 172.21.15.40:0/3226406731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3f142d7d0 con 0x55e3f1735fb0 2024-04-26T23:09:38.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.034+0000 7fdf35000640 1 --2- 172.21.15.40:0/3226406731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 0x55e3f17061f0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c000c00 tx=0x7fdf1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b25b07da235dbec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.034+0000 7fdf16a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf16a00640 1 -- 172.21.15.40:0/3226406731 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c018050 con 0x55e3f1735fb0 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf16a00640 1 -- 172.21.15.40:0/3226406731 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fdf1c003820 con 0x55e3f1735fb0 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/3226406731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf1c010d60 con 0x55e3f1735fb0 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf16a00640 1 -- 172.21.15.40:0/3226406731 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fdf1c003a20 con 0x55e3f1735fb0 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf3b3d4740 10 client.9345 fetch_fsmap learned FSMap version 119 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/3226406731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) -- 0x55e3f13633f0 con 0x55e3f1735fb0 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf16a00640 1 -- 172.21.15.40:0/3226406731 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf1c010d60 con 0x55e3f1735fb0 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf16a00640 1 -- 172.21.15.40:0/3226406731 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 119) ==== 3550+0+0 (secure 0 0 0) 0x7fdf1c032020 con 0x55e3f1735fb0 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf3b3d4740 10 client.9345 fetch_fsmap finished waiting for FSMap version 119 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf3b3d4740 10 client.9345 resolve_mds: resolved 7:1 to role '7:1' aka daemon mds.c 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf3b3d4740 20 client.9345 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/3226406731 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55e3f1719de0 0x55e3f17038a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf3b3d4740 4 client.9345 mds_command: new command op to 8949 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:38.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.035+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/3226406731 --> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3f165f7e0 con 0x55e3f1719de0 2024-04-26T23:09:38.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.036+0000 7fdf35a00640 1 --2- 172.21.15.40:0/3226406731 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55e3f1719de0 0x55e3f17038a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.036+0000 7fdf35a00640 1 --2- 172.21.15.40:0/3226406731 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55e3f1719de0 0x55e3f17038a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d2b5eaceffc13576 server_cookie=7daafc77d2a02bd6 in_seq=0 out_seq=0 2024-04-26T23:09:38.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.036+0000 7fdf16a00640 10 client.9345 ms_handle_connect on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:38.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.037+0000 7fdf16a00640 1 -- 172.21.15.40:0/3226406731 <== mds.1 v2:172.21.15.40:6832/3250599834 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e3f165f7e0 con 0x55e3f1719de0 2024-04-26T23:09:38.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.037+0000 7fdf16a00640 10 client.9345 handle_command_reply: tid=0 2024-04-26T23:09:38.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.037+0000 7fdf3b3d4740 1 client.9345 shutdown 2024-04-26T23:09:38.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.037+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/3226406731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 msgr2=0x55e3f17061f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.037+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/3226406731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 0x55e3f17061f0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c000c00 tx=0x7fdf1c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.037+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/3226406731 shutdown_connections 2024-04-26T23:09:38.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.037+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/3226406731 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55e3f1719de0 0x55e3f17038a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.037+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/3226406731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f171b580 0x55e3f16e7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.037+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/3226406731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1735fb0 0x55e3f17061f0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.037+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/3226406731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1718710 0x55e3f1716d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.037+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/3226406731 >> 172.21.15.40:0/3226406731 conn(0x55e3f16ee3f0 msgr2=0x55e3f16ed970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.037+0000 7fdf16a00640 0 client.9345 ms_handle_reset on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.038+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/3226406731 shutdown_connections 2024-04-26T23:09:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.038+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/3226406731 wait complete. 2024-04-26T23:09:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.038+0000 7fdf3b3d4740 20 client.9345 trim_cache size 0 max 16384 2024-04-26T23:09:38.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.057+0000 7fdf3b3d4740 1 Processor -- start 2024-04-26T23:09:38.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.057+0000 7fdf3b3d4740 1 -- start start 2024-04-26T23:09:38.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf3b3d4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1718710 0x55e3f1733710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf3b3d4740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f1735fb0 0x55e3f1715320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf3b3d4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1717c40 0x55e3f1715860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf3b3d4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf302001a0 con 0x55e3f1718710 2024-04-26T23:09:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf3b3d4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf301ff340 con 0x55e3f1717c40 2024-04-26T23:09:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf3b3d4740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdf301ffa70 con 0x55e3f1735fb0 2024-04-26T23:09:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf35a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1718710 0x55e3f1733710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf35a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1718710 0x55e3f1733710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43282/0 (socket says 172.21.15.40:43282) 2024-04-26T23:09:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf35a00640 1 -- 172.21.15.40:0/4257296544 learned_addr learned my addr 172.21.15.40:0/4257296544 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf35a00640 1 -- 172.21.15.40:0/4257296544 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f1735fb0 msgr2=0x55e3f1715320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf35a00640 1 --2- 172.21.15.40:0/4257296544 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f1735fb0 0x55e3f1715320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf35a00640 1 -- 172.21.15.40:0/4257296544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1717c40 msgr2=0x55e3f1715860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf35a00640 1 --2- 172.21.15.40:0/4257296544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1717c40 0x55e3f1715860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.058+0000 7fdf35a00640 1 -- 172.21.15.40:0/4257296544 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3f16e7cd0 con 0x55e3f1718710 2024-04-26T23:09:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.059+0000 7fdf35a00640 1 --2- 172.21.15.40:0/4257296544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1718710 0x55e3f1733710 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fdf24005590 tx=0x7fdf24012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6fb608427104b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.059+0000 7fdf21600640 1 -- 172.21.15.40:0/4257296544 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf24019040 con 0x55e3f1718710 2024-04-26T23:09:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.059+0000 7fdf21600640 1 -- 172.21.15.40:0/4257296544 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf2402a020 con 0x55e3f1718710 2024-04-26T23:09:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.059+0000 7fdf21600640 1 -- 172.21.15.40:0/4257296544 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf24003770 con 0x55e3f1718710 2024-04-26T23:09:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.059+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/4257296544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1718710 msgr2=0x55e3f1733710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.059+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/4257296544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1718710 0x55e3f1733710 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fdf24005590 tx=0x7fdf24012000 comp rx=0 tx=0).stop 2024-04-26T23:09:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.059+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/4257296544 shutdown_connections 2024-04-26T23:09:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.059+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/4257296544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1717c40 0x55e3f1715860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.059+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/4257296544 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f1735fb0 0x55e3f1715320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.059+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/4257296544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1718710 0x55e3f1733710 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.059+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/4257296544 >> 172.21.15.40:0/4257296544 conn(0x55e3f16ee3f0 msgr2=0x55e3f1706000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.060+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/4257296544 shutdown_connections 2024-04-26T23:09:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.060+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/4257296544 wait complete. 2024-04-26T23:09:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.060+0000 7fdf3b3d4740 1 -- start start 2024-04-26T23:09:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.060+0000 7fdf3b3d4740 1 Processor -- start 2024-04-26T23:09:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.061+0000 7fdf3b3d4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1735fb0 0x55e3f1715250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.061+0000 7fdf3b3d4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1717190 0x55e3f1715790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.061+0000 7fdf3b3d4740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f17355c0 0x55e3f1746f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.061+0000 7fdf3b3d4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e3f165efd0 con 0x55e3f1717190 2024-04-26T23:09:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.061+0000 7fdf3b3d4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf301fe380 con 0x55e3f1735fb0 2024-04-26T23:09:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.061+0000 7fdf3b3d4740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55e3f172ad20 con 0x55e3f17355c0 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.061+0000 7fdf3b3d4740 10 client.0 _ll_register_callbacks cb 0x7fdf1c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.062+0000 7fdf35000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1717190 0x55e3f1715790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.062+0000 7fdf35000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1717190 0x55e3f1715790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43296/0 (socket says 172.21.15.40:43296) 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.062+0000 7fdf35000640 1 -- 172.21.15.40:0/1019714618 learned_addr learned my addr 172.21.15.40:0/1019714618 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.062+0000 7fdf35000640 1 -- 172.21.15.40:0/1019714618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f17355c0 msgr2=0x55e3f1746f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.062+0000 7fdf36400640 1 --2- 172.21.15.40:0/1019714618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f17355c0 0x55e3f1746f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.062+0000 7fdf35a00640 1 --2- 172.21.15.40:0/1019714618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1735fb0 0x55e3f1715250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.062+0000 7fdf35000640 1 --2- 172.21.15.40:0/1019714618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f17355c0 0x55e3f1746f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.062+0000 7fdf35000640 1 -- 172.21.15.40:0/1019714618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1735fb0 msgr2=0x55e3f1715250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.062+0000 7fdf35000640 1 --2- 172.21.15.40:0/1019714618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1735fb0 0x55e3f1715250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.062+0000 7fdf35000640 1 -- 172.21.15.40:0/1019714618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3f13230f0 con 0x55e3f1717190 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.062+0000 7fdf35000640 1 --2- 172.21.15.40:0/1019714618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1717190 0x55e3f1715790 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c008010 tx=0x7fdf1c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca64e94424bd530 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.062+0000 7fdf16a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:38.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf16a00640 1 -- 172.21.15.40:0/1019714618 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c018040 con 0x55e3f1717190 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf16a00640 1 -- 172.21.15.40:0/1019714618 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fdf1c01d660 con 0x55e3f1717190 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/1019714618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf2402a020 con 0x55e3f1717190 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf16a00640 1 -- 172.21.15.40:0/1019714618 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fdf1c040050 con 0x55e3f1717190 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf3b3d4740 10 client.9351 fetch_fsmap learned FSMap version 119 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/1019714618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) -- 0x55e3f1619a30 con 0x55e3f1717190 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf16a00640 1 -- 172.21.15.40:0/1019714618 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf2402a020 con 0x55e3f1717190 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf16a00640 1 -- 172.21.15.40:0/1019714618 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 119) ==== 3550+0+0 (secure 0 0 0) 0x7fdf1c043020 con 0x55e3f1717190 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf3b3d4740 10 client.9351 fetch_fsmap finished waiting for FSMap version 119 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf3b3d4740 10 client.9351 resolve_mds: resolved 7:1 to role '7:1' aka daemon mds.c 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf3b3d4740 20 client.9351 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/1019714618 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55e3f1704920 0x55e3f1747460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf3b3d4740 4 client.9351 mds_command: new command op to 8949 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.063+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/1019714618 --> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e3f1659650 con 0x55e3f1704920 2024-04-26T23:09:38.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.064+0000 7fdf35a00640 1 --2- 172.21.15.40:0/1019714618 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55e3f1704920 0x55e3f1747460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.064+0000 7fdf35a00640 1 --2- 172.21.15.40:0/1019714618 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55e3f1704920 0x55e3f1747460 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc1b3116e9fe5780 server_cookie=7218635775e43b12 in_seq=0 out_seq=0 2024-04-26T23:09:38.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.064+0000 7fdf16a00640 10 client.9351 ms_handle_connect on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:38.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.064+0000 7fdf16a00640 1 -- 172.21.15.40:0/1019714618 <== mds.1 v2:172.21.15.40:6832/3250599834 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55e3f1659650 con 0x55e3f1704920 2024-04-26T23:09:38.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.064+0000 7fdf16a00640 10 client.9351 handle_command_reply: tid=0 2024-04-26T23:09:38.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.064+0000 7fdf3b3d4740 1 client.9351 shutdown 2024-04-26T23:09:38.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.065+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/1019714618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1717190 msgr2=0x55e3f1715790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.065+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/1019714618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1717190 0x55e3f1715790 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c008010 tx=0x7fdf1c009190 comp rx=0 tx=0).stop 2024-04-26T23:09:38.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.065+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/1019714618 shutdown_connections 2024-04-26T23:09:38.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.065+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/1019714618 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55e3f1704920 0x55e3f1747460 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.065+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/1019714618 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55e3f17355c0 0x55e3f1746f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.065+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/1019714618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e3f1717190 0x55e3f1715790 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.065+0000 7fdf3b3d4740 1 --2- 172.21.15.40:0/1019714618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e3f1735fb0 0x55e3f1715250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.065+0000 7fdf16a00640 0 client.9351 ms_handle_reset on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.065+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/1019714618 >> 172.21.15.40:0/1019714618 conn(0x55e3f16ee3f0 msgr2=0x55e3f16e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.065+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/1019714618 shutdown_connections 2024-04-26T23:09:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.065+0000 7fdf3b3d4740 1 -- 172.21.15.40:0/1019714618 wait complete. 2024-04-26T23:09:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.065+0000 7fdf3b3d4740 20 client.9351 trim_cache size 0 max 16384 2024-04-26T23:09:38.069 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:38.069 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-26T23:09:38.069 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:09:38.069 INFO:teuthology.orchestra.run.smithi040.stdout: "a9827531-70a9-4d6f-b12a-2d71fa886d03": { 2024-04-26T23:09:38.070 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:09:38.070 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "a9827531-70a9-4d6f-b12a-2d71fa886d03", 2024-04-26T23:09:38.070 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:09:38.070 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:38.070 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:38.070 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.066+0000 7fdf36e00640 1 -- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdf1006c920 msgr2=0x7fdf1006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.066+0000 7fdf36e00640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdf1006c920 0x7fdf1006ede0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fdf301568b0 tx=0x7fdf24009040 comp rx=0 tx=0).stop 2024-04-26T23:09:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.066+0000 7fdf36e00640 1 -- 172.21.15.40:0/311644245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf300cb860 msgr2=0x7fdf300cbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.066+0000 7fdf36e00640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf300cb860 0x7fdf300cbce0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fdf2802b500 tx=0x7fdf28001dc0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.066+0000 7fdf36e00640 1 -- 172.21.15.40:0/311644245 shutdown_connections 2024-04-26T23:09:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.066+0000 7fdf36e00640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fdf1006c920 0x7fdf1006ede0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.066+0000 7fdf36e00640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf300cb860 0x7fdf300cbce0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.066+0000 7fdf36e00640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf30156900 0x7fdf3014fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.066+0000 7fdf36e00640 1 --2- 172.21.15.40:0/311644245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf18004d40 0x7fdf3009cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.066+0000 7fdf36e00640 1 -- 172.21.15.40:0/311644245 >> 172.21.15.40:0/311644245 conn(0x7fdf300d2bf0 msgr2=0x7fdf300cc910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.066+0000 7fdf36e00640 1 -- 172.21.15.40:0/311644245 shutdown_connections 2024-04-26T23:09:38.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.066+0000 7fdf36e00640 1 -- 172.21.15.40:0/311644245 wait complete. 2024-04-26T23:09:38.084 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:09:38.084 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:09:38.084 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.7:2 scrub status 2024-04-26T23:09:38.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.284+0000 7fa107200640 1 Processor -- start 2024-04-26T23:09:38.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.284+0000 7fa107200640 1 -- start start 2024-04-26T23:09:38.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.285+0000 7fa107200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa1080d1770 0x7fa10815ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.285+0000 7fa107200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1080d1b50 0x7fa10815f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.285+0000 7fa107200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1080d5d30 con 0x7fa1080b6470 2024-04-26T23:09:38.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.285+0000 7fa107200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa1080d3760 con 0x7fa1080d1770 2024-04-26T23:09:38.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.285+0000 7fa107200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa1080d09f0 con 0x7fa1080d1b50 2024-04-26T23:09:38.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.286+0000 7fa105400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa1080b6470 0x7fa1080b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44858/0 (socket says 172.21.15.40:44858) 2024-04-26T23:09:38.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.286+0000 7fa105400640 1 -- 172.21.15.40:0/435362523 learned_addr learned my addr 172.21.15.40:0/435362523 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.286+0000 7fa106800640 1 --2- 172.21.15.40:0/435362523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1080d1b50 0x7fa10815f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.286+0000 7fa105e00640 1 --2- 172.21.15.40:0/435362523 >> v2:172.21.15.103:3300/0 conn(0x7fa1080d1770 0x7fa10815ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.286+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3071978674 0 0) 0x7fa1080d5d30 con 0x7fa1080b6470 2024-04-26T23:09:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.286+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0e8003520 con 0x7fa1080b6470 2024-04-26T23:09:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2260503696 0 0) 0x7fa0e8003520 con 0x7fa1080b6470 2024-04-26T23:09:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1080d1b50 msgr2=0x7fa10815f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 --2- 172.21.15.40:0/435362523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1080d1b50 0x7fa10815f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 >> v2:172.21.15.103:3300/0 conn(0x7fa1080d1770 msgr2=0x7fa10815ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 --2- 172.21.15.40:0/435362523 >> v2:172.21.15.103:3300/0 conn(0x7fa1080d1770 0x7fa10815ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1080d5a70 con 0x7fa1080b6470 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fa0f0001670 con 0x7fa1080b6470 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 >> v1:172.21.15.40:6789/0 conn(0x7fa1080b6470 legacy=0x7fa1080b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 --2- 172.21.15.40:0/435362523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0e80039b0 0x7fa0e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 --2- 172.21.15.40:0/435362523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1080d1b50 0x7fa0e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 --2- 172.21.15.40:0/435362523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0e8004d40 0x7fa0e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa1080d3760 con 0x7fa0e80039b0 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1080d09f0 con 0x7fa0e8004d40 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa0e800b9c0 con 0x7fa1080d1b50 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.287+0000 7fa105e00640 1 --2- 172.21.15.40:0/435362523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0e80039b0 0x7fa0e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.288+0000 7fa105e00640 1 -- 172.21.15.40:0/435362523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1080d1b50 msgr2=0x7fa0e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.288+0000 7fa105e00640 1 --2- 172.21.15.40:0/435362523 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1080d1b50 0x7fa0e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.288+0000 7fa105e00640 1 -- 172.21.15.40:0/435362523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0e8004d40 msgr2=0x7fa0e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.288+0000 7fa105e00640 1 --2- 172.21.15.40:0/435362523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0e8004d40 0x7fa0e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.288+0000 7fa105e00640 1 -- 172.21.15.40:0/435362523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e8003520 con 0x7fa0e80039b0 2024-04-26T23:09:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.289+0000 7fa105e00640 1 --2- 172.21.15.40:0/435362523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0e80039b0 0x7fa0e8003dd0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009ac0 tx=0x7fa0e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68fc2530dfec8719 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.289+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8036070 con 0x7fa0e80039b0 2024-04-26T23:09:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.289+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f8031400 con 0x7fa0e80039b0 2024-04-26T23:09:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.289+0000 7fa104a00640 1 -- 172.21.15.40:0/435362523 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8009690 con 0x7fa0e80039b0 2024-04-26T23:09:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.290+0000 7fa107200640 1 -- 172.21.15.40:0/435362523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0e80039b0 msgr2=0x7fa0e8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.290+0000 7fa107200640 1 --2- 172.21.15.40:0/435362523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0e80039b0 0x7fa0e8003dd0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009ac0 tx=0x7fa0e8005830 comp rx=0 tx=0).stop 2024-04-26T23:09:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.290+0000 7fa106800640 1 -- 172.21.15.40:0/435362523 reap_dead start 2024-04-26T23:09:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.290+0000 7fa107200640 1 -- 172.21.15.40:0/435362523 shutdown_connections 2024-04-26T23:09:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.290+0000 7fa107200640 1 -- 172.21.15.40:0/435362523 >> 172.21.15.40:0/435362523 conn(0x7fa1080d4dd0 msgr2=0x7fa1080d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.290+0000 7fa107200640 1 -- 172.21.15.40:0/435362523 shutdown_connections 2024-04-26T23:09:38.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.290+0000 7fa107200640 1 -- 172.21.15.40:0/435362523 wait complete. 2024-04-26T23:09:38.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.291+0000 7fa107200640 1 Processor -- start 2024-04-26T23:09:38.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.292+0000 7fa107200640 1 -- start start 2024-04-26T23:09:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.292+0000 7fa107200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0e80039b0 0x7fa1081f6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa107200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1080b96e0 0x7fa1081f58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa107200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1081f7cb0 0x7fa1081e8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa107200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa1080d3bf0 con 0x7fa0e80039b0 2024-04-26T23:09:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa107200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1080d2670 con 0x7fa1081f7cb0 2024-04-26T23:09:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa107200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa1080d3d70 con 0x7fa1080b96e0 2024-04-26T23:09:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa105e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0e80039b0 0x7fa1081f6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa105e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0e80039b0 0x7fa1081f6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43320/0 (socket says 172.21.15.40:43320) 2024-04-26T23:09:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa105e00640 1 -- 172.21.15.40:0/254980930 learned_addr learned my addr 172.21.15.40:0/254980930 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa105400640 1 --2- 172.21.15.40:0/254980930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1080b96e0 0x7fa1081f58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa105e00640 1 -- 172.21.15.40:0/254980930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1080b96e0 msgr2=0x7fa1081f58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa105e00640 1 --2- 172.21.15.40:0/254980930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1080b96e0 0x7fa1081f58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa105e00640 1 -- 172.21.15.40:0/254980930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1081f7cb0 msgr2=0x7fa1081e8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa105e00640 1 --2- 172.21.15.40:0/254980930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1081f7cb0 0x7fa1081e8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.293+0000 7fa105e00640 1 -- 172.21.15.40:0/254980930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1081f6550 con 0x7fa0e80039b0 2024-04-26T23:09:38.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.294+0000 7fa105e00640 1 --2- 172.21.15.40:0/254980930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0e80039b0 0x7fa1081f6010 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009ac0 tx=0x7fa0f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.294+0000 7fa0fe000640 1 -- 172.21.15.40:0/254980930 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8040020 con 0x7fa0e80039b0 2024-04-26T23:09:38.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.294+0000 7fa0fe000640 1 -- 172.21.15.40:0/254980930 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f80038a0 con 0x7fa0e80039b0 2024-04-26T23:09:38.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.294+0000 7fa0fe000640 1 -- 172.21.15.40:0/254980930 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8036040 con 0x7fa0e80039b0 2024-04-26T23:09:38.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.294+0000 7fa107200640 1 -- 172.21.15.40:0/254980930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1081f5e10 con 0x7fa0e80039b0 2024-04-26T23:09:38.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.294+0000 7fa107200640 1 -- 172.21.15.40:0/254980930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa108154f00 con 0x7fa0e80039b0 2024-04-26T23:09:38.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.297+0000 7fa0fe000640 1 -- 172.21.15.40:0/254980930 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa0f8043020 con 0x7fa0e80039b0 2024-04-26T23:09:38.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.298+0000 7fa0fe000640 1 --2- 172.21.15.40:0/254980930 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa0e406c8f0 0x7fa0e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.298+0000 7fa0fe000640 1 -- 172.21.15.40:0/254980930 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fa0f80abb90 con 0x7fa0e80039b0 2024-04-26T23:09:38.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.298+0000 7fa10f96b740 1 Processor -- start 2024-04-26T23:09:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.298+0000 7fa105400640 1 --2- 172.21.15.40:0/254980930 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa0e406c8f0 0x7fa0e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.299+0000 7fa10f96b740 1 -- start start 2024-04-26T23:09:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa10f96b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22c710 0x561cef22caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa10f96b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef249fb0 0x561cef24a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa10f96b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef2023f0 0x561cef248670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa10f96b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561cef112030 con 0x561cef2023f0 2024-04-26T23:09:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa10f96b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561cef12d7f0 con 0x561cef22c710 2024-04-26T23:09:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa10f96b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561cef0e87e0 con 0x561cef249fb0 2024-04-26T23:09:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa105400640 1 --2- 172.21.15.40:0/254980930 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa0e406c8f0 0x7fa0e406edb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0000f60 tx=0x7fa0f0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa106800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef2023f0 0x561cef248670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa106800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef2023f0 0x561cef248670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43324/0 (socket says 172.21.15.40:43324) 2024-04-26T23:09:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa106800640 1 -- 172.21.15.40:0/3941112343 learned_addr learned my addr 172.21.15.40:0/3941112343 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa106800640 1 -- 172.21.15.40:0/3941112343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef249fb0 msgr2=0x561cef24a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa106800640 1 --2- 172.21.15.40:0/3941112343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef249fb0 0x561cef24a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa106800640 1 -- 172.21.15.40:0/3941112343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22c710 msgr2=0x561cef22caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa105e00640 1 --2- 172.21.15.40:0/3941112343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22c710 0x561cef22caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa106800640 1 --2- 172.21.15.40:0/3941112343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22c710 0x561cef22caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa106800640 1 -- 172.21.15.40:0/3941112343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cef0f4050 con 0x561cef2023f0 2024-04-26T23:09:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.300+0000 7fa105e00640 1 --2- 172.21.15.40:0/3941112343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22c710 0x561cef22caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.301+0000 7fa106800640 1 --2- 172.21.15.40:0/3941112343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef2023f0 0x561cef248670 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fa1080bae80 tx=0x7fa10000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125d49a58ba74ed4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.301+0000 7fa0f6600640 1 -- 172.21.15.40:0/3941112343 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa100018030 con 0x561cef2023f0 2024-04-26T23:09:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.301+0000 7fa0f6600640 1 -- 172.21.15.40:0/3941112343 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa10000edb0 con 0x561cef2023f0 2024-04-26T23:09:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.301+0000 7fa0f6600640 1 -- 172.21.15.40:0/3941112343 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa100003bd0 con 0x561cef2023f0 2024-04-26T23:09:38.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.301+0000 7fa10f96b740 1 -- 172.21.15.40:0/3941112343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef2023f0 msgr2=0x561cef248670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.301+0000 7fa10f96b740 1 --2- 172.21.15.40:0/3941112343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef2023f0 0x561cef248670 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fa1080bae80 tx=0x7fa10000b040 comp rx=0 tx=0).stop 2024-04-26T23:09:38.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.302+0000 7fa10f96b740 1 -- 172.21.15.40:0/3941112343 shutdown_connections 2024-04-26T23:09:38.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.302+0000 7fa10f96b740 1 --2- 172.21.15.40:0/3941112343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef2023f0 0x561cef248670 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.302+0000 7fa10f96b740 1 --2- 172.21.15.40:0/3941112343 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef249fb0 0x561cef24a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.302+0000 7fa10f96b740 1 --2- 172.21.15.40:0/3941112343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22c710 0x561cef22caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.302+0000 7fa10f96b740 1 -- 172.21.15.40:0/3941112343 >> 172.21.15.40:0/3941112343 conn(0x561cef1fed20 msgr2=0x561cef247520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.302+0000 7fa10f96b740 1 -- 172.21.15.40:0/3941112343 shutdown_connections 2024-04-26T23:09:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.302+0000 7fa10f96b740 1 -- 172.21.15.40:0/3941112343 wait complete. 2024-04-26T23:09:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.302+0000 7fa10f96b740 1 -- start start 2024-04-26T23:09:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.303+0000 7fa10f96b740 1 Processor -- start 2024-04-26T23:09:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa10f96b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef22c710 0x561cef21df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa10f96b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef249fb0 0x561cef22f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa10f96b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef22fd10 0x561cef24aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa10f96b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561cef16a200 con 0x561cef22c710 2024-04-26T23:09:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa10f96b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561cef164ee0 con 0x561cef249fb0 2024-04-26T23:09:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa10f96b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561cef171e70 con 0x561cef22fd10 2024-04-26T23:09:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa10f96b740 10 client.0 _ll_register_callbacks cb 0x561cef15b270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa105e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef22c710 0x561cef21df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa105e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef22c710 0x561cef21df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43330/0 (socket says 172.21.15.40:43330) 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa105e00640 1 -- 172.21.15.40:0/393599580 learned_addr learned my addr 172.21.15.40:0/393599580 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa105e00640 1 -- 172.21.15.40:0/393599580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef22fd10 msgr2=0x561cef24aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa105e00640 1 --2- 172.21.15.40:0/393599580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef22fd10 0x561cef24aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa105e00640 1 -- 172.21.15.40:0/393599580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef249fb0 msgr2=0x561cef22f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa105e00640 1 --2- 172.21.15.40:0/393599580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef249fb0 0x561cef22f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.304+0000 7fa105e00640 1 -- 172.21.15.40:0/393599580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ceef417d0 con 0x561cef22c710 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.305+0000 7fa105e00640 1 --2- 172.21.15.40:0/393599580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef22c710 0x561cef21df20 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80a9fc0 tx=0x7fa0f807a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9058f66b3ffa1b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.305+0000 7fa0eea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.305+0000 7fa0eea00640 1 -- 172.21.15.40:0/393599580 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80823a0 con 0x561cef22c710 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.305+0000 7fa0eea00640 1 -- 172.21.15.40:0/393599580 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fa0f8077ac0 con 0x561cef22c710 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.305+0000 7fa10f96b740 1 -- 172.21.15.40:0/393599580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa10000edb0 con 0x561cef22c710 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.305+0000 7fa0eea00640 1 -- 172.21.15.40:0/393599580 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fa0f8003c90 con 0x561cef22c710 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.305+0000 7fa10f96b740 10 client.9363 fetch_fsmap learned FSMap version 119 2024-04-26T23:09:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.305+0000 7fa10f96b740 1 -- 172.21.15.40:0/393599580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) -- 0x561ceee773f0 con 0x561cef22c710 2024-04-26T23:09:38.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.305+0000 7fa0eea00640 1 -- 172.21.15.40:0/393599580 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa10000edb0 con 0x561cef22c710 2024-04-26T23:09:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.306+0000 7fa0eea00640 1 -- 172.21.15.40:0/393599580 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 119) ==== 3550+0+0 (secure 0 0 0) 0x7fa0f8081f10 con 0x561cef22c710 2024-04-26T23:09:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.306+0000 7fa10f96b740 10 client.9363 fetch_fsmap finished waiting for FSMap version 119 2024-04-26T23:09:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.306+0000 7fa10f96b740 10 client.9363 resolve_mds: resolved 7:2 to role '7:2' aka daemon mds.d 2024-04-26T23:09:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.306+0000 7fa10f96b740 20 client.9363 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.306+0000 7fa10f96b740 1 --2- 172.21.15.40:0/393599580 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x561cef257100 0x561cef257530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.306+0000 7fa10f96b740 4 client.9363 mds_command: new command op to 6598 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.306+0000 7fa10f96b740 1 -- 172.21.15.40:0/393599580 --> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cef1737e0 con 0x561cef257100 2024-04-26T23:09:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.307+0000 7fa105400640 1 --2- 172.21.15.40:0/393599580 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x561cef257100 0x561cef257530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.308+0000 7fa105400640 1 --2- 172.21.15.40:0/393599580 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x561cef257100 0x561cef257530 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=747066fc2d327886 server_cookie=be0f80215f0efe54 in_seq=0 out_seq=0 2024-04-26T23:09:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.308+0000 7fa0eea00640 10 client.9363 ms_handle_connect on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:38.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.309+0000 7fa0eea00640 1 -- 172.21.15.40:0/393599580 <== mds.2 v2:172.21.15.103:6835/2022020922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561cef1737e0 con 0x561cef257100 2024-04-26T23:09:38.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.309+0000 7fa0eea00640 10 client.9363 handle_command_reply: tid=0 2024-04-26T23:09:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.309+0000 7fa10f96b740 1 client.9363 shutdown 2024-04-26T23:09:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.310+0000 7fa10f96b740 1 -- 172.21.15.40:0/393599580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef22c710 msgr2=0x561cef21df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.310+0000 7fa10f96b740 1 --2- 172.21.15.40:0/393599580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef22c710 0x561cef21df20 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80a9fc0 tx=0x7fa0f807a000 comp rx=0 tx=0).stop 2024-04-26T23:09:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.310+0000 7fa10f96b740 1 -- 172.21.15.40:0/393599580 shutdown_connections 2024-04-26T23:09:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.310+0000 7fa10f96b740 1 --2- 172.21.15.40:0/393599580 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x561cef257100 0x561cef257530 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.310+0000 7fa10f96b740 1 --2- 172.21.15.40:0/393599580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef22fd10 0x561cef24aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.310+0000 7fa10f96b740 1 --2- 172.21.15.40:0/393599580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef249fb0 0x561cef22f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.310+0000 7fa10f96b740 1 --2- 172.21.15.40:0/393599580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef22c710 0x561cef21df20 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.310+0000 7fa0eea00640 0 client.9363 ms_handle_reset on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.310+0000 7fa10f96b740 1 -- 172.21.15.40:0/393599580 >> 172.21.15.40:0/393599580 conn(0x561cef2023f0 msgr2=0x561cef246cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.310+0000 7fa10f96b740 1 -- 172.21.15.40:0/393599580 shutdown_connections 2024-04-26T23:09:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.310+0000 7fa10f96b740 1 -- 172.21.15.40:0/393599580 wait complete. 2024-04-26T23:09:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.310+0000 7fa10f96b740 20 client.9363 trim_cache size 0 max 16384 2024-04-26T23:09:38.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.329+0000 7fa10f96b740 1 Processor -- start 2024-04-26T23:09:38.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.329+0000 7fa10f96b740 1 -- start start 2024-04-26T23:09:38.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.330+0000 7fa10f96b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22c710 0x561cef1fc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.330+0000 7fa10f96b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef22fd10 0x561cef24a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.330+0000 7fa10f96b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 0x561cef24a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.330+0000 7fa10f96b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa108202410 con 0x561cef249fb0 2024-04-26T23:09:38.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.330+0000 7fa10f96b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1082015b0 con 0x561cef22c710 2024-04-26T23:09:38.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.330+0000 7fa10f96b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa108201ce0 con 0x561cef22fd10 2024-04-26T23:09:38.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.331+0000 7fa106800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 0x561cef24a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.331+0000 7fa106800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 0x561cef24a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43340/0 (socket says 172.21.15.40:43340) 2024-04-26T23:09:38.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.331+0000 7fa106800640 1 -- 172.21.15.40:0/2337074152 learned_addr learned my addr 172.21.15.40:0/2337074152 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.331+0000 7fa106800640 1 -- 172.21.15.40:0/2337074152 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef22fd10 msgr2=0x561cef24a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.331+0000 7fa106800640 1 --2- 172.21.15.40:0/2337074152 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef22fd10 0x561cef24a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.331+0000 7fa106800640 1 -- 172.21.15.40:0/2337074152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22c710 msgr2=0x561cef1fc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.331+0000 7fa106800640 1 --2- 172.21.15.40:0/2337074152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22c710 0x561cef1fc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.331+0000 7fa106800640 1 -- 172.21.15.40:0/2337074152 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ceee370f0 con 0x561cef249fb0 2024-04-26T23:09:38.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.331+0000 7fa106800640 1 --2- 172.21.15.40:0/2337074152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 0x561cef24a910 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fa100016000 tx=0x7fa100004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721b72810788d994 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.331+0000 7fa0f5c00640 1 -- 172.21.15.40:0/2337074152 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10001d670 con 0x561cef249fb0 2024-04-26T23:09:38.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.332+0000 7fa0f5c00640 1 -- 172.21.15.40:0/2337074152 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa100023070 con 0x561cef249fb0 2024-04-26T23:09:38.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.332+0000 7fa0f5c00640 1 -- 172.21.15.40:0/2337074152 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10001da50 con 0x561cef249fb0 2024-04-26T23:09:38.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.332+0000 7fa10f96b740 1 -- 172.21.15.40:0/2337074152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 msgr2=0x561cef24a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.332+0000 7fa10f96b740 1 --2- 172.21.15.40:0/2337074152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 0x561cef24a910 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fa100016000 tx=0x7fa100004a40 comp rx=0 tx=0).stop 2024-04-26T23:09:38.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.332+0000 7fa10f96b740 1 -- 172.21.15.40:0/2337074152 shutdown_connections 2024-04-26T23:09:38.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.332+0000 7fa10f96b740 1 --2- 172.21.15.40:0/2337074152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 0x561cef24a910 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.332+0000 7fa10f96b740 1 --2- 172.21.15.40:0/2337074152 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef22fd10 0x561cef24a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.332+0000 7fa10f96b740 1 --2- 172.21.15.40:0/2337074152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22c710 0x561cef1fc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.332+0000 7fa10f96b740 1 -- 172.21.15.40:0/2337074152 >> 172.21.15.40:0/2337074152 conn(0x561cef2023f0 msgr2=0x561cef245ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.332+0000 7fa10f96b740 1 -- 172.21.15.40:0/2337074152 shutdown_connections 2024-04-26T23:09:38.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.332+0000 7fa10f96b740 1 -- 172.21.15.40:0/2337074152 wait complete. 2024-04-26T23:09:38.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.333+0000 7fa10f96b740 1 -- start start 2024-04-26T23:09:38.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.333+0000 7fa10f96b740 1 Processor -- start 2024-04-26T23:09:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.334+0000 7fa10f96b740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef22c710 0x561cef2af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.334+0000 7fa10f96b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22fd10 0x561cef21c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.334+0000 7fa10f96b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 0x561cef247400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.334+0000 7fa10f96b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561cef172fd0 con 0x561cef249fb0 2024-04-26T23:09:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.334+0000 7fa10f96b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa108200690 con 0x561cef22fd10 2024-04-26T23:09:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.334+0000 7fa10f96b740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561cef2406c0 con 0x561cef22c710 2024-04-26T23:09:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.334+0000 7fa10f96b740 10 client.0 _ll_register_callbacks cb 0x7fa0f8075a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.334+0000 7fa106800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 0x561cef247400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.334+0000 7fa106800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 0x561cef247400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43344/0 (socket says 172.21.15.40:43344) 2024-04-26T23:09:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.334+0000 7fa106800640 1 -- 172.21.15.40:0/700953282 learned_addr learned my addr 172.21.15.40:0/700953282 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.335+0000 7fa106800640 1 -- 172.21.15.40:0/700953282 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef22c710 msgr2=0x561cef2af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.335+0000 7fa105e00640 1 --2- 172.21.15.40:0/700953282 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef22c710 0x561cef2af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.335+0000 7fa106800640 1 --2- 172.21.15.40:0/700953282 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef22c710 0x561cef2af3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.335+0000 7fa106800640 1 -- 172.21.15.40:0/700953282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22fd10 msgr2=0x561cef21c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.335+0000 7fa105400640 1 --2- 172.21.15.40:0/700953282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22fd10 0x561cef21c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.335+0000 7fa106800640 1 --2- 172.21.15.40:0/700953282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22fd10 0x561cef21c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.335+0000 7fa106800640 1 -- 172.21.15.40:0/700953282 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cef12da30 con 0x561cef249fb0 2024-04-26T23:09:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.335+0000 7fa106800640 1 --2- 172.21.15.40:0/700953282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 0x561cef247400 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fa10808f320 tx=0x7fa0f0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef068f13f3205f19 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.335+0000 7fa0eea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.335+0000 7fa0eea00640 1 -- 172.21.15.40:0/700953282 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa100018050 con 0x561cef249fb0 2024-04-26T23:09:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.335+0000 7fa0eea00640 1 -- 172.21.15.40:0/700953282 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fa100003a10 con 0x561cef249fb0 2024-04-26T23:09:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.336+0000 7fa10f96b740 1 -- 172.21.15.40:0/700953282 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa100023070 con 0x561cef249fb0 2024-04-26T23:09:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.336+0000 7fa0eea00640 1 -- 172.21.15.40:0/700953282 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fa100003c10 con 0x561cef249fb0 2024-04-26T23:09:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.336+0000 7fa10f96b740 10 client.9369 fetch_fsmap learned FSMap version 119 2024-04-26T23:09:38.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.336+0000 7fa10f96b740 1 -- 172.21.15.40:0/700953282 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) -- 0x561cef23d750 con 0x561cef249fb0 2024-04-26T23:09:38.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.336+0000 7fa0eea00640 1 -- 172.21.15.40:0/700953282 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa100023070 con 0x561cef249fb0 2024-04-26T23:09:38.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.336+0000 7fa0eea00640 1 -- 172.21.15.40:0/700953282 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 119) ==== 3550+0+0 (secure 0 0 0) 0x7fa100004030 con 0x561cef249fb0 2024-04-26T23:09:38.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.336+0000 7fa10f96b740 10 client.9369 fetch_fsmap finished waiting for FSMap version 119 2024-04-26T23:09:38.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.336+0000 7fa10f96b740 10 client.9369 resolve_mds: resolved 7:2 to role '7:2' aka daemon mds.d 2024-04-26T23:09:38.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.336+0000 7fa10f96b740 20 client.9369 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:38.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.336+0000 7fa10f96b740 1 --2- 172.21.15.40:0/700953282 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x561cef247940 0x561cef24a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.336+0000 7fa10f96b740 4 client.9369 mds_command: new command op to 6598 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:09:38.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.336+0000 7fa10f96b740 1 -- 172.21.15.40:0/700953282 --> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561cef16d650 con 0x561cef247940 2024-04-26T23:09:38.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.337+0000 7fa105e00640 1 --2- 172.21.15.40:0/700953282 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x561cef247940 0x561cef24a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa105e00640 1 --2- 172.21.15.40:0/700953282 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x561cef247940 0x561cef24a630 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=36ea5506c93fe4a7 server_cookie=7ae970067bcf7bb3 in_seq=0 out_seq=0 2024-04-26T23:09:38.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa0eea00640 10 client.9369 ms_handle_connect on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:38.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa0eea00640 1 -- 172.21.15.40:0/700953282 <== mds.2 v2:172.21.15.103:6835/2022020922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x561cef16d650 con 0x561cef247940 2024-04-26T23:09:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa0eea00640 10 client.9369 handle_command_reply: tid=0 2024-04-26T23:09:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa10f96b740 1 client.9369 shutdown 2024-04-26T23:09:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa10f96b740 1 -- 172.21.15.40:0/700953282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 msgr2=0x561cef247400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa10f96b740 1 --2- 172.21.15.40:0/700953282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 0x561cef247400 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fa10808f320 tx=0x7fa0f0001a40 comp rx=0 tx=0).stop 2024-04-26T23:09:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa10f96b740 1 -- 172.21.15.40:0/700953282 shutdown_connections 2024-04-26T23:09:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa10f96b740 1 --2- 172.21.15.40:0/700953282 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x561cef247940 0x561cef24a630 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa10f96b740 1 --2- 172.21.15.40:0/700953282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cef249fb0 0x561cef247400 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa10f96b740 1 --2- 172.21.15.40:0/700953282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561cef22fd10 0x561cef21c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa10f96b740 1 --2- 172.21.15.40:0/700953282 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561cef22c710 0x561cef2af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa0eea00640 0 client.9369 ms_handle_reset on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.338+0000 7fa10f96b740 1 -- 172.21.15.40:0/700953282 >> 172.21.15.40:0/700953282 conn(0x561cef2023f0 msgr2=0x561cef22f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.339+0000 7fa10f96b740 1 -- 172.21.15.40:0/700953282 shutdown_connections 2024-04-26T23:09:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.339+0000 7fa10f96b740 1 -- 172.21.15.40:0/700953282 wait complete. 2024-04-26T23:09:38.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.339+0000 7fa10f96b740 20 client.9369 trim_cache size 0 max 16384 2024-04-26T23:09:38.342 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-26T23:09:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:09:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: "a9827531-70a9-4d6f-b12a-2d71fa886d03": { 2024-04-26T23:09:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:09:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "a9827531-70a9-4d6f-b12a-2d71fa886d03", 2024-04-26T23:09:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:09:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:38.342 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:38.343 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:38.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.339+0000 7fa107200640 1 -- 172.21.15.40:0/254980930 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa0e406c8f0 msgr2=0x7fa0e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.339+0000 7fa107200640 1 --2- 172.21.15.40:0/254980930 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa0e406c8f0 0x7fa0e406edb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0000f60 tx=0x7fa0f0009040 comp rx=0 tx=0).stop 2024-04-26T23:09:38.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.339+0000 7fa107200640 1 -- 172.21.15.40:0/254980930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0e80039b0 msgr2=0x7fa1081f6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.339+0000 7fa107200640 1 --2- 172.21.15.40:0/254980930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0e80039b0 0x7fa1081f6010 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009ac0 tx=0x7fa0f8004730 comp rx=0 tx=0).stop 2024-04-26T23:09:38.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.339+0000 7fa107200640 1 -- 172.21.15.40:0/254980930 shutdown_connections 2024-04-26T23:09:38.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.339+0000 7fa107200640 1 --2- 172.21.15.40:0/254980930 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa0e406c8f0 0x7fa0e406edb0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.339+0000 7fa107200640 1 --2- 172.21.15.40:0/254980930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1081f7cb0 0x7fa1081e8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.339+0000 7fa107200640 1 --2- 172.21.15.40:0/254980930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa1080b96e0 0x7fa1081f58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.339+0000 7fa107200640 1 --2- 172.21.15.40:0/254980930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0e80039b0 0x7fa1081f6010 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.339+0000 7fa107200640 1 -- 172.21.15.40:0/254980930 >> 172.21.15.40:0/254980930 conn(0x7fa1080d4dd0 msgr2=0x7fa1080b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.340+0000 7fa107200640 1 -- 172.21.15.40:0/254980930 shutdown_connections 2024-04-26T23:09:38.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.340+0000 7fa107200640 1 -- 172.21.15.40:0/254980930 wait complete. 2024-04-26T23:09:38.356 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:09:38.356 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:09:38.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-26T23:09:38.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.566+0000 7f4e8ee00640 1 Processor -- start 2024-04-26T23:09:38.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.566+0000 7f4e8ee00640 1 -- start start 2024-04-26T23:09:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.567+0000 7f4e8ee00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4e880cfaf0 0x7f4e88073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.568+0000 7f4e8ee00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e88161880 con 0x7f4e8806dd40 2024-04-26T23:09:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.568+0000 7f4e8ee00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e880cff80 con 0x7f4e88160a50 2024-04-26T23:09:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.568+0000 7f4e8ee00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4e88071ce0 con 0x7f4e880cfaf0 2024-04-26T23:09:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.568+0000 7f4e8e400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4e8806dd40 0x7f4e88072730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44872/0 (socket says 172.21.15.40:44872) 2024-04-26T23:09:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.568+0000 7f4e8e400640 1 -- 172.21.15.40:0/2998530727 learned_addr learned my addr 172.21.15.40:0/2998530727 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:38.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.568+0000 7f4e8d000640 1 --2- 172.21.15.40:0/2998530727 >> v2:172.21.15.103:3300/0 conn(0x7f4e880cfaf0 0x7f4e88073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.568+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4152618831 0 0) 0x7f4e88161880 con 0x7f4e8806dd40 2024-04-26T23:09:38.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.568+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e70003520 con 0x7f4e8806dd40 2024-04-26T23:09:38.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3808200668 0 0) 0x7f4e70003520 con 0x7f4e8806dd40 2024-04-26T23:09:38.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 >> v2:172.21.15.103:3300/0 conn(0x7f4e880cfaf0 msgr2=0x7f4e88073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e87e00640 1 --2- 172.21.15.40:0/2998530727 >> v2:172.21.15.103:3300/0 conn(0x7f4e880cfaf0 0x7f4e88073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 >> v1:172.21.15.103:6789/0 conn(0x7f4e88160a50 legacy=0x7f4e88160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e880d0260 con 0x7f4e8806dd40 2024-04-26T23:09:38.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f4e800019f0 con 0x7f4e8806dd40 2024-04-26T23:09:38.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e8d000640 1 --2- 172.21.15.40:0/2998530727 >> v2:172.21.15.103:3300/0 conn(0x7f4e880cfaf0 0x7f4e88073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:09:38.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 >> v1:172.21.15.40:6789/0 conn(0x7f4e8806dd40 legacy=0x7f4e88072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e87e00640 1 --2- 172.21.15.40:0/2998530727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e700039b0 0x7f4e70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e87e00640 1 --2- 172.21.15.40:0/2998530727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e70004800 0x7f4e70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e87e00640 1 --2- 172.21.15.40:0/2998530727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e700051c0 0x7f4e7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88071ce0 con 0x7f4e700051c0 2024-04-26T23:09:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7000c110 con 0x7f4e70004800 2024-04-26T23:09:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4e7000c310 con 0x7f4e700039b0 2024-04-26T23:09:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.569+0000 7f4e8e400640 1 --2- 172.21.15.40:0/2998530727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e700051c0 0x7f4e7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.570+0000 7f4e8e400640 1 -- 172.21.15.40:0/2998530727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e700039b0 msgr2=0x7f4e70003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.570+0000 7f4e8da00640 1 --2- 172.21.15.40:0/2998530727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e700039b0 0x7f4e70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.570+0000 7f4e8d000640 1 --2- 172.21.15.40:0/2998530727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e70004800 0x7f4e70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.570+0000 7f4e8e400640 1 --2- 172.21.15.40:0/2998530727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e700039b0 0x7f4e70003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.570+0000 7f4e8e400640 1 -- 172.21.15.40:0/2998530727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e70004800 msgr2=0x7f4e70004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.570+0000 7f4e8e400640 1 --2- 172.21.15.40:0/2998530727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e70004800 0x7f4e70004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.570+0000 7f4e8e400640 1 -- 172.21.15.40:0/2998530727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e70003520 con 0x7f4e700051c0 2024-04-26T23:09:38.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.570+0000 7f4e8da00640 1 --2- 172.21.15.40:0/2998530727 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e700039b0 0x7f4e70003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:38.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.571+0000 7f4e8e400640 1 -- 172.21.15.40:0/2998530727 reap_dead start 2024-04-26T23:09:38.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.571+0000 7f4e8e400640 1 --2- 172.21.15.40:0/2998530727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e700051c0 0x7f4e7000b9c0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000a850 tx=0x7f4e70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a07b0e755d4aaf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.571+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8000a4a0 con 0x7f4e700051c0 2024-04-26T23:09:38.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.571+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e8002f070 con 0x7f4e700051c0 2024-04-26T23:09:38.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.571+0000 7f4e87e00640 1 -- 172.21.15.40:0/2998530727 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8002bce0 con 0x7f4e700051c0 2024-04-26T23:09:38.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.572+0000 7f4e8ee00640 1 -- 172.21.15.40:0/2998530727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e700051c0 msgr2=0x7f4e7000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.572+0000 7f4e8ee00640 1 --2- 172.21.15.40:0/2998530727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e700051c0 0x7f4e7000b9c0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000a850 tx=0x7f4e70005830 comp rx=0 tx=0).stop 2024-04-26T23:09:38.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.572+0000 7f4e8ee00640 1 -- 172.21.15.40:0/2998530727 shutdown_connections 2024-04-26T23:09:38.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.572+0000 7f4e8ee00640 1 --2- 172.21.15.40:0/2998530727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e700051c0 0x7f4e7000b9c0 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.572+0000 7f4e8ee00640 1 -- 172.21.15.40:0/2998530727 >> 172.21.15.40:0/2998530727 conn(0x7f4e880b2720 msgr2=0x7f4e880b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.572+0000 7f4e8ee00640 1 -- 172.21.15.40:0/2998530727 shutdown_connections 2024-04-26T23:09:38.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.572+0000 7f4e8ee00640 1 -- 172.21.15.40:0/2998530727 wait complete. 2024-04-26T23:09:38.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.574+0000 7f4e8ee00640 1 Processor -- start 2024-04-26T23:09:38.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.574+0000 7f4e8ee00640 1 -- start start 2024-04-26T23:09:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8ee00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e700051c0 0x7f4e8815ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8ee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e880b0590 0x7f4e8813bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8ee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e8815dac0 0x7f4e8815df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8ee00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8806e1f0 con 0x7f4e700051c0 2024-04-26T23:09:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8ee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4e880722a0 con 0x7f4e8815dac0 2024-04-26T23:09:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8ee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4e881468f0 con 0x7f4e880b0590 2024-04-26T23:09:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e700051c0 0x7f4e8815ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e700051c0 0x7f4e8815ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43368/0 (socket says 172.21.15.40:43368) 2024-04-26T23:09:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8da00640 1 -- 172.21.15.40:0/2285182297 learned_addr learned my addr 172.21.15.40:0/2285182297 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8da00640 1 -- 172.21.15.40:0/2285182297 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e880b0590 msgr2=0x7f4e8813bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8da00640 1 --2- 172.21.15.40:0/2285182297 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e880b0590 0x7f4e8813bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8da00640 1 -- 172.21.15.40:0/2285182297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e8815dac0 msgr2=0x7f4e8815df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8da00640 1 --2- 172.21.15.40:0/2285182297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e8815dac0 0x7f4e8815df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.575+0000 7f4e8da00640 1 -- 172.21.15.40:0/2285182297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e880b0d50 con 0x7f4e700051c0 2024-04-26T23:09:38.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.576+0000 7f4e8da00640 1 --2- 172.21.15.40:0/2285182297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e700051c0 0x7f4e8815ea20 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00d7d0 tx=0x7f4e7c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.576+0000 7f4e85600640 1 -- 172.21.15.40:0/2285182297 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c005270 con 0x7f4e700051c0 2024-04-26T23:09:38.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.576+0000 7f4e85600640 1 -- 172.21.15.40:0/2285182297 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e7c017070 con 0x7f4e700051c0 2024-04-26T23:09:38.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.576+0000 7f4e85600640 1 -- 172.21.15.40:0/2285182297 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c00b650 con 0x7f4e700051c0 2024-04-26T23:09:38.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.576+0000 7f4e8ee00640 1 -- 172.21.15.40:0/2285182297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e880b1030 con 0x7f4e700051c0 2024-04-26T23:09:38.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.576+0000 7f4e8ee00640 1 -- 172.21.15.40:0/2285182297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e880b15f0 con 0x7f4e700051c0 2024-04-26T23:09:38.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.579+0000 7f4e85600640 1 -- 172.21.15.40:0/2285182297 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4e7c00b810 con 0x7f4e700051c0 2024-04-26T23:09:38.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.580+0000 7f4e85600640 1 --2- 172.21.15.40:0/2285182297 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4e6806c920 0x7f4e6806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.580+0000 7f4e85600640 1 -- 172.21.15.40:0/2285182297 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f4e7c08b790 con 0x7f4e700051c0 2024-04-26T23:09:38.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.581+0000 7f4e93223740 1 Processor -- start 2024-04-26T23:09:38.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.581+0000 7f4e8d000640 1 --2- 172.21.15.40:0/2285182297 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4e6806c920 0x7f4e6806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.581+0000 7f4e93223740 1 -- start start 2024-04-26T23:09:38.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.582+0000 7f4e93223740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434bc710 0x5584434bcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.582+0000 7f4e93223740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434d9fb0 0x5584434da500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.582+0000 7f4e93223740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434923f0 0x5584434d8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.582+0000 7f4e93223740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5584433a2030 con 0x5584434bc710 2024-04-26T23:09:38.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.582+0000 7f4e93223740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5584433bd7f0 con 0x5584434d9fb0 2024-04-26T23:09:38.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.582+0000 7f4e93223740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5584433787e0 con 0x5584434923f0 2024-04-26T23:09:38.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.582+0000 7f4e8d000640 1 --2- 172.21.15.40:0/2285182297 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4e6806c920 0x7f4e6806ede0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4e88140690 tx=0x7f4e74006df0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.582+0000 7f4e8da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434bc710 0x5584434bcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.582+0000 7f4e8da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434bc710 0x5584434bcaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43378/0 (socket says 172.21.15.40:43378) 2024-04-26T23:09:38.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.582+0000 7f4e8da00640 1 -- 172.21.15.40:0/2907791814 learned_addr learned my addr 172.21.15.40:0/2907791814 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.582+0000 7f4e8e400640 1 --2- 172.21.15.40:0/2907791814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434923f0 0x5584434d8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.583+0000 7f4e8d000640 1 --2- 172.21.15.40:0/2907791814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434d9fb0 0x5584434da500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.583+0000 7f4e8da00640 1 -- 172.21.15.40:0/2907791814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434923f0 msgr2=0x5584434d8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.583+0000 7f4e8da00640 1 --2- 172.21.15.40:0/2907791814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434923f0 0x5584434d8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.583+0000 7f4e8da00640 1 -- 172.21.15.40:0/2907791814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434d9fb0 msgr2=0x5584434da500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.583+0000 7f4e8da00640 1 --2- 172.21.15.40:0/2907791814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434d9fb0 0x5584434da500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.583+0000 7f4e8da00640 1 -- 172.21.15.40:0/2907791814 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558443384050 con 0x5584434bc710 2024-04-26T23:09:38.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.583+0000 7f4e8da00640 1 --2- 172.21.15.40:0/2907791814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434bc710 0x5584434bcaf0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0592b0 tx=0x7f4e7c010390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2262e4ee8bbcdf03 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.583+0000 7f4e7a000640 1 -- 172.21.15.40:0/2907791814 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c071020 con 0x5584434bc710 2024-04-26T23:09:38.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.583+0000 7f4e7a000640 1 -- 172.21.15.40:0/2907791814 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e7c05d6d0 con 0x5584434bc710 2024-04-26T23:09:38.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.583+0000 7f4e7a000640 1 -- 172.21.15.40:0/2907791814 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c061f00 con 0x5584434bc710 2024-04-26T23:09:38.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.584+0000 7f4e93223740 1 -- 172.21.15.40:0/2907791814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434bc710 msgr2=0x5584434bcaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.584+0000 7f4e93223740 1 --2- 172.21.15.40:0/2907791814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434bc710 0x5584434bcaf0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0592b0 tx=0x7f4e7c010390 comp rx=0 tx=0).stop 2024-04-26T23:09:38.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.584+0000 7f4e93223740 1 -- 172.21.15.40:0/2907791814 shutdown_connections 2024-04-26T23:09:38.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.584+0000 7f4e93223740 1 --2- 172.21.15.40:0/2907791814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434923f0 0x5584434d8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.584+0000 7f4e93223740 1 --2- 172.21.15.40:0/2907791814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434d9fb0 0x5584434da500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.584+0000 7f4e93223740 1 --2- 172.21.15.40:0/2907791814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434bc710 0x5584434bcaf0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.584+0000 7f4e93223740 1 -- 172.21.15.40:0/2907791814 >> 172.21.15.40:0/2907791814 conn(0x55844348ed20 msgr2=0x5584434d7520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.584+0000 7f4e93223740 1 -- 172.21.15.40:0/2907791814 shutdown_connections 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.584+0000 7f4e93223740 1 -- 172.21.15.40:0/2907791814 wait complete. 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.585+0000 7f4e93223740 1 -- start start 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.585+0000 7f4e93223740 1 Processor -- start 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.586+0000 7f4e93223740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 0x5584434cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.586+0000 7f4e93223740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 0x5584434acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.586+0000 7f4e93223740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x55844348b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.586+0000 7f4e93223740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5584433fa200 con 0x5584434d2d30 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.586+0000 7f4e93223740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5584433f4ee0 con 0x5584434bc710 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.586+0000 7f4e93223740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558443401e70 con 0x5584434d9fb0 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.586+0000 7f4e93223740 10 client.0 _ll_register_callbacks cb 0x5584433eb270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:38.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.586+0000 7f4e8e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x55844348b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.586+0000 7f4e8e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x55844348b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43380/0 (socket says 172.21.15.40:43380) 2024-04-26T23:09:38.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.586+0000 7f4e8e400640 1 -- 172.21.15.40:0/4160742936 learned_addr learned my addr 172.21.15.40:0/4160742936 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e8da00640 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 0x5584434cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e8e400640 1 -- 172.21.15.40:0/4160742936 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 msgr2=0x5584434acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e8d000640 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 0x5584434acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e8e400640 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 0x5584434acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e8e400640 1 -- 172.21.15.40:0/4160742936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 msgr2=0x5584434cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e8e400640 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 0x5584434cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e8e400640 1 -- 172.21.15.40:0/4160742936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584431d17d0 con 0x5584434d2d30 2024-04-26T23:09:38.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e8da00640 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 0x5584434cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:38.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e8d000640 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 0x5584434acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:38.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e8e400640 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x55844348b960 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000a970 tx=0x7f4e80001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8595f99ddc181498 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e6ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:38.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e6ea00640 1 -- 172.21.15.40:0/4160742936 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8002ba90 con 0x5584434d2d30 2024-04-26T23:09:38.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e6ea00640 1 -- 172.21.15.40:0/4160742936 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f4e8002bc30 con 0x5584434d2d30 2024-04-26T23:09:38.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.587+0000 7f4e93223740 1 -- 172.21.15.40:0/4160742936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e7c05d6d0 con 0x5584434d2d30 2024-04-26T23:09:38.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.588+0000 7f4e6ea00640 1 -- 172.21.15.40:0/4160742936 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f4e80048050 con 0x5584434d2d30 2024-04-26T23:09:38.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.588+0000 7f4e93223740 10 client.9381 fetch_fsmap learned FSMap version 119 2024-04-26T23:09:38.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.588+0000 7f4e93223740 1 -- 172.21.15.40:0/4160742936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) -- 0x5584431073f0 con 0x5584434d2d30 2024-04-26T23:09:38.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.588+0000 7f4e6ea00640 1 -- 172.21.15.40:0/4160742936 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e7c05d6d0 con 0x5584434d2d30 2024-04-26T23:09:38.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.588+0000 7f4e6ea00640 1 -- 172.21.15.40:0/4160742936 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 119) ==== 3550+0+0 (secure 0 0 0) 0x7f4e8004b020 con 0x5584434d2d30 2024-04-26T23:09:38.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.588+0000 7f4e93223740 10 client.9381 fetch_fsmap finished waiting for FSMap version 119 2024-04-26T23:09:38.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.588+0000 7f4e93223740 10 client.9381 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:38.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.588+0000 7f4e93223740 20 client.9381 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:38.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.588+0000 7f4e93223740 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5584434f1bb0 0x5584434f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.588+0000 7f4e93223740 4 client.9381 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:38.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.588+0000 7f4e93223740 1 -- 172.21.15.40:0/4160742936 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584434037e0 con 0x5584434f1bb0 2024-04-26T23:09:38.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.588+0000 7f4e8da00640 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5584434f1bb0 0x5584434f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.589+0000 7f4e8da00640 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5584434f1bb0 0x5584434f3fe0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8d4fc84cacb862b server_cookie=d4a1de03cb69a3c1 in_seq=0 out_seq=0 2024-04-26T23:09:38.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.589+0000 7f4e6ea00640 10 client.9381 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:38.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e6ea00640 1 -- 172.21.15.40:0/4160742936 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5584434037e0 con 0x5584434f1bb0 2024-04-26T23:09:38.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e6ea00640 10 client.9381 handle_command_reply: tid=0 2024-04-26T23:09:38.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e93223740 1 client.9381 shutdown 2024-04-26T23:09:38.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e93223740 1 -- 172.21.15.40:0/4160742936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 msgr2=0x55844348b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e93223740 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x55844348b960 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000a970 tx=0x7f4e80001dc0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e93223740 1 -- 172.21.15.40:0/4160742936 shutdown_connections 2024-04-26T23:09:38.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e93223740 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x55844348b960 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e93223740 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 0x5584434acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e93223740 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5584434f1bb0 0x5584434f3fe0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e93223740 1 --2- 172.21.15.40:0/4160742936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 0x5584434cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e93223740 1 -- 172.21.15.40:0/4160742936 >> 172.21.15.40:0/4160742936 conn(0x5584434923f0 msgr2=0x5584434db330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e6ea00640 0 client.9381 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:38.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e93223740 1 -- 172.21.15.40:0/4160742936 shutdown_connections 2024-04-26T23:09:38.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e93223740 1 -- 172.21.15.40:0/4160742936 wait complete. 2024-04-26T23:09:38.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.590+0000 7f4e93223740 20 client.9381 trim_cache size 0 max 16384 2024-04-26T23:09:38.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.610+0000 7f4e93223740 1 Processor -- start 2024-04-26T23:09:38.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.610+0000 7f4e93223740 1 -- start start 2024-04-26T23:09:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e93223740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 0x5584434d8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e93223740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x5584434d9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e93223740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 0x5584434ba5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e93223740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4e881fdf90 con 0x5584434d2d30 2024-04-26T23:09:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e93223740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4e881fd130 con 0x5584434bc710 2024-04-26T23:09:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e93223740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4e881fd860 con 0x5584434d9fb0 2024-04-26T23:09:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e8d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x5584434d9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e8d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x5584434d9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43386/0 (socket says 172.21.15.40:43386) 2024-04-26T23:09:38.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e8d000640 1 -- 172.21.15.40:0/3039397974 learned_addr learned my addr 172.21.15.40:0/3039397974 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e8d000640 1 -- 172.21.15.40:0/3039397974 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 msgr2=0x5584434ba5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e8d000640 1 --2- 172.21.15.40:0/3039397974 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 0x5584434ba5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e8d000640 1 -- 172.21.15.40:0/3039397974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 msgr2=0x5584434d8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e8d000640 1 --2- 172.21.15.40:0/3039397974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 0x5584434d8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.611+0000 7f4e8d000640 1 -- 172.21.15.40:0/3039397974 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584430c70f0 con 0x5584434d2d30 2024-04-26T23:09:38.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.612+0000 7f4e8d000640 1 --2- 172.21.15.40:0/3039397974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x5584434d9040 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f4e7400f9f0 tx=0x7f4e74015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393217c12aa3a067 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.612+0000 7f4e79600640 1 -- 172.21.15.40:0/3039397974 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e74004020 con 0x5584434d2d30 2024-04-26T23:09:38.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.612+0000 7f4e79600640 1 -- 172.21.15.40:0/3039397974 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e7401b070 con 0x5584434d2d30 2024-04-26T23:09:38.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.612+0000 7f4e79600640 1 -- 172.21.15.40:0/3039397974 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e740043e0 con 0x5584434d2d30 2024-04-26T23:09:38.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.613+0000 7f4e93223740 1 -- 172.21.15.40:0/3039397974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 msgr2=0x5584434d9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.613+0000 7f4e93223740 1 --2- 172.21.15.40:0/3039397974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x5584434d9040 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f4e7400f9f0 tx=0x7f4e74015000 comp rx=0 tx=0).stop 2024-04-26T23:09:38.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.613+0000 7f4e93223740 1 -- 172.21.15.40:0/3039397974 shutdown_connections 2024-04-26T23:09:38.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.613+0000 7f4e93223740 1 --2- 172.21.15.40:0/3039397974 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 0x5584434ba5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.613+0000 7f4e93223740 1 --2- 172.21.15.40:0/3039397974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x5584434d9040 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.613+0000 7f4e93223740 1 --2- 172.21.15.40:0/3039397974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 0x5584434d8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.613+0000 7f4e93223740 1 -- 172.21.15.40:0/3039397974 >> 172.21.15.40:0/3039397974 conn(0x5584434923f0 msgr2=0x5584434ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.613+0000 7f4e93223740 1 -- 172.21.15.40:0/3039397974 shutdown_connections 2024-04-26T23:09:38.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.613+0000 7f4e93223740 1 -- 172.21.15.40:0/3039397974 wait complete. 2024-04-26T23:09:38.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.614+0000 7f4e93223740 1 -- start start 2024-04-26T23:09:38.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.614+0000 7f4e93223740 1 Processor -- start 2024-04-26T23:09:38.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.615+0000 7f4e93223740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 0x5584434b96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.615+0000 7f4e93223740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x5584434ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.615+0000 7f4e93223740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 0x558443550760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.615+0000 7f4e93223740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558443402fd0 con 0x5584434d2d30 2024-04-26T23:09:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.615+0000 7f4e93223740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4e881fc1b0 con 0x5584434bc710 2024-04-26T23:09:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.615+0000 7f4e93223740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558443551250 con 0x5584434d9fb0 2024-04-26T23:09:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.615+0000 7f4e93223740 10 client.0 _ll_register_callbacks cb 0x7f4e700036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.615+0000 7f4e8d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x5584434ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.615+0000 7f4e8d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x5584434ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43400/0 (socket says 172.21.15.40:43400) 2024-04-26T23:09:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.615+0000 7f4e8d000640 1 -- 172.21.15.40:0/2960002658 learned_addr learned my addr 172.21.15.40:0/2960002658 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.616+0000 7f4e8d000640 1 -- 172.21.15.40:0/2960002658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 msgr2=0x558443550760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.616+0000 7f4e8d000640 1 --2- 172.21.15.40:0/2960002658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 0x558443550760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.616+0000 7f4e8d000640 1 -- 172.21.15.40:0/2960002658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 msgr2=0x5584434b96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.616+0000 7f4e8d000640 1 --2- 172.21.15.40:0/2960002658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 0x5584434b96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.616+0000 7f4e8d000640 1 -- 172.21.15.40:0/2960002658 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584433bda30 con 0x5584434d2d30 2024-04-26T23:09:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.616+0000 7f4e8d000640 1 --2- 172.21.15.40:0/2960002658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x5584434ced10 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f4e74014830 tx=0x7f4e74008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4159e985667dd913 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.616+0000 7f4e6ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.616+0000 7f4e6ea00640 1 -- 172.21.15.40:0/2960002658 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7402f030 con 0x5584434d2d30 2024-04-26T23:09:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.616+0000 7f4e6ea00640 1 -- 172.21.15.40:0/2960002658 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f4e74027050 con 0x5584434d2d30 2024-04-26T23:09:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.616+0000 7f4e93223740 1 -- 172.21.15.40:0/2960002658 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e7401b070 con 0x5584434d2d30 2024-04-26T23:09:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.616+0000 7f4e6ea00640 1 -- 172.21.15.40:0/2960002658 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7f4e74039050 con 0x5584434d2d30 2024-04-26T23:09:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.617+0000 7f4e93223740 10 client.9387 fetch_fsmap learned FSMap version 119 2024-04-26T23:09:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.617+0000 7f4e93223740 1 -- 172.21.15.40:0/2960002658 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) -- 0x5584434d4c70 con 0x5584434d2d30 2024-04-26T23:09:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.617+0000 7f4e6ea00640 1 -- 172.21.15.40:0/2960002658 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e7401b070 con 0x5584434d2d30 2024-04-26T23:09:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.617+0000 7f4e6ea00640 1 -- 172.21.15.40:0/2960002658 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 119) ==== 3550+0+0 (secure 0 0 0) 0x7f4e7403c020 con 0x5584434d2d30 2024-04-26T23:09:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.617+0000 7f4e93223740 10 client.9387 fetch_fsmap finished waiting for FSMap version 119 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.617+0000 7f4e93223740 10 client.9387 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.617+0000 7f4e93223740 20 client.9387 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.617+0000 7f4e93223740 1 --2- 172.21.15.40:0/2960002658 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5584434d8b00 0x5584434d8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.617+0000 7f4e93223740 4 client.9387 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.617+0000 7f4e93223740 1 -- 172.21.15.40:0/2960002658 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5584433fd650 con 0x5584434d8b00 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.617+0000 7f4e8da00640 1 --2- 172.21.15.40:0/2960002658 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5584434d8b00 0x5584434d8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e8da00640 1 --2- 172.21.15.40:0/2960002658 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5584434d8b00 0x5584434d8f70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb5b47923e0629c1 server_cookie=b77cc6b0e3c0cb91 in_seq=0 out_seq=0 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e6ea00640 10 client.9387 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e6ea00640 1 -- 172.21.15.40:0/2960002658 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5584433fd650 con 0x5584434d8b00 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e6ea00640 10 client.9387 handle_command_reply: tid=0 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e93223740 1 client.9387 shutdown 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e93223740 1 -- 172.21.15.40:0/2960002658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 msgr2=0x5584434ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e93223740 1 --2- 172.21.15.40:0/2960002658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x5584434ced10 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f4e74014830 tx=0x7f4e74008040 comp rx=0 tx=0).stop 2024-04-26T23:09:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e93223740 1 -- 172.21.15.40:0/2960002658 shutdown_connections 2024-04-26T23:09:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e93223740 1 --2- 172.21.15.40:0/2960002658 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5584434d9fb0 0x558443550760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e93223740 1 --2- 172.21.15.40:0/2960002658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5584434d2d30 0x5584434ced10 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e93223740 1 --2- 172.21.15.40:0/2960002658 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5584434d8b00 0x5584434d8f70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e93223740 1 --2- 172.21.15.40:0/2960002658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5584434bc710 0x5584434b96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e93223740 1 -- 172.21.15.40:0/2960002658 >> 172.21.15.40:0/2960002658 conn(0x5584434923f0 msgr2=0x5584434ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e6ea00640 0 client.9387 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e93223740 1 -- 172.21.15.40:0/2960002658 shutdown_connections 2024-04-26T23:09:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.618+0000 7f4e93223740 1 -- 172.21.15.40:0/2960002658 wait complete. 2024-04-26T23:09:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.619+0000 7f4e93223740 20 client.9387 trim_cache size 0 max 16384 2024-04-26T23:09:38.622 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-26T23:09:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:09:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "a9827531-70a9-4d6f-b12a-2d71fa886d03": { 2024-04-26T23:09:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:09:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "a9827531-70a9-4d6f-b12a-2d71fa886d03", 2024-04-26T23:09:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:09:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:38.623 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:38.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.619+0000 7f4e8ee00640 1 -- 172.21.15.40:0/2285182297 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4e6806c920 msgr2=0x7f4e6806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.619+0000 7f4e8ee00640 1 --2- 172.21.15.40:0/2285182297 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4e6806c920 0x7f4e6806ede0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4e88140690 tx=0x7f4e74006df0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.619+0000 7f4e8ee00640 1 -- 172.21.15.40:0/2285182297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e700051c0 msgr2=0x7f4e8815ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.619+0000 7f4e8ee00640 1 --2- 172.21.15.40:0/2285182297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e700051c0 0x7f4e8815ea20 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00d7d0 tx=0x7f4e7c00dc90 comp rx=0 tx=0).stop 2024-04-26T23:09:38.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.619+0000 7f4e8ee00640 1 -- 172.21.15.40:0/2285182297 shutdown_connections 2024-04-26T23:09:38.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.619+0000 7f4e8ee00640 1 --2- 172.21.15.40:0/2285182297 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4e6806c920 0x7f4e6806ede0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.619+0000 7f4e8ee00640 1 --2- 172.21.15.40:0/2285182297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e8815dac0 0x7f4e8815df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.619+0000 7f4e8ee00640 1 --2- 172.21.15.40:0/2285182297 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4e880b0590 0x7f4e8813bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.619+0000 7f4e8ee00640 1 --2- 172.21.15.40:0/2285182297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e700051c0 0x7f4e8815ea20 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.619+0000 7f4e8ee00640 1 -- 172.21.15.40:0/2285182297 >> 172.21.15.40:0/2285182297 conn(0x7f4e880b2720 msgr2=0x7f4e880d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.619+0000 7f4e8ee00640 1 -- 172.21.15.40:0/2285182297 shutdown_connections 2024-04-26T23:09:38.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.619+0000 7f4e8ee00640 1 -- 172.21.15.40:0/2285182297 wait complete. 2024-04-26T23:09:38.635 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:09:38.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:38.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.826+0000 7f80f0a00640 1 Processor -- start 2024-04-26T23:09:38.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.826+0000 7f80f0a00640 1 -- start start 2024-04-26T23:09:38.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.828+0000 7f80f0a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f80ec0c41d0 0x7f80ec0cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.828+0000 7f80f0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80ec1678f0 0x7f80ec0cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.828+0000 7f80f0a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80ec163a50 con 0x7f80ec073f30 2024-04-26T23:09:38.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.828+0000 7f80f0a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f80ec0cf7a0 con 0x7f80ec0c41d0 2024-04-26T23:09:38.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.828+0000 7f80f0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f80ec0ceb90 con 0x7f80ec1678f0 2024-04-26T23:09:38.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.829+0000 7f80ea800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80ec1678f0 0x7f80ec0cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.829+0000 7f80e9e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f80ec0c41d0 0x7f80ec0cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.829+0000 7f80e9400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f80ec073f30 0x7f80ec1671e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:53120/0 (socket says 172.21.15.40:53120) 2024-04-26T23:09:38.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.829+0000 7f80e9400640 1 -- 172.21.15.40:0/584273944 learned_addr learned my addr 172.21.15.40:0/584273944 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:38.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.830+0000 7f80ea800640 1 -- 172.21.15.40:0/584273944 >> v2:172.21.15.103:3300/0 conn(0x7f80ec0c41d0 msgr2=0x7f80ec0cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:38.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.830+0000 7f80ea800640 1 --2- 172.21.15.40:0/584273944 >> v2:172.21.15.103:3300/0 conn(0x7f80ec0c41d0 0x7f80ec0cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.830+0000 7f80ea800640 1 -- 172.21.15.40:0/584273944 >> v1:172.21.15.103:6789/0 conn(0x7f80ec073f30 legacy=0x7f80ec1671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.830+0000 7f80ea800640 1 -- 172.21.15.40:0/584273944 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ec0cee70 con 0x7f80ec1678f0 2024-04-26T23:09:38.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.831+0000 7f80ea800640 1 --2- 172.21.15.40:0/584273944 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80ec1678f0 0x7f80ec0cc240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f80e40097a0 tx=0x7f80e402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88a0ab853638fb22 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.832+0000 7f80e8a00640 1 -- 172.21.15.40:0/584273944 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4037070 con 0x7f80ec1678f0 2024-04-26T23:09:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.832+0000 7f80e8a00640 1 -- 172.21.15.40:0/584273944 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80e4032440 con 0x7f80ec1678f0 2024-04-26T23:09:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.832+0000 7f80e8a00640 1 -- 172.21.15.40:0/584273944 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4032810 con 0x7f80ec1678f0 2024-04-26T23:09:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.833+0000 7f80f0a00640 1 -- 172.21.15.40:0/584273944 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80ec1678f0 msgr2=0x7f80ec0cc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.833+0000 7f80f0a00640 1 --2- 172.21.15.40:0/584273944 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80ec1678f0 0x7f80ec0cc240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f80e40097a0 tx=0x7f80e402a180 comp rx=0 tx=0).stop 2024-04-26T23:09:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.833+0000 7f80f0a00640 1 -- 172.21.15.40:0/584273944 shutdown_connections 2024-04-26T23:09:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.833+0000 7f80f0a00640 1 --2- 172.21.15.40:0/584273944 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80ec1678f0 0x7f80ec0cc240 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.833+0000 7f80f0a00640 1 --2- 172.21.15.40:0/584273944 >> v2:172.21.15.103:3300/0 conn(0x7f80ec0c41d0 0x7f80ec0cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.833+0000 7f80f0a00640 1 -- 172.21.15.40:0/584273944 >> 172.21.15.40:0/584273944 conn(0x7f80ec0d0be0 msgr2=0x7f80ec0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.833+0000 7f80f0a00640 1 -- 172.21.15.40:0/584273944 shutdown_connections 2024-04-26T23:09:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.833+0000 7f80f0a00640 1 -- 172.21.15.40:0/584273944 wait complete. 2024-04-26T23:09:38.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.835+0000 7f80f0a00640 1 Processor -- start 2024-04-26T23:09:38.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.835+0000 7f80f0a00640 1 -- start start 2024-04-26T23:09:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.837+0000 7f80f0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80ec073f30 0x7f80ec0c0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.837+0000 7f80f0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80ec0c41d0 0x7f80ec0c3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.837+0000 7f80f0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80ec1678f0 0x7f80ec1f96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.837+0000 7f80f0a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f80ec0d0170 con 0x7f80ec073f30 2024-04-26T23:09:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.837+0000 7f80f0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f80ec0cf430 con 0x7f80ec1678f0 2024-04-26T23:09:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.837+0000 7f80f0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f80ec0ce7b0 con 0x7f80ec0c41d0 2024-04-26T23:09:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.837+0000 7f80e9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80ec073f30 0x7f80ec0c0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.837+0000 7f80e9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80ec073f30 0x7f80ec0c0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43406/0 (socket says 172.21.15.40:43406) 2024-04-26T23:09:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.837+0000 7f80e9e00640 1 -- 172.21.15.40:0/2528310813 learned_addr learned my addr 172.21.15.40:0/2528310813 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.837+0000 7f80ea800640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80ec1678f0 0x7f80ec1f96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.838+0000 7f80e9400640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80ec0c41d0 0x7f80ec0c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.838+0000 7f80e9e00640 1 -- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80ec0c41d0 msgr2=0x7f80ec0c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.838+0000 7f80e9e00640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80ec0c41d0 0x7f80ec0c3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.838+0000 7f80e9e00640 1 -- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80ec1678f0 msgr2=0x7f80ec1f96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:38.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.838+0000 7f80e9e00640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80ec1678f0 0x7f80ec1f96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:38.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.838+0000 7f80e9e00640 1 -- 172.21.15.40:0/2528310813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ec1fa0c0 con 0x7f80ec073f30 2024-04-26T23:09:38.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.838+0000 7f80e9e00640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80ec073f30 0x7f80ec0c0100 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00e9b0 tx=0x7f80dc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.838+0000 7f80e2000640 1 -- 172.21.15.40:0/2528310813 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc017070 con 0x7f80ec073f30 2024-04-26T23:09:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.838+0000 7f80e2000640 1 -- 172.21.15.40:0/2528310813 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80dc00c9a0 con 0x7f80ec073f30 2024-04-26T23:09:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.838+0000 7f80f0a00640 1 -- 172.21.15.40:0/2528310813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80ec1fa3a0 con 0x7f80ec073f30 2024-04-26T23:09:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.838+0000 7f80e2000640 1 -- 172.21.15.40:0/2528310813 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc00cd90 con 0x7f80ec073f30 2024-04-26T23:09:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.839+0000 7f80f0a00640 1 -- 172.21.15.40:0/2528310813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80ec0c2bf0 con 0x7f80ec073f30 2024-04-26T23:09:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.840+0000 7f80f0a00640 1 -- 172.21.15.40:0/2528310813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80ec0694f0 con 0x7f80ec073f30 2024-04-26T23:09:38.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.841+0000 7f80e2000640 1 -- 172.21.15.40:0/2528310813 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f80dc01a070 con 0x7f80ec073f30 2024-04-26T23:09:38.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.841+0000 7f80e2000640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f80c806c8f0 0x7f80c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:38.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.842+0000 7f80e2000640 1 -- 172.21.15.40:0/2528310813 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f80dc08c210 con 0x7f80ec073f30 2024-04-26T23:09:38.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.842+0000 7f80e9400640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f80c806c8f0 0x7f80c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:38.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.843+0000 7f80e9400640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f80c806c8f0 0x7f80c806edb0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f80ec165590 tx=0x7f80d4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:38.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:38.847+0000 7f80e2000640 1 -- 172.21.15.40:0/2528310813 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f80dc05a390 con 0x7f80ec073f30 2024-04-26T23:09:39.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.081+0000 7f80f0a00640 1 -- 172.21.15.40:0/2528310813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80ec0d0430 con 0x7f80ec073f30 2024-04-26T23:09:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.083+0000 7f80e2000640 1 -- 172.21.15.40:0/2528310813 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+5236 (secure 0 0 0) 0x7f80dc059c90 con 0x7f80ec073f30 2024-04-26T23:09:39.083 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:39.083 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":119,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:36.229817+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955,8949,6598]},"id":7}]} 2024-04-26T23:09:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 119 2024-04-26T23:09:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.085+0000 7f80f0a00640 1 -- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f80c806c8f0 msgr2=0x7f80c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.085+0000 7f80f0a00640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f80c806c8f0 0x7f80c806edb0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f80ec165590 tx=0x7f80d4009040 comp rx=0 tx=0).stop 2024-04-26T23:09:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.085+0000 7f80f0a00640 1 -- 172.21.15.40:0/2528310813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80ec073f30 msgr2=0x7f80ec0c0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.085+0000 7f80f0a00640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80ec073f30 0x7f80ec0c0100 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00e9b0 tx=0x7f80dc00ee70 comp rx=0 tx=0).stop 2024-04-26T23:09:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.085+0000 7f80f0a00640 1 -- 172.21.15.40:0/2528310813 shutdown_connections 2024-04-26T23:09:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.085+0000 7f80f0a00640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f80c806c8f0 0x7f80c806edb0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.085+0000 7f80f0a00640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80ec1678f0 0x7f80ec1f96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.085+0000 7f80f0a00640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80ec0c41d0 0x7f80ec0c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.085+0000 7f80f0a00640 1 --2- 172.21.15.40:0/2528310813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80ec073f30 0x7f80ec0c0100 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.085+0000 7f80f0a00640 1 -- 172.21.15.40:0/2528310813 >> 172.21.15.40:0/2528310813 conn(0x7f80ec0d0be0 msgr2=0x7f80ec0cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.085+0000 7f80f0a00640 1 -- 172.21.15.40:0/2528310813 shutdown_connections 2024-04-26T23:09:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.085+0000 7f80f0a00640 1 -- 172.21.15.40:0/2528310813 wait complete. 2024-04-26T23:09:39.103 DEBUG:tasks.cephfs.test_multimds_misc:current active=['a', 'c', 'd'] 2024-04-26T23:09:39.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:09:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.312+0000 7fb5b1400640 1 Processor -- start 2024-04-26T23:09:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.312+0000 7fb5b1400640 1 -- start start 2024-04-26T23:09:39.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.314+0000 7fb5b1400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5ac145970 0x7fb5ac1612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:39.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.314+0000 7fb5b1400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5ac0d06b0 0x7fb5ac0d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:39.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.314+0000 7fb5b1400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5ac0b59f0 con 0x7fb5ac146060 2024-04-26T23:09:39.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.314+0000 7fb5b1400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb5ac0ceed0 con 0x7fb5ac145970 2024-04-26T23:09:39.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.314+0000 7fb5b1400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb5ac0d1040 con 0x7fb5ac0d06b0 2024-04-26T23:09:39.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.315+0000 7fb5abe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5ac145970 0x7fb5ac1612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:39.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.315+0000 7fb5abe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5ac145970 0x7fb5ac1612b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43416/0 (socket says 172.21.15.40:43416) 2024-04-26T23:09:39.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.315+0000 7fb5abe00640 1 -- 172.21.15.40:0/3184398213 learned_addr learned my addr 172.21.15.40:0/3184398213 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:39.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.315+0000 7fb5b0a00640 1 --2- 172.21.15.40:0/3184398213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5ac0d06b0 0x7fb5ac0d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:39.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.315+0000 7fb5abe00640 1 -- 172.21.15.40:0/3184398213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5ac0d06b0 msgr2=0x7fb5ac0d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:39.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.315+0000 7fb5abe00640 1 --2- 172.21.15.40:0/3184398213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5ac0d06b0 0x7fb5ac0d0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:39.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.315+0000 7fb5abe00640 1 -- 172.21.15.40:0/3184398213 >> v1:172.21.15.103:6789/0 conn(0x7fb5ac146060 legacy=0x7fb5ac0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:39.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.315+0000 7fb5abe00640 1 -- 172.21.15.40:0/3184398213 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac1617f0 con 0x7fb5ac145970 2024-04-26T23:09:39.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.316+0000 7fb5abe00640 1 --2- 172.21.15.40:0/3184398213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ac145970 0x7fb5ac1612b0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009870 tx=0x7fb59c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124358a23dd31adc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:39.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.317+0000 7fb5aaa00640 1 -- 172.21.15.40:0/3184398213 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c037070 con 0x7fb5ac145970 2024-04-26T23:09:39.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.317+0000 7fb5aaa00640 1 -- 172.21.15.40:0/3184398213 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb59c032440 con 0x7fb5ac145970 2024-04-26T23:09:39.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.317+0000 7fb5aaa00640 1 -- 172.21.15.40:0/3184398213 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c032810 con 0x7fb5ac145970 2024-04-26T23:09:39.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.317+0000 7fb5b1400640 1 -- 172.21.15.40:0/3184398213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ac145970 msgr2=0x7fb5ac1612b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:39.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.317+0000 7fb5b1400640 1 --2- 172.21.15.40:0/3184398213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ac145970 0x7fb5ac1612b0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009870 tx=0x7fb59c02a250 comp rx=0 tx=0).stop 2024-04-26T23:09:39.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.317+0000 7fb5b1400640 1 -- 172.21.15.40:0/3184398213 shutdown_connections 2024-04-26T23:09:39.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.317+0000 7fb5b1400640 1 --2- 172.21.15.40:0/3184398213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5ac0d06b0 0x7fb5ac0d0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:39.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.317+0000 7fb5b1400640 1 --2- 172.21.15.40:0/3184398213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ac145970 0x7fb5ac1612b0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:39.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.317+0000 7fb5b1400640 1 -- 172.21.15.40:0/3184398213 >> 172.21.15.40:0/3184398213 conn(0x7fb5ac075eb0 msgr2=0x7fb5ac068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:39.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.318+0000 7fb5b1400640 1 -- 172.21.15.40:0/3184398213 shutdown_connections 2024-04-26T23:09:39.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.318+0000 7fb5b1400640 1 -- 172.21.15.40:0/3184398213 wait complete. 2024-04-26T23:09:39.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.319+0000 7fb5b1400640 1 Processor -- start 2024-04-26T23:09:39.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.319+0000 7fb5b1400640 1 -- start start 2024-04-26T23:09:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b1400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5ac0d06b0 0x7fb5ac1401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5ac145970 0x7fb5ac13b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ac146060 0x7fb5ac1f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b1400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac0b5c60 con 0x7fb5ac146060 2024-04-26T23:09:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b1400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac0b5f40 con 0x7fb5ac145970 2024-04-26T23:09:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b1400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb5ac0cf100 con 0x7fb5ac0d06b0 2024-04-26T23:09:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ac146060 0x7fb5ac1f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ac146060 0x7fb5ac1f4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43430/0 (socket says 172.21.15.40:43430) 2024-04-26T23:09:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b0a00640 1 -- 172.21.15.40:0/3621324171 learned_addr learned my addr 172.21.15.40:0/3621324171 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b0a00640 1 -- 172.21.15.40:0/3621324171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5ac0d06b0 msgr2=0x7fb5ac1401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b0a00640 1 --2- 172.21.15.40:0/3621324171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5ac0d06b0 0x7fb5ac1401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b0a00640 1 -- 172.21.15.40:0/3621324171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5ac145970 msgr2=0x7fb5ac13b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b0a00640 1 --2- 172.21.15.40:0/3621324171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5ac145970 0x7fb5ac13b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.321+0000 7fb5b0a00640 1 -- 172.21.15.40:0/3621324171 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac13a190 con 0x7fb5ac146060 2024-04-26T23:09:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.322+0000 7fb5b0a00640 1 --2- 172.21.15.40:0/3621324171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ac146060 0x7fb5ac1f4a20 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400cd70 tx=0x7fb5a400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.322+0000 7fb5a3e00640 1 -- 172.21.15.40:0/3621324171 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4003b90 con 0x7fb5ac146060 2024-04-26T23:09:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.322+0000 7fb5a3e00640 1 -- 172.21.15.40:0/3621324171 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a4003d30 con 0x7fb5ac146060 2024-04-26T23:09:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.322+0000 7fb5a3e00640 1 -- 172.21.15.40:0/3621324171 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4004240 con 0x7fb5ac146060 2024-04-26T23:09:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.322+0000 7fb5b1400640 1 -- 172.21.15.40:0/3621324171 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ac13a470 con 0x7fb5ac146060 2024-04-26T23:09:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.322+0000 7fb5b1400640 1 -- 172.21.15.40:0/3621324171 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ac13aa30 con 0x7fb5ac146060 2024-04-26T23:09:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.324+0000 7fb5b1400640 1 -- 172.21.15.40:0/3621324171 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5ac069370 con 0x7fb5ac146060 2024-04-26T23:09:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.325+0000 7fb5a3e00640 1 -- 172.21.15.40:0/3621324171 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb5a40043e0 con 0x7fb5ac146060 2024-04-26T23:09:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.326+0000 7fb5a3e00640 1 --2- 172.21.15.40:0/3621324171 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb58c06c940 0x7fb58c06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.326+0000 7fb5a3e00640 1 -- 172.21.15.40:0/3621324171 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fb5a408c730 con 0x7fb5ac146060 2024-04-26T23:09:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.326+0000 7fb5abe00640 1 --2- 172.21.15.40:0/3621324171 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb58c06c940 0x7fb58c06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:39.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.327+0000 7fb5abe00640 1 --2- 172.21.15.40:0/3621324171 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb58c06c940 0x7fb58c06ee00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb59c034000 tx=0x7fb59c02c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:39.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.330+0000 7fb5a3e00640 1 -- 172.21.15.40:0/3621324171 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fb5a405aab0 con 0x7fb5ac146060 2024-04-26T23:09:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.611+0000 7fb5b1400640 1 -- 172.21.15.40:0/3621324171 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fb5ac0d1400 con 0x7fb5ac146060 2024-04-26T23:09:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.612+0000 7fb5a3e00640 1 -- 172.21.15.40:0/3621324171 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1160 (secure 0 0 0) 0x7fb5a405a3b0 con 0x7fb5ac146060 2024-04-26T23:09:39.612 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:39.612 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":539,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":103,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":62,"data_bytes":668440,"bytes_used":267108352,"bytes_avail":772827004928,"bytes_total":773094113280,"write_bytes_sec":5032,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":119,"id":7,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":7,"rank":2,"name":"d","status":"up:active","gid":6598},{"filesystem_id":7,"rank":1,"name":"c","status":"up:active","gid":8949},{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8955}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":28,"modified":"2024-04-26T23:08:53.319384+0000","services":{}},"progress_events":{}} 2024-04-26T23:09:39.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.615+0000 7fb5b1400640 1 -- 172.21.15.40:0/3621324171 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb58c06c940 msgr2=0x7fb58c06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:39.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.615+0000 7fb5b1400640 1 --2- 172.21.15.40:0/3621324171 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb58c06c940 0x7fb58c06ee00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb59c034000 tx=0x7fb59c02c000 comp rx=0 tx=0).stop 2024-04-26T23:09:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.615+0000 7fb5b1400640 1 -- 172.21.15.40:0/3621324171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ac146060 msgr2=0x7fb5ac1f4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.615+0000 7fb5b1400640 1 --2- 172.21.15.40:0/3621324171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ac146060 0x7fb5ac1f4a20 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400cd70 tx=0x7fb5a400a660 comp rx=0 tx=0).stop 2024-04-26T23:09:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.615+0000 7fb5b1400640 1 -- 172.21.15.40:0/3621324171 shutdown_connections 2024-04-26T23:09:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.615+0000 7fb5b1400640 1 --2- 172.21.15.40:0/3621324171 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb58c06c940 0x7fb58c06ee00 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.615+0000 7fb5b1400640 1 --2- 172.21.15.40:0/3621324171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ac146060 0x7fb5ac1f4a20 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.615+0000 7fb5b1400640 1 --2- 172.21.15.40:0/3621324171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5ac145970 0x7fb5ac13b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.615+0000 7fb5b1400640 1 --2- 172.21.15.40:0/3621324171 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5ac0d06b0 0x7fb5ac1401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.615+0000 7fb5b1400640 1 -- 172.21.15.40:0/3621324171 >> 172.21.15.40:0/3621324171 conn(0x7fb5ac075eb0 msgr2=0x7fb5ac0b4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.615+0000 7fb5b1400640 1 -- 172.21.15.40:0/3621324171 shutdown_connections 2024-04-26T23:09:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:39.615+0000 7fb5b1400640 1 -- 172.21.15.40:0/3621324171 wait complete. 2024-04-26T23:09:40.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:40.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.858+0000 7ff67f200640 1 Processor -- start 2024-04-26T23:09:40.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.858+0000 7ff67f200640 1 -- start start 2024-04-26T23:09:40.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.860+0000 7ff67f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6800af7a0 0x7ff6800afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:40.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.860+0000 7ff67f200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff680071fe0 0x7ff680073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:40.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.860+0000 7ff67f200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6800d0820 con 0x7ff68015ce60 2024-04-26T23:09:40.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.860+0000 7ff67f200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7ff6800cc450 con 0x7ff680071fe0 2024-04-26T23:09:40.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.860+0000 7ff67f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff6801614c0 con 0x7ff6800af7a0 2024-04-26T23:09:40.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.860+0000 7ff67de00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff68015ce60 0x7ff6800ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44882/0 (socket says 172.21.15.40:44882) 2024-04-26T23:09:40.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.860+0000 7ff67de00640 1 -- 172.21.15.40:0/1889530305 learned_addr learned my addr 172.21.15.40:0/1889530305 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:40.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.860+0000 7ff67e800640 1 --2- 172.21.15.40:0/1889530305 >> v2:172.21.15.40:3300/0 conn(0x7ff680071fe0 0x7ff680073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:40.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.860+0000 7ff67d400640 1 --2- 172.21.15.40:0/1889530305 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6800af7a0 0x7ff6800afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:40.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.861+0000 7ff67ca00640 1 -- 172.21.15.40:0/1889530305 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1868844731 0 0) 0x7ff6800d0820 con 0x7ff68015ce60 2024-04-26T23:09:40.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.861+0000 7ff67ca00640 1 -- 172.21.15.40:0/1889530305 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff664003520 con 0x7ff68015ce60 2024-04-26T23:09:40.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.861+0000 7ff67e800640 1 -- 172.21.15.40:0/1889530305 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6800af7a0 msgr2=0x7ff6800afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:40.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.861+0000 7ff67e800640 1 --2- 172.21.15.40:0/1889530305 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6800af7a0 0x7ff6800afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:40.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.861+0000 7ff67e800640 1 -- 172.21.15.40:0/1889530305 >> v1:172.21.15.40:6789/0 conn(0x7ff68015ce60 legacy=0x7ff6800ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:40.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.861+0000 7ff67e800640 1 -- 172.21.15.40:0/1889530305 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff680161760 con 0x7ff680071fe0 2024-04-26T23:09:40.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.862+0000 7ff67e800640 1 --2- 172.21.15.40:0/1889530305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff680071fe0 0x7ff680073760 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7ff6780097a0 tx=0x7ff67802a0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf6c8a07e00d04a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:40.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.862+0000 7ff67ca00640 1 -- 172.21.15.40:0/1889530305 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678037070 con 0x7ff680071fe0 2024-04-26T23:09:40.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.862+0000 7ff67ca00640 1 -- 172.21.15.40:0/1889530305 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff67802ae40 con 0x7ff680071fe0 2024-04-26T23:09:40.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.862+0000 7ff67ca00640 1 -- 172.21.15.40:0/1889530305 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678032640 con 0x7ff680071fe0 2024-04-26T23:09:40.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.863+0000 7ff67f200640 1 -- 172.21.15.40:0/1889530305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff680071fe0 msgr2=0x7ff680073760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.863+0000 7ff67f200640 1 --2- 172.21.15.40:0/1889530305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff680071fe0 0x7ff680073760 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7ff6780097a0 tx=0x7ff67802a0f0 comp rx=0 tx=0).stop 2024-04-26T23:09:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.863+0000 7ff67f200640 1 -- 172.21.15.40:0/1889530305 shutdown_connections 2024-04-26T23:09:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.863+0000 7ff67f200640 1 --2- 172.21.15.40:0/1889530305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff680071fe0 0x7ff680073760 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.863+0000 7ff67f200640 1 --2- 172.21.15.40:0/1889530305 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6800af7a0 0x7ff6800afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.863+0000 7ff67f200640 1 -- 172.21.15.40:0/1889530305 >> 172.21.15.40:0/1889530305 conn(0x7ff6800d0a10 msgr2=0x7ff6800d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.863+0000 7ff67f200640 1 -- 172.21.15.40:0/1889530305 shutdown_connections 2024-04-26T23:09:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.863+0000 7ff67f200640 1 -- 172.21.15.40:0/1889530305 wait complete. 2024-04-26T23:09:40.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.865+0000 7ff67f200640 1 Processor -- start 2024-04-26T23:09:40.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.865+0000 7ff67f200640 1 -- start start 2024-04-26T23:09:40.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.866+0000 7ff67f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff680071fe0 0x7ff680140200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:40.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.866+0000 7ff67f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6800af7a0 0x7ff6800c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:40.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.866+0000 7ff67f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff68015ce60 0x7ff68014ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:40.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.866+0000 7ff67f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff6800cc1f0 con 0x7ff680071fe0 2024-04-26T23:09:40.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.866+0000 7ff67f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff6800cc070 con 0x7ff68015ce60 2024-04-26T23:09:40.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.866+0000 7ff67f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff680073ca0 con 0x7ff6800af7a0 2024-04-26T23:09:40.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.866+0000 7ff67de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff680071fe0 0x7ff680140200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:40.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.866+0000 7ff67de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff680071fe0 0x7ff680140200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43448/0 (socket says 172.21.15.40:43448) 2024-04-26T23:09:40.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.866+0000 7ff67de00640 1 -- 172.21.15.40:0/1326215357 learned_addr learned my addr 172.21.15.40:0/1326215357 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:40.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.867+0000 7ff67e800640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff68015ce60 0x7ff68014ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:40.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.867+0000 7ff67d400640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6800af7a0 0x7ff6800c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:40.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.867+0000 7ff67de00640 1 -- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6800af7a0 msgr2=0x7ff6800c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:40.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.867+0000 7ff67de00640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6800af7a0 0x7ff6800c6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:40.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.867+0000 7ff67de00640 1 -- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff68015ce60 msgr2=0x7ff68014ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:40.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.867+0000 7ff67de00640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff68015ce60 0x7ff68014ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:40.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.867+0000 7ff67de00640 1 -- 172.21.15.40:0/1326215357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff68015a4f0 con 0x7ff680071fe0 2024-04-26T23:09:40.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.867+0000 7ff67e800640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff68015ce60 0x7ff68014ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:40.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.867+0000 7ff67de00640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff680071fe0 0x7ff680140200 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7ff67000caa0 tx=0x7ff67000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:40.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.867+0000 7ff676000640 1 -- 172.21.15.40:0/1326215357 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670017070 con 0x7ff680071fe0 2024-04-26T23:09:40.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.867+0000 7ff676000640 1 -- 172.21.15.40:0/1326215357 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff67000ac00 con 0x7ff680071fe0 2024-04-26T23:09:40.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.867+0000 7ff676000640 1 -- 172.21.15.40:0/1326215357 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670012400 con 0x7ff680071fe0 2024-04-26T23:09:40.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.867+0000 7ff67f200640 1 -- 172.21.15.40:0/1326215357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff68015a7d0 con 0x7ff680071fe0 2024-04-26T23:09:40.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.868+0000 7ff67f200640 1 -- 172.21.15.40:0/1326215357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff68015ad90 con 0x7ff680071fe0 2024-04-26T23:09:40.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.869+0000 7ff67f200640 1 -- 172.21.15.40:0/1326215357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff680069370 con 0x7ff680071fe0 2024-04-26T23:09:40.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.870+0000 7ff676000640 1 -- 172.21.15.40:0/1326215357 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff67001a070 con 0x7ff680071fe0 2024-04-26T23:09:40.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.871+0000 7ff676000640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff65c06c8f0 0x7ff65c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:40.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.871+0000 7ff676000640 1 -- 172.21.15.40:0/1326215357 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7ff67008bbf0 con 0x7ff680071fe0 2024-04-26T23:09:40.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.872+0000 7ff67d400640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff65c06c8f0 0x7ff65c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:40.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.873+0000 7ff67d400640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff65c06c8f0 0x7ff65c06edb0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff6800c7a10 tx=0x7ff668008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:40.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:40.876+0000 7ff676000640 1 -- 172.21.15.40:0/1326215357 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff670059df0 con 0x7ff680071fe0 2024-04-26T23:09:41.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.121+0000 7ff67f200640 1 -- 172.21.15.40:0/1326215357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6800d01f0 con 0x7ff680071fe0 2024-04-26T23:09:41.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.122+0000 7ff676000640 1 -- 172.21.15.40:0/1326215357 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+5236 (secure 0 0 0) 0x7ff6700596f0 con 0x7ff680071fe0 2024-04-26T23:09:41.124 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:41.124 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":119,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:36.229817+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955,8949,6598]},"id":7}]} 2024-04-26T23:09:41.124 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 119 2024-04-26T23:09:41.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.125+0000 7ff67f200640 1 -- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff65c06c8f0 msgr2=0x7ff65c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:41.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.125+0000 7ff67f200640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff65c06c8f0 0x7ff65c06edb0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff6800c7a10 tx=0x7ff668008040 comp rx=0 tx=0).stop 2024-04-26T23:09:41.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.125+0000 7ff67f200640 1 -- 172.21.15.40:0/1326215357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff680071fe0 msgr2=0x7ff680140200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:41.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.125+0000 7ff67f200640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff680071fe0 0x7ff680140200 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7ff67000caa0 tx=0x7ff67000cf60 comp rx=0 tx=0).stop 2024-04-26T23:09:41.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.125+0000 7ff67f200640 1 -- 172.21.15.40:0/1326215357 shutdown_connections 2024-04-26T23:09:41.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.125+0000 7ff67f200640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff65c06c8f0 0x7ff65c06edb0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.125+0000 7ff67f200640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff68015ce60 0x7ff68014ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.125+0000 7ff67f200640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6800af7a0 0x7ff6800c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.125+0000 7ff67f200640 1 --2- 172.21.15.40:0/1326215357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff680071fe0 0x7ff680140200 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.125+0000 7ff67f200640 1 -- 172.21.15.40:0/1326215357 >> 172.21.15.40:0/1326215357 conn(0x7ff6800d0a10 msgr2=0x7ff680160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:41.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.125+0000 7ff67f200640 1 -- 172.21.15.40:0/1326215357 shutdown_connections 2024-04-26T23:09:41.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.126+0000 7ff67f200640 1 -- 172.21.15.40:0/1326215357 wait complete. 2024-04-26T23:09:41.145 DEBUG:tasks.cephfs.test_multimds_misc:current active=['a', 'c', 'd'] 2024-04-26T23:09:41.145 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:09:41.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.341+0000 7f3e62800640 1 Processor -- start 2024-04-26T23:09:41.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.341+0000 7f3e62800640 1 -- start start 2024-04-26T23:09:41.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.342+0000 7f3e62800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c071e60 0x7f3e5c06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:41.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.343+0000 7f3e62800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e5c15d7c0 con 0x7f3e5c06fe20 2024-04-26T23:09:41.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.343+0000 7f3e62800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e5c15daa0 con 0x7f3e5c0cc480 2024-04-26T23:09:41.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.343+0000 7f3e62800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3e5c15dd80 con 0x7f3e5c071e60 2024-04-26T23:09:41.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.343+0000 7f3e52a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3e5c06fe20 0x7f3e5c072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44886/0 (socket says 172.21.15.40:44886) 2024-04-26T23:09:41.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.343+0000 7f3e52a00640 1 -- 172.21.15.40:0/1121485478 learned_addr learned my addr 172.21.15.40:0/1121485478 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:41.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.343+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3985928771 0 0) 0x7f3e5c15d7c0 con 0x7f3e5c06fe20 2024-04-26T23:09:41.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.343+0000 7f3e5b400640 1 --2- 172.21.15.40:0/1121485478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c071e60 0x7f3e5c06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:41.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.343+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e40003520 con 0x7f3e5c06fe20 2024-04-26T23:09:41.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 437540766 0 0) 0x7f3e40003520 con 0x7f3e5c06fe20 2024-04-26T23:09:41.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c071e60 msgr2=0x7f3e5c06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5aa00640 1 --2- 172.21.15.40:0/1121485478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c071e60 0x7f3e5c06f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 >> v1:172.21.15.103:6789/0 conn(0x7f3e5c0cc480 legacy=0x7f3e5c166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5c15e060 con 0x7f3e5c06fe20 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f3e440019f0 con 0x7f3e5c06fe20 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 >> v1:172.21.15.40:6789/0 conn(0x7f3e5c06fe20 legacy=0x7f3e5c072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5aa00640 1 --2- 172.21.15.40:0/1121485478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e400039e0 0x7f3e40003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5aa00640 1 --2- 172.21.15.40:0/1121485478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c071e60 0x7f3e40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5aa00640 1 --2- 172.21.15.40:0/1121485478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e40004d40 0x7f3e400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c15dd80 con 0x7f3e400039e0 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4000ff60 con 0x7f3e40004d40 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3e40010160 con 0x7f3e5c071e60 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5b400640 1 --2- 172.21.15.40:0/1121485478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e400039e0 0x7f3e40003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e5b400640 1 -- 172.21.15.40:0/1121485478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c071e60 msgr2=0x7f3e40004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:41.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.344+0000 7f3e52a00640 1 --2- 172.21.15.40:0/1121485478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c071e60 0x7f3e40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:41.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.345+0000 7f3e5b400640 1 --2- 172.21.15.40:0/1121485478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c071e60 0x7f3e40004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.345+0000 7f3e5b400640 1 -- 172.21.15.40:0/1121485478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e40004d40 msgr2=0x7f3e400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:41.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.345+0000 7f3e5b400640 1 --2- 172.21.15.40:0/1121485478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e40004d40 0x7f3e400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.345+0000 7f3e5b400640 1 -- 172.21.15.40:0/1121485478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e40003520 con 0x7f3e400039e0 2024-04-26T23:09:41.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.345+0000 7f3e52a00640 1 --2- 172.21.15.40:0/1121485478 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c071e60 0x7f3e40004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:41.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.346+0000 7f3e5b400640 1 --2- 172.21.15.40:0/1121485478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e400039e0 0x7f3e40003de0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00ba30 tx=0x7f3e40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29deede02e2b10ff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:41.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.346+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c030070 con 0x7f3e400039e0 2024-04-26T23:09:41.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.346+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e4c02ade0 con 0x7f3e400039e0 2024-04-26T23:09:41.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.346+0000 7f3e5aa00640 1 -- 172.21.15.40:0/1121485478 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c00b600 con 0x7f3e400039e0 2024-04-26T23:09:41.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.346+0000 7f3e62800640 1 -- 172.21.15.40:0/1121485478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e400039e0 msgr2=0x7f3e40003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:41.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.346+0000 7f3e62800640 1 --2- 172.21.15.40:0/1121485478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e400039e0 0x7f3e40003de0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00ba30 tx=0x7f3e40005830 comp rx=0 tx=0).stop 2024-04-26T23:09:41.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.346+0000 7f3e5be00640 1 -- 172.21.15.40:0/1121485478 reap_dead start 2024-04-26T23:09:41.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.347+0000 7f3e62800640 1 -- 172.21.15.40:0/1121485478 shutdown_connections 2024-04-26T23:09:41.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.347+0000 7f3e62800640 1 -- 172.21.15.40:0/1121485478 >> 172.21.15.40:0/1121485478 conn(0x7f3e5c0d0be0 msgr2=0x7f3e5c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:41.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.347+0000 7f3e62800640 1 -- 172.21.15.40:0/1121485478 shutdown_connections 2024-04-26T23:09:41.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.347+0000 7f3e62800640 1 -- 172.21.15.40:0/1121485478 wait complete. 2024-04-26T23:09:41.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.348+0000 7f3e62800640 1 Processor -- start 2024-04-26T23:09:41.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.349+0000 7f3e62800640 1 -- start start 2024-04-26T23:09:41.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.349+0000 7f3e62800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e400039e0 0x7f3e5c15a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:41.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.349+0000 7f3e62800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c0b6340 0x7f3e5c14bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:41.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.349+0000 7f3e62800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e5c0b3e20 0x7f3e5c0b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:41.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.350+0000 7f3e62800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c15d570 con 0x7f3e400039e0 2024-04-26T23:09:41.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.350+0000 7f3e62800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c15d3f0 con 0x7f3e5c0b3e20 2024-04-26T23:09:41.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.350+0000 7f3e62800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3e5c072950 con 0x7f3e5c0b6340 2024-04-26T23:09:41.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.350+0000 7f3e5b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e400039e0 0x7f3e5c15a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:41.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.350+0000 7f3e5b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e400039e0 0x7f3e5c15a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43480/0 (socket says 172.21.15.40:43480) 2024-04-26T23:09:41.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.350+0000 7f3e5b400640 1 -- 172.21.15.40:0/2974667777 learned_addr learned my addr 172.21.15.40:0/2974667777 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:41.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.350+0000 7f3e52a00640 1 --2- 172.21.15.40:0/2974667777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c0b6340 0x7f3e5c14bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:41.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.350+0000 7f3e5b400640 1 -- 172.21.15.40:0/2974667777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c0b6340 msgr2=0x7f3e5c14bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:41.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.350+0000 7f3e5b400640 1 --2- 172.21.15.40:0/2974667777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c0b6340 0x7f3e5c14bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.350+0000 7f3e5b400640 1 -- 172.21.15.40:0/2974667777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e5c0b3e20 msgr2=0x7f3e5c0b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:41.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.350+0000 7f3e5b400640 1 --2- 172.21.15.40:0/2974667777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e5c0b3e20 0x7f3e5c0b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.350+0000 7f3e5b400640 1 -- 172.21.15.40:0/2974667777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5c14c440 con 0x7f3e400039e0 2024-04-26T23:09:41.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.351+0000 7f3e5b400640 1 --2- 172.21.15.40:0/2974667777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e400039e0 0x7f3e5c15a000 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00bb20 tx=0x7f3e4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:41.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.351+0000 7f3e53e00640 1 -- 172.21.15.40:0/2974667777 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c043030 con 0x7f3e400039e0 2024-04-26T23:09:41.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.351+0000 7f3e53e00640 1 -- 172.21.15.40:0/2974667777 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e4c02abe0 con 0x7f3e400039e0 2024-04-26T23:09:41.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.351+0000 7f3e53e00640 1 -- 172.21.15.40:0/2974667777 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c00b600 con 0x7f3e400039e0 2024-04-26T23:09:41.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.351+0000 7f3e62800640 1 -- 172.21.15.40:0/2974667777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e5c15a700 con 0x7f3e400039e0 2024-04-26T23:09:41.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.351+0000 7f3e62800640 1 -- 172.21.15.40:0/2974667777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e5c15acc0 con 0x7f3e400039e0 2024-04-26T23:09:41.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.352+0000 7f3e62800640 1 -- 172.21.15.40:0/2974667777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e5c0694f0 con 0x7f3e400039e0 2024-04-26T23:09:41.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.353+0000 7f3e53e00640 1 -- 172.21.15.40:0/2974667777 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3e4c046070 con 0x7f3e400039e0 2024-04-26T23:09:41.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.354+0000 7f3e53e00640 1 --2- 172.21.15.40:0/2974667777 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e3806c8f0 0x7f3e3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:41.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.354+0000 7f3e53e00640 1 -- 172.21.15.40:0/2974667777 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f3e4c0ac700 con 0x7f3e400039e0 2024-04-26T23:09:41.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.355+0000 7f3e52a00640 1 --2- 172.21.15.40:0/2974667777 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e3806c8f0 0x7f3e3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:41.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.356+0000 7f3e52a00640 1 --2- 172.21.15.40:0/2974667777 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e3806c8f0 0x7f3e3806edb0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c1f4120 tx=0x7f3e44009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:41.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.359+0000 7f3e53e00640 1 -- 172.21.15.40:0/2974667777 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f3e4c07a900 con 0x7f3e400039e0 2024-04-26T23:09:41.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.636+0000 7f3e62800640 1 -- 172.21.15.40:0/2974667777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f3e5c073df0 con 0x7f3e400039e0 2024-04-26T23:09:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.637+0000 7f3e53e00640 1 -- 172.21.15.40:0/2974667777 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1180 (secure 0 0 0) 0x7f3e4c07a200 con 0x7f3e400039e0 2024-04-26T23:09:41.637 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:41.637 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":541,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":103,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":700759,"bytes_used":267173888,"bytes_avail":772826939392,"bytes_total":773094113280,"read_bytes_sec":0,"write_bytes_sec":10150,"read_op_per_sec":0,"write_op_per_sec":8},"fsmap":{"epoch":119,"id":7,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":7,"rank":2,"name":"d","status":"up:active","gid":6598},{"filesystem_id":7,"rank":1,"name":"c","status":"up:active","gid":8949},{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8955}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":28,"modified":"2024-04-26T23:08:53.319384+0000","services":{}},"progress_events":{}} 2024-04-26T23:09:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.639+0000 7f3e62800640 1 -- 172.21.15.40:0/2974667777 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e3806c8f0 msgr2=0x7f3e3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.639+0000 7f3e62800640 1 --2- 172.21.15.40:0/2974667777 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e3806c8f0 0x7f3e3806edb0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c1f4120 tx=0x7f3e44009040 comp rx=0 tx=0).stop 2024-04-26T23:09:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.639+0000 7f3e62800640 1 -- 172.21.15.40:0/2974667777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e400039e0 msgr2=0x7f3e5c15a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.639+0000 7f3e62800640 1 --2- 172.21.15.40:0/2974667777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e400039e0 0x7f3e5c15a000 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00bb20 tx=0x7f3e4c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.639+0000 7f3e62800640 1 -- 172.21.15.40:0/2974667777 shutdown_connections 2024-04-26T23:09:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.639+0000 7f3e62800640 1 --2- 172.21.15.40:0/2974667777 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f3e3806c8f0 0x7f3e3806edb0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.639+0000 7f3e62800640 1 --2- 172.21.15.40:0/2974667777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e5c0b3e20 0x7f3e5c0b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.639+0000 7f3e62800640 1 --2- 172.21.15.40:0/2974667777 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e5c0b6340 0x7f3e5c14bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.640+0000 7f3e62800640 1 --2- 172.21.15.40:0/2974667777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e400039e0 0x7f3e5c15a000 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.640+0000 7f3e62800640 1 -- 172.21.15.40:0/2974667777 >> 172.21.15.40:0/2974667777 conn(0x7f3e5c0d0be0 msgr2=0x7f3e5c0b1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.640+0000 7f3e62800640 1 -- 172.21.15.40:0/2974667777 shutdown_connections 2024-04-26T23:09:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:41.640+0000 7f3e62800640 1 -- 172.21.15.40:0/2974667777 wait complete. 2024-04-26T23:09:42.657 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:42.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.877+0000 7f2067e00640 1 Processor -- start 2024-04-26T23:09:42.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.877+0000 7f2067e00640 1 -- start start 2024-04-26T23:09:42.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.879+0000 7f2067e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20680d3740 0x7f20680b9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:42.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.879+0000 7f2067e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2068162c40 con 0x7f20680b6220 2024-04-26T23:09:42.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.879+0000 7f2067e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20680d2d80 con 0x7f20680d32e0 2024-04-26T23:09:42.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.879+0000 7f2067e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f20680b7c80 con 0x7f20680d3740 2024-04-26T23:09:42.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.879+0000 7f2066a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f20680b6220 0x7f20680d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44900/0 (socket says 172.21.15.40:44900) 2024-04-26T23:09:42.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.879+0000 7f2066a00640 1 -- 172.21.15.40:0/775286859 learned_addr learned my addr 172.21.15.40:0/775286859 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:42.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.880+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3265633728 0 0) 0x7f2068162c40 con 0x7f20680b6220 2024-04-26T23:09:42.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.880+0000 7f2067400640 1 --2- 172.21.15.40:0/775286859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20680d3740 0x7f20680b9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:42.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.880+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f204c003520 con 0x7f20680b6220 2024-04-26T23:09:42.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.880+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 672132290 0 0) 0x7f204c003520 con 0x7f20680b6220 2024-04-26T23:09:42.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.880+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20680d3740 msgr2=0x7f20680b9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:42.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.880+0000 7f2065600640 1 --2- 172.21.15.40:0/775286859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20680d3740 0x7f20680b9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:42.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.880+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 >> v1:172.21.15.103:6789/0 conn(0x7f20680d32e0 legacy=0x7f20680b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:42.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.880+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20680b7f60 con 0x7f20680b6220 2024-04-26T23:09:42.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.880+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f2058001690 con 0x7f20680b6220 2024-04-26T23:09:42.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.880+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 >> v1:172.21.15.40:6789/0 conn(0x7f20680b6220 legacy=0x7f20680d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:42.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.880+0000 7f2065600640 1 --2- 172.21.15.40:0/775286859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20680d3740 0x7f204c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:42.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.880+0000 7f2065600640 1 --2- 172.21.15.40:0/775286859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c003fb0 0x7f204c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:42.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.881+0000 7f2065600640 1 --2- 172.21.15.40:0/775286859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204c004d40 0x7f204c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:42.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.881+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f20680b7c80 con 0x7f204c003fb0 2024-04-26T23:09:42.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.881+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f204c00bb00 con 0x7f204c004d40 2024-04-26T23:09:42.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.881+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f204c00bd00 con 0x7f20680d3740 2024-04-26T23:09:42.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.881+0000 7f2066000640 1 --2- 172.21.15.40:0/775286859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c003fb0 0x7f204c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:42.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.881+0000 7f2067400640 1 --2- 172.21.15.40:0/775286859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204c004d40 0x7f204c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:42.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.881+0000 7f2066a00640 1 --2- 172.21.15.40:0/775286859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20680d3740 0x7f204c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:42.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.881+0000 7f2066000640 1 -- 172.21.15.40:0/775286859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20680d3740 msgr2=0x7f204c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:42.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.881+0000 7f2066000640 1 --2- 172.21.15.40:0/775286859 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20680d3740 0x7f204c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:42.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.881+0000 7f2066000640 1 -- 172.21.15.40:0/775286859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204c004d40 msgr2=0x7f204c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:42.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.881+0000 7f2066000640 1 --2- 172.21.15.40:0/775286859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204c004d40 0x7f204c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:42.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.881+0000 7f2066000640 1 -- 172.21.15.40:0/775286859 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204c003520 con 0x7f204c003fb0 2024-04-26T23:09:42.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.882+0000 7f2066000640 1 --2- 172.21.15.40:0/775286859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c003fb0 0x7f204c004800 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f2054001750 tx=0x7f204c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188f52dd70d5a75f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:42.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.883+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2054001370 con 0x7f204c003fb0 2024-04-26T23:09:42.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.883+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20540052d0 con 0x7f204c003fb0 2024-04-26T23:09:42.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.883+0000 7f2065600640 1 -- 172.21.15.40:0/775286859 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205402ab20 con 0x7f204c003fb0 2024-04-26T23:09:42.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.883+0000 7f2067e00640 1 -- 172.21.15.40:0/775286859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c003fb0 msgr2=0x7f204c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:42.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.883+0000 7f2067e00640 1 --2- 172.21.15.40:0/775286859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c003fb0 0x7f204c004800 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f2054001750 tx=0x7f204c005830 comp rx=0 tx=0).stop 2024-04-26T23:09:42.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.883+0000 7f2067400640 1 -- 172.21.15.40:0/775286859 reap_dead start 2024-04-26T23:09:42.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.883+0000 7f2067e00640 1 -- 172.21.15.40:0/775286859 shutdown_connections 2024-04-26T23:09:42.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.883+0000 7f2067e00640 1 -- 172.21.15.40:0/775286859 >> 172.21.15.40:0/775286859 conn(0x7f20680d4d90 msgr2=0x7f20680d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:42.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.883+0000 7f2067e00640 1 -- 172.21.15.40:0/775286859 shutdown_connections 2024-04-26T23:09:42.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.883+0000 7f2067e00640 1 -- 172.21.15.40:0/775286859 wait complete. 2024-04-26T23:09:42.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.885+0000 7f2067e00640 1 Processor -- start 2024-04-26T23:09:42.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.885+0000 7f2067e00640 1 -- start start 2024-04-26T23:09:42.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.886+0000 7f2067e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204c003fb0 0x7f20681e82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:42.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.886+0000 7f2067e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20680b3970 0x7f206815f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:42.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.886+0000 7f2067e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20681f56e0 0x7f20680cf860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:42.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.886+0000 7f2067e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f20680d3bd0 con 0x7f20681f56e0 2024-04-26T23:09:42.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.886+0000 7f2067e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f20680b8520 con 0x7f204c003fb0 2024-04-26T23:09:42.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.886+0000 7f2067e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f20680d0aa0 con 0x7f20680b3970 2024-04-26T23:09:42.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.886+0000 7f2067400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20681f56e0 0x7f20680cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:42.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.887+0000 7f2067400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20681f56e0 0x7f20680cf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43496/0 (socket says 172.21.15.40:43496) 2024-04-26T23:09:42.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.887+0000 7f2067400640 1 -- 172.21.15.40:0/2812765042 learned_addr learned my addr 172.21.15.40:0/2812765042 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:42.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.887+0000 7f2066000640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20680b3970 0x7f206815f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:42.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.887+0000 7f2066a00640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204c003fb0 0x7f20681e82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:42.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.887+0000 7f2067400640 1 -- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20680b3970 msgr2=0x7f206815f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:42.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.887+0000 7f2067400640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20680b3970 0x7f206815f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:42.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.887+0000 7f2067400640 1 -- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204c003fb0 msgr2=0x7f20681e82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:42.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.887+0000 7f2067400640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204c003fb0 0x7f20681e82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:42.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.887+0000 7f2067400640 1 -- 172.21.15.40:0/2812765042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20680b2bb0 con 0x7f20681f56e0 2024-04-26T23:09:42.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.887+0000 7f2067400640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20681f56e0 0x7f20680cf860 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f20680d17e0 tx=0x7f20600115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:42.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.888+0000 7f205ea00640 1 -- 172.21.15.40:0/2812765042 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2060011e40 con 0x7f20681f56e0 2024-04-26T23:09:42.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.888+0000 7f205ea00640 1 -- 172.21.15.40:0/2812765042 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f206001b070 con 0x7f20681f56e0 2024-04-26T23:09:42.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.888+0000 7f205ea00640 1 -- 172.21.15.40:0/2812765042 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2060003990 con 0x7f20681f56e0 2024-04-26T23:09:42.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.888+0000 7f2067e00640 1 -- 172.21.15.40:0/2812765042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20680b2e90 con 0x7f20681f56e0 2024-04-26T23:09:42.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.888+0000 7f2067e00640 1 -- 172.21.15.40:0/2812765042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20681f5b70 con 0x7f20681f56e0 2024-04-26T23:09:42.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.889+0000 7f2067e00640 1 -- 172.21.15.40:0/2812765042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2068069370 con 0x7f20681f56e0 2024-04-26T23:09:42.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.890+0000 7f205ea00640 1 -- 172.21.15.40:0/2812765042 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2060005d10 con 0x7f20681f56e0 2024-04-26T23:09:42.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.892+0000 7f205ea00640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f204806c920 0x7f204806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:42.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.892+0000 7f205ea00640 1 -- 172.21.15.40:0/2812765042 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f206008c340 con 0x7f20681f56e0 2024-04-26T23:09:42.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.892+0000 7f2066a00640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f204806c920 0x7f204806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:42.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.893+0000 7f2066a00640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f204806c920 0x7f204806ede0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f20680cb3c0 tx=0x7f2058009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:42.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:42.896+0000 7f205ea00640 1 -- 172.21.15.40:0/2812765042 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2060025020 con 0x7f20681f56e0 2024-04-26T23:09:43.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.133+0000 7f2067e00640 1 -- 172.21.15.40:0/2812765042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20680d30c0 con 0x7f20681f56e0 2024-04-26T23:09:43.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.135+0000 7f205ea00640 1 -- 172.21.15.40:0/2812765042 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+5236 (secure 0 0 0) 0x7f2060014350 con 0x7f20681f56e0 2024-04-26T23:09:43.135 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:43.135 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":119,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:36.229817+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955,8949,6598]},"id":7}]} 2024-04-26T23:09:43.136 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 119 2024-04-26T23:09:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.137+0000 7f2067e00640 1 -- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f204806c920 msgr2=0x7f204806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.137+0000 7f2067e00640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f204806c920 0x7f204806ede0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f20680cb3c0 tx=0x7f2058009040 comp rx=0 tx=0).stop 2024-04-26T23:09:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.137+0000 7f2067e00640 1 -- 172.21.15.40:0/2812765042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20681f56e0 msgr2=0x7f20680cf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.137+0000 7f2067e00640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20681f56e0 0x7f20680cf860 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f20680d17e0 tx=0x7f20600115a0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.138+0000 7f2067e00640 1 -- 172.21.15.40:0/2812765042 shutdown_connections 2024-04-26T23:09:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.138+0000 7f2067e00640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f204806c920 0x7f204806ede0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.138+0000 7f2067e00640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20681f56e0 0x7f20680cf860 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.138+0000 7f2067e00640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f20680b3970 0x7f206815f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.138+0000 7f2067e00640 1 --2- 172.21.15.40:0/2812765042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f204c003fb0 0x7f20681e82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.138+0000 7f2067e00640 1 -- 172.21.15.40:0/2812765042 >> 172.21.15.40:0/2812765042 conn(0x7f20680d4d90 msgr2=0x7f20680ba5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.138+0000 7f2067e00640 1 -- 172.21.15.40:0/2812765042 shutdown_connections 2024-04-26T23:09:43.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.138+0000 7f2067e00640 1 -- 172.21.15.40:0/2812765042 wait complete. 2024-04-26T23:09:43.155 DEBUG:tasks.cephfs.test_multimds_misc:current active=['a', 'c', 'd'] 2024-04-26T23:09:43.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:09:43.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:09:43.194+0000 7f36ffc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-26T23:09:43.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.358+0000 7fe967a00640 1 Processor -- start 2024-04-26T23:09:43.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.358+0000 7fe967a00640 1 -- start start 2024-04-26T23:09:43.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.359+0000 7fe967a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe960162880 0x7fe9600d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:43.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.359+0000 7fe967a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9600d3bc0 0x7fe9600b8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:43.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.359+0000 7fe967a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9600b7e50 con 0x7fe9600b5f20 2024-04-26T23:09:43.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.359+0000 7fe967a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fe9600d0580 con 0x7fe9600d3bc0 2024-04-26T23:09:43.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.359+0000 7fe967a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe9600d1800 con 0x7fe960162880 2024-04-26T23:09:43.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.360+0000 7fe965a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9600d3bc0 0x7fe9600b8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:43.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.360+0000 7fe965a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9600d3bc0 0x7fe9600b8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43506/0 (socket says 172.21.15.40:43506) 2024-04-26T23:09:43.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.360+0000 7fe965a00640 1 -- 172.21.15.40:0/924300607 learned_addr learned my addr 172.21.15.40:0/924300607 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:43.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.360+0000 7fe965000640 1 --2- 172.21.15.40:0/924300607 >> v2:172.21.15.103:3300/0 conn(0x7fe960162880 0x7fe9600d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:43.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.360+0000 7fe965a00640 1 -- 172.21.15.40:0/924300607 >> v2:172.21.15.103:3300/0 conn(0x7fe960162880 msgr2=0x7fe9600d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:43.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.360+0000 7fe965a00640 1 --2- 172.21.15.40:0/924300607 >> v2:172.21.15.103:3300/0 conn(0x7fe960162880 0x7fe9600d2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.360+0000 7fe965a00640 1 -- 172.21.15.40:0/924300607 >> v1:172.21.15.103:6789/0 conn(0x7fe9600b5f20 legacy=0x7fe9600d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:43.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.360+0000 7fe965a00640 1 -- 172.21.15.40:0/924300607 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9600d07d0 con 0x7fe9600d3bc0 2024-04-26T23:09:43.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.360+0000 7fe965000640 1 --2- 172.21.15.40:0/924300607 >> v2:172.21.15.103:3300/0 conn(0x7fe960162880 0x7fe9600d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.361+0000 7fe965a00640 1 --2- 172.21.15.40:0/924300607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9600d3bc0 0x7fe9600b8300 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fe96008d490 tx=0x7fe95802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88fa74bac2238575 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.362+0000 7fe95f400640 1 -- 172.21.15.40:0/924300607 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe958036070 con 0x7fe9600d3bc0 2024-04-26T23:09:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.362+0000 7fe95f400640 1 -- 172.21.15.40:0/924300607 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe958031600 con 0x7fe9600d3bc0 2024-04-26T23:09:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.362+0000 7fe95f400640 1 -- 172.21.15.40:0/924300607 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9580319d0 con 0x7fe9600d3bc0 2024-04-26T23:09:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.362+0000 7fe967a00640 1 -- 172.21.15.40:0/924300607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9600d3bc0 msgr2=0x7fe9600b8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.362+0000 7fe967a00640 1 --2- 172.21.15.40:0/924300607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9600d3bc0 0x7fe9600b8300 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fe96008d490 tx=0x7fe95802a6f0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.362+0000 7fe967a00640 1 -- 172.21.15.40:0/924300607 shutdown_connections 2024-04-26T23:09:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.362+0000 7fe967a00640 1 --2- 172.21.15.40:0/924300607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9600d3bc0 0x7fe9600b8300 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.362+0000 7fe967a00640 1 --2- 172.21.15.40:0/924300607 >> v2:172.21.15.103:3300/0 conn(0x7fe960162880 0x7fe9600d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.362+0000 7fe967a00640 1 -- 172.21.15.40:0/924300607 >> 172.21.15.40:0/924300607 conn(0x7fe9600d4f60 msgr2=0x7fe9600d5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.363+0000 7fe967a00640 1 -- 172.21.15.40:0/924300607 shutdown_connections 2024-04-26T23:09:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.363+0000 7fe967a00640 1 -- 172.21.15.40:0/924300607 wait complete. 2024-04-26T23:09:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.364+0000 7fe967a00640 1 Processor -- start 2024-04-26T23:09:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.365+0000 7fe967a00640 1 -- start start 2024-04-26T23:09:43.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe967a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9600b5f20 0x7fe9601f5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:43.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe967a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9600d3bc0 0x7fe960153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:43.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe967a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe960162880 0x7fe9600b5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe967a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe960162c60 con 0x7fe9600b5f20 2024-04-26T23:09:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe967a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe9600d0d90 con 0x7fe9600d3bc0 2024-04-26T23:09:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe967a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe960162de0 con 0x7fe960162880 2024-04-26T23:09:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe965000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9600b5f20 0x7fe9601f5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe965000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9600b5f20 0x7fe9601f5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43508/0 (socket says 172.21.15.40:43508) 2024-04-26T23:09:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe965000640 1 -- 172.21.15.40:0/2961004741 learned_addr learned my addr 172.21.15.40:0/2961004741 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:43.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe95fe00640 1 --2- 172.21.15.40:0/2961004741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9600d3bc0 0x7fe960153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:43.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe965000640 1 -- 172.21.15.40:0/2961004741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe960162880 msgr2=0x7fe9600b5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:43.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe965000640 1 --2- 172.21.15.40:0/2961004741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe960162880 0x7fe9600b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe965000640 1 -- 172.21.15.40:0/2961004741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9600d3bc0 msgr2=0x7fe960153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:43.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe965000640 1 --2- 172.21.15.40:0/2961004741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9600d3bc0 0x7fe960153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.366+0000 7fe965000640 1 -- 172.21.15.40:0/2961004741 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9600b1b50 con 0x7fe9600b5f20 2024-04-26T23:09:43.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.367+0000 7fe965000640 1 --2- 172.21.15.40:0/2961004741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9600b5f20 0x7fe9601f5160 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fe95000c7f0 tx=0x7fe95000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.367+0000 7fe95cc00640 1 -- 172.21.15.40:0/2961004741 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe950015070 con 0x7fe9600b5f20 2024-04-26T23:09:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.367+0000 7fe95cc00640 1 -- 172.21.15.40:0/2961004741 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe950007a30 con 0x7fe9600b5f20 2024-04-26T23:09:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.367+0000 7fe95cc00640 1 -- 172.21.15.40:0/2961004741 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe950007e00 con 0x7fe9600b5f20 2024-04-26T23:09:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.367+0000 7fe967a00640 1 -- 172.21.15.40:0/2961004741 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9601f56a0 con 0x7fe9600b5f20 2024-04-26T23:09:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.367+0000 7fe967a00640 1 -- 172.21.15.40:0/2961004741 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9600b1df0 con 0x7fe9600b5f20 2024-04-26T23:09:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.368+0000 7fe967a00640 1 -- 172.21.15.40:0/2961004741 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9600694f0 con 0x7fe9600b5f20 2024-04-26T23:09:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.370+0000 7fe95cc00640 1 -- 172.21.15.40:0/2961004741 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe950018070 con 0x7fe9600b5f20 2024-04-26T23:09:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.371+0000 7fe95cc00640 1 --2- 172.21.15.40:0/2961004741 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe94006c8f0 0x7fe94006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.371+0000 7fe95cc00640 1 -- 172.21.15.40:0/2961004741 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fe95008bcf0 con 0x7fe9600b5f20 2024-04-26T23:09:43.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.372+0000 7fe95fe00640 1 --2- 172.21.15.40:0/2961004741 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe94006c8f0 0x7fe94006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:43.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.374+0000 7fe95fe00640 1 --2- 172.21.15.40:0/2961004741 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe94006c8f0 0x7fe94006edb0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fe96015e1e0 tx=0x7fe94c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:43.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.376+0000 7fe95cc00640 1 -- 172.21.15.40:0/2961004741 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fe950059ef0 con 0x7fe9600b5f20 2024-04-26T23:09:43.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.651+0000 7fe967a00640 1 -- 172.21.15.40:0/2961004741 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fe960161c90 con 0x7fe9600b5f20 2024-04-26T23:09:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.652+0000 7fe95cc00640 1 -- 172.21.15.40:0/2961004741 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1297 (secure 0 0 0) 0x7fe9500600c0 con 0x7fe9600b5f20 2024-04-26T23:09:43.652 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:43.652 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":543,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":103,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":700759,"bytes_used":267173888,"bytes_avail":772826939392,"bytes_total":773094113280,"read_bytes_sec":0,"write_bytes_sec":6738,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":120,"id":7,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":7,"rank":2,"name":"d","status":"up:active","gid":6598},{"filesystem_id":7,"rank":1,"name":"c","status":"up:active","gid":8949},{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8955}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":28,"modified":"2024-04-26T23:08:53.319384+0000","services":{}},"progress_events":{}} 2024-04-26T23:09:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.654+0000 7fe967a00640 1 -- 172.21.15.40:0/2961004741 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe94006c8f0 msgr2=0x7fe94006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.654+0000 7fe967a00640 1 --2- 172.21.15.40:0/2961004741 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe94006c8f0 0x7fe94006edb0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fe96015e1e0 tx=0x7fe94c008040 comp rx=0 tx=0).stop 2024-04-26T23:09:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.654+0000 7fe967a00640 1 -- 172.21.15.40:0/2961004741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9600b5f20 msgr2=0x7fe9601f5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.654+0000 7fe967a00640 1 --2- 172.21.15.40:0/2961004741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9600b5f20 0x7fe9601f5160 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fe95000c7f0 tx=0x7fe95000cbf0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.655+0000 7fe967a00640 1 -- 172.21.15.40:0/2961004741 shutdown_connections 2024-04-26T23:09:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.655+0000 7fe967a00640 1 --2- 172.21.15.40:0/2961004741 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fe94006c8f0 0x7fe94006edb0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.655+0000 7fe967a00640 1 --2- 172.21.15.40:0/2961004741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe960162880 0x7fe9600b5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.655+0000 7fe967a00640 1 --2- 172.21.15.40:0/2961004741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9600d3bc0 0x7fe960153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.655+0000 7fe967a00640 1 --2- 172.21.15.40:0/2961004741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9600b5f20 0x7fe9601f5160 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.655+0000 7fe967a00640 1 -- 172.21.15.40:0/2961004741 >> 172.21.15.40:0/2961004741 conn(0x7fe9600d4f60 msgr2=0x7fe9600d5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.655+0000 7fe967a00640 1 -- 172.21.15.40:0/2961004741 shutdown_connections 2024-04-26T23:09:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:43.655+0000 7fe967a00640 1 -- 172.21.15.40:0/2961004741 wait complete. 2024-04-26T23:09:44.675 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:44.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.871+0000 7f2b09600640 1 Processor -- start 2024-04-26T23:09:44.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.871+0000 7f2b09600640 1 -- start start 2024-04-26T23:09:44.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.872+0000 7f2b09600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b040b6220 0x7f2b040d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.872+0000 7f2b09600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b04162c40 con 0x7f2b040d08d0 2024-04-26T23:09:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.872+0000 7f2b09600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b040d2d80 con 0x7f2b040d39f0 2024-04-26T23:09:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.872+0000 7f2b09600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f2b040b7c80 con 0x7f2b040b6220 2024-04-26T23:09:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.873+0000 7f2b03e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b040b6220 0x7f2b040d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.873+0000 7f2b03e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b040b6220 0x7f2b040d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39534/0 (socket says 172.21.15.40:39534) 2024-04-26T23:09:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.873+0000 7f2b03e00640 1 -- 172.21.15.40:0/3918096627 learned_addr learned my addr 172.21.15.40:0/3918096627 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:44.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.873+0000 7f2b03e00640 1 -- 172.21.15.40:0/3918096627 >> v1:172.21.15.103:6789/0 conn(0x7f2b040d39f0 legacy=0x7f2b040b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.873+0000 7f2b03e00640 1 -- 172.21.15.40:0/3918096627 >> v1:172.21.15.40:6789/0 conn(0x7f2b040d08d0 legacy=0x7f2b040d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.873+0000 7f2b03e00640 1 -- 172.21.15.40:0/3918096627 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b040b7f60 con 0x7f2b040b6220 2024-04-26T23:09:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.874+0000 7f2b03e00640 1 --2- 172.21.15.40:0/3918096627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b040b6220 0x7f2b040d0390 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f2af4009e50 tx=0x7f2af402a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca1ac1701cbe2ab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.874+0000 7f2b03400640 1 -- 172.21.15.40:0/3918096627 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af4036070 con 0x7f2b040b6220 2024-04-26T23:09:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.874+0000 7f2b03400640 1 -- 172.21.15.40:0/3918096627 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2af4031600 con 0x7f2b040b6220 2024-04-26T23:09:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.874+0000 7f2b03400640 1 -- 172.21.15.40:0/3918096627 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af40319d0 con 0x7f2b040b6220 2024-04-26T23:09:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.875+0000 7f2b09600640 1 -- 172.21.15.40:0/3918096627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b040b6220 msgr2=0x7f2b040d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.875+0000 7f2b09600640 1 --2- 172.21.15.40:0/3918096627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b040b6220 0x7f2b040d0390 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f2af4009e50 tx=0x7f2af402a760 comp rx=0 tx=0).stop 2024-04-26T23:09:44.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.875+0000 7f2b09600640 1 -- 172.21.15.40:0/3918096627 shutdown_connections 2024-04-26T23:09:44.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.875+0000 7f2b09600640 1 --2- 172.21.15.40:0/3918096627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b040b6220 0x7f2b040d0390 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:44.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.875+0000 7f2b09600640 1 -- 172.21.15.40:0/3918096627 >> 172.21.15.40:0/3918096627 conn(0x7f2b040d4d90 msgr2=0x7f2b040d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:44.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.875+0000 7f2b09600640 1 -- 172.21.15.40:0/3918096627 shutdown_connections 2024-04-26T23:09:44.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.875+0000 7f2b09600640 1 -- 172.21.15.40:0/3918096627 wait complete. 2024-04-26T23:09:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.877+0000 7f2b09600640 1 Processor -- start 2024-04-26T23:09:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.877+0000 7f2b09600640 1 -- start start 2024-04-26T23:09:44.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.878+0000 7f2b09600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b040b6220 0x7f2b04161560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:44.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.878+0000 7f2b09600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b040d08d0 0x7f2b0409f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:44.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.878+0000 7f2b09600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b040d39f0 0x7f2b040cf860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:44.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.878+0000 7f2b09600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2b040b9430 con 0x7f2b040d08d0 2024-04-26T23:09:44.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.878+0000 7f2b09600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2b040b8520 con 0x7f2b040d39f0 2024-04-26T23:09:44.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.878+0000 7f2b09600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2b040b95b0 con 0x7f2b040b6220 2024-04-26T23:09:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.879+0000 7f2afb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b040d08d0 0x7f2b0409f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.879+0000 7f2afb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b040d08d0 0x7f2b0409f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39548/0 (socket says 172.21.15.40:39548) 2024-04-26T23:09:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.879+0000 7f2afb400640 1 -- 172.21.15.40:0/2460850420 learned_addr learned my addr 172.21.15.40:0/2460850420 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.879+0000 7f2afb400640 1 -- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b040b6220 msgr2=0x7f2b04161560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.879+0000 7f2b03e00640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b040b6220 0x7f2b04161560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.879+0000 7f2b08c00640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b040d39f0 0x7f2b040cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.879+0000 7f2afb400640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b040b6220 0x7f2b04161560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.879+0000 7f2afb400640 1 -- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b040d39f0 msgr2=0x7f2b040cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.879+0000 7f2afb400640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b040d39f0 0x7f2b040cf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.879+0000 7f2afb400640 1 -- 172.21.15.40:0/2460850420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0409fca0 con 0x7f2b040d08d0 2024-04-26T23:09:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.879+0000 7f2afb400640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b040d08d0 0x7f2b0409f220 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f2aec00c400 tx=0x7f2aec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:44.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.880+0000 7f2b00c00640 1 -- 172.21.15.40:0/2460850420 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aec0052d0 con 0x7f2b040d08d0 2024-04-26T23:09:44.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.880+0000 7f2b00c00640 1 -- 172.21.15.40:0/2460850420 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aec005490 con 0x7f2b040d08d0 2024-04-26T23:09:44.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.880+0000 7f2b00c00640 1 -- 172.21.15.40:0/2460850420 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aec007260 con 0x7f2b040d08d0 2024-04-26T23:09:44.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.880+0000 7f2b09600640 1 -- 172.21.15.40:0/2460850420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b0409ff80 con 0x7f2b040d08d0 2024-04-26T23:09:44.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.880+0000 7f2b09600640 1 -- 172.21.15.40:0/2460850420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b040bb650 con 0x7f2b040d08d0 2024-04-26T23:09:44.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.881+0000 7f2b09600640 1 -- 172.21.15.40:0/2460850420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b04069370 con 0x7f2b040d08d0 2024-04-26T23:09:44.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.883+0000 7f2b00c00640 1 -- 172.21.15.40:0/2460850420 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2aec007400 con 0x7f2b040d08d0 2024-04-26T23:09:44.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.884+0000 7f2b00c00640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2ae006c8f0 0x7f2ae006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:44.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.884+0000 7f2b00c00640 1 -- 172.21.15.40:0/2460850420 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f2aec08b280 con 0x7f2b040d08d0 2024-04-26T23:09:44.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.884+0000 7f2b03e00640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2ae006c8f0 0x7f2ae006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:44.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.886+0000 7f2b03e00640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2ae006c8f0 0x7f2ae006edb0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2af4033000 tx=0x7f2af4009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:44.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:44.888+0000 7f2b00c00640 1 -- 172.21.15.40:0/2460850420 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2aec059510 con 0x7f2b040d08d0 2024-04-26T23:09:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.128+0000 7f2b09600640 1 -- 172.21.15.40:0/2460850420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b040d30c0 con 0x7f2b040d08d0 2024-04-26T23:09:45.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.129+0000 7f2b00c00640 1 -- 172.21.15.40:0/2460850420 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+5238 (secure 0 0 0) 0x7f2aec058e10 con 0x7f2b040d08d0 2024-04-26T23:09:45.444 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:45.444 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":121,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:43.778286+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":5,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955,8949,6598]},"id":7}]} 2024-04-26T23:09:45.447 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 121 2024-04-26T23:09:45.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.132+0000 7f2b09600640 1 -- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2ae006c8f0 msgr2=0x7f2ae006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:45.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.132+0000 7f2b09600640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2ae006c8f0 0x7f2ae006edb0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2af4033000 tx=0x7f2af4009000 comp rx=0 tx=0).stop 2024-04-26T23:09:45.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.132+0000 7f2b09600640 1 -- 172.21.15.40:0/2460850420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b040d08d0 msgr2=0x7f2b0409f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:45.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.132+0000 7f2b09600640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b040d08d0 0x7f2b0409f220 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f2aec00c400 tx=0x7f2aec00c8c0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.132+0000 7f2b09600640 1 -- 172.21.15.40:0/2460850420 shutdown_connections 2024-04-26T23:09:45.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.132+0000 7f2b09600640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2ae006c8f0 0x7f2ae006edb0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.132+0000 7f2b09600640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b040d39f0 0x7f2b040cf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.132+0000 7f2b09600640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b040d08d0 0x7f2b0409f220 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.132+0000 7f2b09600640 1 --2- 172.21.15.40:0/2460850420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b040b6220 0x7f2b04161560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.132+0000 7f2b09600640 1 -- 172.21.15.40:0/2460850420 >> 172.21.15.40:0/2460850420 conn(0x7f2b040d4d90 msgr2=0x7f2b04162340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:45.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.132+0000 7f2b09600640 1 -- 172.21.15.40:0/2460850420 shutdown_connections 2024-04-26T23:09:45.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.132+0000 7f2b09600640 1 -- 172.21.15.40:0/2460850420 wait complete. 2024-04-26T23:09:45.450 DEBUG:tasks.cephfs.test_multimds_misc:current active=['a', 'c', 'd'] 2024-04-26T23:09:45.450 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:09:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.648+0000 7f8725200640 1 Processor -- start 2024-04-26T23:09:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.648+0000 7f8725200640 1 -- start start 2024-04-26T23:09:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.649+0000 7f8725200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f87200d0c80 0x7f87200b5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.649+0000 7f8725200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f872015f9c0 con 0x7f87200d17d0 2024-04-26T23:09:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.649+0000 7f8725200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87200d5730 con 0x7f87200d2400 2024-04-26T23:09:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.649+0000 7f8725200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f87200d37b0 con 0x7f87200d0c80 2024-04-26T23:09:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.650+0000 7f871f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f87200d17d0 0x7f87200b6180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35740/0 (socket says 172.21.15.40:35740) 2024-04-26T23:09:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.650+0000 7f871f400640 1 -- 172.21.15.40:0/1729047250 learned_addr learned my addr 172.21.15.40:0/1729047250 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.650+0000 7f871fe00640 1 --2- 172.21.15.40:0/1729047250 >> v2:172.21.15.103:3300/0 conn(0x7f87200d0c80 0x7f87200b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.650+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319364323 0 0) 0x7f872015f9c0 con 0x7f87200d17d0 2024-04-26T23:09:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.650+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8704003520 con 0x7f87200d17d0 2024-04-26T23:09:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1368447283 0 0) 0x7f8704003520 con 0x7f87200d17d0 2024-04-26T23:09:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 >> v2:172.21.15.103:3300/0 conn(0x7f87200d0c80 msgr2=0x7f87200b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871e000640 1 --2- 172.21.15.40:0/1729047250 >> v2:172.21.15.103:3300/0 conn(0x7f87200d0c80 0x7f87200b5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 >> v1:172.21.15.103:6789/0 conn(0x7f87200d2400 legacy=0x7f87200d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87200d5a10 con 0x7f87200d17d0 2024-04-26T23:09:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f8710001670 con 0x7f87200d17d0 2024-04-26T23:09:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 >> v1:172.21.15.40:6789/0 conn(0x7f87200d17d0 legacy=0x7f87200b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871e000640 1 --2- 172.21.15.40:0/1729047250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87040039b0 0x7f8704003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871e000640 1 --2- 172.21.15.40:0/1729047250 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8704004800 0x7f8704004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871e000640 1 --2- 172.21.15.40:0/1729047250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87040051c0 0x7f870400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f87200d37b0 con 0x7f87040051c0 2024-04-26T23:09:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f870400c160 con 0x7f87040039b0 2024-04-26T23:09:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f870400c360 con 0x7f8704004800 2024-04-26T23:09:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871fe00640 1 --2- 172.21.15.40:0/1729047250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87040051c0 0x7f870400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871f400640 1 --2- 172.21.15.40:0/1729047250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87040039b0 0x7f8704003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871fe00640 1 -- 172.21.15.40:0/1729047250 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8704004800 msgr2=0x7f8704004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871fe00640 1 --2- 172.21.15.40:0/1729047250 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8704004800 0x7f8704004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871fe00640 1 -- 172.21.15.40:0/1729047250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87040039b0 msgr2=0x7f8704003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.651+0000 7f871fe00640 1 --2- 172.21.15.40:0/1729047250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87040039b0 0x7f8704003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.652+0000 7f871fe00640 1 -- 172.21.15.40:0/1729047250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8704003520 con 0x7f87040051c0 2024-04-26T23:09:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.652+0000 7f871f400640 1 --2- 172.21.15.40:0/1729047250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87040039b0 0x7f8704003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.652+0000 7f871fe00640 1 -- 172.21.15.40:0/1729047250 reap_dead start 2024-04-26T23:09:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.652+0000 7f871fe00640 1 --2- 172.21.15.40:0/1729047250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87040051c0 0x7f870400b9c0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f8718009fc0 tx=0x7f8704005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b0d56e50579d67 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.653+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718035020 con 0x7f87040051c0 2024-04-26T23:09:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.653+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8718003bd0 con 0x7f87040051c0 2024-04-26T23:09:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.653+0000 7f871e000640 1 -- 172.21.15.40:0/1729047250 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718009c10 con 0x7f87040051c0 2024-04-26T23:09:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.653+0000 7f8725200640 1 -- 172.21.15.40:0/1729047250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87040051c0 msgr2=0x7f870400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.653+0000 7f8725200640 1 --2- 172.21.15.40:0/1729047250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87040051c0 0x7f870400b9c0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f8718009fc0 tx=0x7f8704005830 comp rx=0 tx=0).stop 2024-04-26T23:09:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.653+0000 7f8725200640 1 -- 172.21.15.40:0/1729047250 shutdown_connections 2024-04-26T23:09:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.653+0000 7f8725200640 1 --2- 172.21.15.40:0/1729047250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87040051c0 0x7f870400b9c0 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.653+0000 7f8725200640 1 -- 172.21.15.40:0/1729047250 >> 172.21.15.40:0/1729047250 conn(0x7f872009e210 msgr2=0x7f87200d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.654+0000 7f8725200640 1 -- 172.21.15.40:0/1729047250 shutdown_connections 2024-04-26T23:09:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.654+0000 7f8725200640 1 -- 172.21.15.40:0/1729047250 wait complete. 2024-04-26T23:09:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.655+0000 7f8725200640 1 Processor -- start 2024-04-26T23:09:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.656+0000 7f8725200640 1 -- start start 2024-04-26T23:09:45.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.657+0000 7f8725200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87040051c0 0x7f8720160ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:45.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.657+0000 7f8725200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87201572f0 0x7f8720161670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:45.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.657+0000 7f8725200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87200b8f90 0x7f87200b9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:45.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.657+0000 7f8725200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f87200d10f0 con 0x7f87040051c0 2024-04-26T23:09:45.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.657+0000 7f8725200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87200d3d70 con 0x7f87201572f0 2024-04-26T23:09:45.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.657+0000 7f8725200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f87200d1270 con 0x7f87200b8f90 2024-04-26T23:09:45.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.657+0000 7f871f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87040051c0 0x7f8720160ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:45.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.657+0000 7f871f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87040051c0 0x7f8720160ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39570/0 (socket says 172.21.15.40:39570) 2024-04-26T23:09:45.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.657+0000 7f871f400640 1 -- 172.21.15.40:0/4113452364 learned_addr learned my addr 172.21.15.40:0/4113452364 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.658+0000 7f871ea00640 1 --2- 172.21.15.40:0/4113452364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87201572f0 0x7f8720161670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.658+0000 7f871f400640 1 -- 172.21.15.40:0/4113452364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87200b8f90 msgr2=0x7f87200b9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.658+0000 7f871f400640 1 --2- 172.21.15.40:0/4113452364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87200b8f90 0x7f87200b9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.658+0000 7f871f400640 1 -- 172.21.15.40:0/4113452364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87201572f0 msgr2=0x7f8720161670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.658+0000 7f871f400640 1 --2- 172.21.15.40:0/4113452364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87201572f0 0x7f8720161670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.658+0000 7f871f400640 1 -- 172.21.15.40:0/4113452364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8720161bb0 con 0x7f87040051c0 2024-04-26T23:09:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.658+0000 7f871f400640 1 --2- 172.21.15.40:0/4113452364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87040051c0 0x7f8720160ea0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f871000d7d0 tx=0x7f871000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.658+0000 7f8717400640 1 -- 172.21.15.40:0/4113452364 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710019070 con 0x7f87040051c0 2024-04-26T23:09:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.659+0000 7f8717400640 1 -- 172.21.15.40:0/4113452364 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87100074e0 con 0x7f87040051c0 2024-04-26T23:09:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.659+0000 7f8717400640 1 -- 172.21.15.40:0/4113452364 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87100049e0 con 0x7f87040051c0 2024-04-26T23:09:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.659+0000 7f8725200640 1 -- 172.21.15.40:0/4113452364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8720161e30 con 0x7f87040051c0 2024-04-26T23:09:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.659+0000 7f8725200640 1 -- 172.21.15.40:0/4113452364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87201623f0 con 0x7f87040051c0 2024-04-26T23:09:45.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.660+0000 7f8725200640 1 -- 172.21.15.40:0/4113452364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8720069370 con 0x7f87040051c0 2024-04-26T23:09:45.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.661+0000 7f8717400640 1 -- 172.21.15.40:0/4113452364 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8710004b80 con 0x7f87040051c0 2024-04-26T23:09:45.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.662+0000 7f8717400640 1 --2- 172.21.15.40:0/4113452364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f870006c920 0x7f870006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:45.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.662+0000 7f8717400640 1 -- 172.21.15.40:0/4113452364 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f871008b640 con 0x7f87040051c0 2024-04-26T23:09:45.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.663+0000 7f871ea00640 1 --2- 172.21.15.40:0/4113452364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f870006c920 0x7f870006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:45.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.664+0000 7f871ea00640 1 --2- 172.21.15.40:0/4113452364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f870006c920 0x7f870006ede0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f87200ba860 tx=0x7f870c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:45.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.667+0000 7f8717400640 1 -- 172.21.15.40:0/4113452364 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8720161bb0 con 0x7f87040051c0 2024-04-26T23:09:45.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.934+0000 7f8725200640 1 -- 172.21.15.40:0/4113452364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f87200d5db0 con 0x7f87040051c0 2024-04-26T23:09:45.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.934+0000 7f8717400640 1 -- 172.21.15.40:0/4113452364 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1968 (secure 0 0 0) 0x7f87100598d0 con 0x7f87040051c0 2024-04-26T23:09:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:45.935 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":545,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":103,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":706274,"bytes_used":267182080,"bytes_avail":772826931200,"bytes_total":773094113280,"read_bytes_sec":0,"write_bytes_sec":7421,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":122,"id":7,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":7,"rank":2,"name":"d","status":"up:active","gid":6598},{"filesystem_id":7,"rank":1,"name":"c","status":"up:active","gid":8949},{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8955}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":29,"modified":"2024-04-26T23:09:43.338130+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:09:45.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.937+0000 7f8725200640 1 -- 172.21.15.40:0/4113452364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f870006c920 msgr2=0x7f870006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:45.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.937+0000 7f8725200640 1 --2- 172.21.15.40:0/4113452364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f870006c920 0x7f870006ede0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f87200ba860 tx=0x7f870c009040 comp rx=0 tx=0).stop 2024-04-26T23:09:45.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.937+0000 7f8725200640 1 -- 172.21.15.40:0/4113452364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87040051c0 msgr2=0x7f8720160ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:45.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.937+0000 7f8725200640 1 --2- 172.21.15.40:0/4113452364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87040051c0 0x7f8720160ea0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f871000d7d0 tx=0x7f871000dc90 comp rx=0 tx=0).stop 2024-04-26T23:09:45.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.937+0000 7f8725200640 1 -- 172.21.15.40:0/4113452364 shutdown_connections 2024-04-26T23:09:45.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.937+0000 7f8725200640 1 --2- 172.21.15.40:0/4113452364 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f870006c920 0x7f870006ede0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.937+0000 7f8725200640 1 --2- 172.21.15.40:0/4113452364 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f87200b8f90 0x7f87200b9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.937+0000 7f8725200640 1 --2- 172.21.15.40:0/4113452364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87201572f0 0x7f8720161670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.937+0000 7f8725200640 1 --2- 172.21.15.40:0/4113452364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87040051c0 0x7f8720160ea0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:45.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.937+0000 7f8725200640 1 -- 172.21.15.40:0/4113452364 >> 172.21.15.40:0/4113452364 conn(0x7f872009e210 msgr2=0x7f87200b4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:45.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.938+0000 7f8725200640 1 -- 172.21.15.40:0/4113452364 shutdown_connections 2024-04-26T23:09:45.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:45.938+0000 7f8725200640 1 -- 172.21.15.40:0/4113452364 wait complete. 2024-04-26T23:09:45.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'paused paths [/d1/d2/d3]', 'c': 'paused paths [/d1/d2/d3]', 'd': 'paused paths [/d1/d2/d3]'} 2024-04-26T23:09:45.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:45.956 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub abort 2024-04-26T23:09:46.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.158+0000 7f7836400640 1 Processor -- start 2024-04-26T23:09:46.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.158+0000 7f7836400640 1 -- start start 2024-04-26T23:09:46.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.159+0000 7f7836400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f783015f060 0x7f783015f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.159+0000 7f7836400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f78300d1f70 0x7f78300d3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.159+0000 7f7836400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78300d5c20 con 0x7f78300d3640 2024-04-26T23:09:46.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.159+0000 7f7836400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f78300c99b0 con 0x7f78300d1f70 2024-04-26T23:09:46.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.159+0000 7f7836400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f78300cfc40 con 0x7f783015f060 2024-04-26T23:09:46.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.160+0000 7f782fe00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f78300d3640 0x7f78300d1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:35332/0 (socket says 172.21.15.40:35332) 2024-04-26T23:09:46.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.160+0000 7f7835000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f783015f060 0x7f783015f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.160+0000 7f782fe00640 1 -- 172.21.15.40:0/914786291 learned_addr learned my addr 172.21.15.40:0/914786291 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:46.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.160+0000 7f7835a00640 1 --2- 172.21.15.40:0/914786291 >> v2:172.21.15.103:3300/0 conn(0x7f78300d1f70 0x7f78300d3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.161+0000 7f7835000640 1 -- 172.21.15.40:0/914786291 >> v2:172.21.15.103:3300/0 conn(0x7f78300d1f70 msgr2=0x7f78300d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.161+0000 7f7835000640 1 --2- 172.21.15.40:0/914786291 >> v2:172.21.15.103:3300/0 conn(0x7f78300d1f70 0x7f78300d3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.161+0000 7f7835000640 1 -- 172.21.15.40:0/914786291 >> v1:172.21.15.103:6789/0 conn(0x7f78300d3640 legacy=0x7f78300d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.161+0000 7f7835000640 1 -- 172.21.15.40:0/914786291 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78300cff20 con 0x7f783015f060 2024-04-26T23:09:46.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.162+0000 7f7835000640 1 --2- 172.21.15.40:0/914786291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f783015f060 0x7f783015f440 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f78240097a0 tx=0x7f782402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cadcf81db6e32b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.163+0000 7f782f400640 1 -- 172.21.15.40:0/914786291 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7824037070 con 0x7f783015f060 2024-04-26T23:09:46.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.163+0000 7f782f400640 1 -- 172.21.15.40:0/914786291 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7824032440 con 0x7f783015f060 2024-04-26T23:09:46.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.163+0000 7f782f400640 1 -- 172.21.15.40:0/914786291 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7824032810 con 0x7f783015f060 2024-04-26T23:09:46.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.163+0000 7f7836400640 1 -- 172.21.15.40:0/914786291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f783015f060 msgr2=0x7f783015f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.163+0000 7f7836400640 1 --2- 172.21.15.40:0/914786291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f783015f060 0x7f783015f440 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f78240097a0 tx=0x7f782402a180 comp rx=0 tx=0).stop 2024-04-26T23:09:46.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.163+0000 7f7836400640 1 -- 172.21.15.40:0/914786291 shutdown_connections 2024-04-26T23:09:46.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.163+0000 7f7836400640 1 --2- 172.21.15.40:0/914786291 >> v2:172.21.15.103:3300/0 conn(0x7f78300d1f70 0x7f78300d3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.163+0000 7f7836400640 1 --2- 172.21.15.40:0/914786291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f783015f060 0x7f783015f440 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.163+0000 7f7836400640 1 -- 172.21.15.40:0/914786291 >> 172.21.15.40:0/914786291 conn(0x7f78300d4e50 msgr2=0x7f78300d5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.163+0000 7f7836400640 1 -- 172.21.15.40:0/914786291 shutdown_connections 2024-04-26T23:09:46.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.163+0000 7f7836400640 1 -- 172.21.15.40:0/914786291 wait complete. 2024-04-26T23:09:46.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.165+0000 7f7836400640 1 Processor -- start 2024-04-26T23:09:46.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.165+0000 7f7836400640 1 -- start start 2024-04-26T23:09:46.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.166+0000 7f7836400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78300d1f70 0x7f78300ba3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.166+0000 7f7836400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78300d3640 0x7f783015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.166+0000 7f7836400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f783015f060 0x7f78300b8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.166+0000 7f7836400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f78300d0ce0 con 0x7f78300d1f70 2024-04-26T23:09:46.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.166+0000 7f7836400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f78300d04e0 con 0x7f78300d3640 2024-04-26T23:09:46.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.166+0000 7f7836400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f78300b9120 con 0x7f783015f060 2024-04-26T23:09:46.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.166+0000 7f7835000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78300d1f70 0x7f78300ba3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.167+0000 7f7835000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78300d1f70 0x7f78300ba3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39572/0 (socket says 172.21.15.40:39572) 2024-04-26T23:09:46.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.167+0000 7f7835000640 1 -- 172.21.15.40:0/3472529363 learned_addr learned my addr 172.21.15.40:0/3472529363 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.167+0000 7f7835000640 1 -- 172.21.15.40:0/3472529363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f783015f060 msgr2=0x7f78300b8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.167+0000 7f7835000640 1 --2- 172.21.15.40:0/3472529363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f783015f060 0x7f78300b8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.167+0000 7f7835000640 1 -- 172.21.15.40:0/3472529363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78300d3640 msgr2=0x7f783015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.167+0000 7f7835000640 1 --2- 172.21.15.40:0/3472529363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78300d3640 0x7f783015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.167+0000 7f7835000640 1 -- 172.21.15.40:0/3472529363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78300b9400 con 0x7f78300d1f70 2024-04-26T23:09:46.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.167+0000 7f7835000640 1 --2- 172.21.15.40:0/3472529363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78300d1f70 0x7f78300ba3e0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f7824000c00 tx=0x7f78240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.167+0000 7f782cc00640 1 -- 172.21.15.40:0/3472529363 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7824037040 con 0x7f78300d1f70 2024-04-26T23:09:46.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.168+0000 7f782cc00640 1 -- 172.21.15.40:0/3472529363 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7824042070 con 0x7f78300d1f70 2024-04-26T23:09:46.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.168+0000 7f782cc00640 1 -- 172.21.15.40:0/3472529363 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78240308a0 con 0x7f78300d1f70 2024-04-26T23:09:46.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.168+0000 7f7836400640 1 -- 172.21.15.40:0/3472529363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78300b9be0 con 0x7f78300d1f70 2024-04-26T23:09:46.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.168+0000 7f7836400640 1 -- 172.21.15.40:0/3472529363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7830157770 con 0x7f78300d1f70 2024-04-26T23:09:46.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.170+0000 7f782cc00640 1 -- 172.21.15.40:0/3472529363 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7824030a40 con 0x7f78300d1f70 2024-04-26T23:09:46.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.171+0000 7f782cc00640 1 --2- 172.21.15.40:0/3472529363 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f781006c8f0 0x7f781006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.171+0000 7f782cc00640 1 -- 172.21.15.40:0/3472529363 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f78240ac6b0 con 0x7f78300d1f70 2024-04-26T23:09:46.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.171+0000 7f783a7b2740 1 Processor -- start 2024-04-26T23:09:46.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.172+0000 7f782fe00640 1 --2- 172.21.15.40:0/3472529363 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f781006c8f0 0x7f781006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.172+0000 7f783a7b2740 1 -- start start 2024-04-26T23:09:46.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.172+0000 7f783a7b2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fa140 0x55b3945fa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.172+0000 7f783a7b2740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3946179e0 0x55b394617f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.172+0000 7f783a7b2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945cfe20 0x55b3946160a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.172+0000 7f783a7b2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b394533900 con 0x55b3945cfe20 2024-04-26T23:09:46.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.172+0000 7f783a7b2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b3944d01d0 con 0x55b3945fa140 2024-04-26T23:09:46.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.172+0000 7f783a7b2740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b39452f8c0 con 0x55b3946179e0 2024-04-26T23:09:46.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.173+0000 7f7835a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945cfe20 0x55b3946160a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.173+0000 7f7835a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945cfe20 0x55b3946160a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39580/0 (socket says 172.21.15.40:39580) 2024-04-26T23:09:46.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.173+0000 7f7835a00640 1 -- 172.21.15.40:0/565943449 learned_addr learned my addr 172.21.15.40:0/565943449 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.173+0000 7f782fe00640 1 --2- 172.21.15.40:0/3472529363 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f781006c8f0 0x7f781006edb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f781c000f60 tx=0x7f781c007a80 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.173+0000 7f7835000640 1 --2- 172.21.15.40:0/565943449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fa140 0x55b3945fa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.173+0000 7f7835a00640 1 -- 172.21.15.40:0/565943449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3946179e0 msgr2=0x55b394617f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.173+0000 7f7835a00640 1 --2- 172.21.15.40:0/565943449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3946179e0 0x55b394617f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.173+0000 7f7835a00640 1 -- 172.21.15.40:0/565943449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fa140 msgr2=0x55b3945fa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.173+0000 7f7835a00640 1 --2- 172.21.15.40:0/565943449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fa140 0x55b3945fa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.173+0000 7f7835a00640 1 -- 172.21.15.40:0/565943449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b39430f7d0 con 0x55b3945cfe20 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.173+0000 7f7835000640 1 --2- 172.21.15.40:0/565943449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fa140 0x55b3945fa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.173+0000 7f7835a00640 1 --2- 172.21.15.40:0/565943449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945cfe20 0x55b3946160a0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f782800e270 tx=0x7f782800e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6baa21593121ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.174+0000 7f7821600640 1 -- 172.21.15.40:0/565943449 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828018020 con 0x55b3945cfe20 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.174+0000 7f7821600640 1 -- 172.21.15.40:0/565943449 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7828014890 con 0x55b3945cfe20 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.174+0000 7f7821600640 1 -- 172.21.15.40:0/565943449 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828014c80 con 0x55b3945cfe20 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.174+0000 7f783a7b2740 1 -- 172.21.15.40:0/565943449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945cfe20 msgr2=0x55b3946160a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.174+0000 7f783a7b2740 1 --2- 172.21.15.40:0/565943449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945cfe20 0x55b3946160a0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f782800e270 tx=0x7f782800e730 comp rx=0 tx=0).stop 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.174+0000 7f783a7b2740 1 -- 172.21.15.40:0/565943449 shutdown_connections 2024-04-26T23:09:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.174+0000 7f783a7b2740 1 --2- 172.21.15.40:0/565943449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945cfe20 0x55b3946160a0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.174+0000 7f783a7b2740 1 --2- 172.21.15.40:0/565943449 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3946179e0 0x55b394617f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.174+0000 7f783a7b2740 1 --2- 172.21.15.40:0/565943449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fa140 0x55b3945fa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.174+0000 7f783a7b2740 1 -- 172.21.15.40:0/565943449 >> 172.21.15.40:0/565943449 conn(0x55b3945cc750 msgr2=0x55b394614f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.174+0000 7f783a7b2740 1 -- 172.21.15.40:0/565943449 shutdown_connections 2024-04-26T23:09:46.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.175+0000 7f783a7b2740 1 -- 172.21.15.40:0/565943449 wait complete. 2024-04-26T23:09:46.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.175+0000 7f783a7b2740 1 -- start start 2024-04-26T23:09:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.175+0000 7f783a7b2740 1 Processor -- start 2024-04-26T23:09:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.176+0000 7f783a7b2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fa140 0x55b3945fc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.176+0000 7f783a7b2740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3946179e0 0x55b3945e5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.176+0000 7f783a7b2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fdf00 0x55b3945e5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.176+0000 7f783a7b2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b3945d0bc0 con 0x55b3945fdf00 2024-04-26T23:09:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.176+0000 7f783a7b2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b3944bb120 con 0x55b3945fa140 2024-04-26T23:09:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.176+0000 7f783a7b2740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b3945ca680 con 0x55b3946179e0 2024-04-26T23:09:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.177+0000 7f783a7b2740 10 client.0 _ll_register_callbacks cb 0x55b3944f3750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.177+0000 7f7835a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fdf00 0x55b3945e5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.177+0000 7f7835a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fdf00 0x55b3945e5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39596/0 (socket says 172.21.15.40:39596) 2024-04-26T23:09:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.177+0000 7f7835a00640 1 -- 172.21.15.40:0/3322884876 learned_addr learned my addr 172.21.15.40:0/3322884876 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.177+0000 7f7835000640 1 --2- 172.21.15.40:0/3322884876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fa140 0x55b3945fc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.177+0000 7f782fe00640 1 --2- 172.21.15.40:0/3322884876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3946179e0 0x55b3945e5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.177+0000 7f7835a00640 1 -- 172.21.15.40:0/3322884876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3946179e0 msgr2=0x55b3945e5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.177+0000 7f7835a00640 1 --2- 172.21.15.40:0/3322884876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3946179e0 0x55b3945e5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.177+0000 7f7835a00640 1 -- 172.21.15.40:0/3322884876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fa140 msgr2=0x55b3945fc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.177+0000 7f7835a00640 1 --2- 172.21.15.40:0/3322884876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fa140 0x55b3945fc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.177+0000 7f7835a00640 1 -- 172.21.15.40:0/3322884876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3942050f0 con 0x55b3945fdf00 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.177+0000 7f7835a00640 1 --2- 172.21.15.40:0/3322884876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fdf00 0x55b3945e5f60 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f7828000f80 tx=0x7f7828009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5b8e50624c0b23 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.178+0000 7f7816000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.178+0000 7f7816000640 1 -- 172.21.15.40:0/3322884876 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828003ba0 con 0x55b3945fdf00 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.178+0000 7f7816000640 1 -- 172.21.15.40:0/3322884876 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f7828003d40 con 0x55b3945fdf00 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.178+0000 7f783a7b2740 1 -- 172.21.15.40:0/3322884876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7828014890 con 0x55b3945fdf00 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.178+0000 7f7816000640 1 -- 172.21.15.40:0/3322884876 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f7828031050 con 0x55b3945fdf00 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.178+0000 7f783a7b2740 10 client.9447 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.178+0000 7f783a7b2740 1 -- 172.21.15.40:0/3322884876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55b39436ff60 con 0x55b3945fdf00 2024-04-26T23:09:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.178+0000 7f7816000640 1 -- 172.21.15.40:0/3322884876 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7828014890 con 0x55b3945fdf00 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.178+0000 7f7816000640 1 -- 172.21.15.40:0/3322884876 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7f7828034020 con 0x55b3945fdf00 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.179+0000 7f783a7b2740 10 client.9447 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.179+0000 7f783a7b2740 10 client.9447 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.179+0000 7f783a7b2740 20 client.9447 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.179+0000 7f783a7b2740 1 --2- 172.21.15.40:0/3322884876 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b39460e040 0x55b39460bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.179+0000 7f783a7b2740 4 client.9447 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.179+0000 7f783a7b2740 1 -- 172.21.15.40:0/3322884876 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b394541510 con 0x55b39460e040 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.179+0000 7f7835000640 1 --2- 172.21.15.40:0/3322884876 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b39460e040 0x55b39460bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.179+0000 7f7835000640 1 --2- 172.21.15.40:0/3322884876 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b39460e040 0x55b39460bf10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=255a163f656d54d2 server_cookie=144dec9fbe9c9cbb in_seq=0 out_seq=0 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.179+0000 7f7816000640 10 client.9447 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.180+0000 7f7816000640 1 -- 172.21.15.40:0/3322884876 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b394541510 con 0x55b39460e040 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.180+0000 7f7816000640 10 client.9447 handle_command_reply: tid=0 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.180+0000 7f783a7b2740 1 client.9447 shutdown 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.180+0000 7f783a7b2740 1 -- 172.21.15.40:0/3322884876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fdf00 msgr2=0x55b3945e5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.180+0000 7f783a7b2740 1 --2- 172.21.15.40:0/3322884876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fdf00 0x55b3945e5f60 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f7828000f80 tx=0x7f7828009750 comp rx=0 tx=0).stop 2024-04-26T23:09:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.180+0000 7f783a7b2740 1 -- 172.21.15.40:0/3322884876 shutdown_connections 2024-04-26T23:09:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.180+0000 7f783a7b2740 1 --2- 172.21.15.40:0/3322884876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fdf00 0x55b3945e5f60 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.180+0000 7f783a7b2740 1 --2- 172.21.15.40:0/3322884876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3946179e0 0x55b3945e5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.180+0000 7f783a7b2740 1 --2- 172.21.15.40:0/3322884876 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b39460e040 0x55b39460bf10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.180+0000 7f783a7b2740 1 --2- 172.21.15.40:0/3322884876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fa140 0x55b3945fc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.180+0000 7f783a7b2740 1 -- 172.21.15.40:0/3322884876 >> 172.21.15.40:0/3322884876 conn(0x55b3945cfe20 msgr2=0x55b3946184d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.180+0000 7f7816000640 0 client.9447 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.181+0000 7f783a7b2740 1 -- 172.21.15.40:0/3322884876 shutdown_connections 2024-04-26T23:09:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.181+0000 7f783a7b2740 1 -- 172.21.15.40:0/3322884876 wait complete. 2024-04-26T23:09:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.181+0000 7f783a7b2740 20 client.9447 trim_cache size 0 max 16384 2024-04-26T23:09:46.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.199+0000 7f783a7b2740 1 Processor -- start 2024-04-26T23:09:46.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.199+0000 7f783a7b2740 1 -- start start 2024-04-26T23:09:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f783a7b2740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3945fa140 0x55b3945d03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f783a7b2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fdf00 0x55b394627690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f783a7b2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3946179e0 0x55b394627bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f783a7b2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7830201ca0 con 0x55b3946179e0 2024-04-26T23:09:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f783a7b2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7830200e40 con 0x55b3945fdf00 2024-04-26T23:09:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f783a7b2740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7830201570 con 0x55b3945fa140 2024-04-26T23:09:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f7835a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3946179e0 0x55b394627bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f7835a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3946179e0 0x55b394627bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39598/0 (socket says 172.21.15.40:39598) 2024-04-26T23:09:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f7835a00640 1 -- 172.21.15.40:0/2223342411 learned_addr learned my addr 172.21.15.40:0/2223342411 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f7835a00640 1 -- 172.21.15.40:0/2223342411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3945fa140 msgr2=0x55b3945d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f7835a00640 1 --2- 172.21.15.40:0/2223342411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3945fa140 0x55b3945d03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f7835a00640 1 -- 172.21.15.40:0/2223342411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fdf00 msgr2=0x55b394627690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f7835a00640 1 --2- 172.21.15.40:0/2223342411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fdf00 0x55b394627690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.200+0000 7f7835a00640 1 -- 172.21.15.40:0/2223342411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b39460f640 con 0x55b3946179e0 2024-04-26T23:09:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.201+0000 7f7835a00640 1 --2- 172.21.15.40:0/2223342411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3946179e0 0x55b394627bd0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f78280070b0 tx=0x7f7828020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f8bf22f7b8f304 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.201+0000 7f7820c00640 1 -- 172.21.15.40:0/2223342411 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782803a070 con 0x55b3946179e0 2024-04-26T23:09:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.201+0000 7f7820c00640 1 -- 172.21.15.40:0/2223342411 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7828014690 con 0x55b3946179e0 2024-04-26T23:09:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.201+0000 7f7820c00640 1 -- 172.21.15.40:0/2223342411 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828018070 con 0x55b3946179e0 2024-04-26T23:09:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.201+0000 7f783a7b2740 1 -- 172.21.15.40:0/2223342411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3946179e0 msgr2=0x55b394627bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.201+0000 7f783a7b2740 1 --2- 172.21.15.40:0/2223342411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3946179e0 0x55b394627bd0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f78280070b0 tx=0x7f7828020040 comp rx=0 tx=0).stop 2024-04-26T23:09:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.201+0000 7f783a7b2740 1 -- 172.21.15.40:0/2223342411 shutdown_connections 2024-04-26T23:09:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.201+0000 7f783a7b2740 1 --2- 172.21.15.40:0/2223342411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3946179e0 0x55b394627bd0 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.202+0000 7f783a7b2740 1 --2- 172.21.15.40:0/2223342411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3945fdf00 0x55b394627690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.202+0000 7f783a7b2740 1 --2- 172.21.15.40:0/2223342411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3945fa140 0x55b3945d03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.202+0000 7f783a7b2740 1 -- 172.21.15.40:0/2223342411 >> 172.21.15.40:0/2223342411 conn(0x55b3945cfe20 msgr2=0x55b3946186c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.202+0000 7f783a7b2740 1 -- 172.21.15.40:0/2223342411 shutdown_connections 2024-04-26T23:09:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.202+0000 7f783a7b2740 1 -- 172.21.15.40:0/2223342411 wait complete. 2024-04-26T23:09:46.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.202+0000 7f783a7b2740 1 -- start start 2024-04-26T23:09:46.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.202+0000 7f783a7b2740 1 Processor -- start 2024-04-26T23:09:46.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.203+0000 7f783a7b2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fa140 0x55b39468f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.203+0000 7f783a7b2740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3945fdf00 0x55b394616440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.203+0000 7f783a7b2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3946179e0 0x55b394616980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.203+0000 7f783a7b2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b39460d780 con 0x55b3945fa140 2024-04-26T23:09:46.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.203+0000 7f783a7b2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f78301f8b60 con 0x55b3946179e0 2024-04-26T23:09:46.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.203+0000 7f783a7b2740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b3945f9d90 con 0x55b3945fdf00 2024-04-26T23:09:46.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.204+0000 7f783a7b2740 10 client.0 _ll_register_callbacks cb 0x7f782800e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.204+0000 7f7835000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fa140 0x55b39468f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.204+0000 7f7835000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fa140 0x55b39468f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39612/0 (socket says 172.21.15.40:39612) 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.204+0000 7f7835000640 1 -- 172.21.15.40:0/649240891 learned_addr learned my addr 172.21.15.40:0/649240891 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.204+0000 7f7835000640 1 -- 172.21.15.40:0/649240891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3945fdf00 msgr2=0x55b394616440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.204+0000 7f782fe00640 1 --2- 172.21.15.40:0/649240891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3945fdf00 0x55b394616440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.204+0000 7f7835000640 1 --2- 172.21.15.40:0/649240891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3945fdf00 0x55b394616440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.204+0000 7f7835000640 1 -- 172.21.15.40:0/649240891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3946179e0 msgr2=0x55b394616980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.204+0000 7f7835000640 1 --2- 172.21.15.40:0/649240891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3946179e0 0x55b394616980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.204+0000 7f7835000640 1 -- 172.21.15.40:0/649240891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3941de1b0 con 0x55b3945fa140 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.204+0000 7f7835000640 1 --2- 172.21.15.40:0/649240891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fa140 0x55b39468f250 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f782402a1c0 tx=0x7f782407a970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac5b081a67b5ef2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.204+0000 7f7816000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f7816000640 1 -- 172.21.15.40:0/649240891 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7824082d90 con 0x55b3945fa140 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f7816000640 1 -- 172.21.15.40:0/649240891 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f78240aaa30 con 0x55b3945fa140 2024-04-26T23:09:46.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f783a7b2740 1 -- 172.21.15.40:0/649240891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7828014690 con 0x55b3945fa140 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f7816000640 1 -- 172.21.15.40:0/649240891 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f7824099050 con 0x55b3945fa140 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f783a7b2740 10 client.9453 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f783a7b2740 1 -- 172.21.15.40:0/649240891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55b3945e3f30 con 0x55b3945fa140 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f7816000640 1 -- 172.21.15.40:0/649240891 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7828014690 con 0x55b3945fa140 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f7816000640 1 -- 172.21.15.40:0/649240891 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7f782403a070 con 0x55b3945fa140 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f783a7b2740 10 client.9453 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f783a7b2740 10 client.9453 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f783a7b2740 20 client.9453 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f783a7b2740 1 --2- 172.21.15.40:0/649240891 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b39460f220 0x55b39460bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f783a7b2740 4 client.9453 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "scrub abort"}] 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.205+0000 7f783a7b2740 1 -- 172.21.15.40:0/649240891 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "scrub abort"}) -- 0x55b39453b380 con 0x55b39460f220 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.206+0000 7f782fe00640 1 --2- 172.21.15.40:0/649240891 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b39460f220 0x55b39460bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.206+0000 7f7816000640 10 client.9453 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:46.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.206+0000 7f782fe00640 1 --2- 172.21.15.40:0/649240891 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b39460f220 0x55b39460bc50 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e3bb5850b2792d2 server_cookie=b71c7e5302e3c602 in_seq=0 out_seq=0 2024-04-26T23:09:46.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.206+0000 7f7816000640 1 -- 172.21.15.40:0/649240891 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b39453b380 con 0x55b39460f220 2024-04-26T23:09:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.206+0000 7f7816000640 10 client.9453 handle_command_reply: tid=0 2024-04-26T23:09:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.207+0000 7f783a7b2740 1 client.9453 shutdown 2024-04-26T23:09:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.207+0000 7f783a7b2740 1 -- 172.21.15.40:0/649240891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fa140 msgr2=0x55b39468f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.207+0000 7f783a7b2740 1 --2- 172.21.15.40:0/649240891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fa140 0x55b39468f250 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f782402a1c0 tx=0x7f782407a970 comp rx=0 tx=0).stop 2024-04-26T23:09:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.207+0000 7f783a7b2740 1 -- 172.21.15.40:0/649240891 shutdown_connections 2024-04-26T23:09:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.207+0000 7f783a7b2740 1 --2- 172.21.15.40:0/649240891 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b39460f220 0x55b39460bc50 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.207+0000 7f783a7b2740 1 --2- 172.21.15.40:0/649240891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3946179e0 0x55b394616980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.207+0000 7f783a7b2740 1 --2- 172.21.15.40:0/649240891 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b3945fdf00 0x55b394616440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.207+0000 7f783a7b2740 1 --2- 172.21.15.40:0/649240891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3945fa140 0x55b39468f250 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.207+0000 7f7816000640 0 client.9453 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.207+0000 7f783a7b2740 1 -- 172.21.15.40:0/649240891 >> 172.21.15.40:0/649240891 conn(0x55b3945cfe20 msgr2=0x55b394618510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.207+0000 7f783a7b2740 1 -- 172.21.15.40:0/649240891 shutdown_connections 2024-04-26T23:09:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.207+0000 7f783a7b2740 1 -- 172.21.15.40:0/649240891 wait complete. 2024-04-26T23:09:46.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.207+0000 7f783a7b2740 20 client.9453 trim_cache size 0 max 16384 2024-04-26T23:09:46.211 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:46.211 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:09:46.211 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.208+0000 7f7836400640 1 -- 172.21.15.40:0/3472529363 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f781006c8f0 msgr2=0x7f781006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.208+0000 7f7836400640 1 --2- 172.21.15.40:0/3472529363 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f781006c8f0 0x7f781006edb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f781c000f60 tx=0x7f781c007a80 comp rx=0 tx=0).stop 2024-04-26T23:09:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.208+0000 7f7836400640 1 -- 172.21.15.40:0/3472529363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78300d1f70 msgr2=0x7f78300ba3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.208+0000 7f7836400640 1 --2- 172.21.15.40:0/3472529363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78300d1f70 0x7f78300ba3e0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f7824000c00 tx=0x7f78240046c0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.208+0000 7f7836400640 1 -- 172.21.15.40:0/3472529363 shutdown_connections 2024-04-26T23:09:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.208+0000 7f7836400640 1 --2- 172.21.15.40:0/3472529363 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f781006c8f0 0x7f781006edb0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.208+0000 7f7836400640 1 --2- 172.21.15.40:0/3472529363 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f783015f060 0x7f78300b8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.208+0000 7f7836400640 1 --2- 172.21.15.40:0/3472529363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78300d3640 0x7f783015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.208+0000 7f7836400640 1 --2- 172.21.15.40:0/3472529363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78300d1f70 0x7f78300ba3e0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.208+0000 7f7836400640 1 -- 172.21.15.40:0/3472529363 >> 172.21.15.40:0/3472529363 conn(0x7f78300d4e50 msgr2=0x7f783006e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.208+0000 7f7836400640 1 -- 172.21.15.40:0/3472529363 shutdown_connections 2024-04-26T23:09:46.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.208+0000 7f7836400640 1 -- 172.21.15.40:0/3472529363 wait complete. 2024-04-26T23:09:46.224 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:09:46.224 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.7:1 scrub status 2024-04-26T23:09:46.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.412+0000 7fba0b200640 1 Processor -- start 2024-04-26T23:09:46.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.412+0000 7fba0b200640 1 -- start start 2024-04-26T23:09:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.413+0000 7fba0b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c0d1770 0x7fba0c15ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.413+0000 7fba0b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fba0c0b6470 0x7fba0c15f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.413+0000 7fba0b200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba0c0d5d30 con 0x7fba0c0d1b50 2024-04-26T23:09:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.413+0000 7fba0b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fba0c0d3760 con 0x7fba0c0b6470 2024-04-26T23:09:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.413+0000 7fba0b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fba0c0d09f0 con 0x7fba0c0d1770 2024-04-26T23:09:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.414+0000 7fba0a800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fba0c0d1b50 0x7fba0c0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35744/0 (socket says 172.21.15.40:35744) 2024-04-26T23:09:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.414+0000 7fba0a800640 1 -- 172.21.15.40:0/4134298747 learned_addr learned my addr 172.21.15.40:0/4134298747 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.414+0000 7fba09e00640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c0d1770 0x7fba0c15ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.414+0000 7fba09400640 1 --2- 172.21.15.40:0/4134298747 >> v2:172.21.15.103:3300/0 conn(0x7fba0c0b6470 0x7fba0c15f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.414+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288608658 0 0) 0x7fba0c0d5d30 con 0x7fba0c0d1b50 2024-04-26T23:09:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.414+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9f0003520 con 0x7fba0c0d1b50 2024-04-26T23:09:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4140438278 0 0) 0x7fb9f0003520 con 0x7fba0c0d1b50 2024-04-26T23:09:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c0d1770 msgr2=0x7fba0c15ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c0d1770 0x7fba0c15ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 >> v2:172.21.15.103:3300/0 conn(0x7fba0c0b6470 msgr2=0x7fba0c15f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 --2- 172.21.15.40:0/4134298747 >> v2:172.21.15.103:3300/0 conn(0x7fba0c0b6470 0x7fba0c15f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0c0d5a70 con 0x7fba0c0d1b50 2024-04-26T23:09:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fba040019f0 con 0x7fba0c0d1b50 2024-04-26T23:09:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 >> v1:172.21.15.40:6789/0 conn(0x7fba0c0d1b50 legacy=0x7fba0c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9f00039b0 0x7fb9f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c0d1770 0x7fb9f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9f0004d40 0x7fb9f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c0d3760 con 0x7fb9f0004d40 2024-04-26T23:09:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c0d09f0 con 0x7fb9f00039b0 2024-04-26T23:09:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb9f000b9c0 con 0x7fba0c0d1770 2024-04-26T23:09:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.415+0000 7fba0a800640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9f0004d40 0x7fb9f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.416+0000 7fba09400640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c0d1770 0x7fb9f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.416+0000 7fba0a800640 1 -- 172.21.15.40:0/4134298747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c0d1770 msgr2=0x7fb9f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.416+0000 7fba0a800640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c0d1770 0x7fb9f0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.416+0000 7fba0a800640 1 -- 172.21.15.40:0/4134298747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9f00039b0 msgr2=0x7fb9f0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.416+0000 7fba09e00640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9f00039b0 0x7fb9f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.416+0000 7fba0a800640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9f00039b0 0x7fb9f0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.416+0000 7fba0a800640 1 -- 172.21.15.40:0/4134298747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f0003520 con 0x7fb9f0004d40 2024-04-26T23:09:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.416+0000 7fba09400640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c0d1770 0x7fb9f0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.416+0000 7fba09e00640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9f00039b0 0x7fb9f0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.417+0000 7fba0a800640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9f0004d40 0x7fb9f00051c0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fba0400a870 tx=0x7fb9f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ef61c9e2401caf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.417+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0400a4c0 con 0x7fb9f0004d40 2024-04-26T23:09:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.417+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba0402f070 con 0x7fb9f0004d40 2024-04-26T23:09:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.417+0000 7fba08a00640 1 -- 172.21.15.40:0/4134298747 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0402bce0 con 0x7fb9f0004d40 2024-04-26T23:09:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.417+0000 7fba0b200640 1 -- 172.21.15.40:0/4134298747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9f0004d40 msgr2=0x7fb9f00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.417+0000 7fba0b200640 1 --2- 172.21.15.40:0/4134298747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9f0004d40 0x7fb9f00051c0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fba0400a870 tx=0x7fb9f0005830 comp rx=0 tx=0).stop 2024-04-26T23:09:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.418+0000 7fba0a800640 1 -- 172.21.15.40:0/4134298747 reap_dead start 2024-04-26T23:09:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.418+0000 7fba0b200640 1 -- 172.21.15.40:0/4134298747 shutdown_connections 2024-04-26T23:09:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.418+0000 7fba0b200640 1 -- 172.21.15.40:0/4134298747 >> 172.21.15.40:0/4134298747 conn(0x7fba0c0d4dd0 msgr2=0x7fba0c0d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.418+0000 7fba0b200640 1 -- 172.21.15.40:0/4134298747 shutdown_connections 2024-04-26T23:09:46.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.418+0000 7fba0b200640 1 -- 172.21.15.40:0/4134298747 wait complete. 2024-04-26T23:09:46.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.419+0000 7fba0b200640 1 Processor -- start 2024-04-26T23:09:46.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.420+0000 7fba0b200640 1 -- start start 2024-04-26T23:09:46.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.420+0000 7fba0b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9f0004d40 0x7fba0c14de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.420+0000 7fba0b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c09eab0 0x7fba0c09ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba0b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba0c0a0a00 0x7fba0c0a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba0b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c0d3bf0 con 0x7fb9f0004d40 2024-04-26T23:09:46.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba0b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c0d2670 con 0x7fba0c0a0a00 2024-04-26T23:09:46.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba0b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fba0c0d3d70 con 0x7fba0c09eab0 2024-04-26T23:09:46.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba09e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9f0004d40 0x7fba0c14de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba09e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9f0004d40 0x7fba0c14de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39636/0 (socket says 172.21.15.40:39636) 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba09e00640 1 -- 172.21.15.40:0/523610700 learned_addr learned my addr 172.21.15.40:0/523610700 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba09400640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c09eab0 0x7fba0c09ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba0a800640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba0c0a0a00 0x7fba0c0a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba09e00640 1 -- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c09eab0 msgr2=0x7fba0c09ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba09e00640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c09eab0 0x7fba0c09ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba09e00640 1 -- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba0c0a0a00 msgr2=0x7fba0c0a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba09e00640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba0c0a0a00 0x7fba0c0a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba09e00640 1 -- 172.21.15.40:0/523610700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0c14e560 con 0x7fb9f0004d40 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.421+0000 7fba0a800640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba0c0a0a00 0x7fba0c0a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.422+0000 7fba09e00640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9f0004d40 0x7fba0c14de60 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc0108e0 tx=0x7fb9fc010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.422+0000 7fba02000640 1 -- 172.21.15.40:0/523610700 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc00f040 con 0x7fb9f0004d40 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.422+0000 7fba02000640 1 -- 172.21.15.40:0/523610700 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9fc00ec20 con 0x7fb9f0004d40 2024-04-26T23:09:46.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.422+0000 7fba02000640 1 -- 172.21.15.40:0/523610700 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc01c400 con 0x7fb9f0004d40 2024-04-26T23:09:46.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.422+0000 7fba0b200640 1 -- 172.21.15.40:0/523610700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0c14e840 con 0x7fb9f0004d40 2024-04-26T23:09:46.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.422+0000 7fba0b200640 1 -- 172.21.15.40:0/523610700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba0c14ee00 con 0x7fb9f0004d40 2024-04-26T23:09:46.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.424+0000 7fba02000640 1 -- 172.21.15.40:0/523610700 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb9fc01c5a0 con 0x7fb9f0004d40 2024-04-26T23:09:46.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.425+0000 7fba02000640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb9ec06c8f0 0x7fb9ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.425+0000 7fba02000640 1 -- 172.21.15.40:0/523610700 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fb9fc08c4a0 con 0x7fb9f0004d40 2024-04-26T23:09:46.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.426+0000 7fba13e64740 1 Processor -- start 2024-04-26T23:09:46.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.426+0000 7fba13e64740 1 -- start start 2024-04-26T23:09:46.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.426+0000 7fba09400640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb9ec06c8f0 0x7fb9ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba13e64740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd316710 0x55fbdd316af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba13e64740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd333fb0 0x55fbdd334500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba13e64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd2ec3f0 0x55fbdd332670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba13e64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fbdd1fc030 con 0x55fbdd2ec3f0 2024-04-26T23:09:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba13e64740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fbdd2177f0 con 0x55fbdd316710 2024-04-26T23:09:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba13e64740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55fbdd1d27e0 con 0x55fbdd333fb0 2024-04-26T23:09:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba0a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd2ec3f0 0x55fbdd332670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba0a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd2ec3f0 0x55fbdd332670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39642/0 (socket says 172.21.15.40:39642) 2024-04-26T23:09:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba0a800640 1 -- 172.21.15.40:0/1558530439 learned_addr learned my addr 172.21.15.40:0/1558530439 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba09400640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb9ec06c8f0 0x7fb9ec06edb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fba0c0cc6e0 tx=0x7fb9f8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba09e00640 1 --2- 172.21.15.40:0/1558530439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd316710 0x55fbdd316af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba09400640 1 --2- 172.21.15.40:0/1558530439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd333fb0 0x55fbdd334500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba0a800640 1 -- 172.21.15.40:0/1558530439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd333fb0 msgr2=0x55fbdd334500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba0a800640 1 --2- 172.21.15.40:0/1558530439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd333fb0 0x55fbdd334500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba0a800640 1 -- 172.21.15.40:0/1558530439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd316710 msgr2=0x55fbdd316af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba0a800640 1 --2- 172.21.15.40:0/1558530439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd316710 0x55fbdd316af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.427+0000 7fba0a800640 1 -- 172.21.15.40:0/1558530439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbdd1de050 con 0x55fbdd2ec3f0 2024-04-26T23:09:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.428+0000 7fba0a800640 1 --2- 172.21.15.40:0/1558530439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd2ec3f0 0x55fbdd332670 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fba0402b530 tx=0x7fba0402d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1141ed9bb9e635 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.428+0000 7fb9f6a00640 1 -- 172.21.15.40:0/1558530439 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba04001730 con 0x55fbdd2ec3f0 2024-04-26T23:09:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.428+0000 7fb9f6a00640 1 -- 172.21.15.40:0/1558530439 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba0402f040 con 0x55fbdd2ec3f0 2024-04-26T23:09:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.428+0000 7fb9f6a00640 1 -- 172.21.15.40:0/1558530439 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0402b880 con 0x55fbdd2ec3f0 2024-04-26T23:09:46.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.428+0000 7fba13e64740 1 -- 172.21.15.40:0/1558530439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd2ec3f0 msgr2=0x55fbdd332670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.428+0000 7fba13e64740 1 --2- 172.21.15.40:0/1558530439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd2ec3f0 0x55fbdd332670 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fba0402b530 tx=0x7fba0402d000 comp rx=0 tx=0).stop 2024-04-26T23:09:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.429+0000 7fba13e64740 1 -- 172.21.15.40:0/1558530439 shutdown_connections 2024-04-26T23:09:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.429+0000 7fba13e64740 1 --2- 172.21.15.40:0/1558530439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd2ec3f0 0x55fbdd332670 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.429+0000 7fba13e64740 1 --2- 172.21.15.40:0/1558530439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd333fb0 0x55fbdd334500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.429+0000 7fba13e64740 1 --2- 172.21.15.40:0/1558530439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd316710 0x55fbdd316af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.429+0000 7fba13e64740 1 -- 172.21.15.40:0/1558530439 >> 172.21.15.40:0/1558530439 conn(0x55fbdd2e8d20 msgr2=0x55fbdd331520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.429+0000 7fba13e64740 1 -- 172.21.15.40:0/1558530439 shutdown_connections 2024-04-26T23:09:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.429+0000 7fba13e64740 1 -- 172.21.15.40:0/1558530439 wait complete. 2024-04-26T23:09:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.429+0000 7fba13e64740 1 -- start start 2024-04-26T23:09:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.430+0000 7fba13e64740 1 Processor -- start 2024-04-26T23:09:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.430+0000 7fba13e64740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd316710 0x55fbdd312a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.430+0000 7fba13e64740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd333fb0 0x55fbdd319850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.430+0000 7fba13e64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 0x55fbdd332bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.430+0000 7fba13e64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fbdd254200 con 0x55fbdd332770 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba13e64740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fbdd24eee0 con 0x55fbdd316710 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba13e64740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55fbdd25be70 con 0x55fbdd333fb0 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba13e64740 10 client.0 _ll_register_callbacks cb 0x55fbdd245270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba0a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 0x55fbdd332bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba0a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 0x55fbdd332bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39654/0 (socket says 172.21.15.40:39654) 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba0a800640 1 -- 172.21.15.40:0/2884652183 learned_addr learned my addr 172.21.15.40:0/2884652183 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba09e00640 1 --2- 172.21.15.40:0/2884652183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd316710 0x55fbdd312a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba0a800640 1 -- 172.21.15.40:0/2884652183 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd333fb0 msgr2=0x55fbdd319850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba0a800640 1 --2- 172.21.15.40:0/2884652183 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd333fb0 0x55fbdd319850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba0a800640 1 -- 172.21.15.40:0/2884652183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd316710 msgr2=0x55fbdd312a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba0a800640 1 --2- 172.21.15.40:0/2884652183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd316710 0x55fbdd312a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba0a800640 1 -- 172.21.15.40:0/2884652183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbdd02b7d0 con 0x55fbdd332770 2024-04-26T23:09:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba09e00640 1 --2- 172.21.15.40:0/2884652183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd316710 0x55fbdd312a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fba0a800640 1 --2- 172.21.15.40:0/2884652183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 0x55fbdd332bf0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fba04042040 tx=0x7fba04001e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde7566786196486 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.431+0000 7fb9eb400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fb9eb400640 1 -- 172.21.15.40:0/2884652183 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0402bc90 con 0x55fbdd332770 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fb9eb400640 1 -- 172.21.15.40:0/2884652183 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fba04052c40 con 0x55fbdd332770 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fba13e64740 1 -- 172.21.15.40:0/2884652183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba0402f040 con 0x55fbdd332770 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fb9eb400640 1 -- 172.21.15.40:0/2884652183 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7fba04052e40 con 0x55fbdd332770 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fba13e64740 10 client.9465 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fba13e64740 1 -- 172.21.15.40:0/2884652183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55fbdcf613f0 con 0x55fbdd332770 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fb9eb400640 1 -- 172.21.15.40:0/2884652183 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba0402f040 con 0x55fbdd332770 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fb9eb400640 1 -- 172.21.15.40:0/2884652183 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7fba04034470 con 0x55fbdd332770 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fba13e64740 10 client.9465 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fba13e64740 10 client.9465 resolve_mds: resolved 7:1 to role '7:1' aka daemon mds.c 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fba13e64740 20 client.9465 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fba13e64740 1 --2- 172.21.15.40:0/2884652183 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55fbdd2e5e30 0x55fbdd2e62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fba13e64740 4 client.9465 mds_command: new command op to 8949 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.432+0000 7fba13e64740 1 -- 172.21.15.40:0/2884652183 --> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbdd25d7e0 con 0x55fbdd2e5e30 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.433+0000 7fba09e00640 1 --2- 172.21.15.40:0/2884652183 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55fbdd2e5e30 0x55fbdd2e62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.433+0000 7fba09e00640 1 --2- 172.21.15.40:0/2884652183 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55fbdd2e5e30 0x55fbdd2e62a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8be6975af01dac5a server_cookie=f4f8b2b7135b1b50 in_seq=0 out_seq=0 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.433+0000 7fb9eb400640 10 client.9465 ms_handle_connect on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.434+0000 7fb9eb400640 1 -- 172.21.15.40:0/2884652183 <== mds.1 v2:172.21.15.40:6832/3250599834 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fbdd25d7e0 con 0x55fbdd2e5e30 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.434+0000 7fb9eb400640 10 client.9465 handle_command_reply: tid=0 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.434+0000 7fba13e64740 1 client.9465 shutdown 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.434+0000 7fba13e64740 1 -- 172.21.15.40:0/2884652183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 msgr2=0x55fbdd332bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.434+0000 7fba13e64740 1 --2- 172.21.15.40:0/2884652183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 0x55fbdd332bf0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fba04042040 tx=0x7fba04001e30 comp rx=0 tx=0).stop 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.434+0000 7fba13e64740 1 -- 172.21.15.40:0/2884652183 shutdown_connections 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.434+0000 7fba13e64740 1 --2- 172.21.15.40:0/2884652183 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55fbdd2e5e30 0x55fbdd2e62a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.434+0000 7fba13e64740 1 --2- 172.21.15.40:0/2884652183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 0x55fbdd332bf0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.434+0000 7fba13e64740 1 --2- 172.21.15.40:0/2884652183 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd333fb0 0x55fbdd319850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.434+0000 7fba13e64740 1 --2- 172.21.15.40:0/2884652183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd316710 0x55fbdd312a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.434+0000 7fba13e64740 1 -- 172.21.15.40:0/2884652183 >> 172.21.15.40:0/2884652183 conn(0x55fbdd2ec3f0 msgr2=0x55fbdd330ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.434+0000 7fb9eb400640 0 client.9465 ms_handle_reset on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.434+0000 7fba13e64740 1 -- 172.21.15.40:0/2884652183 shutdown_connections 2024-04-26T23:09:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.435+0000 7fba13e64740 1 -- 172.21.15.40:0/2884652183 wait complete. 2024-04-26T23:09:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.435+0000 7fba13e64740 20 client.9465 trim_cache size 0 max 16384 2024-04-26T23:09:46.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.453+0000 7fba13e64740 1 Processor -- start 2024-04-26T23:09:46.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.453+0000 7fba13e64740 1 -- start start 2024-04-26T23:09:46.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.454+0000 7fba13e64740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd316710 0x55fbdd34b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.454+0000 7fba13e64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 0x55fbdd3403f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.454+0000 7fba13e64740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd333fb0 0x55fbdd340930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.454+0000 7fba13e64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c202210 con 0x55fbdd332770 2024-04-26T23:09:46.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.454+0000 7fba13e64740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c2013b0 con 0x55fbdd333fb0 2024-04-26T23:09:46.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.454+0000 7fba13e64740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fba0c201ae0 con 0x55fbdd316710 2024-04-26T23:09:46.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.454+0000 7fba09400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 0x55fbdd3403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.455+0000 7fba09400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 0x55fbdd3403f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39666/0 (socket says 172.21.15.40:39666) 2024-04-26T23:09:46.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.455+0000 7fba09400640 1 -- 172.21.15.40:0/3961382360 learned_addr learned my addr 172.21.15.40:0/3961382360 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.455+0000 7fba09e00640 1 --2- 172.21.15.40:0/3961382360 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd316710 0x55fbdd34b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.455+0000 7fba0a800640 1 --2- 172.21.15.40:0/3961382360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd333fb0 0x55fbdd340930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.455+0000 7fba09400640 1 -- 172.21.15.40:0/3961382360 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd316710 msgr2=0x55fbdd34b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.455+0000 7fba09400640 1 --2- 172.21.15.40:0/3961382360 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd316710 0x55fbdd34b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.455+0000 7fba09400640 1 -- 172.21.15.40:0/3961382360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd333fb0 msgr2=0x55fbdd340930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.455+0000 7fba09400640 1 --2- 172.21.15.40:0/3961382360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd333fb0 0x55fbdd340930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.455+0000 7fba09400640 1 -- 172.21.15.40:0/3961382360 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbdcf210f0 con 0x55fbdd332770 2024-04-26T23:09:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.455+0000 7fba09400640 1 --2- 172.21.15.40:0/3961382360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 0x55fbdd3403f0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8012cd0 tx=0x7fb9f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da8ba660a29cc36 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.456+0000 7fb9f6000640 1 -- 172.21.15.40:0/3961382360 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f800f040 con 0x55fbdd332770 2024-04-26T23:09:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.456+0000 7fb9f6000640 1 -- 172.21.15.40:0/3961382360 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f8027070 con 0x55fbdd332770 2024-04-26T23:09:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.456+0000 7fb9f6000640 1 -- 172.21.15.40:0/3961382360 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8022440 con 0x55fbdd332770 2024-04-26T23:09:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.456+0000 7fba13e64740 1 -- 172.21.15.40:0/3961382360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 msgr2=0x55fbdd3403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.456+0000 7fba13e64740 1 --2- 172.21.15.40:0/3961382360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 0x55fbdd3403f0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8012cd0 tx=0x7fb9f80046c0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.456+0000 7fba13e64740 1 -- 172.21.15.40:0/3961382360 shutdown_connections 2024-04-26T23:09:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.456+0000 7fba13e64740 1 --2- 172.21.15.40:0/3961382360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd333fb0 0x55fbdd340930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.456+0000 7fba13e64740 1 --2- 172.21.15.40:0/3961382360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd332770 0x55fbdd3403f0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.456+0000 7fba13e64740 1 --2- 172.21.15.40:0/3961382360 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd316710 0x55fbdd34b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.456+0000 7fba13e64740 1 -- 172.21.15.40:0/3961382360 >> 172.21.15.40:0/3961382360 conn(0x55fbdd2ec3f0 msgr2=0x55fbdd330b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.456+0000 7fba13e64740 1 -- 172.21.15.40:0/3961382360 shutdown_connections 2024-04-26T23:09:46.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.457+0000 7fba13e64740 1 -- 172.21.15.40:0/3961382360 wait complete. 2024-04-26T23:09:46.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.457+0000 7fba13e64740 1 -- start start 2024-04-26T23:09:46.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.457+0000 7fba13e64740 1 Processor -- start 2024-04-26T23:09:46.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.458+0000 7fba13e64740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd316710 0x55fbdd3173d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.458+0000 7fba13e64740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd333fb0 0x55fbdd3314f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.458+0000 7fba13e64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd317910 0x55fbdd3403f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.458+0000 7fba13e64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fbdd25cfd0 con 0x55fbdd317910 2024-04-26T23:09:46.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.458+0000 7fba13e64740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c2004a0 con 0x55fbdd333fb0 2024-04-26T23:09:46.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.458+0000 7fba13e64740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55fbdd319c10 con 0x55fbdd316710 2024-04-26T23:09:46.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.458+0000 7fba13e64740 10 client.0 _ll_register_callbacks cb 0x7fb9f8009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:46.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.458+0000 7fba0a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd317910 0x55fbdd3403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fba0a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd317910 0x55fbdd3403f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39676/0 (socket says 172.21.15.40:39676) 2024-04-26T23:09:46.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fba0a800640 1 -- 172.21.15.40:0/17955758 learned_addr learned my addr 172.21.15.40:0/17955758 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fba09e00640 1 --2- 172.21.15.40:0/17955758 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd316710 0x55fbdd3173d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fba09400640 1 --2- 172.21.15.40:0/17955758 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd333fb0 0x55fbdd3314f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fba0a800640 1 -- 172.21.15.40:0/17955758 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd316710 msgr2=0x55fbdd3173d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fba0a800640 1 --2- 172.21.15.40:0/17955758 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd316710 0x55fbdd3173d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fba0a800640 1 -- 172.21.15.40:0/17955758 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd333fb0 msgr2=0x55fbdd3314f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fba0a800640 1 --2- 172.21.15.40:0/17955758 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd333fb0 0x55fbdd3314f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fba0a800640 1 -- 172.21.15.40:0/17955758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbdd217a30 con 0x55fbdd317910 2024-04-26T23:09:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fba0a800640 1 --2- 172.21.15.40:0/17955758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd317910 0x55fbdd3403f0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fba0402b520 tx=0x7fba0403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66448f692535e07f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fb9eb400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fb9eb400640 1 -- 172.21.15.40:0/17955758 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba04034b70 con 0x55fbdd317910 2024-04-26T23:09:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fb9eb400640 1 -- 172.21.15.40:0/17955758 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fba0402ba90 con 0x55fbdd317910 2024-04-26T23:09:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.459+0000 7fba13e64740 1 -- 172.21.15.40:0/17955758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9f8027070 con 0x55fbdd317910 2024-04-26T23:09:46.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.460+0000 7fb9eb400640 1 -- 172.21.15.40:0/17955758 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7fba04066050 con 0x55fbdd317910 2024-04-26T23:09:46.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.460+0000 7fba13e64740 10 client.9471 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:46.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.460+0000 7fba13e64740 1 -- 172.21.15.40:0/17955758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55fbdd331ba0 con 0x55fbdd317910 2024-04-26T23:09:46.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.460+0000 7fb9eb400640 1 -- 172.21.15.40:0/17955758 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f8027070 con 0x55fbdd317910 2024-04-26T23:09:46.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.460+0000 7fb9eb400640 1 -- 172.21.15.40:0/17955758 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7fba04069020 con 0x55fbdd317910 2024-04-26T23:09:46.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.460+0000 7fba13e64740 10 client.9471 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:46.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.460+0000 7fba13e64740 10 client.9471 resolve_mds: resolved 7:1 to role '7:1' aka daemon mds.c 2024-04-26T23:09:46.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.460+0000 7fba13e64740 20 client.9471 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:46.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.460+0000 7fba13e64740 1 --2- 172.21.15.40:0/17955758 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55fbdd2e64f0 0x55fbdd340930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.460+0000 7fba13e64740 4 client.9471 mds_command: new command op to 8949 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:09:46.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.460+0000 7fba13e64740 1 -- 172.21.15.40:0/17955758 --> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fbdd257650 con 0x55fbdd2e64f0 2024-04-26T23:09:46.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.460+0000 7fba09e00640 1 --2- 172.21.15.40:0/17955758 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55fbdd2e64f0 0x55fbdd340930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.461+0000 7fba09e00640 1 --2- 172.21.15.40:0/17955758 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55fbdd2e64f0 0x55fbdd340930 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=feecc2b7d81848d0 server_cookie=2cbfb43308a0e0f1 in_seq=0 out_seq=0 2024-04-26T23:09:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.461+0000 7fb9eb400640 10 client.9471 ms_handle_connect on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.461+0000 7fb9eb400640 1 -- 172.21.15.40:0/17955758 <== mds.1 v2:172.21.15.40:6832/3250599834 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55fbdd257650 con 0x55fbdd2e64f0 2024-04-26T23:09:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.461+0000 7fb9eb400640 10 client.9471 handle_command_reply: tid=0 2024-04-26T23:09:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.461+0000 7fba13e64740 1 client.9471 shutdown 2024-04-26T23:09:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.461+0000 7fba13e64740 1 -- 172.21.15.40:0/17955758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd317910 msgr2=0x55fbdd3403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.461+0000 7fba13e64740 1 --2- 172.21.15.40:0/17955758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd317910 0x55fbdd3403f0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fba0402b520 tx=0x7fba0403d000 comp rx=0 tx=0).stop 2024-04-26T23:09:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.461+0000 7fba13e64740 1 -- 172.21.15.40:0/17955758 shutdown_connections 2024-04-26T23:09:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.461+0000 7fba13e64740 1 --2- 172.21.15.40:0/17955758 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55fbdd2e64f0 0x55fbdd340930 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.461+0000 7fba13e64740 1 --2- 172.21.15.40:0/17955758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbdd317910 0x55fbdd3403f0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.461+0000 7fba13e64740 1 --2- 172.21.15.40:0/17955758 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbdd333fb0 0x55fbdd3314f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.461+0000 7fba13e64740 1 --2- 172.21.15.40:0/17955758 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55fbdd316710 0x55fbdd3173d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.461+0000 7fba13e64740 1 -- 172.21.15.40:0/17955758 >> 172.21.15.40:0/17955758 conn(0x55fbdd2ec3f0 msgr2=0x55fbdd34b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fb9eb400640 0 client.9471 ms_handle_reset on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fba13e64740 1 -- 172.21.15.40:0/17955758 shutdown_connections 2024-04-26T23:09:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fba13e64740 1 -- 172.21.15.40:0/17955758 wait complete. 2024-04-26T23:09:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fba13e64740 20 client.9471 trim_cache size 0 max 16384 2024-04-26T23:09:46.466 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:46.466 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-26T23:09:46.466 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:09:46.466 INFO:teuthology.orchestra.run.smithi040.stdout: "a9827531-70a9-4d6f-b12a-2d71fa886d03": { 2024-04-26T23:09:46.467 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:09:46.467 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "a9827531-70a9-4d6f-b12a-2d71fa886d03", 2024-04-26T23:09:46.467 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:09:46.467 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:46.467 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:46.467 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:46.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fba0b200640 1 -- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb9ec06c8f0 msgr2=0x7fb9ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fba0b200640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb9ec06c8f0 0x7fb9ec06edb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fba0c0cc6e0 tx=0x7fb9f8008040 comp rx=0 tx=0).stop 2024-04-26T23:09:46.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fba0b200640 1 -- 172.21.15.40:0/523610700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9f0004d40 msgr2=0x7fba0c14de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fba0b200640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9f0004d40 0x7fba0c14de60 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc0108e0 tx=0x7fb9fc010da0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fba0b200640 1 -- 172.21.15.40:0/523610700 shutdown_connections 2024-04-26T23:09:46.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fba0b200640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb9ec06c8f0 0x7fb9ec06edb0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fba0b200640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba0c0a0a00 0x7fba0c0a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fba0b200640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fba0c09eab0 0x7fba0c09ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fba0b200640 1 --2- 172.21.15.40:0/523610700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9f0004d40 0x7fba0c14de60 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.462+0000 7fba0b200640 1 -- 172.21.15.40:0/523610700 >> 172.21.15.40:0/523610700 conn(0x7fba0c0d4dd0 msgr2=0x7fba0c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.463+0000 7fba0b200640 1 -- 172.21.15.40:0/523610700 shutdown_connections 2024-04-26T23:09:46.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.463+0000 7fba0b200640 1 -- 172.21.15.40:0/523610700 wait complete. 2024-04-26T23:09:46.478 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:09:46.478 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:09:46.479 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.7:2 scrub status 2024-04-26T23:09:46.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.677+0000 7f9a49600640 1 Processor -- start 2024-04-26T23:09:46.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.677+0000 7f9a49600640 1 -- start start 2024-04-26T23:09:46.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.679+0000 7f9a49600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9a44071e60 0x7f9a4406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.679+0000 7f9a49600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9a4406fe20 0x7f9a44072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.679+0000 7f9a49600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a4415d7c0 con 0x7f9a44072780 2024-04-26T23:09:46.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.679+0000 7f9a49600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f9a4415daa0 con 0x7f9a44071e60 2024-04-26T23:09:46.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.679+0000 7f9a49600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9a4415dd80 con 0x7f9a4406fe20 2024-04-26T23:09:46.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.679+0000 7f9a42a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9a44071e60 0x7f9a4406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.679+0000 7f9a43400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9a44072780 0x7f9a440cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35752/0 (socket says 172.21.15.40:35752) 2024-04-26T23:09:46.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.679+0000 7f9a43400640 1 -- 172.21.15.40:0/1913291916 learned_addr learned my addr 172.21.15.40:0/1913291916 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.680+0000 7f9a3be00640 1 --2- 172.21.15.40:0/1913291916 >> v2:172.21.15.103:3300/0 conn(0x7f9a4406fe20 0x7f9a44072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.680+0000 7f9a42a00640 1 -- 172.21.15.40:0/1913291916 >> v2:172.21.15.103:3300/0 conn(0x7f9a4406fe20 msgr2=0x7f9a44072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.680+0000 7f9a42a00640 1 --2- 172.21.15.40:0/1913291916 >> v2:172.21.15.103:3300/0 conn(0x7f9a4406fe20 0x7f9a44072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.680+0000 7f9a42000640 1 -- 172.21.15.40:0/1913291916 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1339362210 0 0) 0x7f9a4415d7c0 con 0x7f9a44072780 2024-04-26T23:09:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.680+0000 7f9a42a00640 1 -- 172.21.15.40:0/1913291916 >> v1:172.21.15.40:6789/0 conn(0x7f9a44072780 legacy=0x7f9a440cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.680+0000 7f9a42a00640 1 -- 172.21.15.40:0/1913291916 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a4415e060 con 0x7f9a44071e60 2024-04-26T23:09:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.681+0000 7f9a42a00640 1 --2- 172.21.15.40:0/1913291916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a44071e60 0x7f9a4406f8e0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f9a440b3820 tx=0x7f9a3402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2590908ed5c35e85 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.682+0000 7f9a42000640 1 -- 172.21.15.40:0/1913291916 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a34037070 con 0x7f9a44071e60 2024-04-26T23:09:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.682+0000 7f9a42000640 1 -- 172.21.15.40:0/1913291916 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a34032440 con 0x7f9a44071e60 2024-04-26T23:09:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.682+0000 7f9a42000640 1 -- 172.21.15.40:0/1913291916 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a34032810 con 0x7f9a44071e60 2024-04-26T23:09:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.682+0000 7f9a49600640 1 -- 172.21.15.40:0/1913291916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a44071e60 msgr2=0x7f9a4406f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.682+0000 7f9a49600640 1 --2- 172.21.15.40:0/1913291916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a44071e60 0x7f9a4406f8e0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f9a440b3820 tx=0x7f9a3402a1a0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.682+0000 7f9a49600640 1 -- 172.21.15.40:0/1913291916 shutdown_connections 2024-04-26T23:09:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.682+0000 7f9a49600640 1 --2- 172.21.15.40:0/1913291916 >> v2:172.21.15.103:3300/0 conn(0x7f9a4406fe20 0x7f9a44072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.682+0000 7f9a49600640 1 --2- 172.21.15.40:0/1913291916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a44071e60 0x7f9a4406f8e0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.682+0000 7f9a49600640 1 -- 172.21.15.40:0/1913291916 >> 172.21.15.40:0/1913291916 conn(0x7f9a440d0be0 msgr2=0x7f9a440d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.682+0000 7f9a49600640 1 -- 172.21.15.40:0/1913291916 shutdown_connections 2024-04-26T23:09:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.683+0000 7f9a49600640 1 -- 172.21.15.40:0/1913291916 wait complete. 2024-04-26T23:09:46.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.684+0000 7f9a49600640 1 Processor -- start 2024-04-26T23:09:46.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.684+0000 7f9a49600640 1 -- start start 2024-04-26T23:09:46.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.685+0000 7f9a49600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a4406fe20 0x7f9a440b6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.685+0000 7f9a49600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a44071e60 0x7f9a440b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.685+0000 7f9a49600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a44072780 0x7f9a440c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.685+0000 7f9a49600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4415d3f0 con 0x7f9a4406fe20 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.685+0000 7f9a49600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a44072bd0 con 0x7f9a44072780 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.685+0000 7f9a49600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9a4415d570 con 0x7f9a44071e60 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.686+0000 7f9a42a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a4406fe20 0x7f9a440b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.686+0000 7f9a42a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a4406fe20 0x7f9a440b6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39698/0 (socket says 172.21.15.40:39698) 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.686+0000 7f9a42a00640 1 -- 172.21.15.40:0/2857600461 learned_addr learned my addr 172.21.15.40:0/2857600461 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.686+0000 7f9a43400640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a44072780 0x7f9a440c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.686+0000 7f9a3be00640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a44071e60 0x7f9a440b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.686+0000 7f9a42a00640 1 -- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a44071e60 msgr2=0x7f9a440b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.686+0000 7f9a42a00640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a44071e60 0x7f9a440b3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.686+0000 7f9a42a00640 1 -- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a44072780 msgr2=0x7f9a440c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.686+0000 7f9a42a00640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a44072780 0x7f9a440c8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.686+0000 7f9a42a00640 1 -- 172.21.15.40:0/2857600461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a440c7d50 con 0x7f9a4406fe20 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.686+0000 7f9a43400640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a44072780 0x7f9a440c8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:46.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.686+0000 7f9a42a00640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a4406fe20 0x7f9a440b6070 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f9a34000c00 tx=0x7f9a340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.687+0000 7f9a3aa00640 1 -- 172.21.15.40:0/2857600461 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a34037040 con 0x7f9a4406fe20 2024-04-26T23:09:46.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.687+0000 7f9a3aa00640 1 -- 172.21.15.40:0/2857600461 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a34042070 con 0x7f9a4406fe20 2024-04-26T23:09:46.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.687+0000 7f9a3aa00640 1 -- 172.21.15.40:0/2857600461 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a34030bd0 con 0x7f9a4406fe20 2024-04-26T23:09:46.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.687+0000 7f9a49600640 1 -- 172.21.15.40:0/2857600461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a440c8030 con 0x7f9a4406fe20 2024-04-26T23:09:46.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.687+0000 7f9a49600640 1 -- 172.21.15.40:0/2857600461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a440c85f0 con 0x7f9a4406fe20 2024-04-26T23:09:46.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.689+0000 7f9a3aa00640 1 -- 172.21.15.40:0/2857600461 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a34047460 con 0x7f9a4406fe20 2024-04-26T23:09:46.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.690+0000 7f9a3aa00640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a2006c8f0 0x7f9a2006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.690+0000 7f9a3aa00640 1 -- 172.21.15.40:0/2857600461 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f9a340ac890 con 0x7f9a4406fe20 2024-04-26T23:09:46.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.691+0000 7f9a4c422740 1 Processor -- start 2024-04-26T23:09:46.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.691+0000 7f9a3be00640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a2006c8f0 0x7f9a2006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.691+0000 7f9a4c422740 1 -- start start 2024-04-26T23:09:46.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.692+0000 7f9a4c422740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 0x55c986cb6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.692+0000 7f9a4c422740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cd3de0 0x55c986cd4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.692+0000 7f9a4c422740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986c8c220 0x55c986cd24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.692+0000 7f9a4c422740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c986b9c030 con 0x55c986cb6540 2024-04-26T23:09:46.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.692+0000 7f9a4c422740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c986bb77f0 con 0x55c986c8c220 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.692+0000 7f9a4c422740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c986b727e0 con 0x55c986cd3de0 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.692+0000 7f9a3be00640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a2006c8f0 0x7f9a2006edb0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9a441589e0 tx=0x7f9a2c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.692+0000 7f9a42a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 0x55c986cb6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.692+0000 7f9a42a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 0x55c986cb6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39706/0 (socket says 172.21.15.40:39706) 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.692+0000 7f9a42a00640 1 -- 172.21.15.40:0/4227343432 learned_addr learned my addr 172.21.15.40:0/4227343432 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.693+0000 7f9a43400640 1 --2- 172.21.15.40:0/4227343432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986c8c220 0x55c986cd24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.693+0000 7f9a3be00640 1 --2- 172.21.15.40:0/4227343432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cd3de0 0x55c986cd4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.693+0000 7f9a42a00640 1 -- 172.21.15.40:0/4227343432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cd3de0 msgr2=0x55c986cd4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.693+0000 7f9a42a00640 1 --2- 172.21.15.40:0/4227343432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cd3de0 0x55c986cd4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.693+0000 7f9a42a00640 1 -- 172.21.15.40:0/4227343432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986c8c220 msgr2=0x55c986cd24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.693+0000 7f9a42a00640 1 --2- 172.21.15.40:0/4227343432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986c8c220 0x55c986cd24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.693+0000 7f9a42a00640 1 -- 172.21.15.40:0/4227343432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c986b7e050 con 0x55c986cb6540 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.693+0000 7f9a3be00640 1 --2- 172.21.15.40:0/4227343432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cd3de0 0x55c986cd4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:46.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.693+0000 7f9a42a00640 1 --2- 172.21.15.40:0/4227343432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 0x55c986cb6920 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f9a340091d0 tx=0x7f9a34079d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e8022b8e2d3b32 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.693+0000 7f9a33000640 1 -- 172.21.15.40:0/4227343432 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a340a9070 con 0x55c986cb6540 2024-04-26T23:09:46.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.693+0000 7f9a33000640 1 -- 172.21.15.40:0/4227343432 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a3407e7d0 con 0x55c986cb6540 2024-04-26T23:09:46.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.693+0000 7f9a33000640 1 -- 172.21.15.40:0/4227343432 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a34083000 con 0x55c986cb6540 2024-04-26T23:09:46.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.694+0000 7f9a4c422740 1 -- 172.21.15.40:0/4227343432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 msgr2=0x55c986cb6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.694+0000 7f9a4c422740 1 --2- 172.21.15.40:0/4227343432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 0x55c986cb6920 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f9a340091d0 tx=0x7f9a34079d80 comp rx=0 tx=0).stop 2024-04-26T23:09:46.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.694+0000 7f9a4c422740 1 -- 172.21.15.40:0/4227343432 shutdown_connections 2024-04-26T23:09:46.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.694+0000 7f9a4c422740 1 --2- 172.21.15.40:0/4227343432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986c8c220 0x55c986cd24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.694+0000 7f9a4c422740 1 --2- 172.21.15.40:0/4227343432 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cd3de0 0x55c986cd4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.694+0000 7f9a4c422740 1 --2- 172.21.15.40:0/4227343432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 0x55c986cb6920 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.694+0000 7f9a4c422740 1 -- 172.21.15.40:0/4227343432 >> 172.21.15.40:0/4227343432 conn(0x55c986c88b50 msgr2=0x55c986cd1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.694+0000 7f9a4c422740 1 -- 172.21.15.40:0/4227343432 shutdown_connections 2024-04-26T23:09:46.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.694+0000 7f9a4c422740 1 -- 172.21.15.40:0/4227343432 wait complete. 2024-04-26T23:09:46.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.695+0000 7f9a4c422740 1 -- start start 2024-04-26T23:09:46.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.695+0000 7f9a4c422740 1 Processor -- start 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.696+0000 7f9a4c422740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 0x55c986ce1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.696+0000 7f9a4c422740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cd3de0 0x55c986ca43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.696+0000 7f9a4c422740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986ce20f0 0x55c986ca48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.696+0000 7f9a4c422740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c986bf4200 con 0x55c986cb6540 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.696+0000 7f9a4c422740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c986beeee0 con 0x55c986ce20f0 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.696+0000 7f9a4c422740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c986bfbe70 con 0x55c986cd3de0 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.696+0000 7f9a4c422740 10 client.0 _ll_register_callbacks cb 0x55c986be5270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.696+0000 7f9a42a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 0x55c986ce1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.696+0000 7f9a42a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 0x55c986ce1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39722/0 (socket says 172.21.15.40:39722) 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.696+0000 7f9a42a00640 1 -- 172.21.15.40:0/1880620982 learned_addr learned my addr 172.21.15.40:0/1880620982 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.697+0000 7f9a42a00640 1 -- 172.21.15.40:0/1880620982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cd3de0 msgr2=0x55c986ca43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.697+0000 7f9a42a00640 1 --2- 172.21.15.40:0/1880620982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cd3de0 0x55c986ca43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.697+0000 7f9a42a00640 1 -- 172.21.15.40:0/1880620982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986ce20f0 msgr2=0x55c986ca48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.697+0000 7f9a42a00640 1 --2- 172.21.15.40:0/1880620982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986ce20f0 0x55c986ca48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.697+0000 7f9a42a00640 1 -- 172.21.15.40:0/1880620982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9869cb7d0 con 0x55c986cb6540 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.697+0000 7f9a42a00640 1 --2- 172.21.15.40:0/1880620982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 0x55c986ce1980 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f9a3407a3b0 tx=0x7f9a34006ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ef17263df7c4d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.697+0000 7f9a27400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.697+0000 7f9a27400640 1 -- 172.21.15.40:0/1880620982 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a340a9070 con 0x55c986cb6540 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.697+0000 7f9a27400640 1 -- 172.21.15.40:0/1880620982 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f9a34076a90 con 0x55c986cb6540 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.697+0000 7f9a4c422740 1 -- 172.21.15.40:0/1880620982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a3407e7d0 con 0x55c986cb6540 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.697+0000 7f9a27400640 1 -- 172.21.15.40:0/1880620982 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f9a34045360 con 0x55c986cb6540 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.697+0000 7f9a4c422740 10 client.9486 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.697+0000 7f9a4c422740 1 -- 172.21.15.40:0/1880620982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55c9869013f0 con 0x55c986cb6540 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.698+0000 7f9a27400640 1 -- 172.21.15.40:0/1880620982 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a3407e7d0 con 0x55c986cb6540 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.698+0000 7f9a27400640 1 -- 172.21.15.40:0/1880620982 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7f9a3403a030 con 0x55c986cb6540 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.698+0000 7f9a4c422740 10 client.9486 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.698+0000 7f9a4c422740 10 client.9486 resolve_mds: resolved 7:2 to role '7:2' aka daemon mds.d 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.698+0000 7f9a4c422740 20 client.9486 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:46.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.698+0000 7f9a4c422740 1 --2- 172.21.15.40:0/1880620982 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55c986cb4930 0x55c986cb4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.698+0000 7f9a4c422740 4 client.9486 mds_command: new command op to 6598 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:46.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.698+0000 7f9a4c422740 1 -- 172.21.15.40:0/1880620982 --> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c986bfd7e0 con 0x55c986cb4930 2024-04-26T23:09:46.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.699+0000 7f9a3be00640 1 --2- 172.21.15.40:0/1880620982 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55c986cb4930 0x55c986cb4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.700+0000 7f9a3be00640 1 --2- 172.21.15.40:0/1880620982 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55c986cb4930 0x55c986cb4d70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8c037a1b407f77f7 server_cookie=9a74775bbec0fe92 in_seq=0 out_seq=0 2024-04-26T23:09:46.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.700+0000 7f9a27400640 10 client.9486 ms_handle_connect on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:46.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a27400640 1 -- 172.21.15.40:0/1880620982 <== mds.2 v2:172.21.15.103:6835/2022020922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c986bfd7e0 con 0x55c986cb4930 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a27400640 10 client.9486 handle_command_reply: tid=0 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a4c422740 1 client.9486 shutdown 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a4c422740 1 -- 172.21.15.40:0/1880620982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 msgr2=0x55c986ce1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a4c422740 1 --2- 172.21.15.40:0/1880620982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 0x55c986ce1980 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f9a3407a3b0 tx=0x7f9a34006ed0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a4c422740 1 -- 172.21.15.40:0/1880620982 shutdown_connections 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a4c422740 1 --2- 172.21.15.40:0/1880620982 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55c986cb4930 0x55c986cb4d70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a4c422740 1 --2- 172.21.15.40:0/1880620982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986ce20f0 0x55c986ca48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a4c422740 1 --2- 172.21.15.40:0/1880620982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cd3de0 0x55c986ca43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a27400640 0 client.9486 ms_handle_reset on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a4c422740 1 --2- 172.21.15.40:0/1880620982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cb6540 0x55c986ce1980 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a4c422740 1 -- 172.21.15.40:0/1880620982 >> 172.21.15.40:0/1880620982 conn(0x55c986c8c220 msgr2=0x55c986c857b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a4c422740 1 -- 172.21.15.40:0/1880620982 shutdown_connections 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a4c422740 1 -- 172.21.15.40:0/1880620982 wait complete. 2024-04-26T23:09:46.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.702+0000 7f9a4c422740 20 client.9486 trim_cache size 0 max 16384 2024-04-26T23:09:46.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.722+0000 7f9a4c422740 1 Processor -- start 2024-04-26T23:09:46.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.722+0000 7f9a4c422740 1 -- start start 2024-04-26T23:09:46.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.722+0000 7f9a4c422740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cb6540 0x55c986c87890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.722+0000 7f9a4c422740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cd3de0 0x55c986c8c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.723+0000 7f9a4c422740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986ce20f0 0x55c986ccfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.723+0000 7f9a4c422740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9a44201d80 con 0x55c986cd3de0 2024-04-26T23:09:46.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.723+0000 7f9a4c422740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a44200f20 con 0x55c986ce20f0 2024-04-26T23:09:46.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.723+0000 7f9a4c422740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9a44201650 con 0x55c986cb6540 2024-04-26T23:09:46.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.723+0000 7f9a3be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cd3de0 0x55c986c8c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.723+0000 7f9a3be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cd3de0 0x55c986c8c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39732/0 (socket says 172.21.15.40:39732) 2024-04-26T23:09:46.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.723+0000 7f9a3be00640 1 -- 172.21.15.40:0/3643785737 learned_addr learned my addr 172.21.15.40:0/3643785737 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.723+0000 7f9a3be00640 1 -- 172.21.15.40:0/3643785737 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cb6540 msgr2=0x55c986c87890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.723+0000 7f9a3be00640 1 --2- 172.21.15.40:0/3643785737 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cb6540 0x55c986c87890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.723+0000 7f9a3be00640 1 -- 172.21.15.40:0/3643785737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986ce20f0 msgr2=0x55c986ccfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.723+0000 7f9a3be00640 1 --2- 172.21.15.40:0/3643785737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986ce20f0 0x55c986ccfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.723+0000 7f9a3be00640 1 -- 172.21.15.40:0/3643785737 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9868c10f0 con 0x55c986cd3de0 2024-04-26T23:09:46.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.724+0000 7f9a3be00640 1 --2- 172.21.15.40:0/3643785737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cd3de0 0x55c986c8c7f0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c006fd0 tx=0x7f9a2c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa14a3003962d9a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.724+0000 7f9a32600640 1 -- 172.21.15.40:0/3643785737 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c00f040 con 0x55c986cd3de0 2024-04-26T23:09:46.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.724+0000 7f9a32600640 1 -- 172.21.15.40:0/3643785737 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a2c02b070 con 0x55c986cd3de0 2024-04-26T23:09:46.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.724+0000 7f9a32600640 1 -- 172.21.15.40:0/3643785737 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c004270 con 0x55c986cd3de0 2024-04-26T23:09:46.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.724+0000 7f9a4c422740 1 -- 172.21.15.40:0/3643785737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cd3de0 msgr2=0x55c986c8c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.724+0000 7f9a4c422740 1 --2- 172.21.15.40:0/3643785737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cd3de0 0x55c986c8c7f0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c006fd0 tx=0x7f9a2c011000 comp rx=0 tx=0).stop 2024-04-26T23:09:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.725+0000 7f9a4c422740 1 -- 172.21.15.40:0/3643785737 shutdown_connections 2024-04-26T23:09:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.725+0000 7f9a4c422740 1 --2- 172.21.15.40:0/3643785737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986ce20f0 0x55c986ccfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.725+0000 7f9a4c422740 1 --2- 172.21.15.40:0/3643785737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986cd3de0 0x55c986c8c7f0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.725+0000 7f9a4c422740 1 --2- 172.21.15.40:0/3643785737 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cb6540 0x55c986c87890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.725+0000 7f9a4c422740 1 -- 172.21.15.40:0/3643785737 >> 172.21.15.40:0/3643785737 conn(0x55c986c8c220 msgr2=0x55c986cd1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.725+0000 7f9a4c422740 1 -- 172.21.15.40:0/3643785737 shutdown_connections 2024-04-26T23:09:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.725+0000 7f9a4c422740 1 -- 172.21.15.40:0/3643785737 wait complete. 2024-04-26T23:09:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.725+0000 7f9a4c422740 1 -- start start 2024-04-26T23:09:46.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.726+0000 7f9a4c422740 1 Processor -- start 2024-04-26T23:09:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a4c422740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cb6540 0x55c986cd2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a4c422740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986cd3de0 0x55c986cd22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a4c422740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986ce20f0 0x55c986cd2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a4c422740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c986bfcfd0 con 0x55c986ce20f0 2024-04-26T23:09:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a4c422740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a441f8c60 con 0x55c986cd3de0 2024-04-26T23:09:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a4c422740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55c986c89790 con 0x55c986cb6540 2024-04-26T23:09:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a4c422740 10 client.0 _ll_register_callbacks cb 0x7f9a340a7070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a43400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986ce20f0 0x55c986cd2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a43400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986ce20f0 0x55c986cd2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39736/0 (socket says 172.21.15.40:39736) 2024-04-26T23:09:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a43400640 1 -- 172.21.15.40:0/3706269989 learned_addr learned my addr 172.21.15.40:0/3706269989 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a43400640 1 -- 172.21.15.40:0/3706269989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cb6540 msgr2=0x55c986cd2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a43400640 1 --2- 172.21.15.40:0/3706269989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cb6540 0x55c986cd2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a43400640 1 -- 172.21.15.40:0/3706269989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986cd3de0 msgr2=0x55c986cd22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a43400640 1 --2- 172.21.15.40:0/3706269989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986cd3de0 0x55c986cd22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.727+0000 7f9a43400640 1 -- 172.21.15.40:0/3706269989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c986bb7a30 con 0x55c986ce20f0 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.728+0000 7f9a43400640 1 --2- 172.21.15.40:0/3706269989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986ce20f0 0x55c986cd2810 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c007b30 tx=0x7f9a3c007b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5d67751ef384d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.728+0000 7f9a27400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.728+0000 7f9a27400640 1 -- 172.21.15.40:0/3706269989 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c0133f0 con 0x55c986ce20f0 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.728+0000 7f9a27400640 1 -- 172.21.15.40:0/3706269989 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f9a3c013590 con 0x55c986ce20f0 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.728+0000 7f9a4c422740 1 -- 172.21.15.40:0/3706269989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a2c02b070 con 0x55c986ce20f0 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.728+0000 7f9a27400640 1 -- 172.21.15.40:0/3706269989 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f9a3c020050 con 0x55c986ce20f0 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.728+0000 7f9a4c422740 10 client.9492 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.728+0000 7f9a4c422740 1 -- 172.21.15.40:0/3706269989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55c986cc7630 con 0x55c986ce20f0 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.728+0000 7f9a27400640 1 -- 172.21.15.40:0/3706269989 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a2c02b070 con 0x55c986ce20f0 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.728+0000 7f9a27400640 1 -- 172.21.15.40:0/3706269989 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7f9a3c023020 con 0x55c986ce20f0 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.728+0000 7f9a4c422740 10 client.9492 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.728+0000 7f9a4c422740 10 client.9492 resolve_mds: resolved 7:2 to role '7:2' aka daemon mds.d 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.728+0000 7f9a4c422740 20 client.9492 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:46.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.729+0000 7f9a4c422740 1 --2- 172.21.15.40:0/3706269989 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55c986d2cbd0 0x55c986cf3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.729+0000 7f9a4c422740 4 client.9492 mds_command: new command op to 6598 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:09:46.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.729+0000 7f9a4c422740 1 -- 172.21.15.40:0/3706269989 --> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c986bf7650 con 0x55c986d2cbd0 2024-04-26T23:09:46.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.729+0000 7f9a42a00640 1 --2- 172.21.15.40:0/3706269989 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55c986d2cbd0 0x55c986cf3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.730+0000 7f9a42a00640 1 --2- 172.21.15.40:0/3706269989 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55c986d2cbd0 0x55c986cf3b70 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fac4257cdbe5a07e server_cookie=86e6d68c9e864008 in_seq=0 out_seq=0 2024-04-26T23:09:46.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.730+0000 7f9a27400640 10 client.9492 ms_handle_connect on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:46.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a27400640 1 -- 172.21.15.40:0/3706269989 <== mds.2 v2:172.21.15.103:6835/2022020922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55c986bf7650 con 0x55c986d2cbd0 2024-04-26T23:09:46.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a27400640 10 client.9492 handle_command_reply: tid=0 2024-04-26T23:09:46.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a4c422740 1 client.9492 shutdown 2024-04-26T23:09:46.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a4c422740 1 -- 172.21.15.40:0/3706269989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986ce20f0 msgr2=0x55c986cd2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a4c422740 1 --2- 172.21.15.40:0/3706269989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986ce20f0 0x55c986cd2810 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c007b30 tx=0x7f9a3c007b60 comp rx=0 tx=0).stop 2024-04-26T23:09:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a4c422740 1 -- 172.21.15.40:0/3706269989 shutdown_connections 2024-04-26T23:09:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a4c422740 1 --2- 172.21.15.40:0/3706269989 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55c986d2cbd0 0x55c986cf3b70 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a4c422740 1 --2- 172.21.15.40:0/3706269989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c986ce20f0 0x55c986cd2810 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a4c422740 1 --2- 172.21.15.40:0/3706269989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c986cd3de0 0x55c986cd22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a4c422740 1 --2- 172.21.15.40:0/3706269989 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55c986cb6540 0x55c986cd2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a27400640 0 client.9492 ms_handle_reset on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a4c422740 1 -- 172.21.15.40:0/3706269989 >> 172.21.15.40:0/3706269989 conn(0x55c986c8c220 msgr2=0x55c986c875b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a4c422740 1 -- 172.21.15.40:0/3706269989 shutdown_connections 2024-04-26T23:09:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a4c422740 1 -- 172.21.15.40:0/3706269989 wait complete. 2024-04-26T23:09:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.731+0000 7f9a4c422740 20 client.9492 trim_cache size 0 max 16384 2024-04-26T23:09:46.734 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:46.734 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-26T23:09:46.734 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:09:46.734 INFO:teuthology.orchestra.run.smithi040.stdout: "a9827531-70a9-4d6f-b12a-2d71fa886d03": { 2024-04-26T23:09:46.734 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:09:46.734 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "a9827531-70a9-4d6f-b12a-2d71fa886d03", 2024-04-26T23:09:46.734 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:09:46.734 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:46.734 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:46.734 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:46.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.732+0000 7f9a49600640 1 -- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a2006c8f0 msgr2=0x7f9a2006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.732+0000 7f9a49600640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a2006c8f0 0x7f9a2006edb0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9a441589e0 tx=0x7f9a2c008040 comp rx=0 tx=0).stop 2024-04-26T23:09:46.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.732+0000 7f9a49600640 1 -- 172.21.15.40:0/2857600461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a4406fe20 msgr2=0x7f9a440b6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.732+0000 7f9a49600640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a4406fe20 0x7f9a440b6070 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f9a34000c00 tx=0x7f9a340046c0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.732+0000 7f9a49600640 1 -- 172.21.15.40:0/2857600461 shutdown_connections 2024-04-26T23:09:46.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.732+0000 7f9a49600640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9a2006c8f0 0x7f9a2006edb0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.732+0000 7f9a49600640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a44072780 0x7f9a440c8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.732+0000 7f9a49600640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9a44071e60 0x7f9a440b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.732+0000 7f9a49600640 1 --2- 172.21.15.40:0/2857600461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a4406fe20 0x7f9a440b6070 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.732+0000 7f9a49600640 1 -- 172.21.15.40:0/2857600461 >> 172.21.15.40:0/2857600461 conn(0x7f9a440d0be0 msgr2=0x7f9a440b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.732+0000 7f9a49600640 1 -- 172.21.15.40:0/2857600461 shutdown_connections 2024-04-26T23:09:46.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.732+0000 7f9a49600640 1 -- 172.21.15.40:0/2857600461 wait complete. 2024-04-26T23:09:46.749 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:09:46.750 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:09:46.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-26T23:09:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.955+0000 7f8787200640 1 Processor -- start 2024-04-26T23:09:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.955+0000 7f8787200640 1 -- start start 2024-04-26T23:09:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.957+0000 7f8787200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8780072ce0 0x7f878015f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.957+0000 7f8787200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87801604e0 con 0x7f8780073600 2024-04-26T23:09:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.957+0000 7f8787200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8780072200 con 0x7f878015fcb0 2024-04-26T23:09:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.957+0000 7f8787200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8780072850 con 0x7f8780072ce0 2024-04-26T23:09:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.957+0000 7f8785e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8780073600 0x7f87800739e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35756/0 (socket says 172.21.15.40:35756) 2024-04-26T23:09:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.957+0000 7f8785e00640 1 -- 172.21.15.40:0/1060613788 learned_addr learned my addr 172.21.15.40:0/1060613788 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:46.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.957+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845128023 0 0) 0x7f87801604e0 con 0x7f8780073600 2024-04-26T23:09:46.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8785400640 1 --2- 172.21.15.40:0/1060613788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8780072ce0 0x7f878015f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8768003520 con 0x7f8780073600 2024-04-26T23:09:46.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1064449168 0 0) 0x7f8768003520 con 0x7f8780073600 2024-04-26T23:09:46.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8780072ce0 msgr2=0x7f878015f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 --2- 172.21.15.40:0/1060613788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8780072ce0 0x7f878015f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 >> v1:172.21.15.103:6789/0 conn(0x7f878015fcb0 legacy=0x7f87800ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87800724e0 con 0x7f8780073600 2024-04-26T23:09:46.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f8774001670 con 0x7f8780073600 2024-04-26T23:09:46.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8785400640 1 --2- 172.21.15.40:0/1060613788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8780072ce0 0x7f878015f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:09:46.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 >> v1:172.21.15.40:6789/0 conn(0x7f8780073600 legacy=0x7f87800739e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 --2- 172.21.15.40:0/1060613788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87680039e0 0x7f8768003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 --2- 172.21.15.40:0/1060613788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8768004800 0x7f8768004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 --2- 172.21.15.40:0/1060613788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8780072ce0 0x7f8768005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8780072850 con 0x7f8768004800 2024-04-26T23:09:46.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f876800bb50 con 0x7f87680039e0 2024-04-26T23:09:46.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f876800bd00 con 0x7f8780072ce0 2024-04-26T23:09:46.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.958+0000 7f8785400640 1 --2- 172.21.15.40:0/1060613788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8768004800 0x7f8768004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.959+0000 7f8785e00640 1 --2- 172.21.15.40:0/1060613788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87680039e0 0x7f8768003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.959+0000 7f8786800640 1 --2- 172.21.15.40:0/1060613788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8780072ce0 0x7f8768005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.959+0000 7f8785400640 1 -- 172.21.15.40:0/1060613788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8780072ce0 msgr2=0x7f8768005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.959+0000 7f8785400640 1 --2- 172.21.15.40:0/1060613788 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8780072ce0 0x7f8768005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.959+0000 7f8785400640 1 -- 172.21.15.40:0/1060613788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87680039e0 msgr2=0x7f8768003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.959+0000 7f8785400640 1 --2- 172.21.15.40:0/1060613788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87680039e0 0x7f8768003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.959+0000 7f8785400640 1 -- 172.21.15.40:0/1060613788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8768003520 con 0x7f8768004800 2024-04-26T23:09:46.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.960+0000 7f8785400640 1 --2- 172.21.15.40:0/1060613788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8768004800 0x7f8768004c80 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f87700098a0 tx=0x7f8768005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae9dcfcbbaf5f3a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.960+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877000f040 con 0x7f8768004800 2024-04-26T23:09:46.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.960+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f877002ad00 con 0x7f8768004800 2024-04-26T23:09:46.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.960+0000 7f8784a00640 1 -- 172.21.15.40:0/1060613788 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8770004250 con 0x7f8768004800 2024-04-26T23:09:46.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.961+0000 7f8787200640 1 -- 172.21.15.40:0/1060613788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8768004800 msgr2=0x7f8768004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.961+0000 7f8787200640 1 --2- 172.21.15.40:0/1060613788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8768004800 0x7f8768004c80 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f87700098a0 tx=0x7f8768005830 comp rx=0 tx=0).stop 2024-04-26T23:09:46.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.961+0000 7f8786800640 1 -- 172.21.15.40:0/1060613788 reap_dead start 2024-04-26T23:09:46.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.961+0000 7f8787200640 1 -- 172.21.15.40:0/1060613788 shutdown_connections 2024-04-26T23:09:46.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.961+0000 7f8787200640 1 -- 172.21.15.40:0/1060613788 >> 172.21.15.40:0/1060613788 conn(0x7f87800b2720 msgr2=0x7f87800b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.961+0000 7f8787200640 1 -- 172.21.15.40:0/1060613788 shutdown_connections 2024-04-26T23:09:46.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.961+0000 7f8787200640 1 -- 172.21.15.40:0/1060613788 wait complete. 2024-04-26T23:09:46.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.963+0000 7f8787200640 1 Processor -- start 2024-04-26T23:09:46.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.963+0000 7f8787200640 1 -- start start 2024-04-26T23:09:46.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.964+0000 7f8787200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8768004800 0x7f87800c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.964+0000 7f8787200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87800cf6a0 0x7f87801f9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.964+0000 7f8787200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878015b9d0 0x7f87800c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.964+0000 7f8787200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f87801602a0 con 0x7f87800cf6a0 2024-04-26T23:09:46.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.964+0000 7f8787200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8780160120 con 0x7f8768004800 2024-04-26T23:09:46.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.964+0000 7f8787200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8780071e20 con 0x7f878015b9d0 2024-04-26T23:09:46.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.964+0000 7f8785400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87800cf6a0 0x7f87801f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.964+0000 7f8785400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87800cf6a0 0x7f87801f9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39746/0 (socket says 172.21.15.40:39746) 2024-04-26T23:09:46.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.964+0000 7f8785400640 1 -- 172.21.15.40:0/538222422 learned_addr learned my addr 172.21.15.40:0/538222422 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.965+0000 7f8785e00640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8768004800 0x7f87800c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.965+0000 7f8785400640 1 -- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878015b9d0 msgr2=0x7f87800c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.965+0000 7f8785400640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878015b9d0 0x7f87800c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.965+0000 7f8785400640 1 -- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8768004800 msgr2=0x7f87800c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.965+0000 7f8785400640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8768004800 0x7f87800c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.965+0000 7f8785400640 1 -- 172.21.15.40:0/538222422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878015c520 con 0x7f87800cf6a0 2024-04-26T23:09:46.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.965+0000 7f8785e00640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8768004800 0x7f87800c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:46.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.965+0000 7f8785400640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87800cf6a0 0x7f87801f9220 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f87700099c0 tx=0x7f8770004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.965+0000 7f877a000640 1 -- 172.21.15.40:0/538222422 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8770041070 con 0x7f87800cf6a0 2024-04-26T23:09:46.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.965+0000 7f877a000640 1 -- 172.21.15.40:0/538222422 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f877000f000 con 0x7f87800cf6a0 2024-04-26T23:09:46.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.965+0000 7f8787200640 1 -- 172.21.15.40:0/538222422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87801f9760 con 0x7f87800cf6a0 2024-04-26T23:09:46.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.965+0000 7f877a000640 1 -- 172.21.15.40:0/538222422 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8770003bd0 con 0x7f87800cf6a0 2024-04-26T23:09:46.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.965+0000 7f8787200640 1 -- 172.21.15.40:0/538222422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87800c5b30 con 0x7f87800cf6a0 2024-04-26T23:09:46.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.969+0000 7f877a000640 1 -- 172.21.15.40:0/538222422 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8770044070 con 0x7f87800cf6a0 2024-04-26T23:09:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.969+0000 7f877a000640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f876406c8f0 0x7f876406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.970+0000 7f877a000640 1 -- 172.21.15.40:0/538222422 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f87700ac550 con 0x7f87800cf6a0 2024-04-26T23:09:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.970+0000 7f878b622740 1 Processor -- start 2024-04-26T23:09:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.970+0000 7f878b622740 1 -- start start 2024-04-26T23:09:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.970+0000 7f8785e00640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f876406c8f0 0x7f876406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.971+0000 7f878b622740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a93710 0x55b310aaf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.971+0000 7f878b622740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 0x55b310a93af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.971+0000 7f878b622740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310aa48d0 0x55b310aa4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.971+0000 7f878b622740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b310979030 con 0x55b310a93710 2024-04-26T23:09:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.971+0000 7f878b622740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b3109947f0 con 0x55b310ab1550 2024-04-26T23:09:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.971+0000 7f878b622740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b31094f7e0 con 0x55b310aa48d0 2024-04-26T23:09:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.971+0000 7f8785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a93710 0x55b310aaf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.971+0000 7f8785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a93710 0x55b310aaf0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39750/0 (socket says 172.21.15.40:39750) 2024-04-26T23:09:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.971+0000 7f8785e00640 1 -- 172.21.15.40:0/4046910426 learned_addr learned my addr 172.21.15.40:0/4046910426 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.972+0000 7f8785e00640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f876406c8f0 0x7f876406edb0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8774001c00 tx=0x7f8774000f90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.972+0000 7f8786800640 1 --2- 172.21.15.40:0/4046910426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310aa48d0 0x55b310aa4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.972+0000 7f8785e00640 1 -- 172.21.15.40:0/4046910426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310aa48d0 msgr2=0x55b310aa4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.972+0000 7f8785e00640 1 --2- 172.21.15.40:0/4046910426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310aa48d0 0x55b310aa4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.972+0000 7f8785400640 1 --2- 172.21.15.40:0/4046910426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 0x55b310a93af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.972+0000 7f8785e00640 1 -- 172.21.15.40:0/4046910426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 msgr2=0x55b310a93af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.972+0000 7f8785e00640 1 --2- 172.21.15.40:0/4046910426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 0x55b310a93af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.972+0000 7f8785e00640 1 -- 172.21.15.40:0/4046910426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b31095b050 con 0x55b310a93710 2024-04-26T23:09:46.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.972+0000 7f8786800640 1 --2- 172.21.15.40:0/4046910426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310aa48d0 0x55b310aa4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:46.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.972+0000 7f8785e00640 1 --2- 172.21.15.40:0/4046910426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a93710 0x55b310aaf0f0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f877400fb70 tx=0x7f8774012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c610a4bbc750c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.972+0000 7f876e600640 1 -- 172.21.15.40:0/4046910426 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774026070 con 0x55b310a93710 2024-04-26T23:09:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.972+0000 7f876e600640 1 -- 172.21.15.40:0/4046910426 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8774019070 con 0x55b310a93710 2024-04-26T23:09:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.972+0000 7f876e600640 1 -- 172.21.15.40:0/4046910426 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877401b070 con 0x55b310a93710 2024-04-26T23:09:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.973+0000 7f878b622740 1 -- 172.21.15.40:0/4046910426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a93710 msgr2=0x55b310aaf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.973+0000 7f878b622740 1 --2- 172.21.15.40:0/4046910426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a93710 0x55b310aaf0f0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f877400fb70 tx=0x7f8774012000 comp rx=0 tx=0).stop 2024-04-26T23:09:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.973+0000 7f878b622740 1 -- 172.21.15.40:0/4046910426 shutdown_connections 2024-04-26T23:09:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.973+0000 7f878b622740 1 --2- 172.21.15.40:0/4046910426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310aa48d0 0x55b310aa4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.973+0000 7f878b622740 1 --2- 172.21.15.40:0/4046910426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 0x55b310a93af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.973+0000 7f878b622740 1 --2- 172.21.15.40:0/4046910426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a93710 0x55b310aaf0f0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.973+0000 7f878b622740 1 -- 172.21.15.40:0/4046910426 >> 172.21.15.40:0/4046910426 conn(0x55b310a65d20 msgr2=0x55b310aae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.973+0000 7f878b622740 1 -- 172.21.15.40:0/4046910426 shutdown_connections 2024-04-26T23:09:46.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.973+0000 7f878b622740 1 -- 172.21.15.40:0/4046910426 wait complete. 2024-04-26T23:09:46.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.974+0000 7f878b622740 1 -- start start 2024-04-26T23:09:46.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.974+0000 7f878b622740 1 Processor -- start 2024-04-26T23:09:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.975+0000 7f878b622740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 0x55b310a96bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.975+0000 7f878b622740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 0x55b310a679a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.975+0000 7f878b622740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310a90460 0x55b310a67ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.975+0000 7f878b622740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b3109d1200 con 0x55b310aa48d0 2024-04-26T23:09:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.975+0000 7f878b622740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b3109cbee0 con 0x55b310ab1550 2024-04-26T23:09:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.975+0000 7f878b622740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b3109d8e70 con 0x55b310a90460 2024-04-26T23:09:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.975+0000 7f878b622740 10 client.0 _ll_register_callbacks cb 0x55b3109c2270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.975+0000 7f8785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 0x55b310a96bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.976+0000 7f8785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 0x55b310a96bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39752/0 (socket says 172.21.15.40:39752) 2024-04-26T23:09:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.976+0000 7f8785e00640 1 -- 172.21.15.40:0/92195596 learned_addr learned my addr 172.21.15.40:0/92195596 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.976+0000 7f8785e00640 1 -- 172.21.15.40:0/92195596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310a90460 msgr2=0x55b310a67ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.976+0000 7f8786800640 1 --2- 172.21.15.40:0/92195596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310a90460 0x55b310a67ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.976+0000 7f8785400640 1 --2- 172.21.15.40:0/92195596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 0x55b310a679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.976+0000 7f8785e00640 1 --2- 172.21.15.40:0/92195596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310a90460 0x55b310a67ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.976+0000 7f8785e00640 1 -- 172.21.15.40:0/92195596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 msgr2=0x55b310a679a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.976+0000 7f8785e00640 1 --2- 172.21.15.40:0/92195596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 0x55b310a679a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.976+0000 7f8785e00640 1 -- 172.21.15.40:0/92195596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3107a87d0 con 0x55b310aa48d0 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.976+0000 7f8785e00640 1 --2- 172.21.15.40:0/92195596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 0x55b310a96bc0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f8774001eb0 tx=0x7f8774001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ee831c440a886c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.976+0000 7f8762a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.977+0000 7f8762a00640 1 -- 172.21.15.40:0/92195596 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774030060 con 0x55b310aa48d0 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.977+0000 7f8762a00640 1 -- 172.21.15.40:0/92195596 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f8774020950 con 0x55b310aa48d0 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.977+0000 7f878b622740 1 -- 172.21.15.40:0/92195596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8774019070 con 0x55b310aa48d0 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.977+0000 7f8762a00640 1 -- 172.21.15.40:0/92195596 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f877401b070 con 0x55b310aa48d0 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.977+0000 7f878b622740 10 client.9504 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.977+0000 7f878b622740 1 -- 172.21.15.40:0/92195596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55b3106de3f0 con 0x55b310aa48d0 2024-04-26T23:09:46.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.977+0000 7f8762a00640 1 -- 172.21.15.40:0/92195596 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8774019070 con 0x55b310aa48d0 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.977+0000 7f8762a00640 1 -- 172.21.15.40:0/92195596 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7f8774013050 con 0x55b310aa48d0 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.977+0000 7f878b622740 10 client.9504 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.977+0000 7f878b622740 10 client.9504 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.978+0000 7f878b622740 20 client.9504 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.978+0000 7f878b622740 1 --2- 172.21.15.40:0/92195596 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b310aae4d0 0x55b310aae940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.978+0000 7f878b622740 4 client.9504 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.978+0000 7f878b622740 1 -- 172.21.15.40:0/92195596 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3109da7e0 con 0x55b310aae4d0 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.978+0000 7f8785400640 1 --2- 172.21.15.40:0/92195596 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b310aae4d0 0x55b310aae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.978+0000 7f8785400640 1 --2- 172.21.15.40:0/92195596 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b310aae4d0 0x55b310aae940 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d0a76f5bfe7ab25 server_cookie=ca9f1255f860af30 in_seq=0 out_seq=0 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.978+0000 7f8762a00640 10 client.9504 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.979+0000 7f8762a00640 1 -- 172.21.15.40:0/92195596 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b3109da7e0 con 0x55b310aae4d0 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.979+0000 7f8762a00640 10 client.9504 handle_command_reply: tid=0 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.979+0000 7f878b622740 1 client.9504 shutdown 2024-04-26T23:09:46.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.979+0000 7f878b622740 1 -- 172.21.15.40:0/92195596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 msgr2=0x55b310a96bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.979+0000 7f878b622740 1 --2- 172.21.15.40:0/92195596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 0x55b310a96bc0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f8774001eb0 tx=0x7f8774001970 comp rx=0 tx=0).stop 2024-04-26T23:09:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.979+0000 7f878b622740 1 -- 172.21.15.40:0/92195596 shutdown_connections 2024-04-26T23:09:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.979+0000 7f878b622740 1 --2- 172.21.15.40:0/92195596 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310a90460 0x55b310a67ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.979+0000 7f878b622740 1 --2- 172.21.15.40:0/92195596 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b310aae4d0 0x55b310aae940 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.979+0000 7f878b622740 1 --2- 172.21.15.40:0/92195596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 0x55b310a679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.979+0000 7f878b622740 1 --2- 172.21.15.40:0/92195596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 0x55b310a96bc0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.979+0000 7f8762a00640 0 client.9504 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.979+0000 7f878b622740 1 -- 172.21.15.40:0/92195596 >> 172.21.15.40:0/92195596 conn(0x55b310a93710 msgr2=0x55b310a631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.980+0000 7f878b622740 1 -- 172.21.15.40:0/92195596 shutdown_connections 2024-04-26T23:09:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.980+0000 7f878b622740 1 -- 172.21.15.40:0/92195596 wait complete. 2024-04-26T23:09:46.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.980+0000 7f878b622740 20 client.9504 trim_cache size 0 max 16384 2024-04-26T23:09:46.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.999+0000 7f878b622740 1 Processor -- start 2024-04-26T23:09:46.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:46.999+0000 7f878b622740 1 -- start start 2024-04-26T23:09:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.000+0000 7f878b622740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310a90460 0x55b310ac3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.000+0000 7f878b622740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 0x55b310a95830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.000+0000 7f878b622740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 0x55b310aa3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.000+0000 7f878b622740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f87800ad250 con 0x55b310aa48d0 2024-04-26T23:09:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.000+0000 7f878b622740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87800ac3f0 con 0x55b310ab1550 2024-04-26T23:09:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.000+0000 7f878b622740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f87800acb20 con 0x55b310a90460 2024-04-26T23:09:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.000+0000 7f8785400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 0x55b310a95830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.000+0000 7f8785400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 0x55b310a95830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39758/0 (socket says 172.21.15.40:39758) 2024-04-26T23:09:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.000+0000 7f8785400640 1 -- 172.21.15.40:0/1687675227 learned_addr learned my addr 172.21.15.40:0/1687675227 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.000+0000 7f8785e00640 1 --2- 172.21.15.40:0/1687675227 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310a90460 0x55b310ac3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.001+0000 7f8786800640 1 --2- 172.21.15.40:0/1687675227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 0x55b310aa3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.001+0000 7f8785400640 1 -- 172.21.15.40:0/1687675227 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310a90460 msgr2=0x55b310ac3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.001+0000 7f8785400640 1 --2- 172.21.15.40:0/1687675227 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310a90460 0x55b310ac3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.001+0000 7f8785400640 1 -- 172.21.15.40:0/1687675227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 msgr2=0x55b310aa3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.001+0000 7f8785400640 1 --2- 172.21.15.40:0/1687675227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 0x55b310aa3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.001+0000 7f8785400640 1 -- 172.21.15.40:0/1687675227 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b31069e0f0 con 0x55b310aa48d0 2024-04-26T23:09:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.001+0000 7f8785e00640 1 --2- 172.21.15.40:0/1687675227 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310a90460 0x55b310ac3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:09:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.001+0000 7f8785400640 1 --2- 172.21.15.40:0/1687675227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 0x55b310a95830 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f8770076420 tx=0x7f87700aa720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd89bba244e3419 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.001+0000 7f876dc00640 1 -- 172.21.15.40:0/1687675227 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877004b040 con 0x55b310aa48d0 2024-04-26T23:09:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.001+0000 7f876dc00640 1 -- 172.21.15.40:0/1687675227 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87700aa960 con 0x55b310aa48d0 2024-04-26T23:09:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.001+0000 7f876dc00640 1 -- 172.21.15.40:0/1687675227 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8770003d70 con 0x55b310aa48d0 2024-04-26T23:09:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.002+0000 7f878b622740 1 -- 172.21.15.40:0/1687675227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 msgr2=0x55b310a95830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.002+0000 7f878b622740 1 --2- 172.21.15.40:0/1687675227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 0x55b310a95830 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f8770076420 tx=0x7f87700aa720 comp rx=0 tx=0).stop 2024-04-26T23:09:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.002+0000 7f878b622740 1 -- 172.21.15.40:0/1687675227 shutdown_connections 2024-04-26T23:09:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.002+0000 7f878b622740 1 --2- 172.21.15.40:0/1687675227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310ab1550 0x55b310aa3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.002+0000 7f878b622740 1 --2- 172.21.15.40:0/1687675227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310aa48d0 0x55b310a95830 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.002+0000 7f878b622740 1 --2- 172.21.15.40:0/1687675227 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310a90460 0x55b310ac3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.002+0000 7f878b622740 1 -- 172.21.15.40:0/1687675227 >> 172.21.15.40:0/1687675227 conn(0x55b310a93710 msgr2=0x55b310a92d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.002+0000 7f878b622740 1 -- 172.21.15.40:0/1687675227 shutdown_connections 2024-04-26T23:09:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.002+0000 7f878b622740 1 -- 172.21.15.40:0/1687675227 wait complete. 2024-04-26T23:09:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.003+0000 7f878b622740 1 -- start start 2024-04-26T23:09:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.003+0000 7f878b622740 1 Processor -- start 2024-04-26T23:09:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.004+0000 7f878b622740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a90460 0x55b310aacb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.004+0000 7f878b622740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310aa48d0 0x55b310aad0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.004+0000 7f878b622740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310ab1550 0x55b310acf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.004+0000 7f878b622740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b3109d9fd0 con 0x55b310a90460 2024-04-26T23:09:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.004+0000 7f878b622740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87801f60c0 con 0x55b310aa48d0 2024-04-26T23:09:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.004+0000 7f878b622740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55b310ab1930 con 0x55b310ab1550 2024-04-26T23:09:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.004+0000 7f878b622740 10 client.0 _ll_register_callbacks cb 0x7f8774024070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.004+0000 7f8785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a90460 0x55b310aacb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.004+0000 7f8785e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a90460 0x55b310aacb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39764/0 (socket says 172.21.15.40:39764) 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.004+0000 7f8785e00640 1 -- 172.21.15.40:0/218737582 learned_addr learned my addr 172.21.15.40:0/218737582 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.005+0000 7f8786800640 1 --2- 172.21.15.40:0/218737582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310ab1550 0x55b310acf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.005+0000 7f8785e00640 1 -- 172.21.15.40:0/218737582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310ab1550 msgr2=0x55b310acf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.005+0000 7f8785400640 1 --2- 172.21.15.40:0/218737582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310aa48d0 0x55b310aad0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.005+0000 7f8785e00640 1 --2- 172.21.15.40:0/218737582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310ab1550 0x55b310acf8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.005+0000 7f8785e00640 1 -- 172.21.15.40:0/218737582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310aa48d0 msgr2=0x55b310aad0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.005+0000 7f8785e00640 1 --2- 172.21.15.40:0/218737582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310aa48d0 0x55b310aad0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.005+0000 7f8785e00640 1 -- 172.21.15.40:0/218737582 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b310994a30 con 0x55b310a90460 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.005+0000 7f8785e00640 1 --2- 172.21.15.40:0/218737582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a90460 0x55b310aacb70 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f8774014800 tx=0x7f8774015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52918805d6adbf07 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.005+0000 7f8762a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.005+0000 7f8762a00640 1 -- 172.21.15.40:0/218737582 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877403f060 con 0x55b310a90460 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.005+0000 7f8762a00640 1 -- 172.21.15.40:0/218737582 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7f8774014df0 con 0x55b310a90460 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.005+0000 7f878b622740 1 -- 172.21.15.40:0/218737582 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87700aa960 con 0x55b310a90460 2024-04-26T23:09:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.006+0000 7f8762a00640 1 -- 172.21.15.40:0/218737582 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f8774045050 con 0x55b310a90460 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.006+0000 7f878b622740 10 client.9510 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.006+0000 7f878b622740 1 -- 172.21.15.40:0/218737582 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55b310b2fa90 con 0x55b310a90460 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.006+0000 7f8762a00640 1 -- 172.21.15.40:0/218737582 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87700aa960 con 0x55b310a90460 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.006+0000 7f8762a00640 1 -- 172.21.15.40:0/218737582 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7f8774048020 con 0x55b310a90460 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.006+0000 7f878b622740 10 client.9510 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.006+0000 7f878b622740 10 client.9510 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.006+0000 7f878b622740 20 client.9510 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.006+0000 7f878b622740 1 --2- 172.21.15.40:0/218737582 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b310ab2750 0x55b310acfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.006+0000 7f878b622740 4 client.9510 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.006+0000 7f878b622740 1 -- 172.21.15.40:0/218737582 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b3109d4650 con 0x55b310ab2750 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.006+0000 7f8785400640 1 --2- 172.21.15.40:0/218737582 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b310ab2750 0x55b310acfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f8785400640 1 --2- 172.21.15.40:0/218737582 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b310ab2750 0x55b310acfe00 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=537ed13907ba01b8 server_cookie=f952c1ea6b2adf2a in_seq=0 out_seq=0 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f8762a00640 10 client.9510 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f8762a00640 1 -- 172.21.15.40:0/218737582 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55b3109d4650 con 0x55b310ab2750 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f8762a00640 10 client.9510 handle_command_reply: tid=0 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f878b622740 1 client.9510 shutdown 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f878b622740 1 -- 172.21.15.40:0/218737582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a90460 msgr2=0x55b310aacb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f878b622740 1 --2- 172.21.15.40:0/218737582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a90460 0x55b310aacb70 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f8774014800 tx=0x7f8774015000 comp rx=0 tx=0).stop 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f878b622740 1 -- 172.21.15.40:0/218737582 shutdown_connections 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f878b622740 1 --2- 172.21.15.40:0/218737582 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55b310ab1550 0x55b310acf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f878b622740 1 --2- 172.21.15.40:0/218737582 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x55b310ab2750 0x55b310acfe00 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f878b622740 1 --2- 172.21.15.40:0/218737582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b310aa48d0 0x55b310aad0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f878b622740 1 --2- 172.21.15.40:0/218737582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b310a90460 0x55b310aacb70 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f878b622740 1 -- 172.21.15.40:0/218737582 >> 172.21.15.40:0/218737582 conn(0x55b310a93710 msgr2=0x55b310ac2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.007+0000 7f8762a00640 0 client.9510 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.008+0000 7f878b622740 1 -- 172.21.15.40:0/218737582 shutdown_connections 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.008+0000 7f878b622740 1 -- 172.21.15.40:0/218737582 wait complete. 2024-04-26T23:09:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.008+0000 7f878b622740 20 client.9510 trim_cache size 0 max 16384 2024-04-26T23:09:47.012 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-26T23:09:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:09:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: "a9827531-70a9-4d6f-b12a-2d71fa886d03": { 2024-04-26T23:09:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:09:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "a9827531-70a9-4d6f-b12a-2d71fa886d03", 2024-04-26T23:09:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:09:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:47.012 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.008+0000 7f8787200640 1 -- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f876406c8f0 msgr2=0x7f876406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.008+0000 7f8787200640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f876406c8f0 0x7f876406edb0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8774001c00 tx=0x7f8774000f90 comp rx=0 tx=0).stop 2024-04-26T23:09:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.008+0000 7f8787200640 1 -- 172.21.15.40:0/538222422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87800cf6a0 msgr2=0x7f87801f9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.008+0000 7f8787200640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87800cf6a0 0x7f87801f9220 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f87700099c0 tx=0x7f8770004730 comp rx=0 tx=0).stop 2024-04-26T23:09:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.008+0000 7f8787200640 1 -- 172.21.15.40:0/538222422 shutdown_connections 2024-04-26T23:09:47.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.008+0000 7f8787200640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f876406c8f0 0x7f876406edb0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.008+0000 7f8787200640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f878015b9d0 0x7f87800c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.008+0000 7f8787200640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87800cf6a0 0x7f87801f9220 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.008+0000 7f8787200640 1 --2- 172.21.15.40:0/538222422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8768004800 0x7f87800c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.008+0000 7f8787200640 1 -- 172.21.15.40:0/538222422 >> 172.21.15.40:0/538222422 conn(0x7f87800b2720 msgr2=0x7f878015ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.009+0000 7f8787200640 1 -- 172.21.15.40:0/538222422 shutdown_connections 2024-04-26T23:09:47.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.009+0000 7f8787200640 1 -- 172.21.15.40:0/538222422 wait complete. 2024-04-26T23:09:47.025 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:09:47.025 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:09:47.025 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.7:1 scrub status 2024-04-26T23:09:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.232+0000 7fc545000640 1 Processor -- start 2024-04-26T23:09:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.232+0000 7fc545000640 1 -- start start 2024-04-26T23:09:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.234+0000 7fc545000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc54015d110 0x7fc5400d36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.234+0000 7fc545000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc5400d18c0 0x7fc5400b2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.234+0000 7fc545000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5400cfd40 con 0x7fc5400d09b0 2024-04-26T23:09:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.234+0000 7fc545000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc5400ceb20 con 0x7fc5400d18c0 2024-04-26T23:09:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.234+0000 7fc545000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc54015d4f0 con 0x7fc54015d110 2024-04-26T23:09:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.234+0000 7fc53ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc5400d09b0 0x7fc5400d11b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35764/0 (socket says 172.21.15.40:35764) 2024-04-26T23:09:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.234+0000 7fc53ea00640 1 -- 172.21.15.40:0/3495281654 learned_addr learned my addr 172.21.15.40:0/3495281654 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.234+0000 7fc53fe00640 1 --2- 172.21.15.40:0/3495281654 >> v2:172.21.15.40:3300/0 conn(0x7fc5400d18c0 0x7fc5400b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.234+0000 7fc53f400640 1 --2- 172.21.15.40:0/3495281654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc54015d110 0x7fc5400d36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.235+0000 7fc53e000640 1 -- 172.21.15.40:0/3495281654 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3950284036 0 0) 0x7fc5400cfd40 con 0x7fc5400d09b0 2024-04-26T23:09:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.235+0000 7fc53e000640 1 -- 172.21.15.40:0/3495281654 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc524003520 con 0x7fc5400d09b0 2024-04-26T23:09:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.235+0000 7fc53fe00640 1 -- 172.21.15.40:0/3495281654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc54015d110 msgr2=0x7fc5400d36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.235+0000 7fc53fe00640 1 --2- 172.21.15.40:0/3495281654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc54015d110 0x7fc5400d36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.235+0000 7fc53fe00640 1 -- 172.21.15.40:0/3495281654 >> v1:172.21.15.40:6789/0 conn(0x7fc5400d09b0 legacy=0x7fc5400d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.235+0000 7fc53fe00640 1 -- 172.21.15.40:0/3495281654 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5400cffe0 con 0x7fc5400d18c0 2024-04-26T23:09:47.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.236+0000 7fc53fe00640 1 --2- 172.21.15.40:0/3495281654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5400d18c0 0x7fc5400b2760 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fc538009d90 tx=0x7fc53802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bce5f370461ff41 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.237+0000 7fc53e000640 1 -- 172.21.15.40:0/3495281654 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc538031600 con 0x7fc5400d18c0 2024-04-26T23:09:47.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.237+0000 7fc53e000640 1 -- 172.21.15.40:0/3495281654 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5380317a0 con 0x7fc5400d18c0 2024-04-26T23:09:47.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.237+0000 7fc53e000640 1 -- 172.21.15.40:0/3495281654 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc538031b50 con 0x7fc5400d18c0 2024-04-26T23:09:47.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.237+0000 7fc545000640 1 -- 172.21.15.40:0/3495281654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5400d18c0 msgr2=0x7fc5400b2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.237+0000 7fc545000640 1 --2- 172.21.15.40:0/3495281654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5400d18c0 0x7fc5400b2760 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fc538009d90 tx=0x7fc53802a5d0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.237+0000 7fc545000640 1 -- 172.21.15.40:0/3495281654 shutdown_connections 2024-04-26T23:09:47.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.237+0000 7fc545000640 1 --2- 172.21.15.40:0/3495281654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5400d18c0 0x7fc5400b2760 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.237+0000 7fc545000640 1 --2- 172.21.15.40:0/3495281654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc54015d110 0x7fc5400d36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.237+0000 7fc545000640 1 -- 172.21.15.40:0/3495281654 >> 172.21.15.40:0/3495281654 conn(0x7fc5400d2bd0 msgr2=0x7fc5400d2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.237+0000 7fc545000640 1 -- 172.21.15.40:0/3495281654 shutdown_connections 2024-04-26T23:09:47.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.237+0000 7fc545000640 1 -- 172.21.15.40:0/3495281654 wait complete. 2024-04-26T23:09:47.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.239+0000 7fc545000640 1 Processor -- start 2024-04-26T23:09:47.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.239+0000 7fc545000640 1 -- start start 2024-04-26T23:09:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.240+0000 7fc545000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5400d09b0 0x7fc54009f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.240+0000 7fc545000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5400d18c0 0x7fc54009cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.240+0000 7fc545000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc54015d110 0x7fc5400cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.240+0000 7fc545000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc5400b2ca0 con 0x7fc5400d18c0 2024-04-26T23:09:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.240+0000 7fc545000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc5400b2240 con 0x7fc5400d09b0 2024-04-26T23:09:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.240+0000 7fc545000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc5400b2e20 con 0x7fc54015d110 2024-04-26T23:09:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.240+0000 7fc53ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5400d18c0 0x7fc54009cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.241+0000 7fc53ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5400d18c0 0x7fc54009cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39772/0 (socket says 172.21.15.40:39772) 2024-04-26T23:09:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.241+0000 7fc53ea00640 1 -- 172.21.15.40:0/2193440029 learned_addr learned my addr 172.21.15.40:0/2193440029 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.241+0000 7fc53f400640 1 --2- 172.21.15.40:0/2193440029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5400d09b0 0x7fc54009f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.241+0000 7fc53ea00640 1 -- 172.21.15.40:0/2193440029 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc54015d110 msgr2=0x7fc5400cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.241+0000 7fc53ea00640 1 --2- 172.21.15.40:0/2193440029 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc54015d110 0x7fc5400cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.241+0000 7fc53ea00640 1 -- 172.21.15.40:0/2193440029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5400d09b0 msgr2=0x7fc54009f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.241+0000 7fc53ea00640 1 --2- 172.21.15.40:0/2193440029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5400d09b0 0x7fc54009f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.241+0000 7fc53ea00640 1 -- 172.21.15.40:0/2193440029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5400cae20 con 0x7fc5400d18c0 2024-04-26T23:09:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.241+0000 7fc53ea00640 1 --2- 172.21.15.40:0/2193440029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5400d18c0 0x7fc54009cc70 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00c430 tx=0x7fc52c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.241+0000 7fc537400640 1 -- 172.21.15.40:0/2193440029 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c017070 con 0x7fc5400d18c0 2024-04-26T23:09:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.241+0000 7fc537400640 1 -- 172.21.15.40:0/2193440029 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc52c005ce0 con 0x7fc5400d18c0 2024-04-26T23:09:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.242+0000 7fc537400640 1 -- 172.21.15.40:0/2193440029 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0072f0 con 0x7fc5400d18c0 2024-04-26T23:09:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.242+0000 7fc545000640 1 -- 172.21.15.40:0/2193440029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5400cb160 con 0x7fc5400d18c0 2024-04-26T23:09:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.242+0000 7fc545000640 1 -- 172.21.15.40:0/2193440029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5400cb9a0 con 0x7fc5400d18c0 2024-04-26T23:09:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.244+0000 7fc537400640 1 -- 172.21.15.40:0/2193440029 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc52c01a070 con 0x7fc5400d18c0 2024-04-26T23:09:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.245+0000 7fc537400640 1 --2- 172.21.15.40:0/2193440029 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc52006c8f0 0x7fc52006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.245+0000 7fc537400640 1 -- 172.21.15.40:0/2193440029 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fc52c08b130 con 0x7fc5400d18c0 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.246+0000 7fc549598740 1 Processor -- start 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.246+0000 7fc53f400640 1 --2- 172.21.15.40:0/2193440029 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc52006c8f0 0x7fc52006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.246+0000 7fc549598740 1 -- start start 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc549598740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d91b710 0x55a02d91baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc549598740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 0x55a02d939500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc549598740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d8f13f0 0x55a02d937670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc549598740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a02d801030 con 0x55a02d938fb0 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc549598740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a02d81c7f0 con 0x55a02d91b710 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc549598740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a02d7d77e0 con 0x55a02d8f13f0 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc53f400640 1 --2- 172.21.15.40:0/2193440029 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc52006c8f0 0x7fc52006edb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc530006fd0 tx=0x7fc530008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc53ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 0x55a02d939500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc53ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 0x55a02d939500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39786/0 (socket says 172.21.15.40:39786) 2024-04-26T23:09:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc53ea00640 1 -- 172.21.15.40:0/2853000875 learned_addr learned my addr 172.21.15.40:0/2853000875 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc53f400640 1 --2- 172.21.15.40:0/2853000875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d91b710 0x55a02d91baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc53ea00640 1 -- 172.21.15.40:0/2853000875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d8f13f0 msgr2=0x55a02d937670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc53ea00640 1 --2- 172.21.15.40:0/2853000875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d8f13f0 0x55a02d937670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc53ea00640 1 -- 172.21.15.40:0/2853000875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d91b710 msgr2=0x55a02d91baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc53ea00640 1 --2- 172.21.15.40:0/2853000875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d91b710 0x55a02d91baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc53ea00640 1 -- 172.21.15.40:0/2853000875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a02d7e3050 con 0x55a02d938fb0 2024-04-26T23:09:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.247+0000 7fc53f400640 1 --2- 172.21.15.40:0/2853000875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d91b710 0x55a02d91baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.248+0000 7fc53ea00640 1 --2- 172.21.15.40:0/2853000875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 0x55a02d939500 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fc52c061410 tx=0x7fc52c058e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67dbddd388454f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.248+0000 7fc52be00640 1 -- 172.21.15.40:0/2853000875 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c071020 con 0x55a02d938fb0 2024-04-26T23:09:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.248+0000 7fc52be00640 1 -- 172.21.15.40:0/2853000875 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc52c057060 con 0x55a02d938fb0 2024-04-26T23:09:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.248+0000 7fc52be00640 1 -- 172.21.15.40:0/2853000875 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0553d0 con 0x55a02d938fb0 2024-04-26T23:09:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.248+0000 7fc549598740 1 -- 172.21.15.40:0/2853000875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 msgr2=0x55a02d939500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.248+0000 7fc549598740 1 --2- 172.21.15.40:0/2853000875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 0x55a02d939500 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fc52c061410 tx=0x7fc52c058e20 comp rx=0 tx=0).stop 2024-04-26T23:09:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.249+0000 7fc549598740 1 -- 172.21.15.40:0/2853000875 shutdown_connections 2024-04-26T23:09:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.249+0000 7fc549598740 1 --2- 172.21.15.40:0/2853000875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d8f13f0 0x55a02d937670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.249+0000 7fc549598740 1 --2- 172.21.15.40:0/2853000875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 0x55a02d939500 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.249+0000 7fc549598740 1 --2- 172.21.15.40:0/2853000875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d91b710 0x55a02d91baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.249+0000 7fc549598740 1 -- 172.21.15.40:0/2853000875 >> 172.21.15.40:0/2853000875 conn(0x55a02d8edd20 msgr2=0x55a02d936520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.249+0000 7fc549598740 1 -- 172.21.15.40:0/2853000875 shutdown_connections 2024-04-26T23:09:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.249+0000 7fc549598740 1 -- 172.21.15.40:0/2853000875 wait complete. 2024-04-26T23:09:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.249+0000 7fc549598740 1 -- start start 2024-04-26T23:09:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.250+0000 7fc549598740 1 Processor -- start 2024-04-26T23:09:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc549598740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 0x55a02d917720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc549598740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d938fb0 0x55a02d939aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc549598740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 0x55a02d939fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc549598740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a02d859200 con 0x55a02d91b710 2024-04-26T23:09:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc549598740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a02d853ee0 con 0x55a02d938fb0 2024-04-26T23:09:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc549598740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a02d860e70 con 0x55a02d917c60 2024-04-26T23:09:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc549598740 10 client.0 _ll_register_callbacks cb 0x55a02d84a270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc53f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 0x55a02d917720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc53f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 0x55a02d917720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39802/0 (socket says 172.21.15.40:39802) 2024-04-26T23:09:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc53f400640 1 -- 172.21.15.40:0/2860972509 learned_addr learned my addr 172.21.15.40:0/2860972509 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc53f400640 1 -- 172.21.15.40:0/2860972509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 msgr2=0x55a02d939fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc53f400640 1 --2- 172.21.15.40:0/2860972509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 0x55a02d939fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc53f400640 1 -- 172.21.15.40:0/2860972509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d938fb0 msgr2=0x55a02d939aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc53f400640 1 --2- 172.21.15.40:0/2860972509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d938fb0 0x55a02d939aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.251+0000 7fc53f400640 1 -- 172.21.15.40:0/2860972509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a02d6307d0 con 0x55a02d91b710 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.252+0000 7fc53f400640 1 --2- 172.21.15.40:0/2860972509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 0x55a02d917720 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fc53000eda0 tx=0x7fc530004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6480b558eb794f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.252+0000 7fc528c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.252+0000 7fc528c00640 1 -- 172.21.15.40:0/2860972509 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc53000f040 con 0x55a02d91b710 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.252+0000 7fc549598740 1 -- 172.21.15.40:0/2860972509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc52c057060 con 0x55a02d91b710 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.252+0000 7fc528c00640 1 -- 172.21.15.40:0/2860972509 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fc530015d50 con 0x55a02d91b710 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.252+0000 7fc528c00640 1 -- 172.21.15.40:0/2860972509 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7fc53002d050 con 0x55a02d91b710 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.252+0000 7fc549598740 10 client.9525 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.252+0000 7fc549598740 1 -- 172.21.15.40:0/2860972509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55a02d5663f0 con 0x55a02d91b710 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.253+0000 7fc528c00640 1 -- 172.21.15.40:0/2860972509 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc52c057060 con 0x55a02d91b710 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.253+0000 7fc528c00640 1 -- 172.21.15.40:0/2860972509 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7fc530030020 con 0x55a02d91b710 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.253+0000 7fc549598740 10 client.9525 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.253+0000 7fc549598740 10 client.9525 resolve_mds: resolved 7:1 to role '7:1' aka daemon mds.c 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.253+0000 7fc549598740 20 client.9525 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.253+0000 7fc549598740 1 --2- 172.21.15.40:0/2860972509 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55a02d950640 0x55a02d952a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.253+0000 7fc549598740 4 client.9525 mds_command: new command op to 8949 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.253+0000 7fc549598740 1 -- 172.21.15.40:0/2860972509 --> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a02d8627e0 con 0x55a02d950640 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.253+0000 7fc53ea00640 1 --2- 172.21.15.40:0/2860972509 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55a02d950640 0x55a02d952a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.254+0000 7fc53ea00640 1 --2- 172.21.15.40:0/2860972509 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55a02d950640 0x55a02d952a90 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4eb2eec8cfeb386d server_cookie=68f5e7ab2ca32c56 in_seq=0 out_seq=0 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.254+0000 7fc528c00640 10 client.9525 ms_handle_connect on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.254+0000 7fc528c00640 1 -- 172.21.15.40:0/2860972509 <== mds.1 v2:172.21.15.40:6832/3250599834 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a02d8627e0 con 0x55a02d950640 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.254+0000 7fc528c00640 10 client.9525 handle_command_reply: tid=0 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.254+0000 7fc549598740 1 client.9525 shutdown 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.254+0000 7fc549598740 1 -- 172.21.15.40:0/2860972509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 msgr2=0x55a02d917720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.254+0000 7fc549598740 1 --2- 172.21.15.40:0/2860972509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 0x55a02d917720 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fc53000eda0 tx=0x7fc530004730 comp rx=0 tx=0).stop 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.255+0000 7fc549598740 1 -- 172.21.15.40:0/2860972509 shutdown_connections 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.255+0000 7fc549598740 1 --2- 172.21.15.40:0/2860972509 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55a02d950640 0x55a02d952a90 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.255+0000 7fc549598740 1 --2- 172.21.15.40:0/2860972509 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 0x55a02d939fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.255+0000 7fc549598740 1 --2- 172.21.15.40:0/2860972509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d938fb0 0x55a02d939aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.255+0000 7fc549598740 1 --2- 172.21.15.40:0/2860972509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 0x55a02d917720 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.255+0000 7fc549598740 1 -- 172.21.15.40:0/2860972509 >> 172.21.15.40:0/2860972509 conn(0x55a02d8f13f0 msgr2=0x55a02d8ea960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.255+0000 7fc528c00640 0 client.9525 ms_handle_reset on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.255+0000 7fc549598740 1 -- 172.21.15.40:0/2860972509 shutdown_connections 2024-04-26T23:09:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.255+0000 7fc549598740 1 -- 172.21.15.40:0/2860972509 wait complete. 2024-04-26T23:09:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.255+0000 7fc549598740 20 client.9525 trim_cache size 0 max 16384 2024-04-26T23:09:47.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.274+0000 7fc549598740 1 Processor -- start 2024-04-26T23:09:47.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.274+0000 7fc549598740 1 -- start start 2024-04-26T23:09:47.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.275+0000 7fc549598740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 0x55a02d9456d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.275+0000 7fc549598740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d91b710 0x55a02d9368a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.275+0000 7fc549598740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 0x55a02d930310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.275+0000 7fc549598740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc540201c80 con 0x55a02d938fb0 2024-04-26T23:09:47.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.275+0000 7fc549598740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc540200e20 con 0x55a02d91b710 2024-04-26T23:09:47.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.275+0000 7fc549598740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc540201550 con 0x55a02d917c60 2024-04-26T23:09:47.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.276+0000 7fc53fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 0x55a02d930310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.276+0000 7fc53fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 0x55a02d930310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39818/0 (socket says 172.21.15.40:39818) 2024-04-26T23:09:47.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.276+0000 7fc53fe00640 1 -- 172.21.15.40:0/926974874 learned_addr learned my addr 172.21.15.40:0/926974874 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.276+0000 7fc53f400640 1 --2- 172.21.15.40:0/926974874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 0x55a02d9456d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.276+0000 7fc53ea00640 1 --2- 172.21.15.40:0/926974874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d91b710 0x55a02d9368a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.276+0000 7fc53fe00640 1 -- 172.21.15.40:0/926974874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 msgr2=0x55a02d9456d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.276+0000 7fc53fe00640 1 --2- 172.21.15.40:0/926974874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 0x55a02d9456d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.276+0000 7fc53fe00640 1 -- 172.21.15.40:0/926974874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d91b710 msgr2=0x55a02d9368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.276+0000 7fc53fe00640 1 --2- 172.21.15.40:0/926974874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d91b710 0x55a02d9368a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.276+0000 7fc53fe00640 1 -- 172.21.15.40:0/926974874 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a02d5260f0 con 0x55a02d938fb0 2024-04-26T23:09:47.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.276+0000 7fc53f400640 1 --2- 172.21.15.40:0/926974874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 0x55a02d9456d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:47.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.276+0000 7fc53fe00640 1 --2- 172.21.15.40:0/926974874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 0x55a02d930310 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fc53802a610 tx=0x7fc538004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e402299503f5006 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.277+0000 7fc52b400640 1 -- 172.21.15.40:0/926974874 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc538040070 con 0x55a02d938fb0 2024-04-26T23:09:47.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.277+0000 7fc52b400640 1 -- 172.21.15.40:0/926974874 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc538045660 con 0x55a02d938fb0 2024-04-26T23:09:47.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.277+0000 7fc52b400640 1 -- 172.21.15.40:0/926974874 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc538009450 con 0x55a02d938fb0 2024-04-26T23:09:47.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.277+0000 7fc549598740 1 -- 172.21.15.40:0/926974874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 msgr2=0x55a02d930310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.277+0000 7fc549598740 1 --2- 172.21.15.40:0/926974874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 0x55a02d930310 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fc53802a610 tx=0x7fc538004a40 comp rx=0 tx=0).stop 2024-04-26T23:09:47.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.277+0000 7fc549598740 1 -- 172.21.15.40:0/926974874 shutdown_connections 2024-04-26T23:09:47.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.277+0000 7fc549598740 1 --2- 172.21.15.40:0/926974874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d938fb0 0x55a02d930310 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.277+0000 7fc549598740 1 --2- 172.21.15.40:0/926974874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d91b710 0x55a02d9368a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.277+0000 7fc549598740 1 --2- 172.21.15.40:0/926974874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 0x55a02d9456d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.277+0000 7fc549598740 1 -- 172.21.15.40:0/926974874 >> 172.21.15.40:0/926974874 conn(0x55a02d8f13f0 msgr2=0x55a02d92ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.278+0000 7fc549598740 1 -- 172.21.15.40:0/926974874 shutdown_connections 2024-04-26T23:09:47.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.278+0000 7fc549598740 1 -- 172.21.15.40:0/926974874 wait complete. 2024-04-26T23:09:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.278+0000 7fc549598740 1 -- start start 2024-04-26T23:09:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.278+0000 7fc549598740 1 Processor -- start 2024-04-26T23:09:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.279+0000 7fc549598740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 0x55a02d9af000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.279+0000 7fc549598740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 0x55a02d930310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.279+0000 7fc549598740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d938fb0 0x55a02d930850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.279+0000 7fc549598740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a02d861fd0 con 0x55a02d91b710 2024-04-26T23:09:47.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.279+0000 7fc549598740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc5401f8ae0 con 0x55a02d938fb0 2024-04-26T23:09:47.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.279+0000 7fc549598740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a02d91bc10 con 0x55a02d917c60 2024-04-26T23:09:47.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.279+0000 7fc549598740 10 client.0 _ll_register_callbacks cb 0x7fc530009480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:47.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.280+0000 7fc53ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 0x55a02d930310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.280+0000 7fc53ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 0x55a02d930310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39824/0 (socket says 172.21.15.40:39824) 2024-04-26T23:09:47.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.280+0000 7fc53ea00640 1 -- 172.21.15.40:0/3869418505 learned_addr learned my addr 172.21.15.40:0/3869418505 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.280+0000 7fc53f400640 1 --2- 172.21.15.40:0/3869418505 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 0x55a02d9af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.280+0000 7fc53fe00640 1 --2- 172.21.15.40:0/3869418505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d938fb0 0x55a02d930850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.280+0000 7fc53ea00640 1 -- 172.21.15.40:0/3869418505 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 msgr2=0x55a02d9af000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.280+0000 7fc53ea00640 1 --2- 172.21.15.40:0/3869418505 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 0x55a02d9af000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.280+0000 7fc53ea00640 1 -- 172.21.15.40:0/3869418505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d938fb0 msgr2=0x55a02d930850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.280+0000 7fc53ea00640 1 --2- 172.21.15.40:0/3869418505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d938fb0 0x55a02d930850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.280+0000 7fc53ea00640 1 -- 172.21.15.40:0/3869418505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a02d81ca30 con 0x55a02d91b710 2024-04-26T23:09:47.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.280+0000 7fc53ea00640 1 --2- 172.21.15.40:0/3869418505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 0x55a02d930310 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0895b0 tx=0x7fc52c01e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f21b1b7ddc9630 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.280+0000 7fc528c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:47.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc528c00640 1 -- 172.21.15.40:0/3869418505 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c071040 con 0x55a02d91b710 2024-04-26T23:09:47.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc528c00640 1 -- 172.21.15.40:0/3869418505 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fc52c057060 con 0x55a02d91b710 2024-04-26T23:09:47.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc549598740 1 -- 172.21.15.40:0/3869418505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc538045660 con 0x55a02d91b710 2024-04-26T23:09:47.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc528c00640 1 -- 172.21.15.40:0/3869418505 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7fc52c07e050 con 0x55a02d91b710 2024-04-26T23:09:47.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc549598740 10 client.9531 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:47.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc549598740 1 -- 172.21.15.40:0/3869418505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55a02d91f200 con 0x55a02d91b710 2024-04-26T23:09:47.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc528c00640 1 -- 172.21.15.40:0/3869418505 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc538045660 con 0x55a02d91b710 2024-04-26T23:09:47.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc528c00640 1 -- 172.21.15.40:0/3869418505 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7fc52c081020 con 0x55a02d91b710 2024-04-26T23:09:47.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc549598740 10 client.9531 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:47.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc549598740 10 client.9531 resolve_mds: resolved 7:1 to role '7:1' aka daemon mds.c 2024-04-26T23:09:47.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc549598740 20 client.9531 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:47.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc549598740 1 --2- 172.21.15.40:0/3869418505 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55a02d91aab0 0x55a02d9af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc549598740 4 client.9531 mds_command: new command op to 8949 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:09:47.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.281+0000 7fc549598740 1 -- 172.21.15.40:0/3869418505 --> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a02d85c650 con 0x55a02d91aab0 2024-04-26T23:09:47.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.282+0000 7fc53f400640 1 --2- 172.21.15.40:0/3869418505 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55a02d91aab0 0x55a02d9af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.282+0000 7fc53f400640 1 --2- 172.21.15.40:0/3869418505 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55a02d91aab0 0x55a02d9af930 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d6e76b290abe61c7 server_cookie=6c6f7f1e664911ca in_seq=0 out_seq=0 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.282+0000 7fc528c00640 10 client.9531 ms_handle_connect on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.282+0000 7fc528c00640 1 -- 172.21.15.40:0/3869418505 <== mds.1 v2:172.21.15.40:6832/3250599834 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55a02d85c650 con 0x55a02d91aab0 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc528c00640 10 client.9531 handle_command_reply: tid=0 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc549598740 1 client.9531 shutdown 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc549598740 1 -- 172.21.15.40:0/3869418505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 msgr2=0x55a02d930310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc549598740 1 --2- 172.21.15.40:0/3869418505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 0x55a02d930310 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0895b0 tx=0x7fc52c01e040 comp rx=0 tx=0).stop 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc549598740 1 -- 172.21.15.40:0/3869418505 shutdown_connections 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc549598740 1 --2- 172.21.15.40:0/3869418505 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55a02d91aab0 0x55a02d9af930 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc549598740 1 --2- 172.21.15.40:0/3869418505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a02d938fb0 0x55a02d930850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc549598740 1 --2- 172.21.15.40:0/3869418505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a02d91b710 0x55a02d930310 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc549598740 1 --2- 172.21.15.40:0/3869418505 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a02d917c60 0x55a02d9af000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc528c00640 0 client.9531 ms_handle_reset on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc549598740 1 -- 172.21.15.40:0/3869418505 >> 172.21.15.40:0/3869418505 conn(0x55a02d8f13f0 msgr2=0x55a02d945750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc549598740 1 -- 172.21.15.40:0/3869418505 shutdown_connections 2024-04-26T23:09:47.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc549598740 1 -- 172.21.15.40:0/3869418505 wait complete. 2024-04-26T23:09:47.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.283+0000 7fc549598740 20 client.9531 trim_cache size 0 max 16384 2024-04-26T23:09:47.288 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:47.288 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-26T23:09:47.288 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:09:47.288 INFO:teuthology.orchestra.run.smithi040.stdout: "a9827531-70a9-4d6f-b12a-2d71fa886d03": { 2024-04-26T23:09:47.288 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:09:47.288 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "a9827531-70a9-4d6f-b12a-2d71fa886d03", 2024-04-26T23:09:47.288 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:09:47.288 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:47.289 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:47.289 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:47.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.284+0000 7fc545000640 1 -- 172.21.15.40:0/2193440029 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc52006c8f0 msgr2=0x7fc52006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.284+0000 7fc545000640 1 --2- 172.21.15.40:0/2193440029 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc52006c8f0 0x7fc52006edb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc530006fd0 tx=0x7fc530008040 comp rx=0 tx=0).stop 2024-04-26T23:09:47.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.284+0000 7fc545000640 1 -- 172.21.15.40:0/2193440029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5400d18c0 msgr2=0x7fc54009cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.284+0000 7fc545000640 1 --2- 172.21.15.40:0/2193440029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5400d18c0 0x7fc54009cc70 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00c430 tx=0x7fc52c00c8f0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.284+0000 7fc545000640 1 -- 172.21.15.40:0/2193440029 shutdown_connections 2024-04-26T23:09:47.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.284+0000 7fc545000640 1 --2- 172.21.15.40:0/2193440029 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc52006c8f0 0x7fc52006edb0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.284+0000 7fc545000640 1 --2- 172.21.15.40:0/2193440029 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc54015d110 0x7fc5400cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.284+0000 7fc545000640 1 --2- 172.21.15.40:0/2193440029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5400d18c0 0x7fc54009cc70 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.284+0000 7fc545000640 1 --2- 172.21.15.40:0/2193440029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5400d09b0 0x7fc54009f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.284+0000 7fc545000640 1 -- 172.21.15.40:0/2193440029 >> 172.21.15.40:0/2193440029 conn(0x7fc5400d2bd0 msgr2=0x7fc54015c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.284+0000 7fc545000640 1 -- 172.21.15.40:0/2193440029 shutdown_connections 2024-04-26T23:09:47.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.284+0000 7fc545000640 1 -- 172.21.15.40:0/2193440029 wait complete. 2024-04-26T23:09:47.301 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:09:47.301 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:09:47.302 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.7:2 scrub status 2024-04-26T23:09:47.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.512+0000 7fd042400640 1 Processor -- start 2024-04-26T23:09:47.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.513+0000 7fd042400640 1 -- start start 2024-04-26T23:09:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.514+0000 7fd042400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd03c0cfaf0 0x7fd03c073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.514+0000 7fd042400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd03c06dd40 0x7fd03c072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.514+0000 7fd042400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03c161880 con 0x7fd03c160a50 2024-04-26T23:09:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.514+0000 7fd042400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fd03c071ce0 con 0x7fd03c06dd40 2024-04-26T23:09:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.514+0000 7fd042400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd03c0722a0 con 0x7fd03c0cfaf0 2024-04-26T23:09:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.515+0000 7fd041a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd03c06dd40 0x7fd03c072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.515+0000 7fd041a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd03c06dd40 0x7fd03c072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39828/0 (socket says 172.21.15.40:39828) 2024-04-26T23:09:47.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.515+0000 7fd041a00640 1 -- 172.21.15.40:0/342566059 learned_addr learned my addr 172.21.15.40:0/342566059 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.515+0000 7fd041a00640 1 -- 172.21.15.40:0/342566059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd03c0cfaf0 msgr2=0x7fd03c073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.515+0000 7fd03be00640 1 --2- 172.21.15.40:0/342566059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd03c0cfaf0 0x7fd03c073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.515+0000 7fd041a00640 1 --2- 172.21.15.40:0/342566059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd03c0cfaf0 0x7fd03c073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.515+0000 7fd041a00640 1 -- 172.21.15.40:0/342566059 >> v1:172.21.15.103:6789/0 conn(0x7fd03c160a50 legacy=0x7fd03c160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.515+0000 7fd041a00640 1 -- 172.21.15.40:0/342566059 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03c0cff80 con 0x7fd03c06dd40 2024-04-26T23:09:47.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.516+0000 7fd03be00640 1 --2- 172.21.15.40:0/342566059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd03c0cfaf0 0x7fd03c073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:47.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.517+0000 7fd041a00640 1 --2- 172.21.15.40:0/342566059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd03c06dd40 0x7fd03c072730 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fd034009e50 tx=0x7fd03402a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f0170d01340cd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.517+0000 7fd03b400640 1 -- 172.21.15.40:0/342566059 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd034036070 con 0x7fd03c06dd40 2024-04-26T23:09:47.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.517+0000 7fd03b400640 1 -- 172.21.15.40:0/342566059 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd034031600 con 0x7fd03c06dd40 2024-04-26T23:09:47.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.517+0000 7fd03b400640 1 -- 172.21.15.40:0/342566059 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0340319d0 con 0x7fd03c06dd40 2024-04-26T23:09:47.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.518+0000 7fd042400640 1 -- 172.21.15.40:0/342566059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd03c06dd40 msgr2=0x7fd03c072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.518+0000 7fd042400640 1 --2- 172.21.15.40:0/342566059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd03c06dd40 0x7fd03c072730 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fd034009e50 tx=0x7fd03402a6d0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.518+0000 7fd042400640 1 -- 172.21.15.40:0/342566059 shutdown_connections 2024-04-26T23:09:47.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.518+0000 7fd042400640 1 --2- 172.21.15.40:0/342566059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd03c0cfaf0 0x7fd03c073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.518+0000 7fd042400640 1 --2- 172.21.15.40:0/342566059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd03c06dd40 0x7fd03c072730 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.518+0000 7fd042400640 1 -- 172.21.15.40:0/342566059 >> 172.21.15.40:0/342566059 conn(0x7fd03c0b2720 msgr2=0x7fd03c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.518+0000 7fd042400640 1 -- 172.21.15.40:0/342566059 shutdown_connections 2024-04-26T23:09:47.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.518+0000 7fd042400640 1 -- 172.21.15.40:0/342566059 wait complete. 2024-04-26T23:09:47.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.520+0000 7fd042400640 1 Processor -- start 2024-04-26T23:09:47.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.521+0000 7fd042400640 1 -- start start 2024-04-26T23:09:47.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.522+0000 7fd042400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd03c06dd40 0x7fd03c134660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.522+0000 7fd042400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd03c160a50 0x7fd03c15a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.522+0000 7fd042400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd03c15f650 0x7fd03c15fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.522+0000 7fd042400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c071f30 con 0x7fd03c06dd40 2024-04-26T23:09:47.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.522+0000 7fd042400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c073f90 con 0x7fd03c160a50 2024-04-26T23:09:47.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.522+0000 7fd042400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd03c06e1f0 con 0x7fd03c15f650 2024-04-26T23:09:47.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.522+0000 7fd041000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd03c06dd40 0x7fd03c134660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.522+0000 7fd041000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd03c06dd40 0x7fd03c134660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39832/0 (socket says 172.21.15.40:39832) 2024-04-26T23:09:47.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.522+0000 7fd041000640 1 -- 172.21.15.40:0/3352025648 learned_addr learned my addr 172.21.15.40:0/3352025648 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.523+0000 7fd041a00640 1 --2- 172.21.15.40:0/3352025648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd03c15f650 0x7fd03c15fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.523+0000 7fd041000640 1 -- 172.21.15.40:0/3352025648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd03c15f650 msgr2=0x7fd03c15fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.523+0000 7fd041000640 1 --2- 172.21.15.40:0/3352025648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd03c15f650 0x7fd03c15fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.523+0000 7fd041000640 1 -- 172.21.15.40:0/3352025648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd03c160a50 msgr2=0x7fd03c15a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.523+0000 7fd041000640 1 --2- 172.21.15.40:0/3352025648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd03c160a50 0x7fd03c15a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.523+0000 7fd041000640 1 -- 172.21.15.40:0/3352025648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03c134ba0 con 0x7fd03c06dd40 2024-04-26T23:09:47.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.523+0000 7fd041000640 1 --2- 172.21.15.40:0/3352025648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd03c06dd40 0x7fd03c134660 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fd03000b3f0 tx=0x7fd03000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.523+0000 7fd038c00640 1 -- 172.21.15.40:0/3352025648 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd030015020 con 0x7fd03c06dd40 2024-04-26T23:09:47.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.524+0000 7fd038c00640 1 -- 172.21.15.40:0/3352025648 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd030011830 con 0x7fd03c06dd40 2024-04-26T23:09:47.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.524+0000 7fd038c00640 1 -- 172.21.15.40:0/3352025648 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd030011c20 con 0x7fd03c06dd40 2024-04-26T23:09:47.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.524+0000 7fd042400640 1 -- 172.21.15.40:0/3352025648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd03c142de0 con 0x7fd03c06dd40 2024-04-26T23:09:47.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.524+0000 7fd042400640 1 -- 172.21.15.40:0/3352025648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd03c1433a0 con 0x7fd03c06dd40 2024-04-26T23:09:47.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.526+0000 7fd038c00640 1 -- 172.21.15.40:0/3352025648 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd030018020 con 0x7fd03c06dd40 2024-04-26T23:09:47.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.527+0000 7fd038c00640 1 --2- 172.21.15.40:0/3352025648 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd01c06c8f0 0x7fd01c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.527+0000 7fd038c00640 1 -- 172.21.15.40:0/3352025648 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fd03008b980 con 0x7fd03c06dd40 2024-04-26T23:09:47.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.528+0000 7fd046833740 1 Processor -- start 2024-04-26T23:09:47.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.528+0000 7fd046833740 1 -- start start 2024-04-26T23:09:47.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.528+0000 7fd03be00640 1 --2- 172.21.15.40:0/3352025648 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd01c06c8f0 0x7fd01c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.529+0000 7fd046833740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 0x564050d8aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.529+0000 7fd046833740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050da8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.529+0000 7fd046833740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d603f0 0x564050da6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.529+0000 7fd046833740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564050c70030 con 0x564050da7fb0 2024-04-26T23:09:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.529+0000 7fd046833740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564050c8b7f0 con 0x564050d8a710 2024-04-26T23:09:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.529+0000 7fd046833740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564050c467e0 con 0x564050d603f0 2024-04-26T23:09:47.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.529+0000 7fd03be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050da8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.529+0000 7fd03be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050da8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39844/0 (socket says 172.21.15.40:39844) 2024-04-26T23:09:47.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.529+0000 7fd03be00640 1 -- 172.21.15.40:0/280181709 learned_addr learned my addr 172.21.15.40:0/280181709 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.529+0000 7fd03be00640 1 --2- 172.21.15.40:0/3352025648 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd01c06c8f0 0x7fd01c06edb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd028006fd0 tx=0x7fd028008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.529+0000 7fd041000640 1 --2- 172.21.15.40:0/280181709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 0x564050d8aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.529+0000 7fd041a00640 1 --2- 172.21.15.40:0/280181709 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d603f0 0x564050da6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.530+0000 7fd03be00640 1 -- 172.21.15.40:0/280181709 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d603f0 msgr2=0x564050da6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.530+0000 7fd03be00640 1 --2- 172.21.15.40:0/280181709 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d603f0 0x564050da6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.530+0000 7fd03be00640 1 -- 172.21.15.40:0/280181709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 msgr2=0x564050d8aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.530+0000 7fd03be00640 1 --2- 172.21.15.40:0/280181709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 0x564050d8aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.530+0000 7fd03be00640 1 -- 172.21.15.40:0/280181709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564050c52050 con 0x564050da7fb0 2024-04-26T23:09:47.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.530+0000 7fd03be00640 1 --2- 172.21.15.40:0/280181709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050da8500 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fd02800cac0 tx=0x7fd02800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b5f52f031096a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.530+0000 7fd02d600640 1 -- 172.21.15.40:0/280181709 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd028022070 con 0x564050da7fb0 2024-04-26T23:09:47.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.530+0000 7fd02d600640 1 -- 172.21.15.40:0/280181709 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd028013b40 con 0x564050da7fb0 2024-04-26T23:09:47.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.530+0000 7fd02d600640 1 -- 172.21.15.40:0/280181709 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd028019040 con 0x564050da7fb0 2024-04-26T23:09:47.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.531+0000 7fd046833740 1 -- 172.21.15.40:0/280181709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 msgr2=0x564050da8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.531+0000 7fd046833740 1 --2- 172.21.15.40:0/280181709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050da8500 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fd02800cac0 tx=0x7fd02800f000 comp rx=0 tx=0).stop 2024-04-26T23:09:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.531+0000 7fd046833740 1 -- 172.21.15.40:0/280181709 shutdown_connections 2024-04-26T23:09:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.531+0000 7fd046833740 1 --2- 172.21.15.40:0/280181709 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d603f0 0x564050da6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.531+0000 7fd046833740 1 --2- 172.21.15.40:0/280181709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050da8500 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.531+0000 7fd046833740 1 --2- 172.21.15.40:0/280181709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 0x564050d8aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.531+0000 7fd046833740 1 -- 172.21.15.40:0/280181709 >> 172.21.15.40:0/280181709 conn(0x564050d5cd20 msgr2=0x564050da5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.531+0000 7fd046833740 1 -- 172.21.15.40:0/280181709 shutdown_connections 2024-04-26T23:09:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.531+0000 7fd046833740 1 -- 172.21.15.40:0/280181709 wait complete. 2024-04-26T23:09:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.532+0000 7fd046833740 1 -- start start 2024-04-26T23:09:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.532+0000 7fd046833740 1 Processor -- start 2024-04-26T23:09:47.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.533+0000 7fd046833740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d8a710 0x564050db5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.533+0000 7fd046833740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050d8e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.533+0000 7fd046833740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d78f90 0x564050d9c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.533+0000 7fd046833740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564050cc8200 con 0x564050da7fb0 2024-04-26T23:09:47.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.533+0000 7fd046833740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564050cc2ee0 con 0x564050d78f90 2024-04-26T23:09:47.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.533+0000 7fd046833740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564050ccfe70 con 0x564050d8a710 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.533+0000 7fd046833740 10 client.0 _ll_register_callbacks cb 0x564050cb9270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.533+0000 7fd03be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050d8e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.534+0000 7fd03be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050d8e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39856/0 (socket says 172.21.15.40:39856) 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.534+0000 7fd03be00640 1 -- 172.21.15.40:0/4227253075 learned_addr learned my addr 172.21.15.40:0/4227253075 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.534+0000 7fd041000640 1 --2- 172.21.15.40:0/4227253075 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d8a710 0x564050db5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.534+0000 7fd041a00640 1 --2- 172.21.15.40:0/4227253075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d78f90 0x564050d9c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.534+0000 7fd03be00640 1 -- 172.21.15.40:0/4227253075 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d8a710 msgr2=0x564050db5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.534+0000 7fd03be00640 1 --2- 172.21.15.40:0/4227253075 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d8a710 0x564050db5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.534+0000 7fd03be00640 1 -- 172.21.15.40:0/4227253075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d78f90 msgr2=0x564050d9c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.534+0000 7fd03be00640 1 --2- 172.21.15.40:0/4227253075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d78f90 0x564050d9c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.534+0000 7fd03be00640 1 -- 172.21.15.40:0/4227253075 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564050a9f7d0 con 0x564050da7fb0 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.534+0000 7fd03be00640 1 --2- 172.21.15.40:0/4227253075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050d8e620 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fd02800a000 tx=0x7fd028006c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770543fe9952a565 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.534+0000 7fd022000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.535+0000 7fd022000640 1 -- 172.21.15.40:0/4227253075 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02802c020 con 0x564050da7fb0 2024-04-26T23:09:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.535+0000 7fd022000640 1 -- 172.21.15.40:0/4227253075 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fd028028470 con 0x564050da7fb0 2024-04-26T23:09:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.535+0000 7fd046833740 1 -- 172.21.15.40:0/4227253075 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd028013b40 con 0x564050da7fb0 2024-04-26T23:09:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.535+0000 7fd022000640 1 -- 172.21.15.40:0/4227253075 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7fd028036050 con 0x564050da7fb0 2024-04-26T23:09:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.535+0000 7fd046833740 10 client.9543 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.535+0000 7fd046833740 1 -- 172.21.15.40:0/4227253075 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x5640509d53f0 con 0x564050da7fb0 2024-04-26T23:09:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.535+0000 7fd022000640 1 -- 172.21.15.40:0/4227253075 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd028013b40 con 0x564050da7fb0 2024-04-26T23:09:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.535+0000 7fd022000640 1 -- 172.21.15.40:0/4227253075 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7fd028039020 con 0x564050da7fb0 2024-04-26T23:09:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.535+0000 7fd046833740 10 client.9543 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.536+0000 7fd046833740 10 client.9543 resolve_mds: resolved 7:2 to role '7:2' aka daemon mds.d 2024-04-26T23:09:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.536+0000 7fd046833740 20 client.9543 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.536+0000 7fd046833740 1 --2- 172.21.15.40:0/4227253075 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x564050d59ef0 0x564050d8cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.536+0000 7fd046833740 4 client.9543 mds_command: new command op to 6598 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.536+0000 7fd046833740 1 -- 172.21.15.40:0/4227253075 --> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564050cd17e0 con 0x564050d59ef0 2024-04-26T23:09:47.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.536+0000 7fd041000640 1 --2- 172.21.15.40:0/4227253075 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x564050d59ef0 0x564050d8cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.537+0000 7fd041000640 1 --2- 172.21.15.40:0/4227253075 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x564050d59ef0 0x564050d8cbc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3e78b8545539494e server_cookie=1e08ece13fcc3e25 in_seq=0 out_seq=0 2024-04-26T23:09:47.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.537+0000 7fd022000640 10 client.9543 ms_handle_connect on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:47.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.539+0000 7fd022000640 1 -- 172.21.15.40:0/4227253075 <== mds.2 v2:172.21.15.103:6835/2022020922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564050cd17e0 con 0x564050d59ef0 2024-04-26T23:09:47.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.539+0000 7fd022000640 10 client.9543 handle_command_reply: tid=0 2024-04-26T23:09:47.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.540+0000 7fd046833740 1 client.9543 shutdown 2024-04-26T23:09:47.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.540+0000 7fd046833740 1 -- 172.21.15.40:0/4227253075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 msgr2=0x564050d8e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.540+0000 7fd046833740 1 --2- 172.21.15.40:0/4227253075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050d8e620 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fd02800a000 tx=0x7fd028006c90 comp rx=0 tx=0).stop 2024-04-26T23:09:47.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.540+0000 7fd046833740 1 -- 172.21.15.40:0/4227253075 shutdown_connections 2024-04-26T23:09:47.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.540+0000 7fd046833740 1 --2- 172.21.15.40:0/4227253075 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x564050d59ef0 0x564050d8cbc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.540+0000 7fd046833740 1 --2- 172.21.15.40:0/4227253075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d78f90 0x564050d9c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.540+0000 7fd046833740 1 --2- 172.21.15.40:0/4227253075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050d8e620 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.540+0000 7fd046833740 1 --2- 172.21.15.40:0/4227253075 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d8a710 0x564050db5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.540+0000 7fd022000640 0 client.9543 ms_handle_reset on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:47.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.540+0000 7fd046833740 1 -- 172.21.15.40:0/4227253075 >> 172.21.15.40:0/4227253075 conn(0x564050d603f0 msgr2=0x564050da64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.540+0000 7fd046833740 1 -- 172.21.15.40:0/4227253075 shutdown_connections 2024-04-26T23:09:47.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.540+0000 7fd046833740 1 -- 172.21.15.40:0/4227253075 wait complete. 2024-04-26T23:09:47.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.540+0000 7fd046833740 20 client.9543 trim_cache size 0 max 16384 2024-04-26T23:09:47.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.560+0000 7fd046833740 1 Processor -- start 2024-04-26T23:09:47.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.560+0000 7fd046833740 1 -- start start 2024-04-26T23:09:47.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.560+0000 7fd046833740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d78f90 0x564050da5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.560+0000 7fd046833740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 0x564050d56350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.560+0000 7fd046833740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050d619b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.560+0000 7fd046833740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c201d00 con 0x564050da7fb0 2024-04-26T23:09:47.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.561+0000 7fd046833740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c200ea0 con 0x564050d8a710 2024-04-26T23:09:47.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.561+0000 7fd046833740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd03c2015d0 con 0x564050d78f90 2024-04-26T23:09:47.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.561+0000 7fd041a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050d619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.561+0000 7fd041a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050d619b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39864/0 (socket says 172.21.15.40:39864) 2024-04-26T23:09:47.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.561+0000 7fd041a00640 1 -- 172.21.15.40:0/3678421899 learned_addr learned my addr 172.21.15.40:0/3678421899 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.561+0000 7fd041000640 1 --2- 172.21.15.40:0/3678421899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d78f90 0x564050da5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.561+0000 7fd03be00640 1 --2- 172.21.15.40:0/3678421899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 0x564050d56350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.561+0000 7fd041a00640 1 -- 172.21.15.40:0/3678421899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d78f90 msgr2=0x564050da5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.561+0000 7fd041a00640 1 --2- 172.21.15.40:0/3678421899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d78f90 0x564050da5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.561+0000 7fd041a00640 1 -- 172.21.15.40:0/3678421899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 msgr2=0x564050d56350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.561+0000 7fd041a00640 1 --2- 172.21.15.40:0/3678421899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 0x564050d56350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.561+0000 7fd041a00640 1 -- 172.21.15.40:0/3678421899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640509950f0 con 0x564050da7fb0 2024-04-26T23:09:47.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.562+0000 7fd041a00640 1 --2- 172.21.15.40:0/3678421899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050d619b0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fd034034040 tx=0x7fd03402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618f52b70ac33a37 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.562+0000 7fd02cc00640 1 -- 172.21.15.40:0/3678421899 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd034036040 con 0x564050da7fb0 2024-04-26T23:09:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.562+0000 7fd02cc00640 1 -- 172.21.15.40:0/3678421899 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd03403c700 con 0x564050da7fb0 2024-04-26T23:09:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.562+0000 7fd02cc00640 1 -- 172.21.15.40:0/3678421899 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03403caf0 con 0x564050da7fb0 2024-04-26T23:09:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.563+0000 7fd046833740 1 -- 172.21.15.40:0/3678421899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 msgr2=0x564050d619b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.563+0000 7fd046833740 1 --2- 172.21.15.40:0/3678421899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050d619b0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fd034034040 tx=0x7fd03402e000 comp rx=0 tx=0).stop 2024-04-26T23:09:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.563+0000 7fd046833740 1 -- 172.21.15.40:0/3678421899 shutdown_connections 2024-04-26T23:09:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.563+0000 7fd046833740 1 --2- 172.21.15.40:0/3678421899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050da7fb0 0x564050d619b0 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.563+0000 7fd046833740 1 --2- 172.21.15.40:0/3678421899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 0x564050d56350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.563+0000 7fd046833740 1 --2- 172.21.15.40:0/3678421899 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050d78f90 0x564050da5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.563+0000 7fd046833740 1 -- 172.21.15.40:0/3678421899 >> 172.21.15.40:0/3678421899 conn(0x564050d603f0 msgr2=0x564050da6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.563+0000 7fd046833740 1 -- 172.21.15.40:0/3678421899 shutdown_connections 2024-04-26T23:09:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.563+0000 7fd046833740 1 -- 172.21.15.40:0/3678421899 wait complete. 2024-04-26T23:09:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.564+0000 7fd046833740 1 -- start start 2024-04-26T23:09:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.564+0000 7fd046833740 1 Processor -- start 2024-04-26T23:09:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.565+0000 7fd046833740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050d78f90 0x564050da4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.565+0000 7fd046833740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 0x564050da8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.565+0000 7fd046833740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050da7fb0 0x564050e07fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.565+0000 7fd046833740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564050cd0fd0 con 0x564050d78f90 2024-04-26T23:09:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.565+0000 7fd046833740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c132d20 con 0x564050d8a710 2024-04-26T23:09:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.565+0000 7fd046833740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x564050e20650 con 0x564050da7fb0 2024-04-26T23:09:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.565+0000 7fd046833740 10 client.0 _ll_register_callbacks cb 0x7fd028020070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd041000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050d78f90 0x564050da4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd041000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050d78f90 0x564050da4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39880/0 (socket says 172.21.15.40:39880) 2024-04-26T23:09:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd041000640 1 -- 172.21.15.40:0/3927921846 learned_addr learned my addr 172.21.15.40:0/3927921846 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd03be00640 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 0x564050da8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd041000640 1 -- 172.21.15.40:0/3927921846 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050da7fb0 msgr2=0x564050e07fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd041a00640 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050da7fb0 0x564050e07fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd041000640 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050da7fb0 0x564050e07fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd041000640 1 -- 172.21.15.40:0/3927921846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 msgr2=0x564050da8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd041000640 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 0x564050da8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd041000640 1 -- 172.21.15.40:0/3927921846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564050c8ba30 con 0x564050d78f90 2024-04-26T23:09:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd03be00640 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 0x564050da8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd041000640 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050d78f90 0x564050da4820 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fd030017470 tx=0x7fd03005d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f79289bf8f5e3d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd022000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.566+0000 7fd022000640 1 -- 172.21.15.40:0/3927921846 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd030055af0 con 0x564050d78f90 2024-04-26T23:09:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.567+0000 7fd022000640 1 -- 172.21.15.40:0/3927921846 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fd030084e20 con 0x564050d78f90 2024-04-26T23:09:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.567+0000 7fd046833740 1 -- 172.21.15.40:0/3927921846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd03403c700 con 0x564050d78f90 2024-04-26T23:09:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.567+0000 7fd022000640 1 -- 172.21.15.40:0/3927921846 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7fd030055c90 con 0x564050d78f90 2024-04-26T23:09:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.567+0000 7fd046833740 10 client.9549 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.567+0000 7fd046833740 1 -- 172.21.15.40:0/3927921846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x564050e20390 con 0x564050d78f90 2024-04-26T23:09:47.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.567+0000 7fd022000640 1 -- 172.21.15.40:0/3927921846 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd03403c700 con 0x564050d78f90 2024-04-26T23:09:47.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.567+0000 7fd022000640 1 -- 172.21.15.40:0/3927921846 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7fd0300588d0 con 0x564050d78f90 2024-04-26T23:09:47.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.567+0000 7fd046833740 10 client.9549 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:47.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.567+0000 7fd046833740 10 client.9549 resolve_mds: resolved 7:2 to role '7:2' aka daemon mds.d 2024-04-26T23:09:47.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.567+0000 7fd046833740 20 client.9549 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:47.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.567+0000 7fd046833740 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x564050e08500 0x564050d5e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.567+0000 7fd046833740 4 client.9549 mds_command: new command op to 6598 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:09:47.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.567+0000 7fd046833740 1 -- 172.21.15.40:0/3927921846 --> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564050ccb650 con 0x564050e08500 2024-04-26T23:09:47.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.568+0000 7fd03be00640 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x564050e08500 0x564050d5e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.569+0000 7fd03be00640 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x564050e08500 0x564050d5e2c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3be346479ca9e98b server_cookie=5120411374a123f3 in_seq=0 out_seq=0 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.569+0000 7fd022000640 10 client.9549 ms_handle_connect on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd022000640 1 -- 172.21.15.40:0/3927921846 <== mds.2 v2:172.21.15.103:6835/2022020922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x564050ccb650 con 0x564050e08500 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd022000640 10 client.9549 handle_command_reply: tid=0 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd046833740 1 client.9549 shutdown 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd046833740 1 -- 172.21.15.40:0/3927921846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050d78f90 msgr2=0x564050da4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd046833740 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050d78f90 0x564050da4820 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fd030017470 tx=0x7fd03005d9a0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd046833740 1 -- 172.21.15.40:0/3927921846 shutdown_connections 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd046833740 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x564050e08500 0x564050d5e2c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd046833740 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x564050da7fb0 0x564050e07fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd046833740 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564050d8a710 0x564050da8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd046833740 1 --2- 172.21.15.40:0/3927921846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564050d78f90 0x564050da4820 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd046833740 1 -- 172.21.15.40:0/3927921846 >> 172.21.15.40:0/3927921846 conn(0x564050d603f0 msgr2=0x564050da57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd022000640 0 client.9549 ms_handle_reset on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:47.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd046833740 1 -- 172.21.15.40:0/3927921846 shutdown_connections 2024-04-26T23:09:47.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd046833740 1 -- 172.21.15.40:0/3927921846 wait complete. 2024-04-26T23:09:47.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.570+0000 7fd046833740 20 client.9549 trim_cache size 0 max 16384 2024-04-26T23:09:47.573 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:47.573 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-26T23:09:47.573 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:09:47.573 INFO:teuthology.orchestra.run.smithi040.stdout: "a9827531-70a9-4d6f-b12a-2d71fa886d03": { 2024-04-26T23:09:47.573 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:09:47.574 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "a9827531-70a9-4d6f-b12a-2d71fa886d03", 2024-04-26T23:09:47.574 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:09:47.574 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:47.574 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:47.574 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:47.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.571+0000 7fd042400640 1 -- 172.21.15.40:0/3352025648 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd01c06c8f0 msgr2=0x7fd01c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.571+0000 7fd042400640 1 --2- 172.21.15.40:0/3352025648 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd01c06c8f0 0x7fd01c06edb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd028006fd0 tx=0x7fd028008040 comp rx=0 tx=0).stop 2024-04-26T23:09:47.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.571+0000 7fd042400640 1 -- 172.21.15.40:0/3352025648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd03c06dd40 msgr2=0x7fd03c134660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.571+0000 7fd042400640 1 --2- 172.21.15.40:0/3352025648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd03c06dd40 0x7fd03c134660 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fd03000b3f0 tx=0x7fd03000b8b0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.571+0000 7fd042400640 1 -- 172.21.15.40:0/3352025648 shutdown_connections 2024-04-26T23:09:47.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.571+0000 7fd042400640 1 --2- 172.21.15.40:0/3352025648 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd01c06c8f0 0x7fd01c06edb0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.571+0000 7fd042400640 1 --2- 172.21.15.40:0/3352025648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd03c15f650 0x7fd03c15fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.571+0000 7fd042400640 1 --2- 172.21.15.40:0/3352025648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd03c160a50 0x7fd03c15a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.571+0000 7fd042400640 1 --2- 172.21.15.40:0/3352025648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd03c06dd40 0x7fd03c134660 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.571+0000 7fd042400640 1 -- 172.21.15.40:0/3352025648 >> 172.21.15.40:0/3352025648 conn(0x7fd03c0b2720 msgr2=0x7fd03c0aed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.571+0000 7fd042400640 1 -- 172.21.15.40:0/3352025648 shutdown_connections 2024-04-26T23:09:47.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.571+0000 7fd042400640 1 -- 172.21.15.40:0/3352025648 wait complete. 2024-04-26T23:09:47.591 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:09:47.591 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:09:47.591 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-26T23:09:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.806+0000 7fcb21000640 1 Processor -- start 2024-04-26T23:09:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.806+0000 7fcb21000640 1 -- start start 2024-04-26T23:09:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.808+0000 7fcb21000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c073b70 0x7fcb1c0aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.808+0000 7fcb21000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcb1c0cbb50 0x7fcb1c0af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.808+0000 7fcb21000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb1c072c00 con 0x7fcb1c15f380 2024-04-26T23:09:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.808+0000 7fcb21000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcb1c0729b0 con 0x7fcb1c0cbb50 2024-04-26T23:09:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.808+0000 7fcb21000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcb1c072ee0 con 0x7fcb1c073b70 2024-04-26T23:09:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.809+0000 7fcb13e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c073b70 0x7fcb1c0aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.809+0000 7fcb13e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c073b70 0x7fcb1c0aecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:41344/0 (socket says 172.21.15.40:41344) 2024-04-26T23:09:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.809+0000 7fcb13e00640 1 -- 172.21.15.40:0/1077112059 learned_addr learned my addr 172.21.15.40:0/1077112059 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.809+0000 7fcb19e00640 1 --1- 172.21.15.40:0/1077112059 >> v1:172.21.15.103:6789/0 conn(0x7fcb1c15f380 0x7fcb1c0cb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:35348/0 (socket says 172.21.15.40:35348) 2024-04-26T23:09:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.809+0000 7fcb1a800640 1 --2- 172.21.15.40:0/1077112059 >> v2:172.21.15.103:3300/0 conn(0x7fcb1c0cbb50 0x7fcb1c0af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.810+0000 7fcb13e00640 1 -- 172.21.15.40:0/1077112059 >> v2:172.21.15.103:3300/0 conn(0x7fcb1c0cbb50 msgr2=0x7fcb1c0af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.810+0000 7fcb13e00640 1 --2- 172.21.15.40:0/1077112059 >> v2:172.21.15.103:3300/0 conn(0x7fcb1c0cbb50 0x7fcb1c0af1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.810+0000 7fcb13e00640 1 -- 172.21.15.40:0/1077112059 >> v1:172.21.15.103:6789/0 conn(0x7fcb1c15f380 legacy=0x7fcb1c0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.810+0000 7fcb13e00640 1 -- 172.21.15.40:0/1077112059 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1c0731c0 con 0x7fcb1c073b70 2024-04-26T23:09:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.810+0000 7fcb19400640 1 -- 172.21.15.40:0/1077112059 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1434716341 0 0) 0x7fcb1c072c00 con 0x7fcb1c15f380 2024-04-26T23:09:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.811+0000 7fcb13e00640 1 --2- 172.21.15.40:0/1077112059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c073b70 0x7fcb1c0aecb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009870 tx=0x7fcb0402a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fcede7a0da56131 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.812+0000 7fcb19400640 1 -- 172.21.15.40:0/1077112059 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04037070 con 0x7fcb1c073b70 2024-04-26T23:09:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.812+0000 7fcb19400640 1 -- 172.21.15.40:0/1077112059 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb04032440 con 0x7fcb1c073b70 2024-04-26T23:09:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.812+0000 7fcb19400640 1 -- 172.21.15.40:0/1077112059 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04032810 con 0x7fcb1c073b70 2024-04-26T23:09:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.813+0000 7fcb21000640 1 -- 172.21.15.40:0/1077112059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c073b70 msgr2=0x7fcb1c0aecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.813+0000 7fcb21000640 1 --2- 172.21.15.40:0/1077112059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c073b70 0x7fcb1c0aecb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009870 tx=0x7fcb0402a250 comp rx=0 tx=0).stop 2024-04-26T23:09:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.813+0000 7fcb21000640 1 -- 172.21.15.40:0/1077112059 shutdown_connections 2024-04-26T23:09:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.813+0000 7fcb21000640 1 --2- 172.21.15.40:0/1077112059 >> v2:172.21.15.103:3300/0 conn(0x7fcb1c0cbb50 0x7fcb1c0af1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.813+0000 7fcb21000640 1 --2- 172.21.15.40:0/1077112059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c073b70 0x7fcb1c0aecb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.813+0000 7fcb21000640 1 -- 172.21.15.40:0/1077112059 >> 172.21.15.40:0/1077112059 conn(0x7fcb1c0d0be0 msgr2=0x7fcb1c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.813+0000 7fcb21000640 1 -- 172.21.15.40:0/1077112059 shutdown_connections 2024-04-26T23:09:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.813+0000 7fcb21000640 1 -- 172.21.15.40:0/1077112059 wait complete. 2024-04-26T23:09:47.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.815+0000 7fcb21000640 1 Processor -- start 2024-04-26T23:09:47.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.815+0000 7fcb21000640 1 -- start start 2024-04-26T23:09:47.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.816+0000 7fcb21000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c0cbb50 0x7fcb1c1f8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.816+0000 7fcb21000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c15f380 0x7fcb1c1500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.816+0000 7fcb21000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb1c152c90 0x7fcb1c153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.816+0000 7fcb21000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c073620 con 0x7fcb1c152c90 2024-04-26T23:09:47.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.816+0000 7fcb21000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c0d0830 con 0x7fcb1c0cbb50 2024-04-26T23:09:47.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.816+0000 7fcb21000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcb1c0737a0 con 0x7fcb1c15f380 2024-04-26T23:09:47.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.816+0000 7fcb1a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb1c152c90 0x7fcb1c153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.816+0000 7fcb1a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb1c152c90 0x7fcb1c153110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39886/0 (socket says 172.21.15.40:39886) 2024-04-26T23:09:47.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.816+0000 7fcb1a800640 1 -- 172.21.15.40:0/2344221897 learned_addr learned my addr 172.21.15.40:0/2344221897 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.816+0000 7fcb13e00640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c15f380 0x7fcb1c1500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.816+0000 7fcb19e00640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c0cbb50 0x7fcb1c1f8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.817+0000 7fcb1a800640 1 -- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c15f380 msgr2=0x7fcb1c1500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.817+0000 7fcb1a800640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c15f380 0x7fcb1c1500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.817+0000 7fcb1a800640 1 -- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c0cbb50 msgr2=0x7fcb1c1f8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.817+0000 7fcb1a800640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c0cbb50 0x7fcb1c1f8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.817+0000 7fcb1a800640 1 -- 172.21.15.40:0/2344221897 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1c1f8710 con 0x7fcb1c152c90 2024-04-26T23:09:47.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.817+0000 7fcb19e00640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c0cbb50 0x7fcb1c1f8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:47.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.817+0000 7fcb1a800640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb1c152c90 0x7fcb1c153110 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009e30 tx=0x7fcb1400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.817+0000 7fcb12000640 1 -- 172.21.15.40:0/2344221897 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb14018020 con 0x7fcb1c152c90 2024-04-26T23:09:47.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.817+0000 7fcb12000640 1 -- 172.21.15.40:0/2344221897 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb14014480 con 0x7fcb1c152c90 2024-04-26T23:09:47.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.817+0000 7fcb12000640 1 -- 172.21.15.40:0/2344221897 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb14014870 con 0x7fcb1c152c90 2024-04-26T23:09:47.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.817+0000 7fcb21000640 1 -- 172.21.15.40:0/2344221897 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb1c153750 con 0x7fcb1c152c90 2024-04-26T23:09:47.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.817+0000 7fcb21000640 1 -- 172.21.15.40:0/2344221897 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb1c1f89d0 con 0x7fcb1c152c90 2024-04-26T23:09:47.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.821+0000 7fcb12000640 1 -- 172.21.15.40:0/2344221897 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcb14005d10 con 0x7fcb1c152c90 2024-04-26T23:09:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.821+0000 7fcb12000640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcaf806c8f0 0x7fcaf806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.822+0000 7fcb12000640 1 -- 172.21.15.40:0/2344221897 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fcb1408cba0 con 0x7fcb1c152c90 2024-04-26T23:09:47.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.822+0000 7fcb23fcf740 1 Processor -- start 2024-04-26T23:09:47.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.822+0000 7fcb23fcf740 1 -- start start 2024-04-26T23:09:47.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.822+0000 7fcb19e00640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcaf806c8f0 0x7fcaf806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.823+0000 7fcb23fcf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19a1540 0x5632c19a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.823+0000 7fcb23fcf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19bede0 0x5632c19bf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.823+0000 7fcb23fcf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c1977220 0x5632c19bd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.823+0000 7fcb23fcf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5632c1887030 con 0x5632c1977220 2024-04-26T23:09:47.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.823+0000 7fcb23fcf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5632c18a27f0 con 0x5632c19bede0 2024-04-26T23:09:47.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.823+0000 7fcb23fcf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5632c185d7e0 con 0x5632c19a1540 2024-04-26T23:09:47.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.823+0000 7fcb1a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c1977220 0x5632c19bd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.823+0000 7fcb1a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c1977220 0x5632c19bd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39902/0 (socket says 172.21.15.40:39902) 2024-04-26T23:09:47.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.823+0000 7fcb1a800640 1 -- 172.21.15.40:0/609454708 learned_addr learned my addr 172.21.15.40:0/609454708 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.824+0000 7fcb19e00640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcaf806c8f0 0x7fcaf806edb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c000f60 tx=0x7fcb0c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.824+0000 7fcb1a800640 1 -- 172.21.15.40:0/609454708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19a1540 msgr2=0x5632c19a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.824+0000 7fcb1a800640 1 --2- 172.21.15.40:0/609454708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19a1540 0x5632c19a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.824+0000 7fcb13e00640 1 --2- 172.21.15.40:0/609454708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19bede0 0x5632c19bf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.824+0000 7fcb1a800640 1 -- 172.21.15.40:0/609454708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19bede0 msgr2=0x5632c19bf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.824+0000 7fcb1a800640 1 --2- 172.21.15.40:0/609454708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19bede0 0x5632c19bf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.824+0000 7fcb1a800640 1 -- 172.21.15.40:0/609454708 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632c1869050 con 0x5632c1977220 2024-04-26T23:09:47.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.824+0000 7fcb1a800640 1 --2- 172.21.15.40:0/609454708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c1977220 0x5632c19bd4a0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fcb14056b90 tx=0x7fcb1405a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b5fa8903d257da server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.824+0000 7fcb0aa00640 1 -- 172.21.15.40:0/609454708 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb14089070 con 0x5632c1977220 2024-04-26T23:09:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.824+0000 7fcb0aa00640 1 -- 172.21.15.40:0/609454708 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb14063310 con 0x5632c1977220 2024-04-26T23:09:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.824+0000 7fcb0aa00640 1 -- 172.21.15.40:0/609454708 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb14058a30 con 0x5632c1977220 2024-04-26T23:09:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.825+0000 7fcb23fcf740 1 -- 172.21.15.40:0/609454708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c1977220 msgr2=0x5632c19bd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.825+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/609454708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c1977220 0x5632c19bd4a0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fcb14056b90 tx=0x7fcb1405a630 comp rx=0 tx=0).stop 2024-04-26T23:09:47.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.825+0000 7fcb23fcf740 1 -- 172.21.15.40:0/609454708 shutdown_connections 2024-04-26T23:09:47.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.825+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/609454708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c1977220 0x5632c19bd4a0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.825+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/609454708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19bede0 0x5632c19bf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.825+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/609454708 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19a1540 0x5632c19a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.825+0000 7fcb23fcf740 1 -- 172.21.15.40:0/609454708 >> 172.21.15.40:0/609454708 conn(0x5632c1973b50 msgr2=0x5632c19bc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.825+0000 7fcb23fcf740 1 -- 172.21.15.40:0/609454708 shutdown_connections 2024-04-26T23:09:47.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.825+0000 7fcb23fcf740 1 -- 172.21.15.40:0/609454708 wait complete. 2024-04-26T23:09:47.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.826+0000 7fcb23fcf740 1 -- start start 2024-04-26T23:09:47.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.826+0000 7fcb23fcf740 1 Processor -- start 2024-04-26T23:09:47.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.827+0000 7fcb23fcf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c19a1540 0x5632c1975800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.827+0000 7fcb23fcf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19bede0 0x5632c19707b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.827+0000 7fcb23fcf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c199c400 0x5632c1970cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.827+0000 7fcb23fcf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5632c18df200 con 0x5632c19a1540 2024-04-26T23:09:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.827+0000 7fcb23fcf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5632c18d9ee0 con 0x5632c199c400 2024-04-26T23:09:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.827+0000 7fcb23fcf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5632c18e6e70 con 0x5632c19bede0 2024-04-26T23:09:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.828+0000 7fcb23fcf740 10 client.0 _ll_register_callbacks cb 0x5632c18d0270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.828+0000 7fcb19e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c19a1540 0x5632c1975800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.828+0000 7fcb19e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c19a1540 0x5632c1975800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39918/0 (socket says 172.21.15.40:39918) 2024-04-26T23:09:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.828+0000 7fcb19e00640 1 -- 172.21.15.40:0/2627646578 learned_addr learned my addr 172.21.15.40:0/2627646578 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.828+0000 7fcb13e00640 1 --2- 172.21.15.40:0/2627646578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19bede0 0x5632c19707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.828+0000 7fcb1a800640 1 --2- 172.21.15.40:0/2627646578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c199c400 0x5632c1970cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.828+0000 7fcb19e00640 1 -- 172.21.15.40:0/2627646578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19bede0 msgr2=0x5632c19707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.828+0000 7fcb19e00640 1 --2- 172.21.15.40:0/2627646578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19bede0 0x5632c19707b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.828+0000 7fcb19e00640 1 -- 172.21.15.40:0/2627646578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c199c400 msgr2=0x5632c1970cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.828+0000 7fcb19e00640 1 --2- 172.21.15.40:0/2627646578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c199c400 0x5632c1970cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.828+0000 7fcb19e00640 1 -- 172.21.15.40:0/2627646578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632c16b67d0 con 0x5632c19a1540 2024-04-26T23:09:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.829+0000 7fcb19e00640 1 --2- 172.21.15.40:0/2627646578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c19a1540 0x5632c1975800 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c013750 tx=0x7fcb0c013b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86cb04ba5a1ce192 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.829+0000 7fcaff400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.829+0000 7fcaff400640 1 -- 172.21.15.40:0/2627646578 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c024060 con 0x5632c19a1540 2024-04-26T23:09:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.829+0000 7fcaff400640 1 -- 172.21.15.40:0/2627646578 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fcb0c01fc40 con 0x5632c19a1540 2024-04-26T23:09:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.829+0000 7fcb23fcf740 1 -- 172.21.15.40:0/2627646578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb14063310 con 0x5632c19a1540 2024-04-26T23:09:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.829+0000 7fcaff400640 1 -- 172.21.15.40:0/2627646578 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7fcb0c019050 con 0x5632c19a1540 2024-04-26T23:09:47.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.829+0000 7fcb23fcf740 10 client.9558 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:47.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.829+0000 7fcb23fcf740 1 -- 172.21.15.40:0/2627646578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x5632c15ec3f0 con 0x5632c19a1540 2024-04-26T23:09:47.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.829+0000 7fcaff400640 1 -- 172.21.15.40:0/2627646578 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb14063310 con 0x5632c19a1540 2024-04-26T23:09:47.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.830+0000 7fcaff400640 1 -- 172.21.15.40:0/2627646578 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7fcb0c0070e0 con 0x5632c19a1540 2024-04-26T23:09:47.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.830+0000 7fcb23fcf740 10 client.9558 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:47.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.830+0000 7fcb23fcf740 10 client.9558 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:47.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.830+0000 7fcb23fcf740 20 client.9558 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:47.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.830+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/2627646578 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5632c1975d40 0x5632c19b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.830+0000 7fcb23fcf740 4 client.9558 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:47.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.830+0000 7fcb23fcf740 1 -- 172.21.15.40:0/2627646578 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632c18e87e0 con 0x5632c1975d40 2024-04-26T23:09:47.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.830+0000 7fcb13e00640 1 --2- 172.21.15.40:0/2627646578 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5632c1975d40 0x5632c19b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.831+0000 7fcb13e00640 1 --2- 172.21.15.40:0/2627646578 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5632c1975d40 0x5632c19b38d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a94286dff8663e6 server_cookie=3ad1770db07953ae in_seq=0 out_seq=0 2024-04-26T23:09:47.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.831+0000 7fcaff400640 10 client.9558 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:47.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.831+0000 7fcaff400640 1 -- 172.21.15.40:0/2627646578 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632c18e87e0 con 0x5632c1975d40 2024-04-26T23:09:47.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.831+0000 7fcaff400640 10 client.9558 handle_command_reply: tid=0 2024-04-26T23:09:47.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.831+0000 7fcb23fcf740 1 client.9558 shutdown 2024-04-26T23:09:47.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.832+0000 7fcb23fcf740 1 -- 172.21.15.40:0/2627646578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c19a1540 msgr2=0x5632c1975800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.832+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/2627646578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c19a1540 0x5632c1975800 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c013750 tx=0x7fcb0c013b50 comp rx=0 tx=0).stop 2024-04-26T23:09:47.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.832+0000 7fcb23fcf740 1 -- 172.21.15.40:0/2627646578 shutdown_connections 2024-04-26T23:09:47.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.832+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/2627646578 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5632c1975d40 0x5632c19b38d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.832+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/2627646578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c199c400 0x5632c1970cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.832+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/2627646578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19bede0 0x5632c19707b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.832+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/2627646578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c19a1540 0x5632c1975800 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.832+0000 7fcaff400640 0 client.9558 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:47.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.832+0000 7fcb23fcf740 1 -- 172.21.15.40:0/2627646578 >> 172.21.15.40:0/2627646578 conn(0x5632c1977220 msgr2=0x5632c19750f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.832+0000 7fcb23fcf740 1 -- 172.21.15.40:0/2627646578 shutdown_connections 2024-04-26T23:09:47.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.832+0000 7fcb23fcf740 1 -- 172.21.15.40:0/2627646578 wait complete. 2024-04-26T23:09:47.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.832+0000 7fcb23fcf740 20 client.9558 trim_cache size 0 max 16384 2024-04-26T23:09:47.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.851+0000 7fcb23fcf740 1 Processor -- start 2024-04-26T23:09:47.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.852+0000 7fcb23fcf740 1 -- start start 2024-04-26T23:09:47.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.852+0000 7fcb23fcf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 0x5632c19707b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.852+0000 7fcb23fcf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19a1540 0x5632c1970cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.852+0000 7fcb23fcf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19bede0 0x5632c19993f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.852+0000 7fcb23fcf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5632c199dfa0 con 0x5632c199c400 2024-04-26T23:09:47.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.852+0000 7fcb23fcf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c201d60 con 0x5632c19a1540 2024-04-26T23:09:47.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.852+0000 7fcb23fcf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcb1c200f00 con 0x5632c19bede0 2024-04-26T23:09:47.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.853+0000 7fcb19e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 0x5632c19707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.853+0000 7fcb19e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 0x5632c19707b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39920/0 (socket says 172.21.15.40:39920) 2024-04-26T23:09:47.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.853+0000 7fcb19e00640 1 -- 172.21.15.40:0/4021609732 learned_addr learned my addr 172.21.15.40:0/4021609732 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.853+0000 7fcb19e00640 1 -- 172.21.15.40:0/4021609732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19bede0 msgr2=0x5632c19993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.853+0000 7fcb19e00640 1 --2- 172.21.15.40:0/4021609732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19bede0 0x5632c19993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.853+0000 7fcb13e00640 1 --2- 172.21.15.40:0/4021609732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19a1540 0x5632c1970cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.853+0000 7fcb19e00640 1 -- 172.21.15.40:0/4021609732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19a1540 msgr2=0x5632c1970cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.853+0000 7fcb19e00640 1 --2- 172.21.15.40:0/4021609732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19a1540 0x5632c1970cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.853+0000 7fcb19e00640 1 -- 172.21.15.40:0/4021609732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632c15ac0f0 con 0x5632c199c400 2024-04-26T23:09:47.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.853+0000 7fcb13e00640 1 --2- 172.21.15.40:0/4021609732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19a1540 0x5632c1970cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:47.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.854+0000 7fcb19e00640 1 --2- 172.21.15.40:0/4021609732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 0x5632c19707b0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00d520 tx=0x7fcb0c005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6499b3fb3b8926fd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.854+0000 7fcb0a000640 1 -- 172.21.15.40:0/4021609732 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c0071e0 con 0x5632c199c400 2024-04-26T23:09:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.854+0000 7fcb0a000640 1 -- 172.21.15.40:0/4021609732 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb0c017070 con 0x5632c199c400 2024-04-26T23:09:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.854+0000 7fcb0a000640 1 -- 172.21.15.40:0/4021609732 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c012030 con 0x5632c199c400 2024-04-26T23:09:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.855+0000 7fcb23fcf740 1 -- 172.21.15.40:0/4021609732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 msgr2=0x5632c19707b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.855+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/4021609732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 0x5632c19707b0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00d520 tx=0x7fcb0c005000 comp rx=0 tx=0).stop 2024-04-26T23:09:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.855+0000 7fcb23fcf740 1 -- 172.21.15.40:0/4021609732 shutdown_connections 2024-04-26T23:09:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.855+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/4021609732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19bede0 0x5632c19993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.855+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/4021609732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19a1540 0x5632c1970cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.855+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/4021609732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 0x5632c19707b0 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.855+0000 7fcb23fcf740 1 -- 172.21.15.40:0/4021609732 >> 172.21.15.40:0/4021609732 conn(0x5632c1977220 msgr2=0x5632c19b38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.855+0000 7fcb23fcf740 1 -- 172.21.15.40:0/4021609732 shutdown_connections 2024-04-26T23:09:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.855+0000 7fcb23fcf740 1 -- 172.21.15.40:0/4021609732 wait complete. 2024-04-26T23:09:47.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.855+0000 7fcb23fcf740 1 -- start start 2024-04-26T23:09:47.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.856+0000 7fcb23fcf740 1 Processor -- start 2024-04-26T23:09:47.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb23fcf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 0x5632c19deaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb23fcf740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19a1540 0x5632c1974fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb23fcf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19bede0 0x5632c1975500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb23fcf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c0c51d0 con 0x5632c199c400 2024-04-26T23:09:47.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb23fcf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c201630 con 0x5632c19bede0 2024-04-26T23:09:47.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb23fcf740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5632c18e7fd0 con 0x5632c19a1540 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb23fcf740 10 client.0 _ll_register_callbacks cb 0x7fcb0c0013b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb19e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 0x5632c19deaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb19e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 0x5632c19deaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39936/0 (socket says 172.21.15.40:39936) 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb19e00640 1 -- 172.21.15.40:0/575783188 learned_addr learned my addr 172.21.15.40:0/575783188 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb19e00640 1 -- 172.21.15.40:0/575783188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19a1540 msgr2=0x5632c1974fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb13e00640 1 --2- 172.21.15.40:0/575783188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19a1540 0x5632c1974fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb19e00640 1 --2- 172.21.15.40:0/575783188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19a1540 0x5632c1974fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb19e00640 1 -- 172.21.15.40:0/575783188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19bede0 msgr2=0x5632c1975500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb19e00640 1 --2- 172.21.15.40:0/575783188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19bede0 0x5632c1975500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.857+0000 7fcb19e00640 1 -- 172.21.15.40:0/575783188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632c18a2a30 con 0x5632c199c400 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.858+0000 7fcb19e00640 1 --2- 172.21.15.40:0/575783188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 0x5632c19deaf0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c0136f0 tx=0x7fcb0c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e346c50aebfc52 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.858+0000 7fcaff400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.858+0000 7fcaff400640 1 -- 172.21.15.40:0/575783188 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c0047e0 con 0x5632c199c400 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.858+0000 7fcaff400640 1 -- 172.21.15.40:0/575783188 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fcb0c004980 con 0x5632c199c400 2024-04-26T23:09:47.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.858+0000 7fcb23fcf740 1 -- 172.21.15.40:0/575783188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb0c017070 con 0x5632c199c400 2024-04-26T23:09:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.858+0000 7fcaff400640 1 -- 172.21.15.40:0/575783188 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7fcb0c004be0 con 0x5632c199c400 2024-04-26T23:09:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.858+0000 7fcb23fcf740 10 client.9564 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.858+0000 7fcb23fcf740 1 -- 172.21.15.40:0/575783188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x5632c1970c80 con 0x5632c199c400 2024-04-26T23:09:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.859+0000 7fcaff400640 1 -- 172.21.15.40:0/575783188 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb0c017070 con 0x5632c199c400 2024-04-26T23:09:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.859+0000 7fcaff400640 1 -- 172.21.15.40:0/575783188 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7fcb0c03a020 con 0x5632c199c400 2024-04-26T23:09:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.859+0000 7fcb23fcf740 10 client.9564 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.859+0000 7fcb23fcf740 10 client.9564 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.859+0000 7fcb23fcf740 20 client.9564 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.859+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/575783188 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5632c19be9a0 0x5632c1a38980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.859+0000 7fcb23fcf740 4 client.9564 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:09:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.859+0000 7fcb23fcf740 1 -- 172.21.15.40:0/575783188 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5632c18e2650 con 0x5632c19be9a0 2024-04-26T23:09:47.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.859+0000 7fcb13e00640 1 --2- 172.21.15.40:0/575783188 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5632c19be9a0 0x5632c1a38980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:47.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcb13e00640 1 --2- 172.21.15.40:0/575783188 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5632c19be9a0 0x5632c1a38980 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40b1f91845ea91e5 server_cookie=7fb67fe44e34b0cc in_seq=0 out_seq=0 2024-04-26T23:09:47.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcaff400640 10 client.9564 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:47.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcaff400640 1 -- 172.21.15.40:0/575783188 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5632c18e2650 con 0x5632c19be9a0 2024-04-26T23:09:47.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcaff400640 10 client.9564 handle_command_reply: tid=0 2024-04-26T23:09:47.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcb23fcf740 1 client.9564 shutdown 2024-04-26T23:09:47.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcb23fcf740 1 -- 172.21.15.40:0/575783188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 msgr2=0x5632c19deaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/575783188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 0x5632c19deaf0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c0136f0 tx=0x7fcb0c013000 comp rx=0 tx=0).stop 2024-04-26T23:09:47.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcb23fcf740 1 -- 172.21.15.40:0/575783188 shutdown_connections 2024-04-26T23:09:47.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/575783188 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5632c19be9a0 0x5632c1a38980 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/575783188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5632c19bede0 0x5632c1975500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/575783188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5632c19a1540 0x5632c1974fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcb23fcf740 1 --2- 172.21.15.40:0/575783188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5632c199c400 0x5632c19deaf0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcaff400640 0 client.9564 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcb23fcf740 1 -- 172.21.15.40:0/575783188 >> 172.21.15.40:0/575783188 conn(0x5632c1977220 msgr2=0x5632c19bc250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.860+0000 7fcb23fcf740 1 -- 172.21.15.40:0/575783188 shutdown_connections 2024-04-26T23:09:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.861+0000 7fcb23fcf740 1 -- 172.21.15.40:0/575783188 wait complete. 2024-04-26T23:09:47.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.861+0000 7fcb23fcf740 20 client.9564 trim_cache size 0 max 16384 2024-04-26T23:09:47.864 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:47.864 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-26T23:09:47.864 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:09:47.864 INFO:teuthology.orchestra.run.smithi040.stdout: "a9827531-70a9-4d6f-b12a-2d71fa886d03": { 2024-04-26T23:09:47.864 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:09:47.864 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "a9827531-70a9-4d6f-b12a-2d71fa886d03", 2024-04-26T23:09:47.865 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:09:47.865 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:47.865 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:47.865 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:47.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.861+0000 7fcb21000640 1 -- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcaf806c8f0 msgr2=0x7fcaf806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.861+0000 7fcb21000640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcaf806c8f0 0x7fcaf806edb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c000f60 tx=0x7fcb0c009040 comp rx=0 tx=0).stop 2024-04-26T23:09:47.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.861+0000 7fcb21000640 1 -- 172.21.15.40:0/2344221897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb1c152c90 msgr2=0x7fcb1c153110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:47.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.861+0000 7fcb21000640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb1c152c90 0x7fcb1c153110 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009e30 tx=0x7fcb1400c490 comp rx=0 tx=0).stop 2024-04-26T23:09:47.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.861+0000 7fcb21000640 1 -- 172.21.15.40:0/2344221897 shutdown_connections 2024-04-26T23:09:47.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.861+0000 7fcb21000640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcaf806c8f0 0x7fcaf806edb0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.861+0000 7fcb21000640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb1c152c90 0x7fcb1c153110 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.861+0000 7fcb21000640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c15f380 0x7fcb1c1500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.861+0000 7fcb21000640 1 --2- 172.21.15.40:0/2344221897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c0cbb50 0x7fcb1c1f8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:47.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.861+0000 7fcb21000640 1 -- 172.21.15.40:0/2344221897 >> 172.21.15.40:0/2344221897 conn(0x7fcb1c0d0be0 msgr2=0x7fcb1c0aecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:47.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.862+0000 7fcb21000640 1 -- 172.21.15.40:0/2344221897 shutdown_connections 2024-04-26T23:09:47.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:47.862+0000 7fcb21000640 1 -- 172.21.15.40:0/2344221897 wait complete. 2024-04-26T23:09:47.880 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:09:47.880 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:48.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.092+0000 7fb443200640 1 Processor -- start 2024-04-26T23:09:48.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.092+0000 7fb443200640 1 -- start start 2024-04-26T23:09:48.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.094+0000 7fb443200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb44415f2d0 0x7fb4440afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:48.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.094+0000 7fb443200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4440cc590 0x7fb4440cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:48.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.094+0000 7fb443200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4440af7b0 con 0x7fb4440ce5e0 2024-04-26T23:09:48.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.094+0000 7fb443200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb4440afa90 con 0x7fb44415f2d0 2024-04-26T23:09:48.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.094+0000 7fb443200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb4440cbd90 con 0x7fb4440cc590 2024-04-26T23:09:48.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.094+0000 7fb441e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb44415f2d0 0x7fb4440afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:48.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.094+0000 7fb441400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb4440ce5e0 0x7fb4440c58b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35770/0 (socket says 172.21.15.40:35770) 2024-04-26T23:09:48.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.094+0000 7fb441400640 1 -- 172.21.15.40:0/1420867838 learned_addr learned my addr 172.21.15.40:0/1420867838 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:48.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.095+0000 7fb441e00640 1 -- 172.21.15.40:0/1420867838 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4440cc590 msgr2=0x7fb4440cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:48.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.095+0000 7fb442800640 1 --2- 172.21.15.40:0/1420867838 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4440cc590 0x7fb4440cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:48.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.095+0000 7fb440a00640 1 -- 172.21.15.40:0/1420867838 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3577360935 0 0) 0x7fb4440af7b0 con 0x7fb4440ce5e0 2024-04-26T23:09:48.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.095+0000 7fb441e00640 1 --2- 172.21.15.40:0/1420867838 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4440cc590 0x7fb4440cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.095+0000 7fb441e00640 1 -- 172.21.15.40:0/1420867838 >> v1:172.21.15.40:6789/0 conn(0x7fb4440ce5e0 legacy=0x7fb4440c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:48.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.095+0000 7fb441e00640 1 -- 172.21.15.40:0/1420867838 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4440cc070 con 0x7fb44415f2d0 2024-04-26T23:09:48.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.095+0000 7fb442800640 1 --2- 172.21.15.40:0/1420867838 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4440cc590 0x7fb4440cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:48.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.096+0000 7fb441e00640 1 --2- 172.21.15.40:0/1420867838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb44415f2d0 0x7fb4440afdf0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fb4340097a0 tx=0x7fb43402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9809b54ceace593 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:48.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.096+0000 7fb440a00640 1 -- 172.21.15.40:0/1420867838 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb434037070 con 0x7fb44415f2d0 2024-04-26T23:09:48.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.096+0000 7fb440a00640 1 -- 172.21.15.40:0/1420867838 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb434032440 con 0x7fb44415f2d0 2024-04-26T23:09:48.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.096+0000 7fb440a00640 1 -- 172.21.15.40:0/1420867838 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb434032810 con 0x7fb44415f2d0 2024-04-26T23:09:48.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.097+0000 7fb443200640 1 -- 172.21.15.40:0/1420867838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb44415f2d0 msgr2=0x7fb4440afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:48.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.097+0000 7fb443200640 1 --2- 172.21.15.40:0/1420867838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb44415f2d0 0x7fb4440afdf0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fb4340097a0 tx=0x7fb43402a180 comp rx=0 tx=0).stop 2024-04-26T23:09:48.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.097+0000 7fb443200640 1 -- 172.21.15.40:0/1420867838 shutdown_connections 2024-04-26T23:09:48.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.097+0000 7fb443200640 1 --2- 172.21.15.40:0/1420867838 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4440cc590 0x7fb4440cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.097+0000 7fb443200640 1 --2- 172.21.15.40:0/1420867838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb44415f2d0 0x7fb4440afdf0 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.097+0000 7fb443200640 1 -- 172.21.15.40:0/1420867838 >> 172.21.15.40:0/1420867838 conn(0x7fb4440d0a10 msgr2=0x7fb4440d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:48.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.097+0000 7fb443200640 1 -- 172.21.15.40:0/1420867838 shutdown_connections 2024-04-26T23:09:48.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.097+0000 7fb443200640 1 -- 172.21.15.40:0/1420867838 wait complete. 2024-04-26T23:09:48.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.099+0000 7fb443200640 1 Processor -- start 2024-04-26T23:09:48.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.099+0000 7fb443200640 1 -- start start 2024-04-26T23:09:48.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb443200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4440cc590 0x7fb4440b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb443200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4440ce5e0 0x7fb44415e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb443200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44415f2d0 0x7fb4441476e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb443200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb4440c5fc0 con 0x7fb4440cc590 2024-04-26T23:09:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb443200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb44406e270 con 0x7fb4440ce5e0 2024-04-26T23:09:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb443200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb444147c20 con 0x7fb44415f2d0 2024-04-26T23:09:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb441e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4440cc590 0x7fb4440b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb441e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4440cc590 0x7fb4440b63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39952/0 (socket says 172.21.15.40:39952) 2024-04-26T23:09:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb441e00640 1 -- 172.21.15.40:0/687882546 learned_addr learned my addr 172.21.15.40:0/687882546 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb441e00640 1 -- 172.21.15.40:0/687882546 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44415f2d0 msgr2=0x7fb4441476e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb441e00640 1 --2- 172.21.15.40:0/687882546 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44415f2d0 0x7fb4441476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb441e00640 1 -- 172.21.15.40:0/687882546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4440ce5e0 msgr2=0x7fb44415e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb441e00640 1 --2- 172.21.15.40:0/687882546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4440ce5e0 0x7fb44415e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.100+0000 7fb441e00640 1 -- 172.21.15.40:0/687882546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4440b5350 con 0x7fb4440cc590 2024-04-26T23:09:48.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.101+0000 7fb441e00640 1 --2- 172.21.15.40:0/687882546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4440cc590 0x7fb4440b63f0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fb434035040 tx=0x7fb4340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:48.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.101+0000 7fb43a000640 1 -- 172.21.15.40:0/687882546 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb434037040 con 0x7fb4440cc590 2024-04-26T23:09:48.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.101+0000 7fb43a000640 1 -- 172.21.15.40:0/687882546 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb434030770 con 0x7fb4440cc590 2024-04-26T23:09:48.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.101+0000 7fb43a000640 1 -- 172.21.15.40:0/687882546 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb434030b80 con 0x7fb4440cc590 2024-04-26T23:09:48.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.101+0000 7fb443200640 1 -- 172.21.15.40:0/687882546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4440b5630 con 0x7fb4440cc590 2024-04-26T23:09:48.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.101+0000 7fb443200640 1 -- 172.21.15.40:0/687882546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4440b5bf0 con 0x7fb4440cc590 2024-04-26T23:09:48.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.102+0000 7fb443200640 1 -- 172.21.15.40:0/687882546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb444069370 con 0x7fb4440cc590 2024-04-26T23:09:48.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.103+0000 7fb43a000640 1 -- 172.21.15.40:0/687882546 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb434044020 con 0x7fb4440cc590 2024-04-26T23:09:48.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.104+0000 7fb43a000640 1 --2- 172.21.15.40:0/687882546 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb42006c8f0 0x7fb42006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:48.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.104+0000 7fb43a000640 1 -- 172.21.15.40:0/687882546 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6101+0+0 (secure 0 0 0) 0x7fb4340ac390 con 0x7fb4440cc590 2024-04-26T23:09:48.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.105+0000 7fb441400640 1 --2- 172.21.15.40:0/687882546 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb42006c8f0 0x7fb42006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:48.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.106+0000 7fb441400640 1 --2- 172.21.15.40:0/687882546 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb42006c8f0 0x7fb42006edb0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb42c001560 tx=0x7fb42c007ca0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:48.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.109+0000 7fb43a000640 1 -- 172.21.15.40:0/687882546 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fb434032e20 con 0x7fb4440cc590 2024-04-26T23:09:48.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.361+0000 7fb443200640 1 -- 172.21.15.40:0/687882546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb44406de10 con 0x7fb4440cc590 2024-04-26T23:09:48.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.362+0000 7fb43a000640 1 -- 172.21.15.40:0/687882546 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5239 (secure 0 0 0) 0x7fb4340313a0 con 0x7fb4440cc590 2024-04-26T23:09:48.363 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:48.363 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:44.797258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[6598,8949,8955]},"id":7}]} 2024-04-26T23:09:48.363 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 122 2024-04-26T23:09:48.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.365+0000 7fb443200640 1 -- 172.21.15.40:0/687882546 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb42006c8f0 msgr2=0x7fb42006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:48.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.365+0000 7fb443200640 1 --2- 172.21.15.40:0/687882546 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb42006c8f0 0x7fb42006edb0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb42c001560 tx=0x7fb42c007ca0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.365+0000 7fb443200640 1 -- 172.21.15.40:0/687882546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4440cc590 msgr2=0x7fb4440b63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:48.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.365+0000 7fb443200640 1 --2- 172.21.15.40:0/687882546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4440cc590 0x7fb4440b63f0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fb434035040 tx=0x7fb4340046c0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.366+0000 7fb443200640 1 -- 172.21.15.40:0/687882546 shutdown_connections 2024-04-26T23:09:48.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.366+0000 7fb443200640 1 --2- 172.21.15.40:0/687882546 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb42006c8f0 0x7fb42006edb0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.366+0000 7fb443200640 1 --2- 172.21.15.40:0/687882546 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44415f2d0 0x7fb4441476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.366+0000 7fb443200640 1 --2- 172.21.15.40:0/687882546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4440ce5e0 0x7fb44415e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.366+0000 7fb443200640 1 --2- 172.21.15.40:0/687882546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4440cc590 0x7fb4440b63f0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.366+0000 7fb443200640 1 -- 172.21.15.40:0/687882546 >> 172.21.15.40:0/687882546 conn(0x7fb4440d0a10 msgr2=0x7fb4440c7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:48.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.366+0000 7fb443200640 1 -- 172.21.15.40:0/687882546 shutdown_connections 2024-04-26T23:09:48.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.366+0000 7fb443200640 1 -- 172.21.15.40:0/687882546 wait complete. 2024-04-26T23:09:48.386 DEBUG:tasks.cephfs.test_multimds_misc:current active=['a', 'c', 'd'] 2024-04-26T23:09:48.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:09:48.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.585+0000 7f6d03e00640 1 Processor -- start 2024-04-26T23:09:48.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.585+0000 7f6d03e00640 1 -- start start 2024-04-26T23:09:48.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.587+0000 7f6d03e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6d0415c160 0x7f6d040b4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:48.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.587+0000 7f6d03e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6d0415c5c0 0x7f6d040b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:48.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.587+0000 7f6d03e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0415cb90 con 0x7f6d04073e10 2024-04-26T23:09:48.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.587+0000 7f6d03e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f6d04153040 con 0x7f6d0415c160 2024-04-26T23:09:48.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.587+0000 7f6d03e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6d0415fc40 con 0x7f6d0415c5c0 2024-04-26T23:09:48.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.587+0000 7f6cfa000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6d0415c160 0x7f6d040b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:48.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.587+0000 7f6cfa000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6d0415c160 0x7f6d040b4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39954/0 (socket says 172.21.15.40:39954) 2024-04-26T23:09:48.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.587+0000 7f6cfa000640 1 -- 172.21.15.40:0/1248313367 learned_addr learned my addr 172.21.15.40:0/1248313367 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:48.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.587+0000 7f6cfa000640 1 -- 172.21.15.40:0/1248313367 >> v2:172.21.15.103:3300/0 conn(0x7f6d0415c5c0 msgr2=0x7f6d040b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:48.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.587+0000 7f6cfa000640 1 --2- 172.21.15.40:0/1248313367 >> v2:172.21.15.103:3300/0 conn(0x7f6d0415c5c0 0x7f6d040b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.588+0000 7f6cfa000640 1 -- 172.21.15.40:0/1248313367 >> v1:172.21.15.103:6789/0 conn(0x7f6d04073e10 legacy=0x7f6d041600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:48.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.588+0000 7f6cfa000640 1 -- 172.21.15.40:0/1248313367 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0415ce70 con 0x7f6d0415c160 2024-04-26T23:09:48.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.589+0000 7f6cfa000640 1 --2- 172.21.15.40:0/1248313367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d0415c160 0x7f6d040b4840 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f6d04159ca0 tx=0x7f6cec02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2691d10bc4592a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:48.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.589+0000 7f6d02000640 1 -- 172.21.15.40:0/1248313367 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec037070 con 0x7f6d0415c160 2024-04-26T23:09:48.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.589+0000 7f6d02000640 1 -- 172.21.15.40:0/1248313367 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cec032440 con 0x7f6d0415c160 2024-04-26T23:09:48.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.589+0000 7f6d02000640 1 -- 172.21.15.40:0/1248313367 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec032810 con 0x7f6d0415c160 2024-04-26T23:09:48.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.590+0000 7f6d03e00640 1 -- 172.21.15.40:0/1248313367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d0415c160 msgr2=0x7f6d040b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:48.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.590+0000 7f6d03e00640 1 --2- 172.21.15.40:0/1248313367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d0415c160 0x7f6d040b4840 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f6d04159ca0 tx=0x7f6cec02a230 comp rx=0 tx=0).stop 2024-04-26T23:09:48.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.590+0000 7f6d03e00640 1 -- 172.21.15.40:0/1248313367 shutdown_connections 2024-04-26T23:09:48.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.590+0000 7f6d03e00640 1 --2- 172.21.15.40:0/1248313367 >> v2:172.21.15.103:3300/0 conn(0x7f6d0415c5c0 0x7f6d040b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.590+0000 7f6d03e00640 1 --2- 172.21.15.40:0/1248313367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d0415c160 0x7f6d040b4840 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.590+0000 7f6d03e00640 1 -- 172.21.15.40:0/1248313367 >> 172.21.15.40:0/1248313367 conn(0x7f6d04075eb0 msgr2=0x7f6d04068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:48.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.590+0000 7f6d03e00640 1 -- 172.21.15.40:0/1248313367 shutdown_connections 2024-04-26T23:09:48.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.590+0000 7f6d03e00640 1 -- 172.21.15.40:0/1248313367 wait complete. 2024-04-26T23:09:48.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.592+0000 7f6d03e00640 1 Processor -- start 2024-04-26T23:09:48.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.592+0000 7f6d03e00640 1 -- start start 2024-04-26T23:09:48.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.593+0000 7f6d03e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d04073e10 0x7f6d04143b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:48.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.593+0000 7f6d03e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d0415c160 0x7f6d040cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:48.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.593+0000 7f6d03e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d0415c5c0 0x7f6d04159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:48.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.593+0000 7f6d03e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d04073540 con 0x7f6d0415c160 2024-04-26T23:09:48.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.593+0000 7f6d03e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6d040733c0 con 0x7f6d04073e10 2024-04-26T23:09:48.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.593+0000 7f6d03e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6d0415d150 con 0x7f6d0415c5c0 2024-04-26T23:09:48.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.593+0000 7f6cfa000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d0415c160 0x7f6d040cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:48.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.593+0000 7f6cfa000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d0415c160 0x7f6d040cdee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39968/0 (socket says 172.21.15.40:39968) 2024-04-26T23:09:48.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.593+0000 7f6cfa000640 1 -- 172.21.15.40:0/3855767828 learned_addr learned my addr 172.21.15.40:0/3855767828 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:48.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.594+0000 7f6d02a00640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d04073e10 0x7f6d04143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:48.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.594+0000 7f6cfa000640 1 -- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d0415c5c0 msgr2=0x7f6d04159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:48.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.594+0000 7f6cfa000640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d0415c5c0 0x7f6d04159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.594+0000 7f6cfa000640 1 -- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d04073e10 msgr2=0x7f6d04143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:48.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.594+0000 7f6cfa000640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d04073e10 0x7f6d04143b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.594+0000 7f6cfa000640 1 -- 172.21.15.40:0/3855767828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d04142e20 con 0x7f6d0415c160 2024-04-26T23:09:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.594+0000 7f6d02a00640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d04073e10 0x7f6d04143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.594+0000 7f6cfa000640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d0415c160 0x7f6d040cdee0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f6cec02a270 tx=0x7f6cec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.594+0000 7f6cfbe00640 1 -- 172.21.15.40:0/3855767828 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec041070 con 0x7f6d0415c160 2024-04-26T23:09:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.594+0000 7f6cfbe00640 1 -- 172.21.15.40:0/3855767828 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cec03c870 con 0x7f6d0415c160 2024-04-26T23:09:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.594+0000 7f6cfbe00640 1 -- 172.21.15.40:0/3855767828 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec037040 con 0x7f6d0415c160 2024-04-26T23:09:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.594+0000 7f6d03e00640 1 -- 172.21.15.40:0/3855767828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d04143100 con 0x7f6d0415c160 2024-04-26T23:09:48.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.594+0000 7f6d03e00640 1 -- 172.21.15.40:0/3855767828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d0415a4a0 con 0x7f6d0415c160 2024-04-26T23:09:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.596+0000 7f6d03e00640 1 -- 172.21.15.40:0/3855767828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d04069370 con 0x7f6d0415c160 2024-04-26T23:09:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.597+0000 7f6cfbe00640 1 -- 172.21.15.40:0/3855767828 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6cec044070 con 0x7f6d0415c160 2024-04-26T23:09:48.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.597+0000 7f6cfbe00640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6ce006c920 0x7f6ce006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:48.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.598+0000 7f6cfbe00640 1 -- 172.21.15.40:0/3855767828 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 6130+0+0 (secure 0 0 0) 0x7f6cec0ac8a0 con 0x7f6d0415c160 2024-04-26T23:09:48.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.598+0000 7f6d02a00640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6ce006c920 0x7f6ce006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:48.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.599+0000 7f6d02a00640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6ce006c920 0x7f6ce006ede0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4001dc0 tx=0x7f6cf4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:48.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.603+0000 7f6cfbe00640 1 -- 172.21.15.40:0/3855767828 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f6cec07ab10 con 0x7f6d0415c160 2024-04-26T23:09:48.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.891+0000 7f6d03e00640 1 -- 172.21.15.40:0/3855767828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f6d041678d0 con 0x7f6d0415c160 2024-04-26T23:09:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.892+0000 7f6cfbe00640 1 -- 172.21.15.40:0/3855767828 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1968 (secure 0 0 0) 0x7f6cec07a410 con 0x7f6d0415c160 2024-04-26T23:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":548,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":104,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":706274,"bytes_used":267182080,"bytes_avail":772826931200,"bytes_total":773094113280,"read_bytes_sec":0,"write_bytes_sec":5800,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":122,"id":7,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":7,"rank":2,"name":"d","status":"up:active","gid":6598},{"filesystem_id":7,"rank":1,"name":"c","status":"up:active","gid":8949},{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8955}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":29,"modified":"2024-04-26T23:09:43.338130+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:09:48.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.895+0000 7f6d03e00640 1 -- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6ce006c920 msgr2=0x7f6ce006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.895+0000 7f6d03e00640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6ce006c920 0x7f6ce006ede0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4001dc0 tx=0x7f6cf4009040 comp rx=0 tx=0).stop 2024-04-26T23:09:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.895+0000 7f6d03e00640 1 -- 172.21.15.40:0/3855767828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d0415c160 msgr2=0x7f6d040cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.895+0000 7f6d03e00640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d0415c160 0x7f6d040cdee0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f6cec02a270 tx=0x7f6cec0046c0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.896+0000 7f6d03e00640 1 -- 172.21.15.40:0/3855767828 shutdown_connections 2024-04-26T23:09:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.896+0000 7f6d03e00640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6ce006c920 0x7f6ce006ede0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.896+0000 7f6d03e00640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d0415c5c0 0x7f6d04159f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.896+0000 7f6d03e00640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d0415c160 0x7f6d040cdee0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.896+0000 7f6d03e00640 1 --2- 172.21.15.40:0/3855767828 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d04073e10 0x7f6d04143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.896+0000 7f6d03e00640 1 -- 172.21.15.40:0/3855767828 >> 172.21.15.40:0/3855767828 conn(0x7f6d04075eb0 msgr2=0x7f6d040b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.896+0000 7f6d03e00640 1 -- 172.21.15.40:0/3855767828 shutdown_connections 2024-04-26T23:09:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:48.896+0000 7f6d03e00640 1 -- 172.21.15.40:0/3855767828 wait complete. 2024-04-26T23:09:48.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'paused paths [/d1/d2/d3]', 'c': 'paused paths [/d1/d2/d3]', 'd': 'paused paths [/d1/d2/d3]'} 2024-04-26T23:09:48.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:09:48.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub resume 2024-04-26T23:09:49.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.123+0000 7f0a39000640 1 Processor -- start 2024-04-26T23:09:49.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.123+0000 7f0a39000640 1 -- start start 2024-04-26T23:09:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.124+0000 7f0a39000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0a34073a50 0x7f0a34072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.124+0000 7f0a39000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a34161880 con 0x7f0a34160a50 2024-04-26T23:09:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.124+0000 7f0a39000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a340cff80 con 0x7f0a340cfaf0 2024-04-26T23:09:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.124+0000 7f0a39000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f0a34071ce0 con 0x7f0a34073a50 2024-04-26T23:09:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.125+0000 7f0a33400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0a34160a50 0x7f0a34160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35774/0 (socket says 172.21.15.40:35774) 2024-04-26T23:09:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.125+0000 7f0a33400640 1 -- 172.21.15.40:0/2898440376 learned_addr learned my addr 172.21.15.40:0/2898440376 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:49.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.125+0000 7f0a33e00640 1 --2- 172.21.15.40:0/2898440376 >> v2:172.21.15.40:3300/0 conn(0x7f0a34073a50 0x7f0a34072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.125+0000 7f0a32000640 1 -- 172.21.15.40:0/2898440376 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1718597876 0 0) 0x7f0a34161880 con 0x7f0a34160a50 2024-04-26T23:09:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.125+0000 7f0a33e00640 1 -- 172.21.15.40:0/2898440376 >> v1:172.21.15.103:6789/0 conn(0x7f0a340cfaf0 legacy=0x7f0a3406dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.125+0000 7f0a33e00640 1 -- 172.21.15.40:0/2898440376 >> v1:172.21.15.40:6789/0 conn(0x7f0a34160a50 legacy=0x7f0a34160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.125+0000 7f0a33e00640 1 -- 172.21.15.40:0/2898440376 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a340d0260 con 0x7f0a34073a50 2024-04-26T23:09:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.126+0000 7f0a33e00640 1 --2- 172.21.15.40:0/2898440376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a34073a50 0x7f0a34072730 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0097a0 tx=0x7f0a2c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6300b04578683263 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.127+0000 7f0a32000640 1 -- 172.21.15.40:0/2898440376 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c037070 con 0x7f0a34073a50 2024-04-26T23:09:49.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.127+0000 7f0a32000640 1 -- 172.21.15.40:0/2898440376 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a2c032440 con 0x7f0a34073a50 2024-04-26T23:09:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.127+0000 7f0a32000640 1 -- 172.21.15.40:0/2898440376 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c032810 con 0x7f0a34073a50 2024-04-26T23:09:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.127+0000 7f0a39000640 1 -- 172.21.15.40:0/2898440376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a34073a50 msgr2=0x7f0a34072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.127+0000 7f0a39000640 1 --2- 172.21.15.40:0/2898440376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a34073a50 0x7f0a34072730 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0097a0 tx=0x7f0a2c02a230 comp rx=0 tx=0).stop 2024-04-26T23:09:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.127+0000 7f0a39000640 1 -- 172.21.15.40:0/2898440376 shutdown_connections 2024-04-26T23:09:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.127+0000 7f0a39000640 1 --2- 172.21.15.40:0/2898440376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a34073a50 0x7f0a34072730 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.127+0000 7f0a39000640 1 -- 172.21.15.40:0/2898440376 >> 172.21.15.40:0/2898440376 conn(0x7f0a340b2720 msgr2=0x7f0a340b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.127+0000 7f0a39000640 1 -- 172.21.15.40:0/2898440376 shutdown_connections 2024-04-26T23:09:49.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.127+0000 7f0a39000640 1 -- 172.21.15.40:0/2898440376 wait complete. 2024-04-26T23:09:49.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.129+0000 7f0a39000640 1 Processor -- start 2024-04-26T23:09:49.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.129+0000 7f0a39000640 1 -- start start 2024-04-26T23:09:49.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.130+0000 7f0a39000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a34073a50 0x7f0a3413a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.130+0000 7f0a39000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a34160a50 0x7f0a3413c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.130+0000 7f0a39000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a3413c940 0x7f0a34144a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.130+0000 7f0a39000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0a34073f00 con 0x7f0a34073a50 2024-04-26T23:09:49.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.130+0000 7f0a39000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a340722a0 con 0x7f0a34160a50 2024-04-26T23:09:49.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.130+0000 7f0a39000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0a341468f0 con 0x7f0a3413c940 2024-04-26T23:09:49.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.131+0000 7f0a33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a34073a50 0x7f0a3413a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.131+0000 7f0a33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a34073a50 0x7f0a3413a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39990/0 (socket says 172.21.15.40:39990) 2024-04-26T23:09:49.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.131+0000 7f0a33400640 1 -- 172.21.15.40:0/1892654656 learned_addr learned my addr 172.21.15.40:0/1892654656 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.131+0000 7f0a32a00640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a34160a50 0x7f0a3413c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.131+0000 7f0a33e00640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a3413c940 0x7f0a34144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.131+0000 7f0a33400640 1 -- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a3413c940 msgr2=0x7f0a34144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.131+0000 7f0a33400640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a3413c940 0x7f0a34144a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.131+0000 7f0a33400640 1 -- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a34160a50 msgr2=0x7f0a3413c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.131+0000 7f0a33400640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a34160a50 0x7f0a3413c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.131+0000 7f0a33400640 1 -- 172.21.15.40:0/1892654656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3413cf50 con 0x7f0a34073a50 2024-04-26T23:09:49.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.132+0000 7f0a33400640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a34073a50 0x7f0a3413a6c0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f0a2400c780 tx=0x7f0a2400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.132+0000 7f0a2b400640 1 -- 172.21.15.40:0/1892654656 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24018070 con 0x7f0a34073a50 2024-04-26T23:09:49.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.132+0000 7f0a2b400640 1 -- 172.21.15.40:0/1892654656 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a2400a640 con 0x7f0a34073a50 2024-04-26T23:09:49.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.132+0000 7f0a2b400640 1 -- 172.21.15.40:0/1892654656 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2400aa70 con 0x7f0a34073a50 2024-04-26T23:09:49.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.132+0000 7f0a39000640 1 -- 172.21.15.40:0/1892654656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a340b34f0 con 0x7f0a34073a50 2024-04-26T23:09:49.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.132+0000 7f0a39000640 1 -- 172.21.15.40:0/1892654656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a3413d230 con 0x7f0a34073a50 2024-04-26T23:09:49.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.135+0000 7f0a2b400640 1 -- 172.21.15.40:0/1892654656 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0a2400ac10 con 0x7f0a34073a50 2024-04-26T23:09:49.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.136+0000 7f0a2b400640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0a1006c920 0x7f0a1006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.136+0000 7f0a2b400640 1 -- 172.21.15.40:0/1892654656 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 6130+0+0 (secure 0 0 0) 0x7f0a2408b380 con 0x7f0a34073a50 2024-04-26T23:09:49.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.136+0000 7f0a3d43c740 1 Processor -- start 2024-04-26T23:09:49.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.136+0000 7f0a3d43c740 1 -- start start 2024-04-26T23:09:49.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.136+0000 7f0a32a00640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0a1006c920 0x7f0a1006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.137+0000 7f0a3d43c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb71af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.137+0000 7f0a3d43c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bb8f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.137+0000 7f0a3d43c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb473f0 0x56134bb8d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.137+0000 7f0a3d43c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56134ba57030 con 0x56134bb71710 2024-04-26T23:09:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.137+0000 7f0a3d43c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56134ba727f0 con 0x56134bb8efb0 2024-04-26T23:09:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.137+0000 7f0a3d43c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x56134ba2d7e0 con 0x56134bb473f0 2024-04-26T23:09:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.138+0000 7f0a33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb71af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.138+0000 7f0a33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb71af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39996/0 (socket says 172.21.15.40:39996) 2024-04-26T23:09:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.138+0000 7f0a33400640 1 -- 172.21.15.40:0/1532097336 learned_addr learned my addr 172.21.15.40:0/1532097336 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.138+0000 7f0a32a00640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0a1006c920 0x7f0a1006ede0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0a3415ae00 tx=0x7f0a1c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.138+0000 7f0a32a00640 1 --2- 172.21.15.40:0/1532097336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bb8f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.138+0000 7f0a33e00640 1 --2- 172.21.15.40:0/1532097336 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb473f0 0x56134bb8d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.138+0000 7f0a33400640 1 -- 172.21.15.40:0/1532097336 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb473f0 msgr2=0x56134bb8d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.138+0000 7f0a33400640 1 --2- 172.21.15.40:0/1532097336 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb473f0 0x56134bb8d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.138+0000 7f0a33400640 1 -- 172.21.15.40:0/1532097336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 msgr2=0x56134bb8f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.138+0000 7f0a33400640 1 --2- 172.21.15.40:0/1532097336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bb8f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.138+0000 7f0a33400640 1 -- 172.21.15.40:0/1532097336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56134ba39050 con 0x56134bb71710 2024-04-26T23:09:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.138+0000 7f0a33400640 1 --2- 172.21.15.40:0/1532097336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb71af0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f0a24004fd0 tx=0x7f0a240594d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102dee3fae31af1a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.139+0000 7f0a23e00640 1 -- 172.21.15.40:0/1532097336 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2405c040 con 0x56134bb71710 2024-04-26T23:09:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.139+0000 7f0a23e00640 1 -- 172.21.15.40:0/1532097336 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a24061a40 con 0x56134bb71710 2024-04-26T23:09:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.139+0000 7f0a23e00640 1 -- 172.21.15.40:0/1532097336 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2408b070 con 0x56134bb71710 2024-04-26T23:09:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.139+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1532097336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 msgr2=0x56134bb71af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.139+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1532097336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb71af0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f0a24004fd0 tx=0x7f0a240594d0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.139+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1532097336 shutdown_connections 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.139+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1532097336 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb473f0 0x56134bb8d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.139+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1532097336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bb8f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.139+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1532097336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb71af0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.139+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1532097336 >> 172.21.15.40:0/1532097336 conn(0x56134bb43d20 msgr2=0x56134bb8c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.140+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1532097336 shutdown_connections 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.140+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1532097336 wait complete. 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.140+0000 7f0a3d43c740 1 -- start start 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.141+0000 7f0a3d43c740 1 Processor -- start 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a3d43c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb64040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a3d43c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bb748f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a3d43c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb84f00 0x56134bb74e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a3d43c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56134baaf200 con 0x56134bb71710 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a3d43c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56134baa9ee0 con 0x56134bb8efb0 2024-04-26T23:09:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a3d43c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x56134bab6e70 con 0x56134bb84f00 2024-04-26T23:09:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a3d43c740 10 client.0 _ll_register_callbacks cb 0x56134baa0270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb64040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb64040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40002/0 (socket says 172.21.15.40:40002) 2024-04-26T23:09:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a33400640 1 -- 172.21.15.40:0/1388088037 learned_addr learned my addr 172.21.15.40:0/1388088037 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a33400640 1 -- 172.21.15.40:0/1388088037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb84f00 msgr2=0x56134bb74e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a33400640 1 --2- 172.21.15.40:0/1388088037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb84f00 0x56134bb74e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a32a00640 1 --2- 172.21.15.40:0/1388088037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bb748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a33400640 1 -- 172.21.15.40:0/1388088037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 msgr2=0x56134bb748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a33400640 1 --2- 172.21.15.40:0/1388088037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bb748f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.142+0000 7f0a33400640 1 -- 172.21.15.40:0/1388088037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56134b8867d0 con 0x56134bb71710 2024-04-26T23:09:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.143+0000 7f0a33400640 1 --2- 172.21.15.40:0/1388088037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb64040 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f0a24059680 tx=0x7f0a24014070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9cf8be5aa989265 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.143+0000 7f0a20c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.143+0000 7f0a20c00640 1 -- 172.21.15.40:0/1388088037 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2405c040 con 0x56134bb71710 2024-04-26T23:09:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.143+0000 7f0a20c00640 1 -- 172.21.15.40:0/1388088037 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 6130+0+0 (secure 0 0 0) 0x7f0a2408b070 con 0x56134bb71710 2024-04-26T23:09:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.143+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1388088037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a24061a40 con 0x56134bb71710 2024-04-26T23:09:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.143+0000 7f0a20c00640 1 -- 172.21.15.40:0/1388088037 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f0a24080050 con 0x56134bb71710 2024-04-26T23:09:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.143+0000 7f0a3d43c740 10 client.9594 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.143+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1388088037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x56134b7bc3f0 con 0x56134bb71710 2024-04-26T23:09:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.143+0000 7f0a20c00640 1 -- 172.21.15.40:0/1388088037 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a24061a40 con 0x56134bb71710 2024-04-26T23:09:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.143+0000 7f0a20c00640 1 -- 172.21.15.40:0/1388088037 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7f0a24078220 con 0x56134bb71710 2024-04-26T23:09:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.144+0000 7f0a3d43c740 10 client.9594 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.144+0000 7f0a3d43c740 10 client.9594 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.144+0000 7f0a3d43c740 20 client.9594 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.144+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1388088037 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x56134bb9c590 0x56134bb835d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.144+0000 7f0a3d43c740 4 client.9594 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.144+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1388088037 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56134bab87e0 con 0x56134bb9c590 2024-04-26T23:09:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.144+0000 7f0a32a00640 1 --2- 172.21.15.40:0/1388088037 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x56134bb9c590 0x56134bb835d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.145+0000 7f0a32a00640 1 --2- 172.21.15.40:0/1388088037 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x56134bb9c590 0x56134bb835d0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fa99b2cf16f4423 server_cookie=2b5ca335704c3dbe in_seq=0 out_seq=0 2024-04-26T23:09:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.145+0000 7f0a20c00640 10 client.9594 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.145+0000 7f0a20c00640 1 -- 172.21.15.40:0/1388088037 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56134bab87e0 con 0x56134bb9c590 2024-04-26T23:09:49.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.145+0000 7f0a20c00640 10 client.9594 handle_command_reply: tid=0 2024-04-26T23:09:49.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.145+0000 7f0a3d43c740 1 client.9594 shutdown 2024-04-26T23:09:49.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.146+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1388088037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 msgr2=0x56134bb64040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.146+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1388088037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb64040 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f0a24059680 tx=0x7f0a24014070 comp rx=0 tx=0).stop 2024-04-26T23:09:49.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.146+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1388088037 shutdown_connections 2024-04-26T23:09:49.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.146+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1388088037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb84f00 0x56134bb74e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.146+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1388088037 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x56134bb9c590 0x56134bb835d0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.146+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1388088037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bb748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.146+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1388088037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb64040 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.146+0000 7f0a20c00640 0 client.9594 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:49.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.146+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1388088037 >> 172.21.15.40:0/1388088037 conn(0x56134bb473f0 msgr2=0x56134bb411f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.146+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1388088037 shutdown_connections 2024-04-26T23:09:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.146+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1388088037 wait complete. 2024-04-26T23:09:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.146+0000 7f0a3d43c740 20 client.9594 trim_cache size 0 max 16384 2024-04-26T23:09:49.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.165+0000 7f0a3d43c740 1 Processor -- start 2024-04-26T23:09:49.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.165+0000 7f0a3d43c740 1 -- start start 2024-04-26T23:09:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.166+0000 7f0a3d43c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb743e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.166+0000 7f0a3d43c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb84f00 0x56134bb41d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.166+0000 7f0a3d43c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bc00490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.166+0000 7f0a3d43c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0a34201da0 con 0x56134bb71710 2024-04-26T23:09:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.166+0000 7f0a3d43c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a34200f40 con 0x56134bb8efb0 2024-04-26T23:09:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.166+0000 7f0a3d43c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0a34201670 con 0x56134bb84f00 2024-04-26T23:09:49.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.167+0000 7f0a33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.167+0000 7f0a33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb743e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40012/0 (socket says 172.21.15.40:40012) 2024-04-26T23:09:49.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.167+0000 7f0a33400640 1 -- 172.21.15.40:0/1882061077 learned_addr learned my addr 172.21.15.40:0/1882061077 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.167+0000 7f0a33e00640 1 --2- 172.21.15.40:0/1882061077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bc00490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.167+0000 7f0a32a00640 1 --2- 172.21.15.40:0/1882061077 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb84f00 0x56134bb41d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.167+0000 7f0a33400640 1 -- 172.21.15.40:0/1882061077 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb84f00 msgr2=0x56134bb41d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.167+0000 7f0a33400640 1 --2- 172.21.15.40:0/1882061077 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb84f00 0x56134bb41d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.167+0000 7f0a33400640 1 -- 172.21.15.40:0/1882061077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 msgr2=0x56134bc00490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.167+0000 7f0a33400640 1 --2- 172.21.15.40:0/1882061077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bc00490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.167+0000 7f0a33400640 1 -- 172.21.15.40:0/1882061077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56134b77c0f0 con 0x56134bb71710 2024-04-26T23:09:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.168+0000 7f0a33400640 1 --2- 172.21.15.40:0/1882061077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb743e0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f0a24058f90 tx=0x7f0a24087000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927e7e3393273732 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.168+0000 7f0a23400640 1 -- 172.21.15.40:0/1882061077 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2405c040 con 0x56134bb71710 2024-04-26T23:09:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.168+0000 7f0a23400640 1 -- 172.21.15.40:0/1882061077 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a24078070 con 0x56134bb71710 2024-04-26T23:09:49.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.168+0000 7f0a23400640 1 -- 172.21.15.40:0/1882061077 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24089370 con 0x56134bb71710 2024-04-26T23:09:49.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.169+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1882061077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 msgr2=0x56134bb743e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.169+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1882061077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb743e0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f0a24058f90 tx=0x7f0a24087000 comp rx=0 tx=0).stop 2024-04-26T23:09:49.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.169+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1882061077 shutdown_connections 2024-04-26T23:09:49.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.169+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1882061077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bc00490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.169+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1882061077 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb84f00 0x56134bb41d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.169+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/1882061077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bb743e0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.169+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1882061077 >> 172.21.15.40:0/1882061077 conn(0x56134bb473f0 msgr2=0x56134bb633c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.169+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1882061077 shutdown_connections 2024-04-26T23:09:49.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.169+0000 7f0a3d43c740 1 -- 172.21.15.40:0/1882061077 wait complete. 2024-04-26T23:09:49.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.170+0000 7f0a3d43c740 1 -- start start 2024-04-26T23:09:49.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.170+0000 7f0a3d43c740 1 Processor -- start 2024-04-26T23:09:49.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a3d43c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bc028e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a3d43c740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb84f00 0x56134bb743e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a3d43c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bb74920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a3d43c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56134bab7fd0 con 0x56134bb71710 2024-04-26T23:09:49.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a3d43c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3415f830 con 0x56134bb8efb0 2024-04-26T23:09:49.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a3d43c740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x56134bb41620 con 0x56134bb84f00 2024-04-26T23:09:49.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a3d43c740 10 client.0 _ll_register_callbacks cb 0x7f0a24058c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:49.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bc028e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bc028e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40022/0 (socket says 172.21.15.40:40022) 2024-04-26T23:09:49.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a33400640 1 -- 172.21.15.40:0/4174030266 learned_addr learned my addr 172.21.15.40:0/4174030266 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a33400640 1 -- 172.21.15.40:0/4174030266 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb84f00 msgr2=0x56134bb743e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a33400640 1 --2- 172.21.15.40:0/4174030266 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb84f00 0x56134bb743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a33400640 1 -- 172.21.15.40:0/4174030266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 msgr2=0x56134bb74920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a33400640 1 --2- 172.21.15.40:0/4174030266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bb74920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.171+0000 7f0a33400640 1 -- 172.21.15.40:0/4174030266 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56134ba72a30 con 0x56134bb71710 2024-04-26T23:09:49.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.172+0000 7f0a33400640 1 --2- 172.21.15.40:0/4174030266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bc028e0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f0a24059680 tx=0x7f0a2400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83aaf4e20abcc1ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.172+0000 7f0a20c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:49.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.172+0000 7f0a20c00640 1 -- 172.21.15.40:0/4174030266 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2405c070 con 0x56134bb71710 2024-04-26T23:09:49.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.172+0000 7f0a20c00640 1 -- 172.21.15.40:0/4174030266 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 6130+0+0 (secure 0 0 0) 0x7f0a24058220 con 0x56134bb71710 2024-04-26T23:09:49.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.172+0000 7f0a3d43c740 1 -- 172.21.15.40:0/4174030266 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a24078070 con 0x56134bb71710 2024-04-26T23:09:49.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.172+0000 7f0a20c00640 1 -- 172.21.15.40:0/4174030266 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f0a24065050 con 0x56134bb71710 2024-04-26T23:09:49.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.172+0000 7f0a3d43c740 10 client.9600 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:49.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.172+0000 7f0a3d43c740 1 -- 172.21.15.40:0/4174030266 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x56134ba72a30 con 0x56134bb71710 2024-04-26T23:09:49.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.172+0000 7f0a20c00640 1 -- 172.21.15.40:0/4174030266 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a24078070 con 0x56134bb71710 2024-04-26T23:09:49.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.172+0000 7f0a20c00640 1 -- 172.21.15.40:0/4174030266 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7f0a24068020 con 0x56134bb71710 2024-04-26T23:09:49.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.173+0000 7f0a3d43c740 10 client.9600 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:49.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.173+0000 7f0a3d43c740 10 client.9600 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:49.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.173+0000 7f0a3d43c740 20 client.9600 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:49.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.173+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/4174030266 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x56134bc06010 0x56134bb81eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.173+0000 7f0a3d43c740 4 client.9600 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "scrub resume"}] 2024-04-26T23:09:49.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.173+0000 7f0a3d43c740 1 -- 172.21.15.40:0/4174030266 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "scrub resume"}) -- 0x56134bb42b40 con 0x56134bc06010 2024-04-26T23:09:49.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.173+0000 7f0a32a00640 1 --2- 172.21.15.40:0/4174030266 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x56134bc06010 0x56134bb81eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.173+0000 7f0a32a00640 1 --2- 172.21.15.40:0/4174030266 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x56134bc06010 0x56134bb81eb0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=607a3f2e71ac7880 server_cookie=fbe82a931ee99dab in_seq=0 out_seq=0 2024-04-26T23:09:49.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.173+0000 7f0a20c00640 10 client.9600 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:49.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a20c00640 1 -- 172.21.15.40:0/4174030266 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x56134bb42b40 con 0x56134bc06010 2024-04-26T23:09:49.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a20c00640 10 client.9600 handle_command_reply: tid=0 2024-04-26T23:09:49.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a3d43c740 1 client.9600 shutdown 2024-04-26T23:09:49.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a3d43c740 1 -- 172.21.15.40:0/4174030266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 msgr2=0x56134bc028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/4174030266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bc028e0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f0a24059680 tx=0x7f0a2400cf40 comp rx=0 tx=0).stop 2024-04-26T23:09:49.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a3d43c740 1 -- 172.21.15.40:0/4174030266 shutdown_connections 2024-04-26T23:09:49.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/4174030266 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x56134bc06010 0x56134bb81eb0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/4174030266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56134bb8efb0 0x56134bb74920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/4174030266 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56134bb84f00 0x56134bb743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a3d43c740 1 --2- 172.21.15.40:0/4174030266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56134bb71710 0x56134bc028e0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a20c00640 0 client.9600 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:49.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a3d43c740 1 -- 172.21.15.40:0/4174030266 >> 172.21.15.40:0/4174030266 conn(0x56134bb473f0 msgr2=0x56134bb41d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a3d43c740 1 -- 172.21.15.40:0/4174030266 shutdown_connections 2024-04-26T23:09:49.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.174+0000 7f0a3d43c740 1 -- 172.21.15.40:0/4174030266 wait complete. 2024-04-26T23:09:49.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.175+0000 7f0a3d43c740 20 client.9600 trim_cache size 0 max 16384 2024-04-26T23:09:49.181 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0 2024-04-26T23:09:49.181 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:49.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.175+0000 7f0a39000640 1 -- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0a1006c920 msgr2=0x7f0a1006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.175+0000 7f0a39000640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0a1006c920 0x7f0a1006ede0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0a3415ae00 tx=0x7f0a1c009040 comp rx=0 tx=0).stop 2024-04-26T23:09:49.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.175+0000 7f0a39000640 1 -- 172.21.15.40:0/1892654656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a34073a50 msgr2=0x7f0a3413a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.175+0000 7f0a39000640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a34073a50 0x7f0a3413a6c0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f0a2400c780 tx=0x7f0a2400cc40 comp rx=0 tx=0).stop 2024-04-26T23:09:49.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.175+0000 7f0a39000640 1 -- 172.21.15.40:0/1892654656 shutdown_connections 2024-04-26T23:09:49.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.175+0000 7f0a39000640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0a1006c920 0x7f0a1006ede0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.175+0000 7f0a39000640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a3413c940 0x7f0a34144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.175+0000 7f0a39000640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a34160a50 0x7f0a3413c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.175+0000 7f0a39000640 1 --2- 172.21.15.40:0/1892654656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a34073a50 0x7f0a3413a6c0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.175+0000 7f0a39000640 1 -- 172.21.15.40:0/1892654656 >> 172.21.15.40:0/1892654656 conn(0x7f0a340b2720 msgr2=0x7f0a3415d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.175+0000 7f0a39000640 1 -- 172.21.15.40:0/1892654656 shutdown_connections 2024-04-26T23:09:49.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.175+0000 7f0a39000640 1 -- 172.21.15.40:0/1892654656 wait complete. 2024-04-26T23:09:49.195 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:09:49.195 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.7:1 scrub status 2024-04-26T23:09:49.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:09:49.371 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:09:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.402+0000 7f490d600640 1 Processor -- start 2024-04-26T23:09:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.402+0000 7f490d600640 1 -- start start 2024-04-26T23:09:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.403+0000 7f490d600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4908071e60 0x7f490806f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.403+0000 7f490d600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f490806fe20 0x7f4908072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.403+0000 7f490d600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f490815d7c0 con 0x7f4908072780 2024-04-26T23:09:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.403+0000 7f490d600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f490815daa0 con 0x7f490806fe20 2024-04-26T23:09:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.403+0000 7f490d600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f490815dd80 con 0x7f4908071e60 2024-04-26T23:09:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.404+0000 7f4906000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f490806fe20 0x7f4908072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.404+0000 7f4906000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f490806fe20 0x7f4908072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40024/0 (socket says 172.21.15.40:40024) 2024-04-26T23:09:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.404+0000 7f4906000640 1 -- 172.21.15.40:0/2928036568 learned_addr learned my addr 172.21.15.40:0/2928036568 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.404+0000 7f4907400640 1 --1- 172.21.15.40:0/2928036568 >> v1:172.21.15.40:6789/0 conn(0x7f4908072780 0x7f49080cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35776/0 (socket says 172.21.15.40:35776) 2024-04-26T23:09:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.404+0000 7f4906a00640 1 --2- 172.21.15.40:0/2928036568 >> v2:172.21.15.103:3300/0 conn(0x7f4908071e60 0x7f490806f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.404+0000 7f4906000640 1 -- 172.21.15.40:0/2928036568 >> v2:172.21.15.103:3300/0 conn(0x7f4908071e60 msgr2=0x7f490806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.404+0000 7f4906000640 1 --2- 172.21.15.40:0/2928036568 >> v2:172.21.15.103:3300/0 conn(0x7f4908071e60 0x7f490806f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.404+0000 7f4906000640 1 -- 172.21.15.40:0/2928036568 >> v1:172.21.15.40:6789/0 conn(0x7f4908072780 legacy=0x7f49080cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.404+0000 7f4906000640 1 -- 172.21.15.40:0/2928036568 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f490815e060 con 0x7f490806fe20 2024-04-26T23:09:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.404+0000 7f4906a00640 1 --2- 172.21.15.40:0/2928036568 >> v2:172.21.15.103:3300/0 conn(0x7f4908071e60 0x7f490806f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:09:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.405+0000 7f4906000640 1 --2- 172.21.15.40:0/2928036568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490806fe20 0x7f4908072240 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f49080b3700 tx=0x7f48f402a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353d04141cba23b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.406+0000 7f4905600640 1 -- 172.21.15.40:0/2928036568 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f4037070 con 0x7f490806fe20 2024-04-26T23:09:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.406+0000 7f4905600640 1 -- 172.21.15.40:0/2928036568 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48f4032440 con 0x7f490806fe20 2024-04-26T23:09:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.406+0000 7f4905600640 1 -- 172.21.15.40:0/2928036568 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f4032810 con 0x7f490806fe20 2024-04-26T23:09:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.406+0000 7f490d600640 1 -- 172.21.15.40:0/2928036568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490806fe20 msgr2=0x7f4908072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.406+0000 7f490d600640 1 --2- 172.21.15.40:0/2928036568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490806fe20 0x7f4908072240 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f49080b3700 tx=0x7f48f402a280 comp rx=0 tx=0).stop 2024-04-26T23:09:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.407+0000 7f490d600640 1 -- 172.21.15.40:0/2928036568 shutdown_connections 2024-04-26T23:09:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.407+0000 7f490d600640 1 --2- 172.21.15.40:0/2928036568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490806fe20 0x7f4908072240 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.407+0000 7f490d600640 1 --2- 172.21.15.40:0/2928036568 >> v2:172.21.15.103:3300/0 conn(0x7f4908071e60 0x7f490806f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.407+0000 7f490d600640 1 -- 172.21.15.40:0/2928036568 >> 172.21.15.40:0/2928036568 conn(0x7f49080d0be0 msgr2=0x7f49080d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.407+0000 7f490d600640 1 -- 172.21.15.40:0/2928036568 shutdown_connections 2024-04-26T23:09:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.407+0000 7f490d600640 1 -- 172.21.15.40:0/2928036568 wait complete. 2024-04-26T23:09:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.409+0000 7f490d600640 1 Processor -- start 2024-04-26T23:09:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.409+0000 7f490d600640 1 -- start start 2024-04-26T23:09:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.410+0000 7f490d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490806fe20 0x7f49080afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.410+0000 7f490d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4908071e60 0x7f4908150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.410+0000 7f490d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4908072780 0x7f49080c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.410+0000 7f490d600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f490815d3f0 con 0x7f4908071e60 2024-04-26T23:09:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.410+0000 7f490d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4908072bd0 con 0x7f490806fe20 2024-04-26T23:09:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.410+0000 7f490d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f490815d570 con 0x7f4908072780 2024-04-26T23:09:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.410+0000 7f4906000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4908071e60 0x7f4908150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.410+0000 7f4906000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4908071e60 0x7f4908150f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40028/0 (socket says 172.21.15.40:40028) 2024-04-26T23:09:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.410+0000 7f4906000640 1 -- 172.21.15.40:0/2208471082 learned_addr learned my addr 172.21.15.40:0/2208471082 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.410+0000 7f4906a00640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490806fe20 0x7f49080afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.411+0000 7f4907400640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4908072780 0x7f49080c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.411+0000 7f4906000640 1 -- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4908072780 msgr2=0x7f49080c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.411+0000 7f4906000640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4908072780 0x7f49080c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.411+0000 7f4906000640 1 -- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490806fe20 msgr2=0x7f49080afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.411+0000 7f4906000640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490806fe20 0x7f49080afd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.411+0000 7f4906000640 1 -- 172.21.15.40:0/2208471082 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49080aecc0 con 0x7f4908071e60 2024-04-26T23:09:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.411+0000 7f4907400640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4908072780 0x7f49080c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.411+0000 7f4906000640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4908071e60 0x7f4908150f50 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f48f402a2c0 tx=0x7f48f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.412+0000 7f48fea00640 1 -- 172.21.15.40:0/2208471082 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f4041020 con 0x7f4908071e60 2024-04-26T23:09:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.412+0000 7f48fea00640 1 -- 172.21.15.40:0/2208471082 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48f40306c0 con 0x7f4908071e60 2024-04-26T23:09:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.412+0000 7f48fea00640 1 -- 172.21.15.40:0/2208471082 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f4037040 con 0x7f4908071e60 2024-04-26T23:09:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.412+0000 7f490d600640 1 -- 172.21.15.40:0/2208471082 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49080aefa0 con 0x7f4908071e60 2024-04-26T23:09:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.412+0000 7f490d600640 1 -- 172.21.15.40:0/2208471082 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49080af560 con 0x7f4908071e60 2024-04-26T23:09:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.415+0000 7f48fea00640 1 -- 172.21.15.40:0/2208471082 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f48f4044020 con 0x7f4908071e60 2024-04-26T23:09:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.416+0000 7f48fea00640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48e806c8f0 0x7f48e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.416+0000 7f48fea00640 1 -- 172.21.15.40:0/2208471082 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 6130+0+0 (secure 0 0 0) 0x7f48f40ac4f0 con 0x7f4908071e60 2024-04-26T23:09:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.416+0000 7f49103c1740 1 Processor -- start 2024-04-26T23:09:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.416+0000 7f49103c1740 1 -- start start 2024-04-26T23:09:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.416+0000 7f4906a00640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48e806c8f0 0x7f48e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.417+0000 7f49103c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8bd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.417+0000 7f49103c1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8dade0 0x55919a8db330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.417+0000 7f4906a00640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48e806c8f0 0x7f48e806edb0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f48f8006fd0 tx=0x7f48f8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.417+0000 7f49103c1740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a893220 0x55919a8d94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.417+0000 7f49103c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55919a7a3030 con 0x55919a8bd540 2024-04-26T23:09:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.417+0000 7f49103c1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55919a7be7f0 con 0x55919a8dade0 2024-04-26T23:09:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.417+0000 7f49103c1740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55919a7797e0 con 0x55919a893220 2024-04-26T23:09:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.418+0000 7f4906a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8bd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.418+0000 7f4906a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8bd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40044/0 (socket says 172.21.15.40:40044) 2024-04-26T23:09:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.418+0000 7f4906a00640 1 -- 172.21.15.40:0/4028465270 learned_addr learned my addr 172.21.15.40:0/4028465270 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.418+0000 7f4907400640 1 --2- 172.21.15.40:0/4028465270 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a893220 0x55919a8d94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.418+0000 7f4906000640 1 --2- 172.21.15.40:0/4028465270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8dade0 0x55919a8db330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.418+0000 7f4906a00640 1 -- 172.21.15.40:0/4028465270 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a893220 msgr2=0x55919a8d94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.418+0000 7f4906a00640 1 --2- 172.21.15.40:0/4028465270 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a893220 0x55919a8d94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.418+0000 7f4906a00640 1 -- 172.21.15.40:0/4028465270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8dade0 msgr2=0x55919a8db330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.418+0000 7f4906a00640 1 --2- 172.21.15.40:0/4028465270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8dade0 0x55919a8db330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.418+0000 7f4906a00640 1 -- 172.21.15.40:0/4028465270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55919a785050 con 0x55919a8bd540 2024-04-26T23:09:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.419+0000 7f4906a00640 1 --2- 172.21.15.40:0/4028465270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8bd920 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f48f8004650 tx=0x7f48f8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd88dec8539afd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.419+0000 7f48f3000640 1 -- 172.21.15.40:0/4028465270 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f800f040 con 0x55919a8bd540 2024-04-26T23:09:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.419+0000 7f48f3000640 1 -- 172.21.15.40:0/4028465270 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48f8026070 con 0x55919a8bd540 2024-04-26T23:09:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.419+0000 7f48f3000640 1 -- 172.21.15.40:0/4028465270 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f8015dc0 con 0x55919a8bd540 2024-04-26T23:09:49.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.420+0000 7f49103c1740 1 -- 172.21.15.40:0/4028465270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 msgr2=0x55919a8bd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.420+0000 7f49103c1740 1 --2- 172.21.15.40:0/4028465270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8bd920 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f48f8004650 tx=0x7f48f8014000 comp rx=0 tx=0).stop 2024-04-26T23:09:49.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.420+0000 7f49103c1740 1 -- 172.21.15.40:0/4028465270 shutdown_connections 2024-04-26T23:09:49.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.420+0000 7f49103c1740 1 --2- 172.21.15.40:0/4028465270 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a893220 0x55919a8d94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.420+0000 7f49103c1740 1 --2- 172.21.15.40:0/4028465270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8dade0 0x55919a8db330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.420+0000 7f49103c1740 1 --2- 172.21.15.40:0/4028465270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8bd920 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.420+0000 7f49103c1740 1 -- 172.21.15.40:0/4028465270 >> 172.21.15.40:0/4028465270 conn(0x55919a88fb50 msgr2=0x55919a8d8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.420+0000 7f49103c1740 1 -- 172.21.15.40:0/4028465270 shutdown_connections 2024-04-26T23:09:49.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.420+0000 7f49103c1740 1 -- 172.21.15.40:0/4028465270 wait complete. 2024-04-26T23:09:49.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.422+0000 7f49103c1740 1 -- start start 2024-04-26T23:09:49.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.422+0000 7f49103c1740 1 Processor -- start 2024-04-26T23:09:49.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.424+0000 7f49103c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8abcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.424+0000 7f49103c1740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8dade0 0x55919a8db8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.424+0000 7f49103c1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8a97b0 0x55919a8dbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.424+0000 7f49103c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55919a7fb200 con 0x55919a8bd540 2024-04-26T23:09:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.424+0000 7f49103c1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55919a7f5ee0 con 0x55919a8a97b0 2024-04-26T23:09:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.424+0000 7f49103c1740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55919a802e70 con 0x55919a8dade0 2024-04-26T23:09:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.424+0000 7f49103c1740 10 client.0 _ll_register_callbacks cb 0x55919a7ec270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.424+0000 7f4906a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8abcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.424+0000 7f4906a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8abcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40058/0 (socket says 172.21.15.40:40058) 2024-04-26T23:09:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.424+0000 7f4906a00640 1 -- 172.21.15.40:0/2814205755 learned_addr learned my addr 172.21.15.40:0/2814205755 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.424+0000 7f4907400640 1 --2- 172.21.15.40:0/2814205755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8a97b0 0x55919a8dbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.425+0000 7f4906a00640 1 -- 172.21.15.40:0/2814205755 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8dade0 msgr2=0x55919a8db8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.425+0000 7f4906a00640 1 --2- 172.21.15.40:0/2814205755 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8dade0 0x55919a8db8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.425+0000 7f4906a00640 1 -- 172.21.15.40:0/2814205755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8a97b0 msgr2=0x55919a8dbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.425+0000 7f4906a00640 1 --2- 172.21.15.40:0/2814205755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8a97b0 0x55919a8dbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.425+0000 7f4906a00640 1 -- 172.21.15.40:0/2814205755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55919a5d27d0 con 0x55919a8bd540 2024-04-26T23:09:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.425+0000 7f4906a00640 1 --2- 172.21.15.40:0/2814205755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8abcf0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f48f8023040 tx=0x7f48f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e95342c30d43b63 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.425+0000 7f48e7400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.425+0000 7f48e7400640 1 -- 172.21.15.40:0/2814205755 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f802f070 con 0x55919a8bd540 2024-04-26T23:09:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.425+0000 7f48e7400640 1 -- 172.21.15.40:0/2814205755 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f48f8005da0 con 0x55919a8bd540 2024-04-26T23:09:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.425+0000 7f49103c1740 1 -- 172.21.15.40:0/2814205755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48f8026070 con 0x55919a8bd540 2024-04-26T23:09:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.426+0000 7f48e7400640 1 -- 172.21.15.40:0/2814205755 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f48f803c050 con 0x55919a8bd540 2024-04-26T23:09:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.426+0000 7f49103c1740 10 client.9615 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.426+0000 7f49103c1740 1 -- 172.21.15.40:0/2814205755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55919a5083f0 con 0x55919a8bd540 2024-04-26T23:09:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.426+0000 7f48e7400640 1 -- 172.21.15.40:0/2814205755 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48f8026070 con 0x55919a8bd540 2024-04-26T23:09:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.426+0000 7f48e7400640 1 -- 172.21.15.40:0/2814205755 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7f48f803f020 con 0x55919a8bd540 2024-04-26T23:09:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.426+0000 7f49103c1740 10 client.9615 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.426+0000 7f49103c1740 10 client.9615 resolve_mds: resolved 7:1 to role '7:1' aka daemon mds.c 2024-04-26T23:09:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.426+0000 7f49103c1740 20 client.9615 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.426+0000 7f49103c1740 1 --2- 172.21.15.40:0/2814205755 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55919a8bbd10 0x55919a8bc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.426+0000 7f49103c1740 4 client.9615 mds_command: new command op to 8949 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.426+0000 7f49103c1740 1 -- 172.21.15.40:0/2814205755 --> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55919a8047e0 con 0x55919a8bbd10 2024-04-26T23:09:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.426+0000 7f4906000640 1 --2- 172.21.15.40:0/2814205755 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55919a8bbd10 0x55919a8bc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.427+0000 7f4906000640 1 --2- 172.21.15.40:0/2814205755 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55919a8bbd10 0x55919a8bc120 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74d110b7bffebd42 server_cookie=57957f6d516d3e97 in_seq=0 out_seq=0 2024-04-26T23:09:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.427+0000 7f48e7400640 10 client.9615 ms_handle_connect on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.428+0000 7f48e7400640 1 -- 172.21.15.40:0/2814205755 <== mds.1 v2:172.21.15.40:6832/3250599834 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55919a8047e0 con 0x55919a8bbd10 2024-04-26T23:09:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.428+0000 7f48e7400640 10 client.9615 handle_command_reply: tid=0 2024-04-26T23:09:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.428+0000 7f49103c1740 1 client.9615 shutdown 2024-04-26T23:09:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.428+0000 7f49103c1740 1 -- 172.21.15.40:0/2814205755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 msgr2=0x55919a8abcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.428+0000 7f49103c1740 1 --2- 172.21.15.40:0/2814205755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8abcf0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f48f8023040 tx=0x7f48f8004730 comp rx=0 tx=0).stop 2024-04-26T23:09:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.428+0000 7f49103c1740 1 -- 172.21.15.40:0/2814205755 shutdown_connections 2024-04-26T23:09:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.428+0000 7f49103c1740 1 --2- 172.21.15.40:0/2814205755 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55919a8bbd10 0x55919a8bc120 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.428+0000 7f49103c1740 1 --2- 172.21.15.40:0/2814205755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8a97b0 0x55919a8dbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.428+0000 7f49103c1740 1 --2- 172.21.15.40:0/2814205755 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8dade0 0x55919a8db8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.428+0000 7f49103c1740 1 --2- 172.21.15.40:0/2814205755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8abcf0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.428+0000 7f48e7400640 0 client.9615 ms_handle_reset on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.428+0000 7f49103c1740 1 -- 172.21.15.40:0/2814205755 >> 172.21.15.40:0/2814205755 conn(0x55919a893220 msgr2=0x55919a8910f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.429+0000 7f49103c1740 1 -- 172.21.15.40:0/2814205755 shutdown_connections 2024-04-26T23:09:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.429+0000 7f49103c1740 1 -- 172.21.15.40:0/2814205755 wait complete. 2024-04-26T23:09:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.429+0000 7f49103c1740 20 client.9615 trim_cache size 0 max 16384 2024-04-26T23:09:49.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.448+0000 7f49103c1740 1 Processor -- start 2024-04-26T23:09:49.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.448+0000 7f49103c1740 1 -- start start 2024-04-26T23:09:49.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.449+0000 7f49103c1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8a97b0 0x55919a8e6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.449+0000 7f49103c1740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8bd540 0x55919a8e71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.449+0000 7f49103c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8dade0 0x55919a8d1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.449+0000 7f49103c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4908201fe0 con 0x55919a8dade0 2024-04-26T23:09:49.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.449+0000 7f49103c1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4908201180 con 0x55919a8a97b0 2024-04-26T23:09:49.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.449+0000 7f49103c1740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f49082018b0 con 0x55919a8bd540 2024-04-26T23:09:49.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.450+0000 7f4907400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8dade0 0x55919a8d1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.450+0000 7f4907400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8dade0 0x55919a8d1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40062/0 (socket says 172.21.15.40:40062) 2024-04-26T23:09:49.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.450+0000 7f4907400640 1 -- 172.21.15.40:0/4071225073 learned_addr learned my addr 172.21.15.40:0/4071225073 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.450+0000 7f4906000640 1 --2- 172.21.15.40:0/4071225073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8bd540 0x55919a8e71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.450+0000 7f4906a00640 1 --2- 172.21.15.40:0/4071225073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8a97b0 0x55919a8e6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.450+0000 7f4907400640 1 -- 172.21.15.40:0/4071225073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8bd540 msgr2=0x55919a8e71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.450+0000 7f4907400640 1 --2- 172.21.15.40:0/4071225073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8bd540 0x55919a8e71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.450+0000 7f4907400640 1 -- 172.21.15.40:0/4071225073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8a97b0 msgr2=0x55919a8e6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.450+0000 7f4907400640 1 --2- 172.21.15.40:0/4071225073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8a97b0 0x55919a8e6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.450+0000 7f4907400640 1 -- 172.21.15.40:0/4071225073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55919a4c80f0 con 0x55919a8dade0 2024-04-26T23:09:49.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.450+0000 7f4907400640 1 --2- 172.21.15.40:0/4071225073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8dade0 0x55919a8d1160 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f4900005f90 tx=0x7f4900001d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408164afdbee31fe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.451+0000 7f48f2600640 1 -- 172.21.15.40:0/4071225073 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49000077a0 con 0x55919a8dade0 2024-04-26T23:09:49.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.451+0000 7f48f2600640 1 -- 172.21.15.40:0/4071225073 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4900007940 con 0x55919a8dade0 2024-04-26T23:09:49.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.451+0000 7f48f2600640 1 -- 172.21.15.40:0/4071225073 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4900015660 con 0x55919a8dade0 2024-04-26T23:09:49.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.451+0000 7f49103c1740 1 -- 172.21.15.40:0/4071225073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8dade0 msgr2=0x55919a8d1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.451+0000 7f49103c1740 1 --2- 172.21.15.40:0/4071225073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8dade0 0x55919a8d1160 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f4900005f90 tx=0x7f4900001d30 comp rx=0 tx=0).stop 2024-04-26T23:09:49.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.452+0000 7f49103c1740 1 -- 172.21.15.40:0/4071225073 shutdown_connections 2024-04-26T23:09:49.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.452+0000 7f49103c1740 1 --2- 172.21.15.40:0/4071225073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8dade0 0x55919a8d1160 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.452+0000 7f49103c1740 1 --2- 172.21.15.40:0/4071225073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8bd540 0x55919a8e71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.452+0000 7f49103c1740 1 --2- 172.21.15.40:0/4071225073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8a97b0 0x55919a8e6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.452+0000 7f49103c1740 1 -- 172.21.15.40:0/4071225073 >> 172.21.15.40:0/4071225073 conn(0x55919a893220 msgr2=0x55919a893600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.452+0000 7f49103c1740 1 -- 172.21.15.40:0/4071225073 shutdown_connections 2024-04-26T23:09:49.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.452+0000 7f49103c1740 1 -- 172.21.15.40:0/4071225073 wait complete. 2024-04-26T23:09:49.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.452+0000 7f49103c1740 1 -- start start 2024-04-26T23:09:49.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.453+0000 7f49103c1740 1 Processor -- start 2024-04-26T23:09:49.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.454+0000 7f49103c1740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8a97b0 0x55919a8db240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.454+0000 7f49103c1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8d8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.454+0000 7f49103c1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8dade0 0x55919a8d86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.454+0000 7f49103c1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55919a803fd0 con 0x55919a8bd540 2024-04-26T23:09:49.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.454+0000 7f49103c1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f49080d0780 con 0x55919a8dade0 2024-04-26T23:09:49.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.454+0000 7f49103c1740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55919a8fb1c0 con 0x55919a8a97b0 2024-04-26T23:09:49.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.454+0000 7f49103c1740 10 client.0 _ll_register_callbacks cb 0x7f48f40094d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:49.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.454+0000 7f4906000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8d8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.454+0000 7f4906000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8d8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40066/0 (socket says 172.21.15.40:40066) 2024-04-26T23:09:49.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.454+0000 7f4906000640 1 -- 172.21.15.40:0/4274010836 learned_addr learned my addr 172.21.15.40:0/4274010836 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.454+0000 7f4907400640 1 --2- 172.21.15.40:0/4274010836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8dade0 0x55919a8d86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.454+0000 7f4906a00640 1 --2- 172.21.15.40:0/4274010836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8a97b0 0x55919a8db240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.455+0000 7f4906000640 1 -- 172.21.15.40:0/4274010836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8a97b0 msgr2=0x55919a8db240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.455+0000 7f4906000640 1 --2- 172.21.15.40:0/4274010836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8a97b0 0x55919a8db240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.455+0000 7f4906000640 1 -- 172.21.15.40:0/4274010836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8dade0 msgr2=0x55919a8d86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.455+0000 7f4906000640 1 --2- 172.21.15.40:0/4274010836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8dade0 0x55919a8d86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.455+0000 7f4906000640 1 -- 172.21.15.40:0/4274010836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55919a7bea30 con 0x55919a8bd540 2024-04-26T23:09:49.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.455+0000 7f4906000640 1 --2- 172.21.15.40:0/4274010836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8d8160 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f48f4082680 tx=0x7f48f4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2b858ec9efe89b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.455+0000 7f48e7400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:49.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.455+0000 7f48e7400640 1 -- 172.21.15.40:0/4274010836 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f4097060 con 0x55919a8bd540 2024-04-26T23:09:49.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.455+0000 7f48e7400640 1 -- 172.21.15.40:0/4274010836 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f48f40782d0 con 0x55919a8bd540 2024-04-26T23:09:49.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.455+0000 7f49103c1740 1 -- 172.21.15.40:0/4274010836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4900007940 con 0x55919a8bd540 2024-04-26T23:09:49.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.455+0000 7f48e7400640 1 -- 172.21.15.40:0/4274010836 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f48f4076640 con 0x55919a8bd540 2024-04-26T23:09:49.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.455+0000 7f49103c1740 10 client.9621 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:49.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.455+0000 7f49103c1740 1 -- 172.21.15.40:0/4274010836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55919a8faf00 con 0x55919a8bd540 2024-04-26T23:09:49.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.456+0000 7f48e7400640 1 -- 172.21.15.40:0/4274010836 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4900007940 con 0x55919a8bd540 2024-04-26T23:09:49.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.456+0000 7f48e7400640 1 -- 172.21.15.40:0/4274010836 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7f48f403a030 con 0x55919a8bd540 2024-04-26T23:09:49.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.456+0000 7f49103c1740 10 client.9621 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:49.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.456+0000 7f49103c1740 10 client.9621 resolve_mds: resolved 7:1 to role '7:1' aka daemon mds.c 2024-04-26T23:09:49.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.456+0000 7f49103c1740 20 client.9621 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:49.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.456+0000 7f49103c1740 1 --2- 172.21.15.40:0/4274010836 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55919a950e40 0x55919a88c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.456+0000 7f49103c1740 4 client.9621 mds_command: new command op to 8949 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:09:49.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.456+0000 7f49103c1740 1 -- 172.21.15.40:0/4274010836 --> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55919a8d38e0 con 0x55919a950e40 2024-04-26T23:09:49.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.456+0000 7f4906a00640 1 --2- 172.21.15.40:0/4274010836 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55919a950e40 0x55919a88c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f48e7400640 10 client.9621 ms_handle_connect on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:49.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f4906a00640 1 --2- 172.21.15.40:0/4274010836 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55919a950e40 0x55919a88c7b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2ece3fe292501248 server_cookie=9a3a3d1462198f2b in_seq=0 out_seq=0 2024-04-26T23:09:49.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f48e7400640 1 -- 172.21.15.40:0/4274010836 <== mds.1 v2:172.21.15.40:6832/3250599834 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55919a8d38e0 con 0x55919a950e40 2024-04-26T23:09:49.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f48e7400640 10 client.9621 handle_command_reply: tid=0 2024-04-26T23:09:49.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f49103c1740 1 client.9621 shutdown 2024-04-26T23:09:49.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f49103c1740 1 -- 172.21.15.40:0/4274010836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 msgr2=0x55919a8d8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f49103c1740 1 --2- 172.21.15.40:0/4274010836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8d8160 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f48f4082680 tx=0x7f48f4009580 comp rx=0 tx=0).stop 2024-04-26T23:09:49.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f49103c1740 1 -- 172.21.15.40:0/4274010836 shutdown_connections 2024-04-26T23:09:49.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f49103c1740 1 --2- 172.21.15.40:0/4274010836 >> [v2:172.21.15.40:6832/3250599834,v1:172.21.15.40:6833/3250599834] conn(0x55919a950e40 0x55919a88c7b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f49103c1740 1 --2- 172.21.15.40:0/4274010836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55919a8dade0 0x55919a8d86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f49103c1740 1 --2- 172.21.15.40:0/4274010836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55919a8bd540 0x55919a8d8160 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f48e7400640 0 client.9621 ms_handle_reset on v2:172.21.15.40:6832/3250599834 2024-04-26T23:09:49.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f49103c1740 1 --2- 172.21.15.40:0/4274010836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55919a8a97b0 0x55919a8db240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f49103c1740 1 -- 172.21.15.40:0/4274010836 >> 172.21.15.40:0/4274010836 conn(0x55919a893220 msgr2=0x55919a889040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f49103c1740 1 -- 172.21.15.40:0/4274010836 shutdown_connections 2024-04-26T23:09:49.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.457+0000 7f49103c1740 1 -- 172.21.15.40:0/4274010836 wait complete. 2024-04-26T23:09:49.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.458+0000 7f49103c1740 20 client.9621 trim_cache size 0 max 16384 2024-04-26T23:09:49.465 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:49.466 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-26T23:09:49.466 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:09:49.466 INFO:teuthology.orchestra.run.smithi040.stdout: "a9827531-70a9-4d6f-b12a-2d71fa886d03": { 2024-04-26T23:09:49.466 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:09:49.466 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "a9827531-70a9-4d6f-b12a-2d71fa886d03", 2024-04-26T23:09:49.466 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:09:49.466 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:49.466 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:49.466 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:49.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.458+0000 7f490d600640 1 -- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48e806c8f0 msgr2=0x7f48e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.458+0000 7f490d600640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48e806c8f0 0x7f48e806edb0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f48f8006fd0 tx=0x7f48f8008040 comp rx=0 tx=0).stop 2024-04-26T23:09:49.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.458+0000 7f490d600640 1 -- 172.21.15.40:0/2208471082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4908071e60 msgr2=0x7f4908150f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.458+0000 7f490d600640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4908071e60 0x7f4908150f50 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f48f402a2c0 tx=0x7f48f40046c0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.458+0000 7f490d600640 1 -- 172.21.15.40:0/2208471082 shutdown_connections 2024-04-26T23:09:49.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.458+0000 7f490d600640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f48e806c8f0 0x7f48e806edb0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.458+0000 7f490d600640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4908072780 0x7f49080c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.458+0000 7f490d600640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4908071e60 0x7f4908150f50 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.458+0000 7f490d600640 1 --2- 172.21.15.40:0/2208471082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490806fe20 0x7f49080afd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.458+0000 7f490d600640 1 -- 172.21.15.40:0/2208471082 >> 172.21.15.40:0/2208471082 conn(0x7f49080d0be0 msgr2=0x7f490815e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.459+0000 7f490d600640 1 -- 172.21.15.40:0/2208471082 shutdown_connections 2024-04-26T23:09:49.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.459+0000 7f490d600640 1 -- 172.21.15.40:0/2208471082 wait complete. 2024-04-26T23:09:49.476 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:09:49.476 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:09:49.477 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.7:2 scrub status 2024-04-26T23:09:49.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.745+0000 7fb4fa200640 1 Processor -- start 2024-04-26T23:09:49.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.746+0000 7fb4fa200640 1 -- start start 2024-04-26T23:09:49.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.747+0000 7fb4fa200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4f4071e60 0x7fb4f406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.747+0000 7fb4fa200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4f415d7c0 con 0x7fb4f406fe20 2024-04-26T23:09:49.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.747+0000 7fb4fa200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4f415daa0 con 0x7fb4f40cc480 2024-04-26T23:09:49.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.747+0000 7fb4fa200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f415dd80 con 0x7fb4f4071e60 2024-04-26T23:09:49.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.747+0000 7fb4f2a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb4f406fe20 0x7fb4f4072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35782/0 (socket says 172.21.15.40:35782) 2024-04-26T23:09:49.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.747+0000 7fb4f2a00640 1 -- 172.21.15.40:0/3133634235 learned_addr learned my addr 172.21.15.40:0/3133634235 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:49.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.748+0000 7fb4f3400640 1 --2- 172.21.15.40:0/3133634235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4f4071e60 0x7fb4f406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.748+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1886728006 0 0) 0x7fb4f415d7c0 con 0x7fb4f406fe20 2024-04-26T23:09:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.748+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4d8003520 con 0x7fb4f406fe20 2024-04-26T23:09:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.748+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 831145535 0 0) 0x7fb4d8003520 con 0x7fb4f406fe20 2024-04-26T23:09:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4f4071e60 msgr2=0x7fb4f406f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f2000640 1 --2- 172.21.15.40:0/3133634235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4f4071e60 0x7fb4f406f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 >> v1:172.21.15.103:6789/0 conn(0x7fb4f40cc480 legacy=0x7fb4f4166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f415e060 con 0x7fb4f406fe20 2024-04-26T23:09:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fb4e0001670 con 0x7fb4f406fe20 2024-04-26T23:09:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 >> v1:172.21.15.40:6789/0 conn(0x7fb4f406fe20 legacy=0x7fb4f4072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f2000640 1 --2- 172.21.15.40:0/3133634235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d80039e0 0x7fb4d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f2000640 1 --2- 172.21.15.40:0/3133634235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80043d0 0x7fb4d80048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f2000640 1 --2- 172.21.15.40:0/3133634235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d80050a0 0x7fb4d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f415dd80 con 0x7fb4d80050a0 2024-04-26T23:09:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d800bfb0 con 0x7fb4d80043d0 2024-04-26T23:09:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb4d800c1b0 con 0x7fb4d80039e0 2024-04-26T23:09:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f3e00640 1 --2- 172.21.15.40:0/3133634235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d80050a0 0x7fb4d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f3400640 1 --2- 172.21.15.40:0/3133634235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d80039e0 0x7fb4d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.749+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/3133634235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80043d0 0x7fb4d80048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.750+0000 7fb4f3e00640 1 -- 172.21.15.40:0/3133634235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d80039e0 msgr2=0x7fb4d8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.750+0000 7fb4f3e00640 1 --2- 172.21.15.40:0/3133634235 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4d80039e0 0x7fb4d8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.750+0000 7fb4f3e00640 1 -- 172.21.15.40:0/3133634235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80043d0 msgr2=0x7fb4d80048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.750+0000 7fb4f3e00640 1 --2- 172.21.15.40:0/3133634235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80043d0 0x7fb4d80048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.750+0000 7fb4f3e00640 1 -- 172.21.15.40:0/3133634235 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d8003520 con 0x7fb4d80050a0 2024-04-26T23:09:49.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.750+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/3133634235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80043d0 0x7fb4d80048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.751+0000 7fb4f3e00640 1 --2- 172.21.15.40:0/3133634235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d80050a0 0x7fb4d800b9c0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec001720 tx=0x7fb4d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5952309ae24cf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.751+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec001370 con 0x7fb4d80050a0 2024-04-26T23:09:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.751+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4ec0052d0 con 0x7fb4d80050a0 2024-04-26T23:09:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.751+0000 7fb4f2000640 1 -- 172.21.15.40:0/3133634235 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec02abc0 con 0x7fb4d80050a0 2024-04-26T23:09:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.751+0000 7fb4fa200640 1 -- 172.21.15.40:0/3133634235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d80050a0 msgr2=0x7fb4d800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.752+0000 7fb4fa200640 1 --2- 172.21.15.40:0/3133634235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d80050a0 0x7fb4d800b9c0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec001720 tx=0x7fb4d8005830 comp rx=0 tx=0).stop 2024-04-26T23:09:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.752+0000 7fb4f3e00640 1 -- 172.21.15.40:0/3133634235 reap_dead start 2024-04-26T23:09:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.752+0000 7fb4fa200640 1 -- 172.21.15.40:0/3133634235 shutdown_connections 2024-04-26T23:09:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.752+0000 7fb4fa200640 1 -- 172.21.15.40:0/3133634235 >> 172.21.15.40:0/3133634235 conn(0x7fb4f40d0be0 msgr2=0x7fb4f40d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.752+0000 7fb4fa200640 1 -- 172.21.15.40:0/3133634235 shutdown_connections 2024-04-26T23:09:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.752+0000 7fb4fa200640 1 -- 172.21.15.40:0/3133634235 wait complete. 2024-04-26T23:09:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.754+0000 7fb4fa200640 1 Processor -- start 2024-04-26T23:09:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.754+0000 7fb4fa200640 1 -- start start 2024-04-26T23:09:49.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4fa200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d80050a0 0x7fb4f41584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4fa200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4f40c7610 0x7fb4f41f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4fa200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f40b5980 0x7fb4f40b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4fa200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f415d570 con 0x7fb4d80050a0 2024-04-26T23:09:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4fa200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f415d3f0 con 0x7fb4f40b5980 2024-04-26T23:09:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4fa200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f4072950 con 0x7fb4f40c7610 2024-04-26T23:09:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4f3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d80050a0 0x7fb4f41584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4f3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d80050a0 0x7fb4f41584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40076/0 (socket says 172.21.15.40:40076) 2024-04-26T23:09:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4f3400640 1 -- 172.21.15.40:0/1050815751 learned_addr learned my addr 172.21.15.40:0/1050815751 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4f40c7610 0x7fb4f41f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4f3e00640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f40b5980 0x7fb4f40b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4f3400640 1 -- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4f40c7610 msgr2=0x7fb4f41f3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4f3400640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4f40c7610 0x7fb4f41f3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4f3400640 1 -- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f40b5980 msgr2=0x7fb4f40b5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4f3400640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f40b5980 0x7fb4f40b5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4f3400640 1 -- 172.21.15.40:0/1050815751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f40c7a90 con 0x7fb4d80050a0 2024-04-26T23:09:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4f3e00640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f40b5980 0x7fb4f40b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.755+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4f40c7610 0x7fb4f41f3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.756+0000 7fb4f3400640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d80050a0 0x7fb4f41584a0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4007f80 tx=0x7fb4e4011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.756+0000 7fb4eb400640 1 -- 172.21.15.40:0/1050815751 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e401a020 con 0x7fb4d80050a0 2024-04-26T23:09:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.756+0000 7fb4eb400640 1 -- 172.21.15.40:0/1050815751 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4e4016470 con 0x7fb4d80050a0 2024-04-26T23:09:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.756+0000 7fb4fa200640 1 -- 172.21.15.40:0/1050815751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f40c9350 con 0x7fb4d80050a0 2024-04-26T23:09:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.756+0000 7fb4eb400640 1 -- 172.21.15.40:0/1050815751 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4016840 con 0x7fb4d80050a0 2024-04-26T23:09:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.756+0000 7fb4fa200640 1 -- 172.21.15.40:0/1050815751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f41f3d90 con 0x7fb4d80050a0 2024-04-26T23:09:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.758+0000 7fb4eb400640 1 -- 172.21.15.40:0/1050815751 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb4e4005d10 con 0x7fb4d80050a0 2024-04-26T23:09:49.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.760+0000 7fb4eb400640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4d406c8f0 0x7fb4d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.760+0000 7fb4eb400640 1 -- 172.21.15.40:0/1050815751 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7fb4e408ccf0 con 0x7fb4d80050a0 2024-04-26T23:09:49.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.760+0000 7fb4fd050740 1 Processor -- start 2024-04-26T23:09:49.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.760+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4d406c8f0 0x7fb4d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.760+0000 7fb4fd050740 1 -- start start 2024-04-26T23:09:49.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.761+0000 7fb4fd050740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffa8950 0x55724ffa8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.761+0000 7fb4fd050740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffc61f0 0x55724ffc6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.761+0000 7fb4fd050740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ff7e630 0x55724ffc48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.761+0000 7fb4fd050740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55724fe8a030 con 0x55724ffa8950 2024-04-26T23:09:49.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.761+0000 7fb4fd050740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55724fea57f0 con 0x55724ff7e630 2024-04-26T23:09:49.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.761+0000 7fb4fd050740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55724fe607e0 con 0x55724ffc61f0 2024-04-26T23:09:49.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.762+0000 7fb4f3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffa8950 0x55724ffa8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.762+0000 7fb4f3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffa8950 0x55724ffa8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40078/0 (socket says 172.21.15.40:40078) 2024-04-26T23:09:49.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.762+0000 7fb4f3400640 1 -- 172.21.15.40:0/2013867624 learned_addr learned my addr 172.21.15.40:0/2013867624 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.762+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4d406c8f0 0x7fb4d406edb0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0001200 tx=0x7fb4e0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.762+0000 7fb4f3e00640 1 --2- 172.21.15.40:0/2013867624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ff7e630 0x55724ffc48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.762+0000 7fb4f3400640 1 -- 172.21.15.40:0/2013867624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffc61f0 msgr2=0x55724ffc6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.762+0000 7fb4f3400640 1 --2- 172.21.15.40:0/2013867624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffc61f0 0x55724ffc6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.762+0000 7fb4f3400640 1 -- 172.21.15.40:0/2013867624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ff7e630 msgr2=0x55724ffc48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.762+0000 7fb4f3400640 1 --2- 172.21.15.40:0/2013867624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ff7e630 0x55724ffc48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.762+0000 7fb4f3400640 1 -- 172.21.15.40:0/2013867624 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55724fe6c050 con 0x55724ffa8950 2024-04-26T23:09:49.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.762+0000 7fb4f3e00640 1 --2- 172.21.15.40:0/2013867624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ff7e630 0x55724ffc48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:49.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.762+0000 7fb4f3400640 1 --2- 172.21.15.40:0/2013867624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffa8950 0x55724ffa8d30 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4056c30 tx=0x7fb4e405a910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde76ad59096688 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.763+0000 7fb4df400640 1 -- 172.21.15.40:0/2013867624 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4058ad0 con 0x55724ffa8950 2024-04-26T23:09:49.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.763+0000 7fb4df400640 1 -- 172.21.15.40:0/2013867624 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4e4056e40 con 0x55724ffa8950 2024-04-26T23:09:49.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.763+0000 7fb4df400640 1 -- 172.21.15.40:0/2013867624 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4063140 con 0x55724ffa8950 2024-04-26T23:09:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.763+0000 7fb4fd050740 1 -- 172.21.15.40:0/2013867624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffa8950 msgr2=0x55724ffa8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.763+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2013867624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffa8950 0x55724ffa8d30 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4056c30 tx=0x7fb4e405a910 comp rx=0 tx=0).stop 2024-04-26T23:09:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.763+0000 7fb4fd050740 1 -- 172.21.15.40:0/2013867624 shutdown_connections 2024-04-26T23:09:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.763+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2013867624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ff7e630 0x55724ffc48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.763+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2013867624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffc61f0 0x55724ffc6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.763+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2013867624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffa8950 0x55724ffa8d30 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.763+0000 7fb4fd050740 1 -- 172.21.15.40:0/2013867624 >> 172.21.15.40:0/2013867624 conn(0x55724ff7af60 msgr2=0x55724ffc3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.763+0000 7fb4fd050740 1 -- 172.21.15.40:0/2013867624 shutdown_connections 2024-04-26T23:09:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.764+0000 7fb4fd050740 1 -- 172.21.15.40:0/2013867624 wait complete. 2024-04-26T23:09:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.764+0000 7fb4fd050740 1 -- start start 2024-04-26T23:09:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.764+0000 7fb4fd050740 1 Processor -- start 2024-04-26T23:09:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.765+0000 7fb4fd050740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffa8950 0x55724ff990a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.765+0000 7fb4fd050740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 0x55724ffabf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.765+0000 7fb4fd050740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffab960 0x55724ffa72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.765+0000 7fb4fd050740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55724fee2200 con 0x55724ffc61f0 2024-04-26T23:09:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.765+0000 7fb4fd050740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55724fedcee0 con 0x55724ffa8950 2024-04-26T23:09:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.765+0000 7fb4fd050740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55724fee9e70 con 0x55724ffab960 2024-04-26T23:09:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.766+0000 7fb4fd050740 10 client.0 _ll_register_callbacks cb 0x55724fed3270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.766+0000 7fb4f2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 0x55724ffabf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.766+0000 7fb4f2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 0x55724ffabf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40088/0 (socket says 172.21.15.40:40088) 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.766+0000 7fb4f2a00640 1 -- 172.21.15.40:0/2962357889 learned_addr learned my addr 172.21.15.40:0/2962357889 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.766+0000 7fb4f3400640 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffa8950 0x55724ff990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.766+0000 7fb4f3e00640 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffab960 0x55724ffa72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.766+0000 7fb4f2a00640 1 -- 172.21.15.40:0/2962357889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffab960 msgr2=0x55724ffa72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.766+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffab960 0x55724ffa72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.766+0000 7fb4f2a00640 1 -- 172.21.15.40:0/2962357889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffa8950 msgr2=0x55724ff990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.766+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffa8950 0x55724ff990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.766+0000 7fb4f2a00640 1 -- 172.21.15.40:0/2962357889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55724fcb97d0 con 0x55724ffc61f0 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.766+0000 7fb4f3400640 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffa8950 0x55724ff990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.767+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 0x55724ffabf50 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00073d0 tx=0x7fb4e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95eccaa882293389 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.767+0000 7fb4d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.767+0000 7fb4d3e00640 1 -- 172.21.15.40:0/2962357889 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0013890 con 0x55724ffc61f0 2024-04-26T23:09:49.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.767+0000 7fb4d3e00640 1 -- 172.21.15.40:0/2962357889 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7fb4e0023c80 con 0x55724ffc61f0 2024-04-26T23:09:49.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.767+0000 7fb4fd050740 1 -- 172.21.15.40:0/2962357889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4e4056e40 con 0x55724ffc61f0 2024-04-26T23:09:49.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.767+0000 7fb4d3e00640 1 -- 172.21.15.40:0/2962357889 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7fb4e0013a30 con 0x55724ffc61f0 2024-04-26T23:09:49.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.767+0000 7fb4fd050740 10 client.9633 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:49.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.767+0000 7fb4fd050740 1 -- 172.21.15.40:0/2962357889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55724fbef3f0 con 0x55724ffc61f0 2024-04-26T23:09:49.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.768+0000 7fb4d3e00640 1 -- 172.21.15.40:0/2962357889 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4e4056e40 con 0x55724ffc61f0 2024-04-26T23:09:49.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.768+0000 7fb4d3e00640 1 -- 172.21.15.40:0/2962357889 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7fb4e0013bd0 con 0x55724ffc61f0 2024-04-26T23:09:49.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.768+0000 7fb4fd050740 10 client.9633 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:49.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.768+0000 7fb4fd050740 10 client.9633 resolve_mds: resolved 7:2 to role '7:2' aka daemon mds.d 2024-04-26T23:09:49.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.768+0000 7fb4fd050740 20 client.9633 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:49.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.768+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55724ffd3210 0x55724ffd36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.768+0000 7fb4fd050740 4 client.9633 mds_command: new command op to 6598 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:49.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.768+0000 7fb4fd050740 1 -- 172.21.15.40:0/2962357889 --> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55724feeb7e0 con 0x55724ffd3210 2024-04-26T23:09:49.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.769+0000 7fb4f3400640 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55724ffd3210 0x55724ffd36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.770+0000 7fb4f3400640 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55724ffd3210 0x55724ffd36a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1a2c4147b6331350 server_cookie=973d90a138d0d9dd in_seq=0 out_seq=0 2024-04-26T23:09:49.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.770+0000 7fb4d3e00640 10 client.9633 ms_handle_connect on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:49.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.771+0000 7fb4d3e00640 1 -- 172.21.15.40:0/2962357889 <== mds.2 v2:172.21.15.103:6835/2022020922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55724feeb7e0 con 0x55724ffd3210 2024-04-26T23:09:49.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.771+0000 7fb4d3e00640 10 client.9633 handle_command_reply: tid=0 2024-04-26T23:09:49.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.772+0000 7fb4fd050740 1 client.9633 shutdown 2024-04-26T23:09:49.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.772+0000 7fb4fd050740 1 -- 172.21.15.40:0/2962357889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 msgr2=0x55724ffabf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.772+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 0x55724ffabf50 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00073d0 tx=0x7fb4e00057f0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.772+0000 7fb4fd050740 1 -- 172.21.15.40:0/2962357889 shutdown_connections 2024-04-26T23:09:49.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.772+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55724ffd3210 0x55724ffd36a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.772+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffab960 0x55724ffa72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.772+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 0x55724ffabf50 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.772+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2962357889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffa8950 0x55724ff990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.772+0000 7fb4d3e00640 0 client.9633 ms_handle_reset on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:49.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.772+0000 7fb4fd050740 1 -- 172.21.15.40:0/2962357889 >> 172.21.15.40:0/2962357889 conn(0x55724ff7e630 msgr2=0x55724ffc7570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.772+0000 7fb4fd050740 1 -- 172.21.15.40:0/2962357889 shutdown_connections 2024-04-26T23:09:49.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.772+0000 7fb4fd050740 1 -- 172.21.15.40:0/2962357889 wait complete. 2024-04-26T23:09:49.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.772+0000 7fb4fd050740 20 client.9633 trim_cache size 0 max 16384 2024-04-26T23:09:49.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.792+0000 7fb4fd050740 1 Processor -- start 2024-04-26T23:09:49.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.792+0000 7fb4fd050740 1 -- start start 2024-04-26T23:09:49.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.793+0000 7fb4fd050740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffa8950 0x55724ff737b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.793+0000 7fb4fd050740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffab960 0x55724ff73cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.793+0000 7fb4fd050740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 0x55724ffa3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.793+0000 7fb4fd050740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55724ffa8450 con 0x55724ffc61f0 2024-04-26T23:09:49.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.793+0000 7fb4fd050740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55724ff7ad30 con 0x55724ffab960 2024-04-26T23:09:49.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.793+0000 7fb4fd050740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f41fe300 con 0x55724ffa8950 2024-04-26T23:09:49.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.793+0000 7fb4f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 0x55724ffa3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.793+0000 7fb4f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 0x55724ffa3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40090/0 (socket says 172.21.15.40:40090) 2024-04-26T23:09:49.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.793+0000 7fb4f3e00640 1 -- 172.21.15.40:0/2188940339 learned_addr learned my addr 172.21.15.40:0/2188940339 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.794+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/2188940339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffab960 0x55724ff73cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.794+0000 7fb4f3400640 1 --2- 172.21.15.40:0/2188940339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffa8950 0x55724ff737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.794+0000 7fb4f3e00640 1 -- 172.21.15.40:0/2188940339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffa8950 msgr2=0x55724ff737b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.794+0000 7fb4f3e00640 1 --2- 172.21.15.40:0/2188940339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffa8950 0x55724ff737b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.794+0000 7fb4f3e00640 1 -- 172.21.15.40:0/2188940339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffab960 msgr2=0x55724ff73cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.794+0000 7fb4f3e00640 1 --2- 172.21.15.40:0/2188940339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffab960 0x55724ff73cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.794+0000 7fb4f3e00640 1 -- 172.21.15.40:0/2188940339 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55724fbaf0f0 con 0x55724ffc61f0 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.794+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/2188940339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffab960 0x55724ff73cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.794+0000 7fb4f3e00640 1 --2- 172.21.15.40:0/2188940339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 0x55724ffa3710 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec001840 tx=0x7fb4ec02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f23189af5bedc95 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.794+0000 7fb4dea00640 1 -- 172.21.15.40:0/2188940339 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec0013f0 con 0x55724ffc61f0 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.794+0000 7fb4dea00640 1 -- 172.21.15.40:0/2188940339 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4ec0050d0 con 0x55724ffc61f0 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.794+0000 7fb4dea00640 1 -- 172.21.15.40:0/2188940339 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec005480 con 0x55724ffc61f0 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.795+0000 7fb4fd050740 1 -- 172.21.15.40:0/2188940339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 msgr2=0x55724ffa3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.795+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2188940339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 0x55724ffa3710 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec001840 tx=0x7fb4ec02c000 comp rx=0 tx=0).stop 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.795+0000 7fb4fd050740 1 -- 172.21.15.40:0/2188940339 shutdown_connections 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.795+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2188940339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffc61f0 0x55724ffa3710 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.795+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2188940339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffab960 0x55724ff73cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.795+0000 7fb4fd050740 1 --2- 172.21.15.40:0/2188940339 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffa8950 0x55724ff737b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.795+0000 7fb4fd050740 1 -- 172.21.15.40:0/2188940339 >> 172.21.15.40:0/2188940339 conn(0x55724ff7e630 msgr2=0x55724ff98eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.795+0000 7fb4fd050740 1 -- 172.21.15.40:0/2188940339 shutdown_connections 2024-04-26T23:09:49.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.795+0000 7fb4fd050740 1 -- 172.21.15.40:0/2188940339 wait complete. 2024-04-26T23:09:49.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.796+0000 7fb4fd050740 1 -- start start 2024-04-26T23:09:49.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.796+0000 7fb4fd050740 1 Processor -- start 2024-04-26T23:09:49.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.797+0000 7fb4fd050740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffa8950 0x55725003bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.797+0000 7fb4fd050740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffab960 0x55724ff7da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.797+0000 7fb4fd050740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffc61f0 0x55724ff7df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.797+0000 7fb4fd050740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f41fdbd0 con 0x55724ffab960 2024-04-26T23:09:49.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.797+0000 7fb4fd050740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f40d0620 con 0x55724ffc61f0 2024-04-26T23:09:49.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.797+0000 7fb4fd050740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f40caa40 con 0x55724ffa8950 2024-04-26T23:09:49.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.797+0000 7fb4fd050740 10 client.0 _ll_register_callbacks cb 0x7fb4e0007890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:49.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.797+0000 7fb4f2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffab960 0x55724ff7da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.797+0000 7fb4f2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffab960 0x55724ff7da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40092/0 (socket says 172.21.15.40:40092) 2024-04-26T23:09:49.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.797+0000 7fb4f2a00640 1 -- 172.21.15.40:0/3922514191 learned_addr learned my addr 172.21.15.40:0/3922514191 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:49.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.798+0000 7fb4f2a00640 1 -- 172.21.15.40:0/3922514191 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffa8950 msgr2=0x55725003bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.798+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/3922514191 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffa8950 0x55725003bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.798+0000 7fb4f3e00640 1 --2- 172.21.15.40:0/3922514191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffc61f0 0x55724ff7df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.798+0000 7fb4f2a00640 1 -- 172.21.15.40:0/3922514191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffc61f0 msgr2=0x55724ff7df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:49.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.798+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/3922514191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffc61f0 0x55724ff7df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.798+0000 7fb4f2a00640 1 -- 172.21.15.40:0/3922514191 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55724fea5a30 con 0x55724ffab960 2024-04-26T23:09:49.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.798+0000 7fb4f2a00640 1 --2- 172.21.15.40:0/3922514191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffab960 0x55724ff7da30 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x55724ffc5cc0 tx=0x7fb4e0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f670a2095f7d822b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:49.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.798+0000 7fb4d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:49.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.798+0000 7fb4d3e00640 1 -- 172.21.15.40:0/3922514191 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00229f0 con 0x55724ffab960 2024-04-26T23:09:49.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.798+0000 7fb4d3e00640 1 -- 172.21.15.40:0/3922514191 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7fb4e0022b90 con 0x55724ffab960 2024-04-26T23:09:49.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.798+0000 7fb4fd050740 1 -- 172.21.15.40:0/3922514191 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4ec0050d0 con 0x55724ffab960 2024-04-26T23:09:49.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.799+0000 7fb4d3e00640 1 -- 172.21.15.40:0/3922514191 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7fb4e003a050 con 0x55724ffab960 2024-04-26T23:09:49.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.799+0000 7fb4fd050740 10 client.9639 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:49.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.799+0000 7fb4fd050740 1 -- 172.21.15.40:0/3922514191 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x55724ff7f970 con 0x55724ffab960 2024-04-26T23:09:49.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.799+0000 7fb4d3e00640 1 -- 172.21.15.40:0/3922514191 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4ec0050d0 con 0x55724ffab960 2024-04-26T23:09:49.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.799+0000 7fb4d3e00640 1 -- 172.21.15.40:0/3922514191 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7fb4e003d020 con 0x55724ffab960 2024-04-26T23:09:49.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.799+0000 7fb4fd050740 10 client.9639 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:49.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.799+0000 7fb4fd050740 10 client.9639 resolve_mds: resolved 7:2 to role '7:2' aka daemon mds.d 2024-04-26T23:09:49.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.799+0000 7fb4fd050740 20 client.9639 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:49.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.799+0000 7fb4fd050740 1 --2- 172.21.15.40:0/3922514191 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55724ffe6900 0x55724ffa3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:49.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.799+0000 7fb4fd050740 4 client.9639 mds_command: new command op to 6598 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:09:49.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.799+0000 7fb4fd050740 1 -- 172.21.15.40:0/3922514191 --> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55724fee5650 con 0x55724ffe6900 2024-04-26T23:09:49.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.800+0000 7fb4f3400640 1 --2- 172.21.15.40:0/3922514191 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55724ffe6900 0x55724ffa3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:49.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.801+0000 7fb4f3400640 1 --2- 172.21.15.40:0/3922514191 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55724ffe6900 0x55724ffa3710 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=19fa4c0797422cf7 server_cookie=8898aa951b6de5ba in_seq=0 out_seq=0 2024-04-26T23:09:49.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.801+0000 7fb4d3e00640 10 client.9639 ms_handle_connect on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:49.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4d3e00640 1 -- 172.21.15.40:0/3922514191 <== mds.2 v2:172.21.15.103:6835/2022020922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55724fee5650 con 0x55724ffe6900 2024-04-26T23:09:49.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4d3e00640 10 client.9639 handle_command_reply: tid=0 2024-04-26T23:09:49.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4fd050740 1 client.9639 shutdown 2024-04-26T23:09:49.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4fd050740 1 -- 172.21.15.40:0/3922514191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffab960 msgr2=0x55724ff7da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4fd050740 1 --2- 172.21.15.40:0/3922514191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffab960 0x55724ff7da30 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x55724ffc5cc0 tx=0x7fb4e0012000 comp rx=0 tx=0).stop 2024-04-26T23:09:49.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4fd050740 1 -- 172.21.15.40:0/3922514191 shutdown_connections 2024-04-26T23:09:49.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4fd050740 1 --2- 172.21.15.40:0/3922514191 >> [v2:172.21.15.103:6835/2022020922,v1:172.21.15.103:6837/2022020922] conn(0x55724ffe6900 0x55724ffa3710 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4fd050740 1 --2- 172.21.15.40:0/3922514191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55724ffc61f0 0x55724ff7df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4fd050740 1 --2- 172.21.15.40:0/3922514191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55724ffab960 0x55724ff7da30 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4fd050740 1 --2- 172.21.15.40:0/3922514191 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55724ffa8950 0x55725003bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4d3e00640 0 client.9639 ms_handle_reset on v2:172.21.15.103:6835/2022020922 2024-04-26T23:09:49.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4fd050740 1 -- 172.21.15.40:0/3922514191 >> 172.21.15.40:0/3922514191 conn(0x55724ff7e630 msgr2=0x55724ff70060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4fd050740 1 -- 172.21.15.40:0/3922514191 shutdown_connections 2024-04-26T23:09:49.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4fd050740 1 -- 172.21.15.40:0/3922514191 wait complete. 2024-04-26T23:09:49.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.802+0000 7fb4fd050740 20 client.9639 trim_cache size 0 max 16384 2024-04-26T23:09:49.805 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:49.806 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-26T23:09:49.806 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:09:49.806 INFO:teuthology.orchestra.run.smithi040.stdout: "a9827531-70a9-4d6f-b12a-2d71fa886d03": { 2024-04-26T23:09:49.806 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:09:49.806 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "a9827531-70a9-4d6f-b12a-2d71fa886d03", 2024-04-26T23:09:49.806 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:09:49.806 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:49.806 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:49.806 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:49.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.803+0000 7fb4fa200640 1 -- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4d406c8f0 msgr2=0x7fb4d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.803+0000 7fb4fa200640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4d406c8f0 0x7fb4d406edb0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0001200 tx=0x7fb4e0009040 comp rx=0 tx=0).stop 2024-04-26T23:09:49.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.803+0000 7fb4fa200640 1 -- 172.21.15.40:0/1050815751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d80050a0 msgr2=0x7fb4f41584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:49.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.803+0000 7fb4fa200640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d80050a0 0x7fb4f41584a0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4007f80 tx=0x7fb4e4011660 comp rx=0 tx=0).stop 2024-04-26T23:09:49.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.803+0000 7fb4fa200640 1 -- 172.21.15.40:0/1050815751 shutdown_connections 2024-04-26T23:09:49.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.803+0000 7fb4fa200640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb4d406c8f0 0x7fb4d406edb0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.803+0000 7fb4fa200640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f40b5980 0x7fb4f40b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.803+0000 7fb4fa200640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4f40c7610 0x7fb4f41f3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.803+0000 7fb4fa200640 1 --2- 172.21.15.40:0/1050815751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d80050a0 0x7fb4f41584a0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:49.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.803+0000 7fb4fa200640 1 -- 172.21.15.40:0/1050815751 >> 172.21.15.40:0/1050815751 conn(0x7fb4f40d0be0 msgr2=0x7fb4f40cde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:49.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.803+0000 7fb4fa200640 1 -- 172.21.15.40:0/1050815751 shutdown_connections 2024-04-26T23:09:49.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:49.803+0000 7fb4fa200640 1 -- 172.21.15.40:0/1050815751 wait complete. 2024-04-26T23:09:49.822 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:09:49.823 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-26T23:09:49.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-26T23:09:50.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.037+0000 7f7913200640 1 Processor -- start 2024-04-26T23:09:50.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.037+0000 7f7913200640 1 -- start start 2024-04-26T23:09:50.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.039+0000 7f7913200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f79140b4e60 0x7f7914071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.039+0000 7f7913200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7914072220 0x7f791406dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.039+0000 7f7913200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f791406f800 con 0x7f791406f420 2024-04-26T23:09:50.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.039+0000 7f7913200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f7914072600 con 0x7f7914072220 2024-04-26T23:09:50.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.039+0000 7f7913200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f79140b3dc0 con 0x7f79140b4e60 2024-04-26T23:09:50.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.039+0000 7f7912800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7914072220 0x7f791406dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.039+0000 7f7912800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7914072220 0x7f791406dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40104/0 (socket says 172.21.15.40:40104) 2024-04-26T23:09:50.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.039+0000 7f7912800640 1 -- 172.21.15.40:0/3621838912 learned_addr learned my addr 172.21.15.40:0/3621838912 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:50.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.040+0000 7f7911400640 1 --2- 172.21.15.40:0/3621838912 >> v2:172.21.15.103:3300/0 conn(0x7f79140b4e60 0x7f7914071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.040+0000 7f7912800640 1 -- 172.21.15.40:0/3621838912 >> v2:172.21.15.103:3300/0 conn(0x7f79140b4e60 msgr2=0x7f7914071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:50.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.040+0000 7f7912800640 1 --2- 172.21.15.40:0/3621838912 >> v2:172.21.15.103:3300/0 conn(0x7f79140b4e60 0x7f7914071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.040+0000 7f7912800640 1 -- 172.21.15.40:0/3621838912 >> v1:172.21.15.103:6789/0 conn(0x7f791406f420 legacy=0x7f79141644c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:50.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.040+0000 7f7912800640 1 -- 172.21.15.40:0/3621838912 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79140b40a0 con 0x7f7914072220 2024-04-26T23:09:50.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.041+0000 7f7912800640 1 --2- 172.21.15.40:0/3621838912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7914072220 0x7f791406dd40 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f79140b0640 tx=0x7f790c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afddc84a539cc69f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:50.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.042+0000 7f7910a00640 1 -- 172.21.15.40:0/3621838912 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c036070 con 0x7f7914072220 2024-04-26T23:09:50.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.042+0000 7f7910a00640 1 -- 172.21.15.40:0/3621838912 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f790c031600 con 0x7f7914072220 2024-04-26T23:09:50.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.042+0000 7f7910a00640 1 -- 172.21.15.40:0/3621838912 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c0319d0 con 0x7f7914072220 2024-04-26T23:09:50.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.042+0000 7f7913200640 1 -- 172.21.15.40:0/3621838912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7914072220 msgr2=0x7f791406dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.042+0000 7f7913200640 1 --2- 172.21.15.40:0/3621838912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7914072220 0x7f791406dd40 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f79140b0640 tx=0x7f790c02a6f0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.042+0000 7f7913200640 1 -- 172.21.15.40:0/3621838912 shutdown_connections 2024-04-26T23:09:50.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.042+0000 7f7913200640 1 --2- 172.21.15.40:0/3621838912 >> v2:172.21.15.103:3300/0 conn(0x7f79140b4e60 0x7f7914071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.042+0000 7f7913200640 1 --2- 172.21.15.40:0/3621838912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7914072220 0x7f791406dd40 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.042+0000 7f7913200640 1 -- 172.21.15.40:0/3621838912 >> 172.21.15.40:0/3621838912 conn(0x7f79140b2720 msgr2=0x7f79140b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:50.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.042+0000 7f7913200640 1 -- 172.21.15.40:0/3621838912 shutdown_connections 2024-04-26T23:09:50.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.042+0000 7f7913200640 1 -- 172.21.15.40:0/3621838912 wait complete. 2024-04-26T23:09:50.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.044+0000 7f7913200640 1 Processor -- start 2024-04-26T23:09:50.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.044+0000 7f7913200640 1 -- start start 2024-04-26T23:09:50.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.045+0000 7f7913200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f791406f420 0x7f79140ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.045+0000 7f7913200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7914072220 0x7f7914158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.045+0000 7f7913200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79140b4e60 0x7f7914156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.045+0000 7f7913200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f79140b4940 con 0x7f79140b4e60 2024-04-26T23:09:50.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.045+0000 7f7913200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79140b4660 con 0x7f791406f420 2024-04-26T23:09:50.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.045+0000 7f7913200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7914073f50 con 0x7f7914072220 2024-04-26T23:09:50.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.045+0000 7f7912800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79140b4e60 0x7f7914156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.046+0000 7f7912800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79140b4e60 0x7f7914156f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40120/0 (socket says 172.21.15.40:40120) 2024-04-26T23:09:50.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.046+0000 7f7912800640 1 -- 172.21.15.40:0/3145200564 learned_addr learned my addr 172.21.15.40:0/3145200564 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:50.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.046+0000 7f7911e00640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f791406f420 0x7f79140ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.046+0000 7f7911400640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7914072220 0x7f7914158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.046+0000 7f7912800640 1 -- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7914072220 msgr2=0x7f7914158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.046+0000 7f7912800640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7914072220 0x7f7914158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.046+0000 7f7912800640 1 -- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f791406f420 msgr2=0x7f79140ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.046+0000 7f7912800640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f791406f420 0x7f79140ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.046+0000 7f7912800640 1 -- 172.21.15.40:0/3145200564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7914158770 con 0x7f79140b4e60 2024-04-26T23:09:50.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.046+0000 7f7911400640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7914072220 0x7f7914158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:50.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.046+0000 7f7912800640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79140b4e60 0x7f7914156f60 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f790c02ace0 tx=0x7f790c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:50.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.047+0000 7f790a000640 1 -- 172.21.15.40:0/3145200564 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c036050 con 0x7f79140b4e60 2024-04-26T23:09:50.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.047+0000 7f790a000640 1 -- 172.21.15.40:0/3145200564 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f790c042070 con 0x7f79140b4e60 2024-04-26T23:09:50.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.047+0000 7f7913200640 1 -- 172.21.15.40:0/3145200564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7914158a20 con 0x7f79140b4e60 2024-04-26T23:09:50.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.047+0000 7f790a000640 1 -- 172.21.15.40:0/3145200564 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c009510 con 0x7f79140b4e60 2024-04-26T23:09:50.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.047+0000 7f7913200640 1 -- 172.21.15.40:0/3145200564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7914158fe0 con 0x7f79140b4e60 2024-04-26T23:09:50.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.049+0000 7f790a000640 1 -- 172.21.15.40:0/3145200564 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f790c007c50 con 0x7f79140b4e60 2024-04-26T23:09:50.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.050+0000 7f790a000640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f78f406c8f0 0x7f78f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.050+0000 7f790a000640 1 -- 172.21.15.40:0/3145200564 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f790c0accc0 con 0x7f79140b4e60 2024-04-26T23:09:50.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.051+0000 7f7911e00640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f78f406c8f0 0x7f78f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.051+0000 7f791bdb8740 1 Processor -- start 2024-04-26T23:09:50.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.051+0000 7f791bdb8740 1 -- start start 2024-04-26T23:09:50.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.052+0000 7f7911e00640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f78f406c8f0 0x7f78f406edb0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f7904004ea0 tx=0x7f7904009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:50.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.052+0000 7f791bdb8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec05d710 0x5601ec05daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.052+0000 7f791bdb8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec07afb0 0x5601ec07b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.052+0000 7f791bdb8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec0333f0 0x5601ec079670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.052+0000 7f791bdb8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5601ebf43030 con 0x5601ec0333f0 2024-04-26T23:09:50.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.052+0000 7f791bdb8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5601ebf5e7f0 con 0x5601ec05d710 2024-04-26T23:09:50.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.052+0000 7f791bdb8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5601ebf197e0 con 0x5601ec07afb0 2024-04-26T23:09:50.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.052+0000 7f7912800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec0333f0 0x5601ec079670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.052+0000 7f7912800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec0333f0 0x5601ec079670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40130/0 (socket says 172.21.15.40:40130) 2024-04-26T23:09:50.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.052+0000 7f7912800640 1 -- 172.21.15.40:0/2919880750 learned_addr learned my addr 172.21.15.40:0/2919880750 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:50.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.053+0000 7f7911400640 1 --2- 172.21.15.40:0/2919880750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec07afb0 0x5601ec07b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.053+0000 7f7911e00640 1 --2- 172.21.15.40:0/2919880750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec05d710 0x5601ec05daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.053+0000 7f7912800640 1 -- 172.21.15.40:0/2919880750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec07afb0 msgr2=0x5601ec07b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:50.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.053+0000 7f7912800640 1 --2- 172.21.15.40:0/2919880750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec07afb0 0x5601ec07b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.053+0000 7f7912800640 1 -- 172.21.15.40:0/2919880750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec05d710 msgr2=0x5601ec05daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:50.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.053+0000 7f7912800640 1 --2- 172.21.15.40:0/2919880750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec05d710 0x5601ec05daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.053+0000 7f7912800640 1 -- 172.21.15.40:0/2919880750 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601ebf25050 con 0x5601ec0333f0 2024-04-26T23:09:50.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.053+0000 7f7912800640 1 --2- 172.21.15.40:0/2919880750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec0333f0 0x5601ec079670 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f790c082e50 tx=0x7f790c076b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c304486c904fcd64 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:50.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.053+0000 7f78fea00640 1 -- 172.21.15.40:0/2919880750 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c0a9070 con 0x5601ec0333f0 2024-04-26T23:09:50.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.053+0000 7f78fea00640 1 -- 172.21.15.40:0/2919880750 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f790c037040 con 0x5601ec0333f0 2024-04-26T23:09:50.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.053+0000 7f78fea00640 1 -- 172.21.15.40:0/2919880750 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c031600 con 0x5601ec0333f0 2024-04-26T23:09:50.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.054+0000 7f791bdb8740 1 -- 172.21.15.40:0/2919880750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec0333f0 msgr2=0x5601ec079670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.054+0000 7f791bdb8740 1 --2- 172.21.15.40:0/2919880750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec0333f0 0x5601ec079670 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f790c082e50 tx=0x7f790c076b70 comp rx=0 tx=0).stop 2024-04-26T23:09:50.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.054+0000 7f791bdb8740 1 -- 172.21.15.40:0/2919880750 shutdown_connections 2024-04-26T23:09:50.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.054+0000 7f791bdb8740 1 --2- 172.21.15.40:0/2919880750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec0333f0 0x5601ec079670 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.054+0000 7f791bdb8740 1 --2- 172.21.15.40:0/2919880750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec07afb0 0x5601ec07b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.054+0000 7f791bdb8740 1 --2- 172.21.15.40:0/2919880750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec05d710 0x5601ec05daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.054+0000 7f791bdb8740 1 -- 172.21.15.40:0/2919880750 >> 172.21.15.40:0/2919880750 conn(0x5601ec02fd20 msgr2=0x5601ec078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:50.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.054+0000 7f791bdb8740 1 -- 172.21.15.40:0/2919880750 shutdown_connections 2024-04-26T23:09:50.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.054+0000 7f791bdb8740 1 -- 172.21.15.40:0/2919880750 wait complete. 2024-04-26T23:09:50.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.055+0000 7f791bdb8740 1 -- start start 2024-04-26T23:09:50.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.055+0000 7f791bdb8740 1 Processor -- start 2024-04-26T23:09:50.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.056+0000 7f791bdb8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 0x5601ec04b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.056+0000 7f791bdb8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec0604a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.056+0000 7f791bdb8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 0x5601ec058e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.056+0000 7f791bdb8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5601ebf9b200 con 0x5601ec07afb0 2024-04-26T23:09:50.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.056+0000 7f791bdb8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5601ebf95ee0 con 0x5601ec0589a0 2024-04-26T23:09:50.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.056+0000 7f791bdb8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5601ebfa2e70 con 0x5601ec05d710 2024-04-26T23:09:50.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.056+0000 7f791bdb8740 10 client.0 _ll_register_callbacks cb 0x5601ebf8c270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:50.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.057+0000 7f7911400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec0604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.057+0000 7f7911400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec0604a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40136/0 (socket says 172.21.15.40:40136) 2024-04-26T23:09:50.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.057+0000 7f7911400640 1 -- 172.21.15.40:0/732707113 learned_addr learned my addr 172.21.15.40:0/732707113 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:50.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.057+0000 7f7911e00640 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 0x5601ec04b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.057+0000 7f7912800640 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 0x5601ec058e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.057+0000 7f7911400640 1 -- 172.21.15.40:0/732707113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 msgr2=0x5601ec04b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.057+0000 7f7911400640 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 0x5601ec04b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.057+0000 7f7911400640 1 -- 172.21.15.40:0/732707113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 msgr2=0x5601ec058e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.057+0000 7f7911400640 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 0x5601ec058e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.057+0000 7f7911400640 1 -- 172.21.15.40:0/732707113 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601ebd727d0 con 0x5601ec07afb0 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.057+0000 7f7912800640 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 0x5601ec058e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.057+0000 7f7911400640 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec0604a0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f790000deb0 tx=0x7f790000f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d287646a25622 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.057+0000 7f78f3400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.058+0000 7f78f3400640 1 -- 172.21.15.40:0/732707113 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7900018020 con 0x5601ec07afb0 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.058+0000 7f78f3400640 1 -- 172.21.15.40:0/732707113 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f7900004230 con 0x5601ec07afb0 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.058+0000 7f791bdb8740 1 -- 172.21.15.40:0/732707113 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f790c037040 con 0x5601ec07afb0 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.058+0000 7f78f3400640 1 -- 172.21.15.40:0/732707113 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f7900022050 con 0x5601ec07afb0 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.058+0000 7f791bdb8740 10 client.9651 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.058+0000 7f791bdb8740 1 -- 172.21.15.40:0/732707113 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x5601ebca83f0 con 0x5601ec07afb0 2024-04-26T23:09:50.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.058+0000 7f78f3400640 1 -- 172.21.15.40:0/732707113 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f790c037040 con 0x5601ec07afb0 2024-04-26T23:09:50.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.058+0000 7f78f3400640 1 -- 172.21.15.40:0/732707113 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7f7900025020 con 0x5601ec07afb0 2024-04-26T23:09:50.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.059+0000 7f791bdb8740 10 client.9651 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:50.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.059+0000 7f791bdb8740 10 client.9651 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:50.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.059+0000 7f791bdb8740 20 client.9651 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:50.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.059+0000 7f791bdb8740 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5601ec048b00 0x5601ec048f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.059+0000 7f791bdb8740 4 client.9651 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:09:50.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.059+0000 7f791bdb8740 1 -- 172.21.15.40:0/732707113 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601ebfa47e0 con 0x5601ec048b00 2024-04-26T23:09:50.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.059+0000 7f7911e00640 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5601ec048b00 0x5601ec048f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.059+0000 7f7911e00640 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5601ec048b00 0x5601ec048f10 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8700ae8f5f87f85b server_cookie=162181bc8b134e57 in_seq=0 out_seq=0 2024-04-26T23:09:50.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.059+0000 7f78f3400640 10 client.9651 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:50.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.060+0000 7f78f3400640 1 -- 172.21.15.40:0/732707113 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5601ebfa47e0 con 0x5601ec048b00 2024-04-26T23:09:50.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.060+0000 7f78f3400640 10 client.9651 handle_command_reply: tid=0 2024-04-26T23:09:50.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.060+0000 7f791bdb8740 1 client.9651 shutdown 2024-04-26T23:09:50.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.060+0000 7f791bdb8740 1 -- 172.21.15.40:0/732707113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 msgr2=0x5601ec0604a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.061+0000 7f791bdb8740 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec0604a0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f790000deb0 tx=0x7f790000f5b0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.061+0000 7f791bdb8740 1 -- 172.21.15.40:0/732707113 shutdown_connections 2024-04-26T23:09:50.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.061+0000 7f791bdb8740 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5601ec048b00 0x5601ec048f10 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.061+0000 7f791bdb8740 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 0x5601ec058e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.061+0000 7f791bdb8740 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec0604a0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.061+0000 7f791bdb8740 1 --2- 172.21.15.40:0/732707113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 0x5601ec04b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.061+0000 7f78f3400640 0 client.9651 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:50.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.061+0000 7f791bdb8740 1 -- 172.21.15.40:0/732707113 >> 172.21.15.40:0/732707113 conn(0x5601ec0333f0 msgr2=0x5601ec0328e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:50.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.061+0000 7f791bdb8740 1 -- 172.21.15.40:0/732707113 shutdown_connections 2024-04-26T23:09:50.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.061+0000 7f791bdb8740 1 -- 172.21.15.40:0/732707113 wait complete. 2024-04-26T23:09:50.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.061+0000 7f791bdb8740 20 client.9651 trim_cache size 0 max 16384 2024-04-26T23:09:50.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.080+0000 7f791bdb8740 1 Processor -- start 2024-04-26T23:09:50.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.080+0000 7f791bdb8740 1 -- start start 2024-04-26T23:09:50.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.081+0000 7f791bdb8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 0x5601ec060ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.081+0000 7f791bdb8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 0x5601ec0339e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.081+0000 7f791bdb8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec05fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.081+0000 7f791bdb8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7914206180 con 0x5601ec07afb0 2024-04-26T23:09:50.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.081+0000 7f791bdb8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7914205320 con 0x5601ec0589a0 2024-04-26T23:09:50.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.081+0000 7f791bdb8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7914205a50 con 0x5601ec05d710 2024-04-26T23:09:50.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.081+0000 7f7912800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec05fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.082+0000 7f7912800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec05fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40142/0 (socket says 172.21.15.40:40142) 2024-04-26T23:09:50.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.082+0000 7f7912800640 1 -- 172.21.15.40:0/4040059014 learned_addr learned my addr 172.21.15.40:0/4040059014 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:50.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.082+0000 7f7911400640 1 --2- 172.21.15.40:0/4040059014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 0x5601ec0339e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.082+0000 7f7911e00640 1 --2- 172.21.15.40:0/4040059014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 0x5601ec060ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.082+0000 7f7912800640 1 -- 172.21.15.40:0/4040059014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 msgr2=0x5601ec0339e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:50.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.082+0000 7f7912800640 1 --2- 172.21.15.40:0/4040059014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 0x5601ec0339e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.082+0000 7f7912800640 1 -- 172.21.15.40:0/4040059014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 msgr2=0x5601ec060ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:50.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.082+0000 7f7912800640 1 --2- 172.21.15.40:0/4040059014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 0x5601ec060ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.082+0000 7f7912800640 1 -- 172.21.15.40:0/4040059014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601ec049280 con 0x5601ec07afb0 2024-04-26T23:09:50.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.082+0000 7f7911e00640 1 --2- 172.21.15.40:0/4040059014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 0x5601ec060ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:50.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.082+0000 7f7912800640 1 --2- 172.21.15.40:0/4040059014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec05fcd0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f790c049010 tx=0x7f790c07d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865a10beec9fbbc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:50.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.083+0000 7f78fe000640 1 -- 172.21.15.40:0/4040059014 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c0a9070 con 0x5601ec07afb0 2024-04-26T23:09:50.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.083+0000 7f78fe000640 1 -- 172.21.15.40:0/4040059014 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f790c097070 con 0x5601ec07afb0 2024-04-26T23:09:50.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.083+0000 7f78fe000640 1 -- 172.21.15.40:0/4040059014 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c07aea0 con 0x5601ec07afb0 2024-04-26T23:09:50.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.083+0000 7f791bdb8740 1 -- 172.21.15.40:0/4040059014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 msgr2=0x5601ec05fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.083+0000 7f791bdb8740 1 --2- 172.21.15.40:0/4040059014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec05fcd0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f790c049010 tx=0x7f790c07d000 comp rx=0 tx=0).stop 2024-04-26T23:09:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.083+0000 7f791bdb8740 1 -- 172.21.15.40:0/4040059014 shutdown_connections 2024-04-26T23:09:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.083+0000 7f791bdb8740 1 --2- 172.21.15.40:0/4040059014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec05fcd0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.083+0000 7f791bdb8740 1 --2- 172.21.15.40:0/4040059014 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 0x5601ec0339e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.083+0000 7f791bdb8740 1 --2- 172.21.15.40:0/4040059014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 0x5601ec060ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.083+0000 7f791bdb8740 1 -- 172.21.15.40:0/4040059014 >> 172.21.15.40:0/4040059014 conn(0x5601ec0333f0 msgr2=0x5601ec09c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.083+0000 7f791bdb8740 1 -- 172.21.15.40:0/4040059014 shutdown_connections 2024-04-26T23:09:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.083+0000 7f791bdb8740 1 -- 172.21.15.40:0/4040059014 wait complete. 2024-04-26T23:09:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.084+0000 7f791bdb8740 1 -- start start 2024-04-26T23:09:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.084+0000 7f791bdb8740 1 Processor -- start 2024-04-26T23:09:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.085+0000 7f791bdb8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 0x5601ec079c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.085+0000 7f791bdb8740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 0x5601ec055500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.085+0000 7f791bdb8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec0ec4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.085+0000 7f791bdb8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5601ebfa3fd0 con 0x5601ec07afb0 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.085+0000 7f791bdb8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79140b14b0 con 0x5601ec0589a0 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.085+0000 7f791bdb8740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5601ec061430 con 0x5601ec05d710 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.085+0000 7f791bdb8740 10 client.0 _ll_register_callbacks cb 0x7f790000d160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.086+0000 7f7912800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec0ec4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.086+0000 7f7912800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec0ec4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40146/0 (socket says 172.21.15.40:40146) 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.086+0000 7f7912800640 1 -- 172.21.15.40:0/1869885105 learned_addr learned my addr 172.21.15.40:0/1869885105 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.086+0000 7f7911400640 1 --2- 172.21.15.40:0/1869885105 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 0x5601ec055500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.086+0000 7f7912800640 1 -- 172.21.15.40:0/1869885105 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 msgr2=0x5601ec055500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.086+0000 7f7912800640 1 --2- 172.21.15.40:0/1869885105 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 0x5601ec055500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.086+0000 7f7912800640 1 -- 172.21.15.40:0/1869885105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 msgr2=0x5601ec079c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.086+0000 7f7912800640 1 --2- 172.21.15.40:0/1869885105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 0x5601ec079c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.086+0000 7f7912800640 1 -- 172.21.15.40:0/1869885105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601ebc680f0 con 0x5601ec07afb0 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.086+0000 7f7912800640 1 --2- 172.21.15.40:0/1869885105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec0ec4f0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f790c076bb0 tx=0x7f790c009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ac251a48c05073 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.086+0000 7f78f3400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:09:50.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.086+0000 7f78f3400640 1 -- 172.21.15.40:0/1869885105 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c0a9040 con 0x5601ec07afb0 2024-04-26T23:09:50.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f78f3400640 1 -- 172.21.15.40:0/1869885105 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f790c078aa0 con 0x5601ec07afb0 2024-04-26T23:09:50.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f791bdb8740 1 -- 172.21.15.40:0/1869885105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f790c097070 con 0x5601ec07afb0 2024-04-26T23:09:50.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f78f3400640 1 -- 172.21.15.40:0/1869885105 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) ==== 24+0+0 (secure 0 0 0) 0x7f790c081070 con 0x5601ec07afb0 2024-04-26T23:09:50.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f791bdb8740 10 client.9657 fetch_fsmap learned FSMap version 122 2024-04-26T23:09:50.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f791bdb8740 1 -- 172.21.15.40:0/1869885105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) -- 0x5601ebf5ea30 con 0x5601ec07afb0 2024-04-26T23:09:50.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f78f3400640 1 -- 172.21.15.40:0/1869885105 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f790c097070 con 0x5601ec07afb0 2024-04-26T23:09:50.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f78f3400640 1 -- 172.21.15.40:0/1869885105 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 122) ==== 3550+0+0 (secure 0 0 0) 0x7f790c07fce0 con 0x5601ec07afb0 2024-04-26T23:09:50.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f791bdb8740 10 client.9657 fetch_fsmap finished waiting for FSMap version 122 2024-04-26T23:09:50.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f791bdb8740 10 client.9657 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-26T23:09:50.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f791bdb8740 20 client.9657 populate_metadata read hostname 'smithi040' 2024-04-26T23:09:50.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f791bdb8740 1 --2- 172.21.15.40:0/1869885105 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5601ec0e58d0 0x5601ec0e5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f791bdb8740 4 client.9657 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-26T23:09:50.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f791bdb8740 1 -- 172.21.15.40:0/1869885105 --> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5601ebf9e650 con 0x5601ec0e58d0 2024-04-26T23:09:50.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.087+0000 7f7911e00640 1 --2- 172.21.15.40:0/1869885105 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5601ec0e58d0 0x5601ec0e5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f7911e00640 1 --2- 172.21.15.40:0/1869885105 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5601ec0e58d0 0x5601ec0e5ce0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c59a245dff2df85a server_cookie=5b9627404f8190e in_seq=0 out_seq=0 2024-04-26T23:09:50.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f78f3400640 10 client.9657 ms_handle_connect on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:50.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f78f3400640 1 -- 172.21.15.40:0/1869885105 <== mds.0 v2:172.21.15.40:6834/1138762604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5601ebf9e650 con 0x5601ec0e58d0 2024-04-26T23:09:50.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f78f3400640 10 client.9657 handle_command_reply: tid=0 2024-04-26T23:09:50.092 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f36ffc00640 -1 log_channel(cluster) log [ERR] : Health check update: 3 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-26T23:09:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f791bdb8740 1 client.9657 shutdown 2024-04-26T23:09:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f791bdb8740 1 -- 172.21.15.40:0/1869885105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 msgr2=0x5601ec0ec4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f791bdb8740 1 --2- 172.21.15.40:0/1869885105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec0ec4f0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f790c076bb0 tx=0x7f790c009810 comp rx=0 tx=0).stop 2024-04-26T23:09:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f791bdb8740 1 -- 172.21.15.40:0/1869885105 shutdown_connections 2024-04-26T23:09:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f791bdb8740 1 --2- 172.21.15.40:0/1869885105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601ec07afb0 0x5601ec0ec4f0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f791bdb8740 1 --2- 172.21.15.40:0/1869885105 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5601ec05d710 0x5601ec055500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f791bdb8740 1 --2- 172.21.15.40:0/1869885105 >> [v2:172.21.15.40:6834/1138762604,v1:172.21.15.40:6835/1138762604] conn(0x5601ec0e58d0 0x5601ec0e5ce0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f791bdb8740 1 --2- 172.21.15.40:0/1869885105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601ec0589a0 0x5601ec079c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f791bdb8740 1 -- 172.21.15.40:0/1869885105 >> 172.21.15.40:0/1869885105 conn(0x5601ec0333f0 msgr2=0x5601ec0339e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.088+0000 7f78f3400640 0 client.9657 ms_handle_reset on v2:172.21.15.40:6834/1138762604 2024-04-26T23:09:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.089+0000 7f791bdb8740 1 -- 172.21.15.40:0/1869885105 shutdown_connections 2024-04-26T23:09:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.089+0000 7f791bdb8740 1 -- 172.21.15.40:0/1869885105 wait complete. 2024-04-26T23:09:50.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.089+0000 7f791bdb8740 20 client.9657 trim_cache size 0 max 16384 2024-04-26T23:09:50.094 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:09:50.094 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-26T23:09:50.094 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-04-26T23:09:50.094 INFO:teuthology.orchestra.run.smithi040.stdout: "a9827531-70a9-4d6f-b12a-2d71fa886d03": { 2024-04-26T23:09:50.094 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/d1/d2/d3", 2024-04-26T23:09:50.094 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "a9827531-70a9-4d6f-b12a-2d71fa886d03", 2024-04-26T23:09:50.094 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-04-26T23:09:50.094 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:50.095 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:09:50.095 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:09:50.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.090+0000 7f7913200640 1 -- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f78f406c8f0 msgr2=0x7f78f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.090+0000 7f7913200640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f78f406c8f0 0x7f78f406edb0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f7904004ea0 tx=0x7f7904009040 comp rx=0 tx=0).stop 2024-04-26T23:09:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.090+0000 7f7913200640 1 -- 172.21.15.40:0/3145200564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79140b4e60 msgr2=0x7f7914156f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.090+0000 7f7913200640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79140b4e60 0x7f7914156f60 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f790c02ace0 tx=0x7f790c004a40 comp rx=0 tx=0).stop 2024-04-26T23:09:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.090+0000 7f7913200640 1 -- 172.21.15.40:0/3145200564 shutdown_connections 2024-04-26T23:09:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.090+0000 7f7913200640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f78f406c8f0 0x7f78f406edb0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.090+0000 7f7913200640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f79140b4e60 0x7f7914156f60 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.090+0000 7f7913200640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7914072220 0x7f7914158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.090+0000 7f7913200640 1 --2- 172.21.15.40:0/3145200564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f791406f420 0x7f79140ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.090+0000 7f7913200640 1 -- 172.21.15.40:0/3145200564 >> 172.21.15.40:0/3145200564 conn(0x7f79140b2720 msgr2=0x7f791406dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.090+0000 7f7913200640 1 -- 172.21.15.40:0/3145200564 shutdown_connections 2024-04-26T23:09:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.090+0000 7f7913200640 1 -- 172.21.15.40:0/3145200564 wait complete. 2024-04-26T23:09:50.108 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-26T23:09:50.108 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:50.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.322+0000 7f23e9a00640 1 Processor -- start 2024-04-26T23:09:50.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.322+0000 7f23e9a00640 1 -- start start 2024-04-26T23:09:50.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.323+0000 7f23e9a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f23e4156460 0x7f23e40cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.323+0000 7f23e9a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f23e40cba10 0x7f23e406f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.324+0000 7f23e9a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23e40700e0 con 0x7f23e406fcd0 2024-04-26T23:09:50.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.324+0000 7f23e9a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f23e4156840 con 0x7f23e40cba10 2024-04-26T23:09:50.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.324+0000 7f23e9a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f23e40cbf20 con 0x7f23e4156460 2024-04-26T23:09:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.324+0000 7f23e2000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f23e40cba10 0x7f23e406f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.324+0000 7f23e3400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f23e406fcd0 0x7f23e415f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35794/0 (socket says 172.21.15.40:35794) 2024-04-26T23:09:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.324+0000 7f23e3400640 1 -- 172.21.15.40:0/3745606146 learned_addr learned my addr 172.21.15.40:0/3745606146 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.324+0000 7f23e2a00640 1 --2- 172.21.15.40:0/3745606146 >> v2:172.21.15.103:3300/0 conn(0x7f23e4156460 0x7f23e40cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.324+0000 7f23e2000640 1 -- 172.21.15.40:0/3745606146 >> v2:172.21.15.103:3300/0 conn(0x7f23e4156460 msgr2=0x7f23e40cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.324+0000 7f23e2000640 1 --2- 172.21.15.40:0/3745606146 >> v2:172.21.15.103:3300/0 conn(0x7f23e4156460 0x7f23e40cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.324+0000 7f23e2000640 1 -- 172.21.15.40:0/3745606146 >> v1:172.21.15.40:6789/0 conn(0x7f23e406fcd0 legacy=0x7f23e415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.324+0000 7f23e2000640 1 -- 172.21.15.40:0/3745606146 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e40cc200 con 0x7f23e40cba10 2024-04-26T23:09:50.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.326+0000 7f23e2000640 1 --2- 172.21.15.40:0/3745606146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e40cba10 0x7f23e406f790 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f23e40b05d0 tx=0x7f23d002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a737de58ecf08971 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:50.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.326+0000 7f23e1600640 1 -- 172.21.15.40:0/3745606146 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d0037070 con 0x7f23e40cba10 2024-04-26T23:09:50.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.326+0000 7f23e1600640 1 -- 172.21.15.40:0/3745606146 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23d0032440 con 0x7f23e40cba10 2024-04-26T23:09:50.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.326+0000 7f23e1600640 1 -- 172.21.15.40:0/3745606146 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d0032810 con 0x7f23e40cba10 2024-04-26T23:09:50.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.326+0000 7f23e9a00640 1 -- 172.21.15.40:0/3745606146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e40cba10 msgr2=0x7f23e406f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.326+0000 7f23e9a00640 1 --2- 172.21.15.40:0/3745606146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e40cba10 0x7f23e406f790 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f23e40b05d0 tx=0x7f23d002a1a0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.327+0000 7f23e9a00640 1 -- 172.21.15.40:0/3745606146 shutdown_connections 2024-04-26T23:09:50.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.327+0000 7f23e9a00640 1 --2- 172.21.15.40:0/3745606146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e40cba10 0x7f23e406f790 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.327+0000 7f23e9a00640 1 --2- 172.21.15.40:0/3745606146 >> v2:172.21.15.103:3300/0 conn(0x7f23e4156460 0x7f23e40cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.327+0000 7f23e9a00640 1 -- 172.21.15.40:0/3745606146 >> 172.21.15.40:0/3745606146 conn(0x7f23e40d0be0 msgr2=0x7f23e40d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:50.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.327+0000 7f23e9a00640 1 -- 172.21.15.40:0/3745606146 shutdown_connections 2024-04-26T23:09:50.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.327+0000 7f23e9a00640 1 -- 172.21.15.40:0/3745606146 wait complete. 2024-04-26T23:09:50.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.329+0000 7f23e9a00640 1 Processor -- start 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.329+0000 7f23e9a00640 1 -- start start 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e9a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e406fcd0 0x7f23e40b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e9a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e40cba10 0x7f23e40c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4156460 0x7f23e414fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e9a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f23e4072700 con 0x7f23e4156460 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e9a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f23e40cc7c0 con 0x7f23e40cba10 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e9a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f23e406e260 con 0x7f23e406fcd0 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4156460 0x7f23e414fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4156460 0x7f23e414fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40166/0 (socket says 172.21.15.40:40166) 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e3400640 1 -- 172.21.15.40:0/1818819026 learned_addr learned my addr 172.21.15.40:0/1818819026 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e2000640 1 --2- 172.21.15.40:0/1818819026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e40cba10 0x7f23e40c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e3400640 1 -- 172.21.15.40:0/1818819026 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e406fcd0 msgr2=0x7f23e40b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e3400640 1 --2- 172.21.15.40:0/1818819026 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e406fcd0 0x7f23e40b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e3400640 1 -- 172.21.15.40:0/1818819026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e40cba10 msgr2=0x7f23e40c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.330+0000 7f23e3400640 1 --2- 172.21.15.40:0/1818819026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e40cba10 0x7f23e40c9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.331+0000 7f23e3400640 1 -- 172.21.15.40:0/1818819026 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e40c5f40 con 0x7f23e4156460 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.331+0000 7f23e3400640 1 --2- 172.21.15.40:0/1818819026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4156460 0x7f23e414fb40 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00cbc0 tx=0x7f23dc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.331+0000 7f23daa00640 1 -- 172.21.15.40:0/1818819026 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc017070 con 0x7f23e4156460 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.331+0000 7f23daa00640 1 -- 172.21.15.40:0/1818819026 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23dc00ad40 con 0x7f23e4156460 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.331+0000 7f23daa00640 1 -- 172.21.15.40:0/1818819026 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc012600 con 0x7f23e4156460 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.331+0000 7f23e9a00640 1 -- 172.21.15.40:0/1818819026 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23e40c6220 con 0x7f23e4156460 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.331+0000 7f23e9a00640 1 -- 172.21.15.40:0/1818819026 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23e40c6ce0 con 0x7f23e4156460 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.333+0000 7f23e9a00640 1 -- 172.21.15.40:0/1818819026 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23e40c6610 con 0x7f23e4156460 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.334+0000 7f23daa00640 1 -- 172.21.15.40:0/1818819026 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f23dc01a070 con 0x7f23e4156460 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.335+0000 7f23daa00640 1 --2- 172.21.15.40:0/1818819026 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f23c406c8f0 0x7f23c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.335+0000 7f23daa00640 1 -- 172.21.15.40:0/1818819026 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f23dc08b9e0 con 0x7f23e4156460 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.336+0000 7f23e2a00640 1 --2- 172.21.15.40:0/1818819026 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f23c406c8f0 0x7f23c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.337+0000 7f23e2a00640 1 --2- 172.21.15.40:0/1818819026 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f23c406c8f0 0x7f23c406edb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f23d4006fd0 tx=0x7f23d4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:50.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.340+0000 7f23daa00640 1 -- 172.21.15.40:0/1818819026 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f23e40700e0 con 0x7f23e4156460 2024-04-26T23:09:50.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.601+0000 7f23e9a00640 1 -- 172.21.15.40:0/1818819026 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23e40c63f0 con 0x7f23e4156460 2024-04-26T23:09:50.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.602+0000 7f23daa00640 1 -- 172.21.15.40:0/1818819026 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5239 (secure 0 0 0) 0x7f23dc059600 con 0x7f23e4156460 2024-04-26T23:09:50.603 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:50.603 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:44.797258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[6598,8949,8955]},"id":7}]} 2024-04-26T23:09:50.603 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 122 2024-04-26T23:09:50.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.605+0000 7f23e9a00640 1 -- 172.21.15.40:0/1818819026 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f23c406c8f0 msgr2=0x7f23c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.605+0000 7f23e9a00640 1 --2- 172.21.15.40:0/1818819026 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f23c406c8f0 0x7f23c406edb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f23d4006fd0 tx=0x7f23d4008040 comp rx=0 tx=0).stop 2024-04-26T23:09:50.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.605+0000 7f23e9a00640 1 -- 172.21.15.40:0/1818819026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4156460 msgr2=0x7f23e414fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.605+0000 7f23e9a00640 1 --2- 172.21.15.40:0/1818819026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4156460 0x7f23e414fb40 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00cbc0 tx=0x7f23dc00a430 comp rx=0 tx=0).stop 2024-04-26T23:09:50.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.606+0000 7f23e9a00640 1 -- 172.21.15.40:0/1818819026 shutdown_connections 2024-04-26T23:09:50.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.606+0000 7f23e9a00640 1 --2- 172.21.15.40:0/1818819026 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f23c406c8f0 0x7f23c406edb0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.606+0000 7f23e9a00640 1 --2- 172.21.15.40:0/1818819026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4156460 0x7f23e414fb40 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.606+0000 7f23e9a00640 1 --2- 172.21.15.40:0/1818819026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e40cba10 0x7f23e40c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.606+0000 7f23e9a00640 1 --2- 172.21.15.40:0/1818819026 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23e406fcd0 0x7f23e40b5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.606+0000 7f23e9a00640 1 -- 172.21.15.40:0/1818819026 >> 172.21.15.40:0/1818819026 conn(0x7f23e40d0be0 msgr2=0x7f23e4073aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:50.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.606+0000 7f23e9a00640 1 -- 172.21.15.40:0/1818819026 shutdown_connections 2024-04-26T23:09:50.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.606+0000 7f23e9a00640 1 -- 172.21.15.40:0/1818819026 wait complete. 2024-04-26T23:09:50.625 DEBUG:tasks.cephfs.test_multimds_misc:current active=['a', 'c', 'd'] 2024-04-26T23:09:50.626 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:09:50.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.838+0000 7f0025e00640 1 Processor -- start 2024-04-26T23:09:50.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.839+0000 7f0025e00640 1 -- start start 2024-04-26T23:09:50.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.840+0000 7f0025e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0020145970 0x7f00201612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.840+0000 7f0025e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0020146060 0x7f00200b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.840+0000 7f0025e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f00200b3fb0 0x7f00200d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.840+0000 7f0025e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f00200ceed0 con 0x7f0020145970 2024-04-26T23:09:50.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.840+0000 7f0025e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f00200b59f0 con 0x7f00200b3fb0 2024-04-26T23:09:50.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.840+0000 7f0025e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f00201617f0 con 0x7f0020146060 2024-04-26T23:09:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.841+0000 7f0024a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0020145970 0x7f00201612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.841+0000 7f0024a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0020145970 0x7f00201612b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40170/0 (socket says 172.21.15.40:40170) 2024-04-26T23:09:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.841+0000 7f0024a00640 1 -- 172.21.15.40:0/1082407654 learned_addr learned my addr 172.21.15.40:0/1082407654 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.841+0000 7f0024a00640 1 -- 172.21.15.40:0/1082407654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0020146060 msgr2=0x7f00200b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.841+0000 7f0024a00640 1 --2- 172.21.15.40:0/1082407654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0020146060 0x7f00200b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.841+0000 7f0024a00640 1 -- 172.21.15.40:0/1082407654 >> v2:172.21.15.103:3300/0 conn(0x7f00200b3fb0 msgr2=0x7f00200d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.841+0000 7f0024a00640 1 --2- 172.21.15.40:0/1082407654 >> v2:172.21.15.103:3300/0 conn(0x7f00200b3fb0 0x7f00200d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.841+0000 7f0024a00640 1 -- 172.21.15.40:0/1082407654 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00200b5f40 con 0x7f0020145970 2024-04-26T23:09:50.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.842+0000 7f0024a00640 1 --2- 172.21.15.40:0/1082407654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0020145970 0x7f00201612b0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f0010009e50 tx=0x7f001002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96245cc781ff4db8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:50.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.843+0000 7f001f400640 1 -- 172.21.15.40:0/1082407654 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0010036070 con 0x7f0020145970 2024-04-26T23:09:50.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.843+0000 7f001f400640 1 -- 172.21.15.40:0/1082407654 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0010031600 con 0x7f0020145970 2024-04-26T23:09:50.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.843+0000 7f001f400640 1 -- 172.21.15.40:0/1082407654 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00100319d0 con 0x7f0020145970 2024-04-26T23:09:50.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.843+0000 7f0025e00640 1 -- 172.21.15.40:0/1082407654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0020145970 msgr2=0x7f00201612b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.843+0000 7f0025e00640 1 --2- 172.21.15.40:0/1082407654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0020145970 0x7f00201612b0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f0010009e50 tx=0x7f001002a6f0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.843+0000 7f0025e00640 1 -- 172.21.15.40:0/1082407654 shutdown_connections 2024-04-26T23:09:50.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.843+0000 7f0025e00640 1 --2- 172.21.15.40:0/1082407654 >> v2:172.21.15.103:3300/0 conn(0x7f00200b3fb0 0x7f00200d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.843+0000 7f0025e00640 1 --2- 172.21.15.40:0/1082407654 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0020146060 0x7f00200b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.843+0000 7f0025e00640 1 --2- 172.21.15.40:0/1082407654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0020145970 0x7f00201612b0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.844+0000 7f0025e00640 1 -- 172.21.15.40:0/1082407654 >> 172.21.15.40:0/1082407654 conn(0x7f0020075eb0 msgr2=0x7f0020068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:50.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.844+0000 7f0025e00640 1 -- 172.21.15.40:0/1082407654 shutdown_connections 2024-04-26T23:09:50.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.844+0000 7f0025e00640 1 -- 172.21.15.40:0/1082407654 wait complete. 2024-04-26T23:09:50.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.845+0000 7f0025e00640 1 Processor -- start 2024-04-26T23:09:50.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.846+0000 7f0025e00640 1 -- start start 2024-04-26T23:09:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.846+0000 7f0025e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0020145970 0x7f00200727e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.846+0000 7f0025e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0020146060 0x7f0020140a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0025e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00201f5870 0x7f00201f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0025e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00200cf3e0 con 0x7f0020145970 2024-04-26T23:09:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0025e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f00200cf100 con 0x7f00201f5870 2024-04-26T23:09:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0025e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0020145d50 con 0x7f0020146060 2024-04-26T23:09:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0024a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0020145970 0x7f00200727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0024a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0020145970 0x7f00200727e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40186/0 (socket says 172.21.15.40:40186) 2024-04-26T23:09:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0024a00640 1 -- 172.21.15.40:0/1269134554 learned_addr learned my addr 172.21.15.40:0/1269134554 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0025400640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00201f5870 0x7f00201f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f001fe00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0020146060 0x7f0020140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0024a00640 1 -- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0020146060 msgr2=0x7f0020140a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0024a00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0020146060 0x7f0020140a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0024a00640 1 -- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00201f5870 msgr2=0x7f00201f5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0024a00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00201f5870 0x7f00201f5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0024a00640 1 -- 172.21.15.40:0/1269134554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00201f6330 con 0x7f0020145970 2024-04-26T23:09:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f001fe00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0020146060 0x7f0020140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.847+0000 7f0025400640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00201f5870 0x7f00201f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.848+0000 7f0024a00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0020145970 0x7f00200727e0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f0010009e50 tx=0x7f0010004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:50.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.848+0000 7f001cc00640 1 -- 172.21.15.40:0/1269134554 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0010041020 con 0x7f0020145970 2024-04-26T23:09:50.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.848+0000 7f001cc00640 1 -- 172.21.15.40:0/1269134554 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00100094d0 con 0x7f0020145970 2024-04-26T23:09:50.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.848+0000 7f0025e00640 1 -- 172.21.15.40:0/1269134554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00201f6610 con 0x7f0020145970 2024-04-26T23:09:50.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.848+0000 7f001cc00640 1 -- 172.21.15.40:0/1269134554 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0010036040 con 0x7f0020145970 2024-04-26T23:09:50.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.848+0000 7f0025e00640 1 -- 172.21.15.40:0/1269134554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00200722c0 con 0x7f0020145970 2024-04-26T23:09:50.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.849+0000 7f0025e00640 1 -- 172.21.15.40:0/1269134554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0020069370 con 0x7f0020145970 2024-04-26T23:09:50.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.850+0000 7f001cc00640 1 -- 172.21.15.40:0/1269134554 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0010044020 con 0x7f0020145970 2024-04-26T23:09:50.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.851+0000 7f001cc00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f000006c8f0 0x7f000006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:50.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.851+0000 7f001cc00640 1 -- 172.21.15.40:0/1269134554 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f00100acbd0 con 0x7f0020145970 2024-04-26T23:09:50.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.852+0000 7f001fe00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f000006c8f0 0x7f000006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:50.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.853+0000 7f001fe00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f000006c8f0 0x7f000006edb0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f0020073380 tx=0x7f000c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:50.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:50.856+0000 7f001cc00640 1 -- 172.21.15.40:0/1269134554 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f001004b030 con 0x7f0020145970 2024-04-26T23:09:51.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.148+0000 7f0025e00640 1 -- 172.21.15.40:0/1269134554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f00200d1400 con 0x7f0020145970 2024-04-26T23:09:51.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.149+0000 7f001cc00640 1 -- 172.21.15.40:0/1269134554 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1948 (secure 0 0 0) 0x7f001007ae40 con 0x7f0020145970 2024-04-26T23:09:51.149 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:51.150 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":551,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":105,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":706274,"bytes_used":267182080,"bytes_avail":772826931200,"bytes_total":773094113280,"write_bytes_sec":818,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":122,"id":7,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":7,"rank":2,"name":"d","status":"up:active","gid":6598},{"filesystem_id":7,"rank":1,"name":"c","status":"up:active","gid":8949},{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8955}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":29,"modified":"2024-04-26T23:09:43.338130+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:09:51.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.152+0000 7f0025e00640 1 -- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f000006c8f0 msgr2=0x7f000006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:51.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.152+0000 7f0025e00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f000006c8f0 0x7f000006edb0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f0020073380 tx=0x7f000c008040 comp rx=0 tx=0).stop 2024-04-26T23:09:51.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.152+0000 7f0025e00640 1 -- 172.21.15.40:0/1269134554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0020145970 msgr2=0x7f00200727e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:51.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.152+0000 7f0025e00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0020145970 0x7f00200727e0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f0010009e50 tx=0x7f0010004a40 comp rx=0 tx=0).stop 2024-04-26T23:09:51.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.152+0000 7f0025e00640 1 -- 172.21.15.40:0/1269134554 shutdown_connections 2024-04-26T23:09:51.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.152+0000 7f0025e00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f000006c8f0 0x7f000006edb0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:51.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.152+0000 7f0025e00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00201f5870 0x7f00201f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:51.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.152+0000 7f0025e00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0020146060 0x7f0020140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:51.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.152+0000 7f0025e00640 1 --2- 172.21.15.40:0/1269134554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0020145970 0x7f00200727e0 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:51.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.152+0000 7f0025e00640 1 -- 172.21.15.40:0/1269134554 >> 172.21.15.40:0/1269134554 conn(0x7f0020075eb0 msgr2=0x7f0020142e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:51.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.152+0000 7f0025e00640 1 -- 172.21.15.40:0/1269134554 shutdown_connections 2024-04-26T23:09:51.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:51.152+0000 7f0025e00640 1 -- 172.21.15.40:0/1269134554 wait complete. 2024-04-26T23:09:51.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'paused paths [/d1/d2/d3]', 'c': 'paused paths [/d1/d2/d3]', 'd': 'paused paths [/d1/d2/d3]'} 2024-04-26T23:09:52.171 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:52.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.392+0000 7f6c9a000640 1 Processor -- start 2024-04-26T23:09:52.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.393+0000 7f6c9a000640 1 -- start start 2024-04-26T23:09:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.394+0000 7f6c9a000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6c9415f2d0 0x7f6c940afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.394+0000 7f6c9a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c940ce5e0 0x7f6c940c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.394+0000 7f6c9a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6c940c5df0 0x7f6c940cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.394+0000 7f6c9a000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f6c940af7b0 con 0x7f6c940c5df0 2024-04-26T23:09:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.394+0000 7f6c9a000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6c940afa90 con 0x7f6c9415f2d0 2024-04-26T23:09:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.394+0000 7f6c9a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6c940cbd90 con 0x7f6c940ce5e0 2024-04-26T23:09:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.395+0000 7f6c99600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6c940c5df0 0x7f6c940cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.395+0000 7f6c99600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6c940c5df0 0x7f6c940cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40188/0 (socket says 172.21.15.40:40188) 2024-04-26T23:09:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.395+0000 7f6c99600640 1 -- 172.21.15.40:0/1181070580 learned_addr learned my addr 172.21.15.40:0/1181070580 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.395+0000 7f6c93e00640 1 --2- 172.21.15.40:0/1181070580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c940ce5e0 0x7f6c940c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.395+0000 7f6c98c00640 1 --2- 172.21.15.40:0/1181070580 >> v2:172.21.15.103:3300/0 conn(0x7f6c9415f2d0 0x7f6c940afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.395+0000 7f6c99600640 1 -- 172.21.15.40:0/1181070580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c940ce5e0 msgr2=0x7f6c940c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.395+0000 7f6c99600640 1 --2- 172.21.15.40:0/1181070580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c940ce5e0 0x7f6c940c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.396+0000 7f6c99600640 1 -- 172.21.15.40:0/1181070580 >> v2:172.21.15.103:3300/0 conn(0x7f6c9415f2d0 msgr2=0x7f6c940afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.396+0000 7f6c99600640 1 --2- 172.21.15.40:0/1181070580 >> v2:172.21.15.103:3300/0 conn(0x7f6c9415f2d0 0x7f6c940afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.396+0000 7f6c99600640 1 -- 172.21.15.40:0/1181070580 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c940cc070 con 0x7f6c940c5df0 2024-04-26T23:09:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.396+0000 7f6c93e00640 1 --2- 172.21.15.40:0/1181070580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c940ce5e0 0x7f6c940c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:09:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.396+0000 7f6c98c00640 1 --2- 172.21.15.40:0/1181070580 >> v2:172.21.15.103:3300/0 conn(0x7f6c9415f2d0 0x7f6c940afdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:09:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.397+0000 7f6c99600640 1 --2- 172.21.15.40:0/1181070580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c940c5df0 0x7f6c940cc590 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f6c9406a5a0 tx=0x7f6c8c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee2f483d0bc048e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.398+0000 7f6c93400640 1 -- 172.21.15.40:0/1181070580 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c036070 con 0x7f6c940c5df0 2024-04-26T23:09:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.398+0000 7f6c93400640 1 -- 172.21.15.40:0/1181070580 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c8c031600 con 0x7f6c940c5df0 2024-04-26T23:09:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.398+0000 7f6c93400640 1 -- 172.21.15.40:0/1181070580 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0319d0 con 0x7f6c940c5df0 2024-04-26T23:09:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.398+0000 7f6c9a000640 1 -- 172.21.15.40:0/1181070580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c940c5df0 msgr2=0x7f6c940cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.398+0000 7f6c9a000640 1 --2- 172.21.15.40:0/1181070580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c940c5df0 0x7f6c940cc590 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f6c9406a5a0 tx=0x7f6c8c02a6a0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.398+0000 7f6c9a000640 1 -- 172.21.15.40:0/1181070580 shutdown_connections 2024-04-26T23:09:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.398+0000 7f6c9a000640 1 --2- 172.21.15.40:0/1181070580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c940c5df0 0x7f6c940cc590 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.398+0000 7f6c9a000640 1 --2- 172.21.15.40:0/1181070580 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c940ce5e0 0x7f6c940c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.398+0000 7f6c9a000640 1 --2- 172.21.15.40:0/1181070580 >> v2:172.21.15.103:3300/0 conn(0x7f6c9415f2d0 0x7f6c940afdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.398+0000 7f6c9a000640 1 -- 172.21.15.40:0/1181070580 >> 172.21.15.40:0/1181070580 conn(0x7f6c940d0a10 msgr2=0x7f6c940d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.398+0000 7f6c9a000640 1 -- 172.21.15.40:0/1181070580 shutdown_connections 2024-04-26T23:09:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.398+0000 7f6c9a000640 1 -- 172.21.15.40:0/1181070580 wait complete. 2024-04-26T23:09:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.400+0000 7f6c9a000640 1 Processor -- start 2024-04-26T23:09:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.400+0000 7f6c9a000640 1 -- start start 2024-04-26T23:09:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.401+0000 7f6c9a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c940c5df0 0x7f6c941519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:52.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.401+0000 7f6c9a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c940ce5e0 0x7f6c94147430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:52.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.401+0000 7f6c9a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c9415f2d0 0x7f6c94147970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:52.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.401+0000 7f6c9a000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6c940ccad0 con 0x7f6c940c5df0 2024-04-26T23:09:52.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.401+0000 7f6c9a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9406e270 con 0x7f6c9415f2d0 2024-04-26T23:09:52.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.401+0000 7f6c9a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6c940ccc50 con 0x7f6c940ce5e0 2024-04-26T23:09:52.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.401+0000 7f6c98c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c940c5df0 0x7f6c941519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:52.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.402+0000 7f6c98c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c940c5df0 0x7f6c941519d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40204/0 (socket says 172.21.15.40:40204) 2024-04-26T23:09:52.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.402+0000 7f6c98c00640 1 -- 172.21.15.40:0/4013987968 learned_addr learned my addr 172.21.15.40:0/4013987968 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:52.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.402+0000 7f6c99600640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c9415f2d0 0x7f6c94147970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:52.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.402+0000 7f6c93e00640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c940ce5e0 0x7f6c94147430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:52.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.402+0000 7f6c98c00640 1 -- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c940ce5e0 msgr2=0x7f6c94147430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.402+0000 7f6c98c00640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c940ce5e0 0x7f6c94147430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.402+0000 7f6c98c00640 1 -- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c9415f2d0 msgr2=0x7f6c94147970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.402+0000 7f6c98c00640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c9415f2d0 0x7f6c94147970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.402+0000 7f6c98c00640 1 -- 172.21.15.40:0/4013987968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c94072940 con 0x7f6c940c5df0 2024-04-26T23:09:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.402+0000 7f6c99600640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c9415f2d0 0x7f6c94147970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.402+0000 7f6c98c00640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c940c5df0 0x7f6c941519d0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800d7d0 tx=0x7f6c8800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.403+0000 7f6c90c00640 1 -- 172.21.15.40:0/4013987968 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c88004030 con 0x7f6c940c5df0 2024-04-26T23:09:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.403+0000 7f6c90c00640 1 -- 172.21.15.40:0/4013987968 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c880041d0 con 0x7f6c940c5df0 2024-04-26T23:09:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.403+0000 7f6c90c00640 1 -- 172.21.15.40:0/4013987968 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8800b6c0 con 0x7f6c940c5df0 2024-04-26T23:09:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.403+0000 7f6c9a000640 1 -- 172.21.15.40:0/4013987968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c94072c20 con 0x7f6c940c5df0 2024-04-26T23:09:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.403+0000 7f6c9a000640 1 -- 172.21.15.40:0/4013987968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c940734c0 con 0x7f6c940c5df0 2024-04-26T23:09:52.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.404+0000 7f6c9a000640 1 -- 172.21.15.40:0/4013987968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c94072df0 con 0x7f6c940c5df0 2024-04-26T23:09:52.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.405+0000 7f6c90c00640 1 -- 172.21.15.40:0/4013987968 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6c8800b860 con 0x7f6c940c5df0 2024-04-26T23:09:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.406+0000 7f6c90c00640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6c7406c8f0 0x7f6c7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.406+0000 7f6c90c00640 1 -- 172.21.15.40:0/4013987968 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f6c8808bff0 con 0x7f6c940c5df0 2024-04-26T23:09:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.407+0000 7f6c93e00640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6c7406c8f0 0x7f6c7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.408+0000 7f6c93e00640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6c7406c8f0 0x7f6c7406edb0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6c80006fd0 tx=0x7f6c80008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:52.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.410+0000 7f6c90c00640 1 -- 172.21.15.40:0/4013987968 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f6c8805a2a0 con 0x7f6c940c5df0 2024-04-26T23:09:52.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.657+0000 7f6c9a000640 1 -- 172.21.15.40:0/4013987968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c94069370 con 0x7f6c940c5df0 2024-04-26T23:09:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.658+0000 7f6c90c00640 1 -- 172.21.15.40:0/4013987968 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5239 (secure 0 0 0) 0x7f6c88059ba0 con 0x7f6c940c5df0 2024-04-26T23:09:52.659 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:52.659 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:44.797258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[6598,8949,8955]},"id":7}]} 2024-04-26T23:09:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 122 2024-04-26T23:09:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.661+0000 7f6c9a000640 1 -- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6c7406c8f0 msgr2=0x7f6c7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.661+0000 7f6c9a000640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6c7406c8f0 0x7f6c7406edb0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6c80006fd0 tx=0x7f6c80008040 comp rx=0 tx=0).stop 2024-04-26T23:09:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.661+0000 7f6c9a000640 1 -- 172.21.15.40:0/4013987968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c940c5df0 msgr2=0x7f6c941519d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.661+0000 7f6c9a000640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c940c5df0 0x7f6c941519d0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800d7d0 tx=0x7f6c8800dc90 comp rx=0 tx=0).stop 2024-04-26T23:09:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.661+0000 7f6c9a000640 1 -- 172.21.15.40:0/4013987968 shutdown_connections 2024-04-26T23:09:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.661+0000 7f6c9a000640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6c7406c8f0 0x7f6c7406edb0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.661+0000 7f6c9a000640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c9415f2d0 0x7f6c94147970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.661+0000 7f6c9a000640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c940ce5e0 0x7f6c94147430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.661+0000 7f6c9a000640 1 --2- 172.21.15.40:0/4013987968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c940c5df0 0x7f6c941519d0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.661+0000 7f6c9a000640 1 -- 172.21.15.40:0/4013987968 >> 172.21.15.40:0/4013987968 conn(0x7f6c940d0a10 msgr2=0x7f6c940cee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:52.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.662+0000 7f6c9a000640 1 -- 172.21.15.40:0/4013987968 shutdown_connections 2024-04-26T23:09:52.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.662+0000 7f6c9a000640 1 -- 172.21.15.40:0/4013987968 wait complete. 2024-04-26T23:09:52.681 DEBUG:tasks.cephfs.test_multimds_misc:current active=['a', 'c', 'd'] 2024-04-26T23:09:52.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:09:52.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.884+0000 7f8c38800640 1 Processor -- start 2024-04-26T23:09:52.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.884+0000 7f8c38800640 1 -- start start 2024-04-26T23:09:52.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.885+0000 7f8c38800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8c300cc480 0x7f8c30072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:52.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.885+0000 7f8c38800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8c30072780 0x7f8c30166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:52.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.885+0000 7f8c38800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c3015d7c0 con 0x7f8c30071e60 2024-04-26T23:09:52.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.885+0000 7f8c38800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f8c3015daa0 con 0x7f8c300cc480 2024-04-26T23:09:52.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.885+0000 7f8c38800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8c3015dd80 con 0x7f8c30072780 2024-04-26T23:09:52.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.886+0000 7f8c35400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8c300cc480 0x7f8c30072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:52.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.886+0000 7f8c35400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8c300cc480 0x7f8c30072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40210/0 (socket says 172.21.15.40:40210) 2024-04-26T23:09:52.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.886+0000 7f8c35400640 1 -- 172.21.15.40:0/1774924975 learned_addr learned my addr 172.21.15.40:0/1774924975 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:52.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.886+0000 7f8c36800640 1 --2- 172.21.15.40:0/1774924975 >> v2:172.21.15.103:3300/0 conn(0x7f8c30072780 0x7f8c30166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:52.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.886+0000 7f8c35400640 1 -- 172.21.15.40:0/1774924975 >> v2:172.21.15.103:3300/0 conn(0x7f8c30072780 msgr2=0x7f8c30166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:52.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.886+0000 7f8c35400640 1 --2- 172.21.15.40:0/1774924975 >> v2:172.21.15.103:3300/0 conn(0x7f8c30072780 0x7f8c30166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.886+0000 7f8c35400640 1 -- 172.21.15.40:0/1774924975 >> v1:172.21.15.103:6789/0 conn(0x7f8c30071e60 legacy=0x7f8c3006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:52.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.886+0000 7f8c35400640 1 -- 172.21.15.40:0/1774924975 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c3015e060 con 0x7f8c300cc480 2024-04-26T23:09:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.888+0000 7f8c35400640 1 --2- 172.21.15.40:0/1774924975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c300cc480 0x7f8c30072240 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f8c300b06b0 tx=0x7f8c2002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1819c05aa4e085ae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.888+0000 7f8c34a00640 1 -- 172.21.15.40:0/1774924975 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20037070 con 0x7f8c300cc480 2024-04-26T23:09:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.888+0000 7f8c34a00640 1 -- 172.21.15.40:0/1774924975 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c20032440 con 0x7f8c300cc480 2024-04-26T23:09:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.888+0000 7f8c34a00640 1 -- 172.21.15.40:0/1774924975 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20032810 con 0x7f8c300cc480 2024-04-26T23:09:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.888+0000 7f8c38800640 1 -- 172.21.15.40:0/1774924975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c300cc480 msgr2=0x7f8c30072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.888+0000 7f8c38800640 1 --2- 172.21.15.40:0/1774924975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c300cc480 0x7f8c30072240 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f8c300b06b0 tx=0x7f8c2002a230 comp rx=0 tx=0).stop 2024-04-26T23:09:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.889+0000 7f8c38800640 1 -- 172.21.15.40:0/1774924975 shutdown_connections 2024-04-26T23:09:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.889+0000 7f8c38800640 1 --2- 172.21.15.40:0/1774924975 >> v2:172.21.15.103:3300/0 conn(0x7f8c30072780 0x7f8c30166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.889+0000 7f8c38800640 1 --2- 172.21.15.40:0/1774924975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c300cc480 0x7f8c30072240 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.889+0000 7f8c38800640 1 -- 172.21.15.40:0/1774924975 >> 172.21.15.40:0/1774924975 conn(0x7f8c300d0be0 msgr2=0x7f8c300d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:52.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.889+0000 7f8c38800640 1 -- 172.21.15.40:0/1774924975 shutdown_connections 2024-04-26T23:09:52.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.889+0000 7f8c38800640 1 -- 172.21.15.40:0/1774924975 wait complete. 2024-04-26T23:09:52.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.890+0000 7f8c38800640 1 Processor -- start 2024-04-26T23:09:52.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.891+0000 7f8c38800640 1 -- start start 2024-04-26T23:09:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c38800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c30071e60 0x7f8c3014f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c38800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c30072780 0x7f8c300b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c38800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c300cc480 0x7f8c30157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c38800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3015d3f0 con 0x7f8c30072780 2024-04-26T23:09:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c38800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8c30072bd0 con 0x7f8c300cc480 2024-04-26T23:09:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c38800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8c3015d570 con 0x7f8c30071e60 2024-04-26T23:09:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c35400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c30072780 0x7f8c300b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:52.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c35400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c30072780 0x7f8c300b3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40216/0 (socket says 172.21.15.40:40216) 2024-04-26T23:09:52.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c35400640 1 -- 172.21.15.40:0/2877779125 learned_addr learned my addr 172.21.15.40:0/2877779125 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:52.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c35400640 1 -- 172.21.15.40:0/2877779125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c30071e60 msgr2=0x7f8c3014f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:52.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c35e00640 1 --2- 172.21.15.40:0/2877779125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c30071e60 0x7f8c3014f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:52.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c35400640 1 --2- 172.21.15.40:0/2877779125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c30071e60 0x7f8c3014f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c35400640 1 -- 172.21.15.40:0/2877779125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c300cc480 msgr2=0x7f8c30157a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:52.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c35400640 1 --2- 172.21.15.40:0/2877779125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c300cc480 0x7f8c30157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:52.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.892+0000 7f8c35400640 1 -- 172.21.15.40:0/2877779125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c300ce850 con 0x7f8c30072780 2024-04-26T23:09:52.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.893+0000 7f8c35400640 1 --2- 172.21.15.40:0/2877779125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c30072780 0x7f8c300b3880 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f8c20009960 tx=0x7f8c200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:52.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.893+0000 7f8c26000640 1 -- 172.21.15.40:0/2877779125 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20037050 con 0x7f8c30072780 2024-04-26T23:09:52.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.893+0000 7f8c26000640 1 -- 172.21.15.40:0/2877779125 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c20043070 con 0x7f8c30072780 2024-04-26T23:09:52.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.893+0000 7f8c38800640 1 -- 172.21.15.40:0/2877779125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c300ceb30 con 0x7f8c30072780 2024-04-26T23:09:52.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.893+0000 7f8c26000640 1 -- 172.21.15.40:0/2877779125 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20030bd0 con 0x7f8c30072780 2024-04-26T23:09:52.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.893+0000 7f8c38800640 1 -- 172.21.15.40:0/2877779125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c300cf0f0 con 0x7f8c30072780 2024-04-26T23:09:52.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.894+0000 7f8c38800640 1 -- 172.21.15.40:0/2877779125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c300694f0 con 0x7f8c30072780 2024-04-26T23:09:52.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.896+0000 7f8c26000640 1 -- 172.21.15.40:0/2877779125 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8c20048460 con 0x7f8c30072780 2024-04-26T23:09:52.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.896+0000 7f8c26000640 1 --2- 172.21.15.40:0/2877779125 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8c1406c8f0 0x7f8c1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:52.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.897+0000 7f8c26000640 1 -- 172.21.15.40:0/2877779125 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f8c200acbe0 con 0x7f8c30072780 2024-04-26T23:09:52.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.897+0000 7f8c35e00640 1 --2- 172.21.15.40:0/2877779125 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8c1406c8f0 0x7f8c1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:52.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.898+0000 7f8c35e00640 1 --2- 172.21.15.40:0/2877779125 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8c1406c8f0 0x7f8c1406edb0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f8c28007fd0 tx=0x7f8c28009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:52.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:52.901+0000 7f8c26000640 1 -- 172.21.15.40:0/2877779125 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8c2007ad30 con 0x7f8c30072780 2024-04-26T23:09:53.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.184+0000 7f8c38800640 1 -- 172.21.15.40:0/2877779125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f8c3006ce60 con 0x7f8c30072780 2024-04-26T23:09:53.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.185+0000 7f8c26000640 1 -- 172.21.15.40:0/2877779125 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1885 (secure 0 0 0) 0x7f8c2007a630 con 0x7f8c30072780 2024-04-26T23:09:53.185 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:53.185 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":553,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":105,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":706274,"bytes_used":267182080,"bytes_avail":772826931200,"bytes_total":773094113280},"fsmap":{"epoch":122,"id":7,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":7,"rank":2,"name":"d","status":"up:active","gid":6598},{"filesystem_id":7,"rank":1,"name":"c","status":"up:active","gid":8949},{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8955}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":29,"modified":"2024-04-26T23:09:43.338130+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:09:53.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.188+0000 7f8c38800640 1 -- 172.21.15.40:0/2877779125 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8c1406c8f0 msgr2=0x7f8c1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:53.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.188+0000 7f8c38800640 1 --2- 172.21.15.40:0/2877779125 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8c1406c8f0 0x7f8c1406edb0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f8c28007fd0 tx=0x7f8c28009040 comp rx=0 tx=0).stop 2024-04-26T23:09:53.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.188+0000 7f8c38800640 1 -- 172.21.15.40:0/2877779125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c30072780 msgr2=0x7f8c300b3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.188+0000 7f8c38800640 1 --2- 172.21.15.40:0/2877779125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c30072780 0x7f8c300b3880 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f8c20009960 tx=0x7f8c200046c0 comp rx=0 tx=0).stop 2024-04-26T23:09:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.188+0000 7f8c38800640 1 -- 172.21.15.40:0/2877779125 shutdown_connections 2024-04-26T23:09:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.188+0000 7f8c38800640 1 --2- 172.21.15.40:0/2877779125 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f8c1406c8f0 0x7f8c1406edb0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.188+0000 7f8c38800640 1 --2- 172.21.15.40:0/2877779125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c300cc480 0x7f8c30157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.188+0000 7f8c38800640 1 --2- 172.21.15.40:0/2877779125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c30072780 0x7f8c300b3880 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.188+0000 7f8c38800640 1 --2- 172.21.15.40:0/2877779125 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8c30071e60 0x7f8c3014f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.188+0000 7f8c38800640 1 -- 172.21.15.40:0/2877779125 >> 172.21.15.40:0/2877779125 conn(0x7f8c300d0be0 msgr2=0x7f8c30159e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.188+0000 7f8c38800640 1 -- 172.21.15.40:0/2877779125 shutdown_connections 2024-04-26T23:09:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:53.188+0000 7f8c38800640 1 -- 172.21.15.40:0/2877779125 wait complete. 2024-04-26T23:09:53.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'paused paths [/d1/d2/d3]', 'c': 'paused paths [/d1/d2/d3]', 'd': 'paused paths [/d1/d2/d3]'} 2024-04-26T23:09:54.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:54.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.429+0000 7f6086400640 1 Processor -- start 2024-04-26T23:09:54.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.429+0000 7f6086400640 1 -- start start 2024-04-26T23:09:54.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.430+0000 7f6086400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60800cc480 0x7f6080072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.430+0000 7f6086400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6080072780 0x7f6080166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.430+0000 7f6086400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f608015d7c0 con 0x7f6080071e60 2024-04-26T23:09:54.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.430+0000 7f6086400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f608015daa0 con 0x7f6080072780 2024-04-26T23:09:54.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.430+0000 7f6086400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f608015dd80 con 0x7f60800cc480 2024-04-26T23:09:54.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.431+0000 7f607fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6080072780 0x7f6080166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.431+0000 7f607fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6080072780 0x7f6080166ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40232/0 (socket says 172.21.15.40:40232) 2024-04-26T23:09:54.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.431+0000 7f607fe00640 1 -- 172.21.15.40:0/3227145427 learned_addr learned my addr 172.21.15.40:0/3227145427 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:54.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.431+0000 7f607fe00640 1 -- 172.21.15.40:0/3227145427 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60800cc480 msgr2=0x7f6080072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:54.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.431+0000 7f6076a00640 1 --2- 172.21.15.40:0/3227145427 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60800cc480 0x7f6080072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.431+0000 7f607fe00640 1 --2- 172.21.15.40:0/3227145427 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60800cc480 0x7f6080072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.431+0000 7f607fe00640 1 -- 172.21.15.40:0/3227145427 >> v1:172.21.15.103:6789/0 conn(0x7f6080071e60 legacy=0x7f608006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:54.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.431+0000 7f607fe00640 1 -- 172.21.15.40:0/3227145427 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608015e060 con 0x7f6080072780 2024-04-26T23:09:54.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.433+0000 7f607fe00640 1 --2- 172.21.15.40:0/3227145427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6080072780 0x7f6080166ad0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f6078009870 tx=0x7f607802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821e7218313193f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:54.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.433+0000 7f607ea00640 1 -- 172.21.15.40:0/3227145427 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078037070 con 0x7f6080072780 2024-04-26T23:09:54.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.433+0000 7f607ea00640 1 -- 172.21.15.40:0/3227145427 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6078032440 con 0x7f6080072780 2024-04-26T23:09:54.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.433+0000 7f607ea00640 1 -- 172.21.15.40:0/3227145427 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078032810 con 0x7f6080072780 2024-04-26T23:09:54.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.434+0000 7f6086400640 1 -- 172.21.15.40:0/3227145427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6080072780 msgr2=0x7f6080166ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:54.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.434+0000 7f6086400640 1 --2- 172.21.15.40:0/3227145427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6080072780 0x7f6080166ad0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f6078009870 tx=0x7f607802a270 comp rx=0 tx=0).stop 2024-04-26T23:09:54.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.434+0000 7f6086400640 1 -- 172.21.15.40:0/3227145427 shutdown_connections 2024-04-26T23:09:54.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.434+0000 7f6086400640 1 --2- 172.21.15.40:0/3227145427 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60800cc480 0x7f6080072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.434+0000 7f6086400640 1 --2- 172.21.15.40:0/3227145427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6080072780 0x7f6080166ad0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.434+0000 7f6086400640 1 -- 172.21.15.40:0/3227145427 >> 172.21.15.40:0/3227145427 conn(0x7f60800d0be0 msgr2=0x7f60800d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:54.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.434+0000 7f6086400640 1 -- 172.21.15.40:0/3227145427 shutdown_connections 2024-04-26T23:09:54.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.434+0000 7f6086400640 1 -- 172.21.15.40:0/3227145427 wait complete. 2024-04-26T23:09:54.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.435+0000 7f6086400640 1 Processor -- start 2024-04-26T23:09:54.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.436+0000 7f6086400640 1 -- start start 2024-04-26T23:09:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f6086400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071e60 0x7f60800c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f6086400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6080072780 0x7f60800b6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f6086400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800cc480 0x7f60800b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f6086400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f608015d3f0 con 0x7f60800cc480 2024-04-26T23:09:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f6086400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6080072c10 con 0x7f6080072780 2024-04-26T23:09:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f6086400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f608015d570 con 0x7f6080071e60 2024-04-26T23:09:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f607fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800cc480 0x7f60800b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f607fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800cc480 0x7f60800b4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40248/0 (socket says 172.21.15.40:40248) 2024-04-26T23:09:54.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f607fe00640 1 -- 172.21.15.40:0/3971396973 learned_addr learned my addr 172.21.15.40:0/3971396973 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:54.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f6076a00640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6080072780 0x7f60800b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f607f400640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071e60 0x7f60800c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f607fe00640 1 -- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071e60 msgr2=0x7f60800c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:54.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f607fe00640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071e60 0x7f60800c7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f607fe00640 1 -- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6080072780 msgr2=0x7f60800b6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:54.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f607fe00640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6080072780 0x7f60800b6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.437+0000 7f607fe00640 1 -- 172.21.15.40:0/3971396973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60800b39b0 con 0x7f60800cc480 2024-04-26T23:09:54.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.438+0000 7f607f400640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071e60 0x7f60800c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:54.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.438+0000 7f607fe00640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800cc480 0x7f60800b4a50 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f607802a2b0 tx=0x7f60780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.438+0000 7f6077e00640 1 -- 172.21.15.40:0/3971396973 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078037040 con 0x7f60800cc480 2024-04-26T23:09:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.438+0000 7f6077e00640 1 -- 172.21.15.40:0/3971396973 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6078042070 con 0x7f60800cc480 2024-04-26T23:09:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.438+0000 7f6086400640 1 -- 172.21.15.40:0/3971396973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60800b3c90 con 0x7f60800cc480 2024-04-26T23:09:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.438+0000 7f6077e00640 1 -- 172.21.15.40:0/3971396973 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078030bd0 con 0x7f60800cc480 2024-04-26T23:09:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.438+0000 7f6086400640 1 -- 172.21.15.40:0/3971396973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60800b4250 con 0x7f60800cc480 2024-04-26T23:09:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.439+0000 7f6086400640 1 -- 172.21.15.40:0/3971396973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60800694f0 con 0x7f60800cc480 2024-04-26T23:09:54.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.440+0000 7f6077e00640 1 -- 172.21.15.40:0/3971396973 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6078047460 con 0x7f60800cc480 2024-04-26T23:09:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.441+0000 7f6077e00640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f605c06c8f0 0x7f605c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.441+0000 7f6077e00640 1 -- 172.21.15.40:0/3971396973 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f60780ac960 con 0x7f60800cc480 2024-04-26T23:09:54.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.442+0000 7f607f400640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f605c06c8f0 0x7f605c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.443+0000 7f607f400640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f605c06c8f0 0x7f605c06edb0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f60800caa70 tx=0x7f6070009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:54.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.446+0000 7f6077e00640 1 -- 172.21.15.40:0/3971396973 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f607807abd0 con 0x7f60800cc480 2024-04-26T23:09:54.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.699+0000 7f6086400640 1 -- 172.21.15.40:0/3971396973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f608006ce60 con 0x7f60800cc480 2024-04-26T23:09:54.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.701+0000 7f6077e00640 1 -- 172.21.15.40:0/3971396973 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5239 (secure 0 0 0) 0x7f607807a4d0 con 0x7f60800cc480 2024-04-26T23:09:54.701 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:54.701 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:44.797258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[6598,8949,8955]},"id":7}]} 2024-04-26T23:09:54.701 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 122 2024-04-26T23:09:54.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.703+0000 7f6086400640 1 -- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f605c06c8f0 msgr2=0x7f605c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:54.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.703+0000 7f6086400640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f605c06c8f0 0x7f605c06edb0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f60800caa70 tx=0x7f6070009040 comp rx=0 tx=0).stop 2024-04-26T23:09:54.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.704+0000 7f6086400640 1 -- 172.21.15.40:0/3971396973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800cc480 msgr2=0x7f60800b4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:54.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.704+0000 7f6086400640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800cc480 0x7f60800b4a50 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f607802a2b0 tx=0x7f60780046c0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.704+0000 7f6086400640 1 -- 172.21.15.40:0/3971396973 shutdown_connections 2024-04-26T23:09:54.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.704+0000 7f6086400640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f605c06c8f0 0x7f605c06edb0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.704+0000 7f6086400640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60800cc480 0x7f60800b4a50 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.704+0000 7f6086400640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6080072780 0x7f60800b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.704+0000 7f6086400640 1 --2- 172.21.15.40:0/3971396973 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6080071e60 0x7f60800c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.704+0000 7f6086400640 1 -- 172.21.15.40:0/3971396973 >> 172.21.15.40:0/3971396973 conn(0x7f60800d0be0 msgr2=0x7f608015f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:54.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.704+0000 7f6086400640 1 -- 172.21.15.40:0/3971396973 shutdown_connections 2024-04-26T23:09:54.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.704+0000 7f6086400640 1 -- 172.21.15.40:0/3971396973 wait complete. 2024-04-26T23:09:54.725 DEBUG:tasks.cephfs.test_multimds_misc:current active=['a', 'c', 'd'] 2024-04-26T23:09:54.725 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:09:54.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.931+0000 7f2cb7000640 1 Processor -- start 2024-04-26T23:09:54.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.931+0000 7f2cb7000640 1 -- start start 2024-04-26T23:09:54.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.932+0000 7f2cb7000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2cb015f210 0x7f2cb0162e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.932+0000 7f2cb7000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cb00cfcb0 con 0x7f2cb00d0d10 2024-04-26T23:09:54.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.932+0000 7f2cb7000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cb00cff50 con 0x7f2cb00d2b60 2024-04-26T23:09:54.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.932+0000 7f2cb7000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2cb00d0230 con 0x7f2cb015f210 2024-04-26T23:09:54.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.933+0000 7f2cb5200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2cb00d0d10 0x7f2cb01633b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48438/0 (socket says 172.21.15.40:48438) 2024-04-26T23:09:54.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.933+0000 7f2cb5200640 1 -- 172.21.15.40:0/2754116276 learned_addr learned my addr 172.21.15.40:0/2754116276 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:54.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.933+0000 7f2cb5c00640 1 --2- 172.21.15.40:0/2754116276 >> v2:172.21.15.103:3300/0 conn(0x7f2cb015f210 0x7f2cb0162e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.933+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2247642478 0 0) 0x7f2cb00cfcb0 con 0x7f2cb00d0d10 2024-04-26T23:09:54.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.933+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c98003520 con 0x7f2cb00d0d10 2024-04-26T23:09:54.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 177709145 0 0) 0x7f2c98003520 con 0x7f2cb00d0d10 2024-04-26T23:09:54.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 >> v2:172.21.15.103:3300/0 conn(0x7f2cb015f210 msgr2=0x7f2cb0162e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:54.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cafe00640 1 --2- 172.21.15.40:0/2754116276 >> v2:172.21.15.103:3300/0 conn(0x7f2cb015f210 0x7f2cb0162e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 >> v1:172.21.15.103:6789/0 conn(0x7f2cb00d2b60 legacy=0x7f2cb00d3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:54.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb00d0510 con 0x7f2cb00d0d10 2024-04-26T23:09:54.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f2c9c001670 con 0x7f2cb00d0d10 2024-04-26T23:09:54.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 >> v1:172.21.15.40:6789/0 conn(0x7f2cb00d0d10 legacy=0x7f2cb01633b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:54.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cafe00640 1 --2- 172.21.15.40:0/2754116276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c980039b0 0x7f2c98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cafe00640 1 --2- 172.21.15.40:0/2754116276 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c98004800 0x7f2c98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cafe00640 1 --2- 172.21.15.40:0/2754116276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c980051c0 0x7f2c9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb00d0230 con 0x7f2c980051c0 2024-04-26T23:09:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9800c060 con 0x7f2c980039b0 2024-04-26T23:09:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2c9800c260 con 0x7f2c98004800 2024-04-26T23:09:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.934+0000 7f2cb6600640 1 --2- 172.21.15.40:0/2754116276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c980051c0 0x7f2c9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.935+0000 7f2cb5c00640 1 --2- 172.21.15.40:0/2754116276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c980039b0 0x7f2c98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.935+0000 7f2cb6600640 1 -- 172.21.15.40:0/2754116276 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c98004800 msgr2=0x7f2c98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.935+0000 7f2cb5200640 1 --2- 172.21.15.40:0/2754116276 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c98004800 0x7f2c98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.935+0000 7f2cb6600640 1 --2- 172.21.15.40:0/2754116276 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c98004800 0x7f2c98004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.935+0000 7f2cb6600640 1 -- 172.21.15.40:0/2754116276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c980039b0 msgr2=0x7f2c98003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.935+0000 7f2cb6600640 1 --2- 172.21.15.40:0/2754116276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c980039b0 0x7f2c98003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.935+0000 7f2cb6600640 1 -- 172.21.15.40:0/2754116276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c98003520 con 0x7f2c980051c0 2024-04-26T23:09:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.936+0000 7f2cb6600640 1 -- 172.21.15.40:0/2754116276 reap_dead start 2024-04-26T23:09:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.936+0000 7f2cb6600640 1 --2- 172.21.15.40:0/2754116276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c980051c0 0x7f2c9800b9c0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8001750 tx=0x7f2c98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dba6abafc82f863 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.936+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8001370 con 0x7f2c980051c0 2024-04-26T23:09:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.936+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ca8005310 con 0x7f2c980051c0 2024-04-26T23:09:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.936+0000 7f2cafe00640 1 -- 172.21.15.40:0/2754116276 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca802abe0 con 0x7f2c980051c0 2024-04-26T23:09:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.937+0000 7f2cb7000640 1 -- 172.21.15.40:0/2754116276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c980051c0 msgr2=0x7f2c9800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.937+0000 7f2cb7000640 1 --2- 172.21.15.40:0/2754116276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c980051c0 0x7f2c9800b9c0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8001750 tx=0x7f2c98005830 comp rx=0 tx=0).stop 2024-04-26T23:09:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.937+0000 7f2cb7000640 1 -- 172.21.15.40:0/2754116276 shutdown_connections 2024-04-26T23:09:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.937+0000 7f2cb7000640 1 --2- 172.21.15.40:0/2754116276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c980051c0 0x7f2c9800b9c0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.937+0000 7f2cb7000640 1 -- 172.21.15.40:0/2754116276 >> 172.21.15.40:0/2754116276 conn(0x7f2cb00d4f90 msgr2=0x7f2cb00d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.937+0000 7f2cb7000640 1 -- 172.21.15.40:0/2754116276 shutdown_connections 2024-04-26T23:09:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.937+0000 7f2cb7000640 1 -- 172.21.15.40:0/2754116276 wait complete. 2024-04-26T23:09:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.939+0000 7f2cb7000640 1 Processor -- start 2024-04-26T23:09:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.939+0000 7f2cb7000640 1 -- start start 2024-04-26T23:09:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.940+0000 7f2cb7000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c980051c0 0x7f2cb00ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.940+0000 7f2cb7000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cb00ba050 0x7f2cb01f8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.940+0000 7f2cb7000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2cb01f8aa0 0x7f2cb01608e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.940+0000 7f2cb7000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb015f770 con 0x7f2c980051c0 2024-04-26T23:09:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.940+0000 7f2cb7000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb015f5f0 con 0x7f2cb00ba050 2024-04-26T23:09:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.940+0000 7f2cb7000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2cb00d27c0 con 0x7f2cb01f8aa0 2024-04-26T23:09:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.940+0000 7f2cb5c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c980051c0 0x7f2cb00ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.940+0000 7f2cb5c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c980051c0 0x7f2cb00ce840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45870/0 (socket says 172.21.15.40:45870) 2024-04-26T23:09:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.940+0000 7f2cb5c00640 1 -- 172.21.15.40:0/1193857298 learned_addr learned my addr 172.21.15.40:0/1193857298 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.941+0000 7f2cb5200640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cb00ba050 0x7f2cb01f8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.941+0000 7f2cb6600640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2cb01f8aa0 0x7f2cb01608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.941+0000 7f2cb5c00640 1 -- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2cb01f8aa0 msgr2=0x7f2cb01608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.941+0000 7f2cb5c00640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2cb01f8aa0 0x7f2cb01608e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.941+0000 7f2cb5c00640 1 -- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cb00ba050 msgr2=0x7f2cb01f8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:54.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.941+0000 7f2cb5c00640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cb00ba050 0x7f2cb01f8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:54.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.941+0000 7f2cb5c00640 1 -- 172.21.15.40:0/1193857298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb00ccc10 con 0x7f2c980051c0 2024-04-26T23:09:54.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.941+0000 7f2cb5c00640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c980051c0 0x7f2cb00ce840 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4009fb0 tx=0x7f2ca40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:54.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.941+0000 7f2cad600640 1 -- 172.21.15.40:0/1193857298 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca401a020 con 0x7f2c980051c0 2024-04-26T23:09:54.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.941+0000 7f2cad600640 1 -- 172.21.15.40:0/1193857298 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ca4016470 con 0x7f2c980051c0 2024-04-26T23:09:54.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.941+0000 7f2cb7000640 1 -- 172.21.15.40:0/1193857298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cb00ccef0 con 0x7f2c980051c0 2024-04-26T23:09:54.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.942+0000 7f2cad600640 1 -- 172.21.15.40:0/1193857298 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4016860 con 0x7f2c980051c0 2024-04-26T23:09:54.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.942+0000 7f2cb7000640 1 -- 172.21.15.40:0/1193857298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cb00cd4b0 con 0x7f2c980051c0 2024-04-26T23:09:54.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.943+0000 7f2cb7000640 1 -- 172.21.15.40:0/1193857298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cb01f8eb0 con 0x7f2c980051c0 2024-04-26T23:09:54.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.944+0000 7f2cad600640 1 -- 172.21.15.40:0/1193857298 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2ca4005d10 con 0x7f2c980051c0 2024-04-26T23:09:54.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.945+0000 7f2cad600640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2c9006c8f0 0x7f2c9006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:54.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.945+0000 7f2cad600640 1 -- 172.21.15.40:0/1193857298 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f2ca408cb50 con 0x7f2c980051c0 2024-04-26T23:09:54.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.946+0000 7f2cb5200640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2c9006c8f0 0x7f2c9006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:54.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.947+0000 7f2cb5200640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2c9006c8f0 0x7f2c9006edb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c000f60 tx=0x7f2c9c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:54.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:54.950+0000 7f2cad600640 1 -- 172.21.15.40:0/1193857298 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2cb01f8eb0 con 0x7f2c980051c0 2024-04-26T23:09:55.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.228+0000 7f2cb7000640 1 -- 172.21.15.40:0/1193857298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f2cb0069a80 con 0x7f2c980051c0 2024-04-26T23:09:55.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.230+0000 7f2cad600640 1 -- 172.21.15.40:0/1193857298 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1966 (secure 0 0 0) 0x7f2ca4024030 con 0x7f2c980051c0 2024-04-26T23:09:55.230 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:55.230 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":555,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":105,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":712040,"bytes_used":267436032,"bytes_avail":772826677248,"bytes_total":773094113280,"write_bytes_sec":895,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":122,"id":7,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":7,"rank":2,"name":"d","status":"up:active","gid":6598},{"filesystem_id":7,"rank":1,"name":"c","status":"up:active","gid":8949},{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8955}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":30,"modified":"2024-04-26T23:09:53.341661+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:09:55.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.232+0000 7f2cb7000640 1 -- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2c9006c8f0 msgr2=0x7f2c9006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:55.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.232+0000 7f2cb7000640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2c9006c8f0 0x7f2c9006edb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c000f60 tx=0x7f2c9c009040 comp rx=0 tx=0).stop 2024-04-26T23:09:55.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.232+0000 7f2cb7000640 1 -- 172.21.15.40:0/1193857298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c980051c0 msgr2=0x7f2cb00ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:55.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.232+0000 7f2cb7000640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c980051c0 0x7f2cb00ce840 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4009fb0 tx=0x7f2ca40115a0 comp rx=0 tx=0).stop 2024-04-26T23:09:55.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.233+0000 7f2cb7000640 1 -- 172.21.15.40:0/1193857298 shutdown_connections 2024-04-26T23:09:55.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.233+0000 7f2cb7000640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2c9006c8f0 0x7f2c9006edb0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:55.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.233+0000 7f2cb7000640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2cb01f8aa0 0x7f2cb01608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:55.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.233+0000 7f2cb7000640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cb00ba050 0x7f2cb01f8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:55.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.233+0000 7f2cb7000640 1 --2- 172.21.15.40:0/1193857298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c980051c0 0x7f2cb00ce840 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:55.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.233+0000 7f2cb7000640 1 -- 172.21.15.40:0/1193857298 >> 172.21.15.40:0/1193857298 conn(0x7f2cb00d4f90 msgr2=0x7f2cb00d5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:55.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.233+0000 7f2cb7000640 1 -- 172.21.15.40:0/1193857298 shutdown_connections 2024-04-26T23:09:55.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:55.233+0000 7f2cb7000640 1 -- 172.21.15.40:0/1193857298 wait complete. 2024-04-26T23:09:55.252 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'idle+waiting paths [/d1/d2/d3]', 'c': 'idle+waiting paths [/d1/d2/d3]', 'd': 'idle+waiting paths [/d1/d2/d3]'} 2024-04-26T23:09:56.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:56.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.473+0000 7fb540a00640 1 Processor -- start 2024-04-26T23:09:56.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.474+0000 7fb540a00640 1 -- start start 2024-04-26T23:09:56.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.475+0000 7fb540a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb53c06c240 0x7fb53c06c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:56.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.475+0000 7fb540a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb53c0d27e0 con 0x7fb53c0b4ba0 2024-04-26T23:09:56.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.476+0000 7fb540a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb53c0d0400 con 0x7fb53c146790 2024-04-26T23:09:56.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.476+0000 7fb540a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb53c06cbe0 con 0x7fb53c06c240 2024-04-26T23:09:56.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.476+0000 7fb532a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb53c06c240 0x7fb53c06c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:56.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.476+0000 7fb532a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb53c06c240 0x7fb53c06c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45872/0 (socket says 172.21.15.40:45872) 2024-04-26T23:09:56.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.476+0000 7fb532a00640 1 -- 172.21.15.40:0/4193398343 learned_addr learned my addr 172.21.15.40:0/4193398343 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:56.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.476+0000 7fb532a00640 1 -- 172.21.15.40:0/4193398343 >> v1:172.21.15.103:6789/0 conn(0x7fb53c146790 legacy=0x7fb53c0d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:56.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.476+0000 7fb532a00640 1 -- 172.21.15.40:0/4193398343 >> v1:172.21.15.40:6789/0 conn(0x7fb53c0b4ba0 legacy=0x7fb53c0d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:56.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.476+0000 7fb532a00640 1 -- 172.21.15.40:0/4193398343 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53c0d06e0 con 0x7fb53c06c240 2024-04-26T23:09:56.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.477+0000 7fb532a00640 1 --2- 172.21.15.40:0/4193398343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c06c240 0x7fb53c06c6a0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fb524009870 tx=0x7fb52402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c526fd3bf0de4210 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:56.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.478+0000 7fb53aa00640 1 -- 172.21.15.40:0/4193398343 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb524037070 con 0x7fb53c06c240 2024-04-26T23:09:56.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.478+0000 7fb53aa00640 1 -- 172.21.15.40:0/4193398343 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb524032440 con 0x7fb53c06c240 2024-04-26T23:09:56.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.478+0000 7fb53aa00640 1 -- 172.21.15.40:0/4193398343 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb524032810 con 0x7fb53c06c240 2024-04-26T23:09:56.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.478+0000 7fb540a00640 1 -- 172.21.15.40:0/4193398343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c06c240 msgr2=0x7fb53c06c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:56.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.478+0000 7fb540a00640 1 --2- 172.21.15.40:0/4193398343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c06c240 0x7fb53c06c6a0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fb524009870 tx=0x7fb52402a210 comp rx=0 tx=0).stop 2024-04-26T23:09:56.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.479+0000 7fb540a00640 1 -- 172.21.15.40:0/4193398343 shutdown_connections 2024-04-26T23:09:56.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.479+0000 7fb540a00640 1 --2- 172.21.15.40:0/4193398343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c06c240 0x7fb53c06c6a0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:56.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.479+0000 7fb540a00640 1 -- 172.21.15.40:0/4193398343 >> 172.21.15.40:0/4193398343 conn(0x7fb53c068090 msgr2=0x7fb53c068470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:56.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.479+0000 7fb540a00640 1 -- 172.21.15.40:0/4193398343 shutdown_connections 2024-04-26T23:09:56.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.479+0000 7fb540a00640 1 -- 172.21.15.40:0/4193398343 wait complete. 2024-04-26T23:09:56.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.480+0000 7fb540a00640 1 Processor -- start 2024-04-26T23:09:56.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.481+0000 7fb540a00640 1 -- start start 2024-04-26T23:09:56.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.481+0000 7fb540a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb53c06c240 0x7fb53c15d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:56.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb540a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb53c0b4ba0 0x7fb53c0b1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:56.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb540a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c146790 0x7fb53c15c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:56.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb540a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c0d0a50 con 0x7fb53c146790 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb540a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c146b70 con 0x7fb53c0b4ba0 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb540a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb53c0d00b0 con 0x7fb53c06c240 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb53be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c146790 0x7fb53c15c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb53be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c146790 0x7fb53c15c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45886/0 (socket says 172.21.15.40:45886) 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb53be00640 1 -- 172.21.15.40:0/2168509413 learned_addr learned my addr 172.21.15.40:0/2168509413 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb53b400640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb53c06c240 0x7fb53c15d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb532a00640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb53c0b4ba0 0x7fb53c0b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb53be00640 1 -- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb53c06c240 msgr2=0x7fb53c15d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb53be00640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb53c06c240 0x7fb53c15d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb53be00640 1 -- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb53c0b4ba0 msgr2=0x7fb53c0b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.482+0000 7fb53be00640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb53c0b4ba0 0x7fb53c0b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.483+0000 7fb53be00640 1 -- 172.21.15.40:0/2168509413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53c15e810 con 0x7fb53c146790 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.483+0000 7fb53b400640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb53c06c240 0x7fb53c15d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.483+0000 7fb532a00640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb53c0b4ba0 0x7fb53c0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:56.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.483+0000 7fb53be00640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c146790 0x7fb53c15c190 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fb53400c390 tx=0x7fb53400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:56.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.483+0000 7fb533e00640 1 -- 172.21.15.40:0/2168509413 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb534017070 con 0x7fb53c146790 2024-04-26T23:09:56.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.483+0000 7fb533e00640 1 -- 172.21.15.40:0/2168509413 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5340052d0 con 0x7fb53c146790 2024-04-26T23:09:56.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.483+0000 7fb533e00640 1 -- 172.21.15.40:0/2168509413 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb534003810 con 0x7fb53c146790 2024-04-26T23:09:56.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.483+0000 7fb540a00640 1 -- 172.21.15.40:0/2168509413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb53c15dc90 con 0x7fb53c146790 2024-04-26T23:09:56.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.483+0000 7fb540a00640 1 -- 172.21.15.40:0/2168509413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb53c15e530 con 0x7fb53c146790 2024-04-26T23:09:56.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.485+0000 7fb540a00640 1 -- 172.21.15.40:0/2168509413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb53c069a80 con 0x7fb53c146790 2024-04-26T23:09:56.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.485+0000 7fb533e00640 1 -- 172.21.15.40:0/2168509413 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb53401a070 con 0x7fb53c146790 2024-04-26T23:09:56.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.486+0000 7fb533e00640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb51806c8f0 0x7fb51806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:56.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.486+0000 7fb533e00640 1 -- 172.21.15.40:0/2168509413 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7fb53408b0a0 con 0x7fb53c146790 2024-04-26T23:09:56.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.487+0000 7fb53b400640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb51806c8f0 0x7fb51806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:56.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.488+0000 7fb53b400640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb51806c8f0 0x7fb51806edb0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb52c007e90 tx=0x7fb52c0073a0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:56.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.491+0000 7fb533e00640 1 -- 172.21.15.40:0/2168509413 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fb534059280 con 0x7fb53c146790 2024-04-26T23:09:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.738+0000 7fb540a00640 1 -- 172.21.15.40:0/2168509413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb53c0b44d0 con 0x7fb53c146790 2024-04-26T23:09:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.739+0000 7fb533e00640 1 -- 172.21.15.40:0/2168509413 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5239 (secure 0 0 0) 0x7fb534058b80 con 0x7fb53c146790 2024-04-26T23:09:56.740 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:56.740 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:44.797258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[6598,8949,8955]},"id":7}]} 2024-04-26T23:09:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 122 2024-04-26T23:09:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.742+0000 7fb540a00640 1 -- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb51806c8f0 msgr2=0x7fb51806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.742+0000 7fb540a00640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb51806c8f0 0x7fb51806edb0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb52c007e90 tx=0x7fb52c0073a0 comp rx=0 tx=0).stop 2024-04-26T23:09:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.742+0000 7fb540a00640 1 -- 172.21.15.40:0/2168509413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c146790 msgr2=0x7fb53c15c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.742+0000 7fb540a00640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c146790 0x7fb53c15c190 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fb53400c390 tx=0x7fb53400c850 comp rx=0 tx=0).stop 2024-04-26T23:09:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.742+0000 7fb540a00640 1 -- 172.21.15.40:0/2168509413 shutdown_connections 2024-04-26T23:09:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.742+0000 7fb540a00640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb51806c8f0 0x7fb51806edb0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.742+0000 7fb540a00640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c146790 0x7fb53c15c190 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.742+0000 7fb540a00640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb53c0b4ba0 0x7fb53c0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.742+0000 7fb540a00640 1 --2- 172.21.15.40:0/2168509413 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb53c06c240 0x7fb53c15d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.743+0000 7fb540a00640 1 -- 172.21.15.40:0/2168509413 >> 172.21.15.40:0/2168509413 conn(0x7fb53c068090 msgr2=0x7fb53c0b2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.743+0000 7fb540a00640 1 -- 172.21.15.40:0/2168509413 shutdown_connections 2024-04-26T23:09:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.743+0000 7fb540a00640 1 -- 172.21.15.40:0/2168509413 wait complete. 2024-04-26T23:09:56.763 DEBUG:tasks.cephfs.test_multimds_misc:current active=['a', 'c', 'd'] 2024-04-26T23:09:56.764 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:09:56.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.977+0000 7faf8b400640 1 Processor -- start 2024-04-26T23:09:56.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.977+0000 7faf8b400640 1 -- start start 2024-04-26T23:09:56.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.978+0000 7faf8b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faf84073b70 0x7faf840aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:56.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.978+0000 7faf8b400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf84072c00 con 0x7faf840cbb50 2024-04-26T23:09:56.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.978+0000 7faf8b400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf84072ee0 con 0x7faf8415f380 2024-04-26T23:09:56.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.978+0000 7faf8b400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7faf840729b0 con 0x7faf84073b70 2024-04-26T23:09:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.979+0000 7faf89400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7faf840cbb50 0x7faf840af1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48460/0 (socket says 172.21.15.40:48460) 2024-04-26T23:09:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.979+0000 7faf89400640 1 -- 172.21.15.40:0/177933779 learned_addr learned my addr 172.21.15.40:0/177933779 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.979+0000 7faf83e00640 1 --2- 172.21.15.40:0/177933779 >> v2:172.21.15.40:3300/0 conn(0x7faf84073b70 0x7faf840aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.979+0000 7faf83400640 1 -- 172.21.15.40:0/177933779 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4158274304 0 0) 0x7faf84072c00 con 0x7faf840cbb50 2024-04-26T23:09:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.980+0000 7faf83400640 1 -- 172.21.15.40:0/177933779 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf64003520 con 0x7faf840cbb50 2024-04-26T23:09:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.980+0000 7faf83e00640 1 -- 172.21.15.40:0/177933779 >> v1:172.21.15.103:6789/0 conn(0x7faf8415f380 legacy=0x7faf840cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.980+0000 7faf83e00640 1 -- 172.21.15.40:0/177933779 >> v1:172.21.15.40:6789/0 conn(0x7faf840cbb50 legacy=0x7faf840af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.980+0000 7faf83e00640 1 -- 172.21.15.40:0/177933779 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf840731c0 con 0x7faf84073b70 2024-04-26T23:09:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.981+0000 7faf83e00640 1 --2- 172.21.15.40:0/177933779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf84073b70 0x7faf840aecb0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7faf6c009d90 tx=0x7faf6c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec74cd641f015db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.981+0000 7faf83400640 1 -- 172.21.15.40:0/177933779 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c007740 con 0x7faf84073b70 2024-04-26T23:09:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.981+0000 7faf83400640 1 -- 172.21.15.40:0/177933779 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf6c0078e0 con 0x7faf84073b70 2024-04-26T23:09:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.981+0000 7faf83400640 1 -- 172.21.15.40:0/177933779 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c007cb0 con 0x7faf84073b70 2024-04-26T23:09:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.982+0000 7faf8b400640 1 -- 172.21.15.40:0/177933779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf84073b70 msgr2=0x7faf840aecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.982+0000 7faf8b400640 1 --2- 172.21.15.40:0/177933779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf84073b70 0x7faf840aecb0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7faf6c009d90 tx=0x7faf6c02a6f0 comp rx=0 tx=0).stop 2024-04-26T23:09:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.982+0000 7faf8b400640 1 -- 172.21.15.40:0/177933779 shutdown_connections 2024-04-26T23:09:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.982+0000 7faf8b400640 1 --2- 172.21.15.40:0/177933779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf84073b70 0x7faf840aecb0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.982+0000 7faf8b400640 1 -- 172.21.15.40:0/177933779 >> 172.21.15.40:0/177933779 conn(0x7faf840d0be0 msgr2=0x7faf840d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.982+0000 7faf8b400640 1 -- 172.21.15.40:0/177933779 shutdown_connections 2024-04-26T23:09:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.982+0000 7faf8b400640 1 -- 172.21.15.40:0/177933779 wait complete. 2024-04-26T23:09:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.984+0000 7faf8b400640 1 Processor -- start 2024-04-26T23:09:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.984+0000 7faf8b400640 1 -- start start 2024-04-26T23:09:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.985+0000 7faf8b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf840cbb50 0x7faf8414e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.985+0000 7faf8b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf8415f380 0x7faf840c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.985+0000 7faf8b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf840b37b0 0x7faf840b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.985+0000 7faf8b400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faf84073620 con 0x7faf840b37b0 2024-04-26T23:09:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.985+0000 7faf8b400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faf840d08f0 con 0x7faf8415f380 2024-04-26T23:09:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.985+0000 7faf8b400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faf840737a0 con 0x7faf840cbb50 2024-04-26T23:09:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.985+0000 7faf89400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf840b37b0 0x7faf840b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.985+0000 7faf89400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf840b37b0 0x7faf840b3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45906/0 (socket says 172.21.15.40:45906) 2024-04-26T23:09:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.985+0000 7faf89400640 1 -- 172.21.15.40:0/3506085374 learned_addr learned my addr 172.21.15.40:0/3506085374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.986+0000 7faf83e00640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf8415f380 0x7faf840c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.986+0000 7faf88a00640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf840cbb50 0x7faf8414e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.986+0000 7faf89400640 1 -- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf840cbb50 msgr2=0x7faf8414e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.986+0000 7faf89400640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf840cbb50 0x7faf8414e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.986+0000 7faf89400640 1 -- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf8415f380 msgr2=0x7faf840c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.986+0000 7faf89400640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf8415f380 0x7faf840c8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.986+0000 7faf89400640 1 -- 172.21.15.40:0/3506085374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf840c9600 con 0x7faf840b37b0 2024-04-26T23:09:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.986+0000 7faf89400640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf840b37b0 0x7faf840b3c60 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7faf7c00c430 tx=0x7faf7c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.987+0000 7faf80c00640 1 -- 172.21.15.40:0/3506085374 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7c017070 con 0x7faf840b37b0 2024-04-26T23:09:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.987+0000 7faf80c00640 1 -- 172.21.15.40:0/3506085374 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf7c005ce0 con 0x7faf840b37b0 2024-04-26T23:09:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.987+0000 7faf80c00640 1 -- 172.21.15.40:0/3506085374 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7c007310 con 0x7faf840b37b0 2024-04-26T23:09:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.987+0000 7faf8b400640 1 -- 172.21.15.40:0/3506085374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf840c98e0 con 0x7faf840b37b0 2024-04-26T23:09:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.987+0000 7faf8b400640 1 -- 172.21.15.40:0/3506085374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf840c9ea0 con 0x7faf840b37b0 2024-04-26T23:09:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.988+0000 7faf8b400640 1 -- 172.21.15.40:0/3506085374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf840ca6a0 con 0x7faf840b37b0 2024-04-26T23:09:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.989+0000 7faf80c00640 1 -- 172.21.15.40:0/3506085374 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faf7c01a070 con 0x7faf840b37b0 2024-04-26T23:09:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.990+0000 7faf80c00640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faf6006c920 0x7faf6006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.990+0000 7faf80c00640 1 -- 172.21.15.40:0/3506085374 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7faf7c08b1e0 con 0x7faf840b37b0 2024-04-26T23:09:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.991+0000 7faf88a00640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faf6006c920 0x7faf6006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.992+0000 7faf88a00640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faf6006c920 0x7faf6006ede0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7faf74001640 tx=0x7faf74009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:56.995+0000 7faf80c00640 1 -- 172.21.15.40:0/3506085374 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7faf7c0593c0 con 0x7faf840b37b0 2024-04-26T23:09:57.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.284+0000 7faf8b400640 1 -- 172.21.15.40:0/3506085374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7faf840694f0 con 0x7faf840b37b0 2024-04-26T23:09:57.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.285+0000 7faf80c00640 1 -- 172.21.15.40:0/3506085374 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1966 (secure 0 0 0) 0x7faf7c058cc0 con 0x7faf840b37b0 2024-04-26T23:09:57.285 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:57.285 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":557,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":105,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":712040,"bytes_used":267436032,"bytes_avail":772826677248,"bytes_total":773094113280,"write_bytes_sec":895,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":122,"id":7,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":7,"rank":2,"name":"d","status":"up:active","gid":6598},{"filesystem_id":7,"rank":1,"name":"c","status":"up:active","gid":8949},{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8955}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":30,"modified":"2024-04-26T23:09:53.341661+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:09:57.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.288+0000 7faf8b400640 1 -- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faf6006c920 msgr2=0x7faf6006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:57.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.288+0000 7faf8b400640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faf6006c920 0x7faf6006ede0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7faf74001640 tx=0x7faf74009040 comp rx=0 tx=0).stop 2024-04-26T23:09:57.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.288+0000 7faf8b400640 1 -- 172.21.15.40:0/3506085374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf840b37b0 msgr2=0x7faf840b3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:57.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.288+0000 7faf8b400640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf840b37b0 0x7faf840b3c60 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7faf7c00c430 tx=0x7faf7c00c8f0 comp rx=0 tx=0).stop 2024-04-26T23:09:57.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.288+0000 7faf8b400640 1 -- 172.21.15.40:0/3506085374 shutdown_connections 2024-04-26T23:09:57.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.288+0000 7faf8b400640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7faf6006c920 0x7faf6006ede0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:57.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.288+0000 7faf8b400640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf840b37b0 0x7faf840b3c60 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:57.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.288+0000 7faf8b400640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf8415f380 0x7faf840c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:57.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.288+0000 7faf8b400640 1 --2- 172.21.15.40:0/3506085374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faf840cbb50 0x7faf8414e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:57.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.288+0000 7faf8b400640 1 -- 172.21.15.40:0/3506085374 >> 172.21.15.40:0/3506085374 conn(0x7faf840d0be0 msgr2=0x7faf840b14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:57.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.289+0000 7faf8b400640 1 -- 172.21.15.40:0/3506085374 shutdown_connections 2024-04-26T23:09:57.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:57.289+0000 7faf8b400640 1 -- 172.21.15.40:0/3506085374 wait complete. 2024-04-26T23:09:57.309 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'idle+waiting paths [/d1/d2/d3]', 'c': 'idle+waiting paths [/d1/d2/d3]', 'd': 'idle+waiting paths [/d1/d2/d3]'} 2024-04-26T23:09:58.310 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:09:58.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.531+0000 7f287a800640 1 Processor -- start 2024-04-26T23:09:58.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.532+0000 7f287a800640 1 -- start start 2024-04-26T23:09:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.533+0000 7f287a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2874163430 0x7f2874166f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.533+0000 7f287a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28741674a0 0x7f2874167920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.533+0000 7f287a800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2874163810 con 0x7f28740c62c0 2024-04-26T23:09:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.533+0000 7f287a800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f28740cb400 con 0x7f2874163430 2024-04-26T23:09:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.533+0000 7f287a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2874167e60 con 0x7f28741674a0 2024-04-26T23:09:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.534+0000 7f2878a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28741674a0 0x7f2874167920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.534+0000 7f2878a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28741674a0 0x7f2874167920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:38770/0 (socket says 172.21.15.40:38770) 2024-04-26T23:09:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.534+0000 7f2878a00640 1 -- 172.21.15.40:0/76464876 learned_addr learned my addr 172.21.15.40:0/76464876 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.534+0000 7f2879400640 1 --2- 172.21.15.40:0/76464876 >> v2:172.21.15.103:3300/0 conn(0x7f2874163430 0x7f2874166f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:58.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.535+0000 7f2878a00640 1 -- 172.21.15.40:0/76464876 >> v2:172.21.15.103:3300/0 conn(0x7f2874163430 msgr2=0x7f2874166f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:58.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.535+0000 7f2878a00640 1 --2- 172.21.15.40:0/76464876 >> v2:172.21.15.103:3300/0 conn(0x7f2874163430 0x7f2874166f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:58.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.535+0000 7f2878a00640 1 -- 172.21.15.40:0/76464876 >> v1:172.21.15.103:6789/0 conn(0x7f28740c62c0 legacy=0x7f28740c6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:58.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.535+0000 7f2878a00640 1 -- 172.21.15.40:0/76464876 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28740cb100 con 0x7f28741674a0 2024-04-26T23:09:58.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.536+0000 7f2878a00640 1 --2- 172.21.15.40:0/76464876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28741674a0 0x7f2874167920 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f28740cb580 tx=0x7f286002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6736c11c5aa6d835 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.537+0000 7f2873e00640 1 -- 172.21.15.40:0/76464876 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2860037070 con 0x7f28741674a0 2024-04-26T23:09:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.537+0000 7f2873e00640 1 -- 172.21.15.40:0/76464876 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2860032440 con 0x7f28741674a0 2024-04-26T23:09:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.537+0000 7f2873e00640 1 -- 172.21.15.40:0/76464876 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2860032810 con 0x7f28741674a0 2024-04-26T23:09:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.538+0000 7f287a800640 1 -- 172.21.15.40:0/76464876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28741674a0 msgr2=0x7f2874167920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.538+0000 7f287a800640 1 --2- 172.21.15.40:0/76464876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28741674a0 0x7f2874167920 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f28740cb580 tx=0x7f286002a250 comp rx=0 tx=0).stop 2024-04-26T23:09:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.538+0000 7f287a800640 1 -- 172.21.15.40:0/76464876 shutdown_connections 2024-04-26T23:09:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.538+0000 7f287a800640 1 --2- 172.21.15.40:0/76464876 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28741674a0 0x7f2874167920 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.538+0000 7f287a800640 1 --2- 172.21.15.40:0/76464876 >> v2:172.21.15.103:3300/0 conn(0x7f2874163430 0x7f2874166f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.538+0000 7f287a800640 1 -- 172.21.15.40:0/76464876 >> 172.21.15.40:0/76464876 conn(0x7f28740d0a10 msgr2=0x7f28740d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.538+0000 7f287a800640 1 -- 172.21.15.40:0/76464876 shutdown_connections 2024-04-26T23:09:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.538+0000 7f287a800640 1 -- 172.21.15.40:0/76464876 wait complete. 2024-04-26T23:09:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.540+0000 7f287a800640 1 Processor -- start 2024-04-26T23:09:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.540+0000 7f287a800640 1 -- start start 2024-04-26T23:09:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.541+0000 7f287a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28740c62c0 0x7f287415e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.541+0000 7f287a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2874163430 0x7f28740c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.541+0000 7f287a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28741674a0 0x7f2874072c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.541+0000 7f287a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f28740afe80 con 0x7f2874163430 2024-04-26T23:09:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.541+0000 7f287a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f28740ccc60 con 0x7f28740c62c0 2024-04-26T23:09:58.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.541+0000 7f287a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f28740cd1e0 con 0x7f28741674a0 2024-04-26T23:09:58.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.541+0000 7f2878a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2874163430 0x7f28740c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:58.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.541+0000 7f2878a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2874163430 0x7f28740c94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45908/0 (socket says 172.21.15.40:45908) 2024-04-26T23:09:58.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.541+0000 7f2878a00640 1 -- 172.21.15.40:0/3100352382 learned_addr learned my addr 172.21.15.40:0/3100352382 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:58.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.542+0000 7f2879e00640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28741674a0 0x7f2874072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:58.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.542+0000 7f2878a00640 1 -- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28741674a0 msgr2=0x7f2874072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:58.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.542+0000 7f2878a00640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28741674a0 0x7f2874072c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:58.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.542+0000 7f2879400640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28740c62c0 0x7f287415e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:58.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.542+0000 7f2878a00640 1 -- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28740c62c0 msgr2=0x7f287415e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:58.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.542+0000 7f2878a00640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28740c62c0 0x7f287415e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:58.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.542+0000 7f2878a00640 1 -- 172.21.15.40:0/3100352382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28740c8940 con 0x7f2874163430 2024-04-26T23:09:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.542+0000 7f2879e00640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28741674a0 0x7f2874072c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.542+0000 7f2878a00640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2874163430 0x7f28740c94e0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f2860000c00 tx=0x7f28600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.542+0000 7f2871600640 1 -- 172.21.15.40:0/3100352382 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2860037040 con 0x7f2874163430 2024-04-26T23:09:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.542+0000 7f2871600640 1 -- 172.21.15.40:0/3100352382 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2860042070 con 0x7f2874163430 2024-04-26T23:09:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.542+0000 7f287a800640 1 -- 172.21.15.40:0/3100352382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28740c7b80 con 0x7f2874163430 2024-04-26T23:09:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.543+0000 7f2871600640 1 -- 172.21.15.40:0/3100352382 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2860030940 con 0x7f2874163430 2024-04-26T23:09:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.543+0000 7f287a800640 1 -- 172.21.15.40:0/3100352382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28740c8420 con 0x7f2874163430 2024-04-26T23:09:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.544+0000 7f287a800640 1 -- 172.21.15.40:0/3100352382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28740c7d50 con 0x7f2874163430 2024-04-26T23:09:58.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.545+0000 7f2871600640 1 -- 172.21.15.40:0/3100352382 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2860030ae0 con 0x7f2874163430 2024-04-26T23:09:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.546+0000 7f2871600640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f285406c8f0 0x7f285406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.546+0000 7f2871600640 1 -- 172.21.15.40:0/3100352382 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f28600ac630 con 0x7f2874163430 2024-04-26T23:09:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.547+0000 7f2879400640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f285406c8f0 0x7f285406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:58.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.548+0000 7f2879400640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f285406c8f0 0x7f285406edb0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f28740b46d0 tx=0x7f2868008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:58.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.551+0000 7f2871600640 1 -- 172.21.15.40:0/3100352382 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f286007a810 con 0x7f2874163430 2024-04-26T23:09:58.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.802+0000 7f287a800640 1 -- 172.21.15.40:0/3100352382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2874069370 con 0x7f2874163430 2024-04-26T23:09:58.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.803+0000 7f2871600640 1 -- 172.21.15.40:0/3100352382 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5239 (secure 0 0 0) 0x7f286007a110 con 0x7f2874163430 2024-04-26T23:09:58.803 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:58.804 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:44.797258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[6598,8949,8955]},"id":7}]} 2024-04-26T23:09:58.804 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 122 2024-04-26T23:09:58.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.806+0000 7f287a800640 1 -- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f285406c8f0 msgr2=0x7f285406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.806+0000 7f287a800640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f285406c8f0 0x7f285406edb0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f28740b46d0 tx=0x7f2868008040 comp rx=0 tx=0).stop 2024-04-26T23:09:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.806+0000 7f287a800640 1 -- 172.21.15.40:0/3100352382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2874163430 msgr2=0x7f28740c94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.806+0000 7f287a800640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2874163430 0x7f28740c94e0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f2860000c00 tx=0x7f28600046c0 comp rx=0 tx=0).stop 2024-04-26T23:09:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.806+0000 7f287a800640 1 -- 172.21.15.40:0/3100352382 shutdown_connections 2024-04-26T23:09:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.806+0000 7f287a800640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f285406c8f0 0x7f285406edb0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.806+0000 7f287a800640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28741674a0 0x7f2874072c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.806+0000 7f287a800640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2874163430 0x7f28740c94e0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.806+0000 7f287a800640 1 --2- 172.21.15.40:0/3100352382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28740c62c0 0x7f287415e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.806+0000 7f287a800640 1 -- 172.21.15.40:0/3100352382 >> 172.21.15.40:0/3100352382 conn(0x7f28740d0a10 msgr2=0x7f287406f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.806+0000 7f287a800640 1 -- 172.21.15.40:0/3100352382 shutdown_connections 2024-04-26T23:09:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:58.807+0000 7f287a800640 1 -- 172.21.15.40:0/3100352382 wait complete. 2024-04-26T23:09:58.826 DEBUG:tasks.cephfs.test_multimds_misc:current active=['a', 'c', 'd'] 2024-04-26T23:09:58.826 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:09:59.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.036+0000 7f715d800640 1 Processor -- start 2024-04-26T23:09:59.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.037+0000 7f715d800640 1 -- start start 2024-04-26T23:09:59.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.039+0000 7f715d800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7158155f20 0x7f71580d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:59.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.039+0000 7f715d800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715815fca0 con 0x7f71580b5500 2024-04-26T23:09:59.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.039+0000 7f715d800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715815ff80 con 0x7f715815f490 2024-04-26T23:09:59.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.039+0000 7f715d800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7158160260 con 0x7f7158155f20 2024-04-26T23:09:59.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.039+0000 7f715ce00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f71580b5500 0x7f71580b3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48464/0 (socket says 172.21.15.40:48464) 2024-04-26T23:09:59.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.039+0000 7f715ce00640 1 -- 172.21.15.40:0/443025771 learned_addr learned my addr 172.21.15.40:0/443025771 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:09:59.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157e00640 1 --2- 172.21.15.40:0/443025771 >> v2:172.21.15.103:3300/0 conn(0x7f7158155f20 0x7f71580d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:59.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2505702134 0 0) 0x7f715815fca0 con 0x7f71580b5500 2024-04-26T23:09:59.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f713c003520 con 0x7f71580b5500 2024-04-26T23:09:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1892563986 0 0) 0x7f713c003520 con 0x7f71580b5500 2024-04-26T23:09:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 >> v2:172.21.15.103:3300/0 conn(0x7f7158155f20 msgr2=0x7f71580d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 --2- 172.21.15.40:0/443025771 >> v2:172.21.15.103:3300/0 conn(0x7f7158155f20 0x7f71580d11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 >> v1:172.21.15.103:6789/0 conn(0x7f715815f490 legacy=0x7f715806e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715806e040 con 0x7f71580b5500 2024-04-26T23:09:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f71500019f0 con 0x7f71580b5500 2024-04-26T23:09:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 >> v1:172.21.15.40:6789/0 conn(0x7f71580b5500 legacy=0x7f71580b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 --2- 172.21.15.40:0/443025771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f713c003a00 0x7f713c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 --2- 172.21.15.40:0/443025771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f713c004800 0x7f713c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 --2- 172.21.15.40:0/443025771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0051c0 0x7f713c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7158160260 con 0x7f713c0051c0 2024-04-26T23:09:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f713c00c0c0 con 0x7f713c003a00 2024-04-26T23:09:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.040+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f713c00c2c0 con 0x7f713c004800 2024-04-26T23:09:59.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.041+0000 7f715ce00640 1 --2- 172.21.15.40:0/443025771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0051c0 0x7f713c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:59.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.041+0000 7f714fe00640 1 --2- 172.21.15.40:0/443025771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f713c004800 0x7f713c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:59.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.041+0000 7f7157e00640 1 --2- 172.21.15.40:0/443025771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f713c003a00 0x7f713c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:59.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.041+0000 7f715ce00640 1 -- 172.21.15.40:0/443025771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f713c004800 msgr2=0x7f713c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:59.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.041+0000 7f715ce00640 1 --2- 172.21.15.40:0/443025771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f713c004800 0x7f713c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:59.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.041+0000 7f715ce00640 1 -- 172.21.15.40:0/443025771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f713c003a00 msgr2=0x7f713c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:09:59.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.041+0000 7f715ce00640 1 --2- 172.21.15.40:0/443025771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f713c003a00 0x7f713c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:59.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.041+0000 7f715ce00640 1 -- 172.21.15.40:0/443025771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713c003520 con 0x7f713c0051c0 2024-04-26T23:09:59.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.041+0000 7f7157e00640 1 --2- 172.21.15.40:0/443025771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f713c003a00 0x7f713c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:09:59.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.042+0000 7f715ce00640 1 -- 172.21.15.40:0/443025771 reap_dead start 2024-04-26T23:09:59.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.042+0000 7f715ce00640 1 --2- 172.21.15.40:0/443025771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0051c0 0x7f713c00b9c0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f715000a770 tx=0x7f713c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e848f1cdf8d60ff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:59.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.042+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715000a390 con 0x7f713c0051c0 2024-04-26T23:09:59.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.043+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f715002f070 con 0x7f713c0051c0 2024-04-26T23:09:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.043+0000 7f7157400640 1 -- 172.21.15.40:0/443025771 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715002bbe0 con 0x7f713c0051c0 2024-04-26T23:09:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.043+0000 7f715d800640 1 -- 172.21.15.40:0/443025771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0051c0 msgr2=0x7f713c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.043+0000 7f715d800640 1 --2- 172.21.15.40:0/443025771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0051c0 0x7f713c00b9c0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f715000a770 tx=0x7f713c005830 comp rx=0 tx=0).stop 2024-04-26T23:09:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.043+0000 7f715d800640 1 -- 172.21.15.40:0/443025771 shutdown_connections 2024-04-26T23:09:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.043+0000 7f715d800640 1 --2- 172.21.15.40:0/443025771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0051c0 0x7f713c00b9c0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.043+0000 7f715d800640 1 -- 172.21.15.40:0/443025771 >> 172.21.15.40:0/443025771 conn(0x7f7158075eb0 msgr2=0x7f7158068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.043+0000 7f715d800640 1 -- 172.21.15.40:0/443025771 shutdown_connections 2024-04-26T23:09:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.043+0000 7f715d800640 1 -- 172.21.15.40:0/443025771 wait complete. 2024-04-26T23:09:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.045+0000 7f715d800640 1 Processor -- start 2024-04-26T23:09:59.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.045+0000 7f715d800640 1 -- start start 2024-04-26T23:09:59.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.046+0000 7f715d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0051c0 0x7f7158148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:59.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.046+0000 7f715d800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7158158400 0x7f7158158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:59.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.046+0000 7f715d800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f715815a200 0x7f715815a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:59.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.046+0000 7f715d800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f715815fa80 con 0x7f713c0051c0 2024-04-26T23:09:59.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.046+0000 7f715d800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f715815f900 con 0x7f7158158400 2024-04-26T23:09:59.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.046+0000 7f715d800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f71580d11a0 con 0x7f715815a200 2024-04-26T23:09:59.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.047+0000 7f7157e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0051c0 0x7f7158148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:59.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.047+0000 7f7157e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0051c0 0x7f7158148530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45938/0 (socket says 172.21.15.40:45938) 2024-04-26T23:09:59.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.047+0000 7f7157e00640 1 -- 172.21.15.40:0/269508051 learned_addr learned my addr 172.21.15.40:0/269508051 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:09:59.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.047+0000 7f714fe00640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7158158400 0x7f7158158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:59.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.047+0000 7f715ce00640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f715815a200 0x7f715815a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:59.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.047+0000 7f7157e00640 1 -- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f715815a200 msgr2=0x7f715815a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:59.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.047+0000 7f7157e00640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f715815a200 0x7f715815a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:59.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.047+0000 7f7157e00640 1 -- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7158158400 msgr2=0x7f7158158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:59.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.047+0000 7f7157e00640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7158158400 0x7f7158158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:59.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.047+0000 7f7157e00640 1 -- 172.21.15.40:0/269508051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71580afa40 con 0x7f713c0051c0 2024-04-26T23:09:59.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.048+0000 7f7157e00640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0051c0 0x7f7158148530 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f71480108b0 tx=0x7f7148010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:59.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.048+0000 7f7154c00640 1 -- 172.21.15.40:0/269508051 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714800f040 con 0x7f713c0051c0 2024-04-26T23:09:59.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.048+0000 7f7154c00640 1 -- 172.21.15.40:0/269508051 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f714800eb60 con 0x7f713c0051c0 2024-04-26T23:09:59.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.048+0000 7f7154c00640 1 -- 172.21.15.40:0/269508051 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714801c400 con 0x7f713c0051c0 2024-04-26T23:09:59.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.048+0000 7f715d800640 1 -- 172.21.15.40:0/269508051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71580afd20 con 0x7f713c0051c0 2024-04-26T23:09:59.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.048+0000 7f715d800640 1 -- 172.21.15.40:0/269508051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71580b02e0 con 0x7f713c0051c0 2024-04-26T23:09:59.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.049+0000 7f715d800640 1 -- 172.21.15.40:0/269508051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7158069370 con 0x7f713c0051c0 2024-04-26T23:09:59.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.050+0000 7f7154c00640 1 -- 172.21.15.40:0/269508051 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f714801c5a0 con 0x7f713c0051c0 2024-04-26T23:09:59.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.051+0000 7f7154c00640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f713406c920 0x7f713406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:09:59.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.052+0000 7f7154c00640 1 -- 172.21.15.40:0/269508051 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f714808c420 con 0x7f713c0051c0 2024-04-26T23:09:59.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.052+0000 7f714fe00640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f713406c920 0x7f713406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:09:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.053+0000 7f714fe00640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f713406c920 0x7f713406ede0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7158159020 tx=0x7f7140009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:09:59.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.056+0000 7f7154c00640 1 -- 172.21.15.40:0/269508051 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7148018030 con 0x7f713c0051c0 2024-04-26T23:09:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.342+0000 7f715d800640 1 -- 172.21.15.40:0/269508051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f715806f3e0 con 0x7f713c0051c0 2024-04-26T23:09:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.343+0000 7f7154c00640 1 -- 172.21.15.40:0/269508051 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1919 (secure 0 0 0) 0x7f714805a040 con 0x7f713c0051c0 2024-04-26T23:09:59.344 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:09:59.344 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":559,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":105,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":712040,"bytes_used":267436032,"bytes_avail":772826677248,"bytes_total":773094113280,"write_bytes_sec":801,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":122,"id":7,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":7,"rank":2,"name":"d","status":"up:active","gid":6598},{"filesystem_id":7,"rank":1,"name":"c","status":"up:active","gid":8949},{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8955}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":31,"modified":"2024-04-26T23:09:57.342947+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"idle+waiting paths [/d1/d2/d3]"}}}}}},"progress_events":{}} 2024-04-26T23:09:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.346+0000 7f715d800640 1 -- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f713406c920 msgr2=0x7f713406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.346+0000 7f715d800640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f713406c920 0x7f713406ede0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7158159020 tx=0x7f7140009040 comp rx=0 tx=0).stop 2024-04-26T23:09:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.346+0000 7f715d800640 1 -- 172.21.15.40:0/269508051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0051c0 msgr2=0x7f7158148530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:09:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.346+0000 7f715d800640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0051c0 0x7f7158148530 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f71480108b0 tx=0x7f7148010d70 comp rx=0 tx=0).stop 2024-04-26T23:09:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.346+0000 7f715d800640 1 -- 172.21.15.40:0/269508051 shutdown_connections 2024-04-26T23:09:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.346+0000 7f715d800640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f713406c920 0x7f713406ede0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.346+0000 7f715d800640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f715815a200 0x7f715815a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.346+0000 7f715d800640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7158158400 0x7f7158158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.346+0000 7f715d800640 1 --2- 172.21.15.40:0/269508051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f713c0051c0 0x7f7158148530 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:09:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.346+0000 7f715d800640 1 -- 172.21.15.40:0/269508051 >> 172.21.15.40:0/269508051 conn(0x7f7158075eb0 msgr2=0x7f71580d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:09:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.346+0000 7f715d800640 1 -- 172.21.15.40:0/269508051 shutdown_connections 2024-04-26T23:09:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:09:59.346+0000 7f715d800640 1 -- 172.21.15.40:0/269508051 wait complete. 2024-04-26T23:09:59.365 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'idle+waiting paths [/d1/d2/d3]', 'd': 'idle+waiting paths [/d1/d2/d3]'} 2024-04-26T23:09:59.999 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:09:59.999+0000 7f36ffc00640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 3 MDSs report damaged metadata 2024-04-26T23:10:00.367 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:10:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.590+0000 7f5e42800640 1 Processor -- start 2024-04-26T23:10:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.590+0000 7f5e42800640 1 -- start start 2024-04-26T23:10:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.592+0000 7f5e42800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5e3c071e60 0x7f5e3c06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.592+0000 7f5e42800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e3c06fe20 0x7f5e3c072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.592+0000 7f5e42800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e3c15d7c0 con 0x7f5e3c072780 2024-04-26T23:10:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.592+0000 7f5e42800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f5e3c15daa0 con 0x7f5e3c071e60 2024-04-26T23:10:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.592+0000 7f5e42800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5e3c15dd80 con 0x7f5e3c06fe20 2024-04-26T23:10:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.592+0000 7f5e3b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5e3c071e60 0x7f5e3c06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.592+0000 7f5e3b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5e3c071e60 0x7f5e3c06f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45952/0 (socket says 172.21.15.40:45952) 2024-04-26T23:10:00.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.592+0000 7f5e3b400640 1 -- 172.21.15.40:0/1548947069 learned_addr learned my addr 172.21.15.40:0/1548947069 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:00.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.593+0000 7f5e3b400640 1 -- 172.21.15.40:0/1548947069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e3c06fe20 msgr2=0x7f5e3c072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:00.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.592+0000 7f5e3aa00640 1 --2- 172.21.15.40:0/1548947069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e3c06fe20 0x7f5e3c072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:00.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.593+0000 7f5e3b400640 1 --2- 172.21.15.40:0/1548947069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e3c06fe20 0x7f5e3c072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:00.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.593+0000 7f5e3b400640 1 -- 172.21.15.40:0/1548947069 >> v1:172.21.15.103:6789/0 conn(0x7f5e3c072780 legacy=0x7f5e3c0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:00.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.593+0000 7f5e3b400640 1 -- 172.21.15.40:0/1548947069 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e3c15e060 con 0x7f5e3c071e60 2024-04-26T23:10:00.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.593+0000 7f5e3aa00640 1 --2- 172.21.15.40:0/1548947069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e3c06fe20 0x7f5e3c072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:00.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.594+0000 7f5e3b400640 1 --2- 172.21.15.40:0/1548947069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e3c071e60 0x7f5e3c06f8e0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009870 tx=0x7f5e2c02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7db4d8cbe16097b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:00.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.594+0000 7f5e3a000640 1 -- 172.21.15.40:0/1548947069 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c037070 con 0x7f5e3c071e60 2024-04-26T23:10:00.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.594+0000 7f5e3a000640 1 -- 172.21.15.40:0/1548947069 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e2c032440 con 0x7f5e3c071e60 2024-04-26T23:10:00.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.594+0000 7f5e3a000640 1 -- 172.21.15.40:0/1548947069 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c032810 con 0x7f5e3c071e60 2024-04-26T23:10:00.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.595+0000 7f5e42800640 1 -- 172.21.15.40:0/1548947069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e3c071e60 msgr2=0x7f5e3c06f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:00.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.595+0000 7f5e42800640 1 --2- 172.21.15.40:0/1548947069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e3c071e60 0x7f5e3c06f8e0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009870 tx=0x7f5e2c02a200 comp rx=0 tx=0).stop 2024-04-26T23:10:00.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.595+0000 7f5e42800640 1 -- 172.21.15.40:0/1548947069 shutdown_connections 2024-04-26T23:10:00.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.595+0000 7f5e42800640 1 --2- 172.21.15.40:0/1548947069 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e3c06fe20 0x7f5e3c072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:00.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.595+0000 7f5e42800640 1 --2- 172.21.15.40:0/1548947069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e3c071e60 0x7f5e3c06f8e0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:00.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.595+0000 7f5e42800640 1 -- 172.21.15.40:0/1548947069 >> 172.21.15.40:0/1548947069 conn(0x7f5e3c0d0be0 msgr2=0x7f5e3c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:00.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.595+0000 7f5e42800640 1 -- 172.21.15.40:0/1548947069 shutdown_connections 2024-04-26T23:10:00.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.595+0000 7f5e42800640 1 -- 172.21.15.40:0/1548947069 wait complete. 2024-04-26T23:10:00.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.598+0000 7f5e42800640 1 Processor -- start 2024-04-26T23:10:00.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.598+0000 7f5e42800640 1 -- start start 2024-04-26T23:10:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.599+0000 7f5e42800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e3c06fe20 0x7f5e3c1f8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.599+0000 7f5e42800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e3c071e60 0x7f5e3c15c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.599+0000 7f5e42800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e3c072780 0x7f5e3c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.599+0000 7f5e42800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3c15d3f0 con 0x7f5e3c071e60 2024-04-26T23:10:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.599+0000 7f5e42800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3c072c10 con 0x7f5e3c06fe20 2024-04-26T23:10:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.599+0000 7f5e42800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5e3c15d570 con 0x7f5e3c072780 2024-04-26T23:10:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.600+0000 7f5e3aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e3c071e60 0x7f5e3c15c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.600+0000 7f5e3aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e3c071e60 0x7f5e3c15c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45968/0 (socket says 172.21.15.40:45968) 2024-04-26T23:10:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.600+0000 7f5e3aa00640 1 -- 172.21.15.40:0/1242843594 learned_addr learned my addr 172.21.15.40:0/1242843594 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.600+0000 7f5e3b400640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e3c06fe20 0x7f5e3c1f8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.600+0000 7f5e3be00640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e3c072780 0x7f5e3c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.600+0000 7f5e3aa00640 1 -- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e3c072780 msgr2=0x7f5e3c15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.600+0000 7f5e3aa00640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e3c072780 0x7f5e3c15b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.600+0000 7f5e3aa00640 1 -- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e3c06fe20 msgr2=0x7f5e3c1f8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.600+0000 7f5e3aa00640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e3c06fe20 0x7f5e3c1f8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.600+0000 7f5e3aa00640 1 -- 172.21.15.40:0/1242843594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e3c0c8df0 con 0x7f5e3c071e60 2024-04-26T23:10:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.601+0000 7f5e3aa00640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e3c071e60 0x7f5e3c15c8d0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800c7f0 tx=0x7f5e2800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.601+0000 7f5e33400640 1 -- 172.21.15.40:0/1242843594 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e28015070 con 0x7f5e3c071e60 2024-04-26T23:10:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.601+0000 7f5e33400640 1 -- 172.21.15.40:0/1242843594 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e28007a30 con 0x7f5e3c071e60 2024-04-26T23:10:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.601+0000 7f5e33400640 1 -- 172.21.15.40:0/1242843594 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e28007e00 con 0x7f5e3c071e60 2024-04-26T23:10:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.601+0000 7f5e42800640 1 -- 172.21.15.40:0/1242843594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e3c1f8b70 con 0x7f5e3c071e60 2024-04-26T23:10:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.601+0000 7f5e42800640 1 -- 172.21.15.40:0/1242843594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e3c1f9130 con 0x7f5e3c071e60 2024-04-26T23:10:00.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.602+0000 7f5e42800640 1 -- 172.21.15.40:0/1242843594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e3c0694f0 con 0x7f5e3c071e60 2024-04-26T23:10:00.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.603+0000 7f5e33400640 1 -- 172.21.15.40:0/1242843594 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5e28018070 con 0x7f5e3c071e60 2024-04-26T23:10:00.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.605+0000 7f5e33400640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5e1c06c8f0 0x7f5e1c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:00.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.605+0000 7f5e33400640 1 -- 172.21.15.40:0/1242843594 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f5e2808bc60 con 0x7f5e3c071e60 2024-04-26T23:10:00.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.606+0000 7f5e3b400640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5e1c06c8f0 0x7f5e1c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:00.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.607+0000 7f5e3b400640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5e1c06c8f0 0x7f5e1c06edb0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c034000 tx=0x7f5e2c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:00.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.610+0000 7f5e33400640 1 -- 172.21.15.40:0/1242843594 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5e280597f0 con 0x7f5e3c071e60 2024-04-26T23:10:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.854+0000 7f5e42800640 1 -- 172.21.15.40:0/1242843594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e3c06faf0 con 0x7f5e3c071e60 2024-04-26T23:10:00.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.855+0000 7f5e33400640 1 -- 172.21.15.40:0/1242843594 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5239 (secure 0 0 0) 0x7f5e2805daf0 con 0x7f5e3c071e60 2024-04-26T23:10:00.856 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:00.856 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:44.797258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[6598,8949,8955]},"id":7}]} 2024-04-26T23:10:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 122 2024-04-26T23:10:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.858+0000 7f5e42800640 1 -- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5e1c06c8f0 msgr2=0x7f5e1c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.858+0000 7f5e42800640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5e1c06c8f0 0x7f5e1c06edb0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c034000 tx=0x7f5e2c02e000 comp rx=0 tx=0).stop 2024-04-26T23:10:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.858+0000 7f5e42800640 1 -- 172.21.15.40:0/1242843594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e3c071e60 msgr2=0x7f5e3c15c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.858+0000 7f5e42800640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e3c071e60 0x7f5e3c15c8d0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800c7f0 tx=0x7f5e2800cbf0 comp rx=0 tx=0).stop 2024-04-26T23:10:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.858+0000 7f5e42800640 1 -- 172.21.15.40:0/1242843594 shutdown_connections 2024-04-26T23:10:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.858+0000 7f5e42800640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5e1c06c8f0 0x7f5e1c06edb0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.858+0000 7f5e42800640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e3c072780 0x7f5e3c15b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.858+0000 7f5e42800640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e3c071e60 0x7f5e3c15c8d0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.858+0000 7f5e42800640 1 --2- 172.21.15.40:0/1242843594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e3c06fe20 0x7f5e3c1f8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.858+0000 7f5e42800640 1 -- 172.21.15.40:0/1242843594 >> 172.21.15.40:0/1242843594 conn(0x7f5e3c0d0be0 msgr2=0x7f5e3c15f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.858+0000 7f5e42800640 1 -- 172.21.15.40:0/1242843594 shutdown_connections 2024-04-26T23:10:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:00.858+0000 7f5e42800640 1 -- 172.21.15.40:0/1242843594 wait complete. 2024-04-26T23:10:00.877 DEBUG:tasks.cephfs.test_multimds_misc:current active=['a', 'c', 'd'] 2024-04-26T23:10:00.877 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-26T23:10:01.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.082+0000 7f027bc00640 1 Processor -- start 2024-04-26T23:10:01.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.082+0000 7f027bc00640 1 -- start start 2024-04-26T23:10:01.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.084+0000 7f027bc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f027415b600 0x7f0274071f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.084+0000 7f027bc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f02740cf5e0 0x7f02740ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.084+0000 7f027bc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0274072450 con 0x7f02740cbbd0 2024-04-26T23:10:01.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.084+0000 7f027bc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f02740cb9b0 con 0x7f027415b600 2024-04-26T23:10:01.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.084+0000 7f027bc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f027415b9e0 con 0x7f02740cf5e0 2024-04-26T23:10:01.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.084+0000 7f0279200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f027415b600 0x7f0274071f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:01.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.084+0000 7f0279c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f02740cbbd0 0x7f027406f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48472/0 (socket says 172.21.15.40:48472) 2024-04-26T23:10:01.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.084+0000 7f0279200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f027415b600 0x7f0274071f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45972/0 (socket says 172.21.15.40:45972) 2024-04-26T23:10:01.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.084+0000 7f0279c00640 1 -- 172.21.15.40:0/699948890 learned_addr learned my addr 172.21.15.40:0/699948890 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:01.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.085+0000 7f026be00640 1 --2- 172.21.15.40:0/699948890 >> v2:172.21.15.103:3300/0 conn(0x7f02740cf5e0 0x7f02740ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:01.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.085+0000 7f0273e00640 1 -- 172.21.15.40:0/699948890 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2823476667 0 0) 0x7f026c001490 con 0x7f02740cbbd0 2024-04-26T23:10:01.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.085+0000 7f0279200640 1 -- 172.21.15.40:0/699948890 >> v2:172.21.15.103:3300/0 conn(0x7f02740cf5e0 msgr2=0x7f02740ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:01.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.085+0000 7f0279200640 1 --2- 172.21.15.40:0/699948890 >> v2:172.21.15.103:3300/0 conn(0x7f02740cf5e0 0x7f02740ce9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.085+0000 7f0279200640 1 -- 172.21.15.40:0/699948890 >> v1:172.21.15.40:6789/0 conn(0x7f02740cbbd0 legacy=0x7f027406f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:01.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.085+0000 7f0279200640 1 -- 172.21.15.40:0/699948890 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02740d0740 con 0x7f027415b600 2024-04-26T23:10:01.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.086+0000 7f0279200640 1 --2- 172.21.15.40:0/699948890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f027415b600 0x7f0274071f10 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f02740cbb30 tx=0x7f026402b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3fa44b85787ffa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:01.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.086+0000 7f0273e00640 1 -- 172.21.15.40:0/699948890 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026400b040 con 0x7f027415b600 2024-04-26T23:10:01.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.086+0000 7f0273e00640 1 -- 172.21.15.40:0/699948890 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0264030710 con 0x7f027415b600 2024-04-26T23:10:01.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.086+0000 7f0273e00640 1 -- 172.21.15.40:0/699948890 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0264030ae0 con 0x7f027415b600 2024-04-26T23:10:01.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.087+0000 7f027bc00640 1 -- 172.21.15.40:0/699948890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f027415b600 msgr2=0x7f0274071f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:01.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.087+0000 7f027bc00640 1 --2- 172.21.15.40:0/699948890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f027415b600 0x7f0274071f10 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f02740cbb30 tx=0x7f026402b4d0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.087+0000 7f027bc00640 1 -- 172.21.15.40:0/699948890 shutdown_connections 2024-04-26T23:10:01.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.087+0000 7f027bc00640 1 --2- 172.21.15.40:0/699948890 >> v2:172.21.15.103:3300/0 conn(0x7f02740cf5e0 0x7f02740ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.087+0000 7f027bc00640 1 --2- 172.21.15.40:0/699948890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f027415b600 0x7f0274071f10 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.087+0000 7f027bc00640 1 -- 172.21.15.40:0/699948890 >> 172.21.15.40:0/699948890 conn(0x7f02740d0be0 msgr2=0x7f02740d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:01.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.087+0000 7f027bc00640 1 -- 172.21.15.40:0/699948890 shutdown_connections 2024-04-26T23:10:01.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.087+0000 7f027bc00640 1 -- 172.21.15.40:0/699948890 wait complete. 2024-04-26T23:10:01.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.089+0000 7f027bc00640 1 Processor -- start 2024-04-26T23:10:01.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.089+0000 7f027bc00640 1 -- start start 2024-04-26T23:10:01.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.090+0000 7f027bc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02740cbbd0 0x7f02740b5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.090+0000 7f027bc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02740cf5e0 0x7f02741fd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.090+0000 7f027bc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f027415b600 0x7f02741f94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.090+0000 7f027bc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f02740d02f0 con 0x7f02740cbbd0 2024-04-26T23:10:01.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.090+0000 7f027bc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f02740cb640 con 0x7f027415b600 2024-04-26T23:10:01.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.090+0000 7f027bc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f02740d0470 con 0x7f02740cf5e0 2024-04-26T23:10:01.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.090+0000 7f0279200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02740cbbd0 0x7f02740b5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:01.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.090+0000 7f0279200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02740cbbd0 0x7f02740b5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45986/0 (socket says 172.21.15.40:45986) 2024-04-26T23:10:01.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.090+0000 7f0279200640 1 -- 172.21.15.40:0/2083126494 learned_addr learned my addr 172.21.15.40:0/2083126494 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:01.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.091+0000 7f026be00640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02740cf5e0 0x7f02741fd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:01.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.091+0000 7f0279c00640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f027415b600 0x7f02741f94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:01.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.091+0000 7f0279200640 1 -- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02740cf5e0 msgr2=0x7f02741fd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:01.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.091+0000 7f0279200640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02740cf5e0 0x7f02741fd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.091+0000 7f0279200640 1 -- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f027415b600 msgr2=0x7f02741f94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:01.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.091+0000 7f0279200640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f027415b600 0x7f02741f94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.091+0000 7f0279200640 1 -- 172.21.15.40:0/2083126494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02741fd9c0 con 0x7f02740cbbd0 2024-04-26T23:10:01.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.091+0000 7f0279200640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02740cbbd0 0x7f02740b5890 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f0264000c00 tx=0x7f02640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:01.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.091+0000 7f0271600640 1 -- 172.21.15.40:0/2083126494 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026403f070 con 0x7f02740cbbd0 2024-04-26T23:10:01.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.091+0000 7f0271600640 1 -- 172.21.15.40:0/2083126494 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0264032cc0 con 0x7f02740cbbd0 2024-04-26T23:10:01.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.091+0000 7f0271600640 1 -- 172.21.15.40:0/2083126494 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0264006e10 con 0x7f02740cbbd0 2024-04-26T23:10:01.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.091+0000 7f027bc00640 1 -- 172.21.15.40:0/2083126494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02741f9ed0 con 0x7f02740cbbd0 2024-04-26T23:10:01.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.092+0000 7f027bc00640 1 -- 172.21.15.40:0/2083126494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02741fcc80 con 0x7f02740cbbd0 2024-04-26T23:10:01.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.093+0000 7f027bc00640 1 -- 172.21.15.40:0/2083126494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02740694f0 con 0x7f02740cbbd0 2024-04-26T23:10:01.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.094+0000 7f0271600640 1 -- 172.21.15.40:0/2083126494 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0264004250 con 0x7f02740cbbd0 2024-04-26T23:10:01.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.095+0000 7f0271600640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f025006c8f0 0x7f025006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.095+0000 7f0271600640 1 -- 172.21.15.40:0/2083126494 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f02640abe40 con 0x7f02740cbbd0 2024-04-26T23:10:01.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.096+0000 7f026be00640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f025006c8f0 0x7f025006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:01.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.097+0000 7f026be00640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f025006c8f0 0x7f025006edb0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f02740b2170 tx=0x7f025c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:01.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.100+0000 7f0271600640 1 -- 172.21.15.40:0/2083126494 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0264048060 con 0x7f02740cbbd0 2024-04-26T23:10:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.386+0000 7f027bc00640 1 -- 172.21.15.40:0/2083126494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f02740ce7b0 con 0x7f02740cbbd0 2024-04-26T23:10:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.387+0000 7f0271600640 1 -- 172.21.15.40:0/2083126494 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1825 (secure 0 0 0) 0x7f02741fd9c0 con 0x7f02740cbbd0 2024-04-26T23:10:01.388 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:01.388 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","health":{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"3 MDSs report damaged metadata","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":561,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":105,"num_osds":8,"num_up_osds":8,"osd_up_since":1714172452,"num_in_osds":8,"osd_in_since":1714172443,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":70,"data_bytes":712040,"bytes_used":267436032,"bytes_avail":772826677248,"bytes_total":773094113280,"write_bytes_sec":716,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":122,"id":7,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":7,"rank":2,"name":"d","status":"up:active","gid":6598},{"filesystem_id":7,"rank":1,"name":"c","status":"up:active","gid":8949},{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8955}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":32,"modified":"2024-04-26T23:09:59.344017+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T23:10:01.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.390+0000 7f027bc00640 1 -- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f025006c8f0 msgr2=0x7f025006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.390+0000 7f027bc00640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f025006c8f0 0x7f025006edb0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f02740b2170 tx=0x7f025c008040 comp rx=0 tx=0).stop 2024-04-26T23:10:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.390+0000 7f027bc00640 1 -- 172.21.15.40:0/2083126494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02740cbbd0 msgr2=0x7f02740b5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.390+0000 7f027bc00640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02740cbbd0 0x7f02740b5890 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f0264000c00 tx=0x7f02640046c0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.390+0000 7f027bc00640 1 -- 172.21.15.40:0/2083126494 shutdown_connections 2024-04-26T23:10:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.390+0000 7f027bc00640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f025006c8f0 0x7f025006edb0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.390+0000 7f027bc00640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f027415b600 0x7f02741f94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.390+0000 7f027bc00640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02740cf5e0 0x7f02741fd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.390+0000 7f027bc00640 1 --2- 172.21.15.40:0/2083126494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02740cbbd0 0x7f02740b5890 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.390+0000 7f027bc00640 1 -- 172.21.15.40:0/2083126494 >> 172.21.15.40:0/2083126494 conn(0x7f02740d0be0 msgr2=0x7f0274157220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.390+0000 7f027bc00640 1 -- 172.21.15.40:0/2083126494 shutdown_connections 2024-04-26T23:10:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.390+0000 7f027bc00640 1 -- 172.21.15.40:0/2083126494 wait complete. 2024-04-26T23:10:01.409 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-26T23:10:01.409 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-26T23:10:01.409 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:10:01.539 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-26T23:10:01.664 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-26T23:10:01.664 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:10:01.664 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:10:01.664 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:10:01.724 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:10:01.724 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:10:01.777 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:10:01.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.971+0000 7f07f0c00640 1 Processor -- start 2024-04-26T23:10:01.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.971+0000 7f07f0c00640 1 -- start start 2024-04-26T23:10:01.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.973+0000 7f07f0c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07ec073f30 0x7f07ec0cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.973+0000 7f07f0c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07ec163a50 con 0x7f07ec1671e0 2024-04-26T23:10:01.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.973+0000 7f07f0c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07ec0ceb90 con 0x7f07ec0c41d0 2024-04-26T23:10:01.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.973+0000 7f07f0c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f07ec0cf7a0 con 0x7f07ec073f30 2024-04-26T23:10:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.973+0000 7f07ea800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f07ec1671e0 0x7f07ec06ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48484/0 (socket says 172.21.15.40:48484) 2024-04-26T23:10:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.973+0000 7f07ea800640 1 -- 172.21.15.40:0/3799891698 learned_addr learned my addr 172.21.15.40:0/3799891698 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e9400640 1 --2- 172.21.15.40:0/3799891698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07ec073f30 0x7f07ec0cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4199303186 0 0) 0x7f07ec163a50 con 0x7f07ec1671e0 2024-04-26T23:10:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07d0003520 con 0x7f07ec1671e0 2024-04-26T23:10:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2852982465 0 0) 0x7f07d0003520 con 0x7f07ec1671e0 2024-04-26T23:10:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07ec073f30 msgr2=0x7f07ec0cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 --2- 172.21.15.40:0/3799891698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07ec073f30 0x7f07ec0cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 >> v1:172.21.15.103:6789/0 conn(0x7f07ec0c41d0 legacy=0x7f07ec0cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ec0cee70 con 0x7f07ec1671e0 2024-04-26T23:10:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f07e40019f0 con 0x7f07ec1671e0 2024-04-26T23:10:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 >> v1:172.21.15.40:6789/0 conn(0x7f07ec1671e0 legacy=0x7f07ec06ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 --2- 172.21.15.40:0/3799891698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07ec073f30 0x7f07d00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 --2- 172.21.15.40:0/3799891698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07d0003fb0 0x7f07d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 --2- 172.21.15.40:0/3799891698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07d0004d40 0x7f07d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec0cf7a0 con 0x7f07d0003fb0 2024-04-26T23:10:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f07d000bb00 con 0x7f07d0004d40 2024-04-26T23:10:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.974+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f07d000bd00 con 0x7f07ec073f30 2024-04-26T23:10:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.975+0000 7f07e9400640 1 --2- 172.21.15.40:0/3799891698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07d0003fb0 0x7f07d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.975+0000 7f07ea800640 1 --2- 172.21.15.40:0/3799891698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07d0004d40 0x7f07d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.975+0000 7f07e9400640 1 -- 172.21.15.40:0/3799891698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07ec073f30 msgr2=0x7f07d00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.975+0000 7f07e9400640 1 --2- 172.21.15.40:0/3799891698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07ec073f30 0x7f07d00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.975+0000 7f07e9400640 1 -- 172.21.15.40:0/3799891698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07d0004d40 msgr2=0x7f07d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.975+0000 7f07e9400640 1 --2- 172.21.15.40:0/3799891698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07d0004d40 0x7f07d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.975+0000 7f07e9400640 1 -- 172.21.15.40:0/3799891698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d0003520 con 0x7f07d0003fb0 2024-04-26T23:10:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.975+0000 7f07ea800640 1 --2- 172.21.15.40:0/3799891698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07d0004d40 0x7f07d00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.976+0000 7f07e9400640 1 --2- 172.21.15.40:0/3799891698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07d0003fb0 0x7f07d0004800 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f07ec073a10 tx=0x7f07d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2927836e9dae36f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.976+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8036070 con 0x7f07d0003fb0 2024-04-26T23:10:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.977+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d8031400 con 0x7f07d0003fb0 2024-04-26T23:10:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.977+0000 7f07e8a00640 1 -- 172.21.15.40:0/3799891698 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8009690 con 0x7f07d0003fb0 2024-04-26T23:10:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.977+0000 7f07f0c00640 1 -- 172.21.15.40:0/3799891698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07d0003fb0 msgr2=0x7f07d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.977+0000 7f07f0c00640 1 --2- 172.21.15.40:0/3799891698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07d0003fb0 0x7f07d0004800 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f07ec073a10 tx=0x7f07d0005830 comp rx=0 tx=0).stop 2024-04-26T23:10:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.977+0000 7f07ea800640 1 -- 172.21.15.40:0/3799891698 reap_dead start 2024-04-26T23:10:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.977+0000 7f07f0c00640 1 -- 172.21.15.40:0/3799891698 shutdown_connections 2024-04-26T23:10:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.977+0000 7f07f0c00640 1 -- 172.21.15.40:0/3799891698 >> 172.21.15.40:0/3799891698 conn(0x7f07ec0d0be0 msgr2=0x7f07ec0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.977+0000 7f07f0c00640 1 -- 172.21.15.40:0/3799891698 shutdown_connections 2024-04-26T23:10:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.977+0000 7f07f0c00640 1 -- 172.21.15.40:0/3799891698 wait complete. 2024-04-26T23:10:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.979+0000 7f07f0c00640 1 Processor -- start 2024-04-26T23:10:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.979+0000 7f07f0c00640 1 -- start start 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.980+0000 7f07f0c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07d0003fb0 0x7f07ec166530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.980+0000 7f07f0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ec163c60 0x7f07ec1640e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.980+0000 7f07f0c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07ec0c2e00 0x7f07ec0c3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.980+0000 7f07f0c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec0ce7b0 con 0x7f07ec163c60 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.980+0000 7f07f0c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec0cf430 con 0x7f07ec0c2e00 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.980+0000 7f07f0c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f07ec0ce930 con 0x7f07d0003fb0 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.981+0000 7f07e9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ec163c60 0x7f07ec1640e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.981+0000 7f07e9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ec163c60 0x7f07ec1640e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45994/0 (socket says 172.21.15.40:45994) 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.981+0000 7f07e9400640 1 -- 172.21.15.40:0/3943441747 learned_addr learned my addr 172.21.15.40:0/3943441747 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.981+0000 7f07ea800640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07ec0c2e00 0x7f07ec0c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.981+0000 7f07e9400640 1 -- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07d0003fb0 msgr2=0x7f07ec166530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.981+0000 7f07e9e00640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07d0003fb0 0x7f07ec166530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.981+0000 7f07e9400640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07d0003fb0 0x7f07ec166530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.981+0000 7f07e9400640 1 -- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07ec0c2e00 msgr2=0x7f07ec0c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:01.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.981+0000 7f07e9400640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07ec0c2e00 0x7f07ec0c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:01.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.981+0000 7f07e9400640 1 -- 172.21.15.40:0/3943441747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ec1647d0 con 0x7f07ec163c60 2024-04-26T23:10:01.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.981+0000 7f07e9400640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ec163c60 0x7f07ec1640e0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f07d8034040 tx=0x7f07d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:01.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.982+0000 7f07e2000640 1 -- 172.21.15.40:0/3943441747 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8040020 con 0x7f07ec163c60 2024-04-26T23:10:01.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.982+0000 7f07e2000640 1 -- 172.21.15.40:0/3943441747 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d803b680 con 0x7f07ec163c60 2024-04-26T23:10:01.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.982+0000 7f07f0c00640 1 -- 172.21.15.40:0/3943441747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07ec164ab0 con 0x7f07ec163c60 2024-04-26T23:10:01.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.982+0000 7f07e2000640 1 -- 172.21.15.40:0/3943441747 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8036040 con 0x7f07ec163c60 2024-04-26T23:10:01.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.982+0000 7f07f0c00640 1 -- 172.21.15.40:0/3943441747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07ec165070 con 0x7f07ec163c60 2024-04-26T23:10:01.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.983+0000 7f07f0c00640 1 -- 172.21.15.40:0/3943441747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07ec0694f0 con 0x7f07ec163c60 2024-04-26T23:10:01.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.984+0000 7f07e2000640 1 -- 172.21.15.40:0/3943441747 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f07d8043020 con 0x7f07ec163c60 2024-04-26T23:10:01.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.986+0000 7f07e2000640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f07cc06c8f0 0x7f07cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:01.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.987+0000 7f07e2000640 1 -- 172.21.15.40:0/3943441747 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f07d80ac4a0 con 0x7f07ec163c60 2024-04-26T23:10:01.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.987+0000 7f07e9e00640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f07cc06c8f0 0x7f07cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:01.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.988+0000 7f07e9e00640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f07cc06c8f0 0x7f07cc06edb0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f07dc001570 tx=0x7f07dc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:01.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:01.991+0000 7f07e2000640 1 -- 172.21.15.40:0/3943441747 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f07d804a030 con 0x7f07ec163c60 2024-04-26T23:10:02.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.197+0000 7f07f0c00640 1 -- 172.21.15.40:0/3943441747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f07ec072330 con 0x7f07ec163c60 2024-04-26T23:10:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.200+0000 7f07e2000640 1 -- 172.21.15.40:0/3943441747 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+16898 (secure 0 0 0) 0x7f07d807a7b0 con 0x7f07ec163c60 2024-04-26T23:10:02.201 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:02.201 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":105,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:09:49.407703+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:09:03.955148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-26T23:09:05.070627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.2a","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:10:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.203+0000 7f07f0c00640 1 -- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f07cc06c8f0 msgr2=0x7f07cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.203+0000 7f07f0c00640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f07cc06c8f0 0x7f07cc06edb0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f07dc001570 tx=0x7f07dc009040 comp rx=0 tx=0).stop 2024-04-26T23:10:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.203+0000 7f07f0c00640 1 -- 172.21.15.40:0/3943441747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ec163c60 msgr2=0x7f07ec1640e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.203+0000 7f07f0c00640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ec163c60 0x7f07ec1640e0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f07d8034040 tx=0x7f07d8004730 comp rx=0 tx=0).stop 2024-04-26T23:10:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.204+0000 7f07f0c00640 1 -- 172.21.15.40:0/3943441747 shutdown_connections 2024-04-26T23:10:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.204+0000 7f07f0c00640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f07cc06c8f0 0x7f07cc06edb0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.204+0000 7f07f0c00640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07ec0c2e00 0x7f07ec0c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.204+0000 7f07f0c00640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ec163c60 0x7f07ec1640e0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.204+0000 7f07f0c00640 1 --2- 172.21.15.40:0/3943441747 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f07d0003fb0 0x7f07ec166530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.204+0000 7f07f0c00640 1 -- 172.21.15.40:0/3943441747 >> 172.21.15.40:0/3943441747 conn(0x7f07ec0d0be0 msgr2=0x7f07ec163250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.204+0000 7f07f0c00640 1 -- 172.21.15.40:0/3943441747 shutdown_connections 2024-04-26T23:10:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.204+0000 7f07f0c00640 1 -- 172.21.15.40:0/3943441747 wait complete. 2024-04-26T23:10:02.223 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:09:03.955148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:09:05.070627+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:10:02.223 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:10:02.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.437+0000 7ff69ac00640 1 Processor -- start 2024-04-26T23:10:02.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.437+0000 7ff69ac00640 1 -- start start 2024-04-26T23:10:02.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.438+0000 7ff69ac00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff6940cfaf0 0x7ff694073a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:02.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.438+0000 7ff69ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff69406dd40 0x7ff694072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:02.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.438+0000 7ff69ac00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff694161880 con 0x7ff694160a50 2024-04-26T23:10:02.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.438+0000 7ff69ac00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff694071ce0 con 0x7ff6940cfaf0 2024-04-26T23:10:02.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.438+0000 7ff69ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff6940722a0 con 0x7ff69406dd40 2024-04-26T23:10:02.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.439+0000 7ff69a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff69406dd40 0x7ff694072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:02.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.439+0000 7ff698e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff6940cfaf0 0x7ff694073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:02.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.439+0000 7ff69a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff69406dd40 0x7ff694072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:38872/0 (socket says 172.21.15.40:38872) 2024-04-26T23:10:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.439+0000 7ff69a200640 1 -- 172.21.15.40:0/1276172837 learned_addr learned my addr 172.21.15.40:0/1276172837 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.439+0000 7ff699800640 1 --1- 172.21.15.40:0/1276172837 >> v1:172.21.15.103:6789/0 conn(0x7ff694160a50 0x7ff694160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:36618/0 (socket says 172.21.15.40:36618) 2024-04-26T23:10:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.440+0000 7ff698e00640 1 -- 172.21.15.40:0/1276172837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff69406dd40 msgr2=0x7ff694072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.440+0000 7ff698e00640 1 --2- 172.21.15.40:0/1276172837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff69406dd40 0x7ff694072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.440+0000 7ff698e00640 1 -- 172.21.15.40:0/1276172837 >> v1:172.21.15.103:6789/0 conn(0x7ff694160a50 legacy=0x7ff694160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.440+0000 7ff698e00640 1 -- 172.21.15.40:0/1276172837 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6940cff80 con 0x7ff6940cfaf0 2024-04-26T23:10:02.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.440+0000 7ff69a200640 1 --2- 172.21.15.40:0/1276172837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff69406dd40 0x7ff694072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:10:02.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.441+0000 7ff698e00640 1 --2- 172.21.15.40:0/1276172837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6940cfaf0 0x7ff694073a50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff6800097a0 tx=0x7ff68002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d55144683f84f9e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:02.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.442+0000 7ff693e00640 1 -- 172.21.15.40:0/1276172837 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff680037070 con 0x7ff6940cfaf0 2024-04-26T23:10:02.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.442+0000 7ff693e00640 1 -- 172.21.15.40:0/1276172837 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff680032440 con 0x7ff6940cfaf0 2024-04-26T23:10:02.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.442+0000 7ff693e00640 1 -- 172.21.15.40:0/1276172837 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff680032810 con 0x7ff6940cfaf0 2024-04-26T23:10:02.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.443+0000 7ff69ac00640 1 -- 172.21.15.40:0/1276172837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6940cfaf0 msgr2=0x7ff694073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:02.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.443+0000 7ff69ac00640 1 --2- 172.21.15.40:0/1276172837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6940cfaf0 0x7ff694073a50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff6800097a0 tx=0x7ff68002a180 comp rx=0 tx=0).stop 2024-04-26T23:10:02.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.443+0000 7ff69ac00640 1 -- 172.21.15.40:0/1276172837 shutdown_connections 2024-04-26T23:10:02.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.443+0000 7ff69ac00640 1 --2- 172.21.15.40:0/1276172837 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff69406dd40 0x7ff694072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.443+0000 7ff69ac00640 1 --2- 172.21.15.40:0/1276172837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6940cfaf0 0x7ff694073a50 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.443+0000 7ff69ac00640 1 -- 172.21.15.40:0/1276172837 >> 172.21.15.40:0/1276172837 conn(0x7ff6940b2720 msgr2=0x7ff6940b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:02.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.443+0000 7ff69ac00640 1 -- 172.21.15.40:0/1276172837 shutdown_connections 2024-04-26T23:10:02.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.443+0000 7ff69ac00640 1 -- 172.21.15.40:0/1276172837 wait complete. 2024-04-26T23:10:02.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.445+0000 7ff69ac00640 1 Processor -- start 2024-04-26T23:10:02.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.445+0000 7ff69ac00640 1 -- start start 2024-04-26T23:10:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.447+0000 7ff69ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff69406dd40 0x7ff694159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.447+0000 7ff69ac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff694160a50 0x7ff69413cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.447+0000 7ff69ac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff69415a1b0 0x7ff69413d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.447+0000 7ff69ac00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff694071f30 con 0x7ff69415a1b0 2024-04-26T23:10:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.447+0000 7ff69ac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff694073f90 con 0x7ff694160a50 2024-04-26T23:10:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.447+0000 7ff69ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff69406e1f0 con 0x7ff69406dd40 2024-04-26T23:10:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.447+0000 7ff69a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff69415a1b0 0x7ff69413d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.447+0000 7ff69a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff69415a1b0 0x7ff69413d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45996/0 (socket says 172.21.15.40:45996) 2024-04-26T23:10:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.447+0000 7ff69a200640 1 -- 172.21.15.40:0/358037027 learned_addr learned my addr 172.21.15.40:0/358037027 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.447+0000 7ff699800640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff69406dd40 0x7ff694159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.447+0000 7ff698e00640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff694160a50 0x7ff69413cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.448+0000 7ff69a200640 1 -- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff69406dd40 msgr2=0x7ff694159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:02.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.448+0000 7ff69a200640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff69406dd40 0x7ff694159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.448+0000 7ff69a200640 1 -- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff694160a50 msgr2=0x7ff69413cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:02.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.448+0000 7ff69a200640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff694160a50 0x7ff69413cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.448+0000 7ff69a200640 1 -- 172.21.15.40:0/358037027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff69415fd90 con 0x7ff69415a1b0 2024-04-26T23:10:02.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.448+0000 7ff69a200640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff69415a1b0 0x7ff69413d140 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00efc0 tx=0x7ff68c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:02.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.448+0000 7ff691600640 1 -- 172.21.15.40:0/358037027 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c00cbe0 con 0x7ff69415a1b0 2024-04-26T23:10:02.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.448+0000 7ff691600640 1 -- 172.21.15.40:0/358037027 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff68c00cd80 con 0x7ff69415a1b0 2024-04-26T23:10:02.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.448+0000 7ff691600640 1 -- 172.21.15.40:0/358037027 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c003bb0 con 0x7ff69415a1b0 2024-04-26T23:10:02.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.448+0000 7ff69ac00640 1 -- 172.21.15.40:0/358037027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6940b3530 con 0x7ff69415a1b0 2024-04-26T23:10:02.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.449+0000 7ff69ac00640 1 -- 172.21.15.40:0/358037027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6940b3af0 con 0x7ff69415a1b0 2024-04-26T23:10:02.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.450+0000 7ff69ac00640 1 -- 172.21.15.40:0/358037027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff694069370 con 0x7ff69415a1b0 2024-04-26T23:10:02.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.451+0000 7ff691600640 1 -- 172.21.15.40:0/358037027 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff68c01b030 con 0x7ff69415a1b0 2024-04-26T23:10:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.452+0000 7ff691600640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff67406c8f0 0x7ff67406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:02.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.452+0000 7ff691600640 1 -- 172.21.15.40:0/358037027 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7ff68c08c120 con 0x7ff69415a1b0 2024-04-26T23:10:02.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.453+0000 7ff699800640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff67406c8f0 0x7ff67406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:02.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.454+0000 7ff699800640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff67406c8f0 0x7ff67406edb0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff688000f60 tx=0x7ff688007cc0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:02.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.457+0000 7ff691600640 1 -- 172.21.15.40:0/358037027 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff68c022020 con 0x7ff69415a1b0 2024-04-26T23:10:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.669+0000 7ff69ac00640 1 -- 172.21.15.40:0/358037027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff69406fa60 con 0x7ff69415a1b0 2024-04-26T23:10:02.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.670+0000 7ff691600640 1 -- 172.21.15.40:0/358037027 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) ==== 93+0+10 (secure 0 0 0) 0x7ff68c010040 con 0x7ff69415a1b0 2024-04-26T23:10:02.670 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:10:02.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.673+0000 7ff69ac00640 1 -- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff67406c8f0 msgr2=0x7ff67406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:02.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.673+0000 7ff69ac00640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff67406c8f0 0x7ff67406edb0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff688000f60 tx=0x7ff688007cc0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.673+0000 7ff69ac00640 1 -- 172.21.15.40:0/358037027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff69415a1b0 msgr2=0x7ff69413d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:02.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.673+0000 7ff69ac00640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff69415a1b0 0x7ff69413d140 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00efc0 tx=0x7ff68c00c490 comp rx=0 tx=0).stop 2024-04-26T23:10:02.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.674+0000 7ff69ac00640 1 -- 172.21.15.40:0/358037027 shutdown_connections 2024-04-26T23:10:02.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.674+0000 7ff69ac00640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff67406c8f0 0x7ff67406edb0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.674+0000 7ff69ac00640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff69415a1b0 0x7ff69413d140 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.674+0000 7ff69ac00640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff694160a50 0x7ff69413cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.674+0000 7ff69ac00640 1 --2- 172.21.15.40:0/358037027 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff69406dd40 0x7ff694159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.674+0000 7ff69ac00640 1 -- 172.21.15.40:0/358037027 >> 172.21.15.40:0/358037027 conn(0x7ff6940b2720 msgr2=0x7ff6940d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:02.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.674+0000 7ff69ac00640 1 -- 172.21.15.40:0/358037027 shutdown_connections 2024-04-26T23:10:02.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.674+0000 7ff69ac00640 1 -- 172.21.15.40:0/358037027 wait complete. 2024-04-26T23:10:02.692 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:10:02.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.898+0000 7f2e2f600640 1 Processor -- start 2024-04-26T23:10:02.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.898+0000 7f2e2f600640 1 -- start start 2024-04-26T23:10:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.899+0000 7f2e2f600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2e28156460 0x7f2e280cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.899+0000 7f2e2f600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e280cba10 0x7f2e2806f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.899+0000 7f2e2f600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e28156840 con 0x7f2e2806fcd0 2024-04-26T23:10:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.899+0000 7f2e2f600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2e280cbf20 con 0x7f2e28156460 2024-04-26T23:10:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.899+0000 7f2e2f600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2e280cc200 con 0x7f2e280cba10 2024-04-26T23:10:02.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.900+0000 7f2e27e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e280cba10 0x7f2e2806f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:02.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.900+0000 7f2e2d600640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2e2806fcd0 0x7f2e2815f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:36630/0 (socket says 172.21.15.40:36630) 2024-04-26T23:10:02.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.900+0000 7f2e2d600640 1 -- 172.21.15.40:0/1881867774 learned_addr learned my addr 172.21.15.40:0/1881867774 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:02.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.900+0000 7f2e2cc00640 1 --2- 172.21.15.40:0/1881867774 >> v2:172.21.15.103:3300/0 conn(0x7f2e28156460 0x7f2e280cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:02.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.901+0000 7f2e27400640 1 -- 172.21.15.40:0/1881867774 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1271681858 0 0) 0x7f2e28156840 con 0x7f2e2806fcd0 2024-04-26T23:10:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.901+0000 7f2e27400640 1 -- 172.21.15.40:0/1881867774 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e0c003520 con 0x7f2e2806fcd0 2024-04-26T23:10:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.901+0000 7f2e27e00640 1 -- 172.21.15.40:0/1881867774 >> v2:172.21.15.103:3300/0 conn(0x7f2e28156460 msgr2=0x7f2e280cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.901+0000 7f2e27e00640 1 --2- 172.21.15.40:0/1881867774 >> v2:172.21.15.103:3300/0 conn(0x7f2e28156460 0x7f2e280cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.901+0000 7f2e27e00640 1 -- 172.21.15.40:0/1881867774 >> v1:172.21.15.103:6789/0 conn(0x7f2e2806fcd0 legacy=0x7f2e2815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.901+0000 7f2e27e00640 1 -- 172.21.15.40:0/1881867774 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e280cc4e0 con 0x7f2e280cba10 2024-04-26T23:10:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.901+0000 7f2e2cc00640 1 --2- 172.21.15.40:0/1881867774 >> v2:172.21.15.103:3300/0 conn(0x7f2e28156460 0x7f2e280cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:10:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.902+0000 7f2e27e00640 1 --2- 172.21.15.40:0/1881867774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e280cba10 0x7f2e2806f790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2e100097a0 tx=0x7f2e1002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac5176b5ff4e308e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.903+0000 7f2e27400640 1 -- 172.21.15.40:0/1881867774 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e10037070 con 0x7f2e280cba10 2024-04-26T23:10:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.903+0000 7f2e27400640 1 -- 172.21.15.40:0/1881867774 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e10032440 con 0x7f2e280cba10 2024-04-26T23:10:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.903+0000 7f2e27400640 1 -- 172.21.15.40:0/1881867774 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e100327f0 con 0x7f2e280cba10 2024-04-26T23:10:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.903+0000 7f2e2f600640 1 -- 172.21.15.40:0/1881867774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e280cba10 msgr2=0x7f2e2806f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.903+0000 7f2e2f600640 1 --2- 172.21.15.40:0/1881867774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e280cba10 0x7f2e2806f790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2e100097a0 tx=0x7f2e1002a180 comp rx=0 tx=0).stop 2024-04-26T23:10:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.904+0000 7f2e2f600640 1 -- 172.21.15.40:0/1881867774 shutdown_connections 2024-04-26T23:10:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.904+0000 7f2e2f600640 1 --2- 172.21.15.40:0/1881867774 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e280cba10 0x7f2e2806f790 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.904+0000 7f2e2f600640 1 --2- 172.21.15.40:0/1881867774 >> v2:172.21.15.103:3300/0 conn(0x7f2e28156460 0x7f2e280cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.904+0000 7f2e2f600640 1 -- 172.21.15.40:0/1881867774 >> 172.21.15.40:0/1881867774 conn(0x7f2e280d0be0 msgr2=0x7f2e280d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.904+0000 7f2e2f600640 1 -- 172.21.15.40:0/1881867774 shutdown_connections 2024-04-26T23:10:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.904+0000 7f2e2f600640 1 -- 172.21.15.40:0/1881867774 wait complete. 2024-04-26T23:10:02.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.905+0000 7f2e2f600640 1 Processor -- start 2024-04-26T23:10:02.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.906+0000 7f2e2f600640 1 -- start start 2024-04-26T23:10:02.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.906+0000 7f2e2f600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2806fcd0 0x7f2e280c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.906+0000 7f2e2f600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e280cba10 0x7f2e28155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.906+0000 7f2e2f600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e28156460 0x7f2e28151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.906+0000 7f2e2f600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2806e260 con 0x7f2e2806fcd0 2024-04-26T23:10:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.906+0000 7f2e2f600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2e28072700 con 0x7f2e28156460 2024-04-26T23:10:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.907+0000 7f2e2f600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2e2806e3e0 con 0x7f2e280cba10 2024-04-26T23:10:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.907+0000 7f2e2cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2806fcd0 0x7f2e280c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.907+0000 7f2e2cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2806fcd0 0x7f2e280c62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45998/0 (socket says 172.21.15.40:45998) 2024-04-26T23:10:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.907+0000 7f2e2cc00640 1 -- 172.21.15.40:0/360966327 learned_addr learned my addr 172.21.15.40:0/360966327 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.907+0000 7f2e2d600640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e28156460 0x7f2e28151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.907+0000 7f2e27e00640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e280cba10 0x7f2e28155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.907+0000 7f2e2cc00640 1 -- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e280cba10 msgr2=0x7f2e28155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.907+0000 7f2e2cc00640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e280cba10 0x7f2e28155500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.907+0000 7f2e2cc00640 1 -- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e28156460 msgr2=0x7f2e28151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:02.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.907+0000 7f2e2cc00640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e28156460 0x7f2e28151d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:02.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.907+0000 7f2e2cc00640 1 -- 172.21.15.40:0/360966327 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e280c6cb0 con 0x7f2e2806fcd0 2024-04-26T23:10:02.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.907+0000 7f2e2d600640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e28156460 0x7f2e28151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:02.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.908+0000 7f2e2cc00640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2806fcd0 0x7f2e280c62c0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800efc0 tx=0x7f2e1800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:02.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.908+0000 7f2e24c00640 1 -- 172.21.15.40:0/360966327 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18018020 con 0x7f2e2806fcd0 2024-04-26T23:10:02.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.908+0000 7f2e24c00640 1 -- 172.21.15.40:0/360966327 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e18014680 con 0x7f2e2806fcd0 2024-04-26T23:10:02.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.908+0000 7f2e24c00640 1 -- 172.21.15.40:0/360966327 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18014a50 con 0x7f2e2806fcd0 2024-04-26T23:10:02.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.908+0000 7f2e2f600640 1 -- 172.21.15.40:0/360966327 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e280c69d0 con 0x7f2e2806fcd0 2024-04-26T23:10:02.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.908+0000 7f2e2f600640 1 -- 172.21.15.40:0/360966327 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e280c8030 con 0x7f2e2806fcd0 2024-04-26T23:10:02.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.910+0000 7f2e2f600640 1 -- 172.21.15.40:0/360966327 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e280694f0 con 0x7f2e2806fcd0 2024-04-26T23:10:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.911+0000 7f2e24c00640 1 -- 172.21.15.40:0/360966327 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2e18005d10 con 0x7f2e2806fcd0 2024-04-26T23:10:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.912+0000 7f2e24c00640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2e0406c8f0 0x7f2e0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.912+0000 7f2e24c00640 1 -- 172.21.15.40:0/360966327 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f2e1808cbb0 con 0x7f2e2806fcd0 2024-04-26T23:10:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.913+0000 7f2e27e00640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2e0406c8f0 0x7f2e0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.914+0000 7f2e27e00640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2e0406c8f0 0x7f2e0406edb0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2e10034000 tx=0x7f2e1002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:02.916+0000 7f2e24c00640 1 -- 172.21.15.40:0/360966327 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2e18022030 con 0x7f2e2806fcd0 2024-04-26T23:10:03.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.121+0000 7f2e2f600640 1 -- 172.21.15.40:0/360966327 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2e2806de80 con 0x7f2e2806fcd0 2024-04-26T23:10:03.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.121+0000 7f2e24c00640 1 -- 172.21.15.40:0/360966327 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v105) ==== 104+0+11 (secure 0 0 0) 0x7f2e18010040 con 0x7f2e2806fcd0 2024-04-26T23:10:03.122 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:10:03.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.124+0000 7f2e2f600640 1 -- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2e0406c8f0 msgr2=0x7f2e0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:03.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.124+0000 7f2e2f600640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2e0406c8f0 0x7f2e0406edb0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2e10034000 tx=0x7f2e1002e000 comp rx=0 tx=0).stop 2024-04-26T23:10:03.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.124+0000 7f2e2f600640 1 -- 172.21.15.40:0/360966327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2806fcd0 msgr2=0x7f2e280c62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:03.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.124+0000 7f2e2f600640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2806fcd0 0x7f2e280c62c0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800efc0 tx=0x7f2e1800c490 comp rx=0 tx=0).stop 2024-04-26T23:10:03.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.124+0000 7f2e2f600640 1 -- 172.21.15.40:0/360966327 shutdown_connections 2024-04-26T23:10:03.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.124+0000 7f2e2f600640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2e0406c8f0 0x7f2e0406edb0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.124+0000 7f2e2f600640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e28156460 0x7f2e28151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.124+0000 7f2e2f600640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e280cba10 0x7f2e28155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.124+0000 7f2e2f600640 1 --2- 172.21.15.40:0/360966327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2806fcd0 0x7f2e280c62c0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.124+0000 7f2e2f600640 1 -- 172.21.15.40:0/360966327 >> 172.21.15.40:0/360966327 conn(0x7f2e280d0be0 msgr2=0x7f2e28072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:03.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.124+0000 7f2e2f600640 1 -- 172.21.15.40:0/360966327 shutdown_connections 2024-04-26T23:10:03.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.124+0000 7f2e2f600640 1 -- 172.21.15.40:0/360966327 wait complete. 2024-04-26T23:10:03.144 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:10:03.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.360+0000 7f4a8fe00640 1 Processor -- start 2024-04-26T23:10:03.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.360+0000 7f4a8fe00640 1 -- start start 2024-04-26T23:10:03.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.362+0000 7f4a8fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a90073600 0x7f4a90072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:03.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.362+0000 7f4a8fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a900739e0 0x7f4a9015f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.362+0000 7f4a8fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4a9015fcb0 0x7f4a900ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.362+0000 7f4a8fe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f4a90073220 con 0x7f4a900739e0 2024-04-26T23:10:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.362+0000 7f4a8fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4a90072850 con 0x7f4a9015fcb0 2024-04-26T23:10:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.362+0000 7f4a8fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4a901604e0 con 0x7f4a90073600 2024-04-26T23:10:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.362+0000 7f4a8e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a900739e0 0x7f4a9015f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.362+0000 7f4a8e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a900739e0 0x7f4a9015f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46004/0 (socket says 172.21.15.40:46004) 2024-04-26T23:10:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.362+0000 7f4a8e000640 1 -- 172.21.15.40:0/157170857 learned_addr learned my addr 172.21.15.40:0/157170857 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.363+0000 7f4a8ea00640 1 --2- 172.21.15.40:0/157170857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a90073600 0x7f4a90072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.363+0000 7f4a8f400640 1 --2- 172.21.15.40:0/157170857 >> v2:172.21.15.103:3300/0 conn(0x7f4a9015fcb0 0x7f4a900ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.363+0000 7f4a8e000640 1 -- 172.21.15.40:0/157170857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a90073600 msgr2=0x7f4a90072ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.363+0000 7f4a8e000640 1 --2- 172.21.15.40:0/157170857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a90073600 0x7f4a90072ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.363+0000 7f4a8e000640 1 -- 172.21.15.40:0/157170857 >> v2:172.21.15.103:3300/0 conn(0x7f4a9015fcb0 msgr2=0x7f4a900ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.363+0000 7f4a8e000640 1 --2- 172.21.15.40:0/157170857 >> v2:172.21.15.103:3300/0 conn(0x7f4a9015fcb0 0x7f4a900ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.363+0000 7f4a8e000640 1 -- 172.21.15.40:0/157170857 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a90072200 con 0x7f4a900739e0 2024-04-26T23:10:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.363+0000 7f4a8f400640 1 --2- 172.21.15.40:0/157170857 >> v2:172.21.15.103:3300/0 conn(0x7f4a9015fcb0 0x7f4a900ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.363+0000 7f4a8ea00640 1 --2- 172.21.15.40:0/157170857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a90073600 0x7f4a90072ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.364+0000 7f4a8e000640 1 --2- 172.21.15.40:0/157170857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a900739e0 0x7f4a9015f770 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f4a900b0640 tx=0x7f4a7802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607cf40fd5042632 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.364+0000 7f4a8d600640 1 -- 172.21.15.40:0/157170857 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a78037070 con 0x7f4a900739e0 2024-04-26T23:10:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.365+0000 7f4a8d600640 1 -- 172.21.15.40:0/157170857 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a78032440 con 0x7f4a900739e0 2024-04-26T23:10:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.365+0000 7f4a8d600640 1 -- 172.21.15.40:0/157170857 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a78032810 con 0x7f4a900739e0 2024-04-26T23:10:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.365+0000 7f4a8fe00640 1 -- 172.21.15.40:0/157170857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a900739e0 msgr2=0x7f4a9015f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.365+0000 7f4a8fe00640 1 --2- 172.21.15.40:0/157170857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a900739e0 0x7f4a9015f770 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f4a900b0640 tx=0x7f4a7802a1e0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.365+0000 7f4a8fe00640 1 -- 172.21.15.40:0/157170857 shutdown_connections 2024-04-26T23:10:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.365+0000 7f4a8fe00640 1 --2- 172.21.15.40:0/157170857 >> v2:172.21.15.103:3300/0 conn(0x7f4a9015fcb0 0x7f4a900ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.365+0000 7f4a8fe00640 1 --2- 172.21.15.40:0/157170857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a900739e0 0x7f4a9015f770 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.365+0000 7f4a8fe00640 1 --2- 172.21.15.40:0/157170857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a90073600 0x7f4a90072ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.365+0000 7f4a8fe00640 1 -- 172.21.15.40:0/157170857 >> 172.21.15.40:0/157170857 conn(0x7f4a900b2720 msgr2=0x7f4a900b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.365+0000 7f4a8fe00640 1 -- 172.21.15.40:0/157170857 shutdown_connections 2024-04-26T23:10:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.365+0000 7f4a8fe00640 1 -- 172.21.15.40:0/157170857 wait complete. 2024-04-26T23:10:03.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.367+0000 7f4a8fe00640 1 Processor -- start 2024-04-26T23:10:03.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.367+0000 7f4a8fe00640 1 -- start start 2024-04-26T23:10:03.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.368+0000 7f4a8fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a90073600 0x7f4a901482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:03.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.368+0000 7f4a8fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a900739e0 0x7f4a90158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:03.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.368+0000 7f4a8fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a9015fcb0 0x7f4a9015a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:03.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.368+0000 7f4a8fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4a90071e20 con 0x7f4a900739e0 2024-04-26T23:10:03.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.368+0000 7f4a8fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4a90073e60 con 0x7f4a9015fcb0 2024-04-26T23:10:03.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.368+0000 7f4a8fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4a90071fa0 con 0x7f4a90073600 2024-04-26T23:10:03.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.369+0000 7f4a8e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a900739e0 0x7f4a90158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:03.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.369+0000 7f4a8e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a900739e0 0x7f4a90158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46006/0 (socket says 172.21.15.40:46006) 2024-04-26T23:10:03.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.369+0000 7f4a8e000640 1 -- 172.21.15.40:0/3193525673 learned_addr learned my addr 172.21.15.40:0/3193525673 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:03.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.369+0000 7f4a8ea00640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a90073600 0x7f4a901482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:03.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.369+0000 7f4a8e000640 1 -- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a90073600 msgr2=0x7f4a901482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:03.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.369+0000 7f4a8e000640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a90073600 0x7f4a901482e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.369+0000 7f4a8e000640 1 -- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a9015fcb0 msgr2=0x7f4a9015a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.369+0000 7f4a8f400640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a9015fcb0 0x7f4a9015a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.369+0000 7f4a8e000640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a9015fcb0 0x7f4a9015a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.369+0000 7f4a8e000640 1 -- 172.21.15.40:0/3193525673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a900cdb00 con 0x7f4a900739e0 2024-04-26T23:10:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.369+0000 7f4a8e000640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a900739e0 0x7f4a90158070 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f4a7802a220 tx=0x7f4a780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.370+0000 7f4a87400640 1 -- 172.21.15.40:0/3193525673 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a78037040 con 0x7f4a900739e0 2024-04-26T23:10:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.370+0000 7f4a87400640 1 -- 172.21.15.40:0/3193525673 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a78042070 con 0x7f4a900739e0 2024-04-26T23:10:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.370+0000 7f4a87400640 1 -- 172.21.15.40:0/3193525673 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a78030bb0 con 0x7f4a900739e0 2024-04-26T23:10:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.370+0000 7f4a8fe00640 1 -- 172.21.15.40:0/3193525673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a90156570 con 0x7f4a900739e0 2024-04-26T23:10:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.370+0000 7f4a8fe00640 1 -- 172.21.15.40:0/3193525673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a90156b30 con 0x7f4a900739e0 2024-04-26T23:10:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.371+0000 7f4a8fe00640 1 -- 172.21.15.40:0/3193525673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a90069370 con 0x7f4a900739e0 2024-04-26T23:10:03.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.372+0000 7f4a87400640 1 -- 172.21.15.40:0/3193525673 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4a78047460 con 0x7f4a900739e0 2024-04-26T23:10:03.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.373+0000 7f4a87400640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a6c06c920 0x7f4a6c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:03.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.373+0000 7f4a87400640 1 -- 172.21.15.40:0/3193525673 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f4a780ac930 con 0x7f4a900739e0 2024-04-26T23:10:03.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.374+0000 7f4a8ea00640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a6c06c920 0x7f4a6c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:03.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.375+0000 7f4a8ea00640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a6c06c920 0x7f4a6c06ede0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4a90161ec0 tx=0x7f4a80008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:03.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.377+0000 7f4a87400640 1 -- 172.21.15.40:0/3193525673 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f4a7807aba0 con 0x7f4a900739e0 2024-04-26T23:10:03.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.582+0000 7f4a8fe00640 1 -- 172.21.15.40:0/3193525673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4a900b3480 con 0x7f4a900739e0 2024-04-26T23:10:03.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.582+0000 7f4a87400640 1 -- 172.21.15.40:0/3193525673 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v105) ==== 100+0+11 (secure 0 0 0) 0x7f4a7807a4a0 con 0x7f4a900739e0 2024-04-26T23:10:03.583 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:10:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.585+0000 7f4a8fe00640 1 -- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a6c06c920 msgr2=0x7f4a6c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.585+0000 7f4a8fe00640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a6c06c920 0x7f4a6c06ede0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4a90161ec0 tx=0x7f4a80008040 comp rx=0 tx=0).stop 2024-04-26T23:10:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.585+0000 7f4a8fe00640 1 -- 172.21.15.40:0/3193525673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a900739e0 msgr2=0x7f4a90158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.585+0000 7f4a8fe00640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a900739e0 0x7f4a90158070 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f4a7802a220 tx=0x7f4a780046c0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.585+0000 7f4a8fe00640 1 -- 172.21.15.40:0/3193525673 shutdown_connections 2024-04-26T23:10:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.585+0000 7f4a8fe00640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f4a6c06c920 0x7f4a6c06ede0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.585+0000 7f4a8fe00640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a9015fcb0 0x7f4a9015a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.585+0000 7f4a8fe00640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a900739e0 0x7f4a90158070 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.585+0000 7f4a8fe00640 1 --2- 172.21.15.40:0/3193525673 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4a90073600 0x7f4a901482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.585+0000 7f4a8fe00640 1 -- 172.21.15.40:0/3193525673 >> 172.21.15.40:0/3193525673 conn(0x7f4a900b2720 msgr2=0x7f4a900b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.586+0000 7f4a8fe00640 1 -- 172.21.15.40:0/3193525673 shutdown_connections 2024-04-26T23:10:03.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.586+0000 7f4a8fe00640 1 -- 172.21.15.40:0/3193525673 wait complete. 2024-04-26T23:10:03.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:10:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.826+0000 7f5d6fc00640 1 Processor -- start 2024-04-26T23:10:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.826+0000 7f5d6fc00640 1 -- start start 2024-04-26T23:10:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.827+0000 7f5d6fc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d68071e60 0x7f5d6806f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:03.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.827+0000 7f5d6fc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5d6806fe20 0x7f5d68072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:03.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.827+0000 7f5d6fc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d6815d7c0 con 0x7f5d68072780 2024-04-26T23:10:03.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.827+0000 7f5d6fc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5d6815daa0 con 0x7f5d6806fe20 2024-04-26T23:10:03.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.827+0000 7f5d6fc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5d6815dd80 con 0x7f5d68071e60 2024-04-26T23:10:03.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.828+0000 7f5d6d200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d68071e60 0x7f5d6806f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:03.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.828+0000 7f5d6dc00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f5d68072780 0x7f5d680cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:36646/0 (socket says 172.21.15.40:36646) 2024-04-26T23:10:03.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.828+0000 7f5d6dc00640 1 -- 172.21.15.40:0/4114132648 learned_addr learned my addr 172.21.15.40:0/4114132648 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:03.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.828+0000 7f5d67e00640 1 --2- 172.21.15.40:0/4114132648 >> v2:172.21.15.103:3300/0 conn(0x7f5d6806fe20 0x7f5d68072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:03.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.829+0000 7f5d67e00640 1 -- 172.21.15.40:0/4114132648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d68071e60 msgr2=0x7f5d6806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:03.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.829+0000 7f5d67e00640 1 --2- 172.21.15.40:0/4114132648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d68071e60 0x7f5d6806f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.829+0000 7f5d67e00640 1 -- 172.21.15.40:0/4114132648 >> v1:172.21.15.103:6789/0 conn(0x7f5d68072780 legacy=0x7f5d680cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:03.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.829+0000 7f5d67e00640 1 -- 172.21.15.40:0/4114132648 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d6815e060 con 0x7f5d6806fe20 2024-04-26T23:10:03.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.831+0000 7f5d67e00640 1 --2- 172.21.15.40:0/4114132648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d6806fe20 0x7f5d68072240 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5d50009d90 tx=0x7f5d5002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30fd0c1426546f54 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:03.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.831+0000 7f5d67400640 1 -- 172.21.15.40:0/4114132648 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d50007740 con 0x7f5d6806fe20 2024-04-26T23:10:03.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.831+0000 7f5d67400640 1 -- 172.21.15.40:0/4114132648 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d500078e0 con 0x7f5d6806fe20 2024-04-26T23:10:03.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.831+0000 7f5d67400640 1 -- 172.21.15.40:0/4114132648 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d50007cb0 con 0x7f5d6806fe20 2024-04-26T23:10:03.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.832+0000 7f5d6fc00640 1 -- 172.21.15.40:0/4114132648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d6806fe20 msgr2=0x7f5d68072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.832+0000 7f5d6fc00640 1 --2- 172.21.15.40:0/4114132648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d6806fe20 0x7f5d68072240 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5d50009d90 tx=0x7f5d5002a660 comp rx=0 tx=0).stop 2024-04-26T23:10:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.832+0000 7f5d6fc00640 1 -- 172.21.15.40:0/4114132648 shutdown_connections 2024-04-26T23:10:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.832+0000 7f5d6fc00640 1 --2- 172.21.15.40:0/4114132648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d6806fe20 0x7f5d68072240 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.832+0000 7f5d6fc00640 1 --2- 172.21.15.40:0/4114132648 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d68071e60 0x7f5d6806f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.832+0000 7f5d6fc00640 1 -- 172.21.15.40:0/4114132648 >> 172.21.15.40:0/4114132648 conn(0x7f5d680d0be0 msgr2=0x7f5d680d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.832+0000 7f5d6fc00640 1 -- 172.21.15.40:0/4114132648 shutdown_connections 2024-04-26T23:10:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.832+0000 7f5d6fc00640 1 -- 172.21.15.40:0/4114132648 wait complete. 2024-04-26T23:10:03.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.834+0000 7f5d6fc00640 1 Processor -- start 2024-04-26T23:10:03.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.834+0000 7f5d6fc00640 1 -- start start 2024-04-26T23:10:03.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.835+0000 7f5d6fc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6806fe20 0x7f5d6815bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.835+0000 7f5d6fc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d68071e60 0x7f5d681593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.835+0000 7f5d6fc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d68072780 0x7f5d6814eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.835+0000 7f5d6fc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6815d3f0 con 0x7f5d6806fe20 2024-04-26T23:10:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.835+0000 7f5d6fc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5d68072c10 con 0x7f5d68071e60 2024-04-26T23:10:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.835+0000 7f5d6fc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5d6815d570 con 0x7f5d68072780 2024-04-26T23:10:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.835+0000 7f5d6d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6806fe20 0x7f5d6815bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.835+0000 7f5d6d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6806fe20 0x7f5d6815bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46022/0 (socket says 172.21.15.40:46022) 2024-04-26T23:10:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.835+0000 7f5d6d200640 1 -- 172.21.15.40:0/1002167735 learned_addr learned my addr 172.21.15.40:0/1002167735 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.836+0000 7f5d67e00640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d68071e60 0x7f5d681593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.836+0000 7f5d6dc00640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d68072780 0x7f5d6814eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.836+0000 7f5d6d200640 1 -- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d68072780 msgr2=0x7f5d6814eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.836+0000 7f5d6d200640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d68072780 0x7f5d6814eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.836+0000 7f5d6d200640 1 -- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d68071e60 msgr2=0x7f5d681593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.836+0000 7f5d6d200640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d68071e60 0x7f5d681593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.836+0000 7f5d6d200640 1 -- 172.21.15.40:0/1002167735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d68159e70 con 0x7f5d6806fe20 2024-04-26T23:10:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.836+0000 7f5d67e00640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d68071e60 0x7f5d681593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.836+0000 7f5d6d200640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6806fe20 0x7f5d6815bfa0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800e9e0 tx=0x7f5d5800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.837+0000 7f5d64c00640 1 -- 172.21.15.40:0/1002167735 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d58017070 con 0x7f5d6806fe20 2024-04-26T23:10:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.837+0000 7f5d64c00640 1 -- 172.21.15.40:0/1002167735 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d5800c9d0 con 0x7f5d6806fe20 2024-04-26T23:10:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.837+0000 7f5d64c00640 1 -- 172.21.15.40:0/1002167735 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5800cdc0 con 0x7f5d6806fe20 2024-04-26T23:10:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.837+0000 7f5d6fc00640 1 -- 172.21.15.40:0/1002167735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d6815a150 con 0x7f5d6806fe20 2024-04-26T23:10:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.837+0000 7f5d6fc00640 1 -- 172.21.15.40:0/1002167735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d6815a710 con 0x7f5d6806fe20 2024-04-26T23:10:03.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.838+0000 7f5d6fc00640 1 -- 172.21.15.40:0/1002167735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d680694f0 con 0x7f5d6806fe20 2024-04-26T23:10:03.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.839+0000 7f5d64c00640 1 -- 172.21.15.40:0/1002167735 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d5801a070 con 0x7f5d6806fe20 2024-04-26T23:10:03.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.840+0000 7f5d64c00640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5d4406c8f0 0x7f5d4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.841+0000 7f5d64c00640 1 -- 172.21.15.40:0/1002167735 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f5d5808c140 con 0x7f5d6806fe20 2024-04-26T23:10:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.841+0000 7f5d67e00640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5d4406c8f0 0x7f5d4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.842+0000 7f5d67e00640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5d4406c8f0 0x7f5d4406edb0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5d50034000 tx=0x7f5d5002f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:03.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:03.845+0000 7f5d64c00640 1 -- 172.21.15.40:0/1002167735 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5d5805a320 con 0x7f5d6806fe20 2024-04-26T23:10:04.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.066+0000 7f5d6fc00640 1 -- 172.21.15.40:0/1002167735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5d681601b0 con 0x7f5d6806fe20 2024-04-26T23:10:04.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.067+0000 7f5d64c00640 1 -- 172.21.15.40:0/1002167735 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v105) ==== 80+0+0 (secure 0 0 0) 0x7f5d58059c20 con 0x7f5d6806fe20 2024-04-26T23:10:04.067 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-04-26T23:10:04.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.070+0000 7f5d6fc00640 1 -- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5d4406c8f0 msgr2=0x7f5d4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:04.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.070+0000 7f5d6fc00640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5d4406c8f0 0x7f5d4406edb0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5d50034000 tx=0x7f5d5002f000 comp rx=0 tx=0).stop 2024-04-26T23:10:04.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.070+0000 7f5d6fc00640 1 -- 172.21.15.40:0/1002167735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6806fe20 msgr2=0x7f5d6815bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:04.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.070+0000 7f5d6fc00640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6806fe20 0x7f5d6815bfa0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800e9e0 tx=0x7f5d5800eea0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.070+0000 7f5d6fc00640 1 -- 172.21.15.40:0/1002167735 shutdown_connections 2024-04-26T23:10:04.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.070+0000 7f5d6fc00640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5d4406c8f0 0x7f5d4406edb0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.070+0000 7f5d6fc00640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5d68072780 0x7f5d6814eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.070+0000 7f5d6fc00640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d68071e60 0x7f5d681593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.070+0000 7f5d6fc00640 1 --2- 172.21.15.40:0/1002167735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6806fe20 0x7f5d6815bfa0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.070+0000 7f5d6fc00640 1 -- 172.21.15.40:0/1002167735 >> 172.21.15.40:0/1002167735 conn(0x7f5d680d0be0 msgr2=0x7f5d680b06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:04.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.070+0000 7f5d6fc00640 1 -- 172.21.15.40:0/1002167735 shutdown_connections 2024-04-26T23:10:04.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.070+0000 7f5d6fc00640 1 -- 172.21.15.40:0/1002167735 wait complete. 2024-04-26T23:10:04.090 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T23:10:04.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:04.090 DEBUG:teuthology.orchestra.run.smithi040:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-26T23:10:04.193 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:10:04.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:04.193 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T23:10:04.248 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:10:04.248 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:10:04.248 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:10:04.304 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:10:04.304 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:10:04.304 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:10:04.305 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:10:04.365 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:10:04.366 INFO:tasks.cephfs.mount:Terminating background process 2024-04-26T23:10:04.366 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:10:04.366 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:10:04.422 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:10:04.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:10:04.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.643+0000 7f03a3e00640 1 Processor -- start 2024-04-26T23:10:04.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.643+0000 7f03a3e00640 1 -- start start 2024-04-26T23:10:04.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.644+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03a40ce5e0 0x7f03a40afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:04.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.644+0000 7f03a3e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f03a40cc590 0x7f03a40cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.645+0000 7f03a3e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03a40af7b0 con 0x7f03a415f2d0 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.645+0000 7f03a3e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f03a40afa90 con 0x7f03a40cc590 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.645+0000 7f03a3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f03a40cbd90 con 0x7f03a40ce5e0 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.645+0000 7f03a2a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f03a415f2d0 0x7f03a40c58b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48496/0 (socket says 172.21.15.40:48496) 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.645+0000 7f03a2a00640 1 -- 172.21.15.40:0/3916258990 learned_addr learned my addr 172.21.15.40:0/3916258990 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.645+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1506922627 0 0) 0x7f03a40af7b0 con 0x7f03a415f2d0 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.645+0000 7f03a3400640 1 --2- 172.21.15.40:0/3916258990 >> v2:172.21.15.103:3300/0 conn(0x7f03a40cc590 0x7f03a40cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.645+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0388003520 con 0x7f03a415f2d0 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.645+0000 7f03a2000640 1 --2- 172.21.15.40:0/3916258990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03a40ce5e0 0x7f03a40afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4200882459 0 0) 0x7f0388003520 con 0x7f03a415f2d0 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03a40ce5e0 msgr2=0x7f03a40afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 --2- 172.21.15.40:0/3916258990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03a40ce5e0 0x7f03a40afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 >> v2:172.21.15.103:3300/0 conn(0x7f03a40cc590 msgr2=0x7f03a40cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 --2- 172.21.15.40:0/3916258990 >> v2:172.21.15.103:3300/0 conn(0x7f03a40cc590 0x7f03a40cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a40cc070 con 0x7f03a415f2d0 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f0394001670 con 0x7f03a415f2d0 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 >> v1:172.21.15.40:6789/0 conn(0x7f03a415f2d0 legacy=0x7f03a40c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 --2- 172.21.15.40:0/3916258990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03880039b0 0x7f0388003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 --2- 172.21.15.40:0/3916258990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0388004800 0x7f0388004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 --2- 172.21.15.40:0/3916258990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03a40ce5e0 0x7f03880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f03a40afa90 con 0x7f03880039b0 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f03a40cbd90 con 0x7f0388004800 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f038800b9c0 con 0x7f03a40ce5e0 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a2a00640 1 --2- 172.21.15.40:0/3916258990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03880039b0 0x7f0388003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a2a00640 1 -- 172.21.15.40:0/3916258990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03a40ce5e0 msgr2=0x7f03880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a2a00640 1 --2- 172.21.15.40:0/3916258990 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03a40ce5e0 0x7f03880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a2a00640 1 -- 172.21.15.40:0/3916258990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0388004800 msgr2=0x7f0388004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.646+0000 7f03a2a00640 1 --2- 172.21.15.40:0/3916258990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0388004800 0x7f0388004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.647+0000 7f03a2a00640 1 -- 172.21.15.40:0/3916258990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0388003520 con 0x7f03880039b0 2024-04-26T23:10:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.648+0000 7f03a2a00640 1 --2- 172.21.15.40:0/3916258990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03880039b0 0x7f0388003dd0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f039400c290 tx=0x7f0388005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b52ad77f4ac536e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.648+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394007d50 con 0x7f03880039b0 2024-04-26T23:10:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.648+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0394005af0 con 0x7f03880039b0 2024-04-26T23:10:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.648+0000 7f03a1600640 1 -- 172.21.15.40:0/3916258990 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394007070 con 0x7f03880039b0 2024-04-26T23:10:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.648+0000 7f03a3e00640 1 -- 172.21.15.40:0/3916258990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03880039b0 msgr2=0x7f0388003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.648+0000 7f03a3e00640 1 --2- 172.21.15.40:0/3916258990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03880039b0 0x7f0388003dd0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f039400c290 tx=0x7f0388005830 comp rx=0 tx=0).stop 2024-04-26T23:10:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.648+0000 7f03a3400640 1 -- 172.21.15.40:0/3916258990 reap_dead start 2024-04-26T23:10:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.649+0000 7f03a3e00640 1 -- 172.21.15.40:0/3916258990 shutdown_connections 2024-04-26T23:10:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.649+0000 7f03a3e00640 1 -- 172.21.15.40:0/3916258990 >> 172.21.15.40:0/3916258990 conn(0x7f03a40d0a10 msgr2=0x7f03a40d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.649+0000 7f03a3e00640 1 -- 172.21.15.40:0/3916258990 shutdown_connections 2024-04-26T23:10:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.649+0000 7f03a3e00640 1 -- 172.21.15.40:0/3916258990 wait complete. 2024-04-26T23:10:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.651+0000 7f03a3e00640 1 Processor -- start 2024-04-26T23:10:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.651+0000 7f03a3e00640 1 -- start start 2024-04-26T23:10:04.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.652+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03880039b0 0x7f03a414e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.652+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03a40738f0 0x7f03a406f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.652+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a406fb60 0x7f03a4071c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.652+0000 7f03a3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f03a40c5fc0 con 0x7f03a406fb60 2024-04-26T23:10:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.652+0000 7f03a3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f03a406e270 con 0x7f03880039b0 2024-04-26T23:10:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.652+0000 7f03a3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f03a4073460 con 0x7f03a40738f0 2024-04-26T23:10:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.652+0000 7f03a3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a406fb60 0x7f03a4071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.652+0000 7f03a3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a406fb60 0x7f03a4071c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46038/0 (socket says 172.21.15.40:46038) 2024-04-26T23:10:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.652+0000 7f03a3400640 1 -- 172.21.15.40:0/1345370312 learned_addr learned my addr 172.21.15.40:0/1345370312 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.653+0000 7f03a2000640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03a40738f0 0x7f03a406f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.653+0000 7f03a2a00640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03880039b0 0x7f03a414e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.653+0000 7f03a3400640 1 -- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03a40738f0 msgr2=0x7f03a406f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.653+0000 7f03a3400640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03a40738f0 0x7f03a406f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.653+0000 7f03a3400640 1 -- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03880039b0 msgr2=0x7f03a414e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.653+0000 7f03a3400640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03880039b0 0x7f03a414e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.653+0000 7f03a3400640 1 -- 172.21.15.40:0/1345370312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a40b4a70 con 0x7f03a406fb60 2024-04-26T23:10:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.653+0000 7f03a3400640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a406fb60 0x7f03a4071c00 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f03a40d1b10 tx=0x7f039c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.653+0000 7f039aa00640 1 -- 172.21.15.40:0/1345370312 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c00bae0 con 0x7f03a406fb60 2024-04-26T23:10:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.654+0000 7f039aa00640 1 -- 172.21.15.40:0/1345370312 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f039c01a070 con 0x7f03a406fb60 2024-04-26T23:10:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.654+0000 7f03a3e00640 1 -- 172.21.15.40:0/1345370312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a414ebc0 con 0x7f03a406fb60 2024-04-26T23:10:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.654+0000 7f039aa00640 1 -- 172.21.15.40:0/1345370312 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c014400 con 0x7f03a406fb60 2024-04-26T23:10:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.654+0000 7f03a3e00640 1 -- 172.21.15.40:0/1345370312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a4072b30 con 0x7f03a406fb60 2024-04-26T23:10:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.655+0000 7f03a3e00640 1 -- 172.21.15.40:0/1345370312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03a4069370 con 0x7f03a406fb60 2024-04-26T23:10:04.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.656+0000 7f039aa00640 1 -- 172.21.15.40:0/1345370312 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f039c005d10 con 0x7f03a406fb60 2024-04-26T23:10:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.657+0000 7f039aa00640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f038406c8f0 0x7f038406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.657+0000 7f039aa00640 1 -- 172.21.15.40:0/1345370312 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f039c08bf30 con 0x7f03a406fb60 2024-04-26T23:10:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.658+0000 7f03a2a00640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f038406c8f0 0x7f038406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.659+0000 7f03a2a00640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f038406c8f0 0x7f038406edb0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0394033000 tx=0x7f0394009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.662+0000 7f039aa00640 1 -- 172.21.15.40:0/1345370312 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f039c024020 con 0x7f03a406fb60 2024-04-26T23:10:04.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.911+0000 7f03a3e00640 1 -- 172.21.15.40:0/1345370312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03a406de10 con 0x7f03a406fb60 2024-04-26T23:10:04.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.911+0000 7f039aa00640 1 -- 172.21.15.40:0/1345370312 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5239 (secure 0 0 0) 0x7f03a414ebc0 con 0x7f03a406fb60 2024-04-26T23:10:04.912 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:04.912 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:44.797258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[6598,8949,8955]},"id":7}]} 2024-04-26T23:10:04.912 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 122 2024-04-26T23:10:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.914+0000 7f03a3e00640 1 -- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f038406c8f0 msgr2=0x7f038406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.914+0000 7f03a3e00640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f038406c8f0 0x7f038406edb0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0394033000 tx=0x7f0394009030 comp rx=0 tx=0).stop 2024-04-26T23:10:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.914+0000 7f03a3e00640 1 -- 172.21.15.40:0/1345370312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a406fb60 msgr2=0x7f03a4071c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.914+0000 7f03a3e00640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a406fb60 0x7f03a4071c00 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f03a40d1b10 tx=0x7f039c00b480 comp rx=0 tx=0).stop 2024-04-26T23:10:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.915+0000 7f03a3e00640 1 -- 172.21.15.40:0/1345370312 shutdown_connections 2024-04-26T23:10:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.915+0000 7f03a3e00640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f038406c8f0 0x7f038406edb0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.915+0000 7f03a3e00640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a406fb60 0x7f03a4071c00 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.915+0000 7f03a3e00640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03a40738f0 0x7f03a406f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.915+0000 7f03a3e00640 1 --2- 172.21.15.40:0/1345370312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03880039b0 0x7f03a414e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:04.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.915+0000 7f03a3e00640 1 -- 172.21.15.40:0/1345370312 >> 172.21.15.40:0/1345370312 conn(0x7f03a40d0a10 msgr2=0x7f03a40c62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:04.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.915+0000 7f03a3e00640 1 -- 172.21.15.40:0/1345370312 shutdown_connections 2024-04-26T23:10:04.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:04.915+0000 7f03a3e00640 1 -- 172.21.15.40:0/1345370312 wait complete. 2024-04-26T23:10:04.935 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:10:05.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.148+0000 7f76ffa00640 1 Processor -- start 2024-04-26T23:10:05.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.148+0000 7f76ffa00640 1 -- start start 2024-04-26T23:10:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.150+0000 7f76ffa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f76f806fc40 0x7f76f8073360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.150+0000 7f76ffa00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76f8156840 con 0x7f76f8156460 2024-04-26T23:10:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.150+0000 7f76ffa00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76f80cbf20 con 0x7f76f806f790 2024-04-26T23:10:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.150+0000 7f76ffa00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f76f80cc200 con 0x7f76f806fc40 2024-04-26T23:10:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.150+0000 7f76fd000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f76f8156460 0x7f76f80cb4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51838/0 (socket says 172.21.15.40:51838) 2024-04-26T23:10:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.151+0000 7f76fd000640 1 -- 172.21.15.40:0/3619149018 learned_addr learned my addr 172.21.15.40:0/3619149018 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.151+0000 7f76fda00640 1 --2- 172.21.15.40:0/3619149018 >> v2:172.21.15.40:3300/0 conn(0x7f76f806fc40 0x7f76f8073360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.151+0000 7f76f7400640 1 -- 172.21.15.40:0/3619149018 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2760317647 0 0) 0x7f76f8156840 con 0x7f76f8156460 2024-04-26T23:10:05.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.151+0000 7f76f7400640 1 -- 172.21.15.40:0/3619149018 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76dc003520 con 0x7f76f8156460 2024-04-26T23:10:05.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.151+0000 7f76fda00640 1 -- 172.21.15.40:0/3619149018 >> v1:172.21.15.103:6789/0 conn(0x7f76f806f790 legacy=0x7f76f8072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:05.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.151+0000 7f76fda00640 1 -- 172.21.15.40:0/3619149018 >> v1:172.21.15.40:6789/0 conn(0x7f76f8156460 legacy=0x7f76f80cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.151+0000 7f76fda00640 1 -- 172.21.15.40:0/3619149018 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f80cc4e0 con 0x7f76f806fc40 2024-04-26T23:10:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.152+0000 7f76fda00640 1 --2- 172.21.15.40:0/3619149018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76f806fc40 0x7f76f8073360 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f76f0009d90 tx=0x7f76f002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb19641553c56ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.153+0000 7f76f7400640 1 -- 172.21.15.40:0/3619149018 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0007740 con 0x7f76f806fc40 2024-04-26T23:10:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.153+0000 7f76f7400640 1 -- 172.21.15.40:0/3619149018 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76f00078e0 con 0x7f76f806fc40 2024-04-26T23:10:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.153+0000 7f76f7400640 1 -- 172.21.15.40:0/3619149018 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0007cb0 con 0x7f76f806fc40 2024-04-26T23:10:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.153+0000 7f76ffa00640 1 -- 172.21.15.40:0/3619149018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76f806fc40 msgr2=0x7f76f8073360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.153+0000 7f76ffa00640 1 --2- 172.21.15.40:0/3619149018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76f806fc40 0x7f76f8073360 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f76f0009d90 tx=0x7f76f002a6f0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.153+0000 7f76ffa00640 1 -- 172.21.15.40:0/3619149018 shutdown_connections 2024-04-26T23:10:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.153+0000 7f76ffa00640 1 --2- 172.21.15.40:0/3619149018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76f806fc40 0x7f76f8073360 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.153+0000 7f76ffa00640 1 -- 172.21.15.40:0/3619149018 >> 172.21.15.40:0/3619149018 conn(0x7f76f80d0be0 msgr2=0x7f76f80d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.153+0000 7f76ffa00640 1 -- 172.21.15.40:0/3619149018 shutdown_connections 2024-04-26T23:10:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.154+0000 7f76ffa00640 1 -- 172.21.15.40:0/3619149018 wait complete. 2024-04-26T23:10:05.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.155+0000 7f76ffa00640 1 Processor -- start 2024-04-26T23:10:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.156+0000 7f76ffa00640 1 -- start start 2024-04-26T23:10:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.156+0000 7f76ffa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76f806f790 0x7f76f814b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.156+0000 7f76ffa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76f806fc40 0x7f76f81f7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.156+0000 7f76ffa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76f8156460 0x7f76f81f7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.156+0000 7f76ffa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f76f8070050 con 0x7f76f806f790 2024-04-26T23:10:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.156+0000 7f76ffa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f76f8072700 con 0x7f76f806fc40 2024-04-26T23:10:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.156+0000 7f76ffa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f76f80cbbe0 con 0x7f76f8156460 2024-04-26T23:10:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.157+0000 7f76fd000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76f806f790 0x7f76f814b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.157+0000 7f76fd000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76f806f790 0x7f76f814b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38386/0 (socket says 172.21.15.40:38386) 2024-04-26T23:10:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.157+0000 7f76fd000640 1 -- 172.21.15.40:0/426248473 learned_addr learned my addr 172.21.15.40:0/426248473 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.157+0000 7f76f7e00640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76f806fc40 0x7f76f81f7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.157+0000 7f76fda00640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76f8156460 0x7f76f81f7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.157+0000 7f76fd000640 1 -- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76f8156460 msgr2=0x7f76f81f7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:05.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.157+0000 7f76fd000640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76f8156460 0x7f76f81f7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.157+0000 7f76fd000640 1 -- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76f806fc40 msgr2=0x7f76f81f7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:05.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.157+0000 7f76fd000640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76f806fc40 0x7f76f81f7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.157+0000 7f76fd000640 1 -- 172.21.15.40:0/426248473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f80c97b0 con 0x7f76f806f790 2024-04-26T23:10:05.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.158+0000 7f76fd000640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76f806f790 0x7f76f814b920 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f76e800c410 tx=0x7f76e800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:05.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.158+0000 7f76f4c00640 1 -- 172.21.15.40:0/426248473 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e8017070 con 0x7f76f806f790 2024-04-26T23:10:05.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.158+0000 7f76f4c00640 1 -- 172.21.15.40:0/426248473 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76e8005ce0 con 0x7f76f806f790 2024-04-26T23:10:05.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.158+0000 7f76f4c00640 1 -- 172.21.15.40:0/426248473 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e80072f0 con 0x7f76f806f790 2024-04-26T23:10:05.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.158+0000 7f76ffa00640 1 -- 172.21.15.40:0/426248473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f80c9a50 con 0x7f76f806f790 2024-04-26T23:10:05.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.158+0000 7f76ffa00640 1 -- 172.21.15.40:0/426248473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f80ca010 con 0x7f76f806f790 2024-04-26T23:10:05.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.159+0000 7f76ffa00640 1 -- 172.21.15.40:0/426248473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76f80694f0 con 0x7f76f806f790 2024-04-26T23:10:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.160+0000 7f76f4c00640 1 -- 172.21.15.40:0/426248473 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f76e801a070 con 0x7f76f806f790 2024-04-26T23:10:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.162+0000 7f76f4c00640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f76d806c920 0x7f76d806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.162+0000 7f76f4c00640 1 -- 172.21.15.40:0/426248473 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f76e808b1b0 con 0x7f76f806f790 2024-04-26T23:10:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.162+0000 7f76f7e00640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f76d806c920 0x7f76d806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.164+0000 7f76f7e00640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f76d806c920 0x7f76d806ede0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f76f815e900 tx=0x7f76e4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.167+0000 7f76f4c00640 1 -- 172.21.15.40:0/426248473 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f76e8059450 con 0x7f76f806f790 2024-04-26T23:10:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.371+0000 7f76ffa00640 1 -- 172.21.15.40:0/426248473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76f815b4c0 con 0x7f76f806f790 2024-04-26T23:10:05.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.374+0000 7f76f4c00640 1 -- 172.21.15.40:0/426248473 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+16898 (secure 0 0 0) 0x7f76e8058d50 con 0x7f76f806f790 2024-04-26T23:10:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:05.375 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":105,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:09:49.407703+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:09:03.955148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-26T23:09:05.070627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.2a","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:10:05.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.377+0000 7f76ffa00640 1 -- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f76d806c920 msgr2=0x7f76d806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:05.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.377+0000 7f76ffa00640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f76d806c920 0x7f76d806ede0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f76f815e900 tx=0x7f76e4009040 comp rx=0 tx=0).stop 2024-04-26T23:10:05.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.377+0000 7f76ffa00640 1 -- 172.21.15.40:0/426248473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76f806f790 msgr2=0x7f76f814b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:05.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.377+0000 7f76ffa00640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76f806f790 0x7f76f814b920 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f76e800c410 tx=0x7f76e800c8d0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.377+0000 7f76ffa00640 1 -- 172.21.15.40:0/426248473 shutdown_connections 2024-04-26T23:10:05.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.377+0000 7f76ffa00640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f76d806c920 0x7f76d806ede0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.377+0000 7f76ffa00640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f76f8156460 0x7f76f81f7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.377+0000 7f76ffa00640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76f806fc40 0x7f76f81f7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.377+0000 7f76ffa00640 1 --2- 172.21.15.40:0/426248473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76f806f790 0x7f76f814b920 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.377+0000 7f76ffa00640 1 -- 172.21.15.40:0/426248473 >> 172.21.15.40:0/426248473 conn(0x7f76f80d0be0 msgr2=0x7f76f80cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:05.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.378+0000 7f76ffa00640 1 -- 172.21.15.40:0/426248473 shutdown_connections 2024-04-26T23:10:05.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.378+0000 7f76ffa00640 1 -- 172.21.15.40:0/426248473 wait complete. 2024-04-26T23:10:05.399 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:00:53.175324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:09:03.955148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:09:05.070627+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:10:05.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:10:05.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.615+0000 7f1b01800640 1 Processor -- start 2024-04-26T23:10:05.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.616+0000 7f1b01800640 1 -- start start 2024-04-26T23:10:05.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.617+0000 7f1b01800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1afc0b5ab0 0x7f1afc0726f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.617+0000 7f1b01800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc0d0380 0x7f1afc0b4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.617+0000 7f1b01800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1afc0b4cf0 con 0x7f1afc15b7b0 2024-04-26T23:10:05.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.617+0000 7f1b01800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1afc072c30 con 0x7f1afc0b5ab0 2024-04-26T23:10:05.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.617+0000 7f1b01800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1afc0b4fd0 con 0x7f1afc0d0380 2024-04-26T23:10:05.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.618+0000 7f1af3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1afc15b7b0 0x7f1afc0b3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51848/0 (socket says 172.21.15.40:51848) 2024-04-26T23:10:05.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.618+0000 7f1af3e00640 1 -- 172.21.15.40:0/2659809290 learned_addr learned my addr 172.21.15.40:0/2659809290 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.618+0000 7f1afbe00640 1 --2- 172.21.15.40:0/2659809290 >> v2:172.21.15.103:3300/0 conn(0x7f1afc0b5ab0 0x7f1afc0726f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.618+0000 7f1b00e00640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc0d0380 0x7f1afc0b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.618+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1672209483 0 0) 0x7f1afc0b4cf0 con 0x7f1afc15b7b0 2024-04-26T23:10:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.618+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ae0003520 con 0x7f1afc15b7b0 2024-04-26T23:10:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1999362254 0 0) 0x7f1ae0003520 con 0x7f1afc15b7b0 2024-04-26T23:10:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc0d0380 msgr2=0x7f1afc0b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc0d0380 0x7f1afc0b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 >> v2:172.21.15.103:3300/0 conn(0x7f1afc0b5ab0 msgr2=0x7f1afc0726f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 --2- 172.21.15.40:0/2659809290 >> v2:172.21.15.103:3300/0 conn(0x7f1afc0b5ab0 0x7f1afc0726f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1afc0b52b0 con 0x7f1afc15b7b0 2024-04-26T23:10:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f1ae4001670 con 0x7f1afc15b7b0 2024-04-26T23:10:05.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afbe00640 1 --2- 172.21.15.40:0/2659809290 >> v2:172.21.15.103:3300/0 conn(0x7f1afc0b5ab0 0x7f1afc0726f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:10:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1b00e00640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc0d0380 0x7f1afc0b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:10:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 >> v1:172.21.15.40:6789/0 conn(0x7f1afc15b7b0 legacy=0x7f1afc0b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae00039b0 0x7f1ae0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc0d0380 0x7f1ae0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ae0004d40 0x7f1ae00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc0b4fd0 con 0x7f1ae00039b0 2024-04-26T23:10:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae000ba70 con 0x7f1ae0004d40 2024-04-26T23:10:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae000bc70 con 0x7f1afc0d0380 2024-04-26T23:10:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.619+0000 7f1afbe00640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae00039b0 0x7f1ae0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.620+0000 7f1b00e00640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ae0004d40 0x7f1ae00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.620+0000 7f1af3e00640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc0d0380 0x7f1ae0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.620+0000 7f1afbe00640 1 -- 172.21.15.40:0/2659809290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc0d0380 msgr2=0x7f1ae0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.620+0000 7f1afbe00640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc0d0380 0x7f1ae0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.620+0000 7f1afbe00640 1 -- 172.21.15.40:0/2659809290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ae0004d40 msgr2=0x7f1ae00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:05.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.620+0000 7f1afbe00640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ae0004d40 0x7f1ae00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.620+0000 7f1afbe00640 1 -- 172.21.15.40:0/2659809290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae0003520 con 0x7f1ae00039b0 2024-04-26T23:10:05.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.620+0000 7f1b00e00640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ae0004d40 0x7f1ae00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.621+0000 7f1afbe00640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae00039b0 0x7f1ae0003dd0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f1aec0098a0 tx=0x7f1ae0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67b739b8bdb5988 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.622+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec00f040 con 0x7f1ae00039b0 2024-04-26T23:10:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.622+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aec02ad20 con 0x7f1ae00039b0 2024-04-26T23:10:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.622+0000 7f1afb400640 1 -- 172.21.15.40:0/2659809290 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec004250 con 0x7f1ae00039b0 2024-04-26T23:10:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.622+0000 7f1b01800640 1 -- 172.21.15.40:0/2659809290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae00039b0 msgr2=0x7f1ae0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.622+0000 7f1b01800640 1 --2- 172.21.15.40:0/2659809290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ae00039b0 0x7f1ae0003dd0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f1aec0098a0 tx=0x7f1ae0005830 comp rx=0 tx=0).stop 2024-04-26T23:10:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.622+0000 7f1b00e00640 1 -- 172.21.15.40:0/2659809290 reap_dead start 2024-04-26T23:10:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.623+0000 7f1b01800640 1 -- 172.21.15.40:0/2659809290 shutdown_connections 2024-04-26T23:10:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.623+0000 7f1b01800640 1 -- 172.21.15.40:0/2659809290 >> 172.21.15.40:0/2659809290 conn(0x7f1afc0b2720 msgr2=0x7f1afc0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:05.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.623+0000 7f1b01800640 1 -- 172.21.15.40:0/2659809290 shutdown_connections 2024-04-26T23:10:05.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.623+0000 7f1b01800640 1 -- 172.21.15.40:0/2659809290 wait complete. 2024-04-26T23:10:05.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.624+0000 7f1b01800640 1 Processor -- start 2024-04-26T23:10:05.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.625+0000 7f1b01800640 1 -- start start 2024-04-26T23:10:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.625+0000 7f1b01800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1afc146080 0x7f1afc151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.625+0000 7f1b01800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc14a1c0 0x7f1afc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.625+0000 7f1b01800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1afc15d1e0 0x7f1afc15d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.626+0000 7f1b01800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc06ed50 con 0x7f1afc146080 2024-04-26T23:10:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.626+0000 7f1b01800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc072e60 con 0x7f1afc15d1e0 2024-04-26T23:10:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.626+0000 7f1b01800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1afc06eed0 con 0x7f1afc14a1c0 2024-04-26T23:10:05.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.626+0000 7f1afbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1afc146080 0x7f1afc151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.626+0000 7f1afbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1afc146080 0x7f1afc151850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38398/0 (socket says 172.21.15.40:38398) 2024-04-26T23:10:05.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.626+0000 7f1afbe00640 1 -- 172.21.15.40:0/3751981620 learned_addr learned my addr 172.21.15.40:0/3751981620 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:05.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.626+0000 7f1af3e00640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc14a1c0 0x7f1afc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.626+0000 7f1b00e00640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1afc15d1e0 0x7f1afc15d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.626+0000 7f1afbe00640 1 -- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc14a1c0 msgr2=0x7f1afc15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.626+0000 7f1afbe00640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc14a1c0 0x7f1afc15c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.626+0000 7f1afbe00640 1 -- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1afc15d1e0 msgr2=0x7f1afc15d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.626+0000 7f1afbe00640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1afc15d1e0 0x7f1afc15d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.626+0000 7f1afbe00640 1 -- 172.21.15.40:0/3751981620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1afc14a640 con 0x7f1afc146080 2024-04-26T23:10:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.627+0000 7f1afbe00640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1afc146080 0x7f1afc151850 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f1aec02a790 tx=0x7f1aec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:05.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.627+0000 7f1af8c00640 1 -- 172.21.15.40:0/3751981620 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec041070 con 0x7f1afc146080 2024-04-26T23:10:05.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.627+0000 7f1af8c00640 1 -- 172.21.15.40:0/3751981620 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aec00f000 con 0x7f1afc146080 2024-04-26T23:10:05.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.627+0000 7f1b01800640 1 -- 172.21.15.40:0/3751981620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1afc15cd70 con 0x7f1afc146080 2024-04-26T23:10:05.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.627+0000 7f1af8c00640 1 -- 172.21.15.40:0/3751981620 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec0044d0 con 0x7f1afc146080 2024-04-26T23:10:05.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.627+0000 7f1b01800640 1 -- 172.21.15.40:0/3751981620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1afc0b09f0 con 0x7f1afc146080 2024-04-26T23:10:05.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.629+0000 7f1b01800640 1 -- 172.21.15.40:0/3751981620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1afc0763b0 con 0x7f1afc146080 2024-04-26T23:10:05.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.629+0000 7f1af8c00640 1 -- 172.21.15.40:0/3751981620 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1aec044070 con 0x7f1afc146080 2024-04-26T23:10:05.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.630+0000 7f1af8c00640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ad806c8f0 0x7f1ad806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:05.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.631+0000 7f1af8c00640 1 -- 172.21.15.40:0/3751981620 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f1aec0ac610 con 0x7f1afc146080 2024-04-26T23:10:05.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.631+0000 7f1af3e00640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ad806c8f0 0x7f1ad806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:05.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.632+0000 7f1af3e00640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ad806c8f0 0x7f1ad806edb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4000ff0 tx=0x7f1ae4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:05.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.635+0000 7f1af8c00640 1 -- 172.21.15.40:0/3751981620 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1afc0763b0 con 0x7f1afc146080 2024-04-26T23:10:05.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.838+0000 7f1b01800640 1 -- 172.21.15.40:0/3751981620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1afc0b0510 con 0x7f1afc146080 2024-04-26T23:10:05.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.839+0000 7f1af8c00640 1 -- 172.21.15.40:0/3751981620 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) ==== 93+0+10 (secure 0 0 0) 0x7f1aec07a7f0 con 0x7f1afc146080 2024-04-26T23:10:05.840 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:10:05.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.842+0000 7f1b01800640 1 -- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ad806c8f0 msgr2=0x7f1ad806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:05.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.842+0000 7f1b01800640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ad806c8f0 0x7f1ad806edb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4000ff0 tx=0x7f1ae4009040 comp rx=0 tx=0).stop 2024-04-26T23:10:05.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.842+0000 7f1b01800640 1 -- 172.21.15.40:0/3751981620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1afc146080 msgr2=0x7f1afc151850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:05.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.842+0000 7f1b01800640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1afc146080 0x7f1afc151850 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f1aec02a790 tx=0x7f1aec004730 comp rx=0 tx=0).stop 2024-04-26T23:10:05.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.842+0000 7f1b01800640 1 -- 172.21.15.40:0/3751981620 shutdown_connections 2024-04-26T23:10:05.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.842+0000 7f1b01800640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1ad806c8f0 0x7f1ad806edb0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.842+0000 7f1b01800640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1afc15d1e0 0x7f1afc15d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.842+0000 7f1b01800640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1afc14a1c0 0x7f1afc15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.842+0000 7f1b01800640 1 --2- 172.21.15.40:0/3751981620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1afc146080 0x7f1afc151850 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:05.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.842+0000 7f1b01800640 1 -- 172.21.15.40:0/3751981620 >> 172.21.15.40:0/3751981620 conn(0x7f1afc0b2720 msgr2=0x7f1afc071ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:05.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.842+0000 7f1b01800640 1 -- 172.21.15.40:0/3751981620 shutdown_connections 2024-04-26T23:10:05.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:05.842+0000 7f1b01800640 1 -- 172.21.15.40:0/3751981620 wait complete. 2024-04-26T23:10:05.862 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:10:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.082+0000 7fc91da00640 1 Processor -- start 2024-04-26T23:10:06.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.082+0000 7fc91da00640 1 -- start start 2024-04-26T23:10:06.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.084+0000 7fc91da00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc91815bc70 0x7fc9180cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:06.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.084+0000 7fc91da00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc9180730a0 0x7fc9180b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:06.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.084+0000 7fc91da00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc91806dde0 con 0x7fc918071ef0 2024-04-26T23:10:06.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.084+0000 7fc91da00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc91806e3a0 con 0x7fc91815bc70 2024-04-26T23:10:06.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.084+0000 7fc91da00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc9180b4840 con 0x7fc9180730a0 2024-04-26T23:10:06.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.084+0000 7fc917e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc91815bc70 0x7fc9180cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:06.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.084+0000 7fc917400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc918071ef0 0x7fc918072990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51852/0 (socket says 172.21.15.40:51852) 2024-04-26T23:10:06.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.084+0000 7fc917400640 1 -- 172.21.15.40:0/2921350192 learned_addr learned my addr 172.21.15.40:0/2921350192 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:06.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.085+0000 7fc91d000640 1 --2- 172.21.15.40:0/2921350192 >> v2:172.21.15.103:3300/0 conn(0x7fc9180730a0 0x7fc9180b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:06.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.085+0000 7fc917e00640 1 -- 172.21.15.40:0/2921350192 >> v2:172.21.15.103:3300/0 conn(0x7fc9180730a0 msgr2=0x7fc9180b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:06.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.085+0000 7fc917e00640 1 --2- 172.21.15.40:0/2921350192 >> v2:172.21.15.103:3300/0 conn(0x7fc9180730a0 0x7fc9180b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.085+0000 7fc917e00640 1 -- 172.21.15.40:0/2921350192 >> v1:172.21.15.40:6789/0 conn(0x7fc918071ef0 legacy=0x7fc918072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:06.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.085+0000 7fc917e00640 1 -- 172.21.15.40:0/2921350192 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc91806e0c0 con 0x7fc91815bc70 2024-04-26T23:10:06.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.085+0000 7fc916a00640 1 -- 172.21.15.40:0/2921350192 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 294415591 0 0) 0x7fc91806dde0 con 0x7fc918071ef0 2024-04-26T23:10:06.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.086+0000 7fc917e00640 1 --2- 172.21.15.40:0/2921350192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc91815bc70 0x7fc9180cdf30 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fc9180cd7c0 tx=0x7fc90802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198f54413b8c3b4e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:06.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.086+0000 7fc916a00640 1 -- 172.21.15.40:0/2921350192 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc908037070 con 0x7fc91815bc70 2024-04-26T23:10:06.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.087+0000 7fc916a00640 1 -- 172.21.15.40:0/2921350192 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc908032440 con 0x7fc91815bc70 2024-04-26T23:10:06.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.087+0000 7fc916a00640 1 -- 172.21.15.40:0/2921350192 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc908032810 con 0x7fc91815bc70 2024-04-26T23:10:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.087+0000 7fc91da00640 1 -- 172.21.15.40:0/2921350192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc91815bc70 msgr2=0x7fc9180cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.087+0000 7fc91da00640 1 --2- 172.21.15.40:0/2921350192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc91815bc70 0x7fc9180cdf30 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fc9180cd7c0 tx=0x7fc90802a1a0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.087+0000 7fc91da00640 1 -- 172.21.15.40:0/2921350192 shutdown_connections 2024-04-26T23:10:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.087+0000 7fc91da00640 1 --2- 172.21.15.40:0/2921350192 >> v2:172.21.15.103:3300/0 conn(0x7fc9180730a0 0x7fc9180b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.087+0000 7fc91da00640 1 --2- 172.21.15.40:0/2921350192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc91815bc70 0x7fc9180cdf30 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.087+0000 7fc91da00640 1 -- 172.21.15.40:0/2921350192 >> 172.21.15.40:0/2921350192 conn(0x7fc9180b2720 msgr2=0x7fc9180b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.087+0000 7fc91da00640 1 -- 172.21.15.40:0/2921350192 shutdown_connections 2024-04-26T23:10:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.088+0000 7fc91da00640 1 -- 172.21.15.40:0/2921350192 wait complete. 2024-04-26T23:10:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.089+0000 7fc91da00640 1 Processor -- start 2024-04-26T23:10:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.089+0000 7fc91da00640 1 -- start start 2024-04-26T23:10:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.090+0000 7fc91da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc918071ef0 0x7fc918153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.090+0000 7fc91da00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9180730a0 0x7fc91815c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.090+0000 7fc91da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc91815bc70 0x7fc91815ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.090+0000 7fc91da00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc91815c050 con 0x7fc918071ef0 2024-04-26T23:10:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.090+0000 7fc91da00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc91806fad0 con 0x7fc91815bc70 2024-04-26T23:10:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.090+0000 7fc91da00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc91815c1d0 con 0x7fc9180730a0 2024-04-26T23:10:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.091+0000 7fc917e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc918071ef0 0x7fc918153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.091+0000 7fc917e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc918071ef0 0x7fc918153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38414/0 (socket says 172.21.15.40:38414) 2024-04-26T23:10:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.091+0000 7fc917e00640 1 -- 172.21.15.40:0/3252640924 learned_addr learned my addr 172.21.15.40:0/3252640924 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.091+0000 7fc917e00640 1 -- 172.21.15.40:0/3252640924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9180730a0 msgr2=0x7fc91815c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.091+0000 7fc917e00640 1 --2- 172.21.15.40:0/3252640924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9180730a0 0x7fc91815c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.091+0000 7fc917e00640 1 -- 172.21.15.40:0/3252640924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc91815bc70 msgr2=0x7fc91815ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.091+0000 7fc917e00640 1 --2- 172.21.15.40:0/3252640924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc91815bc70 0x7fc91815ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.091+0000 7fc917e00640 1 -- 172.21.15.40:0/3252640924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9181522b0 con 0x7fc918071ef0 2024-04-26T23:10:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.091+0000 7fc917e00640 1 --2- 172.21.15.40:0/3252640924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc918071ef0 0x7fc918153350 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fc908000c00 tx=0x7fc9080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.092+0000 7fc90fe00640 1 -- 172.21.15.40:0/3252640924 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc908037040 con 0x7fc918071ef0 2024-04-26T23:10:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.092+0000 7fc90fe00640 1 -- 172.21.15.40:0/3252640924 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc908042070 con 0x7fc918071ef0 2024-04-26T23:10:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.092+0000 7fc90fe00640 1 -- 172.21.15.40:0/3252640924 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc908030920 con 0x7fc918071ef0 2024-04-26T23:10:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.092+0000 7fc91da00640 1 -- 172.21.15.40:0/3252640924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc918152590 con 0x7fc918071ef0 2024-04-26T23:10:06.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.092+0000 7fc91da00640 1 -- 172.21.15.40:0/3252640924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc918152b50 con 0x7fc918071ef0 2024-04-26T23:10:06.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.093+0000 7fc91da00640 1 -- 172.21.15.40:0/3252640924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9180cd810 con 0x7fc918071ef0 2024-04-26T23:10:06.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.094+0000 7fc90fe00640 1 -- 172.21.15.40:0/3252640924 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc908030ac0 con 0x7fc918071ef0 2024-04-26T23:10:06.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.095+0000 7fc90fe00640 1 --2- 172.21.15.40:0/3252640924 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc8f406c8f0 0x7fc8f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:06.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.095+0000 7fc90fe00640 1 -- 172.21.15.40:0/3252640924 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7fc9080ac6b0 con 0x7fc918071ef0 2024-04-26T23:10:06.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.096+0000 7fc917400640 1 --2- 172.21.15.40:0/3252640924 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc8f406c8f0 0x7fc8f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.097+0000 7fc917400640 1 --2- 172.21.15.40:0/3252640924 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc8f406c8f0 0x7fc8f406edb0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc900001560 tx=0x7fc900007ca0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:06.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.100+0000 7fc90fe00640 1 -- 172.21.15.40:0/3252640924 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc9180cd810 con 0x7fc918071ef0 2024-04-26T23:10:06.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.304+0000 7fc91da00640 1 -- 172.21.15.40:0/3252640924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc9180cd1f0 con 0x7fc918071ef0 2024-04-26T23:10:06.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.305+0000 7fc90fe00640 1 -- 172.21.15.40:0/3252640924 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v105) ==== 104+0+11 (secure 0 0 0) 0x7fc90807a2d0 con 0x7fc918071ef0 2024-04-26T23:10:06.306 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:10:06.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.308+0000 7fc91da00640 1 -- 172.21.15.40:0/3252640924 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc8f406c8f0 msgr2=0x7fc8f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:06.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.308+0000 7fc91da00640 1 --2- 172.21.15.40:0/3252640924 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc8f406c8f0 0x7fc8f406edb0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc900001560 tx=0x7fc900007ca0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.308+0000 7fc91da00640 1 -- 172.21.15.40:0/3252640924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc918071ef0 msgr2=0x7fc918153350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:06.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.308+0000 7fc91da00640 1 --2- 172.21.15.40:0/3252640924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc918071ef0 0x7fc918153350 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fc908000c00 tx=0x7fc9080046c0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.308+0000 7fc91da00640 1 -- 172.21.15.40:0/3252640924 shutdown_connections 2024-04-26T23:10:06.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.308+0000 7fc91da00640 1 --2- 172.21.15.40:0/3252640924 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc8f406c8f0 0x7fc8f406edb0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.308+0000 7fc91da00640 1 --2- 172.21.15.40:0/3252640924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc91815bc70 0x7fc91815ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.308+0000 7fc91da00640 1 --2- 172.21.15.40:0/3252640924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9180730a0 0x7fc91815c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.308+0000 7fc91da00640 1 --2- 172.21.15.40:0/3252640924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc918071ef0 0x7fc918153350 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.308+0000 7fc91da00640 1 -- 172.21.15.40:0/3252640924 >> 172.21.15.40:0/3252640924 conn(0x7fc9180b2720 msgr2=0x7fc9180aef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.308+0000 7fc91da00640 1 -- 172.21.15.40:0/3252640924 shutdown_connections 2024-04-26T23:10:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.308+0000 7fc91da00640 1 -- 172.21.15.40:0/3252640924 wait complete. 2024-04-26T23:10:06.328 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:10:06.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.553+0000 7fcaa1e00640 1 Processor -- start 2024-04-26T23:10:06.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.553+0000 7fcaa1e00640 1 -- start start 2024-04-26T23:10:06.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.554+0000 7fcaa1e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca9c071e60 0x7fca9c06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:06.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.554+0000 7fcaa1e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca9c0cc480 0x7fca9c166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:06.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.554+0000 7fcaa1e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca9c15d7c0 con 0x7fca9c06fe20 2024-04-26T23:10:06.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.554+0000 7fcaa1e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fca9c15daa0 con 0x7fca9c0cc480 2024-04-26T23:10:06.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.554+0000 7fcaa1e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fca9c15dd80 con 0x7fca9c071e60 2024-04-26T23:10:06.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.555+0000 7fca9b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca9c0cc480 0x7fca9c166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:06.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.555+0000 7fca9a000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fca9c06fe20 0x7fca9c072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51868/0 (socket says 172.21.15.40:51868) 2024-04-26T23:10:06.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.555+0000 7fca9a000640 1 -- 172.21.15.40:0/348308342 learned_addr learned my addr 172.21.15.40:0/348308342 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:06.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.555+0000 7fca9aa00640 1 --2- 172.21.15.40:0/348308342 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca9c071e60 0x7fca9c06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:06.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.555+0000 7fca99600640 1 -- 172.21.15.40:0/348308342 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3789791763 0 0) 0x7fca9c15d7c0 con 0x7fca9c06fe20 2024-04-26T23:10:06.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.555+0000 7fca99600640 1 -- 172.21.15.40:0/348308342 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca80003520 con 0x7fca9c06fe20 2024-04-26T23:10:06.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.556+0000 7fca9b400640 1 -- 172.21.15.40:0/348308342 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca9c071e60 msgr2=0x7fca9c06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:06.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.556+0000 7fca9b400640 1 --2- 172.21.15.40:0/348308342 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca9c071e60 0x7fca9c06f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.556+0000 7fca9b400640 1 -- 172.21.15.40:0/348308342 >> v1:172.21.15.40:6789/0 conn(0x7fca9c06fe20 legacy=0x7fca9c072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:06.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.556+0000 7fca9b400640 1 -- 172.21.15.40:0/348308342 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca9c15e060 con 0x7fca9c0cc480 2024-04-26T23:10:06.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.557+0000 7fca9b400640 1 --2- 172.21.15.40:0/348308342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c0cc480 0x7fca9c166ad0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fca94009d90 tx=0x7fca9402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6d85fc013a06ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:06.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.557+0000 7fca99600640 1 -- 172.21.15.40:0/348308342 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94007740 con 0x7fca9c0cc480 2024-04-26T23:10:06.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.557+0000 7fca99600640 1 -- 172.21.15.40:0/348308342 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca940078e0 con 0x7fca9c0cc480 2024-04-26T23:10:06.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.557+0000 7fca99600640 1 -- 172.21.15.40:0/348308342 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94007cb0 con 0x7fca9c0cc480 2024-04-26T23:10:06.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.557+0000 7fcaa1e00640 1 -- 172.21.15.40:0/348308342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c0cc480 msgr2=0x7fca9c166ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:06.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.558+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/348308342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c0cc480 0x7fca9c166ad0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fca94009d90 tx=0x7fca9402a660 comp rx=0 tx=0).stop 2024-04-26T23:10:06.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.558+0000 7fcaa1e00640 1 -- 172.21.15.40:0/348308342 shutdown_connections 2024-04-26T23:10:06.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.558+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/348308342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c0cc480 0x7fca9c166ad0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.558+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/348308342 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca9c071e60 0x7fca9c06f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.558+0000 7fcaa1e00640 1 -- 172.21.15.40:0/348308342 >> 172.21.15.40:0/348308342 conn(0x7fca9c0d0be0 msgr2=0x7fca9c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:06.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.558+0000 7fcaa1e00640 1 -- 172.21.15.40:0/348308342 shutdown_connections 2024-04-26T23:10:06.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.558+0000 7fcaa1e00640 1 -- 172.21.15.40:0/348308342 wait complete. 2024-04-26T23:10:06.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.559+0000 7fcaa1e00640 1 Processor -- start 2024-04-26T23:10:06.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.560+0000 7fcaa1e00640 1 -- start start 2024-04-26T23:10:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.560+0000 7fcaa1e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca9c06fe20 0x7fca9c0d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.561+0000 7fcaa1e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca9c071e60 0x7fca9c0b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.561+0000 7fcaa1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c0cc480 0x7fca9c0b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.561+0000 7fcaa1e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c15d570 con 0x7fca9c0cc480 2024-04-26T23:10:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.561+0000 7fcaa1e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c15d3f0 con 0x7fca9c071e60 2024-04-26T23:10:06.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.561+0000 7fcaa1e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fca9c072950 con 0x7fca9c06fe20 2024-04-26T23:10:06.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.561+0000 7fca9b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c0cc480 0x7fca9c0b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:06.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.561+0000 7fca9b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c0cc480 0x7fca9c0b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38430/0 (socket says 172.21.15.40:38430) 2024-04-26T23:10:06.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.561+0000 7fca9b400640 1 -- 172.21.15.40:0/3534861145 learned_addr learned my addr 172.21.15.40:0/3534861145 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:06.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.561+0000 7fca9aa00640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca9c06fe20 0x7fca9c0d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:06.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.561+0000 7fca9a000640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca9c071e60 0x7fca9c0b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:06.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.562+0000 7fca9b400640 1 -- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca9c06fe20 msgr2=0x7fca9c0d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:06.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.562+0000 7fca9b400640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca9c06fe20 0x7fca9c0d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.562+0000 7fca9b400640 1 -- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca9c071e60 msgr2=0x7fca9c0b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:06.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.562+0000 7fca9b400640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca9c071e60 0x7fca9c0b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.562+0000 7fca9b400640 1 -- 172.21.15.40:0/3534861145 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca9c0aee40 con 0x7fca9c0cc480 2024-04-26T23:10:06.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.562+0000 7fca9a000640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca9c071e60 0x7fca9c0b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:10:06.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.562+0000 7fca9b400640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c0cc480 0x7fca9c0b3dc0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fca94009eb0 tx=0x7fca94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:06.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.562+0000 7fca93400640 1 -- 172.21.15.40:0/3534861145 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9403f070 con 0x7fca9c0cc480 2024-04-26T23:10:06.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.562+0000 7fca93400640 1 -- 172.21.15.40:0/3534861145 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca94031bc0 con 0x7fca9c0cc480 2024-04-26T23:10:06.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.562+0000 7fca93400640 1 -- 172.21.15.40:0/3534861145 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94009450 con 0x7fca9c0cc480 2024-04-26T23:10:06.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.562+0000 7fcaa1e00640 1 -- 172.21.15.40:0/3534861145 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca9c0af120 con 0x7fca9c0cc480 2024-04-26T23:10:06.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.563+0000 7fcaa1e00640 1 -- 172.21.15.40:0/3534861145 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca9c0af6e0 con 0x7fca9c0cc480 2024-04-26T23:10:06.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.564+0000 7fcaa1e00640 1 -- 172.21.15.40:0/3534861145 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca9c0694f0 con 0x7fca9c0cc480 2024-04-26T23:10:06.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.565+0000 7fca93400640 1 -- 172.21.15.40:0/3534861145 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fca9403b410 con 0x7fca9c0cc480 2024-04-26T23:10:06.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.565+0000 7fca93400640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fca7c06c8f0 0x7fca7c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:06.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.566+0000 7fca93400640 1 -- 172.21.15.40:0/3534861145 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7fca940ad230 con 0x7fca9c0cc480 2024-04-26T23:10:06.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.566+0000 7fca9aa00640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fca7c06c8f0 0x7fca7c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:06.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.567+0000 7fca9aa00640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fca7c06c8f0 0x7fca7c06edb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fca8c006fd0 tx=0x7fca8c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:06.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.570+0000 7fca93400640 1 -- 172.21.15.40:0/3534861145 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fca9404b030 con 0x7fca9c0cc480 2024-04-26T23:10:06.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.776+0000 7fcaa1e00640 1 -- 172.21.15.40:0/3534861145 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fca9c072520 con 0x7fca9c0cc480 2024-04-26T23:10:06.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.776+0000 7fca93400640 1 -- 172.21.15.40:0/3534861145 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v105) ==== 100+0+11 (secure 0 0 0) 0x7fca9407ad30 con 0x7fca9c0cc480 2024-04-26T23:10:06.777 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:10:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.779+0000 7fcaa1e00640 1 -- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fca7c06c8f0 msgr2=0x7fca7c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.779+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fca7c06c8f0 0x7fca7c06edb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fca8c006fd0 tx=0x7fca8c008040 comp rx=0 tx=0).stop 2024-04-26T23:10:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.779+0000 7fcaa1e00640 1 -- 172.21.15.40:0/3534861145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c0cc480 msgr2=0x7fca9c0b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.779+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c0cc480 0x7fca9c0b3dc0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fca94009eb0 tx=0x7fca94004a40 comp rx=0 tx=0).stop 2024-04-26T23:10:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.779+0000 7fcaa1e00640 1 -- 172.21.15.40:0/3534861145 shutdown_connections 2024-04-26T23:10:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.779+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fca7c06c8f0 0x7fca7c06edb0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.779+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca9c0cc480 0x7fca9c0b3dc0 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.779+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca9c071e60 0x7fca9c0b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.779+0000 7fcaa1e00640 1 --2- 172.21.15.40:0/3534861145 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca9c06fe20 0x7fca9c0d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:06.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.779+0000 7fcaa1e00640 1 -- 172.21.15.40:0/3534861145 >> 172.21.15.40:0/3534861145 conn(0x7fca9c0d0be0 msgr2=0x7fca9c159f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:06.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.780+0000 7fcaa1e00640 1 -- 172.21.15.40:0/3534861145 shutdown_connections 2024-04-26T23:10:06.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:06.780+0000 7fcaa1e00640 1 -- 172.21.15.40:0/3534861145 wait complete. 2024-04-26T23:10:06.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:10:07.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.013+0000 7f589e000640 1 Processor -- start 2024-04-26T23:10:07.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.013+0000 7f589e000640 1 -- start start 2024-04-26T23:10:07.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.015+0000 7f589e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f58980b4270 0x7f58980af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.015+0000 7f589e000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589815f6e0 con 0x7f589815f300 2024-04-26T23:10:07.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.015+0000 7f589e000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58980d2020 con 0x7f58980aff10 2024-04-26T23:10:07.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.015+0000 7f589e000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f58980b3790 con 0x7f58980b4270 2024-04-26T23:10:07.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.015+0000 7f589cc00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f589815f300 0x7f589806d5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51872/0 (socket says 172.21.15.40:51872) 2024-04-26T23:10:07.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.015+0000 7f589cc00640 1 -- 172.21.15.40:0/1425522140 learned_addr learned my addr 172.21.15.40:0/1425522140 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:07.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.015+0000 7f5897e00640 1 --2- 172.21.15.40:0/1425522140 >> v2:172.21.15.40:3300/0 conn(0x7f58980b4270 0x7f58980af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.016+0000 7f5897400640 1 -- 172.21.15.40:0/1425522140 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660868612 0 0) 0x7f589815f6e0 con 0x7f589815f300 2024-04-26T23:10:07.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.016+0000 7f5897400640 1 -- 172.21.15.40:0/1425522140 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5880003520 con 0x7f589815f300 2024-04-26T23:10:07.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.016+0000 7f5897e00640 1 -- 172.21.15.40:0/1425522140 >> v1:172.21.15.103:6789/0 conn(0x7f58980aff10 legacy=0x7f58980d0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.016+0000 7f5897e00640 1 -- 172.21.15.40:0/1425522140 >> v1:172.21.15.40:6789/0 conn(0x7f589815f300 legacy=0x7f589806d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.016+0000 7f5897e00640 1 -- 172.21.15.40:0/1425522140 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58980b3a70 con 0x7f58980b4270 2024-04-26T23:10:07.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.017+0000 7f5897e00640 1 --2- 172.21.15.40:0/1425522140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58980b4270 0x7f58980af9d0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f58840097a0 tx=0x7f588402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aacb18b74c5d8647 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:07.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.018+0000 7f5897400640 1 -- 172.21.15.40:0/1425522140 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5884037070 con 0x7f58980b4270 2024-04-26T23:10:07.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.018+0000 7f5897400640 1 -- 172.21.15.40:0/1425522140 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5884032440 con 0x7f58980b4270 2024-04-26T23:10:07.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.018+0000 7f5897400640 1 -- 172.21.15.40:0/1425522140 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5884032810 con 0x7f58980b4270 2024-04-26T23:10:07.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.018+0000 7f589e000640 1 -- 172.21.15.40:0/1425522140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58980b4270 msgr2=0x7f58980af9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.018+0000 7f589e000640 1 --2- 172.21.15.40:0/1425522140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58980b4270 0x7f58980af9d0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f58840097a0 tx=0x7f588402a1a0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.018+0000 7f589e000640 1 -- 172.21.15.40:0/1425522140 shutdown_connections 2024-04-26T23:10:07.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.018+0000 7f589e000640 1 --2- 172.21.15.40:0/1425522140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58980b4270 0x7f58980af9d0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.018+0000 7f589e000640 1 -- 172.21.15.40:0/1425522140 >> 172.21.15.40:0/1425522140 conn(0x7f58980d2ca0 msgr2=0x7f58980d30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:07.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.018+0000 7f589e000640 1 -- 172.21.15.40:0/1425522140 shutdown_connections 2024-04-26T23:10:07.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.018+0000 7f589e000640 1 -- 172.21.15.40:0/1425522140 wait complete. 2024-04-26T23:10:07.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.020+0000 7f589e000640 1 Processor -- start 2024-04-26T23:10:07.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.020+0000 7f589e000640 1 -- start start 2024-04-26T23:10:07.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.021+0000 7f589e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58980aff10 0x7f5898158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.021+0000 7f589e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58980b4270 0x7f5898160510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.021+0000 7f589e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f589815f300 0x7f5898160a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.021+0000 7f589e000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58980b04a0 con 0x7f58980aff10 2024-04-26T23:10:07.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.021+0000 7f589e000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f58980b0320 con 0x7f589815f300 2024-04-26T23:10:07.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.021+0000 7f589e000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f58980b0620 con 0x7f58980b4270 2024-04-26T23:10:07.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.022+0000 7f589cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58980aff10 0x7f5898158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.022+0000 7f589cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58980aff10 0x7f5898158520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38450/0 (socket says 172.21.15.40:38450) 2024-04-26T23:10:07.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.022+0000 7f589cc00640 1 -- 172.21.15.40:0/2818311926 learned_addr learned my addr 172.21.15.40:0/2818311926 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.022+0000 7f5897e00640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58980b4270 0x7f5898160510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.022+0000 7f589d600640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f589815f300 0x7f5898160a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.022+0000 7f589cc00640 1 -- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58980b4270 msgr2=0x7f5898160510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.022+0000 7f589cc00640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58980b4270 0x7f5898160510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.022+0000 7f589cc00640 1 -- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f589815f300 msgr2=0x7f5898160a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.022+0000 7f589cc00640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f589815f300 0x7f5898160a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.022+0000 7f589cc00640 1 -- 172.21.15.40:0/2818311926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58980c86e0 con 0x7f58980aff10 2024-04-26T23:10:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.022+0000 7f589d600640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f589815f300 0x7f5898160a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:07.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.022+0000 7f589cc00640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58980aff10 0x7f5898158520 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f588c00c410 tx=0x7f588c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:07.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.023+0000 7f5894c00640 1 -- 172.21.15.40:0/2818311926 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c017070 con 0x7f58980aff10 2024-04-26T23:10:07.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.023+0000 7f5894c00640 1 -- 172.21.15.40:0/2818311926 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588c005ce0 con 0x7f58980aff10 2024-04-26T23:10:07.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.023+0000 7f5894c00640 1 -- 172.21.15.40:0/2818311926 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c0072f0 con 0x7f58980aff10 2024-04-26T23:10:07.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.023+0000 7f589e000640 1 -- 172.21.15.40:0/2818311926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f589815fa30 con 0x7f58980aff10 2024-04-26T23:10:07.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.023+0000 7f589e000640 1 -- 172.21.15.40:0/2818311926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f589815fff0 con 0x7f58980aff10 2024-04-26T23:10:07.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.024+0000 7f589e000640 1 -- 172.21.15.40:0/2818311926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5898069a80 con 0x7f58980aff10 2024-04-26T23:10:07.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.025+0000 7f5894c00640 1 -- 172.21.15.40:0/2818311926 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f588c01a070 con 0x7f58980aff10 2024-04-26T23:10:07.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.026+0000 7f5894c00640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f587806c8f0 0x7f587806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.026+0000 7f5894c00640 1 -- 172.21.15.40:0/2818311926 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f588c08b1a0 con 0x7f58980aff10 2024-04-26T23:10:07.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.027+0000 7f5897e00640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f587806c8f0 0x7f587806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.028+0000 7f5897e00640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f587806c8f0 0x7f587806edb0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5884034000 tx=0x7f588402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:07.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.031+0000 7f5894c00640 1 -- 172.21.15.40:0/2818311926 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f588c0594d0 con 0x7f58980aff10 2024-04-26T23:10:07.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.270+0000 7f589e000640 1 -- 172.21.15.40:0/2818311926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f589806df50 con 0x7f58980aff10 2024-04-26T23:10:07.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.270+0000 7f5894c00640 1 -- 172.21.15.40:0/2818311926 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5239 (secure 0 0 0) 0x7f588c058dd0 con 0x7f58980aff10 2024-04-26T23:10:07.271 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:07.271 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:44.797258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[6598,8949,8955]},"id":7}]} 2024-04-26T23:10:07.271 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 122 2024-04-26T23:10:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.273+0000 7f589e000640 1 -- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f587806c8f0 msgr2=0x7f587806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.273+0000 7f589e000640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f587806c8f0 0x7f587806edb0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5884034000 tx=0x7f588402e000 comp rx=0 tx=0).stop 2024-04-26T23:10:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.273+0000 7f589e000640 1 -- 172.21.15.40:0/2818311926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58980aff10 msgr2=0x7f5898158520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.273+0000 7f589e000640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58980aff10 0x7f5898158520 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f588c00c410 tx=0x7f588c00c8d0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.274+0000 7f589e000640 1 -- 172.21.15.40:0/2818311926 shutdown_connections 2024-04-26T23:10:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.274+0000 7f589e000640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f587806c8f0 0x7f587806edb0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.274+0000 7f589e000640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f589815f300 0x7f5898160a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.274+0000 7f589e000640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58980b4270 0x7f5898160510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.274+0000 7f589e000640 1 --2- 172.21.15.40:0/2818311926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58980aff10 0x7f5898158520 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.274+0000 7f589e000640 1 -- 172.21.15.40:0/2818311926 >> 172.21.15.40:0/2818311926 conn(0x7f58980d2ca0 msgr2=0x7f58980b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.274+0000 7f589e000640 1 -- 172.21.15.40:0/2818311926 shutdown_connections 2024-04-26T23:10:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.274+0000 7f589e000640 1 -- 172.21.15.40:0/2818311926 wait complete. 2024-04-26T23:10:07.292 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:10:07.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.495+0000 7f32d4a00640 1 Processor -- start 2024-04-26T23:10:07.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.496+0000 7f32d4a00640 1 -- start start 2024-04-26T23:10:07.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.497+0000 7f32d4a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f32d00d3740 0x7f32d00b9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.497+0000 7f32d4a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d0162c40 con 0x7f32d00b6220 2024-04-26T23:10:07.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.497+0000 7f32d4a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d00d2d80 con 0x7f32d00d32e0 2024-04-26T23:10:07.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.497+0000 7f32d4a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f32d00b7c80 con 0x7f32d00d3740 2024-04-26T23:10:07.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.497+0000 7f32ce800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f32d00b6220 0x7f32d00d0390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51874/0 (socket says 172.21.15.40:51874) 2024-04-26T23:10:07.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.497+0000 7f32ce800640 1 -- 172.21.15.40:0/2185461434 learned_addr learned my addr 172.21.15.40:0/2185461434 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:07.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1405504593 0 0) 0x7f32d0162c40 con 0x7f32d00b6220 2024-04-26T23:10:07.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32b0003520 con 0x7f32d00b6220 2024-04-26T23:10:07.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cf200640 1 --2- 172.21.15.40:0/2185461434 >> v2:172.21.15.103:3300/0 conn(0x7f32d00d3740 0x7f32d00b9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3397998725 0 0) 0x7f32b0003520 con 0x7f32d00b6220 2024-04-26T23:10:07.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 >> v2:172.21.15.103:3300/0 conn(0x7f32d00d3740 msgr2=0x7f32d00b9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:07.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 --2- 172.21.15.40:0/2185461434 >> v2:172.21.15.103:3300/0 conn(0x7f32d00d3740 0x7f32d00b9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 >> v1:172.21.15.103:6789/0 conn(0x7f32d00d32e0 legacy=0x7f32d00b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d00b7f60 con 0x7f32d00b6220 2024-04-26T23:10:07.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f32c0001670 con 0x7f32d00b6220 2024-04-26T23:10:07.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cf200640 1 --2- 172.21.15.40:0/2185461434 >> v2:172.21.15.103:3300/0 conn(0x7f32d00d3740 0x7f32d00b9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:10:07.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 >> v1:172.21.15.40:6789/0 conn(0x7f32d00b6220 legacy=0x7f32d00d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 --2- 172.21.15.40:0/2185461434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b0003a00 0x7f32b0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 --2- 172.21.15.40:0/2185461434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b0004800 0x7f32b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 --2- 172.21.15.40:0/2185461434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b0005210 0x7f32b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32d00b7c80 con 0x7f32b0005210 2024-04-26T23:10:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32b000c160 con 0x7f32b0003a00 2024-04-26T23:10:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.498+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f32b000c360 con 0x7f32b0004800 2024-04-26T23:10:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.499+0000 7f32cf200640 1 --2- 172.21.15.40:0/2185461434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b0005210 0x7f32b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.499+0000 7f32cde00640 1 --2- 172.21.15.40:0/2185461434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b0004800 0x7f32b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.499+0000 7f32ce800640 1 --2- 172.21.15.40:0/2185461434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b0003a00 0x7f32b0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.499+0000 7f32cf200640 1 -- 172.21.15.40:0/2185461434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b0004800 msgr2=0x7f32b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.499+0000 7f32cf200640 1 --2- 172.21.15.40:0/2185461434 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32b0004800 0x7f32b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.499+0000 7f32cf200640 1 -- 172.21.15.40:0/2185461434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b0003a00 msgr2=0x7f32b0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.499+0000 7f32cf200640 1 --2- 172.21.15.40:0/2185461434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b0003a00 0x7f32b0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.499+0000 7f32cf200640 1 -- 172.21.15.40:0/2185461434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b0003520 con 0x7f32b0005210 2024-04-26T23:10:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.499+0000 7f32ce800640 1 --2- 172.21.15.40:0/2185461434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b0003a00 0x7f32b0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.500+0000 7f32cf200640 1 -- 172.21.15.40:0/2185461434 reap_dead start 2024-04-26T23:10:07.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.500+0000 7f32cf200640 1 --2- 172.21.15.40:0/2185461434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b0005210 0x7f32b000b9c0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f32c80073e0 tx=0x7f32b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf9d9617a77831f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:07.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.500+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8009c80 con 0x7f32b0005210 2024-04-26T23:10:07.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.501+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32c802c680 con 0x7f32b0005210 2024-04-26T23:10:07.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.501+0000 7f32cd400640 1 -- 172.21.15.40:0/2185461434 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c802ca50 con 0x7f32b0005210 2024-04-26T23:10:07.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.501+0000 7f32d4a00640 1 -- 172.21.15.40:0/2185461434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b0005210 msgr2=0x7f32b000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.501+0000 7f32d4a00640 1 --2- 172.21.15.40:0/2185461434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b0005210 0x7f32b000b9c0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f32c80073e0 tx=0x7f32b0005830 comp rx=0 tx=0).stop 2024-04-26T23:10:07.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.501+0000 7f32d4a00640 1 -- 172.21.15.40:0/2185461434 shutdown_connections 2024-04-26T23:10:07.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.501+0000 7f32d4a00640 1 --2- 172.21.15.40:0/2185461434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32b0005210 0x7f32b000b9c0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.501+0000 7f32d4a00640 1 -- 172.21.15.40:0/2185461434 >> 172.21.15.40:0/2185461434 conn(0x7f32d00d4d90 msgr2=0x7f32d00d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:07.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.501+0000 7f32d4a00640 1 -- 172.21.15.40:0/2185461434 shutdown_connections 2024-04-26T23:10:07.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.501+0000 7f32d4a00640 1 -- 172.21.15.40:0/2185461434 wait complete. 2024-04-26T23:10:07.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.503+0000 7f32d4a00640 1 Processor -- start 2024-04-26T23:10:07.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.503+0000 7f32d4a00640 1 -- start start 2024-04-26T23:10:07.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.504+0000 7f32d4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b0005210 0x7f32d00c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.504+0000 7f32d4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32d00c8830 0x7f32d015d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.504+0000 7f32d4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32d015e5e0 0x7f32d0152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.504+0000 7f32d4a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32d00d3b80 con 0x7f32d00c8830 2024-04-26T23:10:07.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.504+0000 7f32d4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32d00b8520 con 0x7f32b0005210 2024-04-26T23:10:07.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.504+0000 7f32d4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f32d00d0390 con 0x7f32d015e5e0 2024-04-26T23:10:07.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.504+0000 7f32cde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32d00c8830 0x7f32d015d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.504+0000 7f32cde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32d00c8830 0x7f32d015d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38470/0 (socket says 172.21.15.40:38470) 2024-04-26T23:10:07.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.504+0000 7f32cde00640 1 -- 172.21.15.40:0/3684165786 learned_addr learned my addr 172.21.15.40:0/3684165786 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:07.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.505+0000 7f32cf200640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32d015e5e0 0x7f32d0152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.505+0000 7f32ce800640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b0005210 0x7f32d00c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.505+0000 7f32cde00640 1 -- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32d015e5e0 msgr2=0x7f32d0152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.505+0000 7f32cde00640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32d015e5e0 0x7f32d0152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.505+0000 7f32cde00640 1 -- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b0005210 msgr2=0x7f32d00c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.505+0000 7f32cde00640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b0005210 0x7f32d00c9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.505+0000 7f32cde00640 1 -- 172.21.15.40:0/3684165786 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d00ca2f0 con 0x7f32d00c8830 2024-04-26T23:10:07.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.505+0000 7f32cde00640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32d00c8830 0x7f32d015d680 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f32b800e4d0 tx=0x7f32b800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:07.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.505+0000 7f32c6a00640 1 -- 172.21.15.40:0/3684165786 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b80052e0 con 0x7f32d00c8830 2024-04-26T23:10:07.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.506+0000 7f32c6a00640 1 -- 172.21.15.40:0/3684165786 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32b8018070 con 0x7f32d00c8830 2024-04-26T23:10:07.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.506+0000 7f32c6a00640 1 -- 172.21.15.40:0/3684165786 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8003810 con 0x7f32d00c8830 2024-04-26T23:10:07.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.506+0000 7f32d4a00640 1 -- 172.21.15.40:0/3684165786 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32d00c8d10 con 0x7f32d00c8830 2024-04-26T23:10:07.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.506+0000 7f32d4a00640 1 -- 172.21.15.40:0/3684165786 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32d00c92d0 con 0x7f32d00c8830 2024-04-26T23:10:07.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.507+0000 7f32d4a00640 1 -- 172.21.15.40:0/3684165786 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32d0069370 con 0x7f32d00c8830 2024-04-26T23:10:07.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.508+0000 7f32c6a00640 1 -- 172.21.15.40:0/3684165786 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f32b80039b0 con 0x7f32d00c8830 2024-04-26T23:10:07.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.509+0000 7f32c6a00640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f32ac06c8f0 0x7f32ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.509+0000 7f32c6a00640 1 -- 172.21.15.40:0/3684165786 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f32b808ba50 con 0x7f32d00c8830 2024-04-26T23:10:07.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.509+0000 7f32ce800640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f32ac06c8f0 0x7f32ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.511+0000 7f32ce800640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f32ac06c8f0 0x7f32ac06edb0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f32c00017e0 tx=0x7f32c0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:07.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.513+0000 7f32c6a00640 1 -- 172.21.15.40:0/3684165786 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f32b8059c30 con 0x7f32d00c8830 2024-04-26T23:10:07.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.729+0000 7f32d4a00640 1 -- 172.21.15.40:0/3684165786 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32d00d4a70 con 0x7f32d00c8830 2024-04-26T23:10:07.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.731+0000 7f32c6a00640 1 -- 172.21.15.40:0/3684165786 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+16898 (secure 0 0 0) 0x7f32b8059530 con 0x7f32d00c8830 2024-04-26T23:10:07.731 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:07.732 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":105,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:09:49.407703+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:09:03.955148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-26T23:09:05.070627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.2a","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:10:07.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.733+0000 7f32d4a00640 1 -- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f32ac06c8f0 msgr2=0x7f32ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.733+0000 7f32d4a00640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f32ac06c8f0 0x7f32ac06edb0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f32c00017e0 tx=0x7f32c0009040 comp rx=0 tx=0).stop 2024-04-26T23:10:07.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.733+0000 7f32d4a00640 1 -- 172.21.15.40:0/3684165786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32d00c8830 msgr2=0x7f32d015d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.733+0000 7f32d4a00640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32d00c8830 0x7f32d015d680 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f32b800e4d0 tx=0x7f32b800e990 comp rx=0 tx=0).stop 2024-04-26T23:10:07.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.734+0000 7f32d4a00640 1 -- 172.21.15.40:0/3684165786 shutdown_connections 2024-04-26T23:10:07.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.734+0000 7f32d4a00640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f32ac06c8f0 0x7f32ac06edb0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.734+0000 7f32d4a00640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32d015e5e0 0x7f32d0152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.734+0000 7f32d4a00640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32d00c8830 0x7f32d015d680 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.734+0000 7f32d4a00640 1 --2- 172.21.15.40:0/3684165786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32b0005210 0x7f32d00c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.734+0000 7f32d4a00640 1 -- 172.21.15.40:0/3684165786 >> 172.21.15.40:0/3684165786 conn(0x7f32d00d4d90 msgr2=0x7f32d01623b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:07.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.734+0000 7f32d4a00640 1 -- 172.21.15.40:0/3684165786 shutdown_connections 2024-04-26T23:10:07.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.734+0000 7f32d4a00640 1 -- 172.21.15.40:0/3684165786 wait complete. 2024-04-26T23:10:07.753 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T23:10:07.753 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:10:07.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.968+0000 7fd28b200640 1 Processor -- start 2024-04-26T23:10:07.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.968+0000 7fd28b200640 1 -- start start 2024-04-26T23:10:07.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.968+0000 7fd28b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd28c0d0d80 0x7fd28c0cb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.968+0000 7fd28b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd28c0d0200 0x7fd28c0c7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.968+0000 7fd28b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd28c0b2590 con 0x7fd28c0cd610 2024-04-26T23:10:07.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.968+0000 7fd28b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd28c0d20e0 con 0x7fd28c0d0d80 2024-04-26T23:10:07.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.968+0000 7fd28b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd28c0d1e00 con 0x7fd28c0d0200 2024-04-26T23:10:07.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.969+0000 7fd283e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd28c0d0d80 0x7fd28c0cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.969+0000 7fd28a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd28c0d0200 0x7fd28c0c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.969+0000 7fd289e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd28c0cd610 0x7fd28c0b1e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:33114/0 (socket says 172.21.15.40:33114) 2024-04-26T23:10:07.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.969+0000 7fd289e00640 1 -- 172.21.15.40:0/2409028020 learned_addr learned my addr 172.21.15.40:0/2409028020 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:07.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.970+0000 7fd283e00640 1 -- 172.21.15.40:0/2409028020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd28c0d0200 msgr2=0x7fd28c0c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:07.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.970+0000 7fd283e00640 1 --2- 172.21.15.40:0/2409028020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd28c0d0200 0x7fd28c0c7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.970+0000 7fd283e00640 1 -- 172.21.15.40:0/2409028020 >> v1:172.21.15.103:6789/0 conn(0x7fd28c0cd610 legacy=0x7fd28c0b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.970+0000 7fd283e00640 1 -- 172.21.15.40:0/2409028020 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28c075e70 con 0x7fd28c0d0d80 2024-04-26T23:10:07.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.970+0000 7fd28a800640 1 --2- 172.21.15.40:0/2409028020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd28c0d0200 0x7fd28c0c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:10:07.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.971+0000 7fd283e00640 1 --2- 172.21.15.40:0/2409028020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd28c0d0d80 0x7fd28c0cb890 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd2740097a0 tx=0x7fd27402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7b09eb87a491471 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.972+0000 7fd289400640 1 -- 172.21.15.40:0/2409028020 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274037070 con 0x7fd28c0d0d80 2024-04-26T23:10:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.972+0000 7fd289400640 1 -- 172.21.15.40:0/2409028020 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd274032440 con 0x7fd28c0d0d80 2024-04-26T23:10:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.972+0000 7fd289400640 1 -- 172.21.15.40:0/2409028020 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274032810 con 0x7fd28c0d0d80 2024-04-26T23:10:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.973+0000 7fd28b200640 1 -- 172.21.15.40:0/2409028020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd28c0d0d80 msgr2=0x7fd28c0cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.973+0000 7fd28b200640 1 --2- 172.21.15.40:0/2409028020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd28c0d0d80 0x7fd28c0cb890 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd2740097a0 tx=0x7fd27402a180 comp rx=0 tx=0).stop 2024-04-26T23:10:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.973+0000 7fd28b200640 1 -- 172.21.15.40:0/2409028020 shutdown_connections 2024-04-26T23:10:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.973+0000 7fd28b200640 1 --2- 172.21.15.40:0/2409028020 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd28c0d0200 0x7fd28c0c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.973+0000 7fd28b200640 1 --2- 172.21.15.40:0/2409028020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd28c0d0d80 0x7fd28c0cb890 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.973+0000 7fd28b200640 1 -- 172.21.15.40:0/2409028020 >> 172.21.15.40:0/2409028020 conn(0x7fd28c0d1240 msgr2=0x7fd28c0d1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.973+0000 7fd28b200640 1 -- 172.21.15.40:0/2409028020 shutdown_connections 2024-04-26T23:10:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.973+0000 7fd28b200640 1 -- 172.21.15.40:0/2409028020 wait complete. 2024-04-26T23:10:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.976+0000 7fd28b200640 1 Processor -- start 2024-04-26T23:10:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.976+0000 7fd28b200640 1 -- start start 2024-04-26T23:10:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd28b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd28c0cd610 0x7fd28c0c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd28b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd28c0d0200 0x7fd28c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd28b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd28c0d0d80 0x7fd28c0b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd28b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c0743b0 con 0x7fd28c0cd610 2024-04-26T23:10:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd28b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c0cd9f0 con 0x7fd28c0d0d80 2024-04-26T23:10:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd28b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd28c074530 con 0x7fd28c0d0200 2024-04-26T23:10:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd289e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd28c0cd610 0x7fd28c0c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd289e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd28c0cd610 0x7fd28c0c91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38482/0 (socket says 172.21.15.40:38482) 2024-04-26T23:10:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd289e00640 1 -- 172.21.15.40:0/3688844987 learned_addr learned my addr 172.21.15.40:0/3688844987 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd28a800640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd28c0d0d80 0x7fd28c0b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd283e00640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd28c0d0200 0x7fd28c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd289e00640 1 -- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd28c0d0200 msgr2=0x7fd28c0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd289e00640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd28c0d0200 0x7fd28c0b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd289e00640 1 -- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd28c0d0d80 msgr2=0x7fd28c0b0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:07.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd289e00640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd28c0d0d80 0x7fd28c0b0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:07.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd289e00640 1 -- 172.21.15.40:0/3688844987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28c0b17b0 con 0x7fd28c0cd610 2024-04-26T23:10:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd289e00640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd28c0cd610 0x7fd28c0c91e0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00c3c0 tx=0x7fd27c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.978+0000 7fd282000640 1 -- 172.21.15.40:0/3688844987 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c017070 con 0x7fd28c0cd610 2024-04-26T23:10:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.979+0000 7fd282000640 1 -- 172.21.15.40:0/3688844987 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27c005ce0 con 0x7fd28c0cd610 2024-04-26T23:10:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.979+0000 7fd28b200640 1 -- 172.21.15.40:0/3688844987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd28c0c8420 con 0x7fd28c0cd610 2024-04-26T23:10:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.979+0000 7fd282000640 1 -- 172.21.15.40:0/3688844987 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c0072f0 con 0x7fd28c0cd610 2024-04-26T23:10:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.979+0000 7fd28b200640 1 -- 172.21.15.40:0/3688844987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd28c0c89e0 con 0x7fd28c0cd610 2024-04-26T23:10:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.980+0000 7fd28b200640 1 -- 172.21.15.40:0/3688844987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd28c069370 con 0x7fd28c0cd610 2024-04-26T23:10:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.981+0000 7fd282000640 1 -- 172.21.15.40:0/3688844987 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd27c01a070 con 0x7fd28c0cd610 2024-04-26T23:10:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.982+0000 7fd282000640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd26806c8f0 0x7fd26806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.982+0000 7fd282000640 1 -- 172.21.15.40:0/3688844987 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7fd27c08b0f0 con 0x7fd28c0cd610 2024-04-26T23:10:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.982+0000 7fd283e00640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd26806c8f0 0x7fd26806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.985+0000 7fd283e00640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd26806c8f0 0x7fd26806edb0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd274034000 tx=0x7fd27402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:07.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:07.986+0000 7fd282000640 1 -- 172.21.15.40:0/3688844987 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fd28c0b2590 con 0x7fd28c0cd610 2024-04-26T23:10:08.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.229+0000 7fd28b200640 1 -- 172.21.15.40:0/3688844987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd28c0c63b0 con 0x7fd28c0cd610 2024-04-26T23:10:08.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.231+0000 7fd282000640 1 -- 172.21.15.40:0/3688844987 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5239 (secure 0 0 0) 0x7fd27c0592d0 con 0x7fd28c0cd610 2024-04-26T23:10:08.231 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:08.231 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:44.797258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[6598,8949,8955]},"id":7}]} 2024-04-26T23:10:08.231 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 122 2024-04-26T23:10:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.233+0000 7fd28b200640 1 -- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd26806c8f0 msgr2=0x7fd26806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.233+0000 7fd28b200640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd26806c8f0 0x7fd26806edb0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd274034000 tx=0x7fd27402e000 comp rx=0 tx=0).stop 2024-04-26T23:10:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.233+0000 7fd28b200640 1 -- 172.21.15.40:0/3688844987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd28c0cd610 msgr2=0x7fd28c0c91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.233+0000 7fd28b200640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd28c0cd610 0x7fd28c0c91e0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00c3c0 tx=0x7fd27c00c880 comp rx=0 tx=0).stop 2024-04-26T23:10:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.234+0000 7fd28b200640 1 -- 172.21.15.40:0/3688844987 shutdown_connections 2024-04-26T23:10:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.234+0000 7fd28b200640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fd26806c8f0 0x7fd26806edb0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.234+0000 7fd28b200640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd28c0d0d80 0x7fd28c0b0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.234+0000 7fd28b200640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd28c0d0200 0x7fd28c0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.234+0000 7fd28b200640 1 --2- 172.21.15.40:0/3688844987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd28c0cd610 0x7fd28c0c91e0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.234+0000 7fd28b200640 1 -- 172.21.15.40:0/3688844987 >> 172.21.15.40:0/3688844987 conn(0x7fd28c0d1240 msgr2=0x7fd28c0d1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:08.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.234+0000 7fd28b200640 1 -- 172.21.15.40:0/3688844987 shutdown_connections 2024-04-26T23:10:08.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.234+0000 7fd28b200640 1 -- 172.21.15.40:0/3688844987 wait complete. 2024-04-26T23:10:08.254 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:10:08.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.457+0000 7fc05e800640 1 Processor -- start 2024-04-26T23:10:08.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.457+0000 7fc05e800640 1 -- start start 2024-04-26T23:10:08.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.458+0000 7fc05e800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc0580d2c20 0x7fc0580d1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:08.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.460+0000 7fc05e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc05815f430 0x7fc0580d3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:08.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.460+0000 7fc05e800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0580cfa90 con 0x7fc05815f050 2024-04-26T23:10:08.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.460+0000 7fc05e800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc0580d5ad0 con 0x7fc05815f430 2024-04-26T23:10:08.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.460+0000 7fc05e800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc0580cfd30 con 0x7fc0580d2c20 2024-04-26T23:10:08.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.460+0000 7fc05de00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc05815f430 0x7fc0580d3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:08.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.460+0000 7fc05de00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc05815f430 0x7fc0580d3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38488/0 (socket says 172.21.15.40:38488) 2024-04-26T23:10:08.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.460+0000 7fc05de00640 1 -- 172.21.15.40:0/3773387935 learned_addr learned my addr 172.21.15.40:0/3773387935 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:08.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.460+0000 7fc05ca00640 1 --2- 172.21.15.40:0/3773387935 >> v2:172.21.15.103:3300/0 conn(0x7fc0580d2c20 0x7fc0580d1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:08.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.460+0000 7fc05de00640 1 -- 172.21.15.40:0/3773387935 >> v2:172.21.15.103:3300/0 conn(0x7fc0580d2c20 msgr2=0x7fc0580d1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:08.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.460+0000 7fc05de00640 1 --2- 172.21.15.40:0/3773387935 >> v2:172.21.15.103:3300/0 conn(0x7fc0580d2c20 0x7fc0580d1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.460+0000 7fc05de00640 1 -- 172.21.15.40:0/3773387935 >> v1:172.21.15.103:6789/0 conn(0x7fc05815f050 legacy=0x7fc0580d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:08.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.460+0000 7fc05de00640 1 -- 172.21.15.40:0/3773387935 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0580d0010 con 0x7fc05815f430 2024-04-26T23:10:08.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.460+0000 7fc05ca00640 1 --2- 172.21.15.40:0/3773387935 >> v2:172.21.15.103:3300/0 conn(0x7fc0580d2c20 0x7fc0580d1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:08.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.461+0000 7fc05de00640 1 --2- 172.21.15.40:0/3773387935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc05815f430 0x7fc0580d3350 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fc05808e270 tx=0x7fc05002a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=778162f7c10d4684 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:08.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.461+0000 7fc057e00640 1 -- 172.21.15.40:0/3773387935 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050037070 con 0x7fc05815f430 2024-04-26T23:10:08.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.461+0000 7fc057e00640 1 -- 172.21.15.40:0/3773387935 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc050032440 con 0x7fc05815f430 2024-04-26T23:10:08.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.461+0000 7fc057e00640 1 -- 172.21.15.40:0/3773387935 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050032810 con 0x7fc05815f430 2024-04-26T23:10:08.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.462+0000 7fc05e800640 1 -- 172.21.15.40:0/3773387935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc05815f430 msgr2=0x7fc0580d3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:08.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.462+0000 7fc05e800640 1 --2- 172.21.15.40:0/3773387935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc05815f430 0x7fc0580d3350 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fc05808e270 tx=0x7fc05002a290 comp rx=0 tx=0).stop 2024-04-26T23:10:08.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.462+0000 7fc05e800640 1 -- 172.21.15.40:0/3773387935 shutdown_connections 2024-04-26T23:10:08.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.462+0000 7fc05e800640 1 --2- 172.21.15.40:0/3773387935 >> v2:172.21.15.103:3300/0 conn(0x7fc0580d2c20 0x7fc0580d1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.462+0000 7fc05e800640 1 --2- 172.21.15.40:0/3773387935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc05815f430 0x7fc0580d3350 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.462+0000 7fc05e800640 1 -- 172.21.15.40:0/3773387935 >> 172.21.15.40:0/3773387935 conn(0x7fc05809e140 msgr2=0x7fc0580d4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:08.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.462+0000 7fc05e800640 1 -- 172.21.15.40:0/3773387935 shutdown_connections 2024-04-26T23:10:08.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.462+0000 7fc05e800640 1 -- 172.21.15.40:0/3773387935 wait complete. 2024-04-26T23:10:08.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.464+0000 7fc05e800640 1 Processor -- start 2024-04-26T23:10:08.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.465+0000 7fc05e800640 1 -- start start 2024-04-26T23:10:08.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0580d2c20 0x7fc0580cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:08.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc05815f050 0x7fc0580cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:08.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc05815f430 0x7fc0580b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:08.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05e800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc0580d5870 con 0x7fc05815f050 2024-04-26T23:10:08.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05e800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc0580d56f0 con 0x7fc0580d2c20 2024-04-26T23:10:08.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05e800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc0580d3890 con 0x7fc05815f430 2024-04-26T23:10:08.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc05815f050 0x7fc0580cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:08.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc05815f050 0x7fc0580cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38492/0 (socket says 172.21.15.40:38492) 2024-04-26T23:10:08.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05ca00640 1 -- 172.21.15.40:0/794794036 learned_addr learned my addr 172.21.15.40:0/794794036 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05d400640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0580d2c20 0x7fc0580cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05de00640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc05815f430 0x7fc0580b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05ca00640 1 -- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc05815f430 msgr2=0x7fc0580b6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05ca00640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc05815f430 0x7fc0580b6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05ca00640 1 -- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0580d2c20 msgr2=0x7fc0580cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05ca00640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0580d2c20 0x7fc0580cb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05ca00640 1 -- 172.21.15.40:0/794794036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0580ba290 con 0x7fc05815f050 2024-04-26T23:10:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05d400640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0580d2c20 0x7fc0580cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:08.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05ca00640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc05815f050 0x7fc0580cbf10 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00cf60 tx=0x7fc04c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:08.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc055600640 1 -- 172.21.15.40:0/794794036 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c01a070 con 0x7fc05815f050 2024-04-26T23:10:08.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc055600640 1 -- 172.21.15.40:0/794794036 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc04c00d840 con 0x7fc05815f050 2024-04-26T23:10:08.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc055600640 1 -- 172.21.15.40:0/794794036 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c00dc10 con 0x7fc05815f050 2024-04-26T23:10:08.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.467+0000 7fc05e800640 1 -- 172.21.15.40:0/794794036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0580ba570 con 0x7fc05815f050 2024-04-26T23:10:08.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.468+0000 7fc05e800640 1 -- 172.21.15.40:0/794794036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0580cc450 con 0x7fc05815f050 2024-04-26T23:10:08.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.469+0000 7fc05e800640 1 -- 172.21.15.40:0/794794036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc058069370 con 0x7fc05815f050 2024-04-26T23:10:08.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.470+0000 7fc055600640 1 -- 172.21.15.40:0/794794036 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc04c01d070 con 0x7fc05815f050 2024-04-26T23:10:08.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.472+0000 7fc055600640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc03c06c8f0 0x7fc03c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:08.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.472+0000 7fc055600640 1 -- 172.21.15.40:0/794794036 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7fc04c08c460 con 0x7fc05815f050 2024-04-26T23:10:08.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.472+0000 7fc05d400640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc03c06c8f0 0x7fc03c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:08.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.473+0000 7fc05d400640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc03c06c8f0 0x7fc03c06edb0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc048003eb0 tx=0x7fc048009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:08.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.475+0000 7fc055600640 1 -- 172.21.15.40:0/794794036 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc04c05a080 con 0x7fc05815f050 2024-04-26T23:10:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.725+0000 7fc05e800640 1 -- 172.21.15.40:0/794794036 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0580d2450 con 0x7fc05815f050 2024-04-26T23:10:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.727+0000 7fc055600640 1 -- 172.21.15.40:0/794794036 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5239 (secure 0 0 0) 0x7fc04c05e380 con 0x7fc05815f050 2024-04-26T23:10:08.727 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:08.727 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:09:06.116498+0000","modified":"2024-04-26T23:09:44.797258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8955,"mds_1":8949,"mds_2":6598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6598":{"gid":6598,"name":"d","rank":2,"incarnation":115,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/2022020922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":2022020922},{"type":"v1","addr":"172.21.15.103:6837","nonce":2022020922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8949":{"gid":8949,"name":"c","rank":1,"incarnation":113,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3250599834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3250599834},{"type":"v1","addr":"172.21.15.40:6833","nonce":3250599834}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6835/1138762604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1138762604},{"type":"v1","addr":"172.21.15.40:6835","nonce":1138762604}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[6598,8949,8955]},"id":7}]} 2024-04-26T23:10:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 122 2024-04-26T23:10:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.729+0000 7fc05e800640 1 -- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc03c06c8f0 msgr2=0x7fc03c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.729+0000 7fc05e800640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc03c06c8f0 0x7fc03c06edb0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc048003eb0 tx=0x7fc048009040 comp rx=0 tx=0).stop 2024-04-26T23:10:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.729+0000 7fc05e800640 1 -- 172.21.15.40:0/794794036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc05815f050 msgr2=0x7fc0580cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.729+0000 7fc05e800640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc05815f050 0x7fc0580cbf10 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00cf60 tx=0x7fc04c00cf90 comp rx=0 tx=0).stop 2024-04-26T23:10:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.730+0000 7fc05e800640 1 -- 172.21.15.40:0/794794036 shutdown_connections 2024-04-26T23:10:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.730+0000 7fc05e800640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc03c06c8f0 0x7fc03c06edb0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.730+0000 7fc05e800640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc05815f430 0x7fc0580b6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.730+0000 7fc05e800640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc05815f050 0x7fc0580cbf10 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.730+0000 7fc05e800640 1 --2- 172.21.15.40:0/794794036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0580d2c20 0x7fc0580cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.730+0000 7fc05e800640 1 -- 172.21.15.40:0/794794036 >> 172.21.15.40:0/794794036 conn(0x7fc05809e140 msgr2=0x7fc0580b53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:08.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.730+0000 7fc05e800640 1 -- 172.21.15.40:0/794794036 shutdown_connections 2024-04-26T23:10:08.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.730+0000 7fc05e800640 1 -- 172.21.15.40:0/794794036 wait complete. 2024-04-26T23:10:08.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:10:08.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.965+0000 7ff7fd400640 1 Processor -- start 2024-04-26T23:10:08.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.965+0000 7ff7fd400640 1 -- start start 2024-04-26T23:10:08.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.967+0000 7ff7fd400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff7f8162810 0x7ff7f8070350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:08.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.967+0000 7ff7fd400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7f80d1b30 0x7ff7f80d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:08.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.967+0000 7ff7fd400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7f80d10d0 con 0x7ff7f80b2340 2024-04-26T23:10:08.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.967+0000 7ff7fd400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff7f80d0750 con 0x7ff7f8162810 2024-04-26T23:10:08.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.967+0000 7ff7fd400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f80d16a0 con 0x7ff7f80d1b30 2024-04-26T23:10:08.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.967+0000 7ff7efe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff7f80b2340 0x7ff7f80cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51882/0 (socket says 172.21.15.40:51882) 2024-04-26T23:10:08.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.967+0000 7ff7efe00640 1 -- 172.21.15.40:0/2550105212 learned_addr learned my addr 172.21.15.40:0/2550105212 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:08.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.967+0000 7ff7f6800640 1 --2- 172.21.15.40:0/2550105212 >> v2:172.21.15.103:3300/0 conn(0x7ff7f8162810 0x7ff7f8070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:08.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.967+0000 7ff7f7200640 1 --2- 172.21.15.40:0/2550105212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7f80d1b30 0x7ff7f80d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:08.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.968+0000 7ff7f5e00640 1 -- 172.21.15.40:0/2550105212 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 369086576 0 0) 0x7ff7f80d10d0 con 0x7ff7f80b2340 2024-04-26T23:10:08.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.968+0000 7ff7f5e00640 1 -- 172.21.15.40:0/2550105212 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7dc003520 con 0x7ff7f80b2340 2024-04-26T23:10:08.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.968+0000 7ff7f6800640 1 -- 172.21.15.40:0/2550105212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7f80d1b30 msgr2=0x7ff7f80d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:08.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.968+0000 7ff7f6800640 1 --2- 172.21.15.40:0/2550105212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7f80d1b30 0x7ff7f80d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.968+0000 7ff7f6800640 1 -- 172.21.15.40:0/2550105212 >> v1:172.21.15.40:6789/0 conn(0x7ff7f80b2340 legacy=0x7ff7f80cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:08.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.968+0000 7ff7f6800640 1 -- 172.21.15.40:0/2550105212 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f80d1370 con 0x7ff7f8162810 2024-04-26T23:10:08.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.969+0000 7ff7f6800640 1 --2- 172.21.15.40:0/2550105212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f8162810 0x7ff7f8070350 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80097a0 tx=0x7ff7e802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36498d6f54a3abb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:08.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.970+0000 7ff7f5e00640 1 -- 172.21.15.40:0/2550105212 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8037070 con 0x7ff7f8162810 2024-04-26T23:10:08.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.970+0000 7ff7f5e00640 1 -- 172.21.15.40:0/2550105212 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e8032440 con 0x7ff7f8162810 2024-04-26T23:10:08.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.970+0000 7ff7f5e00640 1 -- 172.21.15.40:0/2550105212 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8032810 con 0x7ff7f8162810 2024-04-26T23:10:08.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.971+0000 7ff7fd400640 1 -- 172.21.15.40:0/2550105212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f8162810 msgr2=0x7ff7f8070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:08.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.971+0000 7ff7fd400640 1 --2- 172.21.15.40:0/2550105212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f8162810 0x7ff7f8070350 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80097a0 tx=0x7ff7e802a180 comp rx=0 tx=0).stop 2024-04-26T23:10:08.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.971+0000 7ff7fd400640 1 -- 172.21.15.40:0/2550105212 shutdown_connections 2024-04-26T23:10:08.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.971+0000 7ff7fd400640 1 --2- 172.21.15.40:0/2550105212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7f80d1b30 0x7ff7f80d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.971+0000 7ff7fd400640 1 --2- 172.21.15.40:0/2550105212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f8162810 0x7ff7f8070350 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.971+0000 7ff7fd400640 1 -- 172.21.15.40:0/2550105212 >> 172.21.15.40:0/2550105212 conn(0x7ff7f80d2da0 msgr2=0x7ff7f80d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:08.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.971+0000 7ff7fd400640 1 -- 172.21.15.40:0/2550105212 shutdown_connections 2024-04-26T23:10:08.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.971+0000 7ff7fd400640 1 -- 172.21.15.40:0/2550105212 wait complete. 2024-04-26T23:10:08.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.973+0000 7ff7fd400640 1 Processor -- start 2024-04-26T23:10:08.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.973+0000 7ff7fd400640 1 -- start start 2024-04-26T23:10:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.974+0000 7ff7fd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7f80b2340 0x7ff7f8159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.974+0000 7ff7fd400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f80d1b30 0x7ff7f80c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.974+0000 7ff7fd400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7f8162810 0x7ff7f80c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.974+0000 7ff7fd400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f8070890 con 0x7ff7f80b2340 2024-04-26T23:10:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.974+0000 7ff7fd400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f80d0470 con 0x7ff7f80d1b30 2024-04-26T23:10:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.974+0000 7ff7fd400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f80cdb00 con 0x7ff7f8162810 2024-04-26T23:10:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.974+0000 7ff7f6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7f80b2340 0x7ff7f8159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.974+0000 7ff7f6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7f80b2340 0x7ff7f8159c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38502/0 (socket says 172.21.15.40:38502) 2024-04-26T23:10:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.974+0000 7ff7f6800640 1 -- 172.21.15.40:0/2314294910 learned_addr learned my addr 172.21.15.40:0/2314294910 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.975+0000 7ff7efe00640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f80d1b30 0x7ff7f80c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.975+0000 7ff7f7200640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7f8162810 0x7ff7f80c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.975+0000 7ff7f6800640 1 -- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7f8162810 msgr2=0x7ff7f80c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.975+0000 7ff7f6800640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7f8162810 0x7ff7f80c91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.975+0000 7ff7f6800640 1 -- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f80d1b30 msgr2=0x7ff7f80c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.975+0000 7ff7f6800640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f80d1b30 0x7ff7f80c8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.975+0000 7ff7f6800640 1 -- 172.21.15.40:0/2314294910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f80cc340 con 0x7ff7f80b2340 2024-04-26T23:10:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.975+0000 7ff7f6800640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7f80b2340 0x7ff7f8159c10 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009890 tx=0x7ff7e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.975+0000 7ff7eea00640 1 -- 172.21.15.40:0/2314294910 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8037050 con 0x7ff7f80b2340 2024-04-26T23:10:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.976+0000 7ff7eea00640 1 -- 172.21.15.40:0/2314294910 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e8043070 con 0x7ff7f80b2340 2024-04-26T23:10:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.976+0000 7ff7eea00640 1 -- 172.21.15.40:0/2314294910 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e803cb30 con 0x7ff7f80b2340 2024-04-26T23:10:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.976+0000 7ff7fd400640 1 -- 172.21.15.40:0/2314294910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f80cc620 con 0x7ff7f80b2340 2024-04-26T23:10:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.976+0000 7ff7fd400640 1 -- 172.21.15.40:0/2314294910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f80ccbe0 con 0x7ff7f80b2340 2024-04-26T23:10:08.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.977+0000 7ff7fd400640 1 -- 172.21.15.40:0/2314294910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7f80694f0 con 0x7ff7f80b2340 2024-04-26T23:10:08.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.979+0000 7ff7eea00640 1 -- 172.21.15.40:0/2314294910 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff7e803ccd0 con 0x7ff7f80b2340 2024-04-26T23:10:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.981+0000 7ff7eea00640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7d406c8f0 0x7ff7d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.981+0000 7ff7eea00640 1 -- 172.21.15.40:0/2314294910 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7ff7e80ac9b0 con 0x7ff7f80b2340 2024-04-26T23:10:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.981+0000 7ff7efe00640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7d406c8f0 0x7ff7d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:08.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.982+0000 7ff7efe00640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7d406c8f0 0x7ff7d406edb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff7f81f8200 tx=0x7ff7e0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:08.984+0000 7ff7eea00640 1 -- 172.21.15.40:0/2314294910 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7ff7e807acb0 con 0x7ff7f80b2340 2024-04-26T23:10:09.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.191+0000 7ff7fd400640 1 -- 172.21.15.40:0/2314294910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7f80c9b10 con 0x7ff7f80b2340 2024-04-26T23:10:09.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.194+0000 7ff7eea00640 1 -- 172.21.15.40:0/2314294910 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+16898 (secure 0 0 0) 0x7ff7e807a5b0 con 0x7ff7f80b2340 2024-04-26T23:10:09.194 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:09.194 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":105,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:09:49.407703+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:09:03.955148+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-26T23:09:05.070627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.2a","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:10:09.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.197+0000 7ff7fd400640 1 -- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7d406c8f0 msgr2=0x7ff7d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:09.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.197+0000 7ff7fd400640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7d406c8f0 0x7ff7d406edb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff7f81f8200 tx=0x7ff7e0009040 comp rx=0 tx=0).stop 2024-04-26T23:10:09.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.197+0000 7ff7fd400640 1 -- 172.21.15.40:0/2314294910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7f80b2340 msgr2=0x7ff7f8159c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:09.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.197+0000 7ff7fd400640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7f80b2340 0x7ff7f8159c10 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009890 tx=0x7ff7e80046c0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.197+0000 7ff7fd400640 1 -- 172.21.15.40:0/2314294910 shutdown_connections 2024-04-26T23:10:09.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.197+0000 7ff7fd400640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7ff7d406c8f0 0x7ff7d406edb0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.197+0000 7ff7fd400640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff7f8162810 0x7ff7f80c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.197+0000 7ff7fd400640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f80d1b30 0x7ff7f80c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.197+0000 7ff7fd400640 1 --2- 172.21.15.40:0/2314294910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7f80b2340 0x7ff7f8159c10 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.197+0000 7ff7fd400640 1 -- 172.21.15.40:0/2314294910 >> 172.21.15.40:0/2314294910 conn(0x7ff7f80d2da0 msgr2=0x7ff7f80b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:09.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.197+0000 7ff7fd400640 1 -- 172.21.15.40:0/2314294910 shutdown_connections 2024-04-26T23:10:09.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.197+0000 7ff7fd400640 1 -- 172.21.15.40:0/2314294910 wait complete. 2024-04-26T23:10:09.218 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-26T23:10:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.440+0000 7f90fb600640 1 Processor -- start 2024-04-26T23:10:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.440+0000 7f90fb600640 1 -- start start 2024-04-26T23:10:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.441+0000 7f90fb600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90f4156460 0x7f90f40cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.441+0000 7f90fb600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f90f406fea0 0x7f90f4072c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.441+0000 7f90fb600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90f4156840 con 0x7f90f40cba10 2024-04-26T23:10:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.441+0000 7f90fb600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f90f40cbf20 con 0x7f90f4156460 2024-04-26T23:10:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.441+0000 7f90fb600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f90f40cc200 con 0x7f90f406fea0 2024-04-26T23:10:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.442+0000 7f90f3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f90f40cba10 0x7f90f406f790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51890/0 (socket says 172.21.15.40:51890) 2024-04-26T23:10:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.442+0000 7f90f3e00640 1 -- 172.21.15.40:0/180730875 learned_addr learned my addr 172.21.15.40:0/180730875 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.442+0000 7f90f8c00640 1 --2- 172.21.15.40:0/180730875 >> v2:172.21.15.40:3300/0 conn(0x7f90f4156460 0x7f90f40cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.442+0000 7f90f3400640 1 -- 172.21.15.40:0/180730875 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1377932261 0 0) 0x7f90f4156840 con 0x7f90f40cba10 2024-04-26T23:10:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.442+0000 7f90f9600640 1 --2- 172.21.15.40:0/180730875 >> v2:172.21.15.103:3300/0 conn(0x7f90f406fea0 0x7f90f4072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.442+0000 7f90f3400640 1 -- 172.21.15.40:0/180730875 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90d8003520 con 0x7f90f40cba10 2024-04-26T23:10:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.442+0000 7f90f8c00640 1 -- 172.21.15.40:0/180730875 >> v2:172.21.15.103:3300/0 conn(0x7f90f406fea0 msgr2=0x7f90f4072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.442+0000 7f90f8c00640 1 --2- 172.21.15.40:0/180730875 >> v2:172.21.15.103:3300/0 conn(0x7f90f406fea0 0x7f90f4072c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.442+0000 7f90f8c00640 1 -- 172.21.15.40:0/180730875 >> v1:172.21.15.40:6789/0 conn(0x7f90f40cba10 legacy=0x7f90f406f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.442+0000 7f90f8c00640 1 -- 172.21.15.40:0/180730875 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f40cc4e0 con 0x7f90f4156460 2024-04-26T23:10:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.442+0000 7f90f9600640 1 --2- 172.21.15.40:0/180730875 >> v2:172.21.15.103:3300/0 conn(0x7f90f406fea0 0x7f90f4072c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:09.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.444+0000 7f90f8c00640 1 --2- 172.21.15.40:0/180730875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4156460 0x7f90f40cb4d0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f90f40c7c40 tx=0x7f90e402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=742b13179f3a4ab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:09.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.444+0000 7f90f3400640 1 -- 172.21.15.40:0/180730875 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4037070 con 0x7f90f4156460 2024-04-26T23:10:09.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.444+0000 7f90f3400640 1 -- 172.21.15.40:0/180730875 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90e4032440 con 0x7f90f4156460 2024-04-26T23:10:09.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.444+0000 7f90f3400640 1 -- 172.21.15.40:0/180730875 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4032810 con 0x7f90f4156460 2024-04-26T23:10:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.444+0000 7f90fb600640 1 -- 172.21.15.40:0/180730875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4156460 msgr2=0x7f90f40cb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.444+0000 7f90fb600640 1 --2- 172.21.15.40:0/180730875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4156460 0x7f90f40cb4d0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f90f40c7c40 tx=0x7f90e402a1a0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.445+0000 7f90fb600640 1 -- 172.21.15.40:0/180730875 shutdown_connections 2024-04-26T23:10:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.445+0000 7f90fb600640 1 --2- 172.21.15.40:0/180730875 >> v2:172.21.15.103:3300/0 conn(0x7f90f406fea0 0x7f90f4072c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.445+0000 7f90fb600640 1 --2- 172.21.15.40:0/180730875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4156460 0x7f90f40cb4d0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.445+0000 7f90fb600640 1 -- 172.21.15.40:0/180730875 >> 172.21.15.40:0/180730875 conn(0x7f90f40d0be0 msgr2=0x7f90f40d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.445+0000 7f90fb600640 1 -- 172.21.15.40:0/180730875 shutdown_connections 2024-04-26T23:10:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.445+0000 7f90fb600640 1 -- 172.21.15.40:0/180730875 wait complete. 2024-04-26T23:10:09.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.446+0000 7f90fb600640 1 Processor -- start 2024-04-26T23:10:09.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.447+0000 7f90fb600640 1 -- start start 2024-04-26T23:10:09.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.447+0000 7f90fb600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90f406fea0 0x7f90f41f4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:09.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.447+0000 7f90fb600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90f40cba10 0x7f90f40c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:09.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90fb600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4156460 0x7f90f40c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:09.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90fb600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f90f406e260 con 0x7f90f4156460 2024-04-26T23:10:09.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90fb600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f90f4072700 con 0x7f90f406fea0 2024-04-26T23:10:09.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90fb600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f90f406e3e0 con 0x7f90f40cba10 2024-04-26T23:10:09.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90f9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4156460 0x7f90f40c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:09.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90f9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4156460 0x7f90f40c6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38532/0 (socket says 172.21.15.40:38532) 2024-04-26T23:10:09.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90f9600640 1 -- 172.21.15.40:0/980306468 learned_addr learned my addr 172.21.15.40:0/980306468 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:09.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90f3e00640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90f40cba10 0x7f90f40c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:09.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90f8c00640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90f406fea0 0x7f90f41f4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:09.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90f9600640 1 -- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90f40cba10 msgr2=0x7f90f40c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:09.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90f9600640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90f40cba10 0x7f90f40c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90f9600640 1 -- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90f406fea0 msgr2=0x7f90f41f4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:09.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90f9600640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90f406fea0 0x7f90f41f4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90f9600640 1 -- 172.21.15.40:0/980306468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f414e5a0 con 0x7f90f4156460 2024-04-26T23:10:09.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90f8c00640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90f406fea0 0x7f90f41f4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:09.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.448+0000 7f90f3e00640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90f40cba10 0x7f90f40c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:09.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.449+0000 7f90f9600640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4156460 0x7f90f40c6ca0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f90ec007e10 tx=0x7f90ec00c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:09.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.449+0000 7f90f0c00640 1 -- 172.21.15.40:0/980306468 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec00cb30 con 0x7f90f4156460 2024-04-26T23:10:09.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.449+0000 7f90f0c00640 1 -- 172.21.15.40:0/980306468 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90ec00ccf0 con 0x7f90f4156460 2024-04-26T23:10:09.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.449+0000 7f90fb600640 1 -- 172.21.15.40:0/980306468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90f40c72b0 con 0x7f90f4156460 2024-04-26T23:10:09.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.449+0000 7f90f0c00640 1 -- 172.21.15.40:0/980306468 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec015640 con 0x7f90f4156460 2024-04-26T23:10:09.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.449+0000 7f90fb600640 1 -- 172.21.15.40:0/980306468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90f40c7d50 con 0x7f90f4156460 2024-04-26T23:10:09.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.450+0000 7f90fb600640 1 -- 172.21.15.40:0/980306468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90f40694f0 con 0x7f90f4156460 2024-04-26T23:10:09.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.451+0000 7f90f0c00640 1 -- 172.21.15.40:0/980306468 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f90ec0157e0 con 0x7f90f4156460 2024-04-26T23:10:09.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.452+0000 7f90f0c00640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f90d406c920 0x7f90d406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:09.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.452+0000 7f90f0c00640 1 -- 172.21.15.40:0/980306468 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6130+0+0 (secure 0 0 0) 0x7f90ec08c330 con 0x7f90f4156460 2024-04-26T23:10:09.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.453+0000 7f90f8c00640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f90d406c920 0x7f90d406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:09.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.454+0000 7f90f8c00640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f90d406c920 0x7f90d406ede0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f90e4034000 tx=0x7f90e402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.457+0000 7f90f0c00640 1 -- 172.21.15.40:0/980306468 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f90ec05a510 con 0x7f90f4156460 2024-04-26T23:10:09.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.712+0000 7f90fb600640 1 -- 172.21.15.40:0/980306468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f90f406de80 con 0x7f90f4156460 2024-04-26T23:10:09.713 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-26T23:10:09.713+0000 7f36fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T23:10:09.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.717+0000 7f90f0c00640 1 -- 172.21.15.40:0/980306468 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v123) ==== 161+0+0 (secure 0 0 0) 0x7f90ec059e10 con 0x7f90f4156460 2024-04-26T23:10:09.717 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T23:10:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.720+0000 7f90fb600640 1 -- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f90d406c920 msgr2=0x7f90d406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.721+0000 7f90fb600640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f90d406c920 0x7f90d406ede0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f90e4034000 tx=0x7f90e402e000 comp rx=0 tx=0).stop 2024-04-26T23:10:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.721+0000 7f90fb600640 1 -- 172.21.15.40:0/980306468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4156460 msgr2=0x7f90f40c6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.721+0000 7f90fb600640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4156460 0x7f90f40c6ca0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f90ec007e10 tx=0x7f90ec00c420 comp rx=0 tx=0).stop 2024-04-26T23:10:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.721+0000 7f90fb600640 1 -- 172.21.15.40:0/980306468 shutdown_connections 2024-04-26T23:10:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.721+0000 7f90fb600640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f90d406c920 0x7f90d406ede0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.721+0000 7f90fb600640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4156460 0x7f90f40c6ca0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.721+0000 7f90fb600640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f90f40cba10 0x7f90f40c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.721+0000 7f90fb600640 1 --2- 172.21.15.40:0/980306468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90f406fea0 0x7f90f41f4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.721+0000 7f90fb600640 1 -- 172.21.15.40:0/980306468 >> 172.21.15.40:0/980306468 conn(0x7f90f40d0be0 msgr2=0x7f90f40b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.721+0000 7f90fb600640 1 -- 172.21.15.40:0/980306468 shutdown_connections 2024-04-26T23:10:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.721+0000 7f90fb600640 1 -- 172.21.15.40:0/980306468 wait complete. 2024-04-26T23:10:09.741 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T23:10:09.747 INFO:tasks.ceph.mds.c.smithi040.stderr: -1637> 2024-04-26T23:09:37.516+0000 7fa211000640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000004 [...2,head] /d1/d2/d3/d4/d5/ auth{0=1,2=1} v15 ap=1 f(v0 m2024-04-26T23:09:19.921671+0000 1=0+1) n(v2 rc2024-04-26T23:09:20.161667+0000 4=0+4) (inest lock) (ifile excl) caps={6092=pAsLsXsFsx/pAsLsXsFsxcral@4},l=6092 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 authpin=1 scrubqueue=0 0x5638bd7b0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(15)0x10000000004:[<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:09:19.921671+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:09:20.161667+0000 4=0+4)","memory_value.dirstat":"f(v0 m2024-04-26T23:09:19.921671+0000 1=0+1)","memory_value.rstat":"n(v2 rc2024-04-26T23:09:20.161667+0000 4=0+4)","error_str":""},"return_code":-61} 2024-04-26T23:10:09.747 INFO:tasks.ceph.mds.c.smithi040.stderr: -1499> 2024-04-26T23:09:37.519+0000 7fa219200640 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000005 [...2,head] /d1/d2/d3/d4/d5/d6/ auth{0=1,2=1} v11 ap=2 f(v0 m2024-04-26T23:09:19.922671+0000 1=0+1) n(v1 rc2024-04-26T23:09:20.161667+0000 3=0+3) (inest sync->mix g=0,2 scatter_wanted) caps={6092=pAsLsXsFs/pAsLsXsFsxcral@7} | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 export_pin=2 0x5638bd7b1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(15)0x10000000005:[<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:09:19.922671+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:09:19.923671+0000 3=0+3)","memory_value.dirstat":"f(v0 m2024-04-26T23:09:19.922671+0000 1=0+1)","memory_value.rstat":"n(v1 rc2024-04-26T23:09:20.161667+0000 3=0+3)","error_str":""},"return_code":-61} 2024-04-26T23:10:09.747 INFO:tasks.ceph.mds.a.smithi040.stderr: -8122> 2024-04-26T23:09:22.592+0000 7f928e600640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T23:10:09.754 INFO:tasks.ceph.mds.a.smithi040.stderr: -3489> 2024-04-26T23:09:37.515+0000 7f9289600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000002 [...2,head] /d1/d2/d3/ auth{1=1,2=1} v19 ap=1 f(v0 m2024-04-26T23:09:19.919671+0000 1=0+1) n(v3 rc2024-04-26T23:09:20.161667+0000 6=0+6) (inest mix dirty) (ifile excl) caps={6092=pAsLsXsFsx/pAsLsXsFsxcral@4},l=6092 | dirtyscattered=1 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 0x557fb37e1080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(15)0x10000000002:[<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:09:19.919671+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:09:20.161667+0000 6=0+6)","memory_value.dirstat":"f(v0 m2024-04-26T23:09:19.919671+0000 1=0+1)","memory_value.rstat":"n(v3 rc2024-04-26T23:09:20.161667+0000 6=0+6)","error_str":""},"return_code":-61} 2024-04-26T23:10:09.754 INFO:tasks.ceph.mds.a.smithi040.stderr: -3361> 2024-04-26T23:09:37.517+0000 7f9291800640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000003 [...2,head] /d1/d2/d3/d4/ auth{1=1,2=1} v19 ap=2 f(v0 m2024-04-26T23:09:19.920671+0000 1=0+1) n(v4 rc2024-04-26T23:09:20.161667+0000 5=0+5) (inest sync->mix g=1,2) caps={6092=pAsLsXsFs/pAsLsXsFsxcral@8} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 export_pin=1 0x557fb37e0b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(15)0x10000000003:[<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v19>,<0x1/d1 v18>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:09:19.920671+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:09:20.161667+0000 5=0+5)","memory_value.dirstat":"f(v0 m2024-04-26T23:09:19.920671+0000 1=0+1)","memory_value.rstat":"n(v4 rc2024-04-26T23:09:20.161667+0000 5=0+5)","error_str":""},"return_code":-61} 2024-04-26T23:10:09.758 INFO:tasks.ceph.mds.d.smithi103.stderr: -1547> 2024-04-26T23:09:37.517+0000 7fddf5000640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000006 [...2,head] /d1/d2/d3/d4/d5/d6/d7/ auth v4 ap=1 f(v0 m2024-04-26T23:09:19.923671+0000 1=0+1) n(v0 rc2024-04-26T23:09:19.923671+0000 2=0+2) (inest lock) (ifile excl) caps={6092=pAsLsXsFsx/pAsLsXsFsxcral@4},l=6092 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x55cd1e12c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(15)0x10000000006:[<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-26T23:09:19.923671+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-04-26T23:09:19.923671+0000 2=0+2)","memory_value.dirstat":"f(v0 m2024-04-26T23:09:19.923671+0000 1=0+1)","memory_value.rstat":"n(v0 rc2024-04-26T23:09:19.923671+0000 2=0+2)","error_str":""},"return_code":-61} 2024-04-26T23:10:09.758 INFO:tasks.ceph.mds.d.smithi103.stderr: -1526> 2024-04-26T23:09:37.518+0000 7fddf5000640 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000007 [...2,head] /d1/d2/d3/d4/d5/d6/d7/d8/ auth v2 ap=1 f() n(v0 1=0+1) (inest lock) (ifile excl) caps={6092=pAsLsXsFsx/AsLsXsFsx@2},l=6092 | importingcaps=0 dirfrag=0 caps=1 dirtyparent=0 replicated=0 dirty=0 authpin=1 scrubqueue=0 0x55cd1e12c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(15)0x10000000007:[<0x10000000006/d8 v2>,<0x10000000005/d7 v4>,<0x10000000004/d6 v11>,<0x10000000003/d5 v15>,<0x10000000002/d4 v19>,<0x10000000001/d3 v19>,<0x10000000000/d2 v17>,<0x1/d1 v16>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 1=0+1)","error_str":""},"return_code":-61} 2024-04-26T23:10:09.797 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-04-26T23:10:09.811 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-04-26T23:10:09.815 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-26T23:10:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.926+0000 7efe14000640 1 Processor -- start 2024-04-26T23:10:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.927+0000 7efe14000640 1 -- start start 2024-04-26T23:10:09.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.928+0000 7efe14000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe0c071e60 0x7efe0c06f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:09.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.928+0000 7efe14000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe0c06fe20 0x7efe0c072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:09.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.928+0000 7efe14000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0c15d7c0 con 0x7efe0c072780 2024-04-26T23:10:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.928+0000 7efe14000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7efe0c15daa0 con 0x7efe0c071e60 2024-04-26T23:10:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.928+0000 7efe14000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efe0c15dd80 con 0x7efe0c06fe20 2024-04-26T23:10:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.928+0000 7efe11600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe0c071e60 0x7efe0c06f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.928+0000 7efe11600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe0c071e60 0x7efe0c06f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38576/0 (socket says 172.21.15.40:38576) 2024-04-26T23:10:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.928+0000 7efe11600640 1 -- 172.21.15.40:0/2410110140 learned_addr learned my addr 172.21.15.40:0/2410110140 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.928+0000 7efe12000640 1 --1- 172.21.15.40:0/2410110140 >> v1:172.21.15.40:6789/0 conn(0x7efe0c072780 0x7efe0c0cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51914/0 (socket says 172.21.15.40:51914) 2024-04-26T23:10:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.929+0000 7efe10c00640 1 --2- 172.21.15.40:0/2410110140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe0c06fe20 0x7efe0c072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.929+0000 7efe11600640 1 -- 172.21.15.40:0/2410110140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe0c06fe20 msgr2=0x7efe0c072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.929+0000 7efe11600640 1 --2- 172.21.15.40:0/2410110140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe0c06fe20 0x7efe0c072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.929+0000 7efe11600640 1 -- 172.21.15.40:0/2410110140 >> v1:172.21.15.40:6789/0 conn(0x7efe0c072780 legacy=0x7efe0c0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.929+0000 7efe11600640 1 -- 172.21.15.40:0/2410110140 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0c15e060 con 0x7efe0c071e60 2024-04-26T23:10:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.929+0000 7efe10c00640 1 --2- 172.21.15.40:0/2410110140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe0c06fe20 0x7efe0c072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.929+0000 7efe0be00640 1 -- 172.21.15.40:0/2410110140 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3030796796 0 0) 0x7efe0c15d7c0 con 0x7efe0c072780 2024-04-26T23:10:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.930+0000 7efe11600640 1 --2- 172.21.15.40:0/2410110140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c071e60 0x7efe0c06f8e0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7efe00009870 tx=0x7efe0002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8ac0219e6f5f06 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:09.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.930+0000 7efe0be00640 1 -- 172.21.15.40:0/2410110140 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe00037070 con 0x7efe0c071e60 2024-04-26T23:10:09.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.930+0000 7efe0be00640 1 -- 172.21.15.40:0/2410110140 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe00032440 con 0x7efe0c071e60 2024-04-26T23:10:09.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.930+0000 7efe0be00640 1 -- 172.21.15.40:0/2410110140 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe00032810 con 0x7efe0c071e60 2024-04-26T23:10:09.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.931+0000 7efe14000640 1 -- 172.21.15.40:0/2410110140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c071e60 msgr2=0x7efe0c06f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:09.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.931+0000 7efe14000640 1 --2- 172.21.15.40:0/2410110140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c071e60 0x7efe0c06f8e0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7efe00009870 tx=0x7efe0002a250 comp rx=0 tx=0).stop 2024-04-26T23:10:09.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.931+0000 7efe14000640 1 -- 172.21.15.40:0/2410110140 shutdown_connections 2024-04-26T23:10:09.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.931+0000 7efe14000640 1 --2- 172.21.15.40:0/2410110140 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe0c06fe20 0x7efe0c072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.931+0000 7efe14000640 1 --2- 172.21.15.40:0/2410110140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c071e60 0x7efe0c06f8e0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.931+0000 7efe14000640 1 -- 172.21.15.40:0/2410110140 >> 172.21.15.40:0/2410110140 conn(0x7efe0c0d0be0 msgr2=0x7efe0c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:09.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.931+0000 7efe14000640 1 -- 172.21.15.40:0/2410110140 shutdown_connections 2024-04-26T23:10:09.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.931+0000 7efe14000640 1 -- 172.21.15.40:0/2410110140 wait complete. 2024-04-26T23:10:09.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.933+0000 7efe14000640 1 Processor -- start 2024-04-26T23:10:09.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.933+0000 7efe14000640 1 -- start start 2024-04-26T23:10:09.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.934+0000 7efe14000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c06fe20 0x7efe0c14dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:09.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.934+0000 7efe14000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe0c071e60 0x7efe0c0af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:09.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.934+0000 7efe14000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe0c072780 0x7efe0c0c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:09.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.934+0000 7efe14000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c15d3f0 con 0x7efe0c06fe20 2024-04-26T23:10:09.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.934+0000 7efe14000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c072c10 con 0x7efe0c071e60 2024-04-26T23:10:09.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.934+0000 7efe14000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efe0c15d570 con 0x7efe0c072780 2024-04-26T23:10:09.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.934+0000 7efe11600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c06fe20 0x7efe0c14dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:09.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.934+0000 7efe11600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c06fe20 0x7efe0c14dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38584/0 (socket says 172.21.15.40:38584) 2024-04-26T23:10:09.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.934+0000 7efe11600640 1 -- 172.21.15.40:0/2848818963 learned_addr learned my addr 172.21.15.40:0/2848818963 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:09.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.935+0000 7efe11600640 1 -- 172.21.15.40:0/2848818963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe0c072780 msgr2=0x7efe0c0c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:09.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.935+0000 7efe11600640 1 --2- 172.21.15.40:0/2848818963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe0c072780 0x7efe0c0c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.935+0000 7efe11600640 1 -- 172.21.15.40:0/2848818963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe0c071e60 msgr2=0x7efe0c0af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:09.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.935+0000 7efe11600640 1 --2- 172.21.15.40:0/2848818963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe0c071e60 0x7efe0c0af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:09.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.935+0000 7efe11600640 1 -- 172.21.15.40:0/2848818963 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0c0c8230 con 0x7efe0c06fe20 2024-04-26T23:10:09.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.935+0000 7efe11600640 1 --2- 172.21.15.40:0/2848818963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c06fe20 0x7efe0c14dbb0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7efe00000c00 tx=0x7efe000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:09.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.935+0000 7efe09600640 1 -- 172.21.15.40:0/2848818963 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe00037040 con 0x7efe0c06fe20 2024-04-26T23:10:09.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.935+0000 7efe09600640 1 -- 172.21.15.40:0/2848818963 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe00042070 con 0x7efe0c06fe20 2024-04-26T23:10:09.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.935+0000 7efe09600640 1 -- 172.21.15.40:0/2848818963 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe00030bd0 con 0x7efe0c06fe20 2024-04-26T23:10:09.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.935+0000 7efe14000640 1 -- 172.21.15.40:0/2848818963 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe0c0c8510 con 0x7efe0c06fe20 2024-04-26T23:10:09.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.936+0000 7efe14000640 1 -- 172.21.15.40:0/2848818963 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe0c0c8ad0 con 0x7efe0c06fe20 2024-04-26T23:10:09.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.937+0000 7efe14000640 1 -- 172.21.15.40:0/2848818963 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe0c0694f0 con 0x7efe0c06fe20 2024-04-26T23:10:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.938+0000 7efe09600640 1 -- 172.21.15.40:0/2848818963 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efe00047460 con 0x7efe0c06fe20 2024-04-26T23:10:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.939+0000 7efe09600640 1 --2- 172.21.15.40:0/2848818963 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efdec06c8f0 0x7efdec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:09.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.939+0000 7efe09600640 1 -- 172.21.15.40:0/2848818963 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6388+0+0 (secure 0 0 0) 0x7efe000aca40 con 0x7efe0c06fe20 2024-04-26T23:10:09.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.939+0000 7efe10c00640 1 --2- 172.21.15.40:0/2848818963 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efdec06c8f0 0x7efdec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.940+0000 7efe10c00640 1 --2- 172.21.15.40:0/2848818963 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efdec06c8f0 0x7efdec06edb0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7efdf8006fd0 tx=0x7efdf8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:09.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:09.943+0000 7efe09600640 1 -- 172.21.15.40:0/2848818963 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7efe0007abb0 con 0x7efe0c06fe20 2024-04-26T23:10:10.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.196+0000 7efe14000640 1 -- 172.21.15.40:0/2848818963 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7efe0c0c7ec0 con 0x7efe0c06fe20 2024-04-26T23:10:10.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.720+0000 7efe09600640 1 -- 172.21.15.40:0/2848818963 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v124) ==== 104+0+0 (secure 0 0 0) 0x7efe0007a4b0 con 0x7efe0c06fe20 2024-04-26T23:10:10.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.723+0000 7efe14000640 1 -- 172.21.15.40:0/2848818963 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efdec06c8f0 msgr2=0x7efdec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:10.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.723+0000 7efe14000640 1 --2- 172.21.15.40:0/2848818963 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efdec06c8f0 0x7efdec06edb0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7efdf8006fd0 tx=0x7efdf8008040 comp rx=0 tx=0).stop 2024-04-26T23:10:10.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.724+0000 7efe14000640 1 -- 172.21.15.40:0/2848818963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c06fe20 msgr2=0x7efe0c14dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:10.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.724+0000 7efe14000640 1 --2- 172.21.15.40:0/2848818963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c06fe20 0x7efe0c14dbb0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7efe00000c00 tx=0x7efe000046c0 comp rx=0 tx=0).stop 2024-04-26T23:10:10.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.724+0000 7efe14000640 1 -- 172.21.15.40:0/2848818963 shutdown_connections 2024-04-26T23:10:10.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.724+0000 7efe14000640 1 --2- 172.21.15.40:0/2848818963 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efdec06c8f0 0x7efdec06edb0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:10.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.724+0000 7efe14000640 1 --2- 172.21.15.40:0/2848818963 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efe0c072780 0x7efe0c0c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:10.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.724+0000 7efe14000640 1 --2- 172.21.15.40:0/2848818963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe0c071e60 0x7efe0c0af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:10.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.724+0000 7efe14000640 1 --2- 172.21.15.40:0/2848818963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c06fe20 0x7efe0c14dbb0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:10.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.724+0000 7efe14000640 1 -- 172.21.15.40:0/2848818963 >> 172.21.15.40:0/2848818963 conn(0x7efe0c0d0be0 msgr2=0x7efe0c15fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:10.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.724+0000 7efe14000640 1 -- 172.21.15.40:0/2848818963 shutdown_connections 2024-04-26T23:10:10.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:10.724+0000 7efe14000640 1 -- 172.21.15.40:0/2848818963 wait complete. 2024-04-26T23:10:10.747 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T23:10:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.008+0000 7f5e49200640 1 Processor -- start 2024-04-26T23:10:11.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.008+0000 7f5e49200640 1 -- start start 2024-04-26T23:10:11.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.010+0000 7f5e49200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5e440af9d0 0x7f5e4415fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.010+0000 7f5e49200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e440afed0 con 0x7f5e440d0120 2024-04-26T23:10:11.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.010+0000 7f5e49200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e440b4f70 con 0x7f5e44073a30 2024-04-26T23:10:11.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.010+0000 7f5e49200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5e440b58a0 con 0x7f5e440af9d0 2024-04-26T23:10:11.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.010+0000 7f5e43400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5e440d0120 0x7f5e44073320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51928/0 (socket says 172.21.15.40:51928) 2024-04-26T23:10:11.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.010+0000 7f5e43400640 1 -- 172.21.15.40:0/780268778 learned_addr learned my addr 172.21.15.40:0/780268778 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:11.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.011+0000 7f5e43e00640 1 --2- 172.21.15.40:0/780268778 >> v2:172.21.15.103:3300/0 conn(0x7f5e440af9d0 0x7f5e4415fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:11.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.011+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 231615934 0 0) 0x7f5e440afed0 con 0x7f5e440d0120 2024-04-26T23:10:11.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.011+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e28003520 con 0x7f5e440d0120 2024-04-26T23:10:11.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.011+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3057584313 0 0) 0x7f5e28003520 con 0x7f5e440d0120 2024-04-26T23:10:11.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.011+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 >> v2:172.21.15.103:3300/0 conn(0x7f5e440af9d0 msgr2=0x7f5e4415fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:11.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.011+0000 7f5e42000640 1 --2- 172.21.15.40:0/780268778 >> v2:172.21.15.103:3300/0 conn(0x7f5e440af9d0 0x7f5e4415fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.011+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 >> v1:172.21.15.103:6789/0 conn(0x7f5e44073a30 legacy=0x7f5e44071d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.011+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e440b5250 con 0x7f5e440d0120 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.011+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f5e34001670 con 0x7f5e440d0120 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.011+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 >> v1:172.21.15.40:6789/0 conn(0x7f5e440d0120 legacy=0x7f5e44073320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.011+0000 7f5e42000640 1 --2- 172.21.15.40:0/780268778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e280039b0 0x7f5e28003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e42000640 1 --2- 172.21.15.40:0/780268778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e28004800 0x7f5e28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e42000640 1 --2- 172.21.15.40:0/780268778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e280051c0 0x7f5e2800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5e440b58a0 con 0x7f5e28004800 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2800c060 con 0x7f5e280051c0 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5e2800c260 con 0x7f5e280039b0 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e42a00640 1 --2- 172.21.15.40:0/780268778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e28004800 0x7f5e28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e43e00640 1 --2- 172.21.15.40:0/780268778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e280051c0 0x7f5e2800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e43400640 1 --2- 172.21.15.40:0/780268778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e280039b0 0x7f5e28003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e42a00640 1 -- 172.21.15.40:0/780268778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e280039b0 msgr2=0x7f5e28003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:11.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e42a00640 1 --2- 172.21.15.40:0/780268778 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e280039b0 0x7f5e28003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e42a00640 1 -- 172.21.15.40:0/780268778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e280051c0 msgr2=0x7f5e2800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:11.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e42a00640 1 --2- 172.21.15.40:0/780268778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e280051c0 0x7f5e2800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e42a00640 1 -- 172.21.15.40:0/780268778 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e28003520 con 0x7f5e28004800 2024-04-26T23:10:11.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.012+0000 7f5e43e00640 1 -- 172.21.15.40:0/780268778 reap_dead start 2024-04-26T23:10:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.014+0000 7f5e42a00640 1 --2- 172.21.15.40:0/780268778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e28004800 0x7f5e28004c80 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f5e30001750 tx=0x7f5e28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a7a0070357f25d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.014+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30001370 con 0x7f5e28004800 2024-04-26T23:10:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.014+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e300052d0 con 0x7f5e28004800 2024-04-26T23:10:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.014+0000 7f5e42000640 1 -- 172.21.15.40:0/780268778 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3002abc0 con 0x7f5e28004800 2024-04-26T23:10:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.014+0000 7f5e49200640 1 -- 172.21.15.40:0/780268778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e28004800 msgr2=0x7f5e28004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.014+0000 7f5e49200640 1 --2- 172.21.15.40:0/780268778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e28004800 0x7f5e28004c80 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f5e30001750 tx=0x7f5e28005830 comp rx=0 tx=0).stop 2024-04-26T23:10:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.014+0000 7f5e49200640 1 -- 172.21.15.40:0/780268778 shutdown_connections 2024-04-26T23:10:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.014+0000 7f5e49200640 1 --2- 172.21.15.40:0/780268778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e28004800 0x7f5e28004c80 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.014+0000 7f5e49200640 1 -- 172.21.15.40:0/780268778 >> 172.21.15.40:0/780268778 conn(0x7f5e44075f10 msgr2=0x7f5e44068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.015+0000 7f5e49200640 1 -- 172.21.15.40:0/780268778 shutdown_connections 2024-04-26T23:10:11.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.015+0000 7f5e49200640 1 -- 172.21.15.40:0/780268778 wait complete. 2024-04-26T23:10:11.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.017+0000 7f5e49200640 1 Processor -- start 2024-04-26T23:10:11.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.017+0000 7f5e49200640 1 -- start start 2024-04-26T23:10:11.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e49200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e28004800 0x7f5e44140480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e49200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4413d690 0x7f5e44141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e49200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e44142420 0x7f5e4406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e49200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5e440b5d30 con 0x7f5e4413d690 2024-04-26T23:10:11.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e49200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5e440d0500 con 0x7f5e28004800 2024-04-26T23:10:11.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e49200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5e440b5eb0 con 0x7f5e44142420 2024-04-26T23:10:11.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e42a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4413d690 0x7f5e44141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:11.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e42a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4413d690 0x7f5e44141ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38602/0 (socket says 172.21.15.40:38602) 2024-04-26T23:10:11.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e42a00640 1 -- 172.21.15.40:0/4006738318 learned_addr learned my addr 172.21.15.40:0/4006738318 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e42a00640 1 -- 172.21.15.40:0/4006738318 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e44142420 msgr2=0x7f5e4406ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e42a00640 1 --2- 172.21.15.40:0/4006738318 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e44142420 0x7f5e4406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e42a00640 1 -- 172.21.15.40:0/4006738318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e28004800 msgr2=0x7f5e44140480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e42a00640 1 --2- 172.21.15.40:0/4006738318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e28004800 0x7f5e44140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.018+0000 7f5e42a00640 1 -- 172.21.15.40:0/4006738318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e4413f3e0 con 0x7f5e4413d690 2024-04-26T23:10:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.019+0000 7f5e42a00640 1 --2- 172.21.15.40:0/4006738318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4413d690 0x7f5e44141ee0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f5e30001870 tx=0x7f5e30001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.019+0000 7f5e3b400640 1 -- 172.21.15.40:0/4006738318 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30001370 con 0x7f5e4413d690 2024-04-26T23:10:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.019+0000 7f5e3b400640 1 -- 172.21.15.40:0/4006738318 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e30041070 con 0x7f5e4413d690 2024-04-26T23:10:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.019+0000 7f5e49200640 1 -- 172.21.15.40:0/4006738318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e4413f6c0 con 0x7f5e4413d690 2024-04-26T23:10:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.019+0000 7f5e3b400640 1 -- 172.21.15.40:0/4006738318 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3002be50 con 0x7f5e4413d690 2024-04-26T23:10:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.019+0000 7f5e49200640 1 -- 172.21.15.40:0/4006738318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e4413fc80 con 0x7f5e4413d690 2024-04-26T23:10:11.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.020+0000 7f5e49200640 1 -- 172.21.15.40:0/4006738318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e44069370 con 0x7f5e4413d690 2024-04-26T23:10:11.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.021+0000 7f5e3b400640 1 -- 172.21.15.40:0/4006738318 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5e3003a070 con 0x7f5e4413d690 2024-04-26T23:10:11.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.022+0000 7f5e3b400640 1 --2- 172.21.15.40:0/4006738318 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5e2406cac0 0x7f5e2406ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.023+0000 7f5e3b400640 1 -- 172.21.15.40:0/4006738318 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6388+0+0 (secure 0 0 0) 0x7f5e300ab8c0 con 0x7f5e4413d690 2024-04-26T23:10:11.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.023+0000 7f5e43400640 1 --2- 172.21.15.40:0/4006738318 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5e2406cac0 0x7f5e2406ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:11.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.024+0000 7f5e43400640 1 --2- 172.21.15.40:0/4006738318 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5e2406cac0 0x7f5e2406ef80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5e34000ff0 tx=0x7f5e34001430 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:11.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.027+0000 7f5e3b400640 1 -- 172.21.15.40:0/4006738318 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f5e30079910 con 0x7f5e4413d690 2024-04-26T23:10:11.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.235+0000 7f5e49200640 1 -- 172.21.15.40:0/4006738318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5e44146da0 con 0x7f5e4413d690 2024-04-26T23:10:11.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.730+0000 7f5e3b400640 1 -- 172.21.15.40:0/4006738318 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v107) ==== 181+0+0 (secure 0 0 0) 0x7f5e30079210 con 0x7f5e4413d690 2024-04-26T23:10:11.731 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-04-26T23:10:11.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.734+0000 7f5e49200640 1 -- 172.21.15.40:0/4006738318 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5e2406cac0 msgr2=0x7f5e2406ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:11.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.734+0000 7f5e49200640 1 --2- 172.21.15.40:0/4006738318 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5e2406cac0 0x7f5e2406ef80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5e34000ff0 tx=0x7f5e34001430 comp rx=0 tx=0).stop 2024-04-26T23:10:11.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.734+0000 7f5e49200640 1 -- 172.21.15.40:0/4006738318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4413d690 msgr2=0x7f5e44141ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:11.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.734+0000 7f5e49200640 1 --2- 172.21.15.40:0/4006738318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4413d690 0x7f5e44141ee0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f5e30001870 tx=0x7f5e30001120 comp rx=0 tx=0).stop 2024-04-26T23:10:11.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.735+0000 7f5e49200640 1 -- 172.21.15.40:0/4006738318 shutdown_connections 2024-04-26T23:10:11.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.735+0000 7f5e49200640 1 --2- 172.21.15.40:0/4006738318 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f5e2406cac0 0x7f5e2406ef80 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.735+0000 7f5e49200640 1 --2- 172.21.15.40:0/4006738318 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e44142420 0x7f5e4406ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.735+0000 7f5e49200640 1 --2- 172.21.15.40:0/4006738318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4413d690 0x7f5e44141ee0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.735+0000 7f5e49200640 1 --2- 172.21.15.40:0/4006738318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e28004800 0x7f5e44140480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.735+0000 7f5e49200640 1 -- 172.21.15.40:0/4006738318 >> 172.21.15.40:0/4006738318 conn(0x7f5e44075f10 msgr2=0x7f5e440b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:11.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.737+0000 7f5e49200640 1 -- 172.21.15.40:0/4006738318 shutdown_connections 2024-04-26T23:10:11.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.737+0000 7f5e49200640 1 -- 172.21.15.40:0/4006738318 wait complete. 2024-04-26T23:10:11.764 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T23:10:11.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.979+0000 7efc41c00640 1 Processor -- start 2024-04-26T23:10:11.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.979+0000 7efc41c00640 1 -- start start 2024-04-26T23:10:11.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.981+0000 7efc41c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc3c145970 0x7efc3c1612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.981+0000 7efc41c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc3c0b59f0 con 0x7efc3c0d06b0 2024-04-26T23:10:11.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.981+0000 7efc41c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc3c1617f0 con 0x7efc3c146060 2024-04-26T23:10:11.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.981+0000 7efc41c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efc3c0ceed0 con 0x7efc3c145970 2024-04-26T23:10:11.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.981+0000 7efc41200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efc3c0d06b0 0x7efc3c0cf900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51930/0 (socket says 172.21.15.40:51930) 2024-04-26T23:10:11.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.981+0000 7efc41200640 1 -- 172.21.15.40:0/4059786657 learned_addr learned my addr 172.21.15.40:0/4059786657 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:11.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.982+0000 7efc3be00640 1 --2- 172.21.15.40:0/4059786657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc3c145970 0x7efc3c1612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:11.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.982+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3929606562 0 0) 0x7efc3c0b59f0 con 0x7efc3c0d06b0 2024-04-26T23:10:11.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.982+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc20003520 con 0x7efc3c0d06b0 2024-04-26T23:10:11.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.982+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2123559597 0 0) 0x7efc20003520 con 0x7efc3c0d06b0 2024-04-26T23:10:11.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.982+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc3c145970 msgr2=0x7efc3c1612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:11.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.982+0000 7efc3aa00640 1 --2- 172.21.15.40:0/4059786657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc3c145970 0x7efc3c1612b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.982+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 >> v1:172.21.15.103:6789/0 conn(0x7efc3c146060 legacy=0x7efc3c0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.982+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3c0b4180 con 0x7efc3c0d06b0 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.982+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7efc340019f0 con 0x7efc3c0d06b0 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.982+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 >> v1:172.21.15.40:6789/0 conn(0x7efc3c0d06b0 legacy=0x7efc3c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.982+0000 7efc3aa00640 1 --2- 172.21.15.40:0/4059786657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc200039e0 0x7efc20003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.983+0000 7efc3aa00640 1 --2- 172.21.15.40:0/4059786657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc3c145970 0x7efc20004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.983+0000 7efc3aa00640 1 --2- 172.21.15.40:0/4059786657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc20004d40 0x7efc200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.983+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc2000ba70 con 0x7efc200039e0 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.983+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc2000bbf0 con 0x7efc20004d40 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.983+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efc2000bdf0 con 0x7efc3c145970 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.983+0000 7efc3be00640 1 --2- 172.21.15.40:0/4059786657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc200039e0 0x7efc20003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.983+0000 7efc3be00640 1 -- 172.21.15.40:0/4059786657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc3c145970 msgr2=0x7efc20004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.983+0000 7efc3be00640 1 --2- 172.21.15.40:0/4059786657 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc3c145970 0x7efc20004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.983+0000 7efc3be00640 1 -- 172.21.15.40:0/4059786657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc20004d40 msgr2=0x7efc200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:11.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.983+0000 7efc3be00640 1 --2- 172.21.15.40:0/4059786657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc20004d40 0x7efc200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.983+0000 7efc3be00640 1 -- 172.21.15.40:0/4059786657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc20003520 con 0x7efc200039e0 2024-04-26T23:10:11.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.984+0000 7efc3be00640 1 --2- 172.21.15.40:0/4059786657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc200039e0 0x7efc20003e00 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7efc2c009a90 tx=0x7efc20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c6c5bf02a951fb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:11.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.985+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c036070 con 0x7efc200039e0 2024-04-26T23:10:11.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.985+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc2c031400 con 0x7efc200039e0 2024-04-26T23:10:11.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.985+0000 7efc3aa00640 1 -- 172.21.15.40:0/4059786657 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c009690 con 0x7efc200039e0 2024-04-26T23:10:11.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.985+0000 7efc41c00640 1 -- 172.21.15.40:0/4059786657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc200039e0 msgr2=0x7efc20003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:11.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.985+0000 7efc41c00640 1 --2- 172.21.15.40:0/4059786657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc200039e0 0x7efc20003e00 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7efc2c009a90 tx=0x7efc20005830 comp rx=0 tx=0).stop 2024-04-26T23:10:11.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.985+0000 7efc41200640 1 -- 172.21.15.40:0/4059786657 reap_dead start 2024-04-26T23:10:11.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.985+0000 7efc41c00640 1 -- 172.21.15.40:0/4059786657 shutdown_connections 2024-04-26T23:10:11.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.985+0000 7efc41c00640 1 -- 172.21.15.40:0/4059786657 >> 172.21.15.40:0/4059786657 conn(0x7efc3c075eb0 msgr2=0x7efc3c068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:11.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.985+0000 7efc41c00640 1 -- 172.21.15.40:0/4059786657 shutdown_connections 2024-04-26T23:10:11.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.985+0000 7efc41c00640 1 -- 172.21.15.40:0/4059786657 wait complete. 2024-04-26T23:10:11.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.987+0000 7efc41c00640 1 Processor -- start 2024-04-26T23:10:11.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.987+0000 7efc41c00640 1 -- start start 2024-04-26T23:10:11.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.988+0000 7efc41c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc200039e0 0x7efc3c1f5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.988+0000 7efc41c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c1f2580 0x7efc3c1f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.988+0000 7efc41c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc3c1f5070 0x7efc3c1f54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.988+0000 7efc41c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c0cf100 con 0x7efc200039e0 2024-04-26T23:10:11.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.988+0000 7efc41c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c0b5c60 con 0x7efc3c1f2580 2024-04-26T23:10:11.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.988+0000 7efc41c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efc3c0cf3e0 con 0x7efc3c1f5070 2024-04-26T23:10:11.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.988+0000 7efc3be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc200039e0 0x7efc3c1f5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:11.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.989+0000 7efc3be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc200039e0 0x7efc3c1f5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38606/0 (socket says 172.21.15.40:38606) 2024-04-26T23:10:11.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.989+0000 7efc3be00640 1 -- 172.21.15.40:0/3637672595 learned_addr learned my addr 172.21.15.40:0/3637672595 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:11.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.989+0000 7efc3b400640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c1f2580 0x7efc3c1f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:11.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.989+0000 7efc3be00640 1 -- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc3c1f5070 msgr2=0x7efc3c1f54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:11.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.989+0000 7efc41200640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc3c1f5070 0x7efc3c1f54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.989+0000 7efc3be00640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc3c1f5070 0x7efc3c1f54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.989+0000 7efc3be00640 1 -- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c1f2580 msgr2=0x7efc3c1f2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.989+0000 7efc3be00640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c1f2580 0x7efc3c1f2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.989+0000 7efc3be00640 1 -- 172.21.15.40:0/3637672595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3c1f6350 con 0x7efc200039e0 2024-04-26T23:10:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.989+0000 7efc41200640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc3c1f5070 0x7efc3c1f54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.989+0000 7efc3b400640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c1f2580 0x7efc3c1f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.989+0000 7efc3be00640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc200039e0 0x7efc3c1f5b00 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7efc2c009bb0 tx=0x7efc2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.990+0000 7efc33e00640 1 -- 172.21.15.40:0/3637672595 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c040020 con 0x7efc200039e0 2024-04-26T23:10:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.990+0000 7efc33e00640 1 -- 172.21.15.40:0/3637672595 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc2c003bc0 con 0x7efc200039e0 2024-04-26T23:10:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.990+0000 7efc33e00640 1 -- 172.21.15.40:0/3637672595 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c036040 con 0x7efc200039e0 2024-04-26T23:10:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.990+0000 7efc41c00640 1 -- 172.21.15.40:0/3637672595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc3c1f6630 con 0x7efc200039e0 2024-04-26T23:10:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.990+0000 7efc41c00640 1 -- 172.21.15.40:0/3637672595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc3c1f6bf0 con 0x7efc200039e0 2024-04-26T23:10:11.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.991+0000 7efc41c00640 1 -- 172.21.15.40:0/3637672595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc3c069370 con 0x7efc200039e0 2024-04-26T23:10:11.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.992+0000 7efc33e00640 1 -- 172.21.15.40:0/3637672595 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efc2c043020 con 0x7efc200039e0 2024-04-26T23:10:11.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.993+0000 7efc33e00640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc1c06c920 0x7efc1c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:11.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.993+0000 7efc33e00640 1 -- 172.21.15.40:0/3637672595 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5950+0+0 (secure 0 0 0) 0x7efc2c0abfa0 con 0x7efc200039e0 2024-04-26T23:10:11.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.994+0000 7efc3b400640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc1c06c920 0x7efc1c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:11.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.995+0000 7efc3b400640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc1c06c920 0x7efc1c06ede0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7efc3c0b5230 tx=0x7efc28009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:11.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:11.998+0000 7efc33e00640 1 -- 172.21.15.40:0/3637672595 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7efc2c04a030 con 0x7efc200039e0 2024-04-26T23:10:12.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.205+0000 7efc41c00640 1 -- 172.21.15.40:0/3637672595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7efc3c0d1400 con 0x7efc200039e0 2024-04-26T23:10:12.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.734+0000 7efc33e00640 1 -- 172.21.15.40:0/3637672595 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v108) ==== 169+0+0 (secure 0 0 0) 0x7efc2c0044e0 con 0x7efc200039e0 2024-04-26T23:10:12.734 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-04-26T23:10:12.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.740+0000 7efc41c00640 1 -- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc1c06c920 msgr2=0x7efc1c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:12.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.740+0000 7efc41c00640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc1c06c920 0x7efc1c06ede0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7efc3c0b5230 tx=0x7efc28009040 comp rx=0 tx=0).stop 2024-04-26T23:10:12.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.740+0000 7efc41c00640 1 -- 172.21.15.40:0/3637672595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc200039e0 msgr2=0x7efc3c1f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:12.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.740+0000 7efc41c00640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc200039e0 0x7efc3c1f5b00 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7efc2c009bb0 tx=0x7efc2c004730 comp rx=0 tx=0).stop 2024-04-26T23:10:12.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.740+0000 7efc41c00640 1 -- 172.21.15.40:0/3637672595 shutdown_connections 2024-04-26T23:10:12.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.740+0000 7efc41c00640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7efc1c06c920 0x7efc1c06ede0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:12.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.740+0000 7efc41c00640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc3c1f5070 0x7efc3c1f54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:12.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.740+0000 7efc41c00640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c1f2580 0x7efc3c1f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:12.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.740+0000 7efc41c00640 1 --2- 172.21.15.40:0/3637672595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc200039e0 0x7efc3c1f5b00 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:12.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.740+0000 7efc41c00640 1 -- 172.21.15.40:0/3637672595 >> 172.21.15.40:0/3637672595 conn(0x7efc3c075eb0 msgr2=0x7efc3c0b4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:12.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.741+0000 7efc41c00640 1 -- 172.21.15.40:0/3637672595 shutdown_connections 2024-04-26T23:10:12.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.741+0000 7efc41c00640 1 -- 172.21.15.40:0/3637672595 wait complete. 2024-04-26T23:10:12.765 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-26T23:10:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.964+0000 7fcc46600640 1 Processor -- start 2024-04-26T23:10:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.964+0000 7fcc46600640 1 -- start start 2024-04-26T23:10:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.965+0000 7fcc46600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc400c6f10 0x7fcc400c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.965+0000 7fcc46600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcc400c6010 0x7fcc40158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.965+0000 7fcc46600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc400cce10 con 0x7fcc400cc980 2024-04-26T23:10:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.965+0000 7fcc46600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcc400c72f0 con 0x7fcc400c6010 2024-04-26T23:10:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.965+0000 7fcc46600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcc400cb860 con 0x7fcc400c6f10 2024-04-26T23:10:12.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.966+0000 7fcc45200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc400c6f10 0x7fcc400c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:12.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.966+0000 7fcc45c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcc400c6010 0x7fcc40158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:12.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.966+0000 7fcc45200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc400c6f10 0x7fcc400c5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:52962/0 (socket says 172.21.15.40:52962) 2024-04-26T23:10:12.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.966+0000 7fcc45200640 1 -- 172.21.15.40:0/214592353 learned_addr learned my addr 172.21.15.40:0/214592353 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:12.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.967+0000 7fcc45200640 1 -- 172.21.15.40:0/214592353 >> v2:172.21.15.103:3300/0 conn(0x7fcc400c6010 msgr2=0x7fcc40158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:12.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.967+0000 7fcc45200640 1 --2- 172.21.15.40:0/214592353 >> v2:172.21.15.103:3300/0 conn(0x7fcc400c6010 0x7fcc40158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:12.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.967+0000 7fcc45200640 1 -- 172.21.15.40:0/214592353 >> v1:172.21.15.103:6789/0 conn(0x7fcc400cc980 legacy=0x7fcc400cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:12.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.967+0000 7fcc45200640 1 -- 172.21.15.40:0/214592353 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc400cd0f0 con 0x7fcc400c6f10 2024-04-26T23:10:12.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.967+0000 7fcc3f400640 1 -- 172.21.15.40:0/214592353 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 637271398 0 0) 0x7fcc400cce10 con 0x7fcc400cc980 2024-04-26T23:10:12.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.968+0000 7fcc45200640 1 --2- 172.21.15.40:0/214592353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc400c6f10 0x7fcc400c5ad0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009870 tx=0x7fcc2802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14f646699538ae26 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:12.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.969+0000 7fcc3f400640 1 -- 172.21.15.40:0/214592353 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28037070 con 0x7fcc400c6f10 2024-04-26T23:10:12.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.969+0000 7fcc3f400640 1 -- 172.21.15.40:0/214592353 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc28032440 con 0x7fcc400c6f10 2024-04-26T23:10:12.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.969+0000 7fcc3f400640 1 -- 172.21.15.40:0/214592353 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28032810 con 0x7fcc400c6f10 2024-04-26T23:10:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.970+0000 7fcc46600640 1 -- 172.21.15.40:0/214592353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc400c6f10 msgr2=0x7fcc400c5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.970+0000 7fcc46600640 1 --2- 172.21.15.40:0/214592353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc400c6f10 0x7fcc400c5ad0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009870 tx=0x7fcc2802a1c0 comp rx=0 tx=0).stop 2024-04-26T23:10:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.970+0000 7fcc46600640 1 -- 172.21.15.40:0/214592353 shutdown_connections 2024-04-26T23:10:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.970+0000 7fcc46600640 1 --2- 172.21.15.40:0/214592353 >> v2:172.21.15.103:3300/0 conn(0x7fcc400c6010 0x7fcc40158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.970+0000 7fcc46600640 1 --2- 172.21.15.40:0/214592353 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc400c6f10 0x7fcc400c5ad0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.970+0000 7fcc46600640 1 -- 172.21.15.40:0/214592353 >> 172.21.15.40:0/214592353 conn(0x7fcc400d0a10 msgr2=0x7fcc400d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.970+0000 7fcc46600640 1 -- 172.21.15.40:0/214592353 shutdown_connections 2024-04-26T23:10:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.970+0000 7fcc46600640 1 -- 172.21.15.40:0/214592353 wait complete. 2024-04-26T23:10:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.972+0000 7fcc46600640 1 Processor -- start 2024-04-26T23:10:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.972+0000 7fcc46600640 1 -- start start 2024-04-26T23:10:12.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.973+0000 7fcc46600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc400c6010 0x7fcc400b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:12.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.973+0000 7fcc46600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc400c6f10 0x7fcc400b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:12.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.973+0000 7fcc46600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc400b5330 0x7fcc40144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:12.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.973+0000 7fcc46600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcc400cc400 con 0x7fcc400c6f10 2024-04-26T23:10:12.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.973+0000 7fcc46600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcc400cb1a0 con 0x7fcc400c6010 2024-04-26T23:10:12.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.973+0000 7fcc46600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcc400cc580 con 0x7fcc400b5330 2024-04-26T23:10:12.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.973+0000 7fcc3fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc400c6f10 0x7fcc400b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:12.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.973+0000 7fcc3fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc400c6f10 0x7fcc400b4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38612/0 (socket says 172.21.15.40:38612) 2024-04-26T23:10:12.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.973+0000 7fcc3fe00640 1 -- 172.21.15.40:0/3115313287 learned_addr learned my addr 172.21.15.40:0/3115313287 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:12.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.974+0000 7fcc45c00640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc400b5330 0x7fcc40144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:12.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.974+0000 7fcc45200640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc400c6010 0x7fcc400b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:12.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.974+0000 7fcc3fe00640 1 -- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc400b5330 msgr2=0x7fcc40144190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:12.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.974+0000 7fcc3fe00640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc400b5330 0x7fcc40144190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:12.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.974+0000 7fcc3fe00640 1 -- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc400c6010 msgr2=0x7fcc400b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:12.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.974+0000 7fcc3fe00640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc400c6010 0x7fcc400b5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:12.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.974+0000 7fcc3fe00640 1 -- 172.21.15.40:0/3115313287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc401430f0 con 0x7fcc400c6f10 2024-04-26T23:10:12.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.974+0000 7fcc45200640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc400c6010 0x7fcc400b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:12.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.974+0000 7fcc3fe00640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc400c6f10 0x7fcc400b4df0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000c8c0 tx=0x7fcc3000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:12.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.975+0000 7fcc3cc00640 1 -- 172.21.15.40:0/3115313287 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30017070 con 0x7fcc400c6f10 2024-04-26T23:10:12.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.975+0000 7fcc3cc00640 1 -- 172.21.15.40:0/3115313287 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc30003dc0 con 0x7fcc400c6f10 2024-04-26T23:10:12.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.975+0000 7fcc3cc00640 1 -- 172.21.15.40:0/3115313287 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc300052b0 con 0x7fcc400c6f10 2024-04-26T23:10:12.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.975+0000 7fcc46600640 1 -- 172.21.15.40:0/3115313287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc401433d0 con 0x7fcc400c6f10 2024-04-26T23:10:12.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.975+0000 7fcc46600640 1 -- 172.21.15.40:0/3115313287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc40143990 con 0x7fcc400c6f10 2024-04-26T23:10:12.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.976+0000 7fcc46600640 1 -- 172.21.15.40:0/3115313287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc40069370 con 0x7fcc400c6f10 2024-04-26T23:10:12.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.977+0000 7fcc3cc00640 1 -- 172.21.15.40:0/3115313287 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcc3001a070 con 0x7fcc400c6f10 2024-04-26T23:10:12.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.978+0000 7fcc3cc00640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcc1c06c920 0x7fcc1c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:12.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.978+0000 7fcc3cc00640 1 -- 172.21.15.40:0/3115313287 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7fcc3008b2a0 con 0x7fcc400c6f10 2024-04-26T23:10:12.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.979+0000 7fcc45200640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcc1c06c920 0x7fcc1c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:12.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.980+0000 7fcc45200640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcc1c06c920 0x7fcc1c06ede0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fcc28034000 tx=0x7fcc2802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:12.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:12.983+0000 7fcc3cc00640 1 -- 172.21.15.40:0/3115313287 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fcc30059660 con 0x7fcc400c6f10 2024-04-26T23:10:13.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.187+0000 7fcc46600640 1 -- 172.21.15.40:0/3115313287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fcc400cfde0 con 0x7fcc400c6f10 2024-04-26T23:10:13.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.193+0000 7fcc3cc00640 1 -- 172.21.15.40:0/3115313287 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7fcc30058f60 con 0x7fcc400c6f10 2024-04-26T23:10:13.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.196+0000 7fcc46600640 1 -- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcc1c06c920 msgr2=0x7fcc1c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:13.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.196+0000 7fcc46600640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcc1c06c920 0x7fcc1c06ede0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fcc28034000 tx=0x7fcc2802e000 comp rx=0 tx=0).stop 2024-04-26T23:10:13.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.196+0000 7fcc46600640 1 -- 172.21.15.40:0/3115313287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc400c6f10 msgr2=0x7fcc400b4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:13.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.196+0000 7fcc46600640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc400c6f10 0x7fcc400b4df0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000c8c0 tx=0x7fcc3000cd80 comp rx=0 tx=0).stop 2024-04-26T23:10:13.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.196+0000 7fcc46600640 1 -- 172.21.15.40:0/3115313287 shutdown_connections 2024-04-26T23:10:13.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.196+0000 7fcc46600640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fcc1c06c920 0x7fcc1c06ede0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:13.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.196+0000 7fcc46600640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc400b5330 0x7fcc40144190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:13.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.196+0000 7fcc46600640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc400c6f10 0x7fcc400b4df0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:13.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.196+0000 7fcc46600640 1 --2- 172.21.15.40:0/3115313287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc400c6010 0x7fcc400b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:13.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.196+0000 7fcc46600640 1 -- 172.21.15.40:0/3115313287 >> 172.21.15.40:0/3115313287 conn(0x7fcc400d0a10 msgr2=0x7fcc4015d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:13.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.196+0000 7fcc46600640 1 -- 172.21.15.40:0/3115313287 shutdown_connections 2024-04-26T23:10:13.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.196+0000 7fcc46600640 1 -- 172.21.15.40:0/3115313287 wait complete. 2024-04-26T23:10:13.216 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_pause_and_resume_with_abort_mds0' 2024-04-26T23:10:13.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.406+0000 7f1dd6c00640 1 Processor -- start 2024-04-26T23:10:13.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.407+0000 7f1dd6c00640 1 -- start start 2024-04-26T23:10:13.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.409+0000 7f1dd6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1dd00d11e0 0x7f1dd00b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:13.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.409+0000 7f1dd6c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1dd0164650 0x7f1dd00b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:13.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.409+0000 7f1dd6c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dd01638f0 con 0x7f1dd006dea0 2024-04-26T23:10:13.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.409+0000 7f1dd6c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f1dd00d15c0 con 0x7f1dd0164650 2024-04-26T23:10:13.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.409+0000 7f1dd6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1dd006ebd0 con 0x7f1dd00d11e0 2024-04-26T23:10:13.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.409+0000 7f1dd4e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1dd0164650 0x7f1dd00b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.409+0000 7f1dd4e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1dd0164650 0x7f1dd00b3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38628/0 (socket says 172.21.15.40:38628) 2024-04-26T23:10:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.409+0000 7f1dd4e00640 1 -- 172.21.15.40:0/2167597742 learned_addr learned my addr 172.21.15.40:0/2167597742 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:13.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.409+0000 7f1dd6200640 1 --1- 172.21.15.40:0/2167597742 >> v1:172.21.15.40:6789/0 conn(0x7f1dd006dea0 0x7f1dd006e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51944/0 (socket says 172.21.15.40:51944) 2024-04-26T23:10:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.410+0000 7f1dd5800640 1 --2- 172.21.15.40:0/2167597742 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1dd00d11e0 0x7f1dd00b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.410+0000 7f1dd4e00640 1 -- 172.21.15.40:0/2167597742 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1dd00d11e0 msgr2=0x7f1dd00b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.410+0000 7f1dd4e00640 1 --2- 172.21.15.40:0/2167597742 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1dd00d11e0 0x7f1dd00b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.410+0000 7f1dd4e00640 1 -- 172.21.15.40:0/2167597742 >> v1:172.21.15.40:6789/0 conn(0x7f1dd006dea0 legacy=0x7f1dd006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.410+0000 7f1dd4e00640 1 -- 172.21.15.40:0/2167597742 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd0163b70 con 0x7f1dd0164650 2024-04-26T23:10:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.410+0000 7f1dcfe00640 1 -- 172.21.15.40:0/2167597742 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3643367693 0 0) 0x7f1dc8001490 con 0x7f1dd006dea0 2024-04-26T23:10:13.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.410+0000 7f1dd5800640 1 --2- 172.21.15.40:0/2167597742 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1dd00d11e0 0x7f1dd00b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:13.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.411+0000 7f1dd4e00640 1 --2- 172.21.15.40:0/2167597742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd0164650 0x7f1dd00b3fb0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009870 tx=0x7f1dbc02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184c1cc678ba3f74 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:13.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.411+0000 7f1dcfe00640 1 -- 172.21.15.40:0/2167597742 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc037070 con 0x7f1dd0164650 2024-04-26T23:10:13.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.411+0000 7f1dcfe00640 1 -- 172.21.15.40:0/2167597742 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dbc032440 con 0x7f1dd0164650 2024-04-26T23:10:13.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.411+0000 7f1dcfe00640 1 -- 172.21.15.40:0/2167597742 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc032810 con 0x7f1dd0164650 2024-04-26T23:10:13.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.411+0000 7f1dd6c00640 1 -- 172.21.15.40:0/2167597742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd0164650 msgr2=0x7f1dd00b3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.411+0000 7f1dd6c00640 1 --2- 172.21.15.40:0/2167597742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd0164650 0x7f1dd00b3fb0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009870 tx=0x7f1dbc02a250 comp rx=0 tx=0).stop 2024-04-26T23:10:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.412+0000 7f1dd6c00640 1 -- 172.21.15.40:0/2167597742 shutdown_connections 2024-04-26T23:10:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.412+0000 7f1dd6c00640 1 --2- 172.21.15.40:0/2167597742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd0164650 0x7f1dd00b3fb0 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.412+0000 7f1dd6c00640 1 --2- 172.21.15.40:0/2167597742 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1dd00d11e0 0x7f1dd00b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.412+0000 7f1dd6c00640 1 -- 172.21.15.40:0/2167597742 >> 172.21.15.40:0/2167597742 conn(0x7f1dd0075eb0 msgr2=0x7f1dd0068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.412+0000 7f1dd6c00640 1 -- 172.21.15.40:0/2167597742 shutdown_connections 2024-04-26T23:10:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.412+0000 7f1dd6c00640 1 -- 172.21.15.40:0/2167597742 wait complete. 2024-04-26T23:10:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.413+0000 7f1dd6c00640 1 Processor -- start 2024-04-26T23:10:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.414+0000 7f1dd6c00640 1 -- start start 2024-04-26T23:10:13.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.414+0000 7f1dd6c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dd006dea0 0x7f1dd0158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:13.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.414+0000 7f1dd6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1dd00d11e0 0x7f1dd015a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:13.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.414+0000 7f1dd6c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd0164650 0x7f1dd015fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:13.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.415+0000 7f1dd6c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd0166ee0 con 0x7f1dd0164650 2024-04-26T23:10:13.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.415+0000 7f1dd6c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd0164130 con 0x7f1dd006dea0 2024-04-26T23:10:13.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.415+0000 7f1dd6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1dd0167060 con 0x7f1dd00d11e0 2024-04-26T23:10:13.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.415+0000 7f1dd6200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd0164650 0x7f1dd015fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:13.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.415+0000 7f1dd6200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd0164650 0x7f1dd015fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38630/0 (socket says 172.21.15.40:38630) 2024-04-26T23:10:13.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.415+0000 7f1dd6200640 1 -- 172.21.15.40:0/3697807646 learned_addr learned my addr 172.21.15.40:0/3697807646 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:13.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.415+0000 7f1dd4e00640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1dd00d11e0 0x7f1dd015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:13.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.415+0000 7f1dd5800640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dd006dea0 0x7f1dd0158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:13.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.415+0000 7f1dd6200640 1 -- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1dd00d11e0 msgr2=0x7f1dd015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:13.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.415+0000 7f1dd6200640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1dd00d11e0 0x7f1dd015a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:13.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.415+0000 7f1dd6200640 1 -- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dd006dea0 msgr2=0x7f1dd0158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:13.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.415+0000 7f1dd6200640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dd006dea0 0x7f1dd0158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:13.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.415+0000 7f1dd6200640 1 -- 172.21.15.40:0/3697807646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd0160970 con 0x7f1dd0164650 2024-04-26T23:10:13.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.416+0000 7f1dd6200640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd0164650 0x7f1dd015fef0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800c780 tx=0x7f1dc800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:13.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.416+0000 7f1dcd600640 1 -- 172.21.15.40:0/3697807646 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8018070 con 0x7f1dd0164650 2024-04-26T23:10:13.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.416+0000 7f1dcd600640 1 -- 172.21.15.40:0/3697807646 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dc800a640 con 0x7f1dd0164650 2024-04-26T23:10:13.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.416+0000 7f1dcd600640 1 -- 172.21.15.40:0/3697807646 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc800aa30 con 0x7f1dd0164650 2024-04-26T23:10:13.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.416+0000 7f1dd6c00640 1 -- 172.21.15.40:0/3697807646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd00aad40 con 0x7f1dd0164650 2024-04-26T23:10:13.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.416+0000 7f1dd6c00640 1 -- 172.21.15.40:0/3697807646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dd00ab300 con 0x7f1dd0164650 2024-04-26T23:10:13.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.417+0000 7f1dd6c00640 1 -- 172.21.15.40:0/3697807646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dd0069370 con 0x7f1dd0164650 2024-04-26T23:10:13.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.418+0000 7f1dcd600640 1 -- 172.21.15.40:0/3697807646 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1dc800abd0 con 0x7f1dd0164650 2024-04-26T23:10:13.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.419+0000 7f1dcd600640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1db006c8f0 0x7f1db006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:13.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.419+0000 7f1dcd600640 1 -- 172.21.15.40:0/3697807646 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f1dc808af50 con 0x7f1dd0164650 2024-04-26T23:10:13.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.420+0000 7f1dd5800640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1db006c8f0 0x7f1db006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:13.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.421+0000 7f1dd5800640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1db006c8f0 0x7f1db006edb0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4008850 tx=0x7f1dc4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:13.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.425+0000 7f1dcd600640 1 -- 172.21.15.40:0/3697807646 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f1dd01638f0 con 0x7f1dd0164650 2024-04-26T23:10:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.692+0000 7f1dd6c00640 1 -- 172.21.15.40:0/3697807646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_pause_and_resume_with_abort_mds0"]} v 0) -- 0x7f1dd006f3e0 con 0x7f1dd0164650 2024-04-26T23:10:13.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.753+0000 7f1dcd600640 1 -- 172.21.15.40:0/3697807646 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_multimds_misc.TestScrub2.test_scrub_pause_and_resume_with_abort_mds0"]}]=0 v482) ==== 165+0+0 (secure 0 0 0) 0x7f1dc80594d0 con 0x7f1dd0164650 2024-04-26T23:10:13.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.756+0000 7f1dd6c00640 1 -- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1db006c8f0 msgr2=0x7f1db006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:13.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.756+0000 7f1dd6c00640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1db006c8f0 0x7f1db006edb0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4008850 tx=0x7f1dc4008040 comp rx=0 tx=0).stop 2024-04-26T23:10:13.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.756+0000 7f1dd6c00640 1 -- 172.21.15.40:0/3697807646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd0164650 msgr2=0x7f1dd015fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:13.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.756+0000 7f1dd6c00640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd0164650 0x7f1dd015fef0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800c780 tx=0x7f1dc800cc40 comp rx=0 tx=0).stop 2024-04-26T23:10:13.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.756+0000 7f1dd6c00640 1 -- 172.21.15.40:0/3697807646 shutdown_connections 2024-04-26T23:10:13.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.756+0000 7f1dd6c00640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f1db006c8f0 0x7f1db006edb0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:13.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.756+0000 7f1dd6c00640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dd0164650 0x7f1dd015fef0 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:13.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.756+0000 7f1dd6c00640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1dd00d11e0 0x7f1dd015a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:13.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.756+0000 7f1dd6c00640 1 --2- 172.21.15.40:0/3697807646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dd006dea0 0x7f1dd0158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:13.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.756+0000 7f1dd6c00640 1 -- 172.21.15.40:0/3697807646 >> 172.21.15.40:0/3697807646 conn(0x7f1dd0075eb0 msgr2=0x7f1dd0165150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:13.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.756+0000 7f1dd6c00640 1 -- 172.21.15.40:0/3697807646 shutdown_connections 2024-04-26T23:10:13.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:13.757+0000 7f1dd6c00640 1 -- 172.21.15.40:0/3697807646 wait complete. 2024-04-26T23:10:13.776 INFO:tasks.cephfs_test_runner:test_scrub_pause_and_resume_with_abort_mds0 (tasks.cephfs.test_multimds_misc.TestScrub2) ... ok 2024-04-26T23:10:13.776 INFO:tasks.cephfs_test_runner: 2024-04-26T23:10:13.776 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-26T23:10:13.776 INFO:tasks.cephfs_test_runner:Ran 6 tests in 480.552s 2024-04-26T23:10:13.777 INFO:tasks.cephfs_test_runner: 2024-04-26T23:10:13.777 INFO:tasks.cephfs_test_runner:OK 2024-04-26T23:10:13.777 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-26T23:10:13.787 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-26T23:10:13.794 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T23:10:13.795 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:10:13.795 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:10:13.848 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:10:13.848 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:10:13.848 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:10:13.904 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-26T23:10:13.904 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:10:13.904 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:10:13.962 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:10:13.962 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:10:13.962 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:10:14.018 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-26T23:10:14.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:14.018 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:10:14.078 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T23:10:14.079 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-26T23:10:14.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:14.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T23:10:14.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:14.173 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-26T23:10:14.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:14.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2024-04-26T23:10:14.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:14.372 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:10:14.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:14.416 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2024-04-26T23:10:14.546 INFO:teuthology.orchestra.run.smithi040.stderr:Cannot find device "ceph-brx" 2024-04-26T23:10:14.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:10:14.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:14.550 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-26T23:10:14.608 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-26T23:10:14.608 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-26T23:10:14.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:14.609 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-26T23:10:14.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:14.698 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-26T23:10:14.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:14.772 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-04-26T23:10:14.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:14.899 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-26T23:10:14.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:10:14.941 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-04-26T23:10:15.067 INFO:teuthology.orchestra.run.smithi103.stderr:Cannot find device "ceph-brx" 2024-04-26T23:10:15.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:10:15.071 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-26T23:10:15.080 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T23:10:15.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:10:15.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.302+0000 7f0e67200640 1 Processor -- start 2024-04-26T23:10:15.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.302+0000 7f0e67200640 1 -- start start 2024-04-26T23:10:15.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.304+0000 7f0e67200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0e680af7a0 0x7f0e680afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.304+0000 7f0e67200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e680d0820 con 0x7f0e6815ce60 2024-04-26T23:10:15.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.304+0000 7f0e67200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e681614c0 con 0x7f0e68071fe0 2024-04-26T23:10:15.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.304+0000 7f0e67200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0e680cc450 con 0x7f0e680af7a0 2024-04-26T23:10:15.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.304+0000 7f0e65e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0e6815ce60 0x7f0e680ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45860/0 (socket says 172.21.15.40:45860) 2024-04-26T23:10:15.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.304+0000 7f0e65e00640 1 -- 172.21.15.40:0/2383961115 learned_addr learned my addr 172.21.15.40:0/2383961115 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:15.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.305+0000 7f0e5d400640 1 --2- 172.21.15.40:0/2383961115 >> v2:172.21.15.103:3300/0 conn(0x7f0e680af7a0 0x7f0e680afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.305+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1783485021 0 0) 0x7f0e680d0820 con 0x7f0e6815ce60 2024-04-26T23:10:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.305+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e48003520 con 0x7f0e6815ce60 2024-04-26T23:10:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.305+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4215108595 0 0) 0x7f0e48003520 con 0x7f0e6815ce60 2024-04-26T23:10:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.305+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 >> v2:172.21.15.103:3300/0 conn(0x7f0e680af7a0 msgr2=0x7f0e680afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.305+0000 7f0e65400640 1 --2- 172.21.15.40:0/2383961115 >> v2:172.21.15.103:3300/0 conn(0x7f0e680af7a0 0x7f0e680afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.305+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 >> v1:172.21.15.103:6789/0 conn(0x7f0e68071fe0 legacy=0x7f0e68073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.305+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e68161760 con 0x7f0e6815ce60 2024-04-26T23:10:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.305+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f0e58001670 con 0x7f0e6815ce60 2024-04-26T23:10:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.305+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 >> v1:172.21.15.40:6789/0 conn(0x7f0e6815ce60 legacy=0x7f0e680ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.305+0000 7f0e65400640 1 --2- 172.21.15.40:0/2383961115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e48003a00 0x7f0e48003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.305+0000 7f0e65400640 1 --2- 172.21.15.40:0/2383961115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e48004800 0x7f0e48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.305+0000 7f0e65400640 1 --2- 172.21.15.40:0/2383961115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e48005210 0x7f0e4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.306+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0e680cc450 con 0x7f0e48004800 2024-04-26T23:10:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.306+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4800c160 con 0x7f0e48003a00 2024-04-26T23:10:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.306+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0e4800c360 con 0x7f0e48005210 2024-04-26T23:10:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.306+0000 7f0e5d400640 1 --2- 172.21.15.40:0/2383961115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e48004800 0x7f0e48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.306+0000 7f0e65e00640 1 --2- 172.21.15.40:0/2383961115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e48003a00 0x7f0e48003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.306+0000 7f0e5d400640 1 -- 172.21.15.40:0/2383961115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e48005210 msgr2=0x7f0e4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.306+0000 7f0e5d400640 1 --2- 172.21.15.40:0/2383961115 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e48005210 0x7f0e4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.306+0000 7f0e5d400640 1 -- 172.21.15.40:0/2383961115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e48003a00 msgr2=0x7f0e48003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.306+0000 7f0e5d400640 1 --2- 172.21.15.40:0/2383961115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e48003a00 0x7f0e48003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.306+0000 7f0e5d400640 1 -- 172.21.15.40:0/2383961115 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e48003520 con 0x7f0e48004800 2024-04-26T23:10:15.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.306+0000 7f0e66800640 1 -- 172.21.15.40:0/2383961115 reap_dead start 2024-04-26T23:10:15.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.307+0000 7f0e5d400640 1 --2- 172.21.15.40:0/2383961115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e48004800 0x7f0e48004c80 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009a90 tx=0x7f0e48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a803649c50c3c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:15.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.308+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e50036070 con 0x7f0e48004800 2024-04-26T23:10:15.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.308+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e50031400 con 0x7f0e48004800 2024-04-26T23:10:15.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.308+0000 7f0e65400640 1 -- 172.21.15.40:0/2383961115 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e50009690 con 0x7f0e48004800 2024-04-26T23:10:15.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.308+0000 7f0e67200640 1 -- 172.21.15.40:0/2383961115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e48004800 msgr2=0x7f0e48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:15.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.308+0000 7f0e67200640 1 --2- 172.21.15.40:0/2383961115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e48004800 0x7f0e48004c80 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009a90 tx=0x7f0e48005830 comp rx=0 tx=0).stop 2024-04-26T23:10:15.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.308+0000 7f0e67200640 1 -- 172.21.15.40:0/2383961115 shutdown_connections 2024-04-26T23:10:15.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.308+0000 7f0e67200640 1 --2- 172.21.15.40:0/2383961115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e48004800 0x7f0e48004c80 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.308+0000 7f0e67200640 1 -- 172.21.15.40:0/2383961115 >> 172.21.15.40:0/2383961115 conn(0x7f0e680d0a10 msgr2=0x7f0e680d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:15.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.308+0000 7f0e67200640 1 -- 172.21.15.40:0/2383961115 shutdown_connections 2024-04-26T23:10:15.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.308+0000 7f0e67200640 1 -- 172.21.15.40:0/2383961115 wait complete. 2024-04-26T23:10:15.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.310+0000 7f0e67200640 1 Processor -- start 2024-04-26T23:10:15.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.310+0000 7f0e67200640 1 -- start start 2024-04-26T23:10:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.311+0000 7f0e67200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e48004800 0x7f0e68151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.311+0000 7f0e67200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e68149760 0x7f0e68149100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.311+0000 7f0e67200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e680ae8c0 0x7f0e680aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.311+0000 7f0e67200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0e680afde0 con 0x7f0e68149760 2024-04-26T23:10:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.311+0000 7f0e67200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0e680afc60 con 0x7f0e680ae8c0 2024-04-26T23:10:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.311+0000 7f0e67200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0e680aff60 con 0x7f0e48004800 2024-04-26T23:10:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.312+0000 7f0e5d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e68149760 0x7f0e68149100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.312+0000 7f0e5d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e68149760 0x7f0e68149100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58338/0 (socket says 172.21.15.40:58338) 2024-04-26T23:10:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.312+0000 7f0e5d400640 1 -- 172.21.15.40:0/143470800 learned_addr learned my addr 172.21.15.40:0/143470800 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.312+0000 7f0e65e00640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e48004800 0x7f0e68151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.312+0000 7f0e66800640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e680ae8c0 0x7f0e680aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.312+0000 7f0e5d400640 1 -- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e48004800 msgr2=0x7f0e68151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.312+0000 7f0e5d400640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e48004800 0x7f0e68151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.312+0000 7f0e5d400640 1 -- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e680ae8c0 msgr2=0x7f0e680aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.312+0000 7f0e5d400640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e680ae8c0 0x7f0e680aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.312+0000 7f0e5d400640 1 -- 172.21.15.40:0/143470800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e680c93e0 con 0x7f0e68149760 2024-04-26T23:10:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.312+0000 7f0e66800640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e680ae8c0 0x7f0e680aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:15.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.313+0000 7f0e5d400640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e68149760 0x7f0e68149100 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f0e5002ab50 tx=0x7f0e50004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:15.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.313+0000 7f0e5ea00640 1 -- 172.21.15.40:0/143470800 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e50041020 con 0x7f0e68149760 2024-04-26T23:10:15.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.313+0000 7f0e5ea00640 1 -- 172.21.15.40:0/143470800 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e50003a40 con 0x7f0e68149760 2024-04-26T23:10:15.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.313+0000 7f0e5ea00640 1 -- 172.21.15.40:0/143470800 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e50036040 con 0x7f0e68149760 2024-04-26T23:10:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.313+0000 7f0e67200640 1 -- 172.21.15.40:0/143470800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e680c96c0 con 0x7f0e68149760 2024-04-26T23:10:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.313+0000 7f0e67200640 1 -- 172.21.15.40:0/143470800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e680c9c80 con 0x7f0e68149760 2024-04-26T23:10:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.315+0000 7f0e67200640 1 -- 172.21.15.40:0/143470800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e680af280 con 0x7f0e68149760 2024-04-26T23:10:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.315+0000 7f0e5ea00640 1 -- 172.21.15.40:0/143470800 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0e50044020 con 0x7f0e68149760 2024-04-26T23:10:15.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.316+0000 7f0e5ea00640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0e4406c920 0x7f0e4406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.316+0000 7f0e5ea00640 1 -- 172.21.15.40:0/143470800 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f0e500ac3c0 con 0x7f0e68149760 2024-04-26T23:10:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.317+0000 7f0e65e00640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0e4406c920 0x7f0e4406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.318+0000 7f0e65e00640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0e4406c920 0x7f0e4406ede0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0e58001eb0 tx=0x7f0e58009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.321+0000 7f0e5ea00640 1 -- 172.21.15.40:0/143470800 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f0e5004b030 con 0x7f0e68149760 2024-04-26T23:10:15.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.526+0000 7f0e67200640 1 -- 172.21.15.40:0/143470800 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0e6815d710 con 0x7f0e4406c920 2024-04-26T23:10:15.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.528+0000 7f0e5ea00640 1 -- 172.21.15.40:0/143470800 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13511 (secure 0 0 0) 0x7f0e58010830 con 0x7f0e4406c920 2024-04-26T23:10:15.529 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:15.529 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:10:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.531+0000 7f0e67200640 1 -- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0e4406c920 msgr2=0x7f0e4406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.531+0000 7f0e67200640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0e4406c920 0x7f0e4406ede0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0e58001eb0 tx=0x7f0e58009040 comp rx=0 tx=0).stop 2024-04-26T23:10:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.531+0000 7f0e67200640 1 -- 172.21.15.40:0/143470800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e68149760 msgr2=0x7f0e68149100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.531+0000 7f0e67200640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e68149760 0x7f0e68149100 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f0e5002ab50 tx=0x7f0e50004730 comp rx=0 tx=0).stop 2024-04-26T23:10:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.531+0000 7f0e67200640 1 -- 172.21.15.40:0/143470800 shutdown_connections 2024-04-26T23:10:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.531+0000 7f0e67200640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f0e4406c920 0x7f0e4406ede0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.531+0000 7f0e67200640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e680ae8c0 0x7f0e680aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.531+0000 7f0e67200640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e68149760 0x7f0e68149100 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.531+0000 7f0e67200640 1 --2- 172.21.15.40:0/143470800 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0e48004800 0x7f0e68151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.531+0000 7f0e67200640 1 -- 172.21.15.40:0/143470800 >> 172.21.15.40:0/143470800 conn(0x7f0e680d0a10 msgr2=0x7f0e680cbbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.531+0000 7f0e67200640 1 -- 172.21.15.40:0/143470800 shutdown_connections 2024-04-26T23:10:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.532+0000 7f0e67200640 1 -- 172.21.15.40:0/143470800 wait complete. 2024-04-26T23:10:15.551 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-04-26T23:10:15.349925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":261088,"kb_used_data":11624,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713632,"statfs":{"total":773094113280,"available":772826759168,"internally_reserved":0,"allocated":11902976,"data_stored":6199752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-125848,"num_objects":-68,"num_object_clones":0,"num_object_copies":-136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-68,"num_whiteouts":0,"num_read":-1,"num_read_kb":0,"num_write":-258,"num_write_kb":-211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002301"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":340,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-26T23:10:12.749726+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:10:12.749726+0000","last_peered":"2024-04-26T23:10:12.749726+0000","last_clean":"2024-04-26T23:10:12.749726+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:10:12.749726+0000","last_undegraded":"2024-04-26T23:10:12.749726+0000","last_fullsized":"2024-04-26T23:10:12.749726+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574962,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1941504,"data_stored":1220818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574962,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574962,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32484,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1331200,"data_stored":624304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574962,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32484,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1331200,"data_stored":624304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32484,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1331200,"data_stored":624304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574963,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1925120,"data_stored":1214672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:10:15.551 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:10:15.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.766+0000 7f970ae00640 1 Processor -- start 2024-04-26T23:10:15.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.766+0000 7f970ae00640 1 -- start start 2024-04-26T23:10:15.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.767+0000 7f970ae00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f970415f540 0x7f97040d3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.767+0000 7f970ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97040b5ab0 0x7f97040d39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.767+0000 7f970ae00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97040d2d70 con 0x7f97040b3ff0 2024-04-26T23:10:15.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.767+0000 7f970ae00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f970415f920 con 0x7f970415f540 2024-04-26T23:10:15.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.767+0000 7f970ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f97040d0f30 con 0x7f97040b5ab0 2024-04-26T23:10:15.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.768+0000 7f9703400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f97040b3ff0 0x7f97040d1770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45870/0 (socket says 172.21.15.40:45870) 2024-04-26T23:10:15.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.768+0000 7f9703400640 1 -- 172.21.15.40:0/3611297823 learned_addr learned my addr 172.21.15.40:0/3611297823 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:15.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.768+0000 7f9708e00640 1 --2- 172.21.15.40:0/3611297823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97040b5ab0 0x7f97040d39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.768+0000 7f9703e00640 1 --2- 172.21.15.40:0/3611297823 >> v2:172.21.15.103:3300/0 conn(0x7f970415f540 0x7f97040d3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.768+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 758937469 0 0) 0x7f97040d2d70 con 0x7f97040b3ff0 2024-04-26T23:10:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.768+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96e8003520 con 0x7f97040b3ff0 2024-04-26T23:10:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 392557142 0 0) 0x7f96e8003520 con 0x7f97040b3ff0 2024-04-26T23:10:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97040b5ab0 msgr2=0x7f97040d39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 --2- 172.21.15.40:0/3611297823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97040b5ab0 0x7f97040d39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 >> v2:172.21.15.103:3300/0 conn(0x7f970415f540 msgr2=0x7f97040d3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 --2- 172.21.15.40:0/3611297823 >> v2:172.21.15.103:3300/0 conn(0x7f970415f540 0x7f97040d3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97040d3050 con 0x7f97040b3ff0 2024-04-26T23:10:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f96ec001670 con 0x7f97040b3ff0 2024-04-26T23:10:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 >> v1:172.21.15.40:6789/0 conn(0x7f97040b3ff0 legacy=0x7f97040d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 --2- 172.21.15.40:0/3611297823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96e80039b0 0x7f96e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 --2- 172.21.15.40:0/3611297823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97040b5ab0 0x7f96e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 --2- 172.21.15.40:0/3611297823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96e8004d40 0x7f96e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f970415f920 con 0x7f96e80039b0 2024-04-26T23:10:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f97040d0f30 con 0x7f96e8004d40 2024-04-26T23:10:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f96e800b9c0 con 0x7f97040b5ab0 2024-04-26T23:10:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9703e00640 1 --2- 172.21.15.40:0/3611297823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96e80039b0 0x7f96e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9703e00640 1 -- 172.21.15.40:0/3611297823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97040b5ab0 msgr2=0x7f96e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9703e00640 1 --2- 172.21.15.40:0/3611297823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97040b5ab0 0x7f96e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9703e00640 1 -- 172.21.15.40:0/3611297823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96e8004d40 msgr2=0x7f96e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.769+0000 7f9703e00640 1 --2- 172.21.15.40:0/3611297823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96e8004d40 0x7f96e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.770+0000 7f9703e00640 1 -- 172.21.15.40:0/3611297823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e8003520 con 0x7f96e80039b0 2024-04-26T23:10:15.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.771+0000 7f9703e00640 1 --2- 172.21.15.40:0/3611297823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96e80039b0 0x7f96e8003dd0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f96f4009fc0 tx=0x7f96e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ddbf550301f883d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:15.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.771+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4009c10 con 0x7f96e80039b0 2024-04-26T23:10:15.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.771+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96f4036070 con 0x7f96e80039b0 2024-04-26T23:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.771+0000 7f9702a00640 1 -- 172.21.15.40:0/3611297823 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f402d070 con 0x7f96e80039b0 2024-04-26T23:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.771+0000 7f970ae00640 1 -- 172.21.15.40:0/3611297823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96e80039b0 msgr2=0x7f96e8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.771+0000 7f970ae00640 1 --2- 172.21.15.40:0/3611297823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96e80039b0 0x7f96e8003dd0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f96f4009fc0 tx=0x7f96e8005830 comp rx=0 tx=0).stop 2024-04-26T23:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.771+0000 7f9708e00640 1 -- 172.21.15.40:0/3611297823 reap_dead start 2024-04-26T23:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.772+0000 7f970ae00640 1 -- 172.21.15.40:0/3611297823 shutdown_connections 2024-04-26T23:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.772+0000 7f970ae00640 1 -- 172.21.15.40:0/3611297823 >> 172.21.15.40:0/3611297823 conn(0x7f97040d4f90 msgr2=0x7f97040d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.772+0000 7f970ae00640 1 -- 172.21.15.40:0/3611297823 shutdown_connections 2024-04-26T23:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.772+0000 7f970ae00640 1 -- 172.21.15.40:0/3611297823 wait complete. 2024-04-26T23:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.773+0000 7f970ae00640 1 Processor -- start 2024-04-26T23:10:15.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.774+0000 7f970ae00640 1 -- start start 2024-04-26T23:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f970ae00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f970415c980 0x7f97041f85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f970ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97040c93b0 0x7f970406cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f970ae00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f970406d430 0x7f970415faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f970ae00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9704068e50 con 0x7f970415c980 2024-04-26T23:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f970ae00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f97040d0bc0 con 0x7f970406d430 2024-04-26T23:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f970ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f97040d39b0 con 0x7f97040c93b0 2024-04-26T23:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f9703e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f970415c980 0x7f97041f85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f9703e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f970415c980 0x7f97041f85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58368/0 (socket says 172.21.15.40:58368) 2024-04-26T23:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f9703e00640 1 -- 172.21.15.40:0/2196603498 learned_addr learned my addr 172.21.15.40:0/2196603498 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f9703400640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97040c93b0 0x7f970406cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f9708e00640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f970406d430 0x7f970415faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f9703e00640 1 -- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97040c93b0 msgr2=0x7f970406cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f9703e00640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97040c93b0 0x7f970406cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f9703e00640 1 -- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f970406d430 msgr2=0x7f970415faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f9703e00640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f970406d430 0x7f970415faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.775+0000 7f9703e00640 1 -- 172.21.15.40:0/2196603498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97041f8e00 con 0x7f970415c980 2024-04-26T23:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.776+0000 7f9703e00640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f970415c980 0x7f97041f85d0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f96f400e230 tx=0x7f96f4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.776+0000 7f96fbe00640 1 -- 172.21.15.40:0/2196603498 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4009c10 con 0x7f970415c980 2024-04-26T23:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.776+0000 7f96fbe00640 1 -- 172.21.15.40:0/2196603498 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96f4003b90 con 0x7f970415c980 2024-04-26T23:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.776+0000 7f96fbe00640 1 -- 172.21.15.40:0/2196603498 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4004020 con 0x7f970415c980 2024-04-26T23:10:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.776+0000 7f970ae00640 1 -- 172.21.15.40:0/2196603498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97041f90e0 con 0x7f970415c980 2024-04-26T23:10:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.776+0000 7f970ae00640 1 -- 172.21.15.40:0/2196603498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9704160f30 con 0x7f970415c980 2024-04-26T23:10:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.778+0000 7f970ae00640 1 -- 172.21.15.40:0/2196603498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97040694f0 con 0x7f970415c980 2024-04-26T23:10:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.778+0000 7f96fbe00640 1 -- 172.21.15.40:0/2196603498 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f96f4005d10 con 0x7f970415c980 2024-04-26T23:10:15.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.779+0000 7f96fbe00640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f96e006c920 0x7f96e006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:15.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.779+0000 7f96fbe00640 1 -- 172.21.15.40:0/2196603498 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f96f40acd60 con 0x7f970415c980 2024-04-26T23:10:15.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.780+0000 7f9703400640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f96e006c920 0x7f96e006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:15.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.781+0000 7f9703400640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f96e006c920 0x7f96e006ede0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f96ec000ff0 tx=0x7f96ec009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:15.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.784+0000 7f96fbe00640 1 -- 172.21.15.40:0/2196603498 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f96f407b250 con 0x7f970415c980 2024-04-26T23:10:15.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.983+0000 7f970ae00640 1 -- 172.21.15.40:0/2196603498 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f97040cb960 con 0x7f96e006c920 2024-04-26T23:10:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.985+0000 7f96fbe00640 1 -- 172.21.15.40:0/2196603498 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13511 (secure 0 0 0) 0x7f97040cb960 con 0x7f96e006c920 2024-04-26T23:10:15.986 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:15.986 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:10:15.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.988+0000 7f970ae00640 1 -- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f96e006c920 msgr2=0x7f96e006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:15.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.988+0000 7f970ae00640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f96e006c920 0x7f96e006ede0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f96ec000ff0 tx=0x7f96ec009040 comp rx=0 tx=0).stop 2024-04-26T23:10:15.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.988+0000 7f970ae00640 1 -- 172.21.15.40:0/2196603498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f970415c980 msgr2=0x7f97041f85d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.988+0000 7f970ae00640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f970415c980 0x7f97041f85d0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f96f400e230 tx=0x7f96f4009750 comp rx=0 tx=0).stop 2024-04-26T23:10:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.988+0000 7f970ae00640 1 -- 172.21.15.40:0/2196603498 shutdown_connections 2024-04-26T23:10:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.988+0000 7f970ae00640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f96e006c920 0x7f96e006ede0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.988+0000 7f970ae00640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f970406d430 0x7f970415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.988+0000 7f970ae00640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97040c93b0 0x7f970406cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.988+0000 7f970ae00640 1 --2- 172.21.15.40:0/2196603498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f970415c980 0x7f97041f85d0 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.988+0000 7f970ae00640 1 -- 172.21.15.40:0/2196603498 >> 172.21.15.40:0/2196603498 conn(0x7f97040d4f90 msgr2=0x7f97040b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.988+0000 7f970ae00640 1 -- 172.21.15.40:0/2196603498 shutdown_connections 2024-04-26T23:10:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:15.988+0000 7f970ae00640 1 -- 172.21.15.40:0/2196603498 wait complete. 2024-04-26T23:10:16.007 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-04-26T23:10:15.349925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":261088,"kb_used_data":11624,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713632,"statfs":{"total":773094113280,"available":772826759168,"internally_reserved":0,"allocated":11902976,"data_stored":6199752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-125848,"num_objects":-68,"num_object_clones":0,"num_object_copies":-136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-68,"num_whiteouts":0,"num_read":-1,"num_read_kb":0,"num_write":-258,"num_write_kb":-211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002301"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":340,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-26T23:10:12.749726+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:10:12.749726+0000","last_peered":"2024-04-26T23:10:12.749726+0000","last_clean":"2024-04-26T23:10:12.749726+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:10:12.749726+0000","last_undegraded":"2024-04-26T23:10:12.749726+0000","last_fullsized":"2024-04-26T23:10:12.749726+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574962,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1941504,"data_stored":1220818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574962,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574962,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32484,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1331200,"data_stored":624304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574962,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32484,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1331200,"data_stored":624304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32484,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1331200,"data_stored":624304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574963,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1925120,"data_stored":1214672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:10:16.008 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T23:10:16.008 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:10:16.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.211+0000 7f2fd2200640 1 Processor -- start 2024-04-26T23:10:16.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.211+0000 7f2fd2200640 1 -- start start 2024-04-26T23:10:16.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.212+0000 7f2fd2200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fcc0cc480 0x7f2fcc072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.213+0000 7f2fd2200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fcc15d7c0 con 0x7f2fcc071e60 2024-04-26T23:10:16.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.213+0000 7f2fd2200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fcc15daa0 con 0x7f2fcc072780 2024-04-26T23:10:16.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.213+0000 7f2fd2200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2fcc15dd80 con 0x7f2fcc0cc480 2024-04-26T23:10:16.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.213+0000 7f2fcb400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2fcc071e60 0x7f2fcc06f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45872/0 (socket says 172.21.15.40:45872) 2024-04-26T23:10:16.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.213+0000 7f2fcb400640 1 -- 172.21.15.40:0/505769962 learned_addr learned my addr 172.21.15.40:0/505769962 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:16.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.213+0000 7f2fcaa00640 1 --2- 172.21.15.40:0/505769962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fcc0cc480 0x7f2fcc072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:16.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.213+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1659455654 0 0) 0x7f2fcc15d7c0 con 0x7f2fcc071e60 2024-04-26T23:10:16.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.213+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fb0003520 con 0x7f2fcc071e60 2024-04-26T23:10:16.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 418616419 0 0) 0x7f2fb0003520 con 0x7f2fcc071e60 2024-04-26T23:10:16.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fcc0cc480 msgr2=0x7f2fcc072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:16.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fca000640 1 --2- 172.21.15.40:0/505769962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fcc0cc480 0x7f2fcc072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 >> v1:172.21.15.103:6789/0 conn(0x7f2fcc072780 legacy=0x7f2fcc166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fcc15e060 con 0x7f2fcc071e60 2024-04-26T23:10:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f2fbc0019f0 con 0x7f2fcc071e60 2024-04-26T23:10:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 >> v1:172.21.15.40:6789/0 conn(0x7f2fcc071e60 legacy=0x7f2fcc06f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fca000640 1 --2- 172.21.15.40:0/505769962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fcc0cc480 0x7f2fb00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fca000640 1 --2- 172.21.15.40:0/505769962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb0003fb0 0x7f2fb0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fca000640 1 --2- 172.21.15.40:0/505769962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb0004d40 0x7f2fb00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc15dd80 con 0x7f2fb0004d40 2024-04-26T23:10:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb000bb00 con 0x7f2fb0003fb0 2024-04-26T23:10:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb000bd00 con 0x7f2fcc0cc480 2024-04-26T23:10:16.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fcbe00640 1 --2- 172.21.15.40:0/505769962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb0004d40 0x7f2fb00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:16.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.214+0000 7f2fcaa00640 1 --2- 172.21.15.40:0/505769962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb0003fb0 0x7f2fb0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:16.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.215+0000 7f2fcb400640 1 --2- 172.21.15.40:0/505769962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fcc0cc480 0x7f2fb00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:16.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.215+0000 7f2fcbe00640 1 -- 172.21.15.40:0/505769962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fcc0cc480 msgr2=0x7f2fb00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:16.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.215+0000 7f2fcbe00640 1 --2- 172.21.15.40:0/505769962 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fcc0cc480 0x7f2fb00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.215+0000 7f2fcbe00640 1 -- 172.21.15.40:0/505769962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb0003fb0 msgr2=0x7f2fb0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:16.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.215+0000 7f2fcbe00640 1 --2- 172.21.15.40:0/505769962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb0003fb0 0x7f2fb0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.215+0000 7f2fcbe00640 1 -- 172.21.15.40:0/505769962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb0003520 con 0x7f2fb0004d40 2024-04-26T23:10:16.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.215+0000 7f2fcaa00640 1 --2- 172.21.15.40:0/505769962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb0003fb0 0x7f2fb0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:10:16.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.216+0000 7f2fcbe00640 1 --2- 172.21.15.40:0/505769962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb0004d40 0x7f2fb00051c0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4001750 tx=0x7f2fb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ff2cc657ae4a43 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:16.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.216+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4001370 con 0x7f2fb0004d40 2024-04-26T23:10:16.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.216+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fc40052d0 con 0x7f2fb0004d40 2024-04-26T23:10:16.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.216+0000 7f2fca000640 1 -- 172.21.15.40:0/505769962 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc402abe0 con 0x7f2fb0004d40 2024-04-26T23:10:16.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.217+0000 7f2fd2200640 1 -- 172.21.15.40:0/505769962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb0004d40 msgr2=0x7f2fb00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:16.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.217+0000 7f2fd2200640 1 --2- 172.21.15.40:0/505769962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fb0004d40 0x7f2fb00051c0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4001750 tx=0x7f2fb0005830 comp rx=0 tx=0).stop 2024-04-26T23:10:16.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.217+0000 7f2fcbe00640 1 -- 172.21.15.40:0/505769962 reap_dead start 2024-04-26T23:10:16.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.217+0000 7f2fd2200640 1 -- 172.21.15.40:0/505769962 shutdown_connections 2024-04-26T23:10:16.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.217+0000 7f2fd2200640 1 -- 172.21.15.40:0/505769962 >> 172.21.15.40:0/505769962 conn(0x7f2fcc0d0be0 msgr2=0x7f2fcc0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:16.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.217+0000 7f2fd2200640 1 -- 172.21.15.40:0/505769962 shutdown_connections 2024-04-26T23:10:16.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.217+0000 7f2fd2200640 1 -- 172.21.15.40:0/505769962 wait complete. 2024-04-26T23:10:16.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.219+0000 7f2fd2200640 1 Processor -- start 2024-04-26T23:10:16.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.219+0000 7f2fd2200640 1 -- start start 2024-04-26T23:10:16.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.220+0000 7f2fd2200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb0004d40 0x7f2fcc0b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.220+0000 7f2fd2200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fcc15c6a0 0x7f2fcc0cd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.220+0000 7f2fd2200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fcc15b800 0x7f2fcc14eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.220+0000 7f2fd2200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc15d3f0 con 0x7f2fcc15b800 2024-04-26T23:10:16.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.220+0000 7f2fd2200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc072c10 con 0x7f2fb0004d40 2024-04-26T23:10:16.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.220+0000 7f2fd2200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2fcc15d570 con 0x7f2fcc15c6a0 2024-04-26T23:10:16.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.220+0000 7f2fcbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fcc15b800 0x7f2fcc14eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:16.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.220+0000 7f2fcbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fcc15b800 0x7f2fcc14eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58396/0 (socket says 172.21.15.40:58396) 2024-04-26T23:10:16.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.220+0000 7f2fcbe00640 1 -- 172.21.15.40:0/2347468997 learned_addr learned my addr 172.21.15.40:0/2347468997 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:16.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.221+0000 7f2fcaa00640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fcc15c6a0 0x7f2fcc0cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:16.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.221+0000 7f2fcb400640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb0004d40 0x7f2fcc0b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:16.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.221+0000 7f2fcbe00640 1 -- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fcc15c6a0 msgr2=0x7f2fcc0cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:16.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.221+0000 7f2fcbe00640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fcc15c6a0 0x7f2fcc0cd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.221+0000 7f2fcbe00640 1 -- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb0004d40 msgr2=0x7f2fcc0b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:16.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.221+0000 7f2fcbe00640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb0004d40 0x7f2fcc0b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.221+0000 7f2fcbe00640 1 -- 172.21.15.40:0/2347468997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fcc15aa80 con 0x7f2fcc15b800 2024-04-26T23:10:16.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.221+0000 7f2fcbe00640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fcc15b800 0x7f2fcc14eb30 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f2fc402a590 tx=0x7f2fc4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.222+0000 7f2fc3400640 1 -- 172.21.15.40:0/2347468997 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4037040 con 0x7f2fcc15b800 2024-04-26T23:10:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.222+0000 7f2fc3400640 1 -- 172.21.15.40:0/2347468997 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fc4040070 con 0x7f2fcc15b800 2024-04-26T23:10:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.222+0000 7f2fc3400640 1 -- 172.21.15.40:0/2347468997 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4007a30 con 0x7f2fcc15b800 2024-04-26T23:10:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.222+0000 7f2fd2200640 1 -- 172.21.15.40:0/2347468997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fcc15ad20 con 0x7f2fcc15b800 2024-04-26T23:10:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.222+0000 7f2fd2200640 1 -- 172.21.15.40:0/2347468997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fcc15b2e0 con 0x7f2fcc15b800 2024-04-26T23:10:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.223+0000 7f2fd2200640 1 -- 172.21.15.40:0/2347468997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fcc0694f0 con 0x7f2fcc15b800 2024-04-26T23:10:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.224+0000 7f2fc3400640 1 -- 172.21.15.40:0/2347468997 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2fc40052d0 con 0x7f2fcc15b800 2024-04-26T23:10:16.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.225+0000 7f2fc3400640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2fac06c920 0x7f2fac06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.225+0000 7f2fc3400640 1 -- 172.21.15.40:0/2347468997 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f2fc40ab470 con 0x7f2fcc15b800 2024-04-26T23:10:16.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.226+0000 7f2fcb400640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2fac06c920 0x7f2fac06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:16.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.227+0000 7f2fcb400640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2fac06c920 0x7f2fac06ede0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc0013d0 tx=0x7f2fbc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:16.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.230+0000 7f2fc3400640 1 -- 172.21.15.40:0/2347468997 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2fc4079830 con 0x7f2fcc15b800 2024-04-26T23:10:16.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.431+0000 7f2fd2200640 1 -- 172.21.15.40:0/2347468997 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2fcc06ff40 con 0x7f2fac06c920 2024-04-26T23:10:16.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.433+0000 7f2fc3400640 1 -- 172.21.15.40:0/2347468997 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13511 (secure 0 0 0) 0x7f2fcc06ff40 con 0x7f2fac06c920 2024-04-26T23:10:16.434 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:16.434 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:10:16.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.436+0000 7f2fd2200640 1 -- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2fac06c920 msgr2=0x7f2fac06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:16.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.436+0000 7f2fd2200640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2fac06c920 0x7f2fac06ede0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc0013d0 tx=0x7f2fbc009040 comp rx=0 tx=0).stop 2024-04-26T23:10:16.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.436+0000 7f2fd2200640 1 -- 172.21.15.40:0/2347468997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fcc15b800 msgr2=0x7f2fcc14eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:16.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.436+0000 7f2fd2200640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fcc15b800 0x7f2fcc14eb30 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f2fc402a590 tx=0x7f2fc4001120 comp rx=0 tx=0).stop 2024-04-26T23:10:16.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.437+0000 7f2fd2200640 1 -- 172.21.15.40:0/2347468997 shutdown_connections 2024-04-26T23:10:16.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.437+0000 7f2fd2200640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2fac06c920 0x7f2fac06ede0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.437+0000 7f2fd2200640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fcc15b800 0x7f2fcc14eb30 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.437+0000 7f2fd2200640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fcc15c6a0 0x7f2fcc0cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.437+0000 7f2fd2200640 1 --2- 172.21.15.40:0/2347468997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fb0004d40 0x7f2fcc0b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.437+0000 7f2fd2200640 1 -- 172.21.15.40:0/2347468997 >> 172.21.15.40:0/2347468997 conn(0x7f2fcc0d0be0 msgr2=0x7f2fcc159340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:16.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.437+0000 7f2fd2200640 1 -- 172.21.15.40:0/2347468997 shutdown_connections 2024-04-26T23:10:16.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.437+0000 7f2fd2200640 1 -- 172.21.15.40:0/2347468997 wait complete. 2024-04-26T23:10:16.456 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-04-26T23:10:15.349925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":261088,"kb_used_data":11624,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713632,"statfs":{"total":773094113280,"available":772826759168,"internally_reserved":0,"allocated":11902976,"data_stored":6199752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-125848,"num_objects":-68,"num_object_clones":0,"num_object_copies":-136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-68,"num_whiteouts":0,"num_read":-1,"num_read_kb":0,"num_write":-258,"num_write_kb":-211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002301"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":340,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-26T23:10:12.749726+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:10:12.749726+0000","last_peered":"2024-04-26T23:10:12.749726+0000","last_clean":"2024-04-26T23:10:12.749726+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:10:12.749726+0000","last_undegraded":"2024-04-26T23:10:12.749726+0000","last_fullsized":"2024-04-26T23:10:12.749726+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574962,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1941504,"data_stored":1220818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574962,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574962,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32484,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1331200,"data_stored":624304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574962,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32484,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1331200,"data_stored":624304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32484,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1331200,"data_stored":624304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574963,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1925120,"data_stored":1214672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:10:16.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:10:16.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.641+0000 7f389f200640 1 Processor -- start 2024-04-26T23:10:16.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.641+0000 7f389f200640 1 -- start start 2024-04-26T23:10:16.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.642+0000 7f389f200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f389815f380 0x7f38980cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.642+0000 7f389f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38980d0620 0x7f38980cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.642+0000 7f389f200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3898072c00 con 0x7f3898073b70 2024-04-26T23:10:16.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.642+0000 7f389f200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f38980729b0 con 0x7f389815f380 2024-04-26T23:10:16.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.642+0000 7f389f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3898072ee0 con 0x7f38980d0620 2024-04-26T23:10:16.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.643+0000 7f3897400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3898073b70 0x7f38980cede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45882/0 (socket says 172.21.15.40:45882) 2024-04-26T23:10:16.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.643+0000 7f3897400640 1 -- 172.21.15.40:0/702391389 learned_addr learned my addr 172.21.15.40:0/702391389 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.643+0000 7f3897e00640 1 --2- 172.21.15.40:0/702391389 >> v2:172.21.15.40:3300/0 conn(0x7f389815f380 0x7f38980cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.643+0000 7f3897e00640 1 -- 172.21.15.40:0/702391389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38980d0620 msgr2=0x7f38980cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.643+0000 7f3897e00640 1 --2- 172.21.15.40:0/702391389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38980d0620 0x7f38980cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.643+0000 7f3897e00640 1 -- 172.21.15.40:0/702391389 >> v1:172.21.15.40:6789/0 conn(0x7f3898073b70 legacy=0x7f38980cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.643+0000 7f3897e00640 1 -- 172.21.15.40:0/702391389 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38980731c0 con 0x7f389815f380 2024-04-26T23:10:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.644+0000 7f3897e00640 1 --2- 172.21.15.40:0/702391389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389815f380 0x7f38980cb440 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f38880097a0 tx=0x7f388802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c64417aa2b7425 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:16.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.645+0000 7f3896a00640 1 -- 172.21.15.40:0/702391389 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888037070 con 0x7f389815f380 2024-04-26T23:10:16.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.645+0000 7f3896a00640 1 -- 172.21.15.40:0/702391389 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3888032440 con 0x7f389815f380 2024-04-26T23:10:16.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.645+0000 7f3896a00640 1 -- 172.21.15.40:0/702391389 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888032810 con 0x7f389815f380 2024-04-26T23:10:16.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.645+0000 7f389f200640 1 -- 172.21.15.40:0/702391389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389815f380 msgr2=0x7f38980cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:16.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.645+0000 7f389f200640 1 --2- 172.21.15.40:0/702391389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389815f380 0x7f38980cb440 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f38880097a0 tx=0x7f388802a180 comp rx=0 tx=0).stop 2024-04-26T23:10:16.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.645+0000 7f389f200640 1 -- 172.21.15.40:0/702391389 shutdown_connections 2024-04-26T23:10:16.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.645+0000 7f389f200640 1 --2- 172.21.15.40:0/702391389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f38980d0620 0x7f38980cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.645+0000 7f389f200640 1 --2- 172.21.15.40:0/702391389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f389815f380 0x7f38980cb440 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.645+0000 7f389f200640 1 -- 172.21.15.40:0/702391389 >> 172.21.15.40:0/702391389 conn(0x7f38980d0be0 msgr2=0x7f38980d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:16.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.646+0000 7f389f200640 1 -- 172.21.15.40:0/702391389 shutdown_connections 2024-04-26T23:10:16.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.646+0000 7f389f200640 1 -- 172.21.15.40:0/702391389 wait complete. 2024-04-26T23:10:16.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.647+0000 7f389f200640 1 Processor -- start 2024-04-26T23:10:16.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.648+0000 7f389f200640 1 -- start start 2024-04-26T23:10:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.648+0000 7f389f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38980d0620 0x7f389806eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f389f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f389815f380 0x7f38981e4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f389f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38981e49b0 0x7f389815b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f389f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3898073620 con 0x7f38980d0620 2024-04-26T23:10:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f389f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f38980aff50 con 0x7f38981e49b0 2024-04-26T23:10:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f389f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f38980737a0 con 0x7f389815f380 2024-04-26T23:10:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f3897e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38980d0620 0x7f389806eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f3897e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38980d0620 0x7f389806eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58416/0 (socket says 172.21.15.40:58416) 2024-04-26T23:10:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f3897e00640 1 -- 172.21.15.40:0/507469073 learned_addr learned my addr 172.21.15.40:0/507469073 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:16.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f3897400640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f389815f380 0x7f38981e4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:16.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f3897e00640 1 -- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f389815f380 msgr2=0x7f38981e4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:16.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f3897e00640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f389815f380 0x7f38981e4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f3897e00640 1 -- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38981e49b0 msgr2=0x7f389815b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:16.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f3897e00640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38981e49b0 0x7f389815b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f3897e00640 1 -- 172.21.15.40:0/507469073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38981f8600 con 0x7f38980d0620 2024-04-26T23:10:16.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.649+0000 7f3897400640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f389815f380 0x7f38981e4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:16.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.650+0000 7f3897e00640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38980d0620 0x7f389806eb00 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f3888000c00 tx=0x7f38880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:16.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.650+0000 7f388fe00640 1 -- 172.21.15.40:0/507469073 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888037040 con 0x7f38980d0620 2024-04-26T23:10:16.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.650+0000 7f388fe00640 1 -- 172.21.15.40:0/507469073 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3888042070 con 0x7f38980d0620 2024-04-26T23:10:16.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.650+0000 7f389f200640 1 -- 172.21.15.40:0/507469073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38981f88a0 con 0x7f38980d0620 2024-04-26T23:10:16.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.650+0000 7f388fe00640 1 -- 172.21.15.40:0/507469073 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888030a70 con 0x7f38980d0620 2024-04-26T23:10:16.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.650+0000 7f389f200640 1 -- 172.21.15.40:0/507469073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38981f8e60 con 0x7f38980d0620 2024-04-26T23:10:16.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.652+0000 7f389f200640 1 -- 172.21.15.40:0/507469073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38980ccc20 con 0x7f38980d0620 2024-04-26T23:10:16.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.653+0000 7f388fe00640 1 -- 172.21.15.40:0/507469073 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3888030c10 con 0x7f38980d0620 2024-04-26T23:10:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.654+0000 7f388fe00640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f387806c8f0 0x7f387806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.654+0000 7f388fe00640 1 -- 172.21.15.40:0/507469073 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f38880ac4d0 con 0x7f38980d0620 2024-04-26T23:10:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.654+0000 7f3897400640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f387806c8f0 0x7f387806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.656+0000 7f3897400640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f387806c8f0 0x7f387806edb0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3898070920 tx=0x7f3884009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.658+0000 7f388fe00640 1 -- 172.21.15.40:0/507469073 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f38980ccc20 con 0x7f38980d0620 2024-04-26T23:10:16.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.867+0000 7f389f200640 1 -- 172.21.15.40:0/507469073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38980b0ec0 con 0x7f38980d0620 2024-04-26T23:10:16.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.869+0000 7f388fe00640 1 -- 172.21.15.40:0/507469073 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+13529 (secure 0 0 0) 0x7f388807a9b0 con 0x7f38980d0620 2024-04-26T23:10:16.869 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:16.869 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":108,"fsid":"815936ab-d4ac-471e-b069-1a0dc70078ee","created":"2024-04-26T23:00:40.008669+0000","modified":"2024-04-26T23:10:12.731058+0000","last_up_change":"2024-04-26T23:00:52.414901+0000","last_in_change":"2024-04-26T23:00:43.199900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:00:53.175324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"042d5bb0-3c0d-45e5-83d2-a51234697407","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6801","nonce":2897145808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6803","nonce":2897145808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6807","nonce":2897145808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2897145808},{"type":"v1","addr":"172.21.15.40:6805","nonce":2897145808}]},"public_addr":"172.21.15.40:6801/2897145808","cluster_addr":"172.21.15.40:6803/2897145808","heartbeat_back_addr":"172.21.15.40:6807/2897145808","heartbeat_front_addr":"172.21.15.40:6805/2897145808","state":["exists","up"]},{"osd":1,"uuid":"6973225f-5f76-4c21-9204-a664b8eed95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6817","nonce":2323014490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6819","nonce":2323014490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6824","nonce":2323014490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":2323014490},{"type":"v1","addr":"172.21.15.40:6821","nonce":2323014490}]},"public_addr":"172.21.15.40:6817/2323014490","cluster_addr":"172.21.15.40:6819/2323014490","heartbeat_back_addr":"172.21.15.40:6824/2323014490","heartbeat_front_addr":"172.21.15.40:6821/2323014490","state":["exists","up"]},{"osd":2,"uuid":"b6f6d6d1-f67e-4bb3-8b99-fb53f66c1bc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6809","nonce":2108759315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6811","nonce":2108759315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6816","nonce":2108759315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2108759315},{"type":"v1","addr":"172.21.15.40:6813","nonce":2108759315}]},"public_addr":"172.21.15.40:6809/2108759315","cluster_addr":"172.21.15.40:6811/2108759315","heartbeat_back_addr":"172.21.15.40:6816/2108759315","heartbeat_front_addr":"172.21.15.40:6813/2108759315","state":["exists","up"]},{"osd":3,"uuid":"13b77b2e-3161-450c-8aa0-557988e3ce81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6825","nonce":378872239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6827","nonce":378872239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6831","nonce":378872239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":378872239},{"type":"v1","addr":"172.21.15.40:6829","nonce":378872239}]},"public_addr":"172.21.15.40:6825/378872239","cluster_addr":"172.21.15.40:6827/378872239","heartbeat_back_addr":"172.21.15.40:6831/378872239","heartbeat_front_addr":"172.21.15.40:6829/378872239","state":["exists","up"]},{"osd":4,"uuid":"e1c95d9d-13fd-4263-ae34-96862877c738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6803","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6805","nonce":801343304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6807","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6809","nonce":801343304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6815","nonce":801343304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6811","nonce":801343304},{"type":"v1","addr":"172.21.15.103:6813","nonce":801343304}]},"public_addr":"172.21.15.103:6805/801343304","cluster_addr":"172.21.15.103:6809/801343304","heartbeat_back_addr":"172.21.15.103:6815/801343304","heartbeat_front_addr":"172.21.15.103:6813/801343304","state":["exists","up"]},{"osd":5,"uuid":"222ecc63-1194-48ec-abc2-d1f1272c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6801","nonce":157083760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6804","nonce":157083760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6812","nonce":157083760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":157083760},{"type":"v1","addr":"172.21.15.103:6808","nonce":157083760}]},"public_addr":"172.21.15.103:6801/157083760","cluster_addr":"172.21.15.103:6804/157083760","heartbeat_back_addr":"172.21.15.103:6812/157083760","heartbeat_front_addr":"172.21.15.103:6808/157083760","state":["exists","up"]},{"osd":6,"uuid":"4710c272-359f-4fb7-a4bc-0fc444c29a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6817","nonce":293480649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6820","nonce":293480649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6828","nonce":293480649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":293480649},{"type":"v1","addr":"172.21.15.103:6824","nonce":293480649}]},"public_addr":"172.21.15.103:6817/293480649","cluster_addr":"172.21.15.103:6820/293480649","heartbeat_back_addr":"172.21.15.103:6828/293480649","heartbeat_front_addr":"172.21.15.103:6824/293480649","state":["exists","up"]},{"osd":7,"uuid":"6e81e8c8-355c-4411-89ef-c9f95e5dc00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6819","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6821","nonce":3769540391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6823","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6825","nonce":3769540391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6831","nonce":3769540391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":3769540391},{"type":"v1","addr":"172.21.15.103:6829","nonce":3769540391}]},"public_addr":"172.21.15.103:6821/3769540391","cluster_addr":"172.21.15.103:6825/3769540391","heartbeat_back_addr":"172.21.15.103:6831/3769540391","heartbeat_front_addr":"172.21.15.103:6829/3769540391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.834879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.875752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.855267+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.854475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.520837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.576287+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.619315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:00:48.369569+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6833/3250599834":"2024-04-27T23:10:09.714108+0000","172.21.15.40:6832/3250599834":"2024-04-27T23:10:09.714108+0000","172.21.15.40:6835/1138762604":"2024-04-27T23:10:09.714112+0000","172.21.15.40:6834/1138762604":"2024-04-27T23:10:09.714112+0000","172.21.15.103:6837/2022020922":"2024-04-27T23:10:09.714088+0000","172.21.15.103:6835/2022020922":"2024-04-27T23:10:09.714088+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:10:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.871+0000 7f389f200640 1 -- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f387806c8f0 msgr2=0x7f387806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.871+0000 7f389f200640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f387806c8f0 0x7f387806edb0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3898070920 tx=0x7f3884009040 comp rx=0 tx=0).stop 2024-04-26T23:10:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.871+0000 7f389f200640 1 -- 172.21.15.40:0/507469073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38980d0620 msgr2=0x7f389806eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.871+0000 7f389f200640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38980d0620 0x7f389806eb00 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f3888000c00 tx=0x7f38880046c0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.872+0000 7f389f200640 1 -- 172.21.15.40:0/507469073 shutdown_connections 2024-04-26T23:10:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.872+0000 7f389f200640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f387806c8f0 0x7f387806edb0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.872+0000 7f389f200640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38981e49b0 0x7f389815b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.872+0000 7f389f200640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f389815f380 0x7f38981e4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.872+0000 7f389f200640 1 --2- 172.21.15.40:0/507469073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38980d0620 0x7f389806eb00 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.872+0000 7f389f200640 1 -- 172.21.15.40:0/507469073 >> 172.21.15.40:0/507469073 conn(0x7f38980d0be0 msgr2=0x7f38980cb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.872+0000 7f389f200640 1 -- 172.21.15.40:0/507469073 shutdown_connections 2024-04-26T23:10:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:16.872+0000 7f389f200640 1 -- 172.21.15.40:0/507469073 wait complete. 2024-04-26T23:10:17.891 INFO:tasks.ceph:Scrubbing osd.0 2024-04-26T23:10:17.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-26T23:10:18.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.113+0000 7fa2b6a00640 1 Processor -- start 2024-04-26T23:10:18.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.113+0000 7fa2b6a00640 1 -- start start 2024-04-26T23:10:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.115+0000 7fa2b6a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa2b0145970 0x7fa2b01612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.115+0000 7fa2b6a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2b00d06b0 0x7fa2b00d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.115+0000 7fa2b6a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2b00b59f0 con 0x7fa2b0146060 2024-04-26T23:10:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.115+0000 7fa2b6a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa2b00ceed0 con 0x7fa2b0145970 2024-04-26T23:10:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.115+0000 7fa2b6a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b00d1040 con 0x7fa2b00d06b0 2024-04-26T23:10:18.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.115+0000 7fa2b4c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa2b0146060 0x7fa2b00b3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45890/0 (socket says 172.21.15.40:45890) 2024-04-26T23:10:18.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.115+0000 7fa2b4c00640 1 -- 172.21.15.40:0/661175094 learned_addr learned my addr 172.21.15.40:0/661175094 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:18.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.116+0000 7fa2b5600640 1 --2- 172.21.15.40:0/661175094 >> v2:172.21.15.103:3300/0 conn(0x7fa2b0145970 0x7fa2b01612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.116+0000 7fa2b6000640 1 --2- 172.21.15.40:0/661175094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2b00d06b0 0x7fa2b00d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.116+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3964454664 0 0) 0x7fa2b00b59f0 con 0x7fa2b0146060 2024-04-26T23:10:18.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.116+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa294003520 con 0x7fa2b0146060 2024-04-26T23:10:18.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.116+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2891135827 0 0) 0x7fa294003520 con 0x7fa2b0146060 2024-04-26T23:10:18.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.116+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2b00d06b0 msgr2=0x7fa2b00d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:18.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.116+0000 7fa2afe00640 1 --2- 172.21.15.40:0/661175094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2b00d06b0 0x7fa2b00d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.116+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 >> v2:172.21.15.103:3300/0 conn(0x7fa2b0145970 msgr2=0x7fa2b01612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:18.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.116+0000 7fa2afe00640 1 --2- 172.21.15.40:0/661175094 >> v2:172.21.15.103:3300/0 conn(0x7fa2b0145970 0x7fa2b01612b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.116+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b01617f0 con 0x7fa2b0146060 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.116+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7fa29c001670 con 0x7fa2b0146060 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.116+0000 7fa2b5600640 1 --2- 172.21.15.40:0/661175094 >> v2:172.21.15.103:3300/0 conn(0x7fa2b0145970 0x7fa2b01612b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 >> v1:172.21.15.40:6789/0 conn(0x7fa2b0146060 legacy=0x7fa2b00b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2afe00640 1 --2- 172.21.15.40:0/661175094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2b00d06b0 0x7fa2940039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2afe00640 1 --2- 172.21.15.40:0/661175094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa294004030 0x7fa294004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2afe00640 1 --2- 172.21.15.40:0/661175094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa294004d40 0x7fa2940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b00d1040 con 0x7fa294004d40 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa29400baa0 con 0x7fa294004030 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa29400bca0 con 0x7fa2b00d06b0 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2b6000640 1 --2- 172.21.15.40:0/661175094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa294004d40 0x7fa2940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2b5600640 1 --2- 172.21.15.40:0/661175094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2b00d06b0 0x7fa2940039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2b4c00640 1 --2- 172.21.15.40:0/661175094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa294004030 0x7fa294004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2b6000640 1 -- 172.21.15.40:0/661175094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2b00d06b0 msgr2=0x7fa2940039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:18.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2b6000640 1 --2- 172.21.15.40:0/661175094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa2b00d06b0 0x7fa2940039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2b6000640 1 -- 172.21.15.40:0/661175094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa294004030 msgr2=0x7fa294004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:18.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2b6000640 1 --2- 172.21.15.40:0/661175094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa294004030 0x7fa294004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2b6000640 1 -- 172.21.15.40:0/661175094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa294003520 con 0x7fa294004d40 2024-04-26T23:10:18.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.117+0000 7fa2b4c00640 1 --2- 172.21.15.40:0/661175094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa294004030 0x7fa294004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.119+0000 7fa2b6000640 1 --2- 172.21.15.40:0/661175094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa294004d40 0x7fa2940051c0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8007360 tx=0x7fa294005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8cae5bafe3887eb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.119+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8009be0 con 0x7fa294004d40 2024-04-26T23:10:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.119+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2a8036600 con 0x7fa294004d40 2024-04-26T23:10:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.119+0000 7fa2afe00640 1 -- 172.21.15.40:0/661175094 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a80369d0 con 0x7fa294004d40 2024-04-26T23:10:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.119+0000 7fa2b6a00640 1 -- 172.21.15.40:0/661175094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa294004d40 msgr2=0x7fa2940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.119+0000 7fa2b6a00640 1 --2- 172.21.15.40:0/661175094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa294004d40 0x7fa2940051c0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8007360 tx=0x7fa294005830 comp rx=0 tx=0).stop 2024-04-26T23:10:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.119+0000 7fa2b6000640 1 -- 172.21.15.40:0/661175094 reap_dead start 2024-04-26T23:10:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.120+0000 7fa2b6a00640 1 -- 172.21.15.40:0/661175094 shutdown_connections 2024-04-26T23:10:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.120+0000 7fa2b6a00640 1 -- 172.21.15.40:0/661175094 >> 172.21.15.40:0/661175094 conn(0x7fa2b0075eb0 msgr2=0x7fa2b0068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.120+0000 7fa2b6a00640 1 -- 172.21.15.40:0/661175094 shutdown_connections 2024-04-26T23:10:18.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.120+0000 7fa2b6a00640 1 -- 172.21.15.40:0/661175094 wait complete. 2024-04-26T23:10:18.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.122+0000 7fa2b6a00640 1 Processor -- start 2024-04-26T23:10:18.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.122+0000 7fa2b6a00640 1 -- start start 2024-04-26T23:10:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.123+0000 7fa2b6a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa294004d40 0x7fa2b0144b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.123+0000 7fa2b6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2b0073390 0x7fa2b0131a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.123+0000 7fa2b6a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2b00aa4b0 0x7fa2b00aa940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.123+0000 7fa2b6a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b00b5c60 con 0x7fa2b0073390 2024-04-26T23:10:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.123+0000 7fa2b6a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b00b5f40 con 0x7fa2b00aa4b0 2024-04-26T23:10:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.123+0000 7fa2b6a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b00cf100 con 0x7fa294004d40 2024-04-26T23:10:18.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.123+0000 7fa2b4c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2b0073390 0x7fa2b0131a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.123+0000 7fa2b4c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2b0073390 0x7fa2b0131a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58442/0 (socket says 172.21.15.40:58442) 2024-04-26T23:10:18.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.123+0000 7fa2b4c00640 1 -- 172.21.15.40:0/753180223 learned_addr learned my addr 172.21.15.40:0/753180223 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:18.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.124+0000 7fa2b5600640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa294004d40 0x7fa2b0144b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.124+0000 7fa2b4c00640 1 -- 172.21.15.40:0/753180223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa294004d40 msgr2=0x7fa2b0144b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.124+0000 7fa2b6000640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2b00aa4b0 0x7fa2b00aa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.124+0000 7fa2b4c00640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa294004d40 0x7fa2b0144b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.124+0000 7fa2b4c00640 1 -- 172.21.15.40:0/753180223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2b00aa4b0 msgr2=0x7fa2b00aa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.124+0000 7fa2b4c00640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2b00aa4b0 0x7fa2b00aa940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.124+0000 7fa2b4c00640 1 -- 172.21.15.40:0/753180223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b0132330 con 0x7fa2b0073390 2024-04-26T23:10:18.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.124+0000 7fa2b5600640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa294004d40 0x7fa2b0144b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:18.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.124+0000 7fa2b4c00640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2b0073390 0x7fa2b0131a50 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00d8e0 tx=0x7fa29c00dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:18.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.124+0000 7fa2ad600640 1 -- 172.21.15.40:0/753180223 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c019070 con 0x7fa2b0073390 2024-04-26T23:10:18.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.124+0000 7fa2ad600640 1 -- 172.21.15.40:0/753180223 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa29c0074e0 con 0x7fa2b0073390 2024-04-26T23:10:18.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.124+0000 7fa2ad600640 1 -- 172.21.15.40:0/753180223 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c004a20 con 0x7fa2b0073390 2024-04-26T23:10:18.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.124+0000 7fa2b6a00640 1 -- 172.21.15.40:0/753180223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b0132610 con 0x7fa2b0073390 2024-04-26T23:10:18.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.125+0000 7fa2b6a00640 1 -- 172.21.15.40:0/753180223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b0132bd0 con 0x7fa2b0073390 2024-04-26T23:10:18.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.126+0000 7fa2b6a00640 1 -- 172.21.15.40:0/753180223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa2b0069370 con 0x7fa2b0073390 2024-04-26T23:10:18.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.127+0000 7fa2ad600640 1 -- 172.21.15.40:0/753180223 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa29c01c070 con 0x7fa2b0073390 2024-04-26T23:10:18.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.128+0000 7fa2ad600640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa29006c8f0 0x7fa29006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.128+0000 7fa2ad600640 1 -- 172.21.15.40:0/753180223 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7fa29c08b5f0 con 0x7fa2b0073390 2024-04-26T23:10:18.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.128+0000 7fa2ad600640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x7fa290072ed0 0x7fa290075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.128+0000 7fa2ad600640 1 -- 172.21.15.40:0/753180223 --> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2900759e0 con 0x7fa290072ed0 2024-04-26T23:10:18.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.128+0000 7fa2ad600640 1 -- 172.21.15.40:0/753180223 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fa29c092050 con 0x7fa2b0073390 2024-04-26T23:10:18.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.128+0000 7fa2b6000640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x7fa290072ed0 0x7fa290075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.128+0000 7fa2b5600640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa29006c8f0 0x7fa29006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.129+0000 7fa2b6000640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x7fa290072ed0 0x7fa290075310 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:18.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.129+0000 7fa2b5600640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa29006c8f0 0x7fa29006edb0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0068370 tx=0x7fa2a0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:18.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.130+0000 7fa2ad600640 1 -- 172.21.15.40:0/753180223 <== osd.0 v2:172.21.15.40:6800/2897145808 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa2900759e0 con 0x7fa290072ed0 2024-04-26T23:10:18.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.144+0000 7fa2b6a00640 1 -- 172.21.15.40:0/753180223 --> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa2b00d1400 con 0x7fa290072ed0 2024-04-26T23:10:18.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.145+0000 7fa2ad600640 1 -- 172.21.15.40:0/753180223 <== osd.0 v2:172.21.15.40:6800/2897145808 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa2b00d1400 con 0x7fa290072ed0 2024-04-26T23:10:18.145 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:10:18.145 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T23:10:18.145 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:10:18.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.145+0000 7fa2b6a00640 1 -- 172.21.15.40:0/753180223 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x7fa290072ed0 msgr2=0x7fa290075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.145+0000 7fa2b6a00640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.40:6800/2897145808,v1:172.21.15.40:6801/2897145808] conn(0x7fa290072ed0 0x7fa290075310 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.146+0000 7fa2b6a00640 1 -- 172.21.15.40:0/753180223 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa29006c8f0 msgr2=0x7fa29006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.146+0000 7fa2b6a00640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fa29006c8f0 0x7fa29006edb0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0068370 tx=0x7fa2a0008040 comp rx=0 tx=0).stop 2024-04-26T23:10:18.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.146+0000 7fa2b6a00640 1 -- 172.21.15.40:0/753180223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2b0073390 msgr2=0x7fa2b0131a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.146+0000 7fa2b6a00640 1 --2- 172.21.15.40:0/753180223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2b0073390 0x7fa2b0131a50 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00d8e0 tx=0x7fa29c00dda0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.146+0000 7fa2b6000640 1 -- 172.21.15.40:0/753180223 reap_dead start 2024-04-26T23:10:18.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.146+0000 7fa2b6a00640 1 -- 172.21.15.40:0/753180223 shutdown_connections 2024-04-26T23:10:18.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.146+0000 7fa2b6a00640 1 -- 172.21.15.40:0/753180223 >> 172.21.15.40:0/753180223 conn(0x7fa2b0075eb0 msgr2=0x7fa2b0071f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:18.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.146+0000 7fa2b6a00640 1 -- 172.21.15.40:0/753180223 shutdown_connections 2024-04-26T23:10:18.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.146+0000 7fa2b6a00640 1 -- 172.21.15.40:0/753180223 wait complete. 2024-04-26T23:10:18.164 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-26T23:10:18.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.363+0000 7f1845200640 1 Processor -- start 2024-04-26T23:10:18.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.364+0000 7f1845200640 1 -- start start 2024-04-26T23:10:18.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.365+0000 7f1845200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18400af7a0 0x7f18400afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.365+0000 7f1845200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1840071fe0 0x7f1840073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.365+0000 7f1845200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18400d0820 con 0x7f184015ce60 2024-04-26T23:10:18.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.365+0000 7f1845200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f18400cc450 con 0x7f1840071fe0 2024-04-26T23:10:18.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.365+0000 7f1845200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f18401614c0 con 0x7f18400af7a0 2024-04-26T23:10:18.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.366+0000 7f183fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1840071fe0 0x7f1840073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.366+0000 7f183ea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18400af7a0 0x7f18400afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.366+0000 7f183fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1840071fe0 0x7f1840073760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56234/0 (socket says 172.21.15.40:56234) 2024-04-26T23:10:18.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.366+0000 7f183fe00640 1 -- 172.21.15.40:0/1971160987 learned_addr learned my addr 172.21.15.40:0/1971160987 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:18.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.366+0000 7f183f400640 1 --1- 172.21.15.40:0/1971160987 >> v1:172.21.15.103:6789/0 conn(0x7f184015ce60 0x7f18400ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:34318/0 (socket says 172.21.15.40:34318) 2024-04-26T23:10:18.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.367+0000 7f183fe00640 1 -- 172.21.15.40:0/1971160987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18400af7a0 msgr2=0x7f18400afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:18.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.367+0000 7f183fe00640 1 --2- 172.21.15.40:0/1971160987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18400af7a0 0x7f18400afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.367+0000 7f183fe00640 1 -- 172.21.15.40:0/1971160987 >> v1:172.21.15.103:6789/0 conn(0x7f184015ce60 legacy=0x7f18400ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.367+0000 7f183fe00640 1 -- 172.21.15.40:0/1971160987 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1840161760 con 0x7f1840071fe0 2024-04-26T23:10:18.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.367+0000 7f183ea00640 1 --2- 172.21.15.40:0/1971160987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18400af7a0 0x7f18400afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:10:18.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.368+0000 7f183fe00640 1 --2- 172.21.15.40:0/1971160987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1840071fe0 0x7f1840073760 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1838009e50 tx=0x7f183802a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab98336afc515b22 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:18.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.369+0000 7f183e000640 1 -- 172.21.15.40:0/1971160987 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1838036070 con 0x7f1840071fe0 2024-04-26T23:10:18.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.369+0000 7f183e000640 1 -- 172.21.15.40:0/1971160987 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1838031600 con 0x7f1840071fe0 2024-04-26T23:10:18.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.369+0000 7f183e000640 1 -- 172.21.15.40:0/1971160987 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18380319d0 con 0x7f1840071fe0 2024-04-26T23:10:18.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.370+0000 7f1845200640 1 -- 172.21.15.40:0/1971160987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1840071fe0 msgr2=0x7f1840073760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.370+0000 7f1845200640 1 --2- 172.21.15.40:0/1971160987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1840071fe0 0x7f1840073760 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1838009e50 tx=0x7f183802a680 comp rx=0 tx=0).stop 2024-04-26T23:10:18.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.370+0000 7f1845200640 1 -- 172.21.15.40:0/1971160987 shutdown_connections 2024-04-26T23:10:18.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.370+0000 7f1845200640 1 --2- 172.21.15.40:0/1971160987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1840071fe0 0x7f1840073760 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.370+0000 7f1845200640 1 --2- 172.21.15.40:0/1971160987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f18400af7a0 0x7f18400afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.370+0000 7f1845200640 1 -- 172.21.15.40:0/1971160987 >> 172.21.15.40:0/1971160987 conn(0x7f18400d0a10 msgr2=0x7f18400d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:18.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.370+0000 7f1845200640 1 -- 172.21.15.40:0/1971160987 shutdown_connections 2024-04-26T23:10:18.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.370+0000 7f1845200640 1 -- 172.21.15.40:0/1971160987 wait complete. 2024-04-26T23:10:18.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.372+0000 7f1845200640 1 Processor -- start 2024-04-26T23:10:18.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.373+0000 7f1845200640 1 -- start start 2024-04-26T23:10:18.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.374+0000 7f1845200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1840071fe0 0x7f184015a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.374+0000 7f1845200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18400af7a0 0x7f18400c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.374+0000 7f1845200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184015ce60 0x7f1840156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.374+0000 7f1845200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f18400cc1f0 con 0x7f18400af7a0 2024-04-26T23:10:18.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.374+0000 7f1845200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18400cc070 con 0x7f184015ce60 2024-04-26T23:10:18.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.374+0000 7f1845200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1840073ca0 con 0x7f1840071fe0 2024-04-26T23:10:18.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.374+0000 7f183ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18400af7a0 0x7f18400c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.374+0000 7f183ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18400af7a0 0x7f18400c5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58448/0 (socket says 172.21.15.40:58448) 2024-04-26T23:10:18.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.374+0000 7f183ea00640 1 -- 172.21.15.40:0/888196532 learned_addr learned my addr 172.21.15.40:0/888196532 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:18.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.375+0000 7f183fe00640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184015ce60 0x7f1840156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.375+0000 7f183f400640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1840071fe0 0x7f184015a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.375+0000 7f183ea00640 1 -- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1840071fe0 msgr2=0x7f184015a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.375+0000 7f183ea00640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1840071fe0 0x7f184015a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.375+0000 7f183ea00640 1 -- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184015ce60 msgr2=0x7f1840156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.375+0000 7f183ea00640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184015ce60 0x7f1840156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.375+0000 7f183ea00640 1 -- 172.21.15.40:0/888196532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184015f320 con 0x7f18400af7a0 2024-04-26T23:10:18.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.375+0000 7f183ea00640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18400af7a0 0x7f18400c5fa0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f182c00ebe0 tx=0x7f182c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:18.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.376+0000 7f1837400640 1 -- 172.21.15.40:0/888196532 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c017070 con 0x7f18400af7a0 2024-04-26T23:10:18.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.376+0000 7f1837400640 1 -- 172.21.15.40:0/888196532 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182c00cc30 con 0x7f18400af7a0 2024-04-26T23:10:18.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.376+0000 7f1845200640 1 -- 172.21.15.40:0/888196532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18400c65a0 con 0x7f18400af7a0 2024-04-26T23:10:18.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.376+0000 7f1837400640 1 -- 172.21.15.40:0/888196532 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c013470 con 0x7f18400af7a0 2024-04-26T23:10:18.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.376+0000 7f1845200640 1 -- 172.21.15.40:0/888196532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18400c6b60 con 0x7f18400af7a0 2024-04-26T23:10:18.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.377+0000 7f1845200640 1 -- 172.21.15.40:0/888196532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1840069370 con 0x7f18400af7a0 2024-04-26T23:10:18.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.378+0000 7f1837400640 1 -- 172.21.15.40:0/888196532 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f182c01a070 con 0x7f18400af7a0 2024-04-26T23:10:18.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.379+0000 7f1837400640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f182006c8f0 0x7f182006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.379+0000 7f1837400640 1 -- 172.21.15.40:0/888196532 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f182c08ba10 con 0x7f18400af7a0 2024-04-26T23:10:18.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.380+0000 7f183f400640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f182006c8f0 0x7f182006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.381+0000 7f183f400640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f182006c8f0 0x7f182006edb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1830000f60 tx=0x7f1830007ce0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:18.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.383+0000 7f1837400640 1 -- 172.21.15.40:0/888196532 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f182c059f00 con 0x7f18400af7a0 2024-04-26T23:10:18.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.577+0000 7f1845200640 1 -- 172.21.15.40:0/888196532 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f184015c600 con 0x7f182006c8f0 2024-04-26T23:10:18.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.578+0000 7f1837400640 1 -- 172.21.15.40:0/888196532 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f184015c600 con 0x7f182006c8f0 2024-04-26T23:10:18.578 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 0 to deep-scrub 2024-04-26T23:10:18.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.580+0000 7f1845200640 1 -- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f182006c8f0 msgr2=0x7f182006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.580+0000 7f1845200640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f182006c8f0 0x7f182006edb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1830000f60 tx=0x7f1830007ce0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.580+0000 7f1845200640 1 -- 172.21.15.40:0/888196532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18400af7a0 msgr2=0x7f18400c5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.580+0000 7f1845200640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18400af7a0 0x7f18400c5fa0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f182c00ebe0 tx=0x7f182c00c520 comp rx=0 tx=0).stop 2024-04-26T23:10:18.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.581+0000 7f1845200640 1 -- 172.21.15.40:0/888196532 shutdown_connections 2024-04-26T23:10:18.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.581+0000 7f1845200640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f182006c8f0 0x7f182006edb0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.581+0000 7f1845200640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184015ce60 0x7f1840156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.581+0000 7f1845200640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18400af7a0 0x7f18400c5fa0 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.581+0000 7f1845200640 1 --2- 172.21.15.40:0/888196532 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1840071fe0 0x7f184015a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.581+0000 7f1845200640 1 -- 172.21.15.40:0/888196532 >> 172.21.15.40:0/888196532 conn(0x7f18400d0a10 msgr2=0x7f18401592b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:18.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.581+0000 7f1845200640 1 -- 172.21.15.40:0/888196532 shutdown_connections 2024-04-26T23:10:18.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.581+0000 7f1845200640 1 -- 172.21.15.40:0/888196532 wait complete. 2024-04-26T23:10:18.600 INFO:tasks.ceph:Scrubbing osd.1 2024-04-26T23:10:18.600 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-26T23:10:18.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.814+0000 7fee75e00640 1 Processor -- start 2024-04-26T23:10:18.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.814+0000 7fee75e00640 1 -- start start 2024-04-26T23:10:18.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.816+0000 7fee75e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fee70145970 0x7fee701612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.816+0000 7fee75e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fee70146060 0x7fee700b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.816+0000 7fee75e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee700b59f0 con 0x7fee700b3fb0 2024-04-26T23:10:18.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.816+0000 7fee75e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fee700ceed0 con 0x7fee70145970 2024-04-26T23:10:18.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.816+0000 7fee75e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fee701617f0 con 0x7fee70146060 2024-04-26T23:10:18.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.817+0000 7fee74a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fee70145970 0x7fee701612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.817+0000 7fee74a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fee70145970 0x7fee701612b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58454/0 (socket says 172.21.15.40:58454) 2024-04-26T23:10:18.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.817+0000 7fee74a00640 1 -- 172.21.15.40:0/2592786843 learned_addr learned my addr 172.21.15.40:0/2592786843 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:18.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.817+0000 7fee67e00640 1 --2- 172.21.15.40:0/2592786843 >> v2:172.21.15.103:3300/0 conn(0x7fee70146060 0x7fee700b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.817+0000 7fee6fe00640 1 -- 172.21.15.40:0/2592786843 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 890564447 0 0) 0x7fee700b59f0 con 0x7fee700b3fb0 2024-04-26T23:10:18.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.817+0000 7fee6fe00640 1 -- 172.21.15.40:0/2592786843 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee54003520 con 0x7fee700b3fb0 2024-04-26T23:10:18.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.817+0000 7fee74a00640 1 -- 172.21.15.40:0/2592786843 >> v2:172.21.15.103:3300/0 conn(0x7fee70146060 msgr2=0x7fee700b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:18.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.817+0000 7fee74a00640 1 --2- 172.21.15.40:0/2592786843 >> v2:172.21.15.103:3300/0 conn(0x7fee70146060 0x7fee700b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.817+0000 7fee74a00640 1 -- 172.21.15.40:0/2592786843 >> v1:172.21.15.40:6789/0 conn(0x7fee700b3fb0 legacy=0x7fee700d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.817+0000 7fee74a00640 1 -- 172.21.15.40:0/2592786843 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee700b5f40 con 0x7fee70145970 2024-04-26T23:10:18.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.818+0000 7fee74a00640 1 --2- 172.21.15.40:0/2592786843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee70145970 0x7fee701612b0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7fee70071b50 tx=0x7fee6002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7d99c3f80192c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:18.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.819+0000 7fee6fe00640 1 -- 172.21.15.40:0/2592786843 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee60037070 con 0x7fee70145970 2024-04-26T23:10:18.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.819+0000 7fee6fe00640 1 -- 172.21.15.40:0/2592786843 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee60032440 con 0x7fee70145970 2024-04-26T23:10:18.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.819+0000 7fee6fe00640 1 -- 172.21.15.40:0/2592786843 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee600327f0 con 0x7fee70145970 2024-04-26T23:10:18.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.819+0000 7fee75e00640 1 -- 172.21.15.40:0/2592786843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee70145970 msgr2=0x7fee701612b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.819+0000 7fee75e00640 1 --2- 172.21.15.40:0/2592786843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee70145970 0x7fee701612b0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7fee70071b50 tx=0x7fee6002a270 comp rx=0 tx=0).stop 2024-04-26T23:10:18.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.819+0000 7fee75e00640 1 -- 172.21.15.40:0/2592786843 shutdown_connections 2024-04-26T23:10:18.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.819+0000 7fee75e00640 1 --2- 172.21.15.40:0/2592786843 >> v2:172.21.15.103:3300/0 conn(0x7fee70146060 0x7fee700b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.819+0000 7fee75e00640 1 --2- 172.21.15.40:0/2592786843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee70145970 0x7fee701612b0 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.819+0000 7fee75e00640 1 -- 172.21.15.40:0/2592786843 >> 172.21.15.40:0/2592786843 conn(0x7fee70075eb0 msgr2=0x7fee70068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:18.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.819+0000 7fee75e00640 1 -- 172.21.15.40:0/2592786843 shutdown_connections 2024-04-26T23:10:18.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.819+0000 7fee75e00640 1 -- 172.21.15.40:0/2592786843 wait complete. 2024-04-26T23:10:18.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.821+0000 7fee75e00640 1 Processor -- start 2024-04-26T23:10:18.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.821+0000 7fee75e00640 1 -- start start 2024-04-26T23:10:18.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.822+0000 7fee75e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee70145970 0x7fee70132fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.822+0000 7fee75e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee70146060 0x7fee701f85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.822+0000 7fee75e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee701f9020 0x7fee7015e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.822+0000 7fee75e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee700cf3e0 con 0x7fee70146060 2024-04-26T23:10:18.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.822+0000 7fee75e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee700cf100 con 0x7fee701f9020 2024-04-26T23:10:18.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.822+0000 7fee75e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fee70145d50 con 0x7fee70145970 2024-04-26T23:10:18.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.822+0000 7fee67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee70146060 0x7fee701f85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.822+0000 7fee67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee70146060 0x7fee701f85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58460/0 (socket says 172.21.15.40:58460) 2024-04-26T23:10:18.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.822+0000 7fee67e00640 1 -- 172.21.15.40:0/1748822769 learned_addr learned my addr 172.21.15.40:0/1748822769 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:18.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.823+0000 7fee75400640 1 --2- 172.21.15.40:0/1748822769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee701f9020 0x7fee7015e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.823+0000 7fee67e00640 1 -- 172.21.15.40:0/1748822769 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee70145970 msgr2=0x7fee70132fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.823+0000 7fee67e00640 1 --2- 172.21.15.40:0/1748822769 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee70145970 0x7fee70132fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.823+0000 7fee67e00640 1 -- 172.21.15.40:0/1748822769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee701f9020 msgr2=0x7fee7015e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.823+0000 7fee67e00640 1 --2- 172.21.15.40:0/1748822769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee701f9020 0x7fee7015e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.823+0000 7fee67e00640 1 -- 172.21.15.40:0/1748822769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee70133900 con 0x7fee70146060 2024-04-26T23:10:18.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.823+0000 7fee67e00640 1 --2- 172.21.15.40:0/1748822769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee70146060 0x7fee701f85a0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7fee5800e9b0 tx=0x7fee5800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:18.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.823+0000 7fee6d600640 1 -- 172.21.15.40:0/1748822769 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58017070 con 0x7fee70146060 2024-04-26T23:10:18.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.823+0000 7fee6d600640 1 -- 172.21.15.40:0/1748822769 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee5800c9a0 con 0x7fee70146060 2024-04-26T23:10:18.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.823+0000 7fee6d600640 1 -- 172.21.15.40:0/1748822769 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5800cd90 con 0x7fee70146060 2024-04-26T23:10:18.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.823+0000 7fee75e00640 1 -- 172.21.15.40:0/1748822769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee70133ec0 con 0x7fee70146060 2024-04-26T23:10:18.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.824+0000 7fee75e00640 1 -- 172.21.15.40:0/1748822769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee70134480 con 0x7fee70146060 2024-04-26T23:10:18.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.824+0000 7fee75e00640 1 -- 172.21.15.40:0/1748822769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fee7012fc70 con 0x7fee70146060 2024-04-26T23:10:18.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.825+0000 7fee6d600640 1 -- 172.21.15.40:0/1748822769 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fee5801a070 con 0x7fee70146060 2024-04-26T23:10:18.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.826+0000 7fee6d600640 1 --2- 172.21.15.40:0/1748822769 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee4c06c8f0 0x7fee4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.826+0000 7fee6d600640 1 -- 172.21.15.40:0/1748822769 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7fee5808bf00 con 0x7fee70146060 2024-04-26T23:10:18.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.827+0000 7fee6d600640 1 --2- 172.21.15.40:0/1748822769 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x7fee4c072ed0 0x7fee4c075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:18.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.827+0000 7fee6d600640 1 -- 172.21.15.40:0/1748822769 --> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee4c0759e0 con 0x7fee4c072ed0 2024-04-26T23:10:18.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.827+0000 7fee6d600640 1 -- 172.21.15.40:0/1748822769 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fee5808e520 con 0x7fee70146060 2024-04-26T23:10:18.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.827+0000 7fee75400640 1 --2- 172.21.15.40:0/1748822769 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x7fee4c072ed0 0x7fee4c075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.827+0000 7fee74a00640 1 --2- 172.21.15.40:0/1748822769 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee4c06c8f0 0x7fee4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:18.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.827+0000 7fee75400640 1 --2- 172.21.15.40:0/1748822769 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x7fee4c072ed0 0x7fee4c075310 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:18.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.828+0000 7fee74a00640 1 --2- 172.21.15.40:0/1748822769 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee4c06c8f0 0x7fee4c06edb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fee60034000 tx=0x7fee6002c000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:18.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.828+0000 7fee6d600640 1 -- 172.21.15.40:0/1748822769 <== osd.1 v2:172.21.15.40:6815/2323014490 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee4c0759e0 con 0x7fee4c072ed0 2024-04-26T23:10:18.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.843+0000 7fee75e00640 1 -- 172.21.15.40:0/1748822769 --> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fee70069370 con 0x7fee4c072ed0 2024-04-26T23:10:18.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.843+0000 7fee6d600640 1 -- 172.21.15.40:0/1748822769 <== osd.1 v2:172.21.15.40:6815/2323014490 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fee70069370 con 0x7fee4c072ed0 2024-04-26T23:10:18.844 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:10:18.844 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T23:10:18.844 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:10:18.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.844+0000 7fee75e00640 1 -- 172.21.15.40:0/1748822769 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x7fee4c072ed0 msgr2=0x7fee4c075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.844+0000 7fee75e00640 1 --2- 172.21.15.40:0/1748822769 >> [v2:172.21.15.40:6815/2323014490,v1:172.21.15.40:6817/2323014490] conn(0x7fee4c072ed0 0x7fee4c075310 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:18.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.844+0000 7fee75e00640 1 -- 172.21.15.40:0/1748822769 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee4c06c8f0 msgr2=0x7fee4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.844+0000 7fee75e00640 1 --2- 172.21.15.40:0/1748822769 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fee4c06c8f0 0x7fee4c06edb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fee60034000 tx=0x7fee6002c000 comp rx=0 tx=0).stop 2024-04-26T23:10:18.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.844+0000 7fee75e00640 1 -- 172.21.15.40:0/1748822769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee70146060 msgr2=0x7fee701f85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:18.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.844+0000 7fee75e00640 1 --2- 172.21.15.40:0/1748822769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee70146060 0x7fee701f85a0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7fee5800e9b0 tx=0x7fee5800ee70 comp rx=0 tx=0).stop 2024-04-26T23:10:18.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.844+0000 7fee75400640 1 -- 172.21.15.40:0/1748822769 reap_dead start 2024-04-26T23:10:18.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.844+0000 7fee75e00640 1 -- 172.21.15.40:0/1748822769 shutdown_connections 2024-04-26T23:10:18.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.844+0000 7fee75e00640 1 -- 172.21.15.40:0/1748822769 >> 172.21.15.40:0/1748822769 conn(0x7fee70075eb0 msgr2=0x7fee700d06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:18.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.845+0000 7fee75e00640 1 -- 172.21.15.40:0/1748822769 shutdown_connections 2024-04-26T23:10:18.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:18.845+0000 7fee75e00640 1 -- 172.21.15.40:0/1748822769 wait complete. 2024-04-26T23:10:18.862 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-26T23:10:19.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.065+0000 7fc007200640 1 Processor -- start 2024-04-26T23:10:19.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.065+0000 7fc007200640 1 -- start start 2024-04-26T23:10:19.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.067+0000 7fc007200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc0000cba70 0x7fc000073c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.067+0000 7fc007200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc00008c930 0x7fc0000c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.067+0000 7fc007200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0000cfd70 con 0x7fc00008c550 2024-04-26T23:10:19.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.067+0000 7fc007200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc000072350 con 0x7fc00008c930 2024-04-26T23:10:19.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.067+0000 7fc007200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc0000cfb20 con 0x7fc0000cba70 2024-04-26T23:10:19.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.068+0000 7fc005e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc00008c550 0x7fc0000727e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45908/0 (socket says 172.21.15.40:45908) 2024-04-26T23:10:19.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.068+0000 7fc005e00640 1 -- 172.21.15.40:0/543890764 learned_addr learned my addr 172.21.15.40:0/543890764 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:19.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.068+0000 7fc006800640 1 --2- 172.21.15.40:0/543890764 >> v2:172.21.15.40:3300/0 conn(0x7fc00008c930 0x7fc0000c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.068+0000 7fc004a00640 1 -- 172.21.15.40:0/543890764 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3313389669 0 0) 0x7fc0000cfd70 con 0x7fc00008c550 2024-04-26T23:10:19.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.068+0000 7fc005400640 1 --2- 172.21.15.40:0/543890764 >> v2:172.21.15.103:3300/0 conn(0x7fc0000cba70 0x7fc000073c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.074+0000 7fc004a00640 1 -- 172.21.15.40:0/543890764 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfe8007930 con 0x7fc00008c550 2024-04-26T23:10:19.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.074+0000 7fc006800640 1 -- 172.21.15.40:0/543890764 >> v2:172.21.15.103:3300/0 conn(0x7fc0000cba70 msgr2=0x7fc000073c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:19.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.074+0000 7fc006800640 1 --2- 172.21.15.40:0/543890764 >> v2:172.21.15.103:3300/0 conn(0x7fc0000cba70 0x7fc000073c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.074+0000 7fc006800640 1 -- 172.21.15.40:0/543890764 >> v1:172.21.15.40:6789/0 conn(0x7fc00008c550 legacy=0x7fc0000727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.074+0000 7fc006800640 1 -- 172.21.15.40:0/543890764 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc000071d00 con 0x7fc00008c930 2024-04-26T23:10:19.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.074+0000 7fc005400640 1 --2- 172.21.15.40:0/543890764 >> v2:172.21.15.103:3300/0 conn(0x7fc0000cba70 0x7fc000073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:10:19.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.075+0000 7fc006800640 1 --2- 172.21.15.40:0/543890764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00008c930 0x7fc0000c58b0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fc0000b0560 tx=0x7fbffc033300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161b41cc568d02f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:19.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.075+0000 7fc004a00640 1 -- 172.21.15.40:0/543890764 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc00a2d0 con 0x7fc00008c930 2024-04-26T23:10:19.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.075+0000 7fc004a00640 1 -- 172.21.15.40:0/543890764 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbffc033e50 con 0x7fc00008c930 2024-04-26T23:10:19.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.075+0000 7fc004a00640 1 -- 172.21.15.40:0/543890764 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc00a690 con 0x7fc00008c930 2024-04-26T23:10:19.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.076+0000 7fc007200640 1 -- 172.21.15.40:0/543890764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00008c930 msgr2=0x7fc0000c58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.076+0000 7fc007200640 1 --2- 172.21.15.40:0/543890764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00008c930 0x7fc0000c58b0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fc0000b0560 tx=0x7fbffc033300 comp rx=0 tx=0).stop 2024-04-26T23:10:19.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.076+0000 7fc007200640 1 -- 172.21.15.40:0/543890764 shutdown_connections 2024-04-26T23:10:19.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.076+0000 7fc007200640 1 --2- 172.21.15.40:0/543890764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00008c930 0x7fc0000c58b0 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.076+0000 7fc007200640 1 --2- 172.21.15.40:0/543890764 >> v2:172.21.15.103:3300/0 conn(0x7fc0000cba70 0x7fc000073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.076+0000 7fc007200640 1 -- 172.21.15.40:0/543890764 >> 172.21.15.40:0/543890764 conn(0x7fc0000d0a10 msgr2=0x7fc0000d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:19.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.076+0000 7fc007200640 1 -- 172.21.15.40:0/543890764 shutdown_connections 2024-04-26T23:10:19.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.076+0000 7fc007200640 1 -- 172.21.15.40:0/543890764 wait complete. 2024-04-26T23:10:19.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.078+0000 7fc007200640 1 Processor -- start 2024-04-26T23:10:19.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.078+0000 7fc007200640 1 -- start start 2024-04-26T23:10:19.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.079+0000 7fc007200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00008c550 0x7fc0001eb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.079+0000 7fc007200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc00008c930 0x7fc0000ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.079+0000 7fc007200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0000cba70 0x7fc0001eef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.079+0000 7fc007200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc000072ef0 con 0x7fc00008c550 2024-04-26T23:10:19.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.079+0000 7fc007200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc000073940 con 0x7fc0000cba70 2024-04-26T23:10:19.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.079+0000 7fc007200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc000073070 con 0x7fc00008c930 2024-04-26T23:10:19.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.079+0000 7fc005e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00008c550 0x7fc0001eb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.079+0000 7fc005e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00008c550 0x7fc0001eb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58482/0 (socket says 172.21.15.40:58482) 2024-04-26T23:10:19.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.079+0000 7fc005e00640 1 -- 172.21.15.40:0/939504444 learned_addr learned my addr 172.21.15.40:0/939504444 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:19.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.080+0000 7fc005e00640 1 -- 172.21.15.40:0/939504444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc00008c930 msgr2=0x7fc0000ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.080+0000 7fc005e00640 1 --2- 172.21.15.40:0/939504444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc00008c930 0x7fc0000ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.080+0000 7fc005e00640 1 -- 172.21.15.40:0/939504444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0000cba70 msgr2=0x7fc0001eef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.080+0000 7fc005e00640 1 --2- 172.21.15.40:0/939504444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0000cba70 0x7fc0001eef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.080+0000 7fc005e00640 1 -- 172.21.15.40:0/939504444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0001f13f0 con 0x7fc00008c550 2024-04-26T23:10:19.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.080+0000 7fc005e00640 1 --2- 172.21.15.40:0/939504444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00008c550 0x7fc0001eb320 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fbff400c430 tx=0x7fbff400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:19.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.080+0000 7fbffa000640 1 -- 172.21.15.40:0/939504444 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4017070 con 0x7fc00008c550 2024-04-26T23:10:19.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.080+0000 7fbffa000640 1 -- 172.21.15.40:0/939504444 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff4005ce0 con 0x7fc00008c550 2024-04-26T23:10:19.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.080+0000 7fbffa000640 1 -- 172.21.15.40:0/939504444 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff40072f0 con 0x7fc00008c550 2024-04-26T23:10:19.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.080+0000 7fc007200640 1 -- 172.21.15.40:0/939504444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0001f1110 con 0x7fc00008c550 2024-04-26T23:10:19.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.081+0000 7fc007200640 1 -- 172.21.15.40:0/939504444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0000c7f60 con 0x7fc00008c550 2024-04-26T23:10:19.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.082+0000 7fc007200640 1 -- 172.21.15.40:0/939504444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc000069370 con 0x7fc00008c550 2024-04-26T23:10:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.083+0000 7fbffa000640 1 -- 172.21.15.40:0/939504444 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbff401a070 con 0x7fc00008c550 2024-04-26T23:10:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.084+0000 7fbffa000640 1 --2- 172.21.15.40:0/939504444 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfe006c920 0x7fbfe006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.084+0000 7fbffa000640 1 -- 172.21.15.40:0/939504444 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7fbff408afa0 con 0x7fc00008c550 2024-04-26T23:10:19.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.085+0000 7fc005400640 1 --2- 172.21.15.40:0/939504444 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfe006c920 0x7fbfe006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.086+0000 7fc005400640 1 --2- 172.21.15.40:0/939504444 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfe006c920 0x7fbfe006ede0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc0000b2290 tx=0x7fbfec006d80 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:19.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.089+0000 7fbffa000640 1 -- 172.21.15.40:0/939504444 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fbff40593f0 con 0x7fc00008c550 2024-04-26T23:10:19.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.300+0000 7fc007200640 1 -- 172.21.15.40:0/939504444 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fc0000ca460 con 0x7fbfe006c920 2024-04-26T23:10:19.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.301+0000 7fbffa000640 1 -- 172.21.15.40:0/939504444 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc0000ca460 con 0x7fbfe006c920 2024-04-26T23:10:19.301 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 1 to deep-scrub 2024-04-26T23:10:19.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.304+0000 7fc007200640 1 -- 172.21.15.40:0/939504444 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfe006c920 msgr2=0x7fbfe006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.304+0000 7fc007200640 1 --2- 172.21.15.40:0/939504444 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfe006c920 0x7fbfe006ede0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc0000b2290 tx=0x7fbfec006d80 comp rx=0 tx=0).stop 2024-04-26T23:10:19.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.304+0000 7fc007200640 1 -- 172.21.15.40:0/939504444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00008c550 msgr2=0x7fc0001eb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.304+0000 7fc007200640 1 --2- 172.21.15.40:0/939504444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00008c550 0x7fc0001eb320 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fbff400c430 tx=0x7fbff400c8f0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.304+0000 7fc007200640 1 -- 172.21.15.40:0/939504444 shutdown_connections 2024-04-26T23:10:19.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.304+0000 7fc007200640 1 --2- 172.21.15.40:0/939504444 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fbfe006c920 0x7fbfe006ede0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.304+0000 7fc007200640 1 --2- 172.21.15.40:0/939504444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0000cba70 0x7fc0001eef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.304+0000 7fc007200640 1 --2- 172.21.15.40:0/939504444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc00008c930 0x7fc0000ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.304+0000 7fc007200640 1 --2- 172.21.15.40:0/939504444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00008c550 0x7fc0001eb320 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.304+0000 7fc007200640 1 -- 172.21.15.40:0/939504444 >> 172.21.15.40:0/939504444 conn(0x7fc0000d0a10 msgr2=0x7fc0000727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:19.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.304+0000 7fc007200640 1 -- 172.21.15.40:0/939504444 shutdown_connections 2024-04-26T23:10:19.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.304+0000 7fc007200640 1 -- 172.21.15.40:0/939504444 wait complete. 2024-04-26T23:10:19.324 INFO:tasks.ceph:Scrubbing osd.2 2024-04-26T23:10:19.324 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-26T23:10:19.490 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:10:19.496 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:10:19.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.537+0000 7f5415a00640 1 Processor -- start 2024-04-26T23:10:19.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.537+0000 7f5415a00640 1 -- start start 2024-04-26T23:10:19.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.538+0000 7f5415a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f541015f420 0x7f54100d0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.539+0000 7f5415a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f54100d3b60 0x7f54100b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.539+0000 7f5415a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54100d0d50 con 0x7f54100b3d90 2024-04-26T23:10:19.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.539+0000 7f5415a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f54100d0a70 con 0x7f54100d3b60 2024-04-26T23:10:19.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.539+0000 7f5415a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f54100b1bf0 con 0x7f541015f420 2024-04-26T23:10:19.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.539+0000 7f5407e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f54100b3d90 0x7f54100d3450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45910/0 (socket says 172.21.15.40:45910) 2024-04-26T23:10:19.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.539+0000 7f5407e00640 1 -- 172.21.15.40:0/265779763 learned_addr learned my addr 172.21.15.40:0/265779763 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:19.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.539+0000 7f5415000640 1 --2- 172.21.15.40:0/265779763 >> v2:172.21.15.103:3300/0 conn(0x7f54100d3b60 0x7f54100b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.539+0000 7f540fe00640 1 --2- 172.21.15.40:0/265779763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f541015f420 0x7f54100d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.539+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3843243961 0 0) 0x7f54100d0d50 con 0x7f54100b3d90 2024-04-26T23:10:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53f4003520 con 0x7f54100b3d90 2024-04-26T23:10:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 570705336 0 0) 0x7f53f4003520 con 0x7f54100b3d90 2024-04-26T23:10:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f541015f420 msgr2=0x7f54100d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 --2- 172.21.15.40:0/265779763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f541015f420 0x7f54100d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 >> v2:172.21.15.103:3300/0 conn(0x7f54100d3b60 msgr2=0x7f54100b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 --2- 172.21.15.40:0/265779763 >> v2:172.21.15.103:3300/0 conn(0x7f54100d3b60 0x7f54100b26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54100b1ed0 con 0x7f54100b3d90 2024-04-26T23:10:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f53f8001670 con 0x7f54100b3d90 2024-04-26T23:10:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 >> v1:172.21.15.40:6789/0 conn(0x7f54100b3d90 legacy=0x7f54100d3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 --2- 172.21.15.40:0/265779763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f541015f420 0x7f53f40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 --2- 172.21.15.40:0/265779763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53f4003f80 0x7f53f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 --2- 172.21.15.40:0/265779763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f4004d40 0x7f53f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f54100d0a70 con 0x7f53f4004d40 2024-04-26T23:10:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f54100b1bf0 con 0x7f53f4003f80 2024-04-26T23:10:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f53f400ba70 con 0x7f541015f420 2024-04-26T23:10:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.540+0000 7f5415000640 1 --2- 172.21.15.40:0/265779763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f4004d40 0x7f53f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.541+0000 7f5407e00640 1 --2- 172.21.15.40:0/265779763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53f4003f80 0x7f53f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.541+0000 7f540fe00640 1 --2- 172.21.15.40:0/265779763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f541015f420 0x7f53f40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.541+0000 7f5415000640 1 -- 172.21.15.40:0/265779763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f541015f420 msgr2=0x7f53f40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.541+0000 7f5415000640 1 --2- 172.21.15.40:0/265779763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f541015f420 0x7f53f40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.541+0000 7f5415000640 1 -- 172.21.15.40:0/265779763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53f4003f80 msgr2=0x7f53f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.541+0000 7f5415000640 1 --2- 172.21.15.40:0/265779763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53f4003f80 0x7f53f4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.541+0000 7f5415000640 1 -- 172.21.15.40:0/265779763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f4003520 con 0x7f53f4004d40 2024-04-26T23:10:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.541+0000 7f540fe00640 1 --2- 172.21.15.40:0/265779763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f541015f420 0x7f53f40039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:10:19.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.542+0000 7f5415000640 1 --2- 172.21.15.40:0/265779763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f4004d40 0x7f53f40051c0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f541008f110 tx=0x7f53f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f404d8d30469795 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:19.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.542+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408009c10 con 0x7f53f4004d40 2024-04-26T23:10:19.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.542+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5408036070 con 0x7f53f4004d40 2024-04-26T23:10:19.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.542+0000 7f540f400640 1 -- 172.21.15.40:0/265779763 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408031850 con 0x7f53f4004d40 2024-04-26T23:10:19.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.543+0000 7f5415a00640 1 -- 172.21.15.40:0/265779763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f4004d40 msgr2=0x7f53f40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.543+0000 7f5415a00640 1 --2- 172.21.15.40:0/265779763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f4004d40 0x7f53f40051c0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f541008f110 tx=0x7f53f4005830 comp rx=0 tx=0).stop 2024-04-26T23:10:19.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.543+0000 7f5415000640 1 -- 172.21.15.40:0/265779763 reap_dead start 2024-04-26T23:10:19.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.543+0000 7f5415a00640 1 -- 172.21.15.40:0/265779763 shutdown_connections 2024-04-26T23:10:19.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.543+0000 7f5415a00640 1 -- 172.21.15.40:0/265779763 >> 172.21.15.40:0/265779763 conn(0x7f541009e1d0 msgr2=0x7f54100d4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:19.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.543+0000 7f5415a00640 1 -- 172.21.15.40:0/265779763 shutdown_connections 2024-04-26T23:10:19.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.543+0000 7f5415a00640 1 -- 172.21.15.40:0/265779763 wait complete. 2024-04-26T23:10:19.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.545+0000 7f5415a00640 1 Processor -- start 2024-04-26T23:10:19.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.545+0000 7f5415a00640 1 -- start start 2024-04-26T23:10:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.546+0000 7f5415a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53f4004d40 0x7f54100cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.546+0000 7f5415a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54100ba660 0x7f54101533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.546+0000 7f5415a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54100cd3b0 0x7f54100d0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.546+0000 7f5415a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f54100d0440 con 0x7f54100cd3b0 2024-04-26T23:10:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.546+0000 7f5415a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f54100d2e50 con 0x7f54100ba660 2024-04-26T23:10:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.546+0000 7f5415a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f54100d05c0 con 0x7f53f4004d40 2024-04-26T23:10:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.546+0000 7f5415000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54100cd3b0 0x7f54100d0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.546+0000 7f5415000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54100cd3b0 0x7f54100d0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58496/0 (socket says 172.21.15.40:58496) 2024-04-26T23:10:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.546+0000 7f5415000640 1 -- 172.21.15.40:0/1881324663 learned_addr learned my addr 172.21.15.40:0/1881324663 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.547+0000 7f5407e00640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54100ba660 0x7f54101533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.547+0000 7f540fe00640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53f4004d40 0x7f54100cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.547+0000 7f5415000640 1 -- 172.21.15.40:0/1881324663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53f4004d40 msgr2=0x7f54100cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.547+0000 7f5415000640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f53f4004d40 0x7f54100cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.547+0000 7f5415000640 1 -- 172.21.15.40:0/1881324663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54100ba660 msgr2=0x7f54101533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.547+0000 7f5415000640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54100ba660 0x7f54101533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.547+0000 7f5415000640 1 -- 172.21.15.40:0/1881324663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5410153930 con 0x7f54100cd3b0 2024-04-26T23:10:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.547+0000 7f5415000640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54100cd3b0 0x7f54100d0fb0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f5408033040 tx=0x7f5408009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.548+0000 7f540cc00640 1 -- 172.21.15.40:0/1881324663 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408009c10 con 0x7f54100cd3b0 2024-04-26T23:10:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.548+0000 7f540cc00640 1 -- 172.21.15.40:0/1881324663 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5408003990 con 0x7f54100cd3b0 2024-04-26T23:10:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.548+0000 7f540cc00640 1 -- 172.21.15.40:0/1881324663 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408003d80 con 0x7f54100cd3b0 2024-04-26T23:10:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.548+0000 7f5415a00640 1 -- 172.21.15.40:0/1881324663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54100cdc20 con 0x7f54100cd3b0 2024-04-26T23:10:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.548+0000 7f5415a00640 1 -- 172.21.15.40:0/1881324663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54100ce1e0 con 0x7f54100cd3b0 2024-04-26T23:10:19.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.549+0000 7f5415a00640 1 -- 172.21.15.40:0/1881324663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5410069370 con 0x7f54100cd3b0 2024-04-26T23:10:19.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.550+0000 7f540cc00640 1 -- 172.21.15.40:0/1881324663 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5408005d10 con 0x7f54100cd3b0 2024-04-26T23:10:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.552+0000 7f540cc00640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f53ec06c920 0x7f53ec06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.552+0000 7f540cc00640 1 -- 172.21.15.40:0/1881324663 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f54080ad2a0 con 0x7f54100cd3b0 2024-04-26T23:10:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.552+0000 7f540cc00640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] conn(0x7f53ec072f00 0x7f53ec075340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.552+0000 7f540cc00640 1 -- 172.21.15.40:0/1881324663 --> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f540800df90 con 0x7f53ec072f00 2024-04-26T23:10:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.553+0000 7f540cc00640 1 -- 172.21.15.40:0/1881324663 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f54080b4050 con 0x7f54100cd3b0 2024-04-26T23:10:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.553+0000 7f540fe00640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f53ec06c920 0x7f53ec06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.553+0000 7f5407e00640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] conn(0x7f53ec072f00 0x7f53ec075340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.553+0000 7f5407e00640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] conn(0x7f53ec072f00 0x7f53ec075340 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.554+0000 7f540fe00640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f53ec06c920 0x7f53ec06ede0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5400009a90 tx=0x7f5400008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:19.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.554+0000 7f540cc00640 1 -- 172.21.15.40:0/1881324663 <== osd.2 v2:172.21.15.40:6808/2108759315 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f540800df90 con 0x7f53ec072f00 2024-04-26T23:10:19.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.570+0000 7f5415a00640 1 -- 172.21.15.40:0/1881324663 --> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5410069ca0 con 0x7f53ec072f00 2024-04-26T23:10:19.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.571+0000 7f540cc00640 1 -- 172.21.15.40:0/1881324663 <== osd.2 v2:172.21.15.40:6808/2108759315 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5410069ca0 con 0x7f53ec072f00 2024-04-26T23:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T23:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:10:19.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.571+0000 7f5415a00640 1 -- 172.21.15.40:0/1881324663 >> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] conn(0x7f53ec072f00 msgr2=0x7f53ec075340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.571+0000 7f5415a00640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.40:6808/2108759315,v1:172.21.15.40:6809/2108759315] conn(0x7f53ec072f00 0x7f53ec075340 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.572+0000 7f5415a00640 1 -- 172.21.15.40:0/1881324663 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f53ec06c920 msgr2=0x7f53ec06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.572+0000 7f5415a00640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f53ec06c920 0x7f53ec06ede0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5400009a90 tx=0x7f5400008040 comp rx=0 tx=0).stop 2024-04-26T23:10:19.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.572+0000 7f5415a00640 1 -- 172.21.15.40:0/1881324663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54100cd3b0 msgr2=0x7f54100d0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.572+0000 7f5415a00640 1 --2- 172.21.15.40:0/1881324663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54100cd3b0 0x7f54100d0fb0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f5408033040 tx=0x7f5408009750 comp rx=0 tx=0).stop 2024-04-26T23:10:19.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.572+0000 7f5415000640 1 -- 172.21.15.40:0/1881324663 reap_dead start 2024-04-26T23:10:19.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.572+0000 7f5415a00640 1 -- 172.21.15.40:0/1881324663 shutdown_connections 2024-04-26T23:10:19.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.572+0000 7f5415a00640 1 -- 172.21.15.40:0/1881324663 >> 172.21.15.40:0/1881324663 conn(0x7f541009e1d0 msgr2=0x7f54100d3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:19.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.572+0000 7f5415a00640 1 -- 172.21.15.40:0/1881324663 shutdown_connections 2024-04-26T23:10:19.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.572+0000 7f5415a00640 1 -- 172.21.15.40:0/1881324663 wait complete. 2024-04-26T23:10:19.588 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-26T23:10:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.861+0000 7f9ff6e00640 1 Processor -- start 2024-04-26T23:10:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.861+0000 7f9ff6e00640 1 -- start start 2024-04-26T23:10:19.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.862+0000 7f9ff6e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9ff015ce60 0x7f9ff00afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.862+0000 7f9ff6e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ff00af7a0 0x7f9ff00ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.862+0000 7f9ff6e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ff00d0820 con 0x7f9ff0071fe0 2024-04-26T23:10:19.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.862+0000 7f9ff6e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9ff00cc450 con 0x7f9ff015ce60 2024-04-26T23:10:19.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.862+0000 7f9ff6e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ff01614c0 con 0x7f9ff00af7a0 2024-04-26T23:10:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.863+0000 7f9ff5a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9ff015ce60 0x7f9ff00afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.863+0000 7f9ff5a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9ff015ce60 0x7f9ff00afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56302/0 (socket says 172.21.15.40:56302) 2024-04-26T23:10:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.863+0000 7f9ff5a00640 1 -- 172.21.15.40:0/790393186 learned_addr learned my addr 172.21.15.40:0/790393186 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.863+0000 7f9ff5000640 1 --2- 172.21.15.40:0/790393186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ff00af7a0 0x7f9ff00ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.864+0000 7f9ff5a00640 1 -- 172.21.15.40:0/790393186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ff00af7a0 msgr2=0x7f9ff00ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.864+0000 7f9ff5a00640 1 --2- 172.21.15.40:0/790393186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ff00af7a0 0x7f9ff00ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.864+0000 7f9ff5a00640 1 -- 172.21.15.40:0/790393186 >> v1:172.21.15.103:6789/0 conn(0x7f9ff0071fe0 legacy=0x7f9ff0073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.864+0000 7f9ff5a00640 1 -- 172.21.15.40:0/790393186 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff0161760 con 0x7f9ff015ce60 2024-04-26T23:10:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.864+0000 7f9fefe00640 1 -- 172.21.15.40:0/790393186 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1869065493 0 0) 0x7f9ff00d0820 con 0x7f9ff0071fe0 2024-04-26T23:10:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.865+0000 7f9ff5a00640 1 --2- 172.21.15.40:0/790393186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ff015ce60 0x7f9ff00afc60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4009870 tx=0x7f9fe402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f13ab492c579848 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.866+0000 7f9fefe00640 1 -- 172.21.15.40:0/790393186 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4037070 con 0x7f9ff015ce60 2024-04-26T23:10:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.866+0000 7f9fefe00640 1 -- 172.21.15.40:0/790393186 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fe4032440 con 0x7f9ff015ce60 2024-04-26T23:10:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.866+0000 7f9fefe00640 1 -- 172.21.15.40:0/790393186 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4032810 con 0x7f9ff015ce60 2024-04-26T23:10:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.867+0000 7f9ff6e00640 1 -- 172.21.15.40:0/790393186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ff015ce60 msgr2=0x7f9ff00afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.867+0000 7f9ff6e00640 1 --2- 172.21.15.40:0/790393186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ff015ce60 0x7f9ff00afc60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4009870 tx=0x7f9fe402a1c0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.867+0000 7f9ff6e00640 1 -- 172.21.15.40:0/790393186 shutdown_connections 2024-04-26T23:10:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.867+0000 7f9ff6e00640 1 --2- 172.21.15.40:0/790393186 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ff00af7a0 0x7f9ff00ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.867+0000 7f9ff6e00640 1 --2- 172.21.15.40:0/790393186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ff015ce60 0x7f9ff00afc60 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.867+0000 7f9ff6e00640 1 -- 172.21.15.40:0/790393186 >> 172.21.15.40:0/790393186 conn(0x7f9ff00d0a10 msgr2=0x7f9ff00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:19.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.867+0000 7f9ff6e00640 1 -- 172.21.15.40:0/790393186 shutdown_connections 2024-04-26T23:10:19.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.867+0000 7f9ff6e00640 1 -- 172.21.15.40:0/790393186 wait complete. 2024-04-26T23:10:19.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.869+0000 7f9ff6e00640 1 Processor -- start 2024-04-26T23:10:19.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.869+0000 7f9ff6e00640 1 -- start start 2024-04-26T23:10:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.870+0000 7f9ff6e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ff0071fe0 0x7f9ff01561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.870+0000 7f9ff6e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ff00af7a0 0x7f9ff0156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.870+0000 7f9ff6e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ff015ce60 0x7f9ff00c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.870+0000 7f9ff6e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff0073e70 con 0x7f9ff00af7a0 2024-04-26T23:10:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.870+0000 7f9ff6e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff00cf530 con 0x7f9ff015ce60 2024-04-26T23:10:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.870+0000 7f9ff6e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ff0073ff0 con 0x7f9ff0071fe0 2024-04-26T23:10:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.870+0000 7f9ff5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ff00af7a0 0x7f9ff0156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.870+0000 7f9ff5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ff00af7a0 0x7f9ff0156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58504/0 (socket says 172.21.15.40:58504) 2024-04-26T23:10:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.870+0000 7f9ff5000640 1 -- 172.21.15.40:0/885269224 learned_addr learned my addr 172.21.15.40:0/885269224 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.870+0000 7f9ff6400640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ff015ce60 0x7f9ff00c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.871+0000 7f9ff5a00640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ff0071fe0 0x7f9ff01561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.871+0000 7f9ff5000640 1 -- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ff0071fe0 msgr2=0x7f9ff01561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.871+0000 7f9ff5000640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ff0071fe0 0x7f9ff01561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.871+0000 7f9ff5000640 1 -- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ff015ce60 msgr2=0x7f9ff00c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.871+0000 7f9ff5000640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ff015ce60 0x7f9ff00c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.871+0000 7f9ff5000640 1 -- 172.21.15.40:0/885269224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff00c8020 con 0x7f9ff00af7a0 2024-04-26T23:10:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.871+0000 7f9ff6400640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ff015ce60 0x7f9ff00c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.871+0000 7f9ff5000640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ff00af7a0 0x7f9ff0156960 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00e9b0 tx=0x7f9fdc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.871+0000 7f9fed600640 1 -- 172.21.15.40:0/885269224 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc017070 con 0x7f9ff00af7a0 2024-04-26T23:10:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.871+0000 7f9fed600640 1 -- 172.21.15.40:0/885269224 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fdc00c9a0 con 0x7f9ff00af7a0 2024-04-26T23:10:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.871+0000 7f9fed600640 1 -- 172.21.15.40:0/885269224 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc00cd90 con 0x7f9ff00af7a0 2024-04-26T23:10:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.871+0000 7f9ff6e00640 1 -- 172.21.15.40:0/885269224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff00c82c0 con 0x7f9ff00af7a0 2024-04-26T23:10:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.872+0000 7f9ff6e00640 1 -- 172.21.15.40:0/885269224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff00c8880 con 0x7f9ff00af7a0 2024-04-26T23:10:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.873+0000 7f9ff6e00640 1 -- 172.21.15.40:0/885269224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff0069370 con 0x7f9ff00af7a0 2024-04-26T23:10:19.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.874+0000 7f9fed600640 1 -- 172.21.15.40:0/885269224 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9fdc01a070 con 0x7f9ff00af7a0 2024-04-26T23:10:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.875+0000 7f9fed600640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9fd006c8f0 0x7f9fd006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.875+0000 7f9fed600640 1 -- 172.21.15.40:0/885269224 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f9fdc08be00 con 0x7f9ff00af7a0 2024-04-26T23:10:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.875+0000 7f9ff5a00640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9fd006c8f0 0x7f9fd006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.877+0000 7f9ff5a00640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9fd006c8f0 0x7f9fd006edb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4034000 tx=0x7f9fe402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:19.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:19.879+0000 7f9fed600640 1 -- 172.21.15.40:0/885269224 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f9fdc05a2f0 con 0x7f9ff00af7a0 2024-04-26T23:10:20.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.083+0000 7f9ff6e00640 1 -- 172.21.15.40:0/885269224 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f9ff00d01f0 con 0x7f9fd006c8f0 2024-04-26T23:10:20.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.084+0000 7f9fed600640 1 -- 172.21.15.40:0/885269224 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9ff00d01f0 con 0x7f9fd006c8f0 2024-04-26T23:10:20.085 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 2 to deep-scrub 2024-04-26T23:10:20.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.087+0000 7f9ff6e00640 1 -- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9fd006c8f0 msgr2=0x7f9fd006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:20.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.087+0000 7f9ff6e00640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9fd006c8f0 0x7f9fd006edb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4034000 tx=0x7f9fe402e000 comp rx=0 tx=0).stop 2024-04-26T23:10:20.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.087+0000 7f9ff6e00640 1 -- 172.21.15.40:0/885269224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ff00af7a0 msgr2=0x7f9ff0156960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:20.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.087+0000 7f9ff6e00640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ff00af7a0 0x7f9ff0156960 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00e9b0 tx=0x7f9fdc00ee70 comp rx=0 tx=0).stop 2024-04-26T23:10:20.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.087+0000 7f9ff6e00640 1 -- 172.21.15.40:0/885269224 shutdown_connections 2024-04-26T23:10:20.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.088+0000 7f9ff6e00640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f9fd006c8f0 0x7f9fd006edb0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.088+0000 7f9ff6e00640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ff015ce60 0x7f9ff00c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.088+0000 7f9ff6e00640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ff00af7a0 0x7f9ff0156960 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.088+0000 7f9ff6e00640 1 --2- 172.21.15.40:0/885269224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ff0071fe0 0x7f9ff01561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.088+0000 7f9ff6e00640 1 -- 172.21.15.40:0/885269224 >> 172.21.15.40:0/885269224 conn(0x7f9ff00d0a10 msgr2=0x7f9ff00b1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:20.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.088+0000 7f9ff6e00640 1 -- 172.21.15.40:0/885269224 shutdown_connections 2024-04-26T23:10:20.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.088+0000 7f9ff6e00640 1 -- 172.21.15.40:0/885269224 wait complete. 2024-04-26T23:10:20.107 INFO:tasks.ceph:Scrubbing osd.3 2024-04-26T23:10:20.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-26T23:10:20.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.327+0000 7f97f0c00640 1 Processor -- start 2024-04-26T23:10:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.327+0000 7f97f0c00640 1 -- start start 2024-04-26T23:10:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.329+0000 7f97f0c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f97ec14d090 0x7f97ec164620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.329+0000 7f97f0c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97ec0b3df0 0x7f97ec164d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.329+0000 7f97f0c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f97ec1652b0 0x7f97ec0b5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.329+0000 7f97f0c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f97ec164b60 con 0x7f97ec14d090 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.329+0000 7f97f0c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f97ec0b4280 con 0x7f97ec1652b0 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.329+0000 7f97f0c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f97ec0b4560 con 0x7f97ec0b3df0 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.329+0000 7f97e9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f97ec14d090 0x7f97ec164620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.330+0000 7f97e9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f97ec14d090 0x7f97ec164620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58512/0 (socket says 172.21.15.40:58512) 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.330+0000 7f97e9e00640 1 -- 172.21.15.40:0/1353697452 learned_addr learned my addr 172.21.15.40:0/1353697452 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.330+0000 7f97e1400640 1 --2- 172.21.15.40:0/1353697452 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97ec0b3df0 0x7f97ec164d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.330+0000 7f97ea800640 1 --2- 172.21.15.40:0/1353697452 >> v2:172.21.15.103:3300/0 conn(0x7f97ec1652b0 0x7f97ec0b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.330+0000 7f97e9e00640 1 -- 172.21.15.40:0/1353697452 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97ec0b3df0 msgr2=0x7f97ec164d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.330+0000 7f97e9e00640 1 --2- 172.21.15.40:0/1353697452 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97ec0b3df0 0x7f97ec164d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.330+0000 7f97e9e00640 1 -- 172.21.15.40:0/1353697452 >> v2:172.21.15.103:3300/0 conn(0x7f97ec1652b0 msgr2=0x7f97ec0b5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.330+0000 7f97e9e00640 1 --2- 172.21.15.40:0/1353697452 >> v2:172.21.15.103:3300/0 conn(0x7f97ec1652b0 0x7f97ec0b5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.330+0000 7f97e9e00640 1 -- 172.21.15.40:0/1353697452 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ec0b4840 con 0x7f97ec14d090 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.330+0000 7f97ea800640 1 --2- 172.21.15.40:0/1353697452 >> v2:172.21.15.103:3300/0 conn(0x7f97ec1652b0 0x7f97ec0b5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.331+0000 7f97e9e00640 1 --2- 172.21.15.40:0/1353697452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97ec14d090 0x7f97ec164620 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f97ec070350 tx=0x7f97dc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d538cd99b3a4a7b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.331+0000 7f97e9400640 1 -- 172.21.15.40:0/1353697452 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc036070 con 0x7f97ec14d090 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.331+0000 7f97e9400640 1 -- 172.21.15.40:0/1353697452 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97dc031600 con 0x7f97ec14d090 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.331+0000 7f97e9400640 1 -- 172.21.15.40:0/1353697452 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc0319d0 con 0x7f97ec14d090 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.332+0000 7f97f0c00640 1 -- 172.21.15.40:0/1353697452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97ec14d090 msgr2=0x7f97ec164620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.332+0000 7f97f0c00640 1 --2- 172.21.15.40:0/1353697452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97ec14d090 0x7f97ec164620 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f97ec070350 tx=0x7f97dc02a6a0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.332+0000 7f97f0c00640 1 -- 172.21.15.40:0/1353697452 shutdown_connections 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.332+0000 7f97f0c00640 1 --2- 172.21.15.40:0/1353697452 >> v2:172.21.15.103:3300/0 conn(0x7f97ec1652b0 0x7f97ec0b5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.332+0000 7f97f0c00640 1 --2- 172.21.15.40:0/1353697452 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97ec0b3df0 0x7f97ec164d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.332+0000 7f97f0c00640 1 --2- 172.21.15.40:0/1353697452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97ec14d090 0x7f97ec164620 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.332+0000 7f97f0c00640 1 -- 172.21.15.40:0/1353697452 >> 172.21.15.40:0/1353697452 conn(0x7f97ec06f310 msgr2=0x7f97ec06f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.332+0000 7f97f0c00640 1 -- 172.21.15.40:0/1353697452 shutdown_connections 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.332+0000 7f97f0c00640 1 -- 172.21.15.40:0/1353697452 wait complete. 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.334+0000 7f97f0c00640 1 Processor -- start 2024-04-26T23:10:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.334+0000 7f97f0c00640 1 -- start start 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.335+0000 7f97f0c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97ec14d090 0x7f97ec147b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.335+0000 7f97f0c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97ec1652b0 0x7f97ec1639c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.335+0000 7f97f0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97ec143ac0 0x7f97ec143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.335+0000 7f97f0c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec0b5860 con 0x7f97ec143ac0 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.335+0000 7f97f0c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec0737a0 con 0x7f97ec14d090 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.335+0000 7f97f0c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f97ec0b59e0 con 0x7f97ec1652b0 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.336+0000 7f97ea800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97ec143ac0 0x7f97ec143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.336+0000 7f97ea800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97ec143ac0 0x7f97ec143f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58518/0 (socket says 172.21.15.40:58518) 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.336+0000 7f97ea800640 1 -- 172.21.15.40:0/2327482025 learned_addr learned my addr 172.21.15.40:0/2327482025 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.336+0000 7f97e9e00640 1 --2- 172.21.15.40:0/2327482025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97ec14d090 0x7f97ec147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.336+0000 7f97ea800640 1 -- 172.21.15.40:0/2327482025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97ec1652b0 msgr2=0x7f97ec1639c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.336+0000 7f97ea800640 1 --2- 172.21.15.40:0/2327482025 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97ec1652b0 0x7f97ec1639c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.336+0000 7f97ea800640 1 -- 172.21.15.40:0/2327482025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97ec14d090 msgr2=0x7f97ec147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.336+0000 7f97ea800640 1 --2- 172.21.15.40:0/2327482025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97ec14d090 0x7f97ec147b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.336+0000 7f97ea800640 1 -- 172.21.15.40:0/2327482025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ec163f00 con 0x7f97ec143ac0 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.336+0000 7f97ea800640 1 --2- 172.21.15.40:0/2327482025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97ec143ac0 0x7f97ec143f70 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f97e400c7c0 tx=0x7f97e400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.337+0000 7f97e2a00640 1 -- 172.21.15.40:0/2327482025 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4015020 con 0x7f97ec143ac0 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.337+0000 7f97e2a00640 1 -- 172.21.15.40:0/2327482025 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e4004230 con 0x7f97ec143ac0 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.337+0000 7f97e2a00640 1 -- 172.21.15.40:0/2327482025 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4007620 con 0x7f97ec143ac0 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.337+0000 7f97f0c00640 1 -- 172.21.15.40:0/2327482025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97ec1f9260 con 0x7f97ec143ac0 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.337+0000 7f97f0c00640 1 -- 172.21.15.40:0/2327482025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97ec142830 con 0x7f97ec143ac0 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.338+0000 7f97f0c00640 1 -- 172.21.15.40:0/2327482025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f97ec069600 con 0x7f97ec143ac0 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.339+0000 7f97e2a00640 1 -- 172.21.15.40:0/2327482025 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f97e4018020 con 0x7f97ec143ac0 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.340+0000 7f97e2a00640 1 --2- 172.21.15.40:0/2327482025 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f97c806c8f0 0x7f97c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.340+0000 7f97e2a00640 1 -- 172.21.15.40:0/2327482025 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f97e408bac0 con 0x7f97ec143ac0 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.340+0000 7f97e2a00640 1 --2- 172.21.15.40:0/2327482025 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x7f97c8072ed0 0x7f97c8075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.340+0000 7f97e2a00640 1 -- 172.21.15.40:0/2327482025 --> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97c80759e0 con 0x7f97c8072ed0 2024-04-26T23:10:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.340+0000 7f97e2a00640 1 -- 172.21.15.40:0/2327482025 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f97e408de60 con 0x7f97ec143ac0 2024-04-26T23:10:20.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.341+0000 7f97e1400640 1 --2- 172.21.15.40:0/2327482025 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x7f97c8072ed0 0x7f97c8075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:20.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.341+0000 7f97e9e00640 1 --2- 172.21.15.40:0/2327482025 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f97c806c8f0 0x7f97c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:20.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.341+0000 7f97e1400640 1 --2- 172.21.15.40:0/2327482025 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x7f97c8072ed0 0x7f97c8075310 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:20.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.342+0000 7f97e9e00640 1 --2- 172.21.15.40:0/2327482025 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f97c806c8f0 0x7f97c806edb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f97dc033000 tx=0x7f97dc009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:20.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.342+0000 7f97e2a00640 1 -- 172.21.15.40:0/2327482025 <== osd.3 v2:172.21.15.40:6823/378872239 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97c80759e0 con 0x7f97c8072ed0 2024-04-26T23:10:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.357+0000 7f97f0c00640 1 -- 172.21.15.40:0/2327482025 --> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f97ec072050 con 0x7f97c8072ed0 2024-04-26T23:10:20.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.357+0000 7f97e2a00640 1 -- 172.21.15.40:0/2327482025 <== osd.3 v2:172.21.15.40:6823/378872239 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f97ec072050 con 0x7f97c8072ed0 2024-04-26T23:10:20.358 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:10:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T23:10:20.358 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:10:20.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.358+0000 7f97f0c00640 1 -- 172.21.15.40:0/2327482025 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x7f97c8072ed0 msgr2=0x7f97c8075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:20.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.358+0000 7f97f0c00640 1 --2- 172.21.15.40:0/2327482025 >> [v2:172.21.15.40:6823/378872239,v1:172.21.15.40:6825/378872239] conn(0x7f97c8072ed0 0x7f97c8075310 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.358+0000 7f97f0c00640 1 -- 172.21.15.40:0/2327482025 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f97c806c8f0 msgr2=0x7f97c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:20.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.358+0000 7f97f0c00640 1 --2- 172.21.15.40:0/2327482025 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f97c806c8f0 0x7f97c806edb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f97dc033000 tx=0x7f97dc009000 comp rx=0 tx=0).stop 2024-04-26T23:10:20.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.358+0000 7f97f0c00640 1 -- 172.21.15.40:0/2327482025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97ec143ac0 msgr2=0x7f97ec143f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:20.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.358+0000 7f97f0c00640 1 --2- 172.21.15.40:0/2327482025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97ec143ac0 0x7f97ec143f70 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f97e400c7c0 tx=0x7f97e400cc80 comp rx=0 tx=0).stop 2024-04-26T23:10:20.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.358+0000 7f97ea800640 1 -- 172.21.15.40:0/2327482025 reap_dead start 2024-04-26T23:10:20.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.358+0000 7f97f0c00640 1 -- 172.21.15.40:0/2327482025 shutdown_connections 2024-04-26T23:10:20.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.358+0000 7f97f0c00640 1 -- 172.21.15.40:0/2327482025 >> 172.21.15.40:0/2327482025 conn(0x7f97ec06f310 msgr2=0x7f97ec14bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:20.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.358+0000 7f97f0c00640 1 -- 172.21.15.40:0/2327482025 shutdown_connections 2024-04-26T23:10:20.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.358+0000 7f97f0c00640 1 -- 172.21.15.40:0/2327482025 wait complete. 2024-04-26T23:10:20.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-26T23:10:20.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.593+0000 7f7fe6c00640 1 Processor -- start 2024-04-26T23:10:20.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.593+0000 7f7fe6c00640 1 -- start start 2024-04-26T23:10:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.594+0000 7f7fe6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe015ce60 0x7f7fe00afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.595+0000 7f7fe6c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7fe0071fe0 0x7f7fe0073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.595+0000 7f7fe6c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fe00d0820 con 0x7f7fe00af7a0 2024-04-26T23:10:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.595+0000 7f7fe6c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f7fe00cc450 con 0x7f7fe0071fe0 2024-04-26T23:10:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.595+0000 7f7fe6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe01614c0 con 0x7f7fe015ce60 2024-04-26T23:10:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.595+0000 7f7fe6200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7fe0071fe0 0x7f7fe0073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.595+0000 7f7fe6200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7fe0071fe0 0x7f7fe0073760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58520/0 (socket says 172.21.15.40:58520) 2024-04-26T23:10:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.595+0000 7f7fe6200640 1 -- 172.21.15.40:0/1513765004 learned_addr learned my addr 172.21.15.40:0/1513765004 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.595+0000 7f7fe5800640 1 --2- 172.21.15.40:0/1513765004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe015ce60 0x7f7fe00afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.596+0000 7f7fe6200640 1 -- 172.21.15.40:0/1513765004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe015ce60 msgr2=0x7f7fe00afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.596+0000 7f7fe6200640 1 --2- 172.21.15.40:0/1513765004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe015ce60 0x7f7fe00afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.596+0000 7f7fe6200640 1 -- 172.21.15.40:0/1513765004 >> v1:172.21.15.103:6789/0 conn(0x7f7fe00af7a0 legacy=0x7f7fe00ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.596+0000 7f7fe6200640 1 -- 172.21.15.40:0/1513765004 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe0161760 con 0x7f7fe0071fe0 2024-04-26T23:10:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.596+0000 7f7fe5800640 1 --2- 172.21.15.40:0/1513765004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe015ce60 0x7f7fe00afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.597+0000 7f7fe6200640 1 --2- 172.21.15.40:0/1513765004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fe0071fe0 0x7f7fe0073760 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8009870 tx=0x7f7fd802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a2dab084fcd16b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:20.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.597+0000 7f7fdfe00640 1 -- 172.21.15.40:0/1513765004 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8037070 con 0x7f7fe0071fe0 2024-04-26T23:10:20.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.597+0000 7f7fdfe00640 1 -- 172.21.15.40:0/1513765004 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fd8032440 con 0x7f7fe0071fe0 2024-04-26T23:10:20.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.597+0000 7f7fdfe00640 1 -- 172.21.15.40:0/1513765004 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8032810 con 0x7f7fe0071fe0 2024-04-26T23:10:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.598+0000 7f7fe6c00640 1 -- 172.21.15.40:0/1513765004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fe0071fe0 msgr2=0x7f7fe0073760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.598+0000 7f7fe6c00640 1 --2- 172.21.15.40:0/1513765004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fe0071fe0 0x7f7fe0073760 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8009870 tx=0x7f7fd802a180 comp rx=0 tx=0).stop 2024-04-26T23:10:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.598+0000 7f7fe6c00640 1 -- 172.21.15.40:0/1513765004 shutdown_connections 2024-04-26T23:10:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.598+0000 7f7fe6c00640 1 --2- 172.21.15.40:0/1513765004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fe0071fe0 0x7f7fe0073760 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.598+0000 7f7fe6c00640 1 --2- 172.21.15.40:0/1513765004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe015ce60 0x7f7fe00afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.598+0000 7f7fe6c00640 1 -- 172.21.15.40:0/1513765004 >> 172.21.15.40:0/1513765004 conn(0x7f7fe00d0a10 msgr2=0x7f7fe00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.598+0000 7f7fe6c00640 1 -- 172.21.15.40:0/1513765004 shutdown_connections 2024-04-26T23:10:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.598+0000 7f7fe6c00640 1 -- 172.21.15.40:0/1513765004 wait complete. 2024-04-26T23:10:20.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.600+0000 7f7fe6c00640 1 Processor -- start 2024-04-26T23:10:20.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.600+0000 7f7fe6c00640 1 -- start start 2024-04-26T23:10:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.601+0000 7f7fe6c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fe0071fe0 0x7f7fe014b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.601+0000 7f7fe6c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fe00af7a0 0x7f7fe00b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.601+0000 7f7fe6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe015ce60 0x7f7fe00b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.601+0000 7f7fe6c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe00cc1f0 con 0x7f7fe00af7a0 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.601+0000 7f7fe6c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe00cc070 con 0x7f7fe0071fe0 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.601+0000 7f7fe6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe0073ca0 con 0x7f7fe015ce60 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.601+0000 7f7fe4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fe00af7a0 0x7f7fe00b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.601+0000 7f7fe4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fe00af7a0 0x7f7fe00b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58536/0 (socket says 172.21.15.40:58536) 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.601+0000 7f7fe4e00640 1 -- 172.21.15.40:0/2761123903 learned_addr learned my addr 172.21.15.40:0/2761123903 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.602+0000 7f7fe6200640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe015ce60 0x7f7fe00b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.602+0000 7f7fe5800640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fe0071fe0 0x7f7fe014b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.602+0000 7f7fe4e00640 1 -- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe015ce60 msgr2=0x7f7fe00b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.602+0000 7f7fe4e00640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe015ce60 0x7f7fe00b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.602+0000 7f7fe4e00640 1 -- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fe0071fe0 msgr2=0x7f7fe014b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.602+0000 7f7fe4e00640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fe0071fe0 0x7f7fe014b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.602+0000 7f7fe4e00640 1 -- 172.21.15.40:0/2761123903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe014b6f0 con 0x7f7fe00af7a0 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.602+0000 7f7fe5800640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fe0071fe0 0x7f7fe014b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.602+0000 7f7fe4e00640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fe00af7a0 0x7f7fe00b3650 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00c430 tx=0x7f7fcc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.602+0000 7f7fdd600640 1 -- 172.21.15.40:0/2761123903 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc011790 con 0x7f7fe00af7a0 2024-04-26T23:10:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.603+0000 7f7fdd600640 1 -- 172.21.15.40:0/2761123903 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fcc017070 con 0x7f7fe00af7a0 2024-04-26T23:10:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.603+0000 7f7fdd600640 1 -- 172.21.15.40:0/2761123903 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc011bb0 con 0x7f7fe00af7a0 2024-04-26T23:10:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.603+0000 7f7fe6c00640 1 -- 172.21.15.40:0/2761123903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe00b5810 con 0x7f7fe00af7a0 2024-04-26T23:10:20.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.603+0000 7f7fe6c00640 1 -- 172.21.15.40:0/2761123903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe00b6100 con 0x7f7fe00af7a0 2024-04-26T23:10:20.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.604+0000 7f7fe6c00640 1 -- 172.21.15.40:0/2761123903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fe0069370 con 0x7f7fe00af7a0 2024-04-26T23:10:20.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.605+0000 7f7fdd600640 1 -- 172.21.15.40:0/2761123903 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7fcc01c460 con 0x7f7fe00af7a0 2024-04-26T23:10:20.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.606+0000 7f7fdd600640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7fc006c8f0 0x7f7fc006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:20.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.607+0000 7f7fdd600640 1 -- 172.21.15.40:0/2761123903 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f7fcc08b9a0 con 0x7f7fe00af7a0 2024-04-26T23:10:20.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.607+0000 7f7fe5800640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7fc006c8f0 0x7f7fc006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:20.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.608+0000 7f7fe5800640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7fc006c8f0 0x7f7fc006edb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4007f40 tx=0x7f7fd4007490 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.611+0000 7f7fdd600640 1 -- 172.21.15.40:0/2761123903 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f7fcc059e80 con 0x7f7fe00af7a0 2024-04-26T23:10:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.818+0000 7f7fe6c00640 1 -- 172.21.15.40:0/2761123903 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f7fe0152f00 con 0x7f7fc006c8f0 2024-04-26T23:10:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.820+0000 7f7fdd600640 1 -- 172.21.15.40:0/2761123903 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7fe0152f00 con 0x7f7fc006c8f0 2024-04-26T23:10:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 3 to deep-scrub 2024-04-26T23:10:20.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.822+0000 7f7fe6c00640 1 -- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7fc006c8f0 msgr2=0x7f7fc006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:20.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.822+0000 7f7fe6c00640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7fc006c8f0 0x7f7fc006edb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4007f40 tx=0x7f7fd4007490 comp rx=0 tx=0).stop 2024-04-26T23:10:20.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.822+0000 7f7fe6c00640 1 -- 172.21.15.40:0/2761123903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fe00af7a0 msgr2=0x7f7fe00b3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:20.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.822+0000 7f7fe6c00640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fe00af7a0 0x7f7fe00b3650 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00c430 tx=0x7f7fcc00c8f0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.822+0000 7f7fe6c00640 1 -- 172.21.15.40:0/2761123903 shutdown_connections 2024-04-26T23:10:20.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.822+0000 7f7fe6c00640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7fc006c8f0 0x7f7fc006edb0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.822+0000 7f7fe6c00640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fe015ce60 0x7f7fe00b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.822+0000 7f7fe6c00640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fe00af7a0 0x7f7fe00b3650 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.822+0000 7f7fe6c00640 1 --2- 172.21.15.40:0/2761123903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fe0071fe0 0x7f7fe014b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:20.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.822+0000 7f7fe6c00640 1 -- 172.21.15.40:0/2761123903 >> 172.21.15.40:0/2761123903 conn(0x7f7fe00d0a10 msgr2=0x7f7fe00ceda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:20.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.822+0000 7f7fe6c00640 1 -- 172.21.15.40:0/2761123903 shutdown_connections 2024-04-26T23:10:20.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:20.823+0000 7f7fe6c00640 1 -- 172.21.15.40:0/2761123903 wait complete. 2024-04-26T23:10:20.842 INFO:tasks.ceph:Scrubbing osd.4 2024-04-26T23:10:20.843 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-26T23:10:21.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.066+0000 7f7946000640 1 Processor -- start 2024-04-26T23:10:21.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.066+0000 7f7946000640 1 -- start start 2024-04-26T23:10:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.068+0000 7f7946000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7940155f20 0x7f79400d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.068+0000 7f7946000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f794015f490 0x7f794006e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.068+0000 7f7946000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794015fca0 con 0x7f794006eaa0 2024-04-26T23:10:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.068+0000 7f7946000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f794015ff80 con 0x7f794015f490 2024-04-26T23:10:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.068+0000 7f7946000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7940160260 con 0x7f7940155f20 2024-04-26T23:10:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.069+0000 7f7944c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7940155f20 0x7f79400d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.069+0000 7f793fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f794015f490 0x7f794006e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.069+0000 7f7944c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7940155f20 0x7f79400d11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:51296/0 (socket says 172.21.15.40:51296) 2024-04-26T23:10:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.069+0000 7f7944c00640 1 -- 172.21.15.40:0/3842899085 learned_addr learned my addr 172.21.15.40:0/3842899085 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:21.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.070+0000 7f793fe00640 1 -- 172.21.15.40:0/3842899085 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7940155f20 msgr2=0x7f79400d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:21.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.070+0000 7f793fe00640 1 --2- 172.21.15.40:0/3842899085 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7940155f20 0x7f79400d11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.070+0000 7f793fe00640 1 -- 172.21.15.40:0/3842899085 >> v1:172.21.15.103:6789/0 conn(0x7f794006eaa0 legacy=0x7f79400b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.070+0000 7f793fe00640 1 -- 172.21.15.40:0/3842899085 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794006e040 con 0x7f794015f490 2024-04-26T23:10:21.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.071+0000 7f793fe00640 1 --2- 172.21.15.40:0/3842899085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794015f490 0x7f794006e560 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f792c0097a0 tx=0x7f792c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b08fad8e8f7f84c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:21.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.071+0000 7f793f400640 1 -- 172.21.15.40:0/3842899085 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c037070 con 0x7f794015f490 2024-04-26T23:10:21.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.071+0000 7f793f400640 1 -- 172.21.15.40:0/3842899085 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f792c032440 con 0x7f794015f490 2024-04-26T23:10:21.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.071+0000 7f793f400640 1 -- 172.21.15.40:0/3842899085 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c032810 con 0x7f794015f490 2024-04-26T23:10:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.072+0000 7f7946000640 1 -- 172.21.15.40:0/3842899085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794015f490 msgr2=0x7f794006e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.072+0000 7f7946000640 1 --2- 172.21.15.40:0/3842899085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794015f490 0x7f794006e560 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f792c0097a0 tx=0x7f792c02a180 comp rx=0 tx=0).stop 2024-04-26T23:10:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.072+0000 7f7946000640 1 -- 172.21.15.40:0/3842899085 shutdown_connections 2024-04-26T23:10:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.072+0000 7f7946000640 1 --2- 172.21.15.40:0/3842899085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794015f490 0x7f794006e560 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.072+0000 7f7946000640 1 --2- 172.21.15.40:0/3842899085 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7940155f20 0x7f79400d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.072+0000 7f7946000640 1 -- 172.21.15.40:0/3842899085 >> 172.21.15.40:0/3842899085 conn(0x7f7940075eb0 msgr2=0x7f7940068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.072+0000 7f7946000640 1 -- 172.21.15.40:0/3842899085 shutdown_connections 2024-04-26T23:10:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.072+0000 7f7946000640 1 -- 172.21.15.40:0/3842899085 wait complete. 2024-04-26T23:10:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.074+0000 7f7946000640 1 Processor -- start 2024-04-26T23:10:21.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.075+0000 7f7946000640 1 -- start start 2024-04-26T23:10:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.076+0000 7f7946000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794006eaa0 0x7f79400ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.076+0000 7f7946000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940155f20 0x7f79401f8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.076+0000 7f7946000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794015f490 0x7f79401f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.076+0000 7f7946000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f794015fa50 con 0x7f7940155f20 2024-04-26T23:10:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.076+0000 7f7946000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f794015f8d0 con 0x7f794015f490 2024-04-26T23:10:21.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.076+0000 7f7946000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f79400b5c10 con 0x7f794006eaa0 2024-04-26T23:10:21.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.076+0000 7f793fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940155f20 0x7f79401f8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.076+0000 7f793fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940155f20 0x7f79401f8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58544/0 (socket says 172.21.15.40:58544) 2024-04-26T23:10:21.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.076+0000 7f793fe00640 1 -- 172.21.15.40:0/2565680816 learned_addr learned my addr 172.21.15.40:0/2565680816 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:21.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.077+0000 7f7944c00640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794006eaa0 0x7f79400ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.077+0000 7f7945600640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794015f490 0x7f79401f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.077+0000 7f793fe00640 1 -- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794006eaa0 msgr2=0x7f79400ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.077+0000 7f793fe00640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f794006eaa0 0x7f79400ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.077+0000 7f793fe00640 1 -- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794015f490 msgr2=0x7f79401f4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.077+0000 7f793fe00640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794015f490 0x7f79401f4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.077+0000 7f793fe00640 1 -- 172.21.15.40:0/2565680816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79400aef50 con 0x7f7940155f20 2024-04-26T23:10:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.077+0000 7f793fe00640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940155f20 0x7f79401f8880 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f792c000c00 tx=0x7f792c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.078+0000 7f793cc00640 1 -- 172.21.15.40:0/2565680816 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c037040 con 0x7f7940155f20 2024-04-26T23:10:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.078+0000 7f793cc00640 1 -- 172.21.15.40:0/2565680816 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f792c042070 con 0x7f7940155f20 2024-04-26T23:10:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.078+0000 7f793cc00640 1 -- 172.21.15.40:0/2565680816 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c030920 con 0x7f7940155f20 2024-04-26T23:10:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.078+0000 7f7946000640 1 -- 172.21.15.40:0/2565680816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79400af230 con 0x7f7940155f20 2024-04-26T23:10:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.078+0000 7f7946000640 1 -- 172.21.15.40:0/2565680816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79400af7f0 con 0x7f7940155f20 2024-04-26T23:10:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.079+0000 7f7946000640 1 -- 172.21.15.40:0/2565680816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f79401f8dc0 con 0x7f7940155f20 2024-04-26T23:10:21.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.080+0000 7f793cc00640 1 -- 172.21.15.40:0/2565680816 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f792c030ac0 con 0x7f7940155f20 2024-04-26T23:10:21.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.081+0000 7f793cc00640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f792006c8f0 0x7f792006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.081+0000 7f793cc00640 1 -- 172.21.15.40:0/2565680816 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f792c0ac520 con 0x7f7940155f20 2024-04-26T23:10:21.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.081+0000 7f793cc00640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x7f7920072ed0 0x7f7920075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.081+0000 7f793cc00640 1 -- 172.21.15.40:0/2565680816 --> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f792c045e80 con 0x7f7920072ed0 2024-04-26T23:10:21.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.081+0000 7f793cc00640 1 -- 172.21.15.40:0/2565680816 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f792c0b3050 con 0x7f7940155f20 2024-04-26T23:10:21.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.082+0000 7f7944c00640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f792006c8f0 0x7f792006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.082+0000 7f7945600640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x7f7920072ed0 0x7f7920075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.083+0000 7f7944c00640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f792006c8f0 0x7f792006edb0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7934009b60 tx=0x7f7934008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:21.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.083+0000 7f7945600640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x7f7920072ed0 0x7f7920075310 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:21.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.085+0000 7f793cc00640 1 -- 172.21.15.40:0/2565680816 <== osd.4 v2:172.21.15.103:6803/801343304 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f792c045e80 con 0x7f7920072ed0 2024-04-26T23:10:21.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.100+0000 7f7946000640 1 -- 172.21.15.40:0/2565680816 --> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7940069370 con 0x7f7920072ed0 2024-04-26T23:10:21.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.101+0000 7f793cc00640 1 -- 172.21.15.40:0/2565680816 <== osd.4 v2:172.21.15.103:6803/801343304 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7940069370 con 0x7f7920072ed0 2024-04-26T23:10:21.101 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:10:21.101 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T23:10:21.101 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:10:21.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.102+0000 7f7946000640 1 -- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x7f7920072ed0 msgr2=0x7f7920075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.102+0000 7f7946000640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:6803/801343304,v1:172.21.15.103:6805/801343304] conn(0x7f7920072ed0 0x7f7920075310 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.102+0000 7f7946000640 1 -- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f792006c8f0 msgr2=0x7f792006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.102+0000 7f7946000640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f792006c8f0 0x7f792006edb0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7934009b60 tx=0x7f7934008040 comp rx=0 tx=0).stop 2024-04-26T23:10:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.102+0000 7f7946000640 1 -- 172.21.15.40:0/2565680816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940155f20 msgr2=0x7f79401f8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.102+0000 7f7946000640 1 --2- 172.21.15.40:0/2565680816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940155f20 0x7f79401f8880 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f792c000c00 tx=0x7f792c0046c0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.102+0000 7f7945600640 1 -- 172.21.15.40:0/2565680816 reap_dead start 2024-04-26T23:10:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.102+0000 7f7946000640 1 -- 172.21.15.40:0/2565680816 shutdown_connections 2024-04-26T23:10:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.102+0000 7f7946000640 1 -- 172.21.15.40:0/2565680816 >> 172.21.15.40:0/2565680816 conn(0x7f7940075eb0 msgr2=0x7f7940068090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.102+0000 7f7946000640 1 -- 172.21.15.40:0/2565680816 shutdown_connections 2024-04-26T23:10:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.102+0000 7f7946000640 1 -- 172.21.15.40:0/2565680816 wait complete. 2024-04-26T23:10:21.120 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-26T23:10:21.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.327+0000 7f929b400640 1 Processor -- start 2024-04-26T23:10:21.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.328+0000 7f929b400640 1 -- start start 2024-04-26T23:10:21.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.329+0000 7f929b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f929415d1e0 0x7f92940d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.329+0000 7f929b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92940d0f20 0x7f92940d1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.329+0000 7f929b400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92940ce840 con 0x7f92940d18c0 2024-04-26T23:10:21.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.329+0000 7f929b400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f92940cede0 con 0x7f929415d1e0 2024-04-26T23:10:21.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.329+0000 7f929b400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9294070650 con 0x7f92940d0f20 2024-04-26T23:10:21.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.330+0000 7f9298a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f929415d1e0 0x7f92940d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.330+0000 7f9298a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f929415d1e0 0x7f92940d09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58548/0 (socket says 172.21.15.40:58548) 2024-04-26T23:10:21.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.330+0000 7f9298a00640 1 -- 172.21.15.40:0/3916623459 learned_addr learned my addr 172.21.15.40:0/3916623459 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.330+0000 7f9298a00640 1 -- 172.21.15.40:0/3916623459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92940d0f20 msgr2=0x7f92940d1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.330+0000 7f9298a00640 1 --2- 172.21.15.40:0/3916623459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92940d0f20 0x7f92940d1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.330+0000 7f9298a00640 1 -- 172.21.15.40:0/3916623459 >> v1:172.21.15.103:6789/0 conn(0x7f92940d18c0 legacy=0x7f92940cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.330+0000 7f9298a00640 1 -- 172.21.15.40:0/3916623459 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92940ceb00 con 0x7f929415d1e0 2024-04-26T23:10:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.331+0000 7f9298a00640 1 --2- 172.21.15.40:0/3916623459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f929415d1e0 0x7f92940d09e0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f9288009870 tx=0x7f928802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa0ae8623c140df0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:21.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.332+0000 7f9293400640 1 -- 172.21.15.40:0/3916623459 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288037070 con 0x7f929415d1e0 2024-04-26T23:10:21.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.332+0000 7f9293400640 1 -- 172.21.15.40:0/3916623459 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9288032440 con 0x7f929415d1e0 2024-04-26T23:10:21.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.332+0000 7f9293400640 1 -- 172.21.15.40:0/3916623459 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288032810 con 0x7f929415d1e0 2024-04-26T23:10:21.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.332+0000 7f929b400640 1 -- 172.21.15.40:0/3916623459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f929415d1e0 msgr2=0x7f92940d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.332+0000 7f929b400640 1 --2- 172.21.15.40:0/3916623459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f929415d1e0 0x7f92940d09e0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f9288009870 tx=0x7f928802a210 comp rx=0 tx=0).stop 2024-04-26T23:10:21.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.332+0000 7f929b400640 1 -- 172.21.15.40:0/3916623459 shutdown_connections 2024-04-26T23:10:21.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.332+0000 7f929b400640 1 --2- 172.21.15.40:0/3916623459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92940d0f20 0x7f92940d1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.332+0000 7f929b400640 1 --2- 172.21.15.40:0/3916623459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f929415d1e0 0x7f92940d09e0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.332+0000 7f929b400640 1 -- 172.21.15.40:0/3916623459 >> 172.21.15.40:0/3916623459 conn(0x7f92940d2da0 msgr2=0x7f92940d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:21.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.333+0000 7f929b400640 1 -- 172.21.15.40:0/3916623459 shutdown_connections 2024-04-26T23:10:21.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.333+0000 7f929b400640 1 -- 172.21.15.40:0/3916623459 wait complete. 2024-04-26T23:10:21.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.334+0000 7f929b400640 1 Processor -- start 2024-04-26T23:10:21.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.334+0000 7f929b400640 1 -- start start 2024-04-26T23:10:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.335+0000 7f929b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92940d0f20 0x7f92940c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.335+0000 7f929b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92940d18c0 0x7f9294150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.335+0000 7f929b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f929415d1e0 0x7f92940c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.335+0000 7f929b400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9294071af0 con 0x7f92940d18c0 2024-04-26T23:10:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.335+0000 7f929b400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f92940ce4c0 con 0x7f92940d0f20 2024-04-26T23:10:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.335+0000 7f929b400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9294071c70 con 0x7f929415d1e0 2024-04-26T23:10:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.336+0000 7f9293e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92940d18c0 0x7f9294150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.336+0000 7f9293e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92940d18c0 0x7f9294150f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58564/0 (socket says 172.21.15.40:58564) 2024-04-26T23:10:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.336+0000 7f9293e00640 1 -- 172.21.15.40:0/2883344627 learned_addr learned my addr 172.21.15.40:0/2883344627 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.336+0000 7f9298a00640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92940d0f20 0x7f92940c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.336+0000 7f9293e00640 1 -- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f929415d1e0 msgr2=0x7f92940c8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.336+0000 7f9299400640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f929415d1e0 0x7f92940c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.336+0000 7f9293e00640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f929415d1e0 0x7f92940c8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.336+0000 7f9293e00640 1 -- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92940d0f20 msgr2=0x7f92940c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.336+0000 7f9293e00640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92940d0f20 0x7f92940c9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.336+0000 7f9293e00640 1 -- 172.21.15.40:0/2883344627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92940cb060 con 0x7f92940d18c0 2024-04-26T23:10:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.337+0000 7f9293e00640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92940d18c0 0x7f9294150f60 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f928400b410 tx=0x7f928400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.337+0000 7f9290c00640 1 -- 172.21.15.40:0/2883344627 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284015020 con 0x7f92940d18c0 2024-04-26T23:10:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.337+0000 7f9290c00640 1 -- 172.21.15.40:0/2883344627 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9284011850 con 0x7f92940d18c0 2024-04-26T23:10:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.337+0000 7f9290c00640 1 -- 172.21.15.40:0/2883344627 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284011c40 con 0x7f92940d18c0 2024-04-26T23:10:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.337+0000 7f929b400640 1 -- 172.21.15.40:0/2883344627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92940ca450 con 0x7f92940d18c0 2024-04-26T23:10:21.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.337+0000 7f929b400640 1 -- 172.21.15.40:0/2883344627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92940caa10 con 0x7f92940d18c0 2024-04-26T23:10:21.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.338+0000 7f929b400640 1 -- 172.21.15.40:0/2883344627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92940694f0 con 0x7f92940d18c0 2024-04-26T23:10:21.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.339+0000 7f9290c00640 1 -- 172.21.15.40:0/2883344627 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9284018020 con 0x7f92940d18c0 2024-04-26T23:10:21.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.341+0000 7f9290c00640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f927806c8f0 0x7f927806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.341+0000 7f9290c00640 1 -- 172.21.15.40:0/2883344627 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f928408b980 con 0x7f92940d18c0 2024-04-26T23:10:21.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.342+0000 7f9298a00640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f927806c8f0 0x7f927806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.343+0000 7f9298a00640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f927806c8f0 0x7f927806edb0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f9288034000 tx=0x7f928802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:21.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.345+0000 7f9290c00640 1 -- 172.21.15.40:0/2883344627 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f928401f020 con 0x7f92940d18c0 2024-04-26T23:10:21.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.551+0000 7f929b400640 1 -- 172.21.15.40:0/2883344627 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f92940d0c00 con 0x7f927806c8f0 2024-04-26T23:10:21.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.552+0000 7f9290c00640 1 -- 172.21.15.40:0/2883344627 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f92940d0c00 con 0x7f927806c8f0 2024-04-26T23:10:21.552 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 4 to deep-scrub 2024-04-26T23:10:21.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.555+0000 7f929b400640 1 -- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f927806c8f0 msgr2=0x7f927806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.555+0000 7f929b400640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f927806c8f0 0x7f927806edb0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f9288034000 tx=0x7f928802e000 comp rx=0 tx=0).stop 2024-04-26T23:10:21.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.555+0000 7f929b400640 1 -- 172.21.15.40:0/2883344627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92940d18c0 msgr2=0x7f9294150f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.555+0000 7f929b400640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92940d18c0 0x7f9294150f60 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f928400b410 tx=0x7f928400b8d0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.555+0000 7f929b400640 1 -- 172.21.15.40:0/2883344627 shutdown_connections 2024-04-26T23:10:21.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.555+0000 7f929b400640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f927806c8f0 0x7f927806edb0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.555+0000 7f929b400640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f929415d1e0 0x7f92940c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.555+0000 7f929b400640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92940d18c0 0x7f9294150f60 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.555+0000 7f929b400640 1 --2- 172.21.15.40:0/2883344627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92940d0f20 0x7f92940c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.555+0000 7f929b400640 1 -- 172.21.15.40:0/2883344627 >> 172.21.15.40:0/2883344627 conn(0x7f92940d2da0 msgr2=0x7f92940cf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:21.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.555+0000 7f929b400640 1 -- 172.21.15.40:0/2883344627 shutdown_connections 2024-04-26T23:10:21.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.555+0000 7f929b400640 1 -- 172.21.15.40:0/2883344627 wait complete. 2024-04-26T23:10:21.574 INFO:tasks.ceph:Scrubbing osd.5 2024-04-26T23:10:21.575 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-26T23:10:21.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.796+0000 7f7b07e00640 1 Processor -- start 2024-04-26T23:10:21.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.796+0000 7f7b07e00640 1 -- start start 2024-04-26T23:10:21.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.797+0000 7f7b07e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b0815f420 0x7f7b080d0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.797+0000 7f7b07e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b080d0d50 con 0x7f7b080b3d90 2024-04-26T23:10:21.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.797+0000 7f7b07e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b080d0a70 con 0x7f7b080d3b60 2024-04-26T23:10:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.797+0000 7f7b07e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7b080b1bf0 con 0x7f7b0815f420 2024-04-26T23:10:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.797+0000 7f7b06000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7b080b3d90 0x7f7b080d3450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45918/0 (socket says 172.21.15.40:45918) 2024-04-26T23:10:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.797+0000 7f7b06000640 1 -- 172.21.15.40:0/1387568154 learned_addr learned my addr 172.21.15.40:0/1387568154 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.798+0000 7f7b06a00640 1 --2- 172.21.15.40:0/1387568154 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b0815f420 0x7f7b080d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.798+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4173233936 0 0) 0x7f7b080d0d50 con 0x7f7b080b3d90 2024-04-26T23:10:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.798+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7aec003520 con 0x7f7b080b3d90 2024-04-26T23:10:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.798+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2556200016 0 0) 0x7f7aec003520 con 0x7f7b080b3d90 2024-04-26T23:10:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.798+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b0815f420 msgr2=0x7f7b080d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.798+0000 7f7b05600640 1 --2- 172.21.15.40:0/1387568154 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b0815f420 0x7f7b080d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.798+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 >> v1:172.21.15.103:6789/0 conn(0x7f7b080d3b60 legacy=0x7f7b080b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.798+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b080b1ed0 con 0x7f7b080b3d90 2024-04-26T23:10:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.798+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f7af0001670 con 0x7f7b080b3d90 2024-04-26T23:10:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.798+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 >> v1:172.21.15.40:6789/0 conn(0x7f7b080b3d90 legacy=0x7f7b080d3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.799+0000 7f7b05600640 1 --2- 172.21.15.40:0/1387568154 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b0815f420 0x7f7aec0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.799+0000 7f7b05600640 1 --2- 172.21.15.40:0/1387568154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aec003fb0 0x7f7aec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.799+0000 7f7b05600640 1 --2- 172.21.15.40:0/1387568154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7aec004d40 0x7f7aec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.799+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7b080b1bf0 con 0x7f7aec003fb0 2024-04-26T23:10:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.799+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec00bbb0 con 0x7f7aec004d40 2024-04-26T23:10:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.799+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7aec00bdb0 con 0x7f7b0815f420 2024-04-26T23:10:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.799+0000 7f7b06000640 1 --2- 172.21.15.40:0/1387568154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aec003fb0 0x7f7aec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.799+0000 7f7b06000640 1 -- 172.21.15.40:0/1387568154 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b0815f420 msgr2=0x7f7aec0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.799+0000 7f7b06000640 1 --2- 172.21.15.40:0/1387568154 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7b0815f420 0x7f7aec0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.799+0000 7f7b06000640 1 -- 172.21.15.40:0/1387568154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7aec004d40 msgr2=0x7f7aec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.799+0000 7f7b07400640 1 --2- 172.21.15.40:0/1387568154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7aec004d40 0x7f7aec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.799+0000 7f7b06000640 1 --2- 172.21.15.40:0/1387568154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7aec004d40 0x7f7aec0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.799+0000 7f7b06000640 1 -- 172.21.15.40:0/1387568154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aec003520 con 0x7f7aec003fb0 2024-04-26T23:10:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.801+0000 7f7b06000640 1 --2- 172.21.15.40:0/1387568154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aec003fb0 0x7f7aec004800 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f7af000c190 tx=0x7f7aec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7047506772c3efd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.801+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af0007c60 con 0x7f7aec003fb0 2024-04-26T23:10:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.801+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af0005ad0 con 0x7f7aec003fb0 2024-04-26T23:10:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.801+0000 7f7b05600640 1 -- 172.21.15.40:0/1387568154 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af0007070 con 0x7f7aec003fb0 2024-04-26T23:10:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.801+0000 7f7b07e00640 1 -- 172.21.15.40:0/1387568154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aec003fb0 msgr2=0x7f7aec004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.801+0000 7f7b07e00640 1 --2- 172.21.15.40:0/1387568154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aec003fb0 0x7f7aec004800 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f7af000c190 tx=0x7f7aec005830 comp rx=0 tx=0).stop 2024-04-26T23:10:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.802+0000 7f7b07400640 1 -- 172.21.15.40:0/1387568154 reap_dead start 2024-04-26T23:10:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.802+0000 7f7b07e00640 1 -- 172.21.15.40:0/1387568154 shutdown_connections 2024-04-26T23:10:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.802+0000 7f7b07e00640 1 -- 172.21.15.40:0/1387568154 >> 172.21.15.40:0/1387568154 conn(0x7f7b0809e1d0 msgr2=0x7f7b080d4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.802+0000 7f7b07e00640 1 -- 172.21.15.40:0/1387568154 shutdown_connections 2024-04-26T23:10:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.802+0000 7f7b07e00640 1 -- 172.21.15.40:0/1387568154 wait complete. 2024-04-26T23:10:21.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.804+0000 7f7b07e00640 1 Processor -- start 2024-04-26T23:10:21.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.804+0000 7f7b07e00640 1 -- start start 2024-04-26T23:10:21.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.805+0000 7f7b07e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7aec003fb0 0x7f7b080b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.805+0000 7f7b07e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b080b6210 0x7f7b080b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.805+0000 7f7b07e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b080b9d20 0x7f7b081fc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.805+0000 7f7b07e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7b080d0440 con 0x7f7b080b6210 2024-04-26T23:10:21.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.805+0000 7f7b07e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7b080d2e50 con 0x7f7b080b9d20 2024-04-26T23:10:21.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.805+0000 7f7b07e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7b080d05c0 con 0x7f7aec003fb0 2024-04-26T23:10:21.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.805+0000 7f7b06000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b080b6210 0x7f7b080b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.805+0000 7f7b06000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b080b6210 0x7f7b080b97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58596/0 (socket says 172.21.15.40:58596) 2024-04-26T23:10:21.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.805+0000 7f7b06000640 1 -- 172.21.15.40:0/1581960268 learned_addr learned my addr 172.21.15.40:0/1581960268 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:21.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.805+0000 7f7b06a00640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7aec003fb0 0x7f7b080b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.806+0000 7f7b07400640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b080b9d20 0x7f7b081fc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.806+0000 7f7b06000640 1 -- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7aec003fb0 msgr2=0x7f7b080b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.806+0000 7f7b06000640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7aec003fb0 0x7f7b080b92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.806+0000 7f7b06000640 1 -- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b080b9d20 msgr2=0x7f7b081fc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.806+0000 7f7b06000640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b080b9d20 0x7f7b081fc820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.806+0000 7f7b06000640 1 -- 172.21.15.40:0/1581960268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b080b7f00 con 0x7f7b080b6210 2024-04-26T23:10:21.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.806+0000 7f7b06000640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b080b6210 0x7f7b080b97e0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f7af002ad60 tx=0x7f7af0001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:21.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.806+0000 7f7aff400640 1 -- 172.21.15.40:0/1581960268 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af0040020 con 0x7f7b080b6210 2024-04-26T23:10:21.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.806+0000 7f7aff400640 1 -- 172.21.15.40:0/1581960268 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af00013b0 con 0x7f7b080b6210 2024-04-26T23:10:21.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.806+0000 7f7aff400640 1 -- 172.21.15.40:0/1581960268 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af0037040 con 0x7f7b080b6210 2024-04-26T23:10:21.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.806+0000 7f7b07e00640 1 -- 172.21.15.40:0/1581960268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b080cf1c0 con 0x7f7b080b6210 2024-04-26T23:10:21.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.807+0000 7f7b07e00640 1 -- 172.21.15.40:0/1581960268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b080b84a0 con 0x7f7b080b6210 2024-04-26T23:10:21.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.808+0000 7f7b07e00640 1 -- 172.21.15.40:0/1581960268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7b080ba1a0 con 0x7f7b080b6210 2024-04-26T23:10:21.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.808+0000 7f7aff400640 1 -- 172.21.15.40:0/1581960268 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7af0043020 con 0x7f7b080b6210 2024-04-26T23:10:21.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.809+0000 7f7aff400640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7ae406c8f0 0x7f7ae406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.809+0000 7f7aff400640 1 -- 172.21.15.40:0/1581960268 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f7af00ab8c0 con 0x7f7b080b6210 2024-04-26T23:10:21.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.810+0000 7f7aff400640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x7f7ae4072ed0 0x7f7ae4075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:21.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.810+0000 7f7aff400640 1 -- 172.21.15.40:0/1581960268 --> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7af0048e70 con 0x7f7ae4072ed0 2024-04-26T23:10:21.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.810+0000 7f7aff400640 1 -- 172.21.15.40:0/1581960268 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f7af00b02a0 con 0x7f7b080b6210 2024-04-26T23:10:21.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.810+0000 7f7b06a00640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7ae406c8f0 0x7f7ae406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.810+0000 7f7b07400640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x7f7ae4072ed0 0x7f7ae4075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:21.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.811+0000 7f7b06a00640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7ae406c8f0 0x7f7ae406edb0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7af8007520 tx=0x7f7af800d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:21.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.812+0000 7f7b07400640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x7f7ae4072ed0 0x7f7ae4075310 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:21.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.814+0000 7f7aff400640 1 -- 172.21.15.40:0/1581960268 <== osd.5 v2:172.21.15.103:6800/157083760 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7af0048e70 con 0x7f7ae4072ed0 2024-04-26T23:10:21.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.829+0000 7f7b07e00640 1 -- 172.21.15.40:0/1581960268 --> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7b08069370 con 0x7f7ae4072ed0 2024-04-26T23:10:21.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.830+0000 7f7aff400640 1 -- 172.21.15.40:0/1581960268 <== osd.5 v2:172.21.15.103:6800/157083760 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7b08069370 con 0x7f7ae4072ed0 2024-04-26T23:10:21.831 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:10:21.831 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T23:10:21.831 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:10:21.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.831+0000 7f7b07e00640 1 -- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x7f7ae4072ed0 msgr2=0x7f7ae4075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.831+0000 7f7b07e00640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:6800/157083760,v1:172.21.15.103:6801/157083760] conn(0x7f7ae4072ed0 0x7f7ae4075310 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:21.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.831+0000 7f7b07e00640 1 -- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7ae406c8f0 msgr2=0x7f7ae406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.831+0000 7f7b07e00640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f7ae406c8f0 0x7f7ae406edb0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7af8007520 tx=0x7f7af800d040 comp rx=0 tx=0).stop 2024-04-26T23:10:21.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.831+0000 7f7b07e00640 1 -- 172.21.15.40:0/1581960268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b080b6210 msgr2=0x7f7b080b97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:21.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.831+0000 7f7b07e00640 1 --2- 172.21.15.40:0/1581960268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b080b6210 0x7f7b080b97e0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f7af002ad60 tx=0x7f7af0001970 comp rx=0 tx=0).stop 2024-04-26T23:10:21.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.831+0000 7f7b07400640 1 -- 172.21.15.40:0/1581960268 reap_dead start 2024-04-26T23:10:21.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.831+0000 7f7b07e00640 1 -- 172.21.15.40:0/1581960268 shutdown_connections 2024-04-26T23:10:21.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.831+0000 7f7b07e00640 1 -- 172.21.15.40:0/1581960268 >> 172.21.15.40:0/1581960268 conn(0x7f7b0809e1d0 msgr2=0x7f7b0806cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:21.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.831+0000 7f7b07e00640 1 -- 172.21.15.40:0/1581960268 shutdown_connections 2024-04-26T23:10:21.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:21.831+0000 7f7b07e00640 1 -- 172.21.15.40:0/1581960268 wait complete. 2024-04-26T23:10:21.850 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-26T23:10:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.056+0000 7f2f31000640 1 Processor -- start 2024-04-26T23:10:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.056+0000 7f2f31000640 1 -- start start 2024-04-26T23:10:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.057+0000 7f2f31000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f2c15e8c0 0x7f2f2c0ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.057+0000 7f2f31000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f2c0cccc0 0x7f2f2c15ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.057+0000 7f2f31000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2f2c06e5d0 0x7f2f2c15f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.057+0000 7f2f31000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f2f2c0ce5e0 con 0x7f2f2c15e8c0 2024-04-26T23:10:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.057+0000 7f2f31000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2f2c0ced50 con 0x7f2f2c06e5d0 2024-04-26T23:10:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.057+0000 7f2f31000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2f2c0cf530 con 0x7f2f2c0cccc0 2024-04-26T23:10:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.058+0000 7f2f2b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f2c15e8c0 0x7f2f2c0ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.058+0000 7f2f2b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f2c15e8c0 0x7f2f2c0ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58598/0 (socket says 172.21.15.40:58598) 2024-04-26T23:10:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.058+0000 7f2f2b400640 1 -- 172.21.15.40:0/237027179 learned_addr learned my addr 172.21.15.40:0/237027179 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.058+0000 7f2f2be00640 1 --2- 172.21.15.40:0/237027179 >> v2:172.21.15.103:3300/0 conn(0x7f2f2c06e5d0 0x7f2f2c15f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.058+0000 7f2f2aa00640 1 --2- 172.21.15.40:0/237027179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f2c0cccc0 0x7f2f2c15ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.058+0000 7f2f2b400640 1 -- 172.21.15.40:0/237027179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f2c0cccc0 msgr2=0x7f2f2c15ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.058+0000 7f2f2b400640 1 --2- 172.21.15.40:0/237027179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f2c0cccc0 0x7f2f2c15ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.058+0000 7f2f2b400640 1 -- 172.21.15.40:0/237027179 >> v2:172.21.15.103:3300/0 conn(0x7f2f2c06e5d0 msgr2=0x7f2f2c15f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.058+0000 7f2f2b400640 1 --2- 172.21.15.40:0/237027179 >> v2:172.21.15.103:3300/0 conn(0x7f2f2c06e5d0 0x7f2f2c15f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.058+0000 7f2f2b400640 1 -- 172.21.15.40:0/237027179 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2c0c7490 con 0x7f2f2c15e8c0 2024-04-26T23:10:22.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.058+0000 7f2f2aa00640 1 --2- 172.21.15.40:0/237027179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f2c0cccc0 0x7f2f2c15ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.059+0000 7f2f2b400640 1 --2- 172.21.15.40:0/237027179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f2c15e8c0 0x7f2f2c0ceff0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0b6c00 tx=0x7f2f1802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b345f7309fdca93 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.060+0000 7f2f2a000640 1 -- 172.21.15.40:0/237027179 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f18037070 con 0x7f2f2c15e8c0 2024-04-26T23:10:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.060+0000 7f2f2a000640 1 -- 172.21.15.40:0/237027179 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f18032440 con 0x7f2f2c15e8c0 2024-04-26T23:10:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.060+0000 7f2f2a000640 1 -- 172.21.15.40:0/237027179 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f18032810 con 0x7f2f2c15e8c0 2024-04-26T23:10:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.060+0000 7f2f31000640 1 -- 172.21.15.40:0/237027179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f2c15e8c0 msgr2=0x7f2f2c0ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.060+0000 7f2f31000640 1 --2- 172.21.15.40:0/237027179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f2c15e8c0 0x7f2f2c0ceff0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0b6c00 tx=0x7f2f1802a140 comp rx=0 tx=0).stop 2024-04-26T23:10:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.060+0000 7f2f31000640 1 -- 172.21.15.40:0/237027179 shutdown_connections 2024-04-26T23:10:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.060+0000 7f2f31000640 1 --2- 172.21.15.40:0/237027179 >> v2:172.21.15.103:3300/0 conn(0x7f2f2c06e5d0 0x7f2f2c15f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.060+0000 7f2f31000640 1 --2- 172.21.15.40:0/237027179 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f2c0cccc0 0x7f2f2c15ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.060+0000 7f2f31000640 1 --2- 172.21.15.40:0/237027179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f2c15e8c0 0x7f2f2c0ceff0 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.060+0000 7f2f31000640 1 -- 172.21.15.40:0/237027179 >> 172.21.15.40:0/237027179 conn(0x7f2f2c0d0a10 msgr2=0x7f2f2c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.061+0000 7f2f31000640 1 -- 172.21.15.40:0/237027179 shutdown_connections 2024-04-26T23:10:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.061+0000 7f2f31000640 1 -- 172.21.15.40:0/237027179 wait complete. 2024-04-26T23:10:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.062+0000 7f2f31000640 1 Processor -- start 2024-04-26T23:10:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.062+0000 7f2f31000640 1 -- start start 2024-04-26T23:10:22.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.063+0000 7f2f31000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f2c06e5d0 0x7f2f2c159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.063+0000 7f2f31000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f2c0cccc0 0x7f2f2c15c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.063+0000 7f2f31000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f2c15e8c0 0x7f2f2c0d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.063+0000 7f2f31000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c15eca0 con 0x7f2f2c06e5d0 2024-04-26T23:10:22.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.063+0000 7f2f31000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c0ce830 con 0x7f2f2c0cccc0 2024-04-26T23:10:22.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.063+0000 7f2f31000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2f2c06dd40 con 0x7f2f2c15e8c0 2024-04-26T23:10:22.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.064+0000 7f2f2b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f2c06e5d0 0x7f2f2c159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.064+0000 7f2f2b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f2c06e5d0 0x7f2f2c159970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58614/0 (socket says 172.21.15.40:58614) 2024-04-26T23:10:22.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.064+0000 7f2f2b400640 1 -- 172.21.15.40:0/3009100465 learned_addr learned my addr 172.21.15.40:0/3009100465 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:22.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.064+0000 7f2f2aa00640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f2c0cccc0 0x7f2f2c15c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.064+0000 7f2f2be00640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f2c15e8c0 0x7f2f2c0d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.064+0000 7f2f2b400640 1 -- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f2c15e8c0 msgr2=0x7f2f2c0d0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.064+0000 7f2f2b400640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f2c15e8c0 0x7f2f2c0d0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.064+0000 7f2f2b400640 1 -- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f2c0cccc0 msgr2=0x7f2f2c15c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.064+0000 7f2f2b400640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f2c0cccc0 0x7f2f2c15c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.064+0000 7f2f2b400640 1 -- 172.21.15.40:0/3009100465 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2c0c7fe0 con 0x7f2f2c06e5d0 2024-04-26T23:10:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.064+0000 7f2f2aa00640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f2c0cccc0 0x7f2f2c15c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.065+0000 7f2f2b400640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f2c06e5d0 0x7f2f2c159970 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f2f18000c00 tx=0x7f2f180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.065+0000 7f2f23400640 1 -- 172.21.15.40:0/3009100465 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f18041020 con 0x7f2f2c06e5d0 2024-04-26T23:10:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.065+0000 7f2f23400640 1 -- 172.21.15.40:0/3009100465 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f180306c0 con 0x7f2f2c06e5d0 2024-04-26T23:10:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.065+0000 7f2f23400640 1 -- 172.21.15.40:0/3009100465 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f18037040 con 0x7f2f2c06e5d0 2024-04-26T23:10:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.065+0000 7f2f31000640 1 -- 172.21.15.40:0/3009100465 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2c0c82c0 con 0x7f2f2c06e5d0 2024-04-26T23:10:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.065+0000 7f2f31000640 1 -- 172.21.15.40:0/3009100465 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f2c0c8880 con 0x7f2f2c06e5d0 2024-04-26T23:10:22.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.066+0000 7f2f31000640 1 -- 172.21.15.40:0/3009100465 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f2c069370 con 0x7f2f2c06e5d0 2024-04-26T23:10:22.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.067+0000 7f2f23400640 1 -- 172.21.15.40:0/3009100465 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2f18044020 con 0x7f2f2c06e5d0 2024-04-26T23:10:22.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.068+0000 7f2f23400640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f0c06c8f0 0x7f2f0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.068+0000 7f2f23400640 1 -- 172.21.15.40:0/3009100465 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f2f180ac2b0 con 0x7f2f2c06e5d0 2024-04-26T23:10:22.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.069+0000 7f2f2aa00640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f0c06c8f0 0x7f2f0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.070+0000 7f2f2aa00640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f0c06c8f0 0x7f2f0c06edb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c006fd0 tx=0x7f2f1c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:22.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.073+0000 7f2f23400640 1 -- 172.21.15.40:0/3009100465 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2f1804b030 con 0x7f2f2c06e5d0 2024-04-26T23:10:22.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.275+0000 7f2f31000640 1 -- 172.21.15.40:0/3009100465 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f2f2c06ec10 con 0x7f2f0c06c8f0 2024-04-26T23:10:22.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.276+0000 7f2f23400640 1 -- 172.21.15.40:0/3009100465 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2f2c06ec10 con 0x7f2f0c06c8f0 2024-04-26T23:10:22.276 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 5 to deep-scrub 2024-04-26T23:10:22.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.278+0000 7f2f31000640 1 -- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f0c06c8f0 msgr2=0x7f2f0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.278+0000 7f2f31000640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f0c06c8f0 0x7f2f0c06edb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c006fd0 tx=0x7f2f1c008040 comp rx=0 tx=0).stop 2024-04-26T23:10:22.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.278+0000 7f2f31000640 1 -- 172.21.15.40:0/3009100465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f2c06e5d0 msgr2=0x7f2f2c159970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.278+0000 7f2f31000640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f2c06e5d0 0x7f2f2c159970 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f2f18000c00 tx=0x7f2f180046c0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.278+0000 7f2f31000640 1 -- 172.21.15.40:0/3009100465 shutdown_connections 2024-04-26T23:10:22.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.278+0000 7f2f31000640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2f0c06c8f0 0x7f2f0c06edb0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.278+0000 7f2f31000640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2f2c15e8c0 0x7f2f2c0d0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.278+0000 7f2f31000640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f2c0cccc0 0x7f2f2c15c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.278+0000 7f2f31000640 1 --2- 172.21.15.40:0/3009100465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f2c06e5d0 0x7f2f2c159970 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.278+0000 7f2f31000640 1 -- 172.21.15.40:0/3009100465 >> 172.21.15.40:0/3009100465 conn(0x7f2f2c0d0a10 msgr2=0x7f2f2c0b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:22.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.279+0000 7f2f31000640 1 -- 172.21.15.40:0/3009100465 shutdown_connections 2024-04-26T23:10:22.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.279+0000 7f2f31000640 1 -- 172.21.15.40:0/3009100465 wait complete. 2024-04-26T23:10:22.297 INFO:tasks.ceph:Scrubbing osd.6 2024-04-26T23:10:22.298 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-26T23:10:22.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.519+0000 7f6f12200640 1 Processor -- start 2024-04-26T23:10:22.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.519+0000 7f6f12200640 1 -- start start 2024-04-26T23:10:22.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.521+0000 7f6f12200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f0c145520 0x7f6f0c0afde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.521+0000 7f6f12200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0c0af930 con 0x7f6f0c06eba0 2024-04-26T23:10:22.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.521+0000 7f6f12200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0c072160 con 0x7f6f0c0af470 2024-04-26T23:10:22.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.521+0000 7f6f12200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0c072440 con 0x7f6f0c145520 2024-04-26T23:10:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.522+0000 7f6f0be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6f0c06eba0 0x7f6f0c162d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45930/0 (socket says 172.21.15.40:45930) 2024-04-26T23:10:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.522+0000 7f6f0be00640 1 -- 172.21.15.40:0/845056854 learned_addr learned my addr 172.21.15.40:0/845056854 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.522+0000 7f6f0b400640 1 --2- 172.21.15.40:0/845056854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f0c145520 0x7f6f0c0afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.522+0000 7f6f0aa00640 1 -- 172.21.15.40:0/845056854 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1723122832 0 0) 0x7f6f0c0af930 con 0x7f6f0c06eba0 2024-04-26T23:10:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.522+0000 7f6f0aa00640 1 -- 172.21.15.40:0/845056854 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ef0003520 con 0x7f6f0c06eba0 2024-04-26T23:10:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.523+0000 7f6f0aa00640 1 -- 172.21.15.40:0/845056854 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030472844 0 0) 0x7f6f0c072160 con 0x7f6f0c0af470 2024-04-26T23:10:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.523+0000 7f6f0aa00640 1 -- 172.21.15.40:0/845056854 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f0c0af930 con 0x7f6f0c0af470 2024-04-26T23:10:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.523+0000 7f6f0b400640 1 -- 172.21.15.40:0/845056854 >> v1:172.21.15.103:6789/0 conn(0x7f6f0c0af470 legacy=0x7f6f0c06e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.523+0000 7f6f0b400640 1 -- 172.21.15.40:0/845056854 >> v1:172.21.15.40:6789/0 conn(0x7f6f0c06eba0 legacy=0x7f6f0c162d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.523+0000 7f6f0b400640 1 -- 172.21.15.40:0/845056854 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0c148ba0 con 0x7f6f0c145520 2024-04-26T23:10:22.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.524+0000 7f6f0b400640 1 --2- 172.21.15.40:0/845056854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f0c145520 0x7f6f0c0afde0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0097d0 tx=0x7f6efc02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2d24056a7b6974d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:22.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.525+0000 7f6f0aa00640 1 -- 172.21.15.40:0/845056854 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc037070 con 0x7f6f0c145520 2024-04-26T23:10:22.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.525+0000 7f6f0aa00640 1 -- 172.21.15.40:0/845056854 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6efc032440 con 0x7f6f0c145520 2024-04-26T23:10:22.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.525+0000 7f6f0aa00640 1 -- 172.21.15.40:0/845056854 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc032810 con 0x7f6f0c145520 2024-04-26T23:10:22.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.525+0000 7f6f12200640 1 -- 172.21.15.40:0/845056854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f0c145520 msgr2=0x7f6f0c0afde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.525+0000 7f6f12200640 1 --2- 172.21.15.40:0/845056854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f0c145520 0x7f6f0c0afde0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0097d0 tx=0x7f6efc02a180 comp rx=0 tx=0).stop 2024-04-26T23:10:22.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.525+0000 7f6f12200640 1 -- 172.21.15.40:0/845056854 shutdown_connections 2024-04-26T23:10:22.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.525+0000 7f6f12200640 1 --2- 172.21.15.40:0/845056854 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f0c145520 0x7f6f0c0afde0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.525+0000 7f6f12200640 1 -- 172.21.15.40:0/845056854 >> 172.21.15.40:0/845056854 conn(0x7f6f0c06f310 msgr2=0x7f6f0c06f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:22.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.525+0000 7f6f12200640 1 -- 172.21.15.40:0/845056854 shutdown_connections 2024-04-26T23:10:22.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.525+0000 7f6f12200640 1 -- 172.21.15.40:0/845056854 wait complete. 2024-04-26T23:10:22.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.527+0000 7f6f12200640 1 Processor -- start 2024-04-26T23:10:22.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.527+0000 7f6f12200640 1 -- start start 2024-04-26T23:10:22.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.528+0000 7f6f12200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f0c06eba0 0x7f6f0c134d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.528+0000 7f6f12200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0c0af470 0x7f6f0c162390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.528+0000 7f6f12200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f0c145520 0x7f6f0c143270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.528+0000 7f6f12200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c072c40 con 0x7f6f0c145520 2024-04-26T23:10:22.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.528+0000 7f6f12200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c145b10 con 0x7f6f0c0af470 2024-04-26T23:10:22.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.528+0000 7f6f12200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0c072dc0 con 0x7f6f0c06eba0 2024-04-26T23:10:22.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.529+0000 7f6f0be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f0c145520 0x7f6f0c143270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.529+0000 7f6f0be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f0c145520 0x7f6f0c143270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58616/0 (socket says 172.21.15.40:58616) 2024-04-26T23:10:22.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.529+0000 7f6f0be00640 1 -- 172.21.15.40:0/3653641088 learned_addr learned my addr 172.21.15.40:0/3653641088 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:22.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.529+0000 7f6f0b400640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f0c06eba0 0x7f6f0c134d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.529+0000 7f6f03e00640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0c0af470 0x7f6f0c162390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.529+0000 7f6f0be00640 1 -- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f0c06eba0 msgr2=0x7f6f0c134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.529+0000 7f6f0be00640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f0c06eba0 0x7f6f0c134d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.529+0000 7f6f0be00640 1 -- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0c0af470 msgr2=0x7f6f0c162390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.529+0000 7f6f0be00640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0c0af470 0x7f6f0c162390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.529+0000 7f6f0be00640 1 -- 172.21.15.40:0/3653641088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0c0c6260 con 0x7f6f0c145520 2024-04-26T23:10:22.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.529+0000 7f6f0be00640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f0c145520 0x7f6f0c143270 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400cbc0 tx=0x7f6f0400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:22.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.530+0000 7f6f03400640 1 -- 172.21.15.40:0/3653641088 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04017070 con 0x7f6f0c145520 2024-04-26T23:10:22.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.530+0000 7f6f03400640 1 -- 172.21.15.40:0/3653641088 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f0400ace0 con 0x7f6f0c145520 2024-04-26T23:10:22.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.530+0000 7f6f03400640 1 -- 172.21.15.40:0/3653641088 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04012600 con 0x7f6f0c145520 2024-04-26T23:10:22.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.530+0000 7f6f12200640 1 -- 172.21.15.40:0/3653641088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f0c0c6540 con 0x7f6f0c145520 2024-04-26T23:10:22.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.530+0000 7f6f12200640 1 -- 172.21.15.40:0/3653641088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f0c0c6b00 con 0x7f6f0c145520 2024-04-26T23:10:22.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.531+0000 7f6f12200640 1 -- 172.21.15.40:0/3653641088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6f0c069600 con 0x7f6f0c145520 2024-04-26T23:10:22.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.532+0000 7f6f03400640 1 -- 172.21.15.40:0/3653641088 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6f0401a070 con 0x7f6f0c145520 2024-04-26T23:10:22.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.533+0000 7f6f03400640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6ee806c8f0 0x7f6ee806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.533+0000 7f6f03400640 1 -- 172.21.15.40:0/3653641088 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f6f0408bff0 con 0x7f6f0c145520 2024-04-26T23:10:22.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.534+0000 7f6f03400640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x7f6ee8072ed0 0x7f6ee8075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.534+0000 7f6f03400640 1 -- 172.21.15.40:0/3653641088 --> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ee80759e0 con 0x7f6ee8072ed0 2024-04-26T23:10:22.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.534+0000 7f6f03400640 1 -- 172.21.15.40:0/3653641088 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f6f0408c3e0 con 0x7f6f0c145520 2024-04-26T23:10:22.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.534+0000 7f6f0b400640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6ee806c8f0 0x7f6ee806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.534+0000 7f6f03e00640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x7f6ee8072ed0 0x7f6ee8075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.535+0000 7f6f0b400640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6ee806c8f0 0x7f6ee806edb0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6efc034000 tx=0x7f6efc02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:22.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.535+0000 7f6f03e00640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x7f6ee8072ed0 0x7f6ee8075310 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:22.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.537+0000 7f6f03400640 1 -- 172.21.15.40:0/3653641088 <== osd.6 v2:172.21.15.103:6816/293480649 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ee80759e0 con 0x7f6ee8072ed0 2024-04-26T23:10:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.553+0000 7f6f12200640 1 -- 172.21.15.40:0/3653641088 --> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6f0c1352c0 con 0x7f6ee8072ed0 2024-04-26T23:10:22.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.554+0000 7f6f03400640 1 -- 172.21.15.40:0/3653641088 <== osd.6 v2:172.21.15.103:6816/293480649 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6f0c1352c0 con 0x7f6ee8072ed0 2024-04-26T23:10:22.554 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:10:22.554 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T23:10:22.554 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:10:22.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.555+0000 7f6f12200640 1 -- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x7f6ee8072ed0 msgr2=0x7f6ee8075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.555+0000 7f6f12200640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:6816/293480649,v1:172.21.15.103:6817/293480649] conn(0x7f6ee8072ed0 0x7f6ee8075310 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.555+0000 7f6f12200640 1 -- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6ee806c8f0 msgr2=0x7f6ee806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.555+0000 7f6f12200640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6ee806c8f0 0x7f6ee806edb0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6efc034000 tx=0x7f6efc02e000 comp rx=0 tx=0).stop 2024-04-26T23:10:22.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.555+0000 7f6f12200640 1 -- 172.21.15.40:0/3653641088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f0c145520 msgr2=0x7f6f0c143270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.555+0000 7f6f12200640 1 --2- 172.21.15.40:0/3653641088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f0c145520 0x7f6f0c143270 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400cbc0 tx=0x7f6f0400a430 comp rx=0 tx=0).stop 2024-04-26T23:10:22.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.555+0000 7f6f0be00640 1 -- 172.21.15.40:0/3653641088 reap_dead start 2024-04-26T23:10:22.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.555+0000 7f6f12200640 1 -- 172.21.15.40:0/3653641088 shutdown_connections 2024-04-26T23:10:22.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.555+0000 7f6f12200640 1 -- 172.21.15.40:0/3653641088 >> 172.21.15.40:0/3653641088 conn(0x7f6f0c06f310 msgr2=0x7f6f0c06e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:22.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.555+0000 7f6f12200640 1 -- 172.21.15.40:0/3653641088 shutdown_connections 2024-04-26T23:10:22.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.555+0000 7f6f12200640 1 -- 172.21.15.40:0/3653641088 wait complete. 2024-04-26T23:10:22.574 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-26T23:10:22.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.786+0000 7f6dd5c00640 1 Processor -- start 2024-04-26T23:10:22.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.787+0000 7f6dd5c00640 1 -- start start 2024-04-26T23:10:22.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.788+0000 7f6dd5c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dd00c6f10 0x7f6dd00c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.788+0000 7f6dd5c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6dd00c6010 0x7f6dd0158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.788+0000 7f6dd5c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dd00cce10 con 0x7f6dd00cc980 2024-04-26T23:10:22.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.788+0000 7f6dd5c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f6dd00c72f0 con 0x7f6dd00c6010 2024-04-26T23:10:22.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.788+0000 7f6dd5c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6dd00cb860 con 0x7f6dd00c6f10 2024-04-26T23:10:22.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.789+0000 7f6dd5200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6dd00c6010 0x7f6dd0158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.789+0000 7f6dd5200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6dd00c6010 0x7f6dd0158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58628/0 (socket says 172.21.15.40:58628) 2024-04-26T23:10:22.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.789+0000 7f6dd5200640 1 -- 172.21.15.40:0/3003162323 learned_addr learned my addr 172.21.15.40:0/3003162323 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:22.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.789+0000 7f6dcfe00640 1 --2- 172.21.15.40:0/3003162323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dd00c6f10 0x7f6dd00c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.789+0000 7f6dd5200640 1 -- 172.21.15.40:0/3003162323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dd00c6f10 msgr2=0x7f6dd00c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:22.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.789+0000 7f6dd5200640 1 --2- 172.21.15.40:0/3003162323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dd00c6f10 0x7f6dd00c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.789+0000 7f6dd5200640 1 -- 172.21.15.40:0/3003162323 >> v1:172.21.15.103:6789/0 conn(0x7f6dd00cc980 legacy=0x7f6dd00cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:22.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.789+0000 7f6dd5200640 1 -- 172.21.15.40:0/3003162323 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd00cd0f0 con 0x7f6dd00c6010 2024-04-26T23:10:22.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.790+0000 7f6dd5200640 1 --2- 172.21.15.40:0/3003162323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd00c6010 0x7f6dd0158cd0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8009870 tx=0x7f6dc802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87fa511de5e023b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:22.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.791+0000 7f6dcf400640 1 -- 172.21.15.40:0/3003162323 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8037070 con 0x7f6dd00c6010 2024-04-26T23:10:22.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.791+0000 7f6dcf400640 1 -- 172.21.15.40:0/3003162323 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dc8032440 con 0x7f6dd00c6010 2024-04-26T23:10:22.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.791+0000 7f6dcf400640 1 -- 172.21.15.40:0/3003162323 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8032810 con 0x7f6dd00c6010 2024-04-26T23:10:22.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.791+0000 7f6dd5c00640 1 -- 172.21.15.40:0/3003162323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd00c6010 msgr2=0x7f6dd0158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.791+0000 7f6dd5c00640 1 --2- 172.21.15.40:0/3003162323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd00c6010 0x7f6dd0158cd0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8009870 tx=0x7f6dc802a180 comp rx=0 tx=0).stop 2024-04-26T23:10:22.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.791+0000 7f6dd5c00640 1 -- 172.21.15.40:0/3003162323 shutdown_connections 2024-04-26T23:10:22.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.791+0000 7f6dd5c00640 1 --2- 172.21.15.40:0/3003162323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd00c6010 0x7f6dd0158cd0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.791+0000 7f6dd5c00640 1 --2- 172.21.15.40:0/3003162323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dd00c6f10 0x7f6dd00c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.791+0000 7f6dd5c00640 1 -- 172.21.15.40:0/3003162323 >> 172.21.15.40:0/3003162323 conn(0x7f6dd00d0a10 msgr2=0x7f6dd00d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:22.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.791+0000 7f6dd5c00640 1 -- 172.21.15.40:0/3003162323 shutdown_connections 2024-04-26T23:10:22.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.791+0000 7f6dd5c00640 1 -- 172.21.15.40:0/3003162323 wait complete. 2024-04-26T23:10:22.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.793+0000 7f6dd5c00640 1 Processor -- start 2024-04-26T23:10:22.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.793+0000 7f6dd5c00640 1 -- start start 2024-04-26T23:10:22.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.794+0000 7f6dd5c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd00c6010 0x7f6dd0153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.794+0000 7f6dd5c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd00c6f10 0x7f6dd00b5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.794+0000 7f6dd5c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dd00c84f0 0x7f6dd015c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.794+0000 7f6dd5c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd00cc400 con 0x7f6dd00c6f10 2024-04-26T23:10:22.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.794+0000 7f6dd5c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd00cb1a0 con 0x7f6dd00c6010 2024-04-26T23:10:22.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.794+0000 7f6dd5c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6dd00cc580 con 0x7f6dd00c84f0 2024-04-26T23:10:22.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.794+0000 7f6dc7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd00c6f10 0x7f6dd00b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.794+0000 7f6dc7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd00c6f10 0x7f6dd00b5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58634/0 (socket says 172.21.15.40:58634) 2024-04-26T23:10:22.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.794+0000 7f6dc7400640 1 -- 172.21.15.40:0/2401630234 learned_addr learned my addr 172.21.15.40:0/2401630234 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.795+0000 7f6dd5200640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dd00c84f0 0x7f6dd015c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.795+0000 7f6dc7400640 1 -- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dd00c84f0 msgr2=0x7f6dd015c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.795+0000 7f6dcfe00640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd00c6010 0x7f6dd0153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.795+0000 7f6dc7400640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dd00c84f0 0x7f6dd015c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.795+0000 7f6dc7400640 1 -- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd00c6010 msgr2=0x7f6dd0153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.795+0000 7f6dc7400640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd00c6010 0x7f6dd0153ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.795+0000 7f6dc7400640 1 -- 172.21.15.40:0/2401630234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd00c8980 con 0x7f6dd00c6f10 2024-04-26T23:10:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.795+0000 7f6dc7400640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd00c6f10 0x7f6dd00b5330 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f6db800c430 tx=0x7f6db800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.795+0000 7f6dccc00640 1 -- 172.21.15.40:0/2401630234 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db8011790 con 0x7f6dd00c6f10 2024-04-26T23:10:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.795+0000 7f6dccc00640 1 -- 172.21.15.40:0/2401630234 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db8017070 con 0x7f6dd00c6f10 2024-04-26T23:10:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.795+0000 7f6dd5c00640 1 -- 172.21.15.40:0/2401630234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dd00c8c60 con 0x7f6dd00c6f10 2024-04-26T23:10:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.796+0000 7f6dccc00640 1 -- 172.21.15.40:0/2401630234 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db8011bb0 con 0x7f6dd00c6f10 2024-04-26T23:10:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.796+0000 7f6dd5c00640 1 -- 172.21.15.40:0/2401630234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dd00c9220 con 0x7f6dd00c6f10 2024-04-26T23:10:22.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.797+0000 7f6dd5c00640 1 -- 172.21.15.40:0/2401630234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dd00c9600 con 0x7f6dd00c6f10 2024-04-26T23:10:22.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.798+0000 7f6dccc00640 1 -- 172.21.15.40:0/2401630234 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6db801c460 con 0x7f6dd00c6f10 2024-04-26T23:10:22.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.799+0000 7f6dccc00640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6dac06c8f0 0x7f6dac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:22.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.799+0000 7f6dccc00640 1 -- 172.21.15.40:0/2401630234 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f6db808ba00 con 0x7f6dd00c6f10 2024-04-26T23:10:22.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.799+0000 7f6dcfe00640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6dac06c8f0 0x7f6dac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:22.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.801+0000 7f6dcfe00640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6dac06c8f0 0x7f6dac06edb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00c6900 tx=0x7f6dc000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:22.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:22.804+0000 7f6dccc00640 1 -- 172.21.15.40:0/2401630234 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f6dd00c9600 con 0x7f6dd00c6f10 2024-04-26T23:10:23.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.004+0000 7f6dd5c00640 1 -- 172.21.15.40:0/2401630234 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f6dd0069370 con 0x7f6dac06c8f0 2024-04-26T23:10:23.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.006+0000 7f6dccc00640 1 -- 172.21.15.40:0/2401630234 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6dd0069370 con 0x7f6dac06c8f0 2024-04-26T23:10:23.006 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 6 to deep-scrub 2024-04-26T23:10:23.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.008+0000 7f6dd5c00640 1 -- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6dac06c8f0 msgr2=0x7f6dac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.008+0000 7f6dd5c00640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6dac06c8f0 0x7f6dac06edb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00c6900 tx=0x7f6dc000b040 comp rx=0 tx=0).stop 2024-04-26T23:10:23.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.008+0000 7f6dd5c00640 1 -- 172.21.15.40:0/2401630234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd00c6f10 msgr2=0x7f6dd00b5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.008+0000 7f6dd5c00640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd00c6f10 0x7f6dd00b5330 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f6db800c430 tx=0x7f6db800c8f0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.009+0000 7f6dd5c00640 1 -- 172.21.15.40:0/2401630234 shutdown_connections 2024-04-26T23:10:23.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.009+0000 7f6dd5c00640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f6dac06c8f0 0x7f6dac06edb0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.009+0000 7f6dd5c00640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6dd00c84f0 0x7f6dd015c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.009+0000 7f6dd5c00640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd00c6f10 0x7f6dd00b5330 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.009+0000 7f6dd5c00640 1 --2- 172.21.15.40:0/2401630234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd00c6010 0x7f6dd0153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.009+0000 7f6dd5c00640 1 -- 172.21.15.40:0/2401630234 >> 172.21.15.40:0/2401630234 conn(0x7f6dd00d0a10 msgr2=0x7f6dd006e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:23.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.009+0000 7f6dd5c00640 1 -- 172.21.15.40:0/2401630234 shutdown_connections 2024-04-26T23:10:23.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.009+0000 7f6dd5c00640 1 -- 172.21.15.40:0/2401630234 wait complete. 2024-04-26T23:10:23.027 INFO:tasks.ceph:Scrubbing osd.7 2024-04-26T23:10:23.027 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-26T23:10:23.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.241+0000 7fb359c00640 1 Processor -- start 2024-04-26T23:10:23.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.242+0000 7fb359c00640 1 -- start start 2024-04-26T23:10:23.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.243+0000 7fb359c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb35415f420 0x7fb3540d0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.243+0000 7fb359c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3540b3d90 0x7fb3540d3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.243+0000 7fb359c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3540d0d50 con 0x7fb3540d3990 2024-04-26T23:10:23.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.243+0000 7fb359c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fb3540d0a70 con 0x7fb3540b3d90 2024-04-26T23:10:23.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.243+0000 7fb359c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb3540b1bf0 con 0x7fb35415f420 2024-04-26T23:10:23.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.243+0000 7fb353400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3540b3d90 0x7fb3540d3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.244+0000 7fb353400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3540b3d90 0x7fb3540d3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58636/0 (socket says 172.21.15.40:58636) 2024-04-26T23:10:23.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.244+0000 7fb353400640 1 -- 172.21.15.40:0/3452822030 learned_addr learned my addr 172.21.15.40:0/3452822030 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:23.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.244+0000 7fb353e00640 1 --2- 172.21.15.40:0/3452822030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb35415f420 0x7fb3540d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.244+0000 7fb353400640 1 -- 172.21.15.40:0/3452822030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb35415f420 msgr2=0x7fb3540d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:23.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.244+0000 7fb353400640 1 --2- 172.21.15.40:0/3452822030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb35415f420 0x7fb3540d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.244+0000 7fb353400640 1 -- 172.21.15.40:0/3452822030 >> v1:172.21.15.40:6789/0 conn(0x7fb3540d3990 legacy=0x7fb3540d14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.244+0000 7fb353400640 1 -- 172.21.15.40:0/3452822030 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3540b1ed0 con 0x7fb3540b3d90 2024-04-26T23:10:23.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.245+0000 7fb353400640 1 --2- 172.21.15.40:0/3452822030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3540b3d90 0x7fb3540d3450 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fb33c009e50 tx=0x7fb33c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf5a439680b9583 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:23.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.246+0000 7fb352a00640 1 -- 172.21.15.40:0/3452822030 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33c036070 con 0x7fb3540b3d90 2024-04-26T23:10:23.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.246+0000 7fb352a00640 1 -- 172.21.15.40:0/3452822030 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb33c031600 con 0x7fb3540b3d90 2024-04-26T23:10:23.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.246+0000 7fb352a00640 1 -- 172.21.15.40:0/3452822030 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33c0319d0 con 0x7fb3540b3d90 2024-04-26T23:10:23.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.246+0000 7fb359c00640 1 -- 172.21.15.40:0/3452822030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3540b3d90 msgr2=0x7fb3540d3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.246+0000 7fb359c00640 1 --2- 172.21.15.40:0/3452822030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3540b3d90 0x7fb3540d3450 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fb33c009e50 tx=0x7fb33c02a680 comp rx=0 tx=0).stop 2024-04-26T23:10:23.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.246+0000 7fb359c00640 1 -- 172.21.15.40:0/3452822030 shutdown_connections 2024-04-26T23:10:23.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.246+0000 7fb359c00640 1 --2- 172.21.15.40:0/3452822030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3540b3d90 0x7fb3540d3450 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.246+0000 7fb359c00640 1 --2- 172.21.15.40:0/3452822030 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb35415f420 0x7fb3540d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.246+0000 7fb359c00640 1 -- 172.21.15.40:0/3452822030 >> 172.21.15.40:0/3452822030 conn(0x7fb35409e1d0 msgr2=0x7fb3540d4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:23.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.246+0000 7fb359c00640 1 -- 172.21.15.40:0/3452822030 shutdown_connections 2024-04-26T23:10:23.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.246+0000 7fb359c00640 1 -- 172.21.15.40:0/3452822030 wait complete. 2024-04-26T23:10:23.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.248+0000 7fb359c00640 1 Processor -- start 2024-04-26T23:10:23.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.248+0000 7fb359c00640 1 -- start start 2024-04-26T23:10:23.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.249+0000 7fb359c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3540b3d90 0x7fb3540b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.249+0000 7fb359c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3540d3990 0x7fb3540b91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.249+0000 7fb359c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35415f420 0x7fb3540cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.249+0000 7fb359c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb3540b36e0 con 0x7fb35415f420 2024-04-26T23:10:23.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.249+0000 7fb359c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb3540d2e50 con 0x7fb3540d3990 2024-04-26T23:10:23.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.249+0000 7fb359c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb3540b3860 con 0x7fb3540b3d90 2024-04-26T23:10:23.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.250+0000 7fb359200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35415f420 0x7fb3540cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.250+0000 7fb359200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35415f420 0x7fb3540cad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58644/0 (socket says 172.21.15.40:58644) 2024-04-26T23:10:23.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.250+0000 7fb359200640 1 -- 172.21.15.40:0/2514749371 learned_addr learned my addr 172.21.15.40:0/2514749371 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:23.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.250+0000 7fb353400640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3540d3990 0x7fb3540b91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.250+0000 7fb353e00640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3540b3d90 0x7fb3540b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.250+0000 7fb359200640 1 -- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3540b3d90 msgr2=0x7fb3540b7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.250+0000 7fb359200640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3540b3d90 0x7fb3540b7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.250+0000 7fb359200640 1 -- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3540d3990 msgr2=0x7fb3540b91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.250+0000 7fb359200640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3540d3990 0x7fb3540b91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.250+0000 7fb359200640 1 -- 172.21.15.40:0/2514749371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3540b8110 con 0x7fb35415f420 2024-04-26T23:10:23.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.250+0000 7fb359200640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35415f420 0x7fb3540cad20 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c390 tx=0x7fb34c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:23.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.251+0000 7fb34be00640 1 -- 172.21.15.40:0/2514749371 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c017070 con 0x7fb35415f420 2024-04-26T23:10:23.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.251+0000 7fb34be00640 1 -- 172.21.15.40:0/2514749371 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb34c0052d0 con 0x7fb35415f420 2024-04-26T23:10:23.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.251+0000 7fb34be00640 1 -- 172.21.15.40:0/2514749371 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c003810 con 0x7fb35415f420 2024-04-26T23:10:23.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.251+0000 7fb359c00640 1 -- 172.21.15.40:0/2514749371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3540b83f0 con 0x7fb35415f420 2024-04-26T23:10:23.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.251+0000 7fb359c00640 1 -- 172.21.15.40:0/2514749371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3540b89b0 con 0x7fb35415f420 2024-04-26T23:10:23.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.252+0000 7fb359c00640 1 -- 172.21.15.40:0/2514749371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb354069370 con 0x7fb35415f420 2024-04-26T23:10:23.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.253+0000 7fb34be00640 1 -- 172.21.15.40:0/2514749371 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb34c01a070 con 0x7fb35415f420 2024-04-26T23:10:23.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.254+0000 7fb34be00640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb33006c8f0 0x7fb33006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.254+0000 7fb34be00640 1 -- 172.21.15.40:0/2514749371 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7fb34c08ae50 con 0x7fb35415f420 2024-04-26T23:10:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.254+0000 7fb34be00640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x7fb330072ed0 0x7fb330075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.254+0000 7fb34be00640 1 -- 172.21.15.40:0/2514749371 --> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3300759e0 con 0x7fb330072ed0 2024-04-26T23:10:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.254+0000 7fb34be00640 1 -- 172.21.15.40:0/2514749371 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fb34c08d470 con 0x7fb35415f420 2024-04-26T23:10:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.255+0000 7fb353e00640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb33006c8f0 0x7fb33006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.255+0000 7fb353400640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x7fb330072ed0 0x7fb330075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.257+0000 7fb353e00640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb33006c8f0 0x7fb33006edb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb344006fd0 tx=0x7fb344008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:23.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.257+0000 7fb353400640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x7fb330072ed0 0x7fb330075310 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:23.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.259+0000 7fb34be00640 1 -- 172.21.15.40:0/2514749371 <== osd.7 v2:172.21.15.103:6819/3769540391 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3300759e0 con 0x7fb330072ed0 2024-04-26T23:10:23.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.273+0000 7fb359c00640 1 -- 172.21.15.40:0/2514749371 --> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb3540d1880 con 0x7fb330072ed0 2024-04-26T23:10:23.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.274+0000 7fb34be00640 1 -- 172.21.15.40:0/2514749371 <== osd.7 v2:172.21.15.103:6819/3769540391 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb3540d1880 con 0x7fb330072ed0 2024-04-26T23:10:23.275 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:10:23.275 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T23:10:23.275 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:10:23.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.275+0000 7fb359c00640 1 -- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x7fb330072ed0 msgr2=0x7fb330075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.275+0000 7fb359c00640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:6819/3769540391,v1:172.21.15.103:6821/3769540391] conn(0x7fb330072ed0 0x7fb330075310 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.275+0000 7fb359c00640 1 -- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb33006c8f0 msgr2=0x7fb33006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.276+0000 7fb359c00640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fb33006c8f0 0x7fb33006edb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb344006fd0 tx=0x7fb344008040 comp rx=0 tx=0).stop 2024-04-26T23:10:23.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.276+0000 7fb359c00640 1 -- 172.21.15.40:0/2514749371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35415f420 msgr2=0x7fb3540cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.276+0000 7fb359c00640 1 --2- 172.21.15.40:0/2514749371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb35415f420 0x7fb3540cad20 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c390 tx=0x7fb34c00c850 comp rx=0 tx=0).stop 2024-04-26T23:10:23.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.276+0000 7fb359200640 1 -- 172.21.15.40:0/2514749371 reap_dead start 2024-04-26T23:10:23.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.276+0000 7fb359c00640 1 -- 172.21.15.40:0/2514749371 shutdown_connections 2024-04-26T23:10:23.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.276+0000 7fb359c00640 1 -- 172.21.15.40:0/2514749371 >> 172.21.15.40:0/2514749371 conn(0x7fb35409e1d0 msgr2=0x7fb3540d1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:23.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.276+0000 7fb359c00640 1 -- 172.21.15.40:0/2514749371 shutdown_connections 2024-04-26T23:10:23.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.276+0000 7fb359c00640 1 -- 172.21.15.40:0/2514749371 wait complete. 2024-04-26T23:10:23.296 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-26T23:10:23.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.498+0000 7fc88d600640 1 Processor -- start 2024-04-26T23:10:23.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.498+0000 7fc88d600640 1 -- start start 2024-04-26T23:10:23.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.500+0000 7fc88d600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc88815ce60 0x7fc8880afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.500+0000 7fc88d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc888071fe0 0x7fc888073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.500+0000 7fc88d600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8880d0820 con 0x7fc8880af7a0 2024-04-26T23:10:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.500+0000 7fc88d600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7fc8880cc450 con 0x7fc88815ce60 2024-04-26T23:10:23.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.500+0000 7fc88d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc8881614c0 con 0x7fc888071fe0 2024-04-26T23:10:23.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.501+0000 7fc887e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc88815ce60 0x7fc8880afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.501+0000 7fc887e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc88815ce60 0x7fc8880afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58656/0 (socket says 172.21.15.40:58656) 2024-04-26T23:10:23.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.501+0000 7fc887e00640 1 -- 172.21.15.40:0/2991330155 learned_addr learned my addr 172.21.15.40:0/2991330155 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:23.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.501+0000 7fc88cc00640 1 --2- 172.21.15.40:0/2991330155 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc888071fe0 0x7fc888073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.502+0000 7fc887e00640 1 -- 172.21.15.40:0/2991330155 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc888071fe0 msgr2=0x7fc888073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:23.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.502+0000 7fc887e00640 1 --2- 172.21.15.40:0/2991330155 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc888071fe0 0x7fc888073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.502+0000 7fc887e00640 1 -- 172.21.15.40:0/2991330155 >> v1:172.21.15.103:6789/0 conn(0x7fc8880af7a0 legacy=0x7fc8880ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:23.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.502+0000 7fc887e00640 1 -- 172.21.15.40:0/2991330155 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc888161760 con 0x7fc88815ce60 2024-04-26T23:10:23.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.503+0000 7fc887e00640 1 --2- 172.21.15.40:0/2991330155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc88815ce60 0x7fc8880afc60 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fc878009870 tx=0x7fc87802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33657e9fcb5e2d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:23.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.503+0000 7fc887400640 1 -- 172.21.15.40:0/2991330155 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc878037070 con 0x7fc88815ce60 2024-04-26T23:10:23.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.503+0000 7fc887400640 1 -- 172.21.15.40:0/2991330155 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc878032440 con 0x7fc88815ce60 2024-04-26T23:10:23.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.503+0000 7fc887400640 1 -- 172.21.15.40:0/2991330155 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc878032810 con 0x7fc88815ce60 2024-04-26T23:10:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.504+0000 7fc88d600640 1 -- 172.21.15.40:0/2991330155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc88815ce60 msgr2=0x7fc8880afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.504+0000 7fc88d600640 1 --2- 172.21.15.40:0/2991330155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc88815ce60 0x7fc8880afc60 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fc878009870 tx=0x7fc87802a180 comp rx=0 tx=0).stop 2024-04-26T23:10:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.504+0000 7fc88d600640 1 -- 172.21.15.40:0/2991330155 shutdown_connections 2024-04-26T23:10:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.504+0000 7fc88d600640 1 --2- 172.21.15.40:0/2991330155 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc888071fe0 0x7fc888073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.504+0000 7fc88d600640 1 --2- 172.21.15.40:0/2991330155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc88815ce60 0x7fc8880afc60 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.504+0000 7fc88d600640 1 -- 172.21.15.40:0/2991330155 >> 172.21.15.40:0/2991330155 conn(0x7fc8880d0a10 msgr2=0x7fc8880d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.504+0000 7fc88d600640 1 -- 172.21.15.40:0/2991330155 shutdown_connections 2024-04-26T23:10:23.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.504+0000 7fc88d600640 1 -- 172.21.15.40:0/2991330155 wait complete. 2024-04-26T23:10:23.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.506+0000 7fc88d600640 1 Processor -- start 2024-04-26T23:10:23.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.506+0000 7fc88d600640 1 -- start start 2024-04-26T23:10:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc88d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888071fe0 0x7fc88814dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc88d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8880af7a0 0x7fc8881f6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc88d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc88815ce60 0x7fc8881f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc88d600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc8880cc1f0 con 0x7fc888071fe0 2024-04-26T23:10:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc88d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc8880cc070 con 0x7fc88815ce60 2024-04-26T23:10:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc88d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc888073ca0 con 0x7fc8880af7a0 2024-04-26T23:10:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc887e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888071fe0 0x7fc88814dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc887e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888071fe0 0x7fc88814dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58670/0 (socket says 172.21.15.40:58670) 2024-04-26T23:10:23.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc887e00640 1 -- 172.21.15.40:0/2730002397 learned_addr learned my addr 172.21.15.40:0/2730002397 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:23.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc887e00640 1 -- 172.21.15.40:0/2730002397 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8880af7a0 msgr2=0x7fc8881f6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc887e00640 1 --2- 172.21.15.40:0/2730002397 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8880af7a0 0x7fc8881f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc887e00640 1 -- 172.21.15.40:0/2730002397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc88815ce60 msgr2=0x7fc8881f6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc887e00640 1 --2- 172.21.15.40:0/2730002397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc88815ce60 0x7fc8881f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.508+0000 7fc887e00640 1 -- 172.21.15.40:0/2730002397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8880b41e0 con 0x7fc888071fe0 2024-04-26T23:10:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.509+0000 7fc887e00640 1 --2- 172.21.15.40:0/2730002397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888071fe0 0x7fc88814dc20 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fc8780097c0 tx=0x7fc8780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.509+0000 7fc884c00640 1 -- 172.21.15.40:0/2730002397 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc878041020 con 0x7fc888071fe0 2024-04-26T23:10:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.509+0000 7fc884c00640 1 -- 172.21.15.40:0/2730002397 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc87803d620 con 0x7fc888071fe0 2024-04-26T23:10:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.509+0000 7fc88d600640 1 -- 172.21.15.40:0/2730002397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8880b44c0 con 0x7fc888071fe0 2024-04-26T23:10:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.509+0000 7fc884c00640 1 -- 172.21.15.40:0/2730002397 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc878037040 con 0x7fc888071fe0 2024-04-26T23:10:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.509+0000 7fc88d600640 1 -- 172.21.15.40:0/2730002397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8880b4fa0 con 0x7fc888071fe0 2024-04-26T23:10:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.510+0000 7fc88d600640 1 -- 172.21.15.40:0/2730002397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc888069370 con 0x7fc888071fe0 2024-04-26T23:10:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.511+0000 7fc884c00640 1 -- 172.21.15.40:0/2730002397 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc878044020 con 0x7fc888071fe0 2024-04-26T23:10:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.512+0000 7fc884c00640 1 --2- 172.21.15.40:0/2730002397 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc86406c8f0 0x7fc86406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.512+0000 7fc884c00640 1 -- 172.21.15.40:0/2730002397 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7fc8780ac190 con 0x7fc888071fe0 2024-04-26T23:10:23.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.513+0000 7fc87fe00640 1 --2- 172.21.15.40:0/2730002397 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc86406c8f0 0x7fc86406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.514+0000 7fc87fe00640 1 --2- 172.21.15.40:0/2730002397 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc86406c8f0 0x7fc86406edb0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc8880b5da0 tx=0x7fc8700079b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.517+0000 7fc884c00640 1 -- 172.21.15.40:0/2730002397 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7fc87804b030 con 0x7fc888071fe0 2024-04-26T23:10:23.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.721+0000 7fc88d600640 1 -- 172.21.15.40:0/2730002397 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fc8880d01f0 con 0x7fc86406c8f0 2024-04-26T23:10:23.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.723+0000 7fc884c00640 1 -- 172.21.15.40:0/2730002397 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc8880d01f0 con 0x7fc86406c8f0 2024-04-26T23:10:23.723 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 7 to deep-scrub 2024-04-26T23:10:23.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.725+0000 7fc88d600640 1 -- 172.21.15.40:0/2730002397 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc86406c8f0 msgr2=0x7fc86406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.725+0000 7fc88d600640 1 --2- 172.21.15.40:0/2730002397 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc86406c8f0 0x7fc86406edb0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc8880b5da0 tx=0x7fc8700079b0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.725+0000 7fc88d600640 1 -- 172.21.15.40:0/2730002397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888071fe0 msgr2=0x7fc88814dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.725+0000 7fc88d600640 1 --2- 172.21.15.40:0/2730002397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888071fe0 0x7fc88814dc20 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fc8780097c0 tx=0x7fc8780046c0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.725+0000 7fc88d600640 1 -- 172.21.15.40:0/2730002397 shutdown_connections 2024-04-26T23:10:23.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.725+0000 7fc88d600640 1 --2- 172.21.15.40:0/2730002397 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7fc86406c8f0 0x7fc86406edb0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.725+0000 7fc88d600640 1 --2- 172.21.15.40:0/2730002397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc88815ce60 0x7fc8881f6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.725+0000 7fc88d600640 1 --2- 172.21.15.40:0/2730002397 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8880af7a0 0x7fc8881f6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.725+0000 7fc88d600640 1 --2- 172.21.15.40:0/2730002397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc888071fe0 0x7fc88814dc20 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.725+0000 7fc88d600640 1 -- 172.21.15.40:0/2730002397 >> 172.21.15.40:0/2730002397 conn(0x7fc8880d0a10 msgr2=0x7fc888157930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:23.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.726+0000 7fc88d600640 1 -- 172.21.15.40:0/2730002397 shutdown_connections 2024-04-26T23:10:23.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.726+0000 7fc88d600640 1 -- 172.21.15.40:0/2730002397 wait complete. 2024-04-26T23:10:23.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:10:23.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.966+0000 7f2c11800640 1 Processor -- start 2024-04-26T23:10:23.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.966+0000 7f2c11800640 1 -- start start 2024-04-26T23:10:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.967+0000 7f2c11800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c0c15f380 0x7f2c0c0cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.967+0000 7f2c11800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c0c0cfc00 con 0x7f2c0c073b70 2024-04-26T23:10:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.967+0000 7f2c11800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c0c072c00 con 0x7f2c0c0d0620 2024-04-26T23:10:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.967+0000 7f2c11800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2c0c0729b0 con 0x7f2c0c15f380 2024-04-26T23:10:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.968+0000 7f2c03e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2c0c073b70 0x7f2c0c0cede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45950/0 (socket says 172.21.15.40:45950) 2024-04-26T23:10:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.968+0000 7f2c03e00640 1 -- 172.21.15.40:0/352740419 learned_addr learned my addr 172.21.15.40:0/352740419 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.968+0000 7f2c0aa00640 1 --2- 172.21.15.40:0/352740419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c0c15f380 0x7f2c0c0cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.968+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 802610299 0 0) 0x7f2c0c0cfc00 con 0x7f2c0c073b70 2024-04-26T23:10:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.968+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bf0003520 con 0x7f2c0c073b70 2024-04-26T23:10:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2984152799 0 0) 0x7f2bf0003520 con 0x7f2c0c073b70 2024-04-26T23:10:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c0c15f380 msgr2=0x7f2c0c0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0a000640 1 --2- 172.21.15.40:0/352740419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c0c15f380 0x7f2c0c0cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 >> v1:172.21.15.103:6789/0 conn(0x7f2c0c0d0620 legacy=0x7f2c0c0cf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c0c072ee0 con 0x7f2c0c073b70 2024-04-26T23:10:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1434322080 0 0) 0x7f2bf4001670 con 0x7f2c0c073b70 2024-04-26T23:10:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 >> v1:172.21.15.40:6789/0 conn(0x7f2c0c073b70 legacy=0x7f2c0c0cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0a000640 1 --2- 172.21.15.40:0/352740419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c0c15f380 0x7f2bf00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0a000640 1 --2- 172.21.15.40:0/352740419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bf0003fb0 0x7f2bf0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0a000640 1 --2- 172.21.15.40:0/352740419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bf0004d40 0x7f2bf00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c0729b0 con 0x7f2bf0004d40 2024-04-26T23:10:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf000bb00 con 0x7f2bf0003fb0 2024-04-26T23:10:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2bf000bd00 con 0x7f2c0c15f380 2024-04-26T23:10:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.969+0000 7f2c0b400640 1 --2- 172.21.15.40:0/352740419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bf0004d40 0x7f2bf00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.970+0000 7f2c03e00640 1 --2- 172.21.15.40:0/352740419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bf0003fb0 0x7f2bf0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.970+0000 7f2c0aa00640 1 --2- 172.21.15.40:0/352740419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c0c15f380 0x7f2bf00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.970+0000 7f2c0b400640 1 -- 172.21.15.40:0/352740419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c0c15f380 msgr2=0x7f2bf00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.970+0000 7f2c0b400640 1 --2- 172.21.15.40:0/352740419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c0c15f380 0x7f2bf00039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.970+0000 7f2c0b400640 1 -- 172.21.15.40:0/352740419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bf0003fb0 msgr2=0x7f2bf0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.970+0000 7f2c0b400640 1 --2- 172.21.15.40:0/352740419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bf0003fb0 0x7f2bf0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.970+0000 7f2c0b400640 1 -- 172.21.15.40:0/352740419 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf0003520 con 0x7f2bf0004d40 2024-04-26T23:10:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.971+0000 7f2c0b400640 1 --2- 172.21.15.40:0/352740419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bf0004d40 0x7f2bf00051c0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f2c04001e20 tx=0x7f2bf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c6ddbedaf5e0af server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.971+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c04001a70 con 0x7f2bf0004d40 2024-04-26T23:10:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.971+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c0402ae40 con 0x7f2bf0004d40 2024-04-26T23:10:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.971+0000 7f2c0a000640 1 -- 172.21.15.40:0/352740419 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c04003bb0 con 0x7f2bf0004d40 2024-04-26T23:10:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.972+0000 7f2c11800640 1 -- 172.21.15.40:0/352740419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bf0004d40 msgr2=0x7f2bf00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.972+0000 7f2c11800640 1 --2- 172.21.15.40:0/352740419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bf0004d40 0x7f2bf00051c0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f2c04001e20 tx=0x7f2bf0005830 comp rx=0 tx=0).stop 2024-04-26T23:10:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.972+0000 7f2c0b400640 1 -- 172.21.15.40:0/352740419 reap_dead start 2024-04-26T23:10:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.972+0000 7f2c11800640 1 -- 172.21.15.40:0/352740419 shutdown_connections 2024-04-26T23:10:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.972+0000 7f2c11800640 1 -- 172.21.15.40:0/352740419 >> 172.21.15.40:0/352740419 conn(0x7f2c0c0d0be0 msgr2=0x7f2c0c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.972+0000 7f2c11800640 1 -- 172.21.15.40:0/352740419 shutdown_connections 2024-04-26T23:10:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.972+0000 7f2c11800640 1 -- 172.21.15.40:0/352740419 wait complete. 2024-04-26T23:10:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.974+0000 7f2c11800640 1 Processor -- start 2024-04-26T23:10:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.974+0000 7f2c11800640 1 -- start start 2024-04-26T23:10:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.975+0000 7f2c11800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bf0004d40 0x7f2c0c1f4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.975+0000 7f2c11800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c0c0ca940 0x7f2c0c0c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.975+0000 7f2c11800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c0c0c69d0 0x7f2c0c0c6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.975+0000 7f2c11800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c0afe40 con 0x7f2c0c0ca940 2024-04-26T23:10:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.975+0000 7f2c11800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c0afcc0 con 0x7f2bf0004d40 2024-04-26T23:10:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.975+0000 7f2c11800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2c0c0affc0 con 0x7f2c0c0c69d0 2024-04-26T23:10:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.975+0000 7f2c03e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c0c0ca940 0x7f2c0c0c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.975+0000 7f2c03e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c0c0ca940 0x7f2c0c0c8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58684/0 (socket says 172.21.15.40:58684) 2024-04-26T23:10:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.975+0000 7f2c03e00640 1 -- 172.21.15.40:0/3089145306 learned_addr learned my addr 172.21.15.40:0/3089145306 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.975+0000 7f2c0aa00640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bf0004d40 0x7f2c0c1f4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.976+0000 7f2c0b400640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c0c0c69d0 0x7f2c0c0c6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.976+0000 7f2c03e00640 1 -- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c0c0c69d0 msgr2=0x7f2c0c0c6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.976+0000 7f2c03e00640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c0c0c69d0 0x7f2c0c0c6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.976+0000 7f2c03e00640 1 -- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bf0004d40 msgr2=0x7f2c0c1f4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.976+0000 7f2c03e00640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bf0004d40 0x7f2c0c1f4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.976+0000 7f2c03e00640 1 -- 172.21.15.40:0/3089145306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c0c0c9000 con 0x7f2c0c0ca940 2024-04-26T23:10:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.976+0000 7f2c0aa00640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bf0004d40 0x7f2c0c1f4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:10:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.976+0000 7f2c03e00640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c0c0ca940 0x7f2c0c0c8ac0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400e890 tx=0x7f2bf400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.976+0000 7f2c02a00640 1 -- 172.21.15.40:0/3089145306 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf400c870 con 0x7f2c0c0ca940 2024-04-26T23:10:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.976+0000 7f2c02a00640 1 -- 172.21.15.40:0/3089145306 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bf4018070 con 0x7f2c0c0ca940 2024-04-26T23:10:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.977+0000 7f2c02a00640 1 -- 172.21.15.40:0/3089145306 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf400cc70 con 0x7f2c0c0ca940 2024-04-26T23:10:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.977+0000 7f2c11800640 1 -- 172.21.15.40:0/3089145306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c0c0c50c0 con 0x7f2c0c0ca940 2024-04-26T23:10:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.977+0000 7f2c11800640 1 -- 172.21.15.40:0/3089145306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c0c0c9e60 con 0x7f2c0c0ca940 2024-04-26T23:10:23.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.978+0000 7f2c11800640 1 -- 172.21.15.40:0/3089145306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c0c0694f0 con 0x7f2c0c0ca940 2024-04-26T23:10:23.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.979+0000 7f2c02a00640 1 -- 172.21.15.40:0/3089145306 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2bf401d460 con 0x7f2c0c0ca940 2024-04-26T23:10:23.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.980+0000 7f2c02a00640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2be806c8f0 0x7f2be806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:23.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.980+0000 7f2c02a00640 1 -- 172.21.15.40:0/3089145306 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f2bf408b9f0 con 0x7f2c0c0ca940 2024-04-26T23:10:23.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.980+0000 7f2c0aa00640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2be806c8f0 0x7f2be806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.982+0000 7f2c0aa00640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2be806c8f0 0x7f2be806edb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc009ad0 tx=0x7f2bfc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:23.984+0000 7f2c02a00640 1 -- 172.21.15.40:0/3089145306 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f2bf4059ed0 con 0x7f2c0c0ca940 2024-04-26T23:10:24.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.185+0000 7f2c11800640 1 -- 172.21.15.40:0/3089145306 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2c0c0b0df0 con 0x7f2be806c8f0 2024-04-26T23:10:24.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.188+0000 7f2c02a00640 1 -- 172.21.15.40:0/3089145306 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13370 (secure 0 0 0) 0x7f2c0c0b0df0 con 0x7f2be806c8f0 2024-04-26T23:10:24.188 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:24.189 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:10:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.191+0000 7f2c11800640 1 -- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2be806c8f0 msgr2=0x7f2be806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.191+0000 7f2c11800640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2be806c8f0 0x7f2be806edb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc009ad0 tx=0x7f2bfc008040 comp rx=0 tx=0).stop 2024-04-26T23:10:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.191+0000 7f2c11800640 1 -- 172.21.15.40:0/3089145306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c0c0ca940 msgr2=0x7f2c0c0c8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:24.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.191+0000 7f2c11800640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c0c0ca940 0x7f2c0c0c8ac0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400e890 tx=0x7f2bf400ed50 comp rx=0 tx=0).stop 2024-04-26T23:10:24.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.191+0000 7f2c11800640 1 -- 172.21.15.40:0/3089145306 shutdown_connections 2024-04-26T23:10:24.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.191+0000 7f2c11800640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f2be806c8f0 0x7f2be806edb0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:24.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.191+0000 7f2c11800640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c0c0c69d0 0x7f2c0c0c6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:24.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.191+0000 7f2c11800640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c0c0ca940 0x7f2c0c0c8ac0 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:24.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.191+0000 7f2c11800640 1 --2- 172.21.15.40:0/3089145306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bf0004d40 0x7f2c0c1f4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:24.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.191+0000 7f2c11800640 1 -- 172.21.15.40:0/3089145306 >> 172.21.15.40:0/3089145306 conn(0x7f2c0c0d0be0 msgr2=0x7f2c0c074ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:24.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.191+0000 7f2c11800640 1 -- 172.21.15.40:0/3089145306 shutdown_connections 2024-04-26T23:10:24.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:24.191+0000 7f2c11800640 1 -- 172.21.15.40:0/3089145306 wait complete. 2024-04-26T23:10:24.211 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-04-26T23:10:23.352041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":261144,"kb_used_data":11688,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713576,"statfs":{"total":773094113280,"available":772826701824,"internally_reserved":0,"allocated":11968512,"data_stored":6224336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.612999"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":340,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-26T23:10:12.749726+0000","last_change":"2024-04-26T23:00:55.030722+0000","last_active":"2024-04-26T23:10:12.749726+0000","last_peered":"2024-04-26T23:10:12.749726+0000","last_clean":"2024-04-26T23:10:12.749726+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:10:12.749726+0000","last_undegraded":"2024-04-26T23:10:12.749726+0000","last_fullsized":"2024-04-26T23:10:12.749726+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:00:53.435963+0000","last_clean_scrub_stamp":"2024-04-26T23:00:53.435963+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:29:36.734161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574964,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1941504,"data_stored":1220818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1941504,"data_stored":1220818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:10:24.214 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-26T23:00:53.435963+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=23, tm_min=0, tm_sec=53, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=23, tm_min=10, tm_sec=16, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T23:10:24.214 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-26T23:10:44.216 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:10:44.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.438+0000 7f8442e00640 1 Processor -- start 2024-04-26T23:10:44.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.438+0000 7f8442e00640 1 -- start start 2024-04-26T23:10:44.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.440+0000 7f8442e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f843c15ce60 0x7f843c0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:44.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.440+0000 7f8442e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f843c071fe0 0x7f843c073760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:44.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.440+0000 7f8442e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f843c0d0820 con 0x7f843c0af7a0 2024-04-26T23:10:44.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.440+0000 7f8442e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f843c0cc450 con 0x7f843c071fe0 2024-04-26T23:10:44.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.440+0000 7f8442e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f843c1614c0 con 0x7f843c15ce60 2024-04-26T23:10:44.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.441+0000 7f8441a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f843c15ce60 0x7f843c0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:44.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.441+0000 7f8441a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f843c15ce60 0x7f843c0afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.40:37518/0 (socket says 172.21.15.40:37518) 2024-04-26T23:10:44.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.441+0000 7f8441a00640 1 -- 172.21.15.40:0/1571206768 learned_addr learned my addr 172.21.15.40:0/1571206768 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:44.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.441+0000 7f8442400640 1 --2- 172.21.15.40:0/1571206768 >> v2:172.21.15.103:3300/0 conn(0x7f843c071fe0 0x7f843c073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:44.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.441+0000 7f8441a00640 1 -- 172.21.15.40:0/1571206768 >> v2:172.21.15.103:3300/0 conn(0x7f843c071fe0 msgr2=0x7f843c073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:44.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.441+0000 7f8441a00640 1 --2- 172.21.15.40:0/1571206768 >> v2:172.21.15.103:3300/0 conn(0x7f843c071fe0 0x7f843c073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.441+0000 7f8441a00640 1 -- 172.21.15.40:0/1571206768 >> v1:172.21.15.103:6789/0 conn(0x7f843c0af7a0 legacy=0x7f843c0ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:44.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.441+0000 7f8441a00640 1 -- 172.21.15.40:0/1571206768 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f843c161760 con 0x7f843c15ce60 2024-04-26T23:10:44.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.443+0000 7f8441a00640 1 --2- 172.21.15.40:0/1571206768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f843c15ce60 0x7f843c0afc60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8430009e50 tx=0x7f843002a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1eeca8320aa5692 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:44.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.443+0000 7f843be00640 1 -- 172.21.15.40:0/1571206768 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8430036070 con 0x7f843c15ce60 2024-04-26T23:10:44.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.444+0000 7f843be00640 1 -- 172.21.15.40:0/1571206768 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8430031600 con 0x7f843c15ce60 2024-04-26T23:10:44.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.444+0000 7f843be00640 1 -- 172.21.15.40:0/1571206768 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84300319d0 con 0x7f843c15ce60 2024-04-26T23:10:44.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.444+0000 7f8442e00640 1 -- 172.21.15.40:0/1571206768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f843c15ce60 msgr2=0x7f843c0afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:44.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.444+0000 7f8442e00640 1 --2- 172.21.15.40:0/1571206768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f843c15ce60 0x7f843c0afc60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8430009e50 tx=0x7f843002a680 comp rx=0 tx=0).stop 2024-04-26T23:10:44.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.444+0000 7f8442e00640 1 -- 172.21.15.40:0/1571206768 shutdown_connections 2024-04-26T23:10:44.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.444+0000 7f8442e00640 1 --2- 172.21.15.40:0/1571206768 >> v2:172.21.15.103:3300/0 conn(0x7f843c071fe0 0x7f843c073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.444+0000 7f8442e00640 1 --2- 172.21.15.40:0/1571206768 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f843c15ce60 0x7f843c0afc60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.444+0000 7f8442e00640 1 -- 172.21.15.40:0/1571206768 >> 172.21.15.40:0/1571206768 conn(0x7f843c0d0a10 msgr2=0x7f843c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:44.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.444+0000 7f8442e00640 1 -- 172.21.15.40:0/1571206768 shutdown_connections 2024-04-26T23:10:44.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.445+0000 7f8442e00640 1 -- 172.21.15.40:0/1571206768 wait complete. 2024-04-26T23:10:44.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.446+0000 7f8442e00640 1 Processor -- start 2024-04-26T23:10:44.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.446+0000 7f8442e00640 1 -- start start 2024-04-26T23:10:44.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.447+0000 7f8442e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f843c071fe0 0x7f843c0ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:44.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.447+0000 7f8442e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c0af7a0 0x7f843c0aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:44.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.447+0000 7f8442e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f843c15ce60 0x7f843c15f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:44.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.447+0000 7f8442e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f843c0cc1f0 con 0x7f843c0af7a0 2024-04-26T23:10:44.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.447+0000 7f8442e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f843c0cc070 con 0x7f843c071fe0 2024-04-26T23:10:44.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.447+0000 7f8442e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f843c073ca0 con 0x7f843c15ce60 2024-04-26T23:10:44.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.448+0000 7f8441000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c0af7a0 0x7f843c0aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:44.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.448+0000 7f8441000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c0af7a0 0x7f843c0aee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34972/0 (socket says 172.21.15.40:34972) 2024-04-26T23:10:44.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.448+0000 7f8441000640 1 -- 172.21.15.40:0/2082988157 learned_addr learned my addr 172.21.15.40:0/2082988157 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:44.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.448+0000 7f8441a00640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f843c071fe0 0x7f843c0ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.448+0000 7f8442400640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f843c15ce60 0x7f843c15f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.448+0000 7f8441000640 1 -- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f843c15ce60 msgr2=0x7f843c15f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.448+0000 7f8441000640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f843c15ce60 0x7f843c15f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.448+0000 7f8441000640 1 -- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f843c071fe0 msgr2=0x7f843c0ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.448+0000 7f8441000640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f843c071fe0 0x7f843c0ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.448+0000 7f8441000640 1 -- 172.21.15.40:0/2082988157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f843c0c7e60 con 0x7f843c0af7a0 2024-04-26T23:10:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.448+0000 7f8442400640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f843c15ce60 0x7f843c15f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.449+0000 7f8441000640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c0af7a0 0x7f843c0aee30 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f842800c430 tx=0x7f842800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.449+0000 7f8439600640 1 -- 172.21.15.40:0/2082988157 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84280052d0 con 0x7f843c0af7a0 2024-04-26T23:10:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.449+0000 7f8439600640 1 -- 172.21.15.40:0/2082988157 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8428005490 con 0x7f843c0af7a0 2024-04-26T23:10:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.449+0000 7f8442e00640 1 -- 172.21.15.40:0/2082988157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f843c0c8140 con 0x7f843c0af7a0 2024-04-26T23:10:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.449+0000 7f8439600640 1 -- 172.21.15.40:0/2082988157 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8428007260 con 0x7f843c0af7a0 2024-04-26T23:10:44.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.449+0000 7f8442e00640 1 -- 172.21.15.40:0/2082988157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f843c0c8700 con 0x7f843c0af7a0 2024-04-26T23:10:44.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.451+0000 7f8442e00640 1 -- 172.21.15.40:0/2082988157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f843c069370 con 0x7f843c0af7a0 2024-04-26T23:10:44.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.451+0000 7f8439600640 1 -- 172.21.15.40:0/2082988157 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8428007400 con 0x7f843c0af7a0 2024-04-26T23:10:44.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.453+0000 7f8439600640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f841c06c8f0 0x7f841c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:44.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.453+0000 7f8439600640 1 -- 172.21.15.40:0/2082988157 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f842808aeb0 con 0x7f843c0af7a0 2024-04-26T23:10:44.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.453+0000 7f8441a00640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f841c06c8f0 0x7f841c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.455+0000 7f8441a00640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f841c06c8f0 0x7f841c06edb0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8430033000 tx=0x7f8430009000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:44.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.458+0000 7f8439600640 1 -- 172.21.15.40:0/2082988157 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f8428058de0 con 0x7f843c0af7a0 2024-04-26T23:10:44.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.661+0000 7f8442e00640 1 -- 172.21.15.40:0/2082988157 --> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f843c0cee70 con 0x7f841c06c8f0 2024-04-26T23:10:44.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.663+0000 7f8439600640 1 -- 172.21.15.40:0/2082988157 <== mgr.4111 v2:172.21.15.103:6832/1151751020 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13379 (secure 0 0 0) 0x7f843c0cee70 con 0x7f841c06c8f0 2024-04-26T23:10:44.664 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:44.664 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:10:44.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.666+0000 7f8442e00640 1 -- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f841c06c8f0 msgr2=0x7f841c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:44.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.666+0000 7f8442e00640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f841c06c8f0 0x7f841c06edb0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8430033000 tx=0x7f8430009000 comp rx=0 tx=0).stop 2024-04-26T23:10:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.666+0000 7f8442e00640 1 -- 172.21.15.40:0/2082988157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c0af7a0 msgr2=0x7f843c0aee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.666+0000 7f8442e00640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c0af7a0 0x7f843c0aee30 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f842800c430 tx=0x7f842800c8f0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.666+0000 7f8442e00640 1 -- 172.21.15.40:0/2082988157 shutdown_connections 2024-04-26T23:10:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.666+0000 7f8442e00640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f841c06c8f0 0x7f841c06edb0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.666+0000 7f8442e00640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f843c15ce60 0x7f843c15f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.666+0000 7f8442e00640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f843c0af7a0 0x7f843c0aee30 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.666+0000 7f8442e00640 1 --2- 172.21.15.40:0/2082988157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f843c071fe0 0x7f843c0ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.666+0000 7f8442e00640 1 -- 172.21.15.40:0/2082988157 >> 172.21.15.40:0/2082988157 conn(0x7f843c0d0a10 msgr2=0x7f843c1581d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.666+0000 7f8442e00640 1 -- 172.21.15.40:0/2082988157 shutdown_connections 2024-04-26T23:10:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.666+0000 7f8442e00640 1 -- 172.21.15.40:0/2082988157 wait complete. 2024-04-26T23:10:44.687 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-04-26T23:10:43.358156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":261128,"kb_used_data":11688,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713592,"statfs":{"total":773094113280,"available":772826718208,"internally_reserved":0,"allocated":11968512,"data_stored":6224336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003618"},"pg_stats":[{"pgid":"1.0","version":"108'119","reported_seq":351,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-26T23:10:23.907337+0000","last_change":"2024-04-26T23:10:23.907470+0000","last_active":"2024-04-26T23:10:23.907337+0000","last_peered":"2024-04-26T23:10:23.907337+0000","last_clean":"2024-04-26T23:10:23.907337+0000","last_became_active":"2024-04-26T23:00:55.028192+0000","last_became_peered":"2024-04-26T23:00:55.028192+0000","last_unstale":"2024-04-26T23:10:23.907337+0000","last_undegraded":"2024-04-26T23:10:23.907337+0000","last_fullsized":"2024-04-26T23:10:23.907337+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'119","last_scrub_stamp":"2024-04-26T23:10:23.907133+0000","last_deep_scrub":"108'119","last_deep_scrub_stamp":"2024-04-26T23:10:23.907133+0000","last_clean_scrub_stamp":"2024-04-26T23:10:23.907133+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:38:20.356288+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574968,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33072,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338768,"statfs":{"total":96636764160,"available":96602898432,"internally_reserved":0,"allocated":1941504,"data_stored":1220818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":630450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574969,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33072,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338768,"statfs":{"total":96636764160,"available":96602898432,"internally_reserved":0,"allocated":1941504,"data_stored":1220818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:10:44.688 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-26T23:10:44.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.895+0000 7f7375600640 1 Processor -- start 2024-04-26T23:10:44.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.895+0000 7f7375600640 1 -- start start 2024-04-26T23:10:44.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.897+0000 7f7375600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73700cc480 0x7f7370072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:44.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.897+0000 7f7375600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7370072780 0x7f7370166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:44.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.897+0000 7f7375600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f737015d7c0 con 0x7f7370071e60 2024-04-26T23:10:44.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.897+0000 7f7375600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f737015daa0 con 0x7f7370072780 2024-04-26T23:10:44.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.897+0000 7f7375600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f737015dd80 con 0x7f73700cc480 2024-04-26T23:10:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.898+0000 7f736ea00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7370071e60 0x7f737006f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.40:52570/0 (socket says 172.21.15.40:52570) 2024-04-26T23:10:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.898+0000 7f736ea00640 1 -- 172.21.15.40:0/3919016690 learned_addr learned my addr 172.21.15.40:0/3919016690 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-26T23:10:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.898+0000 7f7366000640 1 --2- 172.21.15.40:0/3919016690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73700cc480 0x7f7370072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:44.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.898+0000 7f736f400640 1 --2- 172.21.15.40:0/3919016690 >> v2:172.21.15.103:3300/0 conn(0x7f7370072780 0x7f7370166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:44.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.898+0000 7f7366000640 1 -- 172.21.15.40:0/3919016690 >> v2:172.21.15.103:3300/0 conn(0x7f7370072780 msgr2=0x7f7370166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:44.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.898+0000 7f7366000640 1 --2- 172.21.15.40:0/3919016690 >> v2:172.21.15.103:3300/0 conn(0x7f7370072780 0x7f7370166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.898+0000 7f7366000640 1 -- 172.21.15.40:0/3919016690 >> v1:172.21.15.103:6789/0 conn(0x7f7370071e60 legacy=0x7f737006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:44.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.898+0000 7f736e000640 1 -- 172.21.15.40:0/3919016690 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 291466459 0 0) 0x7f737015d7c0 con 0x7f7370071e60 2024-04-26T23:10:44.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.899+0000 7f7366000640 1 -- 172.21.15.40:0/3919016690 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737015e060 con 0x7f73700cc480 2024-04-26T23:10:44.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.900+0000 7f7366000640 1 --2- 172.21.15.40:0/3919016690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73700cc480 0x7f7370072240 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7358009d90 tx=0x7f735802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=135827ccb3d9bdc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:44.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.900+0000 7f736e000640 1 -- 172.21.15.40:0/3919016690 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7358007740 con 0x7f73700cc480 2024-04-26T23:10:44.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.900+0000 7f736e000640 1 -- 172.21.15.40:0/3919016690 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73580078e0 con 0x7f73700cc480 2024-04-26T23:10:44.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.900+0000 7f736e000640 1 -- 172.21.15.40:0/3919016690 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7358007cb0 con 0x7f73700cc480 2024-04-26T23:10:44.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.901+0000 7f7375600640 1 -- 172.21.15.40:0/3919016690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73700cc480 msgr2=0x7f7370072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:44.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.901+0000 7f7375600640 1 --2- 172.21.15.40:0/3919016690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73700cc480 0x7f7370072240 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7358009d90 tx=0x7f735802a660 comp rx=0 tx=0).stop 2024-04-26T23:10:44.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.901+0000 7f7375600640 1 -- 172.21.15.40:0/3919016690 shutdown_connections 2024-04-26T23:10:44.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.901+0000 7f7375600640 1 --2- 172.21.15.40:0/3919016690 >> v2:172.21.15.103:3300/0 conn(0x7f7370072780 0x7f7370166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.901+0000 7f7375600640 1 --2- 172.21.15.40:0/3919016690 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73700cc480 0x7f7370072240 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.901+0000 7f7375600640 1 -- 172.21.15.40:0/3919016690 >> 172.21.15.40:0/3919016690 conn(0x7f73700d0be0 msgr2=0x7f73700d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:44.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.901+0000 7f7375600640 1 -- 172.21.15.40:0/3919016690 shutdown_connections 2024-04-26T23:10:44.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.901+0000 7f7375600640 1 -- 172.21.15.40:0/3919016690 wait complete. 2024-04-26T23:10:44.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.904+0000 7f7375600640 1 Processor -- start 2024-04-26T23:10:44.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.904+0000 7f7375600640 1 -- start start 2024-04-26T23:10:44.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.905+0000 7f7375600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7370071e60 0x7f73701efc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:44.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.905+0000 7f7375600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7370072780 0x7f73700b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:44.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.905+0000 7f7375600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73700cc480 0x7f73700b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:44.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.905+0000 7f7375600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f737015d3f0 con 0x7f7370072780 2024-04-26T23:10:44.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.905+0000 7f7375600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7370072c10 con 0x7f7370071e60 2024-04-26T23:10:44.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.905+0000 7f7375600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f737015d570 con 0x7f73700cc480 2024-04-26T23:10:44.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.906+0000 7f7366000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7370072780 0x7f73700b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:44.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.906+0000 7f7366000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7370072780 0x7f73700b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54996/0 (socket says 172.21.15.40:54996) 2024-04-26T23:10:44.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.906+0000 7f7366000640 1 -- 172.21.15.40:0/1386152384 learned_addr learned my addr 172.21.15.40:0/1386152384 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:44.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.906+0000 7f736ea00640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7370071e60 0x7f73701efc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:44.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.906+0000 7f736f400640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73700cc480 0x7f73700b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.906+0000 7f7366000640 1 -- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73700cc480 msgr2=0x7f73700b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.906+0000 7f7366000640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73700cc480 0x7f73700b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.906+0000 7f7366000640 1 -- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7370071e60 msgr2=0x7f73701efc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.906+0000 7f7366000640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7370071e60 0x7f73701efc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.906+0000 7f7366000640 1 -- 172.21.15.40:0/1386152384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73701f94a0 con 0x7f7370072780 2024-04-26T23:10:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.906+0000 7f736f400640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73700cc480 0x7f73700b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.907+0000 7f7366000640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7370072780 0x7f73700b0aa0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f7358009d90 tx=0x7f7358004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.907+0000 7f7367400640 1 -- 172.21.15.40:0/1386152384 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735803f070 con 0x7f7370072780 2024-04-26T23:10:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.907+0000 7f7367400640 1 -- 172.21.15.40:0/1386152384 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7358031c40 con 0x7f7370072780 2024-04-26T23:10:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.907+0000 7f7367400640 1 -- 172.21.15.40:0/1386152384 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7358009450 con 0x7f7370072780 2024-04-26T23:10:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.907+0000 7f7375600640 1 -- 172.21.15.40:0/1386152384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73701f9740 con 0x7f7370072780 2024-04-26T23:10:44.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.907+0000 7f7375600640 1 -- 172.21.15.40:0/1386152384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73701f9d00 con 0x7f7370072780 2024-04-26T23:10:44.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.908+0000 7f7375600640 1 -- 172.21.15.40:0/1386152384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73700694f0 con 0x7f7370072780 2024-04-26T23:10:44.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.909+0000 7f7367400640 1 -- 172.21.15.40:0/1386152384 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7358043410 con 0x7f7370072780 2024-04-26T23:10:44.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.910+0000 7f7367400640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f734c06c8f0 0x7f734c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:44.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.911+0000 7f7367400640 1 -- 172.21.15.40:0/1386152384 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f73580acf10 con 0x7f7370072780 2024-04-26T23:10:44.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.911+0000 7f736ea00640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f734c06c8f0 0x7f734c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:44.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.912+0000 7f736ea00640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f734c06c8f0 0x7f734c06edb0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f7360000f60 tx=0x7f7360007b50 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:44.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:44.915+0000 7f7367400640 1 -- 172.21.15.40:0/1386152384 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f735807b360 con 0x7f7370072780 2024-04-26T23:10:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.120+0000 7f7375600640 1 -- 172.21.15.40:0/1386152384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f7370160570 con 0x7f7370072780 2024-04-26T23:10:45.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.127+0000 7f7367400640 1 -- 172.21.15.40:0/1386152384 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v22)=0 v22) ==== 123+0+0 (secure 0 0 0) 0x7f735807ac60 con 0x7f7370072780 2024-04-26T23:10:45.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.127+0000 7f7367400640 1 -- 172.21.15.40:0/1386152384 <== mon.0 v2:172.21.15.40:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f735807f010 con 0x7f7370072780 2024-04-26T23:10:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.130+0000 7f7375600640 1 -- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f734c06c8f0 msgr2=0x7f734c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.130+0000 7f7375600640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f734c06c8f0 0x7f734c06edb0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f7360000f60 tx=0x7f7360007b50 comp rx=0 tx=0).stop 2024-04-26T23:10:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.130+0000 7f7375600640 1 -- 172.21.15.40:0/1386152384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7370072780 msgr2=0x7f73700b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.130+0000 7f7375600640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7370072780 0x7f73700b0aa0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f7358009d90 tx=0x7f7358004a40 comp rx=0 tx=0).stop 2024-04-26T23:10:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.130+0000 7f7375600640 1 -- 172.21.15.40:0/1386152384 shutdown_connections 2024-04-26T23:10:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.130+0000 7f7375600640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f734c06c8f0 0x7f734c06edb0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.130+0000 7f7375600640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f73700cc480 0x7f73700b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.130+0000 7f7375600640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7370072780 0x7f73700b0aa0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.130+0000 7f7375600640 1 --2- 172.21.15.40:0/1386152384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7370071e60 0x7f73701efc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.130+0000 7f7375600640 1 -- 172.21.15.40:0/1386152384 >> 172.21.15.40:0/1386152384 conn(0x7f73700d0be0 msgr2=0x7f73700ce7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.130+0000 7f7375600640 1 -- 172.21.15.40:0/1386152384 shutdown_connections 2024-04-26T23:10:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.131+0000 7f7375600640 1 -- 172.21.15.40:0/1386152384 wait complete. 2024-04-26T23:10:45.151 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T23:10:45.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:10:45.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.361+0000 7f1636600640 1 Processor -- start 2024-04-26T23:10:45.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.361+0000 7f1636600640 1 -- start start 2024-04-26T23:10:45.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.362+0000 7f1636600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f16300ce5e0 0x7f16300afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:45.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.363+0000 7f1636600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f16300cc590 0x7f16300cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:45.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.363+0000 7f1636600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16300af7b0 con 0x7f163015f2d0 2024-04-26T23:10:45.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.363+0000 7f1636600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 -- 0x7f16300afa90 con 0x7f16300ce5e0 2024-04-26T23:10:45.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.363+0000 7f1636600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f16300cbd90 con 0x7f16300cc590 2024-04-26T23:10:45.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.363+0000 7f162fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f16300ce5e0 0x7f16300afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:45.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.363+0000 7f162fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f16300ce5e0 0x7f16300afdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55008/0 (socket says 172.21.15.40:55008) 2024-04-26T23:10:45.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.363+0000 7f162fe00640 1 -- 172.21.15.40:0/3749852924 learned_addr learned my addr 172.21.15.40:0/3749852924 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:45.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.363+0000 7f1635c00640 1 --2- 172.21.15.40:0/3749852924 >> v2:172.21.15.103:3300/0 conn(0x7f16300cc590 0x7f16300cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:45.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.363+0000 7f162fe00640 1 -- 172.21.15.40:0/3749852924 >> v2:172.21.15.103:3300/0 conn(0x7f16300cc590 msgr2=0x7f16300cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:45.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.363+0000 7f162fe00640 1 --2- 172.21.15.40:0/3749852924 >> v2:172.21.15.103:3300/0 conn(0x7f16300cc590 0x7f16300cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.363+0000 7f162fe00640 1 -- 172.21.15.40:0/3749852924 >> v1:172.21.15.103:6789/0 conn(0x7f163015f2d0 legacy=0x7f16300c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:10:45.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.363+0000 7f162fe00640 1 -- 172.21.15.40:0/3749852924 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16300cc070 con 0x7f16300ce5e0 2024-04-26T23:10:45.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.365+0000 7f162fe00640 1 --2- 172.21.15.40:0/3749852924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16300ce5e0 0x7f16300afdf0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f16300c7a50 tx=0x7f162002c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa2b33a06c13e1d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:45.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.365+0000 7f162f400640 1 -- 172.21.15.40:0/3749852924 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f162000b2d0 con 0x7f16300ce5e0 2024-04-26T23:10:45.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.365+0000 7f162f400640 1 -- 172.21.15.40:0/3749852924 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f162000b470 con 0x7f16300ce5e0 2024-04-26T23:10:45.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.365+0000 7f162f400640 1 -- 172.21.15.40:0/3749852924 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1620009840 con 0x7f16300ce5e0 2024-04-26T23:10:45.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.366+0000 7f1636600640 1 -- 172.21.15.40:0/3749852924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16300ce5e0 msgr2=0x7f16300afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:45.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.366+0000 7f1636600640 1 --2- 172.21.15.40:0/3749852924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16300ce5e0 0x7f16300afdf0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f16300c7a50 tx=0x7f162002c8f0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.366+0000 7f1636600640 1 -- 172.21.15.40:0/3749852924 shutdown_connections 2024-04-26T23:10:45.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.366+0000 7f1636600640 1 --2- 172.21.15.40:0/3749852924 >> v2:172.21.15.103:3300/0 conn(0x7f16300cc590 0x7f16300cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.366+0000 7f1636600640 1 --2- 172.21.15.40:0/3749852924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16300ce5e0 0x7f16300afdf0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.366+0000 7f1636600640 1 -- 172.21.15.40:0/3749852924 >> 172.21.15.40:0/3749852924 conn(0x7f16300d0a10 msgr2=0x7f16300d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:45.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.366+0000 7f1636600640 1 -- 172.21.15.40:0/3749852924 shutdown_connections 2024-04-26T23:10:45.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.366+0000 7f1636600640 1 -- 172.21.15.40:0/3749852924 wait complete. 2024-04-26T23:10:45.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.368+0000 7f1636600640 1 Processor -- start 2024-04-26T23:10:45.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.368+0000 7f1636600640 1 -- start start 2024-04-26T23:10:45.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.369+0000 7f1636600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16300cc590 0x7f16301580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:45.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.369+0000 7f1636600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16300ce5e0 0x7f163006f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:45.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.369+0000 7f1636600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f163015f2d0 0x7f1630154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:45.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.369+0000 7f1636600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f16300c5fc0 con 0x7f163015f2d0 2024-04-26T23:10:45.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.369+0000 7f1636600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f163006e270 con 0x7f16300cc590 2024-04-26T23:10:45.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.369+0000 7f1636600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f163015c280 con 0x7f16300ce5e0 2024-04-26T23:10:45.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.369+0000 7f1635c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f163015f2d0 0x7f1630154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.369+0000 7f1635c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f163015f2d0 0x7f1630154fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55012/0 (socket says 172.21.15.40:55012) 2024-04-26T23:10:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.369+0000 7f1635c00640 1 -- 172.21.15.40:0/1671096330 learned_addr learned my addr 172.21.15.40:0/1671096330 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:10:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.370+0000 7f1635200640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16300cc590 0x7f16301580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.370+0000 7f162fe00640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16300ce5e0 0x7f163006f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.370+0000 7f1635c00640 1 -- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16300ce5e0 msgr2=0x7f163006f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.370+0000 7f1635c00640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16300ce5e0 0x7f163006f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.370+0000 7f1635c00640 1 -- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16300cc590 msgr2=0x7f16301580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.370+0000 7f1635c00640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16300cc590 0x7f16301580f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.370+0000 7f1635c00640 1 -- 172.21.15.40:0/1671096330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1630157630 con 0x7f163015f2d0 2024-04-26T23:10:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.370+0000 7f1635200640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16300cc590 0x7f16301580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:10:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.370+0000 7f1635c00640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f163015f2d0 0x7f1630154fc0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f162800c7f0 tx=0x7f162800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.370+0000 7f162cc00640 1 -- 172.21.15.40:0/1671096330 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1628015070 con 0x7f163015f2d0 2024-04-26T23:10:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.370+0000 7f162cc00640 1 -- 172.21.15.40:0/1671096330 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f1628004220 con 0x7f163015f2d0 2024-04-26T23:10:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.370+0000 7f162cc00640 1 -- 172.21.15.40:0/1671096330 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1628003610 con 0x7f163015f2d0 2024-04-26T23:10:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.371+0000 7f1636600640 1 -- 172.21.15.40:0/1671096330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16300b4310 con 0x7f163015f2d0 2024-04-26T23:10:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.371+0000 7f1636600640 1 -- 172.21.15.40:0/1671096330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1630157bd0 con 0x7f163015f2d0 2024-04-26T23:10:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.372+0000 7f1636600640 1 -- 172.21.15.40:0/1671096330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16300b4710 con 0x7f163015f2d0 2024-04-26T23:10:45.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.373+0000 7f162cc00640 1 -- 172.21.15.40:0/1671096330 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f16280039c0 con 0x7f163015f2d0 2024-04-26T23:10:45.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.374+0000 7f162cc00640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f160c06c8f0 0x7f160c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:10:45.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.374+0000 7f162cc00640 1 -- 172.21.15.40:0/1671096330 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5507+0+0 (secure 0 0 0) 0x7f162808bbd0 con 0x7f163015f2d0 2024-04-26T23:10:45.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.375+0000 7f1635200640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f160c06c8f0 0x7f160c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:10:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.376+0000 7f1635200640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f160c06c8f0 0x7f160c06edb0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f161800a830 tx=0x7f1618008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:10:45.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.379+0000 7f162cc00640 1 -- 172.21.15.40:0/1671096330 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187370 (secure 0 0 0) 0x7f16300b4710 con 0x7f163015f2d0 2024-04-26T23:10:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.627+0000 7f1636600640 1 -- 172.21.15.40:0/1671096330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1630069370 con 0x7f163015f2d0 2024-04-26T23:10:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.628+0000 7f162cc00640 1 -- 172.21.15.40:0/1671096330 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+3877 (secure 0 0 0) 0x7f1628059f90 con 0x7f163015f2d0 2024-04-26T23:10:45.629 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:10:45.629 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":124,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6836/647482000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":647482000},{"type":"v1","addr":"172.21.15.103:6836","nonce":647482000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3698092105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6835","nonce":3698092105},{"type":"v1","addr":"172.21.15.103:6837","nonce":3698092105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":9846,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1329506056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1329506056},{"type":"v1","addr":"172.21.15.40:6833","nonce":1329506056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":9855,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1716979532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1716979532},{"type":"v1","addr":"172.21.15.40:6835","nonce":1716979532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124}],"filesystems":[]} 2024-04-26T23:10:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 124 2024-04-26T23:10:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.632+0000 7f1636600640 1 -- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f160c06c8f0 msgr2=0x7f160c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.632+0000 7f1636600640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f160c06c8f0 0x7f160c06edb0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f161800a830 tx=0x7f1618008040 comp rx=0 tx=0).stop 2024-04-26T23:10:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.632+0000 7f1636600640 1 -- 172.21.15.40:0/1671096330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f163015f2d0 msgr2=0x7f1630154fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:10:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.632+0000 7f1636600640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f163015f2d0 0x7f1630154fc0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f162800c7f0 tx=0x7f162800ccb0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.632+0000 7f1636600640 1 -- 172.21.15.40:0/1671096330 shutdown_connections 2024-04-26T23:10:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.632+0000 7f1636600640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:6832/1151751020,v1:172.21.15.103:6833/1151751020] conn(0x7f160c06c8f0 0x7f160c06edb0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.632+0000 7f1636600640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f163015f2d0 0x7f1630154fc0 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.632+0000 7f1636600640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f16300ce5e0 0x7f163006f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.632+0000 7f1636600640 1 --2- 172.21.15.40:0/1671096330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16300cc590 0x7f16301580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:10:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.632+0000 7f1636600640 1 -- 172.21.15.40:0/1671096330 >> 172.21.15.40:0/1671096330 conn(0x7f16300d0a10 msgr2=0x7f16300cee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:10:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.632+0000 7f1636600640 1 -- 172.21.15.40:0/1671096330 shutdown_connections 2024-04-26T23:10:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:10:45.632+0000 7f1636600640 1 -- 172.21.15.40:0/1671096330 wait complete. 2024-04-26T23:10:45.652 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-26T23:10:45.653 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-26T23:10:45.653 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-26T23:10:45.653 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:45.673 INFO:tasks.ceph.mds.a:Stopped 2024-04-26T23:10:45.673 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-26T23:10:45.673 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:45.739 INFO:tasks.ceph.mds.c:Stopped 2024-04-26T23:10:45.739 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-26T23:10:45.739 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:45.765 INFO:tasks.ceph.mds.b:Stopped 2024-04-26T23:10:45.765 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-26T23:10:45.765 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:45.830 INFO:tasks.ceph.mds.d:Stopped 2024-04-26T23:10:45.830 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-26T23:10:45.830 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-26T23:10:45.831 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:45.911 INFO:tasks.ceph.osd.0:Stopped 2024-04-26T23:10:45.911 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-26T23:10:45.911 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:45.986 INFO:tasks.ceph.osd.1:Stopped 2024-04-26T23:10:45.987 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-26T23:10:45.987 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:46.049 INFO:tasks.ceph.osd.2:Stopped 2024-04-26T23:10:46.050 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-26T23:10:46.050 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:46.124 INFO:tasks.ceph.osd.3:Stopped 2024-04-26T23:10:46.124 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-26T23:10:46.124 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:46.181 INFO:tasks.ceph.osd.4:Stopped 2024-04-26T23:10:46.182 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-26T23:10:46.182 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:46.251 INFO:tasks.ceph.osd.5:Stopped 2024-04-26T23:10:46.252 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-26T23:10:46.252 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:46.315 INFO:tasks.ceph.osd.6:Stopped 2024-04-26T23:10:46.316 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-26T23:10:46.316 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:46.409 INFO:tasks.ceph.osd.7:Stopped 2024-04-26T23:10:46.409 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-26T23:10:46.409 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-26T23:10:46.409 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:46.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-26T23:10:46.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-26T23:10:46.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-04-26T23:10:46.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-04-26T23:10:46.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-04-26T23:10:46.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-04-26T23:10:46.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-04-26T23:10:49.630 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:10:49.639 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:10:52.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~6s 2024-04-26T23:10:52.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-26T23:10:52.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-04-26T23:10:52.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-04-26T23:10:52.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-04-26T23:10:52.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-04-26T23:10:52.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-04-26T23:10:52.511 INFO:tasks.ceph.mgr.y:Stopped 2024-04-26T23:10:52.511 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-26T23:10:52.511 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:57.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-04-26T23:10:57.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-04-26T23:10:58.612 INFO:tasks.ceph.mgr.x:Stopped 2024-04-26T23:10:58.613 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-26T23:10:58.613 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T23:10:58.613 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:58.665 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T23:10:58.666 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T23:10:58.666 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:58.709 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T23:10:58.709 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T23:10:58.709 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:10:58.746 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T23:10:58.746 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-26T23:10:58.746 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Scrub error on inode' | head -n 1 2024-04-26T23:10:58.856 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:10:58.856 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-26T23:10:59.035 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:10:59.036 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-26T23:10:59.216 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:10:59.217 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-26T23:10:59.354 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:10:59.354 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-26T23:10:59.529 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T23:10:59.530 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-26T23:10:59.753 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T23:10:59.753 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-26T23:10:59.877 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T23:10:59.878 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-26T23:11:00.023 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T23:11:00.023 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-26T23:11:00.183 INFO:tasks.ceph:Archiving mon data... 2024-04-26T23:11:00.191 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279/data/mon.a.tgz 2024-04-26T23:11:00.191 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-04-26T23:11:00.248 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.0O5UbBBaUG 2024-04-26T23:11:00.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0O5UbBBaUG 2024-04-26T23:11:00.436 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.0O5UbBBaUG 2024-04-26T23:11:00.586 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.0O5UbBBaUG is 389KB 2024-04-26T23:11:00.699 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.0O5UbBBaUG 2024-04-26T23:11:00.760 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279/data/mon.b.tgz 2024-04-26T23:11:00.760 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-04-26T23:11:00.815 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.mU9daWJ59B 2024-04-26T23:11:00.815 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.mU9daWJ59B 2024-04-26T23:11:00.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.mU9daWJ59B 2024-04-26T23:11:01.119 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.mU9daWJ59B is 389KB 2024-04-26T23:11:01.226 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.mU9daWJ59B 2024-04-26T23:11:01.282 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279/data/mon.c.tgz 2024-04-26T23:11:01.282 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-04-26T23:11:01.334 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.Zn3kCc3JaG 2024-04-26T23:11:01.334 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Zn3kCc3JaG 2024-04-26T23:11:01.501 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.Zn3kCc3JaG 2024-04-26T23:11:01.637 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.Zn3kCc3JaG is 389KB 2024-04-26T23:11:01.743 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.Zn3kCc3JaG 2024-04-26T23:11:01.798 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-26T23:11:01.798 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T23:11:01.805 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T23:11:01.943 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T23:11:01.999 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T23:11:01.999 INFO:tasks.ceph:Archiving crash dumps... 2024-04-26T23:11:02.001 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279/remote/smithi040/crash 2024-04-26T23:11:02.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T23:11:02.092 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279/remote/smithi103/crash 2024-04-26T23:11:02.093 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T23:11:02.178 INFO:tasks.ceph:Shutting down logrotate 2024-04-26T23:11:02.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:11:02.185 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-26T23:11:02.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-26T23:11:02.386 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-26T23:11:02.472 INFO:tasks.ceph:Compressing logs... 2024-04-26T23:11:02.472 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T23:11:02.480 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T23:11:02.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-04-26T23:11:02.560 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.39586.log 2024-04-26T23:11:02.561 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33341.log 2024-04-26T23:11:02.562 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.1.39586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.39586.log.gz 2024-04-26T23:11:02.562 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33218.log 2024-04-26T23:11:02.563 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.39567.log 88.1% 2024-04-26T23:11:02.563 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33341.log.gz 2024-04-26T23:11:02.564 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33218.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T23:11:02.564 INFO:teuthology.orchestra.run.smithi103.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33218.log.gz 2024-04-26T23:11:02.564 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.1.39567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.39567.log.gz 2024-04-26T23:11:02.565 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47508.log 2024-04-26T23:11:02.565 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T23:11:02.566 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50740.log 2024-04-26T23:11:02.566 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-26T23:11:02.567 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-26T23:11:02.567 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71421.log 2024-04-26T23:11:02.567 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47508.log.gz 2024-04-26T23:11:02.568 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-26T23:11:02.568 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60473.log 2024-04-26T23:11:02.569 INFO:teuthology.orchestra.run.smithi040.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50740.log.gz 2024-04-26T23:11:02.569 INFO:teuthology.orchestra.run.smithi103.stderr: 92.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T23:11:02.569 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33421.log 2024-04-26T23:11:02.570 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71421.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71421.log.gz 2024-04-26T23:11:02.570 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58607.log 2024-04-26T23:11:02.571 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-26T23:11:02.571 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T23:11:02.571 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42650.log 2024-04-26T23:11:02.572 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60473.log.gz 2024-04-26T23:11:02.573 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33381.log 87.6% 2024-04-26T23:11:02.573 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33421.log.gz 2024-04-26T23:11:02.573 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71505.log 2024-04-26T23:11:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58607.log.gz 2024-04-26T23:11:02.574 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58953.log 2024-04-26T23:11:02.574 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-26T23:11:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.42650.log.gz/var/log/ceph/ceph-client.admin.71505.log: 2024-04-26T23:11:02.575 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46693.log 2024-04-26T23:11:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71505.log.gz 2024-04-26T23:11:02.576 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33381.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-26T23:11:02.576 INFO:teuthology.orchestra.run.smithi103.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33381.log.gz 2024-04-26T23:11:02.577 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57099.log 2024-04-26T23:11:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58953.log.gz 2024-04-26T23:11:02.577 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-26T23:11:02.578 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64645.log 2024-04-26T23:11:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46693.log.gz 2024-04-26T23:11:02.578 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.57099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44230.log 2024-04-26T23:11:02.579 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33261.log 2024-04-26T23:11:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.57099.log.gz 2024-04-26T23:11:02.580 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.64645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50908.log 2024-04-26T23:11:02.580 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64645.log.gz 2024-04-26T23:11:02.581 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.33261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.39985.log 2024-04-26T23:11:02.582 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46339.log 2024-04-26T23:11:02.582 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44230.log.gz 2024-04-26T23:11:02.582 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50908.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50908.log.gz 2024-04-26T23:11:02.582 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41311.log 2024-04-26T23:11:02.583 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64045.log 2024-04-26T23:11:02.583 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46339.log.gz 2024-04-26T23:11:02.584 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.3.39985.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33261.log.gz 2024-04-26T23:11:02.585 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70951.log 2024-04-26T23:11:02.585 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41311.log.gz 2024-04-26T23:11:02.586 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.39981.log 2024-04-26T23:11:02.586 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.64045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48487.log 2024-04-26T23:11:02.586 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64045.log.gz 2024-04-26T23:11:02.587 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33170.log 2024-04-26T23:11:02.588 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.70951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41918.log 2024-04-26T23:11:02.588 INFO:teuthology.orchestra.run.smithi040.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.70951.log.gz 2024-04-26T23:11:02.588 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58523.log 2024-04-26T23:11:02.588 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48487.log.gz 2024-04-26T23:11:02.590 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68086.log 2024-04-26T23:11:02.590 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41918.log.gz 2024-04-26T23:11:02.590 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0%/var/log/ceph/ceph-client.admin.33170.log: -- replaced with /var/log/ceph/ceph-client.3.39985.log.gz 2024-04-26T23:11:02.591 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50824.log 2024-04-26T23:11:02.591 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58523.log.gz 2024-04-26T23:11:02.592 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.3.39981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33130.log 2024-04-26T23:11:02.593 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48823.log 2024-04-26T23:11:02.593 INFO:teuthology.orchestra.run.smithi040.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.68086.log.gz 2024-04-26T23:11:02.593 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44902.log 2024-04-26T23:11:02.593 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50824.log.gz 2024-04-26T23:11:02.594 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73658.log 2024-04-26T23:11:02.594 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48823.log.gz 2024-04-26T23:11:02.595 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53297.log 2024-04-26T23:11:02.596 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0%/var/log/ceph/ceph-client.admin.33130.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33130.log.gz 2024-04-26T23:11:02.596 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33301.log 2024-04-26T23:11:02.596 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/ceph-client.3.39981.log.gz 2024-04-26T23:11:02.597 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44902.log.gz 2024-04-26T23:11:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73658.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73658.log.gz 2024-04-26T23:11:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46255.log 2024-04-26T23:11:02.598 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33170.log.gz 2024-04-26T23:11:02.598 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73948.log 2024-04-26T23:11:02.598 INFO:teuthology.orchestra.run.smithi040.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.53297.log.gz 2024-04-26T23:11:02.598 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66745.log 2024-04-26T23:11:02.599 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46255.log.gz 2024-04-26T23:11:02.600 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.33301.log: /var/log/ceph/ceph-mds.d.log 2024-04-26T23:11:02.601 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56069.log 2024-04-26T23:11:02.601 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73948.log.gz 2024-04-26T23:11:02.601 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41043.log 2024-04-26T23:11:02.601 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66745.log.gz 2024-04-26T23:11:02.601 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48625.log 2024-04-26T23:11:02.602 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56069.log.gz 2024-04-26T23:11:02.603 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mds.d.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.33301.log.gz -5 2024-04-26T23:11:02.604 INFO:teuthology.orchestra.run.smithi103.stderr: --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-26T23:11:02.604 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68927.log 2024-04-26T23:11:02.604 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41043.log.gz 2024-04-26T23:11:02.604 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38893.log 2024-04-26T23:11:02.605 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9%/var/log/ceph/ceph-client.admin.68927.log: -- replaced with /var/log/ceph/ceph-client.admin.48625.log.gz 2024-04-26T23:11:02.605 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54861.log 2024-04-26T23:11:02.605 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68927.log.gz 2024-04-26T23:11:02.606 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38893.log: 87.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.38893.log.gz -5 2024-04-26T23:11:02.606 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38481.log 2024-04-26T23:11:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47446.log 2024-04-26T23:11:02.608 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54861.log.gz 2024-04-26T23:11:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69196.log 2024-04-26T23:11:02.608 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38481.log.gz 2024-04-26T23:11:02.610 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45631.log 2024-04-26T23:11:02.610 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47446.log.gz 2024-04-26T23:11:02.611 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.69196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45757.log 2024-04-26T23:11:02.612 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69196.log.gz 2024-04-26T23:11:02.612 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63463.log 2024-04-26T23:11:02.612 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45631.log.gz 2024-04-26T23:11:02.613 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47192.log 2024-04-26T23:11:02.613 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45757.log.gz 2024-04-26T23:11:02.614 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39934.log 2024-04-26T23:11:02.614 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63463.log.gz 2024-04-26T23:11:02.615 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58481.log 2024-04-26T23:11:02.615 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47192.log.gz 2024-04-26T23:11:02.616 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72922.log 2024-04-26T23:11:02.616 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39934.log.gz 2024-04-26T23:11:02.617 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66703.log 2024-04-26T23:11:02.617 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58481.log.gz 2024-04-26T23:11:02.618 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44818.log 2024-04-26T23:11:02.619 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72922.log.gz 2024-04-26T23:11:02.619 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55888.log 2024-04-26T23:11:02.619 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66703.log.gz 2024-04-26T23:11:02.620 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46087.log 2024-04-26T23:11:02.621 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44818.log.gz 2024-04-26T23:11:02.621 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38666.log 2024-04-26T23:11:02.622 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55888.log.gz 2024-04-26T23:11:02.623 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45013.log 2024-04-26T23:11:02.623 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46087.log.gz 2024-04-26T23:11:02.624 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54252.log 2024-04-26T23:11:02.624 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38666.log.gz 2024-04-26T23:11:02.625 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45013.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45013.log.gz 2024-04-26T23:11:02.625 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69748.log 2024-04-26T23:11:02.626 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68597.log 2024-04-26T23:11:02.627 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54252.log.gz 2024-04-26T23:11:02.627 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.69748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.69748.log.gz 2024-04-26T23:11:02.627 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52712.log 2024-04-26T23:11:02.629 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44692.log 89.3% 2024-04-26T23:11:02.629 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68597.log.gz 2024-04-26T23:11:02.630 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42524.log 2024-04-26T23:11:02.630 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52712.log.gz 2024-04-26T23:11:02.631 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46633.log 2024-04-26T23:11:02.631 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44692.log.gz 2024-04-26T23:11:02.632 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72879.log 2024-04-26T23:11:02.632 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42524.log.gz 2024-04-26T23:11:02.633 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65796.log 2024-04-26T23:11:02.634 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46633.log.gz 2024-04-26T23:11:02.634 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45168.log 2024-04-26T23:11:02.635 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72879.log.gz 2024-04-26T23:11:02.636 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.65796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56925.log 2024-04-26T23:11:02.636 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.65796.log.gz 2024-04-26T23:11:02.637 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71805.log 2024-04-26T23:11:02.637 INFO:teuthology.orchestra.run.smithi040.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45168.log.gz 2024-04-26T23:11:02.638 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73144.log 2024-04-26T23:11:02.638 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56925.log.gz 2024-04-26T23:11:02.639 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69406.log 2024-04-26T23:11:02.639 INFO:teuthology.orchestra.run.smithi040.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.71805.log.gz 2024-04-26T23:11:02.640 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45373.log 2024-04-26T23:11:02.640 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73144.log.gz 2024-04-26T23:11:02.641 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.69406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53445.log 2024-04-26T23:11:02.641 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69406.log.gz 2024-04-26T23:11:02.642 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45373.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45373.log.gz 2024-04-26T23:11:02.642 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.74716.log 2024-04-26T23:11:02.643 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47150.log 2024-04-26T23:11:02.644 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53445.log.gz 2024-04-26T23:11:02.644 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43152.log 2024-04-26T23:11:02.645 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74716.log.gz 2024-04-26T23:11:02.646 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51914.log 2024-04-26T23:11:02.646 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47150.log.gz 2024-04-26T23:11:02.647 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.43152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45074.log 2024-04-26T23:11:02.647 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43152.log.gz 2024-04-26T23:11:02.648 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48067.log 2024-04-26T23:11:02.648 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51914.log.gz 2024-04-26T23:11:02.649 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45074.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45074.log.gz 2024-04-26T23:11:02.649 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57917.log 2024-04-26T23:11:02.650 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51788.log 2024-04-26T23:11:02.651 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48067.log.gz 2024-04-26T23:11:02.652 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.57917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67512.log 2024-04-26T23:11:02.652 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57917.log.gz 2024-04-26T23:11:02.653 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42422.log 2024-04-26T23:11:02.653 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51788.log.gz 2024-04-26T23:11:02.654 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65862.log 2024-04-26T23:11:02.654 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67512.log.gz 2024-04-26T23:11:02.655 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51640.log 2024-04-26T23:11:02.655 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42422.log.gz 2024-04-26T23:11:02.656 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.65862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71337.log 2024-04-26T23:11:02.657 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mon.b.log: 90.6% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-26T23:11:02.657 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.65862.log.gz 2024-04-26T23:11:02.657 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47731.log 2024-04-26T23:11:02.657 INFO:teuthology.orchestra.run.smithi040.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51640.log.gz 2024-04-26T23:11:02.658 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47022.log 2024-04-26T23:11:02.658 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71337.log.gz 2024-04-26T23:11:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58827.log 2024-04-26T23:11:02.660 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47731.log.gz 2024-04-26T23:11:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47022.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.47022.log.gz -5 2024-04-26T23:11:02.661 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.75052.log 2024-04-26T23:11:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68376.log 2024-04-26T23:11:02.662 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58827.log.gz 2024-04-26T23:11:02.663 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.75052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66829.log 2024-04-26T23:11:02.663 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75052.log.gz 2024-04-26T23:11:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74267.log 2024-04-26T23:11:02.664 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68376.log.gz 2024-04-26T23:11:02.665 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55463.log 2024-04-26T23:11:02.665 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66829.log.gz 2024-04-26T23:11:02.666 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72285.log 2024-04-26T23:11:02.667 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74267.log.gz 2024-04-26T23:11:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55463.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55463.log.gz 2024-04-26T23:11:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46003.log 2024-04-26T23:11:02.668 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41316.log 2024-04-26T23:11:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46003.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.72285.log.gz 2024-04-26T23:11:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51704.log 2024-04-26T23:11:02.669 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46003.log.gz 2024-04-26T23:11:02.670 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38807.log 2024-04-26T23:11:02.670 INFO:teuthology.orchestra.run.smithi040.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41316.log.gz 2024-04-26T23:11:02.671 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51704.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51704.log.gz 2024-04-26T23:11:02.672 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62371.log 2024-04-26T23:11:02.673 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44398.log 2024-04-26T23:11:02.673 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38807.log.gz 2024-04-26T23:11:02.674 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45415.log 2024-04-26T23:11:02.674 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62371.log.gz 2024-04-26T23:11:02.675 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63590.log 2024-04-26T23:11:02.675 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44398.log.gz 2024-04-26T23:11:02.676 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57954.log 2024-04-26T23:11:02.676 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45415.log.gz 2024-04-26T23:11:02.677 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61139.log 2024-04-26T23:11:02.677 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63590.log.gz 2024-04-26T23:11:02.678 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.57954.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T23:11:02.678 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57954.log.gz 2024-04-26T23:11:02.679 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.61139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66787.log 2024-04-26T23:11:02.680 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.61139.log.gz 2024-04-26T23:11:02.680 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68843.log 2024-04-26T23:11:02.681 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59950.log 2024-04-26T23:11:02.681 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66787.log.gz 2024-04-26T23:11:02.682 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44650.log 2024-04-26T23:11:02.682 INFO:teuthology.orchestra.run.smithi040.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T23:11:02.682 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68843.log.gz 2024-04-26T23:11:02.683 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52298.log 2024-04-26T23:11:02.683 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59950.log.gz 2024-04-26T23:11:02.684 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71073.log 88.6% 2024-04-26T23:11:02.684 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44650.log.gz 2024-04-26T23:11:02.685 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.52298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53371.log 2024-04-26T23:11:02.685 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.52298.log.gz 2024-04-26T23:11:02.686 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68163.log 2024-04-26T23:11:02.687 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53371.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.71073.log.gz 2024-04-26T23:11:02.687 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74674.log 2024-04-26T23:11:02.688 INFO:teuthology.orchestra.run.smithi040.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.53371.log.gz 2024-04-26T23:11:02.688 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62913.log 2024-04-26T23:11:02.689 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68163.log.gz 2024-04-26T23:11:02.690 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46591.log 2024-04-26T23:11:02.690 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74674.log.gz 2024-04-26T23:11:02.691 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73406.log 2024-04-26T23:11:02.691 INFO:teuthology.orchestra.run.smithi040.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62913.log.gz 2024-04-26T23:11:02.692 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50062.log 2024-04-26T23:11:02.692 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46591.log.gz 2024-04-26T23:11:02.693 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71547.log 2024-04-26T23:11:02.693 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.73406.log.gz 2024-04-26T23:11:02.694 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43936.log 2024-04-26T23:11:02.694 INFO:teuthology.orchestra.run.smithi040.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50062.log.gz 2024-04-26T23:11:02.695 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71631.log 2024-04-26T23:11:02.695 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71547.log.gz 2024-04-26T23:11:02.696 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61926.log 2024-04-26T23:11:02.696 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43936.log.gz 2024-04-26T23:11:02.697 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46900.log 2024-04-26T23:11:02.697 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71631.log.gz 2024-04-26T23:11:02.698 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.61926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52294.log 2024-04-26T23:11:02.698 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.61926.log.gz 2024-04-26T23:11:02.699 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46900.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46900.log.gzgzip 2024-04-26T23:11:02.699 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54777.log 2024-04-26T23:11:02.700 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.52294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45673.log 2024-04-26T23:11:02.700 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.52294.log.gz 2024-04-26T23:11:02.701 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41374.log 2024-04-26T23:11:02.702 INFO:teuthology.orchestra.run.smithi040.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54777.log.gz 2024-04-26T23:11:02.702 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54693.log 2024-04-26T23:11:02.703 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45673.log.gz 2024-04-26T23:11:02.703 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66186.log 2024-04-26T23:11:02.704 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41374.log.gz 2024-04-26T23:11:02.705 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54693.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54693.log.gz 2024-04-26T23:11:02.705 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-26T23:11:02.706 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60173.log 2024-04-26T23:11:02.706 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66186.log.gz 2024-04-26T23:11:02.707 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40448.log 2024-04-26T23:11:02.708 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48109.log 2024-04-26T23:11:02.709 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60173.log.gz 2024-04-26T23:11:02.709 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40448.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40448.log.gz 2024-04-26T23:11:02.709 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38710.log 2024-04-26T23:11:02.710 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60986.log 2024-04-26T23:11:02.710 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48109.log.gz 2024-04-26T23:11:02.711 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38710.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38710.log.gz 2024-04-26T23:11:02.712 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69015.log 2024-04-26T23:11:02.712 INFO:teuthology.orchestra.run.smithi103.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-26T23:11:02.713 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50782.log 2024-04-26T23:11:02.713 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60986.log.gz 2024-04-26T23:11:02.713 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.69015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68758.log 2024-04-26T23:11:02.713 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69015.log.gz 2024-04-26T23:11:02.714 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66661.log 2024-04-26T23:11:02.714 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50782.log.gz 2024-04-26T23:11:02.716 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50534.log 2024-04-26T23:11:02.716 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68758.log.gz 2024-04-26T23:11:02.716 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53618.log 2024-04-26T23:11:02.717 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66661.log.gz 2024-04-26T23:11:02.718 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40490.log 2024-04-26T23:11:02.718 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50534.log.gz 2024-04-26T23:11:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74968.log 2024-04-26T23:11:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40490.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.53618.log.gz 2024-04-26T23:11:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55803.log 2024-04-26T23:11:02.720 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40490.log.gz 2024-04-26T23:11:02.721 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65600.log 2024-04-26T23:11:02.721 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74968.log.gz 2024-04-26T23:11:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59453.log 2024-04-26T23:11:02.722 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55803.log.gz 2024-04-26T23:11:02.723 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.65600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67386.log 2024-04-26T23:11:02.723 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.65600.log.gz 2024-04-26T23:11:02.724 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61205.log 2024-04-26T23:11:02.724 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59453.log.gz 2024-04-26T23:11:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67386.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67386.log.gz 2024-04-26T23:11:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69066.log 2024-04-26T23:11:02.726 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.61205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55974.log 2024-04-26T23:11:02.727 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.69066.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.61205.log.gz 2024-04-26T23:11:02.727 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67834.log 2024-04-26T23:11:02.727 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69066.log.gz 2024-04-26T23:11:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41389.log 2024-04-26T23:11:02.728 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55974.log.gz 2024-04-26T23:11:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73060.log 2024-04-26T23:11:02.730 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67834.log.gz 2024-04-26T23:11:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42044.log 2024-04-26T23:11:02.730 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41389.log.gz 2024-04-26T23:11:02.731 INFO:teuthology.orchestra.run.smithi103.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-26T23:11:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74224.log 2024-04-26T23:11:02.732 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73060.log.gz 2024-04-26T23:11:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65674.log 2024-04-26T23:11:02.733 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42044.log.gz 2024-04-26T23:11:02.734 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74182.log 2024-04-26T23:11:02.734 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74224.log.gz 2024-04-26T23:11:02.734 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.65674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42254.log 2024-04-26T23:11:02.735 INFO:teuthology.orchestra.run.smithi040.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.65674.log.gz 2024-04-26T23:11:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63877.log 2024-04-26T23:11:02.736 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74182.log.gz 2024-04-26T23:11:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44104.log 2024-04-26T23:11:02.737 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42254.log.gz 2024-04-26T23:11:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40244.log 2024-04-26T23:11:02.738 INFO:teuthology.orchestra.run.smithi040.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63877.log.gz 2024-04-26T23:11:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41045.log 2024-04-26T23:11:02.740 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44104.log.gz 2024-04-26T23:11:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45331.log 2024-04-26T23:11:02.740 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40244.log.gz 2024-04-26T23:11:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40370.log 2024-04-26T23:11:02.741 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41045.log.gz 2024-04-26T23:11:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55337.log 2024-04-26T23:11:02.742 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45331.log.gz 2024-04-26T23:11:02.743 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67774.log 2024-04-26T23:11:02.743 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40370.log.gz 2024-04-26T23:11:02.744 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44566.log 2024-04-26T23:11:02.744 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55337.log.gz 2024-04-26T23:11:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53131.log 2024-04-26T23:11:02.746 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67774.log.gz 2024-04-26T23:11:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68460.log 2024-04-26T23:11:02.746 INFO:teuthology.orchestra.run.smithi040.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44566.log.gz 2024-04-26T23:11:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52646.log 2024-04-26T23:11:02.748 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.53131.log.gz 2024-04-26T23:11:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74446.log 2024-04-26T23:11:02.749 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68460.log.gz 2024-04-26T23:11:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40700.log 2024-04-26T23:11:02.750 INFO:teuthology.orchestra.run.smithi040.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.52646.log.gz 2024-04-26T23:11:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53886.log 2024-04-26T23:11:02.750 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74446.log.gz 2024-04-26T23:11:02.751 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51417.log 2024-04-26T23:11:02.752 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40700.log.gz 2024-04-26T23:11:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59704.log 2024-04-26T23:11:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68800.log 2024-04-26T23:11:02.754 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.53886.log.gz 2024-04-26T23:11:02.754 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51417.log.gz 2024-04-26T23:11:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59704.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T23:11:02.755 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59704.log.gz 2024-04-26T23:11:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70030.log 2024-04-26T23:11:02.756 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68800.log.gz 2024-04-26T23:11:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57843.log 2024-04-26T23:11:02.758 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.70030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44776.log 2024-04-26T23:11:02.758 INFO:teuthology.orchestra.run.smithi040.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.70030.log.gz 2024-04-26T23:11:02.759 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.57843.log: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gzgzip 2024-04-26T23:11:02.759 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.71271.log 2024-04-26T23:11:02.759 INFO:teuthology.orchestra.run.smithi040.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.57843.log.gz 2024-04-26T23:11:02.760 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63302.log 2024-04-26T23:11:02.760 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44776.log.gz 2024-04-26T23:11:02.761 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70877.log 2024-04-26T23:11:02.762 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.71271.log.gz 2024-04-26T23:11:02.762 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60857.log 2024-04-26T23:11:02.763 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63302.log.gz 2024-04-26T23:11:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.70877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41705.log 2024-04-26T23:11:02.764 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.70877.log.gz 2024-04-26T23:11:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.60857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.60857.log.gz 2024-04-26T23:11:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56751.log 2024-04-26T23:11:02.765 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44440.log 2024-04-26T23:11:02.766 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.56751.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41705.log.gz 2024-04-26T23:11:02.767 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50950.log 2024-04-26T23:11:02.767 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.56751.log.gz 2024-04-26T23:11:02.768 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54557.log 2024-04-26T23:11:02.768 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44440.log.gz 2024-04-26T23:11:02.769 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47276.log 2024-04-26T23:11:02.769 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50950.log.gz 2024-04-26T23:11:02.770 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72585.log 2024-04-26T23:11:02.770 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54557.log.gz 2024-04-26T23:11:02.771 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47689.log 2024-04-26T23:11:02.771 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47276.log.gz 2024-04-26T23:11:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67165.log 2024-04-26T23:11:02.772 INFO:teuthology.orchestra.run.smithi040.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.72585.log.gz 2024-04-26T23:11:02.773 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47689.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47689.log.gz 2024-04-26T23:11:02.773 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44146.log 2024-04-26T23:11:02.774 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50240.log 2024-04-26T23:11:02.775 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67165.log.gz 2024-04-26T23:11:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60034.log 2024-04-26T23:11:02.775 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44146.log.gz 2024-04-26T23:11:02.777 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63831.log 2024-04-26T23:11:02.777 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50240.log.gz 2024-04-26T23:11:02.778 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67081.log 2024-04-26T23:11:02.778 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60034.log.gz 2024-04-26T23:11:02.779 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61754.log 2024-04-26T23:11:02.779 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63831.log.gz 2024-04-26T23:11:02.780 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67081.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67081.log.gz 2024-04-26T23:11:02.780 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38935.log 2024-04-26T23:11:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.61754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62455.log 2024-04-26T23:11:02.781 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61754.log.gz 2024-04-26T23:11:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42380.log 2024-04-26T23:11:02.782 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38935.log.gz 2024-04-26T23:11:02.783 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66595.log 2024-04-26T23:11:02.783 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62455.log.gz 2024-04-26T23:11:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42814.log 2024-04-26T23:11:02.784 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42380.log.gz 2024-04-26T23:11:02.785 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39211.log 2024-04-26T23:11:02.786 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.66595.log.gz 2024-04-26T23:11:02.786 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42814.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42814.log.gz 2024-04-26T23:11:02.786 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54945.log 2024-04-26T23:11:02.787 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73186.log 2024-04-26T23:11:02.788 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54945.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39211.log.gz 2024-04-26T23:11:02.789 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54945.log.gz 2024-04-26T23:11:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52427.log 2024-04-26T23:11:02.790 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73186.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.73186.log.gz 2024-04-26T23:11:02.790 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51129.log 2024-04-26T23:11:02.791 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43978.log 2024-04-26T23:11:02.792 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52427.log.gz 2024-04-26T23:11:02.792 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57603.log 2024-04-26T23:11:02.792 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51129.log.gz 2024-04-26T23:11:02.793 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61622.log 2024-04-26T23:11:02.793 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43978.log.gz 2024-04-26T23:11:02.794 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.57603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68250.log 2024-04-26T23:11:02.794 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.57603.log.gz 2024-04-26T23:11:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.61622.log: -- /var/log/ceph/ceph-client.admin.49260.log 2024-04-26T23:11:02.796 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60853.log 2024-04-26T23:11:02.796 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.61622.log.gz 2024-04-26T23:11:02.797 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68250.log.gz 2024-04-26T23:11:02.798 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56409.log 2024-04-26T23:11:02.798 INFO:teuthology.orchestra.run.smithi040.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.49260.log.gz 2024-04-26T23:11:02.798 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.60853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57033.log 2024-04-26T23:11:02.798 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.60853.log.gz 2024-04-26T23:11:02.799 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72369.log 2024-04-26T23:11:02.800 INFO:teuthology.orchestra.run.smithi040.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56409.log.gz 2024-04-26T23:11:02.800 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.57033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49716.log 2024-04-26T23:11:02.801 INFO:teuthology.orchestra.run.smithi040.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.57033.log.gz 2024-04-26T23:11:02.801 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46381.log 2024-04-26T23:11:02.802 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49716.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.72369.log.gz 2024-04-26T23:11:02.802 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63676.log 2024-04-26T23:11:02.803 INFO:teuthology.orchestra.run.smithi040.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.49716.log.gz 2024-04-26T23:11:02.803 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74800.log 2024-04-26T23:11:02.804 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46381.log.gz 2024-04-26T23:11:02.804 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58296.log 2024-04-26T23:11:02.805 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63676.log.gz 2024-04-26T23:11:02.805 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55379.log 2024-04-26T23:11:02.806 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74800.log.gz 2024-04-26T23:11:02.806 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58296.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-26T23:11:02.807 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.58296.log.gz 2024-04-26T23:11:02.807 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73812.log 2024-04-26T23:11:02.808 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55379.log.gz 2024-04-26T23:11:02.809 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47899.log 2024-04-26T23:11:02.810 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73812.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73812.log.gz 2024-04-26T23:11:02.810 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40328.log 2024-04-26T23:11:02.811 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52514.log 2024-04-26T23:11:02.811 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47899.log.gz 2024-04-26T23:11:02.812 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40328.log: gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.40532.log -- replaced with /var/log/ceph/ceph-client.admin.40328.log.gz 2024-04-26T23:11:02.812 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-26T23:11:02.813 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74926.log 2024-04-26T23:11:02.813 INFO:teuthology.orchestra.run.smithi040.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.52514.log.gz 2024-04-26T23:11:02.814 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71205.log 2024-04-26T23:11:02.814 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40532.log.gz 2024-04-26T23:11:02.815 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69877.log 2024-04-26T23:11:02.815 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74926.log.gz 2024-04-26T23:11:02.816 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55149.log 2024-04-26T23:11:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.69877.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.71205.log.gz 2024-04-26T23:11:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47601.log 2024-04-26T23:11:02.818 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69877.log.gz 2024-04-26T23:11:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51956.log 2024-04-26T23:11:02.819 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55149.log.gz 2024-04-26T23:11:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61861.log 2024-04-26T23:11:02.820 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47601.log.gz 2024-04-26T23:11:02.821 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61556.log 2024-04-26T23:11:02.821 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51956.log.gz 2024-04-26T23:11:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.61861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42212.log 2024-04-26T23:11:02.822 INFO:teuthology.orchestra.run.smithi040.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.61861.log.gz 2024-04-26T23:11:02.823 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.61556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51746.log 2024-04-26T23:11:02.824 INFO:teuthology.orchestra.run.smithi040.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.61556.log.gz 2024-04-26T23:11:02.824 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41763.log 2024-04-26T23:11:02.824 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42212.log.gz 2024-04-26T23:11:02.825 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61991.log 2024-04-26T23:11:02.825 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51746.log.gz 2024-04-26T23:11:02.827 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55761.log 2024-04-26T23:11:02.827 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41763.log.gz 2024-04-26T23:11:02.828 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.61991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51830.log 2024-04-26T23:11:02.828 INFO:teuthology.orchestra.run.smithi040.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.61991.log.gz 2024-04-26T23:11:02.829 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68971.log 2024-04-26T23:11:02.829 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55761.log.gz 2024-04-26T23:11:02.830 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51830.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51830.log.gz 2024-04-26T23:11:02.830 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46171.log 2024-04-26T23:11:02.831 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65173.log 2024-04-26T23:11:02.831 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68971.log.gz 2024-04-26T23:11:02.832 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46171.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46171.log.gz 2024-04-26T23:11:02.832 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48409.log 2024-04-26T23:11:02.833 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.65173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39103.log 2024-04-26T23:11:02.834 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.48409.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.65173.log.gz 2024-04-26T23:11:02.834 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48409.log.gz 2024-04-26T23:11:02.834 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73990.log 2024-04-26T23:11:02.835 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39103.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39103.log.gz 2024-04-26T23:11:02.835 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73700.log 2024-04-26T23:11:02.837 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44482.log 2024-04-26T23:11:02.837 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73990.log.gz 2024-04-26T23:11:02.838 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73700.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73700.log.gz 2024-04-26T23:11:02.838 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64819.log 2024-04-26T23:11:02.839 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39680.log 2024-04-26T23:11:02.839 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44482.log.gz 2024-04-26T23:11:02.840 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.64819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62413.log 2024-04-26T23:11:02.840 INFO:teuthology.orchestra.run.smithi040.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.64819.log.gz 2024-04-26T23:11:02.841 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69150.log 2024-04-26T23:11:02.842 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54515.log 2024-04-26T23:11:02.842 INFO:teuthology.orchestra.run.smithi040.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62413.log.gz 2024-04-26T23:11:02.843 INFO:teuthology.orchestra.run.smithi040.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.39680.log.gz 2024-04-26T23:11:02.843 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.69150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69364.log 2024-04-26T23:11:02.844 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69150.log.gz 2024-04-26T23:11:02.844 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64885.log 2024-04-26T23:11:02.844 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54515.log.gz 2024-04-26T23:11:02.845 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.69364.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-26T23:11:02.845 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69364.log.gz 2024-04-26T23:11:02.846 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.64885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62539.log 2024-04-26T23:11:02.846 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64885.log.gz 2024-04-26T23:11:02.847 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64003.log 2024-04-26T23:11:02.849 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51034.log 2024-04-26T23:11:02.849 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62539.log.gz 2024-04-26T23:11:02.850 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.64003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51998.log 2024-04-26T23:11:02.850 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64003.log.gz 2024-04-26T23:11:02.851 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47941.log 2024-04-26T23:11:02.852 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51034.log.gz 2024-04-26T23:11:02.852 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70180.log 2024-04-26T23:11:02.852 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51998.log.gz 2024-04-26T23:11:02.854 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40574.log 2024-04-26T23:11:02.854 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47941.log.gz 2024-04-26T23:11:02.855 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.70180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50408.log 2024-04-26T23:11:02.855 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70180.log.gz 2024-04-26T23:11:02.856 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60078.log 2024-04-26T23:11:02.856 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40574.log.gz 2024-04-26T23:11:02.857 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49511.log 2024-04-26T23:11:02.857 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50408.log.gz 2024-04-26T23:11:02.858 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57165.log 2024-04-26T23:11:02.859 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60078.log.gz 2024-04-26T23:11:02.860 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51459.log 2024-04-26T23:11:02.860 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49511.log.gz 2024-04-26T23:11:02.861 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.57165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49978.log 2024-04-26T23:11:02.861 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57165.log.gz 2024-04-26T23:11:02.862 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47857.log 2024-04-26T23:11:02.862 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51459.log.gz 2024-04-26T23:11:02.863 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49978.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49978.log.gz 2024-04-26T23:11:02.863 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69922.log 2024-04-26T23:11:02.864 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50618.log 2024-04-26T23:11:02.865 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47857.log.gz 2024-04-26T23:11:02.866 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.69922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39019.log 2024-04-26T23:11:02.866 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69922.log.gz 2024-04-26T23:11:02.867 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40160.log 2024-04-26T23:11:02.867 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50618.log.gz 2024-04-26T23:11:02.868 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66060.log 2024-04-26T23:11:02.869 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39019.log.gz 2024-04-26T23:11:02.870 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40102.log 2024-04-26T23:11:02.870 INFO:teuthology.orchestra.run.smithi040.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40160.log.gz 2024-04-26T23:11:02.871 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43894.log 2024-04-26T23:11:02.871 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66060.log.gz 2024-04-26T23:11:02.872 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44020.log 2024-04-26T23:11:02.872 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40102.log.gz 2024-04-26T23:11:02.873 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46549.log 2024-04-26T23:11:02.873 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43894.log.gz 2024-04-26T23:11:02.875 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44020.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-26T23:11:02.875 INFO:teuthology.orchestra.run.smithi040.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44020.log.gz 2024-04-26T23:11:02.876 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62955.log 2024-04-26T23:11:02.877 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46549.log.gz 2024-04-26T23:11:02.877 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63505.log 2024-04-26T23:11:02.878 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69556.log 2024-04-26T23:11:02.879 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62955.log.gz 2024-04-26T23:11:02.880 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65730.log 2024-04-26T23:11:02.880 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63505.log.gz 2024-04-26T23:11:02.881 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.69556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41029.log 2024-04-26T23:11:02.881 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69556.log.gz 2024-04-26T23:11:02.882 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.65730.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-26T23:11:02.884 INFO:teuthology.orchestra.run.smithi040.stderr: 91.1%/var/log/ceph/ceph-client.admin.41029.log: -- replaced with /var/log/ceph/ceph-client.admin.65730.log.gz 2024-04-26T23:11:02.884 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41620.log 2024-04-26T23:11:02.884 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41029.log.gz 2024-04-26T23:11:02.885 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50450.log 2024-04-26T23:11:02.886 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72135.log 2024-04-26T23:11:02.887 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41620.log.gz 2024-04-26T23:11:02.888 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53482.log 2024-04-26T23:11:02.888 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50450.log.gz 2024-04-26T23:11:02.889 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60122.log 2024-04-26T23:11:02.889 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72135.log.gz 2024-04-26T23:11:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58230.log 2024-04-26T23:11:02.891 INFO:teuthology.orchestra.run.smithi040.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.53482.log.gz 2024-04-26T23:11:02.892 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46980.log 2024-04-26T23:11:02.892 INFO:teuthology.orchestra.run.smithi040.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60122.log.gz 2024-04-26T23:11:02.894 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58230.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.58230.log.gz 2024-04-26T23:11:02.894 INFO:teuthology.orchestra.run.smithi103.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-26T23:11:02.894 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46980.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46980.log.gz 2024-04-26T23:11:02.895 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67960.log 2024-04-26T23:11:02.895 INFO:teuthology.orchestra.run.smithi040.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-26T23:11:02.896 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44524.log 2024-04-26T23:11:02.897 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49642.log 2024-04-26T23:11:02.898 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67960.log.gz 2024-04-26T23:11:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54473.log 2024-04-26T23:11:02.899 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44524.log.gz 2024-04-26T23:11:02.900 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63039.log 2024-04-26T23:11:02.900 INFO:teuthology.orchestra.run.smithi040.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.49642.log.gz 2024-04-26T23:11:02.901 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59285.log 2024-04-26T23:11:02.902 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54473.log.gz 2024-04-26T23:11:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38622.log 2024-04-26T23:11:02.903 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63039.log.gz 2024-04-26T23:11:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59285.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59285.log.gz 2024-04-26T23:11:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.43593.log 2024-04-26T23:11:02.906 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38622.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49595.log 2024-04-26T23:11:02.906 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38622.log.gz 2024-04-26T23:11:02.907 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.2.43593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67039.log 2024-04-26T23:11:02.907 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.2.43593.log.gz 2024-04-26T23:11:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64467.log 2024-04-26T23:11:02.910 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67039.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.49595.log.gz 2024-04-26T23:11:02.910 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63720.log 2024-04-26T23:11:02.910 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67039.log.gz 2024-04-26T23:11:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.64467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64467.log.gz 2024-04-26T23:11:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-26T23:11:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63789.log 2024-04-26T23:11:02.912 INFO:teuthology.orchestra.run.smithi040.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63720.log.gz 2024-04-26T23:11:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41047.log 2024-04-26T23:11:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47983.log 2024-04-26T23:11:02.915 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63789.log.gz 2024-04-26T23:11:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73532.log 2024-04-26T23:11:02.916 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41047.log.gz 2024-04-26T23:11:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47983.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47983.log.gz 2024-04-26T23:11:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51332.log 2024-04-26T23:11:02.919 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73532.log.gz 2024-04-26T23:11:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69490.log 2024-04-26T23:11:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57471.log 2024-04-26T23:11:02.922 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51332.log.gz 2024-04-26T23:11:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.69490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48151.log 2024-04-26T23:11:02.924 INFO:teuthology.orchestra.run.smithi040.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.69490.log.gz 2024-04-26T23:11:02.924 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.57471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66463.log 2024-04-26T23:11:02.925 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48151.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.57471.log.gz 2024-04-26T23:11:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69108.log 2024-04-26T23:11:02.926 INFO:teuthology.orchestra.run.smithi040.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.48151.log.gz 2024-04-26T23:11:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66463.log: gzip -5 --verbose -- 91.1% -- replaced with /var/log/ceph/ceph-client.admin.66463.log.gz 2024-04-26T23:11:02.928 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.70600.log 2024-04-26T23:11:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.69108.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69108.log.gz 2024-04-26T23:11:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72753.log 2024-04-26T23:11:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.70600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49790.log 2024-04-26T23:11:02.931 INFO:teuthology.orchestra.run.smithi040.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.70600.log.gz 2024-04-26T23:11:02.932 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72753.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72753.log.gz 2024-04-26T23:11:02.933 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48691.log 2024-04-26T23:11:02.934 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49790.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49790.log.gz 2024-04-26T23:11:02.934 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63961.log 2024-04-26T23:11:02.935 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50866.log 2024-04-26T23:11:02.936 INFO:teuthology.orchestra.run.smithi040.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.48691.log.gz 2024-04-26T23:11:02.937 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63961.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40826.log 2024-04-26T23:11:02.937 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63961.log.gz 2024-04-26T23:11:02.938 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50866.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50866.log.gz 2024-04-26T23:11:02.939 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46507.log 2024-04-26T23:11:02.940 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54389.log 2024-04-26T23:11:02.940 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40826.log.gz 2024-04-26T23:11:02.941 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71589.log 2024-04-26T23:11:02.941 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46507.log.gz 2024-04-26T23:11:02.943 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54389.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54389.log.gz 2024-04-26T23:11:02.943 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59411.log 2024-04-26T23:11:02.943 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71589.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71589.log.gz 2024-04-26T23:11:02.944 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41960.log 2024-04-26T23:11:02.945 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61796.log 2024-04-26T23:11:02.945 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59411.log.gz 2024-04-26T23:11:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68208.log 2024-04-26T23:11:02.947 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41960.log.gz 2024-04-26T23:11:02.948 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.61796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47234.log 2024-04-26T23:11:02.949 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.61796.log.gz 2024-04-26T23:11:02.950 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68208.log: gzip -5 --verbose -- 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68208.log.gz 2024-04-26T23:11:02.950 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.70803.log 2024-04-26T23:11:02.950 INFO:teuthology.orchestra.run.smithi103.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-26T23:11:02.951 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47234.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55029.log 2024-04-26T23:11:02.951 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47234.log.gz 2024-04-26T23:11:02.953 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.70803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58743.log 2024-04-26T23:11:02.953 INFO:teuthology.orchestra.run.smithi040.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.70803.log.gz 2024-04-26T23:11:02.954 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55029.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55029.log.gz 2024-04-26T23:11:02.954 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63548.log 2024-04-26T23:11:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58743.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57806.log 2024-04-26T23:11:02.956 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58743.log.gz 2024-04-26T23:11:02.957 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74530.log 2024-04-26T23:11:02.957 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63548.log.gz 2024-04-26T23:11:02.958 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.57806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38761.log 2024-04-26T23:11:02.958 INFO:teuthology.orchestra.run.smithi040.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.57806.log.gz 2024-04-26T23:11:02.960 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74530.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74530.log.gz 2024-04-26T23:11:02.960 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44860.log 2024-04-26T23:11:02.961 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38761.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38761.log.gz 2024-04-26T23:11:02.962 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41714.log 2024-04-26T23:11:02.963 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38851.log 2024-04-26T23:11:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44860.log.gz 2024-04-26T23:11:02.965 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52472.log 2024-04-26T23:11:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41714.log.gz 2024-04-26T23:11:02.966 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74758.log 2024-04-26T23:11:02.967 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38851.log: 89.2%/var/log/ceph/ceph-client.admin.52472.log: -- replaced with /var/log/ceph/ceph-client.admin.38851.log.gz 2024-04-26T23:11:02.968 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52472.log.gz 2024-04-26T23:11:02.968 INFO:teuthology.orchestra.run.smithi040.stderr: -- /var/log/ceph/ceph-client.admin.49753.log 2024-04-26T23:11:02.969 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74758.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74758.log.gz 2024-04-26T23:11:02.969 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73490.log 2024-04-26T23:11:02.970 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51547.log 2024-04-26T23:11:02.971 INFO:teuthology.orchestra.run.smithi040.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49753.log.gz 2024-04-26T23:11:02.972 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73448.log 2024-04-26T23:11:02.972 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73490.log.gz 2024-04-26T23:11:02.973 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51547.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51547.log.gz 2024-04-26T23:11:02.974 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49827.log 2024-04-26T23:11:02.974 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73448.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73448.log.gz 2024-04-26T23:11:02.975 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75135.log 2024-04-26T23:11:02.977 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60515.log 2024-04-26T23:11:02.977 INFO:teuthology.orchestra.run.smithi040.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.49827.log.gz 2024-04-26T23:11:02.978 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.75135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44062.log 2024-04-26T23:11:02.978 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75135.log.gz 2024-04-26T23:11:02.979 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60515.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60515.log.gz 2024-04-26T23:11:02.979 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46213.log 2024-04-26T23:11:02.981 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44062.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39145.log 2024-04-26T23:11:02.981 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44062.log.gz 2024-04-26T23:11:02.982 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56967.log 2024-04-26T23:11:02.982 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46213.log.gz 2024-04-26T23:11:02.984 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40658.log 2024-04-26T23:11:02.984 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56967.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.39145.log.gz 2024-04-26T23:11:02.986 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.56967.log.gz 2024-04-26T23:11:02.986 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56153.log 2024-04-26T23:11:02.987 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40658.log.gz 2024-04-26T23:11:02.987 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64471.log 2024-04-26T23:11:02.989 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62869.log 2024-04-26T23:11:02.989 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56153.log.gz 2024-04-26T23:11:02.990 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.64471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64471.log.gz 2024-04-26T23:11:02.991 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74884.log/var/log/ceph/ceph-client.admin.62869.log: 2024-04-26T23:11:02.991 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62869.log.gz 2024-04-26T23:11:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68885.log 2024-04-26T23:11:02.993 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72965.log 2024-04-26T23:11:02.993 INFO:teuthology.orchestra.run.smithi040.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74884.log.gz 2024-04-26T23:11:02.995 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40018.log 2024-04-26T23:11:02.995 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68885.log.gz 2024-04-26T23:11:02.997 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72965.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72965.log.gz 2024-04-26T23:11:02.997 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58911.log 2024-04-26T23:11:02.998 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40018.log.gz 2024-04-26T23:11:02.998 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42296.log 2024-04-26T23:11:03.000 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65239.log 2024-04-26T23:11:03.000 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58911.log.gz 2024-04-26T23:11:03.001 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42296.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42296.log.gz 2024-04-26T23:11:03.002 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38438.log 2024-04-26T23:11:03.003 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.65239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56018.log 2024-04-26T23:11:03.003 INFO:teuthology.orchestra.run.smithi040.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.65239.log.gz 2024-04-26T23:11:03.004 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38438.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26181.log 2024-04-26T23:11:03.005 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38438.log.gz 2024-04-26T23:11:03.006 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60557.log 2024-04-26T23:11:03.006 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56018.log.gz 2024-04-26T23:11:03.007 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.tmp-client.admin.26181.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26181.log.gz 2024-04-26T23:11:03.007 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62749.log 2024-04-26T23:11:03.009 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55930.log 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60557.log.gz 2024-04-26T23:11:03.009 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-26T23:11:03.010 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62749.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62749.log.gz 2024-04-26T23:11:03.010 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71379.log 2024-04-26T23:11:03.011 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63165.log 2024-04-26T23:11:03.012 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55930.log.gz 2024-04-26T23:11:03.013 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44272.log 2024-04-26T23:11:03.013 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71379.log.gz 2024-04-26T23:11:03.014 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63165.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63165.log.gz 2024-04-26T23:11:03.015 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44272.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44272.log.gz 2024-04-26T23:11:03.016 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48757.log 2024-04-26T23:11:03.017 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42086.log 2024-04-26T23:11:03.019 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62193.log 2024-04-26T23:11:03.019 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0%/var/log/ceph/ceph-client.admin.42086.log: -- replaced with /var/log/ceph/ceph-client.admin.48757.log.gz 2024-04-26T23:11:03.021 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42086.log.gz 2024-04-26T23:11:03.021 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59784.log 2024-04-26T23:11:03.022 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66355.log 2024-04-26T23:11:03.022 INFO:teuthology.orchestra.run.smithi040.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62193.log.gz 2024-04-26T23:11:03.023 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59784.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59784.log.gz 2024-04-26T23:11:03.024 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.66355.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.67732.log 2024-04-26T23:11:03.024 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66355.log.gz 2024-04-26T23:11:03.025 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38315.log 2024-04-26T23:11:03.027 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59121.log 2024-04-26T23:11:03.027 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67732.log.gz 2024-04-26T23:11:03.028 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38315.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38315.log.gz 2024-04-26T23:11:03.029 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54431.log 2024-04-26T23:11:03.029 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59121.log.gz 2024-04-26T23:11:03.030 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66270.log 2024-04-26T23:11:03.031 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46297.log 2024-04-26T23:11:03.032 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54431.log.gz 2024-04-26T23:11:03.033 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50020.log 2024-04-26T23:11:03.033 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66270.log.gz 2024-04-26T23:11:03.034 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46297.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46297.log.gz 2024-04-26T23:11:03.035 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64171.log 2024-04-26T23:11:03.036 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50020.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-26T23:11:03.036 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50020.log.gz 2024-04-26T23:11:03.037 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.64171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41049.log 2024-04-26T23:11:03.038 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.64171.log.gz 2024-04-26T23:11:03.039 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62791.log 2024-04-26T23:11:03.040 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41049.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41049.log.gz 2024-04-26T23:11:03.041 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62791.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62791.log.gz 2024-04-26T23:11:03.042 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67876.log 2024-04-26T23:11:03.044 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70096.log 2024-04-26T23:11:03.045 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67876.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67876.log.gz 2024-04-26T23:11:03.046 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71007.log 2024-04-26T23:11:03.047 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.70096.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.70096.log.gz 2024-04-26T23:11:03.048 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.71007.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.72711.log 2024-04-26T23:11:03.049 INFO:teuthology.orchestra.run.smithi040.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.71007.log.gz 2024-04-26T23:11:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67428.log 2024-04-26T23:11:03.051 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72711.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72711.log.gz 2024-04-26T23:11:03.052 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70840.log 2024-04-26T23:11:03.053 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67428.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67428.log.gz 2024-04-26T23:11:03.054 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62581.log 2024-04-26T23:11:03.055 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.70840.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70840.log.gz 2024-04-26T23:11:03.056 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73228.log 2024-04-26T23:11:03.057 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62581.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62581.log.gz 2024-04-26T23:11:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60257.log/var/log/ceph/ceph-client.admin.73228.log: 2024-04-26T23:11:03.059 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73228.log.gz 2024-04-26T23:11:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45253.log 2024-04-26T23:11:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60257.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60257.log.gz 2024-04-26T23:11:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40616.log 2024-04-26T23:11:03.062 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45253.log.gz 2024-04-26T23:11:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38138.log 2024-04-26T23:11:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40616.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40616.log.gz 2024-04-26T23:11:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58076.log 2024-04-26T23:11:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38138.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38138.log.gz 2024-04-26T23:11:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66313.log 2024-04-26T23:11:03.069 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58076.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.58076.log.gz 2024-04-26T23:11:03.070 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66313.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66313.log.gz 2024-04-26T23:11:03.070 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53952.log 2024-04-26T23:11:03.072 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54084.log 2024-04-26T23:11:03.074 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53952.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.53952.log.gz 2024-04-26T23:11:03.075 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41626.log 2024-04-26T23:11:03.075 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54084.log.gz 2024-04-26T23:11:03.076 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69322.log 2024-04-26T23:11:03.077 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41626.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41626.log.gz 2024-04-26T23:11:03.078 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74488.log/var/log/ceph/ceph-client.admin.69322.log: 2024-04-26T23:11:03.079 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69322.log.gz 2024-04-26T23:11:03.080 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68002.log 2024-04-26T23:11:03.082 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74488.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74488.log.gz 2024-04-26T23:11:03.082 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.68002.log: -- /var/log/ceph/ceph-client.admin.52580.log 2024-04-26T23:11:03.083 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68002.log.gz 2024-04-26T23:11:03.084 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.43589.log 2024-04-26T23:11:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47773.log 2024-04-26T23:11:03.086 INFO:teuthology.orchestra.run.smithi040.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.52580.log.gz 2024-04-26T23:11:03.087 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.2.43589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.43589.log.gz 2024-04-26T23:11:03.088 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59163.log 2024-04-26T23:11:03.089 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47773.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47773.log.gz 2024-04-26T23:11:03.090 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.59163.log: /var/log/ceph/ceph-client.admin.68502.log 2024-04-26T23:11:03.091 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59163.log.gz 2024-04-26T23:11:03.092 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72003.log 2024-04-26T23:11:03.094 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68502.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68502.log.gz 2024-04-26T23:11:03.094 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44734.log 2024-04-26T23:11:03.096 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72003.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.72003.log.gz 2024-04-26T23:11:03.097 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41761.log 2024-04-26T23:11:03.098 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44734.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44734.log.gz 2024-04-26T23:11:03.099 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66529.log 2024-04-26T23:11:03.099 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41761.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41761.log.gz 2024-04-26T23:11:03.100 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72669.log 2024-04-26T23:11:03.102 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66529.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.66529.log.gz 2024-04-26T23:11:03.103 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72669.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72669.log.gz 2024-04-26T23:11:03.104 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38977.log 2024-04-26T23:11:03.105 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56199.log 2024-04-26T23:11:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38977.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38977.log.gz 2024-04-26T23:11:03.107 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45799.log 2024-04-26T23:11:03.108 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56199.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56199.log.gz 2024-04-26T23:11:03.109 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60347.log 2024-04-26T23:11:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45799.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45799.log.gz 2024-04-26T23:11:03.111 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51503.log 2024-04-26T23:11:03.112 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60347.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60347.log.gz 2024-04-26T23:11:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51598.log 2024-04-26T23:11:03.115 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51503.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51503.log.gz 2024-04-26T23:11:03.115 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53408.log 2024-04-26T23:11:03.116 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51598.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51598.log.gz 2024-04-26T23:11:03.117 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42128.log 2024-04-26T23:11:03.119 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53408.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.53408.log.gz 2024-04-26T23:11:03.119 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61031.log 2024-04-26T23:11:03.120 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42128.log.gz 2024-04-26T23:11:03.120 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54903.log 2024-04-26T23:11:03.123 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.61031.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61031.log.gz 2024-04-26T23:11:03.123 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42002.log/var/log/ceph/ceph-client.admin.54903.log: 2024-04-26T23:11:03.123 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54903.log.gz 2024-04-26T23:11:03.124 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62235.log 2024-04-26T23:11:03.126 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42002.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42002.log.gz 2024-04-26T23:11:03.127 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50282.log 2024-04-26T23:11:03.127 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62235.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62235.log.gz 2024-04-26T23:11:03.129 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64129.log 2024-04-26T23:11:03.131 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50282.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50282.log.gz 2024-04-26T23:11:03.131 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.64129.log: --verbose -- /var/log/ceph/ceph-client.admin.51872.log 2024-04-26T23:11:03.131 INFO:teuthology.orchestra.run.smithi040.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64129.log.gz 2024-04-26T23:11:03.132 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59866.log 2024-04-26T23:11:03.134 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51872.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51872.log.gz 2024-04-26T23:11:03.135 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.59866.log: /var/log/ceph/ceph-client.admin.63919.log 2024-04-26T23:11:03.135 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59866.log.gz 2024-04-26T23:11:03.136 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73616.log 2024-04-26T23:11:03.138 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63919.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63919.log.gz 2024-04-26T23:11:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.73616.log: -- /var/log/ceph/ceph-client.admin.41050.log 2024-04-26T23:11:03.139 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73616.log.gz 2024-04-26T23:11:03.141 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48405.log 2024-04-26T23:11:03.142 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41050.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41050.log.gz 2024-04-26T23:11:03.143 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.0.48405.log: --verbose -- /var/log/ceph/ceph-client.admin.58162.log 2024-04-26T23:11:03.143 INFO:teuthology.orchestra.run.smithi040.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48405.log.gz 2024-04-26T23:11:03.145 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45919.log 2024-04-26T23:11:03.146 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58162.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.58162.log.gz 2024-04-26T23:11:03.148 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56283.log 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45919.log.gz 2024-04-26T23:11:03.148 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-26T23:11:03.149 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59992.log 2024-04-26T23:11:03.150 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56283.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56283.log.gz 2024-04-26T23:11:03.152 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70534.log 2024-04-26T23:11:03.152 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59992.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59992.log.gz 2024-04-26T23:11:03.155 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.70534.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.70534.log.gz 2024-04-26T23:11:03.155 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55600.log 2024-04-26T23:11:03.157 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56493.log 2024-04-26T23:11:03.158 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55600.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55600.log.gz 2024-04-26T23:11:03.159 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.56493.log: /var/log/ceph/ceph-client.admin.55421.log 2024-04-26T23:11:03.160 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.56493.log.gz 2024-04-26T23:11:03.161 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67123.log 2024-04-26T23:11:03.162 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55421.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55421.log.gz 2024-04-26T23:11:03.164 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67123.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67123.log.gz 2024-04-26T23:11:03.164 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41034.log 2024-04-26T23:11:03.166 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54018.log 2024-04-26T23:11:03.167 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41034.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41034.log.gz 2024-04-26T23:11:03.168 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38538.log 2024-04-26T23:11:03.169 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54018.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54018.log.gz 2024-04-26T23:11:03.170 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50104.log 2024-04-26T23:11:03.171 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38538.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38538.log.gz 2024-04-26T23:11:03.172 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71673.log 2024-04-26T23:11:03.173 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50104.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50104.log.gz 2024-04-26T23:11:03.174 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59233.log 2024-04-26T23:11:03.175 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71673.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.71673.log.gz 2024-04-26T23:11:03.176 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54819.log/var/log/ceph/ceph-client.admin.59233.log: 2024-04-26T23:11:03.177 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59233.log.gz 2024-04-26T23:11:03.178 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59564.log 2024-04-26T23:11:03.179 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54819.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54819.log.gz 2024-04-26T23:11:03.179 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58010.log 2024-04-26T23:11:03.181 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59564.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59564.log.gz 2024-04-26T23:11:03.181 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46129.log 2024-04-26T23:11:03.183 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58010.log: 91.0%/var/log/ceph/ceph-client.admin.46129.log: -- replaced with /var/log/ceph/ceph-client.admin.58010.log.gz 2024-04-26T23:11:03.184 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46129.log.gz 2024-04-26T23:11:03.184 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47108.log 2024-04-26T23:11:03.186 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52040.log 2024-04-26T23:11:03.188 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47108.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47108.log.gz 2024-04-26T23:11:03.188 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43768.log 2024-04-26T23:11:03.189 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52040.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.52040.log.gz 2024-04-26T23:11:03.190 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68044.log 2024-04-26T23:11:03.191 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43768.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43768.log.gz 2024-04-26T23:11:03.193 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68044.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68044.log.gz 2024-04-26T23:11:03.193 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44608.log 2024-04-26T23:11:03.195 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43726.log 2024-04-26T23:11:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44608.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44608.log.gz 2024-04-26T23:11:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49936.log 2024-04-26T23:11:03.198 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43726.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43726.log.gz 2024-04-26T23:11:03.199 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57537.log 2024-04-26T23:11:03.200 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49936.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49936.log.gz 2024-04-26T23:11:03.201 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56829.log 2024-04-26T23:11:03.202 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.57537.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.57537.log.gz 2024-04-26T23:11:03.203 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.56829.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.68677.log 2024-04-26T23:11:03.203 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56829.log.gz 2024-04-26T23:11:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48217.log 2024-04-26T23:11:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68677.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68677.log.gz 2024-04-26T23:11:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60389.log 2024-04-26T23:11:03.208 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48217.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48217.log.gz 2024-04-26T23:11:03.210 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60389.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60389.log.gz 2024-04-26T23:11:03.210 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68292.log 2024-04-26T23:11:03.212 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45119.log 2024-04-26T23:11:03.214 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68292.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68292.log.gz 2024-04-26T23:11:03.214 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41390.log 2024-04-26T23:11:03.215 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45119.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45119.log.gz 2024-04-26T23:11:03.216 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66913.log 2024-04-26T23:11:03.217 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41390.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41390.log.gz 2024-04-26T23:11:03.218 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72501.log 2024-04-26T23:11:03.219 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66913.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66913.log.gz 2024-04-26T23:11:03.220 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54651.log 2024-04-26T23:11:03.221 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72501.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72501.log.gz 2024-04-26T23:11:03.222 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62707.log 2024-04-26T23:11:03.222 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54651.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54651.log.gz 2024-04-26T23:11:03.223 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64087.log 2024-04-26T23:11:03.225 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62707.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62707.log.gz 2024-04-26T23:11:03.225 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.64087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60935.log 2024-04-26T23:11:03.226 INFO:teuthology.orchestra.run.smithi040.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64087.log.gz 2024-04-26T23:11:03.227 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47559.log 2024-04-26T23:11:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60935.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60935.log.gz 2024-04-26T23:11:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.47559.log: -- /var/log/ceph/ceph-client.admin.66228.log 2024-04-26T23:11:03.230 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47559.log.gz 2024-04-26T23:11:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54735.log 2024-04-26T23:11:03.233 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66228.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66228.log.gz 2024-04-26T23:11:03.233 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.54735.log: -- /var/log/ceph/ceph-client.admin.59327.log 2024-04-26T23:11:03.234 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54735.log.gz 2024-04-26T23:11:03.235 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54294.log 2024-04-26T23:11:03.237 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59327.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59327.log.gz 2024-04-26T23:11:03.238 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46045.log/var/log/ceph/ceph-client.admin.54294.log: 2024-04-26T23:11:03.238 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54294.log.gz 2024-04-26T23:11:03.239 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61688.log 2024-04-26T23:11:03.240 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46045.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46045.log.gz 2024-04-26T23:11:03.242 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.61688.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50492.log 2024-04-26T23:11:03.242 INFO:teuthology.orchestra.run.smithi040.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.61688.log.gz 2024-04-26T23:11:03.243 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53334.log 2024-04-26T23:11:03.245 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50492.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50492.log.gz 2024-04-26T23:11:03.246 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53754.log/var/log/ceph/ceph-client.admin.53334.log: 2024-04-26T23:11:03.246 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53334.log.gz 2024-04-26T23:11:03.247 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43285.log 2024-04-26T23:11:03.249 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53754.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.53754.log.gz 2024-04-26T23:11:03.250 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40742.log 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43285.log.gz 2024-04-26T23:11:03.250 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-26T23:11:03.252 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70468.log 2024-04-26T23:11:03.253 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40742.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40742.log.gz 2024-04-26T23:11:03.254 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64279.log 2024-04-26T23:11:03.255 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.70468.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.70468.log.gz 2024-04-26T23:11:03.257 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.64279.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64279.log.gz 2024-04-26T23:11:03.257 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65471.log 2024-04-26T23:11:03.259 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69744.log 2024-04-26T23:11:03.261 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.65471.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.65471.log.gz 2024-04-26T23:11:03.262 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41662.log 2024-04-26T23:11:03.262 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.69744.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.69744.log.gz 2024-04-26T23:11:03.263 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55505.log 2024-04-26T23:11:03.265 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41662.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41662.log.gz 2024-04-26T23:11:03.266 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44314.log 2024-04-26T23:11:03.266 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55505.log.gz 2024-04-26T23:11:03.267 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69964.log 2024-04-26T23:11:03.268 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44314.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44314.log.gz 2024-04-26T23:11:03.269 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.69964.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42566.log 2024-04-26T23:11:03.270 INFO:teuthology.orchestra.run.smithi040.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.69964.log.gz 2024-04-26T23:11:03.271 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46465.log 2024-04-26T23:11:03.272 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42566.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42566.log.gz 2024-04-26T23:11:03.273 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56325.log/var/log/ceph/ceph-client.admin.46465.log: 2024-04-26T23:11:03.274 INFO:teuthology.orchestra.run.smithi040.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46465.log.gz 2024-04-26T23:11:03.275 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64600.log 2024-04-26T23:11:03.276 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56325.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56325.log.gz 2024-04-26T23:11:03.277 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.64600.log: --verbose -- /var/log/ceph/ceph-client.admin.55071.log 2024-04-26T23:11:03.277 INFO:teuthology.orchestra.run.smithi040.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64600.log.gz 2024-04-26T23:11:03.279 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48025.log 2024-04-26T23:11:03.280 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55071.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55071.log.gz 2024-04-26T23:11:03.281 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.48025.log: -- /var/log/ceph/ceph-client.admin.50366.log 2024-04-26T23:11:03.282 INFO:teuthology.orchestra.run.smithi040.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48025.log.gz 2024-04-26T23:11:03.283 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63632.log 2024-04-26T23:11:03.284 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50366.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50366.log.gz 2024-04-26T23:11:03.285 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41315.log 2024-04-26T23:11:03.285 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63632.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63632.log.gz 2024-04-26T23:11:03.287 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69826.log 2024-04-26T23:11:03.288 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41315.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41315.log.gz 2024-04-26T23:11:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47318.log 2024-04-26T23:11:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.69826.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69826.log.gz 2024-04-26T23:11:03.290 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72627.log 2024-04-26T23:11:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47318.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47318.log.gz 2024-04-26T23:11:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.72627.log: -- /var/log/ceph/ceph-client.admin.57880.log 2024-04-26T23:11:03.293 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72627.log.gz 2024-04-26T23:11:03.294 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60599.log 2024-04-26T23:11:03.296 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.57880.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.57880.log.gz 2024-04-26T23:11:03.296 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59908.log 2024-04-26T23:11:03.298 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.60599.log.gz 2024-04-26T23:11:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69448.log 2024-04-26T23:11:03.299 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59908.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59908.log.gz 2024-04-26T23:11:03.300 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54987.log 2024-04-26T23:11:03.301 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.69448.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69448.log.gz 2024-04-26T23:11:03.302 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62497.log 2024-04-26T23:11:03.303 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54987.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54987.log.gz 2024-04-26T23:11:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62497.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62497.log.gz 2024-04-26T23:11:03.305 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67470.log 2024-04-26T23:11:03.307 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75010.log 2024-04-26T23:11:03.308 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67470.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67470.log.gz 2024-04-26T23:11:03.309 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72177.log 2024-04-26T23:11:03.310 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.75010.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75010.log.gz 2024-04-26T23:11:03.310 INFO:teuthology.orchestra.run.smithi040.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-26T23:11:03.310 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67291.log 2024-04-26T23:11:03.311 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72177.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72177.log.gz 2024-04-26T23:11:03.311 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38396.log 2024-04-26T23:11:03.312 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55846.log 2024-04-26T23:11:03.313 INFO:teuthology.orchestra.run.smithi040.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67291.log.gz 2024-04-26T23:11:03.314 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53688.log 2024-04-26T23:11:03.314 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38396.log.gz 2024-04-26T23:11:03.315 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55846.log: gzip -5 --verbose -- 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55846.log.gz 2024-04-26T23:11:03.315 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.46423.log 2024-04-26T23:11:03.316 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67207.log 2024-04-26T23:11:03.317 INFO:teuthology.orchestra.run.smithi040.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.53688.log.gz 2024-04-26T23:11:03.318 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46423.log: gzip -5 --verbose -- 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46423.log.gz 2024-04-26T23:11:03.318 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.40286.log 2024-04-26T23:11:03.319 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67207.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67207.log.gz 2024-04-26T23:11:03.319 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72837.log 2024-04-26T23:11:03.321 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39976.log 2024-04-26T23:11:03.321 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40286.log.gz 2024-04-26T23:11:03.322 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72837.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72837.log.gz 2024-04-26T23:11:03.323 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39976.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39976.log.gz 2024-04-26T23:11:03.324 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73364.log 2024-04-26T23:11:03.324 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67648.log 2024-04-26T23:11:03.326 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47066.log 2024-04-26T23:11:03.326 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73364.log.gz 2024-04-26T23:11:03.327 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71139.log 2024-04-26T23:11:03.327 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67648.log.gz 2024-04-26T23:11:03.328 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47066.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47066.log.gz 2024-04-26T23:11:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71139.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.71139.log.gz 2024-04-26T23:11:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66397.log 2024-04-26T23:11:03.332 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70766.log 2024-04-26T23:11:03.333 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58565.log 2024-04-26T23:11:03.334 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.66397.log.gz 2024-04-26T23:11:03.335 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.70766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74590.log 2024-04-26T23:11:03.335 INFO:teuthology.orchestra.run.smithi040.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.70766.log.gz 2024-04-26T23:11:03.336 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63382.log 2024-04-26T23:11:03.336 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58565.log.gz 2024-04-26T23:11:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74590.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74590.log.gz 2024-04-26T23:11:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73742.log 2024-04-26T23:11:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51375.log 2024-04-26T23:11:03.339 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63382.log.gz 2024-04-26T23:11:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73742.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73742.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68334.log 2024-04-26T23:11:03.340 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-26T23:11:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51375.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51375.log.gz 2024-04-26T23:11:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45457.log 2024-04-26T23:11:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74632.log 2024-04-26T23:11:03.343 INFO:teuthology.orchestra.run.smithi040.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68334.log.gz 2024-04-26T23:11:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38580.log 2024-04-26T23:11:03.343 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45457.log.gz 2024-04-26T23:11:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74632.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74632.log.gz 2024-04-26T23:11:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73864.log 2024-04-26T23:11:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59079.log 2024-04-26T23:11:03.346 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38580.log.gz 2024-04-26T23:11:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57769.log 2024-04-26T23:11:03.348 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73864.log.gz 2024-04-26T23:11:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59079.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59079.log.gz 2024-04-26T23:11:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.57769.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57769.log.gz 2024-04-26T23:11:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40868.log 2024-04-26T23:11:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62277.log 2024-04-26T23:11:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42464.log 2024-04-26T23:11:03.353 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40868.log.gz 2024-04-26T23:11:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62277.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62277.log.gz 2024-04-26T23:11:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46736.log 2024-04-26T23:11:03.355 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45715.log 2024-04-26T23:11:03.355 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42464.log.gz 2024-04-26T23:11:03.356 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33268.log 2024-04-26T23:11:03.357 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46736.log.gz 2024-04-26T23:11:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45715.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45715.log.gz 2024-04-26T23:11:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73102.log 2024-04-26T23:11:03.359 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.33268.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.65637.log 2024-04-26T23:11:03.359 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33268.log.gz 2024-04-26T23:11:03.360 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73102.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.73102.log.gz -5 2024-04-26T23:11:03.360 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55680.log 2024-04-26T23:11:03.362 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.65637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61073.log 2024-04-26T23:11:03.362 INFO:teuthology.orchestra.run.smithi040.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.65637.log.gz 2024-04-26T23:11:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55680.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.55680.log.gz -5 2024-04-26T23:11:03.363 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47815.log 2024-04-26T23:11:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.61073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66102.log 2024-04-26T23:11:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47815.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.61073.log.gz 2024-04-26T23:11:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63123.log 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47815.log.gz 2024-04-26T23:11:03.366 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-26T23:11:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66102.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66102.log.gz 2024-04-26T23:11:03.368 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39061.log 2024-04-26T23:11:03.368 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67249.log 2024-04-26T23:11:03.369 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63123.log.gz 2024-04-26T23:11:03.370 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66955.log 2024-04-26T23:11:03.370 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39061.log.gz 2024-04-26T23:11:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33368.log 2024-04-26T23:11:03.371 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67249.log.gz 2024-04-26T23:11:03.372 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71463.log 2024-04-26T23:11:03.372 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66955.log.gz 2024-04-26T23:11:03.374 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33368.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33368.log.gz 2024-04-26T23:11:03.375 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.71463.log: --verbose -- /var/log/ceph/ceph-client.admin.55295.log 2024-04-26T23:11:03.375 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71463.log.gz 2024-04-26T23:11:03.376 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64549.log 2024-04-26T23:11:03.377 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55295.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40784.log 2024-04-26T23:11:03.377 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55295.log.gz 2024-04-26T23:11:03.379 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.64549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73906.log 2024-04-26T23:11:03.379 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64549.log.gz 2024-04-26T23:11:03.380 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40784.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40784.log.gz 2024-04-26T23:11:03.380 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56559.log 2024-04-26T23:11:03.381 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73906.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73906.log.gz 2024-04-26T23:11:03.381 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72069.log 2024-04-26T23:11:03.383 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.56559.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56451.log 2024-04-26T23:11:03.383 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56559.log.gz 2024-04-26T23:11:03.384 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72069.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.72069.log.gz 2024-04-26T23:11:03.385 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45210.log 2024-04-26T23:11:03.385 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56451.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56451.log.gz 2024-04-26T23:11:03.386 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73574.log 2024-04-26T23:11:03.387 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43133.log 2024-04-26T23:11:03.387 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45210.log.gz 2024-04-26T23:11:03.389 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69280.log 2024-04-26T23:11:03.389 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73574.log.gz 2024-04-26T23:11:03.389 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.43133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43133.log.gz 2024-04-26T23:11:03.390 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52376.log 2024-04-26T23:11:03.391 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.69280.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47647.log 2024-04-26T23:11:03.392 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69280.log.gz 2024-04-26T23:11:03.392 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52376.log: gzip -5 89.2% --verbose -- /var/log/ceph/ceph-client.admin.63207.log 2024-04-26T23:11:03.393 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52376.log.gz 2024-04-26T23:11:03.393 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47647.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47647.log.gz 2024-04-26T23:11:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63207.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63207.log.gz 2024-04-26T23:11:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50686.log 2024-04-26T23:11:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67554.log 2024-04-26T23:11:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56241.log 2024-04-26T23:11:03.398 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50686.log.gz 2024-04-26T23:11:03.399 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58362.log 2024-04-26T23:11:03.399 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.67554.log.gz 2024-04-26T23:11:03.400 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58869.log 2024-04-26T23:11:03.400 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56241.log.gz 2024-04-26T23:11:03.401 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58362.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.58362.log.gz 2024-04-26T23:11:03.402 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44356.log 2024-04-26T23:11:03.402 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58869.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58869.log.gz 2024-04-26T23:11:03.403 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48583.log 2024-04-26T23:11:03.404 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44356.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44356.log.gz -5 2024-04-26T23:11:03.404 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56367.log 2024-04-26T23:11:03.405 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68418.log 2024-04-26T23:11:03.405 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48583.log.gz 2024-04-26T23:11:03.406 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56367.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39892.log 2024-04-26T23:11:03.407 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56367.log.gz 2024-04-26T23:11:03.407 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.68418.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.68418.log.gz 2024-04-26T23:11:03.408 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66144.log 2024-04-26T23:11:03.409 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41038.log 2024-04-26T23:11:03.410 INFO:teuthology.orchestra.run.smithi040.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39892.log.gz 2024-04-26T23:11:03.410 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73322.log 2024-04-26T23:11:03.410 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66144.log.gz 2024-04-26T23:11:03.411 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41038.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41038.log.gz 2024-04-26T23:11:03.412 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42170.log 2024-04-26T23:11:03.413 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.73322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44188.log 2024-04-26T23:11:03.413 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73322.log.gz 2024-04-26T23:11:03.415 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50324.log 2024-04-26T23:11:03.415 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42170.log.gz 2024-04-26T23:11:03.415 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44188.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44188.log.gz 2024-04-26T23:11:03.416 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42338.log 2024-04-26T23:11:03.417 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40202.log 2024-04-26T23:11:03.417 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50324.log.gz 2024-04-26T23:11:03.418 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42338.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42338.log.gz 2024-04-26T23:11:03.419 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40202.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40202.log.gz 2024-04-26T23:11:03.419 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65928.log 2024-04-26T23:11:03.420 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65545.log 2024-04-26T23:11:03.422 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.65928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74404.log 2024-04-26T23:11:03.422 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.65928.log.gz 2024-04-26T23:11:03.423 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.65545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50198.log 2024-04-26T23:11:03.423 INFO:teuthology.orchestra.run.smithi040.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.65545.log.gz 2024-04-26T23:11:03.424 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74404.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74404.log.gz 2024-04-26T23:11:03.425 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71937.log 2024-04-26T23:11:03.425 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50198.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50198.log.gz 2024-04-26T23:11:03.426 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-26T23:11:03.427 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59037.log 2024-04-26T23:11:03.428 INFO:teuthology.orchestra.run.smithi040.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.71937.log.gz 2024-04-26T23:11:03.430 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.59037.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59037.log.gz 2024-04-26T23:11:03.430 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40060.log 2024-04-26T23:11:03.432 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67918.log 2024-04-26T23:11:03.433 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40060.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40060.log.gz 2024-04-26T23:11:03.434 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74312.log 2024-04-26T23:11:03.435 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67918.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67918.log.gz 2024-04-26T23:11:03.436 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72543.log/var/log/ceph/ceph-client.admin.74312.log: 2024-04-26T23:11:03.436 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74312.log.gz 2024-04-26T23:11:03.437 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53538.log 2024-04-26T23:11:03.438 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72543.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72543.log.gz 2024-04-26T23:11:03.439 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64753.log 2024-04-26T23:11:03.440 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53538.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.53538.log.gz 2024-04-26T23:11:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.64753.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.64753.log.gz 2024-04-26T23:11:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67690.log 2024-04-26T23:11:03.445 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65994.log 2024-04-26T23:11:03.446 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.67690.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67690.log.gz 2024-04-26T23:11:03.446 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62997.log 2024-04-26T23:11:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.65994.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.65994.log.gz 2024-04-26T23:11:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60665.log 2024-04-26T23:11:03.449 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62997.log.gz 2024-04-26T23:11:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60665.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60665.log.gz 2024-04-26T23:11:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64213.log 2024-04-26T23:11:03.453 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42608.log 2024-04-26T23:11:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.64213.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.64213.log.gz 2024-04-26T23:11:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45961.log 2024-04-26T23:11:03.455 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42608.log.gz 2024-04-26T23:11:03.457 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41378.log 2024-04-26T23:11:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45961.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45961.log.gz 2024-04-26T23:11:03.459 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41378.log: -- /var/log/ceph/ceph-client.admin.45541.log 2024-04-26T23:11:03.459 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41378.log.gz 2024-04-26T23:11:03.460 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62109.log 2024-04-26T23:11:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45541.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45541.log.gz 2024-04-26T23:11:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51209.log/var/log/ceph/ceph-client.admin.62109.log: 2024-04-26T23:11:03.463 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62109.log.gz 2024-04-26T23:11:03.464 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59369.log 2024-04-26T23:11:03.465 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51209.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51209.log.gz 2024-04-26T23:11:03.467 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41313.log 2024-04-26T23:11:03.467 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59369.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59369.log.gz 2024-04-26T23:11:03.468 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69238.log 2024-04-26T23:11:03.469 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41313.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41313.log.gz 2024-04-26T23:11:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64687.log/var/log/ceph/ceph-client.admin.69238.log: 2024-04-26T23:11:03.471 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69238.log.gz 2024-04-26T23:11:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72435.log 2024-04-26T23:11:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.64687.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.64687.log.gz 2024-04-26T23:11:03.474 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54168.log 2024-04-26T23:11:03.475 INFO:teuthology.orchestra.run.smithi040.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.72435.log.gz 2024-04-26T23:11:03.476 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48538.log 2024-04-26T23:11:03.477 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54168.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54168.log.gz 2024-04-26T23:11:03.478 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72795.log 2024-04-26T23:11:03.479 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48538.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48538.log.gz 2024-04-26T23:11:03.480 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59514.log 2024-04-26T23:11:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.72795.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72795.log.gz 2024-04-26T23:11:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58995.log 2024-04-26T23:11:03.482 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.59514.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59514.log.gz 2024-04-26T23:11:03.483 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63081.log 2024-04-26T23:11:03.484 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58995.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58995.log.gz 2024-04-26T23:11:03.485 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46778.log 2024-04-26T23:11:03.486 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.63081.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63081.log.gz 2024-04-26T23:11:03.487 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49679.log 2024-04-26T23:11:03.488 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46778.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46778.log.gz 2024-04-26T23:11:03.490 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49679.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49679.log.gz 2024-04-26T23:11:03.490 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33000.log 2024-04-26T23:11:03.492 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54126.log 2024-04-26T23:11:03.494 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33000.log: 93.3% -- replaced with /var/log/ceph/ceph-client.admin.33000.log.gz 2024-04-26T23:11:03.494 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.54126.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.60431.log 2024-04-26T23:11:03.495 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54126.log.gz 2024-04-26T23:11:03.495 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62151.log 2024-04-26T23:11:03.498 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60431.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60431.log.gz 2024-04-26T23:11:03.498 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56111.log 2024-04-26T23:11:03.498 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62151.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62151.log.gz 2024-04-26T23:11:03.499 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38219.log 2024-04-26T23:11:03.501 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56111.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56111.log.gz 2024-04-26T23:11:03.501 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.38219.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.58649.log 2024-04-26T23:11:03.502 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38219.log.gz 2024-04-26T23:11:03.503 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72219.log 2024-04-26T23:11:03.505 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58649.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58649.log.gz 2024-04-26T23:11:03.505 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51290.log/var/log/ceph/ceph-client.admin.72219.log: 2024-04-26T23:11:03.507 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.72219.log.gz 2024-04-26T23:11:03.507 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49194.log 2024-04-26T23:11:03.508 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51290.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51290.log.gz 2024-04-26T23:11:03.510 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49194.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49194.log.gz 2024-04-26T23:11:03.511 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55253.log 2024-04-26T23:11:03.512 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70914.log 2024-04-26T23:11:03.513 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55253.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55253.log.gz 2024-04-26T23:11:03.514 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74362.log 2024-04-26T23:11:03.515 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.70914.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.70914.log.gz 2024-04-26T23:11:03.516 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61271.log 2024-04-26T23:11:03.517 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74362.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74362.log.gz 2024-04-26T23:11:03.518 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74842.log 2024-04-26T23:11:03.519 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.61271.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61271.log.gz 2024-04-26T23:11:03.519 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71739.log 2024-04-26T23:11:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.74842.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74842.log.gz 2024-04-26T23:11:03.521 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65305.log 2024-04-26T23:11:03.523 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71739.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.71739.log.gz 2024-04-26T23:11:03.524 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.65305.log: --verbose -- /var/log/ceph/ceph-client.admin.52999.log 2024-04-26T23:11:03.525 INFO:teuthology.orchestra.run.smithi040.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.65305.log.gz 2024-04-26T23:11:03.525 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59609.log 2024-04-26T23:11:03.527 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52999.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.52999.log.gz 2024-04-26T23:11:03.528 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.59609.log: --verbose -- /var/log/ceph/ceph-client.admin.66871.log 2024-04-26T23:11:03.528 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59609.log.gz 2024-04-26T23:11:03.529 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56747.log 2024-04-26T23:11:03.531 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66871.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66871.log.gz 2024-04-26T23:11:03.532 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50992.log 2024-04-26T23:11:03.533 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.56747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.56747.log.gz 2024-04-26T23:11:03.533 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52106.log 2024-04-26T23:11:03.534 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50992.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50992.log.gz 2024-04-26T23:11:03.535 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49445.log 2024-04-26T23:11:03.536 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52106.log.gz 2024-04-26T23:11:03.538 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49445.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.49445.log.gz 2024-04-26T23:11:03.538 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58785.log 2024-04-26T23:11:03.540 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47402.log 2024-04-26T23:11:03.541 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.58785.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58785.log.gz 2024-04-26T23:11:03.542 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41652.log 2024-04-26T23:11:03.543 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47402.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47402.log.gz 2024-04-26T23:11:03.543 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66997.log 2024-04-26T23:11:03.544 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41652.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41652.log.gz 2024-04-26T23:11:03.545 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60305.log 2024-04-26T23:11:03.547 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.66997.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66997.log.gz 2024-04-26T23:11:03.547 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46820.log 2024-04-26T23:11:03.548 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60305.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60305.log.gz 2024-04-26T23:11:03.549 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50576.log 2024-04-26T23:11:03.551 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46820.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46820.log.gz 2024-04-26T23:11:03.551 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43810.log 2024-04-26T23:11:03.552 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50576.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50576.log.gz 2024-04-26T23:11:03.553 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47360.log 2024-04-26T23:11:03.553 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43810.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43810.log.gz 2024-04-26T23:11:03.554 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55211.log 2024-04-26T23:11:03.555 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47360.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47360.log.gz 2024-04-26T23:11:03.556 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75095.log 2024-04-26T23:11:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55211.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55211.log.gz 2024-04-26T23:11:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75177.log 2024-04-26T23:11:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.75095.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75095.log.gz 2024-04-26T23:11:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.75177.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75177.log.gz 2024-04-26T23:11:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71871.log 2024-04-26T23:11:03.563 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65508.log 2024-04-26T23:11:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.71871.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.71871.log.gz 2024-04-26T23:11:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.65508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45499.log 2024-04-26T23:11:03.566 INFO:teuthology.orchestra.run.smithi040.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.65508.log.gz 2024-04-26T23:11:03.566 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53820.log 2024-04-26T23:11:03.568 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45499.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45499.log.gz 2024-04-26T23:11:03.570 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54210.log 2024-04-26T23:11:03.570 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.53820.log.gz 2024-04-26T23:11:03.571 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53065.log 2024-04-26T23:11:03.572 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54210.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54210.log.gz 2024-04-26T23:11:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.53065.log: /var/log/ceph/ceph-client.admin.49128.log 2024-04-26T23:11:03.574 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.53065.log.gz 2024-04-26T23:11:03.575 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62623.log 2024-04-26T23:11:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49128.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49128.log.gz 2024-04-26T23:11:03.577 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60215.log 2024-04-26T23:11:03.578 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62623.log.gz 2024-04-26T23:11:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56880.log 2024-04-26T23:11:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.60215.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60215.log.gz 2024-04-26T23:11:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.56880.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.62665.log 2024-04-26T23:11:03.581 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56880.log.gz 2024-04-26T23:11:03.582 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45841.log 2024-04-26T23:11:03.583 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.62665.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62665.log.gz 2024-04-26T23:11:03.584 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.45841.log: /var/log/ceph/ceph-client.admin.43852.log 2024-04-26T23:11:03.585 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45841.log.gz 2024-04-26T23:11:03.587 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43852.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43852.log.gz 2024-04-26T23:11:03.685 INFO:teuthology.orchestra.run.smithi040.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-26T23:11:03.744 INFO:teuthology.orchestra.run.smithi103.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-26T23:11:03.751 INFO:teuthology.orchestra.run.smithi103.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-26T23:11:03.778 INFO:teuthology.orchestra.run.smithi103.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-26T23:11:03.815 INFO:teuthology.orchestra.run.smithi103.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-26T23:11:03.819 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-26T23:11:03.819 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m1.289s 2024-04-26T23:11:03.819 INFO:teuthology.orchestra.run.smithi103.stderr:user 0m5.381s 2024-04-26T23:11:03.819 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m0.504s 2024-04-26T23:11:03.962 INFO:teuthology.orchestra.run.smithi040.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-26T23:11:04.019 INFO:teuthology.orchestra.run.smithi040.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-26T23:11:04.248 INFO:teuthology.orchestra.run.smithi040.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-26T23:11:04.374 INFO:teuthology.orchestra.run.smithi040.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-26T23:11:04.382 INFO:teuthology.orchestra.run.smithi040.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-26T23:11:04.386 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-26T23:11:04.386 INFO:teuthology.orchestra.run.smithi040.stderr:real 0m1.855s 2024-04-26T23:11:04.387 INFO:teuthology.orchestra.run.smithi040.stderr:user 0m7.195s 2024-04-26T23:11:04.387 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m2.356s 2024-04-26T23:11:04.388 INFO:tasks.ceph:Archiving logs... 2024-04-26T23:11:04.388 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279/remote/smithi040/log 2024-04-26T23:11:04.389 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T23:11:05.915 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279/remote/smithi103/log 2024-04-26T23:11:05.916 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T23:11:06.719 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T23:11:06.728 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T23:11:06.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T23:11:06.737 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T23:11:06.877 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T23:11:06.887 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T23:11:06.887 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T23:11:06.894 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T23:11:06.945 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-04-26T23:11:06.949 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-26T23:11:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T23:11:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-26T23:11:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv01.front.sepia.ceph.com 3 6 377 18 +1335us[+1334us] +/- 43ms 2024-04-26T23:11:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 21 +269us[ +268us] +/- 77ms 2024-04-26T23:11:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -3057us[-3058us] +/- 89ms 2024-04-26T23:11:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:11:06.959 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T23:11:06.959 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-26T23:11:06.959 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv01.front.sepia.ceph.com 3 6 377 21 +1339us[+1356us] +/- 43ms 2024-04-26T23:11:06.959 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 19 +305us[ +305us] +/- 77ms 2024-04-26T23:11:06.959 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -3037us[-3037us] +/- 89ms 2024-04-26T23:11:06.959 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:11:06.960 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T23:11:06.970 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T23:11:06.970 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T23:11:06.979 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T23:11:07.196 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T23:11:07.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T23:11:07.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:11:07.485 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T23:11:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:11:07.576 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T23:11:07.586 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T23:11:07.595 INFO:teuthology.task.internal:Duration was 1121.049258 seconds 2024-04-26T23:11:07.595 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T23:11:07.604 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T23:11:07.604 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T23:11:07.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T23:11:07.741 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T23:11:07.765 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T23:11:08.291 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T23:11:08.291 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:11:08.291 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T23:11:08.361 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T23:11:08.362 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T23:11:08.438 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T23:11:08.438 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T23:11:08.446 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T23:11:08.534 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T23:11:08.534 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T23:11:08.541 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T23:11:08.740 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T23:11:08.756 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T23:11:08.756 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T23:11:08.846 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T23:11:08.930 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T23:11:08.941 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T23:11:08.949 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T23:11:09.040 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-04-26T23:11:09.047 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-04-26T23:11:09.092 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T23:11:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:11:09.144 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T23:11:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:11:09.199 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T23:11:09.210 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T23:11:09.210 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279/remote/smithi040 2024-04-26T23:11:09.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T23:11:09.312 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675279/remote/smithi103 2024-04-26T23:11:09.312 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T23:11:09.407 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T23:11:09.407 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T23:11:09.415 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T23:11:09.469 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T23:11:09.479 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T23:11:09.479 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T23:11:09.490 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T23:11:09.490 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T23:11:09.495 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T23:11:09.545 INFO:teuthology.orchestra.run.smithi040.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 23:11 /home/ubuntu/cephtest 2024-04-26T23:11:09.548 INFO:teuthology.orchestra.run.smithi103.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 23:11 /home/ubuntu/cephtest 2024-04-26T23:11:09.551 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T23:11:09.562 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T23:11:09.590 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 client.1-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/multimds_misc} duration: 1121.0492579936981 flavor: default owner: scheduled_rishabh@teuthology success: true 2024-04-26T23:11:09.590 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T23:11:09.676 INFO:teuthology.run:pass